[PATCH v2 00/22] Intel FPGA Device Drivers

[PATCH v2 00/22] Intel FPGA Device Drivers

[Wu Hao]

Hi All,

Here is v2 patch-series adding drivers for Intel FPGA devices.

The Intel FPGA driver provides interfaces for userspace applications to
configure, enumerate, open, and access FPGA accelerators on platforms
equipped with Intel(R) PCIe based FPGA solutions and enables system
level management functions such as FPGA partial reconfiguration, power
management and virtualization.

This patch series only adds the basic functions for FPGA accelerators and
partial reconfiguration. Patches for more functions, e.g power management
and virtualization, will be submitted after this series gets reviewed.

This patch series depends on Alan Tull's patch set v2 on FPGA region
support w/o device tree (see [1]).

Patch 1: add a document for Intel FPGA driver overview, including the HW
architecture, driver organization, device enumeration, virtualization and
opens.

Patch 2: introduce a fpga-dev class. It's used in below Intel FPGA PCIe
device driver, to represent a FPGA device on the system, and all actual
feature devices should be registered as child nodes of this container
fpga-dev device.

Patch 3: remove OF dependency for fpga bridge. allow fpga bridge class to
be used in non device tree case.

Patch 4: add region_id for fpga_image_info data structure, which allows
driver to pass region id information to fpga-mgr for FPGA reconfiguration
function.

Patch 5: add a 'status' sysfs interface to fpga-mgr class, it reflects
the status of the fpga-mgr including reconfiguration errors.

Patch 6-10: implement Intel FPGA PCIe device driver. It walks through the
'Device Feature List' in the PCI Bar, creates the container fpga-dev as
parent and platform devices as children for the feature devices it found.

Patch 11-14: implement Intel FPGA Management Engine (FME) driver. It's a
platform driver matching with the FME platform device created by above
PCIe driver. Sysfs and device file ioctls are exposed as user interfaces
to allow partial reconfiguration to Accelerated Function Units (AFUs) from
user space applications.

Patch 15-17: implement FPGA manager/bridge/region platform drivers for
Intel FPGA Management Engine (FME). These platform drivers match with
platform devices created by above FME driver, they use the generic
fpga-mgr/bridge/region class infrastructure to implement FPGA partial
reconfiguration function.

Patch 18-22: implement Intel FPGA Accelerated Function Unit (AFU) driver.
It's a platform driver matching with AFU platform device created by above
PCIe driver. It provides user interfaces to expose the AFU MMIO region,
map/unmap dma buffer, and control the port which AFU connects to.


Changes from v1:

- Use GPLv2 license instead of Dual BSD/GPL.
- Move the code to drivers/fpga folder.
- Update the intel-fpga.txt documentation for new driver organization.
- Add documentation for new sysfs interfacesa.
- Switch to use common fpga-region interface for partial reconfiguration
  (PR) function in FME. It creates fpga-region/fpga-mgr/fpga-bridge
  platform devices and leave the implementation to their platform drivers.
- Add platform drivers for FME fpga-mgr/bridge/region platform devices.
- Fix kbuild warnings, typos and other comments.

[1] http://marc.info/?l=linux-kernel&m=149269778606030&w=2

Kang Luwei (3):
  fpga: intel: add FPGA Management Engine driver basic framework
  fpga: intel: fme: add header sub feature support
  fpga: intel: fme: add partial reconfiguration sub feature support

Wu Hao (14):
  docs: fpga: add a document for Intel FPGA driver overview
  fpga: add FPGA device framework
  fpga: bridge: remove OF dependency for fpga-bridge
  fpga: mgr: add region_id to fpga_image_info
  fpga: mgr: add status for fpga-mgr
  fpga: intel: pcie: adds fpga_for_each_port callback for fme device
  fpga: intel: fme: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls
    support
  fpga: intel: add fpga manager platform driver for FME
  fpga: intel: add fpga bridge platform driver for FME
  fpga: intel: add fpga region platform driver for FME
  fpga: intel: add FPGA Accelerated Function Unit driver basic framework
  fpga: intel: afu: add header sub feature support
  fpga: intel: afu add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls
    support
  fpga: intel: afu: add FPGA_PORT_DMA_MAP/UNMAP ioctls support

Xiao Guangrong (4):
  fpga: intel: pcie: parse feature list and create platform device for
    features.
  fpga: intel: pcie: add chardev support for feature devices
  fpga: intel: add feature device infrastructure
  fpga: intel: afu: add user afu sub feature support

Zhang Yi (1):
  fpga: intel: add FPGA PCIe device driver

 Documentation/ABI/testing/sysfs-class-fpga-dev     |    5 +
 Documentation/ABI/testing/sysfs-class-fpga-manager |   10 +
 .../ABI/testing/sysfs-platform-intel-fpga-afu      |   16 +
 .../ABI/testing/sysfs-platform-intel-fpga-fme      |   19 +
 .../ABI/testing/sysfs-platform-intel-fpga-fme-mgr  |    8 +
 Documentation/fpga/intel-fpga.txt                  |  256 +++++
 Documentation/ioctl/ioctl-number.txt               |    1 +
 drivers/fpga/Kconfig                               |   73 +-
 drivers/fpga/Makefile                              |   16 +
 drivers/fpga/fpga-dev.c                            |  118 +++
 drivers/fpga/fpga-mgr.c                            |   24 +
 drivers/fpga/intel-afu-dma-region.c                |  372 ++++++++
 drivers/fpga/intel-afu-main.c                      |  469 +++++++++
 drivers/fpga/intel-afu-region.c                    |  127 +++
 drivers/fpga/intel-afu.h                           |   70 ++
 drivers/fpga/intel-feature-dev.c                   |  272 ++++++
 drivers/fpga/intel-feature-dev.h                   |  515 ++++++++++
 drivers/fpga/intel-fme-main.c                      |  265 ++++++
 drivers/fpga/intel-fme-pr.c                        |  494 ++++++++++
 drivers/fpga/intel-fme.h                           |   57 ++
 drivers/fpga/intel-fpga-fme-br.c                   |   77 ++
 drivers/fpga/intel-fpga-fme-mgr.c                  |  307 ++++++
 drivers/fpga/intel-fpga-fme-region.c               |   91 ++
 drivers/fpga/intel-pcie.c                          | 1003 ++++++++++++++++++++
 include/linux/fpga/fpga-dev.h                      |   31 +
 include/linux/fpga/fpga-mgr.h                      |   10 +
 include/uapi/linux/intel-fpga.h                    |  175 ++++
 27 files changed, 4880 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/ABI/testing/sysfs-class-fpga-dev
 create mode 100644 Documentation/ABI/testing/sysfs-platform-intel-fpga-afu
 create mode 100644 Documentation/ABI/testing/sysfs-platform-intel-fpga-fme
 create mode 100644 Documentation/ABI/testing/sysfs-platform-intel-fpga-fme-mgr
 create mode 100644 Documentation/fpga/intel-fpga.txt
 create mode 100644 drivers/fpga/fpga-dev.c
 create mode 100644 drivers/fpga/intel-afu-dma-region.c
 create mode 100644 drivers/fpga/intel-afu-main.c
 create mode 100644 drivers/fpga/intel-afu-region.c
 create mode 100644 drivers/fpga/intel-afu.h
 create mode 100644 drivers/fpga/intel-feature-dev.c
 create mode 100644 drivers/fpga/intel-feature-dev.h
 create mode 100644 drivers/fpga/intel-fme-main.c
 create mode 100644 drivers/fpga/intel-fme-pr.c
 create mode 100644 drivers/fpga/intel-fme.h
 create mode 100644 drivers/fpga/intel-fpga-fme-br.c
 create mode 100644 drivers/fpga/intel-fpga-fme-mgr.c
 create mode 100644 drivers/fpga/intel-fpga-fme-region.c
 create mode 100644 drivers/fpga/intel-pcie.c
 create mode 100644 include/linux/fpga/fpga-dev.h
 create mode 100644 include/uapi/linux/intel-fpga.h

-- 
1.8.3.1

[PATCH v2 01/22] docs: fpga: add a document for Intel FPGA driver overview

[Wu Hao]

Add a document for Intel FPGA driver overview.

Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
----
v2: added FME fpga-mgr/bridge/region platform driver to driver organization.
    updated open discussion per current implementation.
    fixed some typos.
---
 Documentation/fpga/intel-fpga.txt | 256 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 256 insertions(+)
 create mode 100644 Documentation/fpga/intel-fpga.txt

diff --git a/Documentation/fpga/intel-fpga.txt b/Documentation/fpga/intel-fpga.txt
new file mode 100644
index 0000000..4a29470
--- /dev/null
+++ b/Documentation/fpga/intel-fpga.txt
@@ -0,0 +1,256 @@
+===============================================================================
+                    Intel FPGA driver Overview
+-------------------------------------------------------------------------------
+                Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+                Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
+                Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+
+The Intel FPGA driver provides interfaces for userspace applications to
+configure, enumerate, open, and access FPGA accelerators on platforms equipped
+with Intel(R) FPGA PCIe based solutions and enables system level management
+functions such as FPGA reconfiguration, power management, and virtualization.
+
+HW Architecture
+===============
+From the OS's point of view, the FPGA hardware appears as a regular PCIe device.
+The FPGA device memory is organized using a predefined data structure (Device
+Feature List). Features supported by the particular FPGA device are exposed
+through these data structures, as illustrated below:
+
+  +-------------------------------+  +-------------+
+  |              PF               |  |     VF      |
+  +-------------------------------+  +-------------+
+      ^            ^         ^              ^
+      |            |         |              |
++-----|------------|---------|--------------|-------+
+|     |            |         |              |       |
+|  +-----+     +-------+ +-------+      +-------+   |
+|  | FME |     | Port0 | | Port1 |      | Port2 |   |
+|  +-----+     +-------+ +-------+      +-------+   |
+|                  ^         ^              ^       |
+|                  |         |              |       |
+|              +-------+ +------+       +-------+   |
+|              |  AFU  | |  AFU |       |  AFU  |   |
+|              +-------+ +------+       +-------+   |
+|                                                   |
+|                 FPGA PCIe Device                  |
++---------------------------------------------------+
+
+The driver supports PCIe SR-IOV to create virtual functions (VFs) which can be
+used to assign individual accelerators to virtual machines.
+
+FME (FPGA Management Engine)
+============================
+The FPGA Management Engine performs power and thermal management, error
+reporting, reconfiguration, performance reporting, and other infrastructure
+functions. Each FPGA has one FME, which is always accessed through the physical
+function (PF).
+
+User-space applications can acquire exclusive access to the FME using open(),
+and release it using close().
+
+The following functions are exposed through ioctls:
+
+	Get driver API version (FPGA_GET_API_VERSION)
+	Check for extensions (FPGA_CHECK_EXTENSION)
+	Assign port to PF (FPGA_FME_PORT_ASSIGN)
+	Release port from PF (FPGA_FME_PORT_RELEASE)
+	Program bitstream (FPGA_FME_PORT_PR)
+
+More functions are exposed through sysfs
+(/sys/class/fpga/fpga.n/intel-fpga-fme.n/):
+
+	Read bitstream ID (bitstream_id)
+	Read bitstream metadata (bitstream_metadata)
+	Read number of ports (ports_num)
+	Read socket ID (socket_id)
+	Read performance counters (perf/)
+	Power management (power_mgmt/)
+	Thermal management (thermal_mgmt/)
+	Error reporting (errors/)
+
+PORT
+====
+A port represents the interface between the static FPGA fabric (the "blue
+bitstream") and a partially reconfigurable region containing an AFU (the "green
+bitstream"). It controls the communication from SW to the accelerator and
+exposes features such as reset and debug.
+
+A PCIe device may have several ports and each port can be released from PF by
+FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe sriov
+sysfs interface.
+
+AFU
+===
+An AFU is attached to a port and exposes a 256k MMIO region to be used for
+accelerator-specific control registers.
+
+User-space applications can acquire exclusive access to an AFU attached to a
+port by using open() on the port device node, and release it using close().
+
+The following functions are exposed through ioctls:
+
+	Get driver API version (FPGA_GET_API_VERSION)
+	Check for extensions (FPGA_CHECK_EXTENSION)
+	Get port info (FPGA_PORT_GET_INFO)
+	Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
+	Map DMA buffer (FPGA_PORT_DMA_MAP)
+	Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
+	Reset AFU (FPGA_PORT_RESET)
+	Enable UMsg (FPGA_PORT_UMSG_ENABLE)
+	Disable UMsg (FPGA_PORT_UMSG_DISABLE)
+	Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
+	Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)
+
+User-space applications can also mmap() accelerator MMIO regions.
+
+More functions are exposed through sysfs:
+(/sys/class/fpga/fpga.n/intel-fpga-port.m/):
+
+	Read Accelerator GUID (afu_id)
+	Error reporting (errors/)
+
+Partial Reconfiguration
+=======================
+As mentioned above, accelerators can be reconfigured through partial
+reconfiguration of a green bitstream file (GBS). The green bitstream must have
+been generated for the exact blue bitstream and targeted reconfigurable region
+(port) of the FPGA; otherwise, the reconfiguration operation will fail and
+possibly cause system instability. This compatibility can be checked by
+comparing the interface ID noted in the GBS header against the interface ID
+exposed by the FME through sysfs (see above). This check is usually done by
+user-space before calling the reconfiguration IOCTL.
+
+FPGA virtualization
+===================
+To enable accessing an accelerator from applications running in a VM, the
+respective AFU's port needs to be assigned to a VF using the following steps:
+
+ a) The PF owns all AFU ports by default. Any port that needs to be reassigned
+ to a VF must be released from PF firstly through the FPGA_FME_PORT_RELEASE
+ ioctl on the FME device.
+
+ b) Once N ports are released from PF, then user can use below command to
+ enable SRIOV and VFs. Each VF owns only one Port with AFU.
+
+ echo N > $PCI_DEVICE_PATH/sriov_numvfs
+
+ c) Pass through the VFs to VMs
+
+ d) The AFU under VF is accessible from applications in VM (using the same
+ driver inside the VF).
+
+Note the an FME can't be assigned to a VF, thus PR and other management
+functions are only available via the PF.
+
+
+Driver organization
+===================
+
+  +-------++------++------+             |
+  | FME   || FME  || FME  |             |
+  | FPGA  || FPGA || FPGA |             |
+  |Manager||Bridge||Region|             |
+  +-------++------++------+             |
+  +-----------------------+  +--------+ |             +--------+
+  |          FME          |  |  AFU   | |             |  AFU   |
+  |         Module        |  | Module | |             | Module |
+  +-----------------------+  +--------+ |             +--------+
+        +-----------------------+       |      +-----------------------+
+        | FPGA Container Device |       |      | FPGA Container Device |
+        +-----------------------+       |      +-----------------------+
+          +------------------+          |         +------------------+
+          | FPGA PCIE Module |          | Virtual | FPGA PCIE Module |
+          +------------------+   Host   | Machine +------------------+
+ -------------------------------------- | ------------------------------
+           +---------------+            |          +---------------+
+           | PCI PF Device |            |          | PCI VF Device |
+           +---------------+            |          +---------------+
+
+The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe device
+driver is always loaded first once a FPGA PCIE PF or VF device is detected. This
+driver plays an infrastructural role in the driver architecture.  It:
+
+	a) creates FPGA container device as parent of the feature devices.
+	b) walks through the Device Feature List, which is implemented in PCIE
+	   device BAR memory, to discover feature devices and their sub features
+	   and create platform device for them under the container device.
+	c) supports SRIOV.
+	d) introduces the feature device infrastructure, which abstracts
+	   operations for sub features and exposes common functions to feature
+	   device drivers.
+
+The FPGA Management Engine (FME) driver is a platform driver which is loaded
+automatically after FME platform device creation from the PCIE driver. It
+provides the key features for FPGA management, including:
+
+	a) Power and thermal management, error reporting, performance reporting
+	   and other infrastructure functions. Users can access these functions
+	   via sysfs interfaces exposed by FME driver.
+	b) Partial Reconfiguration. The FME driver creates platform devices
+	   for FPGA manager, FPGA bridges and FPGA regions during PR sub
+	   feature initialization; Once it receives an FPGA_FME_PORT_PR ioctl
+	   from user, it invokes the common interface function from FPGA Region
+	   to complete the partial reconfiguration of the bitstream to the given
+	   port.
+	c) Port management for virtualization. The FME driver introduces two
+	   ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
+	   FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the port is
+	   released from the PF, it can be assigned to the VF through the SRIOV
+	   interfaces provided by PCIE driver. (Refer to "FPGA virtualization"
+	   for more details).
+
+Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) driver
+is probed once the AFU platform device is created. The main function of this
+module is to provide an interface for userspace applications to access the
+individual accelerators, including basic reset control on port, AFU MMIO region
+export, dma buffer mapping service, UMsg notification, and remote debug
+functions (see above).
+
+
+Device enumeration
+==================
+This section introduces how applications enumerate the fpga device from
+the sysfs hierarchy under /sys/class/fpga.
+
+In the example below, two Intel(R) FPGA devices are installed in the host. Each
+fpga device has one FME and two ports (AFUs).
+
+For each FPGA device, a device director is created under /sys/class/fpga/:
+
+	/sys/class/fpga/fpga.0
+	/sys/class/fpga/fpga.1
+
+The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's name.
+Application can retrieve name information via the sysfs interface:
+
+	/sys/class/fpga/fpga.0/name
+
+Each node has one FME and two ports (AFUs) as child devices:
+
+	/sys/class/fpga/fpga.0/intel-fpga-fme.0
+	/sys/class/fpga/fpga.0/intel-fpga-port.0
+	/sys/class/fpga/fpga.0/intel-fpga-port.1
+
+	/sys/class/fpga/fpga.1/intel-fpga-fme.1
+	/sys/class/fpga/fpga.1/intel-fpga-port.2
+	/sys/class/fpga/fpga.1/intel-fpga-port.3
+
+In general, the FME/AFU sysfs interfaces are named as follows:
+
+	/sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/
+	/sys/class/fpga/<fpga.n>/<intel-fpga-port.m>/
+
+with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
+ports.
+
+The device nodes used for ioctl() or mmap() can be referenced through:
+
+	/sys/class/fpga/<fpga.n>/<intel-fpga-port.n>/dev
+	/sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/dev
+
+Open discussion
+===============
+FME driver exports one ioctl (FPGA_FME_PORT_PR) for partial reconfiguration to
+user now. In the future, if unified user interfaces for reconfiguration are
+added, FME driver should switch to them from ioctl interface.
-- 
1.8.3.1

[PATCH v2 02/22] fpga: add FPGA device framework

[Wu Hao]

During FPGA device (e.g PCI-based) discovery, platform devices are
registered for different FPGA function units. But the device node path
isn't quite friendly to applications.

Consider this case, applications want to access child device's sysfs file
for some information.

1) Access using bus-based path (e.g PCI)

  /sys/bus/pci/devices/xxxxx/fpga_func_a.0/sysfs_file

  From the path, it's clear which PCI device is the parent, but not perfect
  solution for applications. PCI device BDF is not fixed, application may
  need to search all PCI device to find the actual FPGA Device.

2) Or access using platform device path

  /sys/bus/platform/devices/fpga_func_a.0/sysfs_file

  Applications find the actual function by name easily, but no information
  about which fpga device it belongs to. It's quite confusing if multiple
  FPGA devices are in one system.

'FPGA Device' class is introduced to resolve this problem. Each node under
this class represents a fpga device, which may have one or more child
devices. Applications only need to search under this FPGA Device class
folder to find the child device node it needs.

For example, for the platform has 2 fpga devices, each fpga device has
3 child devices, the hierarchy looks like this.

Two nodes are under /sys/class/fpga/:
/sys/class/fpga/fpga.0
/sys/class/fpga/fpga.1

Each node has 1 function A device and 2 function B devices:
/sys/class/fpga/fpga.0/func_a.0
/sys/class/fpga/fpga.0/func_b.0
/sys/class/fpga/fpga.0/func_b.1

/sys/class/fpga/fpga.1/func_a.1
/sys/class/fpga/fpga.1/func_b.2
/sys/class/fpga/fpga.1/func_b.3

This following APIs are provided by FPGA device framework:
* fpga_dev_create
  Create fpga device under the given parent device.
* fpga_dev_destroy
  Destroy fpga device

The following sysfs files are created:
* /sys/class/fpga/<fpga.x>/name
  Name of the fpga device.

Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
v2: added sysfs documentation for fpga-dev class.
    switched to GPLv2 license.
---
 Documentation/ABI/testing/sysfs-class-fpga-dev |   5 ++
 drivers/fpga/Kconfig                           |   6 ++
 drivers/fpga/Makefile                          |   3 +
 drivers/fpga/fpga-dev.c                        | 118 +++++++++++++++++++++++++
 include/linux/fpga/fpga-dev.h                  |  31 +++++++
 5 files changed, 163 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-class-fpga-dev
 create mode 100644 drivers/fpga/fpga-dev.c
 create mode 100644 include/linux/fpga/fpga-dev.h

diff --git a/Documentation/ABI/testing/sysfs-class-fpga-dev b/Documentation/ABI/testing/sysfs-class-fpga-dev
new file mode 100644
index 0000000..ce162df
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-class-fpga-dev
@@ -0,0 +1,5 @@
+What:		/sys/class/fpga/<device>/name
+Date:		June 2017
+KernelVersion:	4.12
+Contact:	Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+Description:	Name of FPGA device
diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 394c141..ed600d5 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig
@@ -12,6 +12,12 @@ config FPGA
 	  manager drivers.
 
 if FPGA
+config FPGA_DEVICE
+	tristate "FPGA Device Framework"
+	help
+	  Say Y here if you want support for FPGA Devices from the kernel.
+	  The FPGA Device Framework adds a FPGA device class and provide
+	  interfaces to create FPGA devices.
 
 config FPGA_REGION
 	tristate "FPGA Region"
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 85b45d0..8950a8f 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -5,6 +5,9 @@
 # Core FPGA Manager Framework
 obj-$(CONFIG_FPGA)			+= fpga-mgr.o
 
+# FPGA Device Framework
+obj-$(CONFIG_FPGA_DEVICE)		+= fpga-dev.o
+
 # FPGA Manager Drivers
 obj-$(CONFIG_FPGA_MGR_ICE40_SPI)	+= ice40-spi.o
 obj-$(CONFIG_FPGA_MGR_SOCFPGA)		+= socfpga.o
diff --git a/drivers/fpga/fpga-dev.c b/drivers/fpga/fpga-dev.c
new file mode 100644
index 0000000..5778ebc
--- /dev/null
+++ b/drivers/fpga/fpga-dev.c
@@ -0,0 +1,118 @@
+/*
+ * FPGA Device Framework Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * This work is licensed under the terms of the GNU GPL version 2. See
+ * the COPYING file in the top-level directory.
+ */
+#include <linux/device.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/fpga/fpga-dev.h>
+
+static DEFINE_IDA(fpga_dev_ida);
+static struct class *fpga_dev_class;
+
+static ssize_t name_show(struct device *dev,
+			 struct device_attribute *attr, char *buf)
+{
+	struct fpga_dev *fdev = to_fpga_dev(dev);
+
+	return sprintf(buf, "%s\n", fdev->name);
+}
+static DEVICE_ATTR_RO(name);
+
+static struct attribute *fpga_dev_attrs[] = {
+	&dev_attr_name.attr,
+	NULL,
+};
+ATTRIBUTE_GROUPS(fpga_dev);
+
+/**
+ * fpga_dev_create - create a fpga device
+ * @parent: parent device
+ * @name: fpga device name
+ *
+ * Return fpga_dev struct for success, error code otherwise.
+ */
+struct fpga_dev *fpga_dev_create(struct device *parent, const char *name)
+{
+	struct fpga_dev *fdev;
+	int id, ret = 0;
+
+	if (!name || !strlen(name)) {
+		dev_err(parent, "Attempt to register with no name!\n");
+		return ERR_PTR(-EINVAL);
+	}
+
+	fdev = kzalloc(sizeof(*fdev), GFP_KERNEL);
+	if (!fdev)
+		return ERR_PTR(-ENOMEM);
+
+	id = ida_simple_get(&fpga_dev_ida, 0, 0, GFP_KERNEL);
+	if (id < 0) {
+		ret = id;
+		goto error_kfree;
+	}
+
+	fdev->name = name;
+
+	device_initialize(&fdev->dev);
+	fdev->dev.class = fpga_dev_class;
+	fdev->dev.parent = parent;
+	fdev->dev.id = id;
+
+	ret = dev_set_name(&fdev->dev, "fpga.%d", id);
+	if (ret)
+		goto error_device;
+
+	ret = device_add(&fdev->dev);
+	if (ret)
+		goto error_device;
+
+	dev_dbg(fdev->dev.parent, "fpga device [%s] created\n", fdev->name);
+
+	return fdev;
+
+error_device:
+	ida_simple_remove(&fpga_dev_ida, id);
+error_kfree:
+	kfree(fdev);
+
+	return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(fpga_dev_create);
+
+static void fpga_dev_release(struct device *dev)
+{
+	struct fpga_dev *fdev = to_fpga_dev(dev);
+
+	ida_simple_remove(&fpga_dev_ida, fdev->dev.id);
+	kfree(fdev);
+}
+
+static int __init fpga_dev_class_init(void)
+{
+	pr_info("FPGA Device framework\n");
+
+	fpga_dev_class = class_create(THIS_MODULE, "fpga");
+	if (IS_ERR(fpga_dev_class))
+		return PTR_ERR(fpga_dev_class);
+
+	fpga_dev_class->dev_groups = fpga_dev_groups;
+	fpga_dev_class->dev_release = fpga_dev_release;
+
+	return 0;
+}
+
+static void __exit fpga_dev_class_exit(void)
+{
+	class_destroy(fpga_dev_class);
+}
+
+MODULE_DESCRIPTION("FPGA Device framework");
+MODULE_LICENSE("GPL v2");
+
+subsys_initcall(fpga_dev_class_init);
+module_exit(fpga_dev_class_exit);
diff --git a/include/linux/fpga/fpga-dev.h b/include/linux/fpga/fpga-dev.h
new file mode 100644
index 0000000..ee693cc
--- /dev/null
+++ b/include/linux/fpga/fpga-dev.h
@@ -0,0 +1,31 @@
+/*
+ * FPGA Device Driver Header
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * This work is licensed under the terms of the GNU GPL version 2. See
+ * the COPYING file in the top-level directory.
+ */
+#ifndef _LINUX_FPGA_DEV_H
+#define _LINUX_FPGA_DEV_H
+
+/**
+ * struct fpga_dev - fpga device structure
+ * @name: name of fpga device
+ * @dev: fpga device
+ */
+struct fpga_dev {
+	const char *name;
+	struct device dev;
+};
+
+#define to_fpga_dev(d) container_of(d, struct fpga_dev, dev)
+
+struct fpga_dev *fpga_dev_create(struct device *parent, const char *name);
+
+static inline void fpga_dev_destroy(struct fpga_dev *fdev)
+{
+	device_unregister(&fdev->dev);
+}
+
+#endif
-- 
1.8.3.1

[PATCH v2 03/22] fpga: bridge: remove OF dependency for fpga-bridge

[Wu Hao]

This patch removes OF dependency of fpga-bridge, it allows drivers
to use fpga-bridge class without device tree support.

Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/fpga/Kconfig | 1 -
 1 file changed, 1 deletion(-)

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index ed600d5..c1d8f41 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig
@@ -75,7 +75,6 @@ config FPGA_MGR_ZYNQ_FPGA
 
 config FPGA_BRIDGE
 	tristate "FPGA Bridge Framework"
-	depends on OF
 	help
 	  Say Y here if you want to support bridges connected between host
 	  processors and FPGAs or between FPGAs.
-- 
1.8.3.1

[PATCH v2 04/22] fpga: mgr: add region_id to fpga_image_info

[Wu Hao]

This patch adds region_id to fpga_image_info data structure, it
allows driver to pass region id information to fpga-mgr via
fpga_image_info for fpga reconfiguration function.

Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 include/linux/fpga/fpga-mgr.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/linux/fpga/fpga-mgr.h b/include/linux/fpga/fpga-mgr.h
index 66d0e32..b222a57 100644
--- a/include/linux/fpga/fpga-mgr.h
+++ b/include/linux/fpga/fpga-mgr.h
@@ -95,6 +95,7 @@ struct fpga_image_info {
 	struct sg_table *sgt;
 	const char *buf;
 	size_t count;
+	int region_id;
 #ifdef CONFIG_OF
 	struct device_node *overlay;
 #endif
-- 
1.8.3.1

[PATCH v2 05/22] fpga: mgr: add status for fpga-mgr

[Wu Hao]

This patch adds status to fpga-manager data structure, to allow
driver to store full/partial reconfiguration errors and other
status information.

one sysfs interface created for user space application to read
fpga-manager status.

Signed-off-by: Wu Hao <hao.wu@intel.com>
---
 Documentation/ABI/testing/sysfs-class-fpga-manager | 10 +++++++++
 drivers/fpga/fpga-mgr.c                            | 24 ++++++++++++++++++++++
 include/linux/fpga/fpga-mgr.h                      |  9 ++++++++
 3 files changed, 43 insertions(+)

diff --git a/Documentation/ABI/testing/sysfs-class-fpga-manager b/Documentation/ABI/testing/sysfs-class-fpga-manager
index 23056c5..71b083e 100644
--- a/Documentation/ABI/testing/sysfs-class-fpga-manager
+++ b/Documentation/ABI/testing/sysfs-class-fpga-manager
@@ -35,3 +35,13 @@ Description:	Read fpga manager state as a string.
 		* write complete	= Doing post programming steps
 		* write complete error	= Error while doing post programming
 		* operating		= FPGA is programmed and operating
+
+What:		/sys/class/fpga_manager/<fpga>/status
+Date:		June 2017
+KernelVersion:	4.12
+Contact:	Wu Hao <hao.wu@intel.com>
+Description:	Read fpga manager status as a string.
+		If FPGA programming operation fails, it could be due to crc
+		error or incompatible bitstream image. The intent of this
+		interface is to provide more detailed information for FPGA
+		programming errors to userspace.
diff --git a/drivers/fpga/fpga-mgr.c b/drivers/fpga/fpga-mgr.c
index be13cce..2485658 100644
--- a/drivers/fpga/fpga-mgr.c
+++ b/drivers/fpga/fpga-mgr.c
@@ -388,12 +388,36 @@ static ssize_t state_show(struct device *dev,
 	return sprintf(buf, "%s\n", state_str[mgr->state]);
 }
 
+static ssize_t status_show(struct device *dev,
+			   struct device_attribute *attr, char *buf)
+{
+	struct fpga_manager *mgr = to_fpga_manager(dev);
+	int len = 0;
+
+	if (mgr->status & FPGA_MGR_STATUS_OPERATION_ERR)
+		len += sprintf(buf + len, "reconfig operation error\n");
+	if (mgr->status & FPGA_MGR_STATUS_CRC_ERR)
+		len += sprintf(buf + len, "reconfig crc error\n");
+	if (mgr->status & FPGA_MGR_STATUS_INCOMPATIBLE_BS_ERR)
+		len += sprintf(buf + len, "reconfig incompatible BS error\n");
+	if (mgr->status & FPGA_MGR_STATUS_IP_PROTOCOL_ERR)
+		len += sprintf(buf + len, "reconfig IP protocol error\n");
+	if (mgr->status & FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR)
+		len += sprintf(buf + len, "reconfig fifo overflow error\n");
+	if (mgr->status & FPGA_MGR_STATUS_SECURE_LOAD_ERR)
+		len += sprintf(buf + len, "reconfig secure load error\n");
+
+	return len;
+}
+
 static DEVICE_ATTR_RO(name);
 static DEVICE_ATTR_RO(state);
+static DEVICE_ATTR_RO(status);
 
 static struct attribute *fpga_mgr_attrs[] = {
 	&dev_attr_name.attr,
 	&dev_attr_state.attr,
+	&dev_attr_status.attr,
 	NULL,
 };
 ATTRIBUTE_GROUPS(fpga_mgr);
diff --git a/include/linux/fpga/fpga-mgr.h b/include/linux/fpga/fpga-mgr.h
index b222a57..8cb42ac 100644
--- a/include/linux/fpga/fpga-mgr.h
+++ b/include/linux/fpga/fpga-mgr.h
@@ -128,6 +128,14 @@ struct fpga_manager_ops {
 	void (*fpga_remove)(struct fpga_manager *mgr);
 };
 
+/* FPGA manager status: Partial/Full Reconfiguration errors */
+#define FPGA_MGR_STATUS_OPERATION_ERR		BIT(0)
+#define FPGA_MGR_STATUS_CRC_ERR			BIT(1)
+#define FPGA_MGR_STATUS_INCOMPATIBLE_BS_ERR	BIT(2)
+#define FPGA_MGR_STATUS_IP_PROTOCOL_ERR		BIT(3)
+#define FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR	BIT(4)
+#define FPGA_MGR_STATUS_SECURE_LOAD_ERR		BIT(5)
+
 /**
  * struct fpga_manager - fpga manager structure
  * @name: name of low level fpga manager
@@ -142,6 +150,7 @@ struct fpga_manager {
 	struct device dev;
 	struct mutex ref_mutex;
 	enum fpga_mgr_states state;
+	u64 status;
 	const struct fpga_manager_ops *mops;
 	void *priv;
 };
-- 
1.8.3.1

[PATCH v2 06/22] fpga: intel: add FPGA PCIe device driver

[Wu Hao]

From: Zhang Yi <yi.z.zhang@intel.com>

The Intel FPGA device appears as a PCIe device on the system. This patch
implements the basic framework of the driver for Intel PCIe device which
locates between CPU and Accelerated Function Units (AFUs).

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
    switched to GPLv2 license.
    fixed comments from Moritz Fischer.
---
 drivers/fpga/Kconfig      |  28 +++++++++++
 drivers/fpga/Makefile     |   5 ++
 drivers/fpga/intel-pcie.c | 126 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 159 insertions(+)
 create mode 100644 drivers/fpga/intel-pcie.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index c1d8f41..3f3b7f4 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig
@@ -117,6 +117,34 @@ config XILINX_PR_DECOUPLER
 	  region of the FPGA from the busses while that region is
 	  being reprogrammed during partial reconfig.
 
+menuconfig INTEL_FPGA
+	tristate "Intel(R) FPGA support"
+	depends on FPGA_DEVICE
+	help
+	  Select this option to enable driver support for Intel(R)
+	  Field-Programmable Gate Array (FPGA) solutions. This driver
+	  provides interfaces for userspace applications to configure,
+	  enumerate, open, and access FPGA accelerators on platforms
+	  equipped with Intel(R) FPGA solutions and enables system
+	  level management functions such as FPGA reconfiguration,
+	  power management, and virtualization.
+
+	  Say Y if your platform has this technology. Say N if unsure.
+
+if INTEL_FPGA
+
+config INTEL_FPGA_PCI
+	tristate "Intel FPGA PCIe Driver"
+	depends on PCI
+	help
+	  This is the driver for the PCIe device which locates between
+	  CPU and Accelerated Function Units (AFUs) and allows them to
+	  communicate with each other.
+
+	  To compile this as a module, choose M here.
+
+endif
+
 endif # FPGA
 
 endmenu
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 8950a8f..5613133 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -27,3 +27,8 @@ obj-$(CONFIG_XILINX_PR_DECOUPLER)	+= xilinx-pr-decoupler.o
 # High Level Interfaces
 obj-$(CONFIG_FPGA_REGION)		+= fpga-region.o
 obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
+
+# Intel FPGA Support
+obj-$(CONFIG_INTEL_FPGA_PCI)		+= intel-fpga-pci.o
+
+intel-fpga-pci-objs := intel-pcie.o
diff --git a/drivers/fpga/intel-pcie.c b/drivers/fpga/intel-pcie.c
new file mode 100644
index 0000000..f697de4
--- /dev/null
+++ b/drivers/fpga/intel-pcie.c
@@ -0,0 +1,126 @@
+/*
+ * Driver for Intel FPGA PCIe device
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Zhang Yi <Yi.Z.Zhang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include <linux/pci.h>
+#include <linux/types.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/stddef.h>
+#include <linux/errno.h>
+#include <linux/aer.h>
+
+#define DRV_VERSION	"0.8"
+#define DRV_NAME	"intel-fpga-pci"
+
+/* PCI Device ID */
+#define PCIe_DEVICE_ID_PF_INT_5_X	0xBCBD
+#define PCIe_DEVICE_ID_PF_INT_6_X	0xBCC0
+#define PCIe_DEVICE_ID_PF_DSC_1_X	0x09C4
+/* VF Device */
+#define PCIe_DEVICE_ID_VF_INT_5_X	0xBCBF
+#define PCIe_DEVICE_ID_VF_INT_6_X	0xBCC1
+#define PCIe_DEVICE_ID_VF_DSC_1_X	0x09C5
+
+static struct pci_device_id cci_pcie_id_tbl[] = {
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_PF_INT_5_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_VF_INT_5_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_PF_INT_6_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_VF_INT_6_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_PF_DSC_1_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_VF_DSC_1_X),},
+	{0,}
+};
+MODULE_DEVICE_TABLE(pci, cci_pcie_id_tbl);
+
+static
+int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
+{
+	int ret;
+
+	ret = pci_enable_device(pcidev);
+	if (ret < 0) {
+		dev_err(&pcidev->dev, "Failed to enable device %d.\n", ret);
+		return ret;
+	}
+
+	ret = pci_enable_pcie_error_reporting(pcidev);
+	if (ret && ret != -EINVAL)
+		dev_info(&pcidev->dev, "PCIE AER unavailable %d.\n", ret);
+
+	ret = pci_request_regions(pcidev, DRV_NAME);
+	if (ret) {
+		dev_err(&pcidev->dev, "Failed to request regions.\n");
+		goto disable_error_report_exit;
+	}
+
+	pci_set_master(pcidev);
+	pci_save_state(pcidev);
+
+	if (!dma_set_mask(&pcidev->dev, DMA_BIT_MASK(64))) {
+		dma_set_coherent_mask(&pcidev->dev, DMA_BIT_MASK(64));
+	} else if (!dma_set_mask(&pcidev->dev, DMA_BIT_MASK(32))) {
+		dma_set_coherent_mask(&pcidev->dev, DMA_BIT_MASK(32));
+	} else {
+		ret = -EIO;
+		dev_err(&pcidev->dev, "No suitable DMA support available.\n");
+		goto release_region_exit;
+	}
+
+	/* TODO: create and add the platform device per feature list */
+	return 0;
+
+release_region_exit:
+	pci_release_regions(pcidev);
+disable_error_report_exit:
+	pci_disable_pcie_error_reporting(pcidev);
+	pci_disable_device(pcidev);
+	return ret;
+}
+
+static void cci_pci_remove(struct pci_dev *pcidev)
+{
+	pci_release_regions(pcidev);
+	pci_disable_pcie_error_reporting(pcidev);
+	pci_disable_device(pcidev);
+}
+
+static struct pci_driver cci_pci_driver = {
+	.name = DRV_NAME,
+	.id_table = cci_pcie_id_tbl,
+	.probe = cci_pci_probe,
+	.remove = cci_pci_remove,
+};
+
+static int __init ccidrv_init(void)
+{
+	pr_info("Intel(R) FPGA PCIe Driver: Version %s\n", DRV_VERSION);
+
+	return pci_register_driver(&cci_pci_driver);
+}
+
+static void __exit ccidrv_exit(void)
+{
+	pci_unregister_driver(&cci_pci_driver);
+}
+
+module_init(ccidrv_init);
+module_exit(ccidrv_exit);
+
+MODULE_DESCRIPTION("Intel FPGA PCIe Device Driver");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
-- 
1.8.3.1

[PATCH v2 07/22] fpga: intel: pcie: parse feature list and create platform device for features.

[Wu Hao]

From: Xiao Guangrong <guangrong.xiao@linux.intel.com>

Device Feature List structure creates a link list of feature headers
within the MMIO space to provide an extensible way of adding features.

The Intel FPGA PCIe driver walks through the feature headers to enumerate
feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
Function Unit (AFU), and their private sub features. For feature devices,
it creates the platform devices and linked the private sub features into
their platform data.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Kang Luwei <luwei.kang@intel.com>
Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
    switched to GPLv2 license.
    fixed comments from Moritz Fischer.
    fixed kbuild warning, typos and clean up the code.
---
 drivers/fpga/Makefile            |   2 +-
 drivers/fpga/intel-feature-dev.c | 130 ++++++
 drivers/fpga/intel-feature-dev.h | 341 ++++++++++++++++
 drivers/fpga/intel-pcie.c        | 841 ++++++++++++++++++++++++++++++++++++++-
 4 files changed, 1311 insertions(+), 3 deletions(-)
 create mode 100644 drivers/fpga/intel-feature-dev.c
 create mode 100644 drivers/fpga/intel-feature-dev.h

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 5613133..ad24b3d 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -31,4 +31,4 @@ obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
 # Intel FPGA Support
 obj-$(CONFIG_INTEL_FPGA_PCI)		+= intel-fpga-pci.o
 
-intel-fpga-pci-objs := intel-pcie.o
+intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
diff --git a/drivers/fpga/intel-feature-dev.c b/drivers/fpga/intel-feature-dev.c
new file mode 100644
index 0000000..68f9cba
--- /dev/null
+++ b/drivers/fpga/intel-feature-dev.c
@@ -0,0 +1,130 @@
+/*
+ * Intel FPGA Feature Device Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Zhang Yi <yi.z.zhang@intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include "intel-feature-dev.h"
+
+void feature_platform_data_add(struct feature_platform_data *pdata,
+			       int index, const char *name,
+			       int resource_index, void __iomem *ioaddr)
+{
+	WARN_ON(index >= pdata->num);
+
+	pdata->features[index].name = name;
+	pdata->features[index].resource_index = resource_index;
+	pdata->features[index].ioaddr = ioaddr;
+}
+
+struct feature_platform_data *
+feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
+{
+	struct feature_platform_data *pdata;
+
+	pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
+	if (pdata) {
+		pdata->dev = dev;
+		pdata->num = num;
+		mutex_init(&pdata->lock);
+	}
+
+	return pdata;
+}
+
+int fme_feature_num(void)
+{
+	return FME_FEATURE_ID_MAX;
+}
+
+int port_feature_num(void)
+{
+	return PORT_FEATURE_ID_MAX;
+}
+
+int fpga_port_id(struct platform_device *pdev)
+{
+	struct feature_port_header *port_hdr;
+	struct feature_port_capability capability;
+
+	port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
+					       PORT_FEATURE_ID_HEADER);
+	WARN_ON(!port_hdr);
+
+	capability.csr = readq(&port_hdr->capability);
+	return capability.port_number;
+}
+EXPORT_SYMBOL_GPL(fpga_port_id);
+
+/*
+ * Enable Port by clear the port soft reset bit, which is set by default.
+ * The User AFU is unable to respond to any MMIO access while in reset.
+ * __fpga_port_enable function should only be used after __fpga_port_disable
+ * function.
+ */
+void __fpga_port_enable(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct feature_port_header *port_hdr;
+	struct feature_port_control control;
+
+	WARN_ON(!pdata->disable_count);
+
+	if (--pdata->disable_count != 0)
+		return;
+
+	port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
+					       PORT_FEATURE_ID_HEADER);
+	WARN_ON(!port_hdr);
+
+	control.csr = readq(&port_hdr->control);
+	control.port_sftrst = 0x0;
+	writeq(control.csr, &port_hdr->control);
+}
+EXPORT_SYMBOL_GPL(__fpga_port_enable);
+
+#define RST_POLL_INVL 10 /* us */
+#define RST_POLL_TIMEOUT 1000 /* us */
+
+int __fpga_port_disable(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct feature_port_header *port_hdr;
+	struct feature_port_control control;
+
+	if (pdata->disable_count++ != 0)
+		return 0;
+
+	port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
+					       PORT_FEATURE_ID_HEADER);
+	WARN_ON(!port_hdr);
+
+	/* Set port soft reset */
+	control.csr = readq(&port_hdr->control);
+	control.port_sftrst = 0x1;
+	writeq(control.csr, &port_hdr->control);
+
+	/*
+	 * HW sets ack bit to 1 when all outstanding requests have been drained
+	 * on this port and minimum soft reset pulse width has elapsed.
+	 * Driver polls port_soft_reset_ack to determine if reset done by HW.
+	 */
+	if (readq_poll_timeout(&port_hdr->control, control.csr,
+			       (control.port_sftrst_ack == 1),
+			       RST_POLL_INVL, RST_POLL_TIMEOUT)) {
+		dev_err(&pdev->dev, "timeout, fail to reset device\n");
+		return -ETIMEDOUT;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(__fpga_port_disable);
diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
new file mode 100644
index 0000000..f67784a
--- /dev/null
+++ b/drivers/fpga/intel-feature-dev.h
@@ -0,0 +1,341 @@
+/*
+ * Intel FPGA Feature Device Driver Header File
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Zhang Yi <yi.z.zhang@intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#ifndef __INTEL_FPGA_FEATURE_H
+#define __INTEL_FPGA_FEATURE_H
+
+#include <linux/fs.h>
+#include <linux/pci.h>
+#include <linux/uuid.h>
+#include <linux/delay.h>
+#include <linux/iopoll.h>
+#include <linux/platform_device.h>
+
+#ifndef readq
+static inline u64 readq(void __iomem *addr)
+{
+	return readl(addr) + ((u64)readl(addr + 4) << 32);
+}
+#endif
+
+#ifndef writeq
+static inline void writeq(u64 val, void __iomem *addr)
+{
+	writel((u32) (val), addr);
+	writel((u32) (val >> 32), (addr + 4));
+}
+#endif
+
+/* maximum supported number of ports */
+#define MAX_FPGA_PORT_NUM 4
+/* plus one for fme device */
+#define MAX_FEATURE_DEV_NUM	(MAX_FPGA_PORT_NUM + 1)
+
+#define FME_FEATURE_HEADER          "fme_hdr"
+#define FME_FEATURE_THERMAL_MGMT    "fme_thermal"
+#define FME_FEATURE_POWER_MGMT      "fme_power"
+#define FME_FEATURE_GLOBAL_PERF     "fme_gperf"
+#define FME_FEATURE_GLOBAL_ERR      "fme_error"
+#define FME_FEATURE_PR_MGMT         "fme_pr"
+
+#define PORT_FEATURE_HEADER         "port_hdr"
+#define PORT_FEATURE_UAFU           "port_uafu"
+#define PORT_FEATURE_ERR            "port_err"
+#define PORT_FEATURE_UMSG           "port_umsg"
+#define PORT_FEATURE_PR             "port_pr"
+#define PORT_FEATURE_STP            "port_stp"
+
+/* All headers and structures must be byte-packed to match the spec. */
+#pragma pack(1)
+
+/* common header for all features */
+struct feature_header {
+	union {
+		u64 csr;
+		struct {
+			u64 id:12;
+			u64 revision:4;
+			u64 next_header_offset:24; /* offset to next header */
+			u64 rsvdz:20;
+			u64 type:4;		   /* feature type */
+#define FEATURE_TYPE_AFU		0x1
+#define FEATURE_TYPE_PRIVATE		0x3
+		};
+	};
+};
+
+/* common header for non-private features */
+struct feature_afu_header {
+	uuid_le guid;
+	union {
+		u64 csr;
+		struct {
+			u64 next_afu:24;	/* pointer to next afu header */
+			u64 rsvdz:40;
+		};
+	};
+};
+
+/* FME Header Register Set */
+/* FME Capability Register */
+struct feature_fme_capability {
+	union {
+		u64 csr;
+		struct {
+			u64 fabric_verid:8;	/* Fabric version ID */
+			u64 socket_id:1;	/* Socket id */
+			u64 rsvdz1:3;
+			u64 pcie0_link_avl:1;	/* PCIe0 link availability */
+			u64 pcie1_link_avl:1;	/* PCIe1 link availability */
+			u64 coherent_link_avl:1;/* Coherent link availability */
+			u64 rsvdz2:1;
+			u64 iommu_support:1;	/* IOMMU or VT-d supported */
+			u64 num_ports:3;	/* Num of ports implemented */
+			u64 rsvdz3:4;
+			u64 addr_width_bits:6;	/* Address width supported */
+			u64 rsvdz4:2;
+			u64 cache_size:12;	/* Cache size in kb */
+			u64 cache_assoc:4;	/* Cache Associativity */
+			u64 rsvdz5:15;
+			u64 lock_bit:1;		/* Latched lock bit by BIOS */
+		};
+	};
+};
+
+/* FME Port Offset Register */
+struct feature_fme_port {
+	union {
+		u64 csr;
+		struct {
+			u64 port_offset:24;	/* Offset to port header */
+			u64 rsvdz1:8;
+			u64 port_bar:3;		/* Bar id */
+			u64 rsvdz2:20;
+			u64 afu_access_ctrl:1;	/* AFU access type: PF/VF */
+			u64 rsvdz3:4;
+			u64 port_implemented:1;	/* Port implemented or not */
+			u64 rsvdz4:3;
+		};
+	};
+};
+
+struct feature_fme_header {
+	struct feature_header header;
+	struct feature_afu_header afu_header;
+	u64 rsvd[2];
+	struct feature_fme_capability capability;
+	struct feature_fme_port port[MAX_FPGA_PORT_NUM];
+};
+
+/* FME Thermal Sub Feature Register Set */
+struct feature_fme_thermal {
+	struct feature_header header;
+};
+
+/* FME Power Sub Feature Register Set */
+struct feature_fme_power {
+	struct feature_header header;
+};
+
+/* FME Global Performance Sub Feature Register Set */
+struct feature_fme_gperf {
+	struct feature_header header;
+};
+
+/* FME Error Sub Feature Register Set */
+struct feature_fme_err {
+	struct feature_header header;
+};
+
+/* FME Partial Reconfiguration Sub Feature Register Set */
+struct feature_fme_pr {
+	struct feature_header header;
+};
+
+/* PORT Header Register Set */
+/* Port Capability Register */
+struct feature_port_capability {
+	union {
+		u64 csr;
+		struct {
+			u64 port_number:2;	/* Port Number 0-3 */
+			u64 rsvdz1:6;
+			u64 mmio_size:16;	/* User MMIO size in KB */
+			u64 rsvdz2:8;
+			u64 sp_intr_num:4;	/* Supported interrupts num */
+			u64 rsvdz3:28;
+		};
+	};
+};
+
+/* Port Control Register */
+struct feature_port_control {
+	union {
+		u64 csr;
+		struct {
+			u64 port_sftrst:1;	/* Port Soft Reset */
+			u64 rsvdz1:1;
+			u64 latency_tolerance:1;/* '1' >= 40us, '0' < 40us */
+			u64 rsvdz2:1;
+			u64 port_sftrst_ack:1;	/* HW ACK for Soft Reset */
+			u64 rsvdz3:59;
+		};
+	};
+};
+
+struct feature_port_header {
+	struct feature_header header;
+	struct feature_afu_header afu_header;
+	u64 rsvd[2];
+	struct feature_port_capability capability;
+	struct feature_port_control control;
+};
+
+/* PORT Error Sub Feature Register Set */
+struct feature_port_error {
+	struct feature_header header;
+};
+
+/* PORT Unordered Message Sub Feature Register Set */
+struct feature_port_umsg {
+	struct feature_header header;
+};
+
+/* PORT SignalTap Sub Feature Register Set */
+struct feature_port_stp {
+	struct feature_header header;
+};
+
+#pragma pack()
+
+struct feature {
+	const char *name;
+	int resource_index;
+	void __iomem *ioaddr;
+};
+
+struct feature_platform_data {
+	/* list the feature dev to cci_drvdata->port_dev_list. */
+	struct list_head node;
+	struct mutex lock;
+	struct platform_device *dev;
+	unsigned int disable_count;	/* count for port disable */
+
+	int num;			/* number of features */
+	struct feature features[0];
+};
+
+enum fme_feature_id {
+	FME_FEATURE_ID_HEADER = 0x0,
+	FME_FEATURE_ID_THERMAL_MGMT = 0x1,
+	FME_FEATURE_ID_POWER_MGMT = 0x2,
+	FME_FEATURE_ID_GLOBAL_PERF = 0x3,
+	FME_FEATURE_ID_GLOBAL_ERR = 0x4,
+	FME_FEATURE_ID_PR_MGMT = 0x5,
+	FME_FEATURE_ID_MAX = 0x6,
+};
+
+enum port_feature_id {
+	PORT_FEATURE_ID_HEADER = 0x0,
+	PORT_FEATURE_ID_ERROR = 0x1,
+	PORT_FEATURE_ID_UMSG = 0x2,
+	PORT_FEATURE_ID_PR = 0x3,
+	PORT_FEATURE_ID_STP = 0x4,
+	PORT_FEATURE_ID_UAFU = 0x5,
+	PORT_FEATURE_ID_MAX = 0x6,
+};
+
+int fme_feature_num(void);
+int port_feature_num(void);
+
+#define FPGA_FEATURE_DEV_FME		"intel-fpga-fme"
+#define FPGA_FEATURE_DEV_PORT		"intel-fpga-port"
+
+void feature_platform_data_add(struct feature_platform_data *pdata,
+			       int index, const char *name,
+			       int resource_index, void __iomem *ioaddr);
+
+static inline int feature_platform_data_size(const int num)
+{
+	return sizeof(struct feature_platform_data) +
+		num * sizeof(struct feature);
+}
+
+struct feature_platform_data *
+feature_platform_data_alloc_and_init(struct platform_device *dev, int num);
+
+int fpga_port_id(struct platform_device *pdev);
+
+static inline int fpga_port_check_id(struct platform_device *pdev,
+				     void *pport_id)
+{
+	return fpga_port_id(pdev) == *(int *)pport_id;
+}
+
+void __fpga_port_enable(struct platform_device *pdev);
+int __fpga_port_disable(struct platform_device *pdev);
+
+static inline void fpga_port_enable(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	mutex_lock(&pdata->lock);
+	__fpga_port_enable(pdev);
+	mutex_unlock(&pdata->lock);
+}
+
+static inline int fpga_port_disable(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	int ret;
+
+	mutex_lock(&pdata->lock);
+	ret = __fpga_port_disable(pdev);
+	mutex_unlock(&pdata->lock);
+
+	return ret;
+}
+
+static inline int __fpga_port_reset(struct platform_device *pdev)
+{
+	int ret;
+
+	ret = __fpga_port_disable(pdev);
+	if (ret)
+		return ret;
+
+	__fpga_port_enable(pdev);
+	return 0;
+}
+
+static inline int fpga_port_reset(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	int ret;
+
+	mutex_lock(&pdata->lock);
+	ret = __fpga_port_reset(pdev);
+	mutex_unlock(&pdata->lock);
+	return ret;
+}
+
+static inline void __iomem *
+get_feature_ioaddr_by_index(struct device *dev, int index)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(dev);
+
+	return pdata->features[index].ioaddr;
+}
+#endif
diff --git a/drivers/fpga/intel-pcie.c b/drivers/fpga/intel-pcie.c
index f697de4..70b8284 100644
--- a/drivers/fpga/intel-pcie.c
+++ b/drivers/fpga/intel-pcie.c
@@ -23,10 +23,827 @@
 #include <linux/stddef.h>
 #include <linux/errno.h>
 #include <linux/aer.h>
+#include <linux/fpga/fpga-dev.h>
+
+#include "intel-feature-dev.h"
 
 #define DRV_VERSION	"0.8"
 #define DRV_NAME	"intel-fpga-pci"
 
+#define INTEL_FPGA_DEV	"intel-fpga-dev"
+
+static DEFINE_MUTEX(fpga_id_mutex);
+
+enum fpga_id_type {
+	FME_ID,		/* fme id allocation and mapping */
+	PORT_ID,	/* port id allocation and mapping */
+	FPGA_ID_MAX,
+};
+
+/* it is protected by fpga_id_mutex */
+static struct idr fpga_ids[FPGA_ID_MAX];
+
+struct cci_drvdata {
+	struct device *fme_dev;
+
+	struct mutex lock;
+	struct list_head port_dev_list;
+
+	struct list_head regions; /* global list of pci bar mapping region */
+};
+
+/* pci bar mapping info */
+struct cci_pci_region {
+	int bar;
+	void __iomem *ioaddr;	/* pointer to mapped bar region */
+	struct list_head node;
+};
+
+static void fpga_ids_init(void)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
+		idr_init(fpga_ids + i);
+}
+
+static void fpga_ids_destroy(void)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
+		idr_destroy(fpga_ids + i);
+}
+
+static int alloc_fpga_id(enum fpga_id_type type, struct device *dev)
+{
+	int id;
+
+	WARN_ON(type >= FPGA_ID_MAX);
+	mutex_lock(&fpga_id_mutex);
+	id = idr_alloc(fpga_ids + type, dev, 0, 0, GFP_KERNEL);
+	mutex_unlock(&fpga_id_mutex);
+	return id;
+}
+
+static void free_fpga_id(enum fpga_id_type type, int id)
+{
+	WARN_ON(type >= FPGA_ID_MAX);
+	mutex_lock(&fpga_id_mutex);
+	idr_remove(fpga_ids + type, id);
+	mutex_unlock(&fpga_id_mutex);
+}
+
+static void cci_pci_add_port_dev(struct pci_dev *pdev,
+				 struct platform_device *port_dev)
+{
+	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
+	struct feature_platform_data *pdata = dev_get_platdata(&port_dev->dev);
+
+	mutex_lock(&drvdata->lock);
+	list_add(&pdata->node, &drvdata->port_dev_list);
+	get_device(&pdata->dev->dev);
+	mutex_unlock(&drvdata->lock);
+}
+
+static void cci_pci_remove_port_devs(struct pci_dev *pdev)
+{
+	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
+	struct feature_platform_data *pdata, *ptmp;
+
+	mutex_lock(&drvdata->lock);
+	list_for_each_entry_safe(pdata, ptmp, &drvdata->port_dev_list, node) {
+		struct platform_device *port_dev = pdata->dev;
+
+		/* the port should be unregistered first. */
+		WARN_ON(device_is_registered(&port_dev->dev));
+		list_del(&pdata->node);
+		free_fpga_id(PORT_ID, port_dev->id);
+		put_device(&port_dev->dev);
+	}
+	mutex_unlock(&drvdata->lock);
+}
+
+/* info collection during feature dev build. */
+struct build_feature_devs_info {
+	struct pci_dev *pdev;
+
+	/*
+	 * PCI BAR mapping info. Parsing feature list starts from
+	 * BAR 0 and switch to different BARs to parse Port
+	 */
+	void __iomem *ioaddr;
+	void __iomem *ioend;
+	int current_bar;
+
+	/* points to FME header where the port offset is figured out. */
+	void __iomem *pfme_hdr;
+
+	/* the container device for all feature devices */
+	struct fpga_dev *parent_dev;
+
+	/* current feature device */
+	struct platform_device *feature_dev;
+};
+
+static void cci_pci_release_regions(struct pci_dev *pdev)
+{
+	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
+	struct cci_pci_region *tmp, *region;
+
+	list_for_each_entry_safe(region, tmp, &drvdata->regions, node) {
+		list_del(&region->node);
+		if (region->ioaddr)
+			pci_iounmap(pdev, region->ioaddr);
+		devm_kfree(&pdev->dev, region);
+	}
+}
+
+static void __iomem *cci_pci_ioremap_bar(struct pci_dev *pdev, int bar)
+{
+	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
+	struct cci_pci_region *region;
+
+	list_for_each_entry(region, &drvdata->regions, node)
+		if (region->bar == bar) {
+			dev_dbg(&pdev->dev, "BAR %d region exists\n", bar);
+			return region->ioaddr;
+		}
+
+	region = devm_kzalloc(&pdev->dev, sizeof(*region), GFP_KERNEL);
+	if (!region)
+		return NULL;
+
+	region->bar = bar;
+	region->ioaddr = pci_ioremap_bar(pdev, bar);
+	if (!region->ioaddr) {
+		dev_err(&pdev->dev, "can't ioremap memory from BAR %d.\n", bar);
+		devm_kfree(&pdev->dev, region);
+		return NULL;
+	}
+
+	list_add(&region->node, &drvdata->regions);
+	return region->ioaddr;
+}
+
+static int parse_start_from(struct build_feature_devs_info *binfo, int bar)
+{
+	binfo->ioaddr = cci_pci_ioremap_bar(binfo->pdev, bar);
+	if (!binfo->ioaddr)
+		return -ENOMEM;
+
+	binfo->current_bar = bar;
+	binfo->ioend = binfo->ioaddr + pci_resource_len(binfo->pdev, bar);
+	return 0;
+}
+
+static int parse_start(struct build_feature_devs_info *binfo)
+{
+	/* fpga feature list starts from BAR 0 */
+	return parse_start_from(binfo, 0);
+}
+
+/* switch the memory mapping to BAR# @bar */
+static int parse_switch_to(struct build_feature_devs_info *binfo, int bar)
+{
+	return parse_start_from(binfo, bar);
+}
+
+static struct build_feature_devs_info *
+build_info_alloc_and_init(struct pci_dev *pdev)
+{
+	struct build_feature_devs_info *binfo;
+
+	binfo = devm_kzalloc(&pdev->dev, sizeof(*binfo), GFP_KERNEL);
+	if (binfo)
+		binfo->pdev = pdev;
+
+	return binfo;
+}
+
+static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
+{
+	if (!strcmp(pdev->name, FPGA_FEATURE_DEV_FME))
+		return FME_ID;
+
+	if (!strcmp(pdev->name, FPGA_FEATURE_DEV_PORT))
+		return PORT_ID;
+
+	WARN_ON(1);
+	return FPGA_ID_MAX;
+}
+
+/*
+ * register current feature device, it is called when we need to switch to
+ * another feature parsing or we have parsed all features
+ */
+static int build_info_commit_dev(struct build_feature_devs_info *binfo)
+{
+	int ret;
+
+	if (!binfo->feature_dev)
+		return 0;
+
+	ret = platform_device_add(binfo->feature_dev);
+	if (!ret) {
+		struct cci_drvdata *drvdata;
+
+		drvdata = dev_get_drvdata(&binfo->pdev->dev);
+		if (feature_dev_id_type(binfo->feature_dev) == PORT_ID)
+			cci_pci_add_port_dev(binfo->pdev, binfo->feature_dev);
+		else
+			drvdata->fme_dev = get_device(&binfo->feature_dev->dev);
+
+		/*
+		 * reset it to avoid build_info_free() freeing their resource.
+		 *
+		 * The resource of successfully registered feature devices
+		 * will be freed by platform_device_unregister(). See the
+		 * comments in build_info_create_dev().
+		 */
+		binfo->feature_dev = NULL;
+	}
+
+	return ret;
+}
+
+static int
+build_info_create_dev(struct build_feature_devs_info *binfo,
+		      enum fpga_id_type type, int feature_nr, const char *name)
+{
+	struct platform_device *fdev;
+	struct resource *res;
+	struct feature_platform_data *pdata;
+	int ret;
+
+	/* we will create a new device, commit current device first */
+	ret = build_info_commit_dev(binfo);
+	if (ret)
+		return ret;
+
+	/*
+	 * we use -ENODEV as the initialization indicator which indicates
+	 * whether the id need to be reclaimed
+	 */
+	fdev = binfo->feature_dev = platform_device_alloc(name, -ENODEV);
+	if (!fdev)
+		return -ENOMEM;
+
+	fdev->id = alloc_fpga_id(type, &fdev->dev);
+	if (fdev->id < 0)
+		return fdev->id;
+
+	fdev->dev.parent = &binfo->parent_dev->dev;
+
+	/*
+	 * we do not need to care for the memory which is associated with
+	 * the platform device. After calling platform_device_unregister(),
+	 * it will be automatically freed by device's release() callback,
+	 * platform_device_release().
+	 */
+	pdata = feature_platform_data_alloc_and_init(fdev, feature_nr);
+	if (!pdata)
+		return -ENOMEM;
+
+	/*
+	 * the count should be initialized to 0 to make sure
+	 *__fpga_port_enable() following __fpga_port_disable()
+	 * works properly for port device.
+	 * and it should always be 0 for fme device.
+	 */
+	WARN_ON(pdata->disable_count);
+
+	fdev->dev.platform_data = pdata;
+	fdev->num_resources = feature_nr;
+	fdev->resource = kcalloc(feature_nr, sizeof(*res), GFP_KERNEL);
+	if (!fdev->resource)
+		return -ENOMEM;
+
+	return 0;
+}
+
+static int remove_feature_dev(struct device *dev, void *data)
+{
+	struct platform_device *pdev = to_platform_device(dev);
+
+	platform_device_unregister(pdev);
+	return 0;
+}
+
+static int remove_parent_dev(struct device *dev, void *data)
+{
+	/* remove platform devices attached in the parent device */
+	device_for_each_child(dev, NULL, remove_feature_dev);
+	fpga_dev_destroy(to_fpga_dev(dev));
+	return 0;
+}
+
+static void remove_all_devs(struct pci_dev *pdev)
+{
+	/* remove parent device and all its children. */
+	device_for_each_child(&pdev->dev, NULL, remove_parent_dev);
+}
+
+static void build_info_free(struct build_feature_devs_info *binfo)
+{
+	if (!IS_ERR_OR_NULL(binfo->parent_dev))
+		remove_all_devs(binfo->pdev);
+
+	/*
+	 * it is a valid id, free it. See comments in
+	 * build_info_create_dev()
+	 */
+	if (binfo->feature_dev && binfo->feature_dev->id >= 0)
+		free_fpga_id(feature_dev_id_type(binfo->feature_dev),
+			     binfo->feature_dev->id);
+
+	platform_device_put(binfo->feature_dev);
+
+	devm_kfree(&binfo->pdev->dev, binfo);
+}
+
+#define FEATURE_TYPE_AFU	0x1
+#define FEATURE_TYPE_PRIVATE	0x3
+
+/* FME and PORT GUID are fixed */
+#define FEATURE_FME_GUID "f9e17764-38f0-82fe-e346-524ae92aafbf"
+#define FEATURE_PORT_GUID "6b355b87-b06c-9642-eb42-8d139398b43a"
+
+static bool feature_is_fme(struct feature_afu_header *afu_hdr)
+{
+	uuid_le u;
+
+	uuid_le_to_bin(FEATURE_FME_GUID, &u);
+
+	return !uuid_le_cmp(u, afu_hdr->guid);
+}
+
+static bool feature_is_port(struct feature_afu_header *afu_hdr)
+{
+	uuid_le u;
+
+	uuid_le_to_bin(FEATURE_PORT_GUID, &u);
+
+	return !uuid_le_cmp(u, afu_hdr->guid);
+}
+
+/*
+ * UAFU GUID is dynamic as it can be changed after FME downloads different
+ * Green Bitstream to the port, so we treat the unknown GUIDs which are
+ * attached on port's feature list as UAFU.
+ */
+static bool feature_is_UAFU(struct build_feature_devs_info *binfo)
+{
+	if (!binfo->feature_dev ||
+	      feature_dev_id_type(binfo->feature_dev) != PORT_ID)
+		return false;
+
+	return true;
+}
+
+static void
+build_info_add_sub_feature(struct build_feature_devs_info *binfo,
+			   int feature_id, const char *feature_name,
+			   resource_size_t resource_size, void __iomem *start)
+{
+
+	struct platform_device *fdev = binfo->feature_dev;
+	struct feature_platform_data *pdata = dev_get_platdata(&fdev->dev);
+	struct resource *res = &fdev->resource[feature_id];
+
+	res->start = pci_resource_start(binfo->pdev, binfo->current_bar) +
+		start - binfo->ioaddr;
+	res->end = res->start + resource_size - 1;
+	res->flags = IORESOURCE_MEM;
+	res->name = feature_name;
+
+	feature_platform_data_add(pdata, feature_id,
+				  feature_name, feature_id, start);
+}
+
+struct feature_info {
+	const char *name;
+	resource_size_t resource_size;
+	int feature_index;
+};
+
+/* indexed by fme feature IDs which are defined in 'enum fme_feature_id'. */
+static struct feature_info fme_features[] = {
+	{
+		.name = FME_FEATURE_HEADER,
+		.resource_size = sizeof(struct feature_fme_header),
+		.feature_index = FME_FEATURE_ID_HEADER,
+	},
+	{
+		.name = FME_FEATURE_THERMAL_MGMT,
+		.resource_size = sizeof(struct feature_fme_thermal),
+		.feature_index = FME_FEATURE_ID_THERMAL_MGMT,
+	},
+	{
+		.name = FME_FEATURE_POWER_MGMT,
+		.resource_size = sizeof(struct feature_fme_power),
+		.feature_index = FME_FEATURE_ID_POWER_MGMT,
+	},
+	{
+		.name = FME_FEATURE_GLOBAL_PERF,
+		.resource_size = sizeof(struct feature_fme_gperf),
+		.feature_index = FME_FEATURE_ID_GLOBAL_PERF,
+	},
+	{
+		.name = FME_FEATURE_GLOBAL_ERR,
+		.resource_size = sizeof(struct feature_fme_err),
+		.feature_index = FME_FEATURE_ID_GLOBAL_ERR,
+	},
+	{
+		.name = FME_FEATURE_PR_MGMT,
+		.resource_size = sizeof(struct feature_fme_pr),
+		.feature_index = FME_FEATURE_ID_PR_MGMT,
+	}
+};
+
+/* indexed by port feature IDs which are defined in 'enum port_feature_id'. */
+static struct feature_info port_features[] = {
+	{
+		.name = PORT_FEATURE_HEADER,
+		.resource_size = sizeof(struct feature_port_header),
+		.feature_index = PORT_FEATURE_ID_HEADER,
+	},
+	{
+		.name = PORT_FEATURE_ERR,
+		.resource_size = sizeof(struct feature_port_error),
+		.feature_index = PORT_FEATURE_ID_ERROR,
+	},
+	{
+		.name = PORT_FEATURE_UMSG,
+		.resource_size = sizeof(struct feature_port_umsg),
+		.feature_index = PORT_FEATURE_ID_UMSG,
+	},
+	{
+		/* This feature isn't available for now */
+		.name = PORT_FEATURE_PR,
+		.resource_size = 0,
+		.feature_index = PORT_FEATURE_ID_PR,
+	},
+	{
+		.name = PORT_FEATURE_STP,
+		.resource_size = sizeof(struct feature_port_stp),
+		.feature_index = PORT_FEATURE_ID_STP,
+	},
+	{
+		/*
+		 * For User AFU feature, its region size is not fixed, but
+		 * reported by register PortCapability.mmio_size. Resource
+		 * size of UAFU will be set while parse port device.
+		 */
+		.name = PORT_FEATURE_UAFU,
+		.resource_size = 0,
+		.feature_index = PORT_FEATURE_ID_UAFU,
+	},
+};
+
+static int
+create_feature_instance(struct build_feature_devs_info *binfo,
+			void __iomem *start, struct feature_info *finfo)
+{
+	if (binfo->ioend - start < finfo->resource_size)
+		return -EINVAL;
+
+	build_info_add_sub_feature(binfo, finfo->feature_index, finfo->name,
+				   finfo->resource_size, start);
+	return 0;
+}
+
+static int parse_feature_fme(struct build_feature_devs_info *binfo,
+			     void __iomem *start)
+{
+	struct cci_drvdata *drvdata = dev_get_drvdata(&binfo->pdev->dev);
+	int ret;
+
+	ret = build_info_create_dev(binfo, FME_ID, fme_feature_num(),
+					FPGA_FEATURE_DEV_FME);
+	if (ret)
+		return ret;
+
+	if (drvdata->fme_dev) {
+		dev_err(&binfo->pdev->dev, "Multiple FMEs are detected.\n");
+		return -EINVAL;
+	}
+
+	return create_feature_instance(binfo, start,
+				       &fme_features[FME_FEATURE_ID_HEADER]);
+}
+
+static int parse_feature_fme_private(struct build_feature_devs_info *binfo,
+				     struct feature_header *hdr)
+{
+	struct feature_header header;
+
+	header.csr = readq(hdr);
+
+	if (header.id >= ARRAY_SIZE(fme_features)) {
+		dev_info(&binfo->pdev->dev, "FME feature id %x is not supported yet.\n",
+			 header.id);
+		return 0;
+	}
+
+	return create_feature_instance(binfo, hdr, &fme_features[header.id]);
+}
+
+static int parse_feature_port(struct build_feature_devs_info *binfo,
+			     void __iomem *start)
+{
+	int ret;
+
+	ret = build_info_create_dev(binfo, PORT_ID, port_feature_num(),
+					FPGA_FEATURE_DEV_PORT);
+	if (ret)
+		return ret;
+
+	return create_feature_instance(binfo, start,
+				       &port_features[PORT_FEATURE_ID_HEADER]);
+}
+
+static void enable_port_uafu(struct build_feature_devs_info *binfo,
+			     void __iomem *start)
+{
+	enum port_feature_id id = PORT_FEATURE_ID_UAFU;
+	struct feature_port_header *port_hdr;
+	struct feature_port_capability capability;
+
+	port_hdr = (struct feature_port_header *)start;
+	capability.csr = readq(&port_hdr->capability);
+	port_features[id].resource_size = capability.mmio_size << 10;
+
+	/*
+	 * To enable User AFU, driver needs to clear reset bit on related port,
+	 * otherwise the mmio space of this user AFU will be invalid.
+	 */
+	if (port_features[id].resource_size)
+		fpga_port_reset(binfo->feature_dev);
+}
+
+static int parse_feature_port_private(struct build_feature_devs_info *binfo,
+				      struct feature_header *hdr)
+{
+	struct feature_header header;
+	enum port_feature_id id;
+
+	header.csr = readq(hdr);
+	/*
+	 * the region of port feature id is [0x10, 0x13], + 1 to reserve 0
+	 * which is dedicated for port-hdr.
+	 */
+	id = (header.id & 0x000f) + 1;
+
+	if (id >= ARRAY_SIZE(port_features)) {
+		dev_info(&binfo->pdev->dev, "Port feature id %x is not supported yet.\n",
+			 header.id);
+		return 0;
+	}
+
+	return create_feature_instance(binfo, hdr, &port_features[id]);
+}
+
+static int parse_feature_port_uafu(struct build_feature_devs_info *binfo,
+				 struct feature_header *hdr)
+{
+	enum port_feature_id id = PORT_FEATURE_ID_UAFU;
+	int ret;
+
+	if (port_features[id].resource_size) {
+		ret = create_feature_instance(binfo, hdr, &port_features[id]);
+		port_features[id].resource_size = 0;
+	} else {
+		dev_err(&binfo->pdev->dev, "the uafu feature header is mis-configured.\n");
+		ret = -EINVAL;
+	}
+
+	return ret;
+}
+
+static int parse_feature_afus(struct build_feature_devs_info *binfo,
+			      struct feature_header *hdr)
+{
+	int ret;
+	struct feature_afu_header *afu_hdr, header;
+	void __iomem *start;
+	void __iomem *end = binfo->ioend;
+
+	start = hdr;
+	for (; start < end; start += header.next_afu) {
+		if (end - start < (sizeof(*afu_hdr) + sizeof(*hdr)))
+			return -EINVAL;
+
+		hdr = start;
+		afu_hdr = (struct feature_afu_header *) (hdr + 1);
+		header.csr = readq(&afu_hdr->csr);
+
+		if (feature_is_fme(afu_hdr)) {
+			ret = parse_feature_fme(binfo, hdr);
+			binfo->pfme_hdr = hdr;
+			if (ret)
+				return ret;
+		} else if (feature_is_port(afu_hdr)) {
+			ret = parse_feature_port(binfo, hdr);
+			enable_port_uafu(binfo, hdr);
+			if (ret)
+				return ret;
+		} else if (feature_is_UAFU(binfo)) {
+			ret = parse_feature_port_uafu(binfo, hdr);
+			if (ret)
+				return ret;
+		} else
+			dev_info(&binfo->pdev->dev, "AFU GUID %pUl is not supported yet.\n",
+				 afu_hdr->guid.b);
+
+		if (!header.next_afu)
+			break;
+	}
+
+	return 0;
+}
+
+static int parse_feature_private(struct build_feature_devs_info *binfo,
+				 struct feature_header *hdr)
+{
+	struct feature_header header;
+
+	header.csr = readq(hdr);
+
+	if (!binfo->feature_dev) {
+		dev_err(&binfo->pdev->dev, "the private feature %x does not belong to any AFU.\n",
+			header.id);
+		return -EINVAL;
+	}
+
+	switch (feature_dev_id_type(binfo->feature_dev)) {
+	case FME_ID:
+		return parse_feature_fme_private(binfo, hdr);
+	case PORT_ID:
+		return parse_feature_port_private(binfo, hdr);
+	default:
+		dev_info(&binfo->pdev->dev, "private feature %x belonging to AFU %s is not supported yet.\n",
+			 header.id, binfo->feature_dev->name);
+	}
+	return 0;
+}
+
+static int parse_feature(struct build_feature_devs_info *binfo,
+			 struct feature_header *hdr)
+{
+	struct feature_header header;
+	int ret = 0;
+
+	header.csr = readq(hdr);
+
+	switch (header.type) {
+	case FEATURE_TYPE_AFU:
+		ret = parse_feature_afus(binfo, hdr);
+		break;
+	case FEATURE_TYPE_PRIVATE:
+		ret = parse_feature_private(binfo, hdr);
+		break;
+	default:
+		dev_info(&binfo->pdev->dev,
+			 "Feature Type %x is not supported.\n", hdr->type);
+	};
+
+	return ret;
+}
+
+static int
+parse_feature_list(struct build_feature_devs_info *binfo, void __iomem *start)
+{
+	struct feature_header *hdr, header;
+	void __iomem *end = binfo->ioend;
+	int ret = 0;
+
+	for (; start < end; start += header.next_header_offset) {
+		if (end - start < sizeof(*hdr)) {
+			dev_err(&binfo->pdev->dev, "The region is too small to contain a feature.\n");
+			ret =  -EINVAL;
+			break;
+		}
+
+		hdr = (struct feature_header *)start;
+		ret = parse_feature(binfo, hdr);
+		if (ret)
+			break;
+
+		header.csr = readq(hdr);
+		if (!header.next_header_offset)
+			break;
+	}
+
+	return ret;
+}
+
+static int parse_ports_from_fme(struct build_feature_devs_info *binfo)
+{
+	struct feature_fme_header *fme_hdr;
+	struct feature_fme_port port;
+	int i = 0, ret = 0;
+
+	if (binfo->pfme_hdr == NULL) {
+		dev_dbg(&binfo->pdev->dev, "VF is detected.\n");
+		return ret;
+	}
+
+	fme_hdr = binfo->pfme_hdr;
+
+	do {
+		port.csr = readq(&fme_hdr->port[i]);
+		if (!port.port_implemented)
+			break;
+
+		ret = parse_switch_to(binfo, port.port_bar);
+		if (ret)
+			break;
+
+		ret = parse_feature_list(binfo,
+				binfo->ioaddr + port.port_offset);
+		if (ret)
+			break;
+	} while (++i < MAX_FPGA_PORT_NUM);
+
+	return ret;
+}
+
+static int create_init_drvdata(struct pci_dev *pdev)
+{
+	struct cci_drvdata *drvdata;
+
+	drvdata = devm_kzalloc(&pdev->dev, sizeof(*drvdata), GFP_KERNEL);
+	if (!drvdata)
+		return -ENOMEM;
+
+	mutex_init(&drvdata->lock);
+	INIT_LIST_HEAD(&drvdata->port_dev_list);
+	INIT_LIST_HEAD(&drvdata->regions);
+
+	dev_set_drvdata(&pdev->dev, drvdata);
+	return 0;
+}
+
+static void destroy_drvdata(struct pci_dev *pdev)
+{
+	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
+
+	if (drvdata->fme_dev) {
+		/* fme device should be unregistered first. */
+		WARN_ON(device_is_registered(drvdata->fme_dev));
+		free_fpga_id(FME_ID, to_platform_device(drvdata->fme_dev)->id);
+		put_device(drvdata->fme_dev);
+	}
+
+	cci_pci_remove_port_devs(pdev);
+	cci_pci_release_regions(pdev);
+	dev_set_drvdata(&pdev->dev, NULL);
+	devm_kfree(&pdev->dev, drvdata);
+}
+
+static int cci_pci_create_feature_devs(struct pci_dev *pdev)
+{
+	struct build_feature_devs_info *binfo;
+	int ret;
+
+	binfo = build_info_alloc_and_init(pdev);
+	if (!binfo)
+		return -ENOMEM;
+
+	binfo->parent_dev = fpga_dev_create(&pdev->dev, INTEL_FPGA_DEV);
+	if (IS_ERR(binfo->parent_dev)) {
+		ret = PTR_ERR(binfo->parent_dev);
+		goto free_binfo_exit;
+	}
+
+	ret = parse_start(binfo);
+	if (ret)
+		goto free_binfo_exit;
+
+	ret = parse_feature_list(binfo, binfo->ioaddr);
+	if (ret)
+		goto free_binfo_exit;
+
+	ret = parse_ports_from_fme(binfo);
+	if (ret)
+		goto free_binfo_exit;
+
+	ret = build_info_commit_dev(binfo);
+	if (ret)
+		goto free_binfo_exit;
+
+	/*
+	 * everything is okay, reset ->parent_dev to stop it being
+	 * freed by build_info_free()
+	 */
+	binfo->parent_dev = NULL;
+
+free_binfo_exit:
+	build_info_free(binfo);
+	return ret;
+}
+
 /* PCI Device ID */
 #define PCIe_DEVICE_ID_PF_INT_5_X	0xBCBD
 #define PCIe_DEVICE_ID_PF_INT_6_X	0xBCC0
@@ -81,9 +898,18 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
 		goto release_region_exit;
 	}
 
-	/* TODO: create and add the platform device per feature list */
+	ret = create_init_drvdata(pcidev);
+	if (ret)
+		goto release_region_exit;
+
+	ret = cci_pci_create_feature_devs(pcidev);
+	if (ret)
+		goto destroy_drvdata_exit;
+
 	return 0;
 
+destroy_drvdata_exit:
+	destroy_drvdata(pcidev);
 release_region_exit:
 	pci_release_regions(pcidev);
 disable_error_report_exit:
@@ -94,6 +920,8 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
 
 static void cci_pci_remove(struct pci_dev *pcidev)
 {
+	remove_all_devs(pcidev);
+	destroy_drvdata(pcidev);
 	pci_release_regions(pcidev);
 	pci_disable_pcie_error_reporting(pcidev);
 	pci_disable_device(pcidev);
@@ -108,14 +936,23 @@ static void cci_pci_remove(struct pci_dev *pcidev)
 
 static int __init ccidrv_init(void)
 {
+	int ret;
+
 	pr_info("Intel(R) FPGA PCIe Driver: Version %s\n", DRV_VERSION);
 
-	return pci_register_driver(&cci_pci_driver);
+	fpga_ids_init();
+
+	ret = pci_register_driver(&cci_pci_driver);
+	if (ret)
+		fpga_ids_destroy();
+
+	return ret;
 }
 
 static void __exit ccidrv_exit(void)
 {
 	pci_unregister_driver(&cci_pci_driver);
+	fpga_ids_destroy();
 }
 
 module_init(ccidrv_init);
-- 
1.8.3.1

[PATCH v2 08/22] fpga: intel: pcie: add chardev support for feature devices

[Wu Hao]

From: Xiao Guangrong <guangrong.xiao@linux.intel.com>

For feature devices drivers, both the FPGA Management Engine (FME) and
Accelerated Function Unit (AFU) driver need to expose user interfaces via
the device file, for example, mmap and ioctls.

This patch adds chardev support in the pcie driver for feature devices,
FME and AFU. It reserves the chardev regions for FME and AFU, and provide
interfaces for FME and AFU driver to register their device file operations.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
v2: rebased
---
 drivers/fpga/intel-feature-dev.c | 76 ++++++++++++++++++++++++++++++++++++++++
 drivers/fpga/intel-feature-dev.h | 16 +++++++++
 drivers/fpga/intel-pcie.c        | 18 +++++++++-
 3 files changed, 109 insertions(+), 1 deletion(-)

diff --git a/drivers/fpga/intel-feature-dev.c b/drivers/fpga/intel-feature-dev.c
index 68f9cba..45788fe 100644
--- a/drivers/fpga/intel-feature-dev.c
+++ b/drivers/fpga/intel-feature-dev.c
@@ -51,6 +51,82 @@ int port_feature_num(void)
 	return PORT_FEATURE_ID_MAX;
 }
 
+struct fpga_chardev_info {
+	const char *name;
+	dev_t devt;
+};
+
+/* indexed by enum fpga_devt_type */
+struct fpga_chardev_info fpga_chrdevs[] = {
+	{.name = FPGA_FEATURE_DEV_FME},		/* FPGA_DEVT_FME */
+	{.name = FPGA_FEATURE_DEV_PORT},	/* FPGA_DEVT_AFU */
+};
+
+void fpga_chardev_uinit(void)
+{
+	int i;
+
+	for (i = 0; i < FPGA_DEVT_MAX; i++)
+		if (MAJOR(fpga_chrdevs[i].devt)) {
+			unregister_chrdev_region(fpga_chrdevs[i].devt,
+						 MINORMASK);
+			fpga_chrdevs[i].devt = MKDEV(0, 0);
+		}
+}
+
+int fpga_chardev_init(void)
+{
+	int i, ret;
+
+	for (i = 0; i < FPGA_DEVT_MAX; i++) {
+		ret = alloc_chrdev_region(&fpga_chrdevs[i].devt, 0, MINORMASK,
+					  fpga_chrdevs[i].name);
+		if (ret)
+			goto exit;
+	}
+
+	return 0;
+
+exit:
+	fpga_chardev_uinit();
+	return ret;
+}
+
+dev_t fpga_get_devt(enum fpga_devt_type type, int id)
+{
+	WARN_ON(type >= FPGA_DEVT_MAX);
+
+	return MKDEV(MAJOR(fpga_chrdevs[type].devt), id);
+}
+
+int fpga_register_dev_ops(struct platform_device *pdev,
+			  const struct file_operations *fops,
+			  struct module *owner)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	cdev_init(&pdata->cdev, fops);
+	pdata->cdev.owner = owner;
+
+	/*
+	 * set parent to the feature device so that its refcount is
+	 * decreased after the last refcount of cdev is gone, that
+	 * makes sure the feature device is valid during device
+	 * file's life-cycle.
+	 */
+	pdata->cdev.kobj.parent = &pdev->dev.kobj;
+	return cdev_add(&pdata->cdev, pdev->dev.devt, 1);
+}
+EXPORT_SYMBOL_GPL(fpga_register_dev_ops);
+
+void fpga_unregister_dev_ops(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	cdev_del(&pdata->cdev);
+}
+EXPORT_SYMBOL_GPL(fpga_unregister_dev_ops);
+
 int fpga_port_id(struct platform_device *pdev)
 {
 	struct feature_port_header *port_hdr;
diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
index f67784a..d64a51e 100644
--- a/drivers/fpga/intel-feature-dev.h
+++ b/drivers/fpga/intel-feature-dev.h
@@ -17,6 +17,7 @@
 #define __INTEL_FPGA_FEATURE_H
 
 #include <linux/fs.h>
+#include <linux/cdev.h>
 #include <linux/pci.h>
 #include <linux/uuid.h>
 #include <linux/delay.h>
@@ -230,6 +231,7 @@ struct feature_platform_data {
 	/* list the feature dev to cci_drvdata->port_dev_list. */
 	struct list_head node;
 	struct mutex lock;
+	struct cdev cdev;
 	struct platform_device *dev;
 	unsigned int disable_count;	/* count for port disable */
 
@@ -276,6 +278,20 @@ static inline int feature_platform_data_size(const int num)
 struct feature_platform_data *
 feature_platform_data_alloc_and_init(struct platform_device *dev, int num);
 
+enum fpga_devt_type {
+	FPGA_DEVT_FME,
+	FPGA_DEVT_PORT,
+	FPGA_DEVT_MAX,
+};
+
+void fpga_chardev_uinit(void);
+int fpga_chardev_init(void);
+dev_t fpga_get_devt(enum fpga_devt_type type, int id);
+int fpga_register_dev_ops(struct platform_device *pdev,
+			  const struct file_operations *fops,
+			  struct module *owner);
+void fpga_unregister_dev_ops(struct platform_device *pdev);
+
 int fpga_port_id(struct platform_device *pdev);
 
 static inline int fpga_port_check_id(struct platform_device *pdev,
diff --git a/drivers/fpga/intel-pcie.c b/drivers/fpga/intel-pcie.c
index 70b8284..54c0e3a 100644
--- a/drivers/fpga/intel-pcie.c
+++ b/drivers/fpga/intel-pcie.c
@@ -274,8 +274,12 @@ static int build_info_commit_dev(struct build_feature_devs_info *binfo)
 	struct platform_device *fdev;
 	struct resource *res;
 	struct feature_platform_data *pdata;
+	enum fpga_devt_type devt_type = FPGA_DEVT_FME;
 	int ret;
 
+	if (type == PORT_ID)
+		devt_type = FPGA_DEVT_PORT;
+
 	/* we will create a new device, commit current device first */
 	ret = build_info_commit_dev(binfo);
 	if (ret)
@@ -294,6 +298,7 @@ static int build_info_commit_dev(struct build_feature_devs_info *binfo)
 		return fdev->id;
 
 	fdev->dev.parent = &binfo->parent_dev->dev;
+	fdev->dev.devt = fpga_get_devt(devt_type, fdev->id);
 
 	/*
 	 * we do not need to care for the memory which is associated with
@@ -942,16 +947,27 @@ static int __init ccidrv_init(void)
 
 	fpga_ids_init();
 
+	ret = fpga_chardev_init();
+	if (ret)
+		goto exit_ids;
+
 	ret = pci_register_driver(&cci_pci_driver);
 	if (ret)
-		fpga_ids_destroy();
+		goto exit_chardev;
 
+	return 0;
+
+exit_chardev:
+	fpga_chardev_uinit();
+exit_ids:
+	fpga_ids_destroy();
 	return ret;
 }
 
 static void __exit ccidrv_exit(void)
 {
 	pci_unregister_driver(&cci_pci_driver);
+	fpga_chardev_uinit();
 	fpga_ids_destroy();
 }
 
-- 
1.8.3.1

[PATCH v2 09/22] fpga: intel: pcie: adds fpga_for_each_port callback for fme device

[Wu Hao]

For FPGA Management Engine (FME), it requires fpga_for_each_port callback
for actions on ports, so export this function from PCIe driver by adding
the callback to the platform data.

Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
v2: rebased
---
 drivers/fpga/intel-feature-dev.h |  9 +++++++++
 drivers/fpga/intel-pcie.c        | 24 ++++++++++++++++++++++++
 2 files changed, 33 insertions(+)

diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
index d64a51e..06b3fb6 100644
--- a/drivers/fpga/intel-feature-dev.h
+++ b/drivers/fpga/intel-feature-dev.h
@@ -235,6 +235,9 @@ struct feature_platform_data {
 	struct platform_device *dev;
 	unsigned int disable_count;	/* count for port disable */
 
+	struct platform_device *(*fpga_for_each_port)(struct platform_device *,
+			void *, int (*match)(struct platform_device *, void *));
+
 	int num;			/* number of features */
 	struct feature features[0];
 };
@@ -354,4 +357,10 @@ static inline int fpga_port_reset(struct platform_device *pdev)
 
 	return pdata->features[index].ioaddr;
 }
+
+static inline struct device *
+fpga_feature_dev_to_pcidev(struct platform_device *dev)
+{
+	return dev->dev.parent->parent;
+}
 #endif
diff --git a/drivers/fpga/intel-pcie.c b/drivers/fpga/intel-pcie.c
index 54c0e3a..86ea5c7 100644
--- a/drivers/fpga/intel-pcie.c
+++ b/drivers/fpga/intel-pcie.c
@@ -209,6 +209,27 @@ static int parse_switch_to(struct build_feature_devs_info *binfo, int bar)
 	return parse_start_from(binfo, bar);
 }
 
+static struct platform_device *fpga_for_each_port(struct platform_device *pdev,
+		     void *data, int (*match)(struct platform_device *, void *))
+{
+	struct device *pci_dev = fpga_feature_dev_to_pcidev(pdev);
+	struct cci_drvdata *drvdata = dev_get_drvdata(pci_dev);
+	struct feature_platform_data *pdata;
+	struct platform_device *port_dev;
+
+	mutex_lock(&drvdata->lock);
+	list_for_each_entry(pdata, &drvdata->port_dev_list, node) {
+		port_dev = pdata->dev;
+
+		if (match(port_dev, data) && get_device(&port_dev->dev))
+			goto exit;
+	}
+	port_dev = NULL;
+exit:
+	mutex_unlock(&drvdata->lock);
+	return port_dev;
+}
+
 static struct build_feature_devs_info *
 build_info_alloc_and_init(struct pci_dev *pdev)
 {
@@ -310,6 +331,9 @@ static int build_info_commit_dev(struct build_feature_devs_info *binfo)
 	if (!pdata)
 		return -ENOMEM;
 
+	if (type == FME_ID)
+		pdata->fpga_for_each_port = fpga_for_each_port;
+
 	/*
 	 * the count should be initialized to 0 to make sure
 	 *__fpga_port_enable() following __fpga_port_disable()
-- 
1.8.3.1

[PATCH v2 10/22] fpga: intel: add feature device infrastructure

[Wu Hao]

From: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>

This patch abstracts the common operations of the sub features, and defines
the feature_ops data structure, including init, uinit and ioctl function
pointers. And this patch adds some common helper functions for FME and AFU
drivers, e.g feature_dev_use_begin/end which are used to ensure exclusive
usage of the feature device file.

Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Zhang Yi <yi.z.zhang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
v2: rebased
---
 drivers/fpga/intel-feature-dev.c | 66 +++++++++++++++++++++++++++++++++++++
 drivers/fpga/intel-feature-dev.h | 71 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 137 insertions(+)

diff --git a/drivers/fpga/intel-feature-dev.c b/drivers/fpga/intel-feature-dev.c
index 45788fe..833bfa9 100644
--- a/drivers/fpga/intel-feature-dev.c
+++ b/drivers/fpga/intel-feature-dev.c
@@ -51,6 +51,72 @@ int port_feature_num(void)
 	return PORT_FEATURE_ID_MAX;
 }
 
+int fme_feature_to_resource_index(int feature_id)
+{
+	WARN_ON(feature_id >= FME_FEATURE_ID_MAX);
+	return feature_id;
+}
+
+void fpga_dev_feature_uinit(struct platform_device *pdev)
+{
+	struct feature *feature;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	fpga_dev_for_each_feature(pdata, feature)
+		if (feature->ops) {
+			feature->ops->uinit(pdev, feature);
+			feature->ops = NULL;
+		}
+}
+EXPORT_SYMBOL_GPL(fpga_dev_feature_uinit);
+
+static int
+feature_instance_init(struct platform_device *pdev,
+		      struct feature_platform_data *pdata,
+		      struct feature *feature, struct feature_driver *drv)
+{
+	int ret;
+
+	WARN_ON(!feature->ioaddr);
+
+	ret = drv->ops->init(pdev, feature);
+	if (ret)
+		return ret;
+
+	feature->ops = drv->ops;
+	return ret;
+}
+
+int fpga_dev_feature_init(struct platform_device *pdev,
+			  struct feature_driver *feature_drvs)
+{
+	struct feature *feature;
+	struct feature_driver *drv = feature_drvs;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	int ret;
+
+	while (drv->ops) {
+		fpga_dev_for_each_feature(pdata, feature) {
+			/* skip the feature which is not initialized. */
+			if (!feature->name)
+				continue;
+
+			if (!strcmp(drv->name, feature->name)) {
+				ret = feature_instance_init(pdev, pdata,
+							    feature, drv);
+				if (ret)
+					goto exit;
+			}
+		}
+		drv++;
+	}
+	return 0;
+exit:
+	fpga_dev_feature_uinit(pdev);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(fpga_dev_feature_init);
+
 struct fpga_chardev_info {
 	const char *name;
 	dev_t devt;
diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
index 06b3fb6..635b857 100644
--- a/drivers/fpga/intel-feature-dev.h
+++ b/drivers/fpga/intel-feature-dev.h
@@ -221,12 +221,20 @@ struct feature_port_stp {
 
 #pragma pack()
 
+struct feature_driver {
+	const char *name;
+	struct feature_ops *ops;
+};
+
 struct feature {
 	const char *name;
 	int resource_index;
 	void __iomem *ioaddr;
+	struct feature_ops *ops;
 };
 
+#define DEV_STATUS_IN_USE	0
+
 struct feature_platform_data {
 	/* list the feature dev to cci_drvdata->port_dev_list. */
 	struct list_head node;
@@ -234,6 +242,9 @@ struct feature_platform_data {
 	struct cdev cdev;
 	struct platform_device *dev;
 	unsigned int disable_count;	/* count for port disable */
+	unsigned long dev_status;
+
+	void *private;			/* ptr to feature dev private data */
 
 	struct platform_device *(*fpga_for_each_port)(struct platform_device *,
 			void *, int (*match)(struct platform_device *, void *));
@@ -242,6 +253,38 @@ struct feature_platform_data {
 	struct feature features[0];
 };
 
+static inline int feature_dev_use_begin(struct feature_platform_data *pdata)
+{
+	/* Test and set IN_USE flags to ensure file is exclusively used */
+	if (test_and_set_bit_lock(DEV_STATUS_IN_USE, &pdata->dev_status))
+		return -EBUSY;
+
+	return 0;
+}
+
+static inline void feature_dev_use_end(struct feature_platform_data *pdata)
+{
+	clear_bit_unlock(DEV_STATUS_IN_USE, &pdata->dev_status);
+}
+
+static inline void
+fpga_pdata_set_private(struct feature_platform_data *pdata, void *private)
+{
+	pdata->private = private;
+}
+
+static inline void *fpga_pdata_get_private(struct feature_platform_data *pdata)
+{
+	return pdata->private;
+}
+
+struct feature_ops {
+	int (*init)(struct platform_device *pdev, struct feature *feature);
+	void (*uinit)(struct platform_device *pdev, struct feature *feature);
+	long (*ioctl)(struct platform_device *pdev, struct feature *feature,
+				unsigned int cmd, unsigned long arg);
+};
+
 enum fme_feature_id {
 	FME_FEATURE_ID_HEADER = 0x0,
 	FME_FEATURE_ID_THERMAL_MGMT = 0x1,
@@ -281,6 +324,10 @@ static inline int feature_platform_data_size(const int num)
 struct feature_platform_data *
 feature_platform_data_alloc_and_init(struct platform_device *dev, int num);
 
+void fpga_dev_feature_uinit(struct platform_device *pdev);
+int fpga_dev_feature_init(struct platform_device *pdev,
+			  struct feature_driver *feature_drvs);
+
 enum fpga_devt_type {
 	FPGA_DEVT_FME,
 	FPGA_DEVT_PORT,
@@ -350,6 +397,15 @@ static inline int fpga_port_reset(struct platform_device *pdev)
 	return ret;
 }
 
+static inline
+struct platform_device *fpga_inode_to_feature_dev(struct inode *inode)
+{
+	struct feature_platform_data *pdata;
+
+	pdata = container_of(inode->i_cdev, struct feature_platform_data, cdev);
+	return pdata->dev;
+}
+
 static inline void __iomem *
 get_feature_ioaddr_by_index(struct device *dev, int index)
 {
@@ -358,9 +414,24 @@ static inline int fpga_port_reset(struct platform_device *pdev)
 	return pdata->features[index].ioaddr;
 }
 
+static inline bool is_feature_present(struct device *dev, int index)
+{
+	return !!get_feature_ioaddr_by_index(dev, index);
+}
+
 static inline struct device *
 fpga_feature_dev_to_pcidev(struct platform_device *dev)
 {
 	return dev->dev.parent->parent;
 }
+
+static inline struct device *
+fpga_pdata_to_pcidev(struct feature_platform_data *pdata)
+{
+	return fpga_feature_dev_to_pcidev(pdata->dev);
+}
+
+#define fpga_dev_for_each_feature(pdata, feature)			    \
+	for ((feature) = (pdata)->features;				    \
+	   (feature) < (pdata)->features + (pdata)->num; (feature)++)
 #endif
-- 
1.8.3.1

[PATCH v2 11/22] fpga: intel: add FPGA Management Engine driver basic framework

[Wu Hao]

From: Kang Luwei <luwei.kang@intel.com>

The FPGA Management Engine (FME) provides power, thermal management,
performance counters, partial reconfiguration and other functions. For each
function, it is packaged into a private feature linked to the FME feature
device in the 'Device Feature List'. It's a platform device created by
Intel FPGA PCIe driver.

This patch adds the basic framework of FME platform driver. It defines
sub feature drivers to handle the different sub features, including init,
uinit and ioctl. It also registers the file operations for the device file.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Kang Luwei <luwei.kang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
    switched to GPLv2 license.
---
 drivers/fpga/Kconfig          |   8 +++
 drivers/fpga/Makefile         |   2 +
 drivers/fpga/intel-fme-main.c | 156 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 166 insertions(+)
 create mode 100644 drivers/fpga/intel-fme-main.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 3f3b7f4..b91458f 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig
@@ -143,6 +143,14 @@ config INTEL_FPGA_PCI
 
 	  To compile this as a module, choose M here.
 
+config INTEL_FPGA_FME
+	tristate "Intel FPGA FME Driver"
+	depends on INTEL_FPGA_PCI
+	help
+	  This is the driver for FPGA Management Engine which implements
+	  all FPGA platform level management features. There shall be 1
+	  FME per Intel FPGA.
+
 endif
 
 endif # FPGA
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index ad24b3d..09bf281 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -30,5 +30,7 @@ obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
 
 # Intel FPGA Support
 obj-$(CONFIG_INTEL_FPGA_PCI)		+= intel-fpga-pci.o
+obj-$(CONFIG_INTEL_FPGA_FME)		+= intel-fpga-fme.o
 
 intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
+intel-fpga-fme-objs := intel-fme-main.o
diff --git a/drivers/fpga/intel-fme-main.c b/drivers/fpga/intel-fme-main.c
new file mode 100644
index 0000000..c16cf81
--- /dev/null
+++ b/drivers/fpga/intel-fme-main.c
@@ -0,0 +1,156 @@
+/*
+ * Driver for Intel FPGA Management Engine (FME)
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+#include "intel-feature-dev.h"
+
+static int fme_hdr_init(struct platform_device *pdev, struct feature *feature)
+{
+	dev_dbg(&pdev->dev, "FME HDR Init.\n");
+	return 0;
+}
+
+static void fme_hdr_uinit(struct platform_device *pdev, struct feature *feature)
+{
+	dev_dbg(&pdev->dev, "FME HDR UInit.\n");
+}
+
+struct feature_ops fme_hdr_ops = {
+	.init = fme_hdr_init,
+	.uinit = fme_hdr_uinit,
+};
+
+static struct feature_driver fme_feature_drvs[] = {
+	{
+		.name = FME_FEATURE_HEADER,
+		.ops = &fme_hdr_ops,
+	},
+	{
+		.ops = NULL,
+	},
+};
+
+static int fme_open(struct inode *inode, struct file *filp)
+{
+	struct platform_device *fdev = fpga_inode_to_feature_dev(inode);
+	struct feature_platform_data *pdata = dev_get_platdata(&fdev->dev);
+	int ret;
+
+	if (WARN_ON(!pdata))
+		return -ENODEV;
+
+	ret = feature_dev_use_begin(pdata);
+	if (ret)
+		return ret;
+
+	dev_dbg(&fdev->dev, "Device File Open\n");
+	filp->private_data = pdata;
+	return 0;
+}
+
+static int fme_release(struct inode *inode, struct file *filp)
+{
+	struct feature_platform_data *pdata = filp->private_data;
+	struct platform_device *pdev = pdata->dev;
+
+	dev_dbg(&pdev->dev, "Device File Release\n");
+	feature_dev_use_end(pdata);
+	return 0;
+}
+
+static long fme_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
+{
+	struct feature_platform_data *pdata = filp->private_data;
+	struct platform_device *pdev = pdata->dev;
+	struct feature *f;
+	long ret;
+
+	dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
+
+	switch (cmd) {
+	default:
+		/*
+		 * Let sub-feature's ioctl function to handle the cmd
+		 * Sub-feature's ioctl returns -ENODEV when cmd is not
+		 * handled in this sub feature, and returns 0 and other
+		 * error code if cmd is handled.
+		 */
+		fpga_dev_for_each_feature(pdata, f) {
+			if (f->ops && f->ops->ioctl) {
+				ret = f->ops->ioctl(pdev, f, cmd, arg);
+				if (ret == -ENODEV)
+					continue;
+				else
+					return ret;
+			}
+		}
+	}
+
+	return -EINVAL;
+}
+
+static const struct file_operations fme_fops = {
+	.owner		= THIS_MODULE,
+	.open		= fme_open,
+	.release	= fme_release,
+	.unlocked_ioctl = fme_ioctl,
+};
+
+static int fme_probe(struct platform_device *pdev)
+{
+	int ret;
+
+	ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+	if (ret)
+		goto exit;
+
+	ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
+	if (ret)
+		goto feature_uinit;
+
+	return 0;
+
+feature_uinit:
+	fpga_dev_feature_uinit(pdev);
+exit:
+	return ret;
+}
+
+static int fme_remove(struct platform_device *pdev)
+{
+	fpga_dev_feature_uinit(pdev);
+	fpga_unregister_dev_ops(pdev);
+	return 0;
+}
+
+static struct platform_driver fme_driver = {
+	.driver	= {
+		.name    = FPGA_FEATURE_DEV_FME,
+	},
+	.probe   = fme_probe,
+	.remove  = fme_remove,
+};
+
+module_platform_driver(fme_driver);
+
+MODULE_DESCRIPTION("Intel FPGA Management Engine driver");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:intel-fpga-fme");
-- 
1.8.3.1

[PATCH v2 12/22] fpga: intel: fme: add header sub feature support

[Wu Hao]

From: Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

The header register set is always present for FPGA Management Engine (FME),
this patch implements init and uinit function for header sub feature and
introduce several read-only sysfs interfaces for the capability and status.

Sysfs interfaces:
* /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/ports_num
  Read-only. Number of ports implemented

* /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/bitstream_id
  Read-only. Blue Bitstream identifier number

* /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/bitstream_metadata
  Read-only. Blue Bitstream meta data

Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
v2: add sysfs documentation
---
 .../ABI/testing/sysfs-platform-intel-fpga-fme      | 19 ++++++++
 drivers/fpga/intel-feature-dev.h                   |  3 ++
 drivers/fpga/intel-fme-main.c                      | 55 ++++++++++++++++++++++
 3 files changed, 77 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-platform-intel-fpga-fme

diff --git a/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme b/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme
new file mode 100644
index 0000000..783cfa9
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme
@@ -0,0 +1,19 @@
+What:		/sys/bus/platform/devices/intel-fpga-fme.0/ports_num
+Date:		June 2017
+KernelVersion:  4.12
+Contact:	Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+Description:	Read-only. One Intel FPGA device may have more than 1
+		port/Accelerator Function Unit (AFU). It returns the
+		number of ports on the FPGA device when read it.
+
+What:		/sys/bus/platform/devices/intel-fpga-fme.0/bitstream_id
+Date:		June 2017
+KernelVersion:  4.12
+Contact:	Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+Description:	Read-only. It returns Blue Bitstream identifier number.
+
+What:		/sys/bus/platform/devices/intel-fpga-fme.0/bitstream_meta
+Date:		June 2017
+KernelVersion:  4.12
+Contact:	Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+Description:	Read-only. It returns Blue Bitstream meta data.
diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
index 635b857..3f97b75 100644
--- a/drivers/fpga/intel-feature-dev.h
+++ b/drivers/fpga/intel-feature-dev.h
@@ -138,6 +138,9 @@ struct feature_fme_header {
 	u64 rsvd[2];
 	struct feature_fme_capability capability;
 	struct feature_fme_port port[MAX_FPGA_PORT_NUM];
+	u64 rsvd1;
+	u64 bitstream_id;
+	u64 bitstream_md;
 };
 
 /* FME Thermal Sub Feature Register Set */
diff --git a/drivers/fpga/intel-fme-main.c b/drivers/fpga/intel-fme-main.c
index c16cf81..dfbb17c 100644
--- a/drivers/fpga/intel-fme-main.c
+++ b/drivers/fpga/intel-fme-main.c
@@ -21,15 +21,70 @@
 
 #include "intel-feature-dev.h"
 
+static ssize_t ports_num_show(struct device *dev,
+			      struct device_attribute *attr, char *buf)
+{
+	struct feature_fme_header *fme_hdr
+		= get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_HEADER);
+	struct feature_fme_capability fme_capability;
+
+	fme_capability.csr = readq(&fme_hdr->capability);
+
+	return scnprintf(buf, PAGE_SIZE, "%d\n", fme_capability.num_ports);
+}
+static DEVICE_ATTR_RO(ports_num);
+
+static ssize_t bitstream_id_show(struct device *dev,
+				 struct device_attribute *attr, char *buf)
+{
+	struct feature_fme_header *fme_hdr
+		= get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_HEADER);
+	u64 bitstream_id = readq(&fme_hdr->bitstream_id);
+
+	return scnprintf(buf, PAGE_SIZE, "0x%llx\n",
+				(unsigned long long)bitstream_id);
+}
+static DEVICE_ATTR_RO(bitstream_id);
+
+static ssize_t bitstream_metadata_show(struct device *dev,
+				       struct device_attribute *attr, char *buf)
+{
+	struct feature_fme_header *fme_hdr
+		= get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_HEADER);
+	u64 bitstream_md = readq(&fme_hdr->bitstream_md);
+
+	return scnprintf(buf, PAGE_SIZE, "0x%llx\n",
+				(unsigned long long)bitstream_md);
+}
+static DEVICE_ATTR_RO(bitstream_metadata);
+
+static const struct attribute *fme_hdr_attrs[] = {
+	&dev_attr_ports_num.attr,
+	&dev_attr_bitstream_id.attr,
+	&dev_attr_bitstream_metadata.attr,
+	NULL,
+};
+
 static int fme_hdr_init(struct platform_device *pdev, struct feature *feature)
 {
+	struct feature_fme_header *fme_hdr = feature->ioaddr;
+	int ret;
+
 	dev_dbg(&pdev->dev, "FME HDR Init.\n");
+	dev_dbg(&pdev->dev, "FME cap %llx.\n",
+				(unsigned long long)fme_hdr->capability.csr);
+
+	ret = sysfs_create_files(&pdev->dev.kobj, fme_hdr_attrs);
+	if (ret)
+		return ret;
+
 	return 0;
 }
 
 static void fme_hdr_uinit(struct platform_device *pdev, struct feature *feature)
 {
 	dev_dbg(&pdev->dev, "FME HDR UInit.\n");
+	sysfs_remove_files(&pdev->dev.kobj, fme_hdr_attrs);
 }
 
 struct feature_ops fme_hdr_ops = {
-- 
1.8.3.1

[PATCH v2 13/22] fpga: intel: fme: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support

[Wu Hao]

FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which
need to be supported by all feature devices drivers including FME and AFU.
Userspace application can use these ioctl interfaces to get the API info
and check if specific extension is supported or not in current driver.

This patch implements above 2 ioctls in Intel FPGA Management Engine (FME)
driver.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
v2: switched to GPLv2 license.
---
 Documentation/ioctl/ioctl-number.txt |  1 +
 drivers/fpga/intel-fme-main.c        | 12 +++++++++
 include/uapi/linux/intel-fpga.h      | 50 ++++++++++++++++++++++++++++++++++++
 3 files changed, 63 insertions(+)
 create mode 100644 include/uapi/linux/intel-fpga.h

diff --git a/Documentation/ioctl/ioctl-number.txt b/Documentation/ioctl/ioctl-number.txt
index 1e9fcb4..a0d8be5 100644
--- a/Documentation/ioctl/ioctl-number.txt
+++ b/Documentation/ioctl/ioctl-number.txt
@@ -324,6 +324,7 @@ Code  Seq#(hex)	Include File		Comments
 0xB3	00	linux/mmc/ioctl.h
 0xB4	00-0F	linux/gpio.h		<mailto:linux-gpio@vger.kernel.org>
 0xB5	00-0F	uapi/linux/rpmsg.h	<mailto:linux-remoteproc@vger.kernel.org>
+0xB6	all	linux/intel-fpga.h
 0xC0	00-0F	linux/usb/iowarrior.h
 0xCA	00-0F	uapi/misc/cxl.h
 0xCA	80-8F	uapi/scsi/cxlflash_ioctl.h
diff --git a/drivers/fpga/intel-fme-main.c b/drivers/fpga/intel-fme-main.c
index dfbb17c..9fd0baf 100644
--- a/drivers/fpga/intel-fme-main.c
+++ b/drivers/fpga/intel-fme-main.c
@@ -18,6 +18,7 @@
 
 #include <linux/kernel.h>
 #include <linux/module.h>
+#include <linux/intel-fpga.h>
 
 #include "intel-feature-dev.h"
 
@@ -102,6 +103,13 @@ struct feature_ops fme_hdr_ops = {
 	},
 };
 
+static long fme_ioctl_check_extension(struct feature_platform_data *pdata,
+				     unsigned long arg)
+{
+	/* No extension support for now */
+	return 0;
+}
+
 static int fme_open(struct inode *inode, struct file *filp)
 {
 	struct platform_device *fdev = fpga_inode_to_feature_dev(inode);
@@ -140,6 +148,10 @@ static long fme_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 	dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
 
 	switch (cmd) {
+	case FPGA_GET_API_VERSION:
+		return FPGA_API_VERSION;
+	case FPGA_CHECK_EXTENSION:
+		return fme_ioctl_check_extension(pdata, arg);
 	default:
 		/*
 		 * Let sub-feature's ioctl function to handle the cmd
diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
new file mode 100644
index 0000000..e340ef9
--- /dev/null
+++ b/include/uapi/linux/intel-fpga.h
@@ -0,0 +1,50 @@
+/*
+ * Header File for Intel FPGA User API
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Zhang Yi <yi.z.zhang@intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#ifndef _UAPI_LINUX_INTEL_FPGA_H
+#define _UAPI_LINUX_INTEL_FPGA_H
+
+#define FPGA_API_VERSION 0
+
+/*
+ * The IOCTL interface for Intel FPGA is designed for extensibility by
+ * embedding the structure length (argsz) and flags into structures passed
+ * between kernel and userspace. This design referenced the VFIO IOCTL
+ * interface (include/uapi/linux/vfio.h).
+ */
+
+#define FPGA_MAGIC 0xB6
+
+#define FPGA_BASE 0
+
+/**
+ * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
+ *
+ * Report the version of the driver API.
+ * Return: Driver API Version.
+ */
+
+#define FPGA_GET_API_VERSION	_IO(FPGA_MAGIC, FPGA_BASE + 0)
+
+/**
+ * FPGA_CHECK_EXTENSION - _IO(FPGA_MAGIC, FPGA_BASE + 1)
+ *
+ * Check whether an extension is supported.
+ * Return: 0 if not supported, otherwise the extension is supported.
+ */
+
+#define FPGA_CHECK_EXTENSION	_IO(FPGA_MAGIC, FPGA_BASE + 1)
+
+#endif /* _UAPI_INTEL_FPGA_H */
-- 
1.8.3.1

[PATCH v2 14/22] fpga: intel: fme: add partial reconfiguration sub feature support

[Wu Hao]

From: Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Partial Reconfiguration (PR) is the most important function for FME. It
allows reconfiguration for given Port/Accelerated Function Unit (AFU).

It creates platform devices for fpga-mgr, fpga-regions and fpga-bridges,
and invokes fpga-region's interface (fpga_region_program_fpga) for PR
operation once PR request received via ioctl. Below user space interface
is exposed by this sub feature.

Ioctl interface:
* FPGA_FME_PORT_PR
  Do partial reconfiguration per information from userspace, including
  target port(AFU), buffer size and address info. It returns error code
  to userspace if failed. For detailed PR error information, user needs
  to read fpga-mgr's status sysfs interface.

Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
    switched to GPLv2 license.
    removed status from FPGA_FME_PORT_PR ioctl data structure.
    added platform devices creation for fpga-mgr/fpga-region/fpga-bridge.
    switched to fpga-region interface fpga_region_program_fpga for PR.
    fixed comments from Alan Tull on FPGA_MGR_PARTIAL_RECONFIG flag usage.
    fixed kbuild warnings.
---
 drivers/fpga/Makefile           |   2 +-
 drivers/fpga/intel-fme-main.c   |  44 +++-
 drivers/fpga/intel-fme-pr.c     | 494 ++++++++++++++++++++++++++++++++++++++++
 drivers/fpga/intel-fme.h        |  57 +++++
 include/uapi/linux/intel-fpga.h |  27 +++
 5 files changed, 622 insertions(+), 2 deletions(-)
 create mode 100644 drivers/fpga/intel-fme-pr.c
 create mode 100644 drivers/fpga/intel-fme.h

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 09bf281..acda5ca 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -33,4 +33,4 @@ obj-$(CONFIG_INTEL_FPGA_PCI)		+= intel-fpga-pci.o
 obj-$(CONFIG_INTEL_FPGA_FME)		+= intel-fpga-fme.o
 
 intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
-intel-fpga-fme-objs := intel-fme-main.o
+intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
diff --git a/drivers/fpga/intel-fme-main.c b/drivers/fpga/intel-fme-main.c
index 9fd0baf..b35bed1 100644
--- a/drivers/fpga/intel-fme-main.c
+++ b/drivers/fpga/intel-fme-main.c
@@ -21,6 +21,7 @@
 #include <linux/intel-fpga.h>
 
 #include "intel-feature-dev.h"
+#include "intel-fme.h"
 
 static ssize_t ports_num_show(struct device *dev,
 			      struct device_attribute *attr, char *buf)
@@ -99,6 +100,10 @@ struct feature_ops fme_hdr_ops = {
 		.ops = &fme_hdr_ops,
 	},
 	{
+		.name = FME_FEATURE_PR_MGMT,
+		.ops = &pr_mgmt_ops,
+	},
+	{
 		.ops = NULL,
 	},
 };
@@ -180,14 +185,48 @@ static long fme_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 	.unlocked_ioctl = fme_ioctl,
 };
 
+static int fme_dev_init(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_fme *fme;
+
+	fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
+	if (!fme)
+		return -ENOMEM;
+
+	fme->pdata = pdata;
+
+	mutex_lock(&pdata->lock);
+	fpga_pdata_set_private(pdata, fme);
+	mutex_unlock(&pdata->lock);
+	return 0;
+}
+
+static void fme_dev_destroy(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_fme *fme;
+
+	mutex_lock(&pdata->lock);
+	fme = fpga_pdata_get_private(pdata);
+	fpga_pdata_set_private(pdata, NULL);
+	mutex_unlock(&pdata->lock);
+
+	devm_kfree(&pdev->dev, fme);
+}
+
 static int fme_probe(struct platform_device *pdev)
 {
 	int ret;
 
-	ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+	ret = fme_dev_init(pdev);
 	if (ret)
 		goto exit;
 
+	ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+	if (ret)
+		goto dev_destroy;
+
 	ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
 	if (ret)
 		goto feature_uinit;
@@ -196,6 +235,8 @@ static int fme_probe(struct platform_device *pdev)
 
 feature_uinit:
 	fpga_dev_feature_uinit(pdev);
+dev_destroy:
+	fme_dev_destroy(pdev);
 exit:
 	return ret;
 }
@@ -204,6 +245,7 @@ static int fme_remove(struct platform_device *pdev)
 {
 	fpga_dev_feature_uinit(pdev);
 	fpga_unregister_dev_ops(pdev);
+	fme_dev_destroy(pdev);
 	return 0;
 }
 
diff --git a/drivers/fpga/intel-fme-pr.c b/drivers/fpga/intel-fme-pr.c
new file mode 100644
index 0000000..cc40d2a
--- /dev/null
+++ b/drivers/fpga/intel-fme-pr.c
@@ -0,0 +1,494 @@
+/*
+ * Driver for Intel FPGA Management Engine (FME) Partial Reconfiguration
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
+ *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Joseph Grecco <joe.grecco-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Ananda Ravuri <ananda.ravuri-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Henry Mitchel <henry.mitchel-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/vmalloc.h>
+#include <linux/uaccess.h>
+#include <linux/fpga/fpga-mgr.h>
+#include <linux/fpga/fpga-bridge.h>
+#include <linux/fpga/fpga-region.h>
+#include <linux/intel-fpga.h>
+
+#include "intel-feature-dev.h"
+#include "intel-fme.h"
+
+static struct fme_region *
+find_fme_region_by_port_id(struct fpga_fme *fme, int port_id)
+{
+	struct fme_region *fme_region;
+
+	list_for_each_entry(fme_region, &fme->region_list, node)
+		if (fme_region->port_id == port_id)
+			return fme_region;
+
+	return NULL;
+}
+
+static int fpga_fme_region_match(struct device *dev, const void *data)
+{
+	return dev->parent == data;
+}
+
+static struct fpga_region *
+fpga_fme_region_find(struct fpga_fme *fme, int port_id)
+{
+	struct fme_region *fme_region;
+	struct fpga_region *region;
+
+	fme_region = find_fme_region_by_port_id(fme, port_id);
+	if (!fme_region)
+		return NULL;
+
+	region = fpga_region_class_find(NULL, &fme_region->region->dev,
+					fpga_fme_region_match);
+	if (!region)
+		return NULL;
+
+	return region;
+}
+
+static int fme_pr(struct platform_device *pdev, unsigned long arg)
+{
+	void __user *argp = (void __user *)arg;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_fme *fme;
+	struct feature_fme_header *fme_hdr;
+	struct feature_fme_capability fme_capability;
+	struct fpga_image_info *info;
+	struct fpga_region *region;
+	struct fpga_fme_port_pr port_pr;
+	unsigned long minsz;
+	void *buf = NULL;
+	int ret = 0;
+
+	minsz = offsetofend(struct fpga_fme_port_pr, buffer_address);
+
+	if (copy_from_user(&port_pr, argp, minsz))
+		return -EFAULT;
+
+	if (port_pr.argsz < minsz || port_pr.flags)
+		return -EINVAL;
+
+	if (!IS_ALIGNED(port_pr.buffer_size, 4))
+		return -EINVAL;
+
+	/* get fme header region */
+	fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
+					FME_FEATURE_ID_HEADER);
+	if (WARN_ON(!fme_hdr))
+		return -EINVAL;
+
+	/* check port id */
+	fme_capability.csr = readq(&fme_hdr->capability);
+	if (port_pr.port_id >= fme_capability.num_ports) {
+		dev_dbg(&pdev->dev, "port number more than maximum\n");
+		return -EINVAL;
+	}
+
+	if (!access_ok(VERIFY_READ,
+		       (void __user *)(unsigned long)port_pr.buffer_address,
+		       port_pr.buffer_size))
+		return -EFAULT;
+
+	buf = vmalloc(port_pr.buffer_size);
+	if (!buf)
+		return -ENOMEM;
+
+	if (copy_from_user(buf,
+			(void __user *)(unsigned long)port_pr.buffer_address,
+			port_pr.buffer_size)) {
+		ret = -EFAULT;
+		goto free_exit;
+	}
+
+	/* prepare fpga_image_info for PR */
+	info = fpga_image_info_alloc(&pdev->dev);
+	if (IS_ERR(info)) {
+		ret = PTR_ERR(info);
+		goto free_exit;
+	}
+
+	info->flags |= FPGA_MGR_PARTIAL_RECONFIG;
+
+	mutex_lock(&pdata->lock);
+	fme = fpga_pdata_get_private(pdata);
+	/* fme device has been unregistered. */
+	if (!fme) {
+		ret = -EINVAL;
+		goto unlock_exit;
+	}
+
+	region = fpga_fme_region_find(fme, port_pr.port_id);
+	if (!region) {
+		ret = -EINVAL;
+		goto unlock_exit;
+	}
+
+	fpga_image_info_free(&pdev->dev, region->info);
+	region->info = NULL;
+
+	info->buf = buf;
+	info->count = port_pr.buffer_size;
+	info->region_id = port_pr.port_id;
+
+	ret = fpga_region_program_fpga(region, info);
+
+	if (region->get_bridges)
+		fpga_bridges_put(&region->bridge_list);
+
+	put_device(&region->dev);
+unlock_exit:
+	mutex_unlock(&pdata->lock);
+free_exit:
+	vfree(buf);
+	if (copy_to_user((void __user *)arg, &port_pr, minsz))
+		return -EFAULT;
+
+	return ret;
+}
+
+/**
+ * fpga_fme_create_mgr - create fpga mgr platform device as child device
+ *
+ * @pdata: fme platform_device's pdata
+ *
+ * Return: mgr platform device if successful, and error code otherwise.
+ */
+static struct platform_device *
+fpga_fme_create_mgr(struct feature_platform_data *pdata)
+{
+	struct platform_device *mgr, *fme = pdata->dev;
+	struct resource res;
+	struct resource *pres;
+	int ret = -ENOMEM;
+
+	/*
+	 * Each FME has only one fpga-mgr, so allocate platform device using
+	 * the same FME platform device id.
+	 */
+	mgr = platform_device_alloc(INTEL_FPGA_FME_MGR, fme->id);
+	if (!mgr)
+		return ERR_PTR(ret);
+
+	mgr->dev.parent = &fme->dev;
+
+	pres = platform_get_resource_byname(fme, IORESOURCE_MEM,
+					    FME_FEATURE_PR_MGMT);
+	if (!pres) {
+		ret = -ENODEV;
+		goto create_mgr_err;
+	}
+
+	memset(&res, 0, sizeof(struct resource));
+
+	res.start = pres->start;
+	res.end = pres->end;
+	res.name = pres->name;
+	res.flags = IORESOURCE_MEM;
+
+	ret = platform_device_add_resources(mgr, &res, 1);
+	if (ret)
+		goto create_mgr_err;
+
+	ret = platform_device_add(mgr);
+	if (ret)
+		goto create_mgr_err;
+
+	return mgr;
+
+create_mgr_err:
+	platform_device_put(mgr);
+	return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_mgr - destroy fpga mgr platform device
+ *
+ * @mgr: fpga mgr platform device to be destroy
+ */
+static void fpga_fme_destroy_mgr(struct feature_platform_data *pdata)
+{
+	struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+
+	platform_device_unregister(priv->mgr);
+}
+
+/**
+ * fpga_fme_create_bridge - create fme fpga bridge platform device as child
+ *
+ * @pdata: fme platform device's pdata
+ * @port_id: port id for the bridge to be created.
+ *
+ * Return: bridge platform device if successful, and error code otherwise.
+ */
+static struct fme_bridge *
+fpga_fme_create_bridge(struct feature_platform_data *pdata, int port_id)
+{
+	struct device *dev = &pdata->dev->dev;
+	struct fme_br_pdata br_pdata;
+	struct fme_bridge *fme_br;
+	int ret = -ENOMEM;
+
+	fme_br = devm_kzalloc(dev, sizeof(*fme_br), GFP_KERNEL);
+	if (!fme_br)
+		return ERR_PTR(ret);
+
+	br_pdata.port = pdata->fpga_for_each_port(pdata->dev, &port_id,
+						  fpga_port_check_id);
+	if (!br_pdata.port)
+		return ERR_PTR(-ENODEV);
+
+	/*
+	 * Each FPGA device may have more than one port, so allocate platform
+	 * device using the same port platform device id.
+	 */
+	fme_br->br = platform_device_alloc(INTEL_FPGA_FME_BRIDGE,
+					   br_pdata.port->id);
+	if (!fme_br->br) {
+		ret = -ENOMEM;
+		goto create_br_err;
+	}
+
+	fme_br->br->dev.parent = dev;
+
+	ret = platform_device_add_data(fme_br->br, &br_pdata, sizeof(br_pdata));
+	if (ret)
+		goto create_br_err;
+
+	ret = platform_device_add(fme_br->br);
+	if (ret)
+		goto create_br_err;
+
+	return fme_br;
+
+create_br_err:
+	platform_device_put(fme_br->br);
+	put_device(&br_pdata.port->dev);
+	return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_bridge - destroy fpga bridge platform device
+ * @br: fpga bridge platform device to be destroy
+ */
+static void fpga_fme_destroy_bridge(struct feature_platform_data *pdata,
+				    struct fme_bridge *fme_br)
+{
+	struct fme_br_pdata *br_pdata = dev_get_platdata(&fme_br->br->dev);
+
+	put_device(&br_pdata->port->dev);
+	platform_device_unregister(fme_br->br);
+}
+
+static void fpga_fme_destroy_bridges(struct feature_platform_data *pdata)
+{
+	struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+	struct fme_bridge *fbridge, *tmp;
+
+	list_for_each_entry_safe(fbridge, tmp, &priv->bridge_list, node) {
+		list_del(&fbridge->node);
+		fpga_fme_destroy_bridge(pdata, fbridge);
+	}
+}
+
+/**
+ * fpga_fme_create_region - create fpga region platform device as child
+ *
+ * @pdata: fme platform device's pdata
+ * @mgr: mgr platform device needed for region
+ * @br: br platform device needed for region
+ * @port_id: port id
+ *
+ * Return: fme region if successful, and error code otherwise.
+ */
+static struct fme_region *
+fpga_fme_create_region(struct feature_platform_data *pdata,
+		       struct platform_device *mgr,
+		       struct platform_device *br, int port_id)
+{
+	struct device *dev = &pdata->dev->dev;
+	struct fme_region_pdata region_pdata;
+	struct fme_region *fme_region;
+	int ret = -ENOMEM;
+
+	fme_region = devm_kzalloc(dev, sizeof(*fme_region), GFP_KERNEL);
+	if (!fme_region)
+		return ERR_PTR(ret);
+
+	region_pdata.mgr = mgr;
+	region_pdata.br = br;
+
+	/*
+	 * Each FPGA device may have more than one port, so allocate platform
+	 * device using the same port platform device id.
+	 */
+	fme_region->region = platform_device_alloc(INTEL_FPGA_FME_REGION,
+						   br->id);
+	if (!fme_region->region)
+		return ERR_PTR(ret);
+
+	fme_region->region->dev.parent = dev;
+
+	ret = platform_device_add_data(fme_region->region, &region_pdata,
+				       sizeof(region_pdata));
+	if (ret)
+		goto create_region_err;
+
+	ret = platform_device_add(fme_region->region);
+	if (ret)
+		goto create_region_err;
+
+	fme_region->port_id = port_id;
+
+	return fme_region;
+
+create_region_err:
+	platform_device_put(fme_region->region);
+	return ERR_PTR(ret);
+}
+
+/**
+ * fpga_fme_destroy_region - destroy fme region
+ * @region: fme region to be destroy
+ */
+static void fpga_fme_destroy_region(struct feature_platform_data *pdata,
+				    struct fme_region *fme_region)
+{
+	platform_device_unregister(fme_region->region);
+}
+
+static void fpga_fme_destroy_regions(struct feature_platform_data *pdata)
+{
+	struct fpga_fme *priv = fpga_pdata_get_private(pdata);
+	struct fme_region *fme_region, *tmp;
+
+	list_for_each_entry_safe(fme_region, tmp, &priv->region_list, node) {
+		list_del(&fme_region->node);
+		fpga_fme_destroy_region(pdata, fme_region);
+	}
+}
+
+static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct feature_fme_capability fme_capability;
+	struct feature_fme_port port;
+	struct feature_fme_header *fme_hdr;
+	struct platform_device *mgr;
+	struct fme_region *fme_region;
+	struct fme_bridge *fme_br;
+	struct fpga_fme *priv;
+	int ret = -ENODEV, i = 0;
+
+	fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
+					      FME_FEATURE_ID_HEADER);
+	if (WARN_ON(!fme_hdr))
+		return -EINVAL;
+
+	mutex_lock(&pdata->lock);
+	priv = fpga_pdata_get_private(pdata);
+
+	/* Initialize the region and bridge sub device list */
+	INIT_LIST_HEAD(&priv->region_list);
+	INIT_LIST_HEAD(&priv->bridge_list);
+
+	/* Create fpga mgr platform device */
+	mgr = fpga_fme_create_mgr(pdata);
+	if (IS_ERR(mgr)) {
+		dev_err(&pdev->dev, "fail to create fpga mgr pdev\n");
+		goto unlock;
+	}
+
+	priv->mgr = mgr;
+
+	/* Read capability register to check number of regions and bridges */
+	fme_capability.csr = readq(&fme_hdr->capability);
+	for (; i < fme_capability.num_ports; i++) {
+		port.csr = readq(&fme_hdr->port[i]);
+		if (!port.port_implemented)
+			continue;
+
+		/* Create bridge for each port */
+		fme_br = fpga_fme_create_bridge(pdata, i);
+		if (IS_ERR(fme_br)) {
+			ret = PTR_ERR(fme_br);
+			goto destroy_region;
+		}
+
+		list_add(&fme_br->node, &priv->bridge_list);
+
+		/* Create region for each port */
+		fme_region = fpga_fme_create_region(pdata, mgr, fme_br->br, i);
+		if (!fme_region) {
+			ret = PTR_ERR(fme_region);
+			goto destroy_region;
+		}
+
+		list_add(&fme_region->node, &priv->region_list);
+	}
+	mutex_unlock(&pdata->lock);
+
+	return 0;
+
+destroy_region:
+	fpga_fme_destroy_regions(pdata);
+	fpga_fme_destroy_bridges(pdata);
+	fpga_fme_destroy_mgr(pdata);
+unlock:
+	mutex_unlock(&pdata->lock);
+	return ret;
+}
+
+static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_fme *priv;
+
+	mutex_lock(&pdata->lock);
+	priv = fpga_pdata_get_private(pdata);
+
+	fpga_fme_destroy_regions(pdata);
+	fpga_fme_destroy_bridges(pdata);
+	fpga_fme_destroy_mgr(pdata);
+	mutex_unlock(&pdata->lock);
+}
+
+static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
+	unsigned int cmd, unsigned long arg)
+{
+	long ret;
+
+	switch (cmd) {
+	case FPGA_FME_PORT_PR:
+		ret = fme_pr(pdev, arg);
+		break;
+	default:
+		ret = -ENODEV;
+	}
+
+	return ret;
+}
+
+struct feature_ops pr_mgmt_ops = {
+	.init = pr_mgmt_init,
+	.uinit = pr_mgmt_uinit,
+	.ioctl = fme_pr_ioctl,
+};
diff --git a/drivers/fpga/intel-fme.h b/drivers/fpga/intel-fme.h
new file mode 100644
index 0000000..da8cadf
--- /dev/null
+++ b/drivers/fpga/intel-fme.h
@@ -0,0 +1,57 @@
+/*
+ * Header file for Intel FPGA Management Engine (FME) Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
+ *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Joseph Grecco <joe.grecco-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Ananda Ravuri <ananda.ravuri-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Henry Mitchel <henry.mitchel-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#ifndef __INTEL_FME_H
+#define __INTEL_FME_H
+
+#define INTEL_FPGA_FME_MGR		"intel-fpga-fme-mgr"
+#define INTEL_FPGA_FME_BRIDGE		"intel-fpga-fme-bridge"
+#define INTEL_FPGA_FME_REGION		"intel-fpga-fme-region"
+
+struct fpga_fme {
+	struct platform_device *mgr;
+	struct list_head region_list;
+	struct list_head bridge_list;
+	struct feature_platform_data *pdata;
+};
+
+struct fme_region {
+	struct platform_device *region;
+	struct list_head node;
+	int port_id;
+};
+
+struct fme_region_pdata {
+	struct platform_device *mgr;
+	struct platform_device *br;
+	int region_id;
+};
+
+struct fme_bridge {
+	struct platform_device *br;
+	struct list_head node;
+};
+
+struct fme_br_pdata {
+	struct platform_device *port;
+};
+
+extern struct feature_ops pr_mgmt_ops;
+
+#endif
diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
index e340ef9..be295ae 100644
--- a/include/uapi/linux/intel-fpga.h
+++ b/include/uapi/linux/intel-fpga.h
@@ -16,6 +16,8 @@
 #ifndef _UAPI_LINUX_INTEL_FPGA_H
 #define _UAPI_LINUX_INTEL_FPGA_H
 
+#include <linux/types.h>
+
 #define FPGA_API_VERSION 0
 
 /*
@@ -28,6 +30,7 @@
 #define FPGA_MAGIC 0xB6
 
 #define FPGA_BASE 0
+#define FME_BASE 0x80
 
 /**
  * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
@@ -47,4 +50,28 @@
 
 #define FPGA_CHECK_EXTENSION	_IO(FPGA_MAGIC, FPGA_BASE + 1)
 
+/* IOCTLs for FME file descriptor */
+
+/**
+ * FPGA_FME_PORT_PR - _IOW(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
+ *
+ * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
+ * provided by caller.
+ * Return: 0 on success, -errno on failure.
+ * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
+ * some errors during PR, under this case, the user can fetch HW error info
+ * from the status of FME's fpga manager.
+ */
+
+struct fpga_fme_port_pr {
+	/* Input */
+	__u32 argsz;		/* Structure length */
+	__u32 flags;		/* Zero for now */
+	__u32 port_id;
+	__u32 buffer_size;
+	__u64 buffer_address;	/* Userspace address to the buffer for PR */
+};
+
+#define FPGA_FME_PORT_PR	_IO(FPGA_MAGIC, FME_BASE + 0)
+
 #endif /* _UAPI_INTEL_FPGA_H */
-- 
1.8.3.1

[PATCH v2 15/22] fpga: intel: add fpga manager platform driver for FME

[Wu Hao]

This patch adds fpga manager driver for Intel FPGA Management
Engine(FME). It implements fpga_manager_ops for FPGA Partial
Reconfiguration function.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Kang Luwei <luwei.kang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
 .../ABI/testing/sysfs-platform-intel-fpga-fme-mgr  |   8 +
 drivers/fpga/Kconfig                               |   7 +
 drivers/fpga/Makefile                              |   1 +
 drivers/fpga/intel-feature-dev.h                   |  75 +++++
 drivers/fpga/intel-fpga-fme-mgr.c                  | 307 +++++++++++++++++++++
 5 files changed, 398 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-platform-intel-fpga-fme-mgr
 create mode 100644 drivers/fpga/intel-fpga-fme-mgr.c

diff --git a/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme-mgr b/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme-mgr
new file mode 100644
index 0000000..40771fb
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme-mgr
@@ -0,0 +1,8 @@
+What:		/sys/bus/platform/devices/intel-fpga-fme-mgr.0/interface_id
+Date:		June 2017
+KernelVersion:  4.12
+Contact:	Wu Hao <hao.wu@intel.com>
+Description:	Read-only. It returns interface id of partial reconfiguration
+		hardware. Userspace could use this information to check if
+		current hardware is compatible with given image before FPGA
+		programming.
diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index b91458f..6f2f623 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig
@@ -151,6 +151,13 @@ config INTEL_FPGA_FME
 	  all FPGA platform level management features. There shall be 1
 	  FME per Intel FPGA.
 
+config INTEL_FPGA_FME_MGR
+	tristate "Intel FPGA FME Manager Driver"
+	depends on INTEL_FPGA_FME
+	help
+	  Say Y to enable FPGA Manager driver for Intel FPGA Management
+	  Engine.
+
 endif
 
 endif # FPGA
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index acda5ca..d1d588b 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -31,6 +31,7 @@ obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
 # Intel FPGA Support
 obj-$(CONFIG_INTEL_FPGA_PCI)		+= intel-fpga-pci.o
 obj-$(CONFIG_INTEL_FPGA_FME)		+= intel-fpga-fme.o
+obj-$(CONFIG_INTEL_FPGA_FME_MGR)	+= intel-fpga-fme-mgr.o
 
 intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
 intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
index 3f97b75..f33923b 100644
--- a/drivers/fpga/intel-feature-dev.h
+++ b/drivers/fpga/intel-feature-dev.h
@@ -164,8 +164,83 @@ struct feature_fme_err {
 };
 
 /* FME Partial Reconfiguration Sub Feature Register Set */
+/* FME PR Control Register */
+struct feature_fme_pr_ctl {
+	union {
+		u64 csr;
+		struct {
+			u64 pr_reset:1;		/* Reset PR Engine */
+			u64 rsvdz1:3;
+			u64 pr_reset_ack:1;	/* Reset PR Engine Ack */
+			u64 rsvdz2:3;
+			u64 pr_regionid:2;	/* PR Region ID */
+			u64 rsvdz3:2;
+			u64 pr_start_req:1;	/* PR Start Request */
+			u64 pr_push_complete:1;	/* PR Data push complete */
+			u64 pr_kind:1;		/* Load Customer or Intel GBS */
+			u64 rsvdz4:17;
+			u64 config_data:32;
+		};
+	};
+};
+
+/* FME PR Status Register */
+struct feature_fme_pr_status {
+	union {
+		u64 csr;
+		struct {
+			u64 pr_credit:9;	/* Number of PR Credits */
+			u64 rsvdz1:7;
+			u64 pr_status:1;	/* PR Operation status */
+			u64 rsvdz2:3;
+			u64 pr_ctrlr_status:3;	/* Controller status */
+			u64 rsvdz3:1;
+			u64 pr_host_status:4;	/* PR Host status */
+			u64 rsvdz4:36;
+		};
+	};
+};
+
+/* FME PR Data Register */
+struct feature_fme_pr_data {
+	union {
+		u64 csr;
+		struct {
+			/* PR data from the raw-binary file */
+			u64 pr_data_raw:32;
+			u64 rsvd:32;
+		};
+	};
+};
+
+/* FME PR Error Register */
+struct feature_fme_pr_error {
+	union {
+		u64 csr;
+		struct {
+			u64 operation_err:1;	/* Previous PR error detected */
+			u64 crc_err:1;		/* CRC error detected */
+			u64 incompatiable_bs:1; /* Incompatiable Bitstream */
+			u64 protocol_err:1;	/* Data push protocol error */
+			u64 fifo_overflow:1;	/* Data fifo overflow */
+			u64 rsvdz1:1;
+			u64 secure_load_err:1;	/* Secure PR load error */
+			u64 rsvdz2:57;
+		};
+	};
+};
+
 struct feature_fme_pr {
 	struct feature_header header;
+	struct feature_fme_pr_ctl control;
+	struct feature_fme_pr_status status;
+	struct feature_fme_pr_data data;
+	struct feature_fme_pr_error error;
+
+	u64 rsvd[16];
+
+	u64 intfc_id_l;		/* PR interface Id Low */
+	u64 intfc_id_h;		/* PR interface Id High */
 };
 
 /* PORT Header Register Set */
diff --git a/drivers/fpga/intel-fpga-fme-mgr.c b/drivers/fpga/intel-fpga-fme-mgr.c
new file mode 100644
index 0000000..5f93dfb
--- /dev/null
+++ b/drivers/fpga/intel-fpga-fme-mgr.c
@@ -0,0 +1,307 @@
+/*
+ * FPGA Manager Driver for Intel FPGA Management Engine (FME)
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Christopher Rauer <christopher.rauer@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include <linux/module.h>
+#include <linux/iopoll.h>
+#include <linux/fpga/fpga-mgr.h>
+
+#include "intel-feature-dev.h"
+#include "intel-fme.h"
+
+#define PR_WAIT_TIMEOUT   8000000
+#define PR_HOST_STATUS_IDLE	0
+
+struct fme_mgr_priv {
+	void __iomem *ioaddr;
+};
+
+static ssize_t interface_id_show(struct device *dev,
+				 struct device_attribute *attr, char *buf)
+{
+	struct fpga_manager *mgr = dev_get_drvdata(dev);
+	struct fme_mgr_priv *priv = mgr->priv;
+	struct feature_fme_pr *fme_pr;
+	u64 intfc_id_l, intfc_id_h;
+
+	fme_pr = priv->ioaddr;
+
+	intfc_id_l = readq(&fme_pr->intfc_id_l);
+	intfc_id_h = readq(&fme_pr->intfc_id_h);
+
+	return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
+			(unsigned long long)intfc_id_h,
+			(unsigned long long)intfc_id_l);
+}
+static DEVICE_ATTR_RO(interface_id);
+
+static const struct attribute *fme_mgr_attrs[] = {
+	&dev_attr_interface_id.attr,
+	NULL,
+};
+
+static void fme_mgr_pr_update_status(struct fpga_manager *mgr,
+				     struct feature_fme_pr_error err)
+{
+	mgr->status = 0;
+
+	if (err.operation_err)
+		mgr->status |= FPGA_MGR_STATUS_OPERATION_ERR;
+	if (err.crc_err)
+		mgr->status |= FPGA_MGR_STATUS_CRC_ERR;
+	if (err.incompatiable_bs)
+		mgr->status |= FPGA_MGR_STATUS_INCOMPATIBLE_BS_ERR;
+	if (err.protocol_err)
+		mgr->status |= FPGA_MGR_STATUS_IP_PROTOCOL_ERR;
+	if (err.fifo_overflow)
+		mgr->status |= FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR;
+	if (err.secure_load_err)
+		mgr->status |= FPGA_MGR_STATUS_SECURE_LOAD_ERR;
+}
+
+static u64 fme_mgr_pr_err_handle(struct feature_fme_pr *fme_pr)
+{
+	struct feature_fme_pr_status fme_pr_status;
+	struct feature_fme_pr_error fme_pr_error;
+
+	fme_pr_status.csr = readq(&fme_pr->status);
+	if (!fme_pr_status.pr_status)
+		return 0;
+
+	fme_pr_error.csr = readq(&fme_pr->error);
+	writeq(fme_pr_error.csr, &fme_pr->error);
+
+	return fme_pr_error.csr;
+}
+
+static int fme_mgr_write_init(struct fpga_manager *mgr,
+			      struct fpga_image_info *info,
+			      const char *buf, size_t count)
+{
+	struct device *dev = &mgr->dev;
+	struct fme_mgr_priv *priv = mgr->priv;
+	struct feature_fme_pr *fme_pr = priv->ioaddr;
+	struct feature_fme_pr_ctl fme_pr_ctl;
+	struct feature_fme_pr_status fme_pr_status;
+	struct feature_fme_pr_error fme_pr_err;
+
+	if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
+		dev_err(dev, "only support partial reconfiguration.\n");
+		return -EINVAL;
+	}
+
+	dev_dbg(dev, "resetting PR before initiated PR\n");
+
+	fme_pr_ctl.csr = readq(&fme_pr->control);
+	fme_pr_ctl.pr_reset = 1;
+	writeq(fme_pr_ctl.csr, &fme_pr->control);
+
+	if (readq_poll_timeout(&fme_pr->control, fme_pr_ctl.csr,
+			       (fme_pr_ctl.pr_reset_ack == 1),
+			       1, PR_WAIT_TIMEOUT)) {
+		dev_err(dev, "maximum PR timeout\n");
+		return -ETIMEDOUT;
+	}
+
+	fme_pr_ctl.csr = readq(&fme_pr->control);
+	fme_pr_ctl.pr_reset = 0;
+	writeq(fme_pr_ctl.csr, &fme_pr->control);
+
+	dev_dbg(dev,
+		"waiting for PR resource in HW to be initialized and ready\n");
+
+	fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
+
+	if (readq_poll_timeout(&fme_pr->status, fme_pr_status.csr,
+			(fme_pr_status.pr_host_status == PR_HOST_STATUS_IDLE),
+			1, PR_WAIT_TIMEOUT)) {
+		dev_err(dev, "maximum PR timeout\n");
+		return -ETIMEDOUT;
+	}
+
+	dev_dbg(dev, "check and clear previous PR error\n");
+	fme_pr_err.csr = fme_mgr_pr_err_handle(fme_pr);
+	if (fme_pr_err.csr)
+		dev_dbg(dev, "previous PR error detected %llx\n",
+			     (unsigned long long)fme_pr_err.csr);
+
+	/* Clear all PR errors */
+	fme_pr_err.csr = 0;
+	fme_mgr_pr_update_status(mgr, fme_pr_err);
+
+	dev_dbg(dev, "set PR port ID\n");
+
+	fme_pr_ctl.csr = readq(&fme_pr->control);
+	fme_pr_ctl.pr_regionid = info->region_id;
+	writeq(fme_pr_ctl.csr, &fme_pr->control);
+
+	return 0;
+}
+
+static int fme_mgr_write(struct fpga_manager *mgr,
+			 const char *buf, size_t count)
+{
+	struct device *dev = &mgr->dev;
+	struct fme_mgr_priv *priv = mgr->priv;
+	struct feature_fme_pr *fme_pr = priv->ioaddr;
+	struct feature_fme_pr_ctl fme_pr_ctl;
+	struct feature_fme_pr_status fme_pr_status;
+	struct feature_fme_pr_data fme_pr_data;
+	int delay, pr_credit, i = 0;
+
+	dev_dbg(dev, "start request\n");
+
+	fme_pr_ctl.csr = readq(&fme_pr->control);
+	fme_pr_ctl.pr_start_req = 1;
+	writeq(fme_pr_ctl.csr, &fme_pr->control);
+
+	dev_dbg(dev, "pushing data from bitstream to HW\n");
+
+	fme_pr_status.csr = readq(&fme_pr->status);
+	pr_credit = fme_pr_status.pr_credit;
+
+	while (count > 0) {
+		delay = 0;
+		while (pr_credit <= 1) {
+			if (delay++ > PR_WAIT_TIMEOUT) {
+				dev_err(dev, "maximum try\n");
+				return -ETIMEDOUT;
+			}
+			udelay(1);
+
+			fme_pr_status.csr = readq(&fme_pr->status);
+			pr_credit = fme_pr_status.pr_credit;
+		};
+
+		if (count >= 4) {
+			fme_pr_data.rsvd = 0;
+			fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
+			writeq(fme_pr_data.csr, &fme_pr->data);
+			count -= 4;
+			pr_credit--;
+			i++;
+		} else {
+			WARN_ON(1);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int fme_mgr_write_complete(struct fpga_manager *mgr,
+				  struct fpga_image_info *info)
+{
+	struct device *dev = &mgr->dev;
+	struct fme_mgr_priv *priv = mgr->priv;
+	struct feature_fme_pr *fme_pr = priv->ioaddr;
+	struct feature_fme_pr_ctl fme_pr_ctl;
+	struct feature_fme_pr_error fme_pr_err;
+
+	fme_pr_ctl.csr = readq(&fme_pr->control);
+	fme_pr_ctl.pr_push_complete = 1;
+	writeq(fme_pr_ctl.csr, &fme_pr->control);
+
+	dev_dbg(dev, "green bitstream push complete\n");
+	dev_dbg(dev, "waiting for HW to release PR resource\n");
+
+	fme_pr_ctl.pr_start_req = 0;
+
+	if (readq_poll_timeout(&fme_pr->control, fme_pr_ctl.csr,
+			       (fme_pr_ctl.pr_start_req == 0),
+			       1, PR_WAIT_TIMEOUT)) {
+		dev_err(dev, "maximum try.\n");
+		return -ETIMEDOUT;
+	}
+
+	dev_dbg(dev, "PR operation complete, checking status\n");
+	fme_pr_err.csr = fme_mgr_pr_err_handle(fme_pr);
+	if (fme_pr_err.csr) {
+		fme_mgr_pr_update_status(mgr, fme_pr_err);
+		return -EIO;
+	}
+
+	dev_dbg(dev, "PR done successfully\n");
+	return 0;
+}
+
+static enum fpga_mgr_states fme_mgr_state(struct fpga_manager *mgr)
+{
+	return FPGA_MGR_STATE_UNKNOWN;
+}
+
+static const struct fpga_manager_ops fme_mgr_ops = {
+	.write_init = fme_mgr_write_init,
+	.write = fme_mgr_write,
+	.write_complete = fme_mgr_write_complete,
+	.state = fme_mgr_state,
+};
+
+static int fme_mgr_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct fme_mgr_priv *priv;
+	struct resource *res;
+	int ret;
+
+	priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+	priv->ioaddr = devm_ioremap(dev, res->start, resource_size(res));
+	if (IS_ERR(priv->ioaddr))
+		return PTR_ERR(priv->ioaddr);
+
+	ret = sysfs_create_files(&pdev->dev.kobj, fme_mgr_attrs);
+	if (ret)
+		return ret;
+
+	ret = fpga_mgr_register(dev, "Intel FPGA Manager", &fme_mgr_ops, priv);
+	if (ret) {
+		dev_err(dev, "unable to register FPGA manager\n");
+		sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
+		return ret;
+	}
+
+	return 0;
+}
+
+static int fme_mgr_remove(struct platform_device *pdev)
+{
+	fpga_mgr_unregister(&pdev->dev);
+	sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
+
+	return 0;
+}
+
+static struct platform_driver fme_mgr_driver = {
+	.driver	= {
+		.name    = INTEL_FPGA_FME_MGR,
+	},
+	.probe   = fme_mgr_probe,
+	.remove  = fme_mgr_remove,
+};
+
+module_platform_driver(fme_mgr_driver);
+
+MODULE_DESCRIPTION("FPGA Manager for Intel FPGA Management Engine");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:intel-fpga-fme-mgr");
-- 
1.8.3.1

[PATCH v2 16/22] fpga: intel: add fpga bridge platform driver for FME

[Wu Hao]

This patch adds fpga bridge platform driver for Intel FPGA Management
Engine. It implements the enable_set call back for fpga bridge.

Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/fpga/Kconfig             |  7 ++++
 drivers/fpga/Makefile            |  1 +
 drivers/fpga/intel-fpga-fme-br.c | 77 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 85 insertions(+)
 create mode 100644 drivers/fpga/intel-fpga-fme-br.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 6f2f623..05e2a8e 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig
@@ -158,6 +158,13 @@ config INTEL_FPGA_FME_MGR
 	  Say Y to enable FPGA Manager driver for Intel FPGA Management
 	  Engine.
 
+config INTEL_FPGA_FME_BRIDGE
+	tristate "Intel FPGA FME Bridge Driver"
+	depends on INTEL_FPGA_FME && FPGA_BRIDGE
+	help
+	  Say Y to enable FPGA Bridge driver for Intel FPGA Management
+	  Engine.
+
 endif
 
 endif # FPGA
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index d1d588b..131c105 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -32,6 +32,7 @@ obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
 obj-$(CONFIG_INTEL_FPGA_PCI)		+= intel-fpga-pci.o
 obj-$(CONFIG_INTEL_FPGA_FME)		+= intel-fpga-fme.o
 obj-$(CONFIG_INTEL_FPGA_FME_MGR)	+= intel-fpga-fme-mgr.o
+obj-$(CONFIG_INTEL_FPGA_FME_BRIDGE)	+= intel-fpga-fme-br.o
 
 intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
 intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
diff --git a/drivers/fpga/intel-fpga-fme-br.c b/drivers/fpga/intel-fpga-fme-br.c
new file mode 100644
index 0000000..9b922d2
--- /dev/null
+++ b/drivers/fpga/intel-fpga-fme-br.c
@@ -0,0 +1,77 @@
+/*
+ * FPGA Bridge Driver for Intel FPGA Management Engine (FME)
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Joseph Grecco <joe.grecco-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Ananda Ravuri <ananda.ravuri-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Henry Mitchel <henry.mitchel-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include <linux/module.h>
+#include <linux/fpga/fpga-bridge.h>
+
+#include "intel-feature-dev.h"
+#include "intel-fme.h"
+
+static int fme_bridge_enable_set(struct fpga_bridge *bridge, bool enable)
+{
+	struct fme_br_pdata *pdata = bridge->priv;
+	int ret = 0;
+
+	if (enable)
+		fpga_port_enable(pdata->port);
+	else
+		ret = fpga_port_disable(pdata->port);
+
+	return ret;
+}
+
+static const struct fpga_bridge_ops fme_bridge_ops = {
+	.enable_set = fme_bridge_enable_set,
+};
+
+static int fme_br_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct fme_br_pdata *pdata = dev_get_platdata(dev);
+	int ret;
+
+	ret = fpga_bridge_register(dev, "Intel FPGA FME Bridge",
+				   &fme_bridge_ops, pdata);
+	if (ret)
+		dev_err(dev, "unable to register FPGA Bridge\n");
+
+	dev_dbg(dev, "Intel FME FPGA Bridge probed\n");
+
+	return ret;
+}
+
+static int fme_br_remove(struct platform_device *pdev)
+{
+	fpga_bridge_unregister(&pdev->dev);
+
+	return 0;
+}
+
+static struct platform_driver fme_br_driver = {
+	.driver	= {
+		.name    = INTEL_FPGA_FME_BRIDGE,
+	},
+	.probe   = fme_br_probe,
+	.remove  = fme_br_remove,
+};
+
+module_platform_driver(fme_br_driver);
+
+MODULE_DESCRIPTION("FPGA Bridge for Intel FPGA Management Engine");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:intel-fpga-fme-bridge");
-- 
1.8.3.1

[PATCH v2 17/22] fpga: intel: add fpga region platform driver for FME

[Wu Hao]

This patch adds fpga region platform driver for Intel FPGA Management
Engine. It register a fpga region with given fpga manager / bridge
device.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
 drivers/fpga/Kconfig                 |  7 +++
 drivers/fpga/Makefile                |  1 +
 drivers/fpga/intel-fpga-fme-region.c | 91 ++++++++++++++++++++++++++++++++++++
 3 files changed, 99 insertions(+)
 create mode 100644 drivers/fpga/intel-fpga-fme-region.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 05e2a8e..db1a6ea 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig
@@ -165,6 +165,13 @@ config INTEL_FPGA_FME_BRIDGE
 	  Say Y to enable FPGA Bridge driver for Intel FPGA Management
 	  Engine.
 
+config INTEL_FPGA_FME_REGION
+	tristate "Intel FPGA FME Region Driver"
+	depends on INTEL_FPGA_FME && FPGA_REGION
+	help
+	  Say Y to enable FPGA Region driver for Intel FPGA Management
+	  Engine.
+
 endif
 
 endif # FPGA
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 131c105..fd0a71f 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -33,6 +33,7 @@ obj-$(CONFIG_INTEL_FPGA_PCI)		+= intel-fpga-pci.o
 obj-$(CONFIG_INTEL_FPGA_FME)		+= intel-fpga-fme.o
 obj-$(CONFIG_INTEL_FPGA_FME_MGR)	+= intel-fpga-fme-mgr.o
 obj-$(CONFIG_INTEL_FPGA_FME_BRIDGE)	+= intel-fpga-fme-br.o
+obj-$(CONFIG_INTEL_FPGA_FME_REGION)	+= intel-fpga-fme-region.o
 
 intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
 intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
diff --git a/drivers/fpga/intel-fpga-fme-region.c b/drivers/fpga/intel-fpga-fme-region.c
new file mode 100644
index 0000000..2b5ca4d
--- /dev/null
+++ b/drivers/fpga/intel-fpga-fme-region.c
@@ -0,0 +1,91 @@
+/*
+ * FPGA Region Driver for Intel FPGA Management Engine (FME)
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Wu Hao <hao.wu@intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include <linux/module.h>
+#include <linux/fpga/fpga-region.h>
+
+#include "intel-feature-dev.h"
+#include "intel-fme.h"
+
+static int fme_region_get_bridges(struct fpga_region *region,
+				  struct fpga_image_info *info)
+{
+	struct fme_region_pdata *pdata = region->priv;
+	struct device *dev = &pdata->br->dev;
+
+	return fpga_bridge_get_to_list(dev, info, &region->bridge_list);
+}
+
+static int fme_region_probe(struct platform_device *pdev)
+{
+	struct device *dev = &pdev->dev;
+	struct fme_region_pdata *pdata = dev_get_platdata(dev);
+	struct fpga_region *region;
+	struct fpga_manager *mgr;
+	int ret;
+
+	mgr = fpga_mgr_get(&pdata->mgr->dev);
+	if (IS_ERR(mgr))
+		return -EPROBE_DEFER;
+
+	region = devm_kzalloc(dev, sizeof(*region), GFP_KERNEL);
+	if (!region) {
+		ret = -ENOMEM;
+		goto eprobe_mgr_put;
+	}
+
+	region->mgr = mgr;
+	region->get_bridges = fme_region_get_bridges;
+	region->priv = pdata;
+
+	ret = fpga_region_register(dev, region);
+	if (ret)
+		goto eprobe_mgr_put;
+
+	dev_dbg(dev, "Intel FME FPGA Region probed\n");
+
+	return 0;
+
+eprobe_mgr_put:
+	fpga_mgr_put(mgr);
+	return ret;
+}
+
+static int fme_region_remove(struct platform_device *pdev)
+{
+	struct fpga_region *region = dev_get_drvdata(&pdev->dev);
+
+	fpga_mgr_put(region->mgr);
+	fpga_region_unregister(region);
+
+	return 0;
+}
+
+static struct platform_driver fme_region_driver = {
+	.driver	= {
+		.name    = INTEL_FPGA_FME_REGION,
+	},
+	.probe   = fme_region_probe,
+	.remove  = fme_region_remove,
+};
+
+module_platform_driver(fme_region_driver);
+
+MODULE_DESCRIPTION("FPGA Region for Intel FPGA Management Engine");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:intel-fpga-fme-region");
-- 
1.8.3.1

[PATCH v2 18/22] fpga: intel: add FPGA Accelerated Function Unit driver basic framework

[Wu Hao]

On Intel FPGA devices, the Accelerated Function Unit (AFU), can be
reprogrammed for different functions. It connects to the FPGA
infrastructure("blue bistream") via a Port. Port CSRs are implemented
separately from the AFU CSRs to provide control and status of the Port.
Once valid green bitstream is programmed into the AFU, it allows access
to the AFU CSRs in the AFU MMIO space.

This patch only implements basic driver framework for AFU, including
device file operation framework.

Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
    switched to GPLv2 license.
---
 drivers/fpga/Kconfig          |   9 +++
 drivers/fpga/Makefile         |   2 +
 drivers/fpga/intel-afu-main.c | 159 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 170 insertions(+)
 create mode 100644 drivers/fpga/intel-afu-main.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index db1a6ea..4658a13 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig
@@ -172,6 +172,15 @@ config INTEL_FPGA_FME_REGION
 	  Say Y to enable FPGA Region driver for Intel FPGA Management
 	  Engine.
 
+config INTEL_FPGA_AFU
+	tristate "Intel FPGA AFU Driver"
+	depends on INTEL_FPGA_PCI
+	help
+	  This is the driver for FPGA Accelerated Function Unit (AFU) which
+	  implements AFU and Port management features. A User AFU connects
+	  to the FPGA infrastructure via a Port. There may be more than 1
+	  Port/AFU per Intel FPGA.
+
 endif
 
 endif # FPGA
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index fd0a71f..ce08833 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -34,6 +34,8 @@ obj-$(CONFIG_INTEL_FPGA_FME)		+= intel-fpga-fme.o
 obj-$(CONFIG_INTEL_FPGA_FME_MGR)	+= intel-fpga-fme-mgr.o
 obj-$(CONFIG_INTEL_FPGA_FME_BRIDGE)	+= intel-fpga-fme-br.o
 obj-$(CONFIG_INTEL_FPGA_FME_REGION)	+= intel-fpga-fme-region.o
+obj-$(CONFIG_INTEL_FPGA_AFU)		+= intel-fpga-afu.o
 
 intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
 intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
+intel-fpga-afu-objs := intel-afu-main.o
diff --git a/drivers/fpga/intel-afu-main.c b/drivers/fpga/intel-afu-main.c
new file mode 100644
index 0000000..96d0367
--- /dev/null
+++ b/drivers/fpga/intel-afu-main.c
@@ -0,0 +1,159 @@
+/*
+ * Driver for Intel FPGA Accelerated Function Unit (AFU)
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
+ *   Joseph Grecco <joe.grecco-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Ananda Ravuri <ananda.ravuri-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Henry Mitchel <henry.mitchel-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+#include "intel-feature-dev.h"
+
+static int port_hdr_init(struct platform_device *pdev, struct feature *feature)
+{
+	dev_dbg(&pdev->dev, "PORT HDR Init.\n");
+
+	return 0;
+}
+
+static void port_hdr_uinit(struct platform_device *pdev,
+					struct feature *feature)
+{
+	dev_dbg(&pdev->dev, "PORT HDR UInit.\n");
+}
+
+struct feature_ops port_hdr_ops = {
+	.init = port_hdr_init,
+	.uinit = port_hdr_uinit,
+};
+
+static struct feature_driver port_feature_drvs[] = {
+	{
+		.name = PORT_FEATURE_HEADER,
+		.ops = &port_hdr_ops,
+	},
+	{
+		.ops = NULL,
+	}
+};
+
+static int afu_open(struct inode *inode, struct file *filp)
+{
+	struct platform_device *fdev = fpga_inode_to_feature_dev(inode);
+	struct feature_platform_data *pdata;
+	int ret;
+
+	pdata = dev_get_platdata(&fdev->dev);
+	if (WARN_ON(!pdata))
+		return -ENODEV;
+
+	ret = feature_dev_use_begin(pdata);
+	if (ret)
+		return ret;
+
+	dev_dbg(&fdev->dev, "Device File Open\n");
+	filp->private_data = fdev;
+
+	return 0;
+}
+
+static int afu_release(struct inode *inode, struct file *filp)
+{
+	struct platform_device *pdev = filp->private_data;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	dev_dbg(&pdev->dev, "Device File Release\n");
+
+	feature_dev_use_end(pdata);
+	return 0;
+}
+
+static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
+{
+	struct platform_device *pdev = filp->private_data;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct feature *f;
+	long ret;
+
+	dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
+
+	switch (cmd) {
+	default:
+		/*
+		 * Let sub-feature's ioctl function to handle the cmd
+		 * Sub-feature's ioctl returns -ENODEV when cmd is not
+		 * handled in this sub feature, and returns 0 and other
+		 * error code if cmd is handled.
+		 */
+		fpga_dev_for_each_feature(pdata, f)
+			if (f->ops && f->ops->ioctl) {
+				ret = f->ops->ioctl(pdev, f, cmd, arg);
+				if (ret == -ENODEV)
+					continue;
+				else
+					return ret;
+			}
+	}
+
+	return -EINVAL;
+}
+
+static const struct file_operations afu_fops = {
+	.owner = THIS_MODULE,
+	.open = afu_open,
+	.release = afu_release,
+	.unlocked_ioctl = afu_ioctl,
+};
+
+static int afu_probe(struct platform_device *pdev)
+{
+	int ret;
+
+	dev_dbg(&pdev->dev, "%s\n", __func__);
+
+	ret = fpga_dev_feature_init(pdev, port_feature_drvs);
+	if (ret)
+		return ret;
+
+	ret = fpga_register_dev_ops(pdev, &afu_fops, THIS_MODULE);
+	if (ret)
+		fpga_dev_feature_uinit(pdev);
+
+	return ret;
+}
+
+static int afu_remove(struct platform_device *pdev)
+{
+	dev_dbg(&pdev->dev, "%s\n", __func__);
+
+	fpga_dev_feature_uinit(pdev);
+	fpga_unregister_dev_ops(pdev);
+	return 0;
+}
+
+static struct platform_driver afu_driver = {
+	.driver	= {
+		.name    = "intel-fpga-port",
+	},
+	.probe   = afu_probe,
+	.remove  = afu_remove,
+};
+
+module_platform_driver(afu_driver);
+
+MODULE_DESCRIPTION("Intel FPGA Accelerated Function Unit driver");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:intel-fpga-port");
-- 
1.8.3.1

[PATCH v2 19/22] fpga: intel: afu: add header sub feature support

[Wu Hao]

The header register set is always present for the Port/AFU, it is mainly
for capability, control and status of the ports that AFU connected to.

This patch implements header sub feature support. Below user interfaces
are created by this patch.

Sysfs interface:
* /sys/class/fpga/<fpga.x>/<intel-fpga-port.x>/id
  Read-only. Port ID.

Ioctl interface:
* FPGA_PORT_RESET
  Reset the FPGA AFU Port.

Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
v2: add sysfs documentation.
---
 .../ABI/testing/sysfs-platform-intel-fpga-afu      |  7 ++++
 drivers/fpga/intel-afu-main.c                      | 44 +++++++++++++++++++++-
 include/uapi/linux/intel-fpga.h                    | 14 +++++++
 3 files changed, 64 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/ABI/testing/sysfs-platform-intel-fpga-afu

diff --git a/Documentation/ABI/testing/sysfs-platform-intel-fpga-afu b/Documentation/ABI/testing/sysfs-platform-intel-fpga-afu
new file mode 100644
index 0000000..8ad22c9
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-platform-intel-fpga-afu
@@ -0,0 +1,7 @@
+What:		/sys/bus/platform/devices/intel-fpga-port.0/id
+Date:		June 2017
+KernelVersion:  4.12
+Contact:	Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+Description:	Read-only. It returns id of this port. One Intel FPGA device
+		may have more than one port. Userspace could use this id to
+		distinguish different ports under same FPGA device.
diff --git a/drivers/fpga/intel-afu-main.c b/drivers/fpga/intel-afu-main.c
index 96d0367..2a17cde 100644
--- a/drivers/fpga/intel-afu-main.c
+++ b/drivers/fpga/intel-afu-main.c
@@ -18,25 +18,66 @@
 
 #include <linux/kernel.h>
 #include <linux/module.h>
+#include <linux/intel-fpga.h>
 
 #include "intel-feature-dev.h"
 
+static ssize_t
+id_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+	int id = fpga_port_id(to_platform_device(dev));
+
+	return scnprintf(buf, PAGE_SIZE, "%d\n", id);
+}
+static DEVICE_ATTR_RO(id);
+
+static const struct attribute *port_hdr_attrs[] = {
+	&dev_attr_id.attr,
+	NULL,
+};
+
 static int port_hdr_init(struct platform_device *pdev, struct feature *feature)
 {
 	dev_dbg(&pdev->dev, "PORT HDR Init.\n");
 
-	return 0;
+	fpga_port_reset(pdev);
+
+	return sysfs_create_files(&pdev->dev.kobj, port_hdr_attrs);
 }
 
 static void port_hdr_uinit(struct platform_device *pdev,
 					struct feature *feature)
 {
 	dev_dbg(&pdev->dev, "PORT HDR UInit.\n");
+
+	sysfs_remove_files(&pdev->dev.kobj, port_hdr_attrs);
+}
+
+static long
+port_hdr_ioctl(struct platform_device *pdev, struct feature *feature,
+					unsigned int cmd, unsigned long arg)
+{
+	long ret;
+
+	switch (cmd) {
+	case FPGA_PORT_RESET:
+		if (!arg)
+			ret = fpga_port_reset(pdev);
+		else
+			ret = -EINVAL;
+		break;
+	default:
+		dev_dbg(&pdev->dev, "%x cmd not handled", cmd);
+		ret = -ENODEV;
+	}
+
+	return ret;
 }
 
 struct feature_ops port_hdr_ops = {
 	.init = port_hdr_init,
 	.uinit = port_hdr_uinit,
+	.ioctl = port_hdr_ioctl,
 };
 
 static struct feature_driver port_feature_drvs[] = {
@@ -76,6 +117,7 @@ static int afu_release(struct inode *inode, struct file *filp)
 
 	dev_dbg(&pdev->dev, "Device File Release\n");
 
+	fpga_port_reset(pdev);
 	feature_dev_use_end(pdata);
 	return 0;
 }
diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
index be295ae..be5f813 100644
--- a/include/uapi/linux/intel-fpga.h
+++ b/include/uapi/linux/intel-fpga.h
@@ -30,8 +30,11 @@
 #define FPGA_MAGIC 0xB6
 
 #define FPGA_BASE 0
+#define PORT_BASE 0x40
 #define FME_BASE 0x80
 
+/* Common IOCTLs for both FME and AFU file descriptor */
+
 /**
  * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
  *
@@ -50,6 +53,17 @@
 
 #define FPGA_CHECK_EXTENSION	_IO(FPGA_MAGIC, FPGA_BASE + 1)
 
+/* IOCTLs for AFU file descriptor */
+
+/**
+ * FPGA_PORT_RESET - _IO(FPGA_MAGIC, PORT_BASE + 0)
+ *
+ * Reset the FPGA AFU Port. No parameters are supported.
+ * Return: 0 on success, -errno of failure
+ */
+
+#define FPGA_PORT_RESET		_IO(FPGA_MAGIC, PORT_BASE + 0)
+
 /* IOCTLs for FME file descriptor */
 
 /**
-- 
1.8.3.1

[PATCH v2 20/22] fpga: intel: afu add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support

[Wu Hao]

FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which
need to be supported by all feature devices drivers including FME and AFU.
This patch implements above 2 ioctls in Intel FPGA Accelerated Function
Unit (AFU) driver.

Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
v2: rebased
---
 drivers/fpga/intel-afu-main.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/fpga/intel-afu-main.c b/drivers/fpga/intel-afu-main.c
index 2a17cde..22f77f2 100644
--- a/drivers/fpga/intel-afu-main.c
+++ b/drivers/fpga/intel-afu-main.c
@@ -122,6 +122,13 @@ static int afu_release(struct inode *inode, struct file *filp)
 	return 0;
 }
 
+static long afu_ioctl_check_extension(struct feature_platform_data *pdata,
+				     unsigned long arg)
+{
+	/* No extension support for now */
+	return 0;
+}
+
 static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 {
 	struct platform_device *pdev = filp->private_data;
@@ -132,6 +139,10 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 	dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
 
 	switch (cmd) {
+	case FPGA_GET_API_VERSION:
+		return FPGA_API_VERSION;
+	case FPGA_CHECK_EXTENSION:
+		return afu_ioctl_check_extension(pdata, arg);
 	default:
 		/*
 		 * Let sub-feature's ioctl function to handle the cmd
-- 
1.8.3.1

[PATCH v2 21/22] fpga: intel: afu: add user afu sub feature support

[Wu Hao]

From: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>

User Accelerated Function Unit sub feature exposes the MMIO region of
the AFU. After valid green bitstream (GBS) is programmed and port is
enabled, then this MMIO region could be accessed.

This patch adds support to enumerate the AFU MMIO region and expose it
to userspace via mmap file operation. Below interfaces are exposed to user:

Sysfs interface:
* /sys/class/fpga/<fpga.x>/<intel-fpga-port.x>/afu_id
  Read-only. Indicate which green bitstream is programmed to this AFU.

Ioctl interfaces:
* FPGA_PORT_GET_INFO
  Provide info to userspace on the number of supported region.
  Only UAFU region is supported now.

* FPGA_PORT_GET_REGION_INFO
  Provide region information, including access permission, region size,
  offset from the start of device fd.

Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
    add sysfs documentation.
    switched to GPLv2 license.
---
 .../ABI/testing/sysfs-platform-intel-fpga-afu      |   9 +
 drivers/fpga/Makefile                              |   2 +-
 drivers/fpga/intel-afu-main.c                      | 204 ++++++++++++++++++++-
 drivers/fpga/intel-afu-region.c                    | 127 +++++++++++++
 drivers/fpga/intel-afu.h                           |  52 ++++++
 include/uapi/linux/intel-fpga.h                    |  47 +++++
 6 files changed, 437 insertions(+), 4 deletions(-)
 create mode 100644 drivers/fpga/intel-afu-region.c
 create mode 100644 drivers/fpga/intel-afu.h

diff --git a/Documentation/ABI/testing/sysfs-platform-intel-fpga-afu b/Documentation/ABI/testing/sysfs-platform-intel-fpga-afu
index 8ad22c9..6cd8781 100644
--- a/Documentation/ABI/testing/sysfs-platform-intel-fpga-afu
+++ b/Documentation/ABI/testing/sysfs-platform-intel-fpga-afu
@@ -5,3 +5,12 @@ Contact:	Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
 Description:	Read-only. It returns id of this port. One Intel FPGA device
 		may have more than one port. Userspace could use this id to
 		distinguish different ports under same FPGA device.
+
+What:		/sys/bus/platform/devices/intel-fpga-port.0/afu_id
+Date:		June 2017
+KernelVersion:  4.12
+Contact:	Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+Description:	Read-only. User can program different green bitstreams (GBS) to
+		FPGA Accelerator Function Unit (AFU) for different functions.
+		It returns uuid which could be used to identify which GBS is
+		programmed in this AFU.
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index ce08833..45c0538 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -38,4 +38,4 @@ obj-$(CONFIG_INTEL_FPGA_AFU)		+= intel-fpga-afu.o
 
 intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
 intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
-intel-fpga-afu-objs := intel-afu-main.o
+intel-fpga-afu-objs := intel-afu-main.o intel-afu-region.o
diff --git a/drivers/fpga/intel-afu-main.c b/drivers/fpga/intel-afu-main.c
index 22f77f2..8c7aa70 100644
--- a/drivers/fpga/intel-afu-main.c
+++ b/drivers/fpga/intel-afu-main.c
@@ -18,9 +18,10 @@
 
 #include <linux/kernel.h>
 #include <linux/module.h>
+#include <linux/uaccess.h>
 #include <linux/intel-fpga.h>
 
-#include "intel-feature-dev.h"
+#include "intel-afu.h"
 
 static ssize_t
 id_show(struct device *dev, struct device_attribute *attr, char *buf)
@@ -80,12 +81,69 @@ struct feature_ops port_hdr_ops = {
 	.ioctl = port_hdr_ioctl,
 };
 
+static ssize_t
+afu_id_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(dev);
+	struct feature_port_header *port_hdr =
+			get_feature_ioaddr_by_index(dev, PORT_FEATURE_ID_UAFU);
+	u64 guidl;
+	u64 guidh;
+
+	mutex_lock(&pdata->lock);
+	guidl = readq(&port_hdr->afu_header.guid.b[0]);
+	guidh = readq(&port_hdr->afu_header.guid.b[8]);
+	mutex_unlock(&pdata->lock);
+
+	return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n", guidh, guidl);
+}
+static DEVICE_ATTR_RO(afu_id);
+
+static const struct attribute *port_uafu_attrs[] = {
+	&dev_attr_afu_id.attr,
+	NULL
+};
+
+static int port_uafu_init(struct platform_device *pdev, struct feature *feature)
+{
+	struct resource *res = &pdev->resource[feature->resource_index];
+	u32 flags = FPGA_REGION_READ | FPGA_REGION_WRITE | FPGA_REGION_MMAP;
+	int ret;
+
+	dev_dbg(&pdev->dev, "PORT AFU Init.\n");
+
+	ret = afu_region_add(dev_get_platdata(&pdev->dev),
+			     FPGA_PORT_INDEX_UAFU, resource_size(res),
+			     res->start, flags);
+	if (ret)
+		return ret;
+
+	return sysfs_create_files(&pdev->dev.kobj, port_uafu_attrs);
+}
+
+static void port_uafu_uinit(struct platform_device *pdev,
+					struct feature *feature)
+{
+	dev_dbg(&pdev->dev, "PORT AFU UInit.\n");
+
+	sysfs_remove_files(&pdev->dev.kobj, port_uafu_attrs);
+}
+
+struct feature_ops port_uafu_ops = {
+	.init = port_uafu_init,
+	.uinit = port_uafu_uinit,
+};
+
 static struct feature_driver port_feature_drvs[] = {
 	{
 		.name = PORT_FEATURE_HEADER,
 		.ops = &port_hdr_ops,
 	},
 	{
+		.name = PORT_FEATURE_UAFU,
+		.ops = &port_uafu_ops,
+	},
+	{
 		.ops = NULL,
 	}
 };
@@ -129,6 +187,64 @@ static long afu_ioctl_check_extension(struct feature_platform_data *pdata,
 	return 0;
 }
 
+static long
+afu_ioctl_get_info(struct feature_platform_data *pdata, void __user *arg)
+{
+	struct fpga_port_info info;
+	struct fpga_afu *afu;
+	unsigned long minsz;
+
+	minsz = offsetofend(struct fpga_port_info, num_umsgs);
+
+	if (copy_from_user(&info, arg, minsz))
+		return -EFAULT;
+
+	if (info.argsz < minsz)
+		return -EINVAL;
+
+	mutex_lock(&pdata->lock);
+	afu = fpga_pdata_get_private(pdata);
+	info.flags = 0;
+	info.num_regions = afu->num_regions;
+	info.num_umsgs = afu->num_umsgs;
+	mutex_unlock(&pdata->lock);
+
+	if (copy_to_user(arg, &info, sizeof(info)))
+		return -EFAULT;
+
+	return 0;
+}
+
+static long
+afu_ioctl_get_region_info(struct feature_platform_data *pdata, void __user *arg)
+{
+	struct fpga_port_region_info rinfo;
+	struct fpga_afu_region region;
+	unsigned long minsz;
+	long ret;
+
+	minsz = offsetofend(struct fpga_port_region_info, offset);
+
+	if (copy_from_user(&rinfo, arg, minsz))
+		return -EFAULT;
+
+	if (rinfo.argsz < minsz || rinfo.padding)
+		return -EINVAL;
+
+	ret = afu_get_region_by_index(pdata, rinfo.index, &region);
+	if (ret)
+		return ret;
+
+	rinfo.flags = region.flags;
+	rinfo.size = region.size;
+	rinfo.offset = region.offset;
+
+	if (copy_to_user(arg, &rinfo, sizeof(rinfo)))
+		return -EFAULT;
+
+	return 0;
+}
+
 static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 {
 	struct platform_device *pdev = filp->private_data;
@@ -143,6 +259,10 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 		return FPGA_API_VERSION;
 	case FPGA_CHECK_EXTENSION:
 		return afu_ioctl_check_extension(pdata, arg);
+	case FPGA_PORT_GET_INFO:
+		return afu_ioctl_get_info(pdata, (void __user *)arg);
+	case FPGA_PORT_GET_REGION_INFO:
+		return afu_ioctl_get_region_info(pdata, (void __user *)arg);
 	default:
 		/*
 		 * Let sub-feature's ioctl function to handle the cmd
@@ -163,27 +283,104 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 	return -EINVAL;
 }
 
+static int afu_mmap(struct file *filp, struct vm_area_struct *vma)
+{
+	struct fpga_afu_region region;
+	struct platform_device *pdev = filp->private_data;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	u64 size = vma->vm_end - vma->vm_start;
+	u64 offset;
+	int ret;
+
+	if (!(vma->vm_flags & VM_SHARED))
+		return -EINVAL;
+
+	offset = vma->vm_pgoff << PAGE_SHIFT;
+	ret = afu_get_region_by_offset(pdata, offset, size, &region);
+	if (ret)
+		return ret;
+
+	if (!(region.flags & FPGA_REGION_MMAP))
+		return -EINVAL;
+
+	if ((vma->vm_flags & VM_READ) && !(region.flags & FPGA_REGION_READ))
+		return -EPERM;
+
+	if ((vma->vm_flags & VM_WRITE) && !(region.flags & FPGA_REGION_WRITE))
+		return -EPERM;
+
+	vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
+	return remap_pfn_range(vma, vma->vm_start,
+			(region.phys + (offset - region.offset)) >> PAGE_SHIFT,
+			size, vma->vm_page_prot);
+}
+
 static const struct file_operations afu_fops = {
 	.owner = THIS_MODULE,
 	.open = afu_open,
 	.release = afu_release,
 	.unlocked_ioctl = afu_ioctl,
+	.mmap = afu_mmap,
 };
 
+static int afu_dev_init(struct platform_device *pdev)
+{
+	struct fpga_afu *afu;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	afu = devm_kzalloc(&pdev->dev, sizeof(*afu), GFP_KERNEL);
+	if (!afu)
+		return -ENOMEM;
+
+	afu->pdata = pdata;
+
+	mutex_lock(&pdata->lock);
+	fpga_pdata_set_private(pdata, afu);
+	afu_region_init(pdata);
+	mutex_unlock(&pdata->lock);
+	return 0;
+}
+
+static int afu_dev_destroy(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_afu *afu;
+
+	mutex_lock(&pdata->lock);
+	afu = fpga_pdata_get_private(pdata);
+	afu_region_destroy(pdata);
+	fpga_pdata_set_private(pdata, NULL);
+	mutex_unlock(&pdata->lock);
+
+	devm_kfree(&pdev->dev, afu);
+	return 0;
+}
+
 static int afu_probe(struct platform_device *pdev)
 {
 	int ret;
 
 	dev_dbg(&pdev->dev, "%s\n", __func__);
 
+	ret = afu_dev_init(pdev);
+	if (ret)
+		goto exit;
+
 	ret = fpga_dev_feature_init(pdev, port_feature_drvs);
 	if (ret)
-		return ret;
+		goto dev_destroy;
 
 	ret = fpga_register_dev_ops(pdev, &afu_fops, THIS_MODULE);
-	if (ret)
+	if (ret) {
 		fpga_dev_feature_uinit(pdev);
+		goto dev_destroy;
+	}
+
+	return 0;
 
+dev_destroy:
+	afu_dev_destroy(pdev);
+exit:
 	return ret;
 }
 
@@ -193,6 +390,7 @@ static int afu_remove(struct platform_device *pdev)
 
 	fpga_dev_feature_uinit(pdev);
 	fpga_unregister_dev_ops(pdev);
+	afu_dev_destroy(pdev);
 	return 0;
 }
 
diff --git a/drivers/fpga/intel-afu-region.c b/drivers/fpga/intel-afu-region.c
new file mode 100644
index 0000000..702275b
--- /dev/null
+++ b/drivers/fpga/intel-afu-region.c
@@ -0,0 +1,127 @@
+/*
+ * Driver for Intel FPGA Accelerated Function Unit (AFU) Region Management
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include "intel-afu.h"
+
+void afu_region_init(struct feature_platform_data *pdata)
+{
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+
+	INIT_LIST_HEAD(&afu->regions);
+}
+
+#define for_each_region(region, afu)	\
+	list_for_each_entry((region), &(afu)->regions, node)
+static struct fpga_afu_region *get_region_by_index(struct fpga_afu *afu,
+						   u32 region_index)
+{
+	struct fpga_afu_region *region;
+
+	for_each_region(region, afu)
+		if (region->index == region_index)
+			return region;
+
+	return NULL;
+}
+
+int afu_region_add(struct feature_platform_data *pdata, u32 region_index,
+		   u64 region_size, u64 phys, u32 flags)
+{
+	struct fpga_afu_region *region;
+	struct fpga_afu *afu;
+	int ret = 0;
+
+	region = devm_kzalloc(&pdata->dev->dev, sizeof(*region), GFP_KERNEL);
+	if (!region)
+		return -ENOMEM;
+
+	region->index = region_index;
+	region->size = region_size;
+	region->phys = phys;
+	region->flags = flags;
+
+	mutex_lock(&pdata->lock);
+
+	afu = fpga_pdata_get_private(pdata);
+
+	/* check if @index already exists */
+	if (get_region_by_index(afu, region_index)) {
+		mutex_unlock(&pdata->lock);
+		ret = -EEXIST;
+		goto exit;
+	}
+
+	region_size = PAGE_ALIGN(region_size);
+	region->offset = afu->region_cur_offset;
+	list_add(&region->node, &afu->regions);
+
+	afu->region_cur_offset += region_size;
+	afu->num_regions++;
+	mutex_unlock(&pdata->lock);
+	return 0;
+
+exit:
+	devm_kfree(&pdata->dev->dev, region);
+	return ret;
+}
+
+void afu_region_destroy(struct feature_platform_data *pdata)
+{
+	struct fpga_afu_region *tmp, *region;
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+
+	list_for_each_entry_safe(region, tmp, &afu->regions, node)
+		devm_kfree(&pdata->dev->dev, region);
+}
+
+int afu_get_region_by_index(struct feature_platform_data *pdata,
+			    u32 region_index, struct fpga_afu_region *pregion)
+{
+	struct fpga_afu_region *region;
+	struct fpga_afu *afu;
+	int ret = 0;
+
+	mutex_lock(&pdata->lock);
+	afu = fpga_pdata_get_private(pdata);
+	region = get_region_by_index(afu, region_index);
+	if (!region) {
+		ret = -EINVAL;
+		goto exit;
+	}
+	*pregion = *region;
+exit:
+	mutex_unlock(&pdata->lock);
+	return ret;
+}
+
+int afu_get_region_by_offset(struct feature_platform_data *pdata,
+			    u64 offset, u64 size,
+			    struct fpga_afu_region *pregion)
+{
+	struct fpga_afu_region *region;
+	struct fpga_afu *afu;
+	int ret = 0;
+
+	mutex_lock(&pdata->lock);
+	afu = fpga_pdata_get_private(pdata);
+	for_each_region(region, afu)
+		if (region->offset <= offset &&
+		   region->offset + region->size >= offset + size) {
+			*pregion = *region;
+			goto exit;
+		}
+	ret = -EINVAL;
+exit:
+	mutex_unlock(&pdata->lock);
+	return ret;
+}
diff --git a/drivers/fpga/intel-afu.h b/drivers/fpga/intel-afu.h
new file mode 100644
index 0000000..3417780d
--- /dev/null
+++ b/drivers/fpga/intel-afu.h
@@ -0,0 +1,52 @@
+/*
+ * Header file for Intel FPGA Accelerated Function Unit (AFU) Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *     Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *     Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
+ *     Joseph Grecco <joe.grecco-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *     Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *     Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *     Ananda Ravuri <ananda.ravuri-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *     Henry Mitchel <henry.mitchel-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#ifndef __INTEL_AFU_H
+#define __INTEL_AFU_H
+
+#include "intel-feature-dev.h"
+
+struct fpga_afu_region {
+	u32 index;
+	u32 flags;
+	u64 size;
+	u64 offset;
+	u64 phys;
+	struct list_head node;
+};
+
+struct fpga_afu {
+	u64 region_cur_offset;
+	int num_regions;
+	u8 num_umsgs;
+	struct list_head regions;
+
+	struct feature_platform_data *pdata;
+};
+
+void afu_region_init(struct feature_platform_data *pdata);
+int afu_region_add(struct feature_platform_data *pdata, u32 region_index,
+		   u64 region_size, u64 phys, u32 flags);
+void afu_region_destroy(struct feature_platform_data *pdata);
+int afu_get_region_by_index(struct feature_platform_data *pdata,
+			    u32 region_index, struct fpga_afu_region *pregion);
+int afu_get_region_by_offset(struct feature_platform_data *pdata,
+			    u64 offset, u64 size,
+			    struct fpga_afu_region *pregion);
+
+#endif
diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
index be5f813..a2ad332 100644
--- a/include/uapi/linux/intel-fpga.h
+++ b/include/uapi/linux/intel-fpga.h
@@ -64,6 +64,53 @@
 
 #define FPGA_PORT_RESET		_IO(FPGA_MAGIC, PORT_BASE + 0)
 
+/**
+ * FPGA_PORT_GET_INFO - _IOR(FPGA_MAGIC, PORT_BASE + 1, struct fpga_port_info)
+ *
+ * Retrieve information about the fpga port.
+ * Driver fills the info in provided struct fpga_port_info.
+ * Return: 0 on success, -errno on failure.
+ */
+struct fpga_port_info {
+	/* Input */
+	__u32 argsz;		/* Structure length */
+	/* Output */
+	__u32 flags;		/* Zero for now */
+	__u32 num_regions;	/* The number of supported regions */
+	__u32 num_umsgs;	/* The number of allocated umsgs */
+};
+
+#define FPGA_PORT_GET_INFO	_IO(FPGA_MAGIC, PORT_BASE + 1)
+
+/**
+ * FPGA_PORT_GET_REGION_INFO - _IOWR(FPGA_MAGIC, PORT_BASE + 2,
+ *						struct fpga_port_region_info)
+ *
+ * Retrieve information about a device region.
+ * Caller provides struct fpga_port_region_info with index value set.
+ * Driver returns the region info in other fields.
+ * Return: 0 on success, -errno on failure.
+ */
+struct fpga_port_region_info {
+	/* input */
+	__u32 argsz;		/* Structure length */
+	/* Output */
+	__u32 flags;		/* Access permission */
+#define FPGA_REGION_READ	(1 << 0)	/* Region is readable */
+#define FPGA_REGION_WRITE	(1 << 1)	/* Region is writable */
+#define FPGA_REGION_MMAP	(1 << 2)	/* Can be mmaped to userspace */
+	/* Input */
+	__u32 index;		/* Region index */
+#define FPGA_PORT_INDEX_UAFU	0		/* User AFU */
+#define FPGA_PORT_INDEX_STP	1		/* Signal Tap */
+	__u32 padding;
+	/* Output */
+	__u64 size;		/* Region size (bytes) */
+	__u64 offset;		/* Region offset from start of device fd */
+};
+
+#define FPGA_PORT_GET_REGION_INFO	_IO(FPGA_MAGIC, PORT_BASE + 2)
+
 /* IOCTLs for FME file descriptor */
 
 /**
-- 
1.8.3.1

[PATCH v2 22/22] fpga: intel: afu: add FPGA_PORT_DMA_MAP/UNMAP ioctls support

[Wu Hao]

DMA memory regions are required for Accelerated Function Unit (AFU) usage.
These two ioctls allow user space applications to map user memory regions
for dma, and unmap them after use. Iova is returned from driver to user
space application via FPGA_PORT_DMA_MAP ioctl. Application needs to unmap
it after use, otherwise, driver will unmap them in device file release
operation.

All the mapped regions are managed via a rb tree.

Ioctl interfaces:
* FPGA_PORT_DMA_MAP
  Do the dma mapping per user_addr and length which provided by user.
  Return iova in provided struct afu_port_dma_map.

* FPGA_PORT_DMA_UNMAP
  Unmap the dma region per iova provided by user.

Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
    switched to GPLv2 license.
    fixed kbuild warnings.
---
 drivers/fpga/Makefile               |   3 +-
 drivers/fpga/intel-afu-dma-region.c | 372 ++++++++++++++++++++++++++++++++++++
 drivers/fpga/intel-afu-main.c       |  61 +++++-
 drivers/fpga/intel-afu.h            |  18 ++
 include/uapi/linux/intel-fpga.h     |  37 ++++
 5 files changed, 489 insertions(+), 2 deletions(-)
 create mode 100644 drivers/fpga/intel-afu-dma-region.c

diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 45c0538..339d1f3 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -38,4 +38,5 @@ obj-$(CONFIG_INTEL_FPGA_AFU)		+= intel-fpga-afu.o
 
 intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
 intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
-intel-fpga-afu-objs := intel-afu-main.o intel-afu-region.o
+intel-fpga-afu-objs := intel-afu-main.o intel-afu-region.o \
+		       intel-afu-dma-region.o
diff --git a/drivers/fpga/intel-afu-dma-region.c b/drivers/fpga/intel-afu-dma-region.c
new file mode 100644
index 0000000..982a9b5
--- /dev/null
+++ b/drivers/fpga/intel-afu-dma-region.c
@@ -0,0 +1,372 @@
+/*
+ * Driver for Intel FPGA Accelerated Function Unit (AFU) DMA Region Management
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
+ *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
+ *
+ * This work is licensed under the terms of the GNU GPL version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include <linux/sched/signal.h>
+#include <linux/uaccess.h>
+
+#include "intel-afu.h"
+
+static void put_all_pages(struct page **pages, int npages)
+{
+	int i;
+
+	for (i = 0; i < npages; i++)
+		if (pages[i] != NULL)
+			put_page(pages[i]);
+}
+
+void afu_dma_region_init(struct feature_platform_data *pdata)
+{
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+
+	afu->dma_regions = RB_ROOT;
+}
+
+static long afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
+{
+	unsigned long locked, lock_limit;
+	int ret = 0;
+
+	/* the task is exiting. */
+	if (!current->mm)
+		return 0;
+
+	down_write(&current->mm->mmap_sem);
+
+	if (incr) {
+		locked = current->mm->locked_vm + npages;
+		lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
+
+		if (locked > lock_limit && !capable(CAP_IPC_LOCK))
+			ret = -ENOMEM;
+		else
+			current->mm->locked_vm += npages;
+	} else {
+
+		if (WARN_ON_ONCE(npages > current->mm->locked_vm))
+			npages = current->mm->locked_vm;
+		current->mm->locked_vm -= npages;
+	}
+
+	dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
+				incr ? '+' : '-',
+				npages << PAGE_SHIFT,
+				current->mm->locked_vm << PAGE_SHIFT,
+				rlimit(RLIMIT_MEMLOCK),
+				ret ? "- execeeded" : "");
+
+	up_write(&current->mm->mmap_sem);
+
+	return ret;
+}
+
+static long afu_dma_pin_pages(struct feature_platform_data *pdata,
+				struct fpga_afu_dma_region *region)
+{
+	long npages = region->length >> PAGE_SHIFT;
+	struct device *dev = &pdata->dev->dev;
+	long ret, pinned;
+
+	ret = afu_dma_adjust_locked_vm(dev, npages, true);
+	if (ret)
+		return ret;
+
+	region->pages = kcalloc(npages, sizeof(struct page *), GFP_KERNEL);
+	if (!region->pages) {
+		afu_dma_adjust_locked_vm(dev, npages, false);
+		return -ENOMEM;
+	}
+
+	pinned = get_user_pages_fast(region->user_addr, npages, 1,
+					region->pages);
+	if (pinned < 0) {
+		ret = pinned;
+		goto err_put_pages;
+	} else if (pinned != npages) {
+		ret = -EFAULT;
+		goto err;
+	}
+
+	dev_dbg(dev, "%ld pages pinned\n", pinned);
+
+	return 0;
+
+err_put_pages:
+	put_all_pages(region->pages, pinned);
+err:
+	kfree(region->pages);
+	afu_dma_adjust_locked_vm(dev, npages, false);
+	return ret;
+}
+
+static void afu_dma_unpin_pages(struct feature_platform_data *pdata,
+				struct fpga_afu_dma_region *region)
+{
+	long npages = region->length >> PAGE_SHIFT;
+	struct device *dev = &pdata->dev->dev;
+
+	put_all_pages(region->pages, npages);
+	kfree(region->pages);
+	afu_dma_adjust_locked_vm(dev, npages, false);
+
+	dev_dbg(dev, "%ld pages unpinned\n", npages);
+}
+
+static bool afu_dma_check_continuous_pages(struct fpga_afu_dma_region *region)
+{
+	int npages = region->length >> PAGE_SHIFT;
+	int i;
+
+	for (i = 0; i < npages - 1; i++)
+		if (page_to_pfn(region->pages[i]) + 1 !=
+					page_to_pfn(region->pages[i+1]))
+			return false;
+
+	return true;
+}
+
+static bool dma_region_check_iova(struct fpga_afu_dma_region *region,
+				  u64 iova, u64 size)
+{
+	if (!size && region->iova != iova)
+		return false;
+
+	return (region->iova <= iova) &&
+		(region->length + region->iova >= iova + size);
+}
+
+/* Need to be called with pdata->lock held */
+static int afu_dma_region_add(struct feature_platform_data *pdata,
+					struct fpga_afu_dma_region *region)
+{
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+	struct rb_node **new, *parent = NULL;
+
+	dev_dbg(&pdata->dev->dev, "add region (iova = %llx)\n",
+					(unsigned long long)region->iova);
+
+	new = &(afu->dma_regions.rb_node);
+
+	while (*new) {
+		struct fpga_afu_dma_region *this;
+
+		this = container_of(*new, struct fpga_afu_dma_region, node);
+
+		parent = *new;
+
+		if (dma_region_check_iova(this, region->iova, region->length))
+			return -EEXIST;
+
+		if (region->iova < this->iova)
+			new = &((*new)->rb_left);
+		else if (region->iova > this->iova)
+			new = &((*new)->rb_right);
+		else
+			return -EEXIST;
+	}
+
+	rb_link_node(&region->node, parent, new);
+	rb_insert_color(&region->node, &afu->dma_regions);
+
+	return 0;
+}
+
+/* Need to be called with pdata->lock held */
+static void afu_dma_region_remove(struct feature_platform_data *pdata,
+					struct fpga_afu_dma_region *region)
+{
+	struct fpga_afu *afu;
+
+	dev_dbg(&pdata->dev->dev, "del region (iova = %llx)\n",
+					(unsigned long long)region->iova);
+
+	afu = fpga_pdata_get_private(pdata);
+	rb_erase(&region->node, &afu->dma_regions);
+}
+
+/* Need to be called with pdata->lock held */
+void afu_dma_region_destroy(struct feature_platform_data *pdata)
+{
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+	struct rb_node *node = rb_first(&afu->dma_regions);
+	struct fpga_afu_dma_region *region;
+
+	while (node) {
+		region = container_of(node, struct fpga_afu_dma_region, node);
+
+		dev_dbg(&pdata->dev->dev, "del region (iova = %llx)\n",
+					(unsigned long long)region->iova);
+
+		rb_erase(node, &afu->dma_regions);
+
+		if (region->iova)
+			dma_unmap_page(fpga_pdata_to_pcidev(pdata),
+					region->iova, region->length,
+					DMA_BIDIRECTIONAL);
+
+		if (region->pages)
+			afu_dma_unpin_pages(pdata, region);
+
+		node = rb_next(node);
+		kfree(region);
+	}
+}
+
+/*
+ * It finds the dma region from the rbtree based on @iova and @size:
+ * - if @size == 0, it finds the dma region which starts from @iova
+ * - otherwise, it finds the dma region which fully contains
+ *   [@iova, @iova+size)
+ * If nothing is matched returns NULL.
+ *
+ * Need to be called with pdata->lock held.
+ */
+struct fpga_afu_dma_region *
+afu_dma_region_find(struct feature_platform_data *pdata, u64 iova, u64 size)
+{
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+	struct rb_node *node = afu->dma_regions.rb_node;
+	struct device *dev = &pdata->dev->dev;
+
+	while (node) {
+		struct fpga_afu_dma_region *region;
+
+		region = container_of(node, struct fpga_afu_dma_region, node);
+
+		if (dma_region_check_iova(region, iova, size)) {
+			dev_dbg(dev, "find region (iova = %llx)\n",
+				(unsigned long long)region->iova);
+			return region;
+		}
+
+		if (iova < region->iova)
+			node = node->rb_left;
+		else if (iova > region->iova)
+			node = node->rb_right;
+		else
+			/* the iova region is not fully covered. */
+			break;
+	}
+
+	dev_dbg(dev, "region with iova %llx and size %llx is not found\n",
+		(unsigned long long)iova, (unsigned long long)size);
+	return NULL;
+}
+
+static struct fpga_afu_dma_region *
+afu_dma_region_find_iova(struct feature_platform_data *pdata, u64 iova)
+{
+	return afu_dma_region_find(pdata, iova, 0);
+}
+
+long afu_dma_map_region(struct feature_platform_data *pdata,
+		       u64 user_addr, u64 length, u64 *iova)
+{
+	struct fpga_afu_dma_region *region;
+	int ret;
+
+	/*
+	 * Check Inputs, only accept page-aligned user memory region with
+	 * valid length.
+	 */
+	if (!PAGE_ALIGNED(user_addr) || !PAGE_ALIGNED(length) || !length)
+		return -EINVAL;
+
+	/* Check overflow */
+	if (user_addr + length < user_addr)
+		return -EINVAL;
+
+	if (!access_ok(VERIFY_WRITE, (void __user *)(unsigned long)user_addr,
+		       length))
+		return -EINVAL;
+
+	region = kzalloc(sizeof(*region), GFP_KERNEL);
+	if (!region)
+		return -ENOMEM;
+
+	region->user_addr = user_addr;
+	region->length = length;
+
+	/* Pin the user memory region */
+	ret = afu_dma_pin_pages(pdata, region);
+	if (ret) {
+		dev_err(&pdata->dev->dev, "fail to pin memory region\n");
+		goto free_region;
+	}
+
+	/* Only accept continuous pages, return error if no */
+	if (!afu_dma_check_continuous_pages(region)) {
+		dev_err(&pdata->dev->dev, "pages are not continuous\n");
+		ret = -EINVAL;
+		goto unpin_pages;
+	}
+
+	/* As pages are continuous then start to do DMA mapping */
+	region->iova = dma_map_page(fpga_pdata_to_pcidev(pdata),
+				    region->pages[0], 0,
+				    region->length,
+				    DMA_BIDIRECTIONAL);
+	if (dma_mapping_error(&pdata->dev->dev, region->iova)) {
+		dev_err(&pdata->dev->dev, "fail to map dma mapping\n");
+		ret = -EFAULT;
+		goto unpin_pages;
+	}
+
+	*iova = region->iova;
+
+	mutex_lock(&pdata->lock);
+	ret = afu_dma_region_add(pdata, region);
+	mutex_unlock(&pdata->lock);
+	if (ret) {
+		dev_err(&pdata->dev->dev, "fail to add dma region\n");
+		goto unmap_dma;
+	}
+
+	return 0;
+
+unmap_dma:
+	dma_unmap_page(fpga_pdata_to_pcidev(pdata),
+		       region->iova, region->length, DMA_BIDIRECTIONAL);
+unpin_pages:
+	afu_dma_unpin_pages(pdata, region);
+free_region:
+	kfree(region);
+	return ret;
+}
+
+long afu_dma_unmap_region(struct feature_platform_data *pdata, u64 iova)
+{
+	struct fpga_afu_dma_region *region;
+
+	mutex_lock(&pdata->lock);
+	region = afu_dma_region_find_iova(pdata, iova);
+	if (!region) {
+		mutex_unlock(&pdata->lock);
+		return -EINVAL;
+	}
+
+	if (region->in_use) {
+		mutex_unlock(&pdata->lock);
+		return -EBUSY;
+	}
+
+	afu_dma_region_remove(pdata, region);
+	mutex_unlock(&pdata->lock);
+
+	dma_unmap_page(fpga_pdata_to_pcidev(pdata),
+		       region->iova, region->length, DMA_BIDIRECTIONAL);
+	afu_dma_unpin_pages(pdata, region);
+	kfree(region);
+
+	return 0;
+}
diff --git a/drivers/fpga/intel-afu-main.c b/drivers/fpga/intel-afu-main.c
index 8c7aa70..d9f1ebf 100644
--- a/drivers/fpga/intel-afu-main.c
+++ b/drivers/fpga/intel-afu-main.c
@@ -175,7 +175,11 @@ static int afu_release(struct inode *inode, struct file *filp)
 
 	dev_dbg(&pdev->dev, "Device File Release\n");
 
-	fpga_port_reset(pdev);
+	mutex_lock(&pdata->lock);
+	__fpga_port_reset(pdev);
+	afu_dma_region_destroy(pdata);
+	mutex_unlock(&pdata->lock);
+
 	feature_dev_use_end(pdata);
 	return 0;
 }
@@ -245,6 +249,55 @@ static long afu_ioctl_check_extension(struct feature_platform_data *pdata,
 	return 0;
 }
 
+static long
+afu_ioctl_dma_map(struct feature_platform_data *pdata, void __user *arg)
+{
+	struct fpga_port_dma_map map;
+	unsigned long minsz;
+	long ret;
+
+	minsz = offsetofend(struct fpga_port_dma_map, iova);
+
+	if (copy_from_user(&map, arg, minsz))
+		return -EFAULT;
+
+	if (map.argsz < minsz || map.flags)
+		return -EINVAL;
+
+	ret = afu_dma_map_region(pdata, map.user_addr, map.length, &map.iova);
+	if (ret)
+		return ret;
+
+	if (copy_to_user(arg, &map, sizeof(map))) {
+		afu_dma_unmap_region(pdata, map.iova);
+		return -EFAULT;
+	}
+
+	dev_dbg(&pdata->dev->dev, "dma map: ua=%llx, len=%llx, iova=%llx\n",
+				(unsigned long long)map.user_addr,
+				(unsigned long long)map.length,
+				(unsigned long long)map.iova);
+
+	return 0;
+}
+
+static long
+afu_ioctl_dma_unmap(struct feature_platform_data *pdata, void __user *arg)
+{
+	struct fpga_port_dma_unmap unmap;
+	unsigned long minsz;
+
+	minsz = offsetofend(struct fpga_port_dma_unmap, iova);
+
+	if (copy_from_user(&unmap, arg, minsz))
+		return -EFAULT;
+
+	if (unmap.argsz < minsz || unmap.flags)
+		return -EINVAL;
+
+	return afu_dma_unmap_region(pdata, unmap.iova);
+}
+
 static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 {
 	struct platform_device *pdev = filp->private_data;
@@ -263,6 +316,10 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 		return afu_ioctl_get_info(pdata, (void __user *)arg);
 	case FPGA_PORT_GET_REGION_INFO:
 		return afu_ioctl_get_region_info(pdata, (void __user *)arg);
+	case FPGA_PORT_DMA_MAP:
+		return afu_ioctl_dma_map(pdata, (void __user *)arg);
+	case FPGA_PORT_DMA_UNMAP:
+		return afu_ioctl_dma_unmap(pdata, (void __user *)arg);
 	default:
 		/*
 		 * Let sub-feature's ioctl function to handle the cmd
@@ -337,6 +394,7 @@ static int afu_dev_init(struct platform_device *pdev)
 	mutex_lock(&pdata->lock);
 	fpga_pdata_set_private(pdata, afu);
 	afu_region_init(pdata);
+	afu_dma_region_init(pdata);
 	mutex_unlock(&pdata->lock);
 	return 0;
 }
@@ -349,6 +407,7 @@ static int afu_dev_destroy(struct platform_device *pdev)
 	mutex_lock(&pdata->lock);
 	afu = fpga_pdata_get_private(pdata);
 	afu_region_destroy(pdata);
+	afu_dma_region_destroy(pdata);
 	fpga_pdata_set_private(pdata, NULL);
 	mutex_unlock(&pdata->lock);
 
diff --git a/drivers/fpga/intel-afu.h b/drivers/fpga/intel-afu.h
index 3417780d..23f7e24 100644
--- a/drivers/fpga/intel-afu.h
+++ b/drivers/fpga/intel-afu.h
@@ -30,11 +30,21 @@ struct fpga_afu_region {
 	struct list_head node;
 };
 
+struct fpga_afu_dma_region {
+	u64 user_addr;
+	u64 length;
+	u64 iova;
+	struct page **pages;
+	struct rb_node node;
+	bool in_use;
+};
+
 struct fpga_afu {
 	u64 region_cur_offset;
 	int num_regions;
 	u8 num_umsgs;
 	struct list_head regions;
+	struct rb_root dma_regions;
 
 	struct feature_platform_data *pdata;
 };
@@ -49,4 +59,12 @@ int afu_get_region_by_offset(struct feature_platform_data *pdata,
 			    u64 offset, u64 size,
 			    struct fpga_afu_region *pregion);
 
+void afu_dma_region_init(struct feature_platform_data *pdata);
+void afu_dma_region_destroy(struct feature_platform_data *pdata);
+long afu_dma_map_region(struct feature_platform_data *pdata,
+		       u64 user_addr, u64 length, u64 *iova);
+long afu_dma_unmap_region(struct feature_platform_data *pdata, u64 iova);
+struct fpga_afu_dma_region *afu_dma_region_find(
+		struct feature_platform_data *pdata, u64 iova, u64 size);
+
 #endif
diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
index a2ad332..b97ea02 100644
--- a/include/uapi/linux/intel-fpga.h
+++ b/include/uapi/linux/intel-fpga.h
@@ -111,6 +111,43 @@ struct fpga_port_region_info {
 
 #define FPGA_PORT_GET_REGION_INFO	_IO(FPGA_MAGIC, PORT_BASE + 2)
 
+/**
+ * FPGA_PORT_DMA_MAP - _IOWR(FPGA_MAGIC, PORT_BASE + 3,
+ *						struct fpga_port_dma_map)
+ *
+ * Map the dma memory per user_addr and length which are provided by caller.
+ * Driver fills the iova in provided struct afu_port_dma_map.
+ * This interface only accepts page-size aligned user memory for dma mapping.
+ * Return: 0 on success, -errno on failure.
+ */
+struct fpga_port_dma_map {
+	/* Input */
+	__u32 argsz;		/* Structure length */
+	__u32 flags;		/* Zero for now */
+	__u64 user_addr;        /* Process virtual address */
+	__u64 length;           /* Length of mapping (bytes)*/
+	/* Output */
+	__u64 iova;             /* IO virtual address */
+};
+
+#define FPGA_PORT_DMA_MAP	_IO(FPGA_MAGIC, PORT_BASE + 3)
+
+/**
+ * FPGA_PORT_DMA_UNMAP - _IOW(FPGA_MAGIC, PORT_BASE + 4,
+ *						struct fpga_port_dma_unmap)
+ *
+ * Unmap the dma memory per iova provided by caller.
+ * Return: 0 on success, -errno on failure.
+ */
+struct fpga_port_dma_unmap {
+	/* Input */
+	__u32 argsz;		/* Structure length */
+	__u32 flags;		/* Zero for now */
+	__u64 iova;		/* IO virtual address */
+};
+
+#define FPGA_PORT_DMA_UNMAP	_IO(FPGA_MAGIC, PORT_BASE + 4)
+
 /* IOCTLs for FME file descriptor */
 
 /**
-- 
1.8.3.1

Re: [PATCH v2 07/22] fpga: intel: pcie: parse feature list and create platform device for features.

[Moritz Fischer]

Hi Xiao / Wu,

general question, can't you make the /* */ comments
for the structs kernel-doc ([1]) markup instead (see below)?


On Mon, Jun 26, 2017 at 09:52:03AM +0800, Wu Hao wrote:
> From: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> 
> Device Feature List structure creates a link list of feature headers
> within the MMIO space to provide an extensible way of adding features.
> 
> The Intel FPGA PCIe driver walks through the feature headers to enumerate
> feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
> Function Unit (AFU), and their private sub features. For feature devices,
> it creates the platform devices and linked the private sub features into
> their platform data.
> 
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ---
> v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
>     switched to GPLv2 license.
>     fixed comments from Moritz Fischer.
>     fixed kbuild warning, typos and clean up the code.
> ---
>  drivers/fpga/Makefile            |   2 +-
>  drivers/fpga/intel-feature-dev.c | 130 ++++++
>  drivers/fpga/intel-feature-dev.h | 341 ++++++++++++++++
>  drivers/fpga/intel-pcie.c        | 841 ++++++++++++++++++++++++++++++++++++++-
>  4 files changed, 1311 insertions(+), 3 deletions(-)
>  create mode 100644 drivers/fpga/intel-feature-dev.c
>  create mode 100644 drivers/fpga/intel-feature-dev.h
> 
> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> index 5613133..ad24b3d 100644
> --- a/drivers/fpga/Makefile
> +++ b/drivers/fpga/Makefile
> @@ -31,4 +31,4 @@ obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
>  # Intel FPGA Support
>  obj-$(CONFIG_INTEL_FPGA_PCI)		+= intel-fpga-pci.o
>  
> -intel-fpga-pci-objs := intel-pcie.o
> +intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
> diff --git a/drivers/fpga/intel-feature-dev.c b/drivers/fpga/intel-feature-dev.c
> new file mode 100644
> index 0000000..68f9cba
> --- /dev/null
> +++ b/drivers/fpga/intel-feature-dev.c
> @@ -0,0 +1,130 @@
> +/*
> + * Intel FPGA Feature Device Driver
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <luwei.kang@intel.com>
> + *   Zhang Yi <yi.z.zhang@intel.com>
> + *   Wu Hao <hao.wu@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2. See
> + * the COPYING file in the top-level directory.
> + */
> +
> +#include "intel-feature-dev.h"
> +
> +void feature_platform_data_add(struct feature_platform_data *pdata,
> +			       int index, const char *name,
> +			       int resource_index, void __iomem *ioaddr)
> +{
> +	WARN_ON(index >= pdata->num);
> +
> +	pdata->features[index].name = name;
> +	pdata->features[index].resource_index = resource_index;
> +	pdata->features[index].ioaddr = ioaddr;
> +}
> +
> +struct feature_platform_data *
> +feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
> +{
> +	struct feature_platform_data *pdata;
> +
> +	pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
> +	if (pdata) {
> +		pdata->dev = dev;
> +		pdata->num = num;
> +		mutex_init(&pdata->lock);
> +	}
> +
> +	return pdata;
> +}
> +
> +int fme_feature_num(void)
> +{
> +	return FME_FEATURE_ID_MAX;
> +}
> +
> +int port_feature_num(void)
> +{
> +	return PORT_FEATURE_ID_MAX;
> +}
> +
> +int fpga_port_id(struct platform_device *pdev)
> +{
> +	struct feature_port_header *port_hdr;
> +	struct feature_port_capability capability;
> +
> +	port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> +					       PORT_FEATURE_ID_HEADER);
> +	WARN_ON(!port_hdr);
> +
> +	capability.csr = readq(&port_hdr->capability);
> +	return capability.port_number;
> +}
> +EXPORT_SYMBOL_GPL(fpga_port_id);
> +
> +/*
> + * Enable Port by clear the port soft reset bit, which is set by default.
> + * The User AFU is unable to respond to any MMIO access while in reset.
> + * __fpga_port_enable function should only be used after __fpga_port_disable
> + * function.
> + */
> +void __fpga_port_enable(struct platform_device *pdev)
> +{
> +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +	struct feature_port_header *port_hdr;
> +	struct feature_port_control control;
> +
> +	WARN_ON(!pdata->disable_count);
> +
> +	if (--pdata->disable_count != 0)
> +		return;
> +
> +	port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> +					       PORT_FEATURE_ID_HEADER);
> +	WARN_ON(!port_hdr);
> +
> +	control.csr = readq(&port_hdr->control);
> +	control.port_sftrst = 0x0;
> +	writeq(control.csr, &port_hdr->control);
> +}
> +EXPORT_SYMBOL_GPL(__fpga_port_enable);
> +
> +#define RST_POLL_INVL 10 /* us */
> +#define RST_POLL_TIMEOUT 1000 /* us */
> +
> +int __fpga_port_disable(struct platform_device *pdev)
> +{
> +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +	struct feature_port_header *port_hdr;
> +	struct feature_port_control control;
> +
> +	if (pdata->disable_count++ != 0)
> +		return 0;
> +
> +	port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> +					       PORT_FEATURE_ID_HEADER);
> +	WARN_ON(!port_hdr);
> +
> +	/* Set port soft reset */
> +	control.csr = readq(&port_hdr->control);
> +	control.port_sftrst = 0x1;
> +	writeq(control.csr, &port_hdr->control);
> +
> +	/*
> +	 * HW sets ack bit to 1 when all outstanding requests have been drained
> +	 * on this port and minimum soft reset pulse width has elapsed.
> +	 * Driver polls port_soft_reset_ack to determine if reset done by HW.
> +	 */
> +	if (readq_poll_timeout(&port_hdr->control, control.csr,
> +			       (control.port_sftrst_ack == 1),
> +			       RST_POLL_INVL, RST_POLL_TIMEOUT)) {
> +		dev_err(&pdev->dev, "timeout, fail to reset device\n");
> +		return -ETIMEDOUT;
> +	}
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(__fpga_port_disable);
> diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
> new file mode 100644
> index 0000000..f67784a
> --- /dev/null
> +++ b/drivers/fpga/intel-feature-dev.h
> @@ -0,0 +1,341 @@
> +/*
> + * Intel FPGA Feature Device Driver Header File
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <luwei.kang@intel.com>
> + *   Zhang Yi <yi.z.zhang@intel.com>
> + *   Wu Hao <hao.wu@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2. See
> + * the COPYING file in the top-level directory.
> + */
> +
> +#ifndef __INTEL_FPGA_FEATURE_H
> +#define __INTEL_FPGA_FEATURE_H
> +
> +#include <linux/fs.h>
> +#include <linux/pci.h>
> +#include <linux/uuid.h>
> +#include <linux/delay.h>
> +#include <linux/iopoll.h>
> +#include <linux/platform_device.h>
> +
> +#ifndef readq
> +static inline u64 readq(void __iomem *addr)
> +{
> +	return readl(addr) + ((u64)readl(addr + 4) << 32);
> +}
> +#endif
> +
> +#ifndef writeq
> +static inline void writeq(u64 val, void __iomem *addr)
> +{
> +	writel((u32) (val), addr);
> +	writel((u32) (val >> 32), (addr + 4));
> +}
> +#endif
> +
> +/* maximum supported number of ports */
> +#define MAX_FPGA_PORT_NUM 4
> +/* plus one for fme device */
> +#define MAX_FEATURE_DEV_NUM	(MAX_FPGA_PORT_NUM + 1)
> +
> +#define FME_FEATURE_HEADER          "fme_hdr"
> +#define FME_FEATURE_THERMAL_MGMT    "fme_thermal"
> +#define FME_FEATURE_POWER_MGMT      "fme_power"
> +#define FME_FEATURE_GLOBAL_PERF     "fme_gperf"
> +#define FME_FEATURE_GLOBAL_ERR      "fme_error"
> +#define FME_FEATURE_PR_MGMT         "fme_pr"
> +
> +#define PORT_FEATURE_HEADER         "port_hdr"
> +#define PORT_FEATURE_UAFU           "port_uafu"
> +#define PORT_FEATURE_ERR            "port_err"
> +#define PORT_FEATURE_UMSG           "port_umsg"
> +#define PORT_FEATURE_PR             "port_pr"
> +#define PORT_FEATURE_STP            "port_stp"
> +
> +/* All headers and structures must be byte-packed to match the spec. */
> +#pragma pack(1)
> +
> +/* common header for all features */
> +struct feature_header {
> +	union {
> +		u64 csr;
> +		struct {
> +			u64 id:12;
> +			u64 revision:4;
> +			u64 next_header_offset:24; /* offset to next header */
> +			u64 rsvdz:20;
> +			u64 type:4;		   /* feature type */
> +#define FEATURE_TYPE_AFU		0x1
> +#define FEATURE_TYPE_PRIVATE		0x3
> +		};
> +	};
> +};

Here for example.
> +
> +/* common header for non-private features */
> +struct feature_afu_header {
> +	uuid_le guid;
> +	union {
> +		u64 csr;
> +		struct {
> +			u64 next_afu:24;	/* pointer to next afu header */
> +			u64 rsvdz:40;
> +		};
> +	};
> +};
> +
> +/* FME Header Register Set */
> +/* FME Capability Register */
> +struct feature_fme_capability {
> +	union {
> +		u64 csr;
> +		struct {
> +			u64 fabric_verid:8;	/* Fabric version ID */
> +			u64 socket_id:1;	/* Socket id */
> +			u64 rsvdz1:3;
> +			u64 pcie0_link_avl:1;	/* PCIe0 link availability */
> +			u64 pcie1_link_avl:1;	/* PCIe1 link availability */
> +			u64 coherent_link_avl:1;/* Coherent link availability */
> +			u64 rsvdz2:1;
> +			u64 iommu_support:1;	/* IOMMU or VT-d supported */
> +			u64 num_ports:3;	/* Num of ports implemented */
> +			u64 rsvdz3:4;
> +			u64 addr_width_bits:6;	/* Address width supported */
> +			u64 rsvdz4:2;
> +			u64 cache_size:12;	/* Cache size in kb */
> +			u64 cache_assoc:4;	/* Cache Associativity */
> +			u64 rsvdz5:15;
> +			u64 lock_bit:1;		/* Latched lock bit by BIOS */
> +		};
> +	};
> +};
> +
> +/* FME Port Offset Register */
> +struct feature_fme_port {
> +	union {
> +		u64 csr;
> +		struct {
> +			u64 port_offset:24;	/* Offset to port header */
> +			u64 rsvdz1:8;
> +			u64 port_bar:3;		/* Bar id */
> +			u64 rsvdz2:20;
> +			u64 afu_access_ctrl:1;	/* AFU access type: PF/VF */
> +			u64 rsvdz3:4;
> +			u64 port_implemented:1;	/* Port implemented or not */
> +			u64 rsvdz4:3;
> +		};
> +	};
> +};
> +
> +struct feature_fme_header {
> +	struct feature_header header;
> +	struct feature_afu_header afu_header;
> +	u64 rsvd[2];
> +	struct feature_fme_capability capability;
> +	struct feature_fme_port port[MAX_FPGA_PORT_NUM];
> +};
> +
> +/* FME Thermal Sub Feature Register Set */
> +struct feature_fme_thermal {
> +	struct feature_header header;
> +};
> +
> +/* FME Power Sub Feature Register Set */
> +struct feature_fme_power {
> +	struct feature_header header;
> +};
> +
> +/* FME Global Performance Sub Feature Register Set */
> +struct feature_fme_gperf {
> +	struct feature_header header;
> +};
> +
> +/* FME Error Sub Feature Register Set */
> +struct feature_fme_err {
> +	struct feature_header header;
> +};
> +
> +/* FME Partial Reconfiguration Sub Feature Register Set */
> +struct feature_fme_pr {
> +	struct feature_header header;
> +};
> +
> +/* PORT Header Register Set */
> +/* Port Capability Register */
> +struct feature_port_capability {
> +	union {
> +		u64 csr;
> +		struct {
> +			u64 port_number:2;	/* Port Number 0-3 */
> +			u64 rsvdz1:6;
> +			u64 mmio_size:16;	/* User MMIO size in KB */
> +			u64 rsvdz2:8;
> +			u64 sp_intr_num:4;	/* Supported interrupts num */
> +			u64 rsvdz3:28;
> +		};
> +	};
> +};
> +
> +/* Port Control Register */
> +struct feature_port_control {
> +	union {
> +		u64 csr;
> +		struct {
> +			u64 port_sftrst:1;	/* Port Soft Reset */
> +			u64 rsvdz1:1;
> +			u64 latency_tolerance:1;/* '1' >= 40us, '0' < 40us */
> +			u64 rsvdz2:1;
> +			u64 port_sftrst_ack:1;	/* HW ACK for Soft Reset */
> +			u64 rsvdz3:59;
> +		};
> +	};
> +};
> +
> +struct feature_port_header {
> +	struct feature_header header;
> +	struct feature_afu_header afu_header;
> +	u64 rsvd[2];
> +	struct feature_port_capability capability;
> +	struct feature_port_control control;
> +};
> +
> +/* PORT Error Sub Feature Register Set */
> +struct feature_port_error {
> +	struct feature_header header;
> +};
> +
> +/* PORT Unordered Message Sub Feature Register Set */
> +struct feature_port_umsg {
> +	struct feature_header header;
> +};
> +
> +/* PORT SignalTap Sub Feature Register Set */
> +struct feature_port_stp {
> +	struct feature_header header;
> +};
> +
> +#pragma pack()
> +
> +struct feature {
> +	const char *name;
> +	int resource_index;
> +	void __iomem *ioaddr;
> +};
> +
> +struct feature_platform_data {
> +	/* list the feature dev to cci_drvdata->port_dev_list. */
> +	struct list_head node;
> +	struct mutex lock;
> +	struct platform_device *dev;
> +	unsigned int disable_count;	/* count for port disable */
> +
> +	int num;			/* number of features */
> +	struct feature features[0];
> +};
> +
> +enum fme_feature_id {
> +	FME_FEATURE_ID_HEADER = 0x0,
> +	FME_FEATURE_ID_THERMAL_MGMT = 0x1,
> +	FME_FEATURE_ID_POWER_MGMT = 0x2,
> +	FME_FEATURE_ID_GLOBAL_PERF = 0x3,
> +	FME_FEATURE_ID_GLOBAL_ERR = 0x4,
> +	FME_FEATURE_ID_PR_MGMT = 0x5,
> +	FME_FEATURE_ID_MAX = 0x6,
> +};
> +
> +enum port_feature_id {
> +	PORT_FEATURE_ID_HEADER = 0x0,
> +	PORT_FEATURE_ID_ERROR = 0x1,
> +	PORT_FEATURE_ID_UMSG = 0x2,
> +	PORT_FEATURE_ID_PR = 0x3,
> +	PORT_FEATURE_ID_STP = 0x4,
> +	PORT_FEATURE_ID_UAFU = 0x5,
> +	PORT_FEATURE_ID_MAX = 0x6,
> +};
> +
> +int fme_feature_num(void);
> +int port_feature_num(void);
> +
> +#define FPGA_FEATURE_DEV_FME		"intel-fpga-fme"
> +#define FPGA_FEATURE_DEV_PORT		"intel-fpga-port"
> +
> +void feature_platform_data_add(struct feature_platform_data *pdata,
> +			       int index, const char *name,
> +			       int resource_index, void __iomem *ioaddr);
> +
> +static inline int feature_platform_data_size(const int num)
> +{
> +	return sizeof(struct feature_platform_data) +
> +		num * sizeof(struct feature);
> +}
> +
> +struct feature_platform_data *
> +feature_platform_data_alloc_and_init(struct platform_device *dev, int num);
> +
> +int fpga_port_id(struct platform_device *pdev);
> +
> +static inline int fpga_port_check_id(struct platform_device *pdev,
> +				     void *pport_id)
> +{
> +	return fpga_port_id(pdev) == *(int *)pport_id;
> +}
> +
> +void __fpga_port_enable(struct platform_device *pdev);
> +int __fpga_port_disable(struct platform_device *pdev);
> +
> +static inline void fpga_port_enable(struct platform_device *pdev)
> +{
> +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +
> +	mutex_lock(&pdata->lock);
> +	__fpga_port_enable(pdev);
> +	mutex_unlock(&pdata->lock);
> +}
> +
> +static inline int fpga_port_disable(struct platform_device *pdev)
> +{
> +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +	int ret;
> +
> +	mutex_lock(&pdata->lock);
> +	ret = __fpga_port_disable(pdev);
> +	mutex_unlock(&pdata->lock);
> +
> +	return ret;
> +}
> +
> +static inline int __fpga_port_reset(struct platform_device *pdev)
> +{
> +	int ret;
> +
> +	ret = __fpga_port_disable(pdev);
> +	if (ret)
> +		return ret;
> +
> +	__fpga_port_enable(pdev);
> +	return 0;
> +}
> +
> +static inline int fpga_port_reset(struct platform_device *pdev)
> +{
> +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +	int ret;
> +
> +	mutex_lock(&pdata->lock);
> +	ret = __fpga_port_reset(pdev);
> +	mutex_unlock(&pdata->lock);
> +	return ret;
> +}
> +
> +static inline void __iomem *
> +get_feature_ioaddr_by_index(struct device *dev, int index)
> +{
> +	struct feature_platform_data *pdata = dev_get_platdata(dev);
> +
> +	return pdata->features[index].ioaddr;
> +}
> +#endif
> diff --git a/drivers/fpga/intel-pcie.c b/drivers/fpga/intel-pcie.c
> index f697de4..70b8284 100644
> --- a/drivers/fpga/intel-pcie.c
> +++ b/drivers/fpga/intel-pcie.c
> @@ -23,10 +23,827 @@
>  #include <linux/stddef.h>
>  #include <linux/errno.h>
>  #include <linux/aer.h>
> +#include <linux/fpga/fpga-dev.h>
> +
> +#include "intel-feature-dev.h"
>  
>  #define DRV_VERSION	"0.8"
>  #define DRV_NAME	"intel-fpga-pci"
>  
> +#define INTEL_FPGA_DEV	"intel-fpga-dev"
> +
> +static DEFINE_MUTEX(fpga_id_mutex);
> +
> +enum fpga_id_type {
> +	FME_ID,		/* fme id allocation and mapping */
> +	PORT_ID,	/* port id allocation and mapping */
> +	FPGA_ID_MAX,
> +};
> +
> +/* it is protected by fpga_id_mutex */
> +static struct idr fpga_ids[FPGA_ID_MAX];
> +
> +struct cci_drvdata {
> +	struct device *fme_dev;
> +
> +	struct mutex lock;
> +	struct list_head port_dev_list;
> +
> +	struct list_head regions; /* global list of pci bar mapping region */
> +};
> +
> +/* pci bar mapping info */
> +struct cci_pci_region {
> +	int bar;
> +	void __iomem *ioaddr;	/* pointer to mapped bar region */
> +	struct list_head node;
> +};
> +
> +static void fpga_ids_init(void)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> +		idr_init(fpga_ids + i);
> +}
> +
> +static void fpga_ids_destroy(void)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> +		idr_destroy(fpga_ids + i);
> +}
> +
> +static int alloc_fpga_id(enum fpga_id_type type, struct device *dev)
> +{
> +	int id;
> +
> +	WARN_ON(type >= FPGA_ID_MAX);
> +	mutex_lock(&fpga_id_mutex);
> +	id = idr_alloc(fpga_ids + type, dev, 0, 0, GFP_KERNEL);
> +	mutex_unlock(&fpga_id_mutex);
> +	return id;
> +}
> +
> +static void free_fpga_id(enum fpga_id_type type, int id)
> +{
> +	WARN_ON(type >= FPGA_ID_MAX);
> +	mutex_lock(&fpga_id_mutex);
> +	idr_remove(fpga_ids + type, id);
> +	mutex_unlock(&fpga_id_mutex);
> +}
> +
> +static void cci_pci_add_port_dev(struct pci_dev *pdev,
> +				 struct platform_device *port_dev)
> +{
> +	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +	struct feature_platform_data *pdata = dev_get_platdata(&port_dev->dev);
> +
> +	mutex_lock(&drvdata->lock);
> +	list_add(&pdata->node, &drvdata->port_dev_list);
> +	get_device(&pdata->dev->dev);
> +	mutex_unlock(&drvdata->lock);
> +}
> +
> +static void cci_pci_remove_port_devs(struct pci_dev *pdev)
> +{
> +	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +	struct feature_platform_data *pdata, *ptmp;
> +
> +	mutex_lock(&drvdata->lock);
> +	list_for_each_entry_safe(pdata, ptmp, &drvdata->port_dev_list, node) {
> +		struct platform_device *port_dev = pdata->dev;
> +
> +		/* the port should be unregistered first. */
> +		WARN_ON(device_is_registered(&port_dev->dev));
> +		list_del(&pdata->node);
> +		free_fpga_id(PORT_ID, port_dev->id);
> +		put_device(&port_dev->dev);
> +	}
> +	mutex_unlock(&drvdata->lock);
> +}
> +
> +/* info collection during feature dev build. */
> +struct build_feature_devs_info {
> +	struct pci_dev *pdev;
> +
> +	/*
> +	 * PCI BAR mapping info. Parsing feature list starts from
> +	 * BAR 0 and switch to different BARs to parse Port
> +	 */
> +	void __iomem *ioaddr;
> +	void __iomem *ioend;
> +	int current_bar;
> +
> +	/* points to FME header where the port offset is figured out. */
> +	void __iomem *pfme_hdr;
> +
> +	/* the container device for all feature devices */
> +	struct fpga_dev *parent_dev;
> +
> +	/* current feature device */
> +	struct platform_device *feature_dev;
> +};
> +
> +static void cci_pci_release_regions(struct pci_dev *pdev)
> +{
> +	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +	struct cci_pci_region *tmp, *region;
> +
> +	list_for_each_entry_safe(region, tmp, &drvdata->regions, node) {
> +		list_del(&region->node);
> +		if (region->ioaddr)
> +			pci_iounmap(pdev, region->ioaddr);
> +		devm_kfree(&pdev->dev, region);
> +	}
> +}
> +
> +static void __iomem *cci_pci_ioremap_bar(struct pci_dev *pdev, int bar)
> +{
> +	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +	struct cci_pci_region *region;
> +
> +	list_for_each_entry(region, &drvdata->regions, node)
> +		if (region->bar == bar) {
> +			dev_dbg(&pdev->dev, "BAR %d region exists\n", bar);
> +			return region->ioaddr;
> +		}
> +
> +	region = devm_kzalloc(&pdev->dev, sizeof(*region), GFP_KERNEL);
> +	if (!region)
> +		return NULL;
> +
> +	region->bar = bar;
> +	region->ioaddr = pci_ioremap_bar(pdev, bar);
> +	if (!region->ioaddr) {
> +		dev_err(&pdev->dev, "can't ioremap memory from BAR %d.\n", bar);
> +		devm_kfree(&pdev->dev, region);
> +		return NULL;
> +	}
> +
> +	list_add(&region->node, &drvdata->regions);
> +	return region->ioaddr;
> +}
> +
> +static int parse_start_from(struct build_feature_devs_info *binfo, int bar)
> +{
> +	binfo->ioaddr = cci_pci_ioremap_bar(binfo->pdev, bar);
> +	if (!binfo->ioaddr)
> +		return -ENOMEM;
> +
> +	binfo->current_bar = bar;
> +	binfo->ioend = binfo->ioaddr + pci_resource_len(binfo->pdev, bar);
> +	return 0;
> +}
> +
> +static int parse_start(struct build_feature_devs_info *binfo)
> +{
> +	/* fpga feature list starts from BAR 0 */
> +	return parse_start_from(binfo, 0);
> +}
> +
> +/* switch the memory mapping to BAR# @bar */
> +static int parse_switch_to(struct build_feature_devs_info *binfo, int bar)
> +{
> +	return parse_start_from(binfo, bar);
> +}
> +
> +static struct build_feature_devs_info *
> +build_info_alloc_and_init(struct pci_dev *pdev)
> +{
> +	struct build_feature_devs_info *binfo;
> +
> +	binfo = devm_kzalloc(&pdev->dev, sizeof(*binfo), GFP_KERNEL);
> +	if (binfo)
> +		binfo->pdev = pdev;
> +
> +	return binfo;
> +}
> +
> +static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
> +{
> +	if (!strcmp(pdev->name, FPGA_FEATURE_DEV_FME))
> +		return FME_ID;
> +
> +	if (!strcmp(pdev->name, FPGA_FEATURE_DEV_PORT))
> +		return PORT_ID;
> +
> +	WARN_ON(1);
> +	return FPGA_ID_MAX;
> +}
> +
> +/*
> + * register current feature device, it is called when we need to switch to
> + * another feature parsing or we have parsed all features
> + */
> +static int build_info_commit_dev(struct build_feature_devs_info *binfo)
> +{
> +	int ret;
> +
> +	if (!binfo->feature_dev)
> +		return 0;
> +
> +	ret = platform_device_add(binfo->feature_dev);
> +	if (!ret) {
> +		struct cci_drvdata *drvdata;
> +
> +		drvdata = dev_get_drvdata(&binfo->pdev->dev);
> +		if (feature_dev_id_type(binfo->feature_dev) == PORT_ID)
> +			cci_pci_add_port_dev(binfo->pdev, binfo->feature_dev);
> +		else
> +			drvdata->fme_dev = get_device(&binfo->feature_dev->dev);
> +
> +		/*
> +		 * reset it to avoid build_info_free() freeing their resource.
> +		 *
> +		 * The resource of successfully registered feature devices
> +		 * will be freed by platform_device_unregister(). See the
> +		 * comments in build_info_create_dev().
> +		 */
> +		binfo->feature_dev = NULL;
> +	}
> +
> +	return ret;
> +}
> +
> +static int
> +build_info_create_dev(struct build_feature_devs_info *binfo,
> +		      enum fpga_id_type type, int feature_nr, const char *name)
> +{
> +	struct platform_device *fdev;
> +	struct resource *res;
> +	struct feature_platform_data *pdata;
> +	int ret;
> +
> +	/* we will create a new device, commit current device first */
> +	ret = build_info_commit_dev(binfo);
> +	if (ret)
> +		return ret;
> +
> +	/*
> +	 * we use -ENODEV as the initialization indicator which indicates
> +	 * whether the id need to be reclaimed
> +	 */
> +	fdev = binfo->feature_dev = platform_device_alloc(name, -ENODEV);
> +	if (!fdev)
> +		return -ENOMEM;
> +
> +	fdev->id = alloc_fpga_id(type, &fdev->dev);
> +	if (fdev->id < 0)
> +		return fdev->id;
> +
> +	fdev->dev.parent = &binfo->parent_dev->dev;
> +
> +	/*
> +	 * we do not need to care for the memory which is associated with
> +	 * the platform device. After calling platform_device_unregister(),
> +	 * it will be automatically freed by device's release() callback,
> +	 * platform_device_release().
> +	 */
> +	pdata = feature_platform_data_alloc_and_init(fdev, feature_nr);
> +	if (!pdata)
> +		return -ENOMEM;
> +
> +	/*
> +	 * the count should be initialized to 0 to make sure
> +	 *__fpga_port_enable() following __fpga_port_disable()
> +	 * works properly for port device.
> +	 * and it should always be 0 for fme device.
> +	 */
> +	WARN_ON(pdata->disable_count);
> +
> +	fdev->dev.platform_data = pdata;
> +	fdev->num_resources = feature_nr;
> +	fdev->resource = kcalloc(feature_nr, sizeof(*res), GFP_KERNEL);
> +	if (!fdev->resource)
> +		return -ENOMEM;
> +
> +	return 0;
> +}
> +
> +static int remove_feature_dev(struct device *dev, void *data)
> +{
> +	struct platform_device *pdev = to_platform_device(dev);
> +
> +	platform_device_unregister(pdev);
> +	return 0;
> +}
> +
> +static int remove_parent_dev(struct device *dev, void *data)
> +{
> +	/* remove platform devices attached in the parent device */
> +	device_for_each_child(dev, NULL, remove_feature_dev);
> +	fpga_dev_destroy(to_fpga_dev(dev));
> +	return 0;
> +}
> +
> +static void remove_all_devs(struct pci_dev *pdev)
> +{
> +	/* remove parent device and all its children. */
> +	device_for_each_child(&pdev->dev, NULL, remove_parent_dev);
> +}
> +
> +static void build_info_free(struct build_feature_devs_info *binfo)
> +{
> +	if (!IS_ERR_OR_NULL(binfo->parent_dev))
> +		remove_all_devs(binfo->pdev);
> +
> +	/*
> +	 * it is a valid id, free it. See comments in
> +	 * build_info_create_dev()
> +	 */
> +	if (binfo->feature_dev && binfo->feature_dev->id >= 0)
> +		free_fpga_id(feature_dev_id_type(binfo->feature_dev),
> +			     binfo->feature_dev->id);
> +
> +	platform_device_put(binfo->feature_dev);
> +
> +	devm_kfree(&binfo->pdev->dev, binfo);
> +}
> +
> +#define FEATURE_TYPE_AFU	0x1
> +#define FEATURE_TYPE_PRIVATE	0x3
> +
> +/* FME and PORT GUID are fixed */
> +#define FEATURE_FME_GUID "f9e17764-38f0-82fe-e346-524ae92aafbf"
> +#define FEATURE_PORT_GUID "6b355b87-b06c-9642-eb42-8d139398b43a"
> +
> +static bool feature_is_fme(struct feature_afu_header *afu_hdr)
> +{
> +	uuid_le u;
> +
> +	uuid_le_to_bin(FEATURE_FME_GUID, &u);
> +
> +	return !uuid_le_cmp(u, afu_hdr->guid);
> +}
> +
> +static bool feature_is_port(struct feature_afu_header *afu_hdr)
> +{
> +	uuid_le u;
> +
> +	uuid_le_to_bin(FEATURE_PORT_GUID, &u);
> +
> +	return !uuid_le_cmp(u, afu_hdr->guid);
> +}
> +
> +/*
> + * UAFU GUID is dynamic as it can be changed after FME downloads different
> + * Green Bitstream to the port, so we treat the unknown GUIDs which are
> + * attached on port's feature list as UAFU.
> + */
> +static bool feature_is_UAFU(struct build_feature_devs_info *binfo)
> +{
> +	if (!binfo->feature_dev ||
> +	      feature_dev_id_type(binfo->feature_dev) != PORT_ID)
> +		return false;
> +
> +	return true;
> +}
> +
> +static void
> +build_info_add_sub_feature(struct build_feature_devs_info *binfo,
> +			   int feature_id, const char *feature_name,
> +			   resource_size_t resource_size, void __iomem *start)
> +{
> +
> +	struct platform_device *fdev = binfo->feature_dev;
> +	struct feature_platform_data *pdata = dev_get_platdata(&fdev->dev);
> +	struct resource *res = &fdev->resource[feature_id];
> +
> +	res->start = pci_resource_start(binfo->pdev, binfo->current_bar) +
> +		start - binfo->ioaddr;
> +	res->end = res->start + resource_size - 1;
> +	res->flags = IORESOURCE_MEM;
> +	res->name = feature_name;
> +
> +	feature_platform_data_add(pdata, feature_id,
> +				  feature_name, feature_id, start);
> +}
> +
> +struct feature_info {
> +	const char *name;
> +	resource_size_t resource_size;
> +	int feature_index;
> +};
> +
> +/* indexed by fme feature IDs which are defined in 'enum fme_feature_id'. */
> +static struct feature_info fme_features[] = {
> +	{
> +		.name = FME_FEATURE_HEADER,
> +		.resource_size = sizeof(struct feature_fme_header),
> +		.feature_index = FME_FEATURE_ID_HEADER,
> +	},
> +	{
> +		.name = FME_FEATURE_THERMAL_MGMT,
> +		.resource_size = sizeof(struct feature_fme_thermal),
> +		.feature_index = FME_FEATURE_ID_THERMAL_MGMT,
> +	},
> +	{
> +		.name = FME_FEATURE_POWER_MGMT,
> +		.resource_size = sizeof(struct feature_fme_power),
> +		.feature_index = FME_FEATURE_ID_POWER_MGMT,
> +	},
> +	{
> +		.name = FME_FEATURE_GLOBAL_PERF,
> +		.resource_size = sizeof(struct feature_fme_gperf),
> +		.feature_index = FME_FEATURE_ID_GLOBAL_PERF,
> +	},
> +	{
> +		.name = FME_FEATURE_GLOBAL_ERR,
> +		.resource_size = sizeof(struct feature_fme_err),
> +		.feature_index = FME_FEATURE_ID_GLOBAL_ERR,
> +	},
> +	{
> +		.name = FME_FEATURE_PR_MGMT,
> +		.resource_size = sizeof(struct feature_fme_pr),
> +		.feature_index = FME_FEATURE_ID_PR_MGMT,
> +	}
> +};
> +
> +/* indexed by port feature IDs which are defined in 'enum port_feature_id'. */
> +static struct feature_info port_features[] = {
> +	{
> +		.name = PORT_FEATURE_HEADER,
> +		.resource_size = sizeof(struct feature_port_header),
> +		.feature_index = PORT_FEATURE_ID_HEADER,
> +	},
> +	{
> +		.name = PORT_FEATURE_ERR,
> +		.resource_size = sizeof(struct feature_port_error),
> +		.feature_index = PORT_FEATURE_ID_ERROR,
> +	},
> +	{
> +		.name = PORT_FEATURE_UMSG,
> +		.resource_size = sizeof(struct feature_port_umsg),
> +		.feature_index = PORT_FEATURE_ID_UMSG,
> +	},
> +	{
> +		/* This feature isn't available for now */
> +		.name = PORT_FEATURE_PR,
> +		.resource_size = 0,
> +		.feature_index = PORT_FEATURE_ID_PR,
> +	},
> +	{
> +		.name = PORT_FEATURE_STP,
> +		.resource_size = sizeof(struct feature_port_stp),
> +		.feature_index = PORT_FEATURE_ID_STP,
> +	},
> +	{
> +		/*
> +		 * For User AFU feature, its region size is not fixed, but
> +		 * reported by register PortCapability.mmio_size. Resource
> +		 * size of UAFU will be set while parse port device.
> +		 */
> +		.name = PORT_FEATURE_UAFU,
> +		.resource_size = 0,
> +		.feature_index = PORT_FEATURE_ID_UAFU,
> +	},
> +};
> +
> +static int
> +create_feature_instance(struct build_feature_devs_info *binfo,
> +			void __iomem *start, struct feature_info *finfo)
> +{
> +	if (binfo->ioend - start < finfo->resource_size)
> +		return -EINVAL;
> +
> +	build_info_add_sub_feature(binfo, finfo->feature_index, finfo->name,
> +				   finfo->resource_size, start);
> +	return 0;
> +}
> +
> +static int parse_feature_fme(struct build_feature_devs_info *binfo,
> +			     void __iomem *start)
> +{
> +	struct cci_drvdata *drvdata = dev_get_drvdata(&binfo->pdev->dev);
> +	int ret;
> +
> +	ret = build_info_create_dev(binfo, FME_ID, fme_feature_num(),
> +					FPGA_FEATURE_DEV_FME);
> +	if (ret)
> +		return ret;
> +
> +	if (drvdata->fme_dev) {
> +		dev_err(&binfo->pdev->dev, "Multiple FMEs are detected.\n");
> +		return -EINVAL;
> +	}
> +
> +	return create_feature_instance(binfo, start,
> +				       &fme_features[FME_FEATURE_ID_HEADER]);
> +}
> +
> +static int parse_feature_fme_private(struct build_feature_devs_info *binfo,
> +				     struct feature_header *hdr)
> +{
> +	struct feature_header header;
> +
> +	header.csr = readq(hdr);
> +
> +	if (header.id >= ARRAY_SIZE(fme_features)) {
> +		dev_info(&binfo->pdev->dev, "FME feature id %x is not supported yet.\n",
> +			 header.id);
> +		return 0;
> +	}
> +
> +	return create_feature_instance(binfo, hdr, &fme_features[header.id]);
> +}
> +
> +static int parse_feature_port(struct build_feature_devs_info *binfo,
> +			     void __iomem *start)
> +{
> +	int ret;
> +
> +	ret = build_info_create_dev(binfo, PORT_ID, port_feature_num(),
> +					FPGA_FEATURE_DEV_PORT);
> +	if (ret)
> +		return ret;
> +
> +	return create_feature_instance(binfo, start,
> +				       &port_features[PORT_FEATURE_ID_HEADER]);
> +}
> +
> +static void enable_port_uafu(struct build_feature_devs_info *binfo,
> +			     void __iomem *start)
> +{
> +	enum port_feature_id id = PORT_FEATURE_ID_UAFU;
> +	struct feature_port_header *port_hdr;
> +	struct feature_port_capability capability;
> +
> +	port_hdr = (struct feature_port_header *)start;
> +	capability.csr = readq(&port_hdr->capability);
> +	port_features[id].resource_size = capability.mmio_size << 10;
> +
> +	/*
> +	 * To enable User AFU, driver needs to clear reset bit on related port,
> +	 * otherwise the mmio space of this user AFU will be invalid.
> +	 */
> +	if (port_features[id].resource_size)
> +		fpga_port_reset(binfo->feature_dev);
> +}
> +
> +static int parse_feature_port_private(struct build_feature_devs_info *binfo,
> +				      struct feature_header *hdr)
> +{
> +	struct feature_header header;
> +	enum port_feature_id id;
> +
> +	header.csr = readq(hdr);
> +	/*
> +	 * the region of port feature id is [0x10, 0x13], + 1 to reserve 0
> +	 * which is dedicated for port-hdr.
> +	 */
> +	id = (header.id & 0x000f) + 1;
> +
> +	if (id >= ARRAY_SIZE(port_features)) {
> +		dev_info(&binfo->pdev->dev, "Port feature id %x is not supported yet.\n",
> +			 header.id);
> +		return 0;
> +	}
> +
> +	return create_feature_instance(binfo, hdr, &port_features[id]);
> +}
> +
> +static int parse_feature_port_uafu(struct build_feature_devs_info *binfo,
> +				 struct feature_header *hdr)
> +{
> +	enum port_feature_id id = PORT_FEATURE_ID_UAFU;
> +	int ret;
> +
> +	if (port_features[id].resource_size) {
> +		ret = create_feature_instance(binfo, hdr, &port_features[id]);
> +		port_features[id].resource_size = 0;
> +	} else {
> +		dev_err(&binfo->pdev->dev, "the uafu feature header is mis-configured.\n");
> +		ret = -EINVAL;
> +	}
> +
> +	return ret;
> +}
> +
> +static int parse_feature_afus(struct build_feature_devs_info *binfo,
> +			      struct feature_header *hdr)
> +{
> +	int ret;
> +	struct feature_afu_header *afu_hdr, header;
> +	void __iomem *start;
> +	void __iomem *end = binfo->ioend;
> +
> +	start = hdr;
> +	for (; start < end; start += header.next_afu) {
> +		if (end - start < (sizeof(*afu_hdr) + sizeof(*hdr)))
> +			return -EINVAL;
> +
> +		hdr = start;
> +		afu_hdr = (struct feature_afu_header *) (hdr + 1);
> +		header.csr = readq(&afu_hdr->csr);
> +
> +		if (feature_is_fme(afu_hdr)) {
> +			ret = parse_feature_fme(binfo, hdr);
> +			binfo->pfme_hdr = hdr;
> +			if (ret)
> +				return ret;
> +		} else if (feature_is_port(afu_hdr)) {
> +			ret = parse_feature_port(binfo, hdr);
> +			enable_port_uafu(binfo, hdr);
> +			if (ret)
> +				return ret;
> +		} else if (feature_is_UAFU(binfo)) {
> +			ret = parse_feature_port_uafu(binfo, hdr);
> +			if (ret)
> +				return ret;
> +		} else
> +			dev_info(&binfo->pdev->dev, "AFU GUID %pUl is not supported yet.\n",
> +				 afu_hdr->guid.b);
> +
> +		if (!header.next_afu)
> +			break;
> +	}
> +
> +	return 0;
> +}
> +
> +static int parse_feature_private(struct build_feature_devs_info *binfo,
> +				 struct feature_header *hdr)
> +{
> +	struct feature_header header;
> +
> +	header.csr = readq(hdr);
> +
> +	if (!binfo->feature_dev) {
> +		dev_err(&binfo->pdev->dev, "the private feature %x does not belong to any AFU.\n",
> +			header.id);
> +		return -EINVAL;
> +	}
> +
> +	switch (feature_dev_id_type(binfo->feature_dev)) {
> +	case FME_ID:
> +		return parse_feature_fme_private(binfo, hdr);
> +	case PORT_ID:
> +		return parse_feature_port_private(binfo, hdr);
> +	default:
> +		dev_info(&binfo->pdev->dev, "private feature %x belonging to AFU %s is not supported yet.\n",
> +			 header.id, binfo->feature_dev->name);
> +	}
> +	return 0;
> +}
> +
> +static int parse_feature(struct build_feature_devs_info *binfo,
> +			 struct feature_header *hdr)
> +{
> +	struct feature_header header;
> +	int ret = 0;
> +
> +	header.csr = readq(hdr);
> +
> +	switch (header.type) {
> +	case FEATURE_TYPE_AFU:
> +		ret = parse_feature_afus(binfo, hdr);
> +		break;
> +	case FEATURE_TYPE_PRIVATE:
> +		ret = parse_feature_private(binfo, hdr);
> +		break;
> +	default:
> +		dev_info(&binfo->pdev->dev,
> +			 "Feature Type %x is not supported.\n", hdr->type);
> +	};
> +
> +	return ret;
> +}
> +
> +static int
> +parse_feature_list(struct build_feature_devs_info *binfo, void __iomem *start)
> +{
> +	struct feature_header *hdr, header;
> +	void __iomem *end = binfo->ioend;
> +	int ret = 0;
> +
> +	for (; start < end; start += header.next_header_offset) {
> +		if (end - start < sizeof(*hdr)) {
> +			dev_err(&binfo->pdev->dev, "The region is too small to contain a feature.\n");
> +			ret =  -EINVAL;
> +			break;
> +		}
> +
> +		hdr = (struct feature_header *)start;
> +		ret = parse_feature(binfo, hdr);
> +		if (ret)
> +			break;
> +
> +		header.csr = readq(hdr);
> +		if (!header.next_header_offset)
> +			break;
> +	}
> +
> +	return ret;
> +}
> +
> +static int parse_ports_from_fme(struct build_feature_devs_info *binfo)
> +{
> +	struct feature_fme_header *fme_hdr;
> +	struct feature_fme_port port;
> +	int i = 0, ret = 0;
> +
> +	if (binfo->pfme_hdr == NULL) {
> +		dev_dbg(&binfo->pdev->dev, "VF is detected.\n");
> +		return ret;
> +	}
> +
> +	fme_hdr = binfo->pfme_hdr;
> +
> +	do {
> +		port.csr = readq(&fme_hdr->port[i]);
> +		if (!port.port_implemented)
> +			break;
> +
> +		ret = parse_switch_to(binfo, port.port_bar);
> +		if (ret)
> +			break;
> +
> +		ret = parse_feature_list(binfo,
> +				binfo->ioaddr + port.port_offset);
> +		if (ret)
> +			break;
> +	} while (++i < MAX_FPGA_PORT_NUM);
> +
> +	return ret;
> +}
> +
> +static int create_init_drvdata(struct pci_dev *pdev)
> +{
> +	struct cci_drvdata *drvdata;
> +
> +	drvdata = devm_kzalloc(&pdev->dev, sizeof(*drvdata), GFP_KERNEL);
> +	if (!drvdata)
> +		return -ENOMEM;
> +
> +	mutex_init(&drvdata->lock);
> +	INIT_LIST_HEAD(&drvdata->port_dev_list);
> +	INIT_LIST_HEAD(&drvdata->regions);
> +
> +	dev_set_drvdata(&pdev->dev, drvdata);
> +	return 0;
> +}
> +
> +static void destroy_drvdata(struct pci_dev *pdev)
> +{
> +	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +
> +	if (drvdata->fme_dev) {
> +		/* fme device should be unregistered first. */
> +		WARN_ON(device_is_registered(drvdata->fme_dev));
> +		free_fpga_id(FME_ID, to_platform_device(drvdata->fme_dev)->id);
> +		put_device(drvdata->fme_dev);
> +	}
> +
> +	cci_pci_remove_port_devs(pdev);
> +	cci_pci_release_regions(pdev);
> +	dev_set_drvdata(&pdev->dev, NULL);
> +	devm_kfree(&pdev->dev, drvdata);
> +}
> +
> +static int cci_pci_create_feature_devs(struct pci_dev *pdev)
> +{
> +	struct build_feature_devs_info *binfo;
> +	int ret;
> +
> +	binfo = build_info_alloc_and_init(pdev);
> +	if (!binfo)
> +		return -ENOMEM;
> +
> +	binfo->parent_dev = fpga_dev_create(&pdev->dev, INTEL_FPGA_DEV);
> +	if (IS_ERR(binfo->parent_dev)) {
> +		ret = PTR_ERR(binfo->parent_dev);
> +		goto free_binfo_exit;
> +	}
> +
> +	ret = parse_start(binfo);
> +	if (ret)
> +		goto free_binfo_exit;
> +
> +	ret = parse_feature_list(binfo, binfo->ioaddr);
> +	if (ret)
> +		goto free_binfo_exit;
> +
> +	ret = parse_ports_from_fme(binfo);
> +	if (ret)
> +		goto free_binfo_exit;
> +
> +	ret = build_info_commit_dev(binfo);
> +	if (ret)
> +		goto free_binfo_exit;
> +
> +	/*
> +	 * everything is okay, reset ->parent_dev to stop it being
> +	 * freed by build_info_free()
> +	 */
> +	binfo->parent_dev = NULL;
> +
> +free_binfo_exit:
> +	build_info_free(binfo);
> +	return ret;
> +}
> +
>  /* PCI Device ID */
>  #define PCIe_DEVICE_ID_PF_INT_5_X	0xBCBD
>  #define PCIe_DEVICE_ID_PF_INT_6_X	0xBCC0
> @@ -81,9 +898,18 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
>  		goto release_region_exit;
>  	}
>  
> -	/* TODO: create and add the platform device per feature list */
> +	ret = create_init_drvdata(pcidev);
> +	if (ret)
> +		goto release_region_exit;
> +
> +	ret = cci_pci_create_feature_devs(pcidev);
> +	if (ret)
> +		goto destroy_drvdata_exit;
> +
>  	return 0;
>  
> +destroy_drvdata_exit:
> +	destroy_drvdata(pcidev);
>  release_region_exit:
>  	pci_release_regions(pcidev);
>  disable_error_report_exit:
> @@ -94,6 +920,8 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
>  
>  static void cci_pci_remove(struct pci_dev *pcidev)
>  {
> +	remove_all_devs(pcidev);
> +	destroy_drvdata(pcidev);
>  	pci_release_regions(pcidev);
>  	pci_disable_pcie_error_reporting(pcidev);
>  	pci_disable_device(pcidev);
> @@ -108,14 +936,23 @@ static void cci_pci_remove(struct pci_dev *pcidev)
>  
>  static int __init ccidrv_init(void)
>  {
> +	int ret;
> +
>  	pr_info("Intel(R) FPGA PCIe Driver: Version %s\n", DRV_VERSION);
>  
> -	return pci_register_driver(&cci_pci_driver);
> +	fpga_ids_init();
> +
> +	ret = pci_register_driver(&cci_pci_driver);
> +	if (ret)
> +		fpga_ids_destroy();
> +
> +	return ret;
>  }
>  
>  static void __exit ccidrv_exit(void)
>  {
>  	pci_unregister_driver(&cci_pci_driver);
> +	fpga_ids_destroy();
>  }
>  
>  module_init(ccidrv_init);
> -- 
> 1.8.3.1
> 

Thanks,

Moritz

[1] https://www.kernel.org/doc/html/v4.11/doc-guide/kernel-doc.html#writing-kernel-doc-comments

Re: [PATCH v2 07/22] fpga: intel: pcie: parse feature list and create platform device for features.

[Wu Hao]

On Mon, Jun 26, 2017 at 11:42:05AM -0700, Moritz Fischer wrote:
> Hi Xiao / Wu,
> 
> general question, can't you make the /* */ comments
> for the structs kernel-doc ([1]) markup instead (see below)?


Hi Moritz,

Thanks for your review. Sure, will fix this issue for all patches.

Hao

> 
> 
> On Mon, Jun 26, 2017 at 09:52:03AM +0800, Wu Hao wrote:
> > From: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > 
> > Device Feature List structure creates a link list of feature headers
> > within the MMIO space to provide an extensible way of adding features.
> > 
> > The Intel FPGA PCIe driver walks through the feature headers to enumerate
> > feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
> > Function Unit (AFU), and their private sub features. For feature devices,
> > it creates the platform devices and linked the private sub features into
> > their platform data.
> > 
> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> > Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > ---
> > v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
> >     switched to GPLv2 license.
> >     fixed comments from Moritz Fischer.
> >     fixed kbuild warning, typos and clean up the code.
> > ---
> >  drivers/fpga/Makefile            |   2 +-
> >  drivers/fpga/intel-feature-dev.c | 130 ++++++
> >  drivers/fpga/intel-feature-dev.h | 341 ++++++++++++++++
> >  drivers/fpga/intel-pcie.c        | 841 ++++++++++++++++++++++++++++++++++++++-
> >  4 files changed, 1311 insertions(+), 3 deletions(-)
> >  create mode 100644 drivers/fpga/intel-feature-dev.c
> >  create mode 100644 drivers/fpga/intel-feature-dev.h
> > 
> > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> > index 5613133..ad24b3d 100644
> > --- a/drivers/fpga/Makefile
> > +++ b/drivers/fpga/Makefile
> > @@ -31,4 +31,4 @@ obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
> >  # Intel FPGA Support
> >  obj-$(CONFIG_INTEL_FPGA_PCI)		+= intel-fpga-pci.o
> >  
> > -intel-fpga-pci-objs := intel-pcie.o
> > +intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
> > diff --git a/drivers/fpga/intel-feature-dev.c b/drivers/fpga/intel-feature-dev.c
> > new file mode 100644
> > index 0000000..68f9cba
> > --- /dev/null
> > +++ b/drivers/fpga/intel-feature-dev.c
> > @@ -0,0 +1,130 @@
> > +/*
> > + * Intel FPGA Feature Device Driver
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <luwei.kang@intel.com>
> > + *   Zhang Yi <yi.z.zhang@intel.com>
> > + *   Wu Hao <hao.wu@intel.com>
> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > + *
> > + * This work is licensed under the terms of the GNU GPL version 2. See
> > + * the COPYING file in the top-level directory.
> > + */
> > +
> > +#include "intel-feature-dev.h"
> > +
> > +void feature_platform_data_add(struct feature_platform_data *pdata,
> > +			       int index, const char *name,
> > +			       int resource_index, void __iomem *ioaddr)
> > +{
> > +	WARN_ON(index >= pdata->num);
> > +
> > +	pdata->features[index].name = name;
> > +	pdata->features[index].resource_index = resource_index;
> > +	pdata->features[index].ioaddr = ioaddr;
> > +}
> > +
> > +struct feature_platform_data *
> > +feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
> > +{
> > +	struct feature_platform_data *pdata;
> > +
> > +	pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
> > +	if (pdata) {
> > +		pdata->dev = dev;
> > +		pdata->num = num;
> > +		mutex_init(&pdata->lock);
> > +	}
> > +
> > +	return pdata;
> > +}
> > +
> > +int fme_feature_num(void)
> > +{
> > +	return FME_FEATURE_ID_MAX;
> > +}
> > +
> > +int port_feature_num(void)
> > +{
> > +	return PORT_FEATURE_ID_MAX;
> > +}
> > +
> > +int fpga_port_id(struct platform_device *pdev)
> > +{
> > +	struct feature_port_header *port_hdr;
> > +	struct feature_port_capability capability;
> > +
> > +	port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> > +					       PORT_FEATURE_ID_HEADER);
> > +	WARN_ON(!port_hdr);
> > +
> > +	capability.csr = readq(&port_hdr->capability);
> > +	return capability.port_number;
> > +}
> > +EXPORT_SYMBOL_GPL(fpga_port_id);
> > +
> > +/*
> > + * Enable Port by clear the port soft reset bit, which is set by default.
> > + * The User AFU is unable to respond to any MMIO access while in reset.
> > + * __fpga_port_enable function should only be used after __fpga_port_disable
> > + * function.
> > + */
> > +void __fpga_port_enable(struct platform_device *pdev)
> > +{
> > +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +	struct feature_port_header *port_hdr;
> > +	struct feature_port_control control;
> > +
> > +	WARN_ON(!pdata->disable_count);
> > +
> > +	if (--pdata->disable_count != 0)
> > +		return;
> > +
> > +	port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> > +					       PORT_FEATURE_ID_HEADER);
> > +	WARN_ON(!port_hdr);
> > +
> > +	control.csr = readq(&port_hdr->control);
> > +	control.port_sftrst = 0x0;
> > +	writeq(control.csr, &port_hdr->control);
> > +}
> > +EXPORT_SYMBOL_GPL(__fpga_port_enable);
> > +
> > +#define RST_POLL_INVL 10 /* us */
> > +#define RST_POLL_TIMEOUT 1000 /* us */
> > +
> > +int __fpga_port_disable(struct platform_device *pdev)
> > +{
> > +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +	struct feature_port_header *port_hdr;
> > +	struct feature_port_control control;
> > +
> > +	if (pdata->disable_count++ != 0)
> > +		return 0;
> > +
> > +	port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> > +					       PORT_FEATURE_ID_HEADER);
> > +	WARN_ON(!port_hdr);
> > +
> > +	/* Set port soft reset */
> > +	control.csr = readq(&port_hdr->control);
> > +	control.port_sftrst = 0x1;
> > +	writeq(control.csr, &port_hdr->control);
> > +
> > +	/*
> > +	 * HW sets ack bit to 1 when all outstanding requests have been drained
> > +	 * on this port and minimum soft reset pulse width has elapsed.
> > +	 * Driver polls port_soft_reset_ack to determine if reset done by HW.
> > +	 */
> > +	if (readq_poll_timeout(&port_hdr->control, control.csr,
> > +			       (control.port_sftrst_ack == 1),
> > +			       RST_POLL_INVL, RST_POLL_TIMEOUT)) {
> > +		dev_err(&pdev->dev, "timeout, fail to reset device\n");
> > +		return -ETIMEDOUT;
> > +	}
> > +
> > +	return 0;
> > +}
> > +EXPORT_SYMBOL_GPL(__fpga_port_disable);
> > diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
> > new file mode 100644
> > index 0000000..f67784a
> > --- /dev/null
> > +++ b/drivers/fpga/intel-feature-dev.h
> > @@ -0,0 +1,341 @@
> > +/*
> > + * Intel FPGA Feature Device Driver Header File
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <luwei.kang@intel.com>
> > + *   Zhang Yi <yi.z.zhang@intel.com>
> > + *   Wu Hao <hao.wu@intel.com>
> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > + *
> > + * This work is licensed under the terms of the GNU GPL version 2. See
> > + * the COPYING file in the top-level directory.
> > + */
> > +
> > +#ifndef __INTEL_FPGA_FEATURE_H
> > +#define __INTEL_FPGA_FEATURE_H
> > +
> > +#include <linux/fs.h>
> > +#include <linux/pci.h>
> > +#include <linux/uuid.h>
> > +#include <linux/delay.h>
> > +#include <linux/iopoll.h>
> > +#include <linux/platform_device.h>
> > +
> > +#ifndef readq
> > +static inline u64 readq(void __iomem *addr)
> > +{
> > +	return readl(addr) + ((u64)readl(addr + 4) << 32);
> > +}
> > +#endif
> > +
> > +#ifndef writeq
> > +static inline void writeq(u64 val, void __iomem *addr)
> > +{
> > +	writel((u32) (val), addr);
> > +	writel((u32) (val >> 32), (addr + 4));
> > +}
> > +#endif
> > +
> > +/* maximum supported number of ports */
> > +#define MAX_FPGA_PORT_NUM 4
> > +/* plus one for fme device */
> > +#define MAX_FEATURE_DEV_NUM	(MAX_FPGA_PORT_NUM + 1)
> > +
> > +#define FME_FEATURE_HEADER          "fme_hdr"
> > +#define FME_FEATURE_THERMAL_MGMT    "fme_thermal"
> > +#define FME_FEATURE_POWER_MGMT      "fme_power"
> > +#define FME_FEATURE_GLOBAL_PERF     "fme_gperf"
> > +#define FME_FEATURE_GLOBAL_ERR      "fme_error"
> > +#define FME_FEATURE_PR_MGMT         "fme_pr"
> > +
> > +#define PORT_FEATURE_HEADER         "port_hdr"
> > +#define PORT_FEATURE_UAFU           "port_uafu"
> > +#define PORT_FEATURE_ERR            "port_err"
> > +#define PORT_FEATURE_UMSG           "port_umsg"
> > +#define PORT_FEATURE_PR             "port_pr"
> > +#define PORT_FEATURE_STP            "port_stp"
> > +
> > +/* All headers and structures must be byte-packed to match the spec. */
> > +#pragma pack(1)
> > +
> > +/* common header for all features */
> > +struct feature_header {
> > +	union {
> > +		u64 csr;
> > +		struct {
> > +			u64 id:12;
> > +			u64 revision:4;
> > +			u64 next_header_offset:24; /* offset to next header */
> > +			u64 rsvdz:20;
> > +			u64 type:4;		   /* feature type */
> > +#define FEATURE_TYPE_AFU		0x1
> > +#define FEATURE_TYPE_PRIVATE		0x3
> > +		};
> > +	};
> > +};
> 
> Here for example.
> > +
> > +/* common header for non-private features */
> > +struct feature_afu_header {
> > +	uuid_le guid;
> > +	union {
> > +		u64 csr;
> > +		struct {
> > +			u64 next_afu:24;	/* pointer to next afu header */
> > +			u64 rsvdz:40;
> > +		};
> > +	};
> > +};
> > +
> > +/* FME Header Register Set */
> > +/* FME Capability Register */
> > +struct feature_fme_capability {
> > +	union {
> > +		u64 csr;
> > +		struct {
> > +			u64 fabric_verid:8;	/* Fabric version ID */
> > +			u64 socket_id:1;	/* Socket id */
> > +			u64 rsvdz1:3;
> > +			u64 pcie0_link_avl:1;	/* PCIe0 link availability */
> > +			u64 pcie1_link_avl:1;	/* PCIe1 link availability */
> > +			u64 coherent_link_avl:1;/* Coherent link availability */
> > +			u64 rsvdz2:1;
> > +			u64 iommu_support:1;	/* IOMMU or VT-d supported */
> > +			u64 num_ports:3;	/* Num of ports implemented */
> > +			u64 rsvdz3:4;
> > +			u64 addr_width_bits:6;	/* Address width supported */
> > +			u64 rsvdz4:2;
> > +			u64 cache_size:12;	/* Cache size in kb */
> > +			u64 cache_assoc:4;	/* Cache Associativity */
> > +			u64 rsvdz5:15;
> > +			u64 lock_bit:1;		/* Latched lock bit by BIOS */
> > +		};
> > +	};
> > +};
> > +
> > +/* FME Port Offset Register */
> > +struct feature_fme_port {
> > +	union {
> > +		u64 csr;
> > +		struct {
> > +			u64 port_offset:24;	/* Offset to port header */
> > +			u64 rsvdz1:8;
> > +			u64 port_bar:3;		/* Bar id */
> > +			u64 rsvdz2:20;
> > +			u64 afu_access_ctrl:1;	/* AFU access type: PF/VF */
> > +			u64 rsvdz3:4;
> > +			u64 port_implemented:1;	/* Port implemented or not */
> > +			u64 rsvdz4:3;
> > +		};
> > +	};
> > +};
> > +
> > +struct feature_fme_header {
> > +	struct feature_header header;
> > +	struct feature_afu_header afu_header;
> > +	u64 rsvd[2];
> > +	struct feature_fme_capability capability;
> > +	struct feature_fme_port port[MAX_FPGA_PORT_NUM];
> > +};
> > +
> > +/* FME Thermal Sub Feature Register Set */
> > +struct feature_fme_thermal {
> > +	struct feature_header header;
> > +};
> > +
> > +/* FME Power Sub Feature Register Set */
> > +struct feature_fme_power {
> > +	struct feature_header header;
> > +};
> > +
> > +/* FME Global Performance Sub Feature Register Set */
> > +struct feature_fme_gperf {
> > +	struct feature_header header;
> > +};
> > +
> > +/* FME Error Sub Feature Register Set */
> > +struct feature_fme_err {
> > +	struct feature_header header;
> > +};
> > +
> > +/* FME Partial Reconfiguration Sub Feature Register Set */
> > +struct feature_fme_pr {
> > +	struct feature_header header;
> > +};
> > +
> > +/* PORT Header Register Set */
> > +/* Port Capability Register */
> > +struct feature_port_capability {
> > +	union {
> > +		u64 csr;
> > +		struct {
> > +			u64 port_number:2;	/* Port Number 0-3 */
> > +			u64 rsvdz1:6;
> > +			u64 mmio_size:16;	/* User MMIO size in KB */
> > +			u64 rsvdz2:8;
> > +			u64 sp_intr_num:4;	/* Supported interrupts num */
> > +			u64 rsvdz3:28;
> > +		};
> > +	};
> > +};
> > +
> > +/* Port Control Register */
> > +struct feature_port_control {
> > +	union {
> > +		u64 csr;
> > +		struct {
> > +			u64 port_sftrst:1;	/* Port Soft Reset */
> > +			u64 rsvdz1:1;
> > +			u64 latency_tolerance:1;/* '1' >= 40us, '0' < 40us */
> > +			u64 rsvdz2:1;
> > +			u64 port_sftrst_ack:1;	/* HW ACK for Soft Reset */
> > +			u64 rsvdz3:59;
> > +		};
> > +	};
> > +};
> > +
> > +struct feature_port_header {
> > +	struct feature_header header;
> > +	struct feature_afu_header afu_header;
> > +	u64 rsvd[2];
> > +	struct feature_port_capability capability;
> > +	struct feature_port_control control;
> > +};
> > +
> > +/* PORT Error Sub Feature Register Set */
> > +struct feature_port_error {
> > +	struct feature_header header;
> > +};
> > +
> > +/* PORT Unordered Message Sub Feature Register Set */
> > +struct feature_port_umsg {
> > +	struct feature_header header;
> > +};
> > +
> > +/* PORT SignalTap Sub Feature Register Set */
> > +struct feature_port_stp {
> > +	struct feature_header header;
> > +};
> > +
> > +#pragma pack()
> > +
> > +struct feature {
> > +	const char *name;
> > +	int resource_index;
> > +	void __iomem *ioaddr;
> > +};
> > +
> > +struct feature_platform_data {
> > +	/* list the feature dev to cci_drvdata->port_dev_list. */
> > +	struct list_head node;
> > +	struct mutex lock;
> > +	struct platform_device *dev;
> > +	unsigned int disable_count;	/* count for port disable */
> > +
> > +	int num;			/* number of features */
> > +	struct feature features[0];
> > +};
> > +
> > +enum fme_feature_id {
> > +	FME_FEATURE_ID_HEADER = 0x0,
> > +	FME_FEATURE_ID_THERMAL_MGMT = 0x1,
> > +	FME_FEATURE_ID_POWER_MGMT = 0x2,
> > +	FME_FEATURE_ID_GLOBAL_PERF = 0x3,
> > +	FME_FEATURE_ID_GLOBAL_ERR = 0x4,
> > +	FME_FEATURE_ID_PR_MGMT = 0x5,
> > +	FME_FEATURE_ID_MAX = 0x6,
> > +};
> > +
> > +enum port_feature_id {
> > +	PORT_FEATURE_ID_HEADER = 0x0,
> > +	PORT_FEATURE_ID_ERROR = 0x1,
> > +	PORT_FEATURE_ID_UMSG = 0x2,
> > +	PORT_FEATURE_ID_PR = 0x3,
> > +	PORT_FEATURE_ID_STP = 0x4,
> > +	PORT_FEATURE_ID_UAFU = 0x5,
> > +	PORT_FEATURE_ID_MAX = 0x6,
> > +};
> > +
> > +int fme_feature_num(void);
> > +int port_feature_num(void);
> > +
> > +#define FPGA_FEATURE_DEV_FME		"intel-fpga-fme"
> > +#define FPGA_FEATURE_DEV_PORT		"intel-fpga-port"
> > +
> > +void feature_platform_data_add(struct feature_platform_data *pdata,
> > +			       int index, const char *name,
> > +			       int resource_index, void __iomem *ioaddr);
> > +
> > +static inline int feature_platform_data_size(const int num)
> > +{
> > +	return sizeof(struct feature_platform_data) +
> > +		num * sizeof(struct feature);
> > +}
> > +
> > +struct feature_platform_data *
> > +feature_platform_data_alloc_and_init(struct platform_device *dev, int num);
> > +
> > +int fpga_port_id(struct platform_device *pdev);
> > +
> > +static inline int fpga_port_check_id(struct platform_device *pdev,
> > +				     void *pport_id)
> > +{
> > +	return fpga_port_id(pdev) == *(int *)pport_id;
> > +}
> > +
> > +void __fpga_port_enable(struct platform_device *pdev);
> > +int __fpga_port_disable(struct platform_device *pdev);
> > +
> > +static inline void fpga_port_enable(struct platform_device *pdev)
> > +{
> > +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +
> > +	mutex_lock(&pdata->lock);
> > +	__fpga_port_enable(pdev);
> > +	mutex_unlock(&pdata->lock);
> > +}
> > +
> > +static inline int fpga_port_disable(struct platform_device *pdev)
> > +{
> > +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +	int ret;
> > +
> > +	mutex_lock(&pdata->lock);
> > +	ret = __fpga_port_disable(pdev);
> > +	mutex_unlock(&pdata->lock);
> > +
> > +	return ret;
> > +}
> > +
> > +static inline int __fpga_port_reset(struct platform_device *pdev)
> > +{
> > +	int ret;
> > +
> > +	ret = __fpga_port_disable(pdev);
> > +	if (ret)
> > +		return ret;
> > +
> > +	__fpga_port_enable(pdev);
> > +	return 0;
> > +}
> > +
> > +static inline int fpga_port_reset(struct platform_device *pdev)
> > +{
> > +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +	int ret;
> > +
> > +	mutex_lock(&pdata->lock);
> > +	ret = __fpga_port_reset(pdev);
> > +	mutex_unlock(&pdata->lock);
> > +	return ret;
> > +}
> > +
> > +static inline void __iomem *
> > +get_feature_ioaddr_by_index(struct device *dev, int index)
> > +{
> > +	struct feature_platform_data *pdata = dev_get_platdata(dev);
> > +
> > +	return pdata->features[index].ioaddr;
> > +}
> > +#endif
> > diff --git a/drivers/fpga/intel-pcie.c b/drivers/fpga/intel-pcie.c
> > index f697de4..70b8284 100644
> > --- a/drivers/fpga/intel-pcie.c
> > +++ b/drivers/fpga/intel-pcie.c
> > @@ -23,10 +23,827 @@
> >  #include <linux/stddef.h>
> >  #include <linux/errno.h>
> >  #include <linux/aer.h>
> > +#include <linux/fpga/fpga-dev.h>
> > +
> > +#include "intel-feature-dev.h"
> >  
> >  #define DRV_VERSION	"0.8"
> >  #define DRV_NAME	"intel-fpga-pci"
> >  
> > +#define INTEL_FPGA_DEV	"intel-fpga-dev"
> > +
> > +static DEFINE_MUTEX(fpga_id_mutex);
> > +
> > +enum fpga_id_type {
> > +	FME_ID,		/* fme id allocation and mapping */
> > +	PORT_ID,	/* port id allocation and mapping */
> > +	FPGA_ID_MAX,
> > +};
> > +
> > +/* it is protected by fpga_id_mutex */
> > +static struct idr fpga_ids[FPGA_ID_MAX];
> > +
> > +struct cci_drvdata {
> > +	struct device *fme_dev;
> > +
> > +	struct mutex lock;
> > +	struct list_head port_dev_list;
> > +
> > +	struct list_head regions; /* global list of pci bar mapping region */
> > +};
> > +
> > +/* pci bar mapping info */
> > +struct cci_pci_region {
> > +	int bar;
> > +	void __iomem *ioaddr;	/* pointer to mapped bar region */
> > +	struct list_head node;
> > +};
> > +
> > +static void fpga_ids_init(void)
> > +{
> > +	int i;
> > +
> > +	for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> > +		idr_init(fpga_ids + i);
> > +}
> > +
> > +static void fpga_ids_destroy(void)
> > +{
> > +	int i;
> > +
> > +	for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> > +		idr_destroy(fpga_ids + i);
> > +}
> > +
> > +static int alloc_fpga_id(enum fpga_id_type type, struct device *dev)
> > +{
> > +	int id;
> > +
> > +	WARN_ON(type >= FPGA_ID_MAX);
> > +	mutex_lock(&fpga_id_mutex);
> > +	id = idr_alloc(fpga_ids + type, dev, 0, 0, GFP_KERNEL);
> > +	mutex_unlock(&fpga_id_mutex);
> > +	return id;
> > +}
> > +
> > +static void free_fpga_id(enum fpga_id_type type, int id)
> > +{
> > +	WARN_ON(type >= FPGA_ID_MAX);
> > +	mutex_lock(&fpga_id_mutex);
> > +	idr_remove(fpga_ids + type, id);
> > +	mutex_unlock(&fpga_id_mutex);
> > +}
> > +
> > +static void cci_pci_add_port_dev(struct pci_dev *pdev,
> > +				 struct platform_device *port_dev)
> > +{
> > +	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> > +	struct feature_platform_data *pdata = dev_get_platdata(&port_dev->dev);
> > +
> > +	mutex_lock(&drvdata->lock);
> > +	list_add(&pdata->node, &drvdata->port_dev_list);
> > +	get_device(&pdata->dev->dev);
> > +	mutex_unlock(&drvdata->lock);
> > +}
> > +
> > +static void cci_pci_remove_port_devs(struct pci_dev *pdev)
> > +{
> > +	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> > +	struct feature_platform_data *pdata, *ptmp;
> > +
> > +	mutex_lock(&drvdata->lock);
> > +	list_for_each_entry_safe(pdata, ptmp, &drvdata->port_dev_list, node) {
> > +		struct platform_device *port_dev = pdata->dev;
> > +
> > +		/* the port should be unregistered first. */
> > +		WARN_ON(device_is_registered(&port_dev->dev));
> > +		list_del(&pdata->node);
> > +		free_fpga_id(PORT_ID, port_dev->id);
> > +		put_device(&port_dev->dev);
> > +	}
> > +	mutex_unlock(&drvdata->lock);
> > +}
> > +
> > +/* info collection during feature dev build. */
> > +struct build_feature_devs_info {
> > +	struct pci_dev *pdev;
> > +
> > +	/*
> > +	 * PCI BAR mapping info. Parsing feature list starts from
> > +	 * BAR 0 and switch to different BARs to parse Port
> > +	 */
> > +	void __iomem *ioaddr;
> > +	void __iomem *ioend;
> > +	int current_bar;
> > +
> > +	/* points to FME header where the port offset is figured out. */
> > +	void __iomem *pfme_hdr;
> > +
> > +	/* the container device for all feature devices */
> > +	struct fpga_dev *parent_dev;
> > +
> > +	/* current feature device */
> > +	struct platform_device *feature_dev;
> > +};
> > +
> > +static void cci_pci_release_regions(struct pci_dev *pdev)
> > +{
> > +	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> > +	struct cci_pci_region *tmp, *region;
> > +
> > +	list_for_each_entry_safe(region, tmp, &drvdata->regions, node) {
> > +		list_del(&region->node);
> > +		if (region->ioaddr)
> > +			pci_iounmap(pdev, region->ioaddr);
> > +		devm_kfree(&pdev->dev, region);
> > +	}
> > +}
> > +
> > +static void __iomem *cci_pci_ioremap_bar(struct pci_dev *pdev, int bar)
> > +{
> > +	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> > +	struct cci_pci_region *region;
> > +
> > +	list_for_each_entry(region, &drvdata->regions, node)
> > +		if (region->bar == bar) {
> > +			dev_dbg(&pdev->dev, "BAR %d region exists\n", bar);
> > +			return region->ioaddr;
> > +		}
> > +
> > +	region = devm_kzalloc(&pdev->dev, sizeof(*region), GFP_KERNEL);
> > +	if (!region)
> > +		return NULL;
> > +
> > +	region->bar = bar;
> > +	region->ioaddr = pci_ioremap_bar(pdev, bar);
> > +	if (!region->ioaddr) {
> > +		dev_err(&pdev->dev, "can't ioremap memory from BAR %d.\n", bar);
> > +		devm_kfree(&pdev->dev, region);
> > +		return NULL;
> > +	}
> > +
> > +	list_add(&region->node, &drvdata->regions);
> > +	return region->ioaddr;
> > +}
> > +
> > +static int parse_start_from(struct build_feature_devs_info *binfo, int bar)
> > +{
> > +	binfo->ioaddr = cci_pci_ioremap_bar(binfo->pdev, bar);
> > +	if (!binfo->ioaddr)
> > +		return -ENOMEM;
> > +
> > +	binfo->current_bar = bar;
> > +	binfo->ioend = binfo->ioaddr + pci_resource_len(binfo->pdev, bar);
> > +	return 0;
> > +}
> > +
> > +static int parse_start(struct build_feature_devs_info *binfo)
> > +{
> > +	/* fpga feature list starts from BAR 0 */
> > +	return parse_start_from(binfo, 0);
> > +}
> > +
> > +/* switch the memory mapping to BAR# @bar */
> > +static int parse_switch_to(struct build_feature_devs_info *binfo, int bar)
> > +{
> > +	return parse_start_from(binfo, bar);
> > +}
> > +
> > +static struct build_feature_devs_info *
> > +build_info_alloc_and_init(struct pci_dev *pdev)
> > +{
> > +	struct build_feature_devs_info *binfo;
> > +
> > +	binfo = devm_kzalloc(&pdev->dev, sizeof(*binfo), GFP_KERNEL);
> > +	if (binfo)
> > +		binfo->pdev = pdev;
> > +
> > +	return binfo;
> > +}
> > +
> > +static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
> > +{
> > +	if (!strcmp(pdev->name, FPGA_FEATURE_DEV_FME))
> > +		return FME_ID;
> > +
> > +	if (!strcmp(pdev->name, FPGA_FEATURE_DEV_PORT))
> > +		return PORT_ID;
> > +
> > +	WARN_ON(1);
> > +	return FPGA_ID_MAX;
> > +}
> > +
> > +/*
> > + * register current feature device, it is called when we need to switch to
> > + * another feature parsing or we have parsed all features
> > + */
> > +static int build_info_commit_dev(struct build_feature_devs_info *binfo)
> > +{
> > +	int ret;
> > +
> > +	if (!binfo->feature_dev)
> > +		return 0;
> > +
> > +	ret = platform_device_add(binfo->feature_dev);
> > +	if (!ret) {
> > +		struct cci_drvdata *drvdata;
> > +
> > +		drvdata = dev_get_drvdata(&binfo->pdev->dev);
> > +		if (feature_dev_id_type(binfo->feature_dev) == PORT_ID)
> > +			cci_pci_add_port_dev(binfo->pdev, binfo->feature_dev);
> > +		else
> > +			drvdata->fme_dev = get_device(&binfo->feature_dev->dev);
> > +
> > +		/*
> > +		 * reset it to avoid build_info_free() freeing their resource.
> > +		 *
> > +		 * The resource of successfully registered feature devices
> > +		 * will be freed by platform_device_unregister(). See the
> > +		 * comments in build_info_create_dev().
> > +		 */
> > +		binfo->feature_dev = NULL;
> > +	}
> > +
> > +	return ret;
> > +}
> > +
> > +static int
> > +build_info_create_dev(struct build_feature_devs_info *binfo,
> > +		      enum fpga_id_type type, int feature_nr, const char *name)
> > +{
> > +	struct platform_device *fdev;
> > +	struct resource *res;
> > +	struct feature_platform_data *pdata;
> > +	int ret;
> > +
> > +	/* we will create a new device, commit current device first */
> > +	ret = build_info_commit_dev(binfo);
> > +	if (ret)
> > +		return ret;
> > +
> > +	/*
> > +	 * we use -ENODEV as the initialization indicator which indicates
> > +	 * whether the id need to be reclaimed
> > +	 */
> > +	fdev = binfo->feature_dev = platform_device_alloc(name, -ENODEV);
> > +	if (!fdev)
> > +		return -ENOMEM;
> > +
> > +	fdev->id = alloc_fpga_id(type, &fdev->dev);
> > +	if (fdev->id < 0)
> > +		return fdev->id;
> > +
> > +	fdev->dev.parent = &binfo->parent_dev->dev;
> > +
> > +	/*
> > +	 * we do not need to care for the memory which is associated with
> > +	 * the platform device. After calling platform_device_unregister(),
> > +	 * it will be automatically freed by device's release() callback,
> > +	 * platform_device_release().
> > +	 */
> > +	pdata = feature_platform_data_alloc_and_init(fdev, feature_nr);
> > +	if (!pdata)
> > +		return -ENOMEM;
> > +
> > +	/*
> > +	 * the count should be initialized to 0 to make sure
> > +	 *__fpga_port_enable() following __fpga_port_disable()
> > +	 * works properly for port device.
> > +	 * and it should always be 0 for fme device.
> > +	 */
> > +	WARN_ON(pdata->disable_count);
> > +
> > +	fdev->dev.platform_data = pdata;
> > +	fdev->num_resources = feature_nr;
> > +	fdev->resource = kcalloc(feature_nr, sizeof(*res), GFP_KERNEL);
> > +	if (!fdev->resource)
> > +		return -ENOMEM;
> > +
> > +	return 0;
> > +}
> > +
> > +static int remove_feature_dev(struct device *dev, void *data)
> > +{
> > +	struct platform_device *pdev = to_platform_device(dev);
> > +
> > +	platform_device_unregister(pdev);
> > +	return 0;
> > +}
> > +
> > +static int remove_parent_dev(struct device *dev, void *data)
> > +{
> > +	/* remove platform devices attached in the parent device */
> > +	device_for_each_child(dev, NULL, remove_feature_dev);
> > +	fpga_dev_destroy(to_fpga_dev(dev));
> > +	return 0;
> > +}
> > +
> > +static void remove_all_devs(struct pci_dev *pdev)
> > +{
> > +	/* remove parent device and all its children. */
> > +	device_for_each_child(&pdev->dev, NULL, remove_parent_dev);
> > +}
> > +
> > +static void build_info_free(struct build_feature_devs_info *binfo)
> > +{
> > +	if (!IS_ERR_OR_NULL(binfo->parent_dev))
> > +		remove_all_devs(binfo->pdev);
> > +
> > +	/*
> > +	 * it is a valid id, free it. See comments in
> > +	 * build_info_create_dev()
> > +	 */
> > +	if (binfo->feature_dev && binfo->feature_dev->id >= 0)
> > +		free_fpga_id(feature_dev_id_type(binfo->feature_dev),
> > +			     binfo->feature_dev->id);
> > +
> > +	platform_device_put(binfo->feature_dev);
> > +
> > +	devm_kfree(&binfo->pdev->dev, binfo);
> > +}
> > +
> > +#define FEATURE_TYPE_AFU	0x1
> > +#define FEATURE_TYPE_PRIVATE	0x3
> > +
> > +/* FME and PORT GUID are fixed */
> > +#define FEATURE_FME_GUID "f9e17764-38f0-82fe-e346-524ae92aafbf"
> > +#define FEATURE_PORT_GUID "6b355b87-b06c-9642-eb42-8d139398b43a"
> > +
> > +static bool feature_is_fme(struct feature_afu_header *afu_hdr)
> > +{
> > +	uuid_le u;
> > +
> > +	uuid_le_to_bin(FEATURE_FME_GUID, &u);
> > +
> > +	return !uuid_le_cmp(u, afu_hdr->guid);
> > +}
> > +
> > +static bool feature_is_port(struct feature_afu_header *afu_hdr)
> > +{
> > +	uuid_le u;
> > +
> > +	uuid_le_to_bin(FEATURE_PORT_GUID, &u);
> > +
> > +	return !uuid_le_cmp(u, afu_hdr->guid);
> > +}
> > +
> > +/*
> > + * UAFU GUID is dynamic as it can be changed after FME downloads different
> > + * Green Bitstream to the port, so we treat the unknown GUIDs which are
> > + * attached on port's feature list as UAFU.
> > + */
> > +static bool feature_is_UAFU(struct build_feature_devs_info *binfo)
> > +{
> > +	if (!binfo->feature_dev ||
> > +	      feature_dev_id_type(binfo->feature_dev) != PORT_ID)
> > +		return false;
> > +
> > +	return true;
> > +}
> > +
> > +static void
> > +build_info_add_sub_feature(struct build_feature_devs_info *binfo,
> > +			   int feature_id, const char *feature_name,
> > +			   resource_size_t resource_size, void __iomem *start)
> > +{
> > +
> > +	struct platform_device *fdev = binfo->feature_dev;
> > +	struct feature_platform_data *pdata = dev_get_platdata(&fdev->dev);
> > +	struct resource *res = &fdev->resource[feature_id];
> > +
> > +	res->start = pci_resource_start(binfo->pdev, binfo->current_bar) +
> > +		start - binfo->ioaddr;
> > +	res->end = res->start + resource_size - 1;
> > +	res->flags = IORESOURCE_MEM;
> > +	res->name = feature_name;
> > +
> > +	feature_platform_data_add(pdata, feature_id,
> > +				  feature_name, feature_id, start);
> > +}
> > +
> > +struct feature_info {
> > +	const char *name;
> > +	resource_size_t resource_size;
> > +	int feature_index;
> > +};
> > +
> > +/* indexed by fme feature IDs which are defined in 'enum fme_feature_id'. */
> > +static struct feature_info fme_features[] = {
> > +	{
> > +		.name = FME_FEATURE_HEADER,
> > +		.resource_size = sizeof(struct feature_fme_header),
> > +		.feature_index = FME_FEATURE_ID_HEADER,
> > +	},
> > +	{
> > +		.name = FME_FEATURE_THERMAL_MGMT,
> > +		.resource_size = sizeof(struct feature_fme_thermal),
> > +		.feature_index = FME_FEATURE_ID_THERMAL_MGMT,
> > +	},
> > +	{
> > +		.name = FME_FEATURE_POWER_MGMT,
> > +		.resource_size = sizeof(struct feature_fme_power),
> > +		.feature_index = FME_FEATURE_ID_POWER_MGMT,
> > +	},
> > +	{
> > +		.name = FME_FEATURE_GLOBAL_PERF,
> > +		.resource_size = sizeof(struct feature_fme_gperf),
> > +		.feature_index = FME_FEATURE_ID_GLOBAL_PERF,
> > +	},
> > +	{
> > +		.name = FME_FEATURE_GLOBAL_ERR,
> > +		.resource_size = sizeof(struct feature_fme_err),
> > +		.feature_index = FME_FEATURE_ID_GLOBAL_ERR,
> > +	},
> > +	{
> > +		.name = FME_FEATURE_PR_MGMT,
> > +		.resource_size = sizeof(struct feature_fme_pr),
> > +		.feature_index = FME_FEATURE_ID_PR_MGMT,
> > +	}
> > +};
> > +
> > +/* indexed by port feature IDs which are defined in 'enum port_feature_id'. */
> > +static struct feature_info port_features[] = {
> > +	{
> > +		.name = PORT_FEATURE_HEADER,
> > +		.resource_size = sizeof(struct feature_port_header),
> > +		.feature_index = PORT_FEATURE_ID_HEADER,
> > +	},
> > +	{
> > +		.name = PORT_FEATURE_ERR,
> > +		.resource_size = sizeof(struct feature_port_error),
> > +		.feature_index = PORT_FEATURE_ID_ERROR,
> > +	},
> > +	{
> > +		.name = PORT_FEATURE_UMSG,
> > +		.resource_size = sizeof(struct feature_port_umsg),
> > +		.feature_index = PORT_FEATURE_ID_UMSG,
> > +	},
> > +	{
> > +		/* This feature isn't available for now */
> > +		.name = PORT_FEATURE_PR,
> > +		.resource_size = 0,
> > +		.feature_index = PORT_FEATURE_ID_PR,
> > +	},
> > +	{
> > +		.name = PORT_FEATURE_STP,
> > +		.resource_size = sizeof(struct feature_port_stp),
> > +		.feature_index = PORT_FEATURE_ID_STP,
> > +	},
> > +	{
> > +		/*
> > +		 * For User AFU feature, its region size is not fixed, but
> > +		 * reported by register PortCapability.mmio_size. Resource
> > +		 * size of UAFU will be set while parse port device.
> > +		 */
> > +		.name = PORT_FEATURE_UAFU,
> > +		.resource_size = 0,
> > +		.feature_index = PORT_FEATURE_ID_UAFU,
> > +	},
> > +};
> > +
> > +static int
> > +create_feature_instance(struct build_feature_devs_info *binfo,
> > +			void __iomem *start, struct feature_info *finfo)
> > +{
> > +	if (binfo->ioend - start < finfo->resource_size)
> > +		return -EINVAL;
> > +
> > +	build_info_add_sub_feature(binfo, finfo->feature_index, finfo->name,
> > +				   finfo->resource_size, start);
> > +	return 0;
> > +}
> > +
> > +static int parse_feature_fme(struct build_feature_devs_info *binfo,
> > +			     void __iomem *start)
> > +{
> > +	struct cci_drvdata *drvdata = dev_get_drvdata(&binfo->pdev->dev);
> > +	int ret;
> > +
> > +	ret = build_info_create_dev(binfo, FME_ID, fme_feature_num(),
> > +					FPGA_FEATURE_DEV_FME);
> > +	if (ret)
> > +		return ret;
> > +
> > +	if (drvdata->fme_dev) {
> > +		dev_err(&binfo->pdev->dev, "Multiple FMEs are detected.\n");
> > +		return -EINVAL;
> > +	}
> > +
> > +	return create_feature_instance(binfo, start,
> > +				       &fme_features[FME_FEATURE_ID_HEADER]);
> > +}
> > +
> > +static int parse_feature_fme_private(struct build_feature_devs_info *binfo,
> > +				     struct feature_header *hdr)
> > +{
> > +	struct feature_header header;
> > +
> > +	header.csr = readq(hdr);
> > +
> > +	if (header.id >= ARRAY_SIZE(fme_features)) {
> > +		dev_info(&binfo->pdev->dev, "FME feature id %x is not supported yet.\n",
> > +			 header.id);
> > +		return 0;
> > +	}
> > +
> > +	return create_feature_instance(binfo, hdr, &fme_features[header.id]);
> > +}
> > +
> > +static int parse_feature_port(struct build_feature_devs_info *binfo,
> > +			     void __iomem *start)
> > +{
> > +	int ret;
> > +
> > +	ret = build_info_create_dev(binfo, PORT_ID, port_feature_num(),
> > +					FPGA_FEATURE_DEV_PORT);
> > +	if (ret)
> > +		return ret;
> > +
> > +	return create_feature_instance(binfo, start,
> > +				       &port_features[PORT_FEATURE_ID_HEADER]);
> > +}
> > +
> > +static void enable_port_uafu(struct build_feature_devs_info *binfo,
> > +			     void __iomem *start)
> > +{
> > +	enum port_feature_id id = PORT_FEATURE_ID_UAFU;
> > +	struct feature_port_header *port_hdr;
> > +	struct feature_port_capability capability;
> > +
> > +	port_hdr = (struct feature_port_header *)start;
> > +	capability.csr = readq(&port_hdr->capability);
> > +	port_features[id].resource_size = capability.mmio_size << 10;
> > +
> > +	/*
> > +	 * To enable User AFU, driver needs to clear reset bit on related port,
> > +	 * otherwise the mmio space of this user AFU will be invalid.
> > +	 */
> > +	if (port_features[id].resource_size)
> > +		fpga_port_reset(binfo->feature_dev);
> > +}
> > +
> > +static int parse_feature_port_private(struct build_feature_devs_info *binfo,
> > +				      struct feature_header *hdr)
> > +{
> > +	struct feature_header header;
> > +	enum port_feature_id id;
> > +
> > +	header.csr = readq(hdr);
> > +	/*
> > +	 * the region of port feature id is [0x10, 0x13], + 1 to reserve 0
> > +	 * which is dedicated for port-hdr.
> > +	 */
> > +	id = (header.id & 0x000f) + 1;
> > +
> > +	if (id >= ARRAY_SIZE(port_features)) {
> > +		dev_info(&binfo->pdev->dev, "Port feature id %x is not supported yet.\n",
> > +			 header.id);
> > +		return 0;
> > +	}
> > +
> > +	return create_feature_instance(binfo, hdr, &port_features[id]);
> > +}
> > +
> > +static int parse_feature_port_uafu(struct build_feature_devs_info *binfo,
> > +				 struct feature_header *hdr)
> > +{
> > +	enum port_feature_id id = PORT_FEATURE_ID_UAFU;
> > +	int ret;
> > +
> > +	if (port_features[id].resource_size) {
> > +		ret = create_feature_instance(binfo, hdr, &port_features[id]);
> > +		port_features[id].resource_size = 0;
> > +	} else {
> > +		dev_err(&binfo->pdev->dev, "the uafu feature header is mis-configured.\n");
> > +		ret = -EINVAL;
> > +	}
> > +
> > +	return ret;
> > +}
> > +
> > +static int parse_feature_afus(struct build_feature_devs_info *binfo,
> > +			      struct feature_header *hdr)
> > +{
> > +	int ret;
> > +	struct feature_afu_header *afu_hdr, header;
> > +	void __iomem *start;
> > +	void __iomem *end = binfo->ioend;
> > +
> > +	start = hdr;
> > +	for (; start < end; start += header.next_afu) {
> > +		if (end - start < (sizeof(*afu_hdr) + sizeof(*hdr)))
> > +			return -EINVAL;
> > +
> > +		hdr = start;
> > +		afu_hdr = (struct feature_afu_header *) (hdr + 1);
> > +		header.csr = readq(&afu_hdr->csr);
> > +
> > +		if (feature_is_fme(afu_hdr)) {
> > +			ret = parse_feature_fme(binfo, hdr);
> > +			binfo->pfme_hdr = hdr;
> > +			if (ret)
> > +				return ret;
> > +		} else if (feature_is_port(afu_hdr)) {
> > +			ret = parse_feature_port(binfo, hdr);
> > +			enable_port_uafu(binfo, hdr);
> > +			if (ret)
> > +				return ret;
> > +		} else if (feature_is_UAFU(binfo)) {
> > +			ret = parse_feature_port_uafu(binfo, hdr);
> > +			if (ret)
> > +				return ret;
> > +		} else
> > +			dev_info(&binfo->pdev->dev, "AFU GUID %pUl is not supported yet.\n",
> > +				 afu_hdr->guid.b);
> > +
> > +		if (!header.next_afu)
> > +			break;
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static int parse_feature_private(struct build_feature_devs_info *binfo,
> > +				 struct feature_header *hdr)
> > +{
> > +	struct feature_header header;
> > +
> > +	header.csr = readq(hdr);
> > +
> > +	if (!binfo->feature_dev) {
> > +		dev_err(&binfo->pdev->dev, "the private feature %x does not belong to any AFU.\n",
> > +			header.id);
> > +		return -EINVAL;
> > +	}
> > +
> > +	switch (feature_dev_id_type(binfo->feature_dev)) {
> > +	case FME_ID:
> > +		return parse_feature_fme_private(binfo, hdr);
> > +	case PORT_ID:
> > +		return parse_feature_port_private(binfo, hdr);
> > +	default:
> > +		dev_info(&binfo->pdev->dev, "private feature %x belonging to AFU %s is not supported yet.\n",
> > +			 header.id, binfo->feature_dev->name);
> > +	}
> > +	return 0;
> > +}
> > +
> > +static int parse_feature(struct build_feature_devs_info *binfo,
> > +			 struct feature_header *hdr)
> > +{
> > +	struct feature_header header;
> > +	int ret = 0;
> > +
> > +	header.csr = readq(hdr);
> > +
> > +	switch (header.type) {
> > +	case FEATURE_TYPE_AFU:
> > +		ret = parse_feature_afus(binfo, hdr);
> > +		break;
> > +	case FEATURE_TYPE_PRIVATE:
> > +		ret = parse_feature_private(binfo, hdr);
> > +		break;
> > +	default:
> > +		dev_info(&binfo->pdev->dev,
> > +			 "Feature Type %x is not supported.\n", hdr->type);
> > +	};
> > +
> > +	return ret;
> > +}
> > +
> > +static int
> > +parse_feature_list(struct build_feature_devs_info *binfo, void __iomem *start)
> > +{
> > +	struct feature_header *hdr, header;
> > +	void __iomem *end = binfo->ioend;
> > +	int ret = 0;
> > +
> > +	for (; start < end; start += header.next_header_offset) {
> > +		if (end - start < sizeof(*hdr)) {
> > +			dev_err(&binfo->pdev->dev, "The region is too small to contain a feature.\n");
> > +			ret =  -EINVAL;
> > +			break;
> > +		}
> > +
> > +		hdr = (struct feature_header *)start;
> > +		ret = parse_feature(binfo, hdr);
> > +		if (ret)
> > +			break;
> > +
> > +		header.csr = readq(hdr);
> > +		if (!header.next_header_offset)
> > +			break;
> > +	}
> > +
> > +	return ret;
> > +}
> > +
> > +static int parse_ports_from_fme(struct build_feature_devs_info *binfo)
> > +{
> > +	struct feature_fme_header *fme_hdr;
> > +	struct feature_fme_port port;
> > +	int i = 0, ret = 0;
> > +
> > +	if (binfo->pfme_hdr == NULL) {
> > +		dev_dbg(&binfo->pdev->dev, "VF is detected.\n");
> > +		return ret;
> > +	}
> > +
> > +	fme_hdr = binfo->pfme_hdr;
> > +
> > +	do {
> > +		port.csr = readq(&fme_hdr->port[i]);
> > +		if (!port.port_implemented)
> > +			break;
> > +
> > +		ret = parse_switch_to(binfo, port.port_bar);
> > +		if (ret)
> > +			break;
> > +
> > +		ret = parse_feature_list(binfo,
> > +				binfo->ioaddr + port.port_offset);
> > +		if (ret)
> > +			break;
> > +	} while (++i < MAX_FPGA_PORT_NUM);
> > +
> > +	return ret;
> > +}
> > +
> > +static int create_init_drvdata(struct pci_dev *pdev)
> > +{
> > +	struct cci_drvdata *drvdata;
> > +
> > +	drvdata = devm_kzalloc(&pdev->dev, sizeof(*drvdata), GFP_KERNEL);
> > +	if (!drvdata)
> > +		return -ENOMEM;
> > +
> > +	mutex_init(&drvdata->lock);
> > +	INIT_LIST_HEAD(&drvdata->port_dev_list);
> > +	INIT_LIST_HEAD(&drvdata->regions);
> > +
> > +	dev_set_drvdata(&pdev->dev, drvdata);
> > +	return 0;
> > +}
> > +
> > +static void destroy_drvdata(struct pci_dev *pdev)
> > +{
> > +	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> > +
> > +	if (drvdata->fme_dev) {
> > +		/* fme device should be unregistered first. */
> > +		WARN_ON(device_is_registered(drvdata->fme_dev));
> > +		free_fpga_id(FME_ID, to_platform_device(drvdata->fme_dev)->id);
> > +		put_device(drvdata->fme_dev);
> > +	}
> > +
> > +	cci_pci_remove_port_devs(pdev);
> > +	cci_pci_release_regions(pdev);
> > +	dev_set_drvdata(&pdev->dev, NULL);
> > +	devm_kfree(&pdev->dev, drvdata);
> > +}
> > +
> > +static int cci_pci_create_feature_devs(struct pci_dev *pdev)
> > +{
> > +	struct build_feature_devs_info *binfo;
> > +	int ret;
> > +
> > +	binfo = build_info_alloc_and_init(pdev);
> > +	if (!binfo)
> > +		return -ENOMEM;
> > +
> > +	binfo->parent_dev = fpga_dev_create(&pdev->dev, INTEL_FPGA_DEV);
> > +	if (IS_ERR(binfo->parent_dev)) {
> > +		ret = PTR_ERR(binfo->parent_dev);
> > +		goto free_binfo_exit;
> > +	}
> > +
> > +	ret = parse_start(binfo);
> > +	if (ret)
> > +		goto free_binfo_exit;
> > +
> > +	ret = parse_feature_list(binfo, binfo->ioaddr);
> > +	if (ret)
> > +		goto free_binfo_exit;
> > +
> > +	ret = parse_ports_from_fme(binfo);
> > +	if (ret)
> > +		goto free_binfo_exit;
> > +
> > +	ret = build_info_commit_dev(binfo);
> > +	if (ret)
> > +		goto free_binfo_exit;
> > +
> > +	/*
> > +	 * everything is okay, reset ->parent_dev to stop it being
> > +	 * freed by build_info_free()
> > +	 */
> > +	binfo->parent_dev = NULL;
> > +
> > +free_binfo_exit:
> > +	build_info_free(binfo);
> > +	return ret;
> > +}
> > +
> >  /* PCI Device ID */
> >  #define PCIe_DEVICE_ID_PF_INT_5_X	0xBCBD
> >  #define PCIe_DEVICE_ID_PF_INT_6_X	0xBCC0
> > @@ -81,9 +898,18 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
> >  		goto release_region_exit;
> >  	}
> >  
> > -	/* TODO: create and add the platform device per feature list */
> > +	ret = create_init_drvdata(pcidev);
> > +	if (ret)
> > +		goto release_region_exit;
> > +
> > +	ret = cci_pci_create_feature_devs(pcidev);
> > +	if (ret)
> > +		goto destroy_drvdata_exit;
> > +
> >  	return 0;
> >  
> > +destroy_drvdata_exit:
> > +	destroy_drvdata(pcidev);
> >  release_region_exit:
> >  	pci_release_regions(pcidev);
> >  disable_error_report_exit:
> > @@ -94,6 +920,8 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
> >  
> >  static void cci_pci_remove(struct pci_dev *pcidev)
> >  {
> > +	remove_all_devs(pcidev);
> > +	destroy_drvdata(pcidev);
> >  	pci_release_regions(pcidev);
> >  	pci_disable_pcie_error_reporting(pcidev);
> >  	pci_disable_device(pcidev);
> > @@ -108,14 +936,23 @@ static void cci_pci_remove(struct pci_dev *pcidev)
> >  
> >  static int __init ccidrv_init(void)
> >  {
> > +	int ret;
> > +
> >  	pr_info("Intel(R) FPGA PCIe Driver: Version %s\n", DRV_VERSION);
> >  
> > -	return pci_register_driver(&cci_pci_driver);
> > +	fpga_ids_init();
> > +
> > +	ret = pci_register_driver(&cci_pci_driver);
> > +	if (ret)
> > +		fpga_ids_destroy();
> > +
> > +	return ret;
> >  }
> >  
> >  static void __exit ccidrv_exit(void)
> >  {
> >  	pci_unregister_driver(&cci_pci_driver);
> > +	fpga_ids_destroy();
> >  }
> >  
> >  module_init(ccidrv_init);
> > -- 
> > 1.8.3.1
> > 
> 
> Thanks,
> 
> Moritz
> 
> [1] https://www.kernel.org/doc/html/v4.11/doc-guide/kernel-doc.html#writing-kernel-doc-comments
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 07/22] fpga: intel: pcie: parse feature list and create platform device for features.

[Alan Tull]

On Mon, Jun 26, 2017 at 1:42 PM, Moritz Fischer <mdf-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:

Hi Hao,

I'll be out until July 5.  I'll be able to start looking at this v2 after that.

Alan

> Hi Xiao / Wu,
>
> general question, can't you make the /* */ comments
> for the structs kernel-doc ([1]) markup instead (see below)?
>
>
> On Mon, Jun 26, 2017 at 09:52:03AM +0800, Wu Hao wrote:
>> From: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
>>
>> Device Feature List structure creates a link list of feature headers
>> within the MMIO space to provide an extensible way of adding features.
>>
>> The Intel FPGA PCIe driver walks through the feature headers to enumerate
>> feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
>> Function Unit (AFU), and their private sub features. For feature devices,
>> it creates the platform devices and linked the private sub features into
>> their platform data.
>>
>> Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> Signed-off-by: Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> Signed-off-by: Zhang Yi <yi.z.zhang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
>> Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> ---
>> v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
>>     switched to GPLv2 license.
>>     fixed comments from Moritz Fischer.
>>     fixed kbuild warning, typos and clean up the code.
>> ---
>>  drivers/fpga/Makefile            |   2 +-
>>  drivers/fpga/intel-feature-dev.c | 130 ++++++
>>  drivers/fpga/intel-feature-dev.h | 341 ++++++++++++++++
>>  drivers/fpga/intel-pcie.c        | 841 ++++++++++++++++++++++++++++++++++++++-
>>  4 files changed, 1311 insertions(+), 3 deletions(-)
>>  create mode 100644 drivers/fpga/intel-feature-dev.c
>>  create mode 100644 drivers/fpga/intel-feature-dev.h
>>
>> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
>> index 5613133..ad24b3d 100644
>> --- a/drivers/fpga/Makefile
>> +++ b/drivers/fpga/Makefile
>> @@ -31,4 +31,4 @@ obj-$(CONFIG_OF_FPGA_REGION)                += of-fpga-region.o
>>  # Intel FPGA Support
>>  obj-$(CONFIG_INTEL_FPGA_PCI)         += intel-fpga-pci.o
>>
>> -intel-fpga-pci-objs := intel-pcie.o
>> +intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
>> diff --git a/drivers/fpga/intel-feature-dev.c b/drivers/fpga/intel-feature-dev.c
>> new file mode 100644
>> index 0000000..68f9cba
>> --- /dev/null
>> +++ b/drivers/fpga/intel-feature-dev.c
>> @@ -0,0 +1,130 @@
>> +/*
>> + * Intel FPGA Feature Device Driver
>> + *
>> + * Copyright (C) 2017 Intel Corporation, Inc.
>> + *
>> + * Authors:
>> + *   Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> + *   Zhang Yi <yi.z.zhang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> + *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> + *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
>> + *
>> + * This work is licensed under the terms of the GNU GPL version 2. See
>> + * the COPYING file in the top-level directory.
>> + */
>> +
>> +#include "intel-feature-dev.h"
>> +
>> +void feature_platform_data_add(struct feature_platform_data *pdata,
>> +                            int index, const char *name,
>> +                            int resource_index, void __iomem *ioaddr)
>> +{
>> +     WARN_ON(index >= pdata->num);
>> +
>> +     pdata->features[index].name = name;
>> +     pdata->features[index].resource_index = resource_index;
>> +     pdata->features[index].ioaddr = ioaddr;
>> +}
>> +
>> +struct feature_platform_data *
>> +feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
>> +{
>> +     struct feature_platform_data *pdata;
>> +
>> +     pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
>> +     if (pdata) {
>> +             pdata->dev = dev;
>> +             pdata->num = num;
>> +             mutex_init(&pdata->lock);
>> +     }
>> +
>> +     return pdata;
>> +}
>> +
>> +int fme_feature_num(void)
>> +{
>> +     return FME_FEATURE_ID_MAX;
>> +}
>> +
>> +int port_feature_num(void)
>> +{
>> +     return PORT_FEATURE_ID_MAX;
>> +}
>> +
>> +int fpga_port_id(struct platform_device *pdev)
>> +{
>> +     struct feature_port_header *port_hdr;
>> +     struct feature_port_capability capability;
>> +
>> +     port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
>> +                                            PORT_FEATURE_ID_HEADER);
>> +     WARN_ON(!port_hdr);
>> +
>> +     capability.csr = readq(&port_hdr->capability);
>> +     return capability.port_number;
>> +}
>> +EXPORT_SYMBOL_GPL(fpga_port_id);
>> +
>> +/*
>> + * Enable Port by clear the port soft reset bit, which is set by default.
>> + * The User AFU is unable to respond to any MMIO access while in reset.
>> + * __fpga_port_enable function should only be used after __fpga_port_disable
>> + * function.
>> + */
>> +void __fpga_port_enable(struct platform_device *pdev)
>> +{
>> +     struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> +     struct feature_port_header *port_hdr;
>> +     struct feature_port_control control;
>> +
>> +     WARN_ON(!pdata->disable_count);
>> +
>> +     if (--pdata->disable_count != 0)
>> +             return;
>> +
>> +     port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
>> +                                            PORT_FEATURE_ID_HEADER);
>> +     WARN_ON(!port_hdr);
>> +
>> +     control.csr = readq(&port_hdr->control);
>> +     control.port_sftrst = 0x0;
>> +     writeq(control.csr, &port_hdr->control);
>> +}
>> +EXPORT_SYMBOL_GPL(__fpga_port_enable);
>> +
>> +#define RST_POLL_INVL 10 /* us */
>> +#define RST_POLL_TIMEOUT 1000 /* us */
>> +
>> +int __fpga_port_disable(struct platform_device *pdev)
>> +{
>> +     struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> +     struct feature_port_header *port_hdr;
>> +     struct feature_port_control control;
>> +
>> +     if (pdata->disable_count++ != 0)
>> +             return 0;
>> +
>> +     port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
>> +                                            PORT_FEATURE_ID_HEADER);
>> +     WARN_ON(!port_hdr);
>> +
>> +     /* Set port soft reset */
>> +     control.csr = readq(&port_hdr->control);
>> +     control.port_sftrst = 0x1;
>> +     writeq(control.csr, &port_hdr->control);
>> +
>> +     /*
>> +      * HW sets ack bit to 1 when all outstanding requests have been drained
>> +      * on this port and minimum soft reset pulse width has elapsed.
>> +      * Driver polls port_soft_reset_ack to determine if reset done by HW.
>> +      */
>> +     if (readq_poll_timeout(&port_hdr->control, control.csr,
>> +                            (control.port_sftrst_ack == 1),
>> +                            RST_POLL_INVL, RST_POLL_TIMEOUT)) {
>> +             dev_err(&pdev->dev, "timeout, fail to reset device\n");
>> +             return -ETIMEDOUT;
>> +     }
>> +
>> +     return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(__fpga_port_disable);
>> diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
>> new file mode 100644
>> index 0000000..f67784a
>> --- /dev/null
>> +++ b/drivers/fpga/intel-feature-dev.h
>> @@ -0,0 +1,341 @@
>> +/*
>> + * Intel FPGA Feature Device Driver Header File
>> + *
>> + * Copyright (C) 2017 Intel Corporation, Inc.
>> + *
>> + * Authors:
>> + *   Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> + *   Zhang Yi <yi.z.zhang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> + *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> + *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
>> + *
>> + * This work is licensed under the terms of the GNU GPL version 2. See
>> + * the COPYING file in the top-level directory.
>> + */
>> +
>> +#ifndef __INTEL_FPGA_FEATURE_H
>> +#define __INTEL_FPGA_FEATURE_H
>> +
>> +#include <linux/fs.h>
>> +#include <linux/pci.h>
>> +#include <linux/uuid.h>
>> +#include <linux/delay.h>
>> +#include <linux/iopoll.h>
>> +#include <linux/platform_device.h>
>> +
>> +#ifndef readq
>> +static inline u64 readq(void __iomem *addr)
>> +{
>> +     return readl(addr) + ((u64)readl(addr + 4) << 32);
>> +}
>> +#endif
>> +
>> +#ifndef writeq
>> +static inline void writeq(u64 val, void __iomem *addr)
>> +{
>> +     writel((u32) (val), addr);
>> +     writel((u32) (val >> 32), (addr + 4));
>> +}
>> +#endif
>> +
>> +/* maximum supported number of ports */
>> +#define MAX_FPGA_PORT_NUM 4
>> +/* plus one for fme device */
>> +#define MAX_FEATURE_DEV_NUM  (MAX_FPGA_PORT_NUM + 1)
>> +
>> +#define FME_FEATURE_HEADER          "fme_hdr"
>> +#define FME_FEATURE_THERMAL_MGMT    "fme_thermal"
>> +#define FME_FEATURE_POWER_MGMT      "fme_power"
>> +#define FME_FEATURE_GLOBAL_PERF     "fme_gperf"
>> +#define FME_FEATURE_GLOBAL_ERR      "fme_error"
>> +#define FME_FEATURE_PR_MGMT         "fme_pr"
>> +
>> +#define PORT_FEATURE_HEADER         "port_hdr"
>> +#define PORT_FEATURE_UAFU           "port_uafu"
>> +#define PORT_FEATURE_ERR            "port_err"
>> +#define PORT_FEATURE_UMSG           "port_umsg"
>> +#define PORT_FEATURE_PR             "port_pr"
>> +#define PORT_FEATURE_STP            "port_stp"
>> +
>> +/* All headers and structures must be byte-packed to match the spec. */
>> +#pragma pack(1)
>> +
>> +/* common header for all features */
>> +struct feature_header {
>> +     union {
>> +             u64 csr;
>> +             struct {
>> +                     u64 id:12;
>> +                     u64 revision:4;
>> +                     u64 next_header_offset:24; /* offset to next header */
>> +                     u64 rsvdz:20;
>> +                     u64 type:4;                /* feature type */
>> +#define FEATURE_TYPE_AFU             0x1
>> +#define FEATURE_TYPE_PRIVATE         0x3
>> +             };
>> +     };
>> +};
>
> Here for example.
>> +
>> +/* common header for non-private features */
>> +struct feature_afu_header {
>> +     uuid_le guid;
>> +     union {
>> +             u64 csr;
>> +             struct {
>> +                     u64 next_afu:24;        /* pointer to next afu header */
>> +                     u64 rsvdz:40;
>> +             };
>> +     };
>> +};
>> +
>> +/* FME Header Register Set */
>> +/* FME Capability Register */
>> +struct feature_fme_capability {
>> +     union {
>> +             u64 csr;
>> +             struct {
>> +                     u64 fabric_verid:8;     /* Fabric version ID */
>> +                     u64 socket_id:1;        /* Socket id */
>> +                     u64 rsvdz1:3;
>> +                     u64 pcie0_link_avl:1;   /* PCIe0 link availability */
>> +                     u64 pcie1_link_avl:1;   /* PCIe1 link availability */
>> +                     u64 coherent_link_avl:1;/* Coherent link availability */
>> +                     u64 rsvdz2:1;
>> +                     u64 iommu_support:1;    /* IOMMU or VT-d supported */
>> +                     u64 num_ports:3;        /* Num of ports implemented */
>> +                     u64 rsvdz3:4;
>> +                     u64 addr_width_bits:6;  /* Address width supported */
>> +                     u64 rsvdz4:2;
>> +                     u64 cache_size:12;      /* Cache size in kb */
>> +                     u64 cache_assoc:4;      /* Cache Associativity */
>> +                     u64 rsvdz5:15;
>> +                     u64 lock_bit:1;         /* Latched lock bit by BIOS */
>> +             };
>> +     };
>> +};
>> +
>> +/* FME Port Offset Register */
>> +struct feature_fme_port {
>> +     union {
>> +             u64 csr;
>> +             struct {
>> +                     u64 port_offset:24;     /* Offset to port header */
>> +                     u64 rsvdz1:8;
>> +                     u64 port_bar:3;         /* Bar id */
>> +                     u64 rsvdz2:20;
>> +                     u64 afu_access_ctrl:1;  /* AFU access type: PF/VF */
>> +                     u64 rsvdz3:4;
>> +                     u64 port_implemented:1; /* Port implemented or not */
>> +                     u64 rsvdz4:3;
>> +             };
>> +     };
>> +};
>> +
>> +struct feature_fme_header {
>> +     struct feature_header header;
>> +     struct feature_afu_header afu_header;
>> +     u64 rsvd[2];
>> +     struct feature_fme_capability capability;
>> +     struct feature_fme_port port[MAX_FPGA_PORT_NUM];
>> +};
>> +
>> +/* FME Thermal Sub Feature Register Set */
>> +struct feature_fme_thermal {
>> +     struct feature_header header;
>> +};
>> +
>> +/* FME Power Sub Feature Register Set */
>> +struct feature_fme_power {
>> +     struct feature_header header;
>> +};
>> +
>> +/* FME Global Performance Sub Feature Register Set */
>> +struct feature_fme_gperf {
>> +     struct feature_header header;
>> +};
>> +
>> +/* FME Error Sub Feature Register Set */
>> +struct feature_fme_err {
>> +     struct feature_header header;
>> +};
>> +
>> +/* FME Partial Reconfiguration Sub Feature Register Set */
>> +struct feature_fme_pr {
>> +     struct feature_header header;
>> +};
>> +
>> +/* PORT Header Register Set */
>> +/* Port Capability Register */
>> +struct feature_port_capability {
>> +     union {
>> +             u64 csr;
>> +             struct {
>> +                     u64 port_number:2;      /* Port Number 0-3 */
>> +                     u64 rsvdz1:6;
>> +                     u64 mmio_size:16;       /* User MMIO size in KB */
>> +                     u64 rsvdz2:8;
>> +                     u64 sp_intr_num:4;      /* Supported interrupts num */
>> +                     u64 rsvdz3:28;
>> +             };
>> +     };
>> +};
>> +
>> +/* Port Control Register */
>> +struct feature_port_control {
>> +     union {
>> +             u64 csr;
>> +             struct {
>> +                     u64 port_sftrst:1;      /* Port Soft Reset */
>> +                     u64 rsvdz1:1;
>> +                     u64 latency_tolerance:1;/* '1' >= 40us, '0' < 40us */
>> +                     u64 rsvdz2:1;
>> +                     u64 port_sftrst_ack:1;  /* HW ACK for Soft Reset */
>> +                     u64 rsvdz3:59;
>> +             };
>> +     };
>> +};
>> +
>> +struct feature_port_header {
>> +     struct feature_header header;
>> +     struct feature_afu_header afu_header;
>> +     u64 rsvd[2];
>> +     struct feature_port_capability capability;
>> +     struct feature_port_control control;
>> +};
>> +
>> +/* PORT Error Sub Feature Register Set */
>> +struct feature_port_error {
>> +     struct feature_header header;
>> +};
>> +
>> +/* PORT Unordered Message Sub Feature Register Set */
>> +struct feature_port_umsg {
>> +     struct feature_header header;
>> +};
>> +
>> +/* PORT SignalTap Sub Feature Register Set */
>> +struct feature_port_stp {
>> +     struct feature_header header;
>> +};
>> +
>> +#pragma pack()
>> +
>> +struct feature {
>> +     const char *name;
>> +     int resource_index;
>> +     void __iomem *ioaddr;
>> +};
>> +
>> +struct feature_platform_data {
>> +     /* list the feature dev to cci_drvdata->port_dev_list. */
>> +     struct list_head node;
>> +     struct mutex lock;
>> +     struct platform_device *dev;
>> +     unsigned int disable_count;     /* count for port disable */
>> +
>> +     int num;                        /* number of features */
>> +     struct feature features[0];
>> +};
>> +
>> +enum fme_feature_id {
>> +     FME_FEATURE_ID_HEADER = 0x0,
>> +     FME_FEATURE_ID_THERMAL_MGMT = 0x1,
>> +     FME_FEATURE_ID_POWER_MGMT = 0x2,
>> +     FME_FEATURE_ID_GLOBAL_PERF = 0x3,
>> +     FME_FEATURE_ID_GLOBAL_ERR = 0x4,
>> +     FME_FEATURE_ID_PR_MGMT = 0x5,
>> +     FME_FEATURE_ID_MAX = 0x6,
>> +};
>> +
>> +enum port_feature_id {
>> +     PORT_FEATURE_ID_HEADER = 0x0,
>> +     PORT_FEATURE_ID_ERROR = 0x1,
>> +     PORT_FEATURE_ID_UMSG = 0x2,
>> +     PORT_FEATURE_ID_PR = 0x3,
>> +     PORT_FEATURE_ID_STP = 0x4,
>> +     PORT_FEATURE_ID_UAFU = 0x5,
>> +     PORT_FEATURE_ID_MAX = 0x6,
>> +};
>> +
>> +int fme_feature_num(void);
>> +int port_feature_num(void);
>> +
>> +#define FPGA_FEATURE_DEV_FME         "intel-fpga-fme"
>> +#define FPGA_FEATURE_DEV_PORT                "intel-fpga-port"
>> +
>> +void feature_platform_data_add(struct feature_platform_data *pdata,
>> +                            int index, const char *name,
>> +                            int resource_index, void __iomem *ioaddr);
>> +
>> +static inline int feature_platform_data_size(const int num)
>> +{
>> +     return sizeof(struct feature_platform_data) +
>> +             num * sizeof(struct feature);
>> +}
>> +
>> +struct feature_platform_data *
>> +feature_platform_data_alloc_and_init(struct platform_device *dev, int num);
>> +
>> +int fpga_port_id(struct platform_device *pdev);
>> +
>> +static inline int fpga_port_check_id(struct platform_device *pdev,
>> +                                  void *pport_id)
>> +{
>> +     return fpga_port_id(pdev) == *(int *)pport_id;
>> +}
>> +
>> +void __fpga_port_enable(struct platform_device *pdev);
>> +int __fpga_port_disable(struct platform_device *pdev);
>> +
>> +static inline void fpga_port_enable(struct platform_device *pdev)
>> +{
>> +     struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> +
>> +     mutex_lock(&pdata->lock);
>> +     __fpga_port_enable(pdev);
>> +     mutex_unlock(&pdata->lock);
>> +}
>> +
>> +static inline int fpga_port_disable(struct platform_device *pdev)
>> +{
>> +     struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> +     int ret;
>> +
>> +     mutex_lock(&pdata->lock);
>> +     ret = __fpga_port_disable(pdev);
>> +     mutex_unlock(&pdata->lock);
>> +
>> +     return ret;
>> +}
>> +
>> +static inline int __fpga_port_reset(struct platform_device *pdev)
>> +{
>> +     int ret;
>> +
>> +     ret = __fpga_port_disable(pdev);
>> +     if (ret)
>> +             return ret;
>> +
>> +     __fpga_port_enable(pdev);
>> +     return 0;
>> +}
>> +
>> +static inline int fpga_port_reset(struct platform_device *pdev)
>> +{
>> +     struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> +     int ret;
>> +
>> +     mutex_lock(&pdata->lock);
>> +     ret = __fpga_port_reset(pdev);
>> +     mutex_unlock(&pdata->lock);
>> +     return ret;
>> +}
>> +
>> +static inline void __iomem *
>> +get_feature_ioaddr_by_index(struct device *dev, int index)
>> +{
>> +     struct feature_platform_data *pdata = dev_get_platdata(dev);
>> +
>> +     return pdata->features[index].ioaddr;
>> +}
>> +#endif
>> diff --git a/drivers/fpga/intel-pcie.c b/drivers/fpga/intel-pcie.c
>> index f697de4..70b8284 100644
>> --- a/drivers/fpga/intel-pcie.c
>> +++ b/drivers/fpga/intel-pcie.c
>> @@ -23,10 +23,827 @@
>>  #include <linux/stddef.h>
>>  #include <linux/errno.h>
>>  #include <linux/aer.h>
>> +#include <linux/fpga/fpga-dev.h>
>> +
>> +#include "intel-feature-dev.h"
>>
>>  #define DRV_VERSION  "0.8"
>>  #define DRV_NAME     "intel-fpga-pci"
>>
>> +#define INTEL_FPGA_DEV       "intel-fpga-dev"
>> +
>> +static DEFINE_MUTEX(fpga_id_mutex);
>> +
>> +enum fpga_id_type {
>> +     FME_ID,         /* fme id allocation and mapping */
>> +     PORT_ID,        /* port id allocation and mapping */
>> +     FPGA_ID_MAX,
>> +};
>> +
>> +/* it is protected by fpga_id_mutex */
>> +static struct idr fpga_ids[FPGA_ID_MAX];
>> +
>> +struct cci_drvdata {
>> +     struct device *fme_dev;
>> +
>> +     struct mutex lock;
>> +     struct list_head port_dev_list;
>> +
>> +     struct list_head regions; /* global list of pci bar mapping region */
>> +};
>> +
>> +/* pci bar mapping info */
>> +struct cci_pci_region {
>> +     int bar;
>> +     void __iomem *ioaddr;   /* pointer to mapped bar region */
>> +     struct list_head node;
>> +};
>> +
>> +static void fpga_ids_init(void)
>> +{
>> +     int i;
>> +
>> +     for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
>> +             idr_init(fpga_ids + i);
>> +}
>> +
>> +static void fpga_ids_destroy(void)
>> +{
>> +     int i;
>> +
>> +     for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
>> +             idr_destroy(fpga_ids + i);
>> +}
>> +
>> +static int alloc_fpga_id(enum fpga_id_type type, struct device *dev)
>> +{
>> +     int id;
>> +
>> +     WARN_ON(type >= FPGA_ID_MAX);
>> +     mutex_lock(&fpga_id_mutex);
>> +     id = idr_alloc(fpga_ids + type, dev, 0, 0, GFP_KERNEL);
>> +     mutex_unlock(&fpga_id_mutex);
>> +     return id;
>> +}
>> +
>> +static void free_fpga_id(enum fpga_id_type type, int id)
>> +{
>> +     WARN_ON(type >= FPGA_ID_MAX);
>> +     mutex_lock(&fpga_id_mutex);
>> +     idr_remove(fpga_ids + type, id);
>> +     mutex_unlock(&fpga_id_mutex);
>> +}
>> +
>> +static void cci_pci_add_port_dev(struct pci_dev *pdev,
>> +                              struct platform_device *port_dev)
>> +{
>> +     struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
>> +     struct feature_platform_data *pdata = dev_get_platdata(&port_dev->dev);
>> +
>> +     mutex_lock(&drvdata->lock);
>> +     list_add(&pdata->node, &drvdata->port_dev_list);
>> +     get_device(&pdata->dev->dev);
>> +     mutex_unlock(&drvdata->lock);
>> +}
>> +
>> +static void cci_pci_remove_port_devs(struct pci_dev *pdev)
>> +{
>> +     struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
>> +     struct feature_platform_data *pdata, *ptmp;
>> +
>> +     mutex_lock(&drvdata->lock);
>> +     list_for_each_entry_safe(pdata, ptmp, &drvdata->port_dev_list, node) {
>> +             struct platform_device *port_dev = pdata->dev;
>> +
>> +             /* the port should be unregistered first. */
>> +             WARN_ON(device_is_registered(&port_dev->dev));
>> +             list_del(&pdata->node);
>> +             free_fpga_id(PORT_ID, port_dev->id);
>> +             put_device(&port_dev->dev);
>> +     }
>> +     mutex_unlock(&drvdata->lock);
>> +}
>> +
>> +/* info collection during feature dev build. */
>> +struct build_feature_devs_info {
>> +     struct pci_dev *pdev;
>> +
>> +     /*
>> +      * PCI BAR mapping info. Parsing feature list starts from
>> +      * BAR 0 and switch to different BARs to parse Port
>> +      */
>> +     void __iomem *ioaddr;
>> +     void __iomem *ioend;
>> +     int current_bar;
>> +
>> +     /* points to FME header where the port offset is figured out. */
>> +     void __iomem *pfme_hdr;
>> +
>> +     /* the container device for all feature devices */
>> +     struct fpga_dev *parent_dev;
>> +
>> +     /* current feature device */
>> +     struct platform_device *feature_dev;
>> +};
>> +
>> +static void cci_pci_release_regions(struct pci_dev *pdev)
>> +{
>> +     struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
>> +     struct cci_pci_region *tmp, *region;
>> +
>> +     list_for_each_entry_safe(region, tmp, &drvdata->regions, node) {
>> +             list_del(&region->node);
>> +             if (region->ioaddr)
>> +                     pci_iounmap(pdev, region->ioaddr);
>> +             devm_kfree(&pdev->dev, region);
>> +     }
>> +}
>> +
>> +static void __iomem *cci_pci_ioremap_bar(struct pci_dev *pdev, int bar)
>> +{
>> +     struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
>> +     struct cci_pci_region *region;
>> +
>> +     list_for_each_entry(region, &drvdata->regions, node)
>> +             if (region->bar == bar) {
>> +                     dev_dbg(&pdev->dev, "BAR %d region exists\n", bar);
>> +                     return region->ioaddr;
>> +             }
>> +
>> +     region = devm_kzalloc(&pdev->dev, sizeof(*region), GFP_KERNEL);
>> +     if (!region)
>> +             return NULL;
>> +
>> +     region->bar = bar;
>> +     region->ioaddr = pci_ioremap_bar(pdev, bar);
>> +     if (!region->ioaddr) {
>> +             dev_err(&pdev->dev, "can't ioremap memory from BAR %d.\n", bar);
>> +             devm_kfree(&pdev->dev, region);
>> +             return NULL;
>> +     }
>> +
>> +     list_add(&region->node, &drvdata->regions);
>> +     return region->ioaddr;
>> +}
>> +
>> +static int parse_start_from(struct build_feature_devs_info *binfo, int bar)
>> +{
>> +     binfo->ioaddr = cci_pci_ioremap_bar(binfo->pdev, bar);
>> +     if (!binfo->ioaddr)
>> +             return -ENOMEM;
>> +
>> +     binfo->current_bar = bar;
>> +     binfo->ioend = binfo->ioaddr + pci_resource_len(binfo->pdev, bar);
>> +     return 0;
>> +}
>> +
>> +static int parse_start(struct build_feature_devs_info *binfo)
>> +{
>> +     /* fpga feature list starts from BAR 0 */
>> +     return parse_start_from(binfo, 0);
>> +}
>> +
>> +/* switch the memory mapping to BAR# @bar */
>> +static int parse_switch_to(struct build_feature_devs_info *binfo, int bar)
>> +{
>> +     return parse_start_from(binfo, bar);
>> +}
>> +
>> +static struct build_feature_devs_info *
>> +build_info_alloc_and_init(struct pci_dev *pdev)
>> +{
>> +     struct build_feature_devs_info *binfo;
>> +
>> +     binfo = devm_kzalloc(&pdev->dev, sizeof(*binfo), GFP_KERNEL);
>> +     if (binfo)
>> +             binfo->pdev = pdev;
>> +
>> +     return binfo;
>> +}
>> +
>> +static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
>> +{
>> +     if (!strcmp(pdev->name, FPGA_FEATURE_DEV_FME))
>> +             return FME_ID;
>> +
>> +     if (!strcmp(pdev->name, FPGA_FEATURE_DEV_PORT))
>> +             return PORT_ID;
>> +
>> +     WARN_ON(1);
>> +     return FPGA_ID_MAX;
>> +}
>> +
>> +/*
>> + * register current feature device, it is called when we need to switch to
>> + * another feature parsing or we have parsed all features
>> + */
>> +static int build_info_commit_dev(struct build_feature_devs_info *binfo)
>> +{
>> +     int ret;
>> +
>> +     if (!binfo->feature_dev)
>> +             return 0;
>> +
>> +     ret = platform_device_add(binfo->feature_dev);
>> +     if (!ret) {
>> +             struct cci_drvdata *drvdata;
>> +
>> +             drvdata = dev_get_drvdata(&binfo->pdev->dev);
>> +             if (feature_dev_id_type(binfo->feature_dev) == PORT_ID)
>> +                     cci_pci_add_port_dev(binfo->pdev, binfo->feature_dev);
>> +             else
>> +                     drvdata->fme_dev = get_device(&binfo->feature_dev->dev);
>> +
>> +             /*
>> +              * reset it to avoid build_info_free() freeing their resource.
>> +              *
>> +              * The resource of successfully registered feature devices
>> +              * will be freed by platform_device_unregister(). See the
>> +              * comments in build_info_create_dev().
>> +              */
>> +             binfo->feature_dev = NULL;
>> +     }
>> +
>> +     return ret;
>> +}
>> +
>> +static int
>> +build_info_create_dev(struct build_feature_devs_info *binfo,
>> +                   enum fpga_id_type type, int feature_nr, const char *name)
>> +{
>> +     struct platform_device *fdev;
>> +     struct resource *res;
>> +     struct feature_platform_data *pdata;
>> +     int ret;
>> +
>> +     /* we will create a new device, commit current device first */
>> +     ret = build_info_commit_dev(binfo);
>> +     if (ret)
>> +             return ret;
>> +
>> +     /*
>> +      * we use -ENODEV as the initialization indicator which indicates
>> +      * whether the id need to be reclaimed
>> +      */
>> +     fdev = binfo->feature_dev = platform_device_alloc(name, -ENODEV);
>> +     if (!fdev)
>> +             return -ENOMEM;
>> +
>> +     fdev->id = alloc_fpga_id(type, &fdev->dev);
>> +     if (fdev->id < 0)
>> +             return fdev->id;
>> +
>> +     fdev->dev.parent = &binfo->parent_dev->dev;
>> +
>> +     /*
>> +      * we do not need to care for the memory which is associated with
>> +      * the platform device. After calling platform_device_unregister(),
>> +      * it will be automatically freed by device's release() callback,
>> +      * platform_device_release().
>> +      */
>> +     pdata = feature_platform_data_alloc_and_init(fdev, feature_nr);
>> +     if (!pdata)
>> +             return -ENOMEM;
>> +
>> +     /*
>> +      * the count should be initialized to 0 to make sure
>> +      *__fpga_port_enable() following __fpga_port_disable()
>> +      * works properly for port device.
>> +      * and it should always be 0 for fme device.
>> +      */
>> +     WARN_ON(pdata->disable_count);
>> +
>> +     fdev->dev.platform_data = pdata;
>> +     fdev->num_resources = feature_nr;
>> +     fdev->resource = kcalloc(feature_nr, sizeof(*res), GFP_KERNEL);
>> +     if (!fdev->resource)
>> +             return -ENOMEM;
>> +
>> +     return 0;
>> +}
>> +
>> +static int remove_feature_dev(struct device *dev, void *data)
>> +{
>> +     struct platform_device *pdev = to_platform_device(dev);
>> +
>> +     platform_device_unregister(pdev);
>> +     return 0;
>> +}
>> +
>> +static int remove_parent_dev(struct device *dev, void *data)
>> +{
>> +     /* remove platform devices attached in the parent device */
>> +     device_for_each_child(dev, NULL, remove_feature_dev);
>> +     fpga_dev_destroy(to_fpga_dev(dev));
>> +     return 0;
>> +}
>> +
>> +static void remove_all_devs(struct pci_dev *pdev)
>> +{
>> +     /* remove parent device and all its children. */
>> +     device_for_each_child(&pdev->dev, NULL, remove_parent_dev);
>> +}
>> +
>> +static void build_info_free(struct build_feature_devs_info *binfo)
>> +{
>> +     if (!IS_ERR_OR_NULL(binfo->parent_dev))
>> +             remove_all_devs(binfo->pdev);
>> +
>> +     /*
>> +      * it is a valid id, free it. See comments in
>> +      * build_info_create_dev()
>> +      */
>> +     if (binfo->feature_dev && binfo->feature_dev->id >= 0)
>> +             free_fpga_id(feature_dev_id_type(binfo->feature_dev),
>> +                          binfo->feature_dev->id);
>> +
>> +     platform_device_put(binfo->feature_dev);
>> +
>> +     devm_kfree(&binfo->pdev->dev, binfo);
>> +}
>> +
>> +#define FEATURE_TYPE_AFU     0x1
>> +#define FEATURE_TYPE_PRIVATE 0x3
>> +
>> +/* FME and PORT GUID are fixed */
>> +#define FEATURE_FME_GUID "f9e17764-38f0-82fe-e346-524ae92aafbf"
>> +#define FEATURE_PORT_GUID "6b355b87-b06c-9642-eb42-8d139398b43a"
>> +
>> +static bool feature_is_fme(struct feature_afu_header *afu_hdr)
>> +{
>> +     uuid_le u;
>> +
>> +     uuid_le_to_bin(FEATURE_FME_GUID, &u);
>> +
>> +     return !uuid_le_cmp(u, afu_hdr->guid);
>> +}
>> +
>> +static bool feature_is_port(struct feature_afu_header *afu_hdr)
>> +{
>> +     uuid_le u;
>> +
>> +     uuid_le_to_bin(FEATURE_PORT_GUID, &u);
>> +
>> +     return !uuid_le_cmp(u, afu_hdr->guid);
>> +}
>> +
>> +/*
>> + * UAFU GUID is dynamic as it can be changed after FME downloads different
>> + * Green Bitstream to the port, so we treat the unknown GUIDs which are
>> + * attached on port's feature list as UAFU.
>> + */
>> +static bool feature_is_UAFU(struct build_feature_devs_info *binfo)
>> +{
>> +     if (!binfo->feature_dev ||
>> +           feature_dev_id_type(binfo->feature_dev) != PORT_ID)
>> +             return false;
>> +
>> +     return true;
>> +}
>> +
>> +static void
>> +build_info_add_sub_feature(struct build_feature_devs_info *binfo,
>> +                        int feature_id, const char *feature_name,
>> +                        resource_size_t resource_size, void __iomem *start)
>> +{
>> +
>> +     struct platform_device *fdev = binfo->feature_dev;
>> +     struct feature_platform_data *pdata = dev_get_platdata(&fdev->dev);
>> +     struct resource *res = &fdev->resource[feature_id];
>> +
>> +     res->start = pci_resource_start(binfo->pdev, binfo->current_bar) +
>> +             start - binfo->ioaddr;
>> +     res->end = res->start + resource_size - 1;
>> +     res->flags = IORESOURCE_MEM;
>> +     res->name = feature_name;
>> +
>> +     feature_platform_data_add(pdata, feature_id,
>> +                               feature_name, feature_id, start);
>> +}
>> +
>> +struct feature_info {
>> +     const char *name;
>> +     resource_size_t resource_size;
>> +     int feature_index;
>> +};
>> +
>> +/* indexed by fme feature IDs which are defined in 'enum fme_feature_id'. */
>> +static struct feature_info fme_features[] = {
>> +     {
>> +             .name = FME_FEATURE_HEADER,
>> +             .resource_size = sizeof(struct feature_fme_header),
>> +             .feature_index = FME_FEATURE_ID_HEADER,
>> +     },
>> +     {
>> +             .name = FME_FEATURE_THERMAL_MGMT,
>> +             .resource_size = sizeof(struct feature_fme_thermal),
>> +             .feature_index = FME_FEATURE_ID_THERMAL_MGMT,
>> +     },
>> +     {
>> +             .name = FME_FEATURE_POWER_MGMT,
>> +             .resource_size = sizeof(struct feature_fme_power),
>> +             .feature_index = FME_FEATURE_ID_POWER_MGMT,
>> +     },
>> +     {
>> +             .name = FME_FEATURE_GLOBAL_PERF,
>> +             .resource_size = sizeof(struct feature_fme_gperf),
>> +             .feature_index = FME_FEATURE_ID_GLOBAL_PERF,
>> +     },
>> +     {
>> +             .name = FME_FEATURE_GLOBAL_ERR,
>> +             .resource_size = sizeof(struct feature_fme_err),
>> +             .feature_index = FME_FEATURE_ID_GLOBAL_ERR,
>> +     },
>> +     {
>> +             .name = FME_FEATURE_PR_MGMT,
>> +             .resource_size = sizeof(struct feature_fme_pr),
>> +             .feature_index = FME_FEATURE_ID_PR_MGMT,
>> +     }
>> +};
>> +
>> +/* indexed by port feature IDs which are defined in 'enum port_feature_id'. */
>> +static struct feature_info port_features[] = {
>> +     {
>> +             .name = PORT_FEATURE_HEADER,
>> +             .resource_size = sizeof(struct feature_port_header),
>> +             .feature_index = PORT_FEATURE_ID_HEADER,
>> +     },
>> +     {
>> +             .name = PORT_FEATURE_ERR,
>> +             .resource_size = sizeof(struct feature_port_error),
>> +             .feature_index = PORT_FEATURE_ID_ERROR,
>> +     },
>> +     {
>> +             .name = PORT_FEATURE_UMSG,
>> +             .resource_size = sizeof(struct feature_port_umsg),
>> +             .feature_index = PORT_FEATURE_ID_UMSG,
>> +     },
>> +     {
>> +             /* This feature isn't available for now */
>> +             .name = PORT_FEATURE_PR,
>> +             .resource_size = 0,
>> +             .feature_index = PORT_FEATURE_ID_PR,
>> +     },
>> +     {
>> +             .name = PORT_FEATURE_STP,
>> +             .resource_size = sizeof(struct feature_port_stp),
>> +             .feature_index = PORT_FEATURE_ID_STP,
>> +     },
>> +     {
>> +             /*
>> +              * For User AFU feature, its region size is not fixed, but
>> +              * reported by register PortCapability.mmio_size. Resource
>> +              * size of UAFU will be set while parse port device.
>> +              */
>> +             .name = PORT_FEATURE_UAFU,
>> +             .resource_size = 0,
>> +             .feature_index = PORT_FEATURE_ID_UAFU,
>> +     },
>> +};
>> +
>> +static int
>> +create_feature_instance(struct build_feature_devs_info *binfo,
>> +                     void __iomem *start, struct feature_info *finfo)
>> +{
>> +     if (binfo->ioend - start < finfo->resource_size)
>> +             return -EINVAL;
>> +
>> +     build_info_add_sub_feature(binfo, finfo->feature_index, finfo->name,
>> +                                finfo->resource_size, start);
>> +     return 0;
>> +}
>> +
>> +static int parse_feature_fme(struct build_feature_devs_info *binfo,
>> +                          void __iomem *start)
>> +{
>> +     struct cci_drvdata *drvdata = dev_get_drvdata(&binfo->pdev->dev);
>> +     int ret;
>> +
>> +     ret = build_info_create_dev(binfo, FME_ID, fme_feature_num(),
>> +                                     FPGA_FEATURE_DEV_FME);
>> +     if (ret)
>> +             return ret;
>> +
>> +     if (drvdata->fme_dev) {
>> +             dev_err(&binfo->pdev->dev, "Multiple FMEs are detected.\n");
>> +             return -EINVAL;
>> +     }
>> +
>> +     return create_feature_instance(binfo, start,
>> +                                    &fme_features[FME_FEATURE_ID_HEADER]);
>> +}
>> +
>> +static int parse_feature_fme_private(struct build_feature_devs_info *binfo,
>> +                                  struct feature_header *hdr)
>> +{
>> +     struct feature_header header;
>> +
>> +     header.csr = readq(hdr);
>> +
>> +     if (header.id >= ARRAY_SIZE(fme_features)) {
>> +             dev_info(&binfo->pdev->dev, "FME feature id %x is not supported yet.\n",
>> +                      header.id);
>> +             return 0;
>> +     }
>> +
>> +     return create_feature_instance(binfo, hdr, &fme_features[header.id]);
>> +}
>> +
>> +static int parse_feature_port(struct build_feature_devs_info *binfo,
>> +                          void __iomem *start)
>> +{
>> +     int ret;
>> +
>> +     ret = build_info_create_dev(binfo, PORT_ID, port_feature_num(),
>> +                                     FPGA_FEATURE_DEV_PORT);
>> +     if (ret)
>> +             return ret;
>> +
>> +     return create_feature_instance(binfo, start,
>> +                                    &port_features[PORT_FEATURE_ID_HEADER]);
>> +}
>> +
>> +static void enable_port_uafu(struct build_feature_devs_info *binfo,
>> +                          void __iomem *start)
>> +{
>> +     enum port_feature_id id = PORT_FEATURE_ID_UAFU;
>> +     struct feature_port_header *port_hdr;
>> +     struct feature_port_capability capability;
>> +
>> +     port_hdr = (struct feature_port_header *)start;
>> +     capability.csr = readq(&port_hdr->capability);
>> +     port_features[id].resource_size = capability.mmio_size << 10;
>> +
>> +     /*
>> +      * To enable User AFU, driver needs to clear reset bit on related port,
>> +      * otherwise the mmio space of this user AFU will be invalid.
>> +      */
>> +     if (port_features[id].resource_size)
>> +             fpga_port_reset(binfo->feature_dev);
>> +}
>> +
>> +static int parse_feature_port_private(struct build_feature_devs_info *binfo,
>> +                                   struct feature_header *hdr)
>> +{
>> +     struct feature_header header;
>> +     enum port_feature_id id;
>> +
>> +     header.csr = readq(hdr);
>> +     /*
>> +      * the region of port feature id is [0x10, 0x13], + 1 to reserve 0
>> +      * which is dedicated for port-hdr.
>> +      */
>> +     id = (header.id & 0x000f) + 1;
>> +
>> +     if (id >= ARRAY_SIZE(port_features)) {
>> +             dev_info(&binfo->pdev->dev, "Port feature id %x is not supported yet.\n",
>> +                      header.id);
>> +             return 0;
>> +     }
>> +
>> +     return create_feature_instance(binfo, hdr, &port_features[id]);
>> +}
>> +
>> +static int parse_feature_port_uafu(struct build_feature_devs_info *binfo,
>> +                              struct feature_header *hdr)
>> +{
>> +     enum port_feature_id id = PORT_FEATURE_ID_UAFU;
>> +     int ret;
>> +
>> +     if (port_features[id].resource_size) {
>> +             ret = create_feature_instance(binfo, hdr, &port_features[id]);
>> +             port_features[id].resource_size = 0;
>> +     } else {
>> +             dev_err(&binfo->pdev->dev, "the uafu feature header is mis-configured.\n");
>> +             ret = -EINVAL;
>> +     }
>> +
>> +     return ret;
>> +}
>> +
>> +static int parse_feature_afus(struct build_feature_devs_info *binfo,
>> +                           struct feature_header *hdr)
>> +{
>> +     int ret;
>> +     struct feature_afu_header *afu_hdr, header;
>> +     void __iomem *start;
>> +     void __iomem *end = binfo->ioend;
>> +
>> +     start = hdr;
>> +     for (; start < end; start += header.next_afu) {
>> +             if (end - start < (sizeof(*afu_hdr) + sizeof(*hdr)))
>> +                     return -EINVAL;
>> +
>> +             hdr = start;
>> +             afu_hdr = (struct feature_afu_header *) (hdr + 1);
>> +             header.csr = readq(&afu_hdr->csr);
>> +
>> +             if (feature_is_fme(afu_hdr)) {
>> +                     ret = parse_feature_fme(binfo, hdr);
>> +                     binfo->pfme_hdr = hdr;
>> +                     if (ret)
>> +                             return ret;
>> +             } else if (feature_is_port(afu_hdr)) {
>> +                     ret = parse_feature_port(binfo, hdr);
>> +                     enable_port_uafu(binfo, hdr);
>> +                     if (ret)
>> +                             return ret;
>> +             } else if (feature_is_UAFU(binfo)) {
>> +                     ret = parse_feature_port_uafu(binfo, hdr);
>> +                     if (ret)
>> +                             return ret;
>> +             } else
>> +                     dev_info(&binfo->pdev->dev, "AFU GUID %pUl is not supported yet.\n",
>> +                              afu_hdr->guid.b);
>> +
>> +             if (!header.next_afu)
>> +                     break;
>> +     }
>> +
>> +     return 0;
>> +}
>> +
>> +static int parse_feature_private(struct build_feature_devs_info *binfo,
>> +                              struct feature_header *hdr)
>> +{
>> +     struct feature_header header;
>> +
>> +     header.csr = readq(hdr);
>> +
>> +     if (!binfo->feature_dev) {
>> +             dev_err(&binfo->pdev->dev, "the private feature %x does not belong to any AFU.\n",
>> +                     header.id);
>> +             return -EINVAL;
>> +     }
>> +
>> +     switch (feature_dev_id_type(binfo->feature_dev)) {
>> +     case FME_ID:
>> +             return parse_feature_fme_private(binfo, hdr);
>> +     case PORT_ID:
>> +             return parse_feature_port_private(binfo, hdr);
>> +     default:
>> +             dev_info(&binfo->pdev->dev, "private feature %x belonging to AFU %s is not supported yet.\n",
>> +                      header.id, binfo->feature_dev->name);
>> +     }
>> +     return 0;
>> +}
>> +
>> +static int parse_feature(struct build_feature_devs_info *binfo,
>> +                      struct feature_header *hdr)
>> +{
>> +     struct feature_header header;
>> +     int ret = 0;
>> +
>> +     header.csr = readq(hdr);
>> +
>> +     switch (header.type) {
>> +     case FEATURE_TYPE_AFU:
>> +             ret = parse_feature_afus(binfo, hdr);
>> +             break;
>> +     case FEATURE_TYPE_PRIVATE:
>> +             ret = parse_feature_private(binfo, hdr);
>> +             break;
>> +     default:
>> +             dev_info(&binfo->pdev->dev,
>> +                      "Feature Type %x is not supported.\n", hdr->type);
>> +     };
>> +
>> +     return ret;
>> +}
>> +
>> +static int
>> +parse_feature_list(struct build_feature_devs_info *binfo, void __iomem *start)
>> +{
>> +     struct feature_header *hdr, header;
>> +     void __iomem *end = binfo->ioend;
>> +     int ret = 0;
>> +
>> +     for (; start < end; start += header.next_header_offset) {
>> +             if (end - start < sizeof(*hdr)) {
>> +                     dev_err(&binfo->pdev->dev, "The region is too small to contain a feature.\n");
>> +                     ret =  -EINVAL;
>> +                     break;
>> +             }
>> +
>> +             hdr = (struct feature_header *)start;
>> +             ret = parse_feature(binfo, hdr);
>> +             if (ret)
>> +                     break;
>> +
>> +             header.csr = readq(hdr);
>> +             if (!header.next_header_offset)
>> +                     break;
>> +     }
>> +
>> +     return ret;
>> +}
>> +
>> +static int parse_ports_from_fme(struct build_feature_devs_info *binfo)
>> +{
>> +     struct feature_fme_header *fme_hdr;
>> +     struct feature_fme_port port;
>> +     int i = 0, ret = 0;
>> +
>> +     if (binfo->pfme_hdr == NULL) {
>> +             dev_dbg(&binfo->pdev->dev, "VF is detected.\n");
>> +             return ret;
>> +     }
>> +
>> +     fme_hdr = binfo->pfme_hdr;
>> +
>> +     do {
>> +             port.csr = readq(&fme_hdr->port[i]);
>> +             if (!port.port_implemented)
>> +                     break;
>> +
>> +             ret = parse_switch_to(binfo, port.port_bar);
>> +             if (ret)
>> +                     break;
>> +
>> +             ret = parse_feature_list(binfo,
>> +                             binfo->ioaddr + port.port_offset);
>> +             if (ret)
>> +                     break;
>> +     } while (++i < MAX_FPGA_PORT_NUM);
>> +
>> +     return ret;
>> +}
>> +
>> +static int create_init_drvdata(struct pci_dev *pdev)
>> +{
>> +     struct cci_drvdata *drvdata;
>> +
>> +     drvdata = devm_kzalloc(&pdev->dev, sizeof(*drvdata), GFP_KERNEL);
>> +     if (!drvdata)
>> +             return -ENOMEM;
>> +
>> +     mutex_init(&drvdata->lock);
>> +     INIT_LIST_HEAD(&drvdata->port_dev_list);
>> +     INIT_LIST_HEAD(&drvdata->regions);
>> +
>> +     dev_set_drvdata(&pdev->dev, drvdata);
>> +     return 0;
>> +}
>> +
>> +static void destroy_drvdata(struct pci_dev *pdev)
>> +{
>> +     struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
>> +
>> +     if (drvdata->fme_dev) {
>> +             /* fme device should be unregistered first. */
>> +             WARN_ON(device_is_registered(drvdata->fme_dev));
>> +             free_fpga_id(FME_ID, to_platform_device(drvdata->fme_dev)->id);
>> +             put_device(drvdata->fme_dev);
>> +     }
>> +
>> +     cci_pci_remove_port_devs(pdev);
>> +     cci_pci_release_regions(pdev);
>> +     dev_set_drvdata(&pdev->dev, NULL);
>> +     devm_kfree(&pdev->dev, drvdata);
>> +}
>> +
>> +static int cci_pci_create_feature_devs(struct pci_dev *pdev)
>> +{
>> +     struct build_feature_devs_info *binfo;
>> +     int ret;
>> +
>> +     binfo = build_info_alloc_and_init(pdev);
>> +     if (!binfo)
>> +             return -ENOMEM;
>> +
>> +     binfo->parent_dev = fpga_dev_create(&pdev->dev, INTEL_FPGA_DEV);
>> +     if (IS_ERR(binfo->parent_dev)) {
>> +             ret = PTR_ERR(binfo->parent_dev);
>> +             goto free_binfo_exit;
>> +     }
>> +
>> +     ret = parse_start(binfo);
>> +     if (ret)
>> +             goto free_binfo_exit;
>> +
>> +     ret = parse_feature_list(binfo, binfo->ioaddr);
>> +     if (ret)
>> +             goto free_binfo_exit;
>> +
>> +     ret = parse_ports_from_fme(binfo);
>> +     if (ret)
>> +             goto free_binfo_exit;
>> +
>> +     ret = build_info_commit_dev(binfo);
>> +     if (ret)
>> +             goto free_binfo_exit;
>> +
>> +     /*
>> +      * everything is okay, reset ->parent_dev to stop it being
>> +      * freed by build_info_free()
>> +      */
>> +     binfo->parent_dev = NULL;
>> +
>> +free_binfo_exit:
>> +     build_info_free(binfo);
>> +     return ret;
>> +}
>> +
>>  /* PCI Device ID */
>>  #define PCIe_DEVICE_ID_PF_INT_5_X    0xBCBD
>>  #define PCIe_DEVICE_ID_PF_INT_6_X    0xBCC0
>> @@ -81,9 +898,18 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
>>               goto release_region_exit;
>>       }
>>
>> -     /* TODO: create and add the platform device per feature list */
>> +     ret = create_init_drvdata(pcidev);
>> +     if (ret)
>> +             goto release_region_exit;
>> +
>> +     ret = cci_pci_create_feature_devs(pcidev);
>> +     if (ret)
>> +             goto destroy_drvdata_exit;
>> +
>>       return 0;
>>
>> +destroy_drvdata_exit:
>> +     destroy_drvdata(pcidev);
>>  release_region_exit:
>>       pci_release_regions(pcidev);
>>  disable_error_report_exit:
>> @@ -94,6 +920,8 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
>>
>>  static void cci_pci_remove(struct pci_dev *pcidev)
>>  {
>> +     remove_all_devs(pcidev);
>> +     destroy_drvdata(pcidev);
>>       pci_release_regions(pcidev);
>>       pci_disable_pcie_error_reporting(pcidev);
>>       pci_disable_device(pcidev);
>> @@ -108,14 +936,23 @@ static void cci_pci_remove(struct pci_dev *pcidev)
>>
>>  static int __init ccidrv_init(void)
>>  {
>> +     int ret;
>> +
>>       pr_info("Intel(R) FPGA PCIe Driver: Version %s\n", DRV_VERSION);
>>
>> -     return pci_register_driver(&cci_pci_driver);
>> +     fpga_ids_init();
>> +
>> +     ret = pci_register_driver(&cci_pci_driver);
>> +     if (ret)
>> +             fpga_ids_destroy();
>> +
>> +     return ret;
>>  }
>>
>>  static void __exit ccidrv_exit(void)
>>  {
>>       pci_unregister_driver(&cci_pci_driver);
>> +     fpga_ids_destroy();
>>  }
>>
>>  module_init(ccidrv_init);
>> --
>> 1.8.3.1
>>
>
> Thanks,
>
> Moritz
>
> [1] https://www.kernel.org/doc/html/v4.11/doc-guide/kernel-doc.html#writing-kernel-doc-comments

Re: [PATCH v2 01/22] docs: fpga: add a document for Intel FPGA driver overview

[Alan Tull]

On Sun, Jun 25, 2017 at 8:51 PM, Wu Hao <hao.wu@intel.com> wrote:

Hi Hao,

> Add a document for Intel FPGA driver overview.
>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ----
> v2: added FME fpga-mgr/bridge/region platform driver to driver organization.
>     updated open discussion per current implementation.
>     fixed some typos.
> ---
>  Documentation/fpga/intel-fpga.txt | 256 ++++++++++++++++++++++++++++++++++++++
>  1 file changed, 256 insertions(+)
>  create mode 100644 Documentation/fpga/intel-fpga.txt
>
> diff --git a/Documentation/fpga/intel-fpga.txt b/Documentation/fpga/intel-fpga.txt
> new file mode 100644
> index 0000000..4a29470
> --- /dev/null
> +++ b/Documentation/fpga/intel-fpga.txt
> @@ -0,0 +1,256 @@
> +===============================================================================
> +                    Intel FPGA driver Overview
> +-------------------------------------------------------------------------------
> +                Enno Luebbers <enno.luebbers@intel.com>
> +                Xiao Guangrong <guangrong.xiao@linux.intel.com>
> +                Wu Hao <hao.wu@intel.com>
> +
> +The Intel FPGA driver provides interfaces for userspace applications to
> +configure, enumerate, open, and access FPGA accelerators on platforms equipped
> +with Intel(R) FPGA PCIe based solutions and enables system level management
> +functions such as FPGA reconfiguration, power management, and virtualization.
> +
> +HW Architecture
> +===============
> +From the OS's point of view, the FPGA hardware appears as a regular PCIe device.
> +The FPGA device memory is organized using a predefined data structure (Device
> +Feature List). Features supported by the particular FPGA device are exposed
> +through these data structures, as illustrated below:
> +
> +  +-------------------------------+  +-------------+
> +  |              PF               |  |     VF      |
> +  +-------------------------------+  +-------------+
> +      ^            ^         ^              ^
> +      |            |         |              |
> ++-----|------------|---------|--------------|-------+
> +|     |            |         |              |       |
> +|  +-----+     +-------+ +-------+      +-------+   |
> +|  | FME |     | Port0 | | Port1 |      | Port2 |   |
> +|  +-----+     +-------+ +-------+      +-------+   |
> +|                  ^         ^              ^       |
> +|                  |         |              |       |
> +|              +-------+ +------+       +-------+   |
> +|              |  AFU  | |  AFU |       |  AFU  |   |
> +|              +-------+ +------+       +-------+   |
> +|                                                   |
> +|                 FPGA PCIe Device                  |
> ++---------------------------------------------------+
> +
> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which can be
> +used to assign individual accelerators to virtual machines.
> +
> +FME (FPGA Management Engine)
> +============================
> +The FPGA Management Engine performs power and thermal management, error
> +reporting, reconfiguration, performance reporting, and other infrastructure
> +functions. Each FPGA has one FME, which is always accessed through the physical
> +function (PF).
> +
> +User-space applications can acquire exclusive access to the FME using open(),
> +and release it using close().
> +
> +The following functions are exposed through ioctls:
> +
> +       Get driver API version (FPGA_GET_API_VERSION)
> +       Check for extensions (FPGA_CHECK_EXTENSION)
> +       Assign port to PF (FPGA_FME_PORT_ASSIGN)
> +       Release port from PF (FPGA_FME_PORT_RELEASE)
> +       Program bitstream (FPGA_FME_PORT_PR)
> +

I was hoping the API mailing list might have an opinion about this,
but I think adding ioctls to the kernel is discouraged.  Could these
be sysfs?

> +More functions are exposed through sysfs
> +(/sys/class/fpga/fpga.n/intel-fpga-fme.n/):
> +
> +       Read bitstream ID (bitstream_id)
> +       Read bitstream metadata (bitstream_metadata)
> +       Read number of ports (ports_num)
> +       Read socket ID (socket_id)
> +       Read performance counters (perf/)
> +       Power management (power_mgmt/)
> +       Thermal management (thermal_mgmt/)
> +       Error reporting (errors/)
> +
> +PORT
> +====
> +A port represents the interface between the static FPGA fabric (the "blue
> +bitstream") and a partially reconfigurable region containing an AFU (the "green
> +bitstream"). It controls the communication from SW to the accelerator and
> +exposes features such as reset and debug.
> +
> +A PCIe device may have several ports and each port can be released from PF by
> +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe sriov
> +sysfs interface.
> +
> +AFU
> +===
> +An AFU is attached to a port and exposes a 256k MMIO region to be used for
> +accelerator-specific control registers.
> +
> +User-space applications can acquire exclusive access to an AFU attached to a
> +port by using open() on the port device node, and release it using close().
> +
> +The following functions are exposed through ioctls:
> +
> +       Get driver API version (FPGA_GET_API_VERSION)
> +       Check for extensions (FPGA_CHECK_EXTENSION)
> +       Get port info (FPGA_PORT_GET_INFO)
> +       Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
> +       Map DMA buffer (FPGA_PORT_DMA_MAP)
> +       Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
> +       Reset AFU (FPGA_PORT_RESET)
> +       Enable UMsg (FPGA_PORT_UMSG_ENABLE)
> +       Disable UMsg (FPGA_PORT_UMSG_DISABLE)
> +       Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
> +       Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)
> +
> +User-space applications can also mmap() accelerator MMIO regions.
> +
> +More functions are exposed through sysfs:
> +(/sys/class/fpga/fpga.n/intel-fpga-port.m/):
> +
> +       Read Accelerator GUID (afu_id)
> +       Error reporting (errors/)
> +
> +Partial Reconfiguration
> +=======================
> +As mentioned above, accelerators can be reconfigured through partial
> +reconfiguration of a green bitstream file (GBS). The green bitstream must have
> +been generated for the exact blue bitstream and targeted reconfigurable region
> +(port) of the FPGA; otherwise, the reconfiguration operation will fail and
> +possibly cause system instability. This compatibility can be checked by
> +comparing the interface ID noted in the GBS header against the interface ID
> +exposed by the FME through sysfs (see above). This check is usually done by
> +user-space before calling the reconfiguration IOCTL.
> +
> +FPGA virtualization
> +===================
> +To enable accessing an accelerator from applications running in a VM, the
> +respective AFU's port needs to be assigned to a VF using the following steps:
> +
> + a) The PF owns all AFU ports by default. Any port that needs to be reassigned
> + to a VF must be released from PF firstly through the FPGA_FME_PORT_RELEASE
> + ioctl on the FME device.
> +
> + b) Once N ports are released from PF, then user can use below command to
> + enable SRIOV and VFs. Each VF owns only one Port with AFU.
> +
> + echo N > $PCI_DEVICE_PATH/sriov_numvfs
> +
> + c) Pass through the VFs to VMs
> +
> + d) The AFU under VF is accessible from applications in VM (using the same
> + driver inside the VF).
> +
> +Note the an FME can't be assigned to a VF, thus PR and other management
> +functions are only available via the PF.
> +
> +
> +Driver organization
> +===================
> +
> +  +-------++------++------+             |
> +  | FME   || FME  || FME  |             |
> +  | FPGA  || FPGA || FPGA |             |
> +  |Manager||Bridge||Region|             |
> +  +-------++------++------+             |
> +  +-----------------------+  +--------+ |             +--------+
> +  |          FME          |  |  AFU   | |             |  AFU   |
> +  |         Module        |  | Module | |             | Module |
> +  +-----------------------+  +--------+ |             +--------+
> +        +-----------------------+       |      +-----------------------+
> +        | FPGA Container Device |       |      | FPGA Container Device |
> +        +-----------------------+       |      +-----------------------+
> +          +------------------+          |         +------------------+
> +          | FPGA PCIE Module |          | Virtual | FPGA PCIE Module |
> +          +------------------+   Host   | Machine +------------------+
> + -------------------------------------- | ------------------------------
> +           +---------------+            |          +---------------+
> +           | PCI PF Device |            |          | PCI VF Device |
> +           +---------------+            |          +---------------+
> +
> +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe device
> +driver is always loaded first once a FPGA PCIE PF or VF device is detected. This
> +driver plays an infrastructural role in the driver architecture.  It:
> +
> +       a) creates FPGA container device as parent of the feature devices.
> +       b) walks through the Device Feature List, which is implemented in PCIE
> +          device BAR memory, to discover feature devices and their sub features
> +          and create platform device for them under the container device.
> +       c) supports SRIOV.
> +       d) introduces the feature device infrastructure, which abstracts
> +          operations for sub features and exposes common functions to feature
> +          device drivers.
> +
> +The FPGA Management Engine (FME) driver is a platform driver which is loaded
> +automatically after FME platform device creation from the PCIE driver. It
> +provides the key features for FPGA management, including:
> +
> +       a) Power and thermal management, error reporting, performance reporting
> +          and other infrastructure functions. Users can access these functions
> +          via sysfs interfaces exposed by FME driver.
> +       b) Partial Reconfiguration. The FME driver creates platform devices
> +          for FPGA manager, FPGA bridges and FPGA regions during PR sub
> +          feature initialization; Once it receives an FPGA_FME_PORT_PR ioctl
> +          from user, it invokes the common interface function from FPGA Region
> +          to complete the partial reconfiguration of the bitstream to the given
> +          port.
> +       c) Port management for virtualization. The FME driver introduces two
> +          ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
> +          FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the port is
> +          released from the PF, it can be assigned to the VF through the SRIOV
> +          interfaces provided by PCIE driver. (Refer to "FPGA virtualization"
> +          for more details).
> +
> +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) driver
> +is probed once the AFU platform device is created. The main function of this
> +module is to provide an interface for userspace applications to access the
> +individual accelerators, including basic reset control on port, AFU MMIO region
> +export, dma buffer mapping service, UMsg notification, and remote debug
> +functions (see above).
> +
> +
> +Device enumeration
> +==================
> +This section introduces how applications enumerate the fpga device from
> +the sysfs hierarchy under /sys/class/fpga.
> +
> +In the example below, two Intel(R) FPGA devices are installed in the host. Each
> +fpga device has one FME and two ports (AFUs).
> +
> +For each FPGA device, a device director is created under /sys/class/fpga/:
> +
> +       /sys/class/fpga/fpga.0
> +       /sys/class/fpga/fpga.1
> +
> +The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's name.
> +Application can retrieve name information via the sysfs interface:
> +
> +       /sys/class/fpga/fpga.0/name
> +
> +Each node has one FME and two ports (AFUs) as child devices:
> +
> +       /sys/class/fpga/fpga.0/intel-fpga-fme.0
> +       /sys/class/fpga/fpga.0/intel-fpga-port.0
> +       /sys/class/fpga/fpga.0/intel-fpga-port.1
> +
> +       /sys/class/fpga/fpga.1/intel-fpga-fme.1
> +       /sys/class/fpga/fpga.1/intel-fpga-port.2
> +       /sys/class/fpga/fpga.1/intel-fpga-port.3
> +
> +In general, the FME/AFU sysfs interfaces are named as follows:
> +
> +       /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/
> +       /sys/class/fpga/<fpga.n>/<intel-fpga-port.m>/
> +
> +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
> +ports.
> +
> +The device nodes used for ioctl() or mmap() can be referenced through:
> +
> +       /sys/class/fpga/<fpga.n>/<intel-fpga-port.n>/dev
> +       /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/dev
> +
> +Open discussion
> +===============
> +FME driver exports one ioctl (FPGA_FME_PORT_PR) for partial reconfiguration to
> +user now. In the future, if unified user interfaces for reconfiguration are
> +added, FME driver should switch to them from ioctl interface.

Adding an ioctl that will go away in the future?

I think we are going to have to find a different interface for doing
partial reconfiguration.  There was a recent conversation on the
mailing list regarding adding a header to the raw bitstream and
probably using sysfs to present that to the kernel.  I'm working on
getting an RFC together for that.

Alan Tull

Re: [PATCH v2 05/22] fpga: mgr: add status for fpga-mgr

[Alan Tull]

On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

Hi Hao,

> This patch adds status to fpga-manager data structure, to allow
> driver to store full/partial reconfiguration errors and other
> status information.
>
> one sysfs interface created for user space application to read
> fpga-manager status.
>
> Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> ---
>  Documentation/ABI/testing/sysfs-class-fpga-manager | 10 +++++++++
>  drivers/fpga/fpga-mgr.c                            | 24 ++++++++++++++++++++++
>  include/linux/fpga/fpga-mgr.h                      |  9 ++++++++
>  3 files changed, 43 insertions(+)
>
> diff --git a/Documentation/ABI/testing/sysfs-class-fpga-manager b/Documentation/ABI/testing/sysfs-class-fpga-manager
> index 23056c5..71b083e 100644
> --- a/Documentation/ABI/testing/sysfs-class-fpga-manager
> +++ b/Documentation/ABI/testing/sysfs-class-fpga-manager
> @@ -35,3 +35,13 @@ Description: Read fpga manager state as a string.
>                 * write complete        = Doing post programming steps
>                 * write complete error  = Error while doing post programming
>                 * operating             = FPGA is programmed and operating
> +
> +What:          /sys/class/fpga_manager/<fpga>/status
> +Date:          June 2017
> +KernelVersion: 4.12
> +Contact:       Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> +Description:   Read fpga manager status as a string.
> +               If FPGA programming operation fails, it could be due to crc
> +               error or incompatible bitstream image. The intent of this
> +               interface is to provide more detailed information for FPGA
> +               programming errors to userspace.
> diff --git a/drivers/fpga/fpga-mgr.c b/drivers/fpga/fpga-mgr.c
> index be13cce..2485658 100644
> --- a/drivers/fpga/fpga-mgr.c
> +++ b/drivers/fpga/fpga-mgr.c
> @@ -388,12 +388,36 @@ static ssize_t state_show(struct device *dev,
>         return sprintf(buf, "%s\n", state_str[mgr->state]);
>  }
>
> +static ssize_t status_show(struct device *dev,
> +                          struct device_attribute *attr, char *buf)
> +{
> +       struct fpga_manager *mgr = to_fpga_manager(dev);
> +       int len = 0;
> +
> +       if (mgr->status & FPGA_MGR_STATUS_OPERATION_ERR)
> +               len += sprintf(buf + len, "reconfig operation error\n");
> +       if (mgr->status & FPGA_MGR_STATUS_CRC_ERR)
> +               len += sprintf(buf + len, "reconfig crc error\n");
> +       if (mgr->status & FPGA_MGR_STATUS_INCOMPATIBLE_BS_ERR)
> +               len += sprintf(buf + len, "reconfig incompatible BS error\n");
> +       if (mgr->status & FPGA_MGR_STATUS_IP_PROTOCOL_ERR)
> +               len += sprintf(buf + len, "reconfig IP protocol error\n");
> +       if (mgr->status & FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR)
> +               len += sprintf(buf + len, "reconfig fifo overflow error\n");
> +       if (mgr->status & FPGA_MGR_STATUS_SECURE_LOAD_ERR)
> +               len += sprintf(buf + len, "reconfig secure load error\n");
> +
> +       return len;
> +}
> +
>  static DEVICE_ATTR_RO(name);
>  static DEVICE_ATTR_RO(state);
> +static DEVICE_ATTR_RO(status);
>
>  static struct attribute *fpga_mgr_attrs[] = {
>         &dev_attr_name.attr,
>         &dev_attr_state.attr,
> +       &dev_attr_status.attr,
>         NULL,
>  };
>  ATTRIBUTE_GROUPS(fpga_mgr);
> diff --git a/include/linux/fpga/fpga-mgr.h b/include/linux/fpga/fpga-mgr.h
> index b222a57..8cb42ac 100644
> --- a/include/linux/fpga/fpga-mgr.h
> +++ b/include/linux/fpga/fpga-mgr.h
> @@ -128,6 +128,14 @@ struct fpga_manager_ops {
>         void (*fpga_remove)(struct fpga_manager *mgr);
>  };
>
> +/* FPGA manager status: Partial/Full Reconfiguration errors */
> +#define FPGA_MGR_STATUS_OPERATION_ERR          BIT(0)
> +#define FPGA_MGR_STATUS_CRC_ERR                        BIT(1)
> +#define FPGA_MGR_STATUS_INCOMPATIBLE_BS_ERR    BIT(2)

How about ..._INCOMPATIBLE_IMAGE_ERR? :)

> +#define FPGA_MGR_STATUS_IP_PROTOCOL_ERR                BIT(3)
> +#define FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR      BIT(4)
> +#define FPGA_MGR_STATUS_SECURE_LOAD_ERR                BIT(5)
> +
>  /**
>   * struct fpga_manager - fpga manager structure
>   * @name: name of low level fpga manager
> @@ -142,6 +150,7 @@ struct fpga_manager {
>         struct device dev;
>         struct mutex ref_mutex;
>         enum fpga_mgr_states state;
> +       u64 status;

With this implementation, the low level driver sets ops->status and
that could be stale by the time the framework looks at it.  I suggest
adding a function to fpga_manager_ops that returns the status.  That
way whenever the status is requested, the low level driver will have
the opportunity to read status registers.

Besides this, this new sysfs looks helpful.

Alan

>         const struct fpga_manager_ops *mops;
>         void *priv;
>  };
> --
> 1.8.3.1
>

Re: [PATCH v2 17/22] fpga: intel: add fpga region platform driver for FME

[Alan Tull]

On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

Hi Hao,

Thanks for using the region patchset.  This looks good except for one
thing below.

> This patch adds fpga region platform driver for Intel FPGA Management
> Engine. It register a fpga region with given fpga manager / bridge
> device.
>
> Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> ---
>  drivers/fpga/Kconfig                 |  7 +++
>  drivers/fpga/Makefile                |  1 +
>  drivers/fpga/intel-fpga-fme-region.c | 91 ++++++++++++++++++++++++++++++++++++
>  3 files changed, 99 insertions(+)
>  create mode 100644 drivers/fpga/intel-fpga-fme-region.c
>
> diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> index 05e2a8e..db1a6ea 100644
> --- a/drivers/fpga/Kconfig
> +++ b/drivers/fpga/Kconfig
> @@ -165,6 +165,13 @@ config INTEL_FPGA_FME_BRIDGE
>           Say Y to enable FPGA Bridge driver for Intel FPGA Management
>           Engine.
>
> +config INTEL_FPGA_FME_REGION
> +       tristate "Intel FPGA FME Region Driver"
> +       depends on INTEL_FPGA_FME && FPGA_REGION
> +       help
> +         Say Y to enable FPGA Region driver for Intel FPGA Management
> +         Engine.
> +
>  endif
>
>  endif # FPGA
> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> index 131c105..fd0a71f 100644
> --- a/drivers/fpga/Makefile
> +++ b/drivers/fpga/Makefile
> @@ -33,6 +33,7 @@ obj-$(CONFIG_INTEL_FPGA_PCI)          += intel-fpga-pci.o
>  obj-$(CONFIG_INTEL_FPGA_FME)           += intel-fpga-fme.o
>  obj-$(CONFIG_INTEL_FPGA_FME_MGR)       += intel-fpga-fme-mgr.o
>  obj-$(CONFIG_INTEL_FPGA_FME_BRIDGE)    += intel-fpga-fme-br.o
> +obj-$(CONFIG_INTEL_FPGA_FME_REGION)    += intel-fpga-fme-region.o
>
>  intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
>  intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
> diff --git a/drivers/fpga/intel-fpga-fme-region.c b/drivers/fpga/intel-fpga-fme-region.c
> new file mode 100644
> index 0000000..2b5ca4d
> --- /dev/null
> +++ b/drivers/fpga/intel-fpga-fme-region.c
> @@ -0,0 +1,91 @@
> +/*
> + * FPGA Region Driver for Intel FPGA Management Engine (FME)
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *   Joseph Grecco <joe.grecco-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *   Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *   Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *   Ananda Ravuri <ananda.ravuri-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *   Henry Mitchel <henry.mitchel-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2. See
> + * the COPYING file in the top-level directory.
> + */
> +
> +#include <linux/module.h>
> +#include <linux/fpga/fpga-region.h>
> +
> +#include "intel-feature-dev.h"
> +#include "intel-fme.h"
> +
> +static int fme_region_get_bridges(struct fpga_region *region,
> +                                 struct fpga_image_info *info)
> +{
> +       struct fme_region_pdata *pdata = region->priv;
> +       struct device *dev = &pdata->br->dev;
> +
> +       return fpga_bridge_get_to_list(dev, info, &region->bridge_list);
> +}
> +
> +static int fme_region_probe(struct platform_device *pdev)
> +{
> +       struct device *dev = &pdev->dev;
> +       struct fme_region_pdata *pdata = dev_get_platdata(dev);
> +       struct fpga_region *region;
> +       struct fpga_manager *mgr;
> +       int ret;
> +
> +       mgr = fpga_mgr_get(&pdata->mgr->dev);
> +       if (IS_ERR(mgr))
> +               return -EPROBE_DEFER;
> +
> +       region = devm_kzalloc(dev, sizeof(*region), GFP_KERNEL);
> +       if (!region) {
> +               ret = -ENOMEM;
> +               goto eprobe_mgr_put;
> +       }
> +
> +       region->mgr = mgr;
> +       region->get_bridges = fme_region_get_bridges;
> +       region->priv = pdata;
> +
> +       ret = fpga_region_register(dev, region);
> +       if (ret)
> +               goto eprobe_mgr_put;
> +
> +       dev_dbg(dev, "Intel FME FPGA Region probed\n");
> +
> +       return 0;
> +
> +eprobe_mgr_put:
> +       fpga_mgr_put(mgr);
> +       return ret;
> +}
> +
> +static int fme_region_remove(struct platform_device *pdev)
> +{
> +       struct fpga_region *region = dev_get_drvdata(&pdev->dev);
> +
> +       fpga_mgr_put(region->mgr);
> +       fpga_region_unregister(region);

Please unregister before putting the mgr ;)

Alan

> +
> +       return 0;
> +}
> +
> +static struct platform_driver fme_region_driver = {
> +       .driver = {
> +               .name    = INTEL_FPGA_FME_REGION,
> +       },
> +       .probe   = fme_region_probe,
> +       .remove  = fme_region_remove,
> +};
> +
> +module_platform_driver(fme_region_driver);
> +
> +MODULE_DESCRIPTION("FPGA Region for Intel FPGA Management Engine");
> +MODULE_AUTHOR("Intel Corporation");
> +MODULE_LICENSE("GPL v2");
> +MODULE_ALIAS("platform:intel-fpga-fme-region");
> --
> 1.8.3.1
>

Re: [PATCH v2 17/22] fpga: intel: add fpga region platform driver for FME

[Wu Hao]

On Wed, Jul 12, 2017 at 11:09:46AM -0500, Alan Tull wrote:
> On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> 
> Hi Hao,
> 
> Thanks for using the region patchset.  This looks good except for one
> thing below.

Hi Alan

Thank you very much for your review. : )

> 
> > This patch adds fpga region platform driver for Intel FPGA Management
> > Engine. It register a fpga region with given fpga manager / bridge
> > device.
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > ---
> >  drivers/fpga/Kconfig                 |  7 +++
> >  drivers/fpga/Makefile                |  1 +
> >  drivers/fpga/intel-fpga-fme-region.c | 91 ++++++++++++++++++++++++++++++++++++
> >  3 files changed, 99 insertions(+)
> >  create mode 100644 drivers/fpga/intel-fpga-fme-region.c
> >
> > diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> > index 05e2a8e..db1a6ea 100644
> > --- a/drivers/fpga/Kconfig
> > +++ b/drivers/fpga/Kconfig
> > @@ -165,6 +165,13 @@ config INTEL_FPGA_FME_BRIDGE
> >           Say Y to enable FPGA Bridge driver for Intel FPGA Management
> >           Engine.
> >
> > +config INTEL_FPGA_FME_REGION
> > +       tristate "Intel FPGA FME Region Driver"
> > +       depends on INTEL_FPGA_FME && FPGA_REGION
> > +       help
> > +         Say Y to enable FPGA Region driver for Intel FPGA Management
> > +         Engine.
> > +
> >  endif
> >
> >  endif # FPGA
> > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> > index 131c105..fd0a71f 100644
> > --- a/drivers/fpga/Makefile
> > +++ b/drivers/fpga/Makefile
> > @@ -33,6 +33,7 @@ obj-$(CONFIG_INTEL_FPGA_PCI)          += intel-fpga-pci.o
> >  obj-$(CONFIG_INTEL_FPGA_FME)           += intel-fpga-fme.o
> >  obj-$(CONFIG_INTEL_FPGA_FME_MGR)       += intel-fpga-fme-mgr.o
> >  obj-$(CONFIG_INTEL_FPGA_FME_BRIDGE)    += intel-fpga-fme-br.o
> > +obj-$(CONFIG_INTEL_FPGA_FME_REGION)    += intel-fpga-fme-region.o
> >
> >  intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
> >  intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
> > diff --git a/drivers/fpga/intel-fpga-fme-region.c b/drivers/fpga/intel-fpga-fme-region.c
> > new file mode 100644
> > index 0000000..2b5ca4d
> > --- /dev/null
> > +++ b/drivers/fpga/intel-fpga-fme-region.c
> > @@ -0,0 +1,91 @@
> > +/*
> > + * FPGA Region Driver for Intel FPGA Management Engine (FME)
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Joseph Grecco <joe.grecco-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Ananda Ravuri <ananda.ravuri-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Henry Mitchel <henry.mitchel-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *
> > + * This work is licensed under the terms of the GNU GPL version 2. See
> > + * the COPYING file in the top-level directory.
> > + */
> > +
> > +#include <linux/module.h>
> > +#include <linux/fpga/fpga-region.h>
> > +
> > +#include "intel-feature-dev.h"
> > +#include "intel-fme.h"
> > +
> > +static int fme_region_get_bridges(struct fpga_region *region,
> > +                                 struct fpga_image_info *info)
> > +{
> > +       struct fme_region_pdata *pdata = region->priv;
> > +       struct device *dev = &pdata->br->dev;
> > +
> > +       return fpga_bridge_get_to_list(dev, info, &region->bridge_list);
> > +}
> > +
> > +static int fme_region_probe(struct platform_device *pdev)
> > +{
> > +       struct device *dev = &pdev->dev;
> > +       struct fme_region_pdata *pdata = dev_get_platdata(dev);
> > +       struct fpga_region *region;
> > +       struct fpga_manager *mgr;
> > +       int ret;
> > +
> > +       mgr = fpga_mgr_get(&pdata->mgr->dev);
> > +       if (IS_ERR(mgr))
> > +               return -EPROBE_DEFER;
> > +
> > +       region = devm_kzalloc(dev, sizeof(*region), GFP_KERNEL);
> > +       if (!region) {
> > +               ret = -ENOMEM;
> > +               goto eprobe_mgr_put;
> > +       }
> > +
> > +       region->mgr = mgr;
> > +       region->get_bridges = fme_region_get_bridges;
> > +       region->priv = pdata;
> > +
> > +       ret = fpga_region_register(dev, region);
> > +       if (ret)
> > +               goto eprobe_mgr_put;
> > +
> > +       dev_dbg(dev, "Intel FME FPGA Region probed\n");
> > +
> > +       return 0;
> > +
> > +eprobe_mgr_put:
> > +       fpga_mgr_put(mgr);
> > +       return ret;
> > +}
> > +
> > +static int fme_region_remove(struct platform_device *pdev)
> > +{
> > +       struct fpga_region *region = dev_get_drvdata(&pdev->dev);
> > +
> > +       fpga_mgr_put(region->mgr);
> > +       fpga_region_unregister(region);
> 
> Please unregister before putting the mgr ;)
> 

Sure, make sense, I will fix this in the next version.

Thanks
Hao

Re: [PATCH v2 05/22] fpga: mgr: add status for fpga-mgr

[Wu Hao]

On Wed, Jul 12, 2017 at 10:22:17AM -0500, Alan Tull wrote:
> On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> 
> Hi Hao,
> 

Hi Alan

Thanks a lot for your feedback. : )

> > This patch adds status to fpga-manager data structure, to allow
> > driver to store full/partial reconfiguration errors and other
> > status information.
> >
> > one sysfs interface created for user space application to read
> > fpga-manager status.
> >
> > Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > ---
> >  Documentation/ABI/testing/sysfs-class-fpga-manager | 10 +++++++++
> >  drivers/fpga/fpga-mgr.c                            | 24 ++++++++++++++++++++++
> >  include/linux/fpga/fpga-mgr.h                      |  9 ++++++++
> >  3 files changed, 43 insertions(+)
> >
> > diff --git a/Documentation/ABI/testing/sysfs-class-fpga-manager b/Documentation/ABI/testing/sysfs-class-fpga-manager
> > index 23056c5..71b083e 100644
> > --- a/Documentation/ABI/testing/sysfs-class-fpga-manager
> > +++ b/Documentation/ABI/testing/sysfs-class-fpga-manager
> > @@ -35,3 +35,13 @@ Description: Read fpga manager state as a string.
> >                 * write complete        = Doing post programming steps
> >                 * write complete error  = Error while doing post programming
> >                 * operating             = FPGA is programmed and operating
> > +
> > +What:          /sys/class/fpga_manager/<fpga>/status
> > +Date:          June 2017
> > +KernelVersion: 4.12
> > +Contact:       Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > +Description:   Read fpga manager status as a string.
> > +               If FPGA programming operation fails, it could be due to crc
> > +               error or incompatible bitstream image. The intent of this
> > +               interface is to provide more detailed information for FPGA
> > +               programming errors to userspace.
> > diff --git a/drivers/fpga/fpga-mgr.c b/drivers/fpga/fpga-mgr.c
> > index be13cce..2485658 100644
> > --- a/drivers/fpga/fpga-mgr.c
> > +++ b/drivers/fpga/fpga-mgr.c
> > @@ -388,12 +388,36 @@ static ssize_t state_show(struct device *dev,
> >         return sprintf(buf, "%s\n", state_str[mgr->state]);
> >  }
> >
> > +static ssize_t status_show(struct device *dev,
> > +                          struct device_attribute *attr, char *buf)
> > +{
> > +       struct fpga_manager *mgr = to_fpga_manager(dev);
> > +       int len = 0;
> > +
> > +       if (mgr->status & FPGA_MGR_STATUS_OPERATION_ERR)
> > +               len += sprintf(buf + len, "reconfig operation error\n");
> > +       if (mgr->status & FPGA_MGR_STATUS_CRC_ERR)
> > +               len += sprintf(buf + len, "reconfig crc error\n");
> > +       if (mgr->status & FPGA_MGR_STATUS_INCOMPATIBLE_BS_ERR)
> > +               len += sprintf(buf + len, "reconfig incompatible BS error\n");
> > +       if (mgr->status & FPGA_MGR_STATUS_IP_PROTOCOL_ERR)
> > +               len += sprintf(buf + len, "reconfig IP protocol error\n");
> > +       if (mgr->status & FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR)
> > +               len += sprintf(buf + len, "reconfig fifo overflow error\n");
> > +       if (mgr->status & FPGA_MGR_STATUS_SECURE_LOAD_ERR)
> > +               len += sprintf(buf + len, "reconfig secure load error\n");
> > +
> > +       return len;
> > +}
> > +
> >  static DEVICE_ATTR_RO(name);
> >  static DEVICE_ATTR_RO(state);
> > +static DEVICE_ATTR_RO(status);
> >
> >  static struct attribute *fpga_mgr_attrs[] = {
> >         &dev_attr_name.attr,
> >         &dev_attr_state.attr,
> > +       &dev_attr_status.attr,
> >         NULL,
> >  };
> >  ATTRIBUTE_GROUPS(fpga_mgr);
> > diff --git a/include/linux/fpga/fpga-mgr.h b/include/linux/fpga/fpga-mgr.h
> > index b222a57..8cb42ac 100644
> > --- a/include/linux/fpga/fpga-mgr.h
> > +++ b/include/linux/fpga/fpga-mgr.h
> > @@ -128,6 +128,14 @@ struct fpga_manager_ops {
> >         void (*fpga_remove)(struct fpga_manager *mgr);
> >  };
> >
> > +/* FPGA manager status: Partial/Full Reconfiguration errors */
> > +#define FPGA_MGR_STATUS_OPERATION_ERR          BIT(0)
> > +#define FPGA_MGR_STATUS_CRC_ERR                        BIT(1)
> > +#define FPGA_MGR_STATUS_INCOMPATIBLE_BS_ERR    BIT(2)
> 
> How about ..._INCOMPATIBLE_IMAGE_ERR? :)

Hm.. It sounds better to me. : )
Will change it in next version patch set.

> 
> > +#define FPGA_MGR_STATUS_IP_PROTOCOL_ERR                BIT(3)
> > +#define FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR      BIT(4)
> > +#define FPGA_MGR_STATUS_SECURE_LOAD_ERR                BIT(5)
> > +
> >  /**
> >   * struct fpga_manager - fpga manager structure
> >   * @name: name of low level fpga manager
> > @@ -142,6 +150,7 @@ struct fpga_manager {
> >         struct device dev;
> >         struct mutex ref_mutex;
> >         enum fpga_mgr_states state;
> > +       u64 status;
> 
> With this implementation, the low level driver sets ops->status and
> that could be stale by the time the framework looks at it.  I suggest
> adding a function to fpga_manager_ops that returns the status.  That
> way whenever the status is requested, the low level driver will have
> the opportunity to read status registers.
> 
> Besides this, this new sysfs looks helpful.

I get your point, then I will add one member to fpga_manager_ops.

@@ -118,6 +118,7 @@ struct fpga_image_info {
 struct fpga_manager_ops {
        size_t initial_header_size;
        enum fpga_mgr_states (*state)(struct fpga_manager *mgr);
+       u64 (*status)(struct fpga_manager *mgr);
        int (*write_init)(struct fpga_manager *mgr,
                          struct fpga_image_info *info,
                          const char *buf, size_t count);


and add common action to get status (pr error) in complete function.

@@ -139,6 +139,8 @@ static int fpga_mgr_write_complete(struct fpga_manager *mgr,
        if (ret) {
                dev_err(&mgr->dev, "Error after writing image data to FPGA\n");
                mgr->state = FPGA_MGR_STATE_WRITE_COMPLETE_ERR;
+               if (mgr->mops->status)
+                       mgr->status = mgr->mops->status(mgr);
                return ret;
        }
        mgr->state = FPGA_MGR_STATE_OPERATING;

after status is updated, then user could read it from the sysfs interface.

How do you think about this?

Thanks
Hao

Re: [PATCH v2 01/22] docs: fpga: add a document for Intel FPGA driver overview

[Wu Hao]

On Wed, Jul 12, 2017 at 09:51:32AM -0500, Alan Tull wrote:
> On Sun, Jun 25, 2017 at 8:51 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> 
> Hi Hao,
> 
> > Add a document for Intel FPGA driver overview.
> >
> > Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> > Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > ----
> > v2: added FME fpga-mgr/bridge/region platform driver to driver organization.
> >     updated open discussion per current implementation.
> >     fixed some typos.
> > ---
> >  Documentation/fpga/intel-fpga.txt | 256 ++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 256 insertions(+)
> >  create mode 100644 Documentation/fpga/intel-fpga.txt
> >
> > diff --git a/Documentation/fpga/intel-fpga.txt b/Documentation/fpga/intel-fpga.txt
> > new file mode 100644
> > index 0000000..4a29470
> > --- /dev/null
> > +++ b/Documentation/fpga/intel-fpga.txt
> > @@ -0,0 +1,256 @@
> > +===============================================================================
> > +                    Intel FPGA driver Overview
> > +-------------------------------------------------------------------------------
> > +                Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > +                Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> > +                Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > +
> > +The Intel FPGA driver provides interfaces for userspace applications to
> > +configure, enumerate, open, and access FPGA accelerators on platforms equipped
> > +with Intel(R) FPGA PCIe based solutions and enables system level management
> > +functions such as FPGA reconfiguration, power management, and virtualization.
> > +
> > +HW Architecture
> > +===============
> > +From the OS's point of view, the FPGA hardware appears as a regular PCIe device.
> > +The FPGA device memory is organized using a predefined data structure (Device
> > +Feature List). Features supported by the particular FPGA device are exposed
> > +through these data structures, as illustrated below:
> > +
> > +  +-------------------------------+  +-------------+
> > +  |              PF               |  |     VF      |
> > +  +-------------------------------+  +-------------+
> > +      ^            ^         ^              ^
> > +      |            |         |              |
> > ++-----|------------|---------|--------------|-------+
> > +|     |            |         |              |       |
> > +|  +-----+     +-------+ +-------+      +-------+   |
> > +|  | FME |     | Port0 | | Port1 |      | Port2 |   |
> > +|  +-----+     +-------+ +-------+      +-------+   |
> > +|                  ^         ^              ^       |
> > +|                  |         |              |       |
> > +|              +-------+ +------+       +-------+   |
> > +|              |  AFU  | |  AFU |       |  AFU  |   |
> > +|              +-------+ +------+       +-------+   |
> > +|                                                   |
> > +|                 FPGA PCIe Device                  |
> > ++---------------------------------------------------+
> > +
> > +The driver supports PCIe SR-IOV to create virtual functions (VFs) which can be
> > +used to assign individual accelerators to virtual machines.
> > +
> > +FME (FPGA Management Engine)
> > +============================
> > +The FPGA Management Engine performs power and thermal management, error
> > +reporting, reconfiguration, performance reporting, and other infrastructure
> > +functions. Each FPGA has one FME, which is always accessed through the physical
> > +function (PF).
> > +
> > +User-space applications can acquire exclusive access to the FME using open(),
> > +and release it using close().
> > +
> > +The following functions are exposed through ioctls:
> > +
> > +       Get driver API version (FPGA_GET_API_VERSION)
> > +       Check for extensions (FPGA_CHECK_EXTENSION)
> > +       Assign port to PF (FPGA_FME_PORT_ASSIGN)
> > +       Release port from PF (FPGA_FME_PORT_RELEASE)
> > +       Program bitstream (FPGA_FME_PORT_PR)
> > +
> 
> I was hoping the API mailing list might have an opinion about this,
> but I think adding ioctls to the kernel is discouraged.  Could these
> be sysfs?

Hi Alan,

As you see below, we have defined a lot of sysfs interface for device
info, attributes and simple control operations. But for some actions
which requires complex inputs/outputs parameters (e.g a struct with
multiple items) with userspace, ioctls are used. I feel in such cases,
ioctls seem more suitable than sysfs.

> 
> > +More functions are exposed through sysfs
> > +(/sys/class/fpga/fpga.n/intel-fpga-fme.n/):
> > +
> > +       Read bitstream ID (bitstream_id)
> > +       Read bitstream metadata (bitstream_metadata)
> > +       Read number of ports (ports_num)
> > +       Read socket ID (socket_id)
> > +       Read performance counters (perf/)
> > +       Power management (power_mgmt/)
> > +       Thermal management (thermal_mgmt/)
> > +       Error reporting (errors/)
> > +
> > +PORT
> > +====
> > +A port represents the interface between the static FPGA fabric (the "blue
> > +bitstream") and a partially reconfigurable region containing an AFU (the "green
> > +bitstream"). It controls the communication from SW to the accelerator and
> > +exposes features such as reset and debug.
> > +
> > +A PCIe device may have several ports and each port can be released from PF by
> > +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe sriov
> > +sysfs interface.
> > +
> > +AFU
> > +===
> > +An AFU is attached to a port and exposes a 256k MMIO region to be used for
> > +accelerator-specific control registers.
> > +
> > +User-space applications can acquire exclusive access to an AFU attached to a
> > +port by using open() on the port device node, and release it using close().
> > +
> > +The following functions are exposed through ioctls:
> > +
> > +       Get driver API version (FPGA_GET_API_VERSION)
> > +       Check for extensions (FPGA_CHECK_EXTENSION)
> > +       Get port info (FPGA_PORT_GET_INFO)
> > +       Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
> > +       Map DMA buffer (FPGA_PORT_DMA_MAP)
> > +       Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
> > +       Reset AFU (FPGA_PORT_RESET)
> > +       Enable UMsg (FPGA_PORT_UMSG_ENABLE)
> > +       Disable UMsg (FPGA_PORT_UMSG_DISABLE)
> > +       Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
> > +       Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)
> > +
> > +User-space applications can also mmap() accelerator MMIO regions.
> > +
> > +More functions are exposed through sysfs:
> > +(/sys/class/fpga/fpga.n/intel-fpga-port.m/):
> > +
> > +       Read Accelerator GUID (afu_id)
> > +       Error reporting (errors/)
> > +
> > +Partial Reconfiguration
> > +=======================
> > +As mentioned above, accelerators can be reconfigured through partial
> > +reconfiguration of a green bitstream file (GBS). The green bitstream must have
> > +been generated for the exact blue bitstream and targeted reconfigurable region
> > +(port) of the FPGA; otherwise, the reconfiguration operation will fail and
> > +possibly cause system instability. This compatibility can be checked by
> > +comparing the interface ID noted in the GBS header against the interface ID
> > +exposed by the FME through sysfs (see above). This check is usually done by
> > +user-space before calling the reconfiguration IOCTL.
> > +
> > +FPGA virtualization
> > +===================
> > +To enable accessing an accelerator from applications running in a VM, the
> > +respective AFU's port needs to be assigned to a VF using the following steps:
> > +
> > + a) The PF owns all AFU ports by default. Any port that needs to be reassigned
> > + to a VF must be released from PF firstly through the FPGA_FME_PORT_RELEASE
> > + ioctl on the FME device.
> > +
> > + b) Once N ports are released from PF, then user can use below command to
> > + enable SRIOV and VFs. Each VF owns only one Port with AFU.
> > +
> > + echo N > $PCI_DEVICE_PATH/sriov_numvfs
> > +
> > + c) Pass through the VFs to VMs
> > +
> > + d) The AFU under VF is accessible from applications in VM (using the same
> > + driver inside the VF).
> > +
> > +Note the an FME can't be assigned to a VF, thus PR and other management
> > +functions are only available via the PF.
> > +
> > +
> > +Driver organization
> > +===================
> > +
> > +  +-------++------++------+             |
> > +  | FME   || FME  || FME  |             |
> > +  | FPGA  || FPGA || FPGA |             |
> > +  |Manager||Bridge||Region|             |
> > +  +-------++------++------+             |
> > +  +-----------------------+  +--------+ |             +--------+
> > +  |          FME          |  |  AFU   | |             |  AFU   |
> > +  |         Module        |  | Module | |             | Module |
> > +  +-----------------------+  +--------+ |             +--------+
> > +        +-----------------------+       |      +-----------------------+
> > +        | FPGA Container Device |       |      | FPGA Container Device |
> > +        +-----------------------+       |      +-----------------------+
> > +          +------------------+          |         +------------------+
> > +          | FPGA PCIE Module |          | Virtual | FPGA PCIE Module |
> > +          +------------------+   Host   | Machine +------------------+
> > + -------------------------------------- | ------------------------------
> > +           +---------------+            |          +---------------+
> > +           | PCI PF Device |            |          | PCI VF Device |
> > +           +---------------+            |          +---------------+
> > +
> > +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe device
> > +driver is always loaded first once a FPGA PCIE PF or VF device is detected. This
> > +driver plays an infrastructural role in the driver architecture.  It:
> > +
> > +       a) creates FPGA container device as parent of the feature devices.
> > +       b) walks through the Device Feature List, which is implemented in PCIE
> > +          device BAR memory, to discover feature devices and their sub features
> > +          and create platform device for them under the container device.
> > +       c) supports SRIOV.
> > +       d) introduces the feature device infrastructure, which abstracts
> > +          operations for sub features and exposes common functions to feature
> > +          device drivers.
> > +
> > +The FPGA Management Engine (FME) driver is a platform driver which is loaded
> > +automatically after FME platform device creation from the PCIE driver. It
> > +provides the key features for FPGA management, including:
> > +
> > +       a) Power and thermal management, error reporting, performance reporting
> > +          and other infrastructure functions. Users can access these functions
> > +          via sysfs interfaces exposed by FME driver.
> > +       b) Partial Reconfiguration. The FME driver creates platform devices
> > +          for FPGA manager, FPGA bridges and FPGA regions during PR sub
> > +          feature initialization; Once it receives an FPGA_FME_PORT_PR ioctl
> > +          from user, it invokes the common interface function from FPGA Region
> > +          to complete the partial reconfiguration of the bitstream to the given
> > +          port.
> > +       c) Port management for virtualization. The FME driver introduces two
> > +          ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
> > +          FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the port is
> > +          released from the PF, it can be assigned to the VF through the SRIOV
> > +          interfaces provided by PCIE driver. (Refer to "FPGA virtualization"
> > +          for more details).
> > +
> > +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) driver
> > +is probed once the AFU platform device is created. The main function of this
> > +module is to provide an interface for userspace applications to access the
> > +individual accelerators, including basic reset control on port, AFU MMIO region
> > +export, dma buffer mapping service, UMsg notification, and remote debug
> > +functions (see above).
> > +
> > +
> > +Device enumeration
> > +==================
> > +This section introduces how applications enumerate the fpga device from
> > +the sysfs hierarchy under /sys/class/fpga.
> > +
> > +In the example below, two Intel(R) FPGA devices are installed in the host. Each
> > +fpga device has one FME and two ports (AFUs).
> > +
> > +For each FPGA device, a device director is created under /sys/class/fpga/:
> > +
> > +       /sys/class/fpga/fpga.0
> > +       /sys/class/fpga/fpga.1
> > +
> > +The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's name.
> > +Application can retrieve name information via the sysfs interface:
> > +
> > +       /sys/class/fpga/fpga.0/name
> > +
> > +Each node has one FME and two ports (AFUs) as child devices:
> > +
> > +       /sys/class/fpga/fpga.0/intel-fpga-fme.0
> > +       /sys/class/fpga/fpga.0/intel-fpga-port.0
> > +       /sys/class/fpga/fpga.0/intel-fpga-port.1
> > +
> > +       /sys/class/fpga/fpga.1/intel-fpga-fme.1
> > +       /sys/class/fpga/fpga.1/intel-fpga-port.2
> > +       /sys/class/fpga/fpga.1/intel-fpga-port.3
> > +
> > +In general, the FME/AFU sysfs interfaces are named as follows:
> > +
> > +       /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/
> > +       /sys/class/fpga/<fpga.n>/<intel-fpga-port.m>/
> > +
> > +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
> > +ports.
> > +
> > +The device nodes used for ioctl() or mmap() can be referenced through:
> > +
> > +       /sys/class/fpga/<fpga.n>/<intel-fpga-port.n>/dev
> > +       /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/dev
> > +
> > +Open discussion
> > +===============
> > +FME driver exports one ioctl (FPGA_FME_PORT_PR) for partial reconfiguration to
> > +user now. In the future, if unified user interfaces for reconfiguration are
> > +added, FME driver should switch to them from ioctl interface.
> 
> Adding an ioctl that will go away in the future?

Currently there is no common userspace interfaces for PR, so Intel FPGA
device driver implemented its own solution by adding one ioctl for now.
Do you think if this is acceptable for now?

As mentioned in 'open discussion', if common interfaces are introduced in
the future, we can adpat this driver for that, and encourage users to use
common interfaces instead. But if ioctl is used widely at that time, then
we may need to keep this ioctl for compatibility.

> I think we are going to have to find a different interface for doing
> partial reconfiguration.  There was a recent conversation on the
> mailing list regarding adding a header to the raw bitstream and
> probably using sysfs to present that to the kernel.  I'm working on
> getting an RFC together for that.

Great to know this. Looking forward to see that. : )

Thanks
Hao

Re: [PATCH v2 07/22] fpga: intel: pcie: parse feature list and create platform device for features.

[Alan Tull]

On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> From: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
>
> Device Feature List structure creates a link list of feature headers
> within the MMIO space to provide an extensible way of adding features.
>
> The Intel FPGA PCIe driver walks through the feature headers to enumerate
> feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
> Function Unit (AFU), and their private sub features. For feature devices,
> it creates the platform devices and linked the private sub features into
> their platform data.
>
> Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Zhang Yi <yi.z.zhang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> ---
> v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
>     switched to GPLv2 license.
>     fixed comments from Moritz Fischer.
>     fixed kbuild warning, typos and clean up the code.
> ---
>  drivers/fpga/Makefile            |   2 +-
>  drivers/fpga/intel-feature-dev.c | 130 ++++++
>  drivers/fpga/intel-feature-dev.h | 341 ++++++++++++++++
>  drivers/fpga/intel-pcie.c        | 841 ++++++++++++++++++++++++++++++++++++++-
>  4 files changed, 1311 insertions(+), 3 deletions(-)
>  create mode 100644 drivers/fpga/intel-feature-dev.c
>  create mode 100644 drivers/fpga/intel-feature-dev.h
>
> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> index 5613133..ad24b3d 100644
> --- a/drivers/fpga/Makefile
> +++ b/drivers/fpga/Makefile
> @@ -31,4 +31,4 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
>  # Intel FPGA Support
>  obj-$(CONFIG_INTEL_FPGA_PCI)           += intel-fpga-pci.o
>
> -intel-fpga-pci-objs := intel-pcie.o
> +intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
> diff --git a/drivers/fpga/intel-feature-dev.c b/drivers/fpga/intel-feature-dev.c
> new file mode 100644
> index 0000000..68f9cba
> --- /dev/null
> +++ b/drivers/fpga/intel-feature-dev.c
> @@ -0,0 +1,130 @@
> +/*
> + * Intel FPGA Feature Device Driver
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *   Zhang Yi <yi.z.zhang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2. See
> + * the COPYING file in the top-level directory.
> + */
> +
> +#include "intel-feature-dev.h"
> +
> +void feature_platform_data_add(struct feature_platform_data *pdata,
> +                              int index, const char *name,
> +                              int resource_index, void __iomem *ioaddr)
> +{
> +       WARN_ON(index >= pdata->num);
> +
> +       pdata->features[index].name = name;
> +       pdata->features[index].resource_index = resource_index;
> +       pdata->features[index].ioaddr = ioaddr;
> +}
> +
> +struct feature_platform_data *
> +feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
> +{
> +       struct feature_platform_data *pdata;
> +
> +       pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
> +       if (pdata) {
> +               pdata->dev = dev;
> +               pdata->num = num;
> +               mutex_init(&pdata->lock);
> +       }
> +
> +       return pdata;
> +}
> +
> +int fme_feature_num(void)
> +{
> +       return FME_FEATURE_ID_MAX;
> +}
> +
> +int port_feature_num(void)
> +{
> +       return PORT_FEATURE_ID_MAX;
> +}
> +
> +int fpga_port_id(struct platform_device *pdev)
> +{
> +       struct feature_port_header *port_hdr;
> +       struct feature_port_capability capability;
> +
> +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> +                                              PORT_FEATURE_ID_HEADER);
> +       WARN_ON(!port_hdr);
> +
> +       capability.csr = readq(&port_hdr->capability);
> +       return capability.port_number;
> +}
> +EXPORT_SYMBOL_GPL(fpga_port_id);
> +
> +/*
> + * Enable Port by clear the port soft reset bit, which is set by default.
> + * The User AFU is unable to respond to any MMIO access while in reset.
> + * __fpga_port_enable function should only be used after __fpga_port_disable
> + * function.
> + */
> +void __fpga_port_enable(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct feature_port_header *port_hdr;
> +       struct feature_port_control control;
> +
> +       WARN_ON(!pdata->disable_count);
> +
> +       if (--pdata->disable_count != 0)
> +               return;
> +
> +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> +                                              PORT_FEATURE_ID_HEADER);
> +       WARN_ON(!port_hdr);
> +
> +       control.csr = readq(&port_hdr->control);
> +       control.port_sftrst = 0x0;
> +       writeq(control.csr, &port_hdr->control);
> +}
> +EXPORT_SYMBOL_GPL(__fpga_port_enable);
> +
> +#define RST_POLL_INVL 10 /* us */
> +#define RST_POLL_TIMEOUT 1000 /* us */
> +
> +int __fpga_port_disable(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct feature_port_header *port_hdr;
> +       struct feature_port_control control;
> +
> +       if (pdata->disable_count++ != 0)
> +               return 0;
> +
> +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> +                                              PORT_FEATURE_ID_HEADER);
> +       WARN_ON(!port_hdr);
> +
> +       /* Set port soft reset */
> +       control.csr = readq(&port_hdr->control);
> +       control.port_sftrst = 0x1;
> +       writeq(control.csr, &port_hdr->control);
> +
> +       /*
> +        * HW sets ack bit to 1 when all outstanding requests have been drained
> +        * on this port and minimum soft reset pulse width has elapsed.
> +        * Driver polls port_soft_reset_ack to determine if reset done by HW.
> +        */
> +       if (readq_poll_timeout(&port_hdr->control, control.csr,
> +                              (control.port_sftrst_ack == 1),
> +                              RST_POLL_INVL, RST_POLL_TIMEOUT)) {
> +               dev_err(&pdev->dev, "timeout, fail to reset device\n");
> +               return -ETIMEDOUT;
> +       }
> +
> +       return 0;
> +}
> +EXPORT_SYMBOL_GPL(__fpga_port_disable);
> diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
> new file mode 100644
> index 0000000..f67784a
> --- /dev/null
> +++ b/drivers/fpga/intel-feature-dev.h
> @@ -0,0 +1,341 @@
> +/*
> + * Intel FPGA Feature Device Driver Header File
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *   Zhang Yi <yi.z.zhang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2. See
> + * the COPYING file in the top-level directory.
> + */
> +
> +#ifndef __INTEL_FPGA_FEATURE_H
> +#define __INTEL_FPGA_FEATURE_H
> +
> +#include <linux/fs.h>
> +#include <linux/pci.h>
> +#include <linux/uuid.h>
> +#include <linux/delay.h>
> +#include <linux/iopoll.h>
> +#include <linux/platform_device.h>
> +
> +#ifndef readq
> +static inline u64 readq(void __iomem *addr)
> +{
> +       return readl(addr) + ((u64)readl(addr + 4) << 32);
> +}
> +#endif
> +
> +#ifndef writeq
> +static inline void writeq(u64 val, void __iomem *addr)
> +{
> +       writel((u32) (val), addr);
> +       writel((u32) (val >> 32), (addr + 4));
> +}
> +#endif
> +
> +/* maximum supported number of ports */
> +#define MAX_FPGA_PORT_NUM 4
> +/* plus one for fme device */
> +#define MAX_FEATURE_DEV_NUM    (MAX_FPGA_PORT_NUM + 1)
> +
> +#define FME_FEATURE_HEADER          "fme_hdr"
> +#define FME_FEATURE_THERMAL_MGMT    "fme_thermal"
> +#define FME_FEATURE_POWER_MGMT      "fme_power"
> +#define FME_FEATURE_GLOBAL_PERF     "fme_gperf"
> +#define FME_FEATURE_GLOBAL_ERR      "fme_error"
> +#define FME_FEATURE_PR_MGMT         "fme_pr"
> +
> +#define PORT_FEATURE_HEADER         "port_hdr"
> +#define PORT_FEATURE_UAFU           "port_uafu"
> +#define PORT_FEATURE_ERR            "port_err"
> +#define PORT_FEATURE_UMSG           "port_umsg"
> +#define PORT_FEATURE_PR             "port_pr"
> +#define PORT_FEATURE_STP            "port_stp"
> +
> +/* All headers and structures must be byte-packed to match the spec. */
> +#pragma pack(1)
> +
> +/* common header for all features */
> +struct feature_header {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u64 id:12;
> +                       u64 revision:4;
> +                       u64 next_header_offset:24; /* offset to next header */
> +                       u64 rsvdz:20;
> +                       u64 type:4;                /* feature type */
> +#define FEATURE_TYPE_AFU               0x1
> +#define FEATURE_TYPE_PRIVATE           0x3
> +               };
> +       };
> +};

Hi Hao,

Aren't these union's introducing a portability problem?  Can you use
regmap instead?

Alan

> +
> +/* common header for non-private features */
> +struct feature_afu_header {
> +       uuid_le guid;
> +       union {
> +               u64 csr;
> +               struct {
> +                       u64 next_afu:24;        /* pointer to next afu header */
> +                       u64 rsvdz:40;
> +               };
> +       };
> +};
> +
> +/* FME Header Register Set */
> +/* FME Capability Register */
> +struct feature_fme_capability {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u64 fabric_verid:8;     /* Fabric version ID */
> +                       u64 socket_id:1;        /* Socket id */
> +                       u64 rsvdz1:3;
> +                       u64 pcie0_link_avl:1;   /* PCIe0 link availability */
> +                       u64 pcie1_link_avl:1;   /* PCIe1 link availability */
> +                       u64 coherent_link_avl:1;/* Coherent link availability */
> +                       u64 rsvdz2:1;
> +                       u64 iommu_support:1;    /* IOMMU or VT-d supported */
> +                       u64 num_ports:3;        /* Num of ports implemented */
> +                       u64 rsvdz3:4;
> +                       u64 addr_width_bits:6;  /* Address width supported */
> +                       u64 rsvdz4:2;
> +                       u64 cache_size:12;      /* Cache size in kb */
> +                       u64 cache_assoc:4;      /* Cache Associativity */
> +                       u64 rsvdz5:15;
> +                       u64 lock_bit:1;         /* Latched lock bit by BIOS */
> +               };
> +       };
> +};
> +
> +/* FME Port Offset Register */
> +struct feature_fme_port {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u64 port_offset:24;     /* Offset to port header */
> +                       u64 rsvdz1:8;
> +                       u64 port_bar:3;         /* Bar id */
> +                       u64 rsvdz2:20;
> +                       u64 afu_access_ctrl:1;  /* AFU access type: PF/VF */
> +                       u64 rsvdz3:4;
> +                       u64 port_implemented:1; /* Port implemented or not */
> +                       u64 rsvdz4:3;
> +               };
> +       };
> +};
> +
> +struct feature_fme_header {
> +       struct feature_header header;
> +       struct feature_afu_header afu_header;
> +       u64 rsvd[2];
> +       struct feature_fme_capability capability;
> +       struct feature_fme_port port[MAX_FPGA_PORT_NUM];
> +};
> +
> +/* FME Thermal Sub Feature Register Set */
> +struct feature_fme_thermal {
> +       struct feature_header header;
> +};
> +
> +/* FME Power Sub Feature Register Set */
> +struct feature_fme_power {
> +       struct feature_header header;
> +};
> +
> +/* FME Global Performance Sub Feature Register Set */
> +struct feature_fme_gperf {
> +       struct feature_header header;
> +};
> +
> +/* FME Error Sub Feature Register Set */
> +struct feature_fme_err {
> +       struct feature_header header;
> +};
> +
> +/* FME Partial Reconfiguration Sub Feature Register Set */
> +struct feature_fme_pr {
> +       struct feature_header header;
> +};
> +
> +/* PORT Header Register Set */
> +/* Port Capability Register */
> +struct feature_port_capability {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u64 port_number:2;      /* Port Number 0-3 */
> +                       u64 rsvdz1:6;
> +                       u64 mmio_size:16;       /* User MMIO size in KB */
> +                       u64 rsvdz2:8;
> +                       u64 sp_intr_num:4;      /* Supported interrupts num */
> +                       u64 rsvdz3:28;
> +               };
> +       };
> +};
> +
> +/* Port Control Register */
> +struct feature_port_control {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u64 port_sftrst:1;      /* Port Soft Reset */
> +                       u64 rsvdz1:1;
> +                       u64 latency_tolerance:1;/* '1' >= 40us, '0' < 40us */
> +                       u64 rsvdz2:1;
> +                       u64 port_sftrst_ack:1;  /* HW ACK for Soft Reset */
> +                       u64 rsvdz3:59;
> +               };
> +       };
> +};
> +
> +struct feature_port_header {
> +       struct feature_header header;
> +       struct feature_afu_header afu_header;
> +       u64 rsvd[2];
> +       struct feature_port_capability capability;
> +       struct feature_port_control control;
> +};
> +
> +/* PORT Error Sub Feature Register Set */
> +struct feature_port_error {
> +       struct feature_header header;
> +};
> +
> +/* PORT Unordered Message Sub Feature Register Set */
> +struct feature_port_umsg {
> +       struct feature_header header;
> +};
> +
> +/* PORT SignalTap Sub Feature Register Set */
> +struct feature_port_stp {
> +       struct feature_header header;
> +};
> +
> +#pragma pack()
> +
> +struct feature {
> +       const char *name;
> +       int resource_index;
> +       void __iomem *ioaddr;
> +};
> +
> +struct feature_platform_data {
> +       /* list the feature dev to cci_drvdata->port_dev_list. */
> +       struct list_head node;
> +       struct mutex lock;
> +       struct platform_device *dev;
> +       unsigned int disable_count;     /* count for port disable */
> +
> +       int num;                        /* number of features */
> +       struct feature features[0];
> +};
> +
> +enum fme_feature_id {
> +       FME_FEATURE_ID_HEADER = 0x0,
> +       FME_FEATURE_ID_THERMAL_MGMT = 0x1,
> +       FME_FEATURE_ID_POWER_MGMT = 0x2,
> +       FME_FEATURE_ID_GLOBAL_PERF = 0x3,
> +       FME_FEATURE_ID_GLOBAL_ERR = 0x4,
> +       FME_FEATURE_ID_PR_MGMT = 0x5,
> +       FME_FEATURE_ID_MAX = 0x6,
> +};
> +
> +enum port_feature_id {
> +       PORT_FEATURE_ID_HEADER = 0x0,
> +       PORT_FEATURE_ID_ERROR = 0x1,
> +       PORT_FEATURE_ID_UMSG = 0x2,
> +       PORT_FEATURE_ID_PR = 0x3,
> +       PORT_FEATURE_ID_STP = 0x4,
> +       PORT_FEATURE_ID_UAFU = 0x5,
> +       PORT_FEATURE_ID_MAX = 0x6,
> +};
> +
> +int fme_feature_num(void);
> +int port_feature_num(void);
> +
> +#define FPGA_FEATURE_DEV_FME           "intel-fpga-fme"
> +#define FPGA_FEATURE_DEV_PORT          "intel-fpga-port"
> +
> +void feature_platform_data_add(struct feature_platform_data *pdata,
> +                              int index, const char *name,
> +                              int resource_index, void __iomem *ioaddr);
> +
> +static inline int feature_platform_data_size(const int num)
> +{
> +       return sizeof(struct feature_platform_data) +
> +               num * sizeof(struct feature);
> +}
> +
> +struct feature_platform_data *
> +feature_platform_data_alloc_and_init(struct platform_device *dev, int num);
> +
> +int fpga_port_id(struct platform_device *pdev);
> +
> +static inline int fpga_port_check_id(struct platform_device *pdev,
> +                                    void *pport_id)
> +{
> +       return fpga_port_id(pdev) == *(int *)pport_id;
> +}
> +
> +void __fpga_port_enable(struct platform_device *pdev);
> +int __fpga_port_disable(struct platform_device *pdev);
> +
> +static inline void fpga_port_enable(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +
> +       mutex_lock(&pdata->lock);
> +       __fpga_port_enable(pdev);
> +       mutex_unlock(&pdata->lock);
> +}
> +
> +static inline int fpga_port_disable(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       int ret;
> +
> +       mutex_lock(&pdata->lock);
> +       ret = __fpga_port_disable(pdev);
> +       mutex_unlock(&pdata->lock);
> +
> +       return ret;
> +}
> +
> +static inline int __fpga_port_reset(struct platform_device *pdev)
> +{
> +       int ret;
> +
> +       ret = __fpga_port_disable(pdev);
> +       if (ret)
> +               return ret;
> +
> +       __fpga_port_enable(pdev);
> +       return 0;
> +}
> +
> +static inline int fpga_port_reset(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       int ret;
> +
> +       mutex_lock(&pdata->lock);
> +       ret = __fpga_port_reset(pdev);
> +       mutex_unlock(&pdata->lock);
> +       return ret;
> +}
> +
> +static inline void __iomem *
> +get_feature_ioaddr_by_index(struct device *dev, int index)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(dev);
> +
> +       return pdata->features[index].ioaddr;
> +}
> +#endif
> diff --git a/drivers/fpga/intel-pcie.c b/drivers/fpga/intel-pcie.c
> index f697de4..70b8284 100644
> --- a/drivers/fpga/intel-pcie.c
> +++ b/drivers/fpga/intel-pcie.c
> @@ -23,10 +23,827 @@
>  #include <linux/stddef.h>
>  #include <linux/errno.h>
>  #include <linux/aer.h>
> +#include <linux/fpga/fpga-dev.h>
> +
> +#include "intel-feature-dev.h"
>
>  #define DRV_VERSION    "0.8"
>  #define DRV_NAME       "intel-fpga-pci"
>
> +#define INTEL_FPGA_DEV "intel-fpga-dev"
> +
> +static DEFINE_MUTEX(fpga_id_mutex);
> +
> +enum fpga_id_type {
> +       FME_ID,         /* fme id allocation and mapping */
> +       PORT_ID,        /* port id allocation and mapping */
> +       FPGA_ID_MAX,
> +};
> +
> +/* it is protected by fpga_id_mutex */
> +static struct idr fpga_ids[FPGA_ID_MAX];
> +
> +struct cci_drvdata {
> +       struct device *fme_dev;
> +
> +       struct mutex lock;
> +       struct list_head port_dev_list;
> +
> +       struct list_head regions; /* global list of pci bar mapping region */
> +};
> +
> +/* pci bar mapping info */
> +struct cci_pci_region {
> +       int bar;
> +       void __iomem *ioaddr;   /* pointer to mapped bar region */
> +       struct list_head node;
> +};
> +
> +static void fpga_ids_init(void)
> +{
> +       int i;
> +
> +       for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> +               idr_init(fpga_ids + i);
> +}
> +
> +static void fpga_ids_destroy(void)
> +{
> +       int i;
> +
> +       for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> +               idr_destroy(fpga_ids + i);
> +}
> +
> +static int alloc_fpga_id(enum fpga_id_type type, struct device *dev)
> +{
> +       int id;
> +
> +       WARN_ON(type >= FPGA_ID_MAX);
> +       mutex_lock(&fpga_id_mutex);
> +       id = idr_alloc(fpga_ids + type, dev, 0, 0, GFP_KERNEL);
> +       mutex_unlock(&fpga_id_mutex);
> +       return id;
> +}
> +
> +static void free_fpga_id(enum fpga_id_type type, int id)
> +{
> +       WARN_ON(type >= FPGA_ID_MAX);
> +       mutex_lock(&fpga_id_mutex);
> +       idr_remove(fpga_ids + type, id);
> +       mutex_unlock(&fpga_id_mutex);
> +}
> +
> +static void cci_pci_add_port_dev(struct pci_dev *pdev,
> +                                struct platform_device *port_dev)
> +{
> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +       struct feature_platform_data *pdata = dev_get_platdata(&port_dev->dev);
> +
> +       mutex_lock(&drvdata->lock);
> +       list_add(&pdata->node, &drvdata->port_dev_list);
> +       get_device(&pdata->dev->dev);
> +       mutex_unlock(&drvdata->lock);
> +}
> +
> +static void cci_pci_remove_port_devs(struct pci_dev *pdev)
> +{
> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +       struct feature_platform_data *pdata, *ptmp;
> +
> +       mutex_lock(&drvdata->lock);
> +       list_for_each_entry_safe(pdata, ptmp, &drvdata->port_dev_list, node) {
> +               struct platform_device *port_dev = pdata->dev;
> +
> +               /* the port should be unregistered first. */
> +               WARN_ON(device_is_registered(&port_dev->dev));
> +               list_del(&pdata->node);
> +               free_fpga_id(PORT_ID, port_dev->id);
> +               put_device(&port_dev->dev);
> +       }
> +       mutex_unlock(&drvdata->lock);
> +}
> +
> +/* info collection during feature dev build. */
> +struct build_feature_devs_info {
> +       struct pci_dev *pdev;
> +
> +       /*
> +        * PCI BAR mapping info. Parsing feature list starts from
> +        * BAR 0 and switch to different BARs to parse Port
> +        */
> +       void __iomem *ioaddr;
> +       void __iomem *ioend;
> +       int current_bar;
> +
> +       /* points to FME header where the port offset is figured out. */
> +       void __iomem *pfme_hdr;
> +
> +       /* the container device for all feature devices */
> +       struct fpga_dev *parent_dev;
> +
> +       /* current feature device */
> +       struct platform_device *feature_dev;
> +};
> +
> +static void cci_pci_release_regions(struct pci_dev *pdev)
> +{
> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +       struct cci_pci_region *tmp, *region;
> +
> +       list_for_each_entry_safe(region, tmp, &drvdata->regions, node) {
> +               list_del(&region->node);
> +               if (region->ioaddr)
> +                       pci_iounmap(pdev, region->ioaddr);
> +               devm_kfree(&pdev->dev, region);
> +       }
> +}
> +
> +static void __iomem *cci_pci_ioremap_bar(struct pci_dev *pdev, int bar)
> +{
> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +       struct cci_pci_region *region;
> +
> +       list_for_each_entry(region, &drvdata->regions, node)
> +               if (region->bar == bar) {
> +                       dev_dbg(&pdev->dev, "BAR %d region exists\n", bar);
> +                       return region->ioaddr;
> +               }
> +
> +       region = devm_kzalloc(&pdev->dev, sizeof(*region), GFP_KERNEL);
> +       if (!region)
> +               return NULL;
> +
> +       region->bar = bar;
> +       region->ioaddr = pci_ioremap_bar(pdev, bar);
> +       if (!region->ioaddr) {
> +               dev_err(&pdev->dev, "can't ioremap memory from BAR %d.\n", bar);
> +               devm_kfree(&pdev->dev, region);
> +               return NULL;
> +       }
> +
> +       list_add(&region->node, &drvdata->regions);
> +       return region->ioaddr;
> +}
> +
> +static int parse_start_from(struct build_feature_devs_info *binfo, int bar)
> +{
> +       binfo->ioaddr = cci_pci_ioremap_bar(binfo->pdev, bar);
> +       if (!binfo->ioaddr)
> +               return -ENOMEM;
> +
> +       binfo->current_bar = bar;
> +       binfo->ioend = binfo->ioaddr + pci_resource_len(binfo->pdev, bar);
> +       return 0;
> +}
> +
> +static int parse_start(struct build_feature_devs_info *binfo)
> +{
> +       /* fpga feature list starts from BAR 0 */
> +       return parse_start_from(binfo, 0);
> +}
> +
> +/* switch the memory mapping to BAR# @bar */
> +static int parse_switch_to(struct build_feature_devs_info *binfo, int bar)
> +{
> +       return parse_start_from(binfo, bar);
> +}
> +
> +static struct build_feature_devs_info *
> +build_info_alloc_and_init(struct pci_dev *pdev)
> +{
> +       struct build_feature_devs_info *binfo;
> +
> +       binfo = devm_kzalloc(&pdev->dev, sizeof(*binfo), GFP_KERNEL);
> +       if (binfo)
> +               binfo->pdev = pdev;
> +
> +       return binfo;
> +}
> +
> +static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
> +{
> +       if (!strcmp(pdev->name, FPGA_FEATURE_DEV_FME))
> +               return FME_ID;
> +
> +       if (!strcmp(pdev->name, FPGA_FEATURE_DEV_PORT))
> +               return PORT_ID;
> +
> +       WARN_ON(1);
> +       return FPGA_ID_MAX;
> +}
> +
> +/*
> + * register current feature device, it is called when we need to switch to
> + * another feature parsing or we have parsed all features
> + */
> +static int build_info_commit_dev(struct build_feature_devs_info *binfo)
> +{
> +       int ret;
> +
> +       if (!binfo->feature_dev)
> +               return 0;
> +
> +       ret = platform_device_add(binfo->feature_dev);
> +       if (!ret) {
> +               struct cci_drvdata *drvdata;
> +
> +               drvdata = dev_get_drvdata(&binfo->pdev->dev);
> +               if (feature_dev_id_type(binfo->feature_dev) == PORT_ID)
> +                       cci_pci_add_port_dev(binfo->pdev, binfo->feature_dev);
> +               else
> +                       drvdata->fme_dev = get_device(&binfo->feature_dev->dev);
> +
> +               /*
> +                * reset it to avoid build_info_free() freeing their resource.
> +                *
> +                * The resource of successfully registered feature devices
> +                * will be freed by platform_device_unregister(). See the
> +                * comments in build_info_create_dev().
> +                */
> +               binfo->feature_dev = NULL;
> +       }
> +
> +       return ret;
> +}
> +
> +static int
> +build_info_create_dev(struct build_feature_devs_info *binfo,
> +                     enum fpga_id_type type, int feature_nr, const char *name)
> +{
> +       struct platform_device *fdev;
> +       struct resource *res;
> +       struct feature_platform_data *pdata;
> +       int ret;
> +
> +       /* we will create a new device, commit current device first */
> +       ret = build_info_commit_dev(binfo);
> +       if (ret)
> +               return ret;
> +
> +       /*
> +        * we use -ENODEV as the initialization indicator which indicates
> +        * whether the id need to be reclaimed
> +        */
> +       fdev = binfo->feature_dev = platform_device_alloc(name, -ENODEV);
> +       if (!fdev)
> +               return -ENOMEM;
> +
> +       fdev->id = alloc_fpga_id(type, &fdev->dev);
> +       if (fdev->id < 0)
> +               return fdev->id;
> +
> +       fdev->dev.parent = &binfo->parent_dev->dev;
> +
> +       /*
> +        * we do not need to care for the memory which is associated with
> +        * the platform device. After calling platform_device_unregister(),
> +        * it will be automatically freed by device's release() callback,
> +        * platform_device_release().
> +        */
> +       pdata = feature_platform_data_alloc_and_init(fdev, feature_nr);
> +       if (!pdata)
> +               return -ENOMEM;
> +
> +       /*
> +        * the count should be initialized to 0 to make sure
> +        *__fpga_port_enable() following __fpga_port_disable()
> +        * works properly for port device.
> +        * and it should always be 0 for fme device.
> +        */
> +       WARN_ON(pdata->disable_count);
> +
> +       fdev->dev.platform_data = pdata;
> +       fdev->num_resources = feature_nr;
> +       fdev->resource = kcalloc(feature_nr, sizeof(*res), GFP_KERNEL);
> +       if (!fdev->resource)
> +               return -ENOMEM;
> +
> +       return 0;
> +}
> +
> +static int remove_feature_dev(struct device *dev, void *data)
> +{
> +       struct platform_device *pdev = to_platform_device(dev);
> +
> +       platform_device_unregister(pdev);
> +       return 0;
> +}
> +
> +static int remove_parent_dev(struct device *dev, void *data)
> +{
> +       /* remove platform devices attached in the parent device */
> +       device_for_each_child(dev, NULL, remove_feature_dev);
> +       fpga_dev_destroy(to_fpga_dev(dev));
> +       return 0;
> +}
> +
> +static void remove_all_devs(struct pci_dev *pdev)
> +{
> +       /* remove parent device and all its children. */
> +       device_for_each_child(&pdev->dev, NULL, remove_parent_dev);
> +}
> +
> +static void build_info_free(struct build_feature_devs_info *binfo)
> +{
> +       if (!IS_ERR_OR_NULL(binfo->parent_dev))
> +               remove_all_devs(binfo->pdev);
> +
> +       /*
> +        * it is a valid id, free it. See comments in
> +        * build_info_create_dev()
> +        */
> +       if (binfo->feature_dev && binfo->feature_dev->id >= 0)
> +               free_fpga_id(feature_dev_id_type(binfo->feature_dev),
> +                            binfo->feature_dev->id);
> +
> +       platform_device_put(binfo->feature_dev);
> +
> +       devm_kfree(&binfo->pdev->dev, binfo);
> +}
> +
> +#define FEATURE_TYPE_AFU       0x1
> +#define FEATURE_TYPE_PRIVATE   0x3
> +
> +/* FME and PORT GUID are fixed */
> +#define FEATURE_FME_GUID "f9e17764-38f0-82fe-e346-524ae92aafbf"
> +#define FEATURE_PORT_GUID "6b355b87-b06c-9642-eb42-8d139398b43a"
> +
> +static bool feature_is_fme(struct feature_afu_header *afu_hdr)
> +{
> +       uuid_le u;
> +
> +       uuid_le_to_bin(FEATURE_FME_GUID, &u);
> +
> +       return !uuid_le_cmp(u, afu_hdr->guid);
> +}
> +
> +static bool feature_is_port(struct feature_afu_header *afu_hdr)
> +{
> +       uuid_le u;
> +
> +       uuid_le_to_bin(FEATURE_PORT_GUID, &u);
> +
> +       return !uuid_le_cmp(u, afu_hdr->guid);
> +}
> +
> +/*
> + * UAFU GUID is dynamic as it can be changed after FME downloads different
> + * Green Bitstream to the port, so we treat the unknown GUIDs which are
> + * attached on port's feature list as UAFU.
> + */
> +static bool feature_is_UAFU(struct build_feature_devs_info *binfo)
> +{
> +       if (!binfo->feature_dev ||
> +             feature_dev_id_type(binfo->feature_dev) != PORT_ID)
> +               return false;
> +
> +       return true;
> +}
> +
> +static void
> +build_info_add_sub_feature(struct build_feature_devs_info *binfo,
> +                          int feature_id, const char *feature_name,
> +                          resource_size_t resource_size, void __iomem *start)
> +{
> +
> +       struct platform_device *fdev = binfo->feature_dev;
> +       struct feature_platform_data *pdata = dev_get_platdata(&fdev->dev);
> +       struct resource *res = &fdev->resource[feature_id];
> +
> +       res->start = pci_resource_start(binfo->pdev, binfo->current_bar) +
> +               start - binfo->ioaddr;
> +       res->end = res->start + resource_size - 1;
> +       res->flags = IORESOURCE_MEM;
> +       res->name = feature_name;
> +
> +       feature_platform_data_add(pdata, feature_id,
> +                                 feature_name, feature_id, start);
> +}
> +
> +struct feature_info {
> +       const char *name;
> +       resource_size_t resource_size;
> +       int feature_index;
> +};
> +
> +/* indexed by fme feature IDs which are defined in 'enum fme_feature_id'. */
> +static struct feature_info fme_features[] = {
> +       {
> +               .name = FME_FEATURE_HEADER,
> +               .resource_size = sizeof(struct feature_fme_header),
> +               .feature_index = FME_FEATURE_ID_HEADER,
> +       },
> +       {
> +               .name = FME_FEATURE_THERMAL_MGMT,
> +               .resource_size = sizeof(struct feature_fme_thermal),
> +               .feature_index = FME_FEATURE_ID_THERMAL_MGMT,
> +       },
> +       {
> +               .name = FME_FEATURE_POWER_MGMT,
> +               .resource_size = sizeof(struct feature_fme_power),
> +               .feature_index = FME_FEATURE_ID_POWER_MGMT,
> +       },
> +       {
> +               .name = FME_FEATURE_GLOBAL_PERF,
> +               .resource_size = sizeof(struct feature_fme_gperf),
> +               .feature_index = FME_FEATURE_ID_GLOBAL_PERF,
> +       },
> +       {
> +               .name = FME_FEATURE_GLOBAL_ERR,
> +               .resource_size = sizeof(struct feature_fme_err),
> +               .feature_index = FME_FEATURE_ID_GLOBAL_ERR,
> +       },
> +       {
> +               .name = FME_FEATURE_PR_MGMT,
> +               .resource_size = sizeof(struct feature_fme_pr),
> +               .feature_index = FME_FEATURE_ID_PR_MGMT,
> +       }
> +};
> +
> +/* indexed by port feature IDs which are defined in 'enum port_feature_id'. */
> +static struct feature_info port_features[] = {
> +       {
> +               .name = PORT_FEATURE_HEADER,
> +               .resource_size = sizeof(struct feature_port_header),
> +               .feature_index = PORT_FEATURE_ID_HEADER,
> +       },
> +       {
> +               .name = PORT_FEATURE_ERR,
> +               .resource_size = sizeof(struct feature_port_error),
> +               .feature_index = PORT_FEATURE_ID_ERROR,
> +       },
> +       {
> +               .name = PORT_FEATURE_UMSG,
> +               .resource_size = sizeof(struct feature_port_umsg),
> +               .feature_index = PORT_FEATURE_ID_UMSG,
> +       },
> +       {
> +               /* This feature isn't available for now */
> +               .name = PORT_FEATURE_PR,
> +               .resource_size = 0,
> +               .feature_index = PORT_FEATURE_ID_PR,
> +       },
> +       {
> +               .name = PORT_FEATURE_STP,
> +               .resource_size = sizeof(struct feature_port_stp),
> +               .feature_index = PORT_FEATURE_ID_STP,
> +       },
> +       {
> +               /*
> +                * For User AFU feature, its region size is not fixed, but
> +                * reported by register PortCapability.mmio_size. Resource
> +                * size of UAFU will be set while parse port device.
> +                */
> +               .name = PORT_FEATURE_UAFU,
> +               .resource_size = 0,
> +               .feature_index = PORT_FEATURE_ID_UAFU,
> +       },
> +};
> +
> +static int
> +create_feature_instance(struct build_feature_devs_info *binfo,
> +                       void __iomem *start, struct feature_info *finfo)
> +{
> +       if (binfo->ioend - start < finfo->resource_size)
> +               return -EINVAL;
> +
> +       build_info_add_sub_feature(binfo, finfo->feature_index, finfo->name,
> +                                  finfo->resource_size, start);
> +       return 0;
> +}
> +
> +static int parse_feature_fme(struct build_feature_devs_info *binfo,
> +                            void __iomem *start)
> +{
> +       struct cci_drvdata *drvdata = dev_get_drvdata(&binfo->pdev->dev);
> +       int ret;
> +
> +       ret = build_info_create_dev(binfo, FME_ID, fme_feature_num(),
> +                                       FPGA_FEATURE_DEV_FME);
> +       if (ret)
> +               return ret;
> +
> +       if (drvdata->fme_dev) {
> +               dev_err(&binfo->pdev->dev, "Multiple FMEs are detected.\n");
> +               return -EINVAL;
> +       }
> +
> +       return create_feature_instance(binfo, start,
> +                                      &fme_features[FME_FEATURE_ID_HEADER]);
> +}
> +
> +static int parse_feature_fme_private(struct build_feature_devs_info *binfo,
> +                                    struct feature_header *hdr)
> +{
> +       struct feature_header header;
> +
> +       header.csr = readq(hdr);
> +
> +       if (header.id >= ARRAY_SIZE(fme_features)) {
> +               dev_info(&binfo->pdev->dev, "FME feature id %x is not supported yet.\n",
> +                        header.id);
> +               return 0;
> +       }
> +
> +       return create_feature_instance(binfo, hdr, &fme_features[header.id]);
> +}
> +
> +static int parse_feature_port(struct build_feature_devs_info *binfo,
> +                            void __iomem *start)
> +{
> +       int ret;
> +
> +       ret = build_info_create_dev(binfo, PORT_ID, port_feature_num(),
> +                                       FPGA_FEATURE_DEV_PORT);
> +       if (ret)
> +               return ret;
> +
> +       return create_feature_instance(binfo, start,
> +                                      &port_features[PORT_FEATURE_ID_HEADER]);
> +}
> +
> +static void enable_port_uafu(struct build_feature_devs_info *binfo,
> +                            void __iomem *start)
> +{
> +       enum port_feature_id id = PORT_FEATURE_ID_UAFU;
> +       struct feature_port_header *port_hdr;
> +       struct feature_port_capability capability;
> +
> +       port_hdr = (struct feature_port_header *)start;
> +       capability.csr = readq(&port_hdr->capability);
> +       port_features[id].resource_size = capability.mmio_size << 10;
> +
> +       /*
> +        * To enable User AFU, driver needs to clear reset bit on related port,
> +        * otherwise the mmio space of this user AFU will be invalid.
> +        */
> +       if (port_features[id].resource_size)
> +               fpga_port_reset(binfo->feature_dev);
> +}
> +
> +static int parse_feature_port_private(struct build_feature_devs_info *binfo,
> +                                     struct feature_header *hdr)
> +{
> +       struct feature_header header;
> +       enum port_feature_id id;
> +
> +       header.csr = readq(hdr);
> +       /*
> +        * the region of port feature id is [0x10, 0x13], + 1 to reserve 0
> +        * which is dedicated for port-hdr.
> +        */
> +       id = (header.id & 0x000f) + 1;
> +
> +       if (id >= ARRAY_SIZE(port_features)) {
> +               dev_info(&binfo->pdev->dev, "Port feature id %x is not supported yet.\n",
> +                        header.id);
> +               return 0;
> +       }
> +
> +       return create_feature_instance(binfo, hdr, &port_features[id]);
> +}
> +
> +static int parse_feature_port_uafu(struct build_feature_devs_info *binfo,
> +                                struct feature_header *hdr)
> +{
> +       enum port_feature_id id = PORT_FEATURE_ID_UAFU;
> +       int ret;
> +
> +       if (port_features[id].resource_size) {
> +               ret = create_feature_instance(binfo, hdr, &port_features[id]);
> +               port_features[id].resource_size = 0;
> +       } else {
> +               dev_err(&binfo->pdev->dev, "the uafu feature header is mis-configured.\n");
> +               ret = -EINVAL;
> +       }
> +
> +       return ret;
> +}
> +
> +static int parse_feature_afus(struct build_feature_devs_info *binfo,
> +                             struct feature_header *hdr)
> +{
> +       int ret;
> +       struct feature_afu_header *afu_hdr, header;
> +       void __iomem *start;
> +       void __iomem *end = binfo->ioend;
> +
> +       start = hdr;
> +       for (; start < end; start += header.next_afu) {
> +               if (end - start < (sizeof(*afu_hdr) + sizeof(*hdr)))
> +                       return -EINVAL;
> +
> +               hdr = start;
> +               afu_hdr = (struct feature_afu_header *) (hdr + 1);
> +               header.csr = readq(&afu_hdr->csr);
> +
> +               if (feature_is_fme(afu_hdr)) {
> +                       ret = parse_feature_fme(binfo, hdr);
> +                       binfo->pfme_hdr = hdr;
> +                       if (ret)
> +                               return ret;
> +               } else if (feature_is_port(afu_hdr)) {
> +                       ret = parse_feature_port(binfo, hdr);
> +                       enable_port_uafu(binfo, hdr);
> +                       if (ret)
> +                               return ret;
> +               } else if (feature_is_UAFU(binfo)) {
> +                       ret = parse_feature_port_uafu(binfo, hdr);
> +                       if (ret)
> +                               return ret;
> +               } else
> +                       dev_info(&binfo->pdev->dev, "AFU GUID %pUl is not supported yet.\n",
> +                                afu_hdr->guid.b);
> +
> +               if (!header.next_afu)
> +                       break;
> +       }
> +
> +       return 0;
> +}
> +
> +static int parse_feature_private(struct build_feature_devs_info *binfo,
> +                                struct feature_header *hdr)
> +{
> +       struct feature_header header;
> +
> +       header.csr = readq(hdr);
> +
> +       if (!binfo->feature_dev) {
> +               dev_err(&binfo->pdev->dev, "the private feature %x does not belong to any AFU.\n",
> +                       header.id);
> +               return -EINVAL;
> +       }
> +
> +       switch (feature_dev_id_type(binfo->feature_dev)) {
> +       case FME_ID:
> +               return parse_feature_fme_private(binfo, hdr);
> +       case PORT_ID:
> +               return parse_feature_port_private(binfo, hdr);
> +       default:
> +               dev_info(&binfo->pdev->dev, "private feature %x belonging to AFU %s is not supported yet.\n",
> +                        header.id, binfo->feature_dev->name);
> +       }
> +       return 0;
> +}
> +
> +static int parse_feature(struct build_feature_devs_info *binfo,
> +                        struct feature_header *hdr)
> +{
> +       struct feature_header header;
> +       int ret = 0;
> +
> +       header.csr = readq(hdr);
> +
> +       switch (header.type) {
> +       case FEATURE_TYPE_AFU:
> +               ret = parse_feature_afus(binfo, hdr);
> +               break;
> +       case FEATURE_TYPE_PRIVATE:
> +               ret = parse_feature_private(binfo, hdr);
> +               break;
> +       default:
> +               dev_info(&binfo->pdev->dev,
> +                        "Feature Type %x is not supported.\n", hdr->type);
> +       };
> +
> +       return ret;
> +}
> +
> +static int
> +parse_feature_list(struct build_feature_devs_info *binfo, void __iomem *start)
> +{
> +       struct feature_header *hdr, header;
> +       void __iomem *end = binfo->ioend;
> +       int ret = 0;
> +
> +       for (; start < end; start += header.next_header_offset) {
> +               if (end - start < sizeof(*hdr)) {
> +                       dev_err(&binfo->pdev->dev, "The region is too small to contain a feature.\n");
> +                       ret =  -EINVAL;
> +                       break;
> +               }
> +
> +               hdr = (struct feature_header *)start;
> +               ret = parse_feature(binfo, hdr);
> +               if (ret)
> +                       break;
> +
> +               header.csr = readq(hdr);
> +               if (!header.next_header_offset)
> +                       break;
> +       }
> +
> +       return ret;
> +}
> +
> +static int parse_ports_from_fme(struct build_feature_devs_info *binfo)
> +{
> +       struct feature_fme_header *fme_hdr;
> +       struct feature_fme_port port;
> +       int i = 0, ret = 0;
> +
> +       if (binfo->pfme_hdr == NULL) {
> +               dev_dbg(&binfo->pdev->dev, "VF is detected.\n");
> +               return ret;
> +       }
> +
> +       fme_hdr = binfo->pfme_hdr;
> +
> +       do {
> +               port.csr = readq(&fme_hdr->port[i]);
> +               if (!port.port_implemented)
> +                       break;
> +
> +               ret = parse_switch_to(binfo, port.port_bar);
> +               if (ret)
> +                       break;
> +
> +               ret = parse_feature_list(binfo,
> +                               binfo->ioaddr + port.port_offset);
> +               if (ret)
> +                       break;
> +       } while (++i < MAX_FPGA_PORT_NUM);
> +
> +       return ret;
> +}
> +
> +static int create_init_drvdata(struct pci_dev *pdev)
> +{
> +       struct cci_drvdata *drvdata;
> +
> +       drvdata = devm_kzalloc(&pdev->dev, sizeof(*drvdata), GFP_KERNEL);
> +       if (!drvdata)
> +               return -ENOMEM;
> +
> +       mutex_init(&drvdata->lock);
> +       INIT_LIST_HEAD(&drvdata->port_dev_list);
> +       INIT_LIST_HEAD(&drvdata->regions);
> +
> +       dev_set_drvdata(&pdev->dev, drvdata);
> +       return 0;
> +}
> +
> +static void destroy_drvdata(struct pci_dev *pdev)
> +{
> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +
> +       if (drvdata->fme_dev) {
> +               /* fme device should be unregistered first. */
> +               WARN_ON(device_is_registered(drvdata->fme_dev));
> +               free_fpga_id(FME_ID, to_platform_device(drvdata->fme_dev)->id);
> +               put_device(drvdata->fme_dev);
> +       }
> +
> +       cci_pci_remove_port_devs(pdev);
> +       cci_pci_release_regions(pdev);
> +       dev_set_drvdata(&pdev->dev, NULL);
> +       devm_kfree(&pdev->dev, drvdata);
> +}
> +
> +static int cci_pci_create_feature_devs(struct pci_dev *pdev)
> +{
> +       struct build_feature_devs_info *binfo;
> +       int ret;
> +
> +       binfo = build_info_alloc_and_init(pdev);
> +       if (!binfo)
> +               return -ENOMEM;
> +
> +       binfo->parent_dev = fpga_dev_create(&pdev->dev, INTEL_FPGA_DEV);
> +       if (IS_ERR(binfo->parent_dev)) {
> +               ret = PTR_ERR(binfo->parent_dev);
> +               goto free_binfo_exit;
> +       }
> +
> +       ret = parse_start(binfo);
> +       if (ret)
> +               goto free_binfo_exit;
> +
> +       ret = parse_feature_list(binfo, binfo->ioaddr);
> +       if (ret)
> +               goto free_binfo_exit;
> +
> +       ret = parse_ports_from_fme(binfo);
> +       if (ret)
> +               goto free_binfo_exit;
> +
> +       ret = build_info_commit_dev(binfo);
> +       if (ret)
> +               goto free_binfo_exit;
> +
> +       /*
> +        * everything is okay, reset ->parent_dev to stop it being
> +        * freed by build_info_free()
> +        */
> +       binfo->parent_dev = NULL;
> +
> +free_binfo_exit:
> +       build_info_free(binfo);
> +       return ret;
> +}
> +
>  /* PCI Device ID */
>  #define PCIe_DEVICE_ID_PF_INT_5_X      0xBCBD
>  #define PCIe_DEVICE_ID_PF_INT_6_X      0xBCC0
> @@ -81,9 +898,18 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
>                 goto release_region_exit;
>         }
>
> -       /* TODO: create and add the platform device per feature list */
> +       ret = create_init_drvdata(pcidev);
> +       if (ret)
> +               goto release_region_exit;
> +
> +       ret = cci_pci_create_feature_devs(pcidev);
> +       if (ret)
> +               goto destroy_drvdata_exit;
> +
>         return 0;
>
> +destroy_drvdata_exit:
> +       destroy_drvdata(pcidev);
>  release_region_exit:
>         pci_release_regions(pcidev);
>  disable_error_report_exit:
> @@ -94,6 +920,8 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
>
>  static void cci_pci_remove(struct pci_dev *pcidev)
>  {
> +       remove_all_devs(pcidev);
> +       destroy_drvdata(pcidev);
>         pci_release_regions(pcidev);
>         pci_disable_pcie_error_reporting(pcidev);
>         pci_disable_device(pcidev);
> @@ -108,14 +936,23 @@ static void cci_pci_remove(struct pci_dev *pcidev)
>
>  static int __init ccidrv_init(void)
>  {
> +       int ret;
> +
>         pr_info("Intel(R) FPGA PCIe Driver: Version %s\n", DRV_VERSION);
>
> -       return pci_register_driver(&cci_pci_driver);
> +       fpga_ids_init();
> +
> +       ret = pci_register_driver(&cci_pci_driver);
> +       if (ret)
> +               fpga_ids_destroy();
> +
> +       return ret;
>  }
>
>  static void __exit ccidrv_exit(void)
>  {
>         pci_unregister_driver(&cci_pci_driver);
> +       fpga_ids_destroy();
>  }
>
>  module_init(ccidrv_init);
> --
> 1.8.3.1
>

Re: [PATCH v2 07/22] fpga: intel: pcie: parse feature list and create platform device for features.

[Wu Hao]

On Thu, Jul 13, 2017 at 12:52:30PM -0500, Alan Tull wrote:
> On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> > From: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> >
> > Device Feature List structure creates a link list of feature headers
> > within the MMIO space to provide an extensible way of adding features.
> >
> > The Intel FPGA PCIe driver walks through the feature headers to enumerate
> > feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
> > Function Unit (AFU), and their private sub features. For feature devices,
> > it creates the platform devices and linked the private sub features into
> > their platform data.
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Zhang Yi <yi.z.zhang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> > Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > ---
> > v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
> >     switched to GPLv2 license.
> >     fixed comments from Moritz Fischer.
> >     fixed kbuild warning, typos and clean up the code.
> > ---
> >  drivers/fpga/Makefile            |   2 +-
> >  drivers/fpga/intel-feature-dev.c | 130 ++++++
> >  drivers/fpga/intel-feature-dev.h | 341 ++++++++++++++++
> >  drivers/fpga/intel-pcie.c        | 841 ++++++++++++++++++++++++++++++++++++++-
> >  4 files changed, 1311 insertions(+), 3 deletions(-)
> >  create mode 100644 drivers/fpga/intel-feature-dev.c
> >  create mode 100644 drivers/fpga/intel-feature-dev.h
> >
> > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> > index 5613133..ad24b3d 100644
> > --- a/drivers/fpga/Makefile
> > +++ b/drivers/fpga/Makefile
> > @@ -31,4 +31,4 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
> >  # Intel FPGA Support
> >  obj-$(CONFIG_INTEL_FPGA_PCI)           += intel-fpga-pci.o
> >
> > -intel-fpga-pci-objs := intel-pcie.o
> > +intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
> > diff --git a/drivers/fpga/intel-feature-dev.c b/drivers/fpga/intel-feature-dev.c
> > new file mode 100644
> > index 0000000..68f9cba
> > --- /dev/null
> > +++ b/drivers/fpga/intel-feature-dev.c
> > @@ -0,0 +1,130 @@
> > +/*
> > + * Intel FPGA Feature Device Driver
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Zhang Yi <yi.z.zhang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> > + *
> > + * This work is licensed under the terms of the GNU GPL version 2. See
> > + * the COPYING file in the top-level directory.
> > + */
> > +
> > +#include "intel-feature-dev.h"
> > +
> > +void feature_platform_data_add(struct feature_platform_data *pdata,
> > +                              int index, const char *name,
> > +                              int resource_index, void __iomem *ioaddr)
> > +{
> > +       WARN_ON(index >= pdata->num);
> > +
> > +       pdata->features[index].name = name;
> > +       pdata->features[index].resource_index = resource_index;
> > +       pdata->features[index].ioaddr = ioaddr;
> > +}
> > +
> > +struct feature_platform_data *
> > +feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
> > +{
> > +       struct feature_platform_data *pdata;
> > +
> > +       pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
> > +       if (pdata) {
> > +               pdata->dev = dev;
> > +               pdata->num = num;
> > +               mutex_init(&pdata->lock);
> > +       }
> > +
> > +       return pdata;
> > +}
> > +
> > +int fme_feature_num(void)
> > +{
> > +       return FME_FEATURE_ID_MAX;
> > +}
> > +
> > +int port_feature_num(void)
> > +{
> > +       return PORT_FEATURE_ID_MAX;
> > +}
> > +
> > +int fpga_port_id(struct platform_device *pdev)
> > +{
> > +       struct feature_port_header *port_hdr;
> > +       struct feature_port_capability capability;
> > +
> > +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                                              PORT_FEATURE_ID_HEADER);
> > +       WARN_ON(!port_hdr);
> > +
> > +       capability.csr = readq(&port_hdr->capability);
> > +       return capability.port_number;
> > +}
> > +EXPORT_SYMBOL_GPL(fpga_port_id);
> > +
> > +/*
> > + * Enable Port by clear the port soft reset bit, which is set by default.
> > + * The User AFU is unable to respond to any MMIO access while in reset.
> > + * __fpga_port_enable function should only be used after __fpga_port_disable
> > + * function.
> > + */
> > +void __fpga_port_enable(struct platform_device *pdev)
> > +{
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct feature_port_header *port_hdr;
> > +       struct feature_port_control control;
> > +
> > +       WARN_ON(!pdata->disable_count);
> > +
> > +       if (--pdata->disable_count != 0)
> > +               return;
> > +
> > +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                                              PORT_FEATURE_ID_HEADER);
> > +       WARN_ON(!port_hdr);
> > +
> > +       control.csr = readq(&port_hdr->control);
> > +       control.port_sftrst = 0x0;
> > +       writeq(control.csr, &port_hdr->control);
> > +}
> > +EXPORT_SYMBOL_GPL(__fpga_port_enable);
> > +
> > +#define RST_POLL_INVL 10 /* us */
> > +#define RST_POLL_TIMEOUT 1000 /* us */
> > +
> > +int __fpga_port_disable(struct platform_device *pdev)
> > +{
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct feature_port_header *port_hdr;
> > +       struct feature_port_control control;
> > +
> > +       if (pdata->disable_count++ != 0)
> > +               return 0;
> > +
> > +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                                              PORT_FEATURE_ID_HEADER);
> > +       WARN_ON(!port_hdr);
> > +
> > +       /* Set port soft reset */
> > +       control.csr = readq(&port_hdr->control);
> > +       control.port_sftrst = 0x1;
> > +       writeq(control.csr, &port_hdr->control);
> > +
> > +       /*
> > +        * HW sets ack bit to 1 when all outstanding requests have been drained
> > +        * on this port and minimum soft reset pulse width has elapsed.
> > +        * Driver polls port_soft_reset_ack to determine if reset done by HW.
> > +        */
> > +       if (readq_poll_timeout(&port_hdr->control, control.csr,
> > +                              (control.port_sftrst_ack == 1),
> > +                              RST_POLL_INVL, RST_POLL_TIMEOUT)) {
> > +               dev_err(&pdev->dev, "timeout, fail to reset device\n");
> > +               return -ETIMEDOUT;
> > +       }
> > +
> > +       return 0;
> > +}
> > +EXPORT_SYMBOL_GPL(__fpga_port_disable);
> > diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
> > new file mode 100644
> > index 0000000..f67784a
> > --- /dev/null
> > +++ b/drivers/fpga/intel-feature-dev.h
> > @@ -0,0 +1,341 @@
> > +/*
> > + * Intel FPGA Feature Device Driver Header File
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Zhang Yi <yi.z.zhang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> > + *
> > + * This work is licensed under the terms of the GNU GPL version 2. See
> > + * the COPYING file in the top-level directory.
> > + */
> > +
> > +#ifndef __INTEL_FPGA_FEATURE_H
> > +#define __INTEL_FPGA_FEATURE_H
> > +
> > +#include <linux/fs.h>
> > +#include <linux/pci.h>
> > +#include <linux/uuid.h>
> > +#include <linux/delay.h>
> > +#include <linux/iopoll.h>
> > +#include <linux/platform_device.h>
> > +
> > +#ifndef readq
> > +static inline u64 readq(void __iomem *addr)
> > +{
> > +       return readl(addr) + ((u64)readl(addr + 4) << 32);
> > +}
> > +#endif
> > +
> > +#ifndef writeq
> > +static inline void writeq(u64 val, void __iomem *addr)
> > +{
> > +       writel((u32) (val), addr);
> > +       writel((u32) (val >> 32), (addr + 4));
> > +}
> > +#endif
> > +
> > +/* maximum supported number of ports */
> > +#define MAX_FPGA_PORT_NUM 4
> > +/* plus one for fme device */
> > +#define MAX_FEATURE_DEV_NUM    (MAX_FPGA_PORT_NUM + 1)
> > +
> > +#define FME_FEATURE_HEADER          "fme_hdr"
> > +#define FME_FEATURE_THERMAL_MGMT    "fme_thermal"
> > +#define FME_FEATURE_POWER_MGMT      "fme_power"
> > +#define FME_FEATURE_GLOBAL_PERF     "fme_gperf"
> > +#define FME_FEATURE_GLOBAL_ERR      "fme_error"
> > +#define FME_FEATURE_PR_MGMT         "fme_pr"
> > +
> > +#define PORT_FEATURE_HEADER         "port_hdr"
> > +#define PORT_FEATURE_UAFU           "port_uafu"
> > +#define PORT_FEATURE_ERR            "port_err"
> > +#define PORT_FEATURE_UMSG           "port_umsg"
> > +#define PORT_FEATURE_PR             "port_pr"
> > +#define PORT_FEATURE_STP            "port_stp"
> > +
> > +/* All headers and structures must be byte-packed to match the spec. */
> > +#pragma pack(1)
> > +
> > +/* common header for all features */
> > +struct feature_header {
> > +       union {
> > +               u64 csr;
> > +               struct {
> > +                       u64 id:12;
> > +                       u64 revision:4;
> > +                       u64 next_header_offset:24; /* offset to next header */
> > +                       u64 rsvdz:20;
> > +                       u64 type:4;                /* feature type */
> > +#define FEATURE_TYPE_AFU               0x1
> > +#define FEATURE_TYPE_PRIVATE           0x3
> > +               };
> > +       };
> > +};
> 
> Hi Hao,
> 
> Aren't these union's introducing a portability problem?  Can you use
> regmap instead?

Hi Alan

I think it's fine to use union here, as endian conversion should be covered
inside the readq/writeq function. :)

Thanks
Hao

> 
> Alan
> 

Re: [PATCH v2 01/22] docs: fpga: add a document for Intel FPGA driver overview

[Luebbers, Enno]

On Thu, Jul 13, 2017 at 12:25:20PM +0800, Wu Hao wrote:
> On Wed, Jul 12, 2017 at 09:51:32AM -0500, Alan Tull wrote:
> > On Sun, Jun 25, 2017 at 8:51 PM, Wu Hao <hao.wu@intel.com> wrote:
> > 
> > Hi Hao,
> > 
> > > Add a document for Intel FPGA driver overview.
> > >
> > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > > ----
> > > v2: added FME fpga-mgr/bridge/region platform driver to driver organization.
> > >     updated open discussion per current implementation.
> > >     fixed some typos.
> > > ---
> > >  Documentation/fpga/intel-fpga.txt | 256 ++++++++++++++++++++++++++++++++++++++
> > >  1 file changed, 256 insertions(+)
> > >  create mode 100644 Documentation/fpga/intel-fpga.txt
> > >
> > > diff --git a/Documentation/fpga/intel-fpga.txt b/Documentation/fpga/intel-fpga.txt
> > > new file mode 100644
> > > index 0000000..4a29470
> > > --- /dev/null
> > > +++ b/Documentation/fpga/intel-fpga.txt
> > > @@ -0,0 +1,256 @@
> > > +===============================================================================
> > > +                    Intel FPGA driver Overview
> > > +-------------------------------------------------------------------------------
> > > +                Enno Luebbers <enno.luebbers@intel.com>
> > > +                Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > > +                Wu Hao <hao.wu@intel.com>
> > > +
> > > +The Intel FPGA driver provides interfaces for userspace applications to
> > > +configure, enumerate, open, and access FPGA accelerators on platforms equipped
> > > +with Intel(R) FPGA PCIe based solutions and enables system level management
> > > +functions such as FPGA reconfiguration, power management, and virtualization.
> > > +
> > > +HW Architecture
> > > +===============
> > > +From the OS's point of view, the FPGA hardware appears as a regular PCIe device.
> > > +The FPGA device memory is organized using a predefined data structure (Device
> > > +Feature List). Features supported by the particular FPGA device are exposed
> > > +through these data structures, as illustrated below:
> > > +
> > > +  +-------------------------------+  +-------------+
> > > +  |              PF               |  |     VF      |
> > > +  +-------------------------------+  +-------------+
> > > +      ^            ^         ^              ^
> > > +      |            |         |              |
> > > ++-----|------------|---------|--------------|-------+
> > > +|     |            |         |              |       |
> > > +|  +-----+     +-------+ +-------+      +-------+   |
> > > +|  | FME |     | Port0 | | Port1 |      | Port2 |   |
> > > +|  +-----+     +-------+ +-------+      +-------+   |
> > > +|                  ^         ^              ^       |
> > > +|                  |         |              |       |
> > > +|              +-------+ +------+       +-------+   |
> > > +|              |  AFU  | |  AFU |       |  AFU  |   |
> > > +|              +-------+ +------+       +-------+   |
> > > +|                                                   |
> > > +|                 FPGA PCIe Device                  |
> > > ++---------------------------------------------------+
> > > +
> > > +The driver supports PCIe SR-IOV to create virtual functions (VFs) which can be
> > > +used to assign individual accelerators to virtual machines.
> > > +
> > > +FME (FPGA Management Engine)
> > > +============================
> > > +The FPGA Management Engine performs power and thermal management, error
> > > +reporting, reconfiguration, performance reporting, and other infrastructure
> > > +functions. Each FPGA has one FME, which is always accessed through the physical
> > > +function (PF).
> > > +
> > > +User-space applications can acquire exclusive access to the FME using open(),
> > > +and release it using close().
> > > +
> > > +The following functions are exposed through ioctls:
> > > +
> > > +       Get driver API version (FPGA_GET_API_VERSION)
> > > +       Check for extensions (FPGA_CHECK_EXTENSION)
> > > +       Assign port to PF (FPGA_FME_PORT_ASSIGN)
> > > +       Release port from PF (FPGA_FME_PORT_RELEASE)
> > > +       Program bitstream (FPGA_FME_PORT_PR)
> > > +
> > 
> > I was hoping the API mailing list might have an opinion about this,
> > but I think adding ioctls to the kernel is discouraged.  Could these
> > be sysfs?
> 
> Hi Alan,
> 
> As you see below, we have defined a lot of sysfs interface for device
> info, attributes and simple control operations. But for some actions
> which requires complex inputs/outputs parameters (e.g a struct with
> multiple items) with userspace, ioctls are used. I feel in such cases,
> ioctls seem more suitable than sysfs.
> 

Also, we're thinking that some operations require that you first "acquire
ownership" of the respective device, which I believe maps more easily to
open() and ioctls than sysfs.

Thanks
- Enno

Re: [PATCH v2 12/22] fpga: intel: fme: add header sub feature support

[Alan Tull]

On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu@intel.com> wrote:

Hi Hao,

I'm making my way through this (very large) patchset.  Some minor
comments below.

> From: Kang Luwei <luwei.kang@intel.com>
>
> The header register set is always present for FPGA Management Engine (FME),
> this patch implements init and uinit function for header sub feature and
> introduce several read-only sysfs interfaces for the capability and status.
>
> Sysfs interfaces:
> * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/ports_num
>   Read-only. Number of ports implemented
>
> * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/bitstream_id
>   Read-only. Blue Bitstream identifier number

Blue and Green bitstreams are an Intel implementation terminology.  I
see that you've defined them in drivers/fpga, but it will be helpful
to add in "static region" and "partial reconfiguration region" added
in any API documentation files that use the green/blue terminology to
keep it accessible.

>
> * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/bitstream_metadata
>   Read-only. Blue Bitstream meta data
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ---
> v2: add sysfs documentation
> ---
>  .../ABI/testing/sysfs-platform-intel-fpga-fme      | 19 ++++++++
>  drivers/fpga/intel-feature-dev.h                   |  3 ++
>  drivers/fpga/intel-fme-main.c                      | 55 ++++++++++++++++++++++
>  3 files changed, 77 insertions(+)
>  create mode 100644 Documentation/ABI/testing/sysfs-platform-intel-fpga-fme
>
> diff --git a/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme b/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme
> new file mode 100644
> index 0000000..783cfa9
> --- /dev/null
> +++ b/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme
> @@ -0,0 +1,19 @@
> +What:          /sys/bus/platform/devices/intel-fpga-fme.0/ports_num
> +Date:          June 2017
> +KernelVersion:  4.12
> +Contact:       Wu Hao <hao.wu@intel.com>
> +Description:   Read-only. One Intel FPGA device may have more than 1
> +               port/Accelerator Function Unit (AFU). It returns the
> +               number of ports on the FPGA device when read it.
> +
> +What:          /sys/bus/platform/devices/intel-fpga-fme.0/bitstream_id
> +Date:          June 2017
> +KernelVersion:  4.12
> +Contact:       Wu Hao <hao.wu@intel.com>
> +Description:   Read-only. It returns Blue Bitstream identifier number.

Here

> +
> +What:          /sys/bus/platform/devices/intel-fpga-fme.0/bitstream_meta
> +Date:          June 2017
> +KernelVersion:  4.12
> +Contact:       Wu Hao <hao.wu@intel.com>
> +Description:   Read-only. It returns Blue Bitstream meta data.

And here

> diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
> index 635b857..3f97b75 100644
> --- a/drivers/fpga/intel-feature-dev.h
> +++ b/drivers/fpga/intel-feature-dev.h
> @@ -138,6 +138,9 @@ struct feature_fme_header {
>         u64 rsvd[2];
>         struct feature_fme_capability capability;
>         struct feature_fme_port port[MAX_FPGA_PORT_NUM];
> +       u64 rsvd1;
> +       u64 bitstream_id;
> +       u64 bitstream_md;
>  };
>
>  /* FME Thermal Sub Feature Register Set */
> diff --git a/drivers/fpga/intel-fme-main.c b/drivers/fpga/intel-fme-main.c
> index c16cf81..dfbb17c 100644
> --- a/drivers/fpga/intel-fme-main.c
> +++ b/drivers/fpga/intel-fme-main.c
> @@ -21,15 +21,70 @@
>
>  #include "intel-feature-dev.h"
>
> +static ssize_t ports_num_show(struct device *dev,
> +                             struct device_attribute *attr, char *buf)
> +{
> +       struct feature_fme_header *fme_hdr
> +               = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_HEADER);
> +       struct feature_fme_capability fme_capability;
> +
> +       fme_capability.csr = readq(&fme_hdr->capability);
> +
> +       return scnprintf(buf, PAGE_SIZE, "%d\n", fme_capability.num_ports);
> +}
> +static DEVICE_ATTR_RO(ports_num);
> +
> +static ssize_t bitstream_id_show(struct device *dev,
> +                                struct device_attribute *attr, char *buf)
> +{
> +       struct feature_fme_header *fme_hdr
> +               = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_HEADER);
> +       u64 bitstream_id = readq(&fme_hdr->bitstream_id);
> +
> +       return scnprintf(buf, PAGE_SIZE, "0x%llx\n",
> +                               (unsigned long long)bitstream_id);
> +}
> +static DEVICE_ATTR_RO(bitstream_id);
> +
> +static ssize_t bitstream_metadata_show(struct device *dev,
> +                                      struct device_attribute *attr, char *buf)
> +{
> +       struct feature_fme_header *fme_hdr
> +               = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_HEADER);
> +       u64 bitstream_md = readq(&fme_hdr->bitstream_md);
> +
> +       return scnprintf(buf, PAGE_SIZE, "0x%llx\n",
> +                               (unsigned long long)bitstream_md);
> +}
> +static DEVICE_ATTR_RO(bitstream_metadata);
> +
> +static const struct attribute *fme_hdr_attrs[] = {
> +       &dev_attr_ports_num.attr,
> +       &dev_attr_bitstream_id.attr,
> +       &dev_attr_bitstream_metadata.attr,
> +       NULL,
> +};
> +
>  static int fme_hdr_init(struct platform_device *pdev, struct feature *feature)
>  {
> +       struct feature_fme_header *fme_hdr = feature->ioaddr;
> +       int ret;
> +
>         dev_dbg(&pdev->dev, "FME HDR Init.\n");
> +       dev_dbg(&pdev->dev, "FME cap %llx.\n",
> +                               (unsigned long long)fme_hdr->capability.csr);
> +
> +       ret = sysfs_create_files(&pdev->dev.kobj, fme_hdr_attrs);
> +       if (ret)
> +               return ret;
> +
>         return 0;
>  }
>
>  static void fme_hdr_uinit(struct platform_device *pdev, struct feature *feature)
>  {
>         dev_dbg(&pdev->dev, "FME HDR UInit.\n");
> +       sysfs_remove_files(&pdev->dev.kobj, fme_hdr_attrs);
>  }
>
>  struct feature_ops fme_hdr_ops = {
> --
> 1.8.3.1
>

Re: [PATCH v2 07/22] fpga: intel: pcie: parse feature list and create platform device for features.

[Alan Tull]

On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu@intel.com> wrote:
> From: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>
> Device Feature List structure creates a link list of feature headers
> within the MMIO space to provide an extensible way of adding features.
>
> The Intel FPGA PCIe driver walks through the feature headers to enumerate
> feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
> Function Unit (AFU), and their private sub features. For feature devices,
> it creates the platform devices and linked the private sub features into
> their platform data.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ---
> v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
>     switched to GPLv2 license.
>     fixed comments from Moritz Fischer.
>     fixed kbuild warning, typos and clean up the code.
> ---
>  drivers/fpga/Makefile            |   2 +-
>  drivers/fpga/intel-feature-dev.c | 130 ++++++
>  drivers/fpga/intel-feature-dev.h | 341 ++++++++++++++++
>  drivers/fpga/intel-pcie.c        | 841 ++++++++++++++++++++++++++++++++++++++-
>  4 files changed, 1311 insertions(+), 3 deletions(-)
>  create mode 100644 drivers/fpga/intel-feature-dev.c
>  create mode 100644 drivers/fpga/intel-feature-dev.h
>
> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> index 5613133..ad24b3d 100644
> --- a/drivers/fpga/Makefile
> +++ b/drivers/fpga/Makefile
> @@ -31,4 +31,4 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
>  # Intel FPGA Support
>  obj-$(CONFIG_INTEL_FPGA_PCI)           += intel-fpga-pci.o
>
> -intel-fpga-pci-objs := intel-pcie.o
> +intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
> diff --git a/drivers/fpga/intel-feature-dev.c b/drivers/fpga/intel-feature-dev.c
> new file mode 100644
> index 0000000..68f9cba
> --- /dev/null
> +++ b/drivers/fpga/intel-feature-dev.c
> @@ -0,0 +1,130 @@
> +/*
> + * Intel FPGA Feature Device Driver
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <luwei.kang@intel.com>
> + *   Zhang Yi <yi.z.zhang@intel.com>
> + *   Wu Hao <hao.wu@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2. See
> + * the COPYING file in the top-level directory.
> + */
> +
> +#include "intel-feature-dev.h"
> +
> +void feature_platform_data_add(struct feature_platform_data *pdata,
> +                              int index, const char *name,
> +                              int resource_index, void __iomem *ioaddr)
> +{
> +       WARN_ON(index >= pdata->num);
> +
> +       pdata->features[index].name = name;
> +       pdata->features[index].resource_index = resource_index;
> +       pdata->features[index].ioaddr = ioaddr;
> +}
> +
> +struct feature_platform_data *
> +feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
> +{
> +       struct feature_platform_data *pdata;
> +
> +       pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
> +       if (pdata) {
> +               pdata->dev = dev;
> +               pdata->num = num;
> +               mutex_init(&pdata->lock);
> +       }
> +
> +       return pdata;
> +}
> +
> +int fme_feature_num(void)
> +{
> +       return FME_FEATURE_ID_MAX;
> +}
> +
> +int port_feature_num(void)
> +{
> +       return PORT_FEATURE_ID_MAX;
> +}
> +
> +int fpga_port_id(struct platform_device *pdev)
> +{
> +       struct feature_port_header *port_hdr;
> +       struct feature_port_capability capability;
> +
> +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> +                                              PORT_FEATURE_ID_HEADER);
> +       WARN_ON(!port_hdr);
> +
> +       capability.csr = readq(&port_hdr->capability);
> +       return capability.port_number;
> +}
> +EXPORT_SYMBOL_GPL(fpga_port_id);
> +
> +/*
> + * Enable Port by clear the port soft reset bit, which is set by default.
> + * The User AFU is unable to respond to any MMIO access while in reset.
> + * __fpga_port_enable function should only be used after __fpga_port_disable
> + * function.
> + */
> +void __fpga_port_enable(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct feature_port_header *port_hdr;
> +       struct feature_port_control control;
> +
> +       WARN_ON(!pdata->disable_count);
> +
> +       if (--pdata->disable_count != 0)
> +               return;
> +
> +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> +                                              PORT_FEATURE_ID_HEADER);
> +       WARN_ON(!port_hdr);
> +
> +       control.csr = readq(&port_hdr->control);
> +       control.port_sftrst = 0x0;
> +       writeq(control.csr, &port_hdr->control);
> +}
> +EXPORT_SYMBOL_GPL(__fpga_port_enable);
> +
> +#define RST_POLL_INVL 10 /* us */
> +#define RST_POLL_TIMEOUT 1000 /* us */
> +
> +int __fpga_port_disable(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct feature_port_header *port_hdr;
> +       struct feature_port_control control;
> +
> +       if (pdata->disable_count++ != 0)
> +               return 0;
> +
> +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> +                                              PORT_FEATURE_ID_HEADER);
> +       WARN_ON(!port_hdr);
> +
> +       /* Set port soft reset */
> +       control.csr = readq(&port_hdr->control);
> +       control.port_sftrst = 0x1;
> +       writeq(control.csr, &port_hdr->control);
> +
> +       /*
> +        * HW sets ack bit to 1 when all outstanding requests have been drained
> +        * on this port and minimum soft reset pulse width has elapsed.
> +        * Driver polls port_soft_reset_ack to determine if reset done by HW.
> +        */
> +       if (readq_poll_timeout(&port_hdr->control, control.csr,
> +                              (control.port_sftrst_ack == 1),
> +                              RST_POLL_INVL, RST_POLL_TIMEOUT)) {
> +               dev_err(&pdev->dev, "timeout, fail to reset device\n");
> +               return -ETIMEDOUT;
> +       }
> +
> +       return 0;
> +}
> +EXPORT_SYMBOL_GPL(__fpga_port_disable);
> diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
> new file mode 100644
> index 0000000..f67784a
> --- /dev/null
> +++ b/drivers/fpga/intel-feature-dev.h
> @@ -0,0 +1,341 @@
> +/*
> + * Intel FPGA Feature Device Driver Header File
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <luwei.kang@intel.com>
> + *   Zhang Yi <yi.z.zhang@intel.com>
> + *   Wu Hao <hao.wu@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2. See
> + * the COPYING file in the top-level directory.
> + */
> +
> +#ifndef __INTEL_FPGA_FEATURE_H
> +#define __INTEL_FPGA_FEATURE_H
> +
> +#include <linux/fs.h>
> +#include <linux/pci.h>
> +#include <linux/uuid.h>
> +#include <linux/delay.h>
> +#include <linux/iopoll.h>
> +#include <linux/platform_device.h>
> +
> +#ifndef readq
> +static inline u64 readq(void __iomem *addr)
> +{
> +       return readl(addr) + ((u64)readl(addr + 4) << 32);
> +}
> +#endif
> +
> +#ifndef writeq
> +static inline void writeq(u64 val, void __iomem *addr)
> +{
> +       writel((u32) (val), addr);
> +       writel((u32) (val >> 32), (addr + 4));
> +}
> +#endif

Hi Hao,

Was there a reason readq and writeq had to be created for this?  Do
these get used?

Alan

> +
> +/* maximum supported number of ports */
> +#define MAX_FPGA_PORT_NUM 4
> +/* plus one for fme device */
> +#define MAX_FEATURE_DEV_NUM    (MAX_FPGA_PORT_NUM + 1)
> +
> +#define FME_FEATURE_HEADER          "fme_hdr"
> +#define FME_FEATURE_THERMAL_MGMT    "fme_thermal"
> +#define FME_FEATURE_POWER_MGMT      "fme_power"
> +#define FME_FEATURE_GLOBAL_PERF     "fme_gperf"
> +#define FME_FEATURE_GLOBAL_ERR      "fme_error"
> +#define FME_FEATURE_PR_MGMT         "fme_pr"
> +
> +#define PORT_FEATURE_HEADER         "port_hdr"
> +#define PORT_FEATURE_UAFU           "port_uafu"
> +#define PORT_FEATURE_ERR            "port_err"
> +#define PORT_FEATURE_UMSG           "port_umsg"
> +#define PORT_FEATURE_PR             "port_pr"
> +#define PORT_FEATURE_STP            "port_stp"
> +
> +/* All headers and structures must be byte-packed to match the spec. */
> +#pragma pack(1)
> +
> +/* common header for all features */
> +struct feature_header {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u64 id:12;
> +                       u64 revision:4;
> +                       u64 next_header_offset:24; /* offset to next header */
> +                       u64 rsvdz:20;
> +                       u64 type:4;                /* feature type */
> +#define FEATURE_TYPE_AFU               0x1
> +#define FEATURE_TYPE_PRIVATE           0x3
> +               };
> +       };
> +};
> +
> +/* common header for non-private features */
> +struct feature_afu_header {
> +       uuid_le guid;
> +       union {
> +               u64 csr;
> +               struct {
> +                       u64 next_afu:24;        /* pointer to next afu header */
> +                       u64 rsvdz:40;
> +               };
> +       };
> +};
> +
> +/* FME Header Register Set */
> +/* FME Capability Register */
> +struct feature_fme_capability {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u64 fabric_verid:8;     /* Fabric version ID */
> +                       u64 socket_id:1;        /* Socket id */
> +                       u64 rsvdz1:3;
> +                       u64 pcie0_link_avl:1;   /* PCIe0 link availability */
> +                       u64 pcie1_link_avl:1;   /* PCIe1 link availability */
> +                       u64 coherent_link_avl:1;/* Coherent link availability */
> +                       u64 rsvdz2:1;
> +                       u64 iommu_support:1;    /* IOMMU or VT-d supported */
> +                       u64 num_ports:3;        /* Num of ports implemented */
> +                       u64 rsvdz3:4;
> +                       u64 addr_width_bits:6;  /* Address width supported */
> +                       u64 rsvdz4:2;
> +                       u64 cache_size:12;      /* Cache size in kb */
> +                       u64 cache_assoc:4;      /* Cache Associativity */
> +                       u64 rsvdz5:15;
> +                       u64 lock_bit:1;         /* Latched lock bit by BIOS */
> +               };
> +       };
> +};
> +
> +/* FME Port Offset Register */
> +struct feature_fme_port {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u64 port_offset:24;     /* Offset to port header */
> +                       u64 rsvdz1:8;
> +                       u64 port_bar:3;         /* Bar id */
> +                       u64 rsvdz2:20;
> +                       u64 afu_access_ctrl:1;  /* AFU access type: PF/VF */
> +                       u64 rsvdz3:4;
> +                       u64 port_implemented:1; /* Port implemented or not */
> +                       u64 rsvdz4:3;
> +               };
> +       };
> +};
> +
> +struct feature_fme_header {
> +       struct feature_header header;
> +       struct feature_afu_header afu_header;
> +       u64 rsvd[2];
> +       struct feature_fme_capability capability;
> +       struct feature_fme_port port[MAX_FPGA_PORT_NUM];
> +};
> +
> +/* FME Thermal Sub Feature Register Set */
> +struct feature_fme_thermal {
> +       struct feature_header header;
> +};
> +
> +/* FME Power Sub Feature Register Set */
> +struct feature_fme_power {
> +       struct feature_header header;
> +};
> +
> +/* FME Global Performance Sub Feature Register Set */
> +struct feature_fme_gperf {
> +       struct feature_header header;
> +};
> +
> +/* FME Error Sub Feature Register Set */
> +struct feature_fme_err {
> +       struct feature_header header;
> +};
> +
> +/* FME Partial Reconfiguration Sub Feature Register Set */
> +struct feature_fme_pr {
> +       struct feature_header header;
> +};
> +
> +/* PORT Header Register Set */
> +/* Port Capability Register */
> +struct feature_port_capability {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u64 port_number:2;      /* Port Number 0-3 */
> +                       u64 rsvdz1:6;
> +                       u64 mmio_size:16;       /* User MMIO size in KB */
> +                       u64 rsvdz2:8;
> +                       u64 sp_intr_num:4;      /* Supported interrupts num */
> +                       u64 rsvdz3:28;
> +               };
> +       };
> +};
> +
> +/* Port Control Register */
> +struct feature_port_control {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u64 port_sftrst:1;      /* Port Soft Reset */
> +                       u64 rsvdz1:1;
> +                       u64 latency_tolerance:1;/* '1' >= 40us, '0' < 40us */
> +                       u64 rsvdz2:1;
> +                       u64 port_sftrst_ack:1;  /* HW ACK for Soft Reset */
> +                       u64 rsvdz3:59;
> +               };
> +       };
> +};
> +
> +struct feature_port_header {
> +       struct feature_header header;
> +       struct feature_afu_header afu_header;
> +       u64 rsvd[2];
> +       struct feature_port_capability capability;
> +       struct feature_port_control control;
> +};
> +
> +/* PORT Error Sub Feature Register Set */
> +struct feature_port_error {
> +       struct feature_header header;
> +};
> +
> +/* PORT Unordered Message Sub Feature Register Set */
> +struct feature_port_umsg {
> +       struct feature_header header;
> +};
> +
> +/* PORT SignalTap Sub Feature Register Set */
> +struct feature_port_stp {
> +       struct feature_header header;
> +};
> +
> +#pragma pack()
> +
> +struct feature {
> +       const char *name;
> +       int resource_index;
> +       void __iomem *ioaddr;
> +};
> +
> +struct feature_platform_data {
> +       /* list the feature dev to cci_drvdata->port_dev_list. */
> +       struct list_head node;
> +       struct mutex lock;
> +       struct platform_device *dev;
> +       unsigned int disable_count;     /* count for port disable */
> +
> +       int num;                        /* number of features */
> +       struct feature features[0];
> +};
> +
> +enum fme_feature_id {
> +       FME_FEATURE_ID_HEADER = 0x0,
> +       FME_FEATURE_ID_THERMAL_MGMT = 0x1,
> +       FME_FEATURE_ID_POWER_MGMT = 0x2,
> +       FME_FEATURE_ID_GLOBAL_PERF = 0x3,
> +       FME_FEATURE_ID_GLOBAL_ERR = 0x4,
> +       FME_FEATURE_ID_PR_MGMT = 0x5,
> +       FME_FEATURE_ID_MAX = 0x6,
> +};
> +
> +enum port_feature_id {
> +       PORT_FEATURE_ID_HEADER = 0x0,
> +       PORT_FEATURE_ID_ERROR = 0x1,
> +       PORT_FEATURE_ID_UMSG = 0x2,
> +       PORT_FEATURE_ID_PR = 0x3,
> +       PORT_FEATURE_ID_STP = 0x4,
> +       PORT_FEATURE_ID_UAFU = 0x5,
> +       PORT_FEATURE_ID_MAX = 0x6,
> +};
> +
> +int fme_feature_num(void);
> +int port_feature_num(void);
> +
> +#define FPGA_FEATURE_DEV_FME           "intel-fpga-fme"
> +#define FPGA_FEATURE_DEV_PORT          "intel-fpga-port"
> +
> +void feature_platform_data_add(struct feature_platform_data *pdata,
> +                              int index, const char *name,
> +                              int resource_index, void __iomem *ioaddr);
> +
> +static inline int feature_platform_data_size(const int num)
> +{
> +       return sizeof(struct feature_platform_data) +
> +               num * sizeof(struct feature);
> +}
> +
> +struct feature_platform_data *
> +feature_platform_data_alloc_and_init(struct platform_device *dev, int num);
> +
> +int fpga_port_id(struct platform_device *pdev);
> +
> +static inline int fpga_port_check_id(struct platform_device *pdev,
> +                                    void *pport_id)
> +{
> +       return fpga_port_id(pdev) == *(int *)pport_id;
> +}
> +
> +void __fpga_port_enable(struct platform_device *pdev);
> +int __fpga_port_disable(struct platform_device *pdev);
> +
> +static inline void fpga_port_enable(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +
> +       mutex_lock(&pdata->lock);
> +       __fpga_port_enable(pdev);
> +       mutex_unlock(&pdata->lock);
> +}
> +
> +static inline int fpga_port_disable(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       int ret;
> +
> +       mutex_lock(&pdata->lock);
> +       ret = __fpga_port_disable(pdev);
> +       mutex_unlock(&pdata->lock);
> +
> +       return ret;
> +}
> +
> +static inline int __fpga_port_reset(struct platform_device *pdev)
> +{
> +       int ret;
> +
> +       ret = __fpga_port_disable(pdev);
> +       if (ret)
> +               return ret;
> +
> +       __fpga_port_enable(pdev);
> +       return 0;
> +}
> +
> +static inline int fpga_port_reset(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       int ret;
> +
> +       mutex_lock(&pdata->lock);
> +       ret = __fpga_port_reset(pdev);
> +       mutex_unlock(&pdata->lock);
> +       return ret;
> +}
> +
> +static inline void __iomem *
> +get_feature_ioaddr_by_index(struct device *dev, int index)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(dev);
> +
> +       return pdata->features[index].ioaddr;
> +}
> +#endif
> diff --git a/drivers/fpga/intel-pcie.c b/drivers/fpga/intel-pcie.c
> index f697de4..70b8284 100644
> --- a/drivers/fpga/intel-pcie.c
> +++ b/drivers/fpga/intel-pcie.c
> @@ -23,10 +23,827 @@
>  #include <linux/stddef.h>
>  #include <linux/errno.h>
>  #include <linux/aer.h>
> +#include <linux/fpga/fpga-dev.h>
> +
> +#include "intel-feature-dev.h"
>
>  #define DRV_VERSION    "0.8"
>  #define DRV_NAME       "intel-fpga-pci"
>
> +#define INTEL_FPGA_DEV "intel-fpga-dev"
> +
> +static DEFINE_MUTEX(fpga_id_mutex);
> +
> +enum fpga_id_type {
> +       FME_ID,         /* fme id allocation and mapping */
> +       PORT_ID,        /* port id allocation and mapping */
> +       FPGA_ID_MAX,
> +};
> +
> +/* it is protected by fpga_id_mutex */
> +static struct idr fpga_ids[FPGA_ID_MAX];
> +
> +struct cci_drvdata {
> +       struct device *fme_dev;
> +
> +       struct mutex lock;
> +       struct list_head port_dev_list;
> +
> +       struct list_head regions; /* global list of pci bar mapping region */
> +};
> +
> +/* pci bar mapping info */
> +struct cci_pci_region {
> +       int bar;
> +       void __iomem *ioaddr;   /* pointer to mapped bar region */
> +       struct list_head node;
> +};
> +
> +static void fpga_ids_init(void)
> +{
> +       int i;
> +
> +       for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> +               idr_init(fpga_ids + i);
> +}
> +
> +static void fpga_ids_destroy(void)
> +{
> +       int i;
> +
> +       for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> +               idr_destroy(fpga_ids + i);
> +}
> +
> +static int alloc_fpga_id(enum fpga_id_type type, struct device *dev)
> +{
> +       int id;
> +
> +       WARN_ON(type >= FPGA_ID_MAX);
> +       mutex_lock(&fpga_id_mutex);
> +       id = idr_alloc(fpga_ids + type, dev, 0, 0, GFP_KERNEL);
> +       mutex_unlock(&fpga_id_mutex);
> +       return id;
> +}
> +
> +static void free_fpga_id(enum fpga_id_type type, int id)
> +{
> +       WARN_ON(type >= FPGA_ID_MAX);
> +       mutex_lock(&fpga_id_mutex);
> +       idr_remove(fpga_ids + type, id);
> +       mutex_unlock(&fpga_id_mutex);
> +}
> +
> +static void cci_pci_add_port_dev(struct pci_dev *pdev,
> +                                struct platform_device *port_dev)
> +{
> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +       struct feature_platform_data *pdata = dev_get_platdata(&port_dev->dev);
> +
> +       mutex_lock(&drvdata->lock);
> +       list_add(&pdata->node, &drvdata->port_dev_list);
> +       get_device(&pdata->dev->dev);
> +       mutex_unlock(&drvdata->lock);
> +}
> +
> +static void cci_pci_remove_port_devs(struct pci_dev *pdev)
> +{
> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +       struct feature_platform_data *pdata, *ptmp;
> +
> +       mutex_lock(&drvdata->lock);
> +       list_for_each_entry_safe(pdata, ptmp, &drvdata->port_dev_list, node) {
> +               struct platform_device *port_dev = pdata->dev;
> +
> +               /* the port should be unregistered first. */
> +               WARN_ON(device_is_registered(&port_dev->dev));
> +               list_del(&pdata->node);
> +               free_fpga_id(PORT_ID, port_dev->id);
> +               put_device(&port_dev->dev);
> +       }
> +       mutex_unlock(&drvdata->lock);
> +}
> +
> +/* info collection during feature dev build. */
> +struct build_feature_devs_info {
> +       struct pci_dev *pdev;
> +
> +       /*
> +        * PCI BAR mapping info. Parsing feature list starts from
> +        * BAR 0 and switch to different BARs to parse Port
> +        */
> +       void __iomem *ioaddr;
> +       void __iomem *ioend;
> +       int current_bar;
> +
> +       /* points to FME header where the port offset is figured out. */
> +       void __iomem *pfme_hdr;
> +
> +       /* the container device for all feature devices */
> +       struct fpga_dev *parent_dev;
> +
> +       /* current feature device */
> +       struct platform_device *feature_dev;
> +};
> +
> +static void cci_pci_release_regions(struct pci_dev *pdev)
> +{
> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +       struct cci_pci_region *tmp, *region;
> +
> +       list_for_each_entry_safe(region, tmp, &drvdata->regions, node) {
> +               list_del(&region->node);
> +               if (region->ioaddr)
> +                       pci_iounmap(pdev, region->ioaddr);
> +               devm_kfree(&pdev->dev, region);
> +       }
> +}
> +
> +static void __iomem *cci_pci_ioremap_bar(struct pci_dev *pdev, int bar)
> +{
> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +       struct cci_pci_region *region;
> +
> +       list_for_each_entry(region, &drvdata->regions, node)
> +               if (region->bar == bar) {
> +                       dev_dbg(&pdev->dev, "BAR %d region exists\n", bar);
> +                       return region->ioaddr;
> +               }
> +
> +       region = devm_kzalloc(&pdev->dev, sizeof(*region), GFP_KERNEL);
> +       if (!region)
> +               return NULL;
> +
> +       region->bar = bar;
> +       region->ioaddr = pci_ioremap_bar(pdev, bar);
> +       if (!region->ioaddr) {
> +               dev_err(&pdev->dev, "can't ioremap memory from BAR %d.\n", bar);
> +               devm_kfree(&pdev->dev, region);
> +               return NULL;
> +       }
> +
> +       list_add(&region->node, &drvdata->regions);
> +       return region->ioaddr;
> +}
> +
> +static int parse_start_from(struct build_feature_devs_info *binfo, int bar)
> +{
> +       binfo->ioaddr = cci_pci_ioremap_bar(binfo->pdev, bar);
> +       if (!binfo->ioaddr)
> +               return -ENOMEM;
> +
> +       binfo->current_bar = bar;
> +       binfo->ioend = binfo->ioaddr + pci_resource_len(binfo->pdev, bar);
> +       return 0;
> +}
> +
> +static int parse_start(struct build_feature_devs_info *binfo)
> +{
> +       /* fpga feature list starts from BAR 0 */
> +       return parse_start_from(binfo, 0);
> +}
> +
> +/* switch the memory mapping to BAR# @bar */
> +static int parse_switch_to(struct build_feature_devs_info *binfo, int bar)
> +{
> +       return parse_start_from(binfo, bar);
> +}
> +
> +static struct build_feature_devs_info *
> +build_info_alloc_and_init(struct pci_dev *pdev)
> +{
> +       struct build_feature_devs_info *binfo;
> +
> +       binfo = devm_kzalloc(&pdev->dev, sizeof(*binfo), GFP_KERNEL);
> +       if (binfo)
> +               binfo->pdev = pdev;
> +
> +       return binfo;
> +}
> +
> +static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
> +{
> +       if (!strcmp(pdev->name, FPGA_FEATURE_DEV_FME))
> +               return FME_ID;
> +
> +       if (!strcmp(pdev->name, FPGA_FEATURE_DEV_PORT))
> +               return PORT_ID;
> +
> +       WARN_ON(1);
> +       return FPGA_ID_MAX;
> +}
> +
> +/*
> + * register current feature device, it is called when we need to switch to
> + * another feature parsing or we have parsed all features
> + */
> +static int build_info_commit_dev(struct build_feature_devs_info *binfo)
> +{
> +       int ret;
> +
> +       if (!binfo->feature_dev)
> +               return 0;
> +
> +       ret = platform_device_add(binfo->feature_dev);
> +       if (!ret) {
> +               struct cci_drvdata *drvdata;
> +
> +               drvdata = dev_get_drvdata(&binfo->pdev->dev);
> +               if (feature_dev_id_type(binfo->feature_dev) == PORT_ID)
> +                       cci_pci_add_port_dev(binfo->pdev, binfo->feature_dev);
> +               else
> +                       drvdata->fme_dev = get_device(&binfo->feature_dev->dev);
> +
> +               /*
> +                * reset it to avoid build_info_free() freeing their resource.
> +                *
> +                * The resource of successfully registered feature devices
> +                * will be freed by platform_device_unregister(). See the
> +                * comments in build_info_create_dev().
> +                */
> +               binfo->feature_dev = NULL;
> +       }
> +
> +       return ret;
> +}
> +
> +static int
> +build_info_create_dev(struct build_feature_devs_info *binfo,
> +                     enum fpga_id_type type, int feature_nr, const char *name)
> +{
> +       struct platform_device *fdev;
> +       struct resource *res;
> +       struct feature_platform_data *pdata;
> +       int ret;
> +
> +       /* we will create a new device, commit current device first */
> +       ret = build_info_commit_dev(binfo);
> +       if (ret)
> +               return ret;
> +
> +       /*
> +        * we use -ENODEV as the initialization indicator which indicates
> +        * whether the id need to be reclaimed
> +        */
> +       fdev = binfo->feature_dev = platform_device_alloc(name, -ENODEV);
> +       if (!fdev)
> +               return -ENOMEM;
> +
> +       fdev->id = alloc_fpga_id(type, &fdev->dev);
> +       if (fdev->id < 0)
> +               return fdev->id;
> +
> +       fdev->dev.parent = &binfo->parent_dev->dev;
> +
> +       /*
> +        * we do not need to care for the memory which is associated with
> +        * the platform device. After calling platform_device_unregister(),
> +        * it will be automatically freed by device's release() callback,
> +        * platform_device_release().
> +        */
> +       pdata = feature_platform_data_alloc_and_init(fdev, feature_nr);
> +       if (!pdata)
> +               return -ENOMEM;
> +
> +       /*
> +        * the count should be initialized to 0 to make sure
> +        *__fpga_port_enable() following __fpga_port_disable()
> +        * works properly for port device.
> +        * and it should always be 0 for fme device.
> +        */
> +       WARN_ON(pdata->disable_count);
> +
> +       fdev->dev.platform_data = pdata;
> +       fdev->num_resources = feature_nr;
> +       fdev->resource = kcalloc(feature_nr, sizeof(*res), GFP_KERNEL);
> +       if (!fdev->resource)
> +               return -ENOMEM;
> +
> +       return 0;
> +}
> +
> +static int remove_feature_dev(struct device *dev, void *data)
> +{
> +       struct platform_device *pdev = to_platform_device(dev);
> +
> +       platform_device_unregister(pdev);
> +       return 0;
> +}
> +
> +static int remove_parent_dev(struct device *dev, void *data)
> +{
> +       /* remove platform devices attached in the parent device */
> +       device_for_each_child(dev, NULL, remove_feature_dev);
> +       fpga_dev_destroy(to_fpga_dev(dev));
> +       return 0;
> +}
> +
> +static void remove_all_devs(struct pci_dev *pdev)
> +{
> +       /* remove parent device and all its children. */
> +       device_for_each_child(&pdev->dev, NULL, remove_parent_dev);
> +}
> +
> +static void build_info_free(struct build_feature_devs_info *binfo)
> +{
> +       if (!IS_ERR_OR_NULL(binfo->parent_dev))
> +               remove_all_devs(binfo->pdev);
> +
> +       /*
> +        * it is a valid id, free it. See comments in
> +        * build_info_create_dev()
> +        */
> +       if (binfo->feature_dev && binfo->feature_dev->id >= 0)
> +               free_fpga_id(feature_dev_id_type(binfo->feature_dev),
> +                            binfo->feature_dev->id);
> +
> +       platform_device_put(binfo->feature_dev);
> +
> +       devm_kfree(&binfo->pdev->dev, binfo);
> +}
> +
> +#define FEATURE_TYPE_AFU       0x1
> +#define FEATURE_TYPE_PRIVATE   0x3
> +
> +/* FME and PORT GUID are fixed */
> +#define FEATURE_FME_GUID "f9e17764-38f0-82fe-e346-524ae92aafbf"
> +#define FEATURE_PORT_GUID "6b355b87-b06c-9642-eb42-8d139398b43a"
> +
> +static bool feature_is_fme(struct feature_afu_header *afu_hdr)
> +{
> +       uuid_le u;
> +
> +       uuid_le_to_bin(FEATURE_FME_GUID, &u);
> +
> +       return !uuid_le_cmp(u, afu_hdr->guid);
> +}
> +
> +static bool feature_is_port(struct feature_afu_header *afu_hdr)
> +{
> +       uuid_le u;
> +
> +       uuid_le_to_bin(FEATURE_PORT_GUID, &u);
> +
> +       return !uuid_le_cmp(u, afu_hdr->guid);
> +}
> +
> +/*
> + * UAFU GUID is dynamic as it can be changed after FME downloads different
> + * Green Bitstream to the port, so we treat the unknown GUIDs which are
> + * attached on port's feature list as UAFU.
> + */
> +static bool feature_is_UAFU(struct build_feature_devs_info *binfo)
> +{
> +       if (!binfo->feature_dev ||
> +             feature_dev_id_type(binfo->feature_dev) != PORT_ID)
> +               return false;
> +
> +       return true;
> +}
> +
> +static void
> +build_info_add_sub_feature(struct build_feature_devs_info *binfo,
> +                          int feature_id, const char *feature_name,
> +                          resource_size_t resource_size, void __iomem *start)
> +{
> +
> +       struct platform_device *fdev = binfo->feature_dev;
> +       struct feature_platform_data *pdata = dev_get_platdata(&fdev->dev);
> +       struct resource *res = &fdev->resource[feature_id];
> +
> +       res->start = pci_resource_start(binfo->pdev, binfo->current_bar) +
> +               start - binfo->ioaddr;
> +       res->end = res->start + resource_size - 1;
> +       res->flags = IORESOURCE_MEM;
> +       res->name = feature_name;
> +
> +       feature_platform_data_add(pdata, feature_id,
> +                                 feature_name, feature_id, start);
> +}
> +
> +struct feature_info {
> +       const char *name;
> +       resource_size_t resource_size;
> +       int feature_index;
> +};
> +
> +/* indexed by fme feature IDs which are defined in 'enum fme_feature_id'. */
> +static struct feature_info fme_features[] = {
> +       {
> +               .name = FME_FEATURE_HEADER,
> +               .resource_size = sizeof(struct feature_fme_header),
> +               .feature_index = FME_FEATURE_ID_HEADER,
> +       },
> +       {
> +               .name = FME_FEATURE_THERMAL_MGMT,
> +               .resource_size = sizeof(struct feature_fme_thermal),
> +               .feature_index = FME_FEATURE_ID_THERMAL_MGMT,
> +       },
> +       {
> +               .name = FME_FEATURE_POWER_MGMT,
> +               .resource_size = sizeof(struct feature_fme_power),
> +               .feature_index = FME_FEATURE_ID_POWER_MGMT,
> +       },
> +       {
> +               .name = FME_FEATURE_GLOBAL_PERF,
> +               .resource_size = sizeof(struct feature_fme_gperf),
> +               .feature_index = FME_FEATURE_ID_GLOBAL_PERF,
> +       },
> +       {
> +               .name = FME_FEATURE_GLOBAL_ERR,
> +               .resource_size = sizeof(struct feature_fme_err),
> +               .feature_index = FME_FEATURE_ID_GLOBAL_ERR,
> +       },
> +       {
> +               .name = FME_FEATURE_PR_MGMT,
> +               .resource_size = sizeof(struct feature_fme_pr),
> +               .feature_index = FME_FEATURE_ID_PR_MGMT,
> +       }
> +};
> +
> +/* indexed by port feature IDs which are defined in 'enum port_feature_id'. */
> +static struct feature_info port_features[] = {
> +       {
> +               .name = PORT_FEATURE_HEADER,
> +               .resource_size = sizeof(struct feature_port_header),
> +               .feature_index = PORT_FEATURE_ID_HEADER,
> +       },
> +       {
> +               .name = PORT_FEATURE_ERR,
> +               .resource_size = sizeof(struct feature_port_error),
> +               .feature_index = PORT_FEATURE_ID_ERROR,
> +       },
> +       {
> +               .name = PORT_FEATURE_UMSG,
> +               .resource_size = sizeof(struct feature_port_umsg),
> +               .feature_index = PORT_FEATURE_ID_UMSG,
> +       },
> +       {
> +               /* This feature isn't available for now */
> +               .name = PORT_FEATURE_PR,
> +               .resource_size = 0,
> +               .feature_index = PORT_FEATURE_ID_PR,
> +       },
> +       {
> +               .name = PORT_FEATURE_STP,
> +               .resource_size = sizeof(struct feature_port_stp),
> +               .feature_index = PORT_FEATURE_ID_STP,
> +       },
> +       {
> +               /*
> +                * For User AFU feature, its region size is not fixed, but
> +                * reported by register PortCapability.mmio_size. Resource
> +                * size of UAFU will be set while parse port device.
> +                */
> +               .name = PORT_FEATURE_UAFU,
> +               .resource_size = 0,
> +               .feature_index = PORT_FEATURE_ID_UAFU,
> +       },
> +};
> +
> +static int
> +create_feature_instance(struct build_feature_devs_info *binfo,
> +                       void __iomem *start, struct feature_info *finfo)
> +{
> +       if (binfo->ioend - start < finfo->resource_size)
> +               return -EINVAL;
> +
> +       build_info_add_sub_feature(binfo, finfo->feature_index, finfo->name,
> +                                  finfo->resource_size, start);
> +       return 0;
> +}
> +
> +static int parse_feature_fme(struct build_feature_devs_info *binfo,
> +                            void __iomem *start)
> +{
> +       struct cci_drvdata *drvdata = dev_get_drvdata(&binfo->pdev->dev);
> +       int ret;
> +
> +       ret = build_info_create_dev(binfo, FME_ID, fme_feature_num(),
> +                                       FPGA_FEATURE_DEV_FME);
> +       if (ret)
> +               return ret;
> +
> +       if (drvdata->fme_dev) {
> +               dev_err(&binfo->pdev->dev, "Multiple FMEs are detected.\n");
> +               return -EINVAL;
> +       }
> +
> +       return create_feature_instance(binfo, start,
> +                                      &fme_features[FME_FEATURE_ID_HEADER]);
> +}
> +
> +static int parse_feature_fme_private(struct build_feature_devs_info *binfo,
> +                                    struct feature_header *hdr)
> +{
> +       struct feature_header header;
> +
> +       header.csr = readq(hdr);
> +
> +       if (header.id >= ARRAY_SIZE(fme_features)) {
> +               dev_info(&binfo->pdev->dev, "FME feature id %x is not supported yet.\n",
> +                        header.id);
> +               return 0;
> +       }
> +
> +       return create_feature_instance(binfo, hdr, &fme_features[header.id]);
> +}
> +
> +static int parse_feature_port(struct build_feature_devs_info *binfo,
> +                            void __iomem *start)
> +{
> +       int ret;
> +
> +       ret = build_info_create_dev(binfo, PORT_ID, port_feature_num(),
> +                                       FPGA_FEATURE_DEV_PORT);
> +       if (ret)
> +               return ret;
> +
> +       return create_feature_instance(binfo, start,
> +                                      &port_features[PORT_FEATURE_ID_HEADER]);
> +}
> +
> +static void enable_port_uafu(struct build_feature_devs_info *binfo,
> +                            void __iomem *start)
> +{
> +       enum port_feature_id id = PORT_FEATURE_ID_UAFU;
> +       struct feature_port_header *port_hdr;
> +       struct feature_port_capability capability;
> +
> +       port_hdr = (struct feature_port_header *)start;
> +       capability.csr = readq(&port_hdr->capability);
> +       port_features[id].resource_size = capability.mmio_size << 10;
> +
> +       /*
> +        * To enable User AFU, driver needs to clear reset bit on related port,
> +        * otherwise the mmio space of this user AFU will be invalid.
> +        */
> +       if (port_features[id].resource_size)
> +               fpga_port_reset(binfo->feature_dev);
> +}
> +
> +static int parse_feature_port_private(struct build_feature_devs_info *binfo,
> +                                     struct feature_header *hdr)
> +{
> +       struct feature_header header;
> +       enum port_feature_id id;
> +
> +       header.csr = readq(hdr);
> +       /*
> +        * the region of port feature id is [0x10, 0x13], + 1 to reserve 0
> +        * which is dedicated for port-hdr.
> +        */
> +       id = (header.id & 0x000f) + 1;
> +
> +       if (id >= ARRAY_SIZE(port_features)) {
> +               dev_info(&binfo->pdev->dev, "Port feature id %x is not supported yet.\n",
> +                        header.id);
> +               return 0;
> +       }
> +
> +       return create_feature_instance(binfo, hdr, &port_features[id]);
> +}
> +
> +static int parse_feature_port_uafu(struct build_feature_devs_info *binfo,
> +                                struct feature_header *hdr)
> +{
> +       enum port_feature_id id = PORT_FEATURE_ID_UAFU;
> +       int ret;
> +
> +       if (port_features[id].resource_size) {
> +               ret = create_feature_instance(binfo, hdr, &port_features[id]);
> +               port_features[id].resource_size = 0;
> +       } else {
> +               dev_err(&binfo->pdev->dev, "the uafu feature header is mis-configured.\n");
> +               ret = -EINVAL;
> +       }
> +
> +       return ret;
> +}
> +
> +static int parse_feature_afus(struct build_feature_devs_info *binfo,
> +                             struct feature_header *hdr)
> +{
> +       int ret;
> +       struct feature_afu_header *afu_hdr, header;
> +       void __iomem *start;
> +       void __iomem *end = binfo->ioend;
> +
> +       start = hdr;
> +       for (; start < end; start += header.next_afu) {
> +               if (end - start < (sizeof(*afu_hdr) + sizeof(*hdr)))
> +                       return -EINVAL;
> +
> +               hdr = start;
> +               afu_hdr = (struct feature_afu_header *) (hdr + 1);
> +               header.csr = readq(&afu_hdr->csr);
> +
> +               if (feature_is_fme(afu_hdr)) {
> +                       ret = parse_feature_fme(binfo, hdr);
> +                       binfo->pfme_hdr = hdr;
> +                       if (ret)
> +                               return ret;
> +               } else if (feature_is_port(afu_hdr)) {
> +                       ret = parse_feature_port(binfo, hdr);
> +                       enable_port_uafu(binfo, hdr);
> +                       if (ret)
> +                               return ret;
> +               } else if (feature_is_UAFU(binfo)) {
> +                       ret = parse_feature_port_uafu(binfo, hdr);
> +                       if (ret)
> +                               return ret;
> +               } else
> +                       dev_info(&binfo->pdev->dev, "AFU GUID %pUl is not supported yet.\n",
> +                                afu_hdr->guid.b);
> +
> +               if (!header.next_afu)
> +                       break;
> +       }
> +
> +       return 0;
> +}
> +
> +static int parse_feature_private(struct build_feature_devs_info *binfo,
> +                                struct feature_header *hdr)
> +{
> +       struct feature_header header;
> +
> +       header.csr = readq(hdr);
> +
> +       if (!binfo->feature_dev) {
> +               dev_err(&binfo->pdev->dev, "the private feature %x does not belong to any AFU.\n",
> +                       header.id);
> +               return -EINVAL;
> +       }
> +
> +       switch (feature_dev_id_type(binfo->feature_dev)) {
> +       case FME_ID:
> +               return parse_feature_fme_private(binfo, hdr);
> +       case PORT_ID:
> +               return parse_feature_port_private(binfo, hdr);
> +       default:
> +               dev_info(&binfo->pdev->dev, "private feature %x belonging to AFU %s is not supported yet.\n",
> +                        header.id, binfo->feature_dev->name);
> +       }
> +       return 0;
> +}
> +
> +static int parse_feature(struct build_feature_devs_info *binfo,
> +                        struct feature_header *hdr)
> +{
> +       struct feature_header header;
> +       int ret = 0;
> +
> +       header.csr = readq(hdr);
> +
> +       switch (header.type) {
> +       case FEATURE_TYPE_AFU:
> +               ret = parse_feature_afus(binfo, hdr);
> +               break;
> +       case FEATURE_TYPE_PRIVATE:
> +               ret = parse_feature_private(binfo, hdr);
> +               break;
> +       default:
> +               dev_info(&binfo->pdev->dev,
> +                        "Feature Type %x is not supported.\n", hdr->type);
> +       };
> +
> +       return ret;
> +}
> +
> +static int
> +parse_feature_list(struct build_feature_devs_info *binfo, void __iomem *start)
> +{
> +       struct feature_header *hdr, header;
> +       void __iomem *end = binfo->ioend;
> +       int ret = 0;
> +
> +       for (; start < end; start += header.next_header_offset) {
> +               if (end - start < sizeof(*hdr)) {
> +                       dev_err(&binfo->pdev->dev, "The region is too small to contain a feature.\n");
> +                       ret =  -EINVAL;
> +                       break;
> +               }
> +
> +               hdr = (struct feature_header *)start;
> +               ret = parse_feature(binfo, hdr);
> +               if (ret)
> +                       break;
> +
> +               header.csr = readq(hdr);
> +               if (!header.next_header_offset)
> +                       break;
> +       }
> +
> +       return ret;
> +}
> +
> +static int parse_ports_from_fme(struct build_feature_devs_info *binfo)
> +{
> +       struct feature_fme_header *fme_hdr;
> +       struct feature_fme_port port;
> +       int i = 0, ret = 0;
> +
> +       if (binfo->pfme_hdr == NULL) {
> +               dev_dbg(&binfo->pdev->dev, "VF is detected.\n");
> +               return ret;
> +       }
> +
> +       fme_hdr = binfo->pfme_hdr;
> +
> +       do {
> +               port.csr = readq(&fme_hdr->port[i]);
> +               if (!port.port_implemented)
> +                       break;
> +
> +               ret = parse_switch_to(binfo, port.port_bar);
> +               if (ret)
> +                       break;
> +
> +               ret = parse_feature_list(binfo,
> +                               binfo->ioaddr + port.port_offset);
> +               if (ret)
> +                       break;
> +       } while (++i < MAX_FPGA_PORT_NUM);
> +
> +       return ret;
> +}
> +
> +static int create_init_drvdata(struct pci_dev *pdev)
> +{
> +       struct cci_drvdata *drvdata;
> +
> +       drvdata = devm_kzalloc(&pdev->dev, sizeof(*drvdata), GFP_KERNEL);
> +       if (!drvdata)
> +               return -ENOMEM;
> +
> +       mutex_init(&drvdata->lock);
> +       INIT_LIST_HEAD(&drvdata->port_dev_list);
> +       INIT_LIST_HEAD(&drvdata->regions);
> +
> +       dev_set_drvdata(&pdev->dev, drvdata);
> +       return 0;
> +}
> +
> +static void destroy_drvdata(struct pci_dev *pdev)
> +{
> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +
> +       if (drvdata->fme_dev) {
> +               /* fme device should be unregistered first. */
> +               WARN_ON(device_is_registered(drvdata->fme_dev));
> +               free_fpga_id(FME_ID, to_platform_device(drvdata->fme_dev)->id);
> +               put_device(drvdata->fme_dev);
> +       }
> +
> +       cci_pci_remove_port_devs(pdev);
> +       cci_pci_release_regions(pdev);
> +       dev_set_drvdata(&pdev->dev, NULL);
> +       devm_kfree(&pdev->dev, drvdata);
> +}
> +
> +static int cci_pci_create_feature_devs(struct pci_dev *pdev)
> +{
> +       struct build_feature_devs_info *binfo;
> +       int ret;
> +
> +       binfo = build_info_alloc_and_init(pdev);
> +       if (!binfo)
> +               return -ENOMEM;
> +
> +       binfo->parent_dev = fpga_dev_create(&pdev->dev, INTEL_FPGA_DEV);
> +       if (IS_ERR(binfo->parent_dev)) {
> +               ret = PTR_ERR(binfo->parent_dev);
> +               goto free_binfo_exit;
> +       }
> +
> +       ret = parse_start(binfo);
> +       if (ret)
> +               goto free_binfo_exit;
> +
> +       ret = parse_feature_list(binfo, binfo->ioaddr);
> +       if (ret)
> +               goto free_binfo_exit;
> +
> +       ret = parse_ports_from_fme(binfo);
> +       if (ret)
> +               goto free_binfo_exit;
> +
> +       ret = build_info_commit_dev(binfo);
> +       if (ret)
> +               goto free_binfo_exit;
> +
> +       /*
> +        * everything is okay, reset ->parent_dev to stop it being
> +        * freed by build_info_free()
> +        */
> +       binfo->parent_dev = NULL;
> +
> +free_binfo_exit:
> +       build_info_free(binfo);
> +       return ret;
> +}
> +
>  /* PCI Device ID */
>  #define PCIe_DEVICE_ID_PF_INT_5_X      0xBCBD
>  #define PCIe_DEVICE_ID_PF_INT_6_X      0xBCC0
> @@ -81,9 +898,18 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
>                 goto release_region_exit;
>         }
>
> -       /* TODO: create and add the platform device per feature list */
> +       ret = create_init_drvdata(pcidev);
> +       if (ret)
> +               goto release_region_exit;
> +
> +       ret = cci_pci_create_feature_devs(pcidev);
> +       if (ret)
> +               goto destroy_drvdata_exit;
> +
>         return 0;
>
> +destroy_drvdata_exit:
> +       destroy_drvdata(pcidev);
>  release_region_exit:
>         pci_release_regions(pcidev);
>  disable_error_report_exit:
> @@ -94,6 +920,8 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
>
>  static void cci_pci_remove(struct pci_dev *pcidev)
>  {
> +       remove_all_devs(pcidev);
> +       destroy_drvdata(pcidev);
>         pci_release_regions(pcidev);
>         pci_disable_pcie_error_reporting(pcidev);
>         pci_disable_device(pcidev);
> @@ -108,14 +936,23 @@ static void cci_pci_remove(struct pci_dev *pcidev)
>
>  static int __init ccidrv_init(void)
>  {
> +       int ret;
> +
>         pr_info("Intel(R) FPGA PCIe Driver: Version %s\n", DRV_VERSION);
>
> -       return pci_register_driver(&cci_pci_driver);
> +       fpga_ids_init();
> +
> +       ret = pci_register_driver(&cci_pci_driver);
> +       if (ret)
> +               fpga_ids_destroy();
> +
> +       return ret;
>  }
>
>  static void __exit ccidrv_exit(void)
>  {
>         pci_unregister_driver(&cci_pci_driver);
> +       fpga_ids_destroy();
>  }
>
>  module_init(ccidrv_init);
> --
> 1.8.3.1
>

Re: [PATCH v2 01/22] docs: fpga: add a document for Intel FPGA driver overview

[Alan Tull]

On Fri, Jul 14, 2017 at 6:59 PM, Luebbers, Enno <enno.luebbers@intel.com> wrote:
> On Thu, Jul 13, 2017 at 12:25:20PM +0800, Wu Hao wrote:
>> On Wed, Jul 12, 2017 at 09:51:32AM -0500, Alan Tull wrote:
>> > On Sun, Jun 25, 2017 at 8:51 PM, Wu Hao <hao.wu@intel.com> wrote:
>> >
>> > Hi Hao,
>> >
>> > > Add a document for Intel FPGA driver overview.
>> > >
>> > > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
>> > > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> > > Signed-off-by: Wu Hao <hao.wu@intel.com>
>> > > ----
>> > > v2: added FME fpga-mgr/bridge/region platform driver to driver organization.
>> > >     updated open discussion per current implementation.
>> > >     fixed some typos.
>> > > ---
>> > >  Documentation/fpga/intel-fpga.txt | 256 ++++++++++++++++++++++++++++++++++++++
>> > >  1 file changed, 256 insertions(+)
>> > >  create mode 100644 Documentation/fpga/intel-fpga.txt
>> > >
>> > > diff --git a/Documentation/fpga/intel-fpga.txt b/Documentation/fpga/intel-fpga.txt
>> > > new file mode 100644
>> > > index 0000000..4a29470
>> > > --- /dev/null
>> > > +++ b/Documentation/fpga/intel-fpga.txt
>> > > @@ -0,0 +1,256 @@
>> > > +===============================================================================
>> > > +                    Intel FPGA driver Overview
>> > > +-------------------------------------------------------------------------------
>> > > +                Enno Luebbers <enno.luebbers@intel.com>
>> > > +                Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> > > +                Wu Hao <hao.wu@intel.com>
>> > > +
>> > > +The Intel FPGA driver provides interfaces for userspace applications to
>> > > +configure, enumerate, open, and access FPGA accelerators on platforms equipped
>> > > +with Intel(R) FPGA PCIe based solutions and enables system level management
>> > > +functions such as FPGA reconfiguration, power management, and virtualization.
>> > > +
>> > > +HW Architecture
>> > > +===============
>> > > +From the OS's point of view, the FPGA hardware appears as a regular PCIe device.
>> > > +The FPGA device memory is organized using a predefined data structure (Device
>> > > +Feature List). Features supported by the particular FPGA device are exposed
>> > > +through these data structures, as illustrated below:
>> > > +
>> > > +  +-------------------------------+  +-------------+
>> > > +  |              PF               |  |     VF      |
>> > > +  +-------------------------------+  +-------------+
>> > > +      ^            ^         ^              ^
>> > > +      |            |         |              |
>> > > ++-----|------------|---------|--------------|-------+
>> > > +|     |            |         |              |       |
>> > > +|  +-----+     +-------+ +-------+      +-------+   |
>> > > +|  | FME |     | Port0 | | Port1 |      | Port2 |   |
>> > > +|  +-----+     +-------+ +-------+      +-------+   |
>> > > +|                  ^         ^              ^       |
>> > > +|                  |         |              |       |
>> > > +|              +-------+ +------+       +-------+   |
>> > > +|              |  AFU  | |  AFU |       |  AFU  |   |
>> > > +|              +-------+ +------+       +-------+   |
>> > > +|                                                   |
>> > > +|                 FPGA PCIe Device                  |
>> > > ++---------------------------------------------------+
>> > > +
>> > > +The driver supports PCIe SR-IOV to create virtual functions (VFs) which can be
>> > > +used to assign individual accelerators to virtual machines.
>> > > +
>> > > +FME (FPGA Management Engine)
>> > > +============================
>> > > +The FPGA Management Engine performs power and thermal management, error
>> > > +reporting, reconfiguration, performance reporting, and other infrastructure
>> > > +functions. Each FPGA has one FME, which is always accessed through the physical
>> > > +function (PF).
>> > > +
>> > > +User-space applications can acquire exclusive access to the FME using open(),
>> > > +and release it using close().
>> > > +
>> > > +The following functions are exposed through ioctls:
>> > > +
>> > > +       Get driver API version (FPGA_GET_API_VERSION)
>> > > +       Check for extensions (FPGA_CHECK_EXTENSION)
>> > > +       Assign port to PF (FPGA_FME_PORT_ASSIGN)
>> > > +       Release port from PF (FPGA_FME_PORT_RELEASE)
>> > > +       Program bitstream (FPGA_FME_PORT_PR)
>> > > +
>> >
>> > I was hoping the API mailing list might have an opinion about this,
>> > but I think adding ioctls to the kernel is discouraged.  Could these
>> > be sysfs?
>>
>> Hi Alan,
>>
>> As you see below, we have defined a lot of sysfs interface for device
>> info, attributes and simple control operations. But for some actions
>> which requires complex inputs/outputs parameters (e.g a struct with
>> multiple items) with userspace, ioctls are used. I feel in such cases,
>> ioctls seem more suitable than sysfs.
>>
>
> Also, we're thinking that some operations require that you first "acquire
> ownership" of the respective device, which I believe maps more easily to
> open() and ioctls than sysfs.
>
> Thanks
> - Enno

Yes, sysfs doesn't implement open/close so there's no clean way to
have one thing in userspace that owns the interface.  It's a shame.

Alan

RE: [PATCH v2 12/22] fpga: intel: fme: add header sub feature support

[Wu, Hao]

> On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu@intel.com> wrote:
> 
> Hi Hao,
> 
> I'm making my way through this (very large) patchset.  Some minor
> comments below.
> 

Hi Alan

Thanks for your review. : )

> > From: Kang Luwei <luwei.kang@intel.com>
> >
> > The header register set is always present for FPGA Management Engine (FME),
> > this patch implements init and uinit function for header sub feature and
> > introduce several read-only sysfs interfaces for the capability and status.
> >
> > Sysfs interfaces:
> > * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/ports_num
> >   Read-only. Number of ports implemented
> >
> > * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/bitstream_id
> >   Read-only. Blue Bitstream identifier number
> 
> Blue and Green bitstreams are an Intel implementation terminology.  I
> see that you've defined them in drivers/fpga, but it will be helpful
> to add in "static region" and "partial reconfiguration region" added
> in any API documentation files that use the green/blue terminology to
> keep it accessible.	
> 

Sure, thanks for your suggestion, will update it like this.

* /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/bitstream_id
  Read-only. Blue Bitstream (static FPGA region) identifier number

* /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/bitstream_metadata
  Read-only. Blue Bitstream (static FPGA region) meta data

> >
> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > ---
> > v2: add sysfs documentation
> > ---
> >  .../ABI/testing/sysfs-platform-intel-fpga-fme      | 19 ++++++++
> >  drivers/fpga/intel-feature-dev.h                   |  3 ++
> >  drivers/fpga/intel-fme-main.c                      | 55 ++++++++++++++++++++++
> >  3 files changed, 77 insertions(+)
> >  create mode 100644 Documentation/ABI/testing/sysfs-platform-intel-fpga-
> fme
> >
> > diff --git a/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme
> b/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme
> > new file mode 100644
> > index 0000000..783cfa9
> > --- /dev/null
> > +++ b/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme
> > @@ -0,0 +1,19 @@
> > +What:          /sys/bus/platform/devices/intel-fpga-fme.0/ports_num
> > +Date:          June 2017
> > +KernelVersion:  4.12
> > +Contact:       Wu Hao <hao.wu@intel.com>
> > +Description:   Read-only. One Intel FPGA device may have more than 1
> > +               port/Accelerator Function Unit (AFU). It returns the
> > +               number of ports on the FPGA device when read it.
> > +
> > +What:          /sys/bus/platform/devices/intel-fpga-fme.0/bitstream_id
> > +Date:          June 2017
> > +KernelVersion:  4.12
> > +Contact:       Wu Hao <hao.wu@intel.com>
> > +Description:   Read-only. It returns Blue Bitstream identifier number.
> 
> Here

Will update this patch as below.

+Description:   Read-only. It returns Blue Bitstream (static FPGA region)
+               identifier number.

> 
> > +
> > +What:          /sys/bus/platform/devices/intel-fpga-fme.0/bitstream_meta
> > +Date:          June 2017
> > +KernelVersion:  4.12
> > +Contact:       Wu Hao <hao.wu@intel.com>
> > +Description:   Read-only. It returns Blue Bitstream meta data.
> 
> And here

Will update this patch as below.

+Description:   Read-only. It returns Blue Bitstream (static FPGA region)
+               meta data.

Thanks
Hao

RE: [PATCH v2 07/22] fpga: intel: pcie: parse feature list and create platform device for features.

[Wu, Hao]

> On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu@intel.com> wrote:
> > From: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> >
> > Device Feature List structure creates a link list of feature headers
> > within the MMIO space to provide an extensible way of adding features.
> >
> > The Intel FPGA PCIe driver walks through the feature headers to enumerate
> > feature devices, FPGA Management Engine (FME) and FPGA Port for
> Accelerated
> > Function Unit (AFU), and their private sub features. For feature devices,
> > it creates the platform devices and linked the private sub features into
> > their platform data.
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> > Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > ---
> > v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
> >     switched to GPLv2 license.
> >     fixed comments from Moritz Fischer.
> >     fixed kbuild warning, typos and clean up the code.
> > ---
> >  drivers/fpga/Makefile            |   2 +-
> >  drivers/fpga/intel-feature-dev.c | 130 ++++++
> >  drivers/fpga/intel-feature-dev.h | 341 ++++++++++++++++
> >  drivers/fpga/intel-pcie.c        | 841
> ++++++++++++++++++++++++++++++++++++++-
> >  4 files changed, 1311 insertions(+), 3 deletions(-)
> >  create mode 100644 drivers/fpga/intel-feature-dev.c
> >  create mode 100644 drivers/fpga/intel-feature-dev.h
> >
> > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> > index 5613133..ad24b3d 100644
> > --- a/drivers/fpga/Makefile
> > +++ b/drivers/fpga/Makefile
> > @@ -31,4 +31,4 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-
> region.o
> >  # Intel FPGA Support
> >  obj-$(CONFIG_INTEL_FPGA_PCI)           += intel-fpga-pci.o
> >
> > -intel-fpga-pci-objs := intel-pcie.o
> > +intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
> > diff --git a/drivers/fpga/intel-feature-dev.c b/drivers/fpga/intel-feature-dev.c
> > new file mode 100644
> > index 0000000..68f9cba
> > --- /dev/null
> > +++ b/drivers/fpga/intel-feature-dev.c
> > @@ -0,0 +1,130 @@
> > +/*
> > + * Intel FPGA Feature Device Driver
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <luwei.kang@intel.com>
> > + *   Zhang Yi <yi.z.zhang@intel.com>
> > + *   Wu Hao <hao.wu@intel.com>
> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > + *
> > + * This work is licensed under the terms of the GNU GPL version 2. See
> > + * the COPYING file in the top-level directory.
> > + */
> > +
> > +#include "intel-feature-dev.h"
> > +
> > +void feature_platform_data_add(struct feature_platform_data *pdata,
> > +                              int index, const char *name,
> > +                              int resource_index, void __iomem *ioaddr)
> > +{
> > +       WARN_ON(index >= pdata->num);
> > +
> > +       pdata->features[index].name = name;
> > +       pdata->features[index].resource_index = resource_index;
> > +       pdata->features[index].ioaddr = ioaddr;
> > +}
> > +
> > +struct feature_platform_data *
> > +feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
> > +{
> > +       struct feature_platform_data *pdata;
> > +
> > +       pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
> > +       if (pdata) {
> > +               pdata->dev = dev;
> > +               pdata->num = num;
> > +               mutex_init(&pdata->lock);
> > +       }
> > +
> > +       return pdata;
> > +}
> > +
> > +int fme_feature_num(void)
> > +{
> > +       return FME_FEATURE_ID_MAX;
> > +}
> > +
> > +int port_feature_num(void)
> > +{
> > +       return PORT_FEATURE_ID_MAX;
> > +}
> > +
> > +int fpga_port_id(struct platform_device *pdev)
> > +{
> > +       struct feature_port_header *port_hdr;
> > +       struct feature_port_capability capability;
> > +
> > +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                                              PORT_FEATURE_ID_HEADER);
> > +       WARN_ON(!port_hdr);
> > +
> > +       capability.csr = readq(&port_hdr->capability);
> > +       return capability.port_number;
> > +}
> > +EXPORT_SYMBOL_GPL(fpga_port_id);
> > +
> > +/*
> > + * Enable Port by clear the port soft reset bit, which is set by default.
> > + * The User AFU is unable to respond to any MMIO access while in reset.
> > + * __fpga_port_enable function should only be used after
> __fpga_port_disable
> > + * function.
> > + */
> > +void __fpga_port_enable(struct platform_device *pdev)
> > +{
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct feature_port_header *port_hdr;
> > +       struct feature_port_control control;
> > +
> > +       WARN_ON(!pdata->disable_count);
> > +
> > +       if (--pdata->disable_count != 0)
> > +               return;
> > +
> > +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                                              PORT_FEATURE_ID_HEADER);
> > +       WARN_ON(!port_hdr);
> > +
> > +       control.csr = readq(&port_hdr->control);
> > +       control.port_sftrst = 0x0;
> > +       writeq(control.csr, &port_hdr->control);
> > +}
> > +EXPORT_SYMBOL_GPL(__fpga_port_enable);
> > +
> > +#define RST_POLL_INVL 10 /* us */
> > +#define RST_POLL_TIMEOUT 1000 /* us */
> > +
> > +int __fpga_port_disable(struct platform_device *pdev)
> > +{
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct feature_port_header *port_hdr;
> > +       struct feature_port_control control;
> > +
> > +       if (pdata->disable_count++ != 0)
> > +               return 0;
> > +
> > +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                                              PORT_FEATURE_ID_HEADER);
> > +       WARN_ON(!port_hdr);
> > +
> > +       /* Set port soft reset */
> > +       control.csr = readq(&port_hdr->control);
> > +       control.port_sftrst = 0x1;
> > +       writeq(control.csr, &port_hdr->control);
> > +
> > +       /*
> > +        * HW sets ack bit to 1 when all outstanding requests have been drained
> > +        * on this port and minimum soft reset pulse width has elapsed.
> > +        * Driver polls port_soft_reset_ack to determine if reset done by HW.
> > +        */
> > +       if (readq_poll_timeout(&port_hdr->control, control.csr,
> > +                              (control.port_sftrst_ack == 1),
> > +                              RST_POLL_INVL, RST_POLL_TIMEOUT)) {
> > +               dev_err(&pdev->dev, "timeout, fail to reset device\n");
> > +               return -ETIMEDOUT;
> > +       }
> > +
> > +       return 0;
> > +}
> > +EXPORT_SYMBOL_GPL(__fpga_port_disable);
> > diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
> > new file mode 100644
> > index 0000000..f67784a
> > --- /dev/null
> > +++ b/drivers/fpga/intel-feature-dev.h
> > @@ -0,0 +1,341 @@
> > +/*
> > + * Intel FPGA Feature Device Driver Header File
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <luwei.kang@intel.com>
> > + *   Zhang Yi <yi.z.zhang@intel.com>
> > + *   Wu Hao <hao.wu@intel.com>
> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > + *
> > + * This work is licensed under the terms of the GNU GPL version 2. See
> > + * the COPYING file in the top-level directory.
> > + */
> > +
> > +#ifndef __INTEL_FPGA_FEATURE_H
> > +#define __INTEL_FPGA_FEATURE_H
> > +
> > +#include <linux/fs.h>
> > +#include <linux/pci.h>
> > +#include <linux/uuid.h>
> > +#include <linux/delay.h>
> > +#include <linux/iopoll.h>
> > +#include <linux/platform_device.h>
> > +
> > +#ifndef readq
> > +static inline u64 readq(void __iomem *addr)
> > +{
> > +       return readl(addr) + ((u64)readl(addr + 4) << 32);
> > +}
> > +#endif
> > +
> > +#ifndef writeq
> > +static inline void writeq(u64 val, void __iomem *addr)
> > +{
> > +       writel((u32) (val), addr);
> > +       writel((u32) (val >> 32), (addr + 4));
> > +}
> > +#endif
> 
> Hi Hao,
> 
> Was there a reason readq and writeq had to be created for this?  Do
> these get used?
> 

Hi Alan

Thanks for your review.

Driver uses writeq and readq to access HW registers.

Actually this is a problem reported by kbuild against the patch set v1[1].
Writeq/readq are missed in some arch, so I added writeq/readq definitions to
resolve this problem following the same way done by some existing drivers.
After recheck this today, I found I should use linux/io-64-nonatomic-lo-hi.h
instead of rewriting them.  I will fix it in the next version.

[1] http://marc.info/?l=linux-kernel&m=149100396816882&w=2

Thanks
Hao

> Alan

Re: [PATCH v2 01/22] docs: fpga: add a document for Intel FPGA driver overview

[Greg KH]

On Mon, Jul 17, 2017 at 03:14:29PM -0500, Alan Tull wrote:
> > Also, we're thinking that some operations require that you first "acquire
> > ownership" of the respective device, which I believe maps more easily to
> > open() and ioctls than sysfs.
> >
> > Thanks
> > - Enno
> 
> Yes, sysfs doesn't implement open/close so there's no clean way to
> have one thing in userspace that owns the interface.  It's a shame.

No, that was by design :)

Re: [PATCH v2 01/22] docs: fpga: add a document for Intel FPGA driver overview

[Alan Tull]

On Tue, Jul 18, 2017 at 12:22 AM, Greg KH <greg-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> wrote:
> On Mon, Jul 17, 2017 at 03:14:29PM -0500, Alan Tull wrote:
>> > Also, we're thinking that some operations require that you first "acquire
>> > ownership" of the respective device, which I believe maps more easily to
>> > open() and ioctls than sysfs.
>> >
>> > Thanks
>> > - Enno
>>
>> Yes, sysfs doesn't implement open/close so there's no clean way to
>> have one thing in userspace that owns the interface.  It's a shame.
>
> No, that was by design :)

Well that's good to know!  linux-api is copied on this patchset.  I'm
wondering if there is another mailing list that can weigh in on this
new ioctl that is being added to the kernel or if that's the one.

Alan Tull

Re: [PATCH v2 12/22] fpga: intel: fme: add header sub feature support

[Alan Tull]

On Mon, Jul 17, 2017 at 8:17 PM, Wu, Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>> On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>>
>> Hi Hao,
>>
>> I'm making my way through this (very large) patchset.  Some minor
>> comments below.
>>
>
> Hi Alan
>
> Thanks for your review. : )

Hi Hao,

Thanks, this looks good and will be helpful for folks who are new to this.

Alan

>
>> > From: Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> >
>> > The header register set is always present for FPGA Management Engine (FME),
>> > this patch implements init and uinit function for header sub feature and
>> > introduce several read-only sysfs interfaces for the capability and status.
>> >
>> > Sysfs interfaces:
>> > * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/ports_num
>> >   Read-only. Number of ports implemented
>> >
>> > * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/bitstream_id
>> >   Read-only. Blue Bitstream identifier number
>>
>> Blue and Green bitstreams are an Intel implementation terminology.  I
>> see that you've defined them in drivers/fpga, but it will be helpful
>> to add in "static region" and "partial reconfiguration region" added
>> in any API documentation files that use the green/blue terminology to
>> keep it accessible.
>>
>
> Sure, thanks for your suggestion, will update it like this.
>
> * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/bitstream_id
>   Read-only. Blue Bitstream (static FPGA region) identifier number
>
> * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/bitstream_metadata
>   Read-only. Blue Bitstream (static FPGA region) meta data
>
>> >
>> > Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > Signed-off-by: Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
>> > Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > ---
>> > v2: add sysfs documentation
>> > ---
>> >  .../ABI/testing/sysfs-platform-intel-fpga-fme      | 19 ++++++++
>> >  drivers/fpga/intel-feature-dev.h                   |  3 ++
>> >  drivers/fpga/intel-fme-main.c                      | 55 ++++++++++++++++++++++
>> >  3 files changed, 77 insertions(+)
>> >  create mode 100644 Documentation/ABI/testing/sysfs-platform-intel-fpga-
>> fme
>> >
>> > diff --git a/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme
>> b/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme
>> > new file mode 100644
>> > index 0000000..783cfa9
>> > --- /dev/null
>> > +++ b/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme
>> > @@ -0,0 +1,19 @@
>> > +What:          /sys/bus/platform/devices/intel-fpga-fme.0/ports_num
>> > +Date:          June 2017
>> > +KernelVersion:  4.12
>> > +Contact:       Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > +Description:   Read-only. One Intel FPGA device may have more than 1
>> > +               port/Accelerator Function Unit (AFU). It returns the
>> > +               number of ports on the FPGA device when read it.
>> > +
>> > +What:          /sys/bus/platform/devices/intel-fpga-fme.0/bitstream_id
>> > +Date:          June 2017
>> > +KernelVersion:  4.12
>> > +Contact:       Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > +Description:   Read-only. It returns Blue Bitstream identifier number.
>>
>> Here
>
> Will update this patch as below.
>
> +Description:   Read-only. It returns Blue Bitstream (static FPGA region)
> +               identifier number.
>
>>
>> > +
>> > +What:          /sys/bus/platform/devices/intel-fpga-fme.0/bitstream_meta
>> > +Date:          June 2017
>> > +KernelVersion:  4.12
>> > +Contact:       Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > +Description:   Read-only. It returns Blue Bitstream meta data.
>>
>> And here
>
> Will update this patch as below.
>
> +Description:   Read-only. It returns Blue Bitstream (static FPGA region)
> +               meta data.
>
> Thanks
> Hao

Re: [PATCH v2 04/22] fpga: mgr: add region_id to fpga_image_info

[Alan Tull]

On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

Hi Hao,

> This patch adds region_id to fpga_image_info data structure, it
> allows driver to pass region id information to fpga-mgr via
> fpga_image_info for fpga reconfiguration function.
>
> Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> ---
>  include/linux/fpga/fpga-mgr.h | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/include/linux/fpga/fpga-mgr.h b/include/linux/fpga/fpga-mgr.h
> index 66d0e32..b222a57 100644
> --- a/include/linux/fpga/fpga-mgr.h
> +++ b/include/linux/fpga/fpga-mgr.h
> @@ -95,6 +95,7 @@ struct fpga_image_info {
>         struct sg_table *sgt;
>         const char *buf;
>         size_t count;
> +       int region_id;

Please add a line to document this region_id above where all the other
struct members are documented.

Alan

>  #ifdef CONFIG_OF
>         struct device_node *overlay;
>  #endif
> --
> 1.8.3.1
>

Re: [PATCH v2 04/22] fpga: mgr: add region_id to fpga_image_info

[Wu Hao]

On Wed, Jul 26, 2017 at 01:33:53PM -0500, Alan Tull wrote:
> On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> 
> Hi Hao,
> 
> > This patch adds region_id to fpga_image_info data structure, it
> > allows driver to pass region id information to fpga-mgr via
> > fpga_image_info for fpga reconfiguration function.
> >
> > Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > ---
> >  include/linux/fpga/fpga-mgr.h | 1 +
> >  1 file changed, 1 insertion(+)
> >
> > diff --git a/include/linux/fpga/fpga-mgr.h b/include/linux/fpga/fpga-mgr.h
> > index 66d0e32..b222a57 100644
> > --- a/include/linux/fpga/fpga-mgr.h
> > +++ b/include/linux/fpga/fpga-mgr.h
> > @@ -95,6 +95,7 @@ struct fpga_image_info {
> >         struct sg_table *sgt;
> >         const char *buf;
> >         size_t count;
> > +       int region_id;
> 
> Please add a line to document this region_id above where all the other
> struct members are documented.

Hi Alan

Thanks for the comments, I will fix this in the next version.

  * @sgt: scatter/gather table containing FPGA image
  * @buf: contiguous buffer containing FPGA image
  * @count: size of buf
+ * @region_id: id of target region
  * @overlay: Device Tree overlay
  */
 struct fpga_image_info {

Thanks
Hao

> 
> Alan
> 
> >  #ifdef CONFIG_OF
> >         struct device_node *overlay;
> >  #endif
> > --
> > 1.8.3.1
> >

Re: [PATCH v2 02/22] fpga: add FPGA device framework

[Alan Tull]

On Sun, Jun 25, 2017 at 8:51 PM, Wu Hao <hao.wu@intel.com> wrote:

Hi Rob,

I was hoping to pick your brain a bit on a DT question.

> During FPGA device (e.g PCI-based) discovery, platform devices are
> registered for different FPGA function units. But the device node path
> isn't quite friendly to applications.
>
> Consider this case, applications want to access child device's sysfs file
> for some information.
>
> 1) Access using bus-based path (e.g PCI)
>
>   /sys/bus/pci/devices/xxxxx/fpga_func_a.0/sysfs_file
>
>   From the path, it's clear which PCI device is the parent, but not perfect
>   solution for applications. PCI device BDF is not fixed, application may
>   need to search all PCI device to find the actual FPGA Device.
>
> 2) Or access using platform device path
>
>   /sys/bus/platform/devices/fpga_func_a.0/sysfs_file
>
>   Applications find the actual function by name easily, but no information
>   about which fpga device it belongs to. It's quite confusing if multiple
>   FPGA devices are in one system.

There's a proposal for adding sysfs nodes that correspond to each FPGA
device., with the devices located on each FPGA under them.  It makes
it easier to see which device is on which FPGA.

>
> 'FPGA Device' class is introduced to resolve this problem. Each node under
> this class represents a fpga device, which may have one or more child
> devices. Applications only need to search under this FPGA Device class
> folder to find the child device node it needs.
>
> For example, for the platform has 2 fpga devices, each fpga device has
> 3 child devices, the hierarchy looks like this.
>
> Two nodes are under /sys/class/fpga/:
> /sys/class/fpga/fpga.0
> /sys/class/fpga/fpga.1
>
> Each node has 1 function A device and 2 function B devices:
> /sys/class/fpga/fpga.0/func_a.0
> /sys/class/fpga/fpga.0/func_b.0
> /sys/class/fpga/fpga.0/func_b.1
>
> /sys/class/fpga/fpga.1/func_a.1
> /sys/class/fpga/fpga.1/func_b.2
> /sys/class/fpga/fpga.1/func_b.3

I can see the value of having sysfs nodes that correspond to fpga
devices and being able to find devices under them.  I'm thinking what
that would mean for Device Tree when fpga-dev is used on DT enabled
systems.  In Device Tree, what is a fpga-dev?

Currently the DT would have a FPGA bridge corresponding to each FPGA's
hardware bridge and a heirarchy of bridges, regions and devices under
it.  On systems that don't support partial reconfiguration under the
OS (so not main bridge that was controlled by the OS), there would be
a FPGA region, then its child regions, bridges, and devices.

Alan

>
> This following APIs are provided by FPGA device framework:
> * fpga_dev_create
>   Create fpga device under the given parent device.
> * fpga_dev_destroy
>   Destroy fpga device
>
> The following sysfs files are created:
> * /sys/class/fpga/<fpga.x>/name
>   Name of the fpga device.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ---
> v2: added sysfs documentation for fpga-dev class.
>     switched to GPLv2 license.
> ---
>  Documentation/ABI/testing/sysfs-class-fpga-dev |   5 ++
>  drivers/fpga/Kconfig                           |   6 ++
>  drivers/fpga/Makefile                          |   3 +
>  drivers/fpga/fpga-dev.c                        | 118 +++++++++++++++++++++++++
>  include/linux/fpga/fpga-dev.h                  |  31 +++++++
>  5 files changed, 163 insertions(+)
>  create mode 100644 Documentation/ABI/testing/sysfs-class-fpga-dev
>  create mode 100644 drivers/fpga/fpga-dev.c
>  create mode 100644 include/linux/fpga/fpga-dev.h
>
> diff --git a/Documentation/ABI/testing/sysfs-class-fpga-dev b/Documentation/ABI/testing/sysfs-class-fpga-dev
> new file mode 100644
> index 0000000..ce162df
> --- /dev/null
> +++ b/Documentation/ABI/testing/sysfs-class-fpga-dev
> @@ -0,0 +1,5 @@
> +What:          /sys/class/fpga/<device>/name
> +Date:          June 2017
> +KernelVersion: 4.12
> +Contact:       Wu Hao <hao.wu@intel.com>
> +Description:   Name of FPGA device
> diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> index 394c141..ed600d5 100644
> --- a/drivers/fpga/Kconfig
> +++ b/drivers/fpga/Kconfig
> @@ -12,6 +12,12 @@ config FPGA
>           manager drivers.
>
>  if FPGA
> +config FPGA_DEVICE
> +       tristate "FPGA Device Framework"
> +       help
> +         Say Y here if you want support for FPGA Devices from the kernel.
> +         The FPGA Device Framework adds a FPGA device class and provide
> +         interfaces to create FPGA devices.
>
>  config FPGA_REGION
>         tristate "FPGA Region"
> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> index 85b45d0..8950a8f 100644
> --- a/drivers/fpga/Makefile
> +++ b/drivers/fpga/Makefile
> @@ -5,6 +5,9 @@
>  # Core FPGA Manager Framework
>  obj-$(CONFIG_FPGA)                     += fpga-mgr.o
>
> +# FPGA Device Framework
> +obj-$(CONFIG_FPGA_DEVICE)              += fpga-dev.o
> +
>  # FPGA Manager Drivers
>  obj-$(CONFIG_FPGA_MGR_ICE40_SPI)       += ice40-spi.o
>  obj-$(CONFIG_FPGA_MGR_SOCFPGA)         += socfpga.o
> diff --git a/drivers/fpga/fpga-dev.c b/drivers/fpga/fpga-dev.c
> new file mode 100644
> index 0000000..5778ebc
> --- /dev/null
> +++ b/drivers/fpga/fpga-dev.c
> @@ -0,0 +1,118 @@
> +/*
> + * FPGA Device Framework Driver
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * This work is licensed under the terms of the GNU GPL version 2. See
> + * the COPYING file in the top-level directory.
> + */
> +#include <linux/device.h>
> +#include <linux/module.h>
> +#include <linux/slab.h>
> +#include <linux/fpga/fpga-dev.h>
> +
> +static DEFINE_IDA(fpga_dev_ida);
> +static struct class *fpga_dev_class;
> +
> +static ssize_t name_show(struct device *dev,
> +                        struct device_attribute *attr, char *buf)
> +{
> +       struct fpga_dev *fdev = to_fpga_dev(dev);
> +
> +       return sprintf(buf, "%s\n", fdev->name);
> +}
> +static DEVICE_ATTR_RO(name);
> +
> +static struct attribute *fpga_dev_attrs[] = {
> +       &dev_attr_name.attr,
> +       NULL,
> +};
> +ATTRIBUTE_GROUPS(fpga_dev);
> +
> +/**
> + * fpga_dev_create - create a fpga device
> + * @parent: parent device
> + * @name: fpga device name
> + *
> + * Return fpga_dev struct for success, error code otherwise.
> + */
> +struct fpga_dev *fpga_dev_create(struct device *parent, const char *name)
> +{
> +       struct fpga_dev *fdev;
> +       int id, ret = 0;
> +
> +       if (!name || !strlen(name)) {
> +               dev_err(parent, "Attempt to register with no name!\n");
> +               return ERR_PTR(-EINVAL);
> +       }
> +
> +       fdev = kzalloc(sizeof(*fdev), GFP_KERNEL);
> +       if (!fdev)
> +               return ERR_PTR(-ENOMEM);
> +
> +       id = ida_simple_get(&fpga_dev_ida, 0, 0, GFP_KERNEL);
> +       if (id < 0) {
> +               ret = id;
> +               goto error_kfree;
> +       }
> +
> +       fdev->name = name;
> +
> +       device_initialize(&fdev->dev);
> +       fdev->dev.class = fpga_dev_class;
> +       fdev->dev.parent = parent;
> +       fdev->dev.id = id;
> +
> +       ret = dev_set_name(&fdev->dev, "fpga.%d", id);
> +       if (ret)
> +               goto error_device;
> +
> +       ret = device_add(&fdev->dev);
> +       if (ret)
> +               goto error_device;
> +
> +       dev_dbg(fdev->dev.parent, "fpga device [%s] created\n", fdev->name);
> +
> +       return fdev;
> +
> +error_device:
> +       ida_simple_remove(&fpga_dev_ida, id);
> +error_kfree:
> +       kfree(fdev);
> +
> +       return ERR_PTR(ret);
> +}
> +EXPORT_SYMBOL_GPL(fpga_dev_create);
> +
> +static void fpga_dev_release(struct device *dev)
> +{
> +       struct fpga_dev *fdev = to_fpga_dev(dev);
> +
> +       ida_simple_remove(&fpga_dev_ida, fdev->dev.id);
> +       kfree(fdev);
> +}
> +
> +static int __init fpga_dev_class_init(void)
> +{
> +       pr_info("FPGA Device framework\n");
> +
> +       fpga_dev_class = class_create(THIS_MODULE, "fpga");
> +       if (IS_ERR(fpga_dev_class))
> +               return PTR_ERR(fpga_dev_class);
> +
> +       fpga_dev_class->dev_groups = fpga_dev_groups;
> +       fpga_dev_class->dev_release = fpga_dev_release;
> +
> +       return 0;
> +}
> +
> +static void __exit fpga_dev_class_exit(void)
> +{
> +       class_destroy(fpga_dev_class);
> +}
> +
> +MODULE_DESCRIPTION("FPGA Device framework");
> +MODULE_LICENSE("GPL v2");
> +
> +subsys_initcall(fpga_dev_class_init);
> +module_exit(fpga_dev_class_exit);
> diff --git a/include/linux/fpga/fpga-dev.h b/include/linux/fpga/fpga-dev.h
> new file mode 100644
> index 0000000..ee693cc
> --- /dev/null
> +++ b/include/linux/fpga/fpga-dev.h
> @@ -0,0 +1,31 @@
> +/*
> + * FPGA Device Driver Header
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * This work is licensed under the terms of the GNU GPL version 2. See
> + * the COPYING file in the top-level directory.
> + */
> +#ifndef _LINUX_FPGA_DEV_H
> +#define _LINUX_FPGA_DEV_H
> +
> +/**
> + * struct fpga_dev - fpga device structure
> + * @name: name of fpga device
> + * @dev: fpga device
> + */
> +struct fpga_dev {
> +       const char *name;
> +       struct device dev;
> +};
> +
> +#define to_fpga_dev(d) container_of(d, struct fpga_dev, dev)
> +
> +struct fpga_dev *fpga_dev_create(struct device *parent, const char *name);
> +
> +static inline void fpga_dev_destroy(struct fpga_dev *fdev)
> +{
> +       device_unregister(&fdev->dev);
> +}
> +
> +#endif
> --
> 1.8.3.1
>

Re: [PATCH v2 02/22] fpga: add FPGA device framework

[Alan Tull]

On Sun, Jun 25, 2017 at 8:51 PM, Wu Hao <hao.wu@intel.com> wrote:
> During FPGA device (e.g PCI-based) discovery, platform devices are
> registered for different FPGA function units. But the device node path
> isn't quite friendly to applications.
>

Hi Hao,

Please add a document for fpga-dev under Documentation/fpga to help
future users.  The header of this patch explains things pretty well
and could be used with a little formatting for the document.

Alan

Re: [PATCH v2 02/22] fpga: add FPGA device framework

[Rob Herring]

On Thu, Jul 27, 2017 at 11:35 AM, Alan Tull <atull-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> On Sun, Jun 25, 2017 at 8:51 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>
> Hi Rob,
>
> I was hoping to pick your brain a bit on a DT question.
>
>> During FPGA device (e.g PCI-based) discovery, platform devices are
>> registered for different FPGA function units. But the device node path
>> isn't quite friendly to applications.
>>
>> Consider this case, applications want to access child device's sysfs file
>> for some information.
>>
>> 1) Access using bus-based path (e.g PCI)
>>
>>   /sys/bus/pci/devices/xxxxx/fpga_func_a.0/sysfs_file
>>
>>   From the path, it's clear which PCI device is the parent, but not perfect
>>   solution for applications. PCI device BDF is not fixed, application may
>>   need to search all PCI device to find the actual FPGA Device.
>>
>> 2) Or access using platform device path
>>
>>   /sys/bus/platform/devices/fpga_func_a.0/sysfs_file
>>
>>   Applications find the actual function by name easily, but no information
>>   about which fpga device it belongs to. It's quite confusing if multiple
>>   FPGA devices are in one system.
>
> There's a proposal for adding sysfs nodes that correspond to each FPGA
> device., with the devices located on each FPGA under them.  It makes
> it easier to see which device is on which FPGA.

Makes sense.

>> 'FPGA Device' class is introduced to resolve this problem. Each node under
>> this class represents a fpga device, which may have one or more child
>> devices. Applications only need to search under this FPGA Device class
>> folder to find the child device node it needs.
>>
>> For example, for the platform has 2 fpga devices, each fpga device has
>> 3 child devices, the hierarchy looks like this.
>>
>> Two nodes are under /sys/class/fpga/:
>> /sys/class/fpga/fpga.0
>> /sys/class/fpga/fpga.1
>>
>> Each node has 1 function A device and 2 function B devices:
>> /sys/class/fpga/fpga.0/func_a.0
>> /sys/class/fpga/fpga.0/func_b.0
>> /sys/class/fpga/fpga.0/func_b.1
>>
>> /sys/class/fpga/fpga.1/func_a.1
>> /sys/class/fpga/fpga.1/func_b.2
>> /sys/class/fpga/fpga.1/func_b.3

A class is generally what is the function of the device, not how it is
attached. Seems like what you want here is a new bus type if the
existing PCI and platform bus types don't work.

>
> I can see the value of having sysfs nodes that correspond to fpga
> devices and being able to find devices under them.  I'm thinking what
> that would mean for Device Tree when fpga-dev is used on DT enabled
> systems.  In Device Tree, what is a fpga-dev?

Just properly setting the parent struct device on the functions should
be enough to figure out which function is in which fpga. I don't see
why a new class is needed.

> Currently the DT would have a FPGA bridge corresponding to each FPGA's
> hardware bridge and a heirarchy of bridges, regions and devices under
> it.  On systems that don't support partial reconfiguration under the
> OS (so not main bridge that was controlled by the OS), there would be
> a FPGA region, then its child regions, bridges, and devices.

The FPGA bridges could instantiate fpga bus type devices instead of
platform devices. That's really up to Linux and outside the scope of
the bindings.

Rob
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 02/22] fpga: add FPGA device framework

[Wu Hao]

On Thu, Jul 27, 2017 at 11:44:01AM -0500, Alan Tull wrote:
> On Sun, Jun 25, 2017 at 8:51 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> > During FPGA device (e.g PCI-based) discovery, platform devices are
> > registered for different FPGA function units. But the device node path
> > isn't quite friendly to applications.
> >
> 
> Hi Hao,
> 
> Please add a document for fpga-dev under Documentation/fpga to help
> future users.  The header of this patch explains things pretty well
> and could be used with a little formatting for the document.

Sure, will include one document for fpga-dev in the next version patch set.

Thanks
Hao

> 
> Alan

Re: [PATCH v2 00/22] Intel FPGA Device Drivers

[Alan Tull]

On Sun, Jun 25, 2017 at 8:51 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> Hi All,
>
> Here is v2 patch-series adding drivers for Intel FPGA devices.
>
> The Intel FPGA driver provides interfaces for userspace applications to
> configure, enumerate, open, and access FPGA accelerators on platforms
> equipped with Intel(R) PCIe based FPGA solutions and enables system
> level management functions such as FPGA partial reconfiguration, power
> management and virtualization.
>
> This patch series only adds the basic functions for FPGA accelerators and
> partial reconfiguration. Patches for more functions, e.g power management
> and virtualization, will be submitted after this series gets reviewed.
>
> This patch series depends on Alan Tull's patch set v2 on FPGA region
> support w/o device tree (see [1]).

I've posted a branch for review convenience on the linux-fpga
kernel.org repo.  The branch contains this patchset on top of my now
old v2 FPGA region patchset.

Branch name: for-review-next-20170720-intel-fpga-driver-v2

Alan

Re: [PATCH v2 02/22] fpga: add FPGA device framework

[Alan Tull]

On Thu, Jul 27, 2017 at 2:10 PM, Rob Herring <robh+dt-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> On Thu, Jul 27, 2017 at 11:35 AM, Alan Tull <atull-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>> On Sun, Jun 25, 2017 at 8:51 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>>
>> Hi Rob,
>>
>> I was hoping to pick your brain a bit on a DT question.
>>
>>> During FPGA device (e.g PCI-based) discovery, platform devices are
>>> registered for different FPGA function units. But the device node path
>>> isn't quite friendly to applications.
>>>
>>> Consider this case, applications want to access child device's sysfs file
>>> for some information.
>>>
>>> 1) Access using bus-based path (e.g PCI)
>>>
>>>   /sys/bus/pci/devices/xxxxx/fpga_func_a.0/sysfs_file
>>>
>>>   From the path, it's clear which PCI device is the parent, but not perfect
>>>   solution for applications. PCI device BDF is not fixed, application may
>>>   need to search all PCI device to find the actual FPGA Device.
>>>
>>> 2) Or access using platform device path
>>>
>>>   /sys/bus/platform/devices/fpga_func_a.0/sysfs_file
>>>
>>>   Applications find the actual function by name easily, but no information
>>>   about which fpga device it belongs to. It's quite confusing if multiple
>>>   FPGA devices are in one system.
>>
>> There's a proposal for adding sysfs nodes that correspond to each FPGA
>> device., with the devices located on each FPGA under them.  It makes
>> it easier to see which device is on which FPGA.
>
> Makes sense.
>
>>> 'FPGA Device' class is introduced to resolve this problem. Each node under
>>> this class represents a fpga device, which may have one or more child
>>> devices. Applications only need to search under this FPGA Device class
>>> folder to find the child device node it needs.
>>>
>>> For example, for the platform has 2 fpga devices, each fpga device has
>>> 3 child devices, the hierarchy looks like this.
>>>
>>> Two nodes are under /sys/class/fpga/:
>>> /sys/class/fpga/fpga.0
>>> /sys/class/fpga/fpga.1
>>>
>>> Each node has 1 function A device and 2 function B devices:
>>> /sys/class/fpga/fpga.0/func_a.0
>>> /sys/class/fpga/fpga.0/func_b.0
>>> /sys/class/fpga/fpga.0/func_b.1
>>>
>>> /sys/class/fpga/fpga.1/func_a.1
>>> /sys/class/fpga/fpga.1/func_b.2
>>> /sys/class/fpga/fpga.1/func_b.3
>
> A class is generally what is the function of the device, not how it is
> attached. Seems like what you want here is a new bus type if the
> existing PCI and platform bus types don't work.
>
>>
>> I can see the value of having sysfs nodes that correspond to fpga
>> devices and being able to find devices under them.  I'm thinking what
>> that would mean for Device Tree when fpga-dev is used on DT enabled
>> systems.  In Device Tree, what is a fpga-dev?
>
> Just properly setting the parent struct device on the functions should
> be enough to figure out which function is in which fpga. I don't see
> why a new class is needed.
>
>> Currently the DT would have a FPGA bridge corresponding to each FPGA's
>> hardware bridge and a heirarchy of bridges, regions and devices under
>> it.  On systems that don't support partial reconfiguration under the
>> OS (so not main bridge that was controlled by the OS), there would be
>> a FPGA region, then its child regions, bridges, and devices.
>
> The FPGA bridges could instantiate fpga bus type devices instead of
> platform devices.

Yes

Some FPGA use cases already have a base bridge per FPGA that could
serve as this bus.  But this use case has a static FPGA image +
reprogrammable child fpga regions.  There's no base bridge under Linux
since the FPGA was programmed and the bridge enabled before Linux
boots.   An added base bridge that doesn't touch hardware will be
required for this type of use.

> That's really up to Linux and outside the scope of
> the bindings.

Thanks for the feedback.

Alan Tull

>
> Rob

Re: [PATCH v2 22/22] fpga: intel: afu: add FPGA_PORT_DMA_MAP/UNMAP ioctls support

[Alan Tull]

On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu@intel.com> wrote:

Hi Hao,

Please run checkpatch.pl --strict on this.

Could you add some kernel-doc function comments here to help the new
user (or reviewer) get a better handle on what this code is doing?

A few mostly minor comments below.

> DMA memory regions are required for Accelerated Function Unit (AFU) usage.
> These two ioctls allow user space applications to map user memory regions
> for dma, and unmap them after use. Iova is returned from driver to user
> space application via FPGA_PORT_DMA_MAP ioctl. Application needs to unmap
> it after use, otherwise, driver will unmap them in device file release
> operation.
>
> All the mapped regions are managed via a rb tree.

Please note that each afu has its own rb tree (this comment sounds
like there's one rb tree for all) to keep track of its DMA regions.

>
> Ioctl interfaces:
> * FPGA_PORT_DMA_MAP
>   Do the dma mapping per user_addr and length which provided by user.
>   Return iova in provided struct afu_port_dma_map.
>
> * FPGA_PORT_DMA_UNMAP
>   Unmap the dma region per iova provided by user.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ---
> v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
>     switched to GPLv2 license.
>     fixed kbuild warnings.
> ---
>  drivers/fpga/Makefile               |   3 +-
>  drivers/fpga/intel-afu-dma-region.c | 372 ++++++++++++++++++++++++++++++++++++
>  drivers/fpga/intel-afu-main.c       |  61 +++++-
>  drivers/fpga/intel-afu.h            |  18 ++
>  include/uapi/linux/intel-fpga.h     |  37 ++++
>  5 files changed, 489 insertions(+), 2 deletions(-)
>  create mode 100644 drivers/fpga/intel-afu-dma-region.c
>
> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> index 45c0538..339d1f3 100644
> --- a/drivers/fpga/Makefile
> +++ b/drivers/fpga/Makefile
> @@ -38,4 +38,5 @@ obj-$(CONFIG_INTEL_FPGA_AFU)          += intel-fpga-afu.o
>
>  intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
>  intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
> -intel-fpga-afu-objs := intel-afu-main.o intel-afu-region.o
> +intel-fpga-afu-objs := intel-afu-main.o intel-afu-region.o \
> +                      intel-afu-dma-region.o
> diff --git a/drivers/fpga/intel-afu-dma-region.c b/drivers/fpga/intel-afu-dma-region.c
> new file mode 100644
> index 0000000..982a9b5
> --- /dev/null
> +++ b/drivers/fpga/intel-afu-dma-region.c
> @@ -0,0 +1,372 @@
> +/*
> + * Driver for Intel FPGA Accelerated Function Unit (AFU) DMA Region Management
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Wu Hao <hao.wu@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2. See
> + * the COPYING file in the top-level directory.
> + */
> +
> +#include <linux/sched/signal.h>
> +#include <linux/uaccess.h>
> +
> +#include "intel-afu.h"
> +
> +static void put_all_pages(struct page **pages, int npages)
> +{
> +       int i;
> +
> +       for (i = 0; i < npages; i++)
> +               if (pages[i] != NULL)
> +                       put_page(pages[i]);
> +}
> +
> +void afu_dma_region_init(struct feature_platform_data *pdata)
> +{
> +       struct fpga_afu *afu = fpga_pdata_get_private(pdata);
> +
> +       afu->dma_regions = RB_ROOT;
> +}
> +
> +static long afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
> +{
> +       unsigned long locked, lock_limit;
> +       int ret = 0;
> +
> +       /* the task is exiting. */
> +       if (!current->mm)
> +               return 0;
> +
> +       down_write(&current->mm->mmap_sem);
> +
> +       if (incr) {
> +               locked = current->mm->locked_vm + npages;
> +               lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> +
> +               if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> +                       ret = -ENOMEM;
> +               else
> +                       current->mm->locked_vm += npages;
> +       } else {
> +

Don't need to skip a line here.

> +               if (WARN_ON_ONCE(npages > current->mm->locked_vm))
> +                       npages = current->mm->locked_vm;
> +               current->mm->locked_vm -= npages;
> +       }
> +
> +       dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
> +                               incr ? '+' : '-',
> +                               npages << PAGE_SHIFT,
> +                               current->mm->locked_vm << PAGE_SHIFT,
> +                               rlimit(RLIMIT_MEMLOCK),
> +                               ret ? "- execeeded" : "");
> +
> +       up_write(&current->mm->mmap_sem);
> +
> +       return ret;
> +}
> +
> +static long afu_dma_pin_pages(struct feature_platform_data *pdata,
> +                               struct fpga_afu_dma_region *region)

Conventionally, return type is usually int in kernel functions that
return 0 or error code.

> +{
> +       long npages = region->length >> PAGE_SHIFT;
> +       struct device *dev = &pdata->dev->dev;
> +       long ret, pinned;
> +
> +       ret = afu_dma_adjust_locked_vm(dev, npages, true);
> +       if (ret)
> +               return ret;
> +
> +       region->pages = kcalloc(npages, sizeof(struct page *), GFP_KERNEL);
> +       if (!region->pages) {
> +               afu_dma_adjust_locked_vm(dev, npages, false);
> +               return -ENOMEM;
> +       }
> +
> +       pinned = get_user_pages_fast(region->user_addr, npages, 1,
> +                                       region->pages);
> +       if (pinned < 0) {
> +               ret = pinned;

This return value gets all the way to be returned by the ioctl and
isn't used for its value as the number that actually got pinned by the
caller.

> +               goto err_put_pages;
> +       } else if (pinned != npages) {
> +               ret = -EFAULT;
> +               goto err;
> +       }
> +
> +       dev_dbg(dev, "%ld pages pinned\n", pinned);
> +
> +       return 0;
> +
> +err_put_pages:
> +       put_all_pages(region->pages, pinned);
> +err:
> +       kfree(region->pages);
> +       afu_dma_adjust_locked_vm(dev, npages, false);
> +       return ret;
> +}
> +
> +static void afu_dma_unpin_pages(struct feature_platform_data *pdata,
> +                               struct fpga_afu_dma_region *region)
> +{
> +       long npages = region->length >> PAGE_SHIFT;
> +       struct device *dev = &pdata->dev->dev;
> +
> +       put_all_pages(region->pages, npages);
> +       kfree(region->pages);
> +       afu_dma_adjust_locked_vm(dev, npages, false);
> +
> +       dev_dbg(dev, "%ld pages unpinned\n", npages);
> +}
> +
> +static bool afu_dma_check_continuous_pages(struct fpga_afu_dma_region *region)
> +{
> +       int npages = region->length >> PAGE_SHIFT;
> +       int i;
> +
> +       for (i = 0; i < npages - 1; i++)
> +               if (page_to_pfn(region->pages[i]) + 1 !=
> +                                       page_to_pfn(region->pages[i+1]))
> +                       return false;
> +
> +       return true;
> +}
> +
> +static bool dma_region_check_iova(struct fpga_afu_dma_region *region,
> +                                 u64 iova, u64 size)

This function checks that the area defined by iova and size is fully
contained in the region, right?  A comment would help.

> +{
> +       if (!size && region->iova != iova)
> +               return false;
> +
> +       return (region->iova <= iova) &&
> +               (region->length + region->iova >= iova + size);
> +}
> +
> +/* Need to be called with pdata->lock held */
> +static int afu_dma_region_add(struct feature_platform_data *pdata,
> +                                       struct fpga_afu_dma_region *region)
> +{
> +       struct fpga_afu *afu = fpga_pdata_get_private(pdata);
> +       struct rb_node **new, *parent = NULL;
> +
> +       dev_dbg(&pdata->dev->dev, "add region (iova = %llx)\n",
> +                                       (unsigned long long)region->iova);
> +
> +       new = &(afu->dma_regions.rb_node);
> +
> +       while (*new) {
> +               struct fpga_afu_dma_region *this;
> +
> +               this = container_of(*new, struct fpga_afu_dma_region, node);
> +
> +               parent = *new;
> +
> +               if (dma_region_check_iova(this, region->iova, region->length))
> +                       return -EEXIST;
> +
> +               if (region->iova < this->iova)
> +                       new = &((*new)->rb_left);
> +               else if (region->iova > this->iova)
> +                       new = &((*new)->rb_right);
> +               else
> +                       return -EEXIST;
> +       }
> +
> +       rb_link_node(&region->node, parent, new);
> +       rb_insert_color(&region->node, &afu->dma_regions);
> +
> +       return 0;
> +}
> +
> +/* Need to be called with pdata->lock held */
> +static void afu_dma_region_remove(struct feature_platform_data *pdata,
> +                                       struct fpga_afu_dma_region *region)
> +{
> +       struct fpga_afu *afu;
> +
> +       dev_dbg(&pdata->dev->dev, "del region (iova = %llx)\n",
> +                                       (unsigned long long)region->iova);
> +
> +       afu = fpga_pdata_get_private(pdata);
> +       rb_erase(&region->node, &afu->dma_regions);
> +}
> +
> +/* Need to be called with pdata->lock held */
> +void afu_dma_region_destroy(struct feature_platform_data *pdata)
> +{
> +       struct fpga_afu *afu = fpga_pdata_get_private(pdata);
> +       struct rb_node *node = rb_first(&afu->dma_regions);
> +       struct fpga_afu_dma_region *region;
> +
> +       while (node) {
> +               region = container_of(node, struct fpga_afu_dma_region, node);
> +
> +               dev_dbg(&pdata->dev->dev, "del region (iova = %llx)\n",
> +                                       (unsigned long long)region->iova);
> +
> +               rb_erase(node, &afu->dma_regions);
> +
> +               if (region->iova)
> +                       dma_unmap_page(fpga_pdata_to_pcidev(pdata),
> +                                       region->iova, region->length,
> +                                       DMA_BIDIRECTIONAL);
> +
> +               if (region->pages)
> +                       afu_dma_unpin_pages(pdata, region);
> +
> +               node = rb_next(node);
> +               kfree(region);
> +       }
> +}
> +
> +/*
> + * It finds the dma region from the rbtree based on @iova and @size:
> + * - if @size == 0, it finds the dma region which starts from @iova
> + * - otherwise, it finds the dma region which fully contains
> + *   [@iova, @iova+size)
> + * If nothing is matched returns NULL.
> + *
> + * Need to be called with pdata->lock held.
> + */
> +struct fpga_afu_dma_region *
> +afu_dma_region_find(struct feature_platform_data *pdata, u64 iova, u64 size)
> +{
> +       struct fpga_afu *afu = fpga_pdata_get_private(pdata);
> +       struct rb_node *node = afu->dma_regions.rb_node;
> +       struct device *dev = &pdata->dev->dev;
> +
> +       while (node) {
> +               struct fpga_afu_dma_region *region;
> +
> +               region = container_of(node, struct fpga_afu_dma_region, node);
> +
> +               if (dma_region_check_iova(region, iova, size)) {
> +                       dev_dbg(dev, "find region (iova = %llx)\n",
> +                               (unsigned long long)region->iova);
> +                       return region;
> +               }
> +
> +               if (iova < region->iova)
> +                       node = node->rb_left;
> +               else if (iova > region->iova)
> +                       node = node->rb_right;
> +               else
> +                       /* the iova region is not fully covered. */
> +                       break;
> +       }
> +
> +       dev_dbg(dev, "region with iova %llx and size %llx is not found\n",
> +               (unsigned long long)iova, (unsigned long long)size);
> +       return NULL;
> +}
> +
> +static struct fpga_afu_dma_region *
> +afu_dma_region_find_iova(struct feature_platform_data *pdata, u64 iova)
> +{
> +       return afu_dma_region_find(pdata, iova, 0);
> +}
> +
> +long afu_dma_map_region(struct feature_platform_data *pdata,
> +                      u64 user_addr, u64 length, u64 *iova)
> +{
> +       struct fpga_afu_dma_region *region;
> +       int ret;
> +
> +       /*
> +        * Check Inputs, only accept page-aligned user memory region with
> +        * valid length.
> +        */
> +       if (!PAGE_ALIGNED(user_addr) || !PAGE_ALIGNED(length) || !length)
> +               return -EINVAL;
> +
> +       /* Check overflow */
> +       if (user_addr + length < user_addr)
> +               return -EINVAL;
> +
> +       if (!access_ok(VERIFY_WRITE, (void __user *)(unsigned long)user_addr,
> +                      length))
> +               return -EINVAL;
> +
> +       region = kzalloc(sizeof(*region), GFP_KERNEL);
> +       if (!region)
> +               return -ENOMEM;
> +
> +       region->user_addr = user_addr;
> +       region->length = length;
> +
> +       /* Pin the user memory region */
> +       ret = afu_dma_pin_pages(pdata, region);
> +       if (ret) {
> +               dev_err(&pdata->dev->dev, "fail to pin memory region\n");
> +               goto free_region;
> +       }
> +
> +       /* Only accept continuous pages, return error if no */
> +       if (!afu_dma_check_continuous_pages(region)) {
> +               dev_err(&pdata->dev->dev, "pages are not continuous\n");
> +               ret = -EINVAL;
> +               goto unpin_pages;
> +       }
> +
> +       /* As pages are continuous then start to do DMA mapping */
> +       region->iova = dma_map_page(fpga_pdata_to_pcidev(pdata),
> +                                   region->pages[0], 0,
> +                                   region->length,
> +                                   DMA_BIDIRECTIONAL);
> +       if (dma_mapping_error(&pdata->dev->dev, region->iova)) {
> +               dev_err(&pdata->dev->dev, "fail to map dma mapping\n");
> +               ret = -EFAULT;
> +               goto unpin_pages;
> +       }
> +
> +       *iova = region->iova;
> +
> +       mutex_lock(&pdata->lock);
> +       ret = afu_dma_region_add(pdata, region);
> +       mutex_unlock(&pdata->lock);
> +       if (ret) {
> +               dev_err(&pdata->dev->dev, "fail to add dma region\n");
> +               goto unmap_dma;
> +       }
> +
> +       return 0;
> +
> +unmap_dma:
> +       dma_unmap_page(fpga_pdata_to_pcidev(pdata),
> +                      region->iova, region->length, DMA_BIDIRECTIONAL);
> +unpin_pages:
> +       afu_dma_unpin_pages(pdata, region);
> +free_region:
> +       kfree(region);
> +       return ret;
> +}
> +
> +long afu_dma_unmap_region(struct feature_platform_data *pdata, u64 iova)
> +{
> +       struct fpga_afu_dma_region *region;
> +
> +       mutex_lock(&pdata->lock);
> +       region = afu_dma_region_find_iova(pdata, iova);
> +       if (!region) {
> +               mutex_unlock(&pdata->lock);
> +               return -EINVAL;
> +       }
> +
> +       if (region->in_use) {
> +               mutex_unlock(&pdata->lock);
> +               return -EBUSY;
> +       }
> +
> +       afu_dma_region_remove(pdata, region);
> +       mutex_unlock(&pdata->lock);
> +
> +       dma_unmap_page(fpga_pdata_to_pcidev(pdata),
> +                      region->iova, region->length, DMA_BIDIRECTIONAL);
> +       afu_dma_unpin_pages(pdata, region);
> +       kfree(region);
> +
> +       return 0;
> +}
> diff --git a/drivers/fpga/intel-afu-main.c b/drivers/fpga/intel-afu-main.c
> index 8c7aa70..d9f1ebf 100644
> --- a/drivers/fpga/intel-afu-main.c
> +++ b/drivers/fpga/intel-afu-main.c
> @@ -175,7 +175,11 @@ static int afu_release(struct inode *inode, struct file *filp)
>
>         dev_dbg(&pdev->dev, "Device File Release\n");
>
> -       fpga_port_reset(pdev);
> +       mutex_lock(&pdata->lock);
> +       __fpga_port_reset(pdev);
> +       afu_dma_region_destroy(pdata);
> +       mutex_unlock(&pdata->lock);
> +
>         feature_dev_use_end(pdata);
>         return 0;
>  }
> @@ -245,6 +249,55 @@ static long afu_ioctl_check_extension(struct feature_platform_data *pdata,
>         return 0;
>  }
>
> +static long
> +afu_ioctl_dma_map(struct feature_platform_data *pdata, void __user *arg)
> +{
> +       struct fpga_port_dma_map map;
> +       unsigned long minsz;
> +       long ret;
> +
> +       minsz = offsetofend(struct fpga_port_dma_map, iova);
> +
> +       if (copy_from_user(&map, arg, minsz))

Why are you using offsetofend() instead of sizeof(map)?  Is this
struct going to expand beyond iova?  Also below in one place.

> +               return -EFAULT;
> +
> +       if (map.argsz < minsz || map.flags)
> +               return -EINVAL;
> +
> +       ret = afu_dma_map_region(pdata, map.user_addr, map.length, &map.iova);
> +       if (ret)
> +               return ret;
> +
> +       if (copy_to_user(arg, &map, sizeof(map))) {
> +               afu_dma_unmap_region(pdata, map.iova);
> +               return -EFAULT;
> +       }
> +
> +       dev_dbg(&pdata->dev->dev, "dma map: ua=%llx, len=%llx, iova=%llx\n",
> +                               (unsigned long long)map.user_addr,
> +                               (unsigned long long)map.length,
> +                               (unsigned long long)map.iova);
> +
> +       return 0;
> +}
> +
> +static long
> +afu_ioctl_dma_unmap(struct feature_platform_data *pdata, void __user *arg)
> +{
> +       struct fpga_port_dma_unmap unmap;
> +       unsigned long minsz;
> +
> +       minsz = offsetofend(struct fpga_port_dma_unmap, iova);

Here as well.

> +
> +       if (copy_from_user(&unmap, arg, minsz))
> +               return -EFAULT;
> +
> +       if (unmap.argsz < minsz || unmap.flags)
> +               return -EINVAL;
> +
> +       return afu_dma_unmap_region(pdata, unmap.iova);
> +}
> +
>  static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
>  {
>         struct platform_device *pdev = filp->private_data;
> @@ -263,6 +316,10 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
>                 return afu_ioctl_get_info(pdata, (void __user *)arg);
>         case FPGA_PORT_GET_REGION_INFO:
>                 return afu_ioctl_get_region_info(pdata, (void __user *)arg);
> +       case FPGA_PORT_DMA_MAP:
> +               return afu_ioctl_dma_map(pdata, (void __user *)arg);
> +       case FPGA_PORT_DMA_UNMAP:
> +               return afu_ioctl_dma_unmap(pdata, (void __user *)arg);
>         default:
>                 /*
>                  * Let sub-feature's ioctl function to handle the cmd
> @@ -337,6 +394,7 @@ static int afu_dev_init(struct platform_device *pdev)
>         mutex_lock(&pdata->lock);
>         fpga_pdata_set_private(pdata, afu);
>         afu_region_init(pdata);
> +       afu_dma_region_init(pdata);
>         mutex_unlock(&pdata->lock);
>         return 0;
>  }
> @@ -349,6 +407,7 @@ static int afu_dev_destroy(struct platform_device *pdev)
>         mutex_lock(&pdata->lock);
>         afu = fpga_pdata_get_private(pdata);
>         afu_region_destroy(pdata);
> +       afu_dma_region_destroy(pdata);
>         fpga_pdata_set_private(pdata, NULL);
>         mutex_unlock(&pdata->lock);
>
> diff --git a/drivers/fpga/intel-afu.h b/drivers/fpga/intel-afu.h
> index 3417780d..23f7e24 100644
> --- a/drivers/fpga/intel-afu.h
> +++ b/drivers/fpga/intel-afu.h
> @@ -30,11 +30,21 @@ struct fpga_afu_region {
>         struct list_head node;
>  };
>
> +struct fpga_afu_dma_region {
> +       u64 user_addr;
> +       u64 length;
> +       u64 iova;
> +       struct page **pages;
> +       struct rb_node node;
> +       bool in_use;
> +};
> +
>  struct fpga_afu {
>         u64 region_cur_offset;
>         int num_regions;
>         u8 num_umsgs;
>         struct list_head regions;
> +       struct rb_root dma_regions;
>
>         struct feature_platform_data *pdata;
>  };
> @@ -49,4 +59,12 @@ int afu_get_region_by_offset(struct feature_platform_data *pdata,
>                             u64 offset, u64 size,
>                             struct fpga_afu_region *pregion);
>
> +void afu_dma_region_init(struct feature_platform_data *pdata);
> +void afu_dma_region_destroy(struct feature_platform_data *pdata);
> +long afu_dma_map_region(struct feature_platform_data *pdata,
> +                      u64 user_addr, u64 length, u64 *iova);
> +long afu_dma_unmap_region(struct feature_platform_data *pdata, u64 iova);
> +struct fpga_afu_dma_region *afu_dma_region_find(
> +               struct feature_platform_data *pdata, u64 iova, u64 size);
> +
>  #endif
> diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
> index a2ad332..b97ea02 100644
> --- a/include/uapi/linux/intel-fpga.h
> +++ b/include/uapi/linux/intel-fpga.h
> @@ -111,6 +111,43 @@ struct fpga_port_region_info {
>
>  #define FPGA_PORT_GET_REGION_INFO      _IO(FPGA_MAGIC, PORT_BASE + 2)
>
> +/**
> + * FPGA_PORT_DMA_MAP - _IOWR(FPGA_MAGIC, PORT_BASE + 3,
> + *                                             struct fpga_port_dma_map)
> + *
> + * Map the dma memory per user_addr and length which are provided by caller.
> + * Driver fills the iova in provided struct afu_port_dma_map.
> + * This interface only accepts page-size aligned user memory for dma mapping.
> + * Return: 0 on success, -errno on failure.
> + */
> +struct fpga_port_dma_map {
> +       /* Input */
> +       __u32 argsz;            /* Structure length */
> +       __u32 flags;            /* Zero for now */
> +       __u64 user_addr;        /* Process virtual address */
> +       __u64 length;           /* Length of mapping (bytes)*/
> +       /* Output */
> +       __u64 iova;             /* IO virtual address */
> +};
> +
> +#define FPGA_PORT_DMA_MAP      _IO(FPGA_MAGIC, PORT_BASE + 3)
> +
> +/**
> + * FPGA_PORT_DMA_UNMAP - _IOW(FPGA_MAGIC, PORT_BASE + 4,
> + *                                             struct fpga_port_dma_unmap)
> + *
> + * Unmap the dma memory per iova provided by caller.
> + * Return: 0 on success, -errno on failure.
> + */
> +struct fpga_port_dma_unmap {
> +       /* Input */
> +       __u32 argsz;            /* Structure length */
> +       __u32 flags;            /* Zero for now */
> +       __u64 iova;             /* IO virtual address */
> +};
> +
> +#define FPGA_PORT_DMA_UNMAP    _IO(FPGA_MAGIC, PORT_BASE + 4)
> +
>  /* IOCTLs for FME file descriptor */
>
>  /**
> --
> 1.8.3.1
>

Re: [PATCH v2 22/22] fpga: intel: afu: add FPGA_PORT_DMA_MAP/UNMAP ioctls support

[Wu Hao]

On Mon, Jul 31, 2017 at 04:41:26PM -0500, Alan Tull wrote:
> On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

Hi Alan

Thanks a lot for the code review. :)

> 
> Hi Hao,
> 
> Please run checkpatch.pl --strict on this.

Sure, thanks for the suggestion, will run this on all the patches and
fix the items reported.

> 
> Could you add some kernel-doc function comments here to help the new
> user (or reviewer) get a better handle on what this code is doing?

Do make sense to me, sure, will add kernel-doc style function comments.

> 
> A few mostly minor comments below.
> 
> > DMA memory regions are required for Accelerated Function Unit (AFU) usage.
> > These two ioctls allow user space applications to map user memory regions
> > for dma, and unmap them after use. Iova is returned from driver to user
> > space application via FPGA_PORT_DMA_MAP ioctl. Application needs to unmap
> > it after use, otherwise, driver will unmap them in device file release
> > operation.
> >
> > All the mapped regions are managed via a rb tree.
> 
> Please note that each afu has its own rb tree (this comment sounds
> like there's one rb tree for all) to keep track of its DMA regions.

Yes, each afu has its own rb tree to manage the DMA regions, as each afu
could be assigned to a separated virtual machine.

I will improve the description above to avoid confusing things.

> 
> >
> > Ioctl interfaces:
> > * FPGA_PORT_DMA_MAP
> >   Do the dma mapping per user_addr and length which provided by user.
> >   Return iova in provided struct afu_port_dma_map.
> >
> > * FPGA_PORT_DMA_UNMAP
> >   Unmap the dma region per iova provided by user.
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> > Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > ---
> > v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
> >     switched to GPLv2 license.
> >     fixed kbuild warnings.
> > ---
> >  drivers/fpga/Makefile               |   3 +-
> >  drivers/fpga/intel-afu-dma-region.c | 372 ++++++++++++++++++++++++++++++++++++
> >  drivers/fpga/intel-afu-main.c       |  61 +++++-
> >  drivers/fpga/intel-afu.h            |  18 ++
> >  include/uapi/linux/intel-fpga.h     |  37 ++++
> >  5 files changed, 489 insertions(+), 2 deletions(-)
> >  create mode 100644 drivers/fpga/intel-afu-dma-region.c
> >
> > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> > index 45c0538..339d1f3 100644
> > --- a/drivers/fpga/Makefile
> > +++ b/drivers/fpga/Makefile
> > @@ -38,4 +38,5 @@ obj-$(CONFIG_INTEL_FPGA_AFU)          += intel-fpga-afu.o
> >
> >  intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
> >  intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
> > -intel-fpga-afu-objs := intel-afu-main.o intel-afu-region.o
> > +intel-fpga-afu-objs := intel-afu-main.o intel-afu-region.o \
> > +                      intel-afu-dma-region.o
> > diff --git a/drivers/fpga/intel-afu-dma-region.c b/drivers/fpga/intel-afu-dma-region.c
> > new file mode 100644
> > index 0000000..982a9b5
> > --- /dev/null
> > +++ b/drivers/fpga/intel-afu-dma-region.c
> > @@ -0,0 +1,372 @@
> > +/*
> > + * Driver for Intel FPGA Accelerated Function Unit (AFU) DMA Region Management
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> > + *
> > + * This work is licensed under the terms of the GNU GPL version 2. See
> > + * the COPYING file in the top-level directory.
> > + */
> > +
> > +#include <linux/sched/signal.h>
> > +#include <linux/uaccess.h>
> > +
> > +#include "intel-afu.h"
> > +
> > +static void put_all_pages(struct page **pages, int npages)
> > +{
> > +       int i;
> > +
> > +       for (i = 0; i < npages; i++)
> > +               if (pages[i] != NULL)
> > +                       put_page(pages[i]);
> > +}
> > +
> > +void afu_dma_region_init(struct feature_platform_data *pdata)
> > +{
> > +       struct fpga_afu *afu = fpga_pdata_get_private(pdata);
> > +
> > +       afu->dma_regions = RB_ROOT;
> > +}
> > +
> > +static long afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
> > +{
> > +       unsigned long locked, lock_limit;
> > +       int ret = 0;
> > +
> > +       /* the task is exiting. */
> > +       if (!current->mm)
> > +               return 0;
> > +
> > +       down_write(&current->mm->mmap_sem);
> > +
> > +       if (incr) {
> > +               locked = current->mm->locked_vm + npages;
> > +               lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> > +
> > +               if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> > +                       ret = -ENOMEM;
> > +               else
> > +                       current->mm->locked_vm += npages;
> > +       } else {
> > +
> 
> Don't need to skip a line here.

Sorry, will fix this in next version.

> 
> > +               if (WARN_ON_ONCE(npages > current->mm->locked_vm))
> > +                       npages = current->mm->locked_vm;
> > +               current->mm->locked_vm -= npages;
> > +       }
> > +
> > +       dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
> > +                               incr ? '+' : '-',
> > +                               npages << PAGE_SHIFT,
> > +                               current->mm->locked_vm << PAGE_SHIFT,
> > +                               rlimit(RLIMIT_MEMLOCK),
> > +                               ret ? "- execeeded" : "");
> > +
> > +       up_write(&current->mm->mmap_sem);
> > +
> > +       return ret;
> > +}
> > +
> > +static long afu_dma_pin_pages(struct feature_platform_data *pdata,
> > +                               struct fpga_afu_dma_region *region)
> 
> Conventionally, return type is usually int in kernel functions that
> return 0 or error code.

The ioctl function has long return value, so I made these functions to have
the long return value too.

I could update the code to change it to int for sure, I think no impact to
any functionality. : )

> 
> > +{
> > +       long npages = region->length >> PAGE_SHIFT;
> > +       struct device *dev = &pdata->dev->dev;
> > +       long ret, pinned;
> > +
> > +       ret = afu_dma_adjust_locked_vm(dev, npages, true);
> > +       if (ret)
> > +               return ret;
> > +
> > +       region->pages = kcalloc(npages, sizeof(struct page *), GFP_KERNEL);
> > +       if (!region->pages) {
> > +               afu_dma_adjust_locked_vm(dev, npages, false);
> > +               return -ENOMEM;
> > +       }
> > +
> > +       pinned = get_user_pages_fast(region->user_addr, npages, 1,
> > +                                       region->pages);
> > +       if (pinned < 0) {
> > +               ret = pinned;
> 
> This return value gets all the way to be returned by the ioctl and
> isn't used for its value as the number that actually got pinned by the
> caller.

Yes, user application doesn't care how many pages are pinned, it only needs
to know the dma mapping request result for given memory region.

So if get_user_pages_fast returns error code, or only pins a page number which
is smaller than expected, then return error code to userspace application. If
it works as expected, then return 0.

Will put them into function comment in next version.

> 
> > +               goto err_put_pages;
> > +       } else if (pinned != npages) {
> > +               ret = -EFAULT;
> > +               goto err;
> > +       }
> > +
> > +       dev_dbg(dev, "%ld pages pinned\n", pinned);
> > +
> > +       return 0;
> > +
> > +err_put_pages:
> > +       put_all_pages(region->pages, pinned);
> > +err:
> > +       kfree(region->pages);
> > +       afu_dma_adjust_locked_vm(dev, npages, false);
> > +       return ret;
> > +}
> > +
> > +static void afu_dma_unpin_pages(struct feature_platform_data *pdata,
> > +                               struct fpga_afu_dma_region *region)
> > +{
> > +       long npages = region->length >> PAGE_SHIFT;
> > +       struct device *dev = &pdata->dev->dev;
> > +
> > +       put_all_pages(region->pages, npages);
> > +       kfree(region->pages);
> > +       afu_dma_adjust_locked_vm(dev, npages, false);
> > +
> > +       dev_dbg(dev, "%ld pages unpinned\n", npages);
> > +}
> > +
> > +static bool afu_dma_check_continuous_pages(struct fpga_afu_dma_region *region)
> > +{
> > +       int npages = region->length >> PAGE_SHIFT;
> > +       int i;
> > +
> > +       for (i = 0; i < npages - 1; i++)
> > +               if (page_to_pfn(region->pages[i]) + 1 !=
> > +                                       page_to_pfn(region->pages[i+1]))
> > +                       return false;
> > +
> > +       return true;
> > +}
> > +
> > +static bool dma_region_check_iova(struct fpga_afu_dma_region *region,
> > +                                 u64 iova, u64 size)
> 
> This function checks that the area defined by iova and size is fully
> contained in the region, right?  A comment would help.

Yes , will add comment for this function. 

> > +{
> > +       if (!size && region->iova != iova)
> > +               return false;
> > +
> > +       return (region->iova <= iova) &&
> > +               (region->length + region->iova >= iova + size);
> > +}
> > +
> > +/* Need to be called with pdata->lock held */
> > +static int afu_dma_region_add(struct feature_platform_data *pdata,
> > +                                       struct fpga_afu_dma_region *region)
> > +{
> > +       struct fpga_afu *afu = fpga_pdata_get_private(pdata);
> > +       struct rb_node **new, *parent = NULL;
> > +
> > +       dev_dbg(&pdata->dev->dev, "add region (iova = %llx)\n",
> > +                                       (unsigned long long)region->iova);
> > +
> > +       new = &(afu->dma_regions.rb_node);
> > +
> > +       while (*new) {
> > +               struct fpga_afu_dma_region *this;
> > +
> > +               this = container_of(*new, struct fpga_afu_dma_region, node);
> > +
> > +               parent = *new;
> > +
> > +               if (dma_region_check_iova(this, region->iova, region->length))
> > +                       return -EEXIST;
> > +
> > +               if (region->iova < this->iova)
> > +                       new = &((*new)->rb_left);
> > +               else if (region->iova > this->iova)
> > +                       new = &((*new)->rb_right);
> > +               else
> > +                       return -EEXIST;
> > +       }
> > +
> > +       rb_link_node(&region->node, parent, new);
> > +       rb_insert_color(&region->node, &afu->dma_regions);
> > +
> > +       return 0;
> > +}
> > +
> > +/* Need to be called with pdata->lock held */
> > +static void afu_dma_region_remove(struct feature_platform_data *pdata,
> > +                                       struct fpga_afu_dma_region *region)
> > +{
> > +       struct fpga_afu *afu;
> > +
> > +       dev_dbg(&pdata->dev->dev, "del region (iova = %llx)\n",
> > +                                       (unsigned long long)region->iova);
> > +
> > +       afu = fpga_pdata_get_private(pdata);
> > +       rb_erase(&region->node, &afu->dma_regions);
> > +}
> > +
> > +/* Need to be called with pdata->lock held */
> > +void afu_dma_region_destroy(struct feature_platform_data *pdata)
> > +{
> > +       struct fpga_afu *afu = fpga_pdata_get_private(pdata);
> > +       struct rb_node *node = rb_first(&afu->dma_regions);
> > +       struct fpga_afu_dma_region *region;
> > +
> > +       while (node) {
> > +               region = container_of(node, struct fpga_afu_dma_region, node);
> > +
> > +               dev_dbg(&pdata->dev->dev, "del region (iova = %llx)\n",
> > +                                       (unsigned long long)region->iova);
> > +
> > +               rb_erase(node, &afu->dma_regions);
> > +
> > +               if (region->iova)
> > +                       dma_unmap_page(fpga_pdata_to_pcidev(pdata),
> > +                                       region->iova, region->length,
> > +                                       DMA_BIDIRECTIONAL);
> > +
> > +               if (region->pages)
> > +                       afu_dma_unpin_pages(pdata, region);
> > +
> > +               node = rb_next(node);
> > +               kfree(region);
> > +       }
> > +}
> > +
> > +/*
> > + * It finds the dma region from the rbtree based on @iova and @size:
> > + * - if @size == 0, it finds the dma region which starts from @iova
> > + * - otherwise, it finds the dma region which fully contains
> > + *   [@iova, @iova+size)
> > + * If nothing is matched returns NULL.
> > + *
> > + * Need to be called with pdata->lock held.
> > + */
> > +struct fpga_afu_dma_region *
> > +afu_dma_region_find(struct feature_platform_data *pdata, u64 iova, u64 size)
> > +{
> > +       struct fpga_afu *afu = fpga_pdata_get_private(pdata);
> > +       struct rb_node *node = afu->dma_regions.rb_node;
> > +       struct device *dev = &pdata->dev->dev;
> > +
> > +       while (node) {
> > +               struct fpga_afu_dma_region *region;
> > +
> > +               region = container_of(node, struct fpga_afu_dma_region, node);
> > +
> > +               if (dma_region_check_iova(region, iova, size)) {
> > +                       dev_dbg(dev, "find region (iova = %llx)\n",
> > +                               (unsigned long long)region->iova);
> > +                       return region;
> > +               }
> > +
> > +               if (iova < region->iova)
> > +                       node = node->rb_left;
> > +               else if (iova > region->iova)
> > +                       node = node->rb_right;
> > +               else
> > +                       /* the iova region is not fully covered. */
> > +                       break;
> > +       }
> > +
> > +       dev_dbg(dev, "region with iova %llx and size %llx is not found\n",
> > +               (unsigned long long)iova, (unsigned long long)size);
> > +       return NULL;
> > +}
> > +
> > +static struct fpga_afu_dma_region *
> > +afu_dma_region_find_iova(struct feature_platform_data *pdata, u64 iova)
> > +{
> > +       return afu_dma_region_find(pdata, iova, 0);
> > +}
> > +
> > +long afu_dma_map_region(struct feature_platform_data *pdata,
> > +                      u64 user_addr, u64 length, u64 *iova)
> > +{
> > +       struct fpga_afu_dma_region *region;
> > +       int ret;
> > +
> > +       /*
> > +        * Check Inputs, only accept page-aligned user memory region with
> > +        * valid length.
> > +        */
> > +       if (!PAGE_ALIGNED(user_addr) || !PAGE_ALIGNED(length) || !length)
> > +               return -EINVAL;
> > +
> > +       /* Check overflow */
> > +       if (user_addr + length < user_addr)
> > +               return -EINVAL;
> > +
> > +       if (!access_ok(VERIFY_WRITE, (void __user *)(unsigned long)user_addr,
> > +                      length))
> > +               return -EINVAL;
> > +
> > +       region = kzalloc(sizeof(*region), GFP_KERNEL);
> > +       if (!region)
> > +               return -ENOMEM;
> > +
> > +       region->user_addr = user_addr;
> > +       region->length = length;
> > +
> > +       /* Pin the user memory region */
> > +       ret = afu_dma_pin_pages(pdata, region);
> > +       if (ret) {
> > +               dev_err(&pdata->dev->dev, "fail to pin memory region\n");
> > +               goto free_region;
> > +       }
> > +
> > +       /* Only accept continuous pages, return error if no */
> > +       if (!afu_dma_check_continuous_pages(region)) {
> > +               dev_err(&pdata->dev->dev, "pages are not continuous\n");
> > +               ret = -EINVAL;
> > +               goto unpin_pages;
> > +       }
> > +
> > +       /* As pages are continuous then start to do DMA mapping */
> > +       region->iova = dma_map_page(fpga_pdata_to_pcidev(pdata),
> > +                                   region->pages[0], 0,
> > +                                   region->length,
> > +                                   DMA_BIDIRECTIONAL);
> > +       if (dma_mapping_error(&pdata->dev->dev, region->iova)) {
> > +               dev_err(&pdata->dev->dev, "fail to map dma mapping\n");
> > +               ret = -EFAULT;
> > +               goto unpin_pages;
> > +       }
> > +
> > +       *iova = region->iova;
> > +
> > +       mutex_lock(&pdata->lock);
> > +       ret = afu_dma_region_add(pdata, region);
> > +       mutex_unlock(&pdata->lock);
> > +       if (ret) {
> > +               dev_err(&pdata->dev->dev, "fail to add dma region\n");
> > +               goto unmap_dma;
> > +       }
> > +
> > +       return 0;
> > +
> > +unmap_dma:
> > +       dma_unmap_page(fpga_pdata_to_pcidev(pdata),
> > +                      region->iova, region->length, DMA_BIDIRECTIONAL);
> > +unpin_pages:
> > +       afu_dma_unpin_pages(pdata, region);
> > +free_region:
> > +       kfree(region);
> > +       return ret;
> > +}
> > +
> > +long afu_dma_unmap_region(struct feature_platform_data *pdata, u64 iova)
> > +{
> > +       struct fpga_afu_dma_region *region;
> > +
> > +       mutex_lock(&pdata->lock);
> > +       region = afu_dma_region_find_iova(pdata, iova);
> > +       if (!region) {
> > +               mutex_unlock(&pdata->lock);
> > +               return -EINVAL;
> > +       }
> > +
> > +       if (region->in_use) {
> > +               mutex_unlock(&pdata->lock);
> > +               return -EBUSY;
> > +       }
> > +
> > +       afu_dma_region_remove(pdata, region);
> > +       mutex_unlock(&pdata->lock);
> > +
> > +       dma_unmap_page(fpga_pdata_to_pcidev(pdata),
> > +                      region->iova, region->length, DMA_BIDIRECTIONAL);
> > +       afu_dma_unpin_pages(pdata, region);
> > +       kfree(region);
> > +
> > +       return 0;
> > +}
> > diff --git a/drivers/fpga/intel-afu-main.c b/drivers/fpga/intel-afu-main.c
> > index 8c7aa70..d9f1ebf 100644
> > --- a/drivers/fpga/intel-afu-main.c
> > +++ b/drivers/fpga/intel-afu-main.c
> > @@ -175,7 +175,11 @@ static int afu_release(struct inode *inode, struct file *filp)
> >
> >         dev_dbg(&pdev->dev, "Device File Release\n");
> >
> > -       fpga_port_reset(pdev);
> > +       mutex_lock(&pdata->lock);
> > +       __fpga_port_reset(pdev);
> > +       afu_dma_region_destroy(pdata);
> > +       mutex_unlock(&pdata->lock);
> > +
> >         feature_dev_use_end(pdata);
> >         return 0;
> >  }
> > @@ -245,6 +249,55 @@ static long afu_ioctl_check_extension(struct feature_platform_data *pdata,
> >         return 0;
> >  }
> >
> > +static long
> > +afu_ioctl_dma_map(struct feature_platform_data *pdata, void __user *arg)
> > +{
> > +       struct fpga_port_dma_map map;
> > +       unsigned long minsz;
> > +       long ret;
> > +
> > +       minsz = offsetofend(struct fpga_port_dma_map, iova);
> > +
> > +       if (copy_from_user(&map, arg, minsz))
> 
> Why are you using offsetofend() instead of sizeof(map)?  Is this
> struct going to expand beyond iova?  Also below in one place.

Yes, we just want to make these ioctls extendable.

Actually this is following the same style as vfio. Here is one reference
case in vfio[1]. Define some new flags and introduce new members in newer
version API data structure, and new version driver still works with the
applications using old version API data structure. : )

And other ioctls in intel fpga device drivers are using the same style too.

[1] http://marc.info/?l=linux-kernel&m=143348624325058&w=2

> > +               return -EFAULT;
> > +
> > +       if (map.argsz < minsz || map.flags)
> > +               return -EINVAL;
> > +
> > +       ret = afu_dma_map_region(pdata, map.user_addr, map.length, &map.iova);
> > +       if (ret)
> > +               return ret;
> > +
> > +       if (copy_to_user(arg, &map, sizeof(map))) {
> > +               afu_dma_unmap_region(pdata, map.iova);
> > +               return -EFAULT;
> > +       }
> > +
> > +       dev_dbg(&pdata->dev->dev, "dma map: ua=%llx, len=%llx, iova=%llx\n",
> > +                               (unsigned long long)map.user_addr,
> > +                               (unsigned long long)map.length,
> > +                               (unsigned long long)map.iova);
> > +
> > +       return 0;
> > +}
> > +
> > +static long
> > +afu_ioctl_dma_unmap(struct feature_platform_data *pdata, void __user *arg)
> > +{
> > +       struct fpga_port_dma_unmap unmap;
> > +       unsigned long minsz;
> > +
> > +       minsz = offsetofend(struct fpga_port_dma_unmap, iova);
> 
> Here as well.

Same as above.

Thanks
Hao
 
> 
> > +
> > +       if (copy_from_user(&unmap, arg, minsz))
> > +               return -EFAULT;
> > +
> > +       if (unmap.argsz < minsz || unmap.flags)
> > +               return -EINVAL;
> > +
> > +       return afu_dma_unmap_region(pdata, unmap.iova);
> > +}
> > +
> >  static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
> >  {
> >         struct platform_device *pdev = filp->private_data;
> > @@ -263,6 +316,10 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
> >                 return afu_ioctl_get_info(pdata, (void __user *)arg);
> >         case FPGA_PORT_GET_REGION_INFO:
> >                 return afu_ioctl_get_region_info(pdata, (void __user *)arg);
> > +       case FPGA_PORT_DMA_MAP:
> > +               return afu_ioctl_dma_map(pdata, (void __user *)arg);
> > +       case FPGA_PORT_DMA_UNMAP:
> > +               return afu_ioctl_dma_unmap(pdata, (void __user *)arg);
> >         default:
> >                 /*
> >                  * Let sub-feature's ioctl function to handle the cmd
> > @@ -337,6 +394,7 @@ static int afu_dev_init(struct platform_device *pdev)
> >         mutex_lock(&pdata->lock);
> >         fpga_pdata_set_private(pdata, afu);
> >         afu_region_init(pdata);
> > +       afu_dma_region_init(pdata);
> >         mutex_unlock(&pdata->lock);
> >         return 0;
> >  }
> > @@ -349,6 +407,7 @@ static int afu_dev_destroy(struct platform_device *pdev)
> >         mutex_lock(&pdata->lock);
> >         afu = fpga_pdata_get_private(pdata);
> >         afu_region_destroy(pdata);
> > +       afu_dma_region_destroy(pdata);
> >         fpga_pdata_set_private(pdata, NULL);
> >         mutex_unlock(&pdata->lock);
> >
> > diff --git a/drivers/fpga/intel-afu.h b/drivers/fpga/intel-afu.h
> > index 3417780d..23f7e24 100644
> > --- a/drivers/fpga/intel-afu.h
> > +++ b/drivers/fpga/intel-afu.h
> > @@ -30,11 +30,21 @@ struct fpga_afu_region {
> >         struct list_head node;
> >  };
> >
> > +struct fpga_afu_dma_region {
> > +       u64 user_addr;
> > +       u64 length;
> > +       u64 iova;
> > +       struct page **pages;
> > +       struct rb_node node;
> > +       bool in_use;
> > +};
> > +
> >  struct fpga_afu {
> >         u64 region_cur_offset;
> >         int num_regions;
> >         u8 num_umsgs;
> >         struct list_head regions;
> > +       struct rb_root dma_regions;
> >
> >         struct feature_platform_data *pdata;
> >  };
> > @@ -49,4 +59,12 @@ int afu_get_region_by_offset(struct feature_platform_data *pdata,
> >                             u64 offset, u64 size,
> >                             struct fpga_afu_region *pregion);
> >
> > +void afu_dma_region_init(struct feature_platform_data *pdata);
> > +void afu_dma_region_destroy(struct feature_platform_data *pdata);
> > +long afu_dma_map_region(struct feature_platform_data *pdata,
> > +                      u64 user_addr, u64 length, u64 *iova);
> > +long afu_dma_unmap_region(struct feature_platform_data *pdata, u64 iova);
> > +struct fpga_afu_dma_region *afu_dma_region_find(
> > +               struct feature_platform_data *pdata, u64 iova, u64 size);
> > +
> >  #endif
> > diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
> > index a2ad332..b97ea02 100644
> > --- a/include/uapi/linux/intel-fpga.h
> > +++ b/include/uapi/linux/intel-fpga.h
> > @@ -111,6 +111,43 @@ struct fpga_port_region_info {
> >
> >  #define FPGA_PORT_GET_REGION_INFO      _IO(FPGA_MAGIC, PORT_BASE + 2)
> >
> > +/**
> > + * FPGA_PORT_DMA_MAP - _IOWR(FPGA_MAGIC, PORT_BASE + 3,
> > + *                                             struct fpga_port_dma_map)
> > + *
> > + * Map the dma memory per user_addr and length which are provided by caller.
> > + * Driver fills the iova in provided struct afu_port_dma_map.
> > + * This interface only accepts page-size aligned user memory for dma mapping.
> > + * Return: 0 on success, -errno on failure.
> > + */
> > +struct fpga_port_dma_map {
> > +       /* Input */
> > +       __u32 argsz;            /* Structure length */
> > +       __u32 flags;            /* Zero for now */
> > +       __u64 user_addr;        /* Process virtual address */
> > +       __u64 length;           /* Length of mapping (bytes)*/
> > +       /* Output */
> > +       __u64 iova;             /* IO virtual address */
> > +};
> > +
> > +#define FPGA_PORT_DMA_MAP      _IO(FPGA_MAGIC, PORT_BASE + 3)
> > +
> > +/**
> > + * FPGA_PORT_DMA_UNMAP - _IOW(FPGA_MAGIC, PORT_BASE + 4,
> > + *                                             struct fpga_port_dma_unmap)
> > + *
> > + * Unmap the dma memory per iova provided by caller.
> > + * Return: 0 on success, -errno on failure.
> > + */
> > +struct fpga_port_dma_unmap {
> > +       /* Input */
> > +       __u32 argsz;            /* Structure length */
> > +       __u32 flags;            /* Zero for now */
> > +       __u64 iova;             /* IO virtual address */
> > +};
> > +
> > +#define FPGA_PORT_DMA_UNMAP    _IO(FPGA_MAGIC, PORT_BASE + 4)
> > +
> >  /* IOCTLs for FME file descriptor */
> >
> >  /**
> > --
> > 1.8.3.1
> >

Re: [PATCH v2 02/22] fpga: add FPGA device framework

[Wu Hao]

On Mon, Jul 31, 2017 at 04:40:16PM -0500, Alan Tull wrote:
> On Thu, Jul 27, 2017 at 2:10 PM, Rob Herring <robh+dt-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> > On Thu, Jul 27, 2017 at 11:35 AM, Alan Tull <atull-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> >> On Sun, Jun 25, 2017 at 8:51 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> >>
> >> Hi Rob,
> >>
> >> I was hoping to pick your brain a bit on a DT question.
> >>
> >>> During FPGA device (e.g PCI-based) discovery, platform devices are
> >>> registered for different FPGA function units. But the device node path
> >>> isn't quite friendly to applications.
> >>>
> >>> Consider this case, applications want to access child device's sysfs file
> >>> for some information.
> >>>
> >>> 1) Access using bus-based path (e.g PCI)
> >>>
> >>>   /sys/bus/pci/devices/xxxxx/fpga_func_a.0/sysfs_file
> >>>
> >>>   From the path, it's clear which PCI device is the parent, but not perfect
> >>>   solution for applications. PCI device BDF is not fixed, application may
> >>>   need to search all PCI device to find the actual FPGA Device.
> >>>
> >>> 2) Or access using platform device path
> >>>
> >>>   /sys/bus/platform/devices/fpga_func_a.0/sysfs_file
> >>>
> >>>   Applications find the actual function by name easily, but no information
> >>>   about which fpga device it belongs to. It's quite confusing if multiple
> >>>   FPGA devices are in one system.
> >>
> >> There's a proposal for adding sysfs nodes that correspond to each FPGA
> >> device., with the devices located on each FPGA under them.  It makes
> >> it easier to see which device is on which FPGA.
> >
> > Makes sense.
> >
> >>> 'FPGA Device' class is introduced to resolve this problem. Each node under
> >>> this class represents a fpga device, which may have one or more child
> >>> devices. Applications only need to search under this FPGA Device class
> >>> folder to find the child device node it needs.
> >>>
> >>> For example, for the platform has 2 fpga devices, each fpga device has
> >>> 3 child devices, the hierarchy looks like this.
> >>>
> >>> Two nodes are under /sys/class/fpga/:
> >>> /sys/class/fpga/fpga.0
> >>> /sys/class/fpga/fpga.1
> >>>
> >>> Each node has 1 function A device and 2 function B devices:
> >>> /sys/class/fpga/fpga.0/func_a.0
> >>> /sys/class/fpga/fpga.0/func_b.0
> >>> /sys/class/fpga/fpga.0/func_b.1
> >>>
> >>> /sys/class/fpga/fpga.1/func_a.1
> >>> /sys/class/fpga/fpga.1/func_b.2
> >>> /sys/class/fpga/fpga.1/func_b.3
> >
> > A class is generally what is the function of the device, not how it is
> > attached. Seems like what you want here is a new bus type if the
> > existing PCI and platform bus types don't work.
> >
> >>
> >> I can see the value of having sysfs nodes that correspond to fpga
> >> devices and being able to find devices under them.  I'm thinking what
> >> that would mean for Device Tree when fpga-dev is used on DT enabled
> >> systems.  In Device Tree, what is a fpga-dev?
> >
> > Just properly setting the parent struct device on the functions should
> > be enough to figure out which function is in which fpga. I don't see
> > why a new class is needed.
> >
> >> Currently the DT would have a FPGA bridge corresponding to each FPGA's
> >> hardware bridge and a heirarchy of bridges, regions and devices under
> >> it.  On systems that don't support partial reconfiguration under the
> >> OS (so not main bridge that was controlled by the OS), there would be
> >> a FPGA region, then its child regions, bridges, and devices.
> >
> > The FPGA bridges could instantiate fpga bus type devices instead of
> > platform devices.
> 
> Yes
> 
> Some FPGA use cases already have a base bridge per FPGA that could
> serve as this bus.  But this use case has a static FPGA image +
> reprogrammable child fpga regions.  There's no base bridge under Linux
> since the FPGA was programmed and the bridge enabled before Linux
> boots.   An added base bridge that doesn't touch hardware will be
> required for this type of use.

Hi Alan

Does 'base bridge' mentioned above mean a hardware bridge just like
PCIe or USB?

I tried to use fpga bus type device instead of fpga-dev class today,
it works for me, e.g Intel FPGA device PCIe driver could create a
fpga bus type dev as a child of PCIe device and its sysfs path will be
changed to /sys/bus/fpga/devices/fpga.x/ from /sys/class/fpga/fpga.x/.
For now, this fpga bus type device is only used as container device,
so no driver needed for it.

Do you have any concern on this? I see fpga bus type works fine, but
I didn't see other advantages for this case, as we only use it as a
container device to represent a FPGA device in sysfs hierarchy. :)

Thanks
Hao

> 
> > That's really up to Linux and outside the scope of
> > the bindings.
> 
> Thanks for the feedback.
> 
> Alan Tull
> 
> >
> > Rob

Re: [PATCH v2 22/22] fpga: intel: afu: add FPGA_PORT_DMA_MAP/UNMAP ioctls support

[Moritz Fischer]

Hi Wu,

couple of minor things inline below.

On Sun, Jun 25, 2017 at 6:52 PM, Wu Hao <hao.wu@intel.com> wrote:
> DMA memory regions are required for Accelerated Function Unit (AFU) usage.
> These two ioctls allow user space applications to map user memory regions
> for dma, and unmap them after use. Iova is returned from driver to user
> space application via FPGA_PORT_DMA_MAP ioctl. Application needs to unmap
> it after use, otherwise, driver will unmap them in device file release
> operation.
>
> All the mapped regions are managed via a rb tree.
>
> Ioctl interfaces:
> * FPGA_PORT_DMA_MAP
>   Do the dma mapping per user_addr and length which provided by user.
>   Return iova in provided struct afu_port_dma_map.
>
> * FPGA_PORT_DMA_UNMAP
>   Unmap the dma region per iova provided by user.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ---
> v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
>     switched to GPLv2 license.
>     fixed kbuild warnings.
> ---
>  drivers/fpga/Makefile               |   3 +-
>  drivers/fpga/intel-afu-dma-region.c | 372 ++++++++++++++++++++++++++++++++++++
>  drivers/fpga/intel-afu-main.c       |  61 +++++-
>  drivers/fpga/intel-afu.h            |  18 ++
>  include/uapi/linux/intel-fpga.h     |  37 ++++
>  5 files changed, 489 insertions(+), 2 deletions(-)
>  create mode 100644 drivers/fpga/intel-afu-dma-region.c
>
> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> index 45c0538..339d1f3 100644
> --- a/drivers/fpga/Makefile
> +++ b/drivers/fpga/Makefile
> @@ -38,4 +38,5 @@ obj-$(CONFIG_INTEL_FPGA_AFU)          += intel-fpga-afu.o
>
>  intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
>  intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
> -intel-fpga-afu-objs := intel-afu-main.o intel-afu-region.o
> +intel-fpga-afu-objs := intel-afu-main.o intel-afu-region.o \
> +                      intel-afu-dma-region.o
> diff --git a/drivers/fpga/intel-afu-dma-region.c b/drivers/fpga/intel-afu-dma-region.c
> new file mode 100644
> index 0000000..982a9b5
> --- /dev/null
> +++ b/drivers/fpga/intel-afu-dma-region.c
> @@ -0,0 +1,372 @@
> +/*
> + * Driver for Intel FPGA Accelerated Function Unit (AFU) DMA Region Management
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Wu Hao <hao.wu@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2. See
> + * the COPYING file in the top-level directory.
> + */
> +
> +#include <linux/sched/signal.h>
> +#include <linux/uaccess.h>
> +
> +#include "intel-afu.h"
> +
> +static void put_all_pages(struct page **pages, int npages)
> +{
> +       int i;
> +
> +       for (i = 0; i < npages; i++)
> +               if (pages[i] != NULL)

if (pages[i]) would do I think

> +                       put_page(pages[i]);
> +}
> +
> +void afu_dma_region_init(struct feature_platform_data *pdata)
> +{
> +       struct fpga_afu *afu = fpga_pdata_get_private(pdata);
> +
> +       afu->dma_regions = RB_ROOT;
> +}
> +
> +static long afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
> +{
> +       unsigned long locked, lock_limit;
> +       int ret = 0;
> +
> +       /* the task is exiting. */
> +       if (!current->mm)
> +               return 0;
> +
> +       down_write(&current->mm->mmap_sem);
> +
> +       if (incr) {
> +               locked = current->mm->locked_vm + npages;
> +               lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> +
> +               if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> +                       ret = -ENOMEM;
> +               else
> +                       current->mm->locked_vm += npages;
> +       } else {
> +
> +               if (WARN_ON_ONCE(npages > current->mm->locked_vm))
> +                       npages = current->mm->locked_vm;
> +               current->mm->locked_vm -= npages;
> +       }
> +
> +       dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
> +                               incr ? '+' : '-',
> +                               npages << PAGE_SHIFT,
> +                               current->mm->locked_vm << PAGE_SHIFT,
> +                               rlimit(RLIMIT_MEMLOCK),
> +                               ret ? "- execeeded" : "");
> +
> +       up_write(&current->mm->mmap_sem);
> +
> +       return ret;
> +}
> +
> +static long afu_dma_pin_pages(struct feature_platform_data *pdata,
> +                               struct fpga_afu_dma_region *region)
> +{
> +       long npages = region->length >> PAGE_SHIFT;
> +       struct device *dev = &pdata->dev->dev;
> +       long ret, pinned;
> +
> +       ret = afu_dma_adjust_locked_vm(dev, npages, true);
> +       if (ret)
> +               return ret;
> +
> +       region->pages = kcalloc(npages, sizeof(struct page *), GFP_KERNEL);
> +       if (!region->pages) {
> +               afu_dma_adjust_locked_vm(dev, npages, false);

You could probably also just have another label in the error handling path.
> +               return -ENOMEM;
> +       }
> +
> +       pinned = get_user_pages_fast(region->user_addr, npages, 1,
> +                                       region->pages);
> +       if (pinned < 0) {
> +               ret = pinned;
> +               goto err_put_pages;
> +       } else if (pinned != npages) {
> +               ret = -EFAULT;
> +               goto err;
> +       }
> +
> +       dev_dbg(dev, "%ld pages pinned\n", pinned);
> +
> +       return 0;
> +
> +err_put_pages:
> +       put_all_pages(region->pages, pinned);
> +err:
> +       kfree(region->pages);
> +       afu_dma_adjust_locked_vm(dev, npages, false);
> +       return ret;
> +}
> +
> +static void afu_dma_unpin_pages(struct feature_platform_data *pdata,
> +                               struct fpga_afu_dma_region *region)
> +{
> +       long npages = region->length >> PAGE_SHIFT;
> +       struct device *dev = &pdata->dev->dev;
> +
> +       put_all_pages(region->pages, npages);
> +       kfree(region->pages);
> +       afu_dma_adjust_locked_vm(dev, npages, false);
> +
> +       dev_dbg(dev, "%ld pages unpinned\n", npages);
> +}
> +
> +static bool afu_dma_check_continuous_pages(struct fpga_afu_dma_region *region)
> +{
> +       int npages = region->length >> PAGE_SHIFT;
> +       int i;
> +
> +       for (i = 0; i < npages - 1; i++)
> +               if (page_to_pfn(region->pages[i]) + 1 !=
> +                                       page_to_pfn(region->pages[i+1]))
> +                       return false;
> +
> +       return true;
> +}
> +
> +static bool dma_region_check_iova(struct fpga_afu_dma_region *region,
> +                                 u64 iova, u64 size)
> +{
> +       if (!size && region->iova != iova)
> +               return false;
> +
> +       return (region->iova <= iova) &&
> +               (region->length + region->iova >= iova + size);
> +}
> +
> +/* Need to be called with pdata->lock held */

Needs
> +static int afu_dma_region_add(struct feature_platform_data *pdata,
> +                                       struct fpga_afu_dma_region *region)
> +{
> +       struct fpga_afu *afu = fpga_pdata_get_private(pdata);
> +       struct rb_node **new, *parent = NULL;
> +
> +       dev_dbg(&pdata->dev->dev, "add region (iova = %llx)\n",
> +                                       (unsigned long long)region->iova);
> +
> +       new = &(afu->dma_regions.rb_node);
> +
> +       while (*new) {
> +               struct fpga_afu_dma_region *this;
> +
> +               this = container_of(*new, struct fpga_afu_dma_region, node);
> +
> +               parent = *new;
> +
> +               if (dma_region_check_iova(this, region->iova, region->length))
> +                       return -EEXIST;
> +
> +               if (region->iova < this->iova)
> +                       new = &((*new)->rb_left);
> +               else if (region->iova > this->iova)
> +                       new = &((*new)->rb_right);
> +               else
> +                       return -EEXIST;
> +       }
> +
> +       rb_link_node(&region->node, parent, new);
> +       rb_insert_color(&region->node, &afu->dma_regions);
> +
> +       return 0;
> +}
> +
> +/* Need to be called with pdata->lock held */

Ditto
> +static void afu_dma_region_remove(struct feature_platform_data *pdata,
> +                                       struct fpga_afu_dma_region *region)
> +{
> +       struct fpga_afu *afu;
> +
> +       dev_dbg(&pdata->dev->dev, "del region (iova = %llx)\n",
> +                                       (unsigned long long)region->iova);
> +
> +       afu = fpga_pdata_get_private(pdata);
> +       rb_erase(&region->node, &afu->dma_regions);
> +}
> +
> +/* Need to be called with pdata->lock held */

Ditto.
> +void afu_dma_region_destroy(struct feature_platform_data *pdata)
> +{
> +       struct fpga_afu *afu = fpga_pdata_get_private(pdata);
> +       struct rb_node *node = rb_first(&afu->dma_regions);
> +       struct fpga_afu_dma_region *region;
> +
> +       while (node) {
> +               region = container_of(node, struct fpga_afu_dma_region, node);
> +
> +               dev_dbg(&pdata->dev->dev, "del region (iova = %llx)\n",
> +                                       (unsigned long long)region->iova);
> +
> +               rb_erase(node, &afu->dma_regions);
> +
> +               if (region->iova)
> +                       dma_unmap_page(fpga_pdata_to_pcidev(pdata),
> +                                       region->iova, region->length,
> +                                       DMA_BIDIRECTIONAL);
> +
> +               if (region->pages)
> +                       afu_dma_unpin_pages(pdata, region);
> +
> +               node = rb_next(node);
> +               kfree(region);
> +       }
> +}
> +
> +/*
> + * It finds the dma region from the rbtree based on @iova and @size:
> + * - if @size == 0, it finds the dma region which starts from @iova
> + * - otherwise, it finds the dma region which fully contains
> + *   [@iova, @iova+size)
> + * If nothing is matched returns NULL.
> + *
> + * Need to be called with pdata->lock held.
> + */
> +struct fpga_afu_dma_region *
> +afu_dma_region_find(struct feature_platform_data *pdata, u64 iova, u64 size)
> +{
> +       struct fpga_afu *afu = fpga_pdata_get_private(pdata);
> +       struct rb_node *node = afu->dma_regions.rb_node;
> +       struct device *dev = &pdata->dev->dev;
> +
> +       while (node) {
> +               struct fpga_afu_dma_region *region;
> +
> +               region = container_of(node, struct fpga_afu_dma_region, node);
> +
> +               if (dma_region_check_iova(region, iova, size)) {
> +                       dev_dbg(dev, "find region (iova = %llx)\n",
> +                               (unsigned long long)region->iova);
> +                       return region;
> +               }
> +
> +               if (iova < region->iova)
> +                       node = node->rb_left;
> +               else if (iova > region->iova)
> +                       node = node->rb_right;
> +               else
> +                       /* the iova region is not fully covered. */
> +                       break;
> +       }
> +
> +       dev_dbg(dev, "region with iova %llx and size %llx is not found\n",
> +               (unsigned long long)iova, (unsigned long long)size);
> +       return NULL;
> +}
> +
> +static struct fpga_afu_dma_region *
> +afu_dma_region_find_iova(struct feature_platform_data *pdata, u64 iova)
> +{
> +       return afu_dma_region_find(pdata, iova, 0);
> +}
> +
> +long afu_dma_map_region(struct feature_platform_data *pdata,
> +                      u64 user_addr, u64 length, u64 *iova)
> +{
> +       struct fpga_afu_dma_region *region;
> +       int ret;
> +
> +       /*
> +        * Check Inputs, only accept page-aligned user memory region with
> +        * valid length.
> +        */
> +       if (!PAGE_ALIGNED(user_addr) || !PAGE_ALIGNED(length) || !length)
> +               return -EINVAL;
> +
> +       /* Check overflow */
> +       if (user_addr + length < user_addr)
> +               return -EINVAL;
> +
> +       if (!access_ok(VERIFY_WRITE, (void __user *)(unsigned long)user_addr,
> +                      length))
> +               return -EINVAL;
> +
> +       region = kzalloc(sizeof(*region), GFP_KERNEL);
> +       if (!region)
> +               return -ENOMEM;
> +
> +       region->user_addr = user_addr;
> +       region->length = length;
> +
> +       /* Pin the user memory region */
> +       ret = afu_dma_pin_pages(pdata, region);
> +       if (ret) {
> +               dev_err(&pdata->dev->dev, "fail to pin memory region\n");
> +               goto free_region;
> +       }
> +
> +       /* Only accept continuous pages, return error if no */
return error else

> +       if (!afu_dma_check_continuous_pages(region)) {
> +               dev_err(&pdata->dev->dev, "pages are not continuous\n");
> +               ret = -EINVAL;
> +               goto unpin_pages;
> +       }
> +
> +       /* As pages are continuous then start to do DMA mapping */
> +       region->iova = dma_map_page(fpga_pdata_to_pcidev(pdata),
> +                                   region->pages[0], 0,
> +                                   region->length,
> +                                   DMA_BIDIRECTIONAL);
> +       if (dma_mapping_error(&pdata->dev->dev, region->iova)) {
> +               dev_err(&pdata->dev->dev, "fail to map dma mapping\n");
'failed to map for dma' ? :)

> +               ret = -EFAULT;
> +               goto unpin_pages;
> +       }
> +
> +       *iova = region->iova;
> +
> +       mutex_lock(&pdata->lock);
> +       ret = afu_dma_region_add(pdata, region);
> +       mutex_unlock(&pdata->lock);
> +       if (ret) {
> +               dev_err(&pdata->dev->dev, "fail to add dma region\n");
> +               goto unmap_dma;
> +       }
> +
> +       return 0;
> +
> +unmap_dma:
> +       dma_unmap_page(fpga_pdata_to_pcidev(pdata),
> +                      region->iova, region->length, DMA_BIDIRECTIONAL);
> +unpin_pages:
> +       afu_dma_unpin_pages(pdata, region);
> +free_region:
> +       kfree(region);
> +       return ret;
> +}
> +
> +long afu_dma_unmap_region(struct feature_platform_data *pdata, u64 iova)
> +{
> +       struct fpga_afu_dma_region *region;
> +
> +       mutex_lock(&pdata->lock);
> +       region = afu_dma_region_find_iova(pdata, iova);
> +       if (!region) {
> +               mutex_unlock(&pdata->lock);
> +               return -EINVAL;
> +       }
> +
> +       if (region->in_use) {
> +               mutex_unlock(&pdata->lock);
> +               return -EBUSY;
> +       }
> +
> +       afu_dma_region_remove(pdata, region);
> +       mutex_unlock(&pdata->lock);
> +
> +       dma_unmap_page(fpga_pdata_to_pcidev(pdata),
> +                      region->iova, region->length, DMA_BIDIRECTIONAL);
> +       afu_dma_unpin_pages(pdata, region);
> +       kfree(region);
> +
> +       return 0;
> +}
> diff --git a/drivers/fpga/intel-afu-main.c b/drivers/fpga/intel-afu-main.c
> index 8c7aa70..d9f1ebf 100644
> --- a/drivers/fpga/intel-afu-main.c
> +++ b/drivers/fpga/intel-afu-main.c
> @@ -175,7 +175,11 @@ static int afu_release(struct inode *inode, struct file *filp)
>
>         dev_dbg(&pdev->dev, "Device File Release\n");
>
> -       fpga_port_reset(pdev);
> +       mutex_lock(&pdata->lock);
> +       __fpga_port_reset(pdev);
> +       afu_dma_region_destroy(pdata);
> +       mutex_unlock(&pdata->lock);
> +
>         feature_dev_use_end(pdata);
>         return 0;
>  }
> @@ -245,6 +249,55 @@ static long afu_ioctl_check_extension(struct feature_platform_data *pdata,
>         return 0;
>  }
>
> +static long
> +afu_ioctl_dma_map(struct feature_platform_data *pdata, void __user *arg)
> +{
> +       struct fpga_port_dma_map map;
> +       unsigned long minsz;
> +       long ret;
> +
> +       minsz = offsetofend(struct fpga_port_dma_map, iova);
> +
> +       if (copy_from_user(&map, arg, minsz))
> +               return -EFAULT;
> +
> +       if (map.argsz < minsz || map.flags)
> +               return -EINVAL;
> +
> +       ret = afu_dma_map_region(pdata, map.user_addr, map.length, &map.iova);
> +       if (ret)
> +               return ret;
> +
> +       if (copy_to_user(arg, &map, sizeof(map))) {
> +               afu_dma_unmap_region(pdata, map.iova);
> +               return -EFAULT;
> +       }
> +
> +       dev_dbg(&pdata->dev->dev, "dma map: ua=%llx, len=%llx, iova=%llx\n",
> +                               (unsigned long long)map.user_addr,
> +                               (unsigned long long)map.length,
> +                               (unsigned long long)map.iova);
> +
> +       return 0;
> +}
> +
> +static long
> +afu_ioctl_dma_unmap(struct feature_platform_data *pdata, void __user *arg)
> +{
> +       struct fpga_port_dma_unmap unmap;
> +       unsigned long minsz;
> +
> +       minsz = offsetofend(struct fpga_port_dma_unmap, iova);
> +
> +       if (copy_from_user(&unmap, arg, minsz))
> +               return -EFAULT;
> +
> +       if (unmap.argsz < minsz || unmap.flags)
> +               return -EINVAL;
> +
> +       return afu_dma_unmap_region(pdata, unmap.iova);
> +}
> +
>  static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
>  {
>         struct platform_device *pdev = filp->private_data;
> @@ -263,6 +316,10 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
>                 return afu_ioctl_get_info(pdata, (void __user *)arg);
>         case FPGA_PORT_GET_REGION_INFO:
>                 return afu_ioctl_get_region_info(pdata, (void __user *)arg);
> +       case FPGA_PORT_DMA_MAP:
> +               return afu_ioctl_dma_map(pdata, (void __user *)arg);
> +       case FPGA_PORT_DMA_UNMAP:
> +               return afu_ioctl_dma_unmap(pdata, (void __user *)arg);
>         default:
>                 /*
>                  * Let sub-feature's ioctl function to handle the cmd
> @@ -337,6 +394,7 @@ static int afu_dev_init(struct platform_device *pdev)
>         mutex_lock(&pdata->lock);
>         fpga_pdata_set_private(pdata, afu);
>         afu_region_init(pdata);
> +       afu_dma_region_init(pdata);
>         mutex_unlock(&pdata->lock);
>         return 0;
>  }
> @@ -349,6 +407,7 @@ static int afu_dev_destroy(struct platform_device *pdev)
>         mutex_lock(&pdata->lock);
>         afu = fpga_pdata_get_private(pdata);
>         afu_region_destroy(pdata);
> +       afu_dma_region_destroy(pdata);
>         fpga_pdata_set_private(pdata, NULL);
>         mutex_unlock(&pdata->lock);
>
> diff --git a/drivers/fpga/intel-afu.h b/drivers/fpga/intel-afu.h
> index 3417780d..23f7e24 100644
> --- a/drivers/fpga/intel-afu.h
> +++ b/drivers/fpga/intel-afu.h
> @@ -30,11 +30,21 @@ struct fpga_afu_region {
>         struct list_head node;
>  };
>
> +struct fpga_afu_dma_region {
> +       u64 user_addr;
> +       u64 length;
> +       u64 iova;
> +       struct page **pages;
> +       struct rb_node node;
> +       bool in_use;
> +};
> +
>  struct fpga_afu {
>         u64 region_cur_offset;
>         int num_regions;
>         u8 num_umsgs;
>         struct list_head regions;
> +       struct rb_root dma_regions;
>
>         struct feature_platform_data *pdata;
>  };
> @@ -49,4 +59,12 @@ int afu_get_region_by_offset(struct feature_platform_data *pdata,
>                             u64 offset, u64 size,
>                             struct fpga_afu_region *pregion);
>
> +void afu_dma_region_init(struct feature_platform_data *pdata);
> +void afu_dma_region_destroy(struct feature_platform_data *pdata);
> +long afu_dma_map_region(struct feature_platform_data *pdata,
> +                      u64 user_addr, u64 length, u64 *iova);
> +long afu_dma_unmap_region(struct feature_platform_data *pdata, u64 iova);
> +struct fpga_afu_dma_region *afu_dma_region_find(
> +               struct feature_platform_data *pdata, u64 iova, u64 size);
> +
>  #endif
> diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
> index a2ad332..b97ea02 100644
> --- a/include/uapi/linux/intel-fpga.h
> +++ b/include/uapi/linux/intel-fpga.h
> @@ -111,6 +111,43 @@ struct fpga_port_region_info {
>
>  #define FPGA_PORT_GET_REGION_INFO      _IO(FPGA_MAGIC, PORT_BASE + 2)
>
> +/**
> + * FPGA_PORT_DMA_MAP - _IOWR(FPGA_MAGIC, PORT_BASE + 3,
> + *                                             struct fpga_port_dma_map)
> + *
> + * Map the dma memory per user_addr and length which are provided by caller.
> + * Driver fills the iova in provided struct afu_port_dma_map.
> + * This interface only accepts page-size aligned user memory for dma mapping.
> + * Return: 0 on success, -errno on failure.
> + */
> +struct fpga_port_dma_map {
> +       /* Input */
> +       __u32 argsz;            /* Structure length */
> +       __u32 flags;            /* Zero for now */
> +       __u64 user_addr;        /* Process virtual address */
> +       __u64 length;           /* Length of mapping (bytes)*/
> +       /* Output */
> +       __u64 iova;             /* IO virtual address */
> +};
> +
> +#define FPGA_PORT_DMA_MAP      _IO(FPGA_MAGIC, PORT_BASE + 3)
> +
> +/**
> + * FPGA_PORT_DMA_UNMAP - _IOW(FPGA_MAGIC, PORT_BASE + 4,
> + *                                             struct fpga_port_dma_unmap)
> + *
> + * Unmap the dma memory per iova provided by caller.
> + * Return: 0 on success, -errno on failure.
> + */
> +struct fpga_port_dma_unmap {
> +       /* Input */
> +       __u32 argsz;            /* Structure length */
> +       __u32 flags;            /* Zero for now */
> +       __u64 iova;             /* IO virtual address */
> +};
> +
> +#define FPGA_PORT_DMA_UNMAP    _IO(FPGA_MAGIC, PORT_BASE + 4)
> +
>  /* IOCTLs for FME file descriptor */
>
>  /**
> --
> 1.8.3.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Thanks,

Moritz

Re: [PATCH v2 02/22] fpga: add FPGA device framework

[Alan Tull]

On Tue, Aug 1, 2017 at 3:43 AM, Wu Hao <hao.wu@intel.com> wrote:
> On Mon, Jul 31, 2017 at 04:40:16PM -0500, Alan Tull wrote:
>> On Thu, Jul 27, 2017 at 2:10 PM, Rob Herring <robh+dt@kernel.org> wrote:
>> > On Thu, Jul 27, 2017 at 11:35 AM, Alan Tull <atull@kernel.org> wrote:
>> >> On Sun, Jun 25, 2017 at 8:51 PM, Wu Hao <hao.wu@intel.com> wrote:
>> >>
>> >> Hi Rob,
>> >>
>> >> I was hoping to pick your brain a bit on a DT question.
>> >>
>> >>> During FPGA device (e.g PCI-based) discovery, platform devices are
>> >>> registered for different FPGA function units. But the device node path
>> >>> isn't quite friendly to applications.
>> >>>
>> >>> Consider this case, applications want to access child device's sysfs file
>> >>> for some information.
>> >>>
>> >>> 1) Access using bus-based path (e.g PCI)
>> >>>
>> >>>   /sys/bus/pci/devices/xxxxx/fpga_func_a.0/sysfs_file
>> >>>
>> >>>   From the path, it's clear which PCI device is the parent, but not perfect
>> >>>   solution for applications. PCI device BDF is not fixed, application may
>> >>>   need to search all PCI device to find the actual FPGA Device.
>> >>>
>> >>> 2) Or access using platform device path
>> >>>
>> >>>   /sys/bus/platform/devices/fpga_func_a.0/sysfs_file
>> >>>
>> >>>   Applications find the actual function by name easily, but no information
>> >>>   about which fpga device it belongs to. It's quite confusing if multiple
>> >>>   FPGA devices are in one system.
>> >>
>> >> There's a proposal for adding sysfs nodes that correspond to each FPGA
>> >> device., with the devices located on each FPGA under them.  It makes
>> >> it easier to see which device is on which FPGA.
>> >
>> > Makes sense.
>> >
>> >>> 'FPGA Device' class is introduced to resolve this problem. Each node under
>> >>> this class represents a fpga device, which may have one or more child
>> >>> devices. Applications only need to search under this FPGA Device class
>> >>> folder to find the child device node it needs.
>> >>>
>> >>> For example, for the platform has 2 fpga devices, each fpga device has
>> >>> 3 child devices, the hierarchy looks like this.
>> >>>
>> >>> Two nodes are under /sys/class/fpga/:
>> >>> /sys/class/fpga/fpga.0
>> >>> /sys/class/fpga/fpga.1
>> >>>
>> >>> Each node has 1 function A device and 2 function B devices:
>> >>> /sys/class/fpga/fpga.0/func_a.0
>> >>> /sys/class/fpga/fpga.0/func_b.0
>> >>> /sys/class/fpga/fpga.0/func_b.1
>> >>>
>> >>> /sys/class/fpga/fpga.1/func_a.1
>> >>> /sys/class/fpga/fpga.1/func_b.2
>> >>> /sys/class/fpga/fpga.1/func_b.3
>> >
>> > A class is generally what is the function of the device, not how it is
>> > attached. Seems like what you want here is a new bus type if the
>> > existing PCI and platform bus types don't work.
>> >
>> >>
>> >> I can see the value of having sysfs nodes that correspond to fpga
>> >> devices and being able to find devices under them.  I'm thinking what
>> >> that would mean for Device Tree when fpga-dev is used on DT enabled
>> >> systems.  In Device Tree, what is a fpga-dev?
>> >
>> > Just properly setting the parent struct device on the functions should
>> > be enough to figure out which function is in which fpga. I don't see
>> > why a new class is needed.
>> >
>> >> Currently the DT would have a FPGA bridge corresponding to each FPGA's
>> >> hardware bridge and a heirarchy of bridges, regions and devices under
>> >> it.  On systems that don't support partial reconfiguration under the
>> >> OS (so not main bridge that was controlled by the OS), there would be
>> >> a FPGA region, then its child regions, bridges, and devices.
>> >
>> > The FPGA bridges could instantiate fpga bus type devices instead of
>> > platform devices.
>>
>> Yes
>>
>> Some FPGA use cases already have a base bridge per FPGA that could
>> serve as this bus.  But this use case has a static FPGA image +
>> reprogrammable child fpga regions.  There's no base bridge under Linux
>> since the FPGA was programmed and the bridge enabled before Linux
>> boots.   An added base bridge that doesn't touch hardware will be
>> required for this type of use.
>
> Hi Alan
>
> Does 'base bridge' mentioned above mean a hardware bridge just like
> PCIe or USB?

Whatever connects each FPGA to the CPU.  One base bridge per FPGA
device to create the fpga bus type devices.  Each PR region's bridge
would also be a bus.

>
> I tried to use fpga bus type device instead of fpga-dev class today,
> it works for me, e.g Intel FPGA device PCIe driver could create a
> fpga bus type dev as a child of PCIe device and its sysfs path will be
> changed to /sys/bus/fpga/devices/fpga.x/ from /sys/class/fpga/fpga.x/.
> For now, this fpga bus type device is only used as container device,
> so no driver needed for it.

That's great!  I'd like to see the code to try it out with device
tree.  Is it part of fpga-bridge or something separate for now?

>
> Do you have any concern on this? I see fpga bus type works fine, but
> I didn't see other advantages for this case, as we only use it as a
> container device to represent a FPGA device in sysfs hierarchy. :)

I could not see a way to make the fpga-dev class compatible with the
FPGA Device Tree bindings.  This was a red flag. That's why I asked
Rob's opinion.  Sysfs classes collect devices of a specific type
together; busses describe topology.  I think the goal of fpga-dev was
to describe topology.  It's more correct to define this as a bus, not
a class.  If it's done right, it can work for device tree also.

Alan

>
> Thanks
> Hao
>
>>
>> > That's really up to Linux and outside the scope of
>> > the bindings.
>>
>> Thanks for the feedback.
>>
>> Alan Tull
>>
>> >
>> > Rob

Re: [PATCH v2 22/22] fpga: intel: afu: add FPGA_PORT_DMA_MAP/UNMAP ioctls support

[Wu Hao]

On Tue, Aug 01, 2017 at 11:15:48AM -0700, Moritz Fischer wrote:
> Hi Wu,
> 
> couple of minor things inline below.

Hi Moritz,

Thanks a lot for your comments. :)

I will fix all the problems below in the next version patchset.

Thanks
Hao

> 
> On Sun, Jun 25, 2017 at 6:52 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> > DMA memory regions are required for Accelerated Function Unit (AFU) usage.
> > These two ioctls allow user space applications to map user memory regions
> > for dma, and unmap them after use. Iova is returned from driver to user
> > space application via FPGA_PORT_DMA_MAP ioctl. Application needs to unmap
> > it after use, otherwise, driver will unmap them in device file release
> > operation.
> >
> > All the mapped regions are managed via a rb tree.
> >
> > Ioctl interfaces:
> > * FPGA_PORT_DMA_MAP
> >   Do the dma mapping per user_addr and length which provided by user.
> >   Return iova in provided struct afu_port_dma_map.
> >
> > * FPGA_PORT_DMA_UNMAP
> >   Unmap the dma region per iova provided by user.
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> > Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > ---
> > v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
> >     switched to GPLv2 license.
> >     fixed kbuild warnings.
> > ---
> >  drivers/fpga/Makefile               |   3 +-
> >  drivers/fpga/intel-afu-dma-region.c | 372 ++++++++++++++++++++++++++++++++++++
> >  drivers/fpga/intel-afu-main.c       |  61 +++++-
> >  drivers/fpga/intel-afu.h            |  18 ++
> >  include/uapi/linux/intel-fpga.h     |  37 ++++
> >  5 files changed, 489 insertions(+), 2 deletions(-)
> >  create mode 100644 drivers/fpga/intel-afu-dma-region.c
> >
> > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> > index 45c0538..339d1f3 100644
> > --- a/drivers/fpga/Makefile
> > +++ b/drivers/fpga/Makefile
> > @@ -38,4 +38,5 @@ obj-$(CONFIG_INTEL_FPGA_AFU)          += intel-fpga-afu.o
> >
> >  intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
> >  intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
> > -intel-fpga-afu-objs := intel-afu-main.o intel-afu-region.o
> > +intel-fpga-afu-objs := intel-afu-main.o intel-afu-region.o \
> > +                      intel-afu-dma-region.o
> > diff --git a/drivers/fpga/intel-afu-dma-region.c b/drivers/fpga/intel-afu-dma-region.c
> > new file mode 100644
> > index 0000000..982a9b5
> > --- /dev/null
> > +++ b/drivers/fpga/intel-afu-dma-region.c
> > @@ -0,0 +1,372 @@
> > +/*
> > + * Driver for Intel FPGA Accelerated Function Unit (AFU) DMA Region Management
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> > + *
> > + * This work is licensed under the terms of the GNU GPL version 2. See
> > + * the COPYING file in the top-level directory.
> > + */
> > +
> > +#include <linux/sched/signal.h>
> > +#include <linux/uaccess.h>
> > +
> > +#include "intel-afu.h"
> > +
> > +static void put_all_pages(struct page **pages, int npages)
> > +{
> > +       int i;
> > +
> > +       for (i = 0; i < npages; i++)
> > +               if (pages[i] != NULL)
> 
> if (pages[i]) would do I think
> 
> > +                       put_page(pages[i]);
> > +}
> > +
> > +void afu_dma_region_init(struct feature_platform_data *pdata)
> > +{
> > +       struct fpga_afu *afu = fpga_pdata_get_private(pdata);
> > +
> > +       afu->dma_regions = RB_ROOT;
> > +}
> > +
> > +static long afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
> > +{
> > +       unsigned long locked, lock_limit;
> > +       int ret = 0;
> > +
> > +       /* the task is exiting. */
> > +       if (!current->mm)
> > +               return 0;
> > +
> > +       down_write(&current->mm->mmap_sem);
> > +
> > +       if (incr) {
> > +               locked = current->mm->locked_vm + npages;
> > +               lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
> > +
> > +               if (locked > lock_limit && !capable(CAP_IPC_LOCK))
> > +                       ret = -ENOMEM;
> > +               else
> > +                       current->mm->locked_vm += npages;
> > +       } else {
> > +
> > +               if (WARN_ON_ONCE(npages > current->mm->locked_vm))
> > +                       npages = current->mm->locked_vm;
> > +               current->mm->locked_vm -= npages;
> > +       }
> > +
> > +       dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
> > +                               incr ? '+' : '-',
> > +                               npages << PAGE_SHIFT,
> > +                               current->mm->locked_vm << PAGE_SHIFT,
> > +                               rlimit(RLIMIT_MEMLOCK),
> > +                               ret ? "- execeeded" : "");
> > +
> > +       up_write(&current->mm->mmap_sem);
> > +
> > +       return ret;
> > +}
> > +
> > +static long afu_dma_pin_pages(struct feature_platform_data *pdata,
> > +                               struct fpga_afu_dma_region *region)
> > +{
> > +       long npages = region->length >> PAGE_SHIFT;
> > +       struct device *dev = &pdata->dev->dev;
> > +       long ret, pinned;
> > +
> > +       ret = afu_dma_adjust_locked_vm(dev, npages, true);
> > +       if (ret)
> > +               return ret;
> > +
> > +       region->pages = kcalloc(npages, sizeof(struct page *), GFP_KERNEL);
> > +       if (!region->pages) {
> > +               afu_dma_adjust_locked_vm(dev, npages, false);
> 
> You could probably also just have another label in the error handling path.
> > +               return -ENOMEM;
> > +       }
> > +
> > +       pinned = get_user_pages_fast(region->user_addr, npages, 1,
> > +                                       region->pages);
> > +       if (pinned < 0) {
> > +               ret = pinned;
> > +               goto err_put_pages;
> > +       } else if (pinned != npages) {
> > +               ret = -EFAULT;
> > +               goto err;
> > +       }
> > +
> > +       dev_dbg(dev, "%ld pages pinned\n", pinned);
> > +
> > +       return 0;
> > +
> > +err_put_pages:
> > +       put_all_pages(region->pages, pinned);
> > +err:
> > +       kfree(region->pages);
> > +       afu_dma_adjust_locked_vm(dev, npages, false);
> > +       return ret;
> > +}
> > +
> > +static void afu_dma_unpin_pages(struct feature_platform_data *pdata,
> > +                               struct fpga_afu_dma_region *region)
> > +{
> > +       long npages = region->length >> PAGE_SHIFT;
> > +       struct device *dev = &pdata->dev->dev;
> > +
> > +       put_all_pages(region->pages, npages);
> > +       kfree(region->pages);
> > +       afu_dma_adjust_locked_vm(dev, npages, false);
> > +
> > +       dev_dbg(dev, "%ld pages unpinned\n", npages);
> > +}
> > +
> > +static bool afu_dma_check_continuous_pages(struct fpga_afu_dma_region *region)
> > +{
> > +       int npages = region->length >> PAGE_SHIFT;
> > +       int i;
> > +
> > +       for (i = 0; i < npages - 1; i++)
> > +               if (page_to_pfn(region->pages[i]) + 1 !=
> > +                                       page_to_pfn(region->pages[i+1]))
> > +                       return false;
> > +
> > +       return true;
> > +}
> > +
> > +static bool dma_region_check_iova(struct fpga_afu_dma_region *region,
> > +                                 u64 iova, u64 size)
> > +{
> > +       if (!size && region->iova != iova)
> > +               return false;
> > +
> > +       return (region->iova <= iova) &&
> > +               (region->length + region->iova >= iova + size);
> > +}
> > +
> > +/* Need to be called with pdata->lock held */
> 
> Needs
> > +static int afu_dma_region_add(struct feature_platform_data *pdata,
> > +                                       struct fpga_afu_dma_region *region)
> > +{
> > +       struct fpga_afu *afu = fpga_pdata_get_private(pdata);
> > +       struct rb_node **new, *parent = NULL;
> > +
> > +       dev_dbg(&pdata->dev->dev, "add region (iova = %llx)\n",
> > +                                       (unsigned long long)region->iova);
> > +
> > +       new = &(afu->dma_regions.rb_node);
> > +
> > +       while (*new) {
> > +               struct fpga_afu_dma_region *this;
> > +
> > +               this = container_of(*new, struct fpga_afu_dma_region, node);
> > +
> > +               parent = *new;
> > +
> > +               if (dma_region_check_iova(this, region->iova, region->length))
> > +                       return -EEXIST;
> > +
> > +               if (region->iova < this->iova)
> > +                       new = &((*new)->rb_left);
> > +               else if (region->iova > this->iova)
> > +                       new = &((*new)->rb_right);
> > +               else
> > +                       return -EEXIST;
> > +       }
> > +
> > +       rb_link_node(&region->node, parent, new);
> > +       rb_insert_color(&region->node, &afu->dma_regions);
> > +
> > +       return 0;
> > +}
> > +
> > +/* Need to be called with pdata->lock held */
> 
> Ditto
> > +static void afu_dma_region_remove(struct feature_platform_data *pdata,
> > +                                       struct fpga_afu_dma_region *region)
> > +{
> > +       struct fpga_afu *afu;
> > +
> > +       dev_dbg(&pdata->dev->dev, "del region (iova = %llx)\n",
> > +                                       (unsigned long long)region->iova);
> > +
> > +       afu = fpga_pdata_get_private(pdata);
> > +       rb_erase(&region->node, &afu->dma_regions);
> > +}
> > +
> > +/* Need to be called with pdata->lock held */
> 
> Ditto.
> > +void afu_dma_region_destroy(struct feature_platform_data *pdata)
> > +{
> > +       struct fpga_afu *afu = fpga_pdata_get_private(pdata);
> > +       struct rb_node *node = rb_first(&afu->dma_regions);
> > +       struct fpga_afu_dma_region *region;
> > +
> > +       while (node) {
> > +               region = container_of(node, struct fpga_afu_dma_region, node);
> > +
> > +               dev_dbg(&pdata->dev->dev, "del region (iova = %llx)\n",
> > +                                       (unsigned long long)region->iova);
> > +
> > +               rb_erase(node, &afu->dma_regions);
> > +
> > +               if (region->iova)
> > +                       dma_unmap_page(fpga_pdata_to_pcidev(pdata),
> > +                                       region->iova, region->length,
> > +                                       DMA_BIDIRECTIONAL);
> > +
> > +               if (region->pages)
> > +                       afu_dma_unpin_pages(pdata, region);
> > +
> > +               node = rb_next(node);
> > +               kfree(region);
> > +       }
> > +}
> > +
> > +/*
> > + * It finds the dma region from the rbtree based on @iova and @size:
> > + * - if @size == 0, it finds the dma region which starts from @iova
> > + * - otherwise, it finds the dma region which fully contains
> > + *   [@iova, @iova+size)
> > + * If nothing is matched returns NULL.
> > + *
> > + * Need to be called with pdata->lock held.
> > + */
> > +struct fpga_afu_dma_region *
> > +afu_dma_region_find(struct feature_platform_data *pdata, u64 iova, u64 size)
> > +{
> > +       struct fpga_afu *afu = fpga_pdata_get_private(pdata);
> > +       struct rb_node *node = afu->dma_regions.rb_node;
> > +       struct device *dev = &pdata->dev->dev;
> > +
> > +       while (node) {
> > +               struct fpga_afu_dma_region *region;
> > +
> > +               region = container_of(node, struct fpga_afu_dma_region, node);
> > +
> > +               if (dma_region_check_iova(region, iova, size)) {
> > +                       dev_dbg(dev, "find region (iova = %llx)\n",
> > +                               (unsigned long long)region->iova);
> > +                       return region;
> > +               }
> > +
> > +               if (iova < region->iova)
> > +                       node = node->rb_left;
> > +               else if (iova > region->iova)
> > +                       node = node->rb_right;
> > +               else
> > +                       /* the iova region is not fully covered. */
> > +                       break;
> > +       }
> > +
> > +       dev_dbg(dev, "region with iova %llx and size %llx is not found\n",
> > +               (unsigned long long)iova, (unsigned long long)size);
> > +       return NULL;
> > +}
> > +
> > +static struct fpga_afu_dma_region *
> > +afu_dma_region_find_iova(struct feature_platform_data *pdata, u64 iova)
> > +{
> > +       return afu_dma_region_find(pdata, iova, 0);
> > +}
> > +
> > +long afu_dma_map_region(struct feature_platform_data *pdata,
> > +                      u64 user_addr, u64 length, u64 *iova)
> > +{
> > +       struct fpga_afu_dma_region *region;
> > +       int ret;
> > +
> > +       /*
> > +        * Check Inputs, only accept page-aligned user memory region with
> > +        * valid length.
> > +        */
> > +       if (!PAGE_ALIGNED(user_addr) || !PAGE_ALIGNED(length) || !length)
> > +               return -EINVAL;
> > +
> > +       /* Check overflow */
> > +       if (user_addr + length < user_addr)
> > +               return -EINVAL;
> > +
> > +       if (!access_ok(VERIFY_WRITE, (void __user *)(unsigned long)user_addr,
> > +                      length))
> > +               return -EINVAL;
> > +
> > +       region = kzalloc(sizeof(*region), GFP_KERNEL);
> > +       if (!region)
> > +               return -ENOMEM;
> > +
> > +       region->user_addr = user_addr;
> > +       region->length = length;
> > +
> > +       /* Pin the user memory region */
> > +       ret = afu_dma_pin_pages(pdata, region);
> > +       if (ret) {
> > +               dev_err(&pdata->dev->dev, "fail to pin memory region\n");
> > +               goto free_region;
> > +       }
> > +
> > +       /* Only accept continuous pages, return error if no */
> return error else
> 
> > +       if (!afu_dma_check_continuous_pages(region)) {
> > +               dev_err(&pdata->dev->dev, "pages are not continuous\n");
> > +               ret = -EINVAL;
> > +               goto unpin_pages;
> > +       }
> > +
> > +       /* As pages are continuous then start to do DMA mapping */
> > +       region->iova = dma_map_page(fpga_pdata_to_pcidev(pdata),
> > +                                   region->pages[0], 0,
> > +                                   region->length,
> > +                                   DMA_BIDIRECTIONAL);
> > +       if (dma_mapping_error(&pdata->dev->dev, region->iova)) {
> > +               dev_err(&pdata->dev->dev, "fail to map dma mapping\n");
> 'failed to map for dma' ? :)
> 
> > +               ret = -EFAULT;
> > +               goto unpin_pages;
> > +       }
> > +
> > +       *iova = region->iova;
> > +
> > +       mutex_lock(&pdata->lock);
> > +       ret = afu_dma_region_add(pdata, region);
> > +       mutex_unlock(&pdata->lock);
> > +       if (ret) {
> > +               dev_err(&pdata->dev->dev, "fail to add dma region\n");
> > +               goto unmap_dma;
> > +       }
> > +
> > +       return 0;
> > +
> > +unmap_dma:
> > +       dma_unmap_page(fpga_pdata_to_pcidev(pdata),
> > +                      region->iova, region->length, DMA_BIDIRECTIONAL);
> > +unpin_pages:
> > +       afu_dma_unpin_pages(pdata, region);
> > +free_region:
> > +       kfree(region);
> > +       return ret;
> > +}
> > +
> > +long afu_dma_unmap_region(struct feature_platform_data *pdata, u64 iova)
> > +{
> > +       struct fpga_afu_dma_region *region;
> > +
> > +       mutex_lock(&pdata->lock);
> > +       region = afu_dma_region_find_iova(pdata, iova);
> > +       if (!region) {
> > +               mutex_unlock(&pdata->lock);
> > +               return -EINVAL;
> > +       }
> > +
> > +       if (region->in_use) {
> > +               mutex_unlock(&pdata->lock);
> > +               return -EBUSY;
> > +       }
> > +
> > +       afu_dma_region_remove(pdata, region);
> > +       mutex_unlock(&pdata->lock);
> > +
> > +       dma_unmap_page(fpga_pdata_to_pcidev(pdata),
> > +                      region->iova, region->length, DMA_BIDIRECTIONAL);
> > +       afu_dma_unpin_pages(pdata, region);
> > +       kfree(region);
> > +
> > +       return 0;
> > +}
> > diff --git a/drivers/fpga/intel-afu-main.c b/drivers/fpga/intel-afu-main.c
> > index 8c7aa70..d9f1ebf 100644
> > --- a/drivers/fpga/intel-afu-main.c
> > +++ b/drivers/fpga/intel-afu-main.c
> > @@ -175,7 +175,11 @@ static int afu_release(struct inode *inode, struct file *filp)
> >
> >         dev_dbg(&pdev->dev, "Device File Release\n");
> >
> > -       fpga_port_reset(pdev);
> > +       mutex_lock(&pdata->lock);
> > +       __fpga_port_reset(pdev);
> > +       afu_dma_region_destroy(pdata);
> > +       mutex_unlock(&pdata->lock);
> > +
> >         feature_dev_use_end(pdata);
> >         return 0;
> >  }
> > @@ -245,6 +249,55 @@ static long afu_ioctl_check_extension(struct feature_platform_data *pdata,
> >         return 0;
> >  }
> >
> > +static long
> > +afu_ioctl_dma_map(struct feature_platform_data *pdata, void __user *arg)
> > +{
> > +       struct fpga_port_dma_map map;
> > +       unsigned long minsz;
> > +       long ret;
> > +
> > +       minsz = offsetofend(struct fpga_port_dma_map, iova);
> > +
> > +       if (copy_from_user(&map, arg, minsz))
> > +               return -EFAULT;
> > +
> > +       if (map.argsz < minsz || map.flags)
> > +               return -EINVAL;
> > +
> > +       ret = afu_dma_map_region(pdata, map.user_addr, map.length, &map.iova);
> > +       if (ret)
> > +               return ret;
> > +
> > +       if (copy_to_user(arg, &map, sizeof(map))) {
> > +               afu_dma_unmap_region(pdata, map.iova);
> > +               return -EFAULT;
> > +       }
> > +
> > +       dev_dbg(&pdata->dev->dev, "dma map: ua=%llx, len=%llx, iova=%llx\n",
> > +                               (unsigned long long)map.user_addr,
> > +                               (unsigned long long)map.length,
> > +                               (unsigned long long)map.iova);
> > +
> > +       return 0;
> > +}
> > +
> > +static long
> > +afu_ioctl_dma_unmap(struct feature_platform_data *pdata, void __user *arg)
> > +{
> > +       struct fpga_port_dma_unmap unmap;
> > +       unsigned long minsz;
> > +
> > +       minsz = offsetofend(struct fpga_port_dma_unmap, iova);
> > +
> > +       if (copy_from_user(&unmap, arg, minsz))
> > +               return -EFAULT;
> > +
> > +       if (unmap.argsz < minsz || unmap.flags)
> > +               return -EINVAL;
> > +
> > +       return afu_dma_unmap_region(pdata, unmap.iova);
> > +}
> > +
> >  static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
> >  {
> >         struct platform_device *pdev = filp->private_data;
> > @@ -263,6 +316,10 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
> >                 return afu_ioctl_get_info(pdata, (void __user *)arg);
> >         case FPGA_PORT_GET_REGION_INFO:
> >                 return afu_ioctl_get_region_info(pdata, (void __user *)arg);
> > +       case FPGA_PORT_DMA_MAP:
> > +               return afu_ioctl_dma_map(pdata, (void __user *)arg);
> > +       case FPGA_PORT_DMA_UNMAP:
> > +               return afu_ioctl_dma_unmap(pdata, (void __user *)arg);
> >         default:
> >                 /*
> >                  * Let sub-feature's ioctl function to handle the cmd
> > @@ -337,6 +394,7 @@ static int afu_dev_init(struct platform_device *pdev)
> >         mutex_lock(&pdata->lock);
> >         fpga_pdata_set_private(pdata, afu);
> >         afu_region_init(pdata);
> > +       afu_dma_region_init(pdata);
> >         mutex_unlock(&pdata->lock);
> >         return 0;
> >  }
> > @@ -349,6 +407,7 @@ static int afu_dev_destroy(struct platform_device *pdev)
> >         mutex_lock(&pdata->lock);
> >         afu = fpga_pdata_get_private(pdata);
> >         afu_region_destroy(pdata);
> > +       afu_dma_region_destroy(pdata);
> >         fpga_pdata_set_private(pdata, NULL);
> >         mutex_unlock(&pdata->lock);
> >
> > diff --git a/drivers/fpga/intel-afu.h b/drivers/fpga/intel-afu.h
> > index 3417780d..23f7e24 100644
> > --- a/drivers/fpga/intel-afu.h
> > +++ b/drivers/fpga/intel-afu.h
> > @@ -30,11 +30,21 @@ struct fpga_afu_region {
> >         struct list_head node;
> >  };
> >
> > +struct fpga_afu_dma_region {
> > +       u64 user_addr;
> > +       u64 length;
> > +       u64 iova;
> > +       struct page **pages;
> > +       struct rb_node node;
> > +       bool in_use;
> > +};
> > +
> >  struct fpga_afu {
> >         u64 region_cur_offset;
> >         int num_regions;
> >         u8 num_umsgs;
> >         struct list_head regions;
> > +       struct rb_root dma_regions;
> >
> >         struct feature_platform_data *pdata;
> >  };
> > @@ -49,4 +59,12 @@ int afu_get_region_by_offset(struct feature_platform_data *pdata,
> >                             u64 offset, u64 size,
> >                             struct fpga_afu_region *pregion);
> >
> > +void afu_dma_region_init(struct feature_platform_data *pdata);
> > +void afu_dma_region_destroy(struct feature_platform_data *pdata);
> > +long afu_dma_map_region(struct feature_platform_data *pdata,
> > +                      u64 user_addr, u64 length, u64 *iova);
> > +long afu_dma_unmap_region(struct feature_platform_data *pdata, u64 iova);
> > +struct fpga_afu_dma_region *afu_dma_region_find(
> > +               struct feature_platform_data *pdata, u64 iova, u64 size);
> > +
> >  #endif
> > diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
> > index a2ad332..b97ea02 100644
> > --- a/include/uapi/linux/intel-fpga.h
> > +++ b/include/uapi/linux/intel-fpga.h
> > @@ -111,6 +111,43 @@ struct fpga_port_region_info {
> >
> >  #define FPGA_PORT_GET_REGION_INFO      _IO(FPGA_MAGIC, PORT_BASE + 2)
> >
> > +/**
> > + * FPGA_PORT_DMA_MAP - _IOWR(FPGA_MAGIC, PORT_BASE + 3,
> > + *                                             struct fpga_port_dma_map)
> > + *
> > + * Map the dma memory per user_addr and length which are provided by caller.
> > + * Driver fills the iova in provided struct afu_port_dma_map.
> > + * This interface only accepts page-size aligned user memory for dma mapping.
> > + * Return: 0 on success, -errno on failure.
> > + */
> > +struct fpga_port_dma_map {
> > +       /* Input */
> > +       __u32 argsz;            /* Structure length */
> > +       __u32 flags;            /* Zero for now */
> > +       __u64 user_addr;        /* Process virtual address */
> > +       __u64 length;           /* Length of mapping (bytes)*/
> > +       /* Output */
> > +       __u64 iova;             /* IO virtual address */
> > +};
> > +
> > +#define FPGA_PORT_DMA_MAP      _IO(FPGA_MAGIC, PORT_BASE + 3)
> > +
> > +/**
> > + * FPGA_PORT_DMA_UNMAP - _IOW(FPGA_MAGIC, PORT_BASE + 4,
> > + *                                             struct fpga_port_dma_unmap)
> > + *
> > + * Unmap the dma memory per iova provided by caller.
> > + * Return: 0 on success, -errno on failure.
> > + */
> > +struct fpga_port_dma_unmap {
> > +       /* Input */
> > +       __u32 argsz;            /* Structure length */
> > +       __u32 flags;            /* Zero for now */
> > +       __u64 iova;             /* IO virtual address */
> > +};
> > +
> > +#define FPGA_PORT_DMA_UNMAP    _IO(FPGA_MAGIC, PORT_BASE + 4)
> > +
> >  /* IOCTLs for FME file descriptor */
> >
> >  /**
> > --
> > 1.8.3.1
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> Thanks,
> 
> Moritz

Re: [PATCH v2 02/22] fpga: add FPGA device framework

[Wu Hao]

On Tue, Aug 01, 2017 at 04:04:44PM -0500, Alan Tull wrote:
> On Tue, Aug 1, 2017 at 3:43 AM, Wu Hao <hao.wu@intel.com> wrote:
> > On Mon, Jul 31, 2017 at 04:40:16PM -0500, Alan Tull wrote:
> >> On Thu, Jul 27, 2017 at 2:10 PM, Rob Herring <robh+dt@kernel.org> wrote:
> >> > On Thu, Jul 27, 2017 at 11:35 AM, Alan Tull <atull@kernel.org> wrote:
> >> >> On Sun, Jun 25, 2017 at 8:51 PM, Wu Hao <hao.wu@intel.com> wrote:
> >> >>
> >> >> Hi Rob,
> >> >>
> >> >> I was hoping to pick your brain a bit on a DT question.
> >> >>
> >> >>> During FPGA device (e.g PCI-based) discovery, platform devices are
> >> >>> registered for different FPGA function units. But the device node path
> >> >>> isn't quite friendly to applications.
> >> >>>
> >> >>> Consider this case, applications want to access child device's sysfs file
> >> >>> for some information.
> >> >>>
> >> >>> 1) Access using bus-based path (e.g PCI)
> >> >>>
> >> >>>   /sys/bus/pci/devices/xxxxx/fpga_func_a.0/sysfs_file
> >> >>>
> >> >>>   From the path, it's clear which PCI device is the parent, but not perfect
> >> >>>   solution for applications. PCI device BDF is not fixed, application may
> >> >>>   need to search all PCI device to find the actual FPGA Device.
> >> >>>
> >> >>> 2) Or access using platform device path
> >> >>>
> >> >>>   /sys/bus/platform/devices/fpga_func_a.0/sysfs_file
> >> >>>
> >> >>>   Applications find the actual function by name easily, but no information
> >> >>>   about which fpga device it belongs to. It's quite confusing if multiple
> >> >>>   FPGA devices are in one system.
> >> >>
> >> >> There's a proposal for adding sysfs nodes that correspond to each FPGA
> >> >> device., with the devices located on each FPGA under them.  It makes
> >> >> it easier to see which device is on which FPGA.
> >> >
> >> > Makes sense.
> >> >
> >> >>> 'FPGA Device' class is introduced to resolve this problem. Each node under
> >> >>> this class represents a fpga device, which may have one or more child
> >> >>> devices. Applications only need to search under this FPGA Device class
> >> >>> folder to find the child device node it needs.
> >> >>>
> >> >>> For example, for the platform has 2 fpga devices, each fpga device has
> >> >>> 3 child devices, the hierarchy looks like this.
> >> >>>
> >> >>> Two nodes are under /sys/class/fpga/:
> >> >>> /sys/class/fpga/fpga.0
> >> >>> /sys/class/fpga/fpga.1
> >> >>>
> >> >>> Each node has 1 function A device and 2 function B devices:
> >> >>> /sys/class/fpga/fpga.0/func_a.0
> >> >>> /sys/class/fpga/fpga.0/func_b.0
> >> >>> /sys/class/fpga/fpga.0/func_b.1
> >> >>>
> >> >>> /sys/class/fpga/fpga.1/func_a.1
> >> >>> /sys/class/fpga/fpga.1/func_b.2
> >> >>> /sys/class/fpga/fpga.1/func_b.3
> >> >
> >> > A class is generally what is the function of the device, not how it is
> >> > attached. Seems like what you want here is a new bus type if the
> >> > existing PCI and platform bus types don't work.
> >> >
> >> >>
> >> >> I can see the value of having sysfs nodes that correspond to fpga
> >> >> devices and being able to find devices under them.  I'm thinking what
> >> >> that would mean for Device Tree when fpga-dev is used on DT enabled
> >> >> systems.  In Device Tree, what is a fpga-dev?
> >> >
> >> > Just properly setting the parent struct device on the functions should
> >> > be enough to figure out which function is in which fpga. I don't see
> >> > why a new class is needed.
> >> >
> >> >> Currently the DT would have a FPGA bridge corresponding to each FPGA's
> >> >> hardware bridge and a heirarchy of bridges, regions and devices under
> >> >> it.  On systems that don't support partial reconfiguration under the
> >> >> OS (so not main bridge that was controlled by the OS), there would be
> >> >> a FPGA region, then its child regions, bridges, and devices.
> >> >
> >> > The FPGA bridges could instantiate fpga bus type devices instead of
> >> > platform devices.
> >>
> >> Yes
> >>
> >> Some FPGA use cases already have a base bridge per FPGA that could
> >> serve as this bus.  But this use case has a static FPGA image +
> >> reprogrammable child fpga regions.  There's no base bridge under Linux
> >> since the FPGA was programmed and the bridge enabled before Linux
> >> boots.   An added base bridge that doesn't touch hardware will be
> >> required for this type of use.
> >
> > Hi Alan
> >
> > Does 'base bridge' mentioned above mean a hardware bridge just like
> > PCIe or USB?
> 
> Whatever connects each FPGA to the CPU.  One base bridge per FPGA
> device to create the fpga bus type devices.  Each PR region's bridge
> would also be a bus.
> 
> >
> > I tried to use fpga bus type device instead of fpga-dev class today,
> > it works for me, e.g Intel FPGA device PCIe driver could create a
> > fpga bus type dev as a child of PCIe device and its sysfs path will be
> > changed to /sys/bus/fpga/devices/fpga.x/ from /sys/class/fpga/fpga.x/.
> > For now, this fpga bus type device is only used as container device,
> > so no driver needed for it.
> 
> That's great!  I'd like to see the code to try it out with device
> tree.  Is it part of fpga-bridge or something separate for now?
> 

Hi Alan

I just sent the patch I did as a RFC Patch[1] to the mailing list. Please
take a look. I only replaced the original fpga-dev class with new 'fpga'
bus type, and keep the original interface not changed.

[1] http://marc.info/?l=linux-fpga&m=150167682312708&w=2

> >
> > Do you have any concern on this? I see fpga bus type works fine, but
> > I didn't see other advantages for this case, as we only use it as a
> > container device to represent a FPGA device in sysfs hierarchy. :)
> 
> I could not see a way to make the fpga-dev class compatible with the
> FPGA Device Tree bindings.  This was a red flag. That's why I asked
> Rob's opinion.  Sysfs classes collect devices of a specific type
> together; busses describe topology.  I think the goal of fpga-dev was
> to describe topology.  It's more correct to define this as a bus, not
> a class.  If it's done right, it can work for device tree also.

Got it. Thanks. :)

Hao

> 
> Alan
> 
> >
> > Thanks
> > Hao
> >
> >>
> >> > That's really up to Linux and outside the scope of
> >> > the bindings.
> >>
> >> Thanks for the feedback.
> >>
> >> Alan Tull
> >>
> >> >
> >> > Rob

Re: [PATCH v2 02/22] fpga: add FPGA device framework

[Alan Tull]

On Wed, Aug 2, 2017 at 9:07 AM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> On Tue, Aug 01, 2017 at 04:04:44PM -0500, Alan Tull wrote:
>> On Tue, Aug 1, 2017 at 3:43 AM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>> > On Mon, Jul 31, 2017 at 04:40:16PM -0500, Alan Tull wrote:
>> >> On Thu, Jul 27, 2017 at 2:10 PM, Rob Herring <robh+dt-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>> >> > On Thu, Jul 27, 2017 at 11:35 AM, Alan Tull <atull-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>> >> >> On Sun, Jun 25, 2017 at 8:51 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>> >> >>
>> >> >> Hi Rob,
>> >> >>
>> >> >> I was hoping to pick your brain a bit on a DT question.
>> >> >>
>> >> >>> During FPGA device (e.g PCI-based) discovery, platform devices are
>> >> >>> registered for different FPGA function units. But the device node path
>> >> >>> isn't quite friendly to applications.
>> >> >>>
>> >> >>> Consider this case, applications want to access child device's sysfs file
>> >> >>> for some information.
>> >> >>>
>> >> >>> 1) Access using bus-based path (e.g PCI)
>> >> >>>
>> >> >>>   /sys/bus/pci/devices/xxxxx/fpga_func_a.0/sysfs_file
>> >> >>>
>> >> >>>   From the path, it's clear which PCI device is the parent, but not perfect
>> >> >>>   solution for applications. PCI device BDF is not fixed, application may
>> >> >>>   need to search all PCI device to find the actual FPGA Device.
>> >> >>>
>> >> >>> 2) Or access using platform device path
>> >> >>>
>> >> >>>   /sys/bus/platform/devices/fpga_func_a.0/sysfs_file
>> >> >>>
>> >> >>>   Applications find the actual function by name easily, but no information
>> >> >>>   about which fpga device it belongs to. It's quite confusing if multiple
>> >> >>>   FPGA devices are in one system.
>> >> >>
>> >> >> There's a proposal for adding sysfs nodes that correspond to each FPGA
>> >> >> device., with the devices located on each FPGA under them.  It makes
>> >> >> it easier to see which device is on which FPGA.
>> >> >
>> >> > Makes sense.
>> >> >
>> >> >>> 'FPGA Device' class is introduced to resolve this problem. Each node under
>> >> >>> this class represents a fpga device, which may have one or more child
>> >> >>> devices. Applications only need to search under this FPGA Device class
>> >> >>> folder to find the child device node it needs.
>> >> >>>
>> >> >>> For example, for the platform has 2 fpga devices, each fpga device has
>> >> >>> 3 child devices, the hierarchy looks like this.
>> >> >>>
>> >> >>> Two nodes are under /sys/class/fpga/:
>> >> >>> /sys/class/fpga/fpga.0
>> >> >>> /sys/class/fpga/fpga.1
>> >> >>>
>> >> >>> Each node has 1 function A device and 2 function B devices:
>> >> >>> /sys/class/fpga/fpga.0/func_a.0
>> >> >>> /sys/class/fpga/fpga.0/func_b.0
>> >> >>> /sys/class/fpga/fpga.0/func_b.1
>> >> >>>
>> >> >>> /sys/class/fpga/fpga.1/func_a.1
>> >> >>> /sys/class/fpga/fpga.1/func_b.2
>> >> >>> /sys/class/fpga/fpga.1/func_b.3
>> >> >
>> >> > A class is generally what is the function of the device, not how it is
>> >> > attached. Seems like what you want here is a new bus type if the
>> >> > existing PCI and platform bus types don't work.
>> >> >
>> >> >>
>> >> >> I can see the value of having sysfs nodes that correspond to fpga
>> >> >> devices and being able to find devices under them.  I'm thinking what
>> >> >> that would mean for Device Tree when fpga-dev is used on DT enabled
>> >> >> systems.  In Device Tree, what is a fpga-dev?
>> >> >
>> >> > Just properly setting the parent struct device on the functions should
>> >> > be enough to figure out which function is in which fpga. I don't see
>> >> > why a new class is needed.
>> >> >
>> >> >> Currently the DT would have a FPGA bridge corresponding to each FPGA's
>> >> >> hardware bridge and a heirarchy of bridges, regions and devices under
>> >> >> it.  On systems that don't support partial reconfiguration under the
>> >> >> OS (so not main bridge that was controlled by the OS), there would be
>> >> >> a FPGA region, then its child regions, bridges, and devices.
>> >> >
>> >> > The FPGA bridges could instantiate fpga bus type devices instead of
>> >> > platform devices.
>> >>
>> >> Yes
>> >>
>> >> Some FPGA use cases already have a base bridge per FPGA that could
>> >> serve as this bus.  But this use case has a static FPGA image +
>> >> reprogrammable child fpga regions.  There's no base bridge under Linux
>> >> since the FPGA was programmed and the bridge enabled before Linux
>> >> boots.   An added base bridge that doesn't touch hardware will be
>> >> required for this type of use.
>> >
>> > Hi Alan
>> >
>> > Does 'base bridge' mentioned above mean a hardware bridge just like
>> > PCIe or USB?
>>
>> Whatever connects each FPGA to the CPU.  One base bridge per FPGA
>> device to create the fpga bus type devices.  Each PR region's bridge
>> would also be a bus.

device.h [1] says it better than me:

 * A bus is a channel between the processor and one or more devices.
For the
 * purposes of the device model, all devices are connected via a bus,
even if
 * it is an internal, virtual, "platform" bus. Buses can plug into
each other.
 * A USB controller is usually a PCI device, for example. The device
model
 * represents the actual connections between buses and the devices
they control.

A fpga-bridge could be the bus even if the FPGA is on PCIe.  The
advantage of doing it that way is that this bus will be usable for
both embedded and PCIe FPGAs.

Alan

[1] https://github.com/torvalds/linux/blob/master/include/linux/device.h

>>
>> >
>> > I tried to use fpga bus type device instead of fpga-dev class today,
>> > it works for me, e.g Intel FPGA device PCIe driver could create a
>> > fpga bus type dev as a child of PCIe device and its sysfs path will be
>> > changed to /sys/bus/fpga/devices/fpga.x/ from /sys/class/fpga/fpga.x/.
>> > For now, this fpga bus type device is only used as container device,
>> > so no driver needed for it.
>>
>> That's great!  I'd like to see the code to try it out with device
>> tree.  Is it part of fpga-bridge or something separate for now?
>>
>
> Hi Alan
>
> I just sent the patch I did as a RFC Patch[1] to the mailing list. Please
> take a look. I only replaced the original fpga-dev class with new 'fpga'
> bus type, and keep the original interface not changed.
>
> [1] http://marc.info/?l=linux-fpga&m=150167682312708&w=2
>
>> >
>> > Do you have any concern on this? I see fpga bus type works fine, but
>> > I didn't see other advantages for this case, as we only use it as a
>> > container device to represent a FPGA device in sysfs hierarchy. :)
>>
>> I could not see a way to make the fpga-dev class compatible with the
>> FPGA Device Tree bindings.  This was a red flag. That's why I asked
>> Rob's opinion.  Sysfs classes collect devices of a specific type
>> together; busses describe topology.  I think the goal of fpga-dev was
>> to describe topology.  It's more correct to define this as a bus, not
>> a class.  If it's done right, it can work for device tree also.
>
> Got it. Thanks. :)
>
> Hao
>
>>
>> Alan
>>
>> >
>> > Thanks
>> > Hao
>> >
>> >>
>> >> > That's really up to Linux and outside the scope of
>> >> > the bindings.
>> >>
>> >> Thanks for the feedback.
>> >>
>> >> Alan Tull
>> >>
>> >> >
>> >> > Rob
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 03/22] fpga: bridge: remove OF dependency for fpga-bridge

[Alan Tull]

On Sun, Jun 25, 2017 at 8:51 PM, Wu Hao <hao.wu@intel.com> wrote:
> This patch removes OF dependency of fpga-bridge, it allows drivers
> to use fpga-bridge class without device tree support.
>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
Acked-by: Alan Tull <atull@kernel.org>

Re: [PATCH v2 02/22] fpga: add FPGA device framework

[Alan Tull]

On Thu, Jul 27, 2017 at 2:10 PM, Rob Herring <robh+dt-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> On Thu, Jul 27, 2017 at 11:35 AM, Alan Tull <atull-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>> On Sun, Jun 25, 2017 at 8:51 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>>
>> Hi Rob,
>>
>> I was hoping to pick your brain a bit on a DT question.
>>
>>> During FPGA device (e.g PCI-based) discovery, platform devices are
>>> registered for different FPGA function units. But the device node path
>>> isn't quite friendly to applications.
>>>
>>> Consider this case, applications want to access child device's sysfs file
>>> for some information.
>>>
>>> 1) Access using bus-based path (e.g PCI)
>>>
>>>   /sys/bus/pci/devices/xxxxx/fpga_func_a.0/sysfs_file
>>>
>>>   From the path, it's clear which PCI device is the parent, but not perfect
>>>   solution for applications. PCI device BDF is not fixed, application may
>>>   need to search all PCI device to find the actual FPGA Device.
>>>
>>> 2) Or access using platform device path
>>>
>>>   /sys/bus/platform/devices/fpga_func_a.0/sysfs_file
>>>
>>>   Applications find the actual function by name easily, but no information
>>>   about which fpga device it belongs to. It's quite confusing if multiple
>>>   FPGA devices are in one system.
>>
>> There's a proposal for adding sysfs nodes that correspond to each FPGA
>> device., with the devices located on each FPGA under them.  It makes
>> it easier to see which device is on which FPGA.
>
> Makes sense.
>
>>> 'FPGA Device' class is introduced to resolve this problem. Each node under
>>> this class represents a fpga device, which may have one or more child
>>> devices. Applications only need to search under this FPGA Device class
>>> folder to find the child device node it needs.
>>>
>>> For example, for the platform has 2 fpga devices, each fpga device has
>>> 3 child devices, the hierarchy looks like this.
>>>
>>> Two nodes are under /sys/class/fpga/:
>>> /sys/class/fpga/fpga.0
>>> /sys/class/fpga/fpga.1
>>>
>>> Each node has 1 function A device and 2 function B devices:
>>> /sys/class/fpga/fpga.0/func_a.0
>>> /sys/class/fpga/fpga.0/func_b.0
>>> /sys/class/fpga/fpga.0/func_b.1
>>>
>>> /sys/class/fpga/fpga.1/func_a.1
>>> /sys/class/fpga/fpga.1/func_b.2
>>> /sys/class/fpga/fpga.1/func_b.3
>
> A class is generally what is the function of the device, not how it is
> attached. Seems like what you want here is a new bus type if the
> existing PCI and platform bus types don't work.
>
>>
>> I can see the value of having sysfs nodes that correspond to fpga
>> devices and being able to find devices under them.  I'm thinking what
>> that would mean for Device Tree when fpga-dev is used on DT enabled
>> systems.  In Device Tree, what is a fpga-dev?
>
> Just properly setting the parent struct device on the functions should
> be enough to figure out which function is in which fpga. I don't see
> why a new class is needed.
>
>> Currently the DT would have a FPGA bridge corresponding to each FPGA's
>> hardware bridge and a heirarchy of bridges, regions and devices under
>> it.  On systems that don't support partial reconfiguration under the
>> OS (so not main bridge that was controlled by the OS), there would be
>> a FPGA region, then its child regions, bridges, and devices.
>
> The FPGA bridges could instantiate fpga bus type devices instead of
> platform devices.

Seems like of_platform_bus_create() would have to be expanded to
support FPGA bus devices, right?  Or is it acceptable to create
platform devices under the FPGA bus?  I'm still pondering this.
Currently this patchset is creating platform devices under a PCIe bus.

Alan

> That's really up to Linux and outside the scope of
> the bindings.
>
> Rob
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 06/22] fpga: intel: add FPGA PCIe device driver

[Alan Tull]

On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu@intel.com> wrote:

Hi Hao,

Making my way through your patchset.

> From: Zhang Yi <yi.z.zhang@intel.com>
>
> The Intel FPGA device appears as a PCIe device on the system. This patch
> implements the basic framework of the driver for Intel PCIe device which
> locates between CPU and Accelerated Function Units (AFUs).

...which is located between the CPU and...

>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ---
> v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
>     switched to GPLv2 license.
>     fixed comments from Moritz Fischer.
> ---
>  drivers/fpga/Kconfig      |  28 +++++++++++
>  drivers/fpga/Makefile     |   5 ++
>  drivers/fpga/intel-pcie.c | 126 ++++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 159 insertions(+)
>  create mode 100644 drivers/fpga/intel-pcie.c
>
> diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> index c1d8f41..3f3b7f4 100644
> --- a/drivers/fpga/Kconfig
> +++ b/drivers/fpga/Kconfig
> @@ -117,6 +117,34 @@ config XILINX_PR_DECOUPLER
>           region of the FPGA from the busses while that region is
>           being reprogrammed during partial reconfig.
>
> +menuconfig INTEL_FPGA
> +       tristate "Intel(R) FPGA support"
> +       depends on FPGA_DEVICE
> +       help
> +         Select this option to enable driver support for Intel(R)
> +         Field-Programmable Gate Array (FPGA) solutions. This driver

This is confusing since there is FPGA support for Intel FPGA devices
in the kernel already such as any of the Altera support that already
is in this Kconfig.  This description will have to be more specific
that this is one particular Intel solution.

> +         provides interfaces for userspace applications to configure,
> +         enumerate, open, and access FPGA accelerators on platforms
> +         equipped with Intel(R) FPGA solutions and enables system
> +         level management functions such as FPGA reconfiguration,
> +         power management, and virtualization.
> +
> +         Say Y if your platform has this technology. Say N if unsure.
> +
> +if INTEL_FPGA
> +
> +config INTEL_FPGA_PCI
> +       tristate "Intel FPGA PCIe Driver"
> +       depends on PCI
> +       help
> +         This is the driver for the PCIe device which locates between

...which is located between...

> +         CPU and Accelerated Function Units (AFUs) and allows them to
> +         communicate with each other.
> +
> +         To compile this as a module, choose M here.
> +
> +endif
> +
>  endif # FPGA
>
>  endmenu
> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> index 8950a8f..5613133 100644
> --- a/drivers/fpga/Makefile
> +++ b/drivers/fpga/Makefile
> @@ -27,3 +27,8 @@ obj-$(CONFIG_XILINX_PR_DECOUPLER)     += xilinx-pr-decoupler.o
>  # High Level Interfaces
>  obj-$(CONFIG_FPGA_REGION)              += fpga-region.o
>  obj-$(CONFIG_OF_FPGA_REGION)           += of-fpga-region.o
> +
> +# Intel FPGA Support
> +obj-$(CONFIG_INTEL_FPGA_PCI)           += intel-fpga-pci.o
> +
> +intel-fpga-pci-objs := intel-pcie.o
> diff --git a/drivers/fpga/intel-pcie.c b/drivers/fpga/intel-pcie.c
> new file mode 100644
> index 0000000..f697de4
> --- /dev/null
> +++ b/drivers/fpga/intel-pcie.c
> @@ -0,0 +1,126 @@
> +/*
> + * Driver for Intel FPGA PCIe device
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Zhang Yi <Yi.Z.Zhang@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *   Joseph Grecco <joe.grecco@intel.com>
> + *   Enno Luebbers <enno.luebbers@intel.com>
> + *   Tim Whisonant <tim.whisonant@intel.com>
> + *   Ananda Ravuri <ananda.ravuri@intel.com>
> + *   Henry Mitchel <henry.mitchel@intel.com>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2. See
> + * the COPYING file in the top-level directory.
> + */
> +
> +#include <linux/pci.h>
> +#include <linux/types.h>
> +#include <linux/kernel.h>
> +#include <linux/module.h>
> +#include <linux/stddef.h>
> +#include <linux/errno.h>
> +#include <linux/aer.h>
> +
> +#define DRV_VERSION    "0.8"
> +#define DRV_NAME       "intel-fpga-pci"
> +
> +/* PCI Device ID */
> +#define PCIe_DEVICE_ID_PF_INT_5_X      0xBCBD
> +#define PCIe_DEVICE_ID_PF_INT_6_X      0xBCC0
> +#define PCIe_DEVICE_ID_PF_DSC_1_X      0x09C4
> +/* VF Device */
> +#define PCIe_DEVICE_ID_VF_INT_5_X      0xBCBF
> +#define PCIe_DEVICE_ID_VF_INT_6_X      0xBCC1
> +#define PCIe_DEVICE_ID_VF_DSC_1_X      0x09C5
> +
> +static struct pci_device_id cci_pcie_id_tbl[] = {
> +       {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_PF_INT_5_X),},
> +       {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_VF_INT_5_X),},
> +       {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_PF_INT_6_X),},
> +       {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_VF_INT_6_X),},
> +       {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_PF_DSC_1_X),},
> +       {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_VF_DSC_1_X),},
> +       {0,}
> +};
> +MODULE_DEVICE_TABLE(pci, cci_pcie_id_tbl);
> +
> +static
> +int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
> +{
> +       int ret;
> +
> +       ret = pci_enable_device(pcidev);
> +       if (ret < 0) {
> +               dev_err(&pcidev->dev, "Failed to enable device %d.\n", ret);
> +               return ret;
> +       }
> +
> +       ret = pci_enable_pcie_error_reporting(pcidev);
> +       if (ret && ret != -EINVAL)
> +               dev_info(&pcidev->dev, "PCIE AER unavailable %d.\n", ret);
> +
> +       ret = pci_request_regions(pcidev, DRV_NAME);
> +       if (ret) {
> +               dev_err(&pcidev->dev, "Failed to request regions.\n");
> +               goto disable_error_report_exit;
> +       }
> +
> +       pci_set_master(pcidev);
> +       pci_save_state(pcidev);

Is pci_save_state needed here?  Thought it was for going into suspend.

> +
> +       if (!dma_set_mask(&pcidev->dev, DMA_BIT_MASK(64))) {

Please use pci_dma_set_mask()

> +               dma_set_coherent_mask(&pcidev->dev, DMA_BIT_MASK(64));

pci_set_consistent_dma_mask() and check its return code.

> +       } else if (!dma_set_mask(&pcidev->dev, DMA_BIT_MASK(32))) {
> +               dma_set_coherent_mask(&pcidev->dev, DMA_BIT_MASK(32));

Same as above.

> +       } else {
> +               ret = -EIO;
> +               dev_err(&pcidev->dev, "No suitable DMA support available.\n");
> +               goto release_region_exit;
> +       }
> +
> +       /* TODO: create and add the platform device per feature list */
> +       return 0;
> +
> +release_region_exit:
> +       pci_release_regions(pcidev);
> +disable_error_report_exit:
> +       pci_disable_pcie_error_reporting(pcidev);
> +       pci_disable_device(pcidev);
> +       return ret;
> +}
> +
> +static void cci_pci_remove(struct pci_dev *pcidev)
> +{
> +       pci_release_regions(pcidev);
> +       pci_disable_pcie_error_reporting(pcidev);
> +       pci_disable_device(pcidev);
> +}
> +
> +static struct pci_driver cci_pci_driver = {
> +       .name = DRV_NAME,
> +       .id_table = cci_pcie_id_tbl,
> +       .probe = cci_pci_probe,
> +       .remove = cci_pci_remove,
> +};
> +
> +static int __init ccidrv_init(void)
> +{
> +       pr_info("Intel(R) FPGA PCIe Driver: Version %s\n", DRV_VERSION);
> +
> +       return pci_register_driver(&cci_pci_driver);
> +}
> +
> +static void __exit ccidrv_exit(void)
> +{
> +       pci_unregister_driver(&cci_pci_driver);
> +}
> +
> +module_init(ccidrv_init);
> +module_exit(ccidrv_exit);
> +
> +MODULE_DESCRIPTION("Intel FPGA PCIe Device Driver");
> +MODULE_AUTHOR("Intel Corporation");
> +MODULE_LICENSE("GPL v2");
> --
> 1.8.3.1
>

RE: [PATCH v2 06/22] fpga: intel: add FPGA PCIe device driver

[Wu, Hao]

> On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu@intel.com> wrote:
> 
> Hi Hao,
> 
> Making my way through your patchset.

Hi Alan

Thanks a lot for your comments. :)

> 
> > From: Zhang Yi <yi.z.zhang@intel.com>
> >
> > The Intel FPGA device appears as a PCIe device on the system. This patch
> > implements the basic framework of the driver for Intel PCIe device which
> > locates between CPU and Accelerated Function Units (AFUs).
> 
> ...which is located between the CPU and...

Will fix this.

> 
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > ---
> > v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
> >     switched to GPLv2 license.
> >     fixed comments from Moritz Fischer.
> > ---
> >  drivers/fpga/Kconfig      |  28 +++++++++++
> >  drivers/fpga/Makefile     |   5 ++
> >  drivers/fpga/intel-pcie.c | 126
> ++++++++++++++++++++++++++++++++++++++++++++++
> >  3 files changed, 159 insertions(+)
> >  create mode 100644 drivers/fpga/intel-pcie.c
> >
> > diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> > index c1d8f41..3f3b7f4 100644
> > --- a/drivers/fpga/Kconfig
> > +++ b/drivers/fpga/Kconfig
> > @@ -117,6 +117,34 @@ config XILINX_PR_DECOUPLER
> >           region of the FPGA from the busses while that region is
> >           being reprogrammed during partial reconfig.
> >
> > +menuconfig INTEL_FPGA
> > +       tristate "Intel(R) FPGA support"
> > +       depends on FPGA_DEVICE
> > +       help
> > +         Select this option to enable driver support for Intel(R)
> > +         Field-Programmable Gate Array (FPGA) solutions. This driver
> 
> This is confusing since there is FPGA support for Intel FPGA devices
> in the kernel already such as any of the Altera support that already
> is in this Kconfig.  This description will have to be more specific
> that this is one particular Intel solution.

Make sense, will add more specific description here.

> 
> > +         provides interfaces for userspace applications to configure,
> > +         enumerate, open, and access FPGA accelerators on platforms
> > +         equipped with Intel(R) FPGA solutions and enables system
> > +         level management functions such as FPGA reconfiguration,
> > +         power management, and virtualization.
> > +
> > +         Say Y if your platform has this technology. Say N if unsure.
> > +
> > +if INTEL_FPGA
> > +
> > +config INTEL_FPGA_PCI
> > +       tristate "Intel FPGA PCIe Driver"
> > +       depends on PCI
> > +       help
> > +         This is the driver for the PCIe device which locates between
> 
> ...which is located between...

Will fix this.

> 
> > +         CPU and Accelerated Function Units (AFUs) and allows them to
> > +         communicate with each other.
> > +
> > +         To compile this as a module, choose M here.
> > +
> > +endif
> > +
> >  endif # FPGA
> >
> >  endmenu
> > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> > index 8950a8f..5613133 100644
> > --- a/drivers/fpga/Makefile
> > +++ b/drivers/fpga/Makefile
> > @@ -27,3 +27,8 @@ obj-$(CONFIG_XILINX_PR_DECOUPLER)     += xilinx-pr-
> decoupler.o
> >  # High Level Interfaces
> >  obj-$(CONFIG_FPGA_REGION)              += fpga-region.o
> >  obj-$(CONFIG_OF_FPGA_REGION)           += of-fpga-region.o
> > +
> > +# Intel FPGA Support
> > +obj-$(CONFIG_INTEL_FPGA_PCI)           += intel-fpga-pci.o
> > +
> > +intel-fpga-pci-objs := intel-pcie.o
> > diff --git a/drivers/fpga/intel-pcie.c b/drivers/fpga/intel-pcie.c
> > new file mode 100644
> > index 0000000..f697de4
> > --- /dev/null
> > +++ b/drivers/fpga/intel-pcie.c
> > @@ -0,0 +1,126 @@
> > +/*
> > + * Driver for Intel FPGA PCIe device
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Zhang Yi <Yi.Z.Zhang@intel.com>
> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > + *   Joseph Grecco <joe.grecco@intel.com>
> > + *   Enno Luebbers <enno.luebbers@intel.com>
> > + *   Tim Whisonant <tim.whisonant@intel.com>
> > + *   Ananda Ravuri <ananda.ravuri@intel.com>
> > + *   Henry Mitchel <henry.mitchel@intel.com>
> > + *
> > + * This work is licensed under the terms of the GNU GPL version 2. See
> > + * the COPYING file in the top-level directory.
> > + */
> > +
> > +#include <linux/pci.h>
> > +#include <linux/types.h>
> > +#include <linux/kernel.h>
> > +#include <linux/module.h>
> > +#include <linux/stddef.h>
> > +#include <linux/errno.h>
> > +#include <linux/aer.h>
> > +
> > +#define DRV_VERSION    "0.8"
> > +#define DRV_NAME       "intel-fpga-pci"
> > +
> > +/* PCI Device ID */
> > +#define PCIe_DEVICE_ID_PF_INT_5_X      0xBCBD
> > +#define PCIe_DEVICE_ID_PF_INT_6_X      0xBCC0
> > +#define PCIe_DEVICE_ID_PF_DSC_1_X      0x09C4
> > +/* VF Device */
> > +#define PCIe_DEVICE_ID_VF_INT_5_X      0xBCBF
> > +#define PCIe_DEVICE_ID_VF_INT_6_X      0xBCC1
> > +#define PCIe_DEVICE_ID_VF_DSC_1_X      0x09C5
> > +
> > +static struct pci_device_id cci_pcie_id_tbl[] = {
> > +       {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_PF_INT_5_X),},
> > +       {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_VF_INT_5_X),},
> > +       {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_PF_INT_6_X),},
> > +       {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_VF_INT_6_X),},
> > +       {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_PF_DSC_1_X),},
> > +       {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_VF_DSC_1_X),},
> > +       {0,}
> > +};
> > +MODULE_DEVICE_TABLE(pci, cci_pcie_id_tbl);
> > +
> > +static
> > +int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
> > +{
> > +       int ret;
> > +
> > +       ret = pci_enable_device(pcidev);
> > +       if (ret < 0) {
> > +               dev_err(&pcidev->dev, "Failed to enable device %d.\n", ret);
> > +               return ret;
> > +       }
> > +
> > +       ret = pci_enable_pcie_error_reporting(pcidev);
> > +       if (ret && ret != -EINVAL)
> > +               dev_info(&pcidev->dev, "PCIE AER unavailable %d.\n", ret);
> > +
> > +       ret = pci_request_regions(pcidev, DRV_NAME);
> > +       if (ret) {
> > +               dev_err(&pcidev->dev, "Failed to request regions.\n");
> > +               goto disable_error_report_exit;
> > +       }
> > +
> > +       pci_set_master(pcidev);
> > +       pci_save_state(pcidev);
> 
> Is pci_save_state needed here?  Thought it was for going into suspend.
> 

I think no, will remove this in the next version.

> > +
> > +       if (!dma_set_mask(&pcidev->dev, DMA_BIT_MASK(64))) {
> 
> Please use pci_dma_set_mask()

Will fix this.

> 
> > +               dma_set_coherent_mask(&pcidev->dev, DMA_BIT_MASK(64));
> 
> pci_set_consistent_dma_mask() and check its return code.
> 

Will fix this.

> > +       } else if (!dma_set_mask(&pcidev->dev, DMA_BIT_MASK(32))) {
> > +               dma_set_coherent_mask(&pcidev->dev, DMA_BIT_MASK(32));
> 
> Same as above.
>

Will fix this.

Thanks again for the review.

Hao
 
> > +       } else {
> > +               ret = -EIO;
> > +               dev_err(&pcidev->dev, "No suitable DMA support available.\n");
> > +               goto release_region_exit;
> > +       }
> > +
> > +       /* TODO: create and add the platform device per feature list */
> > +       return 0;
> > +
> > +release_region_exit:
> > +       pci_release_regions(pcidev);
> > +disable_error_report_exit:
> > +       pci_disable_pcie_error_reporting(pcidev);
> > +       pci_disable_device(pcidev);
> > +       return ret;
> > +}
> > +
> > +static void cci_pci_remove(struct pci_dev *pcidev)
> > +{
> > +       pci_release_regions(pcidev);
> > +       pci_disable_pcie_error_reporting(pcidev);
> > +       pci_disable_device(pcidev);
> > +}
> > +
> > +static struct pci_driver cci_pci_driver = {
> > +       .name = DRV_NAME,
> > +       .id_table = cci_pcie_id_tbl,
> > +       .probe = cci_pci_probe,
> > +       .remove = cci_pci_remove,
> > +};
> > +
> > +static int __init ccidrv_init(void)
> > +{
> > +       pr_info("Intel(R) FPGA PCIe Driver: Version %s\n", DRV_VERSION);
> > +
> > +       return pci_register_driver(&cci_pci_driver);
> > +}
> > +
> > +static void __exit ccidrv_exit(void)
> > +{
> > +       pci_unregister_driver(&cci_pci_driver);
> > +}
> > +
> > +module_init(ccidrv_init);
> > +module_exit(ccidrv_exit);
> > +
> > +MODULE_DESCRIPTION("Intel FPGA PCIe Device Driver");
> > +MODULE_AUTHOR("Intel Corporation");
> > +MODULE_LICENSE("GPL v2");
> > --
> > 1.8.3.1
> >

Re: [PATCH v2 19/22] fpga: intel: afu: add header sub feature support

[Alan Tull]

 On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu@intel.com> wrote:

Hi Hao,

> The header register set is always present for the Port/AFU, it is mainly
> for capability, control and status of the ports that AFU connected to.

So just to be clear, the reset function is acting on the Port, not the
AFU, right?

>
> This patch implements header sub feature support.

Please add a brief reminder here what the 'header' is.  It's defined
in patch 7 as being part of the feature list, but hardly mentioned
when I grep intel-fpga.txt.

> Below user interfaces
> are created by this patch.
>
> Sysfs interface:
> * /sys/class/fpga/<fpga.x>/<intel-fpga-port.x>/id
>   Read-only. Port ID.
>
> Ioctl interface:
> * FPGA_PORT_RESET
>   Reset the FPGA AFU Port.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ---
> v2: add sysfs documentation.
> ---
>  .../ABI/testing/sysfs-platform-intel-fpga-afu      |  7 ++++
>  drivers/fpga/intel-afu-main.c                      | 44 +++++++++++++++++++++-
>  include/uapi/linux/intel-fpga.h                    | 14 +++++++
>  3 files changed, 64 insertions(+), 1 deletion(-)
>  create mode 100644 Documentation/ABI/testing/sysfs-platform-intel-fpga-afu
>
> diff --git a/Documentation/ABI/testing/sysfs-platform-intel-fpga-afu b/Documentation/ABI/testing/sysfs-platform-intel-fpga-afu
> new file mode 100644
> index 0000000..8ad22c9
> --- /dev/null
> +++ b/Documentation/ABI/testing/sysfs-platform-intel-fpga-afu
> @@ -0,0 +1,7 @@
> +What:          /sys/bus/platform/devices/intel-fpga-port.0/id
> +Date:          June 2017
> +KernelVersion:  4.12
> +Contact:       Wu Hao <hao.wu@intel.com>
> +Description:   Read-only. It returns id of this port. One Intel FPGA device
> +               may have more than one port. Userspace could use this id to
> +               distinguish different ports under same FPGA device.
> diff --git a/drivers/fpga/intel-afu-main.c b/drivers/fpga/intel-afu-main.c
> index 96d0367..2a17cde 100644
> --- a/drivers/fpga/intel-afu-main.c
> +++ b/drivers/fpga/intel-afu-main.c
> @@ -18,25 +18,66 @@
>
>  #include <linux/kernel.h>
>  #include <linux/module.h>
> +#include <linux/intel-fpga.h>
>
>  #include "intel-feature-dev.h"
>
> +static ssize_t
> +id_show(struct device *dev, struct device_attribute *attr, char *buf)
> +{
> +       int id = fpga_port_id(to_platform_device(dev));
> +
> +       return scnprintf(buf, PAGE_SIZE, "%d\n", id);
> +}
> +static DEVICE_ATTR_RO(id);
> +
> +static const struct attribute *port_hdr_attrs[] = {
> +       &dev_attr_id.attr,
> +       NULL,
> +};
> +
>  static int port_hdr_init(struct platform_device *pdev, struct feature *feature)
>  {
>         dev_dbg(&pdev->dev, "PORT HDR Init.\n");
>
> -       return 0;
> +       fpga_port_reset(pdev);

So the port will be reset here, which happens during fme_probe().
IIUC the PR region is empty then, there is just the static region,
right?

> +
> +       return sysfs_create_files(&pdev->dev.kobj, port_hdr_attrs);

Greg wrote an article that there could be a race condition caused by
creating sysfs files this late [1] and I see sysfs_create_files() used
very sparingly in the kernel.  I'm thinking that fpga-bridge should
provide a place to create sysfs files earlier by adding an
attribute_group to fpga_bridge_ops (same for fpga-mgr and fpga-region)
and then fpga_bridge_register could do bridge->dev.groups =
br_ops->groups.  I'll put a patch for that out soon.

>  }
>
>  static void port_hdr_uinit(struct platform_device *pdev,
>                                         struct feature *feature)
>  {
>         dev_dbg(&pdev->dev, "PORT HDR UInit.\n");
> +
> +       sysfs_remove_files(&pdev->dev.kobj, port_hdr_attrs);
> +}
> +
> +static long
> +port_hdr_ioctl(struct platform_device *pdev, struct feature *feature,
> +                                       unsigned int cmd, unsigned long arg)
> +{
> +       long ret;
> +
> +       switch (cmd) {
> +       case FPGA_PORT_RESET:
> +               if (!arg)
> +                       ret = fpga_port_reset(pdev);

fpga_port_reset() disables and reenables traffic on the port.  Is
there ever a time when that would be unsafe to do?  Like while DMA is
happening?  When I see a function called 'reset' exposed to userspace,
I become concerned that hitting that reset at the wrong time could
cause problems.  We've discussed this some, but could you please
remind me when userspace would need to reset the port?  Please add
documentation of what the intended use of this ioctl would be, when it
is valid to request a reset from userspace and when userspace should
never do that.

The pcie code, the AFU file interface, and the bridge code all do port
reset.  This code is spread out over a few patches, but I'll comment
here for now.  I'm trying to keep track of everything that resets the
port.  The port gets reset in afu_probe, fme_probe, the AFU file
release, and intel-pcie.c after parsing the features.  Also the port
is esentially reset after doing reprogramming, since that involves a
bridge disable/enable.   In the v1 review, the issue raised that the
port functionality could be an expansion of fpga-bridge.  If reset
were added to the fpga_bridge_ops and a fpga_bridge_reset API added to
fpga-bridge, then anything in the kernel that owns the bridge could
reset it.  That is of course assuming that this code doesn't need to
reset the port before the fpga-bridge is created.

Thanks,
Alan Tull

[1] http://www.kroah.com/log/blog/2013/06/26/how-to-create-a-sysfs-file-correctly/

> +               else
> +                       ret = -EINVAL;
> +               break;
> +       default:
> +               dev_dbg(&pdev->dev, "%x cmd not handled", cmd);
> +               ret = -ENODEV;
> +       }
> +
> +       return ret;
>  }
>
>  struct feature_ops port_hdr_ops = {
>         .init = port_hdr_init,
>         .uinit = port_hdr_uinit,
> +       .ioctl = port_hdr_ioctl,
>  };
>
>  static struct feature_driver port_feature_drvs[] = {
> @@ -76,6 +117,7 @@ static int afu_release(struct inode *inode, struct file *filp)
>
>         dev_dbg(&pdev->dev, "Device File Release\n");
>
> +       fpga_port_reset(pdev);
>         feature_dev_use_end(pdata);
>         return 0;
>  }
> diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
> index be295ae..be5f813 100644
> --- a/include/uapi/linux/intel-fpga.h
> +++ b/include/uapi/linux/intel-fpga.h
> @@ -30,8 +30,11 @@
>  #define FPGA_MAGIC 0xB6
>
>  #define FPGA_BASE 0
> +#define PORT_BASE 0x40
>  #define FME_BASE 0x80
>
> +/* Common IOCTLs for both FME and AFU file descriptor */
> +
>  /**
>   * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
>   *
> @@ -50,6 +53,17 @@
>
>  #define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)
>
> +/* IOCTLs for AFU file descriptor */
> +
> +/**
> + * FPGA_PORT_RESET - _IO(FPGA_MAGIC, PORT_BASE + 0)
> + *
> + * Reset the FPGA AFU Port. No parameters are supported.
> + * Return: 0 on success, -errno of failure
> + */
> +
> +#define FPGA_PORT_RESET                _IO(FPGA_MAGIC, PORT_BASE + 0)
> +
>  /* IOCTLs for FME file descriptor */
>
>  /**
> --
> 1.8.3.1
>

RE: [PATCH v2 19/22] fpga: intel: afu: add header sub feature support

[Wu, Hao]

>  On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu@intel.com> wrote:
> 
> Hi Hao,
> 
> > The header register set is always present for the Port/AFU, it is mainly
> > for capability, control and status of the ports that AFU connected to.
> 
> So just to be clear, the reset function is acting on the Port, not the
> AFU, right?

Hi Alan

AFU will be reset as well. Once port reset is issued, a compliant HW
AFU will reset all its state and stop sending requests, and HW will set
port reset ack bit when all outstanding requests initiated have been
drained.

Will add some notes here.

> 
> >
> > This patch implements header sub feature support.
> 
> Please add a brief reminder here what the 'header' is.  It's defined
> in patch 7 as being part of the feature list, but hardly mentioned
> when I grep intel-fpga.txt.

Sure, will adds some notes here.

Actually the header sub feature means the registers belong to the
feature device (e.g port and FME), not any sub features (e.g PR, 
Power management).

> 
> > Below user interfaces
> > are created by this patch.
> >
> > Sysfs interface:
> > * /sys/class/fpga/<fpga.x>/<intel-fpga-port.x>/id
> >   Read-only. Port ID.
> >
> > Ioctl interface:
> > * FPGA_PORT_RESET
> >   Reset the FPGA AFU Port.
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > ---
> > v2: add sysfs documentation.
> > ---
> >  .../ABI/testing/sysfs-platform-intel-fpga-afu      |  7 ++++
> >  drivers/fpga/intel-afu-main.c                      | 44 +++++++++++++++++++++-
> >  include/uapi/linux/intel-fpga.h                    | 14 +++++++
> >  3 files changed, 64 insertions(+), 1 deletion(-)
> >  create mode 100644 Documentation/ABI/testing/sysfs-platform-intel-fpga-
> afu
> >
> > diff --git a/Documentation/ABI/testing/sysfs-platform-intel-fpga-afu
> b/Documentation/ABI/testing/sysfs-platform-intel-fpga-afu
> > new file mode 100644
> > index 0000000..8ad22c9
> > --- /dev/null
> > +++ b/Documentation/ABI/testing/sysfs-platform-intel-fpga-afu
> > @@ -0,0 +1,7 @@
> > +What:          /sys/bus/platform/devices/intel-fpga-port.0/id
> > +Date:          June 2017
> > +KernelVersion:  4.12
> > +Contact:       Wu Hao <hao.wu@intel.com>
> > +Description:   Read-only. It returns id of this port. One Intel FPGA device
> > +               may have more than one port. Userspace could use this id to
> > +               distinguish different ports under same FPGA device.
> > diff --git a/drivers/fpga/intel-afu-main.c b/drivers/fpga/intel-afu-main.c
> > index 96d0367..2a17cde 100644
> > --- a/drivers/fpga/intel-afu-main.c
> > +++ b/drivers/fpga/intel-afu-main.c
> > @@ -18,25 +18,66 @@
> >
> >  #include <linux/kernel.h>
> >  #include <linux/module.h>
> > +#include <linux/intel-fpga.h>
> >
> >  #include "intel-feature-dev.h"
> >
> > +static ssize_t
> > +id_show(struct device *dev, struct device_attribute *attr, char *buf)
> > +{
> > +       int id = fpga_port_id(to_platform_device(dev));
> > +
> > +       return scnprintf(buf, PAGE_SIZE, "%d\n", id);
> > +}
> > +static DEVICE_ATTR_RO(id);
> > +
> > +static const struct attribute *port_hdr_attrs[] = {
> > +       &dev_attr_id.attr,
> > +       NULL,
> > +};
> > +
> >  static int port_hdr_init(struct platform_device *pdev, struct feature *feature)
> >  {
> >         dev_dbg(&pdev->dev, "PORT HDR Init.\n");
> >
> > -       return 0;
> > +       fpga_port_reset(pdev);
> 
> So the port will be reset here, which happens during fme_probe().
> IIUC the PR region is empty then, there is just the static region,
> right?

port_hdr_init is invoked during afu_probe() function. The fpga_port_reset
only resets the AFU's state and not empty the PR region. User doesn't need
to program it again after port reset.

The purpose of this reset in port_hdr_init function, is to make sure that we 
could have a clean start whenever the port driver module is loaded. And
similar as the one added in afu release.

> 
> > +
> > +       return sysfs_create_files(&pdev->dev.kobj, port_hdr_attrs);
> 
> Greg wrote an article that there could be a race condition caused by
> creating sysfs files this late [1] and I see sysfs_create_files() used
> very sparingly in the kernel.  I'm thinking that fpga-bridge should
> provide a place to create sysfs files earlier by adding an
> attribute_group to fpga_bridge_ops (same for fpga-mgr and fpga-region)
> and then fpga_bridge_register could do bridge->dev.groups =
> br_ops->groups.  I'll put a patch for that out soon.
> 

Hm... I understand there could be a race condition if creates sysfs files late.
Actually the reasons I prefer to have each sub feature to create its own sysfs
files are, 1) if any sub feature is not present, then related init function won't 
be invoked and related sysfs files won't be created at all. Then end user could
know which sub features are present by checking these sysfs nodes easily. 
2) Another point of view is about extension of each sub feature, for example,
Some new registers introduced when sub feature's revision > n, so each sub
feature's init function could check the sub feature's revision to decide which
sysfs interfaces are needed. I think this should be a flexible way for extension.

> >  }
> >
> >  static void port_hdr_uinit(struct platform_device *pdev,
> >                                         struct feature *feature)
> >  {
> >         dev_dbg(&pdev->dev, "PORT HDR UInit.\n");
> > +
> > +       sysfs_remove_files(&pdev->dev.kobj, port_hdr_attrs);
> > +}
> > +
> > +static long
> > +port_hdr_ioctl(struct platform_device *pdev, struct feature *feature,
> > +                                       unsigned int cmd, unsigned long arg)
> > +{
> > +       long ret;
> > +
> > +       switch (cmd) {
> > +       case FPGA_PORT_RESET:
> > +               if (!arg)
> > +                       ret = fpga_port_reset(pdev);
> 
> fpga_port_reset() disables and reenables traffic on the port.  Is
> there ever a time when that would be unsafe to do?  Like while DMA is
> happening?  When I see a function called 'reset' exposed to userspace,
> I become concerned that hitting that reset at the wrong time could
> cause problems.  We've discussed this some, but could you please
> remind me when userspace would need to reset the port?  Please add
> documentation of what the intended use of this ioctl would be, when it
> is valid to request a reset from userspace and when userspace should
> never do that.

Sure, I will add some more description in uapi header file on this IOCTL.

Per my understanding, this API could be invoked at any time, even there
is some DMA operation or PR operation at the same time. It should not
cause any system level problem, but may cause functional failures ( e.g
DMA or PR operation failure) and it should be recoverable.

> 
> The pcie code, the AFU file interface, and the bridge code all do port
> reset.  This code is spread out over a few patches, but I'll comment
> here for now.  I'm trying to keep track of everything that resets the
> port.  The port gets reset in afu_probe, fme_probe, the AFU file
> release, and intel-pcie.c after parsing the features.  Also the port
> is esentially reset after doing reprogramming, since that involves a
> bridge disable/enable.   In the v1 review, the issue raised that the
> port functionality could be an expansion of fpga-bridge.  If reset
> were added to the fpga_bridge_ops and a fpga_bridge_reset API added to
> fpga-bridge, then anything in the kernel that owns the bridge could
> reset it.  That is of course assuming that this code doesn't need to
> reset the port before the fpga-bridge is created.

Actually in the pcie code, it needs to enable fpga port (put it out of reset,
as port is in reset by default), otherwise the AFU MMIO region is not
accessible. It's only a one time action.

For the AFU interface, it gives application a way to reset the AFU if
anything goes wrong or application wants to make AFU back to 
default state.

For the bridge code, it just makes sure port is in reset during PR, which
is requested by the PR flow of Intel FPGA device.

As you know, the fpga-bridge is created under FME module now, and
port/AFU is another module which could be turned into a VF and assigned
to a virtual machine, even we added a reset ops to fpga-bridge, we still
need an interface to do port reset as fpga-bridges (and FME) are always
in PF in host.  :)

Thanks
Hao

> 
> Thanks,
> Alan Tull
> 
> [1] http://www.kroah.com/log/blog/2013/06/26/how-to-create-a-sysfs-file-
> correctly/
> 
> > +               else
> > +                       ret = -EINVAL;
> > +               break;
> > +       default:
> > +               dev_dbg(&pdev->dev, "%x cmd not handled", cmd);
> > +               ret = -ENODEV;
> > +       }
> > +
> > +       return ret;
> >  }
> >
> >  struct feature_ops port_hdr_ops = {
> >         .init = port_hdr_init,
> >         .uinit = port_hdr_uinit,
> > +       .ioctl = port_hdr_ioctl,
> >  };
> >
> >  static struct feature_driver port_feature_drvs[] = {
> > @@ -76,6 +117,7 @@ static int afu_release(struct inode *inode, struct file
> *filp)
> >
> >         dev_dbg(&pdev->dev, "Device File Release\n");
> >
> > +       fpga_port_reset(pdev);
> >         feature_dev_use_end(pdata);
> >         return 0;
> >  }
> > diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
> > index be295ae..be5f813 100644
> > --- a/include/uapi/linux/intel-fpga.h
> > +++ b/include/uapi/linux/intel-fpga.h
> > @@ -30,8 +30,11 @@
> >  #define FPGA_MAGIC 0xB6
> >
> >  #define FPGA_BASE 0
> > +#define PORT_BASE 0x40
> >  #define FME_BASE 0x80
> >
> > +/* Common IOCTLs for both FME and AFU file descriptor */
> > +
> >  /**
> >   * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
> >   *
> > @@ -50,6 +53,17 @@
> >
> >  #define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)
> >
> > +/* IOCTLs for AFU file descriptor */
> > +
> > +/**
> > + * FPGA_PORT_RESET - _IO(FPGA_MAGIC, PORT_BASE + 0)
> > + *
> > + * Reset the FPGA AFU Port. No parameters are supported.
> > + * Return: 0 on success, -errno of failure
> > + */
> > +
> > +#define FPGA_PORT_RESET                _IO(FPGA_MAGIC, PORT_BASE + 0)
> > +
> >  /* IOCTLs for FME file descriptor */
> >
> >  /**
> > --
> > 1.8.3.1
> >

Re: [PATCH v2 18/22] fpga: intel: add FPGA Accelerated Function Unit driver basic framework

[Alan Tull]

On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

Hi Hao,

Just a few minor things below.  Otherwise, it's fine.

> On Intel FPGA devices, the Accelerated Function Unit (AFU), can be
> reprogrammed for different functions. It connects to the FPGA
> infrastructure("blue bistream") via a Port. Port CSRs are implemented
> separately from the AFU CSRs to provide control and status of the Port.
> Once valid green bitstream is programmed into the AFU, it allows access
> to the AFU CSRs in the AFU MMIO space.
>
> This patch only implements basic driver framework for AFU, including
> device file operation framework.
>
> Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Acked-by: Alan Tull <atull-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>

> ---
> v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
>     switched to GPLv2 license.
> ---
>  drivers/fpga/Kconfig          |   9 +++
>  drivers/fpga/Makefile         |   2 +
>  drivers/fpga/intel-afu-main.c | 159 ++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 170 insertions(+)
>  create mode 100644 drivers/fpga/intel-afu-main.c
>
> diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> index db1a6ea..4658a13 100644
> --- a/drivers/fpga/Kconfig
> +++ b/drivers/fpga/Kconfig
> @@ -172,6 +172,15 @@ config INTEL_FPGA_FME_REGION
>           Say Y to enable FPGA Region driver for Intel FPGA Management
>           Engine.
>
> +config INTEL_FPGA_AFU
> +       tristate "Intel FPGA AFU Driver"
> +       depends on INTEL_FPGA_PCI
> +       help
> +         This is the driver for FPGA Accelerated Function Unit (AFU) which
> +         implements AFU and Port management features. A User AFU connects
> +         to the FPGA infrastructure via a Port. There may be more than 1
> +         Port/AFU per Intel FPGA.
> +
>  endif
>
>  endif # FPGA
> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> index fd0a71f..ce08833 100644
> --- a/drivers/fpga/Makefile
> +++ b/drivers/fpga/Makefile
> @@ -34,6 +34,8 @@ obj-$(CONFIG_INTEL_FPGA_FME)          += intel-fpga-fme.o
>  obj-$(CONFIG_INTEL_FPGA_FME_MGR)       += intel-fpga-fme-mgr.o
>  obj-$(CONFIG_INTEL_FPGA_FME_BRIDGE)    += intel-fpga-fme-br.o
>  obj-$(CONFIG_INTEL_FPGA_FME_REGION)    += intel-fpga-fme-region.o
> +obj-$(CONFIG_INTEL_FPGA_AFU)           += intel-fpga-afu.o
>
>  intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
>  intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
> +intel-fpga-afu-objs := intel-afu-main.o
> diff --git a/drivers/fpga/intel-afu-main.c b/drivers/fpga/intel-afu-main.c
> new file mode 100644
> index 0000000..96d0367
> --- /dev/null
> +++ b/drivers/fpga/intel-afu-main.c
> @@ -0,0 +1,159 @@
> +/*
> + * Driver for Intel FPGA Accelerated Function Unit (AFU)
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> + *   Joseph Grecco <joe.grecco-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *   Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *   Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *   Ananda Ravuri <ananda.ravuri-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *   Henry Mitchel <henry.mitchel-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2. See
> + * the COPYING file in the top-level directory.
> + */
> +
> +#include <linux/kernel.h>
> +#include <linux/module.h>
> +
> +#include "intel-feature-dev.h"
> +
> +static int port_hdr_init(struct platform_device *pdev, struct feature *feature)
> +{
> +       dev_dbg(&pdev->dev, "PORT HDR Init.\n");
> +
> +       return 0;
> +}
> +
> +static void port_hdr_uinit(struct platform_device *pdev,
> +                                       struct feature *feature)
> +{
> +       dev_dbg(&pdev->dev, "PORT HDR UInit.\n");
> +}
> +
> +struct feature_ops port_hdr_ops = {

static const struct?

> +       .init = port_hdr_init,
> +       .uinit = port_hdr_uinit,
> +};
> +
> +static struct feature_driver port_feature_drvs[] = {

static const struct?

> +       {
> +               .name = PORT_FEATURE_HEADER,
> +               .ops = &port_hdr_ops,
> +       },
> +       {
> +               .ops = NULL,
> +       }
> +};
> +
> +static int afu_open(struct inode *inode, struct file *filp)
> +{
> +       struct platform_device *fdev = fpga_inode_to_feature_dev(inode);
> +       struct feature_platform_data *pdata;
> +       int ret;
> +
> +       pdata = dev_get_platdata(&fdev->dev);
> +       if (WARN_ON(!pdata))
> +               return -ENODEV;
> +
> +       ret = feature_dev_use_begin(pdata);
> +       if (ret)
> +               return ret;
> +
> +       dev_dbg(&fdev->dev, "Device File Open\n");
> +       filp->private_data = fdev;
> +
> +       return 0;
> +}
> +
> +static int afu_release(struct inode *inode, struct file *filp)
> +{
> +       struct platform_device *pdev = filp->private_data;
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +
> +       dev_dbg(&pdev->dev, "Device File Release\n");
> +
> +       feature_dev_use_end(pdata);

Please skip a line before return.

> +       return 0;
> +}
> +
> +static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
> +{
> +       struct platform_device *pdev = filp->private_data;
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct feature *f;
> +       long ret;
> +
> +       dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
> +
> +       switch (cmd) {
> +       default:
> +               /*
> +                * Let sub-feature's ioctl function to handle the cmd
> +                * Sub-feature's ioctl returns -ENODEV when cmd is not
> +                * handled in this sub feature, and returns 0 and other
> +                * error code if cmd is handled.
> +                */
> +               fpga_dev_for_each_feature(pdata, f)
> +                       if (f->ops && f->ops->ioctl) {
> +                               ret = f->ops->ioctl(pdev, f, cmd, arg);
> +                               if (ret == -ENODEV)
> +                                       continue;
> +                               else
> +                                       return ret;
> +                       }
> +       }
> +
> +       return -EINVAL;
> +}
> +
> +static const struct file_operations afu_fops = {
> +       .owner = THIS_MODULE,
> +       .open = afu_open,
> +       .release = afu_release,
> +       .unlocked_ioctl = afu_ioctl,
> +};
> +
> +static int afu_probe(struct platform_device *pdev)
> +{
> +       int ret;
> +
> +       dev_dbg(&pdev->dev, "%s\n", __func__);
> +
> +       ret = fpga_dev_feature_init(pdev, port_feature_drvs);
> +       if (ret)
> +               return ret;
> +
> +       ret = fpga_register_dev_ops(pdev, &afu_fops, THIS_MODULE);
> +       if (ret)
> +               fpga_dev_feature_uinit(pdev);
> +
> +       return ret;
> +}
> +
> +static int afu_remove(struct platform_device *pdev)
> +{
> +       dev_dbg(&pdev->dev, "%s\n", __func__);
> +
> +       fpga_dev_feature_uinit(pdev);
> +       fpga_unregister_dev_ops(pdev);

Please skip a line before return.

> +       return 0;
> +}
> +
> +static struct platform_driver afu_driver = {
> +       .driver = {
> +               .name    = "intel-fpga-port",
> +       },
> +       .probe   = afu_probe,
> +       .remove  = afu_remove,
> +};
> +
> +module_platform_driver(afu_driver);
> +
> +MODULE_DESCRIPTION("Intel FPGA Accelerated Function Unit driver");
> +MODULE_AUTHOR("Intel Corporation");
> +MODULE_LICENSE("GPL v2");
> +MODULE_ALIAS("platform:intel-fpga-port");
> --
> 1.8.3.1
>

Re: [PATCH v2 20/22] fpga: intel: afu add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support

[Alan Tull]

On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu@intel.com> wrote:
> FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which
> need to be supported by all feature devices drivers including FME and AFU.
> This patch implements above 2 ioctls in Intel FPGA Accelerated Function
> Unit (AFU) driver.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
Acked-by: Alan Tull <atull@kernel.org>
> ---
> v2: rebased
> ---
>  drivers/fpga/intel-afu-main.c | 11 +++++++++++
>  1 file changed, 11 insertions(+)
>
> diff --git a/drivers/fpga/intel-afu-main.c b/drivers/fpga/intel-afu-main.c
> index 2a17cde..22f77f2 100644
> --- a/drivers/fpga/intel-afu-main.c
> +++ b/drivers/fpga/intel-afu-main.c
> @@ -122,6 +122,13 @@ static int afu_release(struct inode *inode, struct file *filp)
>         return 0;
>  }
>
> +static long afu_ioctl_check_extension(struct feature_platform_data *pdata,
> +                                    unsigned long arg)
> +{
> +       /* No extension support for now */
> +       return 0;
> +}
> +
>  static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
>  {
>         struct platform_device *pdev = filp->private_data;
> @@ -132,6 +139,10 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
>         dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
>
>         switch (cmd) {
> +       case FPGA_GET_API_VERSION:
> +               return FPGA_API_VERSION;
> +       case FPGA_CHECK_EXTENSION:
> +               return afu_ioctl_check_extension(pdata, arg);
>         default:
>                 /*
>                  * Let sub-feature's ioctl function to handle the cmd
> --
> 1.8.3.1
>

Re: [PATCH v2 18/22] fpga: intel: add FPGA Accelerated Function Unit driver basic framework

[Moritz Fischer]

On Sun, Jun 25, 2017 at 6:52 PM, Wu Hao <hao.wu@intel.com> wrote:
> On Intel FPGA devices, the Accelerated Function Unit (AFU), can be
> reprogrammed for different functions. It connects to the FPGA
> infrastructure("blue bistream") via a Port. Port CSRs are implemented
> separately from the AFU CSRs to provide control and status of the Port.
> Once valid green bitstream is programmed into the AFU, it allows access
> to the AFU CSRs in the AFU MMIO space.
>
> This patch only implements basic driver framework for AFU, including
> device file operation framework.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ---
> v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
>     switched to GPLv2 license.
> ---
>  drivers/fpga/Kconfig          |   9 +++
>  drivers/fpga/Makefile         |   2 +
>  drivers/fpga/intel-afu-main.c | 159 ++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 170 insertions(+)
>  create mode 100644 drivers/fpga/intel-afu-main.c
>
> diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> index db1a6ea..4658a13 100644
> --- a/drivers/fpga/Kconfig
> +++ b/drivers/fpga/Kconfig
> @@ -172,6 +172,15 @@ config INTEL_FPGA_FME_REGION
>           Say Y to enable FPGA Region driver for Intel FPGA Management
>           Engine.
>
> +config INTEL_FPGA_AFU
> +       tristate "Intel FPGA AFU Driver"
> +       depends on INTEL_FPGA_PCI
> +       help
> +         This is the driver for FPGA Accelerated Function Unit (AFU) which
> +         implements AFU and Port management features. A User AFU connects
> +         to the FPGA infrastructure via a Port. There may be more than 1
> +         Port/AFU per Intel FPGA.
> +
>  endif
>
>  endif # FPGA
> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> index fd0a71f..ce08833 100644
> --- a/drivers/fpga/Makefile
> +++ b/drivers/fpga/Makefile
> @@ -34,6 +34,8 @@ obj-$(CONFIG_INTEL_FPGA_FME)          += intel-fpga-fme.o
>  obj-$(CONFIG_INTEL_FPGA_FME_MGR)       += intel-fpga-fme-mgr.o
>  obj-$(CONFIG_INTEL_FPGA_FME_BRIDGE)    += intel-fpga-fme-br.o
>  obj-$(CONFIG_INTEL_FPGA_FME_REGION)    += intel-fpga-fme-region.o
> +obj-$(CONFIG_INTEL_FPGA_AFU)           += intel-fpga-afu.o
>
>  intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
>  intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
> +intel-fpga-afu-objs := intel-afu-main.o
> diff --git a/drivers/fpga/intel-afu-main.c b/drivers/fpga/intel-afu-main.c
> new file mode 100644
> index 0000000..96d0367
> --- /dev/null
> +++ b/drivers/fpga/intel-afu-main.c
> @@ -0,0 +1,159 @@
> +/*
> + * Driver for Intel FPGA Accelerated Function Unit (AFU)
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Wu Hao <hao.wu@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *   Joseph Grecco <joe.grecco@intel.com>
> + *   Enno Luebbers <enno.luebbers@intel.com>
> + *   Tim Whisonant <tim.whisonant@intel.com>
> + *   Ananda Ravuri <ananda.ravuri@intel.com>
> + *   Henry Mitchel <henry.mitchel@intel.com>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2. See
> + * the COPYING file in the top-level directory.

Or just: SPDX-License-Identifier: GPL-2.0
> + */
> +
> +#include <linux/kernel.h>
> +#include <linux/module.h>
> +
> +#include "intel-feature-dev.h"
> +
> +static int port_hdr_init(struct platform_device *pdev, struct feature *feature)
> +{
> +       dev_dbg(&pdev->dev, "PORT HDR Init.\n");
> +
> +       return 0;
> +}
> +
> +static void port_hdr_uinit(struct platform_device *pdev,
> +                                       struct feature *feature)
> +{
> +       dev_dbg(&pdev->dev, "PORT HDR UInit.\n");
> +}
> +
> +struct feature_ops port_hdr_ops = {
> +       .init = port_hdr_init,
> +       .uinit = port_hdr_uinit,
> +};
> +
> +static struct feature_driver port_feature_drvs[] = {
> +       {
> +               .name = PORT_FEATURE_HEADER,
> +               .ops = &port_hdr_ops,
> +       },
> +       {
> +               .ops = NULL,
> +       }
> +};
> +
> +static int afu_open(struct inode *inode, struct file *filp)
> +{
> +       struct platform_device *fdev = fpga_inode_to_feature_dev(inode);
> +       struct feature_platform_data *pdata;
> +       int ret;
> +
> +       pdata = dev_get_platdata(&fdev->dev);
> +       if (WARN_ON(!pdata))
> +               return -ENODEV;
> +
> +       ret = feature_dev_use_begin(pdata);
> +       if (ret)
> +               return ret;
> +
> +       dev_dbg(&fdev->dev, "Device File Open\n");
> +       filp->private_data = fdev;
> +
> +       return 0;
> +}
> +
> +static int afu_release(struct inode *inode, struct file *filp)
> +{
> +       struct platform_device *pdev = filp->private_data;
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +
> +       dev_dbg(&pdev->dev, "Device File Release\n");
> +
> +       feature_dev_use_end(pdata);
> +       return 0;
> +}
> +
> +static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
> +{
> +       struct platform_device *pdev = filp->private_data;
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct feature *f;
> +       long ret;
> +
> +       dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
> +
> +       switch (cmd) {
> +       default:
> +               /*
> +                * Let sub-feature's ioctl function to handle the cmd
> +                * Sub-feature's ioctl returns -ENODEV when cmd is not
> +                * handled in this sub feature, and returns 0 and other
> +                * error code if cmd is handled.
> +                */
> +               fpga_dev_for_each_feature(pdata, f)
> +                       if (f->ops && f->ops->ioctl) {
> +                               ret = f->ops->ioctl(pdev, f, cmd, arg);
> +                               if (ret == -ENODEV)
> +                                       continue;
> +                               else
> +                                       return ret;
> +                       }
> +       }
> +
> +       return -EINVAL;
> +}
> +
> +static const struct file_operations afu_fops = {
> +       .owner = THIS_MODULE,
> +       .open = afu_open,
> +       .release = afu_release,
> +       .unlocked_ioctl = afu_ioctl,
> +};
> +
> +static int afu_probe(struct platform_device *pdev)
> +{
> +       int ret;
> +
> +       dev_dbg(&pdev->dev, "%s\n", __func__);
> +
> +       ret = fpga_dev_feature_init(pdev, port_feature_drvs);
> +       if (ret)
> +               return ret;
> +
> +       ret = fpga_register_dev_ops(pdev, &afu_fops, THIS_MODULE);
> +       if (ret)
> +               fpga_dev_feature_uinit(pdev);
> +
> +       return ret;
> +}
> +
> +static int afu_remove(struct platform_device *pdev)
> +{
> +       dev_dbg(&pdev->dev, "%s\n", __func__);
> +
> +       fpga_dev_feature_uinit(pdev);
> +       fpga_unregister_dev_ops(pdev);
> +       return 0;
> +}
> +
> +static struct platform_driver afu_driver = {
> +       .driver = {
> +               .name    = "intel-fpga-port",
> +       },
> +       .probe   = afu_probe,
> +       .remove  = afu_remove,
> +};
> +
> +module_platform_driver(afu_driver);
> +
> +MODULE_DESCRIPTION("Intel FPGA Accelerated Function Unit driver");
> +MODULE_AUTHOR("Intel Corporation");
> +MODULE_LICENSE("GPL v2");
> +MODULE_ALIAS("platform:intel-fpga-port");
> --
> 1.8.3.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Thanks,

Moritz

Re: [PATCH v2 13/22] fpga: intel: fme: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support

[Alan Tull]

On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu@intel.com> wrote:
> FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which
> need to be supported by all feature devices drivers including FME and AFU.
> Userspace application can use these ioctl interfaces to get the API info
> and check if specific extension is supported or not in current driver.
>
> This patch implements above 2 ioctls in Intel FPGA Management Engine (FME)
> driver.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
Acked-by: Alan Tull <atull@kernel.org>
> ---
> v2: switched to GPLv2 license.
> ---
>  Documentation/ioctl/ioctl-number.txt |  1 +
>  drivers/fpga/intel-fme-main.c        | 12 +++++++++
>  include/uapi/linux/intel-fpga.h      | 50 ++++++++++++++++++++++++++++++++++++
>  3 files changed, 63 insertions(+)
>  create mode 100644 include/uapi/linux/intel-fpga.h
>
> diff --git a/Documentation/ioctl/ioctl-number.txt b/Documentation/ioctl/ioctl-number.txt
> index 1e9fcb4..a0d8be5 100644
> --- a/Documentation/ioctl/ioctl-number.txt
> +++ b/Documentation/ioctl/ioctl-number.txt
> @@ -324,6 +324,7 @@ Code  Seq#(hex)     Include File            Comments
>  0xB3   00      linux/mmc/ioctl.h
>  0xB4   00-0F   linux/gpio.h            <mailto:linux-gpio@vger.kernel.org>
>  0xB5   00-0F   uapi/linux/rpmsg.h      <mailto:linux-remoteproc@vger.kernel.org>
> +0xB6   all     linux/intel-fpga.h
>  0xC0   00-0F   linux/usb/iowarrior.h
>  0xCA   00-0F   uapi/misc/cxl.h
>  0xCA   80-8F   uapi/scsi/cxlflash_ioctl.h
> diff --git a/drivers/fpga/intel-fme-main.c b/drivers/fpga/intel-fme-main.c
> index dfbb17c..9fd0baf 100644
> --- a/drivers/fpga/intel-fme-main.c
> +++ b/drivers/fpga/intel-fme-main.c
> @@ -18,6 +18,7 @@
>
>  #include <linux/kernel.h>
>  #include <linux/module.h>
> +#include <linux/intel-fpga.h>
>
>  #include "intel-feature-dev.h"
>
> @@ -102,6 +103,13 @@ struct feature_ops fme_hdr_ops = {
>         },
>  };
>
> +static long fme_ioctl_check_extension(struct feature_platform_data *pdata,
> +                                    unsigned long arg)
> +{
> +       /* No extension support for now */
> +       return 0;
> +}
> +
>  static int fme_open(struct inode *inode, struct file *filp)
>  {
>         struct platform_device *fdev = fpga_inode_to_feature_dev(inode);
> @@ -140,6 +148,10 @@ static long fme_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
>         dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
>
>         switch (cmd) {
> +       case FPGA_GET_API_VERSION:
> +               return FPGA_API_VERSION;
> +       case FPGA_CHECK_EXTENSION:
> +               return fme_ioctl_check_extension(pdata, arg);
>         default:
>                 /*
>                  * Let sub-feature's ioctl function to handle the cmd
> diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
> new file mode 100644
> index 0000000..e340ef9
> --- /dev/null
> +++ b/include/uapi/linux/intel-fpga.h
> @@ -0,0 +1,50 @@
> +/*
> + * Header File for Intel FPGA User API
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <luwei.kang@intel.com>
> + *   Zhang Yi <yi.z.zhang@intel.com>
> + *   Wu Hao <hao.wu@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2. See
> + * the COPYING file in the top-level directory.
> + */
> +
> +#ifndef _UAPI_LINUX_INTEL_FPGA_H
> +#define _UAPI_LINUX_INTEL_FPGA_H
> +
> +#define FPGA_API_VERSION 0
> +
> +/*
> + * The IOCTL interface for Intel FPGA is designed for extensibility by
> + * embedding the structure length (argsz) and flags into structures passed
> + * between kernel and userspace. This design referenced the VFIO IOCTL
> + * interface (include/uapi/linux/vfio.h).
> + */
> +
> +#define FPGA_MAGIC 0xB6
> +
> +#define FPGA_BASE 0
> +
> +/**
> + * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
> + *
> + * Report the version of the driver API.
> + * Return: Driver API Version.
> + */
> +
> +#define FPGA_GET_API_VERSION   _IO(FPGA_MAGIC, FPGA_BASE + 0)
> +
> +/**
> + * FPGA_CHECK_EXTENSION - _IO(FPGA_MAGIC, FPGA_BASE + 1)
> + *
> + * Check whether an extension is supported.
> + * Return: 0 if not supported, otherwise the extension is supported.
> + */
> +
> +#define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)
> +
> +#endif /* _UAPI_INTEL_FPGA_H */
> --
> 1.8.3.1
>

Re: [PATCH v2 20/22] fpga: intel: afu add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support

[Moritz Fischer]

Hi,

On Sun, Jun 25, 2017 at 6:52 PM, Wu Hao <hao.wu@intel.com> wrote:
> FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which
> need to be supported by all feature devices drivers including FME and AFU.
> This patch implements above 2 ioctls in Intel FPGA Accelerated Function
> Unit (AFU) driver.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ---
> v2: rebased
> ---
>  drivers/fpga/intel-afu-main.c | 11 +++++++++++
>  1 file changed, 11 insertions(+)
>
> diff --git a/drivers/fpga/intel-afu-main.c b/drivers/fpga/intel-afu-main.c
> index 2a17cde..22f77f2 100644
> --- a/drivers/fpga/intel-afu-main.c
> +++ b/drivers/fpga/intel-afu-main.c
> @@ -122,6 +122,13 @@ static int afu_release(struct inode *inode, struct file *filp)
>         return 0;
>  }
>
> +static long afu_ioctl_check_extension(struct feature_platform_data *pdata,
> +                                    unsigned long arg)
> +{
> +       /* No extension support for now */

Do you maybe make that an -ENOTSUPP or -EINVAL then?
> +       return 0;
> +}
> +
>  static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
>  {
>         struct platform_device *pdev = filp->private_data;
> @@ -132,6 +139,10 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
>         dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
>
>         switch (cmd) {
> +       case FPGA_GET_API_VERSION:
> +               return FPGA_API_VERSION;
> +       case FPGA_CHECK_EXTENSION:
> +               return afu_ioctl_check_extension(pdata, arg);
>         default:
>                 /*
>                  * Let sub-feature's ioctl function to handle the cmd
> --
> 1.8.3.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Thanks,

Moritz

Re: [PATCH v2 16/22] fpga: intel: add fpga bridge platform driver for FME

[Alan Tull]

On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> This patch adds fpga bridge platform driver for Intel FPGA Management
> Engine. It implements the enable_set call back for fpga bridge.
>
> Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Acked-by: Alan Tull <atull-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> ---
>  drivers/fpga/Kconfig             |  7 ++++
>  drivers/fpga/Makefile            |  1 +
>  drivers/fpga/intel-fpga-fme-br.c | 77 ++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 85 insertions(+)
>  create mode 100644 drivers/fpga/intel-fpga-fme-br.c
>
> diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> index 6f2f623..05e2a8e 100644
> --- a/drivers/fpga/Kconfig
> +++ b/drivers/fpga/Kconfig
> @@ -158,6 +158,13 @@ config INTEL_FPGA_FME_MGR
>           Say Y to enable FPGA Manager driver for Intel FPGA Management
>           Engine.
>
> +config INTEL_FPGA_FME_BRIDGE
> +       tristate "Intel FPGA FME Bridge Driver"
> +       depends on INTEL_FPGA_FME && FPGA_BRIDGE
> +       help
> +         Say Y to enable FPGA Bridge driver for Intel FPGA Management
> +         Engine.
> +
>  endif
>
>  endif # FPGA
> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> index d1d588b..131c105 100644
> --- a/drivers/fpga/Makefile
> +++ b/drivers/fpga/Makefile
> @@ -32,6 +32,7 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
>  obj-$(CONFIG_INTEL_FPGA_PCI)           += intel-fpga-pci.o
>  obj-$(CONFIG_INTEL_FPGA_FME)           += intel-fpga-fme.o
>  obj-$(CONFIG_INTEL_FPGA_FME_MGR)       += intel-fpga-fme-mgr.o
> +obj-$(CONFIG_INTEL_FPGA_FME_BRIDGE)    += intel-fpga-fme-br.o
>
>  intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
>  intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
> diff --git a/drivers/fpga/intel-fpga-fme-br.c b/drivers/fpga/intel-fpga-fme-br.c
> new file mode 100644
> index 0000000..9b922d2
> --- /dev/null
> +++ b/drivers/fpga/intel-fpga-fme-br.c
> @@ -0,0 +1,77 @@
> +/*
> + * FPGA Bridge Driver for Intel FPGA Management Engine (FME)
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *   Joseph Grecco <joe.grecco-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *   Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *   Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *   Ananda Ravuri <ananda.ravuri-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *   Henry Mitchel <henry.mitchel-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2. See
> + * the COPYING file in the top-level directory.
> + */
> +
> +#include <linux/module.h>
> +#include <linux/fpga/fpga-bridge.h>
> +
> +#include "intel-feature-dev.h"
> +#include "intel-fme.h"
> +
> +static int fme_bridge_enable_set(struct fpga_bridge *bridge, bool enable)
> +{
> +       struct fme_br_pdata *pdata = bridge->priv;
> +       int ret = 0;
> +
> +       if (enable)
> +               fpga_port_enable(pdata->port);
> +       else
> +               ret = fpga_port_disable(pdata->port);
> +
> +       return ret;
> +}
> +
> +static const struct fpga_bridge_ops fme_bridge_ops = {
> +       .enable_set = fme_bridge_enable_set,
> +};
> +
> +static int fme_br_probe(struct platform_device *pdev)
> +{
> +       struct device *dev = &pdev->dev;
> +       struct fme_br_pdata *pdata = dev_get_platdata(dev);
> +       int ret;
> +
> +       ret = fpga_bridge_register(dev, "Intel FPGA FME Bridge",
> +                                  &fme_bridge_ops, pdata);
> +       if (ret)
> +               dev_err(dev, "unable to register FPGA Bridge\n");
> +
> +       dev_dbg(dev, "Intel FME FPGA Bridge probed\n");
> +
> +       return ret;
> +}
> +
> +static int fme_br_remove(struct platform_device *pdev)
> +{
> +       fpga_bridge_unregister(&pdev->dev);
> +
> +       return 0;
> +}
> +
> +static struct platform_driver fme_br_driver = {
> +       .driver = {
> +               .name    = INTEL_FPGA_FME_BRIDGE,
> +       },
> +       .probe   = fme_br_probe,
> +       .remove  = fme_br_remove,
> +};
> +
> +module_platform_driver(fme_br_driver);
> +
> +MODULE_DESCRIPTION("FPGA Bridge for Intel FPGA Management Engine");
> +MODULE_AUTHOR("Intel Corporation");
> +MODULE_LICENSE("GPL v2");
> +MODULE_ALIAS("platform:intel-fpga-fme-bridge");
> --
> 1.8.3.1
>

Re: [PATCH v2 16/22] fpga: intel: add fpga bridge platform driver for FME

[Moritz Fischer]

Hi Wu,

looks good. Minor nits inline.

On Sun, Jun 25, 2017 at 6:52 PM, Wu Hao <hao.wu@intel.com> wrote:
> This patch adds fpga bridge platform driver for Intel FPGA Management
> Engine. It implements the enable_set call back for fpga bridge.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>

Reviewed-by: Moritz Fischer <mdf@kernel.org> (With fixes below)
> ---
>  drivers/fpga/Kconfig             |  7 ++++
>  drivers/fpga/Makefile            |  1 +
>  drivers/fpga/intel-fpga-fme-br.c | 77 ++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 85 insertions(+)
>  create mode 100644 drivers/fpga/intel-fpga-fme-br.c
>
> diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> index 6f2f623..05e2a8e 100644
> --- a/drivers/fpga/Kconfig
> +++ b/drivers/fpga/Kconfig
> @@ -158,6 +158,13 @@ config INTEL_FPGA_FME_MGR
>           Say Y to enable FPGA Manager driver for Intel FPGA Management
>           Engine.
>
> +config INTEL_FPGA_FME_BRIDGE
> +       tristate "Intel FPGA FME Bridge Driver"
> +       depends on INTEL_FPGA_FME && FPGA_BRIDGE
> +       help
> +         Say Y to enable FPGA Bridge driver for Intel FPGA Management
> +         Engine.
> +
>  endif
>
>  endif # FPGA
> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> index d1d588b..131c105 100644
> --- a/drivers/fpga/Makefile
> +++ b/drivers/fpga/Makefile
> @@ -32,6 +32,7 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
>  obj-$(CONFIG_INTEL_FPGA_PCI)           += intel-fpga-pci.o
>  obj-$(CONFIG_INTEL_FPGA_FME)           += intel-fpga-fme.o
>  obj-$(CONFIG_INTEL_FPGA_FME_MGR)       += intel-fpga-fme-mgr.o
> +obj-$(CONFIG_INTEL_FPGA_FME_BRIDGE)    += intel-fpga-fme-br.o
>
>  intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
>  intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
> diff --git a/drivers/fpga/intel-fpga-fme-br.c b/drivers/fpga/intel-fpga-fme-br.c
> new file mode 100644
> index 0000000..9b922d2
> --- /dev/null
> +++ b/drivers/fpga/intel-fpga-fme-br.c
> @@ -0,0 +1,77 @@
> +/*
> + * FPGA Bridge Driver for Intel FPGA Management Engine (FME)
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Wu Hao <hao.wu@intel.com>
> + *   Joseph Grecco <joe.grecco@intel.com>
> + *   Enno Luebbers <enno.luebbers@intel.com>
> + *   Tim Whisonant <tim.whisonant@intel.com>
> + *   Ananda Ravuri <ananda.ravuri@intel.com>
> + *   Henry Mitchel <henry.mitchel@intel.com>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2. See
> + * the COPYING file in the top-level directory.

Could replace that with SPDX-Licence-Identifier: GPL-2.0
> + */
> +
> +#include <linux/module.h>
> +#include <linux/fpga/fpga-bridge.h>
> +
> +#include "intel-feature-dev.h"
> +#include "intel-fme.h"
> +
> +static int fme_bridge_enable_set(struct fpga_bridge *bridge, bool enable)
> +{
> +       struct fme_br_pdata *pdata = bridge->priv;
> +       int ret = 0;
> +
> +       if (enable)
> +               fpga_port_enable(pdata->port);
> +       else
> +               ret = fpga_port_disable(pdata->port);
> +
> +       return ret;
> +}
> +
> +static const struct fpga_bridge_ops fme_bridge_ops = {
> +       .enable_set = fme_bridge_enable_set,
> +};
> +
> +static int fme_br_probe(struct platform_device *pdev)
> +{
> +       struct device *dev = &pdev->dev;
> +       struct fme_br_pdata *pdata = dev_get_platdata(dev);
> +       int ret;
> +
> +       ret = fpga_bridge_register(dev, "Intel FPGA FME Bridge",
> +                                  &fme_bridge_ops, pdata);
> +       if (ret)
> +               dev_err(dev, "unable to register FPGA Bridge\n");
> +
> +       dev_dbg(dev, "Intel FME FPGA Bridge probed\n");

Mixed signals here. Did it work or did it not. Either return early,
in case of error. Or get rid of dev_dbg().
> +
> +       return ret;
> +}
> +
> +static int fme_br_remove(struct platform_device *pdev)
> +{
> +       fpga_bridge_unregister(&pdev->dev);
> +
> +       return 0;
> +}
> +
> +static struct platform_driver fme_br_driver = {
> +       .driver = {
> +               .name    = INTEL_FPGA_FME_BRIDGE,
> +       },
> +       .probe   = fme_br_probe,
> +       .remove  = fme_br_remove,
> +};
> +
> +module_platform_driver(fme_br_driver);
> +
> +MODULE_DESCRIPTION("FPGA Bridge for Intel FPGA Management Engine");
> +MODULE_AUTHOR("Intel Corporation");
> +MODULE_LICENSE("GPL v2");
> +MODULE_ALIAS("platform:intel-fpga-fme-bridge");
> --
> 1.8.3.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Thanks,

Moritz

Re: [PATCH v2 09/22] fpga: intel: pcie: adds fpga_for_each_port callback for fme device

[Alan Tull]

On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu@intel.com> wrote:

Hi Hao,

> For FPGA Management Engine (FME), it requires fpga_for_each_port callback
> for actions on ports, so export this function from PCIe driver by adding
> the callback to the platform data.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ---
> v2: rebased
> ---
>  drivers/fpga/intel-feature-dev.h |  9 +++++++++
>  drivers/fpga/intel-pcie.c        | 24 ++++++++++++++++++++++++
>  2 files changed, 33 insertions(+)
>
> diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
> index d64a51e..06b3fb6 100644
> --- a/drivers/fpga/intel-feature-dev.h
> +++ b/drivers/fpga/intel-feature-dev.h
> @@ -235,6 +235,9 @@ struct feature_platform_data {
>         struct platform_device *dev;
>         unsigned int disable_count;     /* count for port disable */
>
> +       struct platform_device *(*fpga_for_each_port)(struct platform_device *,
> +                       void *, int (*match)(struct platform_device *, void *));
> +
>         int num;                        /* number of features */
>         struct feature features[0];
>  };
> @@ -354,4 +357,10 @@ static inline int fpga_port_reset(struct platform_device *pdev)
>
>         return pdata->features[index].ioaddr;
>  }
> +
> +static inline struct device *
> +fpga_feature_dev_to_pcidev(struct platform_device *dev)
> +{
> +       return dev->dev.parent->parent;
> +}
>  #endif
> diff --git a/drivers/fpga/intel-pcie.c b/drivers/fpga/intel-pcie.c
> index 54c0e3a..86ea5c7 100644
> --- a/drivers/fpga/intel-pcie.c
> +++ b/drivers/fpga/intel-pcie.c
> @@ -209,6 +209,27 @@ static int parse_switch_to(struct build_feature_devs_info *binfo, int bar)
>         return parse_start_from(binfo, bar);
>  }
>
> +static struct platform_device *fpga_for_each_port(struct platform_device *pdev,

Regarding the name of this function, isn't this more of a 'find'
function like class_find_device() than a 'for_each' function?

Also please document that the caller will need to put_device().

Thanks,
Alan

> +                    void *data, int (*match)(struct platform_device *, void *))
> +{
> +       struct device *pci_dev = fpga_feature_dev_to_pcidev(pdev);
> +       struct cci_drvdata *drvdata = dev_get_drvdata(pci_dev);
> +       struct feature_platform_data *pdata;
> +       struct platform_device *port_dev;
> +
> +       mutex_lock(&drvdata->lock);
> +       list_for_each_entry(pdata, &drvdata->port_dev_list, node) {
> +               port_dev = pdata->dev;
> +
> +               if (match(port_dev, data) && get_device(&port_dev->dev))
> +                       goto exit;
> +       }
> +       port_dev = NULL;
> +exit:
> +       mutex_unlock(&drvdata->lock);
> +       return port_dev;
> +}
> +
>  static struct build_feature_devs_info *
>  build_info_alloc_and_init(struct pci_dev *pdev)
>  {
> @@ -310,6 +331,9 @@ static int build_info_commit_dev(struct build_feature_devs_info *binfo)
>         if (!pdata)
>                 return -ENOMEM;
>
> +       if (type == FME_ID)
> +               pdata->fpga_for_each_port = fpga_for_each_port;
> +
>         /*
>          * the count should be initialized to 0 to make sure
>          *__fpga_port_enable() following __fpga_port_disable()
> --
> 1.8.3.1
>

Re: [PATCH v2 19/22] fpga: intel: afu: add header sub feature support

[Alan Tull]

On Wed, Aug 16, 2017 at 12:11 AM, Wu, Hao <hao.wu@intel.com> wrote:
>>  On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu@intel.com> wrote:
>>
>> Hi Hao,
>>
>> > The header register set is always present for the Port/AFU, it is mainly
>> > for capability, control and status of the ports that AFU connected to.
>>
>> So just to be clear, the reset function is acting on the Port, not the
>> AFU, right?
>
> Hi Alan
>
> AFU will be reset as well. Once port reset is issued, a compliant HW
> AFU will reset all its state and stop sending requests, and HW will set
> port reset ack bit when all outstanding requests initiated have been
> drained.
>
> Will add some notes here.
>
>>
>> >
>> > This patch implements header sub feature support.
>>
>> Please add a brief reminder here what the 'header' is.  It's defined
>> in patch 7 as being part of the feature list, but hardly mentioned
>> when I grep intel-fpga.txt.
>
> Sure, will adds some notes here.
>
> Actually the header sub feature means the registers belong to the
> feature device (e.g port and FME), not any sub features (e.g PR,
> Power management).
>
>>
>> > Below user interfaces
>> > are created by this patch.
>> >
>> > Sysfs interface:
>> > * /sys/class/fpga/<fpga.x>/<intel-fpga-port.x>/id
>> >   Read-only. Port ID.
>> >
>> > Ioctl interface:
>> > * FPGA_PORT_RESET
>> >   Reset the FPGA AFU Port.
>> >
>> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
>> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
>> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
>> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
>> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> > Signed-off-by: Wu Hao <hao.wu@intel.com>
>> > ---
>> > v2: add sysfs documentation.
>> > ---
>> >  .../ABI/testing/sysfs-platform-intel-fpga-afu      |  7 ++++
>> >  drivers/fpga/intel-afu-main.c                      | 44 +++++++++++++++++++++-
>> >  include/uapi/linux/intel-fpga.h                    | 14 +++++++
>> >  3 files changed, 64 insertions(+), 1 deletion(-)
>> >  create mode 100644 Documentation/ABI/testing/sysfs-platform-intel-fpga-
>> afu
>> >
>> > diff --git a/Documentation/ABI/testing/sysfs-platform-intel-fpga-afu
>> b/Documentation/ABI/testing/sysfs-platform-intel-fpga-afu
>> > new file mode 100644
>> > index 0000000..8ad22c9
>> > --- /dev/null
>> > +++ b/Documentation/ABI/testing/sysfs-platform-intel-fpga-afu
>> > @@ -0,0 +1,7 @@
>> > +What:          /sys/bus/platform/devices/intel-fpga-port.0/id
>> > +Date:          June 2017
>> > +KernelVersion:  4.12
>> > +Contact:       Wu Hao <hao.wu@intel.com>
>> > +Description:   Read-only. It returns id of this port. One Intel FPGA device
>> > +               may have more than one port. Userspace could use this id to
>> > +               distinguish different ports under same FPGA device.
>> > diff --git a/drivers/fpga/intel-afu-main.c b/drivers/fpga/intel-afu-main.c
>> > index 96d0367..2a17cde 100644
>> > --- a/drivers/fpga/intel-afu-main.c
>> > +++ b/drivers/fpga/intel-afu-main.c
>> > @@ -18,25 +18,66 @@
>> >
>> >  #include <linux/kernel.h>
>> >  #include <linux/module.h>
>> > +#include <linux/intel-fpga.h>
>> >
>> >  #include "intel-feature-dev.h"
>> >
>> > +static ssize_t
>> > +id_show(struct device *dev, struct device_attribute *attr, char *buf)
>> > +{
>> > +       int id = fpga_port_id(to_platform_device(dev));
>> > +
>> > +       return scnprintf(buf, PAGE_SIZE, "%d\n", id);
>> > +}
>> > +static DEVICE_ATTR_RO(id);
>> > +
>> > +static const struct attribute *port_hdr_attrs[] = {
>> > +       &dev_attr_id.attr,
>> > +       NULL,
>> > +};
>> > +
>> >  static int port_hdr_init(struct platform_device *pdev, struct feature *feature)
>> >  {
>> >         dev_dbg(&pdev->dev, "PORT HDR Init.\n");
>> >
>> > -       return 0;
>> > +       fpga_port_reset(pdev);
>>
>> So the port will be reset here, which happens during fme_probe().
>> IIUC the PR region is empty then, there is just the static region,
>> right?
>
> port_hdr_init is invoked during afu_probe() function. The fpga_port_reset
> only resets the AFU's state and not empty the PR region. User doesn't need
> to program it again after port reset.
>
> The purpose of this reset in port_hdr_init function, is to make sure that we
> could have a clean start whenever the port driver module is loaded. And
> similar as the one added in afu release.
>
>>
>> > +
>> > +       return sysfs_create_files(&pdev->dev.kobj, port_hdr_attrs);
>>
>> Greg wrote an article that there could be a race condition caused by
>> creating sysfs files this late [1] and I see sysfs_create_files() used
>> very sparingly in the kernel.  I'm thinking that fpga-bridge should
>> provide a place to create sysfs files earlier by adding an
>> attribute_group to fpga_bridge_ops (same for fpga-mgr and fpga-region)
>> and then fpga_bridge_register could do bridge->dev.groups =
>> br_ops->groups.  I'll put a patch for that out soon.
>>
>
> Hm... I understand there could be a race condition if creates sysfs files late.
> Actually the reasons I prefer to have each sub feature to create its own sysfs
> files are, 1) if any sub feature is not present, then related init function won't
> be invoked and related sysfs files won't be created at all. Then end user could
> know which sub features are present by checking these sysfs nodes easily.
> 2) Another point of view is about extension of each sub feature, for example,
> Some new registers introduced when sub feature's revision > n, so each sub
> feature's init function could check the sub feature's revision to decide which
> sysfs interfaces are needed. I think this should be a flexible way for extension.
>
>> >  }
>> >
>> >  static void port_hdr_uinit(struct platform_device *pdev,
>> >                                         struct feature *feature)
>> >  {
>> >         dev_dbg(&pdev->dev, "PORT HDR UInit.\n");
>> > +
>> > +       sysfs_remove_files(&pdev->dev.kobj, port_hdr_attrs);
>> > +}
>> > +
>> > +static long
>> > +port_hdr_ioctl(struct platform_device *pdev, struct feature *feature,
>> > +                                       unsigned int cmd, unsigned long arg)
>> > +{
>> > +       long ret;
>> > +
>> > +       switch (cmd) {
>> > +       case FPGA_PORT_RESET:
>> > +               if (!arg)
>> > +                       ret = fpga_port_reset(pdev);
>>
>> fpga_port_reset() disables and reenables traffic on the port.  Is
>> there ever a time when that would be unsafe to do?  Like while DMA is
>> happening?  When I see a function called 'reset' exposed to userspace,
>> I become concerned that hitting that reset at the wrong time could
>> cause problems.  We've discussed this some, but could you please
>> remind me when userspace would need to reset the port?  Please add
>> documentation of what the intended use of this ioctl would be, when it
>> is valid to request a reset from userspace and when userspace should
>> never do that.
>
> Sure, I will add some more description in uapi header file on this IOCTL.
>
> Per my understanding, this API could be invoked at any time, even there
> is some DMA operation or PR operation at the same time. It should not
> cause any system level problem, but may cause functional failures ( e.g
> DMA or PR operation failure) and it should be recoverable.

OK that was my biggest concern here with the reset.

>
>>
>> The pcie code, the AFU file interface, and the bridge code all do port
>> reset.  This code is spread out over a few patches, but I'll comment
>> here for now.  I'm trying to keep track of everything that resets the
>> port.  The port gets reset in afu_probe, fme_probe, the AFU file
>> release, and intel-pcie.c after parsing the features.  Also the port
>> is esentially reset after doing reprogramming, since that involves a
>> bridge disable/enable.   In the v1 review, the issue raised that the
>> port functionality could be an expansion of fpga-bridge.  If reset
>> were added to the fpga_bridge_ops and a fpga_bridge_reset API added to
>> fpga-bridge, then anything in the kernel that owns the bridge could
>> reset it.  That is of course assuming that this code doesn't need to
>> reset the port before the fpga-bridge is created.
>
> Actually in the pcie code, it needs to enable fpga port (put it out of reset,
> as port is in reset by default), otherwise the AFU MMIO region is not
> accessible. It's only a one time action.
>
> For the AFU interface, it gives application a way to reset the AFU if
> anything goes wrong or application wants to make AFU back to
> default state.
>
> For the bridge code, it just makes sure port is in reset during PR, which
> is requested by the PR flow of Intel FPGA device.
>
> As you know, the fpga-bridge is created under FME module now, and
> port/AFU is another module which could be turned into a VF and assigned
> to a virtual machine, even we added a reset ops to fpga-bridge, we still
> need an interface to do port reset as fpga-bridges (and FME) are always
> in PF in host.  :)

OK

Alan

>
> Thanks
> Hao
>
>>
>> Thanks,
>> Alan Tull
>>
>> [1] http://www.kroah.com/log/blog/2013/06/26/how-to-create-a-sysfs-file-
>> correctly/
>>
>> > +               else
>> > +                       ret = -EINVAL;
>> > +               break;
>> > +       default:
>> > +               dev_dbg(&pdev->dev, "%x cmd not handled", cmd);
>> > +               ret = -ENODEV;
>> > +       }
>> > +
>> > +       return ret;
>> >  }
>> >
>> >  struct feature_ops port_hdr_ops = {
>> >         .init = port_hdr_init,
>> >         .uinit = port_hdr_uinit,
>> > +       .ioctl = port_hdr_ioctl,
>> >  };
>> >
>> >  static struct feature_driver port_feature_drvs[] = {
>> > @@ -76,6 +117,7 @@ static int afu_release(struct inode *inode, struct file
>> *filp)
>> >
>> >         dev_dbg(&pdev->dev, "Device File Release\n");
>> >
>> > +       fpga_port_reset(pdev);
>> >         feature_dev_use_end(pdata);
>> >         return 0;
>> >  }
>> > diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
>> > index be295ae..be5f813 100644
>> > --- a/include/uapi/linux/intel-fpga.h
>> > +++ b/include/uapi/linux/intel-fpga.h
>> > @@ -30,8 +30,11 @@
>> >  #define FPGA_MAGIC 0xB6
>> >
>> >  #define FPGA_BASE 0
>> > +#define PORT_BASE 0x40
>> >  #define FME_BASE 0x80
>> >
>> > +/* Common IOCTLs for both FME and AFU file descriptor */
>> > +
>> >  /**
>> >   * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
>> >   *
>> > @@ -50,6 +53,17 @@
>> >
>> >  #define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)
>> >
>> > +/* IOCTLs for AFU file descriptor */
>> > +
>> > +/**
>> > + * FPGA_PORT_RESET - _IO(FPGA_MAGIC, PORT_BASE + 0)
>> > + *
>> > + * Reset the FPGA AFU Port. No parameters are supported.
>> > + * Return: 0 on success, -errno of failure
>> > + */
>> > +
>> > +#define FPGA_PORT_RESET                _IO(FPGA_MAGIC, PORT_BASE + 0)
>> > +
>> >  /* IOCTLs for FME file descriptor */
>> >
>> >  /**
>> > --
>> > 1.8.3.1
>> >

Re: [PATCH v2 16/22] fpga: intel: add fpga bridge platform driver for FME

[Wu Hao]

On Thu, Aug 17, 2017 at 12:55:37PM -0700, Moritz Fischer wrote:
> Hi Wu,
> 
> looks good. Minor nits inline.

Hi Moritz,

Thanks for your comments. : )

> 
> On Sun, Jun 25, 2017 at 6:52 PM, Wu Hao <hao.wu@intel.com> wrote:
> > This patch adds fpga bridge platform driver for Intel FPGA Management
> > Engine. It implements the enable_set call back for fpga bridge.
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> 
> Reviewed-by: Moritz Fischer <mdf@kernel.org> (With fixes below)
> > ---
> >  drivers/fpga/Kconfig             |  7 ++++
> >  drivers/fpga/Makefile            |  1 +
> >  drivers/fpga/intel-fpga-fme-br.c | 77 ++++++++++++++++++++++++++++++++++++++++
> >  3 files changed, 85 insertions(+)
> >  create mode 100644 drivers/fpga/intel-fpga-fme-br.c
> >
> > diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> > index 6f2f623..05e2a8e 100644
> > --- a/drivers/fpga/Kconfig
> > +++ b/drivers/fpga/Kconfig
> > @@ -158,6 +158,13 @@ config INTEL_FPGA_FME_MGR
> >           Say Y to enable FPGA Manager driver for Intel FPGA Management
> >           Engine.
> >
> > +config INTEL_FPGA_FME_BRIDGE
> > +       tristate "Intel FPGA FME Bridge Driver"
> > +       depends on INTEL_FPGA_FME && FPGA_BRIDGE
> > +       help
> > +         Say Y to enable FPGA Bridge driver for Intel FPGA Management
> > +         Engine.
> > +
> >  endif
> >
> >  endif # FPGA
> > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> > index d1d588b..131c105 100644
> > --- a/drivers/fpga/Makefile
> > +++ b/drivers/fpga/Makefile
> > @@ -32,6 +32,7 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
> >  obj-$(CONFIG_INTEL_FPGA_PCI)           += intel-fpga-pci.o
> >  obj-$(CONFIG_INTEL_FPGA_FME)           += intel-fpga-fme.o
> >  obj-$(CONFIG_INTEL_FPGA_FME_MGR)       += intel-fpga-fme-mgr.o
> > +obj-$(CONFIG_INTEL_FPGA_FME_BRIDGE)    += intel-fpga-fme-br.o
> >
> >  intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
> >  intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
> > diff --git a/drivers/fpga/intel-fpga-fme-br.c b/drivers/fpga/intel-fpga-fme-br.c
> > new file mode 100644
> > index 0000000..9b922d2
> > --- /dev/null
> > +++ b/drivers/fpga/intel-fpga-fme-br.c
> > @@ -0,0 +1,77 @@
> > +/*
> > + * FPGA Bridge Driver for Intel FPGA Management Engine (FME)
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Wu Hao <hao.wu@intel.com>
> > + *   Joseph Grecco <joe.grecco@intel.com>
> > + *   Enno Luebbers <enno.luebbers@intel.com>
> > + *   Tim Whisonant <tim.whisonant@intel.com>
> > + *   Ananda Ravuri <ananda.ravuri@intel.com>
> > + *   Henry Mitchel <henry.mitchel@intel.com>
> > + *
> > + * This work is licensed under the terms of the GNU GPL version 2. See
> > + * the COPYING file in the top-level directory.
> 
> Could replace that with SPDX-Licence-Identifier: GPL-2.0

Sure, I will update the header like this. : )

 * This work is licensed under the terms of the GNU GPL version 2.
 * SPDX-License-Identifier: GPL-2.0
 */

> > + */
> > +
> > +#include <linux/module.h>
> > +#include <linux/fpga/fpga-bridge.h>
> > +
> > +#include "intel-feature-dev.h"
> > +#include "intel-fme.h"
> > +
> > +static int fme_bridge_enable_set(struct fpga_bridge *bridge, bool enable)
> > +{
> > +       struct fme_br_pdata *pdata = bridge->priv;
> > +       int ret = 0;
> > +
> > +       if (enable)
> > +               fpga_port_enable(pdata->port);
> > +       else
> > +               ret = fpga_port_disable(pdata->port);
> > +
> > +       return ret;
> > +}
> > +
> > +static const struct fpga_bridge_ops fme_bridge_ops = {
> > +       .enable_set = fme_bridge_enable_set,
> > +};
> > +
> > +static int fme_br_probe(struct platform_device *pdev)
> > +{
> > +       struct device *dev = &pdev->dev;
> > +       struct fme_br_pdata *pdata = dev_get_platdata(dev);
> > +       int ret;
> > +
> > +       ret = fpga_bridge_register(dev, "Intel FPGA FME Bridge",
> > +                                  &fme_bridge_ops, pdata);
> > +       if (ret)
> > +               dev_err(dev, "unable to register FPGA Bridge\n");
> > +
> > +       dev_dbg(dev, "Intel FME FPGA Bridge probed\n");
> 
> Mixed signals here. Did it work or did it not. Either return early,
> in case of error. Or get rid of dev_dbg().

Agree, will remove this dev_dbg. As fpga_bridge_register will print
some message using dev_info if success.

Thanks
Hao

> > +
> > +       return ret;
> > +}
> > +
> > +static int fme_br_remove(struct platform_device *pdev)
> > +{
> > +       fpga_bridge_unregister(&pdev->dev);
> > +
> > +       return 0;
> > +}
> > +
> > +static struct platform_driver fme_br_driver = {
> > +       .driver = {
> > +               .name    = INTEL_FPGA_FME_BRIDGE,
> > +       },
> > +       .probe   = fme_br_probe,
> > +       .remove  = fme_br_remove,
> > +};
> > +
> > +module_platform_driver(fme_br_driver);
> > +
> > +MODULE_DESCRIPTION("FPGA Bridge for Intel FPGA Management Engine");
> > +MODULE_AUTHOR("Intel Corporation");
> > +MODULE_LICENSE("GPL v2");
> > +MODULE_ALIAS("platform:intel-fpga-fme-bridge");
> > --
> > 1.8.3.1
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> Thanks,
> 
> Moritz

Re: [PATCH v2 20/22] fpga: intel: afu add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support

[Wu Hao]

On Thu, Aug 17, 2017 at 12:12:00PM -0700, Moritz Fischer wrote:
> Hi,
> 
> On Sun, Jun 25, 2017 at 6:52 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> > FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which
> > need to be supported by all feature devices drivers including FME and AFU.
> > This patch implements above 2 ioctls in Intel FPGA Accelerated Function
> > Unit (AFU) driver.
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> > Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > ---
> > v2: rebased
> > ---
> >  drivers/fpga/intel-afu-main.c | 11 +++++++++++
> >  1 file changed, 11 insertions(+)
> >
> > diff --git a/drivers/fpga/intel-afu-main.c b/drivers/fpga/intel-afu-main.c
> > index 2a17cde..22f77f2 100644
> > --- a/drivers/fpga/intel-afu-main.c
> > +++ b/drivers/fpga/intel-afu-main.c
> > @@ -122,6 +122,13 @@ static int afu_release(struct inode *inode, struct file *filp)
> >         return 0;
> >  }
> >
> > +static long afu_ioctl_check_extension(struct feature_platform_data *pdata,
> > +                                    unsigned long arg)
> > +{
> > +       /* No extension support for now */
> 
> Do you maybe make that an -ENOTSUPP or -EINVAL then?

Hi Moritz

User will use this ioctl to check if extension X is supported. It always returns 0
means any extension indicated by the arg, is not supported.

I think we don't have to return an -ENOTSUPP or -EINVAL here, which may confuse
the enduser. : )

Thanks
Hao

> > +       return 0;
> > +}
> > +
> >  static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
> >  {
> >         struct platform_device *pdev = filp->private_data;
> > @@ -132,6 +139,10 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
> >         dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
> >
> >         switch (cmd) {
> > +       case FPGA_GET_API_VERSION:
> > +               return FPGA_API_VERSION;
> > +       case FPGA_CHECK_EXTENSION:
> > +               return afu_ioctl_check_extension(pdata, arg);
> >         default:
> >                 /*
> >                  * Let sub-feature's ioctl function to handle the cmd
> > --
> > 1.8.3.1
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> Thanks,
> 
> Moritz

Re: [PATCH v2 18/22] fpga: intel: add FPGA Accelerated Function Unit driver basic framework

[Wu Hao]

On Thu, Aug 17, 2017 at 02:00:21PM -0500, Alan Tull wrote:
> On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> 
> Hi Hao,
> 
> Just a few minor things below.  Otherwise, it's fine.

Hi Alan

Thanks a lot for the review. :)

> 
> > On Intel FPGA devices, the Accelerated Function Unit (AFU), can be
> > reprogrammed for different functions. It connects to the FPGA
> > infrastructure("blue bistream") via a Port. Port CSRs are implemented
> > separately from the AFU CSRs to provide control and status of the Port.
> > Once valid green bitstream is programmed into the AFU, it allows access
> > to the AFU CSRs in the AFU MMIO space.
> >
> > This patch only implements basic driver framework for AFU, including
> > device file operation framework.
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> > Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Acked-by: Alan Tull <atull-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> 
> > ---
> > v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
> >     switched to GPLv2 license.
> > ---
> >  drivers/fpga/Kconfig          |   9 +++
> >  drivers/fpga/Makefile         |   2 +
> >  drivers/fpga/intel-afu-main.c | 159 ++++++++++++++++++++++++++++++++++++++++++
> >  3 files changed, 170 insertions(+)
> >  create mode 100644 drivers/fpga/intel-afu-main.c
> >
> > diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> > index db1a6ea..4658a13 100644
> > --- a/drivers/fpga/Kconfig
> > +++ b/drivers/fpga/Kconfig
> > @@ -172,6 +172,15 @@ config INTEL_FPGA_FME_REGION
> >           Say Y to enable FPGA Region driver for Intel FPGA Management
> >           Engine.
> >
> > +config INTEL_FPGA_AFU
> > +       tristate "Intel FPGA AFU Driver"
> > +       depends on INTEL_FPGA_PCI
> > +       help
> > +         This is the driver for FPGA Accelerated Function Unit (AFU) which
> > +         implements AFU and Port management features. A User AFU connects
> > +         to the FPGA infrastructure via a Port. There may be more than 1
> > +         Port/AFU per Intel FPGA.
> > +
> >  endif
> >
> >  endif # FPGA
> > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> > index fd0a71f..ce08833 100644
> > --- a/drivers/fpga/Makefile
> > +++ b/drivers/fpga/Makefile
> > @@ -34,6 +34,8 @@ obj-$(CONFIG_INTEL_FPGA_FME)          += intel-fpga-fme.o
> >  obj-$(CONFIG_INTEL_FPGA_FME_MGR)       += intel-fpga-fme-mgr.o
> >  obj-$(CONFIG_INTEL_FPGA_FME_BRIDGE)    += intel-fpga-fme-br.o
> >  obj-$(CONFIG_INTEL_FPGA_FME_REGION)    += intel-fpga-fme-region.o
> > +obj-$(CONFIG_INTEL_FPGA_AFU)           += intel-fpga-afu.o
> >
> >  intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
> >  intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
> > +intel-fpga-afu-objs := intel-afu-main.o
> > diff --git a/drivers/fpga/intel-afu-main.c b/drivers/fpga/intel-afu-main.c
> > new file mode 100644
> > index 0000000..96d0367
> > --- /dev/null
> > +++ b/drivers/fpga/intel-afu-main.c
> > @@ -0,0 +1,159 @@
> > +/*
> > + * Driver for Intel FPGA Accelerated Function Unit (AFU)
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> > + *   Joseph Grecco <joe.grecco-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Ananda Ravuri <ananda.ravuri-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Henry Mitchel <henry.mitchel-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *
> > + * This work is licensed under the terms of the GNU GPL version 2. See
> > + * the COPYING file in the top-level directory.
> > + */
> > +
> > +#include <linux/kernel.h>
> > +#include <linux/module.h>
> > +
> > +#include "intel-feature-dev.h"
> > +
> > +static int port_hdr_init(struct platform_device *pdev, struct feature *feature)
> > +{
> > +       dev_dbg(&pdev->dev, "PORT HDR Init.\n");
> > +
> > +       return 0;
> > +}
> > +
> > +static void port_hdr_uinit(struct platform_device *pdev,
> > +                                       struct feature *feature)
> > +{
> > +       dev_dbg(&pdev->dev, "PORT HDR UInit.\n");
> > +}
> > +
> > +struct feature_ops port_hdr_ops = {
> 
> static const struct?

Will fix this.

> 
> > +       .init = port_hdr_init,
> > +       .uinit = port_hdr_uinit,
> > +};
> > +
> > +static struct feature_driver port_feature_drvs[] = {
> 
> static const struct?

Will fix this.

> 
> > +       {
> > +               .name = PORT_FEATURE_HEADER,
> > +               .ops = &port_hdr_ops,
> > +       },
> > +       {
> > +               .ops = NULL,
> > +       }
> > +};
> > +
> > +static int afu_open(struct inode *inode, struct file *filp)
> > +{
> > +       struct platform_device *fdev = fpga_inode_to_feature_dev(inode);
> > +       struct feature_platform_data *pdata;
> > +       int ret;
> > +
> > +       pdata = dev_get_platdata(&fdev->dev);
> > +       if (WARN_ON(!pdata))
> > +               return -ENODEV;
> > +
> > +       ret = feature_dev_use_begin(pdata);
> > +       if (ret)
> > +               return ret;
> > +
> > +       dev_dbg(&fdev->dev, "Device File Open\n");
> > +       filp->private_data = fdev;
> > +
> > +       return 0;
> > +}
> > +
> > +static int afu_release(struct inode *inode, struct file *filp)
> > +{
> > +       struct platform_device *pdev = filp->private_data;
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +
> > +       dev_dbg(&pdev->dev, "Device File Release\n");
> > +
> > +       feature_dev_use_end(pdata);
> 
> Please skip a line before return.

Will fix this.

> 
> > +       return 0;
> > +}
> > +
> > +static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
> > +{
> > +       struct platform_device *pdev = filp->private_data;
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct feature *f;
> > +       long ret;
> > +
> > +       dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
> > +
> > +       switch (cmd) {
> > +       default:
> > +               /*
> > +                * Let sub-feature's ioctl function to handle the cmd
> > +                * Sub-feature's ioctl returns -ENODEV when cmd is not
> > +                * handled in this sub feature, and returns 0 and other
> > +                * error code if cmd is handled.
> > +                */
> > +               fpga_dev_for_each_feature(pdata, f)
> > +                       if (f->ops && f->ops->ioctl) {
> > +                               ret = f->ops->ioctl(pdev, f, cmd, arg);
> > +                               if (ret == -ENODEV)
> > +                                       continue;
> > +                               else
> > +                                       return ret;
> > +                       }
> > +       }
> > +
> > +       return -EINVAL;
> > +}
> > +
> > +static const struct file_operations afu_fops = {
> > +       .owner = THIS_MODULE,
> > +       .open = afu_open,
> > +       .release = afu_release,
> > +       .unlocked_ioctl = afu_ioctl,
> > +};
> > +
> > +static int afu_probe(struct platform_device *pdev)
> > +{
> > +       int ret;
> > +
> > +       dev_dbg(&pdev->dev, "%s\n", __func__);
> > +
> > +       ret = fpga_dev_feature_init(pdev, port_feature_drvs);
> > +       if (ret)
> > +               return ret;
> > +
> > +       ret = fpga_register_dev_ops(pdev, &afu_fops, THIS_MODULE);
> > +       if (ret)
> > +               fpga_dev_feature_uinit(pdev);
> > +
> > +       return ret;
> > +}
> > +
> > +static int afu_remove(struct platform_device *pdev)
> > +{
> > +       dev_dbg(&pdev->dev, "%s\n", __func__);
> > +
> > +       fpga_dev_feature_uinit(pdev);
> > +       fpga_unregister_dev_ops(pdev);
> 
> Please skip a line before return.

Will fix this.

Thanks
Hao

> 
> > +       return 0;
> > +}
> > +
> > +static struct platform_driver afu_driver = {
> > +       .driver = {
> > +               .name    = "intel-fpga-port",
> > +       },
> > +       .probe   = afu_probe,
> > +       .remove  = afu_remove,
> > +};
> > +
> > +module_platform_driver(afu_driver);
> > +
> > +MODULE_DESCRIPTION("Intel FPGA Accelerated Function Unit driver");
> > +MODULE_AUTHOR("Intel Corporation");
> > +MODULE_LICENSE("GPL v2");
> > +MODULE_ALIAS("platform:intel-fpga-port");
> > --
> > 1.8.3.1
> >

Re: [PATCH v2 18/22] fpga: intel: add FPGA Accelerated Function Unit driver basic framework

[Wu Hao]

On Thu, Aug 17, 2017 at 12:09:51PM -0700, Moritz Fischer wrote:
> On Sun, Jun 25, 2017 at 6:52 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> > On Intel FPGA devices, the Accelerated Function Unit (AFU), can be
> > reprogrammed for different functions. It connects to the FPGA
> > infrastructure("blue bistream") via a Port. Port CSRs are implemented
> > separately from the AFU CSRs to provide control and status of the Port.
> > Once valid green bitstream is programmed into the AFU, it allows access
> > to the AFU CSRs in the AFU MMIO space.
> >
> > This patch only implements basic driver framework for AFU, including
> > device file operation framework.
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> > Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > ---
> > v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
> >     switched to GPLv2 license.
> > ---
> >  drivers/fpga/Kconfig          |   9 +++
> >  drivers/fpga/Makefile         |   2 +
> >  drivers/fpga/intel-afu-main.c | 159 ++++++++++++++++++++++++++++++++++++++++++
> >  3 files changed, 170 insertions(+)
> >  create mode 100644 drivers/fpga/intel-afu-main.c
> >
> > diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> > index db1a6ea..4658a13 100644
> > --- a/drivers/fpga/Kconfig
> > +++ b/drivers/fpga/Kconfig
> > @@ -172,6 +172,15 @@ config INTEL_FPGA_FME_REGION
> >           Say Y to enable FPGA Region driver for Intel FPGA Management
> >           Engine.
> >
> > +config INTEL_FPGA_AFU
> > +       tristate "Intel FPGA AFU Driver"
> > +       depends on INTEL_FPGA_PCI
> > +       help
> > +         This is the driver for FPGA Accelerated Function Unit (AFU) which
> > +         implements AFU and Port management features. A User AFU connects
> > +         to the FPGA infrastructure via a Port. There may be more than 1
> > +         Port/AFU per Intel FPGA.
> > +
> >  endif
> >
> >  endif # FPGA
> > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> > index fd0a71f..ce08833 100644
> > --- a/drivers/fpga/Makefile
> > +++ b/drivers/fpga/Makefile
> > @@ -34,6 +34,8 @@ obj-$(CONFIG_INTEL_FPGA_FME)          += intel-fpga-fme.o
> >  obj-$(CONFIG_INTEL_FPGA_FME_MGR)       += intel-fpga-fme-mgr.o
> >  obj-$(CONFIG_INTEL_FPGA_FME_BRIDGE)    += intel-fpga-fme-br.o
> >  obj-$(CONFIG_INTEL_FPGA_FME_REGION)    += intel-fpga-fme-region.o
> > +obj-$(CONFIG_INTEL_FPGA_AFU)           += intel-fpga-afu.o
> >
> >  intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
> >  intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
> > +intel-fpga-afu-objs := intel-afu-main.o
> > diff --git a/drivers/fpga/intel-afu-main.c b/drivers/fpga/intel-afu-main.c
> > new file mode 100644
> > index 0000000..96d0367
> > --- /dev/null
> > +++ b/drivers/fpga/intel-afu-main.c
> > @@ -0,0 +1,159 @@
> > +/*
> > + * Driver for Intel FPGA Accelerated Function Unit (AFU)
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> > + *   Joseph Grecco <joe.grecco-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Ananda Ravuri <ananda.ravuri-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Henry Mitchel <henry.mitchel-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *
> > + * This work is licensed under the terms of the GNU GPL version 2. See
> > + * the COPYING file in the top-level directory.
> 
> Or just: SPDX-License-Identifier: GPL-2.0

Sure, will update it.

Thanks
Hao

> > + */
> > +
> > +#include <linux/kernel.h>
> > +#include <linux/module.h>
> > +
> > +#include "intel-feature-dev.h"
> > +
> > +static int port_hdr_init(struct platform_device *pdev, struct feature *feature)
> > +{
> > +       dev_dbg(&pdev->dev, "PORT HDR Init.\n");
> > +
> > +       return 0;
> > +}
> > +
> > +static void port_hdr_uinit(struct platform_device *pdev,
> > +                                       struct feature *feature)
> > +{
> > +       dev_dbg(&pdev->dev, "PORT HDR UInit.\n");
> > +}
> > +
> > +struct feature_ops port_hdr_ops = {
> > +       .init = port_hdr_init,
> > +       .uinit = port_hdr_uinit,
> > +};
> > +
> > +static struct feature_driver port_feature_drvs[] = {
> > +       {
> > +               .name = PORT_FEATURE_HEADER,
> > +               .ops = &port_hdr_ops,
> > +       },
> > +       {
> > +               .ops = NULL,
> > +       }
> > +};
> > +
> > +static int afu_open(struct inode *inode, struct file *filp)
> > +{
> > +       struct platform_device *fdev = fpga_inode_to_feature_dev(inode);
> > +       struct feature_platform_data *pdata;
> > +       int ret;
> > +
> > +       pdata = dev_get_platdata(&fdev->dev);
> > +       if (WARN_ON(!pdata))
> > +               return -ENODEV;
> > +
> > +       ret = feature_dev_use_begin(pdata);
> > +       if (ret)
> > +               return ret;
> > +
> > +       dev_dbg(&fdev->dev, "Device File Open\n");
> > +       filp->private_data = fdev;
> > +
> > +       return 0;
> > +}
> > +
> > +static int afu_release(struct inode *inode, struct file *filp)
> > +{
> > +       struct platform_device *pdev = filp->private_data;
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +
> > +       dev_dbg(&pdev->dev, "Device File Release\n");
> > +
> > +       feature_dev_use_end(pdata);
> > +       return 0;
> > +}
> > +
> > +static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
> > +{
> > +       struct platform_device *pdev = filp->private_data;
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct feature *f;
> > +       long ret;
> > +
> > +       dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
> > +
> > +       switch (cmd) {
> > +       default:
> > +               /*
> > +                * Let sub-feature's ioctl function to handle the cmd
> > +                * Sub-feature's ioctl returns -ENODEV when cmd is not
> > +                * handled in this sub feature, and returns 0 and other
> > +                * error code if cmd is handled.
> > +                */
> > +               fpga_dev_for_each_feature(pdata, f)
> > +                       if (f->ops && f->ops->ioctl) {
> > +                               ret = f->ops->ioctl(pdev, f, cmd, arg);
> > +                               if (ret == -ENODEV)
> > +                                       continue;
> > +                               else
> > +                                       return ret;
> > +                       }
> > +       }
> > +
> > +       return -EINVAL;
> > +}
> > +
> > +static const struct file_operations afu_fops = {
> > +       .owner = THIS_MODULE,
> > +       .open = afu_open,
> > +       .release = afu_release,
> > +       .unlocked_ioctl = afu_ioctl,
> > +};
> > +
> > +static int afu_probe(struct platform_device *pdev)
> > +{
> > +       int ret;
> > +
> > +       dev_dbg(&pdev->dev, "%s\n", __func__);
> > +
> > +       ret = fpga_dev_feature_init(pdev, port_feature_drvs);
> > +       if (ret)
> > +               return ret;
> > +
> > +       ret = fpga_register_dev_ops(pdev, &afu_fops, THIS_MODULE);
> > +       if (ret)
> > +               fpga_dev_feature_uinit(pdev);
> > +
> > +       return ret;
> > +}
> > +
> > +static int afu_remove(struct platform_device *pdev)
> > +{
> > +       dev_dbg(&pdev->dev, "%s\n", __func__);
> > +
> > +       fpga_dev_feature_uinit(pdev);
> > +       fpga_unregister_dev_ops(pdev);
> > +       return 0;
> > +}
> > +
> > +static struct platform_driver afu_driver = {
> > +       .driver = {
> > +               .name    = "intel-fpga-port",
> > +       },
> > +       .probe   = afu_probe,
> > +       .remove  = afu_remove,
> > +};
> > +
> > +module_platform_driver(afu_driver);
> > +
> > +MODULE_DESCRIPTION("Intel FPGA Accelerated Function Unit driver");
> > +MODULE_AUTHOR("Intel Corporation");
> > +MODULE_LICENSE("GPL v2");
> > +MODULE_ALIAS("platform:intel-fpga-port");
> > --
> > 1.8.3.1
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> Thanks,
> 
> Moritz

Re: [PATCH v2 09/22] fpga: intel: pcie: adds fpga_for_each_port callback for fme device

[Wu Hao]

On Thu, Aug 17, 2017 at 04:31:56PM -0500, Alan Tull wrote:
> On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> 
> Hi Hao,
> 
> > For FPGA Management Engine (FME), it requires fpga_for_each_port callback
> > for actions on ports, so export this function from PCIe driver by adding
> > the callback to the platform data.
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> > Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > ---
> > v2: rebased
> > ---
> >  drivers/fpga/intel-feature-dev.h |  9 +++++++++
> >  drivers/fpga/intel-pcie.c        | 24 ++++++++++++++++++++++++
> >  2 files changed, 33 insertions(+)
> >
> > diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
> > index d64a51e..06b3fb6 100644
> > --- a/drivers/fpga/intel-feature-dev.h
> > +++ b/drivers/fpga/intel-feature-dev.h
> > @@ -235,6 +235,9 @@ struct feature_platform_data {
> >         struct platform_device *dev;
> >         unsigned int disable_count;     /* count for port disable */
> >
> > +       struct platform_device *(*fpga_for_each_port)(struct platform_device *,
> > +                       void *, int (*match)(struct platform_device *, void *));
> > +
> >         int num;                        /* number of features */
> >         struct feature features[0];
> >  };
> > @@ -354,4 +357,10 @@ static inline int fpga_port_reset(struct platform_device *pdev)
> >
> >         return pdata->features[index].ioaddr;
> >  }
> > +
> > +static inline struct device *
> > +fpga_feature_dev_to_pcidev(struct platform_device *dev)
> > +{
> > +       return dev->dev.parent->parent;
> > +}
> >  #endif
> > diff --git a/drivers/fpga/intel-pcie.c b/drivers/fpga/intel-pcie.c
> > index 54c0e3a..86ea5c7 100644
> > --- a/drivers/fpga/intel-pcie.c
> > +++ b/drivers/fpga/intel-pcie.c
> > @@ -209,6 +209,27 @@ static int parse_switch_to(struct build_feature_devs_info *binfo, int bar)
> >         return parse_start_from(binfo, bar);
> >  }
> >
> > +static struct platform_device *fpga_for_each_port(struct platform_device *pdev,
> 
> Regarding the name of this function, isn't this more of a 'find'
> function like class_find_device() than a 'for_each' function?
> 
> Also please document that the caller will need to put_device().

Yes, agree, fpga_find_port() should be a better name.
I will fix this and add comments for this function.

Thanks for your comments.
Hao

> 
> Thanks,
> Alan
> 
> > +                    void *data, int (*match)(struct platform_device *, void *))
> > +{
> > +       struct device *pci_dev = fpga_feature_dev_to_pcidev(pdev);
> > +       struct cci_drvdata *drvdata = dev_get_drvdata(pci_dev);
> > +       struct feature_platform_data *pdata;
> > +       struct platform_device *port_dev;
> > +
> > +       mutex_lock(&drvdata->lock);
> > +       list_for_each_entry(pdata, &drvdata->port_dev_list, node) {
> > +               port_dev = pdata->dev;
> > +
> > +               if (match(port_dev, data) && get_device(&port_dev->dev))
> > +                       goto exit;
> > +       }
> > +       port_dev = NULL;
> > +exit:
> > +       mutex_unlock(&drvdata->lock);
> > +       return port_dev;
> > +}
> > +
> >  static struct build_feature_devs_info *
> >  build_info_alloc_and_init(struct pci_dev *pdev)
> >  {
> > @@ -310,6 +331,9 @@ static int build_info_commit_dev(struct build_feature_devs_info *binfo)
> >         if (!pdata)
> >                 return -ENOMEM;
> >
> > +       if (type == FME_ID)
> > +               pdata->fpga_for_each_port = fpga_for_each_port;
> > +
> >         /*
> >          * the count should be initialized to 0 to make sure
> >          *__fpga_port_enable() following __fpga_port_disable()
> > --
> > 1.8.3.1
> >

Re: [PATCH v2 07/22] fpga: intel: pcie: parse feature list and create platform device for features.

[Alan Tull]

 a (wh., *()On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu@intel.com> wrote:

Hi Hao,

I'm done with some board bringup so I have time to look at your patchset again.

Something I can't help but notice is that this patchset will be almost
completely reusable for embedded FPGAs if the enumeration is separated
from the pcie code.  After the devices are created, they are just mmio
devices.  That makes this whole scheme available for embedded FPGAs.

The division of things would be that the pcie code would read the
headers and do ioremapping.  Then pass the headers to the enumeration
code to create the devices.  With that division, another platform that
is embedded could have its own code that read the headers and did the
iomapping and then used the separate enumeration code.

Going through intel-pcie.c, the truly pcie specific parts are already
pretty well contained.  The pcie code is in probe,
cci_pci_ioremap_bar, cci_pci_release_regions, and the 'parse_s*'
functions that ultimately call cci_pci_ioremap_bar.  So it all comes
down to the code that calls the 'parse_s*' functions.  That is, the
code the reads the fme header and the port headers.  Currently, if I'm
reading this right, this code ioremaps each region, reads its header,
and creates its device before reading the next header.  So we just
need to change the order a little and ioremap/read all the headers
before starting to create devices.  That separates the pci ioremap
from creating the devices.  So the probe function could ioremap and
read in the fme header.  Then it would ioremap and read in each of the
port headers.  After they are read in, then pass the all these headers
to the separate enumeration code that will parse it and create the
devices.  That part is not pcie specific, right?  With the enumeration
code in a separate file (and assuming that the ioremap has already
happened),  another platform that is embedded could have its probe do
the iomapping and reading the headers, and then call the same
enumeration code.

Actually, there's a bit more to it than that but I think this is
doable.  build_info_add_sub_feature() (which is called by
create_feature_instance) saves off resource info for each sub device.
Does this mean this code is ioremapping again (in
intel-fpga-fme-mgr.c's probe)?

A few more things below...

> From: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>
> Device Feature List structure creates a link list of feature headers
> within the MMIO space to provide an extensible way of adding features.
>
> The Intel FPGA PCIe driver walks through the feature headers to enumerate
> feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
> Function Unit (AFU), and their private sub features. For feature devices,
> it creates the platform devices and linked the private sub features into
> their platform data.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ---
> v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
>     switched to GPLv2 license.
>     fixed comments from Moritz Fischer.
>     fixed kbuild warning, typos and clean up the code.
> ---
>  drivers/fpga/Makefile            |   2 +-
>  drivers/fpga/intel-feature-dev.c | 130 ++++++
>  drivers/fpga/intel-feature-dev.h | 341 ++++++++++++++++
>  drivers/fpga/intel-pcie.c        | 841 ++++++++++++++++++++++++++++++++++++++-
>  4 files changed, 1311 insertions(+), 3 deletions(-)
>  create mode 100644 drivers/fpga/intel-feature-dev.c
>  create mode 100644 drivers/fpga/intel-feature-dev.h
>
> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> index 5613133..ad24b3d 100644
> --- a/drivers/fpga/Makefile
> +++ b/drivers/fpga/Makefile
> @@ -31,4 +31,4 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
>  # Intel FPGA Support
>  obj-$(CONFIG_INTEL_FPGA_PCI)           += intel-fpga-pci.o
>
> -intel-fpga-pci-objs := intel-pcie.o
> +intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
> diff --git a/drivers/fpga/intel-feature-dev.c b/drivers/fpga/intel-feature-dev.c
> new file mode 100644
> index 0000000..68f9cba
> --- /dev/null
> +++ b/drivers/fpga/intel-feature-dev.c
> @@ -0,0 +1,130 @@
> +/*
> + * Intel FPGA Feature Device Driver
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <luwei.kang@intel.com>
> + *   Zhang Yi <yi.z.zhang@intel.com>
> + *   Wu Hao <hao.wu@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2. See
> + * the COPYING file in the top-level directory.
> + */
> +
> +#include "intel-feature-dev.h"
> +
> +void feature_platform_data_add(struct feature_platform_data *pdata,
> +                              int index, const char *name,
> +                              int resource_index, void __iomem *ioaddr)
> +{
> +       WARN_ON(index >= pdata->num);
> +
> +       pdata->features[index].name = name;
> +       pdata->features[index].resource_index = resource_index;
> +       pdata->features[index].ioaddr = ioaddr;
> +}
> +
> +struct feature_platform_data *
> +feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
> +{
> +       struct feature_platform_data *pdata;
> +
> +       pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
> +       if (pdata) {
> +               pdata->dev = dev;
> +               pdata->num = num;
> +               mutex_init(&pdata->lock);
> +       }
> +
> +       return pdata;
> +}
> +
> +int fme_feature_num(void)
> +{
> +       return FME_FEATURE_ID_MAX;
> +}
> +
> +int port_feature_num(void)
> +{
> +       return PORT_FEATURE_ID_MAX;
> +}

Do these need to be functions?  If so, static.  But if you can just
use the #define'd values that's one level of indirection we can get
rid of.  And when I'm going through code where there's extra levels of
indirection that don't do anything, it makes it harder to understand
;)

> +
> +int fpga_port_id(struct platform_device *pdev)
> +{
> +       struct feature_port_header *port_hdr;
> +       struct feature_port_capability capability;
> +
> +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> +                                              PORT_FEATURE_ID_HEADER);
> +       WARN_ON(!port_hdr);
> +
> +       capability.csr = readq(&port_hdr->capability);
> +       return capability.port_number;
> +}
> +EXPORT_SYMBOL_GPL(fpga_port_id);

This is here because every feature is a port, right?

> +
> +/*
> + * Enable Port by clear the port soft reset bit, which is set by default.
> + * The User AFU is unable to respond to any MMIO access while in reset.
> + * __fpga_port_enable function should only be used after __fpga_port_disable
> + * function.
> + */
> +void __fpga_port_enable(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct feature_port_header *port_hdr;
> +       struct feature_port_control control;
> +
> +       WARN_ON(!pdata->disable_count);
> +
> +       if (--pdata->disable_count != 0)
> +               return;
> +
> +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> +                                              PORT_FEATURE_ID_HEADER);
> +       WARN_ON(!port_hdr);
> +
> +       control.csr = readq(&port_hdr->control);
> +       control.port_sftrst = 0x0;
> +       writeq(control.csr, &port_hdr->control);
> +}
> +EXPORT_SYMBOL_GPL(__fpga_port_enable);
> +
> +#define RST_POLL_INVL 10 /* us */
> +#define RST_POLL_TIMEOUT 1000 /* us */
> +
> +int __fpga_port_disable(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct feature_port_header *port_hdr;
> +       struct feature_port_control control;
> +
> +       if (pdata->disable_count++ != 0)
> +               return 0;
> +
> +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> +                                              PORT_FEATURE_ID_HEADER);
> +       WARN_ON(!port_hdr);
> +
> +       /* Set port soft reset */
> +       control.csr = readq(&port_hdr->control);
> +       control.port_sftrst = 0x1;
> +       writeq(control.csr, &port_hdr->control);
> +
> +       /*
> +        * HW sets ack bit to 1 when all outstanding requests have been drained
> +        * on this port and minimum soft reset pulse width has elapsed.
> +        * Driver polls port_soft_reset_ack to determine if reset done by HW.
> +        */
> +       if (readq_poll_timeout(&port_hdr->control, control.csr,
> +                              (control.port_sftrst_ack == 1),
> +                              RST_POLL_INVL, RST_POLL_TIMEOUT)) {
> +               dev_err(&pdev->dev, "timeout, fail to reset device\n");
> +               return -ETIMEDOUT;
> +       }
> +
> +       return 0;
> +}
> +EXPORT_SYMBOL_GPL(__fpga_port_disable);
> diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
> new file mode 100644
> index 0000000..f67784a
> --- /dev/null
> +++ b/drivers/fpga/intel-feature-dev.h
> @@ -0,0 +1,341 @@
> +/*
> + * Intel FPGA Feature Device Driver Header File
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <luwei.kang@intel.com>
> + *   Zhang Yi <yi.z.zhang@intel.com>
> + *   Wu Hao <hao.wu@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2. See
> + * the COPYING file in the top-level directory.
> + */
> +
> +#ifndef __INTEL_FPGA_FEATURE_H
> +#define __INTEL_FPGA_FEATURE_H
> +
> +#include <linux/fs.h>
> +#include <linux/pci.h>
> +#include <linux/uuid.h>
> +#include <linux/delay.h>
> +#include <linux/iopoll.h>
> +#include <linux/platform_device.h>
> +
> +#ifndef readq
> +static inline u64 readq(void __iomem *addr)
> +{
> +       return readl(addr) + ((u64)readl(addr + 4) << 32);
> +}
> +#endif
> +
> +#ifndef writeq
> +static inline void writeq(u64 val, void __iomem *addr)
> +{
> +       writel((u32) (val), addr);
> +       writel((u32) (val >> 32), (addr + 4));
> +}
> +#endif
> +
> +/* maximum supported number of ports */
> +#define MAX_FPGA_PORT_NUM 4
> +/* plus one for fme device */
> +#define MAX_FEATURE_DEV_NUM    (MAX_FPGA_PORT_NUM + 1)
> +
> +#define FME_FEATURE_HEADER          "fme_hdr"
> +#define FME_FEATURE_THERMAL_MGMT    "fme_thermal"
> +#define FME_FEATURE_POWER_MGMT      "fme_power"
> +#define FME_FEATURE_GLOBAL_PERF     "fme_gperf"
> +#define FME_FEATURE_GLOBAL_ERR      "fme_error"
> +#define FME_FEATURE_PR_MGMT         "fme_pr"
> +
> +#define PORT_FEATURE_HEADER         "port_hdr"
> +#define PORT_FEATURE_UAFU           "port_uafu"
> +#define PORT_FEATURE_ERR            "port_err"
> +#define PORT_FEATURE_UMSG           "port_umsg"
> +#define PORT_FEATURE_PR             "port_pr"
> +#define PORT_FEATURE_STP            "port_stp"
> +
> +/* All headers and structures must be byte-packed to match the spec. */
> +#pragma pack(1)
> +
> +/* common header for all features */
> +struct feature_header {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u64 id:12;
> +                       u64 revision:4;
> +                       u64 next_header_offset:24; /* offset to next header */
> +                       u64 rsvdz:20;
> +                       u64 type:4;                /* feature type */
> +#define FEATURE_TYPE_AFU               0x1
> +#define FEATURE_TYPE_PRIVATE           0x3
> +               };
> +       };
> +};
> +
> +/* common header for non-private features */
> +struct feature_afu_header {
> +       uuid_le guid;
> +       union {
> +               u64 csr;
> +               struct {
> +                       u64 next_afu:24;        /* pointer to next afu header */
> +                       u64 rsvdz:40;
> +               };
> +       };
> +};
> +
> +/* FME Header Register Set */
> +/* FME Capability Register */
> +struct feature_fme_capability {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u64 fabric_verid:8;     /* Fabric version ID */
> +                       u64 socket_id:1;        /* Socket id */
> +                       u64 rsvdz1:3;
> +                       u64 pcie0_link_avl:1;   /* PCIe0 link availability */
> +                       u64 pcie1_link_avl:1;   /* PCIe1 link availability */
> +                       u64 coherent_link_avl:1;/* Coherent link availability */
> +                       u64 rsvdz2:1;
> +                       u64 iommu_support:1;    /* IOMMU or VT-d supported */
> +                       u64 num_ports:3;        /* Num of ports implemented */
> +                       u64 rsvdz3:4;
> +                       u64 addr_width_bits:6;  /* Address width supported */
> +                       u64 rsvdz4:2;
> +                       u64 cache_size:12;      /* Cache size in kb */
> +                       u64 cache_assoc:4;      /* Cache Associativity */
> +                       u64 rsvdz5:15;
> +                       u64 lock_bit:1;         /* Latched lock bit by BIOS */
> +               };
> +       };
> +};
> +
> +/* FME Port Offset Register */
> +struct feature_fme_port {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u64 port_offset:24;     /* Offset to port header */
> +                       u64 rsvdz1:8;
> +                       u64 port_bar:3;         /* Bar id */
> +                       u64 rsvdz2:20;
> +                       u64 afu_access_ctrl:1;  /* AFU access type: PF/VF */
> +                       u64 rsvdz3:4;
> +                       u64 port_implemented:1; /* Port implemented or not */
> +                       u64 rsvdz4:3;
> +               };
> +       };
> +};
> +
> +struct feature_fme_header {
> +       struct feature_header header;
> +       struct feature_afu_header afu_header;
> +       u64 rsvd[2];
> +       struct feature_fme_capability capability;
> +       struct feature_fme_port port[MAX_FPGA_PORT_NUM];
> +};
> +
> +/* FME Thermal Sub Feature Register Set */
> +struct feature_fme_thermal {
> +       struct feature_header header;
> +};
> +
> +/* FME Power Sub Feature Register Set */
> +struct feature_fme_power {
> +       struct feature_header header;
> +};
> +
> +/* FME Global Performance Sub Feature Register Set */
> +struct feature_fme_gperf {
> +       struct feature_header header;
> +};
> +
> +/* FME Error Sub Feature Register Set */
> +struct feature_fme_err {
> +       struct feature_header header;
> +};
> +
> +/* FME Partial Reconfiguration Sub Feature Register Set */
> +struct feature_fme_pr {
> +       struct feature_header header;
> +};
> +
> +/* PORT Header Register Set */
> +/* Port Capability Register */
> +struct feature_port_capability {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u64 port_number:2;      /* Port Number 0-3 */
> +                       u64 rsvdz1:6;
> +                       u64 mmio_size:16;       /* User MMIO size in KB */
> +                       u64 rsvdz2:8;
> +                       u64 sp_intr_num:4;      /* Supported interrupts num */
> +                       u64 rsvdz3:28;
> +               };
> +       };
> +};
> +
> +/* Port Control Register */
> +struct feature_port_control {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u64 port_sftrst:1;      /* Port Soft Reset */
> +                       u64 rsvdz1:1;
> +                       u64 latency_tolerance:1;/* '1' >= 40us, '0' < 40us */
> +                       u64 rsvdz2:1;
> +                       u64 port_sftrst_ack:1;  /* HW ACK for Soft Reset */
> +                       u64 rsvdz3:59;
> +               };
> +       };
> +};
> +
> +struct feature_port_header {
> +       struct feature_header header;
> +       struct feature_afu_header afu_header;
> +       u64 rsvd[2];
> +       struct feature_port_capability capability;
> +       struct feature_port_control control;
> +};
> +
> +/* PORT Error Sub Feature Register Set */
> +struct feature_port_error {
> +       struct feature_header header;
> +};
> +
> +/* PORT Unordered Message Sub Feature Register Set */
> +struct feature_port_umsg {
> +       struct feature_header header;
> +};
> +
> +/* PORT SignalTap Sub Feature Register Set */
> +struct feature_port_stp {
> +       struct feature_header header;
> +};
> +
> +#pragma pack()
> +
> +struct feature {
> +       const char *name;
> +       int resource_index;
> +       void __iomem *ioaddr;
> +};
> +
> +struct feature_platform_data {
> +       /* list the feature dev to cci_drvdata->port_dev_list. */
> +       struct list_head node;
> +       struct mutex lock;
> +       struct platform_device *dev;
> +       unsigned int disable_count;     /* count for port disable */
> +
> +       int num;                        /* number of features */
> +       struct feature features[0];
> +};
> +
> +enum fme_feature_id {
> +       FME_FEATURE_ID_HEADER = 0x0,
> +       FME_FEATURE_ID_THERMAL_MGMT = 0x1,
> +       FME_FEATURE_ID_POWER_MGMT = 0x2,
> +       FME_FEATURE_ID_GLOBAL_PERF = 0x3,
> +       FME_FEATURE_ID_GLOBAL_ERR = 0x4,
> +       FME_FEATURE_ID_PR_MGMT = 0x5,
> +       FME_FEATURE_ID_MAX = 0x6,
> +};
> +
> +enum port_feature_id {
> +       PORT_FEATURE_ID_HEADER = 0x0,
> +       PORT_FEATURE_ID_ERROR = 0x1,
> +       PORT_FEATURE_ID_UMSG = 0x2,
> +       PORT_FEATURE_ID_PR = 0x3,
> +       PORT_FEATURE_ID_STP = 0x4,
> +       PORT_FEATURE_ID_UAFU = 0x5,
> +       PORT_FEATURE_ID_MAX = 0x6,
> +};
> +
> +int fme_feature_num(void);
> +int port_feature_num(void);
> +
> +#define FPGA_FEATURE_DEV_FME           "intel-fpga-fme"
> +#define FPGA_FEATURE_DEV_PORT          "intel-fpga-port"
> +
> +void feature_platform_data_add(struct feature_platform_data *pdata,
> +                              int index, const char *name,
> +                              int resource_index, void __iomem *ioaddr);
> +
> +static inline int feature_platform_data_size(const int num)
> +{
> +       return sizeof(struct feature_platform_data) +
> +               num * sizeof(struct feature);
> +}
> +
> +struct feature_platform_data *
> +feature_platform_data_alloc_and_init(struct platform_device *dev, int num);
> +
> +int fpga_port_id(struct platform_device *pdev);
> +
> +static inline int fpga_port_check_id(struct platform_device *pdev,
> +                                    void *pport_id)
> +{
> +       return fpga_port_id(pdev) == *(int *)pport_id;
> +}
> +
> +void __fpga_port_enable(struct platform_device *pdev);
> +int __fpga_port_disable(struct platform_device *pdev);
> +
> +static inline void fpga_port_enable(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +
> +       mutex_lock(&pdata->lock);
> +       __fpga_port_enable(pdev);
> +       mutex_unlock(&pdata->lock);
> +}
> +
> +static inline int fpga_port_disable(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       int ret;
> +
> +       mutex_lock(&pdata->lock);
> +       ret = __fpga_port_disable(pdev);
> +       mutex_unlock(&pdata->lock);
> +
> +       return ret;
> +}
> +
> +static inline int __fpga_port_reset(struct platform_device *pdev)
> +{
> +       int ret;
> +
> +       ret = __fpga_port_disable(pdev);
> +       if (ret)
> +               return ret;
> +
> +       __fpga_port_enable(pdev);
> +       return 0;
> +}
> +
> +static inline int fpga_port_reset(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       int ret;
> +
> +       mutex_lock(&pdata->lock);
> +       ret = __fpga_port_reset(pdev);
> +       mutex_unlock(&pdata->lock);
> +       return ret;
> +}
> +
> +static inline void __iomem *
> +get_feature_ioaddr_by_index(struct device *dev, int index)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(dev);
> +
> +       return pdata->features[index].ioaddr;
> +}
> +#endif
> diff --git a/drivers/fpga/intel-pcie.c b/drivers/fpga/intel-pcie.c
> index f697de4..70b8284 100644
> --- a/drivers/fpga/intel-pcie.c
> +++ b/drivers/fpga/intel-pcie.c
> @@ -23,10 +23,827 @@
>  #include <linux/stddef.h>
>  #include <linux/errno.h>
>  #include <linux/aer.h>
> +#include <linux/fpga/fpga-dev.h>
> +
> +#include "intel-feature-dev.h"
>
>  #define DRV_VERSION    "0.8"
>  #define DRV_NAME       "intel-fpga-pci"
>
> +#define INTEL_FPGA_DEV "intel-fpga-dev"
> +
> +static DEFINE_MUTEX(fpga_id_mutex);
> +
> +enum fpga_id_type {
> +       FME_ID,         /* fme id allocation and mapping */
> +       PORT_ID,        /* port id allocation and mapping */
> +       FPGA_ID_MAX,
> +};
> +
> +/* it is protected by fpga_id_mutex */
> +static struct idr fpga_ids[FPGA_ID_MAX];
> +
> +struct cci_drvdata {
> +       struct device *fme_dev;
> +
> +       struct mutex lock;
> +       struct list_head port_dev_list;
> +
> +       struct list_head regions; /* global list of pci bar mapping region */
> +};
> +
> +/* pci bar mapping info */
> +struct cci_pci_region {
> +       int bar;
> +       void __iomem *ioaddr;   /* pointer to mapped bar region */
> +       struct list_head node;
> +};
> +
> +static void fpga_ids_init(void)
> +{
> +       int i;
> +
> +       for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> +               idr_init(fpga_ids + i);
> +}
> +
> +static void fpga_ids_destroy(void)
> +{
> +       int i;
> +
> +       for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> +               idr_destroy(fpga_ids + i);
> +}
> +
> +static int alloc_fpga_id(enum fpga_id_type type, struct device *dev)
> +{
> +       int id;
> +
> +       WARN_ON(type >= FPGA_ID_MAX);
> +       mutex_lock(&fpga_id_mutex);
> +       id = idr_alloc(fpga_ids + type, dev, 0, 0, GFP_KERNEL);
> +       mutex_unlock(&fpga_id_mutex);
> +       return id;
> +}
> +
> +static void free_fpga_id(enum fpga_id_type type, int id)
> +{
> +       WARN_ON(type >= FPGA_ID_MAX);
> +       mutex_lock(&fpga_id_mutex);
> +       idr_remove(fpga_ids + type, id);
> +       mutex_unlock(&fpga_id_mutex);
> +}
> +
> +static void cci_pci_add_port_dev(struct pci_dev *pdev,
> +                                struct platform_device *port_dev)
> +{
> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +       struct feature_platform_data *pdata = dev_get_platdata(&port_dev->dev);
> +
> +       mutex_lock(&drvdata->lock);
> +       list_add(&pdata->node, &drvdata->port_dev_list);
> +       get_device(&pdata->dev->dev);
> +       mutex_unlock(&drvdata->lock);
> +}
> +
> +static void cci_pci_remove_port_devs(struct pci_dev *pdev)
> +{
> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +       struct feature_platform_data *pdata, *ptmp;
> +
> +       mutex_lock(&drvdata->lock);
> +       list_for_each_entry_safe(pdata, ptmp, &drvdata->port_dev_list, node) {
> +               struct platform_device *port_dev = pdata->dev;
> +
> +               /* the port should be unregistered first. */
> +               WARN_ON(device_is_registered(&port_dev->dev));
> +               list_del(&pdata->node);
> +               free_fpga_id(PORT_ID, port_dev->id);
> +               put_device(&port_dev->dev);
> +       }
> +       mutex_unlock(&drvdata->lock);
> +}
> +
> +/* info collection during feature dev build. */
> +struct build_feature_devs_info {
> +       struct pci_dev *pdev;
> +
> +       /*
> +        * PCI BAR mapping info. Parsing feature list starts from
> +        * BAR 0 and switch to different BARs to parse Port
> +        */
> +       void __iomem *ioaddr;
> +       void __iomem *ioend;
> +       int current_bar;
> +
> +       /* points to FME header where the port offset is figured out. */
> +       void __iomem *pfme_hdr;
> +
> +       /* the container device for all feature devices */
> +       struct fpga_dev *parent_dev;
> +
> +       /* current feature device */
> +       struct platform_device *feature_dev;
> +};
> +
> +static void cci_pci_release_regions(struct pci_dev *pdev)
> +{
> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +       struct cci_pci_region *tmp, *region;
> +
> +       list_for_each_entry_safe(region, tmp, &drvdata->regions, node) {
> +               list_del(&region->node);
> +               if (region->ioaddr)
> +                       pci_iounmap(pdev, region->ioaddr);
> +               devm_kfree(&pdev->dev, region);
> +       }
> +}
> +
> +static void __iomem *cci_pci_ioremap_bar(struct pci_dev *pdev, int bar)
> +{
> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +       struct cci_pci_region *region;
> +
> +       list_for_each_entry(region, &drvdata->regions, node)
> +               if (region->bar == bar) {
> +                       dev_dbg(&pdev->dev, "BAR %d region exists\n", bar);
> +                       return region->ioaddr;
> +               }
> +
> +       region = devm_kzalloc(&pdev->dev, sizeof(*region), GFP_KERNEL);
> +       if (!region)
> +               return NULL;
> +
> +       region->bar = bar;
> +       region->ioaddr = pci_ioremap_bar(pdev, bar);
> +       if (!region->ioaddr) {
> +               dev_err(&pdev->dev, "can't ioremap memory from BAR %d.\n", bar);
> +               devm_kfree(&pdev->dev, region);
> +               return NULL;
> +       }
> +
> +       list_add(&region->node, &drvdata->regions);
> +       return region->ioaddr;
> +}
> +
> +static int parse_start_from(struct build_feature_devs_info *binfo, int bar)
> +{
> +       binfo->ioaddr = cci_pci_ioremap_bar(binfo->pdev, bar);
> +       if (!binfo->ioaddr)
> +               return -ENOMEM;
> +
> +       binfo->current_bar = bar;
> +       binfo->ioend = binfo->ioaddr + pci_resource_len(binfo->pdev, bar);
> +       return 0;
> +}
> +
> +static int parse_start(struct build_feature_devs_info *binfo)
> +{
> +       /* fpga feature list starts from BAR 0 */
> +       return parse_start_from(binfo, 0);
> +}
> +
> +/* switch the memory mapping to BAR# @bar */
> +static int parse_switch_to(struct build_feature_devs_info *binfo, int bar)
> +{
> +       return parse_start_from(binfo, bar);
> +}

parse_switch_to() and parse_start() are both one line wrappers for
parse_start_from() and are only called once.  :-) Please just use
parse_start_from and get rid of parse_start and parse_switch_to.  I
don't think this code is all that super complicated.  But as I'm
grepping through this code, I'm trying to see all the places that do
the same thing; if I have to look for multiple function names, it
takes longer to get what is going on here.

Actually, cci_pci_ioremap_bar is only called in one place -
parse_start_from - which really just adds two lines of code.
parse_start_from could go away.  Just a suggestion.  The reason this
comes up is that cci_pci_ioremap_bar is the more descriptive function
name.  The parse_start/switch function names hide the biggest thing
they do, which is ioremapping.  They don't actually do any parsing ;)

> +
> +static struct build_feature_devs_info *
> +build_info_alloc_and_init(struct pci_dev *pdev)
> +{
> +       struct build_feature_devs_info *binfo;
> +
> +       binfo = devm_kzalloc(&pdev->dev, sizeof(*binfo), GFP_KERNEL);
> +       if (binfo)
> +               binfo->pdev = pdev;

build_info_alloc_and_init() is only doing a devm_kzalloc and is only
called once.  The code would be more readable if you just had the
devm_kzalloc in the function that called this
(cci_pci_create_feature_devs).

> +
> +       return binfo;
> +}
> +
> +static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
> +{
> +       if (!strcmp(pdev->name, FPGA_FEATURE_DEV_FME))
> +               return FME_ID;
> +
> +       if (!strcmp(pdev->name, FPGA_FEATURE_DEV_PORT))
> +               return PORT_ID;
> +
> +       WARN_ON(1);
> +       return FPGA_ID_MAX;
> +}
> +
> +/*
> + * register current feature device, it is called when we need to switch to
> + * another feature parsing or we have parsed all features
> + */
> +static int build_info_commit_dev(struct build_feature_devs_info *binfo)
> +{
> +       int ret;
> +
> +       if (!binfo->feature_dev)
> +               return 0;
> +
> +       ret = platform_device_add(binfo->feature_dev);
> +       if (!ret) {
> +               struct cci_drvdata *drvdata;
> +
> +               drvdata = dev_get_drvdata(&binfo->pdev->dev);
> +               if (feature_dev_id_type(binfo->feature_dev) == PORT_ID)
> +                       cci_pci_add_port_dev(binfo->pdev, binfo->feature_dev);
> +               else
> +                       drvdata->fme_dev = get_device(&binfo->feature_dev->dev);
> +
> +               /*
> +                * reset it to avoid build_info_free() freeing their resource.
> +                *
> +                * The resource of successfully registered feature devices
> +                * will be freed by platform_device_unregister(). See the
> +                * comments in build_info_create_dev().
> +                */
> +               binfo->feature_dev = NULL;
> +       }
> +
> +       return ret;
> +}
> +
> +static int
> +build_info_create_dev(struct build_feature_devs_info *binfo,
> +                     enum fpga_id_type type, int feature_nr, const char *name)
> +{
> +       struct platform_device *fdev;
> +       struct resource *res;
> +       struct feature_platform_data *pdata;
> +       int ret;
> +
> +       /* we will create a new device, commit current device first */
> +       ret = build_info_commit_dev(binfo);
> +       if (ret)
> +               return ret;
> +
> +       /*
> +        * we use -ENODEV as the initialization indicator which indicates
> +        * whether the id need to be reclaimed
> +        */
> +       fdev = binfo->feature_dev = platform_device_alloc(name, -ENODEV);
> +       if (!fdev)
> +               return -ENOMEM;
> +
> +       fdev->id = alloc_fpga_id(type, &fdev->dev);
> +       if (fdev->id < 0)
> +               return fdev->id;
> +
> +       fdev->dev.parent = &binfo->parent_dev->dev;
> +
> +       /*
> +        * we do not need to care for the memory which is associated with
> +        * the platform device. After calling platform_device_unregister(),
> +        * it will be automatically freed by device's release() callback,
> +        * platform_device_release().
> +        */
> +       pdata = feature_platform_data_alloc_and_init(fdev, feature_nr);
> +       if (!pdata)
> +               return -ENOMEM;
> +
> +       /*
> +        * the count should be initialized to 0 to make sure
> +        *__fpga_port_enable() following __fpga_port_disable()
> +        * works properly for port device.
> +        * and it should always be 0 for fme device.
> +        */
> +       WARN_ON(pdata->disable_count);
> +
> +       fdev->dev.platform_data = pdata;
> +       fdev->num_resources = feature_nr;
> +       fdev->resource = kcalloc(feature_nr, sizeof(*res), GFP_KERNEL);
> +       if (!fdev->resource)
> +               return -ENOMEM;
> +
> +       return 0;
> +}
> +
> +static int remove_feature_dev(struct device *dev, void *data)
> +{
> +       struct platform_device *pdev = to_platform_device(dev);
> +
> +       platform_device_unregister(pdev);
> +       return 0;
> +}
> +
> +static int remove_parent_dev(struct device *dev, void *data)
> +{
> +       /* remove platform devices attached in the parent device */
> +       device_for_each_child(dev, NULL, remove_feature_dev);
> +       fpga_dev_destroy(to_fpga_dev(dev));
> +       return 0;
> +}
> +
> +static void remove_all_devs(struct pci_dev *pdev)
> +{
> +       /* remove parent device and all its children. */
> +       device_for_each_child(&pdev->dev, NULL, remove_parent_dev);
> +}
> +
> +static void build_info_free(struct build_feature_devs_info *binfo)
> +{
> +       if (!IS_ERR_OR_NULL(binfo->parent_dev))
> +               remove_all_devs(binfo->pdev);
> +
> +       /*
> +        * it is a valid id, free it. See comments in
> +        * build_info_create_dev()
> +        */
> +       if (binfo->feature_dev && binfo->feature_dev->id >= 0)
> +               free_fpga_id(feature_dev_id_type(binfo->feature_dev),
> +                            binfo->feature_dev->id);
> +
> +       platform_device_put(binfo->feature_dev);
> +
> +       devm_kfree(&binfo->pdev->dev, binfo);
> +}
> +
> +#define FEATURE_TYPE_AFU       0x1
> +#define FEATURE_TYPE_PRIVATE   0x3
> +
> +/* FME and PORT GUID are fixed */
> +#define FEATURE_FME_GUID "f9e17764-38f0-82fe-e346-524ae92aafbf"
> +#define FEATURE_PORT_GUID "6b355b87-b06c-9642-eb42-8d139398b43a"
> +
> +static bool feature_is_fme(struct feature_afu_header *afu_hdr)
> +{
> +       uuid_le u;
> +
> +       uuid_le_to_bin(FEATURE_FME_GUID, &u);
> +
> +       return !uuid_le_cmp(u, afu_hdr->guid);
> +}
> +
> +static bool feature_is_port(struct feature_afu_header *afu_hdr)
> +{
> +       uuid_le u;
> +
> +       uuid_le_to_bin(FEATURE_PORT_GUID, &u);
> +
> +       return !uuid_le_cmp(u, afu_hdr->guid);
> +}
> +
> +/*
> + * UAFU GUID is dynamic as it can be changed after FME downloads different
> + * Green Bitstream to the port, so we treat the unknown GUIDs which are
> + * attached on port's feature list as UAFU.
> + */
> +static bool feature_is_UAFU(struct build_feature_devs_info *binfo)
> +{
> +       if (!binfo->feature_dev ||
> +             feature_dev_id_type(binfo->feature_dev) != PORT_ID)
> +               return false;
> +
> +       return true;
> +}
> +
> +static void
> +build_info_add_sub_feature(struct build_feature_devs_info *binfo,
> +                          int feature_id, const char *feature_name,
> +                          resource_size_t resource_size, void __iomem *start)
> +{
> +
> +       struct platform_device *fdev = binfo->feature_dev;
> +       struct feature_platform_data *pdata = dev_get_platdata(&fdev->dev);
> +       struct resource *res = &fdev->resource[feature_id];
> +
> +       res->start = pci_resource_start(binfo->pdev, binfo->current_bar) +
> +               start - binfo->ioaddr;
> +       res->end = res->start + resource_size - 1;
> +       res->flags = IORESOURCE_MEM;
> +       res->name = feature_name;
> +
> +       feature_platform_data_add(pdata, feature_id,
> +                                 feature_name, feature_id, start);
> +}
> +
> +struct feature_info {
> +       const char *name;
> +       resource_size_t resource_size;
> +       int feature_index;
> +};
> +
> +/* indexed by fme feature IDs which are defined in 'enum fme_feature_id'. */
> +static struct feature_info fme_features[] = {
> +       {
> +               .name = FME_FEATURE_HEADER,
> +               .resource_size = sizeof(struct feature_fme_header),
> +               .feature_index = FME_FEATURE_ID_HEADER,
> +       },
> +       {
> +               .name = FME_FEATURE_THERMAL_MGMT,
> +               .resource_size = sizeof(struct feature_fme_thermal),
> +               .feature_index = FME_FEATURE_ID_THERMAL_MGMT,
> +       },
> +       {
> +               .name = FME_FEATURE_POWER_MGMT,
> +               .resource_size = sizeof(struct feature_fme_power),
> +               .feature_index = FME_FEATURE_ID_POWER_MGMT,
> +       },
> +       {
> +               .name = FME_FEATURE_GLOBAL_PERF,
> +               .resource_size = sizeof(struct feature_fme_gperf),
> +               .feature_index = FME_FEATURE_ID_GLOBAL_PERF,
> +       },
> +       {
> +               .name = FME_FEATURE_GLOBAL_ERR,
> +               .resource_size = sizeof(struct feature_fme_err),
> +               .feature_index = FME_FEATURE_ID_GLOBAL_ERR,
> +       },
> +       {
> +               .name = FME_FEATURE_PR_MGMT,
> +               .resource_size = sizeof(struct feature_fme_pr),
> +               .feature_index = FME_FEATURE_ID_PR_MGMT,
> +       }
> +};
> +
> +/* indexed by port feature IDs which are defined in 'enum port_feature_id'. */
> +static struct feature_info port_features[] = {
> +       {
> +               .name = PORT_FEATURE_HEADER,
> +               .resource_size = sizeof(struct feature_port_header),
> +               .feature_index = PORT_FEATURE_ID_HEADER,
> +       },
> +       {
> +               .name = PORT_FEATURE_ERR,
> +               .resource_size = sizeof(struct feature_port_error),
> +               .feature_index = PORT_FEATURE_ID_ERROR,
> +       },
> +       {
> +               .name = PORT_FEATURE_UMSG,
> +               .resource_size = sizeof(struct feature_port_umsg),
> +               .feature_index = PORT_FEATURE_ID_UMSG,
> +       },
> +       {
> +               /* This feature isn't available for now */
> +               .name = PORT_FEATURE_PR,
> +               .resource_size = 0,
> +               .feature_index = PORT_FEATURE_ID_PR,
> +       },
> +       {
> +               .name = PORT_FEATURE_STP,
> +               .resource_size = sizeof(struct feature_port_stp),
> +               .feature_index = PORT_FEATURE_ID_STP,
> +       },
> +       {
> +               /*
> +                * For User AFU feature, its region size is not fixed, but
> +                * reported by register PortCapability.mmio_size. Resource
> +                * size of UAFU will be set while parse port device.
> +                */
> +               .name = PORT_FEATURE_UAFU,
> +               .resource_size = 0,
> +               .feature_index = PORT_FEATURE_ID_UAFU,
> +       },
> +};
> +
> +static int
> +create_feature_instance(struct build_feature_devs_info *binfo,
> +                       void __iomem *start, struct feature_info *finfo)
> +{
> +       if (binfo->ioend - start < finfo->resource_size)
> +               return -EINVAL;
> +
> +       build_info_add_sub_feature(binfo, finfo->feature_index, finfo->name,
> +                                  finfo->resource_size, start);
> +       return 0;
> +}

build_info_add_sub_feature() is only called one time - here.  Could
you collapse these two functions together?

> +
> +static int parse_feature_fme(struct build_feature_devs_info *binfo,
> +                            void __iomem *start)
> +{
> +       struct cci_drvdata *drvdata = dev_get_drvdata(&binfo->pdev->dev);
> +       int ret;
> +
> +       ret = build_info_create_dev(binfo, FME_ID, fme_feature_num(),
> +                                       FPGA_FEATURE_DEV_FME);
> +       if (ret)
> +               return ret;
> +
> +       if (drvdata->fme_dev) {
> +               dev_err(&binfo->pdev->dev, "Multiple FMEs are detected.\n");
> +               return -EINVAL;
> +       }
> +
> +       return create_feature_instance(binfo, start,
> +                                      &fme_features[FME_FEATURE_ID_HEADER]);
> +}
> +
> +static int parse_feature_fme_private(struct build_feature_devs_info *binfo,
> +                                    struct feature_header *hdr)
> +{
> +       struct feature_header header;
> +
> +       header.csr = readq(hdr);
> +
> +       if (header.id >= ARRAY_SIZE(fme_features)) {
> +               dev_info(&binfo->pdev->dev, "FME feature id %x is not supported yet.\n",
> +                        header.id);
> +               return 0;
> +       }
> +
> +       return create_feature_instance(binfo, hdr, &fme_features[header.id]);
> +}
> +
> +static int parse_feature_port(struct build_feature_devs_info *binfo,
> +                            void __iomem *start)
> +{
> +       int ret;
> +
> +       ret = build_info_create_dev(binfo, PORT_ID, port_feature_num(),
> +                                       FPGA_FEATURE_DEV_PORT);
> +       if (ret)
> +               return ret;
> +
> +       return create_feature_instance(binfo, start,
> +                                      &port_features[PORT_FEATURE_ID_HEADER]);
> +}
> +
> +static void enable_port_uafu(struct build_feature_devs_info *binfo,
> +                            void __iomem *start)
> +{
> +       enum port_feature_id id = PORT_FEATURE_ID_UAFU;
> +       struct feature_port_header *port_hdr;
> +       struct feature_port_capability capability;
> +
> +       port_hdr = (struct feature_port_header *)start;
> +       capability.csr = readq(&port_hdr->capability);
> +       port_features[id].resource_size = capability.mmio_size << 10;
> +
> +       /*
> +        * To enable User AFU, driver needs to clear reset bit on related port,
> +        * otherwise the mmio space of this user AFU will be invalid.
> +        */
> +       if (port_features[id].resource_size)
> +               fpga_port_reset(binfo->feature_dev);
> +}
> +
> +static int parse_feature_port_private(struct build_feature_devs_info *binfo,
> +                                     struct feature_header *hdr)
> +{
> +       struct feature_header header;
> +       enum port_feature_id id;
> +
> +       header.csr = readq(hdr);
> +       /*
> +        * the region of port feature id is [0x10, 0x13], + 1 to reserve 0
> +        * which is dedicated for port-hdr.
> +        */
> +       id = (header.id & 0x000f) + 1;
> +
> +       if (id >= ARRAY_SIZE(port_features)) {
> +               dev_info(&binfo->pdev->dev, "Port feature id %x is not supported yet.\n",
> +                        header.id);
> +               return 0;
> +       }
> +
> +       return create_feature_instance(binfo, hdr, &port_features[id]);
> +}
> +
> +static int parse_feature_port_uafu(struct build_feature_devs_info *binfo,
> +                                struct feature_header *hdr)
> +{
> +       enum port_feature_id id = PORT_FEATURE_ID_UAFU;
> +       int ret;
> +
> +       if (port_features[id].resource_size) {
> +               ret = create_feature_instance(binfo, hdr, &port_features[id]);
> +               port_features[id].resource_size = 0;
> +       } else {
> +               dev_err(&binfo->pdev->dev, "the uafu feature header is mis-configured.\n");
> +               ret = -EINVAL;
> +       }
> +
> +       return ret;
> +}
> +
> +static int parse_feature_afus(struct build_feature_devs_info *binfo,
> +                             struct feature_header *hdr)
> +{
> +       int ret;
> +       struct feature_afu_header *afu_hdr, header;
> +       void __iomem *start;
> +       void __iomem *end = binfo->ioend;
> +
> +       start = hdr;
> +       for (; start < end; start += header.next_afu) {
> +               if (end - start < (sizeof(*afu_hdr) + sizeof(*hdr)))
> +                       return -EINVAL;
> +
> +               hdr = start;
> +               afu_hdr = (struct feature_afu_header *) (hdr + 1);
> +               header.csr = readq(&afu_hdr->csr);
> +
> +               if (feature_is_fme(afu_hdr)) {
> +                       ret = parse_feature_fme(binfo, hdr);
> +                       binfo->pfme_hdr = hdr;
> +                       if (ret)
> +                               return ret;
> +               } else if (feature_is_port(afu_hdr)) {
> +                       ret = parse_feature_port(binfo, hdr);
> +                       enable_port_uafu(binfo, hdr);
> +                       if (ret)
> +                               return ret;
> +               } else if (feature_is_UAFU(binfo)) {
> +                       ret = parse_feature_port_uafu(binfo, hdr);
> +                       if (ret)
> +                               return ret;
> +               } else
> +                       dev_info(&binfo->pdev->dev, "AFU GUID %pUl is not supported yet.\n",
> +                                afu_hdr->guid.b);
> +
> +               if (!header.next_afu)
> +                       break;
> +       }
> +
> +       return 0;
> +}
> +
> +static int parse_feature_private(struct build_feature_devs_info *binfo,
> +                                struct feature_header *hdr)
> +{
> +       struct feature_header header;
> +
> +       header.csr = readq(hdr);
> +
> +       if (!binfo->feature_dev) {
> +               dev_err(&binfo->pdev->dev, "the private feature %x does not belong to any AFU.\n",
> +                       header.id);
> +               return -EINVAL;
> +       }
> +
> +       switch (feature_dev_id_type(binfo->feature_dev)) {
> +       case FME_ID:
> +               return parse_feature_fme_private(binfo, hdr);
> +       case PORT_ID:
> +               return parse_feature_port_private(binfo, hdr);
> +       default:
> +               dev_info(&binfo->pdev->dev, "private feature %x belonging to AFU %s is not supported yet.\n",
> +                        header.id, binfo->feature_dev->name);
> +       }
> +       return 0;
> +}
> +
> +static int parse_feature(struct build_feature_devs_info *binfo,
> +                        struct feature_header *hdr)
> +{
> +       struct feature_header header;
> +       int ret = 0;
> +
> +       header.csr = readq(hdr);
> +
> +       switch (header.type) {
> +       case FEATURE_TYPE_AFU:
> +               ret = parse_feature_afus(binfo, hdr);
> +               break;
> +       case FEATURE_TYPE_PRIVATE:
> +               ret = parse_feature_private(binfo, hdr);
> +               break;
> +       default:
> +               dev_info(&binfo->pdev->dev,
> +                        "Feature Type %x is not supported.\n", hdr->type);
> +       };
> +
> +       return ret;
> +}
> +
> +static int
> +parse_feature_list(struct build_feature_devs_info *binfo, void __iomem *start)
> +{
> +       struct feature_header *hdr, header;
> +       void __iomem *end = binfo->ioend;
> +       int ret = 0;
> +

Maybe a helpful comment that we are stepping through a linked list of features.

> +       for (; start < end; start += header.next_header_offset) {
> +               if (end - start < sizeof(*hdr)) {
> +                       dev_err(&binfo->pdev->dev, "The region is too small to contain a feature.\n");
> +                       ret =  -EINVAL;
> +                       break;
> +               }
> +
> +               hdr = (struct feature_header *)start;
> +               ret = parse_feature(binfo, hdr);
> +               if (ret)
> +                       break;

Instead of parse_feature, this can save off the enumeration info and
continue to read in the linked list.  After all the headers are read
in, call the (separate) enumeration code to step through the saved
headers, parse them, and create the devices.  Since the memory is
iomapped during the process of reading in the headers, the enumeration
code doesn't have to be so pcie specific.  Then this code base is
better set to run on embedded devices also.

> +
> +               header.csr = readq(hdr);
> +               if (!header.next_header_offset)
> +                       break;
> +       }
> +
> +       return ret;
> +}
> +
> +static int parse_ports_from_fme(struct build_feature_devs_info *binfo)
> +{
> +       struct feature_fme_header *fme_hdr;
> +       struct feature_fme_port port;
> +       int i = 0, ret = 0;
> +
> +       if (binfo->pfme_hdr == NULL) {
> +               dev_dbg(&binfo->pdev->dev, "VF is detected.\n");
> +               return ret;
> +       }
> +
> +       fme_hdr = binfo->pfme_hdr;
> +
> +       do {
> +               port.csr = readq(&fme_hdr->port[i]);
> +               if (!port.port_implemented)
> +                       break;
> +
> +               ret = parse_switch_to(binfo, port.port_bar);
> +               if (ret)
> +                       break;
> +
> +               ret = parse_feature_list(binfo,
> +                               binfo->ioaddr + port.port_offset);
> +               if (ret)
> +                       break;
> +       } while (++i < MAX_FPGA_PORT_NUM);
> +
> +       return ret;
> +}
> +
> +static int create_init_drvdata(struct pci_dev *pdev)
> +{
> +       struct cci_drvdata *drvdata;
> +
> +       drvdata = devm_kzalloc(&pdev->dev, sizeof(*drvdata), GFP_KERNEL);
> +       if (!drvdata)
> +               return -ENOMEM;
> +
> +       mutex_init(&drvdata->lock);
> +       INIT_LIST_HEAD(&drvdata->port_dev_list);
> +       INIT_LIST_HEAD(&drvdata->regions);
> +
> +       dev_set_drvdata(&pdev->dev, drvdata);
> +       return 0;
> +}
> +
> +static void destroy_drvdata(struct pci_dev *pdev)
> +{
> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +
> +       if (drvdata->fme_dev) {
> +               /* fme device should be unregistered first. */
> +               WARN_ON(device_is_registered(drvdata->fme_dev));
> +               free_fpga_id(FME_ID, to_platform_device(drvdata->fme_dev)->id);
> +               put_device(drvdata->fme_dev);
> +       }
> +
> +       cci_pci_remove_port_devs(pdev);
> +       cci_pci_release_regions(pdev);
> +       dev_set_drvdata(&pdev->dev, NULL);
> +       devm_kfree(&pdev->dev, drvdata);
> +}
> +
> +static int cci_pci_create_feature_devs(struct pci_dev *pdev)
> +{
> +       struct build_feature_devs_info *binfo;
> +       int ret;
> +
> +       binfo = build_info_alloc_and_init(pdev);
> +       if (!binfo)
> +               return -ENOMEM;
> +
> +       binfo->parent_dev = fpga_dev_create(&pdev->dev, INTEL_FPGA_DEV);
> +       if (IS_ERR(binfo->parent_dev)) {
> +               ret = PTR_ERR(binfo->parent_dev);
> +               goto free_binfo_exit;
> +       }
> +
> +       ret = parse_start(binfo);
> +       if (ret)
> +               goto free_binfo_exit;
> +
> +       ret = parse_feature_list(binfo, binfo->ioaddr);
> +       if (ret)
> +               goto free_binfo_exit;
> +
> +       ret = parse_ports_from_fme(binfo);
> +       if (ret)
> +               goto free_binfo_exit;

So ideally, there would be a function call here that read all the
headers from hardware, ioremapping as it went along.  Then after that,
call the enumeration code to create the devices.

> +
> +       ret = build_info_commit_dev(binfo);
> +       if (ret)
> +               goto free_binfo_exit;
> +
> +       /*
> +        * everything is okay, reset ->parent_dev to stop it being
> +        * freed by build_info_free()
> +        */
> +       binfo->parent_dev = NULL;
> +
> +free_binfo_exit:
> +       build_info_free(binfo);
> +       return ret;
> +}
> +
>  /* PCI Device ID */
>  #define PCIe_DEVICE_ID_PF_INT_5_X      0xBCBD
>  #define PCIe_DEVICE_ID_PF_INT_6_X      0xBCC0
> @@ -81,9 +898,18 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
>                 goto release_region_exit;
>         }
>
> -       /* TODO: create and add the platform device per feature list */
> +       ret = create_init_drvdata(pcidev);
> +       if (ret)
> +               goto release_region_exit;
> +
> +       ret = cci_pci_create_feature_devs(pcidev);
> +       if (ret)
> +               goto destroy_drvdata_exit;
> +
>         return 0;
>
> +destroy_drvdata_exit:
> +       destroy_drvdata(pcidev);
>  release_region_exit:
>         pci_release_regions(pcidev);
>  disable_error_report_exit:
> @@ -94,6 +920,8 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
>
>  static void cci_pci_remove(struct pci_dev *pcidev)
>  {
> +       remove_all_devs(pcidev);
> +       destroy_drvdata(pcidev);
>         pci_release_regions(pcidev);
>         pci_disable_pcie_error_reporting(pcidev);
>         pci_disable_device(pcidev);
> @@ -108,14 +936,23 @@ static void cci_pci_remove(struct pci_dev *pcidev)
>
>  static int __init ccidrv_init(void)
>  {
> +       int ret;
> +
>         pr_info("Intel(R) FPGA PCIe Driver: Version %s\n", DRV_VERSION);
>
> -       return pci_register_driver(&cci_pci_driver);
> +       fpga_ids_init();
> +
> +       ret = pci_register_driver(&cci_pci_driver);
> +       if (ret)
> +               fpga_ids_destroy();
> +
> +       return ret;
>  }
>
>  static void __exit ccidrv_exit(void)
>  {
>         pci_unregister_driver(&cci_pci_driver);
> +       fpga_ids_destroy();
>  }
>
>  module_init(ccidrv_init);
> --
> 1.8.3.1
>

Re: [PATCH v2 03/22] fpga: bridge: remove OF dependency for fpga-bridge

[Moritz Fischer]

[-- Attachment #1: Type: text/plain, Size: 940 bytes --]

Hi,

On Mon, Jun 26, 2017 at 09:51:59AM +0800, Wu Hao wrote:
> This patch removes OF dependency of fpga-bridge, it allows drivers
> to use fpga-bridge class without device tree support.
> 
> Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> ---
>  drivers/fpga/Kconfig | 1 -
>  1 file changed, 1 deletion(-)
> 
> diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> index ed600d5..c1d8f41 100644
> --- a/drivers/fpga/Kconfig
> +++ b/drivers/fpga/Kconfig
> @@ -75,7 +75,6 @@ config FPGA_MGR_ZYNQ_FPGA
>  
>  config FPGA_BRIDGE
>  	tristate "FPGA Bridge Framework"
> -	depends on OF

Shouldn't that dependency be removed in the patch that removes the
dependency, or was the dependency never there in the first place?

>  	help
>  	  Say Y here if you want to support bridges connected between host
>  	  processors and FPGAs or between FPGAs.
> -- 
> 1.8.3.1
> 

Thanks,

Moritz

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v2 03/22] fpga: bridge: remove OF dependency for fpga-bridge

[Alan Tull]

On Thu, Sep 21, 2017 at 2:11 PM, Moritz Fischer <mdf@kernel.org> wrote:

Hi Moritz,

> Hi,
>
> On Mon, Jun 26, 2017 at 09:51:59AM +0800, Wu Hao wrote:
>> This patch removes OF dependency of fpga-bridge, it allows drivers
>> to use fpga-bridge class without device tree support.
>>
>> Signed-off-by: Wu Hao <hao.wu@intel.com>
>> ---
>>  drivers/fpga/Kconfig | 1 -
>>  1 file changed, 1 deletion(-)
>>
>> diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
>> index ed600d5..c1d8f41 100644
>> --- a/drivers/fpga/Kconfig
>> +++ b/drivers/fpga/Kconfig
>> @@ -75,7 +75,6 @@ config FPGA_MGR_ZYNQ_FPGA
>>
>>  config FPGA_BRIDGE
>>       tristate "FPGA Bridge Framework"
>> -     depends on OF
>
> Shouldn't that dependency be removed in the patch that removes the
> dependency, or was the dependency never there in the first place?

It wasn't dependent on OF in the first place.

Thanks,
Alan

>
>>       help
>>         Say Y here if you want to support bridges connected between host
>>         processors and FPGAs or between FPGAs.
>> --
>> 1.8.3.1
>>
>
> Thanks,
>
> Moritz

Re: [PATCH v2 07/22] fpga: intel: pcie: parse feature list and create platform device for features.

[Alan Tull]

)On Wed, Sep 20, 2017 at 4:24 PM, Alan Tull <atull@kernel.org> wrote:

Hi Hao,

A few more minor things below.

>  a (wh., *()On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu@intel.com> wrote:
>
> Hi Hao,
>
> I'm done with some board bringup so I have time to look at your patchset again.
>
> Something I can't help but notice is that this patchset will be almost
> completely reusable for embedded FPGAs if the enumeration is separated
> from the pcie code.  After the devices are created, they are just mmio
> devices.  That makes this whole scheme available for embedded FPGAs.
>
> The division of things would be that the pcie code would read the
> headers and do ioremapping.  Then pass the headers to the enumeration
> code to create the devices.  With that division, another platform that
> is embedded could have its own code that read the headers and did the
> iomapping and then used the separate enumeration code.
>
> Going through intel-pcie.c, the truly pcie specific parts are already
> pretty well contained.  The pcie code is in probe,
> cci_pci_ioremap_bar, cci_pci_release_regions, and the 'parse_s*'
> functions that ultimately call cci_pci_ioremap_bar.  So it all comes
> down to the code that calls the 'parse_s*' functions.  That is, the
> code the reads the fme header and the port headers.  Currently, if I'm
> reading this right, this code ioremaps each region, reads its header,
> and creates its device before reading the next header.  So we just
> need to change the order a little and ioremap/read all the headers
> before starting to create devices.  That separates the pci ioremap
> from creating the devices.  So the probe function could ioremap and
> read in the fme header.  Then it would ioremap and read in each of the
> port headers.  After they are read in, then pass the all these headers
> to the separate enumeration code that will parse it and create the
> devices.  That part is not pcie specific, right?  With the enumeration
> code in a separate file (and assuming that the ioremap has already
> happened),  another platform that is embedded could have its probe do
> the iomapping and reading the headers, and then call the same
> enumeration code.
>
> Actually, there's a bit more to it than that but I think this is
> doable.  build_info_add_sub_feature() (which is called by
> create_feature_instance) saves off resource info for each sub device.
> Does this mean this code is ioremapping again (in
> intel-fpga-fme-mgr.c's probe)?
>
> A few more things below...
>
>> From: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>>
>> Device Feature List structure creates a link list of feature headers
>> within the MMIO space to provide an extensible way of adding features.
>>
>> The Intel FPGA PCIe driver walks through the feature headers to enumerate
>> feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
>> Function Unit (AFU), and their private sub features. For feature devices,
>> it creates the platform devices and linked the private sub features into
>> their platform data.
>>
>> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
>> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
>> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
>> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
>> Signed-off-by: Kang Luwei <luwei.kang@intel.com>
>> Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
>> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> Signed-off-by: Wu Hao <hao.wu@intel.com>
>> ---
>> v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
>>     switched to GPLv2 license.
>>     fixed comments from Moritz Fischer.
>>     fixed kbuild warning, typos and clean up the code.
>> ---
>>  drivers/fpga/Makefile            |   2 +-
>>  drivers/fpga/intel-feature-dev.c | 130 ++++++
>>  drivers/fpga/intel-feature-dev.h | 341 ++++++++++++++++
>>  drivers/fpga/intel-pcie.c        | 841 ++++++++++++++++++++++++++++++++++++++-
>>  4 files changed, 1311 insertions(+), 3 deletions(-)
>>  create mode 100644 drivers/fpga/intel-feature-dev.c
>>  create mode 100644 drivers/fpga/intel-feature-dev.h
>>
>> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
>> index 5613133..ad24b3d 100644
>> --- a/drivers/fpga/Makefile
>> +++ b/drivers/fpga/Makefile
>> @@ -31,4 +31,4 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
>>  # Intel FPGA Support
>>  obj-$(CONFIG_INTEL_FPGA_PCI)           += intel-fpga-pci.o
>>
>> -intel-fpga-pci-objs := intel-pcie.o
>> +intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
>> diff --git a/drivers/fpga/intel-feature-dev.c b/drivers/fpga/intel-feature-dev.c
>> new file mode 100644
>> index 0000000..68f9cba
>> --- /dev/null
>> +++ b/drivers/fpga/intel-feature-dev.c
>> @@ -0,0 +1,130 @@
>> +/*
>> + * Intel FPGA Feature Device Driver
>> + *
>> + * Copyright (C) 2017 Intel Corporation, Inc.
>> + *
>> + * Authors:
>> + *   Kang Luwei <luwei.kang@intel.com>
>> + *   Zhang Yi <yi.z.zhang@intel.com>
>> + *   Wu Hao <hao.wu@intel.com>
>> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> + *
>> + * This work is licensed under the terms of the GNU GPL version 2. See
>> + * the COPYING file in the top-level directory.
>> + */
>> +
>> +#include "intel-feature-dev.h"
>> +
>> +void feature_platform_data_add(struct feature_platform_data *pdata,
>> +                              int index, const char *name,
>> +                              int resource_index, void __iomem *ioaddr)
>> +{
>> +       WARN_ON(index >= pdata->num);
>> +
>> +       pdata->features[index].name = name;
>> +       pdata->features[index].resource_index = resource_index;
>> +       pdata->features[index].ioaddr = ioaddr;
>> +}

This function is only called once.  I understand that it is desirable
to break things up into little functions and avoid overly long
functions, but in this case, the calling function
(build_info_add_sub_feature) is pretty short to begin with.   Someone
reading the code will know what is happening better if you just add
its contents to its calling function.

>> +
>> +struct feature_platform_data *
>> +feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
>> +{
>> +       struct feature_platform_data *pdata;
>> +
>> +       pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
>> +       if (pdata) {
>> +               pdata->dev = dev;
>> +               pdata->num = num;
>> +               mutex_init(&pdata->lock);
>> +       }
>> +
>> +       return pdata;
>> +}

feature_platform_data_alloc_and_init is another function that's small,
called once, and would be better included in its caller.

>> +
>> +int fme_feature_num(void)
>> +{
>> +       return FME_FEATURE_ID_MAX;
>> +}
>> +
>> +int port_feature_num(void)
>> +{
>> +       return PORT_FEATURE_ID_MAX;
>> +}
>
> Do these need to be functions?  If so, static.  But if you can just
> use the #define'd values that's one level of indirection we can get
> rid of.  And when I'm going through code where there's extra levels of
> indirection that don't do anything, it makes it harder to understand
> ;)
>
>> +
>> +int fpga_port_id(struct platform_device *pdev)
>> +{
>> +       struct feature_port_header *port_hdr;
>> +       struct feature_port_capability capability;
>> +
>> +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
>> +                                              PORT_FEATURE_ID_HEADER);
>> +       WARN_ON(!port_hdr);
>> +
>> +       capability.csr = readq(&port_hdr->capability);
>> +       return capability.port_number;
>> +}
>> +EXPORT_SYMBOL_GPL(fpga_port_id);
>
> This is here because every feature is a port, right?
>
>> +
>> +/*
>> + * Enable Port by clear the port soft reset bit, which is set by default.
>> + * The User AFU is unable to respond to any MMIO access while in reset.
>> + * __fpga_port_enable function should only be used after __fpga_port_disable
>> + * function.
>> + */
>> +void __fpga_port_enable(struct platform_device *pdev)
>> +{
>> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> +       struct feature_port_header *port_hdr;
>> +       struct feature_port_control control;
>> +
>> +       WARN_ON(!pdata->disable_count);
>> +
>> +       if (--pdata->disable_count != 0)
>> +               return;
>> +
>> +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
>> +                                              PORT_FEATURE_ID_HEADER);
>> +       WARN_ON(!port_hdr);
>> +
>> +       control.csr = readq(&port_hdr->control);
>> +       control.port_sftrst = 0x0;
>> +       writeq(control.csr, &port_hdr->control);
>> +}
>> +EXPORT_SYMBOL_GPL(__fpga_port_enable);
>> +
>> +#define RST_POLL_INVL 10 /* us */
>> +#define RST_POLL_TIMEOUT 1000 /* us */
>> +
>> +int __fpga_port_disable(struct platform_device *pdev)
>> +{
>> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> +       struct feature_port_header *port_hdr;
>> +       struct feature_port_control control;
>> +
>> +       if (pdata->disable_count++ != 0)
>> +               return 0;
>> +
>> +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
>> +                                              PORT_FEATURE_ID_HEADER);
>> +       WARN_ON(!port_hdr);
>> +
>> +       /* Set port soft reset */
>> +       control.csr = readq(&port_hdr->control);
>> +       control.port_sftrst = 0x1;
>> +       writeq(control.csr, &port_hdr->control);
>> +
>> +       /*
>> +        * HW sets ack bit to 1 when all outstanding requests have been drained
>> +        * on this port and minimum soft reset pulse width has elapsed.
>> +        * Driver polls port_soft_reset_ack to determine if reset done by HW.
>> +        */
>> +       if (readq_poll_timeout(&port_hdr->control, control.csr,
>> +                              (control.port_sftrst_ack == 1),
>> +                              RST_POLL_INVL, RST_POLL_TIMEOUT)) {
>> +               dev_err(&pdev->dev, "timeout, fail to reset device\n");
>> +               return -ETIMEDOUT;
>> +       }
>> +
>> +       return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(__fpga_port_disable);
>> diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
>> new file mode 100644
>> index 0000000..f67784a
>> --- /dev/null
>> +++ b/drivers/fpga/intel-feature-dev.h
>> @@ -0,0 +1,341 @@
>> +/*
>> + * Intel FPGA Feature Device Driver Header File
>> + *
>> + * Copyright (C) 2017 Intel Corporation, Inc.
>> + *
>> + * Authors:
>> + *   Kang Luwei <luwei.kang@intel.com>
>> + *   Zhang Yi <yi.z.zhang@intel.com>
>> + *   Wu Hao <hao.wu@intel.com>
>> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> + *
>> + * This work is licensed under the terms of the GNU GPL version 2. See
>> + * the COPYING file in the top-level directory.
>> + */
>> +
>> +#ifndef __INTEL_FPGA_FEATURE_H
>> +#define __INTEL_FPGA_FEATURE_H
>> +
>> +#include <linux/fs.h>
>> +#include <linux/pci.h>
>> +#include <linux/uuid.h>
>> +#include <linux/delay.h>
>> +#include <linux/iopoll.h>
>> +#include <linux/platform_device.h>
>> +
>> +#ifndef readq
>> +static inline u64 readq(void __iomem *addr)
>> +{
>> +       return readl(addr) + ((u64)readl(addr + 4) << 32);
>> +}
>> +#endif
>> +
>> +#ifndef writeq
>> +static inline void writeq(u64 val, void __iomem *addr)
>> +{
>> +       writel((u32) (val), addr);
>> +       writel((u32) (val >> 32), (addr + 4));
>> +}
>> +#endif
>> +
>> +/* maximum supported number of ports */
>> +#define MAX_FPGA_PORT_NUM 4
>> +/* plus one for fme device */
>> +#define MAX_FEATURE_DEV_NUM    (MAX_FPGA_PORT_NUM + 1)
>> +
>> +#define FME_FEATURE_HEADER          "fme_hdr"
>> +#define FME_FEATURE_THERMAL_MGMT    "fme_thermal"
>> +#define FME_FEATURE_POWER_MGMT      "fme_power"
>> +#define FME_FEATURE_GLOBAL_PERF     "fme_gperf"
>> +#define FME_FEATURE_GLOBAL_ERR      "fme_error"
>> +#define FME_FEATURE_PR_MGMT         "fme_pr"
>> +
>> +#define PORT_FEATURE_HEADER         "port_hdr"
>> +#define PORT_FEATURE_UAFU           "port_uafu"
>> +#define PORT_FEATURE_ERR            "port_err"
>> +#define PORT_FEATURE_UMSG           "port_umsg"
>> +#define PORT_FEATURE_PR             "port_pr"
>> +#define PORT_FEATURE_STP            "port_stp"
>> +
>> +/* All headers and structures must be byte-packed to match the spec. */
>> +#pragma pack(1)
>> +
>> +/* common header for all features */
>> +struct feature_header {
>> +       union {
>> +               u64 csr;
>> +               struct {
>> +                       u64 id:12;
>> +                       u64 revision:4;
>> +                       u64 next_header_offset:24; /* offset to next header */
>> +                       u64 rsvdz:20;
>> +                       u64 type:4;                /* feature type */
>> +#define FEATURE_TYPE_AFU               0x1
>> +#define FEATURE_TYPE_PRIVATE           0x3
>> +               };
>> +       };
>> +};
>> +
>> +/* common header for non-private features */
>> +struct feature_afu_header {
>> +       uuid_le guid;
>> +       union {
>> +               u64 csr;
>> +               struct {
>> +                       u64 next_afu:24;        /* pointer to next afu header */
>> +                       u64 rsvdz:40;
>> +               };
>> +       };
>> +};
>> +
>> +/* FME Header Register Set */
>> +/* FME Capability Register */
>> +struct feature_fme_capability {
>> +       union {
>> +               u64 csr;
>> +               struct {
>> +                       u64 fabric_verid:8;     /* Fabric version ID */
>> +                       u64 socket_id:1;        /* Socket id */
>> +                       u64 rsvdz1:3;
>> +                       u64 pcie0_link_avl:1;   /* PCIe0 link availability */
>> +                       u64 pcie1_link_avl:1;   /* PCIe1 link availability */
>> +                       u64 coherent_link_avl:1;/* Coherent link availability */
>> +                       u64 rsvdz2:1;
>> +                       u64 iommu_support:1;    /* IOMMU or VT-d supported */
>> +                       u64 num_ports:3;        /* Num of ports implemented */
>> +                       u64 rsvdz3:4;
>> +                       u64 addr_width_bits:6;  /* Address width supported */
>> +                       u64 rsvdz4:2;
>> +                       u64 cache_size:12;      /* Cache size in kb */
>> +                       u64 cache_assoc:4;      /* Cache Associativity */
>> +                       u64 rsvdz5:15;
>> +                       u64 lock_bit:1;         /* Latched lock bit by BIOS */
>> +               };
>> +       };
>> +};
>> +
>> +/* FME Port Offset Register */
>> +struct feature_fme_port {
>> +       union {
>> +               u64 csr;
>> +               struct {
>> +                       u64 port_offset:24;     /* Offset to port header */
>> +                       u64 rsvdz1:8;
>> +                       u64 port_bar:3;         /* Bar id */
>> +                       u64 rsvdz2:20;
>> +                       u64 afu_access_ctrl:1;  /* AFU access type: PF/VF */
>> +                       u64 rsvdz3:4;
>> +                       u64 port_implemented:1; /* Port implemented or not */
>> +                       u64 rsvdz4:3;
>> +               };
>> +       };
>> +};
>> +
>> +struct feature_fme_header {
>> +       struct feature_header header;
>> +       struct feature_afu_header afu_header;
>> +       u64 rsvd[2];
>> +       struct feature_fme_capability capability;
>> +       struct feature_fme_port port[MAX_FPGA_PORT_NUM];
>> +};
>> +
>> +/* FME Thermal Sub Feature Register Set */
>> +struct feature_fme_thermal {
>> +       struct feature_header header;
>> +};
>> +
>> +/* FME Power Sub Feature Register Set */
>> +struct feature_fme_power {
>> +       struct feature_header header;
>> +};
>> +
>> +/* FME Global Performance Sub Feature Register Set */
>> +struct feature_fme_gperf {
>> +       struct feature_header header;
>> +};
>> +
>> +/* FME Error Sub Feature Register Set */
>> +struct feature_fme_err {
>> +       struct feature_header header;
>> +};
>> +
>> +/* FME Partial Reconfiguration Sub Feature Register Set */
>> +struct feature_fme_pr {
>> +       struct feature_header header;
>> +};
>> +
>> +/* PORT Header Register Set */
>> +/* Port Capability Register */
>> +struct feature_port_capability {
>> +       union {
>> +               u64 csr;
>> +               struct {
>> +                       u64 port_number:2;      /* Port Number 0-3 */
>> +                       u64 rsvdz1:6;
>> +                       u64 mmio_size:16;       /* User MMIO size in KB */
>> +                       u64 rsvdz2:8;
>> +                       u64 sp_intr_num:4;      /* Supported interrupts num */
>> +                       u64 rsvdz3:28;
>> +               };
>> +       };
>> +};
>> +
>> +/* Port Control Register */
>> +struct feature_port_control {
>> +       union {
>> +               u64 csr;
>> +               struct {
>> +                       u64 port_sftrst:1;      /* Port Soft Reset */
>> +                       u64 rsvdz1:1;
>> +                       u64 latency_tolerance:1;/* '1' >= 40us, '0' < 40us */
>> +                       u64 rsvdz2:1;
>> +                       u64 port_sftrst_ack:1;  /* HW ACK for Soft Reset */
>> +                       u64 rsvdz3:59;
>> +               };
>> +       };
>> +};
>> +
>> +struct feature_port_header {
>> +       struct feature_header header;
>> +       struct feature_afu_header afu_header;
>> +       u64 rsvd[2];
>> +       struct feature_port_capability capability;
>> +       struct feature_port_control control;
>> +};
>> +
>> +/* PORT Error Sub Feature Register Set */
>> +struct feature_port_error {
>> +       struct feature_header header;
>> +};
>> +
>> +/* PORT Unordered Message Sub Feature Register Set */
>> +struct feature_port_umsg {
>> +       struct feature_header header;
>> +};
>> +
>> +/* PORT SignalTap Sub Feature Register Set */
>> +struct feature_port_stp {
>> +       struct feature_header header;
>> +};
>> +
>> +#pragma pack()
>> +
>> +struct feature {
>> +       const char *name;
>> +       int resource_index;
>> +       void __iomem *ioaddr;
>> +};
>> +
>> +struct feature_platform_data {
>> +       /* list the feature dev to cci_drvdata->port_dev_list. */
>> +       struct list_head node;
>> +       struct mutex lock;
>> +       struct platform_device *dev;
>> +       unsigned int disable_count;     /* count for port disable */
>> +
>> +       int num;                        /* number of features */
>> +       struct feature features[0];
>> +};
>> +
>> +enum fme_feature_id {
>> +       FME_FEATURE_ID_HEADER = 0x0,
>> +       FME_FEATURE_ID_THERMAL_MGMT = 0x1,
>> +       FME_FEATURE_ID_POWER_MGMT = 0x2,
>> +       FME_FEATURE_ID_GLOBAL_PERF = 0x3,
>> +       FME_FEATURE_ID_GLOBAL_ERR = 0x4,
>> +       FME_FEATURE_ID_PR_MGMT = 0x5,
>> +       FME_FEATURE_ID_MAX = 0x6,
>> +};
>> +
>> +enum port_feature_id {
>> +       PORT_FEATURE_ID_HEADER = 0x0,
>> +       PORT_FEATURE_ID_ERROR = 0x1,
>> +       PORT_FEATURE_ID_UMSG = 0x2,
>> +       PORT_FEATURE_ID_PR = 0x3,
>> +       PORT_FEATURE_ID_STP = 0x4,
>> +       PORT_FEATURE_ID_UAFU = 0x5,
>> +       PORT_FEATURE_ID_MAX = 0x6,
>> +};
>> +
>> +int fme_feature_num(void);
>> +int port_feature_num(void);
>> +
>> +#define FPGA_FEATURE_DEV_FME           "intel-fpga-fme"
>> +#define FPGA_FEATURE_DEV_PORT          "intel-fpga-port"
>> +
>> +void feature_platform_data_add(struct feature_platform_data *pdata,
>> +                              int index, const char *name,
>> +                              int resource_index, void __iomem *ioaddr);
>> +
>> +static inline int feature_platform_data_size(const int num)
>> +{
>> +       return sizeof(struct feature_platform_data) +
>> +               num * sizeof(struct feature);
>> +}
>> +
>> +struct feature_platform_data *
>> +feature_platform_data_alloc_and_init(struct platform_device *dev, int num);
>> +
>> +int fpga_port_id(struct platform_device *pdev);
>> +
>> +static inline int fpga_port_check_id(struct platform_device *pdev,
>> +                                    void *pport_id)
>> +{
>> +       return fpga_port_id(pdev) == *(int *)pport_id;
>> +}
>> +
>> +void __fpga_port_enable(struct platform_device *pdev);
>> +int __fpga_port_disable(struct platform_device *pdev);
>> +
>> +static inline void fpga_port_enable(struct platform_device *pdev)
>> +{
>> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> +
>> +       mutex_lock(&pdata->lock);
>> +       __fpga_port_enable(pdev);
>> +       mutex_unlock(&pdata->lock);
>> +}
>> +
>> +static inline int fpga_port_disable(struct platform_device *pdev)
>> +{
>> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> +       int ret;
>> +
>> +       mutex_lock(&pdata->lock);
>> +       ret = __fpga_port_disable(pdev);
>> +       mutex_unlock(&pdata->lock);
>> +
>> +       return ret;
>> +}
>> +
>> +static inline int __fpga_port_reset(struct platform_device *pdev)
>> +{
>> +       int ret;
>> +
>> +       ret = __fpga_port_disable(pdev);
>> +       if (ret)
>> +               return ret;
>> +
>> +       __fpga_port_enable(pdev);
>> +       return 0;
>> +}
>> +
>> +static inline int fpga_port_reset(struct platform_device *pdev)
>> +{
>> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> +       int ret;
>> +
>> +       mutex_lock(&pdata->lock);
>> +       ret = __fpga_port_reset(pdev);
>> +       mutex_unlock(&pdata->lock);
>> +       return ret;
>> +}
>> +
>> +static inline void __iomem *
>> +get_feature_ioaddr_by_index(struct device *dev, int index)
>> +{
>> +       struct feature_platform_data *pdata = dev_get_platdata(dev);
>> +
>> +       return pdata->features[index].ioaddr;
>> +}
>> +#endif
>> diff --git a/drivers/fpga/intel-pcie.c b/drivers/fpga/intel-pcie.c
>> index f697de4..70b8284 100644
>> --- a/drivers/fpga/intel-pcie.c
>> +++ b/drivers/fpga/intel-pcie.c
>> @@ -23,10 +23,827 @@
>>  #include <linux/stddef.h>
>>  #include <linux/errno.h>
>>  #include <linux/aer.h>
>> +#include <linux/fpga/fpga-dev.h>
>> +
>> +#include "intel-feature-dev.h"
>>
>>  #define DRV_VERSION    "0.8"
>>  #define DRV_NAME       "intel-fpga-pci"
>>
>> +#define INTEL_FPGA_DEV "intel-fpga-dev"
>> +
>> +static DEFINE_MUTEX(fpga_id_mutex);
>> +
>> +enum fpga_id_type {
>> +       FME_ID,         /* fme id allocation and mapping */
>> +       PORT_ID,        /* port id allocation and mapping */
>> +       FPGA_ID_MAX,
>> +};
>> +
>> +/* it is protected by fpga_id_mutex */
>> +static struct idr fpga_ids[FPGA_ID_MAX];
>> +
>> +struct cci_drvdata {
>> +       struct device *fme_dev;
>> +
>> +       struct mutex lock;
>> +       struct list_head port_dev_list;
>> +
>> +       struct list_head regions; /* global list of pci bar mapping region */
>> +};
>> +
>> +/* pci bar mapping info */
>> +struct cci_pci_region {
>> +       int bar;
>> +       void __iomem *ioaddr;   /* pointer to mapped bar region */
>> +       struct list_head node;
>> +};
>> +
>> +static void fpga_ids_init(void)
>> +{
>> +       int i;
>> +
>> +       for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
>> +               idr_init(fpga_ids + i);
>> +}
>> +
>> +static void fpga_ids_destroy(void)
>> +{
>> +       int i;
>> +
>> +       for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
>> +               idr_destroy(fpga_ids + i);
>> +}
>> +
>> +static int alloc_fpga_id(enum fpga_id_type type, struct device *dev)
>> +{
>> +       int id;
>> +
>> +       WARN_ON(type >= FPGA_ID_MAX);
>> +       mutex_lock(&fpga_id_mutex);
>> +       id = idr_alloc(fpga_ids + type, dev, 0, 0, GFP_KERNEL);
>> +       mutex_unlock(&fpga_id_mutex);
>> +       return id;
>> +}
>> +
>> +static void free_fpga_id(enum fpga_id_type type, int id)
>> +{
>> +       WARN_ON(type >= FPGA_ID_MAX);
>> +       mutex_lock(&fpga_id_mutex);
>> +       idr_remove(fpga_ids + type, id);
>> +       mutex_unlock(&fpga_id_mutex);
>> +}
>> +
>> +static void cci_pci_add_port_dev(struct pci_dev *pdev,
>> +                                struct platform_device *port_dev)
>> +{
>> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
>> +       struct feature_platform_data *pdata = dev_get_platdata(&port_dev->dev);
>> +
>> +       mutex_lock(&drvdata->lock);
>> +       list_add(&pdata->node, &drvdata->port_dev_list);
>> +       get_device(&pdata->dev->dev);
>> +       mutex_unlock(&drvdata->lock);
>> +}
>> +
>> +static void cci_pci_remove_port_devs(struct pci_dev *pdev)
>> +{
>> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
>> +       struct feature_platform_data *pdata, *ptmp;
>> +
>> +       mutex_lock(&drvdata->lock);
>> +       list_for_each_entry_safe(pdata, ptmp, &drvdata->port_dev_list, node) {
>> +               struct platform_device *port_dev = pdata->dev;
>> +
>> +               /* the port should be unregistered first. */
>> +               WARN_ON(device_is_registered(&port_dev->dev));
>> +               list_del(&pdata->node);
>> +               free_fpga_id(PORT_ID, port_dev->id);
>> +               put_device(&port_dev->dev);
>> +       }
>> +       mutex_unlock(&drvdata->lock);
>> +}
>> +
>> +/* info collection during feature dev build. */
>> +struct build_feature_devs_info {
>> +       struct pci_dev *pdev;
>> +
>> +       /*
>> +        * PCI BAR mapping info. Parsing feature list starts from
>> +        * BAR 0 and switch to different BARs to parse Port
>> +        */
>> +       void __iomem *ioaddr;
>> +       void __iomem *ioend;
>> +       int current_bar;
>> +
>> +       /* points to FME header where the port offset is figured out. */
>> +       void __iomem *pfme_hdr;
>> +
>> +       /* the container device for all feature devices */
>> +       struct fpga_dev *parent_dev;
>> +
>> +       /* current feature device */
>> +       struct platform_device *feature_dev;
>> +};
>> +
>> +static void cci_pci_release_regions(struct pci_dev *pdev)
>> +{
>> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
>> +       struct cci_pci_region *tmp, *region;
>> +
>> +       list_for_each_entry_safe(region, tmp, &drvdata->regions, node) {
>> +               list_del(&region->node);
>> +               if (region->ioaddr)
>> +                       pci_iounmap(pdev, region->ioaddr);
>> +               devm_kfree(&pdev->dev, region);
>> +       }
>> +}
>> +
>> +static void __iomem *cci_pci_ioremap_bar(struct pci_dev *pdev, int bar)
>> +{
>> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
>> +       struct cci_pci_region *region;
>> +
>> +       list_for_each_entry(region, &drvdata->regions, node)
>> +               if (region->bar == bar) {
>> +                       dev_dbg(&pdev->dev, "BAR %d region exists\n", bar);
>> +                       return region->ioaddr;
>> +               }
>> +
>> +       region = devm_kzalloc(&pdev->dev, sizeof(*region), GFP_KERNEL);
>> +       if (!region)
>> +               return NULL;
>> +
>> +       region->bar = bar;
>> +       region->ioaddr = pci_ioremap_bar(pdev, bar);
>> +       if (!region->ioaddr) {
>> +               dev_err(&pdev->dev, "can't ioremap memory from BAR %d.\n", bar);
>> +               devm_kfree(&pdev->dev, region);
>> +               return NULL;
>> +       }
>> +
>> +       list_add(&region->node, &drvdata->regions);
>> +       return region->ioaddr;
>> +}
>> +
>> +static int parse_start_from(struct build_feature_devs_info *binfo, int bar)
>> +{
>> +       binfo->ioaddr = cci_pci_ioremap_bar(binfo->pdev, bar);
>> +       if (!binfo->ioaddr)
>> +               return -ENOMEM;
>> +
>> +       binfo->current_bar = bar;
>> +       binfo->ioend = binfo->ioaddr + pci_resource_len(binfo->pdev, bar);
>> +       return 0;
>> +}
>> +
>> +static int parse_start(struct build_feature_devs_info *binfo)
>> +{
>> +       /* fpga feature list starts from BAR 0 */
>> +       return parse_start_from(binfo, 0);
>> +}
>> +
>> +/* switch the memory mapping to BAR# @bar */
>> +static int parse_switch_to(struct build_feature_devs_info *binfo, int bar)
>> +{
>> +       return parse_start_from(binfo, bar);
>> +}
>
> parse_switch_to() and parse_start() are both one line wrappers for
> parse_start_from() and are only called once.  :-) Please just use
> parse_start_from and get rid of parse_start and parse_switch_to.  I
> don't think this code is all that super complicated.  But as I'm
> grepping through this code, I'm trying to see all the places that do
> the same thing; if I have to look for multiple function names, it
> takes longer to get what is going on here.
>
> Actually, cci_pci_ioremap_bar is only called in one place -
> parse_start_from - which really just adds two lines of code.
> parse_start_from could go away.  Just a suggestion.  The reason this
> comes up is that cci_pci_ioremap_bar is the more descriptive function
> name.  The parse_start/switch function names hide the biggest thing
> they do, which is ioremapping.  They don't actually do any parsing ;)
>
>> +
>> +static struct build_feature_devs_info *
>> +build_info_alloc_and_init(struct pci_dev *pdev)
>> +{
>> +       struct build_feature_devs_info *binfo;
>> +
>> +       binfo = devm_kzalloc(&pdev->dev, sizeof(*binfo), GFP_KERNEL);
>> +       if (binfo)
>> +               binfo->pdev = pdev;
>
> build_info_alloc_and_init() is only doing a devm_kzalloc and is only
> called once.  The code would be more readable if you just had the
> devm_kzalloc in the function that called this
> (cci_pci_create_feature_devs).
>
>> +
>> +       return binfo;
>> +}
>> +
>> +static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
>> +{
>> +       if (!strcmp(pdev->name, FPGA_FEATURE_DEV_FME))
>> +               return FME_ID;
>> +
>> +       if (!strcmp(pdev->name, FPGA_FEATURE_DEV_PORT))
>> +               return PORT_ID;
>> +
>> +       WARN_ON(1);
>> +       return FPGA_ID_MAX;
>> +}
>> +
>> +/*
>> + * register current feature device, it is called when we need to switch to
>> + * another feature parsing or we have parsed all features
>> + */
>> +static int build_info_commit_dev(struct build_feature_devs_info *binfo)
>> +{
>> +       int ret;
>> +
>> +       if (!binfo->feature_dev)
>> +               return 0;
>> +
>> +       ret = platform_device_add(binfo->feature_dev);
>> +       if (!ret) {
>> +               struct cci_drvdata *drvdata;
>> +
>> +               drvdata = dev_get_drvdata(&binfo->pdev->dev);
>> +               if (feature_dev_id_type(binfo->feature_dev) == PORT_ID)
>> +                       cci_pci_add_port_dev(binfo->pdev, binfo->feature_dev);
>> +               else
>> +                       drvdata->fme_dev = get_device(&binfo->feature_dev->dev);
>> +
>> +               /*
>> +                * reset it to avoid build_info_free() freeing their resource.
>> +                *
>> +                * The resource of successfully registered feature devices
>> +                * will be freed by platform_device_unregister(). See the
>> +                * comments in build_info_create_dev().
>> +                */
>> +               binfo->feature_dev = NULL;
>> +       }
>> +
>> +       return ret;
>> +}
>> +
>> +static int
>> +build_info_create_dev(struct build_feature_devs_info *binfo,
>> +                     enum fpga_id_type type, int feature_nr, const char *name)
>> +{
>> +       struct platform_device *fdev;
>> +       struct resource *res;
>> +       struct feature_platform_data *pdata;
>> +       int ret;
>> +
>> +       /* we will create a new device, commit current device first */
>> +       ret = build_info_commit_dev(binfo);
>> +       if (ret)
>> +               return ret;
>> +
>> +       /*
>> +        * we use -ENODEV as the initialization indicator which indicates
>> +        * whether the id need to be reclaimed
>> +        */
>> +       fdev = binfo->feature_dev = platform_device_alloc(name, -ENODEV);
>> +       if (!fdev)
>> +               return -ENOMEM;
>> +
>> +       fdev->id = alloc_fpga_id(type, &fdev->dev);
>> +       if (fdev->id < 0)
>> +               return fdev->id;
>> +
>> +       fdev->dev.parent = &binfo->parent_dev->dev;
>> +
>> +       /*
>> +        * we do not need to care for the memory which is associated with
>> +        * the platform device. After calling platform_device_unregister(),
>> +        * it will be automatically freed by device's release() callback,
>> +        * platform_device_release().
>> +        */
>> +       pdata = feature_platform_data_alloc_and_init(fdev, feature_nr);
>> +       if (!pdata)
>> +               return -ENOMEM;
>> +
>> +       /*
>> +        * the count should be initialized to 0 to make sure
>> +        *__fpga_port_enable() following __fpga_port_disable()
>> +        * works properly for port device.
>> +        * and it should always be 0 for fme device.
>> +        */
>> +       WARN_ON(pdata->disable_count);
>> +
>> +       fdev->dev.platform_data = pdata;
>> +       fdev->num_resources = feature_nr;
>> +       fdev->resource = kcalloc(feature_nr, sizeof(*res), GFP_KERNEL);
>> +       if (!fdev->resource)
>> +               return -ENOMEM;
>> +
>> +       return 0;
>> +}
>> +
>> +static int remove_feature_dev(struct device *dev, void *data)
>> +{
>> +       struct platform_device *pdev = to_platform_device(dev);
>> +
>> +       platform_device_unregister(pdev);
>> +       return 0;
>> +}
>> +
>> +static int remove_parent_dev(struct device *dev, void *data)
>> +{
>> +       /* remove platform devices attached in the parent device */
>> +       device_for_each_child(dev, NULL, remove_feature_dev);
>> +       fpga_dev_destroy(to_fpga_dev(dev));
>> +       return 0;
>> +}
>> +
>> +static void remove_all_devs(struct pci_dev *pdev)
>> +{
>> +       /* remove parent device and all its children. */
>> +       device_for_each_child(&pdev->dev, NULL, remove_parent_dev);
>> +}
>> +
>> +static void build_info_free(struct build_feature_devs_info *binfo)
>> +{
>> +       if (!IS_ERR_OR_NULL(binfo->parent_dev))
>> +               remove_all_devs(binfo->pdev);
>> +
>> +       /*
>> +        * it is a valid id, free it. See comments in
>> +        * build_info_create_dev()
>> +        */
>> +       if (binfo->feature_dev && binfo->feature_dev->id >= 0)
>> +               free_fpga_id(feature_dev_id_type(binfo->feature_dev),
>> +                            binfo->feature_dev->id);
>> +
>> +       platform_device_put(binfo->feature_dev);
>> +
>> +       devm_kfree(&binfo->pdev->dev, binfo);
>> +}
>> +
>> +#define FEATURE_TYPE_AFU       0x1
>> +#define FEATURE_TYPE_PRIVATE   0x3
>> +
>> +/* FME and PORT GUID are fixed */
>> +#define FEATURE_FME_GUID "f9e17764-38f0-82fe-e346-524ae92aafbf"
>> +#define FEATURE_PORT_GUID "6b355b87-b06c-9642-eb42-8d139398b43a"
>> +
>> +static bool feature_is_fme(struct feature_afu_header *afu_hdr)
>> +{
>> +       uuid_le u;
>> +
>> +       uuid_le_to_bin(FEATURE_FME_GUID, &u);
>> +
>> +       return !uuid_le_cmp(u, afu_hdr->guid);
>> +}
>> +
>> +static bool feature_is_port(struct feature_afu_header *afu_hdr)
>> +{
>> +       uuid_le u;
>> +
>> +       uuid_le_to_bin(FEATURE_PORT_GUID, &u);
>> +
>> +       return !uuid_le_cmp(u, afu_hdr->guid);
>> +}
>> +
>> +/*
>> + * UAFU GUID is dynamic as it can be changed after FME downloads different
>> + * Green Bitstream to the port, so we treat the unknown GUIDs which are
>> + * attached on port's feature list as UAFU.
>> + */
>> +static bool feature_is_UAFU(struct build_feature_devs_info *binfo)
>> +{
>> +       if (!binfo->feature_dev ||
>> +             feature_dev_id_type(binfo->feature_dev) != PORT_ID)
>> +               return false;
>> +
>> +       return true;
>> +}
>> +
>> +static void
>> +build_info_add_sub_feature(struct build_feature_devs_info *binfo,
>> +                          int feature_id, const char *feature_name,
>> +                          resource_size_t resource_size, void __iomem *start)
>> +{
>> +
>> +       struct platform_device *fdev = binfo->feature_dev;
>> +       struct feature_platform_data *pdata = dev_get_platdata(&fdev->dev);
>> +       struct resource *res = &fdev->resource[feature_id];
>> +
>> +       res->start = pci_resource_start(binfo->pdev, binfo->current_bar) +
>> +               start - binfo->ioaddr;
>> +       res->end = res->start + resource_size - 1;
>> +       res->flags = IORESOURCE_MEM;
>> +       res->name = feature_name;
>> +
>> +       feature_platform_data_add(pdata, feature_id,
>> +                                 feature_name, feature_id, start);
>> +}
>> +
>> +struct feature_info {
>> +       const char *name;
>> +       resource_size_t resource_size;
>> +       int feature_index;
>> +};
>> +
>> +/* indexed by fme feature IDs which are defined in 'enum fme_feature_id'. */
>> +static struct feature_info fme_features[] = {
>> +       {
>> +               .name = FME_FEATURE_HEADER,
>> +               .resource_size = sizeof(struct feature_fme_header),
>> +               .feature_index = FME_FEATURE_ID_HEADER,
>> +       },
>> +       {
>> +               .name = FME_FEATURE_THERMAL_MGMT,
>> +               .resource_size = sizeof(struct feature_fme_thermal),
>> +               .feature_index = FME_FEATURE_ID_THERMAL_MGMT,
>> +       },
>> +       {
>> +               .name = FME_FEATURE_POWER_MGMT,
>> +               .resource_size = sizeof(struct feature_fme_power),
>> +               .feature_index = FME_FEATURE_ID_POWER_MGMT,
>> +       },
>> +       {
>> +               .name = FME_FEATURE_GLOBAL_PERF,
>> +               .resource_size = sizeof(struct feature_fme_gperf),
>> +               .feature_index = FME_FEATURE_ID_GLOBAL_PERF,
>> +       },
>> +       {
>> +               .name = FME_FEATURE_GLOBAL_ERR,
>> +               .resource_size = sizeof(struct feature_fme_err),
>> +               .feature_index = FME_FEATURE_ID_GLOBAL_ERR,
>> +       },
>> +       {
>> +               .name = FME_FEATURE_PR_MGMT,
>> +               .resource_size = sizeof(struct feature_fme_pr),
>> +               .feature_index = FME_FEATURE_ID_PR_MGMT,
>> +       }
>> +};
>> +
>> +/* indexed by port feature IDs which are defined in 'enum port_feature_id'. */
>> +static struct feature_info port_features[] = {
>> +       {
>> +               .name = PORT_FEATURE_HEADER,
>> +               .resource_size = sizeof(struct feature_port_header),
>> +               .feature_index = PORT_FEATURE_ID_HEADER,
>> +       },
>> +       {
>> +               .name = PORT_FEATURE_ERR,
>> +               .resource_size = sizeof(struct feature_port_error),
>> +               .feature_index = PORT_FEATURE_ID_ERROR,
>> +       },
>> +       {
>> +               .name = PORT_FEATURE_UMSG,
>> +               .resource_size = sizeof(struct feature_port_umsg),
>> +               .feature_index = PORT_FEATURE_ID_UMSG,
>> +       },
>> +       {
>> +               /* This feature isn't available for now */
>> +               .name = PORT_FEATURE_PR,
>> +               .resource_size = 0,
>> +               .feature_index = PORT_FEATURE_ID_PR,
>> +       },
>> +       {
>> +               .name = PORT_FEATURE_STP,
>> +               .resource_size = sizeof(struct feature_port_stp),
>> +               .feature_index = PORT_FEATURE_ID_STP,
>> +       },
>> +       {
>> +               /*
>> +                * For User AFU feature, its region size is not fixed, but
>> +                * reported by register PortCapability.mmio_size. Resource
>> +                * size of UAFU will be set while parse port device.
>> +                */
>> +               .name = PORT_FEATURE_UAFU,
>> +               .resource_size = 0,
>> +               .feature_index = PORT_FEATURE_ID_UAFU,
>> +       },
>> +};
>> +
>> +static int
>> +create_feature_instance(struct build_feature_devs_info *binfo,
>> +                       void __iomem *start, struct feature_info *finfo)
>> +{
>> +       if (binfo->ioend - start < finfo->resource_size)
>> +               return -EINVAL;
>> +
>> +       build_info_add_sub_feature(binfo, finfo->feature_index, finfo->name,
>> +                                  finfo->resource_size, start);
>> +       return 0;
>> +}
>
> build_info_add_sub_feature() is only called one time - here.  Could
> you collapse these two functions together?
>
>> +
>> +static int parse_feature_fme(struct build_feature_devs_info *binfo,
>> +                            void __iomem *start)
>> +{
>> +       struct cci_drvdata *drvdata = dev_get_drvdata(&binfo->pdev->dev);
>> +       int ret;
>> +
>> +       ret = build_info_create_dev(binfo, FME_ID, fme_feature_num(),
>> +                                       FPGA_FEATURE_DEV_FME);
>> +       if (ret)
>> +               return ret;
>> +
>> +       if (drvdata->fme_dev) {
>> +               dev_err(&binfo->pdev->dev, "Multiple FMEs are detected.\n");
>> +               return -EINVAL;
>> +       }
>> +
>> +       return create_feature_instance(binfo, start,
>> +                                      &fme_features[FME_FEATURE_ID_HEADER]);
>> +}
>> +
>> +static int parse_feature_fme_private(struct build_feature_devs_info *binfo,
>> +                                    struct feature_header *hdr)
>> +{
>> +       struct feature_header header;
>> +
>> +       header.csr = readq(hdr);
>> +
>> +       if (header.id >= ARRAY_SIZE(fme_features)) {
>> +               dev_info(&binfo->pdev->dev, "FME feature id %x is not supported yet.\n",
>> +                        header.id);
>> +               return 0;
>> +       }
>> +
>> +       return create_feature_instance(binfo, hdr, &fme_features[header.id]);
>> +}
>> +
>> +static int parse_feature_port(struct build_feature_devs_info *binfo,
>> +                            void __iomem *start)
>> +{
>> +       int ret;
>> +
>> +       ret = build_info_create_dev(binfo, PORT_ID, port_feature_num(),
>> +                                       FPGA_FEATURE_DEV_PORT);
>> +       if (ret)
>> +               return ret;
>> +
>> +       return create_feature_instance(binfo, start,
>> +                                      &port_features[PORT_FEATURE_ID_HEADER]);
>> +}
>> +
>> +static void enable_port_uafu(struct build_feature_devs_info *binfo,
>> +                            void __iomem *start)
>> +{
>> +       enum port_feature_id id = PORT_FEATURE_ID_UAFU;
>> +       struct feature_port_header *port_hdr;
>> +       struct feature_port_capability capability;
>> +
>> +       port_hdr = (struct feature_port_header *)start;
>> +       capability.csr = readq(&port_hdr->capability);
>> +       port_features[id].resource_size = capability.mmio_size << 10;
>> +
>> +       /*
>> +        * To enable User AFU, driver needs to clear reset bit on related port,
>> +        * otherwise the mmio space of this user AFU will be invalid.
>> +        */
>> +       if (port_features[id].resource_size)
>> +               fpga_port_reset(binfo->feature_dev);
>> +}
>> +
>> +static int parse_feature_port_private(struct build_feature_devs_info *binfo,
>> +                                     struct feature_header *hdr)
>> +{
>> +       struct feature_header header;
>> +       enum port_feature_id id;
>> +
>> +       header.csr = readq(hdr);
>> +       /*
>> +        * the region of port feature id is [0x10, 0x13], + 1 to reserve 0
>> +        * which is dedicated for port-hdr.
>> +        */
>> +       id = (header.id & 0x000f) + 1;
>> +
>> +       if (id >= ARRAY_SIZE(port_features)) {
>> +               dev_info(&binfo->pdev->dev, "Port feature id %x is not supported yet.\n",
>> +                        header.id);
>> +               return 0;
>> +       }
>> +
>> +       return create_feature_instance(binfo, hdr, &port_features[id]);
>> +}
>> +
>> +static int parse_feature_port_uafu(struct build_feature_devs_info *binfo,
>> +                                struct feature_header *hdr)
>> +{
>> +       enum port_feature_id id = PORT_FEATURE_ID_UAFU;
>> +       int ret;
>> +
>> +       if (port_features[id].resource_size) {
>> +               ret = create_feature_instance(binfo, hdr, &port_features[id]);
>> +               port_features[id].resource_size = 0;
>> +       } else {
>> +               dev_err(&binfo->pdev->dev, "the uafu feature header is mis-configured.\n");
>> +               ret = -EINVAL;
>> +       }
>> +
>> +       return ret;
>> +}
>> +
>> +static int parse_feature_afus(struct build_feature_devs_info *binfo,
>> +                             struct feature_header *hdr)
>> +{
>> +       int ret;
>> +       struct feature_afu_header *afu_hdr, header;
>> +       void __iomem *start;
>> +       void __iomem *end = binfo->ioend;
>> +
>> +       start = hdr;
>> +       for (; start < end; start += header.next_afu) {
>> +               if (end - start < (sizeof(*afu_hdr) + sizeof(*hdr)))
>> +                       return -EINVAL;
>> +
>> +               hdr = start;
>> +               afu_hdr = (struct feature_afu_header *) (hdr + 1);
>> +               header.csr = readq(&afu_hdr->csr);
>> +
>> +               if (feature_is_fme(afu_hdr)) {
>> +                       ret = parse_feature_fme(binfo, hdr);
>> +                       binfo->pfme_hdr = hdr;
>> +                       if (ret)
>> +                               return ret;
>> +               } else if (feature_is_port(afu_hdr)) {
>> +                       ret = parse_feature_port(binfo, hdr);
>> +                       enable_port_uafu(binfo, hdr);
>> +                       if (ret)
>> +                               return ret;
>> +               } else if (feature_is_UAFU(binfo)) {
>> +                       ret = parse_feature_port_uafu(binfo, hdr);
>> +                       if (ret)
>> +                               return ret;
>> +               } else
>> +                       dev_info(&binfo->pdev->dev, "AFU GUID %pUl is not supported yet.\n",
>> +                                afu_hdr->guid.b);
>> +
>> +               if (!header.next_afu)
>> +                       break;
>> +       }
>> +
>> +       return 0;
>> +}
>> +
>> +static int parse_feature_private(struct build_feature_devs_info *binfo,
>> +                                struct feature_header *hdr)
>> +{
>> +       struct feature_header header;
>> +
>> +       header.csr = readq(hdr);
>> +
>> +       if (!binfo->feature_dev) {
>> +               dev_err(&binfo->pdev->dev, "the private feature %x does not belong to any AFU.\n",
>> +                       header.id);
>> +               return -EINVAL;
>> +       }
>> +
>> +       switch (feature_dev_id_type(binfo->feature_dev)) {
>> +       case FME_ID:
>> +               return parse_feature_fme_private(binfo, hdr);
>> +       case PORT_ID:
>> +               return parse_feature_port_private(binfo, hdr);
>> +       default:
>> +               dev_info(&binfo->pdev->dev, "private feature %x belonging to AFU %s is not supported yet.\n",
>> +                        header.id, binfo->feature_dev->name);
>> +       }
>> +       return 0;
>> +}
>> +
>> +static int parse_feature(struct build_feature_devs_info *binfo,
>> +                        struct feature_header *hdr)
>> +{
>> +       struct feature_header header;
>> +       int ret = 0;
>> +
>> +       header.csr = readq(hdr);
>> +
>> +       switch (header.type) {
>> +       case FEATURE_TYPE_AFU:
>> +               ret = parse_feature_afus(binfo, hdr);
>> +               break;
>> +       case FEATURE_TYPE_PRIVATE:
>> +               ret = parse_feature_private(binfo, hdr);
>> +               break;
>> +       default:
>> +               dev_info(&binfo->pdev->dev,
>> +                        "Feature Type %x is not supported.\n", hdr->type);
>> +       };
>> +
>> +       return ret;
>> +}
>> +
>> +static int
>> +parse_feature_list(struct build_feature_devs_info *binfo, void __iomem *start)
>> +{
>> +       struct feature_header *hdr, header;
>> +       void __iomem *end = binfo->ioend;
>> +       int ret = 0;
>> +
>
> Maybe a helpful comment that we are stepping through a linked list of features.
>
>> +       for (; start < end; start += header.next_header_offset) {
>> +               if (end - start < sizeof(*hdr)) {
>> +                       dev_err(&binfo->pdev->dev, "The region is too small to contain a feature.\n");
>> +                       ret =  -EINVAL;
>> +                       break;
>> +               }
>> +
>> +               hdr = (struct feature_header *)start;
>> +               ret = parse_feature(binfo, hdr);
>> +               if (ret)
>> +                       break;
>
> Instead of parse_feature, this can save off the enumeration info and
> continue to read in the linked list.  After all the headers are read
> in, call the (separate) enumeration code to step through the saved
> headers, parse them, and create the devices.  Since the memory is
> iomapped during the process of reading in the headers, the enumeration
> code doesn't have to be so pcie specific.  Then this code base is
> better set to run on embedded devices also.
>
>> +
>> +               header.csr = readq(hdr);
>> +               if (!header.next_header_offset)
>> +                       break;
>> +       }
>> +
>> +       return ret;
>> +}
>> +
>> +static int parse_ports_from_fme(struct build_feature_devs_info *binfo)
>> +{
>> +       struct feature_fme_header *fme_hdr;
>> +       struct feature_fme_port port;
>> +       int i = 0, ret = 0;
>> +
>> +       if (binfo->pfme_hdr == NULL) {
>> +               dev_dbg(&binfo->pdev->dev, "VF is detected.\n");
>> +               return ret;
>> +       }
>> +
>> +       fme_hdr = binfo->pfme_hdr;
>> +
>> +       do {
>> +               port.csr = readq(&fme_hdr->port[i]);
>> +               if (!port.port_implemented)
>> +                       break;
>> +
>> +               ret = parse_switch_to(binfo, port.port_bar);
>> +               if (ret)
>> +                       break;
>> +
>> +               ret = parse_feature_list(binfo,
>> +                               binfo->ioaddr + port.port_offset);
>> +               if (ret)
>> +                       break;
>> +       } while (++i < MAX_FPGA_PORT_NUM);
>> +
>> +       return ret;
>> +}
>> +
>> +static int create_init_drvdata(struct pci_dev *pdev)
>> +{
>> +       struct cci_drvdata *drvdata;
>> +
>> +       drvdata = devm_kzalloc(&pdev->dev, sizeof(*drvdata), GFP_KERNEL);
>> +       if (!drvdata)
>> +               return -ENOMEM;
>> +
>> +       mutex_init(&drvdata->lock);
>> +       INIT_LIST_HEAD(&drvdata->port_dev_list);
>> +       INIT_LIST_HEAD(&drvdata->regions);
>> +
>> +       dev_set_drvdata(&pdev->dev, drvdata);
>> +       return 0;
>> +}
>> +
>> +static void destroy_drvdata(struct pci_dev *pdev)
>> +{
>> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
>> +
>> +       if (drvdata->fme_dev) {
>> +               /* fme device should be unregistered first. */
>> +               WARN_ON(device_is_registered(drvdata->fme_dev));
>> +               free_fpga_id(FME_ID, to_platform_device(drvdata->fme_dev)->id);
>> +               put_device(drvdata->fme_dev);
>> +       }
>> +
>> +       cci_pci_remove_port_devs(pdev);
>> +       cci_pci_release_regions(pdev);
>> +       dev_set_drvdata(&pdev->dev, NULL);
>> +       devm_kfree(&pdev->dev, drvdata);
>> +}
>> +
>> +static int cci_pci_create_feature_devs(struct pci_dev *pdev)
>> +{
>> +       struct build_feature_devs_info *binfo;
>> +       int ret;
>> +
>> +       binfo = build_info_alloc_and_init(pdev);
>> +       if (!binfo)
>> +               return -ENOMEM;
>> +
>> +       binfo->parent_dev = fpga_dev_create(&pdev->dev, INTEL_FPGA_DEV);
>> +       if (IS_ERR(binfo->parent_dev)) {
>> +               ret = PTR_ERR(binfo->parent_dev);
>> +               goto free_binfo_exit;
>> +       }
>> +
>> +       ret = parse_start(binfo);
>> +       if (ret)
>> +               goto free_binfo_exit;
>> +
>> +       ret = parse_feature_list(binfo, binfo->ioaddr);
>> +       if (ret)
>> +               goto free_binfo_exit;
>> +
>> +       ret = parse_ports_from_fme(binfo);
>> +       if (ret)
>> +               goto free_binfo_exit;
>
> So ideally, there would be a function call here that read all the
> headers from hardware, ioremapping as it went along.  Then after that,
> call the enumeration code to create the devices.
>
>> +
>> +       ret = build_info_commit_dev(binfo);
>> +       if (ret)
>> +               goto free_binfo_exit;
>> +
>> +       /*
>> +        * everything is okay, reset ->parent_dev to stop it being
>> +        * freed by build_info_free()
>> +        */
>> +       binfo->parent_dev = NULL;
>> +
>> +free_binfo_exit:
>> +       build_info_free(binfo);
>> +       return ret;
>> +}
>> +
>>  /* PCI Device ID */
>>  #define PCIe_DEVICE_ID_PF_INT_5_X      0xBCBD
>>  #define PCIe_DEVICE_ID_PF_INT_6_X      0xBCC0
>> @@ -81,9 +898,18 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
>>                 goto release_region_exit;
>>         }
>>
>> -       /* TODO: create and add the platform device per feature list */
>> +       ret = create_init_drvdata(pcidev);
>> +       if (ret)
>> +               goto release_region_exit;
>> +
>> +       ret = cci_pci_create_feature_devs(pcidev);
>> +       if (ret)
>> +               goto destroy_drvdata_exit;
>> +
>>         return 0;
>>
>> +destroy_drvdata_exit:
>> +       destroy_drvdata(pcidev);
>>  release_region_exit:
>>         pci_release_regions(pcidev);
>>  disable_error_report_exit:
>> @@ -94,6 +920,8 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
>>
>>  static void cci_pci_remove(struct pci_dev *pcidev)
>>  {
>> +       remove_all_devs(pcidev);
>> +       destroy_drvdata(pcidev);
>>         pci_release_regions(pcidev);
>>         pci_disable_pcie_error_reporting(pcidev);
>>         pci_disable_device(pcidev);
>> @@ -108,14 +936,23 @@ static void cci_pci_remove(struct pci_dev *pcidev)
>>
>>  static int __init ccidrv_init(void)
>>  {
>> +       int ret;
>> +
>>         pr_info("Intel(R) FPGA PCIe Driver: Version %s\n", DRV_VERSION);
>>
>> -       return pci_register_driver(&cci_pci_driver);
>> +       fpga_ids_init();
>> +
>> +       ret = pci_register_driver(&cci_pci_driver);
>> +       if (ret)
>> +               fpga_ids_destroy();
>> +
>> +       return ret;
>>  }
>>
>>  static void __exit ccidrv_exit(void)
>>  {
>>         pci_unregister_driver(&cci_pci_driver);
>> +       fpga_ids_destroy();
>>  }
>>
>>  module_init(ccidrv_init);
>> --
>> 1.8.3.1
>>

Re: [PATCH v2 03/22] fpga: bridge: remove OF dependency for fpga-bridge

[Wu Hao]

On Thu, Sep 21, 2017 at 02:50:07PM -0500, Alan Tull wrote:
> On Thu, Sep 21, 2017 at 2:11 PM, Moritz Fischer <mdf-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> 
> Hi Moritz,
> 
> > Hi,
> >
> > On Mon, Jun 26, 2017 at 09:51:59AM +0800, Wu Hao wrote:
> >> This patch removes OF dependency of fpga-bridge, it allows drivers
> >> to use fpga-bridge class without device tree support.
> >>
> >> Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> ---
> >>  drivers/fpga/Kconfig | 1 -
> >>  1 file changed, 1 deletion(-)
> >>
> >> diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> >> index ed600d5..c1d8f41 100644
> >> --- a/drivers/fpga/Kconfig
> >> +++ b/drivers/fpga/Kconfig
> >> @@ -75,7 +75,6 @@ config FPGA_MGR_ZYNQ_FPGA
> >>
> >>  config FPGA_BRIDGE
> >>       tristate "FPGA Bridge Framework"
> >> -     depends on OF
> >
> > Shouldn't that dependency be removed in the patch that removes the
> > dependency, or was the dependency never there in the first place?
> 
> It wasn't dependent on OF in the first place.

Hi Alan / Moritz

I see this change has been integrated into below patch from Alan.

[PATCH v4 15/18] fpga: region: move device tree support to of-fpga-region.c

so I think I can drop this patch in my next version patch set. : )

Thanks
Hao

> 
> Thanks,
> Alan
> 
> >
> >>       help
> >>         Say Y here if you want to support bridges connected between host
> >>         processors and FPGAs or between FPGAs.
> >> --
> >> 1.8.3.1
> >>
> >
> > Thanks,
> >
> > Moritz
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 07/22] fpga: intel: pcie: parse feature list and create platform device for features.

[Wu Hao]

On Wed, Sep 20, 2017 at 04:24:10PM -0500, Alan Tull wrote:
>  a (wh., *()On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu@intel.com> wrote:
> 
> Hi Hao,
> 
> I'm done with some board bringup so I have time to look at your patchset again.

Hi Alan

Thanks for your time on the review. : )

> 
> Something I can't help but notice is that this patchset will be almost
> completely reusable for embedded FPGAs if the enumeration is separated
> from the pcie code.  After the devices are created, they are just mmio
> devices.  That makes this whole scheme available for embedded FPGAs.
> 
> The division of things would be that the pcie code would read the
> headers and do ioremapping.  Then pass the headers to the enumeration
> code to create the devices.  With that division, another platform that
> is embedded could have its own code that read the headers and did the
> iomapping and then used the separate enumeration code.
> 
> Going through intel-pcie.c, the truly pcie specific parts are already
> pretty well contained.  The pcie code is in probe,
> cci_pci_ioremap_bar, cci_pci_release_regions, and the 'parse_s*'
> functions that ultimately call cci_pci_ioremap_bar.  So it all comes
> down to the code that calls the 'parse_s*' functions.  That is, the
> code the reads the fme header and the port headers.  Currently, if I'm
> reading this right, this code ioremaps each region, reads its header,
> and creates its device before reading the next header.  So we just
> need to change the order a little and ioremap/read all the headers
> before starting to create devices.  That separates the pci ioremap
> from creating the devices.  So the probe function could ioremap and
> read in the fme header.  Then it would ioremap and read in each of the
> port headers.  After they are read in, then pass the all these headers
> to the separate enumeration code that will parse it and create the
> devices.  That part is not pcie specific, right?  With the enumeration
> code in a separate file (and assuming that the ioremap has already
> happened),  another platform that is embedded could have its probe do
> the iomapping and reading the headers, and then call the same
> enumeration code.

So it's suggested that we have some kind of pre-enumeration code in the
intel-pcie.c to do ioremap and locate the address of device feature lists
for FME and Ports, and hand them to common enumeration code. Is my
understanding correct?

I have considered this some time ago, this is doable for current code,
as the pre-enumeration code will be simple (only need to find FME header
and then read some register in FME header register set to locate the Port
header). But if in the future, some pcie specific things are added to some
sub features of FME, then pre-enumeration code has to parse FME and all
its sub features to handle pcie specific things. We will have more
duplicate code on pre-enumeration in intel-pcie.c and common enumeration.
Another point is, it's hard to avoid pcie specific things in sub features,
as they may be designed to use PCIE specific function or resources.
e.g some sub features require the interrupts from PCIE device, like PCIE
MSI / MSIX. So keep enumeration code away from PCI things may be difficult.

> 
> Actually, there's a bit more to it than that but I think this is
> doable.  build_info_add_sub_feature() (which is called by
> create_feature_instance) saves off resource info for each sub device.
> Does this mean this code is ioremapping again (in
> intel-fpga-fme-mgr.c's probe)?

Yes, you are correct. Is it fine to have the MMIO region mapped twice?
or we have to pass the mapped address to intel-fpga-fme-mgr directly?

> 
> A few more things below...
> 
> > From: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> >
> > Device Feature List structure creates a link list of feature headers
> > within the MMIO space to provide an extensible way of adding features.
> >
> > The Intel FPGA PCIe driver walks through the feature headers to enumerate
> > feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
> > Function Unit (AFU), and their private sub features. For feature devices,
> > it creates the platform devices and linked the private sub features into
> > their platform data.
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> > Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > ---
> > v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
> >     switched to GPLv2 license.
> >     fixed comments from Moritz Fischer.
> >     fixed kbuild warning, typos and clean up the code.
> > ---
> >  drivers/fpga/Makefile            |   2 +-
> >  drivers/fpga/intel-feature-dev.c | 130 ++++++
> >  drivers/fpga/intel-feature-dev.h | 341 ++++++++++++++++
> >  drivers/fpga/intel-pcie.c        | 841 ++++++++++++++++++++++++++++++++++++++-
> >  4 files changed, 1311 insertions(+), 3 deletions(-)
> >  create mode 100644 drivers/fpga/intel-feature-dev.c
> >  create mode 100644 drivers/fpga/intel-feature-dev.h
> >
> > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> > index 5613133..ad24b3d 100644
> > --- a/drivers/fpga/Makefile
> > +++ b/drivers/fpga/Makefile
> > @@ -31,4 +31,4 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
> >  # Intel FPGA Support
> >  obj-$(CONFIG_INTEL_FPGA_PCI)           += intel-fpga-pci.o
> >
> > -intel-fpga-pci-objs := intel-pcie.o
> > +intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
> > diff --git a/drivers/fpga/intel-feature-dev.c b/drivers/fpga/intel-feature-dev.c
> > new file mode 100644
> > index 0000000..68f9cba
> > --- /dev/null
> > +++ b/drivers/fpga/intel-feature-dev.c
> > @@ -0,0 +1,130 @@
> > +/*
> > + * Intel FPGA Feature Device Driver
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <luwei.kang@intel.com>
> > + *   Zhang Yi <yi.z.zhang@intel.com>
> > + *   Wu Hao <hao.wu@intel.com>
> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > + *
> > + * This work is licensed under the terms of the GNU GPL version 2. See
> > + * the COPYING file in the top-level directory.
> > + */
> > +
> > +#include "intel-feature-dev.h"
> > +
> > +void feature_platform_data_add(struct feature_platform_data *pdata,
> > +                              int index, const char *name,
> > +                              int resource_index, void __iomem *ioaddr)
> > +{
> > +       WARN_ON(index >= pdata->num);
> > +
> > +       pdata->features[index].name = name;
> > +       pdata->features[index].resource_index = resource_index;
> > +       pdata->features[index].ioaddr = ioaddr;
> > +}
> > +
> > +struct feature_platform_data *
> > +feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
> > +{
> > +       struct feature_platform_data *pdata;
> > +
> > +       pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
> > +       if (pdata) {
> > +               pdata->dev = dev;
> > +               pdata->num = num;
> > +               mutex_init(&pdata->lock);
> > +       }
> > +
> > +       return pdata;
> > +}
> > +
> > +int fme_feature_num(void)
> > +{
> > +       return FME_FEATURE_ID_MAX;
> > +}
> > +
> > +int port_feature_num(void)
> > +{
> > +       return PORT_FEATURE_ID_MAX;
> > +}
> 
> Do these need to be functions?  If so, static.  But if you can just
> use the #define'd values that's one level of indirection we can get
> rid of.  And when I'm going through code where there's extra levels of
> indirection that don't do anything, it makes it harder to understand
> ;)

Sure, will fix this.

> 
> > +
> > +int fpga_port_id(struct platform_device *pdev)
> > +{
> > +       struct feature_port_header *port_hdr;
> > +       struct feature_port_capability capability;
> > +
> > +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                                              PORT_FEATURE_ID_HEADER);
> > +       WARN_ON(!port_hdr);
> > +
> > +       capability.csr = readq(&port_hdr->capability);
> > +       return capability.port_number;
> > +}
> > +EXPORT_SYMBOL_GPL(fpga_port_id);
> 
> This is here because every feature is a port, right?

No, this fpga_port_id function will be used in both afu/port driver and
also the pcie driver, so this function in added here.
The code in pcie driver which uses this function isn't submitted yet, it's
part of the SRIOV support. The pcie driver needs to manage which port could
be turned to VF.

> 
> > +
> > +/*
> > + * Enable Port by clear the port soft reset bit, which is set by default.
> > + * The User AFU is unable to respond to any MMIO access while in reset.
> > + * __fpga_port_enable function should only be used after __fpga_port_disable
> > + * function.
> > + */
> > +void __fpga_port_enable(struct platform_device *pdev)
> > +{
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct feature_port_header *port_hdr;
> > +       struct feature_port_control control;
> > +
> > +       WARN_ON(!pdata->disable_count);
> > +
> > +       if (--pdata->disable_count != 0)
> > +               return;
> > +
> > +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                                              PORT_FEATURE_ID_HEADER);
> > +       WARN_ON(!port_hdr);
> > +
> > +       control.csr = readq(&port_hdr->control);
> > +       control.port_sftrst = 0x0;
> > +       writeq(control.csr, &port_hdr->control);
> > +}
> > +EXPORT_SYMBOL_GPL(__fpga_port_enable);
> > +
> > +#define RST_POLL_INVL 10 /* us */
> > +#define RST_POLL_TIMEOUT 1000 /* us */
> > +
> > +int __fpga_port_disable(struct platform_device *pdev)
> > +{
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct feature_port_header *port_hdr;
> > +       struct feature_port_control control;
> > +
> > +       if (pdata->disable_count++ != 0)
> > +               return 0;
> > +
> > +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                                              PORT_FEATURE_ID_HEADER);
> > +       WARN_ON(!port_hdr);
> > +
> > +       /* Set port soft reset */
> > +       control.csr = readq(&port_hdr->control);
> > +       control.port_sftrst = 0x1;
> > +       writeq(control.csr, &port_hdr->control);
> > +
> > +       /*
> > +        * HW sets ack bit to 1 when all outstanding requests have been drained
> > +        * on this port and minimum soft reset pulse width has elapsed.
> > +        * Driver polls port_soft_reset_ack to determine if reset done by HW.
> > +        */
> > +       if (readq_poll_timeout(&port_hdr->control, control.csr,
> > +                              (control.port_sftrst_ack == 1),
> > +                              RST_POLL_INVL, RST_POLL_TIMEOUT)) {
> > +               dev_err(&pdev->dev, "timeout, fail to reset device\n");
> > +               return -ETIMEDOUT;
> > +       }
> > +
> > +       return 0;
> > +}
> > +EXPORT_SYMBOL_GPL(__fpga_port_disable);
> > diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
> > new file mode 100644
> > index 0000000..f67784a
> > --- /dev/null
> > +++ b/drivers/fpga/intel-feature-dev.h
> > @@ -0,0 +1,341 @@
> > +/*
> > + * Intel FPGA Feature Device Driver Header File
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <luwei.kang@intel.com>
> > + *   Zhang Yi <yi.z.zhang@intel.com>
> > + *   Wu Hao <hao.wu@intel.com>
> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > + *
> > + * This work is licensed under the terms of the GNU GPL version 2. See
> > + * the COPYING file in the top-level directory.
> > + */
> > +
> > +#ifndef __INTEL_FPGA_FEATURE_H
> > +#define __INTEL_FPGA_FEATURE_H
> > +
> > +#include <linux/fs.h>
> > +#include <linux/pci.h>
> > +#include <linux/uuid.h>
> > +#include <linux/delay.h>
> > +#include <linux/iopoll.h>
> > +#include <linux/platform_device.h>
> > +
> > +#ifndef readq
> > +static inline u64 readq(void __iomem *addr)
> > +{
> > +       return readl(addr) + ((u64)readl(addr + 4) << 32);
> > +}
> > +#endif
> > +
> > +#ifndef writeq
> > +static inline void writeq(u64 val, void __iomem *addr)
> > +{
> > +       writel((u32) (val), addr);
> > +       writel((u32) (val >> 32), (addr + 4));
> > +}
> > +#endif
> > +
> > +/* maximum supported number of ports */
> > +#define MAX_FPGA_PORT_NUM 4
> > +/* plus one for fme device */
> > +#define MAX_FEATURE_DEV_NUM    (MAX_FPGA_PORT_NUM + 1)
> > +
> > +#define FME_FEATURE_HEADER          "fme_hdr"
> > +#define FME_FEATURE_THERMAL_MGMT    "fme_thermal"
> > +#define FME_FEATURE_POWER_MGMT      "fme_power"
> > +#define FME_FEATURE_GLOBAL_PERF     "fme_gperf"
> > +#define FME_FEATURE_GLOBAL_ERR      "fme_error"
> > +#define FME_FEATURE_PR_MGMT         "fme_pr"
> > +
> > +#define PORT_FEATURE_HEADER         "port_hdr"
> > +#define PORT_FEATURE_UAFU           "port_uafu"
> > +#define PORT_FEATURE_ERR            "port_err"
> > +#define PORT_FEATURE_UMSG           "port_umsg"
> > +#define PORT_FEATURE_PR             "port_pr"
> > +#define PORT_FEATURE_STP            "port_stp"
> > +
> > +/* All headers and structures must be byte-packed to match the spec. */
> > +#pragma pack(1)
> > +
> > +/* common header for all features */
> > +struct feature_header {
> > +       union {
> > +               u64 csr;
> > +               struct {
> > +                       u64 id:12;
> > +                       u64 revision:4;
> > +                       u64 next_header_offset:24; /* offset to next header */
> > +                       u64 rsvdz:20;
> > +                       u64 type:4;                /* feature type */
> > +#define FEATURE_TYPE_AFU               0x1
> > +#define FEATURE_TYPE_PRIVATE           0x3
> > +               };
> > +       };
> > +};
> > +
> > +/* common header for non-private features */
> > +struct feature_afu_header {
> > +       uuid_le guid;
> > +       union {
> > +               u64 csr;
> > +               struct {
> > +                       u64 next_afu:24;        /* pointer to next afu header */
> > +                       u64 rsvdz:40;
> > +               };
> > +       };
> > +};
> > +
> > +/* FME Header Register Set */
> > +/* FME Capability Register */
> > +struct feature_fme_capability {
> > +       union {
> > +               u64 csr;
> > +               struct {
> > +                       u64 fabric_verid:8;     /* Fabric version ID */
> > +                       u64 socket_id:1;        /* Socket id */
> > +                       u64 rsvdz1:3;
> > +                       u64 pcie0_link_avl:1;   /* PCIe0 link availability */
> > +                       u64 pcie1_link_avl:1;   /* PCIe1 link availability */
> > +                       u64 coherent_link_avl:1;/* Coherent link availability */
> > +                       u64 rsvdz2:1;
> > +                       u64 iommu_support:1;    /* IOMMU or VT-d supported */
> > +                       u64 num_ports:3;        /* Num of ports implemented */
> > +                       u64 rsvdz3:4;
> > +                       u64 addr_width_bits:6;  /* Address width supported */
> > +                       u64 rsvdz4:2;
> > +                       u64 cache_size:12;      /* Cache size in kb */
> > +                       u64 cache_assoc:4;      /* Cache Associativity */
> > +                       u64 rsvdz5:15;
> > +                       u64 lock_bit:1;         /* Latched lock bit by BIOS */
> > +               };
> > +       };
> > +};
> > +
> > +/* FME Port Offset Register */
> > +struct feature_fme_port {
> > +       union {
> > +               u64 csr;
> > +               struct {
> > +                       u64 port_offset:24;     /* Offset to port header */
> > +                       u64 rsvdz1:8;
> > +                       u64 port_bar:3;         /* Bar id */
> > +                       u64 rsvdz2:20;
> > +                       u64 afu_access_ctrl:1;  /* AFU access type: PF/VF */
> > +                       u64 rsvdz3:4;
> > +                       u64 port_implemented:1; /* Port implemented or not */
> > +                       u64 rsvdz4:3;
> > +               };
> > +       };
> > +};
> > +
> > +struct feature_fme_header {
> > +       struct feature_header header;
> > +       struct feature_afu_header afu_header;
> > +       u64 rsvd[2];
> > +       struct feature_fme_capability capability;
> > +       struct feature_fme_port port[MAX_FPGA_PORT_NUM];
> > +};
> > +
> > +/* FME Thermal Sub Feature Register Set */
> > +struct feature_fme_thermal {
> > +       struct feature_header header;
> > +};
> > +
> > +/* FME Power Sub Feature Register Set */
> > +struct feature_fme_power {
> > +       struct feature_header header;
> > +};
> > +
> > +/* FME Global Performance Sub Feature Register Set */
> > +struct feature_fme_gperf {
> > +       struct feature_header header;
> > +};
> > +
> > +/* FME Error Sub Feature Register Set */
> > +struct feature_fme_err {
> > +       struct feature_header header;
> > +};
> > +
> > +/* FME Partial Reconfiguration Sub Feature Register Set */
> > +struct feature_fme_pr {
> > +       struct feature_header header;
> > +};
> > +
> > +/* PORT Header Register Set */
> > +/* Port Capability Register */
> > +struct feature_port_capability {
> > +       union {
> > +               u64 csr;
> > +               struct {
> > +                       u64 port_number:2;      /* Port Number 0-3 */
> > +                       u64 rsvdz1:6;
> > +                       u64 mmio_size:16;       /* User MMIO size in KB */
> > +                       u64 rsvdz2:8;
> > +                       u64 sp_intr_num:4;      /* Supported interrupts num */
> > +                       u64 rsvdz3:28;
> > +               };
> > +       };
> > +};
> > +
> > +/* Port Control Register */
> > +struct feature_port_control {
> > +       union {
> > +               u64 csr;
> > +               struct {
> > +                       u64 port_sftrst:1;      /* Port Soft Reset */
> > +                       u64 rsvdz1:1;
> > +                       u64 latency_tolerance:1;/* '1' >= 40us, '0' < 40us */
> > +                       u64 rsvdz2:1;
> > +                       u64 port_sftrst_ack:1;  /* HW ACK for Soft Reset */
> > +                       u64 rsvdz3:59;
> > +               };
> > +       };
> > +};
> > +
> > +struct feature_port_header {
> > +       struct feature_header header;
> > +       struct feature_afu_header afu_header;
> > +       u64 rsvd[2];
> > +       struct feature_port_capability capability;
> > +       struct feature_port_control control;
> > +};
> > +
> > +/* PORT Error Sub Feature Register Set */
> > +struct feature_port_error {
> > +       struct feature_header header;
> > +};
> > +
> > +/* PORT Unordered Message Sub Feature Register Set */
> > +struct feature_port_umsg {
> > +       struct feature_header header;
> > +};
> > +
> > +/* PORT SignalTap Sub Feature Register Set */
> > +struct feature_port_stp {
> > +       struct feature_header header;
> > +};
> > +
> > +#pragma pack()
> > +
> > +struct feature {
> > +       const char *name;
> > +       int resource_index;
> > +       void __iomem *ioaddr;
> > +};
> > +
> > +struct feature_platform_data {
> > +       /* list the feature dev to cci_drvdata->port_dev_list. */
> > +       struct list_head node;
> > +       struct mutex lock;
> > +       struct platform_device *dev;
> > +       unsigned int disable_count;     /* count for port disable */
> > +
> > +       int num;                        /* number of features */
> > +       struct feature features[0];
> > +};
> > +
> > +enum fme_feature_id {
> > +       FME_FEATURE_ID_HEADER = 0x0,
> > +       FME_FEATURE_ID_THERMAL_MGMT = 0x1,
> > +       FME_FEATURE_ID_POWER_MGMT = 0x2,
> > +       FME_FEATURE_ID_GLOBAL_PERF = 0x3,
> > +       FME_FEATURE_ID_GLOBAL_ERR = 0x4,
> > +       FME_FEATURE_ID_PR_MGMT = 0x5,
> > +       FME_FEATURE_ID_MAX = 0x6,
> > +};
> > +
> > +enum port_feature_id {
> > +       PORT_FEATURE_ID_HEADER = 0x0,
> > +       PORT_FEATURE_ID_ERROR = 0x1,
> > +       PORT_FEATURE_ID_UMSG = 0x2,
> > +       PORT_FEATURE_ID_PR = 0x3,
> > +       PORT_FEATURE_ID_STP = 0x4,
> > +       PORT_FEATURE_ID_UAFU = 0x5,
> > +       PORT_FEATURE_ID_MAX = 0x6,
> > +};
> > +
> > +int fme_feature_num(void);
> > +int port_feature_num(void);
> > +
> > +#define FPGA_FEATURE_DEV_FME           "intel-fpga-fme"
> > +#define FPGA_FEATURE_DEV_PORT          "intel-fpga-port"
> > +
> > +void feature_platform_data_add(struct feature_platform_data *pdata,
> > +                              int index, const char *name,
> > +                              int resource_index, void __iomem *ioaddr);
> > +
> > +static inline int feature_platform_data_size(const int num)
> > +{
> > +       return sizeof(struct feature_platform_data) +
> > +               num * sizeof(struct feature);
> > +}
> > +
> > +struct feature_platform_data *
> > +feature_platform_data_alloc_and_init(struct platform_device *dev, int num);
> > +
> > +int fpga_port_id(struct platform_device *pdev);
> > +
> > +static inline int fpga_port_check_id(struct platform_device *pdev,
> > +                                    void *pport_id)
> > +{
> > +       return fpga_port_id(pdev) == *(int *)pport_id;
> > +}
> > +
> > +void __fpga_port_enable(struct platform_device *pdev);
> > +int __fpga_port_disable(struct platform_device *pdev);
> > +
> > +static inline void fpga_port_enable(struct platform_device *pdev)
> > +{
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +
> > +       mutex_lock(&pdata->lock);
> > +       __fpga_port_enable(pdev);
> > +       mutex_unlock(&pdata->lock);
> > +}
> > +
> > +static inline int fpga_port_disable(struct platform_device *pdev)
> > +{
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       int ret;
> > +
> > +       mutex_lock(&pdata->lock);
> > +       ret = __fpga_port_disable(pdev);
> > +       mutex_unlock(&pdata->lock);
> > +
> > +       return ret;
> > +}
> > +
> > +static inline int __fpga_port_reset(struct platform_device *pdev)
> > +{
> > +       int ret;
> > +
> > +       ret = __fpga_port_disable(pdev);
> > +       if (ret)
> > +               return ret;
> > +
> > +       __fpga_port_enable(pdev);
> > +       return 0;
> > +}
> > +
> > +static inline int fpga_port_reset(struct platform_device *pdev)
> > +{
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       int ret;
> > +
> > +       mutex_lock(&pdata->lock);
> > +       ret = __fpga_port_reset(pdev);
> > +       mutex_unlock(&pdata->lock);
> > +       return ret;
> > +}
> > +
> > +static inline void __iomem *
> > +get_feature_ioaddr_by_index(struct device *dev, int index)
> > +{
> > +       struct feature_platform_data *pdata = dev_get_platdata(dev);
> > +
> > +       return pdata->features[index].ioaddr;
> > +}
> > +#endif
> > diff --git a/drivers/fpga/intel-pcie.c b/drivers/fpga/intel-pcie.c
> > index f697de4..70b8284 100644
> > --- a/drivers/fpga/intel-pcie.c
> > +++ b/drivers/fpga/intel-pcie.c
> > @@ -23,10 +23,827 @@
> >  #include <linux/stddef.h>
> >  #include <linux/errno.h>
> >  #include <linux/aer.h>
> > +#include <linux/fpga/fpga-dev.h>
> > +
> > +#include "intel-feature-dev.h"
> >
> >  #define DRV_VERSION    "0.8"
> >  #define DRV_NAME       "intel-fpga-pci"
> >
> > +#define INTEL_FPGA_DEV "intel-fpga-dev"
> > +
> > +static DEFINE_MUTEX(fpga_id_mutex);
> > +
> > +enum fpga_id_type {
> > +       FME_ID,         /* fme id allocation and mapping */
> > +       PORT_ID,        /* port id allocation and mapping */
> > +       FPGA_ID_MAX,
> > +};
> > +
> > +/* it is protected by fpga_id_mutex */
> > +static struct idr fpga_ids[FPGA_ID_MAX];
> > +
> > +struct cci_drvdata {
> > +       struct device *fme_dev;
> > +
> > +       struct mutex lock;
> > +       struct list_head port_dev_list;
> > +
> > +       struct list_head regions; /* global list of pci bar mapping region */
> > +};
> > +
> > +/* pci bar mapping info */
> > +struct cci_pci_region {
> > +       int bar;
> > +       void __iomem *ioaddr;   /* pointer to mapped bar region */
> > +       struct list_head node;
> > +};
> > +
> > +static void fpga_ids_init(void)
> > +{
> > +       int i;
> > +
> > +       for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> > +               idr_init(fpga_ids + i);
> > +}
> > +
> > +static void fpga_ids_destroy(void)
> > +{
> > +       int i;
> > +
> > +       for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> > +               idr_destroy(fpga_ids + i);
> > +}
> > +
> > +static int alloc_fpga_id(enum fpga_id_type type, struct device *dev)
> > +{
> > +       int id;
> > +
> > +       WARN_ON(type >= FPGA_ID_MAX);
> > +       mutex_lock(&fpga_id_mutex);
> > +       id = idr_alloc(fpga_ids + type, dev, 0, 0, GFP_KERNEL);
> > +       mutex_unlock(&fpga_id_mutex);
> > +       return id;
> > +}
> > +
> > +static void free_fpga_id(enum fpga_id_type type, int id)
> > +{
> > +       WARN_ON(type >= FPGA_ID_MAX);
> > +       mutex_lock(&fpga_id_mutex);
> > +       idr_remove(fpga_ids + type, id);
> > +       mutex_unlock(&fpga_id_mutex);
> > +}
> > +
> > +static void cci_pci_add_port_dev(struct pci_dev *pdev,
> > +                                struct platform_device *port_dev)
> > +{
> > +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> > +       struct feature_platform_data *pdata = dev_get_platdata(&port_dev->dev);
> > +
> > +       mutex_lock(&drvdata->lock);
> > +       list_add(&pdata->node, &drvdata->port_dev_list);
> > +       get_device(&pdata->dev->dev);
> > +       mutex_unlock(&drvdata->lock);
> > +}
> > +
> > +static void cci_pci_remove_port_devs(struct pci_dev *pdev)
> > +{
> > +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> > +       struct feature_platform_data *pdata, *ptmp;
> > +
> > +       mutex_lock(&drvdata->lock);
> > +       list_for_each_entry_safe(pdata, ptmp, &drvdata->port_dev_list, node) {
> > +               struct platform_device *port_dev = pdata->dev;
> > +
> > +               /* the port should be unregistered first. */
> > +               WARN_ON(device_is_registered(&port_dev->dev));
> > +               list_del(&pdata->node);
> > +               free_fpga_id(PORT_ID, port_dev->id);
> > +               put_device(&port_dev->dev);
> > +       }
> > +       mutex_unlock(&drvdata->lock);
> > +}
> > +
> > +/* info collection during feature dev build. */
> > +struct build_feature_devs_info {
> > +       struct pci_dev *pdev;
> > +
> > +       /*
> > +        * PCI BAR mapping info. Parsing feature list starts from
> > +        * BAR 0 and switch to different BARs to parse Port
> > +        */
> > +       void __iomem *ioaddr;
> > +       void __iomem *ioend;
> > +       int current_bar;
> > +
> > +       /* points to FME header where the port offset is figured out. */
> > +       void __iomem *pfme_hdr;
> > +
> > +       /* the container device for all feature devices */
> > +       struct fpga_dev *parent_dev;
> > +
> > +       /* current feature device */
> > +       struct platform_device *feature_dev;
> > +};
> > +
> > +static void cci_pci_release_regions(struct pci_dev *pdev)
> > +{
> > +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> > +       struct cci_pci_region *tmp, *region;
> > +
> > +       list_for_each_entry_safe(region, tmp, &drvdata->regions, node) {
> > +               list_del(&region->node);
> > +               if (region->ioaddr)
> > +                       pci_iounmap(pdev, region->ioaddr);
> > +               devm_kfree(&pdev->dev, region);
> > +       }
> > +}
> > +
> > +static void __iomem *cci_pci_ioremap_bar(struct pci_dev *pdev, int bar)
> > +{
> > +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> > +       struct cci_pci_region *region;
> > +
> > +       list_for_each_entry(region, &drvdata->regions, node)
> > +               if (region->bar == bar) {
> > +                       dev_dbg(&pdev->dev, "BAR %d region exists\n", bar);
> > +                       return region->ioaddr;
> > +               }
> > +
> > +       region = devm_kzalloc(&pdev->dev, sizeof(*region), GFP_KERNEL);
> > +       if (!region)
> > +               return NULL;
> > +
> > +       region->bar = bar;
> > +       region->ioaddr = pci_ioremap_bar(pdev, bar);
> > +       if (!region->ioaddr) {
> > +               dev_err(&pdev->dev, "can't ioremap memory from BAR %d.\n", bar);
> > +               devm_kfree(&pdev->dev, region);
> > +               return NULL;
> > +       }
> > +
> > +       list_add(&region->node, &drvdata->regions);
> > +       return region->ioaddr;
> > +}
> > +
> > +static int parse_start_from(struct build_feature_devs_info *binfo, int bar)
> > +{
> > +       binfo->ioaddr = cci_pci_ioremap_bar(binfo->pdev, bar);
> > +       if (!binfo->ioaddr)
> > +               return -ENOMEM;
> > +
> > +       binfo->current_bar = bar;
> > +       binfo->ioend = binfo->ioaddr + pci_resource_len(binfo->pdev, bar);
> > +       return 0;
> > +}
> > +
> > +static int parse_start(struct build_feature_devs_info *binfo)
> > +{
> > +       /* fpga feature list starts from BAR 0 */
> > +       return parse_start_from(binfo, 0);
> > +}
> > +
> > +/* switch the memory mapping to BAR# @bar */
> > +static int parse_switch_to(struct build_feature_devs_info *binfo, int bar)
> > +{
> > +       return parse_start_from(binfo, bar);
> > +}
> 
> parse_switch_to() and parse_start() are both one line wrappers for
> parse_start_from() and are only called once.  :-) Please just use
> parse_start_from and get rid of parse_start and parse_switch_to.  I
> don't think this code is all that super complicated.  But as I'm
> grepping through this code, I'm trying to see all the places that do
> the same thing; if I have to look for multiple function names, it
> takes longer to get what is going on here.
> 
> Actually, cci_pci_ioremap_bar is only called in one place -
> parse_start_from - which really just adds two lines of code.
> parse_start_from could go away.  Just a suggestion.  The reason this
> comes up is that cci_pci_ioremap_bar is the more descriptive function
> name.  The parse_start/switch function names hide the biggest thing
> they do, which is ioremapping.  They don't actually do any parsing ;)

I think I need to explain a little more on the enumeration steps in this
cci_pci_create_feature_devs function, and we use these names for these
functions.

1) parse_start -> that initializes the enumeration from PCI Bar0.
   In PF case, the first byte of BAR0, will be FME header.
   In VF case, the first byte of BAR0, will be Port header.
   (FME module is always in PF, no FME module in VF).

2) parse_feature_list -> parse one device feature list

   This function parses a feature list, it could be FME device feature
   list (PF) or Port device feature list (VF).
   FME device feature list and Port device feature list are not linked
   together, as they are in differnet BARs.

3) parse_port_from_fme -> parse Port device feature lists if FME presents.

   If FME is detected (PF), read FME registers to know the location (BAR n
   + offset) of each port's header (port device feature list).
   call parse_switch_to(BAR n) and call parse_feature_list to parse the
   port device feature list.

   If FME is not detected (VF), just do nothing.

Do you think if we could keep these functions? We use these names to
help people to understand the enumeration process better, but looks like
I should add more descriptions/comments here to avoid misunderstanding. :)

> 
> > +
> > +static struct build_feature_devs_info *
> > +build_info_alloc_and_init(struct pci_dev *pdev)
> > +{
> > +       struct build_feature_devs_info *binfo;
> > +
> > +       binfo = devm_kzalloc(&pdev->dev, sizeof(*binfo), GFP_KERNEL);
> > +       if (binfo)
> > +               binfo->pdev = pdev;
> 
> build_info_alloc_and_init() is only doing a devm_kzalloc and is only
> called once.  The code would be more readable if you just had the
> devm_kzalloc in the function that called this
> (cci_pci_create_feature_devs).

Sure.

> 
> > +
> > +       return binfo;
> > +}
> > +
> > +static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
> > +{
> > +       if (!strcmp(pdev->name, FPGA_FEATURE_DEV_FME))
> > +               return FME_ID;
> > +
> > +       if (!strcmp(pdev->name, FPGA_FEATURE_DEV_PORT))
> > +               return PORT_ID;
> > +
> > +       WARN_ON(1);
> > +       return FPGA_ID_MAX;
> > +}
> > +
> > +/*
> > + * register current feature device, it is called when we need to switch to
> > + * another feature parsing or we have parsed all features
> > + */
> > +static int build_info_commit_dev(struct build_feature_devs_info *binfo)
> > +{
> > +       int ret;
> > +
> > +       if (!binfo->feature_dev)
> > +               return 0;
> > +
> > +       ret = platform_device_add(binfo->feature_dev);
> > +       if (!ret) {
> > +               struct cci_drvdata *drvdata;
> > +
> > +               drvdata = dev_get_drvdata(&binfo->pdev->dev);
> > +               if (feature_dev_id_type(binfo->feature_dev) == PORT_ID)
> > +                       cci_pci_add_port_dev(binfo->pdev, binfo->feature_dev);
> > +               else
> > +                       drvdata->fme_dev = get_device(&binfo->feature_dev->dev);
> > +
> > +               /*
> > +                * reset it to avoid build_info_free() freeing their resource.
> > +                *
> > +                * The resource of successfully registered feature devices
> > +                * will be freed by platform_device_unregister(). See the
> > +                * comments in build_info_create_dev().
> > +                */
> > +               binfo->feature_dev = NULL;
> > +       }
> > +
> > +       return ret;
> > +}
> > +
> > +static int
> > +build_info_create_dev(struct build_feature_devs_info *binfo,
> > +                     enum fpga_id_type type, int feature_nr, const char *name)
> > +{
> > +       struct platform_device *fdev;
> > +       struct resource *res;
> > +       struct feature_platform_data *pdata;
> > +       int ret;
> > +
> > +       /* we will create a new device, commit current device first */
> > +       ret = build_info_commit_dev(binfo);
> > +       if (ret)
> > +               return ret;
> > +
> > +       /*
> > +        * we use -ENODEV as the initialization indicator which indicates
> > +        * whether the id need to be reclaimed
> > +        */
> > +       fdev = binfo->feature_dev = platform_device_alloc(name, -ENODEV);
> > +       if (!fdev)
> > +               return -ENOMEM;
> > +
> > +       fdev->id = alloc_fpga_id(type, &fdev->dev);
> > +       if (fdev->id < 0)
> > +               return fdev->id;
> > +
> > +       fdev->dev.parent = &binfo->parent_dev->dev;
> > +
> > +       /*
> > +        * we do not need to care for the memory which is associated with
> > +        * the platform device. After calling platform_device_unregister(),
> > +        * it will be automatically freed by device's release() callback,
> > +        * platform_device_release().
> > +        */
> > +       pdata = feature_platform_data_alloc_and_init(fdev, feature_nr);
> > +       if (!pdata)
> > +               return -ENOMEM;
> > +
> > +       /*
> > +        * the count should be initialized to 0 to make sure
> > +        *__fpga_port_enable() following __fpga_port_disable()
> > +        * works properly for port device.
> > +        * and it should always be 0 for fme device.
> > +        */
> > +       WARN_ON(pdata->disable_count);
> > +
> > +       fdev->dev.platform_data = pdata;
> > +       fdev->num_resources = feature_nr;
> > +       fdev->resource = kcalloc(feature_nr, sizeof(*res), GFP_KERNEL);
> > +       if (!fdev->resource)
> > +               return -ENOMEM;
> > +
> > +       return 0;
> > +}
> > +
> > +static int remove_feature_dev(struct device *dev, void *data)
> > +{
> > +       struct platform_device *pdev = to_platform_device(dev);
> > +
> > +       platform_device_unregister(pdev);
> > +       return 0;
> > +}
> > +
> > +static int remove_parent_dev(struct device *dev, void *data)
> > +{
> > +       /* remove platform devices attached in the parent device */
> > +       device_for_each_child(dev, NULL, remove_feature_dev);
> > +       fpga_dev_destroy(to_fpga_dev(dev));
> > +       return 0;
> > +}
> > +
> > +static void remove_all_devs(struct pci_dev *pdev)
> > +{
> > +       /* remove parent device and all its children. */
> > +       device_for_each_child(&pdev->dev, NULL, remove_parent_dev);
> > +}
> > +
> > +static void build_info_free(struct build_feature_devs_info *binfo)
> > +{
> > +       if (!IS_ERR_OR_NULL(binfo->parent_dev))
> > +               remove_all_devs(binfo->pdev);
> > +
> > +       /*
> > +        * it is a valid id, free it. See comments in
> > +        * build_info_create_dev()
> > +        */
> > +       if (binfo->feature_dev && binfo->feature_dev->id >= 0)
> > +               free_fpga_id(feature_dev_id_type(binfo->feature_dev),
> > +                            binfo->feature_dev->id);
> > +
> > +       platform_device_put(binfo->feature_dev);
> > +
> > +       devm_kfree(&binfo->pdev->dev, binfo);
> > +}
> > +
> > +#define FEATURE_TYPE_AFU       0x1
> > +#define FEATURE_TYPE_PRIVATE   0x3
> > +
> > +/* FME and PORT GUID are fixed */
> > +#define FEATURE_FME_GUID "f9e17764-38f0-82fe-e346-524ae92aafbf"
> > +#define FEATURE_PORT_GUID "6b355b87-b06c-9642-eb42-8d139398b43a"
> > +
> > +static bool feature_is_fme(struct feature_afu_header *afu_hdr)
> > +{
> > +       uuid_le u;
> > +
> > +       uuid_le_to_bin(FEATURE_FME_GUID, &u);
> > +
> > +       return !uuid_le_cmp(u, afu_hdr->guid);
> > +}
> > +
> > +static bool feature_is_port(struct feature_afu_header *afu_hdr)
> > +{
> > +       uuid_le u;
> > +
> > +       uuid_le_to_bin(FEATURE_PORT_GUID, &u);
> > +
> > +       return !uuid_le_cmp(u, afu_hdr->guid);
> > +}
> > +
> > +/*
> > + * UAFU GUID is dynamic as it can be changed after FME downloads different
> > + * Green Bitstream to the port, so we treat the unknown GUIDs which are
> > + * attached on port's feature list as UAFU.
> > + */
> > +static bool feature_is_UAFU(struct build_feature_devs_info *binfo)
> > +{
> > +       if (!binfo->feature_dev ||
> > +             feature_dev_id_type(binfo->feature_dev) != PORT_ID)
> > +               return false;
> > +
> > +       return true;
> > +}
> > +
> > +static void
> > +build_info_add_sub_feature(struct build_feature_devs_info *binfo,
> > +                          int feature_id, const char *feature_name,
> > +                          resource_size_t resource_size, void __iomem *start)
> > +{
> > +
> > +       struct platform_device *fdev = binfo->feature_dev;
> > +       struct feature_platform_data *pdata = dev_get_platdata(&fdev->dev);
> > +       struct resource *res = &fdev->resource[feature_id];
> > +
> > +       res->start = pci_resource_start(binfo->pdev, binfo->current_bar) +
> > +               start - binfo->ioaddr;
> > +       res->end = res->start + resource_size - 1;
> > +       res->flags = IORESOURCE_MEM;
> > +       res->name = feature_name;
> > +
> > +       feature_platform_data_add(pdata, feature_id,
> > +                                 feature_name, feature_id, start);
> > +}
> > +
> > +struct feature_info {
> > +       const char *name;
> > +       resource_size_t resource_size;
> > +       int feature_index;
> > +};
> > +
> > +/* indexed by fme feature IDs which are defined in 'enum fme_feature_id'. */
> > +static struct feature_info fme_features[] = {
> > +       {
> > +               .name = FME_FEATURE_HEADER,
> > +               .resource_size = sizeof(struct feature_fme_header),
> > +               .feature_index = FME_FEATURE_ID_HEADER,
> > +       },
> > +       {
> > +               .name = FME_FEATURE_THERMAL_MGMT,
> > +               .resource_size = sizeof(struct feature_fme_thermal),
> > +               .feature_index = FME_FEATURE_ID_THERMAL_MGMT,
> > +       },
> > +       {
> > +               .name = FME_FEATURE_POWER_MGMT,
> > +               .resource_size = sizeof(struct feature_fme_power),
> > +               .feature_index = FME_FEATURE_ID_POWER_MGMT,
> > +       },
> > +       {
> > +               .name = FME_FEATURE_GLOBAL_PERF,
> > +               .resource_size = sizeof(struct feature_fme_gperf),
> > +               .feature_index = FME_FEATURE_ID_GLOBAL_PERF,
> > +       },
> > +       {
> > +               .name = FME_FEATURE_GLOBAL_ERR,
> > +               .resource_size = sizeof(struct feature_fme_err),
> > +               .feature_index = FME_FEATURE_ID_GLOBAL_ERR,
> > +       },
> > +       {
> > +               .name = FME_FEATURE_PR_MGMT,
> > +               .resource_size = sizeof(struct feature_fme_pr),
> > +               .feature_index = FME_FEATURE_ID_PR_MGMT,
> > +       }
> > +};
> > +
> > +/* indexed by port feature IDs which are defined in 'enum port_feature_id'. */
> > +static struct feature_info port_features[] = {
> > +       {
> > +               .name = PORT_FEATURE_HEADER,
> > +               .resource_size = sizeof(struct feature_port_header),
> > +               .feature_index = PORT_FEATURE_ID_HEADER,
> > +       },
> > +       {
> > +               .name = PORT_FEATURE_ERR,
> > +               .resource_size = sizeof(struct feature_port_error),
> > +               .feature_index = PORT_FEATURE_ID_ERROR,
> > +       },
> > +       {
> > +               .name = PORT_FEATURE_UMSG,
> > +               .resource_size = sizeof(struct feature_port_umsg),
> > +               .feature_index = PORT_FEATURE_ID_UMSG,
> > +       },
> > +       {
> > +               /* This feature isn't available for now */
> > +               .name = PORT_FEATURE_PR,
> > +               .resource_size = 0,
> > +               .feature_index = PORT_FEATURE_ID_PR,
> > +       },
> > +       {
> > +               .name = PORT_FEATURE_STP,
> > +               .resource_size = sizeof(struct feature_port_stp),
> > +               .feature_index = PORT_FEATURE_ID_STP,
> > +       },
> > +       {
> > +               /*
> > +                * For User AFU feature, its region size is not fixed, but
> > +                * reported by register PortCapability.mmio_size. Resource
> > +                * size of UAFU will be set while parse port device.
> > +                */
> > +               .name = PORT_FEATURE_UAFU,
> > +               .resource_size = 0,
> > +               .feature_index = PORT_FEATURE_ID_UAFU,
> > +       },
> > +};
> > +
> > +static int
> > +create_feature_instance(struct build_feature_devs_info *binfo,
> > +                       void __iomem *start, struct feature_info *finfo)
> > +{
> > +       if (binfo->ioend - start < finfo->resource_size)
> > +               return -EINVAL;
> > +
> > +       build_info_add_sub_feature(binfo, finfo->feature_index, finfo->name,
> > +                                  finfo->resource_size, start);
> > +       return 0;
> > +}
> 
> build_info_add_sub_feature() is only called one time - here.  Could
> you collapse these two functions together?

Sure.

> 
> > +
> > +static int parse_feature_fme(struct build_feature_devs_info *binfo,
> > +                            void __iomem *start)
> > +{
> > +       struct cci_drvdata *drvdata = dev_get_drvdata(&binfo->pdev->dev);
> > +       int ret;
> > +
> > +       ret = build_info_create_dev(binfo, FME_ID, fme_feature_num(),
> > +                                       FPGA_FEATURE_DEV_FME);
> > +       if (ret)
> > +               return ret;
> > +
> > +       if (drvdata->fme_dev) {
> > +               dev_err(&binfo->pdev->dev, "Multiple FMEs are detected.\n");
> > +               return -EINVAL;
> > +       }
> > +
> > +       return create_feature_instance(binfo, start,
> > +                                      &fme_features[FME_FEATURE_ID_HEADER]);
> > +}
> > +
> > +static int parse_feature_fme_private(struct build_feature_devs_info *binfo,
> > +                                    struct feature_header *hdr)
> > +{
> > +       struct feature_header header;
> > +
> > +       header.csr = readq(hdr);
> > +
> > +       if (header.id >= ARRAY_SIZE(fme_features)) {
> > +               dev_info(&binfo->pdev->dev, "FME feature id %x is not supported yet.\n",
> > +                        header.id);
> > +               return 0;
> > +       }
> > +
> > +       return create_feature_instance(binfo, hdr, &fme_features[header.id]);
> > +}
> > +
> > +static int parse_feature_port(struct build_feature_devs_info *binfo,
> > +                            void __iomem *start)
> > +{
> > +       int ret;
> > +
> > +       ret = build_info_create_dev(binfo, PORT_ID, port_feature_num(),
> > +                                       FPGA_FEATURE_DEV_PORT);
> > +       if (ret)
> > +               return ret;
> > +
> > +       return create_feature_instance(binfo, start,
> > +                                      &port_features[PORT_FEATURE_ID_HEADER]);
> > +}
> > +
> > +static void enable_port_uafu(struct build_feature_devs_info *binfo,
> > +                            void __iomem *start)
> > +{
> > +       enum port_feature_id id = PORT_FEATURE_ID_UAFU;
> > +       struct feature_port_header *port_hdr;
> > +       struct feature_port_capability capability;
> > +
> > +       port_hdr = (struct feature_port_header *)start;
> > +       capability.csr = readq(&port_hdr->capability);
> > +       port_features[id].resource_size = capability.mmio_size << 10;
> > +
> > +       /*
> > +        * To enable User AFU, driver needs to clear reset bit on related port,
> > +        * otherwise the mmio space of this user AFU will be invalid.
> > +        */
> > +       if (port_features[id].resource_size)
> > +               fpga_port_reset(binfo->feature_dev);
> > +}
> > +
> > +static int parse_feature_port_private(struct build_feature_devs_info *binfo,
> > +                                     struct feature_header *hdr)
> > +{
> > +       struct feature_header header;
> > +       enum port_feature_id id;
> > +
> > +       header.csr = readq(hdr);
> > +       /*
> > +        * the region of port feature id is [0x10, 0x13], + 1 to reserve 0
> > +        * which is dedicated for port-hdr.
> > +        */
> > +       id = (header.id & 0x000f) + 1;
> > +
> > +       if (id >= ARRAY_SIZE(port_features)) {
> > +               dev_info(&binfo->pdev->dev, "Port feature id %x is not supported yet.\n",
> > +                        header.id);
> > +               return 0;
> > +       }
> > +
> > +       return create_feature_instance(binfo, hdr, &port_features[id]);
> > +}
> > +
> > +static int parse_feature_port_uafu(struct build_feature_devs_info *binfo,
> > +                                struct feature_header *hdr)
> > +{
> > +       enum port_feature_id id = PORT_FEATURE_ID_UAFU;
> > +       int ret;
> > +
> > +       if (port_features[id].resource_size) {
> > +               ret = create_feature_instance(binfo, hdr, &port_features[id]);
> > +               port_features[id].resource_size = 0;
> > +       } else {
> > +               dev_err(&binfo->pdev->dev, "the uafu feature header is mis-configured.\n");
> > +               ret = -EINVAL;
> > +       }
> > +
> > +       return ret;
> > +}
> > +
> > +static int parse_feature_afus(struct build_feature_devs_info *binfo,
> > +                             struct feature_header *hdr)
> > +{
> > +       int ret;
> > +       struct feature_afu_header *afu_hdr, header;
> > +       void __iomem *start;
> > +       void __iomem *end = binfo->ioend;
> > +
> > +       start = hdr;
> > +       for (; start < end; start += header.next_afu) {
> > +               if (end - start < (sizeof(*afu_hdr) + sizeof(*hdr)))
> > +                       return -EINVAL;
> > +
> > +               hdr = start;
> > +               afu_hdr = (struct feature_afu_header *) (hdr + 1);
> > +               header.csr = readq(&afu_hdr->csr);
> > +
> > +               if (feature_is_fme(afu_hdr)) {
> > +                       ret = parse_feature_fme(binfo, hdr);
> > +                       binfo->pfme_hdr = hdr;
> > +                       if (ret)
> > +                               return ret;
> > +               } else if (feature_is_port(afu_hdr)) {
> > +                       ret = parse_feature_port(binfo, hdr);
> > +                       enable_port_uafu(binfo, hdr);
> > +                       if (ret)
> > +                               return ret;
> > +               } else if (feature_is_UAFU(binfo)) {
> > +                       ret = parse_feature_port_uafu(binfo, hdr);
> > +                       if (ret)
> > +                               return ret;
> > +               } else
> > +                       dev_info(&binfo->pdev->dev, "AFU GUID %pUl is not supported yet.\n",
> > +                                afu_hdr->guid.b);
> > +
> > +               if (!header.next_afu)
> > +                       break;
> > +       }
> > +
> > +       return 0;
> > +}
> > +
> > +static int parse_feature_private(struct build_feature_devs_info *binfo,
> > +                                struct feature_header *hdr)
> > +{
> > +       struct feature_header header;
> > +
> > +       header.csr = readq(hdr);
> > +
> > +       if (!binfo->feature_dev) {
> > +               dev_err(&binfo->pdev->dev, "the private feature %x does not belong to any AFU.\n",
> > +                       header.id);
> > +               return -EINVAL;
> > +       }
> > +
> > +       switch (feature_dev_id_type(binfo->feature_dev)) {
> > +       case FME_ID:
> > +               return parse_feature_fme_private(binfo, hdr);
> > +       case PORT_ID:
> > +               return parse_feature_port_private(binfo, hdr);
> > +       default:
> > +               dev_info(&binfo->pdev->dev, "private feature %x belonging to AFU %s is not supported yet.\n",
> > +                        header.id, binfo->feature_dev->name);
> > +       }
> > +       return 0;
> > +}
> > +
> > +static int parse_feature(struct build_feature_devs_info *binfo,
> > +                        struct feature_header *hdr)
> > +{
> > +       struct feature_header header;
> > +       int ret = 0;
> > +
> > +       header.csr = readq(hdr);
> > +
> > +       switch (header.type) {
> > +       case FEATURE_TYPE_AFU:
> > +               ret = parse_feature_afus(binfo, hdr);
> > +               break;
> > +       case FEATURE_TYPE_PRIVATE:
> > +               ret = parse_feature_private(binfo, hdr);
> > +               break;
> > +       default:
> > +               dev_info(&binfo->pdev->dev,
> > +                        "Feature Type %x is not supported.\n", hdr->type);
> > +       };
> > +
> > +       return ret;
> > +}
> > +
> > +static int
> > +parse_feature_list(struct build_feature_devs_info *binfo, void __iomem *start)
> > +{
> > +       struct feature_header *hdr, header;
> > +       void __iomem *end = binfo->ioend;
> > +       int ret = 0;
> > +
> 
> Maybe a helpful comment that we are stepping through a linked list of features.

Sure, will add more comments.

All kind of features has a common header, they are linked together via the
header.next_header_offset.

If it's a TYPE=AFU feature, it will have one more pointer named NEXT_AFU,
which could point to another TYPE=AFU feature, please refer to function
parse_feature_afus.

If it's a TYPE=private feature, there is no other pointers, please refer
to function parse_feature_private.

> 
> > +       for (; start < end; start += header.next_header_offset) {
> > +               if (end - start < sizeof(*hdr)) {
> > +                       dev_err(&binfo->pdev->dev, "The region is too small to contain a feature.\n");
> > +                       ret =  -EINVAL;
> > +                       break;
> > +               }
> > +
> > +               hdr = (struct feature_header *)start;
> > +               ret = parse_feature(binfo, hdr);
> > +               if (ret)
> > +                       break;
> 
> Instead of parse_feature, this can save off the enumeration info and
> continue to read in the linked list.  After all the headers are read
> in, call the (separate) enumeration code to step through the saved
> headers, parse them, and create the devices.  Since the memory is
> iomapped during the process of reading in the headers, the enumeration
> code doesn't have to be so pcie specific.  Then this code base is
> better set to run on embedded devices also.

Actually, it has parse_feature here, because they're different type of
features. e.g for TYPE = AFU, GUID = PORT, it requires driver to check
its NEXT_AFU pointer to find User AFU (and its header). For private
features, there is no next_afu pointer at all.

> 
> > +
> > +               header.csr = readq(hdr);
> > +               if (!header.next_header_offset)
> > +                       break;
> > +       }
> > +
> > +       return ret;
> > +}
> > +
> > +static int parse_ports_from_fme(struct build_feature_devs_info *binfo)
> > +{
> > +       struct feature_fme_header *fme_hdr;
> > +       struct feature_fme_port port;
> > +       int i = 0, ret = 0;
> > +
> > +       if (binfo->pfme_hdr == NULL) {
> > +               dev_dbg(&binfo->pdev->dev, "VF is detected.\n");
> > +               return ret;
> > +       }
> > +
> > +       fme_hdr = binfo->pfme_hdr;
> > +
> > +       do {
> > +               port.csr = readq(&fme_hdr->port[i]);
> > +               if (!port.port_implemented)
> > +                       break;
> > +
> > +               ret = parse_switch_to(binfo, port.port_bar);
> > +               if (ret)
> > +                       break;
> > +
> > +               ret = parse_feature_list(binfo,
> > +                               binfo->ioaddr + port.port_offset);
> > +               if (ret)
> > +                       break;
> > +       } while (++i < MAX_FPGA_PORT_NUM);
> > +
> > +       return ret;
> > +}
> > +
> > +static int create_init_drvdata(struct pci_dev *pdev)
> > +{
> > +       struct cci_drvdata *drvdata;
> > +
> > +       drvdata = devm_kzalloc(&pdev->dev, sizeof(*drvdata), GFP_KERNEL);
> > +       if (!drvdata)
> > +               return -ENOMEM;
> > +
> > +       mutex_init(&drvdata->lock);
> > +       INIT_LIST_HEAD(&drvdata->port_dev_list);
> > +       INIT_LIST_HEAD(&drvdata->regions);
> > +
> > +       dev_set_drvdata(&pdev->dev, drvdata);
> > +       return 0;
> > +}
> > +
> > +static void destroy_drvdata(struct pci_dev *pdev)
> > +{
> > +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> > +
> > +       if (drvdata->fme_dev) {
> > +               /* fme device should be unregistered first. */
> > +               WARN_ON(device_is_registered(drvdata->fme_dev));
> > +               free_fpga_id(FME_ID, to_platform_device(drvdata->fme_dev)->id);
> > +               put_device(drvdata->fme_dev);
> > +       }
> > +
> > +       cci_pci_remove_port_devs(pdev);
> > +       cci_pci_release_regions(pdev);
> > +       dev_set_drvdata(&pdev->dev, NULL);
> > +       devm_kfree(&pdev->dev, drvdata);
> > +}
> > +
> > +static int cci_pci_create_feature_devs(struct pci_dev *pdev)
> > +{
> > +       struct build_feature_devs_info *binfo;
> > +       int ret;
> > +
> > +       binfo = build_info_alloc_and_init(pdev);
> > +       if (!binfo)
> > +               return -ENOMEM;
> > +
> > +       binfo->parent_dev = fpga_dev_create(&pdev->dev, INTEL_FPGA_DEV);
> > +       if (IS_ERR(binfo->parent_dev)) {
> > +               ret = PTR_ERR(binfo->parent_dev);
> > +               goto free_binfo_exit;
> > +       }
> > +
> > +       ret = parse_start(binfo);
> > +       if (ret)
> > +               goto free_binfo_exit;
> > +
> > +       ret = parse_feature_list(binfo, binfo->ioaddr);
> > +       if (ret)
> > +               goto free_binfo_exit;
> > +
> > +       ret = parse_ports_from_fme(binfo);
> > +       if (ret)
> > +               goto free_binfo_exit;
> 
> So ideally, there would be a function call here that read all the
> headers from hardware, ioremapping as it went along.  Then after that,
> call the enumeration code to create the devices.

If we have a function call here which read all the headers, then it
finish most works of parsing the device feature list. The only thing
that enumeration code to do is to create the devices with prepared
resources. It's fine to me, as we can just save information to this
binfo, and create devices later, but is this matches with what you
think? :)

Thanks
Hao

> 
> > +
> > +       ret = build_info_commit_dev(binfo);
> > +       if (ret)
> > +               goto free_binfo_exit;
> > +
> > +       /*
> > +        * everything is okay, reset ->parent_dev to stop it being
> > +        * freed by build_info_free()
> > +        */
> > +       binfo->parent_dev = NULL;
> > +
> > +free_binfo_exit:
> > +       build_info_free(binfo);
> > +       return ret;
> > +}
> > +
> >  /* PCI Device ID */
> >  #define PCIe_DEVICE_ID_PF_INT_5_X      0xBCBD
> >  #define PCIe_DEVICE_ID_PF_INT_6_X      0xBCC0
> > @@ -81,9 +898,18 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
> >                 goto release_region_exit;
> >         }
> >
> > -       /* TODO: create and add the platform device per feature list */
> > +       ret = create_init_drvdata(pcidev);
> > +       if (ret)
> > +               goto release_region_exit;
> > +
> > +       ret = cci_pci_create_feature_devs(pcidev);
> > +       if (ret)
> > +               goto destroy_drvdata_exit;
> > +
> >         return 0;
> >
> > +destroy_drvdata_exit:
> > +       destroy_drvdata(pcidev);
> >  release_region_exit:
> >         pci_release_regions(pcidev);
> >  disable_error_report_exit:
> > @@ -94,6 +920,8 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
> >
> >  static void cci_pci_remove(struct pci_dev *pcidev)
> >  {
> > +       remove_all_devs(pcidev);
> > +       destroy_drvdata(pcidev);
> >         pci_release_regions(pcidev);
> >         pci_disable_pcie_error_reporting(pcidev);
> >         pci_disable_device(pcidev);
> > @@ -108,14 +936,23 @@ static void cci_pci_remove(struct pci_dev *pcidev)
> >
> >  static int __init ccidrv_init(void)
> >  {
> > +       int ret;
> > +
> >         pr_info("Intel(R) FPGA PCIe Driver: Version %s\n", DRV_VERSION);
> >
> > -       return pci_register_driver(&cci_pci_driver);
> > +       fpga_ids_init();
> > +
> > +       ret = pci_register_driver(&cci_pci_driver);
> > +       if (ret)
> > +               fpga_ids_destroy();
> > +
> > +       return ret;
> >  }
> >
> >  static void __exit ccidrv_exit(void)
> >  {
> >         pci_unregister_driver(&cci_pci_driver);
> > +       fpga_ids_destroy();
> >  }
> >
> >  module_init(ccidrv_init);
> > --
> > 1.8.3.1
> >

Re: [PATCH v2 07/22] fpga: intel: pcie: parse feature list and create platform device for features.

[Wu Hao]

On Thu, Sep 21, 2017 at 02:58:57PM -0500, Alan Tull wrote:
> )On Wed, Sep 20, 2017 at 4:24 PM, Alan Tull <atull-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> 
> Hi Hao,
> 
> A few more minor things below.

Hi Alan

Thanks for your comments, will fix them in the next version.

Thanks
Hao

> 
> >  a (wh., *()On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> >
> > Hi Hao,
> >
> > I'm done with some board bringup so I have time to look at your patchset again.
> >
> > Something I can't help but notice is that this patchset will be almost
> > completely reusable for embedded FPGAs if the enumeration is separated
> > from the pcie code.  After the devices are created, they are just mmio
> > devices.  That makes this whole scheme available for embedded FPGAs.
> >
> > The division of things would be that the pcie code would read the
> > headers and do ioremapping.  Then pass the headers to the enumeration
> > code to create the devices.  With that division, another platform that
> > is embedded could have its own code that read the headers and did the
> > iomapping and then used the separate enumeration code.
> >
> > Going through intel-pcie.c, the truly pcie specific parts are already
> > pretty well contained.  The pcie code is in probe,
> > cci_pci_ioremap_bar, cci_pci_release_regions, and the 'parse_s*'
> > functions that ultimately call cci_pci_ioremap_bar.  So it all comes
> > down to the code that calls the 'parse_s*' functions.  That is, the
> > code the reads the fme header and the port headers.  Currently, if I'm
> > reading this right, this code ioremaps each region, reads its header,
> > and creates its device before reading the next header.  So we just
> > need to change the order a little and ioremap/read all the headers
> > before starting to create devices.  That separates the pci ioremap
> > from creating the devices.  So the probe function could ioremap and
> > read in the fme header.  Then it would ioremap and read in each of the
> > port headers.  After they are read in, then pass the all these headers
> > to the separate enumeration code that will parse it and create the
> > devices.  That part is not pcie specific, right?  With the enumeration
> > code in a separate file (and assuming that the ioremap has already
> > happened),  another platform that is embedded could have its probe do
> > the iomapping and reading the headers, and then call the same
> > enumeration code.
> >
> > Actually, there's a bit more to it than that but I think this is
> > doable.  build_info_add_sub_feature() (which is called by
> > create_feature_instance) saves off resource info for each sub device.
> > Does this mean this code is ioremapping again (in
> > intel-fpga-fme-mgr.c's probe)?
> >
> > A few more things below...
> >
> >> From: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> >>
> >> Device Feature List structure creates a link list of feature headers
> >> within the MMIO space to provide an extensible way of adding features.
> >>
> >> The Intel FPGA PCIe driver walks through the feature headers to enumerate
> >> feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
> >> Function Unit (AFU), and their private sub features. For feature devices,
> >> it creates the platform devices and linked the private sub features into
> >> their platform data.
> >>
> >> Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> Signed-off-by: Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> Signed-off-by: Zhang Yi <yi.z.zhang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> >> Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> ---
> >> v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
> >>     switched to GPLv2 license.
> >>     fixed comments from Moritz Fischer.
> >>     fixed kbuild warning, typos and clean up the code.
> >> ---
> >>  drivers/fpga/Makefile            |   2 +-
> >>  drivers/fpga/intel-feature-dev.c | 130 ++++++
> >>  drivers/fpga/intel-feature-dev.h | 341 ++++++++++++++++
> >>  drivers/fpga/intel-pcie.c        | 841 ++++++++++++++++++++++++++++++++++++++-
> >>  4 files changed, 1311 insertions(+), 3 deletions(-)
> >>  create mode 100644 drivers/fpga/intel-feature-dev.c
> >>  create mode 100644 drivers/fpga/intel-feature-dev.h
> >>
> >> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> >> index 5613133..ad24b3d 100644
> >> --- a/drivers/fpga/Makefile
> >> +++ b/drivers/fpga/Makefile
> >> @@ -31,4 +31,4 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
> >>  # Intel FPGA Support
> >>  obj-$(CONFIG_INTEL_FPGA_PCI)           += intel-fpga-pci.o
> >>
> >> -intel-fpga-pci-objs := intel-pcie.o
> >> +intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
> >> diff --git a/drivers/fpga/intel-feature-dev.c b/drivers/fpga/intel-feature-dev.c
> >> new file mode 100644
> >> index 0000000..68f9cba
> >> --- /dev/null
> >> +++ b/drivers/fpga/intel-feature-dev.c
> >> @@ -0,0 +1,130 @@
> >> +/*
> >> + * Intel FPGA Feature Device Driver
> >> + *
> >> + * Copyright (C) 2017 Intel Corporation, Inc.
> >> + *
> >> + * Authors:
> >> + *   Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> + *   Zhang Yi <yi.z.zhang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> + *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> + *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> >> + *
> >> + * This work is licensed under the terms of the GNU GPL version 2. See
> >> + * the COPYING file in the top-level directory.
> >> + */
> >> +
> >> +#include "intel-feature-dev.h"
> >> +
> >> +void feature_platform_data_add(struct feature_platform_data *pdata,
> >> +                              int index, const char *name,
> >> +                              int resource_index, void __iomem *ioaddr)
> >> +{
> >> +       WARN_ON(index >= pdata->num);
> >> +
> >> +       pdata->features[index].name = name;
> >> +       pdata->features[index].resource_index = resource_index;
> >> +       pdata->features[index].ioaddr = ioaddr;
> >> +}
> 
> This function is only called once.  I understand that it is desirable
> to break things up into little functions and avoid overly long
> functions, but in this case, the calling function
> (build_info_add_sub_feature) is pretty short to begin with.   Someone
> reading the code will know what is happening better if you just add
> its contents to its calling function.
> 
> >> +
> >> +struct feature_platform_data *
> >> +feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
> >> +{
> >> +       struct feature_platform_data *pdata;
> >> +
> >> +       pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
> >> +       if (pdata) {
> >> +               pdata->dev = dev;
> >> +               pdata->num = num;
> >> +               mutex_init(&pdata->lock);
> >> +       }
> >> +
> >> +       return pdata;
> >> +}
> 
> feature_platform_data_alloc_and_init is another function that's small,
> called once, and would be better included in its caller.
> 
> >> +
> >> +int fme_feature_num(void)
> >> +{
> >> +       return FME_FEATURE_ID_MAX;
> >> +}
> >> +
> >> +int port_feature_num(void)
> >> +{
> >> +       return PORT_FEATURE_ID_MAX;
> >> +}
> >
> > Do these need to be functions?  If so, static.  But if you can just
> > use the #define'd values that's one level of indirection we can get
> > rid of.  And when I'm going through code where there's extra levels of
> > indirection that don't do anything, it makes it harder to understand
> > ;)
> >
> >> +
> >> +int fpga_port_id(struct platform_device *pdev)
> >> +{
> >> +       struct feature_port_header *port_hdr;
> >> +       struct feature_port_capability capability;
> >> +
> >> +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> >> +                                              PORT_FEATURE_ID_HEADER);
> >> +       WARN_ON(!port_hdr);
> >> +
> >> +       capability.csr = readq(&port_hdr->capability);
> >> +       return capability.port_number;
> >> +}
> >> +EXPORT_SYMBOL_GPL(fpga_port_id);
> >
> > This is here because every feature is a port, right?
> >
> >> +
> >> +/*
> >> + * Enable Port by clear the port soft reset bit, which is set by default.
> >> + * The User AFU is unable to respond to any MMIO access while in reset.
> >> + * __fpga_port_enable function should only be used after __fpga_port_disable
> >> + * function.
> >> + */
> >> +void __fpga_port_enable(struct platform_device *pdev)
> >> +{
> >> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >> +       struct feature_port_header *port_hdr;
> >> +       struct feature_port_control control;
> >> +
> >> +       WARN_ON(!pdata->disable_count);
> >> +
> >> +       if (--pdata->disable_count != 0)
> >> +               return;
> >> +
> >> +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> >> +                                              PORT_FEATURE_ID_HEADER);
> >> +       WARN_ON(!port_hdr);
> >> +
> >> +       control.csr = readq(&port_hdr->control);
> >> +       control.port_sftrst = 0x0;
> >> +       writeq(control.csr, &port_hdr->control);
> >> +}
> >> +EXPORT_SYMBOL_GPL(__fpga_port_enable);
> >> +
> >> +#define RST_POLL_INVL 10 /* us */
> >> +#define RST_POLL_TIMEOUT 1000 /* us */
> >> +
> >> +int __fpga_port_disable(struct platform_device *pdev)
> >> +{
> >> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >> +       struct feature_port_header *port_hdr;
> >> +       struct feature_port_control control;
> >> +
> >> +       if (pdata->disable_count++ != 0)
> >> +               return 0;
> >> +
> >> +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> >> +                                              PORT_FEATURE_ID_HEADER);
> >> +       WARN_ON(!port_hdr);
> >> +
> >> +       /* Set port soft reset */
> >> +       control.csr = readq(&port_hdr->control);
> >> +       control.port_sftrst = 0x1;
> >> +       writeq(control.csr, &port_hdr->control);
> >> +
> >> +       /*
> >> +        * HW sets ack bit to 1 when all outstanding requests have been drained
> >> +        * on this port and minimum soft reset pulse width has elapsed.
> >> +        * Driver polls port_soft_reset_ack to determine if reset done by HW.
> >> +        */
> >> +       if (readq_poll_timeout(&port_hdr->control, control.csr,
> >> +                              (control.port_sftrst_ack == 1),
> >> +                              RST_POLL_INVL, RST_POLL_TIMEOUT)) {
> >> +               dev_err(&pdev->dev, "timeout, fail to reset device\n");
> >> +               return -ETIMEDOUT;
> >> +       }
> >> +
> >> +       return 0;
> >> +}
> >> +EXPORT_SYMBOL_GPL(__fpga_port_disable);
> >> diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
> >> new file mode 100644
> >> index 0000000..f67784a
> >> --- /dev/null
> >> +++ b/drivers/fpga/intel-feature-dev.h
> >> @@ -0,0 +1,341 @@
> >> +/*
> >> + * Intel FPGA Feature Device Driver Header File
> >> + *
> >> + * Copyright (C) 2017 Intel Corporation, Inc.
> >> + *
> >> + * Authors:
> >> + *   Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> + *   Zhang Yi <yi.z.zhang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> + *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> + *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> >> + *
> >> + * This work is licensed under the terms of the GNU GPL version 2. See
> >> + * the COPYING file in the top-level directory.
> >> + */
> >> +
> >> +#ifndef __INTEL_FPGA_FEATURE_H
> >> +#define __INTEL_FPGA_FEATURE_H
> >> +
> >> +#include <linux/fs.h>
> >> +#include <linux/pci.h>
> >> +#include <linux/uuid.h>
> >> +#include <linux/delay.h>
> >> +#include <linux/iopoll.h>
> >> +#include <linux/platform_device.h>
> >> +
> >> +#ifndef readq
> >> +static inline u64 readq(void __iomem *addr)
> >> +{
> >> +       return readl(addr) + ((u64)readl(addr + 4) << 32);
> >> +}
> >> +#endif
> >> +
> >> +#ifndef writeq
> >> +static inline void writeq(u64 val, void __iomem *addr)
> >> +{
> >> +       writel((u32) (val), addr);
> >> +       writel((u32) (val >> 32), (addr + 4));
> >> +}
> >> +#endif
> >> +
> >> +/* maximum supported number of ports */
> >> +#define MAX_FPGA_PORT_NUM 4
> >> +/* plus one for fme device */
> >> +#define MAX_FEATURE_DEV_NUM    (MAX_FPGA_PORT_NUM + 1)
> >> +
> >> +#define FME_FEATURE_HEADER          "fme_hdr"
> >> +#define FME_FEATURE_THERMAL_MGMT    "fme_thermal"
> >> +#define FME_FEATURE_POWER_MGMT      "fme_power"
> >> +#define FME_FEATURE_GLOBAL_PERF     "fme_gperf"
> >> +#define FME_FEATURE_GLOBAL_ERR      "fme_error"
> >> +#define FME_FEATURE_PR_MGMT         "fme_pr"
> >> +
> >> +#define PORT_FEATURE_HEADER         "port_hdr"
> >> +#define PORT_FEATURE_UAFU           "port_uafu"
> >> +#define PORT_FEATURE_ERR            "port_err"
> >> +#define PORT_FEATURE_UMSG           "port_umsg"
> >> +#define PORT_FEATURE_PR             "port_pr"
> >> +#define PORT_FEATURE_STP            "port_stp"
> >> +
> >> +/* All headers and structures must be byte-packed to match the spec. */
> >> +#pragma pack(1)
> >> +
> >> +/* common header for all features */
> >> +struct feature_header {
> >> +       union {
> >> +               u64 csr;
> >> +               struct {
> >> +                       u64 id:12;
> >> +                       u64 revision:4;
> >> +                       u64 next_header_offset:24; /* offset to next header */
> >> +                       u64 rsvdz:20;
> >> +                       u64 type:4;                /* feature type */
> >> +#define FEATURE_TYPE_AFU               0x1
> >> +#define FEATURE_TYPE_PRIVATE           0x3
> >> +               };
> >> +       };
> >> +};
> >> +
> >> +/* common header for non-private features */
> >> +struct feature_afu_header {
> >> +       uuid_le guid;
> >> +       union {
> >> +               u64 csr;
> >> +               struct {
> >> +                       u64 next_afu:24;        /* pointer to next afu header */
> >> +                       u64 rsvdz:40;
> >> +               };
> >> +       };
> >> +};
> >> +
> >> +/* FME Header Register Set */
> >> +/* FME Capability Register */
> >> +struct feature_fme_capability {
> >> +       union {
> >> +               u64 csr;
> >> +               struct {
> >> +                       u64 fabric_verid:8;     /* Fabric version ID */
> >> +                       u64 socket_id:1;        /* Socket id */
> >> +                       u64 rsvdz1:3;
> >> +                       u64 pcie0_link_avl:1;   /* PCIe0 link availability */
> >> +                       u64 pcie1_link_avl:1;   /* PCIe1 link availability */
> >> +                       u64 coherent_link_avl:1;/* Coherent link availability */
> >> +                       u64 rsvdz2:1;
> >> +                       u64 iommu_support:1;    /* IOMMU or VT-d supported */
> >> +                       u64 num_ports:3;        /* Num of ports implemented */
> >> +                       u64 rsvdz3:4;
> >> +                       u64 addr_width_bits:6;  /* Address width supported */
> >> +                       u64 rsvdz4:2;
> >> +                       u64 cache_size:12;      /* Cache size in kb */
> >> +                       u64 cache_assoc:4;      /* Cache Associativity */
> >> +                       u64 rsvdz5:15;
> >> +                       u64 lock_bit:1;         /* Latched lock bit by BIOS */
> >> +               };
> >> +       };
> >> +};
> >> +
> >> +/* FME Port Offset Register */
> >> +struct feature_fme_port {
> >> +       union {
> >> +               u64 csr;
> >> +               struct {
> >> +                       u64 port_offset:24;     /* Offset to port header */
> >> +                       u64 rsvdz1:8;
> >> +                       u64 port_bar:3;         /* Bar id */
> >> +                       u64 rsvdz2:20;
> >> +                       u64 afu_access_ctrl:1;  /* AFU access type: PF/VF */
> >> +                       u64 rsvdz3:4;
> >> +                       u64 port_implemented:1; /* Port implemented or not */
> >> +                       u64 rsvdz4:3;
> >> +               };
> >> +       };
> >> +};
> >> +
> >> +struct feature_fme_header {
> >> +       struct feature_header header;
> >> +       struct feature_afu_header afu_header;
> >> +       u64 rsvd[2];
> >> +       struct feature_fme_capability capability;
> >> +       struct feature_fme_port port[MAX_FPGA_PORT_NUM];
> >> +};
> >> +
> >> +/* FME Thermal Sub Feature Register Set */
> >> +struct feature_fme_thermal {
> >> +       struct feature_header header;
> >> +};
> >> +
> >> +/* FME Power Sub Feature Register Set */
> >> +struct feature_fme_power {
> >> +       struct feature_header header;
> >> +};
> >> +
> >> +/* FME Global Performance Sub Feature Register Set */
> >> +struct feature_fme_gperf {
> >> +       struct feature_header header;
> >> +};
> >> +
> >> +/* FME Error Sub Feature Register Set */
> >> +struct feature_fme_err {
> >> +       struct feature_header header;
> >> +};
> >> +
> >> +/* FME Partial Reconfiguration Sub Feature Register Set */
> >> +struct feature_fme_pr {
> >> +       struct feature_header header;
> >> +};
> >> +
> >> +/* PORT Header Register Set */
> >> +/* Port Capability Register */
> >> +struct feature_port_capability {
> >> +       union {
> >> +               u64 csr;
> >> +               struct {
> >> +                       u64 port_number:2;      /* Port Number 0-3 */
> >> +                       u64 rsvdz1:6;
> >> +                       u64 mmio_size:16;       /* User MMIO size in KB */
> >> +                       u64 rsvdz2:8;
> >> +                       u64 sp_intr_num:4;      /* Supported interrupts num */
> >> +                       u64 rsvdz3:28;
> >> +               };
> >> +       };
> >> +};
> >> +
> >> +/* Port Control Register */
> >> +struct feature_port_control {
> >> +       union {
> >> +               u64 csr;
> >> +               struct {
> >> +                       u64 port_sftrst:1;      /* Port Soft Reset */
> >> +                       u64 rsvdz1:1;
> >> +                       u64 latency_tolerance:1;/* '1' >= 40us, '0' < 40us */
> >> +                       u64 rsvdz2:1;
> >> +                       u64 port_sftrst_ack:1;  /* HW ACK for Soft Reset */
> >> +                       u64 rsvdz3:59;
> >> +               };
> >> +       };
> >> +};
> >> +
> >> +struct feature_port_header {
> >> +       struct feature_header header;
> >> +       struct feature_afu_header afu_header;
> >> +       u64 rsvd[2];
> >> +       struct feature_port_capability capability;
> >> +       struct feature_port_control control;
> >> +};
> >> +
> >> +/* PORT Error Sub Feature Register Set */
> >> +struct feature_port_error {
> >> +       struct feature_header header;
> >> +};
> >> +
> >> +/* PORT Unordered Message Sub Feature Register Set */
> >> +struct feature_port_umsg {
> >> +       struct feature_header header;
> >> +};
> >> +
> >> +/* PORT SignalTap Sub Feature Register Set */
> >> +struct feature_port_stp {
> >> +       struct feature_header header;
> >> +};
> >> +
> >> +#pragma pack()
> >> +
> >> +struct feature {
> >> +       const char *name;
> >> +       int resource_index;
> >> +       void __iomem *ioaddr;
> >> +};
> >> +
> >> +struct feature_platform_data {
> >> +       /* list the feature dev to cci_drvdata->port_dev_list. */
> >> +       struct list_head node;
> >> +       struct mutex lock;
> >> +       struct platform_device *dev;
> >> +       unsigned int disable_count;     /* count for port disable */
> >> +
> >> +       int num;                        /* number of features */
> >> +       struct feature features[0];
> >> +};
> >> +
> >> +enum fme_feature_id {
> >> +       FME_FEATURE_ID_HEADER = 0x0,
> >> +       FME_FEATURE_ID_THERMAL_MGMT = 0x1,
> >> +       FME_FEATURE_ID_POWER_MGMT = 0x2,
> >> +       FME_FEATURE_ID_GLOBAL_PERF = 0x3,
> >> +       FME_FEATURE_ID_GLOBAL_ERR = 0x4,
> >> +       FME_FEATURE_ID_PR_MGMT = 0x5,
> >> +       FME_FEATURE_ID_MAX = 0x6,
> >> +};
> >> +
> >> +enum port_feature_id {
> >> +       PORT_FEATURE_ID_HEADER = 0x0,
> >> +       PORT_FEATURE_ID_ERROR = 0x1,
> >> +       PORT_FEATURE_ID_UMSG = 0x2,
> >> +       PORT_FEATURE_ID_PR = 0x3,
> >> +       PORT_FEATURE_ID_STP = 0x4,
> >> +       PORT_FEATURE_ID_UAFU = 0x5,
> >> +       PORT_FEATURE_ID_MAX = 0x6,
> >> +};
> >> +
> >> +int fme_feature_num(void);
> >> +int port_feature_num(void);
> >> +
> >> +#define FPGA_FEATURE_DEV_FME           "intel-fpga-fme"
> >> +#define FPGA_FEATURE_DEV_PORT          "intel-fpga-port"
> >> +
> >> +void feature_platform_data_add(struct feature_platform_data *pdata,
> >> +                              int index, const char *name,
> >> +                              int resource_index, void __iomem *ioaddr);
> >> +
> >> +static inline int feature_platform_data_size(const int num)
> >> +{
> >> +       return sizeof(struct feature_platform_data) +
> >> +               num * sizeof(struct feature);
> >> +}
> >> +
> >> +struct feature_platform_data *
> >> +feature_platform_data_alloc_and_init(struct platform_device *dev, int num);
> >> +
> >> +int fpga_port_id(struct platform_device *pdev);
> >> +
> >> +static inline int fpga_port_check_id(struct platform_device *pdev,
> >> +                                    void *pport_id)
> >> +{
> >> +       return fpga_port_id(pdev) == *(int *)pport_id;
> >> +}
> >> +
> >> +void __fpga_port_enable(struct platform_device *pdev);
> >> +int __fpga_port_disable(struct platform_device *pdev);
> >> +
> >> +static inline void fpga_port_enable(struct platform_device *pdev)
> >> +{
> >> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >> +
> >> +       mutex_lock(&pdata->lock);
> >> +       __fpga_port_enable(pdev);
> >> +       mutex_unlock(&pdata->lock);
> >> +}
> >> +
> >> +static inline int fpga_port_disable(struct platform_device *pdev)
> >> +{
> >> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >> +       int ret;
> >> +
> >> +       mutex_lock(&pdata->lock);
> >> +       ret = __fpga_port_disable(pdev);
> >> +       mutex_unlock(&pdata->lock);
> >> +
> >> +       return ret;
> >> +}
> >> +
> >> +static inline int __fpga_port_reset(struct platform_device *pdev)
> >> +{
> >> +       int ret;
> >> +
> >> +       ret = __fpga_port_disable(pdev);
> >> +       if (ret)
> >> +               return ret;
> >> +
> >> +       __fpga_port_enable(pdev);
> >> +       return 0;
> >> +}
> >> +
> >> +static inline int fpga_port_reset(struct platform_device *pdev)
> >> +{
> >> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >> +       int ret;
> >> +
> >> +       mutex_lock(&pdata->lock);
> >> +       ret = __fpga_port_reset(pdev);
> >> +       mutex_unlock(&pdata->lock);
> >> +       return ret;
> >> +}
> >> +
> >> +static inline void __iomem *
> >> +get_feature_ioaddr_by_index(struct device *dev, int index)
> >> +{
> >> +       struct feature_platform_data *pdata = dev_get_platdata(dev);
> >> +
> >> +       return pdata->features[index].ioaddr;
> >> +}
> >> +#endif
> >> diff --git a/drivers/fpga/intel-pcie.c b/drivers/fpga/intel-pcie.c
> >> index f697de4..70b8284 100644
> >> --- a/drivers/fpga/intel-pcie.c
> >> +++ b/drivers/fpga/intel-pcie.c
> >> @@ -23,10 +23,827 @@
> >>  #include <linux/stddef.h>
> >>  #include <linux/errno.h>
> >>  #include <linux/aer.h>
> >> +#include <linux/fpga/fpga-dev.h>
> >> +
> >> +#include "intel-feature-dev.h"
> >>
> >>  #define DRV_VERSION    "0.8"
> >>  #define DRV_NAME       "intel-fpga-pci"
> >>
> >> +#define INTEL_FPGA_DEV "intel-fpga-dev"
> >> +
> >> +static DEFINE_MUTEX(fpga_id_mutex);
> >> +
> >> +enum fpga_id_type {
> >> +       FME_ID,         /* fme id allocation and mapping */
> >> +       PORT_ID,        /* port id allocation and mapping */
> >> +       FPGA_ID_MAX,
> >> +};
> >> +
> >> +/* it is protected by fpga_id_mutex */
> >> +static struct idr fpga_ids[FPGA_ID_MAX];
> >> +
> >> +struct cci_drvdata {
> >> +       struct device *fme_dev;
> >> +
> >> +       struct mutex lock;
> >> +       struct list_head port_dev_list;
> >> +
> >> +       struct list_head regions; /* global list of pci bar mapping region */
> >> +};
> >> +
> >> +/* pci bar mapping info */
> >> +struct cci_pci_region {
> >> +       int bar;
> >> +       void __iomem *ioaddr;   /* pointer to mapped bar region */
> >> +       struct list_head node;
> >> +};
> >> +
> >> +static void fpga_ids_init(void)
> >> +{
> >> +       int i;
> >> +
> >> +       for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> >> +               idr_init(fpga_ids + i);
> >> +}
> >> +
> >> +static void fpga_ids_destroy(void)
> >> +{
> >> +       int i;
> >> +
> >> +       for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> >> +               idr_destroy(fpga_ids + i);
> >> +}
> >> +
> >> +static int alloc_fpga_id(enum fpga_id_type type, struct device *dev)
> >> +{
> >> +       int id;
> >> +
> >> +       WARN_ON(type >= FPGA_ID_MAX);
> >> +       mutex_lock(&fpga_id_mutex);
> >> +       id = idr_alloc(fpga_ids + type, dev, 0, 0, GFP_KERNEL);
> >> +       mutex_unlock(&fpga_id_mutex);
> >> +       return id;
> >> +}
> >> +
> >> +static void free_fpga_id(enum fpga_id_type type, int id)
> >> +{
> >> +       WARN_ON(type >= FPGA_ID_MAX);
> >> +       mutex_lock(&fpga_id_mutex);
> >> +       idr_remove(fpga_ids + type, id);
> >> +       mutex_unlock(&fpga_id_mutex);
> >> +}
> >> +
> >> +static void cci_pci_add_port_dev(struct pci_dev *pdev,
> >> +                                struct platform_device *port_dev)
> >> +{
> >> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> >> +       struct feature_platform_data *pdata = dev_get_platdata(&port_dev->dev);
> >> +
> >> +       mutex_lock(&drvdata->lock);
> >> +       list_add(&pdata->node, &drvdata->port_dev_list);
> >> +       get_device(&pdata->dev->dev);
> >> +       mutex_unlock(&drvdata->lock);
> >> +}
> >> +
> >> +static void cci_pci_remove_port_devs(struct pci_dev *pdev)
> >> +{
> >> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> >> +       struct feature_platform_data *pdata, *ptmp;
> >> +
> >> +       mutex_lock(&drvdata->lock);
> >> +       list_for_each_entry_safe(pdata, ptmp, &drvdata->port_dev_list, node) {
> >> +               struct platform_device *port_dev = pdata->dev;
> >> +
> >> +               /* the port should be unregistered first. */
> >> +               WARN_ON(device_is_registered(&port_dev->dev));
> >> +               list_del(&pdata->node);
> >> +               free_fpga_id(PORT_ID, port_dev->id);
> >> +               put_device(&port_dev->dev);
> >> +       }
> >> +       mutex_unlock(&drvdata->lock);
> >> +}
> >> +
> >> +/* info collection during feature dev build. */
> >> +struct build_feature_devs_info {
> >> +       struct pci_dev *pdev;
> >> +
> >> +       /*
> >> +        * PCI BAR mapping info. Parsing feature list starts from
> >> +        * BAR 0 and switch to different BARs to parse Port
> >> +        */
> >> +       void __iomem *ioaddr;
> >> +       void __iomem *ioend;
> >> +       int current_bar;
> >> +
> >> +       /* points to FME header where the port offset is figured out. */
> >> +       void __iomem *pfme_hdr;
> >> +
> >> +       /* the container device for all feature devices */
> >> +       struct fpga_dev *parent_dev;
> >> +
> >> +       /* current feature device */
> >> +       struct platform_device *feature_dev;
> >> +};
> >> +
> >> +static void cci_pci_release_regions(struct pci_dev *pdev)
> >> +{
> >> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> >> +       struct cci_pci_region *tmp, *region;
> >> +
> >> +       list_for_each_entry_safe(region, tmp, &drvdata->regions, node) {
> >> +               list_del(&region->node);
> >> +               if (region->ioaddr)
> >> +                       pci_iounmap(pdev, region->ioaddr);
> >> +               devm_kfree(&pdev->dev, region);
> >> +       }
> >> +}
> >> +
> >> +static void __iomem *cci_pci_ioremap_bar(struct pci_dev *pdev, int bar)
> >> +{
> >> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> >> +       struct cci_pci_region *region;
> >> +
> >> +       list_for_each_entry(region, &drvdata->regions, node)
> >> +               if (region->bar == bar) {
> >> +                       dev_dbg(&pdev->dev, "BAR %d region exists\n", bar);
> >> +                       return region->ioaddr;
> >> +               }
> >> +
> >> +       region = devm_kzalloc(&pdev->dev, sizeof(*region), GFP_KERNEL);
> >> +       if (!region)
> >> +               return NULL;
> >> +
> >> +       region->bar = bar;
> >> +       region->ioaddr = pci_ioremap_bar(pdev, bar);
> >> +       if (!region->ioaddr) {
> >> +               dev_err(&pdev->dev, "can't ioremap memory from BAR %d.\n", bar);
> >> +               devm_kfree(&pdev->dev, region);
> >> +               return NULL;
> >> +       }
> >> +
> >> +       list_add(&region->node, &drvdata->regions);
> >> +       return region->ioaddr;
> >> +}
> >> +
> >> +static int parse_start_from(struct build_feature_devs_info *binfo, int bar)
> >> +{
> >> +       binfo->ioaddr = cci_pci_ioremap_bar(binfo->pdev, bar);
> >> +       if (!binfo->ioaddr)
> >> +               return -ENOMEM;
> >> +
> >> +       binfo->current_bar = bar;
> >> +       binfo->ioend = binfo->ioaddr + pci_resource_len(binfo->pdev, bar);
> >> +       return 0;
> >> +}
> >> +
> >> +static int parse_start(struct build_feature_devs_info *binfo)
> >> +{
> >> +       /* fpga feature list starts from BAR 0 */
> >> +       return parse_start_from(binfo, 0);
> >> +}
> >> +
> >> +/* switch the memory mapping to BAR# @bar */
> >> +static int parse_switch_to(struct build_feature_devs_info *binfo, int bar)
> >> +{
> >> +       return parse_start_from(binfo, bar);
> >> +}
> >
> > parse_switch_to() and parse_start() are both one line wrappers for
> > parse_start_from() and are only called once.  :-) Please just use
> > parse_start_from and get rid of parse_start and parse_switch_to.  I
> > don't think this code is all that super complicated.  But as I'm
> > grepping through this code, I'm trying to see all the places that do
> > the same thing; if I have to look for multiple function names, it
> > takes longer to get what is going on here.
> >
> > Actually, cci_pci_ioremap_bar is only called in one place -
> > parse_start_from - which really just adds two lines of code.
> > parse_start_from could go away.  Just a suggestion.  The reason this
> > comes up is that cci_pci_ioremap_bar is the more descriptive function
> > name.  The parse_start/switch function names hide the biggest thing
> > they do, which is ioremapping.  They don't actually do any parsing ;)
> >
> >> +
> >> +static struct build_feature_devs_info *
> >> +build_info_alloc_and_init(struct pci_dev *pdev)
> >> +{
> >> +       struct build_feature_devs_info *binfo;
> >> +
> >> +       binfo = devm_kzalloc(&pdev->dev, sizeof(*binfo), GFP_KERNEL);
> >> +       if (binfo)
> >> +               binfo->pdev = pdev;
> >
> > build_info_alloc_and_init() is only doing a devm_kzalloc and is only
> > called once.  The code would be more readable if you just had the
> > devm_kzalloc in the function that called this
> > (cci_pci_create_feature_devs).
> >
> >> +
> >> +       return binfo;
> >> +}
> >> +
> >> +static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
> >> +{
> >> +       if (!strcmp(pdev->name, FPGA_FEATURE_DEV_FME))
> >> +               return FME_ID;
> >> +
> >> +       if (!strcmp(pdev->name, FPGA_FEATURE_DEV_PORT))
> >> +               return PORT_ID;
> >> +
> >> +       WARN_ON(1);
> >> +       return FPGA_ID_MAX;
> >> +}
> >> +
> >> +/*
> >> + * register current feature device, it is called when we need to switch to
> >> + * another feature parsing or we have parsed all features
> >> + */
> >> +static int build_info_commit_dev(struct build_feature_devs_info *binfo)
> >> +{
> >> +       int ret;
> >> +
> >> +       if (!binfo->feature_dev)
> >> +               return 0;
> >> +
> >> +       ret = platform_device_add(binfo->feature_dev);
> >> +       if (!ret) {
> >> +               struct cci_drvdata *drvdata;
> >> +
> >> +               drvdata = dev_get_drvdata(&binfo->pdev->dev);
> >> +               if (feature_dev_id_type(binfo->feature_dev) == PORT_ID)
> >> +                       cci_pci_add_port_dev(binfo->pdev, binfo->feature_dev);
> >> +               else
> >> +                       drvdata->fme_dev = get_device(&binfo->feature_dev->dev);
> >> +
> >> +               /*
> >> +                * reset it to avoid build_info_free() freeing their resource.
> >> +                *
> >> +                * The resource of successfully registered feature devices
> >> +                * will be freed by platform_device_unregister(). See the
> >> +                * comments in build_info_create_dev().
> >> +                */
> >> +               binfo->feature_dev = NULL;
> >> +       }
> >> +
> >> +       return ret;
> >> +}
> >> +
> >> +static int
> >> +build_info_create_dev(struct build_feature_devs_info *binfo,
> >> +                     enum fpga_id_type type, int feature_nr, const char *name)
> >> +{
> >> +       struct platform_device *fdev;
> >> +       struct resource *res;
> >> +       struct feature_platform_data *pdata;
> >> +       int ret;
> >> +
> >> +       /* we will create a new device, commit current device first */
> >> +       ret = build_info_commit_dev(binfo);
> >> +       if (ret)
> >> +               return ret;
> >> +
> >> +       /*
> >> +        * we use -ENODEV as the initialization indicator which indicates
> >> +        * whether the id need to be reclaimed
> >> +        */
> >> +       fdev = binfo->feature_dev = platform_device_alloc(name, -ENODEV);
> >> +       if (!fdev)
> >> +               return -ENOMEM;
> >> +
> >> +       fdev->id = alloc_fpga_id(type, &fdev->dev);
> >> +       if (fdev->id < 0)
> >> +               return fdev->id;
> >> +
> >> +       fdev->dev.parent = &binfo->parent_dev->dev;
> >> +
> >> +       /*
> >> +        * we do not need to care for the memory which is associated with
> >> +        * the platform device. After calling platform_device_unregister(),
> >> +        * it will be automatically freed by device's release() callback,
> >> +        * platform_device_release().
> >> +        */
> >> +       pdata = feature_platform_data_alloc_and_init(fdev, feature_nr);
> >> +       if (!pdata)
> >> +               return -ENOMEM;
> >> +
> >> +       /*
> >> +        * the count should be initialized to 0 to make sure
> >> +        *__fpga_port_enable() following __fpga_port_disable()
> >> +        * works properly for port device.
> >> +        * and it should always be 0 for fme device.
> >> +        */
> >> +       WARN_ON(pdata->disable_count);
> >> +
> >> +       fdev->dev.platform_data = pdata;
> >> +       fdev->num_resources = feature_nr;
> >> +       fdev->resource = kcalloc(feature_nr, sizeof(*res), GFP_KERNEL);
> >> +       if (!fdev->resource)
> >> +               return -ENOMEM;
> >> +
> >> +       return 0;
> >> +}
> >> +
> >> +static int remove_feature_dev(struct device *dev, void *data)
> >> +{
> >> +       struct platform_device *pdev = to_platform_device(dev);
> >> +
> >> +       platform_device_unregister(pdev);
> >> +       return 0;
> >> +}
> >> +
> >> +static int remove_parent_dev(struct device *dev, void *data)
> >> +{
> >> +       /* remove platform devices attached in the parent device */
> >> +       device_for_each_child(dev, NULL, remove_feature_dev);
> >> +       fpga_dev_destroy(to_fpga_dev(dev));
> >> +       return 0;
> >> +}
> >> +
> >> +static void remove_all_devs(struct pci_dev *pdev)
> >> +{
> >> +       /* remove parent device and all its children. */
> >> +       device_for_each_child(&pdev->dev, NULL, remove_parent_dev);
> >> +}
> >> +
> >> +static void build_info_free(struct build_feature_devs_info *binfo)
> >> +{
> >> +       if (!IS_ERR_OR_NULL(binfo->parent_dev))
> >> +               remove_all_devs(binfo->pdev);
> >> +
> >> +       /*
> >> +        * it is a valid id, free it. See comments in
> >> +        * build_info_create_dev()
> >> +        */
> >> +       if (binfo->feature_dev && binfo->feature_dev->id >= 0)
> >> +               free_fpga_id(feature_dev_id_type(binfo->feature_dev),
> >> +                            binfo->feature_dev->id);
> >> +
> >> +       platform_device_put(binfo->feature_dev);
> >> +
> >> +       devm_kfree(&binfo->pdev->dev, binfo);
> >> +}
> >> +
> >> +#define FEATURE_TYPE_AFU       0x1
> >> +#define FEATURE_TYPE_PRIVATE   0x3
> >> +
> >> +/* FME and PORT GUID are fixed */
> >> +#define FEATURE_FME_GUID "f9e17764-38f0-82fe-e346-524ae92aafbf"
> >> +#define FEATURE_PORT_GUID "6b355b87-b06c-9642-eb42-8d139398b43a"
> >> +
> >> +static bool feature_is_fme(struct feature_afu_header *afu_hdr)
> >> +{
> >> +       uuid_le u;
> >> +
> >> +       uuid_le_to_bin(FEATURE_FME_GUID, &u);
> >> +
> >> +       return !uuid_le_cmp(u, afu_hdr->guid);
> >> +}
> >> +
> >> +static bool feature_is_port(struct feature_afu_header *afu_hdr)
> >> +{
> >> +       uuid_le u;
> >> +
> >> +       uuid_le_to_bin(FEATURE_PORT_GUID, &u);
> >> +
> >> +       return !uuid_le_cmp(u, afu_hdr->guid);
> >> +}
> >> +
> >> +/*
> >> + * UAFU GUID is dynamic as it can be changed after FME downloads different
> >> + * Green Bitstream to the port, so we treat the unknown GUIDs which are
> >> + * attached on port's feature list as UAFU.
> >> + */
> >> +static bool feature_is_UAFU(struct build_feature_devs_info *binfo)
> >> +{
> >> +       if (!binfo->feature_dev ||
> >> +             feature_dev_id_type(binfo->feature_dev) != PORT_ID)
> >> +               return false;
> >> +
> >> +       return true;
> >> +}
> >> +
> >> +static void
> >> +build_info_add_sub_feature(struct build_feature_devs_info *binfo,
> >> +                          int feature_id, const char *feature_name,
> >> +                          resource_size_t resource_size, void __iomem *start)
> >> +{
> >> +
> >> +       struct platform_device *fdev = binfo->feature_dev;
> >> +       struct feature_platform_data *pdata = dev_get_platdata(&fdev->dev);
> >> +       struct resource *res = &fdev->resource[feature_id];
> >> +
> >> +       res->start = pci_resource_start(binfo->pdev, binfo->current_bar) +
> >> +               start - binfo->ioaddr;
> >> +       res->end = res->start + resource_size - 1;
> >> +       res->flags = IORESOURCE_MEM;
> >> +       res->name = feature_name;
> >> +
> >> +       feature_platform_data_add(pdata, feature_id,
> >> +                                 feature_name, feature_id, start);
> >> +}
> >> +
> >> +struct feature_info {
> >> +       const char *name;
> >> +       resource_size_t resource_size;
> >> +       int feature_index;
> >> +};
> >> +
> >> +/* indexed by fme feature IDs which are defined in 'enum fme_feature_id'. */
> >> +static struct feature_info fme_features[] = {
> >> +       {
> >> +               .name = FME_FEATURE_HEADER,
> >> +               .resource_size = sizeof(struct feature_fme_header),
> >> +               .feature_index = FME_FEATURE_ID_HEADER,
> >> +       },
> >> +       {
> >> +               .name = FME_FEATURE_THERMAL_MGMT,
> >> +               .resource_size = sizeof(struct feature_fme_thermal),
> >> +               .feature_index = FME_FEATURE_ID_THERMAL_MGMT,
> >> +       },
> >> +       {
> >> +               .name = FME_FEATURE_POWER_MGMT,
> >> +               .resource_size = sizeof(struct feature_fme_power),
> >> +               .feature_index = FME_FEATURE_ID_POWER_MGMT,
> >> +       },
> >> +       {
> >> +               .name = FME_FEATURE_GLOBAL_PERF,
> >> +               .resource_size = sizeof(struct feature_fme_gperf),
> >> +               .feature_index = FME_FEATURE_ID_GLOBAL_PERF,
> >> +       },
> >> +       {
> >> +               .name = FME_FEATURE_GLOBAL_ERR,
> >> +               .resource_size = sizeof(struct feature_fme_err),
> >> +               .feature_index = FME_FEATURE_ID_GLOBAL_ERR,
> >> +       },
> >> +       {
> >> +               .name = FME_FEATURE_PR_MGMT,
> >> +               .resource_size = sizeof(struct feature_fme_pr),
> >> +               .feature_index = FME_FEATURE_ID_PR_MGMT,
> >> +       }
> >> +};
> >> +
> >> +/* indexed by port feature IDs which are defined in 'enum port_feature_id'. */
> >> +static struct feature_info port_features[] = {
> >> +       {
> >> +               .name = PORT_FEATURE_HEADER,
> >> +               .resource_size = sizeof(struct feature_port_header),
> >> +               .feature_index = PORT_FEATURE_ID_HEADER,
> >> +       },
> >> +       {
> >> +               .name = PORT_FEATURE_ERR,
> >> +               .resource_size = sizeof(struct feature_port_error),
> >> +               .feature_index = PORT_FEATURE_ID_ERROR,
> >> +       },
> >> +       {
> >> +               .name = PORT_FEATURE_UMSG,
> >> +               .resource_size = sizeof(struct feature_port_umsg),
> >> +               .feature_index = PORT_FEATURE_ID_UMSG,
> >> +       },
> >> +       {
> >> +               /* This feature isn't available for now */
> >> +               .name = PORT_FEATURE_PR,
> >> +               .resource_size = 0,
> >> +               .feature_index = PORT_FEATURE_ID_PR,
> >> +       },
> >> +       {
> >> +               .name = PORT_FEATURE_STP,
> >> +               .resource_size = sizeof(struct feature_port_stp),
> >> +               .feature_index = PORT_FEATURE_ID_STP,
> >> +       },
> >> +       {
> >> +               /*
> >> +                * For User AFU feature, its region size is not fixed, but
> >> +                * reported by register PortCapability.mmio_size. Resource
> >> +                * size of UAFU will be set while parse port device.
> >> +                */
> >> +               .name = PORT_FEATURE_UAFU,
> >> +               .resource_size = 0,
> >> +               .feature_index = PORT_FEATURE_ID_UAFU,
> >> +       },
> >> +};
> >> +
> >> +static int
> >> +create_feature_instance(struct build_feature_devs_info *binfo,
> >> +                       void __iomem *start, struct feature_info *finfo)
> >> +{
> >> +       if (binfo->ioend - start < finfo->resource_size)
> >> +               return -EINVAL;
> >> +
> >> +       build_info_add_sub_feature(binfo, finfo->feature_index, finfo->name,
> >> +                                  finfo->resource_size, start);
> >> +       return 0;
> >> +}
> >
> > build_info_add_sub_feature() is only called one time - here.  Could
> > you collapse these two functions together?
> >
> >> +
> >> +static int parse_feature_fme(struct build_feature_devs_info *binfo,
> >> +                            void __iomem *start)
> >> +{
> >> +       struct cci_drvdata *drvdata = dev_get_drvdata(&binfo->pdev->dev);
> >> +       int ret;
> >> +
> >> +       ret = build_info_create_dev(binfo, FME_ID, fme_feature_num(),
> >> +                                       FPGA_FEATURE_DEV_FME);
> >> +       if (ret)
> >> +               return ret;
> >> +
> >> +       if (drvdata->fme_dev) {
> >> +               dev_err(&binfo->pdev->dev, "Multiple FMEs are detected.\n");
> >> +               return -EINVAL;
> >> +       }
> >> +
> >> +       return create_feature_instance(binfo, start,
> >> +                                      &fme_features[FME_FEATURE_ID_HEADER]);
> >> +}
> >> +
> >> +static int parse_feature_fme_private(struct build_feature_devs_info *binfo,
> >> +                                    struct feature_header *hdr)
> >> +{
> >> +       struct feature_header header;
> >> +
> >> +       header.csr = readq(hdr);
> >> +
> >> +       if (header.id >= ARRAY_SIZE(fme_features)) {
> >> +               dev_info(&binfo->pdev->dev, "FME feature id %x is not supported yet.\n",
> >> +                        header.id);
> >> +               return 0;
> >> +       }
> >> +
> >> +       return create_feature_instance(binfo, hdr, &fme_features[header.id]);
> >> +}
> >> +
> >> +static int parse_feature_port(struct build_feature_devs_info *binfo,
> >> +                            void __iomem *start)
> >> +{
> >> +       int ret;
> >> +
> >> +       ret = build_info_create_dev(binfo, PORT_ID, port_feature_num(),
> >> +                                       FPGA_FEATURE_DEV_PORT);
> >> +       if (ret)
> >> +               return ret;
> >> +
> >> +       return create_feature_instance(binfo, start,
> >> +                                      &port_features[PORT_FEATURE_ID_HEADER]);
> >> +}
> >> +
> >> +static void enable_port_uafu(struct build_feature_devs_info *binfo,
> >> +                            void __iomem *start)
> >> +{
> >> +       enum port_feature_id id = PORT_FEATURE_ID_UAFU;
> >> +       struct feature_port_header *port_hdr;
> >> +       struct feature_port_capability capability;
> >> +
> >> +       port_hdr = (struct feature_port_header *)start;
> >> +       capability.csr = readq(&port_hdr->capability);
> >> +       port_features[id].resource_size = capability.mmio_size << 10;
> >> +
> >> +       /*
> >> +        * To enable User AFU, driver needs to clear reset bit on related port,
> >> +        * otherwise the mmio space of this user AFU will be invalid.
> >> +        */
> >> +       if (port_features[id].resource_size)
> >> +               fpga_port_reset(binfo->feature_dev);
> >> +}
> >> +
> >> +static int parse_feature_port_private(struct build_feature_devs_info *binfo,
> >> +                                     struct feature_header *hdr)
> >> +{
> >> +       struct feature_header header;
> >> +       enum port_feature_id id;
> >> +
> >> +       header.csr = readq(hdr);
> >> +       /*
> >> +        * the region of port feature id is [0x10, 0x13], + 1 to reserve 0
> >> +        * which is dedicated for port-hdr.
> >> +        */
> >> +       id = (header.id & 0x000f) + 1;
> >> +
> >> +       if (id >= ARRAY_SIZE(port_features)) {
> >> +               dev_info(&binfo->pdev->dev, "Port feature id %x is not supported yet.\n",
> >> +                        header.id);
> >> +               return 0;
> >> +       }
> >> +
> >> +       return create_feature_instance(binfo, hdr, &port_features[id]);
> >> +}
> >> +
> >> +static int parse_feature_port_uafu(struct build_feature_devs_info *binfo,
> >> +                                struct feature_header *hdr)
> >> +{
> >> +       enum port_feature_id id = PORT_FEATURE_ID_UAFU;
> >> +       int ret;
> >> +
> >> +       if (port_features[id].resource_size) {
> >> +               ret = create_feature_instance(binfo, hdr, &port_features[id]);
> >> +               port_features[id].resource_size = 0;
> >> +       } else {
> >> +               dev_err(&binfo->pdev->dev, "the uafu feature header is mis-configured.\n");
> >> +               ret = -EINVAL;
> >> +       }
> >> +
> >> +       return ret;
> >> +}
> >> +
> >> +static int parse_feature_afus(struct build_feature_devs_info *binfo,
> >> +                             struct feature_header *hdr)
> >> +{
> >> +       int ret;
> >> +       struct feature_afu_header *afu_hdr, header;
> >> +       void __iomem *start;
> >> +       void __iomem *end = binfo->ioend;
> >> +
> >> +       start = hdr;
> >> +       for (; start < end; start += header.next_afu) {
> >> +               if (end - start < (sizeof(*afu_hdr) + sizeof(*hdr)))
> >> +                       return -EINVAL;
> >> +
> >> +               hdr = start;
> >> +               afu_hdr = (struct feature_afu_header *) (hdr + 1);
> >> +               header.csr = readq(&afu_hdr->csr);
> >> +
> >> +               if (feature_is_fme(afu_hdr)) {
> >> +                       ret = parse_feature_fme(binfo, hdr);
> >> +                       binfo->pfme_hdr = hdr;
> >> +                       if (ret)
> >> +                               return ret;
> >> +               } else if (feature_is_port(afu_hdr)) {
> >> +                       ret = parse_feature_port(binfo, hdr);
> >> +                       enable_port_uafu(binfo, hdr);
> >> +                       if (ret)
> >> +                               return ret;
> >> +               } else if (feature_is_UAFU(binfo)) {
> >> +                       ret = parse_feature_port_uafu(binfo, hdr);
> >> +                       if (ret)
> >> +                               return ret;
> >> +               } else
> >> +                       dev_info(&binfo->pdev->dev, "AFU GUID %pUl is not supported yet.\n",
> >> +                                afu_hdr->guid.b);
> >> +
> >> +               if (!header.next_afu)
> >> +                       break;
> >> +       }
> >> +
> >> +       return 0;
> >> +}
> >> +
> >> +static int parse_feature_private(struct build_feature_devs_info *binfo,
> >> +                                struct feature_header *hdr)
> >> +{
> >> +       struct feature_header header;
> >> +
> >> +       header.csr = readq(hdr);
> >> +
> >> +       if (!binfo->feature_dev) {
> >> +               dev_err(&binfo->pdev->dev, "the private feature %x does not belong to any AFU.\n",
> >> +                       header.id);
> >> +               return -EINVAL;
> >> +       }
> >> +
> >> +       switch (feature_dev_id_type(binfo->feature_dev)) {
> >> +       case FME_ID:
> >> +               return parse_feature_fme_private(binfo, hdr);
> >> +       case PORT_ID:
> >> +               return parse_feature_port_private(binfo, hdr);
> >> +       default:
> >> +               dev_info(&binfo->pdev->dev, "private feature %x belonging to AFU %s is not supported yet.\n",
> >> +                        header.id, binfo->feature_dev->name);
> >> +       }
> >> +       return 0;
> >> +}
> >> +
> >> +static int parse_feature(struct build_feature_devs_info *binfo,
> >> +                        struct feature_header *hdr)
> >> +{
> >> +       struct feature_header header;
> >> +       int ret = 0;
> >> +
> >> +       header.csr = readq(hdr);
> >> +
> >> +       switch (header.type) {
> >> +       case FEATURE_TYPE_AFU:
> >> +               ret = parse_feature_afus(binfo, hdr);
> >> +               break;
> >> +       case FEATURE_TYPE_PRIVATE:
> >> +               ret = parse_feature_private(binfo, hdr);
> >> +               break;
> >> +       default:
> >> +               dev_info(&binfo->pdev->dev,
> >> +                        "Feature Type %x is not supported.\n", hdr->type);
> >> +       };
> >> +
> >> +       return ret;
> >> +}
> >> +
> >> +static int
> >> +parse_feature_list(struct build_feature_devs_info *binfo, void __iomem *start)
> >> +{
> >> +       struct feature_header *hdr, header;
> >> +       void __iomem *end = binfo->ioend;
> >> +       int ret = 0;
> >> +
> >
> > Maybe a helpful comment that we are stepping through a linked list of features.
> >
> >> +       for (; start < end; start += header.next_header_offset) {
> >> +               if (end - start < sizeof(*hdr)) {
> >> +                       dev_err(&binfo->pdev->dev, "The region is too small to contain a feature.\n");
> >> +                       ret =  -EINVAL;
> >> +                       break;
> >> +               }
> >> +
> >> +               hdr = (struct feature_header *)start;
> >> +               ret = parse_feature(binfo, hdr);
> >> +               if (ret)
> >> +                       break;
> >
> > Instead of parse_feature, this can save off the enumeration info and
> > continue to read in the linked list.  After all the headers are read
> > in, call the (separate) enumeration code to step through the saved
> > headers, parse them, and create the devices.  Since the memory is
> > iomapped during the process of reading in the headers, the enumeration
> > code doesn't have to be so pcie specific.  Then this code base is
> > better set to run on embedded devices also.
> >
> >> +
> >> +               header.csr = readq(hdr);
> >> +               if (!header.next_header_offset)
> >> +                       break;
> >> +       }
> >> +
> >> +       return ret;
> >> +}
> >> +
> >> +static int parse_ports_from_fme(struct build_feature_devs_info *binfo)
> >> +{
> >> +       struct feature_fme_header *fme_hdr;
> >> +       struct feature_fme_port port;
> >> +       int i = 0, ret = 0;
> >> +
> >> +       if (binfo->pfme_hdr == NULL) {
> >> +               dev_dbg(&binfo->pdev->dev, "VF is detected.\n");
> >> +               return ret;
> >> +       }
> >> +
> >> +       fme_hdr = binfo->pfme_hdr;
> >> +
> >> +       do {
> >> +               port.csr = readq(&fme_hdr->port[i]);
> >> +               if (!port.port_implemented)
> >> +                       break;
> >> +
> >> +               ret = parse_switch_to(binfo, port.port_bar);
> >> +               if (ret)
> >> +                       break;
> >> +
> >> +               ret = parse_feature_list(binfo,
> >> +                               binfo->ioaddr + port.port_offset);
> >> +               if (ret)
> >> +                       break;
> >> +       } while (++i < MAX_FPGA_PORT_NUM);
> >> +
> >> +       return ret;
> >> +}
> >> +
> >> +static int create_init_drvdata(struct pci_dev *pdev)
> >> +{
> >> +       struct cci_drvdata *drvdata;
> >> +
> >> +       drvdata = devm_kzalloc(&pdev->dev, sizeof(*drvdata), GFP_KERNEL);
> >> +       if (!drvdata)
> >> +               return -ENOMEM;
> >> +
> >> +       mutex_init(&drvdata->lock);
> >> +       INIT_LIST_HEAD(&drvdata->port_dev_list);
> >> +       INIT_LIST_HEAD(&drvdata->regions);
> >> +
> >> +       dev_set_drvdata(&pdev->dev, drvdata);
> >> +       return 0;
> >> +}
> >> +
> >> +static void destroy_drvdata(struct pci_dev *pdev)
> >> +{
> >> +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> >> +
> >> +       if (drvdata->fme_dev) {
> >> +               /* fme device should be unregistered first. */
> >> +               WARN_ON(device_is_registered(drvdata->fme_dev));
> >> +               free_fpga_id(FME_ID, to_platform_device(drvdata->fme_dev)->id);
> >> +               put_device(drvdata->fme_dev);
> >> +       }
> >> +
> >> +       cci_pci_remove_port_devs(pdev);
> >> +       cci_pci_release_regions(pdev);
> >> +       dev_set_drvdata(&pdev->dev, NULL);
> >> +       devm_kfree(&pdev->dev, drvdata);
> >> +}
> >> +
> >> +static int cci_pci_create_feature_devs(struct pci_dev *pdev)
> >> +{
> >> +       struct build_feature_devs_info *binfo;
> >> +       int ret;
> >> +
> >> +       binfo = build_info_alloc_and_init(pdev);
> >> +       if (!binfo)
> >> +               return -ENOMEM;
> >> +
> >> +       binfo->parent_dev = fpga_dev_create(&pdev->dev, INTEL_FPGA_DEV);
> >> +       if (IS_ERR(binfo->parent_dev)) {
> >> +               ret = PTR_ERR(binfo->parent_dev);
> >> +               goto free_binfo_exit;
> >> +       }
> >> +
> >> +       ret = parse_start(binfo);
> >> +       if (ret)
> >> +               goto free_binfo_exit;
> >> +
> >> +       ret = parse_feature_list(binfo, binfo->ioaddr);
> >> +       if (ret)
> >> +               goto free_binfo_exit;
> >> +
> >> +       ret = parse_ports_from_fme(binfo);
> >> +       if (ret)
> >> +               goto free_binfo_exit;
> >
> > So ideally, there would be a function call here that read all the
> > headers from hardware, ioremapping as it went along.  Then after that,
> > call the enumeration code to create the devices.
> >
> >> +
> >> +       ret = build_info_commit_dev(binfo);
> >> +       if (ret)
> >> +               goto free_binfo_exit;
> >> +
> >> +       /*
> >> +        * everything is okay, reset ->parent_dev to stop it being
> >> +        * freed by build_info_free()
> >> +        */
> >> +       binfo->parent_dev = NULL;
> >> +
> >> +free_binfo_exit:
> >> +       build_info_free(binfo);
> >> +       return ret;
> >> +}
> >> +
> >>  /* PCI Device ID */
> >>  #define PCIe_DEVICE_ID_PF_INT_5_X      0xBCBD
> >>  #define PCIe_DEVICE_ID_PF_INT_6_X      0xBCC0
> >> @@ -81,9 +898,18 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
> >>                 goto release_region_exit;
> >>         }
> >>
> >> -       /* TODO: create and add the platform device per feature list */
> >> +       ret = create_init_drvdata(pcidev);
> >> +       if (ret)
> >> +               goto release_region_exit;
> >> +
> >> +       ret = cci_pci_create_feature_devs(pcidev);
> >> +       if (ret)
> >> +               goto destroy_drvdata_exit;
> >> +
> >>         return 0;
> >>
> >> +destroy_drvdata_exit:
> >> +       destroy_drvdata(pcidev);
> >>  release_region_exit:
> >>         pci_release_regions(pcidev);
> >>  disable_error_report_exit:
> >> @@ -94,6 +920,8 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
> >>
> >>  static void cci_pci_remove(struct pci_dev *pcidev)
> >>  {
> >> +       remove_all_devs(pcidev);
> >> +       destroy_drvdata(pcidev);
> >>         pci_release_regions(pcidev);
> >>         pci_disable_pcie_error_reporting(pcidev);
> >>         pci_disable_device(pcidev);
> >> @@ -108,14 +936,23 @@ static void cci_pci_remove(struct pci_dev *pcidev)
> >>
> >>  static int __init ccidrv_init(void)
> >>  {
> >> +       int ret;
> >> +
> >>         pr_info("Intel(R) FPGA PCIe Driver: Version %s\n", DRV_VERSION);
> >>
> >> -       return pci_register_driver(&cci_pci_driver);
> >> +       fpga_ids_init();
> >> +
> >> +       ret = pci_register_driver(&cci_pci_driver);
> >> +       if (ret)
> >> +               fpga_ids_destroy();
> >> +
> >> +       return ret;
> >>  }
> >>
> >>  static void __exit ccidrv_exit(void)
> >>  {
> >>         pci_unregister_driver(&cci_pci_driver);
> >> +       fpga_ids_destroy();
> >>  }
> >>
> >>  module_init(ccidrv_init);
> >> --
> >> 1.8.3.1
> >>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 03/22] fpga: bridge: remove OF dependency for fpga-bridge

[Alan Tull]

On Thu, Sep 21, 2017 at 9:15 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> On Thu, Sep 21, 2017 at 02:50:07PM -0500, Alan Tull wrote:
>> On Thu, Sep 21, 2017 at 2:11 PM, Moritz Fischer <mdf-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>>
>> Hi Moritz,
>>
>> > Hi,
>> >
>> > On Mon, Jun 26, 2017 at 09:51:59AM +0800, Wu Hao wrote:
>> >> This patch removes OF dependency of fpga-bridge, it allows drivers
>> >> to use fpga-bridge class without device tree support.
>> >>
>> >> Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> >> ---
>> >>  drivers/fpga/Kconfig | 1 -
>> >>  1 file changed, 1 deletion(-)
>> >>
>> >> diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
>> >> index ed600d5..c1d8f41 100644
>> >> --- a/drivers/fpga/Kconfig
>> >> +++ b/drivers/fpga/Kconfig
>> >> @@ -75,7 +75,6 @@ config FPGA_MGR_ZYNQ_FPGA
>> >>
>> >>  config FPGA_BRIDGE
>> >>       tristate "FPGA Bridge Framework"
>> >> -     depends on OF
>> >
>> > Shouldn't that dependency be removed in the patch that removes the
>> > dependency, or was the dependency never there in the first place?
>>
>> It wasn't dependent on OF in the first place.
>
> Hi Alan / Moritz
>
> I see this change has been integrated into below patch from Alan.
>
> [PATCH v4 15/18] fpga: region: move device tree support to of-fpga-region.c
>
> so I think I can drop this patch in my next version patch set. : )

Hi Hao,

Yes, I went ahead and pulled it into my patchset so you can drop it.

Thanks for the patch!

Alan

>
> Thanks
> Hao
>
>>
>> Thanks,
>> Alan
>>
>> >
>> >>       help
>> >>         Say Y here if you want to support bridges connected between host
>> >>         processors and FPGAs or between FPGAs.
>> >> --
>> >> 1.8.3.1
>> >>
>> >
>> > Thanks,
>> >
>> > Moritz
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 03/22] fpga: bridge: remove OF dependency for fpga-bridge

[Moritz Fischer]

[-- Attachment #1: Type: text/plain, Size: 388 bytes --]

On Wed, Aug 02, 2017 at 04:21:19PM -0500, Alan Tull wrote:
> On Sun, Jun 25, 2017 at 8:51 PM, Wu Hao <hao.wu@intel.com> wrote:
> > This patch removes OF dependency of fpga-bridge, it allows drivers
> > to use fpga-bridge class without device tree support.
> >
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> Acked-by: Alan Tull <atull@kernel.org>
Acked-by: Moritz Fischer <mdf@kernel.org>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v2 15/22] fpga: intel: add fpga manager platform driver for FME

[Moritz Fischer]

[-- Attachment #1: Type: text/plain, Size: 15273 bytes --]

Hi Hao,

On Mon, Jun 26, 2017 at 09:52:11AM +0800, Wu Hao wrote:
> This patch adds fpga manager driver for Intel FPGA Management
> Engine(FME). It implements fpga_manager_ops for FPGA Partial
> Reconfiguration function.
> 
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ---
>  .../ABI/testing/sysfs-platform-intel-fpga-fme-mgr  |   8 +
>  drivers/fpga/Kconfig                               |   7 +
>  drivers/fpga/Makefile                              |   1 +
>  drivers/fpga/intel-feature-dev.h                   |  75 +++++
>  drivers/fpga/intel-fpga-fme-mgr.c                  | 307 +++++++++++++++++++++
>  5 files changed, 398 insertions(+)
>  create mode 100644 Documentation/ABI/testing/sysfs-platform-intel-fpga-fme-mgr
>  create mode 100644 drivers/fpga/intel-fpga-fme-mgr.c
> 
> diff --git a/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme-mgr b/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme-mgr
> new file mode 100644
> index 0000000..40771fb
> --- /dev/null
> +++ b/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme-mgr
> @@ -0,0 +1,8 @@
> +What:		/sys/bus/platform/devices/intel-fpga-fme-mgr.0/interface_id
> +Date:		June 2017
> +KernelVersion:  4.12
> +Contact:	Wu Hao <hao.wu@intel.com>
> +Description:	Read-only. It returns interface id of partial reconfiguration
> +		hardware. Userspace could use this information to check if
> +		current hardware is compatible with given image before FPGA
> +		programming.
> diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> index b91458f..6f2f623 100644
> --- a/drivers/fpga/Kconfig
> +++ b/drivers/fpga/Kconfig
> @@ -151,6 +151,13 @@ config INTEL_FPGA_FME
>  	  all FPGA platform level management features. There shall be 1
>  	  FME per Intel FPGA.
>  
> +config INTEL_FPGA_FME_MGR
> +	tristate "Intel FPGA FME Manager Driver"
> +	depends on INTEL_FPGA_FME
> +	help
> +	  Say Y to enable FPGA Manager driver for Intel FPGA Management
> +	  Engine.
> +
>  endif
>  
>  endif # FPGA
> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> index acda5ca..d1d588b 100644
> --- a/drivers/fpga/Makefile
> +++ b/drivers/fpga/Makefile
> @@ -31,6 +31,7 @@ obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
>  # Intel FPGA Support
>  obj-$(CONFIG_INTEL_FPGA_PCI)		+= intel-fpga-pci.o
>  obj-$(CONFIG_INTEL_FPGA_FME)		+= intel-fpga-fme.o
> +obj-$(CONFIG_INTEL_FPGA_FME_MGR)	+= intel-fpga-fme-mgr.o
>  
>  intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
>  intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
> diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
> index 3f97b75..f33923b 100644
> --- a/drivers/fpga/intel-feature-dev.h
> +++ b/drivers/fpga/intel-feature-dev.h
> @@ -164,8 +164,83 @@ struct feature_fme_err {
>  };
>  
>  /* FME Partial Reconfiguration Sub Feature Register Set */
> +/* FME PR Control Register */
> +struct feature_fme_pr_ctl {
> +	union {
> +		u64 csr;
> +		struct {
> +			u64 pr_reset:1;		/* Reset PR Engine */
> +			u64 rsvdz1:3;
> +			u64 pr_reset_ack:1;	/* Reset PR Engine Ack */
> +			u64 rsvdz2:3;
> +			u64 pr_regionid:2;	/* PR Region ID */
> +			u64 rsvdz3:2;
> +			u64 pr_start_req:1;	/* PR Start Request */
> +			u64 pr_push_complete:1;	/* PR Data push complete */
> +			u64 pr_kind:1;		/* Load Customer or Intel GBS */
> +			u64 rsvdz4:17;
> +			u64 config_data:32;
> +		};
> +	};
> +};
> +
> +/* FME PR Status Register */
> +struct feature_fme_pr_status {
> +	union {
> +		u64 csr;
> +		struct {
> +			u64 pr_credit:9;	/* Number of PR Credits */
> +			u64 rsvdz1:7;
> +			u64 pr_status:1;	/* PR Operation status */
> +			u64 rsvdz2:3;
> +			u64 pr_ctrlr_status:3;	/* Controller status */
> +			u64 rsvdz3:1;
> +			u64 pr_host_status:4;	/* PR Host status */
> +			u64 rsvdz4:36;
> +		};
> +	};
> +};
> +
> +/* FME PR Data Register */
> +struct feature_fme_pr_data {
> +	union {
> +		u64 csr;
> +		struct {
> +			/* PR data from the raw-binary file */
> +			u64 pr_data_raw:32;
> +			u64 rsvd:32;
> +		};
> +	};
> +};
> +
> +/* FME PR Error Register */
> +struct feature_fme_pr_error {
> +	union {
> +		u64 csr;
> +		struct {
> +			u64 operation_err:1;	/* Previous PR error detected */
> +			u64 crc_err:1;		/* CRC error detected */
> +			u64 incompatiable_bs:1; /* Incompatiable Bitstream */
> +			u64 protocol_err:1;	/* Data push protocol error */
> +			u64 fifo_overflow:1;	/* Data fifo overflow */
> +			u64 rsvdz1:1;
> +			u64 secure_load_err:1;	/* Secure PR load error */
> +			u64 rsvdz2:57;
> +		};
> +	};
> +};

From what I've seen [1], [2] people weren't too happy with using bitfields,
any reason we can't use explicit masking and shifts? I don't know if
this is still an issue.

> +
>  struct feature_fme_pr {
>  	struct feature_header header;
> +	struct feature_fme_pr_ctl control;
> +	struct feature_fme_pr_status status;
> +	struct feature_fme_pr_data data;
> +	struct feature_fme_pr_error error;
> +
> +	u64 rsvd[16];
> +
> +	u64 intfc_id_l;		/* PR interface Id Low */
> +	u64 intfc_id_h;		/* PR interface Id High */
>  };
>  
>  /* PORT Header Register Set */
> diff --git a/drivers/fpga/intel-fpga-fme-mgr.c b/drivers/fpga/intel-fpga-fme-mgr.c
> new file mode 100644
> index 0000000..5f93dfb
> --- /dev/null
> +++ b/drivers/fpga/intel-fpga-fme-mgr.c
> @@ -0,0 +1,307 @@
> +/*
> + * FPGA Manager Driver for Intel FPGA Management Engine (FME)
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <luwei.kang@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *   Wu Hao <hao.wu@intel.com>
> + *   Joseph Grecco <joe.grecco@intel.com>
> + *   Enno Luebbers <enno.luebbers@intel.com>
> + *   Tim Whisonant <tim.whisonant@intel.com>
> + *   Ananda Ravuri <ananda.ravuri@intel.com>
> + *   Christopher Rauer <christopher.rauer@intel.com>
> + *   Henry Mitchel <henry.mitchel@intel.com>
> + *
> + * This work is licensed under the terms of the GNU GPL version 2. See
> + * the COPYING file in the top-level directory.
> + */
> +
> +#include <linux/module.h>
> +#include <linux/iopoll.h>
> +#include <linux/fpga/fpga-mgr.h>
> +
> +#include "intel-feature-dev.h"
> +#include "intel-fme.h"
> +
> +#define PR_WAIT_TIMEOUT   8000000
> +#define PR_HOST_STATUS_IDLE	0
> +
> +struct fme_mgr_priv {
> +	void __iomem *ioaddr;
> +};
> +
> +static ssize_t interface_id_show(struct device *dev,
> +				 struct device_attribute *attr, char *buf)
> +{
> +	struct fpga_manager *mgr = dev_get_drvdata(dev);
> +	struct fme_mgr_priv *priv = mgr->priv;
> +	struct feature_fme_pr *fme_pr;
> +	u64 intfc_id_l, intfc_id_h;
> +
> +	fme_pr = priv->ioaddr;
> +
> +	intfc_id_l = readq(&fme_pr->intfc_id_l);
> +	intfc_id_h = readq(&fme_pr->intfc_id_h);
> +
> +	return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> +			(unsigned long long)intfc_id_h,
> +			(unsigned long long)intfc_id_l);
> +}
> +static DEVICE_ATTR_RO(interface_id);
> +
> +static const struct attribute *fme_mgr_attrs[] = {
> +	&dev_attr_interface_id.attr,
> +	NULL,
> +};
> +
> +static void fme_mgr_pr_update_status(struct fpga_manager *mgr,
> +				     struct feature_fme_pr_error err)
> +{
> +	mgr->status = 0;
> +
> +	if (err.operation_err)
> +		mgr->status |= FPGA_MGR_STATUS_OPERATION_ERR;

Can we just have a (err & FEATURE_FME_PR_ERROR) here instead ?
> +	if (err.crc_err)
> +		mgr->status |= FPGA_MGR_STATUS_CRC_ERR;

Can we just have a (err & FEATURE_FME_CRC_ERROR) here instead ?
> +	if (err.incompatiable_bs)
> +		mgr->status |= FPGA_MGR_STATUS_INCOMPATIBLE_BS_ERR;
> +	if (err.protocol_err)
> +		mgr->status |= FPGA_MGR_STATUS_IP_PROTOCOL_ERR;
> +	if (err.fifo_overflow)
> +		mgr->status |= FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR;
> +	if (err.secure_load_err)
> +		mgr->status |= FPGA_MGR_STATUS_SECURE_LOAD_ERR;
> +}
> +
> +static u64 fme_mgr_pr_err_handle(struct feature_fme_pr *fme_pr)
> +{
> +	struct feature_fme_pr_status fme_pr_status;
> +	struct feature_fme_pr_error fme_pr_error;
> +
> +	fme_pr_status.csr = readq(&fme_pr->status);
> +	if (!fme_pr_status.pr_status)
> +		return 0;
> +
> +	fme_pr_error.csr = readq(&fme_pr->error);
> +	writeq(fme_pr_error.csr, &fme_pr->error);
> +
> +	return fme_pr_error.csr;
> +}
> +
> +static int fme_mgr_write_init(struct fpga_manager *mgr,
> +			      struct fpga_image_info *info,
> +			      const char *buf, size_t count)
> +{
> +	struct device *dev = &mgr->dev;
> +	struct fme_mgr_priv *priv = mgr->priv;
> +	struct feature_fme_pr *fme_pr = priv->ioaddr;
> +	struct feature_fme_pr_ctl fme_pr_ctl;
> +	struct feature_fme_pr_status fme_pr_status;
> +	struct feature_fme_pr_error fme_pr_err;
> +
> +	if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
> +		dev_err(dev, "only support partial reconfiguration.\n");
> +		return -EINVAL;
> +	}
> +
> +	dev_dbg(dev, "resetting PR before initiated PR\n");
> +
> +	fme_pr_ctl.csr = readq(&fme_pr->control);
> +	fme_pr_ctl.pr_reset = 1;
> +	writeq(fme_pr_ctl.csr, &fme_pr->control);
> +
> +	if (readq_poll_timeout(&fme_pr->control, fme_pr_ctl.csr,
> +			       (fme_pr_ctl.pr_reset_ack == 1),
> +			       1, PR_WAIT_TIMEOUT)) {
> +		dev_err(dev, "maximum PR timeout\n");
> +		return -ETIMEDOUT;
> +	}
> +
> +	fme_pr_ctl.csr = readq(&fme_pr->control);
> +	fme_pr_ctl.pr_reset = 0;
> +	writeq(fme_pr_ctl.csr, &fme_pr->control);
> +
> +	dev_dbg(dev,
> +		"waiting for PR resource in HW to be initialized and ready\n");
> +
> +	fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
> +
> +	if (readq_poll_timeout(&fme_pr->status, fme_pr_status.csr,
> +			(fme_pr_status.pr_host_status == PR_HOST_STATUS_IDLE),
> +			1, PR_WAIT_TIMEOUT)) {
> +		dev_err(dev, "maximum PR timeout\n");
> +		return -ETIMEDOUT;
> +	}
> +
> +	dev_dbg(dev, "check and clear previous PR error\n");
> +	fme_pr_err.csr = fme_mgr_pr_err_handle(fme_pr);
> +	if (fme_pr_err.csr)
> +		dev_dbg(dev, "previous PR error detected %llx\n",
> +			     (unsigned long long)fme_pr_err.csr);
> +
> +	/* Clear all PR errors */
> +	fme_pr_err.csr = 0;
> +	fme_mgr_pr_update_status(mgr, fmCRCe_pr_err);
> +
> +	dev_dbg(dev, "set PR port ID\n");
> +
> +	fme_pr_ctl.csr = readq(&fme_pr->control);
> +	fme_pr_ctl.pr_regionid = info->region_id;
> +	writeq(fme_pr_ctl.csr, &fme_pr->control);
> +
> +	return 0;
> +}
> +
> +static int fme_mgr_write(struct fpga_manager *mgr,
> +			 const char *buf, size_t count)
> +{
> +	struct device *dev = &mgr->dev;
> +	struct fme_mgr_priv *priv = mgr->priv;
> +	struct feature_fme_pr *fme_pr = priv->ioaddr;
> +	struct feature_fme_pr_ctl fme_pr_ctl;
> +	struct feature_fme_pr_status fme_pr_status;
> +	struct feature_fme_pr_data fme_pr_data;
> +	int delay, pr_credit, i = 0;
> +
> +	dev_dbg(dev, "start request\n");
> +
> +	fme_pr_ctl.csr = readq(&fme_pr->control);
> +	fme_pr_ctl.pr_start_req = 1;
> +	writeq(fme_pr_ctl.csr, &fme_pr->control);
> +
> +	dev_dbg(dev, "pushing data from bitstream to HW\n");
> +
> +	fme_pr_status.csr = readq(&fme_pr->status);
> +	pr_credit = fme_pr_status.pr_credit;
> +
> +	while (count > 0) {
> +		delay = 0;
> +		while (pr_credit <= 1) {
> +			if (delay++ > PR_WAIT_TIMEOUT) {
> +				dev_err(dev, "maximum try\n");
> +				return -ETIMEDOUT;
> +			}
> +			udelay(1);
> +
> +			fme_pr_status.csr = readq(&fme_pr->status);
> +			pr_credit = fme_pr_status.pr_credit;
> +		};
> +
> +		if (count >= 4) {
> +			fme_pr_data.rsvd = 0;
> +			fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
> +			writeq(fme_pr_data.csr, &fme_pr->data);
> +			count -= 4;
> +			pr_credit--;
> +			i++;
> +		} else {
> +			WARN_ON(1);
> +			return -EINVAL;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static int fme_mgr_write_complete(struct fpga_manager *mgr,
> +				  struct fpga_image_info *info)
> +{
> +	struct device *dev = &mgr->dev;
> +	struct fme_mgr_priv *priv = mgr->priv;
> +	struct feature_fme_pr *fme_pr = priv->ioaddr;
> +	struct feature_fme_pr_ctl fme_pr_ctl;
> +	struct feature_fme_pr_error fme_pr_err;
> +
> +	fme_pr_ctl.csr = readq(&fme_pr->control);
> +	fme_pr_ctl.pr_push_complete = 1;
> +	writeq(fme_pr_ctl.csr, &fme_pr->control);
> +
> +	dev_dbg(dev, "green bitstream push complete\n");
> +	dev_dbg(dev, "waiting for HW to release PR resource\n");
> +
> +	fme_pr_ctl.pr_start_req = 0;
> +
> +	if (readq_poll_timeout(&fme_pr->control, fme_pr_ctl.csr,
> +			       (fme_pr_ctl.pr_start_req == 0),
> +			       1, PR_WAIT_TIMEOUT)) {
> +		dev_err(dev, "maximum try.\n");
> +		return -ETIMEDOUT;
> +	}
> +
> +	dev_dbg(dev, "PR operation complete, checking status\n");
> +	fme_pr_err.csr = fme_mgr_pr_err_handle(fme_pr);
> +	if (fme_pr_err.csr) {
> +		fme_mgr_pr_update_status(mgr, fme_pr_err);
> +		return -EIO;
> +	}
> +
> +	dev_dbg(dev, "PR done successfully\n");
> +	return 0;
> +}
> +
> +static enum fpga_mgr_states fme_mgr_state(struct fpga_manager *mgr)
> +{
> +	return FPGA_MGR_STATE_UNKNOWN;
> +}
> +
> +static const struct fpga_manager_ops fme_mgr_ops = {
> +	.write_init = fme_mgr_write_init,
> +	.write = fme_mgr_write,
> +	.write_complete = fme_mgr_write_complete,
> +	.state = fme_mgr_state,
> +};
> +
> +static int fme_mgr_probe(struct platform_device *pdev)
> +{
> +	struct device *dev = &pdev->dev;
> +	struct fme_mgr_priv *priv;
> +	struct resource *res;
> +	int ret;
> +
> +	priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
> +	if (!priv)
> +		return -ENOMEM;
> +
> +	res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> +	priv->ioaddr = devm_ioremap(dev, res->start, resource_size(res));
> +	if (IS_ERR(priv->ioaddr))
> +		return PTR_ERR(priv->ioaddr);
> +
> +	ret = sysfs_create_files(&pdev->dev.kobj, fme_mgr_attrs);
> +	if (ret)
> +		return ret;
> +
> +	ret = fpga_mgr_register(dev, "Intel FPGA Manager", &fme_mgr_ops, priv);
> +	if (ret) {
> +		dev_err(dev, "unable to register FPGA manager\n");
> +		sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +
> +static int fme_mgr_remove(struct platform_device *pdev)
> +{
> +	fpga_mgr_unregister(&pdev->dev);
> +	sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> +
> +	return 0;
> +}
> +
> +static struct platform_driver fme_mgr_driver = {
> +	.driver	= {
> +		.name    = INTEL_FPGA_FME_MGR,
> +	},
> +	.probe   = fme_mgr_probe,
> +	.remove  = fme_mgr_remove,
> +};
> +
> +module_platform_driver(fme_mgr_driver);
> +
> +MODULE_DESCRIPTION("FPGA Manager for Intel FPGA Management Engine");
> +MODULE_AUTHOR("Intel Corporation");
> +MODULE_LICENSE("GPL v2");
> +MODULE_ALIAS("platform:intel-fpga-fme-mgr");
> -- 
> 1.8.3.1
> 

[1] http://yarchive.net/comp/linux/bitfields.html
[2] https://lwn.net/Articles/478657/

Thanks,
Moritz

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v2 15/22] fpga: intel: add fpga manager platform driver for FME

[Wu Hao]

On Mon, Sep 25, 2017 at 02:24:57PM -0700, Moritz Fischer wrote:
> Hi Hao,

Hi Moritz

Thanks for your review. :)

> 
> On Mon, Jun 26, 2017 at 09:52:11AM +0800, Wu Hao wrote:
> > This patch adds fpga manager driver for Intel FPGA Management
> > Engine(FME). It implements fpga_manager_ops for FPGA Partial
> > Reconfiguration function.
> > 
> > Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> > Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > ---
> >  .../ABI/testing/sysfs-platform-intel-fpga-fme-mgr  |   8 +
> >  drivers/fpga/Kconfig                               |   7 +
> >  drivers/fpga/Makefile                              |   1 +
> >  drivers/fpga/intel-feature-dev.h                   |  75 +++++
> >  drivers/fpga/intel-fpga-fme-mgr.c                  | 307 +++++++++++++++++++++
> >  5 files changed, 398 insertions(+)
> >  create mode 100644 Documentation/ABI/testing/sysfs-platform-intel-fpga-fme-mgr
> >  create mode 100644 drivers/fpga/intel-fpga-fme-mgr.c
> > 
> > diff --git a/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme-mgr b/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme-mgr
> > new file mode 100644
> > index 0000000..40771fb
> > --- /dev/null
> > +++ b/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme-mgr
> > @@ -0,0 +1,8 @@
> > +What:		/sys/bus/platform/devices/intel-fpga-fme-mgr.0/interface_id
> > +Date:		June 2017
> > +KernelVersion:  4.12
> > +Contact:	Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > +Description:	Read-only. It returns interface id of partial reconfiguration
> > +		hardware. Userspace could use this information to check if
> > +		current hardware is compatible with given image before FPGA
> > +		programming.
> > diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> > index b91458f..6f2f623 100644
> > --- a/drivers/fpga/Kconfig
> > +++ b/drivers/fpga/Kconfig
> > @@ -151,6 +151,13 @@ config INTEL_FPGA_FME
> >  	  all FPGA platform level management features. There shall be 1
> >  	  FME per Intel FPGA.
> >  
> > +config INTEL_FPGA_FME_MGR
> > +	tristate "Intel FPGA FME Manager Driver"
> > +	depends on INTEL_FPGA_FME
> > +	help
> > +	  Say Y to enable FPGA Manager driver for Intel FPGA Management
> > +	  Engine.
> > +
> >  endif
> >  
> >  endif # FPGA
> > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> > index acda5ca..d1d588b 100644
> > --- a/drivers/fpga/Makefile
> > +++ b/drivers/fpga/Makefile
> > @@ -31,6 +31,7 @@ obj-$(CONFIG_OF_FPGA_REGION)		+= of-fpga-region.o
> >  # Intel FPGA Support
> >  obj-$(CONFIG_INTEL_FPGA_PCI)		+= intel-fpga-pci.o
> >  obj-$(CONFIG_INTEL_FPGA_FME)		+= intel-fpga-fme.o
> > +obj-$(CONFIG_INTEL_FPGA_FME_MGR)	+= intel-fpga-fme-mgr.o
> >  
> >  intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
> >  intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
> > diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
> > index 3f97b75..f33923b 100644
> > --- a/drivers/fpga/intel-feature-dev.h
> > +++ b/drivers/fpga/intel-feature-dev.h
> > @@ -164,8 +164,83 @@ struct feature_fme_err {
> >  };
> >  
> >  /* FME Partial Reconfiguration Sub Feature Register Set */
> > +/* FME PR Control Register */
> > +struct feature_fme_pr_ctl {
> > +	union {
> > +		u64 csr;
> > +		struct {
> > +			u64 pr_reset:1;		/* Reset PR Engine */
> > +			u64 rsvdz1:3;
> > +			u64 pr_reset_ack:1;	/* Reset PR Engine Ack */
> > +			u64 rsvdz2:3;
> > +			u64 pr_regionid:2;	/* PR Region ID */
> > +			u64 rsvdz3:2;
> > +			u64 pr_start_req:1;	/* PR Start Request */
> > +			u64 pr_push_complete:1;	/* PR Data push complete */
> > +			u64 pr_kind:1;		/* Load Customer or Intel GBS */
> > +			u64 rsvdz4:17;
> > +			u64 config_data:32;
> > +		};
> > +	};
> > +};
> > +
> > +/* FME PR Status Register */
> > +struct feature_fme_pr_status {
> > +	union {
> > +		u64 csr;
> > +		struct {
> > +			u64 pr_credit:9;	/* Number of PR Credits */
> > +			u64 rsvdz1:7;
> > +			u64 pr_status:1;	/* PR Operation status */
> > +			u64 rsvdz2:3;
> > +			u64 pr_ctrlr_status:3;	/* Controller status */
> > +			u64 rsvdz3:1;
> > +			u64 pr_host_status:4;	/* PR Host status */
> > +			u64 rsvdz4:36;
> > +		};
> > +	};
> > +};
> > +
> > +/* FME PR Data Register */
> > +struct feature_fme_pr_data {
> > +	union {
> > +		u64 csr;
> > +		struct {
> > +			/* PR data from the raw-binary file */
> > +			u64 pr_data_raw:32;
> > +			u64 rsvd:32;
> > +		};
> > +	};
> > +};
> > +
> > +/* FME PR Error Register */
> > +struct feature_fme_pr_error {
> > +	union {
> > +		u64 csr;
> > +		struct {
> > +			u64 operation_err:1;	/* Previous PR error detected */
> > +			u64 crc_err:1;		/* CRC error detected */
> > +			u64 incompatiable_bs:1; /* Incompatiable Bitstream */
> > +			u64 protocol_err:1;	/* Data push protocol error */
> > +			u64 fifo_overflow:1;	/* Data fifo overflow */
> > +			u64 rsvdz1:1;
> > +			u64 secure_load_err:1;	/* Secure PR load error */
> > +			u64 rsvdz2:57;
> > +		};
> > +	};
> > +};
> 
> From what I've seen [1], [2] people weren't too happy with using bitfields,
> any reason we can't use explicit masking and shifts? I don't know if
> this is still an issue.

I think both should work fine in our case, and compiler should give same
result on both. We choose bitfields as it seems to be more readable, and
could save some code. Do you think if we can keep it? :)

> 
> > +
> >  struct feature_fme_pr {
> >  	struct feature_header header;
> > +	struct feature_fme_pr_ctl control;
> > +	struct feature_fme_pr_status status;
> > +	struct feature_fme_pr_data data;
> > +	struct feature_fme_pr_error error;
> > +
> > +	u64 rsvd[16];
> > +
> > +	u64 intfc_id_l;		/* PR interface Id Low */
> > +	u64 intfc_id_h;		/* PR interface Id High */
> >  };
> >  
> >  /* PORT Header Register Set */
> > diff --git a/drivers/fpga/intel-fpga-fme-mgr.c b/drivers/fpga/intel-fpga-fme-mgr.c
> > new file mode 100644
> > index 0000000..5f93dfb
> > --- /dev/null
> > +++ b/drivers/fpga/intel-fpga-fme-mgr.c
> > @@ -0,0 +1,307 @@
> > +/*
> > + * FPGA Manager Driver for Intel FPGA Management Engine (FME)
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> > + *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Joseph Grecco <joe.grecco-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Ananda Ravuri <ananda.ravuri-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *   Henry Mitchel <henry.mitchel-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > + *
> > + * This work is licensed under the terms of the GNU GPL version 2. See
> > + * the COPYING file in the top-level directory.
> > + */
> > +
> > +#include <linux/module.h>
> > +#include <linux/iopoll.h>
> > +#include <linux/fpga/fpga-mgr.h>
> > +
> > +#include "intel-feature-dev.h"
> > +#include "intel-fme.h"
> > +
> > +#define PR_WAIT_TIMEOUT   8000000
> > +#define PR_HOST_STATUS_IDLE	0
> > +
> > +struct fme_mgr_priv {
> > +	void __iomem *ioaddr;
> > +};
> > +
> > +static ssize_t interface_id_show(struct device *dev,
> > +				 struct device_attribute *attr, char *buf)
> > +{
> > +	struct fpga_manager *mgr = dev_get_drvdata(dev);
> > +	struct fme_mgr_priv *priv = mgr->priv;
> > +	struct feature_fme_pr *fme_pr;
> > +	u64 intfc_id_l, intfc_id_h;
> > +
> > +	fme_pr = priv->ioaddr;
> > +
> > +	intfc_id_l = readq(&fme_pr->intfc_id_l);
> > +	intfc_id_h = readq(&fme_pr->intfc_id_h);
> > +
> > +	return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> > +			(unsigned long long)intfc_id_h,
> > +			(unsigned long long)intfc_id_l);
> > +}
> > +static DEVICE_ATTR_RO(interface_id);
> > +
> > +static const struct attribute *fme_mgr_attrs[] = {
> > +	&dev_attr_interface_id.attr,
> > +	NULL,
> > +};
> > +
> > +static void fme_mgr_pr_update_status(struct fpga_manager *mgr,
> > +				     struct feature_fme_pr_error err)
> > +{
> > +	mgr->status = 0;
> > +
> > +	if (err.operation_err)
> > +		mgr->status |= FPGA_MGR_STATUS_OPERATION_ERR;
> 
> Can we just have a (err & FEATURE_FME_PR_ERROR) here instead ?
> > +	if (err.crc_err)
> > +		mgr->status |= FPGA_MGR_STATUS_CRC_ERR;
> 
> Can we just have a (err & FEATURE_FME_CRC_ERROR) here instead ?

As above, do you think if we can keep the origin code here?
Use "if (err.crc_err)" not "if (err & FEATURE_FME_CRC_ERROR)",
it makes the code shorter in most cases. :)

Thanks
Hao

> > +	if (err.incompatiable_bs)
> > +		mgr->status |= FPGA_MGR_STATUS_INCOMPATIBLE_BS_ERR;
> > +	if (err.protocol_err)
> > +		mgr->status |= FPGA_MGR_STATUS_IP_PROTOCOL_ERR;
> > +	if (err.fifo_overflow)
> > +		mgr->status |= FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR;
> > +	if (err.secure_load_err)
> > +		mgr->status |= FPGA_MGR_STATUS_SECURE_LOAD_ERR;
> > +}
> > +
> > +static u64 fme_mgr_pr_err_handle(struct feature_fme_pr *fme_pr)
> > +{
> > +	struct feature_fme_pr_status fme_pr_status;
> > +	struct feature_fme_pr_error fme_pr_error;
> > +
> > +	fme_pr_status.csr = readq(&fme_pr->status);
> > +	if (!fme_pr_status.pr_status)
> > +		return 0;
> > +
> > +	fme_pr_error.csr = readq(&fme_pr->error);
> > +	writeq(fme_pr_error.csr, &fme_pr->error);
> > +
> > +	return fme_pr_error.csr;
> > +}
> > +
> > +static int fme_mgr_write_init(struct fpga_manager *mgr,
> > +			      struct fpga_image_info *info,
> > +			      const char *buf, size_t count)
> > +{
> > +	struct device *dev = &mgr->dev;
> > +	struct fme_mgr_priv *priv = mgr->priv;
> > +	struct feature_fme_pr *fme_pr = priv->ioaddr;
> > +	struct feature_fme_pr_ctl fme_pr_ctl;
> > +	struct feature_fme_pr_status fme_pr_status;
> > +	struct feature_fme_pr_error fme_pr_err;
> > +
> > +	if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
> > +		dev_err(dev, "only support partial reconfiguration.\n");
> > +		return -EINVAL;
> > +	}
> > +
> > +	dev_dbg(dev, "resetting PR before initiated PR\n");
> > +
> > +	fme_pr_ctl.csr = readq(&fme_pr->control);
> > +	fme_pr_ctl.pr_reset = 1;
> > +	writeq(fme_pr_ctl.csr, &fme_pr->control);
> > +
> > +	if (readq_poll_timeout(&fme_pr->control, fme_pr_ctl.csr,
> > +			       (fme_pr_ctl.pr_reset_ack == 1),
> > +			       1, PR_WAIT_TIMEOUT)) {
> > +		dev_err(dev, "maximum PR timeout\n");
> > +		return -ETIMEDOUT;
> > +	}
> > +
> > +	fme_pr_ctl.csr = readq(&fme_pr->control);
> > +	fme_pr_ctl.pr_reset = 0;
> > +	writeq(fme_pr_ctl.csr, &fme_pr->control);
> > +
> > +	dev_dbg(dev,
> > +		"waiting for PR resource in HW to be initialized and ready\n");
> > +
> > +	fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
> > +
> > +	if (readq_poll_timeout(&fme_pr->status, fme_pr_status.csr,
> > +			(fme_pr_status.pr_host_status == PR_HOST_STATUS_IDLE),
> > +			1, PR_WAIT_TIMEOUT)) {
> > +		dev_err(dev, "maximum PR timeout\n");
> > +		return -ETIMEDOUT;
> > +	}
> > +
> > +	dev_dbg(dev, "check and clear previous PR error\n");
> > +	fme_pr_err.csr = fme_mgr_pr_err_handle(fme_pr);
> > +	if (fme_pr_err.csr)
> > +		dev_dbg(dev, "previous PR error detected %llx\n",
> > +			     (unsigned long long)fme_pr_err.csr);
> > +
> > +	/* Clear all PR errors */
> > +	fme_pr_err.csr = 0;
> > +	fme_mgr_pr_update_status(mgr, fmCRCe_pr_err);
> > +
> > +	dev_dbg(dev, "set PR port ID\n");
> > +
> > +	fme_pr_ctl.csr = readq(&fme_pr->control);
> > +	fme_pr_ctl.pr_regionid = info->region_id;
> > +	writeq(fme_pr_ctl.csr, &fme_pr->control);
> > +
> > +	return 0;
> > +}
> > +
> > +static int fme_mgr_write(struct fpga_manager *mgr,
> > +			 const char *buf, size_t count)
> > +{
> > +	struct device *dev = &mgr->dev;
> > +	struct fme_mgr_priv *priv = mgr->priv;
> > +	struct feature_fme_pr *fme_pr = priv->ioaddr;
> > +	struct feature_fme_pr_ctl fme_pr_ctl;
> > +	struct feature_fme_pr_status fme_pr_status;
> > +	struct feature_fme_pr_data fme_pr_data;
> > +	int delay, pr_credit, i = 0;
> > +
> > +	dev_dbg(dev, "start request\n");
> > +
> > +	fme_pr_ctl.csr = readq(&fme_pr->control);
> > +	fme_pr_ctl.pr_start_req = 1;
> > +	writeq(fme_pr_ctl.csr, &fme_pr->control);
> > +
> > +	dev_dbg(dev, "pushing data from bitstream to HW\n");
> > +
> > +	fme_pr_status.csr = readq(&fme_pr->status);
> > +	pr_credit = fme_pr_status.pr_credit;
> > +
> > +	while (count > 0) {
> > +		delay = 0;
> > +		while (pr_credit <= 1) {
> > +			if (delay++ > PR_WAIT_TIMEOUT) {
> > +				dev_err(dev, "maximum try\n");
> > +				return -ETIMEDOUT;
> > +			}
> > +			udelay(1);
> > +
> > +			fme_pr_status.csr = readq(&fme_pr->status);
> > +			pr_credit = fme_pr_status.pr_credit;
> > +		};
> > +
> > +		if (count >= 4) {
> > +			fme_pr_data.rsvd = 0;
> > +			fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
> > +			writeq(fme_pr_data.csr, &fme_pr->data);
> > +			count -= 4;
> > +			pr_credit--;
> > +			i++;
> > +		} else {
> > +			WARN_ON(1);
> > +			return -EINVAL;
> > +		}
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static int fme_mgr_write_complete(struct fpga_manager *mgr,
> > +				  struct fpga_image_info *info)
> > +{
> > +	struct device *dev = &mgr->dev;
> > +	struct fme_mgr_priv *priv = mgr->priv;
> > +	struct feature_fme_pr *fme_pr = priv->ioaddr;
> > +	struct feature_fme_pr_ctl fme_pr_ctl;
> > +	struct feature_fme_pr_error fme_pr_err;
> > +
> > +	fme_pr_ctl.csr = readq(&fme_pr->control);
> > +	fme_pr_ctl.pr_push_complete = 1;
> > +	writeq(fme_pr_ctl.csr, &fme_pr->control);
> > +
> > +	dev_dbg(dev, "green bitstream push complete\n");
> > +	dev_dbg(dev, "waiting for HW to release PR resource\n");
> > +
> > +	fme_pr_ctl.pr_start_req = 0;
> > +
> > +	if (readq_poll_timeout(&fme_pr->control, fme_pr_ctl.csr,
> > +			       (fme_pr_ctl.pr_start_req == 0),
> > +			       1, PR_WAIT_TIMEOUT)) {
> > +		dev_err(dev, "maximum try.\n");
> > +		return -ETIMEDOUT;
> > +	}
> > +
> > +	dev_dbg(dev, "PR operation complete, checking status\n");
> > +	fme_pr_err.csr = fme_mgr_pr_err_handle(fme_pr);
> > +	if (fme_pr_err.csr) {
> > +		fme_mgr_pr_update_status(mgr, fme_pr_err);
> > +		return -EIO;
> > +	}
> > +
> > +	dev_dbg(dev, "PR done successfully\n");
> > +	return 0;
> > +}
> > +
> > +static enum fpga_mgr_states fme_mgr_state(struct fpga_manager *mgr)
> > +{
> > +	return FPGA_MGR_STATE_UNKNOWN;
> > +}
> > +
> > +static const struct fpga_manager_ops fme_mgr_ops = {
> > +	.write_init = fme_mgr_write_init,
> > +	.write = fme_mgr_write,
> > +	.write_complete = fme_mgr_write_complete,
> > +	.state = fme_mgr_state,
> > +};
> > +
> > +static int fme_mgr_probe(struct platform_device *pdev)
> > +{
> > +	struct device *dev = &pdev->dev;
> > +	struct fme_mgr_priv *priv;
> > +	struct resource *res;
> > +	int ret;
> > +
> > +	priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
> > +	if (!priv)
> > +		return -ENOMEM;
> > +
> > +	res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> > +	priv->ioaddr = devm_ioremap(dev, res->start, resource_size(res));
> > +	if (IS_ERR(priv->ioaddr))
> > +		return PTR_ERR(priv->ioaddr);
> > +
> > +	ret = sysfs_create_files(&pdev->dev.kobj, fme_mgr_attrs);
> > +	if (ret)
> > +		return ret;
> > +
> > +	ret = fpga_mgr_register(dev, "Intel FPGA Manager", &fme_mgr_ops, priv);
> > +	if (ret) {
> > +		dev_err(dev, "unable to register FPGA manager\n");
> > +		sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> > +		return ret;
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static int fme_mgr_remove(struct platform_device *pdev)
> > +{
> > +	fpga_mgr_unregister(&pdev->dev);
> > +	sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> > +
> > +	return 0;
> > +}
> > +
> > +static struct platform_driver fme_mgr_driver = {
> > +	.driver	= {
> > +		.name    = INTEL_FPGA_FME_MGR,
> > +	},
> > +	.probe   = fme_mgr_probe,
> > +	.remove  = fme_mgr_remove,
> > +};
> > +
> > +module_platform_driver(fme_mgr_driver);
> > +
> > +MODULE_DESCRIPTION("FPGA Manager for Intel FPGA Management Engine");
> > +MODULE_AUTHOR("Intel Corporation");
> > +MODULE_LICENSE("GPL v2");
> > +MODULE_ALIAS("platform:intel-fpga-fme-mgr");
> > -- 
> > 1.8.3.1
> > 
> 
> [1] http://yarchive.net/comp/linux/bitfields.html
> [2] https://lwn.net/Articles/478657/
> 
> Thanks,
> Moritz

Re: [PATCH v2 15/22] fpga: intel: add fpga manager platform driver for FME

[Alan Tull]

On Tue, Sep 26, 2017 at 8:18 PM, Wu Hao <hao.wu@intel.com> wrote:

Hi Hao,

> On Mon, Sep 25, 2017 at 02:24:57PM -0700, Moritz Fischer wrote:
>> Hi Hao,
>
> Hi Moritz
>
> Thanks for your review. :)
>
>>
>> On Mon, Jun 26, 2017 at 09:52:11AM +0800, Wu Hao wrote:
>> > This patch adds fpga manager driver for Intel FPGA Management
>> > Engine(FME). It implements fpga_manager_ops for FPGA Partial
>> > Reconfiguration function.
>> >
>> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
>> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
>> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
>> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
>> > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
>> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> > Signed-off-by: Wu Hao <hao.wu@intel.com>
>> > ---
>> >  .../ABI/testing/sysfs-platform-intel-fpga-fme-mgr  |   8 +
>> >  drivers/fpga/Kconfig                               |   7 +
>> >  drivers/fpga/Makefile                              |   1 +
>> >  drivers/fpga/intel-feature-dev.h                   |  75 +++++
>> >  drivers/fpga/intel-fpga-fme-mgr.c                  | 307 +++++++++++++++++++++
>> >  5 files changed, 398 insertions(+)
>> >  create mode 100644 Documentation/ABI/testing/sysfs-platform-intel-fpga-fme-mgr
>> >  create mode 100644 drivers/fpga/intel-fpga-fme-mgr.c
>> >
>> > diff --git a/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme-mgr b/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme-mgr
>> > new file mode 100644
>> > index 0000000..40771fb
>> > --- /dev/null
>> > +++ b/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme-mgr
>> > @@ -0,0 +1,8 @@
>> > +What:              /sys/bus/platform/devices/intel-fpga-fme-mgr.0/interface_id
>> > +Date:              June 2017
>> > +KernelVersion:  4.12
>> > +Contact:   Wu Hao <hao.wu@intel.com>
>> > +Description:       Read-only. It returns interface id of partial reconfiguration
>> > +           hardware. Userspace could use this information to check if
>> > +           current hardware is compatible with given image before FPGA
>> > +           programming.
>> > diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
>> > index b91458f..6f2f623 100644
>> > --- a/drivers/fpga/Kconfig
>> > +++ b/drivers/fpga/Kconfig
>> > @@ -151,6 +151,13 @@ config INTEL_FPGA_FME
>> >       all FPGA platform level management features. There shall be 1
>> >       FME per Intel FPGA.
>> >
>> > +config INTEL_FPGA_FME_MGR
>> > +   tristate "Intel FPGA FME Manager Driver"
>> > +   depends on INTEL_FPGA_FME
>> > +   help
>> > +     Say Y to enable FPGA Manager driver for Intel FPGA Management
>> > +     Engine.
>> > +
>> >  endif
>> >
>> >  endif # FPGA
>> > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
>> > index acda5ca..d1d588b 100644
>> > --- a/drivers/fpga/Makefile
>> > +++ b/drivers/fpga/Makefile
>> > @@ -31,6 +31,7 @@ obj-$(CONFIG_OF_FPGA_REGION)              += of-fpga-region.o
>> >  # Intel FPGA Support
>> >  obj-$(CONFIG_INTEL_FPGA_PCI)               += intel-fpga-pci.o
>> >  obj-$(CONFIG_INTEL_FPGA_FME)               += intel-fpga-fme.o
>> > +obj-$(CONFIG_INTEL_FPGA_FME_MGR)   += intel-fpga-fme-mgr.o
>> >
>> >  intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
>> >  intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
>> > diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
>> > index 3f97b75..f33923b 100644
>> > --- a/drivers/fpga/intel-feature-dev.h
>> > +++ b/drivers/fpga/intel-feature-dev.h
>> > @@ -164,8 +164,83 @@ struct feature_fme_err {
>> >  };
>> >
>> >  /* FME Partial Reconfiguration Sub Feature Register Set */
>> > +/* FME PR Control Register */
>> > +struct feature_fme_pr_ctl {
>> > +   union {
>> > +           u64 csr;
>> > +           struct {
>> > +                   u64 pr_reset:1;         /* Reset PR Engine */
>> > +                   u64 rsvdz1:3;
>> > +                   u64 pr_reset_ack:1;     /* Reset PR Engine Ack */
>> > +                   u64 rsvdz2:3;
>> > +                   u64 pr_regionid:2;      /* PR Region ID */
>> > +                   u64 rsvdz3:2;
>> > +                   u64 pr_start_req:1;     /* PR Start Request */
>> > +                   u64 pr_push_complete:1; /* PR Data push complete */
>> > +                   u64 pr_kind:1;          /* Load Customer or Intel GBS */
>> > +                   u64 rsvdz4:17;
>> > +                   u64 config_data:32;
>> > +           };
>> > +   };
>> > +};
>> > +
>> > +/* FME PR Status Register */
>> > +struct feature_fme_pr_status {
>> > +   union {
>> > +           u64 csr;
>> > +           struct {
>> > +                   u64 pr_credit:9;        /* Number of PR Credits */
>> > +                   u64 rsvdz1:7;
>> > +                   u64 pr_status:1;        /* PR Operation status */
>> > +                   u64 rsvdz2:3;
>> > +                   u64 pr_ctrlr_status:3;  /* Controller status */
>> > +                   u64 rsvdz3:1;
>> > +                   u64 pr_host_status:4;   /* PR Host status */
>> > +                   u64 rsvdz4:36;
>> > +           };
>> > +   };
>> > +};
>> > +
>> > +/* FME PR Data Register */
>> > +struct feature_fme_pr_data {
>> > +   union {
>> > +           u64 csr;
>> > +           struct {
>> > +                   /* PR data from the raw-binary file */
>> > +                   u64 pr_data_raw:32;
>> > +                   u64 rsvd:32;
>> > +           };
>> > +   };
>> > +};
>> > +
>> > +/* FME PR Error Register */
>> > +struct feature_fme_pr_error {
>> > +   union {
>> > +           u64 csr;
>> > +           struct {
>> > +                   u64 operation_err:1;    /* Previous PR error detected */
>> > +                   u64 crc_err:1;          /* CRC error detected */
>> > +                   u64 incompatiable_bs:1; /* Incompatiable Bitstream */
>> > +                   u64 protocol_err:1;     /* Data push protocol error */
>> > +                   u64 fifo_overflow:1;    /* Data fifo overflow */
>> > +                   u64 rsvdz1:1;
>> > +                   u64 secure_load_err:1;  /* Secure PR load error */
>> > +                   u64 rsvdz2:57;
>> > +           };
>> > +   };
>> > +};
>>
>> From what I've seen [1], [2] people weren't too happy with using bitfields,
>> any reason we can't use explicit masking and shifts? I don't know if
>> this is still an issue.
>
> I think both should work fine in our case, and compiler should give same
> result on both. We choose bitfields as it seems to be more readable, and
> could save some code. Do you think if we can keep it? :)

I have the same concerns as Moritz here.  I have suggested using
regmap before back in July.  Using bitmasks and shifts would be be
preferable.

Alan

>
>>
>> > +
>> >  struct feature_fme_pr {
>> >     struct feature_header header;
>> > +   struct feature_fme_pr_ctl control;
>> > +   struct feature_fme_pr_status status;
>> > +   struct feature_fme_pr_data data;
>> > +   struct feature_fme_pr_error error;
>> > +
>> > +   u64 rsvd[16];
>> > +
>> > +   u64 intfc_id_l;         /* PR interface Id Low */
>> > +   u64 intfc_id_h;         /* PR interface Id High */
>> >  };
>> >
>> >  /* PORT Header Register Set */
>> > diff --git a/drivers/fpga/intel-fpga-fme-mgr.c b/drivers/fpga/intel-fpga-fme-mgr.c
>> > new file mode 100644
>> > index 0000000..5f93dfb
>> > --- /dev/null
>> > +++ b/drivers/fpga/intel-fpga-fme-mgr.c
>> > @@ -0,0 +1,307 @@
>> > +/*
>> > + * FPGA Manager Driver for Intel FPGA Management Engine (FME)
>> > + *
>> > + * Copyright (C) 2017 Intel Corporation, Inc.
>> > + *
>> > + * Authors:
>> > + *   Kang Luwei <luwei.kang@intel.com>
>> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> > + *   Wu Hao <hao.wu@intel.com>
>> > + *   Joseph Grecco <joe.grecco@intel.com>
>> > + *   Enno Luebbers <enno.luebbers@intel.com>
>> > + *   Tim Whisonant <tim.whisonant@intel.com>
>> > + *   Ananda Ravuri <ananda.ravuri@intel.com>
>> > + *   Christopher Rauer <christopher.rauer@intel.com>
>> > + *   Henry Mitchel <henry.mitchel@intel.com>
>> > + *
>> > + * This work is licensed under the terms of the GNU GPL version 2. See
>> > + * the COPYING file in the top-level directory.
>> > + */
>> > +
>> > +#include <linux/module.h>
>> > +#include <linux/iopoll.h>
>> > +#include <linux/fpga/fpga-mgr.h>
>> > +
>> > +#include "intel-feature-dev.h"
>> > +#include "intel-fme.h"
>> > +
>> > +#define PR_WAIT_TIMEOUT   8000000
>> > +#define PR_HOST_STATUS_IDLE        0
>> > +
>> > +struct fme_mgr_priv {
>> > +   void __iomem *ioaddr;
>> > +};
>> > +
>> > +static ssize_t interface_id_show(struct device *dev,
>> > +                            struct device_attribute *attr, char *buf)
>> > +{
>> > +   struct fpga_manager *mgr = dev_get_drvdata(dev);
>> > +   struct fme_mgr_priv *priv = mgr->priv;
>> > +   struct feature_fme_pr *fme_pr;
>> > +   u64 intfc_id_l, intfc_id_h;
>> > +
>> > +   fme_pr = priv->ioaddr;
>> > +
>> > +   intfc_id_l = readq(&fme_pr->intfc_id_l);
>> > +   intfc_id_h = readq(&fme_pr->intfc_id_h);
>> > +
>> > +   return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
>> > +                   (unsigned long long)intfc_id_h,
>> > +                   (unsigned long long)intfc_id_l);
>> > +}
>> > +static DEVICE_ATTR_RO(interface_id);
>> > +
>> > +static const struct attribute *fme_mgr_attrs[] = {
>> > +   &dev_attr_interface_id.attr,
>> > +   NULL,
>> > +};
>> > +
>> > +static void fme_mgr_pr_update_status(struct fpga_manager *mgr,
>> > +                                struct feature_fme_pr_error err)
>> > +{
>> > +   mgr->status = 0;
>> > +
>> > +   if (err.operation_err)
>> > +           mgr->status |= FPGA_MGR_STATUS_OPERATION_ERR;
>>
>> Can we just have a (err & FEATURE_FME_PR_ERROR) here instead ?
>> > +   if (err.crc_err)
>> > +           mgr->status |= FPGA_MGR_STATUS_CRC_ERR;
>>
>> Can we just have a (err & FEATURE_FME_CRC_ERROR) here instead ?
>
> As above, do you think if we can keep the origin code here?
> Use "if (err.crc_err)" not "if (err & FEATURE_FME_CRC_ERROR)",
> it makes the code shorter in most cases. :)
>
> Thanks
> Hao
>
>> > +   if (err.incompatiable_bs)
>> > +           mgr->status |= FPGA_MGR_STATUS_INCOMPATIBLE_BS_ERR;
>> > +   if (err.protocol_err)
>> > +           mgr->status |= FPGA_MGR_STATUS_IP_PROTOCOL_ERR;
>> > +   if (err.fifo_overflow)
>> > +           mgr->status |= FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR;
>> > +   if (err.secure_load_err)
>> > +           mgr->status |= FPGA_MGR_STATUS_SECURE_LOAD_ERR;
>> > +}
>> > +
>> > +static u64 fme_mgr_pr_err_handle(struct feature_fme_pr *fme_pr)
>> > +{
>> > +   struct feature_fme_pr_status fme_pr_status;
>> > +   struct feature_fme_pr_error fme_pr_error;
>> > +
>> > +   fme_pr_status.csr = readq(&fme_pr->status);
>> > +   if (!fme_pr_status.pr_status)
>> > +           return 0;
>> > +
>> > +   fme_pr_error.csr = readq(&fme_pr->error);
>> > +   writeq(fme_pr_error.csr, &fme_pr->error);
>> > +
>> > +   return fme_pr_error.csr;
>> > +}
>> > +
>> > +static int fme_mgr_write_init(struct fpga_manager *mgr,
>> > +                         struct fpga_image_info *info,
>> > +                         const char *buf, size_t count)
>> > +{
>> > +   struct device *dev = &mgr->dev;
>> > +   struct fme_mgr_priv *priv = mgr->priv;
>> > +   struct feature_fme_pr *fme_pr = priv->ioaddr;
>> > +   struct feature_fme_pr_ctl fme_pr_ctl;
>> > +   struct feature_fme_pr_status fme_pr_status;
>> > +   struct feature_fme_pr_error fme_pr_err;
>> > +
>> > +   if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
>> > +           dev_err(dev, "only support partial reconfiguration.\n");
>> > +           return -EINVAL;
>> > +   }
>> > +
>> > +   dev_dbg(dev, "resetting PR before initiated PR\n");
>> > +
>> > +   fme_pr_ctl.csr = readq(&fme_pr->control);
>> > +   fme_pr_ctl.pr_reset = 1;
>> > +   writeq(fme_pr_ctl.csr, &fme_pr->control);
>> > +
>> > +   if (readq_poll_timeout(&fme_pr->control, fme_pr_ctl.csr,
>> > +                          (fme_pr_ctl.pr_reset_ack == 1),
>> > +                          1, PR_WAIT_TIMEOUT)) {
>> > +           dev_err(dev, "maximum PR timeout\n");
>> > +           return -ETIMEDOUT;
>> > +   }
>> > +
>> > +   fme_pr_ctl.csr = readq(&fme_pr->control);
>> > +   fme_pr_ctl.pr_reset = 0;
>> > +   writeq(fme_pr_ctl.csr, &fme_pr->control);
>> > +
>> > +   dev_dbg(dev,
>> > +           "waiting for PR resource in HW to be initialized and ready\n");
>> > +
>> > +   fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
>> > +
>> > +   if (readq_poll_timeout(&fme_pr->status, fme_pr_status.csr,
>> > +                   (fme_pr_status.pr_host_status == PR_HOST_STATUS_IDLE),
>> > +                   1, PR_WAIT_TIMEOUT)) {
>> > +           dev_err(dev, "maximum PR timeout\n");
>> > +           return -ETIMEDOUT;
>> > +   }
>> > +
>> > +   dev_dbg(dev, "check and clear previous PR error\n");
>> > +   fme_pr_err.csr = fme_mgr_pr_err_handle(fme_pr);
>> > +   if (fme_pr_err.csr)
>> > +           dev_dbg(dev, "previous PR error detected %llx\n",
>> > +                        (unsigned long long)fme_pr_err.csr);
>> > +
>> > +   /* Clear all PR errors */
>> > +   fme_pr_err.csr = 0;
>> > +   fme_mgr_pr_update_status(mgr, fmCRCe_pr_err);
>> > +
>> > +   dev_dbg(dev, "set PR port ID\n");
>> > +
>> > +   fme_pr_ctl.csr = readq(&fme_pr->control);
>> > +   fme_pr_ctl.pr_regionid = info->region_id;
>> > +   writeq(fme_pr_ctl.csr, &fme_pr->control);
>> > +
>> > +   return 0;
>> > +}
>> > +
>> > +static int fme_mgr_write(struct fpga_manager *mgr,
>> > +                    const char *buf, size_t count)
>> > +{
>> > +   struct device *dev = &mgr->dev;
>> > +   struct fme_mgr_priv *priv = mgr->priv;
>> > +   struct feature_fme_pr *fme_pr = priv->ioaddr;
>> > +   struct feature_fme_pr_ctl fme_pr_ctl;
>> > +   struct feature_fme_pr_status fme_pr_status;
>> > +   struct feature_fme_pr_data fme_pr_data;
>> > +   int delay, pr_credit, i = 0;
>> > +
>> > +   dev_dbg(dev, "start request\n");
>> > +
>> > +   fme_pr_ctl.csr = readq(&fme_pr->control);
>> > +   fme_pr_ctl.pr_start_req = 1;
>> > +   writeq(fme_pr_ctl.csr, &fme_pr->control);
>> > +
>> > +   dev_dbg(dev, "pushing data from bitstream to HW\n");
>> > +
>> > +   fme_pr_status.csr = readq(&fme_pr->status);
>> > +   pr_credit = fme_pr_status.pr_credit;
>> > +
>> > +   while (count > 0) {
>> > +           delay = 0;
>> > +           while (pr_credit <= 1) {
>> > +                   if (delay++ > PR_WAIT_TIMEOUT) {
>> > +                           dev_err(dev, "maximum try\n");
>> > +                           return -ETIMEDOUT;
>> > +                   }
>> > +                   udelay(1);
>> > +
>> > +                   fme_pr_status.csr = readq(&fme_pr->status);
>> > +                   pr_credit = fme_pr_status.pr_credit;
>> > +           };
>> > +
>> > +           if (count >= 4) {
>> > +                   fme_pr_data.rsvd = 0;
>> > +                   fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
>> > +                   writeq(fme_pr_data.csr, &fme_pr->data);
>> > +                   count -= 4;
>> > +                   pr_credit--;
>> > +                   i++;
>> > +           } else {
>> > +                   WARN_ON(1);
>> > +                   return -EINVAL;
>> > +           }
>> > +   }
>> > +
>> > +   return 0;
>> > +}
>> > +
>> > +static int fme_mgr_write_complete(struct fpga_manager *mgr,
>> > +                             struct fpga_image_info *info)
>> > +{
>> > +   struct device *dev = &mgr->dev;
>> > +   struct fme_mgr_priv *priv = mgr->priv;
>> > +   struct feature_fme_pr *fme_pr = priv->ioaddr;
>> > +   struct feature_fme_pr_ctl fme_pr_ctl;
>> > +   struct feature_fme_pr_error fme_pr_err;
>> > +
>> > +   fme_pr_ctl.csr = readq(&fme_pr->control);
>> > +   fme_pr_ctl.pr_push_complete = 1;
>> > +   writeq(fme_pr_ctl.csr, &fme_pr->control);
>> > +
>> > +   dev_dbg(dev, "green bitstream push complete\n");
>> > +   dev_dbg(dev, "waiting for HW to release PR resource\n");
>> > +
>> > +   fme_pr_ctl.pr_start_req = 0;
>> > +
>> > +   if (readq_poll_timeout(&fme_pr->control, fme_pr_ctl.csr,
>> > +                          (fme_pr_ctl.pr_start_req == 0),
>> > +                          1, PR_WAIT_TIMEOUT)) {
>> > +           dev_err(dev, "maximum try.\n");
>> > +           return -ETIMEDOUT;
>> > +   }
>> > +
>> > +   dev_dbg(dev, "PR operation complete, checking status\n");
>> > +   fme_pr_err.csr = fme_mgr_pr_err_handle(fme_pr);
>> > +   if (fme_pr_err.csr) {
>> > +           fme_mgr_pr_update_status(mgr, fme_pr_err);
>> > +           return -EIO;
>> > +   }
>> > +
>> > +   dev_dbg(dev, "PR done successfully\n");
>> > +   return 0;
>> > +}
>> > +
>> > +static enum fpga_mgr_states fme_mgr_state(struct fpga_manager *mgr)
>> > +{
>> > +   return FPGA_MGR_STATE_UNKNOWN;
>> > +}
>> > +
>> > +static const struct fpga_manager_ops fme_mgr_ops = {
>> > +   .write_init = fme_mgr_write_init,
>> > +   .write = fme_mgr_write,
>> > +   .write_complete = fme_mgr_write_complete,
>> > +   .state = fme_mgr_state,
>> > +};
>> > +
>> > +static int fme_mgr_probe(struct platform_device *pdev)
>> > +{
>> > +   struct device *dev = &pdev->dev;
>> > +   struct fme_mgr_priv *priv;
>> > +   struct resource *res;
>> > +   int ret;
>> > +
>> > +   priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
>> > +   if (!priv)
>> > +           return -ENOMEM;
>> > +
>> > +   res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
>> > +   priv->ioaddr = devm_ioremap(dev, res->start, resource_size(res));
>> > +   if (IS_ERR(priv->ioaddr))
>> > +           return PTR_ERR(priv->ioaddr);
>> > +
>> > +   ret = sysfs_create_files(&pdev->dev.kobj, fme_mgr_attrs);
>> > +   if (ret)
>> > +           return ret;
>> > +
>> > +   ret = fpga_mgr_register(dev, "Intel FPGA Manager", &fme_mgr_ops, priv);
>> > +   if (ret) {
>> > +           dev_err(dev, "unable to register FPGA manager\n");
>> > +           sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
>> > +           return ret;
>> > +   }
>> > +
>> > +   return 0;
>> > +}
>> > +
>> > +static int fme_mgr_remove(struct platform_device *pdev)
>> > +{
>> > +   fpga_mgr_unregister(&pdev->dev);
>> > +   sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
>> > +
>> > +   return 0;
>> > +}
>> > +
>> > +static struct platform_driver fme_mgr_driver = {
>> > +   .driver = {
>> > +           .name    = INTEL_FPGA_FME_MGR,
>> > +   },
>> > +   .probe   = fme_mgr_probe,
>> > +   .remove  = fme_mgr_remove,
>> > +};
>> > +
>> > +module_platform_driver(fme_mgr_driver);
>> > +
>> > +MODULE_DESCRIPTION("FPGA Manager for Intel FPGA Management Engine");
>> > +MODULE_AUTHOR("Intel Corporation");
>> > +MODULE_LICENSE("GPL v2");
>> > +MODULE_ALIAS("platform:intel-fpga-fme-mgr");
>> > --
>> > 1.8.3.1
>> >
>>
>> [1] http://yarchive.net/comp/linux/bitfields.html
>> [2] https://lwn.net/Articles/478657/
>>
>> Thanks,
>> Moritz
>
>

Re: [PATCH v2 07/22] fpga: intel: pcie: parse feature list and create platform device for features.

[Alan Tull]

On Fri, Sep 22, 2017 at 2:28 AM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> On Wed, Sep 20, 2017 at 04:24:10PM -0500, Alan Tull wrote:

HI Hao,

>>  a (wh., *()On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>>
>> Hi Hao,
>>
>> I'm done with some board bringup so I have time to look at your patchset again.
>
> Hi Alan
>
> Thanks for your time on the review. : )
>
>>
>> Something I can't help but notice is that this patchset will be almost
>> completely reusable for embedded FPGAs if the enumeration is separated
>> from the pcie code.  After the devices are created, they are just mmio
>> devices.  That makes this whole scheme available for embedded FPGAs.
>>
>> The division of things would be that the pcie code would read the
>> headers and do ioremapping.  Then pass the headers to the enumeration
>> code to create the devices.  With that division, another platform that
>> is embedded could have its own code that read the headers and did the
>> iomapping and then used the separate enumeration code.
>>
>> Going through intel-pcie.c, the truly pcie specific parts are already
>> pretty well contained.  The pcie code is in probe,
>> cci_pci_ioremap_bar, cci_pci_release_regions, and the 'parse_s*'
>> functions that ultimately call cci_pci_ioremap_bar.  So it all comes
>> down to the code that calls the 'parse_s*' functions.  That is, the
>> code the reads the fme header and the port headers.  Currently, if I'm
>> reading this right, this code ioremaps each region, reads its header,
>> and creates its device before reading the next header.  So we just
>> need to change the order a little and ioremap/read all the headers
>> before starting to create devices.  That separates the pci ioremap
>> from creating the devices.  So the probe function could ioremap and
>> read in the fme header.  Then it would ioremap and read in each of the
>> port headers.  After they are read in, then pass the all these headers
>> to the separate enumeration code that will parse it and create the
>> devices.  That part is not pcie specific, right?  With the enumeration
>> code in a separate file (and assuming that the ioremap has already
>> happened),  another platform that is embedded could have its probe do
>> the iomapping and reading the headers, and then call the same
>> enumeration code.
>
> So it's suggested that we have some kind of pre-enumeration code in the
> intel-pcie.c to do ioremap and locate the address of device feature lists
> for FME and Ports, and hand them to common enumeration code. Is my
> understanding correct?

I was suggesting reading in the structures and saving that info to
pass to the enumeration code.  Passing the ioremapped addresses would
probably be just as fine.

Please keep in mind that I am just making suggestions about how I
think this can be made to be platform independent.  We can work out
the details, but my main point here is that this infrastructure is
going to need to support embedded FPGAs.  As you suggest below, if we
wait it will become more PCIe-centric and it will be harder later for
this code base to support other parts.

>
> I have considered this some time ago, this is doable for current code,
> as the pre-enumeration code will be simple (only need to find FME header
> and then read some register in FME header register set to locate the Port
> header).

A lot of my intention in reviewing this is to make sure the code is
broadly usable.  That way it will have  a future on other platforms,
for example the Stratix10 SoC.

That is also why it has been important to use the FPGA manager API and
expand it where needed.  (You made some good changes for v2 in this
area, thanks!)   When this infrastructure runs on a SoC, the FME will
be divided up, some of the functionality will appear as it currently
does, but the PR part will use whatever PR hardware the SoC has.

> But if in the future, some pcie specific things are added to some
> sub features of FME, then pre-enumeration code has to parse FME and all
> its sub features to handle pcie specific things. We will have more
> duplicate code on pre-enumeration in intel-pcie.c and common enumeration.
> Another point is, it's hard to avoid pcie specific things in sub features,
> as they may be designed to use PCIE specific function or resources.
> e.g some sub features require the interrupts from PCIE device, like PCIE
> MSI / MSIX. So keep enumeration code away from PCI things may be difficult.
>
>>
>> Actually, there's a bit more to it than that but I think this is
>> doable.  build_info_add_sub_feature() (which is called by
>> create_feature_instance) saves off resource info for each sub device.
>> Does this mean this code is ioremapping again (in
>> intel-fpga-fme-mgr.c's probe)?
>
> Yes, you are correct. Is it fine to have the MMIO region mapped twice?
> or we have to pass the mapped address to intel-fpga-fme-mgr directly?
>
>>
>> A few more things below...
>>
>> > From: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
>> >
>> > Device Feature List structure creates a link list of feature headers
>> > within the MMIO space to provide an extensible way of adding features.
>> >
>> > The Intel FPGA PCIe driver walks through the feature headers to enumerate
>> > feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
>> > Function Unit (AFU), and their private sub features. For feature devices,
>> > it creates the platform devices and linked the private sub features into
>> > their platform data.
>> >
>> > Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > Signed-off-by: Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > Signed-off-by: Zhang Yi <yi.z.zhang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
>> > Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > ---
>> > v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
>> >     switched to GPLv2 license.
>> >     fixed comments from Moritz Fischer.
>> >     fixed kbuild warning, typos and clean up the code.
>> > ---
>> >  drivers/fpga/Makefile            |   2 +-
>> >  drivers/fpga/intel-feature-dev.c | 130 ++++++
>> >  drivers/fpga/intel-feature-dev.h | 341 ++++++++++++++++
>> >  drivers/fpga/intel-pcie.c        | 841 ++++++++++++++++++++++++++++++++++++++-
>> >  4 files changed, 1311 insertions(+), 3 deletions(-)
>> >  create mode 100644 drivers/fpga/intel-feature-dev.c
>> >  create mode 100644 drivers/fpga/intel-feature-dev.h
>> >
>> > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
>> > index 5613133..ad24b3d 100644
>> > --- a/drivers/fpga/Makefile
>> > +++ b/drivers/fpga/Makefile
>> > @@ -31,4 +31,4 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
>> >  # Intel FPGA Support
>> >  obj-$(CONFIG_INTEL_FPGA_PCI)           += intel-fpga-pci.o
>> >
>> > -intel-fpga-pci-objs := intel-pcie.o
>> > +intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
>> > diff --git a/drivers/fpga/intel-feature-dev.c b/drivers/fpga/intel-feature-dev.c
>> > new file mode 100644
>> > index 0000000..68f9cba
>> > --- /dev/null
>> > +++ b/drivers/fpga/intel-feature-dev.c
>> > @@ -0,0 +1,130 @@
>> > +/*
>> > + * Intel FPGA Feature Device Driver
>> > + *
>> > + * Copyright (C) 2017 Intel Corporation, Inc.
>> > + *
>> > + * Authors:
>> > + *   Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > + *   Zhang Yi <yi.z.zhang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > + *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > + *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
>> > + *
>> > + * This work is licensed under the terms of the GNU GPL version 2. See
>> > + * the COPYING file in the top-level directory.
>> > + */
>> > +
>> > +#include "intel-feature-dev.h"
>> > +
>> > +void feature_platform_data_add(struct feature_platform_data *pdata,
>> > +                              int index, const char *name,
>> > +                              int resource_index, void __iomem *ioaddr)
>> > +{
>> > +       WARN_ON(index >= pdata->num);
>> > +
>> > +       pdata->features[index].name = name;
>> > +       pdata->features[index].resource_index = resource_index;
>> > +       pdata->features[index].ioaddr = ioaddr;
>> > +}
>> > +
>> > +struct feature_platform_data *
>> > +feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
>> > +{
>> > +       struct feature_platform_data *pdata;
>> > +
>> > +       pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
>> > +       if (pdata) {
>> > +               pdata->dev = dev;
>> > +               pdata->num = num;
>> > +               mutex_init(&pdata->lock);
>> > +       }
>> > +
>> > +       return pdata;
>> > +}
>> > +
>> > +int fme_feature_num(void)
>> > +{
>> > +       return FME_FEATURE_ID_MAX;
>> > +}
>> > +
>> > +int port_feature_num(void)
>> > +{
>> > +       return PORT_FEATURE_ID_MAX;
>> > +}
>>
>> Do these need to be functions?  If so, static.  But if you can just
>> use the #define'd values that's one level of indirection we can get
>> rid of.  And when I'm going through code where there's extra levels of
>> indirection that don't do anything, it makes it harder to understand
>> ;)
>
> Sure, will fix this.
>
>>
>> > +
>> > +int fpga_port_id(struct platform_device *pdev)
>> > +{
>> > +       struct feature_port_header *port_hdr;
>> > +       struct feature_port_capability capability;
>> > +
>> > +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
>> > +                                              PORT_FEATURE_ID_HEADER);
>> > +       WARN_ON(!port_hdr);
>> > +
>> > +       capability.csr = readq(&port_hdr->capability);
>> > +       return capability.port_number;
>> > +}
>> > +EXPORT_SYMBOL_GPL(fpga_port_id);
>>
>> This is here because every feature is a port, right?
>
> No, this fpga_port_id function will be used in both afu/port driver and
> also the pcie driver, so this function in added here.
> The code in pcie driver which uses this function isn't submitted yet, it's
> part of the SRIOV support. The pcie driver needs to manage which port could
> be turned to VF.
>
>>
>> > +
>> > +/*
>> > + * Enable Port by clear the port soft reset bit, which is set by default.
>> > + * The User AFU is unable to respond to any MMIO access while in reset.
>> > + * __fpga_port_enable function should only be used after __fpga_port_disable
>> > + * function.
>> > + */
>> > +void __fpga_port_enable(struct platform_device *pdev)
>> > +{
>> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> > +       struct feature_port_header *port_hdr;
>> > +       struct feature_port_control control;
>> > +
>> > +       WARN_ON(!pdata->disable_count);
>> > +
>> > +       if (--pdata->disable_count != 0)
>> > +               return;
>> > +
>> > +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
>> > +                                              PORT_FEATURE_ID_HEADER);
>> > +       WARN_ON(!port_hdr);
>> > +
>> > +       control.csr = readq(&port_hdr->control);
>> > +       control.port_sftrst = 0x0;
>> > +       writeq(control.csr, &port_hdr->control);
>> > +}
>> > +EXPORT_SYMBOL_GPL(__fpga_port_enable);
>> > +
>> > +#define RST_POLL_INVL 10 /* us */
>> > +#define RST_POLL_TIMEOUT 1000 /* us */
>> > +
>> > +int __fpga_port_disable(struct platform_device *pdev)
>> > +{
>> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> > +       struct feature_port_header *port_hdr;
>> > +       struct feature_port_control control;
>> > +
>> > +       if (pdata->disable_count++ != 0)
>> > +               return 0;
>> > +
>> > +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
>> > +                                              PORT_FEATURE_ID_HEADER);
>> > +       WARN_ON(!port_hdr);
>> > +
>> > +       /* Set port soft reset */
>> > +       control.csr = readq(&port_hdr->control);
>> > +       control.port_sftrst = 0x1;
>> > +       writeq(control.csr, &port_hdr->control);
>> > +
>> > +       /*
>> > +        * HW sets ack bit to 1 when all outstanding requests have been drained
>> > +        * on this port and minimum soft reset pulse width has elapsed.
>> > +        * Driver polls port_soft_reset_ack to determine if reset done by HW.
>> > +        */
>> > +       if (readq_poll_timeout(&port_hdr->control, control.csr,
>> > +                              (control.port_sftrst_ack == 1),
>> > +                              RST_POLL_INVL, RST_POLL_TIMEOUT)) {
>> > +               dev_err(&pdev->dev, "timeout, fail to reset device\n");
>> > +               return -ETIMEDOUT;
>> > +       }
>> > +
>> > +       return 0;
>> > +}
>> > +EXPORT_SYMBOL_GPL(__fpga_port_disable);
>> > diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
>> > new file mode 100644
>> > index 0000000..f67784a
>> > --- /dev/null
>> > +++ b/drivers/fpga/intel-feature-dev.h
>> > @@ -0,0 +1,341 @@
>> > +/*
>> > + * Intel FPGA Feature Device Driver Header File
>> > + *
>> > + * Copyright (C) 2017 Intel Corporation, Inc.
>> > + *
>> > + * Authors:
>> > + *   Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > + *   Zhang Yi <yi.z.zhang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > + *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > + *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
>> > + *
>> > + * This work is licensed under the terms of the GNU GPL version 2. See
>> > + * the COPYING file in the top-level directory.
>> > + */
>> > +
>> > +#ifndef __INTEL_FPGA_FEATURE_H
>> > +#define __INTEL_FPGA_FEATURE_H
>> > +
>> > +#include <linux/fs.h>
>> > +#include <linux/pci.h>
>> > +#include <linux/uuid.h>
>> > +#include <linux/delay.h>
>> > +#include <linux/iopoll.h>
>> > +#include <linux/platform_device.h>
>> > +
>> > +#ifndef readq
>> > +static inline u64 readq(void __iomem *addr)
>> > +{
>> > +       return readl(addr) + ((u64)readl(addr + 4) << 32);
>> > +}
>> > +#endif
>> > +
>> > +#ifndef writeq
>> > +static inline void writeq(u64 val, void __iomem *addr)
>> > +{
>> > +       writel((u32) (val), addr);
>> > +       writel((u32) (val >> 32), (addr + 4));
>> > +}
>> > +#endif
>> > +
>> > +/* maximum supported number of ports */
>> > +#define MAX_FPGA_PORT_NUM 4
>> > +/* plus one for fme device */
>> > +#define MAX_FEATURE_DEV_NUM    (MAX_FPGA_PORT_NUM + 1)
>> > +
>> > +#define FME_FEATURE_HEADER          "fme_hdr"
>> > +#define FME_FEATURE_THERMAL_MGMT    "fme_thermal"
>> > +#define FME_FEATURE_POWER_MGMT      "fme_power"
>> > +#define FME_FEATURE_GLOBAL_PERF     "fme_gperf"
>> > +#define FME_FEATURE_GLOBAL_ERR      "fme_error"
>> > +#define FME_FEATURE_PR_MGMT         "fme_pr"
>> > +
>> > +#define PORT_FEATURE_HEADER         "port_hdr"
>> > +#define PORT_FEATURE_UAFU           "port_uafu"
>> > +#define PORT_FEATURE_ERR            "port_err"
>> > +#define PORT_FEATURE_UMSG           "port_umsg"
>> > +#define PORT_FEATURE_PR             "port_pr"
>> > +#define PORT_FEATURE_STP            "port_stp"
>> > +
>> > +/* All headers and structures must be byte-packed to match the spec. */
>> > +#pragma pack(1)
>> > +
>> > +/* common header for all features */
>> > +struct feature_header {
>> > +       union {
>> > +               u64 csr;
>> > +               struct {
>> > +                       u64 id:12;
>> > +                       u64 revision:4;
>> > +                       u64 next_header_offset:24; /* offset to next header */
>> > +                       u64 rsvdz:20;
>> > +                       u64 type:4;                /* feature type */
>> > +#define FEATURE_TYPE_AFU               0x1
>> > +#define FEATURE_TYPE_PRIVATE           0x3
>> > +               };
>> > +       };
>> > +};
>> > +
>> > +/* common header for non-private features */
>> > +struct feature_afu_header {
>> > +       uuid_le guid;
>> > +       union {
>> > +               u64 csr;
>> > +               struct {
>> > +                       u64 next_afu:24;        /* pointer to next afu header */
>> > +                       u64 rsvdz:40;
>> > +               };
>> > +       };
>> > +};
>> > +
>> > +/* FME Header Register Set */
>> > +/* FME Capability Register */
>> > +struct feature_fme_capability {
>> > +       union {
>> > +               u64 csr;
>> > +               struct {
>> > +                       u64 fabric_verid:8;     /* Fabric version ID */
>> > +                       u64 socket_id:1;        /* Socket id */
>> > +                       u64 rsvdz1:3;
>> > +                       u64 pcie0_link_avl:1;   /* PCIe0 link availability */
>> > +                       u64 pcie1_link_avl:1;   /* PCIe1 link availability */
>> > +                       u64 coherent_link_avl:1;/* Coherent link availability */
>> > +                       u64 rsvdz2:1;
>> > +                       u64 iommu_support:1;    /* IOMMU or VT-d supported */
>> > +                       u64 num_ports:3;        /* Num of ports implemented */
>> > +                       u64 rsvdz3:4;
>> > +                       u64 addr_width_bits:6;  /* Address width supported */
>> > +                       u64 rsvdz4:2;
>> > +                       u64 cache_size:12;      /* Cache size in kb */
>> > +                       u64 cache_assoc:4;      /* Cache Associativity */
>> > +                       u64 rsvdz5:15;
>> > +                       u64 lock_bit:1;         /* Latched lock bit by BIOS */
>> > +               };
>> > +       };
>> > +};
>> > +
>> > +/* FME Port Offset Register */
>> > +struct feature_fme_port {
>> > +       union {
>> > +               u64 csr;
>> > +               struct {
>> > +                       u64 port_offset:24;     /* Offset to port header */
>> > +                       u64 rsvdz1:8;
>> > +                       u64 port_bar:3;         /* Bar id */
>> > +                       u64 rsvdz2:20;
>> > +                       u64 afu_access_ctrl:1;  /* AFU access type: PF/VF */
>> > +                       u64 rsvdz3:4;
>> > +                       u64 port_implemented:1; /* Port implemented or not */
>> > +                       u64 rsvdz4:3;
>> > +               };
>> > +       };
>> > +};
>> > +
>> > +struct feature_fme_header {
>> > +       struct feature_header header;
>> > +       struct feature_afu_header afu_header;
>> > +       u64 rsvd[2];
>> > +       struct feature_fme_capability capability;
>> > +       struct feature_fme_port port[MAX_FPGA_PORT_NUM];
>> > +};
>> > +
>> > +/* FME Thermal Sub Feature Register Set */
>> > +struct feature_fme_thermal {
>> > +       struct feature_header header;
>> > +};
>> > +
>> > +/* FME Power Sub Feature Register Set */
>> > +struct feature_fme_power {
>> > +       struct feature_header header;
>> > +};
>> > +
>> > +/* FME Global Performance Sub Feature Register Set */
>> > +struct feature_fme_gperf {
>> > +       struct feature_header header;
>> > +};
>> > +
>> > +/* FME Error Sub Feature Register Set */
>> > +struct feature_fme_err {
>> > +       struct feature_header header;
>> > +};
>> > +
>> > +/* FME Partial Reconfiguration Sub Feature Register Set */
>> > +struct feature_fme_pr {
>> > +       struct feature_header header;
>> > +};
>> > +
>> > +/* PORT Header Register Set */
>> > +/* Port Capability Register */
>> > +struct feature_port_capability {
>> > +       union {
>> > +               u64 csr;
>> > +               struct {
>> > +                       u64 port_number:2;      /* Port Number 0-3 */
>> > +                       u64 rsvdz1:6;
>> > +                       u64 mmio_size:16;       /* User MMIO size in KB */
>> > +                       u64 rsvdz2:8;
>> > +                       u64 sp_intr_num:4;      /* Supported interrupts num */
>> > +                       u64 rsvdz3:28;
>> > +               };
>> > +       };
>> > +};
>> > +
>> > +/* Port Control Register */
>> > +struct feature_port_control {
>> > +       union {
>> > +               u64 csr;
>> > +               struct {
>> > +                       u64 port_sftrst:1;      /* Port Soft Reset */
>> > +                       u64 rsvdz1:1;
>> > +                       u64 latency_tolerance:1;/* '1' >= 40us, '0' < 40us */
>> > +                       u64 rsvdz2:1;
>> > +                       u64 port_sftrst_ack:1;  /* HW ACK for Soft Reset */
>> > +                       u64 rsvdz3:59;
>> > +               };
>> > +       };
>> > +};
>> > +
>> > +struct feature_port_header {
>> > +       struct feature_header header;
>> > +       struct feature_afu_header afu_header;
>> > +       u64 rsvd[2];
>> > +       struct feature_port_capability capability;
>> > +       struct feature_port_control control;
>> > +};
>> > +
>> > +/* PORT Error Sub Feature Register Set */
>> > +struct feature_port_error {
>> > +       struct feature_header header;
>> > +};
>> > +
>> > +/* PORT Unordered Message Sub Feature Register Set */
>> > +struct feature_port_umsg {
>> > +       struct feature_header header;
>> > +};
>> > +
>> > +/* PORT SignalTap Sub Feature Register Set */
>> > +struct feature_port_stp {
>> > +       struct feature_header header;
>> > +};
>> > +
>> > +#pragma pack()
>> > +
>> > +struct feature {
>> > +       const char *name;
>> > +       int resource_index;
>> > +       void __iomem *ioaddr;
>> > +};
>> > +
>> > +struct feature_platform_data {
>> > +       /* list the feature dev to cci_drvdata->port_dev_list. */
>> > +       struct list_head node;
>> > +       struct mutex lock;
>> > +       struct platform_device *dev;
>> > +       unsigned int disable_count;     /* count for port disable */
>> > +
>> > +       int num;                        /* number of features */
>> > +       struct feature features[0];
>> > +};
>> > +
>> > +enum fme_feature_id {
>> > +       FME_FEATURE_ID_HEADER = 0x0,
>> > +       FME_FEATURE_ID_THERMAL_MGMT = 0x1,
>> > +       FME_FEATURE_ID_POWER_MGMT = 0x2,
>> > +       FME_FEATURE_ID_GLOBAL_PERF = 0x3,
>> > +       FME_FEATURE_ID_GLOBAL_ERR = 0x4,
>> > +       FME_FEATURE_ID_PR_MGMT = 0x5,
>> > +       FME_FEATURE_ID_MAX = 0x6,
>> > +};
>> > +
>> > +enum port_feature_id {
>> > +       PORT_FEATURE_ID_HEADER = 0x0,
>> > +       PORT_FEATURE_ID_ERROR = 0x1,
>> > +       PORT_FEATURE_ID_UMSG = 0x2,
>> > +       PORT_FEATURE_ID_PR = 0x3,
>> > +       PORT_FEATURE_ID_STP = 0x4,
>> > +       PORT_FEATURE_ID_UAFU = 0x5,
>> > +       PORT_FEATURE_ID_MAX = 0x6,
>> > +};
>> > +
>> > +int fme_feature_num(void);
>> > +int port_feature_num(void);
>> > +
>> > +#define FPGA_FEATURE_DEV_FME           "intel-fpga-fme"
>> > +#define FPGA_FEATURE_DEV_PORT          "intel-fpga-port"
>> > +
>> > +void feature_platform_data_add(struct feature_platform_data *pdata,
>> > +                              int index, const char *name,
>> > +                              int resource_index, void __iomem *ioaddr);
>> > +
>> > +static inline int feature_platform_data_size(const int num)
>> > +{
>> > +       return sizeof(struct feature_platform_data) +
>> > +               num * sizeof(struct feature);
>> > +}
>> > +
>> > +struct feature_platform_data *
>> > +feature_platform_data_alloc_and_init(struct platform_device *dev, int num);
>> > +
>> > +int fpga_port_id(struct platform_device *pdev);
>> > +
>> > +static inline int fpga_port_check_id(struct platform_device *pdev,
>> > +                                    void *pport_id)
>> > +{
>> > +       return fpga_port_id(pdev) == *(int *)pport_id;
>> > +}
>> > +
>> > +void __fpga_port_enable(struct platform_device *pdev);
>> > +int __fpga_port_disable(struct platform_device *pdev);
>> > +
>> > +static inline void fpga_port_enable(struct platform_device *pdev)
>> > +{
>> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> > +
>> > +       mutex_lock(&pdata->lock);
>> > +       __fpga_port_enable(pdev);
>> > +       mutex_unlock(&pdata->lock);
>> > +}
>> > +
>> > +static inline int fpga_port_disable(struct platform_device *pdev)
>> > +{
>> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> > +       int ret;
>> > +
>> > +       mutex_lock(&pdata->lock);
>> > +       ret = __fpga_port_disable(pdev);
>> > +       mutex_unlock(&pdata->lock);
>> > +
>> > +       return ret;
>> > +}
>> > +
>> > +static inline int __fpga_port_reset(struct platform_device *pdev)
>> > +{
>> > +       int ret;
>> > +
>> > +       ret = __fpga_port_disable(pdev);
>> > +       if (ret)
>> > +               return ret;
>> > +
>> > +       __fpga_port_enable(pdev);
>> > +       return 0;
>> > +}
>> > +
>> > +static inline int fpga_port_reset(struct platform_device *pdev)
>> > +{
>> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> > +       int ret;
>> > +
>> > +       mutex_lock(&pdata->lock);
>> > +       ret = __fpga_port_reset(pdev);
>> > +       mutex_unlock(&pdata->lock);
>> > +       return ret;
>> > +}
>> > +
>> > +static inline void __iomem *
>> > +get_feature_ioaddr_by_index(struct device *dev, int index)
>> > +{
>> > +       struct feature_platform_data *pdata = dev_get_platdata(dev);
>> > +
>> > +       return pdata->features[index].ioaddr;
>> > +}
>> > +#endif
>> > diff --git a/drivers/fpga/intel-pcie.c b/drivers/fpga/intel-pcie.c
>> > index f697de4..70b8284 100644
>> > --- a/drivers/fpga/intel-pcie.c
>> > +++ b/drivers/fpga/intel-pcie.c
>> > @@ -23,10 +23,827 @@
>> >  #include <linux/stddef.h>
>> >  #include <linux/errno.h>
>> >  #include <linux/aer.h>
>> > +#include <linux/fpga/fpga-dev.h>
>> > +
>> > +#include "intel-feature-dev.h"
>> >
>> >  #define DRV_VERSION    "0.8"
>> >  #define DRV_NAME       "intel-fpga-pci"
>> >
>> > +#define INTEL_FPGA_DEV "intel-fpga-dev"
>> > +
>> > +static DEFINE_MUTEX(fpga_id_mutex);
>> > +
>> > +enum fpga_id_type {
>> > +       FME_ID,         /* fme id allocation and mapping */
>> > +       PORT_ID,        /* port id allocation and mapping */
>> > +       FPGA_ID_MAX,
>> > +};
>> > +
>> > +/* it is protected by fpga_id_mutex */
>> > +static struct idr fpga_ids[FPGA_ID_MAX];
>> > +
>> > +struct cci_drvdata {
>> > +       struct device *fme_dev;
>> > +
>> > +       struct mutex lock;
>> > +       struct list_head port_dev_list;
>> > +
>> > +       struct list_head regions; /* global list of pci bar mapping region */
>> > +};
>> > +
>> > +/* pci bar mapping info */
>> > +struct cci_pci_region {
>> > +       int bar;
>> > +       void __iomem *ioaddr;   /* pointer to mapped bar region */
>> > +       struct list_head node;
>> > +};
>> > +
>> > +static void fpga_ids_init(void)
>> > +{
>> > +       int i;
>> > +
>> > +       for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
>> > +               idr_init(fpga_ids + i);
>> > +}
>> > +
>> > +static void fpga_ids_destroy(void)
>> > +{
>> > +       int i;
>> > +
>> > +       for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
>> > +               idr_destroy(fpga_ids + i);
>> > +}
>> > +
>> > +static int alloc_fpga_id(enum fpga_id_type type, struct device *dev)
>> > +{
>> > +       int id;
>> > +
>> > +       WARN_ON(type >= FPGA_ID_MAX);
>> > +       mutex_lock(&fpga_id_mutex);
>> > +       id = idr_alloc(fpga_ids + type, dev, 0, 0, GFP_KERNEL);
>> > +       mutex_unlock(&fpga_id_mutex);
>> > +       return id;
>> > +}
>> > +
>> > +static void free_fpga_id(enum fpga_id_type type, int id)
>> > +{
>> > +       WARN_ON(type >= FPGA_ID_MAX);
>> > +       mutex_lock(&fpga_id_mutex);
>> > +       idr_remove(fpga_ids + type, id);
>> > +       mutex_unlock(&fpga_id_mutex);
>> > +}
>> > +
>> > +static void cci_pci_add_port_dev(struct pci_dev *pdev,
>> > +                                struct platform_device *port_dev)
>> > +{
>> > +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
>> > +       struct feature_platform_data *pdata = dev_get_platdata(&port_dev->dev);
>> > +
>> > +       mutex_lock(&drvdata->lock);
>> > +       list_add(&pdata->node, &drvdata->port_dev_list);
>> > +       get_device(&pdata->dev->dev);
>> > +       mutex_unlock(&drvdata->lock);
>> > +}
>> > +
>> > +static void cci_pci_remove_port_devs(struct pci_dev *pdev)
>> > +{
>> > +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
>> > +       struct feature_platform_data *pdata, *ptmp;
>> > +
>> > +       mutex_lock(&drvdata->lock);
>> > +       list_for_each_entry_safe(pdata, ptmp, &drvdata->port_dev_list, node) {
>> > +               struct platform_device *port_dev = pdata->dev;
>> > +
>> > +               /* the port should be unregistered first. */
>> > +               WARN_ON(device_is_registered(&port_dev->dev));
>> > +               list_del(&pdata->node);
>> > +               free_fpga_id(PORT_ID, port_dev->id);
>> > +               put_device(&port_dev->dev);
>> > +       }
>> > +       mutex_unlock(&drvdata->lock);
>> > +}
>> > +
>> > +/* info collection during feature dev build. */
>> > +struct build_feature_devs_info {
>> > +       struct pci_dev *pdev;
>> > +
>> > +       /*
>> > +        * PCI BAR mapping info. Parsing feature list starts from
>> > +        * BAR 0 and switch to different BARs to parse Port
>> > +        */
>> > +       void __iomem *ioaddr;
>> > +       void __iomem *ioend;
>> > +       int current_bar;
>> > +
>> > +       /* points to FME header where the port offset is figured out. */
>> > +       void __iomem *pfme_hdr;
>> > +
>> > +       /* the container device for all feature devices */
>> > +       struct fpga_dev *parent_dev;
>> > +
>> > +       /* current feature device */
>> > +       struct platform_device *feature_dev;
>> > +};
>> > +
>> > +static void cci_pci_release_regions(struct pci_dev *pdev)
>> > +{
>> > +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
>> > +       struct cci_pci_region *tmp, *region;
>> > +
>> > +       list_for_each_entry_safe(region, tmp, &drvdata->regions, node) {
>> > +               list_del(&region->node);
>> > +               if (region->ioaddr)
>> > +                       pci_iounmap(pdev, region->ioaddr);
>> > +               devm_kfree(&pdev->dev, region);
>> > +       }
>> > +}
>> > +
>> > +static void __iomem *cci_pci_ioremap_bar(struct pci_dev *pdev, int bar)
>> > +{
>> > +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
>> > +       struct cci_pci_region *region;
>> > +
>> > +       list_for_each_entry(region, &drvdata->regions, node)
>> > +               if (region->bar == bar) {
>> > +                       dev_dbg(&pdev->dev, "BAR %d region exists\n", bar);
>> > +                       return region->ioaddr;
>> > +               }
>> > +
>> > +       region = devm_kzalloc(&pdev->dev, sizeof(*region), GFP_KERNEL);
>> > +       if (!region)
>> > +               return NULL;
>> > +
>> > +       region->bar = bar;
>> > +       region->ioaddr = pci_ioremap_bar(pdev, bar);
>> > +       if (!region->ioaddr) {
>> > +               dev_err(&pdev->dev, "can't ioremap memory from BAR %d.\n", bar);
>> > +               devm_kfree(&pdev->dev, region);
>> > +               return NULL;
>> > +       }
>> > +
>> > +       list_add(&region->node, &drvdata->regions);
>> > +       return region->ioaddr;
>> > +}
>> > +
>> > +static int parse_start_from(struct build_feature_devs_info *binfo, int bar)
>> > +{
>> > +       binfo->ioaddr = cci_pci_ioremap_bar(binfo->pdev, bar);
>> > +       if (!binfo->ioaddr)
>> > +               return -ENOMEM;
>> > +
>> > +       binfo->current_bar = bar;
>> > +       binfo->ioend = binfo->ioaddr + pci_resource_len(binfo->pdev, bar);
>> > +       return 0;
>> > +}
>> > +
>> > +static int parse_start(struct build_feature_devs_info *binfo)
>> > +{
>> > +       /* fpga feature list starts from BAR 0 */
>> > +       return parse_start_from(binfo, 0);
>> > +}
>> > +
>> > +/* switch the memory mapping to BAR# @bar */
>> > +static int parse_switch_to(struct build_feature_devs_info *binfo, int bar)
>> > +{
>> > +       return parse_start_from(binfo, bar);
>> > +}
>>
>> parse_switch_to() and parse_start() are both one line wrappers for
>> parse_start_from() and are only called once.  :-) Please just use
>> parse_start_from and get rid of parse_start and parse_switch_to.  I
>> don't think this code is all that super complicated.  But as I'm
>> grepping through this code, I'm trying to see all the places that do
>> the same thing; if I have to look for multiple function names, it
>> takes longer to get what is going on here.
>>
>> Actually, cci_pci_ioremap_bar is only called in one place -
>> parse_start_from - which really just adds two lines of code.
>> parse_start_from could go away.  Just a suggestion.  The reason this
>> comes up is that cci_pci_ioremap_bar is the more descriptive function
>> name.  The parse_start/switch function names hide the biggest thing
>> they do, which is ioremapping.  They don't actually do any parsing ;)
>
> I think I need to explain a little more on the enumeration steps in this
> cci_pci_create_feature_devs function, and we use these names for these
> functions.
>
> 1) parse_start -> that initializes the enumeration from PCI Bar0.
>    In PF case, the first byte of BAR0, will be FME header.
>    In VF case, the first byte of BAR0, will be Port header.
>    (FME module is always in PF, no FME module in VF).
>
> 2) parse_feature_list -> parse one device feature list
>
>    This function parses a feature list, it could be FME device feature
>    list (PF) or Port device feature list (VF).
>    FME device feature list and Port device feature list are not linked
>    together, as they are in differnet BARs.
>
> 3) parse_port_from_fme -> parse Port device feature lists if FME presents.
>
>    If FME is detected (PF), read FME registers to know the location (BAR n
>    + offset) of each port's header (port device feature list).
>    call parse_switch_to(BAR n) and call parse_feature_list to parse the
>    port device feature list.
>
>    If FME is not detected (VF), just do nothing.
>
> Do you think if we could keep these functions?

I'm OK with parse_feature_list and parse_port_from_fme, etc.  I
intended to just comment about parse_start, parse_start_from, and
parse_switch_to.  I was suggesting to get rid of parse_start and
parse_switch_to and just use parse_start_from instead.

> We use these names to
> help people to understand the enumeration process better, but looks like
> I should add more descriptions/comments here to avoid misunderstanding. :)
>
>>
>> > +
>> > +static struct build_feature_devs_info *
>> > +build_info_alloc_and_init(struct pci_dev *pdev)
>> > +{
>> > +       struct build_feature_devs_info *binfo;
>> > +
>> > +       binfo = devm_kzalloc(&pdev->dev, sizeof(*binfo), GFP_KERNEL);
>> > +       if (binfo)
>> > +               binfo->pdev = pdev;
>>
>> build_info_alloc_and_init() is only doing a devm_kzalloc and is only
>> called once.  The code would be more readable if you just had the
>> devm_kzalloc in the function that called this
>> (cci_pci_create_feature_devs).
>
> Sure.
>
>>
>> > +
>> > +       return binfo;
>> > +}
>> > +
>> > +static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
>> > +{
>> > +       if (!strcmp(pdev->name, FPGA_FEATURE_DEV_FME))
>> > +               return FME_ID;
>> > +
>> > +       if (!strcmp(pdev->name, FPGA_FEATURE_DEV_PORT))
>> > +               return PORT_ID;
>> > +
>> > +       WARN_ON(1);
>> > +       return FPGA_ID_MAX;
>> > +}
>> > +
>> > +/*
>> > + * register current feature device, it is called when we need to switch to
>> > + * another feature parsing or we have parsed all features
>> > + */
>> > +static int build_info_commit_dev(struct build_feature_devs_info *binfo)
>> > +{
>> > +       int ret;
>> > +
>> > +       if (!binfo->feature_dev)
>> > +               return 0;
>> > +
>> > +       ret = platform_device_add(binfo->feature_dev);
>> > +       if (!ret) {
>> > +               struct cci_drvdata *drvdata;
>> > +
>> > +               drvdata = dev_get_drvdata(&binfo->pdev->dev);
>> > +               if (feature_dev_id_type(binfo->feature_dev) == PORT_ID)
>> > +                       cci_pci_add_port_dev(binfo->pdev, binfo->feature_dev);
>> > +               else
>> > +                       drvdata->fme_dev = get_device(&binfo->feature_dev->dev);
>> > +
>> > +               /*
>> > +                * reset it to avoid build_info_free() freeing their resource.
>> > +                *
>> > +                * The resource of successfully registered feature devices
>> > +                * will be freed by platform_device_unregister(). See the
>> > +                * comments in build_info_create_dev().
>> > +                */
>> > +               binfo->feature_dev = NULL;
>> > +       }
>> > +
>> > +       return ret;
>> > +}
>> > +
>> > +static int
>> > +build_info_create_dev(struct build_feature_devs_info *binfo,
>> > +                     enum fpga_id_type type, int feature_nr, const char *name)
>> > +{
>> > +       struct platform_device *fdev;
>> > +       struct resource *res;
>> > +       struct feature_platform_data *pdata;
>> > +       int ret;
>> > +
>> > +       /* we will create a new device, commit current device first */
>> > +       ret = build_info_commit_dev(binfo);
>> > +       if (ret)
>> > +               return ret;
>> > +
>> > +       /*
>> > +        * we use -ENODEV as the initialization indicator which indicates
>> > +        * whether the id need to be reclaimed
>> > +        */
>> > +       fdev = binfo->feature_dev = platform_device_alloc(name, -ENODEV);
>> > +       if (!fdev)
>> > +               return -ENOMEM;
>> > +
>> > +       fdev->id = alloc_fpga_id(type, &fdev->dev);
>> > +       if (fdev->id < 0)
>> > +               return fdev->id;
>> > +
>> > +       fdev->dev.parent = &binfo->parent_dev->dev;
>> > +
>> > +       /*
>> > +        * we do not need to care for the memory which is associated with
>> > +        * the platform device. After calling platform_device_unregister(),
>> > +        * it will be automatically freed by device's release() callback,
>> > +        * platform_device_release().
>> > +        */
>> > +       pdata = feature_platform_data_alloc_and_init(fdev, feature_nr);
>> > +       if (!pdata)
>> > +               return -ENOMEM;
>> > +
>> > +       /*
>> > +        * the count should be initialized to 0 to make sure
>> > +        *__fpga_port_enable() following __fpga_port_disable()
>> > +        * works properly for port device.
>> > +        * and it should always be 0 for fme device.
>> > +        */
>> > +       WARN_ON(pdata->disable_count);
>> > +
>> > +       fdev->dev.platform_data = pdata;
>> > +       fdev->num_resources = feature_nr;
>> > +       fdev->resource = kcalloc(feature_nr, sizeof(*res), GFP_KERNEL);
>> > +       if (!fdev->resource)
>> > +               return -ENOMEM;
>> > +
>> > +       return 0;
>> > +}
>> > +
>> > +static int remove_feature_dev(struct device *dev, void *data)
>> > +{
>> > +       struct platform_device *pdev = to_platform_device(dev);
>> > +
>> > +       platform_device_unregister(pdev);
>> > +       return 0;
>> > +}
>> > +
>> > +static int remove_parent_dev(struct device *dev, void *data)
>> > +{
>> > +       /* remove platform devices attached in the parent device */
>> > +       device_for_each_child(dev, NULL, remove_feature_dev);
>> > +       fpga_dev_destroy(to_fpga_dev(dev));
>> > +       return 0;
>> > +}
>> > +
>> > +static void remove_all_devs(struct pci_dev *pdev)
>> > +{
>> > +       /* remove parent device and all its children. */
>> > +       device_for_each_child(&pdev->dev, NULL, remove_parent_dev);
>> > +}
>> > +
>> > +static void build_info_free(struct build_feature_devs_info *binfo)
>> > +{
>> > +       if (!IS_ERR_OR_NULL(binfo->parent_dev))
>> > +               remove_all_devs(binfo->pdev);
>> > +
>> > +       /*
>> > +        * it is a valid id, free it. See comments in
>> > +        * build_info_create_dev()
>> > +        */
>> > +       if (binfo->feature_dev && binfo->feature_dev->id >= 0)
>> > +               free_fpga_id(feature_dev_id_type(binfo->feature_dev),
>> > +                            binfo->feature_dev->id);
>> > +
>> > +       platform_device_put(binfo->feature_dev);
>> > +
>> > +       devm_kfree(&binfo->pdev->dev, binfo);
>> > +}
>> > +
>> > +#define FEATURE_TYPE_AFU       0x1
>> > +#define FEATURE_TYPE_PRIVATE   0x3
>> > +
>> > +/* FME and PORT GUID are fixed */
>> > +#define FEATURE_FME_GUID "f9e17764-38f0-82fe-e346-524ae92aafbf"
>> > +#define FEATURE_PORT_GUID "6b355b87-b06c-9642-eb42-8d139398b43a"
>> > +
>> > +static bool feature_is_fme(struct feature_afu_header *afu_hdr)
>> > +{
>> > +       uuid_le u;
>> > +
>> > +       uuid_le_to_bin(FEATURE_FME_GUID, &u);
>> > +
>> > +       return !uuid_le_cmp(u, afu_hdr->guid);
>> > +}
>> > +
>> > +static bool feature_is_port(struct feature_afu_header *afu_hdr)
>> > +{
>> > +       uuid_le u;
>> > +
>> > +       uuid_le_to_bin(FEATURE_PORT_GUID, &u);
>> > +
>> > +       return !uuid_le_cmp(u, afu_hdr->guid);
>> > +}
>> > +
>> > +/*
>> > + * UAFU GUID is dynamic as it can be changed after FME downloads different
>> > + * Green Bitstream to the port, so we treat the unknown GUIDs which are
>> > + * attached on port's feature list as UAFU.
>> > + */
>> > +static bool feature_is_UAFU(struct build_feature_devs_info *binfo)
>> > +{
>> > +       if (!binfo->feature_dev ||
>> > +             feature_dev_id_type(binfo->feature_dev) != PORT_ID)
>> > +               return false;
>> > +
>> > +       return true;
>> > +}
>> > +
>> > +static void
>> > +build_info_add_sub_feature(struct build_feature_devs_info *binfo,
>> > +                          int feature_id, const char *feature_name,
>> > +                          resource_size_t resource_size, void __iomem *start)
>> > +{
>> > +
>> > +       struct platform_device *fdev = binfo->feature_dev;
>> > +       struct feature_platform_data *pdata = dev_get_platdata(&fdev->dev);
>> > +       struct resource *res = &fdev->resource[feature_id];
>> > +
>> > +       res->start = pci_resource_start(binfo->pdev, binfo->current_bar) +
>> > +               start - binfo->ioaddr;
>> > +       res->end = res->start + resource_size - 1;
>> > +       res->flags = IORESOURCE_MEM;
>> > +       res->name = feature_name;
>> > +
>> > +       feature_platform_data_add(pdata, feature_id,
>> > +                                 feature_name, feature_id, start);
>> > +}
>> > +
>> > +struct feature_info {
>> > +       const char *name;
>> > +       resource_size_t resource_size;
>> > +       int feature_index;
>> > +};
>> > +
>> > +/* indexed by fme feature IDs which are defined in 'enum fme_feature_id'. */
>> > +static struct feature_info fme_features[] = {
>> > +       {
>> > +               .name = FME_FEATURE_HEADER,
>> > +               .resource_size = sizeof(struct feature_fme_header),
>> > +               .feature_index = FME_FEATURE_ID_HEADER,
>> > +       },
>> > +       {
>> > +               .name = FME_FEATURE_THERMAL_MGMT,
>> > +               .resource_size = sizeof(struct feature_fme_thermal),
>> > +               .feature_index = FME_FEATURE_ID_THERMAL_MGMT,
>> > +       },
>> > +       {
>> > +               .name = FME_FEATURE_POWER_MGMT,
>> > +               .resource_size = sizeof(struct feature_fme_power),
>> > +               .feature_index = FME_FEATURE_ID_POWER_MGMT,
>> > +       },
>> > +       {
>> > +               .name = FME_FEATURE_GLOBAL_PERF,
>> > +               .resource_size = sizeof(struct feature_fme_gperf),
>> > +               .feature_index = FME_FEATURE_ID_GLOBAL_PERF,
>> > +       },
>> > +       {
>> > +               .name = FME_FEATURE_GLOBAL_ERR,
>> > +               .resource_size = sizeof(struct feature_fme_err),
>> > +               .feature_index = FME_FEATURE_ID_GLOBAL_ERR,
>> > +       },
>> > +       {
>> > +               .name = FME_FEATURE_PR_MGMT,
>> > +               .resource_size = sizeof(struct feature_fme_pr),
>> > +               .feature_index = FME_FEATURE_ID_PR_MGMT,
>> > +       }
>> > +};
>> > +
>> > +/* indexed by port feature IDs which are defined in 'enum port_feature_id'. */
>> > +static struct feature_info port_features[] = {
>> > +       {
>> > +               .name = PORT_FEATURE_HEADER,
>> > +               .resource_size = sizeof(struct feature_port_header),
>> > +               .feature_index = PORT_FEATURE_ID_HEADER,
>> > +       },
>> > +       {
>> > +               .name = PORT_FEATURE_ERR,
>> > +               .resource_size = sizeof(struct feature_port_error),
>> > +               .feature_index = PORT_FEATURE_ID_ERROR,
>> > +       },
>> > +       {
>> > +               .name = PORT_FEATURE_UMSG,
>> > +               .resource_size = sizeof(struct feature_port_umsg),
>> > +               .feature_index = PORT_FEATURE_ID_UMSG,
>> > +       },
>> > +       {
>> > +               /* This feature isn't available for now */
>> > +               .name = PORT_FEATURE_PR,
>> > +               .resource_size = 0,
>> > +               .feature_index = PORT_FEATURE_ID_PR,
>> > +       },
>> > +       {
>> > +               .name = PORT_FEATURE_STP,
>> > +               .resource_size = sizeof(struct feature_port_stp),
>> > +               .feature_index = PORT_FEATURE_ID_STP,
>> > +       },
>> > +       {
>> > +               /*
>> > +                * For User AFU feature, its region size is not fixed, but
>> > +                * reported by register PortCapability.mmio_size. Resource
>> > +                * size of UAFU will be set while parse port device.
>> > +                */
>> > +               .name = PORT_FEATURE_UAFU,
>> > +               .resource_size = 0,
>> > +               .feature_index = PORT_FEATURE_ID_UAFU,
>> > +       },
>> > +};
>> > +
>> > +static int
>> > +create_feature_instance(struct build_feature_devs_info *binfo,
>> > +                       void __iomem *start, struct feature_info *finfo)
>> > +{
>> > +       if (binfo->ioend - start < finfo->resource_size)
>> > +               return -EINVAL;
>> > +
>> > +       build_info_add_sub_feature(binfo, finfo->feature_index, finfo->name,
>> > +                                  finfo->resource_size, start);
>> > +       return 0;
>> > +}
>>
>> build_info_add_sub_feature() is only called one time - here.  Could
>> you collapse these two functions together?
>
> Sure.
>
>>
>> > +
>> > +static int parse_feature_fme(struct build_feature_devs_info *binfo,
>> > +                            void __iomem *start)
>> > +{
>> > +       struct cci_drvdata *drvdata = dev_get_drvdata(&binfo->pdev->dev);
>> > +       int ret;
>> > +
>> > +       ret = build_info_create_dev(binfo, FME_ID, fme_feature_num(),
>> > +                                       FPGA_FEATURE_DEV_FME);
>> > +       if (ret)
>> > +               return ret;
>> > +
>> > +       if (drvdata->fme_dev) {
>> > +               dev_err(&binfo->pdev->dev, "Multiple FMEs are detected.\n");
>> > +               return -EINVAL;
>> > +       }
>> > +
>> > +       return create_feature_instance(binfo, start,
>> > +                                      &fme_features[FME_FEATURE_ID_HEADER]);
>> > +}
>> > +
>> > +static int parse_feature_fme_private(struct build_feature_devs_info *binfo,
>> > +                                    struct feature_header *hdr)
>> > +{
>> > +       struct feature_header header;
>> > +
>> > +       header.csr = readq(hdr);
>> > +
>> > +       if (header.id >= ARRAY_SIZE(fme_features)) {
>> > +               dev_info(&binfo->pdev->dev, "FME feature id %x is not supported yet.\n",
>> > +                        header.id);
>> > +               return 0;
>> > +       }
>> > +
>> > +       return create_feature_instance(binfo, hdr, &fme_features[header.id]);
>> > +}
>> > +
>> > +static int parse_feature_port(struct build_feature_devs_info *binfo,
>> > +                            void __iomem *start)
>> > +{
>> > +       int ret;
>> > +
>> > +       ret = build_info_create_dev(binfo, PORT_ID, port_feature_num(),
>> > +                                       FPGA_FEATURE_DEV_PORT);
>> > +       if (ret)
>> > +               return ret;
>> > +
>> > +       return create_feature_instance(binfo, start,
>> > +                                      &port_features[PORT_FEATURE_ID_HEADER]);
>> > +}
>> > +
>> > +static void enable_port_uafu(struct build_feature_devs_info *binfo,
>> > +                            void __iomem *start)
>> > +{
>> > +       enum port_feature_id id = PORT_FEATURE_ID_UAFU;
>> > +       struct feature_port_header *port_hdr;
>> > +       struct feature_port_capability capability;
>> > +
>> > +       port_hdr = (struct feature_port_header *)start;
>> > +       capability.csr = readq(&port_hdr->capability);
>> > +       port_features[id].resource_size = capability.mmio_size << 10;
>> > +
>> > +       /*
>> > +        * To enable User AFU, driver needs to clear reset bit on related port,
>> > +        * otherwise the mmio space of this user AFU will be invalid.
>> > +        */
>> > +       if (port_features[id].resource_size)
>> > +               fpga_port_reset(binfo->feature_dev);
>> > +}
>> > +
>> > +static int parse_feature_port_private(struct build_feature_devs_info *binfo,
>> > +                                     struct feature_header *hdr)
>> > +{
>> > +       struct feature_header header;
>> > +       enum port_feature_id id;
>> > +
>> > +       header.csr = readq(hdr);
>> > +       /*
>> > +        * the region of port feature id is [0x10, 0x13], + 1 to reserve 0
>> > +        * which is dedicated for port-hdr.
>> > +        */
>> > +       id = (header.id & 0x000f) + 1;
>> > +
>> > +       if (id >= ARRAY_SIZE(port_features)) {
>> > +               dev_info(&binfo->pdev->dev, "Port feature id %x is not supported yet.\n",
>> > +                        header.id);
>> > +               return 0;
>> > +       }
>> > +
>> > +       return create_feature_instance(binfo, hdr, &port_features[id]);
>> > +}
>> > +
>> > +static int parse_feature_port_uafu(struct build_feature_devs_info *binfo,
>> > +                                struct feature_header *hdr)
>> > +{
>> > +       enum port_feature_id id = PORT_FEATURE_ID_UAFU;
>> > +       int ret;
>> > +
>> > +       if (port_features[id].resource_size) {
>> > +               ret = create_feature_instance(binfo, hdr, &port_features[id]);
>> > +               port_features[id].resource_size = 0;
>> > +       } else {
>> > +               dev_err(&binfo->pdev->dev, "the uafu feature header is mis-configured.\n");
>> > +               ret = -EINVAL;
>> > +       }
>> > +
>> > +       return ret;
>> > +}
>> > +
>> > +static int parse_feature_afus(struct build_feature_devs_info *binfo,
>> > +                             struct feature_header *hdr)
>> > +{
>> > +       int ret;
>> > +       struct feature_afu_header *afu_hdr, header;
>> > +       void __iomem *start;
>> > +       void __iomem *end = binfo->ioend;
>> > +
>> > +       start = hdr;
>> > +       for (; start < end; start += header.next_afu) {
>> > +               if (end - start < (sizeof(*afu_hdr) + sizeof(*hdr)))
>> > +                       return -EINVAL;
>> > +
>> > +               hdr = start;
>> > +               afu_hdr = (struct feature_afu_header *) (hdr + 1);
>> > +               header.csr = readq(&afu_hdr->csr);
>> > +
>> > +               if (feature_is_fme(afu_hdr)) {
>> > +                       ret = parse_feature_fme(binfo, hdr);
>> > +                       binfo->pfme_hdr = hdr;
>> > +                       if (ret)
>> > +                               return ret;
>> > +               } else if (feature_is_port(afu_hdr)) {
>> > +                       ret = parse_feature_port(binfo, hdr);
>> > +                       enable_port_uafu(binfo, hdr);
>> > +                       if (ret)
>> > +                               return ret;
>> > +               } else if (feature_is_UAFU(binfo)) {
>> > +                       ret = parse_feature_port_uafu(binfo, hdr);
>> > +                       if (ret)
>> > +                               return ret;
>> > +               } else
>> > +                       dev_info(&binfo->pdev->dev, "AFU GUID %pUl is not supported yet.\n",
>> > +                                afu_hdr->guid.b);
>> > +
>> > +               if (!header.next_afu)
>> > +                       break;
>> > +       }
>> > +
>> > +       return 0;
>> > +}
>> > +
>> > +static int parse_feature_private(struct build_feature_devs_info *binfo,
>> > +                                struct feature_header *hdr)
>> > +{
>> > +       struct feature_header header;
>> > +
>> > +       header.csr = readq(hdr);
>> > +
>> > +       if (!binfo->feature_dev) {
>> > +               dev_err(&binfo->pdev->dev, "the private feature %x does not belong to any AFU.\n",
>> > +                       header.id);
>> > +               return -EINVAL;
>> > +       }
>> > +
>> > +       switch (feature_dev_id_type(binfo->feature_dev)) {
>> > +       case FME_ID:
>> > +               return parse_feature_fme_private(binfo, hdr);
>> > +       case PORT_ID:
>> > +               return parse_feature_port_private(binfo, hdr);
>> > +       default:
>> > +               dev_info(&binfo->pdev->dev, "private feature %x belonging to AFU %s is not supported yet.\n",
>> > +                        header.id, binfo->feature_dev->name);
>> > +       }
>> > +       return 0;
>> > +}
>> > +
>> > +static int parse_feature(struct build_feature_devs_info *binfo,
>> > +                        struct feature_header *hdr)
>> > +{
>> > +       struct feature_header header;
>> > +       int ret = 0;
>> > +
>> > +       header.csr = readq(hdr);
>> > +
>> > +       switch (header.type) {
>> > +       case FEATURE_TYPE_AFU:
>> > +               ret = parse_feature_afus(binfo, hdr);
>> > +               break;
>> > +       case FEATURE_TYPE_PRIVATE:
>> > +               ret = parse_feature_private(binfo, hdr);
>> > +               break;
>> > +       default:
>> > +               dev_info(&binfo->pdev->dev,
>> > +                        "Feature Type %x is not supported.\n", hdr->type);
>> > +       };
>> > +
>> > +       return ret;
>> > +}
>> > +
>> > +static int
>> > +parse_feature_list(struct build_feature_devs_info *binfo, void __iomem *start)
>> > +{
>> > +       struct feature_header *hdr, header;
>> > +       void __iomem *end = binfo->ioend;
>> > +       int ret = 0;
>> > +
>>
>> Maybe a helpful comment that we are stepping through a linked list of features.
>
> Sure, will add more comments.
>
> All kind of features has a common header, they are linked together via the
> header.next_header_offset.
>
> If it's a TYPE=AFU feature, it will have one more pointer named NEXT_AFU,
> which could point to another TYPE=AFU feature, please refer to function
> parse_feature_afus.
>
> If it's a TYPE=private feature, there is no other pointers, please refer
> to function parse_feature_private.

OK, yes that will be helpful.

>
>>
>> > +       for (; start < end; start += header.next_header_offset) {
>> > +               if (end - start < sizeof(*hdr)) {
>> > +                       dev_err(&binfo->pdev->dev, "The region is too small to contain a feature.\n");
>> > +                       ret =  -EINVAL;
>> > +                       break;
>> > +               }
>> > +
>> > +               hdr = (struct feature_header *)start;
>> > +               ret = parse_feature(binfo, hdr);
>> > +               if (ret)
>> > +                       break;
>>
>> Instead of parse_feature, this can save off the enumeration info and
>> continue to read in the linked list.  After all the headers are read
>> in, call the (separate) enumeration code to step through the saved
>> headers, parse them, and create the devices.  Since the memory is
>> iomapped during the process of reading in the headers, the enumeration
>> code doesn't have to be so pcie specific.  Then this code base is
>> better set to run on embedded devices also.
>
> Actually, it has parse_feature here, because they're different type of
> features. e.g for TYPE = AFU, GUID = PORT, it requires driver to check
> its NEXT_AFU pointer to find User AFU (and its header). For private
> features, there is no next_afu pointer at all.
>
>>
>> > +
>> > +               header.csr = readq(hdr);
>> > +               if (!header.next_header_offset)
>> > +                       break;
>> > +       }
>> > +
>> > +       return ret;
>> > +}
>> > +
>> > +static int parse_ports_from_fme(struct build_feature_devs_info *binfo)
>> > +{
>> > +       struct feature_fme_header *fme_hdr;
>> > +       struct feature_fme_port port;
>> > +       int i = 0, ret = 0;
>> > +
>> > +       if (binfo->pfme_hdr == NULL) {
>> > +               dev_dbg(&binfo->pdev->dev, "VF is detected.\n");
>> > +               return ret;
>> > +       }
>> > +
>> > +       fme_hdr = binfo->pfme_hdr;
>> > +
>> > +       do {
>> > +               port.csr = readq(&fme_hdr->port[i]);
>> > +               if (!port.port_implemented)
>> > +                       break;
>> > +
>> > +               ret = parse_switch_to(binfo, port.port_bar);
>> > +               if (ret)
>> > +                       break;
>> > +
>> > +               ret = parse_feature_list(binfo,
>> > +                               binfo->ioaddr + port.port_offset);
>> > +               if (ret)
>> > +                       break;
>> > +       } while (++i < MAX_FPGA_PORT_NUM);
>> > +
>> > +       return ret;
>> > +}
>> > +
>> > +static int create_init_drvdata(struct pci_dev *pdev)
>> > +{
>> > +       struct cci_drvdata *drvdata;
>> > +
>> > +       drvdata = devm_kzalloc(&pdev->dev, sizeof(*drvdata), GFP_KERNEL);
>> > +       if (!drvdata)
>> > +               return -ENOMEM;
>> > +
>> > +       mutex_init(&drvdata->lock);
>> > +       INIT_LIST_HEAD(&drvdata->port_dev_list);
>> > +       INIT_LIST_HEAD(&drvdata->regions);
>> > +
>> > +       dev_set_drvdata(&pdev->dev, drvdata);
>> > +       return 0;
>> > +}
>> > +
>> > +static void destroy_drvdata(struct pci_dev *pdev)
>> > +{
>> > +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
>> > +
>> > +       if (drvdata->fme_dev) {
>> > +               /* fme device should be unregistered first. */
>> > +               WARN_ON(device_is_registered(drvdata->fme_dev));
>> > +               free_fpga_id(FME_ID, to_platform_device(drvdata->fme_dev)->id);
>> > +               put_device(drvdata->fme_dev);
>> > +       }
>> > +
>> > +       cci_pci_remove_port_devs(pdev);
>> > +       cci_pci_release_regions(pdev);
>> > +       dev_set_drvdata(&pdev->dev, NULL);
>> > +       devm_kfree(&pdev->dev, drvdata);
>> > +}
>> > +
>> > +static int cci_pci_create_feature_devs(struct pci_dev *pdev)
>> > +{
>> > +       struct build_feature_devs_info *binfo;
>> > +       int ret;
>> > +
>> > +       binfo = build_info_alloc_and_init(pdev);
>> > +       if (!binfo)
>> > +               return -ENOMEM;
>> > +
>> > +       binfo->parent_dev = fpga_dev_create(&pdev->dev, INTEL_FPGA_DEV);
>> > +       if (IS_ERR(binfo->parent_dev)) {
>> > +               ret = PTR_ERR(binfo->parent_dev);
>> > +               goto free_binfo_exit;
>> > +       }
>> > +
>> > +       ret = parse_start(binfo);
>> > +       if (ret)
>> > +               goto free_binfo_exit;
>> > +
>> > +       ret = parse_feature_list(binfo, binfo->ioaddr);
>> > +       if (ret)
>> > +               goto free_binfo_exit;
>> > +
>> > +       ret = parse_ports_from_fme(binfo);
>> > +       if (ret)
>> > +               goto free_binfo_exit;
>>
>> So ideally, there would be a function call here that read all the
>> headers from hardware, ioremapping as it went along.  Then after that,
>> call the enumeration code to create the devices.
>
> If we have a function call here which read all the headers, then it
> finish most works of parsing the device feature list. The only thing
> that enumeration code to do is to create the devices with prepared
> resources. It's fine to me, as we can just save information to this
> binfo, and create devices later, but is this matches with what you
> think? :)

That is what I'm thinking more or less.   I'm trying to see this
implemented some way where iomapping is separate from creating
devices.  There are drivers in the kernel that have 3 parts: core,
platform, and pci.  The core has most of the functionality specific to
the driver where the platform and pci parts handle iomapping depending
on which bus the device is on.

Alan

>
> Thanks
> Hao
>
>>
>> > +
>> > +       ret = build_info_commit_dev(binfo);
>> > +       if (ret)
>> > +               goto free_binfo_exit;
>> > +
>> > +       /*
>> > +        * everything is okay, reset ->parent_dev to stop it being
>> > +        * freed by build_info_free()
>> > +        */
>> > +       binfo->parent_dev = NULL;
>> > +
>> > +free_binfo_exit:
>> > +       build_info_free(binfo);
>> > +       return ret;
>> > +}
>> > +
>> >  /* PCI Device ID */
>> >  #define PCIe_DEVICE_ID_PF_INT_5_X      0xBCBD
>> >  #define PCIe_DEVICE_ID_PF_INT_6_X      0xBCC0
>> > @@ -81,9 +898,18 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
>> >                 goto release_region_exit;
>> >         }
>> >
>> > -       /* TODO: create and add the platform device per feature list */
>> > +       ret = create_init_drvdata(pcidev);
>> > +       if (ret)
>> > +               goto release_region_exit;
>> > +
>> > +       ret = cci_pci_create_feature_devs(pcidev);
>> > +       if (ret)
>> > +               goto destroy_drvdata_exit;
>> > +
>> >         return 0;
>> >
>> > +destroy_drvdata_exit:
>> > +       destroy_drvdata(pcidev);
>> >  release_region_exit:
>> >         pci_release_regions(pcidev);
>> >  disable_error_report_exit:
>> > @@ -94,6 +920,8 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
>> >
>> >  static void cci_pci_remove(struct pci_dev *pcidev)
>> >  {
>> > +       remove_all_devs(pcidev);
>> > +       destroy_drvdata(pcidev);
>> >         pci_release_regions(pcidev);
>> >         pci_disable_pcie_error_reporting(pcidev);
>> >         pci_disable_device(pcidev);
>> > @@ -108,14 +936,23 @@ static void cci_pci_remove(struct pci_dev *pcidev)
>> >
>> >  static int __init ccidrv_init(void)
>> >  {
>> > +       int ret;
>> > +
>> >         pr_info("Intel(R) FPGA PCIe Driver: Version %s\n", DRV_VERSION);
>> >
>> > -       return pci_register_driver(&cci_pci_driver);
>> > +       fpga_ids_init();
>> > +
>> > +       ret = pci_register_driver(&cci_pci_driver);
>> > +       if (ret)
>> > +               fpga_ids_destroy();
>> > +
>> > +       return ret;
>> >  }
>> >
>> >  static void __exit ccidrv_exit(void)
>> >  {
>> >         pci_unregister_driver(&cci_pci_driver);
>> > +       fpga_ids_destroy();
>> >  }
>> >
>> >  module_init(ccidrv_init);
>> > --
>> > 1.8.3.1
>> >

Re: [PATCH v2 15/22] fpga: intel: add fpga manager platform driver for FME

[Wu Hao]

On Wed, Sep 27, 2017 at 01:54:58PM -0500, Alan Tull wrote:
> On Tue, Sep 26, 2017 at 8:18 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> 
> Hi Hao,
> 
> > On Mon, Sep 25, 2017 at 02:24:57PM -0700, Moritz Fischer wrote:
> >> Hi Hao,
> >
> > Hi Moritz
> >
> > Thanks for your review. :)
> >
> >>
> >> On Mon, Jun 26, 2017 at 09:52:11AM +0800, Wu Hao wrote:
> >> > This patch adds fpga manager driver for Intel FPGA Management
> >> > Engine(FME). It implements fpga_manager_ops for FPGA Partial
> >> > Reconfiguration function.
> >> >
> >> > Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > Signed-off-by: Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> >> > Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > ---
> >> >  .../ABI/testing/sysfs-platform-intel-fpga-fme-mgr  |   8 +
> >> >  drivers/fpga/Kconfig                               |   7 +
> >> >  drivers/fpga/Makefile                              |   1 +
> >> >  drivers/fpga/intel-feature-dev.h                   |  75 +++++
> >> >  drivers/fpga/intel-fpga-fme-mgr.c                  | 307 +++++++++++++++++++++
> >> >  5 files changed, 398 insertions(+)
> >> >  create mode 100644 Documentation/ABI/testing/sysfs-platform-intel-fpga-fme-mgr
> >> >  create mode 100644 drivers/fpga/intel-fpga-fme-mgr.c
> >> >
> >> > diff --git a/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme-mgr b/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme-mgr
> >> > new file mode 100644
> >> > index 0000000..40771fb
> >> > --- /dev/null
> >> > +++ b/Documentation/ABI/testing/sysfs-platform-intel-fpga-fme-mgr
> >> > @@ -0,0 +1,8 @@
> >> > +What:              /sys/bus/platform/devices/intel-fpga-fme-mgr.0/interface_id
> >> > +Date:              June 2017
> >> > +KernelVersion:  4.12
> >> > +Contact:   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > +Description:       Read-only. It returns interface id of partial reconfiguration
> >> > +           hardware. Userspace could use this information to check if
> >> > +           current hardware is compatible with given image before FPGA
> >> > +           programming.
> >> > diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> >> > index b91458f..6f2f623 100644
> >> > --- a/drivers/fpga/Kconfig
> >> > +++ b/drivers/fpga/Kconfig
> >> > @@ -151,6 +151,13 @@ config INTEL_FPGA_FME
> >> >       all FPGA platform level management features. There shall be 1
> >> >       FME per Intel FPGA.
> >> >
> >> > +config INTEL_FPGA_FME_MGR
> >> > +   tristate "Intel FPGA FME Manager Driver"
> >> > +   depends on INTEL_FPGA_FME
> >> > +   help
> >> > +     Say Y to enable FPGA Manager driver for Intel FPGA Management
> >> > +     Engine.
> >> > +
> >> >  endif
> >> >
> >> >  endif # FPGA
> >> > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> >> > index acda5ca..d1d588b 100644
> >> > --- a/drivers/fpga/Makefile
> >> > +++ b/drivers/fpga/Makefile
> >> > @@ -31,6 +31,7 @@ obj-$(CONFIG_OF_FPGA_REGION)              += of-fpga-region.o
> >> >  # Intel FPGA Support
> >> >  obj-$(CONFIG_INTEL_FPGA_PCI)               += intel-fpga-pci.o
> >> >  obj-$(CONFIG_INTEL_FPGA_FME)               += intel-fpga-fme.o
> >> > +obj-$(CONFIG_INTEL_FPGA_FME_MGR)   += intel-fpga-fme-mgr.o
> >> >
> >> >  intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
> >> >  intel-fpga-fme-objs := intel-fme-main.o intel-fme-pr.o
> >> > diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
> >> > index 3f97b75..f33923b 100644
> >> > --- a/drivers/fpga/intel-feature-dev.h
> >> > +++ b/drivers/fpga/intel-feature-dev.h
> >> > @@ -164,8 +164,83 @@ struct feature_fme_err {
> >> >  };
> >> >
> >> >  /* FME Partial Reconfiguration Sub Feature Register Set */
> >> > +/* FME PR Control Register */
> >> > +struct feature_fme_pr_ctl {
> >> > +   union {
> >> > +           u64 csr;
> >> > +           struct {
> >> > +                   u64 pr_reset:1;         /* Reset PR Engine */
> >> > +                   u64 rsvdz1:3;
> >> > +                   u64 pr_reset_ack:1;     /* Reset PR Engine Ack */
> >> > +                   u64 rsvdz2:3;
> >> > +                   u64 pr_regionid:2;      /* PR Region ID */
> >> > +                   u64 rsvdz3:2;
> >> > +                   u64 pr_start_req:1;     /* PR Start Request */
> >> > +                   u64 pr_push_complete:1; /* PR Data push complete */
> >> > +                   u64 pr_kind:1;          /* Load Customer or Intel GBS */
> >> > +                   u64 rsvdz4:17;
> >> > +                   u64 config_data:32;
> >> > +           };
> >> > +   };
> >> > +};
> >> > +
> >> > +/* FME PR Status Register */
> >> > +struct feature_fme_pr_status {
> >> > +   union {
> >> > +           u64 csr;
> >> > +           struct {
> >> > +                   u64 pr_credit:9;        /* Number of PR Credits */
> >> > +                   u64 rsvdz1:7;
> >> > +                   u64 pr_status:1;        /* PR Operation status */
> >> > +                   u64 rsvdz2:3;
> >> > +                   u64 pr_ctrlr_status:3;  /* Controller status */
> >> > +                   u64 rsvdz3:1;
> >> > +                   u64 pr_host_status:4;   /* PR Host status */
> >> > +                   u64 rsvdz4:36;
> >> > +           };
> >> > +   };
> >> > +};
> >> > +
> >> > +/* FME PR Data Register */
> >> > +struct feature_fme_pr_data {
> >> > +   union {
> >> > +           u64 csr;
> >> > +           struct {
> >> > +                   /* PR data from the raw-binary file */
> >> > +                   u64 pr_data_raw:32;
> >> > +                   u64 rsvd:32;
> >> > +           };
> >> > +   };
> >> > +};
> >> > +
> >> > +/* FME PR Error Register */
> >> > +struct feature_fme_pr_error {
> >> > +   union {
> >> > +           u64 csr;
> >> > +           struct {
> >> > +                   u64 operation_err:1;    /* Previous PR error detected */
> >> > +                   u64 crc_err:1;          /* CRC error detected */
> >> > +                   u64 incompatiable_bs:1; /* Incompatiable Bitstream */
> >> > +                   u64 protocol_err:1;     /* Data push protocol error */
> >> > +                   u64 fifo_overflow:1;    /* Data fifo overflow */
> >> > +                   u64 rsvdz1:1;
> >> > +                   u64 secure_load_err:1;  /* Secure PR load error */
> >> > +                   u64 rsvdz2:57;
> >> > +           };
> >> > +   };
> >> > +};
> >>
> >> From what I've seen [1], [2] people weren't too happy with using bitfields,
> >> any reason we can't use explicit masking and shifts? I don't know if
> >> this is still an issue.
> >
> > I think both should work fine in our case, and compiler should give same
> > result on both. We choose bitfields as it seems to be more readable, and
> > could save some code. Do you think if we can keep it? :)
> 
> I have the same concerns as Moritz here.  I have suggested using
> regmap before back in July.  Using bitmasks and shifts would be be
> preferable.

Sure, will switch to bitmasks and shifts in the next version. : )

Thanks
Hao

> 
> Alan
> 
> >
> >>
> >> > +
> >> >  struct feature_fme_pr {
> >> >     struct feature_header header;
> >> > +   struct feature_fme_pr_ctl control;
> >> > +   struct feature_fme_pr_status status;
> >> > +   struct feature_fme_pr_data data;
> >> > +   struct feature_fme_pr_error error;
> >> > +
> >> > +   u64 rsvd[16];
> >> > +
> >> > +   u64 intfc_id_l;         /* PR interface Id Low */
> >> > +   u64 intfc_id_h;         /* PR interface Id High */
> >> >  };
> >> >
> >> >  /* PORT Header Register Set */
> >> > diff --git a/drivers/fpga/intel-fpga-fme-mgr.c b/drivers/fpga/intel-fpga-fme-mgr.c
> >> > new file mode 100644
> >> > index 0000000..5f93dfb
> >> > --- /dev/null
> >> > +++ b/drivers/fpga/intel-fpga-fme-mgr.c
> >> > @@ -0,0 +1,307 @@
> >> > +/*
> >> > + * FPGA Manager Driver for Intel FPGA Management Engine (FME)
> >> > + *
> >> > + * Copyright (C) 2017 Intel Corporation, Inc.
> >> > + *
> >> > + * Authors:
> >> > + *   Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > + *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> >> > + *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > + *   Joseph Grecco <joe.grecco-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > + *   Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > + *   Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > + *   Ananda Ravuri <ananda.ravuri-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > + *   Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > + *   Henry Mitchel <henry.mitchel-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > + *
> >> > + * This work is licensed under the terms of the GNU GPL version 2. See
> >> > + * the COPYING file in the top-level directory.
> >> > + */
> >> > +
> >> > +#include <linux/module.h>
> >> > +#include <linux/iopoll.h>
> >> > +#include <linux/fpga/fpga-mgr.h>
> >> > +
> >> > +#include "intel-feature-dev.h"
> >> > +#include "intel-fme.h"
> >> > +
> >> > +#define PR_WAIT_TIMEOUT   8000000
> >> > +#define PR_HOST_STATUS_IDLE        0
> >> > +
> >> > +struct fme_mgr_priv {
> >> > +   void __iomem *ioaddr;
> >> > +};
> >> > +
> >> > +static ssize_t interface_id_show(struct device *dev,
> >> > +                            struct device_attribute *attr, char *buf)
> >> > +{
> >> > +   struct fpga_manager *mgr = dev_get_drvdata(dev);
> >> > +   struct fme_mgr_priv *priv = mgr->priv;
> >> > +   struct feature_fme_pr *fme_pr;
> >> > +   u64 intfc_id_l, intfc_id_h;
> >> > +
> >> > +   fme_pr = priv->ioaddr;
> >> > +
> >> > +   intfc_id_l = readq(&fme_pr->intfc_id_l);
> >> > +   intfc_id_h = readq(&fme_pr->intfc_id_h);
> >> > +
> >> > +   return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> >> > +                   (unsigned long long)intfc_id_h,
> >> > +                   (unsigned long long)intfc_id_l);
> >> > +}
> >> > +static DEVICE_ATTR_RO(interface_id);
> >> > +
> >> > +static const struct attribute *fme_mgr_attrs[] = {
> >> > +   &dev_attr_interface_id.attr,
> >> > +   NULL,
> >> > +};
> >> > +
> >> > +static void fme_mgr_pr_update_status(struct fpga_manager *mgr,
> >> > +                                struct feature_fme_pr_error err)
> >> > +{
> >> > +   mgr->status = 0;
> >> > +
> >> > +   if (err.operation_err)
> >> > +           mgr->status |= FPGA_MGR_STATUS_OPERATION_ERR;
> >>
> >> Can we just have a (err & FEATURE_FME_PR_ERROR) here instead ?
> >> > +   if (err.crc_err)
> >> > +           mgr->status |= FPGA_MGR_STATUS_CRC_ERR;
> >>
> >> Can we just have a (err & FEATURE_FME_CRC_ERROR) here instead ?
> >
> > As above, do you think if we can keep the origin code here?
> > Use "if (err.crc_err)" not "if (err & FEATURE_FME_CRC_ERROR)",
> > it makes the code shorter in most cases. :)
> >
> > Thanks
> > Hao
> >
> >> > +   if (err.incompatiable_bs)
> >> > +           mgr->status |= FPGA_MGR_STATUS_INCOMPATIBLE_BS_ERR;
> >> > +   if (err.protocol_err)
> >> > +           mgr->status |= FPGA_MGR_STATUS_IP_PROTOCOL_ERR;
> >> > +   if (err.fifo_overflow)
> >> > +           mgr->status |= FPGA_MGR_STATUS_FIFO_OVERFLOW_ERR;
> >> > +   if (err.secure_load_err)
> >> > +           mgr->status |= FPGA_MGR_STATUS_SECURE_LOAD_ERR;
> >> > +}
> >> > +
> >> > +static u64 fme_mgr_pr_err_handle(struct feature_fme_pr *fme_pr)
> >> > +{
> >> > +   struct feature_fme_pr_status fme_pr_status;
> >> > +   struct feature_fme_pr_error fme_pr_error;
> >> > +
> >> > +   fme_pr_status.csr = readq(&fme_pr->status);
> >> > +   if (!fme_pr_status.pr_status)
> >> > +           return 0;
> >> > +
> >> > +   fme_pr_error.csr = readq(&fme_pr->error);
> >> > +   writeq(fme_pr_error.csr, &fme_pr->error);
> >> > +
> >> > +   return fme_pr_error.csr;
> >> > +}
> >> > +
> >> > +static int fme_mgr_write_init(struct fpga_manager *mgr,
> >> > +                         struct fpga_image_info *info,
> >> > +                         const char *buf, size_t count)
> >> > +{
> >> > +   struct device *dev = &mgr->dev;
> >> > +   struct fme_mgr_priv *priv = mgr->priv;
> >> > +   struct feature_fme_pr *fme_pr = priv->ioaddr;
> >> > +   struct feature_fme_pr_ctl fme_pr_ctl;
> >> > +   struct feature_fme_pr_status fme_pr_status;
> >> > +   struct feature_fme_pr_error fme_pr_err;
> >> > +
> >> > +   if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
> >> > +           dev_err(dev, "only support partial reconfiguration.\n");
> >> > +           return -EINVAL;
> >> > +   }
> >> > +
> >> > +   dev_dbg(dev, "resetting PR before initiated PR\n");
> >> > +
> >> > +   fme_pr_ctl.csr = readq(&fme_pr->control);
> >> > +   fme_pr_ctl.pr_reset = 1;
> >> > +   writeq(fme_pr_ctl.csr, &fme_pr->control);
> >> > +
> >> > +   if (readq_poll_timeout(&fme_pr->control, fme_pr_ctl.csr,
> >> > +                          (fme_pr_ctl.pr_reset_ack == 1),
> >> > +                          1, PR_WAIT_TIMEOUT)) {
> >> > +           dev_err(dev, "maximum PR timeout\n");
> >> > +           return -ETIMEDOUT;
> >> > +   }
> >> > +
> >> > +   fme_pr_ctl.csr = readq(&fme_pr->control);
> >> > +   fme_pr_ctl.pr_reset = 0;
> >> > +   writeq(fme_pr_ctl.csr, &fme_pr->control);
> >> > +
> >> > +   dev_dbg(dev,
> >> > +           "waiting for PR resource in HW to be initialized and ready\n");
> >> > +
> >> > +   fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
> >> > +
> >> > +   if (readq_poll_timeout(&fme_pr->status, fme_pr_status.csr,
> >> > +                   (fme_pr_status.pr_host_status == PR_HOST_STATUS_IDLE),
> >> > +                   1, PR_WAIT_TIMEOUT)) {
> >> > +           dev_err(dev, "maximum PR timeout\n");
> >> > +           return -ETIMEDOUT;
> >> > +   }
> >> > +
> >> > +   dev_dbg(dev, "check and clear previous PR error\n");
> >> > +   fme_pr_err.csr = fme_mgr_pr_err_handle(fme_pr);
> >> > +   if (fme_pr_err.csr)
> >> > +           dev_dbg(dev, "previous PR error detected %llx\n",
> >> > +                        (unsigned long long)fme_pr_err.csr);
> >> > +
> >> > +   /* Clear all PR errors */
> >> > +   fme_pr_err.csr = 0;
> >> > +   fme_mgr_pr_update_status(mgr, fmCRCe_pr_err);
> >> > +
> >> > +   dev_dbg(dev, "set PR port ID\n");
> >> > +
> >> > +   fme_pr_ctl.csr = readq(&fme_pr->control);
> >> > +   fme_pr_ctl.pr_regionid = info->region_id;
> >> > +   writeq(fme_pr_ctl.csr, &fme_pr->control);
> >> > +
> >> > +   return 0;
> >> > +}
> >> > +
> >> > +static int fme_mgr_write(struct fpga_manager *mgr,
> >> > +                    const char *buf, size_t count)
> >> > +{
> >> > +   struct device *dev = &mgr->dev;
> >> > +   struct fme_mgr_priv *priv = mgr->priv;
> >> > +   struct feature_fme_pr *fme_pr = priv->ioaddr;
> >> > +   struct feature_fme_pr_ctl fme_pr_ctl;
> >> > +   struct feature_fme_pr_status fme_pr_status;
> >> > +   struct feature_fme_pr_data fme_pr_data;
> >> > +   int delay, pr_credit, i = 0;
> >> > +
> >> > +   dev_dbg(dev, "start request\n");
> >> > +
> >> > +   fme_pr_ctl.csr = readq(&fme_pr->control);
> >> > +   fme_pr_ctl.pr_start_req = 1;
> >> > +   writeq(fme_pr_ctl.csr, &fme_pr->control);
> >> > +
> >> > +   dev_dbg(dev, "pushing data from bitstream to HW\n");
> >> > +
> >> > +   fme_pr_status.csr = readq(&fme_pr->status);
> >> > +   pr_credit = fme_pr_status.pr_credit;
> >> > +
> >> > +   while (count > 0) {
> >> > +           delay = 0;
> >> > +           while (pr_credit <= 1) {
> >> > +                   if (delay++ > PR_WAIT_TIMEOUT) {
> >> > +                           dev_err(dev, "maximum try\n");
> >> > +                           return -ETIMEDOUT;
> >> > +                   }
> >> > +                   udelay(1);
> >> > +
> >> > +                   fme_pr_status.csr = readq(&fme_pr->status);
> >> > +                   pr_credit = fme_pr_status.pr_credit;
> >> > +           };
> >> > +
> >> > +           if (count >= 4) {
> >> > +                   fme_pr_data.rsvd = 0;
> >> > +                   fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
> >> > +                   writeq(fme_pr_data.csr, &fme_pr->data);
> >> > +                   count -= 4;
> >> > +                   pr_credit--;
> >> > +                   i++;
> >> > +           } else {
> >> > +                   WARN_ON(1);
> >> > +                   return -EINVAL;
> >> > +           }
> >> > +   }
> >> > +
> >> > +   return 0;
> >> > +}
> >> > +
> >> > +static int fme_mgr_write_complete(struct fpga_manager *mgr,
> >> > +                             struct fpga_image_info *info)
> >> > +{
> >> > +   struct device *dev = &mgr->dev;
> >> > +   struct fme_mgr_priv *priv = mgr->priv;
> >> > +   struct feature_fme_pr *fme_pr = priv->ioaddr;
> >> > +   struct feature_fme_pr_ctl fme_pr_ctl;
> >> > +   struct feature_fme_pr_error fme_pr_err;
> >> > +
> >> > +   fme_pr_ctl.csr = readq(&fme_pr->control);
> >> > +   fme_pr_ctl.pr_push_complete = 1;
> >> > +   writeq(fme_pr_ctl.csr, &fme_pr->control);
> >> > +
> >> > +   dev_dbg(dev, "green bitstream push complete\n");
> >> > +   dev_dbg(dev, "waiting for HW to release PR resource\n");
> >> > +
> >> > +   fme_pr_ctl.pr_start_req = 0;
> >> > +
> >> > +   if (readq_poll_timeout(&fme_pr->control, fme_pr_ctl.csr,
> >> > +                          (fme_pr_ctl.pr_start_req == 0),
> >> > +                          1, PR_WAIT_TIMEOUT)) {
> >> > +           dev_err(dev, "maximum try.\n");
> >> > +           return -ETIMEDOUT;
> >> > +   }
> >> > +
> >> > +   dev_dbg(dev, "PR operation complete, checking status\n");
> >> > +   fme_pr_err.csr = fme_mgr_pr_err_handle(fme_pr);
> >> > +   if (fme_pr_err.csr) {
> >> > +           fme_mgr_pr_update_status(mgr, fme_pr_err);
> >> > +           return -EIO;
> >> > +   }
> >> > +
> >> > +   dev_dbg(dev, "PR done successfully\n");
> >> > +   return 0;
> >> > +}
> >> > +
> >> > +static enum fpga_mgr_states fme_mgr_state(struct fpga_manager *mgr)
> >> > +{
> >> > +   return FPGA_MGR_STATE_UNKNOWN;
> >> > +}
> >> > +
> >> > +static const struct fpga_manager_ops fme_mgr_ops = {
> >> > +   .write_init = fme_mgr_write_init,
> >> > +   .write = fme_mgr_write,
> >> > +   .write_complete = fme_mgr_write_complete,
> >> > +   .state = fme_mgr_state,
> >> > +};
> >> > +
> >> > +static int fme_mgr_probe(struct platform_device *pdev)
> >> > +{
> >> > +   struct device *dev = &pdev->dev;
> >> > +   struct fme_mgr_priv *priv;
> >> > +   struct resource *res;
> >> > +   int ret;
> >> > +
> >> > +   priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
> >> > +   if (!priv)
> >> > +           return -ENOMEM;
> >> > +
> >> > +   res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> >> > +   priv->ioaddr = devm_ioremap(dev, res->start, resource_size(res));
> >> > +   if (IS_ERR(priv->ioaddr))
> >> > +           return PTR_ERR(priv->ioaddr);
> >> > +
> >> > +   ret = sysfs_create_files(&pdev->dev.kobj, fme_mgr_attrs);
> >> > +   if (ret)
> >> > +           return ret;
> >> > +
> >> > +   ret = fpga_mgr_register(dev, "Intel FPGA Manager", &fme_mgr_ops, priv);
> >> > +   if (ret) {
> >> > +           dev_err(dev, "unable to register FPGA manager\n");
> >> > +           sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> >> > +           return ret;
> >> > +   }
> >> > +
> >> > +   return 0;
> >> > +}
> >> > +
> >> > +static int fme_mgr_remove(struct platform_device *pdev)
> >> > +{
> >> > +   fpga_mgr_unregister(&pdev->dev);
> >> > +   sysfs_remove_files(&pdev->dev.kobj, fme_mgr_attrs);
> >> > +
> >> > +   return 0;
> >> > +}
> >> > +
> >> > +static struct platform_driver fme_mgr_driver = {
> >> > +   .driver = {
> >> > +           .name    = INTEL_FPGA_FME_MGR,
> >> > +   },
> >> > +   .probe   = fme_mgr_probe,
> >> > +   .remove  = fme_mgr_remove,
> >> > +};
> >> > +
> >> > +module_platform_driver(fme_mgr_driver);
> >> > +
> >> > +MODULE_DESCRIPTION("FPGA Manager for Intel FPGA Management Engine");
> >> > +MODULE_AUTHOR("Intel Corporation");
> >> > +MODULE_LICENSE("GPL v2");
> >> > +MODULE_ALIAS("platform:intel-fpga-fme-mgr");
> >> > --
> >> > 1.8.3.1
> >> >
> >>
> >> [1] http://yarchive.net/comp/linux/bitfields.html
> >> [2] https://lwn.net/Articles/478657/
> >>
> >> Thanks,
> >> Moritz
> >
> >

Re: [PATCH v2 07/22] fpga: intel: pcie: parse feature list and create platform device for features.

[Wu Hao]

On Wed, Sep 27, 2017 at 03:27:01PM -0500, Alan Tull wrote:
> On Fri, Sep 22, 2017 at 2:28 AM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> > On Wed, Sep 20, 2017 at 04:24:10PM -0500, Alan Tull wrote:
> 
> HI Hao,

Hi Alan

Thanks for your comments. :)

> 
> >>  a (wh., *()On Sun, Jun 25, 2017 at 8:52 PM, Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> >>
> >> Hi Hao,
> >>
> >> I'm done with some board bringup so I have time to look at your patchset again.
> >
> > Hi Alan
> >
> > Thanks for your time on the review. : )
> >
> >>
> >> Something I can't help but notice is that this patchset will be almost
> >> completely reusable for embedded FPGAs if the enumeration is separated
> >> from the pcie code.  After the devices are created, they are just mmio
> >> devices.  That makes this whole scheme available for embedded FPGAs.
> >>
> >> The division of things would be that the pcie code would read the
> >> headers and do ioremapping.  Then pass the headers to the enumeration
> >> code to create the devices.  With that division, another platform that
> >> is embedded could have its own code that read the headers and did the
> >> iomapping and then used the separate enumeration code.
> >>
> >> Going through intel-pcie.c, the truly pcie specific parts are already
> >> pretty well contained.  The pcie code is in probe,
> >> cci_pci_ioremap_bar, cci_pci_release_regions, and the 'parse_s*'
> >> functions that ultimately call cci_pci_ioremap_bar.  So it all comes
> >> down to the code that calls the 'parse_s*' functions.  That is, the
> >> code the reads the fme header and the port headers.  Currently, if I'm
> >> reading this right, this code ioremaps each region, reads its header,
> >> and creates its device before reading the next header.  So we just
> >> need to change the order a little and ioremap/read all the headers
> >> before starting to create devices.  That separates the pci ioremap
> >> from creating the devices.  So the probe function could ioremap and
> >> read in the fme header.  Then it would ioremap and read in each of the
> >> port headers.  After they are read in, then pass the all these headers
> >> to the separate enumeration code that will parse it and create the
> >> devices.  That part is not pcie specific, right?  With the enumeration
> >> code in a separate file (and assuming that the ioremap has already
> >> happened),  another platform that is embedded could have its probe do
> >> the iomapping and reading the headers, and then call the same
> >> enumeration code.
> >
> > So it's suggested that we have some kind of pre-enumeration code in the
> > intel-pcie.c to do ioremap and locate the address of device feature lists
> > for FME and Ports, and hand them to common enumeration code. Is my
> > understanding correct?
> 
> I was suggesting reading in the structures and saving that info to
> pass to the enumeration code.  Passing the ioremapped addresses would
> probably be just as fine.
> 
> Please keep in mind that I am just making suggestions about how I
> think this can be made to be platform independent.  We can work out
> the details, but my main point here is that this infrastructure is
> going to need to support embedded FPGAs.  As you suggest below, if we
> wait it will become more PCIe-centric and it will be harder later for
> this code base to support other parts.

Agree, understand your point. It would be great that this infrastructure
could support embedded FPGAs as well. So I would like have a try to split
this code, do some pre-enumeration in PCIe driver, and hand required
resources to some common enumeration code which creates the feature
devices. You're right, we should not wait without doing anything as we
already have a goal here. :)

> 
> >
> > I have considered this some time ago, this is doable for current code,
> > as the pre-enumeration code will be simple (only need to find FME header
> > and then read some register in FME header register set to locate the Port
> > header).
> 
> A lot of my intention in reviewing this is to make sure the code is
> broadly usable.  That way it will have  a future on other platforms,
> for example the Stratix10 SoC.
> 
> That is also why it has been important to use the FPGA manager API and
> expand it where needed.  (You made some good changes for v2 in this
> area, thanks!)   When this infrastructure runs on a SoC, the FME will
> be divided up, some of the functionality will appear as it currently
> does, but the PR part will use whatever PR hardware the SoC has.
> 
> > But if in the future, some pcie specific things are added to some
> > sub features of FME, then pre-enumeration code has to parse FME and all
> > its sub features to handle pcie specific things. We will have more
> > duplicate code on pre-enumeration in intel-pcie.c and common enumeration.
> > Another point is, it's hard to avoid pcie specific things in sub features,
> > as they may be designed to use PCIE specific function or resources.
> > e.g some sub features require the interrupts from PCIE device, like PCIE
> > MSI / MSIX. So keep enumeration code away from PCI things may be difficult.
> >
> >>
> >> Actually, there's a bit more to it than that but I think this is
> >> doable.  build_info_add_sub_feature() (which is called by
> >> create_feature_instance) saves off resource info for each sub device.
> >> Does this mean this code is ioremapping again (in
> >> intel-fpga-fme-mgr.c's probe)?
> >
> > Yes, you are correct. Is it fine to have the MMIO region mapped twice?
> > or we have to pass the mapped address to intel-fpga-fme-mgr directly?
> >
> >>
> >> A few more things below...
> >>
> >> > From: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> >> >
> >> > Device Feature List structure creates a link list of feature headers
> >> > within the MMIO space to provide an extensible way of adding features.
> >> >
> >> > The Intel FPGA PCIe driver walks through the feature headers to enumerate
> >> > feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
> >> > Function Unit (AFU), and their private sub features. For feature devices,
> >> > it creates the platform devices and linked the private sub features into
> >> > their platform data.
> >> >
> >> > Signed-off-by: Tim Whisonant <tim.whisonant-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > Signed-off-by: Enno Luebbers <enno.luebbers-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > Signed-off-by: Shiva Rao <shiva.rao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > Signed-off-by: Christopher Rauer <christopher.rauer-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > Signed-off-by: Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > Signed-off-by: Zhang Yi <yi.z.zhang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > Signed-off-by: Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> >> > Signed-off-by: Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > ---
> >> > v2: moved the code to drivers/fpga folder as suggested by Alan Tull.
> >> >     switched to GPLv2 license.
> >> >     fixed comments from Moritz Fischer.
> >> >     fixed kbuild warning, typos and clean up the code.
> >> > ---
> >> >  drivers/fpga/Makefile            |   2 +-
> >> >  drivers/fpga/intel-feature-dev.c | 130 ++++++
> >> >  drivers/fpga/intel-feature-dev.h | 341 ++++++++++++++++
> >> >  drivers/fpga/intel-pcie.c        | 841 ++++++++++++++++++++++++++++++++++++++-
> >> >  4 files changed, 1311 insertions(+), 3 deletions(-)
> >> >  create mode 100644 drivers/fpga/intel-feature-dev.c
> >> >  create mode 100644 drivers/fpga/intel-feature-dev.h
> >> >
> >> > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> >> > index 5613133..ad24b3d 100644
> >> > --- a/drivers/fpga/Makefile
> >> > +++ b/drivers/fpga/Makefile
> >> > @@ -31,4 +31,4 @@ obj-$(CONFIG_OF_FPGA_REGION)          += of-fpga-region.o
> >> >  # Intel FPGA Support
> >> >  obj-$(CONFIG_INTEL_FPGA_PCI)           += intel-fpga-pci.o
> >> >
> >> > -intel-fpga-pci-objs := intel-pcie.o
> >> > +intel-fpga-pci-objs := intel-pcie.o intel-feature-dev.o
> >> > diff --git a/drivers/fpga/intel-feature-dev.c b/drivers/fpga/intel-feature-dev.c
> >> > new file mode 100644
> >> > index 0000000..68f9cba
> >> > --- /dev/null
> >> > +++ b/drivers/fpga/intel-feature-dev.c
> >> > @@ -0,0 +1,130 @@
> >> > +/*
> >> > + * Intel FPGA Feature Device Driver
> >> > + *
> >> > + * Copyright (C) 2017 Intel Corporation, Inc.
> >> > + *
> >> > + * Authors:
> >> > + *   Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > + *   Zhang Yi <yi.z.zhang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > + *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > + *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> >> > + *
> >> > + * This work is licensed under the terms of the GNU GPL version 2. See
> >> > + * the COPYING file in the top-level directory.
> >> > + */
> >> > +
> >> > +#include "intel-feature-dev.h"
> >> > +
> >> > +void feature_platform_data_add(struct feature_platform_data *pdata,
> >> > +                              int index, const char *name,
> >> > +                              int resource_index, void __iomem *ioaddr)
> >> > +{
> >> > +       WARN_ON(index >= pdata->num);
> >> > +
> >> > +       pdata->features[index].name = name;
> >> > +       pdata->features[index].resource_index = resource_index;
> >> > +       pdata->features[index].ioaddr = ioaddr;
> >> > +}
> >> > +
> >> > +struct feature_platform_data *
> >> > +feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
> >> > +{
> >> > +       struct feature_platform_data *pdata;
> >> > +
> >> > +       pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
> >> > +       if (pdata) {
> >> > +               pdata->dev = dev;
> >> > +               pdata->num = num;
> >> > +               mutex_init(&pdata->lock);
> >> > +       }
> >> > +
> >> > +       return pdata;
> >> > +}
> >> > +
> >> > +int fme_feature_num(void)
> >> > +{
> >> > +       return FME_FEATURE_ID_MAX;
> >> > +}
> >> > +
> >> > +int port_feature_num(void)
> >> > +{
> >> > +       return PORT_FEATURE_ID_MAX;
> >> > +}
> >>
> >> Do these need to be functions?  If so, static.  But if you can just
> >> use the #define'd values that's one level of indirection we can get
> >> rid of.  And when I'm going through code where there's extra levels of
> >> indirection that don't do anything, it makes it harder to understand
> >> ;)
> >
> > Sure, will fix this.
> >
> >>
> >> > +
> >> > +int fpga_port_id(struct platform_device *pdev)
> >> > +{
> >> > +       struct feature_port_header *port_hdr;
> >> > +       struct feature_port_capability capability;
> >> > +
> >> > +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> >> > +                                              PORT_FEATURE_ID_HEADER);
> >> > +       WARN_ON(!port_hdr);
> >> > +
> >> > +       capability.csr = readq(&port_hdr->capability);
> >> > +       return capability.port_number;
> >> > +}
> >> > +EXPORT_SYMBOL_GPL(fpga_port_id);
> >>
> >> This is here because every feature is a port, right?
> >
> > No, this fpga_port_id function will be used in both afu/port driver and
> > also the pcie driver, so this function in added here.
> > The code in pcie driver which uses this function isn't submitted yet, it's
> > part of the SRIOV support. The pcie driver needs to manage which port could
> > be turned to VF.
> >
> >>
> >> > +
> >> > +/*
> >> > + * Enable Port by clear the port soft reset bit, which is set by default.
> >> > + * The User AFU is unable to respond to any MMIO access while in reset.
> >> > + * __fpga_port_enable function should only be used after __fpga_port_disable
> >> > + * function.
> >> > + */
> >> > +void __fpga_port_enable(struct platform_device *pdev)
> >> > +{
> >> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >> > +       struct feature_port_header *port_hdr;
> >> > +       struct feature_port_control control;
> >> > +
> >> > +       WARN_ON(!pdata->disable_count);
> >> > +
> >> > +       if (--pdata->disable_count != 0)
> >> > +               return;
> >> > +
> >> > +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> >> > +                                              PORT_FEATURE_ID_HEADER);
> >> > +       WARN_ON(!port_hdr);
> >> > +
> >> > +       control.csr = readq(&port_hdr->control);
> >> > +       control.port_sftrst = 0x0;
> >> > +       writeq(control.csr, &port_hdr->control);
> >> > +}
> >> > +EXPORT_SYMBOL_GPL(__fpga_port_enable);
> >> > +
> >> > +#define RST_POLL_INVL 10 /* us */
> >> > +#define RST_POLL_TIMEOUT 1000 /* us */
> >> > +
> >> > +int __fpga_port_disable(struct platform_device *pdev)
> >> > +{
> >> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >> > +       struct feature_port_header *port_hdr;
> >> > +       struct feature_port_control control;
> >> > +
> >> > +       if (pdata->disable_count++ != 0)
> >> > +               return 0;
> >> > +
> >> > +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> >> > +                                              PORT_FEATURE_ID_HEADER);
> >> > +       WARN_ON(!port_hdr);
> >> > +
> >> > +       /* Set port soft reset */
> >> > +       control.csr = readq(&port_hdr->control);
> >> > +       control.port_sftrst = 0x1;
> >> > +       writeq(control.csr, &port_hdr->control);
> >> > +
> >> > +       /*
> >> > +        * HW sets ack bit to 1 when all outstanding requests have been drained
> >> > +        * on this port and minimum soft reset pulse width has elapsed.
> >> > +        * Driver polls port_soft_reset_ack to determine if reset done by HW.
> >> > +        */
> >> > +       if (readq_poll_timeout(&port_hdr->control, control.csr,
> >> > +                              (control.port_sftrst_ack == 1),
> >> > +                              RST_POLL_INVL, RST_POLL_TIMEOUT)) {
> >> > +               dev_err(&pdev->dev, "timeout, fail to reset device\n");
> >> > +               return -ETIMEDOUT;
> >> > +       }
> >> > +
> >> > +       return 0;
> >> > +}
> >> > +EXPORT_SYMBOL_GPL(__fpga_port_disable);
> >> > diff --git a/drivers/fpga/intel-feature-dev.h b/drivers/fpga/intel-feature-dev.h
> >> > new file mode 100644
> >> > index 0000000..f67784a
> >> > --- /dev/null
> >> > +++ b/drivers/fpga/intel-feature-dev.h
> >> > @@ -0,0 +1,341 @@
> >> > +/*
> >> > + * Intel FPGA Feature Device Driver Header File
> >> > + *
> >> > + * Copyright (C) 2017 Intel Corporation, Inc.
> >> > + *
> >> > + * Authors:
> >> > + *   Kang Luwei <luwei.kang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > + *   Zhang Yi <yi.z.zhang-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > + *   Wu Hao <hao.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> > + *   Xiao Guangrong <guangrong.xiao-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> >> > + *
> >> > + * This work is licensed under the terms of the GNU GPL version 2. See
> >> > + * the COPYING file in the top-level directory.
> >> > + */
> >> > +
> >> > +#ifndef __INTEL_FPGA_FEATURE_H
> >> > +#define __INTEL_FPGA_FEATURE_H
> >> > +
> >> > +#include <linux/fs.h>
> >> > +#include <linux/pci.h>
> >> > +#include <linux/uuid.h>
> >> > +#include <linux/delay.h>
> >> > +#include <linux/iopoll.h>
> >> > +#include <linux/platform_device.h>
> >> > +
> >> > +#ifndef readq
> >> > +static inline u64 readq(void __iomem *addr)
> >> > +{
> >> > +       return readl(addr) + ((u64)readl(addr + 4) << 32);
> >> > +}
> >> > +#endif
> >> > +
> >> > +#ifndef writeq
> >> > +static inline void writeq(u64 val, void __iomem *addr)
> >> > +{
> >> > +       writel((u32) (val), addr);
> >> > +       writel((u32) (val >> 32), (addr + 4));
> >> > +}
> >> > +#endif
> >> > +
> >> > +/* maximum supported number of ports */
> >> > +#define MAX_FPGA_PORT_NUM 4
> >> > +/* plus one for fme device */
> >> > +#define MAX_FEATURE_DEV_NUM    (MAX_FPGA_PORT_NUM + 1)
> >> > +
> >> > +#define FME_FEATURE_HEADER          "fme_hdr"
> >> > +#define FME_FEATURE_THERMAL_MGMT    "fme_thermal"
> >> > +#define FME_FEATURE_POWER_MGMT      "fme_power"
> >> > +#define FME_FEATURE_GLOBAL_PERF     "fme_gperf"
> >> > +#define FME_FEATURE_GLOBAL_ERR      "fme_error"
> >> > +#define FME_FEATURE_PR_MGMT         "fme_pr"
> >> > +
> >> > +#define PORT_FEATURE_HEADER         "port_hdr"
> >> > +#define PORT_FEATURE_UAFU           "port_uafu"
> >> > +#define PORT_FEATURE_ERR            "port_err"
> >> > +#define PORT_FEATURE_UMSG           "port_umsg"
> >> > +#define PORT_FEATURE_PR             "port_pr"
> >> > +#define PORT_FEATURE_STP            "port_stp"
> >> > +
> >> > +/* All headers and structures must be byte-packed to match the spec. */
> >> > +#pragma pack(1)
> >> > +
> >> > +/* common header for all features */
> >> > +struct feature_header {
> >> > +       union {
> >> > +               u64 csr;
> >> > +               struct {
> >> > +                       u64 id:12;
> >> > +                       u64 revision:4;
> >> > +                       u64 next_header_offset:24; /* offset to next header */
> >> > +                       u64 rsvdz:20;
> >> > +                       u64 type:4;                /* feature type */
> >> > +#define FEATURE_TYPE_AFU               0x1
> >> > +#define FEATURE_TYPE_PRIVATE           0x3
> >> > +               };
> >> > +       };
> >> > +};
> >> > +
> >> > +/* common header for non-private features */
> >> > +struct feature_afu_header {
> >> > +       uuid_le guid;
> >> > +       union {
> >> > +               u64 csr;
> >> > +               struct {
> >> > +                       u64 next_afu:24;        /* pointer to next afu header */
> >> > +                       u64 rsvdz:40;
> >> > +               };
> >> > +       };
> >> > +};
> >> > +
> >> > +/* FME Header Register Set */
> >> > +/* FME Capability Register */
> >> > +struct feature_fme_capability {
> >> > +       union {
> >> > +               u64 csr;
> >> > +               struct {
> >> > +                       u64 fabric_verid:8;     /* Fabric version ID */
> >> > +                       u64 socket_id:1;        /* Socket id */
> >> > +                       u64 rsvdz1:3;
> >> > +                       u64 pcie0_link_avl:1;   /* PCIe0 link availability */
> >> > +                       u64 pcie1_link_avl:1;   /* PCIe1 link availability */
> >> > +                       u64 coherent_link_avl:1;/* Coherent link availability */
> >> > +                       u64 rsvdz2:1;
> >> > +                       u64 iommu_support:1;    /* IOMMU or VT-d supported */
> >> > +                       u64 num_ports:3;        /* Num of ports implemented */
> >> > +                       u64 rsvdz3:4;
> >> > +                       u64 addr_width_bits:6;  /* Address width supported */
> >> > +                       u64 rsvdz4:2;
> >> > +                       u64 cache_size:12;      /* Cache size in kb */
> >> > +                       u64 cache_assoc:4;      /* Cache Associativity */
> >> > +                       u64 rsvdz5:15;
> >> > +                       u64 lock_bit:1;         /* Latched lock bit by BIOS */
> >> > +               };
> >> > +       };
> >> > +};
> >> > +
> >> > +/* FME Port Offset Register */
> >> > +struct feature_fme_port {
> >> > +       union {
> >> > +               u64 csr;
> >> > +               struct {
> >> > +                       u64 port_offset:24;     /* Offset to port header */
> >> > +                       u64 rsvdz1:8;
> >> > +                       u64 port_bar:3;         /* Bar id */
> >> > +                       u64 rsvdz2:20;
> >> > +                       u64 afu_access_ctrl:1;  /* AFU access type: PF/VF */
> >> > +                       u64 rsvdz3:4;
> >> > +                       u64 port_implemented:1; /* Port implemented or not */
> >> > +                       u64 rsvdz4:3;
> >> > +               };
> >> > +       };
> >> > +};
> >> > +
> >> > +struct feature_fme_header {
> >> > +       struct feature_header header;
> >> > +       struct feature_afu_header afu_header;
> >> > +       u64 rsvd[2];
> >> > +       struct feature_fme_capability capability;
> >> > +       struct feature_fme_port port[MAX_FPGA_PORT_NUM];
> >> > +};
> >> > +
> >> > +/* FME Thermal Sub Feature Register Set */
> >> > +struct feature_fme_thermal {
> >> > +       struct feature_header header;
> >> > +};
> >> > +
> >> > +/* FME Power Sub Feature Register Set */
> >> > +struct feature_fme_power {
> >> > +       struct feature_header header;
> >> > +};
> >> > +
> >> > +/* FME Global Performance Sub Feature Register Set */
> >> > +struct feature_fme_gperf {
> >> > +       struct feature_header header;
> >> > +};
> >> > +
> >> > +/* FME Error Sub Feature Register Set */
> >> > +struct feature_fme_err {
> >> > +       struct feature_header header;
> >> > +};
> >> > +
> >> > +/* FME Partial Reconfiguration Sub Feature Register Set */
> >> > +struct feature_fme_pr {
> >> > +       struct feature_header header;
> >> > +};
> >> > +
> >> > +/* PORT Header Register Set */
> >> > +/* Port Capability Register */
> >> > +struct feature_port_capability {
> >> > +       union {
> >> > +               u64 csr;
> >> > +               struct {
> >> > +                       u64 port_number:2;      /* Port Number 0-3 */
> >> > +                       u64 rsvdz1:6;
> >> > +                       u64 mmio_size:16;       /* User MMIO size in KB */
> >> > +                       u64 rsvdz2:8;
> >> > +                       u64 sp_intr_num:4;      /* Supported interrupts num */
> >> > +                       u64 rsvdz3:28;
> >> > +               };
> >> > +       };
> >> > +};
> >> > +
> >> > +/* Port Control Register */
> >> > +struct feature_port_control {
> >> > +       union {
> >> > +               u64 csr;
> >> > +               struct {
> >> > +                       u64 port_sftrst:1;      /* Port Soft Reset */
> >> > +                       u64 rsvdz1:1;
> >> > +                       u64 latency_tolerance:1;/* '1' >= 40us, '0' < 40us */
> >> > +                       u64 rsvdz2:1;
> >> > +                       u64 port_sftrst_ack:1;  /* HW ACK for Soft Reset */
> >> > +                       u64 rsvdz3:59;
> >> > +               };
> >> > +       };
> >> > +};
> >> > +
> >> > +struct feature_port_header {
> >> > +       struct feature_header header;
> >> > +       struct feature_afu_header afu_header;
> >> > +       u64 rsvd[2];
> >> > +       struct feature_port_capability capability;
> >> > +       struct feature_port_control control;
> >> > +};
> >> > +
> >> > +/* PORT Error Sub Feature Register Set */
> >> > +struct feature_port_error {
> >> > +       struct feature_header header;
> >> > +};
> >> > +
> >> > +/* PORT Unordered Message Sub Feature Register Set */
> >> > +struct feature_port_umsg {
> >> > +       struct feature_header header;
> >> > +};
> >> > +
> >> > +/* PORT SignalTap Sub Feature Register Set */
> >> > +struct feature_port_stp {
> >> > +       struct feature_header header;
> >> > +};
> >> > +
> >> > +#pragma pack()
> >> > +
> >> > +struct feature {
> >> > +       const char *name;
> >> > +       int resource_index;
> >> > +       void __iomem *ioaddr;
> >> > +};
> >> > +
> >> > +struct feature_platform_data {
> >> > +       /* list the feature dev to cci_drvdata->port_dev_list. */
> >> > +       struct list_head node;
> >> > +       struct mutex lock;
> >> > +       struct platform_device *dev;
> >> > +       unsigned int disable_count;     /* count for port disable */
> >> > +
> >> > +       int num;                        /* number of features */
> >> > +       struct feature features[0];
> >> > +};
> >> > +
> >> > +enum fme_feature_id {
> >> > +       FME_FEATURE_ID_HEADER = 0x0,
> >> > +       FME_FEATURE_ID_THERMAL_MGMT = 0x1,
> >> > +       FME_FEATURE_ID_POWER_MGMT = 0x2,
> >> > +       FME_FEATURE_ID_GLOBAL_PERF = 0x3,
> >> > +       FME_FEATURE_ID_GLOBAL_ERR = 0x4,
> >> > +       FME_FEATURE_ID_PR_MGMT = 0x5,
> >> > +       FME_FEATURE_ID_MAX = 0x6,
> >> > +};
> >> > +
> >> > +enum port_feature_id {
> >> > +       PORT_FEATURE_ID_HEADER = 0x0,
> >> > +       PORT_FEATURE_ID_ERROR = 0x1,
> >> > +       PORT_FEATURE_ID_UMSG = 0x2,
> >> > +       PORT_FEATURE_ID_PR = 0x3,
> >> > +       PORT_FEATURE_ID_STP = 0x4,
> >> > +       PORT_FEATURE_ID_UAFU = 0x5,
> >> > +       PORT_FEATURE_ID_MAX = 0x6,
> >> > +};
> >> > +
> >> > +int fme_feature_num(void);
> >> > +int port_feature_num(void);
> >> > +
> >> > +#define FPGA_FEATURE_DEV_FME           "intel-fpga-fme"
> >> > +#define FPGA_FEATURE_DEV_PORT          "intel-fpga-port"
> >> > +
> >> > +void feature_platform_data_add(struct feature_platform_data *pdata,
> >> > +                              int index, const char *name,
> >> > +                              int resource_index, void __iomem *ioaddr);
> >> > +
> >> > +static inline int feature_platform_data_size(const int num)
> >> > +{
> >> > +       return sizeof(struct feature_platform_data) +
> >> > +               num * sizeof(struct feature);
> >> > +}
> >> > +
> >> > +struct feature_platform_data *
> >> > +feature_platform_data_alloc_and_init(struct platform_device *dev, int num);
> >> > +
> >> > +int fpga_port_id(struct platform_device *pdev);
> >> > +
> >> > +static inline int fpga_port_check_id(struct platform_device *pdev,
> >> > +                                    void *pport_id)
> >> > +{
> >> > +       return fpga_port_id(pdev) == *(int *)pport_id;
> >> > +}
> >> > +
> >> > +void __fpga_port_enable(struct platform_device *pdev);
> >> > +int __fpga_port_disable(struct platform_device *pdev);
> >> > +
> >> > +static inline void fpga_port_enable(struct platform_device *pdev)
> >> > +{
> >> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >> > +
> >> > +       mutex_lock(&pdata->lock);
> >> > +       __fpga_port_enable(pdev);
> >> > +       mutex_unlock(&pdata->lock);
> >> > +}
> >> > +
> >> > +static inline int fpga_port_disable(struct platform_device *pdev)
> >> > +{
> >> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >> > +       int ret;
> >> > +
> >> > +       mutex_lock(&pdata->lock);
> >> > +       ret = __fpga_port_disable(pdev);
> >> > +       mutex_unlock(&pdata->lock);
> >> > +
> >> > +       return ret;
> >> > +}
> >> > +
> >> > +static inline int __fpga_port_reset(struct platform_device *pdev)
> >> > +{
> >> > +       int ret;
> >> > +
> >> > +       ret = __fpga_port_disable(pdev);
> >> > +       if (ret)
> >> > +               return ret;
> >> > +
> >> > +       __fpga_port_enable(pdev);
> >> > +       return 0;
> >> > +}
> >> > +
> >> > +static inline int fpga_port_reset(struct platform_device *pdev)
> >> > +{
> >> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >> > +       int ret;
> >> > +
> >> > +       mutex_lock(&pdata->lock);
> >> > +       ret = __fpga_port_reset(pdev);
> >> > +       mutex_unlock(&pdata->lock);
> >> > +       return ret;
> >> > +}
> >> > +
> >> > +static inline void __iomem *
> >> > +get_feature_ioaddr_by_index(struct device *dev, int index)
> >> > +{
> >> > +       struct feature_platform_data *pdata = dev_get_platdata(dev);
> >> > +
> >> > +       return pdata->features[index].ioaddr;
> >> > +}
> >> > +#endif
> >> > diff --git a/drivers/fpga/intel-pcie.c b/drivers/fpga/intel-pcie.c
> >> > index f697de4..70b8284 100644
> >> > --- a/drivers/fpga/intel-pcie.c
> >> > +++ b/drivers/fpga/intel-pcie.c
> >> > @@ -23,10 +23,827 @@
> >> >  #include <linux/stddef.h>
> >> >  #include <linux/errno.h>
> >> >  #include <linux/aer.h>
> >> > +#include <linux/fpga/fpga-dev.h>
> >> > +
> >> > +#include "intel-feature-dev.h"
> >> >
> >> >  #define DRV_VERSION    "0.8"
> >> >  #define DRV_NAME       "intel-fpga-pci"
> >> >
> >> > +#define INTEL_FPGA_DEV "intel-fpga-dev"
> >> > +
> >> > +static DEFINE_MUTEX(fpga_id_mutex);
> >> > +
> >> > +enum fpga_id_type {
> >> > +       FME_ID,         /* fme id allocation and mapping */
> >> > +       PORT_ID,        /* port id allocation and mapping */
> >> > +       FPGA_ID_MAX,
> >> > +};
> >> > +
> >> > +/* it is protected by fpga_id_mutex */
> >> > +static struct idr fpga_ids[FPGA_ID_MAX];
> >> > +
> >> > +struct cci_drvdata {
> >> > +       struct device *fme_dev;
> >> > +
> >> > +       struct mutex lock;
> >> > +       struct list_head port_dev_list;
> >> > +
> >> > +       struct list_head regions; /* global list of pci bar mapping region */
> >> > +};
> >> > +
> >> > +/* pci bar mapping info */
> >> > +struct cci_pci_region {
> >> > +       int bar;
> >> > +       void __iomem *ioaddr;   /* pointer to mapped bar region */
> >> > +       struct list_head node;
> >> > +};
> >> > +
> >> > +static void fpga_ids_init(void)
> >> > +{
> >> > +       int i;
> >> > +
> >> > +       for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> >> > +               idr_init(fpga_ids + i);
> >> > +}
> >> > +
> >> > +static void fpga_ids_destroy(void)
> >> > +{
> >> > +       int i;
> >> > +
> >> > +       for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> >> > +               idr_destroy(fpga_ids + i);
> >> > +}
> >> > +
> >> > +static int alloc_fpga_id(enum fpga_id_type type, struct device *dev)
> >> > +{
> >> > +       int id;
> >> > +
> >> > +       WARN_ON(type >= FPGA_ID_MAX);
> >> > +       mutex_lock(&fpga_id_mutex);
> >> > +       id = idr_alloc(fpga_ids + type, dev, 0, 0, GFP_KERNEL);
> >> > +       mutex_unlock(&fpga_id_mutex);
> >> > +       return id;
> >> > +}
> >> > +
> >> > +static void free_fpga_id(enum fpga_id_type type, int id)
> >> > +{
> >> > +       WARN_ON(type >= FPGA_ID_MAX);
> >> > +       mutex_lock(&fpga_id_mutex);
> >> > +       idr_remove(fpga_ids + type, id);
> >> > +       mutex_unlock(&fpga_id_mutex);
> >> > +}
> >> > +
> >> > +static void cci_pci_add_port_dev(struct pci_dev *pdev,
> >> > +                                struct platform_device *port_dev)
> >> > +{
> >> > +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> >> > +       struct feature_platform_data *pdata = dev_get_platdata(&port_dev->dev);
> >> > +
> >> > +       mutex_lock(&drvdata->lock);
> >> > +       list_add(&pdata->node, &drvdata->port_dev_list);
> >> > +       get_device(&pdata->dev->dev);
> >> > +       mutex_unlock(&drvdata->lock);
> >> > +}
> >> > +
> >> > +static void cci_pci_remove_port_devs(struct pci_dev *pdev)
> >> > +{
> >> > +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> >> > +       struct feature_platform_data *pdata, *ptmp;
> >> > +
> >> > +       mutex_lock(&drvdata->lock);
> >> > +       list_for_each_entry_safe(pdata, ptmp, &drvdata->port_dev_list, node) {
> >> > +               struct platform_device *port_dev = pdata->dev;
> >> > +
> >> > +               /* the port should be unregistered first. */
> >> > +               WARN_ON(device_is_registered(&port_dev->dev));
> >> > +               list_del(&pdata->node);
> >> > +               free_fpga_id(PORT_ID, port_dev->id);
> >> > +               put_device(&port_dev->dev);
> >> > +       }
> >> > +       mutex_unlock(&drvdata->lock);
> >> > +}
> >> > +
> >> > +/* info collection during feature dev build. */
> >> > +struct build_feature_devs_info {
> >> > +       struct pci_dev *pdev;
> >> > +
> >> > +       /*
> >> > +        * PCI BAR mapping info. Parsing feature list starts from
> >> > +        * BAR 0 and switch to different BARs to parse Port
> >> > +        */
> >> > +       void __iomem *ioaddr;
> >> > +       void __iomem *ioend;
> >> > +       int current_bar;
> >> > +
> >> > +       /* points to FME header where the port offset is figured out. */
> >> > +       void __iomem *pfme_hdr;
> >> > +
> >> > +       /* the container device for all feature devices */
> >> > +       struct fpga_dev *parent_dev;
> >> > +
> >> > +       /* current feature device */
> >> > +       struct platform_device *feature_dev;
> >> > +};
> >> > +
> >> > +static void cci_pci_release_regions(struct pci_dev *pdev)
> >> > +{
> >> > +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> >> > +       struct cci_pci_region *tmp, *region;
> >> > +
> >> > +       list_for_each_entry_safe(region, tmp, &drvdata->regions, node) {
> >> > +               list_del(&region->node);
> >> > +               if (region->ioaddr)
> >> > +                       pci_iounmap(pdev, region->ioaddr);
> >> > +               devm_kfree(&pdev->dev, region);
> >> > +       }
> >> > +}
> >> > +
> >> > +static void __iomem *cci_pci_ioremap_bar(struct pci_dev *pdev, int bar)
> >> > +{
> >> > +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> >> > +       struct cci_pci_region *region;
> >> > +
> >> > +       list_for_each_entry(region, &drvdata->regions, node)
> >> > +               if (region->bar == bar) {
> >> > +                       dev_dbg(&pdev->dev, "BAR %d region exists\n", bar);
> >> > +                       return region->ioaddr;
> >> > +               }
> >> > +
> >> > +       region = devm_kzalloc(&pdev->dev, sizeof(*region), GFP_KERNEL);
> >> > +       if (!region)
> >> > +               return NULL;
> >> > +
> >> > +       region->bar = bar;
> >> > +       region->ioaddr = pci_ioremap_bar(pdev, bar);
> >> > +       if (!region->ioaddr) {
> >> > +               dev_err(&pdev->dev, "can't ioremap memory from BAR %d.\n", bar);
> >> > +               devm_kfree(&pdev->dev, region);
> >> > +               return NULL;
> >> > +       }
> >> > +
> >> > +       list_add(&region->node, &drvdata->regions);
> >> > +       return region->ioaddr;
> >> > +}
> >> > +
> >> > +static int parse_start_from(struct build_feature_devs_info *binfo, int bar)
> >> > +{
> >> > +       binfo->ioaddr = cci_pci_ioremap_bar(binfo->pdev, bar);
> >> > +       if (!binfo->ioaddr)
> >> > +               return -ENOMEM;
> >> > +
> >> > +       binfo->current_bar = bar;
> >> > +       binfo->ioend = binfo->ioaddr + pci_resource_len(binfo->pdev, bar);
> >> > +       return 0;
> >> > +}
> >> > +
> >> > +static int parse_start(struct build_feature_devs_info *binfo)
> >> > +{
> >> > +       /* fpga feature list starts from BAR 0 */
> >> > +       return parse_start_from(binfo, 0);
> >> > +}
> >> > +
> >> > +/* switch the memory mapping to BAR# @bar */
> >> > +static int parse_switch_to(struct build_feature_devs_info *binfo, int bar)
> >> > +{
> >> > +       return parse_start_from(binfo, bar);
> >> > +}
> >>
> >> parse_switch_to() and parse_start() are both one line wrappers for
> >> parse_start_from() and are only called once.  :-) Please just use
> >> parse_start_from and get rid of parse_start and parse_switch_to.  I
> >> don't think this code is all that super complicated.  But as I'm
> >> grepping through this code, I'm trying to see all the places that do
> >> the same thing; if I have to look for multiple function names, it
> >> takes longer to get what is going on here.
> >>
> >> Actually, cci_pci_ioremap_bar is only called in one place -
> >> parse_start_from - which really just adds two lines of code.
> >> parse_start_from could go away.  Just a suggestion.  The reason this
> >> comes up is that cci_pci_ioremap_bar is the more descriptive function
> >> name.  The parse_start/switch function names hide the biggest thing
> >> they do, which is ioremapping.  They don't actually do any parsing ;)
> >
> > I think I need to explain a little more on the enumeration steps in this
> > cci_pci_create_feature_devs function, and we use these names for these
> > functions.
> >
> > 1) parse_start -> that initializes the enumeration from PCI Bar0.
> >    In PF case, the first byte of BAR0, will be FME header.
> >    In VF case, the first byte of BAR0, will be Port header.
> >    (FME module is always in PF, no FME module in VF).
> >
> > 2) parse_feature_list -> parse one device feature list
> >
> >    This function parses a feature list, it could be FME device feature
> >    list (PF) or Port device feature list (VF).
> >    FME device feature list and Port device feature list are not linked
> >    together, as they are in differnet BARs.
> >
> > 3) parse_port_from_fme -> parse Port device feature lists if FME presents.
> >
> >    If FME is detected (PF), read FME registers to know the location (BAR n
> >    + offset) of each port's header (port device feature list).
> >    call parse_switch_to(BAR n) and call parse_feature_list to parse the
> >    port device feature list.
> >
> >    If FME is not detected (VF), just do nothing.
> >
> > Do you think if we could keep these functions?
> 
> I'm OK with parse_feature_list and parse_port_from_fme, etc.  I
> intended to just comment about parse_start, parse_start_from, and
> parse_switch_to.  I was suggesting to get rid of parse_start and
> parse_switch_to and just use parse_start_from instead.

Sure, will fix this.

> 
> > We use these names to
> > help people to understand the enumeration process better, but looks like
> > I should add more descriptions/comments here to avoid misunderstanding. :)
> >
> >>
> >> > +
> >> > +static struct build_feature_devs_info *
> >> > +build_info_alloc_and_init(struct pci_dev *pdev)
> >> > +{
> >> > +       struct build_feature_devs_info *binfo;
> >> > +
> >> > +       binfo = devm_kzalloc(&pdev->dev, sizeof(*binfo), GFP_KERNEL);
> >> > +       if (binfo)
> >> > +               binfo->pdev = pdev;
> >>
> >> build_info_alloc_and_init() is only doing a devm_kzalloc and is only
> >> called once.  The code would be more readable if you just had the
> >> devm_kzalloc in the function that called this
> >> (cci_pci_create_feature_devs).
> >
> > Sure.
> >
> >>
> >> > +
> >> > +       return binfo;
> >> > +}
> >> > +
> >> > +static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
> >> > +{
> >> > +       if (!strcmp(pdev->name, FPGA_FEATURE_DEV_FME))
> >> > +               return FME_ID;
> >> > +
> >> > +       if (!strcmp(pdev->name, FPGA_FEATURE_DEV_PORT))
> >> > +               return PORT_ID;
> >> > +
> >> > +       WARN_ON(1);
> >> > +       return FPGA_ID_MAX;
> >> > +}
> >> > +
> >> > +/*
> >> > + * register current feature device, it is called when we need to switch to
> >> > + * another feature parsing or we have parsed all features
> >> > + */
> >> > +static int build_info_commit_dev(struct build_feature_devs_info *binfo)
> >> > +{
> >> > +       int ret;
> >> > +
> >> > +       if (!binfo->feature_dev)
> >> > +               return 0;
> >> > +
> >> > +       ret = platform_device_add(binfo->feature_dev);
> >> > +       if (!ret) {
> >> > +               struct cci_drvdata *drvdata;
> >> > +
> >> > +               drvdata = dev_get_drvdata(&binfo->pdev->dev);
> >> > +               if (feature_dev_id_type(binfo->feature_dev) == PORT_ID)
> >> > +                       cci_pci_add_port_dev(binfo->pdev, binfo->feature_dev);
> >> > +               else
> >> > +                       drvdata->fme_dev = get_device(&binfo->feature_dev->dev);
> >> > +
> >> > +               /*
> >> > +                * reset it to avoid build_info_free() freeing their resource.
> >> > +                *
> >> > +                * The resource of successfully registered feature devices
> >> > +                * will be freed by platform_device_unregister(). See the
> >> > +                * comments in build_info_create_dev().
> >> > +                */
> >> > +               binfo->feature_dev = NULL;
> >> > +       }
> >> > +
> >> > +       return ret;
> >> > +}
> >> > +
> >> > +static int
> >> > +build_info_create_dev(struct build_feature_devs_info *binfo,
> >> > +                     enum fpga_id_type type, int feature_nr, const char *name)
> >> > +{
> >> > +       struct platform_device *fdev;
> >> > +       struct resource *res;
> >> > +       struct feature_platform_data *pdata;
> >> > +       int ret;
> >> > +
> >> > +       /* we will create a new device, commit current device first */
> >> > +       ret = build_info_commit_dev(binfo);
> >> > +       if (ret)
> >> > +               return ret;
> >> > +
> >> > +       /*
> >> > +        * we use -ENODEV as the initialization indicator which indicates
> >> > +        * whether the id need to be reclaimed
> >> > +        */
> >> > +       fdev = binfo->feature_dev = platform_device_alloc(name, -ENODEV);
> >> > +       if (!fdev)
> >> > +               return -ENOMEM;
> >> > +
> >> > +       fdev->id = alloc_fpga_id(type, &fdev->dev);
> >> > +       if (fdev->id < 0)
> >> > +               return fdev->id;
> >> > +
> >> > +       fdev->dev.parent = &binfo->parent_dev->dev;
> >> > +
> >> > +       /*
> >> > +        * we do not need to care for the memory which is associated with
> >> > +        * the platform device. After calling platform_device_unregister(),
> >> > +        * it will be automatically freed by device's release() callback,
> >> > +        * platform_device_release().
> >> > +        */
> >> > +       pdata = feature_platform_data_alloc_and_init(fdev, feature_nr);
> >> > +       if (!pdata)
> >> > +               return -ENOMEM;
> >> > +
> >> > +       /*
> >> > +        * the count should be initialized to 0 to make sure
> >> > +        *__fpga_port_enable() following __fpga_port_disable()
> >> > +        * works properly for port device.
> >> > +        * and it should always be 0 for fme device.
> >> > +        */
> >> > +       WARN_ON(pdata->disable_count);
> >> > +
> >> > +       fdev->dev.platform_data = pdata;
> >> > +       fdev->num_resources = feature_nr;
> >> > +       fdev->resource = kcalloc(feature_nr, sizeof(*res), GFP_KERNEL);
> >> > +       if (!fdev->resource)
> >> > +               return -ENOMEM;
> >> > +
> >> > +       return 0;
> >> > +}
> >> > +
> >> > +static int remove_feature_dev(struct device *dev, void *data)
> >> > +{
> >> > +       struct platform_device *pdev = to_platform_device(dev);
> >> > +
> >> > +       platform_device_unregister(pdev);
> >> > +       return 0;
> >> > +}
> >> > +
> >> > +static int remove_parent_dev(struct device *dev, void *data)
> >> > +{
> >> > +       /* remove platform devices attached in the parent device */
> >> > +       device_for_each_child(dev, NULL, remove_feature_dev);
> >> > +       fpga_dev_destroy(to_fpga_dev(dev));
> >> > +       return 0;
> >> > +}
> >> > +
> >> > +static void remove_all_devs(struct pci_dev *pdev)
> >> > +{
> >> > +       /* remove parent device and all its children. */
> >> > +       device_for_each_child(&pdev->dev, NULL, remove_parent_dev);
> >> > +}
> >> > +
> >> > +static void build_info_free(struct build_feature_devs_info *binfo)
> >> > +{
> >> > +       if (!IS_ERR_OR_NULL(binfo->parent_dev))
> >> > +               remove_all_devs(binfo->pdev);
> >> > +
> >> > +       /*
> >> > +        * it is a valid id, free it. See comments in
> >> > +        * build_info_create_dev()
> >> > +        */
> >> > +       if (binfo->feature_dev && binfo->feature_dev->id >= 0)
> >> > +               free_fpga_id(feature_dev_id_type(binfo->feature_dev),
> >> > +                            binfo->feature_dev->id);
> >> > +
> >> > +       platform_device_put(binfo->feature_dev);
> >> > +
> >> > +       devm_kfree(&binfo->pdev->dev, binfo);
> >> > +}
> >> > +
> >> > +#define FEATURE_TYPE_AFU       0x1
> >> > +#define FEATURE_TYPE_PRIVATE   0x3
> >> > +
> >> > +/* FME and PORT GUID are fixed */
> >> > +#define FEATURE_FME_GUID "f9e17764-38f0-82fe-e346-524ae92aafbf"
> >> > +#define FEATURE_PORT_GUID "6b355b87-b06c-9642-eb42-8d139398b43a"
> >> > +
> >> > +static bool feature_is_fme(struct feature_afu_header *afu_hdr)
> >> > +{
> >> > +       uuid_le u;
> >> > +
> >> > +       uuid_le_to_bin(FEATURE_FME_GUID, &u);
> >> > +
> >> > +       return !uuid_le_cmp(u, afu_hdr->guid);
> >> > +}
> >> > +
> >> > +static bool feature_is_port(struct feature_afu_header *afu_hdr)
> >> > +{
> >> > +       uuid_le u;
> >> > +
> >> > +       uuid_le_to_bin(FEATURE_PORT_GUID, &u);
> >> > +
> >> > +       return !uuid_le_cmp(u, afu_hdr->guid);
> >> > +}
> >> > +
> >> > +/*
> >> > + * UAFU GUID is dynamic as it can be changed after FME downloads different
> >> > + * Green Bitstream to the port, so we treat the unknown GUIDs which are
> >> > + * attached on port's feature list as UAFU.
> >> > + */
> >> > +static bool feature_is_UAFU(struct build_feature_devs_info *binfo)
> >> > +{
> >> > +       if (!binfo->feature_dev ||
> >> > +             feature_dev_id_type(binfo->feature_dev) != PORT_ID)
> >> > +               return false;
> >> > +
> >> > +       return true;
> >> > +}
> >> > +
> >> > +static void
> >> > +build_info_add_sub_feature(struct build_feature_devs_info *binfo,
> >> > +                          int feature_id, const char *feature_name,
> >> > +                          resource_size_t resource_size, void __iomem *start)
> >> > +{
> >> > +
> >> > +       struct platform_device *fdev = binfo->feature_dev;
> >> > +       struct feature_platform_data *pdata = dev_get_platdata(&fdev->dev);
> >> > +       struct resource *res = &fdev->resource[feature_id];
> >> > +
> >> > +       res->start = pci_resource_start(binfo->pdev, binfo->current_bar) +
> >> > +               start - binfo->ioaddr;
> >> > +       res->end = res->start + resource_size - 1;
> >> > +       res->flags = IORESOURCE_MEM;
> >> > +       res->name = feature_name;
> >> > +
> >> > +       feature_platform_data_add(pdata, feature_id,
> >> > +                                 feature_name, feature_id, start);
> >> > +}
> >> > +
> >> > +struct feature_info {
> >> > +       const char *name;
> >> > +       resource_size_t resource_size;
> >> > +       int feature_index;
> >> > +};
> >> > +
> >> > +/* indexed by fme feature IDs which are defined in 'enum fme_feature_id'. */
> >> > +static struct feature_info fme_features[] = {
> >> > +       {
> >> > +               .name = FME_FEATURE_HEADER,
> >> > +               .resource_size = sizeof(struct feature_fme_header),
> >> > +               .feature_index = FME_FEATURE_ID_HEADER,
> >> > +       },
> >> > +       {
> >> > +               .name = FME_FEATURE_THERMAL_MGMT,
> >> > +               .resource_size = sizeof(struct feature_fme_thermal),
> >> > +               .feature_index = FME_FEATURE_ID_THERMAL_MGMT,
> >> > +       },
> >> > +       {
> >> > +               .name = FME_FEATURE_POWER_MGMT,
> >> > +               .resource_size = sizeof(struct feature_fme_power),
> >> > +               .feature_index = FME_FEATURE_ID_POWER_MGMT,
> >> > +       },
> >> > +       {
> >> > +               .name = FME_FEATURE_GLOBAL_PERF,
> >> > +               .resource_size = sizeof(struct feature_fme_gperf),
> >> > +               .feature_index = FME_FEATURE_ID_GLOBAL_PERF,
> >> > +       },
> >> > +       {
> >> > +               .name = FME_FEATURE_GLOBAL_ERR,
> >> > +               .resource_size = sizeof(struct feature_fme_err),
> >> > +               .feature_index = FME_FEATURE_ID_GLOBAL_ERR,
> >> > +       },
> >> > +       {
> >> > +               .name = FME_FEATURE_PR_MGMT,
> >> > +               .resource_size = sizeof(struct feature_fme_pr),
> >> > +               .feature_index = FME_FEATURE_ID_PR_MGMT,
> >> > +       }
> >> > +};
> >> > +
> >> > +/* indexed by port feature IDs which are defined in 'enum port_feature_id'. */
> >> > +static struct feature_info port_features[] = {
> >> > +       {
> >> > +               .name = PORT_FEATURE_HEADER,
> >> > +               .resource_size = sizeof(struct feature_port_header),
> >> > +               .feature_index = PORT_FEATURE_ID_HEADER,
> >> > +       },
> >> > +       {
> >> > +               .name = PORT_FEATURE_ERR,
> >> > +               .resource_size = sizeof(struct feature_port_error),
> >> > +               .feature_index = PORT_FEATURE_ID_ERROR,
> >> > +       },
> >> > +       {
> >> > +               .name = PORT_FEATURE_UMSG,
> >> > +               .resource_size = sizeof(struct feature_port_umsg),
> >> > +               .feature_index = PORT_FEATURE_ID_UMSG,
> >> > +       },
> >> > +       {
> >> > +               /* This feature isn't available for now */
> >> > +               .name = PORT_FEATURE_PR,
> >> > +               .resource_size = 0,
> >> > +               .feature_index = PORT_FEATURE_ID_PR,
> >> > +       },
> >> > +       {
> >> > +               .name = PORT_FEATURE_STP,
> >> > +               .resource_size = sizeof(struct feature_port_stp),
> >> > +               .feature_index = PORT_FEATURE_ID_STP,
> >> > +       },
> >> > +       {
> >> > +               /*
> >> > +                * For User AFU feature, its region size is not fixed, but
> >> > +                * reported by register PortCapability.mmio_size. Resource
> >> > +                * size of UAFU will be set while parse port device.
> >> > +                */
> >> > +               .name = PORT_FEATURE_UAFU,
> >> > +               .resource_size = 0,
> >> > +               .feature_index = PORT_FEATURE_ID_UAFU,
> >> > +       },
> >> > +};
> >> > +
> >> > +static int
> >> > +create_feature_instance(struct build_feature_devs_info *binfo,
> >> > +                       void __iomem *start, struct feature_info *finfo)
> >> > +{
> >> > +       if (binfo->ioend - start < finfo->resource_size)
> >> > +               return -EINVAL;
> >> > +
> >> > +       build_info_add_sub_feature(binfo, finfo->feature_index, finfo->name,
> >> > +                                  finfo->resource_size, start);
> >> > +       return 0;
> >> > +}
> >>
> >> build_info_add_sub_feature() is only called one time - here.  Could
> >> you collapse these two functions together?
> >
> > Sure.
> >
> >>
> >> > +
> >> > +static int parse_feature_fme(struct build_feature_devs_info *binfo,
> >> > +                            void __iomem *start)
> >> > +{
> >> > +       struct cci_drvdata *drvdata = dev_get_drvdata(&binfo->pdev->dev);
> >> > +       int ret;
> >> > +
> >> > +       ret = build_info_create_dev(binfo, FME_ID, fme_feature_num(),
> >> > +                                       FPGA_FEATURE_DEV_FME);
> >> > +       if (ret)
> >> > +               return ret;
> >> > +
> >> > +       if (drvdata->fme_dev) {
> >> > +               dev_err(&binfo->pdev->dev, "Multiple FMEs are detected.\n");
> >> > +               return -EINVAL;
> >> > +       }
> >> > +
> >> > +       return create_feature_instance(binfo, start,
> >> > +                                      &fme_features[FME_FEATURE_ID_HEADER]);
> >> > +}
> >> > +
> >> > +static int parse_feature_fme_private(struct build_feature_devs_info *binfo,
> >> > +                                    struct feature_header *hdr)
> >> > +{
> >> > +       struct feature_header header;
> >> > +
> >> > +       header.csr = readq(hdr);
> >> > +
> >> > +       if (header.id >= ARRAY_SIZE(fme_features)) {
> >> > +               dev_info(&binfo->pdev->dev, "FME feature id %x is not supported yet.\n",
> >> > +                        header.id);
> >> > +               return 0;
> >> > +       }
> >> > +
> >> > +       return create_feature_instance(binfo, hdr, &fme_features[header.id]);
> >> > +}
> >> > +
> >> > +static int parse_feature_port(struct build_feature_devs_info *binfo,
> >> > +                            void __iomem *start)
> >> > +{
> >> > +       int ret;
> >> > +
> >> > +       ret = build_info_create_dev(binfo, PORT_ID, port_feature_num(),
> >> > +                                       FPGA_FEATURE_DEV_PORT);
> >> > +       if (ret)
> >> > +               return ret;
> >> > +
> >> > +       return create_feature_instance(binfo, start,
> >> > +                                      &port_features[PORT_FEATURE_ID_HEADER]);
> >> > +}
> >> > +
> >> > +static void enable_port_uafu(struct build_feature_devs_info *binfo,
> >> > +                            void __iomem *start)
> >> > +{
> >> > +       enum port_feature_id id = PORT_FEATURE_ID_UAFU;
> >> > +       struct feature_port_header *port_hdr;
> >> > +       struct feature_port_capability capability;
> >> > +
> >> > +       port_hdr = (struct feature_port_header *)start;
> >> > +       capability.csr = readq(&port_hdr->capability);
> >> > +       port_features[id].resource_size = capability.mmio_size << 10;
> >> > +
> >> > +       /*
> >> > +        * To enable User AFU, driver needs to clear reset bit on related port,
> >> > +        * otherwise the mmio space of this user AFU will be invalid.
> >> > +        */
> >> > +       if (port_features[id].resource_size)
> >> > +               fpga_port_reset(binfo->feature_dev);
> >> > +}
> >> > +
> >> > +static int parse_feature_port_private(struct build_feature_devs_info *binfo,
> >> > +                                     struct feature_header *hdr)
> >> > +{
> >> > +       struct feature_header header;
> >> > +       enum port_feature_id id;
> >> > +
> >> > +       header.csr = readq(hdr);
> >> > +       /*
> >> > +        * the region of port feature id is [0x10, 0x13], + 1 to reserve 0
> >> > +        * which is dedicated for port-hdr.
> >> > +        */
> >> > +       id = (header.id & 0x000f) + 1;
> >> > +
> >> > +       if (id >= ARRAY_SIZE(port_features)) {
> >> > +               dev_info(&binfo->pdev->dev, "Port feature id %x is not supported yet.\n",
> >> > +                        header.id);
> >> > +               return 0;
> >> > +       }
> >> > +
> >> > +       return create_feature_instance(binfo, hdr, &port_features[id]);
> >> > +}
> >> > +
> >> > +static int parse_feature_port_uafu(struct build_feature_devs_info *binfo,
> >> > +                                struct feature_header *hdr)
> >> > +{
> >> > +       enum port_feature_id id = PORT_FEATURE_ID_UAFU;
> >> > +       int ret;
> >> > +
> >> > +       if (port_features[id].resource_size) {
> >> > +               ret = create_feature_instance(binfo, hdr, &port_features[id]);
> >> > +               port_features[id].resource_size = 0;
> >> > +       } else {
> >> > +               dev_err(&binfo->pdev->dev, "the uafu feature header is mis-configured.\n");
> >> > +               ret = -EINVAL;
> >> > +       }
> >> > +
> >> > +       return ret;
> >> > +}
> >> > +
> >> > +static int parse_feature_afus(struct build_feature_devs_info *binfo,
> >> > +                             struct feature_header *hdr)
> >> > +{
> >> > +       int ret;
> >> > +       struct feature_afu_header *afu_hdr, header;
> >> > +       void __iomem *start;
> >> > +       void __iomem *end = binfo->ioend;
> >> > +
> >> > +       start = hdr;
> >> > +       for (; start < end; start += header.next_afu) {
> >> > +               if (end - start < (sizeof(*afu_hdr) + sizeof(*hdr)))
> >> > +                       return -EINVAL;
> >> > +
> >> > +               hdr = start;
> >> > +               afu_hdr = (struct feature_afu_header *) (hdr + 1);
> >> > +               header.csr = readq(&afu_hdr->csr);
> >> > +
> >> > +               if (feature_is_fme(afu_hdr)) {
> >> > +                       ret = parse_feature_fme(binfo, hdr);
> >> > +                       binfo->pfme_hdr = hdr;
> >> > +                       if (ret)
> >> > +                               return ret;
> >> > +               } else if (feature_is_port(afu_hdr)) {
> >> > +                       ret = parse_feature_port(binfo, hdr);
> >> > +                       enable_port_uafu(binfo, hdr);
> >> > +                       if (ret)
> >> > +                               return ret;
> >> > +               } else if (feature_is_UAFU(binfo)) {
> >> > +                       ret = parse_feature_port_uafu(binfo, hdr);
> >> > +                       if (ret)
> >> > +                               return ret;
> >> > +               } else
> >> > +                       dev_info(&binfo->pdev->dev, "AFU GUID %pUl is not supported yet.\n",
> >> > +                                afu_hdr->guid.b);
> >> > +
> >> > +               if (!header.next_afu)
> >> > +                       break;
> >> > +       }
> >> > +
> >> > +       return 0;
> >> > +}
> >> > +
> >> > +static int parse_feature_private(struct build_feature_devs_info *binfo,
> >> > +                                struct feature_header *hdr)
> >> > +{
> >> > +       struct feature_header header;
> >> > +
> >> > +       header.csr = readq(hdr);
> >> > +
> >> > +       if (!binfo->feature_dev) {
> >> > +               dev_err(&binfo->pdev->dev, "the private feature %x does not belong to any AFU.\n",
> >> > +                       header.id);
> >> > +               return -EINVAL;
> >> > +       }
> >> > +
> >> > +       switch (feature_dev_id_type(binfo->feature_dev)) {
> >> > +       case FME_ID:
> >> > +               return parse_feature_fme_private(binfo, hdr);
> >> > +       case PORT_ID:
> >> > +               return parse_feature_port_private(binfo, hdr);
> >> > +       default:
> >> > +               dev_info(&binfo->pdev->dev, "private feature %x belonging to AFU %s is not supported yet.\n",
> >> > +                        header.id, binfo->feature_dev->name);
> >> > +       }
> >> > +       return 0;
> >> > +}
> >> > +
> >> > +static int parse_feature(struct build_feature_devs_info *binfo,
> >> > +                        struct feature_header *hdr)
> >> > +{
> >> > +       struct feature_header header;
> >> > +       int ret = 0;
> >> > +
> >> > +       header.csr = readq(hdr);
> >> > +
> >> > +       switch (header.type) {
> >> > +       case FEATURE_TYPE_AFU:
> >> > +               ret = parse_feature_afus(binfo, hdr);
> >> > +               break;
> >> > +       case FEATURE_TYPE_PRIVATE:
> >> > +               ret = parse_feature_private(binfo, hdr);
> >> > +               break;
> >> > +       default:
> >> > +               dev_info(&binfo->pdev->dev,
> >> > +                        "Feature Type %x is not supported.\n", hdr->type);
> >> > +       };
> >> > +
> >> > +       return ret;
> >> > +}
> >> > +
> >> > +static int
> >> > +parse_feature_list(struct build_feature_devs_info *binfo, void __iomem *start)
> >> > +{
> >> > +       struct feature_header *hdr, header;
> >> > +       void __iomem *end = binfo->ioend;
> >> > +       int ret = 0;
> >> > +
> >>
> >> Maybe a helpful comment that we are stepping through a linked list of features.
> >
> > Sure, will add more comments.
> >
> > All kind of features has a common header, they are linked together via the
> > header.next_header_offset.
> >
> > If it's a TYPE=AFU feature, it will have one more pointer named NEXT_AFU,
> > which could point to another TYPE=AFU feature, please refer to function
> > parse_feature_afus.
> >
> > If it's a TYPE=private feature, there is no other pointers, please refer
> > to function parse_feature_private.
> 
> OK, yes that will be helpful.
> 
> >
> >>
> >> > +       for (; start < end; start += header.next_header_offset) {
> >> > +               if (end - start < sizeof(*hdr)) {
> >> > +                       dev_err(&binfo->pdev->dev, "The region is too small to contain a feature.\n");
> >> > +                       ret =  -EINVAL;
> >> > +                       break;
> >> > +               }
> >> > +
> >> > +               hdr = (struct feature_header *)start;
> >> > +               ret = parse_feature(binfo, hdr);
> >> > +               if (ret)
> >> > +                       break;
> >>
> >> Instead of parse_feature, this can save off the enumeration info and
> >> continue to read in the linked list.  After all the headers are read
> >> in, call the (separate) enumeration code to step through the saved
> >> headers, parse them, and create the devices.  Since the memory is
> >> iomapped during the process of reading in the headers, the enumeration
> >> code doesn't have to be so pcie specific.  Then this code base is
> >> better set to run on embedded devices also.
> >
> > Actually, it has parse_feature here, because they're different type of
> > features. e.g for TYPE = AFU, GUID = PORT, it requires driver to check
> > its NEXT_AFU pointer to find User AFU (and its header). For private
> > features, there is no next_afu pointer at all.
> >
> >>
> >> > +
> >> > +               header.csr = readq(hdr);
> >> > +               if (!header.next_header_offset)
> >> > +                       break;
> >> > +       }
> >> > +
> >> > +       return ret;
> >> > +}
> >> > +
> >> > +static int parse_ports_from_fme(struct build_feature_devs_info *binfo)
> >> > +{
> >> > +       struct feature_fme_header *fme_hdr;
> >> > +       struct feature_fme_port port;
> >> > +       int i = 0, ret = 0;
> >> > +
> >> > +       if (binfo->pfme_hdr == NULL) {
> >> > +               dev_dbg(&binfo->pdev->dev, "VF is detected.\n");
> >> > +               return ret;
> >> > +       }
> >> > +
> >> > +       fme_hdr = binfo->pfme_hdr;
> >> > +
> >> > +       do {
> >> > +               port.csr = readq(&fme_hdr->port[i]);
> >> > +               if (!port.port_implemented)
> >> > +                       break;
> >> > +
> >> > +               ret = parse_switch_to(binfo, port.port_bar);
> >> > +               if (ret)
> >> > +                       break;
> >> > +
> >> > +               ret = parse_feature_list(binfo,
> >> > +                               binfo->ioaddr + port.port_offset);
> >> > +               if (ret)
> >> > +                       break;
> >> > +       } while (++i < MAX_FPGA_PORT_NUM);
> >> > +
> >> > +       return ret;
> >> > +}
> >> > +
> >> > +static int create_init_drvdata(struct pci_dev *pdev)
> >> > +{
> >> > +       struct cci_drvdata *drvdata;
> >> > +
> >> > +       drvdata = devm_kzalloc(&pdev->dev, sizeof(*drvdata), GFP_KERNEL);
> >> > +       if (!drvdata)
> >> > +               return -ENOMEM;
> >> > +
> >> > +       mutex_init(&drvdata->lock);
> >> > +       INIT_LIST_HEAD(&drvdata->port_dev_list);
> >> > +       INIT_LIST_HEAD(&drvdata->regions);
> >> > +
> >> > +       dev_set_drvdata(&pdev->dev, drvdata);
> >> > +       return 0;
> >> > +}
> >> > +
> >> > +static void destroy_drvdata(struct pci_dev *pdev)
> >> > +{
> >> > +       struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> >> > +
> >> > +       if (drvdata->fme_dev) {
> >> > +               /* fme device should be unregistered first. */
> >> > +               WARN_ON(device_is_registered(drvdata->fme_dev));
> >> > +               free_fpga_id(FME_ID, to_platform_device(drvdata->fme_dev)->id);
> >> > +               put_device(drvdata->fme_dev);
> >> > +       }
> >> > +
> >> > +       cci_pci_remove_port_devs(pdev);
> >> > +       cci_pci_release_regions(pdev);
> >> > +       dev_set_drvdata(&pdev->dev, NULL);
> >> > +       devm_kfree(&pdev->dev, drvdata);
> >> > +}
> >> > +
> >> > +static int cci_pci_create_feature_devs(struct pci_dev *pdev)
> >> > +{
> >> > +       struct build_feature_devs_info *binfo;
> >> > +       int ret;
> >> > +
> >> > +       binfo = build_info_alloc_and_init(pdev);
> >> > +       if (!binfo)
> >> > +               return -ENOMEM;
> >> > +
> >> > +       binfo->parent_dev = fpga_dev_create(&pdev->dev, INTEL_FPGA_DEV);
> >> > +       if (IS_ERR(binfo->parent_dev)) {
> >> > +               ret = PTR_ERR(binfo->parent_dev);
> >> > +               goto free_binfo_exit;
> >> > +       }
> >> > +
> >> > +       ret = parse_start(binfo);
> >> > +       if (ret)
> >> > +               goto free_binfo_exit;
> >> > +
> >> > +       ret = parse_feature_list(binfo, binfo->ioaddr);
> >> > +       if (ret)
> >> > +               goto free_binfo_exit;
> >> > +
> >> > +       ret = parse_ports_from_fme(binfo);
> >> > +       if (ret)
> >> > +               goto free_binfo_exit;
> >>
> >> So ideally, there would be a function call here that read all the
> >> headers from hardware, ioremapping as it went along.  Then after that,
> >> call the enumeration code to create the devices.
> >
> > If we have a function call here which read all the headers, then it
> > finish most works of parsing the device feature list. The only thing
> > that enumeration code to do is to create the devices with prepared
> > resources. It's fine to me, as we can just save information to this
> > binfo, and create devices later, but is this matches with what you
> > think? :)
> 
> That is what I'm thinking more or less.   I'm trying to see this
> implemented some way where iomapping is separate from creating
> devices.  There are drivers in the kernel that have 3 parts: core,
> platform, and pci.  The core has most of the functionality specific to
> the driver where the platform and pci parts handle iomapping depending
> on which bus the device is on.

Yes, I think the core code here is the enumeration part, as we can re-use
platform drivers for the feature devices created by the enumeration part.
As I mentioned above, will have a try and see. Thanks a lot for these
suggestions. :)

Thanks
Hao

> 
> Alan
> 
> >
> > Thanks
> > Hao
> >
> >>
> >> > +
> >> > +       ret = build_info_commit_dev(binfo);
> >> > +       if (ret)
> >> > +               goto free_binfo_exit;
> >> > +
> >> > +       /*
> >> > +        * everything is okay, reset ->parent_dev to stop it being
> >> > +        * freed by build_info_free()
> >> > +        */
> >> > +       binfo->parent_dev = NULL;
> >> > +
> >> > +free_binfo_exit:
> >> > +       build_info_free(binfo);
> >> > +       return ret;
> >> > +}
> >> > +
> >> >  /* PCI Device ID */
> >> >  #define PCIe_DEVICE_ID_PF_INT_5_X      0xBCBD
> >> >  #define PCIe_DEVICE_ID_PF_INT_6_X      0xBCC0
> >> > @@ -81,9 +898,18 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
> >> >                 goto release_region_exit;
> >> >         }
> >> >
> >> > -       /* TODO: create and add the platform device per feature list */
> >> > +       ret = create_init_drvdata(pcidev);
> >> > +       if (ret)
> >> > +               goto release_region_exit;
> >> > +
> >> > +       ret = cci_pci_create_feature_devs(pcidev);
> >> > +       if (ret)
> >> > +               goto destroy_drvdata_exit;
> >> > +
> >> >         return 0;
> >> >
> >> > +destroy_drvdata_exit:
> >> > +       destroy_drvdata(pcidev);
> >> >  release_region_exit:
> >> >         pci_release_regions(pcidev);
> >> >  disable_error_report_exit:
> >> > @@ -94,6 +920,8 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
> >> >
> >> >  static void cci_pci_remove(struct pci_dev *pcidev)
> >> >  {
> >> > +       remove_all_devs(pcidev);
> >> > +       destroy_drvdata(pcidev);
> >> >         pci_release_regions(pcidev);
> >> >         pci_disable_pcie_error_reporting(pcidev);
> >> >         pci_disable_device(pcidev);
> >> > @@ -108,14 +936,23 @@ static void cci_pci_remove(struct pci_dev *pcidev)
> >> >
> >> >  static int __init ccidrv_init(void)
> >> >  {
> >> > +       int ret;
> >> > +
> >> >         pr_info("Intel(R) FPGA PCIe Driver: Version %s\n", DRV_VERSION);
> >> >
> >> > -       return pci_register_driver(&cci_pci_driver);
> >> > +       fpga_ids_init();
> >> > +
> >> > +       ret = pci_register_driver(&cci_pci_driver);
> >> > +       if (ret)
> >> > +               fpga_ids_destroy();
> >> > +
> >> > +       return ret;
> >> >  }
> >> >
> >> >  static void __exit ccidrv_exit(void)
> >> >  {
> >> >         pci_unregister_driver(&cci_pci_driver);
> >> > +       fpga_ids_destroy();
> >> >  }
> >> >
> >> >  module_init(ccidrv_init);
> >> > --
> >> > 1.8.3.1
> >> >