linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH 00/16] Intel FPGA Device Drivers
@ 2017-03-30 12:08 Wu Hao
  2017-03-30 12:08 ` [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview Wu Hao
                   ` (17 more replies)
  0 siblings, 18 replies; 93+ messages in thread
From: Wu Hao @ 2017-03-30 12:08 UTC (permalink / raw)
  To: atull, moritz.fischer, linux-fpga, linux-kernel
  Cc: luwei.kang, yi.z.zhang, hao.wu

Hi All,

Here is a patch-series adding drivers for Intel FPGA devices.

The Intel FPGA driver provides interfaces for userspace applications to
configure, enumerate, open, and access FPGA accelerators on platforms
equipped with Intel(R) FPGA solutions and enables system level management
functions such as FPGA partial reconfiguration, power management and
virtualization.

This patch series only adds the basic functions for FPGA accelerators and
partial reconfiguration. Patches for more functions, e.g power management
and virtualization, will be submitted after this series gets reviewed.

Patch 1: add a document for Intel FPGA driver overview, including the HW
architecture, driver organization, device enumeration, virtualization and
opens.

Patch 2: introduce a fpga-dev class. It's used in below Intel FPGA PCIe
device driver, to represent a FPGA device on the system, and all actual
feature devices should be registered as child nodes of this container
fpga-dev device.

Patch 3-7: implement Intel FPGA PCIe device driver. It walks through the
'Device Feature List' in the PCI Bar, creates the container fpga-dev as
parent and platform devices as children for the feature devices it found.

Patch 8-11: implement Intel FPGA Management Engine (FME) driver. It's a
platform driver matching with the FME platform device created by above
PCIe driver. Sysfs and device file ioctls are exposed as user interfaces
to allow partial reconfiguration to Accelerated Function Units (AFUs) from
user space applications.

Patch 12-16: implement Intel FPGA Accelerated Function Unit (AFU) driver.
It's a platform driver matching with AFU platform device created by above
PCIe driver. It provides user interfaces to expose the AFU MMIO region,
map/unmap dma buffer, and control the port which AFU connects to.

Kang Luwei (3):
  fpga: intel: add FPGA Management Engine driver basic framework
  fpga: intel: fme: add header sub feature support
  fpga: intel: fme: add partial reconfiguration sub feature support

Wu Hao (8):
  docs: fpga: add a document for Intel FPGA driver overview
  fpga: add FPGA device framework
  fpga: intel: pcie: adds fpga_for_each_port callback for fme device
  fpga: intel: fme: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls
    support
  fpga: intel: add FPGA Accelerated Function Unit driver basic framework
  fpga: intel: afu: add header sub feature support
  fpga: intel: afu add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls
    support
  fpga: intel: afu: add FPGA_PORT_DMA_MAP/UNMAP ioctls support

Xiao Guangrong (4):
  fpga: intel: pcie: parse feature list and create platform device for
    features.
  fpga: intel: pcie: add chardev support for feature devices
  fpga: intel: add feature device infrastructure
  fpga: intel: afu: add user afu sub feature support

Zhang Yi (1):
  fpga: intel: add FPGA PCIe device driver

 Documentation/fpga/intel-fpga.txt    |  259 +++++++++
 Documentation/ioctl/ioctl-number.txt |    1 +
 drivers/fpga/Kconfig                 |    8 +
 drivers/fpga/Makefile                |    6 +
 drivers/fpga/fpga-dev.c              |  120 ++++
 drivers/fpga/intel/Kconfig           |   44 ++
 drivers/fpga/intel/LICENSE.BSD       |   24 +
 drivers/fpga/intel/Makefile          |    7 +
 drivers/fpga/intel/afu-dma-region.c  |  373 +++++++++++++
 drivers/fpga/intel/afu-main.c        |  471 ++++++++++++++++
 drivers/fpga/intel/afu-region.c      |  129 +++++
 drivers/fpga/intel/afu.h             |   72 +++
 drivers/fpga/intel/feature-dev.c     |  281 ++++++++++
 drivers/fpga/intel/feature-dev.h     |  499 +++++++++++++++++
 drivers/fpga/intel/fme-main.c        |  267 +++++++++
 drivers/fpga/intel/fme-pr.c          |  400 ++++++++++++++
 drivers/fpga/intel/fme.h             |   32 ++
 drivers/fpga/intel/pcie.c            | 1006 ++++++++++++++++++++++++++++++++++
 include/linux/fpga/fpga-dev.h        |   34 ++
 include/uapi/linux/intel-fpga.h      |  194 +++++++
 20 files changed, 4227 insertions(+)
 create mode 100644 Documentation/fpga/intel-fpga.txt
 create mode 100644 drivers/fpga/fpga-dev.c
 create mode 100644 drivers/fpga/intel/Kconfig
 create mode 100644 drivers/fpga/intel/LICENSE.BSD
 create mode 100644 drivers/fpga/intel/Makefile
 create mode 100644 drivers/fpga/intel/afu-dma-region.c
 create mode 100644 drivers/fpga/intel/afu-main.c
 create mode 100644 drivers/fpga/intel/afu-region.c
 create mode 100644 drivers/fpga/intel/afu.h
 create mode 100644 drivers/fpga/intel/feature-dev.c
 create mode 100644 drivers/fpga/intel/feature-dev.h
 create mode 100644 drivers/fpga/intel/fme-main.c
 create mode 100644 drivers/fpga/intel/fme-pr.c
 create mode 100644 drivers/fpga/intel/fme.h
 create mode 100644 drivers/fpga/intel/pcie.c
 create mode 100644 include/linux/fpga/fpga-dev.h
 create mode 100644 include/uapi/linux/intel-fpga.h

-- 
2.7.4

^ permalink raw reply	[flat|nested] 93+ messages in thread

* [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview
  2017-03-30 12:08 [PATCH 00/16] Intel FPGA Device Drivers Wu Hao
@ 2017-03-30 12:08 ` Wu Hao
  2017-03-31 18:24   ` matthew.gerlach
  2017-03-30 12:08 ` [PATCH 02/16] fpga: add FPGA device framework Wu Hao
                   ` (16 subsequent siblings)
  17 siblings, 1 reply; 93+ messages in thread
From: Wu Hao @ 2017-03-30 12:08 UTC (permalink / raw)
  To: atull, moritz.fischer, linux-fpga, linux-kernel
  Cc: luwei.kang, yi.z.zhang, hao.wu, Enno Luebbers, Xiao Guangrong

Add a document for Intel FPGA driver overview.

Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
 Documentation/fpga/intel-fpga.txt | 259 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 259 insertions(+)
 create mode 100644 Documentation/fpga/intel-fpga.txt

diff --git a/Documentation/fpga/intel-fpga.txt b/Documentation/fpga/intel-fpga.txt
new file mode 100644
index 0000000..9396cea
--- /dev/null
+++ b/Documentation/fpga/intel-fpga.txt
@@ -0,0 +1,259 @@
+===============================================================================
+                    Intel FPGA driver Overview
+-------------------------------------------------------------------------------
+                Enno Luebbers <enno.luebbers@intel.com>
+                Xiao Guangrong <guangrong.xiao@linux.intel.com>
+                Wu Hao <hao.wu@intel.com>
+
+The Intel FPGA driver provides interfaces for userspace applications to
+configure, enumerate, open, and access FPGA accelerators on platforms equipped
+with Intel(R) FPGA solutions and enables system level management functions such
+as FPGA reconfiguration, power management, and virtualization.
+
+HW Architecture
+===============
+From the OS's point of view, the FPGA hardware appears as a regular PCIe device.
+The FPGA device memory is organized using a predefined data structure (Device
+Feature List). Features supported by the particular FPGA device are exposed
+through these data structures, as illustrated below:
+
+  +-------------------------------+  +-------------+
+  |              PF               |  |     VF      |
+  +-------------------------------+  +-------------+
+      ^            ^         ^              ^
+      |            |         |              |
++-----|------------|---------|--------------|-------+
+|     |            |         |              |       |
+|  +-----+     +-------+ +-------+      +-------+   |
+|  | FME |     | Port0 | | Port1 |      | Port2 |   |
+|  +-----+     +-------+ +-------+      +-------+   |
+|                  ^         ^              ^       |
+|                  |         |              |       |
+|              +-------+ +------+       +-------+   |
+|              |  AFU  | |  AFU |       |  AFU  |   |
+|              +-------+ +------+       +-------+   |
+|                                                   |
+|                 FPGA PCIe Device                  |
++---------------------------------------------------+
+
+The driver supports PCIe SR-IOV to create virtual functions (VFs) which can be
+used to assign individual accelerators to virtual machines .
+
+FME (FPGA Management Engine)
+============================
+The FPGA Management Enging performs power and thermal management, error
+reporting, reconfiguration, performance reporting, and other infrastructure
+functions. Each FPGA has one FME, which is always accessed through the physical
+function (PF).
+
+User-space applications can acquire exclusive access to the FME using open(),
+and release it using close().
+
+The following functions are exposed through ioctls:
+
+	Get driver API version (FPGA_GET_API_VERSION)
+	Check for extensions (FPGA_CHECK_EXTENSION)
+	Assign port to PF (FPGA_FME_PORT_ASSIGN)
+	Release port from PF (FPGA_FME_PORT_RELEASE)
+	Program bitstream (FPGA_FME_PORT_PR)
+
+More functions are exposed through sysfs
+(/sys/class/fpga/fpga.n/intel-fpga-fme.n/):
+
+	Read bitstream ID (bitstream_id)
+	Read bitstream metadata (bitstream_metadata)
+	Read number of ports (ports_num)
+	Read socket ID (socket_id)
+	Read performance counters (perf/)
+	Power management (power_mgmt/)
+	Thermal management (thermal_mgmt/)
+	Error reporting (errors/)
+
+PORT
+====
+A port represents the interface between the static FPGA fabric (the "blue
+bitstream") and a partially reconfigurable region containing an AFU (the "green
+bitstream"). It controls the communication from SW to the accelerator and
+exposes features such as reset and debug.
+
+A PCIe device may have several ports and each port can be released from PF by
+FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe sriov
+sysfs interface.
+
+AFU
+===
+An AFU is attached to a port and exposes a 256k MMIO region to be used for
+accelerator-specific control registers.
+
+User-space applications can acquire exclusive access to an AFU attached to a
+port by using open() on the port device node, and release it using close().
+
+The following functions are exposed through ioctls:
+
+	Get driver API version (FPGA_GET_API_VERSION)
+	Check for extensions (FPGA_CHECK_EXTENSION)
+	Get port info (FPGA_PORT_GET_INFO)
+	Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
+	Map DMA buffer (FPGA_PORT_DMA_MAP)
+	Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
+	Reset AFU (FPGA_PORT_RESET)
+	Enable UMsg (FPGA_PORT_UMSG_ENABLE)
+	Disable UMsg (FPGA_PORT_UMSG_DISABLE)
+	Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
+	Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)
+
+User-space applications can also mmap() accelerator MMIO regions.
+
+More functions are exposed through sysfs:
+(/sys/class/fpga/fpga.n/intel-fpga-port.m/):
+
+	Read Accelerator GUID (afu_id)
+	Error reporting (errors/)
+
+Partial Reconfiguration
+=======================
+As mentioned above, accelerators can be reconfigured through partial
+reconfiguration of a green bitstream file (GBS). The green bitstream must have
+been generated for the exact blue bitstream and targeted reconfigurable region
+(port) of the FPGA; otherwise, the reconfiguration operation will fail and
+possibly cause system instability. This compatibility can be checked by
+comparing the interface ID noted in the GBS header against the interface ID
+exposed by the FME through sysfs (see above). This check is usually done by
+user-space before calling the reconfiguration IOCTL.
+
+FPGA virtualization
+===================
+To enable accessing an accelerator from applications running in a VM, the
+respective AFU's port needs to be assigned to a VF using the following steps:
+
+ a) The PF owns all AFU ports by default. Any port that needs to be reassigned
+ to a VF must be released from PF firstly through the FPGA_FME_PORT_RELEASE
+ ioctl on the FME device.
+
+ b) Once N ports are released from PF, then user can use below command to
+ enable SRIOV and VFs. Each VF owns only one Port with AFU.
+
+ echo N > $PCI_DEVICE_PATH/sriov_numvfs
+
+ c) Pass through the VFs to VMs
+
+ d) The AFU under VF is accessiable from applications in VM (using the same
+ driver inside the VF).
+
+Note the an FME can't be assigned to a VF, thus PR and other management
+functions are only available via the PF.
+
+
+Driver organization
+===================
+
+  +------------------+  +---------+   |             +---------+
+  | +-------+        |  |         |   |             |         |
+  | | FPGA  |  FME   |  |   AFU   |   |             |   AFU   |
+  | |Manager| Module |  |  Module |   |             |  Module |
+  | +-------+        |  |         |   |             |         |
+  +------------------+  +---------+   |             +---------+
+        +-----------------------+     |      +-----------------------+
+        | FPGA Container Device |     |      | FPGA Container Device |
+        +-----------------------+     |      +-----------------------+
+          +------------------+        |         +------------------+
+          | FPGA PCIE Module |        | Virtual | FPGA PCIE Module |
+          +------------------+   Host | Machine +------------------+
+ ------------------------------------ | ------------------------------
+           +---------------+          |          +---------------+
+           | PCI PF Device |          |          | PCI VF Device |
+           +---------------+          |          +---------------+
+
+The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe device
+driver is always loaded first once a FPGA PCIE PF or VF device is detected. This
+driver plays an infrastructural role in the driver architecuture.  It:
+
+	a) creates FPGA container device as parent of the feature devices.
+	b) walks through the Device Feature List, which is implemented in PCIE
+	   device BAR memory, to discover feature devices and their sub features
+	   and create platform device for them under the container device.
+	c) supports SRIOV.
+	d) introduces the feature device infrastructure, which abstracts
+	   operations for sub features and exposes common functions to feature
+	   device drivers.
+
+The FPGA Management Engine (FME) driver is a platform driver which is loaded
+automatically after FME platform device creation from the PCIE driver. It
+provides the key features for FPGA management, including:
+
+	a) Power and thermal management, error reporting, performance reporting
+	   and other infrastructure functions. Users can access these functions
+	   via sysfs interfaces exposed by FME driver.
+	b) Paritial Reconfiguration. The FME driver registers a FPGA Manager
+	   during PR sub feature initialization; once it receives an
+	   FPGA_FME_PORT_PR ioctl from user, it invokes the common interface
+	   function from FPGA Manager to complete the partial reconfiguration of
+	   the bitstream to the given port.
+	c) Port management for virtualization. The FME driver introduces two
+	   ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
+	   FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the port is
+	   released from the PF, it can be assigned to the VF through the SRIOV
+	   interfaces provided by PCIE driver. (Refer to "FPGA virtualization"
+	   for more details).
+
+Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) driver
+is probed once the AFU platform device is created. The main function of this
+module is to provide an interface for userspace applications to access the
+individual accelerators, including basic reset control on port, AFU MMIO region
+export, dma buffer mapping service, UMsg notification, and remote debug
+functions (see above).
+
+
+Device enumeration
+==================
+This section introduces how applications enumerate the fpga device from
+the sysfs hierarchy under /sys/class/fpga.
+
+In the example below, two Intel(R) FPGA devices are installed in the host. Each
+fpga device has one FME and two ports (AFUs).
+
+For each FPGA device, a device director is created under /sys/class/fpga/:
+
+	/sys/class/fpga/fpga.0
+	/sys/class/fpga/fpga.1
+
+The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's name.
+Application can retrieve name information via the sysfs interface:
+
+	/sys/class/fpga/fpga.0/name
+
+Each node has one FME and two ports (AFUs) as child devices:
+
+	/sys/class/fpga/fpga.0/intel-fpga-fme.0
+	/sys/class/fpga/fpga.0/intel-fpga-port.0
+	/sys/class/fpga/fpga.0/intel-fpga-port.1
+
+	/sys/class/fpga/fpga.1/intel-fpga-fme.1
+	/sys/class/fpga/fpga.1/intel-fpga-port.2
+	/sys/class/fpga/fpga.1/intel-fpga-port.3
+
+In general, the FME/AFU sysfs interfaces are named as follows:
+
+	/sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/
+	/sys/class/fpga/<fpga.n>/<intel-fpga-port.m>/
+
+with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
+ports.
+
+The device nodes used for ioctl() or mmap() can be referenced through:
+
+	/sys/class/fpga/<fpga.n>/<intel-fpga-port.n>/dev
+	/sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/dev
+
+
+Open discussions
+================
+The current FME driver does not provide user space access to the FME MMIO
+region, but exposes access through sysfs and ioctls. It also provides an FPGA
+manger interface for partial reconfiguration (PR), but does not make use of
+fpga-regions. User PR requests via the FPGA_FME_PORT_PR ioctl are handled inside
+the FME, and fpga-region depends on device tree which is not used at all. There
+are patches from Alan Tull to separate the device tree specific code and
+introduce a sysfs interface for PR. We plan to add fpga-regions support in the
+driver once the related patches get merged. Then the FME driver should create
+one fpga-region for each Port/AFU.
-- 
2.7.4

^ permalink raw reply	[flat|nested] 93+ messages in thread

* [PATCH 02/16] fpga: add FPGA device framework
  2017-03-30 12:08 [PATCH 00/16] Intel FPGA Device Drivers Wu Hao
  2017-03-30 12:08 ` [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview Wu Hao
@ 2017-03-30 12:08 ` Wu Hao
  2017-03-31  6:09   ` Greg KH
  2017-03-31  6:13   ` Greg KH
  2017-03-30 12:08 ` [PATCH 03/16] fpga: intel: add FPGA PCIe device driver Wu Hao
                   ` (15 subsequent siblings)
  17 siblings, 2 replies; 93+ messages in thread
From: Wu Hao @ 2017-03-30 12:08 UTC (permalink / raw)
  To: atull, moritz.fischer, linux-fpga, linux-kernel
  Cc: luwei.kang, yi.z.zhang, hao.wu, Tim Whisonant, Enno Luebbers,
	Shiva Rao, Christopher Rauer, Xiao Guangrong

During FPGA device (e.g PCI-based) discovery, platform devices are
registered for different FPGA function units. But the device node path
isn't quite friendly to applications.

Consider this case, applications want to access child device's sysfs file
for some information.

1) Access using bus-based path (e.g PCI)

  /sys/bus/pci/devices/xxxxx/fpga_func_a.0/sysfs_file

  From the path, it's clear which PCI device is the parent, but not perfect
  solution for applications. PCI device BDF is not fixed, application may
  need to search all PCI device to find the actual FPGA Device.

2) Or access using platform device path

  /sys/bus/platform/devices/fpga_func_a.0/sysfs_file

  Applications find the actual function by name easily, but no information
  about which fpga device it belongs to. It's quite confusing if multiple
  FPGA devices are in one system.

'FPGA Device' class is introduced to resolve this problem. Each node under
this class represents a fpga device, which may have one or more child
devices. Applications only need to search under this FPGA Device class
folder to find the child device node it needs.

For example, for the platform has 2 fpga devices, each fpga device has
3 child devices, the hierarchy looks like this.

Two nodes are under /sys/class/fpga/:
/sys/class/fpga/fpga.0
/sys/class/fpga/fpga.1

Each node has 1 function A device and 2 function B devices:
/sys/class/fpga/fpga.0/func_a.0
/sys/class/fpga/fpga.0/func_b.0
/sys/class/fpga/fpga.0/func_b.1

/sys/class/fpga/fpga.1/func_a.1
/sys/class/fpga/fpga.1/func_b.2
/sys/class/fpga/fpga.1/func_b.3

This following APIs are provided by FPGA device framework:
* fpga_dev_create
  Create fpga device under the given parent device.
* fpga_dev_destroy
  Destroy fpga device

The following sysfs files are created:
* /sys/class/fpga/<fpga.x>/name
  Name of the fpga device.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
 drivers/fpga/Kconfig          |   6 +++
 drivers/fpga/Makefile         |   3 ++
 drivers/fpga/fpga-dev.c       | 120 ++++++++++++++++++++++++++++++++++++++++++
 include/linux/fpga/fpga-dev.h |  34 ++++++++++++
 4 files changed, 163 insertions(+)
 create mode 100644 drivers/fpga/fpga-dev.c
 create mode 100644 include/linux/fpga/fpga-dev.h

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index ce861a2..d99b640 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig
@@ -12,6 +12,12 @@ config FPGA
 	  manager drivers.
 
 if FPGA
+config FPGA_DEVICE
+	tristate "FPGA Device Framework"
+	help
+	  Say Y here if you want support for FPGA Devices from the kernel.
+	  The FPGA Device Framework adds a FPGA device class and provide
+	  interfaces to create FPGA devices.
 
 config FPGA_REGION
 	tristate "FPGA Region"
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 8df07bc..53a41d2 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -5,6 +5,9 @@
 # Core FPGA Manager Framework
 obj-$(CONFIG_FPGA)			+= fpga-mgr.o
 
+# FPGA Device Framework
+obj-$(CONFIG_FPGA_DEVICE)		+= fpga-dev.o
+
 # FPGA Manager Drivers
 obj-$(CONFIG_FPGA_MGR_SOCFPGA)		+= socfpga.o
 obj-$(CONFIG_FPGA_MGR_SOCFPGA_A10)	+= socfpga-a10.o
diff --git a/drivers/fpga/fpga-dev.c b/drivers/fpga/fpga-dev.c
new file mode 100644
index 0000000..0f4c0ed
--- /dev/null
+++ b/drivers/fpga/fpga-dev.c
@@ -0,0 +1,120 @@
+/*
+ * FPGA Device Framework Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * This work is licensed under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license. See the
+ * LICENSE.BSD file under drivers/fpga/intel for the BSD license and see
+ * the COPYING file in the top-level directory for the GPLv2 license.
+ */
+#include <linux/device.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/fpga/fpga-dev.h>
+
+static DEFINE_IDA(fpga_dev_ida);
+static struct class *fpga_dev_class;
+
+static ssize_t name_show(struct device *dev,
+			 struct device_attribute *attr, char *buf)
+{
+	struct fpga_dev *fdev = to_fpga_dev(dev);
+
+	return sprintf(buf, "%s\n", fdev->name);
+}
+static DEVICE_ATTR_RO(name);
+
+static struct attribute *fpga_dev_attrs[] = {
+	&dev_attr_name.attr,
+	NULL,
+};
+ATTRIBUTE_GROUPS(fpga_dev);
+
+/**
+ * fpga_dev_create - create a fpga device
+ * @parent: parent device
+ * @name: fpga device name
+ *
+ * Return fpga_dev struct for success, error code otherwise.
+ */
+struct fpga_dev *fpga_dev_create(struct device *parent, const char *name)
+{
+	struct fpga_dev *fdev;
+	int id, ret = 0;
+
+	if (!name || !strlen(name)) {
+		dev_err(parent, "Attempt to register with no name!\n");
+		return ERR_PTR(-EINVAL);
+	}
+
+	fdev = kzalloc(sizeof(*fdev), GFP_KERNEL);
+	if (!fdev)
+		return ERR_PTR(-ENOMEM);
+
+	id = ida_simple_get(&fpga_dev_ida, 0, 0, GFP_KERNEL);
+	if (id < 0) {
+		ret = id;
+		goto error_kfree;
+	}
+
+	fdev->name = name;
+
+	device_initialize(&fdev->dev);
+	fdev->dev.class = fpga_dev_class;
+	fdev->dev.parent = parent;
+	fdev->dev.id = id;
+
+	ret = dev_set_name(&fdev->dev, "fpga.%d", id);
+	if (ret)
+		goto error_device;
+
+	ret = device_add(&fdev->dev);
+	if (ret)
+		goto error_device;
+
+	dev_dbg(fdev->dev.parent, "fpga device [%s] created\n", fdev->name);
+
+	return fdev;
+
+error_device:
+	ida_simple_remove(&fpga_dev_ida, id);
+error_kfree:
+	kfree(fdev);
+
+	return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(fpga_dev_create);
+
+static void fpga_dev_release(struct device *dev)
+{
+	struct fpga_dev *fdev = to_fpga_dev(dev);
+
+	ida_simple_remove(&fpga_dev_ida, fdev->dev.id);
+	kfree(fdev);
+}
+
+static int __init fpga_dev_class_init(void)
+{
+	pr_info("FPGA Device framework\n");
+
+	fpga_dev_class = class_create(THIS_MODULE, "fpga");
+	if (IS_ERR(fpga_dev_class))
+		return PTR_ERR(fpga_dev_class);
+
+	fpga_dev_class->dev_groups = fpga_dev_groups;
+	fpga_dev_class->dev_release = fpga_dev_release;
+
+	return 0;
+}
+
+static void __exit fpga_dev_class_exit(void)
+{
+	class_destroy(fpga_dev_class);
+}
+
+MODULE_DESCRIPTION("FPGA Device framework");
+MODULE_LICENSE("Dual BSD/GPL");
+
+subsys_initcall(fpga_dev_class_init);
+module_exit(fpga_dev_class_exit);
diff --git a/include/linux/fpga/fpga-dev.h b/include/linux/fpga/fpga-dev.h
new file mode 100644
index 0000000..7b58356
--- /dev/null
+++ b/include/linux/fpga/fpga-dev.h
@@ -0,0 +1,34 @@
+/*
+ * FPGA Device Driver Header
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * This work is licensed under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license. See the
+ * LICENSE.BSD file under drivers/fpga/intel for the BSD license and see
+ * the COPYING file in the top-level directory for the GPLv2 license.
+ *
+ */
+#ifndef _LINUX_FPGA_DEV_H
+#define _LINUX_FPGA_DEV_H
+
+/**
+ * struct fpga_dev - fpga device structure
+ * @name: name of fpga device
+ * @dev: fpga device
+ */
+struct fpga_dev {
+	const char *name;
+	struct device dev;
+};
+
+#define to_fpga_dev(d) container_of(d, struct fpga_dev, dev)
+
+struct fpga_dev *fpga_dev_create(struct device *parent, const char *name);
+
+static inline void fpga_dev_destroy(struct fpga_dev *fdev)
+{
+	device_unregister(&fdev->dev);
+}
+
+#endif
-- 
2.7.4

^ permalink raw reply	[flat|nested] 93+ messages in thread

* [PATCH 03/16] fpga: intel: add FPGA PCIe device driver
  2017-03-30 12:08 [PATCH 00/16] Intel FPGA Device Drivers Wu Hao
  2017-03-30 12:08 ` [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview Wu Hao
  2017-03-30 12:08 ` [PATCH 02/16] fpga: add FPGA device framework Wu Hao
@ 2017-03-30 12:08 ` Wu Hao
  2017-04-04  2:10   ` Moritz Fischer
  2017-03-30 12:08 ` [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features Wu Hao
                   ` (14 subsequent siblings)
  17 siblings, 1 reply; 93+ messages in thread
From: Wu Hao @ 2017-03-30 12:08 UTC (permalink / raw)
  To: atull, moritz.fischer, linux-fpga, linux-kernel
  Cc: luwei.kang, yi.z.zhang, hao.wu, Tim Whisonant, Enno Luebbers,
	Shiva Rao, Christopher Rauer, Xiao Guangrong

From: Zhang Yi <yi.z.zhang@intel.com>

The Intel FPGA device appears as a PCIe device on the system. This patch
implements the basic framework of the driver for Intel PCIe device which
locates between CPU and Accelerated Function Units (AFUs).

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
 drivers/fpga/Kconfig           |   2 +
 drivers/fpga/Makefile          |   3 +
 drivers/fpga/intel/Kconfig     |  27 +++++++++
 drivers/fpga/intel/LICENSE.BSD |  24 ++++++++
 drivers/fpga/intel/Makefile    |   3 +
 drivers/fpga/intel/pcie.c      | 129 +++++++++++++++++++++++++++++++++++++++++
 6 files changed, 188 insertions(+)
 create mode 100644 drivers/fpga/intel/Kconfig
 create mode 100644 drivers/fpga/intel/LICENSE.BSD
 create mode 100644 drivers/fpga/intel/Makefile
 create mode 100644 drivers/fpga/intel/pcie.c

diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index d99b640..4e49aee 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig
@@ -69,6 +69,8 @@ config ALTERA_FREEZE_BRIDGE
 	  isolate one region of the FPGA from the busses while that
 	  region is being reprogrammed.
 
+source "drivers/fpga/intel/Kconfig"
+
 endif # FPGA
 
 endmenu
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 53a41d2..46f1a5d 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -20,3 +20,6 @@ obj-$(CONFIG_ALTERA_FREEZE_BRIDGE)	+= altera-freeze-bridge.o
 
 # High Level Interfaces
 obj-$(CONFIG_FPGA_REGION)		+= fpga-region.o
+
+# Intel FPGA Support
+obj-$(CONFIG_INTEL_FPGA)		+= intel/
diff --git a/drivers/fpga/intel/Kconfig b/drivers/fpga/intel/Kconfig
new file mode 100644
index 0000000..bf402f3
--- /dev/null
+++ b/drivers/fpga/intel/Kconfig
@@ -0,0 +1,27 @@
+menuconfig INTEL_FPGA
+	tristate "Intel(R) FPGA support"
+	depends on FPGA_DEVICE
+	help
+	  Select this option to enable driver support for Intel(R)
+	  Field-Programmable Gate Array (FPGA) solutions. This driver
+	  provides interfaces for userspace applications to configure,
+	  enumerate, open, and access FPGA accelerators on platforms
+	  equipped with Intel(R) FPGA solutions and enables system
+	  level management functions such as FPGA reconfiguration,
+	  power management, and virtualization.
+
+	  Say Y if your platform has this technology. Say N if unsure.
+
+if INTEL_FPGA
+
+config INTEL_FPGA_PCI
+	tristate "Intel FPGA PCIe Driver"
+	depends on PCI
+	help
+	  This is the driver for the PCIe device which locates between
+	  CPU and Accelerated Function Units (AFUs) and allows them to
+	  communicate with each other.
+
+	  To compile this as a module, choose M here.
+
+endif
diff --git a/drivers/fpga/intel/LICENSE.BSD b/drivers/fpga/intel/LICENSE.BSD
new file mode 100644
index 0000000..309d2b7
--- /dev/null
+++ b/drivers/fpga/intel/LICENSE.BSD
@@ -0,0 +1,24 @@
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+  * Redistributions of source code must retain the above copyright
+    notice, this list of conditions and the following disclaimer.
+  * Redistributions in binary form must reproduce the above copyright
+    notice, this list of conditions and the following disclaimer in
+    the documentation and/or other materials provided with the
+    distribution.
+  * Neither the name of Intel Corporation nor the names of its
+    contributors may be used to endorse or promote products derived
+    from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
new file mode 100644
index 0000000..61fd8ea
--- /dev/null
+++ b/drivers/fpga/intel/Makefile
@@ -0,0 +1,3 @@
+obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
+
+intel-fpga-pci-objs := pcie.o
diff --git a/drivers/fpga/intel/pcie.c b/drivers/fpga/intel/pcie.c
new file mode 100644
index 0000000..132d9da
--- /dev/null
+++ b/drivers/fpga/intel/pcie.c
@@ -0,0 +1,129 @@
+/*
+ * Driver for Intel FPGA PCIe device
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Zhang Yi <Yi.Z.Zhang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ *
+ * This work is licensed under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license. See the
+ * LICENSE.BSD file under this directory for the BSD license and see
+ * the COPYING file in the top-level directory for the GPLv2 license.
+ */
+
+#include <linux/pci.h>
+#include <linux/types.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/stddef.h>
+#include <linux/errno.h>
+#include <linux/aer.h>
+
+#define DRV_VERSION	"EXPERIMENTAL VERSION"
+#define DRV_NAME	"intel-fpga-pci"
+
+/* PCI Device ID */
+#define PCIe_DEVICE_ID_PF_INT_5_X	0xBCBD
+#define PCIe_DEVICE_ID_PF_INT_6_X	0xBCC0
+#define PCIe_DEVICE_ID_PF_DSC_1_X	0x09C4
+/* VF Device */
+#define PCIe_DEVICE_ID_VF_INT_5_X	0xBCBF
+#define PCIe_DEVICE_ID_VF_INT_6_X	0xBCC1
+#define PCIe_DEVICE_ID_VF_DSC_1_X	0x09C5
+
+static struct pci_device_id cci_pcie_id_tbl[] = {
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_PF_INT_5_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_VF_INT_5_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_PF_INT_6_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_VF_INT_6_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_PF_DSC_1_X),},
+	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_VF_DSC_1_X),},
+	{0,}
+};
+MODULE_DEVICE_TABLE(pci, cci_pcie_id_tbl);
+
+static
+int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
+{
+	int ret;
+
+	ret = pci_enable_device(pcidev);
+	if (ret < 0) {
+		dev_err(&pcidev->dev, "Failed to enable device %d.\n", ret);
+		goto exit;
+	}
+
+	ret = pci_enable_pcie_error_reporting(pcidev);
+	if (ret && ret != -EINVAL)
+		dev_info(&pcidev->dev, "PCIE AER unavailable %d.\n", ret);
+
+	ret = pci_request_regions(pcidev, DRV_NAME);
+	if (ret) {
+		dev_err(&pcidev->dev, "Failed to request regions.\n");
+		goto disable_error_report_exit;
+	}
+
+	pci_set_master(pcidev);
+	pci_save_state(pcidev);
+
+	if (!dma_set_mask(&pcidev->dev, DMA_BIT_MASK(64))) {
+		dma_set_coherent_mask(&pcidev->dev, DMA_BIT_MASK(64));
+	} else if (!dma_set_mask(&pcidev->dev, DMA_BIT_MASK(32))) {
+		dma_set_coherent_mask(&pcidev->dev, DMA_BIT_MASK(32));
+	} else {
+		ret = -EIO;
+		dev_err(&pcidev->dev, "No suitable DMA support available.\n");
+		goto release_region_exit;
+	}
+
+	/* TODO: create and add the platform device per feature list */
+	return 0;
+
+release_region_exit:
+	pci_release_regions(pcidev);
+disable_error_report_exit:
+	pci_disable_pcie_error_reporting(pcidev);
+	pci_disable_device(pcidev);
+exit:
+	return ret;
+}
+
+static void cci_pci_remove(struct pci_dev *pcidev)
+{
+	pci_release_regions(pcidev);
+	pci_disable_pcie_error_reporting(pcidev);
+	pci_disable_device(pcidev);
+}
+
+static struct pci_driver cci_pci_driver = {
+	.name = DRV_NAME,
+	.id_table = cci_pcie_id_tbl,
+	.probe = cci_pci_probe,
+	.remove = cci_pci_remove,
+};
+
+static int __init ccidrv_init(void)
+{
+	pr_info("Intel(R) FPGA PCIe Driver: Version %s\n", DRV_VERSION);
+
+	return pci_register_driver(&cci_pci_driver);
+}
+
+static void __exit ccidrv_exit(void)
+{
+	pci_unregister_driver(&cci_pci_driver);
+}
+
+module_init(ccidrv_init);
+module_exit(ccidrv_exit);
+
+MODULE_DESCRIPTION("Intel FPGA PCIe Device Driver");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("Dual BSD/GPL");
-- 
2.7.4

^ permalink raw reply	[flat|nested] 93+ messages in thread

* [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features.
  2017-03-30 12:08 [PATCH 00/16] Intel FPGA Device Drivers Wu Hao
                   ` (2 preceding siblings ...)
  2017-03-30 12:08 ` [PATCH 03/16] fpga: intel: add FPGA PCIe device driver Wu Hao
@ 2017-03-30 12:08 ` Wu Hao
  2017-04-03 21:44   ` Alan Tull
                     ` (3 more replies)
  2017-03-30 12:08 ` [PATCH 05/16] fpga: intel: pcie: add chardev support for feature devices Wu Hao
                   ` (13 subsequent siblings)
  17 siblings, 4 replies; 93+ messages in thread
From: Wu Hao @ 2017-03-30 12:08 UTC (permalink / raw)
  To: atull, moritz.fischer, linux-fpga, linux-kernel
  Cc: luwei.kang, yi.z.zhang, hao.wu, Xiao Guangrong, Tim Whisonant,
	Enno Luebbers, Shiva Rao, Christopher Rauer

From: Xiao Guangrong <guangrong.xiao@linux.intel.com>

Device Featuer List structure creates a link list of feature headers
within the MMIO space to provide an extensiable way of adding features.

The Intel FPGA PCIe driver walks through the feature headers to enumerate
feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
Function Unit (AFU), and their private sub features. For feature devices,
it creates the platform devices and linked the private sub features into
their platform data.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Kang Luwei <luwei.kang@intel.com>
Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
 drivers/fpga/intel/Makefile      |   2 +-
 drivers/fpga/intel/feature-dev.c | 139 +++++++
 drivers/fpga/intel/feature-dev.h | 342 ++++++++++++++++
 drivers/fpga/intel/pcie.c        | 841 ++++++++++++++++++++++++++++++++++++++-
 4 files changed, 1321 insertions(+), 3 deletions(-)
 create mode 100644 drivers/fpga/intel/feature-dev.c
 create mode 100644 drivers/fpga/intel/feature-dev.h

diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
index 61fd8ea..c029940 100644
--- a/drivers/fpga/intel/Makefile
+++ b/drivers/fpga/intel/Makefile
@@ -1,3 +1,3 @@
 obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
 
-intel-fpga-pci-objs := pcie.o
+intel-fpga-pci-objs := pcie.o feature-dev.o
diff --git a/drivers/fpga/intel/feature-dev.c b/drivers/fpga/intel/feature-dev.c
new file mode 100644
index 0000000..6952566
--- /dev/null
+++ b/drivers/fpga/intel/feature-dev.c
@@ -0,0 +1,139 @@
+/*
+ * Intel FPGA Feature Device Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Zhang Yi <yi.z.zhang@intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *
+ * This work is licensed under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license. See the
+ * LICENSE.BSD file under this directory for the BSD license and see
+ * the COPYING file in the top-level directory for the GPLv2 license.
+ */
+
+#include "feature-dev.h"
+
+void feature_platform_data_add(struct feature_platform_data *pdata,
+			       int index, const char *name,
+			       int resource_index, void __iomem *ioaddr)
+{
+	WARN_ON(index >= pdata->num);
+
+	pdata->features[index].name = name;
+	pdata->features[index].resource_index = resource_index;
+	pdata->features[index].ioaddr = ioaddr;
+}
+
+int feature_platform_data_size(int num)
+{
+	return sizeof(struct feature_platform_data) +
+		num * sizeof(struct feature);
+}
+
+struct feature_platform_data *
+feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
+{
+	struct feature_platform_data *pdata;
+
+	pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
+	if (pdata) {
+		pdata->dev = dev;
+		pdata->num = num;
+		mutex_init(&pdata->lock);
+	}
+
+	return pdata;
+}
+
+int fme_feature_num(void)
+{
+	return FME_FEATURE_ID_MAX;
+}
+
+int port_feature_num(void)
+{
+	return PORT_FEATURE_ID_MAX;
+}
+
+int fpga_port_id(struct platform_device *pdev)
+{
+	struct feature_port_header *port_hdr;
+	struct feature_port_capability capability;
+
+	port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
+					       PORT_FEATURE_ID_HEADER);
+	WARN_ON(!port_hdr);
+
+	capability.csr = readq(&port_hdr->capability);
+	return capability.port_number;
+}
+EXPORT_SYMBOL_GPL(fpga_port_id);
+
+/*
+ * Enable Port by clear the port soft reset bit, which is set by default.
+ * The User AFU is unable to respond to any MMIO access while in reset.
+ * __fpga_port_enable function should only be used after __fpga_port_disable
+ * function.
+ */
+void __fpga_port_enable(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct feature_port_header *port_hdr;
+	struct feature_port_control control;
+
+	WARN_ON(!pdata->disable_count);
+
+	if (--pdata->disable_count != 0)
+		return;
+
+	port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
+					       PORT_FEATURE_ID_HEADER);
+	WARN_ON(!port_hdr);
+
+	control.csr = readq(&port_hdr->control);
+	control.port_sftrst = 0x0;
+	writeq(control.csr, &port_hdr->control);
+}
+EXPORT_SYMBOL_GPL(__fpga_port_enable);
+
+#define RST_POLL_INVL 10 /* us */
+#define RST_POLL_TIMEOUT 1000 /* us */
+
+int __fpga_port_disable(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct feature_port_header *port_hdr;
+	struct feature_port_control control;
+
+	if (pdata->disable_count++ != 0)
+		return 0;
+
+	port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
+					       PORT_FEATURE_ID_HEADER);
+	WARN_ON(!port_hdr);
+
+	/* Set port soft reset */
+	control.csr = readq(&port_hdr->control);
+	control.port_sftrst = 0x1;
+	writeq(control.csr, &port_hdr->control);
+
+	/*
+	 * HW sets ack bit to 1 when all outstanding requests have been drained
+	 * on this port and minimum soft reset pulse width has elapsed.
+	 * Driver polls port_soft_reset_ack to determine if reset done by HW.
+	 */
+	control.port_sftrst_ack = 1;
+
+	if (fpga_wait_register_field(port_sftrst_ack, control,
+		&port_hdr->control, RST_POLL_TIMEOUT, RST_POLL_INVL)) {
+		dev_err(&pdev->dev, "timeout, fail to reset device\n");
+		return -ETIMEDOUT;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(__fpga_port_disable);
diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
new file mode 100644
index 0000000..a1e6e7d
--- /dev/null
+++ b/drivers/fpga/intel/feature-dev.h
@@ -0,0 +1,342 @@
+/*
+ * Intel FPGA Feature Device Driver Header File
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Zhang Yi <yi.z.zhang@intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *
+ * This work is licensed under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license. See the
+ * LICENSE.BSD file under this directory for the BSD license and see
+ * the COPYING file in the top-level directory for the GPLv2 license.
+ */
+
+#ifndef __INTEL_FPGA_FEATURE_H
+#define __INTEL_FPGA_FEATURE_H
+
+#include <linux/fs.h>
+#include <linux/pci.h>
+#include <linux/uuid.h>
+#include <linux/delay.h>
+#include <linux/platform_device.h>
+
+/* maximum supported number of ports */
+#define MAX_FPGA_PORT_NUM 4
+/* plus one for fme device */
+#define MAX_FEATURE_DEV_NUM	(MAX_FPGA_PORT_NUM + 1)
+
+#define FME_FEATURE_HEADER          "fme_hdr"
+#define FME_FEATURE_THERMAL_MGMT    "fme_thermal"
+#define FME_FEATURE_POWER_MGMT      "fme_power"
+#define FME_FEATURE_GLOBAL_PERF     "fme_gperf"
+#define FME_FEATURE_GLOBAL_ERR      "fme_error"
+#define FME_FEATURE_PR_MGMT         "fme_pr"
+
+#define PORT_FEATURE_HEADER         "port_hdr"
+#define PORT_FEATURE_UAFU           "port_uafu"
+#define PORT_FEATURE_ERR            "port_err"
+#define PORT_FEATURE_UMSG           "port_umsg"
+#define PORT_FEATURE_PR             "port_pr"
+#define PORT_FEATURE_STP            "port_stp"
+
+/* All headers and structures must be byte-packed to match the spec. */
+#pragma pack(1)
+
+/* common header for all features */
+struct feature_header {
+	union {
+		u64 csr;
+		struct {
+			u16 id:12;
+			u8  revision:4;
+			u32 next_header_offset:24; /* offset to next header */
+			u32 rsvdz:20;
+			u8  type:4;		   /* feature type */
+#define FEATURE_TYPE_AFU		0x1
+#define FEATURE_TYPE_PRIVATE		0x3
+		};
+	};
+};
+
+/* common header for non-private features */
+struct feature_afu_header {
+	uuid_le guid;
+	union {
+		u64 csr;
+		struct {
+			u64 next_afu:24;	/* pointer to next afu header */
+			u64 rsvdz:40;
+		};
+	};
+};
+
+/* FME Header Register Set */
+/* FME Capability Register */
+struct feature_fme_capability {
+	union {
+		u64 csr;
+		struct {
+			u8  fabric_verid;	/* Fabric version ID */
+			u8  socket_id:1;	/* Socket id */
+			u8  rsvdz1:3;
+			u8  pcie0_link_avl:1;	/* PCIe0 link availability */
+			u8  pcie1_link_avl:1;	/* PCIe1 link availability */
+			u8  coherent_link_avl:1;/* Coherent link availability */
+			u8  rsvdz2:1;
+			u8  iommu_support:1;	/* IOMMU or VT-d supported */
+			u8  num_ports:3;	/* Num of ports implemented */
+			u8  rsvdz3:4;
+			u8  addr_width_bits:6;	/* Address width supported */
+			u8  rsvdz4:2;
+			u16 cache_size:12;	/* Cache size in kb */
+			u8  cache_assoc:4;	/* Cache Associativity */
+			u16 rsvdz5:15;
+			u8  lock_bit:1;		/* Latched lock bit by BIOS */
+		};
+	};
+};
+
+/* FME Port Offset Register */
+struct feature_fme_port {
+	union {
+		u64 csr;
+		struct {
+			u32 port_offset:24;	/* Offset to port header */
+			u8  rsvdz1;
+			u8  port_bar:3;		/* Bar id */
+			u32 rsvdz2:20;
+			u8  afu_access_ctrl:1;	/* AFU access type: PF/VF */
+			u8  rsvdz3:4;
+			u8  port_implemented:1;	/* Port implemented or not */
+			u8  rsvdz4:3;
+		};
+	};
+};
+
+struct feature_fme_header {
+	struct feature_header header;
+	struct feature_afu_header afu_header;
+	u64 rsvd[2];
+	struct feature_fme_capability capability;
+	struct feature_fme_port port[MAX_FPGA_PORT_NUM];
+};
+
+/* FME Thermal Sub Feature Register Set */
+struct feature_fme_thermal {
+	struct feature_header header;
+};
+
+/* FME Power Sub Feature Register Set */
+struct feature_fme_power {
+	struct feature_header header;
+};
+
+/* FME Global Performance Sub Feature Register Set */
+struct feature_fme_gperf {
+	struct feature_header header;
+};
+
+/* FME Error Sub Feature Register Set */
+struct feature_fme_err {
+	struct feature_header header;
+};
+
+/* FME Partial Reconfiguration Sub Feature Register Set */
+struct feature_fme_pr {
+	struct feature_header header;
+};
+
+/* PORT Header Register Set */
+/* Port Capability Register */
+struct feature_port_capability {
+	union {
+		u64 csr;
+		struct {
+			u8  port_number:2;	/* Port Number 0-3 */
+			u8  rsvdz1:6;
+			u16 mmio_size;		/* User MMIO size in KB */
+			u8  rsvdz2;
+			u8  sp_intr_num:4;	/* Supported interrupts num */
+			u32 rsvdz3:28;
+		};
+	};
+};
+
+/* Port Control Register */
+struct feature_port_control {
+	union {
+		u64 csr;
+		struct {
+			u8  port_sftrst:1;	/* Port Soft Reset */
+			u8  rsvdz1:1;
+			u8  latency_tolerance:1;/* '1' >= 40us, '0' < 40us */
+			u8  rsvdz2:1;
+			u8  port_sftrst_ack:1;	/* HW ACK for Soft Reset */
+			u64 rsvdz3:59;
+		};
+	};
+};
+
+struct feature_port_header {
+	struct feature_header header;
+	struct feature_afu_header afu_header;
+	u64 rsvd[2];
+	struct feature_port_capability capability;
+	struct feature_port_control control;
+};
+
+/* PORT Error Sub Feature Register Set */
+struct feature_port_error {
+	struct feature_header header;
+};
+
+/* PORT Unordered Message Sub Feature Register Set */
+struct feature_port_umsg {
+	struct feature_header header;
+};
+
+/* PORT SignalTap Sub Feature Register Set */
+struct feature_port_stp {
+	struct feature_header header;
+};
+
+#pragma pack()
+
+struct feature {
+	const char *name;
+	int resource_index;
+	void __iomem *ioaddr;
+};
+
+struct feature_platform_data {
+	/* list the feature dev to cci_drvdata->port_dev_list. */
+	struct list_head node;
+	struct mutex lock;
+	struct platform_device *dev;
+	unsigned int disable_count;	/* count for port disable */
+
+	int num;			/* number of features */
+	struct feature features[0];
+};
+
+enum fme_feature_id {
+	FME_FEATURE_ID_HEADER = 0x0,
+	FME_FEATURE_ID_THERMAL_MGMT = 0x1,
+	FME_FEATURE_ID_POWER_MGMT = 0x2,
+	FME_FEATURE_ID_GLOBAL_PERF = 0x3,
+	FME_FEATURE_ID_GLOBAL_ERR = 0x4,
+	FME_FEATURE_ID_PR_MGMT = 0x5,
+	FME_FEATURE_ID_MAX = 0x6,
+};
+
+enum port_feature_id {
+	PORT_FEATURE_ID_HEADER = 0x0,
+	PORT_FEATURE_ID_ERROR = 0x1,
+	PORT_FEATURE_ID_UMSG = 0x2,
+	PORT_FEATURE_ID_PR = 0x3,
+	PORT_FEATURE_ID_STP = 0x4,
+	PORT_FEATURE_ID_UAFU = 0x5,
+	PORT_FEATURE_ID_MAX = 0x6,
+};
+
+int fme_feature_num(void);
+int port_feature_num(void);
+
+#define FPGA_FEATURE_DEV_FME		"intel-fpga-fme"
+#define FPGA_FEATURE_DEV_PORT		"intel-fpga-port"
+
+void feature_platform_data_add(struct feature_platform_data *pdata,
+			       int index, const char *name,
+			       int resource_index, void __iomem *ioaddr);
+int feature_platform_data_size(int num);
+struct feature_platform_data *
+feature_platform_data_alloc_and_init(struct platform_device *dev, int num);
+
+int fpga_port_id(struct platform_device *pdev);
+
+static inline int fpga_port_check_id(struct platform_device *pdev,
+				     void *pport_id)
+{
+	return fpga_port_id(pdev) == *(int *)pport_id;
+}
+
+void __fpga_port_enable(struct platform_device *pdev);
+int __fpga_port_disable(struct platform_device *pdev);
+
+static inline void fpga_port_enable(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	mutex_lock(&pdata->lock);
+	__fpga_port_enable(pdev);
+	mutex_unlock(&pdata->lock);
+}
+
+static inline int fpga_port_disable(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	int ret;
+
+	mutex_lock(&pdata->lock);
+	ret = __fpga_port_disable(pdev);
+	mutex_unlock(&pdata->lock);
+
+	return ret;
+}
+
+static inline int __fpga_port_reset(struct platform_device *pdev)
+{
+	int ret;
+
+	ret = __fpga_port_disable(pdev);
+	if (ret)
+		return ret;
+
+	__fpga_port_enable(pdev);
+	return 0;
+}
+
+static inline int fpga_port_reset(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	int ret;
+
+	mutex_lock(&pdata->lock);
+	ret = __fpga_port_reset(pdev);
+	mutex_unlock(&pdata->lock);
+	return ret;
+}
+
+static inline void __iomem *
+get_feature_ioaddr_by_index(struct device *dev, int index)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(dev);
+
+	return pdata->features[index].ioaddr;
+}
+
+/*
+ * Wait register's _field to be changed to the given value (_expect's _field)
+ * by polling with given interval and timeout.
+ */
+#define fpga_wait_register_field(_field, _expect, _reg_addr, _timeout, _invl)\
+({									     \
+	int wait = 0;							     \
+	int ret = -ETIMEDOUT;						     \
+	typeof(_expect) value;						     \
+	for (; wait <= _timeout; wait += _invl) {			     \
+		value.csr = readq(_reg_addr);				     \
+		if (_expect._field == value._field) {			     \
+			ret = 0;					     \
+			break;						     \
+		}							     \
+		udelay(_invl);						     \
+	}								     \
+	ret;								     \
+})
+
+#endif
diff --git a/drivers/fpga/intel/pcie.c b/drivers/fpga/intel/pcie.c
index 132d9da..28df63e 100644
--- a/drivers/fpga/intel/pcie.c
+++ b/drivers/fpga/intel/pcie.c
@@ -25,10 +25,827 @@
 #include <linux/stddef.h>
 #include <linux/errno.h>
 #include <linux/aer.h>
+#include <linux/fpga/fpga-dev.h>
+
+#include "feature-dev.h"
 
 #define DRV_VERSION	"EXPERIMENTAL VERSION"
 #define DRV_NAME	"intel-fpga-pci"
 
+#define INTEL_FPGA_DEV	"intel-fpga-dev"
+
+static DEFINE_MUTEX(fpga_id_mutex);
+
+enum fpga_id_type {
+	FME_ID,		/* fme id allocation and mapping */
+	PORT_ID,	/* port id allocation and mapping */
+	FPGA_ID_MAX,
+};
+
+/* it is protected by fpga_id_mutex */
+static struct idr fpga_ids[FPGA_ID_MAX];
+
+struct cci_drvdata {
+	struct device *fme_dev;
+
+	struct mutex lock;
+	struct list_head port_dev_list;
+
+	struct list_head regions; /* global list of pci bar mapping region */
+};
+
+/* pci bar mapping info */
+struct cci_pci_region {
+	int bar;
+	void __iomem *ioaddr;	/* pointer to mapped bar region */
+	struct list_head node;
+};
+
+static void fpga_ids_init(void)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
+		idr_init(fpga_ids + i);
+}
+
+static void fpga_ids_destroy(void)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
+		idr_destroy(fpga_ids + i);
+}
+
+static int alloc_fpga_id(enum fpga_id_type type, struct device *dev)
+{
+	int id;
+
+	WARN_ON(type >= FPGA_ID_MAX);
+	mutex_lock(&fpga_id_mutex);
+	id = idr_alloc(fpga_ids + type, dev, 0, 0, GFP_KERNEL);
+	mutex_unlock(&fpga_id_mutex);
+	return id;
+}
+
+static void free_fpga_id(enum fpga_id_type type, int id)
+{
+	WARN_ON(type >= FPGA_ID_MAX);
+	mutex_lock(&fpga_id_mutex);
+	idr_remove(fpga_ids + type, id);
+	mutex_unlock(&fpga_id_mutex);
+}
+
+static void cci_pci_add_port_dev(struct pci_dev *pdev,
+				 struct platform_device *port_dev)
+{
+	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
+	struct feature_platform_data *pdata = dev_get_platdata(&port_dev->dev);
+
+	mutex_lock(&drvdata->lock);
+	list_add(&pdata->node, &drvdata->port_dev_list);
+	get_device(&pdata->dev->dev);
+	mutex_unlock(&drvdata->lock);
+}
+
+static void cci_pci_remove_port_devs(struct pci_dev *pdev)
+{
+	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
+	struct feature_platform_data *pdata, *ptmp;
+
+	mutex_lock(&drvdata->lock);
+	list_for_each_entry_safe(pdata, ptmp, &drvdata->port_dev_list, node) {
+		struct platform_device *port_dev = pdata->dev;
+
+		/* the port should be unregistered first. */
+		WARN_ON(device_is_registered(&port_dev->dev));
+		list_del(&pdata->node);
+		free_fpga_id(PORT_ID, port_dev->id);
+		put_device(&port_dev->dev);
+	}
+	mutex_unlock(&drvdata->lock);
+}
+
+/* info collection during feature dev build. */
+struct build_feature_devs_info {
+	struct pci_dev *pdev;
+
+	/*
+	 * PCI BAR mapping info. Parsing feature list starts from
+	 * BAR 0 and switch to different BARs to parse Port
+	 */
+	void __iomem *ioaddr;
+	void __iomem *ioend;
+	int current_bar;
+
+	/* points to FME header where the port offset is figured out. */
+	void __iomem *pfme_hdr;
+
+	/* the container device for all feature devices */
+	struct fpga_dev *parent_dev;
+
+	/* current feature device */
+	struct platform_device *feature_dev;
+};
+
+static void cci_pci_release_regions(struct pci_dev *pdev)
+{
+	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
+	struct cci_pci_region *tmp, *region;
+
+	list_for_each_entry_safe(region, tmp, &drvdata->regions, node) {
+		list_del(&region->node);
+		if (region->ioaddr)
+			pci_iounmap(pdev, region->ioaddr);
+		devm_kfree(&pdev->dev, region);
+	}
+}
+
+static void __iomem *cci_pci_ioremap_bar(struct pci_dev *pdev, int bar)
+{
+	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
+	struct cci_pci_region *region;
+
+	list_for_each_entry(region, &drvdata->regions, node)
+		if (region->bar == bar) {
+			dev_dbg(&pdev->dev, "BAR %d region exists\n", bar);
+			return region->ioaddr;
+		}
+
+	region = devm_kzalloc(&pdev->dev, sizeof(*region), GFP_KERNEL);
+	if (!region)
+		return NULL;
+
+	region->bar = bar;
+	region->ioaddr = pci_ioremap_bar(pdev, bar);
+	if (!region->ioaddr) {
+		dev_err(&pdev->dev, "can't ioremap memory from BAR %d.\n", bar);
+		devm_kfree(&pdev->dev, region);
+		return NULL;
+	}
+
+	list_add(&region->node, &drvdata->regions);
+	return region->ioaddr;
+}
+
+static int parse_start_from(struct build_feature_devs_info *binfo, int bar)
+{
+	binfo->ioaddr = cci_pci_ioremap_bar(binfo->pdev, bar);
+	if (!binfo->ioaddr)
+		return -ENOMEM;
+
+	binfo->current_bar = bar;
+	binfo->ioend = binfo->ioaddr + pci_resource_len(binfo->pdev, bar);
+	return 0;
+}
+
+static int parse_start(struct build_feature_devs_info *binfo)
+{
+	/* fpga feature list starts from BAR 0 */
+	return parse_start_from(binfo, 0);
+}
+
+/* switch the memory mapping to BAR# @bar */
+static int parse_switch_to(struct build_feature_devs_info *binfo, int bar)
+{
+	return parse_start_from(binfo, bar);
+}
+
+static struct build_feature_devs_info *
+build_info_alloc_and_init(struct pci_dev *pdev)
+{
+	struct build_feature_devs_info *binfo;
+
+	binfo = devm_kzalloc(&pdev->dev, sizeof(*binfo), GFP_KERNEL);
+	if (binfo)
+		binfo->pdev = pdev;
+
+	return binfo;
+}
+
+static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
+{
+	if (!strcmp(pdev->name, FPGA_FEATURE_DEV_FME))
+		return FME_ID;
+
+	if (!strcmp(pdev->name, FPGA_FEATURE_DEV_PORT))
+		return PORT_ID;
+
+	WARN_ON(1);
+	return FPGA_ID_MAX;
+}
+
+/*
+ * register current feature device, it is called when we need to switch to
+ * another feature parsing or we have parsed all features
+ */
+static int build_info_commit_dev(struct build_feature_devs_info *binfo)
+{
+	int ret;
+
+	if (!binfo->feature_dev)
+		return 0;
+
+	ret = platform_device_add(binfo->feature_dev);
+	if (!ret) {
+		struct cci_drvdata *drvdata;
+
+		drvdata = dev_get_drvdata(&binfo->pdev->dev);
+		if (feature_dev_id_type(binfo->feature_dev) == PORT_ID)
+			cci_pci_add_port_dev(binfo->pdev, binfo->feature_dev);
+		else
+			drvdata->fme_dev = get_device(&binfo->feature_dev->dev);
+
+		/*
+		 * reset it to avoid build_info_free() freeing their resource.
+		 *
+		 * The resource of successfully registered feature devices
+		 * will be freed by platform_device_unregister(). See the
+		 * comments in build_info_create_dev().
+		 */
+		binfo->feature_dev = NULL;
+	}
+
+	return ret;
+}
+
+static int
+build_info_create_dev(struct build_feature_devs_info *binfo,
+		      enum fpga_id_type type, int feature_nr, const char *name)
+{
+	struct platform_device *fdev;
+	struct resource *res;
+	struct feature_platform_data *pdata;
+	int ret;
+
+	/* we will create a new device, commit current device first */
+	ret = build_info_commit_dev(binfo);
+	if (ret)
+		return ret;
+
+	/*
+	 * we use -ENODEV as the initialization indicator which indicates
+	 * whether the id need to be reclaimed
+	 */
+	fdev = binfo->feature_dev = platform_device_alloc(name, -ENODEV);
+	if (!fdev)
+		return -ENOMEM;
+
+	fdev->id = alloc_fpga_id(type, &fdev->dev);
+	if (fdev->id < 0)
+		return fdev->id;
+
+	fdev->dev.parent = &binfo->parent_dev->dev;
+
+	/*
+	 * we need not care the memory which is associated with the
+	 * platform device. After call platform_device_unregister(),
+	 * it will be automatically freed by device's
+	 * release() callback, platform_device_release().
+	 */
+	pdata = feature_platform_data_alloc_and_init(fdev, feature_nr);
+	if (!pdata)
+		return -ENOMEM;
+
+	/*
+	 * the count should be initialized to 0 to make sure
+	 *__fpga_port_enable() following __fpga_port_disable()
+	 * works properly for port device.
+	 * and it should always be 0 for fme device.
+	 */
+	WARN_ON(pdata->disable_count);
+
+	fdev->dev.platform_data = pdata;
+	fdev->num_resources = feature_nr;
+	fdev->resource = kcalloc(feature_nr, sizeof(*res), GFP_KERNEL);
+	if (!fdev->resource)
+		return -ENOMEM;
+
+	return 0;
+}
+
+static int remove_feature_dev(struct device *dev, void *data)
+{
+	struct platform_device *pdev = to_platform_device(dev);
+
+	platform_device_unregister(pdev);
+	return 0;
+}
+
+static int remove_parent_dev(struct device *dev, void *data)
+{
+	/* remove platform devices attached in the parent device */
+	device_for_each_child(dev, NULL, remove_feature_dev);
+	fpga_dev_destroy(to_fpga_dev(dev));
+	return 0;
+}
+
+static void remove_all_devs(struct pci_dev *pdev)
+{
+	/* remove parent device and all its children. */
+	device_for_each_child(&pdev->dev, NULL, remove_parent_dev);
+}
+
+static void build_info_free(struct build_feature_devs_info *binfo)
+{
+	if (!IS_ERR_OR_NULL(binfo->parent_dev))
+		remove_all_devs(binfo->pdev);
+
+	/*
+	 * it is a valid id, free it. See comments in
+	 * build_info_create_dev()
+	 */
+	if (binfo->feature_dev && binfo->feature_dev->id >= 0)
+		free_fpga_id(feature_dev_id_type(binfo->feature_dev),
+			     binfo->feature_dev->id);
+
+	platform_device_put(binfo->feature_dev);
+
+	devm_kfree(&binfo->pdev->dev, binfo);
+}
+
+#define FEATURE_TYPE_AFU	0x1
+#define FEATURE_TYPE_PRIVATE	0x3
+
+/* FME and PORT GUID are fixed */
+#define FEATURE_FME_GUID "f9e17764-38f0-82fe-e346-524ae92aafbf"
+#define FEATURE_PORT_GUID "6b355b87-b06c-9642-eb42-8d139398b43a"
+
+static bool feature_is_fme(struct feature_afu_header *afu_hdr)
+{
+	uuid_le u;
+
+	uuid_le_to_bin(FEATURE_FME_GUID, &u);
+
+	return !uuid_le_cmp(u, afu_hdr->guid);
+}
+
+static bool feature_is_port(struct feature_afu_header *afu_hdr)
+{
+	uuid_le u;
+
+	uuid_le_to_bin(FEATURE_PORT_GUID, &u);
+
+	return !uuid_le_cmp(u, afu_hdr->guid);
+}
+
+/*
+ * UAFU GUID is dynamic as it can be changed after FME downloads different
+ * Green Bitstream to the port, so we treat the unknown GUIDs which are
+ * attached on port's feature list as UAFU.
+ */
+static bool feature_is_UAFU(struct build_feature_devs_info *binfo)
+{
+	if (!binfo->feature_dev ||
+	      feature_dev_id_type(binfo->feature_dev) != PORT_ID)
+		return false;
+
+	return true;
+}
+
+static void
+build_info_add_sub_feature(struct build_feature_devs_info *binfo,
+			   int feature_id, const char *feature_name,
+			   resource_size_t resource_size, void __iomem *start)
+{
+
+	struct platform_device *fdev = binfo->feature_dev;
+	struct feature_platform_data *pdata = dev_get_platdata(&fdev->dev);
+	struct resource *res = &fdev->resource[feature_id];
+
+	res->start = pci_resource_start(binfo->pdev, binfo->current_bar) +
+		start - binfo->ioaddr;
+	res->end = res->start + resource_size - 1;
+	res->flags = IORESOURCE_MEM;
+	res->name = feature_name;
+
+	feature_platform_data_add(pdata, feature_id,
+				  feature_name, feature_id, start);
+}
+
+struct feature_info {
+	const char *name;
+	resource_size_t resource_size;
+	int feature_index;
+};
+
+/* indexed by fme feature IDs which are defined in 'enum fme_feature_id'. */
+static struct feature_info fme_features[] = {
+	{
+		.name = FME_FEATURE_HEADER,
+		.resource_size = sizeof(struct feature_fme_header),
+		.feature_index = FME_FEATURE_ID_HEADER,
+	},
+	{
+		.name = FME_FEATURE_THERMAL_MGMT,
+		.resource_size = sizeof(struct feature_fme_thermal),
+		.feature_index = FME_FEATURE_ID_THERMAL_MGMT,
+	},
+	{
+		.name = FME_FEATURE_POWER_MGMT,
+		.resource_size = sizeof(struct feature_fme_power),
+		.feature_index = FME_FEATURE_ID_POWER_MGMT,
+	},
+	{
+		.name = FME_FEATURE_GLOBAL_PERF,
+		.resource_size = sizeof(struct feature_fme_gperf),
+		.feature_index = FME_FEATURE_ID_GLOBAL_PERF,
+	},
+	{
+		.name = FME_FEATURE_GLOBAL_ERR,
+		.resource_size = sizeof(struct feature_fme_err),
+		.feature_index = FME_FEATURE_ID_GLOBAL_ERR,
+	},
+	{
+		.name = FME_FEATURE_PR_MGMT,
+		.resource_size = sizeof(struct feature_fme_pr),
+		.feature_index = FME_FEATURE_ID_PR_MGMT,
+	}
+};
+
+/* indexed by port feature IDs which are defined in 'enum port_feature_id'. */
+static struct feature_info port_features[] = {
+	{
+		.name = PORT_FEATURE_HEADER,
+		.resource_size = sizeof(struct feature_port_header),
+		.feature_index = PORT_FEATURE_ID_HEADER,
+	},
+	{
+		.name = PORT_FEATURE_ERR,
+		.resource_size = sizeof(struct feature_port_error),
+		.feature_index = PORT_FEATURE_ID_ERROR,
+	},
+	{
+		.name = PORT_FEATURE_UMSG,
+		.resource_size = sizeof(struct feature_port_umsg),
+		.feature_index = PORT_FEATURE_ID_UMSG,
+	},
+	{
+		/* This feature isn't available for now */
+		.name = PORT_FEATURE_PR,
+		.resource_size = 0,
+		.feature_index = PORT_FEATURE_ID_PR,
+	},
+	{
+		.name = PORT_FEATURE_STP,
+		.resource_size = sizeof(struct feature_port_stp),
+		.feature_index = PORT_FEATURE_ID_STP,
+	},
+	{
+		/*
+		 * For User AFU feature, its region size is not fixed, but
+		 * reported by register PortCapability.mmio_size. Resource
+		 * size of UAFU will be set while parse port device.
+		 */
+		.name = PORT_FEATURE_UAFU,
+		.resource_size = 0,
+		.feature_index = PORT_FEATURE_ID_UAFU,
+	},
+};
+
+static int
+create_feature_instance(struct build_feature_devs_info *binfo,
+			void __iomem *start, struct feature_info *finfo)
+{
+	if (binfo->ioend - start < finfo->resource_size)
+		return -EINVAL;
+
+	build_info_add_sub_feature(binfo, finfo->feature_index, finfo->name,
+				   finfo->resource_size, start);
+	return 0;
+}
+
+static int parse_feature_fme(struct build_feature_devs_info *binfo,
+			     void __iomem *start)
+{
+	struct cci_drvdata *drvdata = dev_get_drvdata(&binfo->pdev->dev);
+	int ret;
+
+	ret = build_info_create_dev(binfo, FME_ID, fme_feature_num(),
+					FPGA_FEATURE_DEV_FME);
+	if (ret)
+		return ret;
+
+	if (drvdata->fme_dev) {
+		dev_err(&binfo->pdev->dev, "Multiple FMEs are detected.\n");
+		return -EINVAL;
+	}
+
+	return create_feature_instance(binfo, start,
+				       &fme_features[FME_FEATURE_ID_HEADER]);
+}
+
+static int parse_feature_fme_private(struct build_feature_devs_info *binfo,
+				     struct feature_header *hdr)
+{
+	struct feature_header header;
+
+	header.csr = readq(hdr);
+
+	if (header.id >= ARRAY_SIZE(fme_features)) {
+		dev_info(&binfo->pdev->dev, "FME feature id %x is not supported yet.\n",
+			 header.id);
+		return 0;
+	}
+
+	return create_feature_instance(binfo, hdr, &fme_features[header.id]);
+}
+
+static int parse_feature_port(struct build_feature_devs_info *binfo,
+			     void __iomem *start)
+{
+	int ret;
+
+	ret = build_info_create_dev(binfo, PORT_ID, port_feature_num(),
+					FPGA_FEATURE_DEV_PORT);
+	if (ret)
+		return ret;
+
+	return create_feature_instance(binfo, start,
+				       &port_features[PORT_FEATURE_ID_HEADER]);
+}
+
+static void enable_port_uafu(struct build_feature_devs_info *binfo,
+			     void __iomem *start)
+{
+	enum port_feature_id id = PORT_FEATURE_ID_UAFU;
+	struct feature_port_header *port_hdr;
+	struct feature_port_capability capability;
+
+	port_hdr = (struct feature_port_header *)start;
+	capability.csr = readq(&port_hdr->capability);
+	port_features[id].resource_size = capability.mmio_size << 10;
+
+	/*
+	 * To enable User AFU, driver needs to clear reset bit on related port,
+	 * otherwise the mmio space of this user AFU will be invalid.
+	 */
+	if (port_features[id].resource_size)
+		fpga_port_reset(binfo->feature_dev);
+}
+
+static int parse_feature_port_private(struct build_feature_devs_info *binfo,
+				      struct feature_header *hdr)
+{
+	struct feature_header header;
+	enum port_feature_id id;
+
+	header.csr = readq(hdr);
+	/*
+	 * the region of port feature id is [0x10, 0x13], + 1 to reserve 0
+	 * which is dedicated for port-hdr.
+	 */
+	id = (header.id & 0x000f) + 1;
+
+	if (id >= ARRAY_SIZE(port_features)) {
+		dev_info(&binfo->pdev->dev, "Port feature id %x is not supported yet.\n",
+			 header.id);
+		return 0;
+	}
+
+	return create_feature_instance(binfo, hdr, &port_features[id]);
+}
+
+static int parse_feature_port_uafu(struct build_feature_devs_info *binfo,
+				 struct feature_header *hdr)
+{
+	enum port_feature_id id = PORT_FEATURE_ID_UAFU;
+	int ret;
+
+	if (port_features[id].resource_size) {
+		ret = create_feature_instance(binfo, hdr, &port_features[id]);
+		port_features[id].resource_size = 0;
+	} else {
+		dev_err(&binfo->pdev->dev, "the uafu feature header is mis-configured.\n");
+		ret = -EINVAL;
+	}
+
+	return ret;
+}
+
+static int parse_feature_afus(struct build_feature_devs_info *binfo,
+			      struct feature_header *hdr)
+{
+	int ret;
+	struct feature_afu_header *afu_hdr, header;
+	void __iomem *start;
+	void __iomem *end = binfo->ioend;
+
+	start = hdr;
+	for (; start < end; start += header.next_afu) {
+		if (end - start < (sizeof(*afu_hdr) + sizeof(*hdr)))
+			return -EINVAL;
+
+		hdr = start;
+		afu_hdr = (struct feature_afu_header *) (hdr + 1);
+		header.csr = readq(&afu_hdr->csr);
+
+		if (feature_is_fme(afu_hdr)) {
+			ret = parse_feature_fme(binfo, hdr);
+			binfo->pfme_hdr = hdr;
+			if (ret)
+				return ret;
+		} else if (feature_is_port(afu_hdr)) {
+			ret = parse_feature_port(binfo, hdr);
+			enable_port_uafu(binfo, hdr);
+			if (ret)
+				return ret;
+		} else if (feature_is_UAFU(binfo)) {
+			ret = parse_feature_port_uafu(binfo, hdr);
+			if (ret)
+				return ret;
+		} else
+			dev_info(&binfo->pdev->dev, "AFU GUID %pUl is not supported yet.\n",
+				 afu_hdr->guid.b);
+
+		if (!header.next_afu)
+			break;
+	}
+
+	return 0;
+}
+
+static int parse_feature_private(struct build_feature_devs_info *binfo,
+				 struct feature_header *hdr)
+{
+	struct feature_header header;
+
+	header.csr = readq(hdr);
+
+	if (!binfo->feature_dev) {
+		dev_err(&binfo->pdev->dev, "the private feature %x does not belong to any AFU.\n",
+			header.id);
+		return -EINVAL;
+	}
+
+	switch (feature_dev_id_type(binfo->feature_dev)) {
+	case FME_ID:
+		return parse_feature_fme_private(binfo, hdr);
+	case PORT_ID:
+		return parse_feature_port_private(binfo, hdr);
+	default:
+		dev_info(&binfo->pdev->dev, "private feature %x belonging to AFU %s is not supported yet.\n",
+			 header.id, binfo->feature_dev->name);
+	}
+	return 0;
+}
+
+static int parse_feature(struct build_feature_devs_info *binfo,
+			 struct feature_header *hdr)
+{
+	struct feature_header header;
+	int ret = 0;
+
+	header.csr = readq(hdr);
+
+	switch (header.type) {
+	case FEATURE_TYPE_AFU:
+		ret = parse_feature_afus(binfo, hdr);
+		break;
+	case FEATURE_TYPE_PRIVATE:
+		ret = parse_feature_private(binfo, hdr);
+		break;
+	default:
+		dev_info(&binfo->pdev->dev,
+			 "Feature Type %x is not supported.\n", hdr->type);
+	};
+
+	return ret;
+}
+
+static int
+parse_feature_list(struct build_feature_devs_info *binfo, void __iomem *start)
+{
+	struct feature_header *hdr, header;
+	void __iomem *end = binfo->ioend;
+	int ret = 0;
+
+	for (; start < end; start += header.next_header_offset) {
+		if (end - start < sizeof(*hdr)) {
+			dev_err(&binfo->pdev->dev, "The region is too small to contain a feature.\n");
+			ret =  -EINVAL;
+			break;
+		}
+
+		hdr = (struct feature_header *)start;
+		ret = parse_feature(binfo, hdr);
+		if (ret)
+			break;
+
+		header.csr = readq(hdr);
+		if (!header.next_header_offset)
+			break;
+	}
+
+	return ret;
+}
+
+static int parse_ports_from_fme(struct build_feature_devs_info *binfo)
+{
+	struct feature_fme_header *fme_hdr;
+	struct feature_fme_port port;
+	int i = 0, ret = 0;
+
+	if (binfo->pfme_hdr == NULL) {
+		dev_dbg(&binfo->pdev->dev, "VF is detected.\n");
+		return ret;
+	}
+
+	fme_hdr = binfo->pfme_hdr;
+
+	do {
+		port.csr = readq(&fme_hdr->port[i]);
+		if (!port.port_implemented)
+			break;
+
+		ret = parse_switch_to(binfo, port.port_bar);
+		if (ret)
+			break;
+
+		ret = parse_feature_list(binfo,
+				binfo->ioaddr + port.port_offset);
+		if (ret)
+			break;
+	} while (++i < MAX_FPGA_PORT_NUM);
+
+	return ret;
+}
+
+static int create_init_drvdata(struct pci_dev *pdev)
+{
+	struct cci_drvdata *drvdata;
+
+	drvdata = devm_kzalloc(&pdev->dev, sizeof(*drvdata), GFP_KERNEL);
+	if (!drvdata)
+		return -ENOMEM;
+
+	mutex_init(&drvdata->lock);
+	INIT_LIST_HEAD(&drvdata->port_dev_list);
+	INIT_LIST_HEAD(&drvdata->regions);
+
+	dev_set_drvdata(&pdev->dev, drvdata);
+	return 0;
+}
+
+static void destroy_drvdata(struct pci_dev *pdev)
+{
+	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
+
+	if (drvdata->fme_dev) {
+		/* fme device should be unregistered first. */
+		WARN_ON(device_is_registered(drvdata->fme_dev));
+		free_fpga_id(FME_ID, to_platform_device(drvdata->fme_dev)->id);
+		put_device(drvdata->fme_dev);
+	}
+
+	cci_pci_remove_port_devs(pdev);
+	cci_pci_release_regions(pdev);
+	dev_set_drvdata(&pdev->dev, NULL);
+	devm_kfree(&pdev->dev, drvdata);
+}
+
+static int cci_pci_create_feature_devs(struct pci_dev *pdev)
+{
+	struct build_feature_devs_info *binfo;
+	int ret;
+
+	binfo = build_info_alloc_and_init(pdev);
+	if (!binfo)
+		return -ENOMEM;
+
+	binfo->parent_dev = fpga_dev_create(&pdev->dev, INTEL_FPGA_DEV);
+	if (IS_ERR(binfo->parent_dev)) {
+		ret = PTR_ERR(binfo->parent_dev);
+		goto free_binfo_exit;
+	}
+
+	ret = parse_start(binfo);
+	if (ret)
+		goto free_binfo_exit;
+
+	ret = parse_feature_list(binfo, binfo->ioaddr);
+	if (ret)
+		goto free_binfo_exit;
+
+	ret = parse_ports_from_fme(binfo);
+	if (ret)
+		goto free_binfo_exit;
+
+	ret = build_info_commit_dev(binfo);
+	if (ret)
+		goto free_binfo_exit;
+
+	/*
+	 * everything is okay, reset ->parent_dev to stop it being
+	 * freed by build_info_free()
+	 */
+	binfo->parent_dev = NULL;
+
+free_binfo_exit:
+	build_info_free(binfo);
+	return ret;
+}
+
 /* PCI Device ID */
 #define PCIe_DEVICE_ID_PF_INT_5_X	0xBCBD
 #define PCIe_DEVICE_ID_PF_INT_6_X	0xBCC0
@@ -83,9 +900,18 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
 		goto release_region_exit;
 	}
 
-	/* TODO: create and add the platform device per feature list */
+	ret = create_init_drvdata(pcidev);
+	if (ret)
+		goto release_region_exit;
+
+	ret = cci_pci_create_feature_devs(pcidev);
+	if (ret)
+		goto destroy_drvdata_exit;
+
 	return 0;
 
+destroy_drvdata_exit:
+	destroy_drvdata(pcidev);
 release_region_exit:
 	pci_release_regions(pcidev);
 disable_error_report_exit:
@@ -97,6 +923,8 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
 
 static void cci_pci_remove(struct pci_dev *pcidev)
 {
+	remove_all_devs(pcidev);
+	destroy_drvdata(pcidev);
 	pci_release_regions(pcidev);
 	pci_disable_pcie_error_reporting(pcidev);
 	pci_disable_device(pcidev);
@@ -111,14 +939,23 @@ static struct pci_driver cci_pci_driver = {
 
 static int __init ccidrv_init(void)
 {
+	int ret;
+
 	pr_info("Intel(R) FPGA PCIe Driver: Version %s\n", DRV_VERSION);
 
-	return pci_register_driver(&cci_pci_driver);
+	fpga_ids_init();
+
+	ret = pci_register_driver(&cci_pci_driver);
+	if (ret)
+		fpga_ids_destroy();
+
+	return ret;
 }
 
 static void __exit ccidrv_exit(void)
 {
 	pci_unregister_driver(&cci_pci_driver);
+	fpga_ids_destroy();
 }
 
 module_init(ccidrv_init);
-- 
2.7.4

^ permalink raw reply	[flat|nested] 93+ messages in thread

* [PATCH 05/16] fpga: intel: pcie: add chardev support for feature devices
  2017-03-30 12:08 [PATCH 00/16] Intel FPGA Device Drivers Wu Hao
                   ` (3 preceding siblings ...)
  2017-03-30 12:08 ` [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features Wu Hao
@ 2017-03-30 12:08 ` Wu Hao
  2017-03-30 12:08 ` [PATCH 06/16] fpga: intel: pcie: adds fpga_for_each_port callback for fme device Wu Hao
                   ` (12 subsequent siblings)
  17 siblings, 0 replies; 93+ messages in thread
From: Wu Hao @ 2017-03-30 12:08 UTC (permalink / raw)
  To: atull, moritz.fischer, linux-fpga, linux-kernel
  Cc: luwei.kang, yi.z.zhang, hao.wu, Xiao Guangrong, Tim Whisonant,
	Enno Luebbers, Shiva Rao, Christopher Rauer

From: Xiao Guangrong <guangrong.xiao@linux.intel.com>

For feature devices drivers, both the FPGA Management Engine (FME) and
Accelerated Function Unit (AFU) driver need to expose user interfaces via
the device file, for example, mmap and ioctls.

This patch adds chardev support in the pcie driver for feature devices,
FME and AFU. It reserves the chardev regions for FME and AFU, and provide
interfaces for FME and AFU driver to register their device file operations.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
 drivers/fpga/intel/feature-dev.c | 76 ++++++++++++++++++++++++++++++++++++++++
 drivers/fpga/intel/feature-dev.h | 16 +++++++++
 drivers/fpga/intel/pcie.c        | 18 +++++++++-
 3 files changed, 109 insertions(+), 1 deletion(-)

diff --git a/drivers/fpga/intel/feature-dev.c b/drivers/fpga/intel/feature-dev.c
index 6952566..ada6548 100644
--- a/drivers/fpga/intel/feature-dev.c
+++ b/drivers/fpga/intel/feature-dev.c
@@ -59,6 +59,82 @@ int port_feature_num(void)
 	return PORT_FEATURE_ID_MAX;
 }
 
+struct fpga_chardev_info {
+	const char *name;
+	dev_t devt;
+};
+
+/* indexed by enum fpga_devt_type */
+struct fpga_chardev_info fpga_chrdevs[] = {
+	{.name = FPGA_FEATURE_DEV_FME},		/* FPGA_DEVT_FME */
+	{.name = FPGA_FEATURE_DEV_PORT},	/* FPGA_DEVT_AFU */
+};
+
+void fpga_chardev_uinit(void)
+{
+	int i;
+
+	for (i = 0; i < FPGA_DEVT_MAX; i++)
+		if (MAJOR(fpga_chrdevs[i].devt)) {
+			unregister_chrdev_region(fpga_chrdevs[i].devt,
+						 MINORMASK);
+			fpga_chrdevs[i].devt = MKDEV(0, 0);
+		}
+}
+
+int fpga_chardev_init(void)
+{
+	int i, ret;
+
+	for (i = 0; i < FPGA_DEVT_MAX; i++) {
+		ret = alloc_chrdev_region(&fpga_chrdevs[i].devt, 0, MINORMASK,
+					  fpga_chrdevs[i].name);
+		if (ret)
+			goto exit;
+	}
+
+	return 0;
+
+exit:
+	fpga_chardev_uinit();
+	return ret;
+}
+
+dev_t fpga_get_devt(enum fpga_devt_type type, int id)
+{
+	WARN_ON(type >= FPGA_DEVT_MAX);
+
+	return MKDEV(MAJOR(fpga_chrdevs[type].devt), id);
+}
+
+int fpga_register_dev_ops(struct platform_device *pdev,
+			  const struct file_operations *fops,
+			  struct module *owner)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	cdev_init(&pdata->cdev, fops);
+	pdata->cdev.owner = owner;
+
+	/*
+	 * set parent to the feature device so that its refcount is
+	 * decreased after the last refcount of cdev is gone, that
+	 * makes sure the feature device is valid during device
+	 * file's life-cycle.
+	 */
+	pdata->cdev.kobj.parent = &pdev->dev.kobj;
+	return cdev_add(&pdata->cdev, pdev->dev.devt, 1);
+}
+EXPORT_SYMBOL_GPL(fpga_register_dev_ops);
+
+void fpga_unregister_dev_ops(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	cdev_del(&pdata->cdev);
+}
+EXPORT_SYMBOL_GPL(fpga_unregister_dev_ops);
+
 int fpga_port_id(struct platform_device *pdev)
 {
 	struct feature_port_header *port_hdr;
diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
index a1e6e7d..d1723ff 100644
--- a/drivers/fpga/intel/feature-dev.h
+++ b/drivers/fpga/intel/feature-dev.h
@@ -19,6 +19,7 @@
 #define __INTEL_FPGA_FEATURE_H
 
 #include <linux/fs.h>
+#include <linux/cdev.h>
 #include <linux/pci.h>
 #include <linux/uuid.h>
 #include <linux/delay.h>
@@ -216,6 +217,7 @@ struct feature_platform_data {
 	/* list the feature dev to cci_drvdata->port_dev_list. */
 	struct list_head node;
 	struct mutex lock;
+	struct cdev cdev;
 	struct platform_device *dev;
 	unsigned int disable_count;	/* count for port disable */
 
@@ -256,6 +258,20 @@ int feature_platform_data_size(int num);
 struct feature_platform_data *
 feature_platform_data_alloc_and_init(struct platform_device *dev, int num);
 
+enum fpga_devt_type {
+	FPGA_DEVT_FME,
+	FPGA_DEVT_PORT,
+	FPGA_DEVT_MAX,
+};
+
+void fpga_chardev_uinit(void);
+int fpga_chardev_init(void);
+dev_t fpga_get_devt(enum fpga_devt_type type, int id);
+int fpga_register_dev_ops(struct platform_device *pdev,
+			  const struct file_operations *fops,
+			  struct module *owner);
+void fpga_unregister_dev_ops(struct platform_device *pdev);
+
 int fpga_port_id(struct platform_device *pdev);
 
 static inline int fpga_port_check_id(struct platform_device *pdev,
diff --git a/drivers/fpga/intel/pcie.c b/drivers/fpga/intel/pcie.c
index 28df63e..e3440ca 100644
--- a/drivers/fpga/intel/pcie.c
+++ b/drivers/fpga/intel/pcie.c
@@ -276,8 +276,12 @@ build_info_create_dev(struct build_feature_devs_info *binfo,
 	struct platform_device *fdev;
 	struct resource *res;
 	struct feature_platform_data *pdata;
+	enum fpga_devt_type devt_type = FPGA_DEVT_FME;
 	int ret;
 
+	if (type == PORT_ID)
+		devt_type = FPGA_DEVT_PORT;
+
 	/* we will create a new device, commit current device first */
 	ret = build_info_commit_dev(binfo);
 	if (ret)
@@ -296,6 +300,7 @@ build_info_create_dev(struct build_feature_devs_info *binfo,
 		return fdev->id;
 
 	fdev->dev.parent = &binfo->parent_dev->dev;
+	fdev->dev.devt = fpga_get_devt(devt_type, fdev->id);
 
 	/*
 	 * we need not care the memory which is associated with the
@@ -945,16 +950,27 @@ static int __init ccidrv_init(void)
 
 	fpga_ids_init();
 
+	ret = fpga_chardev_init();
+	if (ret)
+		goto exit_ids;
+
 	ret = pci_register_driver(&cci_pci_driver);
 	if (ret)
-		fpga_ids_destroy();
+		goto exit_chardev;
 
+	return 0;
+
+exit_chardev:
+	fpga_chardev_uinit();
+exit_ids:
+	fpga_ids_destroy();
 	return ret;
 }
 
 static void __exit ccidrv_exit(void)
 {
 	pci_unregister_driver(&cci_pci_driver);
+	fpga_chardev_uinit();
 	fpga_ids_destroy();
 }
 
-- 
2.7.4

^ permalink raw reply	[flat|nested] 93+ messages in thread

* [PATCH 06/16] fpga: intel: pcie: adds fpga_for_each_port callback for fme device
  2017-03-30 12:08 [PATCH 00/16] Intel FPGA Device Drivers Wu Hao
                   ` (4 preceding siblings ...)
  2017-03-30 12:08 ` [PATCH 05/16] fpga: intel: pcie: add chardev support for feature devices Wu Hao
@ 2017-03-30 12:08 ` Wu Hao
  2017-03-30 12:08 ` [PATCH 07/16] fpga: intel: add feature device infrastructure Wu Hao
                   ` (11 subsequent siblings)
  17 siblings, 0 replies; 93+ messages in thread
From: Wu Hao @ 2017-03-30 12:08 UTC (permalink / raw)
  To: atull, moritz.fischer, linux-fpga, linux-kernel
  Cc: luwei.kang, yi.z.zhang, hao.wu, Tim Whisonant, Enno Luebbers,
	Shiva Rao, Christopher Rauer, Xiao Guangrong

For FPGA Management Engine (FME), it requires fpga_for_each_port callback
for actions on ports, so export this function from PCIe driver by adding
the callback to the platform data.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
 drivers/fpga/intel/feature-dev.h |  9 +++++++++
 drivers/fpga/intel/pcie.c        | 24 ++++++++++++++++++++++++
 2 files changed, 33 insertions(+)

diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
index d1723ff..38531f8 100644
--- a/drivers/fpga/intel/feature-dev.h
+++ b/drivers/fpga/intel/feature-dev.h
@@ -221,6 +221,9 @@ struct feature_platform_data {
 	struct platform_device *dev;
 	unsigned int disable_count;	/* count for port disable */
 
+	struct platform_device *(*fpga_for_each_port)(struct platform_device *,
+			void *, int (*match)(struct platform_device *, void *));
+
 	int num;			/* number of features */
 	struct feature features[0];
 };
@@ -335,6 +338,12 @@ get_feature_ioaddr_by_index(struct device *dev, int index)
 	return pdata->features[index].ioaddr;
 }
 
+static inline struct device *
+fpga_feature_dev_to_pcidev(struct platform_device *dev)
+{
+	return dev->dev.parent->parent;
+}
+
 /*
  * Wait register's _field to be changed to the given value (_expect's _field)
  * by polling with given interval and timeout.
diff --git a/drivers/fpga/intel/pcie.c b/drivers/fpga/intel/pcie.c
index e3440ca..f2b458d 100644
--- a/drivers/fpga/intel/pcie.c
+++ b/drivers/fpga/intel/pcie.c
@@ -211,6 +211,27 @@ static int parse_switch_to(struct build_feature_devs_info *binfo, int bar)
 	return parse_start_from(binfo, bar);
 }
 
+static struct platform_device *fpga_for_each_port(struct platform_device *pdev,
+		     void *data, int (*match)(struct platform_device *, void *))
+{
+	struct device *pci_dev = fpga_feature_dev_to_pcidev(pdev);
+	struct cci_drvdata *drvdata = dev_get_drvdata(pci_dev);
+	struct feature_platform_data *pdata;
+	struct platform_device *port_dev;
+
+	mutex_lock(&drvdata->lock);
+	list_for_each_entry(pdata, &drvdata->port_dev_list, node) {
+		port_dev = pdata->dev;
+
+		if (match(port_dev, data) && get_device(&port_dev->dev))
+			goto exit;
+	}
+	port_dev = NULL;
+exit:
+	mutex_unlock(&drvdata->lock);
+	return port_dev;
+}
+
 static struct build_feature_devs_info *
 build_info_alloc_and_init(struct pci_dev *pdev)
 {
@@ -312,6 +333,9 @@ build_info_create_dev(struct build_feature_devs_info *binfo,
 	if (!pdata)
 		return -ENOMEM;
 
+	if (type == FME_ID)
+		pdata->fpga_for_each_port = fpga_for_each_port;
+
 	/*
 	 * the count should be initialized to 0 to make sure
 	 *__fpga_port_enable() following __fpga_port_disable()
-- 
2.7.4

^ permalink raw reply	[flat|nested] 93+ messages in thread

* [PATCH 07/16] fpga: intel: add feature device infrastructure
  2017-03-30 12:08 [PATCH 00/16] Intel FPGA Device Drivers Wu Hao
                   ` (5 preceding siblings ...)
  2017-03-30 12:08 ` [PATCH 06/16] fpga: intel: pcie: adds fpga_for_each_port callback for fme device Wu Hao
@ 2017-03-30 12:08 ` Wu Hao
  2017-03-30 12:08 ` [PATCH 08/16] fpga: intel: add FPGA Management Engine driver basic framework Wu Hao
                   ` (10 subsequent siblings)
  17 siblings, 0 replies; 93+ messages in thread
From: Wu Hao @ 2017-03-30 12:08 UTC (permalink / raw)
  To: atull, moritz.fischer, linux-fpga, linux-kernel
  Cc: luwei.kang, yi.z.zhang, hao.wu, Xiao Guangrong, Tim Whisonant,
	Enno Luebbers, Shiva Rao, Christopher Rauer

From: Xiao Guangrong <guangrong.xiao@linux.intel.com>

This patch abstracts the common operations of the sub features, and defines
the feature_ops data structure, including init, uinit and ioctl function
pointers. And this patch adds some common helper functions for FME and AFU
drivers, e.g feature_dev_use_begin/end which are used to ensure exclusive
usage of the feature device file.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Kang Luwei <luwei.kang@intel.com>
Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
 drivers/fpga/intel/feature-dev.c | 66 +++++++++++++++++++++++++++++++++++++
 drivers/fpga/intel/feature-dev.h | 71 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 137 insertions(+)

diff --git a/drivers/fpga/intel/feature-dev.c b/drivers/fpga/intel/feature-dev.c
index ada6548..d729db8 100644
--- a/drivers/fpga/intel/feature-dev.c
+++ b/drivers/fpga/intel/feature-dev.c
@@ -59,6 +59,72 @@ int port_feature_num(void)
 	return PORT_FEATURE_ID_MAX;
 }
 
+int fme_feature_to_resource_index(int feature_id)
+{
+	WARN_ON(feature_id >= FME_FEATURE_ID_MAX);
+	return feature_id;
+}
+
+void fpga_dev_feature_uinit(struct platform_device *pdev)
+{
+	struct feature *feature;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	fpga_dev_for_each_feature(pdata, feature)
+		if (feature->ops) {
+			feature->ops->uinit(pdev, feature);
+			feature->ops = NULL;
+		}
+}
+EXPORT_SYMBOL_GPL(fpga_dev_feature_uinit);
+
+static int
+feature_instance_init(struct platform_device *pdev,
+		      struct feature_platform_data *pdata,
+		      struct feature *feature, struct feature_driver *drv)
+{
+	int ret;
+
+	WARN_ON(!feature->ioaddr);
+
+	ret = drv->ops->init(pdev, feature);
+	if (ret)
+		return ret;
+
+	feature->ops = drv->ops;
+	return ret;
+}
+
+int fpga_dev_feature_init(struct platform_device *pdev,
+			  struct feature_driver *feature_drvs)
+{
+	struct feature *feature;
+	struct feature_driver *drv = feature_drvs;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	int ret;
+
+	while (drv->ops) {
+		fpga_dev_for_each_feature(pdata, feature) {
+			/* skip the feature which is not initialized. */
+			if (!feature->name)
+				continue;
+
+			if (!strcmp(drv->name, feature->name)) {
+				ret = feature_instance_init(pdev, pdata,
+							    feature, drv);
+				if (ret)
+					goto exit;
+			}
+		}
+		drv++;
+	}
+	return 0;
+exit:
+	fpga_dev_feature_uinit(pdev);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(fpga_dev_feature_init);
+
 struct fpga_chardev_info {
 	const char *name;
 	dev_t devt;
diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
index 38531f8..9d39b94 100644
--- a/drivers/fpga/intel/feature-dev.h
+++ b/drivers/fpga/intel/feature-dev.h
@@ -207,12 +207,20 @@ struct feature_port_stp {
 
 #pragma pack()
 
+struct feature_driver {
+	const char *name;
+	struct feature_ops *ops;
+};
+
 struct feature {
 	const char *name;
 	int resource_index;
 	void __iomem *ioaddr;
+	struct feature_ops *ops;
 };
 
+#define DEV_STATUS_IN_USE	0
+
 struct feature_platform_data {
 	/* list the feature dev to cci_drvdata->port_dev_list. */
 	struct list_head node;
@@ -220,6 +228,9 @@ struct feature_platform_data {
 	struct cdev cdev;
 	struct platform_device *dev;
 	unsigned int disable_count;	/* count for port disable */
+	unsigned long dev_status;
+
+	void *private;			/* ptr to feature dev private data */
 
 	struct platform_device *(*fpga_for_each_port)(struct platform_device *,
 			void *, int (*match)(struct platform_device *, void *));
@@ -228,6 +239,38 @@ struct feature_platform_data {
 	struct feature features[0];
 };
 
+static inline int feature_dev_use_begin(struct feature_platform_data *pdata)
+{
+	/* Test and set IN_USE flags to ensure file is exclusively used */
+	if (test_and_set_bit_lock(DEV_STATUS_IN_USE, &pdata->dev_status))
+		return -EBUSY;
+
+	return 0;
+}
+
+static inline void feature_dev_use_end(struct feature_platform_data *pdata)
+{
+	clear_bit_unlock(DEV_STATUS_IN_USE, &pdata->dev_status);
+}
+
+static inline void
+fpga_pdata_set_private(struct feature_platform_data *pdata, void *private)
+{
+	pdata->private = private;
+}
+
+static inline void *fpga_pdata_get_private(struct feature_platform_data *pdata)
+{
+	return pdata->private;
+}
+
+struct feature_ops {
+	int (*init)(struct platform_device *pdev, struct feature *feature);
+	void (*uinit)(struct platform_device *pdev, struct feature *feature);
+	long (*ioctl)(struct platform_device *pdev, struct feature *feature,
+				unsigned int cmd, unsigned long arg);
+};
+
 enum fme_feature_id {
 	FME_FEATURE_ID_HEADER = 0x0,
 	FME_FEATURE_ID_THERMAL_MGMT = 0x1,
@@ -261,6 +304,10 @@ int feature_platform_data_size(int num);
 struct feature_platform_data *
 feature_platform_data_alloc_and_init(struct platform_device *dev, int num);
 
+void fpga_dev_feature_uinit(struct platform_device *pdev);
+int fpga_dev_feature_init(struct platform_device *pdev,
+			  struct feature_driver *feature_drvs);
+
 enum fpga_devt_type {
 	FPGA_DEVT_FME,
 	FPGA_DEVT_PORT,
@@ -330,6 +377,15 @@ static inline int fpga_port_reset(struct platform_device *pdev)
 	return ret;
 }
 
+static inline
+struct platform_device *fpga_inode_to_feature_dev(struct inode *inode)
+{
+	struct feature_platform_data *pdata;
+
+	pdata = container_of(inode->i_cdev, struct feature_platform_data, cdev);
+	return pdata->dev;
+}
+
 static inline void __iomem *
 get_feature_ioaddr_by_index(struct device *dev, int index)
 {
@@ -338,12 +394,27 @@ get_feature_ioaddr_by_index(struct device *dev, int index)
 	return pdata->features[index].ioaddr;
 }
 
+static inline bool is_feature_present(struct device *dev, int index)
+{
+	return !!get_feature_ioaddr_by_index(dev, index);
+}
+
 static inline struct device *
 fpga_feature_dev_to_pcidev(struct platform_device *dev)
 {
 	return dev->dev.parent->parent;
 }
 
+static inline struct device *
+fpga_pdata_to_pcidev(struct feature_platform_data *pdata)
+{
+	return fpga_feature_dev_to_pcidev(pdata->dev);
+}
+
+#define fpga_dev_for_each_feature(pdata, feature)			    \
+	for ((feature) = (pdata)->features;				    \
+	   (feature) < (pdata)->features + (pdata)->num; (feature)++)
+
 /*
  * Wait register's _field to be changed to the given value (_expect's _field)
  * by polling with given interval and timeout.
-- 
2.7.4

^ permalink raw reply	[flat|nested] 93+ messages in thread

* [PATCH 08/16] fpga: intel: add FPGA Management Engine driver basic framework
  2017-03-30 12:08 [PATCH 00/16] Intel FPGA Device Drivers Wu Hao
                   ` (6 preceding siblings ...)
  2017-03-30 12:08 ` [PATCH 07/16] fpga: intel: add feature device infrastructure Wu Hao
@ 2017-03-30 12:08 ` Wu Hao
  2017-03-30 12:08 ` [PATCH 09/16] fpga: intel: fme: add header sub feature support Wu Hao
                   ` (9 subsequent siblings)
  17 siblings, 0 replies; 93+ messages in thread
From: Wu Hao @ 2017-03-30 12:08 UTC (permalink / raw)
  To: atull, moritz.fischer, linux-fpga, linux-kernel
  Cc: luwei.kang, yi.z.zhang, hao.wu, Tim Whisonant, Enno Luebbers,
	Shiva Rao, Christopher Rauer, Xiao Guangrong

From: Kang Luwei <luwei.kang@intel.com>

The FPGA Management Engine (FME) provides power, thermal management,
performance counters, partial reconfiguration and other functions. For each
function, it is packaged into a private feature linked to the FME feature
device in the 'Device Feature List'. It's a platform device created by
Intel FPGA PCIe driver.

This patch adds the basic framework of FME platform driver. It defines
sub feature drivers to handle the different sub features, including init,
uinit and ioctl. It also registers the file operations for the device file.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Kang Luwei <luwei.kang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
 drivers/fpga/intel/Kconfig    |   8 +++
 drivers/fpga/intel/Makefile   |   2 +
 drivers/fpga/intel/fme-main.c | 158 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 168 insertions(+)
 create mode 100644 drivers/fpga/intel/fme-main.c

diff --git a/drivers/fpga/intel/Kconfig b/drivers/fpga/intel/Kconfig
index bf402f3..62e2160 100644
--- a/drivers/fpga/intel/Kconfig
+++ b/drivers/fpga/intel/Kconfig
@@ -24,4 +24,12 @@ config INTEL_FPGA_PCI
 
 	  To compile this as a module, choose M here.
 
+config INTEL_FPGA_FME
+	tristate "Intel FPGA FME Driver"
+	depends on INTEL_FPGA_PCI
+	help
+	  This is the driver for FPGA Management Engine which implements
+	  all FPGA platform level management features. There shall be 1
+	  FME per Intel FPGA.
+
 endif
diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
index c029940..546861d 100644
--- a/drivers/fpga/intel/Makefile
+++ b/drivers/fpga/intel/Makefile
@@ -1,3 +1,5 @@
 obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
+obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
 
 intel-fpga-pci-objs := pcie.o feature-dev.o
+intel-fpga-fme-objs := fme-main.o
diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
new file mode 100644
index 0000000..c603268
--- /dev/null
+++ b/drivers/fpga/intel/fme-main.c
@@ -0,0 +1,158 @@
+/*
+ * Driver for Intel FPGA Management Engine (FME)
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ *
+ * This work is licensed under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license. See the
+ * LICENSE.BSD file under this directory for the BSD license and see
+ * the COPYING file in the top-level directory for the GPLv2 license.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+#include "feature-dev.h"
+
+static int fme_hdr_init(struct platform_device *pdev, struct feature *feature)
+{
+	dev_dbg(&pdev->dev, "FME HDR Init.\n");
+	return 0;
+}
+
+static void fme_hdr_uinit(struct platform_device *pdev, struct feature *feature)
+{
+	dev_dbg(&pdev->dev, "FME HDR UInit.\n");
+}
+
+struct feature_ops fme_hdr_ops = {
+	.init = fme_hdr_init,
+	.uinit = fme_hdr_uinit,
+};
+
+static struct feature_driver fme_feature_drvs[] = {
+	{
+		.name = FME_FEATURE_HEADER,
+		.ops = &fme_hdr_ops,
+	},
+	{
+		.ops = NULL,
+	},
+};
+
+static int fme_open(struct inode *inode, struct file *filp)
+{
+	struct platform_device *fdev = fpga_inode_to_feature_dev(inode);
+	struct feature_platform_data *pdata = dev_get_platdata(&fdev->dev);
+	int ret;
+
+	if (WARN_ON(!pdata))
+		return -ENODEV;
+
+	ret = feature_dev_use_begin(pdata);
+	if (ret)
+		return ret;
+
+	dev_dbg(&fdev->dev, "Device File Open\n");
+	filp->private_data = pdata;
+	return 0;
+}
+
+static int fme_release(struct inode *inode, struct file *filp)
+{
+	struct feature_platform_data *pdata = filp->private_data;
+	struct platform_device *pdev = pdata->dev;
+
+	dev_dbg(&pdev->dev, "Device File Release\n");
+	feature_dev_use_end(pdata);
+	return 0;
+}
+
+static long fme_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
+{
+	struct feature_platform_data *pdata = filp->private_data;
+	struct platform_device *pdev = pdata->dev;
+	struct feature *f;
+	long ret;
+
+	dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
+
+	switch (cmd) {
+	default:
+		/*
+		 * Let sub-feature's ioctl function to handle the cmd
+		 * Sub-feature's ioctl returns -ENODEV when cmd is not
+		 * handled in this sub feature, and returns 0 and other
+		 * error code if cmd is handled.
+		 */
+		fpga_dev_for_each_feature(pdata, f) {
+			if (f->ops && f->ops->ioctl) {
+				ret = f->ops->ioctl(pdev, f, cmd, arg);
+				if (ret == -ENODEV)
+					continue;
+				else
+					return ret;
+			}
+		}
+	}
+
+	return -EINVAL;
+}
+
+static const struct file_operations fme_fops = {
+	.owner		= THIS_MODULE,
+	.open		= fme_open,
+	.release	= fme_release,
+	.unlocked_ioctl = fme_ioctl,
+};
+
+static int fme_probe(struct platform_device *pdev)
+{
+	int ret;
+
+	ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+	if (ret)
+		goto exit;
+
+	ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
+	if (ret)
+		goto feature_uinit;
+
+	return 0;
+
+feature_uinit:
+	fpga_dev_feature_uinit(pdev);
+exit:
+	return ret;
+}
+
+static int fme_remove(struct platform_device *pdev)
+{
+	fpga_dev_feature_uinit(pdev);
+	fpga_unregister_dev_ops(pdev);
+	return 0;
+}
+
+static struct platform_driver fme_driver = {
+	.driver	= {
+		.name    = FPGA_FEATURE_DEV_FME,
+	},
+	.probe   = fme_probe,
+	.remove  = fme_remove,
+};
+
+module_platform_driver(fme_driver);
+
+MODULE_DESCRIPTION("Intel FPGA Management Engine driver");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("Dual BSD/GPL");
+MODULE_ALIAS("platform:intel-fpga-fme");
-- 
2.7.4

^ permalink raw reply	[flat|nested] 93+ messages in thread

* [PATCH 09/16] fpga: intel: fme: add header sub feature support
  2017-03-30 12:08 [PATCH 00/16] Intel FPGA Device Drivers Wu Hao
                   ` (7 preceding siblings ...)
  2017-03-30 12:08 ` [PATCH 08/16] fpga: intel: add FPGA Management Engine driver basic framework Wu Hao
@ 2017-03-30 12:08 ` Wu Hao
  2017-03-30 12:08 ` [PATCH 10/16] fpga: intel: fme: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support Wu Hao
                   ` (8 subsequent siblings)
  17 siblings, 0 replies; 93+ messages in thread
From: Wu Hao @ 2017-03-30 12:08 UTC (permalink / raw)
  To: atull, moritz.fischer, linux-fpga, linux-kernel
  Cc: luwei.kang, yi.z.zhang, hao.wu, Tim Whisonant, Enno Luebbers,
	Shiva Rao, Christopher Rauer, Xiao Guangrong

From: Kang Luwei <luwei.kang@intel.com>

The header register set is always present for FPGA Management Engine (FME),
this patch implements init and uinit function for header sub feature and
introduce several read-only sysfs interfaces for the capability and status.

Sysfs interfaces:
* /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/ports_num
  Read-only. Number of ports implemented

* /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/bitstream_id
  Read-only. Blue Bitstream identifier number

* /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/bitstream_metadata
  Read-only. Blue Bitstream meta data

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Kang Luwei <luwei.kang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
 drivers/fpga/intel/feature-dev.h |  3 +++
 drivers/fpga/intel/fme-main.c    | 55 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 58 insertions(+)

diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
index 9d39b94..dccc283 100644
--- a/drivers/fpga/intel/feature-dev.h
+++ b/drivers/fpga/intel/feature-dev.h
@@ -124,6 +124,9 @@ struct feature_fme_header {
 	u64 rsvd[2];
 	struct feature_fme_capability capability;
 	struct feature_fme_port port[MAX_FPGA_PORT_NUM];
+	u64 rsvd1;
+	u64 bitstream_id;
+	u64 bitstream_md;
 };
 
 /* FME Thermal Sub Feature Register Set */
diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
index c603268..a7c69fc 100644
--- a/drivers/fpga/intel/fme-main.c
+++ b/drivers/fpga/intel/fme-main.c
@@ -23,15 +23,70 @@
 
 #include "feature-dev.h"
 
+static ssize_t ports_num_show(struct device *dev,
+			      struct device_attribute *attr, char *buf)
+{
+	struct feature_fme_header *fme_hdr
+		= get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_HEADER);
+	struct feature_fme_capability fme_capability;
+
+	fme_capability.csr = readq(&fme_hdr->capability);
+
+	return scnprintf(buf, PAGE_SIZE, "%d\n", fme_capability.num_ports);
+}
+static DEVICE_ATTR_RO(ports_num);
+
+static ssize_t bitstream_id_show(struct device *dev,
+				 struct device_attribute *attr, char *buf)
+{
+	struct feature_fme_header *fme_hdr
+		= get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_HEADER);
+	u64 bitstream_id = readq(&fme_hdr->bitstream_id);
+
+	return scnprintf(buf, PAGE_SIZE, "0x%llx\n",
+				(unsigned long long)bitstream_id);
+}
+static DEVICE_ATTR_RO(bitstream_id);
+
+static ssize_t bitstream_metadata_show(struct device *dev,
+				       struct device_attribute *attr, char *buf)
+{
+	struct feature_fme_header *fme_hdr
+		= get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_HEADER);
+	u64 bitstream_md = readq(&fme_hdr->bitstream_md);
+
+	return scnprintf(buf, PAGE_SIZE, "0x%llx\n",
+				(unsigned long long)bitstream_md);
+}
+static DEVICE_ATTR_RO(bitstream_metadata);
+
+static const struct attribute *fme_hdr_attrs[] = {
+	&dev_attr_ports_num.attr,
+	&dev_attr_bitstream_id.attr,
+	&dev_attr_bitstream_metadata.attr,
+	NULL,
+};
+
 static int fme_hdr_init(struct platform_device *pdev, struct feature *feature)
 {
+	struct feature_fme_header *fme_hdr = feature->ioaddr;
+	int ret;
+
 	dev_dbg(&pdev->dev, "FME HDR Init.\n");
+	dev_dbg(&pdev->dev, "FME cap %llx.\n",
+				(unsigned long long)fme_hdr->capability.csr);
+
+	ret = sysfs_create_files(&pdev->dev.kobj, fme_hdr_attrs);
+	if (ret)
+		return ret;
+
 	return 0;
 }
 
 static void fme_hdr_uinit(struct platform_device *pdev, struct feature *feature)
 {
 	dev_dbg(&pdev->dev, "FME HDR UInit.\n");
+	sysfs_remove_files(&pdev->dev.kobj, fme_hdr_attrs);
 }
 
 struct feature_ops fme_hdr_ops = {
-- 
2.7.4

^ permalink raw reply	[flat|nested] 93+ messages in thread

* [PATCH 10/16] fpga: intel: fme: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support
  2017-03-30 12:08 [PATCH 00/16] Intel FPGA Device Drivers Wu Hao
                   ` (8 preceding siblings ...)
  2017-03-30 12:08 ` [PATCH 09/16] fpga: intel: fme: add header sub feature support Wu Hao
@ 2017-03-30 12:08 ` Wu Hao
  2017-03-30 12:08 ` [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Wu Hao
                   ` (7 subsequent siblings)
  17 siblings, 0 replies; 93+ messages in thread
From: Wu Hao @ 2017-03-30 12:08 UTC (permalink / raw)
  To: atull, moritz.fischer, linux-fpga, linux-kernel
  Cc: luwei.kang, yi.z.zhang, hao.wu, Tim Whisonant, Enno Luebbers,
	Shiva Rao, Christopher Rauer, Xiao Guangrong

FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which
need to be supported by all feature devices drivers including FME and AFU.
Userspace application can use these ioctl interfaces to get the API info
and check if specific extension is supported or not in current driver.

This patch implements above 2 ioctls in Intel FPGA Management Engine (FME)
driver.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
 Documentation/ioctl/ioctl-number.txt |  1 +
 drivers/fpga/intel/fme-main.c        | 12 +++++++++
 include/uapi/linux/intel-fpga.h      | 52 ++++++++++++++++++++++++++++++++++++
 3 files changed, 65 insertions(+)
 create mode 100644 include/uapi/linux/intel-fpga.h

diff --git a/Documentation/ioctl/ioctl-number.txt b/Documentation/ioctl/ioctl-number.txt
index 08244be..462f4a5 100644
--- a/Documentation/ioctl/ioctl-number.txt
+++ b/Documentation/ioctl/ioctl-number.txt
@@ -322,6 +322,7 @@ Code  Seq#(hex)	Include File		Comments
 0xB3	00	linux/mmc/ioctl.h
 0xB4	00-0F	linux/gpio.h		<mailto:linux-gpio@vger.kernel.org>
 0xB5	00-0F	uapi/linux/rpmsg.h	<mailto:linux-remoteproc@vger.kernel.org>
+0xB6	all	linux/intel-fpga.h
 0xC0	00-0F	linux/usb/iowarrior.h
 0xCA	00-0F	uapi/misc/cxl.h
 0xCA	80-8F	uapi/scsi/cxlflash_ioctl.h
diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
index a7c69fc..36d0c4c 100644
--- a/drivers/fpga/intel/fme-main.c
+++ b/drivers/fpga/intel/fme-main.c
@@ -20,6 +20,7 @@
 
 #include <linux/kernel.h>
 #include <linux/module.h>
+#include <linux/intel-fpga.h>
 
 #include "feature-dev.h"
 
@@ -104,6 +105,13 @@ static struct feature_driver fme_feature_drvs[] = {
 	},
 };
 
+static long fme_ioctl_check_extension(struct feature_platform_data *pdata,
+				     unsigned long arg)
+{
+	/* No extension support for now */
+	return 0;
+}
+
 static int fme_open(struct inode *inode, struct file *filp)
 {
 	struct platform_device *fdev = fpga_inode_to_feature_dev(inode);
@@ -142,6 +150,10 @@ static long fme_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 	dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
 
 	switch (cmd) {
+	case FPGA_GET_API_VERSION:
+		return FPGA_API_VERSION;
+	case FPGA_CHECK_EXTENSION:
+		return fme_ioctl_check_extension(pdata, arg);
 	default:
 		/*
 		 * Let sub-feature's ioctl function to handle the cmd
diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
new file mode 100644
index 0000000..992e556
--- /dev/null
+++ b/include/uapi/linux/intel-fpga.h
@@ -0,0 +1,52 @@
+/*
+ * Header File for Intel FPGA User API
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Zhang Yi <yi.z.zhang@intel.com>
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *
+ * This work is licensed under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license. See the
+ * LICENSE.BSD file under drivers/fpga/intel for the BSD license and see
+ * the COPYING file in the top-level directory for the GPLv2 license.
+ */
+
+#ifndef _UAPI_LINUX_INTEL_FPGA_H
+#define _UAPI_LINUX_INTEL_FPGA_H
+
+#define FPGA_API_VERSION 0
+
+/*
+ * The IOCTL interface for Intel FPGA is designed for extensibility by
+ * embedding the structure length (argsz) and flags into structures passed
+ * between kernel and userspace. This design referenced the VFIO IOCTL
+ * interface (include/uapi/linux/vfio.h).
+ */
+
+#define FPGA_MAGIC 0xB6
+
+#define FPGA_BASE 0
+
+/**
+ * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
+ *
+ * Report the version of the driver API.
+ * Return: Driver API Version.
+ */
+
+#define FPGA_GET_API_VERSION	_IO(FPGA_MAGIC, FPGA_BASE + 0)
+
+/**
+ * FPGA_CHECK_EXTENSION - _IO(FPGA_MAGIC, FPGA_BASE + 1)
+ *
+ * Check whether an extension is supported.
+ * Return: 0 if not supported, otherwise the extension is supported.
+ */
+
+#define FPGA_CHECK_EXTENSION	_IO(FPGA_MAGIC, FPGA_BASE + 1)
+
+#endif /* _UAPI_INTEL_FPGA_H */
-- 
2.7.4

^ permalink raw reply	[flat|nested] 93+ messages in thread

* [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-03-30 12:08 [PATCH 00/16] Intel FPGA Device Drivers Wu Hao
                   ` (9 preceding siblings ...)
  2017-03-30 12:08 ` [PATCH 10/16] fpga: intel: fme: add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support Wu Hao
@ 2017-03-30 12:08 ` Wu Hao
  2017-03-30 20:30   ` Alan Tull
                     ` (4 more replies)
  2017-03-30 12:08 ` [PATCH 12/16] fpga: intel: add FPGA Accelerated Function Unit driver basic framework Wu Hao
                   ` (6 subsequent siblings)
  17 siblings, 5 replies; 93+ messages in thread
From: Wu Hao @ 2017-03-30 12:08 UTC (permalink / raw)
  To: atull, moritz.fischer, linux-fpga, linux-kernel
  Cc: luwei.kang, yi.z.zhang, hao.wu, Tim Whisonant, Enno Luebbers,
	Shiva Rao, Christopher Rauer, Alan Tull, Xiao Guangrong

From: Kang Luwei <luwei.kang@intel.com>

Partial Reconfiguration (PR) is the most important function for FME. It
allows reconfiguration for given Port/Accelerated Function Unit (AFU).

This patch adds support for PR sub feature. In this patch, it registers
a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
for PR operation once PR request received via ioctl. Below user space
interfaces are exposed by this sub feature.

Sysfs interface:
* /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
  Read-only. Indicate the hardware interface information. Userspace
  applications need to check this interface to select correct green
  bitstream format before PR.

Ioctl interface:
* FPGA_FME_PORT_PR
  Do partial reconfiguration per information from userspace, including
  target port(AFU), buffer size and address info. It returns the PR status
  (PR error code if failed) to userspace.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Alan Tull <alan.tull@intel.com>
Signed-off-by: Kang Luwei <luwei.kang@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
 drivers/fpga/intel/Makefile      |   2 +-
 drivers/fpga/intel/feature-dev.h |  58 ++++++
 drivers/fpga/intel/fme-main.c    |  44 ++++-
 drivers/fpga/intel/fme-pr.c      | 400 +++++++++++++++++++++++++++++++++++++++
 drivers/fpga/intel/fme.h         |  32 ++++
 include/uapi/linux/intel-fpga.h  |  44 +++++
 6 files changed, 578 insertions(+), 2 deletions(-)
 create mode 100644 drivers/fpga/intel/fme-pr.c
 create mode 100644 drivers/fpga/intel/fme.h

diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
index 546861d..0452cb6 100644
--- a/drivers/fpga/intel/Makefile
+++ b/drivers/fpga/intel/Makefile
@@ -2,4 +2,4 @@ obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
 obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
 
 intel-fpga-pci-objs := pcie.o feature-dev.o
-intel-fpga-fme-objs := fme-main.o
+intel-fpga-fme-objs := fme-main.o fme-pr.o
diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
index dccc283..5a25c915 100644
--- a/drivers/fpga/intel/feature-dev.h
+++ b/drivers/fpga/intel/feature-dev.h
@@ -150,8 +150,66 @@ struct feature_fme_err {
 };
 
 /* FME Partial Reconfiguration Sub Feature Register Set */
+/* FME PR Control Register */
+struct feature_fme_pr_ctl {
+	union {
+		u64 csr;
+		struct {
+			u8  pr_reset:1;		/* Reset PR Engine */
+			u8  rsvdz1:3;
+			u8  pr_reset_ack:1;	/* Reset PR Engine Ack */
+			u8  rsvdz2:3;
+			u8  pr_regionid:2;	/* PR Region ID */
+			u8  rsvdz3:2;
+			u8  pr_start_req:1;	/* PR Start Request */
+			u8  pr_push_complete:1;	/* PR Data push complete */
+			u8  pr_kind:1;		/* Load Customer or Intel GBS */
+			u32 rsvdz4:17;
+			u32 config_data;
+		};
+	};
+};
+
+/* FME PR Status Register */
+struct feature_fme_pr_status {
+	union {
+		u64 csr;
+		struct {
+			u16 pr_credit:9;	/* Number of PR Credits */
+			u8  rsvdz1:7;
+			u8  pr_status:1;	/* PR Operation status */
+			u8  rsvdz2:3;
+			u8  pr_ctrlr_status:3;	/* Controller status */
+			u8  rsvdz3:1;
+			u8  pr_host_status:4;	/* PR Host status */
+			u64 rsvdz4:36;
+		};
+	};
+};
+
+/* FME PR Data Register */
+struct feature_fme_pr_data {
+	union {
+		u64 csr;
+		struct {
+			/* PR data from the raw-binary file */
+			u32 pr_data_raw;
+			u32 rsvd;
+		};
+	};
+};
+
 struct feature_fme_pr {
 	struct feature_header header;
+	struct feature_fme_pr_ctl control;
+	struct feature_fme_pr_status status;
+	struct feature_fme_pr_data data;
+	u64 error;
+
+	u64 rsvd[16];
+
+	u64 intfc_id_l;		/* PR interface Id Low */
+	u64 intfc_id_h;		/* PR interface Id High */
 };
 
 /* PORT Header Register Set */
diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
index 36d0c4c..0d9a7a6 100644
--- a/drivers/fpga/intel/fme-main.c
+++ b/drivers/fpga/intel/fme-main.c
@@ -23,6 +23,7 @@
 #include <linux/intel-fpga.h>
 
 #include "feature-dev.h"
+#include "fme.h"
 
 static ssize_t ports_num_show(struct device *dev,
 			      struct device_attribute *attr, char *buf)
@@ -101,6 +102,10 @@ static struct feature_driver fme_feature_drvs[] = {
 		.ops = &fme_hdr_ops,
 	},
 	{
+		.name = FME_FEATURE_PR_MGMT,
+		.ops = &pr_mgmt_ops,
+	},
+	{
 		.ops = NULL,
 	},
 };
@@ -182,14 +187,48 @@ static const struct file_operations fme_fops = {
 	.unlocked_ioctl = fme_ioctl,
 };
 
+static int fme_dev_init(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_fme *fme;
+
+	fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
+	if (!fme)
+		return -ENOMEM;
+
+	fme->pdata = pdata;
+
+	mutex_lock(&pdata->lock);
+	fpga_pdata_set_private(pdata, fme);
+	mutex_unlock(&pdata->lock);
+	return 0;
+}
+
+static void fme_dev_destroy(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_fme *fme;
+
+	mutex_lock(&pdata->lock);
+	fme = fpga_pdata_get_private(pdata);
+	fpga_pdata_set_private(pdata, NULL);
+	mutex_unlock(&pdata->lock);
+
+	devm_kfree(&pdev->dev, fme);
+}
+
 static int fme_probe(struct platform_device *pdev)
 {
 	int ret;
 
-	ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+	ret = fme_dev_init(pdev);
 	if (ret)
 		goto exit;
 
+	ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+	if (ret)
+		goto dev_destroy;
+
 	ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
 	if (ret)
 		goto feature_uinit;
@@ -198,6 +237,8 @@ static int fme_probe(struct platform_device *pdev)
 
 feature_uinit:
 	fpga_dev_feature_uinit(pdev);
+dev_destroy:
+	fme_dev_destroy(pdev);
 exit:
 	return ret;
 }
@@ -206,6 +247,7 @@ static int fme_remove(struct platform_device *pdev)
 {
 	fpga_dev_feature_uinit(pdev);
 	fpga_unregister_dev_ops(pdev);
+	fme_dev_destroy(pdev);
 	return 0;
 }
 
diff --git a/drivers/fpga/intel/fme-pr.c b/drivers/fpga/intel/fme-pr.c
new file mode 100644
index 0000000..3b44a3e
--- /dev/null
+++ b/drivers/fpga/intel/fme-pr.c
@@ -0,0 +1,400 @@
+/*
+ * Driver for Intel FPGA Management Engine (FME) Partial Reconfiguration
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Christopher Rauer <christopher.rauer@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ *
+ * This work is licensed under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license. See the
+ * LICENSE.BSD file under this directory for the BSD license and see
+ * the COPYING file in the top-level directory for the GPLv2 license.
+ */
+
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/vmalloc.h>
+#include <linux/uaccess.h>
+#include <linux/fpga/fpga-mgr.h>
+#include <linux/intel-fpga.h>
+
+#include "feature-dev.h"
+#include "fme.h"
+
+#define PR_WAIT_TIMEOUT   8000000
+
+#define PR_HOST_STATUS_IDLE	0
+
+DEFINE_FPGA_PR_ERR_MSG(pr_err_msg);
+
+static ssize_t interface_id_show(struct device *dev,
+				 struct device_attribute *attr, char *buf)
+{
+	u64 intfc_id_l, intfc_id_h;
+	struct feature_fme_pr *fme_pr
+		= get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_PR_MGMT);
+
+	intfc_id_l = readq(&fme_pr->intfc_id_l);
+	intfc_id_h = readq(&fme_pr->intfc_id_h);
+
+	return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
+			(unsigned long long)intfc_id_h,
+			(unsigned long long)intfc_id_l);
+}
+static DEVICE_ATTR_RO(interface_id);
+
+static struct attribute *pr_mgmt_attrs[] = {
+	&dev_attr_interface_id.attr,
+	NULL,
+};
+
+struct attribute_group pr_mgmt_attr_group = {
+	.attrs	= pr_mgmt_attrs,
+	.name	= "pr",
+};
+
+static u64
+pr_err_handle(struct platform_device *pdev, struct feature_fme_pr *fme_pr)
+{
+	struct feature_fme_pr_status fme_pr_status;
+	unsigned long err_code;
+	u64 fme_pr_error;
+	int i = 0;
+
+	fme_pr_status.csr = readq(&fme_pr->status);
+	if (!fme_pr_status.pr_status)
+		return 0;
+
+	err_code = fme_pr_error = readq(&fme_pr->error);
+	for_each_set_bit(i, &err_code, PR_MAX_ERR_NUM)
+		dev_dbg(&pdev->dev, "%s\n", pr_err_msg[i]);
+	writeq(fme_pr_error, &fme_pr->error);
+	return fme_pr_error;
+}
+
+static int fme_pr_write_init(struct fpga_manager *mgr,
+		struct fpga_image_info *info, const char *buf, size_t count)
+{
+	struct fpga_fme *fme = mgr->priv;
+	struct platform_device *pdev;
+	struct feature_fme_pr *fme_pr;
+	struct feature_fme_pr_ctl fme_pr_ctl;
+	struct feature_fme_pr_status fme_pr_status;
+
+	pdev = fme->pdata->dev;
+	fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
+				FME_FEATURE_ID_PR_MGMT);
+	if (!fme_pr)
+		return -EINVAL;
+
+	if (WARN_ON(info->flags != FPGA_MGR_PARTIAL_RECONFIG))
+		return -EINVAL;
+
+	dev_dbg(&pdev->dev, "resetting PR before initiated PR\n");
+
+	fme_pr_ctl.csr = readq(&fme_pr->control);
+	fme_pr_ctl.pr_reset = 1;
+	writeq(fme_pr_ctl.csr, &fme_pr->control);
+
+	fme_pr_ctl.pr_reset_ack = 1;
+
+	if (fpga_wait_register_field(pr_reset_ack, fme_pr_ctl,
+				     &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
+		dev_err(&pdev->dev, "maximum PR timeout\n");
+		return -ETIMEDOUT;
+	}
+
+	fme_pr_ctl.csr = readq(&fme_pr->control);
+	fme_pr_ctl.pr_reset = 0;
+	writeq(fme_pr_ctl.csr, &fme_pr->control);
+
+	dev_dbg(&pdev->dev,
+		"waiting for PR resource in HW to be initialized and ready\n");
+
+	fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
+
+	if (fpga_wait_register_field(pr_host_status, fme_pr_status,
+				     &fme_pr->status, PR_WAIT_TIMEOUT, 1)) {
+		dev_err(&pdev->dev, "maximum PR timeout\n");
+		return -ETIMEDOUT;
+	}
+
+	dev_dbg(&pdev->dev, "check if have any previous PR error\n");
+	pr_err_handle(pdev, fme_pr);
+	return 0;
+}
+
+static int fme_pr_write(struct fpga_manager *mgr,
+			const char *buf, size_t count)
+{
+	struct fpga_fme *fme = mgr->priv;
+	struct platform_device *pdev;
+	struct feature_fme_pr *fme_pr;
+	struct feature_fme_pr_ctl fme_pr_ctl;
+	struct feature_fme_pr_status fme_pr_status;
+	struct feature_fme_pr_data fme_pr_data;
+	int delay, pr_credit, i = 0;
+
+	pdev = fme->pdata->dev;
+	fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
+				FME_FEATURE_ID_PR_MGMT);
+
+	dev_dbg(&pdev->dev, "set PR port ID and start request\n");
+
+	fme_pr_ctl.csr = readq(&fme_pr->control);
+	fme_pr_ctl.pr_regionid = fme->port_id;
+	fme_pr_ctl.pr_start_req = 1;
+	writeq(fme_pr_ctl.csr, &fme_pr->control);
+
+	dev_dbg(&pdev->dev, "pushing data from bitstream to HW\n");
+
+	fme_pr_status.csr = readq(&fme_pr->status);
+	pr_credit = fme_pr_status.pr_credit;
+
+	while (count > 0) {
+		delay = 0;
+		while (pr_credit <= 1) {
+			if (delay++ > PR_WAIT_TIMEOUT) {
+				dev_err(&pdev->dev, "maximum try\n");
+				return -ETIMEDOUT;
+			}
+			udelay(1);
+
+			fme_pr_status.csr = readq(&fme_pr->status);
+			pr_credit = fme_pr_status.pr_credit;
+		};
+
+		if (count >= 4) {
+			fme_pr_data.rsvd = 0;
+			fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
+			writeq(fme_pr_data.csr, &fme_pr->data);
+			count -= 4;
+			pr_credit--;
+			i++;
+		} else {
+			WARN_ON(1);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int fme_pr_write_complete(struct fpga_manager *mgr,
+			struct fpga_image_info *info)
+{
+	struct fpga_fme *fme = mgr->priv;
+	struct platform_device *pdev;
+	struct feature_fme_pr *fme_pr;
+	struct feature_fme_pr_ctl fme_pr_ctl;
+
+	pdev = fme->pdata->dev;
+	fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
+				FME_FEATURE_ID_PR_MGMT);
+
+	fme_pr_ctl.csr = readq(&fme_pr->control);
+	fme_pr_ctl.pr_push_complete = 1;
+	writeq(fme_pr_ctl.csr, &fme_pr->control);
+
+	dev_dbg(&pdev->dev, "green bitstream push complete\n");
+	dev_dbg(&pdev->dev, "waiting for HW to release PR resource\n");
+
+	fme_pr_ctl.pr_start_req = 0;
+
+	if (fpga_wait_register_field(pr_start_req, fme_pr_ctl,
+				     &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
+		dev_err(&pdev->dev, "maximum try.\n");
+		return -ETIMEDOUT;
+	}
+
+	dev_dbg(&pdev->dev, "PR operation complete, checking status\n");
+	fme->pr_err = pr_err_handle(pdev, fme_pr);
+	if (fme->pr_err)
+		return -EIO;
+
+	dev_dbg(&pdev->dev, "PR done successfully\n");
+	return 0;
+}
+
+static enum fpga_mgr_states fme_pr_state(struct fpga_manager *mgr)
+{
+	return FPGA_MGR_STATE_UNKNOWN;
+}
+
+static const struct fpga_manager_ops fme_pr_ops = {
+	.write_init = fme_pr_write_init,
+	.write = fme_pr_write,
+	.write_complete = fme_pr_write_complete,
+	.state = fme_pr_state,
+};
+
+static int fme_pr(struct platform_device *pdev, unsigned long arg)
+{
+	void __user *argp = (void __user *)arg;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_fme *fme;
+	struct fpga_manager *mgr;
+	struct feature_fme_header *fme_hdr;
+	struct feature_fme_capability fme_capability;
+	struct fpga_image_info info;
+	struct fpga_fme_port_pr port_pr;
+	struct platform_device *port;
+	unsigned long minsz;
+	void *buf = NULL;
+	int ret = 0;
+
+	minsz = offsetofend(struct fpga_fme_port_pr, status);
+
+	if (copy_from_user(&port_pr, argp, minsz))
+		return -EFAULT;
+
+	if (port_pr.argsz < minsz || port_pr.flags)
+		return -EINVAL;
+
+	if (!IS_ALIGNED(port_pr.buffer_size, 4))
+		return -EINVAL;
+
+	/* get fme header region */
+	fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
+					FME_FEATURE_ID_HEADER);
+	if (WARN_ON(!fme_hdr))
+		return -EINVAL;
+
+	/* check port id */
+	fme_capability.csr = readq(&fme_hdr->capability);
+	if (port_pr.port_id >= fme_capability.num_ports) {
+		dev_dbg(&pdev->dev, "port number more than maximum\n");
+		return -EINVAL;
+	}
+
+	if (!access_ok(VERIFY_READ, port_pr.buffer_address,
+				    port_pr.buffer_size))
+		return -EFAULT;
+
+	buf = vmalloc(port_pr.buffer_size);
+	if (!buf)
+		return -ENOMEM;
+
+	if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
+					       port_pr.buffer_size)) {
+		ret = -EFAULT;
+		goto free_exit;
+	}
+
+	memset(&info, 0, sizeof(struct fpga_image_info));
+	info.flags = FPGA_MGR_PARTIAL_RECONFIG;
+
+	mgr = fpga_mgr_get(&pdev->dev);
+	if (IS_ERR(mgr)) {
+		ret = PTR_ERR(mgr);
+		goto free_exit;
+	}
+
+	mutex_lock(&pdata->lock);
+	fme = fpga_pdata_get_private(pdata);
+	/* fme device has been unregistered. */
+	if (!fme) {
+		ret = -EINVAL;
+		goto unlock_exit;
+	}
+
+	fme->pr_err = 0;
+	fme->port_id = port_pr.port_id;
+
+	/* Find and get port device by index */
+	port = pdata->fpga_for_each_port(pdev, &fme->port_id,
+					 fpga_port_check_id);
+	WARN_ON(!port);
+
+	/* Disable Port before PR */
+	fpga_port_disable(port);
+
+	ret = fpga_mgr_buf_load(mgr, &info, buf, port_pr.buffer_size);
+	port_pr.status = fme->pr_err;
+
+	/* Re-enable Port after PR finished */
+	fpga_port_enable(port);
+
+	put_device(&port->dev);
+
+unlock_exit:
+	mutex_unlock(&pdata->lock);
+	fpga_mgr_put(mgr);
+free_exit:
+	vfree(buf);
+	if (copy_to_user((void __user *)arg, &port_pr, minsz))
+		return -EFAULT;
+	return ret;
+}
+
+static int fpga_fme_pr_probe(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_fme *priv;
+	int ret;
+
+	mutex_lock(&pdata->lock);
+	priv = fpga_pdata_get_private(pdata);
+	ret = fpga_mgr_register(&pdata->dev->dev,
+		"Intel FPGA Manager", &fme_pr_ops, priv);
+	mutex_unlock(&pdata->lock);
+
+	return ret;
+}
+
+static int fpga_fme_pr_remove(struct platform_device *pdev)
+{
+	fpga_mgr_unregister(&pdev->dev);
+	return 0;
+}
+
+static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
+{
+	int ret;
+
+	ret = fpga_fme_pr_probe(pdev);
+	if (ret)
+		return ret;
+
+	ret = sysfs_create_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
+	if (ret)
+		fpga_fme_pr_remove(pdev);
+
+	return ret;
+}
+
+static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
+{
+	sysfs_remove_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
+	fpga_fme_pr_remove(pdev);
+}
+
+static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
+	unsigned int cmd, unsigned long arg)
+{
+	long ret;
+
+	switch (cmd) {
+	case FPGA_FME_PORT_PR:
+		ret = fme_pr(pdev, arg);
+		break;
+	default:
+		ret = -ENODEV;
+	}
+
+	return ret;
+}
+
+struct feature_ops pr_mgmt_ops = {
+	.init = pr_mgmt_init,
+	.uinit = pr_mgmt_uinit,
+	.ioctl = fme_pr_ioctl,
+};
diff --git a/drivers/fpga/intel/fme.h b/drivers/fpga/intel/fme.h
new file mode 100644
index 0000000..d6cb7ce
--- /dev/null
+++ b/drivers/fpga/intel/fme.h
@@ -0,0 +1,32 @@
+/*
+ * Header file for Intel FPGA Management Engine (FME) Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <luwei.kang@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ *
+ * This work is licensed under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license. See the
+ * LICENSE.BSD file under this directory for the BSD license and see
+ * the COPYING file in the top-level directory for the GPLv2 license.
+ */
+
+#ifndef __INTEL_FME_H
+#define __INTEL_FME_H
+
+struct fpga_fme {
+	u8  port_id;
+	u64 pr_err;
+	struct feature_platform_data *pdata;
+};
+
+extern struct feature_ops pr_mgmt_ops;
+
+#endif
diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
index 992e556..77658316 100644
--- a/include/uapi/linux/intel-fpga.h
+++ b/include/uapi/linux/intel-fpga.h
@@ -18,6 +18,8 @@
 #ifndef _UAPI_LINUX_INTEL_FPGA_H
 #define _UAPI_LINUX_INTEL_FPGA_H
 
+#include <linux/types.h>
+
 #define FPGA_API_VERSION 0
 
 /*
@@ -30,6 +32,7 @@
 #define FPGA_MAGIC 0xB6
 
 #define FPGA_BASE 0
+#define FME_BASE 0x80
 
 /**
  * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
@@ -49,4 +52,45 @@
 
 #define FPGA_CHECK_EXTENSION	_IO(FPGA_MAGIC, FPGA_BASE + 1)
 
+/* IOCTLs for FME file descriptor */
+
+/**
+ * FPGA_FME_PORT_PR - _IOWR(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
+ *
+ * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
+ * provided by caller.
+ * Return: 0 on success, -errno on failure.
+ * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
+ * some errors during PR, under this case, the user can fetch HW error code
+ * from fpga_fme_port_pr.status. Each bit on the error code is used as the
+ * index for the array created by DEFINE_FPGA_PR_ERR_MSG().
+ * Otherwise, it is always zero.
+ */
+
+#define DEFINE_FPGA_PR_ERR_MSG(_name_)			\
+static const char * const _name_[] = {			\
+	"PR operation error detected",			\
+	"PR CRC error detected",			\
+	"PR incompatiable bitstream error detected",	\
+	"PR IP protocol error detected",		\
+	"PR FIFO overflow error detected",		\
+	"Reserved",					\
+	"PR secure load error detected",		\
+}
+
+#define PR_MAX_ERR_NUM	7
+
+struct fpga_fme_port_pr {
+	/* Input */
+	__u32 argsz;		/* Structure length */
+	__u32 flags;		/* Zero for now */
+	__u32 port_id;
+	__u32 buffer_size;
+	__u64 buffer_address;	/* Userspace address to the buffer for PR */
+	/* Output */
+	__u64 status;		/* HW error code if ioctl returns -EIO */
+};
+
+#define FPGA_FME_PORT_PR	_IO(FPGA_MAGIC, FME_BASE + 0)
+
 #endif /* _UAPI_INTEL_FPGA_H */
-- 
2.7.4

^ permalink raw reply	[flat|nested] 93+ messages in thread

* [PATCH 12/16] fpga: intel: add FPGA Accelerated Function Unit driver basic framework
  2017-03-30 12:08 [PATCH 00/16] Intel FPGA Device Drivers Wu Hao
                   ` (10 preceding siblings ...)
  2017-03-30 12:08 ` [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Wu Hao
@ 2017-03-30 12:08 ` Wu Hao
  2017-03-30 12:08 ` [PATCH 13/16] fpga: intel: afu: add header sub feature support Wu Hao
                   ` (5 subsequent siblings)
  17 siblings, 0 replies; 93+ messages in thread
From: Wu Hao @ 2017-03-30 12:08 UTC (permalink / raw)
  To: atull, moritz.fischer, linux-fpga, linux-kernel
  Cc: luwei.kang, yi.z.zhang, hao.wu, Tim Whisonant, Enno Luebbers,
	Shiva Rao, Christopher Rauer, Xiao Guangrong

On Intel FPGA devices, the Accelerated Function Unit (AFU), can be
reprogrammed for different functions. It connects to the FPGA
infrastructure("blue bistream") via a Port. Port CSRs are implemented
separately from the AFU CSRs to provide control and status of the Port.
Once valid green bitstream is programmed into the AFU, it allows access
to the AFU CSRs in the AFU MMIO space.

This patch only implements basic driver framework for AFU, including
device file operation framework.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
 drivers/fpga/intel/Kconfig    |   9 +++
 drivers/fpga/intel/Makefile   |   2 +
 drivers/fpga/intel/afu-main.c | 161 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 172 insertions(+)
 create mode 100644 drivers/fpga/intel/afu-main.c

diff --git a/drivers/fpga/intel/Kconfig b/drivers/fpga/intel/Kconfig
index 62e2160..0e7112f 100644
--- a/drivers/fpga/intel/Kconfig
+++ b/drivers/fpga/intel/Kconfig
@@ -32,4 +32,13 @@ config INTEL_FPGA_FME
 	  all FPGA platform level management features. There shall be 1
 	  FME per Intel FPGA.
 
+config INTEL_FPGA_AFU
+	tristate "Intel FPGA AFU Driver"
+	depends on INTEL_FPGA_PCI
+	help
+	  This is the driver for FPGA Accelerated Function Unit (AFU) which
+	  implements AFU and Port management features. A User AFU connects
+	  to the FPGA infrastructure via a Port. There may be more than 1
+	  Port/AFU per Intel FPGA.
+
 endif
diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
index 0452cb6..53a54ab 100644
--- a/drivers/fpga/intel/Makefile
+++ b/drivers/fpga/intel/Makefile
@@ -1,5 +1,7 @@
 obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
 obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
+obj-$(CONFIG_INTEL_FPGA_AFU) += intel-fpga-afu.o
 
 intel-fpga-pci-objs := pcie.o feature-dev.o
 intel-fpga-fme-objs := fme-main.o fme-pr.o
+intel-fpga-afu-objs := afu-main.o
diff --git a/drivers/fpga/intel/afu-main.c b/drivers/fpga/intel/afu-main.c
new file mode 100644
index 0000000..1c2035b
--- /dev/null
+++ b/drivers/fpga/intel/afu-main.c
@@ -0,0 +1,161 @@
+/*
+ * Driver for Intel FPGA Accelerated Function Unit (AFU)
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *   Joseph Grecco <joe.grecco@intel.com>
+ *   Enno Luebbers <enno.luebbers@intel.com>
+ *   Tim Whisonant <tim.whisonant@intel.com>
+ *   Ananda Ravuri <ananda.ravuri@intel.com>
+ *   Henry Mitchel <henry.mitchel@intel.com>
+ *
+ * This work is licensed under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license. See the
+ * LICENSE.BSD file under this directory for the BSD license and see
+ * the COPYING file in the top-level directory for the GPLv2 license.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+#include "feature-dev.h"
+
+static int port_hdr_init(struct platform_device *pdev, struct feature *feature)
+{
+	dev_dbg(&pdev->dev, "PORT HDR Init.\n");
+
+	return 0;
+}
+
+static void port_hdr_uinit(struct platform_device *pdev,
+					struct feature *feature)
+{
+	dev_dbg(&pdev->dev, "PORT HDR UInit.\n");
+}
+
+struct feature_ops port_hdr_ops = {
+	.init = port_hdr_init,
+	.uinit = port_hdr_uinit,
+};
+
+static struct feature_driver port_feature_drvs[] = {
+	{
+		.name = PORT_FEATURE_HEADER,
+		.ops = &port_hdr_ops,
+	},
+	{
+		.ops = NULL,
+	}
+};
+
+static int afu_open(struct inode *inode, struct file *filp)
+{
+	struct platform_device *fdev = fpga_inode_to_feature_dev(inode);
+	struct feature_platform_data *pdata;
+	int ret;
+
+	pdata = dev_get_platdata(&fdev->dev);
+	if (WARN_ON(!pdata))
+		return -ENODEV;
+
+	ret = feature_dev_use_begin(pdata);
+	if (ret)
+		return ret;
+
+	dev_dbg(&fdev->dev, "Device File Open\n");
+	filp->private_data = fdev;
+
+	return 0;
+}
+
+static int afu_release(struct inode *inode, struct file *filp)
+{
+	struct platform_device *pdev = filp->private_data;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	dev_dbg(&pdev->dev, "Device File Release\n");
+
+	feature_dev_use_end(pdata);
+	return 0;
+}
+
+static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
+{
+	struct platform_device *pdev = filp->private_data;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct feature *f;
+	long ret;
+
+	dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
+
+	switch (cmd) {
+	default:
+		/*
+		 * Let sub-feature's ioctl function to handle the cmd
+		 * Sub-feature's ioctl returns -ENODEV when cmd is not
+		 * handled in this sub feature, and returns 0 and other
+		 * error code if cmd is handled.
+		 */
+		fpga_dev_for_each_feature(pdata, f)
+			if (f->ops && f->ops->ioctl) {
+				ret = f->ops->ioctl(pdev, f, cmd, arg);
+				if (ret == -ENODEV)
+					continue;
+				else
+					return ret;
+			}
+	}
+
+	return -EINVAL;
+}
+
+static const struct file_operations afu_fops = {
+	.owner = THIS_MODULE,
+	.open = afu_open,
+	.release = afu_release,
+	.unlocked_ioctl = afu_ioctl,
+};
+
+static int afu_probe(struct platform_device *pdev)
+{
+	int ret;
+
+	dev_dbg(&pdev->dev, "%s\n", __func__);
+
+	ret = fpga_dev_feature_init(pdev, port_feature_drvs);
+	if (ret)
+		return ret;
+
+	ret = fpga_register_dev_ops(pdev, &afu_fops, THIS_MODULE);
+	if (ret)
+		fpga_dev_feature_uinit(pdev);
+
+	return ret;
+}
+
+static int afu_remove(struct platform_device *pdev)
+{
+	dev_dbg(&pdev->dev, "%s\n", __func__);
+
+	fpga_dev_feature_uinit(pdev);
+	fpga_unregister_dev_ops(pdev);
+	return 0;
+}
+
+static struct platform_driver afu_driver = {
+	.driver	= {
+		.name    = "intel-fpga-port",
+	},
+	.probe   = afu_probe,
+	.remove  = afu_remove,
+};
+
+module_platform_driver(afu_driver);
+
+MODULE_DESCRIPTION("Intel FPGA Accelerated Function Unit driver");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_LICENSE("Dual BSD/GPL");
+MODULE_ALIAS("platform:intel-fpga-port");
-- 
2.7.4

^ permalink raw reply	[flat|nested] 93+ messages in thread

* [PATCH 13/16] fpga: intel: afu: add header sub feature support
  2017-03-30 12:08 [PATCH 00/16] Intel FPGA Device Drivers Wu Hao
                   ` (11 preceding siblings ...)
  2017-03-30 12:08 ` [PATCH 12/16] fpga: intel: add FPGA Accelerated Function Unit driver basic framework Wu Hao
@ 2017-03-30 12:08 ` Wu Hao
  2017-03-30 12:08 ` [PATCH 14/16] fpga: intel: afu add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support Wu Hao
                   ` (4 subsequent siblings)
  17 siblings, 0 replies; 93+ messages in thread
From: Wu Hao @ 2017-03-30 12:08 UTC (permalink / raw)
  To: atull, moritz.fischer, linux-fpga, linux-kernel
  Cc: luwei.kang, yi.z.zhang, hao.wu, Tim Whisonant, Enno Luebbers,
	Shiva Rao, Christopher Rauer, Xiao Guangrong

The header register set is always present for the Port/AFU, it is mainly
for capability, control and status of the ports that AFU connected to.

This patch implements header sub feature support. Below user interfaces
are created by this patch.

Sysfs interface:
* /sys/class/fpga/<fpga.x>/<intel-fpga-port.x>/id
  Read-only. Port ID.

Ioctl interface:
* FPGA_PORT_RESET
  Reset the FPGA AFU Port.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
 drivers/fpga/intel/afu-main.c   | 44 ++++++++++++++++++++++++++++++++++++++++-
 include/uapi/linux/intel-fpga.h | 14 +++++++++++++
 2 files changed, 57 insertions(+), 1 deletion(-)

diff --git a/drivers/fpga/intel/afu-main.c b/drivers/fpga/intel/afu-main.c
index 1c2035b..7166d5c 100644
--- a/drivers/fpga/intel/afu-main.c
+++ b/drivers/fpga/intel/afu-main.c
@@ -20,25 +20,66 @@
 
 #include <linux/kernel.h>
 #include <linux/module.h>
+#include <linux/intel-fpga.h>
 
 #include "feature-dev.h"
 
+static ssize_t
+id_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+	int id = fpga_port_id(to_platform_device(dev));
+
+	return scnprintf(buf, PAGE_SIZE, "%d\n", id);
+}
+static DEVICE_ATTR_RO(id);
+
+static const struct attribute *port_hdr_attrs[] = {
+	&dev_attr_id.attr,
+	NULL,
+};
+
 static int port_hdr_init(struct platform_device *pdev, struct feature *feature)
 {
 	dev_dbg(&pdev->dev, "PORT HDR Init.\n");
 
-	return 0;
+	fpga_port_reset(pdev);
+
+	return sysfs_create_files(&pdev->dev.kobj, port_hdr_attrs);
 }
 
 static void port_hdr_uinit(struct platform_device *pdev,
 					struct feature *feature)
 {
 	dev_dbg(&pdev->dev, "PORT HDR UInit.\n");
+
+	sysfs_remove_files(&pdev->dev.kobj, port_hdr_attrs);
+}
+
+static long
+port_hdr_ioctl(struct platform_device *pdev, struct feature *feature,
+					unsigned int cmd, unsigned long arg)
+{
+	long ret;
+
+	switch (cmd) {
+	case FPGA_PORT_RESET:
+		if (!arg)
+			ret = fpga_port_reset(pdev);
+		else
+			ret = -EINVAL;
+		break;
+	default:
+		dev_dbg(&pdev->dev, "%x cmd not handled", cmd);
+		ret = -ENODEV;
+	}
+
+	return ret;
 }
 
 struct feature_ops port_hdr_ops = {
 	.init = port_hdr_init,
 	.uinit = port_hdr_uinit,
+	.ioctl = port_hdr_ioctl,
 };
 
 static struct feature_driver port_feature_drvs[] = {
@@ -78,6 +119,7 @@ static int afu_release(struct inode *inode, struct file *filp)
 
 	dev_dbg(&pdev->dev, "Device File Release\n");
 
+	fpga_port_reset(pdev);
 	feature_dev_use_end(pdata);
 	return 0;
 }
diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
index 77658316..13b2e61 100644
--- a/include/uapi/linux/intel-fpga.h
+++ b/include/uapi/linux/intel-fpga.h
@@ -32,8 +32,11 @@
 #define FPGA_MAGIC 0xB6
 
 #define FPGA_BASE 0
+#define PORT_BASE 0x40
 #define FME_BASE 0x80
 
+/* Common IOCTLs for both FME and AFU file descriptor */
+
 /**
  * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
  *
@@ -52,6 +55,17 @@
 
 #define FPGA_CHECK_EXTENSION	_IO(FPGA_MAGIC, FPGA_BASE + 1)
 
+/* IOCTLs for AFU file descriptor */
+
+/**
+ * FPGA_PORT_RESET - _IO(FPGA_MAGIC, PORT_BASE + 0)
+ *
+ * Reset the FPGA AFU Port. No parameters are supported.
+ * Return: 0 on success, -errno of failure
+ */
+
+#define FPGA_PORT_RESET		_IO(FPGA_MAGIC, PORT_BASE + 0)
+
 /* IOCTLs for FME file descriptor */
 
 /**
-- 
2.7.4

^ permalink raw reply	[flat|nested] 93+ messages in thread

* [PATCH 14/16] fpga: intel: afu add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support
  2017-03-30 12:08 [PATCH 00/16] Intel FPGA Device Drivers Wu Hao
                   ` (12 preceding siblings ...)
  2017-03-30 12:08 ` [PATCH 13/16] fpga: intel: afu: add header sub feature support Wu Hao
@ 2017-03-30 12:08 ` Wu Hao
  2017-03-30 12:08 ` [PATCH 15/16] fpga: intel: afu: add user afu sub feature support Wu Hao
                   ` (3 subsequent siblings)
  17 siblings, 0 replies; 93+ messages in thread
From: Wu Hao @ 2017-03-30 12:08 UTC (permalink / raw)
  To: atull, moritz.fischer, linux-fpga, linux-kernel
  Cc: luwei.kang, yi.z.zhang, hao.wu, Tim Whisonant, Enno Luebbers,
	Shiva Rao, Christopher Rauer, Xiao Guangrong

FPGA_GET_API_VERSION and FPGA_CHECK_EXTENSION ioctls are common ones which
need to be supported by all feature devices drivers including FME and AFU.
This patch implements above 2 ioctls in Intel FPGA Accelerated Function
Unit (AFU) driver.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
 drivers/fpga/intel/afu-main.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/fpga/intel/afu-main.c b/drivers/fpga/intel/afu-main.c
index 7166d5c..89d4b2f 100644
--- a/drivers/fpga/intel/afu-main.c
+++ b/drivers/fpga/intel/afu-main.c
@@ -124,6 +124,13 @@ static int afu_release(struct inode *inode, struct file *filp)
 	return 0;
 }
 
+static long afu_ioctl_check_extension(struct feature_platform_data *pdata,
+				     unsigned long arg)
+{
+	/* No extension support for now */
+	return 0;
+}
+
 static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 {
 	struct platform_device *pdev = filp->private_data;
@@ -134,6 +141,10 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 	dev_dbg(&pdev->dev, "%s cmd 0x%x\n", __func__, cmd);
 
 	switch (cmd) {
+	case FPGA_GET_API_VERSION:
+		return FPGA_API_VERSION;
+	case FPGA_CHECK_EXTENSION:
+		return afu_ioctl_check_extension(pdata, arg);
 	default:
 		/*
 		 * Let sub-feature's ioctl function to handle the cmd
-- 
2.7.4

^ permalink raw reply	[flat|nested] 93+ messages in thread

* [PATCH 15/16] fpga: intel: afu: add user afu sub feature support
  2017-03-30 12:08 [PATCH 00/16] Intel FPGA Device Drivers Wu Hao
                   ` (13 preceding siblings ...)
  2017-03-30 12:08 ` [PATCH 14/16] fpga: intel: afu add FPGA_GET_API_VERSION/CHECK_EXTENSION ioctls support Wu Hao
@ 2017-03-30 12:08 ` Wu Hao
  2017-03-30 12:08 ` [PATCH 16/16] fpga: intel: afu: add FPGA_PORT_DMA_MAP/UNMAP ioctls support Wu Hao
                   ` (2 subsequent siblings)
  17 siblings, 0 replies; 93+ messages in thread
From: Wu Hao @ 2017-03-30 12:08 UTC (permalink / raw)
  To: atull, moritz.fischer, linux-fpga, linux-kernel
  Cc: luwei.kang, yi.z.zhang, hao.wu, Xiao Guangrong, Tim Whisonant,
	Enno Luebbers, Shiva Rao, Christopher Rauer

From: Xiao Guangrong <guangrong.xiao@linux.intel.com>

User Accelerated Function Unit sub feature exposes the MMIO region of
the AFU. After valid green bitstream (GBS) is programmed and port is
enabled, then this MMIO region could be accessed.

This patch adds support to enumerate the AFU MMIO region and expose it
to userspace via mmap file operation. Below interfaces are exposed to user:

Sysfs interface:
* /sys/class/fpga/<fpga.x>/<intel-fpga-port.x>/afu_id
  Read-only. Indicate which green bitstream is programmed to this AFU.

Ioctl interfaces:
* FPGA_PORT_GET_INFO
  Provide info to userspace on the number of supported region.
  Only UAFU region is supported now.

* FPGA_PORT_GET_REGION_INFO
  Provide region information, including access permission, region size,
  offset from the start of device fd.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
 drivers/fpga/intel/Makefile     |   2 +-
 drivers/fpga/intel/afu-main.c   | 204 +++++++++++++++++++++++++++++++++++++++-
 drivers/fpga/intel/afu-region.c | 129 +++++++++++++++++++++++++
 drivers/fpga/intel/afu.h        |  54 +++++++++++
 include/uapi/linux/intel-fpga.h |  47 +++++++++
 5 files changed, 432 insertions(+), 4 deletions(-)
 create mode 100644 drivers/fpga/intel/afu-region.c
 create mode 100644 drivers/fpga/intel/afu.h

diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
index 53a54ab..5c33216 100644
--- a/drivers/fpga/intel/Makefile
+++ b/drivers/fpga/intel/Makefile
@@ -4,4 +4,4 @@ obj-$(CONFIG_INTEL_FPGA_AFU) += intel-fpga-afu.o
 
 intel-fpga-pci-objs := pcie.o feature-dev.o
 intel-fpga-fme-objs := fme-main.o fme-pr.o
-intel-fpga-afu-objs := afu-main.o
+intel-fpga-afu-objs := afu-main.o afu-region.o
diff --git a/drivers/fpga/intel/afu-main.c b/drivers/fpga/intel/afu-main.c
index 89d4b2f..db2aec3 100644
--- a/drivers/fpga/intel/afu-main.c
+++ b/drivers/fpga/intel/afu-main.c
@@ -20,9 +20,10 @@
 
 #include <linux/kernel.h>
 #include <linux/module.h>
+#include <linux/uaccess.h>
 #include <linux/intel-fpga.h>
 
-#include "feature-dev.h"
+#include "afu.h"
 
 static ssize_t
 id_show(struct device *dev, struct device_attribute *attr, char *buf)
@@ -82,12 +83,69 @@ struct feature_ops port_hdr_ops = {
 	.ioctl = port_hdr_ioctl,
 };
 
+static ssize_t
+afu_id_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(dev);
+	struct feature_port_header *port_hdr =
+			get_feature_ioaddr_by_index(dev, PORT_FEATURE_ID_UAFU);
+	u64 guidl;
+	u64 guidh;
+
+	mutex_lock(&pdata->lock);
+	guidl = readq(&port_hdr->afu_header.guid.b[0]);
+	guidh = readq(&port_hdr->afu_header.guid.b[8]);
+	mutex_unlock(&pdata->lock);
+
+	return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n", guidh, guidl);
+}
+static DEVICE_ATTR_RO(afu_id);
+
+static const struct attribute *port_uafu_attrs[] = {
+	&dev_attr_afu_id.attr,
+	NULL
+};
+
+static int port_uafu_init(struct platform_device *pdev, struct feature *feature)
+{
+	struct resource *res = &pdev->resource[feature->resource_index];
+	u32 flags = FPGA_REGION_READ | FPGA_REGION_WRITE | FPGA_REGION_MMAP;
+	int ret;
+
+	dev_dbg(&pdev->dev, "PORT AFU Init.\n");
+
+	ret = afu_region_add(dev_get_platdata(&pdev->dev),
+			     FPGA_PORT_INDEX_UAFU, resource_size(res),
+			     res->start, flags);
+	if (ret)
+		return ret;
+
+	return sysfs_create_files(&pdev->dev.kobj, port_uafu_attrs);
+}
+
+static void port_uafu_uinit(struct platform_device *pdev,
+					struct feature *feature)
+{
+	dev_dbg(&pdev->dev, "PORT AFU UInit.\n");
+
+	sysfs_remove_files(&pdev->dev.kobj, port_uafu_attrs);
+}
+
+struct feature_ops port_uafu_ops = {
+	.init = port_uafu_init,
+	.uinit = port_uafu_uinit,
+};
+
 static struct feature_driver port_feature_drvs[] = {
 	{
 		.name = PORT_FEATURE_HEADER,
 		.ops = &port_hdr_ops,
 	},
 	{
+		.name = PORT_FEATURE_UAFU,
+		.ops = &port_uafu_ops,
+	},
+	{
 		.ops = NULL,
 	}
 };
@@ -131,6 +189,64 @@ static long afu_ioctl_check_extension(struct feature_platform_data *pdata,
 	return 0;
 }
 
+static long
+afu_ioctl_get_info(struct feature_platform_data *pdata, void __user *arg)
+{
+	struct fpga_port_info info;
+	struct fpga_afu *afu;
+	unsigned long minsz;
+
+	minsz = offsetofend(struct fpga_port_info, num_umsgs);
+
+	if (copy_from_user(&info, arg, minsz))
+		return -EFAULT;
+
+	if (info.argsz < minsz)
+		return -EINVAL;
+
+	mutex_lock(&pdata->lock);
+	afu = fpga_pdata_get_private(pdata);
+	info.flags = 0;
+	info.num_regions = afu->num_regions;
+	info.num_umsgs = afu->num_umsgs;
+	mutex_unlock(&pdata->lock);
+
+	if (copy_to_user(arg, &info, sizeof(info)))
+		return -EFAULT;
+
+	return 0;
+}
+
+static long
+afu_ioctl_get_region_info(struct feature_platform_data *pdata, void __user *arg)
+{
+	struct fpga_port_region_info rinfo;
+	struct fpga_afu_region region;
+	unsigned long minsz;
+	long ret;
+
+	minsz = offsetofend(struct fpga_port_region_info, offset);
+
+	if (copy_from_user(&rinfo, arg, minsz))
+		return -EFAULT;
+
+	if (rinfo.argsz < minsz || rinfo.padding)
+		return -EINVAL;
+
+	ret = afu_get_region_by_index(pdata, rinfo.index, &region);
+	if (ret)
+		return ret;
+
+	rinfo.flags = region.flags;
+	rinfo.size = region.size;
+	rinfo.offset = region.offset;
+
+	if (copy_to_user(arg, &rinfo, sizeof(rinfo)))
+		return -EFAULT;
+
+	return 0;
+}
+
 static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 {
 	struct platform_device *pdev = filp->private_data;
@@ -145,6 +261,10 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 		return FPGA_API_VERSION;
 	case FPGA_CHECK_EXTENSION:
 		return afu_ioctl_check_extension(pdata, arg);
+	case FPGA_PORT_GET_INFO:
+		return afu_ioctl_get_info(pdata, (void __user *)arg);
+	case FPGA_PORT_GET_REGION_INFO:
+		return afu_ioctl_get_region_info(pdata, (void __user *)arg);
 	default:
 		/*
 		 * Let sub-feature's ioctl function to handle the cmd
@@ -165,27 +285,104 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 	return -EINVAL;
 }
 
+static int afu_mmap(struct file *filp, struct vm_area_struct *vma)
+{
+	struct fpga_afu_region region;
+	struct platform_device *pdev = filp->private_data;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	u64 size = vma->vm_end - vma->vm_start;
+	u64 offset;
+	int ret;
+
+	if (!(vma->vm_flags & VM_SHARED))
+		return -EINVAL;
+
+	offset = vma->vm_pgoff << PAGE_SHIFT;
+	ret = afu_get_region_by_offset(pdata, offset, size, &region);
+	if (ret)
+		return ret;
+
+	if (!(region.flags & FPGA_REGION_MMAP))
+		return -EINVAL;
+
+	if ((vma->vm_flags & VM_READ) && !(region.flags & FPGA_REGION_READ))
+		return -EPERM;
+
+	if ((vma->vm_flags & VM_WRITE) && !(region.flags & FPGA_REGION_WRITE))
+		return -EPERM;
+
+	vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
+	return remap_pfn_range(vma, vma->vm_start,
+			(region.phys + (offset - region.offset)) >> PAGE_SHIFT,
+			size, vma->vm_page_prot);
+}
+
 static const struct file_operations afu_fops = {
 	.owner = THIS_MODULE,
 	.open = afu_open,
 	.release = afu_release,
 	.unlocked_ioctl = afu_ioctl,
+	.mmap = afu_mmap,
 };
 
+static int afu_dev_init(struct platform_device *pdev)
+{
+	struct fpga_afu *afu;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+	afu = devm_kzalloc(&pdev->dev, sizeof(*afu), GFP_KERNEL);
+	if (!afu)
+		return -ENOMEM;
+
+	afu->pdata = pdata;
+
+	mutex_lock(&pdata->lock);
+	fpga_pdata_set_private(pdata, afu);
+	afu_region_init(pdata);
+	mutex_unlock(&pdata->lock);
+	return 0;
+}
+
+static int afu_dev_destroy(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_afu *afu;
+
+	mutex_lock(&pdata->lock);
+	afu = fpga_pdata_get_private(pdata);
+	afu_region_destroy(pdata);
+	fpga_pdata_set_private(pdata, NULL);
+	mutex_unlock(&pdata->lock);
+
+	devm_kfree(&pdev->dev, afu);
+	return 0;
+}
+
 static int afu_probe(struct platform_device *pdev)
 {
 	int ret;
 
 	dev_dbg(&pdev->dev, "%s\n", __func__);
 
+	ret = afu_dev_init(pdev);
+	if (ret)
+		goto exit;
+
 	ret = fpga_dev_feature_init(pdev, port_feature_drvs);
 	if (ret)
-		return ret;
+		goto dev_destroy;
 
 	ret = fpga_register_dev_ops(pdev, &afu_fops, THIS_MODULE);
-	if (ret)
+	if (ret) {
 		fpga_dev_feature_uinit(pdev);
+		goto dev_destroy;
+	}
+
+	return 0;
 
+dev_destroy:
+	afu_dev_destroy(pdev);
+exit:
 	return ret;
 }
 
@@ -195,6 +392,7 @@ static int afu_remove(struct platform_device *pdev)
 
 	fpga_dev_feature_uinit(pdev);
 	fpga_unregister_dev_ops(pdev);
+	afu_dev_destroy(pdev);
 	return 0;
 }
 
diff --git a/drivers/fpga/intel/afu-region.c b/drivers/fpga/intel/afu-region.c
new file mode 100644
index 0000000..1eec08f2
--- /dev/null
+++ b/drivers/fpga/intel/afu-region.c
@@ -0,0 +1,129 @@
+/*
+ * Driver for Intel FPGA Accelerated Function Unit (AFU) Region Management
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *
+ * This work is licensed under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license. See the
+ * LICENSE.BSD file under this directory for the BSD license and see
+ * the COPYING file in the top-level directory for the GPLv2 license.
+ */
+
+#include "afu.h"
+
+void afu_region_init(struct feature_platform_data *pdata)
+{
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+
+	INIT_LIST_HEAD(&afu->regions);
+}
+
+#define for_each_region(region, afu)	\
+	list_for_each_entry((region), &(afu)->regions, node)
+static struct fpga_afu_region *get_region_by_index(struct fpga_afu *afu,
+						   u32 region_index)
+{
+	struct fpga_afu_region *region;
+
+	for_each_region(region, afu)
+		if (region->index == region_index)
+			return region;
+
+	return NULL;
+}
+
+int afu_region_add(struct feature_platform_data *pdata, u32 region_index,
+		   u64 region_size, u64 phys, u32 flags)
+{
+	struct fpga_afu_region *region;
+	struct fpga_afu *afu;
+	int ret = 0;
+
+	region = devm_kzalloc(&pdata->dev->dev, sizeof(*region), GFP_KERNEL);
+	if (!region)
+		return -ENOMEM;
+
+	region->index = region_index;
+	region->size = region_size;
+	region->phys = phys;
+	region->flags = flags;
+
+	mutex_lock(&pdata->lock);
+
+	afu = fpga_pdata_get_private(pdata);
+
+	/* check if @index already exists */
+	if (get_region_by_index(afu, region_index)) {
+		mutex_unlock(&pdata->lock);
+		ret = -EEXIST;
+		goto exit;
+	}
+
+	region_size = PAGE_ALIGN(region_size);
+	region->offset = afu->region_cur_offset;
+	list_add(&region->node, &afu->regions);
+
+	afu->region_cur_offset += region_size;
+	afu->num_regions++;
+	mutex_unlock(&pdata->lock);
+	return 0;
+
+exit:
+	devm_kfree(&pdata->dev->dev, region);
+	return ret;
+}
+
+void afu_region_destroy(struct feature_platform_data *pdata)
+{
+	struct fpga_afu_region *tmp, *region;
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+
+	list_for_each_entry_safe(region, tmp, &afu->regions, node)
+		devm_kfree(&pdata->dev->dev, region);
+}
+
+int afu_get_region_by_index(struct feature_platform_data *pdata,
+			    u32 region_index, struct fpga_afu_region *pregion)
+{
+	struct fpga_afu_region *region;
+	struct fpga_afu *afu;
+	int ret = 0;
+
+	mutex_lock(&pdata->lock);
+	afu = fpga_pdata_get_private(pdata);
+	region = get_region_by_index(afu, region_index);
+	if (!region) {
+		ret = -EINVAL;
+		goto exit;
+	}
+	*pregion = *region;
+exit:
+	mutex_unlock(&pdata->lock);
+	return ret;
+}
+
+int afu_get_region_by_offset(struct feature_platform_data *pdata,
+			    u64 offset, u64 size,
+			    struct fpga_afu_region *pregion)
+{
+	struct fpga_afu_region *region;
+	struct fpga_afu *afu;
+	int ret = 0;
+
+	mutex_lock(&pdata->lock);
+	afu = fpga_pdata_get_private(pdata);
+	for_each_region(region, afu)
+		if (region->offset <= offset &&
+		   region->offset + region->size >= offset + size) {
+			*pregion = *region;
+			goto exit;
+		}
+	ret = -EINVAL;
+exit:
+	mutex_unlock(&pdata->lock);
+	return ret;
+}
diff --git a/drivers/fpga/intel/afu.h b/drivers/fpga/intel/afu.h
new file mode 100644
index 0000000..fca4dbc
--- /dev/null
+++ b/drivers/fpga/intel/afu.h
@@ -0,0 +1,54 @@
+/*
+ * Header file for Intel FPGA Accelerated Function Unit (AFU) Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *     Wu Hao <hao.wu@intel.com>
+ *     Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *     Joseph Grecco <joe.grecco@intel.com>
+ *     Enno Luebbers <enno.luebbers@intel.com>
+ *     Tim Whisonant <tim.whisonant@intel.com>
+ *     Ananda Ravuri <ananda.ravuri@intel.com>
+ *     Henry Mitchel <henry.mitchel@intel.com>
+ *
+ * This work is licensed under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license. See the
+ * LICENSE.BSD file under this directory for the BSD license and see
+ * the COPYING file in the top-level directory for the GPLv2 license.
+ */
+
+#ifndef __INTEL_AFU_H
+#define __INTEL_AFU_H
+
+#include "feature-dev.h"
+
+struct fpga_afu_region {
+	u32 index;
+	u32 flags;
+	u64 size;
+	u64 offset;
+	u64 phys;
+	struct list_head node;
+};
+
+struct fpga_afu {
+	u64 region_cur_offset;
+	int num_regions;
+	u8 num_umsgs;
+	struct list_head regions;
+
+	struct feature_platform_data *pdata;
+};
+
+void afu_region_init(struct feature_platform_data *pdata);
+int afu_region_add(struct feature_platform_data *pdata, u32 region_index,
+		   u64 region_size, u64 phys, u32 flags);
+void afu_region_destroy(struct feature_platform_data *pdata);
+int afu_get_region_by_index(struct feature_platform_data *pdata,
+			    u32 region_index, struct fpga_afu_region *pregion);
+int afu_get_region_by_offset(struct feature_platform_data *pdata,
+			    u64 offset, u64 size,
+			    struct fpga_afu_region *pregion);
+
+#endif
diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
index 13b2e61..86a5168 100644
--- a/include/uapi/linux/intel-fpga.h
+++ b/include/uapi/linux/intel-fpga.h
@@ -66,6 +66,53 @@
 
 #define FPGA_PORT_RESET		_IO(FPGA_MAGIC, PORT_BASE + 0)
 
+/**
+ * FPGA_PORT_GET_INFO - _IOR(FPGA_MAGIC, PORT_BASE + 1, struct fpga_port_info)
+ *
+ * Retrieve information about the fpga port.
+ * Driver fills the info in provided struct fpga_port_info.
+ * Return: 0 on success, -errno on failure.
+ */
+struct fpga_port_info {
+	/* Input */
+	__u32 argsz;		/* Structure length */
+	/* Output */
+	__u32 flags;		/* Zero for now */
+	__u32 num_regions;	/* The number of supported regions */
+	__u32 num_umsgs;	/* The number of allocated umsgs */
+};
+
+#define FPGA_PORT_GET_INFO	_IO(FPGA_MAGIC, PORT_BASE + 1)
+
+/**
+ * FPGA_PORT_GET_REGION_INFO - _IOWR(FPGA_MAGIC, PORT_BASE + 2,
+ *						struct fpga_port_region_info)
+ *
+ * Retrieve information about a device region.
+ * Caller provides struct fpga_port_region_info with index value set.
+ * Driver returns the region info in other fields.
+ * Return: 0 on success, -errno on failure.
+ */
+struct fpga_port_region_info {
+	/* input */
+	__u32 argsz;		/* Structure length */
+	/* Output */
+	__u32 flags;		/* Access permission */
+#define FPGA_REGION_READ	(1 << 0)	/* Region is readable */
+#define FPGA_REGION_WRITE	(1 << 1)	/* Region is writable */
+#define FPGA_REGION_MMAP	(1 << 2)	/* Can be mmaped to userspace */
+	/* Input */
+	__u32 index;		/* Region index */
+#define FPGA_PORT_INDEX_UAFU	0		/* User AFU */
+#define FPGA_PORT_INDEX_STP	1		/* Signal Tap */
+	__u32 padding;
+	/* Output */
+	__u64 size;		/* Region size (bytes) */
+	__u64 offset;		/* Region offset from start of device fd */
+};
+
+#define FPGA_PORT_GET_REGION_INFO	_IO(FPGA_MAGIC, PORT_BASE + 2)
+
 /* IOCTLs for FME file descriptor */
 
 /**
-- 
2.7.4

^ permalink raw reply	[flat|nested] 93+ messages in thread

* [PATCH 16/16] fpga: intel: afu: add FPGA_PORT_DMA_MAP/UNMAP ioctls support
  2017-03-30 12:08 [PATCH 00/16] Intel FPGA Device Drivers Wu Hao
                   ` (14 preceding siblings ...)
  2017-03-30 12:08 ` [PATCH 15/16] fpga: intel: afu: add user afu sub feature support Wu Hao
@ 2017-03-30 12:08 ` Wu Hao
  2017-04-01  0:00   ` kbuild test robot
  2017-04-01  1:33   ` kbuild test robot
  2017-03-30 17:17 ` [PATCH 00/16] Intel FPGA Device Drivers Moritz Fischer
  2017-04-06 20:27 ` Jerome Glisse
  17 siblings, 2 replies; 93+ messages in thread
From: Wu Hao @ 2017-03-30 12:08 UTC (permalink / raw)
  To: atull, moritz.fischer, linux-fpga, linux-kernel
  Cc: luwei.kang, yi.z.zhang, hao.wu, Tim Whisonant, Enno Luebbers,
	Shiva Rao, Christopher Rauer, Xiao Guangrong

DMA memory regions are required for Accelerated Function Unit (AFU) usage.
These two ioctls allow user space applications to map user memory regions
for dma, and unmap them after use. Iova is returned from driver to user
space application via FPGA_PORT_DMA_MAP ioctl. Application needs to unmap
it after use, otherwise, driver will unmap them in device file release
operation.

All the mapped regions are managed via a rb tree.

Ioctl interfaces:
* FPGA_PORT_DMA_MAP
  Do the dma mapping per user_addr and length which provided by user.
  Return iova in provided struct afu_port_dma_map.

* FPGA_PORT_DMA_UNMAP
  Unmap the dma region per iova provided by user.

Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
Signed-off-by: Shiva Rao <shiva.rao@intel.com>
Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Signed-off-by: Wu Hao <hao.wu@intel.com>
---
 drivers/fpga/intel/Makefile         |   2 +-
 drivers/fpga/intel/afu-dma-region.c | 373 ++++++++++++++++++++++++++++++++++++
 drivers/fpga/intel/afu-main.c       |  61 +++++-
 drivers/fpga/intel/afu.h            |  18 ++
 include/uapi/linux/intel-fpga.h     |  37 ++++
 5 files changed, 489 insertions(+), 2 deletions(-)
 create mode 100644 drivers/fpga/intel/afu-dma-region.c

diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
index 5c33216..26ef583 100644
--- a/drivers/fpga/intel/Makefile
+++ b/drivers/fpga/intel/Makefile
@@ -4,4 +4,4 @@ obj-$(CONFIG_INTEL_FPGA_AFU) += intel-fpga-afu.o
 
 intel-fpga-pci-objs := pcie.o feature-dev.o
 intel-fpga-fme-objs := fme-main.o fme-pr.o
-intel-fpga-afu-objs := afu-main.o afu-region.o
+intel-fpga-afu-objs := afu-main.o afu-region.o afu-dma-region.o
diff --git a/drivers/fpga/intel/afu-dma-region.c b/drivers/fpga/intel/afu-dma-region.c
new file mode 100644
index 0000000..5a525f1
--- /dev/null
+++ b/drivers/fpga/intel/afu-dma-region.c
@@ -0,0 +1,373 @@
+/*
+ * Driver for Intel FPGA Accelerated Function Unit (AFU) DMA Region Management
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Wu Hao <hao.wu@intel.com>
+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
+ *
+ * This work is licensed under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license. See the
+ * LICENSE.BSD file under this directory for the BSD license and see
+ * the COPYING file in the top-level directory for the GPLv2 license.
+ */
+
+#include <linux/sched/signal.h>
+#include <linux/uaccess.h>
+
+#include "afu.h"
+
+static void put_all_pages(struct page **pages, int npages)
+{
+	int i;
+
+	for (i = 0; i < npages; i++)
+		if (pages[i] != NULL)
+			put_page(pages[i]);
+}
+
+void afu_dma_region_init(struct feature_platform_data *pdata)
+{
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+
+	afu->dma_regions = RB_ROOT;
+}
+
+static long afu_dma_adjust_locked_vm(struct device *dev, long npages, bool incr)
+{
+	unsigned long locked, lock_limit;
+	int ret = 0;
+
+	/* the task is exiting. */
+	if (!current->mm)
+		return 0;
+
+	down_write(&current->mm->mmap_sem);
+
+	if (incr) {
+		locked = current->mm->locked_vm + npages;
+		lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
+
+		if (locked > lock_limit && !capable(CAP_IPC_LOCK))
+			ret = -ENOMEM;
+		else
+			current->mm->locked_vm += npages;
+	} else {
+
+		if (WARN_ON_ONCE(npages > current->mm->locked_vm))
+			npages = current->mm->locked_vm;
+		current->mm->locked_vm -= npages;
+	}
+
+	dev_dbg(dev, "[%d] RLIMIT_MEMLOCK %c%ld %ld/%ld%s\n", current->pid,
+				incr ? '+' : '-',
+				npages << PAGE_SHIFT,
+				current->mm->locked_vm << PAGE_SHIFT,
+				rlimit(RLIMIT_MEMLOCK),
+				ret ? "- execeeded" : "");
+
+	up_write(&current->mm->mmap_sem);
+
+	return ret;
+}
+
+static long afu_dma_pin_pages(struct feature_platform_data *pdata,
+				struct fpga_afu_dma_region *region)
+{
+	long npages = region->length >> PAGE_SHIFT;
+	struct device *dev = &pdata->dev->dev;
+	long ret, pinned;
+
+	ret = afu_dma_adjust_locked_vm(dev, npages, true);
+	if (ret)
+		return ret;
+
+	region->pages = kcalloc(npages, sizeof(struct page *), GFP_KERNEL);
+	if (!region->pages) {
+		afu_dma_adjust_locked_vm(dev, npages, false);
+		return -ENOMEM;
+	}
+
+	pinned = get_user_pages_fast(region->user_addr, npages, 1,
+					region->pages);
+	if (pinned < 0) {
+		ret = pinned;
+		goto err_put_pages;
+	} else if (pinned != npages) {
+		ret = -EFAULT;
+		goto err;
+	}
+
+	dev_dbg(dev, "%ld pages pinned\n", pinned);
+
+	return 0;
+
+err_put_pages:
+	put_all_pages(region->pages, pinned);
+err:
+	kfree(region->pages);
+	afu_dma_adjust_locked_vm(dev, npages, false);
+	return ret;
+}
+
+static void afu_dma_unpin_pages(struct feature_platform_data *pdata,
+				struct fpga_afu_dma_region *region)
+{
+	long npages = region->length >> PAGE_SHIFT;
+	struct device *dev = &pdata->dev->dev;
+
+	put_all_pages(region->pages, npages);
+	kfree(region->pages);
+	afu_dma_adjust_locked_vm(dev, npages, false);
+
+	dev_dbg(dev, "%ld pages unpinned\n", npages);
+}
+
+static bool afu_dma_check_continuous_pages(struct fpga_afu_dma_region *region)
+{
+	int npages = region->length >> PAGE_SHIFT;
+	int i;
+
+	for (i = 0; i < npages - 1; i++)
+		if (page_to_pfn(region->pages[i]) + 1 !=
+					page_to_pfn(region->pages[i+1]))
+			return false;
+
+	return true;
+}
+
+static bool dma_region_check_iova(struct fpga_afu_dma_region *region,
+				  u64 iova, u64 size)
+{
+	if (!size && region->iova != iova)
+		return false;
+
+	return (region->iova <= iova) &&
+		(region->length + region->iova >= iova + size);
+}
+
+/* Need to be called with pdata->lock held */
+static int afu_dma_region_add(struct feature_platform_data *pdata,
+					struct fpga_afu_dma_region *region)
+{
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+	struct rb_node **new, *parent = NULL;
+
+	dev_dbg(&pdata->dev->dev, "add region (iova = %llx)\n",
+					(unsigned long long)region->iova);
+
+	new = &(afu->dma_regions.rb_node);
+
+	while (*new) {
+		struct fpga_afu_dma_region *this;
+
+		this = container_of(*new, struct fpga_afu_dma_region, node);
+
+		parent = *new;
+
+		if (dma_region_check_iova(this, region->iova, region->length))
+			return -EEXIST;
+
+		if (region->iova < this->iova)
+			new = &((*new)->rb_left);
+		else if (region->iova > this->iova)
+			new = &((*new)->rb_right);
+		else
+			return -EEXIST;
+	}
+
+	rb_link_node(&region->node, parent, new);
+	rb_insert_color(&region->node, &afu->dma_regions);
+
+	return 0;
+}
+
+/* Need to be called with pdata->lock held */
+static void afu_dma_region_remove(struct feature_platform_data *pdata,
+					struct fpga_afu_dma_region *region)
+{
+	struct fpga_afu *afu;
+
+	dev_dbg(&pdata->dev->dev, "del region (iova = %llx)\n",
+					(unsigned long long)region->iova);
+
+	afu = fpga_pdata_get_private(pdata);
+	rb_erase(&region->node, &afu->dma_regions);
+}
+
+/* Need to be called with pdata->lock held */
+void afu_dma_region_destroy(struct feature_platform_data *pdata)
+{
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+	struct rb_node *node = rb_first(&afu->dma_regions);
+	struct fpga_afu_dma_region *region;
+
+	while (node) {
+		region = container_of(node, struct fpga_afu_dma_region, node);
+
+		dev_dbg(&pdata->dev->dev, "del region (iova = %llx)\n",
+					(unsigned long long)region->iova);
+
+		rb_erase(node, &afu->dma_regions);
+
+		if (region->iova)
+			dma_unmap_page(fpga_pdata_to_pcidev(pdata),
+					region->iova, region->length,
+					DMA_BIDIRECTIONAL);
+
+		if (region->pages)
+			afu_dma_unpin_pages(pdata, region);
+
+		node = rb_next(node);
+		kfree(region);
+	}
+}
+
+/*
+ * It finds the dma region from the rbtree based on @iova and @size:
+ * - if @size == 0, it finds the dma region which starts from @iova
+ * - otherwise, it finds the dma region which fully contains
+ *   [@iova, @iova+size)
+ * If nothing is matched returns NULL.
+ *
+ * Need to be called with pdata->lock held.
+ */
+struct fpga_afu_dma_region *
+afu_dma_region_find(struct feature_platform_data *pdata, u64 iova, u64 size)
+{
+	struct fpga_afu *afu = fpga_pdata_get_private(pdata);
+	struct rb_node *node = afu->dma_regions.rb_node;
+	struct device *dev = &pdata->dev->dev;
+
+	while (node) {
+		struct fpga_afu_dma_region *region;
+
+		region = container_of(node, struct fpga_afu_dma_region, node);
+
+		if (dma_region_check_iova(region, iova, size)) {
+			dev_dbg(dev, "find region (iova = %llx)\n",
+				(unsigned long long)region->iova);
+			return region;
+		}
+
+		if (iova < region->iova)
+			node = node->rb_left;
+		else if (iova > region->iova)
+			node = node->rb_right;
+		else
+			/* the iova region is not fully covered. */
+			break;
+	}
+
+	dev_dbg(dev, "region with iova %llx and size %llx is not found\n",
+		(unsigned long long)iova, (unsigned long long)size);
+	return NULL;
+}
+
+static struct fpga_afu_dma_region *
+afu_dma_region_find_iova(struct feature_platform_data *pdata, u64 iova)
+{
+	return afu_dma_region_find(pdata, iova, 0);
+}
+
+long afu_dma_map_region(struct feature_platform_data *pdata,
+		       u64 user_addr, u64 length, u64 *iova)
+{
+	struct fpga_afu_dma_region *region;
+	int ret;
+
+	/*
+	 * Check Inputs, only accept page-aligned user memory region with
+	 * valid length.
+	 */
+	if (!PAGE_ALIGNED(user_addr) || !PAGE_ALIGNED(length) || !length)
+		return -EINVAL;
+
+	/* Check overflow */
+	if (user_addr + length < user_addr)
+		return -EINVAL;
+
+	if (!access_ok(VERIFY_WRITE, user_addr, length))
+		return -EINVAL;
+
+	region = kzalloc(sizeof(*region), GFP_KERNEL);
+	if (!region)
+		return -ENOMEM;
+
+	region->user_addr = user_addr;
+	region->length = length;
+
+	/* Pin the user memory region */
+	ret = afu_dma_pin_pages(pdata, region);
+	if (ret) {
+		dev_err(&pdata->dev->dev, "fail to pin memory region\n");
+		goto free_region;
+	}
+
+	/* Only accept continuous pages, return error if no */
+	if (!afu_dma_check_continuous_pages(region)) {
+		dev_err(&pdata->dev->dev, "pages are not continuous\n");
+		ret = -EINVAL;
+		goto unpin_pages;
+	}
+
+	/* As pages are continuous then start to do DMA mapping */
+	region->iova = dma_map_page(fpga_pdata_to_pcidev(pdata),
+				    region->pages[0], 0,
+				    region->length,
+				    DMA_BIDIRECTIONAL);
+	if (dma_mapping_error(&pdata->dev->dev, region->iova)) {
+		dev_err(&pdata->dev->dev, "fail to map dma mapping\n");
+		ret = -EFAULT;
+		goto unpin_pages;
+	}
+
+	*iova = region->iova;
+
+	mutex_lock(&pdata->lock);
+	ret = afu_dma_region_add(pdata, region);
+	mutex_unlock(&pdata->lock);
+	if (ret) {
+		dev_err(&pdata->dev->dev, "fail to add dma region\n");
+		goto unmap_dma;
+	}
+
+	return 0;
+
+unmap_dma:
+	dma_unmap_page(fpga_pdata_to_pcidev(pdata),
+		       region->iova, region->length, DMA_BIDIRECTIONAL);
+unpin_pages:
+	afu_dma_unpin_pages(pdata, region);
+free_region:
+	kfree(region);
+	return ret;
+}
+
+long afu_dma_unmap_region(struct feature_platform_data *pdata, u64 iova)
+{
+	struct fpga_afu_dma_region *region;
+
+	mutex_lock(&pdata->lock);
+	region = afu_dma_region_find_iova(pdata, iova);
+	if (!region) {
+		mutex_unlock(&pdata->lock);
+		return -EINVAL;
+	}
+
+	if (region->in_use) {
+		mutex_unlock(&pdata->lock);
+		return -EBUSY;
+	}
+
+	afu_dma_region_remove(pdata, region);
+	mutex_unlock(&pdata->lock);
+
+	dma_unmap_page(fpga_pdata_to_pcidev(pdata),
+		       region->iova, region->length, DMA_BIDIRECTIONAL);
+	afu_dma_unpin_pages(pdata, region);
+	kfree(region);
+
+	return 0;
+}
diff --git a/drivers/fpga/intel/afu-main.c b/drivers/fpga/intel/afu-main.c
index db2aec3..782ff81 100644
--- a/drivers/fpga/intel/afu-main.c
+++ b/drivers/fpga/intel/afu-main.c
@@ -177,7 +177,11 @@ static int afu_release(struct inode *inode, struct file *filp)
 
 	dev_dbg(&pdev->dev, "Device File Release\n");
 
-	fpga_port_reset(pdev);
+	mutex_lock(&pdata->lock);
+	__fpga_port_reset(pdev);
+	afu_dma_region_destroy(pdata);
+	mutex_unlock(&pdata->lock);
+
 	feature_dev_use_end(pdata);
 	return 0;
 }
@@ -247,6 +251,55 @@ afu_ioctl_get_region_info(struct feature_platform_data *pdata, void __user *arg)
 	return 0;
 }
 
+static long
+afu_ioctl_dma_map(struct feature_platform_data *pdata, void __user *arg)
+{
+	struct fpga_port_dma_map map;
+	unsigned long minsz;
+	long ret;
+
+	minsz = offsetofend(struct fpga_port_dma_map, iova);
+
+	if (copy_from_user(&map, arg, minsz))
+		return -EFAULT;
+
+	if (map.argsz < minsz || map.flags)
+		return -EINVAL;
+
+	ret = afu_dma_map_region(pdata, map.user_addr, map.length, &map.iova);
+	if (ret)
+		return ret;
+
+	if (copy_to_user(arg, &map, sizeof(map))) {
+		afu_dma_unmap_region(pdata, map.iova);
+		return -EFAULT;
+	}
+
+	dev_dbg(&pdata->dev->dev, "dma map: ua=%llx, len=%llx, iova=%llx\n",
+				(unsigned long long)map.user_addr,
+				(unsigned long long)map.length,
+				(unsigned long long)map.iova);
+
+	return 0;
+}
+
+static long
+afu_ioctl_dma_unmap(struct feature_platform_data *pdata, void __user *arg)
+{
+	struct fpga_port_dma_unmap unmap;
+	unsigned long minsz;
+
+	minsz = offsetofend(struct fpga_port_dma_unmap, iova);
+
+	if (copy_from_user(&unmap, arg, minsz))
+		return -EFAULT;
+
+	if (unmap.argsz < minsz || unmap.flags)
+		return -EINVAL;
+
+	return afu_dma_unmap_region(pdata, unmap.iova);
+}
+
 static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 {
 	struct platform_device *pdev = filp->private_data;
@@ -265,6 +318,10 @@ static long afu_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 		return afu_ioctl_get_info(pdata, (void __user *)arg);
 	case FPGA_PORT_GET_REGION_INFO:
 		return afu_ioctl_get_region_info(pdata, (void __user *)arg);
+	case FPGA_PORT_DMA_MAP:
+		return afu_ioctl_dma_map(pdata, (void __user *)arg);
+	case FPGA_PORT_DMA_UNMAP:
+		return afu_ioctl_dma_unmap(pdata, (void __user *)arg);
 	default:
 		/*
 		 * Let sub-feature's ioctl function to handle the cmd
@@ -339,6 +396,7 @@ static int afu_dev_init(struct platform_device *pdev)
 	mutex_lock(&pdata->lock);
 	fpga_pdata_set_private(pdata, afu);
 	afu_region_init(pdata);
+	afu_dma_region_init(pdata);
 	mutex_unlock(&pdata->lock);
 	return 0;
 }
@@ -351,6 +409,7 @@ static int afu_dev_destroy(struct platform_device *pdev)
 	mutex_lock(&pdata->lock);
 	afu = fpga_pdata_get_private(pdata);
 	afu_region_destroy(pdata);
+	afu_dma_region_destroy(pdata);
 	fpga_pdata_set_private(pdata, NULL);
 	mutex_unlock(&pdata->lock);
 
diff --git a/drivers/fpga/intel/afu.h b/drivers/fpga/intel/afu.h
index fca4dbc..fb46bd3 100644
--- a/drivers/fpga/intel/afu.h
+++ b/drivers/fpga/intel/afu.h
@@ -32,11 +32,21 @@ struct fpga_afu_region {
 	struct list_head node;
 };
 
+struct fpga_afu_dma_region {
+	u64 user_addr;
+	u64 length;
+	u64 iova;
+	struct page **pages;
+	struct rb_node node;
+	bool in_use;
+};
+
 struct fpga_afu {
 	u64 region_cur_offset;
 	int num_regions;
 	u8 num_umsgs;
 	struct list_head regions;
+	struct rb_root dma_regions;
 
 	struct feature_platform_data *pdata;
 };
@@ -51,4 +61,12 @@ int afu_get_region_by_offset(struct feature_platform_data *pdata,
 			    u64 offset, u64 size,
 			    struct fpga_afu_region *pregion);
 
+void afu_dma_region_init(struct feature_platform_data *pdata);
+void afu_dma_region_destroy(struct feature_platform_data *pdata);
+long afu_dma_map_region(struct feature_platform_data *pdata,
+		       u64 user_addr, u64 length, u64 *iova);
+long afu_dma_unmap_region(struct feature_platform_data *pdata, u64 iova);
+struct fpga_afu_dma_region *afu_dma_region_find(
+		struct feature_platform_data *pdata, u64 iova, u64 size);
+
 #endif
diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
index 86a5168..b584381 100644
--- a/include/uapi/linux/intel-fpga.h
+++ b/include/uapi/linux/intel-fpga.h
@@ -113,6 +113,43 @@ struct fpga_port_region_info {
 
 #define FPGA_PORT_GET_REGION_INFO	_IO(FPGA_MAGIC, PORT_BASE + 2)
 
+/**
+ * FPGA_PORT_DMA_MAP - _IOWR(FPGA_MAGIC, PORT_BASE + 3,
+ *						struct fpga_port_dma_map)
+ *
+ * Map the dma memory per user_addr and length which are provided by caller.
+ * Driver fills the iova in provided struct afu_port_dma_map.
+ * This interface only accepts page-size aligned user memory for dma mapping.
+ * Return: 0 on success, -errno on failure.
+ */
+struct fpga_port_dma_map {
+	/* Input */
+	__u32 argsz;		/* Structure length */
+	__u32 flags;		/* Zero for now */
+	__u64 user_addr;        /* Process virtual address */
+	__u64 length;           /* Length of mapping (bytes)*/
+	/* Output */
+	__u64 iova;             /* IO virtual address */
+};
+
+#define FPGA_PORT_DMA_MAP	_IO(FPGA_MAGIC, PORT_BASE + 3)
+
+/**
+ * FPGA_PORT_DMA_UNMAP - _IOW(FPGA_MAGIC, PORT_BASE + 4,
+ *						struct fpga_port_dma_unmap)
+ *
+ * Unmap the dma memory per iova provided by caller.
+ * Return: 0 on success, -errno on failure.
+ */
+struct fpga_port_dma_unmap {
+	/* Input */
+	__u32 argsz;		/* Structure length */
+	__u32 flags;		/* Zero for now */
+	__u64 iova;		/* IO virtual address */
+};
+
+#define FPGA_PORT_DMA_UNMAP	_IO(FPGA_MAGIC, PORT_BASE + 4)
+
 /* IOCTLs for FME file descriptor */
 
 /**
-- 
2.7.4

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 00/16] Intel FPGA Device Drivers
  2017-03-30 12:08 [PATCH 00/16] Intel FPGA Device Drivers Wu Hao
                   ` (15 preceding siblings ...)
  2017-03-30 12:08 ` [PATCH 16/16] fpga: intel: afu: add FPGA_PORT_DMA_MAP/UNMAP ioctls support Wu Hao
@ 2017-03-30 17:17 ` Moritz Fischer
  2017-04-06 20:27 ` Jerome Glisse
  17 siblings, 0 replies; 93+ messages in thread
From: Moritz Fischer @ 2017-03-30 17:17 UTC (permalink / raw)
  To: Wu Hao
  Cc: atull, moritz.fischer, linux-fpga, linux-kernel, luwei.kang, yi.z.zhang

[-- Attachment #1: Type: text/plain, Size: 2116 bytes --]

On Thu, Mar 30, 2017 at 08:08:00PM +0800, Wu Hao wrote:
> Hi All,
> 
> Here is a patch-series adding drivers for Intel FPGA devices.
> 
> The Intel FPGA driver provides interfaces for userspace applications to
> configure, enumerate, open, and access FPGA accelerators on platforms
> equipped with Intel(R) FPGA solutions and enables system level management
> functions such as FPGA partial reconfiguration, power management and
> virtualization.
> 
> This patch series only adds the basic functions for FPGA accelerators and
> partial reconfiguration. Patches for more functions, e.g power management
> and virtualization, will be submitted after this series gets reviewed.
> 
> Patch 1: add a document for Intel FPGA driver overview, including the HW
> architecture, driver organization, device enumeration, virtualization and
> opens.
> 
> Patch 2: introduce a fpga-dev class. It's used in below Intel FPGA PCIe
> device driver, to represent a FPGA device on the system, and all actual
> feature devices should be registered as child nodes of this container
> fpga-dev device.
> 
> Patch 3-7: implement Intel FPGA PCIe device driver. It walks through the
> 'Device Feature List' in the PCI Bar, creates the container fpga-dev as
> parent and platform devices as children for the feature devices it found.
> 
> Patch 8-11: implement Intel FPGA Management Engine (FME) driver. It's a
> platform driver matching with the FME platform device created by above
> PCIe driver. Sysfs and device file ioctls are exposed as user interfaces
> to allow partial reconfiguration to Accelerated Function Units (AFUs) from
> user space applications.
> 
> Patch 12-16: implement Intel FPGA Accelerated Function Unit (AFU) driver.
> It's a platform driver matching with AFU platform device created by above
> PCIe driver. It provides user interfaces to expose the AFU MMIO region,
> map/unmap dma buffer, and control the port which AFU connects to.

This is exciting stuff. It will take some time to review, though. I marked
the patchset as 'In-Review' in patchwork.

Cheers,

Moritz


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-03-30 12:08 ` [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Wu Hao
@ 2017-03-30 20:30   ` Alan Tull
  2017-03-31  4:11     ` Xiao Guangrong
  2017-03-31 19:10   ` Alan Tull
                     ` (3 subsequent siblings)
  4 siblings, 1 reply; 93+ messages in thread
From: Alan Tull @ 2017-03-30 20:30 UTC (permalink / raw)
  To: Wu Hao
  Cc: Moritz Fischer, linux-fpga, linux-kernel, luwei.kang, yi.z.zhang,
	Tim Whisonant, Enno Luebbers, Shiva Rao, Christopher Rauer,
	Alan Tull, Xiao Guangrong

On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote:
> From: Kang Luwei <luwei.kang@intel.com>
>
> Partial Reconfiguration (PR) is the most important function for FME. It
> allows reconfiguration for given Port/Accelerated Function Unit (AFU).
>
> This patch adds support for PR sub feature. In this patch, it registers
> a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
> for PR operation once PR request received via ioctl. Below user space
> interfaces are exposed by this sub feature.
>
> Sysfs interface:
> * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
>   Read-only. Indicate the hardware interface information. Userspace
>   applications need to check this interface to select correct green
>   bitstream format before PR.
>
> Ioctl interface:
> * FPGA_FME_PORT_PR
>   Do partial reconfiguration per information from userspace, including
>   target port(AFU), buffer size and address info. It returns the PR status
>   (PR error code if failed) to userspace.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Alan Tull <alan.tull@intel.com>

Hi Wu Hao,

Thanks for submitting your patches.

I think there's been a misunderstanding of the meaning of
'Signed-off-by' [1].  I have not signed off on this code or had a hand
in its development.  But I'm happy to get to review it now.  It will
take a bit of time; I expect to be replying next week.

Alan Tull
[1] linux/Documentation/process/5.Posting.rst

> Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-03-30 20:30   ` Alan Tull
@ 2017-03-31  4:11     ` Xiao Guangrong
  2017-03-31  8:50       ` Wu Hao
  0 siblings, 1 reply; 93+ messages in thread
From: Xiao Guangrong @ 2017-03-31  4:11 UTC (permalink / raw)
  To: Alan Tull, Wu Hao
  Cc: Moritz Fischer, linux-fpga, linux-kernel, luwei.kang, yi.z.zhang,
	Tim Whisonant, Enno Luebbers, Shiva Rao, Christopher Rauer,
	Alan Tull, Xiao Guangrong



On 31/03/2017 4:30 AM, Alan Tull wrote:
> On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote:
>> From: Kang Luwei <luwei.kang@intel.com>
>>
>> Partial Reconfiguration (PR) is the most important function for FME. It
>> allows reconfiguration for given Port/Accelerated Function Unit (AFU).
>>
>> This patch adds support for PR sub feature. In this patch, it registers
>> a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
>> for PR operation once PR request received via ioctl. Below user space
>> interfaces are exposed by this sub feature.
>>
>> Sysfs interface:
>> * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
>>   Read-only. Indicate the hardware interface information. Userspace
>>   applications need to check this interface to select correct green
>>   bitstream format before PR.
>>
>> Ioctl interface:
>> * FPGA_FME_PORT_PR
>>   Do partial reconfiguration per information from userspace, including
>>   target port(AFU), buffer size and address info. It returns the PR status
>>   (PR error code if failed) to userspace.
>>
>> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
>> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
>> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
>> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
>> Signed-off-by: Alan Tull <alan.tull@intel.com>
>
> Hi Wu Hao,
>
> Thanks for submitting your patches.
>
> I think there's been a misunderstanding of the meaning of
> 'Signed-off-by' [1].  I have not signed off on this code or had a hand
> in its development.  But I'm happy to get to review it now.  It will
> take a bit of time; I expect to be replying next week.

Hi Alan,

Sorry to confuse you, i think it's because you helped Chris a lot to
implement this interface and we'd like to include your credit as this
way. If you dislike, it will be dropped. :)

Thanks for your review in advance.

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 02/16] fpga: add FPGA device framework
  2017-03-30 12:08 ` [PATCH 02/16] fpga: add FPGA device framework Wu Hao
@ 2017-03-31  6:09   ` Greg KH
  2017-03-31  7:48     ` Wu Hao
  2017-03-31  6:13   ` Greg KH
  1 sibling, 1 reply; 93+ messages in thread
From: Greg KH @ 2017-03-31  6:09 UTC (permalink / raw)
  To: Wu Hao
  Cc: atull, moritz.fischer, linux-fpga, linux-kernel, luwei.kang,
	yi.z.zhang, Tim Whisonant, Enno Luebbers, Shiva Rao,
	Christopher Rauer, Xiao Guangrong

On Thu, Mar 30, 2017 at 08:08:02PM +0800, Wu Hao wrote:
> During FPGA device (e.g PCI-based) discovery, platform devices are
> registered for different FPGA function units. But the device node path
> isn't quite friendly to applications.
> 
> Consider this case, applications want to access child device's sysfs file
> for some information.
> 
> 1) Access using bus-based path (e.g PCI)
> 
>   /sys/bus/pci/devices/xxxxx/fpga_func_a.0/sysfs_file
> 
>   From the path, it's clear which PCI device is the parent, but not perfect
>   solution for applications. PCI device BDF is not fixed, application may
>   need to search all PCI device to find the actual FPGA Device.
> 
> 2) Or access using platform device path
> 
>   /sys/bus/platform/devices/fpga_func_a.0/sysfs_file
> 
>   Applications find the actual function by name easily, but no information
>   about which fpga device it belongs to. It's quite confusing if multiple
>   FPGA devices are in one system.
> 
> 'FPGA Device' class is introduced to resolve this problem. Each node under
> this class represents a fpga device, which may have one or more child
> devices. Applications only need to search under this FPGA Device class
> folder to find the child device node it needs.
> 
> For example, for the platform has 2 fpga devices, each fpga device has
> 3 child devices, the hierarchy looks like this.
> 
> Two nodes are under /sys/class/fpga/:
> /sys/class/fpga/fpga.0
> /sys/class/fpga/fpga.1
> 
> Each node has 1 function A device and 2 function B devices:
> /sys/class/fpga/fpga.0/func_a.0
> /sys/class/fpga/fpga.0/func_b.0
> /sys/class/fpga/fpga.0/func_b.1
> 
> /sys/class/fpga/fpga.1/func_a.1
> /sys/class/fpga/fpga.1/func_b.2
> /sys/class/fpga/fpga.1/func_b.3
> 
> This following APIs are provided by FPGA device framework:
> * fpga_dev_create
>   Create fpga device under the given parent device.
> * fpga_dev_destroy
>   Destroy fpga device
> 
> The following sysfs files are created:
> * /sys/class/fpga/<fpga.x>/name
>   Name of the fpga device.

How does this interact with the existing "fpga class" that is in the
kernel already?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 02/16] fpga: add FPGA device framework
  2017-03-30 12:08 ` [PATCH 02/16] fpga: add FPGA device framework Wu Hao
  2017-03-31  6:09   ` Greg KH
@ 2017-03-31  6:13   ` Greg KH
       [not found]     ` <82D7661F83C1A047AF7DC287873BF1E167C90F1B@SHSMSX101.ccr.corp.intel.com>
  1 sibling, 1 reply; 93+ messages in thread
From: Greg KH @ 2017-03-31  6:13 UTC (permalink / raw)
  To: Wu Hao
  Cc: atull, moritz.fischer, linux-fpga, linux-kernel, luwei.kang,
	yi.z.zhang, Tim Whisonant, Enno Luebbers, Shiva Rao,
	Christopher Rauer, Xiao Guangrong

On Thu, Mar 30, 2017 at 08:08:02PM +0800, Wu Hao wrote:
> During FPGA device (e.g PCI-based) discovery, platform devices are
> registered for different FPGA function units. But the device node path
> isn't quite friendly to applications.
> 
> Consider this case, applications want to access child device's sysfs file
> for some information.
> 
> 1) Access using bus-based path (e.g PCI)
> 
>   /sys/bus/pci/devices/xxxxx/fpga_func_a.0/sysfs_file
> 
>   From the path, it's clear which PCI device is the parent, but not perfect
>   solution for applications. PCI device BDF is not fixed, application may
>   need to search all PCI device to find the actual FPGA Device.
> 
> 2) Or access using platform device path
> 
>   /sys/bus/platform/devices/fpga_func_a.0/sysfs_file
> 
>   Applications find the actual function by name easily, but no information
>   about which fpga device it belongs to. It's quite confusing if multiple
>   FPGA devices are in one system.
> 
> 'FPGA Device' class is introduced to resolve this problem. Each node under
> this class represents a fpga device, which may have one or more child
> devices. Applications only need to search under this FPGA Device class
> folder to find the child device node it needs.
> 
> For example, for the platform has 2 fpga devices, each fpga device has
> 3 child devices, the hierarchy looks like this.
> 
> Two nodes are under /sys/class/fpga/:
> /sys/class/fpga/fpga.0
> /sys/class/fpga/fpga.1
> 
> Each node has 1 function A device and 2 function B devices:
> /sys/class/fpga/fpga.0/func_a.0
> /sys/class/fpga/fpga.0/func_b.0
> /sys/class/fpga/fpga.0/func_b.1
> 
> /sys/class/fpga/fpga.1/func_a.1
> /sys/class/fpga/fpga.1/func_b.2
> /sys/class/fpga/fpga.1/func_b.3
> 
> This following APIs are provided by FPGA device framework:
> * fpga_dev_create
>   Create fpga device under the given parent device.
> * fpga_dev_destroy
>   Destroy fpga device
> 
> The following sysfs files are created:
> * /sys/class/fpga/<fpga.x>/name
>   Name of the fpga device.
> 
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ---
>  drivers/fpga/Kconfig          |   6 +++
>  drivers/fpga/Makefile         |   3 ++
>  drivers/fpga/fpga-dev.c       | 120 ++++++++++++++++++++++++++++++++++++++++++
>  include/linux/fpga/fpga-dev.h |  34 ++++++++++++
>  4 files changed, 163 insertions(+)
>  create mode 100644 drivers/fpga/fpga-dev.c
>  create mode 100644 include/linux/fpga/fpga-dev.h
> 
> diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> index ce861a2..d99b640 100644
> --- a/drivers/fpga/Kconfig
> +++ b/drivers/fpga/Kconfig
> @@ -12,6 +12,12 @@ config FPGA
>  	  manager drivers.
>  
>  if FPGA
> +config FPGA_DEVICE
> +	tristate "FPGA Device Framework"
> +	help
> +	  Say Y here if you want support for FPGA Devices from the kernel.
> +	  The FPGA Device Framework adds a FPGA device class and provide
> +	  interfaces to create FPGA devices.
>  
>  config FPGA_REGION
>  	tristate "FPGA Region"
> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> index 8df07bc..53a41d2 100644
> --- a/drivers/fpga/Makefile
> +++ b/drivers/fpga/Makefile
> @@ -5,6 +5,9 @@
>  # Core FPGA Manager Framework
>  obj-$(CONFIG_FPGA)			+= fpga-mgr.o
>  
> +# FPGA Device Framework
> +obj-$(CONFIG_FPGA_DEVICE)		+= fpga-dev.o
> +
>  # FPGA Manager Drivers
>  obj-$(CONFIG_FPGA_MGR_SOCFPGA)		+= socfpga.o
>  obj-$(CONFIG_FPGA_MGR_SOCFPGA_A10)	+= socfpga-a10.o
> diff --git a/drivers/fpga/fpga-dev.c b/drivers/fpga/fpga-dev.c
> new file mode 100644
> index 0000000..0f4c0ed
> --- /dev/null
> +++ b/drivers/fpga/fpga-dev.c
> @@ -0,0 +1,120 @@
> +/*
> + * FPGA Device Framework Driver
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * This work is licensed under a dual BSD/GPLv2 license. When using or
> + * redistributing this file, you may do so under either license. See the
> + * LICENSE.BSD file under drivers/fpga/intel for the BSD license and see
> + * the COPYING file in the top-level directory for the GPLv2 license.

Really?  A BSD licened file that does EXPORT_SYMBOL_GPL and interacts
directly with the driver core?  Please go talk to some of your lawyers
about this before you resubmit this correctly...


> + */
> +#include <linux/device.h>
> +#include <linux/module.h>
> +#include <linux/slab.h>
> +#include <linux/fpga/fpga-dev.h>
> +
> +static DEFINE_IDA(fpga_dev_ida);
> +static struct class *fpga_dev_class;
> +
> +static ssize_t name_show(struct device *dev,
> +			 struct device_attribute *attr, char *buf)
> +{
> +	struct fpga_dev *fdev = to_fpga_dev(dev);
> +
> +	return sprintf(buf, "%s\n", fdev->name);
> +}
> +static DEVICE_ATTR_RO(name);

There already is a name for the device, it's the directory name.

> +
> +static struct attribute *fpga_dev_attrs[] = {
> +	&dev_attr_name.attr,
> +	NULL,
> +};
> +ATTRIBUTE_GROUPS(fpga_dev);
> +
> +/**
> + * fpga_dev_create - create a fpga device
> + * @parent: parent device
> + * @name: fpga device name
> + *
> + * Return fpga_dev struct for success, error code otherwise.
> + */
> +struct fpga_dev *fpga_dev_create(struct device *parent, const char *name)
> +{
> +	struct fpga_dev *fdev;
> +	int id, ret = 0;
> +
> +	if (!name || !strlen(name)) {
> +		dev_err(parent, "Attempt to register with no name!\n");
> +		return ERR_PTR(-EINVAL);
> +	}
> +
> +	fdev = kzalloc(sizeof(*fdev), GFP_KERNEL);
> +	if (!fdev)
> +		return ERR_PTR(-ENOMEM);
> +
> +	id = ida_simple_get(&fpga_dev_ida, 0, 0, GFP_KERNEL);
> +	if (id < 0) {
> +		ret = id;
> +		goto error_kfree;
> +	}
> +
> +	fdev->name = name;
> +
> +	device_initialize(&fdev->dev);
> +	fdev->dev.class = fpga_dev_class;
> +	fdev->dev.parent = parent;
> +	fdev->dev.id = id;
> +
> +	ret = dev_set_name(&fdev->dev, "fpga.%d", id);
> +	if (ret)
> +		goto error_device;
> +
> +	ret = device_add(&fdev->dev);
> +	if (ret)
> +		goto error_device;
> +
> +	dev_dbg(fdev->dev.parent, "fpga device [%s] created\n", fdev->name);
> +
> +	return fdev;
> +
> +error_device:
> +	ida_simple_remove(&fpga_dev_ida, id);
> +error_kfree:
> +	kfree(fdev);
> +
> +	return ERR_PTR(ret);
> +}
> +EXPORT_SYMBOL_GPL(fpga_dev_create);
> +
> +static void fpga_dev_release(struct device *dev)
> +{
> +	struct fpga_dev *fdev = to_fpga_dev(dev);
> +
> +	ida_simple_remove(&fpga_dev_ida, fdev->dev.id);
> +	kfree(fdev);
> +}
> +
> +static int __init fpga_dev_class_init(void)
> +{
> +	pr_info("FPGA Device framework\n");
> +
> +	fpga_dev_class = class_create(THIS_MODULE, "fpga");
> +	if (IS_ERR(fpga_dev_class))
> +		return PTR_ERR(fpga_dev_class);
> +
> +	fpga_dev_class->dev_groups = fpga_dev_groups;
> +	fpga_dev_class->dev_release = fpga_dev_release;
> +
> +	return 0;
> +}
> +
> +static void __exit fpga_dev_class_exit(void)
> +{
> +	class_destroy(fpga_dev_class);
> +}
> +
> +MODULE_DESCRIPTION("FPGA Device framework");
> +MODULE_LICENSE("Dual BSD/GPL");
> +
> +subsys_initcall(fpga_dev_class_init);
> +module_exit(fpga_dev_class_exit);
> diff --git a/include/linux/fpga/fpga-dev.h b/include/linux/fpga/fpga-dev.h
> new file mode 100644
> index 0000000..7b58356
> --- /dev/null
> +++ b/include/linux/fpga/fpga-dev.h
> @@ -0,0 +1,34 @@
> +/*
> + * FPGA Device Driver Header
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * This work is licensed under a dual BSD/GPLv2 license. When using or
> + * redistributing this file, you may do so under either license. See the
> + * LICENSE.BSD file under drivers/fpga/intel for the BSD license and see
> + * the COPYING file in the top-level directory for the GPLv2 license.

Again with the dual license, please fix up.

> + *
> + */
> +#ifndef _LINUX_FPGA_DEV_H
> +#define _LINUX_FPGA_DEV_H
> +
> +/**
> + * struct fpga_dev - fpga device structure
> + * @name: name of fpga device
> + * @dev: fpga device
> + */
> +struct fpga_dev {
> +	const char *name;
> +	struct device dev;

struct device already has a name, why duplicate it here?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 02/16] fpga: add FPGA device framework
  2017-03-31  6:09   ` Greg KH
@ 2017-03-31  7:48     ` Wu Hao
  2017-03-31  9:03       ` Greg KH
  2017-03-31 19:01       ` matthew.gerlach
  0 siblings, 2 replies; 93+ messages in thread
From: Wu Hao @ 2017-03-31  7:48 UTC (permalink / raw)
  To: Greg KH
  Cc: atull, moritz.fischer, linux-fpga, linux-kernel, luwei.kang,
	yi.z.zhang, Tim Whisonant, Enno Luebbers, Shiva Rao,
	Christopher Rauer, Xiao Guangrong

On Fri, Mar 31, 2017 at 08:09:09AM +0200, Greg KH wrote:
> On Thu, Mar 30, 2017 at 08:08:02PM +0800, Wu Hao wrote:
> > During FPGA device (e.g PCI-based) discovery, platform devices are
> > registered for different FPGA function units. But the device node path
> > isn't quite friendly to applications.
> > 
> > Consider this case, applications want to access child device's sysfs file
> > for some information.
> > 
> > 1) Access using bus-based path (e.g PCI)
> > 
> >   /sys/bus/pci/devices/xxxxx/fpga_func_a.0/sysfs_file
> > 
> >   From the path, it's clear which PCI device is the parent, but not perfect
> >   solution for applications. PCI device BDF is not fixed, application may
> >   need to search all PCI device to find the actual FPGA Device.
> > 
> > 2) Or access using platform device path
> > 
> >   /sys/bus/platform/devices/fpga_func_a.0/sysfs_file
> > 
> >   Applications find the actual function by name easily, but no information
> >   about which fpga device it belongs to. It's quite confusing if multiple
> >   FPGA devices are in one system.
> > 
> > 'FPGA Device' class is introduced to resolve this problem. Each node under
> > this class represents a fpga device, which may have one or more child
> > devices. Applications only need to search under this FPGA Device class
> > folder to find the child device node it needs.
> > 
> > For example, for the platform has 2 fpga devices, each fpga device has
> > 3 child devices, the hierarchy looks like this.
> > 
> > Two nodes are under /sys/class/fpga/:
> > /sys/class/fpga/fpga.0
> > /sys/class/fpga/fpga.1
> > 
> > Each node has 1 function A device and 2 function B devices:
> > /sys/class/fpga/fpga.0/func_a.0
> > /sys/class/fpga/fpga.0/func_b.0
> > /sys/class/fpga/fpga.0/func_b.1
> > 
> > /sys/class/fpga/fpga.1/func_a.1
> > /sys/class/fpga/fpga.1/func_b.2
> > /sys/class/fpga/fpga.1/func_b.3
> > 
> > This following APIs are provided by FPGA device framework:
> > * fpga_dev_create
> >   Create fpga device under the given parent device.
> > * fpga_dev_destroy
> >   Destroy fpga device
> > 
> > The following sysfs files are created:
> > * /sys/class/fpga/<fpga.x>/name
> >   Name of the fpga device.
> 
> How does this interact with the existing "fpga class" that is in the
> kernel already?

The fpga-dev introduced by this patch, is only a container device, and
drivers could register different functions under it. Per my understanding,
the existing "fpga class", including fpga-region, fpga-bridge and 
fpga-manager, is used to provide reconfiguration function for FPGA. So
driver can create child node using this existing "fpga class" to provide
FPGA reconfiguration function, and more nodes under this container for
different functions for given FPGA device.

For Intel FPGA device, partial reconfiguration is only one function of 
Intel FPGA Management Engine (FME). FME driver creates fpga_manager under
below path for partial reconfiguration, and other interfaces for more
functions, e.g power management, virtualization support and etc.

/sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/fpga_manager

Thanks
Hao

> 
> thanks,
> 
> greg k-h

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-03-31  4:11     ` Xiao Guangrong
@ 2017-03-31  8:50       ` Wu Hao
  2017-04-03 20:26         ` Alan Tull
  0 siblings, 1 reply; 93+ messages in thread
From: Wu Hao @ 2017-03-31  8:50 UTC (permalink / raw)
  To: atull
  Cc: Moritz Fischer, linux-fpga, linux-kernel, luwei.kang, yi.z.zhang,
	Tim Whisonant, Enno Luebbers, Shiva Rao, Christopher Rauer,
	Alan Tull, Xiao Guangrong, xiaoguangrong.eric

On Fri, Mar 31, 2017 at 12:11:12PM +0800, Xiao Guangrong wrote:
> On 31/03/2017 4:30 AM, Alan Tull wrote:
> >On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote:
> >>From: Kang Luwei <luwei.kang@intel.com>
> >>
> >>Partial Reconfiguration (PR) is the most important function for FME. It
> >>allows reconfiguration for given Port/Accelerated Function Unit (AFU).
> >>
> >>This patch adds support for PR sub feature. In this patch, it registers
> >>a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
> >>for PR operation once PR request received via ioctl. Below user space
> >>interfaces are exposed by this sub feature.
> >>
> >>Sysfs interface:
> >>* /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
> >>  Read-only. Indicate the hardware interface information. Userspace
> >>  applications need to check this interface to select correct green
> >>  bitstream format before PR.
> >>
> >>Ioctl interface:
> >>* FPGA_FME_PORT_PR
> >>  Do partial reconfiguration per information from userspace, including
> >>  target port(AFU), buffer size and address info. It returns the PR status
> >>  (PR error code if failed) to userspace.
> >>
> >>Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> >>Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> >>Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> >>Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> >>Signed-off-by: Alan Tull <alan.tull@intel.com>
> >
> >Hi Wu Hao,
> >
> >Thanks for submitting your patches.
> >
> >I think there's been a misunderstanding of the meaning of
> >'Signed-off-by' [1].  I have not signed off on this code or had a hand
> >in its development.  But I'm happy to get to review it now.  It will
> >take a bit of time; I expect to be replying next week.
> 
> Hi Alan,
> 
> Sorry to confuse you, i think it's because you helped Chris a lot to
> implement this interface and we'd like to include your credit as this
> way. If you dislike, it will be dropped. :)
> 
> Thanks for your review in advance.
>

Hi Alan,

Sorry about this, we should ask you firstly before doing it this way.
Let me know if you don't like it, I will drop it in the next version.

Many thanks for your time and review on these patches. Look forward
for your feedback and comments. :)

Thanks
Hao 

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 02/16] fpga: add FPGA device framework
  2017-03-31  7:48     ` Wu Hao
@ 2017-03-31  9:03       ` Greg KH
  2017-03-31 12:19         ` Wu Hao
  2017-03-31 19:01       ` matthew.gerlach
  1 sibling, 1 reply; 93+ messages in thread
From: Greg KH @ 2017-03-31  9:03 UTC (permalink / raw)
  To: Wu Hao
  Cc: atull, moritz.fischer, linux-fpga, linux-kernel, luwei.kang,
	yi.z.zhang, Tim Whisonant, Enno Luebbers, Shiva Rao,
	Christopher Rauer, Xiao Guangrong

On Fri, Mar 31, 2017 at 03:48:42PM +0800, Wu Hao wrote:
> On Fri, Mar 31, 2017 at 08:09:09AM +0200, Greg KH wrote:
> > On Thu, Mar 30, 2017 at 08:08:02PM +0800, Wu Hao wrote:
> > > During FPGA device (e.g PCI-based) discovery, platform devices are
> > > registered for different FPGA function units. But the device node path
> > > isn't quite friendly to applications.
> > > 
> > > Consider this case, applications want to access child device's sysfs file
> > > for some information.
> > > 
> > > 1) Access using bus-based path (e.g PCI)
> > > 
> > >   /sys/bus/pci/devices/xxxxx/fpga_func_a.0/sysfs_file
> > > 
> > >   From the path, it's clear which PCI device is the parent, but not perfect
> > >   solution for applications. PCI device BDF is not fixed, application may
> > >   need to search all PCI device to find the actual FPGA Device.
> > > 
> > > 2) Or access using platform device path
> > > 
> > >   /sys/bus/platform/devices/fpga_func_a.0/sysfs_file
> > > 
> > >   Applications find the actual function by name easily, but no information
> > >   about which fpga device it belongs to. It's quite confusing if multiple
> > >   FPGA devices are in one system.
> > > 
> > > 'FPGA Device' class is introduced to resolve this problem. Each node under
> > > this class represents a fpga device, which may have one or more child
> > > devices. Applications only need to search under this FPGA Device class
> > > folder to find the child device node it needs.
> > > 
> > > For example, for the platform has 2 fpga devices, each fpga device has
> > > 3 child devices, the hierarchy looks like this.
> > > 
> > > Two nodes are under /sys/class/fpga/:
> > > /sys/class/fpga/fpga.0
> > > /sys/class/fpga/fpga.1
> > > 
> > > Each node has 1 function A device and 2 function B devices:
> > > /sys/class/fpga/fpga.0/func_a.0
> > > /sys/class/fpga/fpga.0/func_b.0
> > > /sys/class/fpga/fpga.0/func_b.1
> > > 
> > > /sys/class/fpga/fpga.1/func_a.1
> > > /sys/class/fpga/fpga.1/func_b.2
> > > /sys/class/fpga/fpga.1/func_b.3
> > > 
> > > This following APIs are provided by FPGA device framework:
> > > * fpga_dev_create
> > >   Create fpga device under the given parent device.
> > > * fpga_dev_destroy
> > >   Destroy fpga device
> > > 
> > > The following sysfs files are created:
> > > * /sys/class/fpga/<fpga.x>/name
> > >   Name of the fpga device.
> > 
> > How does this interact with the existing "fpga class" that is in the
> > kernel already?
> 
> The fpga-dev introduced by this patch, is only a container device, and
> drivers could register different functions under it. Per my understanding,
> the existing "fpga class", including fpga-region, fpga-bridge and 
> fpga-manager, is used to provide reconfiguration function for FPGA. So
> driver can create child node using this existing "fpga class" to provide
> FPGA reconfiguration function, and more nodes under this container for
> different functions for given FPGA device.
> 
> For Intel FPGA device, partial reconfiguration is only one function of 
> Intel FPGA Management Engine (FME). FME driver creates fpga_manager under
> below path for partial reconfiguration, and other interfaces for more
> functions, e.g power management, virtualization support and etc.
> 
> /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/fpga_manager

So there is now two different levels of fpga class interfaces?

I'm not disagreeing with this, just that it seems a bit confusing, don't
you think?

greg k-h

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 02/16] fpga: add FPGA device framework
  2017-03-31  9:03       ` Greg KH
@ 2017-03-31 12:19         ` Wu Hao
  0 siblings, 0 replies; 93+ messages in thread
From: Wu Hao @ 2017-03-31 12:19 UTC (permalink / raw)
  To: Greg KH
  Cc: atull, moritz.fischer, linux-fpga, linux-kernel, luwei.kang,
	yi.z.zhang, Tim Whisonant, Enno Luebbers, Shiva Rao,
	Christopher Rauer, Xiao Guangrong

On Fri, Mar 31, 2017 at 11:03:28AM +0200, Greg KH wrote:
> On Fri, Mar 31, 2017 at 03:48:42PM +0800, Wu Hao wrote:
> > On Fri, Mar 31, 2017 at 08:09:09AM +0200, Greg KH wrote:
> > > On Thu, Mar 30, 2017 at 08:08:02PM +0800, Wu Hao wrote:
> > > > During FPGA device (e.g PCI-based) discovery, platform devices are
> > > > registered for different FPGA function units. But the device node path
> > > > isn't quite friendly to applications.
> > > > 
> > > > Consider this case, applications want to access child device's sysfs file
> > > > for some information.
> > > > 
> > > > 1) Access using bus-based path (e.g PCI)
> > > > 
> > > >   /sys/bus/pci/devices/xxxxx/fpga_func_a.0/sysfs_file
> > > > 
> > > >   From the path, it's clear which PCI device is the parent, but not perfect
> > > >   solution for applications. PCI device BDF is not fixed, application may
> > > >   need to search all PCI device to find the actual FPGA Device.
> > > > 
> > > > 2) Or access using platform device path
> > > > 
> > > >   /sys/bus/platform/devices/fpga_func_a.0/sysfs_file
> > > > 
> > > >   Applications find the actual function by name easily, but no information
> > > >   about which fpga device it belongs to. It's quite confusing if multiple
> > > >   FPGA devices are in one system.
> > > > 
> > > > 'FPGA Device' class is introduced to resolve this problem. Each node under
> > > > this class represents a fpga device, which may have one or more child
> > > > devices. Applications only need to search under this FPGA Device class
> > > > folder to find the child device node it needs.
> > > > 
> > > > For example, for the platform has 2 fpga devices, each fpga device has
> > > > 3 child devices, the hierarchy looks like this.
> > > > 
> > > > Two nodes are under /sys/class/fpga/:
> > > > /sys/class/fpga/fpga.0
> > > > /sys/class/fpga/fpga.1
> > > > 
> > > > Each node has 1 function A device and 2 function B devices:
> > > > /sys/class/fpga/fpga.0/func_a.0
> > > > /sys/class/fpga/fpga.0/func_b.0
> > > > /sys/class/fpga/fpga.0/func_b.1
> > > > 
> > > > /sys/class/fpga/fpga.1/func_a.1
> > > > /sys/class/fpga/fpga.1/func_b.2
> > > > /sys/class/fpga/fpga.1/func_b.3
> > > > 
> > > > This following APIs are provided by FPGA device framework:
> > > > * fpga_dev_create
> > > >   Create fpga device under the given parent device.
> > > > * fpga_dev_destroy
> > > >   Destroy fpga device
> > > > 
> > > > The following sysfs files are created:
> > > > * /sys/class/fpga/<fpga.x>/name
> > > >   Name of the fpga device.
> > > 
> > > How does this interact with the existing "fpga class" that is in the
> > > kernel already?
> > 
> > The fpga-dev introduced by this patch, is only a container device, and
> > drivers could register different functions under it. Per my understanding,
> > the existing "fpga class", including fpga-region, fpga-bridge and 
> > fpga-manager, is used to provide reconfiguration function for FPGA. So
> > driver can create child node using this existing "fpga class" to provide
> > FPGA reconfiguration function, and more nodes under this container for
> > different functions for given FPGA device.
> > 
> > For Intel FPGA device, partial reconfiguration is only one function of 
> > Intel FPGA Management Engine (FME). FME driver creates fpga_manager under
> > below path for partial reconfiguration, and other interfaces for more
> > functions, e.g power management, virtualization support and etc.
> > 
> > /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/fpga_manager
> 
> So there is now two different levels of fpga class interfaces?
> 
> I'm not disagreeing with this, just that it seems a bit confusing, don't
> you think?

I am not so sure, but the main purpose of fpga-dev, is trying to provide
enduser a more clear sysfs hierarchy reflecting the real hardware. And
fpga-things can be registered to fpga-dev directly if the hardware arch
is simple.

>From enduser point of view, he could find everything of this FPGA device
under /sys/class/fpga/<fpga.x>/, including all fpga-regions, fpga-bridges
and fpga-managers. I feel it is not a bad choice. :)

Thanks
Hao

> 
> greg k-h

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 02/16] fpga: add FPGA device framework
       [not found]     ` <82D7661F83C1A047AF7DC287873BF1E167C90F1B@SHSMSX101.ccr.corp.intel.com>
@ 2017-03-31 13:31       ` Wu Hao
  2017-03-31 14:10         ` Greg KH
  0 siblings, 1 reply; 93+ messages in thread
From: Wu Hao @ 2017-03-31 13:31 UTC (permalink / raw)
  To: greg
  Cc: atull, moritz.fischer, linux-fpga, linux-kernel, luwei.kang,
	yi.z.zhang, tim.whisonant, Enno Luebbers, Shiva Rao,
	Christopher Rauer, guangrong.xiao

> On Thu, Mar 30, 2017 at 08:08:02PM +0800, Wu Hao wrote:
> > During FPGA device (e.g PCI-based) discovery, platform devices are 
> > registered for different FPGA function units. But the device node path 
> > isn't quite friendly to applications.
> > 
> > Consider this case, applications want to access child device's sysfs 
> > file for some information.
> > 
> > 1) Access using bus-based path (e.g PCI)
> > 
> >   /sys/bus/pci/devices/xxxxx/fpga_func_a.0/sysfs_file
> > 
> >   From the path, it's clear which PCI device is the parent, but not perfect
> >   solution for applications. PCI device BDF is not fixed, application may
> >   need to search all PCI device to find the actual FPGA Device.
> > 
> > 2) Or access using platform device path
> > 
> >   /sys/bus/platform/devices/fpga_func_a.0/sysfs_file
> > 
> >   Applications find the actual function by name easily, but no information
> >   about which fpga device it belongs to. It's quite confusing if multiple
> >   FPGA devices are in one system.
> > 
> > 'FPGA Device' class is introduced to resolve this problem. Each node 
> > under this class represents a fpga device, which may have one or more 
> > child devices. Applications only need to search under this FPGA Device 
> > class folder to find the child device node it needs.
> > 
> > For example, for the platform has 2 fpga devices, each fpga device has
> > 3 child devices, the hierarchy looks like this.
> > 
> > Two nodes are under /sys/class/fpga/:
> > /sys/class/fpga/fpga.0
> > /sys/class/fpga/fpga.1
> > 
> > Each node has 1 function A device and 2 function B devices:
> > /sys/class/fpga/fpga.0/func_a.0
> > /sys/class/fpga/fpga.0/func_b.0
> > /sys/class/fpga/fpga.0/func_b.1
> > 
> > /sys/class/fpga/fpga.1/func_a.1
> > /sys/class/fpga/fpga.1/func_b.2
> > /sys/class/fpga/fpga.1/func_b.3
> > 
> > This following APIs are provided by FPGA device framework:
> > * fpga_dev_create
> >   Create fpga device under the given parent device.
> > * fpga_dev_destroy
> >   Destroy fpga device
> > 
> > The following sysfs files are created:
> > * /sys/class/fpga/<fpga.x>/name
> >   Name of the fpga device.
> > 
> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > ---
> >  drivers/fpga/Kconfig          |   6 +++
> >  drivers/fpga/Makefile         |   3 ++
> >  drivers/fpga/fpga-dev.c       | 120 ++++++++++++++++++++++++++++++++++++++++++
> >  include/linux/fpga/fpga-dev.h |  34 ++++++++++++
> >  4 files changed, 163 insertions(+)
> >  create mode 100644 drivers/fpga/fpga-dev.c  create mode 100644 
> > include/linux/fpga/fpga-dev.h
> > 
> > diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig index 
> > ce861a2..d99b640 100644
> > --- a/drivers/fpga/Kconfig
> > +++ b/drivers/fpga/Kconfig
> > @@ -12,6 +12,12 @@ config FPGA
> >  	  manager drivers.
> >  
> >  if FPGA
> > +config FPGA_DEVICE
> > +	tristate "FPGA Device Framework"
> > +	help
> > +	  Say Y here if you want support for FPGA Devices from the kernel.
> > +	  The FPGA Device Framework adds a FPGA device class and provide
> > +	  interfaces to create FPGA devices.
> >  
> >  config FPGA_REGION
> >  	tristate "FPGA Region"
> > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile index 
> > 8df07bc..53a41d2 100644
> > --- a/drivers/fpga/Makefile
> > +++ b/drivers/fpga/Makefile
> > @@ -5,6 +5,9 @@
> >  # Core FPGA Manager Framework
> >  obj-$(CONFIG_FPGA)			+= fpga-mgr.o
> >  
> > +# FPGA Device Framework
> > +obj-$(CONFIG_FPGA_DEVICE)		+= fpga-dev.o
> > +
> >  # FPGA Manager Drivers
> >  obj-$(CONFIG_FPGA_MGR_SOCFPGA)		+= socfpga.o
> >  obj-$(CONFIG_FPGA_MGR_SOCFPGA_A10)	+= socfpga-a10.o
> > diff --git a/drivers/fpga/fpga-dev.c b/drivers/fpga/fpga-dev.c new 
> > file mode 100644 index 0000000..0f4c0ed
> > --- /dev/null
> > +++ b/drivers/fpga/fpga-dev.c
> > @@ -0,0 +1,120 @@
> > +/*
> > + * FPGA Device Framework Driver
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * This work is licensed under a dual BSD/GPLv2 license. When using 
> > +or
> > + * redistributing this file, you may do so under either license. See 
> > +the
> > + * LICENSE.BSD file under drivers/fpga/intel for the BSD license and 
> > +see
> > + * the COPYING file in the top-level directory for the GPLv2 license.
> 
> Really?  A BSD licened file that does EXPORT_SYMBOL_GPL and interacts directly with the driver core?  Please go talk to some of your lawyers about this before you resubmit this correctly...

Sorry, will check and fix this in next version.

> 
> 
> > + */
> > +#include <linux/device.h>
> > +#include <linux/module.h>
> > +#include <linux/slab.h>
> > +#include <linux/fpga/fpga-dev.h>
> > +
> > +static DEFINE_IDA(fpga_dev_ida);
> > +static struct class *fpga_dev_class;
> > +
> > +static ssize_t name_show(struct device *dev,
> > +			 struct device_attribute *attr, char *buf) {
> > +	struct fpga_dev *fdev = to_fpga_dev(dev);
> > +
> > +	return sprintf(buf, "%s\n", fdev->name); } static 
> > +DEVICE_ATTR_RO(name);
> 
> There already is a name for the device, it's the directory name.

For current implementation, the directory will have a common name like

/sys/class/fpga/fpga.0
/sys/class/fpga/fpga.1
/sys/class/fpga/fpga.2
...

For the 'name' sysfs interface, driver can put more device specific info
into this 'name', e.g intel-fpga-dev. Userspace can use this information
to know which kind of FPGA device it is. e.g if applications read the
/sys/class/fpga/fpga.5/name as intel-fpga-dev, it means the 5th fpga
device on the system is a Intel FPGA device, and then application applies
related method to enumerate the accelerators for it.
And other existing fpga class has similar sysfs interface too, so I would
like to keep it aligned with others.

> 
> > +
> > +static struct attribute *fpga_dev_attrs[] = {
> > +	&dev_attr_name.attr,
> > +	NULL,
> > +};
> > +ATTRIBUTE_GROUPS(fpga_dev);
> > +
> > +/**
> > + * fpga_dev_create - create a fpga device
> > + * @parent: parent device
> > + * @name: fpga device name
> > + *
> > + * Return fpga_dev struct for success, error code otherwise.
> > + */
> > +struct fpga_dev *fpga_dev_create(struct device *parent, const char 
> > +*name) {
> > +	struct fpga_dev *fdev;
> > +	int id, ret = 0;
> > +
> > +	if (!name || !strlen(name)) {
> > +		dev_err(parent, "Attempt to register with no name!\n");
> > +		return ERR_PTR(-EINVAL);
> > +	}
> > +
> > +	fdev = kzalloc(sizeof(*fdev), GFP_KERNEL);
> > +	if (!fdev)
> > +		return ERR_PTR(-ENOMEM);
> > +
> > +	id = ida_simple_get(&fpga_dev_ida, 0, 0, GFP_KERNEL);
> > +	if (id < 0) {
> > +		ret = id;
> > +		goto error_kfree;
> > +	}
> > +
> > +	fdev->name = name;
> > +
> > +	device_initialize(&fdev->dev);
> > +	fdev->dev.class = fpga_dev_class;
> > +	fdev->dev.parent = parent;
> > +	fdev->dev.id = id;
> > +
> > +	ret = dev_set_name(&fdev->dev, "fpga.%d", id);
> > +	if (ret)
> > +		goto error_device;
> > +
> > +	ret = device_add(&fdev->dev);
> > +	if (ret)
> > +		goto error_device;
> > +
> > +	dev_dbg(fdev->dev.parent, "fpga device [%s] created\n", fdev->name);
> > +
> > +	return fdev;
> > +
> > +error_device:
> > +	ida_simple_remove(&fpga_dev_ida, id);
> > +error_kfree:
> > +	kfree(fdev);
> > +
> > +	return ERR_PTR(ret);
> > +}
> > +EXPORT_SYMBOL_GPL(fpga_dev_create);
> > +
> > +static void fpga_dev_release(struct device *dev) {
> > +	struct fpga_dev *fdev = to_fpga_dev(dev);
> > +
> > +	ida_simple_remove(&fpga_dev_ida, fdev->dev.id);
> > +	kfree(fdev);
> > +}
> > +
> > +static int __init fpga_dev_class_init(void) {
> > +	pr_info("FPGA Device framework\n");
> > +
> > +	fpga_dev_class = class_create(THIS_MODULE, "fpga");
> > +	if (IS_ERR(fpga_dev_class))
> > +		return PTR_ERR(fpga_dev_class);
> > +
> > +	fpga_dev_class->dev_groups = fpga_dev_groups;
> > +	fpga_dev_class->dev_release = fpga_dev_release;
> > +
> > +	return 0;
> > +}
> > +
> > +static void __exit fpga_dev_class_exit(void) {
> > +	class_destroy(fpga_dev_class);
> > +}
> > +
> > +MODULE_DESCRIPTION("FPGA Device framework"); MODULE_LICENSE("Dual 
> > +BSD/GPL");
> > +
> > +subsys_initcall(fpga_dev_class_init);
> > +module_exit(fpga_dev_class_exit);
> > diff --git a/include/linux/fpga/fpga-dev.h 
> > b/include/linux/fpga/fpga-dev.h new file mode 100644 index 
> > 0000000..7b58356
> > --- /dev/null
> > +++ b/include/linux/fpga/fpga-dev.h
> > @@ -0,0 +1,34 @@
> > +/*
> > + * FPGA Device Driver Header
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * This work is licensed under a dual BSD/GPLv2 license. When using 
> > +or
> > + * redistributing this file, you may do so under either license. See 
> > +the
> > + * LICENSE.BSD file under drivers/fpga/intel for the BSD license and 
> > +see
> > + * the COPYING file in the top-level directory for the GPLv2 license.
> 
> Again with the dual license, please fix up.
>

Sorry, will fix this in the next version.
 
> > + *
> > + */
> > +#ifndef _LINUX_FPGA_DEV_H
> > +#define _LINUX_FPGA_DEV_H
> > +
> > +/**
> > + * struct fpga_dev - fpga device structure
> > + * @name: name of fpga device
> > + * @dev: fpga device
> > + */
> > +struct fpga_dev {
> > +	const char *name;
> > +	struct device dev;
> 
> struct device already has a name, why duplicate it here?

As mentioned above, dev has common name as fpga.x, but 'name' could have
more meaningful information, e.g venodr and device information.

Thanks
Hao

> 
> thanks,
> 
> greg k-h

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 02/16] fpga: add FPGA device framework
  2017-03-31 13:31       ` Wu Hao
@ 2017-03-31 14:10         ` Greg KH
  2017-04-01 11:36           ` Wu Hao
  0 siblings, 1 reply; 93+ messages in thread
From: Greg KH @ 2017-03-31 14:10 UTC (permalink / raw)
  To: Wu Hao
  Cc: atull, moritz.fischer, linux-fpga, linux-kernel, luwei.kang,
	yi.z.zhang, tim.whisonant, Enno Luebbers, Shiva Rao,
	Christopher Rauer, guangrong.xiao

On Fri, Mar 31, 2017 at 09:31:09PM +0800, Wu Hao wrote:
> > On Thu, Mar 30, 2017 at 08:08:02PM +0800, Wu Hao wrote:
> > > +#include <linux/device.h>
> > > +#include <linux/module.h>
> > > +#include <linux/slab.h>
> > > +#include <linux/fpga/fpga-dev.h>
> > > +
> > > +static DEFINE_IDA(fpga_dev_ida);
> > > +static struct class *fpga_dev_class;
> > > +
> > > +static ssize_t name_show(struct device *dev,
> > > +			 struct device_attribute *attr, char *buf) {
> > > +	struct fpga_dev *fdev = to_fpga_dev(dev);
> > > +
> > > +	return sprintf(buf, "%s\n", fdev->name); } static 
> > > +DEVICE_ATTR_RO(name);
> > 
> > There already is a name for the device, it's the directory name.
> 
> For current implementation, the directory will have a common name like
> 
> /sys/class/fpga/fpga.0
> /sys/class/fpga/fpga.1
> /sys/class/fpga/fpga.2
> ...
> 
> For the 'name' sysfs interface, driver can put more device specific info
> into this 'name', e.g intel-fpga-dev. Userspace can use this information
> to know which kind of FPGA device it is. e.g if applications read the
> /sys/class/fpga/fpga.5/name as intel-fpga-dev, it means the 5th fpga
> device on the system is a Intel FPGA device, and then application applies
> related method to enumerate the accelerators for it.
> And other existing fpga class has similar sysfs interface too, so I would
> like to keep it aligned with others.

Ok, then document the heck out of this in Documentation/ABI/ which I
don't think you did for the sysfs files you are creating.

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview
  2017-03-30 12:08 ` [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview Wu Hao
@ 2017-03-31 18:24   ` matthew.gerlach
  2017-03-31 18:38     ` Alan Tull
  0 siblings, 1 reply; 93+ messages in thread
From: matthew.gerlach @ 2017-03-31 18:24 UTC (permalink / raw)
  To: Wu Hao
  Cc: atull, moritz.fischer, linux-fpga, linux-kernel, luwei.kang,
	yi.z.zhang, Enno Luebbers, Xiao Guangrong



On Thu, 30 Mar 2017, Wu Hao wrote:


Hi Wu Hao,

Great documentation. I'm looking forward to diving into the rest of the 
patches. Please see my comments inline.

Matthew Gerlach


> Add a document for Intel FPGA driver overview.
>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ---
> Documentation/fpga/intel-fpga.txt | 259 ++++++++++++++++++++++++++++++++++++++
> 1 file changed, 259 insertions(+)
> create mode 100644 Documentation/fpga/intel-fpga.txt
>
> diff --git a/Documentation/fpga/intel-fpga.txt b/Documentation/fpga/intel-fpga.txt
> new file mode 100644
> index 0000000..9396cea
> --- /dev/null
> +++ b/Documentation/fpga/intel-fpga.txt
> @@ -0,0 +1,259 @@
> +===============================================================================
> +                    Intel FPGA driver Overview
> +-------------------------------------------------------------------------------
> +                Enno Luebbers <enno.luebbers@intel.com>
> +                Xiao Guangrong <guangrong.xiao@linux.intel.com>
> +                Wu Hao <hao.wu@intel.com>
> +
> +The Intel FPGA driver provides interfaces for userspace applications to
> +configure, enumerate, open, and access FPGA accelerators on platforms equipped
> +with Intel(R) FPGA solutions and enables system level management functions such
> +as FPGA reconfiguration, power management, and virtualization.
> +

>From a Linux kernel perspective, I'm not sure this is the best name for
this code.  The name gives me the impression that it is a driver for all
Intel FPGAs, but not all Intel FPGAs are connected to the processor over a
PCIe bus.  The processor could be directely connected like the Arria10
SOCFPGA.  Such a processor could certainly benefit from this accelerator
usage model.  In an extreme case, couldn't a processor in the FPGA,
running Linux, also benefit from this accelerator model?  Is this code a
"FPGA Accelerator Framework"?

> +HW Architecture
> +===============
> +From the OS's point of view, the FPGA hardware appears as a regular PCIe device.
> +The FPGA device memory is organized using a predefined data structure (Device
> +Feature List). Features supported by the particular FPGA device are exposed
> +through these data structures, as illustrated below:
> +
> +  +-------------------------------+  +-------------+
> +  |              PF               |  |     VF      |
> +  +-------------------------------+  +-------------+
> +      ^            ^         ^              ^
> +      |            |         |              |
> ++-----|------------|---------|--------------|-------+
> +|     |            |         |              |       |
> +|  +-----+     +-------+ +-------+      +-------+   |
> +|  | FME |     | Port0 | | Port1 |      | Port2 |   |
> +|  +-----+     +-------+ +-------+      +-------+   |
> +|                  ^         ^              ^       |
> +|                  |         |              |       |
> +|              +-------+ +------+       +-------+   |
> +|              |  AFU  | |  AFU |       |  AFU  |   |
> +|              +-------+ +------+       +-------+   |
> +|                                                   |
> +|                 FPGA PCIe Device                  |
> ++---------------------------------------------------+
> +
> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which can be
> +used to assign individual accelerators to virtual machines .

Does this HW Architecture require an Intel FPGA?  Couldn't any vendors 
FPGA be used as long as it presented itself the PCIe bus the same and 
contained an appropriate Device Feature List?

> +
> +FME (FPGA Management Engine)
> +============================
> +The FPGA Management Enging performs power and thermal management, error
> +reporting, reconfiguration, performance reporting, and other infrastructure
> +functions. Each FPGA has one FME, which is always accessed through the physical
> +function (PF).
> +
> +User-space applications can acquire exclusive access to the FME using open(),
> +and release it using close().
> +
> +The following functions are exposed through ioctls:
> +
> +	Get driver API version (FPGA_GET_API_VERSION)
> +	Check for extensions (FPGA_CHECK_EXTENSION)
> +	Assign port to PF (FPGA_FME_PORT_ASSIGN)
> +	Release port from PF (FPGA_FME_PORT_RELEASE)
> +	Program bitstream (FPGA_FME_PORT_PR)
> +
> +More functions are exposed through sysfs
> +(/sys/class/fpga/fpga.n/intel-fpga-fme.n/):
> +
> +	Read bitstream ID (bitstream_id)
> +	Read bitstream metadata (bitstream_metadata)
> +	Read number of ports (ports_num)
> +	Read socket ID (socket_id)
> +	Read performance counters (perf/)
> +	Power management (power_mgmt/)
> +	Thermal management (thermal_mgmt/)
> +	Error reporting (errors/)
> +
> +PORT
> +====
> +A port represents the interface between the static FPGA fabric (the "blue
> +bitstream") and a partially reconfigurable region containing an AFU (the "green
> +bitstream"). It controls the communication from SW to the accelerator and
> +exposes features such as reset and debug.
> +
> +A PCIe device may have several ports and each port can be released from PF by
> +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe sriov
> +sysfs interface.
> +
> +AFU
> +===
> +An AFU is attached to a port and exposes a 256k MMIO region to be used for
> +accelerator-specific control registers.
> +
> +User-space applications can acquire exclusive access to an AFU attached to a
> +port by using open() on the port device node, and release it using close().
> +
> +The following functions are exposed through ioctls:
> +
> +	Get driver API version (FPGA_GET_API_VERSION)
> +	Check for extensions (FPGA_CHECK_EXTENSION)
> +	Get port info (FPGA_PORT_GET_INFO)
> +	Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
> +	Map DMA buffer (FPGA_PORT_DMA_MAP)
> +	Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
> +	Reset AFU (FPGA_PORT_RESET)
> +	Enable UMsg (FPGA_PORT_UMSG_ENABLE)
> +	Disable UMsg (FPGA_PORT_UMSG_DISABLE)
> +	Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
> +	Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)
> +
> +User-space applications can also mmap() accelerator MMIO regions.
> +
> +More functions are exposed through sysfs:
> +(/sys/class/fpga/fpga.n/intel-fpga-port.m/):
> +
> +	Read Accelerator GUID (afu_id)
> +	Error reporting (errors/)
> +
> +Partial Reconfiguration
> +=======================
> +As mentioned above, accelerators can be reconfigured through partial
> +reconfiguration of a green bitstream file (GBS). The green bitstream must have
> +been generated for the exact blue bitstream and targeted reconfigurable region
> +(port) of the FPGA; otherwise, the reconfiguration operation will fail and
> +possibly cause system instability. This compatibility can be checked by
> +comparing the interface ID noted in the GBS header against the interface ID
> +exposed by the FME through sysfs (see above). This check is usually done by
> +user-space before calling the reconfiguration IOCTL.
> +
> +FPGA virtualization
> +===================
> +To enable accessing an accelerator from applications running in a VM, the
> +respective AFU's port needs to be assigned to a VF using the following steps:
> +
> + a) The PF owns all AFU ports by default. Any port that needs to be reassigned
> + to a VF must be released from PF firstly through the FPGA_FME_PORT_RELEASE
> + ioctl on the FME device.
> +
> + b) Once N ports are released from PF, then user can use below command to
> + enable SRIOV and VFs. Each VF owns only one Port with AFU.
> +
> + echo N > $PCI_DEVICE_PATH/sriov_numvfs
> +
> + c) Pass through the VFs to VMs
> +
> + d) The AFU under VF is accessiable from applications in VM (using the same
> + driver inside the VF).
> +
> +Note the an FME can't be assigned to a VF, thus PR and other management
> +functions are only available via the PF.
> +
> +
> +Driver organization
> +===================
> +
> +  +------------------+  +---------+   |             +---------+
> +  | +-------+        |  |         |   |             |         |
> +  | | FPGA  |  FME   |  |   AFU   |   |             |   AFU   |
> +  | |Manager| Module |  |  Module |   |             |  Module |
> +  | +-------+        |  |         |   |             |         |
> +  +------------------+  +---------+   |             +---------+
> +        +-----------------------+     |      +-----------------------+
> +        | FPGA Container Device |     |      | FPGA Container Device |
> +        +-----------------------+     |      +-----------------------+
> +          +------------------+        |         +------------------+
> +          | FPGA PCIE Module |        | Virtual | FPGA PCIE Module |
> +          +------------------+   Host | Machine +------------------+
> + ------------------------------------ | ------------------------------
> +           +---------------+          |          +---------------+
> +           | PCI PF Device |          |          | PCI VF Device |
> +           +---------------+          |          +---------------+
> +
> +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe device
> +driver is always loaded first once a FPGA PCIE PF or VF device is detected. This
> +driver plays an infrastructural role in the driver architecuture.  It:
> +
> +	a) creates FPGA container device as parent of the feature devices.
> +	b) walks through the Device Feature List, which is implemented in PCIE
> +	   device BAR memory, to discover feature devices and their sub features
> +	   and create platform device for them under the container device.

I really like the idea of creating platform devices for the sub features. 
It is in line with other FPGA use cases.  Platform devices are 
at the heart of device trees used by processors directly connected FPGAs 
and processors inside FPGAs.

> +	c) supports SRIOV.
> +	d) introduces the feature device infrastructure, which abstracts
> +	   operations for sub features and exposes common functions to feature
> +	   device drivers.
> +
> +The FPGA Management Engine (FME) driver is a platform driver which is loaded
> +automatically after FME platform device creation from the PCIE driver. It
> +provides the key features for FPGA management, including:
> +
> +	a) Power and thermal management, error reporting, performance reporting
> +	   and other infrastructure functions. Users can access these functions
> +	   via sysfs interfaces exposed by FME driver.
> +	b) Paritial Reconfiguration. The FME driver registers a FPGA Manager
> +	   during PR sub feature initialization; once it receives an
> +	   FPGA_FME_PORT_PR ioctl from user, it invokes the common interface
> +	   function from FPGA Manager to complete the partial reconfiguration of
> +	   the bitstream to the given port.
> +	c) Port management for virtualization. The FME driver introduces two
> +	   ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
> +	   FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the port is
> +	   released from the PF, it can be assigned to the VF through the SRIOV
> +	   interfaces provided by PCIE driver. (Refer to "FPGA virtualization"
> +	   for more details).
> +
> +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) driver
> +is probed once the AFU platform device is created. The main function of this
> +module is to provide an interface for userspace applications to access the
> +individual accelerators, including basic reset control on port, AFU MMIO region
> +export, dma buffer mapping service, UMsg notification, and remote debug
> +functions (see above).
> +
> +
> +Device enumeration
> +==================
> +This section introduces how applications enumerate the fpga device from
> +the sysfs hierarchy under /sys/class/fpga.
> +
> +In the example below, two Intel(R) FPGA devices are installed in the host. Each
> +fpga device has one FME and two ports (AFUs).
> +
> +For each FPGA device, a device director is created under /sys/class/fpga/:
> +
> +	/sys/class/fpga/fpga.0
> +	/sys/class/fpga/fpga.1
> +
> +The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's name.
> +Application can retrieve name information via the sysfs interface:
> +
> +	/sys/class/fpga/fpga.0/name
> +
> +Each node has one FME and two ports (AFUs) as child devices:
> +
> +	/sys/class/fpga/fpga.0/intel-fpga-fme.0
> +	/sys/class/fpga/fpga.0/intel-fpga-port.0
> +	/sys/class/fpga/fpga.0/intel-fpga-port.1
> +
> +	/sys/class/fpga/fpga.1/intel-fpga-fme.1
> +	/sys/class/fpga/fpga.1/intel-fpga-port.2
> +	/sys/class/fpga/fpga.1/intel-fpga-port.3
> +
> +In general, the FME/AFU sysfs interfaces are named as follows:
> +
> +	/sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/
> +	/sys/class/fpga/<fpga.n>/<intel-fpga-port.m>/
> +
> +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
> +ports.
> +
> +The device nodes used for ioctl() or mmap() can be referenced through:
> +
> +	/sys/class/fpga/<fpga.n>/<intel-fpga-port.n>/dev
> +	/sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/dev
> +
> +
> +Open discussions
> +================
> +The current FME driver does not provide user space access to the FME MMIO
> +region, but exposes access through sysfs and ioctls. It also provides an FPGA
> +manger interface for partial reconfiguration (PR), but does not make use of
> +fpga-regions. User PR requests via the FPGA_FME_PORT_PR ioctl are handled inside
> +the FME, and fpga-region depends on device tree which is not used at all. There
> +are patches from Alan Tull to separate the device tree specific code and

I am currently trying to use those patches in a different driver.  They've 
compiled cleanly in my out of tree pcie module driver against the 3.10 kernel.
I need to actually write the code to create and register the region, but 
Alan's platform driver code should be a good guide for me.  Just need to 
find the time.

> +introduce a sysfs interface for PR. We plan to add fpga-regions support in the
> +driver once the related patches get merged. Then the FME driver should create
> +one fpga-region for each Port/AFU.

Does the FME driver create the fpga-region, or is each region described 
as an entry in the Device Feature List and therefore created by the code 
that enumerates the Device Feature List?

> -- 
> 2.7.4
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview
  2017-03-31 18:24   ` matthew.gerlach
@ 2017-03-31 18:38     ` Alan Tull
  2017-04-01 11:16       ` Wu Hao
  0 siblings, 1 reply; 93+ messages in thread
From: Alan Tull @ 2017-03-31 18:38 UTC (permalink / raw)
  To: matthew.gerlach
  Cc: Wu Hao, Moritz Fischer, linux-fpga, linux-kernel, luwei.kang,
	yi.z.zhang, Enno Luebbers, Xiao Guangrong

On Fri, Mar 31, 2017 at 1:24 PM,  <matthew.gerlach@linux.intel.com> wrote:
>
>
> On Thu, 30 Mar 2017, Wu Hao wrote:
>
>
> Hi Wu Hao,
>
> Great documentation. I'm looking forward to diving into the rest of the
> patches. Please see my comments inline.
>
> Matthew Gerlach
>
>
>> Add a document for Intel FPGA driver overview.
>>
>> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
>> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> Signed-off-by: Wu Hao <hao.wu@intel.com>
>> ---
>> Documentation/fpga/intel-fpga.txt | 259
>> ++++++++++++++++++++++++++++++++++++++
>> 1 file changed, 259 insertions(+)
>> create mode 100644 Documentation/fpga/intel-fpga.txt
>>
>> diff --git a/Documentation/fpga/intel-fpga.txt
>> b/Documentation/fpga/intel-fpga.txt
>> new file mode 100644
>> index 0000000..9396cea
>> --- /dev/null
>> +++ b/Documentation/fpga/intel-fpga.txt
>> @@ -0,0 +1,259 @@
>>
>> +===============================================================================
>> +                    Intel FPGA driver Overview
>>
>> +-------------------------------------------------------------------------------
>> +                Enno Luebbers <enno.luebbers@intel.com>
>> +                Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> +                Wu Hao <hao.wu@intel.com>
>> +
>> +The Intel FPGA driver provides interfaces for userspace applications to
>> +configure, enumerate, open, and access FPGA accelerators on platforms
>> equipped
>> +with Intel(R) FPGA solutions and enables system level management
>> functions such
>> +as FPGA reconfiguration, power management, and virtualization.
>> +
>
>
> From a Linux kernel perspective, I'm not sure this is the best name for
> this code.  The name gives me the impression that it is a driver for all
> Intel FPGAs, but not all Intel FPGAs are connected to the processor over a
> PCIe bus.  The processor could be directely connected like the Arria10
> SOCFPGA.  Such a processor could certainly benefit from this accelerator
> usage model.  In an extreme case, couldn't a processor in the FPGA,
> running Linux, also benefit from this accelerator model?  Is this code a
> "FPGA Accelerator Framework"?
>
>> +HW Architecture
>> +===============
>> +From the OS's point of view, the FPGA hardware appears as a regular PCIe
>> device.
>> +The FPGA device memory is organized using a predefined data structure
>> (Device
>> +Feature List). Features supported by the particular FPGA device are
>> exposed
>> +through these data structures, as illustrated below:
>> +
>> +  +-------------------------------+  +-------------+
>> +  |              PF               |  |     VF      |
>> +  +-------------------------------+  +-------------+
>> +      ^            ^         ^              ^
>> +      |            |         |              |
>> ++-----|------------|---------|--------------|-------+
>> +|     |            |         |              |       |
>> +|  +-----+     +-------+ +-------+      +-------+   |
>> +|  | FME |     | Port0 | | Port1 |      | Port2 |   |
>> +|  +-----+     +-------+ +-------+      +-------+   |
>> +|                  ^         ^              ^       |
>> +|                  |         |              |       |
>> +|              +-------+ +------+       +-------+   |
>> +|              |  AFU  | |  AFU |       |  AFU  |   |
>> +|              +-------+ +------+       +-------+   |
>> +|                                                   |
>> +|                 FPGA PCIe Device                  |
>> ++---------------------------------------------------+
>> +
>> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which
>> can be
>> +used to assign individual accelerators to virtual machines .
>
>
> Does this HW Architecture require an Intel FPGA?  Couldn't any vendors FPGA
> be used as long as it presented itself the PCIe bus the same and contained
> an appropriate Device Feature List?
>
>> +
>> +FME (FPGA Management Engine)
>> +============================
>> +The FPGA Management Enging performs power and thermal management, error
>> +reporting, reconfiguration, performance reporting, and other
>> infrastructure
>> +functions. Each FPGA has one FME, which is always accessed through the
>> physical
>> +function (PF).
>> +
>> +User-space applications can acquire exclusive access to the FME using
>> open(),
>> +and release it using close().
>> +
>> +The following functions are exposed through ioctls:
>> +
>> +       Get driver API version (FPGA_GET_API_VERSION)
>> +       Check for extensions (FPGA_CHECK_EXTENSION)
>> +       Assign port to PF (FPGA_FME_PORT_ASSIGN)
>> +       Release port from PF (FPGA_FME_PORT_RELEASE)
>> +       Program bitstream (FPGA_FME_PORT_PR)
>> +
>> +More functions are exposed through sysfs
>> +(/sys/class/fpga/fpga.n/intel-fpga-fme.n/):
>> +
>> +       Read bitstream ID (bitstream_id)
>> +       Read bitstream metadata (bitstream_metadata)
>> +       Read number of ports (ports_num)
>> +       Read socket ID (socket_id)
>> +       Read performance counters (perf/)
>> +       Power management (power_mgmt/)
>> +       Thermal management (thermal_mgmt/)
>> +       Error reporting (errors/)
>> +
>> +PORT
>> +====
>> +A port represents the interface between the static FPGA fabric (the "blue
>> +bitstream") and a partially reconfigurable region containing an AFU (the
>> "green

Is this an fpga bridge but with added features?

>> +bitstream"). It controls the communication from SW to the accelerator and
>> +exposes features such as reset and debug.
>> +
>> +A PCIe device may have several ports and each port can be released from
>> PF by
>> +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe
>> sriov
>> +sysfs interface.
>> +
>> +AFU
>> +===
>> +An AFU is attached to a port and exposes a 256k MMIO region to be used
>> for
>> +accelerator-specific control registers.
>> +
>> +User-space applications can acquire exclusive access to an AFU attached
>> to a
>> +port by using open() on the port device node, and release it using
>> close().
>> +
>> +The following functions are exposed through ioctls:
>> +
>> +       Get driver API version (FPGA_GET_API_VERSION)
>> +       Check for extensions (FPGA_CHECK_EXTENSION)
>> +       Get port info (FPGA_PORT_GET_INFO)
>> +       Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
>> +       Map DMA buffer (FPGA_PORT_DMA_MAP)
>> +       Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
>> +       Reset AFU (FPGA_PORT_RESET)
>> +       Enable UMsg (FPGA_PORT_UMSG_ENABLE)
>> +       Disable UMsg (FPGA_PORT_UMSG_DISABLE)
>> +       Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
>> +       Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)
>> +
>> +User-space applications can also mmap() accelerator MMIO regions.
>> +
>> +More functions are exposed through sysfs:
>> +(/sys/class/fpga/fpga.n/intel-fpga-port.m/):
>> +
>> +       Read Accelerator GUID (afu_id)
>> +       Error reporting (errors/)
>> +
>> +Partial Reconfiguration
>> +=======================
>> +As mentioned above, accelerators can be reconfigured through partial
>> +reconfiguration of a green bitstream file (GBS). The green bitstream must
>> have
>> +been generated for the exact blue bitstream and targeted reconfigurable
>> region
>> +(port) of the FPGA; otherwise, the reconfiguration operation will fail
>> and
>> +possibly cause system instability. This compatibility can be checked by
>> +comparing the interface ID noted in the GBS header against the interface
>> ID
>> +exposed by the FME through sysfs (see above). This check is usually done
>> by
>> +user-space before calling the reconfiguration IOCTL.
>> +
>> +FPGA virtualization
>> +===================
>> +To enable accessing an accelerator from applications running in a VM, the
>> +respective AFU's port needs to be assigned to a VF using the following
>> steps:
>> +
>> + a) The PF owns all AFU ports by default. Any port that needs to be
>> reassigned
>> + to a VF must be released from PF firstly through the
>> FPGA_FME_PORT_RELEASE
>> + ioctl on the FME device.
>> +
>> + b) Once N ports are released from PF, then user can use below command to
>> + enable SRIOV and VFs. Each VF owns only one Port with AFU.
>> +
>> + echo N > $PCI_DEVICE_PATH/sriov_numvfs
>> +
>> + c) Pass through the VFs to VMs
>> +
>> + d) The AFU under VF is accessiable from applications in VM (using the
>> same
>> + driver inside the VF).
>> +
>> +Note the an FME can't be assigned to a VF, thus PR and other management
>> +functions are only available via the PF.
>> +
>> +
>> +Driver organization
>> +===================
>> +
>> +  +------------------+  +---------+   |             +---------+
>> +  | +-------+        |  |         |   |             |         |
>> +  | | FPGA  |  FME   |  |   AFU   |   |             |   AFU   |
>> +  | |Manager| Module |  |  Module |   |             |  Module |
>> +  | +-------+        |  |         |   |             |         |
>> +  +------------------+  +---------+   |             +---------+
>> +        +-----------------------+     |      +-----------------------+
>> +        | FPGA Container Device |     |      | FPGA Container Device |
>> +        +-----------------------+     |      +-----------------------+
>> +          +------------------+        |         +------------------+
>> +          | FPGA PCIE Module |        | Virtual | FPGA PCIE Module |
>> +          +------------------+   Host | Machine +------------------+
>> + ------------------------------------ | ------------------------------
>> +           +---------------+          |          +---------------+
>> +           | PCI PF Device |          |          | PCI VF Device |
>> +           +---------------+          |          +---------------+
>> +
>> +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe
>> device
>> +driver is always loaded first once a FPGA PCIE PF or VF device is
>> detected. This
>> +driver plays an infrastructural role in the driver architecuture.  It:
>> +
>> +       a) creates FPGA container device as parent of the feature devices.
>> +       b) walks through the Device Feature List, which is implemented in
>> PCIE
>> +          device BAR memory, to discover feature devices and their sub
>> features
>> +          and create platform device for them under the container device.
>
>
> I really like the idea of creating platform devices for the sub features. It
> is in line with other FPGA use cases.  Platform devices are at the heart of
> device trees used by processors directly connected FPGAs and processors
> inside FPGAs.
>
>> +       c) supports SRIOV.
>> +       d) introduces the feature device infrastructure, which abstracts
>> +          operations for sub features and exposes common functions to
>> feature
>> +          device drivers.
>> +
>> +The FPGA Management Engine (FME) driver is a platform driver which is
>> loaded
>> +automatically after FME platform device creation from the PCIE driver. It
>> +provides the key features for FPGA management, including:
>> +
>> +       a) Power and thermal management, error reporting, performance
>> reporting
>> +          and other infrastructure functions. Users can access these
>> functions
>> +          via sysfs interfaces exposed by FME driver.
>> +       b) Paritial Reconfiguration. The FME driver registers a FPGA
>> Manager
>> +          during PR sub feature initialization; once it receives an
>> +          FPGA_FME_PORT_PR ioctl from user, it invokes the common
>> interface
>> +          function from FPGA Manager to complete the partial
>> reconfiguration of
>> +          the bitstream to the given port.
>> +       c) Port management for virtualization. The FME driver introduces
>> two
>> +          ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
>> +          FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the
>> port is
>> +          released from the PF, it can be assigned to the VF through the
>> SRIOV
>> +          interfaces provided by PCIE driver. (Refer to "FPGA
>> virtualization"
>> +          for more details).
>> +
>> +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU)
>> driver
>> +is probed once the AFU platform device is created. The main function of
>> this
>> +module is to provide an interface for userspace applications to access
>> the
>> +individual accelerators, including basic reset control on port, AFU MMIO
>> region
>> +export, dma buffer mapping service, UMsg notification, and remote debug
>> +functions (see above).
>> +
>> +
>> +Device enumeration
>> +==================
>> +This section introduces how applications enumerate the fpga device from
>> +the sysfs hierarchy under /sys/class/fpga.
>> +
>> +In the example below, two Intel(R) FPGA devices are installed in the
>> host. Each
>> +fpga device has one FME and two ports (AFUs).
>> +
>> +For each FPGA device, a device director is created under
>> /sys/class/fpga/:
>> +
>> +       /sys/class/fpga/fpga.0
>> +       /sys/class/fpga/fpga.1
>> +
>> +The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's
>> name.
>> +Application can retrieve name information via the sysfs interface:
>> +
>> +       /sys/class/fpga/fpga.0/name
>> +
>> +Each node has one FME and two ports (AFUs) as child devices:
>> +
>> +       /sys/class/fpga/fpga.0/intel-fpga-fme.0
>> +       /sys/class/fpga/fpga.0/intel-fpga-port.0
>> +       /sys/class/fpga/fpga.0/intel-fpga-port.1
>> +
>> +       /sys/class/fpga/fpga.1/intel-fpga-fme.1
>> +       /sys/class/fpga/fpga.1/intel-fpga-port.2
>> +       /sys/class/fpga/fpga.1/intel-fpga-port.3
>> +
>> +In general, the FME/AFU sysfs interfaces are named as follows:
>> +
>> +       /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/
>> +       /sys/class/fpga/<fpga.n>/<intel-fpga-port.m>/
>> +
>> +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering
>> all
>> +ports.
>> +
>> +The device nodes used for ioctl() or mmap() can be referenced through:
>> +
>> +       /sys/class/fpga/<fpga.n>/<intel-fpga-port.n>/dev
>> +       /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/dev
>> +
>> +
>> +Open discussions
>> +================
>> +The current FME driver does not provide user space access to the FME MMIO
>> +region, but exposes access through sysfs and ioctls. It also provides an
>> FPGA
>> +manger interface for partial reconfiguration (PR), but does not make use
>> of
>> +fpga-regions. User PR requests via the FPGA_FME_PORT_PR ioctl are handled
>> inside
>> +the FME, and fpga-region depends on device tree which is not used at all.
>> There
>> +are patches from Alan Tull to separate the device tree specific code and
>
>
> I am currently trying to use those patches in a different driver.  They've
> compiled cleanly in my out of tree pcie module driver against the 3.10
> kernel.
> I need to actually write the code to create and register the region, but
> Alan's platform driver code should be a good guide for me.  Just need to
> find the time.
>
>> +introduce a sysfs interface for PR. We plan to add fpga-regions support
>> in the
>> +driver once the related patches get merged. Then the FME driver should
>> create
>> +one fpga-region for each Port/AFU.
>
>
> Does the FME driver create the fpga-region, or is each region described as
> an entry in the Device Feature List and therefore created by the code that
> enumerates the Device Feature List?
>
>> --
>> 2.7.4
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 02/16] fpga: add FPGA device framework
  2017-03-31  7:48     ` Wu Hao
  2017-03-31  9:03       ` Greg KH
@ 2017-03-31 19:01       ` matthew.gerlach
  2017-04-01 12:18         ` Wu Hao
  1 sibling, 1 reply; 93+ messages in thread
From: matthew.gerlach @ 2017-03-31 19:01 UTC (permalink / raw)
  To: Wu Hao
  Cc: Greg KH, atull, moritz.fischer, linux-fpga, linux-kernel,
	luwei.kang, yi.z.zhang, Tim Whisonant, Enno Luebbers, Shiva Rao,
	Christopher Rauer, Xiao Guangrong



On Fri, 31 Mar 2017, Wu Hao wrote:

> On Fri, Mar 31, 2017 at 08:09:09AM +0200, Greg KH wrote:
>> On Thu, Mar 30, 2017 at 08:08:02PM +0800, Wu Hao wrote:
>>> During FPGA device (e.g PCI-based) discovery, platform devices are
>>> registered for different FPGA function units. But the device node path
>>> isn't quite friendly to applications.
>>>
>>> Consider this case, applications want to access child device's sysfs file
>>> for some information.
>>>
>>> 1) Access using bus-based path (e.g PCI)
>>>
>>>   /sys/bus/pci/devices/xxxxx/fpga_func_a.0/sysfs_file
>>>
>>>   From the path, it's clear which PCI device is the parent, but not perfect
>>>   solution for applications. PCI device BDF is not fixed, application may
>>>   need to search all PCI device to find the actual FPGA Device.
>>>
>>> 2) Or access using platform device path
>>>
>>>   /sys/bus/platform/devices/fpga_func_a.0/sysfs_file
>>>
>>>   Applications find the actual function by name easily, but no information
>>>   about which fpga device it belongs to. It's quite confusing if multiple
>>>   FPGA devices are in one system.
>>>
>>> 'FPGA Device' class is introduced to resolve this problem. Each node under
>>> this class represents a fpga device, which may have one or more child
>>> devices. Applications only need to search under this FPGA Device class
>>> folder to find the child device node it needs.
>>>
>>> For example, for the platform has 2 fpga devices, each fpga device has
>>> 3 child devices, the hierarchy looks like this.
>>>
>>> Two nodes are under /sys/class/fpga/:
>>> /sys/class/fpga/fpga.0
>>> /sys/class/fpga/fpga.1
>>>
>>> Each node has 1 function A device and 2 function B devices:
>>> /sys/class/fpga/fpga.0/func_a.0
>>> /sys/class/fpga/fpga.0/func_b.0
>>> /sys/class/fpga/fpga.0/func_b.1
>>>
>>> /sys/class/fpga/fpga.1/func_a.1
>>> /sys/class/fpga/fpga.1/func_b.2
>>> /sys/class/fpga/fpga.1/func_b.3
>>>
>>> This following APIs are provided by FPGA device framework:
>>> * fpga_dev_create
>>>   Create fpga device under the given parent device.
>>> * fpga_dev_destroy
>>>   Destroy fpga device
>>>
>>> The following sysfs files are created:
>>> * /sys/class/fpga/<fpga.x>/name
>>>   Name of the fpga device.
>>
>> How does this interact with the existing "fpga class" that is in the
>> kernel already?
>
> The fpga-dev introduced by this patch, is only a container device, and

I completely understand the need for a container device.  The fpga-region 
is also primarily a container, and in some cases the fpga-region may 
represent the entire fpga.  Over time this code may become redundant.

> drivers could register different functions under it. Per my understanding,
> the existing "fpga class", including fpga-region, fpga-bridge and
> fpga-manager, is used to provide reconfiguration function for FPGA. So
> driver can create child node using this existing "fpga class" to provide
> FPGA reconfiguration function, and more nodes under this container for
> different functions for given FPGA device.
>
> For Intel FPGA device, partial reconfiguration is only one function of
> Intel FPGA Management Engine (FME). FME driver creates fpga_manager under
> below path for partial reconfiguration, and other interfaces for more
> functions, e.g power management, virtualization support and etc.
>
> /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/fpga_manager
>
> Thanks
> Hao
>
>>
>> thanks,
>>
>> greg k-h
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-03-30 12:08 ` [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Wu Hao
  2017-03-30 20:30   ` Alan Tull
@ 2017-03-31 19:10   ` Alan Tull
  2017-04-01 11:08     ` Wu Hao
  2017-03-31 23:45   ` kbuild test robot
                     ` (2 subsequent siblings)
  4 siblings, 1 reply; 93+ messages in thread
From: Alan Tull @ 2017-03-31 19:10 UTC (permalink / raw)
  To: Wu Hao
  Cc: Moritz Fischer, linux-fpga, linux-kernel, luwei.kang, yi.z.zhang,
	Tim Whisonant, Enno Luebbers, Shiva Rao, Christopher Rauer,
	Alan Tull, Xiao Guangrong

On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote:
> From: Kang Luwei <luwei.kang@intel.com>
>
> Partial Reconfiguration (PR) is the most important function for FME. It
> allows reconfiguration for given Port/Accelerated Function Unit (AFU).
>
> This patch adds support for PR sub feature. In this patch, it registers
> a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
> for PR operation once PR request received via ioctl. Below user space
> interfaces are exposed by this sub feature.
>
> Sysfs interface:
> * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
>   Read-only. Indicate the hardware interface information. Userspace
>   applications need to check this interface to select correct green
>   bitstream format before PR.
>
> Ioctl interface:
> * FPGA_FME_PORT_PR
>   Do partial reconfiguration per information from userspace, including
>   target port(AFU), buffer size and address info. It returns the PR status
>   (PR error code if failed) to userspace.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Alan Tull <alan.tull@intel.com>
> Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ---
>  drivers/fpga/intel/Makefile      |   2 +-
>  drivers/fpga/intel/feature-dev.h |  58 ++++++
>  drivers/fpga/intel/fme-main.c    |  44 ++++-
>  drivers/fpga/intel/fme-pr.c      | 400 +++++++++++++++++++++++++++++++++++++++
>  drivers/fpga/intel/fme.h         |  32 ++++
>  include/uapi/linux/intel-fpga.h  |  44 +++++
>  6 files changed, 578 insertions(+), 2 deletions(-)
>  create mode 100644 drivers/fpga/intel/fme-pr.c
>  create mode 100644 drivers/fpga/intel/fme.h
>
> diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
> index 546861d..0452cb6 100644
> --- a/drivers/fpga/intel/Makefile
> +++ b/drivers/fpga/intel/Makefile
> @@ -2,4 +2,4 @@ obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
>  obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
>
>  intel-fpga-pci-objs := pcie.o feature-dev.o
> -intel-fpga-fme-objs := fme-main.o
> +intel-fpga-fme-objs := fme-main.o fme-pr.o
> diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
> index dccc283..5a25c915 100644
> --- a/drivers/fpga/intel/feature-dev.h
> +++ b/drivers/fpga/intel/feature-dev.h
> @@ -150,8 +150,66 @@ struct feature_fme_err {
>  };
>
>  /* FME Partial Reconfiguration Sub Feature Register Set */
> +/* FME PR Control Register */
> +struct feature_fme_pr_ctl {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u8  pr_reset:1;         /* Reset PR Engine */
> +                       u8  rsvdz1:3;
> +                       u8  pr_reset_ack:1;     /* Reset PR Engine Ack */
> +                       u8  rsvdz2:3;
> +                       u8  pr_regionid:2;      /* PR Region ID */
> +                       u8  rsvdz3:2;
> +                       u8  pr_start_req:1;     /* PR Start Request */
> +                       u8  pr_push_complete:1; /* PR Data push complete */
> +                       u8  pr_kind:1;          /* Load Customer or Intel GBS */
> +                       u32 rsvdz4:17;
> +                       u32 config_data;
> +               };
> +       };
> +};
> +
> +/* FME PR Status Register */
> +struct feature_fme_pr_status {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u16 pr_credit:9;        /* Number of PR Credits */
> +                       u8  rsvdz1:7;
> +                       u8  pr_status:1;        /* PR Operation status */
> +                       u8  rsvdz2:3;
> +                       u8  pr_ctrlr_status:3;  /* Controller status */
> +                       u8  rsvdz3:1;
> +                       u8  pr_host_status:4;   /* PR Host status */
> +                       u64 rsvdz4:36;
> +               };
> +       };
> +};
> +
> +/* FME PR Data Register */
> +struct feature_fme_pr_data {
> +       union {
> +               u64 csr;
> +               struct {
> +                       /* PR data from the raw-binary file */
> +                       u32 pr_data_raw;
> +                       u32 rsvd;
> +               };
> +       };
> +};
> +
>  struct feature_fme_pr {
>         struct feature_header header;
> +       struct feature_fme_pr_ctl control;
> +       struct feature_fme_pr_status status;
> +       struct feature_fme_pr_data data;
> +       u64 error;
> +
> +       u64 rsvd[16];
> +
> +       u64 intfc_id_l;         /* PR interface Id Low */
> +       u64 intfc_id_h;         /* PR interface Id High */
>  };
>
>  /* PORT Header Register Set */
> diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
> index 36d0c4c..0d9a7a6 100644
> --- a/drivers/fpga/intel/fme-main.c
> +++ b/drivers/fpga/intel/fme-main.c
> @@ -23,6 +23,7 @@
>  #include <linux/intel-fpga.h>
>
>  #include "feature-dev.h"
> +#include "fme.h"
>
>  static ssize_t ports_num_show(struct device *dev,
>                               struct device_attribute *attr, char *buf)
> @@ -101,6 +102,10 @@ static struct feature_driver fme_feature_drvs[] = {
>                 .ops = &fme_hdr_ops,
>         },
>         {
> +               .name = FME_FEATURE_PR_MGMT,
> +               .ops = &pr_mgmt_ops,
> +       },
> +       {
>                 .ops = NULL,
>         },
>  };
> @@ -182,14 +187,48 @@ static const struct file_operations fme_fops = {
>         .unlocked_ioctl = fme_ioctl,
>  };
>
> +static int fme_dev_init(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct fpga_fme *fme;
> +
> +       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
> +       if (!fme)
> +               return -ENOMEM;
> +
> +       fme->pdata = pdata;
> +
> +       mutex_lock(&pdata->lock);
> +       fpga_pdata_set_private(pdata, fme);
> +       mutex_unlock(&pdata->lock);
> +       return 0;
> +}
> +
> +static void fme_dev_destroy(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct fpga_fme *fme;
> +
> +       mutex_lock(&pdata->lock);
> +       fme = fpga_pdata_get_private(pdata);
> +       fpga_pdata_set_private(pdata, NULL);
> +       mutex_unlock(&pdata->lock);
> +
> +       devm_kfree(&pdev->dev, fme);
> +}
> +
>  static int fme_probe(struct platform_device *pdev)
>  {
>         int ret;
>
> -       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> +       ret = fme_dev_init(pdev);
>         if (ret)
>                 goto exit;
>
> +       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> +       if (ret)
> +               goto dev_destroy;
> +
>         ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
>         if (ret)
>                 goto feature_uinit;
> @@ -198,6 +237,8 @@ static int fme_probe(struct platform_device *pdev)
>
>  feature_uinit:
>         fpga_dev_feature_uinit(pdev);
> +dev_destroy:
> +       fme_dev_destroy(pdev);
>  exit:
>         return ret;
>  }
> @@ -206,6 +247,7 @@ static int fme_remove(struct platform_device *pdev)
>  {
>         fpga_dev_feature_uinit(pdev);
>         fpga_unregister_dev_ops(pdev);
> +       fme_dev_destroy(pdev);
>         return 0;
>  }
>
> diff --git a/drivers/fpga/intel/fme-pr.c b/drivers/fpga/intel/fme-pr.c
> new file mode 100644
> index 0000000..3b44a3e
> --- /dev/null
> +++ b/drivers/fpga/intel/fme-pr.c
> @@ -0,0 +1,400 @@
> +/*
> + * Driver for Intel FPGA Management Engine (FME) Partial Reconfiguration
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <luwei.kang@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *   Joseph Grecco <joe.grecco@intel.com>
> + *   Enno Luebbers <enno.luebbers@intel.com>
> + *   Tim Whisonant <tim.whisonant@intel.com>
> + *   Ananda Ravuri <ananda.ravuri@intel.com>
> + *   Christopher Rauer <christopher.rauer@intel.com>
> + *   Henry Mitchel <henry.mitchel@intel.com>
> + *
> + * This work is licensed under a dual BSD/GPLv2 license. When using or
> + * redistributing this file, you may do so under either license. See the
> + * LICENSE.BSD file under this directory for the BSD license and see
> + * the COPYING file in the top-level directory for the GPLv2 license.
> + */
> +
> +#include <linux/types.h>
> +#include <linux/device.h>
> +#include <linux/vmalloc.h>
> +#include <linux/uaccess.h>
> +#include <linux/fpga/fpga-mgr.h>
> +#include <linux/intel-fpga.h>
> +
> +#include "feature-dev.h"
> +#include "fme.h"
> +
> +#define PR_WAIT_TIMEOUT   8000000
> +
> +#define PR_HOST_STATUS_IDLE    0
> +
> +DEFINE_FPGA_PR_ERR_MSG(pr_err_msg);
> +
> +static ssize_t interface_id_show(struct device *dev,
> +                                struct device_attribute *attr, char *buf)
> +{
> +       u64 intfc_id_l, intfc_id_h;
> +       struct feature_fme_pr *fme_pr
> +               = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_PR_MGMT);
> +
> +       intfc_id_l = readq(&fme_pr->intfc_id_l);
> +       intfc_id_h = readq(&fme_pr->intfc_id_h);
> +
> +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> +                       (unsigned long long)intfc_id_h,
> +                       (unsigned long long)intfc_id_l);
> +}
> +static DEVICE_ATTR_RO(interface_id);
> +
> +static struct attribute *pr_mgmt_attrs[] = {
> +       &dev_attr_interface_id.attr,
> +       NULL,
> +};
> +
> +struct attribute_group pr_mgmt_attr_group = {
> +       .attrs  = pr_mgmt_attrs,
> +       .name   = "pr",
> +};
> +
> +static u64
> +pr_err_handle(struct platform_device *pdev, struct feature_fme_pr *fme_pr)
> +{
> +       struct feature_fme_pr_status fme_pr_status;
> +       unsigned long err_code;
> +       u64 fme_pr_error;
> +       int i = 0;
> +
> +       fme_pr_status.csr = readq(&fme_pr->status);
> +       if (!fme_pr_status.pr_status)
> +               return 0;
> +
> +       err_code = fme_pr_error = readq(&fme_pr->error);
> +       for_each_set_bit(i, &err_code, PR_MAX_ERR_NUM)
> +               dev_dbg(&pdev->dev, "%s\n", pr_err_msg[i]);
> +       writeq(fme_pr_error, &fme_pr->error);
> +       return fme_pr_error;
> +}
> +
> +static int fme_pr_write_init(struct fpga_manager *mgr,
> +               struct fpga_image_info *info, const char *buf, size_t count)
> +{
> +       struct fpga_fme *fme = mgr->priv;
> +       struct platform_device *pdev;
> +       struct feature_fme_pr *fme_pr;
> +       struct feature_fme_pr_ctl fme_pr_ctl;
> +       struct feature_fme_pr_status fme_pr_status;
> +
> +       pdev = fme->pdata->dev;
> +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> +                               FME_FEATURE_ID_PR_MGMT);
> +       if (!fme_pr)
> +               return -EINVAL;
> +
> +       if (WARN_ON(info->flags != FPGA_MGR_PARTIAL_RECONFIG))
> +               return -EINVAL;
> +
> +       dev_dbg(&pdev->dev, "resetting PR before initiated PR\n");
> +
> +       fme_pr_ctl.csr = readq(&fme_pr->control);
> +       fme_pr_ctl.pr_reset = 1;
> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> +
> +       fme_pr_ctl.pr_reset_ack = 1;
> +
> +       if (fpga_wait_register_field(pr_reset_ack, fme_pr_ctl,
> +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> +               dev_err(&pdev->dev, "maximum PR timeout\n");
> +               return -ETIMEDOUT;
> +       }
> +
> +       fme_pr_ctl.csr = readq(&fme_pr->control);
> +       fme_pr_ctl.pr_reset = 0;
> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> +
> +       dev_dbg(&pdev->dev,
> +               "waiting for PR resource in HW to be initialized and ready\n");
> +
> +       fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
> +
> +       if (fpga_wait_register_field(pr_host_status, fme_pr_status,
> +                                    &fme_pr->status, PR_WAIT_TIMEOUT, 1)) {
> +               dev_err(&pdev->dev, "maximum PR timeout\n");
> +               return -ETIMEDOUT;
> +       }
> +
> +       dev_dbg(&pdev->dev, "check if have any previous PR error\n");
> +       pr_err_handle(pdev, fme_pr);
> +       return 0;
> +}
> +
> +static int fme_pr_write(struct fpga_manager *mgr,
> +                       const char *buf, size_t count)
> +{
> +       struct fpga_fme *fme = mgr->priv;
> +       struct platform_device *pdev;
> +       struct feature_fme_pr *fme_pr;
> +       struct feature_fme_pr_ctl fme_pr_ctl;
> +       struct feature_fme_pr_status fme_pr_status;
> +       struct feature_fme_pr_data fme_pr_data;
> +       int delay, pr_credit, i = 0;
> +
> +       pdev = fme->pdata->dev;
> +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> +                               FME_FEATURE_ID_PR_MGMT);
> +
> +       dev_dbg(&pdev->dev, "set PR port ID and start request\n");
> +
> +       fme_pr_ctl.csr = readq(&fme_pr->control);
> +       fme_pr_ctl.pr_regionid = fme->port_id;
> +       fme_pr_ctl.pr_start_req = 1;
> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> +
> +       dev_dbg(&pdev->dev, "pushing data from bitstream to HW\n");
> +
> +       fme_pr_status.csr = readq(&fme_pr->status);
> +       pr_credit = fme_pr_status.pr_credit;
> +
> +       while (count > 0) {
> +               delay = 0;
> +               while (pr_credit <= 1) {
> +                       if (delay++ > PR_WAIT_TIMEOUT) {
> +                               dev_err(&pdev->dev, "maximum try\n");
> +                               return -ETIMEDOUT;
> +                       }
> +                       udelay(1);
> +
> +                       fme_pr_status.csr = readq(&fme_pr->status);
> +                       pr_credit = fme_pr_status.pr_credit;
> +               };
> +
> +               if (count >= 4) {
> +                       fme_pr_data.rsvd = 0;
> +                       fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
> +                       writeq(fme_pr_data.csr, &fme_pr->data);
> +                       count -= 4;
> +                       pr_credit--;
> +                       i++;
> +               } else {
> +                       WARN_ON(1);
> +                       return -EINVAL;
> +               }
> +       }
> +
> +       return 0;
> +}
> +
> +static int fme_pr_write_complete(struct fpga_manager *mgr,
> +                       struct fpga_image_info *info)
> +{
> +       struct fpga_fme *fme = mgr->priv;
> +       struct platform_device *pdev;
> +       struct feature_fme_pr *fme_pr;
> +       struct feature_fme_pr_ctl fme_pr_ctl;
> +
> +       pdev = fme->pdata->dev;
> +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> +                               FME_FEATURE_ID_PR_MGMT);
> +
> +       fme_pr_ctl.csr = readq(&fme_pr->control);
> +       fme_pr_ctl.pr_push_complete = 1;
> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> +
> +       dev_dbg(&pdev->dev, "green bitstream push complete\n");
> +       dev_dbg(&pdev->dev, "waiting for HW to release PR resource\n");
> +
> +       fme_pr_ctl.pr_start_req = 0;
> +
> +       if (fpga_wait_register_field(pr_start_req, fme_pr_ctl,
> +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> +               dev_err(&pdev->dev, "maximum try.\n");
> +               return -ETIMEDOUT;
> +       }
> +
> +       dev_dbg(&pdev->dev, "PR operation complete, checking status\n");
> +       fme->pr_err = pr_err_handle(pdev, fme_pr);
> +       if (fme->pr_err)
> +               return -EIO;
> +
> +       dev_dbg(&pdev->dev, "PR done successfully\n");
> +       return 0;
> +}
> +
> +static enum fpga_mgr_states fme_pr_state(struct fpga_manager *mgr)
> +{
> +       return FPGA_MGR_STATE_UNKNOWN;
> +}
> +
> +static const struct fpga_manager_ops fme_pr_ops = {
> +       .write_init = fme_pr_write_init,
> +       .write = fme_pr_write,
> +       .write_complete = fme_pr_write_complete,
> +       .state = fme_pr_state,
> +};
> +
> +static int fme_pr(struct platform_device *pdev, unsigned long arg)
> +{
> +       void __user *argp = (void __user *)arg;
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct fpga_fme *fme;
> +       struct fpga_manager *mgr;
> +       struct feature_fme_header *fme_hdr;
> +       struct feature_fme_capability fme_capability;
> +       struct fpga_image_info info;
> +       struct fpga_fme_port_pr port_pr;
> +       struct platform_device *port;
> +       unsigned long minsz;
> +       void *buf = NULL;
> +       int ret = 0;
> +
> +       minsz = offsetofend(struct fpga_fme_port_pr, status);
> +
> +       if (copy_from_user(&port_pr, argp, minsz))
> +               return -EFAULT;
> +
> +       if (port_pr.argsz < minsz || port_pr.flags)
> +               return -EINVAL;
> +
> +       if (!IS_ALIGNED(port_pr.buffer_size, 4))
> +               return -EINVAL;
> +
> +       /* get fme header region */
> +       fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> +                                       FME_FEATURE_ID_HEADER);
> +       if (WARN_ON(!fme_hdr))
> +               return -EINVAL;
> +
> +       /* check port id */
> +       fme_capability.csr = readq(&fme_hdr->capability);
> +       if (port_pr.port_id >= fme_capability.num_ports) {
> +               dev_dbg(&pdev->dev, "port number more than maximum\n");
> +               return -EINVAL;
> +       }
> +
> +       if (!access_ok(VERIFY_READ, port_pr.buffer_address,
> +                                   port_pr.buffer_size))
> +               return -EFAULT;
> +
> +       buf = vmalloc(port_pr.buffer_size);
> +       if (!buf)
> +               return -ENOMEM;
> +
> +       if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
> +                                              port_pr.buffer_size)) {
> +               ret = -EFAULT;
> +               goto free_exit;
> +       }
> +
> +       memset(&info, 0, sizeof(struct fpga_image_info));
> +       info.flags = FPGA_MGR_PARTIAL_RECONFIG;
> +
> +       mgr = fpga_mgr_get(&pdev->dev);
> +       if (IS_ERR(mgr)) {
> +               ret = PTR_ERR(mgr);
> +               goto free_exit;
> +       }
> +
> +       mutex_lock(&pdata->lock);
> +       fme = fpga_pdata_get_private(pdata);
> +       /* fme device has been unregistered. */
> +       if (!fme) {
> +               ret = -EINVAL;
> +               goto unlock_exit;
> +       }
> +
> +       fme->pr_err = 0;
> +       fme->port_id = port_pr.port_id;

It looks like you're using private data to communicate with the
driver, i.e. there is something you want to do with the fpga manager
framework and it doesn't have that feature.  The better way would be
for us to expand the framework so you don't need to do that.

port_id is the kind of thing that should be communicated to the driver
through fpga_image_info, so we could add that to the struct.  Should
we call it port_id?  Or is there something more generic that may be
useful in the future for other architectures?.

pr_err appears to be machine specific error codes that are
communicated outside your low level driver.  (Calling it pr_err is
extra confusing since Linux already has a commonly name function by
the same name).  The framework has state, but that's not doing what
you want here.  Maybe we could add a framework ops called status so
that status could be communicated from the low level driver. It would
be useful to abstract the machine specific state to a defined enum
that would be part of the fpga mgr framework.  I remember we had
something like that in the earliest version of fpga manager but it got
changed to state rather than status for some reason.

mgr->dev won't work for you for some of the things you are using fme->pdev for?

Alan

> +
> +       /* Find and get port device by index */
> +       port = pdata->fpga_for_each_port(pdev, &fme->port_id,
> +                                        fpga_port_check_id);
> +       WARN_ON(!port);
> +
> +       /* Disable Port before PR */
> +       fpga_port_disable(port);
> +
> +       ret = fpga_mgr_buf_load(mgr, &info, buf, port_pr.buffer_size);
> +       port_pr.status = fme->pr_err;
> +
> +       /* Re-enable Port after PR finished */
> +       fpga_port_enable(port);
> +
> +       put_device(&port->dev);
> +
> +unlock_exit:
> +       mutex_unlock(&pdata->lock);
> +       fpga_mgr_put(mgr);
> +free_exit:
> +       vfree(buf);
> +       if (copy_to_user((void __user *)arg, &port_pr, minsz))
> +               return -EFAULT;
> +       return ret;
> +}
> +
> +static int fpga_fme_pr_probe(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct fpga_fme *priv;
> +       int ret;
> +
> +       mutex_lock(&pdata->lock);
> +       priv = fpga_pdata_get_private(pdata);
> +       ret = fpga_mgr_register(&pdata->dev->dev,
> +               "Intel FPGA Manager", &fme_pr_ops, priv);
> +       mutex_unlock(&pdata->lock);
> +
> +       return ret;
> +}
> +
> +static int fpga_fme_pr_remove(struct platform_device *pdev)
> +{
> +       fpga_mgr_unregister(&pdev->dev);
> +       return 0;
> +}
> +
> +static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
> +{
> +       int ret;
> +
> +       ret = fpga_fme_pr_probe(pdev);
> +       if (ret)
> +               return ret;
> +
> +       ret = sysfs_create_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
> +       if (ret)
> +               fpga_fme_pr_remove(pdev);
> +
> +       return ret;
> +}
> +
> +static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
> +{
> +       sysfs_remove_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
> +       fpga_fme_pr_remove(pdev);
> +}
> +
> +static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
> +       unsigned int cmd, unsigned long arg)
> +{
> +       long ret;
> +
> +       switch (cmd) {
> +       case FPGA_FME_PORT_PR:
> +               ret = fme_pr(pdev, arg);
> +               break;
> +       default:
> +               ret = -ENODEV;
> +       }
> +
> +       return ret;
> +}
> +
> +struct feature_ops pr_mgmt_ops = {
> +       .init = pr_mgmt_init,
> +       .uinit = pr_mgmt_uinit,
> +       .ioctl = fme_pr_ioctl,
> +};
> diff --git a/drivers/fpga/intel/fme.h b/drivers/fpga/intel/fme.h
> new file mode 100644
> index 0000000..d6cb7ce
> --- /dev/null
> +++ b/drivers/fpga/intel/fme.h
> @@ -0,0 +1,32 @@
> +/*
> + * Header file for Intel FPGA Management Engine (FME) Driver
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <luwei.kang@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *   Joseph Grecco <joe.grecco@intel.com>
> + *   Enno Luebbers <enno.luebbers@intel.com>
> + *   Tim Whisonant <tim.whisonant@intel.com>
> + *   Ananda Ravuri <ananda.ravuri@intel.com>
> + *   Henry Mitchel <henry.mitchel@intel.com>
> + *
> + * This work is licensed under a dual BSD/GPLv2 license. When using or
> + * redistributing this file, you may do so under either license. See the
> + * LICENSE.BSD file under this directory for the BSD license and see
> + * the COPYING file in the top-level directory for the GPLv2 license.
> + */
> +
> +#ifndef __INTEL_FME_H
> +#define __INTEL_FME_H
> +
> +struct fpga_fme {
> +       u8  port_id;
> +       u64 pr_err;
> +       struct feature_platform_data *pdata;
> +};
> +
> +extern struct feature_ops pr_mgmt_ops;
> +
> +#endif
> diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
> index 992e556..77658316 100644
> --- a/include/uapi/linux/intel-fpga.h
> +++ b/include/uapi/linux/intel-fpga.h
> @@ -18,6 +18,8 @@
>  #ifndef _UAPI_LINUX_INTEL_FPGA_H
>  #define _UAPI_LINUX_INTEL_FPGA_H
>
> +#include <linux/types.h>
> +
>  #define FPGA_API_VERSION 0
>
>  /*
> @@ -30,6 +32,7 @@
>  #define FPGA_MAGIC 0xB6
>
>  #define FPGA_BASE 0
> +#define FME_BASE 0x80
>
>  /**
>   * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
> @@ -49,4 +52,45 @@
>
>  #define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)
>
> +/* IOCTLs for FME file descriptor */
> +
> +/**
> + * FPGA_FME_PORT_PR - _IOWR(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
> + *
> + * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
> + * provided by caller.
> + * Return: 0 on success, -errno on failure.
> + * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
> + * some errors during PR, under this case, the user can fetch HW error code
> + * from fpga_fme_port_pr.status. Each bit on the error code is used as the
> + * index for the array created by DEFINE_FPGA_PR_ERR_MSG().
> + * Otherwise, it is always zero.
> + */
> +
> +#define DEFINE_FPGA_PR_ERR_MSG(_name_)                 \
> +static const char * const _name_[] = {                 \
> +       "PR operation error detected",                  \
> +       "PR CRC error detected",                        \
> +       "PR incompatiable bitstream error detected",    \
> +       "PR IP protocol error detected",                \
> +       "PR FIFO overflow error detected",              \
> +       "Reserved",                                     \
> +       "PR secure load error detected",                \
> +}
> +
> +#define PR_MAX_ERR_NUM 7
> +
> +struct fpga_fme_port_pr {
> +       /* Input */
> +       __u32 argsz;            /* Structure length */
> +       __u32 flags;            /* Zero for now */
> +       __u32 port_id;
> +       __u32 buffer_size;
> +       __u64 buffer_address;   /* Userspace address to the buffer for PR */
> +       /* Output */
> +       __u64 status;           /* HW error code if ioctl returns -EIO */
> +};
> +
> +#define FPGA_FME_PORT_PR       _IO(FPGA_MAGIC, FME_BASE + 0)
> +
>  #endif /* _UAPI_INTEL_FPGA_H */
> --
> 2.7.4
>

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-03-30 12:08 ` [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Wu Hao
  2017-03-30 20:30   ` Alan Tull
  2017-03-31 19:10   ` Alan Tull
@ 2017-03-31 23:45   ` kbuild test robot
  2017-04-01  1:12   ` kbuild test robot
  2017-04-03 21:24   ` Alan Tull
  4 siblings, 0 replies; 93+ messages in thread
From: kbuild test robot @ 2017-03-31 23:45 UTC (permalink / raw)
  To: Wu Hao
  Cc: kbuild-all, atull, moritz.fischer, linux-fpga, linux-kernel,
	luwei.kang, yi.z.zhang, hao.wu, Tim Whisonant, Enno Luebbers,
	Shiva Rao, Christopher Rauer, Alan Tull, Xiao Guangrong

[-- Attachment #1: Type: text/plain, Size: 6733 bytes --]

Hi Kang,

[auto build test WARNING on linus/master]
[also build test WARNING on v4.11-rc4 next-20170331]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Wu-Hao/Intel-FPGA-Device-Drivers/20170401-052017
config: frv-allmodconfig (attached as .config)
compiler: frv-linux-gcc (GCC) 6.2.0
reproduce:
        wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        make.cross ARCH=frv 

All warnings (new ones prefixed by >>):

   drivers/fpga/intel/fme-pr.c: In function 'interface_id_show':
   drivers/fpga/intel/fme-pr.c:45:15: error: implicit declaration of function 'readq' [-Werror=implicit-function-declaration]
     intfc_id_l = readq(&fme_pr->intfc_id_l);
                  ^~~~~
   drivers/fpga/intel/fme-pr.c: In function 'pr_err_handle':
   drivers/fpga/intel/fme-pr.c:79:2: error: implicit declaration of function 'writeq' [-Werror=implicit-function-declaration]
     writeq(fme_pr_error, &fme_pr->error);
     ^~~~~~
   In file included from include/linux/uaccess.h:5:0,
                    from drivers/fpga/intel/fme-pr.c:25:
   drivers/fpga/intel/fme-pr.c: In function 'fme_pr':
>> arch/frv/include/asm/uaccess.h:63:47: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
    #define access_ok(type,addr,size) (__range_ok((void __user *)(addr), (size)) == 0)
                                                  ^
   arch/frv/include/asm/uaccess.h:61:60: note: in definition of macro '__range_ok'
    #define __range_ok(addr,size) ___range_ok((unsigned long) (addr), (unsigned long) (size))
                                                               ^~~~
>> drivers/fpga/intel/fme-pr.c:278:7: note: in expansion of macro 'access_ok'
     if (!access_ok(VERIFY_READ, port_pr.buffer_address,
          ^~~~~~~~~
   drivers/fpga/intel/fme-pr.c:286:26: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
     if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
                             ^
   cc1: some warnings being treated as errors
--
   drivers/fpga//intel/fme-pr.c: In function 'interface_id_show':
   drivers/fpga//intel/fme-pr.c:45:15: error: implicit declaration of function 'readq' [-Werror=implicit-function-declaration]
     intfc_id_l = readq(&fme_pr->intfc_id_l);
                  ^~~~~
   drivers/fpga//intel/fme-pr.c: In function 'pr_err_handle':
   drivers/fpga//intel/fme-pr.c:79:2: error: implicit declaration of function 'writeq' [-Werror=implicit-function-declaration]
     writeq(fme_pr_error, &fme_pr->error);
     ^~~~~~
   In file included from include/linux/uaccess.h:5:0,
                    from drivers/fpga//intel/fme-pr.c:25:
   drivers/fpga//intel/fme-pr.c: In function 'fme_pr':
>> arch/frv/include/asm/uaccess.h:63:47: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
    #define access_ok(type,addr,size) (__range_ok((void __user *)(addr), (size)) == 0)
                                                  ^
   arch/frv/include/asm/uaccess.h:61:60: note: in definition of macro '__range_ok'
    #define __range_ok(addr,size) ___range_ok((unsigned long) (addr), (unsigned long) (size))
                                                               ^~~~
   drivers/fpga//intel/fme-pr.c:278:7: note: in expansion of macro 'access_ok'
     if (!access_ok(VERIFY_READ, port_pr.buffer_address,
          ^~~~~~~~~
   drivers/fpga//intel/fme-pr.c:286:26: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
     if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
                             ^
   cc1: some warnings being treated as errors

vim +63 arch/frv/include/asm/uaccess.h

^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  47  	return flag;
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  48  
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  49  #else
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  50  
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  51  	if (addr < memory_start ||
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  52  	    addr > memory_end ||
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  53  	    size > memory_end - memory_start ||
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  54  	    addr + size > memory_end)
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  55  		return -EFAULT;
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  56  
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  57  	return 0;
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  58  #endif
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  59  }
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  60  
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  61  #define __range_ok(addr,size) ___range_ok((unsigned long) (addr), (unsigned long) (size))
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  62  
a8a77573c9 include/asm-frv/uaccess.h Al Viro        2006-06-23 @63  #define access_ok(type,addr,size) (__range_ok((void __user *)(addr), (size)) == 0)
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  64  #define __access_ok(addr,size) (__range_ok((addr), (size)) == 0)
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  65  
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  66  /*
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  67   * The exception table consists of pairs of addresses: the first is the
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  68   * address of an instruction that is allowed to fault, and the second is
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  69   * the address at which the program should continue.  No registers are
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  70   * modified, so it is entirely up to the continuation code to figure out
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  71   * what to do.

:::::: The code at line 63 was first introduced by commit
:::::: a8a77573c9e5345bcf6a963858745cd83c923f44 [PATCH] frv: __user infrastructure

:::::: TO: Al Viro <viro@zeniv.linux.org.uk>
:::::: CC: Linus Torvalds <torvalds@g5.osdl.org>

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 47478 bytes --]

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 16/16] fpga: intel: afu: add FPGA_PORT_DMA_MAP/UNMAP ioctls support
  2017-03-30 12:08 ` [PATCH 16/16] fpga: intel: afu: add FPGA_PORT_DMA_MAP/UNMAP ioctls support Wu Hao
@ 2017-04-01  0:00   ` kbuild test robot
  2017-04-01  1:33   ` kbuild test robot
  1 sibling, 0 replies; 93+ messages in thread
From: kbuild test robot @ 2017-04-01  0:00 UTC (permalink / raw)
  To: Wu Hao
  Cc: kbuild-all, atull, moritz.fischer, linux-fpga, linux-kernel,
	luwei.kang, yi.z.zhang, hao.wu, Tim Whisonant, Enno Luebbers,
	Shiva Rao, Christopher Rauer, Xiao Guangrong

[-- Attachment #1: Type: text/plain, Size: 2478 bytes --]

Hi Wu,

[auto build test WARNING on linus/master]
[also build test WARNING on v4.11-rc4 next-20170331]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Wu-Hao/Intel-FPGA-Device-Drivers/20170401-052017
config: frv-allmodconfig (attached as .config)
compiler: frv-linux-gcc (GCC) 6.2.0
reproduce:
        wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        make.cross ARCH=frv 

All warnings (new ones prefixed by >>):

   In file included from include/linux/uaccess.h:5:0,
                    from drivers/fpga/intel/afu-dma-region.c:17:
   drivers/fpga/intel/afu-dma-region.c: In function 'afu_dma_map_region':
   arch/frv/include/asm/uaccess.h:63:47: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
    #define access_ok(type,addr,size) (__range_ok((void __user *)(addr), (size)) == 0)
                                                  ^
   arch/frv/include/asm/uaccess.h:61:60: note: in definition of macro '__range_ok'
    #define __range_ok(addr,size) ___range_ok((unsigned long) (addr), (unsigned long) (size))
                                                               ^~~~
>> drivers/fpga/intel/afu-dma-region.c:291:7: note: in expansion of macro 'access_ok'
     if (!access_ok(VERIFY_WRITE, user_addr, length))
          ^~~~~~~~~

vim +/access_ok +291 drivers/fpga/intel/afu-dma-region.c

   275			       u64 user_addr, u64 length, u64 *iova)
   276	{
   277		struct fpga_afu_dma_region *region;
   278		int ret;
   279	
   280		/*
   281		 * Check Inputs, only accept page-aligned user memory region with
   282		 * valid length.
   283		 */
   284		if (!PAGE_ALIGNED(user_addr) || !PAGE_ALIGNED(length) || !length)
   285			return -EINVAL;
   286	
   287		/* Check overflow */
   288		if (user_addr + length < user_addr)
   289			return -EINVAL;
   290	
 > 291		if (!access_ok(VERIFY_WRITE, user_addr, length))
   292			return -EINVAL;
   293	
   294		region = kzalloc(sizeof(*region), GFP_KERNEL);
   295		if (!region)
   296			return -ENOMEM;
   297	
   298		region->user_addr = user_addr;
   299		region->length = length;

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 47484 bytes --]

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-03-30 12:08 ` [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Wu Hao
                     ` (2 preceding siblings ...)
  2017-03-31 23:45   ` kbuild test robot
@ 2017-04-01  1:12   ` kbuild test robot
  2017-04-03 21:24   ` Alan Tull
  4 siblings, 0 replies; 93+ messages in thread
From: kbuild test robot @ 2017-04-01  1:12 UTC (permalink / raw)
  To: Wu Hao
  Cc: kbuild-all, atull, moritz.fischer, linux-fpga, linux-kernel,
	luwei.kang, yi.z.zhang, hao.wu, Tim Whisonant, Enno Luebbers,
	Shiva Rao, Christopher Rauer, Alan Tull, Xiao Guangrong

[-- Attachment #1: Type: text/plain, Size: 2457 bytes --]

Hi Kang,

[auto build test WARNING on linus/master]
[also build test WARNING on v4.11-rc4 next-20170331]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Wu-Hao/Intel-FPGA-Device-Drivers/20170401-052017
config: sparc-allyesconfig (attached as .config)
compiler: sparc64-linux-gnu-gcc (Debian 6.1.1-9) 6.1.1 20160705
reproduce:
        wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        make.cross ARCH=sparc 

All warnings (new ones prefixed by >>):

   drivers/fpga/intel/fme-pr.c: In function 'fme_pr':
>> drivers/fpga/intel/fme-pr.c:278:30: warning: passing argument 2 of 'access_ok' makes pointer from integer without a cast [-Wint-conversion]
     if (!access_ok(VERIFY_READ, port_pr.buffer_address,
                                 ^~~~~~~
   In file included from arch/sparc/include/asm/uaccess.h:4:0,
                    from include/linux/uaccess.h:5,
                    from drivers/fpga/intel/fme-pr.c:25:
   arch/sparc/include/asm/uaccess_64.h:80:19: note: expected 'const void *' but argument is of type '__u64 {aka long long unsigned int}'
    static inline int access_ok(int type, const void __user * addr, unsigned long size)
                      ^~~~~~~~~

vim +/access_ok +278 drivers/fpga/intel/fme-pr.c

   262		if (!IS_ALIGNED(port_pr.buffer_size, 4))
   263			return -EINVAL;
   264	
   265		/* get fme header region */
   266		fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
   267						FME_FEATURE_ID_HEADER);
   268		if (WARN_ON(!fme_hdr))
   269			return -EINVAL;
   270	
   271		/* check port id */
   272		fme_capability.csr = readq(&fme_hdr->capability);
   273		if (port_pr.port_id >= fme_capability.num_ports) {
   274			dev_dbg(&pdev->dev, "port number more than maximum\n");
   275			return -EINVAL;
   276		}
   277	
 > 278		if (!access_ok(VERIFY_READ, port_pr.buffer_address,
   279					    port_pr.buffer_size))
   280			return -EFAULT;
   281	
   282		buf = vmalloc(port_pr.buffer_size);
   283		if (!buf)
   284			return -ENOMEM;
   285	
   286		if (copy_from_user(buf, (void __user *)port_pr.buffer_address,

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 49798 bytes --]

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 16/16] fpga: intel: afu: add FPGA_PORT_DMA_MAP/UNMAP ioctls support
  2017-03-30 12:08 ` [PATCH 16/16] fpga: intel: afu: add FPGA_PORT_DMA_MAP/UNMAP ioctls support Wu Hao
  2017-04-01  0:00   ` kbuild test robot
@ 2017-04-01  1:33   ` kbuild test robot
  1 sibling, 0 replies; 93+ messages in thread
From: kbuild test robot @ 2017-04-01  1:33 UTC (permalink / raw)
  To: Wu Hao
  Cc: kbuild-all, atull, moritz.fischer, linux-fpga, linux-kernel,
	luwei.kang, yi.z.zhang, hao.wu, Tim Whisonant, Enno Luebbers,
	Shiva Rao, Christopher Rauer, Xiao Guangrong

[-- Attachment #1: Type: text/plain, Size: 2410 bytes --]

Hi Wu,

[auto build test WARNING on linus/master]
[also build test WARNING on v4.11-rc4 next-20170331]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Wu-Hao/Intel-FPGA-Device-Drivers/20170401-052017
config: sparc-allyesconfig (attached as .config)
compiler: sparc64-linux-gnu-gcc (Debian 6.1.1-9) 6.1.1 20160705
reproduce:
        wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        make.cross ARCH=sparc 

All warnings (new ones prefixed by >>):

   drivers/fpga/intel/afu-dma-region.c: In function 'afu_dma_map_region':
>> drivers/fpga/intel/afu-dma-region.c:291:31: warning: passing argument 2 of 'access_ok' makes pointer from integer without a cast [-Wint-conversion]
     if (!access_ok(VERIFY_WRITE, user_addr, length))
                                  ^~~~~~~~~
   In file included from arch/sparc/include/asm/uaccess.h:4:0,
                    from include/linux/uaccess.h:5,
                    from drivers/fpga/intel/afu-dma-region.c:17:
   arch/sparc/include/asm/uaccess_64.h:80:19: note: expected 'const void *' but argument is of type 'u64 {aka long long unsigned int}'
    static inline int access_ok(int type, const void __user * addr, unsigned long size)
                      ^~~~~~~~~

vim +/access_ok +291 drivers/fpga/intel/afu-dma-region.c

   275			       u64 user_addr, u64 length, u64 *iova)
   276	{
   277		struct fpga_afu_dma_region *region;
   278		int ret;
   279	
   280		/*
   281		 * Check Inputs, only accept page-aligned user memory region with
   282		 * valid length.
   283		 */
   284		if (!PAGE_ALIGNED(user_addr) || !PAGE_ALIGNED(length) || !length)
   285			return -EINVAL;
   286	
   287		/* Check overflow */
   288		if (user_addr + length < user_addr)
   289			return -EINVAL;
   290	
 > 291		if (!access_ok(VERIFY_WRITE, user_addr, length))
   292			return -EINVAL;
   293	
   294		region = kzalloc(sizeof(*region), GFP_KERNEL);
   295		if (!region)
   296			return -ENOMEM;
   297	
   298		region->user_addr = user_addr;
   299		region->length = length;

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 49802 bytes --]

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-03-31 19:10   ` Alan Tull
@ 2017-04-01 11:08     ` Wu Hao
  2017-04-03 16:30       ` Alan Tull
  0 siblings, 1 reply; 93+ messages in thread
From: Wu Hao @ 2017-04-01 11:08 UTC (permalink / raw)
  To: Alan Tull
  Cc: Moritz Fischer, linux-fpga, linux-kernel, luwei.kang, yi.z.zhang,
	Tim Whisonant, Enno Luebbers, Shiva Rao, Christopher Rauer,
	Alan Tull, Xiao Guangrong

On Fri, Mar 31, 2017 at 02:10:12PM -0500, Alan Tull wrote:
> On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote:
> > From: Kang Luwei <luwei.kang@intel.com>
> >
> > Partial Reconfiguration (PR) is the most important function for FME. It
> > allows reconfiguration for given Port/Accelerated Function Unit (AFU).
> >
> > This patch adds support for PR sub feature. In this patch, it registers
> > a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
> > for PR operation once PR request received via ioctl. Below user space
> > interfaces are exposed by this sub feature.
> >
> > Sysfs interface:
> > * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
> >   Read-only. Indicate the hardware interface information. Userspace
> >   applications need to check this interface to select correct green
> >   bitstream format before PR.
> >
> > Ioctl interface:
> > * FPGA_FME_PORT_PR
> >   Do partial reconfiguration per information from userspace, including
> >   target port(AFU), buffer size and address info. It returns the PR status
> >   (PR error code if failed) to userspace.
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > Signed-off-by: Alan Tull <alan.tull@intel.com>
> > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > ---
> >  drivers/fpga/intel/Makefile      |   2 +-
> >  drivers/fpga/intel/feature-dev.h |  58 ++++++
> >  drivers/fpga/intel/fme-main.c    |  44 ++++-
> >  drivers/fpga/intel/fme-pr.c      | 400 +++++++++++++++++++++++++++++++++++++++
> >  drivers/fpga/intel/fme.h         |  32 ++++
> >  include/uapi/linux/intel-fpga.h  |  44 +++++
> >  6 files changed, 578 insertions(+), 2 deletions(-)
> >  create mode 100644 drivers/fpga/intel/fme-pr.c
> >  create mode 100644 drivers/fpga/intel/fme.h
> >
> > diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
> > index 546861d..0452cb6 100644
> > --- a/drivers/fpga/intel/Makefile
> > +++ b/drivers/fpga/intel/Makefile
> > @@ -2,4 +2,4 @@ obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
> >  obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
> >
> >  intel-fpga-pci-objs := pcie.o feature-dev.o
> > -intel-fpga-fme-objs := fme-main.o
> > +intel-fpga-fme-objs := fme-main.o fme-pr.o
> > diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
> > index dccc283..5a25c915 100644
> > --- a/drivers/fpga/intel/feature-dev.h
> > +++ b/drivers/fpga/intel/feature-dev.h
> > @@ -150,8 +150,66 @@ struct feature_fme_err {
> >  };
> >
> >  /* FME Partial Reconfiguration Sub Feature Register Set */
> > +/* FME PR Control Register */
> > +struct feature_fme_pr_ctl {
> > +       union {
> > +               u64 csr;
> > +               struct {
> > +                       u8  pr_reset:1;         /* Reset PR Engine */
> > +                       u8  rsvdz1:3;
> > +                       u8  pr_reset_ack:1;     /* Reset PR Engine Ack */
> > +                       u8  rsvdz2:3;
> > +                       u8  pr_regionid:2;      /* PR Region ID */
> > +                       u8  rsvdz3:2;
> > +                       u8  pr_start_req:1;     /* PR Start Request */
> > +                       u8  pr_push_complete:1; /* PR Data push complete */
> > +                       u8  pr_kind:1;          /* Load Customer or Intel GBS */
> > +                       u32 rsvdz4:17;
> > +                       u32 config_data;
> > +               };
> > +       };
> > +};
> > +
> > +/* FME PR Status Register */
> > +struct feature_fme_pr_status {
> > +       union {
> > +               u64 csr;
> > +               struct {
> > +                       u16 pr_credit:9;        /* Number of PR Credits */
> > +                       u8  rsvdz1:7;
> > +                       u8  pr_status:1;        /* PR Operation status */
> > +                       u8  rsvdz2:3;
> > +                       u8  pr_ctrlr_status:3;  /* Controller status */
> > +                       u8  rsvdz3:1;
> > +                       u8  pr_host_status:4;   /* PR Host status */
> > +                       u64 rsvdz4:36;
> > +               };
> > +       };
> > +};
> > +
> > +/* FME PR Data Register */
> > +struct feature_fme_pr_data {
> > +       union {
> > +               u64 csr;
> > +               struct {
> > +                       /* PR data from the raw-binary file */
> > +                       u32 pr_data_raw;
> > +                       u32 rsvd;
> > +               };
> > +       };
> > +};
> > +
> >  struct feature_fme_pr {
> >         struct feature_header header;
> > +       struct feature_fme_pr_ctl control;
> > +       struct feature_fme_pr_status status;
> > +       struct feature_fme_pr_data data;
> > +       u64 error;
> > +
> > +       u64 rsvd[16];
> > +
> > +       u64 intfc_id_l;         /* PR interface Id Low */
> > +       u64 intfc_id_h;         /* PR interface Id High */
> >  };
> >
> >  /* PORT Header Register Set */
> > diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
> > index 36d0c4c..0d9a7a6 100644
> > --- a/drivers/fpga/intel/fme-main.c
> > +++ b/drivers/fpga/intel/fme-main.c
> > @@ -23,6 +23,7 @@
> >  #include <linux/intel-fpga.h>
> >
> >  #include "feature-dev.h"
> > +#include "fme.h"
> >
> >  static ssize_t ports_num_show(struct device *dev,
> >                               struct device_attribute *attr, char *buf)
> > @@ -101,6 +102,10 @@ static struct feature_driver fme_feature_drvs[] = {
> >                 .ops = &fme_hdr_ops,
> >         },
> >         {
> > +               .name = FME_FEATURE_PR_MGMT,
> > +               .ops = &pr_mgmt_ops,
> > +       },
> > +       {
> >                 .ops = NULL,
> >         },
> >  };
> > @@ -182,14 +187,48 @@ static const struct file_operations fme_fops = {
> >         .unlocked_ioctl = fme_ioctl,
> >  };
> >
> > +static int fme_dev_init(struct platform_device *pdev)
> > +{
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct fpga_fme *fme;
> > +
> > +       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
> > +       if (!fme)
> > +               return -ENOMEM;
> > +
> > +       fme->pdata = pdata;
> > +
> > +       mutex_lock(&pdata->lock);
> > +       fpga_pdata_set_private(pdata, fme);
> > +       mutex_unlock(&pdata->lock);
> > +       return 0;
> > +}
> > +
> > +static void fme_dev_destroy(struct platform_device *pdev)
> > +{
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct fpga_fme *fme;
> > +
> > +       mutex_lock(&pdata->lock);
> > +       fme = fpga_pdata_get_private(pdata);
> > +       fpga_pdata_set_private(pdata, NULL);
> > +       mutex_unlock(&pdata->lock);
> > +
> > +       devm_kfree(&pdev->dev, fme);
> > +}
> > +
> >  static int fme_probe(struct platform_device *pdev)
> >  {
> >         int ret;
> >
> > -       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> > +       ret = fme_dev_init(pdev);
> >         if (ret)
> >                 goto exit;
> >
> > +       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> > +       if (ret)
> > +               goto dev_destroy;
> > +
> >         ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
> >         if (ret)
> >                 goto feature_uinit;
> > @@ -198,6 +237,8 @@ static int fme_probe(struct platform_device *pdev)
> >
> >  feature_uinit:
> >         fpga_dev_feature_uinit(pdev);
> > +dev_destroy:
> > +       fme_dev_destroy(pdev);
> >  exit:
> >         return ret;
> >  }
> > @@ -206,6 +247,7 @@ static int fme_remove(struct platform_device *pdev)
> >  {
> >         fpga_dev_feature_uinit(pdev);
> >         fpga_unregister_dev_ops(pdev);
> > +       fme_dev_destroy(pdev);
> >         return 0;
> >  }
> >
> > diff --git a/drivers/fpga/intel/fme-pr.c b/drivers/fpga/intel/fme-pr.c
> > new file mode 100644
> > index 0000000..3b44a3e
> > --- /dev/null
> > +++ b/drivers/fpga/intel/fme-pr.c
> > @@ -0,0 +1,400 @@
> > +/*
> > + * Driver for Intel FPGA Management Engine (FME) Partial Reconfiguration
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <luwei.kang@intel.com>
> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > + *   Joseph Grecco <joe.grecco@intel.com>
> > + *   Enno Luebbers <enno.luebbers@intel.com>
> > + *   Tim Whisonant <tim.whisonant@intel.com>
> > + *   Ananda Ravuri <ananda.ravuri@intel.com>
> > + *   Christopher Rauer <christopher.rauer@intel.com>
> > + *   Henry Mitchel <henry.mitchel@intel.com>
> > + *
> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
> > + * redistributing this file, you may do so under either license. See the
> > + * LICENSE.BSD file under this directory for the BSD license and see
> > + * the COPYING file in the top-level directory for the GPLv2 license.
> > + */
> > +
> > +#include <linux/types.h>
> > +#include <linux/device.h>
> > +#include <linux/vmalloc.h>
> > +#include <linux/uaccess.h>
> > +#include <linux/fpga/fpga-mgr.h>
> > +#include <linux/intel-fpga.h>
> > +
> > +#include "feature-dev.h"
> > +#include "fme.h"
> > +
> > +#define PR_WAIT_TIMEOUT   8000000
> > +
> > +#define PR_HOST_STATUS_IDLE    0
> > +
> > +DEFINE_FPGA_PR_ERR_MSG(pr_err_msg);
> > +
> > +static ssize_t interface_id_show(struct device *dev,
> > +                                struct device_attribute *attr, char *buf)
> > +{
> > +       u64 intfc_id_l, intfc_id_h;
> > +       struct feature_fme_pr *fme_pr
> > +               = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_PR_MGMT);
> > +
> > +       intfc_id_l = readq(&fme_pr->intfc_id_l);
> > +       intfc_id_h = readq(&fme_pr->intfc_id_h);
> > +
> > +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> > +                       (unsigned long long)intfc_id_h,
> > +                       (unsigned long long)intfc_id_l);
> > +}
> > +static DEVICE_ATTR_RO(interface_id);
> > +
> > +static struct attribute *pr_mgmt_attrs[] = {
> > +       &dev_attr_interface_id.attr,
> > +       NULL,
> > +};
> > +
> > +struct attribute_group pr_mgmt_attr_group = {
> > +       .attrs  = pr_mgmt_attrs,
> > +       .name   = "pr",
> > +};
> > +
> > +static u64
> > +pr_err_handle(struct platform_device *pdev, struct feature_fme_pr *fme_pr)
> > +{
> > +       struct feature_fme_pr_status fme_pr_status;
> > +       unsigned long err_code;
> > +       u64 fme_pr_error;
> > +       int i = 0;
> > +
> > +       fme_pr_status.csr = readq(&fme_pr->status);
> > +       if (!fme_pr_status.pr_status)
> > +               return 0;
> > +
> > +       err_code = fme_pr_error = readq(&fme_pr->error);
> > +       for_each_set_bit(i, &err_code, PR_MAX_ERR_NUM)
> > +               dev_dbg(&pdev->dev, "%s\n", pr_err_msg[i]);
> > +       writeq(fme_pr_error, &fme_pr->error);
> > +       return fme_pr_error;
> > +}
> > +
> > +static int fme_pr_write_init(struct fpga_manager *mgr,
> > +               struct fpga_image_info *info, const char *buf, size_t count)
> > +{
> > +       struct fpga_fme *fme = mgr->priv;
> > +       struct platform_device *pdev;
> > +       struct feature_fme_pr *fme_pr;
> > +       struct feature_fme_pr_ctl fme_pr_ctl;
> > +       struct feature_fme_pr_status fme_pr_status;
> > +
> > +       pdev = fme->pdata->dev;
> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                               FME_FEATURE_ID_PR_MGMT);
> > +       if (!fme_pr)
> > +               return -EINVAL;
> > +
> > +       if (WARN_ON(info->flags != FPGA_MGR_PARTIAL_RECONFIG))
> > +               return -EINVAL;
> > +
> > +       dev_dbg(&pdev->dev, "resetting PR before initiated PR\n");
> > +
> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> > +       fme_pr_ctl.pr_reset = 1;
> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> > +
> > +       fme_pr_ctl.pr_reset_ack = 1;
> > +
> > +       if (fpga_wait_register_field(pr_reset_ack, fme_pr_ctl,
> > +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> > +               dev_err(&pdev->dev, "maximum PR timeout\n");
> > +               return -ETIMEDOUT;
> > +       }
> > +
> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> > +       fme_pr_ctl.pr_reset = 0;
> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> > +
> > +       dev_dbg(&pdev->dev,
> > +               "waiting for PR resource in HW to be initialized and ready\n");
> > +
> > +       fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
> > +
> > +       if (fpga_wait_register_field(pr_host_status, fme_pr_status,
> > +                                    &fme_pr->status, PR_WAIT_TIMEOUT, 1)) {
> > +               dev_err(&pdev->dev, "maximum PR timeout\n");
> > +               return -ETIMEDOUT;
> > +       }
> > +
> > +       dev_dbg(&pdev->dev, "check if have any previous PR error\n");
> > +       pr_err_handle(pdev, fme_pr);
> > +       return 0;
> > +}
> > +
> > +static int fme_pr_write(struct fpga_manager *mgr,
> > +                       const char *buf, size_t count)
> > +{
> > +       struct fpga_fme *fme = mgr->priv;
> > +       struct platform_device *pdev;
> > +       struct feature_fme_pr *fme_pr;
> > +       struct feature_fme_pr_ctl fme_pr_ctl;
> > +       struct feature_fme_pr_status fme_pr_status;
> > +       struct feature_fme_pr_data fme_pr_data;
> > +       int delay, pr_credit, i = 0;
> > +
> > +       pdev = fme->pdata->dev;
> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                               FME_FEATURE_ID_PR_MGMT);
> > +
> > +       dev_dbg(&pdev->dev, "set PR port ID and start request\n");
> > +
> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> > +       fme_pr_ctl.pr_regionid = fme->port_id;
> > +       fme_pr_ctl.pr_start_req = 1;
> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> > +
> > +       dev_dbg(&pdev->dev, "pushing data from bitstream to HW\n");
> > +
> > +       fme_pr_status.csr = readq(&fme_pr->status);
> > +       pr_credit = fme_pr_status.pr_credit;
> > +
> > +       while (count > 0) {
> > +               delay = 0;
> > +               while (pr_credit <= 1) {
> > +                       if (delay++ > PR_WAIT_TIMEOUT) {
> > +                               dev_err(&pdev->dev, "maximum try\n");
> > +                               return -ETIMEDOUT;
> > +                       }
> > +                       udelay(1);
> > +
> > +                       fme_pr_status.csr = readq(&fme_pr->status);
> > +                       pr_credit = fme_pr_status.pr_credit;
> > +               };
> > +
> > +               if (count >= 4) {
> > +                       fme_pr_data.rsvd = 0;
> > +                       fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
> > +                       writeq(fme_pr_data.csr, &fme_pr->data);
> > +                       count -= 4;
> > +                       pr_credit--;
> > +                       i++;
> > +               } else {
> > +                       WARN_ON(1);
> > +                       return -EINVAL;
> > +               }
> > +       }
> > +
> > +       return 0;
> > +}
> > +
> > +static int fme_pr_write_complete(struct fpga_manager *mgr,
> > +                       struct fpga_image_info *info)
> > +{
> > +       struct fpga_fme *fme = mgr->priv;
> > +       struct platform_device *pdev;
> > +       struct feature_fme_pr *fme_pr;
> > +       struct feature_fme_pr_ctl fme_pr_ctl;
> > +
> > +       pdev = fme->pdata->dev;
> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                               FME_FEATURE_ID_PR_MGMT);
> > +
> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> > +       fme_pr_ctl.pr_push_complete = 1;
> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> > +
> > +       dev_dbg(&pdev->dev, "green bitstream push complete\n");
> > +       dev_dbg(&pdev->dev, "waiting for HW to release PR resource\n");
> > +
> > +       fme_pr_ctl.pr_start_req = 0;
> > +
> > +       if (fpga_wait_register_field(pr_start_req, fme_pr_ctl,
> > +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> > +               dev_err(&pdev->dev, "maximum try.\n");
> > +               return -ETIMEDOUT;
> > +       }
> > +
> > +       dev_dbg(&pdev->dev, "PR operation complete, checking status\n");
> > +       fme->pr_err = pr_err_handle(pdev, fme_pr);
> > +       if (fme->pr_err)
> > +               return -EIO;
> > +
> > +       dev_dbg(&pdev->dev, "PR done successfully\n");
> > +       return 0;
> > +}
> > +
> > +static enum fpga_mgr_states fme_pr_state(struct fpga_manager *mgr)
> > +{
> > +       return FPGA_MGR_STATE_UNKNOWN;
> > +}
> > +
> > +static const struct fpga_manager_ops fme_pr_ops = {
> > +       .write_init = fme_pr_write_init,
> > +       .write = fme_pr_write,
> > +       .write_complete = fme_pr_write_complete,
> > +       .state = fme_pr_state,
> > +};
> > +
> > +static int fme_pr(struct platform_device *pdev, unsigned long arg)
> > +{
> > +       void __user *argp = (void __user *)arg;
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct fpga_fme *fme;
> > +       struct fpga_manager *mgr;
> > +       struct feature_fme_header *fme_hdr;
> > +       struct feature_fme_capability fme_capability;
> > +       struct fpga_image_info info;
> > +       struct fpga_fme_port_pr port_pr;
> > +       struct platform_device *port;
> > +       unsigned long minsz;
> > +       void *buf = NULL;
> > +       int ret = 0;
> > +
> > +       minsz = offsetofend(struct fpga_fme_port_pr, status);
> > +
> > +       if (copy_from_user(&port_pr, argp, minsz))
> > +               return -EFAULT;
> > +
> > +       if (port_pr.argsz < minsz || port_pr.flags)
> > +               return -EINVAL;
> > +
> > +       if (!IS_ALIGNED(port_pr.buffer_size, 4))
> > +               return -EINVAL;
> > +
> > +       /* get fme header region */
> > +       fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                                       FME_FEATURE_ID_HEADER);
> > +       if (WARN_ON(!fme_hdr))
> > +               return -EINVAL;
> > +
> > +       /* check port id */
> > +       fme_capability.csr = readq(&fme_hdr->capability);
> > +       if (port_pr.port_id >= fme_capability.num_ports) {
> > +               dev_dbg(&pdev->dev, "port number more than maximum\n");
> > +               return -EINVAL;
> > +       }
> > +
> > +       if (!access_ok(VERIFY_READ, port_pr.buffer_address,
> > +                                   port_pr.buffer_size))
> > +               return -EFAULT;
> > +
> > +       buf = vmalloc(port_pr.buffer_size);
> > +       if (!buf)
> > +               return -ENOMEM;
> > +
> > +       if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
> > +                                              port_pr.buffer_size)) {
> > +               ret = -EFAULT;
> > +               goto free_exit;
> > +       }
> > +
> > +       memset(&info, 0, sizeof(struct fpga_image_info));
> > +       info.flags = FPGA_MGR_PARTIAL_RECONFIG;
> > +
> > +       mgr = fpga_mgr_get(&pdev->dev);
> > +       if (IS_ERR(mgr)) {
> > +               ret = PTR_ERR(mgr);
> > +               goto free_exit;
> > +       }
> > +
> > +       mutex_lock(&pdata->lock);
> > +       fme = fpga_pdata_get_private(pdata);
> > +       /* fme device has been unregistered. */
> > +       if (!fme) {
> > +               ret = -EINVAL;
> > +               goto unlock_exit;
> > +       }
> > +
> > +       fme->pr_err = 0;
> > +       fme->port_id = port_pr.port_id;
> 
> It looks like you're using private data to communicate with the
> driver, i.e. there is something you want to do with the fpga manager
> framework and it doesn't have that feature.  The better way would be
> for us to expand the framework so you don't need to do that.
> 
> port_id is the kind of thing that should be communicated to the driver
> through fpga_image_info, so we could add that to the struct.  Should
> we call it port_id?  Or is there something more generic that may be
> useful in the future for other architectures?.

Hi Alan

Thanks for your feedback. :)

As you know, each Intel FPGA device may have more than one accelerator,
and all accelerators share the same fpga_manager (only one FME module).
port_id = 0 means the first accelerator on this fpga devices. So it's
needed by the fpga_manager to distinguish one accelerator from others
for partial reconfiguration.

Adding a member like a 'id' to fpga_image_info definitely works for us
in this case. We can add it this way, but I feel 'id' here seems not
related to fpga image, but characteristic of fpga region. It may be a
little confusing. One rough idea is that keep this info under fpga region
(maybe its private data), and pass the fpga-region to fpga_mgr_buf_load,
then fpga_manager knows the target region for partial reconfiguration.
If consider pr sysfs interface support under fpga-region in the future,
then we don't need to create a new 'id' sysfs file, as fpga-region itself
could provide this kind of info. But I'm not sure if this is the right
direction.

> 
> pr_err appears to be machine specific error codes that are
> communicated outside your low level driver.  (Calling it pr_err is
> extra confusing since Linux already has a commonly name function by
> the same name).  The framework has state, but that's not doing what
> you want here.  Maybe we could add a framework ops called status so
> that status could be communicated from the low level driver. It would
> be useful to abstract the machine specific state to a defined enum
> that would be part of the fpga mgr framework.  I remember we had
> something like that in the earliest version of fpga manager but it got
> changed to state rather than status for some reason.
> 

Adding a status function and a 'status' member to fpga manager sounds good.
But I'm not sure how to abstract these error codes, currently what we have
are all PR error codes for Intel FPGA device, e.g "PR FIFO overflow error",
"PR secure load error" and etc (see include/uapi/linux/intel-fpga.h). Is it
fine to let each driver to define how to use that 'status' for machine
specific status?

> mgr->dev won't work for you for some of the things you are using fme->pdev for?
>

Does 'fme->pdev' mean the platform device for fme? I think we use platform
device to represent the FME module, and FME module may have multiple sub
features, driver discovers these sub features via the 'Device Feature List'
in the PCIe Device Bar. PR is only one of the sub features under FME module,
if any FME module doesn't have PR sub feature, fpga-manager will not be 
created at all. But this will not impact other sub features, e.g thermal
management, error reporting and etc, to create their own interfaces under
the platform device.

Thanks
Hao
 
> Alan
> 
> > +
> > +       /* Find and get port device by index */
> > +       port = pdata->fpga_for_each_port(pdev, &fme->port_id,
> > +                                        fpga_port_check_id);
> > +       WARN_ON(!port);
> > +
> > +       /* Disable Port before PR */
> > +       fpga_port_disable(port);
> > +
> > +       ret = fpga_mgr_buf_load(mgr, &info, buf, port_pr.buffer_size);
> > +       port_pr.status = fme->pr_err;
> > +
> > +       /* Re-enable Port after PR finished */
> > +       fpga_port_enable(port);
> > +
> > +       put_device(&port->dev);
> > +
> > +unlock_exit:
> > +       mutex_unlock(&pdata->lock);
> > +       fpga_mgr_put(mgr);
> > +free_exit:
> > +       vfree(buf);
> > +       if (copy_to_user((void __user *)arg, &port_pr, minsz))
> > +               return -EFAULT;
> > +       return ret;
> > +}
> > +
> > +static int fpga_fme_pr_probe(struct platform_device *pdev)
> > +{
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct fpga_fme *priv;
> > +       int ret;
> > +
> > +       mutex_lock(&pdata->lock);
> > +       priv = fpga_pdata_get_private(pdata);
> > +       ret = fpga_mgr_register(&pdata->dev->dev,
> > +               "Intel FPGA Manager", &fme_pr_ops, priv);
> > +       mutex_unlock(&pdata->lock);
> > +
> > +       return ret;
> > +}
> > +
> > +static int fpga_fme_pr_remove(struct platform_device *pdev)
> > +{
> > +       fpga_mgr_unregister(&pdev->dev);
> > +       return 0;
> > +}
> > +
> > +static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
> > +{
> > +       int ret;
> > +
> > +       ret = fpga_fme_pr_probe(pdev);
> > +       if (ret)
> > +               return ret;
> > +
> > +       ret = sysfs_create_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
> > +       if (ret)
> > +               fpga_fme_pr_remove(pdev);
> > +
> > +       return ret;
> > +}
> > +
> > +static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
> > +{
> > +       sysfs_remove_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
> > +       fpga_fme_pr_remove(pdev);
> > +}
> > +
> > +static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
> > +       unsigned int cmd, unsigned long arg)
> > +{
> > +       long ret;
> > +
> > +       switch (cmd) {
> > +       case FPGA_FME_PORT_PR:
> > +               ret = fme_pr(pdev, arg);
> > +               break;
> > +       default:
> > +               ret = -ENODEV;
> > +       }
> > +
> > +       return ret;
> > +}
> > +
> > +struct feature_ops pr_mgmt_ops = {
> > +       .init = pr_mgmt_init,
> > +       .uinit = pr_mgmt_uinit,
> > +       .ioctl = fme_pr_ioctl,
> > +};
> > diff --git a/drivers/fpga/intel/fme.h b/drivers/fpga/intel/fme.h
> > new file mode 100644
> > index 0000000..d6cb7ce
> > --- /dev/null
> > +++ b/drivers/fpga/intel/fme.h
> > @@ -0,0 +1,32 @@
> > +/*
> > + * Header file for Intel FPGA Management Engine (FME) Driver
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <luwei.kang@intel.com>
> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > + *   Joseph Grecco <joe.grecco@intel.com>
> > + *   Enno Luebbers <enno.luebbers@intel.com>
> > + *   Tim Whisonant <tim.whisonant@intel.com>
> > + *   Ananda Ravuri <ananda.ravuri@intel.com>
> > + *   Henry Mitchel <henry.mitchel@intel.com>
> > + *
> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
> > + * redistributing this file, you may do so under either license. See the
> > + * LICENSE.BSD file under this directory for the BSD license and see
> > + * the COPYING file in the top-level directory for the GPLv2 license.
> > + */
> > +
> > +#ifndef __INTEL_FME_H
> > +#define __INTEL_FME_H
> > +
> > +struct fpga_fme {
> > +       u8  port_id;
> > +       u64 pr_err;
> > +       struct feature_platform_data *pdata;
> > +};
> > +
> > +extern struct feature_ops pr_mgmt_ops;
> > +
> > +#endif
> > diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
> > index 992e556..77658316 100644
> > --- a/include/uapi/linux/intel-fpga.h
> > +++ b/include/uapi/linux/intel-fpga.h
> > @@ -18,6 +18,8 @@
> >  #ifndef _UAPI_LINUX_INTEL_FPGA_H
> >  #define _UAPI_LINUX_INTEL_FPGA_H
> >
> > +#include <linux/types.h>
> > +
> >  #define FPGA_API_VERSION 0
> >
> >  /*
> > @@ -30,6 +32,7 @@
> >  #define FPGA_MAGIC 0xB6
> >
> >  #define FPGA_BASE 0
> > +#define FME_BASE 0x80
> >
> >  /**
> >   * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
> > @@ -49,4 +52,45 @@
> >
> >  #define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)
> >
> > +/* IOCTLs for FME file descriptor */
> > +
> > +/**
> > + * FPGA_FME_PORT_PR - _IOWR(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
> > + *
> > + * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
> > + * provided by caller.
> > + * Return: 0 on success, -errno on failure.
> > + * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
> > + * some errors during PR, under this case, the user can fetch HW error code
> > + * from fpga_fme_port_pr.status. Each bit on the error code is used as the
> > + * index for the array created by DEFINE_FPGA_PR_ERR_MSG().
> > + * Otherwise, it is always zero.
> > + */
> > +
> > +#define DEFINE_FPGA_PR_ERR_MSG(_name_)                 \
> > +static const char * const _name_[] = {                 \
> > +       "PR operation error detected",                  \
> > +       "PR CRC error detected",                        \
> > +       "PR incompatiable bitstream error detected",    \
> > +       "PR IP protocol error detected",                \
> > +       "PR FIFO overflow error detected",              \
> > +       "Reserved",                                     \
> > +       "PR secure load error detected",                \
> > +}
> > +
> > +#define PR_MAX_ERR_NUM 7
> > +
> > +struct fpga_fme_port_pr {
> > +       /* Input */
> > +       __u32 argsz;            /* Structure length */
> > +       __u32 flags;            /* Zero for now */
> > +       __u32 port_id;
> > +       __u32 buffer_size;
> > +       __u64 buffer_address;   /* Userspace address to the buffer for PR */
> > +       /* Output */
> > +       __u64 status;           /* HW error code if ioctl returns -EIO */
> > +};
> > +
> > +#define FPGA_FME_PORT_PR       _IO(FPGA_MAGIC, FME_BASE + 0)
> > +
> >  #endif /* _UAPI_INTEL_FPGA_H */
> > --
> > 2.7.4
> >

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview
  2017-03-31 18:38     ` Alan Tull
@ 2017-04-01 11:16       ` Wu Hao
  2017-04-02 14:41         ` Moritz Fischer
  0 siblings, 1 reply; 93+ messages in thread
From: Wu Hao @ 2017-04-01 11:16 UTC (permalink / raw)
  To: Alan Tull
  Cc: matthew.gerlach, Moritz Fischer, linux-fpga, linux-kernel,
	luwei.kang, yi.z.zhang, Enno Luebbers, Xiao Guangrong

On Fri, Mar 31, 2017 at 01:38:06PM -0500, Alan Tull wrote:
> On Fri, Mar 31, 2017 at 1:24 PM,  <matthew.gerlach@linux.intel.com> wrote:
> >
> >
> > On Thu, 30 Mar 2017, Wu Hao wrote:
> >
> >
> > Hi Wu Hao,
> >
> > Great documentation. I'm looking forward to diving into the rest of the
> > patches. Please see my comments inline.
> >
> > Matthew Gerlach
> >
> >
> >> Add a document for Intel FPGA driver overview.
> >>
> >> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> >> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> >> Signed-off-by: Wu Hao <hao.wu@intel.com>
> >> ---
> >> Documentation/fpga/intel-fpga.txt | 259
> >> ++++++++++++++++++++++++++++++++++++++
> >> 1 file changed, 259 insertions(+)
> >> create mode 100644 Documentation/fpga/intel-fpga.txt
> >>
> >> diff --git a/Documentation/fpga/intel-fpga.txt
> >> b/Documentation/fpga/intel-fpga.txt
> >> new file mode 100644
> >> index 0000000..9396cea
> >> --- /dev/null
> >> +++ b/Documentation/fpga/intel-fpga.txt
> >> @@ -0,0 +1,259 @@
> >>
> >> +===============================================================================
> >> +                    Intel FPGA driver Overview
> >>
> >> +-------------------------------------------------------------------------------
> >> +                Enno Luebbers <enno.luebbers@intel.com>
> >> +                Xiao Guangrong <guangrong.xiao@linux.intel.com>
> >> +                Wu Hao <hao.wu@intel.com>
> >> +
> >> +The Intel FPGA driver provides interfaces for userspace applications to
> >> +configure, enumerate, open, and access FPGA accelerators on platforms
> >> equipped
> >> +with Intel(R) FPGA solutions and enables system level management
> >> functions such
> >> +as FPGA reconfiguration, power management, and virtualization.
> >> +
> >
> >
> > From a Linux kernel perspective, I'm not sure this is the best name for
> > this code.  The name gives me the impression that it is a driver for all
> > Intel FPGAs, but not all Intel FPGAs are connected to the processor over a
> > PCIe bus.  The processor could be directely connected like the Arria10
> > SOCFPGA.  Such a processor could certainly benefit from this accelerator
> > usage model.  In an extreme case, couldn't a processor in the FPGA,
> > running Linux, also benefit from this accelerator model?  Is this code a
> > "FPGA Accelerator Framework"?
> >
> >> +HW Architecture
> >> +===============
> >> +From the OS's point of view, the FPGA hardware appears as a regular PCIe
> >> device.
> >> +The FPGA device memory is organized using a predefined data structure
> >> (Device
> >> +Feature List). Features supported by the particular FPGA device are
> >> exposed
> >> +through these data structures, as illustrated below:
> >> +
> >> +  +-------------------------------+  +-------------+
> >> +  |              PF               |  |     VF      |
> >> +  +-------------------------------+  +-------------+
> >> +      ^            ^         ^              ^
> >> +      |            |         |              |
> >> ++-----|------------|---------|--------------|-------+
> >> +|     |            |         |              |       |
> >> +|  +-----+     +-------+ +-------+      +-------+   |
> >> +|  | FME |     | Port0 | | Port1 |      | Port2 |   |
> >> +|  +-----+     +-------+ +-------+      +-------+   |
> >> +|                  ^         ^              ^       |
> >> +|                  |         |              |       |
> >> +|              +-------+ +------+       +-------+   |
> >> +|              |  AFU  | |  AFU |       |  AFU  |   |
> >> +|              +-------+ +------+       +-------+   |
> >> +|                                                   |
> >> +|                 FPGA PCIe Device                  |
> >> ++---------------------------------------------------+
> >> +
> >> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which
> >> can be
> >> +used to assign individual accelerators to virtual machines .
> >
> >
> > Does this HW Architecture require an Intel FPGA?  Couldn't any vendors FPGA
> > be used as long as it presented itself the PCIe bus the same and contained
> > an appropriate Device Feature List?
> >
> >> +
> >> +FME (FPGA Management Engine)
> >> +============================
> >> +The FPGA Management Enging performs power and thermal management, error
> >> +reporting, reconfiguration, performance reporting, and other
> >> infrastructure
> >> +functions. Each FPGA has one FME, which is always accessed through the
> >> physical
> >> +function (PF).
> >> +
> >> +User-space applications can acquire exclusive access to the FME using
> >> open(),
> >> +and release it using close().
> >> +
> >> +The following functions are exposed through ioctls:
> >> +
> >> +       Get driver API version (FPGA_GET_API_VERSION)
> >> +       Check for extensions (FPGA_CHECK_EXTENSION)
> >> +       Assign port to PF (FPGA_FME_PORT_ASSIGN)
> >> +       Release port from PF (FPGA_FME_PORT_RELEASE)
> >> +       Program bitstream (FPGA_FME_PORT_PR)
> >> +
> >> +More functions are exposed through sysfs
> >> +(/sys/class/fpga/fpga.n/intel-fpga-fme.n/):
> >> +
> >> +       Read bitstream ID (bitstream_id)
> >> +       Read bitstream metadata (bitstream_metadata)
> >> +       Read number of ports (ports_num)
> >> +       Read socket ID (socket_id)
> >> +       Read performance counters (perf/)
> >> +       Power management (power_mgmt/)
> >> +       Thermal management (thermal_mgmt/)
> >> +       Error reporting (errors/)
> >> +
> >> +PORT
> >> +====
> >> +A port represents the interface between the static FPGA fabric (the "blue
> >> +bitstream") and a partially reconfigurable region containing an AFU (the
> >> "green
> 
> Is this an fpga bridge but with added features?

Yes, I think so. As you see the fme_pr function in patch 11, related port needs
to be disabled firstly before fpga_mgr_buf_load for given accelerator.

Thanks
Hao


> 
> >> +bitstream"). It controls the communication from SW to the accelerator and
> >> +exposes features such as reset and debug.
> >> +
> >> +A PCIe device may have several ports and each port can be released from
> >> PF by
> >> +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe
> >> sriov
> >> +sysfs interface.
> >> +
> >> +AFU
> >> +===
> >> +An AFU is attached to a port and exposes a 256k MMIO region to be used
> >> for
> >> +accelerator-specific control registers.
> >> +
> >> +User-space applications can acquire exclusive access to an AFU attached
> >> to a
> >> +port by using open() on the port device node, and release it using
> >> close().
> >> +
> >> +The following functions are exposed through ioctls:
> >> +
> >> +       Get driver API version (FPGA_GET_API_VERSION)
> >> +       Check for extensions (FPGA_CHECK_EXTENSION)
> >> +       Get port info (FPGA_PORT_GET_INFO)
> >> +       Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
> >> +       Map DMA buffer (FPGA_PORT_DMA_MAP)
> >> +       Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
> >> +       Reset AFU (FPGA_PORT_RESET)
> >> +       Enable UMsg (FPGA_PORT_UMSG_ENABLE)
> >> +       Disable UMsg (FPGA_PORT_UMSG_DISABLE)
> >> +       Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
> >> +       Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)
> >> +
> >> +User-space applications can also mmap() accelerator MMIO regions.
> >> +
> >> +More functions are exposed through sysfs:
> >> +(/sys/class/fpga/fpga.n/intel-fpga-port.m/):
> >> +
> >> +       Read Accelerator GUID (afu_id)
> >> +       Error reporting (errors/)
> >> +
> >> +Partial Reconfiguration
> >> +=======================
> >> +As mentioned above, accelerators can be reconfigured through partial
> >> +reconfiguration of a green bitstream file (GBS). The green bitstream must
> >> have
> >> +been generated for the exact blue bitstream and targeted reconfigurable
> >> region
> >> +(port) of the FPGA; otherwise, the reconfiguration operation will fail
> >> and
> >> +possibly cause system instability. This compatibility can be checked by
> >> +comparing the interface ID noted in the GBS header against the interface
> >> ID
> >> +exposed by the FME through sysfs (see above). This check is usually done
> >> by
> >> +user-space before calling the reconfiguration IOCTL.
> >> +
> >> +FPGA virtualization
> >> +===================
> >> +To enable accessing an accelerator from applications running in a VM, the
> >> +respective AFU's port needs to be assigned to a VF using the following
> >> steps:
> >> +
> >> + a) The PF owns all AFU ports by default. Any port that needs to be
> >> reassigned
> >> + to a VF must be released from PF firstly through the
> >> FPGA_FME_PORT_RELEASE
> >> + ioctl on the FME device.
> >> +
> >> + b) Once N ports are released from PF, then user can use below command to
> >> + enable SRIOV and VFs. Each VF owns only one Port with AFU.
> >> +
> >> + echo N > $PCI_DEVICE_PATH/sriov_numvfs
> >> +
> >> + c) Pass through the VFs to VMs
> >> +
> >> + d) The AFU under VF is accessiable from applications in VM (using the
> >> same
> >> + driver inside the VF).
> >> +
> >> +Note the an FME can't be assigned to a VF, thus PR and other management
> >> +functions are only available via the PF.
> >> +
> >> +
> >> +Driver organization
> >> +===================
> >> +
> >> +  +------------------+  +---------+   |             +---------+
> >> +  | +-------+        |  |         |   |             |         |
> >> +  | | FPGA  |  FME   |  |   AFU   |   |             |   AFU   |
> >> +  | |Manager| Module |  |  Module |   |             |  Module |
> >> +  | +-------+        |  |         |   |             |         |
> >> +  +------------------+  +---------+   |             +---------+
> >> +        +-----------------------+     |      +-----------------------+
> >> +        | FPGA Container Device |     |      | FPGA Container Device |
> >> +        +-----------------------+     |      +-----------------------+
> >> +          +------------------+        |         +------------------+
> >> +          | FPGA PCIE Module |        | Virtual | FPGA PCIE Module |
> >> +          +------------------+   Host | Machine +------------------+
> >> + ------------------------------------ | ------------------------------
> >> +           +---------------+          |          +---------------+
> >> +           | PCI PF Device |          |          | PCI VF Device |
> >> +           +---------------+          |          +---------------+
> >> +
> >> +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe
> >> device
> >> +driver is always loaded first once a FPGA PCIE PF or VF device is
> >> detected. This
> >> +driver plays an infrastructural role in the driver architecuture.  It:
> >> +
> >> +       a) creates FPGA container device as parent of the feature devices.
> >> +       b) walks through the Device Feature List, which is implemented in
> >> PCIE
> >> +          device BAR memory, to discover feature devices and their sub
> >> features
> >> +          and create platform device for them under the container device.
> >
> >
> > I really like the idea of creating platform devices for the sub features. It
> > is in line with other FPGA use cases.  Platform devices are at the heart of
> > device trees used by processors directly connected FPGAs and processors
> > inside FPGAs.
> >
> >> +       c) supports SRIOV.
> >> +       d) introduces the feature device infrastructure, which abstracts
> >> +          operations for sub features and exposes common functions to
> >> feature
> >> +          device drivers.
> >> +
> >> +The FPGA Management Engine (FME) driver is a platform driver which is
> >> loaded
> >> +automatically after FME platform device creation from the PCIE driver. It
> >> +provides the key features for FPGA management, including:
> >> +
> >> +       a) Power and thermal management, error reporting, performance
> >> reporting
> >> +          and other infrastructure functions. Users can access these
> >> functions
> >> +          via sysfs interfaces exposed by FME driver.
> >> +       b) Paritial Reconfiguration. The FME driver registers a FPGA
> >> Manager
> >> +          during PR sub feature initialization; once it receives an
> >> +          FPGA_FME_PORT_PR ioctl from user, it invokes the common
> >> interface
> >> +          function from FPGA Manager to complete the partial
> >> reconfiguration of
> >> +          the bitstream to the given port.
> >> +       c) Port management for virtualization. The FME driver introduces
> >> two
> >> +          ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
> >> +          FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the
> >> port is
> >> +          released from the PF, it can be assigned to the VF through the
> >> SRIOV
> >> +          interfaces provided by PCIE driver. (Refer to "FPGA
> >> virtualization"
> >> +          for more details).
> >> +
> >> +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU)
> >> driver
> >> +is probed once the AFU platform device is created. The main function of
> >> this
> >> +module is to provide an interface for userspace applications to access
> >> the
> >> +individual accelerators, including basic reset control on port, AFU MMIO
> >> region
> >> +export, dma buffer mapping service, UMsg notification, and remote debug
> >> +functions (see above).
> >> +
> >> +
> >> +Device enumeration
> >> +==================
> >> +This section introduces how applications enumerate the fpga device from
> >> +the sysfs hierarchy under /sys/class/fpga.
> >> +
> >> +In the example below, two Intel(R) FPGA devices are installed in the
> >> host. Each
> >> +fpga device has one FME and two ports (AFUs).
> >> +
> >> +For each FPGA device, a device director is created under
> >> /sys/class/fpga/:
> >> +
> >> +       /sys/class/fpga/fpga.0
> >> +       /sys/class/fpga/fpga.1
> >> +
> >> +The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's
> >> name.
> >> +Application can retrieve name information via the sysfs interface:
> >> +
> >> +       /sys/class/fpga/fpga.0/name
> >> +
> >> +Each node has one FME and two ports (AFUs) as child devices:
> >> +
> >> +       /sys/class/fpga/fpga.0/intel-fpga-fme.0
> >> +       /sys/class/fpga/fpga.0/intel-fpga-port.0
> >> +       /sys/class/fpga/fpga.0/intel-fpga-port.1
> >> +
> >> +       /sys/class/fpga/fpga.1/intel-fpga-fme.1
> >> +       /sys/class/fpga/fpga.1/intel-fpga-port.2
> >> +       /sys/class/fpga/fpga.1/intel-fpga-port.3
> >> +
> >> +In general, the FME/AFU sysfs interfaces are named as follows:
> >> +
> >> +       /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/
> >> +       /sys/class/fpga/<fpga.n>/<intel-fpga-port.m>/
> >> +
> >> +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering
> >> all
> >> +ports.
> >> +
> >> +The device nodes used for ioctl() or mmap() can be referenced through:
> >> +
> >> +       /sys/class/fpga/<fpga.n>/<intel-fpga-port.n>/dev
> >> +       /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/dev
> >> +
> >> +
> >> +Open discussions
> >> +================
> >> +The current FME driver does not provide user space access to the FME MMIO
> >> +region, but exposes access through sysfs and ioctls. It also provides an
> >> FPGA
> >> +manger interface for partial reconfiguration (PR), but does not make use
> >> of
> >> +fpga-regions. User PR requests via the FPGA_FME_PORT_PR ioctl are handled
> >> inside
> >> +the FME, and fpga-region depends on device tree which is not used at all.
> >> There
> >> +are patches from Alan Tull to separate the device tree specific code and
> >
> >
> > I am currently trying to use those patches in a different driver.  They've
> > compiled cleanly in my out of tree pcie module driver against the 3.10
> > kernel.
> > I need to actually write the code to create and register the region, but
> > Alan's platform driver code should be a good guide for me.  Just need to
> > find the time.
> >
> >> +introduce a sysfs interface for PR. We plan to add fpga-regions support
> >> in the
> >> +driver once the related patches get merged. Then the FME driver should
> >> create
> >> +one fpga-region for each Port/AFU.
> >
> >
> > Does the FME driver create the fpga-region, or is each region described as
> > an entry in the Device Feature List and therefore created by the code that
> > enumerates the Device Feature List?
> >
> >> --
> >> 2.7.4
> >>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>
> >

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 02/16] fpga: add FPGA device framework
  2017-03-31 14:10         ` Greg KH
@ 2017-04-01 11:36           ` Wu Hao
  0 siblings, 0 replies; 93+ messages in thread
From: Wu Hao @ 2017-04-01 11:36 UTC (permalink / raw)
  To: Greg KH
  Cc: atull, moritz.fischer, linux-fpga, linux-kernel, luwei.kang,
	yi.z.zhang, tim.whisonant, Enno Luebbers, Shiva Rao,
	Christopher Rauer, guangrong.xiao

On Fri, Mar 31, 2017 at 04:10:09PM +0200, Greg KH wrote:
> On Fri, Mar 31, 2017 at 09:31:09PM +0800, Wu Hao wrote:
> > > On Thu, Mar 30, 2017 at 08:08:02PM +0800, Wu Hao wrote:
> > > > +#include <linux/device.h>
> > > > +#include <linux/module.h>
> > > > +#include <linux/slab.h>
> > > > +#include <linux/fpga/fpga-dev.h>
> > > > +
> > > > +static DEFINE_IDA(fpga_dev_ida);
> > > > +static struct class *fpga_dev_class;
> > > > +
> > > > +static ssize_t name_show(struct device *dev,
> > > > +			 struct device_attribute *attr, char *buf) {
> > > > +	struct fpga_dev *fdev = to_fpga_dev(dev);
> > > > +
> > > > +	return sprintf(buf, "%s\n", fdev->name); } static 
> > > > +DEVICE_ATTR_RO(name);
> > > 
> > > There already is a name for the device, it's the directory name.
> > 
> > For current implementation, the directory will have a common name like
> > 
> > /sys/class/fpga/fpga.0
> > /sys/class/fpga/fpga.1
> > /sys/class/fpga/fpga.2
> > ...
> > 
> > For the 'name' sysfs interface, driver can put more device specific info
> > into this 'name', e.g intel-fpga-dev. Userspace can use this information
> > to know which kind of FPGA device it is. e.g if applications read the
> > /sys/class/fpga/fpga.5/name as intel-fpga-dev, it means the 5th fpga
> > device on the system is a Intel FPGA device, and then application applies
> > related method to enumerate the accelerators for it.
> > And other existing fpga class has similar sysfs interface too, so I would
> > like to keep it aligned with others.
> 
> Ok, then document the heck out of this in Documentation/ABI/ which I
> don't think you did for the sysfs files you are creating.

Sure, thanks a lot for the reminder.

I will prepare the sysfs docs in the next version.

Thanks
Hao

> 
> thanks,
> 
> greg k-h
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 02/16] fpga: add FPGA device framework
  2017-03-31 19:01       ` matthew.gerlach
@ 2017-04-01 12:18         ` Wu Hao
  2017-07-25 21:32           ` Alan Tull
  0 siblings, 1 reply; 93+ messages in thread
From: Wu Hao @ 2017-04-01 12:18 UTC (permalink / raw)
  To: matthew.gerlach
  Cc: Greg KH, atull, moritz.fischer, linux-fpga, linux-kernel,
	luwei.kang, yi.z.zhang, Tim Whisonant, Enno Luebbers, Shiva Rao,
	Christopher Rauer, Xiao Guangrong

On Fri, Mar 31, 2017 at 12:01:13PM -0700, matthew.gerlach@linux.intel.com wrote:
> On Fri, 31 Mar 2017, Wu Hao wrote:
> >On Fri, Mar 31, 2017 at 08:09:09AM +0200, Greg KH wrote:
> >>On Thu, Mar 30, 2017 at 08:08:02PM +0800, Wu Hao wrote:
> >>>During FPGA device (e.g PCI-based) discovery, platform devices are
> >>>registered for different FPGA function units. But the device node path
> >>>isn't quite friendly to applications.
> >>>
> >>>Consider this case, applications want to access child device's sysfs file
> >>>for some information.
> >>>
> >>>1) Access using bus-based path (e.g PCI)
> >>>
> >>>  /sys/bus/pci/devices/xxxxx/fpga_func_a.0/sysfs_file
> >>>
> >>>  From the path, it's clear which PCI device is the parent, but not perfect
> >>>  solution for applications. PCI device BDF is not fixed, application may
> >>>  need to search all PCI device to find the actual FPGA Device.
> >>>
> >>>2) Or access using platform device path
> >>>
> >>>  /sys/bus/platform/devices/fpga_func_a.0/sysfs_file
> >>>
> >>>  Applications find the actual function by name easily, but no information
> >>>  about which fpga device it belongs to. It's quite confusing if multiple
> >>>  FPGA devices are in one system.
> >>>
> >>>'FPGA Device' class is introduced to resolve this problem. Each node under
> >>>this class represents a fpga device, which may have one or more child
> >>>devices. Applications only need to search under this FPGA Device class
> >>>folder to find the child device node it needs.
> >>>
> >>>For example, for the platform has 2 fpga devices, each fpga device has
> >>>3 child devices, the hierarchy looks like this.
> >>>
> >>>Two nodes are under /sys/class/fpga/:
> >>>/sys/class/fpga/fpga.0
> >>>/sys/class/fpga/fpga.1
> >>>
> >>>Each node has 1 function A device and 2 function B devices:
> >>>/sys/class/fpga/fpga.0/func_a.0
> >>>/sys/class/fpga/fpga.0/func_b.0
> >>>/sys/class/fpga/fpga.0/func_b.1
> >>>
> >>>/sys/class/fpga/fpga.1/func_a.1
> >>>/sys/class/fpga/fpga.1/func_b.2
> >>>/sys/class/fpga/fpga.1/func_b.3
> >>>
> >>>This following APIs are provided by FPGA device framework:
> >>>* fpga_dev_create
> >>>  Create fpga device under the given parent device.
> >>>* fpga_dev_destroy
> >>>  Destroy fpga device
> >>>
> >>>The following sysfs files are created:
> >>>* /sys/class/fpga/<fpga.x>/name
> >>>  Name of the fpga device.
> >>
> >>How does this interact with the existing "fpga class" that is in the
> >>kernel already?
> >
> >The fpga-dev introduced by this patch, is only a container device, and
> 
> I completely understand the need for a container device.  The fpga-region is
> also primarily a container, and in some cases the fpga-region may represent
> the entire fpga.  Over time this code may become redundant.

Thanks a lot for your review and comments.

I feel that the fpga-region implies that it supports reconfiguration, but
in our cases, the Intel FPGA device, doesn't have base fpga-region for
full reconfiguration, but many accelerators with partial reconfiguration
support. A fpga-region brings together everything needed for the
reconfiguration, and a fpga-dev is trying to brings everything on a FPGA
device together, including fpga-region/bridge/manager, access different
accelerators and other function units.

I think it's not mandatory to use fpga-dev, as fpga-dev is just trying to
provide one more option here for some complex hardware.

Thanks
Hao
 
> >drivers could register different functions under it. Per my understanding,
> >the existing "fpga class", including fpga-region, fpga-bridge and
> >fpga-manager, is used to provide reconfiguration function for FPGA. So
> >driver can create child node using this existing "fpga class" to provide
> >FPGA reconfiguration function, and more nodes under this container for
> >different functions for given FPGA device.
> >
> >For Intel FPGA device, partial reconfiguration is only one function of
> >Intel FPGA Management Engine (FME). FME driver creates fpga_manager under
> >below path for partial reconfiguration, and other interfaces for more
> >functions, e.g power management, virtualization support and etc.
> >
> >/sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/fpga_manager
> >
> >Thanks
> >Hao
> >
> >>
> >>thanks,
> >>
> >>greg k-h
> >--
> >To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> >the body of a message to majordomo@vger.kernel.org
> >More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview
  2017-04-01 11:16       ` Wu Hao
@ 2017-04-02 14:41         ` Moritz Fischer
  2017-04-03 20:44           ` Alan Tull
                             ` (2 more replies)
  0 siblings, 3 replies; 93+ messages in thread
From: Moritz Fischer @ 2017-04-02 14:41 UTC (permalink / raw)
  To: Wu Hao
  Cc: Alan Tull, matthew.gerlach, Moritz Fischer, linux-fpga,
	linux-kernel, luwei.kang, yi.z.zhang, Enno Luebbers,
	Xiao Guangrong

On Sat, Apr 01, 2017 at 07:16:19PM +0800, Wu Hao wrote:
> On Fri, Mar 31, 2017 at 01:38:06PM -0500, Alan Tull wrote:
> > On Fri, Mar 31, 2017 at 1:24 PM,  <matthew.gerlach@linux.intel.com> wrote:
> > >
> > >
> > > On Thu, 30 Mar 2017, Wu Hao wrote:
> > >
> > >
> > > Hi Wu Hao,
> > >
> > > Great documentation. I'm looking forward to diving into the rest of the
> > > patches. Please see my comments inline.
> > >
> > > Matthew Gerlach
> > >
> > >
> > >> Add a document for Intel FPGA driver overview.
> > >>
> > >> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > >> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > >> Signed-off-by: Wu Hao <hao.wu@intel.com>
> > >> ---
> > >> Documentation/fpga/intel-fpga.txt | 259
> > >> ++++++++++++++++++++++++++++++++++++++
> > >> 1 file changed, 259 insertions(+)
> > >> create mode 100644 Documentation/fpga/intel-fpga.txt
> > >>
> > >> diff --git a/Documentation/fpga/intel-fpga.txt
> > >> b/Documentation/fpga/intel-fpga.txt
> > >> new file mode 100644
> > >> index 0000000..9396cea
> > >> --- /dev/null
> > >> +++ b/Documentation/fpga/intel-fpga.txt
> > >> @@ -0,0 +1,259 @@
> > >>
> > >> +===============================================================================
> > >> +                    Intel FPGA driver Overview
> > >>
> > >> +-------------------------------------------------------------------------------
> > >> +                Enno Luebbers <enno.luebbers@intel.com>
> > >> +                Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > >> +                Wu Hao <hao.wu@intel.com>
> > >> +
> > >> +The Intel FPGA driver provides interfaces for userspace applications to
> > >> +configure, enumerate, open, and access FPGA accelerators on platforms
> > >> equipped
> > >> +with Intel(R) FPGA solutions and enables system level management
> > >> functions such
> > >> +as FPGA reconfiguration, power management, and virtualization.
> > >> +
> > >
> > >
> > > From a Linux kernel perspective, I'm not sure this is the best name for
> > > this code.  The name gives me the impression that it is a driver for all
> > > Intel FPGAs, but not all Intel FPGAs are connected to the processor over a
> > > PCIe bus.  The processor could be directely connected like the Arria10
> > > SOCFPGA.  Such a processor could certainly benefit from this accelerator
> > > usage model.  In an extreme case, couldn't a processor in the FPGA,
> > > running Linux, also benefit from this accelerator model?  Is this code a
> > > "FPGA Accelerator Framework"?
> > >
> > >> +HW Architecture
> > >> +===============
> > >> +From the OS's point of view, the FPGA hardware appears as a regular PCIe
> > >> device.
> > >> +The FPGA device memory is organized using a predefined data structure
> > >> (Device
> > >> +Feature List). Features supported by the particular FPGA device are
> > >> exposed
> > >> +through these data structures, as illustrated below:
> > >> +
> > >> +  +-------------------------------+  +-------------+
> > >> +  |              PF               |  |     VF      |
> > >> +  +-------------------------------+  +-------------+
> > >> +      ^            ^         ^              ^
> > >> +      |            |         |              |
> > >> ++-----|------------|---------|--------------|-------+
> > >> +|     |            |         |              |       |
> > >> +|  +-----+     +-------+ +-------+      +-------+   |
> > >> +|  | FME |     | Port0 | | Port1 |      | Port2 |   |
> > >> +|  +-----+     +-------+ +-------+      +-------+   |
> > >> +|                  ^         ^              ^       |
> > >> +|                  |         |              |       |
> > >> +|              +-------+ +------+       +-------+   |
> > >> +|              |  AFU  | |  AFU |       |  AFU  |   |
> > >> +|              +-------+ +------+       +-------+   |
> > >> +|                                                   |
> > >> +|                 FPGA PCIe Device                  |
> > >> ++---------------------------------------------------+
> > >> +
> > >> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which
> > >> can be
> > >> +used to assign individual accelerators to virtual machines .
> > >
> > >
> > > Does this HW Architecture require an Intel FPGA?  Couldn't any vendors FPGA
> > > be used as long as it presented itself the PCIe bus the same and contained
> > > an appropriate Device Feature List?

I think this is a good (and important) point. Especially when sysfs
entries & ioctls constituting ABI depend on it.

> > >
> > >> +
> > >> +FME (FPGA Management Engine)
> > >> +============================
> > >> +The FPGA Management Enging performs power and thermal management, error
Enging->Engine
> > >> +reporting, reconfiguration, performance reporting, and other
> > >> infrastructure
> > >> +functions. Each FPGA has one FME, which is always accessed through the
> > >> physical
> > >> +function (PF).
> > >> +
> > >> +User-space applications can acquire exclusive access to the FME using
> > >> open(),
> > >> +and release it using close().
> > >> +
> > >> +The following functions are exposed through ioctls:
> > >> +
> > >> +       Get driver API version (FPGA_GET_API_VERSION)
> > >> +       Check for extensions (FPGA_CHECK_EXTENSION)
> > >> +       Assign port to PF (FPGA_FME_PORT_ASSIGN)
> > >> +       Release port from PF (FPGA_FME_PORT_RELEASE)
> > >> +       Program bitstream (FPGA_FME_PORT_PR)
> > >> +
> > >> +More functions are exposed through sysfs
> > >> +(/sys/class/fpga/fpga.n/intel-fpga-fme.n/):
> > >> +
> > >> +       Read bitstream ID (bitstream_id)
> > >> +       Read bitstream metadata (bitstream_metadata)
> > >> +       Read number of ports (ports_num)
> > >> +       Read socket ID (socket_id)
> > >> +       Read performance counters (perf/)
> > >> +       Power management (power_mgmt/)
> > >> +       Thermal management (thermal_mgmt/)
> > >> +       Error reporting (errors/)
> > >> +
> > >> +PORT
> > >> +====
> > >> +A port represents the interface between the static FPGA fabric (the "blue
> > >> +bitstream") and a partially reconfigurable region containing an AFU (the
> > >> "green
> > 
> > Is this an fpga bridge but with added features?
> 
> Yes, I think so. As you see the fme_pr function in patch 11, related port needs
> to be disabled firstly before fpga_mgr_buf_load for given accelerator.

Can we just extend the bridge to have the additional features, please?

> > >> +bitstream"). It controls the communication from SW to the accelerator and
> > >> +exposes features such as reset and debug.
> > >> +
> > >> +A PCIe device may have several ports and each port can be released from
> > >> PF by
> > >> +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe
> > >> sriov
> > >> +sysfs interface.
> > >> +
> > >> +AFU
> > >> +===
> > >> +An AFU is attached to a port and exposes a 256k MMIO region to be used
> > >> for
> > >> +accelerator-specific control registers.
> > >> +
> > >> +User-space applications can acquire exclusive access to an AFU attached
> > >> to a
> > >> +port by using open() on the port device node, and release it using
> > >> close().
> > >> +
> > >> +The following functions are exposed through ioctls:
> > >> +
> > >> +       Get driver API version (FPGA_GET_API_VERSION)
> > >> +       Check for extensions (FPGA_CHECK_EXTENSION)
> > >> +       Get port info (FPGA_PORT_GET_INFO)
> > >> +       Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
> > >> +       Map DMA buffer (FPGA_PORT_DMA_MAP)
> > >> +       Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
> > >> +       Reset AFU (FPGA_PORT_RESET)
> > >> +       Enable UMsg (FPGA_PORT_UMSG_ENABLE)
> > >> +       Disable UMsg (FPGA_PORT_UMSG_DISABLE)
> > >> +       Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
> > >> +       Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)
> > >> +
> > >> +User-space applications can also mmap() accelerator MMIO regions.
> > >> +
> > >> +More functions are exposed through sysfs:
> > >> +(/sys/class/fpga/fpga.n/intel-fpga-port.m/):
> > >> +
> > >> +       Read Accelerator GUID (afu_id)
> > >> +       Error reporting (errors/)
> > >> +
> > >> +Partial Reconfiguration
> > >> +=======================
> > >> +As mentioned above, accelerators can be reconfigured through partial
> > >> +reconfiguration of a green bitstream file (GBS). The green bitstream must
> > >> have
> > >> +been generated for the exact blue bitstream and targeted reconfigurable
> > >> region
> > >> +(port) of the FPGA; otherwise, the reconfiguration operation will fail
> > >> and
> > >> +possibly cause system instability. This compatibility can be checked by
> > >> +comparing the interface ID noted in the GBS header against the interface
> > >> ID
> > >> +exposed by the FME through sysfs (see above). This check is usually done
> > >> by
> > >> +user-space before calling the reconfiguration IOCTL.
> > >> +
> > >> +FPGA virtualization
> > >> +===================
> > >> +To enable accessing an accelerator from applications running in a VM, the
> > >> +respective AFU's port needs to be assigned to a VF using the following
> > >> steps:
> > >> +
> > >> + a) The PF owns all AFU ports by default. Any port that needs to be
> > >> reassigned
> > >> + to a VF must be released from PF firstly through the
> > >> FPGA_FME_PORT_RELEASE
> > >> + ioctl on the FME device.
> > >> +
> > >> + b) Once N ports are released from PF, then user can use below command to
> > >> + enable SRIOV and VFs. Each VF owns only one Port with AFU.
> > >> +
> > >> + echo N > $PCI_DEVICE_PATH/sriov_numvfs
> > >> +
> > >> + c) Pass through the VFs to VMs
> > >> +
> > >> + d) The AFU under VF is accessiable from applications in VM (using the
> > >> same
> > >> + driver inside the VF).
> > >> +
> > >> +Note the an FME can't be assigned to a VF, thus PR and other management
> > >> +functions are only available via the PF.
> > >> +
> > >> +
> > >> +Driver organization
> > >> +===================
> > >> +
> > >> +  +------------------+  +---------+   |             +---------+
> > >> +  | +-------+        |  |         |   |             |         |
> > >> +  | | FPGA  |  FME   |  |   AFU   |   |             |   AFU   |
> > >> +  | |Manager| Module |  |  Module |   |             |  Module |
> > >> +  | +-------+        |  |         |   |             |         |
> > >> +  +------------------+  +---------+   |             +---------+
> > >> +        +-----------------------+     |      +-----------------------+
> > >> +        | FPGA Container Device |     |      | FPGA Container Device |
> > >> +        +-----------------------+     |      +-----------------------+
> > >> +          +------------------+        |         +------------------+
> > >> +          | FPGA PCIE Module |        | Virtual | FPGA PCIE Module |
> > >> +          +------------------+   Host | Machine +------------------+
> > >> + ------------------------------------ | ------------------------------
> > >> +           +---------------+          |          +---------------+
> > >> +           | PCI PF Device |          |          | PCI VF Device |
> > >> +           +---------------+          |          +---------------+
> > >> +
> > >> +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe
> > >> device
> > >> +driver is always loaded first once a FPGA PCIE PF or VF device is
> > >> detected. This
> > >> +driver plays an infrastructural role in the driver architecuture.  It:
> > >> +
> > >> +       a) creates FPGA container device as parent of the feature devices.
> > >> +       b) walks through the Device Feature List, which is implemented in
> > >> PCIE
> > >> +          device BAR memory, to discover feature devices and their sub
> > >> features
> > >> +          and create platform device for them under the container device.
> > >
> > >
> > > I really like the idea of creating platform devices for the sub features. It
> > > is in line with other FPGA use cases.  Platform devices are at the heart of
> > > device trees used by processors directly connected FPGAs and processors
> > > inside FPGAs.
> > >
> > >> +       c) supports SRIOV.
> > >> +       d) introduces the feature device infrastructure, which abstracts
> > >> +          operations for sub features and exposes common functions to
> > >> feature
> > >> +          device drivers.
> > >> +
> > >> +The FPGA Management Engine (FME) driver is a platform driver which is
> > >> loaded
> > >> +automatically after FME platform device creation from the PCIE driver. It
> > >> +provides the key features for FPGA management, including:
> > >> +
> > >> +       a) Power and thermal management, error reporting, performance
> > >> reporting
> > >> +          and other infrastructure functions. Users can access these
> > >> functions
> > >> +          via sysfs interfaces exposed by FME driver.
> > >> +       b) Paritial Reconfiguration. The FME driver registers a FPGA
> > >> Manager
> > >> +          during PR sub feature initialization; once it receives an
> > >> +          FPGA_FME_PORT_PR ioctl from user, it invokes the common
> > >> interface
> > >> +          function from FPGA Manager to complete the partial
> > >> reconfiguration of
> > >> +          the bitstream to the given port.
> > >> +       c) Port management for virtualization. The FME driver introduces
> > >> two
> > >> +          ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
> > >> +          FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the
> > >> port is
> > >> +          released from the PF, it can be assigned to the VF through the
> > >> SRIOV
> > >> +          interfaces provided by PCIE driver. (Refer to "FPGA
> > >> virtualization"
> > >> +          for more details).
> > >> +
> > >> +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU)
> > >> driver
> > >> +is probed once the AFU platform device is created. The main function of
> > >> this
> > >> +module is to provide an interface for userspace applications to access
> > >> the
> > >> +individual accelerators, including basic reset control on port, AFU MMIO
> > >> region
> > >> +export, dma buffer mapping service, UMsg notification, and remote debug
> > >> +functions (see above).
> > >> +
> > >> +
> > >> +Device enumeration
> > >> +==================
> > >> +This section introduces how applications enumerate the fpga device from
> > >> +the sysfs hierarchy under /sys/class/fpga.
> > >> +
> > >> +In the example below, two Intel(R) FPGA devices are installed in the
> > >> host. Each
> > >> +fpga device has one FME and two ports (AFUs).
> > >> +
> > >> +For each FPGA device, a device director is created under
> > >> /sys/class/fpga/:
> > >> +
> > >> +       /sys/class/fpga/fpga.0
> > >> +       /sys/class/fpga/fpga.1
> > >> +
> > >> +The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's
> > >> name.
> > >> +Application can retrieve name information via the sysfs interface:
> > >> +
> > >> +       /sys/class/fpga/fpga.0/name
> > >> +
> > >> +Each node has one FME and two ports (AFUs) as child devices:
> > >> +
> > >> +       /sys/class/fpga/fpga.0/intel-fpga-fme.0
> > >> +       /sys/class/fpga/fpga.0/intel-fpga-port.0
> > >> +       /sys/class/fpga/fpga.0/intel-fpga-port.1
> > >> +
> > >> +       /sys/class/fpga/fpga.1/intel-fpga-fme.1
> > >> +       /sys/class/fpga/fpga.1/intel-fpga-port.2
> > >> +       /sys/class/fpga/fpga.1/intel-fpga-port.3
> > >> +
> > >> +In general, the FME/AFU sysfs interfaces are named as follows:
> > >> +
> > >> +       /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/
> > >> +       /sys/class/fpga/<fpga.n>/<intel-fpga-port.m>/
> > >> +
> > >> +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering
> > >> all
> > >> +ports.
> > >> +
> > >> +The device nodes used for ioctl() or mmap() can be referenced through:
> > >> +
> > >> +       /sys/class/fpga/<fpga.n>/<intel-fpga-port.n>/dev
> > >> +       /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/dev
> > >> +
> > >> +
> > >> +Open discussions
> > >> +================
> > >> +The current FME driver does not provide user space access to the FME MMIO
> > >> +region, but exposes access through sysfs and ioctls. It also provides an
> > >> FPGA
> > >> +manger interface for partial reconfiguration (PR), but does not make use
> > >> of
> > >> +fpga-regions. User PR requests via the FPGA_FME_PORT_PR ioctl are handled
> > >> inside
> > >> +the FME, and fpga-region depends on device tree which is not used at all.
> > >> There
> > >> +are patches from Alan Tull to separate the device tree specific code and
> > >
> > >
> > > I am currently trying to use those patches in a different driver.  They've
> > > compiled cleanly in my out of tree pcie module driver against the 3.10
> > > kernel.
> > > I need to actually write the code to create and register the region, but
> > > Alan's platform driver code should be a good guide for me.  Just need to
> > > find the time.
> > >
> > >> +introduce a sysfs interface for PR. We plan to add fpga-regions support
> > >> in the
> > >> +driver once the related patches get merged. Then the FME driver should
> > >> create
> > >> +one fpga-region for each Port/AFU.
> > >
> > >
> > > Does the FME driver create the fpga-region, or is each region described as
> > > an entry in the Device Feature List and therefore created by the code that
> > > enumerates the Device Feature List?
> > >
> > >> --
> > >> 2.7.4
> > >>
> > >> --
> > >> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> > >> the body of a message to majordomo@vger.kernel.org
> > >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > >>
> > >

Cheers,
Moritz

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-04-01 11:08     ` Wu Hao
@ 2017-04-03 16:30       ` Alan Tull
  2017-04-04  6:05         ` Wu Hao
  0 siblings, 1 reply; 93+ messages in thread
From: Alan Tull @ 2017-04-03 16:30 UTC (permalink / raw)
  To: Wu Hao
  Cc: Moritz Fischer, linux-fpga, linux-kernel, luwei.kang, yi.z.zhang,
	Tim Whisonant, Enno Luebbers, Shiva Rao, Christopher Rauer,
	Alan Tull, Xiao Guangrong

On Sat, Apr 1, 2017 at 6:08 AM, Wu Hao <hao.wu@intel.com> wrote:
> On Fri, Mar 31, 2017 at 02:10:12PM -0500, Alan Tull wrote:
>> On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote:
>> > From: Kang Luwei <luwei.kang@intel.com>
>> >
>> > Partial Reconfiguration (PR) is the most important function for FME. It
>> > allows reconfiguration for given Port/Accelerated Function Unit (AFU).
>> >
>> > This patch adds support for PR sub feature. In this patch, it registers
>> > a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
>> > for PR operation once PR request received via ioctl. Below user space
>> > interfaces are exposed by this sub feature.
>> >
>> > Sysfs interface:
>> > * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
>> >   Read-only. Indicate the hardware interface information. Userspace
>> >   applications need to check this interface to select correct green
>> >   bitstream format before PR.
>> >
>> > Ioctl interface:
>> > * FPGA_FME_PORT_PR
>> >   Do partial reconfiguration per information from userspace, including
>> >   target port(AFU), buffer size and address info. It returns the PR status
>> >   (PR error code if failed) to userspace.
>> >
>> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
>> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
>> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
>> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
>> > Signed-off-by: Alan Tull <alan.tull@intel.com>
>> > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
>> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> > Signed-off-by: Wu Hao <hao.wu@intel.com>
>> > ---
>> >  drivers/fpga/intel/Makefile      |   2 +-
>> >  drivers/fpga/intel/feature-dev.h |  58 ++++++
>> >  drivers/fpga/intel/fme-main.c    |  44 ++++-
>> >  drivers/fpga/intel/fme-pr.c      | 400 +++++++++++++++++++++++++++++++++++++++
>> >  drivers/fpga/intel/fme.h         |  32 ++++
>> >  include/uapi/linux/intel-fpga.h  |  44 +++++
>> >  6 files changed, 578 insertions(+), 2 deletions(-)
>> >  create mode 100644 drivers/fpga/intel/fme-pr.c
>> >  create mode 100644 drivers/fpga/intel/fme.h
>> >
>> > diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
>> > index 546861d..0452cb6 100644
>> > --- a/drivers/fpga/intel/Makefile
>> > +++ b/drivers/fpga/intel/Makefile
>> > @@ -2,4 +2,4 @@ obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
>> >  obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
>> >
>> >  intel-fpga-pci-objs := pcie.o feature-dev.o
>> > -intel-fpga-fme-objs := fme-main.o
>> > +intel-fpga-fme-objs := fme-main.o fme-pr.o
>> > diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
>> > index dccc283..5a25c915 100644
>> > --- a/drivers/fpga/intel/feature-dev.h
>> > +++ b/drivers/fpga/intel/feature-dev.h
>> > @@ -150,8 +150,66 @@ struct feature_fme_err {
>> >  };
>> >
>> >  /* FME Partial Reconfiguration Sub Feature Register Set */
>> > +/* FME PR Control Register */
>> > +struct feature_fme_pr_ctl {
>> > +       union {
>> > +               u64 csr;
>> > +               struct {
>> > +                       u8  pr_reset:1;         /* Reset PR Engine */
>> > +                       u8  rsvdz1:3;
>> > +                       u8  pr_reset_ack:1;     /* Reset PR Engine Ack */
>> > +                       u8  rsvdz2:3;
>> > +                       u8  pr_regionid:2;      /* PR Region ID */
>> > +                       u8  rsvdz3:2;
>> > +                       u8  pr_start_req:1;     /* PR Start Request */
>> > +                       u8  pr_push_complete:1; /* PR Data push complete */
>> > +                       u8  pr_kind:1;          /* Load Customer or Intel GBS */
>> > +                       u32 rsvdz4:17;
>> > +                       u32 config_data;
>> > +               };
>> > +       };
>> > +};
>> > +
>> > +/* FME PR Status Register */
>> > +struct feature_fme_pr_status {
>> > +       union {
>> > +               u64 csr;
>> > +               struct {
>> > +                       u16 pr_credit:9;        /* Number of PR Credits */
>> > +                       u8  rsvdz1:7;
>> > +                       u8  pr_status:1;        /* PR Operation status */
>> > +                       u8  rsvdz2:3;
>> > +                       u8  pr_ctrlr_status:3;  /* Controller status */
>> > +                       u8  rsvdz3:1;
>> > +                       u8  pr_host_status:4;   /* PR Host status */
>> > +                       u64 rsvdz4:36;
>> > +               };
>> > +       };
>> > +};
>> > +
>> > +/* FME PR Data Register */
>> > +struct feature_fme_pr_data {
>> > +       union {
>> > +               u64 csr;
>> > +               struct {
>> > +                       /* PR data from the raw-binary file */
>> > +                       u32 pr_data_raw;
>> > +                       u32 rsvd;
>> > +               };
>> > +       };
>> > +};
>> > +
>> >  struct feature_fme_pr {
>> >         struct feature_header header;
>> > +       struct feature_fme_pr_ctl control;
>> > +       struct feature_fme_pr_status status;
>> > +       struct feature_fme_pr_data data;
>> > +       u64 error;
>> > +
>> > +       u64 rsvd[16];
>> > +
>> > +       u64 intfc_id_l;         /* PR interface Id Low */
>> > +       u64 intfc_id_h;         /* PR interface Id High */
>> >  };
>> >
>> >  /* PORT Header Register Set */
>> > diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
>> > index 36d0c4c..0d9a7a6 100644
>> > --- a/drivers/fpga/intel/fme-main.c
>> > +++ b/drivers/fpga/intel/fme-main.c
>> > @@ -23,6 +23,7 @@
>> >  #include <linux/intel-fpga.h>
>> >
>> >  #include "feature-dev.h"
>> > +#include "fme.h"
>> >
>> >  static ssize_t ports_num_show(struct device *dev,
>> >                               struct device_attribute *attr, char *buf)
>> > @@ -101,6 +102,10 @@ static struct feature_driver fme_feature_drvs[] = {
>> >                 .ops = &fme_hdr_ops,
>> >         },
>> >         {
>> > +               .name = FME_FEATURE_PR_MGMT,
>> > +               .ops = &pr_mgmt_ops,
>> > +       },
>> > +       {
>> >                 .ops = NULL,
>> >         },
>> >  };
>> > @@ -182,14 +187,48 @@ static const struct file_operations fme_fops = {
>> >         .unlocked_ioctl = fme_ioctl,
>> >  };
>> >
>> > +static int fme_dev_init(struct platform_device *pdev)
>> > +{
>> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> > +       struct fpga_fme *fme;
>> > +
>> > +       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
>> > +       if (!fme)
>> > +               return -ENOMEM;
>> > +
>> > +       fme->pdata = pdata;
>> > +
>> > +       mutex_lock(&pdata->lock);
>> > +       fpga_pdata_set_private(pdata, fme);
>> > +       mutex_unlock(&pdata->lock);
>> > +       return 0;
>> > +}
>> > +
>> > +static void fme_dev_destroy(struct platform_device *pdev)
>> > +{
>> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> > +       struct fpga_fme *fme;
>> > +
>> > +       mutex_lock(&pdata->lock);
>> > +       fme = fpga_pdata_get_private(pdata);
>> > +       fpga_pdata_set_private(pdata, NULL);
>> > +       mutex_unlock(&pdata->lock);
>> > +
>> > +       devm_kfree(&pdev->dev, fme);
>> > +}
>> > +
>> >  static int fme_probe(struct platform_device *pdev)
>> >  {
>> >         int ret;
>> >
>> > -       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
>> > +       ret = fme_dev_init(pdev);
>> >         if (ret)
>> >                 goto exit;
>> >
>> > +       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
>> > +       if (ret)
>> > +               goto dev_destroy;
>> > +
>> >         ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
>> >         if (ret)
>> >                 goto feature_uinit;
>> > @@ -198,6 +237,8 @@ static int fme_probe(struct platform_device *pdev)
>> >
>> >  feature_uinit:
>> >         fpga_dev_feature_uinit(pdev);
>> > +dev_destroy:
>> > +       fme_dev_destroy(pdev);
>> >  exit:
>> >         return ret;
>> >  }
>> > @@ -206,6 +247,7 @@ static int fme_remove(struct platform_device *pdev)
>> >  {
>> >         fpga_dev_feature_uinit(pdev);
>> >         fpga_unregister_dev_ops(pdev);
>> > +       fme_dev_destroy(pdev);
>> >         return 0;
>> >  }
>> >
>> > diff --git a/drivers/fpga/intel/fme-pr.c b/drivers/fpga/intel/fme-pr.c
>> > new file mode 100644
>> > index 0000000..3b44a3e
>> > --- /dev/null
>> > +++ b/drivers/fpga/intel/fme-pr.c
>> > @@ -0,0 +1,400 @@
>> > +/*
>> > + * Driver for Intel FPGA Management Engine (FME) Partial Reconfiguration
>> > + *
>> > + * Copyright (C) 2017 Intel Corporation, Inc.
>> > + *
>> > + * Authors:
>> > + *   Kang Luwei <luwei.kang@intel.com>
>> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> > + *   Joseph Grecco <joe.grecco@intel.com>
>> > + *   Enno Luebbers <enno.luebbers@intel.com>
>> > + *   Tim Whisonant <tim.whisonant@intel.com>
>> > + *   Ananda Ravuri <ananda.ravuri@intel.com>
>> > + *   Christopher Rauer <christopher.rauer@intel.com>
>> > + *   Henry Mitchel <henry.mitchel@intel.com>
>> > + *
>> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
>> > + * redistributing this file, you may do so under either license. See the
>> > + * LICENSE.BSD file under this directory for the BSD license and see
>> > + * the COPYING file in the top-level directory for the GPLv2 license.
>> > + */
>> > +
>> > +#include <linux/types.h>
>> > +#include <linux/device.h>
>> > +#include <linux/vmalloc.h>
>> > +#include <linux/uaccess.h>
>> > +#include <linux/fpga/fpga-mgr.h>
>> > +#include <linux/intel-fpga.h>
>> > +
>> > +#include "feature-dev.h"
>> > +#include "fme.h"
>> > +
>> > +#define PR_WAIT_TIMEOUT   8000000
>> > +
>> > +#define PR_HOST_STATUS_IDLE    0
>> > +
>> > +DEFINE_FPGA_PR_ERR_MSG(pr_err_msg);
>> > +
>> > +static ssize_t interface_id_show(struct device *dev,
>> > +                                struct device_attribute *attr, char *buf)
>> > +{
>> > +       u64 intfc_id_l, intfc_id_h;
>> > +       struct feature_fme_pr *fme_pr
>> > +               = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_PR_MGMT);
>> > +
>> > +       intfc_id_l = readq(&fme_pr->intfc_id_l);
>> > +       intfc_id_h = readq(&fme_pr->intfc_id_h);
>> > +
>> > +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
>> > +                       (unsigned long long)intfc_id_h,
>> > +                       (unsigned long long)intfc_id_l);
>> > +}
>> > +static DEVICE_ATTR_RO(interface_id);
>> > +
>> > +static struct attribute *pr_mgmt_attrs[] = {
>> > +       &dev_attr_interface_id.attr,
>> > +       NULL,
>> > +};
>> > +
>> > +struct attribute_group pr_mgmt_attr_group = {
>> > +       .attrs  = pr_mgmt_attrs,
>> > +       .name   = "pr",
>> > +};
>> > +
>> > +static u64
>> > +pr_err_handle(struct platform_device *pdev, struct feature_fme_pr *fme_pr)
>> > +{
>> > +       struct feature_fme_pr_status fme_pr_status;
>> > +       unsigned long err_code;
>> > +       u64 fme_pr_error;
>> > +       int i = 0;
>> > +
>> > +       fme_pr_status.csr = readq(&fme_pr->status);
>> > +       if (!fme_pr_status.pr_status)
>> > +               return 0;
>> > +
>> > +       err_code = fme_pr_error = readq(&fme_pr->error);
>> > +       for_each_set_bit(i, &err_code, PR_MAX_ERR_NUM)
>> > +               dev_dbg(&pdev->dev, "%s\n", pr_err_msg[i]);
>> > +       writeq(fme_pr_error, &fme_pr->error);
>> > +       return fme_pr_error;
>> > +}
>> > +
>> > +static int fme_pr_write_init(struct fpga_manager *mgr,
>> > +               struct fpga_image_info *info, const char *buf, size_t count)
>> > +{
>> > +       struct fpga_fme *fme = mgr->priv;
>> > +       struct platform_device *pdev;
>> > +       struct feature_fme_pr *fme_pr;
>> > +       struct feature_fme_pr_ctl fme_pr_ctl;
>> > +       struct feature_fme_pr_status fme_pr_status;
>> > +
>> > +       pdev = fme->pdata->dev;
>> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
>> > +                               FME_FEATURE_ID_PR_MGMT);
>> > +       if (!fme_pr)
>> > +               return -EINVAL;
>> > +
>> > +       if (WARN_ON(info->flags != FPGA_MGR_PARTIAL_RECONFIG))
>> > +               return -EINVAL;
>> > +
>> > +       dev_dbg(&pdev->dev, "resetting PR before initiated PR\n");
>> > +
>> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> > +       fme_pr_ctl.pr_reset = 1;
>> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> > +
>> > +       fme_pr_ctl.pr_reset_ack = 1;
>> > +
>> > +       if (fpga_wait_register_field(pr_reset_ack, fme_pr_ctl,
>> > +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
>> > +               dev_err(&pdev->dev, "maximum PR timeout\n");
>> > +               return -ETIMEDOUT;
>> > +       }
>> > +
>> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> > +       fme_pr_ctl.pr_reset = 0;
>> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> > +
>> > +       dev_dbg(&pdev->dev,
>> > +               "waiting for PR resource in HW to be initialized and ready\n");
>> > +
>> > +       fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
>> > +
>> > +       if (fpga_wait_register_field(pr_host_status, fme_pr_status,
>> > +                                    &fme_pr->status, PR_WAIT_TIMEOUT, 1)) {
>> > +               dev_err(&pdev->dev, "maximum PR timeout\n");
>> > +               return -ETIMEDOUT;
>> > +       }
>> > +
>> > +       dev_dbg(&pdev->dev, "check if have any previous PR error\n");
>> > +       pr_err_handle(pdev, fme_pr);
>> > +       return 0;
>> > +}
>> > +
>> > +static int fme_pr_write(struct fpga_manager *mgr,
>> > +                       const char *buf, size_t count)
>> > +{
>> > +       struct fpga_fme *fme = mgr->priv;
>> > +       struct platform_device *pdev;
>> > +       struct feature_fme_pr *fme_pr;
>> > +       struct feature_fme_pr_ctl fme_pr_ctl;
>> > +       struct feature_fme_pr_status fme_pr_status;
>> > +       struct feature_fme_pr_data fme_pr_data;
>> > +       int delay, pr_credit, i = 0;
>> > +
>> > +       pdev = fme->pdata->dev;
>> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
>> > +                               FME_FEATURE_ID_PR_MGMT);
>> > +
>> > +       dev_dbg(&pdev->dev, "set PR port ID and start request\n");
>> > +
>> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> > +       fme_pr_ctl.pr_regionid = fme->port_id;
>> > +       fme_pr_ctl.pr_start_req = 1;
>> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> > +
>> > +       dev_dbg(&pdev->dev, "pushing data from bitstream to HW\n");
>> > +
>> > +       fme_pr_status.csr = readq(&fme_pr->status);
>> > +       pr_credit = fme_pr_status.pr_credit;
>> > +
>> > +       while (count > 0) {
>> > +               delay = 0;
>> > +               while (pr_credit <= 1) {
>> > +                       if (delay++ > PR_WAIT_TIMEOUT) {
>> > +                               dev_err(&pdev->dev, "maximum try\n");
>> > +                               return -ETIMEDOUT;
>> > +                       }
>> > +                       udelay(1);
>> > +
>> > +                       fme_pr_status.csr = readq(&fme_pr->status);
>> > +                       pr_credit = fme_pr_status.pr_credit;
>> > +               };
>> > +
>> > +               if (count >= 4) {
>> > +                       fme_pr_data.rsvd = 0;
>> > +                       fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
>> > +                       writeq(fme_pr_data.csr, &fme_pr->data);
>> > +                       count -= 4;
>> > +                       pr_credit--;
>> > +                       i++;
>> > +               } else {
>> > +                       WARN_ON(1);
>> > +                       return -EINVAL;
>> > +               }
>> > +       }
>> > +
>> > +       return 0;
>> > +}
>> > +
>> > +static int fme_pr_write_complete(struct fpga_manager *mgr,
>> > +                       struct fpga_image_info *info)
>> > +{
>> > +       struct fpga_fme *fme = mgr->priv;
>> > +       struct platform_device *pdev;
>> > +       struct feature_fme_pr *fme_pr;
>> > +       struct feature_fme_pr_ctl fme_pr_ctl;
>> > +
>> > +       pdev = fme->pdata->dev;
>> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
>> > +                               FME_FEATURE_ID_PR_MGMT);
>> > +
>> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> > +       fme_pr_ctl.pr_push_complete = 1;
>> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> > +
>> > +       dev_dbg(&pdev->dev, "green bitstream push complete\n");
>> > +       dev_dbg(&pdev->dev, "waiting for HW to release PR resource\n");
>> > +
>> > +       fme_pr_ctl.pr_start_req = 0;
>> > +
>> > +       if (fpga_wait_register_field(pr_start_req, fme_pr_ctl,
>> > +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
>> > +               dev_err(&pdev->dev, "maximum try.\n");
>> > +               return -ETIMEDOUT;
>> > +       }
>> > +
>> > +       dev_dbg(&pdev->dev, "PR operation complete, checking status\n");
>> > +       fme->pr_err = pr_err_handle(pdev, fme_pr);
>> > +       if (fme->pr_err)
>> > +               return -EIO;
>> > +
>> > +       dev_dbg(&pdev->dev, "PR done successfully\n");
>> > +       return 0;
>> > +}
>> > +
>> > +static enum fpga_mgr_states fme_pr_state(struct fpga_manager *mgr)
>> > +{
>> > +       return FPGA_MGR_STATE_UNKNOWN;
>> > +}
>> > +
>> > +static const struct fpga_manager_ops fme_pr_ops = {
>> > +       .write_init = fme_pr_write_init,
>> > +       .write = fme_pr_write,
>> > +       .write_complete = fme_pr_write_complete,
>> > +       .state = fme_pr_state,
>> > +};
>> > +
>> > +static int fme_pr(struct platform_device *pdev, unsigned long arg)
>> > +{
>> > +       void __user *argp = (void __user *)arg;
>> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> > +       struct fpga_fme *fme;
>> > +       struct fpga_manager *mgr;
>> > +       struct feature_fme_header *fme_hdr;
>> > +       struct feature_fme_capability fme_capability;
>> > +       struct fpga_image_info info;
>> > +       struct fpga_fme_port_pr port_pr;
>> > +       struct platform_device *port;
>> > +       unsigned long minsz;
>> > +       void *buf = NULL;
>> > +       int ret = 0;
>> > +
>> > +       minsz = offsetofend(struct fpga_fme_port_pr, status);
>> > +
>> > +       if (copy_from_user(&port_pr, argp, minsz))
>> > +               return -EFAULT;
>> > +
>> > +       if (port_pr.argsz < minsz || port_pr.flags)
>> > +               return -EINVAL;
>> > +
>> > +       if (!IS_ALIGNED(port_pr.buffer_size, 4))
>> > +               return -EINVAL;
>> > +
>> > +       /* get fme header region */
>> > +       fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
>> > +                                       FME_FEATURE_ID_HEADER);
>> > +       if (WARN_ON(!fme_hdr))
>> > +               return -EINVAL;
>> > +
>> > +       /* check port id */
>> > +       fme_capability.csr = readq(&fme_hdr->capability);
>> > +       if (port_pr.port_id >= fme_capability.num_ports) {
>> > +               dev_dbg(&pdev->dev, "port number more than maximum\n");
>> > +               return -EINVAL;
>> > +       }
>> > +
>> > +       if (!access_ok(VERIFY_READ, port_pr.buffer_address,
>> > +                                   port_pr.buffer_size))
>> > +               return -EFAULT;
>> > +
>> > +       buf = vmalloc(port_pr.buffer_size);
>> > +       if (!buf)
>> > +               return -ENOMEM;
>> > +
>> > +       if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
>> > +                                              port_pr.buffer_size)) {
>> > +               ret = -EFAULT;
>> > +               goto free_exit;
>> > +       }
>> > +
>> > +       memset(&info, 0, sizeof(struct fpga_image_info));
>> > +       info.flags = FPGA_MGR_PARTIAL_RECONFIG;
>> > +
>> > +       mgr = fpga_mgr_get(&pdev->dev);
>> > +       if (IS_ERR(mgr)) {
>> > +               ret = PTR_ERR(mgr);
>> > +               goto free_exit;
>> > +       }
>> > +
>> > +       mutex_lock(&pdata->lock);
>> > +       fme = fpga_pdata_get_private(pdata);
>> > +       /* fme device has been unregistered. */
>> > +       if (!fme) {
>> > +               ret = -EINVAL;
>> > +               goto unlock_exit;
>> > +       }
>> > +
>> > +       fme->pr_err = 0;
>> > +       fme->port_id = port_pr.port_id;
>>
>> It looks like you're using private data to communicate with the
>> driver, i.e. there is something you want to do with the fpga manager
>> framework and it doesn't have that feature.  The better way would be
>> for us to expand the framework so you don't need to do that.
>>
>> port_id is the kind of thing that should be communicated to the driver
>> through fpga_image_info, so we could add that to the struct.  Should
>> we call it port_id?

Let's call it region_id.

>> Or is there something more generic that may be
>> useful in the future for other architectures?.
>
> Hi Alan
>
> Thanks for your feedback. :)
>
> As you know, each Intel FPGA device may have more than one accelerator,
> and all accelerators share the same fpga_manager (only one FME module).
> port_id = 0 means the first accelerator on this fpga devices. So it's
> needed by the fpga_manager to distinguish one accelerator from others
> for partial reconfiguration.
>
> Adding a member like a 'id' to fpga_image_info definitely works for us
> in this case. We can add it this way, but I feel 'id' here seems not
> related to fpga image, but characteristic of fpga region.

The  fpga_image_info struct started life as just image specific info,
but I want it to go in the direction of including parameters needed to
program it this specific time. Otherwise we are stuck having to keep
adding parameters as our use of FPGA develops.  It probably could be
documented better as 'information needed to program a FPGA image'
rather than strictly 'information about this particular FPGA image'.
My patch "fpga-mgr: pass parameters for loading fpga in image info"
goes in this direction by having the buf, firmware name, or sg list
passed in the info for the added fpga_mgr_load() function.  Actually I
should probably simplify the API and get rid of fpga_mgr_buf_load,
fpga_mgr_buf_load_sg, and fpga_mgr_firmware_load and require people to
use fpga_mgr_load (passing all parameters in fpga_image_info).

> It may be a
> little confusing. One rough idea is that keep this info under fpga region
> (maybe its private data), and pass the fpga-region to fpga_mgr_buf_load,

Yes, keep this info in fpga-region.  When the region wants to program
using fpga-mgr, add the region id to fpga_image_info.  I propose
calling it region_id.

> then fpga_manager knows the target region for partial reconfiguration.
> If consider pr sysfs interface support under fpga-region in the future,
> then we don't need to create a new 'id' sysfs file, as fpga-region itself
> could provide this kind of info. But I'm not sure if this is the right
> direction.
>
>>
>> pr_err appears to be machine specific error codes that are
>> communicated outside your low level driver.  (Calling it pr_err is
>> extra confusing since Linux already has a commonly name function by
>> the same name).  The framework has state, but that's not doing what
>> you want here.  Maybe we could add a framework ops called status so
>> that status could be communicated from the low level driver. It would
>> be useful to abstract the machine specific state to a defined enum
>> that would be part of the fpga mgr framework.  I remember we had
>> something like that in the earliest version of fpga manager but it got
>> changed to state rather than status for some reason.
>>
>
> Adding a status function and a 'status' member to fpga manager sounds good.
> But I'm not sure how to abstract these error codes, currently what we have
> are all PR error codes for Intel FPGA device, e.g "PR FIFO overflow error",
> "PR secure load error" and etc (see include/uapi/linux/intel-fpga.h). Is it
> fine to let each driver to define how to use that 'status' for machine
> specific status?

I looked at the list of errors in include/uapi/linux/intel-fpga.h.
They all seem pretty generic to me except I am not clear what "secure
load error" or "IP protocol error" mean and whether other
architectures would have them.  But certainly things like crc error,
incompatible bitstream, fifo overflow are generic.  So let's see if we
can define all these in a way that's generic and just pass up a error
number.  That way upper layers can know how to deal with them
possibly.  I would take the word "PR" off these since the error set
applies whether someone is doing full reconfig or partial reconfig.

>
>> mgr->dev won't work for you for some of the things you are using fme->pdev for?
>>
>
> Does 'fme->pdev' mean the platform device for fme? I think we use platform
> device to represent the FME module, and FME module may have multiple sub
> features, driver discovers these sub features via the 'Device Feature List'
> in the PCIe Device Bar. PR is only one of the sub features under FME module,
> if any FME module doesn't have PR sub feature, fpga-manager will not be
> created at all. But this will not impact other sub features, e.g thermal
> management, error reporting and etc, to create their own interfaces under
> the platform device.

If we need it for private data, that's may be actually OK.   Just
wondering whether the priv was needed.   I think that's the one
element in the private data struct that was used privately instead of
being used to pass info outside the framework.

>
> Thanks
> Hao
>
>> Alan
>>
>> > +
>> > +       /* Find and get port device by index */
>> > +       port = pdata->fpga_for_each_port(pdev, &fme->port_id,
>> > +                                        fpga_port_check_id);
>> > +       WARN_ON(!port);
>> > +
>> > +       /* Disable Port before PR */
>> > +       fpga_port_disable(port);
>> > +
>> > +       ret = fpga_mgr_buf_load(mgr, &info, buf, port_pr.buffer_size);
>> > +       port_pr.status = fme->pr_err;
>> > +
>> > +       /* Re-enable Port after PR finished */
>> > +       fpga_port_enable(port);
>> > +
>> > +       put_device(&port->dev);
>> > +
>> > +unlock_exit:
>> > +       mutex_unlock(&pdata->lock);
>> > +       fpga_mgr_put(mgr);
>> > +free_exit:
>> > +       vfree(buf);
>> > +       if (copy_to_user((void __user *)arg, &port_pr, minsz))
>> > +               return -EFAULT;
>> > +       return ret;
>> > +}
>> > +
>> > +static int fpga_fme_pr_probe(struct platform_device *pdev)
>> > +{
>> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> > +       struct fpga_fme *priv;
>> > +       int ret;
>> > +
>> > +       mutex_lock(&pdata->lock);
>> > +       priv = fpga_pdata_get_private(pdata);
>> > +       ret = fpga_mgr_register(&pdata->dev->dev,
>> > +               "Intel FPGA Manager", &fme_pr_ops, priv);
>> > +       mutex_unlock(&pdata->lock);
>> > +
>> > +       return ret;
>> > +}
>> > +
>> > +static int fpga_fme_pr_remove(struct platform_device *pdev)
>> > +{
>> > +       fpga_mgr_unregister(&pdev->dev);
>> > +       return 0;
>> > +}
>> > +
>> > +static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
>> > +{
>> > +       int ret;
>> > +
>> > +       ret = fpga_fme_pr_probe(pdev);
>> > +       if (ret)
>> > +               return ret;
>> > +
>> > +       ret = sysfs_create_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
>> > +       if (ret)
>> > +               fpga_fme_pr_remove(pdev);
>> > +
>> > +       return ret;
>> > +}
>> > +
>> > +static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
>> > +{
>> > +       sysfs_remove_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
>> > +       fpga_fme_pr_remove(pdev);
>> > +}
>> > +
>> > +static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
>> > +       unsigned int cmd, unsigned long arg)
>> > +{
>> > +       long ret;
>> > +
>> > +       switch (cmd) {
>> > +       case FPGA_FME_PORT_PR:
>> > +               ret = fme_pr(pdev, arg);
>> > +               break;
>> > +       default:
>> > +               ret = -ENODEV;
>> > +       }
>> > +
>> > +       return ret;
>> > +}
>> > +
>> > +struct feature_ops pr_mgmt_ops = {
>> > +       .init = pr_mgmt_init,
>> > +       .uinit = pr_mgmt_uinit,
>> > +       .ioctl = fme_pr_ioctl,
>> > +};
>> > diff --git a/drivers/fpga/intel/fme.h b/drivers/fpga/intel/fme.h
>> > new file mode 100644
>> > index 0000000..d6cb7ce
>> > --- /dev/null
>> > +++ b/drivers/fpga/intel/fme.h
>> > @@ -0,0 +1,32 @@
>> > +/*
>> > + * Header file for Intel FPGA Management Engine (FME) Driver
>> > + *
>> > + * Copyright (C) 2017 Intel Corporation, Inc.
>> > + *
>> > + * Authors:
>> > + *   Kang Luwei <luwei.kang@intel.com>
>> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> > + *   Joseph Grecco <joe.grecco@intel.com>
>> > + *   Enno Luebbers <enno.luebbers@intel.com>
>> > + *   Tim Whisonant <tim.whisonant@intel.com>
>> > + *   Ananda Ravuri <ananda.ravuri@intel.com>
>> > + *   Henry Mitchel <henry.mitchel@intel.com>
>> > + *
>> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
>> > + * redistributing this file, you may do so under either license. See the
>> > + * LICENSE.BSD file under this directory for the BSD license and see
>> > + * the COPYING file in the top-level directory for the GPLv2 license.
>> > + */
>> > +
>> > +#ifndef __INTEL_FME_H
>> > +#define __INTEL_FME_H
>> > +
>> > +struct fpga_fme {
>> > +       u8  port_id;
>> > +       u64 pr_err;
>> > +       struct feature_platform_data *pdata;
>> > +};
>> > +
>> > +extern struct feature_ops pr_mgmt_ops;
>> > +
>> > +#endif
>> > diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
>> > index 992e556..77658316 100644
>> > --- a/include/uapi/linux/intel-fpga.h
>> > +++ b/include/uapi/linux/intel-fpga.h
>> > @@ -18,6 +18,8 @@
>> >  #ifndef _UAPI_LINUX_INTEL_FPGA_H
>> >  #define _UAPI_LINUX_INTEL_FPGA_H
>> >
>> > +#include <linux/types.h>
>> > +
>> >  #define FPGA_API_VERSION 0
>> >
>> >  /*
>> > @@ -30,6 +32,7 @@
>> >  #define FPGA_MAGIC 0xB6
>> >
>> >  #define FPGA_BASE 0
>> > +#define FME_BASE 0x80
>> >
>> >  /**
>> >   * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
>> > @@ -49,4 +52,45 @@
>> >
>> >  #define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)
>> >
>> > +/* IOCTLs for FME file descriptor */
>> > +
>> > +/**
>> > + * FPGA_FME_PORT_PR - _IOWR(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
>> > + *
>> > + * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
>> > + * provided by caller.
>> > + * Return: 0 on success, -errno on failure.
>> > + * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
>> > + * some errors during PR, under this case, the user can fetch HW error code
>> > + * from fpga_fme_port_pr.status. Each bit on the error code is used as the
>> > + * index for the array created by DEFINE_FPGA_PR_ERR_MSG().
>> > + * Otherwise, it is always zero.
>> > + */
>> > +
>> > +#define DEFINE_FPGA_PR_ERR_MSG(_name_)                 \
>> > +static const char * const _name_[] = {                 \
>> > +       "PR operation error detected",                  \
>> > +       "PR CRC error detected",                        \
>> > +       "PR incompatiable bitstream error detected",    \
>> > +       "PR IP protocol error detected",                \
>> > +       "PR FIFO overflow error detected",              \
>> > +       "Reserved",                                     \
>> > +       "PR secure load error detected",                \
>> > +}
>> > +
>> > +#define PR_MAX_ERR_NUM 7
>> > +
>> > +struct fpga_fme_port_pr {
>> > +       /* Input */
>> > +       __u32 argsz;            /* Structure length */
>> > +       __u32 flags;            /* Zero for now */
>> > +       __u32 port_id;
>> > +       __u32 buffer_size;
>> > +       __u64 buffer_address;   /* Userspace address to the buffer for PR */
>> > +       /* Output */
>> > +       __u64 status;           /* HW error code if ioctl returns -EIO */
>> > +};
>> > +
>> > +#define FPGA_FME_PORT_PR       _IO(FPGA_MAGIC, FME_BASE + 0)
>> > +
>> >  #endif /* _UAPI_INTEL_FPGA_H */
>> > --
>> > 2.7.4
>> >

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-03-31  8:50       ` Wu Hao
@ 2017-04-03 20:26         ` Alan Tull
  2017-04-04  5:25           ` Wu Hao
  0 siblings, 1 reply; 93+ messages in thread
From: Alan Tull @ 2017-04-03 20:26 UTC (permalink / raw)
  To: Wu Hao
  Cc: Moritz Fischer, linux-fpga, linux-kernel, luwei.kang, yi.z.zhang,
	Tim Whisonant, Enno Luebbers, Shiva Rao, Christopher Rauer,
	Xiao Guangrong, xiaoguangrong.eric

On Fri, Mar 31, 2017 at 3:50 AM, Wu Hao <hao.wu@intel.com> wrote:
> On Fri, Mar 31, 2017 at 12:11:12PM +0800, Xiao Guangrong wrote:
>> On 31/03/2017 4:30 AM, Alan Tull wrote:
>> >On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote:
>> >>From: Kang Luwei <luwei.kang@intel.com>
>> >>
>> >>Partial Reconfiguration (PR) is the most important function for FME. It
>> >>allows reconfiguration for given Port/Accelerated Function Unit (AFU).
>> >>
>> >>This patch adds support for PR sub feature. In this patch, it registers
>> >>a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
>> >>for PR operation once PR request received via ioctl. Below user space
>> >>interfaces are exposed by this sub feature.
>> >>
>> >>Sysfs interface:
>> >>* /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
>> >>  Read-only. Indicate the hardware interface information. Userspace
>> >>  applications need to check this interface to select correct green
>> >>  bitstream format before PR.
>> >>
>> >>Ioctl interface:
>> >>* FPGA_FME_PORT_PR
>> >>  Do partial reconfiguration per information from userspace, including
>> >>  target port(AFU), buffer size and address info. It returns the PR status
>> >>  (PR error code if failed) to userspace.
>> >>
>> >>Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
>> >>Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
>> >>Signed-off-by: Shiva Rao <shiva.rao@intel.com>
>> >>Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
>> >>Signed-off-by: Alan Tull <alan.tull@intel.com>
>> >
>> >Hi Wu Hao,
>> >
>> >Thanks for submitting your patches.
>> >
>> >I think there's been a misunderstanding of the meaning of
>> >'Signed-off-by' [1].  I have not signed off on this code or had a hand
>> >in its development.  But I'm happy to get to review it now.  It will
>> >take a bit of time; I expect to be replying next week.
>>
>> Hi Alan,
>>
>> Sorry to confuse you, i think it's because you helped Chris a lot to
>> implement this interface and we'd like to include your credit as this
>> way. If you dislike, it will be dropped. :)
>>
>> Thanks for your review in advance.
>>
>
> Hi Alan,
>
> Sorry about this, we should ask you firstly before doing it this way.
> Let me know if you don't like it, I will drop it in the next version.

Yes please drop the signed-off-by: me in the next version.  Also, you
don't need to cc my Intel email address.

Alan

>
> Many thanks for your time and review on these patches. Look forward
> for your feedback and comments. :)
>
> Thanks
> Hao

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview
  2017-04-02 14:41         ` Moritz Fischer
@ 2017-04-03 20:44           ` Alan Tull
  2017-04-04  5:24             ` Wu Hao
  2017-04-04  5:06           ` Wu Hao
  2017-04-11 18:02           ` Alan Tull
  2 siblings, 1 reply; 93+ messages in thread
From: Alan Tull @ 2017-04-03 20:44 UTC (permalink / raw)
  To: Moritz Fischer
  Cc: Wu Hao, matthew.gerlach, Moritz Fischer, linux-fpga,
	linux-kernel, luwei.kang, yi.z.zhang, Enno Luebbers,
	Xiao Guangrong

On Sun, Apr 2, 2017 at 9:41 AM, Moritz Fischer <mdf@kernel.org> wrote:
> On Sat, Apr 01, 2017 at 07:16:19PM +0800, Wu Hao wrote:
>> On Fri, Mar 31, 2017 at 01:38:06PM -0500, Alan Tull wrote:
>> > On Fri, Mar 31, 2017 at 1:24 PM,  <matthew.gerlach@linux.intel.com> wrote:
>> > >
>> > >
>> > > On Thu, 30 Mar 2017, Wu Hao wrote:
>> > >
>> > >
>> > > Hi Wu Hao,
>> > >
>> > > Great documentation. I'm looking forward to diving into the rest of the
>> > > patches. Please see my comments inline.
>> > >
>> > > Matthew Gerlach
>> > >
>> > >
>> > >> Add a document for Intel FPGA driver overview.
>> > >>
>> > >> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
>> > >> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> > >> Signed-off-by: Wu Hao <hao.wu@intel.com>
>> > >> ---
>> > >> Documentation/fpga/intel-fpga.txt | 259
>> > >> ++++++++++++++++++++++++++++++++++++++
>> > >> 1 file changed, 259 insertions(+)
>> > >> create mode 100644 Documentation/fpga/intel-fpga.txt
>> > >>
>> > >> diff --git a/Documentation/fpga/intel-fpga.txt
>> > >> b/Documentation/fpga/intel-fpga.txt
>> > >> new file mode 100644
>> > >> index 0000000..9396cea
>> > >> --- /dev/null
>> > >> +++ b/Documentation/fpga/intel-fpga.txt
>> > >> @@ -0,0 +1,259 @@
>> > >>
>> > >> +===============================================================================
>> > >> +                    Intel FPGA driver Overview
>> > >>
>> > >> +-------------------------------------------------------------------------------
>> > >> +                Enno Luebbers <enno.luebbers@intel.com>
>> > >> +                Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> > >> +                Wu Hao <hao.wu@intel.com>
>> > >> +
>> > >> +The Intel FPGA driver provides interfaces for userspace applications to
>> > >> +configure, enumerate, open, and access FPGA accelerators on platforms
>> > >> equipped
>> > >> +with Intel(R) FPGA solutions and enables system level management
>> > >> functions such
>> > >> +as FPGA reconfiguration, power management, and virtualization.
>> > >> +
>> > >
>> > >
>> > > From a Linux kernel perspective, I'm not sure this is the best name for
>> > > this code.  The name gives me the impression that it is a driver for all
>> > > Intel FPGAs, but not all Intel FPGAs are connected to the processor over a
>> > > PCIe bus.  The processor could be directely connected like the Arria10
>> > > SOCFPGA.  Such a processor could certainly benefit from this accelerator
>> > > usage model.  In an extreme case, couldn't a processor in the FPGA,
>> > > running Linux, also benefit from this accelerator model?  Is this code a
>> > > "FPGA Accelerator Framework"?
>> > >
>> > >> +HW Architecture
>> > >> +===============
>> > >> +From the OS's point of view, the FPGA hardware appears as a regular PCIe
>> > >> device.
>> > >> +The FPGA device memory is organized using a predefined data structure
>> > >> (Device
>> > >> +Feature List). Features supported by the particular FPGA device are
>> > >> exposed
>> > >> +through these data structures, as illustrated below:
>> > >> +
>> > >> +  +-------------------------------+  +-------------+
>> > >> +  |              PF               |  |     VF      |
>> > >> +  +-------------------------------+  +-------------+
>> > >> +      ^            ^         ^              ^
>> > >> +      |            |         |              |
>> > >> ++-----|------------|---------|--------------|-------+
>> > >> +|     |            |         |              |       |
>> > >> +|  +-----+     +-------+ +-------+      +-------+   |
>> > >> +|  | FME |     | Port0 | | Port1 |      | Port2 |   |
>> > >> +|  +-----+     +-------+ +-------+      +-------+   |
>> > >> +|                  ^         ^              ^       |
>> > >> +|                  |         |              |       |
>> > >> +|              +-------+ +------+       +-------+   |
>> > >> +|              |  AFU  | |  AFU |       |  AFU  |   |
>> > >> +|              +-------+ +------+       +-------+   |
>> > >> +|                                                   |
>> > >> +|                 FPGA PCIe Device                  |
>> > >> ++---------------------------------------------------+
>> > >> +
>> > >> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which
>> > >> can be
>> > >> +used to assign individual accelerators to virtual machines .
>> > >
>> > >
>> > > Does this HW Architecture require an Intel FPGA?  Couldn't any vendors FPGA
>> > > be used as long as it presented itself the PCIe bus the same and contained
>> > > an appropriate Device Feature List?
>
> I think this is a good (and important) point. Especially when sysfs
> entries & ioctls constituting ABI depend on it.
>
>> > >
>> > >> +
>> > >> +FME (FPGA Management Engine)
>> > >> +============================
>> > >> +The FPGA Management Enging performs power and thermal management, error
> Enging->Engine
>> > >> +reporting, reconfiguration, performance reporting, and other
>> > >> infrastructure
>> > >> +functions. Each FPGA has one FME, which is always accessed through the
>> > >> physical
>> > >> +function (PF).
>> > >> +
>> > >> +User-space applications can acquire exclusive access to the FME using
>> > >> open(),
>> > >> +and release it using close().
>> > >> +
>> > >> +The following functions are exposed through ioctls:
>> > >> +
>> > >> +       Get driver API version (FPGA_GET_API_VERSION)
>> > >> +       Check for extensions (FPGA_CHECK_EXTENSION)
>> > >> +       Assign port to PF (FPGA_FME_PORT_ASSIGN)
>> > >> +       Release port from PF (FPGA_FME_PORT_RELEASE)
>> > >> +       Program bitstream (FPGA_FME_PORT_PR)
>> > >> +
>> > >> +More functions are exposed through sysfs
>> > >> +(/sys/class/fpga/fpga.n/intel-fpga-fme.n/):
>> > >> +
>> > >> +       Read bitstream ID (bitstream_id)
>> > >> +       Read bitstream metadata (bitstream_metadata)
>> > >> +       Read number of ports (ports_num)
>> > >> +       Read socket ID (socket_id)
>> > >> +       Read performance counters (perf/)
>> > >> +       Power management (power_mgmt/)
>> > >> +       Thermal management (thermal_mgmt/)
>> > >> +       Error reporting (errors/)
>> > >> +
>> > >> +PORT
>> > >> +====
>> > >> +A port represents the interface between the static FPGA fabric (the "blue
>> > >> +bitstream") and a partially reconfigurable region containing an AFU (the
>> > >> "green
>> >
>> > Is this an fpga bridge but with added features?
>>
>> Yes, I think so. As you see the fme_pr function in patch 11, related port needs
>> to be disabled firstly before fpga_mgr_buf_load for given accelerator.
>
> Can we just extend the bridge to have the additional features, please?

OK then this code is taking place of a fpga-region that controls the
bridge (port) and fpga-mgr during fpga programming.

>
>> > >> +bitstream"). It controls the communication from SW to the accelerator and
>> > >> +exposes features such as reset and debug.
>> > >> +
>> > >> +A PCIe device may have several ports and each port can be released from
>> > >> PF by
>> > >> +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe
>> > >> sriov
>> > >> +sysfs interface.
>> > >> +
>> > >> +AFU
>> > >> +===
>> > >> +An AFU is attached to a port and exposes a 256k MMIO region to be used
>> > >> for
>> > >> +accelerator-specific control registers.
>> > >> +
>> > >> +User-space applications can acquire exclusive access to an AFU attached
>> > >> to a
>> > >> +port by using open() on the port device node, and release it using
>> > >> close().
>> > >> +
>> > >> +The following functions are exposed through ioctls:
>> > >> +
>> > >> +       Get driver API version (FPGA_GET_API_VERSION)
>> > >> +       Check for extensions (FPGA_CHECK_EXTENSION)
>> > >> +       Get port info (FPGA_PORT_GET_INFO)
>> > >> +       Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
>> > >> +       Map DMA buffer (FPGA_PORT_DMA_MAP)
>> > >> +       Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
>> > >> +       Reset AFU (FPGA_PORT_RESET)
>> > >> +       Enable UMsg (FPGA_PORT_UMSG_ENABLE)
>> > >> +       Disable UMsg (FPGA_PORT_UMSG_DISABLE)
>> > >> +       Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
>> > >> +       Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)
>> > >> +
>> > >> +User-space applications can also mmap() accelerator MMIO regions.
>> > >> +
>> > >> +More functions are exposed through sysfs:
>> > >> +(/sys/class/fpga/fpga.n/intel-fpga-port.m/):
>> > >> +
>> > >> +       Read Accelerator GUID (afu_id)
>> > >> +       Error reporting (errors/)
>> > >> +
>> > >> +Partial Reconfiguration
>> > >> +=======================
>> > >> +As mentioned above, accelerators can be reconfigured through partial
>> > >> +reconfiguration of a green bitstream file (GBS). The green bitstream must
>> > >> have
>> > >> +been generated for the exact blue bitstream and targeted reconfigurable
>> > >> region
>> > >> +(port) of the FPGA; otherwise, the reconfiguration operation will fail
>> > >> and
>> > >> +possibly cause system instability. This compatibility can be checked by
>> > >> +comparing the interface ID noted in the GBS header against the interface
>> > >> ID
>> > >> +exposed by the FME through sysfs (see above). This check is usually done
>> > >> by
>> > >> +user-space before calling the reconfiguration IOCTL.
>> > >> +
>> > >> +FPGA virtualization
>> > >> +===================
>> > >> +To enable accessing an accelerator from applications running in a VM, the
>> > >> +respective AFU's port needs to be assigned to a VF using the following
>> > >> steps:
>> > >> +
>> > >> + a) The PF owns all AFU ports by default. Any port that needs to be
>> > >> reassigned
>> > >> + to a VF must be released from PF firstly through the
>> > >> FPGA_FME_PORT_RELEASE
>> > >> + ioctl on the FME device.
>> > >> +
>> > >> + b) Once N ports are released from PF, then user can use below command to
>> > >> + enable SRIOV and VFs. Each VF owns only one Port with AFU.
>> > >> +
>> > >> + echo N > $PCI_DEVICE_PATH/sriov_numvfs
>> > >> +
>> > >> + c) Pass through the VFs to VMs
>> > >> +
>> > >> + d) The AFU under VF is accessiable from applications in VM (using the
>> > >> same
>> > >> + driver inside the VF).
>> > >> +
>> > >> +Note the an FME can't be assigned to a VF, thus PR and other management
>> > >> +functions are only available via the PF.
>> > >> +
>> > >> +
>> > >> +Driver organization
>> > >> +===================
>> > >> +
>> > >> +  +------------------+  +---------+   |             +---------+
>> > >> +  | +-------+        |  |         |   |             |         |
>> > >> +  | | FPGA  |  FME   |  |   AFU   |   |             |   AFU   |
>> > >> +  | |Manager| Module |  |  Module |   |             |  Module |
>> > >> +  | +-------+        |  |         |   |             |         |
>> > >> +  +------------------+  +---------+   |             +---------+
>> > >> +        +-----------------------+     |      +-----------------------+
>> > >> +        | FPGA Container Device |     |      | FPGA Container Device |
>> > >> +        +-----------------------+     |      +-----------------------+
>> > >> +          +------------------+        |         +------------------+
>> > >> +          | FPGA PCIE Module |        | Virtual | FPGA PCIE Module |
>> > >> +          +------------------+   Host | Machine +------------------+
>> > >> + ------------------------------------ | ------------------------------
>> > >> +           +---------------+          |          +---------------+
>> > >> +           | PCI PF Device |          |          | PCI VF Device |
>> > >> +           +---------------+          |          +---------------+
>> > >> +
>> > >> +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe
>> > >> device
>> > >> +driver is always loaded first once a FPGA PCIE PF or VF device is
>> > >> detected. This
>> > >> +driver plays an infrastructural role in the driver architecuture.  It:
>> > >> +
>> > >> +       a) creates FPGA container device as parent of the feature devices.
>> > >> +       b) walks through the Device Feature List, which is implemented in
>> > >> PCIE
>> > >> +          device BAR memory, to discover feature devices and their sub
>> > >> features
>> > >> +          and create platform device for them under the container device.
>> > >
>> > >
>> > > I really like the idea of creating platform devices for the sub features. It
>> > > is in line with other FPGA use cases.  Platform devices are at the heart of
>> > > device trees used by processors directly connected FPGAs and processors
>> > > inside FPGAs.
>> > >
>> > >> +       c) supports SRIOV.
>> > >> +       d) introduces the feature device infrastructure, which abstracts
>> > >> +          operations for sub features and exposes common functions to
>> > >> feature
>> > >> +          device drivers.
>> > >> +
>> > >> +The FPGA Management Engine (FME) driver is a platform driver which is
>> > >> loaded
>> > >> +automatically after FME platform device creation from the PCIE driver. It
>> > >> +provides the key features for FPGA management, including:
>> > >> +
>> > >> +       a) Power and thermal management, error reporting, performance
>> > >> reporting
>> > >> +          and other infrastructure functions. Users can access these
>> > >> functions
>> > >> +          via sysfs interfaces exposed by FME driver.
>> > >> +       b) Paritial Reconfiguration. The FME driver registers a FPGA
>> > >> Manager
>> > >> +          during PR sub feature initialization; once it receives an
>> > >> +          FPGA_FME_PORT_PR ioctl from user, it invokes the common
>> > >> interface
>> > >> +          function from FPGA Manager to complete the partial
>> > >> reconfiguration of
>> > >> +          the bitstream to the given port.
>> > >> +       c) Port management for virtualization. The FME driver introduces
>> > >> two
>> > >> +          ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
>> > >> +          FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the
>> > >> port is
>> > >> +          released from the PF, it can be assigned to the VF through the
>> > >> SRIOV
>> > >> +          interfaces provided by PCIE driver. (Refer to "FPGA
>> > >> virtualization"
>> > >> +          for more details).
>> > >> +
>> > >> +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU)
>> > >> driver
>> > >> +is probed once the AFU platform device is created. The main function of
>> > >> this
>> > >> +module is to provide an interface for userspace applications to access
>> > >> the
>> > >> +individual accelerators, including basic reset control on port, AFU MMIO
>> > >> region
>> > >> +export, dma buffer mapping service, UMsg notification, and remote debug
>> > >> +functions (see above).
>> > >> +
>> > >> +
>> > >> +Device enumeration
>> > >> +==================
>> > >> +This section introduces how applications enumerate the fpga device from
>> > >> +the sysfs hierarchy under /sys/class/fpga.
>> > >> +
>> > >> +In the example below, two Intel(R) FPGA devices are installed in the
>> > >> host. Each
>> > >> +fpga device has one FME and two ports (AFUs).
>> > >> +
>> > >> +For each FPGA device, a device director is created under
>> > >> /sys/class/fpga/:
>> > >> +
>> > >> +       /sys/class/fpga/fpga.0
>> > >> +       /sys/class/fpga/fpga.1
>> > >> +
>> > >> +The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's
>> > >> name.
>> > >> +Application can retrieve name information via the sysfs interface:
>> > >> +
>> > >> +       /sys/class/fpga/fpga.0/name
>> > >> +
>> > >> +Each node has one FME and two ports (AFUs) as child devices:
>> > >> +
>> > >> +       /sys/class/fpga/fpga.0/intel-fpga-fme.0
>> > >> +       /sys/class/fpga/fpga.0/intel-fpga-port.0
>> > >> +       /sys/class/fpga/fpga.0/intel-fpga-port.1
>> > >> +
>> > >> +       /sys/class/fpga/fpga.1/intel-fpga-fme.1
>> > >> +       /sys/class/fpga/fpga.1/intel-fpga-port.2
>> > >> +       /sys/class/fpga/fpga.1/intel-fpga-port.3
>> > >> +
>> > >> +In general, the FME/AFU sysfs interfaces are named as follows:
>> > >> +
>> > >> +       /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/
>> > >> +       /sys/class/fpga/<fpga.n>/<intel-fpga-port.m>/
>> > >> +
>> > >> +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering
>> > >> all
>> > >> +ports.
>> > >> +
>> > >> +The device nodes used for ioctl() or mmap() can be referenced through:
>> > >> +
>> > >> +       /sys/class/fpga/<fpga.n>/<intel-fpga-port.n>/dev
>> > >> +       /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/dev
>> > >> +
>> > >> +
>> > >> +Open discussions
>> > >> +================
>> > >> +The current FME driver does not provide user space access to the FME MMIO
>> > >> +region, but exposes access through sysfs and ioctls. It also provides an
>> > >> FPGA
>> > >> +manger interface for partial reconfiguration (PR), but does not make use
>> > >> of
>> > >> +fpga-regions. User PR requests via the FPGA_FME_PORT_PR ioctl are handled
>> > >> inside
>> > >> +the FME, and fpga-region depends on device tree which is not used at all.
>> > >> There
>> > >> +are patches from Alan Tull to separate the device tree specific code and
>> > >
>> > >
>> > > I am currently trying to use those patches in a different driver.  They've
>> > > compiled cleanly in my out of tree pcie module driver against the 3.10
>> > > kernel.
>> > > I need to actually write the code to create and register the region, but
>> > > Alan's platform driver code should be a good guide for me.  Just need to
>> > > find the time.
>> > >
>> > >> +introduce a sysfs interface for PR. We plan to add fpga-regions support
>> > >> in the
>> > >> +driver once the related patches get merged. Then the FME driver should
>> > >> create
>> > >> +one fpga-region for each Port/AFU.
>> > >
>> > >
>> > > Does the FME driver create the fpga-region, or is each region described as
>> > > an entry in the Device Feature List and therefore created by the code that
>> > > enumerates the Device Feature List?
>> > >
>> > >> --
>> > >> 2.7.4
>> > >>
>> > >> --
>> > >> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
>> > >> the body of a message to majordomo@vger.kernel.org
>> > >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> > >>
>> > >
>
> Cheers,
> Moritz

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-03-30 12:08 ` [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support Wu Hao
                     ` (3 preceding siblings ...)
  2017-04-01  1:12   ` kbuild test robot
@ 2017-04-03 21:24   ` Alan Tull
  2017-04-03 22:49     ` matthew.gerlach
  2017-04-04  6:28     ` Wu Hao
  4 siblings, 2 replies; 93+ messages in thread
From: Alan Tull @ 2017-04-03 21:24 UTC (permalink / raw)
  To: Wu Hao
  Cc: Moritz Fischer, linux-fpga, linux-kernel, luwei.kang, yi.z.zhang,
	Tim Whisonant, Enno Luebbers, Shiva Rao, Christopher Rauer,
	Alan Tull, Xiao Guangrong

On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote:
> From: Kang Luwei <luwei.kang@intel.com>
>
> Partial Reconfiguration (PR) is the most important function for FME. It
> allows reconfiguration for given Port/Accelerated Function Unit (AFU).
>
> This patch adds support for PR sub feature. In this patch, it registers
> a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
> for PR operation once PR request received via ioctl.

The code that invokes fpga_mgr_buf_load should be a different layer.

> Below user space
> interfaces are exposed by this sub feature.
>
> Sysfs interface:
> * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
>   Read-only. Indicate the hardware interface information. Userspace
>   applications need to check this interface to select correct green
>   bitstream format before PR.
>
> Ioctl interface:
> * FPGA_FME_PORT_PR
>   Do partial reconfiguration per information from userspace, including
>   target port(AFU), buffer size and address info. It returns the PR status
>   (PR error code if failed) to userspace.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Alan Tull <alan.tull@intel.com>
> Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ---
>  drivers/fpga/intel/Makefile      |   2 +-
>  drivers/fpga/intel/feature-dev.h |  58 ++++++
>  drivers/fpga/intel/fme-main.c    |  44 ++++-
>  drivers/fpga/intel/fme-pr.c      | 400 +++++++++++++++++++++++++++++++++++++++
>  drivers/fpga/intel/fme.h         |  32 ++++
>  include/uapi/linux/intel-fpga.h  |  44 +++++
>  6 files changed, 578 insertions(+), 2 deletions(-)
>  create mode 100644 drivers/fpga/intel/fme-pr.c
>  create mode 100644 drivers/fpga/intel/fme.h
>
> diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
> index 546861d..0452cb6 100644
> --- a/drivers/fpga/intel/Makefile
> +++ b/drivers/fpga/intel/Makefile
> @@ -2,4 +2,4 @@ obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
>  obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
>
>  intel-fpga-pci-objs := pcie.o feature-dev.o
> -intel-fpga-fme-objs := fme-main.o
> +intel-fpga-fme-objs := fme-main.o fme-pr.o
> diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
> index dccc283..5a25c915 100644
> --- a/drivers/fpga/intel/feature-dev.h
> +++ b/drivers/fpga/intel/feature-dev.h
> @@ -150,8 +150,66 @@ struct feature_fme_err {
>  };
>
>  /* FME Partial Reconfiguration Sub Feature Register Set */
> +/* FME PR Control Register */
> +struct feature_fme_pr_ctl {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u8  pr_reset:1;         /* Reset PR Engine */
> +                       u8  rsvdz1:3;
> +                       u8  pr_reset_ack:1;     /* Reset PR Engine Ack */
> +                       u8  rsvdz2:3;
> +                       u8  pr_regionid:2;      /* PR Region ID */
> +                       u8  rsvdz3:2;
> +                       u8  pr_start_req:1;     /* PR Start Request */
> +                       u8  pr_push_complete:1; /* PR Data push complete */
> +                       u8  pr_kind:1;          /* Load Customer or Intel GBS */
> +                       u32 rsvdz4:17;
> +                       u32 config_data;
> +               };
> +       };
> +};
> +
> +/* FME PR Status Register */
> +struct feature_fme_pr_status {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u16 pr_credit:9;        /* Number of PR Credits */
> +                       u8  rsvdz1:7;
> +                       u8  pr_status:1;        /* PR Operation status */
> +                       u8  rsvdz2:3;
> +                       u8  pr_ctrlr_status:3;  /* Controller status */
> +                       u8  rsvdz3:1;
> +                       u8  pr_host_status:4;   /* PR Host status */
> +                       u64 rsvdz4:36;
> +               };
> +       };
> +};
> +
> +/* FME PR Data Register */
> +struct feature_fme_pr_data {
> +       union {
> +               u64 csr;
> +               struct {
> +                       /* PR data from the raw-binary file */
> +                       u32 pr_data_raw;
> +                       u32 rsvd;
> +               };
> +       };
> +};
> +
>  struct feature_fme_pr {
>         struct feature_header header;
> +       struct feature_fme_pr_ctl control;
> +       struct feature_fme_pr_status status;
> +       struct feature_fme_pr_data data;
> +       u64 error;
> +
> +       u64 rsvd[16];
> +
> +       u64 intfc_id_l;         /* PR interface Id Low */
> +       u64 intfc_id_h;         /* PR interface Id High */
>  };
>
>  /* PORT Header Register Set */
> diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
> index 36d0c4c..0d9a7a6 100644
> --- a/drivers/fpga/intel/fme-main.c
> +++ b/drivers/fpga/intel/fme-main.c
> @@ -23,6 +23,7 @@
>  #include <linux/intel-fpga.h>
>
>  #include "feature-dev.h"
> +#include "fme.h"
>
>  static ssize_t ports_num_show(struct device *dev,
>                               struct device_attribute *attr, char *buf)
> @@ -101,6 +102,10 @@ static struct feature_driver fme_feature_drvs[] = {
>                 .ops = &fme_hdr_ops,
>         },
>         {
> +               .name = FME_FEATURE_PR_MGMT,
> +               .ops = &pr_mgmt_ops,
> +       },
> +       {
>                 .ops = NULL,
>         },
>  };
> @@ -182,14 +187,48 @@ static const struct file_operations fme_fops = {
>         .unlocked_ioctl = fme_ioctl,
>  };
>
> +static int fme_dev_init(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct fpga_fme *fme;
> +
> +       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
> +       if (!fme)
> +               return -ENOMEM;
> +
> +       fme->pdata = pdata;
> +
> +       mutex_lock(&pdata->lock);
> +       fpga_pdata_set_private(pdata, fme);
> +       mutex_unlock(&pdata->lock);
> +       return 0;
> +}
> +
> +static void fme_dev_destroy(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct fpga_fme *fme;
> +
> +       mutex_lock(&pdata->lock);
> +       fme = fpga_pdata_get_private(pdata);
> +       fpga_pdata_set_private(pdata, NULL);
> +       mutex_unlock(&pdata->lock);
> +
> +       devm_kfree(&pdev->dev, fme);
> +}
> +
>  static int fme_probe(struct platform_device *pdev)
>  {
>         int ret;
>
> -       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> +       ret = fme_dev_init(pdev);
>         if (ret)
>                 goto exit;
>
> +       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> +       if (ret)
> +               goto dev_destroy;
> +
>         ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
>         if (ret)
>                 goto feature_uinit;
> @@ -198,6 +237,8 @@ static int fme_probe(struct platform_device *pdev)
>
>  feature_uinit:
>         fpga_dev_feature_uinit(pdev);
> +dev_destroy:
> +       fme_dev_destroy(pdev);
>  exit:
>         return ret;
>  }
> @@ -206,6 +247,7 @@ static int fme_remove(struct platform_device *pdev)
>  {
>         fpga_dev_feature_uinit(pdev);
>         fpga_unregister_dev_ops(pdev);
> +       fme_dev_destroy(pdev);
>         return 0;
>  }
>
> diff --git a/drivers/fpga/intel/fme-pr.c b/drivers/fpga/intel/fme-pr.c
> new file mode 100644
> index 0000000..3b44a3e
> --- /dev/null
> +++ b/drivers/fpga/intel/fme-pr.c
> @@ -0,0 +1,400 @@
> +/*
> + * Driver for Intel FPGA Management Engine (FME) Partial Reconfiguration
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <luwei.kang@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *   Joseph Grecco <joe.grecco@intel.com>
> + *   Enno Luebbers <enno.luebbers@intel.com>
> + *   Tim Whisonant <tim.whisonant@intel.com>
> + *   Ananda Ravuri <ananda.ravuri@intel.com>
> + *   Christopher Rauer <christopher.rauer@intel.com>
> + *   Henry Mitchel <henry.mitchel@intel.com>
> + *
> + * This work is licensed under a dual BSD/GPLv2 license. When using or
> + * redistributing this file, you may do so under either license. See the
> + * LICENSE.BSD file under this directory for the BSD license and see
> + * the COPYING file in the top-level directory for the GPLv2 license.
> + */
> +
> +#include <linux/types.h>
> +#include <linux/device.h>
> +#include <linux/vmalloc.h>
> +#include <linux/uaccess.h>
> +#include <linux/fpga/fpga-mgr.h>
> +#include <linux/intel-fpga.h>
> +
> +#include "feature-dev.h"
> +#include "fme.h"
> +
> +#define PR_WAIT_TIMEOUT   8000000
> +
> +#define PR_HOST_STATUS_IDLE    0
> +
> +DEFINE_FPGA_PR_ERR_MSG(pr_err_msg);
> +
> +static ssize_t interface_id_show(struct device *dev,
> +                                struct device_attribute *attr, char *buf)
> +{
> +       u64 intfc_id_l, intfc_id_h;
> +       struct feature_fme_pr *fme_pr
> +               = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_PR_MGMT);
> +
> +       intfc_id_l = readq(&fme_pr->intfc_id_l);
> +       intfc_id_h = readq(&fme_pr->intfc_id_h);
> +
> +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> +                       (unsigned long long)intfc_id_h,
> +                       (unsigned long long)intfc_id_l);
> +}
> +static DEVICE_ATTR_RO(interface_id);
> +
> +static struct attribute *pr_mgmt_attrs[] = {
> +       &dev_attr_interface_id.attr,
> +       NULL,
> +};
> +
> +struct attribute_group pr_mgmt_attr_group = {
> +       .attrs  = pr_mgmt_attrs,
> +       .name   = "pr",
> +};
> +
> +static u64
> +pr_err_handle(struct platform_device *pdev, struct feature_fme_pr *fme_pr)
> +{
> +       struct feature_fme_pr_status fme_pr_status;
> +       unsigned long err_code;
> +       u64 fme_pr_error;
> +       int i = 0;
> +
> +       fme_pr_status.csr = readq(&fme_pr->status);
> +       if (!fme_pr_status.pr_status)
> +               return 0;
> +
> +       err_code = fme_pr_error = readq(&fme_pr->error);
> +       for_each_set_bit(i, &err_code, PR_MAX_ERR_NUM)
> +               dev_dbg(&pdev->dev, "%s\n", pr_err_msg[i]);
> +       writeq(fme_pr_error, &fme_pr->error);
> +       return fme_pr_error;
> +}
> +
> +static int fme_pr_write_init(struct fpga_manager *mgr,
> +               struct fpga_image_info *info, const char *buf, size_t count)
> +{
> +       struct fpga_fme *fme = mgr->priv;
> +       struct platform_device *pdev;
> +       struct feature_fme_pr *fme_pr;
> +       struct feature_fme_pr_ctl fme_pr_ctl;
> +       struct feature_fme_pr_status fme_pr_status;
> +
> +       pdev = fme->pdata->dev;
> +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> +                               FME_FEATURE_ID_PR_MGMT);
> +       if (!fme_pr)
> +               return -EINVAL;
> +
> +       if (WARN_ON(info->flags != FPGA_MGR_PARTIAL_RECONFIG))
> +               return -EINVAL;

flags is bitmapped so please do:

if (WARN_ON(info->flags & FPGA_MGR_PARTIAL_RECONFIG))

> +
> +       dev_dbg(&pdev->dev, "resetting PR before initiated PR\n");
> +
> +       fme_pr_ctl.csr = readq(&fme_pr->control);
> +       fme_pr_ctl.pr_reset = 1;
> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> +
> +       fme_pr_ctl.pr_reset_ack = 1;
> +
> +       if (fpga_wait_register_field(pr_reset_ack, fme_pr_ctl,
> +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> +               dev_err(&pdev->dev, "maximum PR timeout\n");
> +               return -ETIMEDOUT;
> +       }
> +
> +       fme_pr_ctl.csr = readq(&fme_pr->control);
> +       fme_pr_ctl.pr_reset = 0;
> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> +
> +       dev_dbg(&pdev->dev,
> +               "waiting for PR resource in HW to be initialized and ready\n");
> +
> +       fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
> +
> +       if (fpga_wait_register_field(pr_host_status, fme_pr_status,
> +                                    &fme_pr->status, PR_WAIT_TIMEOUT, 1)) {
> +               dev_err(&pdev->dev, "maximum PR timeout\n");
> +               return -ETIMEDOUT;
> +       }
> +
> +       dev_dbg(&pdev->dev, "check if have any previous PR error\n");
> +       pr_err_handle(pdev, fme_pr);
> +       return 0;
> +}
> +
> +static int fme_pr_write(struct fpga_manager *mgr,
> +                       const char *buf, size_t count)
> +{
> +       struct fpga_fme *fme = mgr->priv;
> +       struct platform_device *pdev;
> +       struct feature_fme_pr *fme_pr;
> +       struct feature_fme_pr_ctl fme_pr_ctl;
> +       struct feature_fme_pr_status fme_pr_status;
> +       struct feature_fme_pr_data fme_pr_data;
> +       int delay, pr_credit, i = 0;
> +
> +       pdev = fme->pdata->dev;
> +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> +                               FME_FEATURE_ID_PR_MGMT);
> +
> +       dev_dbg(&pdev->dev, "set PR port ID and start request\n");
> +
> +       fme_pr_ctl.csr = readq(&fme_pr->control);
> +       fme_pr_ctl.pr_regionid = fme->port_id;
> +       fme_pr_ctl.pr_start_req = 1;
> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> +
> +       dev_dbg(&pdev->dev, "pushing data from bitstream to HW\n");
> +
> +       fme_pr_status.csr = readq(&fme_pr->status);
> +       pr_credit = fme_pr_status.pr_credit;
> +
> +       while (count > 0) {
> +               delay = 0;
> +               while (pr_credit <= 1) {
> +                       if (delay++ > PR_WAIT_TIMEOUT) {
> +                               dev_err(&pdev->dev, "maximum try\n");
> +                               return -ETIMEDOUT;
> +                       }
> +                       udelay(1);
> +
> +                       fme_pr_status.csr = readq(&fme_pr->status);
> +                       pr_credit = fme_pr_status.pr_credit;
> +               };
> +
> +               if (count >= 4) {
> +                       fme_pr_data.rsvd = 0;
> +                       fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
> +                       writeq(fme_pr_data.csr, &fme_pr->data);
> +                       count -= 4;
> +                       pr_credit--;
> +                       i++;
> +               } else {
> +                       WARN_ON(1);
> +                       return -EINVAL;
> +               }
> +       }
> +
> +       return 0;
> +}
> +
> +static int fme_pr_write_complete(struct fpga_manager *mgr,
> +                       struct fpga_image_info *info)
> +{
> +       struct fpga_fme *fme = mgr->priv;
> +       struct platform_device *pdev;
> +       struct feature_fme_pr *fme_pr;
> +       struct feature_fme_pr_ctl fme_pr_ctl;
> +
> +       pdev = fme->pdata->dev;
> +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> +                               FME_FEATURE_ID_PR_MGMT);
> +
> +       fme_pr_ctl.csr = readq(&fme_pr->control);
> +       fme_pr_ctl.pr_push_complete = 1;
> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> +
> +       dev_dbg(&pdev->dev, "green bitstream push complete\n");
> +       dev_dbg(&pdev->dev, "waiting for HW to release PR resource\n");
> +
> +       fme_pr_ctl.pr_start_req = 0;
> +
> +       if (fpga_wait_register_field(pr_start_req, fme_pr_ctl,
> +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> +               dev_err(&pdev->dev, "maximum try.\n");
> +               return -ETIMEDOUT;
> +       }
> +
> +       dev_dbg(&pdev->dev, "PR operation complete, checking status\n");
> +       fme->pr_err = pr_err_handle(pdev, fme_pr);
> +       if (fme->pr_err)
> +               return -EIO;
> +
> +       dev_dbg(&pdev->dev, "PR done successfully\n");
> +       return 0;
> +}
> +
> +static enum fpga_mgr_states fme_pr_state(struct fpga_manager *mgr)
> +{
> +       return FPGA_MGR_STATE_UNKNOWN;
> +}
> +
> +static const struct fpga_manager_ops fme_pr_ops = {
> +       .write_init = fme_pr_write_init,
> +       .write = fme_pr_write,
> +       .write_complete = fme_pr_write_complete,
> +       .state = fme_pr_state,
> +};
> +

The following fme_pr() function shouldn't be in a fpga-mgr driver.  It
is calling
the framework that this driver is registered with.
A fpga-mgr low level driver is supposed to be a very low level driver.
This function is controlling the port and calling fpga-mgr layer to do the
programming.  There should be a layer above the fpga-mgr and fpga-bridge
that coordinates these things.  I.e. can you use my proposed fpga-region code?

> +static int fme_pr(struct platform_device *pdev, unsigned long arg)
> +{
> +       void __user *argp = (void __user *)arg;
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct fpga_fme *fme;
> +       struct fpga_manager *mgr;
> +       struct feature_fme_header *fme_hdr;
> +       struct feature_fme_capability fme_capability;
> +       struct fpga_image_info info;
> +       struct fpga_fme_port_pr port_pr;
> +       struct platform_device *port;
> +       unsigned long minsz;
> +       void *buf = NULL;
> +       int ret = 0;
> +
> +       minsz = offsetofend(struct fpga_fme_port_pr, status);
> +
> +       if (copy_from_user(&port_pr, argp, minsz))
> +               return -EFAULT;
> +
> +       if (port_pr.argsz < minsz || port_pr.flags)
> +               return -EINVAL;
> +
> +       if (!IS_ALIGNED(port_pr.buffer_size, 4))
> +               return -EINVAL;
> +
> +       /* get fme header region */
> +       fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> +                                       FME_FEATURE_ID_HEADER);
> +       if (WARN_ON(!fme_hdr))
> +               return -EINVAL;
> +
> +       /* check port id */
> +       fme_capability.csr = readq(&fme_hdr->capability);
> +       if (port_pr.port_id >= fme_capability.num_ports) {
> +               dev_dbg(&pdev->dev, "port number more than maximum\n");
> +               return -EINVAL;
> +       }
> +
> +       if (!access_ok(VERIFY_READ, port_pr.buffer_address,
> +                                   port_pr.buffer_size))
> +               return -EFAULT;
> +
> +       buf = vmalloc(port_pr.buffer_size);
> +       if (!buf)
> +               return -ENOMEM;
> +
> +       if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
> +                                              port_pr.buffer_size)) {
> +               ret = -EFAULT;
> +               goto free_exit;
> +       }
> +
> +       memset(&info, 0, sizeof(struct fpga_image_info));
> +       info.flags = FPGA_MGR_PARTIAL_RECONFIG;
> +
> +       mgr = fpga_mgr_get(&pdev->dev);
> +       if (IS_ERR(mgr)) {
> +               ret = PTR_ERR(mgr);
> +               goto free_exit;
> +       }
> +
> +       mutex_lock(&pdata->lock);
> +       fme = fpga_pdata_get_private(pdata);
> +       /* fme device has been unregistered. */
> +       if (!fme) {
> +               ret = -EINVAL;
> +               goto unlock_exit;
> +       }
> +
> +       fme->pr_err = 0;
> +       fme->port_id = port_pr.port_id;
> +
> +       /* Find and get port device by index */
> +       port = pdata->fpga_for_each_port(pdev, &fme->port_id,
> +                                        fpga_port_check_id);
> +       WARN_ON(!port);
> +
> +       /* Disable Port before PR */
> +       fpga_port_disable(port);
> +
> +       ret = fpga_mgr_buf_load(mgr, &info, buf, port_pr.buffer_size);
> +       port_pr.status = fme->pr_err;
> +
> +       /* Re-enable Port after PR finished */
> +       fpga_port_enable(port);
> +
> +       put_device(&port->dev);
> +
> +unlock_exit:
> +       mutex_unlock(&pdata->lock);
> +       fpga_mgr_put(mgr);
> +free_exit:
> +       vfree(buf);
> +       if (copy_to_user((void __user *)arg, &port_pr, minsz))
> +               return -EFAULT;
> +       return ret;
> +}
> +
> +static int fpga_fme_pr_probe(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct fpga_fme *priv;
> +       int ret;
> +
> +       mutex_lock(&pdata->lock);
> +       priv = fpga_pdata_get_private(pdata);
> +       ret = fpga_mgr_register(&pdata->dev->dev,
> +               "Intel FPGA Manager", &fme_pr_ops, priv);
> +       mutex_unlock(&pdata->lock);
> +
> +       return ret;
> +}
> +
> +static int fpga_fme_pr_remove(struct platform_device *pdev)
> +{
> +       fpga_mgr_unregister(&pdev->dev);
> +       return 0;
> +}
> +
> +static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
> +{
> +       int ret;
> +
> +       ret = fpga_fme_pr_probe(pdev);
> +       if (ret)
> +               return ret;
> +
> +       ret = sysfs_create_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
> +       if (ret)
> +               fpga_fme_pr_remove(pdev);
> +
> +       return ret;
> +}
> +
> +static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
> +{
> +       sysfs_remove_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
> +       fpga_fme_pr_remove(pdev);
> +}
> +
> +static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
> +       unsigned int cmd, unsigned long arg)
> +{
> +       long ret;
> +
> +       switch (cmd) {
> +       case FPGA_FME_PORT_PR:
> +               ret = fme_pr(pdev, arg);
> +               break;
> +       default:
> +               ret = -ENODEV;
> +       }
> +
> +       return ret;
> +}
> +
> +struct feature_ops pr_mgmt_ops = {
> +       .init = pr_mgmt_init,
> +       .uinit = pr_mgmt_uinit,
> +       .ioctl = fme_pr_ioctl,
> +};
> diff --git a/drivers/fpga/intel/fme.h b/drivers/fpga/intel/fme.h
> new file mode 100644
> index 0000000..d6cb7ce
> --- /dev/null
> +++ b/drivers/fpga/intel/fme.h
> @@ -0,0 +1,32 @@
> +/*
> + * Header file for Intel FPGA Management Engine (FME) Driver
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <luwei.kang@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *   Joseph Grecco <joe.grecco@intel.com>
> + *   Enno Luebbers <enno.luebbers@intel.com>
> + *   Tim Whisonant <tim.whisonant@intel.com>
> + *   Ananda Ravuri <ananda.ravuri@intel.com>
> + *   Henry Mitchel <henry.mitchel@intel.com>
> + *
> + * This work is licensed under a dual BSD/GPLv2 license. When using or
> + * redistributing this file, you may do so under either license. See the
> + * LICENSE.BSD file under this directory for the BSD license and see
> + * the COPYING file in the top-level directory for the GPLv2 license.
> + */
> +
> +#ifndef __INTEL_FME_H
> +#define __INTEL_FME_H
> +
> +struct fpga_fme {
> +       u8  port_id;
> +       u64 pr_err;
> +       struct feature_platform_data *pdata;
> +};
> +
> +extern struct feature_ops pr_mgmt_ops;
> +
> +#endif
> diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
> index 992e556..77658316 100644
> --- a/include/uapi/linux/intel-fpga.h
> +++ b/include/uapi/linux/intel-fpga.h
> @@ -18,6 +18,8 @@
>  #ifndef _UAPI_LINUX_INTEL_FPGA_H
>  #define _UAPI_LINUX_INTEL_FPGA_H
>
> +#include <linux/types.h>
> +
>  #define FPGA_API_VERSION 0
>
>  /*
> @@ -30,6 +32,7 @@
>  #define FPGA_MAGIC 0xB6
>
>  #define FPGA_BASE 0
> +#define FME_BASE 0x80
>
>  /**
>   * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
> @@ -49,4 +52,45 @@
>
>  #define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)
>
> +/* IOCTLs for FME file descriptor */
> +
> +/**
> + * FPGA_FME_PORT_PR - _IOWR(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
> + *
> + * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
> + * provided by caller.
> + * Return: 0 on success, -errno on failure.
> + * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
> + * some errors during PR, under this case, the user can fetch HW error code
> + * from fpga_fme_port_pr.status. Each bit on the error code is used as the
> + * index for the array created by DEFINE_FPGA_PR_ERR_MSG().
> + * Otherwise, it is always zero.
> + */
> +
> +#define DEFINE_FPGA_PR_ERR_MSG(_name_)                 \
> +static const char * const _name_[] = {                 \
> +       "PR operation error detected",                  \
> +       "PR CRC error detected",                        \
> +       "PR incompatiable bitstream error detected",    \
> +       "PR IP protocol error detected",                \
> +       "PR FIFO overflow error detected",              \
> +       "Reserved",                                     \
> +       "PR secure load error detected",                \
> +}
> +
> +#define PR_MAX_ERR_NUM 7
> +
> +struct fpga_fme_port_pr {
> +       /* Input */
> +       __u32 argsz;            /* Structure length */
> +       __u32 flags;            /* Zero for now */
> +       __u32 port_id;
> +       __u32 buffer_size;
> +       __u64 buffer_address;   /* Userspace address to the buffer for PR */
> +       /* Output */
> +       __u64 status;           /* HW error code if ioctl returns -EIO */
> +};
> +
> +#define FPGA_FME_PORT_PR       _IO(FPGA_MAGIC, FME_BASE + 0)
> +
>  #endif /* _UAPI_INTEL_FPGA_H */
> --
> 2.7.4
>

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features.
  2017-03-30 12:08 ` [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features Wu Hao
@ 2017-04-03 21:44   ` Alan Tull
  2017-04-05 11:58     ` Wu Hao
  2017-04-04  2:44   ` Moritz Fischer
                     ` (2 subsequent siblings)
  3 siblings, 1 reply; 93+ messages in thread
From: Alan Tull @ 2017-04-03 21:44 UTC (permalink / raw)
  To: Wu Hao
  Cc: Moritz Fischer, linux-fpga, linux-kernel, luwei.kang, yi.z.zhang,
	Xiao Guangrong, Tim Whisonant, Enno Luebbers, Shiva Rao,
	Christopher Rauer

On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote:
> From: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>
> Device Featuer List structure creates a link list of feature headers
> within the MMIO space to provide an extensiable way of adding features.
>
> The Intel FPGA PCIe driver walks through the feature headers to enumerate
> feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
> Function Unit (AFU), and their private sub features. For feature devices,
> it creates the platform devices and linked the private sub features into
> their platform data.
>
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ---
>  drivers/fpga/intel/Makefile      |   2 +-
>  drivers/fpga/intel/feature-dev.c | 139 +++++++
>  drivers/fpga/intel/feature-dev.h | 342 ++++++++++++++++
>  drivers/fpga/intel/pcie.c        | 841 ++++++++++++++++++++++++++++++++++++++-
>  4 files changed, 1321 insertions(+), 3 deletions(-)
>  create mode 100644 drivers/fpga/intel/feature-dev.c
>  create mode 100644 drivers/fpga/intel/feature-dev.h
>
> diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
> index 61fd8ea..c029940 100644
> --- a/drivers/fpga/intel/Makefile
> +++ b/drivers/fpga/intel/Makefile
> @@ -1,3 +1,3 @@
>  obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
>
> -intel-fpga-pci-objs := pcie.o
> +intel-fpga-pci-objs := pcie.o feature-dev.o
> diff --git a/drivers/fpga/intel/feature-dev.c b/drivers/fpga/intel/feature-dev.c
> new file mode 100644
> index 0000000..6952566
> --- /dev/null
> +++ b/drivers/fpga/intel/feature-dev.c
> @@ -0,0 +1,139 @@
> +/*
> + * Intel FPGA Feature Device Driver
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <luwei.kang@intel.com>
> + *   Zhang Yi <yi.z.zhang@intel.com>
> + *   Wu Hao <hao.wu@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *
> + * This work is licensed under a dual BSD/GPLv2 license. When using or
> + * redistributing this file, you may do so under either license. See the
> + * LICENSE.BSD file under this directory for the BSD license and see
> + * the COPYING file in the top-level directory for the GPLv2 license.
> + */
> +
> +#include "feature-dev.h"
> +
> +void feature_platform_data_add(struct feature_platform_data *pdata,
> +                              int index, const char *name,
> +                              int resource_index, void __iomem *ioaddr)
> +{
> +       WARN_ON(index >= pdata->num);
> +
> +       pdata->features[index].name = name;
> +       pdata->features[index].resource_index = resource_index;
> +       pdata->features[index].ioaddr = ioaddr;
> +}
> +
> +int feature_platform_data_size(int num)
> +{
> +       return sizeof(struct feature_platform_data) +
> +               num * sizeof(struct feature);
> +}
> +
> +struct feature_platform_data *
> +feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
> +{
> +       struct feature_platform_data *pdata;
> +
> +       pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
> +       if (pdata) {
> +               pdata->dev = dev;
> +               pdata->num = num;
> +               mutex_init(&pdata->lock);
> +       }
> +
> +       return pdata;
> +}
> +
> +int fme_feature_num(void)
> +{
> +       return FME_FEATURE_ID_MAX;
> +}
> +
> +int port_feature_num(void)
> +{
> +       return PORT_FEATURE_ID_MAX;
> +}
> +
> +int fpga_port_id(struct platform_device *pdev)
> +{
> +       struct feature_port_header *port_hdr;
> +       struct feature_port_capability capability;
> +
> +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> +                                              PORT_FEATURE_ID_HEADER);
> +       WARN_ON(!port_hdr);
> +
> +       capability.csr = readq(&port_hdr->capability);
> +       return capability.port_number;
> +}
> +EXPORT_SYMBOL_GPL(fpga_port_id);
> +
> +/*
> + * Enable Port by clear the port soft reset bit, which is set by default.
> + * The User AFU is unable to respond to any MMIO access while in reset.
> + * __fpga_port_enable function should only be used after __fpga_port_disable
> + * function.
> + */
> +void __fpga_port_enable(struct platform_device *pdev)
> +{

feature-dev.c is handling enumeration and adding port
enable/disable/etc functions for a specific port device.  I see the
port as a fpga-bridge.  The enumeration code should be separate from
the bridge code.  Especially separate from a very specific bridge low
level device driver implementation, otherwise this becomes obsolete as
soon as you have another port device with a different register
implementation.  Even if you handle that, then this enumeration code
isn't useable by other people who are using fpga-bridge.  The
fpga-bridge framework exists to separate low level things like how to
enable/disable a specific bridge device from upper level code that
knows when to enable/disable it (fpga-region).

Alan

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-04-03 21:24   ` Alan Tull
@ 2017-04-03 22:49     ` matthew.gerlach
  2017-04-04  6:48       ` Wu Hao
  2017-04-04  6:28     ` Wu Hao
  1 sibling, 1 reply; 93+ messages in thread
From: matthew.gerlach @ 2017-04-03 22:49 UTC (permalink / raw)
  To: Alan Tull
  Cc: Wu Hao, Moritz Fischer, linux-fpga, linux-kernel, luwei.kang,
	yi.z.zhang, Tim Whisonant, Enno Luebbers, Shiva Rao,
	Christopher Rauer, Alan Tull, Xiao Guangrong



On Mon, 3 Apr 2017, Alan Tull wrote:

> On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote:
>> From: Kang Luwei <luwei.kang@intel.com>
>>
>> Partial Reconfiguration (PR) is the most important function for FME. It
>> allows reconfiguration for given Port/Accelerated Function Unit (AFU).
>>
>> This patch adds support for PR sub feature. In this patch, it registers
>> a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
>> for PR operation once PR request received via ioctl.
>
> The code that invokes fpga_mgr_buf_load should be a different layer.
>
>> Below user space
>> interfaces are exposed by this sub feature.
>>
>> Sysfs interface:
>> * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
>>   Read-only. Indicate the hardware interface information. Userspace
>>   applications need to check this interface to select correct green
>>   bitstream format before PR.
>>
>> Ioctl interface:
>> * FPGA_FME_PORT_PR
>>   Do partial reconfiguration per information from userspace, including
>>   target port(AFU), buffer size and address info. It returns the PR status
>>   (PR error code if failed) to userspace.
>>
>> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
>> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
>> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
>> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
>> Signed-off-by: Alan Tull <alan.tull@intel.com>
>> Signed-off-by: Kang Luwei <luwei.kang@intel.com>
>> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> Signed-off-by: Wu Hao <hao.wu@intel.com>
>> ---
>>  drivers/fpga/intel/Makefile      |   2 +-
>>  drivers/fpga/intel/feature-dev.h |  58 ++++++
>>  drivers/fpga/intel/fme-main.c    |  44 ++++-
>>  drivers/fpga/intel/fme-pr.c      | 400 +++++++++++++++++++++++++++++++++++++++
>>  drivers/fpga/intel/fme.h         |  32 ++++
>>  include/uapi/linux/intel-fpga.h  |  44 +++++
>>  6 files changed, 578 insertions(+), 2 deletions(-)
>>  create mode 100644 drivers/fpga/intel/fme-pr.c
>>  create mode 100644 drivers/fpga/intel/fme.h
>>
>> diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
>> index 546861d..0452cb6 100644
>> --- a/drivers/fpga/intel/Makefile
>> +++ b/drivers/fpga/intel/Makefile
>> @@ -2,4 +2,4 @@ obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
>>  obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
>>
>>  intel-fpga-pci-objs := pcie.o feature-dev.o
>> -intel-fpga-fme-objs := fme-main.o
>> +intel-fpga-fme-objs := fme-main.o fme-pr.o
>> diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
>> index dccc283..5a25c915 100644
>> --- a/drivers/fpga/intel/feature-dev.h
>> +++ b/drivers/fpga/intel/feature-dev.h
>> @@ -150,8 +150,66 @@ struct feature_fme_err {
>>  };
>>
>>  /* FME Partial Reconfiguration Sub Feature Register Set */
>> +/* FME PR Control Register */
>> +struct feature_fme_pr_ctl {
>> +       union {
>> +               u64 csr;
>> +               struct {
>> +                       u8  pr_reset:1;         /* Reset PR Engine */
>> +                       u8  rsvdz1:3;
>> +                       u8  pr_reset_ack:1;     /* Reset PR Engine Ack */
>> +                       u8  rsvdz2:3;
>> +                       u8  pr_regionid:2;      /* PR Region ID */
>> +                       u8  rsvdz3:2;
>> +                       u8  pr_start_req:1;     /* PR Start Request */
>> +                       u8  pr_push_complete:1; /* PR Data push complete */
>> +                       u8  pr_kind:1;          /* Load Customer or Intel GBS */
>> +                       u32 rsvdz4:17;
>> +                       u32 config_data;
>> +               };
>> +       };
>> +};
>> +
>> +/* FME PR Status Register */
>> +struct feature_fme_pr_status {
>> +       union {
>> +               u64 csr;
>> +               struct {
>> +                       u16 pr_credit:9;        /* Number of PR Credits */
>> +                       u8  rsvdz1:7;
>> +                       u8  pr_status:1;        /* PR Operation status */
>> +                       u8  rsvdz2:3;
>> +                       u8  pr_ctrlr_status:3;  /* Controller status */
>> +                       u8  rsvdz3:1;
>> +                       u8  pr_host_status:4;   /* PR Host status */
>> +                       u64 rsvdz4:36;
>> +               };
>> +       };
>> +};
>> +
>> +/* FME PR Data Register */
>> +struct feature_fme_pr_data {
>> +       union {
>> +               u64 csr;
>> +               struct {
>> +                       /* PR data from the raw-binary file */
>> +                       u32 pr_data_raw;
>> +                       u32 rsvd;
>> +               };
>> +       };
>> +};
>> +
>>  struct feature_fme_pr {
>>         struct feature_header header;
>> +       struct feature_fme_pr_ctl control;
>> +       struct feature_fme_pr_status status;
>> +       struct feature_fme_pr_data data;
>> +       u64 error;
>> +
>> +       u64 rsvd[16];
>> +
>> +       u64 intfc_id_l;         /* PR interface Id Low */
>> +       u64 intfc_id_h;         /* PR interface Id High */
>>  };
>>
>>  /* PORT Header Register Set */
>> diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
>> index 36d0c4c..0d9a7a6 100644
>> --- a/drivers/fpga/intel/fme-main.c
>> +++ b/drivers/fpga/intel/fme-main.c
>> @@ -23,6 +23,7 @@
>>  #include <linux/intel-fpga.h>
>>
>>  #include "feature-dev.h"
>> +#include "fme.h"
>>
>>  static ssize_t ports_num_show(struct device *dev,
>>                               struct device_attribute *attr, char *buf)
>> @@ -101,6 +102,10 @@ static struct feature_driver fme_feature_drvs[] = {
>>                 .ops = &fme_hdr_ops,
>>         },
>>         {
>> +               .name = FME_FEATURE_PR_MGMT,
>> +               .ops = &pr_mgmt_ops,
>> +       },
>> +       {
>>                 .ops = NULL,
>>         },
>>  };
>> @@ -182,14 +187,48 @@ static const struct file_operations fme_fops = {
>>         .unlocked_ioctl = fme_ioctl,
>>  };
>>
>> +static int fme_dev_init(struct platform_device *pdev)
>> +{
>> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> +       struct fpga_fme *fme;
>> +
>> +       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
>> +       if (!fme)
>> +               return -ENOMEM;
>> +
>> +       fme->pdata = pdata;
>> +
>> +       mutex_lock(&pdata->lock);
>> +       fpga_pdata_set_private(pdata, fme);
>> +       mutex_unlock(&pdata->lock);
>> +       return 0;
>> +}
>> +
>> +static void fme_dev_destroy(struct platform_device *pdev)
>> +{
>> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> +       struct fpga_fme *fme;
>> +
>> +       mutex_lock(&pdata->lock);
>> +       fme = fpga_pdata_get_private(pdata);
>> +       fpga_pdata_set_private(pdata, NULL);
>> +       mutex_unlock(&pdata->lock);
>> +
>> +       devm_kfree(&pdev->dev, fme);
>> +}
>> +
>>  static int fme_probe(struct platform_device *pdev)
>>  {
>>         int ret;
>>
>> -       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
>> +       ret = fme_dev_init(pdev);
>>         if (ret)
>>                 goto exit;
>>
>> +       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
>> +       if (ret)
>> +               goto dev_destroy;
>> +
>>         ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
>>         if (ret)
>>                 goto feature_uinit;
>> @@ -198,6 +237,8 @@ static int fme_probe(struct platform_device *pdev)
>>
>>  feature_uinit:
>>         fpga_dev_feature_uinit(pdev);
>> +dev_destroy:
>> +       fme_dev_destroy(pdev);
>>  exit:
>>         return ret;
>>  }
>> @@ -206,6 +247,7 @@ static int fme_remove(struct platform_device *pdev)
>>  {
>>         fpga_dev_feature_uinit(pdev);
>>         fpga_unregister_dev_ops(pdev);
>> +       fme_dev_destroy(pdev);
>>         return 0;
>>  }
>>
>> diff --git a/drivers/fpga/intel/fme-pr.c b/drivers/fpga/intel/fme-pr.c
>> new file mode 100644
>> index 0000000..3b44a3e
>> --- /dev/null
>> +++ b/drivers/fpga/intel/fme-pr.c
>> @@ -0,0 +1,400 @@
>> +/*
>> + * Driver for Intel FPGA Management Engine (FME) Partial Reconfiguration
>> + *
>> + * Copyright (C) 2017 Intel Corporation, Inc.
>> + *
>> + * Authors:
>> + *   Kang Luwei <luwei.kang@intel.com>
>> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> + *   Joseph Grecco <joe.grecco@intel.com>
>> + *   Enno Luebbers <enno.luebbers@intel.com>
>> + *   Tim Whisonant <tim.whisonant@intel.com>
>> + *   Ananda Ravuri <ananda.ravuri@intel.com>
>> + *   Christopher Rauer <christopher.rauer@intel.com>
>> + *   Henry Mitchel <henry.mitchel@intel.com>
>> + *
>> + * This work is licensed under a dual BSD/GPLv2 license. When using or
>> + * redistributing this file, you may do so under either license. See the
>> + * LICENSE.BSD file under this directory for the BSD license and see
>> + * the COPYING file in the top-level directory for the GPLv2 license.
>> + */
>> +
>> +#include <linux/types.h>
>> +#include <linux/device.h>
>> +#include <linux/vmalloc.h>
>> +#include <linux/uaccess.h>
>> +#include <linux/fpga/fpga-mgr.h>
>> +#include <linux/intel-fpga.h>
>> +
>> +#include "feature-dev.h"
>> +#include "fme.h"
>> +
>> +#define PR_WAIT_TIMEOUT   8000000
>> +
>> +#define PR_HOST_STATUS_IDLE    0
>> +
>> +DEFINE_FPGA_PR_ERR_MSG(pr_err_msg);
>> +
>> +static ssize_t interface_id_show(struct device *dev,
>> +                                struct device_attribute *attr, char *buf)
>> +{
>> +       u64 intfc_id_l, intfc_id_h;
>> +       struct feature_fme_pr *fme_pr
>> +               = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_PR_MGMT);
>> +
>> +       intfc_id_l = readq(&fme_pr->intfc_id_l);
>> +       intfc_id_h = readq(&fme_pr->intfc_id_h);
>> +
>> +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
>> +                       (unsigned long long)intfc_id_h,
>> +                       (unsigned long long)intfc_id_l);
>> +}
>> +static DEVICE_ATTR_RO(interface_id);
>> +
>> +static struct attribute *pr_mgmt_attrs[] = {
>> +       &dev_attr_interface_id.attr,
>> +       NULL,
>> +};
>> +
>> +struct attribute_group pr_mgmt_attr_group = {
>> +       .attrs  = pr_mgmt_attrs,
>> +       .name   = "pr",
>> +};
>> +
>> +static u64
>> +pr_err_handle(struct platform_device *pdev, struct feature_fme_pr *fme_pr)
>> +{
>> +       struct feature_fme_pr_status fme_pr_status;
>> +       unsigned long err_code;
>> +       u64 fme_pr_error;
>> +       int i = 0;
>> +
>> +       fme_pr_status.csr = readq(&fme_pr->status);
>> +       if (!fme_pr_status.pr_status)
>> +               return 0;
>> +
>> +       err_code = fme_pr_error = readq(&fme_pr->error);
>> +       for_each_set_bit(i, &err_code, PR_MAX_ERR_NUM)
>> +               dev_dbg(&pdev->dev, "%s\n", pr_err_msg[i]);
>> +       writeq(fme_pr_error, &fme_pr->error);
>> +       return fme_pr_error;
>> +}
>> +
>> +static int fme_pr_write_init(struct fpga_manager *mgr,
>> +               struct fpga_image_info *info, const char *buf, size_t count)
>> +{
>> +       struct fpga_fme *fme = mgr->priv;
>> +       struct platform_device *pdev;
>> +       struct feature_fme_pr *fme_pr;
>> +       struct feature_fme_pr_ctl fme_pr_ctl;
>> +       struct feature_fme_pr_status fme_pr_status;
>> +
>> +       pdev = fme->pdata->dev;
>> +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
>> +                               FME_FEATURE_ID_PR_MGMT);
>> +       if (!fme_pr)
>> +               return -EINVAL;
>> +
>> +       if (WARN_ON(info->flags != FPGA_MGR_PARTIAL_RECONFIG))
>> +               return -EINVAL;
>
> flags is bitmapped so please do:
>
> if (WARN_ON(info->flags & FPGA_MGR_PARTIAL_RECONFIG))
>
>> +
>> +       dev_dbg(&pdev->dev, "resetting PR before initiated PR\n");
>> +
>> +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> +       fme_pr_ctl.pr_reset = 1;
>> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> +
>> +       fme_pr_ctl.pr_reset_ack = 1;
>> +
>> +       if (fpga_wait_register_field(pr_reset_ack, fme_pr_ctl,
>> +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
>> +               dev_err(&pdev->dev, "maximum PR timeout\n");
>> +               return -ETIMEDOUT;
>> +       }
>> +
>> +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> +       fme_pr_ctl.pr_reset = 0;
>> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> +
>> +       dev_dbg(&pdev->dev,
>> +               "waiting for PR resource in HW to be initialized and ready\n");
>> +
>> +       fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
>> +
>> +       if (fpga_wait_register_field(pr_host_status, fme_pr_status,
>> +                                    &fme_pr->status, PR_WAIT_TIMEOUT, 1)) {
>> +               dev_err(&pdev->dev, "maximum PR timeout\n");
>> +               return -ETIMEDOUT;
>> +       }
>> +
>> +       dev_dbg(&pdev->dev, "check if have any previous PR error\n");
>> +       pr_err_handle(pdev, fme_pr);
>> +       return 0;
>> +}
>> +
>> +static int fme_pr_write(struct fpga_manager *mgr,
>> +                       const char *buf, size_t count)
>> +{
>> +       struct fpga_fme *fme = mgr->priv;
>> +       struct platform_device *pdev;
>> +       struct feature_fme_pr *fme_pr;
>> +       struct feature_fme_pr_ctl fme_pr_ctl;
>> +       struct feature_fme_pr_status fme_pr_status;
>> +       struct feature_fme_pr_data fme_pr_data;
>> +       int delay, pr_credit, i = 0;
>> +
>> +       pdev = fme->pdata->dev;
>> +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
>> +                               FME_FEATURE_ID_PR_MGMT);
>> +
>> +       dev_dbg(&pdev->dev, "set PR port ID and start request\n");
>> +
>> +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> +       fme_pr_ctl.pr_regionid = fme->port_id;
>> +       fme_pr_ctl.pr_start_req = 1;
>> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> +
>> +       dev_dbg(&pdev->dev, "pushing data from bitstream to HW\n");
>> +
>> +       fme_pr_status.csr = readq(&fme_pr->status);
>> +       pr_credit = fme_pr_status.pr_credit;
>> +
>> +       while (count > 0) {
>> +               delay = 0;
>> +               while (pr_credit <= 1) {
>> +                       if (delay++ > PR_WAIT_TIMEOUT) {
>> +                               dev_err(&pdev->dev, "maximum try\n");
>> +                               return -ETIMEDOUT;
>> +                       }
>> +                       udelay(1);
>> +
>> +                       fme_pr_status.csr = readq(&fme_pr->status);
>> +                       pr_credit = fme_pr_status.pr_credit;
>> +               };
>> +
>> +               if (count >= 4) {
>> +                       fme_pr_data.rsvd = 0;
>> +                       fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
>> +                       writeq(fme_pr_data.csr, &fme_pr->data);
>> +                       count -= 4;
>> +                       pr_credit--;
>> +                       i++;
>> +               } else {
>> +                       WARN_ON(1);
>> +                       return -EINVAL;
>> +               }
>> +       }
>> +
>> +       return 0;
>> +}
>> +
>> +static int fme_pr_write_complete(struct fpga_manager *mgr,
>> +                       struct fpga_image_info *info)
>> +{
>> +       struct fpga_fme *fme = mgr->priv;
>> +       struct platform_device *pdev;
>> +       struct feature_fme_pr *fme_pr;
>> +       struct feature_fme_pr_ctl fme_pr_ctl;
>> +
>> +       pdev = fme->pdata->dev;
>> +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
>> +                               FME_FEATURE_ID_PR_MGMT);
>> +
>> +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> +       fme_pr_ctl.pr_push_complete = 1;
>> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> +
>> +       dev_dbg(&pdev->dev, "green bitstream push complete\n");
>> +       dev_dbg(&pdev->dev, "waiting for HW to release PR resource\n");
>> +
>> +       fme_pr_ctl.pr_start_req = 0;
>> +
>> +       if (fpga_wait_register_field(pr_start_req, fme_pr_ctl,
>> +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
>> +               dev_err(&pdev->dev, "maximum try.\n");
>> +               return -ETIMEDOUT;
>> +       }
>> +
>> +       dev_dbg(&pdev->dev, "PR operation complete, checking status\n");
>> +       fme->pr_err = pr_err_handle(pdev, fme_pr);
>> +       if (fme->pr_err)
>> +               return -EIO;
>> +
>> +       dev_dbg(&pdev->dev, "PR done successfully\n");
>> +       return 0;
>> +}
>> +
>> +static enum fpga_mgr_states fme_pr_state(struct fpga_manager *mgr)
>> +{
>> +       return FPGA_MGR_STATE_UNKNOWN;
>> +}

The functions that implement the fme_pr_ops are really a low level fpga
manager driver for the Altera PR IP component.  A standalone version of
such a driver has been reviewed and Acked.  See the links below.
Could this file use those functions and remove this code?

http://marc.info/?l=linux-kernel&m=149019678925564&w=2
http://marc.info/?l=linux-kernel&m=149019654225457&w=2
http://marc.info/?l=linux-kernel&m=149019598025274&w=2
http://marc.info/?l=linux-kernel&m=149013051007149&w=2

>> +
>> +static const struct fpga_manager_ops fme_pr_ops = {
>> +       .write_init = fme_pr_write_init,
>> +       .write = fme_pr_write,
>> +       .write_complete = fme_pr_write_complete,
>> +       .state = fme_pr_state,
>> +};
>> +
>
> The following fme_pr() function shouldn't be in a fpga-mgr driver.  It
> is calling
> the framework that this driver is registered with.
> A fpga-mgr low level driver is supposed to be a very low level driver.
> This function is controlling the port and calling fpga-mgr layer to do the
> programming.  There should be a layer above the fpga-mgr and fpga-bridge
> that coordinates these things.  I.e. can you use my proposed fpga-region code?
>
>> +static int fme_pr(struct platform_device *pdev, unsigned long arg)
>> +{
>> +       void __user *argp = (void __user *)arg;
>> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> +       struct fpga_fme *fme;
>> +       struct fpga_manager *mgr;
>> +       struct feature_fme_header *fme_hdr;
>> +       struct feature_fme_capability fme_capability;
>> +       struct fpga_image_info info;
>> +       struct fpga_fme_port_pr port_pr;
>> +       struct platform_device *port;
>> +       unsigned long minsz;
>> +       void *buf = NULL;
>> +       int ret = 0;
>> +
>> +       minsz = offsetofend(struct fpga_fme_port_pr, status);
>> +
>> +       if (copy_from_user(&port_pr, argp, minsz))
>> +               return -EFAULT;
>> +
>> +       if (port_pr.argsz < minsz || port_pr.flags)
>> +               return -EINVAL;
>> +
>> +       if (!IS_ALIGNED(port_pr.buffer_size, 4))
>> +               return -EINVAL;
>> +
>> +       /* get fme header region */
>> +       fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
>> +                                       FME_FEATURE_ID_HEADER);
>> +       if (WARN_ON(!fme_hdr))
>> +               return -EINVAL;
>> +
>> +       /* check port id */
>> +       fme_capability.csr = readq(&fme_hdr->capability);
>> +       if (port_pr.port_id >= fme_capability.num_ports) {
>> +               dev_dbg(&pdev->dev, "port number more than maximum\n");
>> +               return -EINVAL;
>> +       }
>> +
>> +       if (!access_ok(VERIFY_READ, port_pr.buffer_address,
>> +                                   port_pr.buffer_size))
>> +               return -EFAULT;
>> +
>> +       buf = vmalloc(port_pr.buffer_size);
>> +       if (!buf)
>> +               return -ENOMEM;
>> +
>> +       if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
>> +                                              port_pr.buffer_size)) {
>> +               ret = -EFAULT;
>> +               goto free_exit;
>> +       }
>> +
>> +       memset(&info, 0, sizeof(struct fpga_image_info));
>> +       info.flags = FPGA_MGR_PARTIAL_RECONFIG;
>> +
>> +       mgr = fpga_mgr_get(&pdev->dev);
>> +       if (IS_ERR(mgr)) {
>> +               ret = PTR_ERR(mgr);
>> +               goto free_exit;
>> +       }
>> +
>> +       mutex_lock(&pdata->lock);
>> +       fme = fpga_pdata_get_private(pdata);
>> +       /* fme device has been unregistered. */
>> +       if (!fme) {
>> +               ret = -EINVAL;
>> +               goto unlock_exit;
>> +       }
>> +
>> +       fme->pr_err = 0;
>> +       fme->port_id = port_pr.port_id;
>> +
>> +       /* Find and get port device by index */
>> +       port = pdata->fpga_for_each_port(pdev, &fme->port_id,
>> +                                        fpga_port_check_id);
>> +       WARN_ON(!port);
>> +
>> +       /* Disable Port before PR */
>> +       fpga_port_disable(port);
>> +
>> +       ret = fpga_mgr_buf_load(mgr, &info, buf, port_pr.buffer_size);
>> +       port_pr.status = fme->pr_err;
>> +
>> +       /* Re-enable Port after PR finished */
>> +       fpga_port_enable(port);
>> +
>> +       put_device(&port->dev);
>> +
>> +unlock_exit:
>> +       mutex_unlock(&pdata->lock);
>> +       fpga_mgr_put(mgr);
>> +free_exit:
>> +       vfree(buf);
>> +       if (copy_to_user((void __user *)arg, &port_pr, minsz))
>> +               return -EFAULT;
>> +       return ret;
>> +}
>> +
>> +static int fpga_fme_pr_probe(struct platform_device *pdev)
>> +{
>> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> +       struct fpga_fme *priv;
>> +       int ret;
>> +
>> +       mutex_lock(&pdata->lock);
>> +       priv = fpga_pdata_get_private(pdata);
>> +       ret = fpga_mgr_register(&pdata->dev->dev,
>> +               "Intel FPGA Manager", &fme_pr_ops, priv);

The call to fpga_mgr_register() above would change to a call to
alt_pr_register() if the standalone version of the PR IP driver were used.

>> +       mutex_unlock(&pdata->lock);
>> +
>> +       return ret;
>> +}
>> +
>> +static int fpga_fme_pr_remove(struct platform_device *pdev)
>> +{
>> +       fpga_mgr_unregister(&pdev->dev);

This call to fpga_mgr_unregister() would change to a call to
alt_pr_unregister() if the standalone version of the PR IP driver were
used.

>> +       return 0;
>> +}
>> +
>> +static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
>> +{
>> +       int ret;
>> +
>> +       ret = fpga_fme_pr_probe(pdev);
>> +       if (ret)
>> +               return ret;
>> +
>> +       ret = sysfs_create_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
>> +       if (ret)
>> +               fpga_fme_pr_remove(pdev);
>> +
>> +       return ret;
>> +}
>> +
>> +static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
>> +{
>> +       sysfs_remove_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
>> +       fpga_fme_pr_remove(pdev);
>> +}
>> +
>> +static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
>> +       unsigned int cmd, unsigned long arg)
>> +{
>> +       long ret;
>> +
>> +       switch (cmd) {
>> +       case FPGA_FME_PORT_PR:
>> +               ret = fme_pr(pdev, arg);
>> +               break;
>> +       default:
>> +               ret = -ENODEV;
>> +       }
>> +
>> +       return ret;
>> +}
>> +
>> +struct feature_ops pr_mgmt_ops = {
>> +       .init = pr_mgmt_init,
>> +       .uinit = pr_mgmt_uinit,
>> +       .ioctl = fme_pr_ioctl,
>> +};
>> diff --git a/drivers/fpga/intel/fme.h b/drivers/fpga/intel/fme.h
>> new file mode 100644
>> index 0000000..d6cb7ce
>> --- /dev/null
>> +++ b/drivers/fpga/intel/fme.h
>> @@ -0,0 +1,32 @@
>> +/*
>> + * Header file for Intel FPGA Management Engine (FME) Driver
>> + *
>> + * Copyright (C) 2017 Intel Corporation, Inc.
>> + *
>> + * Authors:
>> + *   Kang Luwei <luwei.kang@intel.com>
>> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> + *   Joseph Grecco <joe.grecco@intel.com>
>> + *   Enno Luebbers <enno.luebbers@intel.com>
>> + *   Tim Whisonant <tim.whisonant@intel.com>
>> + *   Ananda Ravuri <ananda.ravuri@intel.com>
>> + *   Henry Mitchel <henry.mitchel@intel.com>
>> + *
>> + * This work is licensed under a dual BSD/GPLv2 license. When using or
>> + * redistributing this file, you may do so under either license. See the
>> + * LICENSE.BSD file under this directory for the BSD license and see
>> + * the COPYING file in the top-level directory for the GPLv2 license.
>> + */
>> +
>> +#ifndef __INTEL_FME_H
>> +#define __INTEL_FME_H
>> +
>> +struct fpga_fme {
>> +       u8  port_id;
>> +       u64 pr_err;
>> +       struct feature_platform_data *pdata;
>> +};
>> +
>> +extern struct feature_ops pr_mgmt_ops;
>> +
>> +#endif
>> diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
>> index 992e556..77658316 100644
>> --- a/include/uapi/linux/intel-fpga.h
>> +++ b/include/uapi/linux/intel-fpga.h
>> @@ -18,6 +18,8 @@
>>  #ifndef _UAPI_LINUX_INTEL_FPGA_H
>>  #define _UAPI_LINUX_INTEL_FPGA_H
>>
>> +#include <linux/types.h>
>> +
>>  #define FPGA_API_VERSION 0
>>
>>  /*
>> @@ -30,6 +32,7 @@
>>  #define FPGA_MAGIC 0xB6
>>
>>  #define FPGA_BASE 0
>> +#define FME_BASE 0x80
>>
>>  /**
>>   * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
>> @@ -49,4 +52,45 @@
>>
>>  #define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)
>>
>> +/* IOCTLs for FME file descriptor */
>> +
>> +/**
>> + * FPGA_FME_PORT_PR - _IOWR(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
>> + *
>> + * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
>> + * provided by caller.
>> + * Return: 0 on success, -errno on failure.
>> + * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
>> + * some errors during PR, under this case, the user can fetch HW error code
>> + * from fpga_fme_port_pr.status. Each bit on the error code is used as the
>> + * index for the array created by DEFINE_FPGA_PR_ERR_MSG().
>> + * Otherwise, it is always zero.
>> + */
>> +
>> +#define DEFINE_FPGA_PR_ERR_MSG(_name_)                 \
>> +static const char * const _name_[] = {                 \
>> +       "PR operation error detected",                  \
>> +       "PR CRC error detected",                        \
>> +       "PR incompatiable bitstream error detected",    \
>> +       "PR IP protocol error detected",                \
>> +       "PR FIFO overflow error detected",              \
>> +       "Reserved",                                     \
>> +       "PR secure load error detected",                \
>> +}
>> +
>> +#define PR_MAX_ERR_NUM 7
>> +
>> +struct fpga_fme_port_pr {
>> +       /* Input */
>> +       __u32 argsz;            /* Structure length */
>> +       __u32 flags;            /* Zero for now */
>> +       __u32 port_id;
>> +       __u32 buffer_size;
>> +       __u64 buffer_address;   /* Userspace address to the buffer for PR */
>> +       /* Output */
>> +       __u64 status;           /* HW error code if ioctl returns -EIO */
>> +};
>> +
>> +#define FPGA_FME_PORT_PR       _IO(FPGA_MAGIC, FME_BASE + 0)
>> +
>>  #endif /* _UAPI_INTEL_FPGA_H */
>> --
>> 2.7.4
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 03/16] fpga: intel: add FPGA PCIe device driver
  2017-03-30 12:08 ` [PATCH 03/16] fpga: intel: add FPGA PCIe device driver Wu Hao
@ 2017-04-04  2:10   ` Moritz Fischer
  2017-04-05 13:14     ` Wu, Hao
  0 siblings, 1 reply; 93+ messages in thread
From: Moritz Fischer @ 2017-04-04  2:10 UTC (permalink / raw)
  To: Wu Hao
  Cc: atull, moritz.fischer, linux-fpga, linux-kernel, luwei.kang,
	yi.z.zhang, Tim Whisonant, Enno Luebbers, Shiva Rao,
	Christopher Rauer, Xiao Guangrong

[-- Attachment #1: Type: text/plain, Size: 9777 bytes --]

On Thu, Mar 30, 2017 at 08:08:03PM +0800, Wu Hao wrote:
> From: Zhang Yi <yi.z.zhang@intel.com>
> 
> The Intel FPGA device appears as a PCIe device on the system. This patch
> implements the basic framework of the driver for Intel PCIe device which
> locates between CPU and Accelerated Function Units (AFUs).
> 
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ---
>  drivers/fpga/Kconfig           |   2 +
>  drivers/fpga/Makefile          |   3 +
>  drivers/fpga/intel/Kconfig     |  27 +++++++++
>  drivers/fpga/intel/LICENSE.BSD |  24 ++++++++
>  drivers/fpga/intel/Makefile    |   3 +
>  drivers/fpga/intel/pcie.c      | 129 +++++++++++++++++++++++++++++++++++++++++
>  6 files changed, 188 insertions(+)
>  create mode 100644 drivers/fpga/intel/Kconfig
>  create mode 100644 drivers/fpga/intel/LICENSE.BSD
>  create mode 100644 drivers/fpga/intel/Makefile
>  create mode 100644 drivers/fpga/intel/pcie.c
> 
> diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> index d99b640..4e49aee 100644
> --- a/drivers/fpga/Kconfig
> +++ b/drivers/fpga/Kconfig
> @@ -69,6 +69,8 @@ config ALTERA_FREEZE_BRIDGE
>  	  isolate one region of the FPGA from the busses while that
>  	  region is being reprogrammed.
>  
> +source "drivers/fpga/intel/Kconfig"
> +
>  endif # FPGA
>  
>  endmenu
> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> index 53a41d2..46f1a5d 100644
> --- a/drivers/fpga/Makefile
> +++ b/drivers/fpga/Makefile
> @@ -20,3 +20,6 @@ obj-$(CONFIG_ALTERA_FREEZE_BRIDGE)	+= altera-freeze-bridge.o
>  
>  # High Level Interfaces
>  obj-$(CONFIG_FPGA_REGION)		+= fpga-region.o
> +
> +# Intel FPGA Support
> +obj-$(CONFIG_INTEL_FPGA)		+= intel/
> diff --git a/drivers/fpga/intel/Kconfig b/drivers/fpga/intel/Kconfig
> new file mode 100644
> index 0000000..bf402f3
> --- /dev/null
> +++ b/drivers/fpga/intel/Kconfig
> @@ -0,0 +1,27 @@
> +menuconfig INTEL_FPGA
> +	tristate "Intel(R) FPGA support"
> +	depends on FPGA_DEVICE
> +	help
> +	  Select this option to enable driver support for Intel(R)
> +	  Field-Programmable Gate Array (FPGA) solutions. This driver
> +	  provides interfaces for userspace applications to configure,
> +	  enumerate, open, and access FPGA accelerators on platforms
> +	  equipped with Intel(R) FPGA solutions and enables system
> +	  level management functions such as FPGA reconfiguration,
> +	  power management, and virtualization.
> +
> +	  Say Y if your platform has this technology. Say N if unsure.
> +
> +if INTEL_FPGA
> +
> +config INTEL_FPGA_PCI
> +	tristate "Intel FPGA PCIe Driver"
> +	depends on PCI
> +	help
> +	  This is the driver for the PCIe device which locates between
> +	  CPU and Accelerated Function Units (AFUs) and allows them to
> +	  communicate with each other.
> +
> +	  To compile this as a module, choose M here.
> +
> +endif
> diff --git a/drivers/fpga/intel/LICENSE.BSD b/drivers/fpga/intel/LICENSE.BSD
> new file mode 100644
> index 0000000..309d2b7
> --- /dev/null
> +++ b/drivers/fpga/intel/LICENSE.BSD
> @@ -0,0 +1,24 @@
> +Redistribution and use in source and binary forms, with or without
> +modification, are permitted provided that the following conditions
> +are met:
> +  * Redistributions of source code must retain the above copyright
> +    notice, this list of conditions and the following disclaimer.
> +  * Redistributions in binary form must reproduce the above copyright
> +    notice, this list of conditions and the following disclaimer in
> +    the documentation and/or other materials provided with the
> +    distribution.
> +  * Neither the name of Intel Corporation nor the names of its
> +    contributors may be used to endorse or promote products derived
> +    from this software without specific prior written permission.
> +
> +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
> +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
> +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
> +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
> +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
> +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
> +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
> +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
> +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
> diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
> new file mode 100644
> index 0000000..61fd8ea
> --- /dev/null
> +++ b/drivers/fpga/intel/Makefile
> @@ -0,0 +1,3 @@
> +obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
> +
> +intel-fpga-pci-objs := pcie.o
> diff --git a/drivers/fpga/intel/pcie.c b/drivers/fpga/intel/pcie.c
> new file mode 100644
> index 0000000..132d9da
> --- /dev/null
> +++ b/drivers/fpga/intel/pcie.c
> @@ -0,0 +1,129 @@
> +/*
> + * Driver for Intel FPGA PCIe device
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Zhang Yi <Yi.Z.Zhang@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *   Joseph Grecco <joe.grecco@intel.com>
> + *   Enno Luebbers <enno.luebbers@intel.com>
> + *   Tim Whisonant <tim.whisonant@intel.com>
> + *   Ananda Ravuri <ananda.ravuri@intel.com>
> + *   Henry Mitchel <henry.mitchel@intel.com>
> + *
> + * This work is licensed under a dual BSD/GPLv2 license. When using or
> + * redistributing this file, you may do so under either license. See the
> + * LICENSE.BSD file under this directory for the BSD license and see
> + * the COPYING file in the top-level directory for the GPLv2 license.
> + */
> +
> +#include <linux/pci.h>
> +#include <linux/types.h>
> +#include <linux/kernel.h>
> +#include <linux/module.h>
> +#include <linux/stddef.h>
> +#include <linux/errno.h>
> +#include <linux/aer.h>
> +
> +#define DRV_VERSION	"EXPERIMENTAL VERSION"

Is that a leftover? :)
> +#define DRV_NAME	"intel-fpga-pci"
> +
> +/* PCI Device ID */
> +#define PCIe_DEVICE_ID_PF_INT_5_X	0xBCBD
> +#define PCIe_DEVICE_ID_PF_INT_6_X	0xBCC0
> +#define PCIe_DEVICE_ID_PF_DSC_1_X	0x09C4
> +/* VF Device */
> +#define PCIe_DEVICE_ID_VF_INT_5_X	0xBCBF
> +#define PCIe_DEVICE_ID_VF_INT_6_X	0xBCC1
> +#define PCIe_DEVICE_ID_VF_DSC_1_X	0x09C5
> +
> +static struct pci_device_id cci_pcie_id_tbl[] = {
> +	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_PF_INT_5_X),},
> +	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_VF_INT_5_X),},
> +	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_PF_INT_6_X),},
> +	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_VF_INT_6_X),},
> +	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_PF_DSC_1_X),},
> +	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_VF_DSC_1_X),},
> +	{0,}
> +};
> +MODULE_DEVICE_TABLE(pci, cci_pcie_id_tbl);
> +
> +static
> +int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
> +{
> +	int ret;
> +
> +	ret = pci_enable_device(pcidev);
> +	if (ret < 0) {
> +		dev_err(&pcidev->dev, "Failed to enable device %d.\n", ret);
> +		goto exit;
Why not 'return ret' here ?
> +	}
> +
> +	ret = pci_enable_pcie_error_reporting(pcidev);
> +	if (ret && ret != -EINVAL)
> +		dev_info(&pcidev->dev, "PCIE AER unavailable %d.\n", ret);

What if it is EINVAL?

> +
> +	ret = pci_request_regions(pcidev, DRV_NAME);
> +	if (ret) {
> +		dev_err(&pcidev->dev, "Failed to request regions.\n");
> +		goto disable_error_report_exit;
> +	}
> +
> +	pci_set_master(pcidev);
> +	pci_save_state(pcidev);
> +
> +	if (!dma_set_mask(&pcidev->dev, DMA_BIT_MASK(64))) {
> +		dma_set_coherent_mask(&pcidev->dev, DMA_BIT_MASK(64));
> +	} else if (!dma_set_mask(&pcidev->dev, DMA_BIT_MASK(32))) {
> +		dma_set_coherent_mask(&pcidev->dev, DMA_BIT_MASK(32));
> +	} else {
> +		ret = -EIO;
> +		dev_err(&pcidev->dev, "No suitable DMA support available.\n");
> +		goto release_region_exit;
> +	}
> +
> +	/* TODO: create and add the platform device per feature list */
> +	return 0;
> +
> +release_region_exit:
> +	pci_release_regions(pcidev);
> +disable_error_report_exit:
> +	pci_disable_pcie_error_reporting(pcidev);
> +	pci_disable_device(pcidev);
> +exit:
> +	return ret;
If you return as suggested above, this can go away.

> +}
> +
> +static void cci_pci_remove(struct pci_dev *pcidev)
> +{
> +	pci_release_regions(pcidev);
> +	pci_disable_pcie_error_reporting(pcidev);
> +	pci_disable_device(pcidev);
> +}
> +
> +static struct pci_driver cci_pci_driver = {
> +	.name = DRV_NAME,
> +	.id_table = cci_pcie_id_tbl,
> +	.probe = cci_pci_probe,
> +	.remove = cci_pci_remove,
> +};
> +
> +static int __init ccidrv_init(void)
> +{
> +	pr_info("Intel(R) FPGA PCIe Driver: Version %s\n", DRV_VERSION);
> +
> +	return pci_register_driver(&cci_pci_driver);
> +}
> +
> +static void __exit ccidrv_exit(void)
> +{
> +	pci_unregister_driver(&cci_pci_driver);
> +}
> +
> +module_init(ccidrv_init);
> +module_exit(ccidrv_exit);
> +
> +MODULE_DESCRIPTION("Intel FPGA PCIe Device Driver");
> +MODULE_AUTHOR("Intel Corporation");
> +MODULE_LICENSE("Dual BSD/GPL");
> -- 
> 2.7.4
> 

Cheers,

Moritz

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features.
  2017-03-30 12:08 ` [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features Wu Hao
  2017-04-03 21:44   ` Alan Tull
@ 2017-04-04  2:44   ` Moritz Fischer
  2017-04-05 12:57     ` Wu Hao
  2017-04-04 22:09   ` Alan Tull
  2017-05-04 15:13   ` Li, Yi
  3 siblings, 1 reply; 93+ messages in thread
From: Moritz Fischer @ 2017-04-04  2:44 UTC (permalink / raw)
  To: Wu Hao
  Cc: atull, moritz.fischer, linux-fpga, linux-kernel, luwei.kang,
	yi.z.zhang, Xiao Guangrong, Tim Whisonant, Enno Luebbers,
	Shiva Rao, Christopher Rauer

[-- Attachment #1: Type: text/plain, Size: 42346 bytes --]

Xiao,

few nits inline, I'll need to come back to this once I went over the
rest of the patchset ;-)

On Thu, Mar 30, 2017 at 08:08:04PM +0800, Wu Hao wrote:
> From: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> 
> Device Featuer List structure creates a link list of feature headers
> within the MMIO space to provide an extensiable way of adding features.
> 
> The Intel FPGA PCIe driver walks through the feature headers to enumerate
> feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
> Function Unit (AFU), and their private sub features. For feature devices,
> it creates the platform devices and linked the private sub features into
> their platform data.
> 
> Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> Signed-off-by: Wu Hao <hao.wu@intel.com>
> ---
>  drivers/fpga/intel/Makefile      |   2 +-
>  drivers/fpga/intel/feature-dev.c | 139 +++++++
>  drivers/fpga/intel/feature-dev.h | 342 ++++++++++++++++
>  drivers/fpga/intel/pcie.c        | 841 ++++++++++++++++++++++++++++++++++++++-
>  4 files changed, 1321 insertions(+), 3 deletions(-)
>  create mode 100644 drivers/fpga/intel/feature-dev.c
>  create mode 100644 drivers/fpga/intel/feature-dev.h
> 
> diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
> index 61fd8ea..c029940 100644
> --- a/drivers/fpga/intel/Makefile
> +++ b/drivers/fpga/intel/Makefile
> @@ -1,3 +1,3 @@
>  obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
>  
> -intel-fpga-pci-objs := pcie.o
> +intel-fpga-pci-objs := pcie.o feature-dev.o
> diff --git a/drivers/fpga/intel/feature-dev.c b/drivers/fpga/intel/feature-dev.c
> new file mode 100644
> index 0000000..6952566
> --- /dev/null
> +++ b/drivers/fpga/intel/feature-dev.c
> @@ -0,0 +1,139 @@
> +/*
> + * Intel FPGA Feature Device Driver
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <luwei.kang@intel.com>
> + *   Zhang Yi <yi.z.zhang@intel.com>
> + *   Wu Hao <hao.wu@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *
> + * This work is licensed under a dual BSD/GPLv2 license. When using or
> + * redistributing this file, you may do so under either license. See the
> + * LICENSE.BSD file under this directory for the BSD license and see
> + * the COPYING file in the top-level directory for the GPLv2 license.
> + */
> +
> +#include "feature-dev.h"
> +
> +void feature_platform_data_add(struct feature_platform_data *pdata,
> +			       int index, const char *name,
> +			       int resource_index, void __iomem *ioaddr)
> +{
> +	WARN_ON(index >= pdata->num);
> +
> +	pdata->features[index].name = name;
> +	pdata->features[index].resource_index = resource_index;
> +	pdata->features[index].ioaddr = ioaddr;
> +}
> +
> +int feature_platform_data_size(int num)

static inline? num can be const

> +{
> +	return sizeof(struct feature_platform_data) +
> +		num * sizeof(struct feature);
> +}
> +
> +struct feature_platform_data *
> +feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
> +{
> +	struct feature_platform_data *pdata;
> +
> +	pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
> +	if (pdata) {
> +		pdata->dev = dev;
> +		pdata->num = num;
> +		mutex_init(&pdata->lock);
> +	}
> +
> +	return pdata;
> +}
> +
> +int fme_feature_num(void)

is this used outside of this file? if not -> static
> +{
> +	return FME_FEATURE_ID_MAX;
> +}
> +
> +int port_feature_num(void)

is this used outside of this file? if not -> static
> +{
> +	return PORT_FEATURE_ID_MAX;
> +}
> +
> +int fpga_port_id(struct platform_device *pdev)
> +{
> +	struct feature_port_header *port_hdr;
> +	struct feature_port_capability capability;
> +
> +	port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> +					       PORT_FEATURE_ID_HEADER);
> +	WARN_ON(!port_hdr);
> +
> +	capability.csr = readq(&port_hdr->capability);
> +	return capability.port_number;
> +}
> +EXPORT_SYMBOL_GPL(fpga_port_id);
> +
> +/*
> + * Enable Port by clear the port soft reset bit, which is set by default.
> + * The User AFU is unable to respond to any MMIO access while in reset.
> + * __fpga_port_enable function should only be used after __fpga_port_disable
> + * function.
> + */
> +void __fpga_port_enable(struct platform_device *pdev)
> +{
> +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +	struct feature_port_header *port_hdr;
> +	struct feature_port_control control;
> +
> +	WARN_ON(!pdata->disable_count);
> +
> +	if (--pdata->disable_count != 0)
> +		return;
> +
> +	port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> +					       PORT_FEATURE_ID_HEADER);
> +	WARN_ON(!port_hdr);
> +
> +	control.csr = readq(&port_hdr->control);
> +	control.port_sftrst = 0x0;
> +	writeq(control.csr, &port_hdr->control);
> +}
> +EXPORT_SYMBOL_GPL(__fpga_port_enable);
> +
> +#define RST_POLL_INVL 10 /* us */
> +#define RST_POLL_TIMEOUT 1000 /* us */
> +
> +int __fpga_port_disable(struct platform_device *pdev)
> +{
> +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +	struct feature_port_header *port_hdr;
> +	struct feature_port_control control;
> +
> +	if (pdata->disable_count++ != 0)
> +		return 0;
> +
> +	port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> +					       PORT_FEATURE_ID_HEADER);
> +	WARN_ON(!port_hdr);
> +
> +	/* Set port soft reset */
> +	control.csr = readq(&port_hdr->control);
> +	control.port_sftrst = 0x1;
> +	writeq(control.csr, &port_hdr->control);
> +
> +	/*
> +	 * HW sets ack bit to 1 when all outstanding requests have been drained
> +	 * on this port and minimum soft reset pulse width has elapsed.
> +	 * Driver polls port_soft_reset_ack to determine if reset done by HW.
> +	 */
> +	control.port_sftrst_ack = 1;
> +
> +	if (fpga_wait_register_field(port_sftrst_ack, control,
> +		&port_hdr->control, RST_POLL_TIMEOUT, RST_POLL_INVL)) {
> +		dev_err(&pdev->dev, "timeout, fail to reset device\n");
> +		return -ETIMEDOUT;
> +	}

see iopoll comment above.
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(__fpga_port_disable);
> diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
> new file mode 100644
> index 0000000..a1e6e7d
> --- /dev/null
> +++ b/drivers/fpga/intel/feature-dev.h
> @@ -0,0 +1,342 @@
> +/*
> + * Intel FPGA Feature Device Driver Header File
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <luwei.kang@intel.com>
> + *   Zhang Yi <yi.z.zhang@intel.com>
> + *   Wu Hao <hao.wu@intel.com>
> + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> + *
> + * This work is licensed under a dual BSD/GPLv2 license. When using or
> + * redistributing this file, you may do so under either license. See the
> + * LICENSE.BSD file under this directory for the BSD license and see
> + * the COPYING file in the top-level directory for the GPLv2 license.
> + */
> +
> +#ifndef __INTEL_FPGA_FEATURE_H
> +#define __INTEL_FPGA_FEATURE_H
> +
> +#include <linux/fs.h>
> +#include <linux/pci.h>
> +#include <linux/uuid.h>
> +#include <linux/delay.h>
> +#include <linux/platform_device.h>
> +
> +/* maximum supported number of ports */
> +#define MAX_FPGA_PORT_NUM 4
> +/* plus one for fme device */
> +#define MAX_FEATURE_DEV_NUM	(MAX_FPGA_PORT_NUM + 1)
> +
> +#define FME_FEATURE_HEADER          "fme_hdr"
> +#define FME_FEATURE_THERMAL_MGMT    "fme_thermal"
> +#define FME_FEATURE_POWER_MGMT      "fme_power"
> +#define FME_FEATURE_GLOBAL_PERF     "fme_gperf"
> +#define FME_FEATURE_GLOBAL_ERR      "fme_error"
> +#define FME_FEATURE_PR_MGMT         "fme_pr"
> +
> +#define PORT_FEATURE_HEADER         "port_hdr"
> +#define PORT_FEATURE_UAFU           "port_uafu"
> +#define PORT_FEATURE_ERR            "port_err"
> +#define PORT_FEATURE_UMSG           "port_umsg"
> +#define PORT_FEATURE_PR             "port_pr"
> +#define PORT_FEATURE_STP            "port_stp"
> +
> +/* All headers and structures must be byte-packed to match the spec. */
> +#pragma pack(1)

I recall there being some controversy about this, can't remember which
way people ended up deciding :)
> +
> +/* common header for all features */
> +struct feature_header {
> +	union {
> +		u64 csr;
> +		struct {
> +			u16 id:12;
> +			u8  revision:4;
> +			u32 next_header_offset:24; /* offset to next header */
> +			u32 rsvdz:20;
> +			u8  type:4;		   /* feature type */
> +#define FEATURE_TYPE_AFU		0x1
> +#define FEATURE_TYPE_PRIVATE		0x3
> +		};
> +	};
> +};
> +
> +/* common header for non-private features */
> +struct feature_afu_header {
> +	uuid_le guid;
> +	union {
> +		u64 csr;
> +		struct {
> +			u64 next_afu:24;	/* pointer to next afu header */
> +			u64 rsvdz:40;
> +		};
> +	};
> +};
> +
> +/* FME Header Register Set */
> +/* FME Capability Register */
> +struct feature_fme_capability {
> +	union {
> +		u64 csr;
> +		struct {
> +			u8  fabric_verid;	/* Fabric version ID */
> +			u8  socket_id:1;	/* Socket id */
> +			u8  rsvdz1:3;
> +			u8  pcie0_link_avl:1;	/* PCIe0 link availability */
> +			u8  pcie1_link_avl:1;	/* PCIe1 link availability */
> +			u8  coherent_link_avl:1;/* Coherent link availability */
> +			u8  rsvdz2:1;
> +			u8  iommu_support:1;	/* IOMMU or VT-d supported */
> +			u8  num_ports:3;	/* Num of ports implemented */
> +			u8  rsvdz3:4;
> +			u8  addr_width_bits:6;	/* Address width supported */
> +			u8  rsvdz4:2;
> +			u16 cache_size:12;	/* Cache size in kb */
> +			u8  cache_assoc:4;	/* Cache Associativity */
> +			u16 rsvdz5:15;
> +			u8  lock_bit:1;		/* Latched lock bit by BIOS */
> +		};
> +	};
> +};
> +
> +/* FME Port Offset Register */
> +struct feature_fme_port {
> +	union {
> +		u64 csr;
> +		struct {
> +			u32 port_offset:24;	/* Offset to port header */
> +			u8  rsvdz1;
> +			u8  port_bar:3;		/* Bar id */
> +			u32 rsvdz2:20;
> +			u8  afu_access_ctrl:1;	/* AFU access type: PF/VF */
> +			u8  rsvdz3:4;
> +			u8  port_implemented:1;	/* Port implemented or not */
> +			u8  rsvdz4:3;
> +		};
> +	};
> +};
> +
> +struct feature_fme_header {
> +	struct feature_header header;
> +	struct feature_afu_header afu_header;
> +	u64 rsvd[2];
> +	struct feature_fme_capability capability;
> +	struct feature_fme_port port[MAX_FPGA_PORT_NUM];
> +};
> +
> +/* FME Thermal Sub Feature Register Set */
> +struct feature_fme_thermal {
> +	struct feature_header header;
> +};
> +
> +/* FME Power Sub Feature Register Set */
> +struct feature_fme_power {
> +	struct feature_header header;
> +};
> +
> +/* FME Global Performance Sub Feature Register Set */
> +struct feature_fme_gperf {
> +	struct feature_header header;
> +};
> +
> +/* FME Error Sub Feature Register Set */
> +struct feature_fme_err {
> +	struct feature_header header;
> +};
> +
> +/* FME Partial Reconfiguration Sub Feature Register Set */
> +struct feature_fme_pr {
> +	struct feature_header header;
> +};
> +
> +/* PORT Header Register Set */
> +/* Port Capability Register */
> +struct feature_port_capability {
> +	union {
> +		u64 csr;
> +		struct {
> +			u8  port_number:2;	/* Port Number 0-3 */
> +			u8  rsvdz1:6;
> +			u16 mmio_size;		/* User MMIO size in KB */
> +			u8  rsvdz2;
> +			u8  sp_intr_num:4;	/* Supported interrupts num */
> +			u32 rsvdz3:28;
> +		};
> +	};
> +};
> +
> +/* Port Control Register */
> +struct feature_port_control {
> +	union {
> +		u64 csr;
> +		struct {
> +			u8  port_sftrst:1;	/* Port Soft Reset */
> +			u8  rsvdz1:1;
> +			u8  latency_tolerance:1;/* '1' >= 40us, '0' < 40us */
> +			u8  rsvdz2:1;
> +			u8  port_sftrst_ack:1;	/* HW ACK for Soft Reset */
> +			u64 rsvdz3:59;
> +		};
> +	};
> +};
> +
> +struct feature_port_header {
> +	struct feature_header header;
> +	struct feature_afu_header afu_header;
> +	u64 rsvd[2];
> +	struct feature_port_capability capability;
> +	struct feature_port_control control;
> +};
> +
> +/* PORT Error Sub Feature Register Set */
> +struct feature_port_error {
> +	struct feature_header header;
> +};
> +
> +/* PORT Unordered Message Sub Feature Register Set */
> +struct feature_port_umsg {
> +	struct feature_header header;
> +};
> +
> +/* PORT SignalTap Sub Feature Register Set */
> +struct feature_port_stp {
> +	struct feature_header header;
> +};
> +
> +#pragma pack()
> +
> +struct feature {
> +	const char *name;
> +	int resource_index;
> +	void __iomem *ioaddr;
> +};
> +
> +struct feature_platform_data {
> +	/* list the feature dev to cci_drvdata->port_dev_list. */
> +	struct list_head node;
> +	struct mutex lock;
> +	struct platform_device *dev;
> +	unsigned int disable_count;	/* count for port disable */
> +
> +	int num;			/* number of features */
> +	struct feature features[0];
> +};
> +
> +enum fme_feature_id {
> +	FME_FEATURE_ID_HEADER = 0x0,
> +	FME_FEATURE_ID_THERMAL_MGMT = 0x1,
> +	FME_FEATURE_ID_POWER_MGMT = 0x2,
> +	FME_FEATURE_ID_GLOBAL_PERF = 0x3,
> +	FME_FEATURE_ID_GLOBAL_ERR = 0x4,
> +	FME_FEATURE_ID_PR_MGMT = 0x5,
> +	FME_FEATURE_ID_MAX = 0x6,
> +};
> +
> +enum port_feature_id {
> +	PORT_FEATURE_ID_HEADER = 0x0,
> +	PORT_FEATURE_ID_ERROR = 0x1,
> +	PORT_FEATURE_ID_UMSG = 0x2,
> +	PORT_FEATURE_ID_PR = 0x3,
> +	PORT_FEATURE_ID_STP = 0x4,
> +	PORT_FEATURE_ID_UAFU = 0x5,
> +	PORT_FEATURE_ID_MAX = 0x6,
> +};
> +
> +int fme_feature_num(void);
> +int port_feature_num(void);
> +
> +#define FPGA_FEATURE_DEV_FME		"intel-fpga-fme"
> +#define FPGA_FEATURE_DEV_PORT		"intel-fpga-port"
> +
> +void feature_platform_data_add(struct feature_platform_data *pdata,
> +			       int index, const char *name,
> +			       int resource_index, void __iomem *ioaddr);
> +int feature_platform_data_size(int num);
> +struct feature_platform_data *
> +feature_platform_data_alloc_and_init(struct platform_device *dev, int num);
> +
> +int fpga_port_id(struct platform_device *pdev);
> +
> +static inline int fpga_port_check_id(struct platform_device *pdev,
> +				     void *pport_id)
> +{
> +	return fpga_port_id(pdev) == *(int *)pport_id;
> +}
> +
> +void __fpga_port_enable(struct platform_device *pdev);
> +int __fpga_port_disable(struct platform_device *pdev);
> +
> +static inline void fpga_port_enable(struct platform_device *pdev)
> +{
> +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +
> +	mutex_lock(&pdata->lock);
> +	__fpga_port_enable(pdev);
> +	mutex_unlock(&pdata->lock);
> +}
> +
> +static inline int fpga_port_disable(struct platform_device *pdev)
> +{
> +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +	int ret;
> +
> +	mutex_lock(&pdata->lock);
> +	ret = __fpga_port_disable(pdev);
> +	mutex_unlock(&pdata->lock);
> +
> +	return ret;
> +}
> +
> +static inline int __fpga_port_reset(struct platform_device *pdev)
> +{
> +	int ret;
> +
> +	ret = __fpga_port_disable(pdev);
> +	if (ret)
> +		return ret;
> +
> +	__fpga_port_enable(pdev);
> +	return 0;
> +}
> +
> +static inline int fpga_port_reset(struct platform_device *pdev)
> +{
> +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +	int ret;
> +
> +	mutex_lock(&pdata->lock);
> +	ret = __fpga_port_reset(pdev);
> +	mutex_unlock(&pdata->lock);
> +	return ret;
> +}
> +
> +static inline void __iomem *
> +get_feature_ioaddr_by_index(struct device *dev, int index)
> +{
> +	struct feature_platform_data *pdata = dev_get_platdata(dev);
> +
> +	return pdata->features[index].ioaddr;
> +}
> +
> +/*
> + * Wait register's _field to be changed to the given value (_expect's _field)
> + * by polling with given interval and timeout.
> + */
> +#define fpga_wait_register_field(_field, _expect, _reg_addr, _timeout, _invl)\
> +({									     \
> +	int wait = 0;							     \
> +	int ret = -ETIMEDOUT;						     \
> +	typeof(_expect) value;						     \
> +	for (; wait <= _timeout; wait += _invl) {			     \
> +		value.csr = readq(_reg_addr);				     \
> +		if (_expect._field == value._field) {			     \
> +			ret = 0;					     \
> +			break;						     \
> +		}							     \
> +		udelay(_invl);						     \
> +	}								     \
> +	ret;								     \
> +})

can't you use iopoll and friends instead of this?

> +
> +#endif
> diff --git a/drivers/fpga/intel/pcie.c b/drivers/fpga/intel/pcie.c
> index 132d9da..28df63e 100644
> --- a/drivers/fpga/intel/pcie.c
> +++ b/drivers/fpga/intel/pcie.c
> @@ -25,10 +25,827 @@
>  #include <linux/stddef.h>
>  #include <linux/errno.h>
>  #include <linux/aer.h>
> +#include <linux/fpga/fpga-dev.h>
> +
> +#include "feature-dev.h"
>  
>  #define DRV_VERSION	"EXPERIMENTAL VERSION"
>  #define DRV_NAME	"intel-fpga-pci"
>  
> +#define INTEL_FPGA_DEV	"intel-fpga-dev"
> +
> +static DEFINE_MUTEX(fpga_id_mutex);
> +
> +enum fpga_id_type {
> +	FME_ID,		/* fme id allocation and mapping */
> +	PORT_ID,	/* port id allocation and mapping */
> +	FPGA_ID_MAX,
> +};
> +
> +/* it is protected by fpga_id_mutex */
> +static struct idr fpga_ids[FPGA_ID_MAX];
> +
> +struct cci_drvdata {
> +	struct device *fme_dev;
> +
> +	struct mutex lock;
> +	struct list_head port_dev_list;
> +
> +	struct list_head regions; /* global list of pci bar mapping region */
> +};
> +
> +/* pci bar mapping info */
> +struct cci_pci_region {
> +	int bar;
> +	void __iomem *ioaddr;	/* pointer to mapped bar region */
> +	struct list_head node;
> +};
> +
> +static void fpga_ids_init(void)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> +		idr_init(fpga_ids + i);
> +}
> +
> +static void fpga_ids_destroy(void)
> +{
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> +		idr_destroy(fpga_ids + i);
> +}
> +
> +static int alloc_fpga_id(enum fpga_id_type type, struct device *dev)
> +{
> +	int id;
> +
> +	WARN_ON(type >= FPGA_ID_MAX);
> +	mutex_lock(&fpga_id_mutex);
> +	id = idr_alloc(fpga_ids + type, dev, 0, 0, GFP_KERNEL);
> +	mutex_unlock(&fpga_id_mutex);
> +	return id;
> +}
> +
> +static void free_fpga_id(enum fpga_id_type type, int id)
> +{
> +	WARN_ON(type >= FPGA_ID_MAX);
> +	mutex_lock(&fpga_id_mutex);
> +	idr_remove(fpga_ids + type, id);
> +	mutex_unlock(&fpga_id_mutex);
> +}
> +
> +static void cci_pci_add_port_dev(struct pci_dev *pdev,
> +				 struct platform_device *port_dev)
> +{
> +	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +	struct feature_platform_data *pdata = dev_get_platdata(&port_dev->dev);
> +
> +	mutex_lock(&drvdata->lock);
> +	list_add(&pdata->node, &drvdata->port_dev_list);
> +	get_device(&pdata->dev->dev);
> +	mutex_unlock(&drvdata->lock);
> +}
> +
> +static void cci_pci_remove_port_devs(struct pci_dev *pdev)
> +{
> +	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +	struct feature_platform_data *pdata, *ptmp;
> +
> +	mutex_lock(&drvdata->lock);
> +	list_for_each_entry_safe(pdata, ptmp, &drvdata->port_dev_list, node) {
> +		struct platform_device *port_dev = pdata->dev;
> +
> +		/* the port should be unregistered first. */
> +		WARN_ON(device_is_registered(&port_dev->dev));
> +		list_del(&pdata->node);
> +		free_fpga_id(PORT_ID, port_dev->id);
> +		put_device(&port_dev->dev);
> +	}
> +	mutex_unlock(&drvdata->lock);
> +}
> +
> +/* info collection during feature dev build. */
> +struct build_feature_devs_info {
> +	struct pci_dev *pdev;
> +
> +	/*
> +	 * PCI BAR mapping info. Parsing feature list starts from
> +	 * BAR 0 and switch to different BARs to parse Port
> +	 */
> +	void __iomem *ioaddr;
> +	void __iomem *ioend;
> +	int current_bar;
> +
> +	/* points to FME header where the port offset is figured out. */
> +	void __iomem *pfme_hdr;
> +
> +	/* the container device for all feature devices */
> +	struct fpga_dev *parent_dev;
> +
> +	/* current feature device */
> +	struct platform_device *feature_dev;
> +};
> +
> +static void cci_pci_release_regions(struct pci_dev *pdev)
> +{
> +	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +	struct cci_pci_region *tmp, *region;
> +
> +	list_for_each_entry_safe(region, tmp, &drvdata->regions, node) {
> +		list_del(&region->node);
> +		if (region->ioaddr)
> +			pci_iounmap(pdev, region->ioaddr);
> +		devm_kfree(&pdev->dev, region);
> +	}
> +}
> +
> +static void __iomem *cci_pci_ioremap_bar(struct pci_dev *pdev, int bar)
> +{
> +	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +	struct cci_pci_region *region;
> +
> +	list_for_each_entry(region, &drvdata->regions, node)
> +		if (region->bar == bar) {
> +			dev_dbg(&pdev->dev, "BAR %d region exists\n", bar);
> +			return region->ioaddr;
> +		}
> +
> +	region = devm_kzalloc(&pdev->dev, sizeof(*region), GFP_KERNEL);
> +	if (!region)
> +		return NULL;
> +
> +	region->bar = bar;
> +	region->ioaddr = pci_ioremap_bar(pdev, bar);
> +	if (!region->ioaddr) {
> +		dev_err(&pdev->dev, "can't ioremap memory from BAR %d.\n", bar);
> +		devm_kfree(&pdev->dev, region);
> +		return NULL;
> +	}
> +
> +	list_add(&region->node, &drvdata->regions);
> +	return region->ioaddr;
> +}
> +
> +static int parse_start_from(struct build_feature_devs_info *binfo, int bar)
> +{
> +	binfo->ioaddr = cci_pci_ioremap_bar(binfo->pdev, bar);
> +	if (!binfo->ioaddr)
> +		return -ENOMEM;
> +
> +	binfo->current_bar = bar;
> +	binfo->ioend = binfo->ioaddr + pci_resource_len(binfo->pdev, bar);
> +	return 0;
> +}
> +
> +static int parse_start(struct build_feature_devs_info *binfo)
> +{
> +	/* fpga feature list starts from BAR 0 */
> +	return parse_start_from(binfo, 0);
> +}
> +
> +/* switch the memory mapping to BAR# @bar */
> +static int parse_switch_to(struct build_feature_devs_info *binfo, int bar)
> +{
> +	return parse_start_from(binfo, bar);
> +}
> +
> +static struct build_feature_devs_info *
> +build_info_alloc_and_init(struct pci_dev *pdev)
> +{
> +	struct build_feature_devs_info *binfo;
> +
> +	binfo = devm_kzalloc(&pdev->dev, sizeof(*binfo), GFP_KERNEL);
> +	if (binfo)
> +		binfo->pdev = pdev;
> +
> +	return binfo;
> +}
> +
> +static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
> +{
> +	if (!strcmp(pdev->name, FPGA_FEATURE_DEV_FME))
> +		return FME_ID;
> +
> +	if (!strcmp(pdev->name, FPGA_FEATURE_DEV_PORT))
> +		return PORT_ID;
> +
> +	WARN_ON(1);
> +	return FPGA_ID_MAX;
> +}
> +
> +/*
> + * register current feature device, it is called when we need to switch to
> + * another feature parsing or we have parsed all features
> + */
> +static int build_info_commit_dev(struct build_feature_devs_info *binfo)
> +{
> +	int ret;
> +
> +	if (!binfo->feature_dev)
> +		return 0;
> +
> +	ret = platform_device_add(binfo->feature_dev);
> +	if (!ret) {
> +		struct cci_drvdata *drvdata;
> +
> +		drvdata = dev_get_drvdata(&binfo->pdev->dev);
> +		if (feature_dev_id_type(binfo->feature_dev) == PORT_ID)
> +			cci_pci_add_port_dev(binfo->pdev, binfo->feature_dev);
> +		else
> +			drvdata->fme_dev = get_device(&binfo->feature_dev->dev);
> +
> +		/*
> +		 * reset it to avoid build_info_free() freeing their resource.
> +		 *
> +		 * The resource of successfully registered feature devices
> +		 * will be freed by platform_device_unregister(). See the
> +		 * comments in build_info_create_dev().
> +		 */
> +		binfo->feature_dev = NULL;
> +	}
> +
> +	return ret;
> +}
> +
> +static int
> +build_info_create_dev(struct build_feature_devs_info *binfo,
> +		      enum fpga_id_type type, int feature_nr, const char *name)
> +{
> +	struct platform_device *fdev;
> +	struct resource *res;
> +	struct feature_platform_data *pdata;
> +	int ret;
> +
> +	/* we will create a new device, commit current device first */
> +	ret = build_info_commit_dev(binfo);
> +	if (ret)
> +		return ret;
> +
> +	/*
> +	 * we use -ENODEV as the initialization indicator which indicates
> +	 * whether the id need to be reclaimed
> +	 */
> +	fdev = binfo->feature_dev = platform_device_alloc(name, -ENODEV);
> +	if (!fdev)
> +		return -ENOMEM;
> +
> +	fdev->id = alloc_fpga_id(type, &fdev->dev);
> +	if (fdev->id < 0)
> +		return fdev->id;
> +
> +	fdev->dev.parent = &binfo->parent_dev->dev;
> +
> +	/*
> +	 * we need not care the memory which is associated with the
we do not need to care *for* the memory ;-)
> +	 * platform device. After call platform_device_unregister(),
After calling ...
> +	 * it will be automatically freed by device's
> +	 * release() callback, platform_device_release().
> +	 */
> +	pdata = feature_platform_data_alloc_and_init(fdev, feature_nr);
> +	if (!pdata)
> +		return -ENOMEM;
> +
> +	/*
> +	 * the count should be initialized to 0 to make sure
> +	 *__fpga_port_enable() following __fpga_port_disable()
> +	 * works properly for port device.
> +	 * and it should always be 0 for fme device.
> +	 */
> +	WARN_ON(pdata->disable_count);
> +
> +	fdev->dev.platform_data = pdata;
> +	fdev->num_resources = feature_nr;
> +	fdev->resource = kcalloc(feature_nr, sizeof(*res), GFP_KERNEL);
> +	if (!fdev->resource)
> +		return -ENOMEM;
> +
> +	return 0;
> +}
> +
> +static int remove_feature_dev(struct device *dev, void *data)
> +{
> +	struct platform_device *pdev = to_platform_device(dev);
> +
> +	platform_device_unregister(pdev);
> +	return 0;
> +}
> +
> +static int remove_parent_dev(struct device *dev, void *data)
> +{
> +	/* remove platform devices attached in the parent device */
> +	device_for_each_child(dev, NULL, remove_feature_dev);
> +	fpga_dev_destroy(to_fpga_dev(dev));
> +	return 0;
> +}
> +
> +static void remove_all_devs(struct pci_dev *pdev)
> +{
> +	/* remove parent device and all its children. */
> +	device_for_each_child(&pdev->dev, NULL, remove_parent_dev);
> +}
> +
> +static void build_info_free(struct build_feature_devs_info *binfo)
> +{
> +	if (!IS_ERR_OR_NULL(binfo->parent_dev))
> +		remove_all_devs(binfo->pdev);
> +
> +	/*
> +	 * it is a valid id, free it. See comments in
> +	 * build_info_create_dev()
> +	 */
> +	if (binfo->feature_dev && binfo->feature_dev->id >= 0)
> +		free_fpga_id(feature_dev_id_type(binfo->feature_dev),
> +			     binfo->feature_dev->id);
> +
> +	platform_device_put(binfo->feature_dev);
> +
> +	devm_kfree(&binfo->pdev->dev, binfo);
> +}
> +
> +#define FEATURE_TYPE_AFU	0x1
> +#define FEATURE_TYPE_PRIVATE	0x3
> +
> +/* FME and PORT GUID are fixed */
> +#define FEATURE_FME_GUID "f9e17764-38f0-82fe-e346-524ae92aafbf"
> +#define FEATURE_PORT_GUID "6b355b87-b06c-9642-eb42-8d139398b43a"
> +
> +static bool feature_is_fme(struct feature_afu_header *afu_hdr)
> +{
> +	uuid_le u;
> +
> +	uuid_le_to_bin(FEATURE_FME_GUID, &u);
> +
> +	return !uuid_le_cmp(u, afu_hdr->guid);
> +}
> +
> +static bool feature_is_port(struct feature_afu_header *afu_hdr)
> +{
> +	uuid_le u;
> +
> +	uuid_le_to_bin(FEATURE_PORT_GUID, &u);
> +
> +	return !uuid_le_cmp(u, afu_hdr->guid);
> +}
> +
> +/*
> + * UAFU GUID is dynamic as it can be changed after FME downloads different
> + * Green Bitstream to the port, so we treat the unknown GUIDs which are
> + * attached on port's feature list as UAFU.
> + */
> +static bool feature_is_UAFU(struct build_feature_devs_info *binfo)
> +{
> +	if (!binfo->feature_dev ||
> +	      feature_dev_id_type(binfo->feature_dev) != PORT_ID)
> +		return false;
> +
> +	return true;
> +}
> +
> +static void
> +build_info_add_sub_feature(struct build_feature_devs_info *binfo,
> +			   int feature_id, const char *feature_name,
> +			   resource_size_t resource_size, void __iomem *start)
> +{
> +
> +	struct platform_device *fdev = binfo->feature_dev;
> +	struct feature_platform_data *pdata = dev_get_platdata(&fdev->dev);
> +	struct resource *res = &fdev->resource[feature_id];
> +
> +	res->start = pci_resource_start(binfo->pdev, binfo->current_bar) +
> +		start - binfo->ioaddr;
> +	res->end = res->start + resource_size - 1;
> +	res->flags = IORESOURCE_MEM;
> +	res->name = feature_name;
> +
> +	feature_platform_data_add(pdata, feature_id,
> +				  feature_name, feature_id, start);
> +}
> +
> +struct feature_info {
> +	const char *name;
> +	resource_size_t resource_size;
> +	int feature_index;
> +};
> +
> +/* indexed by fme feature IDs which are defined in 'enum fme_feature_id'. */
> +static struct feature_info fme_features[] = {
> +	{
> +		.name = FME_FEATURE_HEADER,
> +		.resource_size = sizeof(struct feature_fme_header),
> +		.feature_index = FME_FEATURE_ID_HEADER,
> +	},
> +	{
> +		.name = FME_FEATURE_THERMAL_MGMT,
> +		.resource_size = sizeof(struct feature_fme_thermal),
> +		.feature_index = FME_FEATURE_ID_THERMAL_MGMT,
> +	},
> +	{
> +		.name = FME_FEATURE_POWER_MGMT,
> +		.resource_size = sizeof(struct feature_fme_power),
> +		.feature_index = FME_FEATURE_ID_POWER_MGMT,
> +	},
> +	{
> +		.name = FME_FEATURE_GLOBAL_PERF,
> +		.resource_size = sizeof(struct feature_fme_gperf),
> +		.feature_index = FME_FEATURE_ID_GLOBAL_PERF,
> +	},
> +	{
> +		.name = FME_FEATURE_GLOBAL_ERR,
> +		.resource_size = sizeof(struct feature_fme_err),
> +		.feature_index = FME_FEATURE_ID_GLOBAL_ERR,
> +	},
> +	{
> +		.name = FME_FEATURE_PR_MGMT,
> +		.resource_size = sizeof(struct feature_fme_pr),
> +		.feature_index = FME_FEATURE_ID_PR_MGMT,
> +	}
> +};
> +
> +/* indexed by port feature IDs which are defined in 'enum port_feature_id'. */
> +static struct feature_info port_features[] = {

Could probably be const
> +	{
> +		.name = PORT_FEATURE_HEADER,
> +		.resource_size = sizeof(struct feature_port_header),
> +		.feature_index = PORT_FEATURE_ID_HEADER,
> +	},
> +	{
> +		.name = PORT_FEATURE_ERR,
> +		.resource_size = sizeof(struct feature_port_error),
> +		.feature_index = PORT_FEATURE_ID_ERROR,
> +	},
> +	{
> +		.name = PORT_FEATURE_UMSG,
> +		.resource_size = sizeof(struct feature_port_umsg),
> +		.feature_index = PORT_FEATURE_ID_UMSG,
> +	},
> +	{
> +		/* This feature isn't available for now */
> +		.name = PORT_FEATURE_PR,
> +		.resource_size = 0,
> +		.feature_index = PORT_FEATURE_ID_PR,
> +	},
> +	{
> +		.name = PORT_FEATURE_STP,
> +		.resource_size = sizeof(struct feature_port_stp),
> +		.feature_index = PORT_FEATURE_ID_STP,
> +	},
> +	{
> +		/*
> +		 * For User AFU feature, its region size is not fixed, but
> +		 * reported by register PortCapability.mmio_size. Resource
> +		 * size of UAFU will be set while parse port device.
> +		 */
> +		.name = PORT_FEATURE_UAFU,
> +		.resource_size = 0,
> +		.feature_index = PORT_FEATURE_ID_UAFU,
> +	},
> +};
> +
> +static int
> +create_feature_instance(struct build_feature_devs_info *binfo,
> +			void __iomem *start, struct feature_info *finfo)
> +{
> +	if (binfo->ioend - start < finfo->resource_size)
> +		return -EINVAL;
> +
> +	build_info_add_sub_feature(binfo, finfo->feature_index, finfo->name,
> +				   finfo->resource_size, start);
> +	return 0;
> +}
> +
> +static int parse_feature_fme(struct build_feature_devs_info *binfo,
> +			     void __iomem *start)
> +{
> +	struct cci_drvdata *drvdata = dev_get_drvdata(&binfo->pdev->dev);
> +	int ret;
> +
> +	ret = build_info_create_dev(binfo, FME_ID, fme_feature_num(),
> +					FPGA_FEATURE_DEV_FME);
> +	if (ret)
> +		return ret;
> +
> +	if (drvdata->fme_dev) {
> +		dev_err(&binfo->pdev->dev, "Multiple FMEs are detected.\n");
> +		return -EINVAL;
> +	}
> +
> +	return create_feature_instance(binfo, start,
> +				       &fme_features[FME_FEATURE_ID_HEADER]);
> +}
> +
> +static int parse_feature_fme_private(struct build_feature_devs_info *binfo,
> +				     struct feature_header *hdr)
> +{
> +	struct feature_header header;
> +
> +	header.csr = readq(hdr);
> +
> +	if (header.id >= ARRAY_SIZE(fme_features)) {
> +		dev_info(&binfo->pdev->dev, "FME feature id %x is not supported yet.\n",
> +			 header.id);
> +		return 0;
> +	}
> +
> +	return create_feature_instance(binfo, hdr, &fme_features[header.id]);
> +}
> +
> +static int parse_feature_port(struct build_feature_devs_info *binfo,
> +			     void __iomem *start)
> +{
> +	int ret;
> +
> +	ret = build_info_create_dev(binfo, PORT_ID, port_feature_num(),
> +					FPGA_FEATURE_DEV_PORT);
> +	if (ret)
> +		return ret;
> +
> +	return create_feature_instance(binfo, start,
> +				       &port_features[PORT_FEATURE_ID_HEADER]);
> +}
> +
> +static void enable_port_uafu(struct build_feature_devs_info *binfo,
> +			     void __iomem *start)
> +{
> +	enum port_feature_id id = PORT_FEATURE_ID_UAFU;
> +	struct feature_port_header *port_hdr;
> +	struct feature_port_capability capability;
> +
> +	port_hdr = (struct feature_port_header *)start;
> +	capability.csr = readq(&port_hdr->capability);
> +	port_features[id].resource_size = capability.mmio_size << 10;
> +
> +	/*
> +	 * To enable User AFU, driver needs to clear reset bit on related port,
> +	 * otherwise the mmio space of this user AFU will be invalid.
> +	 */
> +	if (port_features[id].resource_size)
> +		fpga_port_reset(binfo->feature_dev);
> +}
> +
> +static int parse_feature_port_private(struct build_feature_devs_info *binfo,
> +				      struct feature_header *hdr)
> +{
> +	struct feature_header header;
> +	enum port_feature_id id;
> +
> +	header.csr = readq(hdr);
> +	/*
> +	 * the region of port feature id is [0x10, 0x13], + 1 to reserve 0
> +	 * which is dedicated for port-hdr.
> +	 */
> +	id = (header.id & 0x000f) + 1;
> +
> +	if (id >= ARRAY_SIZE(port_features)) {
> +		dev_info(&binfo->pdev->dev, "Port feature id %x is not supported yet.\n",
> +			 header.id);
> +		return 0;
> +	}
> +
> +	return create_feature_instance(binfo, hdr, &port_features[id]);
> +}
> +
> +static int parse_feature_port_uafu(struct build_feature_devs_info *binfo,
> +				 struct feature_header *hdr)
> +{
> +	enum port_feature_id id = PORT_FEATURE_ID_UAFU;
> +	int ret;
> +
> +	if (port_features[id].resource_size) {
> +		ret = create_feature_instance(binfo, hdr, &port_features[id]);
> +		port_features[id].resource_size = 0;
> +	} else {
> +		dev_err(&binfo->pdev->dev, "the uafu feature header is mis-configured.\n");
> +		ret = -EINVAL;
> +	}
> +
> +	return ret;
> +}
> +
> +static int parse_feature_afus(struct build_feature_devs_info *binfo,
> +			      struct feature_header *hdr)
> +{
> +	int ret;
> +	struct feature_afu_header *afu_hdr, header;
> +	void __iomem *start;
> +	void __iomem *end = binfo->ioend;
> +
> +	start = hdr;
> +	for (; start < end; start += header.next_afu) {
> +		if (end - start < (sizeof(*afu_hdr) + sizeof(*hdr)))
> +			return -EINVAL;
> +
> +		hdr = start;
> +		afu_hdr = (struct feature_afu_header *) (hdr + 1);
> +		header.csr = readq(&afu_hdr->csr);
> +
> +		if (feature_is_fme(afu_hdr)) {
> +			ret = parse_feature_fme(binfo, hdr);
> +			binfo->pfme_hdr = hdr;
> +			if (ret)
> +				return ret;
> +		} else if (feature_is_port(afu_hdr)) {
> +			ret = parse_feature_port(binfo, hdr);
> +			enable_port_uafu(binfo, hdr);
> +			if (ret)
> +				return ret;
> +		} else if (feature_is_UAFU(binfo)) {
> +			ret = parse_feature_port_uafu(binfo, hdr);
> +			if (ret)
> +				return ret;
> +		} else
> +			dev_info(&binfo->pdev->dev, "AFU GUID %pUl is not supported yet.\n",
> +				 afu_hdr->guid.b);
> +
> +		if (!header.next_afu)
> +			break;
> +	}
> +
> +	return 0;
> +}
> +
> +static int parse_feature_private(struct build_feature_devs_info *binfo,
> +				 struct feature_header *hdr)
> +{
> +	struct feature_header header;
> +
> +	header.csr = readq(hdr);
> +
> +	if (!binfo->feature_dev) {
> +		dev_err(&binfo->pdev->dev, "the private feature %x does not belong to any AFU.\n",
> +			header.id);
> +		return -EINVAL;
> +	}
> +
> +	switch (feature_dev_id_type(binfo->feature_dev)) {
> +	case FME_ID:
> +		return parse_feature_fme_private(binfo, hdr);
> +	case PORT_ID:
> +		return parse_feature_port_private(binfo, hdr);
> +	default:
> +		dev_info(&binfo->pdev->dev, "private feature %x belonging to AFU %s is not supported yet.\n",
> +			 header.id, binfo->feature_dev->name);
> +	}
> +	return 0;
> +}
> +
> +static int parse_feature(struct build_feature_devs_info *binfo,
> +			 struct feature_header *hdr)
> +{
> +	struct feature_header header;
> +	int ret = 0;
> +
> +	header.csr = readq(hdr);
> +
> +	switch (header.type) {
> +	case FEATURE_TYPE_AFU:
> +		ret = parse_feature_afus(binfo, hdr);
> +		break;
> +	case FEATURE_TYPE_PRIVATE:
> +		ret = parse_feature_private(binfo, hdr);
> +		break;
> +	default:
> +		dev_info(&binfo->pdev->dev,
> +			 "Feature Type %x is not supported.\n", hdr->type);
> +	};
> +
> +	return ret;
> +}
> +
> +static int
> +parse_feature_list(struct build_feature_devs_info *binfo, void __iomem *start)
> +{
> +	struct feature_header *hdr, header;
> +	void __iomem *end = binfo->ioend;
> +	int ret = 0;
> +
> +	for (; start < end; start += header.next_header_offset) {
> +		if (end - start < sizeof(*hdr)) {
> +			dev_err(&binfo->pdev->dev, "The region is too small to contain a feature.\n");
> +			ret =  -EINVAL;
> +			break;
> +		}
> +
> +		hdr = (struct feature_header *)start;
> +		ret = parse_feature(binfo, hdr);
> +		if (ret)
> +			break;
> +
> +		header.csr = readq(hdr);
> +		if (!header.next_header_offset)
> +			break;
> +	}
> +
> +	return ret;
> +}
> +
> +static int parse_ports_from_fme(struct build_feature_devs_info *binfo)
> +{
> +	struct feature_fme_header *fme_hdr;
> +	struct feature_fme_port port;
> +	int i = 0, ret = 0;
> +
> +	if (binfo->pfme_hdr == NULL) {
> +		dev_dbg(&binfo->pdev->dev, "VF is detected.\n");
> +		return ret;
> +	}
> +
> +	fme_hdr = binfo->pfme_hdr;
> +
> +	do {
> +		port.csr = readq(&fme_hdr->port[i]);
> +		if (!port.port_implemented)
> +			break;
> +
> +		ret = parse_switch_to(binfo, port.port_bar);
> +		if (ret)
> +			break;
> +
> +		ret = parse_feature_list(binfo,
> +				binfo->ioaddr + port.port_offset);
> +		if (ret)
> +			break;
> +	} while (++i < MAX_FPGA_PORT_NUM);
> +
> +	return ret;
> +}
> +
> +static int create_init_drvdata(struct pci_dev *pdev)
> +{
> +	struct cci_drvdata *drvdata;
> +
> +	drvdata = devm_kzalloc(&pdev->dev, sizeof(*drvdata), GFP_KERNEL);
> +	if (!drvdata)
> +		return -ENOMEM;
> +
> +	mutex_init(&drvdata->lock);
> +	INIT_LIST_HEAD(&drvdata->port_dev_list);
> +	INIT_LIST_HEAD(&drvdata->regions);
> +
> +	dev_set_drvdata(&pdev->dev, drvdata);
> +	return 0;
> +}
> +
> +static void destroy_drvdata(struct pci_dev *pdev)
> +{
> +	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> +
> +	if (drvdata->fme_dev) {
> +		/* fme device should be unregistered first. */
> +		WARN_ON(device_is_registered(drvdata->fme_dev));
> +		free_fpga_id(FME_ID, to_platform_device(drvdata->fme_dev)->id);
> +		put_device(drvdata->fme_dev);
> +	}
> +
> +	cci_pci_remove_port_devs(pdev);
> +	cci_pci_release_regions(pdev);
> +	dev_set_drvdata(&pdev->dev, NULL);
> +	devm_kfree(&pdev->dev, drvdata);
> +}
> +
> +static int cci_pci_create_feature_devs(struct pci_dev *pdev)
> +{
> +	struct build_feature_devs_info *binfo;
> +	int ret;
> +
> +	binfo = build_info_alloc_and_init(pdev);
> +	if (!binfo)
> +		return -ENOMEM;
> +
> +	binfo->parent_dev = fpga_dev_create(&pdev->dev, INTEL_FPGA_DEV);
> +	if (IS_ERR(binfo->parent_dev)) {
> +		ret = PTR_ERR(binfo->parent_dev);
> +		goto free_binfo_exit;
> +	}
> +
> +	ret = parse_start(binfo);
> +	if (ret)
> +		goto free_binfo_exit;
> +
> +	ret = parse_feature_list(binfo, binfo->ioaddr);
> +	if (ret)
> +		goto free_binfo_exit;
> +
> +	ret = parse_ports_from_fme(binfo);
> +	if (ret)
> +		goto free_binfo_exit;
> +
> +	ret = build_info_commit_dev(binfo);
> +	if (ret)
> +		goto free_binfo_exit;
> +
> +	/*
> +	 * everything is okay, reset ->parent_dev to stop it being
> +	 * freed by build_info_free()
> +	 */
> +	binfo->parent_dev = NULL;
> +
> +free_binfo_exit:
> +	build_info_free(binfo);
> +	return ret;
> +}
> +
>  /* PCI Device ID */
>  #define PCIe_DEVICE_ID_PF_INT_5_X	0xBCBD
>  #define PCIe_DEVICE_ID_PF_INT_6_X	0xBCC0
> @@ -83,9 +900,18 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
>  		goto release_region_exit;
>  	}
>  
> -	/* TODO: create and add the platform device per feature list */
> +	ret = create_init_drvdata(pcidev);
> +	if (ret)
> +		goto release_region_exit;
> +
> +	ret = cci_pci_create_feature_devs(pcidev);
> +	if (ret)
> +		goto destroy_drvdata_exit;
> +
>  	return 0;
>  
> +destroy_drvdata_exit:
> +	destroy_drvdata(pcidev);
>  release_region_exit:
>  	pci_release_regions(pcidev);
>  disable_error_report_exit:
> @@ -97,6 +923,8 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
>  
>  static void cci_pci_remove(struct pci_dev *pcidev)
>  {
> +	remove_all_devs(pcidev);
> +	destroy_drvdata(pcidev);
>  	pci_release_regions(pcidev);
>  	pci_disable_pcie_error_reporting(pcidev);
>  	pci_disable_device(pcidev);
> @@ -111,14 +939,23 @@ static struct pci_driver cci_pci_driver = {
>  
>  static int __init ccidrv_init(void)
>  {
> +	int ret;
> +
>  	pr_info("Intel(R) FPGA PCIe Driver: Version %s\n", DRV_VERSION);
>  
> -	return pci_register_driver(&cci_pci_driver);
> +	fpga_ids_init();
> +
> +	ret = pci_register_driver(&cci_pci_driver);
> +	if (ret)
> +		fpga_ids_destroy();
> +
> +	return ret;
>  }
>  
>  static void __exit ccidrv_exit(void)
>  {
>  	pci_unregister_driver(&cci_pci_driver);
> +	fpga_ids_destroy();
>  }
>  
>  module_init(ccidrv_init);
> -- 
> 2.7.4
> 

Thanks,

Moritz

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview
  2017-04-02 14:41         ` Moritz Fischer
  2017-04-03 20:44           ` Alan Tull
@ 2017-04-04  5:06           ` Wu Hao
  2017-04-11 18:02           ` Alan Tull
  2 siblings, 0 replies; 93+ messages in thread
From: Wu Hao @ 2017-04-04  5:06 UTC (permalink / raw)
  To: Moritz Fischer
  Cc: Alan Tull, matthew.gerlach, Moritz Fischer, linux-fpga,
	linux-kernel, luwei.kang, yi.z.zhang, Enno Luebbers,
	Xiao Guangrong

On Sun, Apr 02, 2017 at 07:41:46AM -0700, Moritz Fischer wrote:
> On Sat, Apr 01, 2017 at 07:16:19PM +0800, Wu Hao wrote:
> > On Fri, Mar 31, 2017 at 01:38:06PM -0500, Alan Tull wrote:
> > > On Fri, Mar 31, 2017 at 1:24 PM,  <matthew.gerlach@linux.intel.com> wrote:
> > > >
> > > >
> > > > On Thu, 30 Mar 2017, Wu Hao wrote:
> > > >
> > > >
> > > > Hi Wu Hao,
> > > >
> > > > Great documentation. I'm looking forward to diving into the rest of the
> > > > patches. Please see my comments inline.
> > > >
> > > > Matthew Gerlach
> > > >
> > > >
> > > >> Add a document for Intel FPGA driver overview.
> > > >>
> > > >> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > > >> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > > >> Signed-off-by: Wu Hao <hao.wu@intel.com>
> > > >> ---
> > > >> Documentation/fpga/intel-fpga.txt | 259
> > > >> ++++++++++++++++++++++++++++++++++++++
> > > >> 1 file changed, 259 insertions(+)
> > > >> create mode 100644 Documentation/fpga/intel-fpga.txt
> > > >>
> > > >> diff --git a/Documentation/fpga/intel-fpga.txt
> > > >> b/Documentation/fpga/intel-fpga.txt
> > > >> new file mode 100644
> > > >> index 0000000..9396cea
> > > >> --- /dev/null
> > > >> +++ b/Documentation/fpga/intel-fpga.txt
> > > >> @@ -0,0 +1,259 @@
> > > >>
> > > >> +===============================================================================
> > > >> +                    Intel FPGA driver Overview
> > > >>
> > > >> +-------------------------------------------------------------------------------
> > > >> +                Enno Luebbers <enno.luebbers@intel.com>
> > > >> +                Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > > >> +                Wu Hao <hao.wu@intel.com>
> > > >> +
> > > >> +The Intel FPGA driver provides interfaces for userspace applications to
> > > >> +configure, enumerate, open, and access FPGA accelerators on platforms
> > > >> equipped
> > > >> +with Intel(R) FPGA solutions and enables system level management
> > > >> functions such
> > > >> +as FPGA reconfiguration, power management, and virtualization.
> > > >> +
> > > >
> > > >
> > > > From a Linux kernel perspective, I'm not sure this is the best name for
> > > > this code.  The name gives me the impression that it is a driver for all
> > > > Intel FPGAs, but not all Intel FPGAs are connected to the processor over a
> > > > PCIe bus.  The processor could be directely connected like the Arria10
> > > > SOCFPGA.  Such a processor could certainly benefit from this accelerator
> > > > usage model.  In an extreme case, couldn't a processor in the FPGA,
> > > > running Linux, also benefit from this accelerator model?  Is this code a
> > > > "FPGA Accelerator Framework"?
> > > >
> > > >> +HW Architecture
> > > >> +===============
> > > >> +From the OS's point of view, the FPGA hardware appears as a regular PCIe
> > > >> device.
> > > >> +The FPGA device memory is organized using a predefined data structure
> > > >> (Device
> > > >> +Feature List). Features supported by the particular FPGA device are
> > > >> exposed
> > > >> +through these data structures, as illustrated below:
> > > >> +
> > > >> +  +-------------------------------+  +-------------+
> > > >> +  |              PF               |  |     VF      |
> > > >> +  +-------------------------------+  +-------------+
> > > >> +      ^            ^         ^              ^
> > > >> +      |            |         |              |
> > > >> ++-----|------------|---------|--------------|-------+
> > > >> +|     |            |         |              |       |
> > > >> +|  +-----+     +-------+ +-------+      +-------+   |
> > > >> +|  | FME |     | Port0 | | Port1 |      | Port2 |   |
> > > >> +|  +-----+     +-------+ +-------+      +-------+   |
> > > >> +|                  ^         ^              ^       |
> > > >> +|                  |         |              |       |
> > > >> +|              +-------+ +------+       +-------+   |
> > > >> +|              |  AFU  | |  AFU |       |  AFU  |   |
> > > >> +|              +-------+ +------+       +-------+   |
> > > >> +|                                                   |
> > > >> +|                 FPGA PCIe Device                  |
> > > >> ++---------------------------------------------------+
> > > >> +
> > > >> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which
> > > >> can be
> > > >> +used to assign individual accelerators to virtual machines .
> > > >
> > > >
> > > > Does this HW Architecture require an Intel FPGA?  Couldn't any vendors FPGA
> > > > be used as long as it presented itself the PCIe bus the same and contained
> > > > an appropriate Device Feature List?
> 
> I think this is a good (and important) point. Especially when sysfs
> entries & ioctls constituting ABI depend on it.

Thanks for your feedback and comments.

I'm not sure if any vendors FPGA will resue the same. But if the same FME
or Port/AFU module implemented in their hardwares, then they should be
able to use our FME/AFU drivers.

AFU driver creates interfaces for FPGA accelerators. And FME driver creates
interfaces for FPGA partial reconfiguration and other management functions.
As mentioned in 'Open discussions' below, we may need to switch to the ABI
(sysfs) of fpga-region for PR.

> 
> > > >
> > > >> +
> > > >> +FME (FPGA Management Engine)
> > > >> +============================
> > > >> +The FPGA Management Enging performs power and thermal management, error
> Enging->Engine
> > > >> +reporting, reconfiguration, performance reporting, and other
> > > >> infrastructure
> > > >> +functions. Each FPGA has one FME, which is always accessed through the
> > > >> physical
> > > >> +function (PF).
> > > >> +
> > > >> +User-space applications can acquire exclusive access to the FME using
> > > >> open(),
> > > >> +and release it using close().
> > > >> +
> > > >> +The following functions are exposed through ioctls:
> > > >> +
> > > >> +       Get driver API version (FPGA_GET_API_VERSION)
> > > >> +       Check for extensions (FPGA_CHECK_EXTENSION)
> > > >> +       Assign port to PF (FPGA_FME_PORT_ASSIGN)
> > > >> +       Release port from PF (FPGA_FME_PORT_RELEASE)
> > > >> +       Program bitstream (FPGA_FME_PORT_PR)
> > > >> +
> > > >> +More functions are exposed through sysfs
> > > >> +(/sys/class/fpga/fpga.n/intel-fpga-fme.n/):
> > > >> +
> > > >> +       Read bitstream ID (bitstream_id)
> > > >> +       Read bitstream metadata (bitstream_metadata)
> > > >> +       Read number of ports (ports_num)
> > > >> +       Read socket ID (socket_id)
> > > >> +       Read performance counters (perf/)
> > > >> +       Power management (power_mgmt/)
> > > >> +       Thermal management (thermal_mgmt/)
> > > >> +       Error reporting (errors/)
> > > >> +
> > > >> +PORT
> > > >> +====
> > > >> +A port represents the interface between the static FPGA fabric (the "blue
> > > >> +bitstream") and a partially reconfigurable region containing an AFU (the
> > > >> "green
> > > 
> > > Is this an fpga bridge but with added features?
> > 
> > Yes, I think so. As you see the fme_pr function in patch 11, related port needs
> > to be disabled firstly before fpga_mgr_buf_load for given accelerator.
> 
> Can we just extend the bridge to have the additional features, please?

As described in this document, in Intel FPGA device, there are two types
of modudles, FME which provides management function including PR and AFU
which provides the access to the FPGA accelerators, and other advanced
features (e.g error reporting, debug, reset and etc) required by user
space applications.

I think FME needs to create fpga-bridge as well as fpga-manager and
regions. Then enable/disable bridge action could be covered automatically
by fpga-region function. (e.g fpga_region_program_fpga).

Thanks
Hao

> 
> > > >> +bitstream"). It controls the communication from SW to the accelerator and
> > > >> +exposes features such as reset and debug.
> > > >> +
> > > >> +A PCIe device may have several ports and each port can be released from
> > > >> PF by
> > > >> +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe
> > > >> sriov
> > > >> +sysfs interface.
> > > >> +
> > > >> +AFU
> > > >> +===
> > > >> +An AFU is attached to a port and exposes a 256k MMIO region to be used
> > > >> for
> > > >> +accelerator-specific control registers.
> > > >> +
> > > >> +User-space applications can acquire exclusive access to an AFU attached
> > > >> to a
> > > >> +port by using open() on the port device node, and release it using
> > > >> close().
> > > >> +
> > > >> +The following functions are exposed through ioctls:
> > > >> +
> > > >> +       Get driver API version (FPGA_GET_API_VERSION)
> > > >> +       Check for extensions (FPGA_CHECK_EXTENSION)
> > > >> +       Get port info (FPGA_PORT_GET_INFO)
> > > >> +       Get MMIO region info (FPGA_PORT_GET_REGION_INFO)
> > > >> +       Map DMA buffer (FPGA_PORT_DMA_MAP)
> > > >> +       Unmap DMA buffer (FPGA_PORT_DMA_UNMAP)
> > > >> +       Reset AFU (FPGA_PORT_RESET)
> > > >> +       Enable UMsg (FPGA_PORT_UMSG_ENABLE)
> > > >> +       Disable UMsg (FPGA_PORT_UMSG_DISABLE)
> > > >> +       Set UMsg mode (FPGA_PORT_UMSG_SET_MODE)
> > > >> +       Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR)
> > > >> +
> > > >> +User-space applications can also mmap() accelerator MMIO regions.
> > > >> +
> > > >> +More functions are exposed through sysfs:
> > > >> +(/sys/class/fpga/fpga.n/intel-fpga-port.m/):
> > > >> +
> > > >> +       Read Accelerator GUID (afu_id)
> > > >> +       Error reporting (errors/)
> > > >> +
> > > >> +Partial Reconfiguration
> > > >> +=======================
> > > >> +As mentioned above, accelerators can be reconfigured through partial
> > > >> +reconfiguration of a green bitstream file (GBS). The green bitstream must
> > > >> have
> > > >> +been generated for the exact blue bitstream and targeted reconfigurable
> > > >> region
> > > >> +(port) of the FPGA; otherwise, the reconfiguration operation will fail
> > > >> and
> > > >> +possibly cause system instability. This compatibility can be checked by
> > > >> +comparing the interface ID noted in the GBS header against the interface
> > > >> ID
> > > >> +exposed by the FME through sysfs (see above). This check is usually done
> > > >> by
> > > >> +user-space before calling the reconfiguration IOCTL.
> > > >> +
> > > >> +FPGA virtualization
> > > >> +===================
> > > >> +To enable accessing an accelerator from applications running in a VM, the
> > > >> +respective AFU's port needs to be assigned to a VF using the following
> > > >> steps:
> > > >> +
> > > >> + a) The PF owns all AFU ports by default. Any port that needs to be
> > > >> reassigned
> > > >> + to a VF must be released from PF firstly through the
> > > >> FPGA_FME_PORT_RELEASE
> > > >> + ioctl on the FME device.
> > > >> +
> > > >> + b) Once N ports are released from PF, then user can use below command to
> > > >> + enable SRIOV and VFs. Each VF owns only one Port with AFU.
> > > >> +
> > > >> + echo N > $PCI_DEVICE_PATH/sriov_numvfs
> > > >> +
> > > >> + c) Pass through the VFs to VMs
> > > >> +
> > > >> + d) The AFU under VF is accessiable from applications in VM (using the
> > > >> same
> > > >> + driver inside the VF).
> > > >> +
> > > >> +Note the an FME can't be assigned to a VF, thus PR and other management
> > > >> +functions are only available via the PF.
> > > >> +
> > > >> +
> > > >> +Driver organization
> > > >> +===================
> > > >> +
> > > >> +  +------------------+  +---------+   |             +---------+
> > > >> +  | +-------+        |  |         |   |             |         |
> > > >> +  | | FPGA  |  FME   |  |   AFU   |   |             |   AFU   |
> > > >> +  | |Manager| Module |  |  Module |   |             |  Module |
> > > >> +  | +-------+        |  |         |   |             |         |
> > > >> +  +------------------+  +---------+   |             +---------+
> > > >> +        +-----------------------+     |      +-----------------------+
> > > >> +        | FPGA Container Device |     |      | FPGA Container Device |
> > > >> +        +-----------------------+     |      +-----------------------+
> > > >> +          +------------------+        |         +------------------+
> > > >> +          | FPGA PCIE Module |        | Virtual | FPGA PCIE Module |
> > > >> +          +------------------+   Host | Machine +------------------+
> > > >> + ------------------------------------ | ------------------------------
> > > >> +           +---------------+          |          +---------------+
> > > >> +           | PCI PF Device |          |          | PCI VF Device |
> > > >> +           +---------------+          |          +---------------+
> > > >> +
> > > >> +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe
> > > >> device
> > > >> +driver is always loaded first once a FPGA PCIE PF or VF device is
> > > >> detected. This
> > > >> +driver plays an infrastructural role in the driver architecuture.  It:
> > > >> +
> > > >> +       a) creates FPGA container device as parent of the feature devices.
> > > >> +       b) walks through the Device Feature List, which is implemented in
> > > >> PCIE
> > > >> +          device BAR memory, to discover feature devices and their sub
> > > >> features
> > > >> +          and create platform device for them under the container device.
> > > >
> > > >
> > > > I really like the idea of creating platform devices for the sub features. It
> > > > is in line with other FPGA use cases.  Platform devices are at the heart of
> > > > device trees used by processors directly connected FPGAs and processors
> > > > inside FPGAs.
> > > >
> > > >> +       c) supports SRIOV.
> > > >> +       d) introduces the feature device infrastructure, which abstracts
> > > >> +          operations for sub features and exposes common functions to
> > > >> feature
> > > >> +          device drivers.
> > > >> +
> > > >> +The FPGA Management Engine (FME) driver is a platform driver which is
> > > >> loaded
> > > >> +automatically after FME platform device creation from the PCIE driver. It
> > > >> +provides the key features for FPGA management, including:
> > > >> +
> > > >> +       a) Power and thermal management, error reporting, performance
> > > >> reporting
> > > >> +          and other infrastructure functions. Users can access these
> > > >> functions
> > > >> +          via sysfs interfaces exposed by FME driver.
> > > >> +       b) Paritial Reconfiguration. The FME driver registers a FPGA
> > > >> Manager
> > > >> +          during PR sub feature initialization; once it receives an
> > > >> +          FPGA_FME_PORT_PR ioctl from user, it invokes the common
> > > >> interface
> > > >> +          function from FPGA Manager to complete the partial
> > > >> reconfiguration of
> > > >> +          the bitstream to the given port.
> > > >> +       c) Port management for virtualization. The FME driver introduces
> > > >> two
> > > >> +          ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and
> > > >> +          FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the
> > > >> port is
> > > >> +          released from the PF, it can be assigned to the VF through the
> > > >> SRIOV
> > > >> +          interfaces provided by PCIE driver. (Refer to "FPGA
> > > >> virtualization"
> > > >> +          for more details).
> > > >> +
> > > >> +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU)
> > > >> driver
> > > >> +is probed once the AFU platform device is created. The main function of
> > > >> this
> > > >> +module is to provide an interface for userspace applications to access
> > > >> the
> > > >> +individual accelerators, including basic reset control on port, AFU MMIO
> > > >> region
> > > >> +export, dma buffer mapping service, UMsg notification, and remote debug
> > > >> +functions (see above).
> > > >> +
> > > >> +
> > > >> +Device enumeration
> > > >> +==================
> > > >> +This section introduces how applications enumerate the fpga device from
> > > >> +the sysfs hierarchy under /sys/class/fpga.
> > > >> +
> > > >> +In the example below, two Intel(R) FPGA devices are installed in the
> > > >> host. Each
> > > >> +fpga device has one FME and two ports (AFUs).
> > > >> +
> > > >> +For each FPGA device, a device director is created under
> > > >> /sys/class/fpga/:
> > > >> +
> > > >> +       /sys/class/fpga/fpga.0
> > > >> +       /sys/class/fpga/fpga.1
> > > >> +
> > > >> +The Intel(R) FPGA device driver exposes "intel-fpga-dev" as the FPGA's
> > > >> name.
> > > >> +Application can retrieve name information via the sysfs interface:
> > > >> +
> > > >> +       /sys/class/fpga/fpga.0/name
> > > >> +
> > > >> +Each node has one FME and two ports (AFUs) as child devices:
> > > >> +
> > > >> +       /sys/class/fpga/fpga.0/intel-fpga-fme.0
> > > >> +       /sys/class/fpga/fpga.0/intel-fpga-port.0
> > > >> +       /sys/class/fpga/fpga.0/intel-fpga-port.1
> > > >> +
> > > >> +       /sys/class/fpga/fpga.1/intel-fpga-fme.1
> > > >> +       /sys/class/fpga/fpga.1/intel-fpga-port.2
> > > >> +       /sys/class/fpga/fpga.1/intel-fpga-port.3
> > > >> +
> > > >> +In general, the FME/AFU sysfs interfaces are named as follows:
> > > >> +
> > > >> +       /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/
> > > >> +       /sys/class/fpga/<fpga.n>/<intel-fpga-port.m>/
> > > >> +
> > > >> +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering
> > > >> all
> > > >> +ports.
> > > >> +
> > > >> +The device nodes used for ioctl() or mmap() can be referenced through:
> > > >> +
> > > >> +       /sys/class/fpga/<fpga.n>/<intel-fpga-port.n>/dev
> > > >> +       /sys/class/fpga/<fpga.n>/<intel-fpga-fme.n>/dev
> > > >> +
> > > >> +
> > > >> +Open discussions
> > > >> +================
> > > >> +The current FME driver does not provide user space access to the FME MMIO
> > > >> +region, but exposes access through sysfs and ioctls. It also provides an
> > > >> FPGA
> > > >> +manger interface for partial reconfiguration (PR), but does not make use
> > > >> of
> > > >> +fpga-regions. User PR requests via the FPGA_FME_PORT_PR ioctl are handled
> > > >> inside
> > > >> +the FME, and fpga-region depends on device tree which is not used at all.
> > > >> There
> > > >> +are patches from Alan Tull to separate the device tree specific code and
> > > >
> > > >
> > > > I am currently trying to use those patches in a different driver.  They've
> > > > compiled cleanly in my out of tree pcie module driver against the 3.10
> > > > kernel.
> > > > I need to actually write the code to create and register the region, but
> > > > Alan's platform driver code should be a good guide for me.  Just need to
> > > > find the time.
> > > >
> > > >> +introduce a sysfs interface for PR. We plan to add fpga-regions support
> > > >> in the
> > > >> +driver once the related patches get merged. Then the FME driver should
> > > >> create
> > > >> +one fpga-region for each Port/AFU.
> > > >
> > > >
> > > > Does the FME driver create the fpga-region, or is each region described as
> > > > an entry in the Device Feature List and therefore created by the code that
> > > > enumerates the Device Feature List?
> > > >
> > > >> --
> > > >> 2.7.4
> > > >>
> > > >> --
> > > >> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> > > >> the body of a message to majordomo@vger.kernel.org
> > > >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > >>
> > > >
> 
> Cheers,
> Moritz

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview
  2017-04-03 20:44           ` Alan Tull
@ 2017-04-04  5:24             ` Wu Hao
  0 siblings, 0 replies; 93+ messages in thread
From: Wu Hao @ 2017-04-04  5:24 UTC (permalink / raw)
  To: Alan Tull
  Cc: Moritz Fischer, matthew.gerlach, Moritz Fischer, linux-fpga,
	linux-kernel, luwei.kang, yi.z.zhang, Enno Luebbers,
	Xiao Guangrong

On Mon, Apr 03, 2017 at 03:44:17PM -0500, Alan Tull wrote:
> On Sun, Apr 2, 2017 at 9:41 AM, Moritz Fischer <mdf@kernel.org> wrote:
> > On Sat, Apr 01, 2017 at 07:16:19PM +0800, Wu Hao wrote:
> >> On Fri, Mar 31, 2017 at 01:38:06PM -0500, Alan Tull wrote:
> >> > On Fri, Mar 31, 2017 at 1:24 PM,  <matthew.gerlach@linux.intel.com> wrote:
> >> > >
> >> > >
> >> > > On Thu, 30 Mar 2017, Wu Hao wrote:
> >> > >
> >> > >
> >> > > Hi Wu Hao,
> >> > >
> >> > > Great documentation. I'm looking forward to diving into the rest of the
> >> > > patches. Please see my comments inline.
> >> > >
> >> > > Matthew Gerlach
> >> > >
> >> > >
> >> > >> Add a document for Intel FPGA driver overview.
> >> > >>
> >> > >> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> >> > >> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> >> > >> Signed-off-by: Wu Hao <hao.wu@intel.com>
> >> > >> ---
> >> > >> Documentation/fpga/intel-fpga.txt | 259
> >> > >> ++++++++++++++++++++++++++++++++++++++
> >> > >> 1 file changed, 259 insertions(+)
> >> > >> create mode 100644 Documentation/fpga/intel-fpga.txt
> >> > >>
> >> > >> diff --git a/Documentation/fpga/intel-fpga.txt
> >> > >> b/Documentation/fpga/intel-fpga.txt
> >> > >> new file mode 100644
> >> > >> index 0000000..9396cea
> >> > >> --- /dev/null
> >> > >> +++ b/Documentation/fpga/intel-fpga.txt
> >> > >> @@ -0,0 +1,259 @@
> >> > >>
> >> > >> +===============================================================================
> >> > >> +                    Intel FPGA driver Overview
> >> > >>
> >> > >> +-------------------------------------------------------------------------------
> >> > >> +                Enno Luebbers <enno.luebbers@intel.com>
> >> > >> +                Xiao Guangrong <guangrong.xiao@linux.intel.com>
> >> > >> +                Wu Hao <hao.wu@intel.com>
> >> > >> +
> >> > >> +The Intel FPGA driver provides interfaces for userspace applications to
> >> > >> +configure, enumerate, open, and access FPGA accelerators on platforms
> >> > >> equipped
> >> > >> +with Intel(R) FPGA solutions and enables system level management
> >> > >> functions such
> >> > >> +as FPGA reconfiguration, power management, and virtualization.
> >> > >> +
> >> > >
> >> > >
> >> > > From a Linux kernel perspective, I'm not sure this is the best name for
> >> > > this code.  The name gives me the impression that it is a driver for all
> >> > > Intel FPGAs, but not all Intel FPGAs are connected to the processor over a
> >> > > PCIe bus.  The processor could be directely connected like the Arria10
> >> > > SOCFPGA.  Such a processor could certainly benefit from this accelerator
> >> > > usage model.  In an extreme case, couldn't a processor in the FPGA,
> >> > > running Linux, also benefit from this accelerator model?  Is this code a
> >> > > "FPGA Accelerator Framework"?
> >> > >
> >> > >> +HW Architecture
> >> > >> +===============
> >> > >> +From the OS's point of view, the FPGA hardware appears as a regular PCIe
> >> > >> device.
> >> > >> +The FPGA device memory is organized using a predefined data structure
> >> > >> (Device
> >> > >> +Feature List). Features supported by the particular FPGA device are
> >> > >> exposed
> >> > >> +through these data structures, as illustrated below:
> >> > >> +
> >> > >> +  +-------------------------------+  +-------------+
> >> > >> +  |              PF               |  |     VF      |
> >> > >> +  +-------------------------------+  +-------------+
> >> > >> +      ^            ^         ^              ^
> >> > >> +      |            |         |              |
> >> > >> ++-----|------------|---------|--------------|-------+
> >> > >> +|     |            |         |              |       |
> >> > >> +|  +-----+     +-------+ +-------+      +-------+   |
> >> > >> +|  | FME |     | Port0 | | Port1 |      | Port2 |   |
> >> > >> +|  +-----+     +-------+ +-------+      +-------+   |
> >> > >> +|                  ^         ^              ^       |
> >> > >> +|                  |         |              |       |
> >> > >> +|              +-------+ +------+       +-------+   |
> >> > >> +|              |  AFU  | |  AFU |       |  AFU  |   |
> >> > >> +|              +-------+ +------+       +-------+   |
> >> > >> +|                                                   |
> >> > >> +|                 FPGA PCIe Device                  |
> >> > >> ++---------------------------------------------------+
> >> > >> +
> >> > >> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which
> >> > >> can be
> >> > >> +used to assign individual accelerators to virtual machines .
> >> > >
> >> > >
> >> > > Does this HW Architecture require an Intel FPGA?  Couldn't any vendors FPGA
> >> > > be used as long as it presented itself the PCIe bus the same and contained
> >> > > an appropriate Device Feature List?
> >
> > I think this is a good (and important) point. Especially when sysfs
> > entries & ioctls constituting ABI depend on it.
> >
> >> > >
> >> > >> +
> >> > >> +FME (FPGA Management Engine)
> >> > >> +============================
> >> > >> +The FPGA Management Enging performs power and thermal management, error
> > Enging->Engine
> >> > >> +reporting, reconfiguration, performance reporting, and other
> >> > >> infrastructure
> >> > >> +functions. Each FPGA has one FME, which is always accessed through the
> >> > >> physical
> >> > >> +function (PF).
> >> > >> +
> >> > >> +User-space applications can acquire exclusive access to the FME using
> >> > >> open(),
> >> > >> +and release it using close().
> >> > >> +
> >> > >> +The following functions are exposed through ioctls:
> >> > >> +
> >> > >> +       Get driver API version (FPGA_GET_API_VERSION)
> >> > >> +       Check for extensions (FPGA_CHECK_EXTENSION)
> >> > >> +       Assign port to PF (FPGA_FME_PORT_ASSIGN)
> >> > >> +       Release port from PF (FPGA_FME_PORT_RELEASE)
> >> > >> +       Program bitstream (FPGA_FME_PORT_PR)
> >> > >> +
> >> > >> +More functions are exposed through sysfs
> >> > >> +(/sys/class/fpga/fpga.n/intel-fpga-fme.n/):
> >> > >> +
> >> > >> +       Read bitstream ID (bitstream_id)
> >> > >> +       Read bitstream metadata (bitstream_metadata)
> >> > >> +       Read number of ports (ports_num)
> >> > >> +       Read socket ID (socket_id)
> >> > >> +       Read performance counters (perf/)
> >> > >> +       Power management (power_mgmt/)
> >> > >> +       Thermal management (thermal_mgmt/)
> >> > >> +       Error reporting (errors/)
> >> > >> +
> >> > >> +PORT
> >> > >> +====
> >> > >> +A port represents the interface between the static FPGA fabric (the "blue
> >> > >> +bitstream") and a partially reconfigurable region containing an AFU (the
> >> > >> "green
> >> >
> >> > Is this an fpga bridge but with added features?
> >>
> >> Yes, I think so. As you see the fme_pr function in patch 11, related port needs
> >> to be disabled firstly before fpga_mgr_buf_load for given accelerator.
> >
> > Can we just extend the bridge to have the additional features, please?
> 
> OK then this code is taking place of a fpga-region that controls the
> bridge (port) and fpga-mgr during fpga programming.
> 

As mentioned in last email replied to Moritz, I prefer to have fpga-bridge
in FME module together with fpga-region and fpga-manager, and reuse fpga
region related function for PR. Other functions which required by user
space applications when access the FPGA acclerator, should be covered in
AFU driver.

Please notice that In VF case (e.g in virtual machine), there is no FME
at all, but only FPGA accelerators (AFUs). Create a duplciate fpga-bridge
in AFU driver seems not useful.

Thanks
Hao

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-04-03 20:26         ` Alan Tull
@ 2017-04-04  5:25           ` Wu Hao
  0 siblings, 0 replies; 93+ messages in thread
From: Wu Hao @ 2017-04-04  5:25 UTC (permalink / raw)
  To: Alan Tull
  Cc: Moritz Fischer, linux-fpga, linux-kernel, luwei.kang, yi.z.zhang,
	Tim Whisonant, Enno Luebbers, Shiva Rao, Christopher Rauer,
	Xiao Guangrong, xiaoguangrong.eric

On Mon, Apr 03, 2017 at 03:26:14PM -0500, Alan Tull wrote:
> On Fri, Mar 31, 2017 at 3:50 AM, Wu Hao <hao.wu@intel.com> wrote:
> > On Fri, Mar 31, 2017 at 12:11:12PM +0800, Xiao Guangrong wrote:
> >> On 31/03/2017 4:30 AM, Alan Tull wrote:
> >> >On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote:
> >> >>From: Kang Luwei <luwei.kang@intel.com>
> >> >>
> >> >>Partial Reconfiguration (PR) is the most important function for FME. It
> >> >>allows reconfiguration for given Port/Accelerated Function Unit (AFU).
> >> >>
> >> >>This patch adds support for PR sub feature. In this patch, it registers
> >> >>a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
> >> >>for PR operation once PR request received via ioctl. Below user space
> >> >>interfaces are exposed by this sub feature.
> >> >>
> >> >>Sysfs interface:
> >> >>* /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
> >> >>  Read-only. Indicate the hardware interface information. Userspace
> >> >>  applications need to check this interface to select correct green
> >> >>  bitstream format before PR.
> >> >>
> >> >>Ioctl interface:
> >> >>* FPGA_FME_PORT_PR
> >> >>  Do partial reconfiguration per information from userspace, including
> >> >>  target port(AFU), buffer size and address info. It returns the PR status
> >> >>  (PR error code if failed) to userspace.
> >> >>
> >> >>Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> >> >>Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> >> >>Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> >> >>Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> >> >>Signed-off-by: Alan Tull <alan.tull@intel.com>
> >> >
> >> >Hi Wu Hao,
> >> >
> >> >Thanks for submitting your patches.
> >> >
> >> >I think there's been a misunderstanding of the meaning of
> >> >'Signed-off-by' [1].  I have not signed off on this code or had a hand
> >> >in its development.  But I'm happy to get to review it now.  It will
> >> >take a bit of time; I expect to be replying next week.
> >>
> >> Hi Alan,
> >>
> >> Sorry to confuse you, i think it's because you helped Chris a lot to
> >> implement this interface and we'd like to include your credit as this
> >> way. If you dislike, it will be dropped. :)
> >>
> >> Thanks for your review in advance.
> >>
> >
> > Hi Alan,
> >
> > Sorry about this, we should ask you firstly before doing it this way.
> > Let me know if you don't like it, I will drop it in the next version.
> 
> Yes please drop the signed-off-by: me in the next version.  Also, you
> don't need to cc my Intel email address.
> 

Sure, will do.

Hao

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-04-03 16:30       ` Alan Tull
@ 2017-04-04  6:05         ` Wu Hao
  2017-04-04 22:37           ` Alan Tull
  0 siblings, 1 reply; 93+ messages in thread
From: Wu Hao @ 2017-04-04  6:05 UTC (permalink / raw)
  To: Alan Tull
  Cc: Moritz Fischer, linux-fpga, linux-kernel, luwei.kang, yi.z.zhang,
	Tim Whisonant, Enno Luebbers, Shiva Rao, Christopher Rauer,
	Alan Tull, Xiao Guangrong

On Mon, Apr 03, 2017 at 11:30:55AM -0500, Alan Tull wrote:
> On Sat, Apr 1, 2017 at 6:08 AM, Wu Hao <hao.wu@intel.com> wrote:
> > On Fri, Mar 31, 2017 at 02:10:12PM -0500, Alan Tull wrote:
> >> On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote:
> >> > From: Kang Luwei <luwei.kang@intel.com>
> >> >
> >> > Partial Reconfiguration (PR) is the most important function for FME. It
> >> > allows reconfiguration for given Port/Accelerated Function Unit (AFU).
> >> >
> >> > This patch adds support for PR sub feature. In this patch, it registers
> >> > a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
> >> > for PR operation once PR request received via ioctl. Below user space
> >> > interfaces are exposed by this sub feature.
> >> >
> >> > Sysfs interface:
> >> > * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
> >> >   Read-only. Indicate the hardware interface information. Userspace
> >> >   applications need to check this interface to select correct green
> >> >   bitstream format before PR.
> >> >
> >> > Ioctl interface:
> >> > * FPGA_FME_PORT_PR
> >> >   Do partial reconfiguration per information from userspace, including
> >> >   target port(AFU), buffer size and address info. It returns the PR status
> >> >   (PR error code if failed) to userspace.
> >> >
> >> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> >> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> >> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> >> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> >> > Signed-off-by: Alan Tull <alan.tull@intel.com>
> >> > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> >> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> >> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> >> > ---
> >> >  drivers/fpga/intel/Makefile      |   2 +-
> >> >  drivers/fpga/intel/feature-dev.h |  58 ++++++
> >> >  drivers/fpga/intel/fme-main.c    |  44 ++++-
> >> >  drivers/fpga/intel/fme-pr.c      | 400 +++++++++++++++++++++++++++++++++++++++
> >> >  drivers/fpga/intel/fme.h         |  32 ++++
> >> >  include/uapi/linux/intel-fpga.h  |  44 +++++
> >> >  6 files changed, 578 insertions(+), 2 deletions(-)
> >> >  create mode 100644 drivers/fpga/intel/fme-pr.c
> >> >  create mode 100644 drivers/fpga/intel/fme.h
> >> >
> >> > diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
> >> > index 546861d..0452cb6 100644
> >> > --- a/drivers/fpga/intel/Makefile
> >> > +++ b/drivers/fpga/intel/Makefile
> >> > @@ -2,4 +2,4 @@ obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
> >> >  obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
> >> >
> >> >  intel-fpga-pci-objs := pcie.o feature-dev.o
> >> > -intel-fpga-fme-objs := fme-main.o
> >> > +intel-fpga-fme-objs := fme-main.o fme-pr.o
> >> > diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
> >> > index dccc283..5a25c915 100644
> >> > --- a/drivers/fpga/intel/feature-dev.h
> >> > +++ b/drivers/fpga/intel/feature-dev.h
> >> > @@ -150,8 +150,66 @@ struct feature_fme_err {
> >> >  };
> >> >
> >> >  /* FME Partial Reconfiguration Sub Feature Register Set */
> >> > +/* FME PR Control Register */
> >> > +struct feature_fme_pr_ctl {
> >> > +       union {
> >> > +               u64 csr;
> >> > +               struct {
> >> > +                       u8  pr_reset:1;         /* Reset PR Engine */
> >> > +                       u8  rsvdz1:3;
> >> > +                       u8  pr_reset_ack:1;     /* Reset PR Engine Ack */
> >> > +                       u8  rsvdz2:3;
> >> > +                       u8  pr_regionid:2;      /* PR Region ID */
> >> > +                       u8  rsvdz3:2;
> >> > +                       u8  pr_start_req:1;     /* PR Start Request */
> >> > +                       u8  pr_push_complete:1; /* PR Data push complete */
> >> > +                       u8  pr_kind:1;          /* Load Customer or Intel GBS */
> >> > +                       u32 rsvdz4:17;
> >> > +                       u32 config_data;
> >> > +               };
> >> > +       };
> >> > +};
> >> > +
> >> > +/* FME PR Status Register */
> >> > +struct feature_fme_pr_status {
> >> > +       union {
> >> > +               u64 csr;
> >> > +               struct {
> >> > +                       u16 pr_credit:9;        /* Number of PR Credits */
> >> > +                       u8  rsvdz1:7;
> >> > +                       u8  pr_status:1;        /* PR Operation status */
> >> > +                       u8  rsvdz2:3;
> >> > +                       u8  pr_ctrlr_status:3;  /* Controller status */
> >> > +                       u8  rsvdz3:1;
> >> > +                       u8  pr_host_status:4;   /* PR Host status */
> >> > +                       u64 rsvdz4:36;
> >> > +               };
> >> > +       };
> >> > +};
> >> > +
> >> > +/* FME PR Data Register */
> >> > +struct feature_fme_pr_data {
> >> > +       union {
> >> > +               u64 csr;
> >> > +               struct {
> >> > +                       /* PR data from the raw-binary file */
> >> > +                       u32 pr_data_raw;
> >> > +                       u32 rsvd;
> >> > +               };
> >> > +       };
> >> > +};
> >> > +
> >> >  struct feature_fme_pr {
> >> >         struct feature_header header;
> >> > +       struct feature_fme_pr_ctl control;
> >> > +       struct feature_fme_pr_status status;
> >> > +       struct feature_fme_pr_data data;
> >> > +       u64 error;
> >> > +
> >> > +       u64 rsvd[16];
> >> > +
> >> > +       u64 intfc_id_l;         /* PR interface Id Low */
> >> > +       u64 intfc_id_h;         /* PR interface Id High */
> >> >  };
> >> >
> >> >  /* PORT Header Register Set */
> >> > diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
> >> > index 36d0c4c..0d9a7a6 100644
> >> > --- a/drivers/fpga/intel/fme-main.c
> >> > +++ b/drivers/fpga/intel/fme-main.c
> >> > @@ -23,6 +23,7 @@
> >> >  #include <linux/intel-fpga.h>
> >> >
> >> >  #include "feature-dev.h"
> >> > +#include "fme.h"
> >> >
> >> >  static ssize_t ports_num_show(struct device *dev,
> >> >                               struct device_attribute *attr, char *buf)
> >> > @@ -101,6 +102,10 @@ static struct feature_driver fme_feature_drvs[] = {
> >> >                 .ops = &fme_hdr_ops,
> >> >         },
> >> >         {
> >> > +               .name = FME_FEATURE_PR_MGMT,
> >> > +               .ops = &pr_mgmt_ops,
> >> > +       },
> >> > +       {
> >> >                 .ops = NULL,
> >> >         },
> >> >  };
> >> > @@ -182,14 +187,48 @@ static const struct file_operations fme_fops = {
> >> >         .unlocked_ioctl = fme_ioctl,
> >> >  };
> >> >
> >> > +static int fme_dev_init(struct platform_device *pdev)
> >> > +{
> >> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >> > +       struct fpga_fme *fme;
> >> > +
> >> > +       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
> >> > +       if (!fme)
> >> > +               return -ENOMEM;
> >> > +
> >> > +       fme->pdata = pdata;
> >> > +
> >> > +       mutex_lock(&pdata->lock);
> >> > +       fpga_pdata_set_private(pdata, fme);
> >> > +       mutex_unlock(&pdata->lock);
> >> > +       return 0;
> >> > +}
> >> > +
> >> > +static void fme_dev_destroy(struct platform_device *pdev)
> >> > +{
> >> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >> > +       struct fpga_fme *fme;
> >> > +
> >> > +       mutex_lock(&pdata->lock);
> >> > +       fme = fpga_pdata_get_private(pdata);
> >> > +       fpga_pdata_set_private(pdata, NULL);
> >> > +       mutex_unlock(&pdata->lock);
> >> > +
> >> > +       devm_kfree(&pdev->dev, fme);
> >> > +}
> >> > +
> >> >  static int fme_probe(struct platform_device *pdev)
> >> >  {
> >> >         int ret;
> >> >
> >> > -       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> >> > +       ret = fme_dev_init(pdev);
> >> >         if (ret)
> >> >                 goto exit;
> >> >
> >> > +       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> >> > +       if (ret)
> >> > +               goto dev_destroy;
> >> > +
> >> >         ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
> >> >         if (ret)
> >> >                 goto feature_uinit;
> >> > @@ -198,6 +237,8 @@ static int fme_probe(struct platform_device *pdev)
> >> >
> >> >  feature_uinit:
> >> >         fpga_dev_feature_uinit(pdev);
> >> > +dev_destroy:
> >> > +       fme_dev_destroy(pdev);
> >> >  exit:
> >> >         return ret;
> >> >  }
> >> > @@ -206,6 +247,7 @@ static int fme_remove(struct platform_device *pdev)
> >> >  {
> >> >         fpga_dev_feature_uinit(pdev);
> >> >         fpga_unregister_dev_ops(pdev);
> >> > +       fme_dev_destroy(pdev);
> >> >         return 0;
> >> >  }
> >> >
> >> > diff --git a/drivers/fpga/intel/fme-pr.c b/drivers/fpga/intel/fme-pr.c
> >> > new file mode 100644
> >> > index 0000000..3b44a3e
> >> > --- /dev/null
> >> > +++ b/drivers/fpga/intel/fme-pr.c
> >> > @@ -0,0 +1,400 @@
> >> > +/*
> >> > + * Driver for Intel FPGA Management Engine (FME) Partial Reconfiguration
> >> > + *
> >> > + * Copyright (C) 2017 Intel Corporation, Inc.
> >> > + *
> >> > + * Authors:
> >> > + *   Kang Luwei <luwei.kang@intel.com>
> >> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> >> > + *   Joseph Grecco <joe.grecco@intel.com>
> >> > + *   Enno Luebbers <enno.luebbers@intel.com>
> >> > + *   Tim Whisonant <tim.whisonant@intel.com>
> >> > + *   Ananda Ravuri <ananda.ravuri@intel.com>
> >> > + *   Christopher Rauer <christopher.rauer@intel.com>
> >> > + *   Henry Mitchel <henry.mitchel@intel.com>
> >> > + *
> >> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
> >> > + * redistributing this file, you may do so under either license. See the
> >> > + * LICENSE.BSD file under this directory for the BSD license and see
> >> > + * the COPYING file in the top-level directory for the GPLv2 license.
> >> > + */
> >> > +
> >> > +#include <linux/types.h>
> >> > +#include <linux/device.h>
> >> > +#include <linux/vmalloc.h>
> >> > +#include <linux/uaccess.h>
> >> > +#include <linux/fpga/fpga-mgr.h>
> >> > +#include <linux/intel-fpga.h>
> >> > +
> >> > +#include "feature-dev.h"
> >> > +#include "fme.h"
> >> > +
> >> > +#define PR_WAIT_TIMEOUT   8000000
> >> > +
> >> > +#define PR_HOST_STATUS_IDLE    0
> >> > +
> >> > +DEFINE_FPGA_PR_ERR_MSG(pr_err_msg);
> >> > +
> >> > +static ssize_t interface_id_show(struct device *dev,
> >> > +                                struct device_attribute *attr, char *buf)
> >> > +{
> >> > +       u64 intfc_id_l, intfc_id_h;
> >> > +       struct feature_fme_pr *fme_pr
> >> > +               = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_PR_MGMT);
> >> > +
> >> > +       intfc_id_l = readq(&fme_pr->intfc_id_l);
> >> > +       intfc_id_h = readq(&fme_pr->intfc_id_h);
> >> > +
> >> > +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> >> > +                       (unsigned long long)intfc_id_h,
> >> > +                       (unsigned long long)intfc_id_l);
> >> > +}
> >> > +static DEVICE_ATTR_RO(interface_id);
> >> > +
> >> > +static struct attribute *pr_mgmt_attrs[] = {
> >> > +       &dev_attr_interface_id.attr,
> >> > +       NULL,
> >> > +};
> >> > +
> >> > +struct attribute_group pr_mgmt_attr_group = {
> >> > +       .attrs  = pr_mgmt_attrs,
> >> > +       .name   = "pr",
> >> > +};
> >> > +
> >> > +static u64
> >> > +pr_err_handle(struct platform_device *pdev, struct feature_fme_pr *fme_pr)
> >> > +{
> >> > +       struct feature_fme_pr_status fme_pr_status;
> >> > +       unsigned long err_code;
> >> > +       u64 fme_pr_error;
> >> > +       int i = 0;
> >> > +
> >> > +       fme_pr_status.csr = readq(&fme_pr->status);
> >> > +       if (!fme_pr_status.pr_status)
> >> > +               return 0;
> >> > +
> >> > +       err_code = fme_pr_error = readq(&fme_pr->error);
> >> > +       for_each_set_bit(i, &err_code, PR_MAX_ERR_NUM)
> >> > +               dev_dbg(&pdev->dev, "%s\n", pr_err_msg[i]);
> >> > +       writeq(fme_pr_error, &fme_pr->error);
> >> > +       return fme_pr_error;
> >> > +}
> >> > +
> >> > +static int fme_pr_write_init(struct fpga_manager *mgr,
> >> > +               struct fpga_image_info *info, const char *buf, size_t count)
> >> > +{
> >> > +       struct fpga_fme *fme = mgr->priv;
> >> > +       struct platform_device *pdev;
> >> > +       struct feature_fme_pr *fme_pr;
> >> > +       struct feature_fme_pr_ctl fme_pr_ctl;
> >> > +       struct feature_fme_pr_status fme_pr_status;
> >> > +
> >> > +       pdev = fme->pdata->dev;
> >> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> >> > +                               FME_FEATURE_ID_PR_MGMT);
> >> > +       if (!fme_pr)
> >> > +               return -EINVAL;
> >> > +
> >> > +       if (WARN_ON(info->flags != FPGA_MGR_PARTIAL_RECONFIG))
> >> > +               return -EINVAL;
> >> > +
> >> > +       dev_dbg(&pdev->dev, "resetting PR before initiated PR\n");
> >> > +
> >> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> >> > +       fme_pr_ctl.pr_reset = 1;
> >> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> >> > +
> >> > +       fme_pr_ctl.pr_reset_ack = 1;
> >> > +
> >> > +       if (fpga_wait_register_field(pr_reset_ack, fme_pr_ctl,
> >> > +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> >> > +               dev_err(&pdev->dev, "maximum PR timeout\n");
> >> > +               return -ETIMEDOUT;
> >> > +       }
> >> > +
> >> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> >> > +       fme_pr_ctl.pr_reset = 0;
> >> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> >> > +
> >> > +       dev_dbg(&pdev->dev,
> >> > +               "waiting for PR resource in HW to be initialized and ready\n");
> >> > +
> >> > +       fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
> >> > +
> >> > +       if (fpga_wait_register_field(pr_host_status, fme_pr_status,
> >> > +                                    &fme_pr->status, PR_WAIT_TIMEOUT, 1)) {
> >> > +               dev_err(&pdev->dev, "maximum PR timeout\n");
> >> > +               return -ETIMEDOUT;
> >> > +       }
> >> > +
> >> > +       dev_dbg(&pdev->dev, "check if have any previous PR error\n");
> >> > +       pr_err_handle(pdev, fme_pr);
> >> > +       return 0;
> >> > +}
> >> > +
> >> > +static int fme_pr_write(struct fpga_manager *mgr,
> >> > +                       const char *buf, size_t count)
> >> > +{
> >> > +       struct fpga_fme *fme = mgr->priv;
> >> > +       struct platform_device *pdev;
> >> > +       struct feature_fme_pr *fme_pr;
> >> > +       struct feature_fme_pr_ctl fme_pr_ctl;
> >> > +       struct feature_fme_pr_status fme_pr_status;
> >> > +       struct feature_fme_pr_data fme_pr_data;
> >> > +       int delay, pr_credit, i = 0;
> >> > +
> >> > +       pdev = fme->pdata->dev;
> >> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> >> > +                               FME_FEATURE_ID_PR_MGMT);
> >> > +
> >> > +       dev_dbg(&pdev->dev, "set PR port ID and start request\n");
> >> > +
> >> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> >> > +       fme_pr_ctl.pr_regionid = fme->port_id;
> >> > +       fme_pr_ctl.pr_start_req = 1;
> >> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> >> > +
> >> > +       dev_dbg(&pdev->dev, "pushing data from bitstream to HW\n");
> >> > +
> >> > +       fme_pr_status.csr = readq(&fme_pr->status);
> >> > +       pr_credit = fme_pr_status.pr_credit;
> >> > +
> >> > +       while (count > 0) {
> >> > +               delay = 0;
> >> > +               while (pr_credit <= 1) {
> >> > +                       if (delay++ > PR_WAIT_TIMEOUT) {
> >> > +                               dev_err(&pdev->dev, "maximum try\n");
> >> > +                               return -ETIMEDOUT;
> >> > +                       }
> >> > +                       udelay(1);
> >> > +
> >> > +                       fme_pr_status.csr = readq(&fme_pr->status);
> >> > +                       pr_credit = fme_pr_status.pr_credit;
> >> > +               };
> >> > +
> >> > +               if (count >= 4) {
> >> > +                       fme_pr_data.rsvd = 0;
> >> > +                       fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
> >> > +                       writeq(fme_pr_data.csr, &fme_pr->data);
> >> > +                       count -= 4;
> >> > +                       pr_credit--;
> >> > +                       i++;
> >> > +               } else {
> >> > +                       WARN_ON(1);
> >> > +                       return -EINVAL;
> >> > +               }
> >> > +       }
> >> > +
> >> > +       return 0;
> >> > +}
> >> > +
> >> > +static int fme_pr_write_complete(struct fpga_manager *mgr,
> >> > +                       struct fpga_image_info *info)
> >> > +{
> >> > +       struct fpga_fme *fme = mgr->priv;
> >> > +       struct platform_device *pdev;
> >> > +       struct feature_fme_pr *fme_pr;
> >> > +       struct feature_fme_pr_ctl fme_pr_ctl;
> >> > +
> >> > +       pdev = fme->pdata->dev;
> >> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> >> > +                               FME_FEATURE_ID_PR_MGMT);
> >> > +
> >> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> >> > +       fme_pr_ctl.pr_push_complete = 1;
> >> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> >> > +
> >> > +       dev_dbg(&pdev->dev, "green bitstream push complete\n");
> >> > +       dev_dbg(&pdev->dev, "waiting for HW to release PR resource\n");
> >> > +
> >> > +       fme_pr_ctl.pr_start_req = 0;
> >> > +
> >> > +       if (fpga_wait_register_field(pr_start_req, fme_pr_ctl,
> >> > +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> >> > +               dev_err(&pdev->dev, "maximum try.\n");
> >> > +               return -ETIMEDOUT;
> >> > +       }
> >> > +
> >> > +       dev_dbg(&pdev->dev, "PR operation complete, checking status\n");
> >> > +       fme->pr_err = pr_err_handle(pdev, fme_pr);
> >> > +       if (fme->pr_err)
> >> > +               return -EIO;
> >> > +
> >> > +       dev_dbg(&pdev->dev, "PR done successfully\n");
> >> > +       return 0;
> >> > +}
> >> > +
> >> > +static enum fpga_mgr_states fme_pr_state(struct fpga_manager *mgr)
> >> > +{
> >> > +       return FPGA_MGR_STATE_UNKNOWN;
> >> > +}
> >> > +
> >> > +static const struct fpga_manager_ops fme_pr_ops = {
> >> > +       .write_init = fme_pr_write_init,
> >> > +       .write = fme_pr_write,
> >> > +       .write_complete = fme_pr_write_complete,
> >> > +       .state = fme_pr_state,
> >> > +};
> >> > +
> >> > +static int fme_pr(struct platform_device *pdev, unsigned long arg)
> >> > +{
> >> > +       void __user *argp = (void __user *)arg;
> >> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >> > +       struct fpga_fme *fme;
> >> > +       struct fpga_manager *mgr;
> >> > +       struct feature_fme_header *fme_hdr;
> >> > +       struct feature_fme_capability fme_capability;
> >> > +       struct fpga_image_info info;
> >> > +       struct fpga_fme_port_pr port_pr;
> >> > +       struct platform_device *port;
> >> > +       unsigned long minsz;
> >> > +       void *buf = NULL;
> >> > +       int ret = 0;
> >> > +
> >> > +       minsz = offsetofend(struct fpga_fme_port_pr, status);
> >> > +
> >> > +       if (copy_from_user(&port_pr, argp, minsz))
> >> > +               return -EFAULT;
> >> > +
> >> > +       if (port_pr.argsz < minsz || port_pr.flags)
> >> > +               return -EINVAL;
> >> > +
> >> > +       if (!IS_ALIGNED(port_pr.buffer_size, 4))
> >> > +               return -EINVAL;
> >> > +
> >> > +       /* get fme header region */
> >> > +       fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> >> > +                                       FME_FEATURE_ID_HEADER);
> >> > +       if (WARN_ON(!fme_hdr))
> >> > +               return -EINVAL;
> >> > +
> >> > +       /* check port id */
> >> > +       fme_capability.csr = readq(&fme_hdr->capability);
> >> > +       if (port_pr.port_id >= fme_capability.num_ports) {
> >> > +               dev_dbg(&pdev->dev, "port number more than maximum\n");
> >> > +               return -EINVAL;
> >> > +       }
> >> > +
> >> > +       if (!access_ok(VERIFY_READ, port_pr.buffer_address,
> >> > +                                   port_pr.buffer_size))
> >> > +               return -EFAULT;
> >> > +
> >> > +       buf = vmalloc(port_pr.buffer_size);
> >> > +       if (!buf)
> >> > +               return -ENOMEM;
> >> > +
> >> > +       if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
> >> > +                                              port_pr.buffer_size)) {
> >> > +               ret = -EFAULT;
> >> > +               goto free_exit;
> >> > +       }
> >> > +
> >> > +       memset(&info, 0, sizeof(struct fpga_image_info));
> >> > +       info.flags = FPGA_MGR_PARTIAL_RECONFIG;
> >> > +
> >> > +       mgr = fpga_mgr_get(&pdev->dev);
> >> > +       if (IS_ERR(mgr)) {
> >> > +               ret = PTR_ERR(mgr);
> >> > +               goto free_exit;
> >> > +       }
> >> > +
> >> > +       mutex_lock(&pdata->lock);
> >> > +       fme = fpga_pdata_get_private(pdata);
> >> > +       /* fme device has been unregistered. */
> >> > +       if (!fme) {
> >> > +               ret = -EINVAL;
> >> > +               goto unlock_exit;
> >> > +       }
> >> > +
> >> > +       fme->pr_err = 0;
> >> > +       fme->port_id = port_pr.port_id;
> >>
> >> It looks like you're using private data to communicate with the
> >> driver, i.e. there is something you want to do with the fpga manager
> >> framework and it doesn't have that feature.  The better way would be
> >> for us to expand the framework so you don't need to do that.
> >>
> >> port_id is the kind of thing that should be communicated to the driver
> >> through fpga_image_info, so we could add that to the struct.  Should
> >> we call it port_id?
> 
> Let's call it region_id.
> 
> >> Or is there something more generic that may be
> >> useful in the future for other architectures?.
> >
> > Hi Alan
> >
> > Thanks for your feedback. :)
> >
> > As you know, each Intel FPGA device may have more than one accelerator,
> > and all accelerators share the same fpga_manager (only one FME module).
> > port_id = 0 means the first accelerator on this fpga devices. So it's
> > needed by the fpga_manager to distinguish one accelerator from others
> > for partial reconfiguration.
> >
> > Adding a member like a 'id' to fpga_image_info definitely works for us
> > in this case. We can add it this way, but I feel 'id' here seems not
> > related to fpga image, but characteristic of fpga region.
> 
> The  fpga_image_info struct started life as just image specific info,
> but I want it to go in the direction of including parameters needed to
> program it this specific time. Otherwise we are stuck having to keep
> adding parameters as our use of FPGA develops.  It probably could be
> documented better as 'information needed to program a FPGA image'
> rather than strictly 'information about this particular FPGA image'.
> My patch "fpga-mgr: pass parameters for loading fpga in image info"
> goes in this direction by having the buf, firmware name, or sg list
> passed in the info for the added fpga_mgr_load() function.  Actually I
> should probably simplify the API and get rid of fpga_mgr_buf_load,
> fpga_mgr_buf_load_sg, and fpga_mgr_firmware_load and require people to
> use fpga_mgr_load (passing all parameters in fpga_image_info).
>

Make sense.
 
> > It may be a
> > little confusing. One rough idea is that keep this info under fpga region
> > (maybe its private data), and pass the fpga-region to fpga_mgr_buf_load,
> 
> Yes, keep this info in fpga-region.  When the region wants to program
> using fpga-mgr, add the region id to fpga_image_info.  I propose
> calling it region_id.

Hm.. Do we need a function which moves info from region to image info?

Another idea is, add a priv to fpga_image_info, and use a common function
to pass the fpga_region's priv to fpga_image_info's priv before PR.
fpga-mgr then knows fpga_region priv info from the fpga_image_info.

> 
> > then fpga_manager knows the target region for partial reconfiguration.
> > If consider pr sysfs interface support under fpga-region in the future,
> > then we don't need to create a new 'id' sysfs file, as fpga-region itself
> > could provide this kind of info. But I'm not sure if this is the right
> > direction.
> >
> >>
> >> pr_err appears to be machine specific error codes that are
> >> communicated outside your low level driver.  (Calling it pr_err is
> >> extra confusing since Linux already has a commonly name function by
> >> the same name).  The framework has state, but that's not doing what
> >> you want here.  Maybe we could add a framework ops called status so
> >> that status could be communicated from the low level driver. It would
> >> be useful to abstract the machine specific state to a defined enum
> >> that would be part of the fpga mgr framework.  I remember we had
> >> something like that in the earliest version of fpga manager but it got
> >> changed to state rather than status for some reason.
> >>
> >
> > Adding a status function and a 'status' member to fpga manager sounds good.
> > But I'm not sure how to abstract these error codes, currently what we have
> > are all PR error codes for Intel FPGA device, e.g "PR FIFO overflow error",
> > "PR secure load error" and etc (see include/uapi/linux/intel-fpga.h). Is it
> > fine to let each driver to define how to use that 'status' for machine
> > specific status?
> 
> I looked at the list of errors in include/uapi/linux/intel-fpga.h.
> They all seem pretty generic to me except I am not clear what "secure
> load error" or "IP protocol error" mean and whether other
> architectures would have them.  But certainly things like crc error,
> incompatible bitstream, fifo overflow are generic.  So let's see if we
> can define all these in a way that's generic and just pass up a error
> number.  That way upper layers can know how to deal with them
> possibly.  I would take the word "PR" off these since the error set
> applies whether someone is doing full reconfig or partial reconfig.

Sure, good to me, we can make it this way and see.
Thanks for the suggestions. :)

Hao

> 
> >
> >> mgr->dev won't work for you for some of the things you are using fme->pdev for?
> >>
> >
> > Does 'fme->pdev' mean the platform device for fme? I think we use platform
> > device to represent the FME module, and FME module may have multiple sub
> > features, driver discovers these sub features via the 'Device Feature List'
> > in the PCIe Device Bar. PR is only one of the sub features under FME module,
> > if any FME module doesn't have PR sub feature, fpga-manager will not be
> > created at all. But this will not impact other sub features, e.g thermal
> > management, error reporting and etc, to create their own interfaces under
> > the platform device.
> 
> If we need it for private data, that's may be actually OK.   Just
> wondering whether the priv was needed.   I think that's the one
> element in the private data struct that was used privately instead of
> being used to pass info outside the framework.
> 
> >
> > Thanks
> > Hao
> >
> >> Alan
> >>
> >> > +
> >> > +       /* Find and get port device by index */
> >> > +       port = pdata->fpga_for_each_port(pdev, &fme->port_id,
> >> > +                                        fpga_port_check_id);
> >> > +       WARN_ON(!port);
> >> > +
> >> > +       /* Disable Port before PR */
> >> > +       fpga_port_disable(port);
> >> > +
> >> > +       ret = fpga_mgr_buf_load(mgr, &info, buf, port_pr.buffer_size);
> >> > +       port_pr.status = fme->pr_err;
> >> > +
> >> > +       /* Re-enable Port after PR finished */
> >> > +       fpga_port_enable(port);
> >> > +
> >> > +       put_device(&port->dev);
> >> > +
> >> > +unlock_exit:
> >> > +       mutex_unlock(&pdata->lock);
> >> > +       fpga_mgr_put(mgr);
> >> > +free_exit:
> >> > +       vfree(buf);
> >> > +       if (copy_to_user((void __user *)arg, &port_pr, minsz))
> >> > +               return -EFAULT;
> >> > +       return ret;
> >> > +}
> >> > +
> >> > +static int fpga_fme_pr_probe(struct platform_device *pdev)
> >> > +{
> >> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >> > +       struct fpga_fme *priv;
> >> > +       int ret;
> >> > +
> >> > +       mutex_lock(&pdata->lock);
> >> > +       priv = fpga_pdata_get_private(pdata);
> >> > +       ret = fpga_mgr_register(&pdata->dev->dev,
> >> > +               "Intel FPGA Manager", &fme_pr_ops, priv);
> >> > +       mutex_unlock(&pdata->lock);
> >> > +
> >> > +       return ret;
> >> > +}
> >> > +
> >> > +static int fpga_fme_pr_remove(struct platform_device *pdev)
> >> > +{
> >> > +       fpga_mgr_unregister(&pdev->dev);
> >> > +       return 0;
> >> > +}
> >> > +
> >> > +static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
> >> > +{
> >> > +       int ret;
> >> > +
> >> > +       ret = fpga_fme_pr_probe(pdev);
> >> > +       if (ret)
> >> > +               return ret;
> >> > +
> >> > +       ret = sysfs_create_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
> >> > +       if (ret)
> >> > +               fpga_fme_pr_remove(pdev);
> >> > +
> >> > +       return ret;
> >> > +}
> >> > +
> >> > +static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
> >> > +{
> >> > +       sysfs_remove_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
> >> > +       fpga_fme_pr_remove(pdev);
> >> > +}
> >> > +
> >> > +static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
> >> > +       unsigned int cmd, unsigned long arg)
> >> > +{
> >> > +       long ret;
> >> > +
> >> > +       switch (cmd) {
> >> > +       case FPGA_FME_PORT_PR:
> >> > +               ret = fme_pr(pdev, arg);
> >> > +               break;
> >> > +       default:
> >> > +               ret = -ENODEV;
> >> > +       }
> >> > +
> >> > +       return ret;
> >> > +}
> >> > +
> >> > +struct feature_ops pr_mgmt_ops = {
> >> > +       .init = pr_mgmt_init,
> >> > +       .uinit = pr_mgmt_uinit,
> >> > +       .ioctl = fme_pr_ioctl,
> >> > +};
> >> > diff --git a/drivers/fpga/intel/fme.h b/drivers/fpga/intel/fme.h
> >> > new file mode 100644
> >> > index 0000000..d6cb7ce
> >> > --- /dev/null
> >> > +++ b/drivers/fpga/intel/fme.h
> >> > @@ -0,0 +1,32 @@
> >> > +/*
> >> > + * Header file for Intel FPGA Management Engine (FME) Driver
> >> > + *
> >> > + * Copyright (C) 2017 Intel Corporation, Inc.
> >> > + *
> >> > + * Authors:
> >> > + *   Kang Luwei <luwei.kang@intel.com>
> >> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> >> > + *   Joseph Grecco <joe.grecco@intel.com>
> >> > + *   Enno Luebbers <enno.luebbers@intel.com>
> >> > + *   Tim Whisonant <tim.whisonant@intel.com>
> >> > + *   Ananda Ravuri <ananda.ravuri@intel.com>
> >> > + *   Henry Mitchel <henry.mitchel@intel.com>
> >> > + *
> >> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
> >> > + * redistributing this file, you may do so under either license. See the
> >> > + * LICENSE.BSD file under this directory for the BSD license and see
> >> > + * the COPYING file in the top-level directory for the GPLv2 license.
> >> > + */
> >> > +
> >> > +#ifndef __INTEL_FME_H
> >> > +#define __INTEL_FME_H
> >> > +
> >> > +struct fpga_fme {
> >> > +       u8  port_id;
> >> > +       u64 pr_err;
> >> > +       struct feature_platform_data *pdata;
> >> > +};
> >> > +
> >> > +extern struct feature_ops pr_mgmt_ops;
> >> > +
> >> > +#endif
> >> > diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
> >> > index 992e556..77658316 100644
> >> > --- a/include/uapi/linux/intel-fpga.h
> >> > +++ b/include/uapi/linux/intel-fpga.h
> >> > @@ -18,6 +18,8 @@
> >> >  #ifndef _UAPI_LINUX_INTEL_FPGA_H
> >> >  #define _UAPI_LINUX_INTEL_FPGA_H
> >> >
> >> > +#include <linux/types.h>
> >> > +
> >> >  #define FPGA_API_VERSION 0
> >> >
> >> >  /*
> >> > @@ -30,6 +32,7 @@
> >> >  #define FPGA_MAGIC 0xB6
> >> >
> >> >  #define FPGA_BASE 0
> >> > +#define FME_BASE 0x80
> >> >
> >> >  /**
> >> >   * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
> >> > @@ -49,4 +52,45 @@
> >> >
> >> >  #define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)
> >> >
> >> > +/* IOCTLs for FME file descriptor */
> >> > +
> >> > +/**
> >> > + * FPGA_FME_PORT_PR - _IOWR(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
> >> > + *
> >> > + * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
> >> > + * provided by caller.
> >> > + * Return: 0 on success, -errno on failure.
> >> > + * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
> >> > + * some errors during PR, under this case, the user can fetch HW error code
> >> > + * from fpga_fme_port_pr.status. Each bit on the error code is used as the
> >> > + * index for the array created by DEFINE_FPGA_PR_ERR_MSG().
> >> > + * Otherwise, it is always zero.
> >> > + */
> >> > +
> >> > +#define DEFINE_FPGA_PR_ERR_MSG(_name_)                 \
> >> > +static const char * const _name_[] = {                 \
> >> > +       "PR operation error detected",                  \
> >> > +       "PR CRC error detected",                        \
> >> > +       "PR incompatiable bitstream error detected",    \
> >> > +       "PR IP protocol error detected",                \
> >> > +       "PR FIFO overflow error detected",              \
> >> > +       "Reserved",                                     \
> >> > +       "PR secure load error detected",                \
> >> > +}
> >> > +
> >> > +#define PR_MAX_ERR_NUM 7
> >> > +
> >> > +struct fpga_fme_port_pr {
> >> > +       /* Input */
> >> > +       __u32 argsz;            /* Structure length */
> >> > +       __u32 flags;            /* Zero for now */
> >> > +       __u32 port_id;
> >> > +       __u32 buffer_size;
> >> > +       __u64 buffer_address;   /* Userspace address to the buffer for PR */
> >> > +       /* Output */
> >> > +       __u64 status;           /* HW error code if ioctl returns -EIO */
> >> > +};
> >> > +
> >> > +#define FPGA_FME_PORT_PR       _IO(FPGA_MAGIC, FME_BASE + 0)
> >> > +
> >> >  #endif /* _UAPI_INTEL_FPGA_H */
> >> > --
> >> > 2.7.4
> >> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-04-03 21:24   ` Alan Tull
  2017-04-03 22:49     ` matthew.gerlach
@ 2017-04-04  6:28     ` Wu Hao
  1 sibling, 0 replies; 93+ messages in thread
From: Wu Hao @ 2017-04-04  6:28 UTC (permalink / raw)
  To: Alan Tull
  Cc: Moritz Fischer, linux-fpga, linux-kernel, luwei.kang, yi.z.zhang,
	Tim Whisonant, Enno Luebbers, Shiva Rao, Christopher Rauer,
	Alan Tull, Xiao Guangrong

On Mon, Apr 03, 2017 at 04:24:13PM -0500, Alan Tull wrote:
> On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote:
> > From: Kang Luwei <luwei.kang@intel.com>
> >
> > Partial Reconfiguration (PR) is the most important function for FME. It
> > allows reconfiguration for given Port/Accelerated Function Unit (AFU).
> >
> > This patch adds support for PR sub feature. In this patch, it registers
> > a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
> > for PR operation once PR request received via ioctl.
> 
> The code that invokes fpga_mgr_buf_load should be a different layer.

Please see my feedback below.

> 
> > Below user space
> > interfaces are exposed by this sub feature.
> >
> > Sysfs interface:
> > * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
> >   Read-only. Indicate the hardware interface information. Userspace
> >   applications need to check this interface to select correct green
> >   bitstream format before PR.
> >
> > Ioctl interface:
> > * FPGA_FME_PORT_PR
> >   Do partial reconfiguration per information from userspace, including
> >   target port(AFU), buffer size and address info. It returns the PR status
> >   (PR error code if failed) to userspace.
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > Signed-off-by: Alan Tull <alan.tull@intel.com>
> > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > ---
> >  drivers/fpga/intel/Makefile      |   2 +-
> >  drivers/fpga/intel/feature-dev.h |  58 ++++++
> >  drivers/fpga/intel/fme-main.c    |  44 ++++-
> >  drivers/fpga/intel/fme-pr.c      | 400 +++++++++++++++++++++++++++++++++++++++
> >  drivers/fpga/intel/fme.h         |  32 ++++
> >  include/uapi/linux/intel-fpga.h  |  44 +++++
> >  6 files changed, 578 insertions(+), 2 deletions(-)
> >  create mode 100644 drivers/fpga/intel/fme-pr.c
> >  create mode 100644 drivers/fpga/intel/fme.h
> >
> > diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
> > index 546861d..0452cb6 100644
> > --- a/drivers/fpga/intel/Makefile
> > +++ b/drivers/fpga/intel/Makefile
> > @@ -2,4 +2,4 @@ obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
> >  obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
> >
> >  intel-fpga-pci-objs := pcie.o feature-dev.o
> > -intel-fpga-fme-objs := fme-main.o
> > +intel-fpga-fme-objs := fme-main.o fme-pr.o
> > diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
> > index dccc283..5a25c915 100644
> > --- a/drivers/fpga/intel/feature-dev.h
> > +++ b/drivers/fpga/intel/feature-dev.h
> > @@ -150,8 +150,66 @@ struct feature_fme_err {
> >  };
> >
> >  /* FME Partial Reconfiguration Sub Feature Register Set */
> > +/* FME PR Control Register */
> > +struct feature_fme_pr_ctl {
> > +       union {
> > +               u64 csr;
> > +               struct {
> > +                       u8  pr_reset:1;         /* Reset PR Engine */
> > +                       u8  rsvdz1:3;
> > +                       u8  pr_reset_ack:1;     /* Reset PR Engine Ack */
> > +                       u8  rsvdz2:3;
> > +                       u8  pr_regionid:2;      /* PR Region ID */
> > +                       u8  rsvdz3:2;
> > +                       u8  pr_start_req:1;     /* PR Start Request */
> > +                       u8  pr_push_complete:1; /* PR Data push complete */
> > +                       u8  pr_kind:1;          /* Load Customer or Intel GBS */
> > +                       u32 rsvdz4:17;
> > +                       u32 config_data;
> > +               };
> > +       };
> > +};
> > +
> > +/* FME PR Status Register */
> > +struct feature_fme_pr_status {
> > +       union {
> > +               u64 csr;
> > +               struct {
> > +                       u16 pr_credit:9;        /* Number of PR Credits */
> > +                       u8  rsvdz1:7;
> > +                       u8  pr_status:1;        /* PR Operation status */
> > +                       u8  rsvdz2:3;
> > +                       u8  pr_ctrlr_status:3;  /* Controller status */
> > +                       u8  rsvdz3:1;
> > +                       u8  pr_host_status:4;   /* PR Host status */
> > +                       u64 rsvdz4:36;
> > +               };
> > +       };
> > +};
> > +
> > +/* FME PR Data Register */
> > +struct feature_fme_pr_data {
> > +       union {
> > +               u64 csr;
> > +               struct {
> > +                       /* PR data from the raw-binary file */
> > +                       u32 pr_data_raw;
> > +                       u32 rsvd;
> > +               };
> > +       };
> > +};
> > +
> >  struct feature_fme_pr {
> >         struct feature_header header;
> > +       struct feature_fme_pr_ctl control;
> > +       struct feature_fme_pr_status status;
> > +       struct feature_fme_pr_data data;
> > +       u64 error;
> > +
> > +       u64 rsvd[16];
> > +
> > +       u64 intfc_id_l;         /* PR interface Id Low */
> > +       u64 intfc_id_h;         /* PR interface Id High */
> >  };
> >
> >  /* PORT Header Register Set */
> > diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
> > index 36d0c4c..0d9a7a6 100644
> > --- a/drivers/fpga/intel/fme-main.c
> > +++ b/drivers/fpga/intel/fme-main.c
> > @@ -23,6 +23,7 @@
> >  #include <linux/intel-fpga.h>
> >
> >  #include "feature-dev.h"
> > +#include "fme.h"
> >
> >  static ssize_t ports_num_show(struct device *dev,
> >                               struct device_attribute *attr, char *buf)
> > @@ -101,6 +102,10 @@ static struct feature_driver fme_feature_drvs[] = {
> >                 .ops = &fme_hdr_ops,
> >         },
> >         {
> > +               .name = FME_FEATURE_PR_MGMT,
> > +               .ops = &pr_mgmt_ops,
> > +       },
> > +       {
> >                 .ops = NULL,
> >         },
> >  };
> > @@ -182,14 +187,48 @@ static const struct file_operations fme_fops = {
> >         .unlocked_ioctl = fme_ioctl,
> >  };
> >
> > +static int fme_dev_init(struct platform_device *pdev)
> > +{
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct fpga_fme *fme;
> > +
> > +       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
> > +       if (!fme)
> > +               return -ENOMEM;
> > +
> > +       fme->pdata = pdata;
> > +
> > +       mutex_lock(&pdata->lock);
> > +       fpga_pdata_set_private(pdata, fme);
> > +       mutex_unlock(&pdata->lock);
> > +       return 0;
> > +}
> > +
> > +static void fme_dev_destroy(struct platform_device *pdev)
> > +{
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct fpga_fme *fme;
> > +
> > +       mutex_lock(&pdata->lock);
> > +       fme = fpga_pdata_get_private(pdata);
> > +       fpga_pdata_set_private(pdata, NULL);
> > +       mutex_unlock(&pdata->lock);
> > +
> > +       devm_kfree(&pdev->dev, fme);
> > +}
> > +
> >  static int fme_probe(struct platform_device *pdev)
> >  {
> >         int ret;
> >
> > -       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> > +       ret = fme_dev_init(pdev);
> >         if (ret)
> >                 goto exit;
> >
> > +       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> > +       if (ret)
> > +               goto dev_destroy;
> > +
> >         ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
> >         if (ret)
> >                 goto feature_uinit;
> > @@ -198,6 +237,8 @@ static int fme_probe(struct platform_device *pdev)
> >
> >  feature_uinit:
> >         fpga_dev_feature_uinit(pdev);
> > +dev_destroy:
> > +       fme_dev_destroy(pdev);
> >  exit:
> >         return ret;
> >  }
> > @@ -206,6 +247,7 @@ static int fme_remove(struct platform_device *pdev)
> >  {
> >         fpga_dev_feature_uinit(pdev);
> >         fpga_unregister_dev_ops(pdev);
> > +       fme_dev_destroy(pdev);
> >         return 0;
> >  }
> >
> > diff --git a/drivers/fpga/intel/fme-pr.c b/drivers/fpga/intel/fme-pr.c
> > new file mode 100644
> > index 0000000..3b44a3e
> > --- /dev/null
> > +++ b/drivers/fpga/intel/fme-pr.c
> > @@ -0,0 +1,400 @@
> > +/*
> > + * Driver for Intel FPGA Management Engine (FME) Partial Reconfiguration
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <luwei.kang@intel.com>
> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > + *   Joseph Grecco <joe.grecco@intel.com>
> > + *   Enno Luebbers <enno.luebbers@intel.com>
> > + *   Tim Whisonant <tim.whisonant@intel.com>
> > + *   Ananda Ravuri <ananda.ravuri@intel.com>
> > + *   Christopher Rauer <christopher.rauer@intel.com>
> > + *   Henry Mitchel <henry.mitchel@intel.com>
> > + *
> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
> > + * redistributing this file, you may do so under either license. See the
> > + * LICENSE.BSD file under this directory for the BSD license and see
> > + * the COPYING file in the top-level directory for the GPLv2 license.
> > + */
> > +
> > +#include <linux/types.h>
> > +#include <linux/device.h>
> > +#include <linux/vmalloc.h>
> > +#include <linux/uaccess.h>
> > +#include <linux/fpga/fpga-mgr.h>
> > +#include <linux/intel-fpga.h>
> > +
> > +#include "feature-dev.h"
> > +#include "fme.h"
> > +
> > +#define PR_WAIT_TIMEOUT   8000000
> > +
> > +#define PR_HOST_STATUS_IDLE    0
> > +
> > +DEFINE_FPGA_PR_ERR_MSG(pr_err_msg);
> > +
> > +static ssize_t interface_id_show(struct device *dev,
> > +                                struct device_attribute *attr, char *buf)
> > +{
> > +       u64 intfc_id_l, intfc_id_h;
> > +       struct feature_fme_pr *fme_pr
> > +               = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_PR_MGMT);
> > +
> > +       intfc_id_l = readq(&fme_pr->intfc_id_l);
> > +       intfc_id_h = readq(&fme_pr->intfc_id_h);
> > +
> > +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> > +                       (unsigned long long)intfc_id_h,
> > +                       (unsigned long long)intfc_id_l);
> > +}
> > +static DEVICE_ATTR_RO(interface_id);
> > +
> > +static struct attribute *pr_mgmt_attrs[] = {
> > +       &dev_attr_interface_id.attr,
> > +       NULL,
> > +};
> > +
> > +struct attribute_group pr_mgmt_attr_group = {
> > +       .attrs  = pr_mgmt_attrs,
> > +       .name   = "pr",
> > +};
> > +
> > +static u64
> > +pr_err_handle(struct platform_device *pdev, struct feature_fme_pr *fme_pr)
> > +{
> > +       struct feature_fme_pr_status fme_pr_status;
> > +       unsigned long err_code;
> > +       u64 fme_pr_error;
> > +       int i = 0;
> > +
> > +       fme_pr_status.csr = readq(&fme_pr->status);
> > +       if (!fme_pr_status.pr_status)
> > +               return 0;
> > +
> > +       err_code = fme_pr_error = readq(&fme_pr->error);
> > +       for_each_set_bit(i, &err_code, PR_MAX_ERR_NUM)
> > +               dev_dbg(&pdev->dev, "%s\n", pr_err_msg[i]);
> > +       writeq(fme_pr_error, &fme_pr->error);
> > +       return fme_pr_error;
> > +}
> > +
> > +static int fme_pr_write_init(struct fpga_manager *mgr,
> > +               struct fpga_image_info *info, const char *buf, size_t count)
> > +{
> > +       struct fpga_fme *fme = mgr->priv;
> > +       struct platform_device *pdev;
> > +       struct feature_fme_pr *fme_pr;
> > +       struct feature_fme_pr_ctl fme_pr_ctl;
> > +       struct feature_fme_pr_status fme_pr_status;
> > +
> > +       pdev = fme->pdata->dev;
> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                               FME_FEATURE_ID_PR_MGMT);
> > +       if (!fme_pr)
> > +               return -EINVAL;
> > +
> > +       if (WARN_ON(info->flags != FPGA_MGR_PARTIAL_RECONFIG))
> > +               return -EINVAL;
> 
> flags is bitmapped so please do:
> 
> if (WARN_ON(info->flags & FPGA_MGR_PARTIAL_RECONFIG))

Thanks for the review.

Will fix this in the next version.

> 
> > +
> > +       dev_dbg(&pdev->dev, "resetting PR before initiated PR\n");
> > +
> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> > +       fme_pr_ctl.pr_reset = 1;
> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> > +
> > +       fme_pr_ctl.pr_reset_ack = 1;
> > +
> > +       if (fpga_wait_register_field(pr_reset_ack, fme_pr_ctl,
> > +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> > +               dev_err(&pdev->dev, "maximum PR timeout\n");
> > +               return -ETIMEDOUT;
> > +       }
> > +
> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> > +       fme_pr_ctl.pr_reset = 0;
> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> > +
> > +       dev_dbg(&pdev->dev,
> > +               "waiting for PR resource in HW to be initialized and ready\n");
> > +
> > +       fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
> > +
> > +       if (fpga_wait_register_field(pr_host_status, fme_pr_status,
> > +                                    &fme_pr->status, PR_WAIT_TIMEOUT, 1)) {
> > +               dev_err(&pdev->dev, "maximum PR timeout\n");
> > +               return -ETIMEDOUT;
> > +       }
> > +
> > +       dev_dbg(&pdev->dev, "check if have any previous PR error\n");
> > +       pr_err_handle(pdev, fme_pr);
> > +       return 0;
> > +}
> > +
> > +static int fme_pr_write(struct fpga_manager *mgr,
> > +                       const char *buf, size_t count)
> > +{
> > +       struct fpga_fme *fme = mgr->priv;
> > +       struct platform_device *pdev;
> > +       struct feature_fme_pr *fme_pr;
> > +       struct feature_fme_pr_ctl fme_pr_ctl;
> > +       struct feature_fme_pr_status fme_pr_status;
> > +       struct feature_fme_pr_data fme_pr_data;
> > +       int delay, pr_credit, i = 0;
> > +
> > +       pdev = fme->pdata->dev;
> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                               FME_FEATURE_ID_PR_MGMT);
> > +
> > +       dev_dbg(&pdev->dev, "set PR port ID and start request\n");
> > +
> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> > +       fme_pr_ctl.pr_regionid = fme->port_id;
> > +       fme_pr_ctl.pr_start_req = 1;
> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> > +
> > +       dev_dbg(&pdev->dev, "pushing data from bitstream to HW\n");
> > +
> > +       fme_pr_status.csr = readq(&fme_pr->status);
> > +       pr_credit = fme_pr_status.pr_credit;
> > +
> > +       while (count > 0) {
> > +               delay = 0;
> > +               while (pr_credit <= 1) {
> > +                       if (delay++ > PR_WAIT_TIMEOUT) {
> > +                               dev_err(&pdev->dev, "maximum try\n");
> > +                               return -ETIMEDOUT;
> > +                       }
> > +                       udelay(1);
> > +
> > +                       fme_pr_status.csr = readq(&fme_pr->status);
> > +                       pr_credit = fme_pr_status.pr_credit;
> > +               };
> > +
> > +               if (count >= 4) {
> > +                       fme_pr_data.rsvd = 0;
> > +                       fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
> > +                       writeq(fme_pr_data.csr, &fme_pr->data);
> > +                       count -= 4;
> > +                       pr_credit--;
> > +                       i++;
> > +               } else {
> > +                       WARN_ON(1);
> > +                       return -EINVAL;
> > +               }
> > +       }
> > +
> > +       return 0;
> > +}
> > +
> > +static int fme_pr_write_complete(struct fpga_manager *mgr,
> > +                       struct fpga_image_info *info)
> > +{
> > +       struct fpga_fme *fme = mgr->priv;
> > +       struct platform_device *pdev;
> > +       struct feature_fme_pr *fme_pr;
> > +       struct feature_fme_pr_ctl fme_pr_ctl;
> > +
> > +       pdev = fme->pdata->dev;
> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                               FME_FEATURE_ID_PR_MGMT);
> > +
> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> > +       fme_pr_ctl.pr_push_complete = 1;
> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> > +
> > +       dev_dbg(&pdev->dev, "green bitstream push complete\n");
> > +       dev_dbg(&pdev->dev, "waiting for HW to release PR resource\n");
> > +
> > +       fme_pr_ctl.pr_start_req = 0;
> > +
> > +       if (fpga_wait_register_field(pr_start_req, fme_pr_ctl,
> > +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> > +               dev_err(&pdev->dev, "maximum try.\n");
> > +               return -ETIMEDOUT;
> > +       }
> > +
> > +       dev_dbg(&pdev->dev, "PR operation complete, checking status\n");
> > +       fme->pr_err = pr_err_handle(pdev, fme_pr);
> > +       if (fme->pr_err)
> > +               return -EIO;
> > +
> > +       dev_dbg(&pdev->dev, "PR done successfully\n");
> > +       return 0;
> > +}
> > +
> > +static enum fpga_mgr_states fme_pr_state(struct fpga_manager *mgr)
> > +{
> > +       return FPGA_MGR_STATE_UNKNOWN;
> > +}
> > +
> > +static const struct fpga_manager_ops fme_pr_ops = {
> > +       .write_init = fme_pr_write_init,
> > +       .write = fme_pr_write,
> > +       .write_complete = fme_pr_write_complete,
> > +       .state = fme_pr_state,
> > +};
> > +
> 
> The following fme_pr() function shouldn't be in a fpga-mgr driver.  It
> is calling
> the framework that this driver is registered with.
> A fpga-mgr low level driver is supposed to be a very low level driver.
> This function is controlling the port and calling fpga-mgr layer to do the
> programming.  There should be a layer above the fpga-mgr and fpga-bridge
> that coordinates these things.  I.e. can you use my proposed fpga-region code?
> 

Hm.. as mentioned in the intel-fpga.txt documentation 'open discussion'.
Current FME doesn't use fpga-region as related patches are not merged.
But we will make the related changes to let FME create all fpga-regions,
bridges and managers for PR function on given Intel FPGA device per
suggestion.

Please notice below fme_pr is still needed to provide ioctl interface to
enduser for PR, it should invoke fpga_region_program_fpga or equivalent
for PR instead of fpga_mgr_buf_load then.
And if in the future, the sysfs for PR is ready, then fme_pr may not be
needed, as user can do PR via fpga-region sysfs interface.

Thanks
Hao

> > +static int fme_pr(struct platform_device *pdev, unsigned long arg)
> > +{
> > +       void __user *argp = (void __user *)arg;
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct fpga_fme *fme;
> > +       struct fpga_manager *mgr;
> > +       struct feature_fme_header *fme_hdr;
> > +       struct feature_fme_capability fme_capability;
> > +       struct fpga_image_info info;
> > +       struct fpga_fme_port_pr port_pr;
> > +       struct platform_device *port;
> > +       unsigned long minsz;
> > +       void *buf = NULL;
> > +       int ret = 0;
> > +
> > +       minsz = offsetofend(struct fpga_fme_port_pr, status);
> > +
> > +       if (copy_from_user(&port_pr, argp, minsz))
> > +               return -EFAULT;
> > +
> > +       if (port_pr.argsz < minsz || port_pr.flags)
> > +               return -EINVAL;
> > +
> > +       if (!IS_ALIGNED(port_pr.buffer_size, 4))
> > +               return -EINVAL;
> > +
> > +       /* get fme header region */
> > +       fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                                       FME_FEATURE_ID_HEADER);
> > +       if (WARN_ON(!fme_hdr))
> > +               return -EINVAL;
> > +
> > +       /* check port id */
> > +       fme_capability.csr = readq(&fme_hdr->capability);
> > +       if (port_pr.port_id >= fme_capability.num_ports) {
> > +               dev_dbg(&pdev->dev, "port number more than maximum\n");
> > +               return -EINVAL;
> > +       }
> > +
> > +       if (!access_ok(VERIFY_READ, port_pr.buffer_address,
> > +                                   port_pr.buffer_size))
> > +               return -EFAULT;
> > +
> > +       buf = vmalloc(port_pr.buffer_size);
> > +       if (!buf)
> > +               return -ENOMEM;
> > +
> > +       if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
> > +                                              port_pr.buffer_size)) {
> > +               ret = -EFAULT;
> > +               goto free_exit;
> > +       }
> > +
> > +       memset(&info, 0, sizeof(struct fpga_image_info));
> > +       info.flags = FPGA_MGR_PARTIAL_RECONFIG;
> > +
> > +       mgr = fpga_mgr_get(&pdev->dev);
> > +       if (IS_ERR(mgr)) {
> > +               ret = PTR_ERR(mgr);
> > +               goto free_exit;
> > +       }
> > +
> > +       mutex_lock(&pdata->lock);
> > +       fme = fpga_pdata_get_private(pdata);
> > +       /* fme device has been unregistered. */
> > +       if (!fme) {
> > +               ret = -EINVAL;
> > +               goto unlock_exit;
> > +       }
> > +
> > +       fme->pr_err = 0;
> > +       fme->port_id = port_pr.port_id;
> > +
> > +       /* Find and get port device by index */
> > +       port = pdata->fpga_for_each_port(pdev, &fme->port_id,
> > +                                        fpga_port_check_id);
> > +       WARN_ON(!port);
> > +
> > +       /* Disable Port before PR */
> > +       fpga_port_disable(port);
> > +
> > +       ret = fpga_mgr_buf_load(mgr, &info, buf, port_pr.buffer_size);
> > +       port_pr.status = fme->pr_err;
> > +
> > +       /* Re-enable Port after PR finished */
> > +       fpga_port_enable(port);
> > +
> > +       put_device(&port->dev);
> > +
> > +unlock_exit:
> > +       mutex_unlock(&pdata->lock);
> > +       fpga_mgr_put(mgr);
> > +free_exit:
> > +       vfree(buf);
> > +       if (copy_to_user((void __user *)arg, &port_pr, minsz))
> > +               return -EFAULT;
> > +       return ret;
> > +}
> > +
> > +static int fpga_fme_pr_probe(struct platform_device *pdev)
> > +{
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct fpga_fme *priv;
> > +       int ret;
> > +
> > +       mutex_lock(&pdata->lock);
> > +       priv = fpga_pdata_get_private(pdata);
> > +       ret = fpga_mgr_register(&pdata->dev->dev,
> > +               "Intel FPGA Manager", &fme_pr_ops, priv);
> > +       mutex_unlock(&pdata->lock);
> > +
> > +       return ret;
> > +}
> > +
> > +static int fpga_fme_pr_remove(struct platform_device *pdev)
> > +{
> > +       fpga_mgr_unregister(&pdev->dev);
> > +       return 0;
> > +}
> > +
> > +static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
> > +{
> > +       int ret;
> > +
> > +       ret = fpga_fme_pr_probe(pdev);
> > +       if (ret)
> > +               return ret;
> > +
> > +       ret = sysfs_create_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
> > +       if (ret)
> > +               fpga_fme_pr_remove(pdev);
> > +
> > +       return ret;
> > +}
> > +
> > +static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
> > +{
> > +       sysfs_remove_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
> > +       fpga_fme_pr_remove(pdev);
> > +}
> > +
> > +static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
> > +       unsigned int cmd, unsigned long arg)
> > +{
> > +       long ret;
> > +
> > +       switch (cmd) {
> > +       case FPGA_FME_PORT_PR:
> > +               ret = fme_pr(pdev, arg);
> > +               break;
> > +       default:
> > +               ret = -ENODEV;
> > +       }
> > +
> > +       return ret;
> > +}
> > +
> > +struct feature_ops pr_mgmt_ops = {
> > +       .init = pr_mgmt_init,
> > +       .uinit = pr_mgmt_uinit,
> > +       .ioctl = fme_pr_ioctl,
> > +};
> > diff --git a/drivers/fpga/intel/fme.h b/drivers/fpga/intel/fme.h
> > new file mode 100644
> > index 0000000..d6cb7ce
> > --- /dev/null
> > +++ b/drivers/fpga/intel/fme.h
> > @@ -0,0 +1,32 @@
> > +/*
> > + * Header file for Intel FPGA Management Engine (FME) Driver
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <luwei.kang@intel.com>
> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > + *   Joseph Grecco <joe.grecco@intel.com>
> > + *   Enno Luebbers <enno.luebbers@intel.com>
> > + *   Tim Whisonant <tim.whisonant@intel.com>
> > + *   Ananda Ravuri <ananda.ravuri@intel.com>
> > + *   Henry Mitchel <henry.mitchel@intel.com>
> > + *
> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
> > + * redistributing this file, you may do so under either license. See the
> > + * LICENSE.BSD file under this directory for the BSD license and see
> > + * the COPYING file in the top-level directory for the GPLv2 license.
> > + */
> > +
> > +#ifndef __INTEL_FME_H
> > +#define __INTEL_FME_H
> > +
> > +struct fpga_fme {
> > +       u8  port_id;
> > +       u64 pr_err;
> > +       struct feature_platform_data *pdata;
> > +};
> > +
> > +extern struct feature_ops pr_mgmt_ops;
> > +
> > +#endif
> > diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
> > index 992e556..77658316 100644
> > --- a/include/uapi/linux/intel-fpga.h
> > +++ b/include/uapi/linux/intel-fpga.h
> > @@ -18,6 +18,8 @@
> >  #ifndef _UAPI_LINUX_INTEL_FPGA_H
> >  #define _UAPI_LINUX_INTEL_FPGA_H
> >
> > +#include <linux/types.h>
> > +
> >  #define FPGA_API_VERSION 0
> >
> >  /*
> > @@ -30,6 +32,7 @@
> >  #define FPGA_MAGIC 0xB6
> >
> >  #define FPGA_BASE 0
> > +#define FME_BASE 0x80
> >
> >  /**
> >   * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
> > @@ -49,4 +52,45 @@
> >
> >  #define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)
> >
> > +/* IOCTLs for FME file descriptor */
> > +
> > +/**
> > + * FPGA_FME_PORT_PR - _IOWR(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
> > + *
> > + * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
> > + * provided by caller.
> > + * Return: 0 on success, -errno on failure.
> > + * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
> > + * some errors during PR, under this case, the user can fetch HW error code
> > + * from fpga_fme_port_pr.status. Each bit on the error code is used as the
> > + * index for the array created by DEFINE_FPGA_PR_ERR_MSG().
> > + * Otherwise, it is always zero.
> > + */
> > +
> > +#define DEFINE_FPGA_PR_ERR_MSG(_name_)                 \
> > +static const char * const _name_[] = {                 \
> > +       "PR operation error detected",                  \
> > +       "PR CRC error detected",                        \
> > +       "PR incompatiable bitstream error detected",    \
> > +       "PR IP protocol error detected",                \
> > +       "PR FIFO overflow error detected",              \
> > +       "Reserved",                                     \
> > +       "PR secure load error detected",                \
> > +}
> > +
> > +#define PR_MAX_ERR_NUM 7
> > +
> > +struct fpga_fme_port_pr {
> > +       /* Input */
> > +       __u32 argsz;            /* Structure length */
> > +       __u32 flags;            /* Zero for now */
> > +       __u32 port_id;
> > +       __u32 buffer_size;
> > +       __u64 buffer_address;   /* Userspace address to the buffer for PR */
> > +       /* Output */
> > +       __u64 status;           /* HW error code if ioctl returns -EIO */
> > +};
> > +
> > +#define FPGA_FME_PORT_PR       _IO(FPGA_MAGIC, FME_BASE + 0)
> > +
> >  #endif /* _UAPI_INTEL_FPGA_H */
> > --
> > 2.7.4
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-04-03 22:49     ` matthew.gerlach
@ 2017-04-04  6:48       ` Wu Hao
  0 siblings, 0 replies; 93+ messages in thread
From: Wu Hao @ 2017-04-04  6:48 UTC (permalink / raw)
  To: matthew.gerlach
  Cc: Alan Tull, Moritz Fischer, linux-fpga, linux-kernel, luwei.kang,
	yi.z.zhang, Tim Whisonant, Enno Luebbers, Shiva Rao,
	Christopher Rauer, Alan Tull, Xiao Guangrong

On Mon, Apr 03, 2017 at 03:49:41PM -0700, matthew.gerlach@linux.intel.com wrote:
> 
> 
> On Mon, 3 Apr 2017, Alan Tull wrote:
> 
> >On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote:
> >>From: Kang Luwei <luwei.kang@intel.com>
> >>
> >>Partial Reconfiguration (PR) is the most important function for FME. It
> >>allows reconfiguration for given Port/Accelerated Function Unit (AFU).
> >>
> >>This patch adds support for PR sub feature. In this patch, it registers
> >>a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
> >>for PR operation once PR request received via ioctl.
> >
> >The code that invokes fpga_mgr_buf_load should be a different layer.
> >
> >>Below user space
> >>interfaces are exposed by this sub feature.
> >>
> >>Sysfs interface:
> >>* /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
> >>  Read-only. Indicate the hardware interface information. Userspace
> >>  applications need to check this interface to select correct green
> >>  bitstream format before PR.
> >>
> >>Ioctl interface:
> >>* FPGA_FME_PORT_PR
> >>  Do partial reconfiguration per information from userspace, including
> >>  target port(AFU), buffer size and address info. It returns the PR status
> >>  (PR error code if failed) to userspace.
> >>
> >>Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> >>Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> >>Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> >>Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> >>Signed-off-by: Alan Tull <alan.tull@intel.com>
> >>Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> >>Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> >>Signed-off-by: Wu Hao <hao.wu@intel.com>
> >>---
> >> drivers/fpga/intel/Makefile      |   2 +-
> >> drivers/fpga/intel/feature-dev.h |  58 ++++++
> >> drivers/fpga/intel/fme-main.c    |  44 ++++-
> >> drivers/fpga/intel/fme-pr.c      | 400 +++++++++++++++++++++++++++++++++++++++
> >> drivers/fpga/intel/fme.h         |  32 ++++
> >> include/uapi/linux/intel-fpga.h  |  44 +++++
> >> 6 files changed, 578 insertions(+), 2 deletions(-)
> >> create mode 100644 drivers/fpga/intel/fme-pr.c
> >> create mode 100644 drivers/fpga/intel/fme.h
> >>
> >>diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
> >>index 546861d..0452cb6 100644
> >>--- a/drivers/fpga/intel/Makefile
> >>+++ b/drivers/fpga/intel/Makefile
> >>@@ -2,4 +2,4 @@ obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
> >> obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
> >>
> >> intel-fpga-pci-objs := pcie.o feature-dev.o
> >>-intel-fpga-fme-objs := fme-main.o
> >>+intel-fpga-fme-objs := fme-main.o fme-pr.o
> >>diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
> >>index dccc283..5a25c915 100644
> >>--- a/drivers/fpga/intel/feature-dev.h
> >>+++ b/drivers/fpga/intel/feature-dev.h
> >>@@ -150,8 +150,66 @@ struct feature_fme_err {
> >> };
> >>
> >> /* FME Partial Reconfiguration Sub Feature Register Set */
> >>+/* FME PR Control Register */
> >>+struct feature_fme_pr_ctl {
> >>+       union {
> >>+               u64 csr;
> >>+               struct {
> >>+                       u8  pr_reset:1;         /* Reset PR Engine */
> >>+                       u8  rsvdz1:3;
> >>+                       u8  pr_reset_ack:1;     /* Reset PR Engine Ack */
> >>+                       u8  rsvdz2:3;
> >>+                       u8  pr_regionid:2;      /* PR Region ID */
> >>+                       u8  rsvdz3:2;
> >>+                       u8  pr_start_req:1;     /* PR Start Request */
> >>+                       u8  pr_push_complete:1; /* PR Data push complete */
> >>+                       u8  pr_kind:1;          /* Load Customer or Intel GBS */
> >>+                       u32 rsvdz4:17;
> >>+                       u32 config_data;
> >>+               };
> >>+       };
> >>+};
> >>+
> >>+/* FME PR Status Register */
> >>+struct feature_fme_pr_status {
> >>+       union {
> >>+               u64 csr;
> >>+               struct {
> >>+                       u16 pr_credit:9;        /* Number of PR Credits */
> >>+                       u8  rsvdz1:7;
> >>+                       u8  pr_status:1;        /* PR Operation status */
> >>+                       u8  rsvdz2:3;
> >>+                       u8  pr_ctrlr_status:3;  /* Controller status */
> >>+                       u8  rsvdz3:1;
> >>+                       u8  pr_host_status:4;   /* PR Host status */
> >>+                       u64 rsvdz4:36;
> >>+               };
> >>+       };
> >>+};
> >>+
> >>+/* FME PR Data Register */
> >>+struct feature_fme_pr_data {
> >>+       union {
> >>+               u64 csr;
> >>+               struct {
> >>+                       /* PR data from the raw-binary file */
> >>+                       u32 pr_data_raw;
> >>+                       u32 rsvd;
> >>+               };
> >>+       };
> >>+};
> >>+
> >> struct feature_fme_pr {
> >>        struct feature_header header;
> >>+       struct feature_fme_pr_ctl control;
> >>+       struct feature_fme_pr_status status;
> >>+       struct feature_fme_pr_data data;
> >>+       u64 error;
> >>+
> >>+       u64 rsvd[16];
> >>+
> >>+       u64 intfc_id_l;         /* PR interface Id Low */
> >>+       u64 intfc_id_h;         /* PR interface Id High */
> >> };
> >>
> >> /* PORT Header Register Set */
> >>diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
> >>index 36d0c4c..0d9a7a6 100644
> >>--- a/drivers/fpga/intel/fme-main.c
> >>+++ b/drivers/fpga/intel/fme-main.c
> >>@@ -23,6 +23,7 @@
> >> #include <linux/intel-fpga.h>
> >>
> >> #include "feature-dev.h"
> >>+#include "fme.h"
> >>
> >> static ssize_t ports_num_show(struct device *dev,
> >>                              struct device_attribute *attr, char *buf)
> >>@@ -101,6 +102,10 @@ static struct feature_driver fme_feature_drvs[] = {
> >>                .ops = &fme_hdr_ops,
> >>        },
> >>        {
> >>+               .name = FME_FEATURE_PR_MGMT,
> >>+               .ops = &pr_mgmt_ops,
> >>+       },
> >>+       {
> >>                .ops = NULL,
> >>        },
> >> };
> >>@@ -182,14 +187,48 @@ static const struct file_operations fme_fops = {
> >>        .unlocked_ioctl = fme_ioctl,
> >> };
> >>
> >>+static int fme_dev_init(struct platform_device *pdev)
> >>+{
> >>+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >>+       struct fpga_fme *fme;
> >>+
> >>+       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
> >>+       if (!fme)
> >>+               return -ENOMEM;
> >>+
> >>+       fme->pdata = pdata;
> >>+
> >>+       mutex_lock(&pdata->lock);
> >>+       fpga_pdata_set_private(pdata, fme);
> >>+       mutex_unlock(&pdata->lock);
> >>+       return 0;
> >>+}
> >>+
> >>+static void fme_dev_destroy(struct platform_device *pdev)
> >>+{
> >>+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >>+       struct fpga_fme *fme;
> >>+
> >>+       mutex_lock(&pdata->lock);
> >>+       fme = fpga_pdata_get_private(pdata);
> >>+       fpga_pdata_set_private(pdata, NULL);
> >>+       mutex_unlock(&pdata->lock);
> >>+
> >>+       devm_kfree(&pdev->dev, fme);
> >>+}
> >>+
> >> static int fme_probe(struct platform_device *pdev)
> >> {
> >>        int ret;
> >>
> >>-       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> >>+       ret = fme_dev_init(pdev);
> >>        if (ret)
> >>                goto exit;
> >>
> >>+       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> >>+       if (ret)
> >>+               goto dev_destroy;
> >>+
> >>        ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
> >>        if (ret)
> >>                goto feature_uinit;
> >>@@ -198,6 +237,8 @@ static int fme_probe(struct platform_device *pdev)
> >>
> >> feature_uinit:
> >>        fpga_dev_feature_uinit(pdev);
> >>+dev_destroy:
> >>+       fme_dev_destroy(pdev);
> >> exit:
> >>        return ret;
> >> }
> >>@@ -206,6 +247,7 @@ static int fme_remove(struct platform_device *pdev)
> >> {
> >>        fpga_dev_feature_uinit(pdev);
> >>        fpga_unregister_dev_ops(pdev);
> >>+       fme_dev_destroy(pdev);
> >>        return 0;
> >> }
> >>
> >>diff --git a/drivers/fpga/intel/fme-pr.c b/drivers/fpga/intel/fme-pr.c
> >>new file mode 100644
> >>index 0000000..3b44a3e
> >>--- /dev/null
> >>+++ b/drivers/fpga/intel/fme-pr.c
> >>@@ -0,0 +1,400 @@
> >>+/*
> >>+ * Driver for Intel FPGA Management Engine (FME) Partial Reconfiguration
> >>+ *
> >>+ * Copyright (C) 2017 Intel Corporation, Inc.
> >>+ *
> >>+ * Authors:
> >>+ *   Kang Luwei <luwei.kang@intel.com>
> >>+ *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> >>+ *   Joseph Grecco <joe.grecco@intel.com>
> >>+ *   Enno Luebbers <enno.luebbers@intel.com>
> >>+ *   Tim Whisonant <tim.whisonant@intel.com>
> >>+ *   Ananda Ravuri <ananda.ravuri@intel.com>
> >>+ *   Christopher Rauer <christopher.rauer@intel.com>
> >>+ *   Henry Mitchel <henry.mitchel@intel.com>
> >>+ *
> >>+ * This work is licensed under a dual BSD/GPLv2 license. When using or
> >>+ * redistributing this file, you may do so under either license. See the
> >>+ * LICENSE.BSD file under this directory for the BSD license and see
> >>+ * the COPYING file in the top-level directory for the GPLv2 license.
> >>+ */
> >>+
> >>+#include <linux/types.h>
> >>+#include <linux/device.h>
> >>+#include <linux/vmalloc.h>
> >>+#include <linux/uaccess.h>
> >>+#include <linux/fpga/fpga-mgr.h>
> >>+#include <linux/intel-fpga.h>
> >>+
> >>+#include "feature-dev.h"
> >>+#include "fme.h"
> >>+
> >>+#define PR_WAIT_TIMEOUT   8000000
> >>+
> >>+#define PR_HOST_STATUS_IDLE    0
> >>+
> >>+DEFINE_FPGA_PR_ERR_MSG(pr_err_msg);
> >>+
> >>+static ssize_t interface_id_show(struct device *dev,
> >>+                                struct device_attribute *attr, char *buf)
> >>+{
> >>+       u64 intfc_id_l, intfc_id_h;
> >>+       struct feature_fme_pr *fme_pr
> >>+               = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_PR_MGMT);
> >>+
> >>+       intfc_id_l = readq(&fme_pr->intfc_id_l);
> >>+       intfc_id_h = readq(&fme_pr->intfc_id_h);
> >>+
> >>+       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> >>+                       (unsigned long long)intfc_id_h,
> >>+                       (unsigned long long)intfc_id_l);
> >>+}
> >>+static DEVICE_ATTR_RO(interface_id);
> >>+
> >>+static struct attribute *pr_mgmt_attrs[] = {
> >>+       &dev_attr_interface_id.attr,
> >>+       NULL,
> >>+};
> >>+
> >>+struct attribute_group pr_mgmt_attr_group = {
> >>+       .attrs  = pr_mgmt_attrs,
> >>+       .name   = "pr",
> >>+};
> >>+
> >>+static u64
> >>+pr_err_handle(struct platform_device *pdev, struct feature_fme_pr *fme_pr)
> >>+{
> >>+       struct feature_fme_pr_status fme_pr_status;
> >>+       unsigned long err_code;
> >>+       u64 fme_pr_error;
> >>+       int i = 0;
> >>+
> >>+       fme_pr_status.csr = readq(&fme_pr->status);
> >>+       if (!fme_pr_status.pr_status)
> >>+               return 0;
> >>+
> >>+       err_code = fme_pr_error = readq(&fme_pr->error);
> >>+       for_each_set_bit(i, &err_code, PR_MAX_ERR_NUM)
> >>+               dev_dbg(&pdev->dev, "%s\n", pr_err_msg[i]);
> >>+       writeq(fme_pr_error, &fme_pr->error);
> >>+       return fme_pr_error;
> >>+}
> >>+
> >>+static int fme_pr_write_init(struct fpga_manager *mgr,
> >>+               struct fpga_image_info *info, const char *buf, size_t count)
> >>+{
> >>+       struct fpga_fme *fme = mgr->priv;
> >>+       struct platform_device *pdev;
> >>+       struct feature_fme_pr *fme_pr;
> >>+       struct feature_fme_pr_ctl fme_pr_ctl;
> >>+       struct feature_fme_pr_status fme_pr_status;
> >>+
> >>+       pdev = fme->pdata->dev;
> >>+       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> >>+                               FME_FEATURE_ID_PR_MGMT);
> >>+       if (!fme_pr)
> >>+               return -EINVAL;
> >>+
> >>+       if (WARN_ON(info->flags != FPGA_MGR_PARTIAL_RECONFIG))
> >>+               return -EINVAL;
> >
> >flags is bitmapped so please do:
> >
> >if (WARN_ON(info->flags & FPGA_MGR_PARTIAL_RECONFIG))
> >
> >>+
> >>+       dev_dbg(&pdev->dev, "resetting PR before initiated PR\n");
> >>+
> >>+       fme_pr_ctl.csr = readq(&fme_pr->control);
> >>+       fme_pr_ctl.pr_reset = 1;
> >>+       writeq(fme_pr_ctl.csr, &fme_pr->control);
> >>+
> >>+       fme_pr_ctl.pr_reset_ack = 1;
> >>+
> >>+       if (fpga_wait_register_field(pr_reset_ack, fme_pr_ctl,
> >>+                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> >>+               dev_err(&pdev->dev, "maximum PR timeout\n");
> >>+               return -ETIMEDOUT;
> >>+       }
> >>+
> >>+       fme_pr_ctl.csr = readq(&fme_pr->control);
> >>+       fme_pr_ctl.pr_reset = 0;
> >>+       writeq(fme_pr_ctl.csr, &fme_pr->control);
> >>+
> >>+       dev_dbg(&pdev->dev,
> >>+               "waiting for PR resource in HW to be initialized and ready\n");
> >>+
> >>+       fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
> >>+
> >>+       if (fpga_wait_register_field(pr_host_status, fme_pr_status,
> >>+                                    &fme_pr->status, PR_WAIT_TIMEOUT, 1)) {
> >>+               dev_err(&pdev->dev, "maximum PR timeout\n");
> >>+               return -ETIMEDOUT;
> >>+       }
> >>+
> >>+       dev_dbg(&pdev->dev, "check if have any previous PR error\n");
> >>+       pr_err_handle(pdev, fme_pr);
> >>+       return 0;
> >>+}
> >>+
> >>+static int fme_pr_write(struct fpga_manager *mgr,
> >>+                       const char *buf, size_t count)
> >>+{
> >>+       struct fpga_fme *fme = mgr->priv;
> >>+       struct platform_device *pdev;
> >>+       struct feature_fme_pr *fme_pr;
> >>+       struct feature_fme_pr_ctl fme_pr_ctl;
> >>+       struct feature_fme_pr_status fme_pr_status;
> >>+       struct feature_fme_pr_data fme_pr_data;
> >>+       int delay, pr_credit, i = 0;
> >>+
> >>+       pdev = fme->pdata->dev;
> >>+       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> >>+                               FME_FEATURE_ID_PR_MGMT);
> >>+
> >>+       dev_dbg(&pdev->dev, "set PR port ID and start request\n");
> >>+
> >>+       fme_pr_ctl.csr = readq(&fme_pr->control);
> >>+       fme_pr_ctl.pr_regionid = fme->port_id;
> >>+       fme_pr_ctl.pr_start_req = 1;
> >>+       writeq(fme_pr_ctl.csr, &fme_pr->control);
> >>+
> >>+       dev_dbg(&pdev->dev, "pushing data from bitstream to HW\n");
> >>+
> >>+       fme_pr_status.csr = readq(&fme_pr->status);
> >>+       pr_credit = fme_pr_status.pr_credit;
> >>+
> >>+       while (count > 0) {
> >>+               delay = 0;
> >>+               while (pr_credit <= 1) {
> >>+                       if (delay++ > PR_WAIT_TIMEOUT) {
> >>+                               dev_err(&pdev->dev, "maximum try\n");
> >>+                               return -ETIMEDOUT;
> >>+                       }
> >>+                       udelay(1);
> >>+
> >>+                       fme_pr_status.csr = readq(&fme_pr->status);
> >>+                       pr_credit = fme_pr_status.pr_credit;
> >>+               };
> >>+
> >>+               if (count >= 4) {
> >>+                       fme_pr_data.rsvd = 0;
> >>+                       fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
> >>+                       writeq(fme_pr_data.csr, &fme_pr->data);
> >>+                       count -= 4;
> >>+                       pr_credit--;
> >>+                       i++;
> >>+               } else {
> >>+                       WARN_ON(1);
> >>+                       return -EINVAL;
> >>+               }
> >>+       }
> >>+
> >>+       return 0;
> >>+}
> >>+
> >>+static int fme_pr_write_complete(struct fpga_manager *mgr,
> >>+                       struct fpga_image_info *info)
> >>+{
> >>+       struct fpga_fme *fme = mgr->priv;
> >>+       struct platform_device *pdev;
> >>+       struct feature_fme_pr *fme_pr;
> >>+       struct feature_fme_pr_ctl fme_pr_ctl;
> >>+
> >>+       pdev = fme->pdata->dev;
> >>+       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> >>+                               FME_FEATURE_ID_PR_MGMT);
> >>+
> >>+       fme_pr_ctl.csr = readq(&fme_pr->control);
> >>+       fme_pr_ctl.pr_push_complete = 1;
> >>+       writeq(fme_pr_ctl.csr, &fme_pr->control);
> >>+
> >>+       dev_dbg(&pdev->dev, "green bitstream push complete\n");
> >>+       dev_dbg(&pdev->dev, "waiting for HW to release PR resource\n");
> >>+
> >>+       fme_pr_ctl.pr_start_req = 0;
> >>+
> >>+       if (fpga_wait_register_field(pr_start_req, fme_pr_ctl,
> >>+                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> >>+               dev_err(&pdev->dev, "maximum try.\n");
> >>+               return -ETIMEDOUT;
> >>+       }
> >>+
> >>+       dev_dbg(&pdev->dev, "PR operation complete, checking status\n");
> >>+       fme->pr_err = pr_err_handle(pdev, fme_pr);
> >>+       if (fme->pr_err)
> >>+               return -EIO;
> >>+
> >>+       dev_dbg(&pdev->dev, "PR done successfully\n");
> >>+       return 0;
> >>+}
> >>+
> >>+static enum fpga_mgr_states fme_pr_state(struct fpga_manager *mgr)
> >>+{
> >>+       return FPGA_MGR_STATE_UNKNOWN;
> >>+}
> 
> The functions that implement the fme_pr_ops are really a low level fpga
> manager driver for the Altera PR IP component.  A standalone version of
> such a driver has been reviewed and Acked.  See the links below.
> Could this file use those functions and remove this code?
> 
> http://marc.info/?l=linux-kernel&m=149019678925564&w=2
> http://marc.info/?l=linux-kernel&m=149019654225457&w=2
> http://marc.info/?l=linux-kernel&m=149019598025274&w=2
> http://marc.info/?l=linux-kernel&m=149013051007149&w=2
> 

Thanks for the info, I just checked these code quickly, but it seems the
register definitions are totally different, so I feel that we may not be
able to reuse this driver.

Thanks
Hao

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features.
  2017-03-30 12:08 ` [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features Wu Hao
  2017-04-03 21:44   ` Alan Tull
  2017-04-04  2:44   ` Moritz Fischer
@ 2017-04-04 22:09   ` Alan Tull
  2017-04-05 14:09     ` Wu Hao
  2017-05-04 15:13   ` Li, Yi
  3 siblings, 1 reply; 93+ messages in thread
From: Alan Tull @ 2017-04-04 22:09 UTC (permalink / raw)
  To: Wu Hao
  Cc: Moritz Fischer, linux-fpga, linux-kernel, luwei.kang, yi.z.zhang,
	Xiao Guangrong, Tim Whisonant, Enno Luebbers, Shiva Rao,
	Christopher Rauer

On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote:
> From: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>
> Device Featuer List structure creates a link list of feature headers
> within the MMIO space to provide an extensiable way of adding features.
>
> The Intel FPGA PCIe driver walks through the feature headers to enumerate
> feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
> Function Unit (AFU), and their private sub features. For feature devices,
> it creates the platform devices and linked the private sub features into
> their platform data.
>

I'm still looking at this code and it's pretty new to me, but I think
it would be desirable and not really hard to separate the code
that enumerates from all the fixed feature code.  So pcie.c could see
that there is a pci device out there that it knows has these memory
mapped enumeration structs.  So it goes and reads the structs, parses
them, and knows what drivers to probe.  The FME and AFU and other
fpga device drivers could register their guids with the framework
and be discoverable in that way.

That way if you need to implement a different FME or anything else, it
could be added with a new guid to this framework and would get
enumerated.  I'm thinking of the future and of more general usability
of this code.

Then the enumeration code wouldn't have to be 'intel' code or even
code dedicated to FME's and AFU's.  Any FPGA with a PCIe
port that has the right id's could have this struct and use this
enumeration method.  Actually if the parse* enumeration code could be in a
separate file as helper functions for the pcie code, this stuff would
be structured for future support this of the same framework on
embedded FPGA devices.

Alan

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-04-04  6:05         ` Wu Hao
@ 2017-04-04 22:37           ` Alan Tull
  2017-04-05 11:40             ` Wu, Hao
  0 siblings, 1 reply; 93+ messages in thread
From: Alan Tull @ 2017-04-04 22:37 UTC (permalink / raw)
  To: Wu Hao
  Cc: Moritz Fischer, linux-fpga, linux-kernel, luwei.kang, yi.z.zhang,
	Tim Whisonant, Enno Luebbers, Shiva Rao, Christopher Rauer,
	Alan Tull, Xiao Guangrong

On Tue, Apr 4, 2017 at 1:05 AM, Wu Hao <hao.wu@intel.com> wrote:
> On Mon, Apr 03, 2017 at 11:30:55AM -0500, Alan Tull wrote:
>> On Sat, Apr 1, 2017 at 6:08 AM, Wu Hao <hao.wu@intel.com> wrote:
>> > On Fri, Mar 31, 2017 at 02:10:12PM -0500, Alan Tull wrote:
>> >> On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote:
>> >> > From: Kang Luwei <luwei.kang@intel.com>
>> >> >
>> >> > Partial Reconfiguration (PR) is the most important function for FME. It
>> >> > allows reconfiguration for given Port/Accelerated Function Unit (AFU).
>> >> >
>> >> > This patch adds support for PR sub feature. In this patch, it registers
>> >> > a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
>> >> > for PR operation once PR request received via ioctl. Below user space
>> >> > interfaces are exposed by this sub feature.
>> >> >
>> >> > Sysfs interface:
>> >> > * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
>> >> >   Read-only. Indicate the hardware interface information. Userspace
>> >> >   applications need to check this interface to select correct green
>> >> >   bitstream format before PR.
>> >> >
>> >> > Ioctl interface:
>> >> > * FPGA_FME_PORT_PR
>> >> >   Do partial reconfiguration per information from userspace, including
>> >> >   target port(AFU), buffer size and address info. It returns the PR status
>> >> >   (PR error code if failed) to userspace.
>> >> >
>> >> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
>> >> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
>> >> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
>> >> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
>> >> > Signed-off-by: Alan Tull <alan.tull@intel.com>
>> >> > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
>> >> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> >> > Signed-off-by: Wu Hao <hao.wu@intel.com>
>> >> > ---
>> >> >  drivers/fpga/intel/Makefile      |   2 +-
>> >> >  drivers/fpga/intel/feature-dev.h |  58 ++++++
>> >> >  drivers/fpga/intel/fme-main.c    |  44 ++++-
>> >> >  drivers/fpga/intel/fme-pr.c      | 400 +++++++++++++++++++++++++++++++++++++++
>> >> >  drivers/fpga/intel/fme.h         |  32 ++++
>> >> >  include/uapi/linux/intel-fpga.h  |  44 +++++
>> >> >  6 files changed, 578 insertions(+), 2 deletions(-)
>> >> >  create mode 100644 drivers/fpga/intel/fme-pr.c
>> >> >  create mode 100644 drivers/fpga/intel/fme.h
>> >> >
>> >> > diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
>> >> > index 546861d..0452cb6 100644
>> >> > --- a/drivers/fpga/intel/Makefile
>> >> > +++ b/drivers/fpga/intel/Makefile
>> >> > @@ -2,4 +2,4 @@ obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
>> >> >  obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
>> >> >
>> >> >  intel-fpga-pci-objs := pcie.o feature-dev.o
>> >> > -intel-fpga-fme-objs := fme-main.o
>> >> > +intel-fpga-fme-objs := fme-main.o fme-pr.o
>> >> > diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
>> >> > index dccc283..5a25c915 100644
>> >> > --- a/drivers/fpga/intel/feature-dev.h
>> >> > +++ b/drivers/fpga/intel/feature-dev.h
>> >> > @@ -150,8 +150,66 @@ struct feature_fme_err {
>> >> >  };
>> >> >
>> >> >  /* FME Partial Reconfiguration Sub Feature Register Set */
>> >> > +/* FME PR Control Register */
>> >> > +struct feature_fme_pr_ctl {
>> >> > +       union {
>> >> > +               u64 csr;
>> >> > +               struct {
>> >> > +                       u8  pr_reset:1;         /* Reset PR Engine */
>> >> > +                       u8  rsvdz1:3;
>> >> > +                       u8  pr_reset_ack:1;     /* Reset PR Engine Ack */
>> >> > +                       u8  rsvdz2:3;
>> >> > +                       u8  pr_regionid:2;      /* PR Region ID */
>> >> > +                       u8  rsvdz3:2;
>> >> > +                       u8  pr_start_req:1;     /* PR Start Request */
>> >> > +                       u8  pr_push_complete:1; /* PR Data push complete */
>> >> > +                       u8  pr_kind:1;          /* Load Customer or Intel GBS */
>> >> > +                       u32 rsvdz4:17;
>> >> > +                       u32 config_data;
>> >> > +               };
>> >> > +       };
>> >> > +};
>> >> > +
>> >> > +/* FME PR Status Register */
>> >> > +struct feature_fme_pr_status {
>> >> > +       union {
>> >> > +               u64 csr;
>> >> > +               struct {
>> >> > +                       u16 pr_credit:9;        /* Number of PR Credits */
>> >> > +                       u8  rsvdz1:7;
>> >> > +                       u8  pr_status:1;        /* PR Operation status */
>> >> > +                       u8  rsvdz2:3;
>> >> > +                       u8  pr_ctrlr_status:3;  /* Controller status */
>> >> > +                       u8  rsvdz3:1;
>> >> > +                       u8  pr_host_status:4;   /* PR Host status */
>> >> > +                       u64 rsvdz4:36;
>> >> > +               };
>> >> > +       };
>> >> > +};
>> >> > +
>> >> > +/* FME PR Data Register */
>> >> > +struct feature_fme_pr_data {
>> >> > +       union {
>> >> > +               u64 csr;
>> >> > +               struct {
>> >> > +                       /* PR data from the raw-binary file */
>> >> > +                       u32 pr_data_raw;
>> >> > +                       u32 rsvd;
>> >> > +               };
>> >> > +       };
>> >> > +};
>> >> > +
>> >> >  struct feature_fme_pr {
>> >> >         struct feature_header header;
>> >> > +       struct feature_fme_pr_ctl control;
>> >> > +       struct feature_fme_pr_status status;
>> >> > +       struct feature_fme_pr_data data;
>> >> > +       u64 error;
>> >> > +
>> >> > +       u64 rsvd[16];
>> >> > +
>> >> > +       u64 intfc_id_l;         /* PR interface Id Low */
>> >> > +       u64 intfc_id_h;         /* PR interface Id High */
>> >> >  };
>> >> >
>> >> >  /* PORT Header Register Set */
>> >> > diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
>> >> > index 36d0c4c..0d9a7a6 100644
>> >> > --- a/drivers/fpga/intel/fme-main.c
>> >> > +++ b/drivers/fpga/intel/fme-main.c
>> >> > @@ -23,6 +23,7 @@
>> >> >  #include <linux/intel-fpga.h>
>> >> >
>> >> >  #include "feature-dev.h"
>> >> > +#include "fme.h"
>> >> >
>> >> >  static ssize_t ports_num_show(struct device *dev,
>> >> >                               struct device_attribute *attr, char *buf)
>> >> > @@ -101,6 +102,10 @@ static struct feature_driver fme_feature_drvs[] = {
>> >> >                 .ops = &fme_hdr_ops,
>> >> >         },
>> >> >         {
>> >> > +               .name = FME_FEATURE_PR_MGMT,
>> >> > +               .ops = &pr_mgmt_ops,
>> >> > +       },
>> >> > +       {
>> >> >                 .ops = NULL,
>> >> >         },
>> >> >  };
>> >> > @@ -182,14 +187,48 @@ static const struct file_operations fme_fops = {
>> >> >         .unlocked_ioctl = fme_ioctl,
>> >> >  };
>> >> >
>> >> > +static int fme_dev_init(struct platform_device *pdev)
>> >> > +{
>> >> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> >> > +       struct fpga_fme *fme;
>> >> > +
>> >> > +       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
>> >> > +       if (!fme)
>> >> > +               return -ENOMEM;
>> >> > +
>> >> > +       fme->pdata = pdata;
>> >> > +
>> >> > +       mutex_lock(&pdata->lock);
>> >> > +       fpga_pdata_set_private(pdata, fme);
>> >> > +       mutex_unlock(&pdata->lock);
>> >> > +       return 0;
>> >> > +}
>> >> > +
>> >> > +static void fme_dev_destroy(struct platform_device *pdev)
>> >> > +{
>> >> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> >> > +       struct fpga_fme *fme;
>> >> > +
>> >> > +       mutex_lock(&pdata->lock);
>> >> > +       fme = fpga_pdata_get_private(pdata);
>> >> > +       fpga_pdata_set_private(pdata, NULL);
>> >> > +       mutex_unlock(&pdata->lock);
>> >> > +
>> >> > +       devm_kfree(&pdev->dev, fme);
>> >> > +}
>> >> > +
>> >> >  static int fme_probe(struct platform_device *pdev)
>> >> >  {
>> >> >         int ret;
>> >> >
>> >> > -       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
>> >> > +       ret = fme_dev_init(pdev);
>> >> >         if (ret)
>> >> >                 goto exit;
>> >> >
>> >> > +       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
>> >> > +       if (ret)
>> >> > +               goto dev_destroy;
>> >> > +
>> >> >         ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
>> >> >         if (ret)
>> >> >                 goto feature_uinit;
>> >> > @@ -198,6 +237,8 @@ static int fme_probe(struct platform_device *pdev)
>> >> >
>> >> >  feature_uinit:
>> >> >         fpga_dev_feature_uinit(pdev);
>> >> > +dev_destroy:
>> >> > +       fme_dev_destroy(pdev);
>> >> >  exit:
>> >> >         return ret;
>> >> >  }
>> >> > @@ -206,6 +247,7 @@ static int fme_remove(struct platform_device *pdev)
>> >> >  {
>> >> >         fpga_dev_feature_uinit(pdev);
>> >> >         fpga_unregister_dev_ops(pdev);
>> >> > +       fme_dev_destroy(pdev);
>> >> >         return 0;
>> >> >  }
>> >> >
>> >> > diff --git a/drivers/fpga/intel/fme-pr.c b/drivers/fpga/intel/fme-pr.c
>> >> > new file mode 100644
>> >> > index 0000000..3b44a3e
>> >> > --- /dev/null
>> >> > +++ b/drivers/fpga/intel/fme-pr.c
>> >> > @@ -0,0 +1,400 @@
>> >> > +/*
>> >> > + * Driver for Intel FPGA Management Engine (FME) Partial Reconfiguration
>> >> > + *
>> >> > + * Copyright (C) 2017 Intel Corporation, Inc.
>> >> > + *
>> >> > + * Authors:
>> >> > + *   Kang Luwei <luwei.kang@intel.com>
>> >> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> >> > + *   Joseph Grecco <joe.grecco@intel.com>
>> >> > + *   Enno Luebbers <enno.luebbers@intel.com>
>> >> > + *   Tim Whisonant <tim.whisonant@intel.com>
>> >> > + *   Ananda Ravuri <ananda.ravuri@intel.com>
>> >> > + *   Christopher Rauer <christopher.rauer@intel.com>
>> >> > + *   Henry Mitchel <henry.mitchel@intel.com>
>> >> > + *
>> >> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
>> >> > + * redistributing this file, you may do so under either license. See the
>> >> > + * LICENSE.BSD file under this directory for the BSD license and see
>> >> > + * the COPYING file in the top-level directory for the GPLv2 license.
>> >> > + */
>> >> > +
>> >> > +#include <linux/types.h>
>> >> > +#include <linux/device.h>
>> >> > +#include <linux/vmalloc.h>
>> >> > +#include <linux/uaccess.h>
>> >> > +#include <linux/fpga/fpga-mgr.h>
>> >> > +#include <linux/intel-fpga.h>
>> >> > +
>> >> > +#include "feature-dev.h"
>> >> > +#include "fme.h"
>> >> > +
>> >> > +#define PR_WAIT_TIMEOUT   8000000
>> >> > +
>> >> > +#define PR_HOST_STATUS_IDLE    0
>> >> > +
>> >> > +DEFINE_FPGA_PR_ERR_MSG(pr_err_msg);
>> >> > +
>> >> > +static ssize_t interface_id_show(struct device *dev,
>> >> > +                                struct device_attribute *attr, char *buf)
>> >> > +{
>> >> > +       u64 intfc_id_l, intfc_id_h;
>> >> > +       struct feature_fme_pr *fme_pr
>> >> > +               = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_PR_MGMT);
>> >> > +
>> >> > +       intfc_id_l = readq(&fme_pr->intfc_id_l);
>> >> > +       intfc_id_h = readq(&fme_pr->intfc_id_h);
>> >> > +
>> >> > +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
>> >> > +                       (unsigned long long)intfc_id_h,
>> >> > +                       (unsigned long long)intfc_id_l);
>> >> > +}
>> >> > +static DEVICE_ATTR_RO(interface_id);
>> >> > +
>> >> > +static struct attribute *pr_mgmt_attrs[] = {
>> >> > +       &dev_attr_interface_id.attr,
>> >> > +       NULL,
>> >> > +};
>> >> > +
>> >> > +struct attribute_group pr_mgmt_attr_group = {
>> >> > +       .attrs  = pr_mgmt_attrs,
>> >> > +       .name   = "pr",
>> >> > +};
>> >> > +
>> >> > +static u64
>> >> > +pr_err_handle(struct platform_device *pdev, struct feature_fme_pr *fme_pr)
>> >> > +{
>> >> > +       struct feature_fme_pr_status fme_pr_status;
>> >> > +       unsigned long err_code;
>> >> > +       u64 fme_pr_error;
>> >> > +       int i = 0;
>> >> > +
>> >> > +       fme_pr_status.csr = readq(&fme_pr->status);
>> >> > +       if (!fme_pr_status.pr_status)
>> >> > +               return 0;
>> >> > +
>> >> > +       err_code = fme_pr_error = readq(&fme_pr->error);
>> >> > +       for_each_set_bit(i, &err_code, PR_MAX_ERR_NUM)
>> >> > +               dev_dbg(&pdev->dev, "%s\n", pr_err_msg[i]);
>> >> > +       writeq(fme_pr_error, &fme_pr->error);
>> >> > +       return fme_pr_error;
>> >> > +}
>> >> > +
>> >> > +static int fme_pr_write_init(struct fpga_manager *mgr,
>> >> > +               struct fpga_image_info *info, const char *buf, size_t count)
>> >> > +{
>> >> > +       struct fpga_fme *fme = mgr->priv;
>> >> > +       struct platform_device *pdev;
>> >> > +       struct feature_fme_pr *fme_pr;
>> >> > +       struct feature_fme_pr_ctl fme_pr_ctl;
>> >> > +       struct feature_fme_pr_status fme_pr_status;
>> >> > +
>> >> > +       pdev = fme->pdata->dev;
>> >> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
>> >> > +                               FME_FEATURE_ID_PR_MGMT);
>> >> > +       if (!fme_pr)
>> >> > +               return -EINVAL;
>> >> > +
>> >> > +       if (WARN_ON(info->flags != FPGA_MGR_PARTIAL_RECONFIG))
>> >> > +               return -EINVAL;
>> >> > +
>> >> > +       dev_dbg(&pdev->dev, "resetting PR before initiated PR\n");
>> >> > +
>> >> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> >> > +       fme_pr_ctl.pr_reset = 1;
>> >> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> >> > +
>> >> > +       fme_pr_ctl.pr_reset_ack = 1;
>> >> > +
>> >> > +       if (fpga_wait_register_field(pr_reset_ack, fme_pr_ctl,
>> >> > +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
>> >> > +               dev_err(&pdev->dev, "maximum PR timeout\n");
>> >> > +               return -ETIMEDOUT;
>> >> > +       }
>> >> > +
>> >> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> >> > +       fme_pr_ctl.pr_reset = 0;
>> >> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> >> > +
>> >> > +       dev_dbg(&pdev->dev,
>> >> > +               "waiting for PR resource in HW to be initialized and ready\n");
>> >> > +
>> >> > +       fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
>> >> > +
>> >> > +       if (fpga_wait_register_field(pr_host_status, fme_pr_status,
>> >> > +                                    &fme_pr->status, PR_WAIT_TIMEOUT, 1)) {
>> >> > +               dev_err(&pdev->dev, "maximum PR timeout\n");
>> >> > +               return -ETIMEDOUT;
>> >> > +       }
>> >> > +
>> >> > +       dev_dbg(&pdev->dev, "check if have any previous PR error\n");
>> >> > +       pr_err_handle(pdev, fme_pr);
>> >> > +       return 0;
>> >> > +}
>> >> > +
>> >> > +static int fme_pr_write(struct fpga_manager *mgr,
>> >> > +                       const char *buf, size_t count)
>> >> > +{
>> >> > +       struct fpga_fme *fme = mgr->priv;
>> >> > +       struct platform_device *pdev;
>> >> > +       struct feature_fme_pr *fme_pr;
>> >> > +       struct feature_fme_pr_ctl fme_pr_ctl;
>> >> > +       struct feature_fme_pr_status fme_pr_status;
>> >> > +       struct feature_fme_pr_data fme_pr_data;
>> >> > +       int delay, pr_credit, i = 0;
>> >> > +
>> >> > +       pdev = fme->pdata->dev;
>> >> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
>> >> > +                               FME_FEATURE_ID_PR_MGMT);
>> >> > +
>> >> > +       dev_dbg(&pdev->dev, "set PR port ID and start request\n");
>> >> > +
>> >> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> >> > +       fme_pr_ctl.pr_regionid = fme->port_id;
>> >> > +       fme_pr_ctl.pr_start_req = 1;
>> >> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> >> > +
>> >> > +       dev_dbg(&pdev->dev, "pushing data from bitstream to HW\n");
>> >> > +
>> >> > +       fme_pr_status.csr = readq(&fme_pr->status);
>> >> > +       pr_credit = fme_pr_status.pr_credit;
>> >> > +
>> >> > +       while (count > 0) {
>> >> > +               delay = 0;
>> >> > +               while (pr_credit <= 1) {
>> >> > +                       if (delay++ > PR_WAIT_TIMEOUT) {
>> >> > +                               dev_err(&pdev->dev, "maximum try\n");
>> >> > +                               return -ETIMEDOUT;
>> >> > +                       }
>> >> > +                       udelay(1);
>> >> > +
>> >> > +                       fme_pr_status.csr = readq(&fme_pr->status);
>> >> > +                       pr_credit = fme_pr_status.pr_credit;
>> >> > +               };
>> >> > +
>> >> > +               if (count >= 4) {
>> >> > +                       fme_pr_data.rsvd = 0;
>> >> > +                       fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
>> >> > +                       writeq(fme_pr_data.csr, &fme_pr->data);
>> >> > +                       count -= 4;
>> >> > +                       pr_credit--;
>> >> > +                       i++;
>> >> > +               } else {
>> >> > +                       WARN_ON(1);
>> >> > +                       return -EINVAL;
>> >> > +               }
>> >> > +       }
>> >> > +
>> >> > +       return 0;
>> >> > +}
>> >> > +
>> >> > +static int fme_pr_write_complete(struct fpga_manager *mgr,
>> >> > +                       struct fpga_image_info *info)
>> >> > +{
>> >> > +       struct fpga_fme *fme = mgr->priv;
>> >> > +       struct platform_device *pdev;
>> >> > +       struct feature_fme_pr *fme_pr;
>> >> > +       struct feature_fme_pr_ctl fme_pr_ctl;
>> >> > +
>> >> > +       pdev = fme->pdata->dev;
>> >> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
>> >> > +                               FME_FEATURE_ID_PR_MGMT);
>> >> > +
>> >> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> >> > +       fme_pr_ctl.pr_push_complete = 1;
>> >> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> >> > +
>> >> > +       dev_dbg(&pdev->dev, "green bitstream push complete\n");
>> >> > +       dev_dbg(&pdev->dev, "waiting for HW to release PR resource\n");
>> >> > +
>> >> > +       fme_pr_ctl.pr_start_req = 0;
>> >> > +
>> >> > +       if (fpga_wait_register_field(pr_start_req, fme_pr_ctl,
>> >> > +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
>> >> > +               dev_err(&pdev->dev, "maximum try.\n");
>> >> > +               return -ETIMEDOUT;
>> >> > +       }
>> >> > +
>> >> > +       dev_dbg(&pdev->dev, "PR operation complete, checking status\n");
>> >> > +       fme->pr_err = pr_err_handle(pdev, fme_pr);
>> >> > +       if (fme->pr_err)
>> >> > +               return -EIO;
>> >> > +
>> >> > +       dev_dbg(&pdev->dev, "PR done successfully\n");
>> >> > +       return 0;
>> >> > +}
>> >> > +
>> >> > +static enum fpga_mgr_states fme_pr_state(struct fpga_manager *mgr)
>> >> > +{
>> >> > +       return FPGA_MGR_STATE_UNKNOWN;
>> >> > +}
>> >> > +
>> >> > +static const struct fpga_manager_ops fme_pr_ops = {
>> >> > +       .write_init = fme_pr_write_init,
>> >> > +       .write = fme_pr_write,
>> >> > +       .write_complete = fme_pr_write_complete,
>> >> > +       .state = fme_pr_state,
>> >> > +};
>> >> > +
>> >> > +static int fme_pr(struct platform_device *pdev, unsigned long arg)
>> >> > +{
>> >> > +       void __user *argp = (void __user *)arg;
>> >> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> >> > +       struct fpga_fme *fme;
>> >> > +       struct fpga_manager *mgr;
>> >> > +       struct feature_fme_header *fme_hdr;
>> >> > +       struct feature_fme_capability fme_capability;
>> >> > +       struct fpga_image_info info;
>> >> > +       struct fpga_fme_port_pr port_pr;
>> >> > +       struct platform_device *port;
>> >> > +       unsigned long minsz;
>> >> > +       void *buf = NULL;
>> >> > +       int ret = 0;
>> >> > +
>> >> > +       minsz = offsetofend(struct fpga_fme_port_pr, status);
>> >> > +
>> >> > +       if (copy_from_user(&port_pr, argp, minsz))
>> >> > +               return -EFAULT;
>> >> > +
>> >> > +       if (port_pr.argsz < minsz || port_pr.flags)
>> >> > +               return -EINVAL;
>> >> > +
>> >> > +       if (!IS_ALIGNED(port_pr.buffer_size, 4))
>> >> > +               return -EINVAL;
>> >> > +
>> >> > +       /* get fme header region */
>> >> > +       fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
>> >> > +                                       FME_FEATURE_ID_HEADER);
>> >> > +       if (WARN_ON(!fme_hdr))
>> >> > +               return -EINVAL;
>> >> > +
>> >> > +       /* check port id */
>> >> > +       fme_capability.csr = readq(&fme_hdr->capability);
>> >> > +       if (port_pr.port_id >= fme_capability.num_ports) {
>> >> > +               dev_dbg(&pdev->dev, "port number more than maximum\n");
>> >> > +               return -EINVAL;
>> >> > +       }
>> >> > +
>> >> > +       if (!access_ok(VERIFY_READ, port_pr.buffer_address,
>> >> > +                                   port_pr.buffer_size))
>> >> > +               return -EFAULT;
>> >> > +
>> >> > +       buf = vmalloc(port_pr.buffer_size);
>> >> > +       if (!buf)
>> >> > +               return -ENOMEM;
>> >> > +
>> >> > +       if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
>> >> > +                                              port_pr.buffer_size)) {
>> >> > +               ret = -EFAULT;
>> >> > +               goto free_exit;
>> >> > +       }
>> >> > +
>> >> > +       memset(&info, 0, sizeof(struct fpga_image_info));
>> >> > +       info.flags = FPGA_MGR_PARTIAL_RECONFIG;
>> >> > +
>> >> > +       mgr = fpga_mgr_get(&pdev->dev);
>> >> > +       if (IS_ERR(mgr)) {
>> >> > +               ret = PTR_ERR(mgr);
>> >> > +               goto free_exit;
>> >> > +       }
>> >> > +
>> >> > +       mutex_lock(&pdata->lock);
>> >> > +       fme = fpga_pdata_get_private(pdata);
>> >> > +       /* fme device has been unregistered. */
>> >> > +       if (!fme) {
>> >> > +               ret = -EINVAL;
>> >> > +               goto unlock_exit;
>> >> > +       }
>> >> > +
>> >> > +       fme->pr_err = 0;
>> >> > +       fme->port_id = port_pr.port_id;
>> >>
>> >> It looks like you're using private data to communicate with the
>> >> driver, i.e. there is something you want to do with the fpga manager
>> >> framework and it doesn't have that feature.  The better way would be
>> >> for us to expand the framework so you don't need to do that.
>> >>
>> >> port_id is the kind of thing that should be communicated to the driver
>> >> through fpga_image_info, so we could add that to the struct.  Should
>> >> we call it port_id?
>>
>> Let's call it region_id.
>>
>> >> Or is there something more generic that may be
>> >> useful in the future for other architectures?.
>> >
>> > Hi Alan
>> >
>> > Thanks for your feedback. :)
>> >
>> > As you know, each Intel FPGA device may have more than one accelerator,
>> > and all accelerators share the same fpga_manager (only one FME module).
>> > port_id = 0 means the first accelerator on this fpga devices. So it's
>> > needed by the fpga_manager to distinguish one accelerator from others
>> > for partial reconfiguration.
>> >
>> > Adding a member like a 'id' to fpga_image_info definitely works for us
>> > in this case. We can add it this way, but I feel 'id' here seems not
>> > related to fpga image, but characteristic of fpga region.
>>
>> The  fpga_image_info struct started life as just image specific info,
>> but I want it to go in the direction of including parameters needed to
>> program it this specific time. Otherwise we are stuck having to keep
>> adding parameters as our use of FPGA develops.  It probably could be
>> documented better as 'information needed to program a FPGA image'
>> rather than strictly 'information about this particular FPGA image'.
>> My patch "fpga-mgr: pass parameters for loading fpga in image info"
>> goes in this direction by having the buf, firmware name, or sg list
>> passed in the info for the added fpga_mgr_load() function.  Actually I
>> should probably simplify the API and get rid of fpga_mgr_buf_load,
>> fpga_mgr_buf_load_sg, and fpga_mgr_firmware_load and require people to
>> use fpga_mgr_load (passing all parameters in fpga_image_info).
>>
>
> Make sense.
>
>> > It may be a
>> > little confusing. One rough idea is that keep this info under fpga region
>> > (maybe its private data), and pass the fpga-region to fpga_mgr_buf_load,
>>
>> Yes, keep this info in fpga-region.  When the region wants to program
>> using fpga-mgr, add the region id to fpga_image_info.  I propose
>> calling it region_id.
>
> Hm.. Do we need a function which moves info from region to image info?

No, just code that sets that variable in the struct before calling the
fpga_region_program_fpga function.

>
> Another idea is, add a priv to fpga_image_info, and use a common function
> to pass the fpga_region's priv to fpga_image_info's priv before PR.
> fpga-mgr then knows fpga_region priv info from the fpga_image_info.
>

Adding priv would make the interface for fpga-mgr non-uniform.  The point
of having a fpga-mgr framework is that there
is the potential of the upper layers working for different FPGA devices.
If the interface for each FPGA device were different, that would then
be broken.

>>
>> > then fpga_manager knows the target region for partial reconfiguration.
>> > If consider pr sysfs interface support under fpga-region in the future,
>> > then we don't need to create a new 'id' sysfs file, as fpga-region itself
>> > could provide this kind of info. But I'm not sure if this is the right
>> > direction.
>> >
>> >>
>> >> pr_err appears to be machine specific error codes that are
>> >> communicated outside your low level driver.  (Calling it pr_err is
>> >> extra confusing since Linux already has a commonly name function by
>> >> the same name).  The framework has state, but that's not doing what
>> >> you want here.  Maybe we could add a framework ops called status so
>> >> that status could be communicated from the low level driver. It would
>> >> be useful to abstract the machine specific state to a defined enum
>> >> that would be part of the fpga mgr framework.  I remember we had
>> >> something like that in the earliest version of fpga manager but it got
>> >> changed to state rather than status for some reason.
>> >>
>> >
>> > Adding a status function and a 'status' member to fpga manager sounds good.
>> > But I'm not sure how to abstract these error codes, currently what we have
>> > are all PR error codes for Intel FPGA device, e.g "PR FIFO overflow error",
>> > "PR secure load error" and etc (see include/uapi/linux/intel-fpga.h). Is it
>> > fine to let each driver to define how to use that 'status' for machine
>> > specific status?
>>
>> I looked at the list of errors in include/uapi/linux/intel-fpga.h.
>> They all seem pretty generic to me except I am not clear what "secure
>> load error" or "IP protocol error" mean and whether other
>> architectures would have them.  But certainly things like crc error,
>> incompatible bitstream, fifo overflow are generic.  So let's see if we
>> can define all these in a way that's generic and just pass up a error
>> number.  That way upper layers can know how to deal with them
>> possibly.  I would take the word "PR" off these since the error set
>> applies whether someone is doing full reconfig or partial reconfig.
>
> Sure, good to me, we can make it this way and see.
> Thanks for the suggestions. :)
>
> Hao

^ permalink raw reply	[flat|nested] 93+ messages in thread

* RE: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-04-04 22:37           ` Alan Tull
@ 2017-04-05 11:40             ` Wu, Hao
  2017-04-05 15:26               ` Alan Tull
  0 siblings, 1 reply; 93+ messages in thread
From: Wu, Hao @ 2017-04-05 11:40 UTC (permalink / raw)
  To: Alan Tull
  Cc: Moritz Fischer, linux-fpga, linux-kernel, Kang, Luwei, Zhang,
	Yi Z, Whisonant, Tim, Luebbers, Enno, Rao, Shiva, Rauer,
	Christopher, Tull, Alan, Xiao Guangrong

> >> The  fpga_image_info struct started life as just image specific info,
> >> but I want it to go in the direction of including parameters needed to
> >> program it this specific time. Otherwise we are stuck having to keep
> >> adding parameters as our use of FPGA develops.  It probably could be
> >> documented better as 'information needed to program a FPGA image'
> >> rather than strictly 'information about this particular FPGA image'.
> >> My patch "fpga-mgr: pass parameters for loading fpga in image info"
> >> goes in this direction by having the buf, firmware name, or sg list
> >> passed in the info for the added fpga_mgr_load() function.  Actually I
> >> should probably simplify the API and get rid of fpga_mgr_buf_load,
> >> fpga_mgr_buf_load_sg, and fpga_mgr_firmware_load and require people to
> >> use fpga_mgr_load (passing all parameters in fpga_image_info).
> >>
> >
> > Make sense.
> >
> >> > It may be a
> >> > little confusing. One rough idea is that keep this info under fpga region
> >> > (maybe its private data), and pass the fpga-region to fpga_mgr_buf_load,
> >>
> >> Yes, keep this info in fpga-region.  When the region wants to program
> >> using fpga-mgr, add the region id to fpga_image_info.  I propose
> >> calling it region_id.
> >
> > Hm.. Do we need a function which moves info from region to image info?
> 
> No, just code that sets that variable in the struct before calling the
> fpga_region_program_fpga function.
>
> >
> > Another idea is, add a priv to fpga_image_info, and use a common function
> > to pass the fpga_region's priv to fpga_image_info's priv before PR.
> > fpga-mgr then knows fpga_region priv info from the fpga_image_info.
> >
> 
> Adding priv would make the interface for fpga-mgr non-uniform.  The point
> of having a fpga-mgr framework is that there
> is the potential of the upper layers working for different FPGA devices.
> If the interface for each FPGA device were different, that would then
> be broken.
> 

I mean drivers can register their own fpga-mgr ops, and handle priv of
fpga_image_info in driver specific way for pr (e.g write_init function).
We don't need to change the any upper layer interfaces.

If you prefer the region_id for fpga_image_info, we can go with region_id
for sure. : )

Thanks
Hao

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features.
  2017-04-03 21:44   ` Alan Tull
@ 2017-04-05 11:58     ` Wu Hao
  2017-04-11 20:21       ` Alan Tull
  0 siblings, 1 reply; 93+ messages in thread
From: Wu Hao @ 2017-04-05 11:58 UTC (permalink / raw)
  To: Alan Tull
  Cc: Moritz Fischer, linux-fpga, linux-kernel, luwei.kang, yi.z.zhang,
	Xiao Guangrong, Tim Whisonant, Enno Luebbers, Shiva Rao,
	Christopher Rauer

On Mon, Apr 03, 2017 at 04:44:15PM -0500, Alan Tull wrote:
> On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote:
> > From: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> >
> > Device Featuer List structure creates a link list of feature headers
> > within the MMIO space to provide an extensiable way of adding features.
> >
> > The Intel FPGA PCIe driver walks through the feature headers to enumerate
> > feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
> > Function Unit (AFU), and their private sub features. For feature devices,
> > it creates the platform devices and linked the private sub features into
> > their platform data.
> >
> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> > Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > ---
> >  drivers/fpga/intel/Makefile      |   2 +-
> >  drivers/fpga/intel/feature-dev.c | 139 +++++++
> >  drivers/fpga/intel/feature-dev.h | 342 ++++++++++++++++
> >  drivers/fpga/intel/pcie.c        | 841 ++++++++++++++++++++++++++++++++++++++-
> >  4 files changed, 1321 insertions(+), 3 deletions(-)
> >  create mode 100644 drivers/fpga/intel/feature-dev.c
> >  create mode 100644 drivers/fpga/intel/feature-dev.h
> >
> > diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
> > index 61fd8ea..c029940 100644
> > --- a/drivers/fpga/intel/Makefile
> > +++ b/drivers/fpga/intel/Makefile
> > @@ -1,3 +1,3 @@
> >  obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
> >
> > -intel-fpga-pci-objs := pcie.o
> > +intel-fpga-pci-objs := pcie.o feature-dev.o
> > diff --git a/drivers/fpga/intel/feature-dev.c b/drivers/fpga/intel/feature-dev.c
> > new file mode 100644
> > index 0000000..6952566
> > --- /dev/null
> > +++ b/drivers/fpga/intel/feature-dev.c
> > @@ -0,0 +1,139 @@
> > +/*
> > + * Intel FPGA Feature Device Driver
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <luwei.kang@intel.com>
> > + *   Zhang Yi <yi.z.zhang@intel.com>
> > + *   Wu Hao <hao.wu@intel.com>
> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > + *
> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
> > + * redistributing this file, you may do so under either license. See the
> > + * LICENSE.BSD file under this directory for the BSD license and see
> > + * the COPYING file in the top-level directory for the GPLv2 license.
> > + */
> > +
> > +#include "feature-dev.h"
> > +
> > +void feature_platform_data_add(struct feature_platform_data *pdata,
> > +                              int index, const char *name,
> > +                              int resource_index, void __iomem *ioaddr)
> > +{
> > +       WARN_ON(index >= pdata->num);
> > +
> > +       pdata->features[index].name = name;
> > +       pdata->features[index].resource_index = resource_index;
> > +       pdata->features[index].ioaddr = ioaddr;
> > +}
> > +
> > +int feature_platform_data_size(int num)
> > +{
> > +       return sizeof(struct feature_platform_data) +
> > +               num * sizeof(struct feature);
> > +}
> > +
> > +struct feature_platform_data *
> > +feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
> > +{
> > +       struct feature_platform_data *pdata;
> > +
> > +       pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
> > +       if (pdata) {
> > +               pdata->dev = dev;
> > +               pdata->num = num;
> > +               mutex_init(&pdata->lock);
> > +       }
> > +
> > +       return pdata;
> > +}
> > +
> > +int fme_feature_num(void)
> > +{
> > +       return FME_FEATURE_ID_MAX;
> > +}
> > +
> > +int port_feature_num(void)
> > +{
> > +       return PORT_FEATURE_ID_MAX;
> > +}
> > +
> > +int fpga_port_id(struct platform_device *pdev)
> > +{
> > +       struct feature_port_header *port_hdr;
> > +       struct feature_port_capability capability;
> > +
> > +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                                              PORT_FEATURE_ID_HEADER);
> > +       WARN_ON(!port_hdr);
> > +
> > +       capability.csr = readq(&port_hdr->capability);
> > +       return capability.port_number;
> > +}
> > +EXPORT_SYMBOL_GPL(fpga_port_id);
> > +
> > +/*
> > + * Enable Port by clear the port soft reset bit, which is set by default.
> > + * The User AFU is unable to respond to any MMIO access while in reset.
> > + * __fpga_port_enable function should only be used after __fpga_port_disable
> > + * function.
> > + */
> > +void __fpga_port_enable(struct platform_device *pdev)
> > +{
> 
> feature-dev.c is handling enumeration and adding port
> enable/disable/etc functions for a specific port device.  I see the
> port as a fpga-bridge.  The enumeration code should be separate from
> the bridge code.  Especially separate from a very specific bridge low
> level device driver implementation, otherwise this becomes obsolete as
> soon as you have another port device with a different register
> implementation.  Even if you handle that, then this enumeration code
> isn't useable by other people who are using fpga-bridge.  The
> fpga-bridge framework exists to separate low level things like how to
> enable/disable a specific bridge device from upper level code that
> knows when to enable/disable it (fpga-region).

Hi Alan

The major purpose of feature-dev is to create infrastructure for feature
device. Please refer to patch 7.  It abstracts feature device common
data structures and functions in feature-dev.c for FME and AFU now (but
may be more in the future). The reason we add port enable/disable/etc
fuctions there for code reuse. e.g FME driver needs port enable/disable
for PR (in the future, to implement the fpga bridge enable_set function),
AFU driver needs similar code to implement reset interface for user space
application, PCIe driver needs port enable to make AFU MMIO region valid,
then it can access Device Feature Header inside this AFU MMIO during
enumeration. Other fpga_port_* are all for code reuse purpose too. So we
should not put these function in the same feature-dev.c but a seperated
file?

Thanks
Hao

> 
> Alan

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features.
  2017-04-04  2:44   ` Moritz Fischer
@ 2017-04-05 12:57     ` Wu Hao
  0 siblings, 0 replies; 93+ messages in thread
From: Wu Hao @ 2017-04-05 12:57 UTC (permalink / raw)
  To: Moritz Fischer
  Cc: atull, moritz.fischer, linux-fpga, linux-kernel, luwei.kang,
	yi.z.zhang, Xiao Guangrong, Tim Whisonant, Enno Luebbers,
	Shiva Rao, Christopher Rauer

On Mon, Apr 03, 2017 at 07:44:55PM -0700, Moritz Fischer wrote:
> Xiao,
> 
> few nits inline, I'll need to come back to this once I went over the
> rest of the patchset ;-)

Sure, Thanks for your comments and review. :)

> 
> On Thu, Mar 30, 2017 at 08:08:04PM +0800, Wu Hao wrote:
> > From: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > 
> > Device Featuer List structure creates a link list of feature headers
> > within the MMIO space to provide an extensiable way of adding features.
> > 
> > The Intel FPGA PCIe driver walks through the feature headers to enumerate
> > feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
> > Function Unit (AFU), and their private sub features. For feature devices,
> > it creates the platform devices and linked the private sub features into
> > their platform data.
> > 
> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
> > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
> > Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > Signed-off-by: Wu Hao <hao.wu@intel.com>
> > ---
> >  drivers/fpga/intel/Makefile      |   2 +-
> >  drivers/fpga/intel/feature-dev.c | 139 +++++++
> >  drivers/fpga/intel/feature-dev.h | 342 ++++++++++++++++
> >  drivers/fpga/intel/pcie.c        | 841 ++++++++++++++++++++++++++++++++++++++-
> >  4 files changed, 1321 insertions(+), 3 deletions(-)
> >  create mode 100644 drivers/fpga/intel/feature-dev.c
> >  create mode 100644 drivers/fpga/intel/feature-dev.h
> > 
> > diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
> > index 61fd8ea..c029940 100644
> > --- a/drivers/fpga/intel/Makefile
> > +++ b/drivers/fpga/intel/Makefile
> > @@ -1,3 +1,3 @@
> >  obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
> >  
> > -intel-fpga-pci-objs := pcie.o
> > +intel-fpga-pci-objs := pcie.o feature-dev.o
> > diff --git a/drivers/fpga/intel/feature-dev.c b/drivers/fpga/intel/feature-dev.c
> > new file mode 100644
> > index 0000000..6952566
> > --- /dev/null
> > +++ b/drivers/fpga/intel/feature-dev.c
> > @@ -0,0 +1,139 @@
> > +/*
> > + * Intel FPGA Feature Device Driver
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <luwei.kang@intel.com>
> > + *   Zhang Yi <yi.z.zhang@intel.com>
> > + *   Wu Hao <hao.wu@intel.com>
> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > + *
> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
> > + * redistributing this file, you may do so under either license. See the
> > + * LICENSE.BSD file under this directory for the BSD license and see
> > + * the COPYING file in the top-level directory for the GPLv2 license.
> > + */
> > +
> > +#include "feature-dev.h"
> > +
> > +void feature_platform_data_add(struct feature_platform_data *pdata,
> > +			       int index, const char *name,
> > +			       int resource_index, void __iomem *ioaddr)
> > +{
> > +	WARN_ON(index >= pdata->num);
> > +
> > +	pdata->features[index].name = name;
> > +	pdata->features[index].resource_index = resource_index;
> > +	pdata->features[index].ioaddr = ioaddr;
> > +}
> > +
> > +int feature_platform_data_size(int num)
> 
> static inline? num can be const

Sure. will fix this.

> 
> > +{
> > +	return sizeof(struct feature_platform_data) +
> > +		num * sizeof(struct feature);
> > +}
> > +
> > +struct feature_platform_data *
> > +feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
> > +{
> > +	struct feature_platform_data *pdata;
> > +
> > +	pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
> > +	if (pdata) {
> > +		pdata->dev = dev;
> > +		pdata->num = num;
> > +		mutex_init(&pdata->lock);
> > +	}
> > +
> > +	return pdata;
> > +}
> > +
> > +int fme_feature_num(void)
> 
> is this used outside of this file? if not -> static

Yes, used in pcie.c

> > +{
> > +	return FME_FEATURE_ID_MAX;
> > +}
> > +
> > +int port_feature_num(void)
> 
> is this used outside of this file? if not -> static

Yes, used in pcie.c

> > +{
> > +	return PORT_FEATURE_ID_MAX;
> > +}
> > +
> > +int fpga_port_id(struct platform_device *pdev)
> > +{
> > +	struct feature_port_header *port_hdr;
> > +	struct feature_port_capability capability;
> > +
> > +	port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> > +					       PORT_FEATURE_ID_HEADER);
> > +	WARN_ON(!port_hdr);
> > +
> > +	capability.csr = readq(&port_hdr->capability);
> > +	return capability.port_number;
> > +}
> > +EXPORT_SYMBOL_GPL(fpga_port_id);
> > +
> > +/*
> > + * Enable Port by clear the port soft reset bit, which is set by default.
> > + * The User AFU is unable to respond to any MMIO access while in reset.
> > + * __fpga_port_enable function should only be used after __fpga_port_disable
> > + * function.
> > + */
> > +void __fpga_port_enable(struct platform_device *pdev)
> > +{
> > +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +	struct feature_port_header *port_hdr;
> > +	struct feature_port_control control;
> > +
> > +	WARN_ON(!pdata->disable_count);
> > +
> > +	if (--pdata->disable_count != 0)
> > +		return;
> > +
> > +	port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> > +					       PORT_FEATURE_ID_HEADER);
> > +	WARN_ON(!port_hdr);
> > +
> > +	control.csr = readq(&port_hdr->control);
> > +	control.port_sftrst = 0x0;
> > +	writeq(control.csr, &port_hdr->control);
> > +}
> > +EXPORT_SYMBOL_GPL(__fpga_port_enable);
> > +
> > +#define RST_POLL_INVL 10 /* us */
> > +#define RST_POLL_TIMEOUT 1000 /* us */
> > +
> > +int __fpga_port_disable(struct platform_device *pdev)
> > +{
> > +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +	struct feature_port_header *port_hdr;
> > +	struct feature_port_control control;
> > +
> > +	if (pdata->disable_count++ != 0)
> > +		return 0;
> > +
> > +	port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> > +					       PORT_FEATURE_ID_HEADER);
> > +	WARN_ON(!port_hdr);
> > +
> > +	/* Set port soft reset */
> > +	control.csr = readq(&port_hdr->control);
> > +	control.port_sftrst = 0x1;
> > +	writeq(control.csr, &port_hdr->control);
> > +
> > +	/*
> > +	 * HW sets ack bit to 1 when all outstanding requests have been drained
> > +	 * on this port and minimum soft reset pulse width has elapsed.
> > +	 * Driver polls port_soft_reset_ack to determine if reset done by HW.
> > +	 */
> > +	control.port_sftrst_ack = 1;
> > +
> > +	if (fpga_wait_register_field(port_sftrst_ack, control,
> > +		&port_hdr->control, RST_POLL_TIMEOUT, RST_POLL_INVL)) {
> > +		dev_err(&pdev->dev, "timeout, fail to reset device\n");
> > +		return -ETIMEDOUT;
> > +	}
> 
> see iopoll comment above.

Yes, will fix this.

> > +
> > +	return 0;
> > +}
> > +EXPORT_SYMBOL_GPL(__fpga_port_disable);
> > diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
> > new file mode 100644
> > index 0000000..a1e6e7d
> > --- /dev/null
> > +++ b/drivers/fpga/intel/feature-dev.h
> > @@ -0,0 +1,342 @@
> > +/*
> > + * Intel FPGA Feature Device Driver Header File
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <luwei.kang@intel.com>
> > + *   Zhang Yi <yi.z.zhang@intel.com>
> > + *   Wu Hao <hao.wu@intel.com>
> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
> > + *
> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
> > + * redistributing this file, you may do so under either license. See the
> > + * LICENSE.BSD file under this directory for the BSD license and see
> > + * the COPYING file in the top-level directory for the GPLv2 license.
> > + */
> > +
> > +#ifndef __INTEL_FPGA_FEATURE_H
> > +#define __INTEL_FPGA_FEATURE_H
> > +
> > +#include <linux/fs.h>
> > +#include <linux/pci.h>
> > +#include <linux/uuid.h>
> > +#include <linux/delay.h>
> > +#include <linux/platform_device.h>
> > +
> > +/* maximum supported number of ports */
> > +#define MAX_FPGA_PORT_NUM 4
> > +/* plus one for fme device */
> > +#define MAX_FEATURE_DEV_NUM	(MAX_FPGA_PORT_NUM + 1)
> > +
> > +#define FME_FEATURE_HEADER          "fme_hdr"
> > +#define FME_FEATURE_THERMAL_MGMT    "fme_thermal"
> > +#define FME_FEATURE_POWER_MGMT      "fme_power"
> > +#define FME_FEATURE_GLOBAL_PERF     "fme_gperf"
> > +#define FME_FEATURE_GLOBAL_ERR      "fme_error"
> > +#define FME_FEATURE_PR_MGMT         "fme_pr"
> > +
> > +#define PORT_FEATURE_HEADER         "port_hdr"
> > +#define PORT_FEATURE_UAFU           "port_uafu"
> > +#define PORT_FEATURE_ERR            "port_err"
> > +#define PORT_FEATURE_UMSG           "port_umsg"
> > +#define PORT_FEATURE_PR             "port_pr"
> > +#define PORT_FEATURE_STP            "port_stp"
> > +
> > +/* All headers and structures must be byte-packed to match the spec. */
> > +#pragma pack(1)
> 
> I recall there being some controversy about this, can't remember which
> way people ended up deciding :)

It seems __packed is better?

> > +
> > +/* common header for all features */
> > +struct feature_header {
> > +	union {
> > +		u64 csr;
> > +		struct {
> > +			u16 id:12;
> > +			u8  revision:4;
> > +			u32 next_header_offset:24; /* offset to next header */
> > +			u32 rsvdz:20;
> > +			u8  type:4;		   /* feature type */
> > +#define FEATURE_TYPE_AFU		0x1
> > +#define FEATURE_TYPE_PRIVATE		0x3
> > +		};
> > +	};
> > +};
> > +
> > +/* common header for non-private features */
> > +struct feature_afu_header {
> > +	uuid_le guid;
> > +	union {
> > +		u64 csr;
> > +		struct {
> > +			u64 next_afu:24;	/* pointer to next afu header */
> > +			u64 rsvdz:40;
> > +		};
> > +	};
> > +};
> > +
> > +/* FME Header Register Set */
> > +/* FME Capability Register */
> > +struct feature_fme_capability {
> > +	union {
> > +		u64 csr;
> > +		struct {
> > +			u8  fabric_verid;	/* Fabric version ID */
> > +			u8  socket_id:1;	/* Socket id */
> > +			u8  rsvdz1:3;
> > +			u8  pcie0_link_avl:1;	/* PCIe0 link availability */
> > +			u8  pcie1_link_avl:1;	/* PCIe1 link availability */
> > +			u8  coherent_link_avl:1;/* Coherent link availability */
> > +			u8  rsvdz2:1;
> > +			u8  iommu_support:1;	/* IOMMU or VT-d supported */
> > +			u8  num_ports:3;	/* Num of ports implemented */
> > +			u8  rsvdz3:4;
> > +			u8  addr_width_bits:6;	/* Address width supported */
> > +			u8  rsvdz4:2;
> > +			u16 cache_size:12;	/* Cache size in kb */
> > +			u8  cache_assoc:4;	/* Cache Associativity */
> > +			u16 rsvdz5:15;
> > +			u8  lock_bit:1;		/* Latched lock bit by BIOS */
> > +		};
> > +	};
> > +};
> > +
> > +/* FME Port Offset Register */
> > +struct feature_fme_port {
> > +	union {
> > +		u64 csr;
> > +		struct {
> > +			u32 port_offset:24;	/* Offset to port header */
> > +			u8  rsvdz1;
> > +			u8  port_bar:3;		/* Bar id */
> > +			u32 rsvdz2:20;
> > +			u8  afu_access_ctrl:1;	/* AFU access type: PF/VF */
> > +			u8  rsvdz3:4;
> > +			u8  port_implemented:1;	/* Port implemented or not */
> > +			u8  rsvdz4:3;
> > +		};
> > +	};
> > +};
> > +
> > +struct feature_fme_header {
> > +	struct feature_header header;
> > +	struct feature_afu_header afu_header;
> > +	u64 rsvd[2];
> > +	struct feature_fme_capability capability;
> > +	struct feature_fme_port port[MAX_FPGA_PORT_NUM];
> > +};
> > +
> > +/* FME Thermal Sub Feature Register Set */
> > +struct feature_fme_thermal {
> > +	struct feature_header header;
> > +};
> > +
> > +/* FME Power Sub Feature Register Set */
> > +struct feature_fme_power {
> > +	struct feature_header header;
> > +};
> > +
> > +/* FME Global Performance Sub Feature Register Set */
> > +struct feature_fme_gperf {
> > +	struct feature_header header;
> > +};
> > +
> > +/* FME Error Sub Feature Register Set */
> > +struct feature_fme_err {
> > +	struct feature_header header;
> > +};
> > +
> > +/* FME Partial Reconfiguration Sub Feature Register Set */
> > +struct feature_fme_pr {
> > +	struct feature_header header;
> > +};
> > +
> > +/* PORT Header Register Set */
> > +/* Port Capability Register */
> > +struct feature_port_capability {
> > +	union {
> > +		u64 csr;
> > +		struct {
> > +			u8  port_number:2;	/* Port Number 0-3 */
> > +			u8  rsvdz1:6;
> > +			u16 mmio_size;		/* User MMIO size in KB */
> > +			u8  rsvdz2;
> > +			u8  sp_intr_num:4;	/* Supported interrupts num */
> > +			u32 rsvdz3:28;
> > +		};
> > +	};
> > +};
> > +
> > +/* Port Control Register */
> > +struct feature_port_control {
> > +	union {
> > +		u64 csr;
> > +		struct {
> > +			u8  port_sftrst:1;	/* Port Soft Reset */
> > +			u8  rsvdz1:1;
> > +			u8  latency_tolerance:1;/* '1' >= 40us, '0' < 40us */
> > +			u8  rsvdz2:1;
> > +			u8  port_sftrst_ack:1;	/* HW ACK for Soft Reset */
> > +			u64 rsvdz3:59;
> > +		};
> > +	};
> > +};
> > +
> > +struct feature_port_header {
> > +	struct feature_header header;
> > +	struct feature_afu_header afu_header;
> > +	u64 rsvd[2];
> > +	struct feature_port_capability capability;
> > +	struct feature_port_control control;
> > +};
> > +
> > +/* PORT Error Sub Feature Register Set */
> > +struct feature_port_error {
> > +	struct feature_header header;
> > +};
> > +
> > +/* PORT Unordered Message Sub Feature Register Set */
> > +struct feature_port_umsg {
> > +	struct feature_header header;
> > +};
> > +
> > +/* PORT SignalTap Sub Feature Register Set */
> > +struct feature_port_stp {
> > +	struct feature_header header;
> > +};
> > +
> > +#pragma pack()
> > +
> > +struct feature {
> > +	const char *name;
> > +	int resource_index;
> > +	void __iomem *ioaddr;
> > +};
> > +
> > +struct feature_platform_data {
> > +	/* list the feature dev to cci_drvdata->port_dev_list. */
> > +	struct list_head node;
> > +	struct mutex lock;
> > +	struct platform_device *dev;
> > +	unsigned int disable_count;	/* count for port disable */
> > +
> > +	int num;			/* number of features */
> > +	struct feature features[0];
> > +};
> > +
> > +enum fme_feature_id {
> > +	FME_FEATURE_ID_HEADER = 0x0,
> > +	FME_FEATURE_ID_THERMAL_MGMT = 0x1,
> > +	FME_FEATURE_ID_POWER_MGMT = 0x2,
> > +	FME_FEATURE_ID_GLOBAL_PERF = 0x3,
> > +	FME_FEATURE_ID_GLOBAL_ERR = 0x4,
> > +	FME_FEATURE_ID_PR_MGMT = 0x5,
> > +	FME_FEATURE_ID_MAX = 0x6,
> > +};
> > +
> > +enum port_feature_id {
> > +	PORT_FEATURE_ID_HEADER = 0x0,
> > +	PORT_FEATURE_ID_ERROR = 0x1,
> > +	PORT_FEATURE_ID_UMSG = 0x2,
> > +	PORT_FEATURE_ID_PR = 0x3,
> > +	PORT_FEATURE_ID_STP = 0x4,
> > +	PORT_FEATURE_ID_UAFU = 0x5,
> > +	PORT_FEATURE_ID_MAX = 0x6,
> > +};
> > +
> > +int fme_feature_num(void);
> > +int port_feature_num(void);
> > +
> > +#define FPGA_FEATURE_DEV_FME		"intel-fpga-fme"
> > +#define FPGA_FEATURE_DEV_PORT		"intel-fpga-port"
> > +
> > +void feature_platform_data_add(struct feature_platform_data *pdata,
> > +			       int index, const char *name,
> > +			       int resource_index, void __iomem *ioaddr);
> > +int feature_platform_data_size(int num);
> > +struct feature_platform_data *
> > +feature_platform_data_alloc_and_init(struct platform_device *dev, int num);
> > +
> > +int fpga_port_id(struct platform_device *pdev);
> > +
> > +static inline int fpga_port_check_id(struct platform_device *pdev,
> > +				     void *pport_id)
> > +{
> > +	return fpga_port_id(pdev) == *(int *)pport_id;
> > +}
> > +
> > +void __fpga_port_enable(struct platform_device *pdev);
> > +int __fpga_port_disable(struct platform_device *pdev);
> > +
> > +static inline void fpga_port_enable(struct platform_device *pdev)
> > +{
> > +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +
> > +	mutex_lock(&pdata->lock);
> > +	__fpga_port_enable(pdev);
> > +	mutex_unlock(&pdata->lock);
> > +}
> > +
> > +static inline int fpga_port_disable(struct platform_device *pdev)
> > +{
> > +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +	int ret;
> > +
> > +	mutex_lock(&pdata->lock);
> > +	ret = __fpga_port_disable(pdev);
> > +	mutex_unlock(&pdata->lock);
> > +
> > +	return ret;
> > +}
> > +
> > +static inline int __fpga_port_reset(struct platform_device *pdev)
> > +{
> > +	int ret;
> > +
> > +	ret = __fpga_port_disable(pdev);
> > +	if (ret)
> > +		return ret;
> > +
> > +	__fpga_port_enable(pdev);
> > +	return 0;
> > +}
> > +
> > +static inline int fpga_port_reset(struct platform_device *pdev)
> > +{
> > +	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +	int ret;
> > +
> > +	mutex_lock(&pdata->lock);
> > +	ret = __fpga_port_reset(pdev);
> > +	mutex_unlock(&pdata->lock);
> > +	return ret;
> > +}
> > +
> > +static inline void __iomem *
> > +get_feature_ioaddr_by_index(struct device *dev, int index)
> > +{
> > +	struct feature_platform_data *pdata = dev_get_platdata(dev);
> > +
> > +	return pdata->features[index].ioaddr;
> > +}
> > +
> > +/*
> > + * Wait register's _field to be changed to the given value (_expect's _field)
> > + * by polling with given interval and timeout.
> > + */
> > +#define fpga_wait_register_field(_field, _expect, _reg_addr, _timeout, _invl)\
> > +({									     \
> > +	int wait = 0;							     \
> > +	int ret = -ETIMEDOUT;						     \
> > +	typeof(_expect) value;						     \
> > +	for (; wait <= _timeout; wait += _invl) {			     \
> > +		value.csr = readq(_reg_addr);				     \
> > +		if (_expect._field == value._field) {			     \
> > +			ret = 0;					     \
> > +			break;						     \
> > +		}							     \
> > +		udelay(_invl);						     \
> > +	}								     \
> > +	ret;								     \
> > +})
> 
> can't you use iopoll and friends instead of this?

Yes, will fix this.

Thanks for the recommendation.

> 
> > +
> > +#endif
> > diff --git a/drivers/fpga/intel/pcie.c b/drivers/fpga/intel/pcie.c
> > index 132d9da..28df63e 100644
> > --- a/drivers/fpga/intel/pcie.c
> > +++ b/drivers/fpga/intel/pcie.c
> > @@ -25,10 +25,827 @@
> >  #include <linux/stddef.h>
> >  #include <linux/errno.h>
> >  #include <linux/aer.h>
> > +#include <linux/fpga/fpga-dev.h>
> > +
> > +#include "feature-dev.h"
> >  
> >  #define DRV_VERSION	"EXPERIMENTAL VERSION"
> >  #define DRV_NAME	"intel-fpga-pci"
> >  
> > +#define INTEL_FPGA_DEV	"intel-fpga-dev"
> > +
> > +static DEFINE_MUTEX(fpga_id_mutex);
> > +
> > +enum fpga_id_type {
> > +	FME_ID,		/* fme id allocation and mapping */
> > +	PORT_ID,	/* port id allocation and mapping */
> > +	FPGA_ID_MAX,
> > +};
> > +
> > +/* it is protected by fpga_id_mutex */
> > +static struct idr fpga_ids[FPGA_ID_MAX];
> > +
> > +struct cci_drvdata {
> > +	struct device *fme_dev;
> > +
> > +	struct mutex lock;
> > +	struct list_head port_dev_list;
> > +
> > +	struct list_head regions; /* global list of pci bar mapping region */
> > +};
> > +
> > +/* pci bar mapping info */
> > +struct cci_pci_region {
> > +	int bar;
> > +	void __iomem *ioaddr;	/* pointer to mapped bar region */
> > +	struct list_head node;
> > +};
> > +
> > +static void fpga_ids_init(void)
> > +{
> > +	int i;
> > +
> > +	for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> > +		idr_init(fpga_ids + i);
> > +}
> > +
> > +static void fpga_ids_destroy(void)
> > +{
> > +	int i;
> > +
> > +	for (i = 0; i < ARRAY_SIZE(fpga_ids); i++)
> > +		idr_destroy(fpga_ids + i);
> > +}
> > +
> > +static int alloc_fpga_id(enum fpga_id_type type, struct device *dev)
> > +{
> > +	int id;
> > +
> > +	WARN_ON(type >= FPGA_ID_MAX);
> > +	mutex_lock(&fpga_id_mutex);
> > +	id = idr_alloc(fpga_ids + type, dev, 0, 0, GFP_KERNEL);
> > +	mutex_unlock(&fpga_id_mutex);
> > +	return id;
> > +}
> > +
> > +static void free_fpga_id(enum fpga_id_type type, int id)
> > +{
> > +	WARN_ON(type >= FPGA_ID_MAX);
> > +	mutex_lock(&fpga_id_mutex);
> > +	idr_remove(fpga_ids + type, id);
> > +	mutex_unlock(&fpga_id_mutex);
> > +}
> > +
> > +static void cci_pci_add_port_dev(struct pci_dev *pdev,
> > +				 struct platform_device *port_dev)
> > +{
> > +	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> > +	struct feature_platform_data *pdata = dev_get_platdata(&port_dev->dev);
> > +
> > +	mutex_lock(&drvdata->lock);
> > +	list_add(&pdata->node, &drvdata->port_dev_list);
> > +	get_device(&pdata->dev->dev);
> > +	mutex_unlock(&drvdata->lock);
> > +}
> > +
> > +static void cci_pci_remove_port_devs(struct pci_dev *pdev)
> > +{
> > +	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> > +	struct feature_platform_data *pdata, *ptmp;
> > +
> > +	mutex_lock(&drvdata->lock);
> > +	list_for_each_entry_safe(pdata, ptmp, &drvdata->port_dev_list, node) {
> > +		struct platform_device *port_dev = pdata->dev;
> > +
> > +		/* the port should be unregistered first. */
> > +		WARN_ON(device_is_registered(&port_dev->dev));
> > +		list_del(&pdata->node);
> > +		free_fpga_id(PORT_ID, port_dev->id);
> > +		put_device(&port_dev->dev);
> > +	}
> > +	mutex_unlock(&drvdata->lock);
> > +}
> > +
> > +/* info collection during feature dev build. */
> > +struct build_feature_devs_info {
> > +	struct pci_dev *pdev;
> > +
> > +	/*
> > +	 * PCI BAR mapping info. Parsing feature list starts from
> > +	 * BAR 0 and switch to different BARs to parse Port
> > +	 */
> > +	void __iomem *ioaddr;
> > +	void __iomem *ioend;
> > +	int current_bar;
> > +
> > +	/* points to FME header where the port offset is figured out. */
> > +	void __iomem *pfme_hdr;
> > +
> > +	/* the container device for all feature devices */
> > +	struct fpga_dev *parent_dev;
> > +
> > +	/* current feature device */
> > +	struct platform_device *feature_dev;
> > +};
> > +
> > +static void cci_pci_release_regions(struct pci_dev *pdev)
> > +{
> > +	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> > +	struct cci_pci_region *tmp, *region;
> > +
> > +	list_for_each_entry_safe(region, tmp, &drvdata->regions, node) {
> > +		list_del(&region->node);
> > +		if (region->ioaddr)
> > +			pci_iounmap(pdev, region->ioaddr);
> > +		devm_kfree(&pdev->dev, region);
> > +	}
> > +}
> > +
> > +static void __iomem *cci_pci_ioremap_bar(struct pci_dev *pdev, int bar)
> > +{
> > +	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> > +	struct cci_pci_region *region;
> > +
> > +	list_for_each_entry(region, &drvdata->regions, node)
> > +		if (region->bar == bar) {
> > +			dev_dbg(&pdev->dev, "BAR %d region exists\n", bar);
> > +			return region->ioaddr;
> > +		}
> > +
> > +	region = devm_kzalloc(&pdev->dev, sizeof(*region), GFP_KERNEL);
> > +	if (!region)
> > +		return NULL;
> > +
> > +	region->bar = bar;
> > +	region->ioaddr = pci_ioremap_bar(pdev, bar);
> > +	if (!region->ioaddr) {
> > +		dev_err(&pdev->dev, "can't ioremap memory from BAR %d.\n", bar);
> > +		devm_kfree(&pdev->dev, region);
> > +		return NULL;
> > +	}
> > +
> > +	list_add(&region->node, &drvdata->regions);
> > +	return region->ioaddr;
> > +}
> > +
> > +static int parse_start_from(struct build_feature_devs_info *binfo, int bar)
> > +{
> > +	binfo->ioaddr = cci_pci_ioremap_bar(binfo->pdev, bar);
> > +	if (!binfo->ioaddr)
> > +		return -ENOMEM;
> > +
> > +	binfo->current_bar = bar;
> > +	binfo->ioend = binfo->ioaddr + pci_resource_len(binfo->pdev, bar);
> > +	return 0;
> > +}
> > +
> > +static int parse_start(struct build_feature_devs_info *binfo)
> > +{
> > +	/* fpga feature list starts from BAR 0 */
> > +	return parse_start_from(binfo, 0);
> > +}
> > +
> > +/* switch the memory mapping to BAR# @bar */
> > +static int parse_switch_to(struct build_feature_devs_info *binfo, int bar)
> > +{
> > +	return parse_start_from(binfo, bar);
> > +}
> > +
> > +static struct build_feature_devs_info *
> > +build_info_alloc_and_init(struct pci_dev *pdev)
> > +{
> > +	struct build_feature_devs_info *binfo;
> > +
> > +	binfo = devm_kzalloc(&pdev->dev, sizeof(*binfo), GFP_KERNEL);
> > +	if (binfo)
> > +		binfo->pdev = pdev;
> > +
> > +	return binfo;
> > +}
> > +
> > +static enum fpga_id_type feature_dev_id_type(struct platform_device *pdev)
> > +{
> > +	if (!strcmp(pdev->name, FPGA_FEATURE_DEV_FME))
> > +		return FME_ID;
> > +
> > +	if (!strcmp(pdev->name, FPGA_FEATURE_DEV_PORT))
> > +		return PORT_ID;
> > +
> > +	WARN_ON(1);
> > +	return FPGA_ID_MAX;
> > +}
> > +
> > +/*
> > + * register current feature device, it is called when we need to switch to
> > + * another feature parsing or we have parsed all features
> > + */
> > +static int build_info_commit_dev(struct build_feature_devs_info *binfo)
> > +{
> > +	int ret;
> > +
> > +	if (!binfo->feature_dev)
> > +		return 0;
> > +
> > +	ret = platform_device_add(binfo->feature_dev);
> > +	if (!ret) {
> > +		struct cci_drvdata *drvdata;
> > +
> > +		drvdata = dev_get_drvdata(&binfo->pdev->dev);
> > +		if (feature_dev_id_type(binfo->feature_dev) == PORT_ID)
> > +			cci_pci_add_port_dev(binfo->pdev, binfo->feature_dev);
> > +		else
> > +			drvdata->fme_dev = get_device(&binfo->feature_dev->dev);
> > +
> > +		/*
> > +		 * reset it to avoid build_info_free() freeing their resource.
> > +		 *
> > +		 * The resource of successfully registered feature devices
> > +		 * will be freed by platform_device_unregister(). See the
> > +		 * comments in build_info_create_dev().
> > +		 */
> > +		binfo->feature_dev = NULL;
> > +	}
> > +
> > +	return ret;
> > +}
> > +
> > +static int
> > +build_info_create_dev(struct build_feature_devs_info *binfo,
> > +		      enum fpga_id_type type, int feature_nr, const char *name)
> > +{
> > +	struct platform_device *fdev;
> > +	struct resource *res;
> > +	struct feature_platform_data *pdata;
> > +	int ret;
> > +
> > +	/* we will create a new device, commit current device first */
> > +	ret = build_info_commit_dev(binfo);
> > +	if (ret)
> > +		return ret;
> > +
> > +	/*
> > +	 * we use -ENODEV as the initialization indicator which indicates
> > +	 * whether the id need to be reclaimed
> > +	 */
> > +	fdev = binfo->feature_dev = platform_device_alloc(name, -ENODEV);
> > +	if (!fdev)
> > +		return -ENOMEM;
> > +
> > +	fdev->id = alloc_fpga_id(type, &fdev->dev);
> > +	if (fdev->id < 0)
> > +		return fdev->id;
> > +
> > +	fdev->dev.parent = &binfo->parent_dev->dev;
> > +
> > +	/*
> > +	 * we need not care the memory which is associated with the
> we do not need to care *for* the memory ;-)
> > +	 * platform device. After call platform_device_unregister(),
> After calling ...

Will fix them. Thanks.

> > +	 * it will be automatically freed by device's
> > +	 * release() callback, platform_device_release().
> > +	 */
> > +	pdata = feature_platform_data_alloc_and_init(fdev, feature_nr);
> > +	if (!pdata)
> > +		return -ENOMEM;
> > +
> > +	/*
> > +	 * the count should be initialized to 0 to make sure
> > +	 *__fpga_port_enable() following __fpga_port_disable()
> > +	 * works properly for port device.
> > +	 * and it should always be 0 for fme device.
> > +	 */
> > +	WARN_ON(pdata->disable_count);
> > +
> > +	fdev->dev.platform_data = pdata;
> > +	fdev->num_resources = feature_nr;
> > +	fdev->resource = kcalloc(feature_nr, sizeof(*res), GFP_KERNEL);
> > +	if (!fdev->resource)
> > +		return -ENOMEM;
> > +
> > +	return 0;
> > +}
> > +
> > +static int remove_feature_dev(struct device *dev, void *data)
> > +{
> > +	struct platform_device *pdev = to_platform_device(dev);
> > +
> > +	platform_device_unregister(pdev);
> > +	return 0;
> > +}
> > +
> > +static int remove_parent_dev(struct device *dev, void *data)
> > +{
> > +	/* remove platform devices attached in the parent device */
> > +	device_for_each_child(dev, NULL, remove_feature_dev);
> > +	fpga_dev_destroy(to_fpga_dev(dev));
> > +	return 0;
> > +}
> > +
> > +static void remove_all_devs(struct pci_dev *pdev)
> > +{
> > +	/* remove parent device and all its children. */
> > +	device_for_each_child(&pdev->dev, NULL, remove_parent_dev);
> > +}
> > +
> > +static void build_info_free(struct build_feature_devs_info *binfo)
> > +{
> > +	if (!IS_ERR_OR_NULL(binfo->parent_dev))
> > +		remove_all_devs(binfo->pdev);
> > +
> > +	/*
> > +	 * it is a valid id, free it. See comments in
> > +	 * build_info_create_dev()
> > +	 */
> > +	if (binfo->feature_dev && binfo->feature_dev->id >= 0)
> > +		free_fpga_id(feature_dev_id_type(binfo->feature_dev),
> > +			     binfo->feature_dev->id);
> > +
> > +	platform_device_put(binfo->feature_dev);
> > +
> > +	devm_kfree(&binfo->pdev->dev, binfo);
> > +}
> > +
> > +#define FEATURE_TYPE_AFU	0x1
> > +#define FEATURE_TYPE_PRIVATE	0x3
> > +
> > +/* FME and PORT GUID are fixed */
> > +#define FEATURE_FME_GUID "f9e17764-38f0-82fe-e346-524ae92aafbf"
> > +#define FEATURE_PORT_GUID "6b355b87-b06c-9642-eb42-8d139398b43a"
> > +
> > +static bool feature_is_fme(struct feature_afu_header *afu_hdr)
> > +{
> > +	uuid_le u;
> > +
> > +	uuid_le_to_bin(FEATURE_FME_GUID, &u);
> > +
> > +	return !uuid_le_cmp(u, afu_hdr->guid);
> > +}
> > +
> > +static bool feature_is_port(struct feature_afu_header *afu_hdr)
> > +{
> > +	uuid_le u;
> > +
> > +	uuid_le_to_bin(FEATURE_PORT_GUID, &u);
> > +
> > +	return !uuid_le_cmp(u, afu_hdr->guid);
> > +}
> > +
> > +/*
> > + * UAFU GUID is dynamic as it can be changed after FME downloads different
> > + * Green Bitstream to the port, so we treat the unknown GUIDs which are
> > + * attached on port's feature list as UAFU.
> > + */
> > +static bool feature_is_UAFU(struct build_feature_devs_info *binfo)
> > +{
> > +	if (!binfo->feature_dev ||
> > +	      feature_dev_id_type(binfo->feature_dev) != PORT_ID)
> > +		return false;
> > +
> > +	return true;
> > +}
> > +
> > +static void
> > +build_info_add_sub_feature(struct build_feature_devs_info *binfo,
> > +			   int feature_id, const char *feature_name,
> > +			   resource_size_t resource_size, void __iomem *start)
> > +{
> > +
> > +	struct platform_device *fdev = binfo->feature_dev;
> > +	struct feature_platform_data *pdata = dev_get_platdata(&fdev->dev);
> > +	struct resource *res = &fdev->resource[feature_id];
> > +
> > +	res->start = pci_resource_start(binfo->pdev, binfo->current_bar) +
> > +		start - binfo->ioaddr;
> > +	res->end = res->start + resource_size - 1;
> > +	res->flags = IORESOURCE_MEM;
> > +	res->name = feature_name;
> > +
> > +	feature_platform_data_add(pdata, feature_id,
> > +				  feature_name, feature_id, start);
> > +}
> > +
> > +struct feature_info {
> > +	const char *name;
> > +	resource_size_t resource_size;
> > +	int feature_index;
> > +};
> > +
> > +/* indexed by fme feature IDs which are defined in 'enum fme_feature_id'. */
> > +static struct feature_info fme_features[] = {
> > +	{
> > +		.name = FME_FEATURE_HEADER,
> > +		.resource_size = sizeof(struct feature_fme_header),
> > +		.feature_index = FME_FEATURE_ID_HEADER,
> > +	},
> > +	{
> > +		.name = FME_FEATURE_THERMAL_MGMT,
> > +		.resource_size = sizeof(struct feature_fme_thermal),
> > +		.feature_index = FME_FEATURE_ID_THERMAL_MGMT,
> > +	},
> > +	{
> > +		.name = FME_FEATURE_POWER_MGMT,
> > +		.resource_size = sizeof(struct feature_fme_power),
> > +		.feature_index = FME_FEATURE_ID_POWER_MGMT,
> > +	},
> > +	{
> > +		.name = FME_FEATURE_GLOBAL_PERF,
> > +		.resource_size = sizeof(struct feature_fme_gperf),
> > +		.feature_index = FME_FEATURE_ID_GLOBAL_PERF,
> > +	},
> > +	{
> > +		.name = FME_FEATURE_GLOBAL_ERR,
> > +		.resource_size = sizeof(struct feature_fme_err),
> > +		.feature_index = FME_FEATURE_ID_GLOBAL_ERR,
> > +	},
> > +	{
> > +		.name = FME_FEATURE_PR_MGMT,
> > +		.resource_size = sizeof(struct feature_fme_pr),
> > +		.feature_index = FME_FEATURE_ID_PR_MGMT,
> > +	}
> > +};
> > +
> > +/* indexed by port feature IDs which are defined in 'enum port_feature_id'. */
> > +static struct feature_info port_features[] = {
> 
> Could probably be const

The resource_size will be updated during the enumeration.
Driver reads register to know the MMIO region size for accelerator.

Thanks
Hao

> > +	{
> > +		.name = PORT_FEATURE_HEADER,
> > +		.resource_size = sizeof(struct feature_port_header),
> > +		.feature_index = PORT_FEATURE_ID_HEADER,
> > +	},
> > +	{
> > +		.name = PORT_FEATURE_ERR,
> > +		.resource_size = sizeof(struct feature_port_error),
> > +		.feature_index = PORT_FEATURE_ID_ERROR,
> > +	},
> > +	{
> > +		.name = PORT_FEATURE_UMSG,
> > +		.resource_size = sizeof(struct feature_port_umsg),
> > +		.feature_index = PORT_FEATURE_ID_UMSG,
> > +	},
> > +	{
> > +		/* This feature isn't available for now */
> > +		.name = PORT_FEATURE_PR,
> > +		.resource_size = 0,
> > +		.feature_index = PORT_FEATURE_ID_PR,
> > +	},
> > +	{
> > +		.name = PORT_FEATURE_STP,
> > +		.resource_size = sizeof(struct feature_port_stp),
> > +		.feature_index = PORT_FEATURE_ID_STP,
> > +	},
> > +	{
> > +		/*
> > +		 * For User AFU feature, its region size is not fixed, but
> > +		 * reported by register PortCapability.mmio_size. Resource
> > +		 * size of UAFU will be set while parse port device.
> > +		 */
> > +		.name = PORT_FEATURE_UAFU,
> > +		.resource_size = 0,
> > +		.feature_index = PORT_FEATURE_ID_UAFU,
> > +	},
> > +};
> > +
> > +static int
> > +create_feature_instance(struct build_feature_devs_info *binfo,
> > +			void __iomem *start, struct feature_info *finfo)
> > +{
> > +	if (binfo->ioend - start < finfo->resource_size)
> > +		return -EINVAL;
> > +
> > +	build_info_add_sub_feature(binfo, finfo->feature_index, finfo->name,
> > +				   finfo->resource_size, start);
> > +	return 0;
> > +}
> > +
> > +static int parse_feature_fme(struct build_feature_devs_info *binfo,
> > +			     void __iomem *start)
> > +{
> > +	struct cci_drvdata *drvdata = dev_get_drvdata(&binfo->pdev->dev);
> > +	int ret;
> > +
> > +	ret = build_info_create_dev(binfo, FME_ID, fme_feature_num(),
> > +					FPGA_FEATURE_DEV_FME);
> > +	if (ret)
> > +		return ret;
> > +
> > +	if (drvdata->fme_dev) {
> > +		dev_err(&binfo->pdev->dev, "Multiple FMEs are detected.\n");
> > +		return -EINVAL;
> > +	}
> > +
> > +	return create_feature_instance(binfo, start,
> > +				       &fme_features[FME_FEATURE_ID_HEADER]);
> > +}
> > +
> > +static int parse_feature_fme_private(struct build_feature_devs_info *binfo,
> > +				     struct feature_header *hdr)
> > +{
> > +	struct feature_header header;
> > +
> > +	header.csr = readq(hdr);
> > +
> > +	if (header.id >= ARRAY_SIZE(fme_features)) {
> > +		dev_info(&binfo->pdev->dev, "FME feature id %x is not supported yet.\n",
> > +			 header.id);
> > +		return 0;
> > +	}
> > +
> > +	return create_feature_instance(binfo, hdr, &fme_features[header.id]);
> > +}
> > +
> > +static int parse_feature_port(struct build_feature_devs_info *binfo,
> > +			     void __iomem *start)
> > +{
> > +	int ret;
> > +
> > +	ret = build_info_create_dev(binfo, PORT_ID, port_feature_num(),
> > +					FPGA_FEATURE_DEV_PORT);
> > +	if (ret)
> > +		return ret;
> > +
> > +	return create_feature_instance(binfo, start,
> > +				       &port_features[PORT_FEATURE_ID_HEADER]);
> > +}
> > +
> > +static void enable_port_uafu(struct build_feature_devs_info *binfo,
> > +			     void __iomem *start)
> > +{
> > +	enum port_feature_id id = PORT_FEATURE_ID_UAFU;
> > +	struct feature_port_header *port_hdr;
> > +	struct feature_port_capability capability;
> > +
> > +	port_hdr = (struct feature_port_header *)start;
> > +	capability.csr = readq(&port_hdr->capability);
> > +	port_features[id].resource_size = capability.mmio_size << 10;
> > +
> > +	/*
> > +	 * To enable User AFU, driver needs to clear reset bit on related port,
> > +	 * otherwise the mmio space of this user AFU will be invalid.
> > +	 */
> > +	if (port_features[id].resource_size)
> > +		fpga_port_reset(binfo->feature_dev);
> > +}
> > +
> > +static int parse_feature_port_private(struct build_feature_devs_info *binfo,
> > +				      struct feature_header *hdr)
> > +{
> > +	struct feature_header header;
> > +	enum port_feature_id id;
> > +
> > +	header.csr = readq(hdr);
> > +	/*
> > +	 * the region of port feature id is [0x10, 0x13], + 1 to reserve 0
> > +	 * which is dedicated for port-hdr.
> > +	 */
> > +	id = (header.id & 0x000f) + 1;
> > +
> > +	if (id >= ARRAY_SIZE(port_features)) {
> > +		dev_info(&binfo->pdev->dev, "Port feature id %x is not supported yet.\n",
> > +			 header.id);
> > +		return 0;
> > +	}
> > +
> > +	return create_feature_instance(binfo, hdr, &port_features[id]);
> > +}
> > +
> > +static int parse_feature_port_uafu(struct build_feature_devs_info *binfo,
> > +				 struct feature_header *hdr)
> > +{
> > +	enum port_feature_id id = PORT_FEATURE_ID_UAFU;
> > +	int ret;
> > +
> > +	if (port_features[id].resource_size) {
> > +		ret = create_feature_instance(binfo, hdr, &port_features[id]);
> > +		port_features[id].resource_size = 0;
> > +	} else {
> > +		dev_err(&binfo->pdev->dev, "the uafu feature header is mis-configured.\n");
> > +		ret = -EINVAL;
> > +	}
> > +
> > +	return ret;
> > +}
> > +
> > +static int parse_feature_afus(struct build_feature_devs_info *binfo,
> > +			      struct feature_header *hdr)
> > +{
> > +	int ret;
> > +	struct feature_afu_header *afu_hdr, header;
> > +	void __iomem *start;
> > +	void __iomem *end = binfo->ioend;
> > +
> > +	start = hdr;
> > +	for (; start < end; start += header.next_afu) {
> > +		if (end - start < (sizeof(*afu_hdr) + sizeof(*hdr)))
> > +			return -EINVAL;
> > +
> > +		hdr = start;
> > +		afu_hdr = (struct feature_afu_header *) (hdr + 1);
> > +		header.csr = readq(&afu_hdr->csr);
> > +
> > +		if (feature_is_fme(afu_hdr)) {
> > +			ret = parse_feature_fme(binfo, hdr);
> > +			binfo->pfme_hdr = hdr;
> > +			if (ret)
> > +				return ret;
> > +		} else if (feature_is_port(afu_hdr)) {
> > +			ret = parse_feature_port(binfo, hdr);
> > +			enable_port_uafu(binfo, hdr);
> > +			if (ret)
> > +				return ret;
> > +		} else if (feature_is_UAFU(binfo)) {
> > +			ret = parse_feature_port_uafu(binfo, hdr);
> > +			if (ret)
> > +				return ret;
> > +		} else
> > +			dev_info(&binfo->pdev->dev, "AFU GUID %pUl is not supported yet.\n",
> > +				 afu_hdr->guid.b);
> > +
> > +		if (!header.next_afu)
> > +			break;
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +static int parse_feature_private(struct build_feature_devs_info *binfo,
> > +				 struct feature_header *hdr)
> > +{
> > +	struct feature_header header;
> > +
> > +	header.csr = readq(hdr);
> > +
> > +	if (!binfo->feature_dev) {
> > +		dev_err(&binfo->pdev->dev, "the private feature %x does not belong to any AFU.\n",
> > +			header.id);
> > +		return -EINVAL;
> > +	}
> > +
> > +	switch (feature_dev_id_type(binfo->feature_dev)) {
> > +	case FME_ID:
> > +		return parse_feature_fme_private(binfo, hdr);
> > +	case PORT_ID:
> > +		return parse_feature_port_private(binfo, hdr);
> > +	default:
> > +		dev_info(&binfo->pdev->dev, "private feature %x belonging to AFU %s is not supported yet.\n",
> > +			 header.id, binfo->feature_dev->name);
> > +	}
> > +	return 0;
> > +}
> > +
> > +static int parse_feature(struct build_feature_devs_info *binfo,
> > +			 struct feature_header *hdr)
> > +{
> > +	struct feature_header header;
> > +	int ret = 0;
> > +
> > +	header.csr = readq(hdr);
> > +
> > +	switch (header.type) {
> > +	case FEATURE_TYPE_AFU:
> > +		ret = parse_feature_afus(binfo, hdr);
> > +		break;
> > +	case FEATURE_TYPE_PRIVATE:
> > +		ret = parse_feature_private(binfo, hdr);
> > +		break;
> > +	default:
> > +		dev_info(&binfo->pdev->dev,
> > +			 "Feature Type %x is not supported.\n", hdr->type);
> > +	};
> > +
> > +	return ret;
> > +}
> > +
> > +static int
> > +parse_feature_list(struct build_feature_devs_info *binfo, void __iomem *start)
> > +{
> > +	struct feature_header *hdr, header;
> > +	void __iomem *end = binfo->ioend;
> > +	int ret = 0;
> > +
> > +	for (; start < end; start += header.next_header_offset) {
> > +		if (end - start < sizeof(*hdr)) {
> > +			dev_err(&binfo->pdev->dev, "The region is too small to contain a feature.\n");
> > +			ret =  -EINVAL;
> > +			break;
> > +		}
> > +
> > +		hdr = (struct feature_header *)start;
> > +		ret = parse_feature(binfo, hdr);
> > +		if (ret)
> > +			break;
> > +
> > +		header.csr = readq(hdr);
> > +		if (!header.next_header_offset)
> > +			break;
> > +	}
> > +
> > +	return ret;
> > +}
> > +
> > +static int parse_ports_from_fme(struct build_feature_devs_info *binfo)
> > +{
> > +	struct feature_fme_header *fme_hdr;
> > +	struct feature_fme_port port;
> > +	int i = 0, ret = 0;
> > +
> > +	if (binfo->pfme_hdr == NULL) {
> > +		dev_dbg(&binfo->pdev->dev, "VF is detected.\n");
> > +		return ret;
> > +	}
> > +
> > +	fme_hdr = binfo->pfme_hdr;
> > +
> > +	do {
> > +		port.csr = readq(&fme_hdr->port[i]);
> > +		if (!port.port_implemented)
> > +			break;
> > +
> > +		ret = parse_switch_to(binfo, port.port_bar);
> > +		if (ret)
> > +			break;
> > +
> > +		ret = parse_feature_list(binfo,
> > +				binfo->ioaddr + port.port_offset);
> > +		if (ret)
> > +			break;
> > +	} while (++i < MAX_FPGA_PORT_NUM);
> > +
> > +	return ret;
> > +}
> > +
> > +static int create_init_drvdata(struct pci_dev *pdev)
> > +{
> > +	struct cci_drvdata *drvdata;
> > +
> > +	drvdata = devm_kzalloc(&pdev->dev, sizeof(*drvdata), GFP_KERNEL);
> > +	if (!drvdata)
> > +		return -ENOMEM;
> > +
> > +	mutex_init(&drvdata->lock);
> > +	INIT_LIST_HEAD(&drvdata->port_dev_list);
> > +	INIT_LIST_HEAD(&drvdata->regions);
> > +
> > +	dev_set_drvdata(&pdev->dev, drvdata);
> > +	return 0;
> > +}
> > +
> > +static void destroy_drvdata(struct pci_dev *pdev)
> > +{
> > +	struct cci_drvdata *drvdata = dev_get_drvdata(&pdev->dev);
> > +
> > +	if (drvdata->fme_dev) {
> > +		/* fme device should be unregistered first. */
> > +		WARN_ON(device_is_registered(drvdata->fme_dev));
> > +		free_fpga_id(FME_ID, to_platform_device(drvdata->fme_dev)->id);
> > +		put_device(drvdata->fme_dev);
> > +	}
> > +
> > +	cci_pci_remove_port_devs(pdev);
> > +	cci_pci_release_regions(pdev);
> > +	dev_set_drvdata(&pdev->dev, NULL);
> > +	devm_kfree(&pdev->dev, drvdata);
> > +}
> > +
> > +static int cci_pci_create_feature_devs(struct pci_dev *pdev)
> > +{
> > +	struct build_feature_devs_info *binfo;
> > +	int ret;
> > +
> > +	binfo = build_info_alloc_and_init(pdev);
> > +	if (!binfo)
> > +		return -ENOMEM;
> > +
> > +	binfo->parent_dev = fpga_dev_create(&pdev->dev, INTEL_FPGA_DEV);
> > +	if (IS_ERR(binfo->parent_dev)) {
> > +		ret = PTR_ERR(binfo->parent_dev);
> > +		goto free_binfo_exit;
> > +	}
> > +
> > +	ret = parse_start(binfo);
> > +	if (ret)
> > +		goto free_binfo_exit;
> > +
> > +	ret = parse_feature_list(binfo, binfo->ioaddr);
> > +	if (ret)
> > +		goto free_binfo_exit;
> > +
> > +	ret = parse_ports_from_fme(binfo);
> > +	if (ret)
> > +		goto free_binfo_exit;
> > +
> > +	ret = build_info_commit_dev(binfo);
> > +	if (ret)
> > +		goto free_binfo_exit;
> > +
> > +	/*
> > +	 * everything is okay, reset ->parent_dev to stop it being
> > +	 * freed by build_info_free()
> > +	 */
> > +	binfo->parent_dev = NULL;
> > +
> > +free_binfo_exit:
> > +	build_info_free(binfo);
> > +	return ret;
> > +}
> > +
> >  /* PCI Device ID */
> >  #define PCIe_DEVICE_ID_PF_INT_5_X	0xBCBD
> >  #define PCIe_DEVICE_ID_PF_INT_6_X	0xBCC0
> > @@ -83,9 +900,18 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
> >  		goto release_region_exit;
> >  	}
> >  
> > -	/* TODO: create and add the platform device per feature list */
> > +	ret = create_init_drvdata(pcidev);
> > +	if (ret)
> > +		goto release_region_exit;
> > +
> > +	ret = cci_pci_create_feature_devs(pcidev);
> > +	if (ret)
> > +		goto destroy_drvdata_exit;
> > +
> >  	return 0;
> >  
> > +destroy_drvdata_exit:
> > +	destroy_drvdata(pcidev);
> >  release_region_exit:
> >  	pci_release_regions(pcidev);
> >  disable_error_report_exit:
> > @@ -97,6 +923,8 @@ int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
> >  
> >  static void cci_pci_remove(struct pci_dev *pcidev)
> >  {
> > +	remove_all_devs(pcidev);
> > +	destroy_drvdata(pcidev);
> >  	pci_release_regions(pcidev);
> >  	pci_disable_pcie_error_reporting(pcidev);
> >  	pci_disable_device(pcidev);
> > @@ -111,14 +939,23 @@ static struct pci_driver cci_pci_driver = {
> >  
> >  static int __init ccidrv_init(void)
> >  {
> > +	int ret;
> > +
> >  	pr_info("Intel(R) FPGA PCIe Driver: Version %s\n", DRV_VERSION);
> >  
> > -	return pci_register_driver(&cci_pci_driver);
> > +	fpga_ids_init();
> > +
> > +	ret = pci_register_driver(&cci_pci_driver);
> > +	if (ret)
> > +		fpga_ids_destroy();
> > +
> > +	return ret;
> >  }
> >  
> >  static void __exit ccidrv_exit(void)
> >  {
> >  	pci_unregister_driver(&cci_pci_driver);
> > +	fpga_ids_destroy();
> >  }
> >  
> >  module_init(ccidrv_init);
> > -- 
> > 2.7.4
> > 
> 
> Thanks,
> 
> Moritz

^ permalink raw reply	[flat|nested] 93+ messages in thread

* RE: [PATCH 03/16] fpga: intel: add FPGA PCIe device driver
  2017-04-04  2:10   ` Moritz Fischer
@ 2017-04-05 13:14     ` Wu, Hao
  0 siblings, 0 replies; 93+ messages in thread
From: Wu, Hao @ 2017-04-05 13:14 UTC (permalink / raw)
  To: Moritz Fischer
  Cc: atull, moritz.fischer, linux-fpga, linux-kernel, Kang, Luwei,
	Zhang, Yi Z, Whisonant, Tim, Luebbers, Enno, Rao, Shiva, Rauer,
	Christopher, Xiao Guangrong, Wu, Hao

> > +#include <linux/pci.h>
> > +#include <linux/types.h>
> > +#include <linux/kernel.h>
> > +#include <linux/module.h>
> > +#include <linux/stddef.h>
> > +#include <linux/errno.h>
> > +#include <linux/aer.h>
> > +
> > +#define DRV_VERSION	"EXPERIMENTAL VERSION"
> 
> Is that a leftover? :)

Sorry, will fix this.

> > +#define DRV_NAME	"intel-fpga-pci"
> > +
> > +/* PCI Device ID */
> > +#define PCIe_DEVICE_ID_PF_INT_5_X	0xBCBD
> > +#define PCIe_DEVICE_ID_PF_INT_6_X	0xBCC0
> > +#define PCIe_DEVICE_ID_PF_DSC_1_X	0x09C4
> > +/* VF Device */
> > +#define PCIe_DEVICE_ID_VF_INT_5_X	0xBCBF
> > +#define PCIe_DEVICE_ID_VF_INT_6_X	0xBCC1
> > +#define PCIe_DEVICE_ID_VF_DSC_1_X	0x09C5
> > +
> > +static struct pci_device_id cci_pcie_id_tbl[] = {
> > +	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_PF_INT_5_X),},
> > +	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_VF_INT_5_X),},
> > +	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_PF_INT_6_X),},
> > +	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_VF_INT_6_X),},
> > +	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_PF_DSC_1_X),},
> > +	{PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCIe_DEVICE_ID_VF_DSC_1_X),},
> > +	{0,}
> > +};
> > +MODULE_DEVICE_TABLE(pci, cci_pcie_id_tbl);
> > +
> > +static
> > +int cci_pci_probe(struct pci_dev *pcidev, const struct pci_device_id *pcidevid)
> > +{
> > +	int ret;
> > +
> > +	ret = pci_enable_device(pcidev);
> > +	if (ret < 0) {
> > +		dev_err(&pcidev->dev, "Failed to enable device %d.\n", ret);
> > +		goto exit;
> Why not 'return ret' here ?

Yes, you are right, will fix this.

> > +	}
> > +
> > +	ret = pci_enable_pcie_error_reporting(pcidev);
> > +	if (ret && ret != -EINVAL)
> > +		dev_info(&pcidev->dev, "PCIE AER unavailable %d.\n", ret);
> 
> What if it is EINVAL?

pci_enable_pcie_error_reporting is always return -EINVAL when CONFIG_PCIEAER is not selected.
Then we don't need this boring message. : )

> 
> > +
> > +	ret = pci_request_regions(pcidev, DRV_NAME);
> > +	if (ret) {
> > +		dev_err(&pcidev->dev, "Failed to request regions.\n");
> > +		goto disable_error_report_exit;
> > +	}
> > +
> > +	pci_set_master(pcidev);
> > +	pci_save_state(pcidev);
> > +
> > +	if (!dma_set_mask(&pcidev->dev, DMA_BIT_MASK(64))) {
> > +		dma_set_coherent_mask(&pcidev->dev, DMA_BIT_MASK(64));
> > +	} else if (!dma_set_mask(&pcidev->dev, DMA_BIT_MASK(32))) {
> > +		dma_set_coherent_mask(&pcidev->dev, DMA_BIT_MASK(32));
> > +	} else {
> > +		ret = -EIO;
> > +		dev_err(&pcidev->dev, "No suitable DMA support available.\n");
> > +		goto release_region_exit;
> > +	}
> > +
> > +	/* TODO: create and add the platform device per feature list */
> > +	return 0;
> > +
> > +release_region_exit:
> > +	pci_release_regions(pcidev);
> > +disable_error_report_exit:
> > +	pci_disable_pcie_error_reporting(pcidev);
> > +	pci_disable_device(pcidev);
> > +exit:
> > +	return ret;
> If you return as suggested above, this can go away.

Yes, you are right. Will fix this in next version.
Thanks a lot for your review and comments. : )

Hao

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features.
  2017-04-04 22:09   ` Alan Tull
@ 2017-04-05 14:09     ` Wu Hao
  0 siblings, 0 replies; 93+ messages in thread
From: Wu Hao @ 2017-04-05 14:09 UTC (permalink / raw)
  To: Alan Tull
  Cc: Moritz Fischer, linux-fpga, linux-kernel, luwei.kang, yi.z.zhang,
	Xiao Guangrong, Tim Whisonant, Enno Luebbers, Shiva Rao,
	Christopher Rauer

On Tue, Apr 04, 2017 at 05:09:23PM -0500, Alan Tull wrote:
> On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote:
> > From: Xiao Guangrong <guangrong.xiao@linux.intel.com>
> >
> > Device Featuer List structure creates a link list of feature headers
> > within the MMIO space to provide an extensiable way of adding features.
> >
> > The Intel FPGA PCIe driver walks through the feature headers to enumerate
> > feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
> > Function Unit (AFU), and their private sub features. For feature devices,
> > it creates the platform devices and linked the private sub features into
> > their platform data.
> >
> 
> I'm still looking at this code and it's pretty new to me, but I think
> it would be desirable and not really hard to separate the code
> that enumerates from all the fixed feature code.  So pcie.c could see
> that there is a pci device out there that it knows has these memory
> mapped enumeration structs.  So it goes and reads the structs, parses
> them, and knows what drivers to probe.  The FME and AFU and other
> fpga device drivers could register their guids with the framework
> and be discoverable in that way.
> 
> That way if you need to implement a different FME or anything else, it
> could be added with a new guid to this framework and would get
> enumerated.  I'm thinking of the future and of more general usability
> of this code.
>
> Then the enumeration code wouldn't have to be 'intel' code or even
> code dedicated to FME's and AFU's.  Any FPGA with a PCIe
> port that has the right id's could have this struct and use this
> enumeration method.  Actually if the parse* enumeration code could be in a
> separate file as helper functions for the pcie code, this stuff would
> be structured for future support this of the same framework on
> embedded FPGA devices.
> 

Hi Alan

Thank you very much for the review and comments.

Actually I am not sure if the 'Device Feature List' is designed for common
usage or not, but per current implementation of Port/AFU and FME, they did
not use the exact same way for enumeration. e.g PCIe driver reads register
under Port to know the AFU MMIO region size. So for each module, it has
its own method to enumerate and prepare the resource for its platform
device. Other developers may not be able to use them for a new module.

>From the whole device's point of view, do enumeration for all modules is
still a device specific thing. e.g 'Device Feature List' of the FME is in
PCI BAR0, but the location of Port/AFU's 'Device Feature List' is not
linked to FME's Device Feature List, but given (PCI BAR +  offset) by a 
FME register. So the process of enumeration may be totally different
in another device with different module.

I don't expect FME can be used without PCIE or Port/AFU now, as it ties
to them so closely. e.g give PCI Bar info for port, registers to support
PCI SRIOV function. And so does PCIe and AFU driver, e.g PCIe driver is
not only handling the enumeration, but also manage ports and access FME
registers for SRIOV function support (sriov related code is not included
in this patch set).

If we have any PCIE FPGA has FME, then we can reuse this Intel FPGA
driver directly, but if no FME, then most code can't be reused.

I fully understand your point for code reuse and agree with you that
parse* enumeration code could be in a common place as helper function.
But for others, I still have concern due to current hardware
implementation. Anyway I will continue consideration on this.

Thanks
Hao


> Alan

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-04-05 11:40             ` Wu, Hao
@ 2017-04-05 15:26               ` Alan Tull
  2017-04-05 15:39                 ` Alan Tull
  0 siblings, 1 reply; 93+ messages in thread
From: Alan Tull @ 2017-04-05 15:26 UTC (permalink / raw)
  To: Wu, Hao
  Cc: Moritz Fischer, linux-fpga, linux-kernel, Kang, Luwei, Zhang,
	Yi Z, Whisonant, Tim, Luebbers, Enno, Rao, Shiva, Rauer,
	Christopher, Tull, Alan, Xiao Guangrong

On Wed, Apr 5, 2017 at 6:40 AM, Wu, Hao <hao.wu@intel.com> wrote:
>> >> The  fpga_image_info struct started life as just image specific info,
>> >> but I want it to go in the direction of including parameters needed to
>> >> program it this specific time. Otherwise we are stuck having to keep
>> >> adding parameters as our use of FPGA develops.  It probably could be
>> >> documented better as 'information needed to program a FPGA image'
>> >> rather than strictly 'information about this particular FPGA image'.
>> >> My patch "fpga-mgr: pass parameters for loading fpga in image info"
>> >> goes in this direction by having the buf, firmware name, or sg list
>> >> passed in the info for the added fpga_mgr_load() function.  Actually I
>> >> should probably simplify the API and get rid of fpga_mgr_buf_load,
>> >> fpga_mgr_buf_load_sg, and fpga_mgr_firmware_load and require people to
>> >> use fpga_mgr_load (passing all parameters in fpga_image_info).
>> >>
>> >
>> > Make sense.
>> >
>> >> > It may be a
>> >> > little confusing. One rough idea is that keep this info under fpga region
>> >> > (maybe its private data), and pass the fpga-region to fpga_mgr_buf_load,
>> >>
>> >> Yes, keep this info in fpga-region.  When the region wants to program
>> >> using fpga-mgr, add the region id to fpga_image_info.  I propose
>> >> calling it region_id.
>> >
>> > Hm.. Do we need a function which moves info from region to image info?
>>
>> No, just code that sets that variable in the struct before calling the
>> fpga_region_program_fpga function.
>>
>> >
>> > Another idea is, add a priv to fpga_image_info, and use a common function
>> > to pass the fpga_region's priv to fpga_image_info's priv before PR.
>> > fpga-mgr then knows fpga_region priv info from the fpga_image_info.
>> >
>>
>> Adding priv would make the interface for fpga-mgr non-uniform.  The point
>> of having a fpga-mgr framework is that there
>> is the potential of the upper layers working for different FPGA devices.
>> If the interface for each FPGA device were different, that would then
>> be broken.
>>
>
> I mean drivers can register their own fpga-mgr ops, and handle priv of
> fpga_image_info in driver specific way for pr (e.g write_init function).
> We don't need to change the any upper layer interfaces.

I'm trying to avoid driver specific ways of doing things.  Think of this
all as a set of blocks and we want to be able to switch out individual
blocks in the future.  It's future-proofing and also making code more
generally usable.

fpga_mgr_info is part of the interface for calls to fpga-mgr to do
reprogramming.  My patchset will push it further in that direction
as pointers to the image are added to fpga_mgr_info.

Adding 'priv' to fpga_mgr_info makes that interface specific to a this driver.
It's better to add a region_id variable to fpga_mgr_info that may not be used by
all fpga-mgr drivers.  The current model is that a fpga-mgr driver checks
fpga_mgr_info flags to see if its correct.  The fpga-mgr driver can check
any other needed fpga_mgr_info variables and return error if the params
look invalid.  And ignore any it doesn't need.

I don't think priv belongs in fpga_image_info.  priv tends to be information
for a specific instance of a driver that can have several instances.  priv
usually stores a driver's memory mappings, interrupts, etc for that instance.
It's called private info as it is info that other blocks don't need to know
and don't get to look at.  It's private.  So priv as in interface strikes me as
not private any more and is sort of a red flag.

Alan

>
> If you prefer the region_id for fpga_image_info, we can go with region_id
> for sure. : )
>
> Thanks
> Hao

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-04-05 15:26               ` Alan Tull
@ 2017-04-05 15:39                 ` Alan Tull
  2017-04-06 10:57                   ` Wu Hao
  0 siblings, 1 reply; 93+ messages in thread
From: Alan Tull @ 2017-04-05 15:39 UTC (permalink / raw)
  To: Wu, Hao
  Cc: Moritz Fischer, linux-fpga, linux-kernel, Kang, Luwei, Zhang,
	Yi Z, Whisonant, Tim, Luebbers, Enno, Rao, Shiva, Rauer,
	Christopher, Tull, Alan, Xiao Guangrong

On Wed, Apr 5, 2017 at 10:26 AM, Alan Tull <atull@kernel.org> wrote:
> On Wed, Apr 5, 2017 at 6:40 AM, Wu, Hao <hao.wu@intel.com> wrote:
>>> >> The  fpga_image_info struct started life as just image specific info,
>>> >> but I want it to go in the direction of including parameters needed to
>>> >> program it this specific time. Otherwise we are stuck having to keep
>>> >> adding parameters as our use of FPGA develops.  It probably could be
>>> >> documented better as 'information needed to program a FPGA image'
>>> >> rather than strictly 'information about this particular FPGA image'.
>>> >> My patch "fpga-mgr: pass parameters for loading fpga in image info"
>>> >> goes in this direction by having the buf, firmware name, or sg list
>>> >> passed in the info for the added fpga_mgr_load() function.  Actually I
>>> >> should probably simplify the API and get rid of fpga_mgr_buf_load,
>>> >> fpga_mgr_buf_load_sg, and fpga_mgr_firmware_load and require people to
>>> >> use fpga_mgr_load (passing all parameters in fpga_image_info).
>>> >>
>>> >
>>> > Make sense.
>>> >
>>> >> > It may be a
>>> >> > little confusing. One rough idea is that keep this info under fpga region
>>> >> > (maybe its private data), and pass the fpga-region to fpga_mgr_buf_load,
>>> >>
>>> >> Yes, keep this info in fpga-region.  When the region wants to program
>>> >> using fpga-mgr, add the region id to fpga_image_info.  I propose
>>> >> calling it region_id.
>>> >
>>> > Hm.. Do we need a function which moves info from region to image info?
>>>
>>> No, just code that sets that variable in the struct before calling the
>>> fpga_region_program_fpga function.
>>>
>>> >
>>> > Another idea is, add a priv to fpga_image_info, and use a common function
>>> > to pass the fpga_region's priv to fpga_image_info's priv before PR.
>>> > fpga-mgr then knows fpga_region priv info from the fpga_image_info.
>>> >
>>>
>>> Adding priv would make the interface for fpga-mgr non-uniform.  The point
>>> of having a fpga-mgr framework is that there
>>> is the potential of the upper layers working for different FPGA devices.
>>> If the interface for each FPGA device were different, that would then
>>> be broken.
>>>
>>
>> I mean drivers can register their own fpga-mgr ops, and handle priv of
>> fpga_image_info in driver specific way for pr (e.g write_init function).
>> We don't need to change the any upper layer interfaces.
>
> I'm trying to avoid driver specific ways of doing things.  Think of this
> all as a set of blocks and we want to be able to switch out individual
> blocks in the future.  It's future-proofing and also making code more
> generally usable.
>
> fpga_mgr_info is part of the interface for calls to fpga-mgr to do
> reprogramming.  My patchset will push it further in that direction
> as pointers to the image are added to fpga_mgr_info.
>
> Adding 'priv' to fpga_mgr_info makes that interface specific to a this driver.
> It's better to add a region_id variable to fpga_mgr_info that may not be used by
> all fpga-mgr drivers.  The current model is that a fpga-mgr driver checks
> fpga_mgr_info flags to see if its correct.  The fpga-mgr driver can check
> any other needed fpga_mgr_info variables and return error if the params
> look invalid.  And ignore any it doesn't need.
>
> I don't think priv belongs in fpga_image_info.  priv tends to be information
> for a specific instance of a driver that can have several instances.  priv
> usually stores a driver's memory mappings, interrupts, etc for that instance.
> It's called private info as it is info that other blocks don't need to know
> and don't get to look at.  It's private.  So priv as in interface strikes me as
> not private any more and is sort of a red flag.
>
> Alan

Besides that, I see that region_id is needed both for the fme_pr and
for determining which port to disable/enable.  So adding it to the
fpga_image_info would allow the fpga-region to figure out which
bridge to control as well as pass it to the mgr.

>
>>
>> If you prefer the region_id for fpga_image_info, we can go with region_id
>> for sure. : )
>>
>> Thanks
>> Hao

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-04-05 15:39                 ` Alan Tull
@ 2017-04-06 10:57                   ` Wu Hao
  2017-04-06 19:27                     ` Alan Tull
  0 siblings, 1 reply; 93+ messages in thread
From: Wu Hao @ 2017-04-06 10:57 UTC (permalink / raw)
  To: Alan Tull
  Cc: Moritz Fischer, linux-fpga, linux-kernel, Kang, Luwei, Zhang,
	Yi Z, Whisonant, Tim, Luebbers, Enno, Rao, Shiva, Rauer,
	Christopher, Tull, Alan, Xiao Guangrong

On Wed, Apr 05, 2017 at 10:39:05AM -0500, Alan Tull wrote:
> On Wed, Apr 5, 2017 at 10:26 AM, Alan Tull <atull@kernel.org> wrote:
> > On Wed, Apr 5, 2017 at 6:40 AM, Wu, Hao <hao.wu@intel.com> wrote:
> >>> >> The  fpga_image_info struct started life as just image specific info,
> >>> >> but I want it to go in the direction of including parameters needed to
> >>> >> program it this specific time. Otherwise we are stuck having to keep
> >>> >> adding parameters as our use of FPGA develops.  It probably could be
> >>> >> documented better as 'information needed to program a FPGA image'
> >>> >> rather than strictly 'information about this particular FPGA image'.
> >>> >> My patch "fpga-mgr: pass parameters for loading fpga in image info"
> >>> >> goes in this direction by having the buf, firmware name, or sg list
> >>> >> passed in the info for the added fpga_mgr_load() function.  Actually I
> >>> >> should probably simplify the API and get rid of fpga_mgr_buf_load,
> >>> >> fpga_mgr_buf_load_sg, and fpga_mgr_firmware_load and require people to
> >>> >> use fpga_mgr_load (passing all parameters in fpga_image_info).
> >>> >>
> >>> >
> >>> > Make sense.
> >>> >
> >>> >> > It may be a
> >>> >> > little confusing. One rough idea is that keep this info under fpga region
> >>> >> > (maybe its private data), and pass the fpga-region to fpga_mgr_buf_load,
> >>> >>
> >>> >> Yes, keep this info in fpga-region.  When the region wants to program
> >>> >> using fpga-mgr, add the region id to fpga_image_info.  I propose
> >>> >> calling it region_id.
> >>> >
> >>> > Hm.. Do we need a function which moves info from region to image info?
> >>>
> >>> No, just code that sets that variable in the struct before calling the
> >>> fpga_region_program_fpga function.
> >>>
> >>> >
> >>> > Another idea is, add a priv to fpga_image_info, and use a common function
> >>> > to pass the fpga_region's priv to fpga_image_info's priv before PR.
> >>> > fpga-mgr then knows fpga_region priv info from the fpga_image_info.
> >>> >
> >>>
> >>> Adding priv would make the interface for fpga-mgr non-uniform.  The point
> >>> of having a fpga-mgr framework is that there
> >>> is the potential of the upper layers working for different FPGA devices.
> >>> If the interface for each FPGA device were different, that would then
> >>> be broken.
> >>>
> >>
> >> I mean drivers can register their own fpga-mgr ops, and handle priv of
> >> fpga_image_info in driver specific way for pr (e.g write_init function).
> >> We don't need to change the any upper layer interfaces.
> >
> > I'm trying to avoid driver specific ways of doing things.  Think of this
> > all as a set of blocks and we want to be able to switch out individual
> > blocks in the future.  It's future-proofing and also making code more
> > generally usable.
> >
> > fpga_mgr_info is part of the interface for calls to fpga-mgr to do
> > reprogramming.  My patchset will push it further in that direction
> > as pointers to the image are added to fpga_mgr_info.
> >
> > Adding 'priv' to fpga_mgr_info makes that interface specific to a this driver.
> > It's better to add a region_id variable to fpga_mgr_info that may not be used by
> > all fpga-mgr drivers.  The current model is that a fpga-mgr driver checks
> > fpga_mgr_info flags to see if its correct.  The fpga-mgr driver can check
> > any other needed fpga_mgr_info variables and return error if the params
> > look invalid.  And ignore any it doesn't need.
> >
> > I don't think priv belongs in fpga_image_info.  priv tends to be information
> > for a specific instance of a driver that can have several instances.  priv
> > usually stores a driver's memory mappings, interrupts, etc for that instance.
> > It's called private info as it is info that other blocks don't need to know
> > and don't get to look at.  It's private.  So priv as in interface strikes me as
> > not private any more and is sort of a red flag.

Thanks a lot for the details. It's more clear to me now. :)

> >
> > Alan
> 
> Besides that, I see that region_id is needed both for the fme_pr and
> for determining which port to disable/enable.  So adding it to the
> fpga_image_info would allow the fpga-region to figure out which
> bridge to control as well as pass it to the mgr.
> 

Do we need to add an 'id' to fpga-bridge too?

Per my understanding this FME driver should create fpga-region for each
accelerator, and make sure each fpga-region has correct region_id. When
this fpga-region wants to be programmed by fpga-mgr, then it add this
info to fpga_image_info, and fpga-mgr knows which region to program via
this fpga_image_info.

For each fpga-region, FME driver needs to create one fpga-bridge and link
it to region's bridge_list. When fpga_region_program_fpga is invoked for
PR. This fpga-bridge could be disabled before PR and re-enabled after PR
automatically. If fpga-bridge contains this 'id' information, then driver
knows which port to enable/disable to implement of the enable_set function
under this fpga-bridge.

Thanks
Hao

> >
> >>
> >> If you prefer the region_id for fpga_image_info, we can go with region_id
> >> for sure. : )
> >>
> >> Thanks
> >> Hao

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-04-06 10:57                   ` Wu Hao
@ 2017-04-06 19:27                     ` Alan Tull
  2017-04-07  5:56                       ` Wu Hao
  0 siblings, 1 reply; 93+ messages in thread
From: Alan Tull @ 2017-04-06 19:27 UTC (permalink / raw)
  To: Wu Hao
  Cc: Moritz Fischer, linux-fpga, linux-kernel, Kang, Luwei, Zhang,
	Yi Z, Whisonant, Tim, Luebbers, Enno, Rao, Shiva, Rauer,
	Christopher, Tull, Alan, Xiao Guangrong

On Thu, Apr 6, 2017 at 5:57 AM, Wu Hao <hao.wu@intel.com> wrote:
> On Wed, Apr 05, 2017 at 10:39:05AM -0500, Alan Tull wrote:
>> On Wed, Apr 5, 2017 at 10:26 AM, Alan Tull <atull@kernel.org> wrote:
>> > On Wed, Apr 5, 2017 at 6:40 AM, Wu, Hao <hao.wu@intel.com> wrote:
>> >>> >> The  fpga_image_info struct started life as just image specific info,
>> >>> >> but I want it to go in the direction of including parameters needed to
>> >>> >> program it this specific time. Otherwise we are stuck having to keep
>> >>> >> adding parameters as our use of FPGA develops.  It probably could be
>> >>> >> documented better as 'information needed to program a FPGA image'
>> >>> >> rather than strictly 'information about this particular FPGA image'.
>> >>> >> My patch "fpga-mgr: pass parameters for loading fpga in image info"
>> >>> >> goes in this direction by having the buf, firmware name, or sg list
>> >>> >> passed in the info for the added fpga_mgr_load() function.  Actually I
>> >>> >> should probably simplify the API and get rid of fpga_mgr_buf_load,
>> >>> >> fpga_mgr_buf_load_sg, and fpga_mgr_firmware_load and require people to
>> >>> >> use fpga_mgr_load (passing all parameters in fpga_image_info).
>> >>> >>
>> >>> >
>> >>> > Make sense.
>> >>> >
>> >>> >> > It may be a
>> >>> >> > little confusing. One rough idea is that keep this info under fpga region
>> >>> >> > (maybe its private data), and pass the fpga-region to fpga_mgr_buf_load,
>> >>> >>
>> >>> >> Yes, keep this info in fpga-region.  When the region wants to program
>> >>> >> using fpga-mgr, add the region id to fpga_image_info.  I propose
>> >>> >> calling it region_id.
>> >>> >
>> >>> > Hm.. Do we need a function which moves info from region to image info?
>> >>>
>> >>> No, just code that sets that variable in the struct before calling the
>> >>> fpga_region_program_fpga function.
>> >>>
>> >>> >
>> >>> > Another idea is, add a priv to fpga_image_info, and use a common function
>> >>> > to pass the fpga_region's priv to fpga_image_info's priv before PR.
>> >>> > fpga-mgr then knows fpga_region priv info from the fpga_image_info.
>> >>> >
>> >>>
>> >>> Adding priv would make the interface for fpga-mgr non-uniform.  The point
>> >>> of having a fpga-mgr framework is that there
>> >>> is the potential of the upper layers working for different FPGA devices.
>> >>> If the interface for each FPGA device were different, that would then
>> >>> be broken.
>> >>>
>> >>
>> >> I mean drivers can register their own fpga-mgr ops, and handle priv of
>> >> fpga_image_info in driver specific way for pr (e.g write_init function).
>> >> We don't need to change the any upper layer interfaces.
>> >
>> > I'm trying to avoid driver specific ways of doing things.  Think of this
>> > all as a set of blocks and we want to be able to switch out individual
>> > blocks in the future.  It's future-proofing and also making code more
>> > generally usable.
>> >
>> > fpga_mgr_info is part of the interface for calls to fpga-mgr to do
>> > reprogramming.  My patchset will push it further in that direction
>> > as pointers to the image are added to fpga_mgr_info.
>> >
>> > Adding 'priv' to fpga_mgr_info makes that interface specific to a this driver.
>> > It's better to add a region_id variable to fpga_mgr_info that may not be used by
>> > all fpga-mgr drivers.  The current model is that a fpga-mgr driver checks
>> > fpga_mgr_info flags to see if its correct.  The fpga-mgr driver can check
>> > any other needed fpga_mgr_info variables and return error if the params
>> > look invalid.  And ignore any it doesn't need.
>> >
>> > I don't think priv belongs in fpga_image_info.  priv tends to be information
>> > for a specific instance of a driver that can have several instances.  priv
>> > usually stores a driver's memory mappings, interrupts, etc for that instance.
>> > It's called private info as it is info that other blocks don't need to know
>> > and don't get to look at.  It's private.  So priv as in interface strikes me as
>> > not private any more and is sort of a red flag.
>
> Thanks a lot for the details. It's more clear to me now. :)
>
>> >
>> > Alan
>>
>> Besides that, I see that region_id is needed both for the fme_pr and
>> for determining which port to disable/enable.  So adding it to the
>> fpga_image_info would allow the fpga-region to figure out which
>> bridge to control as well as pass it to the mgr.
>>
>
> Do we need to add an 'id' to fpga-bridge too?

You don't have to, but you could.  See below.

>
> Per my understanding this FME driver should create fpga-region for each
> accelerator, and make sure each fpga-region has correct region_id. When
> this fpga-region wants to be programmed by fpga-mgr, then it add this
> info to fpga_image_info, and fpga-mgr knows which region to program via
> this fpga_image_info.

The fpga_image_info is passed both to fpga-mgr and fpga-bridge.  So
both will get the region_id through the info.

>
> For each fpga-region, FME driver needs to create one fpga-bridge and link
> it to region's bridge_list. When fpga_region_program_fpga is invoked for
> PR. This fpga-bridge could be disabled before PR and re-enabled after PR
> automatically. If fpga-bridge contains this 'id' information, then driver
> knows which port to enable/disable to implement of the enable_set function
> under this fpga-bridge.

Could implement it either way: the bridge could either use the
bridge_id that has been passed to it in the info or code that creates
the bridge could create private data and pass it to
fpga_bridge_register().  But the bridge will have the id because it
will be passed in the info, so it's not really needed.

Alan

>
> Thanks
> Hao
>
>> >
>> >>
>> >> If you prefer the region_id for fpga_image_info, we can go with region_id
>> >> for sure. : )
>> >>
>> >> Thanks
>> >> Hao

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 00/16] Intel FPGA Device Drivers
  2017-03-30 12:08 [PATCH 00/16] Intel FPGA Device Drivers Wu Hao
                   ` (16 preceding siblings ...)
  2017-03-30 17:17 ` [PATCH 00/16] Intel FPGA Device Drivers Moritz Fischer
@ 2017-04-06 20:27 ` Jerome Glisse
  2017-04-11 19:38   ` Luebbers, Enno
  17 siblings, 1 reply; 93+ messages in thread
From: Jerome Glisse @ 2017-04-06 20:27 UTC (permalink / raw)
  To: Wu Hao
  Cc: atull, moritz.fischer, linux-fpga, linux-kernel, luwei.kang, yi.z.zhang

On Thu, Mar 30, 2017 at 08:08:00PM +0800, Wu Hao wrote:
> Hi All,
> 
> Here is a patch-series adding drivers for Intel FPGA devices.
> 
> The Intel FPGA driver provides interfaces for userspace applications to
> configure, enumerate, open, and access FPGA accelerators on platforms
> equipped with Intel(R) FPGA solutions and enables system level management
> functions such as FPGA partial reconfiguration, power management and
> virtualization.
> 
> This patch series only adds the basic functions for FPGA accelerators and
> partial reconfiguration. Patches for more functions, e.g power management
> and virtualization, will be submitted after this series gets reviewed.
> 
> Patch 1: add a document for Intel FPGA driver overview, including the HW
> architecture, driver organization, device enumeration, virtualization and
> opens.
> 
> Patch 2: introduce a fpga-dev class. It's used in below Intel FPGA PCIe
> device driver, to represent a FPGA device on the system, and all actual
> feature devices should be registered as child nodes of this container
> fpga-dev device.
> 
> Patch 3-7: implement Intel FPGA PCIe device driver. It walks through the
> 'Device Feature List' in the PCI Bar, creates the container fpga-dev as
> parent and platform devices as children for the feature devices it found.
> 
> Patch 8-11: implement Intel FPGA Management Engine (FME) driver. It's a
> platform driver matching with the FME platform device created by above
> PCIe driver. Sysfs and device file ioctls are exposed as user interfaces
> to allow partial reconfiguration to Accelerated Function Units (AFUs) from
> user space applications.

Do we have an open source toolchain to generate the FPGA configuration
(bitstream) ? As it is required for the GPU sub-system that any driver
API must comes with open source userspace.

Or are FPGA given a free pass ?


Cheers,
Jérôme

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 11/16] fpga: intel: fme: add partial reconfiguration sub feature support
  2017-04-06 19:27                     ` Alan Tull
@ 2017-04-07  5:56                       ` Wu Hao
  0 siblings, 0 replies; 93+ messages in thread
From: Wu Hao @ 2017-04-07  5:56 UTC (permalink / raw)
  To: Alan Tull
  Cc: Moritz Fischer, linux-fpga, linux-kernel, Kang, Luwei, Zhang,
	Yi Z, Whisonant, Tim, Luebbers, Enno, Rao, Shiva, Rauer,
	Christopher, Tull, Alan, Xiao Guangrong

On Thu, Apr 06, 2017 at 02:27:42PM -0500, Alan Tull wrote:
> On Thu, Apr 6, 2017 at 5:57 AM, Wu Hao <hao.wu@intel.com> wrote:
> > On Wed, Apr 05, 2017 at 10:39:05AM -0500, Alan Tull wrote:
> >> On Wed, Apr 5, 2017 at 10:26 AM, Alan Tull <atull@kernel.org> wrote:
> >> > On Wed, Apr 5, 2017 at 6:40 AM, Wu, Hao <hao.wu@intel.com> wrote:
> >> >>> >> The  fpga_image_info struct started life as just image specific info,
> >> >>> >> but I want it to go in the direction of including parameters needed to
> >> >>> >> program it this specific time. Otherwise we are stuck having to keep
> >> >>> >> adding parameters as our use of FPGA develops.  It probably could be
> >> >>> >> documented better as 'information needed to program a FPGA image'
> >> >>> >> rather than strictly 'information about this particular FPGA image'.
> >> >>> >> My patch "fpga-mgr: pass parameters for loading fpga in image info"
> >> >>> >> goes in this direction by having the buf, firmware name, or sg list
> >> >>> >> passed in the info for the added fpga_mgr_load() function.  Actually I
> >> >>> >> should probably simplify the API and get rid of fpga_mgr_buf_load,
> >> >>> >> fpga_mgr_buf_load_sg, and fpga_mgr_firmware_load and require people to
> >> >>> >> use fpga_mgr_load (passing all parameters in fpga_image_info).
> >> >>> >>
> >> >>> >
> >> >>> > Make sense.
> >> >>> >
> >> >>> >> > It may be a
> >> >>> >> > little confusing. One rough idea is that keep this info under fpga region
> >> >>> >> > (maybe its private data), and pass the fpga-region to fpga_mgr_buf_load,
> >> >>> >>
> >> >>> >> Yes, keep this info in fpga-region.  When the region wants to program
> >> >>> >> using fpga-mgr, add the region id to fpga_image_info.  I propose
> >> >>> >> calling it region_id.
> >> >>> >
> >> >>> > Hm.. Do we need a function which moves info from region to image info?
> >> >>>
> >> >>> No, just code that sets that variable in the struct before calling the
> >> >>> fpga_region_program_fpga function.
> >> >>>
> >> >>> >
> >> >>> > Another idea is, add a priv to fpga_image_info, and use a common function
> >> >>> > to pass the fpga_region's priv to fpga_image_info's priv before PR.
> >> >>> > fpga-mgr then knows fpga_region priv info from the fpga_image_info.
> >> >>> >
> >> >>>
> >> >>> Adding priv would make the interface for fpga-mgr non-uniform.  The point
> >> >>> of having a fpga-mgr framework is that there
> >> >>> is the potential of the upper layers working for different FPGA devices.
> >> >>> If the interface for each FPGA device were different, that would then
> >> >>> be broken.
> >> >>>
> >> >>
> >> >> I mean drivers can register their own fpga-mgr ops, and handle priv of
> >> >> fpga_image_info in driver specific way for pr (e.g write_init function).
> >> >> We don't need to change the any upper layer interfaces.
> >> >
> >> > I'm trying to avoid driver specific ways of doing things.  Think of this
> >> > all as a set of blocks and we want to be able to switch out individual
> >> > blocks in the future.  It's future-proofing and also making code more
> >> > generally usable.
> >> >
> >> > fpga_mgr_info is part of the interface for calls to fpga-mgr to do
> >> > reprogramming.  My patchset will push it further in that direction
> >> > as pointers to the image are added to fpga_mgr_info.
> >> >
> >> > Adding 'priv' to fpga_mgr_info makes that interface specific to a this driver.
> >> > It's better to add a region_id variable to fpga_mgr_info that may not be used by
> >> > all fpga-mgr drivers.  The current model is that a fpga-mgr driver checks
> >> > fpga_mgr_info flags to see if its correct.  The fpga-mgr driver can check
> >> > any other needed fpga_mgr_info variables and return error if the params
> >> > look invalid.  And ignore any it doesn't need.
> >> >
> >> > I don't think priv belongs in fpga_image_info.  priv tends to be information
> >> > for a specific instance of a driver that can have several instances.  priv
> >> > usually stores a driver's memory mappings, interrupts, etc for that instance.
> >> > It's called private info as it is info that other blocks don't need to know
> >> > and don't get to look at.  It's private.  So priv as in interface strikes me as
> >> > not private any more and is sort of a red flag.
> >
> > Thanks a lot for the details. It's more clear to me now. :)
> >
> >> >
> >> > Alan
> >>
> >> Besides that, I see that region_id is needed both for the fme_pr and
> >> for determining which port to disable/enable.  So adding it to the
> >> fpga_image_info would allow the fpga-region to figure out which
> >> bridge to control as well as pass it to the mgr.
> >>
> >
> > Do we need to add an 'id' to fpga-bridge too?
> 
> You don't have to, but you could.  See below.
> 
> >
> > Per my understanding this FME driver should create fpga-region for each
> > accelerator, and make sure each fpga-region has correct region_id. When
> > this fpga-region wants to be programmed by fpga-mgr, then it add this
> > info to fpga_image_info, and fpga-mgr knows which region to program via
> > this fpga_image_info.
> 
> The fpga_image_info is passed both to fpga-mgr and fpga-bridge.  So
> both will get the region_id through the info.
> 
> >
> > For each fpga-region, FME driver needs to create one fpga-bridge and link
> > it to region's bridge_list. When fpga_region_program_fpga is invoked for
> > PR. This fpga-bridge could be disabled before PR and re-enabled after PR
> > automatically. If fpga-bridge contains this 'id' information, then driver
> > knows which port to enable/disable to implement of the enable_set function
> > under this fpga-bridge.
> 
> Could implement it either way: the bridge could either use the
> bridge_id that has been passed to it in the info or code that creates
> the bridge could create private data and pass it to
> fpga_bridge_register().  But the bridge will have the id because it
> will be passed in the info, so it's not really needed.

Thanks for the suggestion. :)

The reason I was considering adding id to fpga_bridge is, in the case
driver creates all bridges (and regions/manager) in the initialization
code, but at that time, nobody knows the actual fpga_image_info.

In order to pass fpga_image_info to the bridges of give region during PR
(e.g in fpga_region_program_fpga function), then driver must implement
region->get_bridges function, but I feel we may not need get_bridges
function for this case, as all bridges have already been linked to
fpga-region during initialization.

So I prefer the second method that keep this id in the private data. :)

Thanks
Hao

> 
> Alan
> 
> >
> > Thanks
> > Hao
> >
> >> >
> >> >>
> >> >> If you prefer the region_id for fpga_image_info, we can go with region_id
> >> >> for sure. : )
> >> >>
> >> >> Thanks
> >> >> Hao
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview
  2017-04-02 14:41         ` Moritz Fischer
  2017-04-03 20:44           ` Alan Tull
  2017-04-04  5:06           ` Wu Hao
@ 2017-04-11 18:02           ` Alan Tull
  2017-04-12  3:22             ` Wu, Hao
  2 siblings, 1 reply; 93+ messages in thread
From: Alan Tull @ 2017-04-11 18:02 UTC (permalink / raw)
  To: Moritz Fischer
  Cc: Wu Hao, matthew.gerlach, Moritz Fischer, linux-fpga,
	linux-kernel, Kang, Luwei, Zhang, Yi Z, Enno Luebbers,
	Xiao Guangrong

On Sun, Apr 2, 2017 at 9:41 AM, Moritz Fischer <mdf@kernel.org> wrote:
> On Sat, Apr 01, 2017 at 07:16:19PM +0800, Wu Hao wrote:
>> On Fri, Mar 31, 2017 at 01:38:06PM -0500, Alan Tull wrote:
>> > On Fri, Mar 31, 2017 at 1:24 PM,  <matthew.gerlach@linux.intel.com> wrote:
>> > >
>> > >
>> > > On Thu, 30 Mar 2017, Wu Hao wrote:
>> > >
>> > >
>> > > Hi Wu Hao,
>> > >
>> > > Great documentation. I'm looking forward to diving into the rest of the
>> > > patches. Please see my comments inline.
>> > >
>> > > Matthew Gerlach
>> > >
>> > >
>> > >> Add a document for Intel FPGA driver overview.
>> > >>
>> > >> Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
>> > >> Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> > >> Signed-off-by: Wu Hao <hao.wu@intel.com>
>> > >> ---
>> > >> Documentation/fpga/intel-fpga.txt | 259
>> > >> ++++++++++++++++++++++++++++++++++++++
>> > >> 1 file changed, 259 insertions(+)
>> > >> create mode 100644 Documentation/fpga/intel-fpga.txt
>> > >>
>> > >> diff --git a/Documentation/fpga/intel-fpga.txt
>> > >> b/Documentation/fpga/intel-fpga.txt
>> > >> new file mode 100644
>> > >> index 0000000..9396cea
>> > >> --- /dev/null
>> > >> +++ b/Documentation/fpga/intel-fpga.txt
>> > >> @@ -0,0 +1,259 @@
>> > >>
>> > >> +===============================================================================
>> > >> +                    Intel FPGA driver Overview
>> > >>
>> > >> +-------------------------------------------------------------------------------
>> > >> +                Enno Luebbers <enno.luebbers@intel.com>
>> > >> +                Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> > >> +                Wu Hao <hao.wu@intel.com>
>> > >> +
>> > >> +The Intel FPGA driver provides interfaces for userspace applications to
>> > >> +configure, enumerate, open, and access FPGA accelerators on platforms
>> > >> equipped
>> > >> +with Intel(R) FPGA solutions and enables system level management
>> > >> functions such
>> > >> +as FPGA reconfiguration, power management, and virtualization.
>> > >> +
>> > >
>> > >
>> > > From a Linux kernel perspective, I'm not sure this is the best name for
>> > > this code.  The name gives me the impression that it is a driver for all
>> > > Intel FPGAs, but not all Intel FPGAs are connected to the processor over a
>> > > PCIe bus.  The processor could be directely connected like the Arria10
>> > > SOCFPGA.  Such a processor could certainly benefit from this accelerator
>> > > usage model.  In an extreme case, couldn't a processor in the FPGA,
>> > > running Linux, also benefit from this accelerator model?  Is this code a
>> > > "FPGA Accelerator Framework"?
>> > >
>> > >> +HW Architecture
>> > >> +===============
>> > >> +From the OS's point of view, the FPGA hardware appears as a regular PCIe
>> > >> device.
>> > >> +The FPGA device memory is organized using a predefined data structure
>> > >> (Device
>> > >> +Feature List). Features supported by the particular FPGA device are
>> > >> exposed
>> > >> +through these data structures, as illustrated below:
>> > >> +
>> > >> +  +-------------------------------+  +-------------+
>> > >> +  |              PF               |  |     VF      |
>> > >> +  +-------------------------------+  +-------------+
>> > >> +      ^            ^         ^              ^
>> > >> +      |            |         |              |
>> > >> ++-----|------------|---------|--------------|-------+
>> > >> +|     |            |         |              |       |
>> > >> +|  +-----+     +-------+ +-------+      +-------+   |
>> > >> +|  | FME |     | Port0 | | Port1 |      | Port2 |   |
>> > >> +|  +-----+     +-------+ +-------+      +-------+   |
>> > >> +|                  ^         ^              ^       |
>> > >> +|                  |         |              |       |
>> > >> +|              +-------+ +------+       +-------+   |
>> > >> +|              |  AFU  | |  AFU |       |  AFU  |   |
>> > >> +|              +-------+ +------+       +-------+   |
>> > >> +|                                                   |
>> > >> +|                 FPGA PCIe Device                  |
>> > >> ++---------------------------------------------------+
>> > >> +
>> > >> +The driver supports PCIe SR-IOV to create virtual functions (VFs) which
>> > >> can be
>> > >> +used to assign individual accelerators to virtual machines .
>> > >
>> > >
>> > > Does this HW Architecture require an Intel FPGA?  Couldn't any vendors FPGA
>> > > be used as long as it presented itself the PCIe bus the same and contained
>> > > an appropriate Device Feature List?
>
> I think this is a good (and important) point. Especially when sysfs
> entries & ioctls constituting ABI depend on it.

Please cc linux-api@vger.kernel.org on your next version of this as
linux/Documentation/process/adding-syscalls.rst specifies for new system calls.

Alan

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 00/16] Intel FPGA Device Drivers
  2017-04-06 20:27 ` Jerome Glisse
@ 2017-04-11 19:38   ` Luebbers, Enno
  2017-04-12 13:29     ` Jerome Glisse
  0 siblings, 1 reply; 93+ messages in thread
From: Luebbers, Enno @ 2017-04-11 19:38 UTC (permalink / raw)
  To: Jerome Glisse
  Cc: Wu Hao, atull, moritz.fischer, linux-fpga, linux-kernel,
	luwei.kang, yi.z.zhang

Hi Jerome,

On Thu, Apr 06, 2017 at 04:27:00PM -0400, Jerome Glisse wrote:

> Do we have an open source toolchain to generate the FPGA configuration
> (bitstream) ? As it is required for the GPU sub-system that any driver
> API must comes with open source userspace.

As far as I know, no FPGA vendor currently provides an open-source version of
their FPGA synthesis tools - there are, however, free (as in beer) versions
available for download that can be used for generating FPGA bitstreams. Also,
there are a number of projects to replace parts of the vendor tools with open
alternatives (yosys comes to mind, which I believe recently added initial
support for synthesizing logic for Intel FPGAs).

As an aside, we are also working on an open-source user-space library that would
allow you to use this driver to load existing accelerator bitstreams as well as
enumerate and access accelerators present in the system. This would enable
workflows where users have access to e.g. a library of FPGA accelerator
bitstreams and want to write applications that take advantage of these
accelerators, even without having access to an FPGA synthesis tool.

Thanks
- Enno

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features.
  2017-04-05 11:58     ` Wu Hao
@ 2017-04-11 20:21       ` Alan Tull
  2017-04-13  4:12         ` Wu, Hao
  0 siblings, 1 reply; 93+ messages in thread
From: Alan Tull @ 2017-04-11 20:21 UTC (permalink / raw)
  To: Wu Hao
  Cc: Moritz Fischer, linux-fpga, linux-kernel, Kang, Luwei, Zhang,
	Yi Z, Xiao Guangrong, Tim Whisonant, Enno Luebbers, Shiva Rao,
	Christopher Rauer

On Wed, Apr 5, 2017 at 6:58 AM, Wu Hao <hao.wu@intel.com> wrote:
> On Mon, Apr 03, 2017 at 04:44:15PM -0500, Alan Tull wrote:
>> On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote:
>> > From: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> >
>> > Device Featuer List structure creates a link list of feature headers
>> > within the MMIO space to provide an extensiable way of adding features.
>> >
>> > The Intel FPGA PCIe driver walks through the feature headers to enumerate
>> > feature devices, FPGA Management Engine (FME) and FPGA Port for Accelerated
>> > Function Unit (AFU), and their private sub features. For feature devices,
>> > it creates the platform devices and linked the private sub features into
>> > their platform data.
>> >
>> > Signed-off-by: Tim Whisonant <tim.whisonant@intel.com>
>> > Signed-off-by: Enno Luebbers <enno.luebbers@intel.com>
>> > Signed-off-by: Shiva Rao <shiva.rao@intel.com>
>> > Signed-off-by: Christopher Rauer <christopher.rauer@intel.com>
>> > Signed-off-by: Kang Luwei <luwei.kang@intel.com>
>> > Signed-off-by: Zhang Yi <yi.z.zhang@intel.com>
>> > Signed-off-by: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> > Signed-off-by: Wu Hao <hao.wu@intel.com>
>> > ---
>> >  drivers/fpga/intel/Makefile      |   2 +-
>> >  drivers/fpga/intel/feature-dev.c | 139 +++++++
>> >  drivers/fpga/intel/feature-dev.h | 342 ++++++++++++++++
>> >  drivers/fpga/intel/pcie.c        | 841 ++++++++++++++++++++++++++++++++++++++-
>> >  4 files changed, 1321 insertions(+), 3 deletions(-)
>> >  create mode 100644 drivers/fpga/intel/feature-dev.c
>> >  create mode 100644 drivers/fpga/intel/feature-dev.h
>> >
>> > diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
>> > index 61fd8ea..c029940 100644
>> > --- a/drivers/fpga/intel/Makefile
>> > +++ b/drivers/fpga/intel/Makefile
>> > @@ -1,3 +1,3 @@
>> >  obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
>> >
>> > -intel-fpga-pci-objs := pcie.o
>> > +intel-fpga-pci-objs := pcie.o feature-dev.o
>> > diff --git a/drivers/fpga/intel/feature-dev.c b/drivers/fpga/intel/feature-dev.c
>> > new file mode 100644
>> > index 0000000..6952566
>> > --- /dev/null
>> > +++ b/drivers/fpga/intel/feature-dev.c
>> > @@ -0,0 +1,139 @@
>> > +/*
>> > + * Intel FPGA Feature Device Driver
>> > + *
>> > + * Copyright (C) 2017 Intel Corporation, Inc.
>> > + *
>> > + * Authors:
>> > + *   Kang Luwei <luwei.kang@intel.com>
>> > + *   Zhang Yi <yi.z.zhang@intel.com>
>> > + *   Wu Hao <hao.wu@intel.com>
>> > + *   Xiao Guangrong <guangrong.xiao@linux.intel.com>
>> > + *
>> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
>> > + * redistributing this file, you may do so under either license. See the
>> > + * LICENSE.BSD file under this directory for the BSD license and see
>> > + * the COPYING file in the top-level directory for the GPLv2 license.
>> > + */
>> > +
>> > +#include "feature-dev.h"
>> > +
>> > +void feature_platform_data_add(struct feature_platform_data *pdata,
>> > +                              int index, const char *name,
>> > +                              int resource_index, void __iomem *ioaddr)
>> > +{
>> > +       WARN_ON(index >= pdata->num);
>> > +
>> > +       pdata->features[index].name = name;
>> > +       pdata->features[index].resource_index = resource_index;
>> > +       pdata->features[index].ioaddr = ioaddr;
>> > +}
>> > +
>> > +int feature_platform_data_size(int num)
>> > +{
>> > +       return sizeof(struct feature_platform_data) +
>> > +               num * sizeof(struct feature);
>> > +}
>> > +
>> > +struct feature_platform_data *
>> > +feature_platform_data_alloc_and_init(struct platform_device *dev, int num)
>> > +{
>> > +       struct feature_platform_data *pdata;
>> > +
>> > +       pdata = kzalloc(feature_platform_data_size(num), GFP_KERNEL);
>> > +       if (pdata) {
>> > +               pdata->dev = dev;
>> > +               pdata->num = num;
>> > +               mutex_init(&pdata->lock);
>> > +       }
>> > +
>> > +       return pdata;
>> > +}
>> > +
>> > +int fme_feature_num(void)
>> > +{
>> > +       return FME_FEATURE_ID_MAX;
>> > +}
>> > +
>> > +int port_feature_num(void)
>> > +{
>> > +       return PORT_FEATURE_ID_MAX;
>> > +}
>> > +
>> > +int fpga_port_id(struct platform_device *pdev)
>> > +{
>> > +       struct feature_port_header *port_hdr;
>> > +       struct feature_port_capability capability;
>> > +
>> > +       port_hdr = get_feature_ioaddr_by_index(&pdev->dev,
>> > +                                              PORT_FEATURE_ID_HEADER);
>> > +       WARN_ON(!port_hdr);
>> > +
>> > +       capability.csr = readq(&port_hdr->capability);
>> > +       return capability.port_number;
>> > +}
>> > +EXPORT_SYMBOL_GPL(fpga_port_id);
>> > +
>> > +/*
>> > + * Enable Port by clear the port soft reset bit, which is set by default.
>> > + * The User AFU is unable to respond to any MMIO access while in reset.
>> > + * __fpga_port_enable function should only be used after __fpga_port_disable
>> > + * function.
>> > + */
>> > +void __fpga_port_enable(struct platform_device *pdev)
>> > +{
>>
>> feature-dev.c is handling enumeration and adding port
>> enable/disable/etc functions for a specific port device.  I see the
>> port as a fpga-bridge.  The enumeration code should be separate from
>> the bridge code.  Especially separate from a very specific bridge low
>> level device driver implementation, otherwise this becomes obsolete as
>> soon as you have another port device with a different register
>> implementation.  Even if you handle that, then this enumeration code
>> isn't useable by other people who are using fpga-bridge.  The
>> fpga-bridge framework exists to separate low level things like how to
>> enable/disable a specific bridge device from upper level code that
>> knows when to enable/disable it (fpga-region).
>
> Hi Alan
>
> The major purpose of feature-dev is to create infrastructure for feature
> device. Please refer to patch 7.  It abstracts feature device common
> data structures and functions in feature-dev.c for FME and AFU now (but
> may be more in the future). The reason we add port enable/disable/etc
> fuctions there for code reuse. e.g FME driver needs port enable/disable
> for PR (in the future, to implement the fpga bridge enable_set function),

We partly discussed this elsewhere.  If the port is implemented as a
fpga-bridge and controlled by an fpga-region, that handles the
enable/disable during PR.

> AFU driver needs similar code to implement reset interface for user space
> application,

I would have thought that you would only want to reset the PR region
right when PR has been completed. Or is there a reason to reset the
port separate from programming the PR region?

> PCIe driver needs port enable to make AFU MMIO region valid,
> then it can access Device Feature Header inside this AFU MMIO during
> enumeration.

If the port/fpga-bridge is controlled by a fpga-region, then the
bridge is enabled once PR is finished.  As long as the AFU code knows
that the fpga-region has been successfully programmed, it knows that
the region can be accessed for the Device Function Header.

> Other fpga_port_* are all for code reuse purpose too. So we
> should not put these function in the same feature-dev.c but a seperated
> file?

What is the plan for this code when you need to support a different
FME in the future? For instance, on a new FPGA device.  I am
suggesting that we need to figure out how to make the port a
fpga-bridge and have it a separate module and separate out the
enumeration code from anything else.  So that this same code can be
used with different bridges.

I've said elsewhere that I'm hoping that this enumeration scheme can
be the central part here and be expandable so that people can use
other fpga-mgr's and fpga-bridges with it.  That will make this code
reusable for future generations of your same project as will as other
FPGA projects.

Alan

>
> Thanks
> Hao
>
>>
>> Alan

^ permalink raw reply	[flat|nested] 93+ messages in thread

* RE: [PATCH 01/16] docs: fpga: add a document for Intel FPGA driver overview
  2017-04-11 18:02           ` Alan Tull
@ 2017-04-12  3:22             ` Wu, Hao
  0 siblings, 0 replies; 93+ messages in thread
From: Wu, Hao @ 2017-04-12  3:22 UTC (permalink / raw)
  To: Alan Tull, Moritz Fischer
  Cc: matthew.gerlach, Moritz Fischer, linux-fpga, linux-kernel, Kang,
	Luwei, Zhang, Yi Z, Luebbers, Enno, Xiao Guangrong

> >> > >
> >> > >
> >> > > Does this HW Architecture require an Intel FPGA?  Couldn't any vendors
> FPGA
> >> > > be used as long as it presented itself the PCIe bus the same and contained
> >> > > an appropriate Device Feature List?
> >
> > I think this is a good (and important) point. Especially when sysfs
> > entries & ioctls constituting ABI depend on it.
> 
> Please cc linux-api@vger.kernel.org on your next version of this as
> linux/Documentation/process/adding-syscalls.rst specifies for new system calls.
> 

Sure, will cc linux-api@vger.kernel.org on the next version patch set.

Thanks
Hao

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 00/16] Intel FPGA Device Drivers
  2017-04-11 19:38   ` Luebbers, Enno
@ 2017-04-12 13:29     ` Jerome Glisse
  2017-04-12 14:46       ` Moritz Fischer
  0 siblings, 1 reply; 93+ messages in thread
From: Jerome Glisse @ 2017-04-12 13:29 UTC (permalink / raw)
  To: Luebbers, Enno
  Cc: Wu Hao, atull, moritz.fischer, linux-fpga, linux-kernel,
	luwei.kang, yi.z.zhang

On Tue, Apr 11, 2017 at 12:38:10PM -0700, Luebbers, Enno wrote:
> Hi Jerome,
> 
> On Thu, Apr 06, 2017 at 04:27:00PM -0400, Jerome Glisse wrote:
> 
> > Do we have an open source toolchain to generate the FPGA configuration
> > (bitstream) ? As it is required for the GPU sub-system that any driver
> > API must comes with open source userspace.
> 
> As far as I know, no FPGA vendor currently provides an open-source version of
> their FPGA synthesis tools - there are, however, free (as in beer) versions
> available for download that can be used for generating FPGA bitstreams. Also,
> there are a number of projects to replace parts of the vendor tools with open
> alternatives (yosys comes to mind, which I believe recently added initial
> support for synthesizing logic for Intel FPGAs).
> 
> As an aside, we are also working on an open-source user-space library that would
> allow you to use this driver to load existing accelerator bitstreams as well as
> enumerate and access accelerators present in the system. This would enable
> workflows where users have access to e.g. a library of FPGA accelerator
> bitstreams and want to write applications that take advantage of these
> accelerators, even without having access to an FPGA synthesis tool.

Yosys is not an open source toolchain is use quartus at least that is my
understand from this commit:

https://github.com/cliffordwolf/yosys/commit/c27dcc1e47fa00cd415893c9d3f637a5d5865988


It is like if on GPU we only had close source compiler for the GPU
instructions set. So FPGA is definitly following different rules than
open source upstream GPU kernel driver abides to.

I see this as highly problematic if not only for security purposes
there is no way for anyone to audit how secure and sane the API you
want to expose to userspace. Those FPGA might have connection to
memory bus or device bus and thus they might get access to any memory.

So i am baffle on how anyone can do any serious review of anything the
fpga driver are doing.

Jérôme

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 00/16] Intel FPGA Device Drivers
  2017-04-12 13:29     ` Jerome Glisse
@ 2017-04-12 14:46       ` Moritz Fischer
  2017-04-12 15:37         ` Jerome Glisse
  0 siblings, 1 reply; 93+ messages in thread
From: Moritz Fischer @ 2017-04-12 14:46 UTC (permalink / raw)
  To: Jerome Glisse
  Cc: Luebbers, Enno, Wu Hao, Alan Tull, linux-fpga,
	Linux Kernel Mailing List, luwei.kang, yi.z.zhang

Hi Jerome,

On Wed, Apr 12, 2017 at 6:29 AM, Jerome Glisse <jglisse@redhat.com> wrote:
> On Tue, Apr 11, 2017 at 12:38:10PM -0700, Luebbers, Enno wrote:
>> Hi Jerome,
>>
>> On Thu, Apr 06, 2017 at 04:27:00PM -0400, Jerome Glisse wrote:
>>
>> > Do we have an open source toolchain to generate the FPGA configuration
>> > (bitstream) ? As it is required for the GPU sub-system that any driver
>> > API must comes with open source userspace.

I think the comparison lacks. No one seems to be bothered by the fact
that the GPUs
hardware is built using closed source CAD tools, even if open source drivers are
available. From an OS perspective the FPGA is hardware.
(Reconfigurable) but hardware.

A better comparison from my point of view would be loading a binary
firmware image ...

>> As far as I know, no FPGA vendor currently provides an open-source version of
>> their FPGA synthesis tools - there are, however, free (as in beer) versions
>> available for download that can be used for generating FPGA bitstreams. Also,
>> there are a number of projects to replace parts of the vendor tools with open
>> alternatives (yosys comes to mind, which I believe recently added initial
>> support for synthesizing logic for Intel FPGAs).
>>
>> As an aside, we are also working on an open-source user-space library that would
>> allow you to use this driver to load existing accelerator bitstreams as well as
>> enumerate and access accelerators present in the system. This would enable
>> workflows where users have access to e.g. a library of FPGA accelerator
>> bitstreams and want to write applications that take advantage of these
>> accelerators, even without having access to an FPGA synthesis tool.
>
> Yosys is not an open source toolchain is use quartus at least that is my
> understand from this commit:
> https://github.com/cliffordwolf/yosys/commit/c27dcc1e47fa00cd415893c9d3f637a5d5865988
>
>
> It is like if on GPU we only had close source compiler for the GPU
> instructions set. So FPGA is definitly following different rules than
> open source upstream GPU kernel driver abides to.
>
> I see this as highly problematic if not only for security purposes
> there is no way for anyone to audit how secure and sane the API you
> want to expose to userspace. Those FPGA might have connection to
> memory bus or device bus and thus they might get access to any memory.

It's up to the user to plug a specific piece of hardware into their
machine. After that it is
up to the user to decide whether he wants to load a bitstream that he
doesn't have the
source code for and that he needs to compile with closed source software.
Do you know if NVIDIA has backdoors in their GPU, Intel in their NIC,
or AMD in their
processor? What about that RTC, do you have the source code they
synthesized their
ASIC design from?

Maybe my mental model on FPGAs is different from yours so feel free to
disagree ;-)

Moritz

^ permalink raw reply	[flat|nested] 93+ messages in thread

* Re: [PATCH 00/16] Intel FPGA Device Drivers
  2017-04-12 14:46       ` Moritz Fischer
@ 2017-04-12 15:37         ` Jerome Glisse
  2017-04-14 19:48           ` Luebbers, Enno
  0 siblings, 1 reply; 93+ messages in thread
From: Jerome Glisse @ 2017-04-12 15:37 UTC (permalink / raw)
  To: Moritz Fischer
  Cc: Luebbers, Enno, Wu Hao, Alan Tull, linux-fpga,
	Linux Kernel Mailing List, luwei.kang, yi.z.zhang

On Wed, Apr 12, 2017 at 07:46:19AM -0700, Moritz Fischer wrote:
> Hi Jerome,
> 
> On Wed, Apr 12, 2017 at 6:29 AM, Jerome Glisse <jglisse@redhat.com> wrote:
> > On Tue, Apr 11, 2017 at 12:38:10PM -0700, Luebbers, Enno wrote:
> >> Hi Jerome,
> >>
> >> On Thu, Apr 06, 2017 at 04:27:00PM -0400, Jerome Glisse wrote:
> >>
> >> > Do we have an open source toolchain to generate the FPGA configuration
> >> > (bitstream) ? As it is required for the GPU sub-system that any driver
> >> > API must comes with open source userspace.
> 
> I think the comparison lacks. No one seems to be bothered by the fact
> that the GPUs hardware is built using closed source CAD tools, even if
> open source drivers are available. From an OS perspective the FPGA is
> hardware. (Reconfigurable) but hardware.
> 
> A better comparison from my point of view would be loading a binary
> firmware image ...

I disagree here, Altera or Xilinx provide OpenCL implementation that allows
to use FPGA. Moreover FPGA are way more capable that micro-controller on
which vast majority of firmware are running.


> >> As far as I know, no FPGA vendor currently provides an open-source version of
> >> their FPGA synthesis tools - there are, however, free (as in beer) versions
> >> available for download that can be used for generating FPGA bitstreams. Also,
> >> there are a number of projects to replace parts of the vendor tools with open
> >> alternatives (yosys comes to mind, which I believe recently added initial
> >> support for synthesizing logic for Intel FPGAs).
> >>
> >> As an aside, we are also working on an open-source user-space library that would
> >> allow you to use this driver to load existing accelerator bitstreams as well as
> >> enumerate and access accelerators present in the system. This would enable
> >> workflows where users have access to e.g. a library of FPGA accelerator
> >> bitstreams and want to write applications that take advantage of these
> >> accelerators, even without having access to an FPGA synthesis tool.
> >
> > Yosys is not an open source toolchain is use quartus at least that is my
> > understand from this commit:
> > https://github.com/cliffordwolf/yosys/commit/c27dcc1e47fa00cd415893c9d3f637a5d5865988
> >
> >
> > It is like if on GPU we only had close source compiler for the GPU
> > instructions set. So FPGA is definitly following different rules than
> > open source upstream GPU kernel driver abides to.
> >
> > I see this as highly problematic if not only for security purposes
> > there is no way for anyone to audit how secure and sane the API you
> > want to expose to userspace. Those FPGA might have connection to
> > memory bus or device bus and thus they might get access to any memory.
> 
> It's up to the user to plug a specific piece of hardware into their
> machine. After that it is up to the user to decide whether he wants
> to load a bitstream that he doesn't have the  source code for and
> that he needs to compile with closed source software. Do you know if
> NVIDIA has backdoors in their GPU, Intel in their NIC, or AMD in their
> processor? What about that RTC, do you have the source code they
> synthesized their ASIC design from?

User do not always know what program their executing. Think someone browsing
some random website, through javascript program you might be able to escape
the web browser and look for fpga device file, if on is present then it
might be able to load a bitstream that would allow it to overwritte system
memory and gain root privilege.

Even if you restrict the ioctl to upload bitstream to root user or some
privilege user, you have to think about VM world. Someone from inside a
vm that has access to an fpga device might be able to upload a bitstream
that would allow to escape the vm and gain root privilege on the host
operating system.

Security expert have proven over and over that no matter how unlikely a
path looks to you, no matter how hard, they will eventualy leverage it.


So you ignoring this issue doesn't make it less real. Pretending that
"secret" bitstream protect you or that only user will load a bitstream
is burying your head in the sand.


So yes knowing the bitstream and links the device has is necessary to be
able to evaluate the security implication of what the kernel fpga driver
are doing.

Cheers,
Jérôme

^ permalink raw reply	[flat|nested] 93+ messages in thread

* RE: [PATCH 04/16] fpga: intel: pcie: parse feature list and create platform device for features.
  2017-04-11 20:21       ` Alan Tull
@ 2017-04-13  4:12         ` Wu, Hao
  0 siblings, 0 replies; 93+ messages in thread
From: Wu, Hao @ 2017-04-13  4:12 UTC (permalink / raw)
  To: 'Alan Tull', Luebbers, Enno
  Cc: Moritz Fischer, linux-fpga, linux-kernel, Kang, Luwei, Zhang,
	Yi Z, Xiao Guangrong, Whisonant, Tim, Rao, Shiva, Rauer,
	Christopher

> On Wed, Apr 5, 2017 at 6:58 AM, Wu Hao <hao.wu@intel.com> wrote:
> > On Mon, Apr 03, 2017 at 04:44:15PM -0500, Alan Tull wrote:
> >> On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <hao.wu@intel.com> wrote:
> >> > From: Xiao Guangrong <guangrong.xiao@linux.intel.com>