[PATCH 0/6] introduce prgdev abstraction library

[PATCH 0/6] introduce prgdev abstraction library

[Chen Jing D(Mark)]

These patch set intend to introduce a DPDK generic programming device layer,
called prgdev, to provide an abstract, generic APIs for applications to
program hardware without knowing the details of programmable devices. From
driver's perspective, they'll try to adapt their functions to the abstract
APIs defined in prgdev.

The major purpose of prgdev is to help DPDK users to dynamically load/upgrade
RTL images for FPGA devices, or upgrade firmware for programmble NICs, without
breaking DPDK application running.


Chen Jing D(Mark) (5):
  prgdev: introduce new library
  prgdev: add debug macro for prgdev
  prgdev: add bus probe and remove functions
  prgdev: add prgdev API exposed to application
  prgdev: add ABI control info

Chen, Jing D (1):
  doc: introduction to prgdev

 config/common_base                       |    7 +
 doc/guides/prog_guide/index.rst          |    1 +
 doc/guides/prog_guide/prgdev_lib.rst     |  465 ++++++++++++++++++++++++++++++
 lib/Makefile                             |    1 +
 lib/librte_eal/common/include/rte_log.h  |    1 +
 lib/librte_prgdev/Makefile               |   57 ++++
 lib/librte_prgdev/rte_prgdev.c           |  459 +++++++++++++++++++++++++++++
 lib/librte_prgdev/rte_prgdev.h           |  401 ++++++++++++++++++++++++++
 lib/librte_prgdev/rte_prgdev_version.map |   19 ++
 mk/rte.app.mk                            |    1 +
 10 files changed, 1412 insertions(+), 0 deletions(-)
 create mode 100644 doc/guides/prog_guide/prgdev_lib.rst
 create mode 100644 lib/librte_prgdev/Makefile
 create mode 100644 lib/librte_prgdev/rte_prgdev.c
 create mode 100644 lib/librte_prgdev/rte_prgdev.h
 create mode 100644 lib/librte_prgdev/rte_prgdev_version.map

-- 
1.7.7.6

[PATCH 1/6] prgdev: introduce new library

[Chen Jing D(Mark)]

Change configuration files, add new library prgdev into compile
system.

Signed-off-by: Chen Jing D(Mark) <jing.d.chen@intel.com>
Signed-off-by: Gerald Rogers <gerald.rogers@intel.com>
---
 config/common_base         |    7 +++++
 lib/Makefile               |    1 +
 lib/librte_prgdev/Makefile |   57 ++++++++++++++++++++++++++++++++++++++++++++
 mk/rte.app.mk              |    1 +
 4 files changed, 66 insertions(+), 0 deletions(-)
 create mode 100644 lib/librte_prgdev/Makefile
 create mode 100644 lib/librte_prgdev/rte_prgdev.c
 create mode 100644 lib/librte_prgdev/rte_prgdev.h

diff --git a/config/common_base b/config/common_base
index aeee13e..71e0000 100644
--- a/config/common_base
+++ b/config/common_base
@@ -377,6 +377,13 @@ CONFIG_RTE_CRYPTO_MAX_DEVS=64
 CONFIG_RTE_CRYPTODEV_NAME_LEN=64
 
 #
+# Compile generic prgdev  device library
+#
+CONFIG_RTE_LIBRTE_PRGDEV=y
+CONFIG_RTE_LIBRTE_PRGDEV_DEBUG=n
+CONFIG_RTE_PRGDEV_MAX_DEVS=128
+
+#
 # Compile PMD for ARMv8 Crypto device
 #
 CONFIG_RTE_LIBRTE_PMD_ARMV8_CRYPTO=n
diff --git a/lib/Makefile b/lib/Makefile
index 4178325..5c62944 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -59,6 +59,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_TABLE) += librte_table
 DIRS-$(CONFIG_RTE_LIBRTE_PIPELINE) += librte_pipeline
 DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += librte_reorder
 DIRS-$(CONFIG_RTE_LIBRTE_PDUMP) += librte_pdump
+DIRS-$(CONFIG_RTE_LIBRTE_PRGDEV) += librte_prgdev
 
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y)
 DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
diff --git a/lib/librte_prgdev/Makefile b/lib/librte_prgdev/Makefile
new file mode 100644
index 0000000..2c7e179
--- /dev/null
+++ b/lib/librte_prgdev/Makefile
@@ -0,0 +1,57 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2016-2017 Intel Corporation. All rights reserved.
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or without
+#   modification, are permitted provided that the following conditions
+#   are met:
+#
+#     * Redistributions of source code must retain the above copyright
+#       notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright
+#       notice, this list of conditions and the following disclaimer in
+#       the documentation and/or other materials provided with the
+#       distribution.
+#     * Neither the name of Intel Corporation nor the names of its
+#       contributors may be used to endorse or promote products derived
+#       from this software without specific prior written permission.
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+#
+# library name
+#
+LIB = librte_prgdev.a
+
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+
+EXPORT_MAP := rte_prgdev_version.map
+
+LIBABIVER := 3
+
+#
+# all source are stored in SRCS-y
+#
+SRCS-$(CONFIG_RTE_LIBRTE_PIPELINE) := rte_prgdev.c
+
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_PIPELINE)-include += rte_prgdev.h
+
+# this lib depends upon:
+DEPDIRS-$(CONFIG_RTE_LIBRTE_PIPELINE) += lib/librte_eal
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_prgdev/rte_prgdev.c b/lib/librte_prgdev/rte_prgdev.c
new file mode 100644
index 0000000..e69de29
diff --git a/lib/librte_prgdev/rte_prgdev.h b/lib/librte_prgdev/rte_prgdev.h
new file mode 100644
index 0000000..e69de29
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index d46a33e..79ab842 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -99,6 +99,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_RING)           += -lrte_ring
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL)            += -lrte_eal
 _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
 _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PRGDEV)         += -lrte_prgdev
 
 ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),n)
 # plugins (link only if static libraries)
-- 
1.7.7.6

[PATCH 2/6] prgdev: add debug macro for prgdev

[Chen Jing D(Mark)]

Add debug macro in rte_log.h to support prgdev.
Add debug message macro for debugging.

Signed-off-by: Chen Jing D(Mark) <jing.d.chen@intel.com>
Signed-off-by: Gerald Rogers <gerald.rogers@intel.com>
---
 lib/librte_eal/common/include/rte_log.h |    1 +
 lib/librte_prgdev/rte_prgdev.h          |   67 +++++++++++++++++++++++++++++++
 2 files changed, 68 insertions(+), 0 deletions(-)

diff --git a/lib/librte_eal/common/include/rte_log.h b/lib/librte_eal/common/include/rte_log.h
index 954b96c..7422962 100644
--- a/lib/librte_eal/common/include/rte_log.h
+++ b/lib/librte_eal/common/include/rte_log.h
@@ -80,6 +80,7 @@ struct rte_logs {
 #define RTE_LOGTYPE_MBUF    0x00010000 /**< Log related to mbuf. */
 #define RTE_LOGTYPE_CRYPTODEV 0x00020000 /**< Log related to cryptodev. */
 #define RTE_LOGTYPE_EFD     0x00040000 /**< Log related to EFD. */
+#define RTE_LOGTYPE_PRG    0x00080000 /**< Log related to prg device. */
 
 /* these log types can be used in an application */
 #define RTE_LOGTYPE_USER1   0x01000000 /**< User-defined log type 1. */
diff --git a/lib/librte_prgdev/rte_prgdev.h b/lib/librte_prgdev/rte_prgdev.h
index e69de29..52f8a45 100644
--- a/lib/librte_prgdev/rte_prgdev.h
+++ b/lib/librte_prgdev/rte_prgdev.h
@@ -0,0 +1,67 @@
+/*-
+ *   BSD LICENSE
+ *
+ *   Copyright(c) 2016-2017 Intel Corporation. All rights reserved.
+ *   All rights reserved.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * Neither the name of Intel Corporation nor the names of its
+ *       contributors may be used to endorse or promote products derived
+ *       from this software without specific prior written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _RTE_PRGDEV_H_
+#define _RTE_PRGDEV_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdint.h>
+
+#include <rte_dev.h>
+#include <rte_log.h>
+#include <rte_pci.h>
+#include <rte_dev.h>
+#include <rte_errno.h>
+/* Logging Macros */
+#define PRG_LOG_ERR(...) \
+	RTE_LOG(ERR, PRG, \
+		RTE_FMT("%s() line %u: " RTE_FMT_HEAD(__VA_ARGS__,) "\n", \
+			__func__, __LINE__, RTE_FMT_TAIL(__VA_ARGS__,)))
+
+#ifdef RTE_LIBRTE_PRGDEV_DEBUG
+#define PRG_LOG_DEBUG(...) \
+	RTE_LOG(DEBUG, PRG, \
+		RTE_FMT("%s() line %u: " RTE_FMT_HEAD(__VA_ARGS__,) "\n", \
+			__func__, __LINE__, RTE_FMT_TAIL(__VA_ARGS__,)))
+#else
+#define PRG_LOG_DEBUG(...) (void)0
+#endif
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PRGDEV_H_ */
-- 
1.7.7.6

[PATCH 3/6] prgdev: add bus probe and remove functions

[Chen Jing D(Mark)]

This patch adds 4 major functions: pci_probe/pci_remove,
prgdev_allocate/prgdev_release to help drivers registering
programmable devices.

pci_probe functions helps drivers to allocate prgdev structures
and trigger an initialization function that drivers provided.
This function will be called when bus scan action performed or
hotplug occured.

prgdev_allocate will allocate a prgdev structures, but won't
perform an intialization of the device. Comparing pci_probe
function which will be triggered by bus scan, it can be called
at any time by driver.

Signed-off-by: Chen Jing D(Mark) <jing.d.chen@intel.com>
Signed-off-by: Gerald Rogers <gerald.rogers@intel.com>
---
 lib/librte_prgdev/rte_prgdev.c |  233 ++++++++++++++++++++++++++++++++++++++++
 lib/librte_prgdev/rte_prgdev.h |  228 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 461 insertions(+), 0 deletions(-)

diff --git a/lib/librte_prgdev/rte_prgdev.c b/lib/librte_prgdev/rte_prgdev.c
index e69de29..03465f9 100644
--- a/lib/librte_prgdev/rte_prgdev.c
+++ b/lib/librte_prgdev/rte_prgdev.c
@@ -0,0 +1,233 @@
+/*-
+ *   BSD LICENSE
+ *
+ *   Copyright(c) 2016-2017 Intel Corporation. All rights reserved.
+ *   All rights reserved.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * Neither the name of Intel Corporation nor the names of its
+ *       contributors may be used to endorse or promote products derived
+ *       from this software without specific prior written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <ctype.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <stdarg.h>
+#include <errno.h>
+#include <stdint.h>
+#include <inttypes.h>
+#include <netinet/in.h>
+
+#include <rte_log.h>
+#include <rte_debug.h>
+#include <rte_pci.h>
+#include <rte_memory.h>
+#include <rte_memcpy.h>
+#include <rte_memzone.h>
+#include <rte_launch.h>
+#include <rte_eal.h>
+#include <rte_per_lcore.h>
+#include <rte_lcore.h>
+#include <rte_common.h>
+#include <rte_mempool.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_errno.h>
+#include "rte_prgdev.h"
+
+static struct rte_prgdev rte_prgdev_devices[RTE_PRGDEV_MAX_DEVS];
+static uint8_t nb_devs;
+
+static struct rte_prgdev *rte_prgdev_allocated(char *name);
+
+int
+rte_prgdev_pci_probe(struct rte_pci_driver *pci_drv,
+		      struct rte_pci_device *pci_dev)
+{
+	struct prgdev_driver *prgdrv;
+	struct rte_prgdev *prgdev;
+	char prgdev_name[PRGDEV_NAME_MAX_LEN];
+	int retval;
+
+	/* Not support secondary process to operate on prgdev */
+	RTE_PRG_PRIMARY_PROC_OR_ERR_RET(-EINVAL);
+
+	prgdrv = (struct prgdev_driver *)pci_drv;
+	if (prgdrv == NULL)
+		return -ENODEV;
+
+	rte_eal_pci_device_name(&pci_dev->addr, prgdev_name,
+			sizeof(prgdev_name));
+
+	prgdev = rte_prgdev_allocate(prgdev_name);
+	if (prgdev == NULL)
+		return -1;
+
+	prgdev->dev_data.dev_private =
+			rte_zmalloc_socket(
+					"prgdevprivate structure",
+					prgdrv->dev_private_size,
+					RTE_CACHE_LINE_SIZE,
+					rte_socket_id());
+
+	if (prgdev->dev_data.dev_private == NULL)
+		rte_panic("Cannot allocate memzone for private "
+				"device data");
+
+	prgdev->device = &pci_dev->device;
+	prgdev->driver = prgdrv;
+
+
+	/* Invoke PMD device initialization function */
+	retval = (*prgdrv->prgdev_init)(prgdev);
+	if (retval == 0)
+		return 0;
+
+	PRG_LOG_ERR("driver %s: prgdev_init(vendor_id=0x%x device_id=0x%x)"
+			" failed", pci_drv->driver.name,
+			(unsigned) pci_dev->id.vendor_id,
+			(unsigned) pci_dev->id.device_id);
+
+	rte_free(prgdev->dev_data.dev_private);
+
+	rte_prgdev_release(prgdev);
+
+	return -ENXIO;
+}
+
+int
+rte_prgdev_pci_remove(struct rte_pci_device *pci_dev)
+{
+	struct prgdev_driver *prgdrv;
+	struct rte_prgdev *prgdev;
+	char prgdev_name[PRGDEV_NAME_MAX_LEN];
+	int ret;
+
+	if (pci_dev == NULL)
+		return -EINVAL;
+
+	/* Not support secondary process to operate on prgdev */
+	RTE_PRG_PRIMARY_PROC_OR_ERR_RET(-EINVAL);
+
+	rte_eal_pci_device_name(&pci_dev->addr, prgdev_name,
+			sizeof(prgdev_name));
+
+
+	prgdev = rte_prgdev_allocated(prgdev_name);
+	if (prgdev == NULL)
+		return -ENODEV;
+
+	prgdrv = (struct prgdev_driver *)pci_dev->driver;
+	if (prgdrv == NULL)
+		return -ENODEV;
+
+	/* Invoke device uninit function */
+	if (*prgdrv->prgdev_uninit) {
+		ret = (*prgdrv->prgdev_uninit)(prgdev);
+		if (ret)
+			return ret;
+	}
+
+	/* free prgdev device */
+	rte_prgdev_release(prgdev);
+
+	rte_free(prgdev->dev_data.dev_private);
+
+	return 0;
+}
+
+static uint8_t
+prgdev_find_free_dev_slot(void)
+{
+	unsigned i;
+
+	for (i = 0; i < RTE_PRGDEV_MAX_DEVS; i++) {
+		if (rte_prgdev_devices[i].attached == PRGDEV_DETACHED)
+			return i;
+	}
+	return RTE_PRGDEV_MAX_DEVS;
+}
+
+static struct rte_prgdev *
+rte_prgdev_allocated(char *name)
+{
+	unsigned i;
+
+	for (i = 0; i < RTE_PRGDEV_MAX_DEVS; i++) {
+		if ((rte_prgdev_devices[i].attached == PRGDEV_ATTACHED) &&
+		    strcmp(rte_prgdev_devices[i].dev_data.name, name) == 0)
+			return &rte_prgdev_devices[i];
+	}
+	return NULL;
+}
+
+struct rte_prgdev *
+rte_prgdev_allocate(char *name)
+{
+	uint8_t dev_id;
+	struct rte_prgdev *prg_dev;
+
+	/* Not support secondary process to operate on prgdev */
+	RTE_PRG_PRIMARY_PROC_OR_ERR_RET(NULL);
+	dev_id = prgdev_find_free_dev_slot();
+	if (dev_id == RTE_PRGDEV_MAX_DEVS) {
+		PRG_LOG_DEBUG("Reached maximum number of prgdev numbers");
+		return NULL;
+	}
+
+	if (rte_prgdev_allocated(name) != NULL) {
+		PRG_LOG_DEBUG("Prgdev Device with name %s already allocated!\n",
+				name);
+		return NULL;
+	}
+
+	prg_dev = &rte_prgdev_devices[dev_id];
+	memset(prg_dev, 0, sizeof(*prg_dev));
+	prg_dev->attached = PRGDEV_ATTACHED;
+	snprintf(prg_dev->dev_data.name, sizeof(prg_dev->dev_data.name), "%s",
+			name);
+	prg_dev->dev_data.devid = dev_id;
+	prg_dev->dev_data.numa_node = rte_socket_id();
+	nb_devs++;
+	return prg_dev;
+}
+
+int
+rte_prgdev_release(struct rte_prgdev *prg_dev)
+{
+	/* Not support secondary process to operate on prgdev */
+	RTE_PRG_PRIMARY_PROC_OR_ERR_RET(-EINVAL);
+
+	if (prg_dev == NULL)
+		return -EINVAL;
+
+	prg_dev->attached = PRGDEV_ATTACHED;
+	prg_dev->dev_ops = NULL;
+	prg_dev->driver = NULL;
+	nb_devs--;
+	return 0;
+}
+
diff --git a/lib/librte_prgdev/rte_prgdev.h b/lib/librte_prgdev/rte_prgdev.h
index 52f8a45..c25cfef 100644
--- a/lib/librte_prgdev/rte_prgdev.h
+++ b/lib/librte_prgdev/rte_prgdev.h
@@ -45,6 +45,9 @@
 #include <rte_pci.h>
 #include <rte_dev.h>
 #include <rte_errno.h>
+
+#define PRGDEV_NAME_MAX_LEN	(64)
+
 /* Logging Macros */
 #define PRG_LOG_ERR(...) \
 	RTE_LOG(ERR, PRG, \
@@ -60,6 +63,231 @@
 #define PRG_LOG_DEBUG(...) (void)0
 #endif
 
+/*
+ * reflect the device status
+ */
+enum rte_prgdev_stat {
+	RTE_PRG_STAT_UNKNOWN = 0,  /** Device in a unkown state */
+	RTE_PRG_STAT_READY,   /** Device is ready to be programmed. */
+	RTE_PRG_STAT_OPEN, /*Device is opened, waiting for programming */
+	RTE_PRG_STAT_ERASING, /** Device is being programmed. */
+	/** Device is performing functionalities and can't be programmed. */
+	RTE_PRG_STAT_RUNNING,
+};
+
+/* Reflect the function block attributes */
+/* Block is readable */
+#define RTE_PRG_BLK_ATTR_RD	0x00000001
+/* Block is writable */
+#define RTE_PRG_BLK_ATTR_WR	0x00000002
+/* Block is readable and writable */
+#define RTE_PRG_FUNC_ATTR_RDWR	(RTE_PRG_BLK_ATTR_RD & RTE_PRG_BLK_ATTR_WR)
+
+struct rte_prgdev_blk_info {
+	unsigned int size;     /* the block size in bytes */
+	unsigned int version;  /* It's optional */
+	unsigned int flags;    /* Flags to indicate blk is readable/writable */
+};
+
+#define MAX_SIGNATURE_LEN	256
+#define MAX_BLK_NUM		256
+
+struct rte_prgdev;
+struct rte_prgdev_info;
+
+/*
+ * reflect the firmware status after programming with new image
+ */
+enum rte_prg_fwstat {
+	RTE_PRGDEV_FWSTAT_OK = 0,  /** Image are running well */
+	RTE_PRGDEV_FWSTAT_ERR = -1,   /** Image has error. */
+	RTE_PRGDEV_FWSTAT_ERR_VALID = -2, /** Image is not valid. */
+	RTE_PRGDEV_FWSTAT_ERR_CKSUM = -3, /** Image checksum is not correct. */
+	RTE_PRGDEV_FWSTAT_ERR_LEN = -4, /** Image length is not correct. */
+};
+
+/*
+*     prg_dev_init routine
+*
+*     returns : 0 success, non zero failure.
+*/
+typedef int (*prgdev_init_t)(struct rte_prgdev *prg_dev);
+
+/*
+*     prg_dev_uninit routine
+*
+*     returns : 0 success, non zero failure.
+*/
+typedef int (*prgdev_uninit_t)(struct rte_prgdev *prg_dev);
+
+
+/**< @internal query infos of the device. */
+typedef int (*prgdev_get_info_t)(struct rte_prgdev *prg_dev,
+			struct rte_prgdev_info *info);
+
+/**< @internal Open device for programming, acquiring on-die image, etc. */
+typedef int (*prgdev_open_t)(struct rte_prgdev *prg_dev);
+
+/**< @internal Download image from host to programmable device. */
+typedef int (*prgdev_img_download_t)(struct rte_prgdev *prg_dev,
+		uint8_t *buffer_ptr, uint32_t buf_len);
+
+/**< @internal Upload image from programmable device to host. */
+typedef int (*prgdev_img_upload_t)(struct rte_prgdev *prg_dev,
+		uint8_t *buffer_ptr, uint32_t buf_len, uint32_t *act_len);
+
+/**< @internal Check if the downloaded image works in expected way. */
+typedef int (*prgdev_check_stat_t)(struct rte_prgdev *prg_dev,
+			enum rte_prg_fwstat *stat);
+
+/**< @internal free up resources or whatever to do to hardware. */
+typedef int (*prgdev_close_t)(struct rte_prgdev *prg_dev);
+
+/**< @internal bind personalities with drivers. */
+typedef int (*prgdev_bind_t)(struct rte_prgdev *prg_dev);
+
+/**< @internal unbind all personalities except prgdev from drivers. */
+typedef int (*prgdev_unbind_t)(struct rte_prgdev *prg_dev);
+
+/** prgdev device operations function pointer table */
+struct rte_prgdev_ops {
+	prgdev_get_info_t	prg_infos_get;	/**< Get device info. */
+	prgdev_open_t		prg_open;	/**< Open device. */
+	prgdev_img_download_t	prg_download;	/**< download Image. */
+	prgdev_img_upload_t	prg_upload;	/**< upload device. */
+	prgdev_check_stat_t	prg_check_stat;	/**< Get image status. */
+	prgdev_close_t		prg_close;	/**< Open device. */
+	prgdev_bind_t		prg_bind;	/**< bind drivers */
+	prgdev_unbind_t		prg_unbind;	/**< unbind drivers */
+};
+
+/**
+ * @internal
+ * The structure associated with a programmable device driver.
+ *
+ * - An *rte_pci_driver* structure (which must be the first field).
+ *
+ * - The *prgdev_init_t* function invoked for each matching PCI device.
+ *
+ * - The *prgdev_uninit_t* function invoked for each matching PCI device.
+ *
+ * - The size of the private data to allocate for each matching device.
+ */
+struct prgdev_driver {
+	struct rte_pci_driver pci_drv;    /**< The PMD is also a PCI driver. */
+	prgdev_init_t  prgdev_init;      /**< Device init function. */
+	prgdev_uninit_t prgdev_uninit;  /**< Device uninit function. */
+	unsigned int dev_private_size;    /**< Size of device private data. */
+};
+
+struct rte_prgdev_info {
+	const char *driver_name; /**< Device Driver name. */
+	/* Programable device HW version number. it's possible that app
+	 *  have dependency to HW version.
+	 */
+	struct rte_pci_device *pci_dev;	/**< PCI information. */
+	uint16_t hw_ver_major;  /**< major version number */
+	uint16_t hw_ver_minor;  /**< minor version number */
+
+	/* A array to store hardware, firmware, running image info.Each device
+	* can define and interpret the info. For example, a device can define
+	* byte[3:0] as signature for on-die personality, byte[5:4] is the major
+	* version, byte[7:6] is minor version. if 0xFFEA1000 is a signature for
+	* virtio personality, major version is 0x0001, minor version is 0x0004.
+	* then signature can be defined :
+	* sig_num = 8;
+	* signature[7:0]= {00, 0x10, 0xEA, 0xFF, 0x00, 0x01, 0x00, 0x04};
+	*/
+	unsigned int sig_num;  /** < the valid signature length in bytes> */
+	char signature[MAX_SIGNATURE_LEN];
+	enum rte_prgdev_stat status;
+
+	/**< number of blocks within device */
+	unsigned int blk_num;
+	/**< block info */
+	struct rte_prgdev_blk_info blk_info[MAX_BLK_NUM];
+};
+
+struct rte_prgdev_data {
+	char name[PRGDEV_NAME_MAX_LEN];
+	unsigned int devid; /**< Index to bound host interface, or 0 if none. */
+	void *dev_private;  /**< prgdev -specific private data */
+	int numa_node;  /**< NUMA node connection */
+	const char *drv_name;   /**< Driver name */
+};
+
+enum {
+	PRGDEV_DETACHED = 0,
+	PRGDEV_ATTACHED
+};
+
+struct rte_prgdev {
+	struct rte_prgdev_data dev_data;
+	const struct prgdev_driver *driver;
+	struct rte_prgdev_ops *dev_ops;
+	/**< Supported features */
+	uint64_t feature_flags;
+	struct rte_device *device;
+	uint8_t attached; /**< Flag indicating the device is attached */
+} __rte_cache_aligned;
+
+/* Macros to check for valid device */
+#define RTE_PRG_VALID_DEVID_OR_ERR_RET(dev_id, retval) do { \
+	if (!rte_prgdev_is_valid_dev(dev_id)) { \
+		PRG_LOG_DEBUG("Invalid dev_id=%d", dev_id); \
+		return retval; \
+	} \
+} while (0)
+
+#define RTE_PRG_FUNC_PTR_OR_ERR_RET(func, retval) do { \
+	if ((func) == NULL) { \
+		PRG_LOG_DEBUG("Function not supported\n"); \
+		return retval; \
+	} \
+} while (0)
+
+#define RTE_PRG_PRIMARY_PROC_OR_ERR_RET(retval) do { \
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) { \
+		PRG_LOG_DEBUG("prgdev doesn't support secondary process"); \
+		return retval; \
+	} \
+} while (0)
+
+/**
+ * Wrapper for use by pci drivers as a .probe function to attach to a prgdev
+ * interface.
+ */
+int rte_prgdev_pci_probe(struct rte_pci_driver *pci_drv,
+			    struct rte_pci_device *pci_dev);
+
+/**
+ * Wrapper for use by pci drivers as a .remove function to detach a prgdev
+ * interface.
+ */
+int rte_prgdev_pci_remove(struct rte_pci_device *pci_dev);
+
+/**
+ * @internal
+ * Allocates a new prgdev slot for an programmable device and returns the
+ * pointer to that slot for the driver to use.
+ *
+ * @param name: Unique identifier name for each Ethernet device
+ * @return
+ *   - Slot in the rte_dev_devices array for a new device;
+ */
+struct rte_prgdev *rte_prgdev_allocate(char *name);
+
+/**
+ * @internal
+ * Release the specified prgdev device.
+ *
+ * @param eth_dev
+ * The *prg_dev* pointer is the address of the *rte_prgdev* structure.
+ * @return
+ *   - 0 on success, negative on error
+ */
+int rte_prgdev_release(struct rte_prgdev *prg_dev);
+
 #ifdef __cplusplus
 }
 #endif
-- 
1.7.7.6

[PATCH 4/6] prgdev: add prgdev API exposed to application

[Chen Jing D(Mark)]

Add a series of API and implementations that application can
operate on a programmble device. The major functions are download
and upload an image to/from device.

Signed-off-by: Chen Jing D(Mark) <jing.d.chen@intel.com>
Signed-off-by: Gerald Rogers <gerald.rogers@intel.com>
---
 lib/librte_prgdev/rte_prgdev.c |  226 ++++++++++++++++++++++++++++++++++++++++
 lib/librte_prgdev/rte_prgdev.h |  106 +++++++++++++++++++
 2 files changed, 332 insertions(+), 0 deletions(-)

diff --git a/lib/librte_prgdev/rte_prgdev.c b/lib/librte_prgdev/rte_prgdev.c
index 03465f9..558e97b 100644
--- a/lib/librte_prgdev/rte_prgdev.c
+++ b/lib/librte_prgdev/rte_prgdev.c
@@ -231,3 +231,229 @@ struct rte_prgdev *
 	return 0;
 }
 
+int
+rte_prgdev_is_valid_dev(uint8_t dev_id)
+{
+	/* Not support secondary process to operate on prgdev */
+	RTE_PRG_PRIMARY_PROC_OR_ERR_RET(-EINVAL);
+	RTE_PRG_VALID_DEVID_OR_ERR_RET(dev_id, -EINVAL);
+
+	if (dev_id >= RTE_PRGDEV_MAX_DEVS ||
+		rte_prgdev_devices[dev_id].attached != PRGDEV_ATTACHED)
+		return 0;
+	else
+		return 1;
+}
+
+uint8_t
+rte_prgdev_count(void)
+{
+	/* Not support secondary process to operate on prgdev */
+	RTE_PRG_PRIMARY_PROC_OR_ERR_RET(-EINVAL);
+
+	return nb_devs;
+}
+
+static inline int
+prgdev_devid_check(uint8_t dev_id, struct rte_prgdev **dev)
+{
+	/* Not support secondary process to operate on prgdev */
+	RTE_PRG_PRIMARY_PROC_OR_ERR_RET(-EINVAL);
+	RTE_PRG_VALID_DEVID_OR_ERR_RET(dev_id, -EINVAL);
+
+	if (dev_id >= nb_devs) {
+		PRG_LOG_ERR("Invalid dev_id=%d", dev_id);
+		return -EINVAL;
+	}
+
+	*dev = &rte_prgdev_devices[dev_id];
+	return 0;
+}
+
+int
+rte_prgdev_info_get(uint8_t dev_id,
+			struct rte_prgdev_info *info)
+{
+	struct rte_prgdev *dev;
+	int ret;
+
+	ret = prgdev_devid_check(dev_id, &dev);
+	if (ret)
+		return ret;
+
+	memset(info, 0, sizeof(*info));
+
+	RTE_PRG_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->prg_infos_get, -ENOTSUP);
+	(*dev->dev_ops->prg_infos_get)(dev, info);
+
+	info->pci_dev = RTE_DEV_TO_PCI(dev->device);
+	if (dev->driver)
+		info->driver_name = dev->driver->pci_drv.driver.name;
+	return 0;
+}
+
+int
+rte_prgdev_open(uint8_t dev_id)
+{
+	struct rte_prgdev *dev;
+	struct rte_prgdev_info dev_info;
+	int ret;
+
+	ret = prgdev_devid_check(dev_id, &dev);
+	if (ret)
+		return ret;
+
+	RTE_PRG_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->prg_infos_get, -ENOTSUP);
+	RTE_PRG_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->prg_open, -ENOTSUP);
+
+	(*dev->dev_ops->prg_infos_get)(dev, &dev_info);
+
+	if (dev_info.status != RTE_PRG_STAT_READY)
+		return -1;
+
+	return (*dev->dev_ops->prg_open)(dev);
+}
+
+int
+rte_prgdev_img_download(uint8_t dev_id,
+			uint8_t *buffer_ptr, uint32_t buf_len)
+{
+	struct rte_prgdev *dev;
+	struct rte_prgdev_info dev_info;
+	int ret;
+
+	ret = prgdev_devid_check(dev_id, &dev);
+	if (ret)
+		return ret;
+
+	if (buffer_ptr == NULL || buf_len == 0)
+		return -EINVAL;
+
+	RTE_PRG_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->prg_download, -ENOTSUP);
+	RTE_PRG_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->prg_infos_get, -ENOTSUP);
+
+	(*dev->dev_ops->prg_infos_get)(dev, &dev_info);
+
+	if (dev_info.status != RTE_PRG_STAT_OPEN)
+		return -1;
+
+	return (*dev->dev_ops->prg_download)(dev, buffer_ptr, buf_len);
+}
+
+int
+rte_prgdev_img_upload(uint8_t dev_id, uint8_t *buffer_ptr,
+		uint32_t buf_len, uint32_t *act_len)
+{
+	struct rte_prgdev *dev;
+	struct rte_prgdev_info dev_info;
+	int ret;
+
+	ret = prgdev_devid_check(dev_id, &dev);
+	if (ret)
+		return ret;
+
+	if (buffer_ptr == NULL || buf_len == 0 || act_len == NULL)
+		return -EINVAL;
+
+	RTE_PRG_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->prg_upload, -ENOTSUP);
+	RTE_PRG_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->prg_infos_get, -ENOTSUP);
+
+	(*dev->dev_ops->prg_infos_get)(dev, &dev_info);
+
+	if (dev_info.status != RTE_PRG_STAT_OPEN)
+		return -1;
+
+	return (*dev->dev_ops->prg_upload)(dev, buffer_ptr, buf_len, act_len);
+}
+
+int
+rte_prgdev_check_stat(uint8_t dev_id, enum rte_prg_fwstat *stat)
+{
+	struct rte_prgdev *dev;
+	struct rte_prgdev_info dev_info;
+	int ret;
+
+	ret = prgdev_devid_check(dev_id, &dev);
+	if (ret)
+		return ret;
+
+	RTE_PRG_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->prg_check_stat, -ENOTSUP);
+	RTE_PRG_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->prg_infos_get, -ENOTSUP);
+
+	(*dev->dev_ops->prg_infos_get)(dev, &dev_info);
+
+	/* Only can check downloaded image running status after device is open,
+	* but prgdev doesn't has the capability to check whether an image is
+	* downloaded prior to this operation.
+	*/
+	if (dev_info.status != RTE_PRG_STAT_OPEN)
+		return -EINVAL;
+
+	return (*dev->dev_ops->prg_check_stat)(dev, stat);
+}
+
+int
+rte_prgdev_close(uint8_t dev_id)
+{
+	struct rte_prgdev *dev;
+	struct rte_prgdev_info dev_info;
+	int ret;
+
+	ret = prgdev_devid_check(dev_id, &dev);
+	if (ret)
+		return ret;
+
+	RTE_PRG_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->prg_infos_get, -ENOTSUP);
+	RTE_PRG_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->prg_close, -ENOTSUP);
+
+	(*dev->dev_ops->prg_infos_get)(dev, &dev_info);
+
+	if (dev_info.status != RTE_PRG_STAT_OPEN)
+		return -EINVAL;
+
+	return (*dev->dev_ops->prg_close)(dev);
+}
+
+int
+rte_prgdev_bind(uint8_t dev_id)
+{
+	struct rte_prgdev *dev;
+	struct rte_prgdev_info dev_info;
+	int ret;
+
+	ret = prgdev_devid_check(dev_id, &dev);
+	if (ret)
+		return ret;
+
+	RTE_PRG_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->prg_infos_get, -ENOTSUP);
+	RTE_PRG_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->prg_bind, -ENOTSUP);
+
+	(*dev->dev_ops->prg_infos_get)(dev, &dev_info);
+
+	if (dev_info.status != RTE_PRG_STAT_READY)
+		return -EINVAL;
+
+	return (*dev->dev_ops->prg_bind)(dev);
+}
+
+int
+rte_prgdev_unbind(uint8_t dev_id)
+{
+	struct rte_prgdev *dev;
+	struct rte_prgdev_info dev_info;
+	int ret;
+
+	ret = prgdev_devid_check(dev_id, &dev);
+	if (ret)
+		return ret;
+
+	RTE_PRG_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->prg_infos_get, -ENOTSUP);
+	RTE_PRG_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->prg_unbind, -ENOTSUP);
+
+	(*dev->dev_ops->prg_infos_get)(dev, &dev_info);
+
+	if (dev_info.status != RTE_PRG_STAT_READY)
+		return -EINVAL;
+
+	return (*dev->dev_ops->prg_unbind)(dev);
+}
diff --git a/lib/librte_prgdev/rte_prgdev.h b/lib/librte_prgdev/rte_prgdev.h
index c25cfef..db57512 100644
--- a/lib/librte_prgdev/rte_prgdev.h
+++ b/lib/librte_prgdev/rte_prgdev.h
@@ -288,6 +288,112 @@ int rte_prgdev_pci_probe(struct rte_pci_driver *pci_drv,
  */
 int rte_prgdev_release(struct rte_prgdev *prg_dev);
 
+/*
+*  Query what personality is in the device.
+*
+*  @param device_id
+*   The port identifier of the programmable device.
+*  @param info
+*   A pointer to a structure of type *rte_prg_dev_info* to be filled with
+*   the information of the programmable device.
+*/
+int rte_prgdev_info_get(uint8_t device_id,
+			struct rte_prgdev_info *info);
+
+/**
+ * Check if dev_id of device is attached
+ *
+ * @param dev_id
+ *   The device identifier of the programmable device
+ * @return
+ *   - 0 if device is out of range or not attached
+ *   - 1 if device is attached
+ */
+int rte_prgdev_is_valid_dev(uint8_t dev_id);
+
+/**
+ * Get the total number of programmable devices that have been successfully
+ * initialized.
+ * If the application unplugs a device using hotplug function, The enabled
+ * device numbers may be noncontiguous. In the case, the applications need to
+ * manage enabled port by themselves.
+ *
+ * @return
+ *   - The total number of usable programmable devices.
+ */
+uint8_t rte_prgdev_count(void);
+
+/*
+*  Open device for programming, acquiring on-die image, etc.
+*  Need to call this function first
+*  prior to calling other functionalities.
+*  In case the device is performing some tasks, it's device's decision on
+*  what result is returned.
+*/
+int rte_prgdev_open(uint8_t device_id);
+
+/*
+* Download image from host to programmable device.
+*
+* @param device_id
+*   The port identifier of the programmable device.
+* @param buffer_ptr
+*   A pointer to a buffer that stored the image ready downloading to device
+* @param buf_len
+*   the total image length in bytes.
+*/
+
+int rte_prgdev_img_download(uint8_t device_id,
+			uint8_t *buffer_ptr, uint32_t buf_len);
+
+/*
+* Upload image from programmable device to host.
+*
+* @param device_id
+*   The port identifier of the programmable device.
+* @param buffer_ptr
+*   A pointer to a buffer that store uploaded image
+* @param buf_len
+*   the total buffer length in bytes.
+* @param act_len
+*   pointer to the actual image length in bytes.
+*/
+
+int rte_prgdev_img_upload(uint8_t device_id, uint8_t *buffer_ptr,
+		uint32_t buf_len, uint32_t *act_len);
+
+/*
+*  Check if the downloaded image running on die works in expected way, optional
+*   function.
+* @param device_id
+* @param stat: pointer to image status. Output parameters.
+*   The port identifier of the programmable device.
+*/
+int rte_prgdev_check_stat(uint8_t device_id, enum rte_prg_fwstat *stat);
+
+/*
+* Called to free up resources or whatever to do to hardware
+* after an erase or load of the program.
+* @param device_id
+*   The device identifier of the programmable device.
+*/
+int rte_prgdev_close(uint8_t device_id);
+
+/*
+* Called to bind a programmable device with drivers after close function is
+* called.
+* @param device_id
+*   The device identifier of the programmable device.
+*/
+int rte_prgdev_bind(uint8_t device_id);
+
+/*
+* Called to unbind all functions except prgdev from drivers.
+* @param device_id
+*   The device identifier of the programmable device.
+*/
+int rte_prgdev_unbind(uint8_t device_id);
+
 #ifdef __cplusplus
 }
 #endif
-- 
1.7.7.6

[PATCH 5/6] prgdev: add ABI control info

[Chen Jing D(Mark)]

Add rte_prgdev_version.map file.

Signed-off-by: Chen Jing D(Mark) <jing.d.chen@intel.com>
Signed-off-by: Gerald Rogers <gerald.rogers@intel.com>
---
 lib/librte_prgdev/rte_prgdev_version.map |   19 +++++++++++++++++++
 1 files changed, 19 insertions(+), 0 deletions(-)
 create mode 100644 lib/librte_prgdev/rte_prgdev_version.map

diff --git a/lib/librte_prgdev/rte_prgdev_version.map b/lib/librte_prgdev/rte_prgdev_version.map
new file mode 100644
index 0000000..51dc15a
--- /dev/null
+++ b/lib/librte_prgdev/rte_prgdev_version.map
@@ -0,0 +1,19 @@
+DPDK_17.05 {
+	global:
+
+	rte_prgdev_pci_probe;
+	rte_prgdev_pci_remove;
+	rte_prgdev_allocate;
+	rte_prgdev_release;
+	rte_prgdev_info_get;
+	rte_prgdev_is_valid_dev;
+	rte_prgdev_open;
+	rte_prgdev_img_download;
+	rte_prgdev_img_upload;
+	rte_prgdev_check_stat;
+	rte_prgdev_close;
+	rte_prgdev_bind;
+	rte_prgdev_unbind;
+
+	local: *;
+};
-- 
1.7.7.6

[PATCH 6/6] doc: introduction to prgdev

[Chen Jing D(Mark)]

From: "Chen, Jing D" <jing.d.chen@intel.com>

This is the documentation to describe what prgdev is, how to use
prgdev API and accomplish an image download.

Signed-off-by: Chen Jing D(Mark) <jing.d.chen@intel.com>
Signed-off-by: Gerald Rogers <gerald.rogers@intel.com>
Signed-off-by: John Mcnamara <john.mcnamara@intel.com>
---
 doc/guides/prog_guide/index.rst      |    1 +
 doc/guides/prog_guide/prgdev_lib.rst |  465 ++++++++++++++++++++++++++++++++++
 2 files changed, 466 insertions(+), 0 deletions(-)
 create mode 100644 doc/guides/prog_guide/prgdev_lib.rst

diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index 77f427e..4a24264 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -60,6 +60,7 @@ Programmer's Guide
     qos_framework
     power_man
     packet_classif_access_ctrl
+    prgdev_lib
     packet_framework
     vhost_lib
     port_hotplug_framework
diff --git a/doc/guides/prog_guide/prgdev_lib.rst b/doc/guides/prog_guide/prgdev_lib.rst
new file mode 100644
index 0000000..e2d1e2e
--- /dev/null
+++ b/doc/guides/prog_guide/prgdev_lib.rst
@@ -0,0 +1,465 @@
+Overview
+========
+
+More and more devices start to support on-die programming, which include
+NIC, GPU, FPGA, etc. This capability has a requirement to drivers that
+introduce a API for application to download/upload image to/from the HW.
+So, it's necessary to provide a generic API to perform image download, upload,
+validation, etc.
+
+This RFC patch intends to introduce a DPDK generic programming device layer,
+called prgdev below, to provide an abstract, generic APIs for applications to
+program hardware without knowing the details of programmable devices. From
+driver's perspective, they'll try to adapt their functions to the abstract
+APIs defined in prgdev.
+
+The major purpose of prgdev is to help DPDK users to load/upgrade RTL images
+for FPGA devices, or upgrade firmware for programmble NICs.
+
+But those devices that have the capability to do on-die programming may
+expose versatile talents to apps. Add/delete/update entries in flow table,
+update/overwrite the firmware/microcode, add a new algorithm, update profiles,
+etc, which is hard to be abstracted as generic behaviors. So, prgdev won't
+include the APIs to perform device-unique actions in current stage until it
+became popular. On the contratry, it only provides the simple capability to
+download a segment of buffer(image) from host to on-die device, or vice versa.
+The image could be an algorithm, a flow table in the on-die SRAM, a firmware
+image running in the device, a function within a pipeline, etc. prgdev won't
+try to inteprete the content and it's transparent to prgdev. Apps and devices
+can communicate with a language they can understand each other, which is not
+provided by prgdev to decide what needs to be programmed.
+
+When the set of APIs is introduced, a general question is why we need it in
+DPDK community? Why we can't use offline tool to perform same thing? The answer
+is the prgdev provide a generic, online API to applications, in the meanwile,
+offers a capability to switch driver dynamically when downloaded image changed
+personality and a new driver is required. Comparing offline tool, it have online
+programmability (see below examples); Comparing online tool, it provides a
+generic API for many HWs; Comparing generic online tool/API for many products,
+it provides a capability to switch driver dynamically.
+
+There are various means to reach same goal, we'll try to find the best/better
+way to approach. All above advantages will help prgdev to be a 'better choice'.
+
+
+Scope
+-----
+
+The target devices include but not limited to programmable NIC, FPGA, GPU.
+Any devices, having the capability to store/load a piece of info to/from
+the deivce then changed hardware behavior, and applicable to prgdev programming
+model, could be registered as a prgdev device.
+
+The device can be programmed (dynamic) within DPDK or have been prior programmed
+(static) prior to a DPDK application being launched.
+
+Static example: In a static example a program, bitstream or other firmware
+would have been loaded prior to the start of a DPDK application. The prgdev
+would recognize the firmware loaded, and instantiate higher level interfaces
+(example : PMD, L3 forwarding, flow, etc.).
+
+Dynamic example: In a dynamic example, the programmability of the program,
+bitstream or other firmware would be loaded by the DPDK application upon start
+, or during execution. The prgdev would offer the interface to allow the DPDK
+application to program the device.
+
+
+Dyanamic bind/unbind
+--------------------
+
+Besides the simple API to upload/download image, the prgdev also introduces a
+mechanism internally to switch drivers and register/unregister device on the
+fly. With this mechanism, apps can use the programmable device, unbind other
+personalities on demand, then program it and bind it back with new personalities.
+Apps can follow below examples to simply complete the whole process.
+
+Note that bind/unbind actions are different concept from that a whole device
+attach/detach. For example, ``rte_eal_dev_detach()``, which will try to detach the
+drivers with device and remove the whole device from specific class of devices
+(ethdev, for example). Then, the whole device is invisible until a new 'probe'
+is activated.
+
+During the whole procedure of image upload/download, prgdev handler is always
+valid and apps can use it operate on programmable device. The reason why unbind
+is necessary is it may break the hardware when programming is in progress while
+other functions are active. Using programmble NIC as an example, it's a little
+impossible to receive/forward packets for hardware while updating firmware. In
+this case, apps need to detach ethdev function before firmware upgrading. Again,
+prgdev handler is still valid, it's function-level detach, different from
+device-level detach. After finishing image download, original function needs to
+attach back, either use same or different drivers, depends on personalities
+changed or not. For a programmble NIC, the driver may be the same. For FPGA, it
+may not, see below examples to get more details.
+
+Another reason to provide bind/unbind action is programmble devices, like FPGA,
+are not identified driver by 'vendor ID' and 'device ID', they might not be
+changed in all the ways, even FPGA is fully programmed. So, it depends on
+internal mechanism of FPGA, not 'vendor ID' and 'device ID' to identify proper
+drivers, either a register value or signature, depending on FPGA hardware
+design. In this case, EAL or other bus driver doesn't have the capability to
+identify proper driver for FPGA device. With prgdev introduced, while FPGA is
+always a prgdev, FPGA can use prgdev as primary driver, to find proper function
+driver.
+
+The bind/unbind actions are not trying to break the general initialization
+procedure in DPDK. prgdev still follow current EAL initializatio model to probe
+drivers for devices. See below examples for details.
+
+It's not true that bind/unbind is always necessary for programming. If device
+has a capability to perform specific functions and program in parallel, then
+apps needn't do bind/unbind actions. See 'dynamic program' section below.
+
+Authentication
+--------------
+
+Authentication and permissions for programming the device are outside the scope
+of this API. It is dependent upon a different mechanism not to be supported by
+DPDK. Application can utilize a different mechanism to authenticate user and
+or 'program' as validated for loading.
+
+Misc
+----
+
+PCI-E Vendor & Device ID are fixed in some devices such as the FPGA and GPU. The
+device can be loaded with a personality, and it would be up to this programmable
+device to load the appropriate "personality driver" based upon an identification
+mechanism to work with the loaded personality.
+
+
+prgdev API
+==========
+
+Below is the major APIs exposed to apps.
+
+- ``rte_prgdev_get_info``:
+  Acquire general info of the prgdev.
+- ``rte_prgdev_open``:
+  Open a prgdev for later programming. If failed, needs to call below 'unbind'
+  function to detach other functions except prgdev first.
+- ``rte_prgdev_img_download``:
+  Download a piece of buffer from host to the device after 'open' succeded.
+- ``rte_prgdev_img_upload``:
+  Upload a piece of information from the device to host.
+- ``rte_prgdev_check_status``:
+  Even download process is successful, it's necessary to check the status of the
+  downloaded image if worked in expected way by this function. Harware or driver
+  may not have the capability to check that status, so it's optional to driver.
+- ``rte_prgdev_close``:
+  After upload/download actions, close the prgdev.
+- ``rte_prgdev_bind``:
+  After 'close' action, app can call it to attach the 'new' personality with
+  proper drivers, then expose to app with new functions. prgdev driver will be
+  responsible for finding proper drivers to drive the functions. Desired driver
+  will be responsible for exposing the functions to apps.
+- ``rte_prgdev_unbind``:
+  When app are using the device, they need to call this function to unbind the
+  misc functions except prgdev with drivers prior to call 'open' function.
+
+
+Initialization procedure
+========================
+
+In this section, there 2 examples, FPGA and programmable NIC, used as examples
+to describe the initialization procedure on how programmable devices are probed
+, initialized and registered as different devices. In which, FPGA has 2
+different possible cases, blank and prior programmed FPGAs. Note that prgdev
+depends and follow on EAL initialization process to probe drivers for prgdev.
+
+Blank FPGA
+----------
+
+Below is the diagram on how Blank FPGA are probed and registered as a prgdev
+device that expose to apps.
+
+When PCI probe seeks driver for the FPGA device, FPGA drive will compare vendor
+ID and device ID, then perform HW initialization if both ID match. After then,
+it registered in prgdev abstract layer as a new device with a set of helper
+function ops.
+
+After that, apps will observe single device: prgdev device. Until now, the FPGA
+device is a simply device without any functionalities from application's
+perspective.
+
+In the following action, app will call ``prgdev bind`` function with the device
+ID of the FPGA device to bind the function within the FPGA to specific driver.
+FPGA driver will read AFU (Accelerated Function Unit) or signature from hardware
+and find potential function. Since it's a blank without any prior programming, a
+simply SUCCESS return::
+
+              Abstraction                 FPGA driver             Hardware
+                 layer
+                   +                       +                       +
+                   |                       |                       |
+                   +-----eal pci probe----->                       |
+                   |                       |                       | +------------+
+                   |                       |                       | |  FPGA dev  |
+                   |                       +--HW initialization----> |            |
+    +------------+ |                       |                       | |+----------+|
+    | prgdev dev | <--prgdev_ops_register--+                       | ||  AFU id  ||
+    +------------+ |                       |                       | |+----------+|
+                   |                       |                       | +------------+
+                   +-----prgdev bind------->                       |
+                   |                       |                       |
+                   |                       +--Read and parse AFU--->
+                   |                       |                       |
+                   |                       <--No personality found-+
+                   |                       |                       |
+                   <-------SUCCESS---------+                       |
+                   |                       |                       |
+                   +                       +                       +
+
+prior programmed FPGA
+---------------------
+
+For a prior programmed FPGA, all execution flow is similar to blank FPGA case,
+except the action of reading signature, as below flow table indicates. It will
+read valid functions, crypto and ethdev, as an example, then find proper driver
+to initialize the functions, then register as different ports.
+
+After initialization, app will observe 3 kinds of devices, a prgdev device, a
+ethdev device and a cryptodev device, which all comes from a single PCIE device::
+
+
+              Abstraction            FPGA driver               Hardware
+                 layer
+                   +                       +                       +
+                   |                       |                       |
+                   +----eal pci probe------>                       |
+                   |                       |                       | +-------------+
+                   |                       |                       | |  FPGA dev   |
+                   |                       +--HW initialization----> |             |
+    +------------+ |                       |                       | +-------------+
+    | prgdev dev | <--prgdev_ops_register--+                       | ||  AFU id   ||
+    +------------+ |                       |                       | |-------------|
+                   |                       |                       | +-------------+
+                   +-----prgdev bind------->                       |
+                   |                       |                       |
+                   |                       +--Read and parse AFU--->
+                   |                       |                       |
+                   |                       <--eth and crypto found-+
+    +------------+ |                       |                       |
+    | ethdev port| <---ethdev attach-------+                       |
+    +------------+ |                       |                       |
+                   |                       |                       |
+    +------------+ |                       |                       |
+    | crypto port| <---crypto attach-------+                       |
+    +------------+ |                       |                       |
+                   |                       |                       |
+                   +                       +                       +
+
+
+Programmable NIC
+----------------
+
+Comparing FPGA, programmable NIC has a nature of being a ethdev device. The
+initialization procedure is much more simpler than FPGA. Below is a flow table
+to describe this.
+
+The 'eal pci probe' procedure is similar, it registered itself as a prgdev after
+checking that the NIC is programmable. What is different from FPGA is it
+registered itself as a ethdev device in the following action due to the NIC
+nature.
+
+Then, app still follow the common process to call ``prgdev bind``, while the NIC
+driver does nothing and just return a SUCCESS::
+
+              Abstraction            NIC driver                Hardware
+                 layer
+                   +                       +                       +
+                   |                       |                       |
+                   +----eal pci probe------>                       |
+                   |                       |                       | +-------------+
+                   |                       +--HW initialization----> |Programmable |
+                   |                       |                       | |     NIC     |
+    +------------+ |                       +---Capability check----> +-------------+
+    | prgdev dev | <--prgdev_ops_register--+                       |
+    +------------+ |                       |                       |
+                   |                       |                       |
+    +------------+ |                       |                       |
+    | ethdev port| <---ethdev attach-------+                       |
+    +------------+ |                       |                       |
+                   |                       |                       |
+                   +-----prgdev bind------->                       |
+                   |                       |                       |
+                   |                       |                       |
+                   <--------SUCCESS--------+                       |
+                   |                       |                       |
+                   +                       +                       +
+
+
+Program procedure
+=================
+
+This section will describe how apps finish an image downloading procedure with
+the prgdev API. Similar to previous section, it will use several FPGA and NIC
+examples to show the common and different steps.
+
+Needs to point out, ``rte_prgdev_img_download`` function didn't carry the info
+of the image, or where downloading the image to. The interpretion info of the
+image can be hidden into the buffer itself.
+
+Common programming steps
+------------------------
+
+Before reaching details, it needs to introduce the common steps for apps to
+follow in order to download the image to the hardware.
+
+Before programming, apps need to call function ``rte_prgdev_open`` to open the
+programming interface. If failed, either functions residing in PCI-E device is
+in use or the PCI-E device is not ready, the programming steps are different.
+
+Below is the steps that 'open' succeeds:
+
+#. rte_prgdev_open --> SUCCESS
+#. rte_prgdev_img_download
+#. rte_prgdev_check_status
+#. rte_prgdev_close
+#. rte_prgdev_bind
+
+If failed to open the programming interface, apps need to follow below steps:
+
+#. rte_prgdev_open --> FAILURE
+#. Close all related functions
+#. ret_prgdev_unbind
+#. rte_prgdev_open --> SUCCESS
+#. rte_prgdev_img_download
+#. rte_prgdev_check_status
+#. rte_prgdev_close
+#. rte_prgdev_bind
+
+
+Detach functions
+----------------
+
+As discussed before, apps need to follow steps to detach all the other functions
+except prgdev when meeting error for 'open' operation. Below is a FPGA example
+with 'ethdev' and 'cryptodev' function attached.
+
+After meeting error, app needs to close ethdev port and cryptodev port first.
+Then, use 'prgdev unbind' to detach all the functions that registered previously
+. FPGA driver will call each 'detach' function for each registered device class
+respectively. After then, the 'ethdev' and 'cryptodev' ports will not be
+available::
+
+
+              Abstraction             FPGA driver              Hardware
+                 layer
+                   +                       +                       +
+    +------------+ |                       |                       |
+    | prgdev dev | +-------prgdev open----->                       |
+    |            | <-------ERR_IN_USE------+                       | +-------------+
+    +------------+ |                       |                       | |  FPGA dev   |
+                   |                       |                       | | +---------+ |
+    +------------+ |                       |                       | | | AFU id  | |
+    | ethdev port| +---rte_ethdev_close---->                       | | +---------+ |
+    |            |                         +------ethdev close-----> +-------------+
+    +------------+ <---------SUCCESS-------+                       |
+                   |                       |                       |
+    +------------+ |                       |                       |
+    | crypto port| +---crypto port close--->                       |
+    |            | |                       +---crypto func close--->
+    +------------+ <---------SUCCESS-------+                       |
+                   |                       |                       |
+                   +-----prgdev unbind----->                       |
+                   |                       |                       |
+    + - - - - - -+ |                       |                       |
+      ethdev port  <-----ethdev detach-----+                       |
+    +- - - - - - + |                       |                       |
+                   |                       |                       |
+    + - - - - - -+ |                       |                       |
+      crypto port  <----cryptodev detach---+                       |
+    +- - - - - - + |                       |                       |
+                   +                       +                       +
+
+Device programming
+------------------
+
+After 'open' prgdev sucessfully, apps follow 'image download', 'check status',
+'close' sequence respectively. After then, they need to 'bind' new functions
+back with 'prgdev bind' call.
+
+Still re-use previous FPGA example, below is the execution sequence. After image
+downloading and 'bind' action, 2 new 'virtio' and 'flow' functions are
+registered and 2 new ports are presented to apps::
+
+
+              Abstraction             FPGA driver              Hardware
+                 layer
+                   +                       +                       + +-------------+
+    +------------+ |                       |                       | |  FPGA dev   |
+    |            | +-----prgdev open------->                       | | +---------+ |
+    |            | <-------SUCCESS---------+                       | | | ethdev  | |
+    |            | |                       |                       | | +---------+ |
+    |            | +--prgdev_img_download-->                       | | +---------+ |
+    |            | |                       +----Download image-----> | | crypto  | |
+    |            | |                       <--------SUCCESS--------+ | +---------+ |
+    |            | <---------SUCCESS-------+                       | +-------------+
+    | prgdev dev | |                       |                       |        |
+    |            | +---prgdev_chk_status--->                       |        |
+    |            | |                       +--Verify image status-->        |
+    |            | |                       <--------SUCCESS--------+        |
+    |            | <--------SUCCESS--------+                       |        |
+    |            | +-------prgdev close---->                       |        v
+    |            | |                       |                       | +-------------+
+    |            | +-------prgdev bind----->                       | |  FPGA dev   |
+    +------------+ |                       |                       | | +---------+ |
+                   |                       <--Virtio and flow dev--+ | | virtio  | |
+    +------------+ |                       |                       | | +---------+ |
+    | ethdev port| <-----ethdev attach-----+                       | | +---------+ |
+    +------------+ |                       |                       | | |Flow dev | |
+                   |                       |                       | | +---------+ |
+    +------------+ |                       |                       | +-------------+
+    | Flow port  | <-----Flow dev attach---+                       |
+    +------------+ |                       |                       |
+                   +                       +                       +
+
+
+Dyanamic program
+----------------
+
+If prgdev device has a capability to be programmed while functions are in use,
+then we say it supports dynamic program. It's possible to update a profile while
+device is used as a ethdev device.
+
+The device can claim this capability by set a DYNAMIC_PROGRAM flag in function
+``rte_prgdev_get_info``.
+
+The dynamic programming procedure is the same as previous section shows, while
+ignore the steps to 'close' and 'unbind' actions when device is in use.
+
+Note that prgdev driver needs to consider if it can support dynamic program to
+avoid any hardware or software error.
+
+
+Source code layout
+==================
+
+Below is the suggested source code layout.
+
+- 'prg NIC' under 'net' is the PMD NIC directory that support prgdev. This
+  doesn't change programmable NIC's directory and remains the same. The PMD
+  driver should include 2 parts, one part is the code to support rte_ethdev API,
+  another part is the code to support rte_prgdev API.
+- 'librte_prgdev' under 'lib' is the new directory store definition and
+  implementation of the generic prgdev API.
+- 'prg' under 'drivers' is the directory to store specfic implementation
+  driver to support rte_prgdev. Each sub-directory could be a prgdev driver to
+  support specific hardware product.
+
+::
+
+                    +---------+                         +---------+
+                    | drivers |                         |   lib   |
+                    +---------+                         +----+----+
+                         |                                   |
+              +----------+-----------+                       |
+              |                      |                       |
+          +---v---+              +---v----+          +-------v-------+
+          |  net  |              |   prg  |          | librte_prgdev |
+          +---+---+              +---+----+          +---------------+
+              |                      |
+        +-----+----+           +-----+----+
+        |          |           |          |
+    +---v----+ +---v----+  +---v----+ +---v----+
+    |  i40e  | | prg NIC|  |FPGA ABC| |GPU XYZ |
+    +--------+ +--------+  +--------+ +--------+
-- 
1.7.7.6

Re: [PATCH 6/6] doc: introduction to prgdev

[Thomas Monjalon]

2017-03-02 12:03, Chen Jing D:
> +Overview
> +========

I think the first review pass of this series must be focused on the
overview you give here (thanks for the detailed explanations).

I'll try to sum up while commenting.

[...]
The target is programmable devices.

> +The major purpose of prgdev is to help DPDK users to load/upgrade RTL images
> +for FPGA devices, or upgrade firmware for programmble NICs.

This is a control plane feature.
You want to have it in DPDK to allow dynamic programmation while running
the device, right?

[...]
> +When the set of APIs is introduced, a general question is why we need it in
> +DPDK community?

Good question :)

[...]
> +Any devices, having the capability to store/load a piece of info to/from
> +the deivce then changed hardware behavior, and applicable to prgdev programming
> +model, could be registered as a prgdev device.
> +
> +The device can be programmed (dynamic) within DPDK or have been prior programmed
> +(static) prior to a DPDK application being launched.
[...]
> +Besides the simple API to upload/download image, the prgdev also introduces a
> +mechanism internally to switch drivers and register/unregister device on the
> +fly. With this mechanism, apps can use the programmable device, unbind other
> +personalities on demand, then program it and bind it back with new personalities.
> +Apps can follow below examples to simply complete the whole process.

I disagree about the specific bind/unbind for prgdev.
We must improve binding inside DPDK for every devices.

> +Note that bind/unbind actions are different concept from that a whole device
> +attach/detach. For example, ``rte_eal_dev_detach()``, which will try to detach the
> +drivers with device and remove the whole device from specific class of devices
> +(ethdev, for example). Then, the whole device is invisible until a new 'probe'
> +is activated.

I do not understand.

> +During the whole procedure of image upload/download, prgdev handler is always
> +valid and apps can use it operate on programmable device. The reason why unbind
> +is necessary is it may break the hardware when programming is in progress while
> +other functions are active. Using programmble NIC as an example, it's a little
> +impossible to receive/forward packets for hardware while updating firmware. In
> +this case, apps need to detach ethdev function before firmware upgrading. Again,
> +prgdev handler is still valid, it's function-level detach, different from
> +device-level detach. After finishing image download, original function needs to
> +attach back, either use same or different drivers, depends on personalities
> +changed or not. For a programmble NIC, the driver may be the same. For FPGA, it
> +may not, see below examples to get more details.

If the personality of the device is changed, it must be seen as a new device
from e.g. the ethdev point of view, while keeping the same prgdev device.
In other words, a device can have several interfaces at the same time
(ethdev, cryptodev, eventdev, prgdev, whatever).
I think we can dynamically create/destroy some interfaces while keeping track
of the underlying device.

> +Another reason to provide bind/unbind action is programmble devices, like FPGA,
> +are not identified driver by 'vendor ID' and 'device ID', they might not be
> +changed in all the ways, even FPGA is fully programmed. So, it depends on
> +internal mechanism of FPGA, not 'vendor ID' and 'device ID' to identify proper
> +drivers, either a register value or signature, depending on FPGA hardware
> +design. In this case, EAL or other bus driver doesn't have the capability to
> +identify proper driver for FPGA device. With prgdev introduced, while FPGA is
> +always a prgdev, FPGA can use prgdev as primary driver, to find proper function
> +driver.

You mean prgdev should help the bus layer to map the right driver interface?
It looks weird and dangerous. The standard way to identify a PCI device is
to look at its IDs. Other unknown methods must be strongly discussed.

Re: [PATCH 6/6] doc: introduction to prgdev

[Chen, Jing D]

Hi, Thomas,

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> Sent: Monday, March 6, 2017 11:27 PM
> To: Chen, Jing D <jing.d.chen@intel.com>
> Cc: dev@dpdk.org; Liang, Cunming <cunming.liang@intel.com>; Rogers,
> Gerald <gerald.rogers@intel.com>; Wiles, Keith <keith.wiles@intel.com>;
> Richardson, Bruce <bruce.richardson@intel.com>; Mcnamara, John
> <john.mcnamara@intel.com>
> Subject: Re: [dpdk-dev] [PATCH 6/6] doc: introduction to prgdev
> 
> 2017-03-02 12:03, Chen Jing D:
> > +Overview
> > +========
> 
> I think the first review pass of this series must be focused on the overview
> you give here (thanks for the detailed explanations).
> 
> I'll try to sum up while commenting.
> 
> [...]
> The target is programmable devices.
> 
> > +The major purpose of prgdev is to help DPDK users to load/upgrade RTL
> > +images for FPGA devices, or upgrade firmware for programmble NICs.
> 
> This is a control plane feature.
> You want to have it in DPDK to allow dynamic programmation while running
> the device, right?

Exactly right.

> 
> [...]
> > +When the set of APIs is introduced, a general question is why we need
> > +it in DPDK community?
> 
> Good question :)
> 
> [...]
> > +Any devices, having the capability to store/load a piece of info
> > +to/from the deivce then changed hardware behavior, and applicable to
> > +prgdev programming model, could be registered as a prgdev device.
> > +
> > +The device can be programmed (dynamic) within DPDK or have been prior
> > +programmed
> > +(static) prior to a DPDK application being launched.
> [...]
> > +Besides the simple API to upload/download image, the prgdev also
> > +introduces a mechanism internally to switch drivers and
> > +register/unregister device on the fly. With this mechanism, apps can
> > +use the programmable device, unbind other personalities on demand,
> then program it and bind it back with new personalities.
> > +Apps can follow below examples to simply complete the whole process.
> 
> I disagree about the specific bind/unbind for prgdev.
> We must improve binding inside DPDK for every devices.

First of all, let us try to imagine what is the safe status for device before apps
intending to download some image to it - apps wouldn't operate on the device, 
any behaviors like configuring registers, receive/transmit data may impair the
device or make the download failed. 
Following first answer to prevent app accessing device during image
downloading, how can we achieve that? Detach drivers with device is a smart
idea, right? But the problem is how can apps use prgdev API to download image
after all drivers detached with the device?
So, the final question is how can we just detached others driver except prgdev
one? I can't find answer in current DPDK framework, that's why I'd like to introduce
bind/unbind functions to detach other drivers from the device.

I'm open to this problem. If any suggestion or mechanism can help on it, I can
remove these 2.

BTW, not all devices or image download actions need the device to detach the 
device, depending on hardware implementation.

> 
> > +Note that bind/unbind actions are different concept from that a whole
> > +device attach/detach. For example, ``rte_eal_dev_detach()``, which
> > +will try to detach the drivers with device and remove the whole
> > +device from specific class of devices (ethdev, for example). Then, the
> whole device is invisible until a new 'probe'
> > +is activated.
> 
> I do not understand.

See above explanations.

> 
> > +During the whole procedure of image upload/download, prgdev handler
> > +is always valid and apps can use it operate on programmable device.
> > +The reason why unbind is necessary is it may break the hardware when
> > +programming is in progress while other functions are active. Using
> > +programmble NIC as an example, it's a little impossible to
> > +receive/forward packets for hardware while updating firmware. In this
> > +case, apps need to detach ethdev function before firmware upgrading.
> > +Again, prgdev handler is still valid, it's function-level detach,
> > +different from device-level detach. After finishing image download,
> > +original function needs to attach back, either use same or different
> > +drivers, depends on personalities changed or not. For a programmble NIC,
> the driver may be the same. For FPGA, it may not, see below examples to get
> more details.
> 
> If the personality of the device is changed, it must be seen as a new device
> from e.g. the ethdev point of view, while keeping the same prgdev device.
> In other words, a device can have several interfaces at the same time (ethdev,
> cryptodev, eventdev, prgdev, whatever).
> I think we can dynamically create/destroy some interfaces while keeping
> track of the underlying device.

Fully agree. But current PCI device with 'vendor_id' and 'device_id' ony can
be bind to single driver at a time. So I add 'rte_prgdev_allocate/release' to
support register/unregister prgdev dynamically without BDF.

> 
> > +Another reason to provide bind/unbind action is programmble devices,
> > +like FPGA, are not identified driver by 'vendor ID' and 'device ID',
> > +they might not be changed in all the ways, even FPGA is fully
> > +programmed. So, it depends on internal mechanism of FPGA, not 'vendor
> > +ID' and 'device ID' to identify proper drivers, either a register
> > +value or signature, depending on FPGA hardware design. In this case,
> > +EAL or other bus driver doesn't have the capability to identify
> > +proper driver for FPGA device. With prgdev introduced, while FPGA is
> > +always a prgdev, FPGA can use prgdev as primary driver, to find proper
> function driver.
> 
> You mean prgdev should help the bus layer to map the right driver interface?
> It looks weird and dangerous. The standard way to identify a PCI device is to
> look at its IDs. Other unknown methods must be strongly discussed.

For programmable Ethernet device, it's not truce. But for FPGA, it's. When FPGA
is produced, the device ID indicate what model it is and won't be changed
anyway, even being reprogrammed. It used some not-generic mechanism, like
AFU id to distinguish the personalities. So, for FPGA, a prgdev driver can be used
as primary driver to identify personalities and then register to specific devices.

Re: [PATCH 6/6] doc: introduction to prgdev

[Bruce Richardson]

On Tue, Mar 07, 2017 at 10:34:06AM +0000, Chen, Jing D wrote:
> Hi, Thomas,
> 
> > -----Original Message-----
> > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > Sent: Monday, March 6, 2017 11:27 PM
> > To: Chen, Jing D <jing.d.chen@intel.com>
> > Cc: dev@dpdk.org; Liang, Cunming <cunming.liang@intel.com>; Rogers,
> > Gerald <gerald.rogers@intel.com>; Wiles, Keith <keith.wiles@intel.com>;
> > Richardson, Bruce <bruce.richardson@intel.com>; Mcnamara, John
> > <john.mcnamara@intel.com>
> > Subject: Re: [dpdk-dev] [PATCH 6/6] doc: introduction to prgdev
> > 
> > 2017-03-02 12:03, Chen Jing D:
> > > +Overview
> > > +========
> > 
> > I think the first review pass of this series must be focused on the overview
> > you give here (thanks for the detailed explanations).
> > 
> > I'll try to sum up while commenting.
> > 
> > [...]
> > The target is programmable devices.
> > 
> > > +The major purpose of prgdev is to help DPDK users to load/upgrade RTL
> > > +images for FPGA devices, or upgrade firmware for programmble NICs.
> > 
> > This is a control plane feature.
> > You want to have it in DPDK to allow dynamic programmation while running
> > the device, right?
> 
> Exactly right.
> 
> > 
> > [...]
> > > +When the set of APIs is introduced, a general question is why we need
> > > +it in DPDK community?
> > 
> > Good question :)
> > 
> > [...]
> > > +Any devices, having the capability to store/load a piece of info
> > > +to/from the deivce then changed hardware behavior, and applicable to
> > > +prgdev programming model, could be registered as a prgdev device.
> > > +
> > > +The device can be programmed (dynamic) within DPDK or have been prior
> > > +programmed
> > > +(static) prior to a DPDK application being launched.
> > [...]
> > > +Besides the simple API to upload/download image, the prgdev also
> > > +introduces a mechanism internally to switch drivers and
> > > +register/unregister device on the fly. With this mechanism, apps can
> > > +use the programmable device, unbind other personalities on demand,
> > then program it and bind it back with new personalities.
> > > +Apps can follow below examples to simply complete the whole process.
> > 
> > I disagree about the specific bind/unbind for prgdev.
> > We must improve binding inside DPDK for every devices.
> 
> First of all, let us try to imagine what is the safe status for device before apps
> intending to download some image to it - apps wouldn't operate on the device, 
> any behaviors like configuring registers, receive/transmit data may impair the
> device or make the download failed. 
> Following first answer to prevent app accessing device during image
> downloading, how can we achieve that? Detach drivers with device is a smart
> idea, right? But the problem is how can apps use prgdev API to download image
> after all drivers detached with the device?
> So, the final question is how can we just detached others driver except prgdev
> one? I can't find answer in current DPDK framework, that's why I'd like to introduce
> bind/unbind functions to detach other drivers from the device.
> 
> I'm open to this problem. If any suggestion or mechanism can help on it, I can
> remove these 2.
> 
> BTW, not all devices or image download actions need the device to detach the 
> device, depending on hardware implementation.
> 
> > 
> > > +Note that bind/unbind actions are different concept from that a whole
> > > +device attach/detach. For example, ``rte_eal_dev_detach()``, which
> > > +will try to detach the drivers with device and remove the whole
> > > +device from specific class of devices (ethdev, for example). Then, the
> > whole device is invisible until a new 'probe'
> > > +is activated.
> > 
> > I do not understand.
> 
> See above explanations.
> 
> > 
> > > +During the whole procedure of image upload/download, prgdev handler
> > > +is always valid and apps can use it operate on programmable device.
> > > +The reason why unbind is necessary is it may break the hardware when
> > > +programming is in progress while other functions are active. Using
> > > +programmble NIC as an example, it's a little impossible to
> > > +receive/forward packets for hardware while updating firmware. In this
> > > +case, apps need to detach ethdev function before firmware upgrading.
> > > +Again, prgdev handler is still valid, it's function-level detach,
> > > +different from device-level detach. After finishing image download,
> > > +original function needs to attach back, either use same or different
> > > +drivers, depends on personalities changed or not. For a programmble NIC,
> > the driver may be the same. For FPGA, it may not, see below examples to get
> > more details.
> > 
> > If the personality of the device is changed, it must be seen as a new device
> > from e.g. the ethdev point of view, while keeping the same prgdev device.
> > In other words, a device can have several interfaces at the same time (ethdev,
> > cryptodev, eventdev, prgdev, whatever).
> > I think we can dynamically create/destroy some interfaces while keeping
> > track of the underlying device.
> 
> Fully agree. But current PCI device with 'vendor_id' and 'device_id' ony can
> be bind to single driver at a time. So I add 'rte_prgdev_allocate/release' to
> support register/unregister prgdev dynamically without BDF.
> 
> > 
> > > +Another reason to provide bind/unbind action is programmble devices,
> > > +like FPGA, are not identified driver by 'vendor ID' and 'device ID',
> > > +they might not be changed in all the ways, even FPGA is fully
> > > +programmed. So, it depends on internal mechanism of FPGA, not 'vendor
> > > +ID' and 'device ID' to identify proper drivers, either a register
> > > +value or signature, depending on FPGA hardware design. In this case,
> > > +EAL or other bus driver doesn't have the capability to identify
> > > +proper driver for FPGA device. With prgdev introduced, while FPGA is
> > > +always a prgdev, FPGA can use prgdev as primary driver, to find proper
> > function driver.
> > 
> > You mean prgdev should help the bus layer to map the right driver interface?
> > It looks weird and dangerous. The standard way to identify a PCI device is to
> > look at its IDs. Other unknown methods must be strongly discussed.
> 
> For programmable Ethernet device, it's not truce. But for FPGA, it's. When FPGA
> is produced, the device ID indicate what model it is and won't be changed
> anyway, even being reprogrammed. It used some not-generic mechanism, like
> AFU id to distinguish the personalities. So, for FPGA, a prgdev driver can be used
> as primary driver to identify personalities and then register to specific devices.

Sounds like we would need an FPGA bus driver in that case. I think that
would be a better solution than having a specific device driver loading
other drivers.

/Bruce

Re: [PATCH 6/6] doc: introduction to prgdev

[Thomas Monjalon]

2017-03-07 11:12, Bruce Richardson:
> On Tue, Mar 07, 2017 at 10:34:06AM +0000, Chen, Jing D wrote:
> > From: Thomas Monjalon
> > > > +Any devices, having the capability to store/load a piece of info
> > > > +to/from the deivce then changed hardware behavior, and applicable to
> > > > +prgdev programming model, could be registered as a prgdev device.
> > > > +
> > > > +The device can be programmed (dynamic) within DPDK or have been prior
> > > > +programmed
> > > > +(static) prior to a DPDK application being launched.
> > > [...]
> > > > +Besides the simple API to upload/download image, the prgdev also
> > > > +introduces a mechanism internally to switch drivers and
> > > > +register/unregister device on the fly. With this mechanism, apps can
> > > > +use the programmable device, unbind other personalities on demand,
> > > then program it and bind it back with new personalities.
> > > > +Apps can follow below examples to simply complete the whole process.
> > > 
> > > I disagree about the specific bind/unbind for prgdev.
> > > We must improve binding inside DPDK for every devices.
> > 
> > First of all, let us try to imagine what is the safe status for device before apps
> > intending to download some image to it - apps wouldn't operate on the device, 
> > any behaviors like configuring registers, receive/transmit data may impair the
> > device or make the download failed. 
> > Following first answer to prevent app accessing device during image
> > downloading, how can we achieve that? Detach drivers with device is a smart
> > idea, right? But the problem is how can apps use prgdev API to download image
> > after all drivers detached with the device?
> > So, the final question is how can we just detached others driver except prgdev
> > one? I can't find answer in current DPDK framework, that's why I'd like to introduce
> > bind/unbind functions to detach other drivers from the device.
> > 
> > I'm open to this problem. If any suggestion or mechanism can help on it, I can
> > remove these 2.
> > 
> > BTW, not all devices or image download actions need the device to detach the 
> > device, depending on hardware implementation.

Thanks for the explanations.
It sounds interesting.

> > > > +Another reason to provide bind/unbind action is programmble devices,
> > > > +like FPGA, are not identified driver by 'vendor ID' and 'device ID',
> > > > +they might not be changed in all the ways, even FPGA is fully
> > > > +programmed. So, it depends on internal mechanism of FPGA, not 'vendor
> > > > +ID' and 'device ID' to identify proper drivers, either a register
> > > > +value or signature, depending on FPGA hardware design. In this case,
> > > > +EAL or other bus driver doesn't have the capability to identify
> > > > +proper driver for FPGA device. With prgdev introduced, while FPGA is
> > > > +always a prgdev, FPGA can use prgdev as primary driver, to find proper
> > > function driver.
> > > 
> > > You mean prgdev should help the bus layer to map the right driver interface?
> > > It looks weird and dangerous. The standard way to identify a PCI device is to
> > > look at its IDs. Other unknown methods must be strongly discussed.
> > 
> > For programmable Ethernet device, it's not truce. But for FPGA, it's. When FPGA
> > is produced, the device ID indicate what model it is and won't be changed
> > anyway, even being reprogrammed. It used some not-generic mechanism, like
> > AFU id to distinguish the personalities. So, for FPGA, a prgdev driver can be used
> > as primary driver to identify personalities and then register to specific devices.
> 
> Sounds like we would need an FPGA bus driver in that case. I think that
> would be a better solution than having a specific device driver loading
> other drivers.

Good suggestion. What do you think of a FPGA bus to consider more parameters
than the PCI BDF when matching a device driver?

How other systems manage the drivers for a FPGA-programmed device?

Re: [PATCH 6/6] doc: introduction to prgdev

[Chen, Jing D]

> > >
> > > > +Another reason to provide bind/unbind action is programmble
> > > > +devices, like FPGA, are not identified driver by 'vendor ID' and
> > > > +'device ID', they might not be changed in all the ways, even FPGA
> > > > +is fully programmed. So, it depends on internal mechanism of
> > > > +FPGA, not 'vendor ID' and 'device ID' to identify proper drivers,
> > > > +either a register value or signature, depending on FPGA hardware
> > > > +design. In this case, EAL or other bus driver doesn't have the
> > > > +capability to identify proper driver for FPGA device. With prgdev
> > > > +introduced, while FPGA is always a prgdev, FPGA can use prgdev as
> > > > +primary driver, to find proper
> > > function driver.
> > >
> > > You mean prgdev should help the bus layer to map the right driver
> interface?
> > > It looks weird and dangerous. The standard way to identify a PCI
> > > device is to look at its IDs. Other unknown methods must be strongly
> discussed.
> >
> > For programmable Ethernet device, it's not truce. But for FPGA, it's.
> > When FPGA is produced, the device ID indicate what model it is and
> > won't be changed anyway, even being reprogrammed. It used some
> > not-generic mechanism, like AFU id to distinguish the personalities.
> > So, for FPGA, a prgdev driver can be used as primary driver to identify
> personalities and then register to specific devices.
> 
> Sounds like we would need an FPGA bus driver in that case. I think that would
> be a better solution than having a specific device driver loading other drivers.
>

I don't object to introduce a pseudo bus for FPGA, but it's a matter of work that
FPGA driver needs to consider, not in scope of prgdev.
Besides that, I can see DPDK EAL will do other bus probe first, then PCI bus
probe, which is not friendly to introduce pseudo bus for an actual PCI device.
 
> /Bruce