* [RFC 0/2] Prgdev API
@ 2017-01-20 3:21 Chen Jing D(Mark)
2017-01-20 3:21 ` [RFC 1/2] doc: introduction to prgdev Chen Jing D(Mark)
2017-01-20 3:21 ` [RFC 2/2] prgdev: introduce generic prgdev API Chen Jing D(Mark)
0 siblings, 2 replies; 10+ messages in thread
From: Chen Jing D(Mark) @ 2017-01-20 3:21 UTC (permalink / raw)
To: dev; +Cc: keith.wiles, gerald.rogers, Chen Jing D(Mark)
This RFC patch intends to introduce an new abstraction layer API
called prgdev to program programmable devices. Please refer to doc
and API for details.
Chen Jing D(Mark) (2):
doc: introduction to prgdev
prgdev: introduce generic prgdev API
doc/guides/prog_guide/prgdev_lib.rst | 457 ++++++++++++++++++++++++++++++++++
lib/librte_prgdev/rte_prgdev.h | 242 ++++++++++++++++++
2 files changed, 699 insertions(+), 0 deletions(-)
create mode 100644 doc/guides/prog_guide/prgdev_lib.rst
create mode 100644 lib/librte_prgdev/rte_prgdev.h
--
1.7.7.6
^ permalink raw reply [flat|nested] 10+ messages in thread
* [RFC 1/2] doc: introduction to prgdev
2017-01-20 3:21 [RFC 0/2] Prgdev API Chen Jing D(Mark)
@ 2017-01-20 3:21 ` Chen Jing D(Mark)
2017-02-01 11:41 ` Jan Blunck
2017-02-08 3:30 ` Liang, Cunming
2017-01-20 3:21 ` [RFC 2/2] prgdev: introduce generic prgdev API Chen Jing D(Mark)
1 sibling, 2 replies; 10+ messages in thread
From: Chen Jing D(Mark) @ 2017-01-20 3:21 UTC (permalink / raw)
To: dev; +Cc: keith.wiles, gerald.rogers, Chen Jing D(Mark)
This is the documentation to describe what prgdev is, how to use
prgdev API and accomplish an image download.
Signed-off-by: Chen Jing D(Mark) <jing.d.chen@intel.com>
---
doc/guides/prog_guide/prgdev_lib.rst | 457 ++++++++++++++++++++++++++++++++++
1 files changed, 457 insertions(+), 0 deletions(-)
create mode 100644 doc/guides/prog_guide/prgdev_lib.rst
diff --git a/doc/guides/prog_guide/prgdev_lib.rst b/doc/guides/prog_guide/prgdev_lib.rst
new file mode 100644
index 0000000..3917c18
--- /dev/null
+++ b/doc/guides/prog_guide/prgdev_lib.rst
@@ -0,0 +1,457 @@
+Overview
+========
+
+More and more devices start to support on-die programming, which include
+NIC, GPU, FPGA, etc. This capability has a requirement to drivers that
+introduce a API for application to download/upload image to/from the HW.
+So, it's necessary to provide a generic API to perform image download, upload,
+validation, etc.
+
+This RFC patch intends to introduce a DPDK generic programming device layer,
+called prgdev below, to provide an abstract, generic APIs for applications to
+program hardware without knowing the details of programmable devices. From
+driver's perspective, they'll try to adapt their functions to the abstract
+APIs defined in prgdev.
+
+The major purpose of prgdev is to help DPDK users to load/upgrade RTL images
+for FPGA devices, or upgrade firmware for programmble NICs.
+
+But those devices that have the capability to do on-die programming may
+expose versatile talents to apps. Add/delete/update entries in flow table,
+update/overwrite the firmware/microcode, add a new algorithm, update profiles
+, etc, which is hard to be abstracted as generic behaviors. So, prgdev won't
+include the APIs to perform device-unique actions in current stage until it
+became popular. On the contratry, it only provides the simple capability to
+download a segment of buffer(image) from host to on-die device, or vice versa.
+The image could be an algorithm, a flow table in the on-die SRAM, a firmware
+image running in the device, a function within a pipeline, etc. prgdev won't
+try to inteprete the content and it's transparent to prgdev. Apps and devices
+can communicate with a language they can understand each other, which is not
+provided by prgdev to decide what needs to be programmed.
+
+When the set of APIs is introduced, a general question is why we need it in
+DPDK community? Why we can't use offline tool to perform same thing? The answer
+is the prgdev provide a generic, online API to applications, in the meanwile,
+offers a capability to switch driver dynamically when downloaded image changed
+personality and a new driver is required. Comparing offline tool, it have online
+programmability (see below examples); Comparing online tool, it provides a
+generic API for many HWs; Comparing generic online tool/API for many products,
+it provides a capability to switch driver dynamically.
+
+There are various means to reach same goal, we'll try to find the best/better
+way to approach. All above advantages will help prgdev to be a 'better choice'.
+
+
+Scope
+-----
+
+The target devices include but not limited to programmable NIC, FPGA, GPU.
+Any devices, having the capability to store/load a piece of info to/from
+the deivce then changed hardware behavior, and applicable to prgdev programming
+model, could be registered as a prgdev device.
+
+The device can be programmed (dynamic) within DPDK or have been prior programmed
+(static) prior to a DPDK application being launched.
+
+Static example: In a static example a program, bitstream or other firmware
+would have been loaded prior to the start of a DPDK application. The prgdev
+would recognize the firmware loaded, and instantiate higher level interfaces
+(example : PMD, L3 forwarding, flow, etc.).
+
+Dynamic example: In a dynamic example, the programmability of the program,
+bitstream or other firmware would be loaded by the DPDK application upon start
+, or during execution. The prgdev would offer the interface to allow the DPDK
+application to program the device.
+
+Dyanamic bind/unbind
+--------------------
+
+Besides the simple API to upload/download image, the prgdev also introduces a
+mechanism internally to switch drivers and register/unregister device on the
+fly. With this mechanism, apps can use the programmable device, unbind other
+personalities on demand, then program it and bind it back with new personalities
+. Apps can follow below examples to simply complete the whole process.
+
+Note that bind/unbind actions are different concept from that a whole device
+attach/detach. For example, rte_eal_dev_detach(), which will try to detach the
+drivers with device and remove the whole device from specific class of devices
+(ethdev, for example). Then, the whole device is invisible until a new 'probe'
+is activated.
+
+During the whole procedure of image upload/download, prgdev handler is always
+valid and apps can use it operate on programmable device. The reason why unbind
+is necessary is it may break the hardware when programming is in progress while
+other functions are active. Using programmble NIC as an example, it's a little
+impossible to receive/forward packets for hardware while updating firmware. In
+this case, apps need to detach ethdev function before firmware upgrading. Again,
+prgdev handler is still valid, it's function-level detach, different from
+device-level detach. After finishing image download, original function needs to
+attach back, either use same or different drivers, depends on personalities
+changed or not. For a programmble NIC, the driver may be the same. For FPGA, it
+may not, see below examples to get more details.
+
+Another reason to provide bind/unbind action is programmble devices, like FPGA,
+are not identified driver by 'vendor ID' and 'device ID', they might not be
+changed in all the ways, even FPGA is fully programmed. So, it depends on
+internal mechanism of FPGA, not 'vendor ID' and 'device ID' to identify proper
+drivers, either a register value or signature, depending on FPGA hardware
+design. In this case, EAL or other bus driver doesn't have the capability to
+identify proper driver for FPGA device. With prgdev introduced, while FPGA is
+always a prgdev, FPGA can use prgdev as primary driver, to find proper function
+driver.
+
+The bind/unbind actions are not trying to break the general initialization
+procedure in DPDK. prgdev still follow current EAL initializatio model to probe
+drivers for devices. See below examples for details.
+
+It's not true that bind/unbind is always necessary for programming. If device
+has a capability to perform specific functions and program in parallel, then
+apps needn't do bind/unbind actions. See 'dynamic program' section below.
+
+Authentication
+--------------
+
+Authentication and permissions for programming the device are outside the scope
+of this API. It is dependent upon a different mechanism not to be supported by
+DPDK. Application can utilize a different mechanism to authenticate user and
+or 'program' as validated for loading.
+
+Misc
+----
+
+PCI-E Vendor & Device ID are fixed in some devices such as the FPGA and GPU. The
+device can be loaded with a personality, and it would be up to this programmable
+device to load the appropriate "personality driver" based upon an identification
+mechanism to work with the loaded personality.
+
+
+prgdev API
+==========
+
+Below is the major APIs exposed to apps.
+
+- rte_prgdev_get_info
+ Acquire general info of the prgdev.
+- rte_prgdev_open
+ Open a prgdev for later programming. If failed, needs to call below 'unbind'
+ function to detach other functions except prgdev first.
+- rte_prgdev_img_download
+ Download a piece of buffer from host to the device after 'open' succeded.
+- rte_prgdev_img_upload
+ Upload a piece of information from the device to host.
+- rte_prgdev_check_status
+ Even download process is successful, it's necessary to check the status of the
+ downloaded image if worked in expected way by this function. Harware or driver
+ may not have the capability to check that status, so it's optional to driver.
+- rte_prgdev_close
+ After upload/download actions, close the prgdev.
+- rte_prgdev_bind
+ After 'close' action, app can call it to attach the 'new' personality with
+ proper drivers, then expose to app with new functions. prgdev driver will be
+ responsible for finding proper drivers to drive the functions. Desired driver
+ will be responsible for exposing the functions to apps.
+- rte_prgdev_unbind
+ When app are using the device, they need to call this function to unbind the
+ misc functions except prgdev with drivers prior to call 'open' function.
+
+Initialization procedure
+========================
+In this section, there 2 examples, FPGA and programmable NIC, used as examples
+to describe the initialization procedure on how programmable devices are probed
+, initialized and registered as different devices. In which, FPGA has 2
+different possible cases, blank and prior programmed FPGAs. Note that prgdev
+depends and follow on EAL initialization process to probe drivers for prgdev.
+
+Blank FPGA
+----------
+Below is the diagram on how Blank FPGA are probed and registered as a prgdev
+device that expose to apps.
+
+When PCI probe seeks driver for the FPGA device, FPGA drive will compare vendor
+ID and device ID, then perform HW initialization if both ID match. After then,
+it registered in prgdev abstract layer as a new device with a set of helper
+function ops.
+
+After that, apps will observe single device: prgdev device. Until now, the FPGA
+device is a simply device without any functionalities from application's
+perspective.
+
+In the following action, app will call 'prgdev bind' function with the device
+ID of the FPGA device to bind the function within the FPGA to specific driver.
+FPGA driver will read AFU (Accelerated Function Unit) or signature from hardware
+and find potential function. Since it's a blank without any prior programming, a
+simply SUCCESS return.
+
+ Abstraction FPGA driver Hardware
+ layer
+ + + +
+ | | |
+ +-----eal pci probe-----> |
+ | | | +------------+
+ | | | | FPGA dev |
+ | +--HW initialization----> | |
++------------+ | | | |+----------+|
+| prgdev dev | <--prgdev_ops_register--+ | || AFU id ||
++------------+ | | | |+----------+|
+ | | | +------------+
+ +-----prgdev bind-------> |
+ | | |
+ | +--Read and parse AFU--->
+ | | |
+ | <--No personality found-+
+ | | |
+ <-------SUCCESS---------+ |
+ | | |
+ + + +
+
+prior programmed FPGA
+---------------------
+
+For a prior programmed FPGA, all execution flow is similar to blank FPGA case,
+except the action of reading signature, as below flow table indicates. It will
+read valid functions, crypto and ethdev, as an example, then find proper driver
+to initialize the functions, then register as different ports.
+
+After initialization, app will observe 3 kinds of devices, a prgdev device, a
+ethdev device and a cryptodev device, which all comes from a single PCIE device.
+
+
+ Abstraction FPGA driver Hardware
+ layer
+ + + +
+ | | |
+ +----eal pci probe------> |
+ | | | +-------------+
+ | | | | FPGA dev |
+ | +--HW initialization----> | |
++------------+ | | | +-------------+
+| prgdev dev | <--prgdev_ops_register--+ | || AFU id ||
++------------+ | | | |-------------|
+ | | | +-------------+
+ +-----prgdev bind-------> |
+ | | |
+ | +--Read and parse AFU--->
+ | | |
+ | <--eth and crypto found-+
++------------+ | | |
+| ethdev port| <---ethdev attach-------+ |
++------------+ | | |
+ | | |
++------------+ | | |
+| crypto port| <---crypto attach-------+ |
++------------+ | | |
+ | | |
+ + + +
+
+
+Programmable NIC
+----------------
+
+Comparing FPGA, programmable NIC has a nature of being a ethdev device. The
+initialization procedure is much more simpler than FPGA. Below is a flow table
+to describe this.
+
+The 'eal pci probe' procedure is similar, it registered itself as a prgdev after
+checking that the NIC is programmable. What is different from FPGA is it
+registered itself as a ethdev device in the following action due to the NIC
+nature.
+
+Then, app still follow the common process to call 'prgdev bind', while the NIC
+driver does nothing and just return a SUCCESS.
+
+ Abstraction NIC driver Hardware
+ layer
+ + + +
+ | | |
+ +----eal pci probe------> |
+ | | | +-------------+
+ | +--HW initialization----> |Programmable |
+ | | | | NIC |
++------------+ | +---Capability check----> +-------------+
+| prgdev dev | <--prgdev_ops_register--+ |
++------------+ | | |
+ | | |
++------------+ | | |
+| ethdev port| <---ethdev attach-------+ |
++------------+ | | |
+ | | |
+ +-----prgdev bind-------> |
+ | | |
+ | | |
+ <--------SUCCESS--------+ |
+ | | |
+ + + +
+
+
+Program procedure
+=================
+
+This section will describe how apps finish an image downloading procedure with
+the prgdev API. Similar to previous section, it will use several FPGA and NIC
+examples to show the common and different steps.
+
+Needs to point out, 'rte_prgdev_img_download' function didn't carry the info
+of the image, or where downloading the image to. The interpretion info of the
+image can be hidden into the buffer itself.
+
+Common programming steps
+------------------------
+
+Before reaching details, it needs to introduce the common steps for apps to
+follow in order to download the image to the hardware.
+
+Before programming, apps need to call function 'rte_prgdev_open' to open the
+programming interface. If failed, either functions residing in PCI-E device is
+in use or the PCI-E device is not ready, the programming steps are different.
+
+Below is the steps that 'open' succeeds:
+1. rte_prgdev_open --> SUCCESS
+2. rte_prgdev_img_download
+3. rte_prgdev_check_status
+4. rte_prgdev_close
+5. rte_prgdev_bind
+
+If failed to open the programming interface, apps need to follow below steps:
+1. rte_prgdev_open --> FAILURE
+2. Close all related functions
+3. ret_prgdev_unbind
+4. rte_prgdev_open --> SUCCESS
+5. rte_prgdev_img_download
+6. rte_prgdev_check_status
+7. rte_prgdev_close
+8. rte_prgdev_bind
+
+Detach functions
+----------------
+
+As discussed before, apps need to follow steps to detach all the other functions
+except prgdev when meeting error for 'open' operation. Below is a FPGA example
+with 'ethdev' and 'cryptodev' function attached.
+
+After meeting error, app needs to close ethdev port and cryptodev port first.
+Then, use 'prgdev unbind' to detach all the functions that registered previously
+. FPGA driver will call each 'detach' function for each registered device class
+respectively. After then, the 'ethdev' and 'cryptodev' ports will not be
+available.
+
+
+ Abstraction FPGA driver Hardware
+ layer
+ + + +
++------------+ | | |
+| prgdev dev | +-------prgdev open-----> |
+| | <-------ERR_IN_USE------+ | +-------------+
++------------+ | | | | FPGA dev |
+ | | | | +---------+ |
++------------+ | | | | | AFU id | |
+| ethdev port| +---rte_ethdev_close----> | | +---------+ |
+| | +------ethdev close-----> +-------------+
++------------+ <---------SUCCESS-------+ |
+ | | |
++------------+ | | |
+| crypto port| +---crypto port close---> |
+| | | +---crypto func close--->
++------------+ <---------SUCCESS-------+ |
+ | | |
+ +-----prgdev unbind-----> |
+ | | |
++ - - - - - -+ | | |
+ ethdev port <-----ethdev detach-----+ |
++- - - - - - + | | |
+ | | |
++ - - - - - -+ | | |
+ crypto port <----cryptodev detach---+ |
++- - - - - - + | | |
+ + + +
+
+Device programming
+------------------
+
+After 'open' prgdev sucessfully, apps follow 'image download', 'check status',
+'close' sequence respectively. After then, they need to 'bind' new functions
+back with 'prgdev bind' call.
+
+Still re-use previous FPGA example, below is the execution sequence. After image
+downloading and 'bind' action, 2 new 'virtio' and 'flow' functions are
+registered and 2 new ports are presented to apps.
+
+
+ Abstraction FPGA driver Hardware
+ layer
+ + + + +-------------+
++------------+ | | | | FPGA dev |
+| | +-----prgdev open-------> | | +---------+ |
+| | <-------SUCCESS---------+ | | | ethdev | |
+| | | | | | +---------+ |
+| | +--prgdev_img_download--> | | +---------+ |
+| | | +----Download image-----> | | crypto | |
+| | | <--------SUCCESS--------+ | +---------+ |
+| | <---------SUCCESS-------+ | +-------------+
+| prgdev dev | | | | |
+| | +---prgdev_chk_status---> | |
+| | | +--Verify image status--> |
+| | | <--------SUCCESS--------+ |
+| | <--------SUCCESS--------+ | |
+| | +-------prgdev close----> | v
+| | | | | +-------------+
+| | +-------prgdev bind-----> | | FPGA dev |
++------------+ | | | | +---------+ |
+ | <--Virtio and flow dev--+ | | virtio | |
++------------+ | | | | +---------+ |
+| ethdev port| <-----ethdev attach-----+ | | +---------+ |
++------------+ | | | | |Flow dev | |
+ | | | | +---------+ |
++------------+ | | | +-------------+
+| Flow port | <-----Flow dev attach---+ |
++------------+ | | |
+ + + +
+
+
+Dyanamic program
+----------------
+
+If prgdev device has a capability to be programmed while functions are in use,
+then we say it supports dynamic program. It's possible to update a profile while
+device is used as a ethdev device.
+
+The device can claim this capability by set a DYNAMIC_PROGRAM flag in function
+'rte_prgdev_get_info'.
+
+The dynamic programming procedure is the same as previous section shows, while
+ignore the steps to 'close' and 'unbind' actions when device is in use.
+
+Note that prgdev driver needs to consider if it can support dynamic program to
+avoid any hardware or software error.
+
+
+Source code layout
+==================
+
+Below is the suggested source code layout.
+- 'prg NIC' under 'net' is the PMD NIC directory that support prgdev. This
+ doesn't change programmable NIC's directory and remains the same. The PMD
+ driver should include 2 parts, one part is the code to support rte_ethdev API,
+ another part is the code to support rte_prgdev API.
+- 'librte_prgdev' under 'lib' is the new directory store definition and
+ implementation of the generic prgdev API.
+- 'prg' under 'drivers' is the directory to store specfic implementation
+ driver to support rte_prgdev. Each sub-directory could be a prgdev driver to
+ support specific hardware product.
+
+ +---------+ +---------+
+ | drivers | | lib |
+ +---------+ +----+----+
+ | |
+ +----------+-----------+ |
+ | | |
+ +---v---+ +---v----+ +-------v-------+
+ | net | | prg | | librte_prgdev |
+ +---+---+ +---+----+ +---------------+
+ | |
+ +-----+----+ +-----+----+
+ | | | |
++---v----+ +---v----+ +---v----+ +---v----+
+| i40e | | prg NIC| |FPGA ABC| |GPU XYZ |
++--------+ +--------+ +--------+ +--------+
+
+
--
1.7.7.6
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [RFC 2/2] prgdev: introduce generic prgdev API
2017-01-20 3:21 [RFC 0/2] Prgdev API Chen Jing D(Mark)
2017-01-20 3:21 ` [RFC 1/2] doc: introduction to prgdev Chen Jing D(Mark)
@ 2017-01-20 3:21 ` Chen Jing D(Mark)
2017-02-08 22:49 ` Stephen Hemminger
1 sibling, 1 reply; 10+ messages in thread
From: Chen Jing D(Mark) @ 2017-01-20 3:21 UTC (permalink / raw)
To: dev; +Cc: keith.wiles, gerald.rogers, Chen Jing D(Mark)
A new file to define prgdev API prototype and corresponding data
structures.
Signed-off-by: Chen Jing D(Mark) <jing.d.chen@intel.com>
---
lib/librte_prgdev/rte_prgdev.h | 242 ++++++++++++++++++++++++++++++++++++++++
1 files changed, 242 insertions(+), 0 deletions(-)
create mode 100644 lib/librte_prgdev/rte_prgdev.h
diff --git a/lib/librte_prgdev/rte_prgdev.h b/lib/librte_prgdev/rte_prgdev.h
new file mode 100644
index 0000000..849aba4
--- /dev/null
+++ b/lib/librte_prgdev/rte_prgdev.h
@@ -0,0 +1,242 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016-2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _RTE_PRGDEV_H_
+#define _RTE_PRGDEV_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/*
+ * reflect the device status
+ */
+enum rte_prg_devstat {
+ RTE_PRGDEV_STAT_UNKNOWN = 0, /** Device in a unkown state */
+ RTE_PRGDEV_STAT_READY, /** Device is ready to be programming. */
+ RTE_PRGDEV_STAT_ERASING, /** Device is ready being programmed. */
+ /** Device is performing functionalities and can't be programmed. */
+ RTE_PRGDEV_STAT_RUNNING,
+};
+
+/* Reflect the function block attributes */
+/* Block is readable */
+#define RTE_PGR_FUNC_ATTR_RD 0x00000001
+/* Block is writable */
+#define RTE_PGR_FUNC_ATTR_WR 0x00000002
+/* Block is readable and writable */
+#define RTE_PGR_FUNC_ATTR_RDWR (RTE_PGR_FUNC_ATTR_RD & RTE_PGR_FUNC_ATTR_WR)
+
+struct rte_prg_blk_info {
+ unsigned int size; /* the block size in bytes */
+ unsigned int version; /* It's optional */
+ unsigned int flags; /* Flags to indicate block isreadable/writable */
+};
+
+#define MAX_SIGNATURE_LEN 256
+#define MAX_BLK_NUM 256
+
+struct rte_prg_dev_info {
+ struct rte_pci_device *pci_dev; /**< Device PCI information. */
+ const char *driver_name; /**< Device Driver name. */
+ unsigned int devid; /**< Index to bound host interface, or 0 if none. */
+ /* Programable device HW version number. it's possible that app
+ * have dependency to HW version.
+ */
+ uint16_t hw_ver_major; /* major version number */
+ uint16_t hw_ver_minor; /* minor version number */
+
+ /* A array to store hardware, firmware, running image info.Each device
+ * can define and interpret the info. For example, a device can define
+ * byte[3:0] as signature for on-die personality, byte[5:4] is the major
+ * version, byte[7:6] is minor version. if 0xFFEA1000 is a signature for
+ * virtio personality, major version is 0x0001, minor version is 0x0004.
+ * then signature can be defined :
+ * sig_num = 8;
+ * signature[7:0]= {00, 0x10, 0xEA, 0xFF, 0x00, 0x01, 0x00, 0x04};
+ */
+ unsigned int sig_num; /** < the valid signature length in bytes> */
+ char signature[MAX_SIGNATURE_LEN];
+ enum rte_prg_devstat status;
+
+ /* number of blocks within device */
+ unsigned int blk_num;
+ /* block info */
+ struct rte_prg_blk_info blk_info[MAX_BLK_NUM];
+};
+
+struct rte_prg_dev {
+ struct rte_prg_dev_info prg_dev_info;
+} __rte_cache_aligned;
+
+/*
+* prg_dev_init routine
+*
+* returns : 0 success, non zero failure.
+*/
+
+typedef int (*prg_dev_init_t)(struct rte_prg_dev *prg_dev);
+
+/*
+* prg_dev_uninit routine
+*
+* returns : 0 success, non zero failure.
+*/
+
+
+typedef int (*prg_dev_uninit_t)(struct rte_prg_dev *prg_dev);
+
+/**
+ * @internal
+ * The structure associated with a programmable driver.
+ *
+ * Each prg driver acts as a PCI driver and is represented by a generic
+ * eth_driver* structure that holds:
+ *
+ * - An *rte_pci_driver* structure (which must be the first field).
+ *
+ * - The *prg_dev_init_t* function invoked for each matching PCI device.
+ *
+ * - The *prg_dev_remove* function invoked for each matching PCI device.
+ *
+ * - The size of the private data to allocate for each matching device.
+ */
+
+struct prg_driver {
+ struct rte_pci_driver pci_drv; /**< The PRG is also a PCI driver. */
+ prg_dev_init_t prg_dev_probe; /**< Device probe function. */
+ prg_dev_uninit_t prg_dev_remove; /**< Device remove function. */
+ unsigned int dev_private_size; /**< Size of device private data. */
+};
+
+
+/*
+* Query what personality is in the device.
+*
+* @param device_id
+* The port identifier of the programmable device.
+* @param info
+* A pointer to a structure of type *rte_prg_dev_info* to be filled with
+* the information of the programmable device.
+*/
+
+int rte_prgdev_get_info(uint8_t device_id,
+ struct rte_prg_dev_info *info);
+
+/*
+* Open device for programming, acquiring on-die image, etc.
+* Need to call this function first
+* prior to calling other functionalities.
+* In case the device is performing some tasks, it's device's decision on
+* what result is returned.
+*/
+
+int rte_prgdev_open(uint8_t device_id);
+
+
+/*
+* Download image from host to programmable device.
+*
+* @param device_id
+* The port identifier of the programmable device.
+* @param buffer_ptr
+* A pointer to a buffer that stored the image ready downloading to device
+* @param buf_len
+* the total image length in bytes.
+*/
+
+int rte_prgdev_img_download(uint8_t device_id,
+ uint8_t *buffer_ptr, uint32_t buf_len);
+
+/*
+* Upload image from programmable device to host.
+*
+* @param device_id
+* The port identifier of the programmable device.
+* @param buffer_ptr
+* A pointer to a buffer that store uploaded image
+* @param buf_len
+* the total buffer length in bytes.
+* @param act_len
+* pointer to the actual image length in bytes.
+*/
+
+int rte_prgdev_img_upload(uint8_t device_id, uint8_t *buffer_ptr,
+ uint32_t buf_len,uint32_t *act_len);
+
+/*
+ * reflect the device status
+ */
+
+enum rte_prg_fwstat {
+ RTE_PRGDEV_FWSTAT_OK = 0, /** Image are running well */
+ RTE_PRGDEV_FWSTAT_ERR = -1, /** Image has error. */
+ RTE_PRGDEV_FWSTAT_ERR_VALID = -2, /** Image is not valid. */
+ RTE_PRGDEV_FWSTAT_ERR_CKSUM = -3, /** Image checksum is not correct. */
+ RTE_PRGDEV_FWSTAT_ERR_LEN = -4, /** Image length is not correct. */
+};
+
+/*
+* Check if the downloaded image running on die works in expected way, optional
+* function.
+* @param device_id
+* The port identifier of the programmable device.
+*/
+int rte_prgdev_check_stat(uint8_t device_id, enum rte_prg_fwstat *stat);
+
+/*
+* Called to free up resources or whatever to do to hardware
+* after an erase or load of the program.
+* @param device_id
+* The port identifier of the programmable device.
+*/
+int rte_prgdev_close(uint8_t device_id);
+
+/*
+* Called to bind a programmable device to drivers after close function is called.
+* @param device_id
+* The port identifier of the programmable device.
+*/
+int rte_prgdev_bind(uint8_t device_id);
+/*
+* Called to unbind all functions except prgdev from drivers.
+* @param device_id
+* The port identifier of the programmable device.
+*/
+rte_prgdev_unbind(uint8_t device_id);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_PRGDEV_H_ */
--
1.7.7.6
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [RFC 1/2] doc: introduction to prgdev
2017-01-20 3:21 ` [RFC 1/2] doc: introduction to prgdev Chen Jing D(Mark)
@ 2017-02-01 11:41 ` Jan Blunck
2017-02-03 9:21 ` Liang, Cunming
2017-02-06 9:16 ` Chen, Jing D
2017-02-08 3:30 ` Liang, Cunming
1 sibling, 2 replies; 10+ messages in thread
From: Jan Blunck @ 2017-02-01 11:41 UTC (permalink / raw)
To: Chen Jing D(Mark); +Cc: dev, keith.wiles, gerald.rogers
On Fri, Jan 20, 2017 at 4:21 AM, Chen Jing D(Mark)
<jing.d.chen@intel.com> wrote:
> This is the documentation to describe what prgdev is, how to use
> prgdev API and accomplish an image download.
>
> Signed-off-by: Chen Jing D(Mark) <jing.d.chen@intel.com>
> ---
> doc/guides/prog_guide/prgdev_lib.rst | 457 ++++++++++++++++++++++++++++++++++
> 1 files changed, 457 insertions(+), 0 deletions(-)
> create mode 100644 doc/guides/prog_guide/prgdev_lib.rst
>
> diff --git a/doc/guides/prog_guide/prgdev_lib.rst b/doc/guides/prog_guide/prgdev_lib.rst
> new file mode 100644
> index 0000000..3917c18
> --- /dev/null
> +++ b/doc/guides/prog_guide/prgdev_lib.rst
> @@ -0,0 +1,457 @@
> +Overview
> +========
> +
> +More and more devices start to support on-die programming, which include
> +NIC, GPU, FPGA, etc. This capability has a requirement to drivers that
> +introduce a API for application to download/upload image to/from the HW.
> +So, it's necessary to provide a generic API to perform image download, upload,
> +validation, etc.
> +
> +This RFC patch intends to introduce a DPDK generic programming device layer,
> +called prgdev below, to provide an abstract, generic APIs for applications to
> +program hardware without knowing the details of programmable devices. From
> +driver's perspective, they'll try to adapt their functions to the abstract
> +APIs defined in prgdev.
> +
> +The major purpose of prgdev is to help DPDK users to load/upgrade RTL images
> +for FPGA devices, or upgrade firmware for programmble NICs.
> +
> +But those devices that have the capability to do on-die programming may
> +expose versatile talents to apps. Add/delete/update entries in flow table,
> +update/overwrite the firmware/microcode, add a new algorithm, update profiles
> +, etc, which is hard to be abstracted as generic behaviors. So, prgdev won't
> +include the APIs to perform device-unique actions in current stage until it
> +became popular. On the contratry, it only provides the simple capability to
> +download a segment of buffer(image) from host to on-die device, or vice versa.
> +The image could be an algorithm, a flow table in the on-die SRAM, a firmware
> +image running in the device, a function within a pipeline, etc. prgdev won't
> +try to inteprete the content and it's transparent to prgdev. Apps and devices
> +can communicate with a language they can understand each other, which is not
> +provided by prgdev to decide what needs to be programmed.
> +
> +When the set of APIs is introduced, a general question is why we need it in
> +DPDK community? Why we can't use offline tool to perform same thing? The answer
> +is the prgdev provide a generic, online API to applications, in the meanwile,
> +offers a capability to switch driver dynamically when downloaded image changed
> +personality and a new driver is required. Comparing offline tool, it have online
> +programmability (see below examples); Comparing online tool, it provides a
> +generic API for many HWs; Comparing generic online tool/API for many products,
> +it provides a capability to switch driver dynamically.
> +
> +There are various means to reach same goal, we'll try to find the best/better
> +way to approach. All above advantages will help prgdev to be a 'better choice'.
> +
>From my point of view this doesn't belong into the DPDK. On Linux this
is traditionally handled by udev and you already have the freedom to
use userspace applications to program a device requiring firmware in
that case. I don't believe that modeling this in the DPDK explicitly
is the right thing to do.
Especially if the device supports changing personality it is required
to unplug the existing personality before reprogramming. You can do
this already today. Also writing OOB firmware data that changes
configuration should be possible today by handling interrupts.
Maybe we can come up with an example application that demonstrates how
the different infrastructure components could get orchestrated. Do you
have a device in mind that supports this?
Regards,
Jan
> +
> +Scope
> +-----
> +
> +The target devices include but not limited to programmable NIC, FPGA, GPU.
> +Any devices, having the capability to store/load a piece of info to/from
> +the deivce then changed hardware behavior, and applicable to prgdev programming
> +model, could be registered as a prgdev device.
> +
> +The device can be programmed (dynamic) within DPDK or have been prior programmed
> +(static) prior to a DPDK application being launched.
> +
> +Static example: In a static example a program, bitstream or other firmware
> +would have been loaded prior to the start of a DPDK application. The prgdev
> +would recognize the firmware loaded, and instantiate higher level interfaces
> +(example : PMD, L3 forwarding, flow, etc.).
> +
> +Dynamic example: In a dynamic example, the programmability of the program,
> +bitstream or other firmware would be loaded by the DPDK application upon start
> +, or during execution. The prgdev would offer the interface to allow the DPDK
> +application to program the device.
> +
> +Dyanamic bind/unbind
> +--------------------
> +
> +Besides the simple API to upload/download image, the prgdev also introduces a
> +mechanism internally to switch drivers and register/unregister device on the
> +fly. With this mechanism, apps can use the programmable device, unbind other
> +personalities on demand, then program it and bind it back with new personalities
> +. Apps can follow below examples to simply complete the whole process.
> +
> +Note that bind/unbind actions are different concept from that a whole device
> +attach/detach. For example, rte_eal_dev_detach(), which will try to detach the
> +drivers with device and remove the whole device from specific class of devices
> +(ethdev, for example). Then, the whole device is invisible until a new 'probe'
> +is activated.
> +
> +During the whole procedure of image upload/download, prgdev handler is always
> +valid and apps can use it operate on programmable device. The reason why unbind
> +is necessary is it may break the hardware when programming is in progress while
> +other functions are active. Using programmble NIC as an example, it's a little
> +impossible to receive/forward packets for hardware while updating firmware. In
> +this case, apps need to detach ethdev function before firmware upgrading. Again,
> +prgdev handler is still valid, it's function-level detach, different from
> +device-level detach. After finishing image download, original function needs to
> +attach back, either use same or different drivers, depends on personalities
> +changed or not. For a programmble NIC, the driver may be the same. For FPGA, it
> +may not, see below examples to get more details.
> +
> +Another reason to provide bind/unbind action is programmble devices, like FPGA,
> +are not identified driver by 'vendor ID' and 'device ID', they might not be
> +changed in all the ways, even FPGA is fully programmed. So, it depends on
> +internal mechanism of FPGA, not 'vendor ID' and 'device ID' to identify proper
> +drivers, either a register value or signature, depending on FPGA hardware
> +design. In this case, EAL or other bus driver doesn't have the capability to
> +identify proper driver for FPGA device. With prgdev introduced, while FPGA is
> +always a prgdev, FPGA can use prgdev as primary driver, to find proper function
> +driver.
> +
> +The bind/unbind actions are not trying to break the general initialization
> +procedure in DPDK. prgdev still follow current EAL initializatio model to probe
> +drivers for devices. See below examples for details.
> +
> +It's not true that bind/unbind is always necessary for programming. If device
> +has a capability to perform specific functions and program in parallel, then
> +apps needn't do bind/unbind actions. See 'dynamic program' section below.
> +
> +Authentication
> +--------------
> +
> +Authentication and permissions for programming the device are outside the scope
> +of this API. It is dependent upon a different mechanism not to be supported by
> +DPDK. Application can utilize a different mechanism to authenticate user and
> +or 'program' as validated for loading.
> +
> +Misc
> +----
> +
> +PCI-E Vendor & Device ID are fixed in some devices such as the FPGA and GPU. The
> +device can be loaded with a personality, and it would be up to this programmable
> +device to load the appropriate "personality driver" based upon an identification
> +mechanism to work with the loaded personality.
> +
> +
> +prgdev API
> +==========
> +
> +Below is the major APIs exposed to apps.
> +
> +- rte_prgdev_get_info
> + Acquire general info of the prgdev.
> +- rte_prgdev_open
> + Open a prgdev for later programming. If failed, needs to call below 'unbind'
> + function to detach other functions except prgdev first.
> +- rte_prgdev_img_download
> + Download a piece of buffer from host to the device after 'open' succeded.
> +- rte_prgdev_img_upload
> + Upload a piece of information from the device to host.
> +- rte_prgdev_check_status
> + Even download process is successful, it's necessary to check the status of the
> + downloaded image if worked in expected way by this function. Harware or driver
> + may not have the capability to check that status, so it's optional to driver.
> +- rte_prgdev_close
> + After upload/download actions, close the prgdev.
> +- rte_prgdev_bind
> + After 'close' action, app can call it to attach the 'new' personality with
> + proper drivers, then expose to app with new functions. prgdev driver will be
> + responsible for finding proper drivers to drive the functions. Desired driver
> + will be responsible for exposing the functions to apps.
> +- rte_prgdev_unbind
> + When app are using the device, they need to call this function to unbind the
> + misc functions except prgdev with drivers prior to call 'open' function.
> +
> +Initialization procedure
> +========================
> +In this section, there 2 examples, FPGA and programmable NIC, used as examples
> +to describe the initialization procedure on how programmable devices are probed
> +, initialized and registered as different devices. In which, FPGA has 2
> +different possible cases, blank and prior programmed FPGAs. Note that prgdev
> +depends and follow on EAL initialization process to probe drivers for prgdev.
> +
> +Blank FPGA
> +----------
> +Below is the diagram on how Blank FPGA are probed and registered as a prgdev
> +device that expose to apps.
> +
> +When PCI probe seeks driver for the FPGA device, FPGA drive will compare vendor
> +ID and device ID, then perform HW initialization if both ID match. After then,
> +it registered in prgdev abstract layer as a new device with a set of helper
> +function ops.
> +
> +After that, apps will observe single device: prgdev device. Until now, the FPGA
> +device is a simply device without any functionalities from application's
> +perspective.
> +
> +In the following action, app will call 'prgdev bind' function with the device
> +ID of the FPGA device to bind the function within the FPGA to specific driver.
> +FPGA driver will read AFU (Accelerated Function Unit) or signature from hardware
> +and find potential function. Since it's a blank without any prior programming, a
> +simply SUCCESS return.
> +
> + Abstraction FPGA driver Hardware
> + layer
> + + + +
> + | | |
> + +-----eal pci probe-----> |
> + | | | +------------+
> + | | | | FPGA dev |
> + | +--HW initialization----> | |
> ++------------+ | | | |+----------+|
> +| prgdev dev | <--prgdev_ops_register--+ | || AFU id ||
> ++------------+ | | | |+----------+|
> + | | | +------------+
> + +-----prgdev bind-------> |
> + | | |
> + | +--Read and parse AFU--->
> + | | |
> + | <--No personality found-+
> + | | |
> + <-------SUCCESS---------+ |
> + | | |
> + + + +
> +
> +prior programmed FPGA
> +---------------------
> +
> +For a prior programmed FPGA, all execution flow is similar to blank FPGA case,
> +except the action of reading signature, as below flow table indicates. It will
> +read valid functions, crypto and ethdev, as an example, then find proper driver
> +to initialize the functions, then register as different ports.
> +
> +After initialization, app will observe 3 kinds of devices, a prgdev device, a
> +ethdev device and a cryptodev device, which all comes from a single PCIE device.
> +
> +
> + Abstraction FPGA driver Hardware
> + layer
> + + + +
> + | | |
> + +----eal pci probe------> |
> + | | | +-------------+
> + | | | | FPGA dev |
> + | +--HW initialization----> | |
> ++------------+ | | | +-------------+
> +| prgdev dev | <--prgdev_ops_register--+ | || AFU id ||
> ++------------+ | | | |-------------|
> + | | | +-------------+
> + +-----prgdev bind-------> |
> + | | |
> + | +--Read and parse AFU--->
> + | | |
> + | <--eth and crypto found-+
> ++------------+ | | |
> +| ethdev port| <---ethdev attach-------+ |
> ++------------+ | | |
> + | | |
> ++------------+ | | |
> +| crypto port| <---crypto attach-------+ |
> ++------------+ | | |
> + | | |
> + + + +
> +
> +
> +Programmable NIC
> +----------------
> +
> +Comparing FPGA, programmable NIC has a nature of being a ethdev device. The
> +initialization procedure is much more simpler than FPGA. Below is a flow table
> +to describe this.
> +
> +The 'eal pci probe' procedure is similar, it registered itself as a prgdev after
> +checking that the NIC is programmable. What is different from FPGA is it
> +registered itself as a ethdev device in the following action due to the NIC
> +nature.
> +
> +Then, app still follow the common process to call 'prgdev bind', while the NIC
> +driver does nothing and just return a SUCCESS.
> +
> + Abstraction NIC driver Hardware
> + layer
> + + + +
> + | | |
> + +----eal pci probe------> |
> + | | | +-------------+
> + | +--HW initialization----> |Programmable |
> + | | | | NIC |
> ++------------+ | +---Capability check----> +-------------+
> +| prgdev dev | <--prgdev_ops_register--+ |
> ++------------+ | | |
> + | | |
> ++------------+ | | |
> +| ethdev port| <---ethdev attach-------+ |
> ++------------+ | | |
> + | | |
> + +-----prgdev bind-------> |
> + | | |
> + | | |
> + <--------SUCCESS--------+ |
> + | | |
> + + + +
> +
> +
> +Program procedure
> +=================
> +
> +This section will describe how apps finish an image downloading procedure with
> +the prgdev API. Similar to previous section, it will use several FPGA and NIC
> +examples to show the common and different steps.
> +
> +Needs to point out, 'rte_prgdev_img_download' function didn't carry the info
> +of the image, or where downloading the image to. The interpretion info of the
> +image can be hidden into the buffer itself.
> +
> +Common programming steps
> +------------------------
> +
> +Before reaching details, it needs to introduce the common steps for apps to
> +follow in order to download the image to the hardware.
> +
> +Before programming, apps need to call function 'rte_prgdev_open' to open the
> +programming interface. If failed, either functions residing in PCI-E device is
> +in use or the PCI-E device is not ready, the programming steps are different.
> +
> +Below is the steps that 'open' succeeds:
> +1. rte_prgdev_open --> SUCCESS
> +2. rte_prgdev_img_download
> +3. rte_prgdev_check_status
> +4. rte_prgdev_close
> +5. rte_prgdev_bind
> +
> +If failed to open the programming interface, apps need to follow below steps:
> +1. rte_prgdev_open --> FAILURE
> +2. Close all related functions
> +3. ret_prgdev_unbind
> +4. rte_prgdev_open --> SUCCESS
> +5. rte_prgdev_img_download
> +6. rte_prgdev_check_status
> +7. rte_prgdev_close
> +8. rte_prgdev_bind
> +
> +Detach functions
> +----------------
> +
> +As discussed before, apps need to follow steps to detach all the other functions
> +except prgdev when meeting error for 'open' operation. Below is a FPGA example
> +with 'ethdev' and 'cryptodev' function attached.
> +
> +After meeting error, app needs to close ethdev port and cryptodev port first.
> +Then, use 'prgdev unbind' to detach all the functions that registered previously
> +. FPGA driver will call each 'detach' function for each registered device class
> +respectively. After then, the 'ethdev' and 'cryptodev' ports will not be
> +available.
> +
> +
> + Abstraction FPGA driver Hardware
> + layer
> + + + +
> ++------------+ | | |
> +| prgdev dev | +-------prgdev open-----> |
> +| | <-------ERR_IN_USE------+ | +-------------+
> ++------------+ | | | | FPGA dev |
> + | | | | +---------+ |
> ++------------+ | | | | | AFU id | |
> +| ethdev port| +---rte_ethdev_close----> | | +---------+ |
> +| | +------ethdev close-----> +-------------+
> ++------------+ <---------SUCCESS-------+ |
> + | | |
> ++------------+ | | |
> +| crypto port| +---crypto port close---> |
> +| | | +---crypto func close--->
> ++------------+ <---------SUCCESS-------+ |
> + | | |
> + +-----prgdev unbind-----> |
> + | | |
> ++ - - - - - -+ | | |
> + ethdev port <-----ethdev detach-----+ |
> ++- - - - - - + | | |
> + | | |
> ++ - - - - - -+ | | |
> + crypto port <----cryptodev detach---+ |
> ++- - - - - - + | | |
> + + + +
> +
> +Device programming
> +------------------
> +
> +After 'open' prgdev sucessfully, apps follow 'image download', 'check status',
> +'close' sequence respectively. After then, they need to 'bind' new functions
> +back with 'prgdev bind' call.
> +
> +Still re-use previous FPGA example, below is the execution sequence. After image
> +downloading and 'bind' action, 2 new 'virtio' and 'flow' functions are
> +registered and 2 new ports are presented to apps.
> +
> +
> + Abstraction FPGA driver Hardware
> + layer
> + + + + +-------------+
> ++------------+ | | | | FPGA dev |
> +| | +-----prgdev open-------> | | +---------+ |
> +| | <-------SUCCESS---------+ | | | ethdev | |
> +| | | | | | +---------+ |
> +| | +--prgdev_img_download--> | | +---------+ |
> +| | | +----Download image-----> | | crypto | |
> +| | | <--------SUCCESS--------+ | +---------+ |
> +| | <---------SUCCESS-------+ | +-------------+
> +| prgdev dev | | | | |
> +| | +---prgdev_chk_status---> | |
> +| | | +--Verify image status--> |
> +| | | <--------SUCCESS--------+ |
> +| | <--------SUCCESS--------+ | |
> +| | +-------prgdev close----> | v
> +| | | | | +-------------+
> +| | +-------prgdev bind-----> | | FPGA dev |
> ++------------+ | | | | +---------+ |
> + | <--Virtio and flow dev--+ | | virtio | |
> ++------------+ | | | | +---------+ |
> +| ethdev port| <-----ethdev attach-----+ | | +---------+ |
> ++------------+ | | | | |Flow dev | |
> + | | | | +---------+ |
> ++------------+ | | | +-------------+
> +| Flow port | <-----Flow dev attach---+ |
> ++------------+ | | |
> + + + +
> +
> +
> +Dyanamic program
> +----------------
> +
> +If prgdev device has a capability to be programmed while functions are in use,
> +then we say it supports dynamic program. It's possible to update a profile while
> +device is used as a ethdev device.
> +
> +The device can claim this capability by set a DYNAMIC_PROGRAM flag in function
> +'rte_prgdev_get_info'.
> +
> +The dynamic programming procedure is the same as previous section shows, while
> +ignore the steps to 'close' and 'unbind' actions when device is in use.
> +
> +Note that prgdev driver needs to consider if it can support dynamic program to
> +avoid any hardware or software error.
> +
> +
> +Source code layout
> +==================
> +
> +Below is the suggested source code layout.
> +- 'prg NIC' under 'net' is the PMD NIC directory that support prgdev. This
> + doesn't change programmable NIC's directory and remains the same. The PMD
> + driver should include 2 parts, one part is the code to support rte_ethdev API,
> + another part is the code to support rte_prgdev API.
> +- 'librte_prgdev' under 'lib' is the new directory store definition and
> + implementation of the generic prgdev API.
> +- 'prg' under 'drivers' is the directory to store specfic implementation
> + driver to support rte_prgdev. Each sub-directory could be a prgdev driver to
> + support specific hardware product.
> +
> + +---------+ +---------+
> + | drivers | | lib |
> + +---------+ +----+----+
> + | |
> + +----------+-----------+ |
> + | | |
> + +---v---+ +---v----+ +-------v-------+
> + | net | | prg | | librte_prgdev |
> + +---+---+ +---+----+ +---------------+
> + | |
> + +-----+----+ +-----+----+
> + | | | |
> ++---v----+ +---v----+ +---v----+ +---v----+
> +| i40e | | prg NIC| |FPGA ABC| |GPU XYZ |
> ++--------+ +--------+ +--------+ +--------+
> +
> +
> --
> 1.7.7.6
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC 1/2] doc: introduction to prgdev
2017-02-01 11:41 ` Jan Blunck
@ 2017-02-03 9:21 ` Liang, Cunming
2017-02-06 9:16 ` Chen, Jing D
1 sibling, 0 replies; 10+ messages in thread
From: Liang, Cunming @ 2017-02-03 9:21 UTC (permalink / raw)
To: Jan Blunck, Chen Jing D(Mark); +Cc: dev, keith.wiles, gerald.rogers
Hi,
On 2/1/2017 7:41 PM, Jan Blunck wrote:
> On Fri, Jan 20, 2017 at 4:21 AM, Chen Jing D(Mark)
> <jing.d.chen@intel.com> wrote:
>> This is the documentation to describe what prgdev is, how to use
>> prgdev API and accomplish an image download.
>>
>> Signed-off-by: Chen Jing D(Mark) <jing.d.chen@intel.com>
>> ---
>> doc/guides/prog_guide/prgdev_lib.rst | 457 ++++++++++++++++++++++++++++++++++
>> 1 files changed, 457 insertions(+), 0 deletions(-)
>> create mode 100644 doc/guides/prog_guide/prgdev_lib.rst
[...]
> From my point of view this doesn't belong into the DPDK. On Linux this
> is traditionally handled by udev and you already have the freedom to
> use userspace applications to program a device requiring firmware in
> that case. I don't believe that modeling this in the DPDK explicitly
> is the right thing to do.
Good point, but not sure udev has user space device driver support or not.
>
> Especially if the device supports changing personality it is required
> to unplug the existing personality before reprogramming. You can do
> this already today. Also writing OOB firmware data that changes
> configuration should be possible today by handling interrupts.
It's going to allow changing personality in DPDK user space runtime.
If the personality is not belong to a device but part of the component,
unplug isn't helpful too much.
>
> Maybe we can come up with an example application that demonstrates how
> the different infrastructure components could get orchestrated. Do you
> have a device in mind that supports this?
The coming Purley platform has SKU for Xeon-FPGA.
The FPGA connecting with Xeon has dedicated pcie device id.
The AFU personality for packet I/O depends on the RTL image.
Changing the personality in runtime could be one of the situation.
Regards,
Cunming
>
> Regards,
> Jan
>
[...]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC 1/2] doc: introduction to prgdev
2017-02-01 11:41 ` Jan Blunck
2017-02-03 9:21 ` Liang, Cunming
@ 2017-02-06 9:16 ` Chen, Jing D
1 sibling, 0 replies; 10+ messages in thread
From: Chen, Jing D @ 2017-02-06 9:16 UTC (permalink / raw)
To: Jan Blunck; +Cc: dev, keith.wiles, gerald.rogers
Hi, Jan,
On 2/1/2017 7:41 PM, Jan Blunck wrote:
> On Fri, Jan 20, 2017 at 4:21 AM, Chen Jing D(Mark)
> <jing.d.chen@intel.com> wrote:
>> This is the documentation to describe what prgdev is, how to use
>> prgdev API and accomplish an image download.
>>
>> Signed-off-by: Chen Jing D(Mark) <jing.d.chen@intel.com>
>> ---
>> doc/guides/prog_guide/prgdev_lib.rst | 457 ++++++++++++++++++++++++++++++++++
>> 1 files changed, 457 insertions(+), 0 deletions(-)
>> create mode 100644 doc/guides/prog_guide/prgdev_lib.rst
>>
>> diff --git a/doc/guides/prog_guide/prgdev_lib.rst b/doc/guides/prog_guide/prgdev_lib.rst
>> new file mode 100644
>> index 0000000..3917c18
>> --- /dev/null
>> +++ b/doc/guides/prog_guide/prgdev_lib.rst
>> @@ -0,0 +1,457 @@
>> +Overview
>> +========
>> +
>> +More and more devices start to support on-die programming, which include
>> +NIC, GPU, FPGA, etc. This capability has a requirement to drivers that
>> +introduce a API for application to download/upload image to/from the HW.
>> +So, it's necessary to provide a generic API to perform image download, upload,
>> +validation, etc.
>> +
>> +This RFC patch intends to introduce a DPDK generic programming device layer,
>> +called prgdev below, to provide an abstract, generic APIs for applications to
>> +program hardware without knowing the details of programmable devices. From
>> +driver's perspective, they'll try to adapt their functions to the abstract
>> +APIs defined in prgdev.
>> +
>> +The major purpose of prgdev is to help DPDK users to load/upgrade RTL images
>> +for FPGA devices, or upgrade firmware for programmble NICs.
>> +
>> +But those devices that have the capability to do on-die programming may
>> +expose versatile talents to apps. Add/delete/update entries in flow table,
>> +update/overwrite the firmware/microcode, add a new algorithm, update profiles
>> +, etc, which is hard to be abstracted as generic behaviors. So, prgdev won't
>> +include the APIs to perform device-unique actions in current stage until it
>> +became popular. On the contratry, it only provides the simple capability to
>> +download a segment of buffer(image) from host to on-die device, or vice versa.
>> +The image could be an algorithm, a flow table in the on-die SRAM, a firmware
>> +image running in the device, a function within a pipeline, etc. prgdev won't
>> +try to inteprete the content and it's transparent to prgdev. Apps and devices
>> +can communicate with a language they can understand each other, which is not
>> +provided by prgdev to decide what needs to be programmed.
>> +
>> +When the set of APIs is introduced, a general question is why we need it in
>> +DPDK community? Why we can't use offline tool to perform same thing? The answer
>> +is the prgdev provide a generic, online API to applications, in the meanwile,
>> +offers a capability to switch driver dynamically when downloaded image changed
>> +personality and a new driver is required. Comparing offline tool, it have online
>> +programmability (see below examples); Comparing online tool, it provides a
>> +generic API for many HWs; Comparing generic online tool/API for many products,
>> +it provides a capability to switch driver dynamically.
>> +
>> +There are various means to reach same goal, we'll try to find the best/better
>> +way to approach. All above advantages will help prgdev to be a 'better choice'.
>> +
> From my point of view this doesn't belong into the DPDK. On Linux this
> is traditionally handled by udev and you already have the freedom to
> use userspace applications to program a device requiring firmware in
> that case. I don't believe that modeling this in the DPDK explicitly
> is the right thing to do.
Can you kindly educate me what udev can do? It's for NIC firmware upgrade?
This prgdev is not only for NIC, the major use case is FPGA, but I found
programmable
NIC is also applicable to the API model and can be added into the scope.
>
> Especially if the device supports changing personality it is required
> to unplug the existing personality before reprogramming. You can do
> this already today. Also writing OOB firmware data that changes
> configuration should be possible today by handling interrupts.
What if application is using DPDK, can we do in the same manner without
leaving
DPDK instance?
> Maybe we can come up with an example application that demonstrates how
> the different infrastructure components could get orchestrated. Do you
> have a device in mind that supports this?
As Cunming suggested, the incoming FPGA is the desired device to use
this API. OVS
that is using DPDK can benefit from the API and download personalities
in running
time for blank FPGA, or upgrade for prior-programmed case.
>
> Regards,
> Jan
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC 1/2] doc: introduction to prgdev
2017-01-20 3:21 ` [RFC 1/2] doc: introduction to prgdev Chen Jing D(Mark)
2017-02-01 11:41 ` Jan Blunck
@ 2017-02-08 3:30 ` Liang, Cunming
1 sibling, 0 replies; 10+ messages in thread
From: Liang, Cunming @ 2017-02-08 3:30 UTC (permalink / raw)
To: dev; +Cc: keith.wiles@intel.com; Chen Jing D(Mark), gerald.rogers
On 1/20/2017 11:21 AM, Chen Jing D(Mark) wrote:
> This is the documentation to describe what prgdev is, how to use
> prgdev API and accomplish an image download.
>
> Signed-off-by: Chen Jing D(Mark) <jing.d.chen@intel.com>
> ---
> doc/guides/prog_guide/prgdev_lib.rst | 457 ++++++++++++++++++++++++++++++++++
> 1 files changed, 457 insertions(+), 0 deletions(-)
> create mode 100644 doc/guides/prog_guide/prgdev_lib.rst
>
> diff --git a/doc/guides/prog_guide/prgdev_lib.rst b/doc/guides/prog_guide/prgdev_lib.rst
> new file mode 100644
> index 0000000..3917c18
> --- /dev/null
> +++ b/doc/guides/prog_guide/prgdev_lib.rst
> @@ -0,0 +1,457 @@
> +Overview
> +========
> +
[...]
> +When the set of APIs is introduced, a general question is why we need it in
> +DPDK community? Why we can't use offline tool to perform same thing? The answer
> +is the prgdev provide a generic, online API to applications, in the meanwile,
> +offers a capability to switch driver dynamically when downloaded image changed
> +personality and a new driver is required. Comparing offline tool, it have online
> +programmability (see below examples); Comparing online tool, it provides a
> +generic API for many HWs; Comparing generic online tool/API for many products,
> +it provides a capability to switch driver dynamically.
> +
> +There are various means to reach same goal, we'll try to find the best/better
> +way to approach. All above advantages will help prgdev to be a 'better choice'.
> +
>
One more notes.
DPDK takes over the devices in user space. The legacy tools usually
download the personality by kernel driver. They runs out of DPDK
context. When a DPDK process is running on top of the device, operation
to the device by the solo tool during the time causes resource conflict.
It's one of the motivations to have the native API allowing programming
the personality within DPDK context. Otherwise, it has to exit the
process if any personality change is required. Manually detaching the
device before using the solo tool may ease the conflict. However it
still limits the situation if the device allows multiple programmable
instances(e.g. AFUs in FPGA) which are working independently and
shouldn't be impact by each other.
[...]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC 2/2] prgdev: introduce generic prgdev API
2017-01-20 3:21 ` [RFC 2/2] prgdev: introduce generic prgdev API Chen Jing D(Mark)
@ 2017-02-08 22:49 ` Stephen Hemminger
2017-02-08 23:06 ` Wiles, Keith
0 siblings, 1 reply; 10+ messages in thread
From: Stephen Hemminger @ 2017-02-08 22:49 UTC (permalink / raw)
To: Chen Jing D(Mark); +Cc: dev, keith.wiles, gerald.rogers
On Fri, 20 Jan 2017 11:21:38 +0800
"Chen Jing D(Mark)" <jing.d.chen@intel.com> wrote:
> +struct rte_prg_dev_info {
> + struct rte_pci_device *pci_dev; /**< Device PCI information. */
NAK
No new API should refer to PCI devices. We are trying to make all references
to devices generic. Crypto suceeded, ether is in process, but no new ones please.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC 2/2] prgdev: introduce generic prgdev API
2017-02-08 22:49 ` Stephen Hemminger
@ 2017-02-08 23:06 ` Wiles, Keith
2017-02-09 2:23 ` Chen, Jing D
0 siblings, 1 reply; 10+ messages in thread
From: Wiles, Keith @ 2017-02-08 23:06 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: Chen, Jing D, dev, Rogers, Gerald
> On Feb 8, 2017, at 4:49 PM, Stephen Hemminger <stephen@networkplumber.org> wrote:
>
> On Fri, 20 Jan 2017 11:21:38 +0800
> "Chen Jing D(Mark)" <jing.d.chen@intel.com> wrote:
>
>> +struct rte_prg_dev_info {
>> + struct rte_pci_device *pci_dev; /**< Device PCI information. */
>
> NAK
>
> No new API should refer to PCI devices. We are trying to make all references
> to devices generic. Crypto suceeded, ether is in process, but no new ones please.
The design is not PCI centric and should not be PCI centric, it is just what they started with months ago. So please look past the PCI bits as they have not integrated into the latest code base yet.
Regards,
Keith
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC 2/2] prgdev: introduce generic prgdev API
2017-02-08 23:06 ` Wiles, Keith
@ 2017-02-09 2:23 ` Chen, Jing D
0 siblings, 0 replies; 10+ messages in thread
From: Chen, Jing D @ 2017-02-09 2:23 UTC (permalink / raw)
To: Wiles, Keith, Stephen Hemminger; +Cc: dev, Rogers, Gerald
Hi, Keith, Stephen,
> -----Original Message-----
> From: Wiles, Keith
> Sent: Thursday, February 9, 2017 7:07 AM
> To: Stephen Hemminger <stephen@networkplumber.org>
> Cc: Chen, Jing D <jing.d.chen@intel.com>; dev@dpdk.org; Rogers, Gerald
> <gerald.rogers@intel.com>
> Subject: Re: [dpdk-dev] [RFC 2/2] prgdev: introduce generic prgdev API
>
>
> > On Feb 8, 2017, at 4:49 PM, Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> >
> > On Fri, 20 Jan 2017 11:21:38 +0800
> > "Chen Jing D(Mark)" <jing.d.chen@intel.com> wrote:
> >
> >> +struct rte_prg_dev_info {
> >> + struct rte_pci_device *pci_dev; /**< Device PCI information. */
> >
> > NAK
> >
> > No new API should refer to PCI devices. We are trying to make all
> > references to devices generic. Crypto suceeded, ether is in process, but no
> new ones please.
>
> The design is not PCI centric and should not be PCI centric, it is just what they
> started with months ago. So please look past the PCI bits as they have not
> integrated into the latest code base yet.
>
I got you. Will remove the reference in formal patch.
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2017-02-09 2:23 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-01-20 3:21 [RFC 0/2] Prgdev API Chen Jing D(Mark)
2017-01-20 3:21 ` [RFC 1/2] doc: introduction to prgdev Chen Jing D(Mark)
2017-02-01 11:41 ` Jan Blunck
2017-02-03 9:21 ` Liang, Cunming
2017-02-06 9:16 ` Chen, Jing D
2017-02-08 3:30 ` Liang, Cunming
2017-01-20 3:21 ` [RFC 2/2] prgdev: introduce generic prgdev API Chen Jing D(Mark)
2017-02-08 22:49 ` Stephen Hemminger
2017-02-08 23:06 ` Wiles, Keith
2017-02-09 2:23 ` Chen, Jing D
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.