[dpdk-dev] [PATCH v2 16/16] doc: update telemetry documentation

[Ciara Power <ciara.power@intel.com>]

The existing documentation for Telemetry is updated, and further
documentation is added.

Signed-off-by: Ciara Power <ciara.power@intel.com>
---
 doc/api/doxy-api-index.md                 |   1 +
 doc/guides/howto/telemetry.rst            | 108 ++++++++++------------
 doc/guides/linux_gsg/eal_args.include.rst |   8 ++
 doc/guides/linux_gsg/sys_reqs.rst         |   2 -
 doc/guides/prog_guide/index.rst           |   1 +
 doc/guides/prog_guide/telemetry_lib.rst   |  62 +++++++++++++
 doc/guides/rel_notes/release_20_05.rst    |  15 +++
 7 files changed, 138 insertions(+), 59 deletions(-)
 create mode 100644 doc/guides/prog_guide/telemetry_lib.rst

diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index dff496be09..0bbdbe8706 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -170,6 +170,7 @@ The public API headers are grouped by topics:
 - **debug**:
   [jobstats]           (@ref rte_jobstats.h),
   [telemetry]          (@ref rte_telemetry.h),
+  [telemetry-json]     (@ref rte_telemetry_json.h),
   [pdump]              (@ref rte_pdump.h),
   [hexdump]            (@ref rte_hexdump.h),
   [debug]              (@ref rte_debug.h),
diff --git a/doc/guides/howto/telemetry.rst b/doc/guides/howto/telemetry.rst
index cacc082161..4f39a4df56 100644
--- a/doc/guides/howto/telemetry.rst
+++ b/doc/guides/howto/telemetry.rst
@@ -1,86 +1,80 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
-    Copyright(c) 2018 Intel Corporation.
+    Copyright(c) 2020 Intel Corporation.
 
-DPDK Telemetry API User Guide
+
+DPDK Telemetry User Guide
 ==============================
 
-This document describes how the Data Plane Development Kit(DPDK) Telemetry API
-is used for querying port statistics from incoming traffic.
+The Telemetry library provides users with the ability to query DPDK for
+telemetry information, currently including information such as ethdev stats,
+ethdev port list, and eal parameters.
+
+.. Note::
+
+   This library is experimental and the output format may change in the future.
+
 
-Introduction
-------------
+Telemetry Interface
+-------------------
 
-The ``librte_telemetry`` provides the functionality so that users may query
-metrics from incoming port traffic and global stats(application stats).
-The application which initializes packet forwarding will act as the server,
-sending metrics to the requesting application which acts as the client.
+The :ref:`librte_telemetry <telemetry_library>` opens a socket with path
+*<runtime_directory>/dpdk_telemetry.<version>*. The version represents the
+telemetry version, the latest is v2. For example, a client would connect to a
+socket with path  */var/run/dpdk/\*/dpdk_telemetry.v2* (when the primary process
+is run by a root user).
 
 
-In DPDK, applications are used to initialize the ``telemetry``. To view incoming
-traffic on featured ports, the application should be run first (ie. after ports
-are configured). Once the application is running, the service assurance agent
-(for example the collectd plugin) should be run to begin querying the API.
+Telemetry Initialization
+------------------------
 
-A client connects their Service Assurance application to the DPDK application
-via a UNIX socket. Once a connection is established, a client can send JSON
-messages to the DPDK application requesting metrics via another UNIX client.
-This request is then handled and parsed if valid. The response is then
-formatted in JSON and sent back to the requesting client.
+The library is enabled by default, however an EAL flag to enable the library
+exists, to provide backward compatibility for the previous telemetry library
+interface.
 
-Pre-requisites
-~~~~~~~~~~~~~~
+.. code-block:: console
 
-* Python >= 2.5
+    --telemetry
 
-* Jansson library for JSON serialization
+A flag exists to disable Telemetry also.
 
-Test Environment
-----------------
+.. code-block:: console
 
-``telemetry`` offers a range of selftests that a client can run within
-the DPDK application.
+    --no-telemetry
 
-Selftests are disabled by default. They can be enabled by setting the 'selftest'
-variable to 1 in rte_telemetry_initial_accept().
 
-Note: this 'hardcoded' value is temporary.
+Running Telemetry
+-----------------
 
-Configuration
--------------
+The following steps show how to run an application with telemetry support,
+and query information using the telemetry client python script.
 
-Enable the telemetry API by modifying the following config option before
-building DPDK::
+#. Launch testpmd as the primary application with telemetry.
 
-        CONFIG_RTE_LIBRTE_TELEMETRY=y
+   .. code-block:: console
 
-Note: Meson will pick this up automatically if ``libjansson`` is available.
+        ./app/dpdk-testpmd
 
-Running the Application
------------------------
+#. Launch the telemetry client script.
 
-The following steps demonstrate how to run the ``telemetry`` API  to query all
-statistics on all active ports, using the ``telemetry_client`` python script
-to query.
-Note: This guide assumes packet generation is applicable and the user is
-testing with testpmd as a DPDK primary application to forward packets, although
-any DPDK application is applicable.
+   .. code-block:: console
 
-#. Launch testpmd as the primary application with ``telemetry``.::
+        python usertools/dpdk-telemetry.py
 
-        ./app/testpmd --telemetry
+#. When connected, the script displays the following, waiting for user input.
 
-#. Launch the ``telemetry`` python script with a client filepath.::
+   .. code-block:: console
 
-        python usertools/telemetry_client.py /var/run/some_client
+        Connecting to /var/run/dpdk/rte/dpdk_telemetry.v2
+        {"pid": 60285, "version": "DPDK 20.05.0-rc0", "max_output_len": 16384}
+        -->
 
-   The client filepath is going to be used to setup our UNIX connection with the
-   DPDK primary application, in this case ``testpmd``
-   This will initialize a menu where a client can proceed to recursively query
-   statistics, request statistics once or unregister the file_path, thus exiting
-   the menu.
+#. The user can now input commands to send across the socket, and receive the
+   response.
 
-#. Send traffic to any or all available ports from a traffic generator.
-   Select a query option(recursive or singular polling or global stats).
-   The metrics will then be displayed on the client terminal in JSON format.
+   .. code-block:: console
 
-#. Once finished, unregister the client using the menu command.
+        --> /
+        {"/": ["/", "/eal/app_params", "/eal/params", "/ethdev/list",
+        "/ethdev/link_status", "/ethdev/xstats", "/info"]}
+        --> /ethdev/list
+        {"/ethdev/list": [0, 1]}
diff --git a/doc/guides/linux_gsg/eal_args.include.rst b/doc/guides/linux_gsg/eal_args.include.rst
index ed8b0e35b0..a6938c160d 100644
--- a/doc/guides/linux_gsg/eal_args.include.rst
+++ b/doc/guides/linux_gsg/eal_args.include.rst
@@ -150,3 +150,11 @@ Other options
 *   ``mbuf-pool-ops-name``:
 
     Pool ops name for mbuf to use.
+
+*    ``--telemetry``:
+
+    Enable telemetry (enabled by default).
+
+*    ``--no-telemetry``:
+
+    Disable telemetry.
diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
index 7c47ec04ce..a124656bcb 100644
--- a/doc/guides/linux_gsg/sys_reqs.rst
+++ b/doc/guides/linux_gsg/sys_reqs.rst
@@ -95,8 +95,6 @@ For libraries the additional dependencies include:
 
 *   libarchive: for some unit tests using tar to get their resources.
 
-*   jansson: to compile and use the telemetry library.
-
 *   libelf: to compile and use the bpf library.
 
 For poll-mode drivers, the additional dependencies for each driver can be
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index fb250abf51..7e7b06ce5f 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -57,6 +57,7 @@ Programmer's Guide
     metrics_lib
     bpf_lib
     ipsec_lib
+    telemetry_lib
     source_org
     dev_kit_build_system
     dev_kit_root_make_help
diff --git a/doc/guides/prog_guide/telemetry_lib.rst b/doc/guides/prog_guide/telemetry_lib.rst
new file mode 100644
index 0000000000..cb4f058473
--- /dev/null
+++ b/doc/guides/prog_guide/telemetry_lib.rst
@@ -0,0 +1,62 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2020 Intel Corporation.
+
+.. _telemetry_library:
+
+
+Telemetry Library
+=================
+
+The Telemetry library provides an interface to retrieve information from a
+variety of DPDK libraries. The library provides this information via socket
+connection, taking requests from a connected client and replying with the JSON
+response containing the requested telemetry information.
+
+Telemetry is enabled to run by default when running a DPDK application, and the
+telemetry information from enabled libraries is made available. Libraries are
+responsible for registering their own commands, and providing the callback
+function that will format the library specific stats into the correct JSON
+response format, when requested.
+
+
+Registering Commands
+--------------------
+
+Libraries and applications must register commands to make their information
+available via the Telemetry library. This involves providing a string command
+in the required format ("/library/command"), and the callback function that
+will handle formatting the information when required. An example showing ethdev
+commands being registered is shown below:
+
+.. code-block:: c
+
+    rte_telemetry_register_cmd("/ethdev/list", handle_port_list);
+    rte_telemetry_register_cmd("/ethdev/xstats", handle_port_xstats);
+
+
+Formatting JSON response
+------------------------
+
+The callback function provided by the library must format its telemetry
+information in a valid JSON format. The Telemetry library provides a JSON
+utilities API to build up the response. In the event of the output buffer being
+too small to hold the telemetry information in full, the API functions maintain
+correct JSON formatting regardless. For example, the ethdev library provides a
+list of available ethdev ports in a JSON response, constructed using the
+following functions to build up the list:
+
+.. code-block:: c
+
+    used = rte_tel_json_empty_array(buffer, buf_len, used);
+    RTE_ETH_FOREACH_DEV(port_id)
+        used = rte_tel_json_add_array_int(buffer, buf_len, used, port_id);
+
+The resulting response that is returned to the client shows the list of ports
+constructed above by the handler function in ethdev:
+
+.. code-block:: console
+
+    {"/ethdev/list": [0, 1]}
+
+For more information on the range of JSON functions available in the API,
+please refer to the docs.
diff --git a/doc/guides/rel_notes/release_20_05.rst b/doc/guides/rel_notes/release_20_05.rst
index 000bbf501e..083e706224 100644
--- a/doc/guides/rel_notes/release_20_05.rst
+++ b/doc/guides/rel_notes/release_20_05.rst
@@ -62,6 +62,21 @@ New Features
 
   * Added support for matching on IPv4 Time To Live and IPv6 Hop Limit.
 
+* **Updated Telemetry Library.**
+
+  The updated Telemetry library has many improvements on the original version
+  to make it more accessible and scalable:
+
+  * It enables DPDK libraries and applications provide their own specific
+    telemetry information, rather than being limited to what could be reported
+    through the metrics library.
+
+  * It is no longer dependent on the external Jansson library, which allows
+    Telemetry be enabled by default.
+
+  * The socket handling has been simplified making it easier for clients to
+    connect and retrieve information.
+
 
 Removed Items
 -------------
-- 
2.17.1