[PATCH v10 00/12] Introduce Telemetry Library

[Harry van Haaren <harry.van.haaren@intel.com>]

This patchset introduces a Telemetry library for DPDK Service Assurance.
This library provides an easy way to query DPDK Ethdev metrics.

The telemetry library provides a method for a service assurance component
to retrieve metrics from a DPDK packet forwarding application.
Communicating from the service assurance component to DPDK is done using a
UNIX domain socket, passing a JSON formatted string. A reply is sent (again
a JSON formatted string) of the current DPDK metrics.

The telemetry component makes use of the existing rte_metrics library to
query values. The values to be transmitted via the telemetry infrastructure
must be present in the Metrics library. Currently the ethdev values are
pushed to the metrics library, and the queried from there  there is an open
question on how applications would like this to occur. Currently only
ethdev to metrics functionality is implemented, however other subsystems
like crypto, eventdev, keepalive etc can use similar mechanisms.

Exposing DPDK Telemetry via a socket interface enables service assurance
agents like collectd to consume data from DPDK. This is vital for
monitoring, fault-detection, and error reporting. A collectd plugin has
been created to interact with the DPDK Telemetry component, showing how it
can be used in practice. The collectd plugin will be upstreamed to collectd
at a later stage.  A small python script is provided in
./usertools/telemetry_client.py to quick-start using DPDK Telemetry.

Note: Despite opterr being set to 0, --telemetry said to be 'unrecognized'
as a startup print. This is a cosmetic issue and will be addressed in the
future.

---
v2:
   - Reworked telemetry as part of EAL instead of using vdev (Gaetan)
   - Refactored rte_telemetry_command (Gaetan)
   - Added MAINTAINERS file entry (Stephen)
   - Updated docs to reflect vdev to eal rework
   - Removed collectd patch from patchset (Thomas)
   - General code clean up from v1 feedback

v3:
  - Reworked registering with eal and moved to rte_param (Gaetan)
  - Added BSD implementation for rte_param (Gaetan)
  - Updated the paths to align with the new runtime file location (Mattias)
  - Fixed pointer checks to align with the coding style 1.8.1 (Mattias)
  - Added missing decref's and close's (Mattias)
  - Fixed runtime issue in Meson (was not recognising flag due to linking)
  - More general clean up

v4:
  - Added Doxygen comments for rte_param.h (Thomas)
  - Made eal_get_runtime_dir a public function to use outside of EAL (Thomas)
  - Reworked telemetry to get path using rte_eal_get_runtime_dir (Thomas)
  - Fixed checkpatch coding style error

v5:
  - Moved the BUF_SIZE define to fix build (Harry)
  - Set default config for telemetry to 'n' (Harry)
  - Improved Doxygen comments (Thomas)
  - Cleaned up rte_param struct (Thomas)

v6:
  - Renamed rte_param to rte_option (Thomas)
  - Moved internal functions to eal_private.h (Gaetan)
  - Added fail check for pthread_attr_init() (Mattias)
  - Changed socket implementation to SOCK_SEQPACKET (Mattias)
  - Added check to avoid option duplicates (Gaetan)
  - Removed telemetry example from Doxygen comment (Gaetan)
  - General Doxygen clean-up (Thomas)
  - General code clean-up (Mattias)

v7:
  - Fix object leak (Mattias)
  - General clean-up (Mattias)
  - Fix rte_option header define
  - Added release note update
  - Rebased

v8:
  - Added missing experimental tags
  - Added Gaetan's Ack from ML

v9:
  - Added doxygen to build (Thomas)
  - Added doxygen @file tag (Thomas)
  - Detect jansson library dynamically with Make/pkg-config (Thomas)
  - Make jansson optional build-dep for Meson (Thomas/Bruce)

v10:
  - Location of Telemetry in doxygen (Thomas)
  - Linking jansson in patch of first usage (Thomas)
  - Meson detection in patch of first usage (Thomas)
  - Removed pkg-config check in Make due to issues with 32/64 bit (Thomas)
  - MAINTAINERS file entires introduced earlier and split (Thomas)
  - Disable Telemetry library in Make by default due to jansson dep (Thomas)

Ciara Power (9):
  telemetry: initial telemetry infrastructure
  telemetry: add initial connection socket
  telemetry: add client feature and sockets
  telemetry: add parser for client socket messages
  telemetry: update metrics before sending stats
  telemetry: format json response when sending stats
  telemetry: add ability to disable selftest
  doc: add telemetry documentation
  usertools: add client python script for telemetry

Kevin Laatz (3):
  eal: add option register infrastructure
  eal: make get runtime dir function public
  build: add dependency on telemetry to apps in meson

 MAINTAINERS                                   |    5 +
 app/meson.build                               |    4 +-
 app/pdump/meson.build                         |    2 +-
 app/proc-info/meson.build                     |    2 +-
 app/test-bbdev/meson.build                    |    2 +-
 app/test-crypto-perf/meson.build              |    2 +-
 app/test-pmd/meson.build                      |    2 +-
 config/common_base                            |    5 +
 config/meson.build                            |    3 +
 doc/api/doxy-api-index.md                     |    1 +
 doc/api/doxy-api.conf.in                      |    1 +
 doc/guides/howto/index.rst                    |    1 +
 doc/guides/howto/telemetry.rst                |   85 +
 doc/guides/rel_notes/release_18_11.rst        |    6 +
 lib/Makefile                                  |    2 +
 lib/librte_eal/bsdapp/eal/Makefile            |    1 +
 lib/librte_eal/bsdapp/eal/eal.c               |   16 +-
 lib/librte_eal/common/Makefile                |    1 +
 lib/librte_eal/common/eal_filesystem.h        |   15 +-
 lib/librte_eal/common/eal_private.h           |   21 +
 lib/librte_eal/common/include/rte_eal.h       |    9 +
 lib/librte_eal/common/include/rte_option.h    |   63 +
 lib/librte_eal/common/meson.build             |    2 +
 lib/librte_eal/common/rte_option.c            |   54 +
 lib/librte_eal/linuxapp/eal/Makefile          |    1 +
 lib/librte_eal/linuxapp/eal/eal.c             |   16 +-
 lib/librte_eal/rte_eal_version.map            |    2 +
 lib/librte_telemetry/Makefile                 |   30 +
 lib/librte_telemetry/meson.build              |   15 +
 lib/librte_telemetry/rte_telemetry.c          | 1815 +++++++++++++++++
 lib/librte_telemetry/rte_telemetry.h          |   66 +
 lib/librte_telemetry/rte_telemetry_internal.h |   81 +
 lib/librte_telemetry/rte_telemetry_parser.c   |  586 ++++++
 lib/librte_telemetry/rte_telemetry_parser.h   |   14 +
 .../rte_telemetry_parser_test.c               |  534 +++++
 .../rte_telemetry_parser_test.h               |   39 +
 .../rte_telemetry_socket_tests.h              |   36 +
 .../rte_telemetry_version.map                 |   10 +
 lib/meson.build                               |    3 +-
 meson.build                                   |    2 +
 mk/rte.app.mk                                 |    3 +-
 usertools/dpdk-telemetry-client.py            |  116 ++
 42 files changed, 3654 insertions(+), 20 deletions(-)
 create mode 100644 doc/guides/howto/telemetry.rst
 create mode 100644 lib/librte_eal/common/include/rte_option.h
 create mode 100644 lib/librte_eal/common/rte_option.c
 create mode 100644 lib/librte_telemetry/Makefile
 create mode 100644 lib/librte_telemetry/meson.build
 create mode 100644 lib/librte_telemetry/rte_telemetry.c
 create mode 100644 lib/librte_telemetry/rte_telemetry.h
 create mode 100644 lib/librte_telemetry/rte_telemetry_internal.h
 create mode 100644 lib/librte_telemetry/rte_telemetry_parser.c
 create mode 100644 lib/librte_telemetry/rte_telemetry_parser.h
 create mode 100644 lib/librte_telemetry/rte_telemetry_parser_test.c
 create mode 100644 lib/librte_telemetry/rte_telemetry_parser_test.h
 create mode 100644 lib/librte_telemetry/rte_telemetry_socket_tests.h
 create mode 100644 lib/librte_telemetry/rte_telemetry_version.map
 create mode 100644 usertools/dpdk-telemetry-client.py

-- 
2.17.1