[PATCH 0/2] testpmd: simulating noisy host environment

[PATCH 0/2] testpmd: simulating noisy host environment

[Jens Freimann]

This patch set proposes enhancements to testpmd to simulate
more realistic behavior of a guest machine engaged in receiving
and sending packets performing Virtual Network Function (VNF).

The goal is to enable simple of measuring performance impact on cache and
memory footprint utilization from various VNF co-located on the
same host machine.

This series of patches adds the new command line switches to
testpmd:

--buffersize-before-sending [packet numbers]

Keep the mbuf in a FIFO and forward the over flooding packets from the
FIFO. This queue is per TX-queue (after all other packet processing).

--flush-timer [delay]
Flush the packet queue if no packets have been seen during
[delay]. As long as packets are seen, the timer is reset.


Options to simulate route lookups:

--memory-footprint [size]
Size of the VNF internal memory (MB), in which the random
read/write will be done, allocated by rte_malloc (hugepages).

--random-w-memory-access-per-packet [num]
Number of random writes in memory per packet should be
performed, simulating hit-flags update. 64 bits per write,
all write in different cache lines.

--random-r-memory-access-per-packet [num]
Number of random reads in memory per packet should be
performed, simulating FIB/table lookups. 64 bits per read,
all write in different cache lines.

--random-rw-memory-access-per-packet [num]
Number of random reads and writes in memory per packet should
be performed, simulating stats update. 64 bits per read-write, all
reads and writes in different cache lines.

Comments are appreciated. 

Should parameter names be prefixed so they
won't be confused with other functionality? 

regards,
Jens 


Jens Freimann (2):
  testpmd: add parameters buffersize-before-send and flush-timeout
  testpmd: add code to simulate noisy neighbour memory usage

 app/test-pmd/fifo.h       |  43 ++++++++++++++++++
 app/test-pmd/iofwd.c      | 109 +++++++++++++++++++++++++++++++++++++++++++++-
 app/test-pmd/parameters.c |  57 +++++++++++++++++++++++-
 app/test-pmd/testpmd.c    |  78 +++++++++++++++++++++++++++++++++
 app/test-pmd/testpmd.h    |  20 +++++++++
 config/common_base        |   1 +
 6 files changed, 306 insertions(+), 2 deletions(-)
 create mode 100644 app/test-pmd/fifo.h

-- 
2.14.3

[PATCH 1/2] testpmd: add parameters buffersize-before-send and flush-timeout

[Jens Freimann]

Create a fifo to buffer received packets. Once it flows over put
those packets into the actual tx queue. The fifo is created per tx
queue and its size can be set with the --buffersize-before-sending
commandline parameter.

A second commandline parameter is used to set a timeout in
milliseconds after which the fifo is flushed.

--buffersize-before-sending [packet numbers]
Keep the mbuf in a FIFO and forward the over flooding packets from the
FIFO. This queue is per TX-queue (after all other packet processing).

--flush-timer [delay]
Flush the packet queue if no packets have been seen during
[delay]. As long as packets are seen, the timer is reset.

Signed-off-by: Jens Freimann <jfreimann@redhat.com>
---
 app/test-pmd/fifo.h       | 43 ++++++++++++++++++++++++++++++++++
 app/test-pmd/iofwd.c      | 59 ++++++++++++++++++++++++++++++++++++++++++++++-
 app/test-pmd/parameters.c | 21 ++++++++++++++++-
 app/test-pmd/testpmd.c    | 48 ++++++++++++++++++++++++++++++++++++++
 app/test-pmd/testpmd.h    | 15 ++++++++++++
 config/common_base        |  1 +
 6 files changed, 185 insertions(+), 2 deletions(-)
 create mode 100644 app/test-pmd/fifo.h

diff --git a/app/test-pmd/fifo.h b/app/test-pmd/fifo.h
new file mode 100644
index 000000000..01415f98c
--- /dev/null
+++ b/app/test-pmd/fifo.h
@@ -0,0 +1,43 @@
+#ifndef __FIFO_H
+#define __FIFO_H
+#include <stdint.h>
+#include <rte_mbuf.h>
+#include <rte_ring.h>
+#include "testpmd.h"
+
+#define FIFO_COUNT_MAX 1024
+/**
+ * Add elements to fifo. Return number of written elements
+ */
+static inline unsigned
+fifo_put(struct rte_ring *r, struct rte_mbuf **data, unsigned num)
+{
+
+	return rte_ring_enqueue_burst(r, (void **)data, num, NULL);
+}
+
+/**
+ * Get elements from fifo. Return number of read elements
+ */
+static inline unsigned
+fifo_get(struct rte_ring *r, struct rte_mbuf **data, unsigned num)
+{
+	return rte_ring_dequeue_burst(r, (void **) data, num, NULL);
+}
+
+static inline unsigned
+fifo_count(struct rte_ring *r)
+{
+	return rte_ring_count(r);
+}
+
+static inline int
+fifo_full(struct rte_ring *r)
+{
+	return rte_ring_full(r);
+}
+
+
+struct rte_ring * fifo_init(uint32_t qi, uint32_t pi);
+
+#endif 
diff --git a/app/test-pmd/iofwd.c b/app/test-pmd/iofwd.c
index 9dce76efe..85fa000f7 100644
--- a/app/test-pmd/iofwd.c
+++ b/app/test-pmd/iofwd.c
@@ -36,6 +36,7 @@
 #include <rte_flow.h>
 
 #include "testpmd.h"
+#include "fifo.h"
 
 /*
  * Forwarding of packets in I/O mode.
@@ -48,7 +49,7 @@ pkt_burst_io_forward(struct fwd_stream *fs)
 {
 	struct rte_mbuf *pkts_burst[MAX_PKT_BURST];
 	uint16_t nb_rx;
-	uint16_t nb_tx;
+	uint16_t nb_tx = 0;
 	uint32_t retry;
 
 #ifdef RTE_TEST_PMD_RECORD_CORE_CYCLES
@@ -56,6 +57,15 @@ pkt_burst_io_forward(struct fwd_stream *fs)
 	uint64_t end_tsc;
 	uint64_t core_cycles;
 #endif
+#ifdef RTE_TEST_PMD_NOISY
+	const uint64_t freq_khz = rte_get_timer_hz() / 1000;
+	struct noisy_config *ncf = &noisy_cfg[fs->tx_queue];
+	struct rte_mbuf *tmp_pkts[MAX_PKT_BURST];
+	uint16_t nb_enqd;
+	uint16_t nb_deqd = 0;
+	uint64_t delta_ms;
+	uint64_t now;
+#endif
 
 #ifdef RTE_TEST_PMD_RECORD_CORE_CYCLES
 	start_tsc = rte_rdtsc();
@@ -73,8 +83,55 @@ pkt_burst_io_forward(struct fwd_stream *fs)
 #ifdef RTE_TEST_PMD_RECORD_BURST_STATS
 	fs->rx_burst_stats.pkt_burst_spread[nb_rx]++;
 #endif
+#ifdef RTE_TEST_PMD_NOISY
+	if (bsize_before_send > 0) {
+		if (rte_ring_free_count(ncf->f) >= nb_rx) {
+			/* enqueue into fifo */
+			nb_enqd = fifo_put(ncf->f, pkts_burst, nb_rx);
+			if (nb_enqd < nb_rx)
+				nb_rx = nb_enqd;
+		} else {
+			/* fifo is full, dequeue first */
+			nb_deqd = fifo_get(ncf->f, tmp_pkts, nb_rx);
+			/* enqueue into fifo */
+			nb_enqd = fifo_put(ncf->f, pkts_burst, nb_deqd);
+			if (nb_enqd < nb_rx)
+				nb_rx = nb_enqd;
+			if (nb_deqd > 0)
+				nb_tx = rte_eth_tx_burst(fs->tx_port,
+						fs->tx_queue, tmp_pkts,
+						nb_deqd);
+		}
+	} else {
+		nb_tx = rte_eth_tx_burst(fs->tx_port, fs->tx_queue,
+				pkts_burst, nb_rx);
+	}
+
+	/*
+	 * TX burst queue drain
+	 */
+	if (ncf->prev_time == 0) {
+		now = ncf->prev_time = rte_get_timer_cycles();
+	} else {
+		now = rte_get_timer_cycles();
+	}
+	delta_ms = (now - ncf->prev_time) / freq_khz;
+	if (unlikely(delta_ms >= flush_timer) && flush_timer > 0 && (nb_tx == 0)) {
+		while (fifo_count(ncf->f) > 0) {
+			nb_deqd = fifo_get(ncf->f, tmp_pkts, nb_rx);
+			nb_tx = rte_eth_tx_burst(fs->tx_port, fs->tx_queue,
+						 tmp_pkts, nb_deqd);
+			if(rte_ring_empty(ncf->f))
+				break;
+		}
+		ncf->prev_time = now;
+	}
+	if (nb_tx < nb_rx && fs->retry_enabled)
+		*pkts_burst = *tmp_pkts;
+#else
 	nb_tx = rte_eth_tx_burst(fs->tx_port, fs->tx_queue,
 			pkts_burst, nb_rx);
+#endif
 	/*
 	 * Retry if necessary
 	 */
diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index 2192bdcdf..df0db933a 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -621,6 +621,8 @@ launch_args_parse(int argc, char** argv)
 		{ "print-event",		1, 0, 0 },
 		{ "mask-event",			1, 0, 0 },
 		{ "tx-offloads",		1, 0, 0 },
+		{ "buffersize-before-sending",  1, 0, 0 },
+		{ "flush-timer",                1, 0, 0 },
 		{ 0, 0, 0, 0 },
 	};
 
@@ -1101,7 +1103,24 @@ launch_args_parse(int argc, char** argv)
 					rte_exit(EXIT_FAILURE,
 						 "invalid mask-event argument\n");
 				}
-
+#ifdef RTE_TEST_PMD_NOISY
+			if (!strcmp(lgopts[opt_idx].name, "buffersize-before-sending")) {
+				n = atoi(optarg);
+				if (n > 0)
+					bsize_before_send = (uint16_t) n;
+				else
+					rte_exit(EXIT_FAILURE,
+						 "buffersize-before-sending must be > 0\n");
+			}
+			if (!strcmp(lgopts[opt_idx].name, "flush-timer")) {
+				n = atoi(optarg);
+				if (n >= 0)
+					flush_timer = (uint16_t) n;
+				else
+					rte_exit(EXIT_FAILURE,
+						 "flush-timer must be > 0\n");
+			}
+#endif
 			break;
 		case 'h':
 			usage(argv[0]);
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 4c0e2586c..7b8ffdc9c 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -59,6 +59,9 @@
 #ifdef RTE_LIBRTE_LATENCY_STATS
 #include <rte_latencystats.h>
 #endif
+#ifdef RTE_TEST_PMD_NOISY
+#include "fifo.h"
+#endif
 
 #include "testpmd.h"
 
@@ -249,6 +252,18 @@ int16_t tx_free_thresh = RTE_PMD_PARAM_UNSET;
  */
 int16_t tx_rs_thresh = RTE_PMD_PARAM_UNSET;
 
+#ifdef RTE_TEST_PMD_NOISY
+/*
+ * Configurable value of buffered packed before sending.
+ */
+uint16_t bsize_before_send = 0;
+
+/*
+ * Configurable value of packet buffer timeout.
+ */
+uint16_t flush_timer = 0;
+#endif
+
 /*
  * Receive Side Scaling (RSS) configuration.
  */
@@ -401,6 +416,20 @@ static int all_ports_started(void);
 struct gso_status gso_ports[RTE_MAX_ETHPORTS];
 uint16_t gso_max_segment_size = ETHER_MAX_LEN - ETHER_CRC_LEN;
 
+#ifdef RTE_TEST_PMD_NOISY
+#define STRSIZE 256
+#define NOISY_RING "noisy_ring_%d:%d\n"
+struct rte_ring * fifo_init(uint32_t qi, uint32_t pi)
+{
+	struct noisy_config *n = &noisy_cfg[qi];
+	char name[STRSIZE];
+
+	snprintf(name, STRSIZE, NOISY_RING, pi, qi);
+	n->f = rte_ring_create(name, bsize_before_send, rte_socket_id(), 0);
+	return n->f;
+}
+#endif
+
 /*
  * Helper function to check if socket is already discovered.
  * If yes, return positive value. If not, return zero.
@@ -1584,6 +1613,16 @@ start_port(portid_t pid)
 				return -1;
 			}
 		}
+#ifdef RTE_TEST_PMD_NOISY
+		noisy_cfg = (struct noisy_config *) rte_zmalloc("testpmd noisy fifo and timers",
+					nb_txq * sizeof(struct noisy_config),
+					RTE_CACHE_LINE_SIZE);
+		if (noisy_cfg == NULL) {
+			rte_exit(EXIT_FAILURE,
+					"rte_zmalloc(%d struct noisy_config) failed\n",
+					(int)(nb_txq * sizeof(struct noisy_config)));
+		}
+#endif
 		if (port->need_reconfig_queues > 0) {
 			port->need_reconfig_queues = 0;
 			port->tx_conf.txq_flags = ETH_TXQ_FLAGS_IGNORE;
@@ -1591,6 +1630,11 @@ start_port(portid_t pid)
 			port->tx_conf.offloads = port->dev_conf.txmode.offloads;
 			/* setup tx queues */
 			for (qi = 0; qi < nb_txq; qi++) {
+#ifdef RTE_TEST_PMD_NOISY
+				if (!fifo_init(qi, pi) && bsize_before_send > 0)
+					rte_exit(EXIT_FAILURE, "%s\n",
+						 rte_strerror(rte_errno));
+#endif
 				if ((numa_support) &&
 					(txring_numa[pi] != NUMA_NO_CONFIG))
 					diag = rte_eth_tx_queue_setup(pi, qi,
@@ -1755,6 +1799,10 @@ stop_port(portid_t pid)
 			RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
 			printf("Port %d can not be set into stopped\n", pi);
 		need_check_link_status = 1;
+
+#ifdef  RTE_TEST_PMD_NOISY
+		rte_free(noisy_cfg);
+#endif
 	}
 	if (need_check_link_status && !no_link_check)
 		check_all_ports_link_status(RTE_PORT_ALL);
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 153abea05..a6c1a17bb 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -9,6 +9,9 @@
 #include <rte_bus_pci.h>
 #include <rte_gro.h>
 #include <rte_gso.h>
+#ifdef RTE_TEST_PMD_NOISY
+#include "fifo.h"
+#endif
 
 #define RTE_PORT_ALL            (~(portid_t)0x0)
 
@@ -109,6 +112,15 @@ struct fwd_stream {
 #endif
 };
 
+#ifdef RTE_TEST_PMD_NOISY
+struct noisy_config {
+	struct rte_ring *f;
+	uint64_t prev_time;
+};
+struct noisy_config *noisy_cfg;
+#endif
+
+
 /** Descriptor for a single flow. */
 struct port_flow {
 	size_t size; /**< Allocated space including data[]. */
@@ -382,6 +394,9 @@ extern int8_t rx_drop_en;
 extern int16_t tx_free_thresh;
 extern int16_t tx_rs_thresh;
 
+extern uint16_t bsize_before_send;
+extern uint16_t flush_timer;
+
 extern uint8_t dcb_config;
 extern uint8_t dcb_test;
 
diff --git a/config/common_base b/config/common_base
index c09c7cf88..2fd17439b 100644
--- a/config/common_base
+++ b/config/common_base
@@ -806,6 +806,7 @@ CONFIG_RTE_PROC_INFO=n
 CONFIG_RTE_TEST_PMD=y
 CONFIG_RTE_TEST_PMD_RECORD_CORE_CYCLES=n
 CONFIG_RTE_TEST_PMD_RECORD_BURST_STATS=n
+CONFIG_RTE_TEST_PMD_NOISY=y
 
 #
 # Compile the bbdev test application
-- 
2.14.3

[PATCH 2/2] testpmd: add code to simulate noisy neighbour memory usage

[Jens Freimann]

Add several options to simulate route lookups (memory reads) in tables
that can be quite large, as well as route hit statistics update.
These options simulates the while stack traversal and
will trash the cache. Memory access is random.

Options to simulate route lookups:

--memory-footprint [size]
Size of the VNF internal memory (MB), in which the random
read/write will be done, allocated by rte_malloc (hugepages).

--nb-rnd-write [num]
Number of random writes in memory per packet should be
performed, simulating hit-flags update. 64 bits per write,
all write in different cache lines.

--nb-rnd-read [num]
Number of random reads in memory per packet should be
performed, simulating FIB/table lookups. 64 bits per read,
all write in different cache lines.

--nb-rnd-read-write [num]
Number of random reads and writes in memory per packet should
be performed, simulating stats update. 64 bits per read-write, all
reads and writes in different cache lines.

Signed-off-by: Jens Freimann <jfreimann@redhat.com>
---
 app/test-pmd/iofwd.c      | 50 +++++++++++++++++++++++++++++++++++++++++++++++
 app/test-pmd/parameters.c | 36 ++++++++++++++++++++++++++++++++++
 app/test-pmd/testpmd.c    | 30 ++++++++++++++++++++++++++++
 app/test-pmd/testpmd.h    |  5 +++++
 4 files changed, 121 insertions(+)

diff --git a/app/test-pmd/iofwd.c b/app/test-pmd/iofwd.c
index 85fa000f7..e69727e2c 100644
--- a/app/test-pmd/iofwd.c
+++ b/app/test-pmd/iofwd.c
@@ -34,10 +34,57 @@
 #include <rte_ethdev.h>
 #include <rte_string_fns.h>
 #include <rte_flow.h>
+#include <rte_malloc.h>
 
 #include "testpmd.h"
 #include "fifo.h"
 
+static inline void
+do_write(char *vnf_mem)
+{
+	uint64_t i = rte_rand();
+	uint64_t w = rte_rand();
+
+	vnf_mem[i % ((vnf_memory_footprint * 1024 * 1024 ) /
+			RTE_CACHE_LINE_SIZE)] = w;
+}
+
+static inline void
+do_read(char *vnf_mem)
+{
+	uint64_t i = rte_rand();
+	uint64_t r = 0;
+
+	r = vnf_mem[i % ((vnf_memory_footprint * 1024 * 1024 ) /
+			RTE_CACHE_LINE_SIZE)];
+	r++;
+}
+
+static inline void
+do_rw(char *vnf_mem)
+{
+	do_read(vnf_mem);
+	do_write(vnf_mem);
+}
+
+/*
+ * Simulate route lookups as defined by commandline parameters
+ */
+static void
+sim_memory_lookups(struct noisy_config *ncf, uint16_t nb_pkts)
+{
+	uint16_t i,j;
+
+	for (i = 0; i < nb_pkts; i++) {
+		for (j = 0; j < nb_rnd_write; j++)
+			do_write(ncf->vnf_mem);
+		for (j = 0; j < nb_rnd_read; j++)
+			do_read(ncf->vnf_mem);
+		for (j = 0; j < nb_rnd_read_write; j++)
+			do_rw(ncf->vnf_mem);
+	}
+}
+
 /*
  * Forwarding of packets in I/O mode.
  * Forward packets "as-is".
@@ -107,6 +154,9 @@ pkt_burst_io_forward(struct fwd_stream *fs)
 				pkts_burst, nb_rx);
 	}
 
+	/* simulate noisy vnf by trashing cache lines, simulate route lookups */
+	sim_memory_lookups(ncf, nb_rx);
+
 	/*
 	 * TX burst queue drain
 	 */
diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index df0db933a..78e146164 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -623,6 +623,10 @@ launch_args_parse(int argc, char** argv)
 		{ "tx-offloads",		1, 0, 0 },
 		{ "buffersize-before-sending",  1, 0, 0 },
 		{ "flush-timer",                1, 0, 0 },
+		{ "memory-footprint",           1, 0, 0 },
+		{ "nb-rnd-write",               1, 0, 0 },
+		{ "nb-rnd-read",                1, 0, 0 },
+		{ "nb-rnd-read-write",          1, 0, 0 },
 		{ 0, 0, 0, 0 },
 	};
 
@@ -1120,6 +1124,38 @@ launch_args_parse(int argc, char** argv)
 					rte_exit(EXIT_FAILURE,
 						 "flush-timer must be > 0\n");
 			}
+			if (!strcmp(lgopts[opt_idx].name, "memory-footprint")) {
+				n = atoi(optarg);
+				if (n > 0)
+					vnf_memory_footprint = (uint16_t) n;
+				else
+					rte_exit(EXIT_FAILURE,
+						 "memory-footprint must be > 0\n");
+			}
+			if (!strcmp(lgopts[opt_idx].name, "nb-rnd-write")) {
+				n = atoi(optarg);
+				if (n > 0)
+					nb_rnd_write = (uint16_t) n;
+				else
+					rte_exit(EXIT_FAILURE,
+						 "nb-rnd-write must be > 0\n");
+			}
+			if (!strcmp(lgopts[opt_idx].name, "nb-rnd-read")) {
+				n = atoi(optarg);
+				if (n > 0)
+					nb_rnd_read = (uint16_t) n;
+				else
+					rte_exit(EXIT_FAILURE,
+						 "nb-rnd-read must be > 0\n");
+			}
+			if (!strcmp(lgopts[opt_idx].name, "nb-rnd-read-write")) {
+				n = atoi(optarg);
+				if (n > 0)
+					nb_rnd_read_write = (uint16_t) n;
+				else
+					rte_exit(EXIT_FAILURE,
+						 "nb-rnd-read-write must be > 0\n");
+			}
 #endif
 			break;
 		case 'h':
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 7b8ffdc9c..d33139a89 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -262,6 +262,30 @@ uint16_t bsize_before_send = 0;
  * Configurable value of packet buffer timeout.
  */
 uint16_t flush_timer = 0;
+
+/*
+ * Configurable value for size of VNF internal memory area
+ * used for simulating noisy neighbour behaviour
+ */
+uint64_t vnf_memory_footprint = 0;
+
+/*
+ * Configurable value of number of random writes done in
+ * VNF simulation memory area.
+ */
+uint64_t nb_rnd_write = 0;
+
+/*
+ * Configurable value of number of random reads done in
+ * VNF simulation memory area.
+ */
+uint64_t nb_rnd_read = 0;
+
+/*
+ * Configurable value of number of random reads/wirtes done in
+ * VNF simulation memory area.
+ */
+uint64_t nb_rnd_read_write = 0;
 #endif
 
 /*
@@ -426,6 +450,12 @@ struct rte_ring * fifo_init(uint32_t qi, uint32_t pi)
 
 	snprintf(name, STRSIZE, NOISY_RING, pi, qi);
 	n->f = rte_ring_create(name, bsize_before_send, rte_socket_id(), 0);
+	n->vnf_mem = (char *) rte_zmalloc("vnf sim memory",
+			 vnf_memory_footprint * 1024 * 1024,
+			 RTE_CACHE_LINE_SIZE);
+	if (n->vnf_mem == NULL)
+		printf("allocating vnf memory failed\n");
+
 	return n->f;
 }
 #endif
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index a6c1a17bb..0411b5266 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -116,6 +116,7 @@ struct fwd_stream {
 struct noisy_config {
 	struct rte_ring *f;
 	uint64_t prev_time;
+	char *vnf_mem;
 };
 struct noisy_config *noisy_cfg;
 #endif
@@ -396,6 +397,10 @@ extern int16_t tx_rs_thresh;
 
 extern uint16_t bsize_before_send;
 extern uint16_t flush_timer;
+extern uint64_t vnf_memory_footprint;
+extern uint64_t nb_rnd_write;
+extern uint64_t nb_rnd_read;
+extern uint64_t nb_rnd_read_write;
 
 extern uint8_t dcb_config;
 extern uint8_t dcb_test;
-- 
2.14.3

Re: [PATCH 1/2] testpmd: add parameters buffersize-before-send and flush-timeout

[Ananyev, Konstantin]

Hi,

> 
> Create a fifo to buffer received packets. Once it flows over put
> those packets into the actual tx queue. The fifo is created per tx
> queue and its size can be set with the --buffersize-before-sending
> commandline parameter.
> 
> A second commandline parameter is used to set a timeout in
> milliseconds after which the fifo is flushed.
> 
> --buffersize-before-sending [packet numbers]
> Keep the mbuf in a FIFO and forward the over flooding packets from the
> FIFO. This queue is per TX-queue (after all other packet processing).
> 
> --flush-timer [delay]
> Flush the packet queue if no packets have been seen during
> [delay]. As long as packets are seen, the timer is reset.
> 

I understand your desire to have some realistic fwd scenario,
but why it all have to be put in iowfd mode?
 iowfd is the simplest one, mainly used to test raw PMD pefomance
in nearly ideal conditions.
Why not to create your own forwarding mode (as most people do)?
That way you'll have your 'real world app' test scenario,
while keeping iofwd code small and simple.
Konstantin 

> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
> ---
>  app/test-pmd/fifo.h       | 43 ++++++++++++++++++++++++++++++++++
>  app/test-pmd/iofwd.c      | 59 ++++++++++++++++++++++++++++++++++++++++++++++-
>  app/test-pmd/parameters.c | 21 ++++++++++++++++-
>  app/test-pmd/testpmd.c    | 48 ++++++++++++++++++++++++++++++++++++++
>  app/test-pmd/testpmd.h    | 15 ++++++++++++
>  config/common_base        |  1 +
>  6 files changed, 185 insertions(+), 2 deletions(-)
>  create mode 100644 app/test-pmd/fifo.h
> 
> diff --git a/app/test-pmd/fifo.h b/app/test-pmd/fifo.h
> new file mode 100644
> index 000000000..01415f98c
> --- /dev/null
> +++ b/app/test-pmd/fifo.h
> @@ -0,0 +1,43 @@
> +#ifndef __FIFO_H
> +#define __FIFO_H
> +#include <stdint.h>
> +#include <rte_mbuf.h>
> +#include <rte_ring.h>
> +#include "testpmd.h"
> +
> +#define FIFO_COUNT_MAX 1024
> +/**
> + * Add elements to fifo. Return number of written elements
> + */
> +static inline unsigned
> +fifo_put(struct rte_ring *r, struct rte_mbuf **data, unsigned num)
> +{
> +
> +	return rte_ring_enqueue_burst(r, (void **)data, num, NULL);
> +}
> +
> +/**
> + * Get elements from fifo. Return number of read elements
> + */
> +static inline unsigned
> +fifo_get(struct rte_ring *r, struct rte_mbuf **data, unsigned num)
> +{
> +	return rte_ring_dequeue_burst(r, (void **) data, num, NULL);
> +}
> +
> +static inline unsigned
> +fifo_count(struct rte_ring *r)
> +{
> +	return rte_ring_count(r);
> +}
> +
> +static inline int
> +fifo_full(struct rte_ring *r)
> +{
> +	return rte_ring_full(r);
> +}
> +
> +
> +struct rte_ring * fifo_init(uint32_t qi, uint32_t pi);
> +
> +#endif
> diff --git a/app/test-pmd/iofwd.c b/app/test-pmd/iofwd.c
> index 9dce76efe..85fa000f7 100644
> --- a/app/test-pmd/iofwd.c
> +++ b/app/test-pmd/iofwd.c
> @@ -36,6 +36,7 @@
>  #include <rte_flow.h>
> 
>  #include "testpmd.h"
> +#include "fifo.h"
> 
>  /*
>   * Forwarding of packets in I/O mode.
> @@ -48,7 +49,7 @@ pkt_burst_io_forward(struct fwd_stream *fs)
>  {
>  	struct rte_mbuf *pkts_burst[MAX_PKT_BURST];
>  	uint16_t nb_rx;
> -	uint16_t nb_tx;
> +	uint16_t nb_tx = 0;
>  	uint32_t retry;
> 
>  #ifdef RTE_TEST_PMD_RECORD_CORE_CYCLES
> @@ -56,6 +57,15 @@ pkt_burst_io_forward(struct fwd_stream *fs)
>  	uint64_t end_tsc;
>  	uint64_t core_cycles;
>  #endif
> +#ifdef RTE_TEST_PMD_NOISY
> +	const uint64_t freq_khz = rte_get_timer_hz() / 1000;
> +	struct noisy_config *ncf = &noisy_cfg[fs->tx_queue];
> +	struct rte_mbuf *tmp_pkts[MAX_PKT_BURST];
> +	uint16_t nb_enqd;
> +	uint16_t nb_deqd = 0;
> +	uint64_t delta_ms;
> +	uint64_t now;
> +#endif
> 
>  #ifdef RTE_TEST_PMD_RECORD_CORE_CYCLES
>  	start_tsc = rte_rdtsc();
> @@ -73,8 +83,55 @@ pkt_burst_io_forward(struct fwd_stream *fs)
>  #ifdef RTE_TEST_PMD_RECORD_BURST_STATS
>  	fs->rx_burst_stats.pkt_burst_spread[nb_rx]++;
>  #endif
> +#ifdef RTE_TEST_PMD_NOISY
> +	if (bsize_before_send > 0) {
> +		if (rte_ring_free_count(ncf->f) >= nb_rx) {
> +			/* enqueue into fifo */
> +			nb_enqd = fifo_put(ncf->f, pkts_burst, nb_rx);
> +			if (nb_enqd < nb_rx)
> +				nb_rx = nb_enqd;
> +		} else {
> +			/* fifo is full, dequeue first */
> +			nb_deqd = fifo_get(ncf->f, tmp_pkts, nb_rx);
> +			/* enqueue into fifo */
> +			nb_enqd = fifo_put(ncf->f, pkts_burst, nb_deqd);
> +			if (nb_enqd < nb_rx)
> +				nb_rx = nb_enqd;
> +			if (nb_deqd > 0)
> +				nb_tx = rte_eth_tx_burst(fs->tx_port,
> +						fs->tx_queue, tmp_pkts,
> +						nb_deqd);
> +		}
> +	} else {
> +		nb_tx = rte_eth_tx_burst(fs->tx_port, fs->tx_queue,
> +				pkts_burst, nb_rx);
> +	}
> +
> +	/*
> +	 * TX burst queue drain
> +	 */
> +	if (ncf->prev_time == 0) {
> +		now = ncf->prev_time = rte_get_timer_cycles();
> +	} else {
> +		now = rte_get_timer_cycles();
> +	}
> +	delta_ms = (now - ncf->prev_time) / freq_khz;
> +	if (unlikely(delta_ms >= flush_timer) && flush_timer > 0 && (nb_tx == 0)) {
> +		while (fifo_count(ncf->f) > 0) {
> +			nb_deqd = fifo_get(ncf->f, tmp_pkts, nb_rx);
> +			nb_tx = rte_eth_tx_burst(fs->tx_port, fs->tx_queue,
> +						 tmp_pkts, nb_deqd);
> +			if(rte_ring_empty(ncf->f))
> +				break;
> +		}
> +		ncf->prev_time = now;
> +	}
> +	if (nb_tx < nb_rx && fs->retry_enabled)
> +		*pkts_burst = *tmp_pkts;
> +#else
>  	nb_tx = rte_eth_tx_burst(fs->tx_port, fs->tx_queue,
>  			pkts_burst, nb_rx);
> +#endif
>  	/*
>  	 * Retry if necessary
>  	 */
> diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
> index 2192bdcdf..df0db933a 100644
> --- a/app/test-pmd/parameters.c
> +++ b/app/test-pmd/parameters.c
> @@ -621,6 +621,8 @@ launch_args_parse(int argc, char** argv)
>  		{ "print-event",		1, 0, 0 },
>  		{ "mask-event",			1, 0, 0 },
>  		{ "tx-offloads",		1, 0, 0 },
> +		{ "buffersize-before-sending",  1, 0, 0 },
> +		{ "flush-timer",                1, 0, 0 },
>  		{ 0, 0, 0, 0 },
>  	};
> 
> @@ -1101,7 +1103,24 @@ launch_args_parse(int argc, char** argv)
>  					rte_exit(EXIT_FAILURE,
>  						 "invalid mask-event argument\n");
>  				}
> -
> +#ifdef RTE_TEST_PMD_NOISY
> +			if (!strcmp(lgopts[opt_idx].name, "buffersize-before-sending")) {
> +				n = atoi(optarg);
> +				if (n > 0)
> +					bsize_before_send = (uint16_t) n;
> +				else
> +					rte_exit(EXIT_FAILURE,
> +						 "buffersize-before-sending must be > 0\n");
> +			}
> +			if (!strcmp(lgopts[opt_idx].name, "flush-timer")) {
> +				n = atoi(optarg);
> +				if (n >= 0)
> +					flush_timer = (uint16_t) n;
> +				else
> +					rte_exit(EXIT_FAILURE,
> +						 "flush-timer must be > 0\n");
> +			}
> +#endif
>  			break;
>  		case 'h':
>  			usage(argv[0]);
> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
> index 4c0e2586c..7b8ffdc9c 100644
> --- a/app/test-pmd/testpmd.c
> +++ b/app/test-pmd/testpmd.c
> @@ -59,6 +59,9 @@
>  #ifdef RTE_LIBRTE_LATENCY_STATS
>  #include <rte_latencystats.h>
>  #endif
> +#ifdef RTE_TEST_PMD_NOISY
> +#include "fifo.h"
> +#endif
> 
>  #include "testpmd.h"
> 
> @@ -249,6 +252,18 @@ int16_t tx_free_thresh = RTE_PMD_PARAM_UNSET;
>   */
>  int16_t tx_rs_thresh = RTE_PMD_PARAM_UNSET;
> 
> +#ifdef RTE_TEST_PMD_NOISY
> +/*
> + * Configurable value of buffered packed before sending.
> + */
> +uint16_t bsize_before_send = 0;
> +
> +/*
> + * Configurable value of packet buffer timeout.
> + */
> +uint16_t flush_timer = 0;
> +#endif
> +
>  /*
>   * Receive Side Scaling (RSS) configuration.
>   */
> @@ -401,6 +416,20 @@ static int all_ports_started(void);
>  struct gso_status gso_ports[RTE_MAX_ETHPORTS];
>  uint16_t gso_max_segment_size = ETHER_MAX_LEN - ETHER_CRC_LEN;
> 
> +#ifdef RTE_TEST_PMD_NOISY
> +#define STRSIZE 256
> +#define NOISY_RING "noisy_ring_%d:%d\n"
> +struct rte_ring * fifo_init(uint32_t qi, uint32_t pi)
> +{
> +	struct noisy_config *n = &noisy_cfg[qi];
> +	char name[STRSIZE];
> +
> +	snprintf(name, STRSIZE, NOISY_RING, pi, qi);
> +	n->f = rte_ring_create(name, bsize_before_send, rte_socket_id(), 0);
> +	return n->f;
> +}
> +#endif
> +
>  /*
>   * Helper function to check if socket is already discovered.
>   * If yes, return positive value. If not, return zero.
> @@ -1584,6 +1613,16 @@ start_port(portid_t pid)
>  				return -1;
>  			}
>  		}
> +#ifdef RTE_TEST_PMD_NOISY
> +		noisy_cfg = (struct noisy_config *) rte_zmalloc("testpmd noisy fifo and timers",
> +					nb_txq * sizeof(struct noisy_config),
> +					RTE_CACHE_LINE_SIZE);
> +		if (noisy_cfg == NULL) {
> +			rte_exit(EXIT_FAILURE,
> +					"rte_zmalloc(%d struct noisy_config) failed\n",
> +					(int)(nb_txq * sizeof(struct noisy_config)));
> +		}
> +#endif
>  		if (port->need_reconfig_queues > 0) {
>  			port->need_reconfig_queues = 0;
>  			port->tx_conf.txq_flags = ETH_TXQ_FLAGS_IGNORE;
> @@ -1591,6 +1630,11 @@ start_port(portid_t pid)
>  			port->tx_conf.offloads = port->dev_conf.txmode.offloads;
>  			/* setup tx queues */
>  			for (qi = 0; qi < nb_txq; qi++) {
> +#ifdef RTE_TEST_PMD_NOISY
> +				if (!fifo_init(qi, pi) && bsize_before_send > 0)
> +					rte_exit(EXIT_FAILURE, "%s\n",
> +						 rte_strerror(rte_errno));
> +#endif
>  				if ((numa_support) &&
>  					(txring_numa[pi] != NUMA_NO_CONFIG))
>  					diag = rte_eth_tx_queue_setup(pi, qi,
> @@ -1755,6 +1799,10 @@ stop_port(portid_t pid)
>  			RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>  			printf("Port %d can not be set into stopped\n", pi);
>  		need_check_link_status = 1;
> +
> +#ifdef  RTE_TEST_PMD_NOISY
> +		rte_free(noisy_cfg);
> +#endif
>  	}
>  	if (need_check_link_status && !no_link_check)
>  		check_all_ports_link_status(RTE_PORT_ALL);
> diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
> index 153abea05..a6c1a17bb 100644
> --- a/app/test-pmd/testpmd.h
> +++ b/app/test-pmd/testpmd.h
> @@ -9,6 +9,9 @@
>  #include <rte_bus_pci.h>
>  #include <rte_gro.h>
>  #include <rte_gso.h>
> +#ifdef RTE_TEST_PMD_NOISY
> +#include "fifo.h"
> +#endif
> 
>  #define RTE_PORT_ALL            (~(portid_t)0x0)
> 
> @@ -109,6 +112,15 @@ struct fwd_stream {
>  #endif
>  };
> 
> +#ifdef RTE_TEST_PMD_NOISY
> +struct noisy_config {
> +	struct rte_ring *f;
> +	uint64_t prev_time;
> +};
> +struct noisy_config *noisy_cfg;
> +#endif
> +
> +
>  /** Descriptor for a single flow. */
>  struct port_flow {
>  	size_t size; /**< Allocated space including data[]. */
> @@ -382,6 +394,9 @@ extern int8_t rx_drop_en;
>  extern int16_t tx_free_thresh;
>  extern int16_t tx_rs_thresh;
> 
> +extern uint16_t bsize_before_send;
> +extern uint16_t flush_timer;
> +
>  extern uint8_t dcb_config;
>  extern uint8_t dcb_test;
> 
> diff --git a/config/common_base b/config/common_base
> index c09c7cf88..2fd17439b 100644
> --- a/config/common_base
> +++ b/config/common_base
> @@ -806,6 +806,7 @@ CONFIG_RTE_PROC_INFO=n
>  CONFIG_RTE_TEST_PMD=y
>  CONFIG_RTE_TEST_PMD_RECORD_CORE_CYCLES=n
>  CONFIG_RTE_TEST_PMD_RECORD_BURST_STATS=n
> +CONFIG_RTE_TEST_PMD_NOISY=y
> 
>  #
>  # Compile the bbdev test application
> --
> 2.14.3

Re: [PATCH 1/2] testpmd: add parameters buffersize-before-send and flush-timeout

[Ferruh Yigit]

On 4/12/2018 3:57 PM, Ananyev, Konstantin wrote:
> Hi,
> 
>>
>> Create a fifo to buffer received packets. Once it flows over put
>> those packets into the actual tx queue. The fifo is created per tx
>> queue and its size can be set with the --buffersize-before-sending
>> commandline parameter.
>>
>> A second commandline parameter is used to set a timeout in
>> milliseconds after which the fifo is flushed.
>>
>> --buffersize-before-sending [packet numbers]
>> Keep the mbuf in a FIFO and forward the over flooding packets from the
>> FIFO. This queue is per TX-queue (after all other packet processing).
>>
>> --flush-timer [delay]
>> Flush the packet queue if no packets have been seen during
>> [delay]. As long as packets are seen, the timer is reset.
>>
> 
> I understand your desire to have some realistic fwd scenario,
> but why it all have to be put in iowfd mode?
>  iowfd is the simplest one, mainly used to test raw PMD pefomance
> in nearly ideal conditions.
> Why not to create your own forwarding mode (as most people do)?
> That way you'll have your 'real world app' test scenario,
> while keeping iofwd code small and simple.

+1 to having own forwarding mode for noisy neighbor, and leaving iofwd simple.

> Konstantin 
> 
>> Signed-off-by: Jens Freimann <jfreimann@redhat.com>

<...>

Re: [PATCH 1/2] testpmd: add parameters buffersize-before-send and flush-timeout

[Jens Freimann]

On Thu, Apr 12, 2018 at 05:30:54PM +0100, Ferruh Yigit wrote:
>On 4/12/2018 3:57 PM, Ananyev, Konstantin wrote:
>> Hi,
>>
>>>
>>> Create a fifo to buffer received packets. Once it flows over put
>>> those packets into the actual tx queue. The fifo is created per tx
>>> queue and its size can be set with the --buffersize-before-sending
>>> commandline parameter.
>>>
>>> A second commandline parameter is used to set a timeout in
>>> milliseconds after which the fifo is flushed.
>>>
>>> --buffersize-before-sending [packet numbers]
>>> Keep the mbuf in a FIFO and forward the over flooding packets from the
>>> FIFO. This queue is per TX-queue (after all other packet processing).
>>>
>>> --flush-timer [delay]
>>> Flush the packet queue if no packets have been seen during
>>> [delay]. As long as packets are seen, the timer is reset.
>>>
>>
>> I understand your desire to have some realistic fwd scenario,
>> but why it all have to be put in iowfd mode?
>>  iowfd is the simplest one, mainly used to test raw PMD pefomance
>> in nearly ideal conditions.
>> Why not to create your own forwarding mode (as most people do)?
>> That way you'll have your 'real world app' test scenario,
>> while keeping iofwd code small and simple.
>
>+1 to having own forwarding mode for noisy neighbor, and leaving iofwd simple.

I'm okay with a new forwarding mode. I'll send a v2. 
Thanks for the review Konstantin and Ferruh!

regards,
Jens