[RFC PATCH 0/4] Convert lpm data from header to resource

[RFC PATCH 0/4] Convert lpm data from header to resource

[Bruce Richardson]

This patchset is based on the work done by Jan Viktorin to add resource
handling to the test app. [1]
It takes the existing lpm large routing table information and converts it
from a header file included at compile time to a resource linked in. This
improves things in two ways:
1. Improves DPDK build time
2. Removes approx 1 million lines of code from our LOC counts, as the header
file no longer counts as code, but more correctly as data.

Before and after approx code stats (with some additional patchsets applied).
Notice how the app folder has dropped from being the biggest component with
6x the drivers code to second place with less code than the drivers.
Future work is to repeat the same process for lpm6.

SLOC    Directory       SLOC-by-Language (Sorted)
1237464 app             ansic=1236324,python=1140
192107  drivers         ansic=192107
108109  lib             ansic=107968,python=141
65447   examples        ansic=65312,sh=135
849     tools           python=444,sh=405
799     scripts         sh=799
77      doc             python=75,sh=2
0       config          (none)
0       mk              (none)
0       pkg             (none)
0       top_dir         (none)


Totals grouped by language (dominant language first):
ansic:      1601711 (99.80%)
python:        1800 (0.11%)
sh:            1341 (0.08%)

-----------

SLOC    Directory       SLOC-by-Language (Sorted)
192107  drivers         ansic=192107
160646  app             ansic=159506,python=1140
108109  lib             ansic=107968,python=141
65447   examples        ansic=65312,sh=135
849     tools           python=444,sh=405
799     scripts         sh=799
77      doc             python=75,sh=2
0       config          (none)
0       mk              (none)
0       pkg             (none)
0       top_dir         (none)


Totals grouped by language (dominant language first):
ansic:       524893 (99.41%)
python:        1800 (0.34%)
sh:            1341 (0.25%)

generated using David A. Wheeler's 'SLOCCount'

[1] http://dpdk.org/ml/archives/dev/2016-April/038145.html

Bruce Richardson (4):
  test: add lpm routes as a linked resource
  test: make all lpm routes be of unsigned type
  test: change lpm test to use routes as resource
  test: change lpm routes file from header to data file

 app/test/Makefile                                  |  1 +
 app/test/test_lpm.c                                | 73 +++++++++++++++++++++-
 .../{test_lpm_routes.h => test_lpm_routes.dat}     |  2 +-
 3 files changed, 74 insertions(+), 2 deletions(-)
 rename app/test/{test_lpm_routes.h => test_lpm_routes.dat} (99%)

-- 
2.5.5

[RFC PATCH 1/4] test: add lpm routes as a linked resource

[Bruce Richardson]

Since we now have the ability to store resource data directly inside
the test binary, take the test_lpm_routes.h header file and make it a
test resource. Later commits will then switch the test C file over to use
this rather than including it directly.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/Makefile   | 1 +
 app/test/test_lpm.c | 3 +++
 2 files changed, 4 insertions(+)

diff --git a/app/test/Makefile b/app/test/Makefile
index 90acd63..94ba990 100644
--- a/app/test/Makefile
+++ b/app/test/Makefile
@@ -125,6 +125,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_HASH) += test_hash_perf.c
 SRCS-$(CONFIG_RTE_LIBRTE_HASH) += test_hash_functions.c
 SRCS-$(CONFIG_RTE_LIBRTE_HASH) += test_hash_scaling.c
 
+$(eval $(call resource,test_lpm_data,test_lpm_routes.h))
 SRCS-$(CONFIG_RTE_LIBRTE_LPM) += test_lpm.c
 SRCS-$(CONFIG_RTE_LIBRTE_LPM) += test_lpm6.c
 
diff --git a/app/test/test_lpm.c b/app/test/test_lpm.c
index 40fbbc6..d2aa5de 100644
--- a/app/test/test_lpm.c
+++ b/app/test/test_lpm.c
@@ -46,11 +46,14 @@
 #include <time.h>
 
 #include "test.h"
+#include "resource.h"
 
 #include "rte_lpm.h"
 #include "test_lpm_routes.h"
 #include "test_xmmt_ops.h"
 
+REGISTER_LINKED_RESOURCE(test_lpm_data)
+
 #define TEST_LPM_ASSERT(cond) do {                                            \
 	if (!(cond)) {                                                        \
 		printf("Error at line %d: \n", __LINE__);                     \
-- 
2.5.5

[RFC PATCH 2/4] test: make all lpm routes be of unsigned type

[Bruce Richardson]

The one route that was different to the others in the test_lpm_routes.h
file was the entry "{0, 8}" which was the only route without a "U" after
the IP part. Add in the extra "U" to that entry so that it can be used
as a check character when parsing routes manually.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/test_lpm_routes.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/app/test/test_lpm_routes.h b/app/test/test_lpm_routes.h
index 023b0f9..1decfee 100644
--- a/app/test/test_lpm_routes.h
+++ b/app/test/test_lpm_routes.h
@@ -281825,7 +281825,7 @@ static const struct route_rule large_route_table[] =
 	{3536977920U, 17},
 	{3392992768U, 23},
 	{3341675008U, 23},
-	{0, 8},
+	{0U, 8},
 	{3326316168U, 30},
 	{3326316108U, 30},
 	{3326316060U, 30},
-- 
2.5.5

[RFC PATCH 3/4] test: change lpm test to use routes as resource

[Bruce Richardson]

Change the lpm autotest to use the routes data from the resource data
stored in the binary rather than including it directly into the C file
as a C header. This speeds up compile and link time, without changing
the test results.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/test_lpm.c | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 69 insertions(+), 1 deletion(-)

diff --git a/app/test/test_lpm.c b/app/test/test_lpm.c
index d2aa5de..4234a52 100644
--- a/app/test/test_lpm.c
+++ b/app/test/test_lpm.c
@@ -34,6 +34,7 @@
 #include <stdio.h>
 #include <stdint.h>
 #include <stdlib.h>
+#include <string.h>
 #include <errno.h>
 #include <sys/queue.h>
 
@@ -44,16 +45,23 @@
 #include <rte_branch_prediction.h>
 #include <rte_ip.h>
 #include <time.h>
+#include <rte_malloc.h>
 
 #include "test.h"
 #include "resource.h"
 
 #include "rte_lpm.h"
-#include "test_lpm_routes.h"
 #include "test_xmmt_ops.h"
 
 REGISTER_LINKED_RESOURCE(test_lpm_data)
 
+struct route_rule {
+	uint32_t ip;
+	uint8_t depth;
+};
+static struct route_rule *large_route_table;
+static unsigned int NUM_ROUTE_ENTRIES;
+
 #define TEST_LPM_ASSERT(cond) do {                                            \
 	if (!(cond)) {                                                        \
 		printf("Error at line %d: \n", __LINE__);                     \
@@ -1427,6 +1435,64 @@ perf_test(void)
 	return PASS;
 }
 
+static int
+load_large_route_table(void)
+{
+	const struct resource *r;
+	const char *lpm_data;
+
+	r = resource_find("test_lpm_data");
+	TEST_ASSERT_NOT_NULL(r, "No large lpm table data found");
+
+	/* the routing table size is going to be less than the size of the
+	 * resource, since text extries are more verbose. Allocate this as
+	 * the max size, and shrink the allocation later
+	 */
+	large_route_table = rte_malloc(NULL, resource_size(r), 0);
+	if (large_route_table == NULL)
+		return -1;
+
+	/* parse the lpm table. All entries are of format:
+	 *  {IP-as-decimal-unsigned, depth}
+	 * For example:
+	 *  {1234567U, 24},
+	 * We use the "U" and "}" characters as format check characters,
+	 * after parsing each number.
+	 */
+	for (lpm_data = r->beg; lpm_data < r->end; lpm_data++) {
+		if (*lpm_data == '{') {
+			char *endptr;
+
+			lpm_data++;
+			large_route_table[NUM_ROUTE_ENTRIES].ip = \
+				strtoul(lpm_data, &endptr, 0);
+			if (*endptr != 'U') {
+				if (NUM_ROUTE_ENTRIES > 0)
+					printf("Failed parse of %s\n",
+						strndupa(lpm_data, 12));
+				continue;
+			}
+
+			lpm_data = endptr + 2; /* skip U and , */
+			large_route_table[NUM_ROUTE_ENTRIES].depth = \
+				strtoul(lpm_data, &endptr, 0);
+			if (*endptr != '}') {
+				if (NUM_ROUTE_ENTRIES > 0)
+					printf("Failed parse of %s\n",
+						strndupa(lpm_data, 5));
+				continue;
+			}
+
+			NUM_ROUTE_ENTRIES++;
+		}
+	}
+
+	large_route_table = rte_realloc(large_route_table,
+		sizeof(large_route_table[0]) * NUM_ROUTE_ENTRIES, 0);
+	printf("Read %u route entries\n", NUM_ROUTE_ENTRIES);
+	return 0;
+}
+
 /*
  * Do all unit and performance tests.
  */
@@ -1437,6 +1503,8 @@ test_lpm(void)
 	unsigned i;
 	int status, global_status = 0;
 
+	TEST_ASSERT_SUCCESS(load_large_route_table(), "Error loading lpm table");
+
 	for (i = 0; i < NUM_LPM_TESTS; i++) {
 		status = tests[i]();
 		if (status < 0) {
-- 
2.5.5

[RFC PATCH 4/4] test: change lpm routes file from header to data file

[Bruce Richardson]

Change the file extension of the test_lpm_routes file from .h to .dat.
This makes the lines-of-code counts for DPDK more realistic as they are not
affected by the huge counts from the lpm data.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/Makefile                                   | 2 +-
 app/test/{test_lpm_routes.h => test_lpm_routes.dat} | 0
 2 files changed, 1 insertion(+), 1 deletion(-)
 rename app/test/{test_lpm_routes.h => test_lpm_routes.dat} (100%)

diff --git a/app/test/Makefile b/app/test/Makefile
index 94ba990..9107a91 100644
--- a/app/test/Makefile
+++ b/app/test/Makefile
@@ -125,7 +125,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_HASH) += test_hash_perf.c
 SRCS-$(CONFIG_RTE_LIBRTE_HASH) += test_hash_functions.c
 SRCS-$(CONFIG_RTE_LIBRTE_HASH) += test_hash_scaling.c
 
-$(eval $(call resource,test_lpm_data,test_lpm_routes.h))
+$(eval $(call resource,test_lpm_data,test_lpm_routes.dat))
 SRCS-$(CONFIG_RTE_LIBRTE_LPM) += test_lpm.c
 SRCS-$(CONFIG_RTE_LIBRTE_LPM) += test_lpm6.c
 
diff --git a/app/test/test_lpm_routes.h b/app/test/test_lpm_routes.dat
similarity index 100%
rename from app/test/test_lpm_routes.h
rename to app/test/test_lpm_routes.dat
-- 
2.5.5

Re: [RFC PATCH 3/4] test: change lpm test to use routes as resource

[Bruce Richardson]

On Fri, May 06, 2016 at 01:51:31PM +0100, Bruce Richardson wrote:
> Change the lpm autotest to use the routes data from the resource data
> stored in the binary rather than including it directly into the C file
> as a C header. This speeds up compile and link time, without changing
> the test results.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---
>  app/test/test_lpm.c | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 69 insertions(+), 1 deletion(-)
> 
> diff --git a/app/test/test_lpm.c b/app/test/test_lpm.c
> index d2aa5de..4234a52 100644
> --- a/app/test/test_lpm.c
> +++ b/app/test/test_lpm.c
> @@ -34,6 +34,7 @@
>  #include <stdio.h>
>  #include <stdint.h>
>  #include <stdlib.h>
> +#include <string.h>
>  #include <errno.h>
>  #include <sys/queue.h>
>  
> @@ -44,16 +45,23 @@
>  #include <rte_branch_prediction.h>
>  #include <rte_ip.h>
>  #include <time.h>
> +#include <rte_malloc.h>
>  
>  #include "test.h"
>  #include "resource.h"
>  
>  #include "rte_lpm.h"
> -#include "test_lpm_routes.h"
>  #include "test_xmmt_ops.h"
>  
>  REGISTER_LINKED_RESOURCE(test_lpm_data)
>  
> +struct route_rule {
> +	uint32_t ip;
> +	uint8_t depth;
> +};
> +static struct route_rule *large_route_table;
> +static unsigned int NUM_ROUTE_ENTRIES;
> +
Just as an extra note, I had planned a 5th and final patch to rename this variable
to have a lower-case name, but I forgot to do so when preparing this RFC. If this
patchset seems good to people, I'll include that cleanup in a proper V1 submission.

/Bruce

Re: [RFC PATCH 0/4] Convert lpm data from header to resource

[Thomas Monjalon]

2016-05-06 13:51, Bruce Richardson:
> It takes the existing lpm large routing table information and converts it
> from a header file included at compile time to a resource linked in. This
> improves things in two ways:
> 1. Improves DPDK build time
> 2. Removes approx 1 million lines of code from our LOC counts, as the header
> file no longer counts as code, but more correctly as data.

Good improvement!
Is it possible to go further and
3. Reduce size of the DPDK tarball
by removing some test data?
What is the benefit of having 21 MB and 7 MB of routes data?

[PATCH 0/4] Convert lpm data from header to resource

[Nikita Kozlov]

Re-submission of Bruce Richardson patch set:
http://dpdk.org/dev/patchwork/patch/12484/

This patchset is a rebase to the current head and use of 
strndupa (not posix) was replaced.

Original cover letter:

This patchset is based on the work done by Jan Viktorin to add resource
handling to the test app. [1]
It takes the existing lpm large routing table information and converts it
from a header file included at compile time to a resource linked in. This
improves things in two ways:
1. Improves DPDK build time
2. Removes approx 1 million lines of code from our LOC counts, as the header
file no longer counts as code, but more correctly as data.

Before and after approx code stats (with some additional patchsets applied).
Notice how the app folder has dropped from being the biggest component with
6x the drivers code to second place with less code than the drivers.
Future work is to repeat the same process for lpm6.

SLOC    Directory       SLOC-by-Language (Sorted)
1237464 app             ansic=1236324,python=1140
192107  drivers         ansic=192107
108109  lib             ansic=107968,python=141
65447   examples        ansic=65312,sh=135
849     tools           python=444,sh=405
799     scripts         sh=799
77      doc             python=75,sh=2
0       config          (none)
0       mk              (none)
0       pkg             (none)
0       top_dir         (none)


Totals grouped by language (dominant language first):
ansic:      1601711 (99.80%)
python:        1800 (0.11%)
sh:            1341 (0.08%)

-----------

SLOC    Directory       SLOC-by-Language (Sorted)
192107  drivers         ansic=192107
160646  app             ansic=159506,python=1140
108109  lib             ansic=107968,python=141
65447   examples        ansic=65312,sh=135
849     tools           python=444,sh=405
799     scripts         sh=799
77      doc             python=75,sh=2
0       config          (none)
0       mk              (none)
0       pkg             (none)
0       top_dir         (none)


Totals grouped by language (dominant language first):
ansic:       524893 (99.41%)
python:        1800 (0.34%)
sh:            1341 (0.25%)

generated using David A. Wheeler's 'SLOCCount'

[1] http://dpdk.org/ml/archives/dev/2016-April/038145.html

Nikita Kozlov (4):
  test: make all lpm routes be of unsigned type
  test: change lpm routes file from header to data file
  test: change lpm test to use routes as resource
  test: change lpm perf test to use routes as resource

 app/test/Makefile                                  |  1 +
 app/test/test_lpm.c                                | 73 ++++++++++++++++++-
 app/test/test_lpm_perf.c                           | 83 ++++++++++++++++++++--
 .../{test_lpm_routes.h => test_lpm_routes.dat}     |  2 +-
 4 files changed, 150 insertions(+), 9 deletions(-)
 rename app/test/{test_lpm_routes.h => test_lpm_routes.dat} (99%)

-- 
2.8.1

[PATCH 1/4] test: make all lpm routes be of unsigned type

[Nikita Kozlov]

The one route that was different to the others in the test_lpm_routes.h
file was the entry "{0, 8}" which was the only route without a "U" after
the IP part. Add in the extra "U" to that entry so that it can be used
as a check character when parsing routes manually.

Signed-off-by: Nikita Kozlov <nikita@elyzion.net>
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/test_lpm_routes.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/app/test/test_lpm_routes.h b/app/test/test_lpm_routes.h
index 023b0f9..1decfee 100644
--- a/app/test/test_lpm_routes.h
+++ b/app/test/test_lpm_routes.h
@@ -281825,7 +281825,7 @@ static const struct route_rule large_route_table[] =
 	{3536977920U, 17},
 	{3392992768U, 23},
 	{3341675008U, 23},
-	{0, 8},
+	{0U, 8},
 	{3326316168U, 30},
 	{3326316108U, 30},
 	{3326316060U, 30},
-- 
2.8.1

[PATCH 2/4] test: change lpm routes file from header to data file

[Nikita Kozlov]

Change the file extension of the test_lpm_routes file from .h to .dat.
This makes the lines-of-code counts for DPDK more realistic as they are not
affected by the huge counts from the lpm data.

Signed-off-by: Nikita Kozlov <nikita@elyzion.net>
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/Makefile                                   | 1 +
 app/test/{test_lpm_routes.h => test_lpm_routes.dat} | 0
 2 files changed, 1 insertion(+)
 rename app/test/{test_lpm_routes.h => test_lpm_routes.dat} (100%)

diff --git a/app/test/Makefile b/app/test/Makefile
index 7e4d484..a3f317d 100644
--- a/app/test/Makefile
+++ b/app/test/Makefile
@@ -118,6 +118,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_HASH) += test_hash_perf.c
 SRCS-$(CONFIG_RTE_LIBRTE_HASH) += test_hash_functions.c
 SRCS-$(CONFIG_RTE_LIBRTE_HASH) += test_hash_scaling.c
 
+$(eval $(call linked_resource,test_lpm_data,test_lpm_routes.dat))
 SRCS-$(CONFIG_RTE_LIBRTE_LPM) += test_lpm.c
 SRCS-$(CONFIG_RTE_LIBRTE_LPM) += test_lpm_perf.c
 SRCS-$(CONFIG_RTE_LIBRTE_LPM) += test_lpm6.c
diff --git a/app/test/test_lpm_routes.h b/app/test/test_lpm_routes.dat
similarity index 100%
rename from app/test/test_lpm_routes.h
rename to app/test/test_lpm_routes.dat
-- 
2.8.1

[PATCH 3/4] test: change lpm test to use routes as resource

[Nikita Kozlov]

Change the lpm autotest to use the routes data from the resource data
stored in the binary rather than including it directly into the C file
as a C header. This speeds up compile and link time, without changing
the test results.

Signed-off-by: Nikita Kozlov <nikita@elyzion.net>
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/test_lpm.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 72 insertions(+), 1 deletion(-)

diff --git a/app/test/test_lpm.c b/app/test/test_lpm.c
index f6930fb..de54720 100644
--- a/app/test/test_lpm.c
+++ b/app/test/test_lpm.c
@@ -34,13 +34,26 @@
 #include <stdio.h>
 #include <stdint.h>
 #include <stdlib.h>
+#include <string.h>
 
 #include <rte_lpm.h>
+#include <rte_ip.h>
+#include <rte_malloc.h>
 
 #include "test.h"
-#include "test_lpm_routes.h"
+#include "resource.h"
+
 #include "test_xmmt_ops.h"
 
+REGISTER_LINKED_RESOURCE(test_lpm_data)
+
+struct route_rule {
+	uint32_t ip;
+	uint8_t depth;
+};
+static struct route_rule *large_route_table;
+static unsigned int num_route_entries;
+
 #define TEST_LPM_ASSERT(cond) do {                                            \
 	if (!(cond)) {                                                        \
 		printf("Error at line %d: \n", __LINE__);                     \
@@ -1217,6 +1230,62 @@ test17(void)
 	return PASS;
 }
 
+static int
+load_large_route_table(void)
+{
+	const struct resource *r;
+	const char *lpm_data;
+
+	r = resource_find("test_lpm_data");
+	TEST_ASSERT_NOT_NULL(r, "No large lpm table data found");
+
+	/* the routing table size is going to be less than the size of the
+	 * resource, since text extries are more verbose. Allocate this as
+	 * the max size, and shrink the allocation later
+	 */
+	large_route_table = rte_malloc(NULL, resource_size(r), 0);
+	if (large_route_table == NULL)
+		return -1;
+
+	/* parse the lpm table. All entries are of format:
+	 *  {IP-as-decimal-unsigned, depth}
+	 * For example:
+	 *  {1234567U, 24},
+	 * We use the "U" and "}" characters as format check characters,
+	 * after parsing each number.
+	 */
+	for (lpm_data = r->begin; lpm_data < r->end; lpm_data++) {
+		if (*lpm_data == '{') {
+			char *endptr;
+
+			lpm_data++;
+			large_route_table[num_route_entries].ip = \
+				strtoul(lpm_data, &endptr, 0);
+			if (*endptr != 'U') {
+				if (num_route_entries > 0)
+					printf("Failed parse of %.*s\n", 12, lpm_data);
+				continue;
+			}
+
+			lpm_data = endptr + 2; /* skip U and , */
+			large_route_table[num_route_entries].depth = \
+				strtoul(lpm_data, &endptr, 0);
+			if (*endptr != '}') {
+				if (num_route_entries > 0)
+					printf("Failed parse of %.*s\n",5, lpm_data);
+				continue;
+			}
+
+			num_route_entries++;
+		}
+	}
+
+	large_route_table = rte_realloc(large_route_table,
+		sizeof(large_route_table[0]) * num_route_entries, 0);
+	printf("Read %u route entries\n", num_route_entries);
+	return 0;
+}
+
 /*
  * Do all unit tests.
  */
@@ -1227,6 +1296,8 @@ test_lpm(void)
 	unsigned i;
 	int status, global_status = 0;
 
+	TEST_ASSERT_SUCCESS(load_large_route_table(), "Error loading lpm table");
+
 	for (i = 0; i < NUM_LPM_TESTS; i++) {
 		status = tests[i]();
 		if (status < 0) {
-- 
2.8.1

[PATCH 4/4] test: change lpm perf test to use routes as resource

[Nikita Kozlov]

Change the lpm perf autotest to use the routes data from the resource data
stored in the binary rather than including it directly into the C file
as a C header. This speeds up compile and link time, without changing
the test results.

Signed-off-by: Nikita Kozlov <nikita@elyzion.net>
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/test_lpm_perf.c | 83 ++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 76 insertions(+), 7 deletions(-)

diff --git a/app/test/test_lpm_perf.c b/app/test/test_lpm_perf.c
index 41da811..f72250f 100644
--- a/app/test/test_lpm_perf.c
+++ b/app/test/test_lpm_perf.c
@@ -34,16 +34,28 @@
 #include <stdio.h>
 #include <stdint.h>
 #include <stdlib.h>
+#include <string.h>
 
 #include <rte_cycles.h>
 #include <rte_random.h>
 #include <rte_branch_prediction.h>
 #include <rte_lpm.h>
+#include <rte_ip.h>
+#include <rte_malloc.h>
 
 #include "test.h"
-#include "test_lpm_routes.h"
+#include "resource.h"
 #include "test_xmmt_ops.h"
 
+REGISTER_LINKED_RESOURCE(test_lpm_data)
+
+struct route_rule {
+	uint32_t ip;
+	uint8_t depth;
+};
+static struct route_rule *large_route_table;
+static unsigned int num_route_entries;
+
 #define TEST_LPM_ASSERT(cond) do {                                            \
 	if (!(cond)) {                                                        \
 		printf("Error at line %d: \n", __LINE__);                     \
@@ -80,6 +92,62 @@ print_route_distribution(const struct route_rule *table, uint32_t n)
 }
 
 static int
+load_large_route_table(void)
+{
+	const struct resource *r;
+	const char *lpm_data;
+
+	r = resource_find("test_lpm_data");
+	TEST_ASSERT_NOT_NULL(r, "No large lpm table data found");
+
+	/* the routing table size is going to be less than the size of the
+	 * resource, since text extries are more verbose. Allocate this as
+	 * the max size, and shrink the allocation later
+	 */
+	large_route_table = rte_malloc(NULL, resource_size(r), 0);
+	if (large_route_table == NULL)
+		return -1;
+
+	/* parse the lpm table. All entries are of format:
+	 *  {IP-as-decimal-unsigned, depth}
+	 * For example:
+	 *  {1234567U, 24},
+	 * We use the "U" and "}" characters as format check characters,
+	 * after parsing each number.
+	 */
+	for (lpm_data = r->begin; lpm_data < r->end; lpm_data++) {
+		if (*lpm_data == '{') {
+			char *endptr;
+
+			lpm_data++;
+			large_route_table[num_route_entries].ip = \
+				strtoul(lpm_data, &endptr, 0);
+			if (*endptr != 'U') {
+				if (num_route_entries > 0)
+					printf("Failed parse of %.*s\n", 12, lpm_data);
+				continue;
+			}
+
+			lpm_data = endptr + 2; /* skip U and , */
+			large_route_table[num_route_entries].depth = \
+				strtoul(lpm_data, &endptr, 0);
+			if (*endptr != '}') {
+				if (num_route_entries > 0)
+					printf("Failed parse of %.*s\n",5, lpm_data);
+				continue;
+			}
+
+			num_route_entries++;
+		}
+	}
+
+	large_route_table = rte_realloc(large_route_table,
+		sizeof(large_route_table[0]) * num_route_entries, 0);
+	printf("Read %u route entries\n", num_route_entries);
+	return 0;
+}
+
+static int
 test_lpm_perf(void)
 {
 	struct rte_lpm *lpm = NULL;
@@ -97,9 +165,10 @@ test_lpm_perf(void)
 
 	rte_srand(rte_rdtsc());
 
-	printf("No. routes = %u\n", (unsigned) NUM_ROUTE_ENTRIES);
+	TEST_ASSERT_SUCCESS(load_large_route_table(), "Error loading lpm table");
+	printf("No. routes = %u\n", (unsigned) num_route_entries);
 
-	print_route_distribution(large_route_table, (uint32_t) NUM_ROUTE_ENTRIES);
+	print_route_distribution(large_route_table, (uint32_t) num_route_entries);
 
 	lpm = rte_lpm_create(__func__, SOCKET_ID_ANY, &config);
 	TEST_LPM_ASSERT(lpm != NULL);
@@ -107,7 +176,7 @@ test_lpm_perf(void)
 	/* Measue add. */
 	begin = rte_rdtsc();
 
-	for (i = 0; i < NUM_ROUTE_ENTRIES; i++) {
+	for (i = 0; i < num_route_entries; i++) {
 		if (rte_lpm_add(lpm, large_route_table[i].ip,
 				large_route_table[i].depth, next_hop_add) == 0)
 			status++;
@@ -136,7 +205,7 @@ test_lpm_perf(void)
 			(unsigned) cache_line_counter, (unsigned) cache_line_counter * 64);
 
 	printf("Average LPM Add: %g cycles\n",
-			(double)total_time / NUM_ROUTE_ENTRIES);
+			(double)total_time / num_route_entries);
 
 	/* Measure single Lookup */
 	total_time = 0;
@@ -225,7 +294,7 @@ test_lpm_perf(void)
 	status = 0;
 	begin = rte_rdtsc();
 
-	for (i = 0; i < NUM_ROUTE_ENTRIES; i++) {
+	for (i = 0; i < num_route_entries; i++) {
 		/* rte_lpm_delete(lpm, ip, depth) */
 		status += rte_lpm_delete(lpm, large_route_table[i].ip,
 				large_route_table[i].depth);
@@ -234,7 +303,7 @@ test_lpm_perf(void)
 	total_time += rte_rdtsc() - begin;
 
 	printf("Average LPM Delete: %g cycles\n",
-			(double)total_time / NUM_ROUTE_ENTRIES);
+			(double)total_time / num_route_entries);
 
 	rte_lpm_delete_all(lpm);
 	rte_lpm_free(lpm);
-- 
2.8.1

Re: [PATCH 0/4] Convert lpm data from header to resource

[Thomas Monjalon]

2016-06-14 15:03, Nikita Kozlov:
> Re-submission of Bruce Richardson patch set:
> http://dpdk.org/dev/patchwork/patch/12484/
> 
> This patchset is a rebase to the current head and use of 
> strndupa (not posix) was replaced.
[...]
> Nikita Kozlov (4):
>   test: make all lpm routes be of unsigned type
>   test: change lpm routes file from header to data file
>   test: change lpm test to use routes as resource
>   test: change lpm perf test to use routes as resource

Please fix the authorship and add your Signed-off at the end of the
commits you have touched.

There are few checkpatch warnings. It complains about unnecessary
line continuations and some spacing.

Thanks

[PATCH v2 0/5] change lpm route table from code to data

[Bruce Richardson]

This patchset takes the existing lpm large routing table information and
converts it from a header file included at compile time to a resource linked
in. This improves things in two ways:
1. Improves DPDK build time
2. Removes approx 1 million lines of code from our LOC counts, as the header
   file no longer counts as code, but more correctly as data.

Before and after approx code stats (with some additional patchsets applied)
are given below.  Notice how the app folder has dropped from being the biggest
component with 6x the drivers code to second place with less code than the
drivers.

Future work is to repeat the same process for lpm6. Additional work for both
lpm and lpm6 is to remove the routing tables entirely and replace them with
auto-generated data that gives the roughly the same results. That would
shrink the overall size of DPDK releases, and is the desired end state.

---

This patchset is based directly on the original RFC submitted:
http://www.dpdk.org/ml/archives/dev/2016-May/038478.html.

Updates in this version compared to RFC:
* rebased on dpdk.org HEAD. The changes are now mostly to test_lpm_perf.c
  rather than test_lpm.c
* removed use of strndupa() as suggest by Nikita Koslov

---

## Before:

SLOC    Directory       SLOC-by-Language (Sorted)
1237464 app             ansic=1236324,python=1140
192107  drivers         ansic=192107
108109  lib             ansic=107968,python=141
65447   examples        ansic=65312,sh=135
849     tools           python=444,sh=405
799     scripts         sh=799
77      doc             python=75,sh=2
0       config          (none)
0       mk              (none)
0       pkg             (none)
0       top_dir         (none)


Totals grouped by language (dominant language first):
ansic:      1601711 (99.80%)
python:        1800 (0.11%)
sh:            1341 (0.08%)

-----------

## After

SLOC    Directory       SLOC-by-Language (Sorted)
192107  drivers         ansic=192107
160646  app             ansic=159506,python=1140
108109  lib             ansic=107968,python=141
65447   examples        ansic=65312,sh=135
849     tools           python=444,sh=405
799     scripts         sh=799
77      doc             python=75,sh=2
0       config          (none)
0       mk              (none)
0       pkg             (none)
0       top_dir         (none)


Totals grouped by language (dominant language first):
ansic:       524893 (99.41%)
python:        1800 (0.34%)
sh:            1341 (0.25%)

generated using David A. Wheeler's 'SLOCCount'

Bruce Richardson (5):
  test: fix unneeded routes table include from lpm test
  test: make all lpm routes be of unsigned type
  test: add lpm routes as a linked resource
  test: change lpm test to use routes as resource
  test: change lpm routes file from header to data file

 app/test/Makefile                                  |  1 +
 app/test/test_lpm.c                                |  2 +-
 app/test/test_lpm_perf.c                           | 78 +++++++++++++++++++++-
 .../{test_lpm_routes.h => test_lpm_routes.dat}     |  2 +-
 4 files changed, 80 insertions(+), 3 deletions(-)
 rename app/test/{test_lpm_routes.h => test_lpm_routes.dat} (99%)

-- 
2.5.5

[PATCH v2 1/5] test: fix unneeded routes table include from lpm test

[Bruce Richardson]

The large routing table header file in only needed by the performance
tests and not by the functional tests to remove the include of
test_lpm_routes.h from test_lpm.c.

Fixes: 62dac65a3265 ("app/test: isolate lpm performance cases")

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/test_lpm.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/app/test/test_lpm.c b/app/test/test_lpm.c
index f6930fb..8d1bae9 100644
--- a/app/test/test_lpm.c
+++ b/app/test/test_lpm.c
@@ -36,9 +36,9 @@
 #include <stdlib.h>
 
 #include <rte_lpm.h>
+#include <rte_ip.h>
 
 #include "test.h"
-#include "test_lpm_routes.h"
 #include "test_xmmt_ops.h"
 
 #define TEST_LPM_ASSERT(cond) do {                                            \
-- 
2.5.5

[PATCH v2 2/5] test: make all lpm routes be of unsigned type

[Bruce Richardson]

The one route that was different to the others in the test_lpm_routes.h
file was the entry "{0, 8}" which was the only route without a "U" after
the IP part. Add in the extra "U" to that entry so that it can be used
as a check character when parsing routes manually.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/test_lpm_routes.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/app/test/test_lpm_routes.h b/app/test/test_lpm_routes.h
index 023b0f9..1decfee 100644
--- a/app/test/test_lpm_routes.h
+++ b/app/test/test_lpm_routes.h
@@ -281825,7 +281825,7 @@ static const struct route_rule large_route_table[] =
 	{3536977920U, 17},
 	{3392992768U, 23},
 	{3341675008U, 23},
-	{0, 8},
+	{0U, 8},
 	{3326316168U, 30},
 	{3326316108U, 30},
 	{3326316060U, 30},
-- 
2.5.5

[PATCH v2 3/5] test: add lpm routes as a linked resource

[Bruce Richardson]

Since we now have the ability to store resource data directly inside
the test binary, take the test_lpm_routes.h header file and make it a
test resource. Later commits will then switch the test C file over to use
this rather than including it directly.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/Makefile        | 1 +
 app/test/test_lpm_perf.c | 3 +++
 2 files changed, 4 insertions(+)

diff --git a/app/test/Makefile b/app/test/Makefile
index 2de8c7a..a147db4 100644
--- a/app/test/Makefile
+++ b/app/test/Makefile
@@ -127,6 +127,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_HASH) += test_hash_functions.c
 SRCS-$(CONFIG_RTE_LIBRTE_HASH) += test_hash_scaling.c
 SRCS-$(CONFIG_RTE_LIBRTE_HASH) += test_hash_multiwriter.c
 
+$(eval $(call linked_resource,test_lpm_data,test_lpm_routes.h))
 SRCS-$(CONFIG_RTE_LIBRTE_LPM) += test_lpm.c
 SRCS-$(CONFIG_RTE_LIBRTE_LPM) += test_lpm_perf.c
 SRCS-$(CONFIG_RTE_LIBRTE_LPM) += test_lpm6.c
diff --git a/app/test/test_lpm_perf.c b/app/test/test_lpm_perf.c
index 41da811..0e64919 100644
--- a/app/test/test_lpm_perf.c
+++ b/app/test/test_lpm_perf.c
@@ -41,9 +41,12 @@
 #include <rte_lpm.h>
 
 #include "test.h"
+#include "resource.h"
 #include "test_lpm_routes.h"
 #include "test_xmmt_ops.h"
 
+REGISTER_LINKED_RESOURCE(test_lpm_data)
+
 #define TEST_LPM_ASSERT(cond) do {                                            \
 	if (!(cond)) {                                                        \
 		printf("Error at line %d: \n", __LINE__);                     \
-- 
2.5.5

[PATCH v2 4/5] test: change lpm test to use routes as resource

[Bruce Richardson]

Change the lpm autotest to use the routes data from the resource data
stored in the binary rather than including it directly into the C file
as a C header. This speeds up compile and link time, without changing
the test results.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/test_lpm_perf.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 74 insertions(+), 1 deletion(-)

diff --git a/app/test/test_lpm_perf.c b/app/test/test_lpm_perf.c
index 0e64919..ad70b02 100644
--- a/app/test/test_lpm_perf.c
+++ b/app/test/test_lpm_perf.c
@@ -34,17 +34,25 @@
 #include <stdio.h>
 #include <stdint.h>
 #include <stdlib.h>
+#include <string.h>
 
 #include <rte_cycles.h>
 #include <rte_random.h>
+#include <rte_malloc.h>
 #include <rte_branch_prediction.h>
 #include <rte_lpm.h>
 
 #include "test.h"
 #include "resource.h"
-#include "test_lpm_routes.h"
 #include "test_xmmt_ops.h"
 
+struct route_rule {
+	uint32_t ip;
+	uint8_t depth;
+};
+static struct route_rule *large_route_table;
+static unsigned int NUM_ROUTE_ENTRIES;
+
 REGISTER_LINKED_RESOURCE(test_lpm_data)
 
 #define TEST_LPM_ASSERT(cond) do {                                            \
@@ -83,6 +91,68 @@ print_route_distribution(const struct route_rule *table, uint32_t n)
 }
 
 static int
+load_large_route_table(void)
+{
+	const struct resource *r;
+	const char *lpm_data;
+
+	r = resource_find("test_lpm_data");
+	TEST_ASSERT_NOT_NULL(r, "No large lpm table data found");
+
+	/* the routing table size is going to be less than the size of the
+	 * resource, since text extries are more verbose. Allocate this as
+	 * the max size, and shrink the allocation later
+	 */
+	large_route_table = rte_malloc(NULL, resource_size(r), 0);
+	if (large_route_table == NULL)
+		return -1;
+
+	/* parse the lpm table. All entries are of format:
+	 *  {IP-as-decimal-unsigned, depth}
+	 * For example:
+	 *  {1234567U, 24},
+	 * We use the "U" and "}" characters as format check characters,
+	 * after parsing each number.
+	 */
+	for (lpm_data = r->begin; lpm_data < r->end; lpm_data++) {
+		if (*lpm_data == '{') {
+			char *endptr;
+
+			lpm_data++;
+			large_route_table[NUM_ROUTE_ENTRIES].ip = \
+				strtoul(lpm_data, &endptr, 0);
+			if (*endptr != 'U') {
+				if (NUM_ROUTE_ENTRIES > 0) {
+					char *value = strndup(lpm_data, 12);
+					printf("Failed parse of %s\n", value);
+					free(value);
+				}
+				continue;
+			}
+
+			lpm_data = endptr + 2; /* skip U and , */
+			large_route_table[NUM_ROUTE_ENTRIES].depth = \
+				strtoul(lpm_data, &endptr, 0);
+			if (*endptr != '}') {
+				if (NUM_ROUTE_ENTRIES > 0) {
+					char *value = strndup(lpm_data, 5);
+					printf("Failed parse of %s\n", value);
+					free(value);
+				}
+				continue;
+			}
+
+			NUM_ROUTE_ENTRIES++;
+		}
+	}
+
+	large_route_table = rte_realloc(large_route_table,
+		sizeof(large_route_table[0]) * NUM_ROUTE_ENTRIES, 0);
+	printf("Read %u route entries\n", NUM_ROUTE_ENTRIES);
+	return 0;
+}
+
+static int
 test_lpm_perf(void)
 {
 	struct rte_lpm *lpm = NULL;
@@ -98,6 +168,9 @@ test_lpm_perf(void)
 	uint64_t cache_line_counter = 0;
 	int64_t count = 0;
 
+	TEST_ASSERT_SUCCESS(load_large_route_table(),
+		"Error loading lpm table");
+
 	rte_srand(rte_rdtsc());
 
 	printf("No. routes = %u\n", (unsigned) NUM_ROUTE_ENTRIES);
-- 
2.5.5

[PATCH v2 5/5] test: change lpm routes file from header to data file

[Bruce Richardson]

Change the file extension of the test_lpm_routes file from .h to .dat.
This makes the lines-of-code counts for DPDK more realistic as they are not
affected by the huge counts from the lpm data.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/Makefile                                   | 2 +-
 app/test/{test_lpm_routes.h => test_lpm_routes.dat} | 0
 2 files changed, 1 insertion(+), 1 deletion(-)
 rename app/test/{test_lpm_routes.h => test_lpm_routes.dat} (100%)

diff --git a/app/test/Makefile b/app/test/Makefile
index a147db4..cc8c205 100644
--- a/app/test/Makefile
+++ b/app/test/Makefile
@@ -127,7 +127,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_HASH) += test_hash_functions.c
 SRCS-$(CONFIG_RTE_LIBRTE_HASH) += test_hash_scaling.c
 SRCS-$(CONFIG_RTE_LIBRTE_HASH) += test_hash_multiwriter.c
 
-$(eval $(call linked_resource,test_lpm_data,test_lpm_routes.h))
+$(eval $(call linked_resource,test_lpm_data,test_lpm_routes.dat))
 SRCS-$(CONFIG_RTE_LIBRTE_LPM) += test_lpm.c
 SRCS-$(CONFIG_RTE_LIBRTE_LPM) += test_lpm_perf.c
 SRCS-$(CONFIG_RTE_LIBRTE_LPM) += test_lpm6.c
diff --git a/app/test/test_lpm_routes.h b/app/test/test_lpm_routes.dat
similarity index 100%
rename from app/test/test_lpm_routes.h
rename to app/test/test_lpm_routes.dat
-- 
2.5.5

Re: [PATCH v2 5/5] test: change lpm routes file from header to data file

[Thomas Monjalon]

2016-07-14 17:06, Bruce Richardson:
> Change the file extension of the test_lpm_routes file from .h to .dat.
> This makes the lines-of-code counts for DPDK more realistic as they are not
> affected by the huge counts from the lpm data.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---
>  app/test/Makefile                                   | 2 +-
>  app/test/{test_lpm_routes.h => test_lpm_routes.dat} | 0

The size of the git tree significantly increase after renaming this file:
	git -c gc.reflogExpire=0 -c gc.reflogExpireUnreachable=0
	-c gc.rerereresolved=0 -c gc.rerereunresolved=0 -c gc.pruneExpire=now gc
	--aggressive && du -sk .git
28272   .git
	git am
	git -c gc.reflogExpire=0 -c gc.reflogExpireUnreachable=0
	-c gc.rerereresolved=0 -c gc.rerereunresolved=0 -c gc.pruneExpire=now gc
	--aggressive && du -sk .git
33188   .git

I think it would be better to replace this huge file by a script generating
the test resource.

Re: [PATCH v2 5/5] test: change lpm routes file from header to data file

[Richardson, Bruce]

> On 16 Jul 2016, at 00:05, Thomas Monjalon <thomas.monjalon@6wind.com> wrote:
> 
> 2016-07-14 17:06, Bruce Richardson:
>> Change the file extension of the test_lpm_routes file from .h to .dat.
>> This makes the lines-of-code counts for DPDK more realistic as they are not
>> affected by the huge counts from the lpm data.
>> 
>> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
>> ---
>> app/test/Makefile                                   | 2 +-
>> app/test/{test_lpm_routes.h => test_lpm_routes.dat} | 0
> 
> The size of the git tree significantly increase after renaming this file:
>    git -c gc.reflogExpire=0 -c gc.reflogExpireUnreachable=0
>    -c gc.rerereresolved=0 -c gc.rerereunresolved=0 -c gc.pruneExpire=now gc
>    --aggressive && du -sk .git
> 28272   .git
>    git am
>    git -c gc.reflogExpire=0 -c gc.reflogExpireUnreachable=0
>    -c gc.rerereresolved=0 -c gc.rerereunresolved=0 -c gc.pruneExpire=now gc
>    --aggressive && du -sk .git
> 33188   .git
> 
> I think it would be better to replace this huge file by a script generating
> the test resource.
> 
Completely agree. This was a temporary fix anyway so if it causes git bloat lets just wait for the permanent fix.

Bruce