[PATCH 0/7] examples/ipsec-secgw: make app to use ipsec library

[PATCH 0/7] examples/ipsec-secgw: make app to use ipsec library

[Konstantin Ananyev]

This patch series targets 19.02 release.

This patch series depends on the patch series:
http://patches.dpdk.org/patch/48044/
http://patches.dpdk.org/patch/48045/

http://patches.dpdk.org/patch/48143/
http://patches.dpdk.org/patch/48144/
http://patches.dpdk.org/patch/48145/
http://patches.dpdk.org/patch/48146/
http://patches.dpdk.org/patch/48147/
http://patches.dpdk.org/patch/48148/
http://patches.dpdk.org/patch/48149/
http://patches.dpdk.org/patch/48150/
http://patches.dpdk.org/patch/48151/

to be applied first.

That series contians few bug-fixes and changes to make ipsec-secgw
to utilize librte_ipsec library:
     - changes in the related data structures.
     - changes in the initialization code.
     - changes in the data-path code.
     - new command-line parameters to enable librte_ipsec codepath
       and related features.
     - test scripts to help automate ipsec-secgw functional testing.

Note that right now by default current (non-librte_ipsec) code-path
will be used. User has to run application with new command-line option ('-l')
to enable new codepath.
The main reason for that:
  - current librte_ipsec doesn't support all ipsec algorithms
    and features that the app does.
  - allow users to run both versions in parallel for some time
    to figure out any functional or performance degradation with the
    new code.

Konstantin Ananyev (7):
  examples/ipsec-secgw: avoid to request unused TX offloads
  examples/ipsec-secgw: allow to specify neighbor mac address
  examples/ipsec-secgw: fix crypto-op might never get dequeued
  examples/ipsec-secgw: fix outbound codepath for single SA
  examples/ipsec-secgw: make app to use ipsec library
  examples/ipsec-secgw: make data-path to use ipsec library
  examples/ipsec-secgw: add scripts for functional test

 examples/ipsec-secgw/Makefile                 |   1 +
 examples/ipsec-secgw/ipsec-secgw.c            | 387 +++++++++++++-----
 examples/ipsec-secgw/ipsec.c                  |  62 +--
 examples/ipsec-secgw/ipsec.h                  |  57 +++
 examples/ipsec-secgw/ipsec_process.c          | 266 ++++++++++++
 examples/ipsec-secgw/meson.build              |   6 +-
 examples/ipsec-secgw/parser.c                 |  75 ++++
 examples/ipsec-secgw/parser.h                 |   8 +-
 examples/ipsec-secgw/sa.c                     | 241 ++++++++++-
 examples/ipsec-secgw/test/common_defs.sh      | 113 +++++
 examples/ipsec-secgw/test/data_rxtx.sh        |  70 ++++
 examples/ipsec-secgw/test/linux_test1.sh      |  62 +++
 examples/ipsec-secgw/test/run_test.sh         |  48 +++
 .../test/trs_aescbc_sha1_common_defs.sh       |  42 ++
 .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  38 ++
 .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/trs_aescbc_sha1_esn_defs.sh          |  36 ++
 .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
 .../test/trs_aesgcm_common_defs.sh            |  38 ++
 examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  37 ++
 .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  36 ++
 .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
 .../test/tun_aescbc_sha1_common_defs.sh       |  42 ++
 .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  38 ++
 .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/tun_aescbc_sha1_esn_defs.sh          |  38 ++
 .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
 .../test/tun_aesgcm_common_defs.sh            |  38 ++
 examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  38 ++
 .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  38 ++
 .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
 33 files changed, 1770 insertions(+), 125 deletions(-)
 create mode 100644 examples/ipsec-secgw/ipsec_process.c
 create mode 100644 examples/ipsec-secgw/test/common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test1.sh
 create mode 100644 examples/ipsec-secgw/test/run_test.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh

-- 
2.17.1

[PATCH 1/7] examples/ipsec-secgw: avoid to request unused TX offloads

[Konstantin Ananyev]

ipsec-secgw always enables TX offloads
(DEV_TX_OFFLOAD_MULTI_SEGS, DEV_TX_OFFLOAD_SECURITY),
even when they are not requested by the config.
That causes many PMD to choose full-featured TX function,
which in many cases is much slower then one without offloads.
That patch adds checks to enabled extra HW offloads, only when
they were requested.
Plus it enables DEV_TX_OFFLOAD_IPV4_CKSUM,
only when other HW TX ofloads are going to be enabled.
Otherwise SW version of ip cksum calculation is used.
That allows to use vector TX function, when inline-ipsec is not
requested.

Signed-off-by: Remy Horton <remy.horton@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 44 +++++++++++++++--------
 examples/ipsec-secgw/ipsec.h       |  6 ++++
 examples/ipsec-secgw/sa.c          | 56 ++++++++++++++++++++++++++++++
 3 files changed, 91 insertions(+), 15 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 1bc0b5b50..cfc2b05e5 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -208,8 +208,6 @@ static struct rte_eth_conf port_conf = {
 	},
 	.txmode = {
 		.mq_mode = ETH_MQ_TX_NONE,
-		.offloads = (DEV_TX_OFFLOAD_IPV4_CKSUM |
-			     DEV_TX_OFFLOAD_MULTI_SEGS),
 	},
 };
 
@@ -315,7 +313,8 @@ prepare_traffic(struct rte_mbuf **pkts, struct ipsec_traffic *t,
 }
 
 static inline void
-prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
+prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port,
+		const struct lcore_conf *qconf)
 {
 	struct ip *ip;
 	struct ether_hdr *ethhdr;
@@ -325,14 +324,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
 	ethhdr = (struct ether_hdr *)rte_pktmbuf_prepend(pkt, ETHER_HDR_LEN);
 
 	if (ip->ip_v == IPVERSION) {
-		pkt->ol_flags |= PKT_TX_IP_CKSUM | PKT_TX_IPV4;
+		pkt->ol_flags |= qconf->outbound.ipv4_offloads;
 		pkt->l3_len = sizeof(struct ip);
 		pkt->l2_len = ETHER_HDR_LEN;
 
 		ip->ip_sum = 0;
+
+		/* calculate IPv4 cksum in SW */
+		if ((pkt->ol_flags & PKT_TX_IP_CKSUM) == 0)
+			ip->ip_sum = rte_ipv4_cksum((struct ipv4_hdr *)ip);
+
 		ethhdr->ether_type = rte_cpu_to_be_16(ETHER_TYPE_IPv4);
 	} else {
-		pkt->ol_flags |= PKT_TX_IPV6;
+		pkt->ol_flags |= qconf->outbound.ipv6_offloads;
 		pkt->l3_len = sizeof(struct ip6_hdr);
 		pkt->l2_len = ETHER_HDR_LEN;
 
@@ -346,18 +350,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
 }
 
 static inline void
-prepare_tx_burst(struct rte_mbuf *pkts[], uint16_t nb_pkts, uint16_t port)
+prepare_tx_burst(struct rte_mbuf *pkts[], uint16_t nb_pkts, uint16_t port,
+		const struct lcore_conf *qconf)
 {
 	int32_t i;
 	const int32_t prefetch_offset = 2;
 
 	for (i = 0; i < (nb_pkts - prefetch_offset); i++) {
 		rte_mbuf_prefetch_part2(pkts[i + prefetch_offset]);
-		prepare_tx_pkt(pkts[i], port);
+		prepare_tx_pkt(pkts[i], port, qconf);
 	}
 	/* Process left packets */
 	for (; i < nb_pkts; i++)
-		prepare_tx_pkt(pkts[i], port);
+		prepare_tx_pkt(pkts[i], port, qconf);
 }
 
 /* Send burst of packets on an output interface */
@@ -371,7 +376,7 @@ send_burst(struct lcore_conf *qconf, uint16_t n, uint16_t port)
 	queueid = qconf->tx_queue_id[port];
 	m_table = (struct rte_mbuf **)qconf->tx_mbufs[port].m_table;
 
-	prepare_tx_burst(m_table, n, port);
+	prepare_tx_burst(m_table, n, port, qconf);
 
 	ret = rte_eth_tx_burst(port, queueid, m_table, n);
 	if (unlikely(ret < n)) {
@@ -1543,7 +1548,7 @@ cryptodevs_init(void)
 }
 
 static void
-port_init(uint16_t portid)
+port_init(uint16_t portid, uint64_t req_rx_offloads, uint64_t req_tx_offloads)
 {
 	struct rte_eth_dev_info dev_info;
 	struct rte_eth_txconf *txconf;
@@ -1584,10 +1589,10 @@ port_init(uint16_t portid)
 		local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_JUMBO_FRAME;
 	}
 
-	if (dev_info.rx_offload_capa & DEV_RX_OFFLOAD_SECURITY)
-		local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_SECURITY;
-	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_SECURITY)
-		local_port_conf.txmode.offloads |= DEV_TX_OFFLOAD_SECURITY;
+	/* Capabilities will already have been checked.. */
+	local_port_conf.rxmode.offloads |= req_rx_offloads;
+	local_port_conf.txmode.offloads |= req_tx_offloads;
+
 	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_MBUF_FAST_FREE)
 		local_port_conf.txmode.offloads |=
 			DEV_TX_OFFLOAD_MBUF_FAST_FREE;
@@ -1639,6 +1644,13 @@ port_init(uint16_t portid)
 
 		qconf = &lcore_conf[lcore_id];
 		qconf->tx_queue_id[portid] = tx_queueid;
+
+		/* Pre-populate pkt offloads based on capabilities */
+		qconf->outbound.ipv4_offloads = PKT_TX_IPV4;
+		qconf->outbound.ipv6_offloads = PKT_TX_IPV6;
+		if (req_tx_offloads & DEV_TX_OFFLOAD_IPV4_CKSUM)
+			qconf->outbound.ipv4_offloads |= PKT_TX_IP_CKSUM;
+
 		tx_queueid++;
 
 		/* init RX queues */
@@ -1749,6 +1761,7 @@ main(int32_t argc, char **argv)
 	uint32_t lcore_id;
 	uint8_t socket_id;
 	uint16_t portid;
+	uint64_t req_rx_offloads, req_tx_offloads;
 
 	/* init EAL */
 	ret = rte_eal_init(argc, argv);
@@ -1804,7 +1817,8 @@ main(int32_t argc, char **argv)
 		if ((enabled_port_mask & (1 << portid)) == 0)
 			continue;
 
-		port_init(portid);
+		sa_check_offloads(portid, &req_rx_offloads, &req_tx_offloads);
+		port_init(portid, req_rx_offloads, req_tx_offloads);
 	}
 
 	cryptodevs_init();
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index c998c8076..9b1586f52 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -146,6 +146,8 @@ struct ipsec_ctx {
 	struct rte_mempool *session_pool;
 	struct rte_mbuf *ol_pkts[MAX_PKT_BURST] __rte_aligned(sizeof(void *));
 	uint16_t ol_pkts_cnt;
+	uint64_t ipv4_offloads;
+	uint64_t ipv6_offloads;
 };
 
 struct cdev_key {
@@ -239,4 +241,8 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id);
 void
 rt_init(struct socket_ctx *ctx, int32_t socket_id);
 
+int
+sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
+		uint64_t *tx_offloads);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index d2d3550a4..ff8c4b829 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -1017,3 +1017,59 @@ outbound_sa_lookup(struct sa_ctx *sa_ctx, uint32_t sa_idx[],
 	for (i = 0; i < nb_pkts; i++)
 		sa[i] = &sa_ctx->sa[sa_idx[i]];
 }
+
+/*
+ * Select HW offloads to be used.
+ */
+int
+sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
+		uint64_t *tx_offloads)
+{
+	struct ipsec_sa *rule;
+	uint32_t idx_sa;
+	struct rte_eth_dev_info dev_info;
+
+	rte_eth_dev_info_get(port_id, &dev_info);
+
+	*rx_offloads = 0;
+	*tx_offloads = 0;
+
+	/* Check for inbound rules that use offloads and use this port */
+	for (idx_sa = 0; idx_sa < nb_sa_in; idx_sa++) {
+		rule = &sa_in[idx_sa];
+		if ((rule->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				rule->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL)
+				&& rule->portid == port_id) {
+			if ((dev_info.rx_offload_capa & DEV_RX_OFFLOAD_SECURITY)
+					== 0) {
+				RTE_LOG(WARNING, PORT,
+					"HW RX IPSec is not supported\n");
+				return -EINVAL;
+			}
+			*rx_offloads |= DEV_RX_OFFLOAD_SECURITY;
+		}
+	}
+
+	/* Check for outbound rules that use offloads and use this port */
+	for (idx_sa = 0; idx_sa < nb_sa_out; idx_sa++) {
+		rule = &sa_out[idx_sa];
+		if ((rule->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				rule->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL)
+				&& rule->portid == port_id) {
+			if ((dev_info.tx_offload_capa & DEV_TX_OFFLOAD_SECURITY)
+					== 0) {
+				RTE_LOG(WARNING, PORT,
+					"HW TX IPSec is not supported\n");
+				return -EINVAL;
+			}
+			*tx_offloads |= DEV_TX_OFFLOAD_SECURITY;
+			/* Enable HW IPv4 cksum as well, if it is available */
+			if (dev_info.tx_offload_capa &
+					DEV_TX_OFFLOAD_IPV4_CKSUM)
+				*tx_offloads |= DEV_TX_OFFLOAD_IPV4_CKSUM;
+		}
+	}
+	return 0;
+}
-- 
2.17.1

[PATCH v2 0/7] examples/ipsec-secgw: make app to use ipsec library

[Konstantin Ananyev]

This patch series depends on the patch series:

Add 'try' semantics for RD and WR locking
http://patches.dpdk.org/patch/48044/
http://patches.dpdk.org/patch/48045/

ipsec: new library for IPsec data-path processing
http://patches.dpdk.org/patch/48436/
http://patches.dpdk.org/patch/48437/
http://patches.dpdk.org/patch/48438/
http://patches.dpdk.org/patch/48439/
http://patches.dpdk.org/patch/48440/
http://patches.dpdk.org/patch/48442/
http://patches.dpdk.org/patch/48441/
http://patches.dpdk.org/patch/48443/
http://patches.dpdk.org/patch/48444/

to be applied first.


v1 -> v2
 - Several bug fixes

That series contians few bug-fixes and changes to make ipsec-secgw
to utilize librte_ipsec library:
     - changes in the related data structures.
     - changes in the initialization code.
     - changes in the data-path code.
     - new command-line parameters to enable librte_ipsec codepath
       and related features.
     - test scripts to help automate ipsec-secgw functional testing.

Note that right now by default current (non-librte_ipsec) code-path
will be used. User has to run application with new command-line option
('-l')
to enable new codepath.
The main reason for that:
  - current librte_ipsec doesn't support all ipsec algorithms
    and features that the app does.
  - allow users to run both versions in parallel for some time
    to figure out any functional or performance degradation with the
    new code.

Test scripts were run with the following crypto devices:
 - aesni_mb
 - aesni_gcm
 - qat

Konstantin Ananyev (7):
  examples/ipsec-secgw: avoid to request unused TX offloads
  examples/ipsec-secgw: allow to specify neighbor mac address
  examples/ipsec-secgw: fix crypto-op might never get dequeued
  examples/ipsec-secgw: fix outbound codepath for single SA
  examples/ipsec-secgw: make app to use ipsec library
  examples/ipsec-secgw: make data-path to use ipsec library
  examples/ipsec-secgw: add scripts for functional test

 examples/ipsec-secgw/Makefile                 |   1 +
 examples/ipsec-secgw/ipsec-secgw.c            | 391 +++++++++++++-----
 examples/ipsec-secgw/ipsec.c                  |  62 +--
 examples/ipsec-secgw/ipsec.h                  |  57 +++
 examples/ipsec-secgw/ipsec_process.c          | 266 ++++++++++++
 examples/ipsec-secgw/meson.build              |   6 +-
 examples/ipsec-secgw/parser.c                 |  75 ++++
 examples/ipsec-secgw/parser.h                 |   8 +-
 examples/ipsec-secgw/sa.c                     | 241 ++++++++++-
 examples/ipsec-secgw/test/common_defs.sh      | 113 +++++
 examples/ipsec-secgw/test/data_rxtx.sh        |  70 ++++
 examples/ipsec-secgw/test/linux_test1.sh      |  62 +++
 examples/ipsec-secgw/test/run_test.sh         |  48 +++
 .../test/trs_aescbc_sha1_common_defs.sh       |  42 ++
 .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  38 ++
 .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/trs_aescbc_sha1_esn_defs.sh          |  36 ++
 .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
 .../test/trs_aesgcm_common_defs.sh            |  38 ++
 examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  37 ++
 .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  36 ++
 .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
 .../test/tun_aescbc_sha1_common_defs.sh       |  42 ++
 .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  38 ++
 .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/tun_aescbc_sha1_esn_defs.sh          |  38 ++
 .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
 .../test/tun_aesgcm_common_defs.sh            |  38 ++
 examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  38 ++
 .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  38 ++
 .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
 33 files changed, 1774 insertions(+), 125 deletions(-)
 create mode 100644 examples/ipsec-secgw/ipsec_process.c
 create mode 100644 examples/ipsec-secgw/test/common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test1.sh
 create mode 100644 examples/ipsec-secgw/test/run_test.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh

-- 
2.17.1

[PATCH v2 1/7] examples/ipsec-secgw: avoid to request unused TX offloads

[Konstantin Ananyev]

ipsec-secgw always enables TX offloads
(DEV_TX_OFFLOAD_MULTI_SEGS, DEV_TX_OFFLOAD_SECURITY),
even when they are not requested by the config.
That causes many PMD to choose full-featured TX function,
which in many cases is much slower then one without offloads.
That patch adds checks to enabled extra HW offloads, only when
they were requested.
Plus it enables DEV_TX_OFFLOAD_IPV4_CKSUM,
only when other HW TX ofloads are going to be enabled.
Otherwise SW version of ip cksum calculation is used.
That allows to use vector TX function, when inline-ipsec is not
requested.

Signed-off-by: Remy Horton <remy.horton@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 44 +++++++++++++++--------
 examples/ipsec-secgw/ipsec.h       |  6 ++++
 examples/ipsec-secgw/sa.c          | 56 ++++++++++++++++++++++++++++++
 3 files changed, 91 insertions(+), 15 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 1bc0b5b50..cfc2b05e5 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -208,8 +208,6 @@ static struct rte_eth_conf port_conf = {
 	},
 	.txmode = {
 		.mq_mode = ETH_MQ_TX_NONE,
-		.offloads = (DEV_TX_OFFLOAD_IPV4_CKSUM |
-			     DEV_TX_OFFLOAD_MULTI_SEGS),
 	},
 };
 
@@ -315,7 +313,8 @@ prepare_traffic(struct rte_mbuf **pkts, struct ipsec_traffic *t,
 }
 
 static inline void
-prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
+prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port,
+		const struct lcore_conf *qconf)
 {
 	struct ip *ip;
 	struct ether_hdr *ethhdr;
@@ -325,14 +324,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
 	ethhdr = (struct ether_hdr *)rte_pktmbuf_prepend(pkt, ETHER_HDR_LEN);
 
 	if (ip->ip_v == IPVERSION) {
-		pkt->ol_flags |= PKT_TX_IP_CKSUM | PKT_TX_IPV4;
+		pkt->ol_flags |= qconf->outbound.ipv4_offloads;
 		pkt->l3_len = sizeof(struct ip);
 		pkt->l2_len = ETHER_HDR_LEN;
 
 		ip->ip_sum = 0;
+
+		/* calculate IPv4 cksum in SW */
+		if ((pkt->ol_flags & PKT_TX_IP_CKSUM) == 0)
+			ip->ip_sum = rte_ipv4_cksum((struct ipv4_hdr *)ip);
+
 		ethhdr->ether_type = rte_cpu_to_be_16(ETHER_TYPE_IPv4);
 	} else {
-		pkt->ol_flags |= PKT_TX_IPV6;
+		pkt->ol_flags |= qconf->outbound.ipv6_offloads;
 		pkt->l3_len = sizeof(struct ip6_hdr);
 		pkt->l2_len = ETHER_HDR_LEN;
 
@@ -346,18 +350,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
 }
 
 static inline void
-prepare_tx_burst(struct rte_mbuf *pkts[], uint16_t nb_pkts, uint16_t port)
+prepare_tx_burst(struct rte_mbuf *pkts[], uint16_t nb_pkts, uint16_t port,
+		const struct lcore_conf *qconf)
 {
 	int32_t i;
 	const int32_t prefetch_offset = 2;
 
 	for (i = 0; i < (nb_pkts - prefetch_offset); i++) {
 		rte_mbuf_prefetch_part2(pkts[i + prefetch_offset]);
-		prepare_tx_pkt(pkts[i], port);
+		prepare_tx_pkt(pkts[i], port, qconf);
 	}
 	/* Process left packets */
 	for (; i < nb_pkts; i++)
-		prepare_tx_pkt(pkts[i], port);
+		prepare_tx_pkt(pkts[i], port, qconf);
 }
 
 /* Send burst of packets on an output interface */
@@ -371,7 +376,7 @@ send_burst(struct lcore_conf *qconf, uint16_t n, uint16_t port)
 	queueid = qconf->tx_queue_id[port];
 	m_table = (struct rte_mbuf **)qconf->tx_mbufs[port].m_table;
 
-	prepare_tx_burst(m_table, n, port);
+	prepare_tx_burst(m_table, n, port, qconf);
 
 	ret = rte_eth_tx_burst(port, queueid, m_table, n);
 	if (unlikely(ret < n)) {
@@ -1543,7 +1548,7 @@ cryptodevs_init(void)
 }
 
 static void
-port_init(uint16_t portid)
+port_init(uint16_t portid, uint64_t req_rx_offloads, uint64_t req_tx_offloads)
 {
 	struct rte_eth_dev_info dev_info;
 	struct rte_eth_txconf *txconf;
@@ -1584,10 +1589,10 @@ port_init(uint16_t portid)
 		local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_JUMBO_FRAME;
 	}
 
-	if (dev_info.rx_offload_capa & DEV_RX_OFFLOAD_SECURITY)
-		local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_SECURITY;
-	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_SECURITY)
-		local_port_conf.txmode.offloads |= DEV_TX_OFFLOAD_SECURITY;
+	/* Capabilities will already have been checked.. */
+	local_port_conf.rxmode.offloads |= req_rx_offloads;
+	local_port_conf.txmode.offloads |= req_tx_offloads;
+
 	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_MBUF_FAST_FREE)
 		local_port_conf.txmode.offloads |=
 			DEV_TX_OFFLOAD_MBUF_FAST_FREE;
@@ -1639,6 +1644,13 @@ port_init(uint16_t portid)
 
 		qconf = &lcore_conf[lcore_id];
 		qconf->tx_queue_id[portid] = tx_queueid;
+
+		/* Pre-populate pkt offloads based on capabilities */
+		qconf->outbound.ipv4_offloads = PKT_TX_IPV4;
+		qconf->outbound.ipv6_offloads = PKT_TX_IPV6;
+		if (req_tx_offloads & DEV_TX_OFFLOAD_IPV4_CKSUM)
+			qconf->outbound.ipv4_offloads |= PKT_TX_IP_CKSUM;
+
 		tx_queueid++;
 
 		/* init RX queues */
@@ -1749,6 +1761,7 @@ main(int32_t argc, char **argv)
 	uint32_t lcore_id;
 	uint8_t socket_id;
 	uint16_t portid;
+	uint64_t req_rx_offloads, req_tx_offloads;
 
 	/* init EAL */
 	ret = rte_eal_init(argc, argv);
@@ -1804,7 +1817,8 @@ main(int32_t argc, char **argv)
 		if ((enabled_port_mask & (1 << portid)) == 0)
 			continue;
 
-		port_init(portid);
+		sa_check_offloads(portid, &req_rx_offloads, &req_tx_offloads);
+		port_init(portid, req_rx_offloads, req_tx_offloads);
 	}
 
 	cryptodevs_init();
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index c998c8076..9b1586f52 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -146,6 +146,8 @@ struct ipsec_ctx {
 	struct rte_mempool *session_pool;
 	struct rte_mbuf *ol_pkts[MAX_PKT_BURST] __rte_aligned(sizeof(void *));
 	uint16_t ol_pkts_cnt;
+	uint64_t ipv4_offloads;
+	uint64_t ipv6_offloads;
 };
 
 struct cdev_key {
@@ -239,4 +241,8 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id);
 void
 rt_init(struct socket_ctx *ctx, int32_t socket_id);
 
+int
+sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
+		uint64_t *tx_offloads);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index d2d3550a4..ff8c4b829 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -1017,3 +1017,59 @@ outbound_sa_lookup(struct sa_ctx *sa_ctx, uint32_t sa_idx[],
 	for (i = 0; i < nb_pkts; i++)
 		sa[i] = &sa_ctx->sa[sa_idx[i]];
 }
+
+/*
+ * Select HW offloads to be used.
+ */
+int
+sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
+		uint64_t *tx_offloads)
+{
+	struct ipsec_sa *rule;
+	uint32_t idx_sa;
+	struct rte_eth_dev_info dev_info;
+
+	rte_eth_dev_info_get(port_id, &dev_info);
+
+	*rx_offloads = 0;
+	*tx_offloads = 0;
+
+	/* Check for inbound rules that use offloads and use this port */
+	for (idx_sa = 0; idx_sa < nb_sa_in; idx_sa++) {
+		rule = &sa_in[idx_sa];
+		if ((rule->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				rule->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL)
+				&& rule->portid == port_id) {
+			if ((dev_info.rx_offload_capa & DEV_RX_OFFLOAD_SECURITY)
+					== 0) {
+				RTE_LOG(WARNING, PORT,
+					"HW RX IPSec is not supported\n");
+				return -EINVAL;
+			}
+			*rx_offloads |= DEV_RX_OFFLOAD_SECURITY;
+		}
+	}
+
+	/* Check for outbound rules that use offloads and use this port */
+	for (idx_sa = 0; idx_sa < nb_sa_out; idx_sa++) {
+		rule = &sa_out[idx_sa];
+		if ((rule->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				rule->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL)
+				&& rule->portid == port_id) {
+			if ((dev_info.tx_offload_capa & DEV_TX_OFFLOAD_SECURITY)
+					== 0) {
+				RTE_LOG(WARNING, PORT,
+					"HW TX IPSec is not supported\n");
+				return -EINVAL;
+			}
+			*tx_offloads |= DEV_TX_OFFLOAD_SECURITY;
+			/* Enable HW IPv4 cksum as well, if it is available */
+			if (dev_info.tx_offload_capa &
+					DEV_TX_OFFLOAD_IPV4_CKSUM)
+				*tx_offloads |= DEV_TX_OFFLOAD_IPV4_CKSUM;
+		}
+	}
+	return 0;
+}
-- 
2.17.1

[PATCH v3 0/8] examples/ipsec-secgw: make app to use ipsec library

[Konstantin Ananyev]

This patch series depends on the patch series:

Add 'try' semantics for RD and WR locking
http://patches.dpdk.org/patch/48044/
http://patches.dpdk.org/patch/48045/

ipsec: new library for IPsec data-path processing
http://patches.dpdk.org/patch/48566/
http://patches.dpdk.org/patch/48567/
http://patches.dpdk.org/patch/48568/
http://patches.dpdk.org/patch/48569/
http://patches.dpdk.org/patch/48570/
http://patches.dpdk.org/patch/48571/
http://patches.dpdk.org/patch/48572/
http://patches.dpdk.org/patch/48573/
http://patches.dpdk.org/patch/48574/

to be applied first.

v2 -> v3
 - add IPv6 cases into test scripts
 - fixes for IPv6 support
 - fixes for inline-crypto support
 - some code restructuring

v1 -> v2
 - Several bug fixes

That series contians few bug-fixes and changes to make ipsec-secgw
to utilize librte_ipsec library:
     - changes in the related data structures.
     - changes in the initialization code.
     - changes in the data-path code.
     - new command-line parameters to enable librte_ipsec codepath
       and related features.
     - test scripts to help automate ipsec-secgw functional testing.

Note that right now by default current (non-librte_ipsec) code-path
will be used. User has to run application with new command-line option
('-l')
to enable new codepath.
The main reason for that:
  - current librte_ipsec doesn't support all ipsec algorithms
    and features that the app does.
  - allow users to run both versions in parallel for some time
    to figure out any functional or performance degradation with the
    new code.

Test scripts were run with the following crypto devices:
 - aesni_mb
 - aesni_gcm
 - qat

Konstantin Ananyev (8):
  examples/ipsec-secgw: avoid to request unused TX offloads
  examples/ipsec-secgw: allow to specify neighbor mac address
  examples/ipsec-secgw: fix crypto-op might never get dequeued
  examples/ipsec-secgw: fix outbound codepath for single SA
  examples/ipsec-secgw: make local variables static
  examples/ipsec-secgw: make app to use ipsec library
  examples/ipsec-secgw: make data-path to use ipsec library
  examples/ipsec-secgw: add scripts for functional test

 examples/ipsec-secgw/Makefile                 |   1 +
 examples/ipsec-secgw/ipsec-secgw.c            | 397 +++++++++++++-----
 examples/ipsec-secgw/ipsec.c                  |  62 ++-
 examples/ipsec-secgw/ipsec.h                  |  67 +++
 examples/ipsec-secgw/ipsec_process.c          | 341 +++++++++++++++
 examples/ipsec-secgw/meson.build              |   6 +-
 examples/ipsec-secgw/parser.c                 |  75 ++++
 examples/ipsec-secgw/parser.h                 |   8 +-
 examples/ipsec-secgw/sa.c                     | 284 ++++++++++++-
 examples/ipsec-secgw/sp4.c                    |  35 +-
 examples/ipsec-secgw/sp6.c                    |  35 +-
 examples/ipsec-secgw/test/common_defs.sh      | 149 +++++++
 examples/ipsec-secgw/test/data_rxtx.sh        | 118 ++++++
 examples/ipsec-secgw/test/linux_test4.sh      |  63 +++
 examples/ipsec-secgw/test/linux_test6.sh      |  63 +++
 examples/ipsec-secgw/test/run_test.sh         |  80 ++++
 .../test/trs_aescbc_sha1_common_defs.sh       |  69 +++
 .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  67 +++
 .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/trs_aescbc_sha1_esn_defs.sh          |  66 +++
 .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
 .../test/trs_aesgcm_common_defs.sh            |  60 +++
 examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  66 +++
 .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  66 +++
 .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
 .../test/tun_aescbc_sha1_common_defs.sh       |  68 +++
 .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  70 +++
 .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/tun_aescbc_sha1_esn_defs.sh          |  70 +++
 .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
 .../test/tun_aesgcm_common_defs.sh            |  60 +++
 examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  70 +++
 .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  70 +++
 .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
 36 files changed, 2488 insertions(+), 138 deletions(-)
 create mode 100644 examples/ipsec-secgw/ipsec_process.c
 create mode 100644 examples/ipsec-secgw/test/common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test4.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test6.sh
 create mode 100644 examples/ipsec-secgw/test/run_test.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh

-- 
2.17.1

Re: [PATCH v3 0/8] examples/ipsec-secgw: make app to use ipsec library

[Radu Nicolau]

On 12/6/2018 3:54 PM, Konstantin Ananyev wrote:
> This patch series depends on the patch series:
>
> Add 'try' semantics for RD and WR locking
> http://patches.dpdk.org/patch/48044/
> http://patches.dpdk.org/patch/48045/
>
> ipsec: new library for IPsec data-path processing
> http://patches.dpdk.org/patch/48566/
> http://patches.dpdk.org/patch/48567/
> http://patches.dpdk.org/patch/48568/
> http://patches.dpdk.org/patch/48569/
> http://patches.dpdk.org/patch/48570/
> http://patches.dpdk.org/patch/48571/
> http://patches.dpdk.org/patch/48572/
> http://patches.dpdk.org/patch/48573/
> http://patches.dpdk.org/patch/48574/
>
> to be applied first.
>
> v2 -> v3
>   - add IPv6 cases into test scripts
>   - fixes for IPv6 support
>   - fixes for inline-crypto support
>   - some code restructuring
>
> v1 -> v2
>   - Several bug fixes
>
> That series contians few bug-fixes and changes to make ipsec-secgw
> to utilize librte_ipsec library:
>       - changes in the related data structures.
>       - changes in the initialization code.
>       - changes in the data-path code.
>       - new command-line parameters to enable librte_ipsec codepath
>         and related features.
>       - test scripts to help automate ipsec-secgw functional testing.
>
> Note that right now by default current (non-librte_ipsec) code-path
> will be used. User has to run application with new command-line option
> ('-l')
> to enable new codepath.
> The main reason for that:
>    - current librte_ipsec doesn't support all ipsec algorithms
>      and features that the app does.
>    - allow users to run both versions in parallel for some time
>      to figure out any functional or performance degradation with the
>      new code.
>
> Test scripts were run with the following crypto devices:
>   - aesni_mb
>   - aesni_gcm
>   - qat
>
Series Acked-by: Radu Nicolau <radu.nicolau@intel.com 
<mailto:radu.nicolau@intel.com>>

[PATCH v4 0/9] examples/ipsec-secgw: make app to use ipsec library

[Konstantin Ananyev]

This patch series depends on the patch series:

Add 'try' semantics for RD and WR locking
http://patches.dpdk.org/patch/48044/
http://patches.dpdk.org/patch/48045/

ipsec: new library for IPsec data-path processing
http://patches.dpdk.org/patch/48900/
http://patches.dpdk.org/patch/48901/
http://patches.dpdk.org/patch/48902/
http://patches.dpdk.org/patch/48903/
http://patches.dpdk.org/patch/48904/
http://patches.dpdk.org/patch/48905/
http://patches.dpdk.org/patch/48906/
http://patches.dpdk.org/patch/48907/
http://patches.dpdk.org/patch/48908/
http://patches.dpdk.org/patch/48910/

to be applied first.

v3 ->v4
 - fix few issues with the test scripts
 - update docs

v2 -> v3
 - add IPv6 cases into test scripts
 - fixes for IPv6 support
 - fixes for inline-crypto support
 - some code restructuring

v1 -> v2
 - Several bug fixes

That series contians few bug-fixes and changes to make ipsec-secgw
to utilize librte_ipsec library:
     - changes in the related data structures.
     - changes in the initialization code.
     - changes in the data-path code.
     - new command-line parameters to enable librte_ipsec codepath
       and related features.
     - test scripts to help automate ipsec-secgw functional testing.

Note that right now by default current (non-librte_ipsec) code-path
will be used. User has to run application with new command-line option
('-l')
to enable new codepath.
The main reason for that:
  - current librte_ipsec doesn't support all ipsec algorithms
    and features that the app does.
  - allow users to run both versions in parallel for some time
    to figure out any functional or performance degradation with the
    new code.

Test scripts were run with the following crypto devices:
 - aesni_mb
 - aesni_gcm
 - qat

Konstantin Ananyev (9):
  examples/ipsec-secgw: avoid to request unused TX offloads
  examples/ipsec-secgw: allow to specify neighbor mac address
  examples/ipsec-secgw: fix crypto-op might never get dequeued
  examples/ipsec-secgw: fix outbound codepath for single SA
  examples/ipsec-secgw: make local variables static
  examples/ipsec-secgw: make app to use ipsec library
  examples/ipsec-secgw: make data-path to use ipsec library
  examples/ipsec-secgw: add scripts for functional test
  doc: update ipsec-secgw guide and relelase notes

 doc/guides/rel_notes/release_19_02.rst        |  14 +
 doc/guides/sample_app_ug/ipsec_secgw.rst      | 101 ++++-
 examples/ipsec-secgw/Makefile                 |   1 +
 examples/ipsec-secgw/ipsec-secgw.c            | 397 +++++++++++++-----
 examples/ipsec-secgw/ipsec.c                  |  62 ++-
 examples/ipsec-secgw/ipsec.h                  |  67 +++
 examples/ipsec-secgw/ipsec_process.c          | 341 +++++++++++++++
 examples/ipsec-secgw/meson.build              |   6 +-
 examples/ipsec-secgw/parser.c                 |  75 ++++
 examples/ipsec-secgw/parser.h                 |   8 +-
 examples/ipsec-secgw/sa.c                     | 284 ++++++++++++-
 examples/ipsec-secgw/sp4.c                    |  35 +-
 examples/ipsec-secgw/sp6.c                    |  35 +-
 examples/ipsec-secgw/test/common_defs.sh      | 153 +++++++
 examples/ipsec-secgw/test/data_rxtx.sh        |  62 +++
 examples/ipsec-secgw/test/linux_test4.sh      |  63 +++
 examples/ipsec-secgw/test/linux_test6.sh      |  64 +++
 examples/ipsec-secgw/test/run_test.sh         |  80 ++++
 .../test/trs_aescbc_sha1_common_defs.sh       |  69 +++
 .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  67 +++
 .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/trs_aescbc_sha1_esn_defs.sh          |  66 +++
 .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
 .../test/trs_aesgcm_common_defs.sh            |  60 +++
 examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  66 +++
 .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  66 +++
 .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
 .../test/tun_aescbc_sha1_common_defs.sh       |  68 +++
 .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  70 +++
 .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/tun_aescbc_sha1_esn_defs.sh          |  70 +++
 .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
 .../test/tun_aesgcm_common_defs.sh            |  60 +++
 examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  70 +++
 .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  70 +++
 .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
 38 files changed, 2550 insertions(+), 140 deletions(-)
 create mode 100644 examples/ipsec-secgw/ipsec_process.c
 create mode 100644 examples/ipsec-secgw/test/common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test4.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test6.sh
 create mode 100644 examples/ipsec-secgw/test/run_test.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh

-- 
2.17.1

[PATCH v4 1/9] examples/ipsec-secgw: avoid to request unused TX offloads

[Konstantin Ananyev]

ipsec-secgw always enables TX offloads
(DEV_TX_OFFLOAD_MULTI_SEGS, DEV_TX_OFFLOAD_SECURITY),
even when they are not requested by the config.
That causes many PMD to choose full-featured TX function,
which in many cases is much slower then one without offloads.
That patch adds checks to enabled extra HW offloads, only when
they were requested.
Plus it enables DEV_TX_OFFLOAD_IPV4_CKSUM,
only when other HW TX ofloads are going to be enabled.
Otherwise SW version of ip cksum calculation is used.
That allows to use vector TX function, when inline-ipsec is not
requested.

Signed-off-by: Remy Horton <remy.horton@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 44 +++++++++++++++--------
 examples/ipsec-secgw/ipsec.h       |  6 ++++
 examples/ipsec-secgw/sa.c          | 56 ++++++++++++++++++++++++++++++
 3 files changed, 91 insertions(+), 15 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 1bc0b5b50..cfc2b05e5 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -208,8 +208,6 @@ static struct rte_eth_conf port_conf = {
 	},
 	.txmode = {
 		.mq_mode = ETH_MQ_TX_NONE,
-		.offloads = (DEV_TX_OFFLOAD_IPV4_CKSUM |
-			     DEV_TX_OFFLOAD_MULTI_SEGS),
 	},
 };
 
@@ -315,7 +313,8 @@ prepare_traffic(struct rte_mbuf **pkts, struct ipsec_traffic *t,
 }
 
 static inline void
-prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
+prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port,
+		const struct lcore_conf *qconf)
 {
 	struct ip *ip;
 	struct ether_hdr *ethhdr;
@@ -325,14 +324,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
 	ethhdr = (struct ether_hdr *)rte_pktmbuf_prepend(pkt, ETHER_HDR_LEN);
 
 	if (ip->ip_v == IPVERSION) {
-		pkt->ol_flags |= PKT_TX_IP_CKSUM | PKT_TX_IPV4;
+		pkt->ol_flags |= qconf->outbound.ipv4_offloads;
 		pkt->l3_len = sizeof(struct ip);
 		pkt->l2_len = ETHER_HDR_LEN;
 
 		ip->ip_sum = 0;
+
+		/* calculate IPv4 cksum in SW */
+		if ((pkt->ol_flags & PKT_TX_IP_CKSUM) == 0)
+			ip->ip_sum = rte_ipv4_cksum((struct ipv4_hdr *)ip);
+
 		ethhdr->ether_type = rte_cpu_to_be_16(ETHER_TYPE_IPv4);
 	} else {
-		pkt->ol_flags |= PKT_TX_IPV6;
+		pkt->ol_flags |= qconf->outbound.ipv6_offloads;
 		pkt->l3_len = sizeof(struct ip6_hdr);
 		pkt->l2_len = ETHER_HDR_LEN;
 
@@ -346,18 +350,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
 }
 
 static inline void
-prepare_tx_burst(struct rte_mbuf *pkts[], uint16_t nb_pkts, uint16_t port)
+prepare_tx_burst(struct rte_mbuf *pkts[], uint16_t nb_pkts, uint16_t port,
+		const struct lcore_conf *qconf)
 {
 	int32_t i;
 	const int32_t prefetch_offset = 2;
 
 	for (i = 0; i < (nb_pkts - prefetch_offset); i++) {
 		rte_mbuf_prefetch_part2(pkts[i + prefetch_offset]);
-		prepare_tx_pkt(pkts[i], port);
+		prepare_tx_pkt(pkts[i], port, qconf);
 	}
 	/* Process left packets */
 	for (; i < nb_pkts; i++)
-		prepare_tx_pkt(pkts[i], port);
+		prepare_tx_pkt(pkts[i], port, qconf);
 }
 
 /* Send burst of packets on an output interface */
@@ -371,7 +376,7 @@ send_burst(struct lcore_conf *qconf, uint16_t n, uint16_t port)
 	queueid = qconf->tx_queue_id[port];
 	m_table = (struct rte_mbuf **)qconf->tx_mbufs[port].m_table;
 
-	prepare_tx_burst(m_table, n, port);
+	prepare_tx_burst(m_table, n, port, qconf);
 
 	ret = rte_eth_tx_burst(port, queueid, m_table, n);
 	if (unlikely(ret < n)) {
@@ -1543,7 +1548,7 @@ cryptodevs_init(void)
 }
 
 static void
-port_init(uint16_t portid)
+port_init(uint16_t portid, uint64_t req_rx_offloads, uint64_t req_tx_offloads)
 {
 	struct rte_eth_dev_info dev_info;
 	struct rte_eth_txconf *txconf;
@@ -1584,10 +1589,10 @@ port_init(uint16_t portid)
 		local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_JUMBO_FRAME;
 	}
 
-	if (dev_info.rx_offload_capa & DEV_RX_OFFLOAD_SECURITY)
-		local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_SECURITY;
-	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_SECURITY)
-		local_port_conf.txmode.offloads |= DEV_TX_OFFLOAD_SECURITY;
+	/* Capabilities will already have been checked.. */
+	local_port_conf.rxmode.offloads |= req_rx_offloads;
+	local_port_conf.txmode.offloads |= req_tx_offloads;
+
 	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_MBUF_FAST_FREE)
 		local_port_conf.txmode.offloads |=
 			DEV_TX_OFFLOAD_MBUF_FAST_FREE;
@@ -1639,6 +1644,13 @@ port_init(uint16_t portid)
 
 		qconf = &lcore_conf[lcore_id];
 		qconf->tx_queue_id[portid] = tx_queueid;
+
+		/* Pre-populate pkt offloads based on capabilities */
+		qconf->outbound.ipv4_offloads = PKT_TX_IPV4;
+		qconf->outbound.ipv6_offloads = PKT_TX_IPV6;
+		if (req_tx_offloads & DEV_TX_OFFLOAD_IPV4_CKSUM)
+			qconf->outbound.ipv4_offloads |= PKT_TX_IP_CKSUM;
+
 		tx_queueid++;
 
 		/* init RX queues */
@@ -1749,6 +1761,7 @@ main(int32_t argc, char **argv)
 	uint32_t lcore_id;
 	uint8_t socket_id;
 	uint16_t portid;
+	uint64_t req_rx_offloads, req_tx_offloads;
 
 	/* init EAL */
 	ret = rte_eal_init(argc, argv);
@@ -1804,7 +1817,8 @@ main(int32_t argc, char **argv)
 		if ((enabled_port_mask & (1 << portid)) == 0)
 			continue;
 
-		port_init(portid);
+		sa_check_offloads(portid, &req_rx_offloads, &req_tx_offloads);
+		port_init(portid, req_rx_offloads, req_tx_offloads);
 	}
 
 	cryptodevs_init();
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index c998c8076..9b1586f52 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -146,6 +146,8 @@ struct ipsec_ctx {
 	struct rte_mempool *session_pool;
 	struct rte_mbuf *ol_pkts[MAX_PKT_BURST] __rte_aligned(sizeof(void *));
 	uint16_t ol_pkts_cnt;
+	uint64_t ipv4_offloads;
+	uint64_t ipv6_offloads;
 };
 
 struct cdev_key {
@@ -239,4 +241,8 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id);
 void
 rt_init(struct socket_ctx *ctx, int32_t socket_id);
 
+int
+sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
+		uint64_t *tx_offloads);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index d2d3550a4..ff8c4b829 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -1017,3 +1017,59 @@ outbound_sa_lookup(struct sa_ctx *sa_ctx, uint32_t sa_idx[],
 	for (i = 0; i < nb_pkts; i++)
 		sa[i] = &sa_ctx->sa[sa_idx[i]];
 }
+
+/*
+ * Select HW offloads to be used.
+ */
+int
+sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
+		uint64_t *tx_offloads)
+{
+	struct ipsec_sa *rule;
+	uint32_t idx_sa;
+	struct rte_eth_dev_info dev_info;
+
+	rte_eth_dev_info_get(port_id, &dev_info);
+
+	*rx_offloads = 0;
+	*tx_offloads = 0;
+
+	/* Check for inbound rules that use offloads and use this port */
+	for (idx_sa = 0; idx_sa < nb_sa_in; idx_sa++) {
+		rule = &sa_in[idx_sa];
+		if ((rule->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				rule->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL)
+				&& rule->portid == port_id) {
+			if ((dev_info.rx_offload_capa & DEV_RX_OFFLOAD_SECURITY)
+					== 0) {
+				RTE_LOG(WARNING, PORT,
+					"HW RX IPSec is not supported\n");
+				return -EINVAL;
+			}
+			*rx_offloads |= DEV_RX_OFFLOAD_SECURITY;
+		}
+	}
+
+	/* Check for outbound rules that use offloads and use this port */
+	for (idx_sa = 0; idx_sa < nb_sa_out; idx_sa++) {
+		rule = &sa_out[idx_sa];
+		if ((rule->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				rule->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL)
+				&& rule->portid == port_id) {
+			if ((dev_info.tx_offload_capa & DEV_TX_OFFLOAD_SECURITY)
+					== 0) {
+				RTE_LOG(WARNING, PORT,
+					"HW TX IPSec is not supported\n");
+				return -EINVAL;
+			}
+			*tx_offloads |= DEV_TX_OFFLOAD_SECURITY;
+			/* Enable HW IPv4 cksum as well, if it is available */
+			if (dev_info.tx_offload_capa &
+					DEV_TX_OFFLOAD_IPV4_CKSUM)
+				*tx_offloads |= DEV_TX_OFFLOAD_IPV4_CKSUM;
+		}
+	}
+	return 0;
+}
-- 
2.17.1

Re: [PATCH v4 1/9] examples/ipsec-secgw: avoid to request unused TX offloads

[Akhil Goyal]

Hi Konstantin,

On 12/14/2018 10:10 PM, Konstantin Ananyev wrote:
> ipsec-secgw always enables TX offloads
> (DEV_TX_OFFLOAD_MULTI_SEGS, DEV_TX_OFFLOAD_SECURITY),
> even when they are not requested by the config.
> That causes many PMD to choose full-featured TX function,
> which in many cases is much slower then one without offloads.
> That patch adds checks to enabled extra HW offloads, only when
> they were requested.
> Plus it enables DEV_TX_OFFLOAD_IPV4_CKSUM,
> only when other HW TX ofloads are going to be enabled.
> Otherwise SW version of ip cksum calculation is used.
> That allows to use vector TX function, when inline-ipsec is not
> requested.
>
> Signed-off-by: Remy Horton <remy.horton@intel.com>
> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> Acked-by: Radu Nicolau <radu.nicolau@intel.com>
> ---
>   examples/ipsec-secgw/ipsec-secgw.c | 44 +++++++++++++++--------
>   examples/ipsec-secgw/ipsec.h       |  6 ++++
>   examples/ipsec-secgw/sa.c          | 56 ++++++++++++++++++++++++++++++
>   3 files changed, 91 insertions(+), 15 deletions(-)
>
> diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
> index 1bc0b5b50..cfc2b05e5 100644
> --- a/examples/ipsec-secgw/ipsec-secgw.c
> +++ b/examples/ipsec-secgw/ipsec-secgw.c
> @@ -208,8 +208,6 @@ static struct rte_eth_conf port_conf = {
>   	},
>   	.txmode = {
>   		.mq_mode = ETH_MQ_TX_NONE,
> -		.offloads = (DEV_TX_OFFLOAD_IPV4_CKSUM |
> -			     DEV_TX_OFFLOAD_MULTI_SEGS),
I believe this is disabling checksum offload for all cases and then 
enabling only for inline crypto and inline proto.
This is breaking lookaside proto and lookaside none cases. Please 
correct me if I am wrong.
So a NACK for this if my understanding is correct.
>   	},
>   };
>   
> @@ -315,7 +313,8 @@ prepare_traffic(struct rte_mbuf **pkts, struct ipsec_traffic *t,
>   }
>   
>   static inline void
> -prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
> +prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port,
> +		const struct lcore_conf *qconf)
>   {
>   	struct ip *ip;
>   	struct ether_hdr *ethhdr;
> @@ -325,14 +324,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
>   	ethhdr = (struct ether_hdr *)rte_pktmbuf_prepend(pkt, ETHER_HDR_LEN);
>   
>   	if (ip->ip_v == IPVERSION) {
> -		pkt->ol_flags |= PKT_TX_IP_CKSUM | PKT_TX_IPV4;
> +		pkt->ol_flags |= qconf->outbound.ipv4_offloads;
>   		pkt->l3_len = sizeof(struct ip);
>   		pkt->l2_len = ETHER_HDR_LEN;
>   
>   		ip->ip_sum = 0;
> +
> +		/* calculate IPv4 cksum in SW */
> +		if ((pkt->ol_flags & PKT_TX_IP_CKSUM) == 0)
> +			ip->ip_sum = rte_ipv4_cksum((struct ipv4_hdr *)ip);
> +
>   		ethhdr->ether_type = rte_cpu_to_be_16(ETHER_TYPE_IPv4);
>   	} else {
> -		pkt->ol_flags |= PKT_TX_IPV6;
> +		pkt->ol_flags |= qconf->outbound.ipv6_offloads;
>   		pkt->l3_len = sizeof(struct ip6_hdr);
>   		pkt->l2_len = ETHER_HDR_LEN;
>   
> @@ -346,18 +350,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
>   }
>   
>   static inline void
> -prepare_tx_burst(struct rte_mbuf *pkts[], uint16_t nb_pkts, uint16_t port)
> +prepare_tx_burst(struct rte_mbuf *pkts[], uint16_t nb_pkts, uint16_t port,
> +		const struct lcore_conf *qconf)
>   {
>   	int32_t i;
>   	const int32_t prefetch_offset = 2;
>   
>   	for (i = 0; i < (nb_pkts - prefetch_offset); i++) {
>   		rte_mbuf_prefetch_part2(pkts[i + prefetch_offset]);
> -		prepare_tx_pkt(pkts[i], port);
> +		prepare_tx_pkt(pkts[i], port, qconf);
>   	}
>   	/* Process left packets */
>   	for (; i < nb_pkts; i++)
> -		prepare_tx_pkt(pkts[i], port);
> +		prepare_tx_pkt(pkts[i], port, qconf);
>   }
>   
>   /* Send burst of packets on an output interface */
> @@ -371,7 +376,7 @@ send_burst(struct lcore_conf *qconf, uint16_t n, uint16_t port)
>   	queueid = qconf->tx_queue_id[port];
>   	m_table = (struct rte_mbuf **)qconf->tx_mbufs[port].m_table;
>   
> -	prepare_tx_burst(m_table, n, port);
> +	prepare_tx_burst(m_table, n, port, qconf);
>   
>   	ret = rte_eth_tx_burst(port, queueid, m_table, n);
>   	if (unlikely(ret < n)) {
> @@ -1543,7 +1548,7 @@ cryptodevs_init(void)
>   }
>   
>   static void
> -port_init(uint16_t portid)
> +port_init(uint16_t portid, uint64_t req_rx_offloads, uint64_t req_tx_offloads)
>   {
>   	struct rte_eth_dev_info dev_info;
>   	struct rte_eth_txconf *txconf;
> @@ -1584,10 +1589,10 @@ port_init(uint16_t portid)
>   		local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_JUMBO_FRAME;
>   	}
>   
> -	if (dev_info.rx_offload_capa & DEV_RX_OFFLOAD_SECURITY)
> -		local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_SECURITY;
> -	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_SECURITY)
> -		local_port_conf.txmode.offloads |= DEV_TX_OFFLOAD_SECURITY;
> +	/* Capabilities will already have been checked.. */
> +	local_port_conf.rxmode.offloads |= req_rx_offloads;
> +	local_port_conf.txmode.offloads |= req_tx_offloads;
> +
>   	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_MBUF_FAST_FREE)
>   		local_port_conf.txmode.offloads |=
>   			DEV_TX_OFFLOAD_MBUF_FAST_FREE;
> @@ -1639,6 +1644,13 @@ port_init(uint16_t portid)
>   
>   		qconf = &lcore_conf[lcore_id];
>   		qconf->tx_queue_id[portid] = tx_queueid;
> +
> +		/* Pre-populate pkt offloads based on capabilities */
> +		qconf->outbound.ipv4_offloads = PKT_TX_IPV4;
> +		qconf->outbound.ipv6_offloads = PKT_TX_IPV6;
> +		if (req_tx_offloads & DEV_TX_OFFLOAD_IPV4_CKSUM)
> +			qconf->outbound.ipv4_offloads |= PKT_TX_IP_CKSUM;
> +
>   		tx_queueid++;
>   
>   		/* init RX queues */
> @@ -1749,6 +1761,7 @@ main(int32_t argc, char **argv)
>   	uint32_t lcore_id;
>   	uint8_t socket_id;
>   	uint16_t portid;
> +	uint64_t req_rx_offloads, req_tx_offloads;
>   
>   	/* init EAL */
>   	ret = rte_eal_init(argc, argv);
> @@ -1804,7 +1817,8 @@ main(int32_t argc, char **argv)
>   		if ((enabled_port_mask & (1 << portid)) == 0)
>   			continue;
>   
> -		port_init(portid);
> +		sa_check_offloads(portid, &req_rx_offloads, &req_tx_offloads);
> +		port_init(portid, req_rx_offloads, req_tx_offloads);
>   	}
>   
>   	cryptodevs_init();
> diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
> index c998c8076..9b1586f52 100644
> --- a/examples/ipsec-secgw/ipsec.h
> +++ b/examples/ipsec-secgw/ipsec.h
> @@ -146,6 +146,8 @@ struct ipsec_ctx {
>   	struct rte_mempool *session_pool;
>   	struct rte_mbuf *ol_pkts[MAX_PKT_BURST] __rte_aligned(sizeof(void *));
>   	uint16_t ol_pkts_cnt;
> +	uint64_t ipv4_offloads;
> +	uint64_t ipv6_offloads;
>   };
>   
>   struct cdev_key {
> @@ -239,4 +241,8 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id);
>   void
>   rt_init(struct socket_ctx *ctx, int32_t socket_id);
>   
> +int
> +sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
> +		uint64_t *tx_offloads);
> +
>   #endif /* __IPSEC_H__ */
> diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
> index d2d3550a4..ff8c4b829 100644
> --- a/examples/ipsec-secgw/sa.c
> +++ b/examples/ipsec-secgw/sa.c
> @@ -1017,3 +1017,59 @@ outbound_sa_lookup(struct sa_ctx *sa_ctx, uint32_t sa_idx[],
>   	for (i = 0; i < nb_pkts; i++)
>   		sa[i] = &sa_ctx->sa[sa_idx[i]];
>   }
> +
> +/*
> + * Select HW offloads to be used.
> + */
> +int
> +sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
> +		uint64_t *tx_offloads)
> +{
> +	struct ipsec_sa *rule;
> +	uint32_t idx_sa;
> +	struct rte_eth_dev_info dev_info;
> +
> +	rte_eth_dev_info_get(port_id, &dev_info);
> +
> +	*rx_offloads = 0;
> +	*tx_offloads = 0;
> +
> +	/* Check for inbound rules that use offloads and use this port */
> +	for (idx_sa = 0; idx_sa < nb_sa_in; idx_sa++) {
> +		rule = &sa_in[idx_sa];
> +		if ((rule->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
> +				rule->type ==
> +				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL)
> +				&& rule->portid == port_id) {
> +			if ((dev_info.rx_offload_capa & DEV_RX_OFFLOAD_SECURITY)
> +					== 0) {
> +				RTE_LOG(WARNING, PORT,
> +					"HW RX IPSec is not supported\n");
> +				return -EINVAL;
> +			}
> +			*rx_offloads |= DEV_RX_OFFLOAD_SECURITY;
> +		}
> +	}
> +
> +	/* Check for outbound rules that use offloads and use this port */
> +	for (idx_sa = 0; idx_sa < nb_sa_out; idx_sa++) {
> +		rule = &sa_out[idx_sa];
> +		if ((rule->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
> +				rule->type ==
> +				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL)
> +				&& rule->portid == port_id) {
> +			if ((dev_info.tx_offload_capa & DEV_TX_OFFLOAD_SECURITY)
> +					== 0) {
> +				RTE_LOG(WARNING, PORT,
> +					"HW TX IPSec is not supported\n");
> +				return -EINVAL;
> +			}
> +			*tx_offloads |= DEV_TX_OFFLOAD_SECURITY;
> +			/* Enable HW IPv4 cksum as well, if it is available */
> +			if (dev_info.tx_offload_capa &
> +					DEV_TX_OFFLOAD_IPV4_CKSUM)
> +				*tx_offloads |= DEV_TX_OFFLOAD_IPV4_CKSUM;
> +		}
> +	}
> +	return 0;
> +}

Re: [PATCH v4 1/9] examples/ipsec-secgw: avoid to request unused TX offloads

[Ananyev, Konstantin]

> -----Original Message-----
> From: Akhil Goyal [mailto:akhil.goyal@nxp.com]
> Sent: Friday, December 21, 2018 1:57 PM
> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
> Cc: Nicolau, Radu <radu.nicolau@intel.com>; Horton, Remy <remy.horton@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v4 1/9] examples/ipsec-secgw: avoid to request unused TX offloads
> 
> Hi Konstantin,
> 
> On 12/14/2018 10:10 PM, Konstantin Ananyev wrote:
> > ipsec-secgw always enables TX offloads
> > (DEV_TX_OFFLOAD_MULTI_SEGS, DEV_TX_OFFLOAD_SECURITY),
> > even when they are not requested by the config.
> > That causes many PMD to choose full-featured TX function,
> > which in many cases is much slower then one without offloads.
> > That patch adds checks to enabled extra HW offloads, only when
> > they were requested.
> > Plus it enables DEV_TX_OFFLOAD_IPV4_CKSUM,
> > only when other HW TX ofloads are going to be enabled.
> > Otherwise SW version of ip cksum calculation is used.
> > That allows to use vector TX function, when inline-ipsec is not
> > requested.
> >
> > Signed-off-by: Remy Horton <remy.horton@intel.com>
> > Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> > Acked-by: Radu Nicolau <radu.nicolau@intel.com>
> > ---
> >   examples/ipsec-secgw/ipsec-secgw.c | 44 +++++++++++++++--------
> >   examples/ipsec-secgw/ipsec.h       |  6 ++++
> >   examples/ipsec-secgw/sa.c          | 56 ++++++++++++++++++++++++++++++
> >   3 files changed, 91 insertions(+), 15 deletions(-)
> >
> > diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
> > index 1bc0b5b50..cfc2b05e5 100644
> > --- a/examples/ipsec-secgw/ipsec-secgw.c
> > +++ b/examples/ipsec-secgw/ipsec-secgw.c
> > @@ -208,8 +208,6 @@ static struct rte_eth_conf port_conf = {
> >   	},
> >   	.txmode = {
> >   		.mq_mode = ETH_MQ_TX_NONE,
> > -		.offloads = (DEV_TX_OFFLOAD_IPV4_CKSUM |
> > -			     DEV_TX_OFFLOAD_MULTI_SEGS),
> I believe this is disabling checksum offload for all cases and then
> enabling only for inline crypto and inline proto.

Yes.

> This is breaking lookaside proto and lookaside none cases. Please
> correct me if I am wrong.

Why breaking?
For cases when HW cksum offload is disabled, IPv4 cksum calculation
will be done in SW, see below:
prepare_tx_pkt(...)
{
   ...
    +
    +		/* calculate IPv4 cksum in SW */
    +		if ((pkt->ol_flags & PKT_TX_IP_CKSUM) == 0)
    +			ip->ip_sum = rte_ipv4_cksum((struct ipv4_hdr *)ip);


We tested lookaside-none case quite extensively - all works well,
in fact on Intel NICs it became even a bit faster because of that change
(though not much). 
Disabling HW offloads when they are not really required has 2 benefits:
 1) allows app to be run on NICs without HW offloads support.
 2) allows dev_configure() for TX path to select simple/vector TX functions
     which for many NICs are significantly faster. 

Konstantin

> So a NACK for this if my understanding is correct.
> >   	},
> >   };
> >
> > @@ -315,7 +313,8 @@ prepare_traffic(struct rte_mbuf **pkts, struct ipsec_traffic *t,
> >   }
> >
> >   static inline void
> > -prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
> > +prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port,
> > +		const struct lcore_conf *qconf)
> >   {
> >   	struct ip *ip;
> >   	struct ether_hdr *ethhdr;
> > @@ -325,14 +324,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
> >   	ethhdr = (struct ether_hdr *)rte_pktmbuf_prepend(pkt, ETHER_HDR_LEN);
> >
> >   	if (ip->ip_v == IPVERSION) {
> > -		pkt->ol_flags |= PKT_TX_IP_CKSUM | PKT_TX_IPV4;
> > +		pkt->ol_flags |= qconf->outbound.ipv4_offloads;
> >   		pkt->l3_len = sizeof(struct ip);
> >   		pkt->l2_len = ETHER_HDR_LEN;
> >
> >   		ip->ip_sum = 0;
> > +
> > +		/* calculate IPv4 cksum in SW */
> > +		if ((pkt->ol_flags & PKT_TX_IP_CKSUM) == 0)
> > +			ip->ip_sum = rte_ipv4_cksum((struct ipv4_hdr *)ip);
> > +
> >   		ethhdr->ether_type = rte_cpu_to_be_16(ETHER_TYPE_IPv4);
> >   	} else {
> > -		pkt->ol_flags |= PKT_TX_IPV6;
> > +		pkt->ol_flags |= qconf->outbound.ipv6_offloads;
> >   		pkt->l3_len = sizeof(struct ip6_hdr);
> >   		pkt->l2_len = ETHER_HDR_LEN;
> >
> > @@ -346,18 +350,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
> >   }
> >
> >   static inline void
> > -prepare_tx_burst(struct rte_mbuf *pkts[], uint16_t nb_pkts, uint16_t port)
> > +prepare_tx_burst(struct rte_mbuf *pkts[], uint16_t nb_pkts, uint16_t port,
> > +		const struct lcore_conf *qconf)
> >   {
> >   	int32_t i;
> >   	const int32_t prefetch_offset = 2;
> >
> >   	for (i = 0; i < (nb_pkts - prefetch_offset); i++) {
> >   		rte_mbuf_prefetch_part2(pkts[i + prefetch_offset]);
> > -		prepare_tx_pkt(pkts[i], port);
> > +		prepare_tx_pkt(pkts[i], port, qconf);
> >   	}
> >   	/* Process left packets */
> >   	for (; i < nb_pkts; i++)
> > -		prepare_tx_pkt(pkts[i], port);
> > +		prepare_tx_pkt(pkts[i], port, qconf);
> >   }
> >
> >   /* Send burst of packets on an output interface */
> > @@ -371,7 +376,7 @@ send_burst(struct lcore_conf *qconf, uint16_t n, uint16_t port)
> >   	queueid = qconf->tx_queue_id[port];
> >   	m_table = (struct rte_mbuf **)qconf->tx_mbufs[port].m_table;
> >
> > -	prepare_tx_burst(m_table, n, port);
> > +	prepare_tx_burst(m_table, n, port, qconf);
> >
> >   	ret = rte_eth_tx_burst(port, queueid, m_table, n);
> >   	if (unlikely(ret < n)) {
> > @@ -1543,7 +1548,7 @@ cryptodevs_init(void)
> >   }
> >
> >   static void
> > -port_init(uint16_t portid)
> > +port_init(uint16_t portid, uint64_t req_rx_offloads, uint64_t req_tx_offloads)
> >   {
> >   	struct rte_eth_dev_info dev_info;
> >   	struct rte_eth_txconf *txconf;
> > @@ -1584,10 +1589,10 @@ port_init(uint16_t portid)
> >   		local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_JUMBO_FRAME;
> >   	}
> >
> > -	if (dev_info.rx_offload_capa & DEV_RX_OFFLOAD_SECURITY)
> > -		local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_SECURITY;
> > -	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_SECURITY)
> > -		local_port_conf.txmode.offloads |= DEV_TX_OFFLOAD_SECURITY;
> > +	/* Capabilities will already have been checked.. */
> > +	local_port_conf.rxmode.offloads |= req_rx_offloads;
> > +	local_port_conf.txmode.offloads |= req_tx_offloads;
> > +
> >   	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_MBUF_FAST_FREE)
> >   		local_port_conf.txmode.offloads |=
> >   			DEV_TX_OFFLOAD_MBUF_FAST_FREE;
> > @@ -1639,6 +1644,13 @@ port_init(uint16_t portid)
> >
> >   		qconf = &lcore_conf[lcore_id];
> >   		qconf->tx_queue_id[portid] = tx_queueid;
> > +
> > +		/* Pre-populate pkt offloads based on capabilities */
> > +		qconf->outbound.ipv4_offloads = PKT_TX_IPV4;
> > +		qconf->outbound.ipv6_offloads = PKT_TX_IPV6;
> > +		if (req_tx_offloads & DEV_TX_OFFLOAD_IPV4_CKSUM)
> > +			qconf->outbound.ipv4_offloads |= PKT_TX_IP_CKSUM;
> > +
> >   		tx_queueid++;
> >
> >   		/* init RX queues */
> > @@ -1749,6 +1761,7 @@ main(int32_t argc, char **argv)
> >   	uint32_t lcore_id;
> >   	uint8_t socket_id;
> >   	uint16_t portid;
> > +	uint64_t req_rx_offloads, req_tx_offloads;
> >
> >   	/* init EAL */
> >   	ret = rte_eal_init(argc, argv);
> > @@ -1804,7 +1817,8 @@ main(int32_t argc, char **argv)
> >   		if ((enabled_port_mask & (1 << portid)) == 0)
> >   			continue;
> >
> > -		port_init(portid);
> > +		sa_check_offloads(portid, &req_rx_offloads, &req_tx_offloads);
> > +		port_init(portid, req_rx_offloads, req_tx_offloads);
> >   	}
> >
> >   	cryptodevs_init();
> > diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
> > index c998c8076..9b1586f52 100644
> > --- a/examples/ipsec-secgw/ipsec.h
> > +++ b/examples/ipsec-secgw/ipsec.h
> > @@ -146,6 +146,8 @@ struct ipsec_ctx {
> >   	struct rte_mempool *session_pool;
> >   	struct rte_mbuf *ol_pkts[MAX_PKT_BURST] __rte_aligned(sizeof(void *));
> >   	uint16_t ol_pkts_cnt;
> > +	uint64_t ipv4_offloads;
> > +	uint64_t ipv6_offloads;
> >   };
> >
> >   struct cdev_key {
> > @@ -239,4 +241,8 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id);
> >   void
> >   rt_init(struct socket_ctx *ctx, int32_t socket_id);
> >
> > +int
> > +sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
> > +		uint64_t *tx_offloads);
> > +
> >   #endif /* __IPSEC_H__ */
> > diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
> > index d2d3550a4..ff8c4b829 100644
> > --- a/examples/ipsec-secgw/sa.c
> > +++ b/examples/ipsec-secgw/sa.c
> > @@ -1017,3 +1017,59 @@ outbound_sa_lookup(struct sa_ctx *sa_ctx, uint32_t sa_idx[],
> >   	for (i = 0; i < nb_pkts; i++)
> >   		sa[i] = &sa_ctx->sa[sa_idx[i]];
> >   }
> > +
> > +/*
> > + * Select HW offloads to be used.
> > + */
> > +int
> > +sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
> > +		uint64_t *tx_offloads)
> > +{
> > +	struct ipsec_sa *rule;
> > +	uint32_t idx_sa;
> > +	struct rte_eth_dev_info dev_info;
> > +
> > +	rte_eth_dev_info_get(port_id, &dev_info);
> > +
> > +	*rx_offloads = 0;
> > +	*tx_offloads = 0;
> > +
> > +	/* Check for inbound rules that use offloads and use this port */
> > +	for (idx_sa = 0; idx_sa < nb_sa_in; idx_sa++) {
> > +		rule = &sa_in[idx_sa];
> > +		if ((rule->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
> > +				rule->type ==
> > +				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL)
> > +				&& rule->portid == port_id) {
> > +			if ((dev_info.rx_offload_capa & DEV_RX_OFFLOAD_SECURITY)
> > +					== 0) {
> > +				RTE_LOG(WARNING, PORT,
> > +					"HW RX IPSec is not supported\n");
> > +				return -EINVAL;
> > +			}
> > +			*rx_offloads |= DEV_RX_OFFLOAD_SECURITY;
> > +		}
> > +	}
> > +
> > +	/* Check for outbound rules that use offloads and use this port */
> > +	for (idx_sa = 0; idx_sa < nb_sa_out; idx_sa++) {
> > +		rule = &sa_out[idx_sa];
> > +		if ((rule->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
> > +				rule->type ==
> > +				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL)
> > +				&& rule->portid == port_id) {
> > +			if ((dev_info.tx_offload_capa & DEV_TX_OFFLOAD_SECURITY)
> > +					== 0) {
> > +				RTE_LOG(WARNING, PORT,
> > +					"HW TX IPSec is not supported\n");
> > +				return -EINVAL;
> > +			}
> > +			*tx_offloads |= DEV_TX_OFFLOAD_SECURITY;
> > +			/* Enable HW IPv4 cksum as well, if it is available */
> > +			if (dev_info.tx_offload_capa &
> > +					DEV_TX_OFFLOAD_IPV4_CKSUM)
> > +				*tx_offloads |= DEV_TX_OFFLOAD_IPV4_CKSUM;
> > +		}
> > +	}
> > +	return 0;
> > +}

Re: [PATCH v4 1/9] examples/ipsec-secgw: avoid to request unused TX offloads

[Akhil Goyal]

On 12/21/2018 9:28 PM, Ananyev, Konstantin wrote:
>
>> -----Original Message-----
>> From: Akhil Goyal [mailto:akhil.goyal@nxp.com]
>> Sent: Friday, December 21, 2018 1:57 PM
>> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
>> Cc: Nicolau, Radu <radu.nicolau@intel.com>; Horton, Remy <remy.horton@intel.com>
>> Subject: Re: [dpdk-dev] [PATCH v4 1/9] examples/ipsec-secgw: avoid to request unused TX offloads
>>
>> Hi Konstantin,
>>
>> On 12/14/2018 10:10 PM, Konstantin Ananyev wrote:
>>> ipsec-secgw always enables TX offloads
>>> (DEV_TX_OFFLOAD_MULTI_SEGS, DEV_TX_OFFLOAD_SECURITY),
>>> even when they are not requested by the config.
>>> That causes many PMD to choose full-featured TX function,
>>> which in many cases is much slower then one without offloads.
>>> That patch adds checks to enabled extra HW offloads, only when
>>> they were requested.
>>> Plus it enables DEV_TX_OFFLOAD_IPV4_CKSUM,
>>> only when other HW TX ofloads are going to be enabled.
>>> Otherwise SW version of ip cksum calculation is used.
>>> That allows to use vector TX function, when inline-ipsec is not
>>> requested.
>>>
>>> Signed-off-by: Remy Horton <remy.horton@intel.com>
>>> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
>>> Acked-by: Radu Nicolau <radu.nicolau@intel.com>
>>> ---
>>>    examples/ipsec-secgw/ipsec-secgw.c | 44 +++++++++++++++--------
>>>    examples/ipsec-secgw/ipsec.h       |  6 ++++
>>>    examples/ipsec-secgw/sa.c          | 56 ++++++++++++++++++++++++++++++
>>>    3 files changed, 91 insertions(+), 15 deletions(-)
>>>
>>> diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
>>> index 1bc0b5b50..cfc2b05e5 100644
>>> --- a/examples/ipsec-secgw/ipsec-secgw.c
>>> +++ b/examples/ipsec-secgw/ipsec-secgw.c
>>> @@ -208,8 +208,6 @@ static struct rte_eth_conf port_conf = {
>>>    	},
>>>    	.txmode = {
>>>    		.mq_mode = ETH_MQ_TX_NONE,
>>> -		.offloads = (DEV_TX_OFFLOAD_IPV4_CKSUM |
>>> -			     DEV_TX_OFFLOAD_MULTI_SEGS),
>> I believe this is disabling checksum offload for all cases and then
>> enabling only for inline crypto and inline proto.
> Yes.
>
>> This is breaking lookaside proto and lookaside none cases. Please
>> correct me if I am wrong.
> Why breaking?
reduction in performance is kind of breaking the code.
> For cases when HW cksum offload is disabled, IPv4 cksum calculation
> will be done in SW, see below:
> prepare_tx_pkt(...)
> {
>     ...
>      +
>      +		/* calculate IPv4 cksum in SW */
>      +		if ((pkt->ol_flags & PKT_TX_IP_CKSUM) == 0)
>      +			ip->ip_sum = rte_ipv4_cksum((struct ipv4_hdr *)ip);
>
>
> We tested lookaside-none case quite extensively - all works well,
> in fact on Intel NICs it became even a bit faster because of that change
> (though not much).
yes, it may work well on one hardware, but may not perform good in other 
hardware where cores are limited.
> Disabling HW offloads when they are not really required has 2 benefits:
>   1) allows app to be run on NICs without HW offloads support.
>   2) allows dev_configure() for TX path to select simple/vector TX functions
>       which for many NICs are significantly faster.
>
> Konstantin
>
>> So a NACK for this if my understanding is correct.
>>

Re: [PATCH v4 1/9] examples/ipsec-secgw: avoid to request unused TX offloads

[Ananyev, Konstantin]

> >>
> >> On 12/14/2018 10:10 PM, Konstantin Ananyev wrote:
> >>> ipsec-secgw always enables TX offloads
> >>> (DEV_TX_OFFLOAD_MULTI_SEGS, DEV_TX_OFFLOAD_SECURITY),
> >>> even when they are not requested by the config.
> >>> That causes many PMD to choose full-featured TX function,
> >>> which in many cases is much slower then one without offloads.
> >>> That patch adds checks to enabled extra HW offloads, only when
> >>> they were requested.
> >>> Plus it enables DEV_TX_OFFLOAD_IPV4_CKSUM,
> >>> only when other HW TX ofloads are going to be enabled.
> >>> Otherwise SW version of ip cksum calculation is used.
> >>> That allows to use vector TX function, when inline-ipsec is not
> >>> requested.
> >>>
> >>> Signed-off-by: Remy Horton <remy.horton@intel.com>
> >>> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> >>> Acked-by: Radu Nicolau <radu.nicolau@intel.com>
> >>> ---
> >>>    examples/ipsec-secgw/ipsec-secgw.c | 44 +++++++++++++++--------
> >>>    examples/ipsec-secgw/ipsec.h       |  6 ++++
> >>>    examples/ipsec-secgw/sa.c          | 56 ++++++++++++++++++++++++++++++
> >>>    3 files changed, 91 insertions(+), 15 deletions(-)
> >>>
> >>> diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
> >>> index 1bc0b5b50..cfc2b05e5 100644
> >>> --- a/examples/ipsec-secgw/ipsec-secgw.c
> >>> +++ b/examples/ipsec-secgw/ipsec-secgw.c
> >>> @@ -208,8 +208,6 @@ static struct rte_eth_conf port_conf = {
> >>>    	},
> >>>    	.txmode = {
> >>>    		.mq_mode = ETH_MQ_TX_NONE,
> >>> -		.offloads = (DEV_TX_OFFLOAD_IPV4_CKSUM |
> >>> -			     DEV_TX_OFFLOAD_MULTI_SEGS),
> >> I believe this is disabling checksum offload for all cases and then
> >> enabling only for inline crypto and inline proto.
> > Yes.
> >
> >> This is breaking lookaside proto and lookaside none cases. Please
> >> correct me if I am wrong.
> > Why breaking?
> reduction in performance is kind of breaking the code.

I didn’t observe any performance drop with that patch.
In fact there was a tiny improvement (see below).
Did you see any regression with this patch on your HW?

> > For cases when HW cksum offload is disabled, IPv4 cksum calculation
> > will be done in SW, see below:
> > prepare_tx_pkt(...)
> > {
> >     ...
> >      +
> >      +		/* calculate IPv4 cksum in SW */
> >      +		if ((pkt->ol_flags & PKT_TX_IP_CKSUM) == 0)
> >      +			ip->ip_sum = rte_ipv4_cksum((struct ipv4_hdr *)ip);
> >
> >
> > We tested lookaside-none case quite extensively - all works well,
> > in fact on Intel NICs it became even a bit faster because of that change
> > (though not much).
> yes, it may work well on one hardware, but may not perform good in other
> hardware where cores are limited.

Could you elaborate a bit more what do you mean by 'cores are limited' here?
Do you mean that for some low end cpus calculating IPv4 cksum in SW is too expensive?
Note that prepare_tx_pkts() and friends read/write L2/L3 packet headers anyway -
so IPv4 header will be in L1 cache already.

> > Disabling HW offloads when they are not really required has 2 benefits:
> >   1) allows app to be run on NICs without HW offloads support.
> >   2) allows dev_configure() for TX path to select simple/vector TX functions
> >       which for many NICs are significantly faster.
> >
> > Konstantin
> >
> >> So a NACK for this if my understanding is correct.
> >>

Re: [PATCH v4 1/9] examples/ipsec-secgw: avoid to request unused TX offloads

[Akhil Goyal]

On 12/24/2018 3:49 PM, Ananyev, Konstantin wrote:
>
>>>> On 12/14/2018 10:10 PM, Konstantin Ananyev wrote:
>>>>> ipsec-secgw always enables TX offloads
>>>>> (DEV_TX_OFFLOAD_MULTI_SEGS, DEV_TX_OFFLOAD_SECURITY),
>>>>> even when they are not requested by the config.
>>>>> That causes many PMD to choose full-featured TX function,
>>>>> which in many cases is much slower then one without offloads.
>>>>> That patch adds checks to enabled extra HW offloads, only when
>>>>> they were requested.
>>>>> Plus it enables DEV_TX_OFFLOAD_IPV4_CKSUM,
>>>>> only when other HW TX ofloads are going to be enabled.
>>>>> Otherwise SW version of ip cksum calculation is used.
>>>>> That allows to use vector TX function, when inline-ipsec is not
>>>>> requested.
>>>>>
>>>>> Signed-off-by: Remy Horton <remy.horton@intel.com>
>>>>> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
>>>>> Acked-by: Radu Nicolau <radu.nicolau@intel.com>
>>>>> ---
>>>>>     examples/ipsec-secgw/ipsec-secgw.c | 44 +++++++++++++++--------
>>>>>     examples/ipsec-secgw/ipsec.h       |  6 ++++
>>>>>     examples/ipsec-secgw/sa.c          | 56 ++++++++++++++++++++++++++++++
>>>>>     3 files changed, 91 insertions(+), 15 deletions(-)
>>>>>
>>>>> diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
>>>>> index 1bc0b5b50..cfc2b05e5 100644
>>>>> --- a/examples/ipsec-secgw/ipsec-secgw.c
>>>>> +++ b/examples/ipsec-secgw/ipsec-secgw.c
>>>>> @@ -208,8 +208,6 @@ static struct rte_eth_conf port_conf = {
>>>>>     	},
>>>>>     	.txmode = {
>>>>>     		.mq_mode = ETH_MQ_TX_NONE,
>>>>> -		.offloads = (DEV_TX_OFFLOAD_IPV4_CKSUM |
>>>>> -			     DEV_TX_OFFLOAD_MULTI_SEGS),
>>>> I believe this is disabling checksum offload for all cases and then
>>>> enabling only for inline crypto and inline proto.
>>> Yes.
>>>
>>>> This is breaking lookaside proto and lookaside none cases. Please
>>>> correct me if I am wrong.
>>> Why breaking?
>> reduction in performance is kind of breaking the code.
> I didn’t observe any performance drop with that patch.
> In fact there was a tiny improvement (see below).
> Did you see any regression with this patch on your HW?
NXP hardware are low -end to mid end devices and we are always 
bottleneck by core cycles.
So we would like to have as much offloads to HW as possible.
>
>>> For cases when HW cksum offload is disabled, IPv4 cksum calculation
>>> will be done in SW, see below:
>>> prepare_tx_pkt(...)
>>> {
>>>      ...
>>>       +
>>>       +		/* calculate IPv4 cksum in SW */
>>>       +		if ((pkt->ol_flags & PKT_TX_IP_CKSUM) == 0)
>>>       +			ip->ip_sum = rte_ipv4_cksum((struct ipv4_hdr *)ip);
>>>
>>>
>>> We tested lookaside-none case quite extensively - all works well,
>>> in fact on Intel NICs it became even a bit faster because of that change
>>> (though not much).
>> yes, it may work well on one hardware, but may not perform good in other
>> hardware where cores are limited.
> Could you elaborate a bit more what do you mean by 'cores are limited' here?
we have single core devices as well on which we run ipsec-secgw.
> Do you mean that for some low end cpus calculating IPv4 cksum in SW is too expensive?
yes, limited by core cycles and not by HW
> Note that prepare_tx_pkts() and friends read/write L2/L3 packet headers anyway -
> so IPv4 header will be in L1 cache already.
Agreed, but still it will consume some cycles which are more than that 
of HW.
>
>>> Disabling HW offloads when they are not really required has 2 benefits:
>>>    1) allows app to be run on NICs without HW offloads support.
>>>    2) allows dev_configure() for TX path to select simple/vector TX functions
>>>        which for many NICs are significantly faster.
>>>
>>> Konstantin
>>>
>>>> So a NACK for this if my understanding is correct.
>>>>

Re: [PATCH v4 1/9] examples/ipsec-secgw: avoid to request unused TX offloads

[Akhil Goyal]

On 12/24/2018 4:24 PM, Akhil Goyal wrote:
>
> On 12/24/2018 3:49 PM, Ananyev, Konstantin wrote:
>>>>> On 12/14/2018 10:10 PM, Konstantin Ananyev wrote:
>>>>>> ipsec-secgw always enables TX offloads
>>>>>> (DEV_TX_OFFLOAD_MULTI_SEGS, DEV_TX_OFFLOAD_SECURITY),
>>>>>> even when they are not requested by the config.
>>>>>> That causes many PMD to choose full-featured TX function,
>>>>>> which in many cases is much slower then one without offloads.
>>>>>> That patch adds checks to enabled extra HW offloads, only when
>>>>>> they were requested.
>>>>>> Plus it enables DEV_TX_OFFLOAD_IPV4_CKSUM,
>>>>>> only when other HW TX ofloads are going to be enabled.
>>>>>> Otherwise SW version of ip cksum calculation is used.
>>>>>> That allows to use vector TX function, when inline-ipsec is not
>>>>>> requested.
>>>>>>
>>>>>> Signed-off-by: Remy Horton <remy.horton@intel.com>
>>>>>> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
>>>>>> Acked-by: Radu Nicolau <radu.nicolau@intel.com>
>>>>>> ---
>>>>>>      examples/ipsec-secgw/ipsec-secgw.c | 44 +++++++++++++++--------
>>>>>>      examples/ipsec-secgw/ipsec.h       |  6 ++++
>>>>>>      examples/ipsec-secgw/sa.c          | 56 ++++++++++++++++++++++++++++++
>>>>>>      3 files changed, 91 insertions(+), 15 deletions(-)
>>>>>>
>>>>>> diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
>>>>>> index 1bc0b5b50..cfc2b05e5 100644
>>>>>> --- a/examples/ipsec-secgw/ipsec-secgw.c
>>>>>> +++ b/examples/ipsec-secgw/ipsec-secgw.c
>>>>>> @@ -208,8 +208,6 @@ static struct rte_eth_conf port_conf = {
>>>>>>      	},
>>>>>>      	.txmode = {
>>>>>>      		.mq_mode = ETH_MQ_TX_NONE,
>>>>>> -		.offloads = (DEV_TX_OFFLOAD_IPV4_CKSUM |
>>>>>> -			     DEV_TX_OFFLOAD_MULTI_SEGS),
>>>>> I believe this is disabling checksum offload for all cases and then
>>>>> enabling only for inline crypto and inline proto.
>>>> Yes.
>>>>
>>>>> This is breaking lookaside proto and lookaside none cases. Please
>>>>> correct me if I am wrong.
>>>> Why breaking?
>>> reduction in performance is kind of breaking the code.
>> I didn’t observe any performance drop with that patch.
>> In fact there was a tiny improvement (see below).
>> Did you see any regression with this patch on your HW?
> NXP hardware are low -end to mid end devices and we are always
> bottleneck by core cycles.
> So we would like to have as much offloads to HW as possible.
>>>> For cases when HW cksum offload is disabled, IPv4 cksum calculation
>>>> will be done in SW, see below:
>>>> prepare_tx_pkt(...)
>>>> {
>>>>       ...
>>>>        +
>>>>        +		/* calculate IPv4 cksum in SW */
>>>>        +		if ((pkt->ol_flags & PKT_TX_IP_CKSUM) == 0)
>>>>        +			ip->ip_sum = rte_ipv4_cksum((struct ipv4_hdr *)ip);
>>>>
>>>>
>>>> We tested lookaside-none case quite extensively - all works well,
>>>> in fact on Intel NICs it became even a bit faster because of that change
>>>> (though not much).
>>> yes, it may work well on one hardware, but may not perform good in other
>>> hardware where cores are limited.
>> Could you elaborate a bit more what do you mean by 'cores are limited' here?
> we have single core devices as well on which we run ipsec-secgw.
>> Do you mean that for some low end cpus calculating IPv4 cksum in SW is too expensive?
> yes, limited by core cycles and not by HW
>> Note that prepare_tx_pkts() and friends read/write L2/L3 packet headers anyway -
>> so IPv4 header will be in L1 cache already.
> Agreed, but still it will consume some cycles which are more than that
> of HW.
Impact will be lesser in lookaside none, but will be more in lookaside 
offload.
>>>> Disabling HW offloads when they are not really required has 2 benefits:
>>>>     1) allows app to be run on NICs without HW offloads support.
>>>>     2) allows dev_configure() for TX path to select simple/vector TX functions
>>>>         which for many NICs are significantly faster.
>>>>
>>>> Konstantin
>>>>
>>>>> So a NACK for this if my understanding is correct.
>>>>>

Re: [PATCH v4 1/9] examples/ipsec-secgw: avoid to request unused TX offloads

[Ananyev, Konstantin]

> -----Original Message-----
> From: Akhil Goyal [mailto:akhil.goyal@nxp.com]
> Sent: Monday, December 24, 2018 10:54 AM
> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
> Cc: Nicolau, Radu <radu.nicolau@intel.com>; Horton, Remy <remy.horton@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v4 1/9] examples/ipsec-secgw: avoid to request unused TX offloads
> 
> 
> 
> On 12/24/2018 3:49 PM, Ananyev, Konstantin wrote:
> >
> >>>> On 12/14/2018 10:10 PM, Konstantin Ananyev wrote:
> >>>>> ipsec-secgw always enables TX offloads
> >>>>> (DEV_TX_OFFLOAD_MULTI_SEGS, DEV_TX_OFFLOAD_SECURITY),
> >>>>> even when they are not requested by the config.
> >>>>> That causes many PMD to choose full-featured TX function,
> >>>>> which in many cases is much slower then one without offloads.
> >>>>> That patch adds checks to enabled extra HW offloads, only when
> >>>>> they were requested.
> >>>>> Plus it enables DEV_TX_OFFLOAD_IPV4_CKSUM,
> >>>>> only when other HW TX ofloads are going to be enabled.
> >>>>> Otherwise SW version of ip cksum calculation is used.
> >>>>> That allows to use vector TX function, when inline-ipsec is not
> >>>>> requested.
> >>>>>
> >>>>> Signed-off-by: Remy Horton <remy.horton@intel.com>
> >>>>> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> >>>>> Acked-by: Radu Nicolau <radu.nicolau@intel.com>
> >>>>> ---
> >>>>>     examples/ipsec-secgw/ipsec-secgw.c | 44 +++++++++++++++--------
> >>>>>     examples/ipsec-secgw/ipsec.h       |  6 ++++
> >>>>>     examples/ipsec-secgw/sa.c          | 56 ++++++++++++++++++++++++++++++
> >>>>>     3 files changed, 91 insertions(+), 15 deletions(-)
> >>>>>
> >>>>> diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
> >>>>> index 1bc0b5b50..cfc2b05e5 100644
> >>>>> --- a/examples/ipsec-secgw/ipsec-secgw.c
> >>>>> +++ b/examples/ipsec-secgw/ipsec-secgw.c
> >>>>> @@ -208,8 +208,6 @@ static struct rte_eth_conf port_conf = {
> >>>>>     	},
> >>>>>     	.txmode = {
> >>>>>     		.mq_mode = ETH_MQ_TX_NONE,
> >>>>> -		.offloads = (DEV_TX_OFFLOAD_IPV4_CKSUM |
> >>>>> -			     DEV_TX_OFFLOAD_MULTI_SEGS),
> >>>> I believe this is disabling checksum offload for all cases and then
> >>>> enabling only for inline crypto and inline proto.
> >>> Yes.
> >>>
> >>>> This is breaking lookaside proto and lookaside none cases. Please
> >>>> correct me if I am wrong.
> >>> Why breaking?
> >> reduction in performance is kind of breaking the code.
> > I didn’t observe any performance drop with that patch.
> > In fact there was a tiny improvement (see below).
> > Did you see any regression with this patch on your HW?
> NXP hardware are low -end to mid end devices and we are always
> bottleneck by core cycles.
> So we would like to have as much offloads to HW as possible.

Ok, then I suppose we need to introduce new cmd-line options,
Something like: --txoffloads=<tx_offload_mask> --rx_offloads=<rx_offload_mask>
to keep everyone happy.
Are you ok with that?
Konstantin 

> >
> >>> For cases when HW cksum offload is disabled, IPv4 cksum calculation
> >>> will be done in SW, see below:
> >>> prepare_tx_pkt(...)
> >>> {
> >>>      ...
> >>>       +
> >>>       +		/* calculate IPv4 cksum in SW */
> >>>       +		if ((pkt->ol_flags & PKT_TX_IP_CKSUM) == 0)
> >>>       +			ip->ip_sum = rte_ipv4_cksum((struct ipv4_hdr *)ip);
> >>>
> >>>
> >>> We tested lookaside-none case quite extensively - all works well,
> >>> in fact on Intel NICs it became even a bit faster because of that change
> >>> (though not much).
> >> yes, it may work well on one hardware, but may not perform good in other
> >> hardware where cores are limited.
> > Could you elaborate a bit more what do you mean by 'cores are limited' here?
> we have single core devices as well on which we run ipsec-secgw.
> > Do you mean that for some low end cpus calculating IPv4 cksum in SW is too expensive?
> yes, limited by core cycles and not by HW
> > Note that prepare_tx_pkts() and friends read/write L2/L3 packet headers anyway -
> > so IPv4 header will be in L1 cache already.
> Agreed, but still it will consume some cycles which are more than that
> of HW.
> >
> >>> Disabling HW offloads when they are not really required has 2 benefits:
> >>>    1) allows app to be run on NICs without HW offloads support.
> >>>    2) allows dev_configure() for TX path to select simple/vector TX functions
> >>>        which for many NICs are significantly faster.
> >>>
> >>> Konstantin
> >>>
> >>>> So a NACK for this if my understanding is correct.
> >>>>

Re: [PATCH v4 1/9] examples/ipsec-secgw: avoid to request unused TX offloads

[Akhil Goyal]

On 12/24/2018 4:52 PM, Ananyev, Konstantin wrote:
>
>> -----Original Message-----
>> From: Akhil Goyal [mailto:akhil.goyal@nxp.com]
>> Sent: Monday, December 24, 2018 10:54 AM
>> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
>> Cc: Nicolau, Radu <radu.nicolau@intel.com>; Horton, Remy <remy.horton@intel.com>
>> Subject: Re: [dpdk-dev] [PATCH v4 1/9] examples/ipsec-secgw: avoid to request unused TX offloads
>>
>>
>>
>> On 12/24/2018 3:49 PM, Ananyev, Konstantin wrote:
>>>>>> On 12/14/2018 10:10 PM, Konstantin Ananyev wrote:
>>>>>>> ipsec-secgw always enables TX offloads
>>>>>>> (DEV_TX_OFFLOAD_MULTI_SEGS, DEV_TX_OFFLOAD_SECURITY),
>>>>>>> even when they are not requested by the config.
>>>>>>> That causes many PMD to choose full-featured TX function,
>>>>>>> which in many cases is much slower then one without offloads.
>>>>>>> That patch adds checks to enabled extra HW offloads, only when
>>>>>>> they were requested.
>>>>>>> Plus it enables DEV_TX_OFFLOAD_IPV4_CKSUM,
>>>>>>> only when other HW TX ofloads are going to be enabled.
>>>>>>> Otherwise SW version of ip cksum calculation is used.
>>>>>>> That allows to use vector TX function, when inline-ipsec is not
>>>>>>> requested.
>>>>>>>
>>>>>>> Signed-off-by: Remy Horton <remy.horton@intel.com>
>>>>>>> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
>>>>>>> Acked-by: Radu Nicolau <radu.nicolau@intel.com>
>>>>>>> ---
>>>>>>>      examples/ipsec-secgw/ipsec-secgw.c | 44 +++++++++++++++--------
>>>>>>>      examples/ipsec-secgw/ipsec.h       |  6 ++++
>>>>>>>      examples/ipsec-secgw/sa.c          | 56 ++++++++++++++++++++++++++++++
>>>>>>>      3 files changed, 91 insertions(+), 15 deletions(-)
>>>>>>>
>>>>>>> diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
>>>>>>> index 1bc0b5b50..cfc2b05e5 100644
>>>>>>> --- a/examples/ipsec-secgw/ipsec-secgw.c
>>>>>>> +++ b/examples/ipsec-secgw/ipsec-secgw.c
>>>>>>> @@ -208,8 +208,6 @@ static struct rte_eth_conf port_conf = {
>>>>>>>      	},
>>>>>>>      	.txmode = {
>>>>>>>      		.mq_mode = ETH_MQ_TX_NONE,
>>>>>>> -		.offloads = (DEV_TX_OFFLOAD_IPV4_CKSUM |
>>>>>>> -			     DEV_TX_OFFLOAD_MULTI_SEGS),
>>>>>> I believe this is disabling checksum offload for all cases and then
>>>>>> enabling only for inline crypto and inline proto.
>>>>> Yes.
>>>>>
>>>>>> This is breaking lookaside proto and lookaside none cases. Please
>>>>>> correct me if I am wrong.
>>>>> Why breaking?
>>>> reduction in performance is kind of breaking the code.
>>> I didn’t observe any performance drop with that patch.
>>> In fact there was a tiny improvement (see below).
>>> Did you see any regression with this patch on your HW?
>> NXP hardware are low -end to mid end devices and we are always
>> bottleneck by core cycles.
>> So we would like to have as much offloads to HW as possible.
> Ok, then I suppose we need to introduce new cmd-line options,
> Something like: --txoffloads=<tx_offload_mask> --rx_offloads=<rx_offload_mask>
> to keep everyone happy.
> Are you ok with that?
I think it should be taken from the PMD capabilities. cmd line for every 
parameter will make it very complex.
> Konstantin
>
>>>>> For cases when HW cksum offload is disabled, IPv4 cksum calculation
>>>>> will be done in SW, see below:
>>>>> prepare_tx_pkt(...)
>>>>> {
>>>>>       ...
>>>>>        +
>>>>>        +		/* calculate IPv4 cksum in SW */
>>>>>        +		if ((pkt->ol_flags & PKT_TX_IP_CKSUM) == 0)
>>>>>        +			ip->ip_sum = rte_ipv4_cksum((struct ipv4_hdr *)ip);
>>>>>
>>>>>
>>>>> We tested lookaside-none case quite extensively - all works well,
>>>>> in fact on Intel NICs it became even a bit faster because of that change
>>>>> (though not much).
>>>> yes, it may work well on one hardware, but may not perform good in other
>>>> hardware where cores are limited.
>>> Could you elaborate a bit more what do you mean by 'cores are limited' here?
>> we have single core devices as well on which we run ipsec-secgw.
>>> Do you mean that for some low end cpus calculating IPv4 cksum in SW is too expensive?
>> yes, limited by core cycles and not by HW
>>> Note that prepare_tx_pkts() and friends read/write L2/L3 packet headers anyway -
>>> so IPv4 header will be in L1 cache already.
>> Agreed, but still it will consume some cycles which are more than that
>> of HW.
>>>>> Disabling HW offloads when they are not really required has 2 benefits:
>>>>>     1) allows app to be run on NICs without HW offloads support.
>>>>>     2) allows dev_configure() for TX path to select simple/vector TX functions
>>>>>         which for many NICs are significantly faster.
>>>>>
>>>>> Konstantin
>>>>>
>>>>>> So a NACK for this if my understanding is correct.
>>>>>>

Re: [PATCH v4 1/9] examples/ipsec-secgw: avoid to request unused TX offloads

[Ananyev, Konstantin]

> -----Original Message-----
> From: Akhil Goyal [mailto:akhil.goyal@nxp.com]
> Sent: Monday, December 24, 2018 11:25 AM
> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
> Cc: Nicolau, Radu <radu.nicolau@intel.com>; Horton, Remy <remy.horton@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v4 1/9] examples/ipsec-secgw: avoid to request unused TX offloads
> 
> 
> 
> On 12/24/2018 4:52 PM, Ananyev, Konstantin wrote:
> >
> >> -----Original Message-----
> >> From: Akhil Goyal [mailto:akhil.goyal@nxp.com]
> >> Sent: Monday, December 24, 2018 10:54 AM
> >> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
> >> Cc: Nicolau, Radu <radu.nicolau@intel.com>; Horton, Remy <remy.horton@intel.com>
> >> Subject: Re: [dpdk-dev] [PATCH v4 1/9] examples/ipsec-secgw: avoid to request unused TX offloads
> >>
> >>
> >>
> >> On 12/24/2018 3:49 PM, Ananyev, Konstantin wrote:
> >>>>>> On 12/14/2018 10:10 PM, Konstantin Ananyev wrote:
> >>>>>>> ipsec-secgw always enables TX offloads
> >>>>>>> (DEV_TX_OFFLOAD_MULTI_SEGS, DEV_TX_OFFLOAD_SECURITY),
> >>>>>>> even when they are not requested by the config.
> >>>>>>> That causes many PMD to choose full-featured TX function,
> >>>>>>> which in many cases is much slower then one without offloads.
> >>>>>>> That patch adds checks to enabled extra HW offloads, only when
> >>>>>>> they were requested.
> >>>>>>> Plus it enables DEV_TX_OFFLOAD_IPV4_CKSUM,
> >>>>>>> only when other HW TX ofloads are going to be enabled.
> >>>>>>> Otherwise SW version of ip cksum calculation is used.
> >>>>>>> That allows to use vector TX function, when inline-ipsec is not
> >>>>>>> requested.
> >>>>>>>
> >>>>>>> Signed-off-by: Remy Horton <remy.horton@intel.com>
> >>>>>>> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> >>>>>>> Acked-by: Radu Nicolau <radu.nicolau@intel.com>
> >>>>>>> ---
> >>>>>>>      examples/ipsec-secgw/ipsec-secgw.c | 44 +++++++++++++++--------
> >>>>>>>      examples/ipsec-secgw/ipsec.h       |  6 ++++
> >>>>>>>      examples/ipsec-secgw/sa.c          | 56 ++++++++++++++++++++++++++++++
> >>>>>>>      3 files changed, 91 insertions(+), 15 deletions(-)
> >>>>>>>
> >>>>>>> diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
> >>>>>>> index 1bc0b5b50..cfc2b05e5 100644
> >>>>>>> --- a/examples/ipsec-secgw/ipsec-secgw.c
> >>>>>>> +++ b/examples/ipsec-secgw/ipsec-secgw.c
> >>>>>>> @@ -208,8 +208,6 @@ static struct rte_eth_conf port_conf = {
> >>>>>>>      	},
> >>>>>>>      	.txmode = {
> >>>>>>>      		.mq_mode = ETH_MQ_TX_NONE,
> >>>>>>> -		.offloads = (DEV_TX_OFFLOAD_IPV4_CKSUM |
> >>>>>>> -			     DEV_TX_OFFLOAD_MULTI_SEGS),
> >>>>>> I believe this is disabling checksum offload for all cases and then
> >>>>>> enabling only for inline crypto and inline proto.
> >>>>> Yes.
> >>>>>
> >>>>>> This is breaking lookaside proto and lookaside none cases. Please
> >>>>>> correct me if I am wrong.
> >>>>> Why breaking?
> >>>> reduction in performance is kind of breaking the code.
> >>> I didn’t observe any performance drop with that patch.
> >>> In fact there was a tiny improvement (see below).
> >>> Did you see any regression with this patch on your HW?
> >> NXP hardware are low -end to mid end devices and we are always
> >> bottleneck by core cycles.
> >> So we would like to have as much offloads to HW as possible.
> > Ok, then I suppose we need to introduce new cmd-line options,
> > Something like: --txoffloads=<tx_offload_mask> --rx_offloads=<rx_offload_mask>
> > to keep everyone happy.
> > Are you ok with that?
> I think it should be taken from the PMD capabilities. 

Don't see how?
Let say, Intel NICs do support HW IPv4 cksum offload, but we don't want
to enable it on its own - only if IPsec offload is also enabled.
From other side you want HW IPv4 cksum offload to be always enabled
if present.
As I can see, to fulfill everyone needs we need to provide user ability
to specify which HW offloads to use. 

> cmd line for every
> parameter will make it very complex.

> > Konstantin
> >
> >>>>> For cases when HW cksum offload is disabled, IPv4 cksum calculation
> >>>>> will be done in SW, see below:
> >>>>> prepare_tx_pkt(...)
> >>>>> {
> >>>>>       ...
> >>>>>        +
> >>>>>        +		/* calculate IPv4 cksum in SW */
> >>>>>        +		if ((pkt->ol_flags & PKT_TX_IP_CKSUM) == 0)
> >>>>>        +			ip->ip_sum = rte_ipv4_cksum((struct ipv4_hdr *)ip);
> >>>>>
> >>>>>
> >>>>> We tested lookaside-none case quite extensively - all works well,
> >>>>> in fact on Intel NICs it became even a bit faster because of that change
> >>>>> (though not much).
> >>>> yes, it may work well on one hardware, but may not perform good in other
> >>>> hardware where cores are limited.
> >>> Could you elaborate a bit more what do you mean by 'cores are limited' here?
> >> we have single core devices as well on which we run ipsec-secgw.
> >>> Do you mean that for some low end cpus calculating IPv4 cksum in SW is too expensive?
> >> yes, limited by core cycles and not by HW
> >>> Note that prepare_tx_pkts() and friends read/write L2/L3 packet headers anyway -
> >>> so IPv4 header will be in L1 cache already.
> >> Agreed, but still it will consume some cycles which are more than that
> >> of HW.
> >>>>> Disabling HW offloads when they are not really required has 2 benefits:
> >>>>>     1) allows app to be run on NICs without HW offloads support.
> >>>>>     2) allows dev_configure() for TX path to select simple/vector TX functions
> >>>>>         which for many NICs are significantly faster.
> >>>>>
> >>>>> Konstantin
> >>>>>
> >>>>>> So a NACK for this if my understanding is correct.
> >>>>>>

Re: [PATCH v4 1/9] examples/ipsec-secgw: avoid to request unused TX offloads

[Akhil Goyal]

On 12/24/2018 5:07 PM, Ananyev, Konstantin wrote:
>
>> -----Original Message-----
>> From: Akhil Goyal [mailto:akhil.goyal@nxp.com]
>> Sent: Monday, December 24, 2018 11:25 AM
>> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
>> Cc: Nicolau, Radu <radu.nicolau@intel.com>; Horton, Remy <remy.horton@intel.com>
>> Subject: Re: [dpdk-dev] [PATCH v4 1/9] examples/ipsec-secgw: avoid to request unused TX offloads
>>
>>
>>
>> On 12/24/2018 4:52 PM, Ananyev, Konstantin wrote:
>>>> -----Original Message-----
>>>> From: Akhil Goyal [mailto:akhil.goyal@nxp.com]
>>>> Sent: Monday, December 24, 2018 10:54 AM
>>>> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
>>>> Cc: Nicolau, Radu <radu.nicolau@intel.com>; Horton, Remy <remy.horton@intel.com>
>>>> Subject: Re: [dpdk-dev] [PATCH v4 1/9] examples/ipsec-secgw: avoid to request unused TX offloads
>>>>
>>>>
>>>>
>>>> On 12/24/2018 3:49 PM, Ananyev, Konstantin wrote:
>>>>>>>> On 12/14/2018 10:10 PM, Konstantin Ananyev wrote:
>>>>>>>>> ipsec-secgw always enables TX offloads
>>>>>>>>> (DEV_TX_OFFLOAD_MULTI_SEGS, DEV_TX_OFFLOAD_SECURITY),
>>>>>>>>> even when they are not requested by the config.
>>>>>>>>> That causes many PMD to choose full-featured TX function,
>>>>>>>>> which in many cases is much slower then one without offloads.
>>>>>>>>> That patch adds checks to enabled extra HW offloads, only when
>>>>>>>>> they were requested.
>>>>>>>>> Plus it enables DEV_TX_OFFLOAD_IPV4_CKSUM,
>>>>>>>>> only when other HW TX ofloads are going to be enabled.
>>>>>>>>> Otherwise SW version of ip cksum calculation is used.
>>>>>>>>> That allows to use vector TX function, when inline-ipsec is not
>>>>>>>>> requested.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Remy Horton <remy.horton@intel.com>
>>>>>>>>> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
>>>>>>>>> Acked-by: Radu Nicolau <radu.nicolau@intel.com>
>>>>>>>>> ---
>>>>>>>>>       examples/ipsec-secgw/ipsec-secgw.c | 44 +++++++++++++++--------
>>>>>>>>>       examples/ipsec-secgw/ipsec.h       |  6 ++++
>>>>>>>>>       examples/ipsec-secgw/sa.c          | 56 ++++++++++++++++++++++++++++++
>>>>>>>>>       3 files changed, 91 insertions(+), 15 deletions(-)
>>>>>>>>>
>>>>>>>>> diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
>>>>>>>>> index 1bc0b5b50..cfc2b05e5 100644
>>>>>>>>> --- a/examples/ipsec-secgw/ipsec-secgw.c
>>>>>>>>> +++ b/examples/ipsec-secgw/ipsec-secgw.c
>>>>>>>>> @@ -208,8 +208,6 @@ static struct rte_eth_conf port_conf = {
>>>>>>>>>       	},
>>>>>>>>>       	.txmode = {
>>>>>>>>>       		.mq_mode = ETH_MQ_TX_NONE,
>>>>>>>>> -		.offloads = (DEV_TX_OFFLOAD_IPV4_CKSUM |
>>>>>>>>> -			     DEV_TX_OFFLOAD_MULTI_SEGS),
>>>>>>>> I believe this is disabling checksum offload for all cases and then
>>>>>>>> enabling only for inline crypto and inline proto.
>>>>>>> Yes.
>>>>>>>
>>>>>>>> This is breaking lookaside proto and lookaside none cases. Please
>>>>>>>> correct me if I am wrong.
>>>>>>> Why breaking?
>>>>>> reduction in performance is kind of breaking the code.
>>>>> I didn’t observe any performance drop with that patch.
>>>>> In fact there was a tiny improvement (see below).
>>>>> Did you see any regression with this patch on your HW?
>>>> NXP hardware are low -end to mid end devices and we are always
>>>> bottleneck by core cycles.
>>>> So we would like to have as much offloads to HW as possible.
>>> Ok, then I suppose we need to introduce new cmd-line options,
>>> Something like: --txoffloads=<tx_offload_mask> --rx_offloads=<rx_offload_mask>
>>> to keep everyone happy.
>>> Are you ok with that?
>> I think it should be taken from the PMD capabilities.
> Don't see how?
> Let say, Intel NICs do support HW IPv4 cksum offload, but we don't want
> to enable it on its own - only if IPsec offload is also enabled.
>  From other side you want HW IPv4 cksum offload to be always enabled
> if present.
> As I can see, to fulfill everyone needs we need to provide user ability
> to specify which HW offloads to use.
if there is no other way to resolve it, then probably you can add an 
optional parameter to disable offloads when it is required.
I believe default behavior should not be changed.
>
>> cmd line for every
>> parameter will make it very complex.
>>> Konstantin
>>>
>>>>>>> For cases when HW cksum offload is disabled, IPv4 cksum calculation
>>>>>>> will be done in SW, see below:
>>>>>>> prepare_tx_pkt(...)
>>>>>>> {
>>>>>>>        ...
>>>>>>>         +
>>>>>>>         +		/* calculate IPv4 cksum in SW */
>>>>>>>         +		if ((pkt->ol_flags & PKT_TX_IP_CKSUM) == 0)
>>>>>>>         +			ip->ip_sum = rte_ipv4_cksum((struct ipv4_hdr *)ip);
>>>>>>>
>>>>>>>
>>>>>>> We tested lookaside-none case quite extensively - all works well,
>>>>>>> in fact on Intel NICs it became even a bit faster because of that change
>>>>>>> (though not much).
>>>>>> yes, it may work well on one hardware, but may not perform good in other
>>>>>> hardware where cores are limited.
>>>>> Could you elaborate a bit more what do you mean by 'cores are limited' here?
>>>> we have single core devices as well on which we run ipsec-secgw.
>>>>> Do you mean that for some low end cpus calculating IPv4 cksum in SW is too expensive?
>>>> yes, limited by core cycles and not by HW
>>>>> Note that prepare_tx_pkts() and friends read/write L2/L3 packet headers anyway -
>>>>> so IPv4 header will be in L1 cache already.
>>>> Agreed, but still it will consume some cycles which are more than that
>>>> of HW.
>>>>>>> Disabling HW offloads when they are not really required has 2 benefits:
>>>>>>>      1) allows app to be run on NICs without HW offloads support.
>>>>>>>      2) allows dev_configure() for TX path to select simple/vector TX functions
>>>>>>>          which for many NICs are significantly faster.
>>>>>>>
>>>>>>> Konstantin
>>>>>>>
>>>>>>>> So a NACK for this if my understanding is correct.
>>>>>>>>

[PATCH v5 00/10] examples/ipsec-secgw: make app to use ipsec library

[Konstantin Ananyev]

This patch series depends on the patch series:

ipsec: new library for IPsec data-path processing
http://patches.dpdk.org/patch/49332/
http://patches.dpdk.org/patch/49333/
http://patches.dpdk.org/patch/49334/
http://patches.dpdk.org/patch/49335/
http://patches.dpdk.org/patch/49336/
http://patches.dpdk.org/patch/49337/
http://patches.dpdk.org/patch/49338/
http://patches.dpdk.org/patch/49339/
http://patches.dpdk.org/patch/49340/
http://patches.dpdk.org/patch/49341/

to be applied first.

v4 -> v5
- Address Akhil comments:
     documentation update
     spell checks spacing etc.
     introduce rxoffload/txoffload parameters
     single SA for ipv6
     update Makefile

v3 -> v4
 - fix few issues with the test scripts
 - update docs

v2 -> v3
 - add IPv6 cases into test scripts
 - fixes for IPv6 support
 - fixes for inline-crypto support
 - some code restructuring

v1 -> v2
 - Several bug fixes

That series contians few bug-fixes and changes to make ipsec-secgw
to utilize librte_ipsec library:
     - changes in the related data structures.
     - changes in the initialization code.
     - changes in the data-path code.
     - new command-line parameters to enable librte_ipsec codepath
       and related features.
     - test scripts to help automate ipsec-secgw functional testing.

Note that right now by default current (non-librte_ipsec) code-path
will be used. User has to run application with new command-line option
('-l')
to enable new codepath.
The main reason for that:
  - current librte_ipsec doesn't support all ipsec algorithms
    and features that the app does.
  - allow users to run both versions in parallel for some time
    to figure out any functional or performance degradation with the
    new code.

Test scripts were run with the following crypto devices:
 - aesni_mb
 - aesni_gcm
 - qat

Konstantin Ananyev (10):
  examples/ipsec-secgw: allow user to disable some RX/TX offloads
  examples/ipsec-secgw: allow to specify neighbour mac address
  examples/ipsec-secgw: fix crypto-op might never get dequeued
  examples/ipsec-secgw: fix outbound codepath for single SA
  examples/ipsec-secgw: make local variables static
  examples/ipsec-secgw: fix inbound SA checking
  examples/ipsec-secgw: make app to use ipsec library
  examples/ipsec-secgw: make data-path to use ipsec library
  examples/ipsec-secgw: add scripts for functional test
  doc: update ipsec-secgw guide and relelase notes

 doc/guides/rel_notes/release_19_02.rst        |  14 +
 doc/guides/sample_app_ug/ipsec_secgw.rst      | 159 +++++-
 examples/ipsec-secgw/Makefile                 |   5 +-
 examples/ipsec-secgw/ipsec-secgw.c            | 480 ++++++++++++++----
 examples/ipsec-secgw/ipsec.c                  |  62 ++-
 examples/ipsec-secgw/ipsec.h                  |  67 +++
 examples/ipsec-secgw/ipsec_process.c          | 341 +++++++++++++
 examples/ipsec-secgw/meson.build              |   6 +-
 examples/ipsec-secgw/parser.c                 |  91 ++++
 examples/ipsec-secgw/parser.h                 |   8 +-
 examples/ipsec-secgw/sa.c                     | 263 +++++++++-
 examples/ipsec-secgw/sp4.c                    |  35 +-
 examples/ipsec-secgw/sp6.c                    |  35 +-
 examples/ipsec-secgw/test/common_defs.sh      | 153 ++++++
 examples/ipsec-secgw/test/data_rxtx.sh        |  62 +++
 examples/ipsec-secgw/test/linux_test4.sh      |  63 +++
 examples/ipsec-secgw/test/linux_test6.sh      |  64 +++
 examples/ipsec-secgw/test/run_test.sh         |  80 +++
 .../test/trs_aescbc_sha1_common_defs.sh       |  69 +++
 .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  67 +++
 .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/trs_aescbc_sha1_esn_defs.sh          |  66 +++
 .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
 .../test/trs_aesgcm_common_defs.sh            |  60 +++
 examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  66 +++
 .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  66 +++
 .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
 .../test/tun_aescbc_sha1_common_defs.sh       |  68 +++
 .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  70 +++
 .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/tun_aescbc_sha1_esn_defs.sh          |  70 +++
 .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
 .../test/tun_aesgcm_common_defs.sh            |  60 +++
 examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  70 +++
 .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  70 +++
 .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
 38 files changed, 2685 insertions(+), 145 deletions(-)
 create mode 100644 examples/ipsec-secgw/ipsec_process.c
 create mode 100644 examples/ipsec-secgw/test/common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test4.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test6.sh
 create mode 100644 examples/ipsec-secgw/test/run_test.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh

-- 
2.17.1

Re: [PATCH v5 00/10] examples/ipsec-secgw: make app to use ipsec library

[Akhil Goyal]

Hi Konstantin,

I just got results on running the ipsec-secgw on NXP hardware.

it seems there is a drop of around 10% for lookaside proto with normal 
command(i.e. without -l option)

with -l option, I got a seg fault while running traffic. gdb suggest 
that pkt_func is not filled up and is NULL.
#1  0x00000000004689bc in rte_ipsec_pkt_crypto_prepare (ss=0x17ad82d80, 
mb=0xffffffffe498, cop=0xffffffffdfc0, num=1)
     at 
/home/akhil/netperf/dpdk_up/dpdk-next-crypto/arm64-dpaa-linuxapp-gcc/include/rte_ipsec.h:115
(gdb)  p /x *ss
$1 = {sa = 0x17ad7ea40, type = 0x3, {crypto = {ses = 0x165a4e900}, 
security = {ses = 0x165a4e900, ctx = 0x0, ol_flags = 0x0}}, pkt_func = {
     prepare = 0x0, process = 0x0}}




On 12/28/2018 9:03 PM, Konstantin Ananyev wrote:
> This patch series depends on the patch series:
>
> ipsec: new library for IPsec data-path processing
> http://patches.dpdk.org/patch/49332/
> http://patches.dpdk.org/patch/49333/
> http://patches.dpdk.org/patch/49334/
> http://patches.dpdk.org/patch/49335/
> http://patches.dpdk.org/patch/49336/
> http://patches.dpdk.org/patch/49337/
> http://patches.dpdk.org/patch/49338/
> http://patches.dpdk.org/patch/49339/
> http://patches.dpdk.org/patch/49340/
> http://patches.dpdk.org/patch/49341/
>
> to be applied first.
>
> v4 -> v5
> - Address Akhil comments:
>       documentation update
>       spell checks spacing etc.
>       introduce rxoffload/txoffload parameters
>       single SA for ipv6
>       update Makefile
>
> v3 -> v4
>   - fix few issues with the test scripts
>   - update docs
>
> v2 -> v3
>   - add IPv6 cases into test scripts
>   - fixes for IPv6 support
>   - fixes for inline-crypto support
>   - some code restructuring
>
> v1 -> v2
>   - Several bug fixes
>
> That series contians few bug-fixes and changes to make ipsec-secgw
> to utilize librte_ipsec library:
>       - changes in the related data structures.
>       - changes in the initialization code.
>       - changes in the data-path code.
>       - new command-line parameters to enable librte_ipsec codepath
>         and related features.
>       - test scripts to help automate ipsec-secgw functional testing.
>
> Note that right now by default current (non-librte_ipsec) code-path
> will be used. User has to run application with new command-line option
> ('-l')
> to enable new codepath.
> The main reason for that:
>    - current librte_ipsec doesn't support all ipsec algorithms
>      and features that the app does.
>    - allow users to run both versions in parallel for some time
>      to figure out any functional or performance degradation with the
>      new code.
>
> Test scripts were run with the following crypto devices:
>   - aesni_mb
>   - aesni_gcm
>   - qat
>
> Konstantin Ananyev (10):
>    examples/ipsec-secgw: allow user to disable some RX/TX offloads
>    examples/ipsec-secgw: allow to specify neighbour mac address
>    examples/ipsec-secgw: fix crypto-op might never get dequeued
>    examples/ipsec-secgw: fix outbound codepath for single SA
>    examples/ipsec-secgw: make local variables static
>    examples/ipsec-secgw: fix inbound SA checking
>    examples/ipsec-secgw: make app to use ipsec library
>    examples/ipsec-secgw: make data-path to use ipsec library
>    examples/ipsec-secgw: add scripts for functional test
>    doc: update ipsec-secgw guide and relelase notes
>
>   doc/guides/rel_notes/release_19_02.rst        |  14 +
>   doc/guides/sample_app_ug/ipsec_secgw.rst      | 159 +++++-
>   examples/ipsec-secgw/Makefile                 |   5 +-
>   examples/ipsec-secgw/ipsec-secgw.c            | 480 ++++++++++++++----
>   examples/ipsec-secgw/ipsec.c                  |  62 ++-
>   examples/ipsec-secgw/ipsec.h                  |  67 +++
>   examples/ipsec-secgw/ipsec_process.c          | 341 +++++++++++++
>   examples/ipsec-secgw/meson.build              |   6 +-
>   examples/ipsec-secgw/parser.c                 |  91 ++++
>   examples/ipsec-secgw/parser.h                 |   8 +-
>   examples/ipsec-secgw/sa.c                     | 263 +++++++++-
>   examples/ipsec-secgw/sp4.c                    |  35 +-
>   examples/ipsec-secgw/sp6.c                    |  35 +-
>   examples/ipsec-secgw/test/common_defs.sh      | 153 ++++++
>   examples/ipsec-secgw/test/data_rxtx.sh        |  62 +++
>   examples/ipsec-secgw/test/linux_test4.sh      |  63 +++
>   examples/ipsec-secgw/test/linux_test6.sh      |  64 +++
>   examples/ipsec-secgw/test/run_test.sh         |  80 +++
>   .../test/trs_aescbc_sha1_common_defs.sh       |  69 +++
>   .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  67 +++
>   .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
>   .../test/trs_aescbc_sha1_esn_defs.sh          |  66 +++
>   .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
>   .../test/trs_aesgcm_common_defs.sh            |  60 +++
>   examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  66 +++
>   .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
>   .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  66 +++
>   .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
>   .../test/tun_aescbc_sha1_common_defs.sh       |  68 +++
>   .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  70 +++
>   .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
>   .../test/tun_aescbc_sha1_esn_defs.sh          |  70 +++
>   .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
>   .../test/tun_aesgcm_common_defs.sh            |  60 +++
>   examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  70 +++
>   .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
>   .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  70 +++
>   .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
>   38 files changed, 2685 insertions(+), 145 deletions(-)
>   create mode 100644 examples/ipsec-secgw/ipsec_process.c
>   create mode 100644 examples/ipsec-secgw/test/common_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
>   create mode 100644 examples/ipsec-secgw/test/linux_test4.sh
>   create mode 100644 examples/ipsec-secgw/test/linux_test6.sh
>   create mode 100644 examples/ipsec-secgw/test/run_test.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh
>

[PATCH v5 01/10] examples/ipsec-secgw: allow user to disable some RX/TX offloads

[Konstantin Ananyev]

Right now ipsec-secgw always enables TX offloads
(DEV_TX_OFFLOAD_MULTI_SEGS, DEV_TX_OFFLOAD_SECURITY),
even when they are not requested by the config.
That causes many PMD to choose full-featured TX function,
which in many cases is much slower then one without offloads.
That patch adds ability for the user to disable unneeded HW offloads.
If DEV_TX_OFFLOAD_IPV4_CKSUM is disabled by user, then
SW version of ip cksum calculation is used.
That allows to use vector TX function, when inline-ipsec is not
requested.

Signed-off-by: Remy Horton <remy.horton@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 doc/guides/sample_app_ug/ipsec_secgw.rst |  12 +++
 examples/ipsec-secgw/ipsec-secgw.c       | 126 ++++++++++++++++++++---
 examples/ipsec-secgw/ipsec.h             |   6 ++
 examples/ipsec-secgw/sa.c                |  35 +++++++
 4 files changed, 164 insertions(+), 15 deletions(-)

diff --git a/doc/guides/sample_app_ug/ipsec_secgw.rst b/doc/guides/sample_app_ug/ipsec_secgw.rst
index 4869a011d..244bde2de 100644
--- a/doc/guides/sample_app_ug/ipsec_secgw.rst
+++ b/doc/guides/sample_app_ug/ipsec_secgw.rst
@@ -95,6 +95,8 @@ The application has a number of command line options::
                         -p PORTMASK -P -u PORTMASK -j FRAMESIZE
                         --config (port,queue,lcore)[,(port,queue,lcore]
                         --single-sa SAIDX
+                        --rxoffload MASK
+                        --txoffload MASK
                         -f CONFIG_FILE_PATH
 
 Where:
@@ -119,6 +121,16 @@ Where:
     on both Inbound and Outbound. This option is meant for debugging/performance
     purposes.
 
+*   ``--rxoffload MASK``: RX HW offload capabilities to enable/use on this port
+    (bitmask of DEV_RX_OFFLOAD_* values). It is an optional parameter and
+    allows user to disable some of the RX HW offload capabilities.
+    By default all HW RX offloads are enabled.
+
+*   ``--txoffload MASK``: TX HW offload capabilities to enable/use on this port
+    (bitmask of DEV_TX_OFFLOAD_* values). It is an optional parameter and
+    allows user to disable some of the TX HW offload capabilities.
+    By default all HW TX offloads are enabled.
+
 *   ``-f CONFIG_FILE_PATH``: the full path of text-based file containing all
     configuration items for running the application (See Configuration file
     syntax section below). ``-f CONFIG_FILE_PATH`` **must** be specified.
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 1bc0b5b50..f5db3da0c 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -124,6 +124,8 @@ struct ethaddr_info ethaddr_tbl[RTE_MAX_ETHPORTS] = {
 #define CMD_LINE_OPT_CONFIG		"config"
 #define CMD_LINE_OPT_SINGLE_SA		"single-sa"
 #define CMD_LINE_OPT_CRYPTODEV_MASK	"cryptodev_mask"
+#define CMD_LINE_OPT_RX_OFFLOAD		"rxoffload"
+#define CMD_LINE_OPT_TX_OFFLOAD		"txoffload"
 
 enum {
 	/* long options mapped to a short option */
@@ -135,12 +137,16 @@ enum {
 	CMD_LINE_OPT_CONFIG_NUM,
 	CMD_LINE_OPT_SINGLE_SA_NUM,
 	CMD_LINE_OPT_CRYPTODEV_MASK_NUM,
+	CMD_LINE_OPT_RX_OFFLOAD_NUM,
+	CMD_LINE_OPT_TX_OFFLOAD_NUM,
 };
 
 static const struct option lgopts[] = {
 	{CMD_LINE_OPT_CONFIG, 1, 0, CMD_LINE_OPT_CONFIG_NUM},
 	{CMD_LINE_OPT_SINGLE_SA, 1, 0, CMD_LINE_OPT_SINGLE_SA_NUM},
 	{CMD_LINE_OPT_CRYPTODEV_MASK, 1, 0, CMD_LINE_OPT_CRYPTODEV_MASK_NUM},
+	{CMD_LINE_OPT_RX_OFFLOAD, 1, 0, CMD_LINE_OPT_RX_OFFLOAD_NUM},
+	{CMD_LINE_OPT_TX_OFFLOAD, 1, 0, CMD_LINE_OPT_TX_OFFLOAD_NUM},
 	{NULL, 0, 0, 0}
 };
 
@@ -155,6 +161,13 @@ static uint32_t single_sa;
 static uint32_t single_sa_idx;
 static uint32_t frame_size;
 
+/*
+ * RX/TX HW offload capabilities to enable/use on ethernet ports.
+ * By default all capabilities are enabled.
+ */
+static uint64_t dev_rx_offload = UINT64_MAX;
+static uint64_t dev_tx_offload = UINT64_MAX;
+
 struct lcore_rx_queue {
 	uint16_t port_id;
 	uint8_t queue_id;
@@ -208,8 +221,6 @@ static struct rte_eth_conf port_conf = {
 	},
 	.txmode = {
 		.mq_mode = ETH_MQ_TX_NONE,
-		.offloads = (DEV_TX_OFFLOAD_IPV4_CKSUM |
-			     DEV_TX_OFFLOAD_MULTI_SEGS),
 	},
 };
 
@@ -315,7 +326,8 @@ prepare_traffic(struct rte_mbuf **pkts, struct ipsec_traffic *t,
 }
 
 static inline void
-prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
+prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port,
+		const struct lcore_conf *qconf)
 {
 	struct ip *ip;
 	struct ether_hdr *ethhdr;
@@ -325,14 +337,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
 	ethhdr = (struct ether_hdr *)rte_pktmbuf_prepend(pkt, ETHER_HDR_LEN);
 
 	if (ip->ip_v == IPVERSION) {
-		pkt->ol_flags |= PKT_TX_IP_CKSUM | PKT_TX_IPV4;
+		pkt->ol_flags |= qconf->outbound.ipv4_offloads;
 		pkt->l3_len = sizeof(struct ip);
 		pkt->l2_len = ETHER_HDR_LEN;
 
 		ip->ip_sum = 0;
+
+		/* calculate IPv4 cksum in SW */
+		if ((pkt->ol_flags & PKT_TX_IP_CKSUM) == 0)
+			ip->ip_sum = rte_ipv4_cksum((struct ipv4_hdr *)ip);
+
 		ethhdr->ether_type = rte_cpu_to_be_16(ETHER_TYPE_IPv4);
 	} else {
-		pkt->ol_flags |= PKT_TX_IPV6;
+		pkt->ol_flags |= qconf->outbound.ipv6_offloads;
 		pkt->l3_len = sizeof(struct ip6_hdr);
 		pkt->l2_len = ETHER_HDR_LEN;
 
@@ -346,18 +363,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
 }
 
 static inline void
-prepare_tx_burst(struct rte_mbuf *pkts[], uint16_t nb_pkts, uint16_t port)
+prepare_tx_burst(struct rte_mbuf *pkts[], uint16_t nb_pkts, uint16_t port,
+		const struct lcore_conf *qconf)
 {
 	int32_t i;
 	const int32_t prefetch_offset = 2;
 
 	for (i = 0; i < (nb_pkts - prefetch_offset); i++) {
 		rte_mbuf_prefetch_part2(pkts[i + prefetch_offset]);
-		prepare_tx_pkt(pkts[i], port);
+		prepare_tx_pkt(pkts[i], port, qconf);
 	}
 	/* Process left packets */
 	for (; i < nb_pkts; i++)
-		prepare_tx_pkt(pkts[i], port);
+		prepare_tx_pkt(pkts[i], port, qconf);
 }
 
 /* Send burst of packets on an output interface */
@@ -371,7 +389,7 @@ send_burst(struct lcore_conf *qconf, uint16_t n, uint16_t port)
 	queueid = qconf->tx_queue_id[port];
 	m_table = (struct rte_mbuf **)qconf->tx_mbufs[port].m_table;
 
-	prepare_tx_burst(m_table, n, port);
+	prepare_tx_burst(m_table, n, port, qconf);
 
 	ret = rte_eth_tx_burst(port, queueid, m_table, n);
 	if (unlikely(ret < n)) {
@@ -954,6 +972,8 @@ print_usage(const char *prgname)
 		" --config (port,queue,lcore)[,(port,queue,lcore)]"
 		" [--single-sa SAIDX]"
 		" [--cryptodev_mask MASK]"
+		" [--" CMD_LINE_OPT_RX_OFFLOAD " RX_OFFLOAD_MASK]"
+		" [--" CMD_LINE_OPT_TX_OFFLOAD " TX_OFFLOAD_MASK]"
 		"\n\n"
 		"  -p PORTMASK: Hexadecimal bitmask of ports to configure\n"
 		"  -P : Enable promiscuous mode\n"
@@ -966,10 +986,31 @@ print_usage(const char *prgname)
 		"                     bypassing the SP\n"
 		"  --cryptodev_mask MASK: Hexadecimal bitmask of the crypto\n"
 		"                         devices to configure\n"
+		"  --" CMD_LINE_OPT_RX_OFFLOAD
+		": bitmask of the RX HW offload capabilities to enable/use\n"
+		"                         (DEV_RX_OFFLOAD_*)\n"
+		"  --" CMD_LINE_OPT_TX_OFFLOAD
+		": bitmask of the TX HW offload capabilities to enable/use\n"
+		"                         (DEV_TX_OFFLOAD_*)\n"
 		"\n",
 		prgname);
 }
 
+static int
+parse_mask(const char *str, uint64_t *val)
+{
+	char *end;
+	unsigned long t;
+
+	errno = 0;
+	t = strtoul(str, &end, 0);
+	if (errno != 0 || end[0] != 0)
+		return -EINVAL;
+
+	*val = t;
+	return 0;
+}
+
 static int32_t
 parse_portmask(const char *portmask)
 {
@@ -1156,6 +1197,24 @@ parse_args(int32_t argc, char **argv)
 			/* else */
 			enabled_cryptodev_mask = ret;
 			break;
+		case CMD_LINE_OPT_RX_OFFLOAD_NUM:
+			ret = parse_mask(optarg, &dev_rx_offload);
+			if (ret != 0) {
+				printf("Invalid argument for \'%s\': %s\n",
+					CMD_LINE_OPT_RX_OFFLOAD, optarg);
+				print_usage(prgname);
+				return -1;
+			}
+			break;
+		case CMD_LINE_OPT_TX_OFFLOAD_NUM:
+			ret = parse_mask(optarg, &dev_tx_offload);
+			if (ret != 0) {
+				printf("Invalid argument for \'%s\': %s\n",
+					CMD_LINE_OPT_TX_OFFLOAD, optarg);
+				print_usage(prgname);
+				return -1;
+			}
+			break;
 		default:
 			print_usage(prgname);
 			return -1;
@@ -1543,7 +1602,7 @@ cryptodevs_init(void)
 }
 
 static void
-port_init(uint16_t portid)
+port_init(uint16_t portid, uint64_t req_rx_offloads, uint64_t req_tx_offloads)
 {
 	struct rte_eth_dev_info dev_info;
 	struct rte_eth_txconf *txconf;
@@ -1556,6 +1615,10 @@ port_init(uint16_t portid)
 
 	rte_eth_dev_info_get(portid, &dev_info);
 
+	/* limit allowed HW offloafs, as user requested */
+	dev_info.rx_offload_capa &= dev_rx_offload;
+	dev_info.tx_offload_capa &= dev_tx_offload;
+
 	printf("Configuring device port %u:\n", portid);
 
 	rte_eth_macaddr_get(portid, &ethaddr);
@@ -1584,14 +1647,38 @@ port_init(uint16_t portid)
 		local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_JUMBO_FRAME;
 	}
 
-	if (dev_info.rx_offload_capa & DEV_RX_OFFLOAD_SECURITY)
-		local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_SECURITY;
-	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_SECURITY)
-		local_port_conf.txmode.offloads |= DEV_TX_OFFLOAD_SECURITY;
+	local_port_conf.rxmode.offloads |= req_rx_offloads;
+	local_port_conf.txmode.offloads |= req_tx_offloads;
+
+	/* Check that all required capabilities are supported */
+	if ((local_port_conf.rxmode.offloads & dev_info.rx_offload_capa) !=
+			local_port_conf.rxmode.offloads)
+		rte_exit(EXIT_FAILURE,
+			"Error: port %u required RX offloads: 0x%" PRIx64
+			", avaialbe RX offloads: 0x%" PRIx64 "\n",
+			portid, local_port_conf.rxmode.offloads,
+			dev_info.rx_offload_capa);
+
+	if ((local_port_conf.txmode.offloads & dev_info.tx_offload_capa) !=
+			local_port_conf.txmode.offloads)
+		rte_exit(EXIT_FAILURE,
+			"Error: port %u required TX offloads: 0x%" PRIx64
+			", avaialbe TX offloads: 0x%" PRIx64 "\n",
+			portid, local_port_conf.txmode.offloads,
+			dev_info.tx_offload_capa);
+
 	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_MBUF_FAST_FREE)
 		local_port_conf.txmode.offloads |=
 			DEV_TX_OFFLOAD_MBUF_FAST_FREE;
 
+	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_IPV4_CKSUM)
+		local_port_conf.txmode.offloads |= DEV_TX_OFFLOAD_IPV4_CKSUM;
+
+	printf("port %u configurng rx_offloads=0x%" PRIx64
+		", tx_offloads=0x%" PRIx64 "\n",
+		portid, local_port_conf.rxmode.offloads,
+		local_port_conf.txmode.offloads);
+
 	local_port_conf.rx_adv_conf.rss_conf.rss_hf &=
 		dev_info.flow_type_rss_offloads;
 	if (local_port_conf.rx_adv_conf.rss_conf.rss_hf !=
@@ -1639,6 +1726,13 @@ port_init(uint16_t portid)
 
 		qconf = &lcore_conf[lcore_id];
 		qconf->tx_queue_id[portid] = tx_queueid;
+
+		/* Pre-populate pkt offloads based on capabilities */
+		qconf->outbound.ipv4_offloads = PKT_TX_IPV4;
+		qconf->outbound.ipv6_offloads = PKT_TX_IPV6;
+		if (req_tx_offloads & DEV_TX_OFFLOAD_IPV4_CKSUM)
+			qconf->outbound.ipv4_offloads |= PKT_TX_IP_CKSUM;
+
 		tx_queueid++;
 
 		/* init RX queues */
@@ -1749,6 +1843,7 @@ main(int32_t argc, char **argv)
 	uint32_t lcore_id;
 	uint8_t socket_id;
 	uint16_t portid;
+	uint64_t req_rx_offloads, req_tx_offloads;
 
 	/* init EAL */
 	ret = rte_eal_init(argc, argv);
@@ -1804,7 +1899,8 @@ main(int32_t argc, char **argv)
 		if ((enabled_port_mask & (1 << portid)) == 0)
 			continue;
 
-		port_init(portid);
+		sa_check_offloads(portid, &req_rx_offloads, &req_tx_offloads);
+		port_init(portid, req_rx_offloads, req_tx_offloads);
 	}
 
 	cryptodevs_init();
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index c998c8076..9b1586f52 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -146,6 +146,8 @@ struct ipsec_ctx {
 	struct rte_mempool *session_pool;
 	struct rte_mbuf *ol_pkts[MAX_PKT_BURST] __rte_aligned(sizeof(void *));
 	uint16_t ol_pkts_cnt;
+	uint64_t ipv4_offloads;
+	uint64_t ipv6_offloads;
 };
 
 struct cdev_key {
@@ -239,4 +241,8 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id);
 void
 rt_init(struct socket_ctx *ctx, int32_t socket_id);
 
+int
+sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
+		uint64_t *tx_offloads);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index d2d3550a4..f6271bc60 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -1017,3 +1017,38 @@ outbound_sa_lookup(struct sa_ctx *sa_ctx, uint32_t sa_idx[],
 	for (i = 0; i < nb_pkts; i++)
 		sa[i] = &sa_ctx->sa[sa_idx[i]];
 }
+
+/*
+ * Select HW offloads to be used.
+ */
+int
+sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
+		uint64_t *tx_offloads)
+{
+	struct ipsec_sa *rule;
+	uint32_t idx_sa;
+
+	*rx_offloads = 0;
+	*tx_offloads = 0;
+
+	/* Check for inbound rules that use offloads and use this port */
+	for (idx_sa = 0; idx_sa < nb_sa_in; idx_sa++) {
+		rule = &sa_in[idx_sa];
+		if ((rule->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				rule->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL)
+				&& rule->portid == port_id)
+			*rx_offloads |= DEV_RX_OFFLOAD_SECURITY;
+	}
+
+	/* Check for outbound rules that use offloads and use this port */
+	for (idx_sa = 0; idx_sa < nb_sa_out; idx_sa++) {
+		rule = &sa_out[idx_sa];
+		if ((rule->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				rule->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL)
+				&& rule->portid == port_id)
+			*tx_offloads |= DEV_TX_OFFLOAD_SECURITY;
+	}
+	return 0;
+}
-- 
2.17.1

Re: [PATCH v5 01/10] examples/ipsec-secgw: allow user to disable some RX/TX offloads

[Akhil Goyal]

On 12/28/2018 9:03 PM, Konstantin Ananyev wrote:
> Right now ipsec-secgw always enables TX offloads
> (DEV_TX_OFFLOAD_MULTI_SEGS, DEV_TX_OFFLOAD_SECURITY),
> even when they are not requested by the config.
> That causes many PMD to choose full-featured TX function,
> which in many cases is much slower then one without offloads.
> That patch adds ability for the user to disable unneeded HW offloads.
> If DEV_TX_OFFLOAD_IPV4_CKSUM is disabled by user, then
> SW version of ip cksum calculation is used.
> That allows to use vector TX function, when inline-ipsec is not
> requested.
>
> Signed-off-by: Remy Horton <remy.horton@intel.com>
> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> Acked-by: Radu Nicolau <radu.nicolau@intel.com>
> ---
>   doc/guides/sample_app_ug/ipsec_secgw.rst |  12 +++
>   examples/ipsec-secgw/ipsec-secgw.c       | 126 ++++++++++++++++++++---
>   examples/ipsec-secgw/ipsec.h             |   6 ++
>   examples/ipsec-secgw/sa.c                |  35 +++++++
>   4 files changed, 164 insertions(+), 15 deletions(-)
>
> diff --git a/doc/guides/sample_app_ug/ipsec_secgw.rst b/doc/guides/sample_app_ug/ipsec_secgw.rst
> index 4869a011d..244bde2de 100644
> --- a/doc/guides/sample_app_ug/ipsec_secgw.rst
> +++ b/doc/guides/sample_app_ug/ipsec_secgw.rst
> @@ -95,6 +95,8 @@ The application has a number of command line options::
>                           -p PORTMASK -P -u PORTMASK -j FRAMESIZE
>                           --config (port,queue,lcore)[,(port,queue,lcore]
>                           --single-sa SAIDX
> +                        --rxoffload MASK
> +                        --txoffload MASK
>                           -f CONFIG_FILE_PATH
>   
>   Where:
> @@ -119,6 +121,16 @@ Where:
>       on both Inbound and Outbound. This option is meant for debugging/performance
>       purposes.
>   
> +*   ``--rxoffload MASK``: RX HW offload capabilities to enable/use on this port
> +    (bitmask of DEV_RX_OFFLOAD_* values). It is an optional parameter and
> +    allows user to disable some of the RX HW offload capabilities.
> +    By default all HW RX offloads are enabled.
> +
> +*   ``--txoffload MASK``: TX HW offload capabilities to enable/use on this port
> +    (bitmask of DEV_TX_OFFLOAD_* values). It is an optional parameter and
> +    allows user to disable some of the TX HW offload capabilities.
> +    By default all HW TX offloads are enabled.
> +
by default all should not be enabled. It should remain as it was before 
your patchset.
It would be good if you can add a link to ethdev section where it 
explain different offloads available.

If anyone needs to change the default behavior, a patch need to be 
submitted to add that.
>   *   ``-f CONFIG_FILE_PATH``: the full path of text-based file containing all
>       configuration items for running the application (See Configuration file
>       syntax section below). ``-f CONFIG_FILE_PATH`` **must** be specified.
> diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
> index 1bc0b5b50..f5db3da0c 100644
> --- a/examples/ipsec-secgw/ipsec-secgw.c
> +++ b/examples/ipsec-secgw/ipsec-secgw.c
> @@ -124,6 +124,8 @@ struct ethaddr_info ethaddr_tbl[RTE_MAX_ETHPORTS] = {
>   #define CMD_LINE_OPT_CONFIG		"config"
>   #define CMD_LINE_OPT_SINGLE_SA		"single-sa"
>   #define CMD_LINE_OPT_CRYPTODEV_MASK	"cryptodev_mask"
> +#define CMD_LINE_OPT_RX_OFFLOAD		"rxoffload"
> +#define CMD_LINE_OPT_TX_OFFLOAD		"txoffload"
>   
>   enum {
>   	/* long options mapped to a short option */
> @@ -135,12 +137,16 @@ enum {
>   	CMD_LINE_OPT_CONFIG_NUM,
>   	CMD_LINE_OPT_SINGLE_SA_NUM,
>   	CMD_LINE_OPT_CRYPTODEV_MASK_NUM,
> +	CMD_LINE_OPT_RX_OFFLOAD_NUM,
> +	CMD_LINE_OPT_TX_OFFLOAD_NUM,
>   };
>   
>   static const struct option lgopts[] = {
>   	{CMD_LINE_OPT_CONFIG, 1, 0, CMD_LINE_OPT_CONFIG_NUM},
>   	{CMD_LINE_OPT_SINGLE_SA, 1, 0, CMD_LINE_OPT_SINGLE_SA_NUM},
>   	{CMD_LINE_OPT_CRYPTODEV_MASK, 1, 0, CMD_LINE_OPT_CRYPTODEV_MASK_NUM},
> +	{CMD_LINE_OPT_RX_OFFLOAD, 1, 0, CMD_LINE_OPT_RX_OFFLOAD_NUM},
> +	{CMD_LINE_OPT_TX_OFFLOAD, 1, 0, CMD_LINE_OPT_TX_OFFLOAD_NUM},
>   	{NULL, 0, 0, 0}
>   };
>   
> @@ -155,6 +161,13 @@ static uint32_t single_sa;
>   static uint32_t single_sa_idx;
>   static uint32_t frame_size;
>   
> +/*
> + * RX/TX HW offload capabilities to enable/use on ethernet ports.
> + * By default all capabilities are enabled.
> + */
> +static uint64_t dev_rx_offload = UINT64_MAX;
> +static uint64_t dev_tx_offload = UINT64_MAX;
> +
>   struct lcore_rx_queue {
>   	uint16_t port_id;
>   	uint8_t queue_id;
> @@ -208,8 +221,6 @@ static struct rte_eth_conf port_conf = {
>   	},
>   	.txmode = {
>   		.mq_mode = ETH_MQ_TX_NONE,
> -		.offloads = (DEV_TX_OFFLOAD_IPV4_CKSUM |
> -			     DEV_TX_OFFLOAD_MULTI_SEGS),
>   	},
>   };
>   
> @@ -315,7 +326,8 @@ prepare_traffic(struct rte_mbuf **pkts, struct ipsec_traffic *t,
>   }
>   
>   static inline void
> -prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
> +prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port,
> +		const struct lcore_conf *qconf)
>   {
>   	struct ip *ip;
>   	struct ether_hdr *ethhdr;
> @@ -325,14 +337,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
>   	ethhdr = (struct ether_hdr *)rte_pktmbuf_prepend(pkt, ETHER_HDR_LEN);
>   
>   	if (ip->ip_v == IPVERSION) {
> -		pkt->ol_flags |= PKT_TX_IP_CKSUM | PKT_TX_IPV4;
> +		pkt->ol_flags |= qconf->outbound.ipv4_offloads;
>   		pkt->l3_len = sizeof(struct ip);
>   		pkt->l2_len = ETHER_HDR_LEN;
>   
>   		ip->ip_sum = 0;
> +
> +		/* calculate IPv4 cksum in SW */
> +		if ((pkt->ol_flags & PKT_TX_IP_CKSUM) == 0)
> +			ip->ip_sum = rte_ipv4_cksum((struct ipv4_hdr *)ip);
> +
something is not correct, on my hardware I still see rte_ipv4_cksum 
getting called.
This result in around 2% drop due to this patch.
>   		ethhdr->ether_type = rte_cpu_to_be_16(ETHER_TYPE_IPv4);
>   	} else {
> -		pkt->ol_flags |= PKT_TX_IPV6;
> +		pkt->ol_flags |= qconf->outbound.ipv6_offloads;
>   		pkt->l3_len = sizeof(struct ip6_hdr);
>   		pkt->l2_len = ETHER_HDR_LEN;
>   
> @@ -346,18 +363,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
>   }
>   
>   static inline void
> -prepare_tx_burst(struct rte_mbuf *pkts[], uint16_t nb_pkts, uint16_t port)
> +prepare_tx_burst(struct rte_mbuf *pkts[], uint16_t nb_pkts, uint16_t port,
> +		const struct lcore_conf *qconf)
>   {
>   	int32_t i;
>   	const int32_t prefetch_offset = 2;
>   
>   	for (i = 0; i < (nb_pkts - prefetch_offset); i++) {
>   		rte_mbuf_prefetch_part2(pkts[i + prefetch_offset]);
> -		prepare_tx_pkt(pkts[i], port);
> +		prepare_tx_pkt(pkts[i], port, qconf);
>   	}
>   	/* Process left packets */
>   	for (; i < nb_pkts; i++)
> -		prepare_tx_pkt(pkts[i], port);
> +		prepare_tx_pkt(pkts[i], port, qconf);
>   }
>   
>   /* Send burst of packets on an output interface */
> @@ -371,7 +389,7 @@ send_burst(struct lcore_conf *qconf, uint16_t n, uint16_t port)
>   	queueid = qconf->tx_queue_id[port];
>   	m_table = (struct rte_mbuf **)qconf->tx_mbufs[port].m_table;
>   
> -	prepare_tx_burst(m_table, n, port);
> +	prepare_tx_burst(m_table, n, port, qconf);
>   
>   	ret = rte_eth_tx_burst(port, queueid, m_table, n);
>   	if (unlikely(ret < n)) {
> @@ -954,6 +972,8 @@ print_usage(const char *prgname)
>   		" --config (port,queue,lcore)[,(port,queue,lcore)]"
>   		" [--single-sa SAIDX]"
>   		" [--cryptodev_mask MASK]"
> +		" [--" CMD_LINE_OPT_RX_OFFLOAD " RX_OFFLOAD_MASK]"
> +		" [--" CMD_LINE_OPT_TX_OFFLOAD " TX_OFFLOAD_MASK]"
>   		"\n\n"
>   		"  -p PORTMASK: Hexadecimal bitmask of ports to configure\n"
>   		"  -P : Enable promiscuous mode\n"
> @@ -966,10 +986,31 @@ print_usage(const char *prgname)
>   		"                     bypassing the SP\n"
>   		"  --cryptodev_mask MASK: Hexadecimal bitmask of the crypto\n"
>   		"                         devices to configure\n"
> +		"  --" CMD_LINE_OPT_RX_OFFLOAD
> +		": bitmask of the RX HW offload capabilities to enable/use\n"
> +		"                         (DEV_RX_OFFLOAD_*)\n"
> +		"  --" CMD_LINE_OPT_TX_OFFLOAD
> +		": bitmask of the TX HW offload capabilities to enable/use\n"
> +		"                         (DEV_TX_OFFLOAD_*)\n"
>   		"\n",
>   		prgname);
>   }
>   
> +static int
> +parse_mask(const char *str, uint64_t *val)
> +{
> +	char *end;
> +	unsigned long t;
> +
> +	errno = 0;
> +	t = strtoul(str, &end, 0);
> +	if (errno != 0 || end[0] != 0)
> +		return -EINVAL;
> +
> +	*val = t;
> +	return 0;
> +}
> +
>   static int32_t
>   parse_portmask(const char *portmask)
>   {
> @@ -1156,6 +1197,24 @@ parse_args(int32_t argc, char **argv)
>   			/* else */
>   			enabled_cryptodev_mask = ret;
>   			break;
> +		case CMD_LINE_OPT_RX_OFFLOAD_NUM:
> +			ret = parse_mask(optarg, &dev_rx_offload);
> +			if (ret != 0) {
> +				printf("Invalid argument for \'%s\': %s\n",
> +					CMD_LINE_OPT_RX_OFFLOAD, optarg);
> +				print_usage(prgname);
> +				return -1;
> +			}
> +			break;
> +		case CMD_LINE_OPT_TX_OFFLOAD_NUM:
> +			ret = parse_mask(optarg, &dev_tx_offload);
> +			if (ret != 0) {
> +				printf("Invalid argument for \'%s\': %s\n",
> +					CMD_LINE_OPT_TX_OFFLOAD, optarg);
> +				print_usage(prgname);
> +				return -1;
> +			}
> +			break;
>   		default:
>   			print_usage(prgname);
>   			return -1;
> @@ -1543,7 +1602,7 @@ cryptodevs_init(void)
>   }
>   
>   static void
> -port_init(uint16_t portid)
> +port_init(uint16_t portid, uint64_t req_rx_offloads, uint64_t req_tx_offloads)
>   {
>   	struct rte_eth_dev_info dev_info;
>   	struct rte_eth_txconf *txconf;
> @@ -1556,6 +1615,10 @@ port_init(uint16_t portid)
>   
>   	rte_eth_dev_info_get(portid, &dev_info);
>   
> +	/* limit allowed HW offloafs, as user requested */
> +	dev_info.rx_offload_capa &= dev_rx_offload;
> +	dev_info.tx_offload_capa &= dev_tx_offload;
> +
>   	printf("Configuring device port %u:\n", portid);
>   
>   	rte_eth_macaddr_get(portid, &ethaddr);
> @@ -1584,14 +1647,38 @@ port_init(uint16_t portid)
>   		local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_JUMBO_FRAME;
>   	}
>   
> -	if (dev_info.rx_offload_capa & DEV_RX_OFFLOAD_SECURITY)
> -		local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_SECURITY;
> -	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_SECURITY)
> -		local_port_conf.txmode.offloads |= DEV_TX_OFFLOAD_SECURITY;
> +	local_port_conf.rxmode.offloads |= req_rx_offloads;
> +	local_port_conf.txmode.offloads |= req_tx_offloads;
> +
> +	/* Check that all required capabilities are supported */
> +	if ((local_port_conf.rxmode.offloads & dev_info.rx_offload_capa) !=
> +			local_port_conf.rxmode.offloads)
> +		rte_exit(EXIT_FAILURE,
> +			"Error: port %u required RX offloads: 0x%" PRIx64
> +			", avaialbe RX offloads: 0x%" PRIx64 "\n",
> +			portid, local_port_conf.rxmode.offloads,
> +			dev_info.rx_offload_capa);
> +
> +	if ((local_port_conf.txmode.offloads & dev_info.tx_offload_capa) !=
> +			local_port_conf.txmode.offloads)
> +		rte_exit(EXIT_FAILURE,
> +			"Error: port %u required TX offloads: 0x%" PRIx64
> +			", avaialbe TX offloads: 0x%" PRIx64 "\n",
> +			portid, local_port_conf.txmode.offloads,
> +			dev_info.tx_offload_capa);
> +
>   	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_MBUF_FAST_FREE)
>   		local_port_conf.txmode.offloads |=
>   			DEV_TX_OFFLOAD_MBUF_FAST_FREE;
>   
> +	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_IPV4_CKSUM)
> +		local_port_conf.txmode.offloads |= DEV_TX_OFFLOAD_IPV4_CKSUM;
> +
> +	printf("port %u configurng rx_offloads=0x%" PRIx64
> +		", tx_offloads=0x%" PRIx64 "\n",
> +		portid, local_port_conf.rxmode.offloads,
> +		local_port_conf.txmode.offloads);
> +
>   	local_port_conf.rx_adv_conf.rss_conf.rss_hf &=
>   		dev_info.flow_type_rss_offloads;
>   	if (local_port_conf.rx_adv_conf.rss_conf.rss_hf !=
> @@ -1639,6 +1726,13 @@ port_init(uint16_t portid)
>   
>   		qconf = &lcore_conf[lcore_id];
>   		qconf->tx_queue_id[portid] = tx_queueid;
> +
> +		/* Pre-populate pkt offloads based on capabilities */
> +		qconf->outbound.ipv4_offloads = PKT_TX_IPV4;
> +		qconf->outbound.ipv6_offloads = PKT_TX_IPV6;
> +		if (req_tx_offloads & DEV_TX_OFFLOAD_IPV4_CKSUM)
> +			qconf->outbound.ipv4_offloads |= PKT_TX_IP_CKSUM;
I believe this is the issue, this should not be req_tx_offload. 
req_tx_offloads cannot have DEV_TX_OFFLOAD_IPV4_CKSUM set for lookaside 
mode.
> +
>   		tx_queueid++;
>   
>   		/* init RX queues */
> @@ -1749,6 +1843,7 @@ main(int32_t argc, char **argv)
>   	uint32_t lcore_id;
>   	uint8_t socket_id;
>   	uint16_t portid;
> +	uint64_t req_rx_offloads, req_tx_offloads;
>   
>   	/* init EAL */
>   	ret = rte_eal_init(argc, argv);
> @@ -1804,7 +1899,8 @@ main(int32_t argc, char **argv)
>   		if ((enabled_port_mask & (1 << portid)) == 0)
>   			continue;
>   
> -		port_init(portid);
> +		sa_check_offloads(portid, &req_rx_offloads, &req_tx_offloads);
> +		port_init(portid, req_rx_offloads, req_tx_offloads);
>   	}
>   
>   	cryptodevs_init();
> diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
> index c998c8076..9b1586f52 100644
> --- a/examples/ipsec-secgw/ipsec.h
> +++ b/examples/ipsec-secgw/ipsec.h
> @@ -146,6 +146,8 @@ struct ipsec_ctx {
>   	struct rte_mempool *session_pool;
>   	struct rte_mbuf *ol_pkts[MAX_PKT_BURST] __rte_aligned(sizeof(void *));
>   	uint16_t ol_pkts_cnt;
> +	uint64_t ipv4_offloads;
> +	uint64_t ipv6_offloads;
>   };
>   
>   struct cdev_key {
> @@ -239,4 +241,8 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id);
>   void
>   rt_init(struct socket_ctx *ctx, int32_t socket_id);
>   
> +int
> +sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
> +		uint64_t *tx_offloads);
> +
>   #endif /* __IPSEC_H__ */
> diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
> index d2d3550a4..f6271bc60 100644
> --- a/examples/ipsec-secgw/sa.c
> +++ b/examples/ipsec-secgw/sa.c
> @@ -1017,3 +1017,38 @@ outbound_sa_lookup(struct sa_ctx *sa_ctx, uint32_t sa_idx[],
>   	for (i = 0; i < nb_pkts; i++)
>   		sa[i] = &sa_ctx->sa[sa_idx[i]];
>   }
> +
> +/*
> + * Select HW offloads to be used.
> + */
> +int
> +sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
> +		uint64_t *tx_offloads)
> +{
> +	struct ipsec_sa *rule;
> +	uint32_t idx_sa;
> +
> +	*rx_offloads = 0;
> +	*tx_offloads = 0;
> +
> +	/* Check for inbound rules that use offloads and use this port */
> +	for (idx_sa = 0; idx_sa < nb_sa_in; idx_sa++) {
> +		rule = &sa_in[idx_sa];
> +		if ((rule->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
> +				rule->type ==
> +				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL)
> +				&& rule->portid == port_id)
> +			*rx_offloads |= DEV_RX_OFFLOAD_SECURITY;
> +	}
> +
> +	/* Check for outbound rules that use offloads and use this port */
> +	for (idx_sa = 0; idx_sa < nb_sa_out; idx_sa++) {
> +		rule = &sa_out[idx_sa];
> +		if ((rule->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
> +				rule->type ==
> +				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL)
> +				&& rule->portid == port_id)
> +			*tx_offloads |= DEV_TX_OFFLOAD_SECURITY;
> +	}
> +	return 0;
> +}

Re: [PATCH v5 01/10] examples/ipsec-secgw: allow user to disable some RX/TX offloads

[Ananyev, Konstantin]

> -----Original Message-----
> From: Akhil Goyal [mailto:akhil.goyal@nxp.com]
> Sent: Wednesday, January 2, 2019 1:43 PM
> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
> Cc: Horton, Remy <remy.horton@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v5 01/10] examples/ipsec-secgw: allow user to disable some RX/TX offloads
> 
> 
> 
> On 12/28/2018 9:03 PM, Konstantin Ananyev wrote:
> > Right now ipsec-secgw always enables TX offloads
> > (DEV_TX_OFFLOAD_MULTI_SEGS, DEV_TX_OFFLOAD_SECURITY),
> > even when they are not requested by the config.
> > That causes many PMD to choose full-featured TX function,
> > which in many cases is much slower then one without offloads.
> > That patch adds ability for the user to disable unneeded HW offloads.
> > If DEV_TX_OFFLOAD_IPV4_CKSUM is disabled by user, then
> > SW version of ip cksum calculation is used.
> > That allows to use vector TX function, when inline-ipsec is not
> > requested.
> >
> > Signed-off-by: Remy Horton <remy.horton@intel.com>
> > Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> > Acked-by: Radu Nicolau <radu.nicolau@intel.com>
> > ---
> >   doc/guides/sample_app_ug/ipsec_secgw.rst |  12 +++
> >   examples/ipsec-secgw/ipsec-secgw.c       | 126 ++++++++++++++++++++---
> >   examples/ipsec-secgw/ipsec.h             |   6 ++
> >   examples/ipsec-secgw/sa.c                |  35 +++++++
> >   4 files changed, 164 insertions(+), 15 deletions(-)
> >
> > diff --git a/doc/guides/sample_app_ug/ipsec_secgw.rst b/doc/guides/sample_app_ug/ipsec_secgw.rst
> > index 4869a011d..244bde2de 100644
> > --- a/doc/guides/sample_app_ug/ipsec_secgw.rst
> > +++ b/doc/guides/sample_app_ug/ipsec_secgw.rst
> > @@ -95,6 +95,8 @@ The application has a number of command line options::
> >                           -p PORTMASK -P -u PORTMASK -j FRAMESIZE
> >                           --config (port,queue,lcore)[,(port,queue,lcore]
> >                           --single-sa SAIDX
> > +                        --rxoffload MASK
> > +                        --txoffload MASK
> >                           -f CONFIG_FILE_PATH
> >
> >   Where:
> > @@ -119,6 +121,16 @@ Where:
> >       on both Inbound and Outbound. This option is meant for debugging/performance
> >       purposes.
> >
> > +*   ``--rxoffload MASK``: RX HW offload capabilities to enable/use on this port
> > +    (bitmask of DEV_RX_OFFLOAD_* values). It is an optional parameter and
> > +    allows user to disable some of the RX HW offload capabilities.
> > +    By default all HW RX offloads are enabled.
> > +
> > +*   ``--txoffload MASK``: TX HW offload capabilities to enable/use on this port
> > +    (bitmask of DEV_TX_OFFLOAD_* values). It is an optional parameter and
> > +    allows user to disable some of the TX HW offload capabilities.
> > +    By default all HW TX offloads are enabled.
> > +
> by default all should not be enabled. It should remain as it was before
> your patchset.

It remains as it was before the patchset.
By default we initialize dev_rx_offload/dev_tx_offloads to (uint64_t)-1;
Then: 
dev_info.rx_offload_capa &= dev_rx_offload;
to disable unwanted capabilities.
Then:
if ((local_port_conf.rxmode.offloads & dev_info.rx_offload_capa)
to check that all required capabilities are supported.

> It would be good if you can add a link to ethdev section where it
> explain different offloads available.
> 
> If anyone needs to change the default behavior, a patch need to be
> submitted to add that.
> >   *   ``-f CONFIG_FILE_PATH``: the full path of text-based file containing all
> >       configuration items for running the application (See Configuration file
> >       syntax section below). ``-f CONFIG_FILE_PATH`` **must** be specified.
> > diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
> > index 1bc0b5b50..f5db3da0c 100644
> > --- a/examples/ipsec-secgw/ipsec-secgw.c
> > +++ b/examples/ipsec-secgw/ipsec-secgw.c
> > @@ -124,6 +124,8 @@ struct ethaddr_info ethaddr_tbl[RTE_MAX_ETHPORTS] = {
> >   #define CMD_LINE_OPT_CONFIG		"config"
> >   #define CMD_LINE_OPT_SINGLE_SA		"single-sa"
> >   #define CMD_LINE_OPT_CRYPTODEV_MASK	"cryptodev_mask"
> > +#define CMD_LINE_OPT_RX_OFFLOAD		"rxoffload"
> > +#define CMD_LINE_OPT_TX_OFFLOAD		"txoffload"
> >
> >   enum {
> >   	/* long options mapped to a short option */
> > @@ -135,12 +137,16 @@ enum {
> >   	CMD_LINE_OPT_CONFIG_NUM,
> >   	CMD_LINE_OPT_SINGLE_SA_NUM,
> >   	CMD_LINE_OPT_CRYPTODEV_MASK_NUM,
> > +	CMD_LINE_OPT_RX_OFFLOAD_NUM,
> > +	CMD_LINE_OPT_TX_OFFLOAD_NUM,
> >   };
> >
> >   static const struct option lgopts[] = {
> >   	{CMD_LINE_OPT_CONFIG, 1, 0, CMD_LINE_OPT_CONFIG_NUM},
> >   	{CMD_LINE_OPT_SINGLE_SA, 1, 0, CMD_LINE_OPT_SINGLE_SA_NUM},
> >   	{CMD_LINE_OPT_CRYPTODEV_MASK, 1, 0, CMD_LINE_OPT_CRYPTODEV_MASK_NUM},
> > +	{CMD_LINE_OPT_RX_OFFLOAD, 1, 0, CMD_LINE_OPT_RX_OFFLOAD_NUM},
> > +	{CMD_LINE_OPT_TX_OFFLOAD, 1, 0, CMD_LINE_OPT_TX_OFFLOAD_NUM},
> >   	{NULL, 0, 0, 0}
> >   };
> >
> > @@ -155,6 +161,13 @@ static uint32_t single_sa;
> >   static uint32_t single_sa_idx;
> >   static uint32_t frame_size;
> >
> > +/*
> > + * RX/TX HW offload capabilities to enable/use on ethernet ports.
> > + * By default all capabilities are enabled.
> > + */
> > +static uint64_t dev_rx_offload = UINT64_MAX;
> > +static uint64_t dev_tx_offload = UINT64_MAX;
> > +
> >   struct lcore_rx_queue {
> >   	uint16_t port_id;
> >   	uint8_t queue_id;
> > @@ -208,8 +221,6 @@ static struct rte_eth_conf port_conf = {
> >   	},
> >   	.txmode = {
> >   		.mq_mode = ETH_MQ_TX_NONE,
> > -		.offloads = (DEV_TX_OFFLOAD_IPV4_CKSUM |
> > -			     DEV_TX_OFFLOAD_MULTI_SEGS),
> >   	},
> >   };
> >
> > @@ -315,7 +326,8 @@ prepare_traffic(struct rte_mbuf **pkts, struct ipsec_traffic *t,
> >   }
> >
> >   static inline void
> > -prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
> > +prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port,
> > +		const struct lcore_conf *qconf)
> >   {
> >   	struct ip *ip;
> >   	struct ether_hdr *ethhdr;
> > @@ -325,14 +337,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
> >   	ethhdr = (struct ether_hdr *)rte_pktmbuf_prepend(pkt, ETHER_HDR_LEN);
> >
> >   	if (ip->ip_v == IPVERSION) {
> > -		pkt->ol_flags |= PKT_TX_IP_CKSUM | PKT_TX_IPV4;
> > +		pkt->ol_flags |= qconf->outbound.ipv4_offloads;
> >   		pkt->l3_len = sizeof(struct ip);
> >   		pkt->l2_len = ETHER_HDR_LEN;
> >
> >   		ip->ip_sum = 0;
> > +
> > +		/* calculate IPv4 cksum in SW */
> > +		if ((pkt->ol_flags & PKT_TX_IP_CKSUM) == 0)
> > +			ip->ip_sum = rte_ipv4_cksum((struct ipv4_hdr *)ip);
> > +
> something is not correct, on my hardware I still see rte_ipv4_cksum
> getting called.

My bad - screwed part of Remy's changes while introducing offloads options.
Will fix it in v6.
Konstantin

[PATCH v6 00/10] examples/ipsec-secgw: make app to use ipsec library

[Konstantin Ananyev]

This patch series depends on the patch series:

ipsec: new library for IPsec data-path processing
http://patches.dpdk.org/patch/49410/
http://patches.dpdk.org/patch/49411/
http://patches.dpdk.org/patch/49412/
http://patches.dpdk.org/patch/49413/
http://patches.dpdk.org/patch/49414/
http://patches.dpdk.org/patch/49415/
http://patches.dpdk.org/patch/49416/
http://patches.dpdk.org/patch/49417/
http://patches.dpdk.org/patch/49418/
http://patches.dpdk.org/patch/49419/

to be applied first.

v5 -> v6
  Address issues reported by Akhil:
     segfault when using lookaside-proto device
     HW IPv4 cksum offload not enabled by default
     crypto-dev dequeue() is called to often

v4 -> v5
- Address Akhil comments:
     documentation update
     spell checks spacing etc.
     introduce rxoffload/txoffload parameters
     single SA for ipv6
     update Makefile

v3 -> v4
 - fix few issues with the test scripts
 - update docs

v2 -> v3
 - add IPv6 cases into test scripts
 - fixes for IPv6 support
 - fixes for inline-crypto support
 - some code restructuring

v1 -> v2
 - Several bug fixes

That series contians few bug-fixes and changes to make ipsec-secgw
to utilize librte_ipsec library:
     - changes in the related data structures.
     - changes in the initialization code.
     - changes in the data-path code.
     - new command-line parameters to enable librte_ipsec codepath
       and related features.
     - test scripts to help automate ipsec-secgw functional testing.

Note that right now by default current (non-librte_ipsec) code-path
will be used. User has to run application with new command-line option
('-l')
to enable new codepath.
The main reason for that:
  - current librte_ipsec doesn't support all ipsec algorithms
    and features that the app does.
  - allow users to run both versions in parallel for some time
    to figure out any functional or performance degradation with the
    new code.

Test scripts were run with the following crypto devices:
 - aesni_mb
 - aesni_gcm
 - qat

Konstantin Ananyev (10):
  examples/ipsec-secgw: allow user to disable some RX/TX offloads
  examples/ipsec-secgw: allow to specify neighbour mac address
  examples/ipsec-secgw: fix crypto-op might never get dequeued
  examples/ipsec-secgw: fix outbound codepath for single SA
  examples/ipsec-secgw: make local variables static
  examples/ipsec-secgw: fix inbound SA checking
  examples/ipsec-secgw: make app to use ipsec library
  examples/ipsec-secgw: make data-path to use ipsec library
  examples/ipsec-secgw: add scripts for functional test
  doc: update ipsec-secgw guide and relelase notes

 doc/guides/rel_notes/release_19_02.rst        |  14 +
 doc/guides/sample_app_ug/ipsec_secgw.rst      | 159 +++++-
 examples/ipsec-secgw/Makefile                 |   5 +-
 examples/ipsec-secgw/ipsec-secgw.c            | 480 ++++++++++++++----
 examples/ipsec-secgw/ipsec.c                  | 101 ++--
 examples/ipsec-secgw/ipsec.h                  |  67 +++
 examples/ipsec-secgw/ipsec_process.c          | 357 +++++++++++++
 examples/ipsec-secgw/meson.build              |   6 +-
 examples/ipsec-secgw/parser.c                 |  91 ++++
 examples/ipsec-secgw/parser.h                 |   8 +-
 examples/ipsec-secgw/sa.c                     | 263 +++++++++-
 examples/ipsec-secgw/sp4.c                    |  35 +-
 examples/ipsec-secgw/sp6.c                    |  35 +-
 examples/ipsec-secgw/test/common_defs.sh      | 153 ++++++
 examples/ipsec-secgw/test/data_rxtx.sh        |  62 +++
 examples/ipsec-secgw/test/linux_test4.sh      |  63 +++
 examples/ipsec-secgw/test/linux_test6.sh      |  64 +++
 examples/ipsec-secgw/test/run_test.sh         |  80 +++
 .../test/trs_aescbc_sha1_common_defs.sh       |  69 +++
 .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  67 +++
 .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/trs_aescbc_sha1_esn_defs.sh          |  66 +++
 .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
 .../test/trs_aesgcm_common_defs.sh            |  60 +++
 examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  66 +++
 .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  66 +++
 .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
 .../test/tun_aescbc_sha1_common_defs.sh       |  68 +++
 .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  70 +++
 .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/tun_aescbc_sha1_esn_defs.sh          |  70 +++
 .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
 .../test/tun_aesgcm_common_defs.sh            |  60 +++
 examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  70 +++
 .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  70 +++
 .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
 38 files changed, 2727 insertions(+), 158 deletions(-)
 create mode 100644 examples/ipsec-secgw/ipsec_process.c
 create mode 100644 examples/ipsec-secgw/test/common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test4.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test6.sh
 create mode 100644 examples/ipsec-secgw/test/run_test.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh

-- 
2.17.1

Re: [PATCH v6 00/10] examples/ipsec-secgw: make app to use ipsec library

[Akhil Goyal]

Hi Konstantin,

With this patchset, I am seeing a 3% drop in legacy mode lookaside and 
12% drop with -l option.
I am debugging this. Will let you know if I find something.

-Akhil

On 1/4/2019 1:55 AM, Konstantin Ananyev wrote:
> This patch series depends on the patch series:
>
> ipsec: new library for IPsec data-path processing
> http://patches.dpdk.org/patch/49410/
> http://patches.dpdk.org/patch/49411/
> http://patches.dpdk.org/patch/49412/
> http://patches.dpdk.org/patch/49413/
> http://patches.dpdk.org/patch/49414/
> http://patches.dpdk.org/patch/49415/
> http://patches.dpdk.org/patch/49416/
> http://patches.dpdk.org/patch/49417/
> http://patches.dpdk.org/patch/49418/
> http://patches.dpdk.org/patch/49419/
>
> to be applied first.
>
> v5 -> v6
>    Address issues reported by Akhil:
>       segfault when using lookaside-proto device
>       HW IPv4 cksum offload not enabled by default
>       crypto-dev dequeue() is called to often
>
> v4 -> v5
> - Address Akhil comments:
>       documentation update
>       spell checks spacing etc.
>       introduce rxoffload/txoffload parameters
>       single SA for ipv6
>       update Makefile
>
> v3 -> v4
>   - fix few issues with the test scripts
>   - update docs
>
> v2 -> v3
>   - add IPv6 cases into test scripts
>   - fixes for IPv6 support
>   - fixes for inline-crypto support
>   - some code restructuring
>
> v1 -> v2
>   - Several bug fixes
>
> That series contians few bug-fixes and changes to make ipsec-secgw
> to utilize librte_ipsec library:
>       - changes in the related data structures.
>       - changes in the initialization code.
>       - changes in the data-path code.
>       - new command-line parameters to enable librte_ipsec codepath
>         and related features.
>       - test scripts to help automate ipsec-secgw functional testing.
>
> Note that right now by default current (non-librte_ipsec) code-path
> will be used. User has to run application with new command-line option
> ('-l')
> to enable new codepath.
> The main reason for that:
>    - current librte_ipsec doesn't support all ipsec algorithms
>      and features that the app does.
>    - allow users to run both versions in parallel for some time
>      to figure out any functional or performance degradation with the
>      new code.
>
> Test scripts were run with the following crypto devices:
>   - aesni_mb
>   - aesni_gcm
>   - qat
>
> Konstantin Ananyev (10):
>    examples/ipsec-secgw: allow user to disable some RX/TX offloads
>    examples/ipsec-secgw: allow to specify neighbour mac address
>    examples/ipsec-secgw: fix crypto-op might never get dequeued
>    examples/ipsec-secgw: fix outbound codepath for single SA
>    examples/ipsec-secgw: make local variables static
>    examples/ipsec-secgw: fix inbound SA checking
>    examples/ipsec-secgw: make app to use ipsec library
>    examples/ipsec-secgw: make data-path to use ipsec library
>    examples/ipsec-secgw: add scripts for functional test
>    doc: update ipsec-secgw guide and relelase notes
>
>   doc/guides/rel_notes/release_19_02.rst        |  14 +
>   doc/guides/sample_app_ug/ipsec_secgw.rst      | 159 +++++-
>   examples/ipsec-secgw/Makefile                 |   5 +-
>   examples/ipsec-secgw/ipsec-secgw.c            | 480 ++++++++++++++----
>   examples/ipsec-secgw/ipsec.c                  | 101 ++--
>   examples/ipsec-secgw/ipsec.h                  |  67 +++
>   examples/ipsec-secgw/ipsec_process.c          | 357 +++++++++++++
>   examples/ipsec-secgw/meson.build              |   6 +-
>   examples/ipsec-secgw/parser.c                 |  91 ++++
>   examples/ipsec-secgw/parser.h                 |   8 +-
>   examples/ipsec-secgw/sa.c                     | 263 +++++++++-
>   examples/ipsec-secgw/sp4.c                    |  35 +-
>   examples/ipsec-secgw/sp6.c                    |  35 +-
>   examples/ipsec-secgw/test/common_defs.sh      | 153 ++++++
>   examples/ipsec-secgw/test/data_rxtx.sh        |  62 +++
>   examples/ipsec-secgw/test/linux_test4.sh      |  63 +++
>   examples/ipsec-secgw/test/linux_test6.sh      |  64 +++
>   examples/ipsec-secgw/test/run_test.sh         |  80 +++
>   .../test/trs_aescbc_sha1_common_defs.sh       |  69 +++
>   .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  67 +++
>   .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
>   .../test/trs_aescbc_sha1_esn_defs.sh          |  66 +++
>   .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
>   .../test/trs_aesgcm_common_defs.sh            |  60 +++
>   examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  66 +++
>   .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
>   .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  66 +++
>   .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
>   .../test/tun_aescbc_sha1_common_defs.sh       |  68 +++
>   .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  70 +++
>   .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
>   .../test/tun_aescbc_sha1_esn_defs.sh          |  70 +++
>   .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
>   .../test/tun_aesgcm_common_defs.sh            |  60 +++
>   examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  70 +++
>   .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
>   .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  70 +++
>   .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
>   38 files changed, 2727 insertions(+), 158 deletions(-)
>   create mode 100644 examples/ipsec-secgw/ipsec_process.c
>   create mode 100644 examples/ipsec-secgw/test/common_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
>   create mode 100644 examples/ipsec-secgw/test/linux_test4.sh
>   create mode 100644 examples/ipsec-secgw/test/linux_test6.sh
>   create mode 100644 examples/ipsec-secgw/test/run_test.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh
>

Re: [PATCH v6 00/10] examples/ipsec-secgw: make app to use ipsec library

[Ananyev, Konstantin]

Hi Akhil,

> 
> Hi Konstantin,
> 
> With this patchset, I am seeing a 3% drop in legacy mode lookaside and
> 12% drop with -l option.
> I am debugging this. Will let you know if I find something.

Ok, thanks.
For legacy mode do you know which patch in the series causing 3% drop?
Is it still: " fix crypto-op might never get dequeued" or a different one?
Konstantin 


> 
> -Akhil
> 
> On 1/4/2019 1:55 AM, Konstantin Ananyev wrote:
> > This patch series depends on the patch series:
> >
> > ipsec: new library for IPsec data-path processing
> > http://patches.dpdk.org/patch/49410/
> > http://patches.dpdk.org/patch/49411/
> > http://patches.dpdk.org/patch/49412/
> > http://patches.dpdk.org/patch/49413/
> > http://patches.dpdk.org/patch/49414/
> > http://patches.dpdk.org/patch/49415/
> > http://patches.dpdk.org/patch/49416/
> > http://patches.dpdk.org/patch/49417/
> > http://patches.dpdk.org/patch/49418/
> > http://patches.dpdk.org/patch/49419/
> >
> > to be applied first.
> >
> > v5 -> v6
> >    Address issues reported by Akhil:
> >       segfault when using lookaside-proto device
> >       HW IPv4 cksum offload not enabled by default
> >       crypto-dev dequeue() is called to often
> >
> > v4 -> v5
> > - Address Akhil comments:
> >       documentation update
> >       spell checks spacing etc.
> >       introduce rxoffload/txoffload parameters
> >       single SA for ipv6
> >       update Makefile
> >
> > v3 -> v4
> >   - fix few issues with the test scripts
> >   - update docs
> >
> > v2 -> v3
> >   - add IPv6 cases into test scripts
> >   - fixes for IPv6 support
> >   - fixes for inline-crypto support
> >   - some code restructuring
> >
> > v1 -> v2
> >   - Several bug fixes
> >
> > That series contians few bug-fixes and changes to make ipsec-secgw
> > to utilize librte_ipsec library:
> >       - changes in the related data structures.
> >       - changes in the initialization code.
> >       - changes in the data-path code.
> >       - new command-line parameters to enable librte_ipsec codepath
> >         and related features.
> >       - test scripts to help automate ipsec-secgw functional testing.
> >
> > Note that right now by default current (non-librte_ipsec) code-path
> > will be used. User has to run application with new command-line option
> > ('-l')
> > to enable new codepath.
> > The main reason for that:
> >    - current librte_ipsec doesn't support all ipsec algorithms
> >      and features that the app does.
> >    - allow users to run both versions in parallel for some time
> >      to figure out any functional or performance degradation with the
> >      new code.
> >
> > Test scripts were run with the following crypto devices:
> >   - aesni_mb
> >   - aesni_gcm
> >   - qat
> >
> > Konstantin Ananyev (10):
> >    examples/ipsec-secgw: allow user to disable some RX/TX offloads
> >    examples/ipsec-secgw: allow to specify neighbour mac address
> >    examples/ipsec-secgw: fix crypto-op might never get dequeued
> >    examples/ipsec-secgw: fix outbound codepath for single SA
> >    examples/ipsec-secgw: make local variables static
> >    examples/ipsec-secgw: fix inbound SA checking
> >    examples/ipsec-secgw: make app to use ipsec library
> >    examples/ipsec-secgw: make data-path to use ipsec library
> >    examples/ipsec-secgw: add scripts for functional test
> >    doc: update ipsec-secgw guide and relelase notes
> >
> >   doc/guides/rel_notes/release_19_02.rst        |  14 +
> >   doc/guides/sample_app_ug/ipsec_secgw.rst      | 159 +++++-
> >   examples/ipsec-secgw/Makefile                 |   5 +-
> >   examples/ipsec-secgw/ipsec-secgw.c            | 480 ++++++++++++++----
> >   examples/ipsec-secgw/ipsec.c                  | 101 ++--
> >   examples/ipsec-secgw/ipsec.h                  |  67 +++
> >   examples/ipsec-secgw/ipsec_process.c          | 357 +++++++++++++
> >   examples/ipsec-secgw/meson.build              |   6 +-
> >   examples/ipsec-secgw/parser.c                 |  91 ++++
> >   examples/ipsec-secgw/parser.h                 |   8 +-
> >   examples/ipsec-secgw/sa.c                     | 263 +++++++++-
> >   examples/ipsec-secgw/sp4.c                    |  35 +-
> >   examples/ipsec-secgw/sp6.c                    |  35 +-
> >   examples/ipsec-secgw/test/common_defs.sh      | 153 ++++++
> >   examples/ipsec-secgw/test/data_rxtx.sh        |  62 +++
> >   examples/ipsec-secgw/test/linux_test4.sh      |  63 +++
> >   examples/ipsec-secgw/test/linux_test6.sh      |  64 +++
> >   examples/ipsec-secgw/test/run_test.sh         |  80 +++
> >   .../test/trs_aescbc_sha1_common_defs.sh       |  69 +++
> >   .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  67 +++
> >   .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
> >   .../test/trs_aescbc_sha1_esn_defs.sh          |  66 +++
> >   .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
> >   .../test/trs_aesgcm_common_defs.sh            |  60 +++
> >   examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  66 +++
> >   .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
> >   .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  66 +++
> >   .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
> >   .../test/tun_aescbc_sha1_common_defs.sh       |  68 +++
> >   .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  70 +++
> >   .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
> >   .../test/tun_aescbc_sha1_esn_defs.sh          |  70 +++
> >   .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
> >   .../test/tun_aesgcm_common_defs.sh            |  60 +++
> >   examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  70 +++
> >   .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
> >   .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  70 +++
> >   .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
> >   38 files changed, 2727 insertions(+), 158 deletions(-)
> >   create mode 100644 examples/ipsec-secgw/ipsec_process.c
> >   create mode 100644 examples/ipsec-secgw/test/common_defs.sh
> >   create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
> >   create mode 100644 examples/ipsec-secgw/test/linux_test4.sh
> >   create mode 100644 examples/ipsec-secgw/test/linux_test6.sh
> >   create mode 100644 examples/ipsec-secgw/test/run_test.sh
> >   create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
> >   create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
> >   create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
> >   create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
> >   create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
> >   create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
> >   create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
> >   create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
> >   create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
> >   create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
> >   create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
> >   create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
> >   create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
> >   create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
> >   create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
> >   create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
> >   create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
> >   create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
> >   create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
> >   create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh
> >

Re: [PATCH v6 00/10] examples/ipsec-secgw: make app to use ipsec library

[Akhil Goyal]

On 1/4/2019 5:59 PM, Ananyev, Konstantin wrote:
> Hi Akhil,
>
>> Hi Konstantin,
>>
>> With this patchset, I am seeing a 3% drop in legacy mode lookaside and
>> 12% drop with -l option.
>> I am debugging this. Will let you know if I find something.
> Ok, thanks.
> For legacy mode do you know which patch in the series causing 3% drop?
> Is it still: " fix crypto-op might never get dequeued" or a different one?
> Konstantin
>
for legacy mode you may consider this diff in 3/10

diff --git a/examples/ipsec-secgw/ipsec-secgw.c 
b/examples/ipsec-secgw/ipsec-secgw.c
index 4f7a77d8d..d183862b8 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -1015,10 +1015,13 @@ main_loop(__attribute__((unused)) void *dummy)

                         if (nb_rx > 0)
                                 process_pkts(qconf, pkts, nb_rx, portid);
-               }
+//             }

-               drain_inbound_crypto_queues(qconf, &qconf->inbound);
-               drain_outbound_crypto_queues(qconf, &qconf->outbound);
+                       if (UNPROTECTED_PORT(portid))
+                               drain_inbound_crypto_queues(qconf, 
&qconf->inbound);
+                       else
+                               drain_outbound_crypto_queues(qconf, 
&qconf->outbound);
+               }
         }
  }

The 3% gap was on single core performance. But on multi cores scenario, 
there is no significant drop.

But I see a bigger functional problem in case of non-legacy mode.
I am trying a multi tunnel scenario with 32 kind of flows on one side 
each with different tunnels.
The flows are kind of dest ip: 192.168.101.1 ........... 192.168.x.1  
.........  192.168.132.1.
Each IP has a different tunnel.
All are pumped from same port. I can see that some of the IPs are not 
getting forwarded.
like 192.168.103.1, 192.168.104.1, 192.168.105.1, 192.168.114.1, 
192.168.115.1, 192.168.116.1, 192.168.122.1, 192.168.125.1, 
192.168.126.1, 192.168.130.1

The same setup/flows works perfectly fine with legacy mode.


>> -Akhil
>>
>> On 1/4/2019 1:55 AM, Konstantin Ananyev wrote:
>>> This patch series depends on the patch series:
>>>
>>> ipsec: new library for IPsec data-path processing
>>> http://patches.dpdk.org/patch/49410/
>>> http://patches.dpdk.org/patch/49411/
>>> http://patches.dpdk.org/patch/49412/
>>> http://patches.dpdk.org/patch/49413/
>>> http://patches.dpdk.org/patch/49414/
>>> http://patches.dpdk.org/patch/49415/
>>> http://patches.dpdk.org/patch/49416/
>>> http://patches.dpdk.org/patch/49417/
>>> http://patches.dpdk.org/patch/49418/
>>> http://patches.dpdk.org/patch/49419/
>>>
>>> to be applied first.
>>>
>>> v5 -> v6
>>>     Address issues reported by Akhil:
>>>        segfault when using lookaside-proto device
>>>        HW IPv4 cksum offload not enabled by default
>>>        crypto-dev dequeue() is called to often
>>>
>>> v4 -> v5
>>> - Address Akhil comments:
>>>        documentation update
>>>        spell checks spacing etc.
>>>        introduce rxoffload/txoffload parameters
>>>        single SA for ipv6
>>>        update Makefile
>>>
>>> v3 -> v4
>>>    - fix few issues with the test scripts
>>>    - update docs
>>>
>>> v2 -> v3
>>>    - add IPv6 cases into test scripts
>>>    - fixes for IPv6 support
>>>    - fixes for inline-crypto support
>>>    - some code restructuring
>>>
>>> v1 -> v2
>>>    - Several bug fixes
>>>
>>> That series contians few bug-fixes and changes to make ipsec-secgw
>>> to utilize librte_ipsec library:
>>>        - changes in the related data structures.
>>>        - changes in the initialization code.
>>>        - changes in the data-path code.
>>>        - new command-line parameters to enable librte_ipsec codepath
>>>          and related features.
>>>        - test scripts to help automate ipsec-secgw functional testing.
>>>
>>> Note that right now by default current (non-librte_ipsec) code-path
>>> will be used. User has to run application with new command-line option
>>> ('-l')
>>> to enable new codepath.
>>> The main reason for that:
>>>     - current librte_ipsec doesn't support all ipsec algorithms
>>>       and features that the app does.
>>>     - allow users to run both versions in parallel for some time
>>>       to figure out any functional or performance degradation with the
>>>       new code.
>>>
>>> Test scripts were run with the following crypto devices:
>>>    - aesni_mb
>>>    - aesni_gcm
>>>    - qat
>>>
>>> Konstantin Ananyev (10):
>>>     examples/ipsec-secgw: allow user to disable some RX/TX offloads
>>>     examples/ipsec-secgw: allow to specify neighbour mac address
>>>     examples/ipsec-secgw: fix crypto-op might never get dequeued
>>>     examples/ipsec-secgw: fix outbound codepath for single SA
>>>     examples/ipsec-secgw: make local variables static
>>>     examples/ipsec-secgw: fix inbound SA checking
>>>     examples/ipsec-secgw: make app to use ipsec library
>>>     examples/ipsec-secgw: make data-path to use ipsec library
>>>     examples/ipsec-secgw: add scripts for functional test
>>>     doc: update ipsec-secgw guide and relelase notes
>>>
>>>    doc/guides/rel_notes/release_19_02.rst        |  14 +
>>>    doc/guides/sample_app_ug/ipsec_secgw.rst      | 159 +++++-
>>>    examples/ipsec-secgw/Makefile                 |   5 +-
>>>    examples/ipsec-secgw/ipsec-secgw.c            | 480 ++++++++++++++----
>>>    examples/ipsec-secgw/ipsec.c                  | 101 ++--
>>>    examples/ipsec-secgw/ipsec.h                  |  67 +++
>>>    examples/ipsec-secgw/ipsec_process.c          | 357 +++++++++++++
>>>    examples/ipsec-secgw/meson.build              |   6 +-
>>>    examples/ipsec-secgw/parser.c                 |  91 ++++
>>>    examples/ipsec-secgw/parser.h                 |   8 +-
>>>    examples/ipsec-secgw/sa.c                     | 263 +++++++++-
>>>    examples/ipsec-secgw/sp4.c                    |  35 +-
>>>    examples/ipsec-secgw/sp6.c                    |  35 +-
>>>    examples/ipsec-secgw/test/common_defs.sh      | 153 ++++++
>>>    examples/ipsec-secgw/test/data_rxtx.sh        |  62 +++
>>>    examples/ipsec-secgw/test/linux_test4.sh      |  63 +++
>>>    examples/ipsec-secgw/test/linux_test6.sh      |  64 +++
>>>    examples/ipsec-secgw/test/run_test.sh         |  80 +++
>>>    .../test/trs_aescbc_sha1_common_defs.sh       |  69 +++
>>>    .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  67 +++
>>>    .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
>>>    .../test/trs_aescbc_sha1_esn_defs.sh          |  66 +++
>>>    .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
>>>    .../test/trs_aesgcm_common_defs.sh            |  60 +++
>>>    examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  66 +++
>>>    .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
>>>    .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  66 +++
>>>    .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
>>>    .../test/tun_aescbc_sha1_common_defs.sh       |  68 +++
>>>    .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  70 +++
>>>    .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
>>>    .../test/tun_aescbc_sha1_esn_defs.sh          |  70 +++
>>>    .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
>>>    .../test/tun_aesgcm_common_defs.sh            |  60 +++
>>>    examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  70 +++
>>>    .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
>>>    .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  70 +++
>>>    .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
>>>    38 files changed, 2727 insertions(+), 158 deletions(-)
>>>    create mode 100644 examples/ipsec-secgw/ipsec_process.c
>>>    create mode 100644 examples/ipsec-secgw/test/common_defs.sh
>>>    create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
>>>    create mode 100644 examples/ipsec-secgw/test/linux_test4.sh
>>>    create mode 100644 examples/ipsec-secgw/test/linux_test6.sh
>>>    create mode 100644 examples/ipsec-secgw/test/run_test.sh
>>>    create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
>>>    create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
>>>    create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
>>>    create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
>>>    create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
>>>    create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
>>>    create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
>>>    create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
>>>    create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
>>>    create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
>>>    create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
>>>    create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
>>>    create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
>>>    create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
>>>    create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
>>>    create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
>>>    create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
>>>    create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
>>>    create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
>>>    create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh
>>>

Re: [PATCH v6 00/10] examples/ipsec-secgw: make app to use ipsec library

[Akhil Goyal]

On 1/4/2019 8:10 PM, Akhil Goyal wrote:
>
> On 1/4/2019 5:59 PM, Ananyev, Konstantin wrote:
>> Hi Akhil,
>>
>>> Hi Konstantin,
>>>
>>> With this patchset, I am seeing a 3% drop in legacy mode lookaside and
>>> 12% drop with -l option.
>>> I am debugging this. Will let you know if I find something.
>> Ok, thanks.
>> For legacy mode do you know which patch in the series causing 3% drop?
>> Is it still: " fix crypto-op might never get dequeued" or a different one?
>> Konstantin
>>
> for legacy mode you may consider this diff in 3/10
>
> diff --git a/examples/ipsec-secgw/ipsec-secgw.c
> b/examples/ipsec-secgw/ipsec-secgw.c
> index 4f7a77d8d..d183862b8 100644
> --- a/examples/ipsec-secgw/ipsec-secgw.c
> +++ b/examples/ipsec-secgw/ipsec-secgw.c
> @@ -1015,10 +1015,13 @@ main_loop(__attribute__((unused)) void *dummy)
>
>                           if (nb_rx > 0)
>                                   process_pkts(qconf, pkts, nb_rx, portid);
> -               }
> +//             }
>
> -               drain_inbound_crypto_queues(qconf, &qconf->inbound);
> -               drain_outbound_crypto_queues(qconf, &qconf->outbound);
> +                       if (UNPROTECTED_PORT(portid))
> +                               drain_inbound_crypto_queues(qconf,
> &qconf->inbound);
> +                       else
> +                               drain_outbound_crypto_queues(qconf,
> &qconf->outbound);
> +               }
>           }
>    }
>
> The 3% gap was on single core performance. But on multi cores scenario,
> there is no significant drop.
>
> But I see a bigger functional problem in case of non-legacy mode.
> I am trying a multi tunnel scenario with 32 kind of flows on one side
> each with different tunnels.
> The flows are kind of dest ip: 192.168.101.1 ........... 192.168.x.1
> .........  192.168.132.1.
> Each IP has a different tunnel.
> All are pumped from same port. I can see that some of the IPs are not
> getting forwarded.
> like 192.168.103.1, 192.168.104.1, 192.168.105.1, 192.168.114.1,
> 192.168.115.1, 192.168.116.1, 192.168.122.1, 192.168.125.1,
> 192.168.126.1, 192.168.130.1
>
> The same setup/flows works perfectly fine with legacy mode.

I see that the IPs are not fixed and changes on every run. Probably, the 
inbound /outbound queues gets flushed when they do not intend to on that 
particular core and packets get discarded as they come from non-destined 
port.
Also the issue does not come if I add the above suggested change.

>
>
>>> -Akhil
>>>
>>> On 1/4/2019 1:55 AM, Konstantin Ananyev wrote:
>>>> This patch series depends on the patch series:
>>>>
>>>> ipsec: new library for IPsec data-path processing
>>>> http://patches.dpdk.org/patch/49410/
>>>> http://patches.dpdk.org/patch/49411/
>>>> http://patches.dpdk.org/patch/49412/
>>>> http://patches.dpdk.org/patch/49413/
>>>> http://patches.dpdk.org/patch/49414/
>>>> http://patches.dpdk.org/patch/49415/
>>>> http://patches.dpdk.org/patch/49416/
>>>> http://patches.dpdk.org/patch/49417/
>>>> http://patches.dpdk.org/patch/49418/
>>>> http://patches.dpdk.org/patch/49419/
>>>>
>>>> to be applied first.
>>>>
>>>> v5 -> v6
>>>>      Address issues reported by Akhil:
>>>>         segfault when using lookaside-proto device
>>>>         HW IPv4 cksum offload not enabled by default
>>>>         crypto-dev dequeue() is called to often
>>>>
>>>> v4 -> v5
>>>> - Address Akhil comments:
>>>>         documentation update
>>>>         spell checks spacing etc.
>>>>         introduce rxoffload/txoffload parameters
>>>>         single SA for ipv6
>>>>         update Makefile
>>>>
>>>> v3 -> v4
>>>>     - fix few issues with the test scripts
>>>>     - update docs
>>>>
>>>> v2 -> v3
>>>>     - add IPv6 cases into test scripts
>>>>     - fixes for IPv6 support
>>>>     - fixes for inline-crypto support
>>>>     - some code restructuring
>>>>
>>>> v1 -> v2
>>>>     - Several bug fixes
>>>>
>>>> That series contians few bug-fixes and changes to make ipsec-secgw
>>>> to utilize librte_ipsec library:
>>>>         - changes in the related data structures.
>>>>         - changes in the initialization code.
>>>>         - changes in the data-path code.
>>>>         - new command-line parameters to enable librte_ipsec codepath
>>>>           and related features.
>>>>         - test scripts to help automate ipsec-secgw functional testing.
>>>>
>>>> Note that right now by default current (non-librte_ipsec) code-path
>>>> will be used. User has to run application with new command-line option
>>>> ('-l')
>>>> to enable new codepath.
>>>> The main reason for that:
>>>>      - current librte_ipsec doesn't support all ipsec algorithms
>>>>        and features that the app does.
>>>>      - allow users to run both versions in parallel for some time
>>>>        to figure out any functional or performance degradation with the
>>>>        new code.
>>>>
>>>> Test scripts were run with the following crypto devices:
>>>>     - aesni_mb
>>>>     - aesni_gcm
>>>>     - qat
>>>>
>>>> Konstantin Ananyev (10):
>>>>      examples/ipsec-secgw: allow user to disable some RX/TX offloads
>>>>      examples/ipsec-secgw: allow to specify neighbour mac address
>>>>      examples/ipsec-secgw: fix crypto-op might never get dequeued
>>>>      examples/ipsec-secgw: fix outbound codepath for single SA
>>>>      examples/ipsec-secgw: make local variables static
>>>>      examples/ipsec-secgw: fix inbound SA checking
>>>>      examples/ipsec-secgw: make app to use ipsec library
>>>>      examples/ipsec-secgw: make data-path to use ipsec library
>>>>      examples/ipsec-secgw: add scripts for functional test
>>>>      doc: update ipsec-secgw guide and relelase notes
>>>>
>>>>     doc/guides/rel_notes/release_19_02.rst        |  14 +
>>>>     doc/guides/sample_app_ug/ipsec_secgw.rst      | 159 +++++-
>>>>     examples/ipsec-secgw/Makefile                 |   5 +-
>>>>     examples/ipsec-secgw/ipsec-secgw.c            | 480 ++++++++++++++----
>>>>     examples/ipsec-secgw/ipsec.c                  | 101 ++--
>>>>     examples/ipsec-secgw/ipsec.h                  |  67 +++
>>>>     examples/ipsec-secgw/ipsec_process.c          | 357 +++++++++++++
>>>>     examples/ipsec-secgw/meson.build              |   6 +-
>>>>     examples/ipsec-secgw/parser.c                 |  91 ++++
>>>>     examples/ipsec-secgw/parser.h                 |   8 +-
>>>>     examples/ipsec-secgw/sa.c                     | 263 +++++++++-
>>>>     examples/ipsec-secgw/sp4.c                    |  35 +-
>>>>     examples/ipsec-secgw/sp6.c                    |  35 +-
>>>>     examples/ipsec-secgw/test/common_defs.sh      | 153 ++++++
>>>>     examples/ipsec-secgw/test/data_rxtx.sh        |  62 +++
>>>>     examples/ipsec-secgw/test/linux_test4.sh      |  63 +++
>>>>     examples/ipsec-secgw/test/linux_test6.sh      |  64 +++
>>>>     examples/ipsec-secgw/test/run_test.sh         |  80 +++
>>>>     .../test/trs_aescbc_sha1_common_defs.sh       |  69 +++
>>>>     .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  67 +++
>>>>     .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
>>>>     .../test/trs_aescbc_sha1_esn_defs.sh          |  66 +++
>>>>     .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
>>>>     .../test/trs_aesgcm_common_defs.sh            |  60 +++
>>>>     examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  66 +++
>>>>     .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
>>>>     .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  66 +++
>>>>     .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
>>>>     .../test/tun_aescbc_sha1_common_defs.sh       |  68 +++
>>>>     .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  70 +++
>>>>     .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
>>>>     .../test/tun_aescbc_sha1_esn_defs.sh          |  70 +++
>>>>     .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
>>>>     .../test/tun_aesgcm_common_defs.sh            |  60 +++
>>>>     examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  70 +++
>>>>     .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
>>>>     .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  70 +++
>>>>     .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
>>>>     38 files changed, 2727 insertions(+), 158 deletions(-)
>>>>     create mode 100644 examples/ipsec-secgw/ipsec_process.c
>>>>     create mode 100644 examples/ipsec-secgw/test/common_defs.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/linux_test4.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/linux_test6.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/run_test.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
>>>>     create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh
>>>>

Re: [PATCH v6 00/10] examples/ipsec-secgw: make app to use ipsec library

[Ananyev, Konstantin]

> On 1/4/2019 5:59 PM, Ananyev, Konstantin wrote:
> > Hi Akhil,
> >
> >> Hi Konstantin,
> >>
> >> With this patchset, I am seeing a 3% drop in legacy mode lookaside and
> >> 12% drop with -l option.
> >> I am debugging this. Will let you know if I find something.
> > Ok, thanks.
> > For legacy mode do you know which patch in the series causing 3% drop?
> > Is it still: " fix crypto-op might never get dequeued" or a different one?
> > Konstantin
> >
> for legacy mode you may consider this diff in 3/10
> 
> diff --git a/examples/ipsec-secgw/ipsec-secgw.c
> b/examples/ipsec-secgw/ipsec-secgw.c
> index 4f7a77d8d..d183862b8 100644
> --- a/examples/ipsec-secgw/ipsec-secgw.c
> +++ b/examples/ipsec-secgw/ipsec-secgw.c
> @@ -1015,10 +1015,13 @@ main_loop(__attribute__((unused)) void *dummy)
> 
>                          if (nb_rx > 0)
>                                  process_pkts(qconf, pkts, nb_rx, portid);
> -               }
> +//             }
> 
> -               drain_inbound_crypto_queues(qconf, &qconf->inbound);
> -               drain_outbound_crypto_queues(qconf, &qconf->outbound);
> +                       if (UNPROTECTED_PORT(portid))
> +                               drain_inbound_crypto_queues(qconf,
> &qconf->inbound);
> +                       else
> +                               drain_outbound_crypto_queues(qconf,
> &qconf->outbound);
> +               }
>          }
>   }
> 
> The 3% gap was on single core performance. But on multi cores scenario,
> there is no significant drop.
> 
> But I see a bigger functional problem in case of non-legacy mode.
> I am trying a multi tunnel scenario with 32 kind of flows on one side
> each with different tunnels.
> The flows are kind of dest ip: 192.168.101.1 ........... 192.168.x.1
> .........  192.168.132.1.
> Each IP has a different tunnel.
> All are pumped from same port. I can see that some of the IPs are not
> getting forwarded.
> like 192.168.103.1, 192.168.104.1, 192.168.105.1, 192.168.114.1,
> 192.168.115.1, 192.168.116.1, 192.168.122.1, 192.168.125.1,
> 192.168.126.1, 192.168.130.1
> 
> The same setup/flows works perfectly fine with legacy mode.

I'll try to reproduce it on my box.
To clarify:
   you are using one core, one port.
   each dest ip assigned to different SA
  (i.e. for dst=192.168.103.1, spi=1,  dst=192.168.104.1, spi=2, ...)
  and for some SAs packets are not forwarded.
  I presume each rx burst contains packets from different SAs.
 correct?
 Is that failure for inbound or outbound packets or both?
 Any particular details about mode(tunnel/transport), algorithm? 
   
Thanks
Konstantin

Re: [PATCH v6 00/10] examples/ipsec-secgw: make app to use ipsec library

[Akhil Goyal]

On 1/4/2019 10:34 PM, Ananyev, Konstantin wrote:
>> On 1/4/2019 5:59 PM, Ananyev, Konstantin wrote:
>>> Hi Akhil,
>>>
>>>> Hi Konstantin,
>>>>
>>>> With this patchset, I am seeing a 3% drop in legacy mode lookaside and
>>>> 12% drop with -l option.
>>>> I am debugging this. Will let you know if I find something.
>>> Ok, thanks.
>>> For legacy mode do you know which patch in the series causing 3% drop?
>>> Is it still: " fix crypto-op might never get dequeued" or a different one?
>>> Konstantin
>>>
>> for legacy mode you may consider this diff in 3/10
>>
>> diff --git a/examples/ipsec-secgw/ipsec-secgw.c
>> b/examples/ipsec-secgw/ipsec-secgw.c
>> index 4f7a77d8d..d183862b8 100644
>> --- a/examples/ipsec-secgw/ipsec-secgw.c
>> +++ b/examples/ipsec-secgw/ipsec-secgw.c
>> @@ -1015,10 +1015,13 @@ main_loop(__attribute__((unused)) void *dummy)
>>
>>                           if (nb_rx > 0)
>>                                   process_pkts(qconf, pkts, nb_rx, portid);
>> -               }
>> +//             }
>>
>> -               drain_inbound_crypto_queues(qconf, &qconf->inbound);
>> -               drain_outbound_crypto_queues(qconf, &qconf->outbound);
>> +                       if (UNPROTECTED_PORT(portid))
>> +                               drain_inbound_crypto_queues(qconf,
>> &qconf->inbound);
>> +                       else
>> +                               drain_outbound_crypto_queues(qconf,
>> &qconf->outbound);
>> +               }
>>           }
>>    }
>>
>> The 3% gap was on single core performance. But on multi cores scenario,
>> there is no significant drop.
>>
>> But I see a bigger functional problem in case of non-legacy mode.
>> I am trying a multi tunnel scenario with 32 kind of flows on one side
>> each with different tunnels.
>> The flows are kind of dest ip: 192.168.101.1 ........... 192.168.x.1
>> .........  192.168.132.1.
>> Each IP has a different tunnel.
>> All are pumped from same port. I can see that some of the IPs are not
>> getting forwarded.
>> like 192.168.103.1, 192.168.104.1, 192.168.105.1, 192.168.114.1,
>> 192.168.115.1, 192.168.116.1, 192.168.122.1, 192.168.125.1,
>> 192.168.126.1, 192.168.130.1
>>
>> The same setup/flows works perfectly fine with legacy mode.
> I'll try to reproduce it on my box.
> To clarify:
>     you are using one core, one port.
4 cores one port with 4 queues

./ipsec-secgw -c 0xff -n 1   -- -p 0x3 -P -u 0x2 
--config="(0,0,0),(0,1,1),(0,2,2),(0,3,3),(1,0,4),(1,1,5),(1,2,6),(1,3,7)" 
-f ep0_64X64_proto.cfg
>     each dest ip assigned to different SA
>    (i.e. for dst=192.168.103.1, spi=1,  dst=192.168.104.1, spi=2, ...)
yes
>    and for some SAs packets are not forwarded.
>    I presume each rx burst contains packets from different SAs.
>   correct?
yes
>   Is that failure for inbound or outbound packets or both?
I believe it can be anything, probably because unprotected port is 
trying to dequeue outbound queues.
>   Any particular details about mode(tunnel/transport), algorithm?
tunnel, aes-cbc-hmac-sha1
>     
> Thanks
> Konstantin

[PATCH v6 01/10] examples/ipsec-secgw: allow user to disable some RX/TX offloads

[Konstantin Ananyev]

Right now ipsec-secgw always enables TX offloads
(DEV_TX_OFFLOAD_MULTI_SEGS, DEV_TX_OFFLOAD_SECURITY),
even when they are not requested by the config.
That causes many PMD to choose full-featured TX function,
which in many cases is much slower then one without offloads.
That patch adds ability for the user to disable unneeded HW offloads.
If DEV_TX_OFFLOAD_IPV4_CKSUM is disabled by user, then
SW version of ip cksum calculation is used.
That allows to use vector TX function, when inline-ipsec is not
requested.

Signed-off-by: Remy Horton <remy.horton@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 doc/guides/sample_app_ug/ipsec_secgw.rst |  12 +++
 examples/ipsec-secgw/ipsec-secgw.c       | 126 ++++++++++++++++++++---
 examples/ipsec-secgw/ipsec.h             |   6 ++
 examples/ipsec-secgw/sa.c                |  35 +++++++
 4 files changed, 164 insertions(+), 15 deletions(-)

diff --git a/doc/guides/sample_app_ug/ipsec_secgw.rst b/doc/guides/sample_app_ug/ipsec_secgw.rst
index 4869a011d..244bde2de 100644
--- a/doc/guides/sample_app_ug/ipsec_secgw.rst
+++ b/doc/guides/sample_app_ug/ipsec_secgw.rst
@@ -95,6 +95,8 @@ The application has a number of command line options::
                         -p PORTMASK -P -u PORTMASK -j FRAMESIZE
                         --config (port,queue,lcore)[,(port,queue,lcore]
                         --single-sa SAIDX
+                        --rxoffload MASK
+                        --txoffload MASK
                         -f CONFIG_FILE_PATH
 
 Where:
@@ -119,6 +121,16 @@ Where:
     on both Inbound and Outbound. This option is meant for debugging/performance
     purposes.
 
+*   ``--rxoffload MASK``: RX HW offload capabilities to enable/use on this port
+    (bitmask of DEV_RX_OFFLOAD_* values). It is an optional parameter and
+    allows user to disable some of the RX HW offload capabilities.
+    By default all HW RX offloads are enabled.
+
+*   ``--txoffload MASK``: TX HW offload capabilities to enable/use on this port
+    (bitmask of DEV_TX_OFFLOAD_* values). It is an optional parameter and
+    allows user to disable some of the TX HW offload capabilities.
+    By default all HW TX offloads are enabled.
+
 *   ``-f CONFIG_FILE_PATH``: the full path of text-based file containing all
     configuration items for running the application (See Configuration file
     syntax section below). ``-f CONFIG_FILE_PATH`` **must** be specified.
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 1bc0b5b50..4ccff38d2 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -124,6 +124,8 @@ struct ethaddr_info ethaddr_tbl[RTE_MAX_ETHPORTS] = {
 #define CMD_LINE_OPT_CONFIG		"config"
 #define CMD_LINE_OPT_SINGLE_SA		"single-sa"
 #define CMD_LINE_OPT_CRYPTODEV_MASK	"cryptodev_mask"
+#define CMD_LINE_OPT_RX_OFFLOAD		"rxoffload"
+#define CMD_LINE_OPT_TX_OFFLOAD		"txoffload"
 
 enum {
 	/* long options mapped to a short option */
@@ -135,12 +137,16 @@ enum {
 	CMD_LINE_OPT_CONFIG_NUM,
 	CMD_LINE_OPT_SINGLE_SA_NUM,
 	CMD_LINE_OPT_CRYPTODEV_MASK_NUM,
+	CMD_LINE_OPT_RX_OFFLOAD_NUM,
+	CMD_LINE_OPT_TX_OFFLOAD_NUM,
 };
 
 static const struct option lgopts[] = {
 	{CMD_LINE_OPT_CONFIG, 1, 0, CMD_LINE_OPT_CONFIG_NUM},
 	{CMD_LINE_OPT_SINGLE_SA, 1, 0, CMD_LINE_OPT_SINGLE_SA_NUM},
 	{CMD_LINE_OPT_CRYPTODEV_MASK, 1, 0, CMD_LINE_OPT_CRYPTODEV_MASK_NUM},
+	{CMD_LINE_OPT_RX_OFFLOAD, 1, 0, CMD_LINE_OPT_RX_OFFLOAD_NUM},
+	{CMD_LINE_OPT_TX_OFFLOAD, 1, 0, CMD_LINE_OPT_TX_OFFLOAD_NUM},
 	{NULL, 0, 0, 0}
 };
 
@@ -155,6 +161,13 @@ static uint32_t single_sa;
 static uint32_t single_sa_idx;
 static uint32_t frame_size;
 
+/*
+ * RX/TX HW offload capabilities to enable/use on ethernet ports.
+ * By default all capabilities are enabled.
+ */
+static uint64_t dev_rx_offload = UINT64_MAX;
+static uint64_t dev_tx_offload = UINT64_MAX;
+
 struct lcore_rx_queue {
 	uint16_t port_id;
 	uint8_t queue_id;
@@ -208,8 +221,6 @@ static struct rte_eth_conf port_conf = {
 	},
 	.txmode = {
 		.mq_mode = ETH_MQ_TX_NONE,
-		.offloads = (DEV_TX_OFFLOAD_IPV4_CKSUM |
-			     DEV_TX_OFFLOAD_MULTI_SEGS),
 	},
 };
 
@@ -315,7 +326,8 @@ prepare_traffic(struct rte_mbuf **pkts, struct ipsec_traffic *t,
 }
 
 static inline void
-prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
+prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port,
+		const struct lcore_conf *qconf)
 {
 	struct ip *ip;
 	struct ether_hdr *ethhdr;
@@ -325,14 +337,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
 	ethhdr = (struct ether_hdr *)rte_pktmbuf_prepend(pkt, ETHER_HDR_LEN);
 
 	if (ip->ip_v == IPVERSION) {
-		pkt->ol_flags |= PKT_TX_IP_CKSUM | PKT_TX_IPV4;
+		pkt->ol_flags |= qconf->outbound.ipv4_offloads;
 		pkt->l3_len = sizeof(struct ip);
 		pkt->l2_len = ETHER_HDR_LEN;
 
 		ip->ip_sum = 0;
+
+		/* calculate IPv4 cksum in SW */
+		if ((pkt->ol_flags & PKT_TX_IP_CKSUM) == 0)
+			ip->ip_sum = rte_ipv4_cksum((struct ipv4_hdr *)ip);
+
 		ethhdr->ether_type = rte_cpu_to_be_16(ETHER_TYPE_IPv4);
 	} else {
-		pkt->ol_flags |= PKT_TX_IPV6;
+		pkt->ol_flags |= qconf->outbound.ipv6_offloads;
 		pkt->l3_len = sizeof(struct ip6_hdr);
 		pkt->l2_len = ETHER_HDR_LEN;
 
@@ -346,18 +363,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
 }
 
 static inline void
-prepare_tx_burst(struct rte_mbuf *pkts[], uint16_t nb_pkts, uint16_t port)
+prepare_tx_burst(struct rte_mbuf *pkts[], uint16_t nb_pkts, uint16_t port,
+		const struct lcore_conf *qconf)
 {
 	int32_t i;
 	const int32_t prefetch_offset = 2;
 
 	for (i = 0; i < (nb_pkts - prefetch_offset); i++) {
 		rte_mbuf_prefetch_part2(pkts[i + prefetch_offset]);
-		prepare_tx_pkt(pkts[i], port);
+		prepare_tx_pkt(pkts[i], port, qconf);
 	}
 	/* Process left packets */
 	for (; i < nb_pkts; i++)
-		prepare_tx_pkt(pkts[i], port);
+		prepare_tx_pkt(pkts[i], port, qconf);
 }
 
 /* Send burst of packets on an output interface */
@@ -371,7 +389,7 @@ send_burst(struct lcore_conf *qconf, uint16_t n, uint16_t port)
 	queueid = qconf->tx_queue_id[port];
 	m_table = (struct rte_mbuf **)qconf->tx_mbufs[port].m_table;
 
-	prepare_tx_burst(m_table, n, port);
+	prepare_tx_burst(m_table, n, port, qconf);
 
 	ret = rte_eth_tx_burst(port, queueid, m_table, n);
 	if (unlikely(ret < n)) {
@@ -954,6 +972,8 @@ print_usage(const char *prgname)
 		" --config (port,queue,lcore)[,(port,queue,lcore)]"
 		" [--single-sa SAIDX]"
 		" [--cryptodev_mask MASK]"
+		" [--" CMD_LINE_OPT_RX_OFFLOAD " RX_OFFLOAD_MASK]"
+		" [--" CMD_LINE_OPT_TX_OFFLOAD " TX_OFFLOAD_MASK]"
 		"\n\n"
 		"  -p PORTMASK: Hexadecimal bitmask of ports to configure\n"
 		"  -P : Enable promiscuous mode\n"
@@ -966,10 +986,31 @@ print_usage(const char *prgname)
 		"                     bypassing the SP\n"
 		"  --cryptodev_mask MASK: Hexadecimal bitmask of the crypto\n"
 		"                         devices to configure\n"
+		"  --" CMD_LINE_OPT_RX_OFFLOAD
+		": bitmask of the RX HW offload capabilities to enable/use\n"
+		"                         (DEV_RX_OFFLOAD_*)\n"
+		"  --" CMD_LINE_OPT_TX_OFFLOAD
+		": bitmask of the TX HW offload capabilities to enable/use\n"
+		"                         (DEV_TX_OFFLOAD_*)\n"
 		"\n",
 		prgname);
 }
 
+static int
+parse_mask(const char *str, uint64_t *val)
+{
+	char *end;
+	unsigned long t;
+
+	errno = 0;
+	t = strtoul(str, &end, 0);
+	if (errno != 0 || end[0] != 0)
+		return -EINVAL;
+
+	*val = t;
+	return 0;
+}
+
 static int32_t
 parse_portmask(const char *portmask)
 {
@@ -1156,6 +1197,24 @@ parse_args(int32_t argc, char **argv)
 			/* else */
 			enabled_cryptodev_mask = ret;
 			break;
+		case CMD_LINE_OPT_RX_OFFLOAD_NUM:
+			ret = parse_mask(optarg, &dev_rx_offload);
+			if (ret != 0) {
+				printf("Invalid argument for \'%s\': %s\n",
+					CMD_LINE_OPT_RX_OFFLOAD, optarg);
+				print_usage(prgname);
+				return -1;
+			}
+			break;
+		case CMD_LINE_OPT_TX_OFFLOAD_NUM:
+			ret = parse_mask(optarg, &dev_tx_offload);
+			if (ret != 0) {
+				printf("Invalid argument for \'%s\': %s\n",
+					CMD_LINE_OPT_TX_OFFLOAD, optarg);
+				print_usage(prgname);
+				return -1;
+			}
+			break;
 		default:
 			print_usage(prgname);
 			return -1;
@@ -1543,7 +1602,7 @@ cryptodevs_init(void)
 }
 
 static void
-port_init(uint16_t portid)
+port_init(uint16_t portid, uint64_t req_rx_offloads, uint64_t req_tx_offloads)
 {
 	struct rte_eth_dev_info dev_info;
 	struct rte_eth_txconf *txconf;
@@ -1556,6 +1615,10 @@ port_init(uint16_t portid)
 
 	rte_eth_dev_info_get(portid, &dev_info);
 
+	/* limit allowed HW offloafs, as user requested */
+	dev_info.rx_offload_capa &= dev_rx_offload;
+	dev_info.tx_offload_capa &= dev_tx_offload;
+
 	printf("Configuring device port %u:\n", portid);
 
 	rte_eth_macaddr_get(portid, &ethaddr);
@@ -1584,14 +1647,38 @@ port_init(uint16_t portid)
 		local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_JUMBO_FRAME;
 	}
 
-	if (dev_info.rx_offload_capa & DEV_RX_OFFLOAD_SECURITY)
-		local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_SECURITY;
-	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_SECURITY)
-		local_port_conf.txmode.offloads |= DEV_TX_OFFLOAD_SECURITY;
+	local_port_conf.rxmode.offloads |= req_rx_offloads;
+	local_port_conf.txmode.offloads |= req_tx_offloads;
+
+	/* Check that all required capabilities are supported */
+	if ((local_port_conf.rxmode.offloads & dev_info.rx_offload_capa) !=
+			local_port_conf.rxmode.offloads)
+		rte_exit(EXIT_FAILURE,
+			"Error: port %u required RX offloads: 0x%" PRIx64
+			", avaialbe RX offloads: 0x%" PRIx64 "\n",
+			portid, local_port_conf.rxmode.offloads,
+			dev_info.rx_offload_capa);
+
+	if ((local_port_conf.txmode.offloads & dev_info.tx_offload_capa) !=
+			local_port_conf.txmode.offloads)
+		rte_exit(EXIT_FAILURE,
+			"Error: port %u required TX offloads: 0x%" PRIx64
+			", avaialbe TX offloads: 0x%" PRIx64 "\n",
+			portid, local_port_conf.txmode.offloads,
+			dev_info.tx_offload_capa);
+
 	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_MBUF_FAST_FREE)
 		local_port_conf.txmode.offloads |=
 			DEV_TX_OFFLOAD_MBUF_FAST_FREE;
 
+	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_IPV4_CKSUM)
+		local_port_conf.txmode.offloads |= DEV_TX_OFFLOAD_IPV4_CKSUM;
+
+	printf("port %u configurng rx_offloads=0x%" PRIx64
+		", tx_offloads=0x%" PRIx64 "\n",
+		portid, local_port_conf.rxmode.offloads,
+		local_port_conf.txmode.offloads);
+
 	local_port_conf.rx_adv_conf.rss_conf.rss_hf &=
 		dev_info.flow_type_rss_offloads;
 	if (local_port_conf.rx_adv_conf.rss_conf.rss_hf !=
@@ -1639,6 +1726,13 @@ port_init(uint16_t portid)
 
 		qconf = &lcore_conf[lcore_id];
 		qconf->tx_queue_id[portid] = tx_queueid;
+
+		/* Pre-populate pkt offloads based on capabilities */
+		qconf->outbound.ipv4_offloads = PKT_TX_IPV4;
+		qconf->outbound.ipv6_offloads = PKT_TX_IPV6;
+		if (local_port_conf.txmode.offloads & DEV_TX_OFFLOAD_IPV4_CKSUM)
+			qconf->outbound.ipv4_offloads |= PKT_TX_IP_CKSUM;
+
 		tx_queueid++;
 
 		/* init RX queues */
@@ -1749,6 +1843,7 @@ main(int32_t argc, char **argv)
 	uint32_t lcore_id;
 	uint8_t socket_id;
 	uint16_t portid;
+	uint64_t req_rx_offloads, req_tx_offloads;
 
 	/* init EAL */
 	ret = rte_eal_init(argc, argv);
@@ -1804,7 +1899,8 @@ main(int32_t argc, char **argv)
 		if ((enabled_port_mask & (1 << portid)) == 0)
 			continue;
 
-		port_init(portid);
+		sa_check_offloads(portid, &req_rx_offloads, &req_tx_offloads);
+		port_init(portid, req_rx_offloads, req_tx_offloads);
 	}
 
 	cryptodevs_init();
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index c998c8076..9b1586f52 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -146,6 +146,8 @@ struct ipsec_ctx {
 	struct rte_mempool *session_pool;
 	struct rte_mbuf *ol_pkts[MAX_PKT_BURST] __rte_aligned(sizeof(void *));
 	uint16_t ol_pkts_cnt;
+	uint64_t ipv4_offloads;
+	uint64_t ipv6_offloads;
 };
 
 struct cdev_key {
@@ -239,4 +241,8 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id);
 void
 rt_init(struct socket_ctx *ctx, int32_t socket_id);
 
+int
+sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
+		uint64_t *tx_offloads);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index d2d3550a4..f6271bc60 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -1017,3 +1017,38 @@ outbound_sa_lookup(struct sa_ctx *sa_ctx, uint32_t sa_idx[],
 	for (i = 0; i < nb_pkts; i++)
 		sa[i] = &sa_ctx->sa[sa_idx[i]];
 }
+
+/*
+ * Select HW offloads to be used.
+ */
+int
+sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
+		uint64_t *tx_offloads)
+{
+	struct ipsec_sa *rule;
+	uint32_t idx_sa;
+
+	*rx_offloads = 0;
+	*tx_offloads = 0;
+
+	/* Check for inbound rules that use offloads and use this port */
+	for (idx_sa = 0; idx_sa < nb_sa_in; idx_sa++) {
+		rule = &sa_in[idx_sa];
+		if ((rule->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				rule->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL)
+				&& rule->portid == port_id)
+			*rx_offloads |= DEV_RX_OFFLOAD_SECURITY;
+	}
+
+	/* Check for outbound rules that use offloads and use this port */
+	for (idx_sa = 0; idx_sa < nb_sa_out; idx_sa++) {
+		rule = &sa_out[idx_sa];
+		if ((rule->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				rule->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL)
+				&& rule->portid == port_id)
+			*tx_offloads |= DEV_TX_OFFLOAD_SECURITY;
+	}
+	return 0;
+}
-- 
2.17.1

[PATCH v7 00/10] examples/ipsec-secgw: make app to use ipsec library

[Konstantin Ananyev]

This patch series depends on the patch series:

ipsec: new library for IPsec data-path processing
http://patches.dpdk.org/patch/49410/
http://patches.dpdk.org/patch/49411/
http://patches.dpdk.org/patch/49412/
http://patches.dpdk.org/patch/49413/
http://patches.dpdk.org/patch/49414/
http://patches.dpdk.org/patch/49415/
http://patches.dpdk.org/patch/49416/
http://patches.dpdk.org/patch/49417/
http://patches.dpdk.org/patch/49418/
http://patches.dpdk.org/patch/49419/

to be applied first.

v6 -> v7
  Call drain_crypto_queue() for each processed port
  (reported/suggested by Akhil)
  Add ixgbe inline-ipsec test-case into functiona test scripts
  (for now ipv4 tunnel mode only)

v5 -> v6
  Address issues reported by Akhil:
     segfault when using lookaside-proto device
     HW IPv4 cksum offload not enabled by default
     crypto-dev dequeue() is called to often

v4 -> v5
- Address Akhil comments:
     documentation update
     spell checks spacing etc.
     introduce rxoffload/txoffload parameters
     single SA for ipv6
     update Makefile

v3 -> v4
 - fix few issues with the test scripts
 - update docs

v2 -> v3
 - add IPv6 cases into test scripts
 - fixes for IPv6 support
 - fixes for inline-crypto support
 - some code restructuring

v1 -> v2
 - Several bug fixes

That series contians few bug-fixes and changes to make ipsec-secgw
to utilize librte_ipsec library:
     - changes in the related data structures.
     - changes in the initialization code.
     - changes in the data-path code.
     - new command-line parameters to enable librte_ipsec codepath
       and related features.
     - test scripts to help automate ipsec-secgw functional testing.

Note that right now by default current (non-librte_ipsec) code-path
will be used. User has to run application with new command-line option
('-l')
to enable new codepath.
The main reason for that:
  - current librte_ipsec doesn't support all ipsec algorithms
    and features that the app does.
  - allow users to run both versions in parallel for some time
    to figure out any functional or performance degradation with the
    new code.

Test scripts were run with the following crypto devices:
 - aesni_mb
 - aesni_gcm
 - qat
 - ixgbe inline-ipsec (ipv4 tunnel mode)

Konstantin Ananyev (10):
  examples/ipsec-secgw: allow user to disable some RX/TX offloads
  examples/ipsec-secgw: allow to specify neighbour mac address
  examples/ipsec-secgw: fix crypto-op might never get dequeued
  examples/ipsec-secgw: fix outbound codepath for single SA
  examples/ipsec-secgw: make local variables static
  examples/ipsec-secgw: fix inbound SA checking
  examples/ipsec-secgw: make app to use ipsec library
  examples/ipsec-secgw: make data-path to use ipsec library
  examples/ipsec-secgw: add scripts for functional test
  doc: update ipsec-secgw guide and relelase notes

 doc/guides/rel_notes/release_19_02.rst        |  14 +
 doc/guides/sample_app_ug/ipsec_secgw.rst      | 159 +++++-
 examples/ipsec-secgw/Makefile                 |   5 +-
 examples/ipsec-secgw/ipsec-secgw.c            | 488 ++++++++++++++----
 examples/ipsec-secgw/ipsec.c                  | 101 ++--
 examples/ipsec-secgw/ipsec.h                  |  67 +++
 examples/ipsec-secgw/ipsec_process.c          | 357 +++++++++++++
 examples/ipsec-secgw/meson.build              |   6 +-
 examples/ipsec-secgw/parser.c                 |  91 ++++
 examples/ipsec-secgw/parser.h                 |   8 +-
 examples/ipsec-secgw/sa.c                     | 263 +++++++++-
 examples/ipsec-secgw/sp4.c                    |  35 +-
 examples/ipsec-secgw/sp6.c                    |  35 +-
 examples/ipsec-secgw/test/common_defs.sh      | 162 ++++++
 examples/ipsec-secgw/test/data_rxtx.sh        |  62 +++
 examples/ipsec-secgw/test/linux_test4.sh      |  63 +++
 examples/ipsec-secgw/test/linux_test6.sh      |  64 +++
 examples/ipsec-secgw/test/run_test.sh         |  80 +++
 .../test/trs_aescbc_sha1_common_defs.sh       |  69 +++
 .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  67 +++
 .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/trs_aescbc_sha1_esn_defs.sh          |  66 +++
 .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
 .../test/trs_aesgcm_common_defs.sh            |  60 +++
 examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  76 +++
 .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  66 +++
 .../test/trs_aesgcm_inline_crypto_defs.sh     |   6 +
 .../test/trs_aesgcm_inline_crypto_old_defs.sh |   5 +
 .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
 .../test/tun_aescbc_sha1_common_defs.sh       |  68 +++
 .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  70 +++
 .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/tun_aescbc_sha1_esn_defs.sh          |  70 +++
 .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
 .../test/tun_aesgcm_common_defs.sh            |  60 +++
 examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  80 +++
 .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  70 +++
 .../test/tun_aesgcm_inline_crypto_defs.sh     |   6 +
 .../test/tun_aesgcm_inline_crypto_old_defs.sh |   5 +
 .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
 42 files changed, 2785 insertions(+), 159 deletions(-)
 create mode 100644 examples/ipsec-secgw/ipsec_process.c
 create mode 100644 examples/ipsec-secgw/test/common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test4.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test6.sh
 create mode 100644 examples/ipsec-secgw/test/run_test.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh

-- 
2.17.1

Re: [PATCH v7 00/10] examples/ipsec-secgw: make app to use ipsec library

[Akhil Goyal]

On 1/9/2019 5:14 PM, Konstantin Ananyev wrote:
> This patch series depends on the patch series:
>
> ipsec: new library for IPsec data-path processing
> http://patches.dpdk.org/patch/49410/
> http://patches.dpdk.org/patch/49411/
> http://patches.dpdk.org/patch/49412/
> http://patches.dpdk.org/patch/49413/
> http://patches.dpdk.org/patch/49414/
> http://patches.dpdk.org/patch/49415/
> http://patches.dpdk.org/patch/49416/
> http://patches.dpdk.org/patch/49417/
> http://patches.dpdk.org/patch/49418/
> http://patches.dpdk.org/patch/49419/
>
> to be applied first.
>
> v6 -> v7
>    Call drain_crypto_queue() for each processed port
>    (reported/suggested by Akhil)
>    Add ixgbe inline-ipsec test-case into functiona test scripts
>    (for now ipv4 tunnel mode only)
There is some performance drop for NXP hardware if we use the new 
library mode for lookaside case which can be looked at a later stage 
when library mode is enabled by default and legacy mode code path is 
removed.

So for now, (probably a rebase will required over Fan's patchset)

Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
> v5 -> v6
>    Address issues reported by Akhil:
>       segfault when using lookaside-proto device
>       HW IPv4 cksum offload not enabled by default
>       crypto-dev dequeue() is called to often
>
> v4 -> v5
> - Address Akhil comments:
>       documentation update
>       spell checks spacing etc.
>       introduce rxoffload/txoffload parameters
>       single SA for ipv6
>       update Makefile
>
> v3 -> v4
>   - fix few issues with the test scripts
>   - update docs
>
> v2 -> v3
>   - add IPv6 cases into test scripts
>   - fixes for IPv6 support
>   - fixes for inline-crypto support
>   - some code restructuring
>
> v1 -> v2
>   - Several bug fixes
>
> That series contians few bug-fixes and changes to make ipsec-secgw
> to utilize librte_ipsec library:
>       - changes in the related data structures.
>       - changes in the initialization code.
>       - changes in the data-path code.
>       - new command-line parameters to enable librte_ipsec codepath
>         and related features.
>       - test scripts to help automate ipsec-secgw functional testing.
>
> Note that right now by default current (non-librte_ipsec) code-path
> will be used. User has to run application with new command-line option
> ('-l')
> to enable new codepath.
> The main reason for that:
>    - current librte_ipsec doesn't support all ipsec algorithms
>      and features that the app does.
>    - allow users to run both versions in parallel for some time
>      to figure out any functional or performance degradation with the
>      new code.
>
> Test scripts were run with the following crypto devices:
>   - aesni_mb
>   - aesni_gcm
>   - qat
>   - ixgbe inline-ipsec (ipv4 tunnel mode)
>
> Konstantin Ananyev (10):
>    examples/ipsec-secgw: allow user to disable some RX/TX offloads
>    examples/ipsec-secgw: allow to specify neighbour mac address
>    examples/ipsec-secgw: fix crypto-op might never get dequeued
>    examples/ipsec-secgw: fix outbound codepath for single SA
>    examples/ipsec-secgw: make local variables static
>    examples/ipsec-secgw: fix inbound SA checking
>    examples/ipsec-secgw: make app to use ipsec library
>    examples/ipsec-secgw: make data-path to use ipsec library
>    examples/ipsec-secgw: add scripts for functional test
>    doc: update ipsec-secgw guide and relelase notes
>
>   doc/guides/rel_notes/release_19_02.rst        |  14 +
>   doc/guides/sample_app_ug/ipsec_secgw.rst      | 159 +++++-
>   examples/ipsec-secgw/Makefile                 |   5 +-
>   examples/ipsec-secgw/ipsec-secgw.c            | 488 ++++++++++++++----
>   examples/ipsec-secgw/ipsec.c                  | 101 ++--
>   examples/ipsec-secgw/ipsec.h                  |  67 +++
>   examples/ipsec-secgw/ipsec_process.c          | 357 +++++++++++++
>   examples/ipsec-secgw/meson.build              |   6 +-
>   examples/ipsec-secgw/parser.c                 |  91 ++++
>   examples/ipsec-secgw/parser.h                 |   8 +-
>   examples/ipsec-secgw/sa.c                     | 263 +++++++++-
>   examples/ipsec-secgw/sp4.c                    |  35 +-
>   examples/ipsec-secgw/sp6.c                    |  35 +-
>   examples/ipsec-secgw/test/common_defs.sh      | 162 ++++++
>   examples/ipsec-secgw/test/data_rxtx.sh        |  62 +++
>   examples/ipsec-secgw/test/linux_test4.sh      |  63 +++
>   examples/ipsec-secgw/test/linux_test6.sh      |  64 +++
>   examples/ipsec-secgw/test/run_test.sh         |  80 +++
>   .../test/trs_aescbc_sha1_common_defs.sh       |  69 +++
>   .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  67 +++
>   .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
>   .../test/trs_aescbc_sha1_esn_defs.sh          |  66 +++
>   .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
>   .../test/trs_aesgcm_common_defs.sh            |  60 +++
>   examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  76 +++
>   .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
>   .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  66 +++
>   .../test/trs_aesgcm_inline_crypto_defs.sh     |   6 +
>   .../test/trs_aesgcm_inline_crypto_old_defs.sh |   5 +
>   .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
>   .../test/tun_aescbc_sha1_common_defs.sh       |  68 +++
>   .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  70 +++
>   .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
>   .../test/tun_aescbc_sha1_esn_defs.sh          |  70 +++
>   .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
>   .../test/tun_aesgcm_common_defs.sh            |  60 +++
>   examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  80 +++
>   .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
>   .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  70 +++
>   .../test/tun_aesgcm_inline_crypto_defs.sh     |   6 +
>   .../test/tun_aesgcm_inline_crypto_old_defs.sh |   5 +
>   .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
>   42 files changed, 2785 insertions(+), 159 deletions(-)
>   create mode 100644 examples/ipsec-secgw/ipsec_process.c
>   create mode 100644 examples/ipsec-secgw/test/common_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
>   create mode 100644 examples/ipsec-secgw/test/linux_test4.sh
>   create mode 100644 examples/ipsec-secgw/test/linux_test6.sh
>   create mode 100644 examples/ipsec-secgw/test/run_test.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_old_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_old_defs.sh
>   create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh
>

Re: [PATCH v7 00/10] examples/ipsec-secgw: make app to use ipsec library

[Xu, Yanjie]

The patch series latest versions be tested by yanjie xu,  which work for crypto and inline ipsec cases. 

-----Original Message-----
From: Ananyev, Konstantin 
Sent: Wednesday, January 9, 2019 7:44 PM
To: dev@dpdk.org
Cc: akhil.goyal@nxp.com; De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>; Ananyev, Konstantin <konstantin.ananyev@intel.com>
Subject: [PATCH v7 00/10] examples/ipsec-secgw: make app to use ipsec library

This patch series depends on the patch series:

ipsec: new library for IPsec data-path processing http://patches.dpdk.org/patch/49410/
http://patches.dpdk.org/patch/49411/
http://patches.dpdk.org/patch/49412/
http://patches.dpdk.org/patch/49413/
http://patches.dpdk.org/patch/49414/
http://patches.dpdk.org/patch/49415/
http://patches.dpdk.org/patch/49416/
http://patches.dpdk.org/patch/49417/
http://patches.dpdk.org/patch/49418/
http://patches.dpdk.org/patch/49419/

to be applied first.

v6 -> v7
  Call drain_crypto_queue() for each processed port
  (reported/suggested by Akhil)
  Add ixgbe inline-ipsec test-case into functiona test scripts
  (for now ipv4 tunnel mode only)

v5 -> v6
  Address issues reported by Akhil:
     segfault when using lookaside-proto device
     HW IPv4 cksum offload not enabled by default
     crypto-dev dequeue() is called to often

v4 -> v5
- Address Akhil comments:
     documentation update
     spell checks spacing etc.
     introduce rxoffload/txoffload parameters
     single SA for ipv6
     update Makefile

v3 -> v4
 - fix few issues with the test scripts
 - update docs

v2 -> v3
 - add IPv6 cases into test scripts
 - fixes for IPv6 support
 - fixes for inline-crypto support
 - some code restructuring

v1 -> v2
 - Several bug fixes

That series contians few bug-fixes and changes to make ipsec-secgw to utilize librte_ipsec library:
     - changes in the related data structures.
     - changes in the initialization code.
     - changes in the data-path code.
     - new command-line parameters to enable librte_ipsec codepath
       and related features.
     - test scripts to help automate ipsec-secgw functional testing.

Note that right now by default current (non-librte_ipsec) code-path will be used. User has to run application with new command-line option
('-l')
to enable new codepath.
The main reason for that:
  - current librte_ipsec doesn't support all ipsec algorithms
    and features that the app does.
  - allow users to run both versions in parallel for some time
    to figure out any functional or performance degradation with the
    new code.

Test scripts were run with the following crypto devices:
 - aesni_mb
 - aesni_gcm
 - qat
 - ixgbe inline-ipsec (ipv4 tunnel mode)

Konstantin Ananyev (10):
  examples/ipsec-secgw: allow user to disable some RX/TX offloads
  examples/ipsec-secgw: allow to specify neighbour mac address
  examples/ipsec-secgw: fix crypto-op might never get dequeued
  examples/ipsec-secgw: fix outbound codepath for single SA
  examples/ipsec-secgw: make local variables static
  examples/ipsec-secgw: fix inbound SA checking
  examples/ipsec-secgw: make app to use ipsec library
  examples/ipsec-secgw: make data-path to use ipsec library
  examples/ipsec-secgw: add scripts for functional test
  doc: update ipsec-secgw guide and relelase notes

 doc/guides/rel_notes/release_19_02.rst        |  14 +
 doc/guides/sample_app_ug/ipsec_secgw.rst      | 159 +++++-
 examples/ipsec-secgw/Makefile                 |   5 +-
 examples/ipsec-secgw/ipsec-secgw.c            | 488 ++++++++++++++----
 examples/ipsec-secgw/ipsec.c                  | 101 ++--
 examples/ipsec-secgw/ipsec.h                  |  67 +++
 examples/ipsec-secgw/ipsec_process.c          | 357 +++++++++++++
 examples/ipsec-secgw/meson.build              |   6 +-
 examples/ipsec-secgw/parser.c                 |  91 ++++
 examples/ipsec-secgw/parser.h                 |   8 +-
 examples/ipsec-secgw/sa.c                     | 263 +++++++++-
 examples/ipsec-secgw/sp4.c                    |  35 +-
 examples/ipsec-secgw/sp6.c                    |  35 +-
 examples/ipsec-secgw/test/common_defs.sh      | 162 ++++++
 examples/ipsec-secgw/test/data_rxtx.sh        |  62 +++
 examples/ipsec-secgw/test/linux_test4.sh      |  63 +++
 examples/ipsec-secgw/test/linux_test6.sh      |  64 +++
 examples/ipsec-secgw/test/run_test.sh         |  80 +++
 .../test/trs_aescbc_sha1_common_defs.sh       |  69 +++
 .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  67 +++
 .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/trs_aescbc_sha1_esn_defs.sh          |  66 +++
 .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
 .../test/trs_aesgcm_common_defs.sh            |  60 +++
 examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  76 +++
 .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  66 +++
 .../test/trs_aesgcm_inline_crypto_defs.sh     |   6 +
 .../test/trs_aesgcm_inline_crypto_old_defs.sh |   5 +
 .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
 .../test/tun_aescbc_sha1_common_defs.sh       |  68 +++
 .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  70 +++
 .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/tun_aescbc_sha1_esn_defs.sh          |  70 +++
 .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
 .../test/tun_aesgcm_common_defs.sh            |  60 +++
 examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  80 +++
 .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  70 +++
 .../test/tun_aesgcm_inline_crypto_defs.sh     |   6 +
 .../test/tun_aesgcm_inline_crypto_old_defs.sh |   5 +
 .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
 42 files changed, 2785 insertions(+), 159 deletions(-)  create mode 100644 examples/ipsec-secgw/ipsec_process.c
 create mode 100644 examples/ipsec-secgw/test/common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test4.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test6.sh
 create mode 100644 examples/ipsec-secgw/test/run_test.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh

--
2.17.1

[PATCH v7 01/10] examples/ipsec-secgw: allow user to disable some RX/TX offloads

[Konstantin Ananyev]

Right now ipsec-secgw always enables TX offloads
(DEV_TX_OFFLOAD_MULTI_SEGS, DEV_TX_OFFLOAD_SECURITY),
even when they are not requested by the config.
That causes many PMD to choose full-featured TX function,
which in many cases is much slower then one without offloads.
That patch adds ability for the user to disable unneeded HW offloads.
If DEV_TX_OFFLOAD_IPV4_CKSUM is disabled by user, then
SW version of ip cksum calculation is used.
That allows to use vector TX function, when inline-ipsec is not
requested.

Signed-off-by: Remy Horton <remy.horton@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 doc/guides/sample_app_ug/ipsec_secgw.rst |  12 +++
 examples/ipsec-secgw/ipsec-secgw.c       | 126 ++++++++++++++++++++---
 examples/ipsec-secgw/ipsec.h             |   6 ++
 examples/ipsec-secgw/sa.c                |  35 +++++++
 4 files changed, 164 insertions(+), 15 deletions(-)

diff --git a/doc/guides/sample_app_ug/ipsec_secgw.rst b/doc/guides/sample_app_ug/ipsec_secgw.rst
index 4869a011d..244bde2de 100644
--- a/doc/guides/sample_app_ug/ipsec_secgw.rst
+++ b/doc/guides/sample_app_ug/ipsec_secgw.rst
@@ -95,6 +95,8 @@ The application has a number of command line options::
                         -p PORTMASK -P -u PORTMASK -j FRAMESIZE
                         --config (port,queue,lcore)[,(port,queue,lcore]
                         --single-sa SAIDX
+                        --rxoffload MASK
+                        --txoffload MASK
                         -f CONFIG_FILE_PATH
 
 Where:
@@ -119,6 +121,16 @@ Where:
     on both Inbound and Outbound. This option is meant for debugging/performance
     purposes.
 
+*   ``--rxoffload MASK``: RX HW offload capabilities to enable/use on this port
+    (bitmask of DEV_RX_OFFLOAD_* values). It is an optional parameter and
+    allows user to disable some of the RX HW offload capabilities.
+    By default all HW RX offloads are enabled.
+
+*   ``--txoffload MASK``: TX HW offload capabilities to enable/use on this port
+    (bitmask of DEV_TX_OFFLOAD_* values). It is an optional parameter and
+    allows user to disable some of the TX HW offload capabilities.
+    By default all HW TX offloads are enabled.
+
 *   ``-f CONFIG_FILE_PATH``: the full path of text-based file containing all
     configuration items for running the application (See Configuration file
     syntax section below). ``-f CONFIG_FILE_PATH`` **must** be specified.
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 1bc0b5b50..4ccff38d2 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -124,6 +124,8 @@ struct ethaddr_info ethaddr_tbl[RTE_MAX_ETHPORTS] = {
 #define CMD_LINE_OPT_CONFIG		"config"
 #define CMD_LINE_OPT_SINGLE_SA		"single-sa"
 #define CMD_LINE_OPT_CRYPTODEV_MASK	"cryptodev_mask"
+#define CMD_LINE_OPT_RX_OFFLOAD		"rxoffload"
+#define CMD_LINE_OPT_TX_OFFLOAD		"txoffload"
 
 enum {
 	/* long options mapped to a short option */
@@ -135,12 +137,16 @@ enum {
 	CMD_LINE_OPT_CONFIG_NUM,
 	CMD_LINE_OPT_SINGLE_SA_NUM,
 	CMD_LINE_OPT_CRYPTODEV_MASK_NUM,
+	CMD_LINE_OPT_RX_OFFLOAD_NUM,
+	CMD_LINE_OPT_TX_OFFLOAD_NUM,
 };
 
 static const struct option lgopts[] = {
 	{CMD_LINE_OPT_CONFIG, 1, 0, CMD_LINE_OPT_CONFIG_NUM},
 	{CMD_LINE_OPT_SINGLE_SA, 1, 0, CMD_LINE_OPT_SINGLE_SA_NUM},
 	{CMD_LINE_OPT_CRYPTODEV_MASK, 1, 0, CMD_LINE_OPT_CRYPTODEV_MASK_NUM},
+	{CMD_LINE_OPT_RX_OFFLOAD, 1, 0, CMD_LINE_OPT_RX_OFFLOAD_NUM},
+	{CMD_LINE_OPT_TX_OFFLOAD, 1, 0, CMD_LINE_OPT_TX_OFFLOAD_NUM},
 	{NULL, 0, 0, 0}
 };
 
@@ -155,6 +161,13 @@ static uint32_t single_sa;
 static uint32_t single_sa_idx;
 static uint32_t frame_size;
 
+/*
+ * RX/TX HW offload capabilities to enable/use on ethernet ports.
+ * By default all capabilities are enabled.
+ */
+static uint64_t dev_rx_offload = UINT64_MAX;
+static uint64_t dev_tx_offload = UINT64_MAX;
+
 struct lcore_rx_queue {
 	uint16_t port_id;
 	uint8_t queue_id;
@@ -208,8 +221,6 @@ static struct rte_eth_conf port_conf = {
 	},
 	.txmode = {
 		.mq_mode = ETH_MQ_TX_NONE,
-		.offloads = (DEV_TX_OFFLOAD_IPV4_CKSUM |
-			     DEV_TX_OFFLOAD_MULTI_SEGS),
 	},
 };
 
@@ -315,7 +326,8 @@ prepare_traffic(struct rte_mbuf **pkts, struct ipsec_traffic *t,
 }
 
 static inline void
-prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
+prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port,
+		const struct lcore_conf *qconf)
 {
 	struct ip *ip;
 	struct ether_hdr *ethhdr;
@@ -325,14 +337,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
 	ethhdr = (struct ether_hdr *)rte_pktmbuf_prepend(pkt, ETHER_HDR_LEN);
 
 	if (ip->ip_v == IPVERSION) {
-		pkt->ol_flags |= PKT_TX_IP_CKSUM | PKT_TX_IPV4;
+		pkt->ol_flags |= qconf->outbound.ipv4_offloads;
 		pkt->l3_len = sizeof(struct ip);
 		pkt->l2_len = ETHER_HDR_LEN;
 
 		ip->ip_sum = 0;
+
+		/* calculate IPv4 cksum in SW */
+		if ((pkt->ol_flags & PKT_TX_IP_CKSUM) == 0)
+			ip->ip_sum = rte_ipv4_cksum((struct ipv4_hdr *)ip);
+
 		ethhdr->ether_type = rte_cpu_to_be_16(ETHER_TYPE_IPv4);
 	} else {
-		pkt->ol_flags |= PKT_TX_IPV6;
+		pkt->ol_flags |= qconf->outbound.ipv6_offloads;
 		pkt->l3_len = sizeof(struct ip6_hdr);
 		pkt->l2_len = ETHER_HDR_LEN;
 
@@ -346,18 +363,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
 }
 
 static inline void
-prepare_tx_burst(struct rte_mbuf *pkts[], uint16_t nb_pkts, uint16_t port)
+prepare_tx_burst(struct rte_mbuf *pkts[], uint16_t nb_pkts, uint16_t port,
+		const struct lcore_conf *qconf)
 {
 	int32_t i;
 	const int32_t prefetch_offset = 2;
 
 	for (i = 0; i < (nb_pkts - prefetch_offset); i++) {
 		rte_mbuf_prefetch_part2(pkts[i + prefetch_offset]);
-		prepare_tx_pkt(pkts[i], port);
+		prepare_tx_pkt(pkts[i], port, qconf);
 	}
 	/* Process left packets */
 	for (; i < nb_pkts; i++)
-		prepare_tx_pkt(pkts[i], port);
+		prepare_tx_pkt(pkts[i], port, qconf);
 }
 
 /* Send burst of packets on an output interface */
@@ -371,7 +389,7 @@ send_burst(struct lcore_conf *qconf, uint16_t n, uint16_t port)
 	queueid = qconf->tx_queue_id[port];
 	m_table = (struct rte_mbuf **)qconf->tx_mbufs[port].m_table;
 
-	prepare_tx_burst(m_table, n, port);
+	prepare_tx_burst(m_table, n, port, qconf);
 
 	ret = rte_eth_tx_burst(port, queueid, m_table, n);
 	if (unlikely(ret < n)) {
@@ -954,6 +972,8 @@ print_usage(const char *prgname)
 		" --config (port,queue,lcore)[,(port,queue,lcore)]"
 		" [--single-sa SAIDX]"
 		" [--cryptodev_mask MASK]"
+		" [--" CMD_LINE_OPT_RX_OFFLOAD " RX_OFFLOAD_MASK]"
+		" [--" CMD_LINE_OPT_TX_OFFLOAD " TX_OFFLOAD_MASK]"
 		"\n\n"
 		"  -p PORTMASK: Hexadecimal bitmask of ports to configure\n"
 		"  -P : Enable promiscuous mode\n"
@@ -966,10 +986,31 @@ print_usage(const char *prgname)
 		"                     bypassing the SP\n"
 		"  --cryptodev_mask MASK: Hexadecimal bitmask of the crypto\n"
 		"                         devices to configure\n"
+		"  --" CMD_LINE_OPT_RX_OFFLOAD
+		": bitmask of the RX HW offload capabilities to enable/use\n"
+		"                         (DEV_RX_OFFLOAD_*)\n"
+		"  --" CMD_LINE_OPT_TX_OFFLOAD
+		": bitmask of the TX HW offload capabilities to enable/use\n"
+		"                         (DEV_TX_OFFLOAD_*)\n"
 		"\n",
 		prgname);
 }
 
+static int
+parse_mask(const char *str, uint64_t *val)
+{
+	char *end;
+	unsigned long t;
+
+	errno = 0;
+	t = strtoul(str, &end, 0);
+	if (errno != 0 || end[0] != 0)
+		return -EINVAL;
+
+	*val = t;
+	return 0;
+}
+
 static int32_t
 parse_portmask(const char *portmask)
 {
@@ -1156,6 +1197,24 @@ parse_args(int32_t argc, char **argv)
 			/* else */
 			enabled_cryptodev_mask = ret;
 			break;
+		case CMD_LINE_OPT_RX_OFFLOAD_NUM:
+			ret = parse_mask(optarg, &dev_rx_offload);
+			if (ret != 0) {
+				printf("Invalid argument for \'%s\': %s\n",
+					CMD_LINE_OPT_RX_OFFLOAD, optarg);
+				print_usage(prgname);
+				return -1;
+			}
+			break;
+		case CMD_LINE_OPT_TX_OFFLOAD_NUM:
+			ret = parse_mask(optarg, &dev_tx_offload);
+			if (ret != 0) {
+				printf("Invalid argument for \'%s\': %s\n",
+					CMD_LINE_OPT_TX_OFFLOAD, optarg);
+				print_usage(prgname);
+				return -1;
+			}
+			break;
 		default:
 			print_usage(prgname);
 			return -1;
@@ -1543,7 +1602,7 @@ cryptodevs_init(void)
 }
 
 static void
-port_init(uint16_t portid)
+port_init(uint16_t portid, uint64_t req_rx_offloads, uint64_t req_tx_offloads)
 {
 	struct rte_eth_dev_info dev_info;
 	struct rte_eth_txconf *txconf;
@@ -1556,6 +1615,10 @@ port_init(uint16_t portid)
 
 	rte_eth_dev_info_get(portid, &dev_info);
 
+	/* limit allowed HW offloafs, as user requested */
+	dev_info.rx_offload_capa &= dev_rx_offload;
+	dev_info.tx_offload_capa &= dev_tx_offload;
+
 	printf("Configuring device port %u:\n", portid);
 
 	rte_eth_macaddr_get(portid, &ethaddr);
@@ -1584,14 +1647,38 @@ port_init(uint16_t portid)
 		local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_JUMBO_FRAME;
 	}
 
-	if (dev_info.rx_offload_capa & DEV_RX_OFFLOAD_SECURITY)
-		local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_SECURITY;
-	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_SECURITY)
-		local_port_conf.txmode.offloads |= DEV_TX_OFFLOAD_SECURITY;
+	local_port_conf.rxmode.offloads |= req_rx_offloads;
+	local_port_conf.txmode.offloads |= req_tx_offloads;
+
+	/* Check that all required capabilities are supported */
+	if ((local_port_conf.rxmode.offloads & dev_info.rx_offload_capa) !=
+			local_port_conf.rxmode.offloads)
+		rte_exit(EXIT_FAILURE,
+			"Error: port %u required RX offloads: 0x%" PRIx64
+			", avaialbe RX offloads: 0x%" PRIx64 "\n",
+			portid, local_port_conf.rxmode.offloads,
+			dev_info.rx_offload_capa);
+
+	if ((local_port_conf.txmode.offloads & dev_info.tx_offload_capa) !=
+			local_port_conf.txmode.offloads)
+		rte_exit(EXIT_FAILURE,
+			"Error: port %u required TX offloads: 0x%" PRIx64
+			", avaialbe TX offloads: 0x%" PRIx64 "\n",
+			portid, local_port_conf.txmode.offloads,
+			dev_info.tx_offload_capa);
+
 	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_MBUF_FAST_FREE)
 		local_port_conf.txmode.offloads |=
 			DEV_TX_OFFLOAD_MBUF_FAST_FREE;
 
+	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_IPV4_CKSUM)
+		local_port_conf.txmode.offloads |= DEV_TX_OFFLOAD_IPV4_CKSUM;
+
+	printf("port %u configurng rx_offloads=0x%" PRIx64
+		", tx_offloads=0x%" PRIx64 "\n",
+		portid, local_port_conf.rxmode.offloads,
+		local_port_conf.txmode.offloads);
+
 	local_port_conf.rx_adv_conf.rss_conf.rss_hf &=
 		dev_info.flow_type_rss_offloads;
 	if (local_port_conf.rx_adv_conf.rss_conf.rss_hf !=
@@ -1639,6 +1726,13 @@ port_init(uint16_t portid)
 
 		qconf = &lcore_conf[lcore_id];
 		qconf->tx_queue_id[portid] = tx_queueid;
+
+		/* Pre-populate pkt offloads based on capabilities */
+		qconf->outbound.ipv4_offloads = PKT_TX_IPV4;
+		qconf->outbound.ipv6_offloads = PKT_TX_IPV6;
+		if (local_port_conf.txmode.offloads & DEV_TX_OFFLOAD_IPV4_CKSUM)
+			qconf->outbound.ipv4_offloads |= PKT_TX_IP_CKSUM;
+
 		tx_queueid++;
 
 		/* init RX queues */
@@ -1749,6 +1843,7 @@ main(int32_t argc, char **argv)
 	uint32_t lcore_id;
 	uint8_t socket_id;
 	uint16_t portid;
+	uint64_t req_rx_offloads, req_tx_offloads;
 
 	/* init EAL */
 	ret = rte_eal_init(argc, argv);
@@ -1804,7 +1899,8 @@ main(int32_t argc, char **argv)
 		if ((enabled_port_mask & (1 << portid)) == 0)
 			continue;
 
-		port_init(portid);
+		sa_check_offloads(portid, &req_rx_offloads, &req_tx_offloads);
+		port_init(portid, req_rx_offloads, req_tx_offloads);
 	}
 
 	cryptodevs_init();
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index c998c8076..9b1586f52 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -146,6 +146,8 @@ struct ipsec_ctx {
 	struct rte_mempool *session_pool;
 	struct rte_mbuf *ol_pkts[MAX_PKT_BURST] __rte_aligned(sizeof(void *));
 	uint16_t ol_pkts_cnt;
+	uint64_t ipv4_offloads;
+	uint64_t ipv6_offloads;
 };
 
 struct cdev_key {
@@ -239,4 +241,8 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id);
 void
 rt_init(struct socket_ctx *ctx, int32_t socket_id);
 
+int
+sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
+		uint64_t *tx_offloads);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index d2d3550a4..f6271bc60 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -1017,3 +1017,38 @@ outbound_sa_lookup(struct sa_ctx *sa_ctx, uint32_t sa_idx[],
 	for (i = 0; i < nb_pkts; i++)
 		sa[i] = &sa_ctx->sa[sa_idx[i]];
 }
+
+/*
+ * Select HW offloads to be used.
+ */
+int
+sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
+		uint64_t *tx_offloads)
+{
+	struct ipsec_sa *rule;
+	uint32_t idx_sa;
+
+	*rx_offloads = 0;
+	*tx_offloads = 0;
+
+	/* Check for inbound rules that use offloads and use this port */
+	for (idx_sa = 0; idx_sa < nb_sa_in; idx_sa++) {
+		rule = &sa_in[idx_sa];
+		if ((rule->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				rule->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL)
+				&& rule->portid == port_id)
+			*rx_offloads |= DEV_RX_OFFLOAD_SECURITY;
+	}
+
+	/* Check for outbound rules that use offloads and use this port */
+	for (idx_sa = 0; idx_sa < nb_sa_out; idx_sa++) {
+		rule = &sa_out[idx_sa];
+		if ((rule->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				rule->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL)
+				&& rule->portid == port_id)
+			*tx_offloads |= DEV_TX_OFFLOAD_SECURITY;
+	}
+	return 0;
+}
-- 
2.17.1

[PATCH v8 00/10] examples/ipsec-secgw: make app to use ipsec library

[Konstantin Ananyev]

This patch series depends on the patch series:
ipsec: new library for IPsec data-path processing
to be applied first.

v7 -> v8
  rebase on top of crypto-next
  check-git-log nit-picks 

v6 -> v7
  Call drain_crypto_queue() for each processed port
  (reported/suggested by Akhil)
  Add ixgbe inline-ipsec test-case into functiona test scripts
  (for now ipv4 tunnel mode only)

v5 -> v6
  Address issues reported by Akhil:
     segfault when using lookaside-proto device
     HW IPv4 cksum offload not enabled by default
     crypto-dev dequeue() is called to often

v4 -> v5
- Address Akhil comments:
     documentation update
     spell checks spacing etc.
     introduce rxoffload/txoffload parameters
     single SA for ipv6
     update Makefile

v3 -> v4
 - fix few issues with the test scripts
 - update docs

v2 -> v3
 - add IPv6 cases into test scripts
 - fixes for IPv6 support
 - fixes for inline-crypto support
 - some code restructuring

v1 -> v2
 - Several bug fixes

That series contians few bug-fixes and changes to make ipsec-secgw
to utilize librte_ipsec library:
     - changes in the related data structures.
     - changes in the initialization code.
     - changes in the data-path code.
     - new command-line parameters to enable librte_ipsec codepath
       and related features.
     - test scripts to help automate ipsec-secgw functional testing.

Note that right now by default current (non-librte_ipsec) code-path
will be used. User has to run application with new command-line option
('-l')
to enable new codepath.
The main reason for that:
  - current librte_ipsec doesn't support all ipsec algorithms
    and features that the app does.
  - allow users to run both versions in parallel for some time
    to figure out any functional or performance degradation with the
    new code.

Test scripts were run with the following crypto devices:
 - aesni_mb
 - aesni_gcm
 - qat
 - ixgbe inline-ipsec (ipv4 tunnel mode)

Konstantin Ananyev (10):
  examples/ipsec-secgw: allow user to disable some Rx/Tx offloads
  examples/ipsec-secgw: allow to specify neighbour MAC address
  examples/ipsec-secgw: fix crypto-op might never get dequeued
  examples/ipsec-secgw: fix outbound codepath for single SA
  examples/ipsec-secgw: make local variables static
  examples/ipsec-secgw: fix inbound SA checking
  examples/ipsec-secgw: make app to use ipsec library
  examples/ipsec-secgw: make data-path to use ipsec library
  examples/ipsec-secgw: add scripts for functional test
  doc: update ipsec-secgw guide and relelase notes

 doc/guides/rel_notes/release_19_02.rst        |  14 +
 doc/guides/sample_app_ug/ipsec_secgw.rst      | 159 +++++-
 examples/ipsec-secgw/Makefile                 |   5 +-
 examples/ipsec-secgw/ipsec-secgw.c            | 488 ++++++++++++++----
 examples/ipsec-secgw/ipsec.c                  | 101 ++--
 examples/ipsec-secgw/ipsec.h                  |  67 +++
 examples/ipsec-secgw/ipsec_process.c          | 357 +++++++++++++
 examples/ipsec-secgw/meson.build              |   6 +-
 examples/ipsec-secgw/parser.c                 |  91 ++++
 examples/ipsec-secgw/parser.h                 |   8 +-
 examples/ipsec-secgw/sa.c                     | 263 +++++++++-
 examples/ipsec-secgw/sp4.c                    |  35 +-
 examples/ipsec-secgw/sp6.c                    |  35 +-
 examples/ipsec-secgw/test/common_defs.sh      | 162 ++++++
 examples/ipsec-secgw/test/data_rxtx.sh        |  62 +++
 examples/ipsec-secgw/test/linux_test4.sh      |  63 +++
 examples/ipsec-secgw/test/linux_test6.sh      |  64 +++
 examples/ipsec-secgw/test/run_test.sh         |  80 +++
 .../test/trs_aescbc_sha1_common_defs.sh       |  69 +++
 .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  67 +++
 .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/trs_aescbc_sha1_esn_defs.sh          |  66 +++
 .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
 .../test/trs_aesgcm_common_defs.sh            |  60 +++
 examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  76 +++
 .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  66 +++
 .../test/trs_aesgcm_inline_crypto_defs.sh     |   6 +
 .../test/trs_aesgcm_inline_crypto_old_defs.sh |   5 +
 .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
 .../test/tun_aescbc_sha1_common_defs.sh       |  68 +++
 .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  70 +++
 .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/tun_aescbc_sha1_esn_defs.sh          |  70 +++
 .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
 .../test/tun_aesgcm_common_defs.sh            |  60 +++
 examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  80 +++
 .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  70 +++
 .../test/tun_aesgcm_inline_crypto_defs.sh     |   6 +
 .../test/tun_aesgcm_inline_crypto_old_defs.sh |   5 +
 .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
 42 files changed, 2785 insertions(+), 159 deletions(-)
 create mode 100644 examples/ipsec-secgw/ipsec_process.c
 create mode 100644 examples/ipsec-secgw/test/common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test4.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test6.sh
 create mode 100644 examples/ipsec-secgw/test/run_test.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh

-- 
2.17.1

Re: [PATCH v8 00/10] examples/ipsec-secgw: make app to use ipsec library

[De Lara Guarch, Pablo]

> -----Original Message-----
> From: Ananyev, Konstantin
> Sent: Thursday, January 10, 2019 9:09 PM
> To: dev@dpdk.org
> Cc: akhil.goyal@nxp.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; thomas@monjalon.net; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>
> Subject: [PATCH v8 00/10] examples/ipsec-secgw: make app to use ipsec
> library
> 
> This patch series depends on the patch series:
> ipsec: new library for IPsec data-path processing to be applied first.
> 
> v7 -> v8
>   rebase on top of crypto-next
>   check-git-log nit-picks

Series applied to dpdk-next-crypto.

Thanks,
Pablo

Re: [PATCH v8 00/10] examples/ipsec-secgw: make app to use ipsec library

[Ananyev, Konstantin]

> 
> > -----Original Message-----
> > From: Ananyev, Konstantin
> > Sent: Thursday, January 10, 2019 9:09 PM
> > To: dev@dpdk.org
> > Cc: akhil.goyal@nxp.com; De Lara Guarch, Pablo
> > <pablo.de.lara.guarch@intel.com>; thomas@monjalon.net; Ananyev,
> > Konstantin <konstantin.ananyev@intel.com>
> > Subject: [PATCH v8 00/10] examples/ipsec-secgw: make app to use ipsec
> > library
> >
> > This patch series depends on the patch series:
> > ipsec: new library for IPsec data-path processing to be applied first.
> >
> > v7 -> v8
> >   rebase on top of crypto-next
> >   check-git-log nit-picks
> 
> Series applied to dpdk-next-crypto.
> 
> Thanks,
> Pablo

Thanks a lot Pablo.
Appreciated your help with it.
Konstantin

[PATCH v8 01/10] examples/ipsec-secgw: allow user to disable some Rx/Tx offloads

[Konstantin Ananyev]

Right now ipsec-secgw always enables TX offloads
(DEV_TX_OFFLOAD_MULTI_SEGS, DEV_TX_OFFLOAD_SECURITY),
even when they are not requested by the config.
That causes many PMD to choose full-featured TX function,
which in many cases is much slower then one without offloads.
That patch adds ability for the user to disable unneeded HW offloads.
If DEV_TX_OFFLOAD_IPV4_CKSUM is disabled by user, then
SW version of ip cksum calculation is used.
That allows to use vector TX function, when inline-ipsec is not
requested.

Signed-off-by: Remy Horton <remy.horton@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
---
 doc/guides/sample_app_ug/ipsec_secgw.rst |  12 +++
 examples/ipsec-secgw/ipsec-secgw.c       | 126 ++++++++++++++++++++---
 examples/ipsec-secgw/ipsec.h             |   6 ++
 examples/ipsec-secgw/sa.c                |  35 +++++++
 4 files changed, 164 insertions(+), 15 deletions(-)

diff --git a/doc/guides/sample_app_ug/ipsec_secgw.rst b/doc/guides/sample_app_ug/ipsec_secgw.rst
index 4869a011d..244bde2de 100644
--- a/doc/guides/sample_app_ug/ipsec_secgw.rst
+++ b/doc/guides/sample_app_ug/ipsec_secgw.rst
@@ -95,6 +95,8 @@ The application has a number of command line options::
                         -p PORTMASK -P -u PORTMASK -j FRAMESIZE
                         --config (port,queue,lcore)[,(port,queue,lcore]
                         --single-sa SAIDX
+                        --rxoffload MASK
+                        --txoffload MASK
                         -f CONFIG_FILE_PATH
 
 Where:
@@ -119,6 +121,16 @@ Where:
     on both Inbound and Outbound. This option is meant for debugging/performance
     purposes.
 
+*   ``--rxoffload MASK``: RX HW offload capabilities to enable/use on this port
+    (bitmask of DEV_RX_OFFLOAD_* values). It is an optional parameter and
+    allows user to disable some of the RX HW offload capabilities.
+    By default all HW RX offloads are enabled.
+
+*   ``--txoffload MASK``: TX HW offload capabilities to enable/use on this port
+    (bitmask of DEV_TX_OFFLOAD_* values). It is an optional parameter and
+    allows user to disable some of the TX HW offload capabilities.
+    By default all HW TX offloads are enabled.
+
 *   ``-f CONFIG_FILE_PATH``: the full path of text-based file containing all
     configuration items for running the application (See Configuration file
     syntax section below). ``-f CONFIG_FILE_PATH`` **must** be specified.
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index fc102a396..cd634e1d6 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -124,6 +124,8 @@ struct ethaddr_info ethaddr_tbl[RTE_MAX_ETHPORTS] = {
 #define CMD_LINE_OPT_CONFIG		"config"
 #define CMD_LINE_OPT_SINGLE_SA		"single-sa"
 #define CMD_LINE_OPT_CRYPTODEV_MASK	"cryptodev_mask"
+#define CMD_LINE_OPT_RX_OFFLOAD		"rxoffload"
+#define CMD_LINE_OPT_TX_OFFLOAD		"txoffload"
 
 enum {
 	/* long options mapped to a short option */
@@ -135,12 +137,16 @@ enum {
 	CMD_LINE_OPT_CONFIG_NUM,
 	CMD_LINE_OPT_SINGLE_SA_NUM,
 	CMD_LINE_OPT_CRYPTODEV_MASK_NUM,
+	CMD_LINE_OPT_RX_OFFLOAD_NUM,
+	CMD_LINE_OPT_TX_OFFLOAD_NUM,
 };
 
 static const struct option lgopts[] = {
 	{CMD_LINE_OPT_CONFIG, 1, 0, CMD_LINE_OPT_CONFIG_NUM},
 	{CMD_LINE_OPT_SINGLE_SA, 1, 0, CMD_LINE_OPT_SINGLE_SA_NUM},
 	{CMD_LINE_OPT_CRYPTODEV_MASK, 1, 0, CMD_LINE_OPT_CRYPTODEV_MASK_NUM},
+	{CMD_LINE_OPT_RX_OFFLOAD, 1, 0, CMD_LINE_OPT_RX_OFFLOAD_NUM},
+	{CMD_LINE_OPT_TX_OFFLOAD, 1, 0, CMD_LINE_OPT_TX_OFFLOAD_NUM},
 	{NULL, 0, 0, 0}
 };
 
@@ -155,6 +161,13 @@ static uint32_t single_sa;
 static uint32_t single_sa_idx;
 static uint32_t frame_size;
 
+/*
+ * RX/TX HW offload capabilities to enable/use on ethernet ports.
+ * By default all capabilities are enabled.
+ */
+static uint64_t dev_rx_offload = UINT64_MAX;
+static uint64_t dev_tx_offload = UINT64_MAX;
+
 struct lcore_rx_queue {
 	uint16_t port_id;
 	uint8_t queue_id;
@@ -208,8 +221,6 @@ static struct rte_eth_conf port_conf = {
 	},
 	.txmode = {
 		.mq_mode = ETH_MQ_TX_NONE,
-		.offloads = (DEV_TX_OFFLOAD_IPV4_CKSUM |
-			     DEV_TX_OFFLOAD_MULTI_SEGS),
 	},
 };
 
@@ -315,7 +326,8 @@ prepare_traffic(struct rte_mbuf **pkts, struct ipsec_traffic *t,
 }
 
 static inline void
-prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
+prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port,
+		const struct lcore_conf *qconf)
 {
 	struct ip *ip;
 	struct ether_hdr *ethhdr;
@@ -325,14 +337,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
 	ethhdr = (struct ether_hdr *)rte_pktmbuf_prepend(pkt, ETHER_HDR_LEN);
 
 	if (ip->ip_v == IPVERSION) {
-		pkt->ol_flags |= PKT_TX_IP_CKSUM | PKT_TX_IPV4;
+		pkt->ol_flags |= qconf->outbound.ipv4_offloads;
 		pkt->l3_len = sizeof(struct ip);
 		pkt->l2_len = ETHER_HDR_LEN;
 
 		ip->ip_sum = 0;
+
+		/* calculate IPv4 cksum in SW */
+		if ((pkt->ol_flags & PKT_TX_IP_CKSUM) == 0)
+			ip->ip_sum = rte_ipv4_cksum((struct ipv4_hdr *)ip);
+
 		ethhdr->ether_type = rte_cpu_to_be_16(ETHER_TYPE_IPv4);
 	} else {
-		pkt->ol_flags |= PKT_TX_IPV6;
+		pkt->ol_flags |= qconf->outbound.ipv6_offloads;
 		pkt->l3_len = sizeof(struct ip6_hdr);
 		pkt->l2_len = ETHER_HDR_LEN;
 
@@ -346,18 +363,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
 }
 
 static inline void
-prepare_tx_burst(struct rte_mbuf *pkts[], uint16_t nb_pkts, uint16_t port)
+prepare_tx_burst(struct rte_mbuf *pkts[], uint16_t nb_pkts, uint16_t port,
+		const struct lcore_conf *qconf)
 {
 	int32_t i;
 	const int32_t prefetch_offset = 2;
 
 	for (i = 0; i < (nb_pkts - prefetch_offset); i++) {
 		rte_mbuf_prefetch_part2(pkts[i + prefetch_offset]);
-		prepare_tx_pkt(pkts[i], port);
+		prepare_tx_pkt(pkts[i], port, qconf);
 	}
 	/* Process left packets */
 	for (; i < nb_pkts; i++)
-		prepare_tx_pkt(pkts[i], port);
+		prepare_tx_pkt(pkts[i], port, qconf);
 }
 
 /* Send burst of packets on an output interface */
@@ -371,7 +389,7 @@ send_burst(struct lcore_conf *qconf, uint16_t n, uint16_t port)
 	queueid = qconf->tx_queue_id[port];
 	m_table = (struct rte_mbuf **)qconf->tx_mbufs[port].m_table;
 
-	prepare_tx_burst(m_table, n, port);
+	prepare_tx_burst(m_table, n, port, qconf);
 
 	ret = rte_eth_tx_burst(port, queueid, m_table, n);
 	if (unlikely(ret < n)) {
@@ -958,6 +976,8 @@ print_usage(const char *prgname)
 		" --config (port,queue,lcore)[,(port,queue,lcore)]"
 		" [--single-sa SAIDX]"
 		" [--cryptodev_mask MASK]"
+		" [--" CMD_LINE_OPT_RX_OFFLOAD " RX_OFFLOAD_MASK]"
+		" [--" CMD_LINE_OPT_TX_OFFLOAD " TX_OFFLOAD_MASK]"
 		"\n\n"
 		"  -p PORTMASK: Hexadecimal bitmask of ports to configure\n"
 		"  -P : Enable promiscuous mode\n"
@@ -970,10 +990,31 @@ print_usage(const char *prgname)
 		"                     bypassing the SP\n"
 		"  --cryptodev_mask MASK: Hexadecimal bitmask of the crypto\n"
 		"                         devices to configure\n"
+		"  --" CMD_LINE_OPT_RX_OFFLOAD
+		": bitmask of the RX HW offload capabilities to enable/use\n"
+		"                         (DEV_RX_OFFLOAD_*)\n"
+		"  --" CMD_LINE_OPT_TX_OFFLOAD
+		": bitmask of the TX HW offload capabilities to enable/use\n"
+		"                         (DEV_TX_OFFLOAD_*)\n"
 		"\n",
 		prgname);
 }
 
+static int
+parse_mask(const char *str, uint64_t *val)
+{
+	char *end;
+	unsigned long t;
+
+	errno = 0;
+	t = strtoul(str, &end, 0);
+	if (errno != 0 || end[0] != 0)
+		return -EINVAL;
+
+	*val = t;
+	return 0;
+}
+
 static int32_t
 parse_portmask(const char *portmask)
 {
@@ -1160,6 +1201,24 @@ parse_args(int32_t argc, char **argv)
 			/* else */
 			enabled_cryptodev_mask = ret;
 			break;
+		case CMD_LINE_OPT_RX_OFFLOAD_NUM:
+			ret = parse_mask(optarg, &dev_rx_offload);
+			if (ret != 0) {
+				printf("Invalid argument for \'%s\': %s\n",
+					CMD_LINE_OPT_RX_OFFLOAD, optarg);
+				print_usage(prgname);
+				return -1;
+			}
+			break;
+		case CMD_LINE_OPT_TX_OFFLOAD_NUM:
+			ret = parse_mask(optarg, &dev_tx_offload);
+			if (ret != 0) {
+				printf("Invalid argument for \'%s\': %s\n",
+					CMD_LINE_OPT_TX_OFFLOAD, optarg);
+				print_usage(prgname);
+				return -1;
+			}
+			break;
 		default:
 			print_usage(prgname);
 			return -1;
@@ -1566,7 +1625,7 @@ cryptodevs_init(void)
 }
 
 static void
-port_init(uint16_t portid)
+port_init(uint16_t portid, uint64_t req_rx_offloads, uint64_t req_tx_offloads)
 {
 	struct rte_eth_dev_info dev_info;
 	struct rte_eth_txconf *txconf;
@@ -1579,6 +1638,10 @@ port_init(uint16_t portid)
 
 	rte_eth_dev_info_get(portid, &dev_info);
 
+	/* limit allowed HW offloafs, as user requested */
+	dev_info.rx_offload_capa &= dev_rx_offload;
+	dev_info.tx_offload_capa &= dev_tx_offload;
+
 	printf("Configuring device port %u:\n", portid);
 
 	rte_eth_macaddr_get(portid, &ethaddr);
@@ -1607,14 +1670,38 @@ port_init(uint16_t portid)
 		local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_JUMBO_FRAME;
 	}
 
-	if (dev_info.rx_offload_capa & DEV_RX_OFFLOAD_SECURITY)
-		local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_SECURITY;
-	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_SECURITY)
-		local_port_conf.txmode.offloads |= DEV_TX_OFFLOAD_SECURITY;
+	local_port_conf.rxmode.offloads |= req_rx_offloads;
+	local_port_conf.txmode.offloads |= req_tx_offloads;
+
+	/* Check that all required capabilities are supported */
+	if ((local_port_conf.rxmode.offloads & dev_info.rx_offload_capa) !=
+			local_port_conf.rxmode.offloads)
+		rte_exit(EXIT_FAILURE,
+			"Error: port %u required RX offloads: 0x%" PRIx64
+			", avaialbe RX offloads: 0x%" PRIx64 "\n",
+			portid, local_port_conf.rxmode.offloads,
+			dev_info.rx_offload_capa);
+
+	if ((local_port_conf.txmode.offloads & dev_info.tx_offload_capa) !=
+			local_port_conf.txmode.offloads)
+		rte_exit(EXIT_FAILURE,
+			"Error: port %u required TX offloads: 0x%" PRIx64
+			", avaialbe TX offloads: 0x%" PRIx64 "\n",
+			portid, local_port_conf.txmode.offloads,
+			dev_info.tx_offload_capa);
+
 	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_MBUF_FAST_FREE)
 		local_port_conf.txmode.offloads |=
 			DEV_TX_OFFLOAD_MBUF_FAST_FREE;
 
+	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_IPV4_CKSUM)
+		local_port_conf.txmode.offloads |= DEV_TX_OFFLOAD_IPV4_CKSUM;
+
+	printf("port %u configurng rx_offloads=0x%" PRIx64
+		", tx_offloads=0x%" PRIx64 "\n",
+		portid, local_port_conf.rxmode.offloads,
+		local_port_conf.txmode.offloads);
+
 	local_port_conf.rx_adv_conf.rss_conf.rss_hf &=
 		dev_info.flow_type_rss_offloads;
 	if (local_port_conf.rx_adv_conf.rss_conf.rss_hf !=
@@ -1662,6 +1749,13 @@ port_init(uint16_t portid)
 
 		qconf = &lcore_conf[lcore_id];
 		qconf->tx_queue_id[portid] = tx_queueid;
+
+		/* Pre-populate pkt offloads based on capabilities */
+		qconf->outbound.ipv4_offloads = PKT_TX_IPV4;
+		qconf->outbound.ipv6_offloads = PKT_TX_IPV6;
+		if (local_port_conf.txmode.offloads & DEV_TX_OFFLOAD_IPV4_CKSUM)
+			qconf->outbound.ipv4_offloads |= PKT_TX_IP_CKSUM;
+
 		tx_queueid++;
 
 		/* init RX queues */
@@ -1772,6 +1866,7 @@ main(int32_t argc, char **argv)
 	uint32_t lcore_id;
 	uint8_t socket_id;
 	uint16_t portid;
+	uint64_t req_rx_offloads, req_tx_offloads;
 
 	/* init EAL */
 	ret = rte_eal_init(argc, argv);
@@ -1827,7 +1922,8 @@ main(int32_t argc, char **argv)
 		if ((enabled_port_mask & (1 << portid)) == 0)
 			continue;
 
-		port_init(portid);
+		sa_check_offloads(portid, &req_rx_offloads, &req_tx_offloads);
+		port_init(portid, req_rx_offloads, req_tx_offloads);
 	}
 
 	cryptodevs_init();
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index f35552857..b4cbf1013 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -147,6 +147,8 @@ struct ipsec_ctx {
 	struct rte_mempool *session_priv_pool;
 	struct rte_mbuf *ol_pkts[MAX_PKT_BURST] __rte_aligned(sizeof(void *));
 	uint16_t ol_pkts_cnt;
+	uint64_t ipv4_offloads;
+	uint64_t ipv6_offloads;
 };
 
 struct cdev_key {
@@ -241,4 +243,8 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id);
 void
 rt_init(struct socket_ctx *ctx, int32_t socket_id);
 
+int
+sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
+		uint64_t *tx_offloads);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index d2d3550a4..f6271bc60 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -1017,3 +1017,38 @@ outbound_sa_lookup(struct sa_ctx *sa_ctx, uint32_t sa_idx[],
 	for (i = 0; i < nb_pkts; i++)
 		sa[i] = &sa_ctx->sa[sa_idx[i]];
 }
+
+/*
+ * Select HW offloads to be used.
+ */
+int
+sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
+		uint64_t *tx_offloads)
+{
+	struct ipsec_sa *rule;
+	uint32_t idx_sa;
+
+	*rx_offloads = 0;
+	*tx_offloads = 0;
+
+	/* Check for inbound rules that use offloads and use this port */
+	for (idx_sa = 0; idx_sa < nb_sa_in; idx_sa++) {
+		rule = &sa_in[idx_sa];
+		if ((rule->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				rule->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL)
+				&& rule->portid == port_id)
+			*rx_offloads |= DEV_RX_OFFLOAD_SECURITY;
+	}
+
+	/* Check for outbound rules that use offloads and use this port */
+	for (idx_sa = 0; idx_sa < nb_sa_out; idx_sa++) {
+		rule = &sa_out[idx_sa];
+		if ((rule->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				rule->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL)
+				&& rule->portid == port_id)
+			*tx_offloads |= DEV_TX_OFFLOAD_SECURITY;
+	}
+	return 0;
+}
-- 
2.17.1

[PATCH v8 02/10] examples/ipsec-secgw: allow to specify neighbour MAC address

[Konstantin Ananyev]

In some cases it is useful to allow user to specify destination
ether address for outgoing packets.
This patch adds such ability by introducing new 'neigh' config
file option.

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
---
 doc/guides/sample_app_ug/ipsec_secgw.rst | 42 ++++++++++-
 examples/ipsec-secgw/ipsec-secgw.c       | 21 ++++--
 examples/ipsec-secgw/ipsec.h             |  3 +
 examples/ipsec-secgw/parser.c            | 91 ++++++++++++++++++++++++
 examples/ipsec-secgw/parser.h            |  8 +--
 5 files changed, 154 insertions(+), 11 deletions(-)

diff --git a/doc/guides/sample_app_ug/ipsec_secgw.rst b/doc/guides/sample_app_ug/ipsec_secgw.rst
index 244bde2de..61638e733 100644
--- a/doc/guides/sample_app_ug/ipsec_secgw.rst
+++ b/doc/guides/sample_app_ug/ipsec_secgw.rst
@@ -217,7 +217,7 @@ Configurations
 --------------
 
 The following sections provide the syntax of configurations to initialize
-your SP, SA and Routing tables.
+your SP, SA, Routing and Neighbour tables.
 Configurations shall be specified in the configuration file to be passed to
 the application. The file is then parsed by the application. The successful
 parsing will result in the appropriate rules being applied to the tables
@@ -238,8 +238,8 @@ General rule syntax
 
 The parse treats one line in the configuration file as one configuration
 item (unless the line concatenation symbol exists). Every configuration
-item shall follow the syntax of either SP, SA, or Routing rules specified
-below.
+item shall follow the syntax of either SP, SA, Routing or Neighbour
+rules specified below.
 
 The configuration parser supports the following special symbols:
 
@@ -631,3 +631,39 @@ Example SP rules:
     rt ipv4 dst 172.16.1.5/32 port 0
 
     rt ipv6 dst 1111:1111:1111:1111:1111:1111:1111:5555/116 port 0
+
+Neighbour rule syntax
+^^^^^^^^^^^^^^^^^^^^^
+
+The Neighbour rule syntax is shown as follows:
+
+.. code-block:: console
+
+    neigh <port> <dst_mac>
+
+
+where each options means:
+
+``<port>``
+
+ * The output port id
+
+ * Optional: No
+
+ * Syntax: *port X*
+
+``<dst_mac>``
+
+ * The destination ethernet address to use for that port
+
+ * Optional: No
+
+ * Syntax:
+
+   * XX:XX:XX:XX:XX:XX
+
+Example Neighbour rules:
+
+.. code-block:: console
+
+    neigh port 0 DE:AD:BE:EF:01:02
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index cd634e1d6..0921b08d2 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -104,9 +104,9 @@ static uint16_t nb_txd = IPSEC_SECGW_TX_DESC_DEFAULT;
 #define ETHADDR(a, b, c, d, e, f) (__BYTES_TO_UINT64(a, b, c, d, e, f, 0, 0))
 
 #define ETHADDR_TO_UINT64(addr) __BYTES_TO_UINT64( \
-		addr.addr_bytes[0], addr.addr_bytes[1], \
-		addr.addr_bytes[2], addr.addr_bytes[3], \
-		addr.addr_bytes[4], addr.addr_bytes[5], \
+		(addr)->addr_bytes[0], (addr)->addr_bytes[1], \
+		(addr)->addr_bytes[2], (addr)->addr_bytes[3], \
+		(addr)->addr_bytes[4], (addr)->addr_bytes[5], \
 		0, 0)
 
 /* port/source ethernet addr and destination ethernet addr */
@@ -1246,6 +1246,19 @@ print_ethaddr(const char *name, const struct ether_addr *eth_addr)
 	printf("%s%s", name, buf);
 }
 
+/*
+ * Update destination ethaddr for the port.
+ */
+int
+add_dst_ethaddr(uint16_t port, const struct ether_addr *addr)
+{
+	if (port > RTE_DIM(ethaddr_tbl))
+		return -EINVAL;
+
+	ethaddr_tbl[port].dst = ETHADDR_TO_UINT64(addr);
+	return 0;
+}
+
 /* Check the link status of all ports in up to 9s, and print them finally */
 static void
 check_all_ports_link_status(uint32_t port_mask)
@@ -1645,7 +1658,7 @@ port_init(uint16_t portid, uint64_t req_rx_offloads, uint64_t req_tx_offloads)
 	printf("Configuring device port %u:\n", portid);
 
 	rte_eth_macaddr_get(portid, &ethaddr);
-	ethaddr_tbl[portid].src = ETHADDR_TO_UINT64(ethaddr);
+	ethaddr_tbl[portid].src = ETHADDR_TO_UINT64(&ethaddr);
 	print_ethaddr("Address: ", &ethaddr);
 	printf("\n");
 
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index b4cbf1013..311f116bb 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -247,4 +247,7 @@ int
 sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
 		uint64_t *tx_offloads);
 
+int
+add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/parser.c b/examples/ipsec-secgw/parser.c
index 91282ca94..b0a8ee23b 100644
--- a/examples/ipsec-secgw/parser.c
+++ b/examples/ipsec-secgw/parser.c
@@ -306,6 +306,46 @@ parse_range(const char *token, uint16_t *low, uint16_t *high)
 	return 0;
 }
 
+/*
+ * helper function for parse_mac, parse one section of the ether addr.
+ */
+static const char *
+parse_uint8x16(const char *s, uint8_t *v, uint8_t ls)
+{
+	char *end;
+	unsigned long t;
+
+	errno = 0;
+	t = strtoul(s, &end, 16);
+	if (errno != 0 || end[0] != ls || t > UINT8_MAX)
+		return NULL;
+	v[0] = t;
+	return end + 1;
+}
+
+static int
+parse_mac(const char *str, struct ether_addr *addr)
+{
+	uint32_t i;
+
+	static const uint8_t stop_sym[RTE_DIM(addr->addr_bytes)] = {
+		[0] = ':',
+		[1] = ':',
+		[2] = ':',
+		[3] = ':',
+		[4] = ':',
+		[5] = 0,
+	};
+
+	for (i = 0; i != RTE_DIM(addr->addr_bytes); i++) {
+		str = parse_uint8x16(str, addr->addr_bytes + i, stop_sym[i]);
+		if (str == NULL)
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
 /** sp add parse */
 struct cfg_sp_add_cfg_item {
 	cmdline_fixed_string_t sp_keyword;
@@ -444,11 +484,61 @@ cmdline_parse_inst_t cfg_rt_add_rule = {
 	},
 };
 
+/* neigh add parse */
+struct cfg_neigh_add_item {
+	cmdline_fixed_string_t neigh;
+	cmdline_fixed_string_t pstr;
+	uint16_t port;
+	cmdline_fixed_string_t mac;
+};
+
+static void
+cfg_parse_neigh(void *parsed_result, __rte_unused struct cmdline *cl,
+	void *data)
+{
+	int32_t rc;
+	struct cfg_neigh_add_item *res;
+	struct parse_status *st;
+	struct ether_addr mac;
+
+	st = data;
+	res = parsed_result;
+	rc = parse_mac(res->mac, &mac);
+	APP_CHECK(rc == 0, st, "invalid ether addr:%s", res->mac);
+	rc = add_dst_ethaddr(res->port, &mac);
+	APP_CHECK(rc == 0, st, "invalid port numer:%hu", res->port);
+	if (st->status < 0)
+		return;
+}
+
+cmdline_parse_token_string_t cfg_add_neigh_start =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, neigh, "neigh");
+cmdline_parse_token_string_t cfg_add_neigh_pstr =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, pstr, "port");
+cmdline_parse_token_num_t cfg_add_neigh_port =
+	TOKEN_NUM_INITIALIZER(struct cfg_neigh_add_item, port, UINT16);
+cmdline_parse_token_string_t cfg_add_neigh_mac =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, mac, NULL);
+
+cmdline_parse_inst_t cfg_neigh_add_rule = {
+	.f = cfg_parse_neigh,
+	.data = NULL,
+	.help_str = "",
+	.tokens = {
+		(void *)&cfg_add_neigh_start,
+		(void *)&cfg_add_neigh_pstr,
+		(void *)&cfg_add_neigh_port,
+		(void *)&cfg_add_neigh_mac,
+		NULL,
+	},
+};
+
 /** set of cfg items */
 cmdline_parse_ctx_t ipsec_ctx[] = {
 	(cmdline_parse_inst_t *)&cfg_sp_add_rule,
 	(cmdline_parse_inst_t *)&cfg_sa_add_rule,
 	(cmdline_parse_inst_t *)&cfg_rt_add_rule,
+	(cmdline_parse_inst_t *)&cfg_neigh_add_rule,
 	NULL,
 };
 
@@ -474,6 +564,7 @@ parse_cfg_file(const char *cfg_filename)
 	cfg_sp_add_rule.data = &status;
 	cfg_sa_add_rule.data = &status;
 	cfg_rt_add_rule.data = &status;
+	cfg_neigh_add_rule.data = &status;
 
 	do {
 		char oneline[1024];
diff --git a/examples/ipsec-secgw/parser.h b/examples/ipsec-secgw/parser.h
index be02537c5..6b8a10076 100644
--- a/examples/ipsec-secgw/parser.h
+++ b/examples/ipsec-secgw/parser.h
@@ -14,14 +14,14 @@ struct parse_status {
 	char parse_msg[256];
 };
 
-#define	APP_CHECK(exp, status, fmt, ...)				\
+#define	APP_CHECK(exp, st, fmt, ...)					\
 do {									\
 	if (!(exp)) {							\
-		sprintf(status->parse_msg, fmt "\n",			\
+		sprintf((st)->parse_msg, fmt "\n",			\
 			## __VA_ARGS__);				\
-		status->status = -1;					\
+		(st)->status = -1;					\
 	} else								\
-		status->status = 0;					\
+		(st)->status = 0;					\
 } while (0)
 
 #define APP_CHECK_PRESENCE(val, str, status)				\
-- 
2.17.1

[PATCH v8 03/10] examples/ipsec-secgw: fix crypto-op might never get dequeued

[Konstantin Ananyev]

In some cases crypto-ops could never be dequeued from the crypto-device.
The easiest way to reproduce:
start ipsec-secgw with crypto-dev and send to it less then 32 packets.
none packets will be forwarded.
Reason for that is that the application does dequeue() from crypto-queues
only when new packets arrive.
This patch makes the app to call dequeue() on a regular basis.
Also to make code cleaner and easier to understand,
it separates crypto-dev enqueue() and dequeue() code paths.
pkt_process() now only enqueues packets into crypto device,
dequeuing and final processing is done by drain_crypto_queues().

Fixes: c64278c0c18b ("examples/ipsec-secgw: rework processing loop")
Cc: stable@dpdk.org

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 144 ++++++++++++++++++++++++-----
 examples/ipsec-secgw/ipsec.c       |  99 +++++++++++++-------
 examples/ipsec-secgw/ipsec.h       |  11 +++
 3 files changed, 198 insertions(+), 56 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 0921b08d2..0c2005eea 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -469,38 +469,55 @@ inbound_sp_sa(struct sp_ctx *sp, struct sa_ctx *sa, struct traffic_type *ip,
 	ip->num = j;
 }
 
-static inline void
-process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
-		struct ipsec_traffic *traffic)
+static void
+split46_traffic(struct ipsec_traffic *trf, struct rte_mbuf *mb[], uint32_t num)
 {
+	uint32_t i, n4, n6;
+	struct ip *ip;
 	struct rte_mbuf *m;
-	uint16_t idx, nb_pkts_in, i, n_ip4, n_ip6;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
+	n4 = trf->ip4.num;
+	n6 = trf->ip6.num;
 
-	n_ip4 = traffic->ip4.num;
-	n_ip6 = traffic->ip6.num;
+	for (i = 0; i < num; i++) {
+
+		m = mb[i];
+		ip = rte_pktmbuf_mtod(m, struct ip *);
 
-	/* SP/ACL Inbound check ipsec and ip4 */
-	for (i = 0; i < nb_pkts_in; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
 		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-			traffic->ip4.data[idx] = rte_pktmbuf_mtod_offset(m,
+			trf->ip4.pkts[n4] = m;
+			trf->ip4.data[n4] = rte_pktmbuf_mtod_offset(m,
 					uint8_t *, offsetof(struct ip, ip_p));
+			n4++;
 		} else if (ip->ip_v == IP6_VERSION) {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
-			traffic->ip6.data[idx] = rte_pktmbuf_mtod_offset(m,
+			trf->ip6.pkts[n6] = m;
+			trf->ip6.data[n6] = rte_pktmbuf_mtod_offset(m,
 					uint8_t *,
 					offsetof(struct ip6_hdr, ip6_nxt));
+			n6++;
 		} else
 			rte_pktmbuf_free(m);
 	}
 
+	trf->ip4.num = n4;
+	trf->ip6.num = n6;
+}
+
+
+static inline void
+process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
+		struct ipsec_traffic *traffic)
+{
+	uint16_t nb_pkts_in, n_ip4, n_ip6;
+
+	n_ip4 = traffic->ip4.num;
+	n_ip6 = traffic->ip6.num;
+
+	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.num, MAX_PKT_BURST);
+
+	split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+
 	inbound_sp_sa(ipsec_ctx->sp4_ctx, ipsec_ctx->sa_ctx, &traffic->ip4,
 			n_ip4);
 
@@ -795,7 +812,7 @@ process_pkts(struct lcore_conf *qconf, struct rte_mbuf **pkts,
 }
 
 static inline void
-drain_buffers(struct lcore_conf *qconf)
+drain_tx_buffers(struct lcore_conf *qconf)
 {
 	struct buffer *buf;
 	uint32_t portid;
@@ -809,6 +826,81 @@ drain_buffers(struct lcore_conf *qconf)
 	}
 }
 
+static inline void
+drain_crypto_buffers(struct lcore_conf *qconf)
+{
+	uint32_t i;
+	struct ipsec_ctx *ctx;
+
+	/* drain inbound buffers*/
+	ctx = &qconf->inbound;
+	for (i = 0; i != ctx->nb_qps; i++) {
+		if (ctx->tbl[i].len != 0)
+			enqueue_cop_burst(ctx->tbl  + i);
+	}
+
+	/* drain outbound buffers*/
+	ctx = &qconf->outbound;
+	for (i = 0; i != ctx->nb_qps; i++) {
+		if (ctx->tbl[i].len != 0)
+			enqueue_cop_burst(ctx->tbl  + i);
+	}
+}
+
+static void
+drain_inbound_crypto_queues(const struct lcore_conf *qconf,
+		struct ipsec_ctx *ctx)
+{
+	uint32_t n;
+	struct ipsec_traffic trf;
+
+	/* dequeue packets from crypto-queue */
+	n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+			RTE_DIM(trf.ipsec.pkts));
+	if (n == 0)
+		return;
+
+	trf.ip4.num = 0;
+	trf.ip6.num = 0;
+
+	/* split traffic by ipv4-ipv6 */
+	split46_traffic(&trf, trf.ipsec.pkts, n);
+
+	/* process ipv4 packets */
+	inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
+	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+
+	/* process ipv6 packets */
+	inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
+	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+}
+
+static void
+drain_outbound_crypto_queues(const struct lcore_conf *qconf,
+		struct ipsec_ctx *ctx)
+{
+	uint32_t n;
+	struct ipsec_traffic trf;
+
+	/* dequeue packets from crypto-queue */
+	n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+			RTE_DIM(trf.ipsec.pkts));
+	if (n == 0)
+		return;
+
+	trf.ip4.num = 0;
+	trf.ip6.num = 0;
+
+	/* split traffic by ipv4-ipv6 */
+	split46_traffic(&trf, trf.ipsec.pkts, n);
+
+	/* process ipv4 packets */
+	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+
+	/* process ipv6 packets */
+	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+}
+
 /* main processing loop */
 static int32_t
 main_loop(__attribute__((unused)) void *dummy)
@@ -870,12 +962,14 @@ main_loop(__attribute__((unused)) void *dummy)
 		diff_tsc = cur_tsc - prev_tsc;
 
 		if (unlikely(diff_tsc > drain_tsc)) {
-			drain_buffers(qconf);
+			drain_tx_buffers(qconf);
+			drain_crypto_buffers(qconf);
 			prev_tsc = cur_tsc;
 		}
 
-		/* Read packet from RX queues */
 		for (i = 0; i < qconf->nb_rx_queue; ++i) {
+
+			/* Read packets from RX queues */
 			portid = rxql[i].port_id;
 			queueid = rxql[i].queue_id;
 			nb_rx = rte_eth_rx_burst(portid, queueid,
@@ -883,6 +977,14 @@ main_loop(__attribute__((unused)) void *dummy)
 
 			if (nb_rx > 0)
 				process_pkts(qconf, pkts, nb_rx, portid);
+
+			/* dequeue and process completed crypto-ops */
+			if (UNPROTECTED_PORT(portid))
+				drain_inbound_crypto_queues(qconf,
+					&qconf->inbound);
+			else
+				drain_outbound_crypto_queues(qconf,
+					&qconf->outbound);
 		}
 	}
 }
diff --git a/examples/ipsec-secgw/ipsec.c b/examples/ipsec-secgw/ipsec.c
index 9dc6e173c..c1469094a 100644
--- a/examples/ipsec-secgw/ipsec.c
+++ b/examples/ipsec-secgw/ipsec.c
@@ -333,33 +333,35 @@ create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa)
 	return 0;
 }
 
+/*
+ * queue crypto-ops into PMD queue.
+ */
+void
+enqueue_cop_burst(struct cdev_qp *cqp)
+{
+	uint32_t i, len, ret;
+
+	len = cqp->len;
+	ret = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp, cqp->buf, len);
+	if (ret < len) {
+		RTE_LOG_DP(DEBUG, IPSEC, "Cryptodev %u queue %u:"
+			" enqueued %u crypto ops out of %u\n",
+			cqp->id, cqp->qp, ret, len);
+			/* drop packets that we fail to enqueue */
+			for (i = ret; i < len; i++)
+				rte_pktmbuf_free(cqp->buf[i]->sym->m_src);
+	}
+	cqp->in_flight += ret;
+	cqp->len = 0;
+}
+
 static inline void
 enqueue_cop(struct cdev_qp *cqp, struct rte_crypto_op *cop)
 {
-	int32_t ret = 0, i;
-
 	cqp->buf[cqp->len++] = cop;
 
-	if (cqp->len == MAX_PKT_BURST) {
-		int enq_size = cqp->len;
-		if ((cqp->in_flight + enq_size) > MAX_INFLIGHT)
-			enq_size -=
-			    (int)((cqp->in_flight + enq_size) - MAX_INFLIGHT);
-
-		if (enq_size > 0)
-			ret = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp,
-					cqp->buf, enq_size);
-		if (ret < cqp->len) {
-			RTE_LOG_DP(DEBUG, IPSEC, "Cryptodev %u queue %u:"
-					" enqueued %u crypto ops out of %u\n",
-					 cqp->id, cqp->qp,
-					 ret, cqp->len);
-			for (i = ret; i < cqp->len; i++)
-				rte_pktmbuf_free(cqp->buf[i]->sym->m_src);
-		}
-		cqp->in_flight += ret;
-		cqp->len = 0;
-	}
+	if (cqp->len == MAX_PKT_BURST)
+		enqueue_cop_burst(cqp);
 }
 
 static inline void
@@ -473,6 +475,32 @@ ipsec_enqueue(ipsec_xform_fn xform_func, struct ipsec_ctx *ipsec_ctx,
 	}
 }
 
+static inline int32_t
+ipsec_inline_dequeue(ipsec_xform_fn xform_func, struct ipsec_ctx *ipsec_ctx,
+	      struct rte_mbuf *pkts[], uint16_t max_pkts)
+{
+	int32_t nb_pkts, ret;
+	struct ipsec_mbuf_metadata *priv;
+	struct ipsec_sa *sa;
+	struct rte_mbuf *pkt;
+
+	nb_pkts = 0;
+	while (ipsec_ctx->ol_pkts_cnt > 0 && nb_pkts < max_pkts) {
+		pkt = ipsec_ctx->ol_pkts[--ipsec_ctx->ol_pkts_cnt];
+		rte_prefetch0(pkt);
+		priv = get_priv(pkt);
+		sa = priv->sa;
+		ret = xform_func(pkt, sa, &priv->cop);
+		if (unlikely(ret)) {
+			rte_pktmbuf_free(pkt);
+			continue;
+		}
+		pkts[nb_pkts++] = pkt;
+	}
+
+	return nb_pkts;
+}
+
 static inline int
 ipsec_dequeue(ipsec_xform_fn xform_func, struct ipsec_ctx *ipsec_ctx,
 	      struct rte_mbuf *pkts[], uint16_t max_pkts)
@@ -490,19 +518,6 @@ ipsec_dequeue(ipsec_xform_fn xform_func, struct ipsec_ctx *ipsec_ctx,
 		if (ipsec_ctx->last_qp == ipsec_ctx->nb_qps)
 			ipsec_ctx->last_qp %= ipsec_ctx->nb_qps;
 
-		while (ipsec_ctx->ol_pkts_cnt > 0 && nb_pkts < max_pkts) {
-			pkt = ipsec_ctx->ol_pkts[--ipsec_ctx->ol_pkts_cnt];
-			rte_prefetch0(pkt);
-			priv = get_priv(pkt);
-			sa = priv->sa;
-			ret = xform_func(pkt, sa, &priv->cop);
-			if (unlikely(ret)) {
-				rte_pktmbuf_free(pkt);
-				continue;
-			}
-			pkts[nb_pkts++] = pkt;
-		}
-
 		if (cqp->in_flight == 0)
 			continue;
 
@@ -545,6 +560,13 @@ ipsec_inbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 
 	ipsec_enqueue(esp_inbound, ctx, pkts, sas, nb_pkts);
 
+	return ipsec_inline_dequeue(esp_inbound_post, ctx, pkts, len);
+}
+
+uint16_t
+ipsec_inbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len)
+{
 	return ipsec_dequeue(esp_inbound_post, ctx, pkts, len);
 }
 
@@ -558,5 +580,12 @@ ipsec_outbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 
 	ipsec_enqueue(esp_outbound, ctx, pkts, sas, nb_pkts);
 
+	return ipsec_inline_dequeue(esp_outbound_post, ctx, pkts, len);
+}
+
+uint16_t
+ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len)
+{
 	return ipsec_dequeue(esp_outbound_post, ctx, pkts, len);
 }
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 311f116bb..5fe54be93 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -186,6 +186,14 @@ uint16_t
 ipsec_outbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint32_t sa_idx[], uint16_t nb_pkts, uint16_t len);
 
+uint16_t
+ipsec_inbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len);
+
+uint16_t
+ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len);
+
 static inline uint16_t
 ipsec_metadata_size(void)
 {
@@ -250,4 +258,7 @@ sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
 int
 add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
 
+void
+enqueue_cop_burst(struct cdev_qp *cqp);
+
 #endif /* __IPSEC_H__ */
-- 
2.17.1

[PATCH v8 04/10] examples/ipsec-secgw: fix outbound codepath for single SA

[Konstantin Ananyev]

Looking at process_pkts_outbound_nosp() there seems few issues:
- accessing mbuf after it was freed
- invoking ipsec_outbound() for ipv4 packets only
- copying number of packets, but not the mbuf pointers itself

that patch provides fixes for that issues.

Fixes: 906257e965b7 ("examples/ipsec-secgw: support IPv6")
Cc: stable@dpdk.org

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 33 +++++++++++++++++++++---------
 1 file changed, 23 insertions(+), 10 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 0c2005eea..a5dfd1826 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -629,32 +629,45 @@ process_pkts_outbound_nosp(struct ipsec_ctx *ipsec_ctx,
 		struct ipsec_traffic *traffic)
 {
 	struct rte_mbuf *m;
-	uint32_t nb_pkts_out, i;
+	uint32_t nb_pkts_out, i, n;
 	struct ip *ip;
 
 	/* Drop any IPsec traffic from protected ports */
 	for (i = 0; i < traffic->ipsec.num; i++)
 		rte_pktmbuf_free(traffic->ipsec.pkts[i]);
 
-	traffic->ipsec.num = 0;
+	n = 0;
 
-	for (i = 0; i < traffic->ip4.num; i++)
-		traffic->ip4.res[i] = single_sa_idx;
+	for (i = 0; i < traffic->ip4.num; i++) {
+		traffic->ipsec.pkts[n] = traffic->ip4.pkts[i];
+		traffic->ipsec.res[n++] = single_sa_idx;
+	}
 
-	for (i = 0; i < traffic->ip6.num; i++)
-		traffic->ip6.res[i] = single_sa_idx;
+	for (i = 0; i < traffic->ip6.num; i++) {
+		traffic->ipsec.pkts[n] = traffic->ip6.pkts[i];
+		traffic->ipsec.res[n++] = single_sa_idx;
+	}
+
+	traffic->ip4.num = 0;
+	traffic->ip6.num = 0;
+	traffic->ipsec.num = n;
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ip4.pkts,
-			traffic->ip4.res, traffic->ip4.num,
+	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.res, traffic->ipsec.num,
 			MAX_PKT_BURST);
 
 	/* They all sue the same SA (ip4 or ip6 tunnel) */
 	m = traffic->ipsec.pkts[i];
 	ip = rte_pktmbuf_mtod(m, struct ip *);
-	if (ip->ip_v == IPVERSION)
+	if (ip->ip_v == IPVERSION) {
 		traffic->ip4.num = nb_pkts_out;
-	else
+		for (i = 0; i < nb_pkts_out; i++)
+			traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+	} else {
 		traffic->ip6.num = nb_pkts_out;
+		for (i = 0; i < nb_pkts_out; i++)
+			traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+	}
 }
 
 static inline int32_t
-- 
2.17.1

[PATCH v8 05/10] examples/ipsec-secgw: make local variables static

[Konstantin Ananyev]

in sp4.c and sp6.c there are few globals that used only locally.
Define them as static ones.

Cc: stable@dpdk.org

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
---
 examples/ipsec-secgw/sp4.c | 10 +++++-----
 examples/ipsec-secgw/sp6.c | 10 +++++-----
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/examples/ipsec-secgw/sp4.c b/examples/ipsec-secgw/sp4.c
index 8d3d3d8e0..6b05daaa9 100644
--- a/examples/ipsec-secgw/sp4.c
+++ b/examples/ipsec-secgw/sp4.c
@@ -44,7 +44,7 @@ enum {
 	RTE_ACL_IPV4_NUM
 };
 
-struct rte_acl_field_def ip4_defs[NUM_FIELDS_IPV4] = {
+static struct rte_acl_field_def ip4_defs[NUM_FIELDS_IPV4] = {
 	{
 	.type = RTE_ACL_FIELD_TYPE_BITMASK,
 	.size = sizeof(uint8_t),
@@ -85,11 +85,11 @@ struct rte_acl_field_def ip4_defs[NUM_FIELDS_IPV4] = {
 
 RTE_ACL_RULE_DEF(acl4_rules, RTE_DIM(ip4_defs));
 
-struct acl4_rules acl4_rules_out[MAX_ACL_RULE_NUM];
-uint32_t nb_acl4_rules_out;
+static struct acl4_rules acl4_rules_out[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl4_rules_out;
 
-struct acl4_rules acl4_rules_in[MAX_ACL_RULE_NUM];
-uint32_t nb_acl4_rules_in;
+static struct acl4_rules acl4_rules_in[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl4_rules_in;
 
 void
 parse_sp4_tokens(char **tokens, uint32_t n_tokens,
diff --git a/examples/ipsec-secgw/sp6.c b/examples/ipsec-secgw/sp6.c
index 6002afef3..dc5b94c6a 100644
--- a/examples/ipsec-secgw/sp6.c
+++ b/examples/ipsec-secgw/sp6.c
@@ -34,7 +34,7 @@ enum {
 
 #define IP6_ADDR_SIZE 16
 
-struct rte_acl_field_def ip6_defs[IP6_NUM] = {
+static struct rte_acl_field_def ip6_defs[IP6_NUM] = {
 	{
 	.type = RTE_ACL_FIELD_TYPE_BITMASK,
 	.size = sizeof(uint8_t),
@@ -116,11 +116,11 @@ struct rte_acl_field_def ip6_defs[IP6_NUM] = {
 
 RTE_ACL_RULE_DEF(acl6_rules, RTE_DIM(ip6_defs));
 
-struct acl6_rules acl6_rules_out[MAX_ACL_RULE_NUM];
-uint32_t nb_acl6_rules_out;
+static struct acl6_rules acl6_rules_out[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl6_rules_out;
 
-struct acl6_rules acl6_rules_in[MAX_ACL_RULE_NUM];
-uint32_t nb_acl6_rules_in;
+static struct acl6_rules acl6_rules_in[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl6_rules_in;
 
 void
 parse_sp6_tokens(char **tokens, uint32_t n_tokens,
-- 
2.17.1

[PATCH v8 06/10] examples/ipsec-secgw: fix inbound SA checking

[Konstantin Ananyev]

In the inbound_sa_check() make sure that sa pointer stored
inside mbuf private area is not NULL.

Fixes: d299106e8e31 ("examples/ipsec-secgw: add IPsec sample application")
Cc: stable@dpdk.org

Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/sa.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index f6271bc60..839aaca0c 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -947,10 +947,15 @@ int
 inbound_sa_check(struct sa_ctx *sa_ctx, struct rte_mbuf *m, uint32_t sa_idx)
 {
 	struct ipsec_mbuf_metadata *priv;
+	struct ipsec_sa *sa;
 
 	priv = get_priv(m);
+	sa = priv->sa;
+	if (sa != NULL)
+		return (sa_ctx->sa[sa_idx].spi == sa->spi);
 
-	return (sa_ctx->sa[sa_idx].spi == priv->sa->spi);
+	RTE_LOG(ERR, IPSEC, "SA not saved in private data\n");
+	return 0;
 }
 
 static inline void
-- 
2.17.1

[PATCH v8 07/10] examples/ipsec-secgw: make app to use ipsec library

[Konstantin Ananyev]

Changes to make ipsec-secgw to utilize librte_ipsec library.
That patch provides:
 - changes in the related data structures.
 - changes in the initialization code.
 - new command-line parameters to enable librte_ipsec codepath
   and related features.

Note that right now by default current (non-librte_ipsec) code-path will
be used. User has to run application with new command-line option ('-l')
to enable new codepath.
The main reason for that:
 - current librte_ipsec doesn't support all ipsec algorithms
   and features that the app does.
 - allow users to run both versions in parallel for some time
   to figure out any functional or performance degradation with the
   new code.

It is planned to deprecate and remove non-librte_ipsec code path
in future releases.

Signed-off-by: Mohammad Abdul Awal <mohammad.abdul.awal@intel.com>
Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
---
 examples/ipsec-secgw/Makefile      |   4 +-
 examples/ipsec-secgw/ipsec-secgw.c |  51 ++++++-
 examples/ipsec-secgw/ipsec.h       |  24 ++++
 examples/ipsec-secgw/meson.build   |   2 +-
 examples/ipsec-secgw/sa.c          | 221 ++++++++++++++++++++++++++++-
 examples/ipsec-secgw/sp4.c         |  25 ++++
 examples/ipsec-secgw/sp6.c         |  25 ++++
 7 files changed, 344 insertions(+), 8 deletions(-)

diff --git a/examples/ipsec-secgw/Makefile b/examples/ipsec-secgw/Makefile
index 02d41e39a..3918ee63e 100644
--- a/examples/ipsec-secgw/Makefile
+++ b/examples/ipsec-secgw/Makefile
@@ -61,8 +61,8 @@ RTE_TARGET ?= x86_64-native-linuxapp-gcc
 include $(RTE_SDK)/mk/rte.vars.mk
 
 ifneq ($(MAKECMDGOALS),clean)
-ifneq ($(CONFIG_RTE_LIBRTE_SECURITY),y)
-$(error "RTE_LIBRTE_SECURITY is required to build ipsec-secgw")
+ifneq ($(CONFIG_RTE_LIBRTE_IPSEC),y)
+$(error "RTE_LIBRTE_IPSEC is required to build ipsec-secgw")
 endif
 endif
 
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index a5dfd1826..f2254360c 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -168,6 +168,9 @@ static uint32_t frame_size;
 static uint64_t dev_rx_offload = UINT64_MAX;
 static uint64_t dev_tx_offload = UINT64_MAX;
 
+/* application wide librte_ipsec/SA parameters */
+struct app_sa_prm app_sa_prm = {.enable = 0};
+
 struct lcore_rx_queue {
 	uint16_t port_id;
 	uint8_t queue_id;
@@ -1087,6 +1090,10 @@ print_usage(const char *prgname)
 		" [-P]"
 		" [-u PORTMASK]"
 		" [-j FRAMESIZE]"
+		" [-l]"
+		" [-w REPLAY_WINDOW_SIZE]"
+		" [-e]"
+		" [-a]"
 		" -f CONFIG_FILE"
 		" --config (port,queue,lcore)[,(port,queue,lcore)]"
 		" [--single-sa SAIDX]"
@@ -1099,6 +1106,11 @@ print_usage(const char *prgname)
 		"  -u PORTMASK: Hexadecimal bitmask of unprotected ports\n"
 		"  -j FRAMESIZE: Enable jumbo frame with 'FRAMESIZE' as maximum\n"
 		"                packet size\n"
+		"  -l enables code-path that uses librte_ipsec\n"
+		"  -w REPLAY_WINDOW_SIZE specifies IPsec SQN replay window\n"
+		"     size for each SA\n"
+		"  -e enables ESN\n"
+		"  -a enables SA SQN atomic behaviour\n"
 		"  -f CONFIG_FILE: Configuration file\n"
 		"  --config (port,queue,lcore): Rx queue configuration\n"
 		"  --single-sa SAIDX: Use single SA index for outbound traffic,\n"
@@ -1216,6 +1228,20 @@ parse_config(const char *q_arg)
 	return 0;
 }
 
+static void
+print_app_sa_prm(const struct app_sa_prm *prm)
+{
+	printf("librte_ipsec usage: %s\n",
+		(prm->enable == 0) ? "disabled" : "enabled");
+
+	if (prm->enable == 0)
+		return;
+
+	printf("replay window size: %u\n", prm->window_size);
+	printf("ESN: %s\n", (prm->enable_esn == 0) ? "disabled" : "enabled");
+	printf("SA flags: %#" PRIx64 "\n", prm->flags);
+}
+
 static int32_t
 parse_args(int32_t argc, char **argv)
 {
@@ -1227,7 +1253,7 @@ parse_args(int32_t argc, char **argv)
 
 	argvopt = argv;
 
-	while ((opt = getopt_long(argc, argvopt, "p:Pu:f:j:",
+	while ((opt = getopt_long(argc, argvopt, "aelp:Pu:f:j:w:",
 				lgopts, &option_index)) != EOF) {
 
 		switch (opt) {
@@ -1283,6 +1309,21 @@ parse_args(int32_t argc, char **argv)
 			}
 			printf("Enabled jumbo frames size %u\n", frame_size);
 			break;
+		case 'l':
+			app_sa_prm.enable = 1;
+			break;
+		case 'w':
+			app_sa_prm.enable = 1;
+			app_sa_prm.window_size = parse_decimal(optarg);
+			break;
+		case 'e':
+			app_sa_prm.enable = 1;
+			app_sa_prm.enable_esn = 1;
+			break;
+		case 'a':
+			app_sa_prm.enable = 1;
+			app_sa_prm.flags |= RTE_IPSEC_SAFLAG_SQN_ATOM;
+			break;
 		case CMD_LINE_OPT_CONFIG_NUM:
 			ret = parse_config(optarg);
 			if (ret) {
@@ -1345,6 +1386,8 @@ parse_args(int32_t argc, char **argv)
 		return -1;
 	}
 
+	print_app_sa_prm(&app_sa_prm);
+
 	if (optind >= 0)
 		argv[optind-1] = prgname;
 
@@ -2035,12 +2078,14 @@ main(int32_t argc, char **argv)
 		if (socket_ctx[socket_id].mbuf_pool)
 			continue;
 
-		sa_init(&socket_ctx[socket_id], socket_id);
-
+		/* initilaze SPD */
 		sp4_init(&socket_ctx[socket_id], socket_id);
 
 		sp6_init(&socket_ctx[socket_id], socket_id);
 
+		/* initilaze SAD */
+		sa_init(&socket_ctx[socket_id], socket_id);
+
 		rt_init(&socket_ctx[socket_id], socket_id);
 
 		pool_init(&socket_ctx[socket_id], socket_id, NB_MBUF);
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 5fe54be93..1778dd75d 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -11,6 +11,7 @@
 #include <rte_crypto.h>
 #include <rte_security.h>
 #include <rte_flow.h>
+#include <rte_ipsec.h>
 
 #define RTE_LOGTYPE_IPSEC       RTE_LOGTYPE_USER1
 #define RTE_LOGTYPE_IPSEC_ESP   RTE_LOGTYPE_USER2
@@ -70,7 +71,20 @@ struct ip_addr {
 
 #define MAX_KEY_SIZE		32
 
+/*
+ * application wide SA parameters
+ */
+struct app_sa_prm {
+	uint32_t enable; /* use librte_ipsec API for ipsec pkt processing */
+	uint32_t window_size; /* replay window size */
+	uint32_t enable_esn;  /* enable/disable ESN support */
+	uint64_t flags;       /* rte_ipsec_sa_prm.flags */
+};
+
+extern struct app_sa_prm app_sa_prm;
+
 struct ipsec_sa {
+	struct rte_ipsec_session ips; /* one session per sa for now */
 	uint32_t spi;
 	uint32_t cdev_id_qp;
 	uint64_t seq;
@@ -245,6 +259,16 @@ sp4_init(struct socket_ctx *ctx, int32_t socket_id);
 void
 sp6_init(struct socket_ctx *ctx, int32_t socket_id);
 
+/*
+ * Search through SP rules for given SPI.
+ * Returns first rule index if found(greater or equal then zero),
+ * or -ENOENT otherwise.
+ */
+int
+sp4_spi_present(uint32_t spi, int inbound);
+int
+sp6_spi_present(uint32_t spi, int inbound);
+
 void
 sa_init(struct socket_ctx *ctx, int32_t socket_id);
 
diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
index 77d8b298f..31f68fee2 100644
--- a/examples/ipsec-secgw/meson.build
+++ b/examples/ipsec-secgw/meson.build
@@ -6,7 +6,7 @@
 # To build this example as a standalone application with an already-installed
 # DPDK instance, use 'make'
 
-deps += ['security', 'lpm', 'acl', 'hash']
+deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
 allow_experimental_apis = true
 sources = files(
 	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index 839aaca0c..414fcd26c 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -19,6 +19,7 @@
 #include <rte_ip.h>
 #include <rte_random.h>
 #include <rte_ethdev.h>
+#include <rte_malloc.h>
 
 #include "ipsec.h"
 #include "esp.h"
@@ -694,6 +695,7 @@ print_one_sa_rule(const struct ipsec_sa *sa, int inbound)
 }
 
 struct sa_ctx {
+	void *satbl; /* pointer to array of rte_ipsec_sa objects*/
 	struct ipsec_sa sa[IPSEC_SA_MAX_ENTRIES];
 	union {
 		struct {
@@ -764,7 +766,10 @@ sa_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 {
 	struct ipsec_sa *sa;
 	uint32_t i, idx;
-	uint16_t iv_length;
+	uint16_t iv_length, aad_length;
+
+	/* for ESN upper 32 bits of SQN also need to be part of AAD */
+	aad_length = (app_sa_prm.enable_esn != 0) ? sizeof(uint32_t) : 0;
 
 	for (i = 0; i < nb_entries; i++) {
 		idx = SPI2IDX(entries[i].spi);
@@ -808,7 +813,7 @@ sa_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 			sa_ctx->xf[idx].a.aead.iv.offset = IV_OFFSET;
 			sa_ctx->xf[idx].a.aead.iv.length = iv_length;
 			sa_ctx->xf[idx].a.aead.aad_length =
-				sa->aad_len;
+				sa->aad_len + aad_length;
 			sa_ctx->xf[idx].a.aead.digest_length =
 				sa->digest_len;
 
@@ -901,9 +906,205 @@ sa_in_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 	return sa_add_rules(sa_ctx, entries, nb_entries, 1);
 }
 
+/*
+ * helper function, fills parameters that are identical for all SAs
+ */
+static void
+fill_ipsec_app_sa_prm(struct rte_ipsec_sa_prm *prm,
+	const struct app_sa_prm *app_prm)
+{
+	memset(prm, 0, sizeof(*prm));
+
+	prm->flags = app_prm->flags;
+	prm->ipsec_xform.options.esn = app_prm->enable_esn;
+	prm->replay_win_sz = app_prm->window_size;
+}
+
+/*
+ * Helper function, tries to determine next_proto for SPI
+ * by searching though SP rules.
+ */
+static int
+get_spi_proto(uint32_t spi, enum rte_security_ipsec_sa_direction dir)
+{
+	int32_t rc4, rc6;
+
+	rc4 = sp4_spi_present(spi, dir == RTE_SECURITY_IPSEC_SA_DIR_INGRESS);
+	rc6 = sp6_spi_present(spi, dir == RTE_SECURITY_IPSEC_SA_DIR_INGRESS);
+
+	if (rc4 >= 0) {
+		if (rc6 >= 0) {
+			RTE_LOG(ERR, IPSEC,
+				"%s: SPI %u used simultaeously by "
+				"IPv4(%d) and IPv6 (%d) SP rules\n",
+				__func__, spi, rc4, rc6);
+			return -EINVAL;
+		} else
+			return IPPROTO_IPIP;
+	} else if (rc6 < 0) {
+		RTE_LOG(ERR, IPSEC,
+			"%s: SPI %u is not used by any SP rule\n",
+			__func__, spi);
+		return -EINVAL;
+	} else
+		return IPPROTO_IPV6;
+}
+
+static int
+fill_ipsec_sa_prm(struct rte_ipsec_sa_prm *prm, const struct ipsec_sa *ss,
+	const struct ipv4_hdr *v4, struct ipv6_hdr *v6)
+{
+	int32_t rc;
+
+	/*
+	 * Try to get SPI next proto by searching that SPI in SPD.
+	 * probably not the optimal way, but there seems nothing
+	 * better right now.
+	 */
+	rc = get_spi_proto(ss->spi, ss->direction);
+	if (rc < 0)
+		return rc;
+
+	fill_ipsec_app_sa_prm(prm, &app_sa_prm);
+	prm->userdata = (uintptr_t)ss;
+
+	/* setup ipsec xform */
+	prm->ipsec_xform.spi = ss->spi;
+	prm->ipsec_xform.salt = ss->salt;
+	prm->ipsec_xform.direction = ss->direction;
+	prm->ipsec_xform.proto = RTE_SECURITY_IPSEC_SA_PROTO_ESP;
+	prm->ipsec_xform.mode = (ss->flags == TRANSPORT) ?
+		RTE_SECURITY_IPSEC_SA_MODE_TRANSPORT :
+		RTE_SECURITY_IPSEC_SA_MODE_TUNNEL;
+
+	if (ss->flags == IP4_TUNNEL) {
+		prm->ipsec_xform.tunnel.type = RTE_SECURITY_IPSEC_TUNNEL_IPV4;
+		prm->tun.hdr_len = sizeof(*v4);
+		prm->tun.next_proto = rc;
+		prm->tun.hdr = v4;
+	} else if (ss->flags == IP6_TUNNEL) {
+		prm->ipsec_xform.tunnel.type = RTE_SECURITY_IPSEC_TUNNEL_IPV6;
+		prm->tun.hdr_len = sizeof(*v6);
+		prm->tun.next_proto = rc;
+		prm->tun.hdr = v6;
+	} else {
+		/* transport mode */
+		prm->trs.proto = rc;
+	}
+
+	/* setup crypto section */
+	prm->crypto_xform = ss->xforms;
+	return 0;
+}
+
+static void
+fill_ipsec_session(struct rte_ipsec_session *ss, struct rte_ipsec_sa *sa,
+	const struct ipsec_sa *lsa)
+{
+	ss->sa = sa;
+	ss->type = lsa->type;
+
+	/* setup crypto section */
+	if (ss->type == RTE_SECURITY_ACTION_TYPE_NONE) {
+		ss->crypto.ses = lsa->crypto_session;
+	/* setup session action type */
+	} else {
+		ss->security.ses = lsa->sec_session;
+		ss->security.ctx = lsa->security_ctx;
+		ss->security.ol_flags = lsa->ol_flags;
+	}
+}
+
+/*
+ * Initialise related rte_ipsec_sa object.
+ */
+static int
+ipsec_sa_init(struct ipsec_sa *lsa, struct rte_ipsec_sa *sa, uint32_t sa_size)
+{
+	int rc;
+	struct rte_ipsec_sa_prm prm;
+	struct ipv4_hdr v4  = {
+		.version_ihl = IPVERSION << 4 |
+			sizeof(v4) / IPV4_IHL_MULTIPLIER,
+		.time_to_live = IPDEFTTL,
+		.next_proto_id = IPPROTO_ESP,
+		.src_addr = lsa->src.ip.ip4,
+		.dst_addr = lsa->dst.ip.ip4,
+	};
+	struct ipv6_hdr v6 = {
+		.vtc_flow = htonl(IP6_VERSION << 28),
+		.proto = IPPROTO_ESP,
+	};
+
+	if (lsa->flags == IP6_TUNNEL) {
+		memcpy(v6.src_addr, lsa->src.ip.ip6.ip6_b, sizeof(v6.src_addr));
+		memcpy(v6.dst_addr, lsa->dst.ip.ip6.ip6_b, sizeof(v6.dst_addr));
+	}
+
+	rc = fill_ipsec_sa_prm(&prm, lsa, &v4, &v6);
+	if (rc == 0)
+		rc = rte_ipsec_sa_init(sa, &prm, sa_size);
+	if (rc < 0)
+		return rc;
+
+	fill_ipsec_session(&lsa->ips, sa, lsa);
+	return 0;
+}
+
+/*
+ * Allocate space and init rte_ipsec_sa strcutures,
+ * one per session.
+ */
+static int
+ipsec_satbl_init(struct sa_ctx *ctx, const struct ipsec_sa *ent,
+	uint32_t nb_ent, int32_t socket)
+{
+	int32_t rc, sz;
+	uint32_t i, idx;
+	size_t tsz;
+	struct rte_ipsec_sa *sa;
+	struct ipsec_sa *lsa;
+	struct rte_ipsec_sa_prm prm;
+
+	/* determine SA size */
+	idx = SPI2IDX(ent[0].spi);
+	fill_ipsec_sa_prm(&prm, ctx->sa + idx, NULL, NULL);
+	sz = rte_ipsec_sa_size(&prm);
+	if (sz < 0) {
+		RTE_LOG(ERR, IPSEC, "%s(%p, %u, %d): "
+			"failed to determine SA size, error code: %d\n",
+			__func__, ctx, nb_ent, socket, sz);
+		return sz;
+	}
+
+	tsz = sz * nb_ent;
+
+	ctx->satbl = rte_zmalloc_socket(NULL, tsz, RTE_CACHE_LINE_SIZE, socket);
+	if (ctx->satbl == NULL) {
+		RTE_LOG(ERR, IPSEC,
+			"%s(%p, %u, %d): failed to allocate %zu bytes\n",
+			__func__,  ctx, nb_ent, socket, tsz);
+		return -ENOMEM;
+	}
+
+	rc = 0;
+	for (i = 0; i != nb_ent && rc == 0; i++) {
+
+		idx = SPI2IDX(ent[i].spi);
+
+		sa = (struct rte_ipsec_sa *)((uintptr_t)ctx->satbl + sz * i);
+		lsa = ctx->sa + idx;
+
+		rc = ipsec_sa_init(lsa, sa, sz);
+	}
+
+	return rc;
+}
+
 void
 sa_init(struct socket_ctx *ctx, int32_t socket_id)
 {
+	int32_t rc;
 	const char *name;
 
 	if (ctx == NULL)
@@ -926,6 +1127,14 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id)
 				name, socket_id);
 
 		sa_in_add_rules(ctx->sa_in, sa_in, nb_sa_in);
+
+		if (app_sa_prm.enable != 0) {
+			rc = ipsec_satbl_init(ctx->sa_in, sa_in, nb_sa_in,
+				socket_id);
+			if (rc != 0)
+				rte_exit(EXIT_FAILURE,
+					"failed to init inbound SAs\n");
+		}
 	} else
 		RTE_LOG(WARNING, IPSEC, "No SA Inbound rule specified\n");
 
@@ -938,6 +1147,14 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id)
 				name, socket_id);
 
 		sa_out_add_rules(ctx->sa_out, sa_out, nb_sa_out);
+
+		if (app_sa_prm.enable != 0) {
+			rc = ipsec_satbl_init(ctx->sa_out, sa_out, nb_sa_out,
+				socket_id);
+			if (rc != 0)
+				rte_exit(EXIT_FAILURE,
+					"failed to init outbound SAs\n");
+		}
 	} else
 		RTE_LOG(WARNING, IPSEC, "No SA Outbound rule "
 			"specified\n");
diff --git a/examples/ipsec-secgw/sp4.c b/examples/ipsec-secgw/sp4.c
index 6b05daaa9..d1dc64bad 100644
--- a/examples/ipsec-secgw/sp4.c
+++ b/examples/ipsec-secgw/sp4.c
@@ -504,3 +504,28 @@ sp4_init(struct socket_ctx *ctx, int32_t socket_id)
 		RTE_LOG(WARNING, IPSEC, "No IPv4 SP Outbound rule "
 			"specified\n");
 }
+
+/*
+ * Search though SP rules for given SPI.
+ */
+int
+sp4_spi_present(uint32_t spi, int inbound)
+{
+	uint32_t i, num;
+	const struct acl4_rules *acr;
+
+	if (inbound != 0) {
+		acr = acl4_rules_in;
+		num = nb_acl4_rules_in;
+	} else {
+		acr = acl4_rules_out;
+		num = nb_acl4_rules_out;
+	}
+
+	for (i = 0; i != num; i++) {
+		if (acr[i].data.userdata == PROTECT(spi))
+			return i;
+	}
+
+	return -ENOENT;
+}
diff --git a/examples/ipsec-secgw/sp6.c b/examples/ipsec-secgw/sp6.c
index dc5b94c6a..e67d85aaf 100644
--- a/examples/ipsec-secgw/sp6.c
+++ b/examples/ipsec-secgw/sp6.c
@@ -618,3 +618,28 @@ sp6_init(struct socket_ctx *ctx, int32_t socket_id)
 		RTE_LOG(WARNING, IPSEC, "No IPv6 SP Outbound rule "
 			"specified\n");
 }
+
+/*
+ * Search though SP rules for given SPI.
+ */
+int
+sp6_spi_present(uint32_t spi, int inbound)
+{
+	uint32_t i, num;
+	const struct acl6_rules *acr;
+
+	if (inbound != 0) {
+		acr = acl6_rules_in;
+		num = nb_acl6_rules_in;
+	} else {
+		acr = acl6_rules_out;
+		num = nb_acl6_rules_out;
+	}
+
+	for (i = 0; i != num; i++) {
+		if (acr[i].data.userdata == PROTECT(spi))
+			return i;
+	}
+
+	return -ENOENT;
+}
-- 
2.17.1

[PATCH v8 08/10] examples/ipsec-secgw: make data-path to use ipsec library

[Konstantin Ananyev]

Changes to make ipsec-secgw data-path code to utilize librte_ipsec library.
Note that right now by default current (non-librte_ipsec) code-path will
be used. User has to run application with new command-line option ('-l')
to enable new codepath.

Signed-off-by: Mohammad Abdul Awal <mohammad.abdul.awal@intel.com>
Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
---
 examples/ipsec-secgw/Makefile        |   1 +
 examples/ipsec-secgw/ipsec-secgw.c   | 179 ++++++++------
 examples/ipsec-secgw/ipsec.c         |   2 +-
 examples/ipsec-secgw/ipsec.h         |  23 ++
 examples/ipsec-secgw/ipsec_process.c | 357 +++++++++++++++++++++++++++
 examples/ipsec-secgw/meson.build     |   4 +-
 6 files changed, 487 insertions(+), 79 deletions(-)
 create mode 100644 examples/ipsec-secgw/ipsec_process.c

diff --git a/examples/ipsec-secgw/Makefile b/examples/ipsec-secgw/Makefile
index 3918ee63e..08f474da6 100644
--- a/examples/ipsec-secgw/Makefile
+++ b/examples/ipsec-secgw/Makefile
@@ -13,6 +13,7 @@ SRCS-y += sp4.c
 SRCS-y += sp6.c
 SRCS-y += sa.c
 SRCS-y += rt.c
+SRCS-y += ipsec_process.c
 SRCS-y += ipsec-secgw.c
 
 CFLAGS += -gdwarf-2
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index f2254360c..8e7cd1b73 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -229,19 +229,6 @@ static struct rte_eth_conf port_conf = {
 
 static struct socket_ctx socket_ctx[NB_SOCKETS];
 
-struct traffic_type {
-	const uint8_t *data[MAX_PKT_BURST * 2];
-	struct rte_mbuf *pkts[MAX_PKT_BURST * 2];
-	uint32_t res[MAX_PKT_BURST * 2];
-	uint32_t num;
-};
-
-struct ipsec_traffic {
-	struct traffic_type ipsec;
-	struct traffic_type ip4;
-	struct traffic_type ip6;
-};
-
 static inline void
 prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 {
@@ -258,6 +245,8 @@ prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 			t->ip4.data[t->ip4.num] = nlp;
 			t->ip4.pkts[(t->ip4.num)++] = pkt;
 		}
+		pkt->l2_len = 0;
+		pkt->l3_len = sizeof(struct ip);
 	} else if (eth->ether_type == rte_cpu_to_be_16(ETHER_TYPE_IPv6)) {
 		nlp = (uint8_t *)rte_pktmbuf_adj(pkt, ETHER_HDR_LEN);
 		nlp = RTE_PTR_ADD(nlp, offsetof(struct ip6_hdr, ip6_nxt));
@@ -267,6 +256,8 @@ prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 			t->ip6.data[t->ip6.num] = nlp;
 			t->ip6.pkts[(t->ip6.num)++] = pkt;
 		}
+		pkt->l2_len = 0;
+		pkt->l3_len = sizeof(struct ip6_hdr);
 	} else {
 		/* Unknown/Unsupported type, drop the packet */
 		RTE_LOG(ERR, IPSEC, "Unsupported packet type\n");
@@ -516,10 +507,15 @@ process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
 	n_ip4 = traffic->ip4.num;
 	n_ip6 = traffic->ip6.num;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
-
-	split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+	if (app_sa_prm.enable == 0) {
+		nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.num, MAX_PKT_BURST);
+		split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+	} else {
+		inbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
+	}
 
 	inbound_sp_sa(ipsec_ctx->sp4_ctx, ipsec_ctx->sa_ctx, &traffic->ip4,
 			n_ip4);
@@ -575,20 +571,27 @@ process_pkts_outbound(struct ipsec_ctx *ipsec_ctx,
 
 	outbound_sp(ipsec_ctx->sp6_ctx, &traffic->ip6, &traffic->ipsec);
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.res, traffic->ipsec.num,
-			MAX_PKT_BURST);
-
-	for (i = 0; i < nb_pkts_out; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
-		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-		} else {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
+	if (app_sa_prm.enable == 0) {
+
+		nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.res, traffic->ipsec.num,
+				MAX_PKT_BURST);
+
+		for (i = 0; i < nb_pkts_out; i++) {
+			m = traffic->ipsec.pkts[i];
+			struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
+			if (ip->ip_v == IPVERSION) {
+				idx = traffic->ip4.num++;
+				traffic->ip4.pkts[idx] = m;
+			} else {
+				idx = traffic->ip6.num++;
+				traffic->ip6.pkts[idx] = m;
+			}
 		}
+	} else {
+		outbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.res,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -611,19 +614,26 @@ process_pkts_inbound_nosp(struct ipsec_ctx *ipsec_ctx,
 
 	traffic->ip6.num = 0;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
+	if (app_sa_prm.enable == 0) {
 
-	for (i = 0; i < nb_pkts_in; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
-		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-		} else {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
+		nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.num, MAX_PKT_BURST);
+
+		for (i = 0; i < nb_pkts_in; i++) {
+			m = traffic->ipsec.pkts[i];
+			struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
+			if (ip->ip_v == IPVERSION) {
+				idx = traffic->ip4.num++;
+				traffic->ip4.pkts[idx] = m;
+			} else {
+				idx = traffic->ip6.num++;
+				traffic->ip6.pkts[idx] = m;
+			}
 		}
+	} else {
+		inbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -655,21 +665,28 @@ process_pkts_outbound_nosp(struct ipsec_ctx *ipsec_ctx,
 	traffic->ip6.num = 0;
 	traffic->ipsec.num = n;
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.res, traffic->ipsec.num,
-			MAX_PKT_BURST);
+	if (app_sa_prm.enable == 0) {
 
-	/* They all sue the same SA (ip4 or ip6 tunnel) */
-	m = traffic->ipsec.pkts[i];
-	ip = rte_pktmbuf_mtod(m, struct ip *);
-	if (ip->ip_v == IPVERSION) {
-		traffic->ip4.num = nb_pkts_out;
-		for (i = 0; i < nb_pkts_out; i++)
-			traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+		nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.res, traffic->ipsec.num,
+				MAX_PKT_BURST);
+
+		/* They all sue the same SA (ip4 or ip6 tunnel) */
+		m = traffic->ipsec.pkts[0];
+		ip = rte_pktmbuf_mtod(m, struct ip *);
+		if (ip->ip_v == IPVERSION) {
+			traffic->ip4.num = nb_pkts_out;
+			for (i = 0; i < nb_pkts_out; i++)
+				traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+		} else {
+			traffic->ip6.num = nb_pkts_out;
+			for (i = 0; i < nb_pkts_out; i++)
+				traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+		}
 	} else {
-		traffic->ip6.num = nb_pkts_out;
-		for (i = 0; i < nb_pkts_out; i++)
-			traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+		outbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.res,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -870,25 +887,31 @@ drain_inbound_crypto_queues(const struct lcore_conf *qconf,
 	uint32_t n;
 	struct ipsec_traffic trf;
 
-	/* dequeue packets from crypto-queue */
-	n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+	if (app_sa_prm.enable == 0) {
+
+		/* dequeue packets from crypto-queue */
+		n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
 			RTE_DIM(trf.ipsec.pkts));
-	if (n == 0)
-		return;
 
-	trf.ip4.num = 0;
-	trf.ip6.num = 0;
+		trf.ip4.num = 0;
+		trf.ip6.num = 0;
 
-	/* split traffic by ipv4-ipv6 */
-	split46_traffic(&trf, trf.ipsec.pkts, n);
+		/* split traffic by ipv4-ipv6 */
+		split46_traffic(&trf, trf.ipsec.pkts, n);
+	} else
+		ipsec_cqp_process(ctx, &trf);
 
 	/* process ipv4 packets */
-	inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
-	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	if (trf.ip4.num != 0) {
+		inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
+		route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	}
 
 	/* process ipv6 packets */
-	inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
-	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	if (trf.ip6.num != 0) {
+		inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
+		route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	}
 }
 
 static void
@@ -898,23 +921,27 @@ drain_outbound_crypto_queues(const struct lcore_conf *qconf,
 	uint32_t n;
 	struct ipsec_traffic trf;
 
-	/* dequeue packets from crypto-queue */
-	n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+	if (app_sa_prm.enable == 0) {
+
+		/* dequeue packets from crypto-queue */
+		n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
 			RTE_DIM(trf.ipsec.pkts));
-	if (n == 0)
-		return;
 
-	trf.ip4.num = 0;
-	trf.ip6.num = 0;
+		trf.ip4.num = 0;
+		trf.ip6.num = 0;
 
-	/* split traffic by ipv4-ipv6 */
-	split46_traffic(&trf, trf.ipsec.pkts, n);
+		/* split traffic by ipv4-ipv6 */
+		split46_traffic(&trf, trf.ipsec.pkts, n);
+	} else
+		ipsec_cqp_process(ctx, &trf);
 
 	/* process ipv4 packets */
-	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	if (trf.ip4.num != 0)
+		route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
 
 	/* process ipv6 packets */
-	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	if (trf.ip6.num != 0)
+		route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
 }
 
 /* main processing loop */
diff --git a/examples/ipsec-secgw/ipsec.c b/examples/ipsec-secgw/ipsec.c
index c1469094a..4352cb842 100644
--- a/examples/ipsec-secgw/ipsec.c
+++ b/examples/ipsec-secgw/ipsec.c
@@ -39,7 +39,7 @@ set_ipsec_conf(struct ipsec_sa *sa, struct rte_security_ipsec_xform *ipsec)
 	ipsec->esn_soft_limit = IPSEC_OFFLOAD_ESN_SOFTLIMIT;
 }
 
-static inline int
+int
 create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa)
 {
 	struct rte_cryptodev_info cdev_info;
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 1778dd75d..99f49d65f 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -192,6 +192,20 @@ struct cnt_blk {
 	uint32_t cnt;
 } __attribute__((packed));
 
+struct traffic_type {
+	const uint8_t *data[MAX_PKT_BURST * 2];
+	struct rte_mbuf *pkts[MAX_PKT_BURST * 2];
+	struct ipsec_sa *saptr[MAX_PKT_BURST * 2];
+	uint32_t res[MAX_PKT_BURST * 2];
+	uint32_t num;
+};
+
+struct ipsec_traffic {
+	struct traffic_type ipsec;
+	struct traffic_type ip4;
+	struct traffic_type ip6;
+};
+
 uint16_t
 ipsec_inbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint16_t nb_pkts, uint16_t len);
@@ -208,6 +222,12 @@ uint16_t
 ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint16_t len);
 
+void
+ipsec_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf);
+
+void
+ipsec_cqp_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf);
+
 static inline uint16_t
 ipsec_metadata_size(void)
 {
@@ -285,4 +305,7 @@ add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
 void
 enqueue_cop_burst(struct cdev_qp *cqp);
 
+int
+create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/ipsec_process.c b/examples/ipsec-secgw/ipsec_process.c
new file mode 100644
index 000000000..e403c461a
--- /dev/null
+++ b/examples/ipsec-secgw/ipsec_process.c
@@ -0,0 +1,357 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2017 Intel Corporation
+ */
+#include <sys/types.h>
+#include <netinet/in.h>
+#include <netinet/ip.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_log.h>
+#include <rte_cryptodev.h>
+#include <rte_ethdev.h>
+#include <rte_mbuf.h>
+
+#include "ipsec.h"
+
+#define SATP_OUT_IPV4(t)	\
+	((((t) & RTE_IPSEC_SATP_MODE_MASK) == RTE_IPSEC_SATP_MODE_TRANS && \
+	(((t) & RTE_IPSEC_SATP_IPV_MASK) == RTE_IPSEC_SATP_IPV4)) || \
+	((t) & RTE_IPSEC_SATP_MODE_MASK) == RTE_IPSEC_SATP_MODE_TUNLV4)
+
+
+/* helper routine to free bulk of packets */
+static inline void
+free_pkts(struct rte_mbuf *mb[], uint32_t n)
+{
+	uint32_t i;
+
+	for (i = 0; i != n; i++)
+		rte_pktmbuf_free(mb[i]);
+}
+
+/* helper routine to free bulk of crypto-ops and related packets */
+static inline void
+free_cops(struct rte_crypto_op *cop[], uint32_t n)
+{
+	uint32_t i;
+
+	for (i = 0; i != n; i++)
+		rte_pktmbuf_free(cop[i]->sym->m_src);
+}
+
+/* helper routine to enqueue bulk of crypto ops */
+static inline void
+enqueue_cop_bulk(struct cdev_qp *cqp, struct rte_crypto_op *cop[], uint32_t num)
+{
+	uint32_t i, k, len, n;
+
+	len = cqp->len;
+
+	/*
+	 * if cqp is empty and we have enough ops,
+	 * then queue them to the PMD straightway.
+	 */
+	if (num >= RTE_DIM(cqp->buf) * 3 / 4 && len == 0) {
+		n = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp, cop, num);
+		cqp->in_flight += n;
+		free_cops(cop + n, num - n);
+		return;
+	}
+
+	k = 0;
+
+	do {
+		n = RTE_DIM(cqp->buf) - len;
+		n = RTE_MIN(num - k, n);
+
+		/* put packets into cqp */
+		for (i = 0; i != n; i++)
+			cqp->buf[len + i] = cop[k + i];
+
+		len += n;
+		k += n;
+
+		/* if cqp is full then, enqueue crypto-ops to PMD */
+		if (len == RTE_DIM(cqp->buf)) {
+			n = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp,
+					cqp->buf, len);
+			cqp->in_flight += n;
+			free_cops(cqp->buf + n, len - n);
+			len = 0;
+		}
+
+
+	} while (k != num);
+
+	cqp->len = len;
+}
+
+static inline int
+fill_ipsec_session(struct rte_ipsec_session *ss, struct ipsec_ctx *ctx,
+	struct ipsec_sa *sa)
+{
+	int32_t rc;
+
+	/* setup crypto section */
+	if (ss->type == RTE_SECURITY_ACTION_TYPE_NONE) {
+		if (sa->crypto_session == NULL) {
+			rc = create_session(ctx, sa);
+			if (rc != 0)
+				return rc;
+		}
+		ss->crypto.ses = sa->crypto_session;
+	/* setup session action type */
+	} else {
+		if (sa->sec_session == NULL) {
+			rc = create_session(ctx, sa);
+			if (rc != 0)
+				return rc;
+		}
+		ss->security.ses = sa->sec_session;
+		ss->security.ctx = sa->security_ctx;
+		ss->security.ol_flags = sa->ol_flags;
+	}
+
+	rc = rte_ipsec_session_prepare(ss);
+	if (rc != 0)
+		memset(ss, 0, sizeof(*ss));
+
+	return rc;
+}
+
+/*
+ * group input packets byt the SA they belong to.
+ */
+static uint32_t
+sa_group(struct ipsec_sa *sa_ptr[], struct rte_mbuf *pkts[],
+	struct rte_ipsec_group grp[], uint32_t num)
+{
+	uint32_t i, n, spi;
+	void *sa;
+	void * const nosa = &spi;
+
+	sa = nosa;
+	for (i = 0, n = 0; i != num; i++) {
+
+		if (sa != sa_ptr[i]) {
+			grp[n].cnt = pkts + i - grp[n].m;
+			n += (sa != nosa);
+			grp[n].id.ptr = sa_ptr[i];
+			grp[n].m = pkts + i;
+			sa = sa_ptr[i];
+		}
+	}
+
+	/* terminate last group */
+	if (sa != nosa) {
+		grp[n].cnt = pkts + i - grp[n].m;
+		n++;
+	}
+
+	return n;
+}
+
+/*
+ * helper function, splits processed packets into ipv4/ipv6 traffic.
+ */
+static inline void
+copy_to_trf(struct ipsec_traffic *trf, uint64_t satp, struct rte_mbuf *mb[],
+	uint32_t num)
+{
+	uint32_t j, ofs, s;
+	struct traffic_type *out;
+
+	/*
+	 * determine traffic type(ipv4/ipv6) and offset for ACL classify
+	 * based on SA type
+	 */
+	if ((satp & RTE_IPSEC_SATP_DIR_MASK) == RTE_IPSEC_SATP_DIR_IB) {
+		if ((satp & RTE_IPSEC_SATP_IPV_MASK) == RTE_IPSEC_SATP_IPV4) {
+			out = &trf->ip4;
+			ofs = offsetof(struct ip, ip_p);
+		} else {
+			out = &trf->ip6;
+			ofs = offsetof(struct ip6_hdr, ip6_nxt);
+		}
+	} else if (SATP_OUT_IPV4(satp)) {
+		out = &trf->ip4;
+		ofs = offsetof(struct ip, ip_p);
+	} else {
+		out = &trf->ip6;
+		ofs = offsetof(struct ip6_hdr, ip6_nxt);
+	}
+
+	for (j = 0, s = out->num; j != num; j++) {
+		out->data[s + j] = rte_pktmbuf_mtod_offset(mb[j],
+				void *, ofs);
+		out->pkts[s + j] = mb[j];
+	}
+
+	out->num += num;
+}
+
+/*
+ * Process ipsec packets.
+ * If packet belong to SA that is subject of inline-crypto,
+ * then process it immediately.
+ * Otherwise do necessary preparations and queue it to related
+ * crypto-dev queue.
+ */
+void
+ipsec_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf)
+{
+	uint64_t satp;
+	uint32_t i, j, k, n;
+	struct ipsec_sa *sa;
+	struct ipsec_mbuf_metadata *priv;
+	struct rte_ipsec_group *pg;
+	struct rte_ipsec_session *ips;
+	struct cdev_qp *cqp;
+	struct rte_crypto_op *cop[RTE_DIM(trf->ipsec.pkts)];
+	struct rte_ipsec_group grp[RTE_DIM(trf->ipsec.pkts)];
+
+	n = sa_group(trf->ipsec.saptr, trf->ipsec.pkts, grp, trf->ipsec.num);
+
+	for (i = 0; i != n; i++) {
+
+		pg = grp + i;
+		sa = pg->id.ptr;
+
+		/* no valid SA found */
+		if (sa == NULL)
+			k = 0;
+
+		ips = &sa->ips;
+		satp = rte_ipsec_sa_type(ips->sa);
+
+		/* no valid HW session for that SA, try to create one */
+		if (ips->crypto.ses == NULL &&
+				fill_ipsec_session(ips, ctx, sa) != 0)
+			k = 0;
+
+		/* process packets inline */
+		else if (sa->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				sa->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL) {
+
+			/*
+			 * This is just to satisfy inbound_sa_check()
+			 * and get_hop_for_offload_pkt().
+			 * Should be removed in future.
+			 */
+			for (j = 0; j != pg->cnt; j++) {
+				priv = get_priv(pg->m[j]);
+				priv->sa = sa;
+			}
+
+			k = rte_ipsec_pkt_process(ips, pg->m, pg->cnt);
+			copy_to_trf(trf, satp, pg->m, k);
+
+		/* enqueue packets to crypto dev */
+		} else {
+
+			cqp = &ctx->tbl[sa->cdev_id_qp];
+
+			/* for that app each mbuf has it's own crypto op */
+			for (j = 0; j != pg->cnt; j++) {
+				priv = get_priv(pg->m[j]);
+				cop[j] = &priv->cop;
+				/*
+				 * this is just to satisfy inbound_sa_check()
+				 * should be removed in future.
+				 */
+				priv->sa = sa;
+			}
+
+			/* prepare and enqueue crypto ops */
+			k = rte_ipsec_pkt_crypto_prepare(ips, pg->m, cop,
+				pg->cnt);
+			if (k != 0)
+				enqueue_cop_bulk(cqp, cop, k);
+		}
+
+		/* drop packets that cannot be enqueued/processed */
+		if (k != pg->cnt)
+			free_pkts(pg->m + k, pg->cnt - k);
+	}
+}
+
+static inline uint32_t
+cqp_dequeue(struct cdev_qp *cqp, struct rte_crypto_op *cop[], uint32_t num)
+{
+	uint32_t n;
+
+	if (cqp->in_flight == 0)
+		return 0;
+
+	n = rte_cryptodev_dequeue_burst(cqp->id, cqp->qp, cop, num);
+	RTE_ASSERT(cqp->in_flight >= n);
+	cqp->in_flight -= n;
+
+	return n;
+}
+
+static inline uint32_t
+ctx_dequeue(struct ipsec_ctx *ctx, struct rte_crypto_op *cop[], uint32_t num)
+{
+	uint32_t i, n;
+
+	n = 0;
+
+	for (i = ctx->last_qp; n != num && i != ctx->nb_qps; i++)
+		n += cqp_dequeue(ctx->tbl + i, cop + n, num - n);
+
+	for (i = 0; n != num && i != ctx->last_qp; i++)
+		n += cqp_dequeue(ctx->tbl + i, cop + n, num - n);
+
+	ctx->last_qp = i;
+	return n;
+}
+
+/*
+ * dequeue packets from crypto-queues and finalize processing.
+ */
+void
+ipsec_cqp_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf)
+{
+	uint64_t satp;
+	uint32_t i, k, n, ng;
+	struct rte_ipsec_session *ss;
+	struct traffic_type *out;
+	struct rte_ipsec_group *pg;
+	struct rte_crypto_op *cop[RTE_DIM(trf->ipsec.pkts)];
+	struct rte_ipsec_group grp[RTE_DIM(trf->ipsec.pkts)];
+
+	trf->ip4.num = 0;
+	trf->ip6.num = 0;
+
+	out = &trf->ipsec;
+
+	/* dequeue completed crypto-ops */
+	n = ctx_dequeue(ctx, cop, RTE_DIM(cop));
+	if (n == 0)
+		return;
+
+	/* group them by ipsec session */
+	ng = rte_ipsec_pkt_crypto_group((const struct rte_crypto_op **)
+		(uintptr_t)cop, out->pkts, grp, n);
+
+	/* process each group of packets */
+	for (i = 0; i != ng; i++) {
+
+		pg = grp + i;
+		ss = pg->id.ptr;
+		satp = rte_ipsec_sa_type(ss->sa);
+
+		k = rte_ipsec_pkt_process(ss, pg->m, pg->cnt);
+		copy_to_trf(trf, satp, pg->m, k);
+
+		/* free bad packets, if any */
+		free_pkts(pg->m + k, pg->cnt - k);
+
+		n -= pg->cnt;
+	}
+
+	/* we should never have packet with unknown SA here */
+	RTE_VERIFY(n == 0);
+}
diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
index 31f68fee2..81c146ebc 100644
--- a/examples/ipsec-secgw/meson.build
+++ b/examples/ipsec-secgw/meson.build
@@ -9,6 +9,6 @@
 deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
 allow_experimental_apis = true
 sources = files(
-	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
-	'rt.c', 'sa.c', 'sp4.c', 'sp6.c'
+	'esp.c', 'ipsec.c', 'ipsec_process.c', 'ipsec-secgw.c',
+	'parser.c', 'rt.c', 'sa.c', 'sp4.c', 'sp6.c'
 )
-- 
2.17.1

[PATCH v8 09/10] examples/ipsec-secgw: add scripts for functional test

[Konstantin Ananyev]

The purpose of these scripts is to automate ipsec-secgw functional testing.
The scripts require two machines (SUT and DUT) connected through
at least 2 NICs and running linux (so far tested only on Ubuntu 18.04).
Introduced test-cases for the following scenarios:
- Transport/Tunnel modes
- AES-CBC SHA1
- AES-GCM
- ESN on/off
- legacy/librte_ipsec code path

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
---
 examples/ipsec-secgw/test/common_defs.sh      | 162 ++++++++++++++++++
 examples/ipsec-secgw/test/data_rxtx.sh        |  62 +++++++
 examples/ipsec-secgw/test/linux_test4.sh      |  63 +++++++
 examples/ipsec-secgw/test/linux_test6.sh      |  64 +++++++
 examples/ipsec-secgw/test/run_test.sh         |  80 +++++++++
 .../test/trs_aescbc_sha1_common_defs.sh       |  69 ++++++++
 .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  67 ++++++++
 .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/trs_aescbc_sha1_esn_defs.sh          |  66 +++++++
 .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
 .../test/trs_aesgcm_common_defs.sh            |  60 +++++++
 examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  76 ++++++++
 .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  66 +++++++
 .../test/trs_aesgcm_inline_crypto_defs.sh     |   6 +
 .../test/trs_aesgcm_inline_crypto_old_defs.sh |   5 +
 .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
 .../test/tun_aescbc_sha1_common_defs.sh       |  68 ++++++++
 .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  70 ++++++++
 .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/tun_aescbc_sha1_esn_defs.sh          |  70 ++++++++
 .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
 .../test/tun_aesgcm_common_defs.sh            |  60 +++++++
 examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  80 +++++++++
 .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  70 ++++++++
 .../test/tun_aesgcm_inline_crypto_defs.sh     |   6 +
 .../test/tun_aesgcm_inline_crypto_old_defs.sh |   5 +
 .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
 29 files changed, 1315 insertions(+)
 create mode 100644 examples/ipsec-secgw/test/common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test4.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test6.sh
 create mode 100644 examples/ipsec-secgw/test/run_test.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh

diff --git a/examples/ipsec-secgw/test/common_defs.sh b/examples/ipsec-secgw/test/common_defs.sh
new file mode 100644
index 000000000..1ed31f89f
--- /dev/null
+++ b/examples/ipsec-secgw/test/common_defs.sh
@@ -0,0 +1,162 @@
+#! /bin/bash
+
+#check that env vars are properly defined
+
+#check SGW_PATH
+if [[ -z "${SGW_PATH}" || ! -x ${SGW_PATH} ]]; then
+	echo "SGW_PATH is invalid"
+	exit 127
+fi
+
+#check ETH_DEV
+if [[ -z "${ETH_DEV}" ]]; then
+	echo "ETH_DEV is invalid"
+	exit 127
+fi
+
+#setup SGW_LCORE
+SGW_LCORE=${SGW_LCORE:-0}
+
+#check that REMOTE_HOST is reachable
+ssh ${REMOTE_HOST} echo
+st=$?
+if [[ $st -ne 0 ]]; then
+	echo "host ${REMOTE_HOST} is not reachable"
+	exit $st
+fi
+
+#get ether addr of REMOTE_HOST
+REMOTE_MAC=`ssh ${REMOTE_HOST} ip addr show dev ${REMOTE_IFACE}`
+st=$?
+REMOTE_MAC=`echo ${REMOTE_MAC} | sed -e 's/^.*ether //' -e 's/ brd.*$//'`
+if [[ $st -ne 0 || -z "${REMOTE_MAC}" ]]; then
+	echo "coouldn't retrieve ether addr from ${REMOTE_IFACE}"
+	exit 127
+fi
+
+LOCAL_IFACE=dtap0
+
+LOCAL_MAC="00:64:74:61:70:30"
+
+REMOTE_IPV4=192.168.31.14
+LOCAL_IPV4=192.168.31.92
+
+REMOTE_IPV6=fd12:3456:789a:0031:0000:0000:0000:0014
+LOCAL_IPV6=fd12:3456:789a:0031:0000:0000:0000:0092
+
+DPDK_PATH=${RTE_SDK:-${PWD}}
+DPDK_BUILD=${RTE_TARGET:-x86_64-native-linuxapp-gcc}
+
+SGW_OUT_FILE=./ipsec-secgw.out1
+
+SGW_CMD_EAL_PRM="--lcores=${SGW_LCORE} -n 4 ${ETH_DEV}"
+SGW_CMD_CFG="(0,0,${SGW_LCORE}),(1,0,${SGW_LCORE})"
+SGW_CMD_PRM="-p 0x3 -u 1 -P --config=\"${SGW_CMD_CFG}\""
+
+SGW_CFG_FILE=$(tempfile)
+
+# configure local host/ifaces
+config_local_iface()
+{
+	ifconfig ${LOCAL_IFACE} ${LOCAL_IPV4}/24 mtu 1400 up
+	ifconfig ${LOCAL_IFACE}
+
+	ip neigh flush dev ${LOCAL_IFACE}
+	ip neigh add ${REMOTE_IPV4} dev ${LOCAL_IFACE} lladdr ${REMOTE_MAC}
+	ip neigh show dev ${LOCAL_IFACE}
+}
+
+config6_local_iface()
+{
+	config_local_iface
+
+	sysctl -w net.ipv6.conf.${LOCAL_IFACE}.disable_ipv6=0
+	ip addr add  ${LOCAL_IPV6}/64 dev ${LOCAL_IFACE}
+
+	sysctl -w net.ipv6.conf.${LOCAL_IFACE}.mtu=1300
+
+	ip -6 neigh add ${REMOTE_IPV6} dev ${LOCAL_IFACE} lladdr ${REMOTE_MAC}
+	ip neigh show dev ${LOCAL_IFACE}
+}
+
+#configure remote host/iface
+config_remote_iface()
+{
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE} down
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE} ${REMOTE_IPV4}/24 up
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} ip neigh flush dev ${REMOTE_IFACE}
+
+	# by some reason following ip neigh doesn't work for me here properly:
+	#ssh ${REMOTE_HOST} ip neigh add ${LOCAL_IPV4} \
+	#		dev ${REMOTE_IFACE} lladr ${LOCAL_MAC}
+	# so used arp instead.
+	ssh ${REMOTE_HOST} arp -i ${REMOTE_IFACE} -s ${LOCAL_IPV4} ${LOCAL_MAC}
+	ssh ${REMOTE_HOST} ip neigh show dev ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} iptables --flush
+}
+
+config6_remote_iface()
+{
+	config_remote_iface
+
+	ssh ${REMOTE_HOST} sysctl -w \
+		net.ipv6.conf.${REMOTE_IFACE}.disable_ipv6=0
+	ssh ${REMOTE_HOST} ip addr add  ${REMOTE_IPV6}/64 dev ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} ip -6 neigh add ${LOCAL_IPV6} \
+		dev ${REMOTE_IFACE} lladdr ${LOCAL_MAC}
+	ssh ${REMOTE_HOST} ip neigh show dev ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} ip6tables --flush
+}
+
+#configure remote and local host/iface
+config_iface()
+{
+	config_local_iface
+	config_remote_iface
+}
+
+config6_iface()
+{
+	config6_local_iface
+	config6_remote_iface
+}
+
+#start ipsec-secgw
+secgw_start()
+{
+	SGW_EXEC_FILE=$(tempfile)
+	cat <<EOF > ${SGW_EXEC_FILE}
+${SGW_PATH} ${SGW_CMD_EAL_PRM} ${CRYPTO_DEV} \
+--vdev="net_tap0,mac=fixed" \
+-- ${SGW_CMD_PRM} ${SGW_CMD_XPRM} -f ${SGW_CFG_FILE} > \
+${SGW_OUT_FILE} 2>&1 &
+p=\$!
+echo \$p
+EOF
+
+	cat ${SGW_EXEC_FILE}
+	SGW_PID=`/bin/bash -x ${SGW_EXEC_FILE}`
+
+	# wait till ipsec-secgw start properly
+	i=0
+	st=1
+	while [[ $i -ne 10 && st -ne 0 ]]; do
+		sleep 1
+		ifconfig ${LOCAL_IFACE}
+		st=$?
+		let i++
+	done
+}
+
+#stop ipsec-secgw and cleanup
+secgw_stop()
+{
+	kill ${SGW_PID}
+	rm -f ${SGW_EXEC_FILE}
+	rm -f ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/data_rxtx.sh b/examples/ipsec-secgw/test/data_rxtx.sh
new file mode 100644
index 000000000..f23a6d594
--- /dev/null
+++ b/examples/ipsec-secgw/test/data_rxtx.sh
@@ -0,0 +1,62 @@
+#! /bin/bash
+
+TCP_PORT=22222
+
+ping_test1()
+{
+	dst=$1
+
+	i=0
+	st=0
+	while [[ $i -ne 1200 && $st -eq 0 ]];
+	do
+		let i++
+		ping -c 1 -s ${i} ${dst}
+		st=$?
+	done
+
+	if [[ $st -ne 0 ]]; then
+		echo "ERROR: $0 failed for dst=${dst}, sz=${i}"
+	fi
+	return $st;
+}
+
+ping6_test1()
+{
+	dst=$1
+
+	i=0
+	st=0
+	while [[ $i -ne 1200 && $st -eq 0 ]];
+	do
+		let i++
+		ping6 -c 1 -s ${i} ${dst}
+		st=$?
+	done
+
+	if [[ $st -ne 0 ]]; then
+		echo "ERROR: $0 failed for dst=${dst}, sz=${i}"
+	fi
+	return $st;
+}
+
+scp_test1()
+{
+	dst=$1
+
+	for sz in 1234 23456 345678 4567890 56789102 ; do
+		x=`basename $0`.${sz}
+		dd if=/dev/urandom of=${x} bs=${sz} count=1
+		scp ${x} [${dst}]:${x}
+		scp [${dst}]:${x} ${x}.copy1
+		diff -u ${x} ${x}.copy1
+		st=$?
+		rm -f ${x} ${x}.copy1
+		ssh ${REMOTE_HOST} rm -f ${x}
+		if [[ $st -ne 0 ]]; then
+			return $st
+		fi
+	done
+
+	return 0;
+}
diff --git a/examples/ipsec-secgw/test/linux_test4.sh b/examples/ipsec-secgw/test/linux_test4.sh
new file mode 100644
index 000000000..d636f5604
--- /dev/null
+++ b/examples/ipsec-secgw/test/linux_test4.sh
@@ -0,0 +1,63 @@
+#! /bin/bash
+
+# usage:  /bin/bash linux_test4.sh <ipsec_mode>
+# for list of available modes please refer to run_test.sh.
+# ipsec-secgw (IPv4 mode) functional test script.
+#
+# Note that for most of them you required appropriate crypto PMD/device
+# to be avaialble.
+# Also user has to setup properly the following environment variables:
+#  SGW_PATH - path to the ipsec-secgw binary to test
+#  REMOTE_HOST - ip/hostname of the DUT
+#  REMOTE_IFACE - iface name for the test-port on DUT
+#  ETH_DEV - ethernet device to be used on SUT by DPDK ('-w <pci-id>')
+# Also user can optonally setup:
+#  SGW_LCORE - lcore to run ipsec-secgw on (default value is 0)
+#  CRYPTO_DEV - crypto device to be used ('-w <pci-id>')
+#  if none specified appropriate vdevs will be created by the scrit
+#
+# The purpose of the script is to automate ipsec-secgw testing
+# using another system running linux as a DUT.
+# It expects that SUT and DUT are connected through at least 2 NICs.
+# One NIC is expected to be managed by linux both machines,
+# and will be used as a control path
+# Make sure user from SUT can ssh to DUT without entering password.
+# Second NIC (test-port) should be reserved for DPDK on SUT,
+# and should be managed by linux on DUT.
+# The script starts ipsec-secgw with 2 NIC devices: test-port and tap vdev.
+# Then configures local tap iface and remote iface and ipsec policies
+# in the following way:
+# traffic going over test-port in both directions has to be
+# protected by ipsec.
+# raffic going over TAP in both directions doesn't have to be protected.
+# I.E:
+# DUT OS(NIC1)--(ipsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS
+# SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(ipsec)-->(NIC1)DUT OS
+# Then tries to perorm some data transfer using the scheme decribed above.
+#
+
+DIR=`dirname $0`
+MODE=$1
+
+ . ${DIR}/common_defs.sh
+ . ${DIR}/${MODE}_defs.sh
+
+config_secgw
+
+secgw_start
+
+config_iface
+
+config_remote_xfrm
+
+ . ${DIR}/data_rxtx.sh
+
+ping_test1 ${REMOTE_IPV4}
+st=$?
+if [[ $st -eq 0 ]]; then
+	scp_test1 ${REMOTE_IPV4}
+	st=$?
+fi
+
+secgw_stop
+exit $st
diff --git a/examples/ipsec-secgw/test/linux_test6.sh b/examples/ipsec-secgw/test/linux_test6.sh
new file mode 100644
index 000000000..e30f607d8
--- /dev/null
+++ b/examples/ipsec-secgw/test/linux_test6.sh
@@ -0,0 +1,64 @@
+#! /bin/bash
+
+# usage:  /bin/bash linux_test6.sh <ipsec_mode>
+# for list of available modes please refer to run_test.sh.
+# ipsec-secgw (IPv6 mode) functional test script.
+#
+# Note that for most of them you required appropriate crypto PMD/device
+# to be avaialble.
+# Also user has to setup properly the following environment variables:
+#  SGW_PATH - path to the ipsec-secgw binary to test
+#  REMOTE_HOST - ip/hostname of the DUT
+#  REMOTE_IFACE - iface name for the test-port on DUT
+#  ETH_DEV - ethernet device to be used on SUT by DPDK ('-w <pci-id>')
+# Also user can optonally setup:
+#  SGW_LCORE - lcore to run ipsec-secgw on (default value is 0)
+#  CRYPTO_DEV - crypto device to be used ('-w <pci-id>')
+#  if none specified appropriate vdevs will be created by the scrit
+#
+# The purpose of the script is to automate ipsec-secgw testing
+# using another system running linux as a DUT.
+# It expects that SUT and DUT are connected through at least 2 NICs.
+# One NIC is expected to be managed by linux both machines,
+# and will be used as a control path.
+# Make sure user from SUT can ssh to DUT without entering password,
+# also make sure that sshd over ipv6 is enabled.
+# Second NIC (test-port) should be reserved for DPDK on SUT,
+# and should be managed by linux on DUT.
+# The script starts ipsec-secgw with 2 NIC devices: test-port and tap vdev.
+# Then configures local tap iface and remote iface and ipsec policies
+# in the following way:
+# traffic going over test-port in both directions has to be
+# protected by ipsec.
+# raffic going over TAP in both directions doesn't have to be protected.
+# I.E:
+# DUT OS(NIC1)--(ipsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS
+# SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(ipsec)-->(NIC1)DUT OS
+# Then tries to perorm some data transfer using the scheme decribed above.
+#
+
+DIR=`dirname $0`
+MODE=$1
+
+ . ${DIR}/common_defs.sh
+ . ${DIR}/${MODE}_defs.sh
+
+config_secgw
+
+secgw_start
+
+config6_iface
+
+config6_remote_xfrm
+
+ . ${DIR}/data_rxtx.sh
+
+ping6_test1 ${REMOTE_IPV6}
+st=$?
+if [[ $st -eq 0 ]]; then
+	scp_test1 ${REMOTE_IPV6}
+	st=$?
+fi
+
+secgw_stop
+exit $st
diff --git a/examples/ipsec-secgw/test/run_test.sh b/examples/ipsec-secgw/test/run_test.sh
new file mode 100644
index 000000000..6dc0ce54e
--- /dev/null
+++ b/examples/ipsec-secgw/test/run_test.sh
@@ -0,0 +1,80 @@
+#! /bin/bash
+
+# usage: /bin/bash run_test.sh [-46]
+# Run all defined linux_test[4,6].sh test-cases one by one
+# user has to setup properly the following environment variables:
+#  SGW_PATH - path to the ipsec-secgw binary to test
+#  REMOTE_HOST - ip/hostname of the DUT
+#  REMOTE_IFACE - iface name for the test-port on DUT
+#  ETH_DEV - ethernet device to be used on SUT by DPDK ('-w <pci-id>')
+# Also user can optonally setup:
+#  SGW_LCORE - lcore to run ipsec-secgw on (default value is 0)
+#  CRYPTO_DEV - crypto device to be used ('-w <pci-id>')
+#  if none specified appropriate vdevs will be created by the scrit
+# refer to linux_test1.sh for more information
+
+# All supported modes to test.
+# naming convention:
+# 'old' means that ipsec-secgw will run in legacy (non-librte_ipsec mode)
+# 'tun/trs' refer to tunnel/transport mode respectively
+LINUX_TEST="tun_aescbc_sha1 \
+tun_aescbc_sha1_esn \
+tun_aescbc_sha1_esn_atom \
+tun_aesgcm \
+tun_aesgcm_esn \
+tun_aesgcm_esn_atom \
+trs_aescbc_sha1 \
+trs_aescbc_sha1_esn \
+trs_aescbc_sha1_esn_atom \
+trs_aesgcm \
+trs_aesgcm_esn \
+trs_aesgcm_esn_atom \
+tun_aescbc_sha1_old \
+tun_aesgcm_old \
+trs_aescbc_sha1_old \
+trs_aesgcm_old"
+
+DIR=`dirname $0`
+
+# get input options
+st=0
+run4=0
+run6=0
+while [[ ${st} -eq 0 ]]; do
+	getopts ":46" opt
+	st=$?
+	if [[ "${opt}" == "4" ]]; then
+		run4=1
+	elif [[ "${opt}" == "6" ]]; then
+		run6=1
+	fi
+done
+
+if [[ ${run4} -eq 0 && {run6} -eq 0 ]]; then
+	exit 127
+fi
+
+for i in ${LINUX_TEST}; do
+
+	echo "starting test ${i}"
+
+	st4=0
+	if [[ ${run4} -ne 0 ]]; then
+		/bin/bash ${DIR}/linux_test4.sh ${i}
+		st4=$?
+		echo "test4 ${i} finished with status ${st4}"
+	fi
+
+	st6=0
+	if [[ ${run6} -ne 0 ]]; then
+		/bin/bash ${DIR}/linux_test6.sh ${i}
+		st6=$?
+		echo "test6 ${i} finished with status ${st6}"
+	fi
+
+	let "st = st4 + st6"
+	if [[ $st -ne 0 ]]; then
+		echo "ERROR test ${i} FAILED"
+		exit $st
+	fi
+done
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
new file mode 100644
index 000000000..e2621e0df
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
@@ -0,0 +1,69 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_mb0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#SP in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#sp in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+sa in 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#SA out rules
+sa out 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#SA out rules
+sa out 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
new file mode 100644
index 000000000..d68552fce
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
@@ -0,0 +1,67 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
new file mode 100644
index 000000000..f16222e11
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
new file mode 100644
index 000000000..ce7c977a3
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
@@ -0,0 +1,66 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
new file mode 100644
index 000000000..a3abb6103
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
new file mode 100644
index 000000000..f6c5bf5a7
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
@@ -0,0 +1,60 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_gcm0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#SP in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport ${SGW_CFG_XPRM}
+
+sa in 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport ${SGW_CFG_XPRM}
+
+#SA out rules
+sa out 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport ${SGW_CFG_XPRM}
+
+sa out 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport ${SGW_CFG_XPRM}
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_defs.sh
new file mode 100644
index 000000000..a4d902be0
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_defs.sh
@@ -0,0 +1,76 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+
+	# to overcome problem with ipsec-secgw for inline mode,
+	# when first packet(s) will be always dropped.
+	# note that ping will fail here
+	ssh ${REMOTE_HOST} ping -c 1 ${LOCAL_IPV4}
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+
+	# to overcome problem with ipsec-secgw for inline mode,
+	# when first packet(s) will be always dropped.
+	# note that ping will fail here
+	ssh ${REMOTE_HOST} ping -c 1 ${LOCAL_IPV6}
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
new file mode 100644
index 000000000..80d8d63b8
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
new file mode 100644
index 000000000..94958d199
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
@@ -0,0 +1,66 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_defs.sh
new file mode 100644
index 000000000..de6048d68
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_defs.sh
@@ -0,0 +1,6 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_defs.sh
+
+CRYPTO_DEV='--vdev="crypto_null0"'
+SGW_CFG_XPRM='port_id 0 type inline-crypto-offload'
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_old_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_old_defs.sh
new file mode 100644
index 000000000..05230496f
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_inline_crypto_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
new file mode 100644
index 000000000..951e6b68f
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
new file mode 100644
index 000000000..4025da232
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
@@ -0,0 +1,68 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_mb0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#sp in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#sp in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${REMOTE_IPV4} dst ${LOCAL_IPV4}
+
+sa in 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${REMOTE_IPV6} dst ${LOCAL_IPV6}
+
+#SA out rules
+sa out 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${LOCAL_IPV4} dst ${REMOTE_IPV4}
+
+sa out 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${LOCAL_IPV6} dst ${REMOTE_IPV6}
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
new file mode 100644
index 000000000..18aade3a9
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
new file mode 100644
index 000000000..6b4a82149
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
new file mode 100644
index 000000000..28c1125d6
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
new file mode 100644
index 000000000..3c0d8d1b1
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
new file mode 100644
index 000000000..278377967
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
@@ -0,0 +1,60 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_gcm0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#sp in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#sp in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${REMOTE_IPV4} dst ${LOCAL_IPV4} ${SGW_CFG_XPRM}
+
+sa in 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${REMOTE_IPV6} dst ${LOCAL_IPV6} ${SGW_CFG_XPRM}
+
+#SA out rules
+sa out 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${LOCAL_IPV4} dst ${REMOTE_IPV4} ${SGW_CFG_XPRM}
+
+sa out 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${LOCAL_IPV6} dst ${REMOTE_IPV6} ${SGW_CFG_XPRM}
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_defs.sh
new file mode 100644
index 000000000..1764ef681
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_defs.sh
@@ -0,0 +1,80 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+
+	# to overcome problem with ipsec-secgw for inline mode,
+	# when first packet(s) will be always dropped.
+	# note that ping will fail here
+	ssh ${REMOTE_HOST} ping -c 1 ${LOCAL_IPV4}
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+
+	# to overcome problem with ipsec-secgw for inline mode,
+	# when first packet(s) will be always dropped.
+	# note that ping will fail here
+	ssh ${REMOTE_HOST} ping6 -c 1 ${LOCAL_IPV6}
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
new file mode 100644
index 000000000..dab1460c8
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
new file mode 100644
index 000000000..606232349
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_defs.sh
new file mode 100644
index 000000000..eafefe7ae
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_defs.sh
@@ -0,0 +1,6 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_defs.sh
+
+CRYPTO_DEV='--vdev="crypto_null0"'
+SGW_CFG_XPRM='port_id 0 type inline-crypto-offload'
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_old_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_old_defs.sh
new file mode 100644
index 000000000..de659610c
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_inline_crypto_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh
new file mode 100644
index 000000000..e0a015e21
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_defs.sh
+
+SGW_CMD_XPRM=
-- 
2.17.1

[PATCH v8 10/10] doc: update ipsec-secgw guide and relelase notes

[Konstantin Ananyev]

Update ipsec-secgw guide and relelase notes to reflect latest changes.

Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
---
 doc/guides/rel_notes/release_19_02.rst   |  14 +++
 doc/guides/sample_app_ug/ipsec_secgw.rst | 105 ++++++++++++++++++++++-
 2 files changed, 117 insertions(+), 2 deletions(-)

diff --git a/doc/guides/rel_notes/release_19_02.rst b/doc/guides/rel_notes/release_19_02.rst
index fe231152f..d824d66bb 100644
--- a/doc/guides/rel_notes/release_19_02.rst
+++ b/doc/guides/rel_notes/release_19_02.rst
@@ -133,6 +133,20 @@ New Features
 
   See :doc:`../prog_guide/ipsec_lib` for more information.
 
+* **Updated the ipsec-secgw sample application.**
+
+  The ``ipsec-secgw`` sample application has been updated to use the new
+  ``librte_ipsec`` library also added in this release.
+  The original functionality of ipsec-secgw is retained, a new command line
+  parameter ``-l`` has  been added to ipsec-secgw to use the IPsec library,
+  instead of the existing IPsec code in the application.
+
+  The IPsec library does not support all the functionality of the existing
+  ipsec-secgw application, its is planned to add the outstanding functionality
+  in future releases.
+
+  See :doc:`../sample_app_ug/ipsec_secgw` for more information.
+
 
 Removed Items
 -------------
diff --git a/doc/guides/sample_app_ug/ipsec_secgw.rst b/doc/guides/sample_app_ug/ipsec_secgw.rst
index 61638e733..3d784e705 100644
--- a/doc/guides/sample_app_ug/ipsec_secgw.rst
+++ b/doc/guides/sample_app_ug/ipsec_secgw.rst
@@ -76,7 +76,7 @@ Compiling the Application
 
 To compile the sample application see :doc:`compiling`.
 
-The application is located in the ``rpsec-secgw`` sub-directory.
+The application is located in the ``ipsec-secgw`` sub-directory.
 
 #. [Optional] Build the application for debugging:
    This option adds some extra flags, disables compiler optimizations and
@@ -93,6 +93,7 @@ The application has a number of command line options::
 
    ./build/ipsec-secgw [EAL options] --
                         -p PORTMASK -P -u PORTMASK -j FRAMESIZE
+                        -l -w REPLAY_WINOW_SIZE -e -a
                         --config (port,queue,lcore)[,(port,queue,lcore]
                         --single-sa SAIDX
                         --rxoffload MASK
@@ -114,6 +115,18 @@ Where:
     specified as FRAMESIZE. If an invalid value is provided as FRAMESIZE
     then the default value 9000 is used.
 
+*   ``-l``: enables code-path that uses librte_ipsec.
+
+*   ``-w REPLAY_WINOW_SIZE``: specifies the IPsec sequence number replay window
+    size for each Security Association (available only with librte_ipsec
+    code path).
+
+*   ``-e``: enables Security Association extended sequence number processing
+    (available only with librte_ipsec code path).
+
+*   ``-a``: enables Security Association sequence number atomic behaviour
+    (available only with librte_ipsec code path).
+
 *   ``--config (port,queue,lcore)[,(port,queue,lcore)]``: determines which queues
     from which ports are mapped to which cores.
 
@@ -225,7 +238,7 @@ accordingly.
 
 
 Configuration File Syntax
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~
 
 As mention in the overview, the Security Policies are ACL rules.
 The application parsers the rules specified in the configuration file and
@@ -571,6 +584,11 @@ Example SA rules:
     mode ipv4-tunnel src 172.16.1.5 dst 172.16.2.5 \
     type lookaside-protocol-offload port_id 4
 
+    sa in 35 aead_algo aes-128-gcm \
+    aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+    mode ipv4-tunnel src 172.16.2.5 dst 172.16.1.5 \
+    type inline-crypto-offload port_id 0
+
 Routing rule syntax
 ^^^^^^^^^^^^^^^^^^^
 
@@ -667,3 +685,86 @@ Example Neighbour rules:
 .. code-block:: console
 
     neigh port 0 DE:AD:BE:EF:01:02
+
+Test directory
+--------------
+
+The test directory contains scripts for testing the various encryption
+algorithms.
+
+The purpose of the scripts is to automate ipsec-secgw testing
+using another system running linux as a DUT.
+
+The user must setup the following environment variables:
+
+*   ``SGW_PATH``: path to the ipsec-secgw binary to test.
+
+*   ``REMOTE_HOST``: IP address/hostname of the DUT.
+
+*   ``REMOTE_IFACE``: interface name for the test-port on the DUT.
+
+*   ``ETH_DEV``: ethernet device to be used on the SUT by DPDK ('-w <pci-id>')
+
+Also the user can optionally setup:
+
+*   ``SGW_LCORE``: lcore to run ipsec-secgw on (default value is 0)
+
+*   ``CRYPTO_DEV``: crypto device to be used ('-w <pci-id>'). If none specified
+    appropriate vdevs will be created by the script
+
+Note that most of the tests require the appropriate crypto PMD/device to be
+available.
+
+Server configuration
+~~~~~~~~~~~~~~~~~~~~
+
+Two servers are required for the tests, SUT and DUT.
+
+Make sure the user from the SUT can ssh to the DUT without entering the password.
+To enable this feature keys must be setup on the DUT.
+
+``ssh-keygen`` will make a private & public key pair on the SUT.
+
+``ssh-copy-id`` <user name>@<target host name> on the SUT will copy the public
+key to the DUT. It will ask for credentials so that it can upload the public key.
+
+The SUT and DUT are connected through at least 2 NIC ports.
+
+One NIC port is expected to be managed by linux on both machines and will be
+used as a control path.
+
+The second NIC port (test-port) should be bound to DPDK on the SUT, and should
+be managed by linux on the DUT.
+
+The script starts ``ipsec-secgw`` with 2 NIC devices: ``test-port`` and
+``tap vdev``.
+
+It then configures the local tap interface and the remote interface and IPsec
+policies in the following way:
+
+Traffic going over the test-port in both directions has to be protected by IPsec.
+
+Traffic going over the TAP port in both directions does not have to be protected.
+
+i.e:
+
+DUT OS(NIC1)--(IPsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS
+
+SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(IPsec)-->(NIC1)DUT OS
+
+It then tries to perform some data transfer using the scheme decribed above.
+
+usage
+~~~~~
+
+In the ipsec-secgw/test directory
+
+to run one test for IPv4 or IPv6
+
+/bin/bash linux_test(4|6).sh <ipsec_mode>
+
+to run all tests for IPv4 or IPv6
+
+/bin/bash run_test.sh -4|-6
+
+For the list of available modes please refer to run_test.sh.
-- 
2.17.1

Re: [PATCH v8 10/10] doc: update ipsec-secgw guide and relelase notes

[Varghese, Vipin]

Hi Konstantin,

As per 19.02-rc1, documentation has to be updated along with the code base. 

snipped
> --- a/doc/guides/rel_notes/release_19_02.rst
> +++ b/doc/guides/rel_notes/release_19_02.rst
> @@ -133,6 +133,20 @@ New Features
> 
>    See :doc:`../prog_guide/ipsec_lib` for more information.
> 
> +* **Updated the ipsec-secgw sample application.**
> +
> +  The ``ipsec-secgw`` sample application has been updated to use the
> + new  ``librte_ipsec`` library also added in this release.
> +  The original functionality of ipsec-secgw is retained, a new command
> + line  parameter ``-l`` has  been added to ipsec-secgw to use the IPsec
> + library,  instead of the existing IPsec code in the application.
> +
> +  The IPsec library does not support all the functionality of the
> + existing  ipsec-secgw application, its is planned to add the
> + outstanding functionality  in future releases.
> +
> +  See :doc:`../sample_app_ug/ipsec_secgw` for more information.
> +
> 
In my opinion this can come in the first patch 

snipped
>  #. [Optional] Build the application for debugging:
>     This option adds some extra flags, disables compiler optimizations and @@ -
> 93,6 +93,7 @@ The application has a number of command line options::
> 
>     ./build/ipsec-secgw [EAL options] --
>                          -p PORTMASK -P -u PORTMASK -j FRAMESIZE
> +                        -l -w REPLAY_WINOW_SIZE -e -a
This can be added patch which adds the option

>                          --config (port,queue,lcore)[,(port,queue,lcore]
>                          --single-sa SAIDX
>                          --rxoffload MASK @@ -114,6 +115,18 @@ Where:
>      specified as FRAMESIZE. If an invalid value is provided as FRAMESIZE
>      then the default value 9000 is used.
> 
> +*   ``-l``: enables code-path that uses librte_ipsec.
> +
> +*   ``-w REPLAY_WINOW_SIZE``: specifies the IPsec sequence number replay
> window
> +    size for each Security Association (available only with librte_ipsec
> +    code path).
> +
> +*   ``-e``: enables Security Association extended sequence number processing
> +    (available only with librte_ipsec code path).
> +
> +*   ``-a``: enables Security Association sequence number atomic behaviour
> +    (available only with librte_ipsec code path).
> +
>  *   ``--config (port,queue,lcore)[,(port,queue,lcore)]``: determines which
> queues
>      from which ports are mapped to which cores.
> 
> @@ -225,7 +238,7 @@ accordingly.
> 
> 
>  Configuration File Syntax
> -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +~~~~~~~~~~~~~~~~~~~~~~~~~
> 
>  As mention in the overview, the Security Policies are ACL rules.
>  The application parsers the rules specified in the configuration file and @@ -
> 571,6 +584,11 @@ Example SA rules:
>      mode ipv4-tunnel src 172.16.1.5 dst 172.16.2.5 \
>      type lookaside-protocol-offload port_id 4
> 
> +    sa in 35 aead_algo aes-128-gcm \
> +    aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
> +    mode ipv4-tunnel src 172.16.2.5 dst 172.16.1.5 \
> +    type inline-crypto-offload port_id 0
> +
>  Routing rule syntax
>  ^^^^^^^^^^^^^^^^^^^
> 
> @@ -667,3 +685,86 @@ Example Neighbour rules:
>  .. code-block:: console
> 
>      neigh port 0 DE:AD:BE:EF:01:02
> +
> +Test directory
> +--------------
> +
> +The test directory contains scripts for testing the various encryption
> +algorithms.
> +
> +The purpose of the scripts is to automate ipsec-secgw testing using
> +another system running linux as a DUT.
> +
> +The user must setup the following environment variables:
> +
> +*   ``SGW_PATH``: path to the ipsec-secgw binary to test.
> +
> +*   ``REMOTE_HOST``: IP address/hostname of the DUT.
> +
> +*   ``REMOTE_IFACE``: interface name for the test-port on the DUT.
> +
> +*   ``ETH_DEV``: ethernet device to be used on the SUT by DPDK ('-w <pci-id>')
> +
> +Also the user can optionally setup:
> +
> +*   ``SGW_LCORE``: lcore to run ipsec-secgw on (default value is 0)
> +
> +*   ``CRYPTO_DEV``: crypto device to be used ('-w <pci-id>'). If none specified
> +    appropriate vdevs will be created by the script
> +
> +Note that most of the tests require the appropriate crypto PMD/device
> +to be available.
> +
> +Server configuration
> +~~~~~~~~~~~~~~~~~~~~
> +
> +Two servers are required for the tests, SUT and DUT.
> +
> +Make sure the user from the SUT can ssh to the DUT without entering the
> password.
> +To enable this feature keys must be setup on the DUT.
> +
> +``ssh-keygen`` will make a private & public key pair on the SUT.
> +
> +``ssh-copy-id`` <user name>@<target host name> on the SUT will copy the
> +public key to the DUT. It will ask for credentials so that it can upload the
> public key.
> +
> +The SUT and DUT are connected through at least 2 NIC ports.
> +
> +One NIC port is expected to be managed by linux on both machines and
> +will be used as a control path.
> +
> +The second NIC port (test-port) should be bound to DPDK on the SUT, and
> +should be managed by linux on the DUT.
> +
> +The script starts ``ipsec-secgw`` with 2 NIC devices: ``test-port`` and
> +``tap vdev``.
> +
> +It then configures the local tap interface and the remote interface and
> +IPsec policies in the following way:
> +
> +Traffic going over the test-port in both directions has to be protected by
> IPsec.
> +
> +Traffic going over the TAP port in both directions does not have to be
> protected.
> +
> +i.e:
> +
> +DUT OS(NIC1)--(IPsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS
> +
> +SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(IPsec)-->(NIC1)DUT OS
> +
> +It then tries to perform some data transfer using the scheme decribed above.
> +
> +usage
> +~~~~~
> +
> +In the ipsec-secgw/test directory
> +
> +to run one test for IPv4 or IPv6
> +
> +/bin/bash linux_test(4|6).sh <ipsec_mode>
> +
> +to run all tests for IPv4 or IPv6
> +
> +/bin/bash run_test.sh -4|-6
> +
> +For the list of available modes please refer to run_test.sh.
> --
> 2.17.1

Re: [PATCH v8 10/10] doc: update ipsec-secgw guide and relelase notes

[Akhil Goyal]

Hi Vipin,

On 1/11/2019 8:19 AM, Varghese, Vipin wrote:
> Hi Konstantin,
>
> As per 19.02-rc1, documentation has to be updated along with the code base.
I and Pablo, thought about your comment, but the feature that is added 
in this patchset is split across multiple patches (mainly 7/10 and 8/10) 
and it would be improper to add documentation in any of these patches.
So, it would be good to have a separate patch which is added when the 
feature is completely there.

>
> snipped
>> --- a/doc/guides/rel_notes/release_19_02.rst
>> +++ b/doc/guides/rel_notes/release_19_02.rst
>> @@ -133,6 +133,20 @@ New Features
>>
>>     See :doc:`../prog_guide/ipsec_lib` for more information.
>>
>> +* **Updated the ipsec-secgw sample application.**
>> +
>> +  The ``ipsec-secgw`` sample application has been updated to use the
>> + new  ``librte_ipsec`` library also added in this release.
>> +  The original functionality of ipsec-secgw is retained, a new command
>> + line  parameter ``-l`` has  been added to ipsec-secgw to use the IPsec
>> + library,  instead of the existing IPsec code in the application.
>> +
>> +  The IPsec library does not support all the functionality of the
>> + existing  ipsec-secgw application, its is planned to add the
>> + outstanding functionality  in future releases.
>> +
>> +  See :doc:`../sample_app_ug/ipsec_secgw` for more information.
>> +
>>
> In my opinion this can come in the first patch
>
> snipped
>>   #. [Optional] Build the application for debugging:
>>      This option adds some extra flags, disables compiler optimizations and @@ -
>> 93,6 +93,7 @@ The application has a number of command line options::
>>
>>      ./build/ipsec-secgw [EAL options] --
>>                           -p PORTMASK -P -u PORTMASK -j FRAMESIZE
>> +                        -l -w REPLAY_WINOW_SIZE -e -a
> This can be added patch which adds the option
>
>>                           --config (port,queue,lcore)[,(port,queue,lcore]
>>                           --single-sa SAIDX
>>                           --rxoffload MASK @@ -114,6 +115,18 @@ Where:
>>       specified as FRAMESIZE. If an invalid value is provided as FRAMESIZE
>>       then the default value 9000 is used.
>>
>> +*   ``-l``: enables code-path that uses librte_ipsec.
>> +
>> +*   ``-w REPLAY_WINOW_SIZE``: specifies the IPsec sequence number replay
>> window
>> +    size for each Security Association (available only with librte_ipsec
>> +    code path).
>> +
>> +*   ``-e``: enables Security Association extended sequence number processing
>> +    (available only with librte_ipsec code path).
>> +
>> +*   ``-a``: enables Security Association sequence number atomic behaviour
>> +    (available only with librte_ipsec code path).
>> +
>>   *   ``--config (port,queue,lcore)[,(port,queue,lcore)]``: determines which
>> queues
>>       from which ports are mapped to which cores.
>>
>> @@ -225,7 +238,7 @@ accordingly.
>>
>>
>>   Configuration File Syntax
>> -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> +~~~~~~~~~~~~~~~~~~~~~~~~~
>>
>>   As mention in the overview, the Security Policies are ACL rules.
>>   The application parsers the rules specified in the configuration file and @@ -
>> 571,6 +584,11 @@ Example SA rules:
>>       mode ipv4-tunnel src 172.16.1.5 dst 172.16.2.5 \
>>       type lookaside-protocol-offload port_id 4
>>
>> +    sa in 35 aead_algo aes-128-gcm \
>> +    aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
>> +    mode ipv4-tunnel src 172.16.2.5 dst 172.16.1.5 \
>> +    type inline-crypto-offload port_id 0
>> +
>>   Routing rule syntax
>>   ^^^^^^^^^^^^^^^^^^^
>>
>> @@ -667,3 +685,86 @@ Example Neighbour rules:
>>   .. code-block:: console
>>
>>       neigh port 0 DE:AD:BE:EF:01:02
>> +
>> +Test directory
>> +--------------
>> +
>> +The test directory contains scripts for testing the various encryption
>> +algorithms.
>> +
>> +The purpose of the scripts is to automate ipsec-secgw testing using
>> +another system running linux as a DUT.
>> +
>> +The user must setup the following environment variables:
>> +
>> +*   ``SGW_PATH``: path to the ipsec-secgw binary to test.
>> +
>> +*   ``REMOTE_HOST``: IP address/hostname of the DUT.
>> +
>> +*   ``REMOTE_IFACE``: interface name for the test-port on the DUT.
>> +
>> +*   ``ETH_DEV``: ethernet device to be used on the SUT by DPDK ('-w <pci-id>')
>> +
>> +Also the user can optionally setup:
>> +
>> +*   ``SGW_LCORE``: lcore to run ipsec-secgw on (default value is 0)
>> +
>> +*   ``CRYPTO_DEV``: crypto device to be used ('-w <pci-id>'). If none specified
>> +    appropriate vdevs will be created by the script
>> +
>> +Note that most of the tests require the appropriate crypto PMD/device
>> +to be available.
>> +
>> +Server configuration
>> +~~~~~~~~~~~~~~~~~~~~
>> +
>> +Two servers are required for the tests, SUT and DUT.
>> +
>> +Make sure the user from the SUT can ssh to the DUT without entering the
>> password.
>> +To enable this feature keys must be setup on the DUT.
>> +
>> +``ssh-keygen`` will make a private & public key pair on the SUT.
>> +
>> +``ssh-copy-id`` <user name>@<target host name> on the SUT will copy the
>> +public key to the DUT. It will ask for credentials so that it can upload the
>> public key.
>> +
>> +The SUT and DUT are connected through at least 2 NIC ports.
>> +
>> +One NIC port is expected to be managed by linux on both machines and
>> +will be used as a control path.
>> +
>> +The second NIC port (test-port) should be bound to DPDK on the SUT, and
>> +should be managed by linux on the DUT.
>> +
>> +The script starts ``ipsec-secgw`` with 2 NIC devices: ``test-port`` and
>> +``tap vdev``.
>> +
>> +It then configures the local tap interface and the remote interface and
>> +IPsec policies in the following way:
>> +
>> +Traffic going over the test-port in both directions has to be protected by
>> IPsec.
>> +
>> +Traffic going over the TAP port in both directions does not have to be
>> protected.
>> +
>> +i.e:
>> +
>> +DUT OS(NIC1)--(IPsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS
>> +
>> +SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(IPsec)-->(NIC1)DUT OS
>> +
>> +It then tries to perform some data transfer using the scheme decribed above.
>> +
>> +usage
>> +~~~~~
>> +
>> +In the ipsec-secgw/test directory
>> +
>> +to run one test for IPv4 or IPv6
>> +
>> +/bin/bash linux_test(4|6).sh <ipsec_mode>
>> +
>> +to run all tests for IPv4 or IPv6
>> +
>> +/bin/bash run_test.sh -4|-6
>> +
>> +For the list of available modes please refer to run_test.sh.
>> --
>> 2.17.1

Re: [PATCH v8 10/10] doc: update ipsec-secgw guide and relelase notes

[Varghese, Vipin]

Hi Akhil,

Thanks for considering, as mentioned it is suggestion from my end how the document could have been updated.

Thanks
Vipin Varghese

> -----Original Message-----
> From: Akhil Goyal <akhil.goyal@nxp.com>
> Sent: Friday, January 11, 2019 12:26 PM
> To: Varghese, Vipin <vipin.varghese@intel.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; dev@dpdk.org
> Cc: De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>;
> thomas@monjalon.net; Iremonger, Bernard <bernard.iremonger@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v8 10/10] doc: update ipsec-secgw guide and
> relelase notes
> 
> Hi Vipin,
> 
> On 1/11/2019 8:19 AM, Varghese, Vipin wrote:
> > Hi Konstantin,
> >
> > As per 19.02-rc1, documentation has to be updated along with the code
> base.
> I and Pablo, thought about your comment, but the feature that is added in this
> patchset is split across multiple patches (mainly 7/10 and 8/10) and it would
> be improper to add documentation in any of these patches.
> So, it would be good to have a separate patch which is added when the
> feature is completely there.
> 
> >
> > snipped
> >> --- a/doc/guides/rel_notes/release_19_02.rst
> >> +++ b/doc/guides/rel_notes/release_19_02.rst
> >> @@ -133,6 +133,20 @@ New Features
> >>
> >>     See :doc:`../prog_guide/ipsec_lib` for more information.
> >>
> >> +* **Updated the ipsec-secgw sample application.**
> >> +
> >> +  The ``ipsec-secgw`` sample application has been updated to use the
> >> + new  ``librte_ipsec`` library also added in this release.
> >> +  The original functionality of ipsec-secgw is retained, a new
> >> + command line  parameter ``-l`` has  been added to ipsec-secgw to
> >> + use the IPsec library,  instead of the existing IPsec code in the application.
> >> +
> >> +  The IPsec library does not support all the functionality of the
> >> + existing  ipsec-secgw application, its is planned to add the
> >> + outstanding functionality  in future releases.
> >> +
> >> +  See :doc:`../sample_app_ug/ipsec_secgw` for more information.
> >> +
> >>
> > In my opinion this can come in the first patch
> >
> > snipped
> >>   #. [Optional] Build the application for debugging:
> >>      This option adds some extra flags, disables compiler
> >> optimizations and @@ -
> >> 93,6 +93,7 @@ The application has a number of command line options::
> >>
> >>      ./build/ipsec-secgw [EAL options] --
> >>                           -p PORTMASK -P -u PORTMASK -j FRAMESIZE
> >> +                        -l -w REPLAY_WINOW_SIZE -e -a
> > This can be added patch which adds the option
> >
> >>                           --config (port,queue,lcore)[,(port,queue,lcore]
> >>                           --single-sa SAIDX
> >>                           --rxoffload MASK @@ -114,6 +115,18 @@ Where:
> >>       specified as FRAMESIZE. If an invalid value is provided as FRAMESIZE
> >>       then the default value 9000 is used.
> >>
> >> +*   ``-l``: enables code-path that uses librte_ipsec.
> >> +
> >> +*   ``-w REPLAY_WINOW_SIZE``: specifies the IPsec sequence number
> replay
> >> window
> >> +    size for each Security Association (available only with librte_ipsec
> >> +    code path).
> >> +
> >> +*   ``-e``: enables Security Association extended sequence number
> processing
> >> +    (available only with librte_ipsec code path).
> >> +
> >> +*   ``-a``: enables Security Association sequence number atomic behaviour
> >> +    (available only with librte_ipsec code path).
> >> +
> >>   *   ``--config (port,queue,lcore)[,(port,queue,lcore)]``: determines which
> >> queues
> >>       from which ports are mapped to which cores.
> >>
> >> @@ -225,7 +238,7 @@ accordingly.
> >>
> >>
> >>   Configuration File Syntax
> >> -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >> +~~~~~~~~~~~~~~~~~~~~~~~~~
> >>
> >>   As mention in the overview, the Security Policies are ACL rules.
> >>   The application parsers the rules specified in the configuration
> >> file and @@ -
> >> 571,6 +584,11 @@ Example SA rules:
> >>       mode ipv4-tunnel src 172.16.1.5 dst 172.16.2.5 \
> >>       type lookaside-protocol-offload port_id 4
> >>
> >> +    sa in 35 aead_algo aes-128-gcm \
> >> +    aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
> >> +    mode ipv4-tunnel src 172.16.2.5 dst 172.16.1.5 \
> >> +    type inline-crypto-offload port_id 0
> >> +
> >>   Routing rule syntax
> >>   ^^^^^^^^^^^^^^^^^^^
> >>
> >> @@ -667,3 +685,86 @@ Example Neighbour rules:
> >>   .. code-block:: console
> >>
> >>       neigh port 0 DE:AD:BE:EF:01:02
> >> +
> >> +Test directory
> >> +--------------
> >> +
> >> +The test directory contains scripts for testing the various
> >> +encryption algorithms.
> >> +
> >> +The purpose of the scripts is to automate ipsec-secgw testing using
> >> +another system running linux as a DUT.
> >> +
> >> +The user must setup the following environment variables:
> >> +
> >> +*   ``SGW_PATH``: path to the ipsec-secgw binary to test.
> >> +
> >> +*   ``REMOTE_HOST``: IP address/hostname of the DUT.
> >> +
> >> +*   ``REMOTE_IFACE``: interface name for the test-port on the DUT.
> >> +
> >> +*   ``ETH_DEV``: ethernet device to be used on the SUT by DPDK ('-w <pci-
> id>')
> >> +
> >> +Also the user can optionally setup:
> >> +
> >> +*   ``SGW_LCORE``: lcore to run ipsec-secgw on (default value is 0)
> >> +
> >> +*   ``CRYPTO_DEV``: crypto device to be used ('-w <pci-id>'). If none
> specified
> >> +    appropriate vdevs will be created by the script
> >> +
> >> +Note that most of the tests require the appropriate crypto
> >> +PMD/device to be available.
> >> +
> >> +Server configuration
> >> +~~~~~~~~~~~~~~~~~~~~
> >> +
> >> +Two servers are required for the tests, SUT and DUT.
> >> +
> >> +Make sure the user from the SUT can ssh to the DUT without entering
> >> +the
> >> password.
> >> +To enable this feature keys must be setup on the DUT.
> >> +
> >> +``ssh-keygen`` will make a private & public key pair on the SUT.
> >> +
> >> +``ssh-copy-id`` <user name>@<target host name> on the SUT will copy
> >> +the public key to the DUT. It will ask for credentials so that it
> >> +can upload the
> >> public key.
> >> +
> >> +The SUT and DUT are connected through at least 2 NIC ports.
> >> +
> >> +One NIC port is expected to be managed by linux on both machines and
> >> +will be used as a control path.
> >> +
> >> +The second NIC port (test-port) should be bound to DPDK on the SUT,
> >> +and should be managed by linux on the DUT.
> >> +
> >> +The script starts ``ipsec-secgw`` with 2 NIC devices: ``test-port``
> >> +and ``tap vdev``.
> >> +
> >> +It then configures the local tap interface and the remote interface
> >> +and IPsec policies in the following way:
> >> +
> >> +Traffic going over the test-port in both directions has to be
> >> +protected by
> >> IPsec.
> >> +
> >> +Traffic going over the TAP port in both directions does not have to
> >> +be
> >> protected.
> >> +
> >> +i.e:
> >> +
> >> +DUT OS(NIC1)--(IPsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT
> >> +OS
> >> +
> >> +SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(IPsec)-->(NIC1)DUT OS
> >> +
> >> +It then tries to perform some data transfer using the scheme decribed
> above.
> >> +
> >> +usage
> >> +~~~~~
> >> +
> >> +In the ipsec-secgw/test directory
> >> +
> >> +to run one test for IPv4 or IPv6
> >> +
> >> +/bin/bash linux_test(4|6).sh <ipsec_mode>
> >> +
> >> +to run all tests for IPv4 or IPv6
> >> +
> >> +/bin/bash run_test.sh -4|-6
> >> +
> >> +For the list of available modes please refer to run_test.sh.
> >> --
> >> 2.17.1

Re: [PATCH v8 10/10] doc: update ipsec-secgw guide and relelase notes

[Thomas Monjalon]

10/01/2019 22:09, Konstantin Ananyev:
> Update ipsec-secgw guide and relelase notes to reflect latest changes.
> 
> Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> Acked-by: Akhil Goyal <akhil.goyal@nxp.com>
> ---
>  doc/guides/rel_notes/release_19_02.rst   |  14 +++
>  doc/guides/sample_app_ug/ipsec_secgw.rst | 105 ++++++++++++++++++++++-

This new file is missing in MAINTAINERS. I'll add it.

[PATCH v7 02/10] examples/ipsec-secgw: allow to specify neighbour mac address

[Konstantin Ananyev]

In some cases it is useful to allow user to specify destination
ether address for outgoing packets.
This patch adds such ability by introducing new 'neigh' config
file option.

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 doc/guides/sample_app_ug/ipsec_secgw.rst | 42 ++++++++++-
 examples/ipsec-secgw/ipsec-secgw.c       | 21 ++++--
 examples/ipsec-secgw/ipsec.h             |  3 +
 examples/ipsec-secgw/parser.c            | 91 ++++++++++++++++++++++++
 examples/ipsec-secgw/parser.h            |  8 +--
 5 files changed, 154 insertions(+), 11 deletions(-)

diff --git a/doc/guides/sample_app_ug/ipsec_secgw.rst b/doc/guides/sample_app_ug/ipsec_secgw.rst
index 244bde2de..61638e733 100644
--- a/doc/guides/sample_app_ug/ipsec_secgw.rst
+++ b/doc/guides/sample_app_ug/ipsec_secgw.rst
@@ -217,7 +217,7 @@ Configurations
 --------------
 
 The following sections provide the syntax of configurations to initialize
-your SP, SA and Routing tables.
+your SP, SA, Routing and Neighbour tables.
 Configurations shall be specified in the configuration file to be passed to
 the application. The file is then parsed by the application. The successful
 parsing will result in the appropriate rules being applied to the tables
@@ -238,8 +238,8 @@ General rule syntax
 
 The parse treats one line in the configuration file as one configuration
 item (unless the line concatenation symbol exists). Every configuration
-item shall follow the syntax of either SP, SA, or Routing rules specified
-below.
+item shall follow the syntax of either SP, SA, Routing or Neighbour
+rules specified below.
 
 The configuration parser supports the following special symbols:
 
@@ -631,3 +631,39 @@ Example SP rules:
     rt ipv4 dst 172.16.1.5/32 port 0
 
     rt ipv6 dst 1111:1111:1111:1111:1111:1111:1111:5555/116 port 0
+
+Neighbour rule syntax
+^^^^^^^^^^^^^^^^^^^^^
+
+The Neighbour rule syntax is shown as follows:
+
+.. code-block:: console
+
+    neigh <port> <dst_mac>
+
+
+where each options means:
+
+``<port>``
+
+ * The output port id
+
+ * Optional: No
+
+ * Syntax: *port X*
+
+``<dst_mac>``
+
+ * The destination ethernet address to use for that port
+
+ * Optional: No
+
+ * Syntax:
+
+   * XX:XX:XX:XX:XX:XX
+
+Example Neighbour rules:
+
+.. code-block:: console
+
+    neigh port 0 DE:AD:BE:EF:01:02
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 4ccff38d2..e79d17d51 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -104,9 +104,9 @@ static uint16_t nb_txd = IPSEC_SECGW_TX_DESC_DEFAULT;
 #define ETHADDR(a, b, c, d, e, f) (__BYTES_TO_UINT64(a, b, c, d, e, f, 0, 0))
 
 #define ETHADDR_TO_UINT64(addr) __BYTES_TO_UINT64( \
-		addr.addr_bytes[0], addr.addr_bytes[1], \
-		addr.addr_bytes[2], addr.addr_bytes[3], \
-		addr.addr_bytes[4], addr.addr_bytes[5], \
+		(addr)->addr_bytes[0], (addr)->addr_bytes[1], \
+		(addr)->addr_bytes[2], (addr)->addr_bytes[3], \
+		(addr)->addr_bytes[4], (addr)->addr_bytes[5], \
 		0, 0)
 
 /* port/source ethernet addr and destination ethernet addr */
@@ -1242,6 +1242,19 @@ print_ethaddr(const char *name, const struct ether_addr *eth_addr)
 	printf("%s%s", name, buf);
 }
 
+/*
+ * Update destination ethaddr for the port.
+ */
+int
+add_dst_ethaddr(uint16_t port, const struct ether_addr *addr)
+{
+	if (port > RTE_DIM(ethaddr_tbl))
+		return -EINVAL;
+
+	ethaddr_tbl[port].dst = ETHADDR_TO_UINT64(addr);
+	return 0;
+}
+
 /* Check the link status of all ports in up to 9s, and print them finally */
 static void
 check_all_ports_link_status(uint32_t port_mask)
@@ -1622,7 +1635,7 @@ port_init(uint16_t portid, uint64_t req_rx_offloads, uint64_t req_tx_offloads)
 	printf("Configuring device port %u:\n", portid);
 
 	rte_eth_macaddr_get(portid, &ethaddr);
-	ethaddr_tbl[portid].src = ETHADDR_TO_UINT64(ethaddr);
+	ethaddr_tbl[portid].src = ETHADDR_TO_UINT64(&ethaddr);
 	print_ethaddr("Address: ", &ethaddr);
 	printf("\n");
 
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 9b1586f52..580f7876b 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -245,4 +245,7 @@ int
 sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
 		uint64_t *tx_offloads);
 
+int
+add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/parser.c b/examples/ipsec-secgw/parser.c
index 91282ca94..b0a8ee23b 100644
--- a/examples/ipsec-secgw/parser.c
+++ b/examples/ipsec-secgw/parser.c
@@ -306,6 +306,46 @@ parse_range(const char *token, uint16_t *low, uint16_t *high)
 	return 0;
 }
 
+/*
+ * helper function for parse_mac, parse one section of the ether addr.
+ */
+static const char *
+parse_uint8x16(const char *s, uint8_t *v, uint8_t ls)
+{
+	char *end;
+	unsigned long t;
+
+	errno = 0;
+	t = strtoul(s, &end, 16);
+	if (errno != 0 || end[0] != ls || t > UINT8_MAX)
+		return NULL;
+	v[0] = t;
+	return end + 1;
+}
+
+static int
+parse_mac(const char *str, struct ether_addr *addr)
+{
+	uint32_t i;
+
+	static const uint8_t stop_sym[RTE_DIM(addr->addr_bytes)] = {
+		[0] = ':',
+		[1] = ':',
+		[2] = ':',
+		[3] = ':',
+		[4] = ':',
+		[5] = 0,
+	};
+
+	for (i = 0; i != RTE_DIM(addr->addr_bytes); i++) {
+		str = parse_uint8x16(str, addr->addr_bytes + i, stop_sym[i]);
+		if (str == NULL)
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
 /** sp add parse */
 struct cfg_sp_add_cfg_item {
 	cmdline_fixed_string_t sp_keyword;
@@ -444,11 +484,61 @@ cmdline_parse_inst_t cfg_rt_add_rule = {
 	},
 };
 
+/* neigh add parse */
+struct cfg_neigh_add_item {
+	cmdline_fixed_string_t neigh;
+	cmdline_fixed_string_t pstr;
+	uint16_t port;
+	cmdline_fixed_string_t mac;
+};
+
+static void
+cfg_parse_neigh(void *parsed_result, __rte_unused struct cmdline *cl,
+	void *data)
+{
+	int32_t rc;
+	struct cfg_neigh_add_item *res;
+	struct parse_status *st;
+	struct ether_addr mac;
+
+	st = data;
+	res = parsed_result;
+	rc = parse_mac(res->mac, &mac);
+	APP_CHECK(rc == 0, st, "invalid ether addr:%s", res->mac);
+	rc = add_dst_ethaddr(res->port, &mac);
+	APP_CHECK(rc == 0, st, "invalid port numer:%hu", res->port);
+	if (st->status < 0)
+		return;
+}
+
+cmdline_parse_token_string_t cfg_add_neigh_start =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, neigh, "neigh");
+cmdline_parse_token_string_t cfg_add_neigh_pstr =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, pstr, "port");
+cmdline_parse_token_num_t cfg_add_neigh_port =
+	TOKEN_NUM_INITIALIZER(struct cfg_neigh_add_item, port, UINT16);
+cmdline_parse_token_string_t cfg_add_neigh_mac =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, mac, NULL);
+
+cmdline_parse_inst_t cfg_neigh_add_rule = {
+	.f = cfg_parse_neigh,
+	.data = NULL,
+	.help_str = "",
+	.tokens = {
+		(void *)&cfg_add_neigh_start,
+		(void *)&cfg_add_neigh_pstr,
+		(void *)&cfg_add_neigh_port,
+		(void *)&cfg_add_neigh_mac,
+		NULL,
+	},
+};
+
 /** set of cfg items */
 cmdline_parse_ctx_t ipsec_ctx[] = {
 	(cmdline_parse_inst_t *)&cfg_sp_add_rule,
 	(cmdline_parse_inst_t *)&cfg_sa_add_rule,
 	(cmdline_parse_inst_t *)&cfg_rt_add_rule,
+	(cmdline_parse_inst_t *)&cfg_neigh_add_rule,
 	NULL,
 };
 
@@ -474,6 +564,7 @@ parse_cfg_file(const char *cfg_filename)
 	cfg_sp_add_rule.data = &status;
 	cfg_sa_add_rule.data = &status;
 	cfg_rt_add_rule.data = &status;
+	cfg_neigh_add_rule.data = &status;
 
 	do {
 		char oneline[1024];
diff --git a/examples/ipsec-secgw/parser.h b/examples/ipsec-secgw/parser.h
index be02537c5..6b8a10076 100644
--- a/examples/ipsec-secgw/parser.h
+++ b/examples/ipsec-secgw/parser.h
@@ -14,14 +14,14 @@ struct parse_status {
 	char parse_msg[256];
 };
 
-#define	APP_CHECK(exp, status, fmt, ...)				\
+#define	APP_CHECK(exp, st, fmt, ...)					\
 do {									\
 	if (!(exp)) {							\
-		sprintf(status->parse_msg, fmt "\n",			\
+		sprintf((st)->parse_msg, fmt "\n",			\
 			## __VA_ARGS__);				\
-		status->status = -1;					\
+		(st)->status = -1;					\
 	} else								\
-		status->status = 0;					\
+		(st)->status = 0;					\
 } while (0)
 
 #define APP_CHECK_PRESENCE(val, str, status)				\
-- 
2.17.1

[PATCH v7 03/10] examples/ipsec-secgw: fix crypto-op might never get dequeued

[Konstantin Ananyev]

In some cases crypto-ops could never be dequeued from the crypto-device.
The easiest way to reproduce:
start ipsec-secgw with crypto-dev and send to it less then 32 packets.
none packets will be forwarded.
Reason for that is that the application does dequeue() from crypto-queues
only when new packets arrive.
This patch makes the app to call dequeue() on a regular basis.
Also to make code cleaner and easier to understand,
it separates crypto-dev enqueue() and dequeue() code paths.
pkt_process() now only enqueues packets into crypto device,
dequeuing and final processing is done by drain_crypto_queues().

Fixes: c64278c0c18b ("examples/ipsec-secgw: rework processing loop")
Cc: stable@dpdk.org

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 144 ++++++++++++++++++++++++-----
 examples/ipsec-secgw/ipsec.c       |  99 +++++++++++++-------
 examples/ipsec-secgw/ipsec.h       |  11 +++
 3 files changed, 198 insertions(+), 56 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index e79d17d51..f8290c13c 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -469,38 +469,55 @@ inbound_sp_sa(struct sp_ctx *sp, struct sa_ctx *sa, struct traffic_type *ip,
 	ip->num = j;
 }
 
-static inline void
-process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
-		struct ipsec_traffic *traffic)
+static void
+split46_traffic(struct ipsec_traffic *trf, struct rte_mbuf *mb[], uint32_t num)
 {
+	uint32_t i, n4, n6;
+	struct ip *ip;
 	struct rte_mbuf *m;
-	uint16_t idx, nb_pkts_in, i, n_ip4, n_ip6;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
+	n4 = trf->ip4.num;
+	n6 = trf->ip6.num;
 
-	n_ip4 = traffic->ip4.num;
-	n_ip6 = traffic->ip6.num;
+	for (i = 0; i < num; i++) {
+
+		m = mb[i];
+		ip = rte_pktmbuf_mtod(m, struct ip *);
 
-	/* SP/ACL Inbound check ipsec and ip4 */
-	for (i = 0; i < nb_pkts_in; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
 		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-			traffic->ip4.data[idx] = rte_pktmbuf_mtod_offset(m,
+			trf->ip4.pkts[n4] = m;
+			trf->ip4.data[n4] = rte_pktmbuf_mtod_offset(m,
 					uint8_t *, offsetof(struct ip, ip_p));
+			n4++;
 		} else if (ip->ip_v == IP6_VERSION) {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
-			traffic->ip6.data[idx] = rte_pktmbuf_mtod_offset(m,
+			trf->ip6.pkts[n6] = m;
+			trf->ip6.data[n6] = rte_pktmbuf_mtod_offset(m,
 					uint8_t *,
 					offsetof(struct ip6_hdr, ip6_nxt));
+			n6++;
 		} else
 			rte_pktmbuf_free(m);
 	}
 
+	trf->ip4.num = n4;
+	trf->ip6.num = n6;
+}
+
+
+static inline void
+process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
+		struct ipsec_traffic *traffic)
+{
+	uint16_t nb_pkts_in, n_ip4, n_ip6;
+
+	n_ip4 = traffic->ip4.num;
+	n_ip6 = traffic->ip6.num;
+
+	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.num, MAX_PKT_BURST);
+
+	split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+
 	inbound_sp_sa(ipsec_ctx->sp4_ctx, ipsec_ctx->sa_ctx, &traffic->ip4,
 			n_ip4);
 
@@ -795,7 +812,7 @@ process_pkts(struct lcore_conf *qconf, struct rte_mbuf **pkts,
 }
 
 static inline void
-drain_buffers(struct lcore_conf *qconf)
+drain_tx_buffers(struct lcore_conf *qconf)
 {
 	struct buffer *buf;
 	uint32_t portid;
@@ -809,6 +826,81 @@ drain_buffers(struct lcore_conf *qconf)
 	}
 }
 
+static inline void
+drain_crypto_buffers(struct lcore_conf *qconf)
+{
+	uint32_t i;
+	struct ipsec_ctx *ctx;
+
+	/* drain inbound buffers*/
+	ctx = &qconf->inbound;
+	for (i = 0; i != ctx->nb_qps; i++) {
+		if (ctx->tbl[i].len != 0)
+			enqueue_cop_burst(ctx->tbl  + i);
+	}
+
+	/* drain outbound buffers*/
+	ctx = &qconf->outbound;
+	for (i = 0; i != ctx->nb_qps; i++) {
+		if (ctx->tbl[i].len != 0)
+			enqueue_cop_burst(ctx->tbl  + i);
+	}
+}
+
+static void
+drain_inbound_crypto_queues(const struct lcore_conf *qconf,
+		struct ipsec_ctx *ctx)
+{
+	uint32_t n;
+	struct ipsec_traffic trf;
+
+	/* dequeue packets from crypto-queue */
+	n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+			RTE_DIM(trf.ipsec.pkts));
+	if (n == 0)
+		return;
+
+	trf.ip4.num = 0;
+	trf.ip6.num = 0;
+
+	/* split traffic by ipv4-ipv6 */
+	split46_traffic(&trf, trf.ipsec.pkts, n);
+
+	/* process ipv4 packets */
+	inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
+	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+
+	/* process ipv6 packets */
+	inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
+	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+}
+
+static void
+drain_outbound_crypto_queues(const struct lcore_conf *qconf,
+		struct ipsec_ctx *ctx)
+{
+	uint32_t n;
+	struct ipsec_traffic trf;
+
+	/* dequeue packets from crypto-queue */
+	n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+			RTE_DIM(trf.ipsec.pkts));
+	if (n == 0)
+		return;
+
+	trf.ip4.num = 0;
+	trf.ip6.num = 0;
+
+	/* split traffic by ipv4-ipv6 */
+	split46_traffic(&trf, trf.ipsec.pkts, n);
+
+	/* process ipv4 packets */
+	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+
+	/* process ipv6 packets */
+	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+}
+
 /* main processing loop */
 static int32_t
 main_loop(__attribute__((unused)) void *dummy)
@@ -866,12 +958,14 @@ main_loop(__attribute__((unused)) void *dummy)
 		diff_tsc = cur_tsc - prev_tsc;
 
 		if (unlikely(diff_tsc > drain_tsc)) {
-			drain_buffers(qconf);
+			drain_tx_buffers(qconf);
+			drain_crypto_buffers(qconf);
 			prev_tsc = cur_tsc;
 		}
 
-		/* Read packet from RX queues */
 		for (i = 0; i < qconf->nb_rx_queue; ++i) {
+
+			/* Read packets from RX queues */
 			portid = rxql[i].port_id;
 			queueid = rxql[i].queue_id;
 			nb_rx = rte_eth_rx_burst(portid, queueid,
@@ -879,6 +973,14 @@ main_loop(__attribute__((unused)) void *dummy)
 
 			if (nb_rx > 0)
 				process_pkts(qconf, pkts, nb_rx, portid);
+
+			/* dequeue and process completed crypto-ops */
+			if (UNPROTECTED_PORT(portid))
+				drain_inbound_crypto_queues(qconf,
+					&qconf->inbound);
+			else
+				drain_outbound_crypto_queues(qconf,
+					&qconf->outbound);
 		}
 	}
 }
diff --git a/examples/ipsec-secgw/ipsec.c b/examples/ipsec-secgw/ipsec.c
index 3d415f1af..72a29bcb1 100644
--- a/examples/ipsec-secgw/ipsec.c
+++ b/examples/ipsec-secgw/ipsec.c
@@ -333,33 +333,35 @@ create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa)
 	return 0;
 }
 
+/*
+ * queue crypto-ops into PMD queue.
+ */
+void
+enqueue_cop_burst(struct cdev_qp *cqp)
+{
+	uint32_t i, len, ret;
+
+	len = cqp->len;
+	ret = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp, cqp->buf, len);
+	if (ret < len) {
+		RTE_LOG_DP(DEBUG, IPSEC, "Cryptodev %u queue %u:"
+			" enqueued %u crypto ops out of %u\n",
+			cqp->id, cqp->qp, ret, len);
+			/* drop packets that we fail to enqueue */
+			for (i = ret; i < len; i++)
+				rte_pktmbuf_free(cqp->buf[i]->sym->m_src);
+	}
+	cqp->in_flight += ret;
+	cqp->len = 0;
+}
+
 static inline void
 enqueue_cop(struct cdev_qp *cqp, struct rte_crypto_op *cop)
 {
-	int32_t ret = 0, i;
-
 	cqp->buf[cqp->len++] = cop;
 
-	if (cqp->len == MAX_PKT_BURST) {
-		int enq_size = cqp->len;
-		if ((cqp->in_flight + enq_size) > MAX_INFLIGHT)
-			enq_size -=
-			    (int)((cqp->in_flight + enq_size) - MAX_INFLIGHT);
-
-		if (enq_size > 0)
-			ret = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp,
-					cqp->buf, enq_size);
-		if (ret < cqp->len) {
-			RTE_LOG_DP(DEBUG, IPSEC, "Cryptodev %u queue %u:"
-					" enqueued %u crypto ops out of %u\n",
-					 cqp->id, cqp->qp,
-					 ret, cqp->len);
-			for (i = ret; i < cqp->len; i++)
-				rte_pktmbuf_free(cqp->buf[i]->sym->m_src);
-		}
-		cqp->in_flight += ret;
-		cqp->len = 0;
-	}
+	if (cqp->len == MAX_PKT_BURST)
+		enqueue_cop_burst(cqp);
 }
 
 static inline void
@@ -473,6 +475,32 @@ ipsec_enqueue(ipsec_xform_fn xform_func, struct ipsec_ctx *ipsec_ctx,
 	}
 }
 
+static inline int32_t
+ipsec_inline_dequeue(ipsec_xform_fn xform_func, struct ipsec_ctx *ipsec_ctx,
+	      struct rte_mbuf *pkts[], uint16_t max_pkts)
+{
+	int32_t nb_pkts, ret;
+	struct ipsec_mbuf_metadata *priv;
+	struct ipsec_sa *sa;
+	struct rte_mbuf *pkt;
+
+	nb_pkts = 0;
+	while (ipsec_ctx->ol_pkts_cnt > 0 && nb_pkts < max_pkts) {
+		pkt = ipsec_ctx->ol_pkts[--ipsec_ctx->ol_pkts_cnt];
+		rte_prefetch0(pkt);
+		priv = get_priv(pkt);
+		sa = priv->sa;
+		ret = xform_func(pkt, sa, &priv->cop);
+		if (unlikely(ret)) {
+			rte_pktmbuf_free(pkt);
+			continue;
+		}
+		pkts[nb_pkts++] = pkt;
+	}
+
+	return nb_pkts;
+}
+
 static inline int
 ipsec_dequeue(ipsec_xform_fn xform_func, struct ipsec_ctx *ipsec_ctx,
 	      struct rte_mbuf *pkts[], uint16_t max_pkts)
@@ -490,19 +518,6 @@ ipsec_dequeue(ipsec_xform_fn xform_func, struct ipsec_ctx *ipsec_ctx,
 		if (ipsec_ctx->last_qp == ipsec_ctx->nb_qps)
 			ipsec_ctx->last_qp %= ipsec_ctx->nb_qps;
 
-		while (ipsec_ctx->ol_pkts_cnt > 0 && nb_pkts < max_pkts) {
-			pkt = ipsec_ctx->ol_pkts[--ipsec_ctx->ol_pkts_cnt];
-			rte_prefetch0(pkt);
-			priv = get_priv(pkt);
-			sa = priv->sa;
-			ret = xform_func(pkt, sa, &priv->cop);
-			if (unlikely(ret)) {
-				rte_pktmbuf_free(pkt);
-				continue;
-			}
-			pkts[nb_pkts++] = pkt;
-		}
-
 		if (cqp->in_flight == 0)
 			continue;
 
@@ -545,6 +560,13 @@ ipsec_inbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 
 	ipsec_enqueue(esp_inbound, ctx, pkts, sas, nb_pkts);
 
+	return ipsec_inline_dequeue(esp_inbound_post, ctx, pkts, len);
+}
+
+uint16_t
+ipsec_inbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len)
+{
 	return ipsec_dequeue(esp_inbound_post, ctx, pkts, len);
 }
 
@@ -558,5 +580,12 @@ ipsec_outbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 
 	ipsec_enqueue(esp_outbound, ctx, pkts, sas, nb_pkts);
 
+	return ipsec_inline_dequeue(esp_outbound_post, ctx, pkts, len);
+}
+
+uint16_t
+ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len)
+{
 	return ipsec_dequeue(esp_outbound_post, ctx, pkts, len);
 }
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 580f7876b..2f04b7d68 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -184,6 +184,14 @@ uint16_t
 ipsec_outbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint32_t sa_idx[], uint16_t nb_pkts, uint16_t len);
 
+uint16_t
+ipsec_inbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len);
+
+uint16_t
+ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len);
+
 static inline uint16_t
 ipsec_metadata_size(void)
 {
@@ -248,4 +256,7 @@ sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
 int
 add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
 
+void
+enqueue_cop_burst(struct cdev_qp *cqp);
+
 #endif /* __IPSEC_H__ */
-- 
2.17.1

[PATCH v7 04/10] examples/ipsec-secgw: fix outbound codepath for single SA

[Konstantin Ananyev]

Looking at process_pkts_outbound_nosp() there seems few issues:
- accessing mbuf after it was freed
- invoking ipsec_outbound() for ipv4 packets only
- copying number of packets, but not the mbuf pointers itself

that patch provides fixes for that issues.

Fixes: 906257e965b7 ("examples/ipsec-secgw: support IPv6")
Cc: stable@dpdk.org

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 33 +++++++++++++++++++++---------
 1 file changed, 23 insertions(+), 10 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index f8290c13c..1cb4f06b0 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -629,32 +629,45 @@ process_pkts_outbound_nosp(struct ipsec_ctx *ipsec_ctx,
 		struct ipsec_traffic *traffic)
 {
 	struct rte_mbuf *m;
-	uint32_t nb_pkts_out, i;
+	uint32_t nb_pkts_out, i, n;
 	struct ip *ip;
 
 	/* Drop any IPsec traffic from protected ports */
 	for (i = 0; i < traffic->ipsec.num; i++)
 		rte_pktmbuf_free(traffic->ipsec.pkts[i]);
 
-	traffic->ipsec.num = 0;
+	n = 0;
 
-	for (i = 0; i < traffic->ip4.num; i++)
-		traffic->ip4.res[i] = single_sa_idx;
+	for (i = 0; i < traffic->ip4.num; i++) {
+		traffic->ipsec.pkts[n] = traffic->ip4.pkts[i];
+		traffic->ipsec.res[n++] = single_sa_idx;
+	}
 
-	for (i = 0; i < traffic->ip6.num; i++)
-		traffic->ip6.res[i] = single_sa_idx;
+	for (i = 0; i < traffic->ip6.num; i++) {
+		traffic->ipsec.pkts[n] = traffic->ip6.pkts[i];
+		traffic->ipsec.res[n++] = single_sa_idx;
+	}
+
+	traffic->ip4.num = 0;
+	traffic->ip6.num = 0;
+	traffic->ipsec.num = n;
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ip4.pkts,
-			traffic->ip4.res, traffic->ip4.num,
+	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.res, traffic->ipsec.num,
 			MAX_PKT_BURST);
 
 	/* They all sue the same SA (ip4 or ip6 tunnel) */
 	m = traffic->ipsec.pkts[i];
 	ip = rte_pktmbuf_mtod(m, struct ip *);
-	if (ip->ip_v == IPVERSION)
+	if (ip->ip_v == IPVERSION) {
 		traffic->ip4.num = nb_pkts_out;
-	else
+		for (i = 0; i < nb_pkts_out; i++)
+			traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+	} else {
 		traffic->ip6.num = nb_pkts_out;
+		for (i = 0; i < nb_pkts_out; i++)
+			traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+	}
 }
 
 static inline int32_t
-- 
2.17.1

[PATCH v7 05/10] examples/ipsec-secgw: make local variables static

[Konstantin Ananyev]

in sp4.c and sp6.c there are few globals that used only locally.
Define them as static ones.

Cc: stable@dpdk.org

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/sp4.c | 10 +++++-----
 examples/ipsec-secgw/sp6.c | 10 +++++-----
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/examples/ipsec-secgw/sp4.c b/examples/ipsec-secgw/sp4.c
index 8d3d3d8e0..6b05daaa9 100644
--- a/examples/ipsec-secgw/sp4.c
+++ b/examples/ipsec-secgw/sp4.c
@@ -44,7 +44,7 @@ enum {
 	RTE_ACL_IPV4_NUM
 };
 
-struct rte_acl_field_def ip4_defs[NUM_FIELDS_IPV4] = {
+static struct rte_acl_field_def ip4_defs[NUM_FIELDS_IPV4] = {
 	{
 	.type = RTE_ACL_FIELD_TYPE_BITMASK,
 	.size = sizeof(uint8_t),
@@ -85,11 +85,11 @@ struct rte_acl_field_def ip4_defs[NUM_FIELDS_IPV4] = {
 
 RTE_ACL_RULE_DEF(acl4_rules, RTE_DIM(ip4_defs));
 
-struct acl4_rules acl4_rules_out[MAX_ACL_RULE_NUM];
-uint32_t nb_acl4_rules_out;
+static struct acl4_rules acl4_rules_out[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl4_rules_out;
 
-struct acl4_rules acl4_rules_in[MAX_ACL_RULE_NUM];
-uint32_t nb_acl4_rules_in;
+static struct acl4_rules acl4_rules_in[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl4_rules_in;
 
 void
 parse_sp4_tokens(char **tokens, uint32_t n_tokens,
diff --git a/examples/ipsec-secgw/sp6.c b/examples/ipsec-secgw/sp6.c
index 6002afef3..dc5b94c6a 100644
--- a/examples/ipsec-secgw/sp6.c
+++ b/examples/ipsec-secgw/sp6.c
@@ -34,7 +34,7 @@ enum {
 
 #define IP6_ADDR_SIZE 16
 
-struct rte_acl_field_def ip6_defs[IP6_NUM] = {
+static struct rte_acl_field_def ip6_defs[IP6_NUM] = {
 	{
 	.type = RTE_ACL_FIELD_TYPE_BITMASK,
 	.size = sizeof(uint8_t),
@@ -116,11 +116,11 @@ struct rte_acl_field_def ip6_defs[IP6_NUM] = {
 
 RTE_ACL_RULE_DEF(acl6_rules, RTE_DIM(ip6_defs));
 
-struct acl6_rules acl6_rules_out[MAX_ACL_RULE_NUM];
-uint32_t nb_acl6_rules_out;
+static struct acl6_rules acl6_rules_out[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl6_rules_out;
 
-struct acl6_rules acl6_rules_in[MAX_ACL_RULE_NUM];
-uint32_t nb_acl6_rules_in;
+static struct acl6_rules acl6_rules_in[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl6_rules_in;
 
 void
 parse_sp6_tokens(char **tokens, uint32_t n_tokens,
-- 
2.17.1

[PATCH v7 06/10] examples/ipsec-secgw: fix inbound SA checking

[Konstantin Ananyev]

In the inbound_sa_check() make sure that sa pointer stored
inside mbuf private area is not NULL.

Fixes: d299106e8e31 ("examples/ipsec-secgw: add IPsec sample application")
Cc: stable@dpdk.org

Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/sa.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index f6271bc60..839aaca0c 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -947,10 +947,15 @@ int
 inbound_sa_check(struct sa_ctx *sa_ctx, struct rte_mbuf *m, uint32_t sa_idx)
 {
 	struct ipsec_mbuf_metadata *priv;
+	struct ipsec_sa *sa;
 
 	priv = get_priv(m);
+	sa = priv->sa;
+	if (sa != NULL)
+		return (sa_ctx->sa[sa_idx].spi == sa->spi);
 
-	return (sa_ctx->sa[sa_idx].spi == priv->sa->spi);
+	RTE_LOG(ERR, IPSEC, "SA not saved in private data\n");
+	return 0;
 }
 
 static inline void
-- 
2.17.1

[PATCH v7 07/10] examples/ipsec-secgw: make app to use ipsec library

[Konstantin Ananyev]

Changes to make ipsec-secgw to utilize librte_ipsec library.
That patch provides:
 - changes in the related data structures.
 - changes in the initialization code.
 - new command-line parameters to enable librte_ipsec codepath
   and related features.

Note that right now by default current (non-librte_ipsec) code-path will
be used. User has to run application with new command-line option ('-l')
to enable new codepath.
The main reason for that:
 - current librte_ipsec doesn't support all ipsec algorithms
   and features that the app does.
 - allow users to run both versions in parallel for some time
   to figure out any functional or performance degradation with the
   new code.

It is planned to deprecate and remove non-librte_ipsec code path
in future releases.

Signed-off-by: Mohammad Abdul Awal <mohammad.abdul.awal@intel.com>
Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/Makefile      |   4 +-
 examples/ipsec-secgw/ipsec-secgw.c |  51 ++++++-
 examples/ipsec-secgw/ipsec.h       |  24 ++++
 examples/ipsec-secgw/meson.build   |   2 +-
 examples/ipsec-secgw/sa.c          | 221 ++++++++++++++++++++++++++++-
 examples/ipsec-secgw/sp4.c         |  25 ++++
 examples/ipsec-secgw/sp6.c         |  25 ++++
 7 files changed, 344 insertions(+), 8 deletions(-)

diff --git a/examples/ipsec-secgw/Makefile b/examples/ipsec-secgw/Makefile
index 02d41e39a..3918ee63e 100644
--- a/examples/ipsec-secgw/Makefile
+++ b/examples/ipsec-secgw/Makefile
@@ -61,8 +61,8 @@ RTE_TARGET ?= x86_64-native-linuxapp-gcc
 include $(RTE_SDK)/mk/rte.vars.mk
 
 ifneq ($(MAKECMDGOALS),clean)
-ifneq ($(CONFIG_RTE_LIBRTE_SECURITY),y)
-$(error "RTE_LIBRTE_SECURITY is required to build ipsec-secgw")
+ifneq ($(CONFIG_RTE_LIBRTE_IPSEC),y)
+$(error "RTE_LIBRTE_IPSEC is required to build ipsec-secgw")
 endif
 endif
 
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 1cb4f06b0..9174b3e56 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -168,6 +168,9 @@ static uint32_t frame_size;
 static uint64_t dev_rx_offload = UINT64_MAX;
 static uint64_t dev_tx_offload = UINT64_MAX;
 
+/* application wide librte_ipsec/SA parameters */
+struct app_sa_prm app_sa_prm = {.enable = 0};
+
 struct lcore_rx_queue {
 	uint16_t port_id;
 	uint8_t queue_id;
@@ -1083,6 +1086,10 @@ print_usage(const char *prgname)
 		" [-P]"
 		" [-u PORTMASK]"
 		" [-j FRAMESIZE]"
+		" [-l]"
+		" [-w REPLAY_WINDOW_SIZE]"
+		" [-e]"
+		" [-a]"
 		" -f CONFIG_FILE"
 		" --config (port,queue,lcore)[,(port,queue,lcore)]"
 		" [--single-sa SAIDX]"
@@ -1095,6 +1102,11 @@ print_usage(const char *prgname)
 		"  -u PORTMASK: Hexadecimal bitmask of unprotected ports\n"
 		"  -j FRAMESIZE: Enable jumbo frame with 'FRAMESIZE' as maximum\n"
 		"                packet size\n"
+		"  -l enables code-path that uses librte_ipsec\n"
+		"  -w REPLAY_WINDOW_SIZE specifies IPsec SQN replay window\n"
+		"     size for each SA\n"
+		"  -e enables ESN\n"
+		"  -a enables SA SQN atomic behaviour\n"
 		"  -f CONFIG_FILE: Configuration file\n"
 		"  --config (port,queue,lcore): Rx queue configuration\n"
 		"  --single-sa SAIDX: Use single SA index for outbound traffic,\n"
@@ -1212,6 +1224,20 @@ parse_config(const char *q_arg)
 	return 0;
 }
 
+static void
+print_app_sa_prm(const struct app_sa_prm *prm)
+{
+	printf("librte_ipsec usage: %s\n",
+		(prm->enable == 0) ? "disabled" : "enabled");
+
+	if (prm->enable == 0)
+		return;
+
+	printf("replay window size: %u\n", prm->window_size);
+	printf("ESN: %s\n", (prm->enable_esn == 0) ? "disabled" : "enabled");
+	printf("SA flags: %#" PRIx64 "\n", prm->flags);
+}
+
 static int32_t
 parse_args(int32_t argc, char **argv)
 {
@@ -1223,7 +1249,7 @@ parse_args(int32_t argc, char **argv)
 
 	argvopt = argv;
 
-	while ((opt = getopt_long(argc, argvopt, "p:Pu:f:j:",
+	while ((opt = getopt_long(argc, argvopt, "aelp:Pu:f:j:w:",
 				lgopts, &option_index)) != EOF) {
 
 		switch (opt) {
@@ -1279,6 +1305,21 @@ parse_args(int32_t argc, char **argv)
 			}
 			printf("Enabled jumbo frames size %u\n", frame_size);
 			break;
+		case 'l':
+			app_sa_prm.enable = 1;
+			break;
+		case 'w':
+			app_sa_prm.enable = 1;
+			app_sa_prm.window_size = parse_decimal(optarg);
+			break;
+		case 'e':
+			app_sa_prm.enable = 1;
+			app_sa_prm.enable_esn = 1;
+			break;
+		case 'a':
+			app_sa_prm.enable = 1;
+			app_sa_prm.flags |= RTE_IPSEC_SAFLAG_SQN_ATOM;
+			break;
 		case CMD_LINE_OPT_CONFIG_NUM:
 			ret = parse_config(optarg);
 			if (ret) {
@@ -1341,6 +1382,8 @@ parse_args(int32_t argc, char **argv)
 		return -1;
 	}
 
+	print_app_sa_prm(&app_sa_prm);
+
 	if (optind >= 0)
 		argv[optind-1] = prgname;
 
@@ -2012,12 +2055,14 @@ main(int32_t argc, char **argv)
 		if (socket_ctx[socket_id].mbuf_pool)
 			continue;
 
-		sa_init(&socket_ctx[socket_id], socket_id);
-
+		/* initilaze SPD */
 		sp4_init(&socket_ctx[socket_id], socket_id);
 
 		sp6_init(&socket_ctx[socket_id], socket_id);
 
+		/* initilaze SAD */
+		sa_init(&socket_ctx[socket_id], socket_id);
+
 		rt_init(&socket_ctx[socket_id], socket_id);
 
 		pool_init(&socket_ctx[socket_id], socket_id, NB_MBUF);
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 2f04b7d68..8da86c4de 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -11,6 +11,7 @@
 #include <rte_crypto.h>
 #include <rte_security.h>
 #include <rte_flow.h>
+#include <rte_ipsec.h>
 
 #define RTE_LOGTYPE_IPSEC       RTE_LOGTYPE_USER1
 #define RTE_LOGTYPE_IPSEC_ESP   RTE_LOGTYPE_USER2
@@ -70,7 +71,20 @@ struct ip_addr {
 
 #define MAX_KEY_SIZE		32
 
+/*
+ * application wide SA parameters
+ */
+struct app_sa_prm {
+	uint32_t enable; /* use librte_ipsec API for ipsec pkt processing */
+	uint32_t window_size; /* replay window size */
+	uint32_t enable_esn;  /* enable/disable ESN support */
+	uint64_t flags;       /* rte_ipsec_sa_prm.flags */
+};
+
+extern struct app_sa_prm app_sa_prm;
+
 struct ipsec_sa {
+	struct rte_ipsec_session ips; /* one session per sa for now */
 	uint32_t spi;
 	uint32_t cdev_id_qp;
 	uint64_t seq;
@@ -243,6 +257,16 @@ sp4_init(struct socket_ctx *ctx, int32_t socket_id);
 void
 sp6_init(struct socket_ctx *ctx, int32_t socket_id);
 
+/*
+ * Search through SP rules for given SPI.
+ * Returns first rule index if found(greater or equal then zero),
+ * or -ENOENT otherwise.
+ */
+int
+sp4_spi_present(uint32_t spi, int inbound);
+int
+sp6_spi_present(uint32_t spi, int inbound);
+
 void
 sa_init(struct socket_ctx *ctx, int32_t socket_id);
 
diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
index 77d8b298f..31f68fee2 100644
--- a/examples/ipsec-secgw/meson.build
+++ b/examples/ipsec-secgw/meson.build
@@ -6,7 +6,7 @@
 # To build this example as a standalone application with an already-installed
 # DPDK instance, use 'make'
 
-deps += ['security', 'lpm', 'acl', 'hash']
+deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
 allow_experimental_apis = true
 sources = files(
 	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index 839aaca0c..414fcd26c 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -19,6 +19,7 @@
 #include <rte_ip.h>
 #include <rte_random.h>
 #include <rte_ethdev.h>
+#include <rte_malloc.h>
 
 #include "ipsec.h"
 #include "esp.h"
@@ -694,6 +695,7 @@ print_one_sa_rule(const struct ipsec_sa *sa, int inbound)
 }
 
 struct sa_ctx {
+	void *satbl; /* pointer to array of rte_ipsec_sa objects*/
 	struct ipsec_sa sa[IPSEC_SA_MAX_ENTRIES];
 	union {
 		struct {
@@ -764,7 +766,10 @@ sa_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 {
 	struct ipsec_sa *sa;
 	uint32_t i, idx;
-	uint16_t iv_length;
+	uint16_t iv_length, aad_length;
+
+	/* for ESN upper 32 bits of SQN also need to be part of AAD */
+	aad_length = (app_sa_prm.enable_esn != 0) ? sizeof(uint32_t) : 0;
 
 	for (i = 0; i < nb_entries; i++) {
 		idx = SPI2IDX(entries[i].spi);
@@ -808,7 +813,7 @@ sa_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 			sa_ctx->xf[idx].a.aead.iv.offset = IV_OFFSET;
 			sa_ctx->xf[idx].a.aead.iv.length = iv_length;
 			sa_ctx->xf[idx].a.aead.aad_length =
-				sa->aad_len;
+				sa->aad_len + aad_length;
 			sa_ctx->xf[idx].a.aead.digest_length =
 				sa->digest_len;
 
@@ -901,9 +906,205 @@ sa_in_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 	return sa_add_rules(sa_ctx, entries, nb_entries, 1);
 }
 
+/*
+ * helper function, fills parameters that are identical for all SAs
+ */
+static void
+fill_ipsec_app_sa_prm(struct rte_ipsec_sa_prm *prm,
+	const struct app_sa_prm *app_prm)
+{
+	memset(prm, 0, sizeof(*prm));
+
+	prm->flags = app_prm->flags;
+	prm->ipsec_xform.options.esn = app_prm->enable_esn;
+	prm->replay_win_sz = app_prm->window_size;
+}
+
+/*
+ * Helper function, tries to determine next_proto for SPI
+ * by searching though SP rules.
+ */
+static int
+get_spi_proto(uint32_t spi, enum rte_security_ipsec_sa_direction dir)
+{
+	int32_t rc4, rc6;
+
+	rc4 = sp4_spi_present(spi, dir == RTE_SECURITY_IPSEC_SA_DIR_INGRESS);
+	rc6 = sp6_spi_present(spi, dir == RTE_SECURITY_IPSEC_SA_DIR_INGRESS);
+
+	if (rc4 >= 0) {
+		if (rc6 >= 0) {
+			RTE_LOG(ERR, IPSEC,
+				"%s: SPI %u used simultaeously by "
+				"IPv4(%d) and IPv6 (%d) SP rules\n",
+				__func__, spi, rc4, rc6);
+			return -EINVAL;
+		} else
+			return IPPROTO_IPIP;
+	} else if (rc6 < 0) {
+		RTE_LOG(ERR, IPSEC,
+			"%s: SPI %u is not used by any SP rule\n",
+			__func__, spi);
+		return -EINVAL;
+	} else
+		return IPPROTO_IPV6;
+}
+
+static int
+fill_ipsec_sa_prm(struct rte_ipsec_sa_prm *prm, const struct ipsec_sa *ss,
+	const struct ipv4_hdr *v4, struct ipv6_hdr *v6)
+{
+	int32_t rc;
+
+	/*
+	 * Try to get SPI next proto by searching that SPI in SPD.
+	 * probably not the optimal way, but there seems nothing
+	 * better right now.
+	 */
+	rc = get_spi_proto(ss->spi, ss->direction);
+	if (rc < 0)
+		return rc;
+
+	fill_ipsec_app_sa_prm(prm, &app_sa_prm);
+	prm->userdata = (uintptr_t)ss;
+
+	/* setup ipsec xform */
+	prm->ipsec_xform.spi = ss->spi;
+	prm->ipsec_xform.salt = ss->salt;
+	prm->ipsec_xform.direction = ss->direction;
+	prm->ipsec_xform.proto = RTE_SECURITY_IPSEC_SA_PROTO_ESP;
+	prm->ipsec_xform.mode = (ss->flags == TRANSPORT) ?
+		RTE_SECURITY_IPSEC_SA_MODE_TRANSPORT :
+		RTE_SECURITY_IPSEC_SA_MODE_TUNNEL;
+
+	if (ss->flags == IP4_TUNNEL) {
+		prm->ipsec_xform.tunnel.type = RTE_SECURITY_IPSEC_TUNNEL_IPV4;
+		prm->tun.hdr_len = sizeof(*v4);
+		prm->tun.next_proto = rc;
+		prm->tun.hdr = v4;
+	} else if (ss->flags == IP6_TUNNEL) {
+		prm->ipsec_xform.tunnel.type = RTE_SECURITY_IPSEC_TUNNEL_IPV6;
+		prm->tun.hdr_len = sizeof(*v6);
+		prm->tun.next_proto = rc;
+		prm->tun.hdr = v6;
+	} else {
+		/* transport mode */
+		prm->trs.proto = rc;
+	}
+
+	/* setup crypto section */
+	prm->crypto_xform = ss->xforms;
+	return 0;
+}
+
+static void
+fill_ipsec_session(struct rte_ipsec_session *ss, struct rte_ipsec_sa *sa,
+	const struct ipsec_sa *lsa)
+{
+	ss->sa = sa;
+	ss->type = lsa->type;
+
+	/* setup crypto section */
+	if (ss->type == RTE_SECURITY_ACTION_TYPE_NONE) {
+		ss->crypto.ses = lsa->crypto_session;
+	/* setup session action type */
+	} else {
+		ss->security.ses = lsa->sec_session;
+		ss->security.ctx = lsa->security_ctx;
+		ss->security.ol_flags = lsa->ol_flags;
+	}
+}
+
+/*
+ * Initialise related rte_ipsec_sa object.
+ */
+static int
+ipsec_sa_init(struct ipsec_sa *lsa, struct rte_ipsec_sa *sa, uint32_t sa_size)
+{
+	int rc;
+	struct rte_ipsec_sa_prm prm;
+	struct ipv4_hdr v4  = {
+		.version_ihl = IPVERSION << 4 |
+			sizeof(v4) / IPV4_IHL_MULTIPLIER,
+		.time_to_live = IPDEFTTL,
+		.next_proto_id = IPPROTO_ESP,
+		.src_addr = lsa->src.ip.ip4,
+		.dst_addr = lsa->dst.ip.ip4,
+	};
+	struct ipv6_hdr v6 = {
+		.vtc_flow = htonl(IP6_VERSION << 28),
+		.proto = IPPROTO_ESP,
+	};
+
+	if (lsa->flags == IP6_TUNNEL) {
+		memcpy(v6.src_addr, lsa->src.ip.ip6.ip6_b, sizeof(v6.src_addr));
+		memcpy(v6.dst_addr, lsa->dst.ip.ip6.ip6_b, sizeof(v6.dst_addr));
+	}
+
+	rc = fill_ipsec_sa_prm(&prm, lsa, &v4, &v6);
+	if (rc == 0)
+		rc = rte_ipsec_sa_init(sa, &prm, sa_size);
+	if (rc < 0)
+		return rc;
+
+	fill_ipsec_session(&lsa->ips, sa, lsa);
+	return 0;
+}
+
+/*
+ * Allocate space and init rte_ipsec_sa strcutures,
+ * one per session.
+ */
+static int
+ipsec_satbl_init(struct sa_ctx *ctx, const struct ipsec_sa *ent,
+	uint32_t nb_ent, int32_t socket)
+{
+	int32_t rc, sz;
+	uint32_t i, idx;
+	size_t tsz;
+	struct rte_ipsec_sa *sa;
+	struct ipsec_sa *lsa;
+	struct rte_ipsec_sa_prm prm;
+
+	/* determine SA size */
+	idx = SPI2IDX(ent[0].spi);
+	fill_ipsec_sa_prm(&prm, ctx->sa + idx, NULL, NULL);
+	sz = rte_ipsec_sa_size(&prm);
+	if (sz < 0) {
+		RTE_LOG(ERR, IPSEC, "%s(%p, %u, %d): "
+			"failed to determine SA size, error code: %d\n",
+			__func__, ctx, nb_ent, socket, sz);
+		return sz;
+	}
+
+	tsz = sz * nb_ent;
+
+	ctx->satbl = rte_zmalloc_socket(NULL, tsz, RTE_CACHE_LINE_SIZE, socket);
+	if (ctx->satbl == NULL) {
+		RTE_LOG(ERR, IPSEC,
+			"%s(%p, %u, %d): failed to allocate %zu bytes\n",
+			__func__,  ctx, nb_ent, socket, tsz);
+		return -ENOMEM;
+	}
+
+	rc = 0;
+	for (i = 0; i != nb_ent && rc == 0; i++) {
+
+		idx = SPI2IDX(ent[i].spi);
+
+		sa = (struct rte_ipsec_sa *)((uintptr_t)ctx->satbl + sz * i);
+		lsa = ctx->sa + idx;
+
+		rc = ipsec_sa_init(lsa, sa, sz);
+	}
+
+	return rc;
+}
+
 void
 sa_init(struct socket_ctx *ctx, int32_t socket_id)
 {
+	int32_t rc;
 	const char *name;
 
 	if (ctx == NULL)
@@ -926,6 +1127,14 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id)
 				name, socket_id);
 
 		sa_in_add_rules(ctx->sa_in, sa_in, nb_sa_in);
+
+		if (app_sa_prm.enable != 0) {
+			rc = ipsec_satbl_init(ctx->sa_in, sa_in, nb_sa_in,
+				socket_id);
+			if (rc != 0)
+				rte_exit(EXIT_FAILURE,
+					"failed to init inbound SAs\n");
+		}
 	} else
 		RTE_LOG(WARNING, IPSEC, "No SA Inbound rule specified\n");
 
@@ -938,6 +1147,14 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id)
 				name, socket_id);
 
 		sa_out_add_rules(ctx->sa_out, sa_out, nb_sa_out);
+
+		if (app_sa_prm.enable != 0) {
+			rc = ipsec_satbl_init(ctx->sa_out, sa_out, nb_sa_out,
+				socket_id);
+			if (rc != 0)
+				rte_exit(EXIT_FAILURE,
+					"failed to init outbound SAs\n");
+		}
 	} else
 		RTE_LOG(WARNING, IPSEC, "No SA Outbound rule "
 			"specified\n");
diff --git a/examples/ipsec-secgw/sp4.c b/examples/ipsec-secgw/sp4.c
index 6b05daaa9..d1dc64bad 100644
--- a/examples/ipsec-secgw/sp4.c
+++ b/examples/ipsec-secgw/sp4.c
@@ -504,3 +504,28 @@ sp4_init(struct socket_ctx *ctx, int32_t socket_id)
 		RTE_LOG(WARNING, IPSEC, "No IPv4 SP Outbound rule "
 			"specified\n");
 }
+
+/*
+ * Search though SP rules for given SPI.
+ */
+int
+sp4_spi_present(uint32_t spi, int inbound)
+{
+	uint32_t i, num;
+	const struct acl4_rules *acr;
+
+	if (inbound != 0) {
+		acr = acl4_rules_in;
+		num = nb_acl4_rules_in;
+	} else {
+		acr = acl4_rules_out;
+		num = nb_acl4_rules_out;
+	}
+
+	for (i = 0; i != num; i++) {
+		if (acr[i].data.userdata == PROTECT(spi))
+			return i;
+	}
+
+	return -ENOENT;
+}
diff --git a/examples/ipsec-secgw/sp6.c b/examples/ipsec-secgw/sp6.c
index dc5b94c6a..e67d85aaf 100644
--- a/examples/ipsec-secgw/sp6.c
+++ b/examples/ipsec-secgw/sp6.c
@@ -618,3 +618,28 @@ sp6_init(struct socket_ctx *ctx, int32_t socket_id)
 		RTE_LOG(WARNING, IPSEC, "No IPv6 SP Outbound rule "
 			"specified\n");
 }
+
+/*
+ * Search though SP rules for given SPI.
+ */
+int
+sp6_spi_present(uint32_t spi, int inbound)
+{
+	uint32_t i, num;
+	const struct acl6_rules *acr;
+
+	if (inbound != 0) {
+		acr = acl6_rules_in;
+		num = nb_acl6_rules_in;
+	} else {
+		acr = acl6_rules_out;
+		num = nb_acl6_rules_out;
+	}
+
+	for (i = 0; i != num; i++) {
+		if (acr[i].data.userdata == PROTECT(spi))
+			return i;
+	}
+
+	return -ENOENT;
+}
-- 
2.17.1

[PATCH v7 08/10] examples/ipsec-secgw: make data-path to use ipsec library

[Konstantin Ananyev]

Changes to make ipsec-secgw data-path code to utilize librte_ipsec library.
Note that right now by default current (non-librte_ipsec) code-path will
be used. User has to run application with new command-line option ('-l')
to enable new codepath.

Signed-off-by: Mohammad Abdul Awal <mohammad.abdul.awal@intel.com>
Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/Makefile        |   1 +
 examples/ipsec-secgw/ipsec-secgw.c   | 179 ++++++++------
 examples/ipsec-secgw/ipsec.c         |   2 +-
 examples/ipsec-secgw/ipsec.h         |  23 ++
 examples/ipsec-secgw/ipsec_process.c | 357 +++++++++++++++++++++++++++
 examples/ipsec-secgw/meson.build     |   4 +-
 6 files changed, 487 insertions(+), 79 deletions(-)
 create mode 100644 examples/ipsec-secgw/ipsec_process.c

diff --git a/examples/ipsec-secgw/Makefile b/examples/ipsec-secgw/Makefile
index 3918ee63e..08f474da6 100644
--- a/examples/ipsec-secgw/Makefile
+++ b/examples/ipsec-secgw/Makefile
@@ -13,6 +13,7 @@ SRCS-y += sp4.c
 SRCS-y += sp6.c
 SRCS-y += sa.c
 SRCS-y += rt.c
+SRCS-y += ipsec_process.c
 SRCS-y += ipsec-secgw.c
 
 CFLAGS += -gdwarf-2
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 9174b3e56..045437659 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -229,19 +229,6 @@ static struct rte_eth_conf port_conf = {
 
 static struct socket_ctx socket_ctx[NB_SOCKETS];
 
-struct traffic_type {
-	const uint8_t *data[MAX_PKT_BURST * 2];
-	struct rte_mbuf *pkts[MAX_PKT_BURST * 2];
-	uint32_t res[MAX_PKT_BURST * 2];
-	uint32_t num;
-};
-
-struct ipsec_traffic {
-	struct traffic_type ipsec;
-	struct traffic_type ip4;
-	struct traffic_type ip6;
-};
-
 static inline void
 prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 {
@@ -258,6 +245,8 @@ prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 			t->ip4.data[t->ip4.num] = nlp;
 			t->ip4.pkts[(t->ip4.num)++] = pkt;
 		}
+		pkt->l2_len = 0;
+		pkt->l3_len = sizeof(struct ip);
 	} else if (eth->ether_type == rte_cpu_to_be_16(ETHER_TYPE_IPv6)) {
 		nlp = (uint8_t *)rte_pktmbuf_adj(pkt, ETHER_HDR_LEN);
 		nlp = RTE_PTR_ADD(nlp, offsetof(struct ip6_hdr, ip6_nxt));
@@ -267,6 +256,8 @@ prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 			t->ip6.data[t->ip6.num] = nlp;
 			t->ip6.pkts[(t->ip6.num)++] = pkt;
 		}
+		pkt->l2_len = 0;
+		pkt->l3_len = sizeof(struct ip6_hdr);
 	} else {
 		/* Unknown/Unsupported type, drop the packet */
 		RTE_LOG(ERR, IPSEC, "Unsupported packet type\n");
@@ -516,10 +507,15 @@ process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
 	n_ip4 = traffic->ip4.num;
 	n_ip6 = traffic->ip6.num;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
-
-	split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+	if (app_sa_prm.enable == 0) {
+		nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.num, MAX_PKT_BURST);
+		split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+	} else {
+		inbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
+	}
 
 	inbound_sp_sa(ipsec_ctx->sp4_ctx, ipsec_ctx->sa_ctx, &traffic->ip4,
 			n_ip4);
@@ -575,20 +571,27 @@ process_pkts_outbound(struct ipsec_ctx *ipsec_ctx,
 
 	outbound_sp(ipsec_ctx->sp6_ctx, &traffic->ip6, &traffic->ipsec);
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.res, traffic->ipsec.num,
-			MAX_PKT_BURST);
-
-	for (i = 0; i < nb_pkts_out; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
-		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-		} else {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
+	if (app_sa_prm.enable == 0) {
+
+		nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.res, traffic->ipsec.num,
+				MAX_PKT_BURST);
+
+		for (i = 0; i < nb_pkts_out; i++) {
+			m = traffic->ipsec.pkts[i];
+			struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
+			if (ip->ip_v == IPVERSION) {
+				idx = traffic->ip4.num++;
+				traffic->ip4.pkts[idx] = m;
+			} else {
+				idx = traffic->ip6.num++;
+				traffic->ip6.pkts[idx] = m;
+			}
 		}
+	} else {
+		outbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.res,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -611,19 +614,26 @@ process_pkts_inbound_nosp(struct ipsec_ctx *ipsec_ctx,
 
 	traffic->ip6.num = 0;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
+	if (app_sa_prm.enable == 0) {
 
-	for (i = 0; i < nb_pkts_in; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
-		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-		} else {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
+		nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.num, MAX_PKT_BURST);
+
+		for (i = 0; i < nb_pkts_in; i++) {
+			m = traffic->ipsec.pkts[i];
+			struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
+			if (ip->ip_v == IPVERSION) {
+				idx = traffic->ip4.num++;
+				traffic->ip4.pkts[idx] = m;
+			} else {
+				idx = traffic->ip6.num++;
+				traffic->ip6.pkts[idx] = m;
+			}
 		}
+	} else {
+		inbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -655,21 +665,28 @@ process_pkts_outbound_nosp(struct ipsec_ctx *ipsec_ctx,
 	traffic->ip6.num = 0;
 	traffic->ipsec.num = n;
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.res, traffic->ipsec.num,
-			MAX_PKT_BURST);
+	if (app_sa_prm.enable == 0) {
 
-	/* They all sue the same SA (ip4 or ip6 tunnel) */
-	m = traffic->ipsec.pkts[i];
-	ip = rte_pktmbuf_mtod(m, struct ip *);
-	if (ip->ip_v == IPVERSION) {
-		traffic->ip4.num = nb_pkts_out;
-		for (i = 0; i < nb_pkts_out; i++)
-			traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+		nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.res, traffic->ipsec.num,
+				MAX_PKT_BURST);
+
+		/* They all sue the same SA (ip4 or ip6 tunnel) */
+		m = traffic->ipsec.pkts[0];
+		ip = rte_pktmbuf_mtod(m, struct ip *);
+		if (ip->ip_v == IPVERSION) {
+			traffic->ip4.num = nb_pkts_out;
+			for (i = 0; i < nb_pkts_out; i++)
+				traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+		} else {
+			traffic->ip6.num = nb_pkts_out;
+			for (i = 0; i < nb_pkts_out; i++)
+				traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+		}
 	} else {
-		traffic->ip6.num = nb_pkts_out;
-		for (i = 0; i < nb_pkts_out; i++)
-			traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+		outbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.res,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -870,25 +887,31 @@ drain_inbound_crypto_queues(const struct lcore_conf *qconf,
 	uint32_t n;
 	struct ipsec_traffic trf;
 
-	/* dequeue packets from crypto-queue */
-	n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+	if (app_sa_prm.enable == 0) {
+
+		/* dequeue packets from crypto-queue */
+		n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
 			RTE_DIM(trf.ipsec.pkts));
-	if (n == 0)
-		return;
 
-	trf.ip4.num = 0;
-	trf.ip6.num = 0;
+		trf.ip4.num = 0;
+		trf.ip6.num = 0;
 
-	/* split traffic by ipv4-ipv6 */
-	split46_traffic(&trf, trf.ipsec.pkts, n);
+		/* split traffic by ipv4-ipv6 */
+		split46_traffic(&trf, trf.ipsec.pkts, n);
+	} else
+		ipsec_cqp_process(ctx, &trf);
 
 	/* process ipv4 packets */
-	inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
-	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	if (trf.ip4.num != 0) {
+		inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
+		route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	}
 
 	/* process ipv6 packets */
-	inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
-	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	if (trf.ip6.num != 0) {
+		inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
+		route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	}
 }
 
 static void
@@ -898,23 +921,27 @@ drain_outbound_crypto_queues(const struct lcore_conf *qconf,
 	uint32_t n;
 	struct ipsec_traffic trf;
 
-	/* dequeue packets from crypto-queue */
-	n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+	if (app_sa_prm.enable == 0) {
+
+		/* dequeue packets from crypto-queue */
+		n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
 			RTE_DIM(trf.ipsec.pkts));
-	if (n == 0)
-		return;
 
-	trf.ip4.num = 0;
-	trf.ip6.num = 0;
+		trf.ip4.num = 0;
+		trf.ip6.num = 0;
 
-	/* split traffic by ipv4-ipv6 */
-	split46_traffic(&trf, trf.ipsec.pkts, n);
+		/* split traffic by ipv4-ipv6 */
+		split46_traffic(&trf, trf.ipsec.pkts, n);
+	} else
+		ipsec_cqp_process(ctx, &trf);
 
 	/* process ipv4 packets */
-	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	if (trf.ip4.num != 0)
+		route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
 
 	/* process ipv6 packets */
-	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	if (trf.ip6.num != 0)
+		route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
 }
 
 /* main processing loop */
diff --git a/examples/ipsec-secgw/ipsec.c b/examples/ipsec-secgw/ipsec.c
index 72a29bcb1..36ead5cf9 100644
--- a/examples/ipsec-secgw/ipsec.c
+++ b/examples/ipsec-secgw/ipsec.c
@@ -39,7 +39,7 @@ set_ipsec_conf(struct ipsec_sa *sa, struct rte_security_ipsec_xform *ipsec)
 	ipsec->esn_soft_limit = IPSEC_OFFLOAD_ESN_SOFTLIMIT;
 }
 
-static inline int
+int
 create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa)
 {
 	struct rte_cryptodev_info cdev_info;
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 8da86c4de..8629f5045 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -190,6 +190,20 @@ struct cnt_blk {
 	uint32_t cnt;
 } __attribute__((packed));
 
+struct traffic_type {
+	const uint8_t *data[MAX_PKT_BURST * 2];
+	struct rte_mbuf *pkts[MAX_PKT_BURST * 2];
+	struct ipsec_sa *saptr[MAX_PKT_BURST * 2];
+	uint32_t res[MAX_PKT_BURST * 2];
+	uint32_t num;
+};
+
+struct ipsec_traffic {
+	struct traffic_type ipsec;
+	struct traffic_type ip4;
+	struct traffic_type ip6;
+};
+
 uint16_t
 ipsec_inbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint16_t nb_pkts, uint16_t len);
@@ -206,6 +220,12 @@ uint16_t
 ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint16_t len);
 
+void
+ipsec_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf);
+
+void
+ipsec_cqp_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf);
+
 static inline uint16_t
 ipsec_metadata_size(void)
 {
@@ -283,4 +303,7 @@ add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
 void
 enqueue_cop_burst(struct cdev_qp *cqp);
 
+int
+create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/ipsec_process.c b/examples/ipsec-secgw/ipsec_process.c
new file mode 100644
index 000000000..e403c461a
--- /dev/null
+++ b/examples/ipsec-secgw/ipsec_process.c
@@ -0,0 +1,357 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2017 Intel Corporation
+ */
+#include <sys/types.h>
+#include <netinet/in.h>
+#include <netinet/ip.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_log.h>
+#include <rte_cryptodev.h>
+#include <rte_ethdev.h>
+#include <rte_mbuf.h>
+
+#include "ipsec.h"
+
+#define SATP_OUT_IPV4(t)	\
+	((((t) & RTE_IPSEC_SATP_MODE_MASK) == RTE_IPSEC_SATP_MODE_TRANS && \
+	(((t) & RTE_IPSEC_SATP_IPV_MASK) == RTE_IPSEC_SATP_IPV4)) || \
+	((t) & RTE_IPSEC_SATP_MODE_MASK) == RTE_IPSEC_SATP_MODE_TUNLV4)
+
+
+/* helper routine to free bulk of packets */
+static inline void
+free_pkts(struct rte_mbuf *mb[], uint32_t n)
+{
+	uint32_t i;
+
+	for (i = 0; i != n; i++)
+		rte_pktmbuf_free(mb[i]);
+}
+
+/* helper routine to free bulk of crypto-ops and related packets */
+static inline void
+free_cops(struct rte_crypto_op *cop[], uint32_t n)
+{
+	uint32_t i;
+
+	for (i = 0; i != n; i++)
+		rte_pktmbuf_free(cop[i]->sym->m_src);
+}
+
+/* helper routine to enqueue bulk of crypto ops */
+static inline void
+enqueue_cop_bulk(struct cdev_qp *cqp, struct rte_crypto_op *cop[], uint32_t num)
+{
+	uint32_t i, k, len, n;
+
+	len = cqp->len;
+
+	/*
+	 * if cqp is empty and we have enough ops,
+	 * then queue them to the PMD straightway.
+	 */
+	if (num >= RTE_DIM(cqp->buf) * 3 / 4 && len == 0) {
+		n = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp, cop, num);
+		cqp->in_flight += n;
+		free_cops(cop + n, num - n);
+		return;
+	}
+
+	k = 0;
+
+	do {
+		n = RTE_DIM(cqp->buf) - len;
+		n = RTE_MIN(num - k, n);
+
+		/* put packets into cqp */
+		for (i = 0; i != n; i++)
+			cqp->buf[len + i] = cop[k + i];
+
+		len += n;
+		k += n;
+
+		/* if cqp is full then, enqueue crypto-ops to PMD */
+		if (len == RTE_DIM(cqp->buf)) {
+			n = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp,
+					cqp->buf, len);
+			cqp->in_flight += n;
+			free_cops(cqp->buf + n, len - n);
+			len = 0;
+		}
+
+
+	} while (k != num);
+
+	cqp->len = len;
+}
+
+static inline int
+fill_ipsec_session(struct rte_ipsec_session *ss, struct ipsec_ctx *ctx,
+	struct ipsec_sa *sa)
+{
+	int32_t rc;
+
+	/* setup crypto section */
+	if (ss->type == RTE_SECURITY_ACTION_TYPE_NONE) {
+		if (sa->crypto_session == NULL) {
+			rc = create_session(ctx, sa);
+			if (rc != 0)
+				return rc;
+		}
+		ss->crypto.ses = sa->crypto_session;
+	/* setup session action type */
+	} else {
+		if (sa->sec_session == NULL) {
+			rc = create_session(ctx, sa);
+			if (rc != 0)
+				return rc;
+		}
+		ss->security.ses = sa->sec_session;
+		ss->security.ctx = sa->security_ctx;
+		ss->security.ol_flags = sa->ol_flags;
+	}
+
+	rc = rte_ipsec_session_prepare(ss);
+	if (rc != 0)
+		memset(ss, 0, sizeof(*ss));
+
+	return rc;
+}
+
+/*
+ * group input packets byt the SA they belong to.
+ */
+static uint32_t
+sa_group(struct ipsec_sa *sa_ptr[], struct rte_mbuf *pkts[],
+	struct rte_ipsec_group grp[], uint32_t num)
+{
+	uint32_t i, n, spi;
+	void *sa;
+	void * const nosa = &spi;
+
+	sa = nosa;
+	for (i = 0, n = 0; i != num; i++) {
+
+		if (sa != sa_ptr[i]) {
+			grp[n].cnt = pkts + i - grp[n].m;
+			n += (sa != nosa);
+			grp[n].id.ptr = sa_ptr[i];
+			grp[n].m = pkts + i;
+			sa = sa_ptr[i];
+		}
+	}
+
+	/* terminate last group */
+	if (sa != nosa) {
+		grp[n].cnt = pkts + i - grp[n].m;
+		n++;
+	}
+
+	return n;
+}
+
+/*
+ * helper function, splits processed packets into ipv4/ipv6 traffic.
+ */
+static inline void
+copy_to_trf(struct ipsec_traffic *trf, uint64_t satp, struct rte_mbuf *mb[],
+	uint32_t num)
+{
+	uint32_t j, ofs, s;
+	struct traffic_type *out;
+
+	/*
+	 * determine traffic type(ipv4/ipv6) and offset for ACL classify
+	 * based on SA type
+	 */
+	if ((satp & RTE_IPSEC_SATP_DIR_MASK) == RTE_IPSEC_SATP_DIR_IB) {
+		if ((satp & RTE_IPSEC_SATP_IPV_MASK) == RTE_IPSEC_SATP_IPV4) {
+			out = &trf->ip4;
+			ofs = offsetof(struct ip, ip_p);
+		} else {
+			out = &trf->ip6;
+			ofs = offsetof(struct ip6_hdr, ip6_nxt);
+		}
+	} else if (SATP_OUT_IPV4(satp)) {
+		out = &trf->ip4;
+		ofs = offsetof(struct ip, ip_p);
+	} else {
+		out = &trf->ip6;
+		ofs = offsetof(struct ip6_hdr, ip6_nxt);
+	}
+
+	for (j = 0, s = out->num; j != num; j++) {
+		out->data[s + j] = rte_pktmbuf_mtod_offset(mb[j],
+				void *, ofs);
+		out->pkts[s + j] = mb[j];
+	}
+
+	out->num += num;
+}
+
+/*
+ * Process ipsec packets.
+ * If packet belong to SA that is subject of inline-crypto,
+ * then process it immediately.
+ * Otherwise do necessary preparations and queue it to related
+ * crypto-dev queue.
+ */
+void
+ipsec_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf)
+{
+	uint64_t satp;
+	uint32_t i, j, k, n;
+	struct ipsec_sa *sa;
+	struct ipsec_mbuf_metadata *priv;
+	struct rte_ipsec_group *pg;
+	struct rte_ipsec_session *ips;
+	struct cdev_qp *cqp;
+	struct rte_crypto_op *cop[RTE_DIM(trf->ipsec.pkts)];
+	struct rte_ipsec_group grp[RTE_DIM(trf->ipsec.pkts)];
+
+	n = sa_group(trf->ipsec.saptr, trf->ipsec.pkts, grp, trf->ipsec.num);
+
+	for (i = 0; i != n; i++) {
+
+		pg = grp + i;
+		sa = pg->id.ptr;
+
+		/* no valid SA found */
+		if (sa == NULL)
+			k = 0;
+
+		ips = &sa->ips;
+		satp = rte_ipsec_sa_type(ips->sa);
+
+		/* no valid HW session for that SA, try to create one */
+		if (ips->crypto.ses == NULL &&
+				fill_ipsec_session(ips, ctx, sa) != 0)
+			k = 0;
+
+		/* process packets inline */
+		else if (sa->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				sa->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL) {
+
+			/*
+			 * This is just to satisfy inbound_sa_check()
+			 * and get_hop_for_offload_pkt().
+			 * Should be removed in future.
+			 */
+			for (j = 0; j != pg->cnt; j++) {
+				priv = get_priv(pg->m[j]);
+				priv->sa = sa;
+			}
+
+			k = rte_ipsec_pkt_process(ips, pg->m, pg->cnt);
+			copy_to_trf(trf, satp, pg->m, k);
+
+		/* enqueue packets to crypto dev */
+		} else {
+
+			cqp = &ctx->tbl[sa->cdev_id_qp];
+
+			/* for that app each mbuf has it's own crypto op */
+			for (j = 0; j != pg->cnt; j++) {
+				priv = get_priv(pg->m[j]);
+				cop[j] = &priv->cop;
+				/*
+				 * this is just to satisfy inbound_sa_check()
+				 * should be removed in future.
+				 */
+				priv->sa = sa;
+			}
+
+			/* prepare and enqueue crypto ops */
+			k = rte_ipsec_pkt_crypto_prepare(ips, pg->m, cop,
+				pg->cnt);
+			if (k != 0)
+				enqueue_cop_bulk(cqp, cop, k);
+		}
+
+		/* drop packets that cannot be enqueued/processed */
+		if (k != pg->cnt)
+			free_pkts(pg->m + k, pg->cnt - k);
+	}
+}
+
+static inline uint32_t
+cqp_dequeue(struct cdev_qp *cqp, struct rte_crypto_op *cop[], uint32_t num)
+{
+	uint32_t n;
+
+	if (cqp->in_flight == 0)
+		return 0;
+
+	n = rte_cryptodev_dequeue_burst(cqp->id, cqp->qp, cop, num);
+	RTE_ASSERT(cqp->in_flight >= n);
+	cqp->in_flight -= n;
+
+	return n;
+}
+
+static inline uint32_t
+ctx_dequeue(struct ipsec_ctx *ctx, struct rte_crypto_op *cop[], uint32_t num)
+{
+	uint32_t i, n;
+
+	n = 0;
+
+	for (i = ctx->last_qp; n != num && i != ctx->nb_qps; i++)
+		n += cqp_dequeue(ctx->tbl + i, cop + n, num - n);
+
+	for (i = 0; n != num && i != ctx->last_qp; i++)
+		n += cqp_dequeue(ctx->tbl + i, cop + n, num - n);
+
+	ctx->last_qp = i;
+	return n;
+}
+
+/*
+ * dequeue packets from crypto-queues and finalize processing.
+ */
+void
+ipsec_cqp_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf)
+{
+	uint64_t satp;
+	uint32_t i, k, n, ng;
+	struct rte_ipsec_session *ss;
+	struct traffic_type *out;
+	struct rte_ipsec_group *pg;
+	struct rte_crypto_op *cop[RTE_DIM(trf->ipsec.pkts)];
+	struct rte_ipsec_group grp[RTE_DIM(trf->ipsec.pkts)];
+
+	trf->ip4.num = 0;
+	trf->ip6.num = 0;
+
+	out = &trf->ipsec;
+
+	/* dequeue completed crypto-ops */
+	n = ctx_dequeue(ctx, cop, RTE_DIM(cop));
+	if (n == 0)
+		return;
+
+	/* group them by ipsec session */
+	ng = rte_ipsec_pkt_crypto_group((const struct rte_crypto_op **)
+		(uintptr_t)cop, out->pkts, grp, n);
+
+	/* process each group of packets */
+	for (i = 0; i != ng; i++) {
+
+		pg = grp + i;
+		ss = pg->id.ptr;
+		satp = rte_ipsec_sa_type(ss->sa);
+
+		k = rte_ipsec_pkt_process(ss, pg->m, pg->cnt);
+		copy_to_trf(trf, satp, pg->m, k);
+
+		/* free bad packets, if any */
+		free_pkts(pg->m + k, pg->cnt - k);
+
+		n -= pg->cnt;
+	}
+
+	/* we should never have packet with unknown SA here */
+	RTE_VERIFY(n == 0);
+}
diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
index 31f68fee2..81c146ebc 100644
--- a/examples/ipsec-secgw/meson.build
+++ b/examples/ipsec-secgw/meson.build
@@ -9,6 +9,6 @@
 deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
 allow_experimental_apis = true
 sources = files(
-	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
-	'rt.c', 'sa.c', 'sp4.c', 'sp6.c'
+	'esp.c', 'ipsec.c', 'ipsec_process.c', 'ipsec-secgw.c',
+	'parser.c', 'rt.c', 'sa.c', 'sp4.c', 'sp6.c'
 )
-- 
2.17.1

[PATCH v7 09/10] examples/ipsec-secgw: add scripts for functional test

[Konstantin Ananyev]

The purpose of these scripts is to automate ipsec-secgw functional testing.
The scripts require two machines (SUT and DUT) connected through
at least 2 NICs and running linux (so far tested only on Ubuntu 18.04).
Introduced test-cases for the following scenarios:
- Transport/Tunnel modes
- AES-CBC SHA1
- AES-GCM
- ESN on/off
- legacy/librte_ipsec code path

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/test/common_defs.sh      | 162 ++++++++++++++++++
 examples/ipsec-secgw/test/data_rxtx.sh        |  62 +++++++
 examples/ipsec-secgw/test/linux_test4.sh      |  63 +++++++
 examples/ipsec-secgw/test/linux_test6.sh      |  64 +++++++
 examples/ipsec-secgw/test/run_test.sh         |  80 +++++++++
 .../test/trs_aescbc_sha1_common_defs.sh       |  69 ++++++++
 .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  67 ++++++++
 .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/trs_aescbc_sha1_esn_defs.sh          |  66 +++++++
 .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
 .../test/trs_aesgcm_common_defs.sh            |  60 +++++++
 examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  76 ++++++++
 .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  66 +++++++
 .../test/trs_aesgcm_inline_crypto_defs.sh     |   6 +
 .../test/trs_aesgcm_inline_crypto_old_defs.sh |   5 +
 .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
 .../test/tun_aescbc_sha1_common_defs.sh       |  68 ++++++++
 .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  70 ++++++++
 .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/tun_aescbc_sha1_esn_defs.sh          |  70 ++++++++
 .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
 .../test/tun_aesgcm_common_defs.sh            |  60 +++++++
 examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  80 +++++++++
 .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  70 ++++++++
 .../test/tun_aesgcm_inline_crypto_defs.sh     |   6 +
 .../test/tun_aesgcm_inline_crypto_old_defs.sh |   5 +
 .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
 29 files changed, 1315 insertions(+)
 create mode 100644 examples/ipsec-secgw/test/common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test4.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test6.sh
 create mode 100644 examples/ipsec-secgw/test/run_test.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh

diff --git a/examples/ipsec-secgw/test/common_defs.sh b/examples/ipsec-secgw/test/common_defs.sh
new file mode 100644
index 000000000..1ed31f89f
--- /dev/null
+++ b/examples/ipsec-secgw/test/common_defs.sh
@@ -0,0 +1,162 @@
+#! /bin/bash
+
+#check that env vars are properly defined
+
+#check SGW_PATH
+if [[ -z "${SGW_PATH}" || ! -x ${SGW_PATH} ]]; then
+	echo "SGW_PATH is invalid"
+	exit 127
+fi
+
+#check ETH_DEV
+if [[ -z "${ETH_DEV}" ]]; then
+	echo "ETH_DEV is invalid"
+	exit 127
+fi
+
+#setup SGW_LCORE
+SGW_LCORE=${SGW_LCORE:-0}
+
+#check that REMOTE_HOST is reachable
+ssh ${REMOTE_HOST} echo
+st=$?
+if [[ $st -ne 0 ]]; then
+	echo "host ${REMOTE_HOST} is not reachable"
+	exit $st
+fi
+
+#get ether addr of REMOTE_HOST
+REMOTE_MAC=`ssh ${REMOTE_HOST} ip addr show dev ${REMOTE_IFACE}`
+st=$?
+REMOTE_MAC=`echo ${REMOTE_MAC} | sed -e 's/^.*ether //' -e 's/ brd.*$//'`
+if [[ $st -ne 0 || -z "${REMOTE_MAC}" ]]; then
+	echo "coouldn't retrieve ether addr from ${REMOTE_IFACE}"
+	exit 127
+fi
+
+LOCAL_IFACE=dtap0
+
+LOCAL_MAC="00:64:74:61:70:30"
+
+REMOTE_IPV4=192.168.31.14
+LOCAL_IPV4=192.168.31.92
+
+REMOTE_IPV6=fd12:3456:789a:0031:0000:0000:0000:0014
+LOCAL_IPV6=fd12:3456:789a:0031:0000:0000:0000:0092
+
+DPDK_PATH=${RTE_SDK:-${PWD}}
+DPDK_BUILD=${RTE_TARGET:-x86_64-native-linuxapp-gcc}
+
+SGW_OUT_FILE=./ipsec-secgw.out1
+
+SGW_CMD_EAL_PRM="--lcores=${SGW_LCORE} -n 4 ${ETH_DEV}"
+SGW_CMD_CFG="(0,0,${SGW_LCORE}),(1,0,${SGW_LCORE})"
+SGW_CMD_PRM="-p 0x3 -u 1 -P --config=\"${SGW_CMD_CFG}\""
+
+SGW_CFG_FILE=$(tempfile)
+
+# configure local host/ifaces
+config_local_iface()
+{
+	ifconfig ${LOCAL_IFACE} ${LOCAL_IPV4}/24 mtu 1400 up
+	ifconfig ${LOCAL_IFACE}
+
+	ip neigh flush dev ${LOCAL_IFACE}
+	ip neigh add ${REMOTE_IPV4} dev ${LOCAL_IFACE} lladdr ${REMOTE_MAC}
+	ip neigh show dev ${LOCAL_IFACE}
+}
+
+config6_local_iface()
+{
+	config_local_iface
+
+	sysctl -w net.ipv6.conf.${LOCAL_IFACE}.disable_ipv6=0
+	ip addr add  ${LOCAL_IPV6}/64 dev ${LOCAL_IFACE}
+
+	sysctl -w net.ipv6.conf.${LOCAL_IFACE}.mtu=1300
+
+	ip -6 neigh add ${REMOTE_IPV6} dev ${LOCAL_IFACE} lladdr ${REMOTE_MAC}
+	ip neigh show dev ${LOCAL_IFACE}
+}
+
+#configure remote host/iface
+config_remote_iface()
+{
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE} down
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE} ${REMOTE_IPV4}/24 up
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} ip neigh flush dev ${REMOTE_IFACE}
+
+	# by some reason following ip neigh doesn't work for me here properly:
+	#ssh ${REMOTE_HOST} ip neigh add ${LOCAL_IPV4} \
+	#		dev ${REMOTE_IFACE} lladr ${LOCAL_MAC}
+	# so used arp instead.
+	ssh ${REMOTE_HOST} arp -i ${REMOTE_IFACE} -s ${LOCAL_IPV4} ${LOCAL_MAC}
+	ssh ${REMOTE_HOST} ip neigh show dev ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} iptables --flush
+}
+
+config6_remote_iface()
+{
+	config_remote_iface
+
+	ssh ${REMOTE_HOST} sysctl -w \
+		net.ipv6.conf.${REMOTE_IFACE}.disable_ipv6=0
+	ssh ${REMOTE_HOST} ip addr add  ${REMOTE_IPV6}/64 dev ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} ip -6 neigh add ${LOCAL_IPV6} \
+		dev ${REMOTE_IFACE} lladdr ${LOCAL_MAC}
+	ssh ${REMOTE_HOST} ip neigh show dev ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} ip6tables --flush
+}
+
+#configure remote and local host/iface
+config_iface()
+{
+	config_local_iface
+	config_remote_iface
+}
+
+config6_iface()
+{
+	config6_local_iface
+	config6_remote_iface
+}
+
+#start ipsec-secgw
+secgw_start()
+{
+	SGW_EXEC_FILE=$(tempfile)
+	cat <<EOF > ${SGW_EXEC_FILE}
+${SGW_PATH} ${SGW_CMD_EAL_PRM} ${CRYPTO_DEV} \
+--vdev="net_tap0,mac=fixed" \
+-- ${SGW_CMD_PRM} ${SGW_CMD_XPRM} -f ${SGW_CFG_FILE} > \
+${SGW_OUT_FILE} 2>&1 &
+p=\$!
+echo \$p
+EOF
+
+	cat ${SGW_EXEC_FILE}
+	SGW_PID=`/bin/bash -x ${SGW_EXEC_FILE}`
+
+	# wait till ipsec-secgw start properly
+	i=0
+	st=1
+	while [[ $i -ne 10 && st -ne 0 ]]; do
+		sleep 1
+		ifconfig ${LOCAL_IFACE}
+		st=$?
+		let i++
+	done
+}
+
+#stop ipsec-secgw and cleanup
+secgw_stop()
+{
+	kill ${SGW_PID}
+	rm -f ${SGW_EXEC_FILE}
+	rm -f ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/data_rxtx.sh b/examples/ipsec-secgw/test/data_rxtx.sh
new file mode 100644
index 000000000..f23a6d594
--- /dev/null
+++ b/examples/ipsec-secgw/test/data_rxtx.sh
@@ -0,0 +1,62 @@
+#! /bin/bash
+
+TCP_PORT=22222
+
+ping_test1()
+{
+	dst=$1
+
+	i=0
+	st=0
+	while [[ $i -ne 1200 && $st -eq 0 ]];
+	do
+		let i++
+		ping -c 1 -s ${i} ${dst}
+		st=$?
+	done
+
+	if [[ $st -ne 0 ]]; then
+		echo "ERROR: $0 failed for dst=${dst}, sz=${i}"
+	fi
+	return $st;
+}
+
+ping6_test1()
+{
+	dst=$1
+
+	i=0
+	st=0
+	while [[ $i -ne 1200 && $st -eq 0 ]];
+	do
+		let i++
+		ping6 -c 1 -s ${i} ${dst}
+		st=$?
+	done
+
+	if [[ $st -ne 0 ]]; then
+		echo "ERROR: $0 failed for dst=${dst}, sz=${i}"
+	fi
+	return $st;
+}
+
+scp_test1()
+{
+	dst=$1
+
+	for sz in 1234 23456 345678 4567890 56789102 ; do
+		x=`basename $0`.${sz}
+		dd if=/dev/urandom of=${x} bs=${sz} count=1
+		scp ${x} [${dst}]:${x}
+		scp [${dst}]:${x} ${x}.copy1
+		diff -u ${x} ${x}.copy1
+		st=$?
+		rm -f ${x} ${x}.copy1
+		ssh ${REMOTE_HOST} rm -f ${x}
+		if [[ $st -ne 0 ]]; then
+			return $st
+		fi
+	done
+
+	return 0;
+}
diff --git a/examples/ipsec-secgw/test/linux_test4.sh b/examples/ipsec-secgw/test/linux_test4.sh
new file mode 100644
index 000000000..d636f5604
--- /dev/null
+++ b/examples/ipsec-secgw/test/linux_test4.sh
@@ -0,0 +1,63 @@
+#! /bin/bash
+
+# usage:  /bin/bash linux_test4.sh <ipsec_mode>
+# for list of available modes please refer to run_test.sh.
+# ipsec-secgw (IPv4 mode) functional test script.
+#
+# Note that for most of them you required appropriate crypto PMD/device
+# to be avaialble.
+# Also user has to setup properly the following environment variables:
+#  SGW_PATH - path to the ipsec-secgw binary to test
+#  REMOTE_HOST - ip/hostname of the DUT
+#  REMOTE_IFACE - iface name for the test-port on DUT
+#  ETH_DEV - ethernet device to be used on SUT by DPDK ('-w <pci-id>')
+# Also user can optonally setup:
+#  SGW_LCORE - lcore to run ipsec-secgw on (default value is 0)
+#  CRYPTO_DEV - crypto device to be used ('-w <pci-id>')
+#  if none specified appropriate vdevs will be created by the scrit
+#
+# The purpose of the script is to automate ipsec-secgw testing
+# using another system running linux as a DUT.
+# It expects that SUT and DUT are connected through at least 2 NICs.
+# One NIC is expected to be managed by linux both machines,
+# and will be used as a control path
+# Make sure user from SUT can ssh to DUT without entering password.
+# Second NIC (test-port) should be reserved for DPDK on SUT,
+# and should be managed by linux on DUT.
+# The script starts ipsec-secgw with 2 NIC devices: test-port and tap vdev.
+# Then configures local tap iface and remote iface and ipsec policies
+# in the following way:
+# traffic going over test-port in both directions has to be
+# protected by ipsec.
+# raffic going over TAP in both directions doesn't have to be protected.
+# I.E:
+# DUT OS(NIC1)--(ipsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS
+# SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(ipsec)-->(NIC1)DUT OS
+# Then tries to perorm some data transfer using the scheme decribed above.
+#
+
+DIR=`dirname $0`
+MODE=$1
+
+ . ${DIR}/common_defs.sh
+ . ${DIR}/${MODE}_defs.sh
+
+config_secgw
+
+secgw_start
+
+config_iface
+
+config_remote_xfrm
+
+ . ${DIR}/data_rxtx.sh
+
+ping_test1 ${REMOTE_IPV4}
+st=$?
+if [[ $st -eq 0 ]]; then
+	scp_test1 ${REMOTE_IPV4}
+	st=$?
+fi
+
+secgw_stop
+exit $st
diff --git a/examples/ipsec-secgw/test/linux_test6.sh b/examples/ipsec-secgw/test/linux_test6.sh
new file mode 100644
index 000000000..e30f607d8
--- /dev/null
+++ b/examples/ipsec-secgw/test/linux_test6.sh
@@ -0,0 +1,64 @@
+#! /bin/bash
+
+# usage:  /bin/bash linux_test6.sh <ipsec_mode>
+# for list of available modes please refer to run_test.sh.
+# ipsec-secgw (IPv6 mode) functional test script.
+#
+# Note that for most of them you required appropriate crypto PMD/device
+# to be avaialble.
+# Also user has to setup properly the following environment variables:
+#  SGW_PATH - path to the ipsec-secgw binary to test
+#  REMOTE_HOST - ip/hostname of the DUT
+#  REMOTE_IFACE - iface name for the test-port on DUT
+#  ETH_DEV - ethernet device to be used on SUT by DPDK ('-w <pci-id>')
+# Also user can optonally setup:
+#  SGW_LCORE - lcore to run ipsec-secgw on (default value is 0)
+#  CRYPTO_DEV - crypto device to be used ('-w <pci-id>')
+#  if none specified appropriate vdevs will be created by the scrit
+#
+# The purpose of the script is to automate ipsec-secgw testing
+# using another system running linux as a DUT.
+# It expects that SUT and DUT are connected through at least 2 NICs.
+# One NIC is expected to be managed by linux both machines,
+# and will be used as a control path.
+# Make sure user from SUT can ssh to DUT without entering password,
+# also make sure that sshd over ipv6 is enabled.
+# Second NIC (test-port) should be reserved for DPDK on SUT,
+# and should be managed by linux on DUT.
+# The script starts ipsec-secgw with 2 NIC devices: test-port and tap vdev.
+# Then configures local tap iface and remote iface and ipsec policies
+# in the following way:
+# traffic going over test-port in both directions has to be
+# protected by ipsec.
+# raffic going over TAP in both directions doesn't have to be protected.
+# I.E:
+# DUT OS(NIC1)--(ipsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS
+# SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(ipsec)-->(NIC1)DUT OS
+# Then tries to perorm some data transfer using the scheme decribed above.
+#
+
+DIR=`dirname $0`
+MODE=$1
+
+ . ${DIR}/common_defs.sh
+ . ${DIR}/${MODE}_defs.sh
+
+config_secgw
+
+secgw_start
+
+config6_iface
+
+config6_remote_xfrm
+
+ . ${DIR}/data_rxtx.sh
+
+ping6_test1 ${REMOTE_IPV6}
+st=$?
+if [[ $st -eq 0 ]]; then
+	scp_test1 ${REMOTE_IPV6}
+	st=$?
+fi
+
+secgw_stop
+exit $st
diff --git a/examples/ipsec-secgw/test/run_test.sh b/examples/ipsec-secgw/test/run_test.sh
new file mode 100644
index 000000000..6dc0ce54e
--- /dev/null
+++ b/examples/ipsec-secgw/test/run_test.sh
@@ -0,0 +1,80 @@
+#! /bin/bash
+
+# usage: /bin/bash run_test.sh [-46]
+# Run all defined linux_test[4,6].sh test-cases one by one
+# user has to setup properly the following environment variables:
+#  SGW_PATH - path to the ipsec-secgw binary to test
+#  REMOTE_HOST - ip/hostname of the DUT
+#  REMOTE_IFACE - iface name for the test-port on DUT
+#  ETH_DEV - ethernet device to be used on SUT by DPDK ('-w <pci-id>')
+# Also user can optonally setup:
+#  SGW_LCORE - lcore to run ipsec-secgw on (default value is 0)
+#  CRYPTO_DEV - crypto device to be used ('-w <pci-id>')
+#  if none specified appropriate vdevs will be created by the scrit
+# refer to linux_test1.sh for more information
+
+# All supported modes to test.
+# naming convention:
+# 'old' means that ipsec-secgw will run in legacy (non-librte_ipsec mode)
+# 'tun/trs' refer to tunnel/transport mode respectively
+LINUX_TEST="tun_aescbc_sha1 \
+tun_aescbc_sha1_esn \
+tun_aescbc_sha1_esn_atom \
+tun_aesgcm \
+tun_aesgcm_esn \
+tun_aesgcm_esn_atom \
+trs_aescbc_sha1 \
+trs_aescbc_sha1_esn \
+trs_aescbc_sha1_esn_atom \
+trs_aesgcm \
+trs_aesgcm_esn \
+trs_aesgcm_esn_atom \
+tun_aescbc_sha1_old \
+tun_aesgcm_old \
+trs_aescbc_sha1_old \
+trs_aesgcm_old"
+
+DIR=`dirname $0`
+
+# get input options
+st=0
+run4=0
+run6=0
+while [[ ${st} -eq 0 ]]; do
+	getopts ":46" opt
+	st=$?
+	if [[ "${opt}" == "4" ]]; then
+		run4=1
+	elif [[ "${opt}" == "6" ]]; then
+		run6=1
+	fi
+done
+
+if [[ ${run4} -eq 0 && {run6} -eq 0 ]]; then
+	exit 127
+fi
+
+for i in ${LINUX_TEST}; do
+
+	echo "starting test ${i}"
+
+	st4=0
+	if [[ ${run4} -ne 0 ]]; then
+		/bin/bash ${DIR}/linux_test4.sh ${i}
+		st4=$?
+		echo "test4 ${i} finished with status ${st4}"
+	fi
+
+	st6=0
+	if [[ ${run6} -ne 0 ]]; then
+		/bin/bash ${DIR}/linux_test6.sh ${i}
+		st6=$?
+		echo "test6 ${i} finished with status ${st6}"
+	fi
+
+	let "st = st4 + st6"
+	if [[ $st -ne 0 ]]; then
+		echo "ERROR test ${i} FAILED"
+		exit $st
+	fi
+done
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
new file mode 100644
index 000000000..e2621e0df
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
@@ -0,0 +1,69 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_mb0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#SP in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#sp in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+sa in 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#SA out rules
+sa out 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#SA out rules
+sa out 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
new file mode 100644
index 000000000..d68552fce
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
@@ -0,0 +1,67 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
new file mode 100644
index 000000000..f16222e11
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
new file mode 100644
index 000000000..ce7c977a3
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
@@ -0,0 +1,66 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
new file mode 100644
index 000000000..a3abb6103
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
new file mode 100644
index 000000000..f6c5bf5a7
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
@@ -0,0 +1,60 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_gcm0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#SP in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport ${SGW_CFG_XPRM}
+
+sa in 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport ${SGW_CFG_XPRM}
+
+#SA out rules
+sa out 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport ${SGW_CFG_XPRM}
+
+sa out 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport ${SGW_CFG_XPRM}
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_defs.sh
new file mode 100644
index 000000000..a4d902be0
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_defs.sh
@@ -0,0 +1,76 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+
+	# to overcome problem with ipsec-secgw for inline mode,
+	# when first packet(s) will be always dropped.
+	# note that ping will fail here
+	ssh ${REMOTE_HOST} ping -c 1 ${LOCAL_IPV4}
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+
+	# to overcome problem with ipsec-secgw for inline mode,
+	# when first packet(s) will be always dropped.
+	# note that ping will fail here
+	ssh ${REMOTE_HOST} ping -c 1 ${LOCAL_IPV6}
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
new file mode 100644
index 000000000..80d8d63b8
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
new file mode 100644
index 000000000..94958d199
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
@@ -0,0 +1,66 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_defs.sh
new file mode 100644
index 000000000..de6048d68
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_defs.sh
@@ -0,0 +1,6 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_defs.sh
+
+CRYPTO_DEV='--vdev="crypto_null0"'
+SGW_CFG_XPRM='port_id 0 type inline-crypto-offload'
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_old_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_old_defs.sh
new file mode 100644
index 000000000..05230496f
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_inline_crypto_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_inline_crypto_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
new file mode 100644
index 000000000..951e6b68f
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
new file mode 100644
index 000000000..4025da232
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
@@ -0,0 +1,68 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_mb0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#sp in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#sp in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${REMOTE_IPV4} dst ${LOCAL_IPV4}
+
+sa in 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${REMOTE_IPV6} dst ${LOCAL_IPV6}
+
+#SA out rules
+sa out 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${LOCAL_IPV4} dst ${REMOTE_IPV4}
+
+sa out 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${LOCAL_IPV6} dst ${REMOTE_IPV6}
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
new file mode 100644
index 000000000..18aade3a9
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
new file mode 100644
index 000000000..6b4a82149
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
new file mode 100644
index 000000000..28c1125d6
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
new file mode 100644
index 000000000..3c0d8d1b1
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
new file mode 100644
index 000000000..278377967
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
@@ -0,0 +1,60 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_gcm0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#sp in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#sp in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${REMOTE_IPV4} dst ${LOCAL_IPV4} ${SGW_CFG_XPRM}
+
+sa in 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${REMOTE_IPV6} dst ${LOCAL_IPV6} ${SGW_CFG_XPRM}
+
+#SA out rules
+sa out 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${LOCAL_IPV4} dst ${REMOTE_IPV4} ${SGW_CFG_XPRM}
+
+sa out 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${LOCAL_IPV6} dst ${REMOTE_IPV6} ${SGW_CFG_XPRM}
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_defs.sh
new file mode 100644
index 000000000..1764ef681
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_defs.sh
@@ -0,0 +1,80 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+
+	# to overcome problem with ipsec-secgw for inline mode,
+	# when first packet(s) will be always dropped.
+	# note that ping will fail here
+	ssh ${REMOTE_HOST} ping -c 1 ${LOCAL_IPV4}
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+
+	# to overcome problem with ipsec-secgw for inline mode,
+	# when first packet(s) will be always dropped.
+	# note that ping will fail here
+	ssh ${REMOTE_HOST} ping6 -c 1 ${LOCAL_IPV6}
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
new file mode 100644
index 000000000..dab1460c8
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
new file mode 100644
index 000000000..606232349
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_defs.sh
new file mode 100644
index 000000000..eafefe7ae
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_defs.sh
@@ -0,0 +1,6 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_defs.sh
+
+CRYPTO_DEV='--vdev="crypto_null0"'
+SGW_CFG_XPRM='port_id 0 type inline-crypto-offload'
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_old_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_old_defs.sh
new file mode 100644
index 000000000..de659610c
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_inline_crypto_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_inline_crypto_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh
new file mode 100644
index 000000000..e0a015e21
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_defs.sh
+
+SGW_CMD_XPRM=
-- 
2.17.1

[PATCH v7 10/10] doc: update ipsec-secgw guide and relelase notes

[Konstantin Ananyev]

Update ipsec-secgw guide and relelase notes to reflect latest changes.

Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 doc/guides/rel_notes/release_19_02.rst   |  14 +++
 doc/guides/sample_app_ug/ipsec_secgw.rst | 105 ++++++++++++++++++++++-
 2 files changed, 117 insertions(+), 2 deletions(-)

diff --git a/doc/guides/rel_notes/release_19_02.rst b/doc/guides/rel_notes/release_19_02.rst
index 1a9885c44..28dbe3ad0 100644
--- a/doc/guides/rel_notes/release_19_02.rst
+++ b/doc/guides/rel_notes/release_19_02.rst
@@ -116,6 +116,20 @@ New Features
 
   See :doc:`../prog_guide/ipsec_lib` for more information.
 
+* **Updated the ipsec-secgw sample application.**
+
+  The ``ipsec-secgw`` sample application has been updated to use the new
+  ``librte_ipsec`` library also added in this release.
+  The original functionality of ipsec-secgw is retained, a new command line
+  parameter ``-l`` has  been added to ipsec-secgw to use the IPsec library,
+  instead of the existing IPsec code in the application.
+
+  The IPsec library does not support all the functionality of the existing
+  ipsec-secgw application, its is planned to add the outstanding functionality
+  in future releases.
+
+  See :doc:`../sample_app_ug/ipsec_secgw` for more information.
+
 
 Removed Items
 -------------
diff --git a/doc/guides/sample_app_ug/ipsec_secgw.rst b/doc/guides/sample_app_ug/ipsec_secgw.rst
index 61638e733..3d784e705 100644
--- a/doc/guides/sample_app_ug/ipsec_secgw.rst
+++ b/doc/guides/sample_app_ug/ipsec_secgw.rst
@@ -76,7 +76,7 @@ Compiling the Application
 
 To compile the sample application see :doc:`compiling`.
 
-The application is located in the ``rpsec-secgw`` sub-directory.
+The application is located in the ``ipsec-secgw`` sub-directory.
 
 #. [Optional] Build the application for debugging:
    This option adds some extra flags, disables compiler optimizations and
@@ -93,6 +93,7 @@ The application has a number of command line options::
 
    ./build/ipsec-secgw [EAL options] --
                         -p PORTMASK -P -u PORTMASK -j FRAMESIZE
+                        -l -w REPLAY_WINOW_SIZE -e -a
                         --config (port,queue,lcore)[,(port,queue,lcore]
                         --single-sa SAIDX
                         --rxoffload MASK
@@ -114,6 +115,18 @@ Where:
     specified as FRAMESIZE. If an invalid value is provided as FRAMESIZE
     then the default value 9000 is used.
 
+*   ``-l``: enables code-path that uses librte_ipsec.
+
+*   ``-w REPLAY_WINOW_SIZE``: specifies the IPsec sequence number replay window
+    size for each Security Association (available only with librte_ipsec
+    code path).
+
+*   ``-e``: enables Security Association extended sequence number processing
+    (available only with librte_ipsec code path).
+
+*   ``-a``: enables Security Association sequence number atomic behaviour
+    (available only with librte_ipsec code path).
+
 *   ``--config (port,queue,lcore)[,(port,queue,lcore)]``: determines which queues
     from which ports are mapped to which cores.
 
@@ -225,7 +238,7 @@ accordingly.
 
 
 Configuration File Syntax
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~
 
 As mention in the overview, the Security Policies are ACL rules.
 The application parsers the rules specified in the configuration file and
@@ -571,6 +584,11 @@ Example SA rules:
     mode ipv4-tunnel src 172.16.1.5 dst 172.16.2.5 \
     type lookaside-protocol-offload port_id 4
 
+    sa in 35 aead_algo aes-128-gcm \
+    aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+    mode ipv4-tunnel src 172.16.2.5 dst 172.16.1.5 \
+    type inline-crypto-offload port_id 0
+
 Routing rule syntax
 ^^^^^^^^^^^^^^^^^^^
 
@@ -667,3 +685,86 @@ Example Neighbour rules:
 .. code-block:: console
 
     neigh port 0 DE:AD:BE:EF:01:02
+
+Test directory
+--------------
+
+The test directory contains scripts for testing the various encryption
+algorithms.
+
+The purpose of the scripts is to automate ipsec-secgw testing
+using another system running linux as a DUT.
+
+The user must setup the following environment variables:
+
+*   ``SGW_PATH``: path to the ipsec-secgw binary to test.
+
+*   ``REMOTE_HOST``: IP address/hostname of the DUT.
+
+*   ``REMOTE_IFACE``: interface name for the test-port on the DUT.
+
+*   ``ETH_DEV``: ethernet device to be used on the SUT by DPDK ('-w <pci-id>')
+
+Also the user can optionally setup:
+
+*   ``SGW_LCORE``: lcore to run ipsec-secgw on (default value is 0)
+
+*   ``CRYPTO_DEV``: crypto device to be used ('-w <pci-id>'). If none specified
+    appropriate vdevs will be created by the script
+
+Note that most of the tests require the appropriate crypto PMD/device to be
+available.
+
+Server configuration
+~~~~~~~~~~~~~~~~~~~~
+
+Two servers are required for the tests, SUT and DUT.
+
+Make sure the user from the SUT can ssh to the DUT without entering the password.
+To enable this feature keys must be setup on the DUT.
+
+``ssh-keygen`` will make a private & public key pair on the SUT.
+
+``ssh-copy-id`` <user name>@<target host name> on the SUT will copy the public
+key to the DUT. It will ask for credentials so that it can upload the public key.
+
+The SUT and DUT are connected through at least 2 NIC ports.
+
+One NIC port is expected to be managed by linux on both machines and will be
+used as a control path.
+
+The second NIC port (test-port) should be bound to DPDK on the SUT, and should
+be managed by linux on the DUT.
+
+The script starts ``ipsec-secgw`` with 2 NIC devices: ``test-port`` and
+``tap vdev``.
+
+It then configures the local tap interface and the remote interface and IPsec
+policies in the following way:
+
+Traffic going over the test-port in both directions has to be protected by IPsec.
+
+Traffic going over the TAP port in both directions does not have to be protected.
+
+i.e:
+
+DUT OS(NIC1)--(IPsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS
+
+SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(IPsec)-->(NIC1)DUT OS
+
+It then tries to perform some data transfer using the scheme decribed above.
+
+usage
+~~~~~
+
+In the ipsec-secgw/test directory
+
+to run one test for IPv4 or IPv6
+
+/bin/bash linux_test(4|6).sh <ipsec_mode>
+
+to run all tests for IPv4 or IPv6
+
+/bin/bash run_test.sh -4|-6
+
+For the list of available modes please refer to run_test.sh.
-- 
2.17.1

[PATCH v6 02/10] examples/ipsec-secgw: allow to specify neighbour mac address

[Konstantin Ananyev]

In some cases it is useful to allow user to specify destination
ether address for outgoing packets.
This patch adds such ability by introducing new 'neigh' config
file option.

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 doc/guides/sample_app_ug/ipsec_secgw.rst | 42 ++++++++++-
 examples/ipsec-secgw/ipsec-secgw.c       | 21 ++++--
 examples/ipsec-secgw/ipsec.h             |  3 +
 examples/ipsec-secgw/parser.c            | 91 ++++++++++++++++++++++++
 examples/ipsec-secgw/parser.h            |  8 +--
 5 files changed, 154 insertions(+), 11 deletions(-)

diff --git a/doc/guides/sample_app_ug/ipsec_secgw.rst b/doc/guides/sample_app_ug/ipsec_secgw.rst
index 244bde2de..61638e733 100644
--- a/doc/guides/sample_app_ug/ipsec_secgw.rst
+++ b/doc/guides/sample_app_ug/ipsec_secgw.rst
@@ -217,7 +217,7 @@ Configurations
 --------------
 
 The following sections provide the syntax of configurations to initialize
-your SP, SA and Routing tables.
+your SP, SA, Routing and Neighbour tables.
 Configurations shall be specified in the configuration file to be passed to
 the application. The file is then parsed by the application. The successful
 parsing will result in the appropriate rules being applied to the tables
@@ -238,8 +238,8 @@ General rule syntax
 
 The parse treats one line in the configuration file as one configuration
 item (unless the line concatenation symbol exists). Every configuration
-item shall follow the syntax of either SP, SA, or Routing rules specified
-below.
+item shall follow the syntax of either SP, SA, Routing or Neighbour
+rules specified below.
 
 The configuration parser supports the following special symbols:
 
@@ -631,3 +631,39 @@ Example SP rules:
     rt ipv4 dst 172.16.1.5/32 port 0
 
     rt ipv6 dst 1111:1111:1111:1111:1111:1111:1111:5555/116 port 0
+
+Neighbour rule syntax
+^^^^^^^^^^^^^^^^^^^^^
+
+The Neighbour rule syntax is shown as follows:
+
+.. code-block:: console
+
+    neigh <port> <dst_mac>
+
+
+where each options means:
+
+``<port>``
+
+ * The output port id
+
+ * Optional: No
+
+ * Syntax: *port X*
+
+``<dst_mac>``
+
+ * The destination ethernet address to use for that port
+
+ * Optional: No
+
+ * Syntax:
+
+   * XX:XX:XX:XX:XX:XX
+
+Example Neighbour rules:
+
+.. code-block:: console
+
+    neigh port 0 DE:AD:BE:EF:01:02
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 4ccff38d2..e79d17d51 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -104,9 +104,9 @@ static uint16_t nb_txd = IPSEC_SECGW_TX_DESC_DEFAULT;
 #define ETHADDR(a, b, c, d, e, f) (__BYTES_TO_UINT64(a, b, c, d, e, f, 0, 0))
 
 #define ETHADDR_TO_UINT64(addr) __BYTES_TO_UINT64( \
-		addr.addr_bytes[0], addr.addr_bytes[1], \
-		addr.addr_bytes[2], addr.addr_bytes[3], \
-		addr.addr_bytes[4], addr.addr_bytes[5], \
+		(addr)->addr_bytes[0], (addr)->addr_bytes[1], \
+		(addr)->addr_bytes[2], (addr)->addr_bytes[3], \
+		(addr)->addr_bytes[4], (addr)->addr_bytes[5], \
 		0, 0)
 
 /* port/source ethernet addr and destination ethernet addr */
@@ -1242,6 +1242,19 @@ print_ethaddr(const char *name, const struct ether_addr *eth_addr)
 	printf("%s%s", name, buf);
 }
 
+/*
+ * Update destination ethaddr for the port.
+ */
+int
+add_dst_ethaddr(uint16_t port, const struct ether_addr *addr)
+{
+	if (port > RTE_DIM(ethaddr_tbl))
+		return -EINVAL;
+
+	ethaddr_tbl[port].dst = ETHADDR_TO_UINT64(addr);
+	return 0;
+}
+
 /* Check the link status of all ports in up to 9s, and print them finally */
 static void
 check_all_ports_link_status(uint32_t port_mask)
@@ -1622,7 +1635,7 @@ port_init(uint16_t portid, uint64_t req_rx_offloads, uint64_t req_tx_offloads)
 	printf("Configuring device port %u:\n", portid);
 
 	rte_eth_macaddr_get(portid, &ethaddr);
-	ethaddr_tbl[portid].src = ETHADDR_TO_UINT64(ethaddr);
+	ethaddr_tbl[portid].src = ETHADDR_TO_UINT64(&ethaddr);
 	print_ethaddr("Address: ", &ethaddr);
 	printf("\n");
 
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 9b1586f52..580f7876b 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -245,4 +245,7 @@ int
 sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
 		uint64_t *tx_offloads);
 
+int
+add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/parser.c b/examples/ipsec-secgw/parser.c
index 91282ca94..b0a8ee23b 100644
--- a/examples/ipsec-secgw/parser.c
+++ b/examples/ipsec-secgw/parser.c
@@ -306,6 +306,46 @@ parse_range(const char *token, uint16_t *low, uint16_t *high)
 	return 0;
 }
 
+/*
+ * helper function for parse_mac, parse one section of the ether addr.
+ */
+static const char *
+parse_uint8x16(const char *s, uint8_t *v, uint8_t ls)
+{
+	char *end;
+	unsigned long t;
+
+	errno = 0;
+	t = strtoul(s, &end, 16);
+	if (errno != 0 || end[0] != ls || t > UINT8_MAX)
+		return NULL;
+	v[0] = t;
+	return end + 1;
+}
+
+static int
+parse_mac(const char *str, struct ether_addr *addr)
+{
+	uint32_t i;
+
+	static const uint8_t stop_sym[RTE_DIM(addr->addr_bytes)] = {
+		[0] = ':',
+		[1] = ':',
+		[2] = ':',
+		[3] = ':',
+		[4] = ':',
+		[5] = 0,
+	};
+
+	for (i = 0; i != RTE_DIM(addr->addr_bytes); i++) {
+		str = parse_uint8x16(str, addr->addr_bytes + i, stop_sym[i]);
+		if (str == NULL)
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
 /** sp add parse */
 struct cfg_sp_add_cfg_item {
 	cmdline_fixed_string_t sp_keyword;
@@ -444,11 +484,61 @@ cmdline_parse_inst_t cfg_rt_add_rule = {
 	},
 };
 
+/* neigh add parse */
+struct cfg_neigh_add_item {
+	cmdline_fixed_string_t neigh;
+	cmdline_fixed_string_t pstr;
+	uint16_t port;
+	cmdline_fixed_string_t mac;
+};
+
+static void
+cfg_parse_neigh(void *parsed_result, __rte_unused struct cmdline *cl,
+	void *data)
+{
+	int32_t rc;
+	struct cfg_neigh_add_item *res;
+	struct parse_status *st;
+	struct ether_addr mac;
+
+	st = data;
+	res = parsed_result;
+	rc = parse_mac(res->mac, &mac);
+	APP_CHECK(rc == 0, st, "invalid ether addr:%s", res->mac);
+	rc = add_dst_ethaddr(res->port, &mac);
+	APP_CHECK(rc == 0, st, "invalid port numer:%hu", res->port);
+	if (st->status < 0)
+		return;
+}
+
+cmdline_parse_token_string_t cfg_add_neigh_start =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, neigh, "neigh");
+cmdline_parse_token_string_t cfg_add_neigh_pstr =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, pstr, "port");
+cmdline_parse_token_num_t cfg_add_neigh_port =
+	TOKEN_NUM_INITIALIZER(struct cfg_neigh_add_item, port, UINT16);
+cmdline_parse_token_string_t cfg_add_neigh_mac =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, mac, NULL);
+
+cmdline_parse_inst_t cfg_neigh_add_rule = {
+	.f = cfg_parse_neigh,
+	.data = NULL,
+	.help_str = "",
+	.tokens = {
+		(void *)&cfg_add_neigh_start,
+		(void *)&cfg_add_neigh_pstr,
+		(void *)&cfg_add_neigh_port,
+		(void *)&cfg_add_neigh_mac,
+		NULL,
+	},
+};
+
 /** set of cfg items */
 cmdline_parse_ctx_t ipsec_ctx[] = {
 	(cmdline_parse_inst_t *)&cfg_sp_add_rule,
 	(cmdline_parse_inst_t *)&cfg_sa_add_rule,
 	(cmdline_parse_inst_t *)&cfg_rt_add_rule,
+	(cmdline_parse_inst_t *)&cfg_neigh_add_rule,
 	NULL,
 };
 
@@ -474,6 +564,7 @@ parse_cfg_file(const char *cfg_filename)
 	cfg_sp_add_rule.data = &status;
 	cfg_sa_add_rule.data = &status;
 	cfg_rt_add_rule.data = &status;
+	cfg_neigh_add_rule.data = &status;
 
 	do {
 		char oneline[1024];
diff --git a/examples/ipsec-secgw/parser.h b/examples/ipsec-secgw/parser.h
index be02537c5..6b8a10076 100644
--- a/examples/ipsec-secgw/parser.h
+++ b/examples/ipsec-secgw/parser.h
@@ -14,14 +14,14 @@ struct parse_status {
 	char parse_msg[256];
 };
 
-#define	APP_CHECK(exp, status, fmt, ...)				\
+#define	APP_CHECK(exp, st, fmt, ...)					\
 do {									\
 	if (!(exp)) {							\
-		sprintf(status->parse_msg, fmt "\n",			\
+		sprintf((st)->parse_msg, fmt "\n",			\
 			## __VA_ARGS__);				\
-		status->status = -1;					\
+		(st)->status = -1;					\
 	} else								\
-		status->status = 0;					\
+		(st)->status = 0;					\
 } while (0)
 
 #define APP_CHECK_PRESENCE(val, str, status)				\
-- 
2.17.1

[PATCH v6 03/10] examples/ipsec-secgw: fix crypto-op might never get dequeued

[Konstantin Ananyev]

In some cases crypto-ops could never be dequeued from the crypto-device.
The easiest way to reproduce:
start ipsec-secgw with crypto-dev and send to it less then 32 packets.
none packets will be forwarded.
Reason for that is that the application does dequeue() from crypto-queues
only when new packets arrive.
This patch makes the app to call dequeue() on a regular basis.
Also to make code cleaner and easier to understand,
it separates crypto-dev enqueue() and dequeue() code paths.
pkt_process() now only enqueues packets into crypto device,
dequeuing and final processing is done by drain_crypto_queues().

Fixes: c64278c0c18b ("examples/ipsec-secgw: rework processing loop")
Cc: stable@dpdk.org

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 136 ++++++++++++++++++++++++-----
 examples/ipsec-secgw/ipsec.c       |  99 +++++++++++++--------
 examples/ipsec-secgw/ipsec.h       |  11 +++
 3 files changed, 191 insertions(+), 55 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index e79d17d51..2a8ead708 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -469,38 +469,55 @@ inbound_sp_sa(struct sp_ctx *sp, struct sa_ctx *sa, struct traffic_type *ip,
 	ip->num = j;
 }
 
-static inline void
-process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
-		struct ipsec_traffic *traffic)
+static void
+split46_traffic(struct ipsec_traffic *trf, struct rte_mbuf *mb[], uint32_t num)
 {
+	uint32_t i, n4, n6;
+	struct ip *ip;
 	struct rte_mbuf *m;
-	uint16_t idx, nb_pkts_in, i, n_ip4, n_ip6;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
+	n4 = trf->ip4.num;
+	n6 = trf->ip6.num;
 
-	n_ip4 = traffic->ip4.num;
-	n_ip6 = traffic->ip6.num;
+	for (i = 0; i < num; i++) {
+
+		m = mb[i];
+		ip = rte_pktmbuf_mtod(m, struct ip *);
 
-	/* SP/ACL Inbound check ipsec and ip4 */
-	for (i = 0; i < nb_pkts_in; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
 		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-			traffic->ip4.data[idx] = rte_pktmbuf_mtod_offset(m,
+			trf->ip4.pkts[n4] = m;
+			trf->ip4.data[n4] = rte_pktmbuf_mtod_offset(m,
 					uint8_t *, offsetof(struct ip, ip_p));
+			n4++;
 		} else if (ip->ip_v == IP6_VERSION) {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
-			traffic->ip6.data[idx] = rte_pktmbuf_mtod_offset(m,
+			trf->ip6.pkts[n6] = m;
+			trf->ip6.data[n6] = rte_pktmbuf_mtod_offset(m,
 					uint8_t *,
 					offsetof(struct ip6_hdr, ip6_nxt));
+			n6++;
 		} else
 			rte_pktmbuf_free(m);
 	}
 
+	trf->ip4.num = n4;
+	trf->ip6.num = n6;
+}
+
+
+static inline void
+process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
+		struct ipsec_traffic *traffic)
+{
+	uint16_t nb_pkts_in, n_ip4, n_ip6;
+
+	n_ip4 = traffic->ip4.num;
+	n_ip6 = traffic->ip6.num;
+
+	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.num, MAX_PKT_BURST);
+
+	split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+
 	inbound_sp_sa(ipsec_ctx->sp4_ctx, ipsec_ctx->sa_ctx, &traffic->ip4,
 			n_ip4);
 
@@ -795,7 +812,7 @@ process_pkts(struct lcore_conf *qconf, struct rte_mbuf **pkts,
 }
 
 static inline void
-drain_buffers(struct lcore_conf *qconf)
+drain_tx_buffers(struct lcore_conf *qconf)
 {
 	struct buffer *buf;
 	uint32_t portid;
@@ -809,6 +826,81 @@ drain_buffers(struct lcore_conf *qconf)
 	}
 }
 
+static inline void
+drain_crypto_buffers(struct lcore_conf *qconf)
+{
+	uint32_t i;
+	struct ipsec_ctx *ctx;
+
+	/* drain inbound buffers*/
+	ctx = &qconf->inbound;
+	for (i = 0; i != ctx->nb_qps; i++) {
+		if (ctx->tbl[i].len != 0)
+			enqueue_cop_burst(ctx->tbl  + i);
+	}
+
+	/* drain outbound buffers*/
+	ctx = &qconf->outbound;
+	for (i = 0; i != ctx->nb_qps; i++) {
+		if (ctx->tbl[i].len != 0)
+			enqueue_cop_burst(ctx->tbl  + i);
+	}
+}
+
+static void
+drain_inbound_crypto_queues(const struct lcore_conf *qconf,
+		struct ipsec_ctx *ctx)
+{
+	uint32_t n;
+	struct ipsec_traffic trf;
+
+	/* dequeue packets from crypto-queue */
+	n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+			RTE_DIM(trf.ipsec.pkts));
+	if (n == 0)
+		return;
+
+	trf.ip4.num = 0;
+	trf.ip6.num = 0;
+
+	/* split traffic by ipv4-ipv6 */
+	split46_traffic(&trf, trf.ipsec.pkts, n);
+
+	/* process ipv4 packets */
+	inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
+	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+
+	/* process ipv6 packets */
+	inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
+	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+}
+
+static void
+drain_outbound_crypto_queues(const struct lcore_conf *qconf,
+		struct ipsec_ctx *ctx)
+{
+	uint32_t n;
+	struct ipsec_traffic trf;
+
+	/* dequeue packets from crypto-queue */
+	n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+			RTE_DIM(trf.ipsec.pkts));
+	if (n == 0)
+		return;
+
+	trf.ip4.num = 0;
+	trf.ip6.num = 0;
+
+	/* split traffic by ipv4-ipv6 */
+	split46_traffic(&trf, trf.ipsec.pkts, n);
+
+	/* process ipv4 packets */
+	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+
+	/* process ipv6 packets */
+	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+}
+
 /* main processing loop */
 static int32_t
 main_loop(__attribute__((unused)) void *dummy)
@@ -866,7 +958,8 @@ main_loop(__attribute__((unused)) void *dummy)
 		diff_tsc = cur_tsc - prev_tsc;
 
 		if (unlikely(diff_tsc > drain_tsc)) {
-			drain_buffers(qconf);
+			drain_tx_buffers(qconf);
+			drain_crypto_buffers(qconf);
 			prev_tsc = cur_tsc;
 		}
 
@@ -880,6 +973,9 @@ main_loop(__attribute__((unused)) void *dummy)
 			if (nb_rx > 0)
 				process_pkts(qconf, pkts, nb_rx, portid);
 		}
+
+		drain_inbound_crypto_queues(qconf, &qconf->inbound);
+		drain_outbound_crypto_queues(qconf, &qconf->outbound);
 	}
 }
 
diff --git a/examples/ipsec-secgw/ipsec.c b/examples/ipsec-secgw/ipsec.c
index 3d415f1af..72a29bcb1 100644
--- a/examples/ipsec-secgw/ipsec.c
+++ b/examples/ipsec-secgw/ipsec.c
@@ -333,33 +333,35 @@ create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa)
 	return 0;
 }
 
+/*
+ * queue crypto-ops into PMD queue.
+ */
+void
+enqueue_cop_burst(struct cdev_qp *cqp)
+{
+	uint32_t i, len, ret;
+
+	len = cqp->len;
+	ret = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp, cqp->buf, len);
+	if (ret < len) {
+		RTE_LOG_DP(DEBUG, IPSEC, "Cryptodev %u queue %u:"
+			" enqueued %u crypto ops out of %u\n",
+			cqp->id, cqp->qp, ret, len);
+			/* drop packets that we fail to enqueue */
+			for (i = ret; i < len; i++)
+				rte_pktmbuf_free(cqp->buf[i]->sym->m_src);
+	}
+	cqp->in_flight += ret;
+	cqp->len = 0;
+}
+
 static inline void
 enqueue_cop(struct cdev_qp *cqp, struct rte_crypto_op *cop)
 {
-	int32_t ret = 0, i;
-
 	cqp->buf[cqp->len++] = cop;
 
-	if (cqp->len == MAX_PKT_BURST) {
-		int enq_size = cqp->len;
-		if ((cqp->in_flight + enq_size) > MAX_INFLIGHT)
-			enq_size -=
-			    (int)((cqp->in_flight + enq_size) - MAX_INFLIGHT);
-
-		if (enq_size > 0)
-			ret = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp,
-					cqp->buf, enq_size);
-		if (ret < cqp->len) {
-			RTE_LOG_DP(DEBUG, IPSEC, "Cryptodev %u queue %u:"
-					" enqueued %u crypto ops out of %u\n",
-					 cqp->id, cqp->qp,
-					 ret, cqp->len);
-			for (i = ret; i < cqp->len; i++)
-				rte_pktmbuf_free(cqp->buf[i]->sym->m_src);
-		}
-		cqp->in_flight += ret;
-		cqp->len = 0;
-	}
+	if (cqp->len == MAX_PKT_BURST)
+		enqueue_cop_burst(cqp);
 }
 
 static inline void
@@ -473,6 +475,32 @@ ipsec_enqueue(ipsec_xform_fn xform_func, struct ipsec_ctx *ipsec_ctx,
 	}
 }
 
+static inline int32_t
+ipsec_inline_dequeue(ipsec_xform_fn xform_func, struct ipsec_ctx *ipsec_ctx,
+	      struct rte_mbuf *pkts[], uint16_t max_pkts)
+{
+	int32_t nb_pkts, ret;
+	struct ipsec_mbuf_metadata *priv;
+	struct ipsec_sa *sa;
+	struct rte_mbuf *pkt;
+
+	nb_pkts = 0;
+	while (ipsec_ctx->ol_pkts_cnt > 0 && nb_pkts < max_pkts) {
+		pkt = ipsec_ctx->ol_pkts[--ipsec_ctx->ol_pkts_cnt];
+		rte_prefetch0(pkt);
+		priv = get_priv(pkt);
+		sa = priv->sa;
+		ret = xform_func(pkt, sa, &priv->cop);
+		if (unlikely(ret)) {
+			rte_pktmbuf_free(pkt);
+			continue;
+		}
+		pkts[nb_pkts++] = pkt;
+	}
+
+	return nb_pkts;
+}
+
 static inline int
 ipsec_dequeue(ipsec_xform_fn xform_func, struct ipsec_ctx *ipsec_ctx,
 	      struct rte_mbuf *pkts[], uint16_t max_pkts)
@@ -490,19 +518,6 @@ ipsec_dequeue(ipsec_xform_fn xform_func, struct ipsec_ctx *ipsec_ctx,
 		if (ipsec_ctx->last_qp == ipsec_ctx->nb_qps)
 			ipsec_ctx->last_qp %= ipsec_ctx->nb_qps;
 
-		while (ipsec_ctx->ol_pkts_cnt > 0 && nb_pkts < max_pkts) {
-			pkt = ipsec_ctx->ol_pkts[--ipsec_ctx->ol_pkts_cnt];
-			rte_prefetch0(pkt);
-			priv = get_priv(pkt);
-			sa = priv->sa;
-			ret = xform_func(pkt, sa, &priv->cop);
-			if (unlikely(ret)) {
-				rte_pktmbuf_free(pkt);
-				continue;
-			}
-			pkts[nb_pkts++] = pkt;
-		}
-
 		if (cqp->in_flight == 0)
 			continue;
 
@@ -545,6 +560,13 @@ ipsec_inbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 
 	ipsec_enqueue(esp_inbound, ctx, pkts, sas, nb_pkts);
 
+	return ipsec_inline_dequeue(esp_inbound_post, ctx, pkts, len);
+}
+
+uint16_t
+ipsec_inbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len)
+{
 	return ipsec_dequeue(esp_inbound_post, ctx, pkts, len);
 }
 
@@ -558,5 +580,12 @@ ipsec_outbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 
 	ipsec_enqueue(esp_outbound, ctx, pkts, sas, nb_pkts);
 
+	return ipsec_inline_dequeue(esp_outbound_post, ctx, pkts, len);
+}
+
+uint16_t
+ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len)
+{
 	return ipsec_dequeue(esp_outbound_post, ctx, pkts, len);
 }
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 580f7876b..2f04b7d68 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -184,6 +184,14 @@ uint16_t
 ipsec_outbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint32_t sa_idx[], uint16_t nb_pkts, uint16_t len);
 
+uint16_t
+ipsec_inbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len);
+
+uint16_t
+ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len);
+
 static inline uint16_t
 ipsec_metadata_size(void)
 {
@@ -248,4 +256,7 @@ sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
 int
 add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
 
+void
+enqueue_cop_burst(struct cdev_qp *cqp);
+
 #endif /* __IPSEC_H__ */
-- 
2.17.1

[PATCH v6 04/10] examples/ipsec-secgw: fix outbound codepath for single SA

[Konstantin Ananyev]

Looking at process_pkts_outbound_nosp() there seems few issues:
- accessing mbuf after it was freed
- invoking ipsec_outbound() for ipv4 packets only
- copying number of packets, but not the mbuf pointers itself

that patch provides fixes for that issues.

Fixes: 906257e965b7 ("examples/ipsec-secgw: support IPv6")
Cc: stable@dpdk.org

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 33 +++++++++++++++++++++---------
 1 file changed, 23 insertions(+), 10 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 2a8ead708..58059316c 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -629,32 +629,45 @@ process_pkts_outbound_nosp(struct ipsec_ctx *ipsec_ctx,
 		struct ipsec_traffic *traffic)
 {
 	struct rte_mbuf *m;
-	uint32_t nb_pkts_out, i;
+	uint32_t nb_pkts_out, i, n;
 	struct ip *ip;
 
 	/* Drop any IPsec traffic from protected ports */
 	for (i = 0; i < traffic->ipsec.num; i++)
 		rte_pktmbuf_free(traffic->ipsec.pkts[i]);
 
-	traffic->ipsec.num = 0;
+	n = 0;
 
-	for (i = 0; i < traffic->ip4.num; i++)
-		traffic->ip4.res[i] = single_sa_idx;
+	for (i = 0; i < traffic->ip4.num; i++) {
+		traffic->ipsec.pkts[n] = traffic->ip4.pkts[i];
+		traffic->ipsec.res[n++] = single_sa_idx;
+	}
 
-	for (i = 0; i < traffic->ip6.num; i++)
-		traffic->ip6.res[i] = single_sa_idx;
+	for (i = 0; i < traffic->ip6.num; i++) {
+		traffic->ipsec.pkts[n] = traffic->ip6.pkts[i];
+		traffic->ipsec.res[n++] = single_sa_idx;
+	}
+
+	traffic->ip4.num = 0;
+	traffic->ip6.num = 0;
+	traffic->ipsec.num = n;
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ip4.pkts,
-			traffic->ip4.res, traffic->ip4.num,
+	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.res, traffic->ipsec.num,
 			MAX_PKT_BURST);
 
 	/* They all sue the same SA (ip4 or ip6 tunnel) */
 	m = traffic->ipsec.pkts[i];
 	ip = rte_pktmbuf_mtod(m, struct ip *);
-	if (ip->ip_v == IPVERSION)
+	if (ip->ip_v == IPVERSION) {
 		traffic->ip4.num = nb_pkts_out;
-	else
+		for (i = 0; i < nb_pkts_out; i++)
+			traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+	} else {
 		traffic->ip6.num = nb_pkts_out;
+		for (i = 0; i < nb_pkts_out; i++)
+			traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+	}
 }
 
 static inline int32_t
-- 
2.17.1

[PATCH v6 05/10] examples/ipsec-secgw: make local variables static

[Konstantin Ananyev]

in sp4.c and sp6.c there are few globals that used only locally.
Define them as static ones.

Cc: stable@dpdk.org

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/sp4.c | 10 +++++-----
 examples/ipsec-secgw/sp6.c | 10 +++++-----
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/examples/ipsec-secgw/sp4.c b/examples/ipsec-secgw/sp4.c
index 8d3d3d8e0..6b05daaa9 100644
--- a/examples/ipsec-secgw/sp4.c
+++ b/examples/ipsec-secgw/sp4.c
@@ -44,7 +44,7 @@ enum {
 	RTE_ACL_IPV4_NUM
 };
 
-struct rte_acl_field_def ip4_defs[NUM_FIELDS_IPV4] = {
+static struct rte_acl_field_def ip4_defs[NUM_FIELDS_IPV4] = {
 	{
 	.type = RTE_ACL_FIELD_TYPE_BITMASK,
 	.size = sizeof(uint8_t),
@@ -85,11 +85,11 @@ struct rte_acl_field_def ip4_defs[NUM_FIELDS_IPV4] = {
 
 RTE_ACL_RULE_DEF(acl4_rules, RTE_DIM(ip4_defs));
 
-struct acl4_rules acl4_rules_out[MAX_ACL_RULE_NUM];
-uint32_t nb_acl4_rules_out;
+static struct acl4_rules acl4_rules_out[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl4_rules_out;
 
-struct acl4_rules acl4_rules_in[MAX_ACL_RULE_NUM];
-uint32_t nb_acl4_rules_in;
+static struct acl4_rules acl4_rules_in[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl4_rules_in;
 
 void
 parse_sp4_tokens(char **tokens, uint32_t n_tokens,
diff --git a/examples/ipsec-secgw/sp6.c b/examples/ipsec-secgw/sp6.c
index 6002afef3..dc5b94c6a 100644
--- a/examples/ipsec-secgw/sp6.c
+++ b/examples/ipsec-secgw/sp6.c
@@ -34,7 +34,7 @@ enum {
 
 #define IP6_ADDR_SIZE 16
 
-struct rte_acl_field_def ip6_defs[IP6_NUM] = {
+static struct rte_acl_field_def ip6_defs[IP6_NUM] = {
 	{
 	.type = RTE_ACL_FIELD_TYPE_BITMASK,
 	.size = sizeof(uint8_t),
@@ -116,11 +116,11 @@ struct rte_acl_field_def ip6_defs[IP6_NUM] = {
 
 RTE_ACL_RULE_DEF(acl6_rules, RTE_DIM(ip6_defs));
 
-struct acl6_rules acl6_rules_out[MAX_ACL_RULE_NUM];
-uint32_t nb_acl6_rules_out;
+static struct acl6_rules acl6_rules_out[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl6_rules_out;
 
-struct acl6_rules acl6_rules_in[MAX_ACL_RULE_NUM];
-uint32_t nb_acl6_rules_in;
+static struct acl6_rules acl6_rules_in[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl6_rules_in;
 
 void
 parse_sp6_tokens(char **tokens, uint32_t n_tokens,
-- 
2.17.1

[PATCH v6 06/10] examples/ipsec-secgw: fix inbound SA checking

[Konstantin Ananyev]

In the inbound_sa_check() make sure that sa pointer stored
inside mbuf private area is not NULL.

Fixes: d299106e8e31 ("examples/ipsec-secgw: add IPsec sample application")
Cc: stable@dpdk.org

Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/sa.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index f6271bc60..839aaca0c 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -947,10 +947,15 @@ int
 inbound_sa_check(struct sa_ctx *sa_ctx, struct rte_mbuf *m, uint32_t sa_idx)
 {
 	struct ipsec_mbuf_metadata *priv;
+	struct ipsec_sa *sa;
 
 	priv = get_priv(m);
+	sa = priv->sa;
+	if (sa != NULL)
+		return (sa_ctx->sa[sa_idx].spi == sa->spi);
 
-	return (sa_ctx->sa[sa_idx].spi == priv->sa->spi);
+	RTE_LOG(ERR, IPSEC, "SA not saved in private data\n");
+	return 0;
 }
 
 static inline void
-- 
2.17.1

[PATCH v6 07/10] examples/ipsec-secgw: make app to use ipsec library

[Konstantin Ananyev]

Changes to make ipsec-secgw to utilize librte_ipsec library.
That patch provides:
 - changes in the related data structures.
 - changes in the initialization code.
 - new command-line parameters to enable librte_ipsec codepath
   and related features.

Note that right now by default current (non-librte_ipsec) code-path will
be used. User has to run application with new command-line option ('-l')
to enable new codepath.
The main reason for that:
 - current librte_ipsec doesn't support all ipsec algorithms
   and features that the app does.
 - allow users to run both versions in parallel for some time
   to figure out any functional or performance degradation with the
   new code.

It is planned to deprecate and remove non-librte_ipsec code path
in future releases.

Signed-off-by: Mohammad Abdul Awal <mohammad.abdul.awal@intel.com>
Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/Makefile      |   4 +-
 examples/ipsec-secgw/ipsec-secgw.c |  51 ++++++-
 examples/ipsec-secgw/ipsec.h       |  24 ++++
 examples/ipsec-secgw/meson.build   |   2 +-
 examples/ipsec-secgw/sa.c          | 221 ++++++++++++++++++++++++++++-
 examples/ipsec-secgw/sp4.c         |  25 ++++
 examples/ipsec-secgw/sp6.c         |  25 ++++
 7 files changed, 344 insertions(+), 8 deletions(-)

diff --git a/examples/ipsec-secgw/Makefile b/examples/ipsec-secgw/Makefile
index 02d41e39a..3918ee63e 100644
--- a/examples/ipsec-secgw/Makefile
+++ b/examples/ipsec-secgw/Makefile
@@ -61,8 +61,8 @@ RTE_TARGET ?= x86_64-native-linuxapp-gcc
 include $(RTE_SDK)/mk/rte.vars.mk
 
 ifneq ($(MAKECMDGOALS),clean)
-ifneq ($(CONFIG_RTE_LIBRTE_SECURITY),y)
-$(error "RTE_LIBRTE_SECURITY is required to build ipsec-secgw")
+ifneq ($(CONFIG_RTE_LIBRTE_IPSEC),y)
+$(error "RTE_LIBRTE_IPSEC is required to build ipsec-secgw")
 endif
 endif
 
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 58059316c..4b7309954 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -168,6 +168,9 @@ static uint32_t frame_size;
 static uint64_t dev_rx_offload = UINT64_MAX;
 static uint64_t dev_tx_offload = UINT64_MAX;
 
+/* application wide librte_ipsec/SA parameters */
+struct app_sa_prm app_sa_prm = {.enable = 0};
+
 struct lcore_rx_queue {
 	uint16_t port_id;
 	uint8_t queue_id;
@@ -1077,6 +1080,10 @@ print_usage(const char *prgname)
 		" [-P]"
 		" [-u PORTMASK]"
 		" [-j FRAMESIZE]"
+		" [-l]"
+		" [-w REPLAY_WINDOW_SIZE]"
+		" [-e]"
+		" [-a]"
 		" -f CONFIG_FILE"
 		" --config (port,queue,lcore)[,(port,queue,lcore)]"
 		" [--single-sa SAIDX]"
@@ -1089,6 +1096,11 @@ print_usage(const char *prgname)
 		"  -u PORTMASK: Hexadecimal bitmask of unprotected ports\n"
 		"  -j FRAMESIZE: Enable jumbo frame with 'FRAMESIZE' as maximum\n"
 		"                packet size\n"
+		"  -l enables code-path that uses librte_ipsec\n"
+		"  -w REPLAY_WINDOW_SIZE specifies IPsec SQN replay window\n"
+		"     size for each SA\n"
+		"  -e enables ESN\n"
+		"  -a enables SA SQN atomic behaviour\n"
 		"  -f CONFIG_FILE: Configuration file\n"
 		"  --config (port,queue,lcore): Rx queue configuration\n"
 		"  --single-sa SAIDX: Use single SA index for outbound traffic,\n"
@@ -1206,6 +1218,20 @@ parse_config(const char *q_arg)
 	return 0;
 }
 
+static void
+print_app_sa_prm(const struct app_sa_prm *prm)
+{
+	printf("librte_ipsec usage: %s\n",
+		(prm->enable == 0) ? "disabled" : "enabled");
+
+	if (prm->enable == 0)
+		return;
+
+	printf("replay window size: %u\n", prm->window_size);
+	printf("ESN: %s\n", (prm->enable_esn == 0) ? "disabled" : "enabled");
+	printf("SA flags: %#" PRIx64 "\n", prm->flags);
+}
+
 static int32_t
 parse_args(int32_t argc, char **argv)
 {
@@ -1217,7 +1243,7 @@ parse_args(int32_t argc, char **argv)
 
 	argvopt = argv;
 
-	while ((opt = getopt_long(argc, argvopt, "p:Pu:f:j:",
+	while ((opt = getopt_long(argc, argvopt, "aelp:Pu:f:j:w:",
 				lgopts, &option_index)) != EOF) {
 
 		switch (opt) {
@@ -1273,6 +1299,21 @@ parse_args(int32_t argc, char **argv)
 			}
 			printf("Enabled jumbo frames size %u\n", frame_size);
 			break;
+		case 'l':
+			app_sa_prm.enable = 1;
+			break;
+		case 'w':
+			app_sa_prm.enable = 1;
+			app_sa_prm.window_size = parse_decimal(optarg);
+			break;
+		case 'e':
+			app_sa_prm.enable = 1;
+			app_sa_prm.enable_esn = 1;
+			break;
+		case 'a':
+			app_sa_prm.enable = 1;
+			app_sa_prm.flags |= RTE_IPSEC_SAFLAG_SQN_ATOM;
+			break;
 		case CMD_LINE_OPT_CONFIG_NUM:
 			ret = parse_config(optarg);
 			if (ret) {
@@ -1335,6 +1376,8 @@ parse_args(int32_t argc, char **argv)
 		return -1;
 	}
 
+	print_app_sa_prm(&app_sa_prm);
+
 	if (optind >= 0)
 		argv[optind-1] = prgname;
 
@@ -2006,12 +2049,14 @@ main(int32_t argc, char **argv)
 		if (socket_ctx[socket_id].mbuf_pool)
 			continue;
 
-		sa_init(&socket_ctx[socket_id], socket_id);
-
+		/* initilaze SPD */
 		sp4_init(&socket_ctx[socket_id], socket_id);
 
 		sp6_init(&socket_ctx[socket_id], socket_id);
 
+		/* initilaze SAD */
+		sa_init(&socket_ctx[socket_id], socket_id);
+
 		rt_init(&socket_ctx[socket_id], socket_id);
 
 		pool_init(&socket_ctx[socket_id], socket_id, NB_MBUF);
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 2f04b7d68..8da86c4de 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -11,6 +11,7 @@
 #include <rte_crypto.h>
 #include <rte_security.h>
 #include <rte_flow.h>
+#include <rte_ipsec.h>
 
 #define RTE_LOGTYPE_IPSEC       RTE_LOGTYPE_USER1
 #define RTE_LOGTYPE_IPSEC_ESP   RTE_LOGTYPE_USER2
@@ -70,7 +71,20 @@ struct ip_addr {
 
 #define MAX_KEY_SIZE		32
 
+/*
+ * application wide SA parameters
+ */
+struct app_sa_prm {
+	uint32_t enable; /* use librte_ipsec API for ipsec pkt processing */
+	uint32_t window_size; /* replay window size */
+	uint32_t enable_esn;  /* enable/disable ESN support */
+	uint64_t flags;       /* rte_ipsec_sa_prm.flags */
+};
+
+extern struct app_sa_prm app_sa_prm;
+
 struct ipsec_sa {
+	struct rte_ipsec_session ips; /* one session per sa for now */
 	uint32_t spi;
 	uint32_t cdev_id_qp;
 	uint64_t seq;
@@ -243,6 +257,16 @@ sp4_init(struct socket_ctx *ctx, int32_t socket_id);
 void
 sp6_init(struct socket_ctx *ctx, int32_t socket_id);
 
+/*
+ * Search through SP rules for given SPI.
+ * Returns first rule index if found(greater or equal then zero),
+ * or -ENOENT otherwise.
+ */
+int
+sp4_spi_present(uint32_t spi, int inbound);
+int
+sp6_spi_present(uint32_t spi, int inbound);
+
 void
 sa_init(struct socket_ctx *ctx, int32_t socket_id);
 
diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
index 77d8b298f..31f68fee2 100644
--- a/examples/ipsec-secgw/meson.build
+++ b/examples/ipsec-secgw/meson.build
@@ -6,7 +6,7 @@
 # To build this example as a standalone application with an already-installed
 # DPDK instance, use 'make'
 
-deps += ['security', 'lpm', 'acl', 'hash']
+deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
 allow_experimental_apis = true
 sources = files(
 	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index 839aaca0c..414fcd26c 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -19,6 +19,7 @@
 #include <rte_ip.h>
 #include <rte_random.h>
 #include <rte_ethdev.h>
+#include <rte_malloc.h>
 
 #include "ipsec.h"
 #include "esp.h"
@@ -694,6 +695,7 @@ print_one_sa_rule(const struct ipsec_sa *sa, int inbound)
 }
 
 struct sa_ctx {
+	void *satbl; /* pointer to array of rte_ipsec_sa objects*/
 	struct ipsec_sa sa[IPSEC_SA_MAX_ENTRIES];
 	union {
 		struct {
@@ -764,7 +766,10 @@ sa_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 {
 	struct ipsec_sa *sa;
 	uint32_t i, idx;
-	uint16_t iv_length;
+	uint16_t iv_length, aad_length;
+
+	/* for ESN upper 32 bits of SQN also need to be part of AAD */
+	aad_length = (app_sa_prm.enable_esn != 0) ? sizeof(uint32_t) : 0;
 
 	for (i = 0; i < nb_entries; i++) {
 		idx = SPI2IDX(entries[i].spi);
@@ -808,7 +813,7 @@ sa_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 			sa_ctx->xf[idx].a.aead.iv.offset = IV_OFFSET;
 			sa_ctx->xf[idx].a.aead.iv.length = iv_length;
 			sa_ctx->xf[idx].a.aead.aad_length =
-				sa->aad_len;
+				sa->aad_len + aad_length;
 			sa_ctx->xf[idx].a.aead.digest_length =
 				sa->digest_len;
 
@@ -901,9 +906,205 @@ sa_in_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 	return sa_add_rules(sa_ctx, entries, nb_entries, 1);
 }
 
+/*
+ * helper function, fills parameters that are identical for all SAs
+ */
+static void
+fill_ipsec_app_sa_prm(struct rte_ipsec_sa_prm *prm,
+	const struct app_sa_prm *app_prm)
+{
+	memset(prm, 0, sizeof(*prm));
+
+	prm->flags = app_prm->flags;
+	prm->ipsec_xform.options.esn = app_prm->enable_esn;
+	prm->replay_win_sz = app_prm->window_size;
+}
+
+/*
+ * Helper function, tries to determine next_proto for SPI
+ * by searching though SP rules.
+ */
+static int
+get_spi_proto(uint32_t spi, enum rte_security_ipsec_sa_direction dir)
+{
+	int32_t rc4, rc6;
+
+	rc4 = sp4_spi_present(spi, dir == RTE_SECURITY_IPSEC_SA_DIR_INGRESS);
+	rc6 = sp6_spi_present(spi, dir == RTE_SECURITY_IPSEC_SA_DIR_INGRESS);
+
+	if (rc4 >= 0) {
+		if (rc6 >= 0) {
+			RTE_LOG(ERR, IPSEC,
+				"%s: SPI %u used simultaeously by "
+				"IPv4(%d) and IPv6 (%d) SP rules\n",
+				__func__, spi, rc4, rc6);
+			return -EINVAL;
+		} else
+			return IPPROTO_IPIP;
+	} else if (rc6 < 0) {
+		RTE_LOG(ERR, IPSEC,
+			"%s: SPI %u is not used by any SP rule\n",
+			__func__, spi);
+		return -EINVAL;
+	} else
+		return IPPROTO_IPV6;
+}
+
+static int
+fill_ipsec_sa_prm(struct rte_ipsec_sa_prm *prm, const struct ipsec_sa *ss,
+	const struct ipv4_hdr *v4, struct ipv6_hdr *v6)
+{
+	int32_t rc;
+
+	/*
+	 * Try to get SPI next proto by searching that SPI in SPD.
+	 * probably not the optimal way, but there seems nothing
+	 * better right now.
+	 */
+	rc = get_spi_proto(ss->spi, ss->direction);
+	if (rc < 0)
+		return rc;
+
+	fill_ipsec_app_sa_prm(prm, &app_sa_prm);
+	prm->userdata = (uintptr_t)ss;
+
+	/* setup ipsec xform */
+	prm->ipsec_xform.spi = ss->spi;
+	prm->ipsec_xform.salt = ss->salt;
+	prm->ipsec_xform.direction = ss->direction;
+	prm->ipsec_xform.proto = RTE_SECURITY_IPSEC_SA_PROTO_ESP;
+	prm->ipsec_xform.mode = (ss->flags == TRANSPORT) ?
+		RTE_SECURITY_IPSEC_SA_MODE_TRANSPORT :
+		RTE_SECURITY_IPSEC_SA_MODE_TUNNEL;
+
+	if (ss->flags == IP4_TUNNEL) {
+		prm->ipsec_xform.tunnel.type = RTE_SECURITY_IPSEC_TUNNEL_IPV4;
+		prm->tun.hdr_len = sizeof(*v4);
+		prm->tun.next_proto = rc;
+		prm->tun.hdr = v4;
+	} else if (ss->flags == IP6_TUNNEL) {
+		prm->ipsec_xform.tunnel.type = RTE_SECURITY_IPSEC_TUNNEL_IPV6;
+		prm->tun.hdr_len = sizeof(*v6);
+		prm->tun.next_proto = rc;
+		prm->tun.hdr = v6;
+	} else {
+		/* transport mode */
+		prm->trs.proto = rc;
+	}
+
+	/* setup crypto section */
+	prm->crypto_xform = ss->xforms;
+	return 0;
+}
+
+static void
+fill_ipsec_session(struct rte_ipsec_session *ss, struct rte_ipsec_sa *sa,
+	const struct ipsec_sa *lsa)
+{
+	ss->sa = sa;
+	ss->type = lsa->type;
+
+	/* setup crypto section */
+	if (ss->type == RTE_SECURITY_ACTION_TYPE_NONE) {
+		ss->crypto.ses = lsa->crypto_session;
+	/* setup session action type */
+	} else {
+		ss->security.ses = lsa->sec_session;
+		ss->security.ctx = lsa->security_ctx;
+		ss->security.ol_flags = lsa->ol_flags;
+	}
+}
+
+/*
+ * Initialise related rte_ipsec_sa object.
+ */
+static int
+ipsec_sa_init(struct ipsec_sa *lsa, struct rte_ipsec_sa *sa, uint32_t sa_size)
+{
+	int rc;
+	struct rte_ipsec_sa_prm prm;
+	struct ipv4_hdr v4  = {
+		.version_ihl = IPVERSION << 4 |
+			sizeof(v4) / IPV4_IHL_MULTIPLIER,
+		.time_to_live = IPDEFTTL,
+		.next_proto_id = IPPROTO_ESP,
+		.src_addr = lsa->src.ip.ip4,
+		.dst_addr = lsa->dst.ip.ip4,
+	};
+	struct ipv6_hdr v6 = {
+		.vtc_flow = htonl(IP6_VERSION << 28),
+		.proto = IPPROTO_ESP,
+	};
+
+	if (lsa->flags == IP6_TUNNEL) {
+		memcpy(v6.src_addr, lsa->src.ip.ip6.ip6_b, sizeof(v6.src_addr));
+		memcpy(v6.dst_addr, lsa->dst.ip.ip6.ip6_b, sizeof(v6.dst_addr));
+	}
+
+	rc = fill_ipsec_sa_prm(&prm, lsa, &v4, &v6);
+	if (rc == 0)
+		rc = rte_ipsec_sa_init(sa, &prm, sa_size);
+	if (rc < 0)
+		return rc;
+
+	fill_ipsec_session(&lsa->ips, sa, lsa);
+	return 0;
+}
+
+/*
+ * Allocate space and init rte_ipsec_sa strcutures,
+ * one per session.
+ */
+static int
+ipsec_satbl_init(struct sa_ctx *ctx, const struct ipsec_sa *ent,
+	uint32_t nb_ent, int32_t socket)
+{
+	int32_t rc, sz;
+	uint32_t i, idx;
+	size_t tsz;
+	struct rte_ipsec_sa *sa;
+	struct ipsec_sa *lsa;
+	struct rte_ipsec_sa_prm prm;
+
+	/* determine SA size */
+	idx = SPI2IDX(ent[0].spi);
+	fill_ipsec_sa_prm(&prm, ctx->sa + idx, NULL, NULL);
+	sz = rte_ipsec_sa_size(&prm);
+	if (sz < 0) {
+		RTE_LOG(ERR, IPSEC, "%s(%p, %u, %d): "
+			"failed to determine SA size, error code: %d\n",
+			__func__, ctx, nb_ent, socket, sz);
+		return sz;
+	}
+
+	tsz = sz * nb_ent;
+
+	ctx->satbl = rte_zmalloc_socket(NULL, tsz, RTE_CACHE_LINE_SIZE, socket);
+	if (ctx->satbl == NULL) {
+		RTE_LOG(ERR, IPSEC,
+			"%s(%p, %u, %d): failed to allocate %zu bytes\n",
+			__func__,  ctx, nb_ent, socket, tsz);
+		return -ENOMEM;
+	}
+
+	rc = 0;
+	for (i = 0; i != nb_ent && rc == 0; i++) {
+
+		idx = SPI2IDX(ent[i].spi);
+
+		sa = (struct rte_ipsec_sa *)((uintptr_t)ctx->satbl + sz * i);
+		lsa = ctx->sa + idx;
+
+		rc = ipsec_sa_init(lsa, sa, sz);
+	}
+
+	return rc;
+}
+
 void
 sa_init(struct socket_ctx *ctx, int32_t socket_id)
 {
+	int32_t rc;
 	const char *name;
 
 	if (ctx == NULL)
@@ -926,6 +1127,14 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id)
 				name, socket_id);
 
 		sa_in_add_rules(ctx->sa_in, sa_in, nb_sa_in);
+
+		if (app_sa_prm.enable != 0) {
+			rc = ipsec_satbl_init(ctx->sa_in, sa_in, nb_sa_in,
+				socket_id);
+			if (rc != 0)
+				rte_exit(EXIT_FAILURE,
+					"failed to init inbound SAs\n");
+		}
 	} else
 		RTE_LOG(WARNING, IPSEC, "No SA Inbound rule specified\n");
 
@@ -938,6 +1147,14 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id)
 				name, socket_id);
 
 		sa_out_add_rules(ctx->sa_out, sa_out, nb_sa_out);
+
+		if (app_sa_prm.enable != 0) {
+			rc = ipsec_satbl_init(ctx->sa_out, sa_out, nb_sa_out,
+				socket_id);
+			if (rc != 0)
+				rte_exit(EXIT_FAILURE,
+					"failed to init outbound SAs\n");
+		}
 	} else
 		RTE_LOG(WARNING, IPSEC, "No SA Outbound rule "
 			"specified\n");
diff --git a/examples/ipsec-secgw/sp4.c b/examples/ipsec-secgw/sp4.c
index 6b05daaa9..d1dc64bad 100644
--- a/examples/ipsec-secgw/sp4.c
+++ b/examples/ipsec-secgw/sp4.c
@@ -504,3 +504,28 @@ sp4_init(struct socket_ctx *ctx, int32_t socket_id)
 		RTE_LOG(WARNING, IPSEC, "No IPv4 SP Outbound rule "
 			"specified\n");
 }
+
+/*
+ * Search though SP rules for given SPI.
+ */
+int
+sp4_spi_present(uint32_t spi, int inbound)
+{
+	uint32_t i, num;
+	const struct acl4_rules *acr;
+
+	if (inbound != 0) {
+		acr = acl4_rules_in;
+		num = nb_acl4_rules_in;
+	} else {
+		acr = acl4_rules_out;
+		num = nb_acl4_rules_out;
+	}
+
+	for (i = 0; i != num; i++) {
+		if (acr[i].data.userdata == PROTECT(spi))
+			return i;
+	}
+
+	return -ENOENT;
+}
diff --git a/examples/ipsec-secgw/sp6.c b/examples/ipsec-secgw/sp6.c
index dc5b94c6a..e67d85aaf 100644
--- a/examples/ipsec-secgw/sp6.c
+++ b/examples/ipsec-secgw/sp6.c
@@ -618,3 +618,28 @@ sp6_init(struct socket_ctx *ctx, int32_t socket_id)
 		RTE_LOG(WARNING, IPSEC, "No IPv6 SP Outbound rule "
 			"specified\n");
 }
+
+/*
+ * Search though SP rules for given SPI.
+ */
+int
+sp6_spi_present(uint32_t spi, int inbound)
+{
+	uint32_t i, num;
+	const struct acl6_rules *acr;
+
+	if (inbound != 0) {
+		acr = acl6_rules_in;
+		num = nb_acl6_rules_in;
+	} else {
+		acr = acl6_rules_out;
+		num = nb_acl6_rules_out;
+	}
+
+	for (i = 0; i != num; i++) {
+		if (acr[i].data.userdata == PROTECT(spi))
+			return i;
+	}
+
+	return -ENOENT;
+}
-- 
2.17.1

[PATCH v6 08/10] examples/ipsec-secgw: make data-path to use ipsec library

[Konstantin Ananyev]

Changes to make ipsec-secgw data-path code to utilize librte_ipsec library.
Note that right now by default current (non-librte_ipsec) code-path will
be used. User has to run application with new command-line option ('-l')
to enable new codepath.

Signed-off-by: Mohammad Abdul Awal <mohammad.abdul.awal@intel.com>
Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/Makefile        |   1 +
 examples/ipsec-secgw/ipsec-secgw.c   | 179 ++++++++------
 examples/ipsec-secgw/ipsec.c         |   2 +-
 examples/ipsec-secgw/ipsec.h         |  23 ++
 examples/ipsec-secgw/ipsec_process.c | 357 +++++++++++++++++++++++++++
 examples/ipsec-secgw/meson.build     |   4 +-
 6 files changed, 487 insertions(+), 79 deletions(-)
 create mode 100644 examples/ipsec-secgw/ipsec_process.c

diff --git a/examples/ipsec-secgw/Makefile b/examples/ipsec-secgw/Makefile
index 3918ee63e..08f474da6 100644
--- a/examples/ipsec-secgw/Makefile
+++ b/examples/ipsec-secgw/Makefile
@@ -13,6 +13,7 @@ SRCS-y += sp4.c
 SRCS-y += sp6.c
 SRCS-y += sa.c
 SRCS-y += rt.c
+SRCS-y += ipsec_process.c
 SRCS-y += ipsec-secgw.c
 
 CFLAGS += -gdwarf-2
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 4b7309954..4f7a77d8d 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -229,19 +229,6 @@ static struct rte_eth_conf port_conf = {
 
 static struct socket_ctx socket_ctx[NB_SOCKETS];
 
-struct traffic_type {
-	const uint8_t *data[MAX_PKT_BURST * 2];
-	struct rte_mbuf *pkts[MAX_PKT_BURST * 2];
-	uint32_t res[MAX_PKT_BURST * 2];
-	uint32_t num;
-};
-
-struct ipsec_traffic {
-	struct traffic_type ipsec;
-	struct traffic_type ip4;
-	struct traffic_type ip6;
-};
-
 static inline void
 prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 {
@@ -258,6 +245,8 @@ prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 			t->ip4.data[t->ip4.num] = nlp;
 			t->ip4.pkts[(t->ip4.num)++] = pkt;
 		}
+		pkt->l2_len = 0;
+		pkt->l3_len = sizeof(struct ip);
 	} else if (eth->ether_type == rte_cpu_to_be_16(ETHER_TYPE_IPv6)) {
 		nlp = (uint8_t *)rte_pktmbuf_adj(pkt, ETHER_HDR_LEN);
 		nlp = RTE_PTR_ADD(nlp, offsetof(struct ip6_hdr, ip6_nxt));
@@ -267,6 +256,8 @@ prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 			t->ip6.data[t->ip6.num] = nlp;
 			t->ip6.pkts[(t->ip6.num)++] = pkt;
 		}
+		pkt->l2_len = 0;
+		pkt->l3_len = sizeof(struct ip6_hdr);
 	} else {
 		/* Unknown/Unsupported type, drop the packet */
 		RTE_LOG(ERR, IPSEC, "Unsupported packet type\n");
@@ -516,10 +507,15 @@ process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
 	n_ip4 = traffic->ip4.num;
 	n_ip6 = traffic->ip6.num;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
-
-	split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+	if (app_sa_prm.enable == 0) {
+		nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.num, MAX_PKT_BURST);
+		split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+	} else {
+		inbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
+	}
 
 	inbound_sp_sa(ipsec_ctx->sp4_ctx, ipsec_ctx->sa_ctx, &traffic->ip4,
 			n_ip4);
@@ -575,20 +571,27 @@ process_pkts_outbound(struct ipsec_ctx *ipsec_ctx,
 
 	outbound_sp(ipsec_ctx->sp6_ctx, &traffic->ip6, &traffic->ipsec);
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.res, traffic->ipsec.num,
-			MAX_PKT_BURST);
-
-	for (i = 0; i < nb_pkts_out; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
-		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-		} else {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
+	if (app_sa_prm.enable == 0) {
+
+		nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.res, traffic->ipsec.num,
+				MAX_PKT_BURST);
+
+		for (i = 0; i < nb_pkts_out; i++) {
+			m = traffic->ipsec.pkts[i];
+			struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
+			if (ip->ip_v == IPVERSION) {
+				idx = traffic->ip4.num++;
+				traffic->ip4.pkts[idx] = m;
+			} else {
+				idx = traffic->ip6.num++;
+				traffic->ip6.pkts[idx] = m;
+			}
 		}
+	} else {
+		outbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.res,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -611,19 +614,26 @@ process_pkts_inbound_nosp(struct ipsec_ctx *ipsec_ctx,
 
 	traffic->ip6.num = 0;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
+	if (app_sa_prm.enable == 0) {
 
-	for (i = 0; i < nb_pkts_in; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
-		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-		} else {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
+		nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.num, MAX_PKT_BURST);
+
+		for (i = 0; i < nb_pkts_in; i++) {
+			m = traffic->ipsec.pkts[i];
+			struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
+			if (ip->ip_v == IPVERSION) {
+				idx = traffic->ip4.num++;
+				traffic->ip4.pkts[idx] = m;
+			} else {
+				idx = traffic->ip6.num++;
+				traffic->ip6.pkts[idx] = m;
+			}
 		}
+	} else {
+		inbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -655,21 +665,28 @@ process_pkts_outbound_nosp(struct ipsec_ctx *ipsec_ctx,
 	traffic->ip6.num = 0;
 	traffic->ipsec.num = n;
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.res, traffic->ipsec.num,
-			MAX_PKT_BURST);
+	if (app_sa_prm.enable == 0) {
 
-	/* They all sue the same SA (ip4 or ip6 tunnel) */
-	m = traffic->ipsec.pkts[i];
-	ip = rte_pktmbuf_mtod(m, struct ip *);
-	if (ip->ip_v == IPVERSION) {
-		traffic->ip4.num = nb_pkts_out;
-		for (i = 0; i < nb_pkts_out; i++)
-			traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+		nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.res, traffic->ipsec.num,
+				MAX_PKT_BURST);
+
+		/* They all sue the same SA (ip4 or ip6 tunnel) */
+		m = traffic->ipsec.pkts[0];
+		ip = rte_pktmbuf_mtod(m, struct ip *);
+		if (ip->ip_v == IPVERSION) {
+			traffic->ip4.num = nb_pkts_out;
+			for (i = 0; i < nb_pkts_out; i++)
+				traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+		} else {
+			traffic->ip6.num = nb_pkts_out;
+			for (i = 0; i < nb_pkts_out; i++)
+				traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+		}
 	} else {
-		traffic->ip6.num = nb_pkts_out;
-		for (i = 0; i < nb_pkts_out; i++)
-			traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+		outbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.res,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -870,25 +887,31 @@ drain_inbound_crypto_queues(const struct lcore_conf *qconf,
 	uint32_t n;
 	struct ipsec_traffic trf;
 
-	/* dequeue packets from crypto-queue */
-	n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+	if (app_sa_prm.enable == 0) {
+
+		/* dequeue packets from crypto-queue */
+		n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
 			RTE_DIM(trf.ipsec.pkts));
-	if (n == 0)
-		return;
 
-	trf.ip4.num = 0;
-	trf.ip6.num = 0;
+		trf.ip4.num = 0;
+		trf.ip6.num = 0;
 
-	/* split traffic by ipv4-ipv6 */
-	split46_traffic(&trf, trf.ipsec.pkts, n);
+		/* split traffic by ipv4-ipv6 */
+		split46_traffic(&trf, trf.ipsec.pkts, n);
+	} else
+		ipsec_cqp_process(ctx, &trf);
 
 	/* process ipv4 packets */
-	inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
-	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	if (trf.ip4.num != 0) {
+		inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
+		route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	}
 
 	/* process ipv6 packets */
-	inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
-	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	if (trf.ip6.num != 0) {
+		inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
+		route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	}
 }
 
 static void
@@ -898,23 +921,27 @@ drain_outbound_crypto_queues(const struct lcore_conf *qconf,
 	uint32_t n;
 	struct ipsec_traffic trf;
 
-	/* dequeue packets from crypto-queue */
-	n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+	if (app_sa_prm.enable == 0) {
+
+		/* dequeue packets from crypto-queue */
+		n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
 			RTE_DIM(trf.ipsec.pkts));
-	if (n == 0)
-		return;
 
-	trf.ip4.num = 0;
-	trf.ip6.num = 0;
+		trf.ip4.num = 0;
+		trf.ip6.num = 0;
 
-	/* split traffic by ipv4-ipv6 */
-	split46_traffic(&trf, trf.ipsec.pkts, n);
+		/* split traffic by ipv4-ipv6 */
+		split46_traffic(&trf, trf.ipsec.pkts, n);
+	} else
+		ipsec_cqp_process(ctx, &trf);
 
 	/* process ipv4 packets */
-	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	if (trf.ip4.num != 0)
+		route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
 
 	/* process ipv6 packets */
-	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	if (trf.ip6.num != 0)
+		route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
 }
 
 /* main processing loop */
diff --git a/examples/ipsec-secgw/ipsec.c b/examples/ipsec-secgw/ipsec.c
index 72a29bcb1..36ead5cf9 100644
--- a/examples/ipsec-secgw/ipsec.c
+++ b/examples/ipsec-secgw/ipsec.c
@@ -39,7 +39,7 @@ set_ipsec_conf(struct ipsec_sa *sa, struct rte_security_ipsec_xform *ipsec)
 	ipsec->esn_soft_limit = IPSEC_OFFLOAD_ESN_SOFTLIMIT;
 }
 
-static inline int
+int
 create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa)
 {
 	struct rte_cryptodev_info cdev_info;
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 8da86c4de..8629f5045 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -190,6 +190,20 @@ struct cnt_blk {
 	uint32_t cnt;
 } __attribute__((packed));
 
+struct traffic_type {
+	const uint8_t *data[MAX_PKT_BURST * 2];
+	struct rte_mbuf *pkts[MAX_PKT_BURST * 2];
+	struct ipsec_sa *saptr[MAX_PKT_BURST * 2];
+	uint32_t res[MAX_PKT_BURST * 2];
+	uint32_t num;
+};
+
+struct ipsec_traffic {
+	struct traffic_type ipsec;
+	struct traffic_type ip4;
+	struct traffic_type ip6;
+};
+
 uint16_t
 ipsec_inbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint16_t nb_pkts, uint16_t len);
@@ -206,6 +220,12 @@ uint16_t
 ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint16_t len);
 
+void
+ipsec_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf);
+
+void
+ipsec_cqp_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf);
+
 static inline uint16_t
 ipsec_metadata_size(void)
 {
@@ -283,4 +303,7 @@ add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
 void
 enqueue_cop_burst(struct cdev_qp *cqp);
 
+int
+create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/ipsec_process.c b/examples/ipsec-secgw/ipsec_process.c
new file mode 100644
index 000000000..e403c461a
--- /dev/null
+++ b/examples/ipsec-secgw/ipsec_process.c
@@ -0,0 +1,357 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2017 Intel Corporation
+ */
+#include <sys/types.h>
+#include <netinet/in.h>
+#include <netinet/ip.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_log.h>
+#include <rte_cryptodev.h>
+#include <rte_ethdev.h>
+#include <rte_mbuf.h>
+
+#include "ipsec.h"
+
+#define SATP_OUT_IPV4(t)	\
+	((((t) & RTE_IPSEC_SATP_MODE_MASK) == RTE_IPSEC_SATP_MODE_TRANS && \
+	(((t) & RTE_IPSEC_SATP_IPV_MASK) == RTE_IPSEC_SATP_IPV4)) || \
+	((t) & RTE_IPSEC_SATP_MODE_MASK) == RTE_IPSEC_SATP_MODE_TUNLV4)
+
+
+/* helper routine to free bulk of packets */
+static inline void
+free_pkts(struct rte_mbuf *mb[], uint32_t n)
+{
+	uint32_t i;
+
+	for (i = 0; i != n; i++)
+		rte_pktmbuf_free(mb[i]);
+}
+
+/* helper routine to free bulk of crypto-ops and related packets */
+static inline void
+free_cops(struct rte_crypto_op *cop[], uint32_t n)
+{
+	uint32_t i;
+
+	for (i = 0; i != n; i++)
+		rte_pktmbuf_free(cop[i]->sym->m_src);
+}
+
+/* helper routine to enqueue bulk of crypto ops */
+static inline void
+enqueue_cop_bulk(struct cdev_qp *cqp, struct rte_crypto_op *cop[], uint32_t num)
+{
+	uint32_t i, k, len, n;
+
+	len = cqp->len;
+
+	/*
+	 * if cqp is empty and we have enough ops,
+	 * then queue them to the PMD straightway.
+	 */
+	if (num >= RTE_DIM(cqp->buf) * 3 / 4 && len == 0) {
+		n = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp, cop, num);
+		cqp->in_flight += n;
+		free_cops(cop + n, num - n);
+		return;
+	}
+
+	k = 0;
+
+	do {
+		n = RTE_DIM(cqp->buf) - len;
+		n = RTE_MIN(num - k, n);
+
+		/* put packets into cqp */
+		for (i = 0; i != n; i++)
+			cqp->buf[len + i] = cop[k + i];
+
+		len += n;
+		k += n;
+
+		/* if cqp is full then, enqueue crypto-ops to PMD */
+		if (len == RTE_DIM(cqp->buf)) {
+			n = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp,
+					cqp->buf, len);
+			cqp->in_flight += n;
+			free_cops(cqp->buf + n, len - n);
+			len = 0;
+		}
+
+
+	} while (k != num);
+
+	cqp->len = len;
+}
+
+static inline int
+fill_ipsec_session(struct rte_ipsec_session *ss, struct ipsec_ctx *ctx,
+	struct ipsec_sa *sa)
+{
+	int32_t rc;
+
+	/* setup crypto section */
+	if (ss->type == RTE_SECURITY_ACTION_TYPE_NONE) {
+		if (sa->crypto_session == NULL) {
+			rc = create_session(ctx, sa);
+			if (rc != 0)
+				return rc;
+		}
+		ss->crypto.ses = sa->crypto_session;
+	/* setup session action type */
+	} else {
+		if (sa->sec_session == NULL) {
+			rc = create_session(ctx, sa);
+			if (rc != 0)
+				return rc;
+		}
+		ss->security.ses = sa->sec_session;
+		ss->security.ctx = sa->security_ctx;
+		ss->security.ol_flags = sa->ol_flags;
+	}
+
+	rc = rte_ipsec_session_prepare(ss);
+	if (rc != 0)
+		memset(ss, 0, sizeof(*ss));
+
+	return rc;
+}
+
+/*
+ * group input packets byt the SA they belong to.
+ */
+static uint32_t
+sa_group(struct ipsec_sa *sa_ptr[], struct rte_mbuf *pkts[],
+	struct rte_ipsec_group grp[], uint32_t num)
+{
+	uint32_t i, n, spi;
+	void *sa;
+	void * const nosa = &spi;
+
+	sa = nosa;
+	for (i = 0, n = 0; i != num; i++) {
+
+		if (sa != sa_ptr[i]) {
+			grp[n].cnt = pkts + i - grp[n].m;
+			n += (sa != nosa);
+			grp[n].id.ptr = sa_ptr[i];
+			grp[n].m = pkts + i;
+			sa = sa_ptr[i];
+		}
+	}
+
+	/* terminate last group */
+	if (sa != nosa) {
+		grp[n].cnt = pkts + i - grp[n].m;
+		n++;
+	}
+
+	return n;
+}
+
+/*
+ * helper function, splits processed packets into ipv4/ipv6 traffic.
+ */
+static inline void
+copy_to_trf(struct ipsec_traffic *trf, uint64_t satp, struct rte_mbuf *mb[],
+	uint32_t num)
+{
+	uint32_t j, ofs, s;
+	struct traffic_type *out;
+
+	/*
+	 * determine traffic type(ipv4/ipv6) and offset for ACL classify
+	 * based on SA type
+	 */
+	if ((satp & RTE_IPSEC_SATP_DIR_MASK) == RTE_IPSEC_SATP_DIR_IB) {
+		if ((satp & RTE_IPSEC_SATP_IPV_MASK) == RTE_IPSEC_SATP_IPV4) {
+			out = &trf->ip4;
+			ofs = offsetof(struct ip, ip_p);
+		} else {
+			out = &trf->ip6;
+			ofs = offsetof(struct ip6_hdr, ip6_nxt);
+		}
+	} else if (SATP_OUT_IPV4(satp)) {
+		out = &trf->ip4;
+		ofs = offsetof(struct ip, ip_p);
+	} else {
+		out = &trf->ip6;
+		ofs = offsetof(struct ip6_hdr, ip6_nxt);
+	}
+
+	for (j = 0, s = out->num; j != num; j++) {
+		out->data[s + j] = rte_pktmbuf_mtod_offset(mb[j],
+				void *, ofs);
+		out->pkts[s + j] = mb[j];
+	}
+
+	out->num += num;
+}
+
+/*
+ * Process ipsec packets.
+ * If packet belong to SA that is subject of inline-crypto,
+ * then process it immediately.
+ * Otherwise do necessary preparations and queue it to related
+ * crypto-dev queue.
+ */
+void
+ipsec_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf)
+{
+	uint64_t satp;
+	uint32_t i, j, k, n;
+	struct ipsec_sa *sa;
+	struct ipsec_mbuf_metadata *priv;
+	struct rte_ipsec_group *pg;
+	struct rte_ipsec_session *ips;
+	struct cdev_qp *cqp;
+	struct rte_crypto_op *cop[RTE_DIM(trf->ipsec.pkts)];
+	struct rte_ipsec_group grp[RTE_DIM(trf->ipsec.pkts)];
+
+	n = sa_group(trf->ipsec.saptr, trf->ipsec.pkts, grp, trf->ipsec.num);
+
+	for (i = 0; i != n; i++) {
+
+		pg = grp + i;
+		sa = pg->id.ptr;
+
+		/* no valid SA found */
+		if (sa == NULL)
+			k = 0;
+
+		ips = &sa->ips;
+		satp = rte_ipsec_sa_type(ips->sa);
+
+		/* no valid HW session for that SA, try to create one */
+		if (ips->crypto.ses == NULL &&
+				fill_ipsec_session(ips, ctx, sa) != 0)
+			k = 0;
+
+		/* process packets inline */
+		else if (sa->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				sa->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL) {
+
+			/*
+			 * This is just to satisfy inbound_sa_check()
+			 * and get_hop_for_offload_pkt().
+			 * Should be removed in future.
+			 */
+			for (j = 0; j != pg->cnt; j++) {
+				priv = get_priv(pg->m[j]);
+				priv->sa = sa;
+			}
+
+			k = rte_ipsec_pkt_process(ips, pg->m, pg->cnt);
+			copy_to_trf(trf, satp, pg->m, k);
+
+		/* enqueue packets to crypto dev */
+		} else {
+
+			cqp = &ctx->tbl[sa->cdev_id_qp];
+
+			/* for that app each mbuf has it's own crypto op */
+			for (j = 0; j != pg->cnt; j++) {
+				priv = get_priv(pg->m[j]);
+				cop[j] = &priv->cop;
+				/*
+				 * this is just to satisfy inbound_sa_check()
+				 * should be removed in future.
+				 */
+				priv->sa = sa;
+			}
+
+			/* prepare and enqueue crypto ops */
+			k = rte_ipsec_pkt_crypto_prepare(ips, pg->m, cop,
+				pg->cnt);
+			if (k != 0)
+				enqueue_cop_bulk(cqp, cop, k);
+		}
+
+		/* drop packets that cannot be enqueued/processed */
+		if (k != pg->cnt)
+			free_pkts(pg->m + k, pg->cnt - k);
+	}
+}
+
+static inline uint32_t
+cqp_dequeue(struct cdev_qp *cqp, struct rte_crypto_op *cop[], uint32_t num)
+{
+	uint32_t n;
+
+	if (cqp->in_flight == 0)
+		return 0;
+
+	n = rte_cryptodev_dequeue_burst(cqp->id, cqp->qp, cop, num);
+	RTE_ASSERT(cqp->in_flight >= n);
+	cqp->in_flight -= n;
+
+	return n;
+}
+
+static inline uint32_t
+ctx_dequeue(struct ipsec_ctx *ctx, struct rte_crypto_op *cop[], uint32_t num)
+{
+	uint32_t i, n;
+
+	n = 0;
+
+	for (i = ctx->last_qp; n != num && i != ctx->nb_qps; i++)
+		n += cqp_dequeue(ctx->tbl + i, cop + n, num - n);
+
+	for (i = 0; n != num && i != ctx->last_qp; i++)
+		n += cqp_dequeue(ctx->tbl + i, cop + n, num - n);
+
+	ctx->last_qp = i;
+	return n;
+}
+
+/*
+ * dequeue packets from crypto-queues and finalize processing.
+ */
+void
+ipsec_cqp_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf)
+{
+	uint64_t satp;
+	uint32_t i, k, n, ng;
+	struct rte_ipsec_session *ss;
+	struct traffic_type *out;
+	struct rte_ipsec_group *pg;
+	struct rte_crypto_op *cop[RTE_DIM(trf->ipsec.pkts)];
+	struct rte_ipsec_group grp[RTE_DIM(trf->ipsec.pkts)];
+
+	trf->ip4.num = 0;
+	trf->ip6.num = 0;
+
+	out = &trf->ipsec;
+
+	/* dequeue completed crypto-ops */
+	n = ctx_dequeue(ctx, cop, RTE_DIM(cop));
+	if (n == 0)
+		return;
+
+	/* group them by ipsec session */
+	ng = rte_ipsec_pkt_crypto_group((const struct rte_crypto_op **)
+		(uintptr_t)cop, out->pkts, grp, n);
+
+	/* process each group of packets */
+	for (i = 0; i != ng; i++) {
+
+		pg = grp + i;
+		ss = pg->id.ptr;
+		satp = rte_ipsec_sa_type(ss->sa);
+
+		k = rte_ipsec_pkt_process(ss, pg->m, pg->cnt);
+		copy_to_trf(trf, satp, pg->m, k);
+
+		/* free bad packets, if any */
+		free_pkts(pg->m + k, pg->cnt - k);
+
+		n -= pg->cnt;
+	}
+
+	/* we should never have packet with unknown SA here */
+	RTE_VERIFY(n == 0);
+}
diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
index 31f68fee2..81c146ebc 100644
--- a/examples/ipsec-secgw/meson.build
+++ b/examples/ipsec-secgw/meson.build
@@ -9,6 +9,6 @@
 deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
 allow_experimental_apis = true
 sources = files(
-	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
-	'rt.c', 'sa.c', 'sp4.c', 'sp6.c'
+	'esp.c', 'ipsec.c', 'ipsec_process.c', 'ipsec-secgw.c',
+	'parser.c', 'rt.c', 'sa.c', 'sp4.c', 'sp6.c'
 )
-- 
2.17.1

Re: [PATCH v6 08/10] examples/ipsec-secgw: make data-path to use ipsec library

[Akhil Goyal]

On 1/4/2019 1:55 AM, Konstantin Ananyev wrote:
>   	/* process ipv4 packets */
> -	inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
> -	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
> +	if (trf.ip4.num != 0) {
> +		inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
> +		route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
> +	}
>   
>   	/* process ipv6 packets */
> -	inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
> -	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
> +	if (trf.ip6.num != 0) {
> +		inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
> +		route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
> +	}
One more thing.
Why do you need these changes. There are multiple places where this 
change is done.

route6_pkts and route4_pkts already checks that internally for 0 pkts.
inbound_sp_sa also does that.
These are getting called from multiple places. So it is better we add this check inside the function itself instead of outside it.

Re: [PATCH v6 08/10] examples/ipsec-secgw: make data-path to use ipsec library

[Ananyev, Konstantin]

> 
> On 1/4/2019 1:55 AM, Konstantin Ananyev wrote:
> >   	/* process ipv4 packets */
> > -	inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
> > -	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
> > +	if (trf.ip4.num != 0) {
> > +		inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
> > +		route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
> > +	}
> >
> >   	/* process ipv6 packets */
> > -	inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
> > -	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
> > +	if (trf.ip6.num != 0) {
> > +		inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
> > +		route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
> > +	}
> One more thing.
> Why do you need these changes. There are multiple places where this
> change is done.
> 
> route6_pkts and route4_pkts already checks that internally for 0 pkts.
> inbound_sp_sa also does that.
> These are getting called from multiple places. So it is better we add this check inside the function itself instead of outside it.

You right, these checks are not necessary here.
I put these extra checks to avoid unnecessary calls -
there is no guarantee that both route_pkts() and inbound_sp_sa()
will always be inlined by the compiler (they are not that small).
Konstantin

[PATCH v6 09/10] examples/ipsec-secgw: add scripts for functional test

[Konstantin Ananyev]

The purpose of these scripts is to automate ipsec-secgw functional testing.
The scripts require two machines (SUT and DUT) connected through
at least 2 NICs and running linux (so far tested only on Ubuntu 18.04).
Introduced test-cases for the following scenarios:
- Transport/Tunnel modes
- AES-CBC SHA1
- AES-GCM
- ESN on/off
- legacy/librte_ipsec code path

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/test/common_defs.sh      | 153 ++++++++++++++++++
 examples/ipsec-secgw/test/data_rxtx.sh        |  62 +++++++
 examples/ipsec-secgw/test/linux_test4.sh      |  63 ++++++++
 examples/ipsec-secgw/test/linux_test6.sh      |  64 ++++++++
 examples/ipsec-secgw/test/run_test.sh         |  80 +++++++++
 .../test/trs_aescbc_sha1_common_defs.sh       |  69 ++++++++
 .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  67 ++++++++
 .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/trs_aescbc_sha1_esn_defs.sh          |  66 ++++++++
 .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
 .../test/trs_aesgcm_common_defs.sh            |  60 +++++++
 examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  66 ++++++++
 .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  66 ++++++++
 .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
 .../test/tun_aescbc_sha1_common_defs.sh       |  68 ++++++++
 .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  70 ++++++++
 .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/tun_aescbc_sha1_esn_defs.sh          |  70 ++++++++
 .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
 .../test/tun_aesgcm_common_defs.sh            |  60 +++++++
 examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  70 ++++++++
 .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  70 ++++++++
 .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
 25 files changed, 1264 insertions(+)
 create mode 100644 examples/ipsec-secgw/test/common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test4.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test6.sh
 create mode 100644 examples/ipsec-secgw/test/run_test.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh

diff --git a/examples/ipsec-secgw/test/common_defs.sh b/examples/ipsec-secgw/test/common_defs.sh
new file mode 100644
index 000000000..7adfffa19
--- /dev/null
+++ b/examples/ipsec-secgw/test/common_defs.sh
@@ -0,0 +1,153 @@
+#! /bin/bash
+
+#check that env vars are properly defined
+
+#check SGW_PATH
+if [[ -z "${SGW_PATH}" || ! -x ${SGW_PATH} ]]; then
+	echo "SGW_PATH is invalid"
+	exit 127
+fi
+
+#check ETH_DEV
+if [[ -z "${ETH_DEV}" ]]; then
+	echo "ETH_DEV is invalid"
+	exit 127
+fi
+
+#setup SGW_LCORE
+SGW_LCORE=${SGW_LCORE:-0}
+
+#check that REMOTE_HOST is reachable
+ssh ${REMOTE_HOST} echo
+st=$?
+if [[ $st -ne 0 ]]; then
+	echo "host ${REMOTE_HOST} is not reachable"
+	exit $st
+fi
+
+#get ether addr of REMOTE_HOST
+REMOTE_MAC=`ssh ${REMOTE_HOST} ip addr show dev ${REMOTE_IFACE}`
+st=$?
+REMOTE_MAC=`echo ${REMOTE_MAC} | sed -e 's/^.*ether //' -e 's/ brd.*$//'`
+if [[ $st -ne 0 || -z "${REMOTE_MAC}" ]]; then
+	echo "coouldn't retrieve ether addr from ${REMOTE_IFACE}"
+	exit 127
+fi
+
+LOCAL_IFACE=dtap0
+
+LOCAL_MAC="00:64:74:61:70:30"
+
+REMOTE_IPV4=192.168.31.14
+LOCAL_IPV4=192.168.31.92
+
+REMOTE_IPV6=fd12:3456:789a:0031:0000:0000:0000:0014
+LOCAL_IPV6=fd12:3456:789a:0031:0000:0000:0000:0092
+
+DPDK_PATH=${RTE_SDK:-${PWD}}
+DPDK_BUILD=${RTE_TARGET:-x86_64-native-linuxapp-gcc}
+
+SGW_OUT_FILE=./ipsec-secgw.out1
+
+SGW_CMD_EAL_PRM="--lcores=${SGW_LCORE} -n 4 ${ETH_DEV}"
+SGW_CMD_CFG="(0,0,${SGW_LCORE}),(1,0,${SGW_LCORE})"
+SGW_CMD_PRM="-p 0x3 -u 1 -P --config=\"${SGW_CMD_CFG}\""
+
+SGW_CFG_FILE=$(tempfile)
+
+# configure local host/ifaces
+config_local_iface()
+{
+	ifconfig ${LOCAL_IFACE} ${LOCAL_IPV4}/24 mtu 1400 up
+	ifconfig ${LOCAL_IFACE}
+
+	ip neigh flush dev ${LOCAL_IFACE}
+	ip neigh add ${REMOTE_IPV4} dev ${LOCAL_IFACE} lladdr ${REMOTE_MAC}
+	ip neigh show dev ${LOCAL_IFACE}
+}
+
+config6_local_iface()
+{
+	config_local_iface
+
+	sysctl -w net.ipv6.conf.${LOCAL_IFACE}.disable_ipv6=0
+	ip addr add  ${LOCAL_IPV6}/64 dev ${LOCAL_IFACE}
+
+	sysctl -w net.ipv6.conf.${LOCAL_IFACE}.mtu=1300
+
+	ip -6 neigh add ${REMOTE_IPV6} dev ${LOCAL_IFACE} lladdr ${REMOTE_MAC}
+	ip neigh show dev ${LOCAL_IFACE}
+}
+
+#configure remote host/iface
+config_remote_iface()
+{
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE} down
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE} ${REMOTE_IPV4}/24 up
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} ip neigh flush dev ${REMOTE_IFACE}
+
+	# by some reason following ip neigh doesn't work for me here properly:
+	#ssh ${REMOTE_HOST} ip neigh add ${LOCAL_IPV4} \
+	#		dev ${REMOTE_IFACE} lladr ${LOCAL_MAC}
+	# so used arp instead.
+	ssh ${REMOTE_HOST} arp -i ${REMOTE_IFACE} -s ${LOCAL_IPV4} ${LOCAL_MAC}
+	ssh ${REMOTE_HOST} ip neigh show dev ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} iptables --flush
+}
+
+config6_remote_iface()
+{
+	config_remote_iface
+
+	ssh ${REMOTE_HOST} sysctl -w \
+		net.ipv6.conf.${REMOTE_IFACE}.disable_ipv6=0
+	ssh ${REMOTE_HOST} ip addr add  ${REMOTE_IPV6}/64 dev ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} ip -6 neigh add ${LOCAL_IPV6} \
+		dev ${REMOTE_IFACE} lladdr ${LOCAL_MAC}
+	ssh ${REMOTE_HOST} ip neigh show dev ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} ip6tables --flush
+}
+
+#configure remote and local host/iface
+config_iface()
+{
+	config_local_iface
+	config_remote_iface
+}
+
+config6_iface()
+{
+	config6_local_iface
+	config6_remote_iface
+}
+
+#start ipsec-secgw
+secgw_start()
+{
+	SGW_EXEC_FILE=$(tempfile)
+	cat <<EOF > ${SGW_EXEC_FILE}
+${SGW_PATH} ${SGW_CMD_EAL_PRM} ${CRYPTO_DEV} \
+--vdev="net_tap0,mac=fixed" \
+-- ${SGW_CMD_PRM} ${SGW_CMD_XPRM} -f ${SGW_CFG_FILE} > \
+${SGW_OUT_FILE} 2>&1 &
+p=\$!
+echo \$p
+EOF
+
+	cat ${SGW_EXEC_FILE}
+	SGW_PID=`/bin/bash -x ${SGW_EXEC_FILE}`
+	sleep 1
+}
+
+#stop ipsec-secgw and cleanup
+secgw_stop()
+{
+	kill ${SGW_PID}
+	rm -f ${SGW_EXEC_FILE}
+	rm -f ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/data_rxtx.sh b/examples/ipsec-secgw/test/data_rxtx.sh
new file mode 100644
index 000000000..f23a6d594
--- /dev/null
+++ b/examples/ipsec-secgw/test/data_rxtx.sh
@@ -0,0 +1,62 @@
+#! /bin/bash
+
+TCP_PORT=22222
+
+ping_test1()
+{
+	dst=$1
+
+	i=0
+	st=0
+	while [[ $i -ne 1200 && $st -eq 0 ]];
+	do
+		let i++
+		ping -c 1 -s ${i} ${dst}
+		st=$?
+	done
+
+	if [[ $st -ne 0 ]]; then
+		echo "ERROR: $0 failed for dst=${dst}, sz=${i}"
+	fi
+	return $st;
+}
+
+ping6_test1()
+{
+	dst=$1
+
+	i=0
+	st=0
+	while [[ $i -ne 1200 && $st -eq 0 ]];
+	do
+		let i++
+		ping6 -c 1 -s ${i} ${dst}
+		st=$?
+	done
+
+	if [[ $st -ne 0 ]]; then
+		echo "ERROR: $0 failed for dst=${dst}, sz=${i}"
+	fi
+	return $st;
+}
+
+scp_test1()
+{
+	dst=$1
+
+	for sz in 1234 23456 345678 4567890 56789102 ; do
+		x=`basename $0`.${sz}
+		dd if=/dev/urandom of=${x} bs=${sz} count=1
+		scp ${x} [${dst}]:${x}
+		scp [${dst}]:${x} ${x}.copy1
+		diff -u ${x} ${x}.copy1
+		st=$?
+		rm -f ${x} ${x}.copy1
+		ssh ${REMOTE_HOST} rm -f ${x}
+		if [[ $st -ne 0 ]]; then
+			return $st
+		fi
+	done
+
+	return 0;
+}
diff --git a/examples/ipsec-secgw/test/linux_test4.sh b/examples/ipsec-secgw/test/linux_test4.sh
new file mode 100644
index 000000000..d636f5604
--- /dev/null
+++ b/examples/ipsec-secgw/test/linux_test4.sh
@@ -0,0 +1,63 @@
+#! /bin/bash
+
+# usage:  /bin/bash linux_test4.sh <ipsec_mode>
+# for list of available modes please refer to run_test.sh.
+# ipsec-secgw (IPv4 mode) functional test script.
+#
+# Note that for most of them you required appropriate crypto PMD/device
+# to be avaialble.
+# Also user has to setup properly the following environment variables:
+#  SGW_PATH - path to the ipsec-secgw binary to test
+#  REMOTE_HOST - ip/hostname of the DUT
+#  REMOTE_IFACE - iface name for the test-port on DUT
+#  ETH_DEV - ethernet device to be used on SUT by DPDK ('-w <pci-id>')
+# Also user can optonally setup:
+#  SGW_LCORE - lcore to run ipsec-secgw on (default value is 0)
+#  CRYPTO_DEV - crypto device to be used ('-w <pci-id>')
+#  if none specified appropriate vdevs will be created by the scrit
+#
+# The purpose of the script is to automate ipsec-secgw testing
+# using another system running linux as a DUT.
+# It expects that SUT and DUT are connected through at least 2 NICs.
+# One NIC is expected to be managed by linux both machines,
+# and will be used as a control path
+# Make sure user from SUT can ssh to DUT without entering password.
+# Second NIC (test-port) should be reserved for DPDK on SUT,
+# and should be managed by linux on DUT.
+# The script starts ipsec-secgw with 2 NIC devices: test-port and tap vdev.
+# Then configures local tap iface and remote iface and ipsec policies
+# in the following way:
+# traffic going over test-port in both directions has to be
+# protected by ipsec.
+# raffic going over TAP in both directions doesn't have to be protected.
+# I.E:
+# DUT OS(NIC1)--(ipsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS
+# SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(ipsec)-->(NIC1)DUT OS
+# Then tries to perorm some data transfer using the scheme decribed above.
+#
+
+DIR=`dirname $0`
+MODE=$1
+
+ . ${DIR}/common_defs.sh
+ . ${DIR}/${MODE}_defs.sh
+
+config_secgw
+
+secgw_start
+
+config_iface
+
+config_remote_xfrm
+
+ . ${DIR}/data_rxtx.sh
+
+ping_test1 ${REMOTE_IPV4}
+st=$?
+if [[ $st -eq 0 ]]; then
+	scp_test1 ${REMOTE_IPV4}
+	st=$?
+fi
+
+secgw_stop
+exit $st
diff --git a/examples/ipsec-secgw/test/linux_test6.sh b/examples/ipsec-secgw/test/linux_test6.sh
new file mode 100644
index 000000000..e30f607d8
--- /dev/null
+++ b/examples/ipsec-secgw/test/linux_test6.sh
@@ -0,0 +1,64 @@
+#! /bin/bash
+
+# usage:  /bin/bash linux_test6.sh <ipsec_mode>
+# for list of available modes please refer to run_test.sh.
+# ipsec-secgw (IPv6 mode) functional test script.
+#
+# Note that for most of them you required appropriate crypto PMD/device
+# to be avaialble.
+# Also user has to setup properly the following environment variables:
+#  SGW_PATH - path to the ipsec-secgw binary to test
+#  REMOTE_HOST - ip/hostname of the DUT
+#  REMOTE_IFACE - iface name for the test-port on DUT
+#  ETH_DEV - ethernet device to be used on SUT by DPDK ('-w <pci-id>')
+# Also user can optonally setup:
+#  SGW_LCORE - lcore to run ipsec-secgw on (default value is 0)
+#  CRYPTO_DEV - crypto device to be used ('-w <pci-id>')
+#  if none specified appropriate vdevs will be created by the scrit
+#
+# The purpose of the script is to automate ipsec-secgw testing
+# using another system running linux as a DUT.
+# It expects that SUT and DUT are connected through at least 2 NICs.
+# One NIC is expected to be managed by linux both machines,
+# and will be used as a control path.
+# Make sure user from SUT can ssh to DUT without entering password,
+# also make sure that sshd over ipv6 is enabled.
+# Second NIC (test-port) should be reserved for DPDK on SUT,
+# and should be managed by linux on DUT.
+# The script starts ipsec-secgw with 2 NIC devices: test-port and tap vdev.
+# Then configures local tap iface and remote iface and ipsec policies
+# in the following way:
+# traffic going over test-port in both directions has to be
+# protected by ipsec.
+# raffic going over TAP in both directions doesn't have to be protected.
+# I.E:
+# DUT OS(NIC1)--(ipsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS
+# SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(ipsec)-->(NIC1)DUT OS
+# Then tries to perorm some data transfer using the scheme decribed above.
+#
+
+DIR=`dirname $0`
+MODE=$1
+
+ . ${DIR}/common_defs.sh
+ . ${DIR}/${MODE}_defs.sh
+
+config_secgw
+
+secgw_start
+
+config6_iface
+
+config6_remote_xfrm
+
+ . ${DIR}/data_rxtx.sh
+
+ping6_test1 ${REMOTE_IPV6}
+st=$?
+if [[ $st -eq 0 ]]; then
+	scp_test1 ${REMOTE_IPV6}
+	st=$?
+fi
+
+secgw_stop
+exit $st
diff --git a/examples/ipsec-secgw/test/run_test.sh b/examples/ipsec-secgw/test/run_test.sh
new file mode 100644
index 000000000..6dc0ce54e
--- /dev/null
+++ b/examples/ipsec-secgw/test/run_test.sh
@@ -0,0 +1,80 @@
+#! /bin/bash
+
+# usage: /bin/bash run_test.sh [-46]
+# Run all defined linux_test[4,6].sh test-cases one by one
+# user has to setup properly the following environment variables:
+#  SGW_PATH - path to the ipsec-secgw binary to test
+#  REMOTE_HOST - ip/hostname of the DUT
+#  REMOTE_IFACE - iface name for the test-port on DUT
+#  ETH_DEV - ethernet device to be used on SUT by DPDK ('-w <pci-id>')
+# Also user can optonally setup:
+#  SGW_LCORE - lcore to run ipsec-secgw on (default value is 0)
+#  CRYPTO_DEV - crypto device to be used ('-w <pci-id>')
+#  if none specified appropriate vdevs will be created by the scrit
+# refer to linux_test1.sh for more information
+
+# All supported modes to test.
+# naming convention:
+# 'old' means that ipsec-secgw will run in legacy (non-librte_ipsec mode)
+# 'tun/trs' refer to tunnel/transport mode respectively
+LINUX_TEST="tun_aescbc_sha1 \
+tun_aescbc_sha1_esn \
+tun_aescbc_sha1_esn_atom \
+tun_aesgcm \
+tun_aesgcm_esn \
+tun_aesgcm_esn_atom \
+trs_aescbc_sha1 \
+trs_aescbc_sha1_esn \
+trs_aescbc_sha1_esn_atom \
+trs_aesgcm \
+trs_aesgcm_esn \
+trs_aesgcm_esn_atom \
+tun_aescbc_sha1_old \
+tun_aesgcm_old \
+trs_aescbc_sha1_old \
+trs_aesgcm_old"
+
+DIR=`dirname $0`
+
+# get input options
+st=0
+run4=0
+run6=0
+while [[ ${st} -eq 0 ]]; do
+	getopts ":46" opt
+	st=$?
+	if [[ "${opt}" == "4" ]]; then
+		run4=1
+	elif [[ "${opt}" == "6" ]]; then
+		run6=1
+	fi
+done
+
+if [[ ${run4} -eq 0 && {run6} -eq 0 ]]; then
+	exit 127
+fi
+
+for i in ${LINUX_TEST}; do
+
+	echo "starting test ${i}"
+
+	st4=0
+	if [[ ${run4} -ne 0 ]]; then
+		/bin/bash ${DIR}/linux_test4.sh ${i}
+		st4=$?
+		echo "test4 ${i} finished with status ${st4}"
+	fi
+
+	st6=0
+	if [[ ${run6} -ne 0 ]]; then
+		/bin/bash ${DIR}/linux_test6.sh ${i}
+		st6=$?
+		echo "test6 ${i} finished with status ${st6}"
+	fi
+
+	let "st = st4 + st6"
+	if [[ $st -ne 0 ]]; then
+		echo "ERROR test ${i} FAILED"
+		exit $st
+	fi
+done
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
new file mode 100644
index 000000000..e2621e0df
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
@@ -0,0 +1,69 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_mb0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#SP in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#sp in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+sa in 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#SA out rules
+sa out 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#SA out rules
+sa out 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
new file mode 100644
index 000000000..d68552fce
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
@@ -0,0 +1,67 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
new file mode 100644
index 000000000..f16222e11
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
new file mode 100644
index 000000000..ce7c977a3
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
@@ -0,0 +1,66 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
new file mode 100644
index 000000000..a3abb6103
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
new file mode 100644
index 000000000..720e807e4
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
@@ -0,0 +1,60 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_gcm0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#SP in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+sa in 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#SA out rules
+sa out 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+sa out 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_defs.sh
new file mode 100644
index 000000000..8382d3d52
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_defs.sh
@@ -0,0 +1,66 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
new file mode 100644
index 000000000..80d8d63b8
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
new file mode 100644
index 000000000..94958d199
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
@@ -0,0 +1,66 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
new file mode 100644
index 000000000..951e6b68f
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
new file mode 100644
index 000000000..4025da232
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
@@ -0,0 +1,68 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_mb0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#sp in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#sp in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${REMOTE_IPV4} dst ${LOCAL_IPV4}
+
+sa in 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${REMOTE_IPV6} dst ${LOCAL_IPV6}
+
+#SA out rules
+sa out 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${LOCAL_IPV4} dst ${REMOTE_IPV4}
+
+sa out 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${LOCAL_IPV6} dst ${REMOTE_IPV6}
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
new file mode 100644
index 000000000..18aade3a9
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
new file mode 100644
index 000000000..6b4a82149
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
new file mode 100644
index 000000000..28c1125d6
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
new file mode 100644
index 000000000..3c0d8d1b1
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
new file mode 100644
index 000000000..fba68c6a3
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
@@ -0,0 +1,60 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_gcm0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#sp in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#sp in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${REMOTE_IPV4} dst ${LOCAL_IPV4}
+
+sa in 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${REMOTE_IPV6} dst ${LOCAL_IPV6}
+
+#SA out rules
+sa out 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${LOCAL_IPV4} dst ${REMOTE_IPV4}
+
+sa out 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${LOCAL_IPV6} dst ${REMOTE_IPV6}
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_defs.sh
new file mode 100644
index 000000000..8ae65321b
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_defs.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
new file mode 100644
index 000000000..dab1460c8
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
new file mode 100644
index 000000000..606232349
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh
new file mode 100644
index 000000000..e0a015e21
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_defs.sh
+
+SGW_CMD_XPRM=
-- 
2.17.1

[PATCH v6 10/10] doc: update ipsec-secgw guide and relelase notes

[Konstantin Ananyev]

Update ipsec-secgw guide and relelase notes to reflect latest changes.

Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 doc/guides/rel_notes/release_19_02.rst   |  14 +++
 doc/guides/sample_app_ug/ipsec_secgw.rst | 105 ++++++++++++++++++++++-
 2 files changed, 117 insertions(+), 2 deletions(-)

diff --git a/doc/guides/rel_notes/release_19_02.rst b/doc/guides/rel_notes/release_19_02.rst
index 1a9885c44..28dbe3ad0 100644
--- a/doc/guides/rel_notes/release_19_02.rst
+++ b/doc/guides/rel_notes/release_19_02.rst
@@ -116,6 +116,20 @@ New Features
 
   See :doc:`../prog_guide/ipsec_lib` for more information.
 
+* **Updated the ipsec-secgw sample application.**
+
+  The ``ipsec-secgw`` sample application has been updated to use the new
+  ``librte_ipsec`` library also added in this release.
+  The original functionality of ipsec-secgw is retained, a new command line
+  parameter ``-l`` has  been added to ipsec-secgw to use the IPsec library,
+  instead of the existing IPsec code in the application.
+
+  The IPsec library does not support all the functionality of the existing
+  ipsec-secgw application, its is planned to add the outstanding functionality
+  in future releases.
+
+  See :doc:`../sample_app_ug/ipsec_secgw` for more information.
+
 
 Removed Items
 -------------
diff --git a/doc/guides/sample_app_ug/ipsec_secgw.rst b/doc/guides/sample_app_ug/ipsec_secgw.rst
index 61638e733..3d784e705 100644
--- a/doc/guides/sample_app_ug/ipsec_secgw.rst
+++ b/doc/guides/sample_app_ug/ipsec_secgw.rst
@@ -76,7 +76,7 @@ Compiling the Application
 
 To compile the sample application see :doc:`compiling`.
 
-The application is located in the ``rpsec-secgw`` sub-directory.
+The application is located in the ``ipsec-secgw`` sub-directory.
 
 #. [Optional] Build the application for debugging:
    This option adds some extra flags, disables compiler optimizations and
@@ -93,6 +93,7 @@ The application has a number of command line options::
 
    ./build/ipsec-secgw [EAL options] --
                         -p PORTMASK -P -u PORTMASK -j FRAMESIZE
+                        -l -w REPLAY_WINOW_SIZE -e -a
                         --config (port,queue,lcore)[,(port,queue,lcore]
                         --single-sa SAIDX
                         --rxoffload MASK
@@ -114,6 +115,18 @@ Where:
     specified as FRAMESIZE. If an invalid value is provided as FRAMESIZE
     then the default value 9000 is used.
 
+*   ``-l``: enables code-path that uses librte_ipsec.
+
+*   ``-w REPLAY_WINOW_SIZE``: specifies the IPsec sequence number replay window
+    size for each Security Association (available only with librte_ipsec
+    code path).
+
+*   ``-e``: enables Security Association extended sequence number processing
+    (available only with librte_ipsec code path).
+
+*   ``-a``: enables Security Association sequence number atomic behaviour
+    (available only with librte_ipsec code path).
+
 *   ``--config (port,queue,lcore)[,(port,queue,lcore)]``: determines which queues
     from which ports are mapped to which cores.
 
@@ -225,7 +238,7 @@ accordingly.
 
 
 Configuration File Syntax
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~
 
 As mention in the overview, the Security Policies are ACL rules.
 The application parsers the rules specified in the configuration file and
@@ -571,6 +584,11 @@ Example SA rules:
     mode ipv4-tunnel src 172.16.1.5 dst 172.16.2.5 \
     type lookaside-protocol-offload port_id 4
 
+    sa in 35 aead_algo aes-128-gcm \
+    aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+    mode ipv4-tunnel src 172.16.2.5 dst 172.16.1.5 \
+    type inline-crypto-offload port_id 0
+
 Routing rule syntax
 ^^^^^^^^^^^^^^^^^^^
 
@@ -667,3 +685,86 @@ Example Neighbour rules:
 .. code-block:: console
 
     neigh port 0 DE:AD:BE:EF:01:02
+
+Test directory
+--------------
+
+The test directory contains scripts for testing the various encryption
+algorithms.
+
+The purpose of the scripts is to automate ipsec-secgw testing
+using another system running linux as a DUT.
+
+The user must setup the following environment variables:
+
+*   ``SGW_PATH``: path to the ipsec-secgw binary to test.
+
+*   ``REMOTE_HOST``: IP address/hostname of the DUT.
+
+*   ``REMOTE_IFACE``: interface name for the test-port on the DUT.
+
+*   ``ETH_DEV``: ethernet device to be used on the SUT by DPDK ('-w <pci-id>')
+
+Also the user can optionally setup:
+
+*   ``SGW_LCORE``: lcore to run ipsec-secgw on (default value is 0)
+
+*   ``CRYPTO_DEV``: crypto device to be used ('-w <pci-id>'). If none specified
+    appropriate vdevs will be created by the script
+
+Note that most of the tests require the appropriate crypto PMD/device to be
+available.
+
+Server configuration
+~~~~~~~~~~~~~~~~~~~~
+
+Two servers are required for the tests, SUT and DUT.
+
+Make sure the user from the SUT can ssh to the DUT without entering the password.
+To enable this feature keys must be setup on the DUT.
+
+``ssh-keygen`` will make a private & public key pair on the SUT.
+
+``ssh-copy-id`` <user name>@<target host name> on the SUT will copy the public
+key to the DUT. It will ask for credentials so that it can upload the public key.
+
+The SUT and DUT are connected through at least 2 NIC ports.
+
+One NIC port is expected to be managed by linux on both machines and will be
+used as a control path.
+
+The second NIC port (test-port) should be bound to DPDK on the SUT, and should
+be managed by linux on the DUT.
+
+The script starts ``ipsec-secgw`` with 2 NIC devices: ``test-port`` and
+``tap vdev``.
+
+It then configures the local tap interface and the remote interface and IPsec
+policies in the following way:
+
+Traffic going over the test-port in both directions has to be protected by IPsec.
+
+Traffic going over the TAP port in both directions does not have to be protected.
+
+i.e:
+
+DUT OS(NIC1)--(IPsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS
+
+SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(IPsec)-->(NIC1)DUT OS
+
+It then tries to perform some data transfer using the scheme decribed above.
+
+usage
+~~~~~
+
+In the ipsec-secgw/test directory
+
+to run one test for IPv4 or IPv6
+
+/bin/bash linux_test(4|6).sh <ipsec_mode>
+
+to run all tests for IPv4 or IPv6
+
+/bin/bash run_test.sh -4|-6
+
+For the list of available modes please refer to run_test.sh.
-- 
2.17.1

Re: [PATCH v6 10/10] doc: update ipsec-secgw guide and relelase notes

[Varghese, Vipin]

Hi Konstantin,

Sharing information with respect to documentation and code update with respect to rel 19.02-rc1 onwards 'it is now required to combine both code and documentation into a single patch'.

Thanks
Vipin Varghese

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Konstantin Ananyev
> Sent: Friday, January 4, 2019 1:56 AM
> To: dev@dpdk.org; dev@dpdk.org
> Cc: akhil.goyal@nxp.com; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; Iremonger, Bernard
> <bernard.iremonger@intel.com>
> Subject: [dpdk-dev] [PATCH v6 10/10] doc: update ipsec-secgw guide and
> relelase notes
> 
> Update ipsec-secgw guide and relelase notes to reflect latest changes.
> 
> Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> ---
>  doc/guides/rel_notes/release_19_02.rst   |  14 +++
>  doc/guides/sample_app_ug/ipsec_secgw.rst | 105 ++++++++++++++++++++++-
>  2 files changed, 117 insertions(+), 2 deletions(-)
> 
> diff --git a/doc/guides/rel_notes/release_19_02.rst
> b/doc/guides/rel_notes/release_19_02.rst
> index 1a9885c44..28dbe3ad0 100644
> --- a/doc/guides/rel_notes/release_19_02.rst
> +++ b/doc/guides/rel_notes/release_19_02.rst
> @@ -116,6 +116,20 @@ New Features
> 
>    See :doc:`../prog_guide/ipsec_lib` for more information.
> 
> +* **Updated the ipsec-secgw sample application.**
> +
> +  The ``ipsec-secgw`` sample application has been updated to use the
> + new  ``librte_ipsec`` library also added in this release.
> +  The original functionality of ipsec-secgw is retained, a new command
> + line  parameter ``-l`` has  been added to ipsec-secgw to use the IPsec
> + library,  instead of the existing IPsec code in the application.
> +
> +  The IPsec library does not support all the functionality of the
> + existing  ipsec-secgw application, its is planned to add the
> + outstanding functionality  in future releases.
> +
> +  See :doc:`../sample_app_ug/ipsec_secgw` for more information.
> +
> 
>  Removed Items
>  -------------
> diff --git a/doc/guides/sample_app_ug/ipsec_secgw.rst
> b/doc/guides/sample_app_ug/ipsec_secgw.rst
> index 61638e733..3d784e705 100644
> --- a/doc/guides/sample_app_ug/ipsec_secgw.rst
> +++ b/doc/guides/sample_app_ug/ipsec_secgw.rst
> @@ -76,7 +76,7 @@ Compiling the Application
> 
>  To compile the sample application see :doc:`compiling`.
> 
> -The application is located in the ``rpsec-secgw`` sub-directory.
> +The application is located in the ``ipsec-secgw`` sub-directory.
> 
>  #. [Optional] Build the application for debugging:
>     This option adds some extra flags, disables compiler optimizations and @@ -
> 93,6 +93,7 @@ The application has a number of command line options::
> 
>     ./build/ipsec-secgw [EAL options] --
>                          -p PORTMASK -P -u PORTMASK -j FRAMESIZE
> +                        -l -w REPLAY_WINOW_SIZE -e -a
>                          --config (port,queue,lcore)[,(port,queue,lcore]
>                          --single-sa SAIDX
>                          --rxoffload MASK @@ -114,6 +115,18 @@ Where:
>      specified as FRAMESIZE. If an invalid value is provided as FRAMESIZE
>      then the default value 9000 is used.
> 
> +*   ``-l``: enables code-path that uses librte_ipsec.
> +
> +*   ``-w REPLAY_WINOW_SIZE``: specifies the IPsec sequence number replay
> window
> +    size for each Security Association (available only with librte_ipsec
> +    code path).
> +
> +*   ``-e``: enables Security Association extended sequence number processing
> +    (available only with librte_ipsec code path).
> +
> +*   ``-a``: enables Security Association sequence number atomic behaviour
> +    (available only with librte_ipsec code path).
> +
>  *   ``--config (port,queue,lcore)[,(port,queue,lcore)]``: determines which
> queues
>      from which ports are mapped to which cores.
> 
> @@ -225,7 +238,7 @@ accordingly.
> 
> 
>  Configuration File Syntax
> -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +~~~~~~~~~~~~~~~~~~~~~~~~~
> 
>  As mention in the overview, the Security Policies are ACL rules.
>  The application parsers the rules specified in the configuration file and @@ -
> 571,6 +584,11 @@ Example SA rules:
>      mode ipv4-tunnel src 172.16.1.5 dst 172.16.2.5 \
>      type lookaside-protocol-offload port_id 4
> 
> +    sa in 35 aead_algo aes-128-gcm \
> +    aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
> +    mode ipv4-tunnel src 172.16.2.5 dst 172.16.1.5 \
> +    type inline-crypto-offload port_id 0
> +
>  Routing rule syntax
>  ^^^^^^^^^^^^^^^^^^^
> 
> @@ -667,3 +685,86 @@ Example Neighbour rules:
>  .. code-block:: console
> 
>      neigh port 0 DE:AD:BE:EF:01:02
> +
> +Test directory
> +--------------
> +
> +The test directory contains scripts for testing the various encryption
> +algorithms.
> +
> +The purpose of the scripts is to automate ipsec-secgw testing using
> +another system running linux as a DUT.
> +
> +The user must setup the following environment variables:
> +
> +*   ``SGW_PATH``: path to the ipsec-secgw binary to test.
> +
> +*   ``REMOTE_HOST``: IP address/hostname of the DUT.
> +
> +*   ``REMOTE_IFACE``: interface name for the test-port on the DUT.
> +
> +*   ``ETH_DEV``: ethernet device to be used on the SUT by DPDK ('-w <pci-id>')
> +
> +Also the user can optionally setup:
> +
> +*   ``SGW_LCORE``: lcore to run ipsec-secgw on (default value is 0)
> +
> +*   ``CRYPTO_DEV``: crypto device to be used ('-w <pci-id>'). If none specified
> +    appropriate vdevs will be created by the script
> +
> +Note that most of the tests require the appropriate crypto PMD/device
> +to be available.
> +
> +Server configuration
> +~~~~~~~~~~~~~~~~~~~~
> +
> +Two servers are required for the tests, SUT and DUT.
> +
> +Make sure the user from the SUT can ssh to the DUT without entering the
> password.
> +To enable this feature keys must be setup on the DUT.
> +
> +``ssh-keygen`` will make a private & public key pair on the SUT.
> +
> +``ssh-copy-id`` <user name>@<target host name> on the SUT will copy the
> +public key to the DUT. It will ask for credentials so that it can upload the
> public key.
> +
> +The SUT and DUT are connected through at least 2 NIC ports.
> +
> +One NIC port is expected to be managed by linux on both machines and
> +will be used as a control path.
> +
> +The second NIC port (test-port) should be bound to DPDK on the SUT, and
> +should be managed by linux on the DUT.
> +
> +The script starts ``ipsec-secgw`` with 2 NIC devices: ``test-port`` and
> +``tap vdev``.
> +
> +It then configures the local tap interface and the remote interface and
> +IPsec policies in the following way:
> +
> +Traffic going over the test-port in both directions has to be protected by
> IPsec.
> +
> +Traffic going over the TAP port in both directions does not have to be
> protected.
> +
> +i.e:
> +
> +DUT OS(NIC1)--(IPsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS
> +
> +SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(IPsec)-->(NIC1)DUT OS
> +
> +It then tries to perform some data transfer using the scheme decribed above.
> +
> +usage
> +~~~~~
> +
> +In the ipsec-secgw/test directory
> +
> +to run one test for IPv4 or IPv6
> +
> +/bin/bash linux_test(4|6).sh <ipsec_mode>
> +
> +to run all tests for IPv4 or IPv6
> +
> +/bin/bash run_test.sh -4|-6
> +
> +For the list of available modes please refer to run_test.sh.
> --
> 2.17.1

[PATCH v5 02/10] examples/ipsec-secgw: allow to specify neighbour mac address

[Konstantin Ananyev]

In some cases it is useful to allow user to specify destination
ether address for outgoing packets.
This patch adds such ability by introducing new 'neigh' config
file option.

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 doc/guides/sample_app_ug/ipsec_secgw.rst | 42 ++++++++++-
 examples/ipsec-secgw/ipsec-secgw.c       | 21 ++++--
 examples/ipsec-secgw/ipsec.h             |  3 +
 examples/ipsec-secgw/parser.c            | 91 ++++++++++++++++++++++++
 examples/ipsec-secgw/parser.h            |  8 +--
 5 files changed, 154 insertions(+), 11 deletions(-)

diff --git a/doc/guides/sample_app_ug/ipsec_secgw.rst b/doc/guides/sample_app_ug/ipsec_secgw.rst
index 244bde2de..61638e733 100644
--- a/doc/guides/sample_app_ug/ipsec_secgw.rst
+++ b/doc/guides/sample_app_ug/ipsec_secgw.rst
@@ -217,7 +217,7 @@ Configurations
 --------------
 
 The following sections provide the syntax of configurations to initialize
-your SP, SA and Routing tables.
+your SP, SA, Routing and Neighbour tables.
 Configurations shall be specified in the configuration file to be passed to
 the application. The file is then parsed by the application. The successful
 parsing will result in the appropriate rules being applied to the tables
@@ -238,8 +238,8 @@ General rule syntax
 
 The parse treats one line in the configuration file as one configuration
 item (unless the line concatenation symbol exists). Every configuration
-item shall follow the syntax of either SP, SA, or Routing rules specified
-below.
+item shall follow the syntax of either SP, SA, Routing or Neighbour
+rules specified below.
 
 The configuration parser supports the following special symbols:
 
@@ -631,3 +631,39 @@ Example SP rules:
     rt ipv4 dst 172.16.1.5/32 port 0
 
     rt ipv6 dst 1111:1111:1111:1111:1111:1111:1111:5555/116 port 0
+
+Neighbour rule syntax
+^^^^^^^^^^^^^^^^^^^^^
+
+The Neighbour rule syntax is shown as follows:
+
+.. code-block:: console
+
+    neigh <port> <dst_mac>
+
+
+where each options means:
+
+``<port>``
+
+ * The output port id
+
+ * Optional: No
+
+ * Syntax: *port X*
+
+``<dst_mac>``
+
+ * The destination ethernet address to use for that port
+
+ * Optional: No
+
+ * Syntax:
+
+   * XX:XX:XX:XX:XX:XX
+
+Example Neighbour rules:
+
+.. code-block:: console
+
+    neigh port 0 DE:AD:BE:EF:01:02
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index f5db3da0c..274a49cbb 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -104,9 +104,9 @@ static uint16_t nb_txd = IPSEC_SECGW_TX_DESC_DEFAULT;
 #define ETHADDR(a, b, c, d, e, f) (__BYTES_TO_UINT64(a, b, c, d, e, f, 0, 0))
 
 #define ETHADDR_TO_UINT64(addr) __BYTES_TO_UINT64( \
-		addr.addr_bytes[0], addr.addr_bytes[1], \
-		addr.addr_bytes[2], addr.addr_bytes[3], \
-		addr.addr_bytes[4], addr.addr_bytes[5], \
+		(addr)->addr_bytes[0], (addr)->addr_bytes[1], \
+		(addr)->addr_bytes[2], (addr)->addr_bytes[3], \
+		(addr)->addr_bytes[4], (addr)->addr_bytes[5], \
 		0, 0)
 
 /* port/source ethernet addr and destination ethernet addr */
@@ -1242,6 +1242,19 @@ print_ethaddr(const char *name, const struct ether_addr *eth_addr)
 	printf("%s%s", name, buf);
 }
 
+/*
+ * Update destination ethaddr for the port.
+ */
+int
+add_dst_ethaddr(uint16_t port, const struct ether_addr *addr)
+{
+	if (port > RTE_DIM(ethaddr_tbl))
+		return -EINVAL;
+
+	ethaddr_tbl[port].dst = ETHADDR_TO_UINT64(addr);
+	return 0;
+}
+
 /* Check the link status of all ports in up to 9s, and print them finally */
 static void
 check_all_ports_link_status(uint32_t port_mask)
@@ -1622,7 +1635,7 @@ port_init(uint16_t portid, uint64_t req_rx_offloads, uint64_t req_tx_offloads)
 	printf("Configuring device port %u:\n", portid);
 
 	rte_eth_macaddr_get(portid, &ethaddr);
-	ethaddr_tbl[portid].src = ETHADDR_TO_UINT64(ethaddr);
+	ethaddr_tbl[portid].src = ETHADDR_TO_UINT64(&ethaddr);
 	print_ethaddr("Address: ", &ethaddr);
 	printf("\n");
 
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 9b1586f52..580f7876b 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -245,4 +245,7 @@ int
 sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
 		uint64_t *tx_offloads);
 
+int
+add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/parser.c b/examples/ipsec-secgw/parser.c
index 91282ca94..b0a8ee23b 100644
--- a/examples/ipsec-secgw/parser.c
+++ b/examples/ipsec-secgw/parser.c
@@ -306,6 +306,46 @@ parse_range(const char *token, uint16_t *low, uint16_t *high)
 	return 0;
 }
 
+/*
+ * helper function for parse_mac, parse one section of the ether addr.
+ */
+static const char *
+parse_uint8x16(const char *s, uint8_t *v, uint8_t ls)
+{
+	char *end;
+	unsigned long t;
+
+	errno = 0;
+	t = strtoul(s, &end, 16);
+	if (errno != 0 || end[0] != ls || t > UINT8_MAX)
+		return NULL;
+	v[0] = t;
+	return end + 1;
+}
+
+static int
+parse_mac(const char *str, struct ether_addr *addr)
+{
+	uint32_t i;
+
+	static const uint8_t stop_sym[RTE_DIM(addr->addr_bytes)] = {
+		[0] = ':',
+		[1] = ':',
+		[2] = ':',
+		[3] = ':',
+		[4] = ':',
+		[5] = 0,
+	};
+
+	for (i = 0; i != RTE_DIM(addr->addr_bytes); i++) {
+		str = parse_uint8x16(str, addr->addr_bytes + i, stop_sym[i]);
+		if (str == NULL)
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
 /** sp add parse */
 struct cfg_sp_add_cfg_item {
 	cmdline_fixed_string_t sp_keyword;
@@ -444,11 +484,61 @@ cmdline_parse_inst_t cfg_rt_add_rule = {
 	},
 };
 
+/* neigh add parse */
+struct cfg_neigh_add_item {
+	cmdline_fixed_string_t neigh;
+	cmdline_fixed_string_t pstr;
+	uint16_t port;
+	cmdline_fixed_string_t mac;
+};
+
+static void
+cfg_parse_neigh(void *parsed_result, __rte_unused struct cmdline *cl,
+	void *data)
+{
+	int32_t rc;
+	struct cfg_neigh_add_item *res;
+	struct parse_status *st;
+	struct ether_addr mac;
+
+	st = data;
+	res = parsed_result;
+	rc = parse_mac(res->mac, &mac);
+	APP_CHECK(rc == 0, st, "invalid ether addr:%s", res->mac);
+	rc = add_dst_ethaddr(res->port, &mac);
+	APP_CHECK(rc == 0, st, "invalid port numer:%hu", res->port);
+	if (st->status < 0)
+		return;
+}
+
+cmdline_parse_token_string_t cfg_add_neigh_start =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, neigh, "neigh");
+cmdline_parse_token_string_t cfg_add_neigh_pstr =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, pstr, "port");
+cmdline_parse_token_num_t cfg_add_neigh_port =
+	TOKEN_NUM_INITIALIZER(struct cfg_neigh_add_item, port, UINT16);
+cmdline_parse_token_string_t cfg_add_neigh_mac =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, mac, NULL);
+
+cmdline_parse_inst_t cfg_neigh_add_rule = {
+	.f = cfg_parse_neigh,
+	.data = NULL,
+	.help_str = "",
+	.tokens = {
+		(void *)&cfg_add_neigh_start,
+		(void *)&cfg_add_neigh_pstr,
+		(void *)&cfg_add_neigh_port,
+		(void *)&cfg_add_neigh_mac,
+		NULL,
+	},
+};
+
 /** set of cfg items */
 cmdline_parse_ctx_t ipsec_ctx[] = {
 	(cmdline_parse_inst_t *)&cfg_sp_add_rule,
 	(cmdline_parse_inst_t *)&cfg_sa_add_rule,
 	(cmdline_parse_inst_t *)&cfg_rt_add_rule,
+	(cmdline_parse_inst_t *)&cfg_neigh_add_rule,
 	NULL,
 };
 
@@ -474,6 +564,7 @@ parse_cfg_file(const char *cfg_filename)
 	cfg_sp_add_rule.data = &status;
 	cfg_sa_add_rule.data = &status;
 	cfg_rt_add_rule.data = &status;
+	cfg_neigh_add_rule.data = &status;
 
 	do {
 		char oneline[1024];
diff --git a/examples/ipsec-secgw/parser.h b/examples/ipsec-secgw/parser.h
index be02537c5..6b8a10076 100644
--- a/examples/ipsec-secgw/parser.h
+++ b/examples/ipsec-secgw/parser.h
@@ -14,14 +14,14 @@ struct parse_status {
 	char parse_msg[256];
 };
 
-#define	APP_CHECK(exp, status, fmt, ...)				\
+#define	APP_CHECK(exp, st, fmt, ...)					\
 do {									\
 	if (!(exp)) {							\
-		sprintf(status->parse_msg, fmt "\n",			\
+		sprintf((st)->parse_msg, fmt "\n",			\
 			## __VA_ARGS__);				\
-		status->status = -1;					\
+		(st)->status = -1;					\
 	} else								\
-		status->status = 0;					\
+		(st)->status = 0;					\
 } while (0)
 
 #define APP_CHECK_PRESENCE(val, str, status)				\
-- 
2.17.1

[PATCH v5 03/10] examples/ipsec-secgw: fix crypto-op might never get dequeued

[Konstantin Ananyev]

In some cases crypto-ops could never be dequeued from the crypto-device.
The easiest way to reproduce:
start ipsec-secgw with crypto-dev and send to it less then 32 packets.
none packets will be forwarded.
Reason for that is that the application does dequeue() from crypto-queues
only when new packets arrive.
This patch makes sure it calls dequeue() on a regular basis.

Fixes: c64278c0c18b ("examples/ipsec-secgw: rework processing loop")
Cc: stable@dpdk.org

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 136 ++++++++++++++++++++++++-----
 examples/ipsec-secgw/ipsec.c       |  60 ++++++++-----
 examples/ipsec-secgw/ipsec.h       |  11 +++
 3 files changed, 165 insertions(+), 42 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 274a49cbb..797bd6435 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -469,38 +469,55 @@ inbound_sp_sa(struct sp_ctx *sp, struct sa_ctx *sa, struct traffic_type *ip,
 	ip->num = j;
 }
 
-static inline void
-process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
-		struct ipsec_traffic *traffic)
+static void
+split46_traffic(struct ipsec_traffic *trf, struct rte_mbuf *mb[], uint32_t num)
 {
+	uint32_t i, n4, n6;
+	struct ip *ip;
 	struct rte_mbuf *m;
-	uint16_t idx, nb_pkts_in, i, n_ip4, n_ip6;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
+	n4 = trf->ip4.num;
+	n6 = trf->ip6.num;
 
-	n_ip4 = traffic->ip4.num;
-	n_ip6 = traffic->ip6.num;
+	for (i = 0; i < num; i++) {
+
+		m = mb[i];
+		ip = rte_pktmbuf_mtod(m, struct ip *);
 
-	/* SP/ACL Inbound check ipsec and ip4 */
-	for (i = 0; i < nb_pkts_in; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
 		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-			traffic->ip4.data[idx] = rte_pktmbuf_mtod_offset(m,
+			trf->ip4.pkts[n4] = m;
+			trf->ip4.data[n4] = rte_pktmbuf_mtod_offset(m,
 					uint8_t *, offsetof(struct ip, ip_p));
+			n4++;
 		} else if (ip->ip_v == IP6_VERSION) {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
-			traffic->ip6.data[idx] = rte_pktmbuf_mtod_offset(m,
+			trf->ip6.pkts[n6] = m;
+			trf->ip6.data[n6] = rte_pktmbuf_mtod_offset(m,
 					uint8_t *,
 					offsetof(struct ip6_hdr, ip6_nxt));
+			n6++;
 		} else
 			rte_pktmbuf_free(m);
 	}
 
+	trf->ip4.num = n4;
+	trf->ip6.num = n6;
+}
+
+
+static inline void
+process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
+		struct ipsec_traffic *traffic)
+{
+	uint16_t nb_pkts_in, n_ip4, n_ip6;
+
+	n_ip4 = traffic->ip4.num;
+	n_ip6 = traffic->ip6.num;
+
+	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.num, MAX_PKT_BURST);
+
+	split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+
 	inbound_sp_sa(ipsec_ctx->sp4_ctx, ipsec_ctx->sa_ctx, &traffic->ip4,
 			n_ip4);
 
@@ -795,7 +812,7 @@ process_pkts(struct lcore_conf *qconf, struct rte_mbuf **pkts,
 }
 
 static inline void
-drain_buffers(struct lcore_conf *qconf)
+drain_tx_buffers(struct lcore_conf *qconf)
 {
 	struct buffer *buf;
 	uint32_t portid;
@@ -809,6 +826,81 @@ drain_buffers(struct lcore_conf *qconf)
 	}
 }
 
+static inline void
+drain_crypto_buffers(struct lcore_conf *qconf)
+{
+	uint32_t i;
+	struct ipsec_ctx *ctx;
+
+	/* drain inbound buffers*/
+	ctx = &qconf->inbound;
+	for (i = 0; i != ctx->nb_qps; i++) {
+		if (ctx->tbl[i].len != 0)
+			enqueue_cop_burst(ctx->tbl  + i);
+	}
+
+	/* drain outbound buffers*/
+	ctx = &qconf->outbound;
+	for (i = 0; i != ctx->nb_qps; i++) {
+		if (ctx->tbl[i].len != 0)
+			enqueue_cop_burst(ctx->tbl  + i);
+	}
+}
+
+static void
+drain_inbound_crypto_queues(const struct lcore_conf *qconf,
+		struct ipsec_ctx *ctx)
+{
+	uint32_t n;
+	struct ipsec_traffic trf;
+
+	/* dequeue packets from crypto-queue */
+	n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+			RTE_DIM(trf.ipsec.pkts));
+	if (n == 0)
+		return;
+
+	trf.ip4.num = 0;
+	trf.ip6.num = 0;
+
+	/* split traffic by ipv4-ipv6 */
+	split46_traffic(&trf, trf.ipsec.pkts, n);
+
+	/* process ipv4 packets */
+	inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
+	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+
+	/* process ipv6 packets */
+	inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
+	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+}
+
+static void
+drain_outbound_crypto_queues(const struct lcore_conf *qconf,
+		struct ipsec_ctx *ctx)
+{
+	uint32_t n;
+	struct ipsec_traffic trf;
+
+	/* dequeue packets from crypto-queue */
+	n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+			RTE_DIM(trf.ipsec.pkts));
+	if (n == 0)
+		return;
+
+	trf.ip4.num = 0;
+	trf.ip6.num = 0;
+
+	/* split traffic by ipv4-ipv6 */
+	split46_traffic(&trf, trf.ipsec.pkts, n);
+
+	/* process ipv4 packets */
+	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+
+	/* process ipv6 packets */
+	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+}
+
 /* main processing loop */
 static int32_t
 main_loop(__attribute__((unused)) void *dummy)
@@ -866,7 +958,8 @@ main_loop(__attribute__((unused)) void *dummy)
 		diff_tsc = cur_tsc - prev_tsc;
 
 		if (unlikely(diff_tsc > drain_tsc)) {
-			drain_buffers(qconf);
+			drain_tx_buffers(qconf);
+			drain_crypto_buffers(qconf);
 			prev_tsc = cur_tsc;
 		}
 
@@ -880,6 +973,9 @@ main_loop(__attribute__((unused)) void *dummy)
 			if (nb_rx > 0)
 				process_pkts(qconf, pkts, nb_rx, portid);
 		}
+
+		drain_inbound_crypto_queues(qconf, &qconf->inbound);
+		drain_outbound_crypto_queues(qconf, &qconf->outbound);
 	}
 }
 
diff --git a/examples/ipsec-secgw/ipsec.c b/examples/ipsec-secgw/ipsec.c
index 3d415f1af..8bf928a23 100644
--- a/examples/ipsec-secgw/ipsec.c
+++ b/examples/ipsec-secgw/ipsec.c
@@ -333,33 +333,35 @@ create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa)
 	return 0;
 }
 
+/*
+ * queue crypto-ops into PMD queue.
+ */
+void
+enqueue_cop_burst(struct cdev_qp *cqp)
+{
+	uint32_t i, len, ret;
+
+	len = cqp->len;
+	ret = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp, cqp->buf, len);
+	if (ret < len) {
+		RTE_LOG_DP(DEBUG, IPSEC, "Cryptodev %u queue %u:"
+			" enqueued %u crypto ops out of %u\n",
+			cqp->id, cqp->qp, ret, len);
+			/* drop packets that we fail to enqueue */
+			for (i = ret; i < len; i++)
+				rte_pktmbuf_free(cqp->buf[i]->sym->m_src);
+	}
+	cqp->in_flight += ret;
+	cqp->len = 0;
+}
+
 static inline void
 enqueue_cop(struct cdev_qp *cqp, struct rte_crypto_op *cop)
 {
-	int32_t ret = 0, i;
-
 	cqp->buf[cqp->len++] = cop;
 
-	if (cqp->len == MAX_PKT_BURST) {
-		int enq_size = cqp->len;
-		if ((cqp->in_flight + enq_size) > MAX_INFLIGHT)
-			enq_size -=
-			    (int)((cqp->in_flight + enq_size) - MAX_INFLIGHT);
-
-		if (enq_size > 0)
-			ret = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp,
-					cqp->buf, enq_size);
-		if (ret < cqp->len) {
-			RTE_LOG_DP(DEBUG, IPSEC, "Cryptodev %u queue %u:"
-					" enqueued %u crypto ops out of %u\n",
-					 cqp->id, cqp->qp,
-					 ret, cqp->len);
-			for (i = ret; i < cqp->len; i++)
-				rte_pktmbuf_free(cqp->buf[i]->sym->m_src);
-		}
-		cqp->in_flight += ret;
-		cqp->len = 0;
-	}
+	if (cqp->len == MAX_PKT_BURST)
+		enqueue_cop_burst(cqp);
 }
 
 static inline void
@@ -548,6 +550,13 @@ ipsec_inbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 	return ipsec_dequeue(esp_inbound_post, ctx, pkts, len);
 }
 
+uint16_t
+ipsec_inbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len)
+{
+	return ipsec_dequeue(esp_inbound_post, ctx, pkts, len);
+}
+
 uint16_t
 ipsec_outbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint32_t sa_idx[], uint16_t nb_pkts, uint16_t len)
@@ -560,3 +569,10 @@ ipsec_outbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 
 	return ipsec_dequeue(esp_outbound_post, ctx, pkts, len);
 }
+
+uint16_t
+ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len)
+{
+	return ipsec_dequeue(esp_outbound_post, ctx, pkts, len);
+}
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 580f7876b..2f04b7d68 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -184,6 +184,14 @@ uint16_t
 ipsec_outbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint32_t sa_idx[], uint16_t nb_pkts, uint16_t len);
 
+uint16_t
+ipsec_inbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len);
+
+uint16_t
+ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len);
+
 static inline uint16_t
 ipsec_metadata_size(void)
 {
@@ -248,4 +256,7 @@ sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
 int
 add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
 
+void
+enqueue_cop_burst(struct cdev_qp *cqp);
+
 #endif /* __IPSEC_H__ */
-- 
2.17.1

Re: [PATCH v5 03/10] examples/ipsec-secgw: fix crypto-op might never get dequeued

[Akhil Goyal]

On 12/28/2018 9:03 PM, Konstantin Ananyev wrote:
> In some cases crypto-ops could never be dequeued from the crypto-device.
> The easiest way to reproduce:
> start ipsec-secgw with crypto-dev and send to it less then 32 packets.
> none packets will be forwarded.
> Reason for that is that the application does dequeue() from crypto-queues
> only when new packets arrive.
> This patch makes sure it calls dequeue() on a regular basis.
>
> Fixes: c64278c0c18b ("examples/ipsec-secgw: rework processing loop")
> Cc: stable@dpdk.org
>
> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> Acked-by: Radu Nicolau <radu.nicolau@intel.com>
> ---
>   examples/ipsec-secgw/ipsec-secgw.c | 136 ++++++++++++++++++++++++-----
>   examples/ipsec-secgw/ipsec.c       |  60 ++++++++-----
>   examples/ipsec-secgw/ipsec.h       |  11 +++
>   3 files changed, 165 insertions(+), 42 deletions(-)
[snip]
> +
>   /* main processing loop */
>   static int32_t
>   main_loop(__attribute__((unused)) void *dummy)
> @@ -866,7 +958,8 @@ main_loop(__attribute__((unused)) void *dummy)
>   		diff_tsc = cur_tsc - prev_tsc;
>   
>   		if (unlikely(diff_tsc > drain_tsc)) {
> -			drain_buffers(qconf);
> +			drain_tx_buffers(qconf);
> +			drain_crypto_buffers(qconf);
>   			prev_tsc = cur_tsc;
>   		}
>   
> @@ -880,6 +973,9 @@ main_loop(__attribute__((unused)) void *dummy)
>   			if (nb_rx > 0)
>   				process_pkts(qconf, pkts, nb_rx, portid);
>   		}
> +
> +		drain_inbound_crypto_queues(qconf, &qconf->inbound);
> +		drain_outbound_crypto_queues(qconf, &qconf->outbound);

drain_inbound_crypto_queues and drain_outbound_crypto_queues should be called based on diff_tsc.
moving these two lines above after  drain_crypto_buffers will improve the performance drop caused due to this patch.

Re: [PATCH v5 03/10] examples/ipsec-secgw: fix crypto-op might never get dequeued

[Ananyev, Konstantin]

> 
> On 12/28/2018 9:03 PM, Konstantin Ananyev wrote:
> > In some cases crypto-ops could never be dequeued from the crypto-device.
> > The easiest way to reproduce:
> > start ipsec-secgw with crypto-dev and send to it less then 32 packets.
> > none packets will be forwarded.
> > Reason for that is that the application does dequeue() from crypto-queues
> > only when new packets arrive.
> > This patch makes sure it calls dequeue() on a regular basis.
> >
> > Fixes: c64278c0c18b ("examples/ipsec-secgw: rework processing loop")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> > Acked-by: Radu Nicolau <radu.nicolau@intel.com>
> > ---
> >   examples/ipsec-secgw/ipsec-secgw.c | 136 ++++++++++++++++++++++++-----
> >   examples/ipsec-secgw/ipsec.c       |  60 ++++++++-----
> >   examples/ipsec-secgw/ipsec.h       |  11 +++
> >   3 files changed, 165 insertions(+), 42 deletions(-)
> [snip]
> > +
> >   /* main processing loop */
> >   static int32_t
> >   main_loop(__attribute__((unused)) void *dummy)
> > @@ -866,7 +958,8 @@ main_loop(__attribute__((unused)) void *dummy)
> >   		diff_tsc = cur_tsc - prev_tsc;
> >
> >   		if (unlikely(diff_tsc > drain_tsc)) {
> > -			drain_buffers(qconf);
> > +			drain_tx_buffers(qconf);
> > +			drain_crypto_buffers(qconf);
> >   			prev_tsc = cur_tsc;
> >   		}
> >
> > @@ -880,6 +973,9 @@ main_loop(__attribute__((unused)) void *dummy)
> >   			if (nb_rx > 0)
> >   				process_pkts(qconf, pkts, nb_rx, portid);
> >   		}
> > +
> > +		drain_inbound_crypto_queues(qconf, &qconf->inbound);
> > +		drain_outbound_crypto_queues(qconf, &qconf->outbound);
> 
> drain_inbound_crypto_queues and drain_outbound_crypto_queues should be called based on diff_tsc.
> moving these two lines above after  drain_crypto_buffers will improve the performance drop caused due to this patch.

Thanks, good to know.
To make what you suggest above to work properly with non-legacy mode ('-l') extra changes
would be needed...  
Do you have an idea - what exactly causing a slowdown?
Just an extra function calls (drain_inbound_crypto_queues/ drain_outbound_crypto_queues)?
Or is that because we do dequeue() from crypto PMD more often then before?  
Konstantin
 

Re: [PATCH v5 03/10] examples/ipsec-secgw: fix crypto-op might never get dequeued

[Akhil Goyal]

On 1/2/2019 7:13 PM, Ananyev, Konstantin wrote:
>
>> On 12/28/2018 9:03 PM, Konstantin Ananyev wrote:
>>> In some cases crypto-ops could never be dequeued from the crypto-device.
>>> The easiest way to reproduce:
>>> start ipsec-secgw with crypto-dev and send to it less then 32 packets.
>>> none packets will be forwarded.
>>> Reason for that is that the application does dequeue() from crypto-queues
>>> only when new packets arrive.
>>> This patch makes sure it calls dequeue() on a regular basis.
>>>
>>> Fixes: c64278c0c18b ("examples/ipsec-secgw: rework processing loop")
>>> Cc: stable@dpdk.org
>>>
>>> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
>>> Acked-by: Radu Nicolau <radu.nicolau@intel.com>
>>> ---
>>>    examples/ipsec-secgw/ipsec-secgw.c | 136 ++++++++++++++++++++++++-----
>>>    examples/ipsec-secgw/ipsec.c       |  60 ++++++++-----
>>>    examples/ipsec-secgw/ipsec.h       |  11 +++
>>>    3 files changed, 165 insertions(+), 42 deletions(-)
>> [snip]
>>> +
>>>    /* main processing loop */
>>>    static int32_t
>>>    main_loop(__attribute__((unused)) void *dummy)
>>> @@ -866,7 +958,8 @@ main_loop(__attribute__((unused)) void *dummy)
>>>    		diff_tsc = cur_tsc - prev_tsc;
>>>
>>>    		if (unlikely(diff_tsc > drain_tsc)) {
>>> -			drain_buffers(qconf);
>>> +			drain_tx_buffers(qconf);
>>> +			drain_crypto_buffers(qconf);
>>>    			prev_tsc = cur_tsc;
>>>    		}
>>>
>>> @@ -880,6 +973,9 @@ main_loop(__attribute__((unused)) void *dummy)
>>>    			if (nb_rx > 0)
>>>    				process_pkts(qconf, pkts, nb_rx, portid);
>>>    		}
>>> +
>>> +		drain_inbound_crypto_queues(qconf, &qconf->inbound);
>>> +		drain_outbound_crypto_queues(qconf, &qconf->outbound);
>> drain_inbound_crypto_queues and drain_outbound_crypto_queues should be called based on diff_tsc.
>> moving these two lines above after  drain_crypto_buffers will improve the performance drop caused due to this patch.
> Thanks, good to know.
> To make what you suggest above to work properly with non-legacy mode ('-l') extra changes
> would be needed...
What changes do you see?
> Do you have an idea - what exactly causing a slowdown?
> Just an extra function calls (drain_inbound_crypto_queues/ drain_outbound_crypto_queues)?
> Or is that because we do dequeue() from crypto PMD more often then before?
I have not profiled it, but it should be because of more dequeues. On a 
single call to dequeue, a burst of packets get dequeued. but now there 
will be a lot more dequeues which have lesser packets than the burst 
size which will deteriorate the performance as it would be wasting the 
dequeue cycles.

This patch is causing around 5% drop out of the 10% that I mentioned in 
the other mail.
With the change that I suggested, I am almost able to get back those 5%.
> Konstantin
>   
>

Re: [PATCH v5 03/10] examples/ipsec-secgw: fix crypto-op might never get dequeued

[Ananyev, Konstantin]

> -----Original Message-----
> From: Akhil Goyal [mailto:akhil.goyal@nxp.com]
> Sent: Wednesday, January 2, 2019 1:51 PM
> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
> Cc: stable@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v5 03/10] examples/ipsec-secgw: fix crypto-op might never get dequeued
> 
> 
> 
> On 1/2/2019 7:13 PM, Ananyev, Konstantin wrote:
> >
> >> On 12/28/2018 9:03 PM, Konstantin Ananyev wrote:
> >>> In some cases crypto-ops could never be dequeued from the crypto-device.
> >>> The easiest way to reproduce:
> >>> start ipsec-secgw with crypto-dev and send to it less then 32 packets.
> >>> none packets will be forwarded.
> >>> Reason for that is that the application does dequeue() from crypto-queues
> >>> only when new packets arrive.
> >>> This patch makes sure it calls dequeue() on a regular basis.
> >>>
> >>> Fixes: c64278c0c18b ("examples/ipsec-secgw: rework processing loop")
> >>> Cc: stable@dpdk.org
> >>>
> >>> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> >>> Acked-by: Radu Nicolau <radu.nicolau@intel.com>
> >>> ---
> >>>    examples/ipsec-secgw/ipsec-secgw.c | 136 ++++++++++++++++++++++++-----
> >>>    examples/ipsec-secgw/ipsec.c       |  60 ++++++++-----
> >>>    examples/ipsec-secgw/ipsec.h       |  11 +++
> >>>    3 files changed, 165 insertions(+), 42 deletions(-)
> >> [snip]
> >>> +
> >>>    /* main processing loop */
> >>>    static int32_t
> >>>    main_loop(__attribute__((unused)) void *dummy)
> >>> @@ -866,7 +958,8 @@ main_loop(__attribute__((unused)) void *dummy)
> >>>    		diff_tsc = cur_tsc - prev_tsc;
> >>>
> >>>    		if (unlikely(diff_tsc > drain_tsc)) {
> >>> -			drain_buffers(qconf);
> >>> +			drain_tx_buffers(qconf);
> >>> +			drain_crypto_buffers(qconf);
> >>>    			prev_tsc = cur_tsc;
> >>>    		}
> >>>
> >>> @@ -880,6 +973,9 @@ main_loop(__attribute__((unused)) void *dummy)
> >>>    			if (nb_rx > 0)
> >>>    				process_pkts(qconf, pkts, nb_rx, portid);
> >>>    		}
> >>> +
> >>> +		drain_inbound_crypto_queues(qconf, &qconf->inbound);
> >>> +		drain_outbound_crypto_queues(qconf, &qconf->outbound);
> >> drain_inbound_crypto_queues and drain_outbound_crypto_queues should be called based on diff_tsc.
> >> moving these two lines above after  drain_crypto_buffers will improve the performance drop caused due to this patch.
> > Thanks, good to know.
> > To make what you suggest above to work properly with non-legacy mode ('-l') extra changes
> > would be needed...
> What changes do you see?

Non-legacy mode relies on a drain_crypto_queues() to dequeuer crypto-ops.
It doesn't do that as part of process_pkts().
It is doable, but it means I have to rework my patches a bit.

> > Do you have an idea - what exactly causing a slowdown?
> > Just an extra function calls (drain_inbound_crypto_queues/ drain_outbound_crypto_queues)?
> > Or is that because we do dequeue() from crypto PMD more often then before?
> I have not profiled it, but it should be because of more dequeues. On a
> single call to dequeue, a burst of packets get dequeued. but now there
> will be a lot more dequeues which have lesser packets than the burst
> size which will deteriorate the performance as it would be wasting the
> dequeue cycles.
> 
> This patch is causing around 5% drop out of the 10% that I mentioned in
> the other mail.
> With the change that I suggested, I am almost able to get back those 5%.

Great, any idea what causing other 5%?
Konstantin

Re: [PATCH v5 03/10] examples/ipsec-secgw: fix crypto-op might never get dequeued

[Ananyev, Konstantin]

Hi Akhil,

> >>> In some cases crypto-ops could never be dequeued from the crypto-device.
> >>> The easiest way to reproduce:
> >>> start ipsec-secgw with crypto-dev and send to it less then 32 packets.
> >>> none packets will be forwarded.
> >>> Reason for that is that the application does dequeue() from crypto-queues
> >>> only when new packets arrive.
> >>> This patch makes sure it calls dequeue() on a regular basis.
> >>>
> >>> Fixes: c64278c0c18b ("examples/ipsec-secgw: rework processing loop")
> >>> Cc: stable@dpdk.org
> >>>
> >>> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> >>> Acked-by: Radu Nicolau <radu.nicolau@intel.com>
> >>> ---
> >>>    examples/ipsec-secgw/ipsec-secgw.c | 136 ++++++++++++++++++++++++-----
> >>>    examples/ipsec-secgw/ipsec.c       |  60 ++++++++-----
> >>>    examples/ipsec-secgw/ipsec.h       |  11 +++
> >>>    3 files changed, 165 insertions(+), 42 deletions(-)
> >> [snip]
> >>> +
> >>>    /* main processing loop */
> >>>    static int32_t
> >>>    main_loop(__attribute__((unused)) void *dummy)
> >>> @@ -866,7 +958,8 @@ main_loop(__attribute__((unused)) void *dummy)
> >>>    		diff_tsc = cur_tsc - prev_tsc;
> >>>
> >>>    		if (unlikely(diff_tsc > drain_tsc)) {
> >>> -			drain_buffers(qconf);
> >>> +			drain_tx_buffers(qconf);
> >>> +			drain_crypto_buffers(qconf);
> >>>    			prev_tsc = cur_tsc;
> >>>    		}
> >>>
> >>> @@ -880,6 +973,9 @@ main_loop(__attribute__((unused)) void *dummy)
> >>>    			if (nb_rx > 0)
> >>>    				process_pkts(qconf, pkts, nb_rx, portid);
> >>>    		}
> >>> +
> >>> +		drain_inbound_crypto_queues(qconf, &qconf->inbound);
> >>> +		drain_outbound_crypto_queues(qconf, &qconf->outbound);
> >> drain_inbound_crypto_queues and drain_outbound_crypto_queues should be called based on diff_tsc.
> >> moving these two lines above after  drain_crypto_buffers will improve the performance drop caused due to this patch.
> > Thanks, good to know.
> > To make what you suggest above to work properly with non-legacy mode ('-l') extra changes
> > would be needed...
> What changes do you see?
> > Do you have an idea - what exactly causing a slowdown?
> > Just an extra function calls (drain_inbound_crypto_queues/ drain_outbound_crypto_queues)?
> > Or is that because we do dequeue() from crypto PMD more often then before?
> I have not profiled it, but it should be because of more dequeues. On a
> single call to dequeue, a burst of packets get dequeued. but now there
> will be a lot more dequeues which have lesser packets than the burst
> size which will deteriorate the performance as it would be wasting the
> dequeue cycles.
> 
> This patch is causing around 5% drop out of the 10% that I mentioned in
> the other mail.
> With the change that I suggested, I am almost able to get back those 5%.

After another thought, I suppose we have 2 options here:
1. move drain_crypto_queues into timeout section
    (as you suggested above).
2. Make pkt_process() not to dequeue packets from crypto-dev
    (leave that to drain_crypto_queues).

For V6 I choose option #2 - it seems a bit cleaner and easier to me,
plus on my box it is ~1-1.5% faster than option #1.
Also V6 contains few other fixes for the issues you reported.
Please give it a try, if you'll still find it slower than #1 on your HW -
NP to use option #1.
Konstantin

[PATCH v5 04/10] examples/ipsec-secgw: fix outbound codepath for single SA

[Konstantin Ananyev]

Looking at process_pkts_outbound_nosp() there seems few issues:
- accessing mbuf after it was freed
- invoking ipsec_outbound() for ipv4 packets only
- copying number of packets, but not the mbuf pointers itself

that patch provides fixes for that issues.

Fixes: 906257e965b7 ("examples/ipsec-secgw: support IPv6")
Cc: stable@dpdk.org

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 33 +++++++++++++++++++++---------
 1 file changed, 23 insertions(+), 10 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 797bd6435..cc812d502 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -629,32 +629,45 @@ process_pkts_outbound_nosp(struct ipsec_ctx *ipsec_ctx,
 		struct ipsec_traffic *traffic)
 {
 	struct rte_mbuf *m;
-	uint32_t nb_pkts_out, i;
+	uint32_t nb_pkts_out, i, n;
 	struct ip *ip;
 
 	/* Drop any IPsec traffic from protected ports */
 	for (i = 0; i < traffic->ipsec.num; i++)
 		rte_pktmbuf_free(traffic->ipsec.pkts[i]);
 
-	traffic->ipsec.num = 0;
+	n = 0;
 
-	for (i = 0; i < traffic->ip4.num; i++)
-		traffic->ip4.res[i] = single_sa_idx;
+	for (i = 0; i < traffic->ip4.num; i++) {
+		traffic->ipsec.pkts[n] = traffic->ip4.pkts[i];
+		traffic->ipsec.res[n++] = single_sa_idx;
+	}
 
-	for (i = 0; i < traffic->ip6.num; i++)
-		traffic->ip6.res[i] = single_sa_idx;
+	for (i = 0; i < traffic->ip6.num; i++) {
+		traffic->ipsec.pkts[n] = traffic->ip6.pkts[i];
+		traffic->ipsec.res[n++] = single_sa_idx;
+	}
+
+	traffic->ip4.num = 0;
+	traffic->ip6.num = 0;
+	traffic->ipsec.num = n;
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ip4.pkts,
-			traffic->ip4.res, traffic->ip4.num,
+	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.res, traffic->ipsec.num,
 			MAX_PKT_BURST);
 
 	/* They all sue the same SA (ip4 or ip6 tunnel) */
 	m = traffic->ipsec.pkts[i];
 	ip = rte_pktmbuf_mtod(m, struct ip *);
-	if (ip->ip_v == IPVERSION)
+	if (ip->ip_v == IPVERSION) {
 		traffic->ip4.num = nb_pkts_out;
-	else
+		for (i = 0; i < nb_pkts_out; i++)
+			traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+	} else {
 		traffic->ip6.num = nb_pkts_out;
+		for (i = 0; i < nb_pkts_out; i++)
+			traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+	}
 }
 
 static inline int32_t
-- 
2.17.1

[PATCH v5 05/10] examples/ipsec-secgw: make local variables static

[Konstantin Ananyev]

in sp4.c and sp6.c there are few globals that used only locally.
Define them as static ones.

Cc: stable@dpdk.org

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/sp4.c | 10 +++++-----
 examples/ipsec-secgw/sp6.c | 10 +++++-----
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/examples/ipsec-secgw/sp4.c b/examples/ipsec-secgw/sp4.c
index 8d3d3d8e0..6b05daaa9 100644
--- a/examples/ipsec-secgw/sp4.c
+++ b/examples/ipsec-secgw/sp4.c
@@ -44,7 +44,7 @@ enum {
 	RTE_ACL_IPV4_NUM
 };
 
-struct rte_acl_field_def ip4_defs[NUM_FIELDS_IPV4] = {
+static struct rte_acl_field_def ip4_defs[NUM_FIELDS_IPV4] = {
 	{
 	.type = RTE_ACL_FIELD_TYPE_BITMASK,
 	.size = sizeof(uint8_t),
@@ -85,11 +85,11 @@ struct rte_acl_field_def ip4_defs[NUM_FIELDS_IPV4] = {
 
 RTE_ACL_RULE_DEF(acl4_rules, RTE_DIM(ip4_defs));
 
-struct acl4_rules acl4_rules_out[MAX_ACL_RULE_NUM];
-uint32_t nb_acl4_rules_out;
+static struct acl4_rules acl4_rules_out[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl4_rules_out;
 
-struct acl4_rules acl4_rules_in[MAX_ACL_RULE_NUM];
-uint32_t nb_acl4_rules_in;
+static struct acl4_rules acl4_rules_in[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl4_rules_in;
 
 void
 parse_sp4_tokens(char **tokens, uint32_t n_tokens,
diff --git a/examples/ipsec-secgw/sp6.c b/examples/ipsec-secgw/sp6.c
index 6002afef3..dc5b94c6a 100644
--- a/examples/ipsec-secgw/sp6.c
+++ b/examples/ipsec-secgw/sp6.c
@@ -34,7 +34,7 @@ enum {
 
 #define IP6_ADDR_SIZE 16
 
-struct rte_acl_field_def ip6_defs[IP6_NUM] = {
+static struct rte_acl_field_def ip6_defs[IP6_NUM] = {
 	{
 	.type = RTE_ACL_FIELD_TYPE_BITMASK,
 	.size = sizeof(uint8_t),
@@ -116,11 +116,11 @@ struct rte_acl_field_def ip6_defs[IP6_NUM] = {
 
 RTE_ACL_RULE_DEF(acl6_rules, RTE_DIM(ip6_defs));
 
-struct acl6_rules acl6_rules_out[MAX_ACL_RULE_NUM];
-uint32_t nb_acl6_rules_out;
+static struct acl6_rules acl6_rules_out[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl6_rules_out;
 
-struct acl6_rules acl6_rules_in[MAX_ACL_RULE_NUM];
-uint32_t nb_acl6_rules_in;
+static struct acl6_rules acl6_rules_in[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl6_rules_in;
 
 void
 parse_sp6_tokens(char **tokens, uint32_t n_tokens,
-- 
2.17.1

[PATCH v5 06/10] examples/ipsec-secgw: fix inbound SA checking

[Konstantin Ananyev]

In the inbound_sa_check() make sure that sa pointer stored
inside mbuf private area is not NULL.

Fixes: d299106e8e31 ("examples/ipsec-secgw: add IPsec sample application")
Cc: stable@dpdk.org

Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/sa.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index f6271bc60..839aaca0c 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -947,10 +947,15 @@ int
 inbound_sa_check(struct sa_ctx *sa_ctx, struct rte_mbuf *m, uint32_t sa_idx)
 {
 	struct ipsec_mbuf_metadata *priv;
+	struct ipsec_sa *sa;
 
 	priv = get_priv(m);
+	sa = priv->sa;
+	if (sa != NULL)
+		return (sa_ctx->sa[sa_idx].spi == sa->spi);
 
-	return (sa_ctx->sa[sa_idx].spi == priv->sa->spi);
+	RTE_LOG(ERR, IPSEC, "SA not saved in private data\n");
+	return 0;
 }
 
 static inline void
-- 
2.17.1

[PATCH v5 07/10] examples/ipsec-secgw: make app to use ipsec library

[Konstantin Ananyev]

Changes to make ipsec-secgw to utilize librte_ipsec library.
That patch provides:
 - changes in the related data structures.
 - changes in the initialization code.
 - new command-line parameters to enable librte_ipsec codepath
   and related features.

Note that right now by default current (non-librte_ipsec) code-path will
be used. User has to run application with new command-line option ('-l')
to enable new codepath.
The main reason for that:
 - current librte_ipsec doesn't support all ipsec algorithms
   and features that the app does.
 - allow users to run both versions in parallel for some time
   to figure out any functional or performance degradation with the
   new code.

It is planned to deprecate and remove non-librte_ipsec code path
in future releases.

Signed-off-by: Mohammad Abdul Awal <mohammad.abdul.awal@intel.com>
Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/Makefile      |   4 +-
 examples/ipsec-secgw/ipsec-secgw.c |  51 ++++++-
 examples/ipsec-secgw/ipsec.h       |  24 ++++
 examples/ipsec-secgw/meson.build   |   2 +-
 examples/ipsec-secgw/sa.c          | 221 ++++++++++++++++++++++++++++-
 examples/ipsec-secgw/sp4.c         |  25 ++++
 examples/ipsec-secgw/sp6.c         |  25 ++++
 7 files changed, 344 insertions(+), 8 deletions(-)

diff --git a/examples/ipsec-secgw/Makefile b/examples/ipsec-secgw/Makefile
index 02d41e39a..3918ee63e 100644
--- a/examples/ipsec-secgw/Makefile
+++ b/examples/ipsec-secgw/Makefile
@@ -61,8 +61,8 @@ RTE_TARGET ?= x86_64-native-linuxapp-gcc
 include $(RTE_SDK)/mk/rte.vars.mk
 
 ifneq ($(MAKECMDGOALS),clean)
-ifneq ($(CONFIG_RTE_LIBRTE_SECURITY),y)
-$(error "RTE_LIBRTE_SECURITY is required to build ipsec-secgw")
+ifneq ($(CONFIG_RTE_LIBRTE_IPSEC),y)
+$(error "RTE_LIBRTE_IPSEC is required to build ipsec-secgw")
 endif
 endif
 
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index cc812d502..c2c9cb2bd 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -168,6 +168,9 @@ static uint32_t frame_size;
 static uint64_t dev_rx_offload = UINT64_MAX;
 static uint64_t dev_tx_offload = UINT64_MAX;
 
+/* application wide librte_ipsec/SA parameters */
+struct app_sa_prm app_sa_prm = {.enable = 0};
+
 struct lcore_rx_queue {
 	uint16_t port_id;
 	uint8_t queue_id;
@@ -1077,6 +1080,10 @@ print_usage(const char *prgname)
 		" [-P]"
 		" [-u PORTMASK]"
 		" [-j FRAMESIZE]"
+		" [-l]"
+		" [-w REPLAY_WINDOW_SIZE]"
+		" [-e]"
+		" [-a]"
 		" -f CONFIG_FILE"
 		" --config (port,queue,lcore)[,(port,queue,lcore)]"
 		" [--single-sa SAIDX]"
@@ -1089,6 +1096,11 @@ print_usage(const char *prgname)
 		"  -u PORTMASK: Hexadecimal bitmask of unprotected ports\n"
 		"  -j FRAMESIZE: Enable jumbo frame with 'FRAMESIZE' as maximum\n"
 		"                packet size\n"
+		"  -l enables code-path that uses librte_ipsec\n"
+		"  -w REPLAY_WINDOW_SIZE specifies IPsec SQN replay window\n"
+		"     size for each SA\n"
+		"  -e enables ESN\n"
+		"  -a enables SA SQN atomic behaviour\n"
 		"  -f CONFIG_FILE: Configuration file\n"
 		"  --config (port,queue,lcore): Rx queue configuration\n"
 		"  --single-sa SAIDX: Use single SA index for outbound traffic,\n"
@@ -1206,6 +1218,20 @@ parse_config(const char *q_arg)
 	return 0;
 }
 
+static void
+print_app_sa_prm(const struct app_sa_prm *prm)
+{
+	printf("librte_ipsec usage: %s\n",
+		(prm->enable == 0) ? "disabled" : "enabled");
+
+	if (prm->enable == 0)
+		return;
+
+	printf("replay window size: %u\n", prm->window_size);
+	printf("ESN: %s\n", (prm->enable_esn == 0) ? "disabled" : "enabled");
+	printf("SA flags: %#" PRIx64 "\n", prm->flags);
+}
+
 static int32_t
 parse_args(int32_t argc, char **argv)
 {
@@ -1217,7 +1243,7 @@ parse_args(int32_t argc, char **argv)
 
 	argvopt = argv;
 
-	while ((opt = getopt_long(argc, argvopt, "p:Pu:f:j:",
+	while ((opt = getopt_long(argc, argvopt, "aelp:Pu:f:j:w:",
 				lgopts, &option_index)) != EOF) {
 
 		switch (opt) {
@@ -1273,6 +1299,21 @@ parse_args(int32_t argc, char **argv)
 			}
 			printf("Enabled jumbo frames size %u\n", frame_size);
 			break;
+		case 'l':
+			app_sa_prm.enable = 1;
+			break;
+		case 'w':
+			app_sa_prm.enable = 1;
+			app_sa_prm.window_size = parse_decimal(optarg);
+			break;
+		case 'e':
+			app_sa_prm.enable = 1;
+			app_sa_prm.enable_esn = 1;
+			break;
+		case 'a':
+			app_sa_prm.enable = 1;
+			app_sa_prm.flags |= RTE_IPSEC_SAFLAG_SQN_ATOM;
+			break;
 		case CMD_LINE_OPT_CONFIG_NUM:
 			ret = parse_config(optarg);
 			if (ret) {
@@ -1335,6 +1376,8 @@ parse_args(int32_t argc, char **argv)
 		return -1;
 	}
 
+	print_app_sa_prm(&app_sa_prm);
+
 	if (optind >= 0)
 		argv[optind-1] = prgname;
 
@@ -2006,12 +2049,14 @@ main(int32_t argc, char **argv)
 		if (socket_ctx[socket_id].mbuf_pool)
 			continue;
 
-		sa_init(&socket_ctx[socket_id], socket_id);
-
+		/* initilaze SPD */
 		sp4_init(&socket_ctx[socket_id], socket_id);
 
 		sp6_init(&socket_ctx[socket_id], socket_id);
 
+		/* initilaze SAD */
+		sa_init(&socket_ctx[socket_id], socket_id);
+
 		rt_init(&socket_ctx[socket_id], socket_id);
 
 		pool_init(&socket_ctx[socket_id], socket_id, NB_MBUF);
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 2f04b7d68..8da86c4de 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -11,6 +11,7 @@
 #include <rte_crypto.h>
 #include <rte_security.h>
 #include <rte_flow.h>
+#include <rte_ipsec.h>
 
 #define RTE_LOGTYPE_IPSEC       RTE_LOGTYPE_USER1
 #define RTE_LOGTYPE_IPSEC_ESP   RTE_LOGTYPE_USER2
@@ -70,7 +71,20 @@ struct ip_addr {
 
 #define MAX_KEY_SIZE		32
 
+/*
+ * application wide SA parameters
+ */
+struct app_sa_prm {
+	uint32_t enable; /* use librte_ipsec API for ipsec pkt processing */
+	uint32_t window_size; /* replay window size */
+	uint32_t enable_esn;  /* enable/disable ESN support */
+	uint64_t flags;       /* rte_ipsec_sa_prm.flags */
+};
+
+extern struct app_sa_prm app_sa_prm;
+
 struct ipsec_sa {
+	struct rte_ipsec_session ips; /* one session per sa for now */
 	uint32_t spi;
 	uint32_t cdev_id_qp;
 	uint64_t seq;
@@ -243,6 +257,16 @@ sp4_init(struct socket_ctx *ctx, int32_t socket_id);
 void
 sp6_init(struct socket_ctx *ctx, int32_t socket_id);
 
+/*
+ * Search through SP rules for given SPI.
+ * Returns first rule index if found(greater or equal then zero),
+ * or -ENOENT otherwise.
+ */
+int
+sp4_spi_present(uint32_t spi, int inbound);
+int
+sp6_spi_present(uint32_t spi, int inbound);
+
 void
 sa_init(struct socket_ctx *ctx, int32_t socket_id);
 
diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
index 77d8b298f..31f68fee2 100644
--- a/examples/ipsec-secgw/meson.build
+++ b/examples/ipsec-secgw/meson.build
@@ -6,7 +6,7 @@
 # To build this example as a standalone application with an already-installed
 # DPDK instance, use 'make'
 
-deps += ['security', 'lpm', 'acl', 'hash']
+deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
 allow_experimental_apis = true
 sources = files(
 	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index 839aaca0c..414fcd26c 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -19,6 +19,7 @@
 #include <rte_ip.h>
 #include <rte_random.h>
 #include <rte_ethdev.h>
+#include <rte_malloc.h>
 
 #include "ipsec.h"
 #include "esp.h"
@@ -694,6 +695,7 @@ print_one_sa_rule(const struct ipsec_sa *sa, int inbound)
 }
 
 struct sa_ctx {
+	void *satbl; /* pointer to array of rte_ipsec_sa objects*/
 	struct ipsec_sa sa[IPSEC_SA_MAX_ENTRIES];
 	union {
 		struct {
@@ -764,7 +766,10 @@ sa_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 {
 	struct ipsec_sa *sa;
 	uint32_t i, idx;
-	uint16_t iv_length;
+	uint16_t iv_length, aad_length;
+
+	/* for ESN upper 32 bits of SQN also need to be part of AAD */
+	aad_length = (app_sa_prm.enable_esn != 0) ? sizeof(uint32_t) : 0;
 
 	for (i = 0; i < nb_entries; i++) {
 		idx = SPI2IDX(entries[i].spi);
@@ -808,7 +813,7 @@ sa_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 			sa_ctx->xf[idx].a.aead.iv.offset = IV_OFFSET;
 			sa_ctx->xf[idx].a.aead.iv.length = iv_length;
 			sa_ctx->xf[idx].a.aead.aad_length =
-				sa->aad_len;
+				sa->aad_len + aad_length;
 			sa_ctx->xf[idx].a.aead.digest_length =
 				sa->digest_len;
 
@@ -901,9 +906,205 @@ sa_in_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 	return sa_add_rules(sa_ctx, entries, nb_entries, 1);
 }
 
+/*
+ * helper function, fills parameters that are identical for all SAs
+ */
+static void
+fill_ipsec_app_sa_prm(struct rte_ipsec_sa_prm *prm,
+	const struct app_sa_prm *app_prm)
+{
+	memset(prm, 0, sizeof(*prm));
+
+	prm->flags = app_prm->flags;
+	prm->ipsec_xform.options.esn = app_prm->enable_esn;
+	prm->replay_win_sz = app_prm->window_size;
+}
+
+/*
+ * Helper function, tries to determine next_proto for SPI
+ * by searching though SP rules.
+ */
+static int
+get_spi_proto(uint32_t spi, enum rte_security_ipsec_sa_direction dir)
+{
+	int32_t rc4, rc6;
+
+	rc4 = sp4_spi_present(spi, dir == RTE_SECURITY_IPSEC_SA_DIR_INGRESS);
+	rc6 = sp6_spi_present(spi, dir == RTE_SECURITY_IPSEC_SA_DIR_INGRESS);
+
+	if (rc4 >= 0) {
+		if (rc6 >= 0) {
+			RTE_LOG(ERR, IPSEC,
+				"%s: SPI %u used simultaeously by "
+				"IPv4(%d) and IPv6 (%d) SP rules\n",
+				__func__, spi, rc4, rc6);
+			return -EINVAL;
+		} else
+			return IPPROTO_IPIP;
+	} else if (rc6 < 0) {
+		RTE_LOG(ERR, IPSEC,
+			"%s: SPI %u is not used by any SP rule\n",
+			__func__, spi);
+		return -EINVAL;
+	} else
+		return IPPROTO_IPV6;
+}
+
+static int
+fill_ipsec_sa_prm(struct rte_ipsec_sa_prm *prm, const struct ipsec_sa *ss,
+	const struct ipv4_hdr *v4, struct ipv6_hdr *v6)
+{
+	int32_t rc;
+
+	/*
+	 * Try to get SPI next proto by searching that SPI in SPD.
+	 * probably not the optimal way, but there seems nothing
+	 * better right now.
+	 */
+	rc = get_spi_proto(ss->spi, ss->direction);
+	if (rc < 0)
+		return rc;
+
+	fill_ipsec_app_sa_prm(prm, &app_sa_prm);
+	prm->userdata = (uintptr_t)ss;
+
+	/* setup ipsec xform */
+	prm->ipsec_xform.spi = ss->spi;
+	prm->ipsec_xform.salt = ss->salt;
+	prm->ipsec_xform.direction = ss->direction;
+	prm->ipsec_xform.proto = RTE_SECURITY_IPSEC_SA_PROTO_ESP;
+	prm->ipsec_xform.mode = (ss->flags == TRANSPORT) ?
+		RTE_SECURITY_IPSEC_SA_MODE_TRANSPORT :
+		RTE_SECURITY_IPSEC_SA_MODE_TUNNEL;
+
+	if (ss->flags == IP4_TUNNEL) {
+		prm->ipsec_xform.tunnel.type = RTE_SECURITY_IPSEC_TUNNEL_IPV4;
+		prm->tun.hdr_len = sizeof(*v4);
+		prm->tun.next_proto = rc;
+		prm->tun.hdr = v4;
+	} else if (ss->flags == IP6_TUNNEL) {
+		prm->ipsec_xform.tunnel.type = RTE_SECURITY_IPSEC_TUNNEL_IPV6;
+		prm->tun.hdr_len = sizeof(*v6);
+		prm->tun.next_proto = rc;
+		prm->tun.hdr = v6;
+	} else {
+		/* transport mode */
+		prm->trs.proto = rc;
+	}
+
+	/* setup crypto section */
+	prm->crypto_xform = ss->xforms;
+	return 0;
+}
+
+static void
+fill_ipsec_session(struct rte_ipsec_session *ss, struct rte_ipsec_sa *sa,
+	const struct ipsec_sa *lsa)
+{
+	ss->sa = sa;
+	ss->type = lsa->type;
+
+	/* setup crypto section */
+	if (ss->type == RTE_SECURITY_ACTION_TYPE_NONE) {
+		ss->crypto.ses = lsa->crypto_session;
+	/* setup session action type */
+	} else {
+		ss->security.ses = lsa->sec_session;
+		ss->security.ctx = lsa->security_ctx;
+		ss->security.ol_flags = lsa->ol_flags;
+	}
+}
+
+/*
+ * Initialise related rte_ipsec_sa object.
+ */
+static int
+ipsec_sa_init(struct ipsec_sa *lsa, struct rte_ipsec_sa *sa, uint32_t sa_size)
+{
+	int rc;
+	struct rte_ipsec_sa_prm prm;
+	struct ipv4_hdr v4  = {
+		.version_ihl = IPVERSION << 4 |
+			sizeof(v4) / IPV4_IHL_MULTIPLIER,
+		.time_to_live = IPDEFTTL,
+		.next_proto_id = IPPROTO_ESP,
+		.src_addr = lsa->src.ip.ip4,
+		.dst_addr = lsa->dst.ip.ip4,
+	};
+	struct ipv6_hdr v6 = {
+		.vtc_flow = htonl(IP6_VERSION << 28),
+		.proto = IPPROTO_ESP,
+	};
+
+	if (lsa->flags == IP6_TUNNEL) {
+		memcpy(v6.src_addr, lsa->src.ip.ip6.ip6_b, sizeof(v6.src_addr));
+		memcpy(v6.dst_addr, lsa->dst.ip.ip6.ip6_b, sizeof(v6.dst_addr));
+	}
+
+	rc = fill_ipsec_sa_prm(&prm, lsa, &v4, &v6);
+	if (rc == 0)
+		rc = rte_ipsec_sa_init(sa, &prm, sa_size);
+	if (rc < 0)
+		return rc;
+
+	fill_ipsec_session(&lsa->ips, sa, lsa);
+	return 0;
+}
+
+/*
+ * Allocate space and init rte_ipsec_sa strcutures,
+ * one per session.
+ */
+static int
+ipsec_satbl_init(struct sa_ctx *ctx, const struct ipsec_sa *ent,
+	uint32_t nb_ent, int32_t socket)
+{
+	int32_t rc, sz;
+	uint32_t i, idx;
+	size_t tsz;
+	struct rte_ipsec_sa *sa;
+	struct ipsec_sa *lsa;
+	struct rte_ipsec_sa_prm prm;
+
+	/* determine SA size */
+	idx = SPI2IDX(ent[0].spi);
+	fill_ipsec_sa_prm(&prm, ctx->sa + idx, NULL, NULL);
+	sz = rte_ipsec_sa_size(&prm);
+	if (sz < 0) {
+		RTE_LOG(ERR, IPSEC, "%s(%p, %u, %d): "
+			"failed to determine SA size, error code: %d\n",
+			__func__, ctx, nb_ent, socket, sz);
+		return sz;
+	}
+
+	tsz = sz * nb_ent;
+
+	ctx->satbl = rte_zmalloc_socket(NULL, tsz, RTE_CACHE_LINE_SIZE, socket);
+	if (ctx->satbl == NULL) {
+		RTE_LOG(ERR, IPSEC,
+			"%s(%p, %u, %d): failed to allocate %zu bytes\n",
+			__func__,  ctx, nb_ent, socket, tsz);
+		return -ENOMEM;
+	}
+
+	rc = 0;
+	for (i = 0; i != nb_ent && rc == 0; i++) {
+
+		idx = SPI2IDX(ent[i].spi);
+
+		sa = (struct rte_ipsec_sa *)((uintptr_t)ctx->satbl + sz * i);
+		lsa = ctx->sa + idx;
+
+		rc = ipsec_sa_init(lsa, sa, sz);
+	}
+
+	return rc;
+}
+
 void
 sa_init(struct socket_ctx *ctx, int32_t socket_id)
 {
+	int32_t rc;
 	const char *name;
 
 	if (ctx == NULL)
@@ -926,6 +1127,14 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id)
 				name, socket_id);
 
 		sa_in_add_rules(ctx->sa_in, sa_in, nb_sa_in);
+
+		if (app_sa_prm.enable != 0) {
+			rc = ipsec_satbl_init(ctx->sa_in, sa_in, nb_sa_in,
+				socket_id);
+			if (rc != 0)
+				rte_exit(EXIT_FAILURE,
+					"failed to init inbound SAs\n");
+		}
 	} else
 		RTE_LOG(WARNING, IPSEC, "No SA Inbound rule specified\n");
 
@@ -938,6 +1147,14 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id)
 				name, socket_id);
 
 		sa_out_add_rules(ctx->sa_out, sa_out, nb_sa_out);
+
+		if (app_sa_prm.enable != 0) {
+			rc = ipsec_satbl_init(ctx->sa_out, sa_out, nb_sa_out,
+				socket_id);
+			if (rc != 0)
+				rte_exit(EXIT_FAILURE,
+					"failed to init outbound SAs\n");
+		}
 	} else
 		RTE_LOG(WARNING, IPSEC, "No SA Outbound rule "
 			"specified\n");
diff --git a/examples/ipsec-secgw/sp4.c b/examples/ipsec-secgw/sp4.c
index 6b05daaa9..d1dc64bad 100644
--- a/examples/ipsec-secgw/sp4.c
+++ b/examples/ipsec-secgw/sp4.c
@@ -504,3 +504,28 @@ sp4_init(struct socket_ctx *ctx, int32_t socket_id)
 		RTE_LOG(WARNING, IPSEC, "No IPv4 SP Outbound rule "
 			"specified\n");
 }
+
+/*
+ * Search though SP rules for given SPI.
+ */
+int
+sp4_spi_present(uint32_t spi, int inbound)
+{
+	uint32_t i, num;
+	const struct acl4_rules *acr;
+
+	if (inbound != 0) {
+		acr = acl4_rules_in;
+		num = nb_acl4_rules_in;
+	} else {
+		acr = acl4_rules_out;
+		num = nb_acl4_rules_out;
+	}
+
+	for (i = 0; i != num; i++) {
+		if (acr[i].data.userdata == PROTECT(spi))
+			return i;
+	}
+
+	return -ENOENT;
+}
diff --git a/examples/ipsec-secgw/sp6.c b/examples/ipsec-secgw/sp6.c
index dc5b94c6a..e67d85aaf 100644
--- a/examples/ipsec-secgw/sp6.c
+++ b/examples/ipsec-secgw/sp6.c
@@ -618,3 +618,28 @@ sp6_init(struct socket_ctx *ctx, int32_t socket_id)
 		RTE_LOG(WARNING, IPSEC, "No IPv6 SP Outbound rule "
 			"specified\n");
 }
+
+/*
+ * Search though SP rules for given SPI.
+ */
+int
+sp6_spi_present(uint32_t spi, int inbound)
+{
+	uint32_t i, num;
+	const struct acl6_rules *acr;
+
+	if (inbound != 0) {
+		acr = acl6_rules_in;
+		num = nb_acl6_rules_in;
+	} else {
+		acr = acl6_rules_out;
+		num = nb_acl6_rules_out;
+	}
+
+	for (i = 0; i != num; i++) {
+		if (acr[i].data.userdata == PROTECT(spi))
+			return i;
+	}
+
+	return -ENOENT;
+}
-- 
2.17.1

[PATCH v5 08/10] examples/ipsec-secgw: make data-path to use ipsec library

[Konstantin Ananyev]

Changes to make ipsec-secgw data-path code to utilize librte_ipsec library.
Note that right now by default current (non-librte_ipsec) code-path will
be used. User has to run application with new command-line option ('-l')
to enable new codepath.

Signed-off-by: Mohammad Abdul Awal <mohammad.abdul.awal@intel.com>
Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/Makefile        |   1 +
 examples/ipsec-secgw/ipsec-secgw.c   | 179 ++++++++------
 examples/ipsec-secgw/ipsec.c         |   2 +-
 examples/ipsec-secgw/ipsec.h         |  23 ++
 examples/ipsec-secgw/ipsec_process.c | 341 +++++++++++++++++++++++++++
 examples/ipsec-secgw/meson.build     |   4 +-
 6 files changed, 471 insertions(+), 79 deletions(-)
 create mode 100644 examples/ipsec-secgw/ipsec_process.c

diff --git a/examples/ipsec-secgw/Makefile b/examples/ipsec-secgw/Makefile
index 3918ee63e..08f474da6 100644
--- a/examples/ipsec-secgw/Makefile
+++ b/examples/ipsec-secgw/Makefile
@@ -13,6 +13,7 @@ SRCS-y += sp4.c
 SRCS-y += sp6.c
 SRCS-y += sa.c
 SRCS-y += rt.c
+SRCS-y += ipsec_process.c
 SRCS-y += ipsec-secgw.c
 
 CFLAGS += -gdwarf-2
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index c2c9cb2bd..9a340acd6 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -229,19 +229,6 @@ static struct rte_eth_conf port_conf = {
 
 static struct socket_ctx socket_ctx[NB_SOCKETS];
 
-struct traffic_type {
-	const uint8_t *data[MAX_PKT_BURST * 2];
-	struct rte_mbuf *pkts[MAX_PKT_BURST * 2];
-	uint32_t res[MAX_PKT_BURST * 2];
-	uint32_t num;
-};
-
-struct ipsec_traffic {
-	struct traffic_type ipsec;
-	struct traffic_type ip4;
-	struct traffic_type ip6;
-};
-
 static inline void
 prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 {
@@ -258,6 +245,8 @@ prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 			t->ip4.data[t->ip4.num] = nlp;
 			t->ip4.pkts[(t->ip4.num)++] = pkt;
 		}
+		pkt->l2_len = 0;
+		pkt->l3_len = sizeof(struct ip);
 	} else if (eth->ether_type == rte_cpu_to_be_16(ETHER_TYPE_IPv6)) {
 		nlp = (uint8_t *)rte_pktmbuf_adj(pkt, ETHER_HDR_LEN);
 		nlp = RTE_PTR_ADD(nlp, offsetof(struct ip6_hdr, ip6_nxt));
@@ -267,6 +256,8 @@ prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 			t->ip6.data[t->ip6.num] = nlp;
 			t->ip6.pkts[(t->ip6.num)++] = pkt;
 		}
+		pkt->l2_len = 0;
+		pkt->l3_len = sizeof(struct ip6_hdr);
 	} else {
 		/* Unknown/Unsupported type, drop the packet */
 		RTE_LOG(ERR, IPSEC, "Unsupported packet type\n");
@@ -516,10 +507,15 @@ process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
 	n_ip4 = traffic->ip4.num;
 	n_ip6 = traffic->ip6.num;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
-
-	split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+	if (app_sa_prm.enable == 0) {
+		nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.num, MAX_PKT_BURST);
+		split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+	} else {
+		inbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
+	}
 
 	inbound_sp_sa(ipsec_ctx->sp4_ctx, ipsec_ctx->sa_ctx, &traffic->ip4,
 			n_ip4);
@@ -575,20 +571,27 @@ process_pkts_outbound(struct ipsec_ctx *ipsec_ctx,
 
 	outbound_sp(ipsec_ctx->sp6_ctx, &traffic->ip6, &traffic->ipsec);
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.res, traffic->ipsec.num,
-			MAX_PKT_BURST);
-
-	for (i = 0; i < nb_pkts_out; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
-		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-		} else {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
+	if (app_sa_prm.enable == 0) {
+
+		nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.res, traffic->ipsec.num,
+				MAX_PKT_BURST);
+
+		for (i = 0; i < nb_pkts_out; i++) {
+			m = traffic->ipsec.pkts[i];
+			struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
+			if (ip->ip_v == IPVERSION) {
+				idx = traffic->ip4.num++;
+				traffic->ip4.pkts[idx] = m;
+			} else {
+				idx = traffic->ip6.num++;
+				traffic->ip6.pkts[idx] = m;
+			}
 		}
+	} else {
+		outbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.res,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -611,19 +614,26 @@ process_pkts_inbound_nosp(struct ipsec_ctx *ipsec_ctx,
 
 	traffic->ip6.num = 0;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
+	if (app_sa_prm.enable == 0) {
 
-	for (i = 0; i < nb_pkts_in; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
-		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-		} else {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
+		nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.num, MAX_PKT_BURST);
+
+		for (i = 0; i < nb_pkts_in; i++) {
+			m = traffic->ipsec.pkts[i];
+			struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
+			if (ip->ip_v == IPVERSION) {
+				idx = traffic->ip4.num++;
+				traffic->ip4.pkts[idx] = m;
+			} else {
+				idx = traffic->ip6.num++;
+				traffic->ip6.pkts[idx] = m;
+			}
 		}
+	} else {
+		inbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -655,21 +665,28 @@ process_pkts_outbound_nosp(struct ipsec_ctx *ipsec_ctx,
 	traffic->ip6.num = 0;
 	traffic->ipsec.num = n;
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.res, traffic->ipsec.num,
-			MAX_PKT_BURST);
+	if (app_sa_prm.enable == 0) {
 
-	/* They all sue the same SA (ip4 or ip6 tunnel) */
-	m = traffic->ipsec.pkts[i];
-	ip = rte_pktmbuf_mtod(m, struct ip *);
-	if (ip->ip_v == IPVERSION) {
-		traffic->ip4.num = nb_pkts_out;
-		for (i = 0; i < nb_pkts_out; i++)
-			traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+		nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.res, traffic->ipsec.num,
+				MAX_PKT_BURST);
+
+		/* They all sue the same SA (ip4 or ip6 tunnel) */
+		m = traffic->ipsec.pkts[0];
+		ip = rte_pktmbuf_mtod(m, struct ip *);
+		if (ip->ip_v == IPVERSION) {
+			traffic->ip4.num = nb_pkts_out;
+			for (i = 0; i < nb_pkts_out; i++)
+				traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+		} else {
+			traffic->ip6.num = nb_pkts_out;
+			for (i = 0; i < nb_pkts_out; i++)
+				traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+		}
 	} else {
-		traffic->ip6.num = nb_pkts_out;
-		for (i = 0; i < nb_pkts_out; i++)
-			traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+		outbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.res,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -870,25 +887,31 @@ drain_inbound_crypto_queues(const struct lcore_conf *qconf,
 	uint32_t n;
 	struct ipsec_traffic trf;
 
-	/* dequeue packets from crypto-queue */
-	n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+	if (app_sa_prm.enable == 0) {
+
+		/* dequeue packets from crypto-queue */
+		n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
 			RTE_DIM(trf.ipsec.pkts));
-	if (n == 0)
-		return;
 
-	trf.ip4.num = 0;
-	trf.ip6.num = 0;
+		trf.ip4.num = 0;
+		trf.ip6.num = 0;
 
-	/* split traffic by ipv4-ipv6 */
-	split46_traffic(&trf, trf.ipsec.pkts, n);
+		/* split traffic by ipv4-ipv6 */
+		split46_traffic(&trf, trf.ipsec.pkts, n);
+	} else
+		ipsec_cqp_process(ctx, &trf);
 
 	/* process ipv4 packets */
-	inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
-	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	if (trf.ip4.num != 0) {
+		inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
+		route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	}
 
 	/* process ipv6 packets */
-	inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
-	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	if (trf.ip6.num != 0) {
+		inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
+		route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	}
 }
 
 static void
@@ -898,23 +921,27 @@ drain_outbound_crypto_queues(const struct lcore_conf *qconf,
 	uint32_t n;
 	struct ipsec_traffic trf;
 
-	/* dequeue packets from crypto-queue */
-	n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+	if (app_sa_prm.enable == 0) {
+
+		/* dequeue packets from crypto-queue */
+		n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
 			RTE_DIM(trf.ipsec.pkts));
-	if (n == 0)
-		return;
 
-	trf.ip4.num = 0;
-	trf.ip6.num = 0;
+		trf.ip4.num = 0;
+		trf.ip6.num = 0;
 
-	/* split traffic by ipv4-ipv6 */
-	split46_traffic(&trf, trf.ipsec.pkts, n);
+		/* split traffic by ipv4-ipv6 */
+		split46_traffic(&trf, trf.ipsec.pkts, n);
+	} else
+		ipsec_cqp_process(ctx, &trf);
 
 	/* process ipv4 packets */
-	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	if (trf.ip4.num != 0)
+		route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
 
 	/* process ipv6 packets */
-	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	if (trf.ip6.num != 0)
+		route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
 }
 
 /* main processing loop */
diff --git a/examples/ipsec-secgw/ipsec.c b/examples/ipsec-secgw/ipsec.c
index 8bf928a23..8e7fc3639 100644
--- a/examples/ipsec-secgw/ipsec.c
+++ b/examples/ipsec-secgw/ipsec.c
@@ -39,7 +39,7 @@ set_ipsec_conf(struct ipsec_sa *sa, struct rte_security_ipsec_xform *ipsec)
 	ipsec->esn_soft_limit = IPSEC_OFFLOAD_ESN_SOFTLIMIT;
 }
 
-static inline int
+int
 create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa)
 {
 	struct rte_cryptodev_info cdev_info;
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 8da86c4de..8629f5045 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -190,6 +190,20 @@ struct cnt_blk {
 	uint32_t cnt;
 } __attribute__((packed));
 
+struct traffic_type {
+	const uint8_t *data[MAX_PKT_BURST * 2];
+	struct rte_mbuf *pkts[MAX_PKT_BURST * 2];
+	struct ipsec_sa *saptr[MAX_PKT_BURST * 2];
+	uint32_t res[MAX_PKT_BURST * 2];
+	uint32_t num;
+};
+
+struct ipsec_traffic {
+	struct traffic_type ipsec;
+	struct traffic_type ip4;
+	struct traffic_type ip6;
+};
+
 uint16_t
 ipsec_inbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint16_t nb_pkts, uint16_t len);
@@ -206,6 +220,12 @@ uint16_t
 ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint16_t len);
 
+void
+ipsec_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf);
+
+void
+ipsec_cqp_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf);
+
 static inline uint16_t
 ipsec_metadata_size(void)
 {
@@ -283,4 +303,7 @@ add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
 void
 enqueue_cop_burst(struct cdev_qp *cqp);
 
+int
+create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/ipsec_process.c b/examples/ipsec-secgw/ipsec_process.c
new file mode 100644
index 000000000..7ab378f6a
--- /dev/null
+++ b/examples/ipsec-secgw/ipsec_process.c
@@ -0,0 +1,341 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2017 Intel Corporation
+ */
+#include <sys/types.h>
+#include <netinet/in.h>
+#include <netinet/ip.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_log.h>
+#include <rte_cryptodev.h>
+#include <rte_ethdev.h>
+#include <rte_mbuf.h>
+
+#include "ipsec.h"
+
+#define SATP_OUT_IPV4(t)	\
+	((((t) & RTE_IPSEC_SATP_MODE_MASK) == RTE_IPSEC_SATP_MODE_TRANS && \
+	(((t) & RTE_IPSEC_SATP_IPV_MASK) == RTE_IPSEC_SATP_IPV4)) || \
+	((t) & RTE_IPSEC_SATP_MODE_MASK) == RTE_IPSEC_SATP_MODE_TUNLV4)
+
+
+/* helper routine to free bulk of packets */
+static inline void
+free_pkts(struct rte_mbuf *mb[], uint32_t n)
+{
+	uint32_t i;
+
+	for (i = 0; i != n; i++)
+		rte_pktmbuf_free(mb[i]);
+}
+
+/* helper routine to free bulk of crypto-ops and related packets */
+static inline void
+free_cops(struct rte_crypto_op *cop[], uint32_t n)
+{
+	uint32_t i;
+
+	for (i = 0; i != n; i++)
+		rte_pktmbuf_free(cop[i]->sym->m_src);
+}
+
+/* helper routine to enqueue bulk of crypto ops */
+static inline void
+enqueue_cop_bulk(struct cdev_qp *cqp, struct rte_crypto_op *cop[], uint32_t num)
+{
+	uint32_t i, k, len, n;
+
+	len = cqp->len;
+
+	/*
+	 * if cqp is empty and we have enough ops,
+	 * then queue them to the PMD straightway.
+	 */
+	if (num >= RTE_DIM(cqp->buf) * 3 / 4 && len == 0) {
+		n = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp, cop, num);
+		cqp->in_flight += n;
+		free_cops(cop + n, num - n);
+		return;
+	}
+
+	k = 0;
+
+	do {
+		n = RTE_DIM(cqp->buf) - len;
+		n = RTE_MIN(num - k, n);
+
+		/* put packets into cqp */
+		for (i = 0; i != n; i++)
+			cqp->buf[len + i] = cop[k + i];
+
+		len += n;
+		k += n;
+
+		/* if cqp is full then, enqueue crypto-ops to PMD */
+		if (len == RTE_DIM(cqp->buf)) {
+			n = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp,
+					cqp->buf, len);
+			cqp->in_flight += n;
+			free_cops(cqp->buf + n, len - n);
+			len = 0;
+		}
+
+
+	} while (k != num);
+
+	cqp->len = len;
+}
+
+static inline int
+fill_ipsec_session(struct rte_ipsec_session *ss, const struct ipsec_sa *sa)
+{
+	/* setup crypto section */
+	if (ss->type == RTE_SECURITY_ACTION_TYPE_NONE) {
+		ss->crypto.ses = sa->crypto_session;
+	/* setup session action type */
+	} else {
+		ss->security.ses = sa->sec_session;
+		ss->security.ctx = sa->security_ctx;
+		ss->security.ol_flags = sa->ol_flags;
+	}
+
+	return rte_ipsec_session_prepare(ss);
+}
+
+/*
+ * group input packets byt the SA they belong to.
+ */
+static uint32_t
+sa_group(struct ipsec_sa *sa_ptr[], struct rte_mbuf *pkts[],
+	struct rte_ipsec_group grp[], uint32_t num)
+{
+	uint32_t i, n, spi;
+	void *sa;
+	void * const nosa = &spi;
+
+	sa = nosa;
+	for (i = 0, n = 0; i != num; i++) {
+
+		if (sa != sa_ptr[i]) {
+			grp[n].cnt = pkts + i - grp[n].m;
+			n += (sa != nosa);
+			grp[n].id.ptr = sa_ptr[i];
+			grp[n].m = pkts + i;
+			sa = sa_ptr[i];
+		}
+	}
+
+	/* terminate last group */
+	if (sa != nosa) {
+		grp[n].cnt = pkts + i - grp[n].m;
+		n++;
+	}
+
+	return n;
+}
+
+/*
+ * helper function, splits processed packets into ipv4/ipv6 traffic.
+ */
+static inline void
+copy_to_trf(struct ipsec_traffic *trf, uint64_t satp, struct rte_mbuf *mb[],
+	uint32_t num)
+{
+	uint32_t j, ofs, s;
+	struct traffic_type *out;
+
+	/*
+	 * determine traffic type(ipv4/ipv6) and offset for ACL classify
+	 * based on SA type
+	 */
+	if ((satp & RTE_IPSEC_SATP_DIR_MASK) == RTE_IPSEC_SATP_DIR_IB) {
+		if ((satp & RTE_IPSEC_SATP_IPV_MASK) == RTE_IPSEC_SATP_IPV4) {
+			out = &trf->ip4;
+			ofs = offsetof(struct ip, ip_p);
+		} else {
+			out = &trf->ip6;
+			ofs = offsetof(struct ip6_hdr, ip6_nxt);
+		}
+	} else if (SATP_OUT_IPV4(satp)) {
+		out = &trf->ip4;
+		ofs = offsetof(struct ip, ip_p);
+	} else {
+		out = &trf->ip6;
+		ofs = offsetof(struct ip6_hdr, ip6_nxt);
+	}
+
+	for (j = 0, s = out->num; j != num; j++) {
+		out->data[s + j] = rte_pktmbuf_mtod_offset(mb[j],
+				void *, ofs);
+		out->pkts[s + j] = mb[j];
+	}
+
+	out->num += num;
+}
+
+/*
+ * Process ipsec packets.
+ * If packet belong to SA that is subject of inline-crypto,
+ * then process it immediately.
+ * Otherwise do necessary preparations and queue it to related
+ * crypto-dev queue.
+ */
+void
+ipsec_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf)
+{
+	uint64_t satp;
+	uint32_t i, j, k, n;
+	struct ipsec_sa *sa;
+	struct ipsec_mbuf_metadata *priv;
+	struct rte_ipsec_group *pg;
+	struct rte_ipsec_session *ips;
+	struct cdev_qp *cqp;
+	struct rte_crypto_op *cop[RTE_DIM(trf->ipsec.pkts)];
+	struct rte_ipsec_group grp[RTE_DIM(trf->ipsec.pkts)];
+
+	n = sa_group(trf->ipsec.saptr, trf->ipsec.pkts, grp, trf->ipsec.num);
+
+	for (i = 0; i != n; i++) {
+
+		pg = grp + i;
+		sa = pg->id.ptr;
+
+		/* no valid SA found */
+		if (sa == NULL)
+			k = 0;
+
+		ips = &sa->ips;
+		satp = rte_ipsec_sa_type(ips->sa);
+
+		/* no valid HW session for that SA, try to create one */
+		if (ips->crypto.ses == NULL &&
+				(create_session(ctx, sa) != 0 ||
+				fill_ipsec_session(ips, sa) != 0))
+			k = 0;
+
+		/* process packets inline */
+		else if (sa->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				sa->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL) {
+
+			/*
+			 * This is just to satisfy inbound_sa_check()
+			 * and get_hop_for_offload_pkt().
+			 * Should be removed in future.
+			 */
+			for (j = 0; j != pg->cnt; j++) {
+				priv = get_priv(pg->m[j]);
+				priv->sa = sa;
+			}
+
+			k = rte_ipsec_pkt_process(ips, pg->m, pg->cnt);
+			copy_to_trf(trf, satp, pg->m, k);
+
+		/* enqueue packets to crypto dev */
+		} else {
+
+			cqp = &ctx->tbl[sa->cdev_id_qp];
+
+			/* for that app each mbuf has it's own crypto op */
+			for (j = 0; j != pg->cnt; j++) {
+				priv = get_priv(pg->m[j]);
+				cop[j] = &priv->cop;
+				/*
+				 * this is just to satisfy inbound_sa_check()
+				 * should be removed in future.
+				 */
+				priv->sa = sa;
+			}
+
+			/* prepare and enqueue crypto ops */
+			k = rte_ipsec_pkt_crypto_prepare(ips, pg->m, cop,
+				pg->cnt);
+			if (k != 0)
+				enqueue_cop_bulk(cqp, cop, k);
+		}
+
+		/* drop packets that cannot be enqueued/processed */
+		if (k != pg->cnt)
+			free_pkts(pg->m + k, pg->cnt - k);
+	}
+}
+
+static inline uint32_t
+cqp_dequeue(struct cdev_qp *cqp, struct rte_crypto_op *cop[], uint32_t num)
+{
+	uint32_t n;
+
+	if (cqp->in_flight == 0)
+		return 0;
+
+	n = rte_cryptodev_dequeue_burst(cqp->id, cqp->qp, cop, num);
+	RTE_ASSERT(cqp->in_flight >= n);
+	cqp->in_flight -= n;
+
+	return n;
+}
+
+static inline uint32_t
+ctx_dequeue(struct ipsec_ctx *ctx, struct rte_crypto_op *cop[], uint32_t num)
+{
+	uint32_t i, n;
+
+	n = 0;
+
+	for (i = ctx->last_qp; n != num && i != ctx->nb_qps; i++)
+		n += cqp_dequeue(ctx->tbl + i, cop + n, num - n);
+
+	for (i = 0; n != num && i != ctx->last_qp; i++)
+		n += cqp_dequeue(ctx->tbl + i, cop + n, num - n);
+
+	ctx->last_qp = i;
+	return n;
+}
+
+/*
+ * dequeue packets from crypto-queues and finalize processing.
+ */
+void
+ipsec_cqp_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf)
+{
+	uint64_t satp;
+	uint32_t i, k, n, ng;
+	struct rte_ipsec_session *ss;
+	struct traffic_type *out;
+	struct rte_ipsec_group *pg;
+	struct rte_crypto_op *cop[RTE_DIM(trf->ipsec.pkts)];
+	struct rte_ipsec_group grp[RTE_DIM(trf->ipsec.pkts)];
+
+	trf->ip4.num = 0;
+	trf->ip6.num = 0;
+
+	out = &trf->ipsec;
+
+	/* dequeue completed crypto-ops */
+	n = ctx_dequeue(ctx, cop, RTE_DIM(cop));
+	if (n == 0)
+		return;
+
+	/* group them by ipsec session */
+	ng = rte_ipsec_pkt_crypto_group((const struct rte_crypto_op **)
+		(uintptr_t)cop, out->pkts, grp, n);
+
+	/* process each group of packets */
+	for (i = 0; i != ng; i++) {
+
+		pg = grp + i;
+		ss = pg->id.ptr;
+		satp = rte_ipsec_sa_type(ss->sa);
+
+		k = rte_ipsec_pkt_process(ss, pg->m, pg->cnt);
+		copy_to_trf(trf, satp, pg->m, k);
+
+		/* free bad packets, if any */
+		free_pkts(pg->m + k, pg->cnt - k);
+
+		n -= pg->cnt;
+	}
+
+	/* we should never have packet with unknown SA here */
+	RTE_VERIFY(n == 0);
+}
diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
index 31f68fee2..81c146ebc 100644
--- a/examples/ipsec-secgw/meson.build
+++ b/examples/ipsec-secgw/meson.build
@@ -9,6 +9,6 @@
 deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
 allow_experimental_apis = true
 sources = files(
-	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
-	'rt.c', 'sa.c', 'sp4.c', 'sp6.c'
+	'esp.c', 'ipsec.c', 'ipsec_process.c', 'ipsec-secgw.c',
+	'parser.c', 'rt.c', 'sa.c', 'sp4.c', 'sp6.c'
 )
-- 
2.17.1

[PATCH v5 09/10] examples/ipsec-secgw: add scripts for functional test

[Konstantin Ananyev]

The purpose of these scripts is to automate ipsec-secgw functional testing.
The scripts require two machines (SUT and DUT) connected through
at least 2 NICs and running linux (so far tested only on Ubuntu 18.04).
Introduced test-cases for the following scenarios:
- Transport/Tunnel modes
- AES-CBC SHA1
- AES-GCM
- ESN on/off
- legacy/librte_ipsec code path

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/test/common_defs.sh      | 153 ++++++++++++++++++
 examples/ipsec-secgw/test/data_rxtx.sh        |  62 +++++++
 examples/ipsec-secgw/test/linux_test4.sh      |  63 ++++++++
 examples/ipsec-secgw/test/linux_test6.sh      |  64 ++++++++
 examples/ipsec-secgw/test/run_test.sh         |  80 +++++++++
 .../test/trs_aescbc_sha1_common_defs.sh       |  69 ++++++++
 .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  67 ++++++++
 .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/trs_aescbc_sha1_esn_defs.sh          |  66 ++++++++
 .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
 .../test/trs_aesgcm_common_defs.sh            |  60 +++++++
 examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  66 ++++++++
 .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  66 ++++++++
 .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
 .../test/tun_aescbc_sha1_common_defs.sh       |  68 ++++++++
 .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  70 ++++++++
 .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/tun_aescbc_sha1_esn_defs.sh          |  70 ++++++++
 .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
 .../test/tun_aesgcm_common_defs.sh            |  60 +++++++
 examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  70 ++++++++
 .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  70 ++++++++
 .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
 25 files changed, 1264 insertions(+)
 create mode 100644 examples/ipsec-secgw/test/common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test4.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test6.sh
 create mode 100644 examples/ipsec-secgw/test/run_test.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh

diff --git a/examples/ipsec-secgw/test/common_defs.sh b/examples/ipsec-secgw/test/common_defs.sh
new file mode 100644
index 000000000..7adfffa19
--- /dev/null
+++ b/examples/ipsec-secgw/test/common_defs.sh
@@ -0,0 +1,153 @@
+#! /bin/bash
+
+#check that env vars are properly defined
+
+#check SGW_PATH
+if [[ -z "${SGW_PATH}" || ! -x ${SGW_PATH} ]]; then
+	echo "SGW_PATH is invalid"
+	exit 127
+fi
+
+#check ETH_DEV
+if [[ -z "${ETH_DEV}" ]]; then
+	echo "ETH_DEV is invalid"
+	exit 127
+fi
+
+#setup SGW_LCORE
+SGW_LCORE=${SGW_LCORE:-0}
+
+#check that REMOTE_HOST is reachable
+ssh ${REMOTE_HOST} echo
+st=$?
+if [[ $st -ne 0 ]]; then
+	echo "host ${REMOTE_HOST} is not reachable"
+	exit $st
+fi
+
+#get ether addr of REMOTE_HOST
+REMOTE_MAC=`ssh ${REMOTE_HOST} ip addr show dev ${REMOTE_IFACE}`
+st=$?
+REMOTE_MAC=`echo ${REMOTE_MAC} | sed -e 's/^.*ether //' -e 's/ brd.*$//'`
+if [[ $st -ne 0 || -z "${REMOTE_MAC}" ]]; then
+	echo "coouldn't retrieve ether addr from ${REMOTE_IFACE}"
+	exit 127
+fi
+
+LOCAL_IFACE=dtap0
+
+LOCAL_MAC="00:64:74:61:70:30"
+
+REMOTE_IPV4=192.168.31.14
+LOCAL_IPV4=192.168.31.92
+
+REMOTE_IPV6=fd12:3456:789a:0031:0000:0000:0000:0014
+LOCAL_IPV6=fd12:3456:789a:0031:0000:0000:0000:0092
+
+DPDK_PATH=${RTE_SDK:-${PWD}}
+DPDK_BUILD=${RTE_TARGET:-x86_64-native-linuxapp-gcc}
+
+SGW_OUT_FILE=./ipsec-secgw.out1
+
+SGW_CMD_EAL_PRM="--lcores=${SGW_LCORE} -n 4 ${ETH_DEV}"
+SGW_CMD_CFG="(0,0,${SGW_LCORE}),(1,0,${SGW_LCORE})"
+SGW_CMD_PRM="-p 0x3 -u 1 -P --config=\"${SGW_CMD_CFG}\""
+
+SGW_CFG_FILE=$(tempfile)
+
+# configure local host/ifaces
+config_local_iface()
+{
+	ifconfig ${LOCAL_IFACE} ${LOCAL_IPV4}/24 mtu 1400 up
+	ifconfig ${LOCAL_IFACE}
+
+	ip neigh flush dev ${LOCAL_IFACE}
+	ip neigh add ${REMOTE_IPV4} dev ${LOCAL_IFACE} lladdr ${REMOTE_MAC}
+	ip neigh show dev ${LOCAL_IFACE}
+}
+
+config6_local_iface()
+{
+	config_local_iface
+
+	sysctl -w net.ipv6.conf.${LOCAL_IFACE}.disable_ipv6=0
+	ip addr add  ${LOCAL_IPV6}/64 dev ${LOCAL_IFACE}
+
+	sysctl -w net.ipv6.conf.${LOCAL_IFACE}.mtu=1300
+
+	ip -6 neigh add ${REMOTE_IPV6} dev ${LOCAL_IFACE} lladdr ${REMOTE_MAC}
+	ip neigh show dev ${LOCAL_IFACE}
+}
+
+#configure remote host/iface
+config_remote_iface()
+{
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE} down
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE} ${REMOTE_IPV4}/24 up
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} ip neigh flush dev ${REMOTE_IFACE}
+
+	# by some reason following ip neigh doesn't work for me here properly:
+	#ssh ${REMOTE_HOST} ip neigh add ${LOCAL_IPV4} \
+	#		dev ${REMOTE_IFACE} lladr ${LOCAL_MAC}
+	# so used arp instead.
+	ssh ${REMOTE_HOST} arp -i ${REMOTE_IFACE} -s ${LOCAL_IPV4} ${LOCAL_MAC}
+	ssh ${REMOTE_HOST} ip neigh show dev ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} iptables --flush
+}
+
+config6_remote_iface()
+{
+	config_remote_iface
+
+	ssh ${REMOTE_HOST} sysctl -w \
+		net.ipv6.conf.${REMOTE_IFACE}.disable_ipv6=0
+	ssh ${REMOTE_HOST} ip addr add  ${REMOTE_IPV6}/64 dev ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} ip -6 neigh add ${LOCAL_IPV6} \
+		dev ${REMOTE_IFACE} lladdr ${LOCAL_MAC}
+	ssh ${REMOTE_HOST} ip neigh show dev ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} ip6tables --flush
+}
+
+#configure remote and local host/iface
+config_iface()
+{
+	config_local_iface
+	config_remote_iface
+}
+
+config6_iface()
+{
+	config6_local_iface
+	config6_remote_iface
+}
+
+#start ipsec-secgw
+secgw_start()
+{
+	SGW_EXEC_FILE=$(tempfile)
+	cat <<EOF > ${SGW_EXEC_FILE}
+${SGW_PATH} ${SGW_CMD_EAL_PRM} ${CRYPTO_DEV} \
+--vdev="net_tap0,mac=fixed" \
+-- ${SGW_CMD_PRM} ${SGW_CMD_XPRM} -f ${SGW_CFG_FILE} > \
+${SGW_OUT_FILE} 2>&1 &
+p=\$!
+echo \$p
+EOF
+
+	cat ${SGW_EXEC_FILE}
+	SGW_PID=`/bin/bash -x ${SGW_EXEC_FILE}`
+	sleep 1
+}
+
+#stop ipsec-secgw and cleanup
+secgw_stop()
+{
+	kill ${SGW_PID}
+	rm -f ${SGW_EXEC_FILE}
+	rm -f ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/data_rxtx.sh b/examples/ipsec-secgw/test/data_rxtx.sh
new file mode 100644
index 000000000..f23a6d594
--- /dev/null
+++ b/examples/ipsec-secgw/test/data_rxtx.sh
@@ -0,0 +1,62 @@
+#! /bin/bash
+
+TCP_PORT=22222
+
+ping_test1()
+{
+	dst=$1
+
+	i=0
+	st=0
+	while [[ $i -ne 1200 && $st -eq 0 ]];
+	do
+		let i++
+		ping -c 1 -s ${i} ${dst}
+		st=$?
+	done
+
+	if [[ $st -ne 0 ]]; then
+		echo "ERROR: $0 failed for dst=${dst}, sz=${i}"
+	fi
+	return $st;
+}
+
+ping6_test1()
+{
+	dst=$1
+
+	i=0
+	st=0
+	while [[ $i -ne 1200 && $st -eq 0 ]];
+	do
+		let i++
+		ping6 -c 1 -s ${i} ${dst}
+		st=$?
+	done
+
+	if [[ $st -ne 0 ]]; then
+		echo "ERROR: $0 failed for dst=${dst}, sz=${i}"
+	fi
+	return $st;
+}
+
+scp_test1()
+{
+	dst=$1
+
+	for sz in 1234 23456 345678 4567890 56789102 ; do
+		x=`basename $0`.${sz}
+		dd if=/dev/urandom of=${x} bs=${sz} count=1
+		scp ${x} [${dst}]:${x}
+		scp [${dst}]:${x} ${x}.copy1
+		diff -u ${x} ${x}.copy1
+		st=$?
+		rm -f ${x} ${x}.copy1
+		ssh ${REMOTE_HOST} rm -f ${x}
+		if [[ $st -ne 0 ]]; then
+			return $st
+		fi
+	done
+
+	return 0;
+}
diff --git a/examples/ipsec-secgw/test/linux_test4.sh b/examples/ipsec-secgw/test/linux_test4.sh
new file mode 100644
index 000000000..d636f5604
--- /dev/null
+++ b/examples/ipsec-secgw/test/linux_test4.sh
@@ -0,0 +1,63 @@
+#! /bin/bash
+
+# usage:  /bin/bash linux_test4.sh <ipsec_mode>
+# for list of available modes please refer to run_test.sh.
+# ipsec-secgw (IPv4 mode) functional test script.
+#
+# Note that for most of them you required appropriate crypto PMD/device
+# to be avaialble.
+# Also user has to setup properly the following environment variables:
+#  SGW_PATH - path to the ipsec-secgw binary to test
+#  REMOTE_HOST - ip/hostname of the DUT
+#  REMOTE_IFACE - iface name for the test-port on DUT
+#  ETH_DEV - ethernet device to be used on SUT by DPDK ('-w <pci-id>')
+# Also user can optonally setup:
+#  SGW_LCORE - lcore to run ipsec-secgw on (default value is 0)
+#  CRYPTO_DEV - crypto device to be used ('-w <pci-id>')
+#  if none specified appropriate vdevs will be created by the scrit
+#
+# The purpose of the script is to automate ipsec-secgw testing
+# using another system running linux as a DUT.
+# It expects that SUT and DUT are connected through at least 2 NICs.
+# One NIC is expected to be managed by linux both machines,
+# and will be used as a control path
+# Make sure user from SUT can ssh to DUT without entering password.
+# Second NIC (test-port) should be reserved for DPDK on SUT,
+# and should be managed by linux on DUT.
+# The script starts ipsec-secgw with 2 NIC devices: test-port and tap vdev.
+# Then configures local tap iface and remote iface and ipsec policies
+# in the following way:
+# traffic going over test-port in both directions has to be
+# protected by ipsec.
+# raffic going over TAP in both directions doesn't have to be protected.
+# I.E:
+# DUT OS(NIC1)--(ipsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS
+# SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(ipsec)-->(NIC1)DUT OS
+# Then tries to perorm some data transfer using the scheme decribed above.
+#
+
+DIR=`dirname $0`
+MODE=$1
+
+ . ${DIR}/common_defs.sh
+ . ${DIR}/${MODE}_defs.sh
+
+config_secgw
+
+secgw_start
+
+config_iface
+
+config_remote_xfrm
+
+ . ${DIR}/data_rxtx.sh
+
+ping_test1 ${REMOTE_IPV4}
+st=$?
+if [[ $st -eq 0 ]]; then
+	scp_test1 ${REMOTE_IPV4}
+	st=$?
+fi
+
+secgw_stop
+exit $st
diff --git a/examples/ipsec-secgw/test/linux_test6.sh b/examples/ipsec-secgw/test/linux_test6.sh
new file mode 100644
index 000000000..e30f607d8
--- /dev/null
+++ b/examples/ipsec-secgw/test/linux_test6.sh
@@ -0,0 +1,64 @@
+#! /bin/bash
+
+# usage:  /bin/bash linux_test6.sh <ipsec_mode>
+# for list of available modes please refer to run_test.sh.
+# ipsec-secgw (IPv6 mode) functional test script.
+#
+# Note that for most of them you required appropriate crypto PMD/device
+# to be avaialble.
+# Also user has to setup properly the following environment variables:
+#  SGW_PATH - path to the ipsec-secgw binary to test
+#  REMOTE_HOST - ip/hostname of the DUT
+#  REMOTE_IFACE - iface name for the test-port on DUT
+#  ETH_DEV - ethernet device to be used on SUT by DPDK ('-w <pci-id>')
+# Also user can optonally setup:
+#  SGW_LCORE - lcore to run ipsec-secgw on (default value is 0)
+#  CRYPTO_DEV - crypto device to be used ('-w <pci-id>')
+#  if none specified appropriate vdevs will be created by the scrit
+#
+# The purpose of the script is to automate ipsec-secgw testing
+# using another system running linux as a DUT.
+# It expects that SUT and DUT are connected through at least 2 NICs.
+# One NIC is expected to be managed by linux both machines,
+# and will be used as a control path.
+# Make sure user from SUT can ssh to DUT without entering password,
+# also make sure that sshd over ipv6 is enabled.
+# Second NIC (test-port) should be reserved for DPDK on SUT,
+# and should be managed by linux on DUT.
+# The script starts ipsec-secgw with 2 NIC devices: test-port and tap vdev.
+# Then configures local tap iface and remote iface and ipsec policies
+# in the following way:
+# traffic going over test-port in both directions has to be
+# protected by ipsec.
+# raffic going over TAP in both directions doesn't have to be protected.
+# I.E:
+# DUT OS(NIC1)--(ipsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS
+# SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(ipsec)-->(NIC1)DUT OS
+# Then tries to perorm some data transfer using the scheme decribed above.
+#
+
+DIR=`dirname $0`
+MODE=$1
+
+ . ${DIR}/common_defs.sh
+ . ${DIR}/${MODE}_defs.sh
+
+config_secgw
+
+secgw_start
+
+config6_iface
+
+config6_remote_xfrm
+
+ . ${DIR}/data_rxtx.sh
+
+ping6_test1 ${REMOTE_IPV6}
+st=$?
+if [[ $st -eq 0 ]]; then
+	scp_test1 ${REMOTE_IPV6}
+	st=$?
+fi
+
+secgw_stop
+exit $st
diff --git a/examples/ipsec-secgw/test/run_test.sh b/examples/ipsec-secgw/test/run_test.sh
new file mode 100644
index 000000000..6dc0ce54e
--- /dev/null
+++ b/examples/ipsec-secgw/test/run_test.sh
@@ -0,0 +1,80 @@
+#! /bin/bash
+
+# usage: /bin/bash run_test.sh [-46]
+# Run all defined linux_test[4,6].sh test-cases one by one
+# user has to setup properly the following environment variables:
+#  SGW_PATH - path to the ipsec-secgw binary to test
+#  REMOTE_HOST - ip/hostname of the DUT
+#  REMOTE_IFACE - iface name for the test-port on DUT
+#  ETH_DEV - ethernet device to be used on SUT by DPDK ('-w <pci-id>')
+# Also user can optonally setup:
+#  SGW_LCORE - lcore to run ipsec-secgw on (default value is 0)
+#  CRYPTO_DEV - crypto device to be used ('-w <pci-id>')
+#  if none specified appropriate vdevs will be created by the scrit
+# refer to linux_test1.sh for more information
+
+# All supported modes to test.
+# naming convention:
+# 'old' means that ipsec-secgw will run in legacy (non-librte_ipsec mode)
+# 'tun/trs' refer to tunnel/transport mode respectively
+LINUX_TEST="tun_aescbc_sha1 \
+tun_aescbc_sha1_esn \
+tun_aescbc_sha1_esn_atom \
+tun_aesgcm \
+tun_aesgcm_esn \
+tun_aesgcm_esn_atom \
+trs_aescbc_sha1 \
+trs_aescbc_sha1_esn \
+trs_aescbc_sha1_esn_atom \
+trs_aesgcm \
+trs_aesgcm_esn \
+trs_aesgcm_esn_atom \
+tun_aescbc_sha1_old \
+tun_aesgcm_old \
+trs_aescbc_sha1_old \
+trs_aesgcm_old"
+
+DIR=`dirname $0`
+
+# get input options
+st=0
+run4=0
+run6=0
+while [[ ${st} -eq 0 ]]; do
+	getopts ":46" opt
+	st=$?
+	if [[ "${opt}" == "4" ]]; then
+		run4=1
+	elif [[ "${opt}" == "6" ]]; then
+		run6=1
+	fi
+done
+
+if [[ ${run4} -eq 0 && {run6} -eq 0 ]]; then
+	exit 127
+fi
+
+for i in ${LINUX_TEST}; do
+
+	echo "starting test ${i}"
+
+	st4=0
+	if [[ ${run4} -ne 0 ]]; then
+		/bin/bash ${DIR}/linux_test4.sh ${i}
+		st4=$?
+		echo "test4 ${i} finished with status ${st4}"
+	fi
+
+	st6=0
+	if [[ ${run6} -ne 0 ]]; then
+		/bin/bash ${DIR}/linux_test6.sh ${i}
+		st6=$?
+		echo "test6 ${i} finished with status ${st6}"
+	fi
+
+	let "st = st4 + st6"
+	if [[ $st -ne 0 ]]; then
+		echo "ERROR test ${i} FAILED"
+		exit $st
+	fi
+done
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
new file mode 100644
index 000000000..e2621e0df
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
@@ -0,0 +1,69 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_mb0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#SP in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#sp in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+sa in 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#SA out rules
+sa out 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#SA out rules
+sa out 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
new file mode 100644
index 000000000..d68552fce
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
@@ -0,0 +1,67 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
new file mode 100644
index 000000000..f16222e11
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
new file mode 100644
index 000000000..ce7c977a3
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
@@ -0,0 +1,66 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
new file mode 100644
index 000000000..a3abb6103
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
new file mode 100644
index 000000000..720e807e4
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
@@ -0,0 +1,60 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_gcm0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#SP in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+sa in 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#SA out rules
+sa out 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+sa out 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_defs.sh
new file mode 100644
index 000000000..8382d3d52
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_defs.sh
@@ -0,0 +1,66 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
new file mode 100644
index 000000000..80d8d63b8
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
new file mode 100644
index 000000000..94958d199
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
@@ -0,0 +1,66 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
new file mode 100644
index 000000000..951e6b68f
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
new file mode 100644
index 000000000..4025da232
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
@@ -0,0 +1,68 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_mb0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#sp in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#sp in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${REMOTE_IPV4} dst ${LOCAL_IPV4}
+
+sa in 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${REMOTE_IPV6} dst ${LOCAL_IPV6}
+
+#SA out rules
+sa out 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${LOCAL_IPV4} dst ${REMOTE_IPV4}
+
+sa out 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${LOCAL_IPV6} dst ${REMOTE_IPV6}
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
new file mode 100644
index 000000000..18aade3a9
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
new file mode 100644
index 000000000..6b4a82149
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
new file mode 100644
index 000000000..28c1125d6
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
new file mode 100644
index 000000000..3c0d8d1b1
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
new file mode 100644
index 000000000..fba68c6a3
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
@@ -0,0 +1,60 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_gcm0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#sp in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#sp in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${REMOTE_IPV4} dst ${LOCAL_IPV4}
+
+sa in 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${REMOTE_IPV6} dst ${LOCAL_IPV6}
+
+#SA out rules
+sa out 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${LOCAL_IPV4} dst ${REMOTE_IPV4}
+
+sa out 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${LOCAL_IPV6} dst ${REMOTE_IPV6}
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_defs.sh
new file mode 100644
index 000000000..8ae65321b
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_defs.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
new file mode 100644
index 000000000..dab1460c8
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
new file mode 100644
index 000000000..606232349
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh
new file mode 100644
index 000000000..e0a015e21
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_defs.sh
+
+SGW_CMD_XPRM=
-- 
2.17.1

[PATCH v5 10/10] doc: update ipsec-secgw guide and relelase notes

[Konstantin Ananyev]

Update ipsec-secgw guide and relelase notes to reflect latest changes.

Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 doc/guides/rel_notes/release_19_02.rst   |  14 +++
 doc/guides/sample_app_ug/ipsec_secgw.rst | 105 ++++++++++++++++++++++-
 2 files changed, 117 insertions(+), 2 deletions(-)

diff --git a/doc/guides/rel_notes/release_19_02.rst b/doc/guides/rel_notes/release_19_02.rst
index 1a9885c44..28dbe3ad0 100644
--- a/doc/guides/rel_notes/release_19_02.rst
+++ b/doc/guides/rel_notes/release_19_02.rst
@@ -116,6 +116,20 @@ New Features
 
   See :doc:`../prog_guide/ipsec_lib` for more information.
 
+* **Updated the ipsec-secgw sample application.**
+
+  The ``ipsec-secgw`` sample application has been updated to use the new
+  ``librte_ipsec`` library also added in this release.
+  The original functionality of ipsec-secgw is retained, a new command line
+  parameter ``-l`` has  been added to ipsec-secgw to use the IPsec library,
+  instead of the existing IPsec code in the application.
+
+  The IPsec library does not support all the functionality of the existing
+  ipsec-secgw application, its is planned to add the outstanding functionality
+  in future releases.
+
+  See :doc:`../sample_app_ug/ipsec_secgw` for more information.
+
 
 Removed Items
 -------------
diff --git a/doc/guides/sample_app_ug/ipsec_secgw.rst b/doc/guides/sample_app_ug/ipsec_secgw.rst
index 61638e733..3d784e705 100644
--- a/doc/guides/sample_app_ug/ipsec_secgw.rst
+++ b/doc/guides/sample_app_ug/ipsec_secgw.rst
@@ -76,7 +76,7 @@ Compiling the Application
 
 To compile the sample application see :doc:`compiling`.
 
-The application is located in the ``rpsec-secgw`` sub-directory.
+The application is located in the ``ipsec-secgw`` sub-directory.
 
 #. [Optional] Build the application for debugging:
    This option adds some extra flags, disables compiler optimizations and
@@ -93,6 +93,7 @@ The application has a number of command line options::
 
    ./build/ipsec-secgw [EAL options] --
                         -p PORTMASK -P -u PORTMASK -j FRAMESIZE
+                        -l -w REPLAY_WINOW_SIZE -e -a
                         --config (port,queue,lcore)[,(port,queue,lcore]
                         --single-sa SAIDX
                         --rxoffload MASK
@@ -114,6 +115,18 @@ Where:
     specified as FRAMESIZE. If an invalid value is provided as FRAMESIZE
     then the default value 9000 is used.
 
+*   ``-l``: enables code-path that uses librte_ipsec.
+
+*   ``-w REPLAY_WINOW_SIZE``: specifies the IPsec sequence number replay window
+    size for each Security Association (available only with librte_ipsec
+    code path).
+
+*   ``-e``: enables Security Association extended sequence number processing
+    (available only with librte_ipsec code path).
+
+*   ``-a``: enables Security Association sequence number atomic behaviour
+    (available only with librte_ipsec code path).
+
 *   ``--config (port,queue,lcore)[,(port,queue,lcore)]``: determines which queues
     from which ports are mapped to which cores.
 
@@ -225,7 +238,7 @@ accordingly.
 
 
 Configuration File Syntax
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~
 
 As mention in the overview, the Security Policies are ACL rules.
 The application parsers the rules specified in the configuration file and
@@ -571,6 +584,11 @@ Example SA rules:
     mode ipv4-tunnel src 172.16.1.5 dst 172.16.2.5 \
     type lookaside-protocol-offload port_id 4
 
+    sa in 35 aead_algo aes-128-gcm \
+    aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+    mode ipv4-tunnel src 172.16.2.5 dst 172.16.1.5 \
+    type inline-crypto-offload port_id 0
+
 Routing rule syntax
 ^^^^^^^^^^^^^^^^^^^
 
@@ -667,3 +685,86 @@ Example Neighbour rules:
 .. code-block:: console
 
     neigh port 0 DE:AD:BE:EF:01:02
+
+Test directory
+--------------
+
+The test directory contains scripts for testing the various encryption
+algorithms.
+
+The purpose of the scripts is to automate ipsec-secgw testing
+using another system running linux as a DUT.
+
+The user must setup the following environment variables:
+
+*   ``SGW_PATH``: path to the ipsec-secgw binary to test.
+
+*   ``REMOTE_HOST``: IP address/hostname of the DUT.
+
+*   ``REMOTE_IFACE``: interface name for the test-port on the DUT.
+
+*   ``ETH_DEV``: ethernet device to be used on the SUT by DPDK ('-w <pci-id>')
+
+Also the user can optionally setup:
+
+*   ``SGW_LCORE``: lcore to run ipsec-secgw on (default value is 0)
+
+*   ``CRYPTO_DEV``: crypto device to be used ('-w <pci-id>'). If none specified
+    appropriate vdevs will be created by the script
+
+Note that most of the tests require the appropriate crypto PMD/device to be
+available.
+
+Server configuration
+~~~~~~~~~~~~~~~~~~~~
+
+Two servers are required for the tests, SUT and DUT.
+
+Make sure the user from the SUT can ssh to the DUT without entering the password.
+To enable this feature keys must be setup on the DUT.
+
+``ssh-keygen`` will make a private & public key pair on the SUT.
+
+``ssh-copy-id`` <user name>@<target host name> on the SUT will copy the public
+key to the DUT. It will ask for credentials so that it can upload the public key.
+
+The SUT and DUT are connected through at least 2 NIC ports.
+
+One NIC port is expected to be managed by linux on both machines and will be
+used as a control path.
+
+The second NIC port (test-port) should be bound to DPDK on the SUT, and should
+be managed by linux on the DUT.
+
+The script starts ``ipsec-secgw`` with 2 NIC devices: ``test-port`` and
+``tap vdev``.
+
+It then configures the local tap interface and the remote interface and IPsec
+policies in the following way:
+
+Traffic going over the test-port in both directions has to be protected by IPsec.
+
+Traffic going over the TAP port in both directions does not have to be protected.
+
+i.e:
+
+DUT OS(NIC1)--(IPsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS
+
+SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(IPsec)-->(NIC1)DUT OS
+
+It then tries to perform some data transfer using the scheme decribed above.
+
+usage
+~~~~~
+
+In the ipsec-secgw/test directory
+
+to run one test for IPv4 or IPv6
+
+/bin/bash linux_test(4|6).sh <ipsec_mode>
+
+to run all tests for IPv4 or IPv6
+
+/bin/bash run_test.sh -4|-6
+
+For the list of available modes please refer to run_test.sh.
-- 
2.17.1

[PATCH v4 2/9] examples/ipsec-secgw: allow to specify neighbor mac address

[Konstantin Ananyev]

In some cases it is useful to allow user to specify destination
ether address for outgoing packets.
This patch adds such ability by introducing new 'neigh' config
file option.

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 21 +++++++--
 examples/ipsec-secgw/ipsec.h       |  3 ++
 examples/ipsec-secgw/parser.c      | 75 ++++++++++++++++++++++++++++++
 examples/ipsec-secgw/parser.h      |  8 ++--
 4 files changed, 99 insertions(+), 8 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index cfc2b05e5..2ed757922 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -104,9 +104,9 @@ static uint16_t nb_txd = IPSEC_SECGW_TX_DESC_DEFAULT;
 #define ETHADDR(a, b, c, d, e, f) (__BYTES_TO_UINT64(a, b, c, d, e, f, 0, 0))
 
 #define ETHADDR_TO_UINT64(addr) __BYTES_TO_UINT64( \
-		addr.addr_bytes[0], addr.addr_bytes[1], \
-		addr.addr_bytes[2], addr.addr_bytes[3], \
-		addr.addr_bytes[4], addr.addr_bytes[5], \
+		(addr)->addr_bytes[0], (addr)->addr_bytes[1], \
+		(addr)->addr_bytes[2], (addr)->addr_bytes[3], \
+		(addr)->addr_bytes[4], (addr)->addr_bytes[5], \
 		0, 0)
 
 /* port/source ethernet addr and destination ethernet addr */
@@ -1188,6 +1188,19 @@ print_ethaddr(const char *name, const struct ether_addr *eth_addr)
 	printf("%s%s", name, buf);
 }
 
+/*
+ * Update destination ethaddr for the port.
+ */
+int
+add_dst_ethaddr(uint16_t port, const struct ether_addr *addr)
+{
+	if (port > RTE_DIM(ethaddr_tbl))
+		return -EINVAL;
+
+	ethaddr_tbl[port].dst = ETHADDR_TO_UINT64(addr);
+	return 0;
+}
+
 /* Check the link status of all ports in up to 9s, and print them finally */
 static void
 check_all_ports_link_status(uint32_t port_mask)
@@ -1564,7 +1577,7 @@ port_init(uint16_t portid, uint64_t req_rx_offloads, uint64_t req_tx_offloads)
 	printf("Configuring device port %u:\n", portid);
 
 	rte_eth_macaddr_get(portid, &ethaddr);
-	ethaddr_tbl[portid].src = ETHADDR_TO_UINT64(ethaddr);
+	ethaddr_tbl[portid].src = ETHADDR_TO_UINT64(&ethaddr);
 	print_ethaddr("Address: ", &ethaddr);
 	printf("\n");
 
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 9b1586f52..580f7876b 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -245,4 +245,7 @@ int
 sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
 		uint64_t *tx_offloads);
 
+int
+add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/parser.c b/examples/ipsec-secgw/parser.c
index 91282ca94..6769a8e6d 100644
--- a/examples/ipsec-secgw/parser.c
+++ b/examples/ipsec-secgw/parser.c
@@ -306,6 +306,30 @@ parse_range(const char *token, uint16_t *low, uint16_t *high)
 	return 0;
 }
 
+#define PARSE_UINT8x16(s, v, l)	                          \
+do {                                                      \
+	char *end;                                        \
+	unsigned long t;                                  \
+	errno = 0;                                        \
+	t = strtoul((s), &end, 16);                       \
+	if (errno != 0 || end[0] != (l) || t > UINT8_MAX) \
+		return -EINVAL;                           \
+	(s) = end + 1;                                    \
+	(v) = t;                                          \
+} while (0)
+
+static int
+parse_mac(const char *str, struct ether_addr *addr)
+{
+	PARSE_UINT8x16(str, addr->addr_bytes[0], ':');
+	PARSE_UINT8x16(str, addr->addr_bytes[1], ':');
+	PARSE_UINT8x16(str, addr->addr_bytes[2], ':');
+	PARSE_UINT8x16(str, addr->addr_bytes[3], ':');
+	PARSE_UINT8x16(str, addr->addr_bytes[4], ':');
+	PARSE_UINT8x16(str, addr->addr_bytes[5], 0);
+	return 0;
+}
+
 /** sp add parse */
 struct cfg_sp_add_cfg_item {
 	cmdline_fixed_string_t sp_keyword;
@@ -444,11 +468,61 @@ cmdline_parse_inst_t cfg_rt_add_rule = {
 	},
 };
 
+/* neigh add parse */
+struct cfg_neigh_add_item {
+	cmdline_fixed_string_t neigh;
+	cmdline_fixed_string_t pstr;
+	uint16_t port;
+	cmdline_fixed_string_t mac;
+};
+
+static void
+cfg_parse_neigh(void *parsed_result, __rte_unused struct cmdline *cl,
+	void *data)
+{
+	int32_t rc;
+	struct cfg_neigh_add_item *res;
+	struct parse_status *st;
+	struct ether_addr mac;
+
+	st = data;
+	res = parsed_result;
+	rc = parse_mac(res->mac, &mac);
+	APP_CHECK(rc == 0, st, "invalid ether addr:%s", res->mac);
+	rc = add_dst_ethaddr(res->port, &mac);
+	APP_CHECK(rc == 0, st, "invalid port numer:%hu", res->port);
+	if (st->status < 0)
+		return;
+}
+
+cmdline_parse_token_string_t cfg_add_neigh_start =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, neigh, "neigh");
+cmdline_parse_token_string_t cfg_add_neigh_pstr =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, pstr, "port");
+cmdline_parse_token_num_t cfg_add_neigh_port =
+	TOKEN_NUM_INITIALIZER(struct cfg_neigh_add_item, port, UINT16);
+cmdline_parse_token_string_t cfg_add_neigh_mac =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, mac, NULL);
+
+cmdline_parse_inst_t cfg_neigh_add_rule = {
+	.f = cfg_parse_neigh,
+	.data = NULL,
+	.help_str = "",
+	.tokens = {
+		(void *)&cfg_add_neigh_start,
+		(void *)&cfg_add_neigh_pstr,
+		(void *)&cfg_add_neigh_port,
+		(void *)&cfg_add_neigh_mac,
+		NULL,
+	},
+};
+
 /** set of cfg items */
 cmdline_parse_ctx_t ipsec_ctx[] = {
 	(cmdline_parse_inst_t *)&cfg_sp_add_rule,
 	(cmdline_parse_inst_t *)&cfg_sa_add_rule,
 	(cmdline_parse_inst_t *)&cfg_rt_add_rule,
+	(cmdline_parse_inst_t *)&cfg_neigh_add_rule,
 	NULL,
 };
 
@@ -474,6 +548,7 @@ parse_cfg_file(const char *cfg_filename)
 	cfg_sp_add_rule.data = &status;
 	cfg_sa_add_rule.data = &status;
 	cfg_rt_add_rule.data = &status;
+	cfg_neigh_add_rule.data = &status;
 
 	do {
 		char oneline[1024];
diff --git a/examples/ipsec-secgw/parser.h b/examples/ipsec-secgw/parser.h
index be02537c5..6b8a10076 100644
--- a/examples/ipsec-secgw/parser.h
+++ b/examples/ipsec-secgw/parser.h
@@ -14,14 +14,14 @@ struct parse_status {
 	char parse_msg[256];
 };
 
-#define	APP_CHECK(exp, status, fmt, ...)				\
+#define	APP_CHECK(exp, st, fmt, ...)					\
 do {									\
 	if (!(exp)) {							\
-		sprintf(status->parse_msg, fmt "\n",			\
+		sprintf((st)->parse_msg, fmt "\n",			\
 			## __VA_ARGS__);				\
-		status->status = -1;					\
+		(st)->status = -1;					\
 	} else								\
-		status->status = 0;					\
+		(st)->status = 0;					\
 } while (0)
 
 #define APP_CHECK_PRESENCE(val, str, status)				\
-- 
2.17.1

Re: [PATCH v4 2/9] examples/ipsec-secgw: allow to specify neighbor mac address

[Akhil Goyal]

Hi,

On 12/14/2018 10:10 PM, Konstantin Ananyev wrote:
> In some cases it is useful to allow user to specify destination
> ether address for outgoing packets.
> This patch adds such ability by introducing new 'neigh' config
> file option.
I do not see an example config file option about how to use this option
and no update is done to documentation for this newly added option.
> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> Acked-by: Radu Nicolau <radu.nicolau@intel.com>
>

[PATCH v4 3/9] examples/ipsec-secgw: fix crypto-op might never get dequeued

[Konstantin Ananyev]

In some cases crypto-ops could never be dequeued from the crypto-device.
The easiest way to reproduce:
start ipsec-secgw with crypto-dev and send to it less then 32 packets.
none packets will be forwarded.
Reason for that is that the application does dequeue() from crypto-queues
only when new packets arrive.
This patch makes sure it calls dequeue() on a regular basis.

Fixes: c64278c0c18b ("examples/ipsec-secgw: rework processing loop")

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 136 ++++++++++++++++++++++++-----
 examples/ipsec-secgw/ipsec.c       |  60 ++++++++-----
 examples/ipsec-secgw/ipsec.h       |  11 +++
 3 files changed, 165 insertions(+), 42 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 2ed757922..62443172a 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -456,38 +456,55 @@ inbound_sp_sa(struct sp_ctx *sp, struct sa_ctx *sa, struct traffic_type *ip,
 	ip->num = j;
 }
 
-static inline void
-process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
-		struct ipsec_traffic *traffic)
+static void
+split46_traffic(struct ipsec_traffic *trf, struct rte_mbuf *mb[], uint32_t num)
 {
+	uint32_t i, n4, n6;
+	struct ip *ip;
 	struct rte_mbuf *m;
-	uint16_t idx, nb_pkts_in, i, n_ip4, n_ip6;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
+	n4 = trf->ip4.num;
+	n6 = trf->ip6.num;
 
-	n_ip4 = traffic->ip4.num;
-	n_ip6 = traffic->ip6.num;
+	for (i = 0; i < num; i++) {
+
+		m = mb[i];
+		ip = rte_pktmbuf_mtod(m, struct ip *);
 
-	/* SP/ACL Inbound check ipsec and ip4 */
-	for (i = 0; i < nb_pkts_in; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
 		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-			traffic->ip4.data[idx] = rte_pktmbuf_mtod_offset(m,
+			trf->ip4.pkts[n4] = m;
+			trf->ip4.data[n4] = rte_pktmbuf_mtod_offset(m,
 					uint8_t *, offsetof(struct ip, ip_p));
+			n4++;
 		} else if (ip->ip_v == IP6_VERSION) {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
-			traffic->ip6.data[idx] = rte_pktmbuf_mtod_offset(m,
+			trf->ip6.pkts[n6] = m;
+			trf->ip6.data[n6] = rte_pktmbuf_mtod_offset(m,
 					uint8_t *,
 					offsetof(struct ip6_hdr, ip6_nxt));
+			n6++;
 		} else
 			rte_pktmbuf_free(m);
 	}
 
+	trf->ip4.num = n4;
+	trf->ip6.num = n6;
+}
+
+
+static inline void
+process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
+		struct ipsec_traffic *traffic)
+{
+	uint16_t nb_pkts_in, n_ip4, n_ip6;
+
+	n_ip4 = traffic->ip4.num;
+	n_ip6 = traffic->ip6.num;
+
+	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.num, MAX_PKT_BURST);
+
+	split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+
 	inbound_sp_sa(ipsec_ctx->sp4_ctx, ipsec_ctx->sa_ctx, &traffic->ip4,
 			n_ip4);
 
@@ -782,7 +799,7 @@ process_pkts(struct lcore_conf *qconf, struct rte_mbuf **pkts,
 }
 
 static inline void
-drain_buffers(struct lcore_conf *qconf)
+drain_tx_buffers(struct lcore_conf *qconf)
 {
 	struct buffer *buf;
 	uint32_t portid;
@@ -796,6 +813,81 @@ drain_buffers(struct lcore_conf *qconf)
 	}
 }
 
+static inline void
+drain_crypto_buffers(struct lcore_conf *qconf)
+{
+	uint32_t i;
+	struct ipsec_ctx *ctx;
+
+	/* drain inbound buffers*/
+	ctx = &qconf->inbound;
+	for (i = 0; i != ctx->nb_qps; i++) {
+		if (ctx->tbl[i].len != 0)
+			enqueue_cop_burst(ctx->tbl  + i);
+	}
+
+	/* drain outbound buffers*/
+	ctx = &qconf->outbound;
+	for (i = 0; i != ctx->nb_qps; i++) {
+		if (ctx->tbl[i].len != 0)
+			enqueue_cop_burst(ctx->tbl  + i);
+	}
+}
+
+static void
+drain_inbound_crypto_queues(const struct lcore_conf *qconf,
+		struct ipsec_ctx *ctx)
+{
+	uint32_t n;
+	struct ipsec_traffic trf;
+
+	/* dequeue packets from crypto-queue */
+	n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+			RTE_DIM(trf.ipsec.pkts));
+	if (n == 0)
+		return;
+
+	trf.ip4.num = 0;
+	trf.ip6.num = 0;
+
+	/* split traffic by ipv4-ipv6 */
+	split46_traffic(&trf, trf.ipsec.pkts, n);
+
+	/* process ipv4 packets */
+	inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
+	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+
+	/* process ipv6 packets */
+	inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
+	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+}
+
+static void
+drain_outbound_crypto_queues(const struct lcore_conf *qconf,
+		struct ipsec_ctx *ctx)
+{
+	uint32_t n;
+	struct ipsec_traffic trf;
+
+	/* dequeue packets from crypto-queue */
+	n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+			RTE_DIM(trf.ipsec.pkts));
+	if (n == 0)
+		return;
+
+	trf.ip4.num = 0;
+	trf.ip6.num = 0;
+
+	/* split traffic by ipv4-ipv6 */
+	split46_traffic(&trf, trf.ipsec.pkts, n);
+
+	/* process ipv4 packets */
+	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+
+	/* process ipv6 packets */
+	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+}
+
 /* main processing loop */
 static int32_t
 main_loop(__attribute__((unused)) void *dummy)
@@ -853,7 +945,8 @@ main_loop(__attribute__((unused)) void *dummy)
 		diff_tsc = cur_tsc - prev_tsc;
 
 		if (unlikely(diff_tsc > drain_tsc)) {
-			drain_buffers(qconf);
+			drain_tx_buffers(qconf);
+			drain_crypto_buffers(qconf);
 			prev_tsc = cur_tsc;
 		}
 
@@ -867,6 +960,9 @@ main_loop(__attribute__((unused)) void *dummy)
 			if (nb_rx > 0)
 				process_pkts(qconf, pkts, nb_rx, portid);
 		}
+
+		drain_inbound_crypto_queues(qconf, &qconf->inbound);
+		drain_outbound_crypto_queues(qconf, &qconf->outbound);
 	}
 }
 
diff --git a/examples/ipsec-secgw/ipsec.c b/examples/ipsec-secgw/ipsec.c
index 3d415f1af..8bf928a23 100644
--- a/examples/ipsec-secgw/ipsec.c
+++ b/examples/ipsec-secgw/ipsec.c
@@ -333,33 +333,35 @@ create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa)
 	return 0;
 }
 
+/*
+ * queue crypto-ops into PMD queue.
+ */
+void
+enqueue_cop_burst(struct cdev_qp *cqp)
+{
+	uint32_t i, len, ret;
+
+	len = cqp->len;
+	ret = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp, cqp->buf, len);
+	if (ret < len) {
+		RTE_LOG_DP(DEBUG, IPSEC, "Cryptodev %u queue %u:"
+			" enqueued %u crypto ops out of %u\n",
+			cqp->id, cqp->qp, ret, len);
+			/* drop packets that we fail to enqueue */
+			for (i = ret; i < len; i++)
+				rte_pktmbuf_free(cqp->buf[i]->sym->m_src);
+	}
+	cqp->in_flight += ret;
+	cqp->len = 0;
+}
+
 static inline void
 enqueue_cop(struct cdev_qp *cqp, struct rte_crypto_op *cop)
 {
-	int32_t ret = 0, i;
-
 	cqp->buf[cqp->len++] = cop;
 
-	if (cqp->len == MAX_PKT_BURST) {
-		int enq_size = cqp->len;
-		if ((cqp->in_flight + enq_size) > MAX_INFLIGHT)
-			enq_size -=
-			    (int)((cqp->in_flight + enq_size) - MAX_INFLIGHT);
-
-		if (enq_size > 0)
-			ret = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp,
-					cqp->buf, enq_size);
-		if (ret < cqp->len) {
-			RTE_LOG_DP(DEBUG, IPSEC, "Cryptodev %u queue %u:"
-					" enqueued %u crypto ops out of %u\n",
-					 cqp->id, cqp->qp,
-					 ret, cqp->len);
-			for (i = ret; i < cqp->len; i++)
-				rte_pktmbuf_free(cqp->buf[i]->sym->m_src);
-		}
-		cqp->in_flight += ret;
-		cqp->len = 0;
-	}
+	if (cqp->len == MAX_PKT_BURST)
+		enqueue_cop_burst(cqp);
 }
 
 static inline void
@@ -548,6 +550,13 @@ ipsec_inbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 	return ipsec_dequeue(esp_inbound_post, ctx, pkts, len);
 }
 
+uint16_t
+ipsec_inbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len)
+{
+	return ipsec_dequeue(esp_inbound_post, ctx, pkts, len);
+}
+
 uint16_t
 ipsec_outbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint32_t sa_idx[], uint16_t nb_pkts, uint16_t len)
@@ -560,3 +569,10 @@ ipsec_outbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 
 	return ipsec_dequeue(esp_outbound_post, ctx, pkts, len);
 }
+
+uint16_t
+ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len)
+{
+	return ipsec_dequeue(esp_outbound_post, ctx, pkts, len);
+}
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 580f7876b..2f04b7d68 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -184,6 +184,14 @@ uint16_t
 ipsec_outbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint32_t sa_idx[], uint16_t nb_pkts, uint16_t len);
 
+uint16_t
+ipsec_inbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len);
+
+uint16_t
+ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len);
+
 static inline uint16_t
 ipsec_metadata_size(void)
 {
@@ -248,4 +256,7 @@ sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
 int
 add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
 
+void
+enqueue_cop_burst(struct cdev_qp *cqp);
+
 #endif /* __IPSEC_H__ */
-- 
2.17.1

Re: [PATCH v4 3/9] examples/ipsec-secgw: fix crypto-op might never get dequeued

[Akhil Goyal]

On 12/14/2018 10:10 PM, Konstantin Ananyev wrote:
> In some cases crypto-ops could never be dequeued from the crypto-device.
> The easiest way to reproduce:
> start ipsec-secgw with crypto-dev and send to it less then 32 packets.
> none packets will be forwarded.
> Reason for that is that the application does dequeue() from crypto-queues
> only when new packets arrive.
> This patch makes sure it calls dequeue() on a regular basis.
>
> Fixes: c64278c0c18b ("examples/ipsec-secgw: rework processing loop")
Thanks for looking into this age long issue of ipsec-secgw. But wouldn't 
this cause packet reordering,
and the packets which are somehow left in the queue will get delayed and 
would be dropped subsequently due to anti-replay late?
> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> Acked-by: Radu Nicolau <radu.nicolau@intel.com>
> ---
>

Re: [PATCH v4 3/9] examples/ipsec-secgw: fix crypto-op might never get dequeued

[Ananyev, Konstantin]

> -----Original Message-----
> From: Akhil Goyal [mailto:akhil.goyal@nxp.com]
> Sent: Friday, December 21, 2018 2:13 PM
> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
> Cc: Nicolau, Radu <radu.nicolau@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v4 3/9] examples/ipsec-secgw: fix crypto-op might never get dequeued
> 
> 
> 
> On 12/14/2018 10:10 PM, Konstantin Ananyev wrote:
> > In some cases crypto-ops could never be dequeued from the crypto-device.
> > The easiest way to reproduce:
> > start ipsec-secgw with crypto-dev and send to it less then 32 packets.
> > none packets will be forwarded.
> > Reason for that is that the application does dequeue() from crypto-queues
> > only when new packets arrive.
> > This patch makes sure it calls dequeue() on a regular basis.
> >
> > Fixes: c64278c0c18b ("examples/ipsec-secgw: rework processing loop")
> Thanks for looking into this age long issue of ipsec-secgw. But wouldn't
> this cause packet reordering,
> and the packets which are somehow left in the queue will get delayed and
> would be dropped subsequently due to anti-replay late?

Could you explain a bit more - how do you think reordering might happen?
Now we always processing packets belonging to particular SA on the same
crypto-dev.
Konstantin

Re: [PATCH v4 3/9] examples/ipsec-secgw: fix crypto-op might never get dequeued

[Akhil Goyal]

On 12/21/2018 8:19 PM, Ananyev, Konstantin wrote:
>
>> -----Original Message-----
>> From: Akhil Goyal [mailto:akhil.goyal@nxp.com]
>> Sent: Friday, December 21, 2018 2:13 PM
>> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
>> Cc: Nicolau, Radu <radu.nicolau@intel.com>
>> Subject: Re: [dpdk-dev] [PATCH v4 3/9] examples/ipsec-secgw: fix crypto-op might never get dequeued
>>
>>
>>
>> On 12/14/2018 10:10 PM, Konstantin Ananyev wrote:
>>> In some cases crypto-ops could never be dequeued from the crypto-device.
>>> The easiest way to reproduce:
>>> start ipsec-secgw with crypto-dev and send to it less then 32 packets.
>>> none packets will be forwarded.
>>> Reason for that is that the application does dequeue() from crypto-queues
>>> only when new packets arrive.
>>> This patch makes sure it calls dequeue() on a regular basis.
>>>
>>> Fixes: c64278c0c18b ("examples/ipsec-secgw: rework processing loop")
>> Thanks for looking into this age long issue of ipsec-secgw. But wouldn't
>> this cause packet reordering,
>> and the packets which are somehow left in the queue will get delayed and
>> would be dropped subsequently due to anti-replay late?
> Could you explain a bit more - how do you think reordering might happen?

I thought any core can pick the remainder of the packets and can give to 
any of the cryptodevs.
If that is assured, then probably we wont face such issues.
> Now we always processing packets belonging to particular SA on the same
> crypto-dev.
> Konstantin

Re: [PATCH v4 3/9] examples/ipsec-secgw: fix crypto-op might never get dequeued

[Ananyev, Konstantin]

> -----Original Message-----
> From: Akhil Goyal [mailto:akhil.goyal@nxp.com]
> Sent: Friday, December 21, 2018 2:57 PM
> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
> Cc: Nicolau, Radu <radu.nicolau@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v4 3/9] examples/ipsec-secgw: fix crypto-op might never get dequeued
> 
> 
> 
> On 12/21/2018 8:19 PM, Ananyev, Konstantin wrote:
> >
> >> -----Original Message-----
> >> From: Akhil Goyal [mailto:akhil.goyal@nxp.com]
> >> Sent: Friday, December 21, 2018 2:13 PM
> >> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
> >> Cc: Nicolau, Radu <radu.nicolau@intel.com>
> >> Subject: Re: [dpdk-dev] [PATCH v4 3/9] examples/ipsec-secgw: fix crypto-op might never get dequeued
> >>
> >>
> >>
> >> On 12/14/2018 10:10 PM, Konstantin Ananyev wrote:
> >>> In some cases crypto-ops could never be dequeued from the crypto-device.
> >>> The easiest way to reproduce:
> >>> start ipsec-secgw with crypto-dev and send to it less then 32 packets.
> >>> none packets will be forwarded.
> >>> Reason for that is that the application does dequeue() from crypto-queues
> >>> only when new packets arrive.
> >>> This patch makes sure it calls dequeue() on a regular basis.
> >>>
> >>> Fixes: c64278c0c18b ("examples/ipsec-secgw: rework processing loop")
> >> Thanks for looking into this age long issue of ipsec-secgw. But wouldn't
> >> this cause packet reordering,
> >> and the packets which are somehow left in the queue will get delayed and
> >> would be dropped subsequently due to anti-replay late?
> > Could you explain a bit more - how do you think reordering might happen?
> 
> I thought any core can pick the remainder of the packets and can give to
> any of the cryptodevs.

No it is not possible with current app design.
eqnueue/dequeue from particular crypto-queue is always done by the same core.

> If that is assured, then probably we wont face such issues.
> > Now we always processing packets belonging to particular SA on the same
> > crypto-dev.
> > Konstantin

[PATCH v4 4/9] examples/ipsec-secgw: fix outbound codepath for single SA

[Konstantin Ananyev]

Looking at process_pkts_outbound_nosp() there seems few issues:
- accessing mbuf after it was freed
- invoking ipsec_outbound() for ipv4 packets only
- copying number of packets, but not the mbuf pointers itself

that patch provides fixes for that issues.

Fixes: 906257e965b7 ("examples/ipsec-secgw: support IPv6")

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 32 ++++++++++++++++++++----------
 1 file changed, 22 insertions(+), 10 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 62443172a..d1da2d5ce 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -616,32 +616,44 @@ process_pkts_outbound_nosp(struct ipsec_ctx *ipsec_ctx,
 		struct ipsec_traffic *traffic)
 {
 	struct rte_mbuf *m;
-	uint32_t nb_pkts_out, i;
+	uint32_t nb_pkts_out, i, n;
 	struct ip *ip;
 
 	/* Drop any IPsec traffic from protected ports */
 	for (i = 0; i < traffic->ipsec.num; i++)
 		rte_pktmbuf_free(traffic->ipsec.pkts[i]);
 
-	traffic->ipsec.num = 0;
+	n = 0;
 
-	for (i = 0; i < traffic->ip4.num; i++)
-		traffic->ip4.res[i] = single_sa_idx;
+	for (i = 0; i < traffic->ip4.num; i++) {
+		traffic->ipsec.pkts[n] = traffic->ip4.pkts[i];
+		traffic->ipsec.res[n++] = single_sa_idx;
+	}
 
-	for (i = 0; i < traffic->ip6.num; i++)
-		traffic->ip6.res[i] = single_sa_idx;
+	for (i = 0; i < traffic->ip6.num; i++) {
+		traffic->ipsec.pkts[n] = traffic->ip6.pkts[i];
+		traffic->ipsec.res[n++] = single_sa_idx;
+	}
+
+	traffic->ip4.num = 0;
+	traffic->ip6.num = 0;
+	traffic->ipsec.num = n;
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ip4.pkts,
-			traffic->ip4.res, traffic->ip4.num,
+	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.res, traffic->ipsec.num,
 			MAX_PKT_BURST);
 
 	/* They all sue the same SA (ip4 or ip6 tunnel) */
 	m = traffic->ipsec.pkts[i];
 	ip = rte_pktmbuf_mtod(m, struct ip *);
-	if (ip->ip_v == IPVERSION)
+	if (ip->ip_v == IPVERSION) {
 		traffic->ip4.num = nb_pkts_out;
-	else
+		for (i = 0; i < nb_pkts_out; i++)
+			traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+	} else {
 		traffic->ip6.num = nb_pkts_out;
+		traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+	}
 }
 
 static inline int32_t
-- 
2.17.1

Re: [PATCH v4 4/9] examples/ipsec-secgw: fix outbound codepath for single SA

[Akhil Goyal]

On 12/14/2018 10:10 PM, Konstantin Ananyev wrote:
> Looking at process_pkts_outbound_nosp() there seems few issues:
> - accessing mbuf after it was freed
> - invoking ipsec_outbound() for ipv4 packets only
> - copying number of packets, but not the mbuf pointers itself
>
> that patch provides fixes for that issues.
>
> Fixes: 906257e965b7 ("examples/ipsec-secgw: support IPv6")
>
> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> Acked-by: Radu Nicolau <radu.nicolau@intel.com>
> ---
>   examples/ipsec-secgw/ipsec-secgw.c | 32 ++++++++++++++++++++----------
>   1 file changed, 22 insertions(+), 10 deletions(-)
>
> diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
> index 62443172a..d1da2d5ce 100644
> --- a/examples/ipsec-secgw/ipsec-secgw.c
> +++ b/examples/ipsec-secgw/ipsec-secgw.c
> @@ -616,32 +616,44 @@ process_pkts_outbound_nosp(struct ipsec_ctx *ipsec_ctx,
>   		struct ipsec_traffic *traffic)
>   {
>   	struct rte_mbuf *m;
> -	uint32_t nb_pkts_out, i;
> +	uint32_t nb_pkts_out, i, n;
>   	struct ip *ip;
>   
>   	/* Drop any IPsec traffic from protected ports */
>   	for (i = 0; i < traffic->ipsec.num; i++)
>   		rte_pktmbuf_free(traffic->ipsec.pkts[i]);
>   
> -	traffic->ipsec.num = 0;
> +	n = 0;
>   
> -	for (i = 0; i < traffic->ip4.num; i++)
> -		traffic->ip4.res[i] = single_sa_idx;
> +	for (i = 0; i < traffic->ip4.num; i++) {
> +		traffic->ipsec.pkts[n] = traffic->ip4.pkts[i];
> +		traffic->ipsec.res[n++] = single_sa_idx;
> +	}
>   
> -	for (i = 0; i < traffic->ip6.num; i++)
> -		traffic->ip6.res[i] = single_sa_idx;
> +	for (i = 0; i < traffic->ip6.num; i++) {
> +		traffic->ipsec.pkts[n] = traffic->ip6.pkts[i];
> +		traffic->ipsec.res[n++] = single_sa_idx;
> +	}
> +
> +	traffic->ip4.num = 0;
> +	traffic->ip6.num = 0;
> +	traffic->ipsec.num = n;
>   
> -	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ip4.pkts,
> -			traffic->ip4.res, traffic->ip4.num,
> +	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
> +			traffic->ipsec.res, traffic->ipsec.num,
>   			MAX_PKT_BURST);
>   
>   	/* They all sue the same SA (ip4 or ip6 tunnel) */
>   	m = traffic->ipsec.pkts[i];
>   	ip = rte_pktmbuf_mtod(m, struct ip *);
> -	if (ip->ip_v == IPVERSION)
> +	if (ip->ip_v == IPVERSION) {
>   		traffic->ip4.num = nb_pkts_out;
> -	else
> +		for (i = 0; i < nb_pkts_out; i++)
> +			traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
> +	} else {
>   		traffic->ip6.num = nb_pkts_out;
> +		traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
you don't need a for loop here??
> +	}
>   }
>   
>   static inline int32_t

Re: [PATCH v4 4/9] examples/ipsec-secgw: fix outbound codepath for single SA

[Ananyev, Konstantin]

> -----Original Message-----
> From: Akhil Goyal [mailto:akhil.goyal@nxp.com]
> Sent: Friday, December 21, 2018 2:26 PM
> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
> Cc: Nicolau, Radu <radu.nicolau@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v4 4/9] examples/ipsec-secgw: fix outbound codepath for single SA
> 
> 
> 
> On 12/14/2018 10:10 PM, Konstantin Ananyev wrote:
> > Looking at process_pkts_outbound_nosp() there seems few issues:
> > - accessing mbuf after it was freed
> > - invoking ipsec_outbound() for ipv4 packets only
> > - copying number of packets, but not the mbuf pointers itself
> >
> > that patch provides fixes for that issues.
> >
> > Fixes: 906257e965b7 ("examples/ipsec-secgw: support IPv6")
> >
> > Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> > Acked-by: Radu Nicolau <radu.nicolau@intel.com>
> > ---
> >   examples/ipsec-secgw/ipsec-secgw.c | 32 ++++++++++++++++++++----------
> >   1 file changed, 22 insertions(+), 10 deletions(-)
> >
> > diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
> > index 62443172a..d1da2d5ce 100644
> > --- a/examples/ipsec-secgw/ipsec-secgw.c
> > +++ b/examples/ipsec-secgw/ipsec-secgw.c
> > @@ -616,32 +616,44 @@ process_pkts_outbound_nosp(struct ipsec_ctx *ipsec_ctx,
> >   		struct ipsec_traffic *traffic)
> >   {
> >   	struct rte_mbuf *m;
> > -	uint32_t nb_pkts_out, i;
> > +	uint32_t nb_pkts_out, i, n;
> >   	struct ip *ip;
> >
> >   	/* Drop any IPsec traffic from protected ports */
> >   	for (i = 0; i < traffic->ipsec.num; i++)
> >   		rte_pktmbuf_free(traffic->ipsec.pkts[i]);
> >
> > -	traffic->ipsec.num = 0;
> > +	n = 0;
> >
> > -	for (i = 0; i < traffic->ip4.num; i++)
> > -		traffic->ip4.res[i] = single_sa_idx;
> > +	for (i = 0; i < traffic->ip4.num; i++) {
> > +		traffic->ipsec.pkts[n] = traffic->ip4.pkts[i];
> > +		traffic->ipsec.res[n++] = single_sa_idx;
> > +	}
> >
> > -	for (i = 0; i < traffic->ip6.num; i++)
> > -		traffic->ip6.res[i] = single_sa_idx;
> > +	for (i = 0; i < traffic->ip6.num; i++) {
> > +		traffic->ipsec.pkts[n] = traffic->ip6.pkts[i];
> > +		traffic->ipsec.res[n++] = single_sa_idx;
> > +	}
> > +
> > +	traffic->ip4.num = 0;
> > +	traffic->ip6.num = 0;
> > +	traffic->ipsec.num = n;
> >
> > -	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ip4.pkts,
> > -			traffic->ip4.res, traffic->ip4.num,
> > +	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
> > +			traffic->ipsec.res, traffic->ipsec.num,
> >   			MAX_PKT_BURST);
> >
> >   	/* They all sue the same SA (ip4 or ip6 tunnel) */
> >   	m = traffic->ipsec.pkts[i];
> >   	ip = rte_pktmbuf_mtod(m, struct ip *);
> > -	if (ip->ip_v == IPVERSION)
> > +	if (ip->ip_v == IPVERSION) {
> >   		traffic->ip4.num = nb_pkts_out;
> > -	else
> > +		for (i = 0; i < nb_pkts_out; i++)
> > +			traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
> > +	} else {
> >   		traffic->ip6.num = nb_pkts_out;
> > +		traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
> you don't need a for loop here??
> > +	}

Yep, missed that.
Will update.

> >   }
> >
> >   static inline int32_t

[PATCH v4 5/9] examples/ipsec-secgw: make local variables static

[Konstantin Ananyev]

in sp4.c and sp6.c there are few globals that used only locally.
Define them as static ones.

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/sp4.c | 10 +++++-----
 examples/ipsec-secgw/sp6.c | 10 +++++-----
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/examples/ipsec-secgw/sp4.c b/examples/ipsec-secgw/sp4.c
index 8d3d3d8e0..6b05daaa9 100644
--- a/examples/ipsec-secgw/sp4.c
+++ b/examples/ipsec-secgw/sp4.c
@@ -44,7 +44,7 @@ enum {
 	RTE_ACL_IPV4_NUM
 };
 
-struct rte_acl_field_def ip4_defs[NUM_FIELDS_IPV4] = {
+static struct rte_acl_field_def ip4_defs[NUM_FIELDS_IPV4] = {
 	{
 	.type = RTE_ACL_FIELD_TYPE_BITMASK,
 	.size = sizeof(uint8_t),
@@ -85,11 +85,11 @@ struct rte_acl_field_def ip4_defs[NUM_FIELDS_IPV4] = {
 
 RTE_ACL_RULE_DEF(acl4_rules, RTE_DIM(ip4_defs));
 
-struct acl4_rules acl4_rules_out[MAX_ACL_RULE_NUM];
-uint32_t nb_acl4_rules_out;
+static struct acl4_rules acl4_rules_out[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl4_rules_out;
 
-struct acl4_rules acl4_rules_in[MAX_ACL_RULE_NUM];
-uint32_t nb_acl4_rules_in;
+static struct acl4_rules acl4_rules_in[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl4_rules_in;
 
 void
 parse_sp4_tokens(char **tokens, uint32_t n_tokens,
diff --git a/examples/ipsec-secgw/sp6.c b/examples/ipsec-secgw/sp6.c
index 6002afef3..dc5b94c6a 100644
--- a/examples/ipsec-secgw/sp6.c
+++ b/examples/ipsec-secgw/sp6.c
@@ -34,7 +34,7 @@ enum {
 
 #define IP6_ADDR_SIZE 16
 
-struct rte_acl_field_def ip6_defs[IP6_NUM] = {
+static struct rte_acl_field_def ip6_defs[IP6_NUM] = {
 	{
 	.type = RTE_ACL_FIELD_TYPE_BITMASK,
 	.size = sizeof(uint8_t),
@@ -116,11 +116,11 @@ struct rte_acl_field_def ip6_defs[IP6_NUM] = {
 
 RTE_ACL_RULE_DEF(acl6_rules, RTE_DIM(ip6_defs));
 
-struct acl6_rules acl6_rules_out[MAX_ACL_RULE_NUM];
-uint32_t nb_acl6_rules_out;
+static struct acl6_rules acl6_rules_out[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl6_rules_out;
 
-struct acl6_rules acl6_rules_in[MAX_ACL_RULE_NUM];
-uint32_t nb_acl6_rules_in;
+static struct acl6_rules acl6_rules_in[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl6_rules_in;
 
 void
 parse_sp6_tokens(char **tokens, uint32_t n_tokens,
-- 
2.17.1

[PATCH v4 6/9] examples/ipsec-secgw: make app to use ipsec library

[Konstantin Ananyev]

Changes to make ipsec-secgw to utilize librte_ipsec library.
That patch provides:
 - changes in the related data structures.
 - changes in the initialization code.
 - new command-line parameters to enable librte_ipsec codepath
   and related features.

Note that right now by default current (non-librte_ipsec) code-path will
be used. User has to run application with new command-line option ('-l')
to enable new codepath.
The main reason for that:
 - current librte_ipsec doesn't support all ipsec algorithms
   and features that the app does.
 - allow users to run both versions in parallel for some time
   to figure out any functional or performance degradation with the
   new code.

It is planned to deprecate and remove non-librte_ipsec code path
in future releases.

Signed-off-by: Mohammad Abdul Awal <mohammad.abdul.awal@intel.com>
Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c |  50 ++++++-
 examples/ipsec-secgw/ipsec.h       |  24 ++++
 examples/ipsec-secgw/meson.build   |   2 +-
 examples/ipsec-secgw/sa.c          | 221 ++++++++++++++++++++++++++++-
 examples/ipsec-secgw/sp4.c         |  25 ++++
 examples/ipsec-secgw/sp6.c         |  25 ++++
 6 files changed, 341 insertions(+), 6 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index d1da2d5ce..48baa5001 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -155,6 +155,9 @@ static uint32_t single_sa;
 static uint32_t single_sa_idx;
 static uint32_t frame_size;
 
+/* application wide librte_ipsec/SA parameters */
+struct app_sa_prm app_sa_prm = {.enable = 0};
+
 struct lcore_rx_queue {
 	uint16_t port_id;
 	uint8_t queue_id;
@@ -1063,6 +1066,10 @@ print_usage(const char *prgname)
 		" [-P]"
 		" [-u PORTMASK]"
 		" [-j FRAMESIZE]"
+		" [-l]"
+		" [-w REPLAY_WINDOW_SIZE]"
+		" [-e]"
+		" [-a]"
 		" -f CONFIG_FILE"
 		" --config (port,queue,lcore)[,(port,queue,lcore)]"
 		" [--single-sa SAIDX]"
@@ -1073,6 +1080,10 @@ print_usage(const char *prgname)
 		"  -u PORTMASK: Hexadecimal bitmask of unprotected ports\n"
 		"  -j FRAMESIZE: Enable jumbo frame with 'FRAMESIZE' as maximum\n"
 		"                packet size\n"
+		"  -l enables code-path that uses librte_ipsec\n"
+		"  -w REPLAY_WINDOW_SIZE specifies IPsec SQN replay window\n"
+		"     size for each SA\n"
+		"  -a enables SA SQN atomic behaviour\n"
 		"  -f CONFIG_FILE: Configuration file\n"
 		"  --config (port,queue,lcore): Rx queue configuration\n"
 		"  --single-sa SAIDX: Use single SA index for outbound traffic,\n"
@@ -1169,6 +1180,20 @@ parse_config(const char *q_arg)
 	return 0;
 }
 
+static void
+print_app_sa_prm(const struct app_sa_prm *prm)
+{
+	printf("librte_ipsec usage: %s\n",
+		(prm->enable == 0) ? "disabled" : "enabled");
+
+	if (prm->enable == 0)
+		return;
+
+	printf("replay window size: %u\n", prm->window_size);
+	printf("ESN: %s\n", (prm->enable_esn == 0) ? "disabled" : "enabled");
+	printf("SA flags: %#" PRIx64 "\n", prm->flags);
+}
+
 static int32_t
 parse_args(int32_t argc, char **argv)
 {
@@ -1180,7 +1205,7 @@ parse_args(int32_t argc, char **argv)
 
 	argvopt = argv;
 
-	while ((opt = getopt_long(argc, argvopt, "p:Pu:f:j:",
+	while ((opt = getopt_long(argc, argvopt, "aelp:Pu:f:j:w:",
 				lgopts, &option_index)) != EOF) {
 
 		switch (opt) {
@@ -1236,6 +1261,21 @@ parse_args(int32_t argc, char **argv)
 			}
 			printf("Enabled jumbo frames size %u\n", frame_size);
 			break;
+		case 'l':
+			app_sa_prm.enable = 1;
+			break;
+		case 'w':
+			app_sa_prm.enable = 1;
+			app_sa_prm.window_size = parse_decimal(optarg);
+			break;
+		case 'e':
+			app_sa_prm.enable = 1;
+			app_sa_prm.enable_esn = 1;
+			break;
+		case 'a':
+			app_sa_prm.enable = 1;
+			app_sa_prm.flags |= RTE_IPSEC_SAFLAG_SQN_ATOM;
+			break;
 		case CMD_LINE_OPT_CONFIG_NUM:
 			ret = parse_config(optarg);
 			if (ret) {
@@ -1280,6 +1320,8 @@ parse_args(int32_t argc, char **argv)
 		return -1;
 	}
 
+	print_app_sa_prm(&app_sa_prm);
+
 	if (optind >= 0)
 		argv[optind-1] = prgname;
 
@@ -1923,12 +1965,14 @@ main(int32_t argc, char **argv)
 		if (socket_ctx[socket_id].mbuf_pool)
 			continue;
 
-		sa_init(&socket_ctx[socket_id], socket_id);
-
+		/* initilaze SPD */
 		sp4_init(&socket_ctx[socket_id], socket_id);
 
 		sp6_init(&socket_ctx[socket_id], socket_id);
 
+		/* initilaze SAD */
+		sa_init(&socket_ctx[socket_id], socket_id);
+
 		rt_init(&socket_ctx[socket_id], socket_id);
 
 		pool_init(&socket_ctx[socket_id], socket_id, NB_MBUF);
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 2f04b7d68..b089fe54b 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -11,6 +11,7 @@
 #include <rte_crypto.h>
 #include <rte_security.h>
 #include <rte_flow.h>
+#include <rte_ipsec.h>
 
 #define RTE_LOGTYPE_IPSEC       RTE_LOGTYPE_USER1
 #define RTE_LOGTYPE_IPSEC_ESP   RTE_LOGTYPE_USER2
@@ -70,7 +71,20 @@ struct ip_addr {
 
 #define MAX_KEY_SIZE		32
 
+/*
+ * application wide SA parameters
+ */
+struct app_sa_prm {
+	uint32_t enable; /* use librte_ipsec API for ipsec pkt processing */
+	uint32_t window_size; /* replay window size */
+	uint32_t enable_esn;  /* enable/disable ESN support */
+	uint64_t flags;       /* rte_ipsec_sa_prm.flags */
+};
+
+extern struct app_sa_prm app_sa_prm;
+
 struct ipsec_sa {
+	struct rte_ipsec_session ips; /* one session per sa for now */
 	uint32_t spi;
 	uint32_t cdev_id_qp;
 	uint64_t seq;
@@ -243,6 +257,16 @@ sp4_init(struct socket_ctx *ctx, int32_t socket_id);
 void
 sp6_init(struct socket_ctx *ctx, int32_t socket_id);
 
+/*
+ * Search though SP rules for given SPI.
+ * Returns first rule index if found(greater or equal then zero),
+ * or -ENOENT otherwise.
+ */
+int
+sp4_spi_present(uint32_t spi, int inbound);
+int
+sp6_spi_present(uint32_t spi, int inbound);
+
 void
 sa_init(struct socket_ctx *ctx, int32_t socket_id);
 
diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
index 77d8b298f..31f68fee2 100644
--- a/examples/ipsec-secgw/meson.build
+++ b/examples/ipsec-secgw/meson.build
@@ -6,7 +6,7 @@
 # To build this example as a standalone application with an already-installed
 # DPDK instance, use 'make'
 
-deps += ['security', 'lpm', 'acl', 'hash']
+deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
 allow_experimental_apis = true
 sources = files(
 	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index ff8c4b829..04eb8bce8 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -19,6 +19,7 @@
 #include <rte_ip.h>
 #include <rte_random.h>
 #include <rte_ethdev.h>
+#include <rte_malloc.h>
 
 #include "ipsec.h"
 #include "esp.h"
@@ -694,6 +695,7 @@ print_one_sa_rule(const struct ipsec_sa *sa, int inbound)
 }
 
 struct sa_ctx {
+	void *satbl; /* pointer to array of rte_ipsec_sa objects*/
 	struct ipsec_sa sa[IPSEC_SA_MAX_ENTRIES];
 	union {
 		struct {
@@ -764,7 +766,10 @@ sa_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 {
 	struct ipsec_sa *sa;
 	uint32_t i, idx;
-	uint16_t iv_length;
+	uint16_t iv_length, aad_length;
+
+	/* for ESN upper 32 bits of SQN also need to be part of AAD */
+	aad_length = (app_sa_prm.enable_esn != 0) ? sizeof(uint32_t) : 0;
 
 	for (i = 0; i < nb_entries; i++) {
 		idx = SPI2IDX(entries[i].spi);
@@ -808,7 +813,7 @@ sa_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 			sa_ctx->xf[idx].a.aead.iv.offset = IV_OFFSET;
 			sa_ctx->xf[idx].a.aead.iv.length = iv_length;
 			sa_ctx->xf[idx].a.aead.aad_length =
-				sa->aad_len;
+				sa->aad_len + aad_length;
 			sa_ctx->xf[idx].a.aead.digest_length =
 				sa->digest_len;
 
@@ -901,9 +906,205 @@ sa_in_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 	return sa_add_rules(sa_ctx, entries, nb_entries, 1);
 }
 
+/*
+ * helper function, fills parameters that are identical for all SAs
+ */
+static void
+fill_ipsec_app_sa_prm(struct rte_ipsec_sa_prm *prm,
+	const struct app_sa_prm *app_prm)
+{
+	memset(prm, 0, sizeof(*prm));
+
+	prm->flags = app_prm->flags;
+	prm->ipsec_xform.options.esn = app_prm->enable_esn;
+	prm->replay_win_sz = app_prm->window_size;
+}
+
+/*
+ * Helper function, tries to determine next_proto for SPI
+ * by searching though SP rules.
+ */
+static int
+get_spi_proto(uint32_t spi, enum rte_security_ipsec_sa_direction dir)
+{
+	int32_t rc4, rc6;
+
+	rc4 = sp4_spi_present(spi, dir == RTE_SECURITY_IPSEC_SA_DIR_INGRESS);
+	rc6 = sp6_spi_present(spi, dir == RTE_SECURITY_IPSEC_SA_DIR_INGRESS);
+
+	if (rc4 >= 0) {
+		if (rc6 >= 0) {
+			RTE_LOG(ERR, IPSEC,
+				"%s: SPI %u used simultaeously by "
+				"IPv4(%d) and IPv6 (%d) SP rules\n",
+				__func__, spi, rc4, rc6);
+			return -EINVAL;
+		} else
+			return IPPROTO_IPIP;
+	} else if (rc6 < 0) {
+		RTE_LOG(ERR, IPSEC,
+			"%s: SPI %u is not used by any SP rule\n",
+			__func__, spi);
+		return -EINVAL;
+	} else
+		return IPPROTO_IPV6;
+}
+
+static int
+fill_ipsec_sa_prm(struct rte_ipsec_sa_prm *prm, const struct ipsec_sa *ss,
+	const struct ipv4_hdr *v4, struct ipv6_hdr *v6)
+{
+	int32_t rc;
+
+	/*
+	 * Try to get SPI next proto by searching that SPI in SPD.
+	 * probably not the optimal way, but there seems nothing
+	 * better right now.
+	 */
+	rc = get_spi_proto(ss->spi, ss->direction);
+	if (rc < 0)
+		return rc;
+
+	fill_ipsec_app_sa_prm(prm, &app_sa_prm);
+	prm->userdata = (uintptr_t)ss;
+
+	/* setup ipsec xform */
+	prm->ipsec_xform.spi = ss->spi;
+	prm->ipsec_xform.salt = ss->salt;
+	prm->ipsec_xform.direction = ss->direction;
+	prm->ipsec_xform.proto = RTE_SECURITY_IPSEC_SA_PROTO_ESP;
+	prm->ipsec_xform.mode = (ss->flags == TRANSPORT) ?
+		RTE_SECURITY_IPSEC_SA_MODE_TRANSPORT :
+		RTE_SECURITY_IPSEC_SA_MODE_TUNNEL;
+
+	if (ss->flags == IP4_TUNNEL) {
+		prm->ipsec_xform.tunnel.type = RTE_SECURITY_IPSEC_TUNNEL_IPV4;
+		prm->tun.hdr_len = sizeof(*v4);
+		prm->tun.next_proto = rc;
+		prm->tun.hdr = v4;
+	} else if (ss->flags == IP6_TUNNEL) {
+		prm->ipsec_xform.tunnel.type = RTE_SECURITY_IPSEC_TUNNEL_IPV6;
+		prm->tun.hdr_len = sizeof(*v6);
+		prm->tun.next_proto = rc;
+		prm->tun.hdr = v6;
+	} else {
+		/* transport mode */
+		prm->trs.proto = rc;
+	}
+
+	/* setup crypto section */
+	prm->crypto_xform = ss->xforms;
+	return 0;
+}
+
+static void
+fill_ipsec_session(struct rte_ipsec_session *ss, struct rte_ipsec_sa *sa,
+	const struct ipsec_sa *lsa)
+{
+	ss->sa = sa;
+	ss->type = lsa->type;
+
+	/* setup crypto section */
+	if (ss->type == RTE_SECURITY_ACTION_TYPE_NONE) {
+		ss->crypto.ses = lsa->crypto_session;
+	/* setup session action type */
+	} else {
+		ss->security.ses = lsa->sec_session;
+		ss->security.ctx = lsa->security_ctx;
+		ss->security.ol_flags = lsa->ol_flags;
+	}
+}
+
+/*
+ * Initialise related rte_ipsec_sa object.
+ */
+static int
+ipsec_sa_init(struct ipsec_sa *lsa, struct rte_ipsec_sa *sa, uint32_t sa_size)
+{
+	int rc;
+	struct rte_ipsec_sa_prm prm;
+	struct ipv4_hdr v4  = {
+		.version_ihl = IPVERSION << 4 |
+			sizeof(v4) / IPV4_IHL_MULTIPLIER,
+		.time_to_live = IPDEFTTL,
+		.next_proto_id = IPPROTO_ESP,
+		.src_addr = lsa->src.ip.ip4,
+		.dst_addr = lsa->dst.ip.ip4,
+	};
+	struct ipv6_hdr v6 = {
+		.vtc_flow = htonl(IP6_VERSION << 28),
+		.proto = IPPROTO_ESP,
+	};
+
+	if (lsa->flags == IP6_TUNNEL) {
+		memcpy(v6.src_addr, lsa->src.ip.ip6.ip6_b, sizeof(v6.src_addr));
+		memcpy(v6.dst_addr, lsa->dst.ip.ip6.ip6_b, sizeof(v6.dst_addr));
+	}
+
+	rc = fill_ipsec_sa_prm(&prm, lsa, &v4, &v6);
+	if (rc == 0)
+		rc = rte_ipsec_sa_init(sa, &prm, sa_size);
+	if (rc < 0)
+		return rc;
+
+	fill_ipsec_session(&lsa->ips, sa, lsa);
+	return 0;
+}
+
+/*
+ * Allocate space and init rte_ipsec_sa strcutures,
+ * one per session.
+ */
+static int
+ipsec_satbl_init(struct sa_ctx *ctx, const struct ipsec_sa *ent,
+	uint32_t nb_ent, int32_t socket)
+{
+	int32_t rc, sz;
+	uint32_t i, idx;
+	size_t tsz;
+	struct rte_ipsec_sa *sa;
+	struct ipsec_sa *lsa;
+	struct rte_ipsec_sa_prm prm;
+
+	/* determine SA size */
+	idx = SPI2IDX(ent[0].spi);
+	fill_ipsec_sa_prm(&prm, ctx->sa + idx, NULL, NULL);
+	sz = rte_ipsec_sa_size(&prm);
+	if (sz < 0) {
+		RTE_LOG(ERR, IPSEC, "%s(%p, %u, %d): "
+			"failed to determine SA size, error code: %d\n",
+			__func__, ctx, nb_ent, socket, sz);
+		return sz;
+	}
+
+	tsz = sz * nb_ent;
+
+	ctx->satbl = rte_zmalloc_socket(NULL, tsz, RTE_CACHE_LINE_SIZE, socket);
+	if (ctx->satbl == NULL) {
+		RTE_LOG(ERR, IPSEC,
+			"%s(%p, %u, %d): failed to allocate %zu bytes\n",
+			__func__,  ctx, nb_ent, socket, tsz);
+		return -ENOMEM;
+	}
+
+	rc = 0;
+	for (i = 0; i != nb_ent && rc == 0; i++) {
+
+		idx = SPI2IDX(ent[i].spi);
+
+		sa = (struct rte_ipsec_sa *)((uintptr_t)ctx->satbl + sz * i);
+		lsa = ctx->sa + idx;
+
+		rc = ipsec_sa_init(lsa, sa, sz);
+	}
+
+	return rc;
+}
+
 void
 sa_init(struct socket_ctx *ctx, int32_t socket_id)
 {
+	int32_t rc;
 	const char *name;
 
 	if (ctx == NULL)
@@ -926,6 +1127,14 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id)
 				name, socket_id);
 
 		sa_in_add_rules(ctx->sa_in, sa_in, nb_sa_in);
+
+		if (app_sa_prm.enable != 0) {
+			rc = ipsec_satbl_init(ctx->sa_in, sa_in, nb_sa_in,
+				socket_id);
+			if (rc != 0)
+				rte_exit(EXIT_FAILURE,
+					"failed to init inbound SAs\n");
+		}
 	} else
 		RTE_LOG(WARNING, IPSEC, "No SA Inbound rule specified\n");
 
@@ -938,6 +1147,14 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id)
 				name, socket_id);
 
 		sa_out_add_rules(ctx->sa_out, sa_out, nb_sa_out);
+
+		if (app_sa_prm.enable != 0) {
+			rc = ipsec_satbl_init(ctx->sa_out, sa_out, nb_sa_out,
+				socket_id);
+			if (rc != 0)
+				rte_exit(EXIT_FAILURE,
+					"failed to init outbound SAs\n");
+		}
 	} else
 		RTE_LOG(WARNING, IPSEC, "No SA Outbound rule "
 			"specified\n");
diff --git a/examples/ipsec-secgw/sp4.c b/examples/ipsec-secgw/sp4.c
index 6b05daaa9..d1dc64bad 100644
--- a/examples/ipsec-secgw/sp4.c
+++ b/examples/ipsec-secgw/sp4.c
@@ -504,3 +504,28 @@ sp4_init(struct socket_ctx *ctx, int32_t socket_id)
 		RTE_LOG(WARNING, IPSEC, "No IPv4 SP Outbound rule "
 			"specified\n");
 }
+
+/*
+ * Search though SP rules for given SPI.
+ */
+int
+sp4_spi_present(uint32_t spi, int inbound)
+{
+	uint32_t i, num;
+	const struct acl4_rules *acr;
+
+	if (inbound != 0) {
+		acr = acl4_rules_in;
+		num = nb_acl4_rules_in;
+	} else {
+		acr = acl4_rules_out;
+		num = nb_acl4_rules_out;
+	}
+
+	for (i = 0; i != num; i++) {
+		if (acr[i].data.userdata == PROTECT(spi))
+			return i;
+	}
+
+	return -ENOENT;
+}
diff --git a/examples/ipsec-secgw/sp6.c b/examples/ipsec-secgw/sp6.c
index dc5b94c6a..e67d85aaf 100644
--- a/examples/ipsec-secgw/sp6.c
+++ b/examples/ipsec-secgw/sp6.c
@@ -618,3 +618,28 @@ sp6_init(struct socket_ctx *ctx, int32_t socket_id)
 		RTE_LOG(WARNING, IPSEC, "No IPv6 SP Outbound rule "
 			"specified\n");
 }
+
+/*
+ * Search though SP rules for given SPI.
+ */
+int
+sp6_spi_present(uint32_t spi, int inbound)
+{
+	uint32_t i, num;
+	const struct acl6_rules *acr;
+
+	if (inbound != 0) {
+		acr = acl6_rules_in;
+		num = nb_acl6_rules_in;
+	} else {
+		acr = acl6_rules_out;
+		num = nb_acl6_rules_out;
+	}
+
+	for (i = 0; i != num; i++) {
+		if (acr[i].data.userdata == PROTECT(spi))
+			return i;
+	}
+
+	return -ENOENT;
+}
-- 
2.17.1

Re: [PATCH v4 6/9] examples/ipsec-secgw: make app to use ipsec library

[Akhil Goyal]

On 12/14/2018 10:10 PM, Konstantin Ananyev wrote:
> Changes to make ipsec-secgw to utilize librte_ipsec library.
> That patch provides:
>   - changes in the related data structures.
>   - changes in the initialization code.
>   - new command-line parameters to enable librte_ipsec codepath
>     and related features.
>
> Note that right now by default current (non-librte_ipsec) code-path will
> be used. User has to run application with new command-line option ('-l')
> to enable new codepath.
> The main reason for that:
>   - current librte_ipsec doesn't support all ipsec algorithms
>     and features that the app does.
>   - allow users to run both versions in parallel for some time
>     to figure out any functional or performance degradation with the
>     new code.
>
> It is planned to deprecate and remove non-librte_ipsec code path
> in future releases.
>
> Signed-off-by: Mohammad Abdul Awal <mohammad.abdul.awal@intel.com>
> Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> Acked-by: Radu Nicolau <radu.nicolau@intel.com>
> ---
>   examples/ipsec-secgw/ipsec-secgw.c |  50 ++++++-
>   examples/ipsec-secgw/ipsec.h       |  24 ++++
>   examples/ipsec-secgw/meson.build   |   2 +-
>   examples/ipsec-secgw/sa.c          | 221 ++++++++++++++++++++++++++++-
>   examples/ipsec-secgw/sp4.c         |  25 ++++
>   examples/ipsec-secgw/sp6.c         |  25 ++++
>   6 files changed, 341 insertions(+), 6 deletions(-)
>
> diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
> index d1da2d5ce..48baa5001 100644
> --- a/examples/ipsec-secgw/ipsec-secgw.c
> +++ b/examples/ipsec-secgw/ipsec-secgw.c
> @@ -155,6 +155,9 @@ static uint32_t single_sa;
>   static uint32_t single_sa_idx;
>   static uint32_t frame_size;
>   
> +/* application wide librte_ipsec/SA parameters */
> +struct app_sa_prm app_sa_prm = {.enable = 0};
> +
>   struct lcore_rx_queue {
>   	uint16_t port_id;
>   	uint8_t queue_id;
> @@ -1063,6 +1066,10 @@ print_usage(const char *prgname)
>   		" [-P]"
>   		" [-u PORTMASK]"
>   		" [-j FRAMESIZE]"
> +		" [-l]"
> +		" [-w REPLAY_WINDOW_SIZE]"
> +		" [-e]"
> +		" [-a]"
>   		" -f CONFIG_FILE"
>   		" --config (port,queue,lcore)[,(port,queue,lcore)]"
>   		" [--single-sa SAIDX]"
> @@ -1073,6 +1080,10 @@ print_usage(const char *prgname)
>   		"  -u PORTMASK: Hexadecimal bitmask of unprotected ports\n"
>   		"  -j FRAMESIZE: Enable jumbo frame with 'FRAMESIZE' as maximum\n"
>   		"                packet size\n"
> +		"  -l enables code-path that uses librte_ipsec\n"
> +		"  -w REPLAY_WINDOW_SIZE specifies IPsec SQN replay window\n"
> +		"     size for each SA\n"
> +		"  -a enables SA SQN atomic behaviour\n"
-e missing
>   		"  -f CONFIG_FILE: Configuration file\n"
>   		"  --config (port,queue,lcore): Rx queue configuration\n"
>   		"  --single-sa SAIDX: Use single SA index for outbound traffic,\n"
> @@ -1169,6 +1180,20 @@ parse_config(const char *q_arg)
>   	return 0;
>   }
>   
> +static void
> +print_app_sa_prm(const struct app_sa_prm *prm)
> +{
> +	printf("librte_ipsec usage: %s\n",
> +		(prm->enable == 0) ? "disabled" : "enabled");
> +
> +	if (prm->enable == 0)
> +		return;
> +
> +	printf("replay window size: %u\n", prm->window_size);
> +	printf("ESN: %s\n", (prm->enable_esn == 0) ? "disabled" : "enabled");
> +	printf("SA flags: %#" PRIx64 "\n", prm->flags);
> +}
> +
>   static int32_t
>   parse_args(int32_t argc, char **argv)
>   {
> @@ -1180,7 +1205,7 @@ parse_args(int32_t argc, char **argv)
>   
>   	argvopt = argv;
>   
> -	while ((opt = getopt_long(argc, argvopt, "p:Pu:f:j:",
> +	while ((opt = getopt_long(argc, argvopt, "aelp:Pu:f:j:w:",
>   				lgopts, &option_index)) != EOF) {
>   
>   		switch (opt) {
> @@ -1236,6 +1261,21 @@ parse_args(int32_t argc, char **argv)
>   			}
>   			printf("Enabled jumbo frames size %u\n", frame_size);
>   			break;
> +		case 'l':
> +			app_sa_prm.enable = 1;
> +			break;
> +		case 'w':
> +			app_sa_prm.enable = 1;
> +			app_sa_prm.window_size = parse_decimal(optarg);
> +			break;
> +		case 'e':
> +			app_sa_prm.enable = 1;
> +			app_sa_prm.enable_esn = 1;
> +			break;
> +		case 'a':
> +			app_sa_prm.enable = 1;
> +			app_sa_prm.flags |= RTE_IPSEC_SAFLAG_SQN_ATOM;
> +			break;
>   		case CMD_LINE_OPT_CONFIG_NUM:
>   			ret = parse_config(optarg);
>   			if (ret) {
> @@ -1280,6 +1320,8 @@ parse_args(int32_t argc, char **argv)
>   		return -1;
>   	}
>   
> +	print_app_sa_prm(&app_sa_prm);
> +
>   	if (optind >= 0)
>   		argv[optind-1] = prgname;
>   
> @@ -1923,12 +1965,14 @@ main(int32_t argc, char **argv)
>   		if (socket_ctx[socket_id].mbuf_pool)
>   			continue;
>   
> -		sa_init(&socket_ctx[socket_id], socket_id);
> -
> +		/* initilaze SPD */
>   		sp4_init(&socket_ctx[socket_id], socket_id);
>   
>   		sp6_init(&socket_ctx[socket_id], socket_id);
>   
> +		/* initilaze SAD */
> +		sa_init(&socket_ctx[socket_id], socket_id);
> +
>   		rt_init(&socket_ctx[socket_id], socket_id);
>   
>   		pool_init(&socket_ctx[socket_id], socket_id, NB_MBUF);
> diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
> index 2f04b7d68..b089fe54b 100644
> --- a/examples/ipsec-secgw/ipsec.h
> +++ b/examples/ipsec-secgw/ipsec.h
> @@ -11,6 +11,7 @@
>   #include <rte_crypto.h>
>   #include <rte_security.h>
>   #include <rte_flow.h>
> +#include <rte_ipsec.h>
>   
>   #define RTE_LOGTYPE_IPSEC       RTE_LOGTYPE_USER1
>   #define RTE_LOGTYPE_IPSEC_ESP   RTE_LOGTYPE_USER2
> @@ -70,7 +71,20 @@ struct ip_addr {
>   
>   #define MAX_KEY_SIZE		32
>   
> +/*
> + * application wide SA parameters
> + */
> +struct app_sa_prm {
> +	uint32_t enable; /* use librte_ipsec API for ipsec pkt processing */
> +	uint32_t window_size; /* replay window size */
> +	uint32_t enable_esn;  /* enable/disable ESN support */
> +	uint64_t flags;       /* rte_ipsec_sa_prm.flags */
> +};
> +
> +extern struct app_sa_prm app_sa_prm;
> +
>   struct ipsec_sa {
> +	struct rte_ipsec_session ips; /* one session per sa for now */
>   	uint32_t spi;
>   	uint32_t cdev_id_qp;
>   	uint64_t seq;
> @@ -243,6 +257,16 @@ sp4_init(struct socket_ctx *ctx, int32_t socket_id);
>   void
>   sp6_init(struct socket_ctx *ctx, int32_t socket_id);
>   
> +/*
> + * Search though SP rules for given SPI.
spell check
> + * Returns first rule index if found(greater or equal then zero),
> + * or -ENOENT otherwise.
> + */
> +int
> +sp4_spi_present(uint32_t spi, int inbound);
> +int
> +sp6_spi_present(uint32_t spi, int inbound);
> +
>   void
>   sa_init(struct socket_ctx *ctx, int32_t socket_id);
>   
> diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
> index 77d8b298f..31f68fee2 100644
> --- a/examples/ipsec-secgw/meson.build
> +++ b/examples/ipsec-secgw/meson.build
> @@ -6,7 +6,7 @@
>   # To build this example as a standalone application with an already-installed
>   # DPDK instance, use 'make'
>   
> -deps += ['security', 'lpm', 'acl', 'hash']
> +deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
>   allow_experimental_apis = true
>   sources = files(
>   	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
>
Makefile should also be updated I guess.

.....
will be running the application and will come back with issues if any.

-Akhil

Re: [PATCH v4 6/9] examples/ipsec-secgw: make app to use ipsec library

[Ananyev, Konstantin]

> > diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
> > index 77d8b298f..31f68fee2 100644
> > --- a/examples/ipsec-secgw/meson.build
> > +++ b/examples/ipsec-secgw/meson.build
> > @@ -6,7 +6,7 @@
> >   # To build this example as a standalone application with an already-installed
> >   # DPDK instance, use 'make'
> >
> > -deps += ['security', 'lpm', 'acl', 'hash']
> > +deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
> >   allow_experimental_apis = true
> >   sources = files(
> >   	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
> >
> Makefile should also be updated I guess.

Anything particular you think is missed?
Konstantin

Re: [PATCH v4 6/9] examples/ipsec-secgw: make app to use ipsec library

[Akhil Goyal]

On 12/24/2018 5:59 PM, Ananyev, Konstantin wrote:
>>> diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
>>> index 77d8b298f..31f68fee2 100644
>>> --- a/examples/ipsec-secgw/meson.build
>>> +++ b/examples/ipsec-secgw/meson.build
>>> @@ -6,7 +6,7 @@
>>>    # To build this example as a standalone application with an already-installed
>>>    # DPDK instance, use 'make'
>>>
>>> -deps += ['security', 'lpm', 'acl', 'hash']
>>> +deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
>>>    allow_experimental_apis = true
>>>    sources = files(
>>>    	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
>>>
>> Makefile should also be updated I guess.
> Anything particular you think is missed?
Will it compile with makefile when IPSEC lib is disabled?
> Konstantin
>

Re: [PATCH v4 6/9] examples/ipsec-secgw: make app to use ipsec library

[Ananyev, Konstantin]

> -----Original Message-----
> From: Akhil Goyal [mailto:akhil.goyal@nxp.com]
> Sent: Monday, December 24, 2018 12:33 PM
> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
> Cc: Nicolau, Radu <radu.nicolau@intel.com>; Awal, Mohammad Abdul <mohammad.abdul.awal@intel.com>; Iremonger, Bernard
> <bernard.iremonger@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v4 6/9] examples/ipsec-secgw: make app to use ipsec library
> 
> 
> 
> On 12/24/2018 5:59 PM, Ananyev, Konstantin wrote:
> >>> diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
> >>> index 77d8b298f..31f68fee2 100644
> >>> --- a/examples/ipsec-secgw/meson.build
> >>> +++ b/examples/ipsec-secgw/meson.build
> >>> @@ -6,7 +6,7 @@
> >>>    # To build this example as a standalone application with an already-installed
> >>>    # DPDK instance, use 'make'
> >>>
> >>> -deps += ['security', 'lpm', 'acl', 'hash']
> >>> +deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
> >>>    allow_experimental_apis = true
> >>>    sources = files(
> >>>    	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
> >>>
> >> Makefile should also be updated I guess.
> > Anything particular you think is missed?
> Will it compile with makefile when IPSEC lib is disabled?

Nope, it wouldn't.

Re: [PATCH v4 6/9] examples/ipsec-secgw: make app to use ipsec library

[Ananyev, Konstantin]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ananyev, Konstantin
> Sent: Monday, December 24, 2018 12:37 PM
> To: Akhil Goyal <akhil.goyal@nxp.com>; dev@dpdk.org
> Cc: Nicolau, Radu <radu.nicolau@intel.com>; Awal, Mohammad Abdul <mohammad.abdul.awal@intel.com>; Iremonger, Bernard
> <bernard.iremonger@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v4 6/9] examples/ipsec-secgw: make app to use ipsec library
> 
> 
> 
> > -----Original Message-----
> > From: Akhil Goyal [mailto:akhil.goyal@nxp.com]
> > Sent: Monday, December 24, 2018 12:33 PM
> > To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
> > Cc: Nicolau, Radu <radu.nicolau@intel.com>; Awal, Mohammad Abdul <mohammad.abdul.awal@intel.com>; Iremonger, Bernard
> > <bernard.iremonger@intel.com>
> > Subject: Re: [dpdk-dev] [PATCH v4 6/9] examples/ipsec-secgw: make app to use ipsec library
> >
> >
> >
> > On 12/24/2018 5:59 PM, Ananyev, Konstantin wrote:
> > >>> diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
> > >>> index 77d8b298f..31f68fee2 100644
> > >>> --- a/examples/ipsec-secgw/meson.build
> > >>> +++ b/examples/ipsec-secgw/meson.build
> > >>> @@ -6,7 +6,7 @@
> > >>>    # To build this example as a standalone application with an already-installed
> > >>>    # DPDK instance, use 'make'
> > >>>
> > >>> -deps += ['security', 'lpm', 'acl', 'hash']
> > >>> +deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
> > >>>    allow_experimental_apis = true
> > >>>    sources = files(
> > >>>    	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
> > >>>
> > >> Makefile should also be updated I guess.
> > > Anything particular you think is missed?
> > Will it compile with makefile when IPSEC lib is disabled?
> 
> Nope, it wouldn't.
> 
Could you be more specific and describe what particular
changes in the Makefile you think are necessary?
Is it a check that librte_ipsec was enabled, like one you have for rte_security:
ifneq ($(CONFIG_RTE_LIBRTE_SECURITY),y)
$(error "RTE_LIBRTE_SECURITY is required to build ipsec-secgw")
endif
?
Something else?
BTW, why  this check above is needed?
Konstantin

Re: [PATCH v4 6/9] examples/ipsec-secgw: make app to use ipsec library

[Akhil Goyal]

On 12/24/2018 6:51 PM, Ananyev, Konstantin wrote:
>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ananyev, Konstantin
>> Sent: Monday, December 24, 2018 12:37 PM
>> To: Akhil Goyal <akhil.goyal@nxp.com>; dev@dpdk.org
>> Cc: Nicolau, Radu <radu.nicolau@intel.com>; Awal, Mohammad Abdul <mohammad.abdul.awal@intel.com>; Iremonger, Bernard
>> <bernard.iremonger@intel.com>
>> Subject: Re: [dpdk-dev] [PATCH v4 6/9] examples/ipsec-secgw: make app to use ipsec library
>>
>>
>>
>>> -----Original Message-----
>>> From: Akhil Goyal [mailto:akhil.goyal@nxp.com]
>>> Sent: Monday, December 24, 2018 12:33 PM
>>> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
>>> Cc: Nicolau, Radu <radu.nicolau@intel.com>; Awal, Mohammad Abdul <mohammad.abdul.awal@intel.com>; Iremonger, Bernard
>>> <bernard.iremonger@intel.com>
>>> Subject: Re: [dpdk-dev] [PATCH v4 6/9] examples/ipsec-secgw: make app to use ipsec library
>>>
>>>
>>>
>>> On 12/24/2018 5:59 PM, Ananyev, Konstantin wrote:
>>>>>> diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
>>>>>> index 77d8b298f..31f68fee2 100644
>>>>>> --- a/examples/ipsec-secgw/meson.build
>>>>>> +++ b/examples/ipsec-secgw/meson.build
>>>>>> @@ -6,7 +6,7 @@
>>>>>>     # To build this example as a standalone application with an already-installed
>>>>>>     # DPDK instance, use 'make'
>>>>>>
>>>>>> -deps += ['security', 'lpm', 'acl', 'hash']
>>>>>> +deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
>>>>>>     allow_experimental_apis = true
>>>>>>     sources = files(
>>>>>>     	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
>>>>>>
>>>>> Makefile should also be updated I guess.
>>>> Anything particular you think is missed?
>>> Will it compile with makefile when IPSEC lib is disabled?
>> Nope, it wouldn't.
>>
> Could you be more specific and describe what particular
> changes in the Makefile you think are necessary?
> Is it a check that librte_ipsec was enabled, like one you have for rte_security:
> ifneq ($(CONFIG_RTE_LIBRTE_SECURITY),y)
> $(error "RTE_LIBRTE_SECURITY is required to build ipsec-secgw")
> endif
> ?
yes
> Something else?
> BTW, why  this check above is needed?
To ensure that user do not compile ipsec-secgw without ipsec/security lib
> Konstantin
>
>

Re: [PATCH v4 6/9] examples/ipsec-secgw: make app to use ipsec library

[Ananyev, Konstantin]

> -----Original Message-----
> From: Akhil Goyal [mailto:akhil.goyal@nxp.com]
> Sent: Monday, December 24, 2018 1:50 PM
> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
> Cc: Nicolau, Radu <radu.nicolau@intel.com>; Awal, Mohammad Abdul <mohammad.abdul.awal@intel.com>; Iremonger, Bernard
> <bernard.iremonger@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v4 6/9] examples/ipsec-secgw: make app to use ipsec library
> 
> 
> 
> On 12/24/2018 6:51 PM, Ananyev, Konstantin wrote:
> >
> >> -----Original Message-----
> >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ananyev, Konstantin
> >> Sent: Monday, December 24, 2018 12:37 PM
> >> To: Akhil Goyal <akhil.goyal@nxp.com>; dev@dpdk.org
> >> Cc: Nicolau, Radu <radu.nicolau@intel.com>; Awal, Mohammad Abdul <mohammad.abdul.awal@intel.com>; Iremonger, Bernard
> >> <bernard.iremonger@intel.com>
> >> Subject: Re: [dpdk-dev] [PATCH v4 6/9] examples/ipsec-secgw: make app to use ipsec library
> >>
> >>
> >>
> >>> -----Original Message-----
> >>> From: Akhil Goyal [mailto:akhil.goyal@nxp.com]
> >>> Sent: Monday, December 24, 2018 12:33 PM
> >>> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
> >>> Cc: Nicolau, Radu <radu.nicolau@intel.com>; Awal, Mohammad Abdul <mohammad.abdul.awal@intel.com>; Iremonger, Bernard
> >>> <bernard.iremonger@intel.com>
> >>> Subject: Re: [dpdk-dev] [PATCH v4 6/9] examples/ipsec-secgw: make app to use ipsec library
> >>>
> >>>
> >>>
> >>> On 12/24/2018 5:59 PM, Ananyev, Konstantin wrote:
> >>>>>> diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
> >>>>>> index 77d8b298f..31f68fee2 100644
> >>>>>> --- a/examples/ipsec-secgw/meson.build
> >>>>>> +++ b/examples/ipsec-secgw/meson.build
> >>>>>> @@ -6,7 +6,7 @@
> >>>>>>     # To build this example as a standalone application with an already-installed
> >>>>>>     # DPDK instance, use 'make'
> >>>>>>
> >>>>>> -deps += ['security', 'lpm', 'acl', 'hash']
> >>>>>> +deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
> >>>>>>     allow_experimental_apis = true
> >>>>>>     sources = files(
> >>>>>>     	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
> >>>>>>
> >>>>> Makefile should also be updated I guess.
> >>>> Anything particular you think is missed?
> >>> Will it compile with makefile when IPSEC lib is disabled?
> >> Nope, it wouldn't.
> >>
> > Could you be more specific and describe what particular
> > changes in the Makefile you think are necessary?
> > Is it a check that librte_ipsec was enabled, like one you have for rte_security:
> > ifneq ($(CONFIG_RTE_LIBRTE_SECURITY),y)
> > $(error "RTE_LIBRTE_SECURITY is required to build ipsec-secgw")
> > endif
> > ?
> yes
> > Something else?
> > BTW, why  this check above is needed?
> To ensure that user do not compile ipsec-secgw without ipsec/security lib

ipsec-secgw depends on a lot of other libs (cryptodev, ethdev, acl, lpm, etc.).
Why only these 2 dependencies require a special check? 

Re: [PATCH v4 6/9] examples/ipsec-secgw: make app to use ipsec library

[Akhil Goyal]

On 12/24/2018 8:31 PM, Ananyev, Konstantin wrote:
>
>> -----Original Message-----
>> From: Akhil Goyal [mailto:akhil.goyal@nxp.com]
>> Sent: Monday, December 24, 2018 1:50 PM
>> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
>> Cc: Nicolau, Radu <radu.nicolau@intel.com>; Awal, Mohammad Abdul <mohammad.abdul.awal@intel.com>; Iremonger, Bernard
>> <bernard.iremonger@intel.com>
>> Subject: Re: [dpdk-dev] [PATCH v4 6/9] examples/ipsec-secgw: make app to use ipsec library
>>
>>
>>
>> On 12/24/2018 6:51 PM, Ananyev, Konstantin wrote:
>>>> -----Original Message-----
>>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ananyev, Konstantin
>>>> Sent: Monday, December 24, 2018 12:37 PM
>>>> To: Akhil Goyal <akhil.goyal@nxp.com>; dev@dpdk.org
>>>> Cc: Nicolau, Radu <radu.nicolau@intel.com>; Awal, Mohammad Abdul <mohammad.abdul.awal@intel.com>; Iremonger, Bernard
>>>> <bernard.iremonger@intel.com>
>>>> Subject: Re: [dpdk-dev] [PATCH v4 6/9] examples/ipsec-secgw: make app to use ipsec library
>>>>
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: Akhil Goyal [mailto:akhil.goyal@nxp.com]
>>>>> Sent: Monday, December 24, 2018 12:33 PM
>>>>> To: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org
>>>>> Cc: Nicolau, Radu <radu.nicolau@intel.com>; Awal, Mohammad Abdul <mohammad.abdul.awal@intel.com>; Iremonger, Bernard
>>>>> <bernard.iremonger@intel.com>
>>>>> Subject: Re: [dpdk-dev] [PATCH v4 6/9] examples/ipsec-secgw: make app to use ipsec library
>>>>>
>>>>>
>>>>>
>>>>> On 12/24/2018 5:59 PM, Ananyev, Konstantin wrote:
>>>>>>>> diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
>>>>>>>> index 77d8b298f..31f68fee2 100644
>>>>>>>> --- a/examples/ipsec-secgw/meson.build
>>>>>>>> +++ b/examples/ipsec-secgw/meson.build
>>>>>>>> @@ -6,7 +6,7 @@
>>>>>>>>      # To build this example as a standalone application with an already-installed
>>>>>>>>      # DPDK instance, use 'make'
>>>>>>>>
>>>>>>>> -deps += ['security', 'lpm', 'acl', 'hash']
>>>>>>>> +deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
>>>>>>>>      allow_experimental_apis = true
>>>>>>>>      sources = files(
>>>>>>>>      	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
>>>>>>>>
>>>>>>> Makefile should also be updated I guess.
>>>>>> Anything particular you think is missed?
>>>>> Will it compile with makefile when IPSEC lib is disabled?
>>>> Nope, it wouldn't.
>>>>
>>> Could you be more specific and describe what particular
>>> changes in the Makefile you think are necessary?
>>> Is it a check that librte_ipsec was enabled, like one you have for rte_security:
>>> ifneq ($(CONFIG_RTE_LIBRTE_SECURITY),y)
>>> $(error "RTE_LIBRTE_SECURITY is required to build ipsec-secgw")
>>> endif
>>> ?
>> yes
>>> Something else?
>>> BTW, why  this check above is needed?
>> To ensure that user do not compile ipsec-secgw without ipsec/security lib
> ipsec-secgw depends on a lot of other libs (cryptodev, ethdev, acl, lpm, etc.).
> Why only these 2 dependencies require a special check?
Radu did this change in 1b028d5e81 (examples/ipsec-secgw: fix build 
without security lib).
probably because the security lib was a new one and in experimental stage.

Re: [PATCH v4 6/9] examples/ipsec-secgw: make app to use ipsec library

[Ananyev, Konstantin]

> >>>>>>>> diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
> >>>>>>>> index 77d8b298f..31f68fee2 100644
> >>>>>>>> --- a/examples/ipsec-secgw/meson.build
> >>>>>>>> +++ b/examples/ipsec-secgw/meson.build
> >>>>>>>> @@ -6,7 +6,7 @@
> >>>>>>>>      # To build this example as a standalone application with an already-installed
> >>>>>>>>      # DPDK instance, use 'make'
> >>>>>>>>
> >>>>>>>> -deps += ['security', 'lpm', 'acl', 'hash']
> >>>>>>>> +deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
> >>>>>>>>      allow_experimental_apis = true
> >>>>>>>>      sources = files(
> >>>>>>>>      	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
> >>>>>>>>
> >>>>>>> Makefile should also be updated I guess.
> >>>>>> Anything particular you think is missed?
> >>>>> Will it compile with makefile when IPSEC lib is disabled?
> >>>> Nope, it wouldn't.
> >>>>
> >>> Could you be more specific and describe what particular
> >>> changes in the Makefile you think are necessary?
> >>> Is it a check that librte_ipsec was enabled, like one you have for rte_security:
> >>> ifneq ($(CONFIG_RTE_LIBRTE_SECURITY),y)
> >>> $(error "RTE_LIBRTE_SECURITY is required to build ipsec-secgw")
> >>> endif
> >>> ?
> >> yes
> >>> Something else?
> >>> BTW, why  this check above is needed?
> >> To ensure that user do not compile ipsec-secgw without ipsec/security lib
> > ipsec-secgw depends on a lot of other libs (cryptodev, ethdev, acl, lpm, etc.).
> > Why only these 2 dependencies require a special check?
> Radu did this change in 1b028d5e81 (examples/ipsec-secgw: fix build
> without security lib).
> probably because the security lib was a new one and in experimental stage.

Ok, I'll replace RTE_LIBRTE_SECURITY with RTE_LIBRTE_IPSEC here.
Konstantin

[PATCH v4 7/9] examples/ipsec-secgw: make data-path to use ipsec library

[Konstantin Ananyev]

Changes to make ipsec-secgw data-path code to utilize librte_ipsec library.
Note that right now by default current (non-librte_ipsec) code-path will
be used. User has to run application with new command-line option ('-l')
to enable new codepath.

Signed-off-by: Mohammad Abdul Awal <mohammad.abdul.awal@intel.com>
Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/Makefile        |   1 +
 examples/ipsec-secgw/ipsec-secgw.c   | 178 ++++++++------
 examples/ipsec-secgw/ipsec.c         |   2 +-
 examples/ipsec-secgw/ipsec.h         |  23 ++
 examples/ipsec-secgw/ipsec_process.c | 341 +++++++++++++++++++++++++++
 examples/ipsec-secgw/meson.build     |   4 +-
 examples/ipsec-secgw/sa.c            |   7 +-
 7 files changed, 477 insertions(+), 79 deletions(-)
 create mode 100644 examples/ipsec-secgw/ipsec_process.c

diff --git a/examples/ipsec-secgw/Makefile b/examples/ipsec-secgw/Makefile
index 02d41e39a..4d6b4cfa4 100644
--- a/examples/ipsec-secgw/Makefile
+++ b/examples/ipsec-secgw/Makefile
@@ -13,6 +13,7 @@ SRCS-y += sp4.c
 SRCS-y += sp6.c
 SRCS-y += sa.c
 SRCS-y += rt.c
+SRCS-y += ipsec_process.c
 SRCS-y += ipsec-secgw.c
 
 CFLAGS += -gdwarf-2
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 48baa5001..53cfe585d 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -216,19 +216,6 @@ static struct rte_eth_conf port_conf = {
 
 static struct socket_ctx socket_ctx[NB_SOCKETS];
 
-struct traffic_type {
-	const uint8_t *data[MAX_PKT_BURST * 2];
-	struct rte_mbuf *pkts[MAX_PKT_BURST * 2];
-	uint32_t res[MAX_PKT_BURST * 2];
-	uint32_t num;
-};
-
-struct ipsec_traffic {
-	struct traffic_type ipsec;
-	struct traffic_type ip4;
-	struct traffic_type ip6;
-};
-
 static inline void
 prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 {
@@ -245,6 +232,8 @@ prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 			t->ip4.data[t->ip4.num] = nlp;
 			t->ip4.pkts[(t->ip4.num)++] = pkt;
 		}
+		pkt->l2_len = 0;
+		pkt->l3_len = sizeof(struct ip);
 	} else if (eth->ether_type == rte_cpu_to_be_16(ETHER_TYPE_IPv6)) {
 		nlp = (uint8_t *)rte_pktmbuf_adj(pkt, ETHER_HDR_LEN);
 		nlp = RTE_PTR_ADD(nlp, offsetof(struct ip6_hdr, ip6_nxt));
@@ -254,6 +243,8 @@ prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 			t->ip6.data[t->ip6.num] = nlp;
 			t->ip6.pkts[(t->ip6.num)++] = pkt;
 		}
+		pkt->l2_len = 0;
+		pkt->l3_len = sizeof(struct ip6_hdr);
 	} else {
 		/* Unknown/Unsupported type, drop the packet */
 		RTE_LOG(ERR, IPSEC, "Unsupported packet type\n");
@@ -503,10 +494,15 @@ process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
 	n_ip4 = traffic->ip4.num;
 	n_ip6 = traffic->ip6.num;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
-
-	split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+	if (app_sa_prm.enable == 0) {
+		nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.num, MAX_PKT_BURST);
+		split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+	} else {
+		inbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
+	}
 
 	inbound_sp_sa(ipsec_ctx->sp4_ctx, ipsec_ctx->sa_ctx, &traffic->ip4,
 			n_ip4);
@@ -562,20 +558,27 @@ process_pkts_outbound(struct ipsec_ctx *ipsec_ctx,
 
 	outbound_sp(ipsec_ctx->sp6_ctx, &traffic->ip6, &traffic->ipsec);
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.res, traffic->ipsec.num,
-			MAX_PKT_BURST);
-
-	for (i = 0; i < nb_pkts_out; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
-		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-		} else {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
+	if (app_sa_prm.enable == 0) {
+
+		nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.res, traffic->ipsec.num,
+				MAX_PKT_BURST);
+
+		for (i = 0; i < nb_pkts_out; i++) {
+			m = traffic->ipsec.pkts[i];
+			struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
+			if (ip->ip_v == IPVERSION) {
+				idx = traffic->ip4.num++;
+				traffic->ip4.pkts[idx] = m;
+			} else {
+				idx = traffic->ip6.num++;
+				traffic->ip6.pkts[idx] = m;
+			}
 		}
+	} else {
+		outbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.res,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -598,19 +601,26 @@ process_pkts_inbound_nosp(struct ipsec_ctx *ipsec_ctx,
 
 	traffic->ip6.num = 0;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
+	if (app_sa_prm.enable == 0) {
 
-	for (i = 0; i < nb_pkts_in; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
-		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-		} else {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
+		nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.num, MAX_PKT_BURST);
+
+		for (i = 0; i < nb_pkts_in; i++) {
+			m = traffic->ipsec.pkts[i];
+			struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
+			if (ip->ip_v == IPVERSION) {
+				idx = traffic->ip4.num++;
+				traffic->ip4.pkts[idx] = m;
+			} else {
+				idx = traffic->ip6.num++;
+				traffic->ip6.pkts[idx] = m;
+			}
 		}
+	} else {
+		inbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -642,20 +652,28 @@ process_pkts_outbound_nosp(struct ipsec_ctx *ipsec_ctx,
 	traffic->ip6.num = 0;
 	traffic->ipsec.num = n;
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.res, traffic->ipsec.num,
-			MAX_PKT_BURST);
+	if (app_sa_prm.enable == 0) {
 
-	/* They all sue the same SA (ip4 or ip6 tunnel) */
-	m = traffic->ipsec.pkts[i];
-	ip = rte_pktmbuf_mtod(m, struct ip *);
-	if (ip->ip_v == IPVERSION) {
-		traffic->ip4.num = nb_pkts_out;
-		for (i = 0; i < nb_pkts_out; i++)
-			traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+		nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.res, traffic->ipsec.num,
+				MAX_PKT_BURST);
+
+		/* They all sue the same SA (ip4 or ip6 tunnel) */
+		m = traffic->ipsec.pkts[0];
+		ip = rte_pktmbuf_mtod(m, struct ip *);
+		if (ip->ip_v == IPVERSION) {
+			traffic->ip4.num = nb_pkts_out;
+			for (i = 0; i < nb_pkts_out; i++)
+				traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+		} else {
+			traffic->ip6.num = nb_pkts_out;
+			for (i = 0; i < nb_pkts_out; i++)
+				traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+		}
 	} else {
-		traffic->ip6.num = nb_pkts_out;
-		traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+		outbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.res,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -856,25 +874,31 @@ drain_inbound_crypto_queues(const struct lcore_conf *qconf,
 	uint32_t n;
 	struct ipsec_traffic trf;
 
-	/* dequeue packets from crypto-queue */
-	n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+	if (app_sa_prm.enable == 0) {
+
+		/* dequeue packets from crypto-queue */
+		n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
 			RTE_DIM(trf.ipsec.pkts));
-	if (n == 0)
-		return;
 
-	trf.ip4.num = 0;
-	trf.ip6.num = 0;
+		trf.ip4.num = 0;
+		trf.ip6.num = 0;
 
-	/* split traffic by ipv4-ipv6 */
-	split46_traffic(&trf, trf.ipsec.pkts, n);
+		/* split traffic by ipv4-ipv6 */
+		split46_traffic(&trf, trf.ipsec.pkts, n);
+	} else
+		ipsec_cqp_process(ctx, &trf);
 
 	/* process ipv4 packets */
-	inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
-	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	if (trf.ip4.num != 0) {
+		inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
+		route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	}
 
 	/* process ipv6 packets */
-	inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
-	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	if (trf.ip6.num != 0) {
+		inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
+		route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	}
 }
 
 static void
@@ -884,23 +908,27 @@ drain_outbound_crypto_queues(const struct lcore_conf *qconf,
 	uint32_t n;
 	struct ipsec_traffic trf;
 
-	/* dequeue packets from crypto-queue */
-	n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+	if (app_sa_prm.enable == 0) {
+
+		/* dequeue packets from crypto-queue */
+		n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
 			RTE_DIM(trf.ipsec.pkts));
-	if (n == 0)
-		return;
 
-	trf.ip4.num = 0;
-	trf.ip6.num = 0;
+		trf.ip4.num = 0;
+		trf.ip6.num = 0;
 
-	/* split traffic by ipv4-ipv6 */
-	split46_traffic(&trf, trf.ipsec.pkts, n);
+		/* split traffic by ipv4-ipv6 */
+		split46_traffic(&trf, trf.ipsec.pkts, n);
+	} else
+		ipsec_cqp_process(ctx, &trf);
 
 	/* process ipv4 packets */
-	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	if (trf.ip4.num != 0)
+		route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
 
 	/* process ipv6 packets */
-	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	if (trf.ip6.num != 0)
+		route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
 }
 
 /* main processing loop */
diff --git a/examples/ipsec-secgw/ipsec.c b/examples/ipsec-secgw/ipsec.c
index 8bf928a23..8e7fc3639 100644
--- a/examples/ipsec-secgw/ipsec.c
+++ b/examples/ipsec-secgw/ipsec.c
@@ -39,7 +39,7 @@ set_ipsec_conf(struct ipsec_sa *sa, struct rte_security_ipsec_xform *ipsec)
 	ipsec->esn_soft_limit = IPSEC_OFFLOAD_ESN_SOFTLIMIT;
 }
 
-static inline int
+int
 create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa)
 {
 	struct rte_cryptodev_info cdev_info;
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index b089fe54b..f0e2e79af 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -190,6 +190,20 @@ struct cnt_blk {
 	uint32_t cnt;
 } __attribute__((packed));
 
+struct traffic_type {
+	const uint8_t *data[MAX_PKT_BURST * 2];
+	struct rte_mbuf *pkts[MAX_PKT_BURST * 2];
+	struct ipsec_sa *saptr[MAX_PKT_BURST * 2];
+	uint32_t res[MAX_PKT_BURST * 2];
+	uint32_t num;
+};
+
+struct ipsec_traffic {
+	struct traffic_type ipsec;
+	struct traffic_type ip4;
+	struct traffic_type ip6;
+};
+
 uint16_t
 ipsec_inbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint16_t nb_pkts, uint16_t len);
@@ -206,6 +220,12 @@ uint16_t
 ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint16_t len);
 
+void
+ipsec_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf);
+
+void
+ipsec_cqp_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf);
+
 static inline uint16_t
 ipsec_metadata_size(void)
 {
@@ -283,4 +303,7 @@ add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
 void
 enqueue_cop_burst(struct cdev_qp *cqp);
 
+int
+create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/ipsec_process.c b/examples/ipsec-secgw/ipsec_process.c
new file mode 100644
index 000000000..7ab378f6a
--- /dev/null
+++ b/examples/ipsec-secgw/ipsec_process.c
@@ -0,0 +1,341 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2017 Intel Corporation
+ */
+#include <sys/types.h>
+#include <netinet/in.h>
+#include <netinet/ip.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_log.h>
+#include <rte_cryptodev.h>
+#include <rte_ethdev.h>
+#include <rte_mbuf.h>
+
+#include "ipsec.h"
+
+#define SATP_OUT_IPV4(t)	\
+	((((t) & RTE_IPSEC_SATP_MODE_MASK) == RTE_IPSEC_SATP_MODE_TRANS && \
+	(((t) & RTE_IPSEC_SATP_IPV_MASK) == RTE_IPSEC_SATP_IPV4)) || \
+	((t) & RTE_IPSEC_SATP_MODE_MASK) == RTE_IPSEC_SATP_MODE_TUNLV4)
+
+
+/* helper routine to free bulk of packets */
+static inline void
+free_pkts(struct rte_mbuf *mb[], uint32_t n)
+{
+	uint32_t i;
+
+	for (i = 0; i != n; i++)
+		rte_pktmbuf_free(mb[i]);
+}
+
+/* helper routine to free bulk of crypto-ops and related packets */
+static inline void
+free_cops(struct rte_crypto_op *cop[], uint32_t n)
+{
+	uint32_t i;
+
+	for (i = 0; i != n; i++)
+		rte_pktmbuf_free(cop[i]->sym->m_src);
+}
+
+/* helper routine to enqueue bulk of crypto ops */
+static inline void
+enqueue_cop_bulk(struct cdev_qp *cqp, struct rte_crypto_op *cop[], uint32_t num)
+{
+	uint32_t i, k, len, n;
+
+	len = cqp->len;
+
+	/*
+	 * if cqp is empty and we have enough ops,
+	 * then queue them to the PMD straightway.
+	 */
+	if (num >= RTE_DIM(cqp->buf) * 3 / 4 && len == 0) {
+		n = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp, cop, num);
+		cqp->in_flight += n;
+		free_cops(cop + n, num - n);
+		return;
+	}
+
+	k = 0;
+
+	do {
+		n = RTE_DIM(cqp->buf) - len;
+		n = RTE_MIN(num - k, n);
+
+		/* put packets into cqp */
+		for (i = 0; i != n; i++)
+			cqp->buf[len + i] = cop[k + i];
+
+		len += n;
+		k += n;
+
+		/* if cqp is full then, enqueue crypto-ops to PMD */
+		if (len == RTE_DIM(cqp->buf)) {
+			n = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp,
+					cqp->buf, len);
+			cqp->in_flight += n;
+			free_cops(cqp->buf + n, len - n);
+			len = 0;
+		}
+
+
+	} while (k != num);
+
+	cqp->len = len;
+}
+
+static inline int
+fill_ipsec_session(struct rte_ipsec_session *ss, const struct ipsec_sa *sa)
+{
+	/* setup crypto section */
+	if (ss->type == RTE_SECURITY_ACTION_TYPE_NONE) {
+		ss->crypto.ses = sa->crypto_session;
+	/* setup session action type */
+	} else {
+		ss->security.ses = sa->sec_session;
+		ss->security.ctx = sa->security_ctx;
+		ss->security.ol_flags = sa->ol_flags;
+	}
+
+	return rte_ipsec_session_prepare(ss);
+}
+
+/*
+ * group input packets byt the SA they belong to.
+ */
+static uint32_t
+sa_group(struct ipsec_sa *sa_ptr[], struct rte_mbuf *pkts[],
+	struct rte_ipsec_group grp[], uint32_t num)
+{
+	uint32_t i, n, spi;
+	void *sa;
+	void * const nosa = &spi;
+
+	sa = nosa;
+	for (i = 0, n = 0; i != num; i++) {
+
+		if (sa != sa_ptr[i]) {
+			grp[n].cnt = pkts + i - grp[n].m;
+			n += (sa != nosa);
+			grp[n].id.ptr = sa_ptr[i];
+			grp[n].m = pkts + i;
+			sa = sa_ptr[i];
+		}
+	}
+
+	/* terminate last group */
+	if (sa != nosa) {
+		grp[n].cnt = pkts + i - grp[n].m;
+		n++;
+	}
+
+	return n;
+}
+
+/*
+ * helper function, splits processed packets into ipv4/ipv6 traffic.
+ */
+static inline void
+copy_to_trf(struct ipsec_traffic *trf, uint64_t satp, struct rte_mbuf *mb[],
+	uint32_t num)
+{
+	uint32_t j, ofs, s;
+	struct traffic_type *out;
+
+	/*
+	 * determine traffic type(ipv4/ipv6) and offset for ACL classify
+	 * based on SA type
+	 */
+	if ((satp & RTE_IPSEC_SATP_DIR_MASK) == RTE_IPSEC_SATP_DIR_IB) {
+		if ((satp & RTE_IPSEC_SATP_IPV_MASK) == RTE_IPSEC_SATP_IPV4) {
+			out = &trf->ip4;
+			ofs = offsetof(struct ip, ip_p);
+		} else {
+			out = &trf->ip6;
+			ofs = offsetof(struct ip6_hdr, ip6_nxt);
+		}
+	} else if (SATP_OUT_IPV4(satp)) {
+		out = &trf->ip4;
+		ofs = offsetof(struct ip, ip_p);
+	} else {
+		out = &trf->ip6;
+		ofs = offsetof(struct ip6_hdr, ip6_nxt);
+	}
+
+	for (j = 0, s = out->num; j != num; j++) {
+		out->data[s + j] = rte_pktmbuf_mtod_offset(mb[j],
+				void *, ofs);
+		out->pkts[s + j] = mb[j];
+	}
+
+	out->num += num;
+}
+
+/*
+ * Process ipsec packets.
+ * If packet belong to SA that is subject of inline-crypto,
+ * then process it immediately.
+ * Otherwise do necessary preparations and queue it to related
+ * crypto-dev queue.
+ */
+void
+ipsec_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf)
+{
+	uint64_t satp;
+	uint32_t i, j, k, n;
+	struct ipsec_sa *sa;
+	struct ipsec_mbuf_metadata *priv;
+	struct rte_ipsec_group *pg;
+	struct rte_ipsec_session *ips;
+	struct cdev_qp *cqp;
+	struct rte_crypto_op *cop[RTE_DIM(trf->ipsec.pkts)];
+	struct rte_ipsec_group grp[RTE_DIM(trf->ipsec.pkts)];
+
+	n = sa_group(trf->ipsec.saptr, trf->ipsec.pkts, grp, trf->ipsec.num);
+
+	for (i = 0; i != n; i++) {
+
+		pg = grp + i;
+		sa = pg->id.ptr;
+
+		/* no valid SA found */
+		if (sa == NULL)
+			k = 0;
+
+		ips = &sa->ips;
+		satp = rte_ipsec_sa_type(ips->sa);
+
+		/* no valid HW session for that SA, try to create one */
+		if (ips->crypto.ses == NULL &&
+				(create_session(ctx, sa) != 0 ||
+				fill_ipsec_session(ips, sa) != 0))
+			k = 0;
+
+		/* process packets inline */
+		else if (sa->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				sa->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL) {
+
+			/*
+			 * This is just to satisfy inbound_sa_check()
+			 * and get_hop_for_offload_pkt().
+			 * Should be removed in future.
+			 */
+			for (j = 0; j != pg->cnt; j++) {
+				priv = get_priv(pg->m[j]);
+				priv->sa = sa;
+			}
+
+			k = rte_ipsec_pkt_process(ips, pg->m, pg->cnt);
+			copy_to_trf(trf, satp, pg->m, k);
+
+		/* enqueue packets to crypto dev */
+		} else {
+
+			cqp = &ctx->tbl[sa->cdev_id_qp];
+
+			/* for that app each mbuf has it's own crypto op */
+			for (j = 0; j != pg->cnt; j++) {
+				priv = get_priv(pg->m[j]);
+				cop[j] = &priv->cop;
+				/*
+				 * this is just to satisfy inbound_sa_check()
+				 * should be removed in future.
+				 */
+				priv->sa = sa;
+			}
+
+			/* prepare and enqueue crypto ops */
+			k = rte_ipsec_pkt_crypto_prepare(ips, pg->m, cop,
+				pg->cnt);
+			if (k != 0)
+				enqueue_cop_bulk(cqp, cop, k);
+		}
+
+		/* drop packets that cannot be enqueued/processed */
+		if (k != pg->cnt)
+			free_pkts(pg->m + k, pg->cnt - k);
+	}
+}
+
+static inline uint32_t
+cqp_dequeue(struct cdev_qp *cqp, struct rte_crypto_op *cop[], uint32_t num)
+{
+	uint32_t n;
+
+	if (cqp->in_flight == 0)
+		return 0;
+
+	n = rte_cryptodev_dequeue_burst(cqp->id, cqp->qp, cop, num);
+	RTE_ASSERT(cqp->in_flight >= n);
+	cqp->in_flight -= n;
+
+	return n;
+}
+
+static inline uint32_t
+ctx_dequeue(struct ipsec_ctx *ctx, struct rte_crypto_op *cop[], uint32_t num)
+{
+	uint32_t i, n;
+
+	n = 0;
+
+	for (i = ctx->last_qp; n != num && i != ctx->nb_qps; i++)
+		n += cqp_dequeue(ctx->tbl + i, cop + n, num - n);
+
+	for (i = 0; n != num && i != ctx->last_qp; i++)
+		n += cqp_dequeue(ctx->tbl + i, cop + n, num - n);
+
+	ctx->last_qp = i;
+	return n;
+}
+
+/*
+ * dequeue packets from crypto-queues and finalize processing.
+ */
+void
+ipsec_cqp_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf)
+{
+	uint64_t satp;
+	uint32_t i, k, n, ng;
+	struct rte_ipsec_session *ss;
+	struct traffic_type *out;
+	struct rte_ipsec_group *pg;
+	struct rte_crypto_op *cop[RTE_DIM(trf->ipsec.pkts)];
+	struct rte_ipsec_group grp[RTE_DIM(trf->ipsec.pkts)];
+
+	trf->ip4.num = 0;
+	trf->ip6.num = 0;
+
+	out = &trf->ipsec;
+
+	/* dequeue completed crypto-ops */
+	n = ctx_dequeue(ctx, cop, RTE_DIM(cop));
+	if (n == 0)
+		return;
+
+	/* group them by ipsec session */
+	ng = rte_ipsec_pkt_crypto_group((const struct rte_crypto_op **)
+		(uintptr_t)cop, out->pkts, grp, n);
+
+	/* process each group of packets */
+	for (i = 0; i != ng; i++) {
+
+		pg = grp + i;
+		ss = pg->id.ptr;
+		satp = rte_ipsec_sa_type(ss->sa);
+
+		k = rte_ipsec_pkt_process(ss, pg->m, pg->cnt);
+		copy_to_trf(trf, satp, pg->m, k);
+
+		/* free bad packets, if any */
+		free_pkts(pg->m + k, pg->cnt - k);
+
+		n -= pg->cnt;
+	}
+
+	/* we should never have packet with unknown SA here */
+	RTE_VERIFY(n == 0);
+}
diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
index 31f68fee2..81c146ebc 100644
--- a/examples/ipsec-secgw/meson.build
+++ b/examples/ipsec-secgw/meson.build
@@ -9,6 +9,6 @@
 deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
 allow_experimental_apis = true
 sources = files(
-	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
-	'rt.c', 'sa.c', 'sp4.c', 'sp6.c'
+	'esp.c', 'ipsec.c', 'ipsec_process.c', 'ipsec-secgw.c',
+	'parser.c', 'rt.c', 'sa.c', 'sp4.c', 'sp6.c'
 )
diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index 04eb8bce8..49da966a3 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -1164,10 +1164,15 @@ int
 inbound_sa_check(struct sa_ctx *sa_ctx, struct rte_mbuf *m, uint32_t sa_idx)
 {
 	struct ipsec_mbuf_metadata *priv;
+	struct ipsec_sa *sa;
 
 	priv = get_priv(m);
+	sa = priv->sa;
+	if (sa != NULL)
+		return (sa_ctx->sa[sa_idx].spi == sa->spi);
 
-	return (sa_ctx->sa[sa_idx].spi == priv->sa->spi);
+	RTE_LOG(ERR, IPSEC, "SA not saved in private data\n");
+	return 0;
 }
 
 static inline void
-- 
2.17.1

Re: [PATCH v4 7/9] examples/ipsec-secgw: make data-path to use ipsec library

[Akhil Goyal]

On 12/14/2018 10:10 PM, Konstantin Ananyev wrote:
> diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
> index 04eb8bce8..49da966a3 100644
> --- a/examples/ipsec-secgw/sa.c
> +++ b/examples/ipsec-secgw/sa.c
> @@ -1164,10 +1164,15 @@ int
>   inbound_sa_check(struct sa_ctx *sa_ctx, struct rte_mbuf *m, uint32_t sa_idx)
>   {
>   	struct ipsec_mbuf_metadata *priv;
> +	struct ipsec_sa *sa;
>   
>   	priv = get_priv(m);
> +	sa = priv->sa;
> +	if (sa != NULL)
> +		return (sa_ctx->sa[sa_idx].spi == sa->spi);
>   
> -	return (sa_ctx->sa[sa_idx].spi == priv->sa->spi);
> +	RTE_LOG(ERR, IPSEC, "SA not saved in private data\n");
> +	return 0;
>   }
I believe this is a fix and not related to this patchset. Should go as a 
fix and CC to stable.

[PATCH v4 8/9] examples/ipsec-secgw: add scripts for functional test

[Konstantin Ananyev]

The purpose of these scripts is to automate ipsec-secgw functional testing.
The scripts require two machines (SUT and DUT) connected through
at least 2 NICs and running linux (so far tested only on Ubuntu 18.04).
Introduced test-cases for the following scenarios:
- Transport/Tunnel modes
- AES-CBC SHA1
- AES-GCM
- ESN on/off
- legacy/librte_ipsec code path

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Acked-by: Radu Nicolau <radu.nicolau@intel.com>
---
 examples/ipsec-secgw/test/common_defs.sh      | 153 ++++++++++++++++++
 examples/ipsec-secgw/test/data_rxtx.sh        |  62 +++++++
 examples/ipsec-secgw/test/linux_test4.sh      |  63 ++++++++
 examples/ipsec-secgw/test/linux_test6.sh      |  64 ++++++++
 examples/ipsec-secgw/test/run_test.sh         |  80 +++++++++
 .../test/trs_aescbc_sha1_common_defs.sh       |  69 ++++++++
 .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  67 ++++++++
 .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/trs_aescbc_sha1_esn_defs.sh          |  66 ++++++++
 .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
 .../test/trs_aesgcm_common_defs.sh            |  60 +++++++
 examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  66 ++++++++
 .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  66 ++++++++
 .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
 .../test/tun_aescbc_sha1_common_defs.sh       |  68 ++++++++
 .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  70 ++++++++
 .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/tun_aescbc_sha1_esn_defs.sh          |  70 ++++++++
 .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
 .../test/tun_aesgcm_common_defs.sh            |  60 +++++++
 examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  70 ++++++++
 .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  70 ++++++++
 .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
 25 files changed, 1264 insertions(+)
 create mode 100644 examples/ipsec-secgw/test/common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test4.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test6.sh
 create mode 100644 examples/ipsec-secgw/test/run_test.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh

diff --git a/examples/ipsec-secgw/test/common_defs.sh b/examples/ipsec-secgw/test/common_defs.sh
new file mode 100644
index 000000000..7adfffa19
--- /dev/null
+++ b/examples/ipsec-secgw/test/common_defs.sh
@@ -0,0 +1,153 @@
+#! /bin/bash
+
+#check that env vars are properly defined
+
+#check SGW_PATH
+if [[ -z "${SGW_PATH}" || ! -x ${SGW_PATH} ]]; then
+	echo "SGW_PATH is invalid"
+	exit 127
+fi
+
+#check ETH_DEV
+if [[ -z "${ETH_DEV}" ]]; then
+	echo "ETH_DEV is invalid"
+	exit 127
+fi
+
+#setup SGW_LCORE
+SGW_LCORE=${SGW_LCORE:-0}
+
+#check that REMOTE_HOST is reachable
+ssh ${REMOTE_HOST} echo
+st=$?
+if [[ $st -ne 0 ]]; then
+	echo "host ${REMOTE_HOST} is not reachable"
+	exit $st
+fi
+
+#get ether addr of REMOTE_HOST
+REMOTE_MAC=`ssh ${REMOTE_HOST} ip addr show dev ${REMOTE_IFACE}`
+st=$?
+REMOTE_MAC=`echo ${REMOTE_MAC} | sed -e 's/^.*ether //' -e 's/ brd.*$//'`
+if [[ $st -ne 0 || -z "${REMOTE_MAC}" ]]; then
+	echo "coouldn't retrieve ether addr from ${REMOTE_IFACE}"
+	exit 127
+fi
+
+LOCAL_IFACE=dtap0
+
+LOCAL_MAC="00:64:74:61:70:30"
+
+REMOTE_IPV4=192.168.31.14
+LOCAL_IPV4=192.168.31.92
+
+REMOTE_IPV6=fd12:3456:789a:0031:0000:0000:0000:0014
+LOCAL_IPV6=fd12:3456:789a:0031:0000:0000:0000:0092
+
+DPDK_PATH=${RTE_SDK:-${PWD}}
+DPDK_BUILD=${RTE_TARGET:-x86_64-native-linuxapp-gcc}
+
+SGW_OUT_FILE=./ipsec-secgw.out1
+
+SGW_CMD_EAL_PRM="--lcores=${SGW_LCORE} -n 4 ${ETH_DEV}"
+SGW_CMD_CFG="(0,0,${SGW_LCORE}),(1,0,${SGW_LCORE})"
+SGW_CMD_PRM="-p 0x3 -u 1 -P --config=\"${SGW_CMD_CFG}\""
+
+SGW_CFG_FILE=$(tempfile)
+
+# configure local host/ifaces
+config_local_iface()
+{
+	ifconfig ${LOCAL_IFACE} ${LOCAL_IPV4}/24 mtu 1400 up
+	ifconfig ${LOCAL_IFACE}
+
+	ip neigh flush dev ${LOCAL_IFACE}
+	ip neigh add ${REMOTE_IPV4} dev ${LOCAL_IFACE} lladdr ${REMOTE_MAC}
+	ip neigh show dev ${LOCAL_IFACE}
+}
+
+config6_local_iface()
+{
+	config_local_iface
+
+	sysctl -w net.ipv6.conf.${LOCAL_IFACE}.disable_ipv6=0
+	ip addr add  ${LOCAL_IPV6}/64 dev ${LOCAL_IFACE}
+
+	sysctl -w net.ipv6.conf.${LOCAL_IFACE}.mtu=1300
+
+	ip -6 neigh add ${REMOTE_IPV6} dev ${LOCAL_IFACE} lladdr ${REMOTE_MAC}
+	ip neigh show dev ${LOCAL_IFACE}
+}
+
+#configure remote host/iface
+config_remote_iface()
+{
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE} down
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE} ${REMOTE_IPV4}/24 up
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} ip neigh flush dev ${REMOTE_IFACE}
+
+	# by some reason following ip neigh doesn't work for me here properly:
+	#ssh ${REMOTE_HOST} ip neigh add ${LOCAL_IPV4} \
+	#		dev ${REMOTE_IFACE} lladr ${LOCAL_MAC}
+	# so used arp instead.
+	ssh ${REMOTE_HOST} arp -i ${REMOTE_IFACE} -s ${LOCAL_IPV4} ${LOCAL_MAC}
+	ssh ${REMOTE_HOST} ip neigh show dev ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} iptables --flush
+}
+
+config6_remote_iface()
+{
+	config_remote_iface
+
+	ssh ${REMOTE_HOST} sysctl -w \
+		net.ipv6.conf.${REMOTE_IFACE}.disable_ipv6=0
+	ssh ${REMOTE_HOST} ip addr add  ${REMOTE_IPV6}/64 dev ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} ip -6 neigh add ${LOCAL_IPV6} \
+		dev ${REMOTE_IFACE} lladdr ${LOCAL_MAC}
+	ssh ${REMOTE_HOST} ip neigh show dev ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} ip6tables --flush
+}
+
+#configure remote and local host/iface
+config_iface()
+{
+	config_local_iface
+	config_remote_iface
+}
+
+config6_iface()
+{
+	config6_local_iface
+	config6_remote_iface
+}
+
+#start ipsec-secgw
+secgw_start()
+{
+	SGW_EXEC_FILE=$(tempfile)
+	cat <<EOF > ${SGW_EXEC_FILE}
+${SGW_PATH} ${SGW_CMD_EAL_PRM} ${CRYPTO_DEV} \
+--vdev="net_tap0,mac=fixed" \
+-- ${SGW_CMD_PRM} ${SGW_CMD_XPRM} -f ${SGW_CFG_FILE} > \
+${SGW_OUT_FILE} 2>&1 &
+p=\$!
+echo \$p
+EOF
+
+	cat ${SGW_EXEC_FILE}
+	SGW_PID=`/bin/bash -x ${SGW_EXEC_FILE}`
+	sleep 1
+}
+
+#stop ipsec-secgw and cleanup
+secgw_stop()
+{
+	kill ${SGW_PID}
+	rm -f ${SGW_EXEC_FILE}
+	rm -f ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/data_rxtx.sh b/examples/ipsec-secgw/test/data_rxtx.sh
new file mode 100644
index 000000000..f23a6d594
--- /dev/null
+++ b/examples/ipsec-secgw/test/data_rxtx.sh
@@ -0,0 +1,62 @@
+#! /bin/bash
+
+TCP_PORT=22222
+
+ping_test1()
+{
+	dst=$1
+
+	i=0
+	st=0
+	while [[ $i -ne 1200 && $st -eq 0 ]];
+	do
+		let i++
+		ping -c 1 -s ${i} ${dst}
+		st=$?
+	done
+
+	if [[ $st -ne 0 ]]; then
+		echo "ERROR: $0 failed for dst=${dst}, sz=${i}"
+	fi
+	return $st;
+}
+
+ping6_test1()
+{
+	dst=$1
+
+	i=0
+	st=0
+	while [[ $i -ne 1200 && $st -eq 0 ]];
+	do
+		let i++
+		ping6 -c 1 -s ${i} ${dst}
+		st=$?
+	done
+
+	if [[ $st -ne 0 ]]; then
+		echo "ERROR: $0 failed for dst=${dst}, sz=${i}"
+	fi
+	return $st;
+}
+
+scp_test1()
+{
+	dst=$1
+
+	for sz in 1234 23456 345678 4567890 56789102 ; do
+		x=`basename $0`.${sz}
+		dd if=/dev/urandom of=${x} bs=${sz} count=1
+		scp ${x} [${dst}]:${x}
+		scp [${dst}]:${x} ${x}.copy1
+		diff -u ${x} ${x}.copy1
+		st=$?
+		rm -f ${x} ${x}.copy1
+		ssh ${REMOTE_HOST} rm -f ${x}
+		if [[ $st -ne 0 ]]; then
+			return $st
+		fi
+	done
+
+	return 0;
+}
diff --git a/examples/ipsec-secgw/test/linux_test4.sh b/examples/ipsec-secgw/test/linux_test4.sh
new file mode 100644
index 000000000..d636f5604
--- /dev/null
+++ b/examples/ipsec-secgw/test/linux_test4.sh
@@ -0,0 +1,63 @@
+#! /bin/bash
+
+# usage:  /bin/bash linux_test4.sh <ipsec_mode>
+# for list of available modes please refer to run_test.sh.
+# ipsec-secgw (IPv4 mode) functional test script.
+#
+# Note that for most of them you required appropriate crypto PMD/device
+# to be avaialble.
+# Also user has to setup properly the following environment variables:
+#  SGW_PATH - path to the ipsec-secgw binary to test
+#  REMOTE_HOST - ip/hostname of the DUT
+#  REMOTE_IFACE - iface name for the test-port on DUT
+#  ETH_DEV - ethernet device to be used on SUT by DPDK ('-w <pci-id>')
+# Also user can optonally setup:
+#  SGW_LCORE - lcore to run ipsec-secgw on (default value is 0)
+#  CRYPTO_DEV - crypto device to be used ('-w <pci-id>')
+#  if none specified appropriate vdevs will be created by the scrit
+#
+# The purpose of the script is to automate ipsec-secgw testing
+# using another system running linux as a DUT.
+# It expects that SUT and DUT are connected through at least 2 NICs.
+# One NIC is expected to be managed by linux both machines,
+# and will be used as a control path
+# Make sure user from SUT can ssh to DUT without entering password.
+# Second NIC (test-port) should be reserved for DPDK on SUT,
+# and should be managed by linux on DUT.
+# The script starts ipsec-secgw with 2 NIC devices: test-port and tap vdev.
+# Then configures local tap iface and remote iface and ipsec policies
+# in the following way:
+# traffic going over test-port in both directions has to be
+# protected by ipsec.
+# raffic going over TAP in both directions doesn't have to be protected.
+# I.E:
+# DUT OS(NIC1)--(ipsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS
+# SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(ipsec)-->(NIC1)DUT OS
+# Then tries to perorm some data transfer using the scheme decribed above.
+#
+
+DIR=`dirname $0`
+MODE=$1
+
+ . ${DIR}/common_defs.sh
+ . ${DIR}/${MODE}_defs.sh
+
+config_secgw
+
+secgw_start
+
+config_iface
+
+config_remote_xfrm
+
+ . ${DIR}/data_rxtx.sh
+
+ping_test1 ${REMOTE_IPV4}
+st=$?
+if [[ $st -eq 0 ]]; then
+	scp_test1 ${REMOTE_IPV4}
+	st=$?
+fi
+
+secgw_stop
+exit $st
diff --git a/examples/ipsec-secgw/test/linux_test6.sh b/examples/ipsec-secgw/test/linux_test6.sh
new file mode 100644
index 000000000..e30f607d8
--- /dev/null
+++ b/examples/ipsec-secgw/test/linux_test6.sh
@@ -0,0 +1,64 @@
+#! /bin/bash
+
+# usage:  /bin/bash linux_test6.sh <ipsec_mode>
+# for list of available modes please refer to run_test.sh.
+# ipsec-secgw (IPv6 mode) functional test script.
+#
+# Note that for most of them you required appropriate crypto PMD/device
+# to be avaialble.
+# Also user has to setup properly the following environment variables:
+#  SGW_PATH - path to the ipsec-secgw binary to test
+#  REMOTE_HOST - ip/hostname of the DUT
+#  REMOTE_IFACE - iface name for the test-port on DUT
+#  ETH_DEV - ethernet device to be used on SUT by DPDK ('-w <pci-id>')
+# Also user can optonally setup:
+#  SGW_LCORE - lcore to run ipsec-secgw on (default value is 0)
+#  CRYPTO_DEV - crypto device to be used ('-w <pci-id>')
+#  if none specified appropriate vdevs will be created by the scrit
+#
+# The purpose of the script is to automate ipsec-secgw testing
+# using another system running linux as a DUT.
+# It expects that SUT and DUT are connected through at least 2 NICs.
+# One NIC is expected to be managed by linux both machines,
+# and will be used as a control path.
+# Make sure user from SUT can ssh to DUT without entering password,
+# also make sure that sshd over ipv6 is enabled.
+# Second NIC (test-port) should be reserved for DPDK on SUT,
+# and should be managed by linux on DUT.
+# The script starts ipsec-secgw with 2 NIC devices: test-port and tap vdev.
+# Then configures local tap iface and remote iface and ipsec policies
+# in the following way:
+# traffic going over test-port in both directions has to be
+# protected by ipsec.
+# raffic going over TAP in both directions doesn't have to be protected.
+# I.E:
+# DUT OS(NIC1)--(ipsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS
+# SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(ipsec)-->(NIC1)DUT OS
+# Then tries to perorm some data transfer using the scheme decribed above.
+#
+
+DIR=`dirname $0`
+MODE=$1
+
+ . ${DIR}/common_defs.sh
+ . ${DIR}/${MODE}_defs.sh
+
+config_secgw
+
+secgw_start
+
+config6_iface
+
+config6_remote_xfrm
+
+ . ${DIR}/data_rxtx.sh
+
+ping6_test1 ${REMOTE_IPV6}
+st=$?
+if [[ $st -eq 0 ]]; then
+	scp_test1 ${REMOTE_IPV6}
+	st=$?
+fi
+
+secgw_stop
+exit $st
diff --git a/examples/ipsec-secgw/test/run_test.sh b/examples/ipsec-secgw/test/run_test.sh
new file mode 100644
index 000000000..6dc0ce54e
--- /dev/null
+++ b/examples/ipsec-secgw/test/run_test.sh
@@ -0,0 +1,80 @@
+#! /bin/bash
+
+# usage: /bin/bash run_test.sh [-46]
+# Run all defined linux_test[4,6].sh test-cases one by one
+# user has to setup properly the following environment variables:
+#  SGW_PATH - path to the ipsec-secgw binary to test
+#  REMOTE_HOST - ip/hostname of the DUT
+#  REMOTE_IFACE - iface name for the test-port on DUT
+#  ETH_DEV - ethernet device to be used on SUT by DPDK ('-w <pci-id>')
+# Also user can optonally setup:
+#  SGW_LCORE - lcore to run ipsec-secgw on (default value is 0)
+#  CRYPTO_DEV - crypto device to be used ('-w <pci-id>')
+#  if none specified appropriate vdevs will be created by the scrit
+# refer to linux_test1.sh for more information
+
+# All supported modes to test.
+# naming convention:
+# 'old' means that ipsec-secgw will run in legacy (non-librte_ipsec mode)
+# 'tun/trs' refer to tunnel/transport mode respectively
+LINUX_TEST="tun_aescbc_sha1 \
+tun_aescbc_sha1_esn \
+tun_aescbc_sha1_esn_atom \
+tun_aesgcm \
+tun_aesgcm_esn \
+tun_aesgcm_esn_atom \
+trs_aescbc_sha1 \
+trs_aescbc_sha1_esn \
+trs_aescbc_sha1_esn_atom \
+trs_aesgcm \
+trs_aesgcm_esn \
+trs_aesgcm_esn_atom \
+tun_aescbc_sha1_old \
+tun_aesgcm_old \
+trs_aescbc_sha1_old \
+trs_aesgcm_old"
+
+DIR=`dirname $0`
+
+# get input options
+st=0
+run4=0
+run6=0
+while [[ ${st} -eq 0 ]]; do
+	getopts ":46" opt
+	st=$?
+	if [[ "${opt}" == "4" ]]; then
+		run4=1
+	elif [[ "${opt}" == "6" ]]; then
+		run6=1
+	fi
+done
+
+if [[ ${run4} -eq 0 && {run6} -eq 0 ]]; then
+	exit 127
+fi
+
+for i in ${LINUX_TEST}; do
+
+	echo "starting test ${i}"
+
+	st4=0
+	if [[ ${run4} -ne 0 ]]; then
+		/bin/bash ${DIR}/linux_test4.sh ${i}
+		st4=$?
+		echo "test4 ${i} finished with status ${st4}"
+	fi
+
+	st6=0
+	if [[ ${run6} -ne 0 ]]; then
+		/bin/bash ${DIR}/linux_test6.sh ${i}
+		st6=$?
+		echo "test6 ${i} finished with status ${st6}"
+	fi
+
+	let "st = st4 + st6"
+	if [[ $st -ne 0 ]]; then
+		echo "ERROR test ${i} FAILED"
+		exit $st
+	fi
+done
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
new file mode 100644
index 000000000..e2621e0df
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
@@ -0,0 +1,69 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_mb0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#SP in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#sp in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+sa in 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#SA out rules
+sa out 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#SA out rules
+sa out 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
new file mode 100644
index 000000000..d68552fce
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
@@ -0,0 +1,67 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
new file mode 100644
index 000000000..f16222e11
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
new file mode 100644
index 000000000..ce7c977a3
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
@@ -0,0 +1,66 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
new file mode 100644
index 000000000..a3abb6103
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
new file mode 100644
index 000000000..720e807e4
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
@@ -0,0 +1,60 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_gcm0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#SP in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+sa in 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#SA out rules
+sa out 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+sa out 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_defs.sh
new file mode 100644
index 000000000..8382d3d52
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_defs.sh
@@ -0,0 +1,66 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
new file mode 100644
index 000000000..80d8d63b8
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
new file mode 100644
index 000000000..94958d199
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
@@ -0,0 +1,66 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
new file mode 100644
index 000000000..951e6b68f
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
new file mode 100644
index 000000000..4025da232
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
@@ -0,0 +1,68 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_mb0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#sp in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#sp in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${REMOTE_IPV4} dst ${LOCAL_IPV4}
+
+sa in 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${REMOTE_IPV6} dst ${LOCAL_IPV6}
+
+#SA out rules
+sa out 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${LOCAL_IPV4} dst ${REMOTE_IPV4}
+
+sa out 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${LOCAL_IPV6} dst ${REMOTE_IPV6}
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
new file mode 100644
index 000000000..18aade3a9
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
new file mode 100644
index 000000000..6b4a82149
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
new file mode 100644
index 000000000..28c1125d6
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
new file mode 100644
index 000000000..3c0d8d1b1
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
new file mode 100644
index 000000000..fba68c6a3
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
@@ -0,0 +1,60 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_gcm0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#sp in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#sp in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${REMOTE_IPV4} dst ${LOCAL_IPV4}
+
+sa in 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${REMOTE_IPV6} dst ${LOCAL_IPV6}
+
+#SA out rules
+sa out 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${LOCAL_IPV4} dst ${REMOTE_IPV4}
+
+sa out 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${LOCAL_IPV6} dst ${REMOTE_IPV6}
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_defs.sh
new file mode 100644
index 000000000..8ae65321b
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_defs.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
new file mode 100644
index 000000000..dab1460c8
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
new file mode 100644
index 000000000..606232349
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh
new file mode 100644
index 000000000..e0a015e21
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_defs.sh
+
+SGW_CMD_XPRM=
-- 
2.17.1

[PATCH v4 9/9] doc: update ipsec-secgw guide and relelase notes

[Konstantin Ananyev]

Update ipsec-secgw guide and relelase notes to reflect latest changes.

Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 doc/guides/rel_notes/release_19_02.rst   |  14 ++++
 doc/guides/sample_app_ug/ipsec_secgw.rst | 101 ++++++++++++++++++++++-
 2 files changed, 113 insertions(+), 2 deletions(-)

diff --git a/doc/guides/rel_notes/release_19_02.rst b/doc/guides/rel_notes/release_19_02.rst
index e88289f73..021034c62 100644
--- a/doc/guides/rel_notes/release_19_02.rst
+++ b/doc/guides/rel_notes/release_19_02.rst
@@ -71,6 +71,20 @@ New Features
 
   See :doc:`../prog_guide/ipsec_lib` for more information.
 
+* **Updated the ipsec-secgw sample application.**
+
+    The ``ipsec-secgw`` sample application has been updated to use the new
+    ``librte_ipsec`` library also added in this release.
+    The original functionality of ipsec-secgw is retained, a new command line
+    parameter ``-l`` has  been added to ipsec-secgw to use the IPsec library,
+    instead of the existing IPsec code in the application.
+
+    The IPsec library does not support all the functionality of the existing
+    ipsec-secgw application, its is planned to add the outstanding functionality
+    in future releases.
+
+    See :doc:`../sample_app_ug/ipsec_secgw` for more information.
+
 Removed Items
 -------------
 
diff --git a/doc/guides/sample_app_ug/ipsec_secgw.rst b/doc/guides/sample_app_ug/ipsec_secgw.rst
index 4869a011d..c159ddac7 100644
--- a/doc/guides/sample_app_ug/ipsec_secgw.rst
+++ b/doc/guides/sample_app_ug/ipsec_secgw.rst
@@ -76,7 +76,7 @@ Compiling the Application
 
 To compile the sample application see :doc:`compiling`.
 
-The application is located in the ``rpsec-secgw`` sub-directory.
+The application is located in the ``ipsec-secgw`` sub-directory.
 
 #. [Optional] Build the application for debugging:
    This option adds some extra flags, disables compiler optimizations and
@@ -112,6 +112,15 @@ Where:
     specified as FRAMESIZE. If an invalid value is provided as FRAMESIZE
     then the default value 9000 is used.
 
+*   ``-l``: enables code-path that uses librte_ipsec.
+
+*   ``-w REPLAY_WINOW_SIZE``: specifies the IPsec sequence number replay window
+    size for each Security Association.
+
+*   ``-e``: enables Security Association extended sequence number processing.
+
+*   ``-a``: enables Security Association sequence number atomic behaviour.
+
 *   ``--config (port,queue,lcore)[,(port,queue,lcore)]``: determines which queues
     from which ports are mapped to which cores.
 
@@ -213,7 +222,7 @@ accordingly.
 
 
 Configuration File Syntax
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~
 
 As mention in the overview, the Security Policies are ACL rules.
 The application parsers the rules specified in the configuration file and
@@ -559,6 +568,11 @@ Example SA rules:
     mode ipv4-tunnel src 172.16.1.5 dst 172.16.2.5 \
     type lookaside-protocol-offload port_id 4
 
+    sa in 35 aead_algo aes-128-gcm \
+    aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+    mode ipv4-tunnel src 172.16.2.5 dst 172.16.1.5 \
+    type inline-crypto-offload port_id 0
+
 Routing rule syntax
 ^^^^^^^^^^^^^^^^^^^
 
@@ -619,3 +633,86 @@ Example SP rules:
     rt ipv4 dst 172.16.1.5/32 port 0
 
     rt ipv6 dst 1111:1111:1111:1111:1111:1111:1111:5555/116 port 0
+
+Test directory
+--------------
+
+The test directory contains scripts for testing the various encryption
+algorithms.
+
+The purpose of the scripts is to automate ipsec-secgw testing
+using another system running linux as a DUT.
+
+The user must setup the following environment variables:
+
+*   ``SGW_PATH``: path to the ipsec-secgw binary to test.
+
+*   ``REMOTE_HOST``: IP address/hostname of the DUT.
+
+*   ``REMOTE_IFACE``: interface name for the test-port on the DUT.
+
+*   ``ETH_DEV``: ethernet device to be used on the SUT by DPDK ('-w <pci-id>')
+
+Also the user can optionally setup:
+
+*   ``SGW_LCORE``: lcore to run ipsec-secgw on (default value is 0)
+
+*   ``CRYPTO_DEV``: crypto device to be used ('-w <pci-id>'). If none specified
+    appropriate vdevs will be created by the script
+
+Note that most of the tests require the appropriate crypto PMD/device to be
+available.
+
+Server configuration
+~~~~~~~~~~~~~~~~~~~~
+
+Two servers are required for the tests, SUT and DUT.
+
+Make sure the user from the SUT can ssh to the DUT without entering the password.
+To enable this feature keys must be setup on the DUT.
+
+``ssh-keygen`` will make a private & public key pair on the SUT.
+
+``ssh-copy-id`` <user name>@<target host name> on the SUT will copy the public
+key to the DUT. It will ask for credentials so that it can upload the public key.
+
+The SUT and DUT are connected through at least 2 NIC ports.
+
+One NIC port is expected to be managed by linux on both machines and will be
+used as a control path.
+
+The second NIC port (test-port) should be bound to DPDK on the SUT, and should
+be managed by linux on the DUT.
+
+The script starts ``ipsec-secgw`` with 2 NIC devices: ``test-port`` and
+``tap vdev``.
+
+It then configures the local tap interface and the remote interface and IPsec
+policies in the following way:
+
+Traffic going over the test-port in both directions has to be protected by IPsec.
+
+Traffic going over the TAP port in both directions does not have to be protected.
+
+i.e:
+
+DUT OS(NIC1)--(IPsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS
+
+SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(IPsec)-->(NIC1)DUT OS
+
+It then tries to perform some data transfer using the scheme decribed above.
+
+usage
+~~~~~
+
+In the ipsec-secgw/test directory
+
+to run one test for IPv4 or IPv6
+
+/bin/bash linux_test(4|6).sh <ipsec_mode>
+
+to run all tests for IPv4 or IPv6
+
+/bin/bash run_test.sh -4|-6
+
+For the list of available modes please refer to run_test.sh.
-- 
2.17.1

[PATCH v3 1/8] examples/ipsec-secgw: avoid to request unused TX offloads

[Konstantin Ananyev]

ipsec-secgw always enables TX offloads
(DEV_TX_OFFLOAD_MULTI_SEGS, DEV_TX_OFFLOAD_SECURITY),
even when they are not requested by the config.
That causes many PMD to choose full-featured TX function,
which in many cases is much slower then one without offloads.
That patch adds checks to enabled extra HW offloads, only when
they were requested.
Plus it enables DEV_TX_OFFLOAD_IPV4_CKSUM,
only when other HW TX ofloads are going to be enabled.
Otherwise SW version of ip cksum calculation is used.
That allows to use vector TX function, when inline-ipsec is not
requested.

Signed-off-by: Remy Horton <remy.horton@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 44 +++++++++++++++--------
 examples/ipsec-secgw/ipsec.h       |  6 ++++
 examples/ipsec-secgw/sa.c          | 56 ++++++++++++++++++++++++++++++
 3 files changed, 91 insertions(+), 15 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 1bc0b5b50..cfc2b05e5 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -208,8 +208,6 @@ static struct rte_eth_conf port_conf = {
 	},
 	.txmode = {
 		.mq_mode = ETH_MQ_TX_NONE,
-		.offloads = (DEV_TX_OFFLOAD_IPV4_CKSUM |
-			     DEV_TX_OFFLOAD_MULTI_SEGS),
 	},
 };
 
@@ -315,7 +313,8 @@ prepare_traffic(struct rte_mbuf **pkts, struct ipsec_traffic *t,
 }
 
 static inline void
-prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
+prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port,
+		const struct lcore_conf *qconf)
 {
 	struct ip *ip;
 	struct ether_hdr *ethhdr;
@@ -325,14 +324,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
 	ethhdr = (struct ether_hdr *)rte_pktmbuf_prepend(pkt, ETHER_HDR_LEN);
 
 	if (ip->ip_v == IPVERSION) {
-		pkt->ol_flags |= PKT_TX_IP_CKSUM | PKT_TX_IPV4;
+		pkt->ol_flags |= qconf->outbound.ipv4_offloads;
 		pkt->l3_len = sizeof(struct ip);
 		pkt->l2_len = ETHER_HDR_LEN;
 
 		ip->ip_sum = 0;
+
+		/* calculate IPv4 cksum in SW */
+		if ((pkt->ol_flags & PKT_TX_IP_CKSUM) == 0)
+			ip->ip_sum = rte_ipv4_cksum((struct ipv4_hdr *)ip);
+
 		ethhdr->ether_type = rte_cpu_to_be_16(ETHER_TYPE_IPv4);
 	} else {
-		pkt->ol_flags |= PKT_TX_IPV6;
+		pkt->ol_flags |= qconf->outbound.ipv6_offloads;
 		pkt->l3_len = sizeof(struct ip6_hdr);
 		pkt->l2_len = ETHER_HDR_LEN;
 
@@ -346,18 +350,19 @@ prepare_tx_pkt(struct rte_mbuf *pkt, uint16_t port)
 }
 
 static inline void
-prepare_tx_burst(struct rte_mbuf *pkts[], uint16_t nb_pkts, uint16_t port)
+prepare_tx_burst(struct rte_mbuf *pkts[], uint16_t nb_pkts, uint16_t port,
+		const struct lcore_conf *qconf)
 {
 	int32_t i;
 	const int32_t prefetch_offset = 2;
 
 	for (i = 0; i < (nb_pkts - prefetch_offset); i++) {
 		rte_mbuf_prefetch_part2(pkts[i + prefetch_offset]);
-		prepare_tx_pkt(pkts[i], port);
+		prepare_tx_pkt(pkts[i], port, qconf);
 	}
 	/* Process left packets */
 	for (; i < nb_pkts; i++)
-		prepare_tx_pkt(pkts[i], port);
+		prepare_tx_pkt(pkts[i], port, qconf);
 }
 
 /* Send burst of packets on an output interface */
@@ -371,7 +376,7 @@ send_burst(struct lcore_conf *qconf, uint16_t n, uint16_t port)
 	queueid = qconf->tx_queue_id[port];
 	m_table = (struct rte_mbuf **)qconf->tx_mbufs[port].m_table;
 
-	prepare_tx_burst(m_table, n, port);
+	prepare_tx_burst(m_table, n, port, qconf);
 
 	ret = rte_eth_tx_burst(port, queueid, m_table, n);
 	if (unlikely(ret < n)) {
@@ -1543,7 +1548,7 @@ cryptodevs_init(void)
 }
 
 static void
-port_init(uint16_t portid)
+port_init(uint16_t portid, uint64_t req_rx_offloads, uint64_t req_tx_offloads)
 {
 	struct rte_eth_dev_info dev_info;
 	struct rte_eth_txconf *txconf;
@@ -1584,10 +1589,10 @@ port_init(uint16_t portid)
 		local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_JUMBO_FRAME;
 	}
 
-	if (dev_info.rx_offload_capa & DEV_RX_OFFLOAD_SECURITY)
-		local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_SECURITY;
-	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_SECURITY)
-		local_port_conf.txmode.offloads |= DEV_TX_OFFLOAD_SECURITY;
+	/* Capabilities will already have been checked.. */
+	local_port_conf.rxmode.offloads |= req_rx_offloads;
+	local_port_conf.txmode.offloads |= req_tx_offloads;
+
 	if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_MBUF_FAST_FREE)
 		local_port_conf.txmode.offloads |=
 			DEV_TX_OFFLOAD_MBUF_FAST_FREE;
@@ -1639,6 +1644,13 @@ port_init(uint16_t portid)
 
 		qconf = &lcore_conf[lcore_id];
 		qconf->tx_queue_id[portid] = tx_queueid;
+
+		/* Pre-populate pkt offloads based on capabilities */
+		qconf->outbound.ipv4_offloads = PKT_TX_IPV4;
+		qconf->outbound.ipv6_offloads = PKT_TX_IPV6;
+		if (req_tx_offloads & DEV_TX_OFFLOAD_IPV4_CKSUM)
+			qconf->outbound.ipv4_offloads |= PKT_TX_IP_CKSUM;
+
 		tx_queueid++;
 
 		/* init RX queues */
@@ -1749,6 +1761,7 @@ main(int32_t argc, char **argv)
 	uint32_t lcore_id;
 	uint8_t socket_id;
 	uint16_t portid;
+	uint64_t req_rx_offloads, req_tx_offloads;
 
 	/* init EAL */
 	ret = rte_eal_init(argc, argv);
@@ -1804,7 +1817,8 @@ main(int32_t argc, char **argv)
 		if ((enabled_port_mask & (1 << portid)) == 0)
 			continue;
 
-		port_init(portid);
+		sa_check_offloads(portid, &req_rx_offloads, &req_tx_offloads);
+		port_init(portid, req_rx_offloads, req_tx_offloads);
 	}
 
 	cryptodevs_init();
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index c998c8076..9b1586f52 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -146,6 +146,8 @@ struct ipsec_ctx {
 	struct rte_mempool *session_pool;
 	struct rte_mbuf *ol_pkts[MAX_PKT_BURST] __rte_aligned(sizeof(void *));
 	uint16_t ol_pkts_cnt;
+	uint64_t ipv4_offloads;
+	uint64_t ipv6_offloads;
 };
 
 struct cdev_key {
@@ -239,4 +241,8 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id);
 void
 rt_init(struct socket_ctx *ctx, int32_t socket_id);
 
+int
+sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
+		uint64_t *tx_offloads);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index d2d3550a4..ff8c4b829 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -1017,3 +1017,59 @@ outbound_sa_lookup(struct sa_ctx *sa_ctx, uint32_t sa_idx[],
 	for (i = 0; i < nb_pkts; i++)
 		sa[i] = &sa_ctx->sa[sa_idx[i]];
 }
+
+/*
+ * Select HW offloads to be used.
+ */
+int
+sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
+		uint64_t *tx_offloads)
+{
+	struct ipsec_sa *rule;
+	uint32_t idx_sa;
+	struct rte_eth_dev_info dev_info;
+
+	rte_eth_dev_info_get(port_id, &dev_info);
+
+	*rx_offloads = 0;
+	*tx_offloads = 0;
+
+	/* Check for inbound rules that use offloads and use this port */
+	for (idx_sa = 0; idx_sa < nb_sa_in; idx_sa++) {
+		rule = &sa_in[idx_sa];
+		if ((rule->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				rule->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL)
+				&& rule->portid == port_id) {
+			if ((dev_info.rx_offload_capa & DEV_RX_OFFLOAD_SECURITY)
+					== 0) {
+				RTE_LOG(WARNING, PORT,
+					"HW RX IPSec is not supported\n");
+				return -EINVAL;
+			}
+			*rx_offloads |= DEV_RX_OFFLOAD_SECURITY;
+		}
+	}
+
+	/* Check for outbound rules that use offloads and use this port */
+	for (idx_sa = 0; idx_sa < nb_sa_out; idx_sa++) {
+		rule = &sa_out[idx_sa];
+		if ((rule->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				rule->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL)
+				&& rule->portid == port_id) {
+			if ((dev_info.tx_offload_capa & DEV_TX_OFFLOAD_SECURITY)
+					== 0) {
+				RTE_LOG(WARNING, PORT,
+					"HW TX IPSec is not supported\n");
+				return -EINVAL;
+			}
+			*tx_offloads |= DEV_TX_OFFLOAD_SECURITY;
+			/* Enable HW IPv4 cksum as well, if it is available */
+			if (dev_info.tx_offload_capa &
+					DEV_TX_OFFLOAD_IPV4_CKSUM)
+				*tx_offloads |= DEV_TX_OFFLOAD_IPV4_CKSUM;
+		}
+	}
+	return 0;
+}
-- 
2.17.1

[PATCH v3 2/8] examples/ipsec-secgw: allow to specify neighbor mac address

[Konstantin Ananyev]

In some cases it is useful to allow user to specify destination
ether address for outgoing packets.
This patch adds such ability by introducing new 'neigh' config
file option.

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 21 +++++++--
 examples/ipsec-secgw/ipsec.h       |  3 ++
 examples/ipsec-secgw/parser.c      | 75 ++++++++++++++++++++++++++++++
 examples/ipsec-secgw/parser.h      |  8 ++--
 4 files changed, 99 insertions(+), 8 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index cfc2b05e5..2ed757922 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -104,9 +104,9 @@ static uint16_t nb_txd = IPSEC_SECGW_TX_DESC_DEFAULT;
 #define ETHADDR(a, b, c, d, e, f) (__BYTES_TO_UINT64(a, b, c, d, e, f, 0, 0))
 
 #define ETHADDR_TO_UINT64(addr) __BYTES_TO_UINT64( \
-		addr.addr_bytes[0], addr.addr_bytes[1], \
-		addr.addr_bytes[2], addr.addr_bytes[3], \
-		addr.addr_bytes[4], addr.addr_bytes[5], \
+		(addr)->addr_bytes[0], (addr)->addr_bytes[1], \
+		(addr)->addr_bytes[2], (addr)->addr_bytes[3], \
+		(addr)->addr_bytes[4], (addr)->addr_bytes[5], \
 		0, 0)
 
 /* port/source ethernet addr and destination ethernet addr */
@@ -1188,6 +1188,19 @@ print_ethaddr(const char *name, const struct ether_addr *eth_addr)
 	printf("%s%s", name, buf);
 }
 
+/*
+ * Update destination ethaddr for the port.
+ */
+int
+add_dst_ethaddr(uint16_t port, const struct ether_addr *addr)
+{
+	if (port > RTE_DIM(ethaddr_tbl))
+		return -EINVAL;
+
+	ethaddr_tbl[port].dst = ETHADDR_TO_UINT64(addr);
+	return 0;
+}
+
 /* Check the link status of all ports in up to 9s, and print them finally */
 static void
 check_all_ports_link_status(uint32_t port_mask)
@@ -1564,7 +1577,7 @@ port_init(uint16_t portid, uint64_t req_rx_offloads, uint64_t req_tx_offloads)
 	printf("Configuring device port %u:\n", portid);
 
 	rte_eth_macaddr_get(portid, &ethaddr);
-	ethaddr_tbl[portid].src = ETHADDR_TO_UINT64(ethaddr);
+	ethaddr_tbl[portid].src = ETHADDR_TO_UINT64(&ethaddr);
 	print_ethaddr("Address: ", &ethaddr);
 	printf("\n");
 
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 9b1586f52..580f7876b 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -245,4 +245,7 @@ int
 sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
 		uint64_t *tx_offloads);
 
+int
+add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/parser.c b/examples/ipsec-secgw/parser.c
index 91282ca94..6769a8e6d 100644
--- a/examples/ipsec-secgw/parser.c
+++ b/examples/ipsec-secgw/parser.c
@@ -306,6 +306,30 @@ parse_range(const char *token, uint16_t *low, uint16_t *high)
 	return 0;
 }
 
+#define PARSE_UINT8x16(s, v, l)	                          \
+do {                                                      \
+	char *end;                                        \
+	unsigned long t;                                  \
+	errno = 0;                                        \
+	t = strtoul((s), &end, 16);                       \
+	if (errno != 0 || end[0] != (l) || t > UINT8_MAX) \
+		return -EINVAL;                           \
+	(s) = end + 1;                                    \
+	(v) = t;                                          \
+} while (0)
+
+static int
+parse_mac(const char *str, struct ether_addr *addr)
+{
+	PARSE_UINT8x16(str, addr->addr_bytes[0], ':');
+	PARSE_UINT8x16(str, addr->addr_bytes[1], ':');
+	PARSE_UINT8x16(str, addr->addr_bytes[2], ':');
+	PARSE_UINT8x16(str, addr->addr_bytes[3], ':');
+	PARSE_UINT8x16(str, addr->addr_bytes[4], ':');
+	PARSE_UINT8x16(str, addr->addr_bytes[5], 0);
+	return 0;
+}
+
 /** sp add parse */
 struct cfg_sp_add_cfg_item {
 	cmdline_fixed_string_t sp_keyword;
@@ -444,11 +468,61 @@ cmdline_parse_inst_t cfg_rt_add_rule = {
 	},
 };
 
+/* neigh add parse */
+struct cfg_neigh_add_item {
+	cmdline_fixed_string_t neigh;
+	cmdline_fixed_string_t pstr;
+	uint16_t port;
+	cmdline_fixed_string_t mac;
+};
+
+static void
+cfg_parse_neigh(void *parsed_result, __rte_unused struct cmdline *cl,
+	void *data)
+{
+	int32_t rc;
+	struct cfg_neigh_add_item *res;
+	struct parse_status *st;
+	struct ether_addr mac;
+
+	st = data;
+	res = parsed_result;
+	rc = parse_mac(res->mac, &mac);
+	APP_CHECK(rc == 0, st, "invalid ether addr:%s", res->mac);
+	rc = add_dst_ethaddr(res->port, &mac);
+	APP_CHECK(rc == 0, st, "invalid port numer:%hu", res->port);
+	if (st->status < 0)
+		return;
+}
+
+cmdline_parse_token_string_t cfg_add_neigh_start =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, neigh, "neigh");
+cmdline_parse_token_string_t cfg_add_neigh_pstr =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, pstr, "port");
+cmdline_parse_token_num_t cfg_add_neigh_port =
+	TOKEN_NUM_INITIALIZER(struct cfg_neigh_add_item, port, UINT16);
+cmdline_parse_token_string_t cfg_add_neigh_mac =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, mac, NULL);
+
+cmdline_parse_inst_t cfg_neigh_add_rule = {
+	.f = cfg_parse_neigh,
+	.data = NULL,
+	.help_str = "",
+	.tokens = {
+		(void *)&cfg_add_neigh_start,
+		(void *)&cfg_add_neigh_pstr,
+		(void *)&cfg_add_neigh_port,
+		(void *)&cfg_add_neigh_mac,
+		NULL,
+	},
+};
+
 /** set of cfg items */
 cmdline_parse_ctx_t ipsec_ctx[] = {
 	(cmdline_parse_inst_t *)&cfg_sp_add_rule,
 	(cmdline_parse_inst_t *)&cfg_sa_add_rule,
 	(cmdline_parse_inst_t *)&cfg_rt_add_rule,
+	(cmdline_parse_inst_t *)&cfg_neigh_add_rule,
 	NULL,
 };
 
@@ -474,6 +548,7 @@ parse_cfg_file(const char *cfg_filename)
 	cfg_sp_add_rule.data = &status;
 	cfg_sa_add_rule.data = &status;
 	cfg_rt_add_rule.data = &status;
+	cfg_neigh_add_rule.data = &status;
 
 	do {
 		char oneline[1024];
diff --git a/examples/ipsec-secgw/parser.h b/examples/ipsec-secgw/parser.h
index be02537c5..6b8a10076 100644
--- a/examples/ipsec-secgw/parser.h
+++ b/examples/ipsec-secgw/parser.h
@@ -14,14 +14,14 @@ struct parse_status {
 	char parse_msg[256];
 };
 
-#define	APP_CHECK(exp, status, fmt, ...)				\
+#define	APP_CHECK(exp, st, fmt, ...)					\
 do {									\
 	if (!(exp)) {							\
-		sprintf(status->parse_msg, fmt "\n",			\
+		sprintf((st)->parse_msg, fmt "\n",			\
 			## __VA_ARGS__);				\
-		status->status = -1;					\
+		(st)->status = -1;					\
 	} else								\
-		status->status = 0;					\
+		(st)->status = 0;					\
 } while (0)
 
 #define APP_CHECK_PRESENCE(val, str, status)				\
-- 
2.17.1

[PATCH v3 3/8] examples/ipsec-secgw: fix crypto-op might never get dequeued

[Konstantin Ananyev]

In some cases crypto-ops could never be dequeued from the crypto-device.
The easiest way to reproduce:
start ipsec-secgw with crypto-dev and send to it less then 32 packets.
none packets will be forwarded.
Reason for that is that the application does dequeue() from crypto-queues
only when new packets arrive.
This patch makes sure it calls dequeue() on a regular basis.

Fixes: c64278c0c18b ("examples/ipsec-secgw: rework processing loop")

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 136 ++++++++++++++++++++++++-----
 examples/ipsec-secgw/ipsec.c       |  60 ++++++++-----
 examples/ipsec-secgw/ipsec.h       |  11 +++
 3 files changed, 165 insertions(+), 42 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 2ed757922..62443172a 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -456,38 +456,55 @@ inbound_sp_sa(struct sp_ctx *sp, struct sa_ctx *sa, struct traffic_type *ip,
 	ip->num = j;
 }
 
-static inline void
-process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
-		struct ipsec_traffic *traffic)
+static void
+split46_traffic(struct ipsec_traffic *trf, struct rte_mbuf *mb[], uint32_t num)
 {
+	uint32_t i, n4, n6;
+	struct ip *ip;
 	struct rte_mbuf *m;
-	uint16_t idx, nb_pkts_in, i, n_ip4, n_ip6;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
+	n4 = trf->ip4.num;
+	n6 = trf->ip6.num;
 
-	n_ip4 = traffic->ip4.num;
-	n_ip6 = traffic->ip6.num;
+	for (i = 0; i < num; i++) {
+
+		m = mb[i];
+		ip = rte_pktmbuf_mtod(m, struct ip *);
 
-	/* SP/ACL Inbound check ipsec and ip4 */
-	for (i = 0; i < nb_pkts_in; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
 		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-			traffic->ip4.data[idx] = rte_pktmbuf_mtod_offset(m,
+			trf->ip4.pkts[n4] = m;
+			trf->ip4.data[n4] = rte_pktmbuf_mtod_offset(m,
 					uint8_t *, offsetof(struct ip, ip_p));
+			n4++;
 		} else if (ip->ip_v == IP6_VERSION) {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
-			traffic->ip6.data[idx] = rte_pktmbuf_mtod_offset(m,
+			trf->ip6.pkts[n6] = m;
+			trf->ip6.data[n6] = rte_pktmbuf_mtod_offset(m,
 					uint8_t *,
 					offsetof(struct ip6_hdr, ip6_nxt));
+			n6++;
 		} else
 			rte_pktmbuf_free(m);
 	}
 
+	trf->ip4.num = n4;
+	trf->ip6.num = n6;
+}
+
+
+static inline void
+process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
+		struct ipsec_traffic *traffic)
+{
+	uint16_t nb_pkts_in, n_ip4, n_ip6;
+
+	n_ip4 = traffic->ip4.num;
+	n_ip6 = traffic->ip6.num;
+
+	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.num, MAX_PKT_BURST);
+
+	split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+
 	inbound_sp_sa(ipsec_ctx->sp4_ctx, ipsec_ctx->sa_ctx, &traffic->ip4,
 			n_ip4);
 
@@ -782,7 +799,7 @@ process_pkts(struct lcore_conf *qconf, struct rte_mbuf **pkts,
 }
 
 static inline void
-drain_buffers(struct lcore_conf *qconf)
+drain_tx_buffers(struct lcore_conf *qconf)
 {
 	struct buffer *buf;
 	uint32_t portid;
@@ -796,6 +813,81 @@ drain_buffers(struct lcore_conf *qconf)
 	}
 }
 
+static inline void
+drain_crypto_buffers(struct lcore_conf *qconf)
+{
+	uint32_t i;
+	struct ipsec_ctx *ctx;
+
+	/* drain inbound buffers*/
+	ctx = &qconf->inbound;
+	for (i = 0; i != ctx->nb_qps; i++) {
+		if (ctx->tbl[i].len != 0)
+			enqueue_cop_burst(ctx->tbl  + i);
+	}
+
+	/* drain outbound buffers*/
+	ctx = &qconf->outbound;
+	for (i = 0; i != ctx->nb_qps; i++) {
+		if (ctx->tbl[i].len != 0)
+			enqueue_cop_burst(ctx->tbl  + i);
+	}
+}
+
+static void
+drain_inbound_crypto_queues(const struct lcore_conf *qconf,
+		struct ipsec_ctx *ctx)
+{
+	uint32_t n;
+	struct ipsec_traffic trf;
+
+	/* dequeue packets from crypto-queue */
+	n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+			RTE_DIM(trf.ipsec.pkts));
+	if (n == 0)
+		return;
+
+	trf.ip4.num = 0;
+	trf.ip6.num = 0;
+
+	/* split traffic by ipv4-ipv6 */
+	split46_traffic(&trf, trf.ipsec.pkts, n);
+
+	/* process ipv4 packets */
+	inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
+	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+
+	/* process ipv6 packets */
+	inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
+	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+}
+
+static void
+drain_outbound_crypto_queues(const struct lcore_conf *qconf,
+		struct ipsec_ctx *ctx)
+{
+	uint32_t n;
+	struct ipsec_traffic trf;
+
+	/* dequeue packets from crypto-queue */
+	n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+			RTE_DIM(trf.ipsec.pkts));
+	if (n == 0)
+		return;
+
+	trf.ip4.num = 0;
+	trf.ip6.num = 0;
+
+	/* split traffic by ipv4-ipv6 */
+	split46_traffic(&trf, trf.ipsec.pkts, n);
+
+	/* process ipv4 packets */
+	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+
+	/* process ipv6 packets */
+	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+}
+
 /* main processing loop */
 static int32_t
 main_loop(__attribute__((unused)) void *dummy)
@@ -853,7 +945,8 @@ main_loop(__attribute__((unused)) void *dummy)
 		diff_tsc = cur_tsc - prev_tsc;
 
 		if (unlikely(diff_tsc > drain_tsc)) {
-			drain_buffers(qconf);
+			drain_tx_buffers(qconf);
+			drain_crypto_buffers(qconf);
 			prev_tsc = cur_tsc;
 		}
 
@@ -867,6 +960,9 @@ main_loop(__attribute__((unused)) void *dummy)
 			if (nb_rx > 0)
 				process_pkts(qconf, pkts, nb_rx, portid);
 		}
+
+		drain_inbound_crypto_queues(qconf, &qconf->inbound);
+		drain_outbound_crypto_queues(qconf, &qconf->outbound);
 	}
 }
 
diff --git a/examples/ipsec-secgw/ipsec.c b/examples/ipsec-secgw/ipsec.c
index 3d415f1af..8bf928a23 100644
--- a/examples/ipsec-secgw/ipsec.c
+++ b/examples/ipsec-secgw/ipsec.c
@@ -333,33 +333,35 @@ create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa)
 	return 0;
 }
 
+/*
+ * queue crypto-ops into PMD queue.
+ */
+void
+enqueue_cop_burst(struct cdev_qp *cqp)
+{
+	uint32_t i, len, ret;
+
+	len = cqp->len;
+	ret = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp, cqp->buf, len);
+	if (ret < len) {
+		RTE_LOG_DP(DEBUG, IPSEC, "Cryptodev %u queue %u:"
+			" enqueued %u crypto ops out of %u\n",
+			cqp->id, cqp->qp, ret, len);
+			/* drop packets that we fail to enqueue */
+			for (i = ret; i < len; i++)
+				rte_pktmbuf_free(cqp->buf[i]->sym->m_src);
+	}
+	cqp->in_flight += ret;
+	cqp->len = 0;
+}
+
 static inline void
 enqueue_cop(struct cdev_qp *cqp, struct rte_crypto_op *cop)
 {
-	int32_t ret = 0, i;
-
 	cqp->buf[cqp->len++] = cop;
 
-	if (cqp->len == MAX_PKT_BURST) {
-		int enq_size = cqp->len;
-		if ((cqp->in_flight + enq_size) > MAX_INFLIGHT)
-			enq_size -=
-			    (int)((cqp->in_flight + enq_size) - MAX_INFLIGHT);
-
-		if (enq_size > 0)
-			ret = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp,
-					cqp->buf, enq_size);
-		if (ret < cqp->len) {
-			RTE_LOG_DP(DEBUG, IPSEC, "Cryptodev %u queue %u:"
-					" enqueued %u crypto ops out of %u\n",
-					 cqp->id, cqp->qp,
-					 ret, cqp->len);
-			for (i = ret; i < cqp->len; i++)
-				rte_pktmbuf_free(cqp->buf[i]->sym->m_src);
-		}
-		cqp->in_flight += ret;
-		cqp->len = 0;
-	}
+	if (cqp->len == MAX_PKT_BURST)
+		enqueue_cop_burst(cqp);
 }
 
 static inline void
@@ -548,6 +550,13 @@ ipsec_inbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 	return ipsec_dequeue(esp_inbound_post, ctx, pkts, len);
 }
 
+uint16_t
+ipsec_inbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len)
+{
+	return ipsec_dequeue(esp_inbound_post, ctx, pkts, len);
+}
+
 uint16_t
 ipsec_outbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint32_t sa_idx[], uint16_t nb_pkts, uint16_t len)
@@ -560,3 +569,10 @@ ipsec_outbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 
 	return ipsec_dequeue(esp_outbound_post, ctx, pkts, len);
 }
+
+uint16_t
+ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len)
+{
+	return ipsec_dequeue(esp_outbound_post, ctx, pkts, len);
+}
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 580f7876b..2f04b7d68 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -184,6 +184,14 @@ uint16_t
 ipsec_outbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint32_t sa_idx[], uint16_t nb_pkts, uint16_t len);
 
+uint16_t
+ipsec_inbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len);
+
+uint16_t
+ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len);
+
 static inline uint16_t
 ipsec_metadata_size(void)
 {
@@ -248,4 +256,7 @@ sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
 int
 add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
 
+void
+enqueue_cop_burst(struct cdev_qp *cqp);
+
 #endif /* __IPSEC_H__ */
-- 
2.17.1

[PATCH v3 4/8] examples/ipsec-secgw: fix outbound codepath for single SA

[Konstantin Ananyev]

Looking at process_pkts_outbound_nosp() there seems few issues:
- accessing mbuf after it was freed
- invoking ipsec_outbound() for ipv4 packets only
- copying number of packets, but not the mbuf pointers itself

that patch provides fixes for that issues.

Fixes: 906257e965b7 ("examples/ipsec-secgw: support IPv6")

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 32 ++++++++++++++++++++----------
 1 file changed, 22 insertions(+), 10 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 62443172a..d1da2d5ce 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -616,32 +616,44 @@ process_pkts_outbound_nosp(struct ipsec_ctx *ipsec_ctx,
 		struct ipsec_traffic *traffic)
 {
 	struct rte_mbuf *m;
-	uint32_t nb_pkts_out, i;
+	uint32_t nb_pkts_out, i, n;
 	struct ip *ip;
 
 	/* Drop any IPsec traffic from protected ports */
 	for (i = 0; i < traffic->ipsec.num; i++)
 		rte_pktmbuf_free(traffic->ipsec.pkts[i]);
 
-	traffic->ipsec.num = 0;
+	n = 0;
 
-	for (i = 0; i < traffic->ip4.num; i++)
-		traffic->ip4.res[i] = single_sa_idx;
+	for (i = 0; i < traffic->ip4.num; i++) {
+		traffic->ipsec.pkts[n] = traffic->ip4.pkts[i];
+		traffic->ipsec.res[n++] = single_sa_idx;
+	}
 
-	for (i = 0; i < traffic->ip6.num; i++)
-		traffic->ip6.res[i] = single_sa_idx;
+	for (i = 0; i < traffic->ip6.num; i++) {
+		traffic->ipsec.pkts[n] = traffic->ip6.pkts[i];
+		traffic->ipsec.res[n++] = single_sa_idx;
+	}
+
+	traffic->ip4.num = 0;
+	traffic->ip6.num = 0;
+	traffic->ipsec.num = n;
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ip4.pkts,
-			traffic->ip4.res, traffic->ip4.num,
+	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.res, traffic->ipsec.num,
 			MAX_PKT_BURST);
 
 	/* They all sue the same SA (ip4 or ip6 tunnel) */
 	m = traffic->ipsec.pkts[i];
 	ip = rte_pktmbuf_mtod(m, struct ip *);
-	if (ip->ip_v == IPVERSION)
+	if (ip->ip_v == IPVERSION) {
 		traffic->ip4.num = nb_pkts_out;
-	else
+		for (i = 0; i < nb_pkts_out; i++)
+			traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+	} else {
 		traffic->ip6.num = nb_pkts_out;
+		traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+	}
 }
 
 static inline int32_t
-- 
2.17.1

[PATCH v3 5/8] examples/ipsec-secgw: make local variables static

[Konstantin Ananyev]

in sp4.c and sp6.c there are few globals that used only locally.
Define them as static ones.

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/sp4.c | 10 +++++-----
 examples/ipsec-secgw/sp6.c | 10 +++++-----
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/examples/ipsec-secgw/sp4.c b/examples/ipsec-secgw/sp4.c
index 8d3d3d8e0..6b05daaa9 100644
--- a/examples/ipsec-secgw/sp4.c
+++ b/examples/ipsec-secgw/sp4.c
@@ -44,7 +44,7 @@ enum {
 	RTE_ACL_IPV4_NUM
 };
 
-struct rte_acl_field_def ip4_defs[NUM_FIELDS_IPV4] = {
+static struct rte_acl_field_def ip4_defs[NUM_FIELDS_IPV4] = {
 	{
 	.type = RTE_ACL_FIELD_TYPE_BITMASK,
 	.size = sizeof(uint8_t),
@@ -85,11 +85,11 @@ struct rte_acl_field_def ip4_defs[NUM_FIELDS_IPV4] = {
 
 RTE_ACL_RULE_DEF(acl4_rules, RTE_DIM(ip4_defs));
 
-struct acl4_rules acl4_rules_out[MAX_ACL_RULE_NUM];
-uint32_t nb_acl4_rules_out;
+static struct acl4_rules acl4_rules_out[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl4_rules_out;
 
-struct acl4_rules acl4_rules_in[MAX_ACL_RULE_NUM];
-uint32_t nb_acl4_rules_in;
+static struct acl4_rules acl4_rules_in[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl4_rules_in;
 
 void
 parse_sp4_tokens(char **tokens, uint32_t n_tokens,
diff --git a/examples/ipsec-secgw/sp6.c b/examples/ipsec-secgw/sp6.c
index 6002afef3..dc5b94c6a 100644
--- a/examples/ipsec-secgw/sp6.c
+++ b/examples/ipsec-secgw/sp6.c
@@ -34,7 +34,7 @@ enum {
 
 #define IP6_ADDR_SIZE 16
 
-struct rte_acl_field_def ip6_defs[IP6_NUM] = {
+static struct rte_acl_field_def ip6_defs[IP6_NUM] = {
 	{
 	.type = RTE_ACL_FIELD_TYPE_BITMASK,
 	.size = sizeof(uint8_t),
@@ -116,11 +116,11 @@ struct rte_acl_field_def ip6_defs[IP6_NUM] = {
 
 RTE_ACL_RULE_DEF(acl6_rules, RTE_DIM(ip6_defs));
 
-struct acl6_rules acl6_rules_out[MAX_ACL_RULE_NUM];
-uint32_t nb_acl6_rules_out;
+static struct acl6_rules acl6_rules_out[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl6_rules_out;
 
-struct acl6_rules acl6_rules_in[MAX_ACL_RULE_NUM];
-uint32_t nb_acl6_rules_in;
+static struct acl6_rules acl6_rules_in[MAX_ACL_RULE_NUM];
+static uint32_t nb_acl6_rules_in;
 
 void
 parse_sp6_tokens(char **tokens, uint32_t n_tokens,
-- 
2.17.1

[PATCH v3 6/8] examples/ipsec-secgw: make app to use ipsec library

[Konstantin Ananyev]

Changes to make ipsec-secgw to utilize librte_ipsec library.
That patch provides:
 - changes in the related data structures.
 - changes in the initialization code.
 - new command-line parameters to enable librte_ipsec codepath
   and related features.

Note that right now by default current (non-librte_ipsec) code-path will
be used. User has to run application with new command-line option ('-l')
to enable new codepath.
The main reason for that:
 - current librte_ipsec doesn't support all ipsec algorithms
   and features that the app does.
 - allow users to run both versions in parallel for some time
   to figure out any functional or performance degradation with the
   new code.

It is planned to deprecate and remove non-librte_ipsec code path
in future releases.

Signed-off-by: Mohammad Abdul Awal <mohammad.abdul.awal@intel.com>
Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c |  50 ++++++-
 examples/ipsec-secgw/ipsec.h       |  24 ++++
 examples/ipsec-secgw/meson.build   |   2 +-
 examples/ipsec-secgw/sa.c          | 221 ++++++++++++++++++++++++++++-
 examples/ipsec-secgw/sp4.c         |  25 ++++
 examples/ipsec-secgw/sp6.c         |  25 ++++
 6 files changed, 341 insertions(+), 6 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index d1da2d5ce..48baa5001 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -155,6 +155,9 @@ static uint32_t single_sa;
 static uint32_t single_sa_idx;
 static uint32_t frame_size;
 
+/* application wide librte_ipsec/SA parameters */
+struct app_sa_prm app_sa_prm = {.enable = 0};
+
 struct lcore_rx_queue {
 	uint16_t port_id;
 	uint8_t queue_id;
@@ -1063,6 +1066,10 @@ print_usage(const char *prgname)
 		" [-P]"
 		" [-u PORTMASK]"
 		" [-j FRAMESIZE]"
+		" [-l]"
+		" [-w REPLAY_WINDOW_SIZE]"
+		" [-e]"
+		" [-a]"
 		" -f CONFIG_FILE"
 		" --config (port,queue,lcore)[,(port,queue,lcore)]"
 		" [--single-sa SAIDX]"
@@ -1073,6 +1080,10 @@ print_usage(const char *prgname)
 		"  -u PORTMASK: Hexadecimal bitmask of unprotected ports\n"
 		"  -j FRAMESIZE: Enable jumbo frame with 'FRAMESIZE' as maximum\n"
 		"                packet size\n"
+		"  -l enables code-path that uses librte_ipsec\n"
+		"  -w REPLAY_WINDOW_SIZE specifies IPsec SQN replay window\n"
+		"     size for each SA\n"
+		"  -a enables SA SQN atomic behaviour\n"
 		"  -f CONFIG_FILE: Configuration file\n"
 		"  --config (port,queue,lcore): Rx queue configuration\n"
 		"  --single-sa SAIDX: Use single SA index for outbound traffic,\n"
@@ -1169,6 +1180,20 @@ parse_config(const char *q_arg)
 	return 0;
 }
 
+static void
+print_app_sa_prm(const struct app_sa_prm *prm)
+{
+	printf("librte_ipsec usage: %s\n",
+		(prm->enable == 0) ? "disabled" : "enabled");
+
+	if (prm->enable == 0)
+		return;
+
+	printf("replay window size: %u\n", prm->window_size);
+	printf("ESN: %s\n", (prm->enable_esn == 0) ? "disabled" : "enabled");
+	printf("SA flags: %#" PRIx64 "\n", prm->flags);
+}
+
 static int32_t
 parse_args(int32_t argc, char **argv)
 {
@@ -1180,7 +1205,7 @@ parse_args(int32_t argc, char **argv)
 
 	argvopt = argv;
 
-	while ((opt = getopt_long(argc, argvopt, "p:Pu:f:j:",
+	while ((opt = getopt_long(argc, argvopt, "aelp:Pu:f:j:w:",
 				lgopts, &option_index)) != EOF) {
 
 		switch (opt) {
@@ -1236,6 +1261,21 @@ parse_args(int32_t argc, char **argv)
 			}
 			printf("Enabled jumbo frames size %u\n", frame_size);
 			break;
+		case 'l':
+			app_sa_prm.enable = 1;
+			break;
+		case 'w':
+			app_sa_prm.enable = 1;
+			app_sa_prm.window_size = parse_decimal(optarg);
+			break;
+		case 'e':
+			app_sa_prm.enable = 1;
+			app_sa_prm.enable_esn = 1;
+			break;
+		case 'a':
+			app_sa_prm.enable = 1;
+			app_sa_prm.flags |= RTE_IPSEC_SAFLAG_SQN_ATOM;
+			break;
 		case CMD_LINE_OPT_CONFIG_NUM:
 			ret = parse_config(optarg);
 			if (ret) {
@@ -1280,6 +1320,8 @@ parse_args(int32_t argc, char **argv)
 		return -1;
 	}
 
+	print_app_sa_prm(&app_sa_prm);
+
 	if (optind >= 0)
 		argv[optind-1] = prgname;
 
@@ -1923,12 +1965,14 @@ main(int32_t argc, char **argv)
 		if (socket_ctx[socket_id].mbuf_pool)
 			continue;
 
-		sa_init(&socket_ctx[socket_id], socket_id);
-
+		/* initilaze SPD */
 		sp4_init(&socket_ctx[socket_id], socket_id);
 
 		sp6_init(&socket_ctx[socket_id], socket_id);
 
+		/* initilaze SAD */
+		sa_init(&socket_ctx[socket_id], socket_id);
+
 		rt_init(&socket_ctx[socket_id], socket_id);
 
 		pool_init(&socket_ctx[socket_id], socket_id, NB_MBUF);
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 2f04b7d68..b089fe54b 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -11,6 +11,7 @@
 #include <rte_crypto.h>
 #include <rte_security.h>
 #include <rte_flow.h>
+#include <rte_ipsec.h>
 
 #define RTE_LOGTYPE_IPSEC       RTE_LOGTYPE_USER1
 #define RTE_LOGTYPE_IPSEC_ESP   RTE_LOGTYPE_USER2
@@ -70,7 +71,20 @@ struct ip_addr {
 
 #define MAX_KEY_SIZE		32
 
+/*
+ * application wide SA parameters
+ */
+struct app_sa_prm {
+	uint32_t enable; /* use librte_ipsec API for ipsec pkt processing */
+	uint32_t window_size; /* replay window size */
+	uint32_t enable_esn;  /* enable/disable ESN support */
+	uint64_t flags;       /* rte_ipsec_sa_prm.flags */
+};
+
+extern struct app_sa_prm app_sa_prm;
+
 struct ipsec_sa {
+	struct rte_ipsec_session ips; /* one session per sa for now */
 	uint32_t spi;
 	uint32_t cdev_id_qp;
 	uint64_t seq;
@@ -243,6 +257,16 @@ sp4_init(struct socket_ctx *ctx, int32_t socket_id);
 void
 sp6_init(struct socket_ctx *ctx, int32_t socket_id);
 
+/*
+ * Search though SP rules for given SPI.
+ * Returns first rule index if found(greater or equal then zero),
+ * or -ENOENT otherwise.
+ */
+int
+sp4_spi_present(uint32_t spi, int inbound);
+int
+sp6_spi_present(uint32_t spi, int inbound);
+
 void
 sa_init(struct socket_ctx *ctx, int32_t socket_id);
 
diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
index 77d8b298f..31f68fee2 100644
--- a/examples/ipsec-secgw/meson.build
+++ b/examples/ipsec-secgw/meson.build
@@ -6,7 +6,7 @@
 # To build this example as a standalone application with an already-installed
 # DPDK instance, use 'make'
 
-deps += ['security', 'lpm', 'acl', 'hash']
+deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
 allow_experimental_apis = true
 sources = files(
 	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index ff8c4b829..04eb8bce8 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -19,6 +19,7 @@
 #include <rte_ip.h>
 #include <rte_random.h>
 #include <rte_ethdev.h>
+#include <rte_malloc.h>
 
 #include "ipsec.h"
 #include "esp.h"
@@ -694,6 +695,7 @@ print_one_sa_rule(const struct ipsec_sa *sa, int inbound)
 }
 
 struct sa_ctx {
+	void *satbl; /* pointer to array of rte_ipsec_sa objects*/
 	struct ipsec_sa sa[IPSEC_SA_MAX_ENTRIES];
 	union {
 		struct {
@@ -764,7 +766,10 @@ sa_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 {
 	struct ipsec_sa *sa;
 	uint32_t i, idx;
-	uint16_t iv_length;
+	uint16_t iv_length, aad_length;
+
+	/* for ESN upper 32 bits of SQN also need to be part of AAD */
+	aad_length = (app_sa_prm.enable_esn != 0) ? sizeof(uint32_t) : 0;
 
 	for (i = 0; i < nb_entries; i++) {
 		idx = SPI2IDX(entries[i].spi);
@@ -808,7 +813,7 @@ sa_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 			sa_ctx->xf[idx].a.aead.iv.offset = IV_OFFSET;
 			sa_ctx->xf[idx].a.aead.iv.length = iv_length;
 			sa_ctx->xf[idx].a.aead.aad_length =
-				sa->aad_len;
+				sa->aad_len + aad_length;
 			sa_ctx->xf[idx].a.aead.digest_length =
 				sa->digest_len;
 
@@ -901,9 +906,205 @@ sa_in_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 	return sa_add_rules(sa_ctx, entries, nb_entries, 1);
 }
 
+/*
+ * helper function, fills parameters that are identical for all SAs
+ */
+static void
+fill_ipsec_app_sa_prm(struct rte_ipsec_sa_prm *prm,
+	const struct app_sa_prm *app_prm)
+{
+	memset(prm, 0, sizeof(*prm));
+
+	prm->flags = app_prm->flags;
+	prm->ipsec_xform.options.esn = app_prm->enable_esn;
+	prm->replay_win_sz = app_prm->window_size;
+}
+
+/*
+ * Helper function, tries to determine next_proto for SPI
+ * by searching though SP rules.
+ */
+static int
+get_spi_proto(uint32_t spi, enum rte_security_ipsec_sa_direction dir)
+{
+	int32_t rc4, rc6;
+
+	rc4 = sp4_spi_present(spi, dir == RTE_SECURITY_IPSEC_SA_DIR_INGRESS);
+	rc6 = sp6_spi_present(spi, dir == RTE_SECURITY_IPSEC_SA_DIR_INGRESS);
+
+	if (rc4 >= 0) {
+		if (rc6 >= 0) {
+			RTE_LOG(ERR, IPSEC,
+				"%s: SPI %u used simultaeously by "
+				"IPv4(%d) and IPv6 (%d) SP rules\n",
+				__func__, spi, rc4, rc6);
+			return -EINVAL;
+		} else
+			return IPPROTO_IPIP;
+	} else if (rc6 < 0) {
+		RTE_LOG(ERR, IPSEC,
+			"%s: SPI %u is not used by any SP rule\n",
+			__func__, spi);
+		return -EINVAL;
+	} else
+		return IPPROTO_IPV6;
+}
+
+static int
+fill_ipsec_sa_prm(struct rte_ipsec_sa_prm *prm, const struct ipsec_sa *ss,
+	const struct ipv4_hdr *v4, struct ipv6_hdr *v6)
+{
+	int32_t rc;
+
+	/*
+	 * Try to get SPI next proto by searching that SPI in SPD.
+	 * probably not the optimal way, but there seems nothing
+	 * better right now.
+	 */
+	rc = get_spi_proto(ss->spi, ss->direction);
+	if (rc < 0)
+		return rc;
+
+	fill_ipsec_app_sa_prm(prm, &app_sa_prm);
+	prm->userdata = (uintptr_t)ss;
+
+	/* setup ipsec xform */
+	prm->ipsec_xform.spi = ss->spi;
+	prm->ipsec_xform.salt = ss->salt;
+	prm->ipsec_xform.direction = ss->direction;
+	prm->ipsec_xform.proto = RTE_SECURITY_IPSEC_SA_PROTO_ESP;
+	prm->ipsec_xform.mode = (ss->flags == TRANSPORT) ?
+		RTE_SECURITY_IPSEC_SA_MODE_TRANSPORT :
+		RTE_SECURITY_IPSEC_SA_MODE_TUNNEL;
+
+	if (ss->flags == IP4_TUNNEL) {
+		prm->ipsec_xform.tunnel.type = RTE_SECURITY_IPSEC_TUNNEL_IPV4;
+		prm->tun.hdr_len = sizeof(*v4);
+		prm->tun.next_proto = rc;
+		prm->tun.hdr = v4;
+	} else if (ss->flags == IP6_TUNNEL) {
+		prm->ipsec_xform.tunnel.type = RTE_SECURITY_IPSEC_TUNNEL_IPV6;
+		prm->tun.hdr_len = sizeof(*v6);
+		prm->tun.next_proto = rc;
+		prm->tun.hdr = v6;
+	} else {
+		/* transport mode */
+		prm->trs.proto = rc;
+	}
+
+	/* setup crypto section */
+	prm->crypto_xform = ss->xforms;
+	return 0;
+}
+
+static void
+fill_ipsec_session(struct rte_ipsec_session *ss, struct rte_ipsec_sa *sa,
+	const struct ipsec_sa *lsa)
+{
+	ss->sa = sa;
+	ss->type = lsa->type;
+
+	/* setup crypto section */
+	if (ss->type == RTE_SECURITY_ACTION_TYPE_NONE) {
+		ss->crypto.ses = lsa->crypto_session;
+	/* setup session action type */
+	} else {
+		ss->security.ses = lsa->sec_session;
+		ss->security.ctx = lsa->security_ctx;
+		ss->security.ol_flags = lsa->ol_flags;
+	}
+}
+
+/*
+ * Initialise related rte_ipsec_sa object.
+ */
+static int
+ipsec_sa_init(struct ipsec_sa *lsa, struct rte_ipsec_sa *sa, uint32_t sa_size)
+{
+	int rc;
+	struct rte_ipsec_sa_prm prm;
+	struct ipv4_hdr v4  = {
+		.version_ihl = IPVERSION << 4 |
+			sizeof(v4) / IPV4_IHL_MULTIPLIER,
+		.time_to_live = IPDEFTTL,
+		.next_proto_id = IPPROTO_ESP,
+		.src_addr = lsa->src.ip.ip4,
+		.dst_addr = lsa->dst.ip.ip4,
+	};
+	struct ipv6_hdr v6 = {
+		.vtc_flow = htonl(IP6_VERSION << 28),
+		.proto = IPPROTO_ESP,
+	};
+
+	if (lsa->flags == IP6_TUNNEL) {
+		memcpy(v6.src_addr, lsa->src.ip.ip6.ip6_b, sizeof(v6.src_addr));
+		memcpy(v6.dst_addr, lsa->dst.ip.ip6.ip6_b, sizeof(v6.dst_addr));
+	}
+
+	rc = fill_ipsec_sa_prm(&prm, lsa, &v4, &v6);
+	if (rc == 0)
+		rc = rte_ipsec_sa_init(sa, &prm, sa_size);
+	if (rc < 0)
+		return rc;
+
+	fill_ipsec_session(&lsa->ips, sa, lsa);
+	return 0;
+}
+
+/*
+ * Allocate space and init rte_ipsec_sa strcutures,
+ * one per session.
+ */
+static int
+ipsec_satbl_init(struct sa_ctx *ctx, const struct ipsec_sa *ent,
+	uint32_t nb_ent, int32_t socket)
+{
+	int32_t rc, sz;
+	uint32_t i, idx;
+	size_t tsz;
+	struct rte_ipsec_sa *sa;
+	struct ipsec_sa *lsa;
+	struct rte_ipsec_sa_prm prm;
+
+	/* determine SA size */
+	idx = SPI2IDX(ent[0].spi);
+	fill_ipsec_sa_prm(&prm, ctx->sa + idx, NULL, NULL);
+	sz = rte_ipsec_sa_size(&prm);
+	if (sz < 0) {
+		RTE_LOG(ERR, IPSEC, "%s(%p, %u, %d): "
+			"failed to determine SA size, error code: %d\n",
+			__func__, ctx, nb_ent, socket, sz);
+		return sz;
+	}
+
+	tsz = sz * nb_ent;
+
+	ctx->satbl = rte_zmalloc_socket(NULL, tsz, RTE_CACHE_LINE_SIZE, socket);
+	if (ctx->satbl == NULL) {
+		RTE_LOG(ERR, IPSEC,
+			"%s(%p, %u, %d): failed to allocate %zu bytes\n",
+			__func__,  ctx, nb_ent, socket, tsz);
+		return -ENOMEM;
+	}
+
+	rc = 0;
+	for (i = 0; i != nb_ent && rc == 0; i++) {
+
+		idx = SPI2IDX(ent[i].spi);
+
+		sa = (struct rte_ipsec_sa *)((uintptr_t)ctx->satbl + sz * i);
+		lsa = ctx->sa + idx;
+
+		rc = ipsec_sa_init(lsa, sa, sz);
+	}
+
+	return rc;
+}
+
 void
 sa_init(struct socket_ctx *ctx, int32_t socket_id)
 {
+	int32_t rc;
 	const char *name;
 
 	if (ctx == NULL)
@@ -926,6 +1127,14 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id)
 				name, socket_id);
 
 		sa_in_add_rules(ctx->sa_in, sa_in, nb_sa_in);
+
+		if (app_sa_prm.enable != 0) {
+			rc = ipsec_satbl_init(ctx->sa_in, sa_in, nb_sa_in,
+				socket_id);
+			if (rc != 0)
+				rte_exit(EXIT_FAILURE,
+					"failed to init inbound SAs\n");
+		}
 	} else
 		RTE_LOG(WARNING, IPSEC, "No SA Inbound rule specified\n");
 
@@ -938,6 +1147,14 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id)
 				name, socket_id);
 
 		sa_out_add_rules(ctx->sa_out, sa_out, nb_sa_out);
+
+		if (app_sa_prm.enable != 0) {
+			rc = ipsec_satbl_init(ctx->sa_out, sa_out, nb_sa_out,
+				socket_id);
+			if (rc != 0)
+				rte_exit(EXIT_FAILURE,
+					"failed to init outbound SAs\n");
+		}
 	} else
 		RTE_LOG(WARNING, IPSEC, "No SA Outbound rule "
 			"specified\n");
diff --git a/examples/ipsec-secgw/sp4.c b/examples/ipsec-secgw/sp4.c
index 6b05daaa9..d1dc64bad 100644
--- a/examples/ipsec-secgw/sp4.c
+++ b/examples/ipsec-secgw/sp4.c
@@ -504,3 +504,28 @@ sp4_init(struct socket_ctx *ctx, int32_t socket_id)
 		RTE_LOG(WARNING, IPSEC, "No IPv4 SP Outbound rule "
 			"specified\n");
 }
+
+/*
+ * Search though SP rules for given SPI.
+ */
+int
+sp4_spi_present(uint32_t spi, int inbound)
+{
+	uint32_t i, num;
+	const struct acl4_rules *acr;
+
+	if (inbound != 0) {
+		acr = acl4_rules_in;
+		num = nb_acl4_rules_in;
+	} else {
+		acr = acl4_rules_out;
+		num = nb_acl4_rules_out;
+	}
+
+	for (i = 0; i != num; i++) {
+		if (acr[i].data.userdata == PROTECT(spi))
+			return i;
+	}
+
+	return -ENOENT;
+}
diff --git a/examples/ipsec-secgw/sp6.c b/examples/ipsec-secgw/sp6.c
index dc5b94c6a..e67d85aaf 100644
--- a/examples/ipsec-secgw/sp6.c
+++ b/examples/ipsec-secgw/sp6.c
@@ -618,3 +618,28 @@ sp6_init(struct socket_ctx *ctx, int32_t socket_id)
 		RTE_LOG(WARNING, IPSEC, "No IPv6 SP Outbound rule "
 			"specified\n");
 }
+
+/*
+ * Search though SP rules for given SPI.
+ */
+int
+sp6_spi_present(uint32_t spi, int inbound)
+{
+	uint32_t i, num;
+	const struct acl6_rules *acr;
+
+	if (inbound != 0) {
+		acr = acl6_rules_in;
+		num = nb_acl6_rules_in;
+	} else {
+		acr = acl6_rules_out;
+		num = nb_acl6_rules_out;
+	}
+
+	for (i = 0; i != num; i++) {
+		if (acr[i].data.userdata == PROTECT(spi))
+			return i;
+	}
+
+	return -ENOENT;
+}
-- 
2.17.1

[PATCH v3 7/8] examples/ipsec-secgw: make data-path to use ipsec library

[Konstantin Ananyev]

Changes to make ipsec-secgw data-path code to utilize librte_ipsec library.
Note that right now by default current (non-librte_ipsec) code-path will
be used. User has to run application with new command-line option ('-l')
to enable new codepath.

Signed-off-by: Mohammad Abdul Awal <mohammad.abdul.awal@intel.com>
Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/Makefile        |   1 +
 examples/ipsec-secgw/ipsec-secgw.c   | 178 ++++++++------
 examples/ipsec-secgw/ipsec.c         |   2 +-
 examples/ipsec-secgw/ipsec.h         |  23 ++
 examples/ipsec-secgw/ipsec_process.c | 341 +++++++++++++++++++++++++++
 examples/ipsec-secgw/meson.build     |   4 +-
 examples/ipsec-secgw/sa.c            |   7 +-
 7 files changed, 477 insertions(+), 79 deletions(-)
 create mode 100644 examples/ipsec-secgw/ipsec_process.c

diff --git a/examples/ipsec-secgw/Makefile b/examples/ipsec-secgw/Makefile
index 02d41e39a..4d6b4cfa4 100644
--- a/examples/ipsec-secgw/Makefile
+++ b/examples/ipsec-secgw/Makefile
@@ -13,6 +13,7 @@ SRCS-y += sp4.c
 SRCS-y += sp6.c
 SRCS-y += sa.c
 SRCS-y += rt.c
+SRCS-y += ipsec_process.c
 SRCS-y += ipsec-secgw.c
 
 CFLAGS += -gdwarf-2
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 48baa5001..53cfe585d 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -216,19 +216,6 @@ static struct rte_eth_conf port_conf = {
 
 static struct socket_ctx socket_ctx[NB_SOCKETS];
 
-struct traffic_type {
-	const uint8_t *data[MAX_PKT_BURST * 2];
-	struct rte_mbuf *pkts[MAX_PKT_BURST * 2];
-	uint32_t res[MAX_PKT_BURST * 2];
-	uint32_t num;
-};
-
-struct ipsec_traffic {
-	struct traffic_type ipsec;
-	struct traffic_type ip4;
-	struct traffic_type ip6;
-};
-
 static inline void
 prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 {
@@ -245,6 +232,8 @@ prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 			t->ip4.data[t->ip4.num] = nlp;
 			t->ip4.pkts[(t->ip4.num)++] = pkt;
 		}
+		pkt->l2_len = 0;
+		pkt->l3_len = sizeof(struct ip);
 	} else if (eth->ether_type == rte_cpu_to_be_16(ETHER_TYPE_IPv6)) {
 		nlp = (uint8_t *)rte_pktmbuf_adj(pkt, ETHER_HDR_LEN);
 		nlp = RTE_PTR_ADD(nlp, offsetof(struct ip6_hdr, ip6_nxt));
@@ -254,6 +243,8 @@ prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 			t->ip6.data[t->ip6.num] = nlp;
 			t->ip6.pkts[(t->ip6.num)++] = pkt;
 		}
+		pkt->l2_len = 0;
+		pkt->l3_len = sizeof(struct ip6_hdr);
 	} else {
 		/* Unknown/Unsupported type, drop the packet */
 		RTE_LOG(ERR, IPSEC, "Unsupported packet type\n");
@@ -503,10 +494,15 @@ process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
 	n_ip4 = traffic->ip4.num;
 	n_ip6 = traffic->ip6.num;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
-
-	split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+	if (app_sa_prm.enable == 0) {
+		nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.num, MAX_PKT_BURST);
+		split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+	} else {
+		inbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
+	}
 
 	inbound_sp_sa(ipsec_ctx->sp4_ctx, ipsec_ctx->sa_ctx, &traffic->ip4,
 			n_ip4);
@@ -562,20 +558,27 @@ process_pkts_outbound(struct ipsec_ctx *ipsec_ctx,
 
 	outbound_sp(ipsec_ctx->sp6_ctx, &traffic->ip6, &traffic->ipsec);
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.res, traffic->ipsec.num,
-			MAX_PKT_BURST);
-
-	for (i = 0; i < nb_pkts_out; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
-		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-		} else {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
+	if (app_sa_prm.enable == 0) {
+
+		nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.res, traffic->ipsec.num,
+				MAX_PKT_BURST);
+
+		for (i = 0; i < nb_pkts_out; i++) {
+			m = traffic->ipsec.pkts[i];
+			struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
+			if (ip->ip_v == IPVERSION) {
+				idx = traffic->ip4.num++;
+				traffic->ip4.pkts[idx] = m;
+			} else {
+				idx = traffic->ip6.num++;
+				traffic->ip6.pkts[idx] = m;
+			}
 		}
+	} else {
+		outbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.res,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -598,19 +601,26 @@ process_pkts_inbound_nosp(struct ipsec_ctx *ipsec_ctx,
 
 	traffic->ip6.num = 0;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
+	if (app_sa_prm.enable == 0) {
 
-	for (i = 0; i < nb_pkts_in; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
-		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-		} else {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
+		nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.num, MAX_PKT_BURST);
+
+		for (i = 0; i < nb_pkts_in; i++) {
+			m = traffic->ipsec.pkts[i];
+			struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
+			if (ip->ip_v == IPVERSION) {
+				idx = traffic->ip4.num++;
+				traffic->ip4.pkts[idx] = m;
+			} else {
+				idx = traffic->ip6.num++;
+				traffic->ip6.pkts[idx] = m;
+			}
 		}
+	} else {
+		inbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -642,20 +652,28 @@ process_pkts_outbound_nosp(struct ipsec_ctx *ipsec_ctx,
 	traffic->ip6.num = 0;
 	traffic->ipsec.num = n;
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.res, traffic->ipsec.num,
-			MAX_PKT_BURST);
+	if (app_sa_prm.enable == 0) {
 
-	/* They all sue the same SA (ip4 or ip6 tunnel) */
-	m = traffic->ipsec.pkts[i];
-	ip = rte_pktmbuf_mtod(m, struct ip *);
-	if (ip->ip_v == IPVERSION) {
-		traffic->ip4.num = nb_pkts_out;
-		for (i = 0; i < nb_pkts_out; i++)
-			traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+		nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.res, traffic->ipsec.num,
+				MAX_PKT_BURST);
+
+		/* They all sue the same SA (ip4 or ip6 tunnel) */
+		m = traffic->ipsec.pkts[0];
+		ip = rte_pktmbuf_mtod(m, struct ip *);
+		if (ip->ip_v == IPVERSION) {
+			traffic->ip4.num = nb_pkts_out;
+			for (i = 0; i < nb_pkts_out; i++)
+				traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+		} else {
+			traffic->ip6.num = nb_pkts_out;
+			for (i = 0; i < nb_pkts_out; i++)
+				traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+		}
 	} else {
-		traffic->ip6.num = nb_pkts_out;
-		traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+		outbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.res,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -856,25 +874,31 @@ drain_inbound_crypto_queues(const struct lcore_conf *qconf,
 	uint32_t n;
 	struct ipsec_traffic trf;
 
-	/* dequeue packets from crypto-queue */
-	n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+	if (app_sa_prm.enable == 0) {
+
+		/* dequeue packets from crypto-queue */
+		n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
 			RTE_DIM(trf.ipsec.pkts));
-	if (n == 0)
-		return;
 
-	trf.ip4.num = 0;
-	trf.ip6.num = 0;
+		trf.ip4.num = 0;
+		trf.ip6.num = 0;
 
-	/* split traffic by ipv4-ipv6 */
-	split46_traffic(&trf, trf.ipsec.pkts, n);
+		/* split traffic by ipv4-ipv6 */
+		split46_traffic(&trf, trf.ipsec.pkts, n);
+	} else
+		ipsec_cqp_process(ctx, &trf);
 
 	/* process ipv4 packets */
-	inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
-	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	if (trf.ip4.num != 0) {
+		inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
+		route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	}
 
 	/* process ipv6 packets */
-	inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
-	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	if (trf.ip6.num != 0) {
+		inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
+		route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	}
 }
 
 static void
@@ -884,23 +908,27 @@ drain_outbound_crypto_queues(const struct lcore_conf *qconf,
 	uint32_t n;
 	struct ipsec_traffic trf;
 
-	/* dequeue packets from crypto-queue */
-	n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+	if (app_sa_prm.enable == 0) {
+
+		/* dequeue packets from crypto-queue */
+		n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
 			RTE_DIM(trf.ipsec.pkts));
-	if (n == 0)
-		return;
 
-	trf.ip4.num = 0;
-	trf.ip6.num = 0;
+		trf.ip4.num = 0;
+		trf.ip6.num = 0;
 
-	/* split traffic by ipv4-ipv6 */
-	split46_traffic(&trf, trf.ipsec.pkts, n);
+		/* split traffic by ipv4-ipv6 */
+		split46_traffic(&trf, trf.ipsec.pkts, n);
+	} else
+		ipsec_cqp_process(ctx, &trf);
 
 	/* process ipv4 packets */
-	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	if (trf.ip4.num != 0)
+		route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
 
 	/* process ipv6 packets */
-	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	if (trf.ip6.num != 0)
+		route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
 }
 
 /* main processing loop */
diff --git a/examples/ipsec-secgw/ipsec.c b/examples/ipsec-secgw/ipsec.c
index 8bf928a23..8e7fc3639 100644
--- a/examples/ipsec-secgw/ipsec.c
+++ b/examples/ipsec-secgw/ipsec.c
@@ -39,7 +39,7 @@ set_ipsec_conf(struct ipsec_sa *sa, struct rte_security_ipsec_xform *ipsec)
 	ipsec->esn_soft_limit = IPSEC_OFFLOAD_ESN_SOFTLIMIT;
 }
 
-static inline int
+int
 create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa)
 {
 	struct rte_cryptodev_info cdev_info;
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index b089fe54b..f0e2e79af 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -190,6 +190,20 @@ struct cnt_blk {
 	uint32_t cnt;
 } __attribute__((packed));
 
+struct traffic_type {
+	const uint8_t *data[MAX_PKT_BURST * 2];
+	struct rte_mbuf *pkts[MAX_PKT_BURST * 2];
+	struct ipsec_sa *saptr[MAX_PKT_BURST * 2];
+	uint32_t res[MAX_PKT_BURST * 2];
+	uint32_t num;
+};
+
+struct ipsec_traffic {
+	struct traffic_type ipsec;
+	struct traffic_type ip4;
+	struct traffic_type ip6;
+};
+
 uint16_t
 ipsec_inbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint16_t nb_pkts, uint16_t len);
@@ -206,6 +220,12 @@ uint16_t
 ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint16_t len);
 
+void
+ipsec_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf);
+
+void
+ipsec_cqp_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf);
+
 static inline uint16_t
 ipsec_metadata_size(void)
 {
@@ -283,4 +303,7 @@ add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
 void
 enqueue_cop_burst(struct cdev_qp *cqp);
 
+int
+create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/ipsec_process.c b/examples/ipsec-secgw/ipsec_process.c
new file mode 100644
index 000000000..7ab378f6a
--- /dev/null
+++ b/examples/ipsec-secgw/ipsec_process.c
@@ -0,0 +1,341 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2017 Intel Corporation
+ */
+#include <sys/types.h>
+#include <netinet/in.h>
+#include <netinet/ip.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_log.h>
+#include <rte_cryptodev.h>
+#include <rte_ethdev.h>
+#include <rte_mbuf.h>
+
+#include "ipsec.h"
+
+#define SATP_OUT_IPV4(t)	\
+	((((t) & RTE_IPSEC_SATP_MODE_MASK) == RTE_IPSEC_SATP_MODE_TRANS && \
+	(((t) & RTE_IPSEC_SATP_IPV_MASK) == RTE_IPSEC_SATP_IPV4)) || \
+	((t) & RTE_IPSEC_SATP_MODE_MASK) == RTE_IPSEC_SATP_MODE_TUNLV4)
+
+
+/* helper routine to free bulk of packets */
+static inline void
+free_pkts(struct rte_mbuf *mb[], uint32_t n)
+{
+	uint32_t i;
+
+	for (i = 0; i != n; i++)
+		rte_pktmbuf_free(mb[i]);
+}
+
+/* helper routine to free bulk of crypto-ops and related packets */
+static inline void
+free_cops(struct rte_crypto_op *cop[], uint32_t n)
+{
+	uint32_t i;
+
+	for (i = 0; i != n; i++)
+		rte_pktmbuf_free(cop[i]->sym->m_src);
+}
+
+/* helper routine to enqueue bulk of crypto ops */
+static inline void
+enqueue_cop_bulk(struct cdev_qp *cqp, struct rte_crypto_op *cop[], uint32_t num)
+{
+	uint32_t i, k, len, n;
+
+	len = cqp->len;
+
+	/*
+	 * if cqp is empty and we have enough ops,
+	 * then queue them to the PMD straightway.
+	 */
+	if (num >= RTE_DIM(cqp->buf) * 3 / 4 && len == 0) {
+		n = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp, cop, num);
+		cqp->in_flight += n;
+		free_cops(cop + n, num - n);
+		return;
+	}
+
+	k = 0;
+
+	do {
+		n = RTE_DIM(cqp->buf) - len;
+		n = RTE_MIN(num - k, n);
+
+		/* put packets into cqp */
+		for (i = 0; i != n; i++)
+			cqp->buf[len + i] = cop[k + i];
+
+		len += n;
+		k += n;
+
+		/* if cqp is full then, enqueue crypto-ops to PMD */
+		if (len == RTE_DIM(cqp->buf)) {
+			n = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp,
+					cqp->buf, len);
+			cqp->in_flight += n;
+			free_cops(cqp->buf + n, len - n);
+			len = 0;
+		}
+
+
+	} while (k != num);
+
+	cqp->len = len;
+}
+
+static inline int
+fill_ipsec_session(struct rte_ipsec_session *ss, const struct ipsec_sa *sa)
+{
+	/* setup crypto section */
+	if (ss->type == RTE_SECURITY_ACTION_TYPE_NONE) {
+		ss->crypto.ses = sa->crypto_session;
+	/* setup session action type */
+	} else {
+		ss->security.ses = sa->sec_session;
+		ss->security.ctx = sa->security_ctx;
+		ss->security.ol_flags = sa->ol_flags;
+	}
+
+	return rte_ipsec_session_prepare(ss);
+}
+
+/*
+ * group input packets byt the SA they belong to.
+ */
+static uint32_t
+sa_group(struct ipsec_sa *sa_ptr[], struct rte_mbuf *pkts[],
+	struct rte_ipsec_group grp[], uint32_t num)
+{
+	uint32_t i, n, spi;
+	void *sa;
+	void * const nosa = &spi;
+
+	sa = nosa;
+	for (i = 0, n = 0; i != num; i++) {
+
+		if (sa != sa_ptr[i]) {
+			grp[n].cnt = pkts + i - grp[n].m;
+			n += (sa != nosa);
+			grp[n].id.ptr = sa_ptr[i];
+			grp[n].m = pkts + i;
+			sa = sa_ptr[i];
+		}
+	}
+
+	/* terminate last group */
+	if (sa != nosa) {
+		grp[n].cnt = pkts + i - grp[n].m;
+		n++;
+	}
+
+	return n;
+}
+
+/*
+ * helper function, splits processed packets into ipv4/ipv6 traffic.
+ */
+static inline void
+copy_to_trf(struct ipsec_traffic *trf, uint64_t satp, struct rte_mbuf *mb[],
+	uint32_t num)
+{
+	uint32_t j, ofs, s;
+	struct traffic_type *out;
+
+	/*
+	 * determine traffic type(ipv4/ipv6) and offset for ACL classify
+	 * based on SA type
+	 */
+	if ((satp & RTE_IPSEC_SATP_DIR_MASK) == RTE_IPSEC_SATP_DIR_IB) {
+		if ((satp & RTE_IPSEC_SATP_IPV_MASK) == RTE_IPSEC_SATP_IPV4) {
+			out = &trf->ip4;
+			ofs = offsetof(struct ip, ip_p);
+		} else {
+			out = &trf->ip6;
+			ofs = offsetof(struct ip6_hdr, ip6_nxt);
+		}
+	} else if (SATP_OUT_IPV4(satp)) {
+		out = &trf->ip4;
+		ofs = offsetof(struct ip, ip_p);
+	} else {
+		out = &trf->ip6;
+		ofs = offsetof(struct ip6_hdr, ip6_nxt);
+	}
+
+	for (j = 0, s = out->num; j != num; j++) {
+		out->data[s + j] = rte_pktmbuf_mtod_offset(mb[j],
+				void *, ofs);
+		out->pkts[s + j] = mb[j];
+	}
+
+	out->num += num;
+}
+
+/*
+ * Process ipsec packets.
+ * If packet belong to SA that is subject of inline-crypto,
+ * then process it immediately.
+ * Otherwise do necessary preparations and queue it to related
+ * crypto-dev queue.
+ */
+void
+ipsec_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf)
+{
+	uint64_t satp;
+	uint32_t i, j, k, n;
+	struct ipsec_sa *sa;
+	struct ipsec_mbuf_metadata *priv;
+	struct rte_ipsec_group *pg;
+	struct rte_ipsec_session *ips;
+	struct cdev_qp *cqp;
+	struct rte_crypto_op *cop[RTE_DIM(trf->ipsec.pkts)];
+	struct rte_ipsec_group grp[RTE_DIM(trf->ipsec.pkts)];
+
+	n = sa_group(trf->ipsec.saptr, trf->ipsec.pkts, grp, trf->ipsec.num);
+
+	for (i = 0; i != n; i++) {
+
+		pg = grp + i;
+		sa = pg->id.ptr;
+
+		/* no valid SA found */
+		if (sa == NULL)
+			k = 0;
+
+		ips = &sa->ips;
+		satp = rte_ipsec_sa_type(ips->sa);
+
+		/* no valid HW session for that SA, try to create one */
+		if (ips->crypto.ses == NULL &&
+				(create_session(ctx, sa) != 0 ||
+				fill_ipsec_session(ips, sa) != 0))
+			k = 0;
+
+		/* process packets inline */
+		else if (sa->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				sa->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL) {
+
+			/*
+			 * This is just to satisfy inbound_sa_check()
+			 * and get_hop_for_offload_pkt().
+			 * Should be removed in future.
+			 */
+			for (j = 0; j != pg->cnt; j++) {
+				priv = get_priv(pg->m[j]);
+				priv->sa = sa;
+			}
+
+			k = rte_ipsec_pkt_process(ips, pg->m, pg->cnt);
+			copy_to_trf(trf, satp, pg->m, k);
+
+		/* enqueue packets to crypto dev */
+		} else {
+
+			cqp = &ctx->tbl[sa->cdev_id_qp];
+
+			/* for that app each mbuf has it's own crypto op */
+			for (j = 0; j != pg->cnt; j++) {
+				priv = get_priv(pg->m[j]);
+				cop[j] = &priv->cop;
+				/*
+				 * this is just to satisfy inbound_sa_check()
+				 * should be removed in future.
+				 */
+				priv->sa = sa;
+			}
+
+			/* prepare and enqueue crypto ops */
+			k = rte_ipsec_pkt_crypto_prepare(ips, pg->m, cop,
+				pg->cnt);
+			if (k != 0)
+				enqueue_cop_bulk(cqp, cop, k);
+		}
+
+		/* drop packets that cannot be enqueued/processed */
+		if (k != pg->cnt)
+			free_pkts(pg->m + k, pg->cnt - k);
+	}
+}
+
+static inline uint32_t
+cqp_dequeue(struct cdev_qp *cqp, struct rte_crypto_op *cop[], uint32_t num)
+{
+	uint32_t n;
+
+	if (cqp->in_flight == 0)
+		return 0;
+
+	n = rte_cryptodev_dequeue_burst(cqp->id, cqp->qp, cop, num);
+	RTE_ASSERT(cqp->in_flight >= n);
+	cqp->in_flight -= n;
+
+	return n;
+}
+
+static inline uint32_t
+ctx_dequeue(struct ipsec_ctx *ctx, struct rte_crypto_op *cop[], uint32_t num)
+{
+	uint32_t i, n;
+
+	n = 0;
+
+	for (i = ctx->last_qp; n != num && i != ctx->nb_qps; i++)
+		n += cqp_dequeue(ctx->tbl + i, cop + n, num - n);
+
+	for (i = 0; n != num && i != ctx->last_qp; i++)
+		n += cqp_dequeue(ctx->tbl + i, cop + n, num - n);
+
+	ctx->last_qp = i;
+	return n;
+}
+
+/*
+ * dequeue packets from crypto-queues and finalize processing.
+ */
+void
+ipsec_cqp_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf)
+{
+	uint64_t satp;
+	uint32_t i, k, n, ng;
+	struct rte_ipsec_session *ss;
+	struct traffic_type *out;
+	struct rte_ipsec_group *pg;
+	struct rte_crypto_op *cop[RTE_DIM(trf->ipsec.pkts)];
+	struct rte_ipsec_group grp[RTE_DIM(trf->ipsec.pkts)];
+
+	trf->ip4.num = 0;
+	trf->ip6.num = 0;
+
+	out = &trf->ipsec;
+
+	/* dequeue completed crypto-ops */
+	n = ctx_dequeue(ctx, cop, RTE_DIM(cop));
+	if (n == 0)
+		return;
+
+	/* group them by ipsec session */
+	ng = rte_ipsec_pkt_crypto_group((const struct rte_crypto_op **)
+		(uintptr_t)cop, out->pkts, grp, n);
+
+	/* process each group of packets */
+	for (i = 0; i != ng; i++) {
+
+		pg = grp + i;
+		ss = pg->id.ptr;
+		satp = rte_ipsec_sa_type(ss->sa);
+
+		k = rte_ipsec_pkt_process(ss, pg->m, pg->cnt);
+		copy_to_trf(trf, satp, pg->m, k);
+
+		/* free bad packets, if any */
+		free_pkts(pg->m + k, pg->cnt - k);
+
+		n -= pg->cnt;
+	}
+
+	/* we should never have packet with unknown SA here */
+	RTE_VERIFY(n == 0);
+}
diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
index 31f68fee2..81c146ebc 100644
--- a/examples/ipsec-secgw/meson.build
+++ b/examples/ipsec-secgw/meson.build
@@ -9,6 +9,6 @@
 deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
 allow_experimental_apis = true
 sources = files(
-	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
-	'rt.c', 'sa.c', 'sp4.c', 'sp6.c'
+	'esp.c', 'ipsec.c', 'ipsec_process.c', 'ipsec-secgw.c',
+	'parser.c', 'rt.c', 'sa.c', 'sp4.c', 'sp6.c'
 )
diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index 04eb8bce8..49da966a3 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -1164,10 +1164,15 @@ int
 inbound_sa_check(struct sa_ctx *sa_ctx, struct rte_mbuf *m, uint32_t sa_idx)
 {
 	struct ipsec_mbuf_metadata *priv;
+	struct ipsec_sa *sa;
 
 	priv = get_priv(m);
+	sa = priv->sa;
+	if (sa != NULL)
+		return (sa_ctx->sa[sa_idx].spi == sa->spi);
 
-	return (sa_ctx->sa[sa_idx].spi == priv->sa->spi);
+	RTE_LOG(ERR, IPSEC, "SA not saved in private data\n");
+	return 0;
 }
 
 static inline void
-- 
2.17.1

[PATCH v3 8/8] examples/ipsec-secgw: add scripts for functional test

[Konstantin Ananyev]

The purpose of these scripts is to automate ipsec-secgw functional testing.
The scripts require two machines (SUT and DUT) connected through
at least 2 NICs and running linux (so far tested only on Ubuntu 18.04).
Introduced test-cases for the following scenarios:
- Transport/Tunnel modes
- AES-CBC SHA1
- AES-GCM
- ESN on/off
- legacy/librte_ipsec code path

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/test/common_defs.sh      | 149 ++++++++++++++++++
 examples/ipsec-secgw/test/data_rxtx.sh        | 118 ++++++++++++++
 examples/ipsec-secgw/test/linux_test4.sh      |  63 ++++++++
 examples/ipsec-secgw/test/linux_test6.sh      |  63 ++++++++
 examples/ipsec-secgw/test/run_test.sh         |  80 ++++++++++
 .../test/trs_aescbc_sha1_common_defs.sh       |  69 ++++++++
 .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  67 ++++++++
 .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/trs_aescbc_sha1_esn_defs.sh          |  66 ++++++++
 .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
 .../test/trs_aesgcm_common_defs.sh            |  60 +++++++
 examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  66 ++++++++
 .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  66 ++++++++
 .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
 .../test/tun_aescbc_sha1_common_defs.sh       |  68 ++++++++
 .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  70 ++++++++
 .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/tun_aescbc_sha1_esn_defs.sh          |  70 ++++++++
 .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
 .../test/tun_aesgcm_common_defs.sh            |  60 +++++++
 examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  70 ++++++++
 .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  70 ++++++++
 .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
 25 files changed, 1315 insertions(+)
 create mode 100644 examples/ipsec-secgw/test/common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test4.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test6.sh
 create mode 100644 examples/ipsec-secgw/test/run_test.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh

diff --git a/examples/ipsec-secgw/test/common_defs.sh b/examples/ipsec-secgw/test/common_defs.sh
new file mode 100644
index 000000000..557381679
--- /dev/null
+++ b/examples/ipsec-secgw/test/common_defs.sh
@@ -0,0 +1,149 @@
+#! /bin/bash
+
+#check that env vars are properly defined
+
+#check SGW_PATH
+if [[ -z "${SGW_PATH}" || ! -x ${SGW_PATH} ]]; then
+	echo "SGW_PATH is invalid"
+	exit 127
+fi
+
+#check ETH_DEV
+if [[ -z "${ETH_DEV}" ]]; then
+	echo "ETH_DEV is invalid"
+	exit 127
+fi
+
+#setup SGW_LCORE
+SGW_LCORE=${SGW_LCORE:-0}
+
+#check that REMOTE_HOST is reachable
+ssh ${REMOTE_HOST} echo
+st=$?
+if [[ $st -ne 0 ]]; then
+	echo "host ${REMOTE_HOST} is not reachable"
+	exit $st
+fi
+
+#get ether addr of REMOTE_HOST
+REMOTE_MAC=`ssh ${REMOTE_HOST} ip addr show dev ${REMOTE_IFACE}`
+st=$?
+REMOTE_MAC=`echo ${REMOTE_MAC} | sed -e 's/^.*ether //' -e 's/ brd.*$//'`
+if [[ $st -ne 0 || -z "${REMOTE_MAC}" ]]; then
+	echo "coouldn't retrieve ether addr from ${REMOTE_IFACE}"
+	exit 127
+fi
+
+LOCAL_IFACE=dtap0
+
+LOCAL_MAC="00:64:74:61:70:30"
+
+REMOTE_IPV4=192.168.31.14
+LOCAL_IPV4=192.168.31.92
+
+REMOTE_IPV6=fd12:3456:789a:0031:0000:0000:0000:0014
+LOCAL_IPV6=fd12:3456:789a:0031:0000:0000:0000:0092
+
+DPDK_PATH=${RTE_SDK:-${PWD}}
+DPDK_BUILD=${RTE_TARGET:-x86_64-native-linuxapp-gcc}
+
+SGW_OUT_FILE=./ipsec-secgw.out1
+
+SGW_CMD_EAL_PRM="--lcores=${SGW_LCORE} -n 4 ${ETH_DEV}"
+SGW_CMD_CFG="(0,0,${SGW_LCORE}),(1,0,${SGW_LCORE})"
+SGW_CMD_PRM="-p 0x3 -u 1 -P --config=\"${SGW_CMD_CFG}\""
+
+SGW_CFG_FILE=$(tempfile)
+
+# configure local host/ifaces
+config_local_iface()
+{
+	ifconfig ${LOCAL_IFACE} ${LOCAL_IPV4}/24 mtu 1400 up
+	ifconfig ${LOCAL_IFACE}
+
+	ip neigh flush dev ${LOCAL_IFACE}
+	ip neigh add ${REMOTE_IPV4} dev ${LOCAL_IFACE} lladdr ${REMOTE_MAC}
+	ip neigh show dev ${LOCAL_IFACE}
+}
+
+config6_local_iface()
+{
+	config_local_iface
+
+	sysctl -w net.ipv6.conf.${LOCAL_IFACE}.disable_ipv6=0
+	ip addr add  ${LOCAL_IPV6}/64 dev ${LOCAL_IFACE}
+
+	sysctl -w net.ipv6.conf.${LOCAL_IFACE}.mtu=1300
+
+	ip -6 neigh add ${REMOTE_IPV6} dev ${LOCAL_IFACE} lladdr ${REMOTE_MAC}
+	ip neigh show dev ${LOCAL_IFACE}
+}
+
+#configure remote host/iface
+config_remote_iface()
+{
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE} down
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE} ${REMOTE_IPV4}/24 up
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} ip neigh flush dev ${REMOTE_IFACE}
+
+	# by some reason following ip neigh doesn't work for me here properly:
+	#ssh ${REMOTE_HOST} ip neigh add ${LOCAL_IPV4} \
+	#		dev ${REMOTE_IFACE} lladr ${LOCAL_MAC}
+	# so used arp instead.
+	ssh ${REMOTE_HOST} arp -i ${REMOTE_IFACE} -s ${LOCAL_IPV4} ${LOCAL_MAC}
+	ssh ${REMOTE_HOST} ip neigh show dev ${REMOTE_IFACE}
+}
+
+config6_remote_iface()
+{
+	config_remote_iface
+
+	ssh ${REMOTE_HOST} sysctl -w \
+		net.ipv6.conf.${REMOTE_IFACE}.disable_ipv6=0
+	ssh ${REMOTE_HOST} ip addr add  ${REMOTE_IPV6}/64 dev ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} ip -6 neigh add ${LOCAL_IPV6} \
+		dev ${REMOTE_IFACE} lladdr ${LOCAL_MAC}
+	ssh ${REMOTE_HOST} ip neigh show dev ${REMOTE_IFACE}
+}
+
+#configure remote and local host/iface
+config_iface()
+{
+	config_local_iface
+	config_remote_iface
+}
+
+config6_iface()
+{
+	config6_local_iface
+	config6_remote_iface
+}
+
+#start ipsec-secgw
+secgw_start()
+{
+	SGW_EXEC_FILE=$(tempfile)
+	cat <<EOF > ${SGW_EXEC_FILE}
+${SGW_PATH} ${SGW_CMD_EAL_PRM} ${CRYPTO_DEV} \
+--vdev="net_tap0,mac=fixed" \
+-- ${SGW_CMD_PRM} ${SGW_CMD_XPRM} -f ${SGW_CFG_FILE} > \
+${SGW_OUT_FILE} 2>&1 &
+p=\$!
+echo \$p
+EOF
+
+	cat ${SGW_EXEC_FILE}
+	SGW_PID=`/bin/bash -x ${SGW_EXEC_FILE}`
+	sleep 1
+}
+
+#stop ipsec-secgw and cleanup
+secgw_stop()
+{
+	kill ${SGW_PID}
+	rm -f ${SGW_EXEC_FILE}
+	rm -f ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/data_rxtx.sh b/examples/ipsec-secgw/test/data_rxtx.sh
new file mode 100644
index 000000000..2072fb900
--- /dev/null
+++ b/examples/ipsec-secgw/test/data_rxtx.sh
@@ -0,0 +1,118 @@
+#! /bin/bash
+
+TCP_PORT=22222
+
+ping_test1()
+{
+	dst=$1
+
+	i=0
+	st=0
+	while [[ $i -ne 1200 && $st -eq 0 ]];
+	do
+		let i++
+		ping -c 1 -s ${i} ${dst}
+		st=$?
+	done
+
+	if [[ $st -ne 0 ]]; then
+		echo "ERROR: $0 failed for dst=${dst}, sz=${i}"
+	fi
+	return $st;
+}
+
+ping6_test1()
+{
+	dst=$1
+
+	i=0
+	st=0
+	while [[ $i -ne 1200 && $st -eq 0 ]];
+	do
+		let i++
+		ping6 -c 1 -s ${i} ${dst}
+		st=$?
+	done
+
+	if [[ $st -ne 0 ]]; then
+		echo "ERROR: $0 failed for dst=${dst}, sz=${i}"
+	fi
+	return $st;
+}
+
+scp_test1()
+{
+	for sz in 1234 23456 345678 4567890 56789102 ; do
+		x=$(tempfile)
+		dd if=/dev/urandom of=${x}.in bs=${sz} count=1
+		scp ${x} ${REMOTE_IPV4}:${x}
+		scp ${REMOTE_IPV4}:${x} ${x}.copy1
+		diff -u ${x} ${x}.copy1
+		st=$?
+		rm -f ${x} ${x}.out
+		ssh ${REMOTE_HOST} rm -f ${x}
+		if [[ $st -ne 0 ]]; then
+			return $st
+		fi
+	done
+
+	return 0;
+}
+
+tcp_test1()
+{
+	for sz in 1234 23456 345678 4567890 56789102 ; do
+		x=`basename $0`.${sz}
+		rcmd=$(tempfile)
+		lcmd=$(tempfile)
+		dd if=/dev/urandom of=${x} bs=${sz} count=1
+		echo "nc -l ${REMOTE_IPV4} ${TCP_PORT} > ${x}; \
+			nc -q 0 ${LOCAL_IPV4} ${TCP_PORT} < ${x}; \
+			rm -f ${x}" > ${rcmd}
+		echo "nc -l ${LOCAL_IPV4} ${TCP_PORT} > ${x}.copy" > ${lcmd}
+		scp ${rcmd} ${REMOTE_HOST}:${rcmd}
+		ssh ${REMOTE_HOST} /bin/bash -x ${rcmd} &
+		/bin/bash -x ${lcmd} &
+		sleep 1
+		nc -q 0 ${REMOTE_IPV4} ${TCP_PORT} < ${x}
+		wait
+		diff -u ${x} ${x}.copy
+		st=$?
+		rm -f ${x} ${x}.copy ${rcmd} ${lcmd}
+		ssh  ${REMOTE_HOST} rm -f ${rcmd}
+		if [[ $st -ne 0 ]]; then
+			return $st
+		fi
+	done
+
+	return $st;
+}
+
+tcp6_test1()
+{
+	for sz in 1234 23456 345678 4567890 56789102 ; do
+		x=`basename $0`.${sz}
+		rcmd=$(tempfile)
+		lcmd=$(tempfile)
+		dd if=/dev/urandom of=${x} bs=${sz} count=1
+		echo "nc -l ${REMOTE_IPV6} ${TCP_PORT} > ${x}; \
+			nc -q 0 ${LOCAL_IPV6} ${TCP_PORT} < ${x}; \
+			rm -f ${x}" > ${rcmd}
+		echo "nc -l ${LOCAL_IPV6} ${TCP_PORT} > ${x}.copy" > ${lcmd}
+		scp ${rcmd} ${REMOTE_HOST}:${rcmd}
+		ssh ${REMOTE_HOST} /bin/bash -x ${rcmd} &
+		/bin/bash -x ${lcmd} &
+		sleep 1
+		nc -q 0 ${REMOTE_IPV6} ${TCP_PORT} < ${x}
+		wait
+		diff -u ${x} ${x}.copy
+		st=$?
+		rm -f ${x} ${x}.copy ${rcmd} ${lcmd}
+		ssh  ${REMOTE_HOST} rm -f ${rcmd}
+		if [[ $st -ne 0 ]]; then
+			return $st
+		fi
+	done
+
+	return $st;
+}
diff --git a/examples/ipsec-secgw/test/linux_test4.sh b/examples/ipsec-secgw/test/linux_test4.sh
new file mode 100644
index 000000000..039fc7c4a
--- /dev/null
+++ b/examples/ipsec-secgw/test/linux_test4.sh
@@ -0,0 +1,63 @@
+#! /bin/bash
+
+# usage:  /bin/bash linux_test4.sh <ipsec_mode>
+# for list of available modes please refer to run_test.sh.
+# ipsec-secgw (IPv4 mode) functional test script.
+#
+# Note that for most of them you required appropriate crypto PMD/device
+# to be avaialble.
+# Also user has to setup properly the following environment variables:
+#  SGW_PATH - path to the ipsec-secgw binary to test
+#  REMOTE_HOST - ip/hostname of the DUT
+#  REMOTE_IFACE - iface name for the test-port on DUT
+#  ETH_DEV - ethernet device to be used on SUT by DPDK ('-w <pci-id>')
+# Also user can optonally setup:
+#  SGW_LCORE - lcore to run ipsec-secgw on (default value is 0)
+#  CRYPTO_DEV - crypto device to be used ('-w <pci-id>')
+#  if none specified appropriate vdevs will be created by the scrit
+#
+# The purpose of the script is to automate ipsec-secgw testing
+# using another system running linux as a DUT.
+# It expects that SUT and DUT are connected through at least 2 NICs.
+# One NIC is expected to be managed by linux both machines,
+# and will be used as a control path
+# (make sure user from SUT can ssh to DUT without entering password).
+# Second NIC (test-port) should be reserved for DPDK on SUT,
+# and should be managed by linux on DUT.
+# The script starts ipsec-secgw with 2 NIC devices: test-port and tap vdev.
+# Then configures local tap iface and remote iface and ipsec policies
+# in the following way:
+# traffic going over test-port in both directions has to be
+# protected by ipsec.
+# raffic going over TAP in both directions doesn't have to be protected.
+# I.E:
+# DUT OS(NIC1)--(ipsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS
+# SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(ipsec)-->(NIC1)DUT OS
+# Then tries to perorm some data transfer using the scheme decribed above.
+#
+
+DIR=`dirname $0`
+MODE=$1
+
+ . ${DIR}/common_defs.sh
+ . ${DIR}/${MODE}_defs.sh
+
+config_secgw
+
+secgw_start
+
+config_iface
+
+config_remote_xfrm
+
+ . ${DIR}/data_rxtx.sh
+
+ping_test1 ${REMOTE_IPV4}
+st=$?
+if [[ $st -eq 0 ]]; then
+	tcp_test1
+	st=$?
+fi
+
+secgw_stop
+exit $st
diff --git a/examples/ipsec-secgw/test/linux_test6.sh b/examples/ipsec-secgw/test/linux_test6.sh
new file mode 100644
index 000000000..425cd66a3
--- /dev/null
+++ b/examples/ipsec-secgw/test/linux_test6.sh
@@ -0,0 +1,63 @@
+#! /bin/bash
+
+# usage:  /bin/bash linux_test6.sh <ipsec_mode>
+# for list of available modes please refer to run_test.sh.
+# ipsec-secgw (IPv6 mode) functional test script.
+#
+# Note that for most of them you required appropriate crypto PMD/device
+# to be avaialble.
+# Also user has to setup properly the following environment variables:
+#  SGW_PATH - path to the ipsec-secgw binary to test
+#  REMOTE_HOST - ip/hostname of the DUT
+#  REMOTE_IFACE - iface name for the test-port on DUT
+#  ETH_DEV - ethernet device to be used on SUT by DPDK ('-w <pci-id>')
+# Also user can optonally setup:
+#  SGW_LCORE - lcore to run ipsec-secgw on (default value is 0)
+#  CRYPTO_DEV - crypto device to be used ('-w <pci-id>')
+#  if none specified appropriate vdevs will be created by the scrit
+#
+# The purpose of the script is to automate ipsec-secgw testing
+# using another system running linux as a DUT.
+# It expects that SUT and DUT are connected through at least 2 NICs.
+# One NIC is expected to be managed by linux both machines,
+# and will be used as a control path
+# (make sure user from SUT can ssh to DUT without entering password).
+# Second NIC (test-port) should be reserved for DPDK on SUT,
+# and should be managed by linux on DUT.
+# The script starts ipsec-secgw with 2 NIC devices: test-port and tap vdev.
+# Then configures local tap iface and remote iface and ipsec policies
+# in the following way:
+# traffic going over test-port in both directions has to be
+# protected by ipsec.
+# raffic going over TAP in both directions doesn't have to be protected.
+# I.E:
+# DUT OS(NIC1)--(ipsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS
+# SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(ipsec)-->(NIC1)DUT OS
+# Then tries to perorm some data transfer using the scheme decribed above.
+#
+
+DIR=`dirname $0`
+MODE=$1
+
+ . ${DIR}/common_defs.sh
+ . ${DIR}/${MODE}_defs.sh
+
+config_secgw
+
+secgw_start
+
+config6_iface
+
+config6_remote_xfrm
+
+ . ${DIR}/data_rxtx.sh
+
+ping6_test1 ${REMOTE_IPV6}
+st=$?
+if [[ $st -eq 0 ]]; then
+	tcp6_test1
+	st=$?
+fi
+
+secgw_stop
+exit $st
diff --git a/examples/ipsec-secgw/test/run_test.sh b/examples/ipsec-secgw/test/run_test.sh
new file mode 100644
index 000000000..6dc0ce54e
--- /dev/null
+++ b/examples/ipsec-secgw/test/run_test.sh
@@ -0,0 +1,80 @@
+#! /bin/bash
+
+# usage: /bin/bash run_test.sh [-46]
+# Run all defined linux_test[4,6].sh test-cases one by one
+# user has to setup properly the following environment variables:
+#  SGW_PATH - path to the ipsec-secgw binary to test
+#  REMOTE_HOST - ip/hostname of the DUT
+#  REMOTE_IFACE - iface name for the test-port on DUT
+#  ETH_DEV - ethernet device to be used on SUT by DPDK ('-w <pci-id>')
+# Also user can optonally setup:
+#  SGW_LCORE - lcore to run ipsec-secgw on (default value is 0)
+#  CRYPTO_DEV - crypto device to be used ('-w <pci-id>')
+#  if none specified appropriate vdevs will be created by the scrit
+# refer to linux_test1.sh for more information
+
+# All supported modes to test.
+# naming convention:
+# 'old' means that ipsec-secgw will run in legacy (non-librte_ipsec mode)
+# 'tun/trs' refer to tunnel/transport mode respectively
+LINUX_TEST="tun_aescbc_sha1 \
+tun_aescbc_sha1_esn \
+tun_aescbc_sha1_esn_atom \
+tun_aesgcm \
+tun_aesgcm_esn \
+tun_aesgcm_esn_atom \
+trs_aescbc_sha1 \
+trs_aescbc_sha1_esn \
+trs_aescbc_sha1_esn_atom \
+trs_aesgcm \
+trs_aesgcm_esn \
+trs_aesgcm_esn_atom \
+tun_aescbc_sha1_old \
+tun_aesgcm_old \
+trs_aescbc_sha1_old \
+trs_aesgcm_old"
+
+DIR=`dirname $0`
+
+# get input options
+st=0
+run4=0
+run6=0
+while [[ ${st} -eq 0 ]]; do
+	getopts ":46" opt
+	st=$?
+	if [[ "${opt}" == "4" ]]; then
+		run4=1
+	elif [[ "${opt}" == "6" ]]; then
+		run6=1
+	fi
+done
+
+if [[ ${run4} -eq 0 && {run6} -eq 0 ]]; then
+	exit 127
+fi
+
+for i in ${LINUX_TEST}; do
+
+	echo "starting test ${i}"
+
+	st4=0
+	if [[ ${run4} -ne 0 ]]; then
+		/bin/bash ${DIR}/linux_test4.sh ${i}
+		st4=$?
+		echo "test4 ${i} finished with status ${st4}"
+	fi
+
+	st6=0
+	if [[ ${run6} -ne 0 ]]; then
+		/bin/bash ${DIR}/linux_test6.sh ${i}
+		st6=$?
+		echo "test6 ${i} finished with status ${st6}"
+	fi
+
+	let "st = st4 + st6"
+	if [[ $st -ne 0 ]]; then
+		echo "ERROR test ${i} FAILED"
+		exit $st
+	fi
+done
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
new file mode 100644
index 000000000..e2621e0df
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
@@ -0,0 +1,69 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_mb0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#SP in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#sp in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+sa in 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#SA out rules
+sa out 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#SA out rules
+sa out 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
new file mode 100644
index 000000000..d68552fce
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
@@ -0,0 +1,67 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
new file mode 100644
index 000000000..f16222e11
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
new file mode 100644
index 000000000..ce7c977a3
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
@@ -0,0 +1,66 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
new file mode 100644
index 000000000..a3abb6103
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
new file mode 100644
index 000000000..720e807e4
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
@@ -0,0 +1,60 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_gcm0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#SP in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+sa in 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#SA out rules
+sa out 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+sa out 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_defs.sh
new file mode 100644
index 000000000..8382d3d52
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_defs.sh
@@ -0,0 +1,66 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
new file mode 100644
index 000000000..80d8d63b8
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
new file mode 100644
index 000000000..94958d199
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
@@ -0,0 +1,66 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
new file mode 100644
index 000000000..951e6b68f
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
new file mode 100644
index 000000000..4025da232
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
@@ -0,0 +1,68 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_mb0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#sp in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#sp in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${REMOTE_IPV4} dst ${LOCAL_IPV4}
+
+sa in 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${REMOTE_IPV6} dst ${LOCAL_IPV6}
+
+#SA out rules
+sa out 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${LOCAL_IPV4} dst ${REMOTE_IPV4}
+
+sa out 9 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${LOCAL_IPV6} dst ${REMOTE_IPV6}
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
new file mode 100644
index 000000000..18aade3a9
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
new file mode 100644
index 000000000..6b4a82149
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
new file mode 100644
index 000000000..28c1125d6
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
new file mode 100644
index 000000000..3c0d8d1b1
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
new file mode 100644
index 000000000..fba68c6a3
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
@@ -0,0 +1,60 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_gcm0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+#sp in IPv4 rules
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#sp in IPv6 rules
+sp ipv6 in esp protect 9 pri 2 src ${REMOTE_IPV6}/128 dst ${LOCAL_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv6 rules
+sp ipv6 out esp protect 9 pri 2 src ${LOCAL_IPV6}/128 dst ${REMOTE_IPV6}/128 \
+sport 0:65535 dport 0:65535
+sp ipv6 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${REMOTE_IPV4} dst ${LOCAL_IPV4}
+
+sa in 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${REMOTE_IPV6} dst ${LOCAL_IPV6}
+
+#SA out rules
+sa out 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${LOCAL_IPV4} dst ${REMOTE_IPV4}
+
+sa out 9 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv6-tunnel src ${LOCAL_IPV6} dst ${REMOTE_IPV6}
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+rt ipv6 dst ${REMOTE_IPV6}/128 port 0
+rt ipv6 dst ${LOCAL_IPV6}/128 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_defs.sh
new file mode 100644
index 000000000..8ae65321b
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_defs.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
new file mode 100644
index 000000000..dab1460c8
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
new file mode 100644
index 000000000..606232349
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
+
+config6_remote_xfrm()
+{
+	config_remote_xfrm
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp mode tunnel reqid 3
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp mode tunnel reqid 4
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV6} dst ${LOCAL_IPV6} \
+proto esp spi 9 reqid 3 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV6} dst ${REMOTE_IPV6} \
+proto esp spi 9 reqid 4 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh
new file mode 100644
index 000000000..e0a015e21
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_defs.sh
+
+SGW_CMD_XPRM=
-- 
2.17.1

[PATCH v2 2/7] examples/ipsec-secgw: allow to specify neighbor mac address

[Konstantin Ananyev]

In some cases it is useful to allow user to specify destination
ether address for outgoing packets.
This patch adds such ability by introducing new 'neigh' config
file option.

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 21 +++++++--
 examples/ipsec-secgw/ipsec.h       |  3 ++
 examples/ipsec-secgw/parser.c      | 75 ++++++++++++++++++++++++++++++
 examples/ipsec-secgw/parser.h      |  8 ++--
 4 files changed, 99 insertions(+), 8 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index cfc2b05e5..2ed757922 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -104,9 +104,9 @@ static uint16_t nb_txd = IPSEC_SECGW_TX_DESC_DEFAULT;
 #define ETHADDR(a, b, c, d, e, f) (__BYTES_TO_UINT64(a, b, c, d, e, f, 0, 0))
 
 #define ETHADDR_TO_UINT64(addr) __BYTES_TO_UINT64( \
-		addr.addr_bytes[0], addr.addr_bytes[1], \
-		addr.addr_bytes[2], addr.addr_bytes[3], \
-		addr.addr_bytes[4], addr.addr_bytes[5], \
+		(addr)->addr_bytes[0], (addr)->addr_bytes[1], \
+		(addr)->addr_bytes[2], (addr)->addr_bytes[3], \
+		(addr)->addr_bytes[4], (addr)->addr_bytes[5], \
 		0, 0)
 
 /* port/source ethernet addr and destination ethernet addr */
@@ -1188,6 +1188,19 @@ print_ethaddr(const char *name, const struct ether_addr *eth_addr)
 	printf("%s%s", name, buf);
 }
 
+/*
+ * Update destination ethaddr for the port.
+ */
+int
+add_dst_ethaddr(uint16_t port, const struct ether_addr *addr)
+{
+	if (port > RTE_DIM(ethaddr_tbl))
+		return -EINVAL;
+
+	ethaddr_tbl[port].dst = ETHADDR_TO_UINT64(addr);
+	return 0;
+}
+
 /* Check the link status of all ports in up to 9s, and print them finally */
 static void
 check_all_ports_link_status(uint32_t port_mask)
@@ -1564,7 +1577,7 @@ port_init(uint16_t portid, uint64_t req_rx_offloads, uint64_t req_tx_offloads)
 	printf("Configuring device port %u:\n", portid);
 
 	rte_eth_macaddr_get(portid, &ethaddr);
-	ethaddr_tbl[portid].src = ETHADDR_TO_UINT64(ethaddr);
+	ethaddr_tbl[portid].src = ETHADDR_TO_UINT64(&ethaddr);
 	print_ethaddr("Address: ", &ethaddr);
 	printf("\n");
 
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 9b1586f52..580f7876b 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -245,4 +245,7 @@ int
 sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
 		uint64_t *tx_offloads);
 
+int
+add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/parser.c b/examples/ipsec-secgw/parser.c
index 91282ca94..6769a8e6d 100644
--- a/examples/ipsec-secgw/parser.c
+++ b/examples/ipsec-secgw/parser.c
@@ -306,6 +306,30 @@ parse_range(const char *token, uint16_t *low, uint16_t *high)
 	return 0;
 }
 
+#define PARSE_UINT8x16(s, v, l)	                          \
+do {                                                      \
+	char *end;                                        \
+	unsigned long t;                                  \
+	errno = 0;                                        \
+	t = strtoul((s), &end, 16);                       \
+	if (errno != 0 || end[0] != (l) || t > UINT8_MAX) \
+		return -EINVAL;                           \
+	(s) = end + 1;                                    \
+	(v) = t;                                          \
+} while (0)
+
+static int
+parse_mac(const char *str, struct ether_addr *addr)
+{
+	PARSE_UINT8x16(str, addr->addr_bytes[0], ':');
+	PARSE_UINT8x16(str, addr->addr_bytes[1], ':');
+	PARSE_UINT8x16(str, addr->addr_bytes[2], ':');
+	PARSE_UINT8x16(str, addr->addr_bytes[3], ':');
+	PARSE_UINT8x16(str, addr->addr_bytes[4], ':');
+	PARSE_UINT8x16(str, addr->addr_bytes[5], 0);
+	return 0;
+}
+
 /** sp add parse */
 struct cfg_sp_add_cfg_item {
 	cmdline_fixed_string_t sp_keyword;
@@ -444,11 +468,61 @@ cmdline_parse_inst_t cfg_rt_add_rule = {
 	},
 };
 
+/* neigh add parse */
+struct cfg_neigh_add_item {
+	cmdline_fixed_string_t neigh;
+	cmdline_fixed_string_t pstr;
+	uint16_t port;
+	cmdline_fixed_string_t mac;
+};
+
+static void
+cfg_parse_neigh(void *parsed_result, __rte_unused struct cmdline *cl,
+	void *data)
+{
+	int32_t rc;
+	struct cfg_neigh_add_item *res;
+	struct parse_status *st;
+	struct ether_addr mac;
+
+	st = data;
+	res = parsed_result;
+	rc = parse_mac(res->mac, &mac);
+	APP_CHECK(rc == 0, st, "invalid ether addr:%s", res->mac);
+	rc = add_dst_ethaddr(res->port, &mac);
+	APP_CHECK(rc == 0, st, "invalid port numer:%hu", res->port);
+	if (st->status < 0)
+		return;
+}
+
+cmdline_parse_token_string_t cfg_add_neigh_start =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, neigh, "neigh");
+cmdline_parse_token_string_t cfg_add_neigh_pstr =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, pstr, "port");
+cmdline_parse_token_num_t cfg_add_neigh_port =
+	TOKEN_NUM_INITIALIZER(struct cfg_neigh_add_item, port, UINT16);
+cmdline_parse_token_string_t cfg_add_neigh_mac =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, mac, NULL);
+
+cmdline_parse_inst_t cfg_neigh_add_rule = {
+	.f = cfg_parse_neigh,
+	.data = NULL,
+	.help_str = "",
+	.tokens = {
+		(void *)&cfg_add_neigh_start,
+		(void *)&cfg_add_neigh_pstr,
+		(void *)&cfg_add_neigh_port,
+		(void *)&cfg_add_neigh_mac,
+		NULL,
+	},
+};
+
 /** set of cfg items */
 cmdline_parse_ctx_t ipsec_ctx[] = {
 	(cmdline_parse_inst_t *)&cfg_sp_add_rule,
 	(cmdline_parse_inst_t *)&cfg_sa_add_rule,
 	(cmdline_parse_inst_t *)&cfg_rt_add_rule,
+	(cmdline_parse_inst_t *)&cfg_neigh_add_rule,
 	NULL,
 };
 
@@ -474,6 +548,7 @@ parse_cfg_file(const char *cfg_filename)
 	cfg_sp_add_rule.data = &status;
 	cfg_sa_add_rule.data = &status;
 	cfg_rt_add_rule.data = &status;
+	cfg_neigh_add_rule.data = &status;
 
 	do {
 		char oneline[1024];
diff --git a/examples/ipsec-secgw/parser.h b/examples/ipsec-secgw/parser.h
index be02537c5..6b8a10076 100644
--- a/examples/ipsec-secgw/parser.h
+++ b/examples/ipsec-secgw/parser.h
@@ -14,14 +14,14 @@ struct parse_status {
 	char parse_msg[256];
 };
 
-#define	APP_CHECK(exp, status, fmt, ...)				\
+#define	APP_CHECK(exp, st, fmt, ...)					\
 do {									\
 	if (!(exp)) {							\
-		sprintf(status->parse_msg, fmt "\n",			\
+		sprintf((st)->parse_msg, fmt "\n",			\
 			## __VA_ARGS__);				\
-		status->status = -1;					\
+		(st)->status = -1;					\
 	} else								\
-		status->status = 0;					\
+		(st)->status = 0;					\
 } while (0)
 
 #define APP_CHECK_PRESENCE(val, str, status)				\
-- 
2.17.1

[PATCH v2 3/7] examples/ipsec-secgw: fix crypto-op might never get dequeued

[Konstantin Ananyev]

In some cases crypto-ops could never be dequeued from the crypto-device.
The easiest way to reproduce:
start ipsec-secgw with crypto-dev and send to it less then 32 packets.
none packets will be forwarded.
Reason for that is that the application does dequeue() from crypto-queues
only when new packets arrive.
This patch makes sure it calls dequeue() on a regular basis.

Fixes: c64278c0c18b ("examples/ipsec-secgw: rework processing loop")

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 136 ++++++++++++++++++++++++-----
 examples/ipsec-secgw/ipsec.c       |  60 ++++++++-----
 examples/ipsec-secgw/ipsec.h       |  11 +++
 3 files changed, 165 insertions(+), 42 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 2ed757922..62443172a 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -456,38 +456,55 @@ inbound_sp_sa(struct sp_ctx *sp, struct sa_ctx *sa, struct traffic_type *ip,
 	ip->num = j;
 }
 
-static inline void
-process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
-		struct ipsec_traffic *traffic)
+static void
+split46_traffic(struct ipsec_traffic *trf, struct rte_mbuf *mb[], uint32_t num)
 {
+	uint32_t i, n4, n6;
+	struct ip *ip;
 	struct rte_mbuf *m;
-	uint16_t idx, nb_pkts_in, i, n_ip4, n_ip6;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
+	n4 = trf->ip4.num;
+	n6 = trf->ip6.num;
 
-	n_ip4 = traffic->ip4.num;
-	n_ip6 = traffic->ip6.num;
+	for (i = 0; i < num; i++) {
+
+		m = mb[i];
+		ip = rte_pktmbuf_mtod(m, struct ip *);
 
-	/* SP/ACL Inbound check ipsec and ip4 */
-	for (i = 0; i < nb_pkts_in; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
 		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-			traffic->ip4.data[idx] = rte_pktmbuf_mtod_offset(m,
+			trf->ip4.pkts[n4] = m;
+			trf->ip4.data[n4] = rte_pktmbuf_mtod_offset(m,
 					uint8_t *, offsetof(struct ip, ip_p));
+			n4++;
 		} else if (ip->ip_v == IP6_VERSION) {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
-			traffic->ip6.data[idx] = rte_pktmbuf_mtod_offset(m,
+			trf->ip6.pkts[n6] = m;
+			trf->ip6.data[n6] = rte_pktmbuf_mtod_offset(m,
 					uint8_t *,
 					offsetof(struct ip6_hdr, ip6_nxt));
+			n6++;
 		} else
 			rte_pktmbuf_free(m);
 	}
 
+	trf->ip4.num = n4;
+	trf->ip6.num = n6;
+}
+
+
+static inline void
+process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
+		struct ipsec_traffic *traffic)
+{
+	uint16_t nb_pkts_in, n_ip4, n_ip6;
+
+	n_ip4 = traffic->ip4.num;
+	n_ip6 = traffic->ip6.num;
+
+	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.num, MAX_PKT_BURST);
+
+	split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+
 	inbound_sp_sa(ipsec_ctx->sp4_ctx, ipsec_ctx->sa_ctx, &traffic->ip4,
 			n_ip4);
 
@@ -782,7 +799,7 @@ process_pkts(struct lcore_conf *qconf, struct rte_mbuf **pkts,
 }
 
 static inline void
-drain_buffers(struct lcore_conf *qconf)
+drain_tx_buffers(struct lcore_conf *qconf)
 {
 	struct buffer *buf;
 	uint32_t portid;
@@ -796,6 +813,81 @@ drain_buffers(struct lcore_conf *qconf)
 	}
 }
 
+static inline void
+drain_crypto_buffers(struct lcore_conf *qconf)
+{
+	uint32_t i;
+	struct ipsec_ctx *ctx;
+
+	/* drain inbound buffers*/
+	ctx = &qconf->inbound;
+	for (i = 0; i != ctx->nb_qps; i++) {
+		if (ctx->tbl[i].len != 0)
+			enqueue_cop_burst(ctx->tbl  + i);
+	}
+
+	/* drain outbound buffers*/
+	ctx = &qconf->outbound;
+	for (i = 0; i != ctx->nb_qps; i++) {
+		if (ctx->tbl[i].len != 0)
+			enqueue_cop_burst(ctx->tbl  + i);
+	}
+}
+
+static void
+drain_inbound_crypto_queues(const struct lcore_conf *qconf,
+		struct ipsec_ctx *ctx)
+{
+	uint32_t n;
+	struct ipsec_traffic trf;
+
+	/* dequeue packets from crypto-queue */
+	n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+			RTE_DIM(trf.ipsec.pkts));
+	if (n == 0)
+		return;
+
+	trf.ip4.num = 0;
+	trf.ip6.num = 0;
+
+	/* split traffic by ipv4-ipv6 */
+	split46_traffic(&trf, trf.ipsec.pkts, n);
+
+	/* process ipv4 packets */
+	inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
+	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+
+	/* process ipv6 packets */
+	inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
+	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+}
+
+static void
+drain_outbound_crypto_queues(const struct lcore_conf *qconf,
+		struct ipsec_ctx *ctx)
+{
+	uint32_t n;
+	struct ipsec_traffic trf;
+
+	/* dequeue packets from crypto-queue */
+	n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+			RTE_DIM(trf.ipsec.pkts));
+	if (n == 0)
+		return;
+
+	trf.ip4.num = 0;
+	trf.ip6.num = 0;
+
+	/* split traffic by ipv4-ipv6 */
+	split46_traffic(&trf, trf.ipsec.pkts, n);
+
+	/* process ipv4 packets */
+	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+
+	/* process ipv6 packets */
+	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+}
+
 /* main processing loop */
 static int32_t
 main_loop(__attribute__((unused)) void *dummy)
@@ -853,7 +945,8 @@ main_loop(__attribute__((unused)) void *dummy)
 		diff_tsc = cur_tsc - prev_tsc;
 
 		if (unlikely(diff_tsc > drain_tsc)) {
-			drain_buffers(qconf);
+			drain_tx_buffers(qconf);
+			drain_crypto_buffers(qconf);
 			prev_tsc = cur_tsc;
 		}
 
@@ -867,6 +960,9 @@ main_loop(__attribute__((unused)) void *dummy)
 			if (nb_rx > 0)
 				process_pkts(qconf, pkts, nb_rx, portid);
 		}
+
+		drain_inbound_crypto_queues(qconf, &qconf->inbound);
+		drain_outbound_crypto_queues(qconf, &qconf->outbound);
 	}
 }
 
diff --git a/examples/ipsec-secgw/ipsec.c b/examples/ipsec-secgw/ipsec.c
index 3d415f1af..8bf928a23 100644
--- a/examples/ipsec-secgw/ipsec.c
+++ b/examples/ipsec-secgw/ipsec.c
@@ -333,33 +333,35 @@ create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa)
 	return 0;
 }
 
+/*
+ * queue crypto-ops into PMD queue.
+ */
+void
+enqueue_cop_burst(struct cdev_qp *cqp)
+{
+	uint32_t i, len, ret;
+
+	len = cqp->len;
+	ret = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp, cqp->buf, len);
+	if (ret < len) {
+		RTE_LOG_DP(DEBUG, IPSEC, "Cryptodev %u queue %u:"
+			" enqueued %u crypto ops out of %u\n",
+			cqp->id, cqp->qp, ret, len);
+			/* drop packets that we fail to enqueue */
+			for (i = ret; i < len; i++)
+				rte_pktmbuf_free(cqp->buf[i]->sym->m_src);
+	}
+	cqp->in_flight += ret;
+	cqp->len = 0;
+}
+
 static inline void
 enqueue_cop(struct cdev_qp *cqp, struct rte_crypto_op *cop)
 {
-	int32_t ret = 0, i;
-
 	cqp->buf[cqp->len++] = cop;
 
-	if (cqp->len == MAX_PKT_BURST) {
-		int enq_size = cqp->len;
-		if ((cqp->in_flight + enq_size) > MAX_INFLIGHT)
-			enq_size -=
-			    (int)((cqp->in_flight + enq_size) - MAX_INFLIGHT);
-
-		if (enq_size > 0)
-			ret = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp,
-					cqp->buf, enq_size);
-		if (ret < cqp->len) {
-			RTE_LOG_DP(DEBUG, IPSEC, "Cryptodev %u queue %u:"
-					" enqueued %u crypto ops out of %u\n",
-					 cqp->id, cqp->qp,
-					 ret, cqp->len);
-			for (i = ret; i < cqp->len; i++)
-				rte_pktmbuf_free(cqp->buf[i]->sym->m_src);
-		}
-		cqp->in_flight += ret;
-		cqp->len = 0;
-	}
+	if (cqp->len == MAX_PKT_BURST)
+		enqueue_cop_burst(cqp);
 }
 
 static inline void
@@ -548,6 +550,13 @@ ipsec_inbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 	return ipsec_dequeue(esp_inbound_post, ctx, pkts, len);
 }
 
+uint16_t
+ipsec_inbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len)
+{
+	return ipsec_dequeue(esp_inbound_post, ctx, pkts, len);
+}
+
 uint16_t
 ipsec_outbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint32_t sa_idx[], uint16_t nb_pkts, uint16_t len)
@@ -560,3 +569,10 @@ ipsec_outbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 
 	return ipsec_dequeue(esp_outbound_post, ctx, pkts, len);
 }
+
+uint16_t
+ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len)
+{
+	return ipsec_dequeue(esp_outbound_post, ctx, pkts, len);
+}
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 580f7876b..2f04b7d68 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -184,6 +184,14 @@ uint16_t
 ipsec_outbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint32_t sa_idx[], uint16_t nb_pkts, uint16_t len);
 
+uint16_t
+ipsec_inbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len);
+
+uint16_t
+ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len);
+
 static inline uint16_t
 ipsec_metadata_size(void)
 {
@@ -248,4 +256,7 @@ sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
 int
 add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
 
+void
+enqueue_cop_burst(struct cdev_qp *cqp);
+
 #endif /* __IPSEC_H__ */
-- 
2.17.1

[PATCH v2 4/7] examples/ipsec-secgw: fix outbound codepath for single SA

[Konstantin Ananyev]

Looking at process_pkts_outbound_nosp() there seems few issues:
- accessing mbuf after it was freed
- invoking ipsec_outbound() for ipv4 packets only
- copying number of packets, but not the mbuf pointers itself

that patch provides fixes for that issues.

Fixes: 906257e965b7 ("examples/ipsec-secgw: support IPv6")

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 32 ++++++++++++++++++++----------
 1 file changed, 22 insertions(+), 10 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 62443172a..d1da2d5ce 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -616,32 +616,44 @@ process_pkts_outbound_nosp(struct ipsec_ctx *ipsec_ctx,
 		struct ipsec_traffic *traffic)
 {
 	struct rte_mbuf *m;
-	uint32_t nb_pkts_out, i;
+	uint32_t nb_pkts_out, i, n;
 	struct ip *ip;
 
 	/* Drop any IPsec traffic from protected ports */
 	for (i = 0; i < traffic->ipsec.num; i++)
 		rte_pktmbuf_free(traffic->ipsec.pkts[i]);
 
-	traffic->ipsec.num = 0;
+	n = 0;
 
-	for (i = 0; i < traffic->ip4.num; i++)
-		traffic->ip4.res[i] = single_sa_idx;
+	for (i = 0; i < traffic->ip4.num; i++) {
+		traffic->ipsec.pkts[n] = traffic->ip4.pkts[i];
+		traffic->ipsec.res[n++] = single_sa_idx;
+	}
 
-	for (i = 0; i < traffic->ip6.num; i++)
-		traffic->ip6.res[i] = single_sa_idx;
+	for (i = 0; i < traffic->ip6.num; i++) {
+		traffic->ipsec.pkts[n] = traffic->ip6.pkts[i];
+		traffic->ipsec.res[n++] = single_sa_idx;
+	}
+
+	traffic->ip4.num = 0;
+	traffic->ip6.num = 0;
+	traffic->ipsec.num = n;
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ip4.pkts,
-			traffic->ip4.res, traffic->ip4.num,
+	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.res, traffic->ipsec.num,
 			MAX_PKT_BURST);
 
 	/* They all sue the same SA (ip4 or ip6 tunnel) */
 	m = traffic->ipsec.pkts[i];
 	ip = rte_pktmbuf_mtod(m, struct ip *);
-	if (ip->ip_v == IPVERSION)
+	if (ip->ip_v == IPVERSION) {
 		traffic->ip4.num = nb_pkts_out;
-	else
+		for (i = 0; i < nb_pkts_out; i++)
+			traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+	} else {
 		traffic->ip6.num = nb_pkts_out;
+		traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+	}
 }
 
 static inline int32_t
-- 
2.17.1

[PATCH v2 5/7] examples/ipsec-secgw: make app to use ipsec library

[Konstantin Ananyev]

Changes to make ipsec-secgw to utilize librte_ipsec library.
That patch provides:
 - changes in the related data structures.
 - changes in the initialization code.
 - new command-line parameters to enable librte_ipsec codepath
   and related features.

Note that right now by default current (non-librte_ipsec) code-path will
be used. User has to run application with new command-line option ('-l')
to enable new codepath.
The main reason for that:
 - current librte_ipsec doesn't support all ipsec algorithms
   and features that the app does.
 - allow users to run both versions in parallel for some time
   to figure out any functional or performance degradation with the
   new code.

It is planned to deprecate and remove non-librte_ipsec code path
in future releases.

Signed-off-by: Mohammad Abdul Awal <mohammad.abdul.awal@intel.com>
Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c |  44 ++++++-
 examples/ipsec-secgw/ipsec.h       |  14 +++
 examples/ipsec-secgw/meson.build   |   2 +-
 examples/ipsec-secgw/sa.c          | 185 ++++++++++++++++++++++++++++-
 4 files changed, 241 insertions(+), 4 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index d1da2d5ce..feef865c9 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -155,6 +155,9 @@ static uint32_t single_sa;
 static uint32_t single_sa_idx;
 static uint32_t frame_size;
 
+/* application wide librte_ipsec/SA parameters */
+struct app_sa_prm app_sa_prm = {.enable = 0};
+
 struct lcore_rx_queue {
 	uint16_t port_id;
 	uint8_t queue_id;
@@ -1063,6 +1066,10 @@ print_usage(const char *prgname)
 		" [-P]"
 		" [-u PORTMASK]"
 		" [-j FRAMESIZE]"
+		" [-l]"
+		" [-w REPLAY_WINDOW_SIZE]"
+		" [-e]"
+		" [-a]"
 		" -f CONFIG_FILE"
 		" --config (port,queue,lcore)[,(port,queue,lcore)]"
 		" [--single-sa SAIDX]"
@@ -1073,6 +1080,10 @@ print_usage(const char *prgname)
 		"  -u PORTMASK: Hexadecimal bitmask of unprotected ports\n"
 		"  -j FRAMESIZE: Enable jumbo frame with 'FRAMESIZE' as maximum\n"
 		"                packet size\n"
+		"  -l enables code-path that uses librte_ipsec\n"
+		"  -w REPLAY_WINDOW_SIZE specifies IPsec SQN replay window\n"
+		"     size for each SA\n"
+		"  -a enables SA SQN atomic behaviour\n"
 		"  -f CONFIG_FILE: Configuration file\n"
 		"  --config (port,queue,lcore): Rx queue configuration\n"
 		"  --single-sa SAIDX: Use single SA index for outbound traffic,\n"
@@ -1169,6 +1180,20 @@ parse_config(const char *q_arg)
 	return 0;
 }
 
+static void
+print_app_sa_prm(const struct app_sa_prm *prm)
+{
+	printf("librte_ipsec usage: %s\n",
+		(prm->enable == 0) ? "disabled" : "enabled");
+
+	if (prm->enable == 0)
+		return;
+
+	printf("replay window size: %u\n", prm->window_size);
+	printf("ESN: %s\n", (prm->enable_esn == 0) ? "disabled" : "enabled");
+	printf("SA flags: %#" PRIx64 "\n", prm->flags);
+}
+
 static int32_t
 parse_args(int32_t argc, char **argv)
 {
@@ -1180,7 +1205,7 @@ parse_args(int32_t argc, char **argv)
 
 	argvopt = argv;
 
-	while ((opt = getopt_long(argc, argvopt, "p:Pu:f:j:",
+	while ((opt = getopt_long(argc, argvopt, "aelp:Pu:f:j:w:",
 				lgopts, &option_index)) != EOF) {
 
 		switch (opt) {
@@ -1236,6 +1261,21 @@ parse_args(int32_t argc, char **argv)
 			}
 			printf("Enabled jumbo frames size %u\n", frame_size);
 			break;
+		case 'l':
+			app_sa_prm.enable = 1;
+			break;
+		case 'w':
+			app_sa_prm.enable = 1;
+			app_sa_prm.window_size = parse_decimal(optarg);
+			break;
+		case 'e':
+			app_sa_prm.enable = 1;
+			app_sa_prm.enable_esn = 1;
+			break;
+		case 'a':
+			app_sa_prm.enable = 1;
+			app_sa_prm.flags |= RTE_IPSEC_SAFLAG_SQN_ATOM;
+			break;
 		case CMD_LINE_OPT_CONFIG_NUM:
 			ret = parse_config(optarg);
 			if (ret) {
@@ -1280,6 +1320,8 @@ parse_args(int32_t argc, char **argv)
 		return -1;
 	}
 
+	print_app_sa_prm(&app_sa_prm);
+
 	if (optind >= 0)
 		argv[optind-1] = prgname;
 
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 2f04b7d68..45485b019 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -11,6 +11,7 @@
 #include <rte_crypto.h>
 #include <rte_security.h>
 #include <rte_flow.h>
+#include <rte_ipsec.h>
 
 #define RTE_LOGTYPE_IPSEC       RTE_LOGTYPE_USER1
 #define RTE_LOGTYPE_IPSEC_ESP   RTE_LOGTYPE_USER2
@@ -70,7 +71,20 @@ struct ip_addr {
 
 #define MAX_KEY_SIZE		32
 
+/*
+ * application wide SA parameters
+ */
+struct app_sa_prm {
+	uint32_t enable; /* use librte_ipsec API for ipsec pkt processing */
+	uint32_t window_size; /* replay window size */
+	uint32_t enable_esn;  /* enable/disable ESN support */
+	uint64_t flags;       /* rte_ipsec_sa_prm.flags */
+};
+
+extern struct app_sa_prm app_sa_prm;
+
 struct ipsec_sa {
+	struct rte_ipsec_session ips; /* one session per sa for now */
 	uint32_t spi;
 	uint32_t cdev_id_qp;
 	uint64_t seq;
diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
index 77d8b298f..31f68fee2 100644
--- a/examples/ipsec-secgw/meson.build
+++ b/examples/ipsec-secgw/meson.build
@@ -6,7 +6,7 @@
 # To build this example as a standalone application with an already-installed
 # DPDK instance, use 'make'
 
-deps += ['security', 'lpm', 'acl', 'hash']
+deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
 allow_experimental_apis = true
 sources = files(
 	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index ff8c4b829..282b40bea 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -19,6 +19,7 @@
 #include <rte_ip.h>
 #include <rte_random.h>
 #include <rte_ethdev.h>
+#include <rte_malloc.h>
 
 #include "ipsec.h"
 #include "esp.h"
@@ -694,6 +695,7 @@ print_one_sa_rule(const struct ipsec_sa *sa, int inbound)
 }
 
 struct sa_ctx {
+	void *satbl; /* pointer to array of rte_ipsec_sa objects*/
 	struct ipsec_sa sa[IPSEC_SA_MAX_ENTRIES];
 	union {
 		struct {
@@ -764,7 +766,10 @@ sa_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 {
 	struct ipsec_sa *sa;
 	uint32_t i, idx;
-	uint16_t iv_length;
+	uint16_t iv_length, aad_length;
+
+	/* for ESN upper 32 bits of SQN also need to be part of AAD */
+	aad_length = (app_sa_prm.enable_esn != 0) ? sizeof(uint32_t) : 0;
 
 	for (i = 0; i < nb_entries; i++) {
 		idx = SPI2IDX(entries[i].spi);
@@ -808,7 +813,7 @@ sa_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 			sa_ctx->xf[idx].a.aead.iv.offset = IV_OFFSET;
 			sa_ctx->xf[idx].a.aead.iv.length = iv_length;
 			sa_ctx->xf[idx].a.aead.aad_length =
-				sa->aad_len;
+				sa->aad_len + aad_length;
 			sa_ctx->xf[idx].a.aead.digest_length =
 				sa->digest_len;
 
@@ -901,9 +906,169 @@ sa_in_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 	return sa_add_rules(sa_ctx, entries, nb_entries, 1);
 }
 
+/*
+ * helper function, fills parameters that are identical for all SAs
+ */
+static void
+fill_ipsec_app_sa_prm(struct rte_ipsec_sa_prm *prm,
+	const struct app_sa_prm *app_prm)
+{
+	memset(prm, 0, sizeof(*prm));
+
+	prm->flags = app_prm->flags;
+	prm->ipsec_xform.options.esn = app_prm->enable_esn;
+	prm->replay_win_sz = app_prm->window_size;
+}
+
+static void
+fill_ipsec_sa_prm(struct rte_ipsec_sa_prm *prm, const struct ipsec_sa *ss,
+	const struct ipv4_hdr *v4, struct ipv6_hdr *v6)
+{
+	fill_ipsec_app_sa_prm(prm, &app_sa_prm);
+
+	prm->userdata = (uintptr_t)ss;
+
+	/* setup ipsec xform */
+	prm->ipsec_xform.spi = ss->spi;
+	prm->ipsec_xform.salt = ss->salt;
+	prm->ipsec_xform.direction = ss->direction;
+	prm->ipsec_xform.proto = RTE_SECURITY_IPSEC_SA_PROTO_ESP;
+	prm->ipsec_xform.mode = (ss->flags == TRANSPORT) ?
+		RTE_SECURITY_IPSEC_SA_MODE_TRANSPORT :
+		RTE_SECURITY_IPSEC_SA_MODE_TUNNEL;
+
+	/*
+	 * !!! this has to be changed.
+	 * Below we fill tun.next_proto based on the assumption that
+	 * input packet IP version is equal to tunnel IP version.
+	 * That could not be the case.
+	 * SA should know it's input IP version.
+	 */
+	if (ss->flags == IP4_TUNNEL) {
+		prm->ipsec_xform.tunnel.type = RTE_SECURITY_IPSEC_TUNNEL_IPV4;
+		prm->tun.hdr_len = sizeof(*v4);
+		prm->tun.next_proto = IPPROTO_IPIP;
+		prm->tun.hdr = v4;
+	} else if (ss->flags == IP6_TUNNEL) {
+		prm->ipsec_xform.tunnel.type = RTE_SECURITY_IPSEC_TUNNEL_IPV6;
+		prm->tun.hdr_len = sizeof(*v6);
+		prm->tun.next_proto = IPPROTO_IPV6;
+		prm->tun.hdr = v6;
+	} else {
+		prm->trs.proto = IPPROTO_IPIP;
+	}
+	/* setup crypto section */
+	prm->crypto_xform = ss->xforms;
+}
+
+static void
+fill_ipsec_session(struct rte_ipsec_session *ss, struct rte_ipsec_sa *sa,
+	const struct ipsec_sa *lsa)
+{
+	ss->sa = sa;
+	ss->type = lsa->type;
+
+	/* setup crypto section */
+	if (ss->type == RTE_SECURITY_ACTION_TYPE_NONE) {
+		ss->crypto.ses = lsa->crypto_session;
+	/* setup session action type */
+	} else {
+		ss->security.ses = lsa->sec_session;
+		ss->security.ctx = lsa->security_ctx;
+		ss->security.ol_flags = lsa->ol_flags;
+	}
+}
+
+/*
+ * Initialise related rte_ipsec_sa object.
+ */
+static int
+ipsec_sa_init(struct ipsec_sa *lsa, struct rte_ipsec_sa *sa, uint32_t sa_size)
+{
+	int rc;
+	struct rte_ipsec_sa_prm prm;
+	struct ipv4_hdr v4  = {
+		.version_ihl = IPVERSION << 4 |
+			sizeof(v4) / IPV4_IHL_MULTIPLIER,
+		.time_to_live = IPDEFTTL,
+		.next_proto_id = IPPROTO_ESP,
+		.src_addr = lsa->src.ip.ip4,
+		.dst_addr = lsa->dst.ip.ip4,
+	};
+	struct ipv6_hdr v6 = {
+		.vtc_flow = htonl(IP6_VERSION << 28),
+		.proto = IPPROTO_ESP,
+	};
+
+	if (lsa->flags == IP6_TUNNEL) {
+		memcpy(v6.src_addr, lsa->src.ip.ip6.ip6_b, sizeof(v6.src_addr));
+		memcpy(v6.dst_addr, lsa->dst.ip.ip6.ip6_b, sizeof(v6.dst_addr));
+	}
+
+	fill_ipsec_sa_prm(&prm, lsa, &v4, &v6);
+	rc = rte_ipsec_sa_init(sa, &prm, sa_size);
+	if (rc < 0)
+		return rc;
+
+	fill_ipsec_session(&lsa->ips, sa, lsa);
+	return 0;
+}
+
+/*
+ * Allocate space and init rte_ipsec_sa strcutures,
+ * one per session.
+ */
+static int
+ipsec_satbl_init(struct sa_ctx *ctx, const struct ipsec_sa *ent,
+	uint32_t nb_ent, int32_t socket)
+{
+	int32_t rc, sz;
+	uint32_t i, idx;
+	size_t tsz;
+	struct rte_ipsec_sa *sa;
+	struct ipsec_sa *lsa;
+	struct rte_ipsec_sa_prm prm;
+
+	/* determine SA size */
+	idx = SPI2IDX(ent[0].spi);
+	fill_ipsec_app_sa_prm(&prm, &app_sa_prm);
+	prm.ipsec_xform.direction = ctx->sa[idx].direction;
+	sz = rte_ipsec_sa_size(&prm);
+	if (sz < 0) {
+		RTE_LOG(ERR, IPSEC, "%s(%p, %u, %d): "
+			"failed to determine SA size, error code: %d\n",
+			__func__, ctx, nb_ent, socket, sz);
+		return sz;
+	}
+
+	tsz = sz * nb_ent;
+
+	ctx->satbl = rte_zmalloc_socket(NULL, tsz, RTE_CACHE_LINE_SIZE, socket);
+	if (ctx->satbl == NULL) {
+		RTE_LOG(ERR, IPSEC,
+			"%s(%p, %u, %d): failed to allocate %zu bytes\n",
+			__func__,  ctx, nb_ent, socket, tsz);
+		return -ENOMEM;
+	}
+
+	rc = 0;
+	for (i = 0; i != nb_ent && rc == 0; i++) {
+
+		idx = SPI2IDX(ent[i].spi);
+
+		sa = (struct rte_ipsec_sa *)((uintptr_t)ctx->satbl + sz * i);
+		lsa = ctx->sa + idx;
+
+		rc = ipsec_sa_init(lsa, sa, sz);
+	}
+
+	return rc;
+}
+
 void
 sa_init(struct socket_ctx *ctx, int32_t socket_id)
 {
+	int32_t rc;
 	const char *name;
 
 	if (ctx == NULL)
@@ -926,6 +1091,14 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id)
 				name, socket_id);
 
 		sa_in_add_rules(ctx->sa_in, sa_in, nb_sa_in);
+
+		if (app_sa_prm.enable != 0) {
+			rc = ipsec_satbl_init(ctx->sa_in, sa_in, nb_sa_in,
+				socket_id);
+			if (rc != 0)
+				rte_exit(EXIT_FAILURE,
+					"failed to init inbound SAs\n");
+		}
 	} else
 		RTE_LOG(WARNING, IPSEC, "No SA Inbound rule specified\n");
 
@@ -938,6 +1111,14 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id)
 				name, socket_id);
 
 		sa_out_add_rules(ctx->sa_out, sa_out, nb_sa_out);
+
+		if (app_sa_prm.enable != 0) {
+			rc = ipsec_satbl_init(ctx->sa_out, sa_out, nb_sa_out,
+				socket_id);
+			if (rc != 0)
+				rte_exit(EXIT_FAILURE,
+					"failed to init outbound SAs\n");
+		}
 	} else
 		RTE_LOG(WARNING, IPSEC, "No SA Outbound rule "
 			"specified\n");
-- 
2.17.1

[PATCH v2 6/7] examples/ipsec-secgw: make data-path to use ipsec library

[Konstantin Ananyev]

Changes to make ipsec-secgw data-path code to utilize librte_ipsec library.
Note that right now by default current (non-librte_ipsec) code-path will
be used. User has to run application with new command-line option ('-l')
to enable new codepath.

Signed-off-by: Mohammad Abdul Awal <mohammad.abdul.awal@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/Makefile        |   1 +
 examples/ipsec-secgw/ipsec-secgw.c   | 178 ++++++++++--------
 examples/ipsec-secgw/ipsec.c         |   2 +-
 examples/ipsec-secgw/ipsec.h         |  23 +++
 examples/ipsec-secgw/ipsec_process.c | 266 +++++++++++++++++++++++++++
 examples/ipsec-secgw/meson.build     |   4 +-
 6 files changed, 396 insertions(+), 78 deletions(-)
 create mode 100644 examples/ipsec-secgw/ipsec_process.c

diff --git a/examples/ipsec-secgw/Makefile b/examples/ipsec-secgw/Makefile
index 02d41e39a..4d6b4cfa4 100644
--- a/examples/ipsec-secgw/Makefile
+++ b/examples/ipsec-secgw/Makefile
@@ -13,6 +13,7 @@ SRCS-y += sp4.c
 SRCS-y += sp6.c
 SRCS-y += sa.c
 SRCS-y += rt.c
+SRCS-y += ipsec_process.c
 SRCS-y += ipsec-secgw.c
 
 CFLAGS += -gdwarf-2
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index feef865c9..074803973 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -216,19 +216,6 @@ static struct rte_eth_conf port_conf = {
 
 static struct socket_ctx socket_ctx[NB_SOCKETS];
 
-struct traffic_type {
-	const uint8_t *data[MAX_PKT_BURST * 2];
-	struct rte_mbuf *pkts[MAX_PKT_BURST * 2];
-	uint32_t res[MAX_PKT_BURST * 2];
-	uint32_t num;
-};
-
-struct ipsec_traffic {
-	struct traffic_type ipsec;
-	struct traffic_type ip4;
-	struct traffic_type ip6;
-};
-
 static inline void
 prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 {
@@ -245,6 +232,8 @@ prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 			t->ip4.data[t->ip4.num] = nlp;
 			t->ip4.pkts[(t->ip4.num)++] = pkt;
 		}
+		pkt->l2_len = 0;
+		pkt->l3_len = sizeof(struct ip);
 	} else if (eth->ether_type == rte_cpu_to_be_16(ETHER_TYPE_IPv6)) {
 		nlp = (uint8_t *)rte_pktmbuf_adj(pkt, ETHER_HDR_LEN);
 		nlp = RTE_PTR_ADD(nlp, offsetof(struct ip6_hdr, ip6_nxt));
@@ -254,6 +243,8 @@ prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 			t->ip6.data[t->ip6.num] = nlp;
 			t->ip6.pkts[(t->ip6.num)++] = pkt;
 		}
+		pkt->l2_len = 0;
+		pkt->l3_len = sizeof(struct ip6_hdr);
 	} else {
 		/* Unknown/Unsupported type, drop the packet */
 		RTE_LOG(ERR, IPSEC, "Unsupported packet type\n");
@@ -503,10 +494,15 @@ process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
 	n_ip4 = traffic->ip4.num;
 	n_ip6 = traffic->ip6.num;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
-
-	split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+	if (app_sa_prm.enable == 0) {
+		nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.num, MAX_PKT_BURST);
+		split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+	} else {
+		inbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
+	}
 
 	inbound_sp_sa(ipsec_ctx->sp4_ctx, ipsec_ctx->sa_ctx, &traffic->ip4,
 			n_ip4);
@@ -562,20 +558,27 @@ process_pkts_outbound(struct ipsec_ctx *ipsec_ctx,
 
 	outbound_sp(ipsec_ctx->sp6_ctx, &traffic->ip6, &traffic->ipsec);
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.res, traffic->ipsec.num,
-			MAX_PKT_BURST);
-
-	for (i = 0; i < nb_pkts_out; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
-		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-		} else {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
+	if (app_sa_prm.enable == 0) {
+
+		nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.res, traffic->ipsec.num,
+				MAX_PKT_BURST);
+
+		for (i = 0; i < nb_pkts_out; i++) {
+			m = traffic->ipsec.pkts[i];
+			struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
+			if (ip->ip_v == IPVERSION) {
+				idx = traffic->ip4.num++;
+				traffic->ip4.pkts[idx] = m;
+			} else {
+				idx = traffic->ip6.num++;
+				traffic->ip6.pkts[idx] = m;
+			}
 		}
+	} else {
+		outbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.res,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -598,19 +601,26 @@ process_pkts_inbound_nosp(struct ipsec_ctx *ipsec_ctx,
 
 	traffic->ip6.num = 0;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
+	if (app_sa_prm.enable == 0) {
 
-	for (i = 0; i < nb_pkts_in; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
-		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-		} else {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
+		nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.num, MAX_PKT_BURST);
+
+		for (i = 0; i < nb_pkts_in; i++) {
+			m = traffic->ipsec.pkts[i];
+			struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
+			if (ip->ip_v == IPVERSION) {
+				idx = traffic->ip4.num++;
+				traffic->ip4.pkts[idx] = m;
+			} else {
+				idx = traffic->ip6.num++;
+				traffic->ip6.pkts[idx] = m;
+			}
 		}
+	} else {
+		inbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -642,20 +652,28 @@ process_pkts_outbound_nosp(struct ipsec_ctx *ipsec_ctx,
 	traffic->ip6.num = 0;
 	traffic->ipsec.num = n;
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.res, traffic->ipsec.num,
-			MAX_PKT_BURST);
+	if (app_sa_prm.enable == 0) {
 
-	/* They all sue the same SA (ip4 or ip6 tunnel) */
-	m = traffic->ipsec.pkts[i];
-	ip = rte_pktmbuf_mtod(m, struct ip *);
-	if (ip->ip_v == IPVERSION) {
-		traffic->ip4.num = nb_pkts_out;
-		for (i = 0; i < nb_pkts_out; i++)
-			traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+		nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.res, traffic->ipsec.num,
+				MAX_PKT_BURST);
+
+		/* They all sue the same SA (ip4 or ip6 tunnel) */
+		m = traffic->ipsec.pkts[0];
+		ip = rte_pktmbuf_mtod(m, struct ip *);
+		if (ip->ip_v == IPVERSION) {
+			traffic->ip4.num = nb_pkts_out;
+			for (i = 0; i < nb_pkts_out; i++)
+				traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+		} else {
+			traffic->ip6.num = nb_pkts_out;
+			for (i = 0; i < nb_pkts_out; i++)
+				traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+		}
 	} else {
-		traffic->ip6.num = nb_pkts_out;
-		traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+		outbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.res,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -856,25 +874,31 @@ drain_inbound_crypto_queues(const struct lcore_conf *qconf,
 	uint32_t n;
 	struct ipsec_traffic trf;
 
-	/* dequeue packets from crypto-queue */
-	n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+	if (app_sa_prm.enable == 0) {
+
+		/* dequeue packets from crypto-queue */
+		n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
 			RTE_DIM(trf.ipsec.pkts));
-	if (n == 0)
-		return;
 
-	trf.ip4.num = 0;
-	trf.ip6.num = 0;
+		trf.ip4.num = 0;
+		trf.ip6.num = 0;
 
-	/* split traffic by ipv4-ipv6 */
-	split46_traffic(&trf, trf.ipsec.pkts, n);
+		/* split traffic by ipv4-ipv6 */
+		split46_traffic(&trf, trf.ipsec.pkts, n);
+	} else
+		ipsec_cqp_process(ctx, &trf);
 
 	/* process ipv4 packets */
-	inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
-	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	if (trf.ip4.num != 0) {
+		inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
+		route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	}
 
 	/* process ipv6 packets */
-	inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
-	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	if (trf.ip6.num != 0) {
+		inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
+		route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	}
 }
 
 static void
@@ -884,23 +908,27 @@ drain_outbound_crypto_queues(const struct lcore_conf *qconf,
 	uint32_t n;
 	struct ipsec_traffic trf;
 
-	/* dequeue packets from crypto-queue */
-	n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+	if (app_sa_prm.enable == 0) {
+
+		/* dequeue packets from crypto-queue */
+		n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
 			RTE_DIM(trf.ipsec.pkts));
-	if (n == 0)
-		return;
 
-	trf.ip4.num = 0;
-	trf.ip6.num = 0;
+		trf.ip4.num = 0;
+		trf.ip6.num = 0;
 
-	/* split traffic by ipv4-ipv6 */
-	split46_traffic(&trf, trf.ipsec.pkts, n);
+		/* split traffic by ipv4-ipv6 */
+		split46_traffic(&trf, trf.ipsec.pkts, n);
+	} else
+		ipsec_cqp_process(ctx, &trf);
 
 	/* process ipv4 packets */
-	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	if (trf.ip4.num != 0)
+		route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
 
 	/* process ipv6 packets */
-	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	if (trf.ip6.num != 0)
+		route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
 }
 
 /* main processing loop */
diff --git a/examples/ipsec-secgw/ipsec.c b/examples/ipsec-secgw/ipsec.c
index 8bf928a23..8e7fc3639 100644
--- a/examples/ipsec-secgw/ipsec.c
+++ b/examples/ipsec-secgw/ipsec.c
@@ -39,7 +39,7 @@ set_ipsec_conf(struct ipsec_sa *sa, struct rte_security_ipsec_xform *ipsec)
 	ipsec->esn_soft_limit = IPSEC_OFFLOAD_ESN_SOFTLIMIT;
 }
 
-static inline int
+int
 create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa)
 {
 	struct rte_cryptodev_info cdev_info;
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 45485b019..c6e460e6f 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -190,6 +190,20 @@ struct cnt_blk {
 	uint32_t cnt;
 } __attribute__((packed));
 
+struct traffic_type {
+	const uint8_t *data[MAX_PKT_BURST * 2];
+	struct rte_mbuf *pkts[MAX_PKT_BURST * 2];
+	struct ipsec_sa *saptr[MAX_PKT_BURST * 2];
+	uint32_t res[MAX_PKT_BURST * 2];
+	uint32_t num;
+};
+
+struct ipsec_traffic {
+	struct traffic_type ipsec;
+	struct traffic_type ip4;
+	struct traffic_type ip6;
+};
+
 uint16_t
 ipsec_inbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint16_t nb_pkts, uint16_t len);
@@ -206,6 +220,12 @@ uint16_t
 ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint16_t len);
 
+void
+ipsec_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf);
+
+void
+ipsec_cqp_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf);
+
 static inline uint16_t
 ipsec_metadata_size(void)
 {
@@ -273,4 +293,7 @@ add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
 void
 enqueue_cop_burst(struct cdev_qp *cqp);
 
+int
+create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/ipsec_process.c b/examples/ipsec-secgw/ipsec_process.c
new file mode 100644
index 000000000..c90e764b3
--- /dev/null
+++ b/examples/ipsec-secgw/ipsec_process.c
@@ -0,0 +1,266 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2017 Intel Corporation
+ */
+#include <sys/types.h>
+#include <netinet/in.h>
+#include <netinet/ip.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_log.h>
+#include <rte_cryptodev.h>
+#include <rte_ethdev.h>
+#include <rte_mbuf.h>
+
+#include "ipsec.h"
+
+#define SATP_OUT_IPV4(t)	\
+	((((t) & RTE_IPSEC_SATP_MODE_MASK) == RTE_IPSEC_SATP_MODE_TRANS && \
+	(((t) & RTE_IPSEC_SATP_IPV_MASK) == RTE_IPSEC_SATP_IPV4)) || \
+	((t) & RTE_IPSEC_SATP_MODE_MASK) == RTE_IPSEC_SATP_MODE_TUNLV4)
+
+
+/* helper routine to free bulk of packets */
+static inline void
+free_pkts(struct rte_mbuf *mb[], uint32_t n)
+{
+	uint32_t i;
+
+	for (i = 0; i != n; i++)
+		rte_pktmbuf_free(mb[i]);
+}
+
+static inline int
+fill_ipsec_session(struct rte_ipsec_session *ss, const struct ipsec_sa *sa)
+{
+	/* setup crypto section */
+	if (ss->type == RTE_SECURITY_ACTION_TYPE_NONE) {
+		ss->crypto.ses = sa->crypto_session;
+	/* setup session action type */
+	} else {
+		ss->security.ses = sa->sec_session;
+		ss->security.ctx = sa->security_ctx;
+		ss->security.ol_flags = sa->ol_flags;
+	}
+
+	return rte_ipsec_session_prepare(ss);
+}
+
+/*
+ * group input packets byt the SA they belong to.
+ */
+static uint32_t
+sa_group(struct ipsec_sa *sa_ptr[], struct rte_mbuf *pkts[],
+	struct rte_ipsec_group grp[], uint32_t num)
+{
+	uint32_t i, n, spi;
+	void *sa;
+	void * const nosa = &spi;
+
+	sa = nosa;
+	for (i = 0, n = 0; i != num; i++) {
+
+		if (sa != sa_ptr[i]) {
+			grp[n].cnt = pkts + i - grp[n].m;
+			n += (sa != nosa);
+			grp[n].id.ptr = sa_ptr[i];
+			grp[n].m = pkts + i;
+			sa = sa_ptr[i];
+		}
+	}
+
+	/* terminate last group */
+	if (sa != nosa) {
+		grp[n].cnt = pkts + i - grp[n].m;
+		n++;
+	}
+
+	return n;
+}
+
+/*
+ * helper function, splits processed packets into ipv4/ipv6 traffic.
+ */
+static inline void
+copy_to_trf(struct ipsec_traffic *trf, uint64_t satp, struct rte_mbuf *mb[],
+	uint32_t num)
+{
+	uint32_t j, ofs, s;
+	struct traffic_type *out;
+
+	/*
+	 * determine traffic type(ipv4/ipv6) and offset for ACL classify
+	 * based on SA type
+	 */
+	if ((satp & RTE_IPSEC_SATP_DIR_MASK) == RTE_IPSEC_SATP_DIR_IB) {
+		if ((satp & RTE_IPSEC_SATP_IPV_MASK) == RTE_IPSEC_SATP_IPV4) {
+			out = &trf->ip4;
+			ofs = offsetof(struct ip, ip_p);
+		} else {
+			out = &trf->ip6;
+			ofs = offsetof(struct ip6_hdr, ip6_nxt);
+		}
+	} else if (SATP_OUT_IPV4(satp)) {
+		out = &trf->ip4;
+		ofs = offsetof(struct ip, ip_p);
+	} else {
+		out = &trf->ip6;
+		ofs = offsetof(struct ip6_hdr, ip6_nxt);
+	}
+
+	for (j = 0, s = out->num; j != num; j++) {
+		out->data[s + j] = rte_pktmbuf_mtod_offset(mb[j],
+				void *, ofs);
+		out->pkts[s + j] = mb[j];
+	}
+
+	out->num += num;
+}
+
+/*
+ * Process ipsec packets.
+ * If packet belong to SA that is subject of inline-crypto,
+ * then process it immediately.
+ * Otherwise do necessary preparations and queue it to related
+ * crypto-dev queue.
+ */
+void
+ipsec_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf)
+{
+	uint64_t satp;
+	uint32_t i, j, k, n;
+	struct ipsec_sa *sa;
+	struct ipsec_mbuf_metadata *priv;
+	struct rte_ipsec_group *pg;
+	struct rte_ipsec_session *ips;
+	struct cdev_qp *cqp;
+	struct rte_crypto_op *cop[RTE_DIM(trf->ipsec.pkts)];
+	struct rte_ipsec_group grp[RTE_DIM(trf->ipsec.pkts)];
+
+	n = sa_group(trf->ipsec.saptr, trf->ipsec.pkts, grp, trf->ipsec.num);
+
+	for (i = 0; i != n; i++) {
+
+		pg = grp + i;
+		sa = pg->id.ptr;
+
+		/* no valid SA found */
+		if (sa == NULL)
+			k = 0;
+
+		ips = &sa->ips;
+		satp = rte_ipsec_sa_type(ips->sa);
+
+		/* no valid HW session for that SA, try to create one */
+		if (ips->crypto.ses == NULL &&
+				(create_session(ctx, sa) != 0 ||
+				fill_ipsec_session(ips, sa) != 0))
+			k = 0;
+
+		/* process packets inline */
+		else if (sa->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				sa->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL) {
+
+			k = rte_ipsec_pkt_process(ips, pg->m, pg->cnt);
+			copy_to_trf(trf, satp, pg->m, k);
+
+		/* enqueue packets to crypto dev */
+		} else {
+
+			cqp = &ctx->tbl[sa->cdev_id_qp];
+
+			/* for that app each mbuf has it's own crypto op */
+			for (j = 0; j != pg->cnt; j++) {
+				priv = get_priv(pg->m[j]);
+				cop[j] = &priv->cop;
+				/*
+				 * this is just to satisfy inbound_sa_check()
+				 * should be removed in future.
+				 */
+				priv->sa = sa;
+			}
+
+			/* prepare and enqueue crypto ops */
+			k = rte_ipsec_pkt_crypto_prepare(ips, pg->m, cop,
+				pg->cnt);
+			if (k != 0)
+				k = rte_cryptodev_enqueue_burst(cqp->id,
+					cqp->qp, cop, k);
+		}
+
+		/* drop packets that cannot be enqueued/processed */
+		if (k != pg->cnt)
+			free_pkts(pg->m + k, pg->cnt - k);
+	}
+}
+
+static inline uint32_t
+cqp_dequeue(struct ipsec_ctx *ctx, struct rte_crypto_op *cop[], uint32_t num)
+{
+	uint32_t i, n;
+	struct cdev_qp *cqp;
+
+	n = 0;
+
+	for (i = ctx->last_qp; n != num && i != ctx->nb_qps; i++) {
+		cqp = ctx->tbl + i;
+		n += rte_cryptodev_dequeue_burst(cqp->id, cqp->qp, cop + n,
+				num - n);
+	}
+	for (i = 0; n != num && i != ctx->last_qp; i++) {
+		cqp = ctx->tbl + i;
+		n += rte_cryptodev_dequeue_burst(cqp->id, cqp->qp, cop + n,
+				num - n);
+	}
+
+	ctx->last_qp = i;
+	return n;
+}
+
+/*
+ * dequeue packets from crypto-queues and finalize processing.
+ */
+void
+ipsec_cqp_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf)
+{
+	uint64_t satp;
+	uint32_t i, k, n, ng;
+	struct rte_ipsec_session *ss;
+	struct traffic_type *out;
+	struct rte_ipsec_group *pg;
+	struct rte_crypto_op *cop[RTE_DIM(trf->ipsec.pkts)];
+	struct rte_ipsec_group grp[RTE_DIM(trf->ipsec.pkts)];
+
+	trf->ip4.num = 0;
+	trf->ip6.num = 0;
+
+	out = &trf->ipsec;
+
+	/* dequeue completed crypto-ops */
+	n = cqp_dequeue(ctx, cop, RTE_DIM(cop));
+	if (n == 0)
+		return;
+
+	/* group them by ipsec session */
+	ng = rte_ipsec_pkt_crypto_group((const struct rte_crypto_op **)
+		(uintptr_t)cop, out->pkts, grp, n);
+
+	/* process each group of packets */
+	for (i = 0; i != ng; i++) {
+
+		pg = grp + i;
+		ss = pg->id.ptr;
+		satp = rte_ipsec_sa_type(ss->sa);
+
+		k = rte_ipsec_pkt_process(ss, pg->m, pg->cnt);
+		copy_to_trf(trf, satp, pg->m, k);
+
+		/* free bad packets, if any */
+		free_pkts(pg->m + k, pg->cnt - k);
+
+		n -= pg->cnt;
+	}
+
+	/* we should never have packet with unknown SA here */
+	RTE_VERIFY(n == 0);
+}
diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
index 31f68fee2..81c146ebc 100644
--- a/examples/ipsec-secgw/meson.build
+++ b/examples/ipsec-secgw/meson.build
@@ -9,6 +9,6 @@
 deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
 allow_experimental_apis = true
 sources = files(
-	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
-	'rt.c', 'sa.c', 'sp4.c', 'sp6.c'
+	'esp.c', 'ipsec.c', 'ipsec_process.c', 'ipsec-secgw.c',
+	'parser.c', 'rt.c', 'sa.c', 'sp4.c', 'sp6.c'
 )
-- 
2.17.1

[PATCH v2 7/7] examples/ipsec-secgw: add scripts for functional test

[Konstantin Ananyev]

The purpose of these scripts is to automate ipsec-secgw functional testing.
The scripts require two machines (SUT and DUT) connected through
at least 2 NICs and running linux (so far tested only on Ubuntu 18.04).
Introduced test-cases for the following scenarios:
- Transport/Tunnel modes
- AES-CBC SHA1
- AES-GCM
- ESN on/off
- legacy/librte_ipsec code path

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/test/common_defs.sh      | 113 ++++++++++++++++++
 examples/ipsec-secgw/test/data_rxtx.sh        |  70 +++++++++++
 examples/ipsec-secgw/test/linux_test1.sh      |  62 ++++++++++
 examples/ipsec-secgw/test/run_test.sh         |  48 ++++++++
 .../test/trs_aescbc_sha1_common_defs.sh       |  42 +++++++
 .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  38 ++++++
 .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/trs_aescbc_sha1_esn_defs.sh          |  36 ++++++
 .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
 .../test/trs_aesgcm_common_defs.sh            |  38 ++++++
 examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  37 ++++++
 .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  36 ++++++
 .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
 .../test/tun_aescbc_sha1_common_defs.sh       |  42 +++++++
 .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  38 ++++++
 .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/tun_aescbc_sha1_esn_defs.sh          |  38 ++++++
 .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
 .../test/tun_aesgcm_common_defs.sh            |  38 ++++++
 examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  38 ++++++
 .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  38 ++++++
 .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
 24 files changed, 792 insertions(+)
 create mode 100644 examples/ipsec-secgw/test/common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test1.sh
 create mode 100644 examples/ipsec-secgw/test/run_test.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh

diff --git a/examples/ipsec-secgw/test/common_defs.sh b/examples/ipsec-secgw/test/common_defs.sh
new file mode 100644
index 000000000..835eca41d
--- /dev/null
+++ b/examples/ipsec-secgw/test/common_defs.sh
@@ -0,0 +1,113 @@
+#! /bin/bash
+
+#check that env vars are properly defined
+
+#check SGW_PATH
+if [[ -z "${SGW_PATH}" || ! -x ${SGW_PATH} ]]; then
+	echo "SGW_PATH is invalid"
+	exit 127
+fi
+
+#check ETH_DEV
+if [[ -z "${ETH_DEV}" ]]; then
+	echo "ETH_DEV is invalid"
+	exit 127
+fi
+
+#setup SGW_LCORE
+SGW_LCORE=${SGW_LCORE:-0}
+
+#check that REMOTE_HOST is reachable
+ssh ${REMOTE_HOST} echo
+st=$?
+if [[ $st -ne 0 ]]; then
+	echo "host ${REMOTE_HOST} is not reachable"
+	exit $st
+fi
+
+#get ether addr of REMOTE_HOST
+REMOTE_MAC=`ssh ${REMOTE_HOST} ip addr show dev ${REMOTE_IFACE}`
+st=$?
+REMOTE_MAC=`echo ${REMOTE_MAC} | sed -e 's/^.*ether //' -e 's/ brd.*$//'`
+if [[ $st -ne 0 || -z "${REMOTE_MAC}" ]]; then
+	echo "coouldn't retrieve ether addr from ${REMOTE_IFACE}"
+	exit 127
+fi
+
+LOCAL_IFACE=dtap0
+
+LOCAL_MAC="00:64:74:61:70:30"
+
+REMOTE_IPV4=192.168.31.14
+LOCAL_IPV4=192.168.31.92
+
+DPDK_PATH=${RTE_SDK:-${PWD}}
+DPDK_BUILD=${RTE_TARGET:-x86_64-native-linuxapp-gcc}
+
+SGW_OUT_FILE=./ipsec-secgw.out1
+
+SGW_CMD_EAL_PRM="--lcores=${SGW_LCORE} -n 4 ${ETH_DEV}"
+SGW_CMD_CFG="(0,0,${SGW_LCORE}),(1,0,${SGW_LCORE})"
+SGW_CMD_PRM="-p 0x3 -u 1 -P --config=\"${SGW_CMD_CFG}\""
+
+SGW_CFG_FILE=$(tempfile)
+
+# configure local host/ifaces
+config_local_iface()
+{
+	ifconfig ${LOCAL_IFACE} ${LOCAL_IPV4}/24 mtu 1440 up
+	ifconfig ${LOCAL_IFACE}
+
+	ip neigh flush dev ${LOCAL_IFACE}
+	arp -i ${LOCAL_IFACE} -s ${REMOTE_IPV4} ${LOCAL_MAC}
+	ip neigh show
+}
+
+#configure remote host/iface
+config_remote_iface()
+{
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE} down
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE} ${REMOTE_IPV4}/24 up
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} ip neigh flush dev ${REMOTE_IFACE}
+
+	# by some reason following ip neigh doesn't work for me here properly:
+	#ssh ${REMOTE_HOST} ip neigh add ${LOCAL_IPV4} lladr ${LOCAL_MAC} \
+		#       dev ${REMOTE_IFACE}
+	# so used arp instead.
+	ssh ${REMOTE_HOST} arp -i ${REMOTE_IFACE} -s ${LOCAL_IPV4} ${LOCAL_MAC}
+	ssh ${REMOTE_HOST} ip neigh show
+}
+
+config_iface()
+{
+	config_local_iface
+	config_remote_iface
+}
+
+#start ipsec-secgw
+secgw_start()
+{
+	SGW_EXEC_FILE=$(tempfile)
+	cat <<EOF > ${SGW_EXEC_FILE}
+${SGW_PATH} ${SGW_CMD_EAL_PRM} ${CRYPTO_DEV} \
+--vdev="net_tap0,mac=fixed" \
+-- ${SGW_CMD_PRM} ${SGW_CMD_XPRM} -f ${SGW_CFG_FILE} > \
+${SGW_OUT_FILE} 2>&1 &
+p=\$!
+echo \$p
+EOF
+
+	cat ${SGW_EXEC_FILE}
+	SGW_PID=`/bin/bash -x ${SGW_EXEC_FILE}`
+	sleep 1
+}
+
+#stop ipsec-secgw and cleanup
+secgw_stop()
+{
+	kill ${SGW_PID}
+	rm -f ${SGW_EXEC_FILE}
+	rm -f ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/data_rxtx.sh b/examples/ipsec-secgw/test/data_rxtx.sh
new file mode 100644
index 000000000..f2ff96eff
--- /dev/null
+++ b/examples/ipsec-secgw/test/data_rxtx.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+TCP_PORT=22222
+
+ping_test1()
+{
+	dst=$1
+
+	i=0
+	st=0
+	while [[ $i -ne 1400 && $st -eq 0 ]];
+	do
+		let i++
+		ping -c 1 -s ${i} ${dst}
+		st=$?
+	done
+
+	if [[ $st -ne 0 ]]; then
+		echo "ERROR: $0 failed for dst=${dst}, sz=${i}"
+	fi
+	return $st;
+}
+
+scp_test1()
+{
+	for sz in 1234 23456 345678 4567890 56789102 ; do
+		x=$(tempfile)
+		dd if=/dev/urandom of=${x}.in bs=${sz} count=1
+		scp ${x} ${REMOTE_IPV4}:${x}
+		scp ${REMOTE_IPV4}:${x} ${x}.copy1
+		diff -u ${x} ${x}.copy1
+		st=$?
+		rm -f ${x} ${x}.out
+		ssh ${REMOTE_HOST} rm -f ${x}
+		if [[ $st -ne 0 ]]; then
+			return $st
+		fi
+	done
+
+	return 0;
+}
+
+tcp_test1()
+{
+	for sz in 1234 23456 345678 4567890 56789102 ; do
+		x=`basename $0`.${sz}
+		rcmd=$(tempfile)
+		lcmd=$(tempfile)
+		dd if=/dev/urandom of=${x} bs=${sz} count=1
+		echo "nc -l ${REMOTE_IPV4} ${TCP_PORT} > ${x}; \
+			nc -q 0 ${LOCAL_IPV4} ${TCP_PORT} < ${x}; \
+			rm -f ${x}" > ${rcmd}
+		echo "nc -l ${LOCAL_IPV4} ${TCP_PORT} > ${x}.copy" > ${lcmd}
+		scp ${rcmd} ${REMOTE_HOST}:${rcmd}
+		ssh ${REMOTE_HOST} /bin/bash -x ${rcmd} &
+		/bin/bash -x ${lcmd} &
+		sleep 1
+		nc -q 0 ${REMOTE_IPV4} ${TCP_PORT} < ${x}
+		wait
+		diff -u ${x} ${x}.copy
+		st=$?
+		rm -f ${x} ${x}.copy ${rcmd} ${lcmd}
+		ssh  ${REMOTE_HOST} rm -f ${rcmd}
+		if [[ $st -ne 0 ]]; then
+			return $st
+		fi
+	done
+
+	return $st;
+}
diff --git a/examples/ipsec-secgw/test/linux_test1.sh b/examples/ipsec-secgw/test/linux_test1.sh
new file mode 100644
index 000000000..8265ba6ea
--- /dev/null
+++ b/examples/ipsec-secgw/test/linux_test1.sh
@@ -0,0 +1,62 @@
+#! /bin/bash
+
+# usage:  /bin/bash linux_test1.sh <ipsec_mode>
+# for list of available modes please refer to run_test.sh.
+#
+# Note that for most of them you required appropriate crypto PMD/device
+# to be avaialble.
+# Also user has to setup properly the following environment variables:
+#  SGW_PATH - path to the ipsec-secgw binary to test
+#  REMOTE_HOST - ip/hostname of the DUT
+#  REMOTE_IFACE - iface name for the test-port on DUT
+#  ETH_DEV - ethernet device to be used on SUT by DPDK ('-w <pci-id>')
+# Also user can optonally setup:
+#  SGW_LCORE - lcore to run ipsec-secgw on (default value is 0)
+#  CRYPTO_DEV - crypto device to be used ('-w <pci-id>')
+#  if none specified appropriate vdevs will be created by the scrit
+#
+# The purpose of the script is to automate ipsec-secgw testing
+# using another system running linux as a DUT.
+# It expects that SUT and DUT are connected through at least 2 NICs.
+# One NIC is expected to be managed by linux both machines,
+# and will be used as a control path
+# (make sure user from SUT can ssh to DUT without entering password).
+# Second NIC (test-port) should be reserved for DPDK on SUT,
+# and should be managed by linux on DUT.
+# The script starts ipsec-secgw with 2 NIC devices: test-port and tap vdev.
+# Then configures local tap iface and remote iface and ipsec policies
+# in the following way:
+# traffic going over test-port in both directions has to be
+# protected by ipsec.
+# raffic going over TAP in both directions doesn't have to be protected.
+# I.E:
+# DUT OS(NIC1)--(ipsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS
+# SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(ipsec)-->(NIC1)DUT OS
+# Then tries to perorm some data transfer using the scheme decribed above.
+#
+
+DIR=`dirname $0`
+MODE=$1
+
+ . ${DIR}/common_defs.sh
+ . ${DIR}/${MODE}_defs.sh
+
+config_secgw
+
+secgw_start
+
+config_iface
+
+config_remote_xfrm
+
+ . ${DIR}/data_rxtx.sh
+
+ping_test1 ${REMOTE_IPV4}
+st=$?
+if [[ $st -eq 0 ]]; then
+	tcp_test1
+	st=$?
+fi
+
+secgw_stop
+exit $st
diff --git a/examples/ipsec-secgw/test/run_test.sh b/examples/ipsec-secgw/test/run_test.sh
new file mode 100644
index 000000000..7d0298819
--- /dev/null
+++ b/examples/ipsec-secgw/test/run_test.sh
@@ -0,0 +1,48 @@
+#! /bin/bash
+
+# usage: /bin/bash run_test.sh
+# Run all defined linux_test1 test-cases one by one
+# user has to setup properly the following environment variables:
+#  SGW_PATH - path to the ipsec-secgw binary to test
+#  REMOTE_HOST - ip/hostname of the DUT
+#  REMOTE_IFACE - iface name for the test-port on DUT
+#  ETH_DEV - ethernet device to be used on SUT by DPDK ('-w <pci-id>')
+# Also user can optonally setup:
+#  SGW_LCORE - lcore to run ipsec-secgw on (default value is 0)
+#  CRYPTO_DEV - crypto device to be used ('-w <pci-id>')
+#  if none specified appropriate vdevs will be created by the scrit
+# refer to linux_test1.sh for more information
+
+# All supported modes to test.
+# naming convention:
+# 'old' means that ipsec-secgw will run in legacy (non-librte_ipsec mode)
+# 'tun/trs' refer to tunnel/transport mode respectively
+LINUX_TEST1="tun_aescbc_sha1 \
+tun_aescbc_sha1_esn \
+tun_aescbc_sha1_esn_atom \
+tun_aesgcm \
+tun_aesgcm_esn \
+tun_aesgcm_esn_atom \
+trs_aescbc_sha1 \
+trs_aescbc_sha1_esn \
+trs_aescbc_sha1_esn_atom \
+trs_aesgcm \
+trs_aesgcm_esn \
+trs_aesgcm_esn_atom \
+tun_aescbc_sha1_old \
+tun_aesgcm_old \
+trs_aescbc_sha1_old \
+trs_aesgcm_old"
+
+DIR=`dirname $0`
+
+for i in ${LINUX_TEST1}; do
+	echo "starting test ${i}"
+	/bin/bash ${DIR}/linux_test1.sh ${i}
+	st=$?
+	echo "test ${i} finished with status ${st}"
+	if [[ $st -ne 0 ]]; then
+		echo "ERROR test ${i} FAILED"
+		exit $st
+	fi
+done
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
new file mode 100644
index 000000000..40bea68a0
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
@@ -0,0 +1,42 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_mb0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#SA out rules
+sa out 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
new file mode 100644
index 000000000..db4720abc
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
@@ -0,0 +1,38 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
new file mode 100644
index 000000000..f16222e11
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
new file mode 100644
index 000000000..04f335023
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
@@ -0,0 +1,36 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
new file mode 100644
index 000000000..a3abb6103
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
new file mode 100644
index 000000000..239782d96
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
@@ -0,0 +1,38 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_gcm0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#SA out rules
+sa out 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_defs.sh
new file mode 100644
index 000000000..bc89522b0
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_defs.sh
@@ -0,0 +1,37 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
new file mode 100644
index 000000000..80d8d63b8
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
new file mode 100644
index 000000000..c0f3021e9
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
@@ -0,0 +1,36 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
new file mode 100644
index 000000000..951e6b68f
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
new file mode 100644
index 000000000..630808c21
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
@@ -0,0 +1,42 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_mb0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+sp ipv4 in esp protect 7 pri 2 src 192.168.0.0/16 dst 192.168.0.0/16 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src 192.168.0.0/16 dst 192.168.0.0/16 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${REMOTE_IPV4} dst ${LOCAL_IPV4}
+
+#SA out rules
+sa out 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${LOCAL_IPV4} dst ${REMOTE_IPV4}
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
new file mode 100644
index 000000000..2f6ffaeeb
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
@@ -0,0 +1,38 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
new file mode 100644
index 000000000..6b4a82149
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
new file mode 100644
index 000000000..0ca7684bc
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
@@ -0,0 +1,38 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
new file mode 100644
index 000000000..3c0d8d1b1
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
new file mode 100644
index 000000000..bebd38145
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
@@ -0,0 +1,38 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_gcm0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+sp ipv4 in esp protect 7 pri 2 src 192.168.0.0/16 dst 192.168.0.0/16 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src 192.168.0.0/16 dst 192.168.0.0/16 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${REMOTE_IPV4} dst ${LOCAL_IPV4}
+
+#SA out rules
+sa out 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${LOCAL_IPV4} dst ${REMOTE_IPV4}
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_defs.sh
new file mode 100644
index 000000000..d0bb9afae
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_defs.sh
@@ -0,0 +1,38 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
new file mode 100644
index 000000000..dab1460c8
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
new file mode 100644
index 000000000..814d5f016
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
@@ -0,0 +1,38 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh
new file mode 100644
index 000000000..e0a015e21
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_defs.sh
+
+SGW_CMD_XPRM=
-- 
2.17.1

[PATCH 2/7] examples/ipsec-secgw: allow to specify neighbor mac address

[Konstantin Ananyev]

In some cases it is useful to allow user to specify destination
ether address for outgoing packets.
This patch adds such ability by introducing new 'neigh' config
file option.

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 21 +++++++--
 examples/ipsec-secgw/ipsec.h       |  3 ++
 examples/ipsec-secgw/parser.c      | 75 ++++++++++++++++++++++++++++++
 examples/ipsec-secgw/parser.h      |  8 ++--
 4 files changed, 99 insertions(+), 8 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index cfc2b05e5..2ed757922 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -104,9 +104,9 @@ static uint16_t nb_txd = IPSEC_SECGW_TX_DESC_DEFAULT;
 #define ETHADDR(a, b, c, d, e, f) (__BYTES_TO_UINT64(a, b, c, d, e, f, 0, 0))
 
 #define ETHADDR_TO_UINT64(addr) __BYTES_TO_UINT64( \
-		addr.addr_bytes[0], addr.addr_bytes[1], \
-		addr.addr_bytes[2], addr.addr_bytes[3], \
-		addr.addr_bytes[4], addr.addr_bytes[5], \
+		(addr)->addr_bytes[0], (addr)->addr_bytes[1], \
+		(addr)->addr_bytes[2], (addr)->addr_bytes[3], \
+		(addr)->addr_bytes[4], (addr)->addr_bytes[5], \
 		0, 0)
 
 /* port/source ethernet addr and destination ethernet addr */
@@ -1188,6 +1188,19 @@ print_ethaddr(const char *name, const struct ether_addr *eth_addr)
 	printf("%s%s", name, buf);
 }
 
+/*
+ * Update destination ethaddr for the port.
+ */
+int
+add_dst_ethaddr(uint16_t port, const struct ether_addr *addr)
+{
+	if (port > RTE_DIM(ethaddr_tbl))
+		return -EINVAL;
+
+	ethaddr_tbl[port].dst = ETHADDR_TO_UINT64(addr);
+	return 0;
+}
+
 /* Check the link status of all ports in up to 9s, and print them finally */
 static void
 check_all_ports_link_status(uint32_t port_mask)
@@ -1564,7 +1577,7 @@ port_init(uint16_t portid, uint64_t req_rx_offloads, uint64_t req_tx_offloads)
 	printf("Configuring device port %u:\n", portid);
 
 	rte_eth_macaddr_get(portid, &ethaddr);
-	ethaddr_tbl[portid].src = ETHADDR_TO_UINT64(ethaddr);
+	ethaddr_tbl[portid].src = ETHADDR_TO_UINT64(&ethaddr);
 	print_ethaddr("Address: ", &ethaddr);
 	printf("\n");
 
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 9b1586f52..580f7876b 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -245,4 +245,7 @@ int
 sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
 		uint64_t *tx_offloads);
 
+int
+add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/parser.c b/examples/ipsec-secgw/parser.c
index 91282ca94..6769a8e6d 100644
--- a/examples/ipsec-secgw/parser.c
+++ b/examples/ipsec-secgw/parser.c
@@ -306,6 +306,30 @@ parse_range(const char *token, uint16_t *low, uint16_t *high)
 	return 0;
 }
 
+#define PARSE_UINT8x16(s, v, l)	                          \
+do {                                                      \
+	char *end;                                        \
+	unsigned long t;                                  \
+	errno = 0;                                        \
+	t = strtoul((s), &end, 16);                       \
+	if (errno != 0 || end[0] != (l) || t > UINT8_MAX) \
+		return -EINVAL;                           \
+	(s) = end + 1;                                    \
+	(v) = t;                                          \
+} while (0)
+
+static int
+parse_mac(const char *str, struct ether_addr *addr)
+{
+	PARSE_UINT8x16(str, addr->addr_bytes[0], ':');
+	PARSE_UINT8x16(str, addr->addr_bytes[1], ':');
+	PARSE_UINT8x16(str, addr->addr_bytes[2], ':');
+	PARSE_UINT8x16(str, addr->addr_bytes[3], ':');
+	PARSE_UINT8x16(str, addr->addr_bytes[4], ':');
+	PARSE_UINT8x16(str, addr->addr_bytes[5], 0);
+	return 0;
+}
+
 /** sp add parse */
 struct cfg_sp_add_cfg_item {
 	cmdline_fixed_string_t sp_keyword;
@@ -444,11 +468,61 @@ cmdline_parse_inst_t cfg_rt_add_rule = {
 	},
 };
 
+/* neigh add parse */
+struct cfg_neigh_add_item {
+	cmdline_fixed_string_t neigh;
+	cmdline_fixed_string_t pstr;
+	uint16_t port;
+	cmdline_fixed_string_t mac;
+};
+
+static void
+cfg_parse_neigh(void *parsed_result, __rte_unused struct cmdline *cl,
+	void *data)
+{
+	int32_t rc;
+	struct cfg_neigh_add_item *res;
+	struct parse_status *st;
+	struct ether_addr mac;
+
+	st = data;
+	res = parsed_result;
+	rc = parse_mac(res->mac, &mac);
+	APP_CHECK(rc == 0, st, "invalid ether addr:%s", res->mac);
+	rc = add_dst_ethaddr(res->port, &mac);
+	APP_CHECK(rc == 0, st, "invalid port numer:%hu", res->port);
+	if (st->status < 0)
+		return;
+}
+
+cmdline_parse_token_string_t cfg_add_neigh_start =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, neigh, "neigh");
+cmdline_parse_token_string_t cfg_add_neigh_pstr =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, pstr, "port");
+cmdline_parse_token_num_t cfg_add_neigh_port =
+	TOKEN_NUM_INITIALIZER(struct cfg_neigh_add_item, port, UINT16);
+cmdline_parse_token_string_t cfg_add_neigh_mac =
+	TOKEN_STRING_INITIALIZER(struct cfg_neigh_add_item, mac, NULL);
+
+cmdline_parse_inst_t cfg_neigh_add_rule = {
+	.f = cfg_parse_neigh,
+	.data = NULL,
+	.help_str = "",
+	.tokens = {
+		(void *)&cfg_add_neigh_start,
+		(void *)&cfg_add_neigh_pstr,
+		(void *)&cfg_add_neigh_port,
+		(void *)&cfg_add_neigh_mac,
+		NULL,
+	},
+};
+
 /** set of cfg items */
 cmdline_parse_ctx_t ipsec_ctx[] = {
 	(cmdline_parse_inst_t *)&cfg_sp_add_rule,
 	(cmdline_parse_inst_t *)&cfg_sa_add_rule,
 	(cmdline_parse_inst_t *)&cfg_rt_add_rule,
+	(cmdline_parse_inst_t *)&cfg_neigh_add_rule,
 	NULL,
 };
 
@@ -474,6 +548,7 @@ parse_cfg_file(const char *cfg_filename)
 	cfg_sp_add_rule.data = &status;
 	cfg_sa_add_rule.data = &status;
 	cfg_rt_add_rule.data = &status;
+	cfg_neigh_add_rule.data = &status;
 
 	do {
 		char oneline[1024];
diff --git a/examples/ipsec-secgw/parser.h b/examples/ipsec-secgw/parser.h
index be02537c5..6b8a10076 100644
--- a/examples/ipsec-secgw/parser.h
+++ b/examples/ipsec-secgw/parser.h
@@ -14,14 +14,14 @@ struct parse_status {
 	char parse_msg[256];
 };
 
-#define	APP_CHECK(exp, status, fmt, ...)				\
+#define	APP_CHECK(exp, st, fmt, ...)					\
 do {									\
 	if (!(exp)) {							\
-		sprintf(status->parse_msg, fmt "\n",			\
+		sprintf((st)->parse_msg, fmt "\n",			\
 			## __VA_ARGS__);				\
-		status->status = -1;					\
+		(st)->status = -1;					\
 	} else								\
-		status->status = 0;					\
+		(st)->status = 0;					\
 } while (0)
 
 #define APP_CHECK_PRESENCE(val, str, status)				\
-- 
2.17.1

[PATCH 3/7] examples/ipsec-secgw: fix crypto-op might never get dequeued

[Konstantin Ananyev]

In some cases crypto-ops could never be dequeued from the crypto-device.
The easiest way to reproduce:
start ipsec-secgw with crypto-dev and send to it less then 32 packets.
none packets will be forwarded.
Reason for that is that the application does dequeue() from crypto-queues
only when new packets arrive.
This patch makes sure it calls dequeue() on a regular basis.

Fixes: c64278c0c18b ("examples/ipsec-secgw: rework processing loop")

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 132 ++++++++++++++++++++++++-----
 examples/ipsec-secgw/ipsec.c       |  60 ++++++++-----
 examples/ipsec-secgw/ipsec.h       |  11 +++
 3 files changed, 161 insertions(+), 42 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 2ed757922..47c728eae 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -456,38 +456,55 @@ inbound_sp_sa(struct sp_ctx *sp, struct sa_ctx *sa, struct traffic_type *ip,
 	ip->num = j;
 }
 
-static inline void
-process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
-		struct ipsec_traffic *traffic)
+static void
+split46_traffic(struct ipsec_traffic *trf, struct rte_mbuf *mb[], uint32_t num)
 {
+	uint32_t i, n4, n6;
+	struct ip *ip;
 	struct rte_mbuf *m;
-	uint16_t idx, nb_pkts_in, i, n_ip4, n_ip6;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
+	n4 = trf->ip4.num;
+	n6 = trf->ip6.num;
 
-	n_ip4 = traffic->ip4.num;
-	n_ip6 = traffic->ip6.num;
+	for (i = 0; i < num; i++) {
+
+		m = mb[i];
+		ip = rte_pktmbuf_mtod(m, struct ip *);
 
-	/* SP/ACL Inbound check ipsec and ip4 */
-	for (i = 0; i < nb_pkts_in; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
 		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-			traffic->ip4.data[idx] = rte_pktmbuf_mtod_offset(m,
+			trf->ip4.pkts[n4] = m;
+			trf->ip4.data[n4] = rte_pktmbuf_mtod_offset(m,
 					uint8_t *, offsetof(struct ip, ip_p));
+			n4++;
 		} else if (ip->ip_v == IP6_VERSION) {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
-			traffic->ip6.data[idx] = rte_pktmbuf_mtod_offset(m,
+			trf->ip6.pkts[n6] = m;
+			trf->ip6.data[n6] = rte_pktmbuf_mtod_offset(m,
 					uint8_t *,
 					offsetof(struct ip6_hdr, ip6_nxt));
+			n6++;
 		} else
 			rte_pktmbuf_free(m);
 	}
 
+	trf->ip4.num = n4;
+	trf->ip6.num = n6;
+}
+
+
+static inline void
+process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
+		struct ipsec_traffic *traffic)
+{
+	uint16_t nb_pkts_in, n_ip4, n_ip6;
+
+	n_ip4 = traffic->ip4.num;
+	n_ip6 = traffic->ip6.num;
+
+	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.num, MAX_PKT_BURST);
+
+	split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+
 	inbound_sp_sa(ipsec_ctx->sp4_ctx, ipsec_ctx->sa_ctx, &traffic->ip4,
 			n_ip4);
 
@@ -782,7 +799,7 @@ process_pkts(struct lcore_conf *qconf, struct rte_mbuf **pkts,
 }
 
 static inline void
-drain_buffers(struct lcore_conf *qconf)
+drain_tx_buffers(struct lcore_conf *qconf)
 {
 	struct buffer *buf;
 	uint32_t portid;
@@ -796,6 +813,77 @@ drain_buffers(struct lcore_conf *qconf)
 	}
 }
 
+static inline void
+drain_crypto_buffers(struct lcore_conf *qconf)
+{
+	uint32_t i;
+	struct ipsec_ctx *ctx;
+
+	/* drain inbound buffers*/
+	ctx = &qconf->inbound;
+	for (i = 0; i != ctx->nb_qps; i++)
+		enqueue_cop_burst(ctx->tbl  + i);
+
+	/* drain outbound buffers*/
+	ctx = &qconf->outbound;
+	for (i = 0; i != ctx->nb_qps; i++)
+		enqueue_cop_burst(ctx->tbl  + i);
+}
+
+static void
+drain_inbound_crypto_queues(const struct lcore_conf *qconf,
+		struct ipsec_ctx *ctx)
+{
+	uint32_t n;
+	struct ipsec_traffic trf;
+
+	/* dequeue packets from crypto-queue */
+	n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+			RTE_DIM(trf.ipsec.pkts));
+	if (n == 0)
+		return;
+
+	trf.ip4.num = 0;
+	trf.ip6.num = 0;
+
+	/* split traffic by ipv4-ipv6 */
+	split46_traffic(&trf, trf.ipsec.pkts, n);
+
+	/* process ipv4 packets */
+	inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
+	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+
+	/* process ipv6 packets */
+	inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
+	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+}
+
+static void
+drain_outbound_crypto_queues(const struct lcore_conf *qconf,
+		struct ipsec_ctx *ctx)
+{
+	uint32_t n;
+	struct ipsec_traffic trf;
+
+	/* dequeue packets from crypto-queue */
+	n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+			RTE_DIM(trf.ipsec.pkts));
+	if (n == 0)
+		return;
+
+	trf.ip4.num = 0;
+	trf.ip6.num = 0;
+
+	/* split traffic by ipv4-ipv6 */
+	split46_traffic(&trf, trf.ipsec.pkts, n);
+
+	/* process ipv4 packets */
+	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+
+	/* process ipv6 packets */
+	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+}
+
 /* main processing loop */
 static int32_t
 main_loop(__attribute__((unused)) void *dummy)
@@ -853,7 +941,8 @@ main_loop(__attribute__((unused)) void *dummy)
 		diff_tsc = cur_tsc - prev_tsc;
 
 		if (unlikely(diff_tsc > drain_tsc)) {
-			drain_buffers(qconf);
+			drain_tx_buffers(qconf);
+			drain_crypto_buffers(qconf);
 			prev_tsc = cur_tsc;
 		}
 
@@ -867,6 +956,9 @@ main_loop(__attribute__((unused)) void *dummy)
 			if (nb_rx > 0)
 				process_pkts(qconf, pkts, nb_rx, portid);
 		}
+
+		drain_inbound_crypto_queues(qconf, &qconf->inbound);
+		drain_outbound_crypto_queues(qconf, &qconf->outbound);
 	}
 }
 
diff --git a/examples/ipsec-secgw/ipsec.c b/examples/ipsec-secgw/ipsec.c
index 3d415f1af..8bf928a23 100644
--- a/examples/ipsec-secgw/ipsec.c
+++ b/examples/ipsec-secgw/ipsec.c
@@ -333,33 +333,35 @@ create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa)
 	return 0;
 }
 
+/*
+ * queue crypto-ops into PMD queue.
+ */
+void
+enqueue_cop_burst(struct cdev_qp *cqp)
+{
+	uint32_t i, len, ret;
+
+	len = cqp->len;
+	ret = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp, cqp->buf, len);
+	if (ret < len) {
+		RTE_LOG_DP(DEBUG, IPSEC, "Cryptodev %u queue %u:"
+			" enqueued %u crypto ops out of %u\n",
+			cqp->id, cqp->qp, ret, len);
+			/* drop packets that we fail to enqueue */
+			for (i = ret; i < len; i++)
+				rte_pktmbuf_free(cqp->buf[i]->sym->m_src);
+	}
+	cqp->in_flight += ret;
+	cqp->len = 0;
+}
+
 static inline void
 enqueue_cop(struct cdev_qp *cqp, struct rte_crypto_op *cop)
 {
-	int32_t ret = 0, i;
-
 	cqp->buf[cqp->len++] = cop;
 
-	if (cqp->len == MAX_PKT_BURST) {
-		int enq_size = cqp->len;
-		if ((cqp->in_flight + enq_size) > MAX_INFLIGHT)
-			enq_size -=
-			    (int)((cqp->in_flight + enq_size) - MAX_INFLIGHT);
-
-		if (enq_size > 0)
-			ret = rte_cryptodev_enqueue_burst(cqp->id, cqp->qp,
-					cqp->buf, enq_size);
-		if (ret < cqp->len) {
-			RTE_LOG_DP(DEBUG, IPSEC, "Cryptodev %u queue %u:"
-					" enqueued %u crypto ops out of %u\n",
-					 cqp->id, cqp->qp,
-					 ret, cqp->len);
-			for (i = ret; i < cqp->len; i++)
-				rte_pktmbuf_free(cqp->buf[i]->sym->m_src);
-		}
-		cqp->in_flight += ret;
-		cqp->len = 0;
-	}
+	if (cqp->len == MAX_PKT_BURST)
+		enqueue_cop_burst(cqp);
 }
 
 static inline void
@@ -548,6 +550,13 @@ ipsec_inbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 	return ipsec_dequeue(esp_inbound_post, ctx, pkts, len);
 }
 
+uint16_t
+ipsec_inbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len)
+{
+	return ipsec_dequeue(esp_inbound_post, ctx, pkts, len);
+}
+
 uint16_t
 ipsec_outbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint32_t sa_idx[], uint16_t nb_pkts, uint16_t len)
@@ -560,3 +569,10 @@ ipsec_outbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 
 	return ipsec_dequeue(esp_outbound_post, ctx, pkts, len);
 }
+
+uint16_t
+ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len)
+{
+	return ipsec_dequeue(esp_outbound_post, ctx, pkts, len);
+}
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 580f7876b..2f04b7d68 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -184,6 +184,14 @@ uint16_t
 ipsec_outbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint32_t sa_idx[], uint16_t nb_pkts, uint16_t len);
 
+uint16_t
+ipsec_inbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len);
+
+uint16_t
+ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
+		uint16_t len);
+
 static inline uint16_t
 ipsec_metadata_size(void)
 {
@@ -248,4 +256,7 @@ sa_check_offloads(uint16_t port_id, uint64_t *rx_offloads,
 int
 add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
 
+void
+enqueue_cop_burst(struct cdev_qp *cqp);
+
 #endif /* __IPSEC_H__ */
-- 
2.17.1

[PATCH 4/7] examples/ipsec-secgw: fix outbound codepath for single SA

[Konstantin Ananyev]

Looking at process_pkts_outbound_nosp() there seems few issues:
- accessing mbuf after it was freed
- invoking ipsec_outbound() for ipv4 packets only
- copying number of packets, but not the mbuf pointers itself

that patch provides fixes for that issues.

Fixes: 906257e965b7 ("examples/ipsec-secgw: support IPv6")

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c | 32 ++++++++++++++++++++----------
 1 file changed, 22 insertions(+), 10 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 47c728eae..793c8f912 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -616,32 +616,44 @@ process_pkts_outbound_nosp(struct ipsec_ctx *ipsec_ctx,
 		struct ipsec_traffic *traffic)
 {
 	struct rte_mbuf *m;
-	uint32_t nb_pkts_out, i;
+	uint32_t nb_pkts_out, i, n;
 	struct ip *ip;
 
 	/* Drop any IPsec traffic from protected ports */
 	for (i = 0; i < traffic->ipsec.num; i++)
 		rte_pktmbuf_free(traffic->ipsec.pkts[i]);
 
-	traffic->ipsec.num = 0;
+	n = 0;
 
-	for (i = 0; i < traffic->ip4.num; i++)
-		traffic->ip4.res[i] = single_sa_idx;
+	for (i = 0; i < traffic->ip4.num; i++) {
+		traffic->ipsec.pkts[n] = traffic->ip4.pkts[i];
+		traffic->ipsec.res[n++] = single_sa_idx;
+	}
 
-	for (i = 0; i < traffic->ip6.num; i++)
-		traffic->ip6.res[i] = single_sa_idx;
+	for (i = 0; i < traffic->ip6.num; i++) {
+		traffic->ipsec.pkts[n] = traffic->ip6.pkts[i];
+		traffic->ipsec.res[n++] = single_sa_idx;
+	}
+
+	traffic->ip4.num = 0;
+	traffic->ip6.num = 0;
+	traffic->ipsec.num = n;
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ip4.pkts,
-			traffic->ip4.res, traffic->ip4.num,
+	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.res, traffic->ipsec.num,
 			MAX_PKT_BURST);
 
 	/* They all sue the same SA (ip4 or ip6 tunnel) */
 	m = traffic->ipsec.pkts[i];
 	ip = rte_pktmbuf_mtod(m, struct ip *);
-	if (ip->ip_v == IPVERSION)
+	if (ip->ip_v == IPVERSION) {
 		traffic->ip4.num = nb_pkts_out;
-	else
+		for (i = 0; i < nb_pkts_out; i++)
+			traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+	} else {
 		traffic->ip6.num = nb_pkts_out;
+		traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+	}
 }
 
 static inline int32_t
-- 
2.17.1

[PATCH 5/7] examples/ipsec-secgw: make app to use ipsec library

[Konstantin Ananyev]

Changes to make ipsec-secgw to utilize librte_ipsec library.
That patch provides:
 - changes in the related data structures.
 - changes in the initialization code.
 - new command-line parameters to enable librte_ipsec codepath
   and related features.

Note that right now by default current (non-librte_ipsec) code-path will
be used. User has to run application with new command-line option ('-l')
to enable new codepath.
The main reason for that:
 - current librte_ipsec doesn't support all ipsec algorithms
   and features that the app does.
 - allow users to run both versions in parallel for some time
   to figure out any functional or performance degradation with the
   new code.

It is planned to deprecate and remove non-librte_ipsec code path
in future releases.

Signed-off-by: Mohammad Abdul Awal <mohammad.abdul.awal@intel.com>
Signed-off-by: Bernard Iremonger <bernard.iremonger@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/ipsec-secgw.c |  44 ++++++-
 examples/ipsec-secgw/ipsec.h       |  14 +++
 examples/ipsec-secgw/meson.build   |   2 +-
 examples/ipsec-secgw/sa.c          | 185 ++++++++++++++++++++++++++++-
 4 files changed, 241 insertions(+), 4 deletions(-)

diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index 793c8f912..d031d365d 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -155,6 +155,9 @@ static uint32_t single_sa;
 static uint32_t single_sa_idx;
 static uint32_t frame_size;
 
+/* application wide librte_ipsec/SA parameters */
+struct app_sa_prm app_sa_prm = {.enable = 0};
+
 struct lcore_rx_queue {
 	uint16_t port_id;
 	uint8_t queue_id;
@@ -1059,6 +1062,10 @@ print_usage(const char *prgname)
 		" [-P]"
 		" [-u PORTMASK]"
 		" [-j FRAMESIZE]"
+		" [-l]"
+		" [-w REPLAY_WINDOW_SIZE]"
+		" [-e]"
+		" [-a]"
 		" -f CONFIG_FILE"
 		" --config (port,queue,lcore)[,(port,queue,lcore)]"
 		" [--single-sa SAIDX]"
@@ -1069,6 +1076,10 @@ print_usage(const char *prgname)
 		"  -u PORTMASK: Hexadecimal bitmask of unprotected ports\n"
 		"  -j FRAMESIZE: Enable jumbo frame with 'FRAMESIZE' as maximum\n"
 		"                packet size\n"
+		"  -l enables code-path that uses librte_ipsec\n"
+		"  -w REPLAY_WINDOW_SIZE specifies IPsec SQN replay window\n"
+		"     size for each SA\n"
+		"  -a enables SA SQN atomic behaviour\n"
 		"  -f CONFIG_FILE: Configuration file\n"
 		"  --config (port,queue,lcore): Rx queue configuration\n"
 		"  --single-sa SAIDX: Use single SA index for outbound traffic,\n"
@@ -1165,6 +1176,20 @@ parse_config(const char *q_arg)
 	return 0;
 }
 
+static void
+print_app_sa_prm(const struct app_sa_prm *prm)
+{
+	printf("librte_ipsec usage: %s\n",
+		(prm->enable == 0) ? "disabled" : "enabled");
+
+	if (prm->enable == 0)
+		return;
+
+	printf("replay window size: %u\n", prm->window_size);
+	printf("ESN: %s\n", (prm->enable_esn == 0) ? "disabled" : "enabled");
+	printf("SA flags: %#" PRIx64 "\n", prm->flags);
+}
+
 static int32_t
 parse_args(int32_t argc, char **argv)
 {
@@ -1176,7 +1201,7 @@ parse_args(int32_t argc, char **argv)
 
 	argvopt = argv;
 
-	while ((opt = getopt_long(argc, argvopt, "p:Pu:f:j:",
+	while ((opt = getopt_long(argc, argvopt, "aelp:Pu:f:j:w:",
 				lgopts, &option_index)) != EOF) {
 
 		switch (opt) {
@@ -1232,6 +1257,21 @@ parse_args(int32_t argc, char **argv)
 			}
 			printf("Enabled jumbo frames size %u\n", frame_size);
 			break;
+		case 'l':
+			app_sa_prm.enable = 1;
+			break;
+		case 'w':
+			app_sa_prm.enable = 1;
+			app_sa_prm.window_size = parse_decimal(optarg);
+			break;
+		case 'e':
+			app_sa_prm.enable = 1;
+			app_sa_prm.enable_esn = 1;
+			break;
+		case 'a':
+			app_sa_prm.enable = 1;
+			app_sa_prm.flags |= RTE_IPSEC_SAFLAG_SQN_ATOM;
+			break;
 		case CMD_LINE_OPT_CONFIG_NUM:
 			ret = parse_config(optarg);
 			if (ret) {
@@ -1276,6 +1316,8 @@ parse_args(int32_t argc, char **argv)
 		return -1;
 	}
 
+	print_app_sa_prm(&app_sa_prm);
+
 	if (optind >= 0)
 		argv[optind-1] = prgname;
 
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 2f04b7d68..45485b019 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -11,6 +11,7 @@
 #include <rte_crypto.h>
 #include <rte_security.h>
 #include <rte_flow.h>
+#include <rte_ipsec.h>
 
 #define RTE_LOGTYPE_IPSEC       RTE_LOGTYPE_USER1
 #define RTE_LOGTYPE_IPSEC_ESP   RTE_LOGTYPE_USER2
@@ -70,7 +71,20 @@ struct ip_addr {
 
 #define MAX_KEY_SIZE		32
 
+/*
+ * application wide SA parameters
+ */
+struct app_sa_prm {
+	uint32_t enable; /* use librte_ipsec API for ipsec pkt processing */
+	uint32_t window_size; /* replay window size */
+	uint32_t enable_esn;  /* enable/disable ESN support */
+	uint64_t flags;       /* rte_ipsec_sa_prm.flags */
+};
+
+extern struct app_sa_prm app_sa_prm;
+
 struct ipsec_sa {
+	struct rte_ipsec_session ips; /* one session per sa for now */
 	uint32_t spi;
 	uint32_t cdev_id_qp;
 	uint64_t seq;
diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
index 77d8b298f..31f68fee2 100644
--- a/examples/ipsec-secgw/meson.build
+++ b/examples/ipsec-secgw/meson.build
@@ -6,7 +6,7 @@
 # To build this example as a standalone application with an already-installed
 # DPDK instance, use 'make'
 
-deps += ['security', 'lpm', 'acl', 'hash']
+deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
 allow_experimental_apis = true
 sources = files(
 	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
diff --git a/examples/ipsec-secgw/sa.c b/examples/ipsec-secgw/sa.c
index ff8c4b829..282b40bea 100644
--- a/examples/ipsec-secgw/sa.c
+++ b/examples/ipsec-secgw/sa.c
@@ -19,6 +19,7 @@
 #include <rte_ip.h>
 #include <rte_random.h>
 #include <rte_ethdev.h>
+#include <rte_malloc.h>
 
 #include "ipsec.h"
 #include "esp.h"
@@ -694,6 +695,7 @@ print_one_sa_rule(const struct ipsec_sa *sa, int inbound)
 }
 
 struct sa_ctx {
+	void *satbl; /* pointer to array of rte_ipsec_sa objects*/
 	struct ipsec_sa sa[IPSEC_SA_MAX_ENTRIES];
 	union {
 		struct {
@@ -764,7 +766,10 @@ sa_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 {
 	struct ipsec_sa *sa;
 	uint32_t i, idx;
-	uint16_t iv_length;
+	uint16_t iv_length, aad_length;
+
+	/* for ESN upper 32 bits of SQN also need to be part of AAD */
+	aad_length = (app_sa_prm.enable_esn != 0) ? sizeof(uint32_t) : 0;
 
 	for (i = 0; i < nb_entries; i++) {
 		idx = SPI2IDX(entries[i].spi);
@@ -808,7 +813,7 @@ sa_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 			sa_ctx->xf[idx].a.aead.iv.offset = IV_OFFSET;
 			sa_ctx->xf[idx].a.aead.iv.length = iv_length;
 			sa_ctx->xf[idx].a.aead.aad_length =
-				sa->aad_len;
+				sa->aad_len + aad_length;
 			sa_ctx->xf[idx].a.aead.digest_length =
 				sa->digest_len;
 
@@ -901,9 +906,169 @@ sa_in_add_rules(struct sa_ctx *sa_ctx, const struct ipsec_sa entries[],
 	return sa_add_rules(sa_ctx, entries, nb_entries, 1);
 }
 
+/*
+ * helper function, fills parameters that are identical for all SAs
+ */
+static void
+fill_ipsec_app_sa_prm(struct rte_ipsec_sa_prm *prm,
+	const struct app_sa_prm *app_prm)
+{
+	memset(prm, 0, sizeof(*prm));
+
+	prm->flags = app_prm->flags;
+	prm->ipsec_xform.options.esn = app_prm->enable_esn;
+	prm->replay_win_sz = app_prm->window_size;
+}
+
+static void
+fill_ipsec_sa_prm(struct rte_ipsec_sa_prm *prm, const struct ipsec_sa *ss,
+	const struct ipv4_hdr *v4, struct ipv6_hdr *v6)
+{
+	fill_ipsec_app_sa_prm(prm, &app_sa_prm);
+
+	prm->userdata = (uintptr_t)ss;
+
+	/* setup ipsec xform */
+	prm->ipsec_xform.spi = ss->spi;
+	prm->ipsec_xform.salt = ss->salt;
+	prm->ipsec_xform.direction = ss->direction;
+	prm->ipsec_xform.proto = RTE_SECURITY_IPSEC_SA_PROTO_ESP;
+	prm->ipsec_xform.mode = (ss->flags == TRANSPORT) ?
+		RTE_SECURITY_IPSEC_SA_MODE_TRANSPORT :
+		RTE_SECURITY_IPSEC_SA_MODE_TUNNEL;
+
+	/*
+	 * !!! this has to be changed.
+	 * Below we fill tun.next_proto based on the assumption that
+	 * input packet IP version is equal to tunnel IP version.
+	 * That could not be the case.
+	 * SA should know it's input IP version.
+	 */
+	if (ss->flags == IP4_TUNNEL) {
+		prm->ipsec_xform.tunnel.type = RTE_SECURITY_IPSEC_TUNNEL_IPV4;
+		prm->tun.hdr_len = sizeof(*v4);
+		prm->tun.next_proto = IPPROTO_IPIP;
+		prm->tun.hdr = v4;
+	} else if (ss->flags == IP6_TUNNEL) {
+		prm->ipsec_xform.tunnel.type = RTE_SECURITY_IPSEC_TUNNEL_IPV6;
+		prm->tun.hdr_len = sizeof(*v6);
+		prm->tun.next_proto = IPPROTO_IPV6;
+		prm->tun.hdr = v6;
+	} else {
+		prm->trs.proto = IPPROTO_IPIP;
+	}
+	/* setup crypto section */
+	prm->crypto_xform = ss->xforms;
+}
+
+static void
+fill_ipsec_session(struct rte_ipsec_session *ss, struct rte_ipsec_sa *sa,
+	const struct ipsec_sa *lsa)
+{
+	ss->sa = sa;
+	ss->type = lsa->type;
+
+	/* setup crypto section */
+	if (ss->type == RTE_SECURITY_ACTION_TYPE_NONE) {
+		ss->crypto.ses = lsa->crypto_session;
+	/* setup session action type */
+	} else {
+		ss->security.ses = lsa->sec_session;
+		ss->security.ctx = lsa->security_ctx;
+		ss->security.ol_flags = lsa->ol_flags;
+	}
+}
+
+/*
+ * Initialise related rte_ipsec_sa object.
+ */
+static int
+ipsec_sa_init(struct ipsec_sa *lsa, struct rte_ipsec_sa *sa, uint32_t sa_size)
+{
+	int rc;
+	struct rte_ipsec_sa_prm prm;
+	struct ipv4_hdr v4  = {
+		.version_ihl = IPVERSION << 4 |
+			sizeof(v4) / IPV4_IHL_MULTIPLIER,
+		.time_to_live = IPDEFTTL,
+		.next_proto_id = IPPROTO_ESP,
+		.src_addr = lsa->src.ip.ip4,
+		.dst_addr = lsa->dst.ip.ip4,
+	};
+	struct ipv6_hdr v6 = {
+		.vtc_flow = htonl(IP6_VERSION << 28),
+		.proto = IPPROTO_ESP,
+	};
+
+	if (lsa->flags == IP6_TUNNEL) {
+		memcpy(v6.src_addr, lsa->src.ip.ip6.ip6_b, sizeof(v6.src_addr));
+		memcpy(v6.dst_addr, lsa->dst.ip.ip6.ip6_b, sizeof(v6.dst_addr));
+	}
+
+	fill_ipsec_sa_prm(&prm, lsa, &v4, &v6);
+	rc = rte_ipsec_sa_init(sa, &prm, sa_size);
+	if (rc < 0)
+		return rc;
+
+	fill_ipsec_session(&lsa->ips, sa, lsa);
+	return 0;
+}
+
+/*
+ * Allocate space and init rte_ipsec_sa strcutures,
+ * one per session.
+ */
+static int
+ipsec_satbl_init(struct sa_ctx *ctx, const struct ipsec_sa *ent,
+	uint32_t nb_ent, int32_t socket)
+{
+	int32_t rc, sz;
+	uint32_t i, idx;
+	size_t tsz;
+	struct rte_ipsec_sa *sa;
+	struct ipsec_sa *lsa;
+	struct rte_ipsec_sa_prm prm;
+
+	/* determine SA size */
+	idx = SPI2IDX(ent[0].spi);
+	fill_ipsec_app_sa_prm(&prm, &app_sa_prm);
+	prm.ipsec_xform.direction = ctx->sa[idx].direction;
+	sz = rte_ipsec_sa_size(&prm);
+	if (sz < 0) {
+		RTE_LOG(ERR, IPSEC, "%s(%p, %u, %d): "
+			"failed to determine SA size, error code: %d\n",
+			__func__, ctx, nb_ent, socket, sz);
+		return sz;
+	}
+
+	tsz = sz * nb_ent;
+
+	ctx->satbl = rte_zmalloc_socket(NULL, tsz, RTE_CACHE_LINE_SIZE, socket);
+	if (ctx->satbl == NULL) {
+		RTE_LOG(ERR, IPSEC,
+			"%s(%p, %u, %d): failed to allocate %zu bytes\n",
+			__func__,  ctx, nb_ent, socket, tsz);
+		return -ENOMEM;
+	}
+
+	rc = 0;
+	for (i = 0; i != nb_ent && rc == 0; i++) {
+
+		idx = SPI2IDX(ent[i].spi);
+
+		sa = (struct rte_ipsec_sa *)((uintptr_t)ctx->satbl + sz * i);
+		lsa = ctx->sa + idx;
+
+		rc = ipsec_sa_init(lsa, sa, sz);
+	}
+
+	return rc;
+}
+
 void
 sa_init(struct socket_ctx *ctx, int32_t socket_id)
 {
+	int32_t rc;
 	const char *name;
 
 	if (ctx == NULL)
@@ -926,6 +1091,14 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id)
 				name, socket_id);
 
 		sa_in_add_rules(ctx->sa_in, sa_in, nb_sa_in);
+
+		if (app_sa_prm.enable != 0) {
+			rc = ipsec_satbl_init(ctx->sa_in, sa_in, nb_sa_in,
+				socket_id);
+			if (rc != 0)
+				rte_exit(EXIT_FAILURE,
+					"failed to init inbound SAs\n");
+		}
 	} else
 		RTE_LOG(WARNING, IPSEC, "No SA Inbound rule specified\n");
 
@@ -938,6 +1111,14 @@ sa_init(struct socket_ctx *ctx, int32_t socket_id)
 				name, socket_id);
 
 		sa_out_add_rules(ctx->sa_out, sa_out, nb_sa_out);
+
+		if (app_sa_prm.enable != 0) {
+			rc = ipsec_satbl_init(ctx->sa_out, sa_out, nb_sa_out,
+				socket_id);
+			if (rc != 0)
+				rte_exit(EXIT_FAILURE,
+					"failed to init outbound SAs\n");
+		}
 	} else
 		RTE_LOG(WARNING, IPSEC, "No SA Outbound rule "
 			"specified\n");
-- 
2.17.1

[PATCH 6/7] examples/ipsec-secgw: make data-path to use ipsec library

[Konstantin Ananyev]

Changes to make ipsec-secgw data-path code to utilize librte_ipsec library.
Note that right now by default current (non-librte_ipsec) code-path will
be used. User has to run application with new command-line option ('-l')
to enable new codepath.

Signed-off-by: Mohammad Abdul Awal <mohammad.abdul.awal@intel.com>
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/Makefile        |   1 +
 examples/ipsec-secgw/ipsec-secgw.c   | 178 ++++++++++--------
 examples/ipsec-secgw/ipsec.c         |   2 +-
 examples/ipsec-secgw/ipsec.h         |  23 +++
 examples/ipsec-secgw/ipsec_process.c | 266 +++++++++++++++++++++++++++
 examples/ipsec-secgw/meson.build     |   4 +-
 6 files changed, 396 insertions(+), 78 deletions(-)
 create mode 100644 examples/ipsec-secgw/ipsec_process.c

diff --git a/examples/ipsec-secgw/Makefile b/examples/ipsec-secgw/Makefile
index 02d41e39a..4d6b4cfa4 100644
--- a/examples/ipsec-secgw/Makefile
+++ b/examples/ipsec-secgw/Makefile
@@ -13,6 +13,7 @@ SRCS-y += sp4.c
 SRCS-y += sp6.c
 SRCS-y += sa.c
 SRCS-y += rt.c
+SRCS-y += ipsec_process.c
 SRCS-y += ipsec-secgw.c
 
 CFLAGS += -gdwarf-2
diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c
index d031d365d..e4e996159 100644
--- a/examples/ipsec-secgw/ipsec-secgw.c
+++ b/examples/ipsec-secgw/ipsec-secgw.c
@@ -216,19 +216,6 @@ static struct rte_eth_conf port_conf = {
 
 static struct socket_ctx socket_ctx[NB_SOCKETS];
 
-struct traffic_type {
-	const uint8_t *data[MAX_PKT_BURST * 2];
-	struct rte_mbuf *pkts[MAX_PKT_BURST * 2];
-	uint32_t res[MAX_PKT_BURST * 2];
-	uint32_t num;
-};
-
-struct ipsec_traffic {
-	struct traffic_type ipsec;
-	struct traffic_type ip4;
-	struct traffic_type ip6;
-};
-
 static inline void
 prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 {
@@ -245,6 +232,8 @@ prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 			t->ip4.data[t->ip4.num] = nlp;
 			t->ip4.pkts[(t->ip4.num)++] = pkt;
 		}
+		pkt->l2_len = 0;
+		pkt->l3_len = sizeof(struct ip);
 	} else if (eth->ether_type == rte_cpu_to_be_16(ETHER_TYPE_IPv6)) {
 		nlp = (uint8_t *)rte_pktmbuf_adj(pkt, ETHER_HDR_LEN);
 		nlp = RTE_PTR_ADD(nlp, offsetof(struct ip6_hdr, ip6_nxt));
@@ -254,6 +243,8 @@ prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t)
 			t->ip6.data[t->ip6.num] = nlp;
 			t->ip6.pkts[(t->ip6.num)++] = pkt;
 		}
+		pkt->l2_len = 0;
+		pkt->l3_len = sizeof(struct ip6_hdr);
 	} else {
 		/* Unknown/Unsupported type, drop the packet */
 		RTE_LOG(ERR, IPSEC, "Unsupported packet type\n");
@@ -503,10 +494,15 @@ process_pkts_inbound(struct ipsec_ctx *ipsec_ctx,
 	n_ip4 = traffic->ip4.num;
 	n_ip6 = traffic->ip6.num;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
-
-	split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+	if (app_sa_prm.enable == 0) {
+		nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.num, MAX_PKT_BURST);
+		split46_traffic(traffic, traffic->ipsec.pkts, nb_pkts_in);
+	} else {
+		inbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
+	}
 
 	inbound_sp_sa(ipsec_ctx->sp4_ctx, ipsec_ctx->sa_ctx, &traffic->ip4,
 			n_ip4);
@@ -562,20 +558,27 @@ process_pkts_outbound(struct ipsec_ctx *ipsec_ctx,
 
 	outbound_sp(ipsec_ctx->sp6_ctx, &traffic->ip6, &traffic->ipsec);
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.res, traffic->ipsec.num,
-			MAX_PKT_BURST);
-
-	for (i = 0; i < nb_pkts_out; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
-		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-		} else {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
+	if (app_sa_prm.enable == 0) {
+
+		nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.res, traffic->ipsec.num,
+				MAX_PKT_BURST);
+
+		for (i = 0; i < nb_pkts_out; i++) {
+			m = traffic->ipsec.pkts[i];
+			struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
+			if (ip->ip_v == IPVERSION) {
+				idx = traffic->ip4.num++;
+				traffic->ip4.pkts[idx] = m;
+			} else {
+				idx = traffic->ip6.num++;
+				traffic->ip6.pkts[idx] = m;
+			}
 		}
+	} else {
+		outbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.res,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -598,19 +601,26 @@ process_pkts_inbound_nosp(struct ipsec_ctx *ipsec_ctx,
 
 	traffic->ip6.num = 0;
 
-	nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.num, MAX_PKT_BURST);
+	if (app_sa_prm.enable == 0) {
 
-	for (i = 0; i < nb_pkts_in; i++) {
-		m = traffic->ipsec.pkts[i];
-		struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
-		if (ip->ip_v == IPVERSION) {
-			idx = traffic->ip4.num++;
-			traffic->ip4.pkts[idx] = m;
-		} else {
-			idx = traffic->ip6.num++;
-			traffic->ip6.pkts[idx] = m;
+		nb_pkts_in = ipsec_inbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.num, MAX_PKT_BURST);
+
+		for (i = 0; i < nb_pkts_in; i++) {
+			m = traffic->ipsec.pkts[i];
+			struct ip *ip = rte_pktmbuf_mtod(m, struct ip *);
+			if (ip->ip_v == IPVERSION) {
+				idx = traffic->ip4.num++;
+				traffic->ip4.pkts[idx] = m;
+			} else {
+				idx = traffic->ip6.num++;
+				traffic->ip6.pkts[idx] = m;
+			}
 		}
+	} else {
+		inbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.pkts,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -642,20 +652,28 @@ process_pkts_outbound_nosp(struct ipsec_ctx *ipsec_ctx,
 	traffic->ip6.num = 0;
 	traffic->ipsec.num = n;
 
-	nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
-			traffic->ipsec.res, traffic->ipsec.num,
-			MAX_PKT_BURST);
+	if (app_sa_prm.enable == 0) {
 
-	/* They all sue the same SA (ip4 or ip6 tunnel) */
-	m = traffic->ipsec.pkts[i];
-	ip = rte_pktmbuf_mtod(m, struct ip *);
-	if (ip->ip_v == IPVERSION) {
-		traffic->ip4.num = nb_pkts_out;
-		for (i = 0; i < nb_pkts_out; i++)
-			traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+		nb_pkts_out = ipsec_outbound(ipsec_ctx, traffic->ipsec.pkts,
+				traffic->ipsec.res, traffic->ipsec.num,
+				MAX_PKT_BURST);
+
+		/* They all sue the same SA (ip4 or ip6 tunnel) */
+		m = traffic->ipsec.pkts[0];
+		ip = rte_pktmbuf_mtod(m, struct ip *);
+		if (ip->ip_v == IPVERSION) {
+			traffic->ip4.num = nb_pkts_out;
+			for (i = 0; i < nb_pkts_out; i++)
+				traffic->ip4.pkts[i] = traffic->ipsec.pkts[i];
+		} else {
+			traffic->ip6.num = nb_pkts_out;
+			for (i = 0; i < nb_pkts_out; i++)
+				traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+		}
 	} else {
-		traffic->ip6.num = nb_pkts_out;
-		traffic->ip6.pkts[i] = traffic->ipsec.pkts[i];
+		outbound_sa_lookup(ipsec_ctx->sa_ctx, traffic->ipsec.res,
+			traffic->ipsec.saptr, traffic->ipsec.num);
+		ipsec_process(ipsec_ctx, traffic);
 	}
 }
 
@@ -852,25 +870,31 @@ drain_inbound_crypto_queues(const struct lcore_conf *qconf,
 	uint32_t n;
 	struct ipsec_traffic trf;
 
-	/* dequeue packets from crypto-queue */
-	n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+	if (app_sa_prm.enable == 0) {
+
+		/* dequeue packets from crypto-queue */
+		n = ipsec_inbound_cqp_dequeue(ctx, trf.ipsec.pkts,
 			RTE_DIM(trf.ipsec.pkts));
-	if (n == 0)
-		return;
 
-	trf.ip4.num = 0;
-	trf.ip6.num = 0;
+		trf.ip4.num = 0;
+		trf.ip6.num = 0;
 
-	/* split traffic by ipv4-ipv6 */
-	split46_traffic(&trf, trf.ipsec.pkts, n);
+		/* split traffic by ipv4-ipv6 */
+		split46_traffic(&trf, trf.ipsec.pkts, n);
+	} else
+		ipsec_cqp_process(ctx, &trf);
 
 	/* process ipv4 packets */
-	inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
-	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	if (trf.ip4.num != 0) {
+		inbound_sp_sa(ctx->sp4_ctx, ctx->sa_ctx, &trf.ip4, 0);
+		route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	}
 
 	/* process ipv6 packets */
-	inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
-	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	if (trf.ip6.num != 0) {
+		inbound_sp_sa(ctx->sp6_ctx, ctx->sa_ctx, &trf.ip6, 0);
+		route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	}
 }
 
 static void
@@ -880,23 +904,27 @@ drain_outbound_crypto_queues(const struct lcore_conf *qconf,
 	uint32_t n;
 	struct ipsec_traffic trf;
 
-	/* dequeue packets from crypto-queue */
-	n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
+	if (app_sa_prm.enable == 0) {
+
+		/* dequeue packets from crypto-queue */
+		n = ipsec_outbound_cqp_dequeue(ctx, trf.ipsec.pkts,
 			RTE_DIM(trf.ipsec.pkts));
-	if (n == 0)
-		return;
 
-	trf.ip4.num = 0;
-	trf.ip6.num = 0;
+		trf.ip4.num = 0;
+		trf.ip6.num = 0;
 
-	/* split traffic by ipv4-ipv6 */
-	split46_traffic(&trf, trf.ipsec.pkts, n);
+		/* split traffic by ipv4-ipv6 */
+		split46_traffic(&trf, trf.ipsec.pkts, n);
+	} else
+		ipsec_cqp_process(ctx, &trf);
 
 	/* process ipv4 packets */
-	route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
+	if (trf.ip4.num != 0)
+		route4_pkts(qconf->rt4_ctx, trf.ip4.pkts, trf.ip4.num);
 
 	/* process ipv6 packets */
-	route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
+	if (trf.ip6.num != 0)
+		route6_pkts(qconf->rt6_ctx, trf.ip6.pkts, trf.ip6.num);
 }
 
 /* main processing loop */
diff --git a/examples/ipsec-secgw/ipsec.c b/examples/ipsec-secgw/ipsec.c
index 8bf928a23..8e7fc3639 100644
--- a/examples/ipsec-secgw/ipsec.c
+++ b/examples/ipsec-secgw/ipsec.c
@@ -39,7 +39,7 @@ set_ipsec_conf(struct ipsec_sa *sa, struct rte_security_ipsec_xform *ipsec)
 	ipsec->esn_soft_limit = IPSEC_OFFLOAD_ESN_SOFTLIMIT;
 }
 
-static inline int
+int
 create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa)
 {
 	struct rte_cryptodev_info cdev_info;
diff --git a/examples/ipsec-secgw/ipsec.h b/examples/ipsec-secgw/ipsec.h
index 45485b019..c6e460e6f 100644
--- a/examples/ipsec-secgw/ipsec.h
+++ b/examples/ipsec-secgw/ipsec.h
@@ -190,6 +190,20 @@ struct cnt_blk {
 	uint32_t cnt;
 } __attribute__((packed));
 
+struct traffic_type {
+	const uint8_t *data[MAX_PKT_BURST * 2];
+	struct rte_mbuf *pkts[MAX_PKT_BURST * 2];
+	struct ipsec_sa *saptr[MAX_PKT_BURST * 2];
+	uint32_t res[MAX_PKT_BURST * 2];
+	uint32_t num;
+};
+
+struct ipsec_traffic {
+	struct traffic_type ipsec;
+	struct traffic_type ip4;
+	struct traffic_type ip6;
+};
+
 uint16_t
 ipsec_inbound(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint16_t nb_pkts, uint16_t len);
@@ -206,6 +220,12 @@ uint16_t
 ipsec_outbound_cqp_dequeue(struct ipsec_ctx *ctx, struct rte_mbuf *pkts[],
 		uint16_t len);
 
+void
+ipsec_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf);
+
+void
+ipsec_cqp_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf);
+
 static inline uint16_t
 ipsec_metadata_size(void)
 {
@@ -273,4 +293,7 @@ add_dst_ethaddr(uint16_t port, const struct ether_addr *addr);
 void
 enqueue_cop_burst(struct cdev_qp *cqp);
 
+int
+create_session(struct ipsec_ctx *ipsec_ctx, struct ipsec_sa *sa);
+
 #endif /* __IPSEC_H__ */
diff --git a/examples/ipsec-secgw/ipsec_process.c b/examples/ipsec-secgw/ipsec_process.c
new file mode 100644
index 000000000..c90e764b3
--- /dev/null
+++ b/examples/ipsec-secgw/ipsec_process.c
@@ -0,0 +1,266 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2016-2017 Intel Corporation
+ */
+#include <sys/types.h>
+#include <netinet/in.h>
+#include <netinet/ip.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_log.h>
+#include <rte_cryptodev.h>
+#include <rte_ethdev.h>
+#include <rte_mbuf.h>
+
+#include "ipsec.h"
+
+#define SATP_OUT_IPV4(t)	\
+	((((t) & RTE_IPSEC_SATP_MODE_MASK) == RTE_IPSEC_SATP_MODE_TRANS && \
+	(((t) & RTE_IPSEC_SATP_IPV_MASK) == RTE_IPSEC_SATP_IPV4)) || \
+	((t) & RTE_IPSEC_SATP_MODE_MASK) == RTE_IPSEC_SATP_MODE_TUNLV4)
+
+
+/* helper routine to free bulk of packets */
+static inline void
+free_pkts(struct rte_mbuf *mb[], uint32_t n)
+{
+	uint32_t i;
+
+	for (i = 0; i != n; i++)
+		rte_pktmbuf_free(mb[i]);
+}
+
+static inline int
+fill_ipsec_session(struct rte_ipsec_session *ss, const struct ipsec_sa *sa)
+{
+	/* setup crypto section */
+	if (ss->type == RTE_SECURITY_ACTION_TYPE_NONE) {
+		ss->crypto.ses = sa->crypto_session;
+	/* setup session action type */
+	} else {
+		ss->security.ses = sa->sec_session;
+		ss->security.ctx = sa->security_ctx;
+		ss->security.ol_flags = sa->ol_flags;
+	}
+
+	return rte_ipsec_session_prepare(ss);
+}
+
+/*
+ * group input packets byt the SA they belong to.
+ */
+static uint32_t
+sa_group(struct ipsec_sa *sa_ptr[], struct rte_mbuf *pkts[],
+	struct rte_ipsec_group grp[], uint32_t num)
+{
+	uint32_t i, n, spi;
+	void *sa;
+	void * const nosa = &spi;
+
+	sa = nosa;
+	for (i = 0, n = 0; i != num; i++) {
+
+		if (sa != sa_ptr[i]) {
+			grp[n].cnt = pkts + i - grp[n].m;
+			n += (sa != nosa);
+			grp[n].id.ptr = sa_ptr[i];
+			grp[n].m = pkts + i;
+			sa = sa_ptr[i];
+		}
+	}
+
+	/* terminate last group */
+	if (sa != nosa) {
+		grp[n].cnt = pkts + i - grp[n].m;
+		n++;
+	}
+
+	return n;
+}
+
+/*
+ * helper function, splits processed packets into ipv4/ipv6 traffic.
+ */
+static inline void
+copy_to_trf(struct ipsec_traffic *trf, uint64_t satp, struct rte_mbuf *mb[],
+	uint32_t num)
+{
+	uint32_t j, ofs, s;
+	struct traffic_type *out;
+
+	/*
+	 * determine traffic type(ipv4/ipv6) and offset for ACL classify
+	 * based on SA type
+	 */
+	if ((satp & RTE_IPSEC_SATP_DIR_MASK) == RTE_IPSEC_SATP_DIR_IB) {
+		if ((satp & RTE_IPSEC_SATP_IPV_MASK) == RTE_IPSEC_SATP_IPV4) {
+			out = &trf->ip4;
+			ofs = offsetof(struct ip, ip_p);
+		} else {
+			out = &trf->ip6;
+			ofs = offsetof(struct ip6_hdr, ip6_nxt);
+		}
+	} else if (SATP_OUT_IPV4(satp)) {
+		out = &trf->ip4;
+		ofs = offsetof(struct ip, ip_p);
+	} else {
+		out = &trf->ip6;
+		ofs = offsetof(struct ip6_hdr, ip6_nxt);
+	}
+
+	for (j = 0, s = out->num; j != num; j++) {
+		out->data[s + j] = rte_pktmbuf_mtod_offset(mb[j],
+				void *, ofs);
+		out->pkts[s + j] = mb[j];
+	}
+
+	out->num += num;
+}
+
+/*
+ * Process ipsec packets.
+ * If packet belong to SA that is subject of inline-crypto,
+ * then process it immediately.
+ * Otherwise do necessary preparations and queue it to related
+ * crypto-dev queue.
+ */
+void
+ipsec_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf)
+{
+	uint64_t satp;
+	uint32_t i, j, k, n;
+	struct ipsec_sa *sa;
+	struct ipsec_mbuf_metadata *priv;
+	struct rte_ipsec_group *pg;
+	struct rte_ipsec_session *ips;
+	struct cdev_qp *cqp;
+	struct rte_crypto_op *cop[RTE_DIM(trf->ipsec.pkts)];
+	struct rte_ipsec_group grp[RTE_DIM(trf->ipsec.pkts)];
+
+	n = sa_group(trf->ipsec.saptr, trf->ipsec.pkts, grp, trf->ipsec.num);
+
+	for (i = 0; i != n; i++) {
+
+		pg = grp + i;
+		sa = pg->id.ptr;
+
+		/* no valid SA found */
+		if (sa == NULL)
+			k = 0;
+
+		ips = &sa->ips;
+		satp = rte_ipsec_sa_type(ips->sa);
+
+		/* no valid HW session for that SA, try to create one */
+		if (ips->crypto.ses == NULL &&
+				(create_session(ctx, sa) != 0 ||
+				fill_ipsec_session(ips, sa) != 0))
+			k = 0;
+
+		/* process packets inline */
+		else if (sa->type == RTE_SECURITY_ACTION_TYPE_INLINE_CRYPTO ||
+				sa->type ==
+				RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL) {
+
+			k = rte_ipsec_pkt_process(ips, pg->m, pg->cnt);
+			copy_to_trf(trf, satp, pg->m, k);
+
+		/* enqueue packets to crypto dev */
+		} else {
+
+			cqp = &ctx->tbl[sa->cdev_id_qp];
+
+			/* for that app each mbuf has it's own crypto op */
+			for (j = 0; j != pg->cnt; j++) {
+				priv = get_priv(pg->m[j]);
+				cop[j] = &priv->cop;
+				/*
+				 * this is just to satisfy inbound_sa_check()
+				 * should be removed in future.
+				 */
+				priv->sa = sa;
+			}
+
+			/* prepare and enqueue crypto ops */
+			k = rte_ipsec_pkt_crypto_prepare(ips, pg->m, cop,
+				pg->cnt);
+			if (k != 0)
+				k = rte_cryptodev_enqueue_burst(cqp->id,
+					cqp->qp, cop, k);
+		}
+
+		/* drop packets that cannot be enqueued/processed */
+		if (k != pg->cnt)
+			free_pkts(pg->m + k, pg->cnt - k);
+	}
+}
+
+static inline uint32_t
+cqp_dequeue(struct ipsec_ctx *ctx, struct rte_crypto_op *cop[], uint32_t num)
+{
+	uint32_t i, n;
+	struct cdev_qp *cqp;
+
+	n = 0;
+
+	for (i = ctx->last_qp; n != num && i != ctx->nb_qps; i++) {
+		cqp = ctx->tbl + i;
+		n += rte_cryptodev_dequeue_burst(cqp->id, cqp->qp, cop + n,
+				num - n);
+	}
+	for (i = 0; n != num && i != ctx->last_qp; i++) {
+		cqp = ctx->tbl + i;
+		n += rte_cryptodev_dequeue_burst(cqp->id, cqp->qp, cop + n,
+				num - n);
+	}
+
+	ctx->last_qp = i;
+	return n;
+}
+
+/*
+ * dequeue packets from crypto-queues and finalize processing.
+ */
+void
+ipsec_cqp_process(struct ipsec_ctx *ctx, struct ipsec_traffic *trf)
+{
+	uint64_t satp;
+	uint32_t i, k, n, ng;
+	struct rte_ipsec_session *ss;
+	struct traffic_type *out;
+	struct rte_ipsec_group *pg;
+	struct rte_crypto_op *cop[RTE_DIM(trf->ipsec.pkts)];
+	struct rte_ipsec_group grp[RTE_DIM(trf->ipsec.pkts)];
+
+	trf->ip4.num = 0;
+	trf->ip6.num = 0;
+
+	out = &trf->ipsec;
+
+	/* dequeue completed crypto-ops */
+	n = cqp_dequeue(ctx, cop, RTE_DIM(cop));
+	if (n == 0)
+		return;
+
+	/* group them by ipsec session */
+	ng = rte_ipsec_pkt_crypto_group((const struct rte_crypto_op **)
+		(uintptr_t)cop, out->pkts, grp, n);
+
+	/* process each group of packets */
+	for (i = 0; i != ng; i++) {
+
+		pg = grp + i;
+		ss = pg->id.ptr;
+		satp = rte_ipsec_sa_type(ss->sa);
+
+		k = rte_ipsec_pkt_process(ss, pg->m, pg->cnt);
+		copy_to_trf(trf, satp, pg->m, k);
+
+		/* free bad packets, if any */
+		free_pkts(pg->m + k, pg->cnt - k);
+
+		n -= pg->cnt;
+	}
+
+	/* we should never have packet with unknown SA here */
+	RTE_VERIFY(n == 0);
+}
diff --git a/examples/ipsec-secgw/meson.build b/examples/ipsec-secgw/meson.build
index 31f68fee2..81c146ebc 100644
--- a/examples/ipsec-secgw/meson.build
+++ b/examples/ipsec-secgw/meson.build
@@ -9,6 +9,6 @@
 deps += ['security', 'lpm', 'acl', 'hash', 'ipsec']
 allow_experimental_apis = true
 sources = files(
-	'esp.c', 'ipsec.c', 'ipsec-secgw.c', 'parser.c',
-	'rt.c', 'sa.c', 'sp4.c', 'sp6.c'
+	'esp.c', 'ipsec.c', 'ipsec_process.c', 'ipsec-secgw.c',
+	'parser.c', 'rt.c', 'sa.c', 'sp4.c', 'sp6.c'
 )
-- 
2.17.1

[PATCH 7/7] examples/ipsec-secgw: add scripts for functional test

[Konstantin Ananyev]

The purpose of these scripts is to automate ipsec-secgw functional testing.
The scripts require two machines (SUT and DUT) connected through
at least 2 NICs and running linux (so far tested only on Ubuntu 18.04).
Introduced test-cases for the following scenarios:
- Transport/Tunnel modes
- AES-CBC SHA1
- AES-GCM
- ESN on/off
- legacy/librte_ipsec code path

Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
 examples/ipsec-secgw/test/common_defs.sh      | 113 ++++++++++++++++++
 examples/ipsec-secgw/test/data_rxtx.sh        |  70 +++++++++++
 examples/ipsec-secgw/test/linux_test1.sh      |  62 ++++++++++
 examples/ipsec-secgw/test/run_test.sh         |  48 ++++++++
 .../test/trs_aescbc_sha1_common_defs.sh       |  42 +++++++
 .../ipsec-secgw/test/trs_aescbc_sha1_defs.sh  |  38 ++++++
 .../test/trs_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/trs_aescbc_sha1_esn_defs.sh          |  36 ++++++
 .../test/trs_aescbc_sha1_old_defs.sh          |   5 +
 .../test/trs_aesgcm_common_defs.sh            |  38 ++++++
 examples/ipsec-secgw/test/trs_aesgcm_defs.sh  |  37 ++++++
 .../test/trs_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/trs_aesgcm_esn_defs.sh   |  36 ++++++
 .../ipsec-secgw/test/trs_aesgcm_old_defs.sh   |   5 +
 .../test/tun_aescbc_sha1_common_defs.sh       |  42 +++++++
 .../ipsec-secgw/test/tun_aescbc_sha1_defs.sh  |  38 ++++++
 .../test/tun_aescbc_sha1_esn_atom_defs.sh     |   5 +
 .../test/tun_aescbc_sha1_esn_defs.sh          |  38 ++++++
 .../test/tun_aescbc_sha1_old_defs.sh          |   5 +
 .../test/tun_aesgcm_common_defs.sh            |  38 ++++++
 examples/ipsec-secgw/test/tun_aesgcm_defs.sh  |  38 ++++++
 .../test/tun_aesgcm_esn_atom_defs.sh          |   5 +
 .../ipsec-secgw/test/tun_aesgcm_esn_defs.sh   |  38 ++++++
 .../ipsec-secgw/test/tun_aesgcm_old_defs.sh   |   5 +
 24 files changed, 792 insertions(+)
 create mode 100644 examples/ipsec-secgw/test/common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/data_rxtx.sh
 create mode 100644 examples/ipsec-secgw/test/linux_test1.sh
 create mode 100644 examples/ipsec-secgw/test/run_test.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
 create mode 100644 examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh

diff --git a/examples/ipsec-secgw/test/common_defs.sh b/examples/ipsec-secgw/test/common_defs.sh
new file mode 100644
index 000000000..835eca41d
--- /dev/null
+++ b/examples/ipsec-secgw/test/common_defs.sh
@@ -0,0 +1,113 @@
+#! /bin/bash
+
+#check that env vars are properly defined
+
+#check SGW_PATH
+if [[ -z "${SGW_PATH}" || ! -x ${SGW_PATH} ]]; then
+	echo "SGW_PATH is invalid"
+	exit 127
+fi
+
+#check ETH_DEV
+if [[ -z "${ETH_DEV}" ]]; then
+	echo "ETH_DEV is invalid"
+	exit 127
+fi
+
+#setup SGW_LCORE
+SGW_LCORE=${SGW_LCORE:-0}
+
+#check that REMOTE_HOST is reachable
+ssh ${REMOTE_HOST} echo
+st=$?
+if [[ $st -ne 0 ]]; then
+	echo "host ${REMOTE_HOST} is not reachable"
+	exit $st
+fi
+
+#get ether addr of REMOTE_HOST
+REMOTE_MAC=`ssh ${REMOTE_HOST} ip addr show dev ${REMOTE_IFACE}`
+st=$?
+REMOTE_MAC=`echo ${REMOTE_MAC} | sed -e 's/^.*ether //' -e 's/ brd.*$//'`
+if [[ $st -ne 0 || -z "${REMOTE_MAC}" ]]; then
+	echo "coouldn't retrieve ether addr from ${REMOTE_IFACE}"
+	exit 127
+fi
+
+LOCAL_IFACE=dtap0
+
+LOCAL_MAC="00:64:74:61:70:30"
+
+REMOTE_IPV4=192.168.31.14
+LOCAL_IPV4=192.168.31.92
+
+DPDK_PATH=${RTE_SDK:-${PWD}}
+DPDK_BUILD=${RTE_TARGET:-x86_64-native-linuxapp-gcc}
+
+SGW_OUT_FILE=./ipsec-secgw.out1
+
+SGW_CMD_EAL_PRM="--lcores=${SGW_LCORE} -n 4 ${ETH_DEV}"
+SGW_CMD_CFG="(0,0,${SGW_LCORE}),(1,0,${SGW_LCORE})"
+SGW_CMD_PRM="-p 0x3 -u 1 -P --config=\"${SGW_CMD_CFG}\""
+
+SGW_CFG_FILE=$(tempfile)
+
+# configure local host/ifaces
+config_local_iface()
+{
+	ifconfig ${LOCAL_IFACE} ${LOCAL_IPV4}/24 mtu 1440 up
+	ifconfig ${LOCAL_IFACE}
+
+	ip neigh flush dev ${LOCAL_IFACE}
+	arp -i ${LOCAL_IFACE} -s ${REMOTE_IPV4} ${LOCAL_MAC}
+	ip neigh show
+}
+
+#configure remote host/iface
+config_remote_iface()
+{
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE} down
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE} ${REMOTE_IPV4}/24 up
+	ssh ${REMOTE_HOST} ifconfig ${REMOTE_IFACE}
+
+	ssh ${REMOTE_HOST} ip neigh flush dev ${REMOTE_IFACE}
+
+	# by some reason following ip neigh doesn't work for me here properly:
+	#ssh ${REMOTE_HOST} ip neigh add ${LOCAL_IPV4} lladr ${LOCAL_MAC} \
+		#       dev ${REMOTE_IFACE}
+	# so used arp instead.
+	ssh ${REMOTE_HOST} arp -i ${REMOTE_IFACE} -s ${LOCAL_IPV4} ${LOCAL_MAC}
+	ssh ${REMOTE_HOST} ip neigh show
+}
+
+config_iface()
+{
+	config_local_iface
+	config_remote_iface
+}
+
+#start ipsec-secgw
+secgw_start()
+{
+	SGW_EXEC_FILE=$(tempfile)
+	cat <<EOF > ${SGW_EXEC_FILE}
+${SGW_PATH} ${SGW_CMD_EAL_PRM} ${CRYPTO_DEV} \
+--vdev="net_tap0,mac=fixed" \
+-- ${SGW_CMD_PRM} ${SGW_CMD_XPRM} -f ${SGW_CFG_FILE} > \
+${SGW_OUT_FILE} 2>&1 &
+p=\$!
+echo \$p
+EOF
+
+	cat ${SGW_EXEC_FILE}
+	SGW_PID=`/bin/bash -x ${SGW_EXEC_FILE}`
+	sleep 1
+}
+
+#stop ipsec-secgw and cleanup
+secgw_stop()
+{
+	kill ${SGW_PID}
+	rm -f ${SGW_EXEC_FILE}
+	rm -f ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/data_rxtx.sh b/examples/ipsec-secgw/test/data_rxtx.sh
new file mode 100644
index 000000000..f2ff96eff
--- /dev/null
+++ b/examples/ipsec-secgw/test/data_rxtx.sh
@@ -0,0 +1,70 @@
+#! /bin/bash
+
+TCP_PORT=22222
+
+ping_test1()
+{
+	dst=$1
+
+	i=0
+	st=0
+	while [[ $i -ne 1400 && $st -eq 0 ]];
+	do
+		let i++
+		ping -c 1 -s ${i} ${dst}
+		st=$?
+	done
+
+	if [[ $st -ne 0 ]]; then
+		echo "ERROR: $0 failed for dst=${dst}, sz=${i}"
+	fi
+	return $st;
+}
+
+scp_test1()
+{
+	for sz in 1234 23456 345678 4567890 56789102 ; do
+		x=$(tempfile)
+		dd if=/dev/urandom of=${x}.in bs=${sz} count=1
+		scp ${x} ${REMOTE_IPV4}:${x}
+		scp ${REMOTE_IPV4}:${x} ${x}.copy1
+		diff -u ${x} ${x}.copy1
+		st=$?
+		rm -f ${x} ${x}.out
+		ssh ${REMOTE_HOST} rm -f ${x}
+		if [[ $st -ne 0 ]]; then
+			return $st
+		fi
+	done
+
+	return 0;
+}
+
+tcp_test1()
+{
+	for sz in 1234 23456 345678 4567890 56789102 ; do
+		x=`basename $0`.${sz}
+		rcmd=$(tempfile)
+		lcmd=$(tempfile)
+		dd if=/dev/urandom of=${x} bs=${sz} count=1
+		echo "nc -l ${REMOTE_IPV4} ${TCP_PORT} > ${x}; \
+			nc -q 0 ${LOCAL_IPV4} ${TCP_PORT} < ${x}; \
+			rm -f ${x}" > ${rcmd}
+		echo "nc -l ${LOCAL_IPV4} ${TCP_PORT} > ${x}.copy" > ${lcmd}
+		scp ${rcmd} ${REMOTE_HOST}:${rcmd}
+		ssh ${REMOTE_HOST} /bin/bash -x ${rcmd} &
+		/bin/bash -x ${lcmd} &
+		sleep 1
+		nc -q 0 ${REMOTE_IPV4} ${TCP_PORT} < ${x}
+		wait
+		diff -u ${x} ${x}.copy
+		st=$?
+		rm -f ${x} ${x}.copy ${rcmd} ${lcmd}
+		ssh  ${REMOTE_HOST} rm -f ${rcmd}
+		if [[ $st -ne 0 ]]; then
+			return $st
+		fi
+	done
+
+	return $st;
+}
diff --git a/examples/ipsec-secgw/test/linux_test1.sh b/examples/ipsec-secgw/test/linux_test1.sh
new file mode 100644
index 000000000..8265ba6ea
--- /dev/null
+++ b/examples/ipsec-secgw/test/linux_test1.sh
@@ -0,0 +1,62 @@
+#! /bin/bash
+
+# usage:  /bin/bash linux_test1.sh <ipsec_mode>
+# for list of available modes please refer to run_test.sh.
+#
+# Note that for most of them you required appropriate crypto PMD/device
+# to be avaialble.
+# Also user has to setup properly the following environment variables:
+#  SGW_PATH - path to the ipsec-secgw binary to test
+#  REMOTE_HOST - ip/hostname of the DUT
+#  REMOTE_IFACE - iface name for the test-port on DUT
+#  ETH_DEV - ethernet device to be used on SUT by DPDK ('-w <pci-id>')
+# Also user can optonally setup:
+#  SGW_LCORE - lcore to run ipsec-secgw on (default value is 0)
+#  CRYPTO_DEV - crypto device to be used ('-w <pci-id>')
+#  if none specified appropriate vdevs will be created by the scrit
+#
+# The purpose of the script is to automate ipsec-secgw testing
+# using another system running linux as a DUT.
+# It expects that SUT and DUT are connected through at least 2 NICs.
+# One NIC is expected to be managed by linux both machines,
+# and will be used as a control path
+# (make sure user from SUT can ssh to DUT without entering password).
+# Second NIC (test-port) should be reserved for DPDK on SUT,
+# and should be managed by linux on DUT.
+# The script starts ipsec-secgw with 2 NIC devices: test-port and tap vdev.
+# Then configures local tap iface and remote iface and ipsec policies
+# in the following way:
+# traffic going over test-port in both directions has to be
+# protected by ipsec.
+# raffic going over TAP in both directions doesn't have to be protected.
+# I.E:
+# DUT OS(NIC1)--(ipsec)-->(NIC1)ipsec-secgw(TAP)--(plain)-->(TAP)SUT OS
+# SUT OS(TAP)--(plain)-->(TAP)psec-secgw(NIC1)--(ipsec)-->(NIC1)DUT OS
+# Then tries to perorm some data transfer using the scheme decribed above.
+#
+
+DIR=`dirname $0`
+MODE=$1
+
+ . ${DIR}/common_defs.sh
+ . ${DIR}/${MODE}_defs.sh
+
+config_secgw
+
+secgw_start
+
+config_iface
+
+config_remote_xfrm
+
+ . ${DIR}/data_rxtx.sh
+
+ping_test1 ${REMOTE_IPV4}
+st=$?
+if [[ $st -eq 0 ]]; then
+	tcp_test1
+	st=$?
+fi
+
+secgw_stop
+exit $st
diff --git a/examples/ipsec-secgw/test/run_test.sh b/examples/ipsec-secgw/test/run_test.sh
new file mode 100644
index 000000000..7d0298819
--- /dev/null
+++ b/examples/ipsec-secgw/test/run_test.sh
@@ -0,0 +1,48 @@
+#! /bin/bash
+
+# usage: /bin/bash run_test.sh
+# Run all defined linux_test1 test-cases one by one
+# user has to setup properly the following environment variables:
+#  SGW_PATH - path to the ipsec-secgw binary to test
+#  REMOTE_HOST - ip/hostname of the DUT
+#  REMOTE_IFACE - iface name for the test-port on DUT
+#  ETH_DEV - ethernet device to be used on SUT by DPDK ('-w <pci-id>')
+# Also user can optonally setup:
+#  SGW_LCORE - lcore to run ipsec-secgw on (default value is 0)
+#  CRYPTO_DEV - crypto device to be used ('-w <pci-id>')
+#  if none specified appropriate vdevs will be created by the scrit
+# refer to linux_test1.sh for more information
+
+# All supported modes to test.
+# naming convention:
+# 'old' means that ipsec-secgw will run in legacy (non-librte_ipsec mode)
+# 'tun/trs' refer to tunnel/transport mode respectively
+LINUX_TEST1="tun_aescbc_sha1 \
+tun_aescbc_sha1_esn \
+tun_aescbc_sha1_esn_atom \
+tun_aesgcm \
+tun_aesgcm_esn \
+tun_aesgcm_esn_atom \
+trs_aescbc_sha1 \
+trs_aescbc_sha1_esn \
+trs_aescbc_sha1_esn_atom \
+trs_aesgcm \
+trs_aesgcm_esn \
+trs_aesgcm_esn_atom \
+tun_aescbc_sha1_old \
+tun_aesgcm_old \
+trs_aescbc_sha1_old \
+trs_aesgcm_old"
+
+DIR=`dirname $0`
+
+for i in ${LINUX_TEST1}; do
+	echo "starting test ${i}"
+	/bin/bash ${DIR}/linux_test1.sh ${i}
+	st=$?
+	echo "test ${i} finished with status ${st}"
+	if [[ $st -ne 0 ]]; then
+		echo "ERROR test ${i} FAILED"
+		exit $st
+	fi
+done
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
new file mode 100644
index 000000000..40bea68a0
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_common_defs.sh
@@ -0,0 +1,42 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_mb0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#SA out rules
+sa out 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
new file mode 100644
index 000000000..db4720abc
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_defs.sh
@@ -0,0 +1,38 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
new file mode 100644
index 000000000..f16222e11
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
new file mode 100644
index 000000000..04f335023
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_esn_defs.sh
@@ -0,0 +1,36 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh b/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
new file mode 100644
index 000000000..a3abb6103
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aescbc_sha1_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aescbc_sha1_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
new file mode 100644
index 000000000..239782d96
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_common_defs.sh
@@ -0,0 +1,38 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_gcm0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+sp ipv4 in esp protect 7 pri 2 src ${REMOTE_IPV4}/32 dst ${LOCAL_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src ${LOCAL_IPV4}/32 dst ${REMOTE_IPV4}/32 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#SA out rules
+sa out 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode transport
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_defs.sh
new file mode 100644
index 000000000..bc89522b0
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_defs.sh
@@ -0,0 +1,37 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
new file mode 100644
index 000000000..80d8d63b8
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
new file mode 100644
index 000000000..c0f3021e9
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_esn_defs.sh
@@ -0,0 +1,36 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl proto esp mode transport reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl proto esp mode transport reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode transport replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh b/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
new file mode 100644
index 000000000..951e6b68f
--- /dev/null
+++ b/examples/ipsec-secgw/test/trs_aesgcm_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/trs_aesgcm_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
new file mode 100644
index 000000000..630808c21
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_common_defs.sh
@@ -0,0 +1,42 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_mb0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+sp ipv4 in esp protect 7 pri 2 src 192.168.0.0/16 dst 192.168.0.0/16 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src 192.168.0.0/16 dst 192.168.0.0/16 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${REMOTE_IPV4} dst ${LOCAL_IPV4}
+
+#SA out rules
+sa out 7 cipher_algo aes-128-cbc \
+cipher_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+auth_algo sha1-hmac \
+auth_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${LOCAL_IPV4} dst ${REMOTE_IPV4}
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
new file mode 100644
index 000000000..2f6ffaeeb
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_defs.sh
@@ -0,0 +1,38 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
new file mode 100644
index 000000000..6b4a82149
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
new file mode 100644
index 000000000..0ca7684bc
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_esn_defs.sh
@@ -0,0 +1,38 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 flag esn \
+auth sha1 0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef \
+enc aes 0xdeadbeefdeadbeefdeadbeefdeadbeef
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh b/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
new file mode 100644
index 000000000..3c0d8d1b1
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aescbc_sha1_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aescbc_sha1_defs.sh
+
+SGW_CMD_XPRM=
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
new file mode 100644
index 000000000..bebd38145
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_common_defs.sh
@@ -0,0 +1,38 @@
+#! /bin/bash
+
+CRYPTO_DEV=${CRYPTO_DEV:-'--vdev="crypto_aesni_gcm0"'}
+
+#generate cfg file for ipsec-secgw
+config_secgw()
+{
+	cat <<EOF > ${SGW_CFG_FILE}
+sp ipv4 in esp protect 7 pri 2 src 192.168.0.0/16 dst 192.168.0.0/16 \
+sport 0:65535 dport 0:65535
+sp ipv4 in esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SP out IPv4 rules
+sp ipv4 out esp protect 7 pri 2 src 192.168.0.0/16 dst 192.168.0.0/16 \
+sport 0:65535 dport 0:65535
+sp ipv4 out esp bypass pri 1 sport 0:65535 dport 0:65535
+
+#SA in rules
+sa in 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${REMOTE_IPV4} dst ${LOCAL_IPV4}
+
+#SA out rules
+sa out 7 aead_algo aes-128-gcm \
+aead_key de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef:de:ad:be:ef \
+mode ipv4-tunnel src ${LOCAL_IPV4} dst ${REMOTE_IPV4}
+
+#Routing rules
+rt ipv4 dst ${REMOTE_IPV4}/32 port 0
+rt ipv4 dst ${LOCAL_IPV4}/32 port 1
+
+#neighbours
+neigh port 0 ${REMOTE_MAC}
+neigh port 1 ${LOCAL_MAC}
+EOF
+
+	cat ${SGW_CFG_FILE}
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_defs.sh
new file mode 100644
index 000000000..d0bb9afae
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_defs.sh
@@ -0,0 +1,38 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
new file mode 100644
index 000000000..dab1460c8
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_esn_atom_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_esn_defs.sh
+
+SGW_CMD_XPRM='-e -a -w 300'
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
new file mode 100644
index 000000000..814d5f016
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_esn_defs.sh
@@ -0,0 +1,38 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_common_defs.sh
+
+SGW_CMD_XPRM='-e -w 300'
+
+config_remote_xfrm()
+{
+	ssh ${REMOTE_HOST} ip xfrm policy flush
+	ssh ${REMOTE_HOST} ip xfrm state flush
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+dir out ptype main action allow \
+tmpl src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp mode tunnel reqid 1
+
+	ssh ${REMOTE_HOST} ip xfrm policy add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+dir in ptype main action allow \
+tmpl src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp mode tunnel reqid 2
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${REMOTE_IPV4} dst ${LOCAL_IPV4} \
+proto esp spi 7 reqid 1 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm state add \
+src ${LOCAL_IPV4} dst ${REMOTE_IPV4} \
+proto esp spi 7 reqid 2 mode tunnel replay-window 64 flag esn \
+aead "rfc4106\(gcm\(aes\)\)" \
+0xdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 128
+
+	ssh ${REMOTE_HOST} ip xfrm policy list
+	ssh ${REMOTE_HOST} ip xfrm state list
+}
diff --git a/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh b/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh
new file mode 100644
index 000000000..e0a015e21
--- /dev/null
+++ b/examples/ipsec-secgw/test/tun_aesgcm_old_defs.sh
@@ -0,0 +1,5 @@
+#! /bin/bash
+
+. ${DIR}/tun_aesgcm_defs.sh
+
+SGW_CMD_XPRM=
-- 
2.17.1