[PATCH 00/11] selftests: Improve nettest and net/fcnal-test.sh

[PATCH 00/11] selftests: Improve nettest and net/fcnal-test.sh

[Leonard Crestez]

This is a series of improvements to nettest and fcnal-test.sh from
tools/testing/selftests/net which make tests run faster. For me this
reduces the runtime from ~40minutes to ~5minutes and makes the tcp
tests very fast.

Some of the early commits are outright bugfixes.

The tests DO NOT pass perfectly for me on latest net-next/master but I
verified that the failures also happen without my changes. Here is the
list of failures:

TEST: ping local, VRF bind - VRF IP                                           [FAIL]
TEST: Raw socket bind to local address - VRF IP                               [FAIL]
TEST: ping out, VRF bind - ns-B IPv6 LLA                                      [FAIL]
TEST: ping out, VRF bind - multicast IP                                       [FAIL]
TEST: TCP socket bind to out of scope local address - ns-A loopback IPv6      [FAIL]
TEST: TCP socket bind to VRF address with device bind - VRF IPv6              [FAIL]

Three of those were not tested by default before my changes, only with
explicit -t 'bind bind6'

This is related to my work on TCP-AO but there are no patch
dependencies
Link: https://lore.kernel.org/netdev/cover.1632240523.git.cdleonard@gmail.com/

Leonard Crestez (11):
  selftests: net/fcnal: Fix {ipv4,ipv6}_bind not run by default
  selftests: net/fcnal: Mark unknown -t or TESTS value as error
  selftests: net/fcnal: Non-zero exit on failures
  selftests: net/fcnal: Use accept_dad=0 to avoid setup sleep
  selftests: net/fcnal: kill_procs via spin instead of sleep
  selftests: net/fcnal: Do not capture do_run_cmd in verbose mode
  selftests: nettest: Implement -k to fork after bind or listen
  selftests: net/fcnal: Replace sleep after server start with -k
  selftests: nettest: Convert timeout to miliseconds
  selftests: nettest: Add NETTEST_CLIENT,SERVER}_TIMEOUT envvars
  selftests: net/fcnal: Reduce client timeout

 tools/testing/selftests/net/fcnal-test.sh | 710 ++++++++--------------
 tools/testing/selftests/net/nettest.c     | 134 +++-
 2 files changed, 378 insertions(+), 466 deletions(-)


base-commit: 0693b27644f04852e46f7f034e3143992b658869
-- 
2.25.1

[PATCH 01/11] selftests: net/fcnal: Fix {ipv4,ipv6}_bind not run by default

[Leonard Crestez]

Inside the TESTS_IPV{4,6} array these are listed as ipv{4,6}_addr_bind
but test running loop only checks for ipv{4,6}_bind and bind{,6} and
ignores everything else.

As a consequence these tests were not run by default, only with an
explicit -t.

Fix inside TESTS_IPV{4,6}.

Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
---
 tools/testing/selftests/net/fcnal-test.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/net/fcnal-test.sh b/tools/testing/selftests/net/fcnal-test.sh
index 13350cd5c8ac..2839bed91afa 100755
--- a/tools/testing/selftests/net/fcnal-test.sh
+++ b/tools/testing/selftests/net/fcnal-test.sh
@@ -3937,12 +3937,12 @@ EOF
 }
 
 ################################################################################
 # main
 
-TESTS_IPV4="ipv4_ping ipv4_tcp ipv4_udp ipv4_addr_bind ipv4_runtime ipv4_netfilter"
-TESTS_IPV6="ipv6_ping ipv6_tcp ipv6_udp ipv6_addr_bind ipv6_runtime ipv6_netfilter"
+TESTS_IPV4="ipv4_ping ipv4_tcp ipv4_udp ipv4_bind ipv4_runtime ipv4_netfilter"
+TESTS_IPV6="ipv6_ping ipv6_tcp ipv6_udp ipv6_bind ipv6_runtime ipv6_netfilter"
 TESTS_OTHER="use_cases"
 
 PAUSE_ON_FAIL=no
 PAUSE=no
 
-- 
2.25.1

[PATCH 02/11] selftests: net/fcnal: Mark unknown -t or TESTS value as error

[Leonard Crestez]

Right now unknown values are completely ignored which is very confusing
and lead to a subset of tests being skipped because of a mispelling.

Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
---
 tools/testing/selftests/net/fcnal-test.sh | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tools/testing/selftests/net/fcnal-test.sh b/tools/testing/selftests/net/fcnal-test.sh
index 2839bed91afa..43ea5c878a85 100755
--- a/tools/testing/selftests/net/fcnal-test.sh
+++ b/tools/testing/selftests/net/fcnal-test.sh
@@ -4005,10 +4005,13 @@ do
 	# setup namespaces and config, but do not run any tests
 	setup)		 setup; exit 0;;
 	vrf_setup)	 setup "yes"; exit 0;;
 
 	help)            echo "Test names: $TESTS"; exit 0;;
+	*)
+		echo "Unknown test '$t'"
+		exit 1
 	esac
 done
 
 cleanup 2>/dev/null
 
-- 
2.25.1

[PATCH 03/11] selftests: net/fcnal: Non-zero exit on failures

[Leonard Crestez]

Test scripts must report 'pass' or 'fail' via exit status, not just
logging.

Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
---
 tools/testing/selftests/net/fcnal-test.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/testing/selftests/net/fcnal-test.sh b/tools/testing/selftests/net/fcnal-test.sh
index 43ea5c878a85..9cf05e6e0d9b 100755
--- a/tools/testing/selftests/net/fcnal-test.sh
+++ b/tools/testing/selftests/net/fcnal-test.sh
@@ -4015,5 +4015,6 @@ done
 
 cleanup 2>/dev/null
 
 printf "\nTests passed: %3d\n" ${nsuccess}
 printf "Tests failed: %3d\n"   ${nfail}
+exit $(( nfail != 0 ))
-- 
2.25.1

[PATCH 04/11] selftests: net/fcnal: Use accept_dad=0 to avoid setup sleep

[Leonard Crestez]

Duplicate Address Detection makes ipv6 addresses unavailable for a short
period after adding (average about 1 second). Adding sleep statements
avoid this but since all addresses in the test environment are
controlled from the same source we can just disable DAD for the entire
namespace.

Unlike sprinkling nodad to all ipv6 address additions this also skips
DAD for link-local-addresses.

Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
---
 tools/testing/selftests/net/fcnal-test.sh | 17 +++++++----------
 1 file changed, 7 insertions(+), 10 deletions(-)

diff --git a/tools/testing/selftests/net/fcnal-test.sh b/tools/testing/selftests/net/fcnal-test.sh
index 9cf05e6e0d9b..0bd60cd3bc06 100755
--- a/tools/testing/selftests/net/fcnal-test.sh
+++ b/tools/testing/selftests/net/fcnal-test.sh
@@ -357,11 +357,11 @@ create_vrf()
 	ip -netns ${ns} link set ${vrf} up
 	ip -netns ${ns} route add vrf ${vrf} unreachable default metric 8192
 	ip -netns ${ns} -6 route add vrf ${vrf} unreachable default metric 8192
 
 	ip -netns ${ns} addr add 127.0.0.1/8 dev ${vrf}
-	ip -netns ${ns} -6 addr add ::1 dev ${vrf} nodad
+	ip -netns ${ns} -6 addr add ::1 dev ${vrf}
 	if [ "${addr}" != "-" ]; then
 		ip -netns ${ns} addr add dev ${vrf} ${addr}
 	fi
 	if [ "${addr6}" != "-" ]; then
 		ip -netns ${ns} -6 addr add dev ${vrf} ${addr6}
@@ -378,10 +378,11 @@ create_ns()
 	local ns=$1
 	local addr=$2
 	local addr6=$3
 
 	ip netns add ${ns}
+	ip netns exec ${ns} sysctl -wq net.ipv6.conf.{all,default}.accept_dad=0
 
 	ip -netns ${ns} link set lo up
 	if [ "${addr}" != "-" ]; then
 		ip -netns ${ns} addr add dev lo ${addr}
 	fi
@@ -490,12 +491,10 @@ setup()
 	# tell ns-B how to get to remote addresses of ns-A
 	ip -netns ${NSB} ro add ${NSA_LO_IP}/32 via ${NSA_IP} dev ${NSB_DEV}
 	ip -netns ${NSB} ro add ${NSA_LO_IP6}/128 via ${NSA_IP6} dev ${NSB_DEV}
 
 	set +e
-
-	sleep 1
 }
 
 setup_lla_only()
 {
 	# make sure we are starting with a clean slate
@@ -520,12 +519,10 @@ setup_lla_only()
 	create_vrf ${NSA} ${VRF} ${VRF_TABLE} "-" "-"
 	ip -netns ${NSA} link set dev ${NSA_DEV} vrf ${VRF}
 	ip -netns ${NSA} link set dev ${NSA_DEV2} vrf ${VRF}
 
 	set +e
-
-	sleep 1
 }
 
 ################################################################################
 # IPv4
 
@@ -3014,11 +3011,11 @@ ipv6_udp_novrf()
 	sleep 1
 	run_cmd_nsb nettest -6 -D -r ${NSA_IP6}
 	log_test $? 0 "UDP in - LLA to GUA"
 
 	run_cmd_nsb ip -6 ro del ${NSA_IP6}/128 dev ${NSB_DEV}
-	run_cmd_nsb ip -6 addr add ${NSB_IP6}/64 dev ${NSB_DEV} nodad
+	run_cmd_nsb ip -6 addr add ${NSB_IP6}/64 dev ${NSB_DEV}
 }
 
 ipv6_udp_vrf()
 {
 	local a
@@ -3292,11 +3289,11 @@ ipv6_udp_vrf()
 	sleep 1
 	run_cmd_nsb nettest -6 -D -r ${NSA_IP6}
 	log_test $? 0 "UDP in - LLA to GUA"
 
 	run_cmd_nsb ip -6 ro del ${NSA_IP6}/128 dev ${NSB_DEV}
-	run_cmd_nsb ip -6 addr add ${NSB_IP6}/64 dev ${NSB_DEV} nodad
+	run_cmd_nsb ip -6 addr add ${NSB_IP6}/64 dev ${NSB_DEV}
 }
 
 ipv6_udp()
 {
         # should not matter, but set to known state
@@ -3742,11 +3739,11 @@ use_case_br()
 	setup_cmd ip addr del dev ${NSA_DEV} ${NSA_IP}/24
 	setup_cmd ip -6 addr del dev ${NSA_DEV} ${NSA_IP6}/64
 
 	setup_cmd ip link add br0 type bridge
 	setup_cmd ip addr add dev br0 ${NSA_IP}/24
-	setup_cmd ip -6 addr add dev br0 ${NSA_IP6}/64 nodad
+	setup_cmd ip -6 addr add dev br0 ${NSA_IP6}/64
 
 	setup_cmd ip li set ${NSA_DEV} master br0
 	setup_cmd ip li set ${NSA_DEV} up
 	setup_cmd ip li set br0 up
 	setup_cmd ip li set br0 vrf ${VRF}
@@ -3791,15 +3788,15 @@ use_case_br()
 
 	setup_cmd ip li set br0 nomaster
 	setup_cmd ip li add br0.100 link br0 type vlan id 100
 	setup_cmd ip li set br0.100 vrf ${VRF} up
 	setup_cmd ip    addr add dev br0.100 172.16.101.1/24
-	setup_cmd ip -6 addr add dev br0.100 2001:db8:101::1/64 nodad
+	setup_cmd ip -6 addr add dev br0.100 2001:db8:101::1/64
 
 	setup_cmd_nsb ip li add vlan100 link ${NSB_DEV} type vlan id 100
 	setup_cmd_nsb ip addr add dev vlan100 172.16.101.2/24
-	setup_cmd_nsb ip -6 addr add dev vlan100 2001:db8:101::2/64 nodad
+	setup_cmd_nsb ip -6 addr add dev vlan100 2001:db8:101::2/64
 	setup_cmd_nsb ip li set vlan100 up
 	sleep 1
 
 	rmmod br_netfilter 2>/dev/null
 
-- 
2.25.1

[PATCH 05/11] selftests: net/fcnal: kill_procs via spin instead of sleep

[Leonard Crestez]

Sleeping for one second after a kill is not necessary and adds up quite
quickly. Replace with a fast loop spinning until pidof returns nothing.

Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
---
 tools/testing/selftests/net/fcnal-test.sh | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/net/fcnal-test.sh b/tools/testing/selftests/net/fcnal-test.sh
index 0bd60cd3bc06..b7fda51deb3f 100755
--- a/tools/testing/selftests/net/fcnal-test.sh
+++ b/tools/testing/selftests/net/fcnal-test.sh
@@ -176,12 +176,19 @@ show_hint()
 	fi
 }
 
 kill_procs()
 {
-	killall nettest ping ping6 >/dev/null 2>&1
-	sleep 1
+	local pids
+	while true; do
+		pids=$(pidof nettest ping ping6)
+		if [[ -z $pids ]]; then
+			break
+		fi
+		kill $pids
+		sleep 0.01
+	done
 }
 
 do_run_cmd()
 {
 	local cmd="$*"
-- 
2.25.1

[PATCH 06/11] selftests: net/fcnal: Do not capture do_run_cmd in verbose mode

[Leonard Crestez]

This way running with -v will show interspersed output from both nettest
client and server. This helps to identify the order of events.

It is also required in order to make nettest fork in the background by
itself because shell capturing does not stop if the target forks.

This also fixes SC2166 warning:
Prefer [ p ] && [ q ] as [ p -a q ] is not well defined.

Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
---
 tools/testing/selftests/net/fcnal-test.sh | 16 +++++++---------
 1 file changed, 7 insertions(+), 9 deletions(-)

diff --git a/tools/testing/selftests/net/fcnal-test.sh b/tools/testing/selftests/net/fcnal-test.sh
index b7fda51deb3f..09cb35e16219 100755
--- a/tools/testing/selftests/net/fcnal-test.sh
+++ b/tools/testing/selftests/net/fcnal-test.sh
@@ -189,21 +189,19 @@ kill_procs()
 	done
 }
 
 do_run_cmd()
 {
-	local cmd="$*"
-	local out
+	local rc cmd="$*"
 
-	if [ "$VERBOSE" = "1" ]; then
+	if [[ "$VERBOSE" = "1" ]]; then
 		echo "COMMAND: ${cmd}"
-	fi
-
-	out=$($cmd 2>&1)
-	rc=$?
-	if [ "$VERBOSE" = "1" -a -n "$out" ]; then
-		echo "$out"
+		$cmd
+		rc=$?
+	else
+		$cmd &> /dev/null
+		rc=$?
 	fi
 
 	return $rc
 }
 
-- 
2.25.1

[PATCH 07/11] selftests: nettest: Implement -k to fork after bind or listen

[Leonard Crestez]

The fcnal-test.sh script launches the nettest server in the background
and sleeps for 1 second to ensure that it has time to get up.

Add an option to nettest to fork into the background as soon as packets
are accepted by the kernel, this means bind() for datagrams or listen()
for TCP.

Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
---
 tools/testing/selftests/net/nettest.c | 34 ++++++++++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/net/nettest.c b/tools/testing/selftests/net/nettest.c
index bd6288302094..576d8bb4c94c 100644
--- a/tools/testing/selftests/net/nettest.c
+++ b/tools/testing/selftests/net/nettest.c
@@ -73,10 +73,11 @@ struct sock_args {
 	unsigned int has_local_ip:1,
 		     has_remote_ip:1,
 		     has_grp:1,
 		     has_expected_laddr:1,
 		     has_expected_raddr:1,
+		     fork_background:1,
 		     bind_test_only:1;
 
 	unsigned short port;
 
 	int type;      /* DGRAM, STREAM, RAW */
@@ -1388,10 +1389,29 @@ static int config_xfrm_policy(int sd, struct sock_args *args)
 	}
 
 	return 0;
 }
 
+static void handle_fork_background(struct sock_args *args)
+{
+	pid_t fork_result;
+	int result;
+
+	if (!args->fork_background)
+		return;
+
+	fork_result = fork();
+	if (fork_result)
+		exit(0);
+	result = setpgid(0, 0);
+	if (result) {
+		log_err_errno("Failed setpgid");
+		exit(1);
+	}
+	log_msg("server running in background\n");
+}
+
 static int lsock_init(struct sock_args *args)
 {
 	long flags;
 	int sd;
 
@@ -1422,10 +1442,12 @@ static int lsock_init(struct sock_args *args)
 	if (args->type == SOCK_STREAM && listen(sd, 1) < 0) {
 		log_err_errno("listen failed");
 		goto err;
 	}
 
+	handle_fork_background(args);
+
 	flags = fcntl(sd, F_GETFL);
 	if ((flags < 0) || (fcntl(sd, F_SETFL, flags|O_NONBLOCK) < 0)) {
 		log_err_errno("Failed to set non-blocking option");
 		goto err;
 	}
@@ -1819,11 +1841,11 @@ static int ipc_parent(int cpid, int fd, struct sock_args *args)
 
 	wait(&status);
 	return client_status;
 }
 
-#define GETOPT_STR  "sr:l:c:p:t:g:P:DRn:M:X:m:d:I:BN:O:SCi6xL:0:1:2:3:Fbq"
+#define GETOPT_STR  "sr:l:c:p:t:g:P:DRn:M:X:m:d:I:BN:O:SCi6xL:0:1:2:3:Fbqk"
 
 static void print_usage(char *prog)
 {
 	printf(
 	"usage: %s OPTS\n"
@@ -1866,10 +1888,11 @@ static void print_usage(char *prog)
 	"    -2 dev        Expected device name (or index) to receive packet\n"
 	"    -3 dev        Expected device name (or index) to receive packets - server mode\n"
 	"\n"
 	"    -b            Bind test only.\n"
 	"    -q            Be quiet. Run test without printing anything.\n"
+	"    -k            Fork server in background after bind or listen.\n"
 	, prog, DEFAULT_PORT);
 }
 
 int main(int argc, char *argv[])
 {
@@ -2017,10 +2040,13 @@ int main(int argc, char *argv[])
 			quiet = 1;
 			break;
 		case 'x':
 			args.use_xfrm = 1;
 			break;
+		case 'k':
+			args.fork_background = 1;
+			break;
 		default:
 			print_usage(argv[0]);
 			return 1;
 		}
 	}
@@ -2058,10 +2084,16 @@ int main(int argc, char *argv[])
 		fprintf(stderr,
 			"Local (server mode) or remote IP (client IP) required\n");
 		return 1;
 	}
 
+	if (args.fork_background && (both_mode || !server_mode)) {
+		fprintf(stderr,
+			"Fork after listen only supported for server mode\n");
+		return 1;
+	}
+
 	if (interactive) {
 		prog_timeout = 0;
 		msg = NULL;
 	}
 
-- 
2.25.1

[PATCH 08/11] selftests: net/fcnal: Replace sleep after server start with -k

[Leonard Crestez]

The -k switch makes the server fork into the background after the listen
call is succesful, this can be used to replace most of the `sleep 1`
statements in this script.

Change performed with a vim command:

s/nettest \(.*-s.*\) &\n\s*sleep 1\n/nettest \1 -k\r

Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
---
 tools/testing/selftests/net/fcnal-test.sh | 641 ++++++++--------------
 1 file changed, 219 insertions(+), 422 deletions(-)

diff --git a/tools/testing/selftests/net/fcnal-test.sh b/tools/testing/selftests/net/fcnal-test.sh
index 09cb35e16219..e73aeb3884c5 100755
--- a/tools/testing/selftests/net/fcnal-test.sh
+++ b/tools/testing/selftests/net/fcnal-test.sh
@@ -804,63 +804,56 @@ ipv4_tcp_md5_novrf()
 	# single address
 	#
 
 	# basic use case
 	log_start
-	run_cmd nettest -s -M ${MD5_PW} -m ${NSB_IP} &
-	sleep 1
+	run_cmd nettest -s -M ${MD5_PW} -m ${NSB_IP} -k
 	run_cmd_nsb nettest -r ${NSA_IP} -X ${MD5_PW}
 	log_test $? 0 "MD5: Single address config"
 
 	# client sends MD5, server not configured
 	log_start
 	show_hint "Should timeout due to MD5 mismatch"
-	run_cmd nettest -s &
-	sleep 1
+	run_cmd nettest -s -k
 	run_cmd_nsb nettest -r ${NSA_IP} -X ${MD5_PW}
 	log_test $? 2 "MD5: Server no config, client uses password"
 
 	# wrong password
 	log_start
 	show_hint "Should timeout since client uses wrong password"
-	run_cmd nettest -s -M ${MD5_PW} -m ${NSB_IP} &
-	sleep 1
+	run_cmd nettest -s -M ${MD5_PW} -m ${NSB_IP} -k
 	run_cmd_nsb nettest -r ${NSA_IP} -X ${MD5_WRONG_PW}
 	log_test $? 2 "MD5: Client uses wrong password"
 
 	# client from different address
 	log_start
 	show_hint "Should timeout due to MD5 mismatch"
-	run_cmd nettest -s -M ${MD5_PW} -m ${NSB_LO_IP} &
-	sleep 1
+	run_cmd nettest -s -M ${MD5_PW} -m ${NSB_LO_IP} -k
 	run_cmd_nsb nettest -r ${NSA_IP} -X ${MD5_PW}
 	log_test $? 2 "MD5: Client address does not match address configured with password"
 
 	#
 	# MD5 extension - prefix length
 	#
 
 	# client in prefix
 	log_start
-	run_cmd nettest -s -M ${MD5_PW} -m ${NS_NET} &
-	sleep 1
+	run_cmd nettest -s -M ${MD5_PW} -m ${NS_NET} -k
 	run_cmd_nsb nettest  -r ${NSA_IP} -X ${MD5_PW}
 	log_test $? 0 "MD5: Prefix config"
 
 	# client in prefix, wrong password
 	log_start
 	show_hint "Should timeout since client uses wrong password"
-	run_cmd nettest -s -M ${MD5_PW} -m ${NS_NET} &
-	sleep 1
+	run_cmd nettest -s -M ${MD5_PW} -m ${NS_NET} -k
 	run_cmd_nsb nettest -r ${NSA_IP} -X ${MD5_WRONG_PW}
 	log_test $? 2 "MD5: Prefix config, client uses wrong password"
 
 	# client outside of prefix
 	log_start
 	show_hint "Should timeout due to MD5 mismatch"
-	run_cmd nettest -s -M ${MD5_PW} -m ${NS_NET} &
-	sleep 1
+	run_cmd nettest -s -M ${MD5_PW} -m ${NS_NET} -k
 	run_cmd_nsb nettest -c ${NSB_LO_IP} -r ${NSA_IP} -X ${MD5_PW}
 	log_test $? 2 "MD5: Prefix config, client address not in configured prefix"
 }
 
 #
@@ -872,127 +865,112 @@ ipv4_tcp_md5()
 	# single address
 	#
 
 	# basic use case
 	log_start
-	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP} &
-	sleep 1
+	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP} -k
 	run_cmd_nsb nettest -r ${NSA_IP} -X ${MD5_PW}
 	log_test $? 0 "MD5: VRF: Single address config"
 
 	# client sends MD5, server not configured
 	log_start
 	show_hint "Should timeout since server does not have MD5 auth"
-	run_cmd nettest -s -I ${VRF} &
-	sleep 1
+	run_cmd nettest -s -I ${VRF} -k
 	run_cmd_nsb nettest -r ${NSA_IP} -X ${MD5_PW}
 	log_test $? 2 "MD5: VRF: Server no config, client uses password"
 
 	# wrong password
 	log_start
 	show_hint "Should timeout since client uses wrong password"
-	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP} &
-	sleep 1
+	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP} -k
 	run_cmd_nsb nettest -r ${NSA_IP} -X ${MD5_WRONG_PW}
 	log_test $? 2 "MD5: VRF: Client uses wrong password"
 
 	# client from different address
 	log_start
 	show_hint "Should timeout since server config differs from client"
-	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NSB_LO_IP} &
-	sleep 1
+	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NSB_LO_IP} -k
 	run_cmd_nsb nettest -r ${NSA_IP} -X ${MD5_PW}
 	log_test $? 2 "MD5: VRF: Client address does not match address configured with password"
 
 	#
 	# MD5 extension - prefix length
 	#
 
 	# client in prefix
 	log_start
-	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET} &
-	sleep 1
+	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET} -k
 	run_cmd_nsb nettest  -r ${NSA_IP} -X ${MD5_PW}
 	log_test $? 0 "MD5: VRF: Prefix config"
 
 	# client in prefix, wrong password
 	log_start
 	show_hint "Should timeout since client uses wrong password"
-	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET} &
-	sleep 1
+	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET} -k
 	run_cmd_nsb nettest -r ${NSA_IP} -X ${MD5_WRONG_PW}
 	log_test $? 2 "MD5: VRF: Prefix config, client uses wrong password"
 
 	# client outside of prefix
 	log_start
 	show_hint "Should timeout since client address is outside of prefix"
-	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET} &
-	sleep 1
+	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET} -k
 	run_cmd_nsb nettest -c ${NSB_LO_IP} -r ${NSA_IP} -X ${MD5_PW}
 	log_test $? 2 "MD5: VRF: Prefix config, client address not in configured prefix"
 
 	#
 	# duplicate config between default VRF and a VRF
 	#
 
 	log_start
-	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP} &
-	run_cmd nettest -s -M ${MD5_WRONG_PW} -m ${NSB_IP} &
-	sleep 1
+	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP} -k
+	run_cmd nettest -s -M ${MD5_WRONG_PW} -m ${NSB_IP} -k
 	run_cmd_nsb nettest  -r ${NSA_IP} -X ${MD5_PW}
 	log_test $? 0 "MD5: VRF: Single address config in default VRF and VRF, conn in VRF"
 
 	log_start
-	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP} &
-	run_cmd nettest -s -M ${MD5_WRONG_PW} -m ${NSB_IP} &
-	sleep 1
+	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP} -k
+	run_cmd nettest -s -M ${MD5_WRONG_PW} -m ${NSB_IP} -k
 	run_cmd_nsc nettest  -r ${NSA_IP} -X ${MD5_WRONG_PW}
 	log_test $? 0 "MD5: VRF: Single address config in default VRF and VRF, conn in default VRF"
 
 	log_start
 	show_hint "Should timeout since client in default VRF uses VRF password"
-	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP} &
-	run_cmd nettest -s -M ${MD5_WRONG_PW} -m ${NSB_IP} &
-	sleep 1
+	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP} -k
+	run_cmd nettest -s -M ${MD5_WRONG_PW} -m ${NSB_IP} -k
 	run_cmd_nsc nettest -r ${NSA_IP} -X ${MD5_PW}
 	log_test $? 2 "MD5: VRF: Single address config in default VRF and VRF, conn in default VRF with VRF pw"
 
 	log_start
 	show_hint "Should timeout since client in VRF uses default VRF password"
-	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP} &
-	run_cmd nettest -s -M ${MD5_WRONG_PW} -m ${NSB_IP} &
-	sleep 1
+	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP} -k
+	run_cmd nettest -s -M ${MD5_WRONG_PW} -m ${NSB_IP} -k
 	run_cmd_nsb nettest -r ${NSA_IP} -X ${MD5_WRONG_PW}
 	log_test $? 2 "MD5: VRF: Single address config in default VRF and VRF, conn in VRF with default VRF pw"
 
 	log_start
-	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET} &
-	run_cmd nettest -s -M ${MD5_WRONG_PW} -m ${NS_NET} &
-	sleep 1
+	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET} -k
+	run_cmd nettest -s -M ${MD5_WRONG_PW} -m ${NS_NET} -k
 	run_cmd_nsb nettest  -r ${NSA_IP} -X ${MD5_PW}
 	log_test $? 0 "MD5: VRF: Prefix config in default VRF and VRF, conn in VRF"
 
 	log_start
-	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET} &
-	run_cmd nettest -s -M ${MD5_WRONG_PW} -m ${NS_NET} &
-	sleep 1
+	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET} -k
+	run_cmd nettest -s -M ${MD5_WRONG_PW} -m ${NS_NET} -k
 	run_cmd_nsc nettest  -r ${NSA_IP} -X ${MD5_WRONG_PW}
 	log_test $? 0 "MD5: VRF: Prefix config in default VRF and VRF, conn in default VRF"
 
 	log_start
 	show_hint "Should timeout since client in default VRF uses VRF password"
-	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET} &
-	run_cmd nettest -s -M ${MD5_WRONG_PW} -m ${NS_NET} &
-	sleep 1
+	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET} -k
+	run_cmd nettest -s -M ${MD5_WRONG_PW} -m ${NS_NET} -k
 	run_cmd_nsc nettest -r ${NSA_IP} -X ${MD5_PW}
 	log_test $? 2 "MD5: VRF: Prefix config in default VRF and VRF, conn in default VRF with VRF pw"
 
 	log_start
 	show_hint "Should timeout since client in VRF uses default VRF password"
-	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET} &
-	run_cmd nettest -s -M ${MD5_WRONG_PW} -m ${NS_NET} &
-	sleep 1
+	run_cmd nettest -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET} -k
+	run_cmd nettest -s -M ${MD5_WRONG_PW} -m ${NS_NET} -k
 	run_cmd_nsb nettest -r ${NSA_IP} -X ${MD5_WRONG_PW}
 	log_test $? 2 "MD5: VRF: Prefix config in default VRF and VRF, conn in VRF with default VRF pw"
 
 	#
 	# negative tests
@@ -1015,20 +993,18 @@ ipv4_tcp_novrf()
 	# server tests
 	#
 	for a in ${NSA_IP} ${NSA_LO_IP}
 	do
 		log_start
-		run_cmd nettest -s &
-		sleep 1
+		run_cmd nettest -s -k
 		run_cmd_nsb nettest -r ${a}
 		log_test_addr ${a} $? 0 "Global server"
 	done
 
 	a=${NSA_IP}
 	log_start
-	run_cmd nettest -s -I ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -s -I ${NSA_DEV} -k
 	run_cmd_nsb nettest -r ${a}
 	log_test_addr ${a} $? 0 "Device server"
 
 	# verify TCP reset sent and received
 	for a in ${NSA_IP} ${NSA_LO_IP}
@@ -1043,18 +1019,16 @@ ipv4_tcp_novrf()
 	# client
 	#
 	for a in ${NSB_IP} ${NSB_LO_IP}
 	do
 		log_start
-		run_cmd_nsb nettest -s &
-		sleep 1
+		run_cmd_nsb nettest -s -k
 		run_cmd nettest -r ${a} -0 ${NSA_IP}
 		log_test_addr ${a} $? 0 "Client"
 
 		log_start
-		run_cmd_nsb nettest -s &
-		sleep 1
+		run_cmd_nsb nettest -s -k
 		run_cmd nettest -r ${a} -d ${NSA_DEV}
 		log_test_addr ${a} $? 0 "Client, device bind"
 
 		log_start
 		show_hint "Should fail 'Connection refused'"
@@ -1071,54 +1045,48 @@ ipv4_tcp_novrf()
 	# local address tests
 	#
 	for a in ${NSA_IP} ${NSA_LO_IP} 127.0.0.1
 	do
 		log_start
-		run_cmd nettest -s &
-		sleep 1
+		run_cmd nettest -s -k
 		run_cmd nettest -r ${a} -0 ${a} -1 ${a}
 		log_test_addr ${a} $? 0 "Global server, local connection"
 	done
 
 	a=${NSA_IP}
 	log_start
-	run_cmd nettest -s -I ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -s -I ${NSA_DEV} -k
 	run_cmd nettest -r ${a} -0 ${a}
 	log_test_addr ${a} $? 0 "Device server, unbound client, local connection"
 
 	for a in ${NSA_LO_IP} 127.0.0.1
 	do
 		log_start
 		show_hint "Should fail 'Connection refused' since addresses on loopback are out of device scope"
-		run_cmd nettest -s -I ${NSA_DEV} &
-		sleep 1
+		run_cmd nettest -s -I ${NSA_DEV} -k
 		run_cmd nettest -r ${a}
 		log_test_addr ${a} $? 1 "Device server, unbound client, local connection"
 	done
 
 	a=${NSA_IP}
 	log_start
-	run_cmd nettest -s &
-	sleep 1
+	run_cmd nettest -s -k
 	run_cmd nettest -r ${a} -0 ${a} -d ${NSA_DEV}
 	log_test_addr ${a} $? 0 "Global server, device client, local connection"
 
 	for a in ${NSA_LO_IP} 127.0.0.1
 	do
 		log_start
 		show_hint "Should fail 'No route to host' since addresses on loopback are out of device scope"
-		run_cmd nettest -s &
-		sleep 1
+		run_cmd nettest -s -k
 		run_cmd nettest -r ${a} -d ${NSA_DEV}
 		log_test_addr ${a} $? 1 "Global server, device client, local connection"
 	done
 
 	a=${NSA_IP}
 	log_start
-	run_cmd nettest -s -I ${NSA_DEV} -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -s -I ${NSA_DEV} -3 ${NSA_DEV} -k
 	run_cmd nettest  -d ${NSA_DEV} -r ${a} -0 ${a}
 	log_test_addr ${a} $? 0 "Device server, device client, local connection"
 
 	log_start
 	show_hint "Should fail 'Connection refused'"
@@ -1142,24 +1110,21 @@ ipv4_tcp_vrf()
 	#
 	for a in ${NSA_IP} ${VRF_IP}
 	do
 		log_start
 		show_hint "Should fail 'Connection refused' since global server with VRF is disabled"
-		run_cmd nettest -s &
-		sleep 1
+		run_cmd nettest -s -k
 		run_cmd_nsb nettest -r ${a}
 		log_test_addr ${a} $? 1 "Global server"
 
 		log_start
-		run_cmd nettest -s -I ${VRF} -3 ${VRF} &
-		sleep 1
+		run_cmd nettest -s -I ${VRF} -3 ${VRF} -k
 		run_cmd_nsb nettest -r ${a}
 		log_test_addr ${a} $? 0 "VRF server"
 
 		log_start
-		run_cmd nettest -s -I ${NSA_DEV} -3 ${NSA_DEV} &
-		sleep 1
+		run_cmd nettest -s -I ${NSA_DEV} -3 ${NSA_DEV} -k
 		run_cmd_nsb nettest -r ${a}
 		log_test_addr ${a} $? 0 "Device server"
 
 		# verify TCP reset received
 		log_start
@@ -1171,12 +1136,11 @@ ipv4_tcp_vrf()
 	# local address tests
 	# (${VRF_IP} and 127.0.0.1 both timeout)
 	a=${NSA_IP}
 	log_start
 	show_hint "Should fail 'Connection refused' since global server with VRF is disabled"
-	run_cmd nettest -s &
-	sleep 1
+	run_cmd nettest -s -k
 	run_cmd nettest -r ${a} -d ${NSA_DEV}
 	log_test_addr ${a} $? 1 "Global server, local connection"
 
 	# run MD5 tests
 	ipv4_tcp_md5
@@ -1189,19 +1153,17 @@ ipv4_tcp_vrf()
 
 	for a in ${NSA_IP} ${VRF_IP}
 	do
 		log_start
 		show_hint "client socket should be bound to VRF"
-		run_cmd nettest -s -3 ${VRF} &
-		sleep 1
+		run_cmd nettest -s -3 ${VRF} -k
 		run_cmd_nsb nettest -r ${a}
 		log_test_addr ${a} $? 0 "Global server"
 
 		log_start
 		show_hint "client socket should be bound to VRF"
-		run_cmd nettest -s -I ${VRF} -3 ${VRF} &
-		sleep 1
+		run_cmd nettest -s -I ${VRF} -3 ${VRF} -k
 		run_cmd_nsb nettest -r ${a}
 		log_test_addr ${a} $? 0 "VRF server"
 
 		# verify TCP reset received
 		log_start
@@ -1211,40 +1173,36 @@ ipv4_tcp_vrf()
 	done
 
 	a=${NSA_IP}
 	log_start
 	show_hint "client socket should be bound to device"
-	run_cmd nettest -s -I ${NSA_DEV} -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -s -I ${NSA_DEV} -3 ${NSA_DEV} -k
 	run_cmd_nsb nettest -r ${a}
 	log_test_addr ${a} $? 0 "Device server"
 
 	# local address tests
 	for a in ${NSA_IP} ${VRF_IP}
 	do
 		log_start
 		show_hint "Should fail 'Connection refused' since client is not bound to VRF"
-		run_cmd nettest -s -I ${VRF} &
-		sleep 1
+		run_cmd nettest -s -I ${VRF} -k
 		run_cmd nettest -r ${a}
 		log_test_addr ${a} $? 1 "Global server, local connection"
 	done
 
 	#
 	# client
 	#
 	for a in ${NSB_IP} ${NSB_LO_IP}
 	do
 		log_start
-		run_cmd_nsb nettest -s &
-		sleep 1
+		run_cmd_nsb nettest -s -k
 		run_cmd nettest -r ${a} -d ${VRF}
 		log_test_addr ${a} $? 0 "Client, VRF bind"
 
 		log_start
-		run_cmd_nsb nettest -s &
-		sleep 1
+		run_cmd_nsb nettest -s -k
 		run_cmd nettest -r ${a} -d ${NSA_DEV}
 		log_test_addr ${a} $? 0 "Client, device bind"
 
 		log_start
 		show_hint "Should fail 'Connection refused'"
@@ -1258,39 +1216,34 @@ ipv4_tcp_vrf()
 	done
 
 	for a in ${NSA_IP} ${VRF_IP} 127.0.0.1
 	do
 		log_start
-		run_cmd nettest -s -I ${VRF} -3 ${VRF} &
-		sleep 1
+		run_cmd nettest -s -I ${VRF} -3 ${VRF} -k
 		run_cmd nettest -r ${a} -d ${VRF} -0 ${a}
 		log_test_addr ${a} $? 0 "VRF server, VRF client, local connection"
 	done
 
 	a=${NSA_IP}
 	log_start
-	run_cmd nettest -s -I ${VRF} -3 ${VRF} &
-	sleep 1
+	run_cmd nettest -s -I ${VRF} -3 ${VRF} -k
 	run_cmd nettest -r ${a} -d ${NSA_DEV} -0 ${a}
 	log_test_addr ${a} $? 0 "VRF server, device client, local connection"
 
 	log_start
 	show_hint "Should fail 'No route to host' since client is out of VRF scope"
-	run_cmd nettest -s -I ${VRF} &
-	sleep 1
+	run_cmd nettest -s -I ${VRF} -k
 	run_cmd nettest -r ${a}
 	log_test_addr ${a} $? 1 "VRF server, unbound client, local connection"
 
 	log_start
-	run_cmd nettest -s -I ${NSA_DEV} -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -s -I ${NSA_DEV} -3 ${NSA_DEV} -k
 	run_cmd nettest -r ${a} -d ${VRF} -0 ${a}
 	log_test_addr ${a} $? 0 "Device server, VRF client, local connection"
 
 	log_start
-	run_cmd nettest -s -I ${NSA_DEV} -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -s -I ${NSA_DEV} -3 ${NSA_DEV} -k
 	run_cmd nettest -r ${a} -d ${NSA_DEV} -0 ${a}
 	log_test_addr ${a} $? 0 "Device server, device client, local connection"
 }
 
 ipv4_tcp()
@@ -1324,12 +1277,11 @@ ipv4_udp_novrf()
 	# server tests
 	#
 	for a in ${NSA_IP} ${NSA_LO_IP}
 	do
 		log_start
-		run_cmd nettest -D -s -3 ${NSA_DEV} &
-		sleep 1
+		run_cmd nettest -D -s -3 ${NSA_DEV} -k
 		run_cmd_nsb nettest -D -r ${a}
 		log_test_addr ${a} $? 0 "Global server"
 
 		log_start
 		show_hint "Should fail 'Connection refused' since there is no server"
@@ -1337,41 +1289,36 @@ ipv4_udp_novrf()
 		log_test_addr ${a} $? 1 "No server"
 	done
 
 	a=${NSA_IP}
 	log_start
-	run_cmd nettest -D -I ${NSA_DEV} -s -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -D -I ${NSA_DEV} -s -3 ${NSA_DEV} -k
 	run_cmd_nsb nettest -D -r ${a}
 	log_test_addr ${a} $? 0 "Device server"
 
 	#
 	# client
 	#
 	for a in ${NSB_IP} ${NSB_LO_IP}
 	do
 		log_start
-		run_cmd_nsb nettest -D -s &
-		sleep 1
+		run_cmd_nsb nettest -D -s -k
 		run_cmd nettest -D -r ${a} -0 ${NSA_IP}
 		log_test_addr ${a} $? 0 "Client"
 
 		log_start
-		run_cmd_nsb nettest -D -s &
-		sleep 1
+		run_cmd_nsb nettest -D -s -k
 		run_cmd nettest -D -r ${a} -d ${NSA_DEV} -0 ${NSA_IP}
 		log_test_addr ${a} $? 0 "Client, device bind"
 
 		log_start
-		run_cmd_nsb nettest -D -s &
-		sleep 1
+		run_cmd_nsb nettest -D -s -k
 		run_cmd nettest -D -r ${a} -d ${NSA_DEV} -C -0 ${NSA_IP}
 		log_test_addr ${a} $? 0 "Client, device send via cmsg"
 
 		log_start
-		run_cmd_nsb nettest -D -s &
-		sleep 1
+		run_cmd_nsb nettest -D -s -k
 		run_cmd nettest -D -r ${a} -d ${NSA_DEV} -S -0 ${NSA_IP}
 		log_test_addr ${a} $? 0 "Client, device bind via IP_UNICAST_IF"
 
 		log_start
 		show_hint "Should fail 'Connection refused'"
@@ -1388,83 +1335,73 @@ ipv4_udp_novrf()
 	# local address tests
 	#
 	for a in ${NSA_IP} ${NSA_LO_IP} 127.0.0.1
 	do
 		log_start
-		run_cmd nettest -D -s &
-		sleep 1
+		run_cmd nettest -D -s -k
 		run_cmd nettest -D -r ${a} -0 ${a} -1 ${a}
 		log_test_addr ${a} $? 0 "Global server, local connection"
 	done
 
 	a=${NSA_IP}
 	log_start
-	run_cmd nettest -s -D -I ${NSA_DEV} -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -s -D -I ${NSA_DEV} -3 ${NSA_DEV} -k
 	run_cmd nettest -D -r ${a}
 	log_test_addr ${a} $? 0 "Device server, unbound client, local connection"
 
 	for a in ${NSA_LO_IP} 127.0.0.1
 	do
 		log_start
 		show_hint "Should fail 'Connection refused' since address is out of device scope"
-		run_cmd nettest -s -D -I ${NSA_DEV} &
-		sleep 1
+		run_cmd nettest -s -D -I ${NSA_DEV} -k
 		run_cmd nettest -D -r ${a}
 		log_test_addr ${a} $? 1 "Device server, unbound client, local connection"
 	done
 
 	a=${NSA_IP}
 	log_start
-	run_cmd nettest -s -D &
-	sleep 1
+	run_cmd nettest -s -D -k
 	run_cmd nettest -D -d ${NSA_DEV} -r ${a}
 	log_test_addr ${a} $? 0 "Global server, device client, local connection"
 
 	log_start
-	run_cmd nettest -s -D &
-	sleep 1
+	run_cmd nettest -s -D -k
 	run_cmd nettest -D -d ${NSA_DEV} -C -r ${a}
 	log_test_addr ${a} $? 0 "Global server, device send via cmsg, local connection"
 
 	log_start
-	run_cmd nettest -s -D &
-	sleep 1
+	run_cmd nettest -s -D -k
 	run_cmd nettest -D -d ${NSA_DEV} -S -r ${a}
 	log_test_addr ${a} $? 0 "Global server, device client via IP_UNICAST_IF, local connection"
 
 	# IPv4 with device bind has really weird behavior - it overrides the
 	# fib lookup, generates an rtable and tries to send the packet. This
 	# causes failures for local traffic at different places
 	for a in ${NSA_LO_IP} 127.0.0.1
 	do
 		log_start
 		show_hint "Should fail since addresses on loopback are out of device scope"
-		run_cmd nettest -D -s &
-		sleep 1
+		run_cmd nettest -D -s -k
 		run_cmd nettest -D -r ${a} -d ${NSA_DEV}
 		log_test_addr ${a} $? 2 "Global server, device client, local connection"
 
 		log_start
 		show_hint "Should fail since addresses on loopback are out of device scope"
-		run_cmd nettest -D -s &
-		sleep 1
+		run_cmd nettest -D -s -k
 		run_cmd nettest -D -r ${a} -d ${NSA_DEV} -C
 		log_test_addr ${a} $? 1 "Global server, device send via cmsg, local connection"
 
 		log_start
 		show_hint "Should fail since addresses on loopback are out of device scope"
-		run_cmd nettest -D -s &
-		sleep 1
+		run_cmd nettest -D -s -k
 		run_cmd nettest -D -r ${a} -d ${NSA_DEV} -S
 		log_test_addr ${a} $? 1 "Global server, device client via IP_UNICAST_IF, local connection"
 	done
 
 	a=${NSA_IP}
 	log_start
-	run_cmd nettest -D -s -I ${NSA_DEV} -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -D -s -I ${NSA_DEV} -3 ${NSA_DEV} -k
 	run_cmd nettest -D -d ${NSA_DEV} -r ${a} -0 ${a}
 	log_test_addr ${a} $? 0 "Device server, device client, local conn"
 
 	log_start
 	run_cmd nettest -D -d ${NSA_DEV} -r ${a}
@@ -1484,63 +1421,55 @@ ipv4_udp_vrf()
 	#
 	for a in ${NSA_IP} ${VRF_IP}
 	do
 		log_start
 		show_hint "Fails because ingress is in a VRF and global server is disabled"
-		run_cmd nettest -D -s &
-		sleep 1
+		run_cmd nettest -D -s -k
 		run_cmd_nsb nettest -D -r ${a}
 		log_test_addr ${a} $? 1 "Global server"
 
 		log_start
-		run_cmd nettest -D -I ${VRF} -s -3 ${NSA_DEV} &
-		sleep 1
+		run_cmd nettest -D -I ${VRF} -s -3 ${NSA_DEV} -k
 		run_cmd_nsb nettest -D -r ${a}
 		log_test_addr ${a} $? 0 "VRF server"
 
 		log_start
-		run_cmd nettest -D -I ${NSA_DEV} -s -3 ${NSA_DEV} &
-		sleep 1
+		run_cmd nettest -D -I ${NSA_DEV} -s -3 ${NSA_DEV} -k
 		run_cmd_nsb nettest -D -r ${a}
 		log_test_addr ${a} $? 0 "Enslaved device server"
 
 		log_start
 		show_hint "Should fail 'Connection refused' since there is no server"
 		run_cmd_nsb nettest -D -r ${a}
 		log_test_addr ${a} $? 1 "No server"
 
 		log_start
 		show_hint "Should fail 'Connection refused' since global server is out of scope"
-		run_cmd nettest -D -s &
-		sleep 1
+		run_cmd nettest -D -s -k
 		run_cmd nettest -D -d ${VRF} -r ${a}
 		log_test_addr ${a} $? 1 "Global server, VRF client, local connection"
 	done
 
 	a=${NSA_IP}
 	log_start
-	run_cmd nettest -s -D -I ${VRF} -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -s -D -I ${VRF} -3 ${NSA_DEV} -k
 	run_cmd nettest -D -d ${VRF} -r ${a}
 	log_test_addr ${a} $? 0 "VRF server, VRF client, local conn"
 
 	log_start
-	run_cmd nettest -s -D -I ${VRF} -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -s -D -I ${VRF} -3 ${NSA_DEV} -k
 	run_cmd nettest -D -d ${NSA_DEV} -r ${a}
 	log_test_addr ${a} $? 0 "VRF server, enslaved device client, local connection"
 
 	a=${NSA_IP}
 	log_start
-	run_cmd nettest -s -D -I ${NSA_DEV} -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -s -D -I ${NSA_DEV} -3 ${NSA_DEV} -k
 	run_cmd nettest -D -d ${VRF} -r ${a}
 	log_test_addr ${a} $? 0 "Enslaved device server, VRF client, local conn"
 
 	log_start
-	run_cmd nettest -s -D -I ${NSA_DEV} -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -s -D -I ${NSA_DEV} -3 ${NSA_DEV} -k
 	run_cmd nettest -D -d ${NSA_DEV} -r ${a}
 	log_test_addr ${a} $? 0 "Enslaved device server, device client, local conn"
 
 	# enable global server
 	log_subsection "Global server enabled"
@@ -1550,24 +1479,21 @@ ipv4_udp_vrf()
 	# server tests
 	#
 	for a in ${NSA_IP} ${VRF_IP}
 	do
 		log_start
-		run_cmd nettest -D -s -3 ${NSA_DEV} &
-		sleep 1
+		run_cmd nettest -D -s -3 ${NSA_DEV} -k
 		run_cmd_nsb nettest -D -r ${a}
 		log_test_addr ${a} $? 0 "Global server"
 
 		log_start
-		run_cmd nettest -D -I ${VRF} -s -3 ${NSA_DEV} &
-		sleep 1
+		run_cmd nettest -D -I ${VRF} -s -3 ${NSA_DEV} -k
 		run_cmd_nsb nettest -D -r ${a}
 		log_test_addr ${a} $? 0 "VRF server"
 
 		log_start
-		run_cmd nettest -D -I ${NSA_DEV} -s -3 ${NSA_DEV} &
-		sleep 1
+		run_cmd nettest -D -I ${NSA_DEV} -s -3 ${NSA_DEV} -k
 		run_cmd_nsb nettest -D -r ${a}
 		log_test_addr ${a} $? 0 "Enslaved device server"
 
 		log_start
 		show_hint "Should fail 'Connection refused'"
@@ -1577,18 +1503,16 @@ ipv4_udp_vrf()
 
 	#
 	# client tests
 	#
 	log_start
-	run_cmd_nsb nettest -D -s &
-	sleep 1
+	run_cmd_nsb nettest -D -s -k
 	run_cmd nettest -d ${VRF} -D -r ${NSB_IP} -1 ${NSA_IP}
 	log_test $? 0 "VRF client"
 
 	log_start
-	run_cmd_nsb nettest -D -s &
-	sleep 1
+	run_cmd_nsb nettest -D -s -k
 	run_cmd nettest -d ${NSA_DEV} -D -r ${NSB_IP} -1 ${NSA_IP}
 	log_test $? 0 "Enslaved device client"
 
 	# negative test - should fail
 	log_start
@@ -1604,53 +1528,46 @@ ipv4_udp_vrf()
 	#
 	# local address tests
 	#
 	a=${NSA_IP}
 	log_start
-	run_cmd nettest -D -s -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -D -s -3 ${NSA_DEV} -k
 	run_cmd nettest -D -d ${VRF} -r ${a}
 	log_test_addr ${a} $? 0 "Global server, VRF client, local conn"
 
 	log_start
-	run_cmd nettest -s -D -I ${VRF} -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -s -D -I ${VRF} -3 ${NSA_DEV} -k
 	run_cmd nettest -D -d ${VRF} -r ${a}
 	log_test_addr ${a} $? 0 "VRF server, VRF client, local conn"
 
 	log_start
-	run_cmd nettest -s -D -I ${VRF} -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -s -D -I ${VRF} -3 ${NSA_DEV} -k
 	run_cmd nettest -D -d ${NSA_DEV} -r ${a}
 	log_test_addr ${a} $? 0 "VRF server, device client, local conn"
 
 	log_start
-	run_cmd nettest -s -D -I ${NSA_DEV} -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -s -D -I ${NSA_DEV} -3 ${NSA_DEV} -k
 	run_cmd nettest -D -d ${VRF} -r ${a}
 	log_test_addr ${a} $? 0 "Enslaved device server, VRF client, local conn"
 
 	log_start
-	run_cmd nettest -s -D -I ${NSA_DEV} -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -s -D -I ${NSA_DEV} -3 ${NSA_DEV} -k
 	run_cmd nettest -D -d ${NSA_DEV} -r ${a}
 	log_test_addr ${a} $? 0 "Enslaved device server, device client, local conn"
 
 	for a in ${VRF_IP} 127.0.0.1
 	do
 		log_start
-		run_cmd nettest -D -s -3 ${VRF} &
-		sleep 1
+		run_cmd nettest -D -s -3 ${VRF} -k
 		run_cmd nettest -D -d ${VRF} -r ${a}
 		log_test_addr ${a} $? 0 "Global server, VRF client, local conn"
 	done
 
 	for a in ${VRF_IP} 127.0.0.1
 	do
 		log_start
-		run_cmd nettest -s -D -I ${VRF} -3 ${VRF} &
-		sleep 1
+		run_cmd nettest -s -D -I ${VRF} -3 ${VRF} -k
 		run_cmd nettest -D -d ${VRF} -r ${a}
 		log_test_addr ${a} $? 0 "VRF server, VRF client, local conn"
 	done
 
 	# negative test - should fail
@@ -1807,12 +1724,11 @@ ipv4_rt()
 	# server tests
 	#
 	for a in ${NSA_IP} ${VRF_IP}
 	do
 		log_start
-		run_cmd nettest ${varg} -s &
-		sleep 1
+		run_cmd nettest ${varg} -s -k
 		run_cmd_nsb nettest ${varg} -r ${a} &
 		sleep 3
 		run_cmd ip link del ${VRF}
 		sleep 1
 		log_test_addr ${a} 0 0 "${desc}, global server"
@@ -1821,12 +1737,11 @@ ipv4_rt()
 	done
 
 	for a in ${NSA_IP} ${VRF_IP}
 	do
 		log_start
-		run_cmd nettest ${varg} -s -I ${VRF} &
-		sleep 1
+		run_cmd nettest ${varg} -s -I ${VRF} -k
 		run_cmd_nsb nettest ${varg} -r ${a} &
 		sleep 3
 		run_cmd ip link del ${VRF}
 		sleep 1
 		log_test_addr ${a} 0 0 "${desc}, VRF server"
@@ -1834,12 +1749,11 @@ ipv4_rt()
 		setup ${with_vrf}
 	done
 
 	a=${NSA_IP}
 	log_start
-	run_cmd nettest ${varg} -s -I ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest ${varg} -s -I ${NSA_DEV} -k
 	run_cmd_nsb nettest ${varg} -r ${a} &
 	sleep 3
 	run_cmd ip link del ${VRF}
 	sleep 1
 	log_test_addr ${a} 0 0 "${desc}, enslaved device server"
@@ -1848,23 +1762,21 @@ ipv4_rt()
 
 	#
 	# client test
 	#
 	log_start
-	run_cmd_nsb nettest ${varg} -s &
-	sleep 1
+	run_cmd_nsb nettest ${varg} -s -k
 	run_cmd nettest ${varg} -d ${VRF} -r ${NSB_IP} &
 	sleep 3
 	run_cmd ip link del ${VRF}
 	sleep 1
 	log_test_addr ${a} 0 0 "${desc}, VRF client"
 
 	setup ${with_vrf}
 
 	log_start
-	run_cmd_nsb nettest ${varg} -s &
-	sleep 1
+	run_cmd_nsb nettest ${varg} -s -k
 	run_cmd nettest ${varg} -d ${NSA_DEV} -r ${NSB_IP} &
 	sleep 3
 	run_cmd ip link del ${VRF}
 	sleep 1
 	log_test_addr ${a} 0 0 "${desc}, enslaved device client"
@@ -1875,12 +1787,11 @@ ipv4_rt()
 	# local address tests
 	#
 	for a in ${NSA_IP} ${VRF_IP}
 	do
 		log_start
-		run_cmd nettest ${varg} -s &
-		sleep 1
+		run_cmd nettest ${varg} -s -k
 		run_cmd nettest ${varg} -d ${VRF} -r ${a} &
 		sleep 3
 		run_cmd ip link del ${VRF}
 		sleep 1
 		log_test_addr ${a} 0 0 "${desc}, global server, VRF client, local"
@@ -1889,12 +1800,11 @@ ipv4_rt()
 	done
 
 	for a in ${NSA_IP} ${VRF_IP}
 	do
 		log_start
-		run_cmd nettest ${varg} -I ${VRF} -s &
-		sleep 1
+		run_cmd nettest ${varg} -I ${VRF} -s -k
 		run_cmd nettest ${varg} -d ${VRF} -r ${a} &
 		sleep 3
 		run_cmd ip link del ${VRF}
 		sleep 1
 		log_test_addr ${a} 0 0 "${desc}, VRF server and client, local"
@@ -1902,34 +1812,31 @@ ipv4_rt()
 		setup ${with_vrf}
 	done
 
 	a=${NSA_IP}
 	log_start
-	run_cmd nettest ${varg} -s &
-	sleep 1
+	run_cmd nettest ${varg} -s -k
 	run_cmd nettest ${varg} -d ${NSA_DEV} -r ${a} &
 	sleep 3
 	run_cmd ip link del ${VRF}
 	sleep 1
 	log_test_addr ${a} 0 0 "${desc}, global server, enslaved device client, local"
 
 	setup ${with_vrf}
 
 	log_start
-	run_cmd nettest ${varg} -I ${VRF} -s &
-	sleep 1
+	run_cmd nettest ${varg} -I ${VRF} -s -k
 	run_cmd nettest ${varg} -d ${NSA_DEV} -r ${a} &
 	sleep 3
 	run_cmd ip link del ${VRF}
 	sleep 1
 	log_test_addr ${a} 0 0 "${desc}, VRF server, enslaved device client, local"
 
 	setup ${with_vrf}
 
 	log_start
-	run_cmd nettest ${varg} -I ${NSA_DEV} -s &
-	sleep 1
+	run_cmd nettest ${varg} -I ${NSA_DEV} -s -k
 	run_cmd nettest ${varg} -d ${NSA_DEV} -r ${a} &
 	sleep 3
 	run_cmd ip link del ${VRF}
 	sleep 1
 	log_test_addr ${a} 0 0 "${desc}, enslaved device server and client, local"
@@ -2268,63 +2175,56 @@ ipv6_tcp_md5_novrf()
 	# single address
 	#
 
 	# basic use case
 	log_start
-	run_cmd nettest -6 -s -M ${MD5_PW} -m ${NSB_IP6} &
-	sleep 1
+	run_cmd nettest -6 -s -M ${MD5_PW} -m ${NSB_IP6} -k
 	run_cmd_nsb nettest -6 -r ${NSA_IP6} -X ${MD5_PW}
 	log_test $? 0 "MD5: Single address config"
 
 	# client sends MD5, server not configured
 	log_start
 	show_hint "Should timeout due to MD5 mismatch"
-	run_cmd nettest -6 -s &
-	sleep 1
+	run_cmd nettest -6 -s -k
 	run_cmd_nsb nettest -6 -r ${NSA_IP6} -X ${MD5_PW}
 	log_test $? 2 "MD5: Server no config, client uses password"
 
 	# wrong password
 	log_start
 	show_hint "Should timeout since client uses wrong password"
-	run_cmd nettest -6 -s -M ${MD5_PW} -m ${NSB_IP6} &
-	sleep 1
+	run_cmd nettest -6 -s -M ${MD5_PW} -m ${NSB_IP6} -k
 	run_cmd_nsb nettest -6 -r ${NSA_IP6} -X ${MD5_WRONG_PW}
 	log_test $? 2 "MD5: Client uses wrong password"
 
 	# client from different address
 	log_start
 	show_hint "Should timeout due to MD5 mismatch"
-	run_cmd nettest -6 -s -M ${MD5_PW} -m ${NSB_LO_IP6} &
-	sleep 1
+	run_cmd nettest -6 -s -M ${MD5_PW} -m ${NSB_LO_IP6} -k
 	run_cmd_nsb nettest -6 -r ${NSA_IP6} -X ${MD5_PW}
 	log_test $? 2 "MD5: Client address does not match address configured with password"
 
 	#
 	# MD5 extension - prefix length
 	#
 
 	# client in prefix
 	log_start
-	run_cmd nettest -6 -s -M ${MD5_PW} -m ${NS_NET6} &
-	sleep 1
+	run_cmd nettest -6 -s -M ${MD5_PW} -m ${NS_NET6} -k
 	run_cmd_nsb nettest -6 -r ${NSA_IP6} -X ${MD5_PW}
 	log_test $? 0 "MD5: Prefix config"
 
 	# client in prefix, wrong password
 	log_start
 	show_hint "Should timeout since client uses wrong password"
-	run_cmd nettest -6 -s -M ${MD5_PW} -m ${NS_NET6} &
-	sleep 1
+	run_cmd nettest -6 -s -M ${MD5_PW} -m ${NS_NET6} -k
 	run_cmd_nsb nettest -6 -r ${NSA_IP6} -X ${MD5_WRONG_PW}
 	log_test $? 2 "MD5: Prefix config, client uses wrong password"
 
 	# client outside of prefix
 	log_start
 	show_hint "Should timeout due to MD5 mismatch"
-	run_cmd nettest -6 -s -M ${MD5_PW} -m ${NS_NET6} &
-	sleep 1
+	run_cmd nettest -6 -s -M ${MD5_PW} -m ${NS_NET6} -k
 	run_cmd_nsb nettest -6 -c ${NSB_LO_IP6} -r ${NSA_IP6} -X ${MD5_PW}
 	log_test $? 2 "MD5: Prefix config, client address not in configured prefix"
 }
 
 #
@@ -2336,127 +2236,112 @@ ipv6_tcp_md5()
 	# single address
 	#
 
 	# basic use case
 	log_start
-	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP6} &
-	sleep 1
+	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP6} -k
 	run_cmd_nsb nettest -6 -r ${NSA_IP6} -X ${MD5_PW}
 	log_test $? 0 "MD5: VRF: Single address config"
 
 	# client sends MD5, server not configured
 	log_start
 	show_hint "Should timeout since server does not have MD5 auth"
-	run_cmd nettest -6 -s -I ${VRF} &
-	sleep 1
+	run_cmd nettest -6 -s -I ${VRF} -k
 	run_cmd_nsb nettest -6 -r ${NSA_IP6} -X ${MD5_PW}
 	log_test $? 2 "MD5: VRF: Server no config, client uses password"
 
 	# wrong password
 	log_start
 	show_hint "Should timeout since client uses wrong password"
-	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP6} &
-	sleep 1
+	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP6} -k
 	run_cmd_nsb nettest -6 -r ${NSA_IP6} -X ${MD5_WRONG_PW}
 	log_test $? 2 "MD5: VRF: Client uses wrong password"
 
 	# client from different address
 	log_start
 	show_hint "Should timeout since server config differs from client"
-	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NSB_LO_IP6} &
-	sleep 1
+	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NSB_LO_IP6} -k
 	run_cmd_nsb nettest -6 -r ${NSA_IP6} -X ${MD5_PW}
 	log_test $? 2 "MD5: VRF: Client address does not match address configured with password"
 
 	#
 	# MD5 extension - prefix length
 	#
 
 	# client in prefix
 	log_start
-	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET6} &
-	sleep 1
+	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET6} -k
 	run_cmd_nsb nettest -6 -r ${NSA_IP6} -X ${MD5_PW}
 	log_test $? 0 "MD5: VRF: Prefix config"
 
 	# client in prefix, wrong password
 	log_start
 	show_hint "Should timeout since client uses wrong password"
-	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET6} &
-	sleep 1
+	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET6} -k
 	run_cmd_nsb nettest -6 -r ${NSA_IP6} -X ${MD5_WRONG_PW}
 	log_test $? 2 "MD5: VRF: Prefix config, client uses wrong password"
 
 	# client outside of prefix
 	log_start
 	show_hint "Should timeout since client address is outside of prefix"
-	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET6} &
-	sleep 1
+	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET6} -k
 	run_cmd_nsb nettest -6 -c ${NSB_LO_IP6} -r ${NSA_IP6} -X ${MD5_PW}
 	log_test $? 2 "MD5: VRF: Prefix config, client address not in configured prefix"
 
 	#
 	# duplicate config between default VRF and a VRF
 	#
 
 	log_start
-	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP6} &
-	run_cmd nettest -6 -s -M ${MD5_WRONG_PW} -m ${NSB_IP6} &
-	sleep 1
+	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP6} -k
+	run_cmd nettest -6 -s -M ${MD5_WRONG_PW} -m ${NSB_IP6} -k
 	run_cmd_nsb nettest -6 -r ${NSA_IP6} -X ${MD5_PW}
 	log_test $? 0 "MD5: VRF: Single address config in default VRF and VRF, conn in VRF"
 
 	log_start
-	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP6} &
-	run_cmd nettest -6 -s -M ${MD5_WRONG_PW} -m ${NSB_IP6} &
-	sleep 1
+	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP6} -k
+	run_cmd nettest -6 -s -M ${MD5_WRONG_PW} -m ${NSB_IP6} -k
 	run_cmd_nsc nettest -6 -r ${NSA_IP6} -X ${MD5_WRONG_PW}
 	log_test $? 0 "MD5: VRF: Single address config in default VRF and VRF, conn in default VRF"
 
 	log_start
 	show_hint "Should timeout since client in default VRF uses VRF password"
-	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP6} &
-	run_cmd nettest -6 -s -M ${MD5_WRONG_PW} -m ${NSB_IP6} &
-	sleep 1
+	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP6} -k
+	run_cmd nettest -6 -s -M ${MD5_WRONG_PW} -m ${NSB_IP6} -k
 	run_cmd_nsc nettest -6 -r ${NSA_IP6} -X ${MD5_PW}
 	log_test $? 2 "MD5: VRF: Single address config in default VRF and VRF, conn in default VRF with VRF pw"
 
 	log_start
 	show_hint "Should timeout since client in VRF uses default VRF password"
-	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP6} &
-	run_cmd nettest -6 -s -M ${MD5_WRONG_PW} -m ${NSB_IP6} &
-	sleep 1
+	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NSB_IP6} -k
+	run_cmd nettest -6 -s -M ${MD5_WRONG_PW} -m ${NSB_IP6} -k
 	run_cmd_nsb nettest -6 -r ${NSA_IP6} -X ${MD5_WRONG_PW}
 	log_test $? 2 "MD5: VRF: Single address config in default VRF and VRF, conn in VRF with default VRF pw"
 
 	log_start
-	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET6} &
-	run_cmd nettest -6 -s -M ${MD5_WRONG_PW} -m ${NS_NET6} &
-	sleep 1
+	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET6} -k
+	run_cmd nettest -6 -s -M ${MD5_WRONG_PW} -m ${NS_NET6} -k
 	run_cmd_nsb nettest -6 -r ${NSA_IP6} -X ${MD5_PW}
 	log_test $? 0 "MD5: VRF: Prefix config in default VRF and VRF, conn in VRF"
 
 	log_start
-	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET6} &
-	run_cmd nettest -6 -s -M ${MD5_WRONG_PW} -m ${NS_NET6} &
-	sleep 1
+	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET6} -k
+	run_cmd nettest -6 -s -M ${MD5_WRONG_PW} -m ${NS_NET6} -k
 	run_cmd_nsc nettest -6 -r ${NSA_IP6} -X ${MD5_WRONG_PW}
 	log_test $? 0 "MD5: VRF: Prefix config in default VRF and VRF, conn in default VRF"
 
 	log_start
 	show_hint "Should timeout since client in default VRF uses VRF password"
-	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET6} &
-	run_cmd nettest -6 -s -M ${MD5_WRONG_PW} -m ${NS_NET6} &
-	sleep 1
+	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET6} -k
+	run_cmd nettest -6 -s -M ${MD5_WRONG_PW} -m ${NS_NET6} -k
 	run_cmd_nsc nettest -6 -r ${NSA_IP6} -X ${MD5_PW}
 	log_test $? 2 "MD5: VRF: Prefix config in default VRF and VRF, conn in default VRF with VRF pw"
 
 	log_start
 	show_hint "Should timeout since client in VRF uses default VRF password"
-	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET6} &
-	run_cmd nettest -6 -s -M ${MD5_WRONG_PW} -m ${NS_NET6} &
-	sleep 1
+	run_cmd nettest -6 -s -I ${VRF} -M ${MD5_PW} -m ${NS_NET6} -k
+	run_cmd nettest -6 -s -M ${MD5_WRONG_PW} -m ${NS_NET6} -k
 	run_cmd_nsb nettest -6 -r ${NSA_IP6} -X ${MD5_WRONG_PW}
 	log_test $? 2 "MD5: VRF: Prefix config in default VRF and VRF, conn in VRF with default VRF pw"
 
 	#
 	# negative tests
@@ -2479,12 +2364,11 @@ ipv6_tcp_novrf()
 	# server tests
 	#
 	for a in ${NSA_IP6} ${NSA_LO_IP6} ${NSA_LINKIP6}%${NSB_DEV}
 	do
 		log_start
-		run_cmd nettest -6 -s &
-		sleep 1
+		run_cmd nettest -6 -s -k
 		run_cmd_nsb nettest -6 -r ${a}
 		log_test_addr ${a} $? 0 "Global server"
 	done
 
 	# verify TCP reset received
@@ -2500,21 +2384,19 @@ ipv6_tcp_novrf()
 	# client
 	#
 	for a in ${NSB_IP6} ${NSB_LO_IP6} ${NSB_LINKIP6}%${NSA_DEV}
 	do
 		log_start
-		run_cmd_nsb nettest -6 -s &
-		sleep 1
+		run_cmd_nsb nettest -6 -s -k
 		run_cmd nettest -6 -r ${a}
 		log_test_addr ${a} $? 0 "Client"
 	done
 
 	for a in ${NSB_IP6} ${NSB_LO_IP6} ${NSB_LINKIP6}%${NSA_DEV}
 	do
 		log_start
-		run_cmd_nsb nettest -6 -s &
-		sleep 1
+		run_cmd_nsb nettest -6 -s -k
 		run_cmd nettest -6 -r ${a} -d ${NSA_DEV}
 		log_test_addr ${a} $? 0 "Client, device bind"
 	done
 
 	for a in ${NSB_IP6} ${NSB_LO_IP6} ${NSB_LINKIP6}%${NSA_DEV}
@@ -2529,55 +2411,49 @@ ipv6_tcp_novrf()
 	# local address tests
 	#
 	for a in ${NSA_IP6} ${NSA_LO_IP6} ::1
 	do
 		log_start
-		run_cmd nettest -6 -s &
-		sleep 1
+		run_cmd nettest -6 -s -k
 		run_cmd nettest -6 -r ${a}
 		log_test_addr ${a} $? 0 "Global server, local connection"
 	done
 
 	a=${NSA_IP6}
 	log_start
-	run_cmd nettest -6 -s -I ${NSA_DEV} -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -6 -s -I ${NSA_DEV} -3 ${NSA_DEV} -k
 	run_cmd nettest -6 -r ${a} -0 ${a}
 	log_test_addr ${a} $? 0 "Device server, unbound client, local connection"
 
 	for a in ${NSA_LO_IP6} ::1
 	do
 		log_start
 		show_hint "Should fail 'Connection refused' since addresses on loopback are out of device scope"
-		run_cmd nettest -6 -s -I ${NSA_DEV} &
-		sleep 1
+		run_cmd nettest -6 -s -I ${NSA_DEV} -k
 		run_cmd nettest -6 -r ${a}
 		log_test_addr ${a} $? 1 "Device server, unbound client, local connection"
 	done
 
 	a=${NSA_IP6}
 	log_start
-	run_cmd nettest -6 -s &
-	sleep 1
+	run_cmd nettest -6 -s -k
 	run_cmd nettest -6 -r ${a} -d ${NSA_DEV} -0 ${a}
 	log_test_addr ${a} $? 0 "Global server, device client, local connection"
 
 	for a in ${NSA_LO_IP6} ::1
 	do
 		log_start
 		show_hint "Should fail 'Connection refused' since addresses on loopback are out of device scope"
-		run_cmd nettest -6 -s &
-		sleep 1
+		run_cmd nettest -6 -s -k
 		run_cmd nettest -6 -r ${a} -d ${NSA_DEV}
 		log_test_addr ${a} $? 1 "Global server, device client, local connection"
 	done
 
 	for a in ${NSA_IP6} ${NSA_LINKIP6}
 	do
 		log_start
-		run_cmd nettest -6 -s -I ${NSA_DEV} -3 ${NSA_DEV} &
-		sleep 1
+		run_cmd nettest -6 -s -I ${NSA_DEV} -3 ${NSA_DEV} -k
 		run_cmd nettest -6  -d ${NSA_DEV} -r ${a}
 		log_test_addr ${a} $? 0 "Device server, device client, local conn"
 	done
 
 	for a in ${NSA_IP6} ${NSA_LINKIP6}
@@ -2605,38 +2481,34 @@ ipv6_tcp_vrf()
 	#
 	for a in ${NSA_IP6} ${VRF_IP6} ${NSA_LINKIP6}%${NSB_DEV}
 	do
 		log_start
 		show_hint "Should fail 'Connection refused' since global server with VRF is disabled"
-		run_cmd nettest -6 -s &
-		sleep 1
+		run_cmd nettest -6 -s -k
 		run_cmd_nsb nettest -6 -r ${a}
 		log_test_addr ${a} $? 1 "Global server"
 	done
 
 	for a in ${NSA_IP6} ${VRF_IP6}
 	do
 		log_start
-		run_cmd nettest -6 -s -I ${VRF} -3 ${VRF} &
-		sleep 1
+		run_cmd nettest -6 -s -I ${VRF} -3 ${VRF} -k
 		run_cmd_nsb nettest -6 -r ${a}
 		log_test_addr ${a} $? 0 "VRF server"
 	done
 
 	# link local is always bound to ingress device
 	a=${NSA_LINKIP6}%${NSB_DEV}
 	log_start
-	run_cmd nettest -6 -s -I ${VRF} -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -6 -s -I ${VRF} -3 ${NSA_DEV} -k
 	run_cmd_nsb nettest -6 -r ${a}
 	log_test_addr ${a} $? 0 "VRF server"
 
 	for a in ${NSA_IP6} ${VRF_IP6} ${NSA_LINKIP6}%${NSB_DEV}
 	do
 		log_start
-		run_cmd nettest -6 -s -I ${NSA_DEV} -3 ${NSA_DEV} &
-		sleep 1
+		run_cmd nettest -6 -s -I ${NSA_DEV} -3 ${NSA_DEV} -k
 		run_cmd_nsb nettest -6 -r ${a}
 		log_test_addr ${a} $? 0 "Device server"
 	done
 
 	# verify TCP reset received
@@ -2650,12 +2522,11 @@ ipv6_tcp_vrf()
 
 	# local address tests
 	a=${NSA_IP6}
 	log_start
 	show_hint "Should fail 'Connection refused' since global server with VRF is disabled"
-	run_cmd nettest -6 -s &
-	sleep 1
+	run_cmd nettest -6 -s -k
 	run_cmd nettest -6 -r ${a} -d ${NSA_DEV}
 	log_test_addr ${a} $? 1 "Global server, local connection"
 
 	# run MD5 tests
 	ipv6_tcp_md5
@@ -2667,44 +2538,39 @@ ipv6_tcp_vrf()
 	set_sysctl net.ipv4.tcp_l3mdev_accept=1
 
 	for a in ${NSA_IP6} ${VRF_IP6}
 	do
 		log_start
-		run_cmd nettest -6 -s -3 ${VRF} &
-		sleep 1
+		run_cmd nettest -6 -s -3 ${VRF} -k
 		run_cmd_nsb nettest -6 -r ${a}
 		log_test_addr ${a} $? 0 "Global server"
 	done
 
 	for a in ${NSA_IP6} ${VRF_IP6}
 	do
 		log_start
-		run_cmd nettest -6 -s -I ${VRF} -3 ${VRF} &
-		sleep 1
+		run_cmd nettest -6 -s -I ${VRF} -3 ${VRF} -k
 		run_cmd_nsb nettest -6 -r ${a}
 		log_test_addr ${a} $? 0 "VRF server"
 	done
 
 	# For LLA, child socket is bound to device
 	a=${NSA_LINKIP6}%${NSB_DEV}
 	log_start
-	run_cmd nettest -6 -s -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -6 -s -3 ${NSA_DEV} -k
 	run_cmd_nsb nettest -6 -r ${a}
 	log_test_addr ${a} $? 0 "Global server"
 
 	log_start
-	run_cmd nettest -6 -s -I ${VRF} -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -6 -s -I ${VRF} -3 ${NSA_DEV} -k
 	run_cmd_nsb nettest -6 -r ${a}
 	log_test_addr ${a} $? 0 "VRF server"
 
 	for a in ${NSA_IP6} ${NSA_LINKIP6}%${NSB_DEV}
 	do
 		log_start
-		run_cmd nettest -6 -s -I ${NSA_DEV} -3 ${NSA_DEV} &
-		sleep 1
+		run_cmd nettest -6 -s -I ${NSA_DEV} -3 ${NSA_DEV} -k
 		run_cmd_nsb nettest -6 -r ${a}
 		log_test_addr ${a} $? 0 "Device server"
 	done
 
 	# verify TCP reset received
@@ -2719,12 +2585,11 @@ ipv6_tcp_vrf()
 	# local address tests
 	for a in ${NSA_IP6} ${VRF_IP6}
 	do
 		log_start
 		show_hint "Fails 'Connection refused' since client is not in VRF"
-		run_cmd nettest -6 -s -I ${VRF} &
-		sleep 1
+		run_cmd nettest -6 -s -I ${VRF} -k
 		run_cmd nettest -6 -r ${a}
 		log_test_addr ${a} $? 1 "Global server, local connection"
 	done
 
 
@@ -2732,29 +2597,26 @@ ipv6_tcp_vrf()
 	# client
 	#
 	for a in ${NSB_IP6} ${NSB_LO_IP6}
 	do
 		log_start
-		run_cmd_nsb nettest -6 -s &
-		sleep 1
+		run_cmd_nsb nettest -6 -s -k
 		run_cmd nettest -6 -r ${a} -d ${VRF}
 		log_test_addr ${a} $? 0 "Client, VRF bind"
 	done
 
 	a=${NSB_LINKIP6}
 	log_start
 	show_hint "Fails since VRF device does not allow linklocal addresses"
-	run_cmd_nsb nettest -6 -s &
-	sleep 1
+	run_cmd_nsb nettest -6 -s -k
 	run_cmd nettest -6 -r ${a} -d ${VRF}
 	log_test_addr ${a} $? 1 "Client, VRF bind"
 
 	for a in ${NSB_IP6} ${NSB_LO_IP6} ${NSB_LINKIP6}
 	do
 		log_start
-		run_cmd_nsb nettest -6 -s &
-		sleep 1
+		run_cmd_nsb nettest -6 -s -k
 		run_cmd nettest -6 -r ${a} -d ${NSA_DEV}
 		log_test_addr ${a} $? 0 "Client, device bind"
 	done
 
 	for a in ${NSB_IP6} ${NSB_LO_IP6}
@@ -2774,42 +2636,37 @@ ipv6_tcp_vrf()
 	done
 
 	for a in ${NSA_IP6} ${VRF_IP6} ::1
 	do
 		log_start
-		run_cmd nettest -6 -s -I ${VRF} -3 ${VRF} &
-		sleep 1
+		run_cmd nettest -6 -s -I ${VRF} -3 ${VRF} -k
 		run_cmd nettest -6 -r ${a} -d ${VRF} -0 ${a}
 		log_test_addr ${a} $? 0 "VRF server, VRF client, local connection"
 	done
 
 	a=${NSA_IP6}
 	log_start
-	run_cmd nettest -6 -s -I ${VRF} -3 ${VRF} &
-	sleep 1
+	run_cmd nettest -6 -s -I ${VRF} -3 ${VRF} -k
 	run_cmd nettest -6 -r ${a} -d ${NSA_DEV} -0 ${a}
 	log_test_addr ${a} $? 0 "VRF server, device client, local connection"
 
 	a=${NSA_IP6}
 	log_start
 	show_hint "Should fail since unbound client is out of VRF scope"
-	run_cmd nettest -6 -s -I ${VRF} &
-	sleep 1
+	run_cmd nettest -6 -s -I ${VRF} -k
 	run_cmd nettest -6 -r ${a}
 	log_test_addr ${a} $? 1 "VRF server, unbound client, local connection"
 
 	log_start
-	run_cmd nettest -6 -s -I ${NSA_DEV} -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -6 -s -I ${NSA_DEV} -3 ${NSA_DEV} -k
 	run_cmd nettest -6 -r ${a} -d ${VRF} -0 ${a}
 	log_test_addr ${a} $? 0 "Device server, VRF client, local connection"
 
 	for a in ${NSA_IP6} ${NSA_LINKIP6}
 	do
 		log_start
-		run_cmd nettest -6 -s -I ${NSA_DEV} -3 ${NSA_DEV} &
-		sleep 1
+		run_cmd nettest -6 -s -I ${NSA_DEV} -3 ${NSA_DEV} -k
 		run_cmd nettest -6 -r ${a} -d ${NSA_DEV} -0 ${a}
 		log_test_addr ${a} $? 0 "Device server, device client, local connection"
 	done
 }
 
@@ -2844,26 +2701,23 @@ ipv6_udp_novrf()
 	# server tests
 	#
 	for a in ${NSA_IP6} ${NSA_LINKIP6}%${NSB_DEV}
 	do
 		log_start
-		run_cmd nettest -6 -D -s -3 ${NSA_DEV} &
-		sleep 1
+		run_cmd nettest -6 -D -s -3 ${NSA_DEV} -k
 		run_cmd_nsb nettest -6 -D -r ${a}
 		log_test_addr ${a} $? 0 "Global server"
 
 		log_start
-		run_cmd nettest -6 -D -I ${NSA_DEV} -s -3 ${NSA_DEV} &
-		sleep 1
+		run_cmd nettest -6 -D -I ${NSA_DEV} -s -3 ${NSA_DEV} -k
 		run_cmd_nsb nettest -6 -D -r ${a}
 		log_test_addr ${a} $? 0 "Device server"
 	done
 
 	a=${NSA_LO_IP6}
 	log_start
-	run_cmd nettest -6 -D -s -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -6 -D -s -3 ${NSA_DEV} -k
 	run_cmd_nsb nettest -6 -D -r ${a}
 	log_test_addr ${a} $? 0 "Global server"
 
 	# should fail since loopback address is out of scope for a device
 	# bound server, but it does not - hence this is more documenting
@@ -2888,30 +2742,26 @@ ipv6_udp_novrf()
 	# client
 	#
 	for a in ${NSB_IP6} ${NSB_LO_IP6} ${NSB_LINKIP6}%${NSA_DEV}
 	do
 		log_start
-		run_cmd_nsb nettest -6 -D -s &
-		sleep 1
+		run_cmd_nsb nettest -6 -D -s -k
 		run_cmd nettest -6 -D -r ${a} -0 ${NSA_IP6}
 		log_test_addr ${a} $? 0 "Client"
 
 		log_start
-		run_cmd_nsb nettest -6 -D -s &
-		sleep 1
+		run_cmd_nsb nettest -6 -D -s -k
 		run_cmd nettest -6 -D -r ${a} -d ${NSA_DEV} -0 ${NSA_IP6}
 		log_test_addr ${a} $? 0 "Client, device bind"
 
 		log_start
-		run_cmd_nsb nettest -6 -D -s &
-		sleep 1
+		run_cmd_nsb nettest -6 -D -s -k
 		run_cmd nettest -6 -D -r ${a} -d ${NSA_DEV} -C -0 ${NSA_IP6}
 		log_test_addr ${a} $? 0 "Client, device send via cmsg"
 
 		log_start
-		run_cmd_nsb nettest -6 -D -s &
-		sleep 1
+		run_cmd_nsb nettest -6 -D -s -k
 		run_cmd nettest -6 -D -r ${a} -d ${NSA_DEV} -S -0 ${NSA_IP6}
 		log_test_addr ${a} $? 0 "Client, device bind via IPV6_UNICAST_IF"
 
 		log_start
 		show_hint "Should fail 'Connection refused'"
@@ -2928,80 +2778,70 @@ ipv6_udp_novrf()
 	# local address tests
 	#
 	for a in ${NSA_IP6} ${NSA_LO_IP6} ::1
 	do
 		log_start
-		run_cmd nettest -6 -D -s &
-		sleep 1
+		run_cmd nettest -6 -D -s -k
 		run_cmd nettest -6 -D -r ${a} -0 ${a} -1 ${a}
 		log_test_addr ${a} $? 0 "Global server, local connection"
 	done
 
 	a=${NSA_IP6}
 	log_start
-	run_cmd nettest -6 -s -D -I ${NSA_DEV} -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -6 -s -D -I ${NSA_DEV} -3 ${NSA_DEV} -k
 	run_cmd nettest -6 -D -r ${a}
 	log_test_addr ${a} $? 0 "Device server, unbound client, local connection"
 
 	for a in ${NSA_LO_IP6} ::1
 	do
 		log_start
 		show_hint "Should fail 'Connection refused' since address is out of device scope"
-		run_cmd nettest -6 -s -D -I ${NSA_DEV} &
-		sleep 1
+		run_cmd nettest -6 -s -D -I ${NSA_DEV} -k
 		run_cmd nettest -6 -D -r ${a}
 		log_test_addr ${a} $? 1 "Device server, local connection"
 	done
 
 	a=${NSA_IP6}
 	log_start
-	run_cmd nettest -6 -s -D &
-	sleep 1
+	run_cmd nettest -6 -s -D -k
 	run_cmd nettest -6 -D -d ${NSA_DEV} -r ${a}
 	log_test_addr ${a} $? 0 "Global server, device client, local connection"
 
 	log_start
-	run_cmd nettest -6 -s -D &
-	sleep 1
+	run_cmd nettest -6 -s -D -k
 	run_cmd nettest -6 -D -d ${NSA_DEV} -C -r ${a}
 	log_test_addr ${a} $? 0 "Global server, device send via cmsg, local connection"
 
 	log_start
-	run_cmd nettest -6 -s -D &
-	sleep 1
+	run_cmd nettest -6 -s -D -k
 	run_cmd nettest -6 -D -d ${NSA_DEV} -S -r ${a}
 	log_test_addr ${a} $? 0 "Global server, device client via IPV6_UNICAST_IF, local connection"
 
 	for a in ${NSA_LO_IP6} ::1
 	do
 		log_start
 		show_hint "Should fail 'No route to host' since addresses on loopback are out of device scope"
-		run_cmd nettest -6 -D -s &
-		sleep 1
+		run_cmd nettest -6 -D -s -k
 		run_cmd nettest -6 -D -r ${a} -d ${NSA_DEV}
 		log_test_addr ${a} $? 1 "Global server, device client, local connection"
 
 		log_start
 		show_hint "Should fail 'No route to host' since addresses on loopback are out of device scope"
-		run_cmd nettest -6 -D -s &
-		sleep 1
+		run_cmd nettest -6 -D -s -k
 		run_cmd nettest -6 -D -r ${a} -d ${NSA_DEV} -C
 		log_test_addr ${a} $? 1 "Global server, device send via cmsg, local connection"
 
 		log_start
 		show_hint "Should fail 'No route to host' since addresses on loopback are out of device scope"
-		run_cmd nettest -6 -D -s &
-		sleep 1
+		run_cmd nettest -6 -D -s -k
 		run_cmd nettest -6 -D -r ${a} -d ${NSA_DEV} -S
 		log_test_addr ${a} $? 1 "Global server, device client via IP_UNICAST_IF, local connection"
 	done
 
 	a=${NSA_IP6}
 	log_start
-	run_cmd nettest -6 -D -s -I ${NSA_DEV} -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -6 -D -s -I ${NSA_DEV} -3 ${NSA_DEV} -k
 	run_cmd nettest -6 -D -d ${NSA_DEV} -r ${a} -0 ${a}
 	log_test_addr ${a} $? 0 "Device server, device client, local conn"
 
 	log_start
 	show_hint "Should fail 'Connection refused'"
@@ -3010,12 +2850,11 @@ ipv6_udp_novrf()
 
 	# LLA to GUA
 	run_cmd_nsb ip -6 addr del ${NSB_IP6}/64 dev ${NSB_DEV}
 	run_cmd_nsb ip -6 ro add ${NSA_IP6}/128 dev ${NSB_DEV}
 	log_start
-	run_cmd nettest -6 -s -D &
-	sleep 1
+	run_cmd nettest -6 -s -D -k
 	run_cmd_nsb nettest -6 -D -r ${NSA_IP6}
 	log_test $? 0 "UDP in - LLA to GUA"
 
 	run_cmd_nsb ip -6 ro del ${NSA_IP6}/128 dev ${NSB_DEV}
 	run_cmd_nsb ip -6 addr add ${NSB_IP6}/64 dev ${NSB_DEV}
@@ -3034,30 +2873,27 @@ ipv6_udp_vrf()
 	#
 	for a in ${NSA_IP6} ${VRF_IP6}
 	do
 		log_start
 		show_hint "Should fail 'Connection refused' since global server is disabled"
-		run_cmd nettest -6 -D -s &
-		sleep 1
+		run_cmd nettest -6 -D -s -k
 		run_cmd_nsb nettest -6 -D -r ${a}
 		log_test_addr ${a} $? 1 "Global server"
 	done
 
 	for a in ${NSA_IP6} ${VRF_IP6}
 	do
 		log_start
-		run_cmd nettest -6 -D -I ${VRF} -s -3 ${NSA_DEV} &
-		sleep 1
+		run_cmd nettest -6 -D -I ${VRF} -s -3 ${NSA_DEV} -k
 		run_cmd_nsb nettest -6 -D -r ${a}
 		log_test_addr ${a} $? 0 "VRF server"
 	done
 
 	for a in ${NSA_IP6} ${VRF_IP6}
 	do
 		log_start
-		run_cmd nettest -6 -D -I ${NSA_DEV} -s -3 ${NSA_DEV} &
-		sleep 1
+		run_cmd nettest -6 -D -I ${NSA_DEV} -s -3 ${NSA_DEV} -k
 		run_cmd_nsb nettest -6 -D -r ${a}
 		log_test_addr ${a} $? 0 "Enslaved device server"
 	done
 
 	# negative test - should fail
@@ -3074,48 +2910,42 @@ ipv6_udp_vrf()
 	#
 	for a in ${NSA_IP6} ${VRF_IP6}
 	do
 		log_start
 		show_hint "Should fail 'Connection refused' since global server is disabled"
-		run_cmd nettest -6 -D -s &
-		sleep 1
+		run_cmd nettest -6 -D -s -k
 		run_cmd nettest -6 -D -d ${VRF} -r ${a}
 		log_test_addr ${a} $? 1 "Global server, VRF client, local conn"
 	done
 
 	for a in ${NSA_IP6} ${VRF_IP6}
 	do
 		log_start
-		run_cmd nettest -6 -D -I ${VRF} -s &
-		sleep 1
+		run_cmd nettest -6 -D -I ${VRF} -s -k
 		run_cmd nettest -6 -D -d ${VRF} -r ${a}
 		log_test_addr ${a} $? 0 "VRF server, VRF client, local conn"
 	done
 
 	a=${NSA_IP6}
 	log_start
 	show_hint "Should fail 'Connection refused' since global server is disabled"
-	run_cmd nettest -6 -D -s &
-	sleep 1
+	run_cmd nettest -6 -D -s -k
 	run_cmd nettest -6 -D -d ${NSA_DEV} -r ${a}
 	log_test_addr ${a} $? 1 "Global server, device client, local conn"
 
 	log_start
-	run_cmd nettest -6 -D -I ${VRF} -s -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -6 -D -I ${VRF} -s -3 ${NSA_DEV} -k
 	run_cmd nettest -6 -D -d ${NSA_DEV} -r ${a}
 	log_test_addr ${a} $? 0 "VRF server, device client, local conn"
 
 	log_start
-	run_cmd nettest -6 -D -I ${NSA_DEV} -s -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -6 -D -I ${NSA_DEV} -s -3 ${NSA_DEV} -k
 	run_cmd nettest -6 -D -d ${VRF} -r ${a}
 	log_test_addr ${a} $? 0 "Enslaved device server, VRF client, local conn"
 
 	log_start
-	run_cmd nettest -6 -D -I ${NSA_DEV} -s -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -6 -D -I ${NSA_DEV} -s -3 ${NSA_DEV} -k
 	run_cmd nettest -6 -D -d ${NSA_DEV} -r ${a}
 	log_test_addr ${a} $? 0 "Enslaved device server, device client, local conn"
 
 	# disable global server
 	log_subsection "Global server enabled"
@@ -3125,30 +2955,27 @@ ipv6_udp_vrf()
 	# server tests
 	#
 	for a in ${NSA_IP6} ${VRF_IP6}
 	do
 		log_start
-		run_cmd nettest -6 -D -s -3 ${NSA_DEV} &
-		sleep 1
+		run_cmd nettest -6 -D -s -3 ${NSA_DEV} -k
 		run_cmd_nsb nettest -6 -D -r ${a}
 		log_test_addr ${a} $? 0 "Global server"
 	done
 
 	for a in ${NSA_IP6} ${VRF_IP6}
 	do
 		log_start
-		run_cmd nettest -6 -D -I ${VRF} -s -3 ${NSA_DEV} &
-		sleep 1
+		run_cmd nettest -6 -D -I ${VRF} -s -3 ${NSA_DEV} -k
 		run_cmd_nsb nettest -6 -D -r ${a}
 		log_test_addr ${a} $? 0 "VRF server"
 	done
 
 	for a in ${NSA_IP6} ${VRF_IP6}
 	do
 		log_start
-		run_cmd nettest -6 -D -I ${NSA_DEV} -s -3 ${NSA_DEV} &
-		sleep 1
+		run_cmd nettest -6 -D -I ${NSA_DEV} -s -3 ${NSA_DEV} -k
 		run_cmd_nsb nettest -6 -D -r ${a}
 		log_test_addr ${a} $? 0 "Enslaved device server"
 	done
 
 	# negative test - should fail
@@ -3161,23 +2988,21 @@ ipv6_udp_vrf()
 
 	#
 	# client tests
 	#
 	log_start
-	run_cmd_nsb nettest -6 -D -s &
-	sleep 1
+	run_cmd_nsb nettest -6 -D -s -k
 	run_cmd nettest -6 -D -d ${VRF} -r ${NSB_IP6}
 	log_test $? 0 "VRF client"
 
 	# negative test - should fail
 	log_start
 	run_cmd nettest -6 -D -d ${VRF} -r ${NSB_IP6}
 	log_test $? 1 "No server, VRF client"
 
 	log_start
-	run_cmd_nsb nettest -6 -D -s &
-	sleep 1
+	run_cmd_nsb nettest -6 -D -s -k
 	run_cmd nettest -6 -D -d ${NSA_DEV} -r ${NSB_IP6}
 	log_test $? 0 "Enslaved device client"
 
 	# negative test - should fail
 	log_start
@@ -3187,32 +3012,28 @@ ipv6_udp_vrf()
 	#
 	# local address tests
 	#
 	a=${NSA_IP6}
 	log_start
-	run_cmd nettest -6 -D -s -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -6 -D -s -3 ${NSA_DEV} -k
 	run_cmd nettest -6 -D -d ${VRF} -r ${a}
 	log_test_addr ${a} $? 0 "Global server, VRF client, local conn"
 
 	#log_start
-	run_cmd nettest -6 -D -I ${VRF} -s -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -6 -D -I ${VRF} -s -3 ${NSA_DEV} -k
 	run_cmd nettest -6 -D -d ${VRF} -r ${a}
 	log_test_addr ${a} $? 0 "VRF server, VRF client, local conn"
 
 
 	a=${VRF_IP6}
 	log_start
-	run_cmd nettest -6 -D -s -3 ${VRF} &
-	sleep 1
+	run_cmd nettest -6 -D -s -3 ${VRF} -k
 	run_cmd nettest -6 -D -d ${VRF} -r ${a}
 	log_test_addr ${a} $? 0 "Global server, VRF client, local conn"
 
 	log_start
-	run_cmd nettest -6 -D -I ${VRF} -s -3 ${VRF} &
-	sleep 1
+	run_cmd nettest -6 -D -I ${VRF} -s -3 ${VRF} -k
 	run_cmd nettest -6 -D -d ${VRF} -r ${a}
 	log_test_addr ${a} $? 0 "VRF server, VRF client, local conn"
 
 	# negative test - should fail
 	for a in ${NSA_IP6} ${VRF_IP6}
@@ -3223,64 +3044,57 @@ ipv6_udp_vrf()
 	done
 
 	# device to global IP
 	a=${NSA_IP6}
 	log_start
-	run_cmd nettest -6 -D -s -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -6 -D -s -3 ${NSA_DEV} -k
 	run_cmd nettest -6 -D -d ${NSA_DEV} -r ${a}
 	log_test_addr ${a} $? 0 "Global server, device client, local conn"
 
 	log_start
-	run_cmd nettest -6 -D -I ${VRF} -s -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -6 -D -I ${VRF} -s -3 ${NSA_DEV} -k
 	run_cmd nettest -6 -D -d ${NSA_DEV} -r ${a}
 	log_test_addr ${a} $? 0 "VRF server, device client, local conn"
 
 	log_start
-	run_cmd nettest -6 -D -I ${NSA_DEV} -s -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -6 -D -I ${NSA_DEV} -s -3 ${NSA_DEV} -k
 	run_cmd nettest -6 -D -d ${VRF} -r ${a}
 	log_test_addr ${a} $? 0 "Device server, VRF client, local conn"
 
 	log_start
-	run_cmd nettest -6 -D -I ${NSA_DEV} -s -3 ${NSA_DEV} &
-	sleep 1
+	run_cmd nettest -6 -D -I ${NSA_DEV} -s -3 ${NSA_DEV} -k
 	run_cmd nettest -6 -D -d ${NSA_DEV} -r ${a}
 	log_test_addr ${a} $? 0 "Device server, device client, local conn"
 
 	log_start
 	run_cmd nettest -6 -D -d ${NSA_DEV} -r ${a}
 	log_test_addr ${a} $? 1 "No server, device client, local conn"
 
 
 	# link local addresses
 	log_start
-	run_cmd nettest -6 -D -s &
-	sleep 1
+	run_cmd nettest -6 -D -s -k
 	run_cmd_nsb nettest -6 -D -d ${NSB_DEV} -r ${NSA_LINKIP6}
 	log_test $? 0 "Global server, linklocal IP"
 
 	log_start
 	run_cmd_nsb nettest -6 -D -d ${NSB_DEV} -r ${NSA_LINKIP6}
 	log_test $? 1 "No server, linklocal IP"
 
 
 	log_start
-	run_cmd_nsb nettest -6 -D -s &
-	sleep 1
+	run_cmd_nsb nettest -6 -D -s -k
 	run_cmd nettest -6 -D -d ${NSA_DEV} -r ${NSB_LINKIP6}
 	log_test $? 0 "Enslaved device client, linklocal IP"
 
 	log_start
 	run_cmd nettest -6 -D -d ${NSA_DEV} -r ${NSB_LINKIP6}
 	log_test $? 1 "No server, device client, peer linklocal IP"
 
 
 	log_start
-	run_cmd nettest -6 -D -s &
-	sleep 1
+	run_cmd nettest -6 -D -s -k
 	run_cmd nettest -6 -D -d ${NSA_DEV} -r ${NSA_LINKIP6}
 	log_test $? 0 "Enslaved device client, local conn - linklocal IP"
 
 	log_start
 	run_cmd nettest -6 -D -d ${NSA_DEV} -r ${NSA_LINKIP6}
@@ -3288,12 +3102,11 @@ ipv6_udp_vrf()
 
 	# LLA to GUA
 	run_cmd_nsb ip -6 addr del ${NSB_IP6}/64 dev ${NSB_DEV}
 	run_cmd_nsb ip -6 ro add ${NSA_IP6}/128 dev ${NSB_DEV}
 	log_start
-	run_cmd nettest -6 -s -D &
-	sleep 1
+	run_cmd nettest -6 -s -D -k
 	run_cmd_nsb nettest -6 -D -r ${NSA_IP6}
 	log_test $? 0 "UDP in - LLA to GUA"
 
 	run_cmd_nsb ip -6 ro del ${NSA_IP6}/128 dev ${NSB_DEV}
 	run_cmd_nsb ip -6 addr add ${NSB_IP6}/64 dev ${NSB_DEV}
@@ -3443,12 +3256,11 @@ ipv6_rt()
 	# server tests
 	#
 	for a in ${NSA_IP6} ${VRF_IP6}
 	do
 		log_start
-		run_cmd nettest ${varg} -s &
-		sleep 1
+		run_cmd nettest ${varg} -s -k
 		run_cmd_nsb nettest ${varg} -r ${a} &
 		sleep 3
 		run_cmd ip link del ${VRF}
 		sleep 1
 		log_test_addr ${a} 0 0 "${desc}, global server"
@@ -3457,12 +3269,11 @@ ipv6_rt()
 	done
 
 	for a in ${NSA_IP6} ${VRF_IP6}
 	do
 		log_start
-		run_cmd nettest ${varg} -I ${VRF} -s &
-		sleep 1
+		run_cmd nettest ${varg} -I ${VRF} -s -k
 		run_cmd_nsb nettest ${varg} -r ${a} &
 		sleep 3
 		run_cmd ip link del ${VRF}
 		sleep 1
 		log_test_addr ${a} 0 0 "${desc}, VRF server"
@@ -3471,12 +3282,11 @@ ipv6_rt()
 	done
 
 	for a in ${NSA_IP6} ${VRF_IP6}
 	do
 		log_start
-		run_cmd nettest ${varg} -I ${NSA_DEV} -s &
-		sleep 1
+		run_cmd nettest ${varg} -I ${NSA_DEV} -s -k
 		run_cmd_nsb nettest ${varg} -r ${a} &
 		sleep 3
 		run_cmd ip link del ${VRF}
 		sleep 1
 		log_test_addr ${a} 0 0 "${desc}, enslaved device server"
@@ -3486,23 +3296,21 @@ ipv6_rt()
 
 	#
 	# client test
 	#
 	log_start
-	run_cmd_nsb nettest ${varg} -s &
-	sleep 1
+	run_cmd_nsb nettest ${varg} -s -k
 	run_cmd nettest ${varg} -d ${VRF} -r ${NSB_IP6} &
 	sleep 3
 	run_cmd ip link del ${VRF}
 	sleep 1
 	log_test  0 0 "${desc}, VRF client"
 
 	setup ${with_vrf}
 
 	log_start
-	run_cmd_nsb nettest ${varg} -s &
-	sleep 1
+	run_cmd_nsb nettest ${varg} -s -k
 	run_cmd nettest ${varg} -d ${NSA_DEV} -r ${NSB_IP6} &
 	sleep 3
 	run_cmd ip link del ${VRF}
 	sleep 1
 	log_test  0 0 "${desc}, enslaved device client"
@@ -3514,12 +3322,11 @@ ipv6_rt()
 	# local address tests
 	#
 	for a in ${NSA_IP6} ${VRF_IP6}
 	do
 		log_start
-		run_cmd nettest ${varg} -s &
-		sleep 1
+		run_cmd nettest ${varg} -s -k
 		run_cmd nettest ${varg} -d ${VRF} -r ${a} &
 		sleep 3
 		run_cmd ip link del ${VRF}
 		sleep 1
 		log_test_addr ${a} 0 0 "${desc}, global server, VRF client"
@@ -3528,12 +3335,11 @@ ipv6_rt()
 	done
 
 	for a in ${NSA_IP6} ${VRF_IP6}
 	do
 		log_start
-		run_cmd nettest ${varg} -I ${VRF} -s &
-		sleep 1
+		run_cmd nettest ${varg} -I ${VRF} -s -k
 		run_cmd nettest ${varg} -d ${VRF} -r ${a} &
 		sleep 3
 		run_cmd ip link del ${VRF}
 		sleep 1
 		log_test_addr ${a} 0 0 "${desc}, VRF server and client"
@@ -3541,34 +3347,31 @@ ipv6_rt()
 		setup ${with_vrf}
 	done
 
 	a=${NSA_IP6}
 	log_start
-	run_cmd nettest ${varg} -s &
-	sleep 1
+	run_cmd nettest ${varg} -s -k
 	run_cmd nettest ${varg} -d ${NSA_DEV} -r ${a} &
 	sleep 3
 	run_cmd ip link del ${VRF}
 	sleep 1
 	log_test_addr ${a} 0 0 "${desc}, global server, device client"
 
 	setup ${with_vrf}
 
 	log_start
-	run_cmd nettest ${varg} -I ${VRF} -s &
-	sleep 1
+	run_cmd nettest ${varg} -I ${VRF} -s -k
 	run_cmd nettest ${varg} -d ${NSA_DEV} -r ${a} &
 	sleep 3
 	run_cmd ip link del ${VRF}
 	sleep 1
 	log_test_addr ${a} 0 0 "${desc}, VRF server, device client"
 
 	setup ${with_vrf}
 
 	log_start
-	run_cmd nettest ${varg} -I ${NSA_DEV} -s &
-	sleep 1
+	run_cmd nettest ${varg} -I ${NSA_DEV} -s -k
 	run_cmd nettest ${varg} -d ${NSA_DEV} -r ${a} &
 	sleep 3
 	run_cmd ip link del ${VRF}
 	sleep 1
 	log_test_addr ${a} 0 0 "${desc}, device server, device client"
@@ -3622,12 +3425,11 @@ netfilter_tcp_reset()
 	local a
 
 	for a in ${NSA_IP} ${VRF_IP}
 	do
 		log_start
-		run_cmd nettest -s &
-		sleep 1
+		run_cmd nettest -s -k
 		run_cmd_nsb nettest -r ${a}
 		log_test_addr ${a} $? 1 "Global server, reject with TCP-reset on Rx"
 	done
 }
 
@@ -3640,12 +3442,11 @@ netfilter_icmp()
 	[ "${stype}" = "UDP" ] && arg="-D"
 
 	for a in ${NSA_IP} ${VRF_IP}
 	do
 		log_start
-		run_cmd nettest ${arg} -s &
-		sleep 1
+		run_cmd nettest ${arg} -s -k
 		run_cmd_nsb nettest ${arg} -r ${a}
 		log_test_addr ${a} $? 1 "Global ${stype} server, Rx reject icmp-port-unreach"
 	done
 }
 
@@ -3679,12 +3480,11 @@ netfilter_tcp6_reset()
 	local a
 
 	for a in ${NSA_IP6} ${VRF_IP6}
 	do
 		log_start
-		run_cmd nettest -6 -s &
-		sleep 1
+		run_cmd nettest -6 -s -k
 		run_cmd_nsb nettest -6 -r ${a}
 		log_test_addr ${a} $? 1 "Global server, reject with TCP-reset on Rx"
 	done
 }
 
@@ -3697,12 +3497,11 @@ netfilter_icmp6()
 	[ "${stype}" = "UDP" ] && arg="$arg -D"
 
 	for a in ${NSA_IP6} ${VRF_IP6}
 	do
 		log_start
-		run_cmd nettest -6 -s ${arg} &
-		sleep 1
+		run_cmd nettest -6 -s ${arg} -k
 		run_cmd_nsb nettest -6 ${arg} -r ${a}
 		log_test_addr ${a} $? 1 "Global ${stype} server, Rx reject icmp-port-unreach"
 	done
 }
 
@@ -3893,17 +3692,15 @@ use_case_snat_on_vrf()
 	local port="12345"
 
 	run_cmd iptables -t nat -A POSTROUTING -p tcp -m tcp --dport ${port} -j SNAT --to-source ${NSA_LO_IP} -o ${VRF}
 	run_cmd ip6tables -t nat -A POSTROUTING -p tcp -m tcp --dport ${port} -j SNAT --to-source ${NSA_LO_IP6} -o ${VRF}
 
-	run_cmd_nsb nettest -s -l ${NSB_IP} -p ${port} &
-	sleep 1
+	run_cmd_nsb nettest -s -l ${NSB_IP} -p ${port} -k
 	run_cmd nettest -d ${VRF} -r ${NSB_IP} -p ${port}
 	log_test $? 0 "IPv4 TCP connection over VRF with SNAT"
 
-	run_cmd_nsb nettest -6 -s -l ${NSB_IP6} -p ${port} &
-	sleep 1
+	run_cmd_nsb nettest -6 -s -l ${NSB_IP6} -p ${port} -k
 	run_cmd nettest -6 -d ${VRF} -r ${NSB_IP6} -p ${port}
 	log_test $? 0 "IPv6 TCP connection over VRF with SNAT"
 
 	# Cleanup
 	run_cmd iptables -t nat -D POSTROUTING -p tcp -m tcp --dport ${port} -j SNAT --to-source ${NSA_LO_IP} -o ${VRF}
-- 
2.25.1

[PATCH 09/11] selftests: nettest: Convert timeout to miliseconds

[Leonard Crestez]

This allows tests to be faster

Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
---
 tools/testing/selftests/net/nettest.c | 52 +++++++++++++++++++++------
 1 file changed, 41 insertions(+), 11 deletions(-)

diff --git a/tools/testing/selftests/net/nettest.c b/tools/testing/selftests/net/nettest.c
index 576d8bb4c94c..eb6c8cf69a74 100644
--- a/tools/testing/selftests/net/nettest.c
+++ b/tools/testing/selftests/net/nettest.c
@@ -125,18 +125,24 @@ struct sock_args {
 	/* ESP in UDP encap test */
 	int use_xfrm;
 };
 
 static int server_mode;
-static unsigned int prog_timeout = 5;
+static unsigned int prog_timeout_ms = 5000;
 static unsigned int interactive;
 static int iter = 1;
 static char *msg = "Hello world!";
 static int msglen;
 static int quiet;
 static int try_broadcast = 1;
 
+static void set_timeval_ms(struct timeval *tv, unsigned long ms)
+{
+	tv->tv_sec = ms / 1000;
+	tv->tv_usec = (ms % 1000) * 1000;
+}
+
 static char *timestamp(char *timebuf, int buflen)
 {
 	time_t now;
 
 	now = time(NULL);
@@ -566,10 +572,29 @@ static int str_to_uint(const char *str, int min, int max, unsigned int *value)
 	}
 
 	return -1;
 }
 
+/* parse seconds with a decimal point as miliseconds */
+static int str_to_msec(const char *str, unsigned int *value)
+{
+	float float_value;
+	char *end;
+
+	float_value = strtof(str, &end);
+
+	/* entire string should be consumed by conversion
+	 * and value should be between min and max
+	 */
+	if (((*end == '\0') || (*end == '\n')) && (end != str)) {
+		*value = float_value * 1000;
+		return 0;
+	}
+
+	return -1;
+}
+
 static int resolve_devices(struct sock_args *args)
 {
 	if (args->dev) {
 		args->ifindex = get_ifidx(args->dev);
 		if (args->ifindex < 0) {
@@ -1165,11 +1190,11 @@ static void set_recv_attr(int sd, int version)
 }
 
 static int msg_loop(int client, int sd, void *addr, socklen_t alen,
 		    struct sock_args *args)
 {
-	struct timeval timeout = { .tv_sec = prog_timeout }, *ptval = NULL;
+	struct timeval timeout, *ptval = NULL;
 	fd_set rfds;
 	int nfds;
 	int rc;
 
 	if (args->type != SOCK_STREAM)
@@ -1182,13 +1207,15 @@ static int msg_loop(int client, int sd, void *addr, socklen_t alen,
 		if (client) {
 			if (send_msg(sd, addr, alen, args))
 				return 1;
 		}
 		if (!interactive) {
+			if (!prog_timeout_ms)
+				set_timeval_ms(&timeout, 5000);
+			else
+				set_timeval_ms(&timeout, prog_timeout_ms);
 			ptval = &timeout;
-			if (!prog_timeout)
-				timeout.tv_sec = 5;
 		}
 	}
 
 	nfds = interactive ? MAX(fileno(stdin), sd)  + 1 : sd + 1;
 	while (1) {
@@ -1479,11 +1506,11 @@ static void ipc_write(int fd, int message)
 }
 
 static int do_server(struct sock_args *args, int ipc_fd)
 {
 	/* ipc_fd = -1 if no parent process to signal */
-	struct timeval timeout = { .tv_sec = prog_timeout }, *ptval = NULL;
+	struct timeval timeout, *ptval = NULL;
 	unsigned char addr[sizeof(struct sockaddr_in6)] = {};
 	socklen_t alen = sizeof(addr);
 	int lsd, csd = -1;
 
 	fd_set rfds;
@@ -1501,12 +1528,14 @@ static int do_server(struct sock_args *args, int ipc_fd)
 	args->dev = args->server_dev;
 	args->expected_dev = args->expected_server_dev;
 	if (resolve_devices(args) || validate_addresses(args))
 		goto err_exit;
 
-	if (prog_timeout)
+	if (prog_timeout_ms) {
+		set_timeval_ms(&timeout, prog_timeout_ms);
 		ptval = &timeout;
+	}
 
 	if (args->has_grp)
 		lsd = msock_server(args);
 	else
 		lsd = lsock_init(args);
@@ -1584,20 +1613,22 @@ static int do_server(struct sock_args *args, int ipc_fd)
 	return 1;
 }
 
 static int wait_for_connect(int sd)
 {
-	struct timeval _tv = { .tv_sec = prog_timeout }, *tv = NULL;
+	struct timeval _tv, *tv = NULL;
 	fd_set wfd;
 	int val = 0, sz = sizeof(val);
 	int rc;
 
 	FD_ZERO(&wfd);
 	FD_SET(sd, &wfd);
 
-	if (prog_timeout)
+	if (prog_timeout_ms) {
+		set_timeval_ms(&_tv, prog_timeout_ms);
 		tv = &_tv;
+	}
 
 	rc = select(FD_SETSIZE, NULL, &wfd, NULL, tv);
 	if (rc == 0) {
 		log_error("connect timed out\n");
 		return -2;
@@ -1945,12 +1976,11 @@ int main(int argc, char *argv[])
 				return 1;
 			}
 			args.port = (unsigned short) tmp;
 			break;
 		case 't':
-			if (str_to_uint(optarg, 0, INT_MAX,
-					&prog_timeout) != 0) {
+			if (str_to_msec(optarg, &prog_timeout_ms) != 0) {
 				fprintf(stderr, "Invalid timeout\n");
 				return 1;
 			}
 			break;
 		case 'D':
@@ -2091,11 +2121,11 @@ int main(int argc, char *argv[])
 			"Fork after listen only supported for server mode\n");
 		return 1;
 	}
 
 	if (interactive) {
-		prog_timeout = 0;
+		prog_timeout_ms = 0;
 		msg = NULL;
 	}
 
 	if (both_mode) {
 		if (pipe(fd) < 0) {
-- 
2.25.1

[PATCH 10/11] selftests: nettest: Add NETTEST_CLIENT,SERVER}_TIMEOUT envvars

[Leonard Crestez]

Move the single "prog_timeout_ms" into sock_args and split into client
and server timeouts.

Add NETTEST_CLIENT_TIMEOUT and NETTEST_SERVER_TIMEOUT which can set a
default value different than the default of 5 seconds.

This allows exporting NETTEST_CLIENT_TIMEOUT=0.1 and running all of
fcnal-test.sh quickly.

A reduced server timeout is less useful, most tests would work fine with
an infinite timeout because nettest is launched in the background and
killed explicitly.

Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
---
 tools/testing/selftests/net/nettest.c | 66 ++++++++++++++++++++++-----
 1 file changed, 54 insertions(+), 12 deletions(-)

diff --git a/tools/testing/selftests/net/nettest.c b/tools/testing/selftests/net/nettest.c
index eb6c8cf69a74..bc5976f842f9 100644
--- a/tools/testing/selftests/net/nettest.c
+++ b/tools/testing/selftests/net/nettest.c
@@ -122,14 +122,16 @@ struct sock_args {
 		struct in6_addr in6;
 	} expected_raddr;
 
 	/* ESP in UDP encap test */
 	int use_xfrm;
+
+	unsigned int client_timeout_ms;
+	unsigned int server_timeout_ms;
 };
 
 static int server_mode;
-static unsigned int prog_timeout_ms = 5000;
 static unsigned int interactive;
 static int iter = 1;
 static char *msg = "Hello world!";
 static int msglen;
 static int quiet;
@@ -1207,14 +1209,21 @@ static int msg_loop(int client, int sd, void *addr, socklen_t alen,
 		if (client) {
 			if (send_msg(sd, addr, alen, args))
 				return 1;
 		}
 		if (!interactive) {
-			if (!prog_timeout_ms)
+			unsigned int timeout_ms;
+
+			if (client)
+				timeout_ms = args->client_timeout_ms;
+			else
+				timeout_ms = args->server_timeout_ms;
+
+			if (!timeout_ms)
 				set_timeval_ms(&timeout, 5000);
 			else
-				set_timeval_ms(&timeout, prog_timeout_ms);
+				set_timeval_ms(&timeout, timeout_ms);
 			ptval = &timeout;
 		}
 	}
 
 	nfds = interactive ? MAX(fileno(stdin), sd)  + 1 : sd + 1;
@@ -1528,12 +1537,12 @@ static int do_server(struct sock_args *args, int ipc_fd)
 	args->dev = args->server_dev;
 	args->expected_dev = args->expected_server_dev;
 	if (resolve_devices(args) || validate_addresses(args))
 		goto err_exit;
 
-	if (prog_timeout_ms) {
-		set_timeval_ms(&timeout, prog_timeout_ms);
+	if (args->server_timeout_ms) {
+		set_timeval_ms(&timeout, args->server_timeout_ms);
 		ptval = &timeout;
 	}
 
 	if (args->has_grp)
 		lsd = msock_server(args);
@@ -1611,22 +1620,22 @@ static int do_server(struct sock_args *args, int ipc_fd)
 err_exit:
 	ipc_write(ipc_fd, 0);
 	return 1;
 }
 
-static int wait_for_connect(int sd)
+static int wait_for_connect(int sd, struct sock_args *args)
 {
 	struct timeval _tv, *tv = NULL;
 	fd_set wfd;
 	int val = 0, sz = sizeof(val);
 	int rc;
 
 	FD_ZERO(&wfd);
 	FD_SET(sd, &wfd);
 
-	if (prog_timeout_ms) {
-		set_timeval_ms(&_tv, prog_timeout_ms);
+	if (args->client_timeout_ms) {
+		set_timeval_ms(&_tv, args->client_timeout_ms);
 		tv = &_tv;
 	}
 
 	rc = select(FD_SETSIZE, NULL, &wfd, NULL, tv);
 	if (rc == 0) {
@@ -1692,11 +1701,11 @@ static int connectsock(void *addr, socklen_t alen, struct sock_args *args)
 		if (errno != EINPROGRESS) {
 			log_err_errno("Failed to connect to remote host");
 			rc = -1;
 			goto err;
 		}
-		rc = wait_for_connect(sd);
+		rc = wait_for_connect(sd, args);
 		if (rc < 0)
 			goto err;
 	}
 out:
 	return sd;
@@ -1883,11 +1892,11 @@ static void print_usage(char *prog)
 	"Required:\n"
 	"    -r addr       remote address to connect to (client mode only)\n"
 	"    -p port       port to connect to (client mode)/listen on (server mode)\n"
 	"                  (default: %d)\n"
 	"    -s            server mode (default: client mode)\n"
-	"    -t            timeout seconds (default: none)\n"
+	"    -t seconds    timeout seconds for both client and server (default: 5.000)\n"
 	"\n"
 	"Optional:\n"
 	"    -B            do both client and server via fork and IPC\n"
 	"    -N ns         set client to network namespace ns (requires root)\n"
 	"    -O ns         set server to network namespace ns (requires root)\n"
@@ -1920,19 +1929,46 @@ static void print_usage(char *prog)
 	"    -3 dev        Expected device name (or index) to receive packets - server mode\n"
 	"\n"
 	"    -b            Bind test only.\n"
 	"    -q            Be quiet. Run test without printing anything.\n"
 	"    -k            Fork server in background after bind or listen.\n"
+	"\n"
+	"Environment Variables:"
+	"\n"
+	"NETTEST_CLIENT_TIMEOUT:  timeouts in seconds for client"
+	"NETTEST_SERVER_TIMEOUT:  timeouts in seconds for server"
 	, prog, DEFAULT_PORT);
 }
 
+int parse_env(struct sock_args *args)
+{
+	const char *str;
+
+	if ((str = getenv("NETTEST_CLIENT_TIMEOUT"))) {
+		if (str_to_msec(str, &args->client_timeout_ms) != 0) {
+			fprintf(stderr, "Invalid NETTEST_CLIENT_TIMEOUT\n");
+			return 1;
+		}
+	}
+	if ((str = getenv("NETTEST_SERVER_TIMEOUT"))) {
+		if (str_to_msec(str, &args->server_timeout_ms) != 0) {
+			fprintf(stderr, "Invalid NETTEST_SERVER_TIMEOUT\n");
+			return 1;
+		}
+	}
+
+	return 0;
+}
+
 int main(int argc, char *argv[])
 {
 	struct sock_args args = {
 		.version = AF_INET,
 		.type    = SOCK_STREAM,
 		.port    = DEFAULT_PORT,
+		.client_timeout_ms = 5000,
+		.server_timeout_ms = 5000,
 	};
 	struct protoent *pe;
 	int both_mode = 0;
 	unsigned int tmp;
 	int forever = 0;
@@ -1941,10 +1977,14 @@ int main(int argc, char *argv[])
 
 	/* process inputs */
 	extern char *optarg;
 	int rc = 0;
 
+	rc = parse_env(&args);
+	if (rc)
+		return rc;
+
 	/*
 	 * process input args
 	 */
 
 	while ((rc = getopt(argc, argv, GETOPT_STR)) != -1) {
@@ -1976,14 +2016,15 @@ int main(int argc, char *argv[])
 				return 1;
 			}
 			args.port = (unsigned short) tmp;
 			break;
 		case 't':
-			if (str_to_msec(optarg, &prog_timeout_ms) != 0) {
+			if (str_to_msec(optarg, &args.client_timeout_ms) != 0) {
 				fprintf(stderr, "Invalid timeout\n");
 				return 1;
 			}
+			args.server_timeout_ms = args.client_timeout_ms;
 			break;
 		case 'D':
 			args.type = SOCK_DGRAM;
 			break;
 		case 'R':
@@ -2121,11 +2162,12 @@ int main(int argc, char *argv[])
 			"Fork after listen only supported for server mode\n");
 		return 1;
 	}
 
 	if (interactive) {
-		prog_timeout_ms = 0;
+		args.client_timeout_ms = 0;
+		args.server_timeout_ms = 0;
 		msg = NULL;
 	}
 
 	if (both_mode) {
 		if (pipe(fd) < 0) {
-- 
2.25.1

[PATCH 11/11] selftests: net/fcnal: Reduce client timeout

[Leonard Crestez]

Reduce default client timeout from 5 seconds to 500 miliseconds.
Can be overridden from environment by exporting NETTEST_CLIENT_TIMEOUT=5

Some tests need ICMP timeouts so pass an explicit -t5 for those.

Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
---
 tools/testing/selftests/net/fcnal-test.sh | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/tools/testing/selftests/net/fcnal-test.sh b/tools/testing/selftests/net/fcnal-test.sh
index e73aeb3884c5..cf5dd96bb9db 100755
--- a/tools/testing/selftests/net/fcnal-test.sh
+++ b/tools/testing/selftests/net/fcnal-test.sh
@@ -40,10 +40,15 @@
 # Kselftest framework requirement - SKIP code is 4.
 ksft_skip=4
 
 VERBOSE=0
 
+# Use a reduced client timeout by default
+# Can be overridden by user
+NETTEST_CLIENT_TIMEOUT=${NETTEST_CLIENT_TIMEOUT:-0.5}
+export NETTEST_CLIENT_TIMEOUT
+
 NSA_DEV=eth1
 NSA_DEV2=eth2
 NSB_DEV=eth1
 NSC_DEV=eth2
 VRF=red
@@ -1076,11 +1081,11 @@ ipv4_tcp_novrf()
 	for a in ${NSA_LO_IP} 127.0.0.1
 	do
 		log_start
 		show_hint "Should fail 'No route to host' since addresses on loopback are out of device scope"
 		run_cmd nettest -s -k
-		run_cmd nettest -r ${a} -d ${NSA_DEV}
+		run_cmd nettest -r ${a} -d ${NSA_DEV} -t5
 		log_test_addr ${a} $? 1 "Global server, device client, local connection"
 	done
 
 	a=${NSA_IP}
 	log_start
@@ -1379,23 +1384,23 @@ ipv4_udp_novrf()
 	for a in ${NSA_LO_IP} 127.0.0.1
 	do
 		log_start
 		show_hint "Should fail since addresses on loopback are out of device scope"
 		run_cmd nettest -D -s -k
-		run_cmd nettest -D -r ${a} -d ${NSA_DEV}
+		run_cmd nettest -D -r ${a} -d ${NSA_DEV} -t5
 		log_test_addr ${a} $? 2 "Global server, device client, local connection"
 
 		log_start
 		show_hint "Should fail since addresses on loopback are out of device scope"
 		run_cmd nettest -D -s -k
-		run_cmd nettest -D -r ${a} -d ${NSA_DEV} -C
+		run_cmd nettest -D -r ${a} -d ${NSA_DEV} -C -t5
 		log_test_addr ${a} $? 1 "Global server, device send via cmsg, local connection"
 
 		log_start
 		show_hint "Should fail since addresses on loopback are out of device scope"
 		run_cmd nettest -D -s -k
-		run_cmd nettest -D -r ${a} -d ${NSA_DEV} -S
+		run_cmd nettest -D -r ${a} -d ${NSA_DEV} -S -t5
 		log_test_addr ${a} $? 1 "Global server, device client via IP_UNICAST_IF, local connection"
 	done
 
 	a=${NSA_IP}
 	log_start
@@ -3443,11 +3448,11 @@ netfilter_icmp()
 
 	for a in ${NSA_IP} ${VRF_IP}
 	do
 		log_start
 		run_cmd nettest ${arg} -s -k
-		run_cmd_nsb nettest ${arg} -r ${a}
+		run_cmd_nsb nettest ${arg} -r ${a} -t5
 		log_test_addr ${a} $? 1 "Global ${stype} server, Rx reject icmp-port-unreach"
 	done
 }
 
 ipv4_netfilter()
@@ -3498,11 +3503,11 @@ netfilter_icmp6()
 
 	for a in ${NSA_IP6} ${VRF_IP6}
 	do
 		log_start
 		run_cmd nettest -6 -s ${arg} -k
-		run_cmd_nsb nettest -6 ${arg} -r ${a}
+		run_cmd_nsb nettest -6 ${arg} -r ${a} -t5
 		log_test_addr ${a} $? 1 "Global ${stype} server, Rx reject icmp-port-unreach"
 	done
 }
 
 ipv6_netfilter()
-- 
2.25.1

Re: [PATCH 01/11] selftests: net/fcnal: Fix {ipv4,ipv6}_bind not run by default

[David Ahern]

On 10/6/21 5:47 AM, Leonard Crestez wrote:
> Inside the TESTS_IPV{4,6} array these are listed as ipv{4,6}_addr_bind
> but test running loop only checks for ipv{4,6}_bind and bind{,6} and
> ignores everything else.
> 
> As a consequence these tests were not run by default, only with an
> explicit -t.
> 
> Fix inside TESTS_IPV{4,6}.
> 
> Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
> ---
>  tools/testing/selftests/net/fcnal-test.sh | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 

Good catch.

Reviewed-by: David Ahern <dsahern@kernel.org>

Re: [PATCH 02/11] selftests: net/fcnal: Mark unknown -t or TESTS value as error

[David Ahern]

On 10/6/21 5:47 AM, Leonard Crestez wrote:
> Right now unknown values are completely ignored which is very confusing
> and lead to a subset of tests being skipped because of a mispelling.
> 
> Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
> ---
>  tools/testing/selftests/net/fcnal-test.sh | 3 +++
>  1 file changed, 3 insertions(+)
> 

Reviewed-by: David Ahern <dsahern@kernel.org>

Re: [PATCH 03/11] selftests: net/fcnal: Non-zero exit on failures

[David Ahern]

On 10/6/21 5:47 AM, Leonard Crestez wrote:
> Test scripts must report 'pass' or 'fail' via exit status, not just
> logging.
> 
> Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
> ---
>  tools/testing/selftests/net/fcnal-test.sh | 1 +
>  1 file changed, 1 insertion(+)
> 

Reviewed-by: David Ahern <dsahern@kernel.org>

Re: [PATCH 04/11] selftests: net/fcnal: Use accept_dad=0 to avoid setup sleep

[David Ahern]

On 10/6/21 5:47 AM, Leonard Crestez wrote:
> Duplicate Address Detection makes ipv6 addresses unavailable for a short
> period after adding (average about 1 second). Adding sleep statements
> avoid this but since all addresses in the test environment are
> controlled from the same source we can just disable DAD for the entire
> namespace.
> 
> Unlike sprinkling nodad to all ipv6 address additions this also skips
> DAD for link-local-addresses.
> 
> Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
> ---
>  tools/testing/selftests/net/fcnal-test.sh | 17 +++++++----------
>  1 file changed, 7 insertions(+), 10 deletions(-)
> 

Reviewed-by: David Ahern <dsahern@kernel.org>

Re: [PATCH 05/11] selftests: net/fcnal: kill_procs via spin instead of sleep

[David Ahern]

On 10/6/21 5:47 AM, Leonard Crestez wrote:
> Sleeping for one second after a kill is not necessary and adds up quite
> quickly. Replace with a fast loop spinning until pidof returns nothing.
> 
> Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
> ---
>  tools/testing/selftests/net/fcnal-test.sh | 11 +++++++++--
>  1 file changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/tools/testing/selftests/net/fcnal-test.sh b/tools/testing/selftests/net/fcnal-test.sh
> index 0bd60cd3bc06..b7fda51deb3f 100755
> --- a/tools/testing/selftests/net/fcnal-test.sh
> +++ b/tools/testing/selftests/net/fcnal-test.sh
> @@ -176,12 +176,19 @@ show_hint()
>  	fi
>  }
>  
>  kill_procs()
>  {
> -	killall nettest ping ping6 >/dev/null 2>&1
> -	sleep 1
> +	local pids
> +	while true; do
> +		pids=$(pidof nettest ping ping6)
> +		if [[ -z $pids ]]; then
> +			break
> +		fi
> +		kill $pids
> +		sleep 0.01
> +	done
>  }
>  
>  do_run_cmd()
>  {
>  	local cmd="$*"
> 

ideally the script keeps track of processes it launches and only kills
those. The original killall was just a stop gap until the process
tracking was added.

Re: [PATCH 06/11] selftests: net/fcnal: Do not capture do_run_cmd in verbose mode

[David Ahern]

On 10/6/21 5:47 AM, Leonard Crestez wrote:
> This way running with -v will show interspersed output from both nettest
> client and server. This helps to identify the order of events.
> 
> It is also required in order to make nettest fork in the background by
> itself because shell capturing does not stop if the target forks.
> 
> This also fixes SC2166 warning:
> Prefer [ p ] && [ q ] as [ p -a q ] is not well defined.
> 
> Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
> ---
>  tools/testing/selftests/net/fcnal-test.sh | 16 +++++++---------
>  1 file changed, 7 insertions(+), 9 deletions(-)
> 

Reviewed-by: David Ahern <dsahern@kernel.org>

Re: [PATCH 08/11] selftests: net/fcnal: Replace sleep after server start with -k

[David Ahern]

On 10/6/21 5:47 AM, Leonard Crestez wrote:
> The -k switch makes the server fork into the background after the listen
> call is succesful, this can be used to replace most of the `sleep 1`
> statements in this script.
> 
> Change performed with a vim command:
> 
> s/nettest \(.*-s.*\) &\n\s*sleep 1\n/nettest \1 -k\r
> 
> Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
> ---
>  tools/testing/selftests/net/fcnal-test.sh | 641 ++++++++--------------
>  1 file changed, 219 insertions(+), 422 deletions(-)
> 

I have a change from January [1] that runs the tests with 1 binary -
takes both client and server side arguments, does the server setup,
switches namespaces as needed and then runs the client side. I got
bogged down validating the before and after which takes a long time
given the number of tests. The output in verbose mode is as important as
the pass / fail. Many of the tests document existing behavior as well as
intended behavior.

You used a search and replace to update the tests. Did you then do the
compare of test results - not pass / fail but output?


[1]
https://github.com/dsahern/linux/commit/8e16fbab1eb224298e3324a9ddf38e27eee439c7

Re: [PATCH 09/11] selftests: nettest: Convert timeout to miliseconds

[David Ahern]

On 10/6/21 5:47 AM, Leonard Crestez wrote:
> This allows tests to be faster
> 
> Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
> ---
>  tools/testing/selftests/net/nettest.c | 52 +++++++++++++++++++++------
>  1 file changed, 41 insertions(+), 11 deletions(-)
> 

Reviewed-by: David Ahern <dsahern@kernel.org>

Re: [PATCH 10/11] selftests: nettest: Add NETTEST_CLIENT,SERVER}_TIMEOUT envvars

[David Ahern]

On 10/6/21 5:47 AM, Leonard Crestez wrote:
> Move the single "prog_timeout_ms" into sock_args and split into client
> and server timeouts.
> 
> Add NETTEST_CLIENT_TIMEOUT and NETTEST_SERVER_TIMEOUT which can set a
> default value different than the default of 5 seconds.
> 
> This allows exporting NETTEST_CLIENT_TIMEOUT=0.1 and running all of
> fcnal-test.sh quickly.

The command takes command line arguments; no need to make 2 special
environment variables.

> 
> A reduced server timeout is less useful, most tests would work fine with
> an infinite timeout because nettest is launched in the background and
> killed explicitly.

The server timeout was only for cleanup (the tests have a very long
history); given the kill on processes launched the server could just
wait forever.

> 
> Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
> ---
>  tools/testing/selftests/net/nettest.c | 66 ++++++++++++++++++++++-----
>  1 file changed, 54 insertions(+), 12 deletions(-)
> 

Re: [PATCH 11/11] selftests: net/fcnal: Reduce client timeout

[David Ahern]

On 10/6/21 5:47 AM, Leonard Crestez wrote:
> Reduce default client timeout from 5 seconds to 500 miliseconds.
> Can be overridden from environment by exporting NETTEST_CLIENT_TIMEOUT=5
> 
> Some tests need ICMP timeouts so pass an explicit -t5 for those.
> 
> Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
> ---
>  tools/testing/selftests/net/fcnal-test.sh | 17 +++++++++++------
>  1 file changed, 11 insertions(+), 6 deletions(-)
> 

The problem with blindly reducing the timeouts is running the script on
a loaded server. Some tests are expected to timeout while for tests a
timeout is a failure.

Re: [PATCH 05/11] selftests: net/fcnal: kill_procs via spin instead of sleep

[Leonard Crestez]

On 06.10.2021 17:45, David Ahern wrote:
> On 10/6/21 5:47 AM, Leonard Crestez wrote:
>> Sleeping for one second after a kill is not necessary and adds up quite
>> quickly. Replace with a fast loop spinning until pidof returns nothing.
>>
>> Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
>> ---
>>   tools/testing/selftests/net/fcnal-test.sh | 11 +++++++++--
>>   1 file changed, 9 insertions(+), 2 deletions(-)
>>
>> diff --git a/tools/testing/selftests/net/fcnal-test.sh b/tools/testing/selftests/net/fcnal-test.sh
>> index 0bd60cd3bc06..b7fda51deb3f 100755
>> --- a/tools/testing/selftests/net/fcnal-test.sh
>> +++ b/tools/testing/selftests/net/fcnal-test.sh
>> @@ -176,12 +176,19 @@ show_hint()
>>   	fi
>>   }
>>   
>>   kill_procs()
>>   {
>> -	killall nettest ping ping6 >/dev/null 2>&1
>> -	sleep 1
>> +	local pids
>> +	while true; do
>> +		pids=$(pidof nettest ping ping6)
>> +		if [[ -z $pids ]]; then
>> +			break
>> +		fi
>> +		kill $pids
>> +		sleep 0.01
>> +	done
>>   }
>>   
>>   do_run_cmd()
>>   {
>>   	local cmd="$*"
>>
> 
> ideally the script keeps track of processes it launches and only kills
> those. The original killall was just a stop gap until the process
> tracking was added.

That's harder to do. This is much faster and not in any way worse than 
killall + sleep.

Some sort of a wrapper would have to added for each process running the 
background, for each run_ping_bg.

If nettest forks by itself then $! won't work, maybe some sort of 
--pid-file switch would be required?

--
Regards,
Leonard

Re: [PATCH 11/11] selftests: net/fcnal: Reduce client timeout

[Leonard Crestez]

On 06.10.2021 18:01, David Ahern wrote:
> On 10/6/21 5:47 AM, Leonard Crestez wrote:
>> Reduce default client timeout from 5 seconds to 500 miliseconds.
>> Can be overridden from environment by exporting NETTEST_CLIENT_TIMEOUT=5
>>
>> Some tests need ICMP timeouts so pass an explicit -t5 for those.
>>
>> Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
>> ---
>>   tools/testing/selftests/net/fcnal-test.sh | 17 +++++++++++------
>>   1 file changed, 11 insertions(+), 6 deletions(-)
>>
> 
> The problem with blindly reducing the timeouts is running the script on
> a loaded server. Some tests are expected to timeout while for tests a
> timeout is a failure.

Keeping the default value "5" would be fine as long as it is possible to 
override externally and get fast results on a mostly-idle machine.

Placing a default value in the environment which is overriden by certain 
tests achieves that.

In theory it would also be possible for fcnal-test.sh to parse as 
"--timeout" option and pass it into every single test but that solution 
would cause much more code churn.

Having default values in environment variables that can still be 
overridden by command-line arguments is a common pattern in many tools. 
It also avoids having to pass-through every flag through every 
intermediate wrapper.

--
Regards,
Leonard

Re: [PATCH 08/11] selftests: net/fcnal: Replace sleep after server start with -k

[Leonard Crestez]

On 06.10.2021 17:54, David Ahern wrote:
> On 10/6/21 5:47 AM, Leonard Crestez wrote:
>> The -k switch makes the server fork into the background after the listen
>> call is succesful, this can be used to replace most of the `sleep 1`
>> statements in this script.
>>
>> Change performed with a vim command:
>>
>> s/nettest \(.*-s.*\) &\n\s*sleep 1\n/nettest \1 -k\r
>>
>> Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
>> ---
>>   tools/testing/selftests/net/fcnal-test.sh | 641 ++++++++--------------
>>   1 file changed, 219 insertions(+), 422 deletions(-)
>>
> 
> I have a change from January [1] that runs the tests with 1 binary -
> takes both client and server side arguments, does the server setup,
> switches namespaces as needed and then runs the client side. I got
> bogged down validating the before and after which takes a long time
> given the number of tests. The output in verbose mode is as important as
> the pass / fail. Many of the tests document existing behavior as well as
> intended behavior.
> 
> You used a search and replace to update the tests. Did you then do the
> compare of test results - not pass / fail but output?

I counted the [FAIL] or [ OK ] markers but not the output of nettest 
itself. I don't know what to look for, I guess I could diff the outputs?

Shouldn't it be sufficient to compare the exit codes of the nettest client?

The output is also modified by a previous change to not capture server 
output separately and instead let it be combined with that of the 
client. That change is required for this one, doing out=$(nettest -k) 
does not return on fork unless the pipe is also closed.

I did not look at your change, mine is relatively minimal because it 
only changes who decide when the server goes into the background: the 
shell script or the server itself. This makes it work very easily even 
for tests with multiple server instances.

--
Regards,
Leonard

Re: [PATCH 11/11] selftests: net/fcnal: Reduce client timeout

[David Ahern]

On 10/6/21 3:26 PM, Leonard Crestez wrote:
> 
> 
> On 06.10.2021 18:01, David Ahern wrote:
>> On 10/6/21 5:47 AM, Leonard Crestez wrote:
>>> Reduce default client timeout from 5 seconds to 500 miliseconds.
>>> Can be overridden from environment by exporting NETTEST_CLIENT_TIMEOUT=5
>>>
>>> Some tests need ICMP timeouts so pass an explicit -t5 for those.
>>>
>>> Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
>>> ---
>>>   tools/testing/selftests/net/fcnal-test.sh | 17 +++++++++++------
>>>   1 file changed, 11 insertions(+), 6 deletions(-)
>>>
>>
>> The problem with blindly reducing the timeouts is running the script on
>> a loaded server. Some tests are expected to timeout while for tests a
>> timeout is a failure.
> 
> Keeping the default value "5" would be fine as long as it is possible to
> override externally and get fast results on a mostly-idle machine.

5 is the default for nettest.c; the test script passes in -t1 for all tests.

> 
> Placing a default value in the environment which is overriden by certain
> tests achieves that.
> 
> In theory it would also be possible for fcnal-test.sh to parse as
> "--timeout" option and pass it into every single test but that solution
> would cause much more code churn.
> 
> Having default values in environment variables that can still be
> overridden by command-line arguments is a common pattern in many tools.
> It also avoids having to pass-through every flag through every
> intermediate wrapper.

I do not agree with env variables here.

Re: [PATCH 08/11] selftests: net/fcnal: Replace sleep after server start with -k

[David Ahern]

On 10/6/21 3:35 PM, Leonard Crestez wrote:
> 
> I counted the [FAIL] or [ OK ] markers but not the output of nettest
> itself. I don't know what to look for, I guess I could diff the outputs?
> 
> Shouldn't it be sufficient to compare the exit codes of the nettest client?

mistakes happen. The 700+ tests that exist were verified by me when I
submitted the script - that each test passes when it should and fails
when it should. "FAIL" has many reasons. I tried to have separate exit
codes for nettest.c to capture the timeouts vs ECONNREFUSED, etc., but I
could easily have made a mistake. scanning the output is the best way.
Most of the 'supposed to fail' tests have a HINT saying why it should fail.

> 
> The output is also modified by a previous change to not capture server
> output separately and instead let it be combined with that of the
> client. That change is required for this one, doing out=$(nettest -k)
> does not return on fork unless the pipe is also closed.
> 
> I did not look at your change, mine is relatively minimal because it
> only changes who decide when the server goes into the background: the
> shell script or the server itself. This makes it work very easily even
> for tests with multiple server instances.

The logging issue is why I went with 1 binary do both server and client
after nettest.c got support for changing namespaces.

Re: [PATCH 11/11] selftests: net/fcnal: Reduce client timeout

[Leonard Crestez]

On 07.10.2021 04:17, David Ahern wrote:
> On 10/6/21 3:26 PM, Leonard Crestez wrote:
>> On 06.10.2021 18:01, David Ahern wrote:
>>> On 10/6/21 5:47 AM, Leonard Crestez wrote:
>>>> Reduce default client timeout from 5 seconds to 500 miliseconds.
>>>> Can be overridden from environment by exporting NETTEST_CLIENT_TIMEOUT=5
>>>>
>>>> Some tests need ICMP timeouts so pass an explicit -t5 for those.
>>>>
>>>> Signed-off-by: Leonard Crestez <cdleonard@gmail.com>
>>>> ---
>>>>    tools/testing/selftests/net/fcnal-test.sh | 17 +++++++++++------
>>>>    1 file changed, 11 insertions(+), 6 deletions(-)
>>>>
>>>
>>> The problem with blindly reducing the timeouts is running the script on
>>> a loaded server. Some tests are expected to timeout while for tests a
>>> timeout is a failure.
>>
>> Keeping the default value "5" would be fine as long as it is possible to
>> override externally and get fast results on a mostly-idle machine.
> 
> 5 is the default for nettest.c; the test script passes in -t1 for all tests.

An explicit -t is only passed for some of the tests

$ grep -c nettest.*-r tools/testing/selftests/net/fcnal-test.sh
243
$ grep -c nettest.*-t tools/testing/selftests/net/fcnal-test.sh
15

>> Placing a default value in the environment which is overriden by certain
>> tests achieves that.
>>
>> In theory it would also be possible for fcnal-test.sh to parse as
>> "--timeout" option and pass it into every single test but that solution
>> would cause much more code churn.
>>
>> Having default values in environment variables that can still be
>> overridden by command-line arguments is a common pattern in many tools.
>> It also avoids having to pass-through every flag through every
>> intermediate wrapper.
> 
> I do not agree with env variables here.

Would you agree with adding an option to fcnal-test.sh which decreases 
timeouts passed to nettest client calls?

Re: [PATCH 11/11] selftests: net/fcnal: Reduce client timeout

[David Ahern]

On 10/7/21 2:52 PM, Leonard Crestez wrote:
> Would you agree with adding an option to fcnal-test.sh which decreases
> timeouts passed to nettest client calls?

sure

Re: [PATCH 08/11] selftests: net/fcnal: Replace sleep after server start with -k

[Leonard Crestez]

On 10/7/21 4:22 AM, David Ahern wrote:
> On 10/6/21 3:35 PM, Leonard Crestez wrote:
>>
>> I counted the [FAIL] or [ OK ] markers but not the output of nettest
>> itself. I don't know what to look for, I guess I could diff the outputs?
>>
>> Shouldn't it be sufficient to compare the exit codes of the nettest client?
> 
> mistakes happen. The 700+ tests that exist were verified by me when I
> submitted the script - that each test passes when it should and fails
> when it should. "FAIL" has many reasons. I tried to have separate exit
> codes for nettest.c to capture the timeouts vs ECONNREFUSED, etc., but I
> could easily have made a mistake. scanning the output is the best way.
> Most of the 'supposed to fail' tests have a HINT saying why it should fail.

It is not good to have a test for which correctness is ambiguous to such 
an extent, it makes reliable future changes difficult. In theory an 
uniform TAP format is supposed to solve this but it is not applied 
inside selftests/net.

I attempted to write a script to compare two logs in their current 
format: https://gitlab.com/cdleonard/kselftest-parse-nettest-fcnal

It does a bunch of nasty scrubbing of irrelevant behavior and got it to 
the point where no diffs are found between repeated runs on the same 
machine.

One nasty issue was that many tests kill processes inside log_test so 
relevant output may be shown either before or after the "TEST: " result 
line. This was solved by associating output until the next ### with 
previous test.

>> The output is also modified by a previous change to not capture server
>> output separately and instead let it be combined with that of the
>> client. That change is required for this one, doing out=$(nettest -k)
>> does not return on fork unless the pipe is also closed.
>>
>> I did not look at your change, mine is relatively minimal because it
>> only changes who decide when the server goes into the background: the
>> shell script or the server itself. This makes it work very easily even
>> for tests with multiple server instances.
> 
> The logging issue is why I went with 1 binary do both server and client
> after nettest.c got support for changing namespaces.

It's possible to just compare the "client" and "server" logs separately 
by sorting them on their prefix.

I think a decent approach would be to do a bulk replace for all 
"run_cmd{,_nsb,_nsc} nettest" with a new "run_nettest" function that 
passes all arguments to nettest itself. That run_nettest function could 
include a leading common "-t" arg that is parsed at the top of 
fcnal-test.sh.

--
Regards,
Leonard