* [net-next PATCH 1/4] samples/pktgen: Add some helper functions
2017-11-01 10:41 [net-next PATCH 0/4] Updates for samples/pktgen Jesper Dangaard Brouer
@ 2017-11-01 10:41 ` Jesper Dangaard Brouer
2017-11-01 10:41 ` [net-next PATCH 2/4] samples/pktgen: add script pktgen_sample06_numa_awared_queue_irq_affinity.sh Jesper Dangaard Brouer
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Jesper Dangaard Brouer @ 2017-11-01 10:41 UTC (permalink / raw)
To: netdev
Cc: Robert Olsson, Jesper Dangaard Brouer, robert.hu, pabeni, kafai, tariqt
From: Robert Hoo <robert.hu@intel.com>
1. given a device, get its NUMA belongings
2. given a device, get its queues' irq numbers.
3. given a NUMA node, get its cpu id list.
Signed-off-by: Robert Hoo <robert.hu@intel.com>
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
samples/pktgen/functions.sh | 43 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 43 insertions(+)
diff --git a/samples/pktgen/functions.sh b/samples/pktgen/functions.sh
index 205e4cde4601..f8bb3cd0f4ce 100644
--- a/samples/pktgen/functions.sh
+++ b/samples/pktgen/functions.sh
@@ -119,3 +119,46 @@ function root_check_run_with_sudo() {
err 4 "cannot perform sudo run of $0"
fi
}
+
+# Exact input device's NUMA node info
+function get_iface_node()
+{
+ local node=$(</sys/class/net/$1/device/numa_node)
+ if [[ $node == -1 ]]; then
+ echo 0
+ else
+ echo $node
+ fi
+}
+
+# Given an Dev/iface, get its queues' irq numbers
+function get_iface_irqs()
+{
+ local IFACE=$1
+ local queues="${IFACE}-.*TxRx"
+
+ irqs=$(grep "$queues" /proc/interrupts | cut -f1 -d:)
+ [ -z "$irqs" ] && irqs=$(grep $IFACE /proc/interrupts | cut -f1 -d:)
+ [ -z "$irqs" ] && irqs=$(for i in `ls -Ux /sys/class/net/$IFACE/device/msi_irqs` ;\
+ do grep "$i:.*TxRx" /proc/interrupts | grep -v fdir | cut -f 1 -d : ;\
+ done)
+ [ -z "$irqs" ] && err 3 "Could not find interrupts for $IFACE"
+
+ echo $irqs
+}
+
+# Given a NUMA node, return cpu ids belonging to it.
+function get_node_cpus()
+{
+ local node=$1
+ local node_cpu_list
+ local node_cpu_range_list=`cut -f1- -d, --output-delimiter=" " \
+ /sys/devices/system/node/node$node/cpulist`
+
+ for cpu_range in $node_cpu_range_list
+ do
+ node_cpu_list="$node_cpu_list "`seq -s " " ${cpu_range//-/ }`
+ done
+
+ echo $node_cpu_list
+}
^ permalink raw reply related [flat|nested] 7+ messages in thread* [net-next PATCH 2/4] samples/pktgen: add script pktgen_sample06_numa_awared_queue_irq_affinity.sh
2017-11-01 10:41 [net-next PATCH 0/4] Updates for samples/pktgen Jesper Dangaard Brouer
2017-11-01 10:41 ` [net-next PATCH 1/4] samples/pktgen: Add some helper functions Jesper Dangaard Brouer
@ 2017-11-01 10:41 ` Jesper Dangaard Brouer
2017-11-01 10:41 ` [net-next PATCH 3/4] samples/pktgen: update sample03, no need for clones when bursting Jesper Dangaard Brouer
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Jesper Dangaard Brouer @ 2017-11-01 10:41 UTC (permalink / raw)
To: netdev
Cc: Robert Olsson, Jesper Dangaard Brouer, robert.hu, pabeni, kafai, tariqt
From: Robert Hoo <robert.hu@intel.com>
This script simply does:
* Detect $DEV's NUMA node belonging.
* Bind each thread (processor of NUMA locality) with each $DEV queue's
irq affinity, 1:1 mapping.
* How many '-t' threads input determines how many queues will be utilized.
If '-f' designates first cpu id, then offset in the NUMA node's cpu list.
(Changes by Jesper: allow changing count from cmdline via '-n')
Signed-off-by: Robert Hoo <robert.hu@intel.com>
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
...tgen_sample06_numa_awared_queue_irq_affinity.sh | 97 ++++++++++++++++++++
1 file changed, 97 insertions(+)
create mode 100755 samples/pktgen/pktgen_sample06_numa_awared_queue_irq_affinity.sh
diff --git a/samples/pktgen/pktgen_sample06_numa_awared_queue_irq_affinity.sh b/samples/pktgen/pktgen_sample06_numa_awared_queue_irq_affinity.sh
new file mode 100755
index 000000000000..353adc17205e
--- /dev/null
+++ b/samples/pktgen/pktgen_sample06_numa_awared_queue_irq_affinity.sh
@@ -0,0 +1,97 @@
+#!/bin/bash
+#
+# Multiqueue: Using pktgen threads for sending on multiple CPUs
+# * adding devices to kernel threads which are in the same NUMA node
+# * bound devices queue's irq affinity to the threads, 1:1 mapping
+# * notice the naming scheme for keeping device names unique
+# * nameing scheme: dev@thread_number
+# * flow variation via random UDP source port
+#
+basedir=`dirname $0`
+source ${basedir}/functions.sh
+root_check_run_with_sudo "$@"
+#
+# Required param: -i dev in $DEV
+source ${basedir}/parameters.sh
+
+# Base Config
+DELAY="0" # Zero means max speed
+[ -z "$COUNT" ] && COUNT="20000000" # Zero means indefinitely
+[ -z "$CLONE_SKB" ] && CLONE_SKB="0"
+
+# Flow variation random source port between min and max
+UDP_MIN=9
+UDP_MAX=109
+
+node=`get_iface_node $DEV`
+irq_array=(`get_iface_irqs $DEV`)
+cpu_array=(`get_node_cpus $node`)
+
+[ $THREADS -gt ${#irq_array[*]} -o $THREADS -gt ${#cpu_array[*]} ] && \
+ err 1 "Thread number $THREADS exceeds: min (${#irq_array[*]},${#cpu_array[*]})"
+
+# (example of setting default params in your script)
+if [ -z "$DEST_IP" ]; then
+ [ -z "$IP6" ] && DEST_IP="198.18.0.42" || DEST_IP="FD00::1"
+fi
+[ -z "$DST_MAC" ] && DST_MAC="90:e2:ba:ff:ff:ff"
+
+# General cleanup everything since last run
+pg_ctrl "reset"
+
+# Threads are specified with parameter -t value in $THREADS
+for ((i = 0; i < $THREADS; i++)); do
+ # The device name is extended with @name, using thread number to
+ # make then unique, but any name will do.
+ # Set the queue's irq affinity to this $thread (processor)
+ # if '-f' is designated, offset cpu id
+ thread=${cpu_array[$((i+F_THREAD))]}
+ dev=${DEV}@${thread}
+ echo $thread > /proc/irq/${irq_array[$i]}/smp_affinity_list
+ info "irq ${irq_array[$i]} is set affinity to `cat /proc/irq/${irq_array[$i]}/smp_affinity_list`"
+
+ # Add remove all other devices and add_device $dev to thread
+ pg_thread $thread "rem_device_all"
+ pg_thread $thread "add_device" $dev
+
+ # select queue and bind the queue and $dev in 1:1 relationship
+ queue_num=$i
+ info "queue number is $queue_num"
+ pg_set $dev "queue_map_min $queue_num"
+ pg_set $dev "queue_map_max $queue_num"
+
+ # Notice config queue to map to cpu (mirrors smp_processor_id())
+ # It is beneficial to map IRQ /proc/irq/*/smp_affinity 1:1 to CPU number
+ pg_set $dev "flag QUEUE_MAP_CPU"
+
+ # Base config of dev
+ pg_set $dev "count $COUNT"
+ pg_set $dev "clone_skb $CLONE_SKB"
+ pg_set $dev "pkt_size $PKT_SIZE"
+ pg_set $dev "delay $DELAY"
+
+ # Flag example disabling timestamping
+ pg_set $dev "flag NO_TIMESTAMP"
+
+ # Destination
+ pg_set $dev "dst_mac $DST_MAC"
+ pg_set $dev "dst$IP6 $DEST_IP"
+
+ # Setup random UDP port src range
+ pg_set $dev "flag UDPSRC_RND"
+ pg_set $dev "udp_src_min $UDP_MIN"
+ pg_set $dev "udp_src_max $UDP_MAX"
+done
+
+# start_run
+echo "Running... ctrl^C to stop" >&2
+pg_ctrl "start"
+echo "Done" >&2
+
+# Print results
+for ((i = 0; i < $THREADS; i++)); do
+ thread=${cpu_array[$((i+F_THREAD))]}
+ dev=${DEV}@${thread}
+ echo "Device: $dev"
+ cat /proc/net/pktgen/$dev | grep -A2 "Result:"
+done
^ permalink raw reply related [flat|nested] 7+ messages in thread* [net-next PATCH 3/4] samples/pktgen: update sample03, no need for clones when bursting
2017-11-01 10:41 [net-next PATCH 0/4] Updates for samples/pktgen Jesper Dangaard Brouer
2017-11-01 10:41 ` [net-next PATCH 1/4] samples/pktgen: Add some helper functions Jesper Dangaard Brouer
2017-11-01 10:41 ` [net-next PATCH 2/4] samples/pktgen: add script pktgen_sample06_numa_awared_queue_irq_affinity.sh Jesper Dangaard Brouer
@ 2017-11-01 10:41 ` Jesper Dangaard Brouer
2017-11-01 10:41 ` [net-next PATCH 4/4] samples/pktgen: remove remaining old pktgen sample scripts Jesper Dangaard Brouer
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Jesper Dangaard Brouer @ 2017-11-01 10:41 UTC (permalink / raw)
To: netdev
Cc: Robert Olsson, Jesper Dangaard Brouer, robert.hu, pabeni, kafai, tariqt
Like sample05, don't use pktgen clone_skb feature when using 'burst' feature,
it is not really needed. This brings the burst users in sync.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
.../pktgen/pktgen_sample03_burst_single_flow.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/samples/pktgen/pktgen_sample03_burst_single_flow.sh b/samples/pktgen/pktgen_sample03_burst_single_flow.sh
index 8d26e0ca683d..8a46daf27966 100755
--- a/samples/pktgen/pktgen_sample03_burst_single_flow.sh
+++ b/samples/pktgen/pktgen_sample03_burst_single_flow.sh
@@ -30,7 +30,7 @@ if [ -z "$DEST_IP" ]; then
fi
[ -z "$DST_MAC" ] && DST_MAC="90:e2:ba:ff:ff:ff"
[ -z "$BURST" ] && BURST=32
-[ -z "$CLONE_SKB" ] && CLONE_SKB="100000"
+[ -z "$CLONE_SKB" ] && CLONE_SKB="0" # No need for clones when bursting
[ -z "$COUNT" ] && COUNT="0" # Zero means indefinitely
# Base Config
^ permalink raw reply related [flat|nested] 7+ messages in thread* [net-next PATCH 4/4] samples/pktgen: remove remaining old pktgen sample scripts
2017-11-01 10:41 [net-next PATCH 0/4] Updates for samples/pktgen Jesper Dangaard Brouer
` (2 preceding siblings ...)
2017-11-01 10:41 ` [net-next PATCH 3/4] samples/pktgen: update sample03, no need for clones when bursting Jesper Dangaard Brouer
@ 2017-11-01 10:41 ` Jesper Dangaard Brouer
2017-11-01 16:28 ` [net-next PATCH 0/4] Updates for samples/pktgen Alexei Starovoitov
2017-11-02 5:20 ` David Miller
5 siblings, 0 replies; 7+ messages in thread
From: Jesper Dangaard Brouer @ 2017-11-01 10:41 UTC (permalink / raw)
To: netdev
Cc: Robert Olsson, Jesper Dangaard Brouer, robert.hu, pabeni, kafai, tariqt
Since commit 0f06a6787e05 ("samples: Add an IPv6 '-6' option to the
pktgen scripts") the newer pktgen_sampleXX script does show howto use
IPv6 with pktgen.
Thus, there is no longer a reason to keep the older sample scripts around.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
samples/pktgen/pktgen.conf-1-1-ip6 | 60 ---------------------------
samples/pktgen/pktgen.conf-1-1-ip6-rdos | 63 ----------------------------
samples/pktgen/pktgen.conf-1-2 | 69 -------------------------------
3 files changed, 192 deletions(-)
delete mode 100755 samples/pktgen/pktgen.conf-1-1-ip6
delete mode 100755 samples/pktgen/pktgen.conf-1-1-ip6-rdos
delete mode 100755 samples/pktgen/pktgen.conf-1-2
diff --git a/samples/pktgen/pktgen.conf-1-1-ip6 b/samples/pktgen/pktgen.conf-1-1-ip6
deleted file mode 100755
index 0b9ffd47fd41..000000000000
--- a/samples/pktgen/pktgen.conf-1-1-ip6
+++ /dev/null
@@ -1,60 +0,0 @@
-#!/bin/bash
-
-#modprobe pktgen
-
-
-function pgset() {
- local result
-
- echo $1 > $PGDEV
-
- result=`cat $PGDEV | fgrep "Result: OK:"`
- if [ "$result" = "" ]; then
- cat $PGDEV | fgrep Result:
- fi
-}
-
-# Config Start Here -----------------------------------------------------------
-
-
-# thread config
-# Each CPU has its own thread. One CPU example. We add eth1.
-# IPv6. Note increase in minimal packet length
-
-PGDEV=/proc/net/pktgen/kpktgend_0
- echo "Removing all devices"
- pgset "rem_device_all"
- echo "Adding eth1"
- pgset "add_device eth1"
-
-
-# device config
-# delay 0
-
-CLONE_SKB="clone_skb 1000000"
-# NIC adds 4 bytes CRC
-PKT_SIZE="pkt_size 66"
-
-# COUNT 0 means forever
-#COUNT="count 0"
-COUNT="count 10000000"
-DELAY="delay 0"
-
-PGDEV=/proc/net/pktgen/eth1
- echo "Configuring $PGDEV"
- pgset "$COUNT"
- pgset "$CLONE_SKB"
- pgset "$PKT_SIZE"
- pgset "$DELAY"
- pgset "dst6 fec0::1"
- pgset "src6 fec0::2"
- pgset "dst_mac 00:04:23:08:91:dc"
-
-# Time to run
-PGDEV=/proc/net/pktgen/pgctrl
-
- echo "Running... ctrl^C to stop"
- trap true INT
- pgset "start"
- echo "Done"
- cat /proc/net/pktgen/eth1
diff --git a/samples/pktgen/pktgen.conf-1-1-ip6-rdos b/samples/pktgen/pktgen.conf-1-1-ip6-rdos
deleted file mode 100755
index ad98e5f40776..000000000000
--- a/samples/pktgen/pktgen.conf-1-1-ip6-rdos
+++ /dev/null
@@ -1,63 +0,0 @@
-#!/bin/bash
-
-#modprobe pktgen
-
-
-function pgset() {
- local result
-
- echo $1 > $PGDEV
-
- result=`cat $PGDEV | fgrep "Result: OK:"`
- if [ "$result" = "" ]; then
- cat $PGDEV | fgrep Result:
- fi
-}
-
-# Config Start Here -----------------------------------------------------------
-
-
-# thread config
-# Each CPU has its own thread. One CPU example. We add eth1.
-# IPv6. Note increase in minimal packet length
-
-PGDEV=/proc/net/pktgen/kpktgend_0
- echo "Removing all devices"
- pgset "rem_device_all"
- echo "Adding eth1"
- pgset "add_device eth1"
-
-
-# device config
-# delay 0 means maximum speed.
-
-# We need to do alloc for every skb since we cannot clone here.
-CLONE_SKB="clone_skb 0"
-
-# NIC adds 4 bytes CRC
-PKT_SIZE="pkt_size 66"
-
-# COUNT 0 means forever
-#COUNT="count 0"
-COUNT="count 10000000"
-DELAY="delay 0"
-
-PGDEV=/proc/net/pktgen/eth1
- echo "Configuring $PGDEV"
- pgset "$COUNT"
- pgset "$CLONE_SKB"
- pgset "$PKT_SIZE"
- pgset "$DELAY"
- pgset "dst6_min fec0::1"
- pgset "dst6_max fec0::FFFF:FFFF"
-
- pgset "dst_mac 00:04:23:08:91:dc"
-
-# Time to run
-PGDEV=/proc/net/pktgen/pgctrl
-
- echo "Running... ctrl^C to stop"
- trap true INT
- pgset "start"
- echo "Done"
- cat /proc/net/pktgen/eth1
diff --git a/samples/pktgen/pktgen.conf-1-2 b/samples/pktgen/pktgen.conf-1-2
deleted file mode 100755
index ba4eb26e168d..000000000000
--- a/samples/pktgen/pktgen.conf-1-2
+++ /dev/null
@@ -1,69 +0,0 @@
-#!/bin/bash
-
-#modprobe pktgen
-
-
-function pgset() {
- local result
-
- echo $1 > $PGDEV
-
- result=`cat $PGDEV | fgrep "Result: OK:"`
- if [ "$result" = "" ]; then
- cat $PGDEV | fgrep Result:
- fi
-}
-
-# Config Start Here -----------------------------------------------------------
-
-
-# thread config
-# One CPU means one thread. One CPU example. We add eth1, eth2 respectivly.
-
-PGDEV=/proc/net/pktgen/kpktgend_0
- echo "Removing all devices"
- pgset "rem_device_all"
- echo "Adding eth1"
- pgset "add_device eth1"
- echo "Adding eth2"
- pgset "add_device eth2"
-
-
-# device config
-# delay 0 means maximum speed.
-
-CLONE_SKB="clone_skb 1000000"
-# NIC adds 4 bytes CRC
-PKT_SIZE="pkt_size 60"
-
-# COUNT 0 means forever
-#COUNT="count 0"
-COUNT="count 10000000"
-DELAY="delay 0"
-
-PGDEV=/proc/net/pktgen/eth1
- echo "Configuring $PGDEV"
- pgset "$COUNT"
- pgset "$CLONE_SKB"
- pgset "$PKT_SIZE"
- pgset "$DELAY"
- pgset "dst 10.10.11.2"
- pgset "dst_mac 00:04:23:08:91:dc"
-
-PGDEV=/proc/net/pktgen/eth2
- echo "Configuring $PGDEV"
- pgset "$COUNT"
- pgset "$CLONE_SKB"
- pgset "$PKT_SIZE"
- pgset "$DELAY"
- pgset "dst 192.168.2.2"
- pgset "dst_mac 00:04:23:08:91:de"
-
-# Time to run
-PGDEV=/proc/net/pktgen/pgctrl
-
- echo "Running... ctrl^C to stop"
- trap true INT
- pgset "start"
- echo "Done"
- cat /proc/net/pktgen/eth1 /proc/net/pktgen/eth2
^ permalink raw reply related [flat|nested] 7+ messages in thread* Re: [net-next PATCH 0/4] Updates for samples/pktgen
2017-11-01 10:41 [net-next PATCH 0/4] Updates for samples/pktgen Jesper Dangaard Brouer
` (3 preceding siblings ...)
2017-11-01 10:41 ` [net-next PATCH 4/4] samples/pktgen: remove remaining old pktgen sample scripts Jesper Dangaard Brouer
@ 2017-11-01 16:28 ` Alexei Starovoitov
2017-11-02 5:20 ` David Miller
5 siblings, 0 replies; 7+ messages in thread
From: Alexei Starovoitov @ 2017-11-01 16:28 UTC (permalink / raw)
To: Jesper Dangaard Brouer
Cc: netdev, Robert Olsson, robert.hu, pabeni, kafai, tariqt
On Wed, Nov 01, 2017 at 11:41:04AM +0100, Jesper Dangaard Brouer wrote:
> This patchset updates samples/pktgen and synchronize with changes
> maintained in https://github.com/netoptimizer/network-testing/
>
> Features wise Robert Hoo <robert.hu@intel.com> added support for
> detecting and determining dev NUMA node IRQs, and added a new script
> named pktgen_sample06_numa_awared_queue_irq_affinity.sh that use these
> features.
>
> Cleanup remove last of the old sample files, as IPv6 is covered by
> existing sample code.
Looks nice. Thanks for doing this work.
Acked-by: Alexei Starovoitov <ast@kernel.org>
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [net-next PATCH 0/4] Updates for samples/pktgen
2017-11-01 10:41 [net-next PATCH 0/4] Updates for samples/pktgen Jesper Dangaard Brouer
` (4 preceding siblings ...)
2017-11-01 16:28 ` [net-next PATCH 0/4] Updates for samples/pktgen Alexei Starovoitov
@ 2017-11-02 5:20 ` David Miller
5 siblings, 0 replies; 7+ messages in thread
From: David Miller @ 2017-11-02 5:20 UTC (permalink / raw)
To: brouer; +Cc: netdev, robert, robert.hu, pabeni, kafai, tariqt
From: Jesper Dangaard Brouer <brouer@redhat.com>
Date: Wed, 01 Nov 2017 11:41:04 +0100
> This patchset updates samples/pktgen and synchronize with changes
> maintained in https://github.com/netoptimizer/network-testing/
>
> Features wise Robert Hoo <robert.hu@intel.com> added support for
> detecting and determining dev NUMA node IRQs, and added a new script
> named pktgen_sample06_numa_awared_queue_irq_affinity.sh that use these
> features.
>
> Cleanup remove last of the old sample files, as IPv6 is covered by
> existing sample code.
Series applied, thanks Jesper.
^ permalink raw reply [flat|nested] 7+ messages in thread