[Fuego] [PATCH 1/5] iperf3: be lax about the cpu utilization

[Fuego] [PATCH 1/5] iperf3: be lax about the cpu utilization

[Daniel Sangorrin]

Tim Bird reported having some failures due to too high CPU
utilization results. Let's use a lax criteria and let users
customize it for their boards.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Benchmark.iperf3/criteria.json | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/engine/tests/Benchmark.iperf3/criteria.json b/engine/tests/Benchmark.iperf3/criteria.json
index 8a50e1e..f4c2a16 100644
--- a/engine/tests/Benchmark.iperf3/criteria.json
+++ b/engine/tests/Benchmark.iperf3/criteria.json
@@ -39,42 +39,42 @@
         {
             "tguid":"cpu_utilization.host.total",
             "reference":{
-                "value":90,
+                "value":100,
                 "operator":"le"
             }
         },
         {
             "tguid":"cpu_utilization.host.system",
             "reference":{
-                "value":90,
+                "value":100,
                 "operator":"le"
             }
         },
         {
             "tguid":"cpu_utilization.host.user",
             "reference":{
-                "value":90,
+                "value":100,
                 "operator":"le"
             }
         },
         {
             "tguid":"cpu_utilization.remote.total",
             "reference":{
-                "value":90,
+                "value":100,
                 "operator":"le"
             }
         },
         {
             "tguid":"cpu_utilization.remote.system",
             "reference":{
-                "value":90,
+                "value":100,
                 "operator":"le"
             }
         },
         {
             "tguid":"cpu_utilization.remote.user",
             "reference":{
-                "value":90,
+                "value":100,
                 "operator":"le"
             }
         }
-- 
2.7.4

[Fuego] [PATCH 2/5] iperf3: use an spdx identifier for the license

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Benchmark.iperf3/test.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/engine/tests/Benchmark.iperf3/test.yaml b/engine/tests/Benchmark.iperf3/test.yaml
index f95d8c1..7a055ce 100644
--- a/engine/tests/Benchmark.iperf3/test.yaml
+++ b/engine/tests/Benchmark.iperf3/test.yaml
@@ -3,7 +3,7 @@ name: Benchmark.iperf3
 description: |
     iPerf3 is a tool for active measurements of the maximum achievable
     bandwidth on IP networks.
-license: three-clause BSD license.
+license: BSD-3-Clause
 author: |
     Jon Dugan, Seth Elliott, Bruce A. Mah, Jeff Poskanzer, Kaustubh Prabhu,
     Mark Ashley, Aaron Brown, Aeneas Jaißle, Susant Sahani, Bruce Simpson,
-- 
2.7.4

[Fuego] [PATCH 3/5] iperf3: save json file instead of using the stdout

[Daniel Sangorrin]

On certain boards/kernels, when using the UDP test/spec, some
extra messages (about not having pacing support) where
sent to the stdout polluting the json output. Fix that by
storing the json output on its own file.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Benchmark.iperf3/fuego_test.sh | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/engine/tests/Benchmark.iperf3/fuego_test.sh b/engine/tests/Benchmark.iperf3/fuego_test.sh
index 9f10743..4a6e98b 100755
--- a/engine/tests/Benchmark.iperf3/fuego_test.sh
+++ b/engine/tests/Benchmark.iperf3/fuego_test.sh
@@ -30,7 +30,8 @@ function test_run {
     fi
 
     echo "Starting iperf3 client on the target (CLIENT IP: $IPADDR)"
-    report "cd $BOARD_TESTDIR/fuego.$TESTDIR; ./iperf3 -V -c $IPERF3_SERVER_IP -f M -J --get-server-output $BENCHMARK_IPERF3_CLIENT_PARAMS" $BOARD_TESTDIR/fuego.$TESTDIR/${TESTDIR}.log
+    cmd "cd $BOARD_TESTDIR/fuego.$TESTDIR; ./iperf3 -V -c $IPERF3_SERVER_IP -f M -J --logfile $BOARD_TESTDIR/fuego.$TESTDIR/output.json --get-server-output $BENCHMARK_IPERF3_CLIENT_PARAMS"
+    report "cat $BOARD_TESTDIR/fuego.$TESTDIR/output.json"
 }
 
 function test_cleanup {
-- 
2.7.4

[Fuego] [PATCH 4/5] iperf3: remove server initialization

[Daniel Sangorrin]

Killing and restarting the server could create race conditions
when running the test at the same time on multiple boards.
Instead we move the initialization to the Docker entrypoint
(see Fuego commit)

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Benchmark.iperf3/fuego_test.sh | 9 ---------
 1 file changed, 9 deletions(-)

diff --git a/engine/tests/Benchmark.iperf3/fuego_test.sh b/engine/tests/Benchmark.iperf3/fuego_test.sh
index 4a6e98b..4c27ce1 100755
--- a/engine/tests/Benchmark.iperf3/fuego_test.sh
+++ b/engine/tests/Benchmark.iperf3/fuego_test.sh
@@ -20,15 +20,6 @@ function test_run {
         return 1
     fi
     echo "Using server ip address: $IPERF3_SERVER_IP"
-
-    # Check if the server ip belongs to us
-    ROUTE=$(ip route get $IPERF3_SERVER_IP | cut -f 1 -d " ")
-    if [ "$ROUTE" = "local" ]; then
-        echo "Starting iperf3 server on localhost (SERVER IP: $IPERF3_SERVER_IP)"
-        killall iperf3 2> /dev/null || true
-        iperf3 -V -s -D $BENCHMARK_IPERF3_SERVER_PARAMS
-    fi
-
     echo "Starting iperf3 client on the target (CLIENT IP: $IPADDR)"
     cmd "cd $BOARD_TESTDIR/fuego.$TESTDIR; ./iperf3 -V -c $IPERF3_SERVER_IP -f M -J --logfile $BOARD_TESTDIR/fuego.$TESTDIR/output.json --get-server-output $BENCHMARK_IPERF3_CLIENT_PARAMS"
     report "cat $BOARD_TESTDIR/fuego.$TESTDIR/output.json"
-- 
2.7.4

[Fuego] [PATCH 5/5] iperf3: add retries if the server is busy

[Daniel Sangorrin]

The iperf3 server can only serve iperf3 clients one at a time.
For that reason, running the test on multiple boards at the
same time would end up in an error for all boards except the
one that arrived first.

One solution would be to start multiple servers using
different ports, and asking Fuego users to write the default
port on their board files.

Here I am taking another solution that does not require
doing that and therefore is less confusing. The idea is
to allow iperf3 clients to retry the connection several
times (with 30 seconds intervals). This approach also has the
benefit that by ensuring that only one test can run at the
same time, the cpu/network utilization of the server is not
affected by other boards.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Benchmark.iperf3/fuego_test.sh | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)

diff --git a/engine/tests/Benchmark.iperf3/fuego_test.sh b/engine/tests/Benchmark.iperf3/fuego_test.sh
index 4c27ce1..528eba8 100755
--- a/engine/tests/Benchmark.iperf3/fuego_test.sh
+++ b/engine/tests/Benchmark.iperf3/fuego_test.sh
@@ -19,9 +19,24 @@ function test_run {
         echo "ERROR: set the server ip on the spec or board file"
         return 1
     fi
-    echo "Using server ip address: $IPERF3_SERVER_IP"
-    echo "Starting iperf3 client on the target (CLIENT IP: $IPADDR)"
-    cmd "cd $BOARD_TESTDIR/fuego.$TESTDIR; ./iperf3 -V -c $IPERF3_SERVER_IP -f M -J --logfile $BOARD_TESTDIR/fuego.$TESTDIR/output.json --get-server-output $BENCHMARK_IPERF3_CLIENT_PARAMS"
+
+    echo "Starting iperf3 client on the target (CLIENT IP: $IPADDR, SERVER IP: $IPERF3_SERVER_IP)"
+    json_file=$(mktemp)
+    for i in 1 2 3 4 5 6; do
+        cmd "$BOARD_TESTDIR/fuego.$TESTDIR/iperf3 -V -c $IPERF3_SERVER_IP -J --logfile $BOARD_TESTDIR/fuego.$TESTDIR/output.json --get-server-output $BENCHMARK_IPERF3_CLIENT_PARAMS" && break || \
+            echo "The server seems busy running another iperf3 test. Trying again in 30 seconds"
+
+        # that was our last try so abort the job
+        if [ $i -eq 6 ]; then
+            abort_job "The server seems busy running another iperf3 test."
+        fi
+
+        # remove the json file before retrying, otherwise it gets appended
+        cmd "rm -f $BOARD_TESTDIR/fuego.$TESTDIR/output.json"
+        sleep 30
+    done
+    rm -f $json_file
+
     report "cat $BOARD_TESTDIR/fuego.$TESTDIR/output.json"
 }
 
-- 
2.7.4

Re: [Fuego] [PATCH 4/5] iperf3: remove server initialization

[Tim.Bird]

Patches 1 through 4 in this series look fine.  1-4 Applied.
 -- Tim


> -----Original Message-----
> From: Daniel Sangorrin
> 
> Killing and restarting the server could create race conditions
> when running the test at the same time on multiple boards.
> Instead we move the initialization to the Docker entrypoint
> (see Fuego commit)
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/tests/Benchmark.iperf3/fuego_test.sh | 9 ---------
>  1 file changed, 9 deletions(-)
> 
> diff --git a/engine/tests/Benchmark.iperf3/fuego_test.sh
> b/engine/tests/Benchmark.iperf3/fuego_test.sh
> index 4a6e98b..4c27ce1 100755
> --- a/engine/tests/Benchmark.iperf3/fuego_test.sh
> +++ b/engine/tests/Benchmark.iperf3/fuego_test.sh
> @@ -20,15 +20,6 @@ function test_run {
>          return 1
>      fi
>      echo "Using server ip address: $IPERF3_SERVER_IP"
> -
> -    # Check if the server ip belongs to us
> -    ROUTE=$(ip route get $IPERF3_SERVER_IP | cut -f 1 -d " ")
> -    if [ "$ROUTE" = "local" ]; then
> -        echo "Starting iperf3 server on localhost (SERVER IP:
> $IPERF3_SERVER_IP)"
> -        killall iperf3 2> /dev/null || true
> -        iperf3 -V -s -D $BENCHMARK_IPERF3_SERVER_PARAMS
> -    fi
> -
>      echo "Starting iperf3 client on the target (CLIENT IP: $IPADDR)"
>      cmd "cd $BOARD_TESTDIR/fuego.$TESTDIR; ./iperf3 -V -c
> $IPERF3_SERVER_IP -f M -J --logfile
> $BOARD_TESTDIR/fuego.$TESTDIR/output.json --get-server-output
> $BENCHMARK_IPERF3_CLIENT_PARAMS"
>      report "cat $BOARD_TESTDIR/fuego.$TESTDIR/output.json"
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH 5/5] iperf3: add retries if the server is busy

[Tim.Bird]

I have some questions.
See below.

> -----Original Message-----
> From: Daniel Sangorrin
> 
> The iperf3 server can only serve iperf3 clients one at a time.
> For that reason, running the test on multiple boards at the
> same time would end up in an error for all boards except the
> one that arrived first.
> 
> One solution would be to start multiple servers using
> different ports, and asking Fuego users to write the default
> port on their board files.
> 
> Here I am taking another solution that does not require
> doing that and therefore is less confusing. The idea is
> to allow iperf3 clients to retry the connection several
> times (with 30 seconds intervals). This approach also has the
> benefit that by ensuring that only one test can run at the
> same time, the cpu/network utilization of the server is not
> affected by other boards.

Regarding the general approach: it looks like this code waits about
3 minutes.  How long does the average iperf test take?  Is a 3-minute
timeout waiting for the service sufficient?

What happens if multiple clients attempt iperf simultaneously?
It would be better if we waited 3 minutes, but reset the counter
if we determined that a different client was using the server.  This
could create a potentially long busywait (even an indefinite wait
in the worst case).  But that might be preferable to erroring out
if there happen to be multiple collisions on the server.
I'm guessing that would be extremely difficult to implement, so maybe
we'll just have to live with occasional retry exhaustion.

> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/tests/Benchmark.iperf3/fuego_test.sh | 21 ++++++++++++++++++-
> --
>  1 file changed, 18 insertions(+), 3 deletions(-)
> 
> diff --git a/engine/tests/Benchmark.iperf3/fuego_test.sh
> b/engine/tests/Benchmark.iperf3/fuego_test.sh
> index 4c27ce1..528eba8 100755
> --- a/engine/tests/Benchmark.iperf3/fuego_test.sh
> +++ b/engine/tests/Benchmark.iperf3/fuego_test.sh
> @@ -19,9 +19,24 @@ function test_run {
>          echo "ERROR: set the server ip on the spec or board file"
>          return 1
>      fi
> -    echo "Using server ip address: $IPERF3_SERVER_IP"
> -    echo "Starting iperf3 client on the target (CLIENT IP: $IPADDR)"
> -    cmd "cd $BOARD_TESTDIR/fuego.$TESTDIR; ./iperf3 -V -c
> $IPERF3_SERVER_IP -f M -J --logfile
> $BOARD_TESTDIR/fuego.$TESTDIR/output.json --get-server-output
> $BENCHMARK_IPERF3_CLIENT_PARAMS"
> +
> +    echo "Starting iperf3 client on the target (CLIENT IP: $IPADDR, SERVER IP:
> $IPERF3_SERVER_IP)"
> +    json_file=$(mktemp)
maybe do the following:

IPERF_RETRIES=6
IPERF_RETRY_LIST=$(seq $IPERF_RETRIES)
for I in ${IPERF_RETRY_LIST}; do

This would isolate the retry count to one location in the code.

> +    for i in 1 2 3 4 5 6; do
> +        cmd "$BOARD_TESTDIR/fuego.$TESTDIR/iperf3 -V -c
> $IPERF3_SERVER_IP -J --logfile
> $BOARD_TESTDIR/fuego.$TESTDIR/output.json --get-server-output
> $BENCHMARK_IPERF3_CLIENT_PARAMS" && break || \

Are there other reasons the command might fail, that we can detect,
besides server busy?  Is there an exit code for that specific case, or
are we just using any failure as possible indicator that the server is
busy?

> +            echo "The server seems busy running another iperf3 test. Trying again
> in 30 seconds"
> +
> +        # that was our last try so abort the job
> +        if [ $i -eq 6 ]; then
I prefer a '-ge', purely for style sake.

If using the above, maybe:
    if [ $i -ge $IPERF_RETRIES ]; then

> +            abort_job "The server seems busy running another iperf3 test."
> +        fi
> +
> +        # remove the json file before retrying, otherwise it gets appended
> +        cmd "rm -f $BOARD_TESTDIR/fuego.$TESTDIR/output.json"
> +        sleep 30
> +    done
> +    rm -f $json_file
> +
>      report "cat $BOARD_TESTDIR/fuego.$TESTDIR/output.json"
>  }
> 
> --
> 2.7.4

Thanks.
 -- Tim

Re: [Fuego] [PATCH 5/5] iperf3: add retries if the server is busy

[Daniel Sangorrin]

> -----Original Message-----
> From: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com]
> Sent: Tuesday, March 20, 2018 1:19 PM
> To: daniel.sangorrin@toshiba.co.jp; fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH 5/5] iperf3: add retries if the server is busy
> 
> I have some questions.
> See below.
> 
> > -----Original Message-----
> > From: Daniel Sangorrin
> >
> > The iperf3 server can only serve iperf3 clients one at a time.
> > For that reason, running the test on multiple boards at the
> > same time would end up in an error for all boards except the
> > one that arrived first.
> >
> > One solution would be to start multiple servers using
> > different ports, and asking Fuego users to write the default
> > port on their board files.
> >
> > Here I am taking another solution that does not require
> > doing that and therefore is less confusing. The idea is
> > to allow iperf3 clients to retry the connection several
> > times (with 30 seconds intervals). This approach also has the
> > benefit that by ensuring that only one test can run at the
> > same time, the cpu/network utilization of the server is not
> > affected by other boards.
> 
> Regarding the general approach: it looks like this code waits about
> 3 minutes.  How long does the average iperf test take?  Is a 3-minute
> timeout waiting for the service sufficient?
> 
> What happens if multiple clients attempt iperf simultaneously?
> It would be better if we waited 3 minutes, but reset the counter
> if we determined that a different client was using the server.  This
> could create a potentially long busywait (even an indefinite wait
> in the worst case).  But that might be preferable to erroring out
> if there happen to be multiple collisions on the server.
> I'm guessing that would be extremely difficult to implement, so maybe
> we'll just have to live with occasional retry exhaustion.

I think it would be hard to implement in a generic way.
If you want I can increase the timeout.

> 
> >
> > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > ---
> >  engine/tests/Benchmark.iperf3/fuego_test.sh | 21 ++++++++++++++++++-
> > --
> >  1 file changed, 18 insertions(+), 3 deletions(-)
> >
> > diff --git a/engine/tests/Benchmark.iperf3/fuego_test.sh
> > b/engine/tests/Benchmark.iperf3/fuego_test.sh
> > index 4c27ce1..528eba8 100755
> > --- a/engine/tests/Benchmark.iperf3/fuego_test.sh
> > +++ b/engine/tests/Benchmark.iperf3/fuego_test.sh
> > @@ -19,9 +19,24 @@ function test_run {
> >          echo "ERROR: set the server ip on the spec or board file"
> >          return 1
> >      fi
> > -    echo "Using server ip address: $IPERF3_SERVER_IP"
> > -    echo "Starting iperf3 client on the target (CLIENT IP: $IPADDR)"
> > -    cmd "cd $BOARD_TESTDIR/fuego.$TESTDIR; ./iperf3 -V -c
> > $IPERF3_SERVER_IP -f M -J --logfile
> > $BOARD_TESTDIR/fuego.$TESTDIR/output.json --get-server-output
> > $BENCHMARK_IPERF3_CLIENT_PARAMS"
> > +
> > +    echo "Starting iperf3 client on the target (CLIENT IP: $IPADDR, SERVER IP:
> > $IPERF3_SERVER_IP)"
> > +    json_file=$(mktemp)
> maybe do the following:
> 
> IPERF_RETRIES=6
> IPERF_RETRY_LIST=$(seq $IPERF_RETRIES)
> for I in ${IPERF_RETRY_LIST}; do
> 
> This would isolate the retry count to one location in the code.
> 
> > +    for i in 1 2 3 4 5 6; do
> > +        cmd "$BOARD_TESTDIR/fuego.$TESTDIR/iperf3 -V -c
> > $IPERF3_SERVER_IP -J --logfile
> > $BOARD_TESTDIR/fuego.$TESTDIR/output.json --get-server-output
> > $BENCHMARK_IPERF3_CLIENT_PARAMS" && break || \
> 
> Are there other reasons the command might fail, that we can detect,
> besides server busy?  Is there an exit code for that specific case, or
> are we just using any failure as possible indicator that the server is
> busy?
> 
> > +            echo "The server seems busy running another iperf3 test. Trying again
> > in 30 seconds"
> > +
> > +        # that was our last try so abort the job
> > +        if [ $i -eq 6 ]; then
> I prefer a '-ge', purely for style sake.
> 
> If using the above, maybe:
>     if [ $i -ge $IPERF_RETRIES ]; then
> 
> > +            abort_job "The server seems busy running another iperf3 test."
> > +        fi
> > +
> > +        # remove the json file before retrying, otherwise it gets appended
> > +        cmd "rm -f $BOARD_TESTDIR/fuego.$TESTDIR/output.json"
> > +        sleep 30
> > +    done
> > +    rm -f $json_file
> > +
> >      report "cat $BOARD_TESTDIR/fuego.$TESTDIR/output.json"
> >  }
> >
> > --
> > 2.7.4
> 
> Thanks.
>  -- Tim