Re: [PATCH v2 3/3] perf-test: add a test for perf-stat --bpf-counters option

From: Song Liu <songliubraving@fb.com>
To: Namhyung Kim <namhyung@kernel.org>
Cc: linux-kernel <linux-kernel@vger.kernel.org>,
	Kernel Team <Kernel-team@fb.com>,
	Arnaldo Carvalho de Melo <acme@kernel.org>,
	"Arnaldo Carvalho de Melo" <acme@redhat.com>,
	Jiri Olsa <jolsa@kernel.org>
Subject: Re: [PATCH v2 3/3] perf-test: add a test for perf-stat --bpf-counters option
Date: Thu, 18 Mar 2021 07:39:08 +0000	[thread overview]
Message-ID: <85FE8519-615F-464F-A2BE-06A16D6BAFD0@fb.com> (raw)
In-Reply-To: <CAM9d7ciqTGRx6kZ3c-+tXi3AwvOj5EG1ig4oTiUy9sDuTA=KEw@mail.gmail.com>

> On Mar 17, 2021, at 11:07 PM, Namhyung Kim <namhyung@kernel.org> wrote:
> 
> On Wed, Mar 17, 2021 at 6:18 AM Song Liu <songliubraving@fb.com> wrote:
>> 
>> Add a test to compare the output of perf-stat with and without option
>> --bpf-counters. If the difference is more than 10%, the test is considered
>> as failed.
>> 
>> For stable results between two runs (w/ and w/o --bpf-counters), the test
>> program should: 1) be long enough for better signal-noise-ratio; 2) not
>> depend on the behavior of IO subsystem (for less noise from caching). So
>> far, the best option we found is stressapptest.
>> 
>> Signed-off-by: Song Liu <songliubraving@fb.com>
>> ---
>> tools/perf/tests/shell/stat_bpf_counters.sh | 34 +++++++++++++++++++++
>> 1 file changed, 34 insertions(+)
>> create mode 100755 tools/perf/tests/shell/stat_bpf_counters.sh
>> 
>> diff --git a/tools/perf/tests/shell/stat_bpf_counters.sh b/tools/perf/tests/shell/stat_bpf_counters.sh
>> new file mode 100755
>> index 0000000000000..c0bcb38d6b53c
>> --- /dev/null
>> +++ b/tools/perf/tests/shell/stat_bpf_counters.sh
>> @@ -0,0 +1,34 @@
>> +#!/bin/sh
>> +# perf stat --bpf-counters test
>> +# SPDX-License-Identifier: GPL-2.0
>> +
>> +set -e
>> +
>> +# check whether $2 is within +/- 10% of $1
>> +compare_number()
>> +{
>> +       first_num=$1
>> +       second_num=$2
>> +
>> +       # upper bound is first_num * 110%
>> +       upper=$(( $first_num + $first_num / 10 ))
>> +       # lower bound is first_num * 90%
>> +       lower=$(( $first_num - $first_num / 10 ))
>> +
>> +       if [ $second_num -gt $upper ] || [ $second_num -lt $lower ]; then
>> +               echo "The difference between $first_num and $second_num are greater than 10%."
>> +               exit 1
>> +       fi
>> +}
>> +
>> +# skip if --bpf-counters is not supported
>> +perf stat --bpf-counters true > /dev/null 2>&1 || exit 2
>> +
>> +# skip if stressapptest is not available
>> +stressapptest -s 1 -M 100 -m 1 > /dev/null 2>&1 || exit 2
> 
> I don't know how popular it is, but we can print some info
> in case we miss it.

I just realized that perf-bench-sched-messaging is a good test to use, 
so we don't need stressapptest. Attached the updated version below.

> 
>> +
>> +base_cycles=$(perf stat --no-big-num -e cycles -- stressapptest -s 3 -M 100 -m 1 2>&1 | grep -e cycles | awk '{print $1}')
>> +bpf_cycles=$(perf stat --no-big-num --bpf-counters -e cycles -- stressapptest -s 3 -M 100 -m 1 2>&1 | grep -e cycles | awk '{print $1}')
> 
> I think just awk '/cycles/ {print $1}' should work.

Thanks! Fixed in the new version. 

Song

cat tools/perf/tests/shell/stat_bpf_counters.sh
#!/bin/sh
# perf stat --bpf-counters test
# SPDX-License-Identifier: GPL-2.0

set -e

# check whether $2 is within +/- 10% of $1
compare_number()
{
        first_num=$1
        second_num=$2

        # upper bound is first_num * 110%
        upper=$(( $first_num + $first_num / 10 ))
        # lower bound is first_num * 90%
        lower=$(( $first_num - $first_num / 10 ))

        if [ $second_num -gt $upper ] || [ $second_num -lt $lower ]; then
                echo "The difference between $first_num and $second_num are greater than 10%."
                exit 1
        fi
}

# skip if --bpf-counters is not supported
perf stat --bpf-counters true > /dev/null 2>&1 || exit 2

base_cycles=$(perf stat --no-big-num -e cycles -- perf bench sched messaging -g 1 -l 100 -t 2>&1 | awk '/cycles/ {print $1}')
bpf_cycles=$(perf stat --no-big-num --bpf-counters -e cycles -- perf bench sched messaging -g 1 -l 100 -t 2>&1 | awk '/cycles/ {print $1}')

compare_number $base_cycles $bpf_cycles
exit 0