Re: [PATCH v2 1/8] simplebench: bench_one(): add slow_limit argument

[John Snow <jsnow@redhat.com>]

On 3/4/21 5:17 AM, Vladimir Sementsov-Ogievskiy wrote:
> Sometimes one of cells in a testing table runs too slow. And we really
> don't want to wait so long. Limit number of runs in this case.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
>   scripts/simplebench/simplebench.py | 29 +++++++++++++++++++++++++----
>   1 file changed, 25 insertions(+), 4 deletions(-)
> 
> diff --git a/scripts/simplebench/simplebench.py b/scripts/simplebench/simplebench.py
> index f61513af90..b153cae274 100644
> --- a/scripts/simplebench/simplebench.py
> +++ b/scripts/simplebench/simplebench.py
> @@ -19,9 +19,11 @@
>   #
>   
>   import statistics
> +import time
>   
>   
> -def bench_one(test_func, test_env, test_case, count=5, initial_run=True):
> +def bench_one(test_func, test_env, test_case, count=5, initial_run=True,
> +              slow_limit=100):
>       """Benchmark one test-case
>   
>       test_func   -- benchmarking function with prototype
> @@ -36,6 +38,8 @@ def bench_one(test_func, test_env, test_case, count=5, initial_run=True):
>       test_case   -- test case - opaque second argument for test_func
>       count       -- how many times to call test_func, to calculate average
>       initial_run -- do initial run of test_func, which don't get into result
> +    slow_limit  -- reduce test runs to 2, if current run exceedes the limit
> +                   (in seconds)
>   

s/exceedes/exceeds, and you need to mention that if the initial run 
exceeds the limit, it will change the behavior to count that result.

It is also possible (conceivably) that the initial run exceeds the 
limit, but subsequent runs don't, so it might be hard to predict how 
many tests it'll actually run.

If you're OK with that behavior, maybe:

"Consider a test run 'slow' once it exceeds this limit, in seconds.
  Stop early once there are two 'slow' runs, including the initial run.
  Slow initial runs will be included in the results."

Lastly, this will change existing behavior -- do we care? Should it 
default to None instead? Should we be able to pass None or 0 to disable 
this behavior?

>       Returns dict with the following fields:
>           'runs':     list of test_func results
> @@ -47,17 +51,34 @@ def bench_one(test_func, test_env, test_case, count=5, initial_run=True):
>           'n-failed': number of failed runs (exists only if at least one run
>                       failed)
>       """
> +    runs = []
> +    i = 0
>       if initial_run:
> +        t = time.time()
> +
>           print('  #initial run:')
> -        print('   ', test_func(test_env, test_case))
> +        res = test_func(test_env, test_case)
> +        print('   ', res)
> +
> +        if time.time() - t > slow_limit:
> +            print('    - initial run is too slow, so it counts')
> +            runs.append(res)
> +            i = 1
> +
> +    for i in range(i, count):
> +        t = time.time()
>   
> -    runs = []
> -    for i in range(count):
>           print('  #run {}'.format(i+1))
>           res = test_func(test_env, test_case)
>           print('   ', res)
>           runs.append(res)
>   
> +        if time.time() - t > slow_limit and len(runs) >= 2:
> +            print('    - run is too slow, and we have enough runs, stop here')
> +            break
> +
> +    count = len(runs)
> +
>       result = {'runs': runs}
>   
>       succeeded = [r for r in runs if ('seconds' in r or 'iops' in r)]
>