Re: [PATCH v7 09/11] iotests: add testrunner.py

From: Kevin Wolf <kwolf@redhat.com>
To: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Cc: qemu-block@nongnu.org, qemu-devel@nongnu.org, mreitz@redhat.com,
	den@openvz.org, jsnow@redhat.com
Subject: Re: [PATCH v7 09/11] iotests: add testrunner.py
Date: Fri, 22 Jan 2021 15:51:26 +0100	[thread overview]
Message-ID: <20210122145126.GI15866@merkur.fritz.box> (raw)
In-Reply-To: <20210116134424.82867-10-vsementsov@virtuozzo.com>

Am 16.01.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben:
> Add TestRunner class, which will run tests in a new python iotests
> running framework.
> 
> There are some differences with current ./check behavior, most
> significant are:
> - Consider all tests self-executable, just run them, don't run python
>   by hand.
> - Elapsed time is cached in json file
> - Elapsed time precision increased a bit
> - use python difflib instead of "diff -w", to ignore spaces at line
>   ends strip lines by hand. Do not ignore other spaces.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Sorry for sending so many individual mails, but I'm running into things
as I actually run the code during the review of the final 'check'
rewrite.

I get this output for a failing test case:

    001   fail       [15:44:41] [15:44:42]   0.5s   (last: 13.2s) output mismatch (see {f_bad})
    --- /home/kwolf/source/qemu/tests/qemu-iotests/001.out

    +++ 001.out.bad

    @@ -1,4 +1,4 @@

    -QA output created by 001
    +Broken QA output created by 001
     Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728

     == reading whole image ==
    Failures: 001

Somehow we get additional empty lines in the diff header that shouldn't
be there. Not sure where it comes from. Especially the empty line after
@@ even corrupts the patch if you try to apply it.

The other problem is obvious enough:

> +        diff = file_diff(str(f_reference), str(f_bad))
> +        if diff:
> +            return TestResult(status='fail', elapsed=elapsed,
> +                              description='output mismatch (see {f_bad})',

This needs to be an f-string.

Kevin


