Re: [PATCH v7 4/8] Acceptance tests: use relative location for tests

["Philippe Mathieu-Daudé" <philmd@redhat.com>]

On 11/11/19 11:11 PM, Cleber Rosa wrote:
> On Mon, Nov 04, 2019 at 07:26:23PM +0100, Philippe Mathieu-Daudé wrote:
>> On 11/4/19 4:13 PM, Cleber Rosa wrote:
>>> An Avocado Test ID[1] is composed by a number of components, but it
>>> starts with the Test Name, usually a file system location that was
>>> given to the loader.
>>>
>>> Because the source directory is being given as a prefix to the
>>> "tests/acceptance" directory containing the acceptance tests, the test
>>> names will needlessly include the directory the user is using to host
>>> the QEMU sources (and/or build tree).
>>>
>>> Let's remove the source dir (or a build dir) from the path given to
>>> the test loader.  This should give more constant names, and when using
>>> result servers and databases, it should give the same test names
>>> across executions from different people or from different directories.
>>
>> Can we strip the full path to directory and only keep the filename in the
>> database? (Thinking about out-of-tree tests).
>>
> 
> Yes, absolutely, but this needs to be done one the Avocado side.  TBH,
> I have ideas to make this go even further, such as:
> 
>   1) the stripping of the "test_" prefix of the test method
> 
>   2) replace the full path to a directory with tests for a "test suite"
>      alias (default to the directory name itself)
> 
>   3) test suite alias will be persisted on test result such as reports
>      or machine, but ommited from the human UI
> 
>   4) full path to directory, exact version of test files (git hash) will
>      all be metadata and not part of the test ID
> 
> Roughly speaking, something which is listed like this currently:
> 
>    $ avocado list tests/acceptance/
>    INSTRUMENTED tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_x86_64_pc
>    ...
> 
> When executed, would be shown as:
> 
>    JOB ID     : fb885e9c3e7dc50534ec380a7e988cbf94233f07
>    JOB LOG    : /home/cleber/avocado/job-results/job-2019-11-11T17.07-fb885e9/job.log
>     (1/1) acceptance/boot_linux_console.py:BootLinuxConsole.x86_64_pc: PASS (2.17 s)

For the particular use case of QEMU, we can also strip the "acceptance/" 
part (and eventually ".py").

>    RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>    JOB TIME   : 2.35 s
> 
> How does that sound?
> 
> - Cleber.
>