RE: [PATCH i-g-t v2 0/2] lib/kunit: Execute test cases synchronously

From: "Cavitt, Jonathan" <jonathan.cavitt@intel.com>
To: Janusz Krzysztofik <janusz.krzysztofik@linux.intel.com>,
	"igt-dev@lists.freedesktop.org" <igt-dev@lists.freedesktop.org>
Cc: "intel-gfx@lists.freedesktop.org"
	<intel-gfx@lists.freedesktop.org>,
	"intel-xe@lists.freedesktop.org" <intel-xe@lists.freedesktop.org>,
	"Kamil Konieczny" <kamil.konieczny@linux.intel.com>,
	Mauro Carvalho Chehab <mchehab@kernel.org>,
	"De Marchi, Lucas" <lucas.demarchi@intel.com>
Subject: RE: [PATCH i-g-t v2 0/2] lib/kunit: Execute test cases synchronously
Date: Thu, 7 Mar 2024 20:58:39 +0000	[thread overview]
Message-ID: <CH0PR11MB544442BD9EDD4DD68D482A5DE5202@CH0PR11MB5444.namprd11.prod.outlook.com> (raw)
In-Reply-To: <20240227151128.16802-4-janusz.krzysztofik@linux.intel.com>


All patches LGTM
Acked-by: Jonathan Cavitt <jonathan.cavitt@intel.com>
-Jonathan Cavitt

-----Original Message-----
From: Intel-gfx <intel-gfx-bounces@lists.freedesktop.org> On Behalf Of Janusz Krzysztofik
Sent: Tuesday, February 27, 2024 7:11 AM
To: igt-dev@lists.freedesktop.org
Cc: intel-gfx@lists.freedesktop.org; intel-xe@lists.freedesktop.org; Kamil Konieczny <kamil.konieczny@linux.intel.com>; Mauro Carvalho Chehab <mchehab@kernel.org>; De Marchi, Lucas <lucas.demarchi@intel.com>
Subject: [PATCH i-g-t v2 0/2] lib/kunit: Execute test cases synchronously
> 
> Up to now we were loading a KUnit test module in test execution mode only
> once per subtest, in background, and then, in parallel with execution of
> test cases while the module was loading, we were looking through dmesg for
> KTAP results from each expected test case.  As a consequence, our IGT
> messages were more or less delayed, never in full sync with kernel
> messages.  Moreover, parsing of KTAP results from already completed test
> cases could be abandoned on a failure from loading the test module or
> kernel taint caused by a subsequent test case.  Also, parsing of KTAP
> results from all subsequent test cases could be abandoned on a failure of
> the parser caused by any test case.  Other than that, if a user requested
> a single dynamic sub-subtest, all test cases were executed anyway while
> results from only one of them that corresponded to the selected dynamic
> sub-subtest were reported.  That way, kernel messages from unrelated test
> cases, not only the selected one, could contribute to dmesg-fail or dmesg-
> warn CI results from that sub-subtest.
> 
> Since recent KUnit implementation is capable of executing only those test
> cases that match a user filter, stop executing all of them asynchronously
> and parsing their KTAP results as they appear.  Instead, reload the test
> module once per each dynamic sub-subtest with a filter that selects a
> specific test case and wait for its completion.  If successful and no
> kernel taint has occurred then parse the whole KTAP report from a single
> test case it has produced and translate it to IGT result of that single
> corresponding sub-subtest.
> 
> v2: Refresh the series on top of changes to KUnit filters handling,
>   - move the code of a new helper from a previous patch 1 to a previous
>     patch 2 which now becomes patch 1,
>   - actually limit the scope of the helper to fetching a KTAP report from
>     a file descriptor, and let the callers decide on how other steps, like
>     setting up filters or loading a test module, and errors they return
>     are handled,
>   - update commit description with a more detailed justification of why we
>     need these changes,
>   - rebase the former patch 1 on top of the new patch 1, update its commit
>     message and description and provide it as patch 2.
> 
> Janusz Krzysztofik (2):
>   lib/kunit: Execute test cases synchronously
>   lib/kunit: Minimize code duplication
> 
>  lib/igt_kmod.c | 172 ++++++++++++++++---------------------------------
>  1 file changed, 54 insertions(+), 118 deletions(-)
> 
> -- 
> 2.43.0
> 
>