archive mirror
 help / color / mirror / Atom feed
From: Richard Purdie <>
Subject: Re: [OE-core] [PATCH v2 0/4] scripts/resulttool/regression: add metadata filtering
Date: Thu, 16 Feb 2023 00:02:44 +0000	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

On Tue, 2023-02-14 at 17:53 +0100, Alexis Lothoré via wrote:
> From: Alexis Lothoré <>
> This v2 does not contain any change in patches content, it only sets the From:
> field correctly. Sorry for the noise.
> This patch serie is a proposal linked to discussion initiated here:
> After integration of some improvements on regression reporting, it has been
> observed that the regression report of version 4.2_M2 is way too big. When
> checking it, it appears that a big part of the report is composed of "missing
> tests" (regression detected because test status changed from "PASS" to "None").
> It is mostly due to oeselftest results, since oeselftest is run multiple time
> for a single build, but not with the same parameters (so not the same tests
> "sets"), so those test sets are not comparable.
> The proposed serie introduce OSELFTEST_METADATA appended to tests results when
> the TEST_TYPE is "oeselftest". An oeselftest result with those metadata looks
> like this:
> 	[...]
> 	"configuration": {
> 		"HOST_DISTRO": "fedora-36",
> 		"HOST_NAME": "fedora36-ty-3",
> 		"LAYERS": {
> 			[...]
> 		},
> 		"MACHINE": "qemux86",
> 		"STARTTIME": "20230126235248",
> 		"TESTSERIES": "qemux86",
> 		"TEST_TYPE": "oeselftest",
> 		    "run_all_tests": true,
> 		    "run_tests": null,
> 		    "skips": null,
> 		    "machine": null,
> 		    "select_tags": ["toolchain-user", "toolchain-system"],
> 		    "exclude_tags": null
> 		} 
>  	}
> 	[...]
> Additionally, the serie now makes resulttool look at a METADATA_MATCH_TABLE,
> which tells that when compared test results have a specific TEST_TYPE, it should
> look for some specific metadata to know if tests can be compared or not. It will
> then remove all the false positive in regression reports due to tests present in
> base results but not found in target results because of skipped tests/excluded
> tags
> * this serie prioritize retro-compatibility: if the base test is older (ie: it
> does not have the needed metadata), it will consider tests as "comparable"
> * additionally to tests added in oeqa test cases, some "best effort" manual
> testing has been done, with the following cases:
>   - run a basic test (e.g: `oeselftest -r tinfoils`), collect test result, break
>     test, collect result, ensure tests are compared. Change oeselftest
>     parameters, ensure tests are not compared
>   - collect base and target tests results from 4.2_M2 regression report,
>     manually add new metadata to some tests, replay regression report, ensure
>     that regressions are kept or discarded depending on the metadata

I think this is heading in the right direction. Firstly, can we put
some kind of test script into OE-Core for making debugging/testing this

I'm wondering if we can take some of the code from qa_send_email and
move it into OE-Core such that I could do something like:

show-regression-report 4.2_M1 4.2_M2

which would then resolve those two tags to commits, find the
testresults repo, fetch the data depth1 then call resulttool regression
on them.

I did that manually to experiment. I realised that if we do something

    if "MACHINE" in base_configuration and "MACHINE" in target_configuration:
        if base_configuration["MACHINE"] != target_configuration["MACHINE"]:
            return False

in metadata_matches() we can skip a lot of mismatched combinations even
with the older test results. I think we also should be able to use some
pattern matching to generate a dummy OESELFTEST_METADATA section for
older data which doesn't have it. For example, the presence of meta_ide
tests indicates one particular type of test. Combined with the MACHINE
match, this should let us compare old and new data? That would mean
metadata_matches() would need to see into the actual results too.

Does that make sense?



  parent reply	other threads:[~2023-02-16  0:02 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-02-14 16:53 [PATCH v2 0/4] scripts/resulttool/regression: add metadata filtering alexis.lothore
2023-02-14 16:53 ` [PATCH v2 1/4] scripts/oe-selftest: append metadata to tests results alexis.lothore
2023-02-14 16:53 ` [PATCH v2 2/4] oeqa/selftest/resulttooltests: fix minor typo alexis.lothore
2023-02-14 16:53 ` [PATCH v2 3/4] scripts/resulttool/regression: add metadata filtering for oeselftest alexis.lothore
2023-02-14 16:53 ` [PATCH v2 4/4] oeqa/selftest/resulttool: add test for metadata filtering on regression alexis.lothore
2023-02-16  0:02 ` Richard Purdie [this message]
2023-02-16  8:56   ` [OE-core] [PATCH v2 0/4] scripts/resulttool/regression: add metadata filtering Alexis Lothoré
2023-02-16 10:33     ` Richard Purdie

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).