[autobuilder][PATCH 0/2] generate regression reports against proper releases

[autobuilder][PATCH 0/2] generate regression reports against proper releases

[Alexis Lothoré]

This work has been initiated while discussing issue #14065
(https://bugzilla.yoctoproject.org/show_bug.cgi?id=14065)
The topic initially covers ptest reports between releases, but after discussions
with R. Purdie (https://lists.yoctoproject.org/g/automated-testing/message/1211),
a general regression report reliability topic emerged. Those first
patches aims to make regression reports more useful during releases by computing
against which revision it would be relevant to check for regressions.

A few notes about those patches :
- Since it affects release process, I struggled to properly test the new behavior,
so testing has been made by stubbing send-qa-email side effects and picking some
samples of build properties and layerinfo.json from autobuilder history to run
manually send-qa-email on my machine. But I will gladly take into account any
advice or procedure to test it more thoroughly
- I also have some uncommitted python unit tests on "previous version
  computation", but current scripts directory does not allow proper unit tests
integration. If appropriate, I could submit some minor reworks to make it viable
for testing and submit those tests
- This new regression checking introduced the need to clone a wider testresults
  history, which is (very) slow on my machine, but I do not now how slow it is
on autobuilder. Could it be an issue for the delivery process ? Does it need to
be smarter on this point ?
- If appropriate, new patch could follow to allow regression reports generation
against multiple revisions : for example we could check for a milestone build
against previous milestone AND previous major release

Alexis Lothoré (2):
  scripts/send-qa-email : clone older history to allow comparing against
    older releases
  scripts/send-qa-email: Generate regression reports against most
    relevant release

 scripts/send-qa-email | 69 ++++++++++++++++++++++++++++++++-----------
 scripts/utils.py      | 47 +++++++++++++++++++++++++++++
 2 files changed, 99 insertions(+), 17 deletions(-)

-- 
2.39.0

[autobuilder][PATCH 1/2] scripts/send-qa-email : clone older history to allow comparing against older releases

[Alexis Lothoré]

In order to generate relevant regression reports, we need to use resulttool
against taggued releases, which can be way older than the last five previous
commits

Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
---
 scripts/send-qa-email | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/send-qa-email b/scripts/send-qa-email
index d2f63dc..c2c5a4b 100755
--- a/scripts/send-qa-email
+++ b/scripts/send-qa-email
@@ -78,10 +78,10 @@ if 'poky' in repos and os.path.exists(resulttool) and args.results_dir:
         elif basebranch:
             cloneopts = ["--branch", basebranch]
         try:
-            subprocess.check_call(["git", "clone", "git@push.yoctoproject.org:yocto-testresults", tempdir, "--depth", "5"] + cloneopts)
+            subprocess.check_call(["git", "clone", "git@push.yoctoproject.org:yocto-testresults", tempdir] + cloneopts)
         except subprocess.CalledProcessError:
             print("No comparision branch found, falling back to master")
-            subprocess.check_call(["git", "clone", "git@push.yoctoproject.org:yocto-testresults", tempdir, "--depth", "5"])
+            subprocess.check_call(["git", "clone", "git@push.yoctoproject.org:yocto-testresults", tempdir])
 
         # If the base comparision branch isn't present regression comparision won't work
         # at least until we can tell the tool to ignore internal branch information
-- 
2.39.0

[autobuilder][PATCH 2/2] scripts/send-qa-email: Generate regression reports against most relevant release

[Alexis Lothoré]

Instead of only generating regressions reports against HEAD of relevant branch, compute
most relevant tag (ie : release) against which we can check for regressions. General rules
introduced are the following :
- milestone release is checked against previous milestone if possible, otherwise
  against major release
- point release  is checked against previous point release if possible,
  otherwise against major release
- major release is checked against previous major release
- a non release build is checked against base branch
Examples :
- 4.1.2.rc1 is checked against yocto-4.1.1
- 4.1.2 is checked against yocto-4.1.1
- 4.1.1.rc1 is checked against yocto-4.1
- 4.1.1 is checked against yocto-4.1
- 4.1 is checked against yocto-4.0
- 4.1.rc4 is checked against yocto-4.0
- 4.1_M2.rc1 is checked against 4.1_M1
- 4.1_M2 is checked against 4.1_M1
- 4.1_M1.rc1 is checked against yocto-4.0
- 4.1_M1 is checked against yocto-4.0

Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
---
 scripts/send-qa-email | 65 +++++++++++++++++++++++++++++++++----------
 scripts/utils.py      | 47 +++++++++++++++++++++++++++++++
 2 files changed, 97 insertions(+), 15 deletions(-)

diff --git a/scripts/send-qa-email b/scripts/send-qa-email
index c2c5a4b..c2bdeb5 100755
--- a/scripts/send-qa-email
+++ b/scripts/send-qa-email
@@ -49,6 +49,54 @@ if os.path.exists(buildtoolsdir):
 
 repodir = os.path.dirname(args.repojson) + "/build/repos"
 
+def get_previous_tag(targetrepodir, version):
+    previousversion = None
+    previousmilestone = None
+    if version:
+        compareversion, comparemilestone, _ = utils.get_version_from_string(version)
+        print(f"Searching before {compareversion}")
+        compareversionminor = compareversion[-1]
+        # After ignoring rc part, if we get a minor to 0 on point release (e.g 4.0.0),
+        # reject last digit since such versions do not exist
+        if len(compareversion) == 3 and compareversionminor == 0:
+            compareversion = compareversion[:-1]
+
+        # Process milestone if not first in current release
+        if comparemilestone and comparemilestone > 1:
+            previousversion = compareversion
+            previousmilestone = comparemilestone-1
+        # Process first milestone or release if not first in major release
+        elif compareversionminor > 0:
+            previousversion = compareversion[:-1] + [compareversion[-1] - 1]
+        # Otherwise : format it as tag (which must exist) and search previous tag
+        else:
+            comparetagstring = utils.get_tag_from_version(compareversion, comparemilestone)
+            return subprocess.check_output(["git", "describe", "--abbrev=0", comparetagstring + "^"], cwd=targetrepodir).decode('utf-8').strip()
+
+        return utils.get_tag_from_version(previousversion, previousmilestone)
+
+    # All other cases : merely check against latest tag reachable
+    defaultbaseversion, _, _ = utils.get_version_from_string(subprocess.check_output(["git", "describe", "--abbrev=0"], cwd=targetrepodir).decode('utf-8').strip())
+    return utils.get_tag_from_version(defaultbaseversion, None)
+
+def get_sha1(targetrepodir, revision):
+    return subprocess.check_output(["git", "rev-list", "-n", "1", revision], cwd=targetrepodir).decode('utf-8').strip()
+
+def generate_regression_report(targetrepodir, basebranch, resultdir, yoctoversion):
+    baseversion = get_previous_tag(targetrepodir, yoctoversion)
+    baserevision = get_sha1(targetrepodir, baseversion)
+    comparerevision = get_sha1(targetrepodir, basebranch)
+    print(f"Compare version : {basebranch} ({comparerevision})")
+    print(f"Base tag : {baseversion} ({baserevision})")
+
+    try:
+        regreport = subprocess.check_output([resulttool, "regression-git", "-B", basebranch, "--commit", baserevision, "--commit2", comparerevision, resultdir])
+        with open(args.results_dir + "/testresult-regressions-report.txt", "wb") as f:
+           f.write(regreport)
+    except subprocess.CalledProcessError as e:
+        error = str(e)
+        print(f"Error while generating report between {baserevision} and {comparerevision} : {error}")
+
 if 'poky' in repos and os.path.exists(resulttool) and args.results_dir:
     # Need the finalised revisions (not 'HEAD')
     targetrepodir = "%s/poky" % (repodir)
@@ -56,16 +104,7 @@ if 'poky' in repos and os.path.exists(resulttool) and args.results_dir:
     branch = repos['poky']['branch']
     repo = repos['poky']['url']
 
-    extraopts = None
     basebranch, comparebranch = utils.getcomparisonbranch(ourconfig, repo, branch)
-    if basebranch:
-        extraopts = " --branch %s --commit %s" % (branch, revision)
-    if comparebranch:
-        extraopts = extraopts + " --branch2 %s" % (comparebranch)
-    elif basebranch:
-        print("No comparision branch found, comparing to %s" % basebranch)
-        extraopts = extraopts + " --branch2 %s" % basebranch
-
     report = subprocess.check_output([resulttool, "report", args.results_dir])
     with open(args.results_dir + "/testresult-report.txt", "wb") as f:
         f.write(report)
@@ -94,8 +133,6 @@ if 'poky' in repos and os.path.exists(resulttool) and args.results_dir:
                 subprocess.check_call(["git", "checkout", "master"], cwd=tempdir)
                 subprocess.check_call(["git", "branch", basebranch], cwd=tempdir)
                 subprocess.check_call(["git", "checkout", basebranch], cwd=tempdir)
-                extraopts = None
-
         subprocess.check_call([resulttool, "store", args.results_dir, tempdir])
         if comparebranch:
             subprocess.check_call(["git", "push", "--all", "--force"], cwd=tempdir)
@@ -104,10 +141,8 @@ if 'poky' in repos and os.path.exists(resulttool) and args.results_dir:
             subprocess.check_call(["git", "push", "--all"], cwd=tempdir)
             subprocess.check_call(["git", "push", "--tags"], cwd=tempdir)
 
-        if extraopts:
-            regreport = subprocess.check_output([resulttool, "regression-git", tempdir] + extraopts.split())
-            with open(args.results_dir + "/testresult-regressions-report.txt", "wb") as f:
-                f.write(regreport)
+        if basebranch:
+            generate_regression_report(targetrepodir, basebranch, tempdir, args.release)
 
     finally:
         subprocess.check_call(["rm", "-rf",  tempdir])
diff --git a/scripts/utils.py b/scripts/utils.py
index c0ad14e..444b3ab 100644
--- a/scripts/utils.py
+++ b/scripts/utils.py
@@ -478,3 +478,50 @@ def setup_buildtools_tarball(ourconfig, workername, btdir, checkonly=False):
                     pass
             subprocess.check_call(["bash", btdlpath, "-d", btdir, "-y"])
         enable_buildtools_tarball(btdir)
+
+def get_string_from_version(version, milestone=None, rc=None):
+    """ Point releases finishing by 0 (e.g 4.0.0, 4.1.0) do no exists,
+    those are major releases
+    """
+    if len(version) == 3 and version[-1] == 0:
+        version = version[:-1]
+
+    result = ".".join(list(map(str, version)))
+    if milestone:
+        result += "_M" + str(milestone)
+    if rc:
+        result += ".rc" + str(rc)
+    return result
+
+def get_tag_from_version(version, milestone):
+    if not milestone:
+        return "yocto-" + get_string_from_version(version, milestone)
+    return get_string_from_version(version, milestone)
+
+
+def get_version_from_string(raw_version):
+    """ Get version as list of int from raw_version.
+
+    Raw version _can_ be prefixed by "yocto-",
+    Raw version _can_ be suffixed by "_MX"
+    Raw version _can_ be suffixed by ".rcY"
+    """
+    version = None
+    milestone = None
+    rc = None
+    if raw_version[:6] == "yocto-":
+        raw_version = raw_version[6:]
+    raw_version = raw_version.split(".")
+    if raw_version[-1][:2] == "rc":
+        rc = int(raw_version[-1][-1])
+        raw_version = raw_version[:-1]
+    if raw_version[-1][-3:-1] == "_M":
+        milestone = int(raw_version[-1][-1])
+        raw_version = raw_version[:-1] + [raw_version[-1][:-3]]
+    version = list(map(int, raw_version))
+    """ Point releases finishing by 0 (e.g 4.0.0, 4.1.0) do no exists,
+    those are major releases
+    """
+    if len(version) == 3 and version[-1] == 0:
+        version = version[:-1]
+    return version, milestone, rc
\ No newline at end of file
-- 
2.39.0

Re: [yocto] [autobuilder][PATCH 1/2] scripts/send-qa-email : clone older history to allow comparing against older releases

[Richard Purdie]

On Fri, 2023-01-13 at 16:06 +0100, Alexis Lothoré via
lists.yoctoproject.org wrote:
> In order to generate relevant regression reports, we need to use resulttool
> against taggued releases, which can be way older than the last five previous
> commits
> 
> Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
> ---
>  scripts/send-qa-email | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/scripts/send-qa-email b/scripts/send-qa-email
> index d2f63dc..c2c5a4b 100755
> --- a/scripts/send-qa-email
> +++ b/scripts/send-qa-email
> @@ -78,10 +78,10 @@ if 'poky' in repos and os.path.exists(resulttool) and args.results_dir:
>          elif basebranch:
>              cloneopts = ["--branch", basebranch]
>          try:
> -            subprocess.check_call(["git", "clone", "git@push.yoctoproject.org:yocto-testresults", tempdir, "--depth", "5"] + cloneopts)
> +            subprocess.check_call(["git", "clone", "git@push.yoctoproject.org:yocto-testresults", tempdir] + cloneopts)
>          except subprocess.CalledProcessError:
>              print("No comparision branch found, falling back to master")
> -            subprocess.check_call(["git", "clone", "git@push.yoctoproject.org:yocto-testresults", tempdir, "--depth", "5"])
> +            subprocess.check_call(["git", "clone", "git@push.yoctoproject.org:yocto-testresults", tempdir])
>  
>          # If the base comparision branch isn't present regression comparision won't work
>          # at least until we can tell the tool to ignore internal branch information

That is a somewhat "easy" option but it might not be practical.

The yocto-testresults repo is large. I tried a clone here and it is
taking rather a while, it starts ok but then starts taking a long time
with a lot of data transfer (over a GB so far) :(.

Can we check the tags with something like ls-remote and then only fetch
what we need to compare against?

Cheers,

Richard

Re: [yocto] [autobuilder][PATCH 0/2] generate regression reports against proper releases

[Richard Purdie]

On Fri, 2023-01-13 at 16:06 +0100, Alexis Lothoré via
lists.yoctoproject.org wrote:
> This work has been initiated while discussing issue #14065
> (https://bugzilla.yoctoproject.org/show_bug.cgi?id=14065)
> The topic initially covers ptest reports between releases, but after discussions
> with R. Purdie (https://lists.yoctoproject.org/g/automated-testing/message/1211),
> a general regression report reliability topic emerged. Those first
> patches aims to make regression reports more useful during releases by computing
> against which revision it would be relevant to check for regressions.
> 
> A few notes about those patches :
> - Since it affects release process, I struggled to properly test the new behavior,
> so testing has been made by stubbing send-qa-email side effects and picking some
> samples of build properties and layerinfo.json from autobuilder history to run
> manually send-qa-email on my machine. But I will gladly take into account any
> advice or procedure to test it more thoroughly
> - I also have some uncommitted python unit tests on "previous version
>   computation", but current scripts directory does not allow proper unit tests
> integration. If appropriate, I could submit some minor reworks to make it viable
> for testing and submit those tests

I'm glad you mentioned this! Yes, I think it is important we add some
tests for this. We can rework these scripts as needed as the project is
their main user at this point.

> - This new regression checking introduced the need to clone a wider testresults
>   history, which is (very) slow on my machine, but I do not now how slow it is
> on autobuilder. Could it be an issue for the delivery process ? Does it need to
> be smarter on this point ?

I think it will be slow on the autobuilder and we're going to have to
do something different to make this work well. My clone still hasn't
finished and is at 5.75GB of data now (15 mins to get the last 4.5GB)
and still going.

> - If appropriate, new patch could follow to allow regression reports generation
> against multiple revisions : for example we could check for a milestone build
> against previous milestone AND previous major release

I think that is the direction we need to go so yes please!

Cheers,

Richard

Re: [yocto] [autobuilder][PATCH 1/2] scripts/send-qa-email : clone older history to allow comparing against older releases

[Richard Purdie]

On Fri, 2023-01-13 at 17:46 +0000, Richard Purdie via
lists.yoctoproject.org wrote:
> On Fri, 2023-01-13 at 16:06 +0100, Alexis Lothoré via
> lists.yoctoproject.org wrote:
> > In order to generate relevant regression reports, we need to use resulttool
> > against taggued releases, which can be way older than the last five previous
> > commits
> > 
> > Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
> > ---
> >  scripts/send-qa-email | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/scripts/send-qa-email b/scripts/send-qa-email
> > index d2f63dc..c2c5a4b 100755
> > --- a/scripts/send-qa-email
> > +++ b/scripts/send-qa-email
> > @@ -78,10 +78,10 @@ if 'poky' in repos and os.path.exists(resulttool) and args.results_dir:
> >          elif basebranch:
> >              cloneopts = ["--branch", basebranch]
> >          try:
> > -            subprocess.check_call(["git", "clone", "git@push.yoctoproject.org:yocto-testresults", tempdir, "--depth", "5"] + cloneopts)
> > +            subprocess.check_call(["git", "clone", "git@push.yoctoproject.org:yocto-testresults", tempdir] + cloneopts)
> >          except subprocess.CalledProcessError:
> >              print("No comparision branch found, falling back to master")
> > -            subprocess.check_call(["git", "clone", "git@push.yoctoproject.org:yocto-testresults", tempdir, "--depth", "5"])
> > +            subprocess.check_call(["git", "clone", "git@push.yoctoproject.org:yocto-testresults", tempdir])
> >  
> >          # If the base comparision branch isn't present regression comparision won't work
> >          # at least until we can tell the tool to ignore internal branch information
> 
> That is a somewhat "easy" option but it might not be practical.
> 
> The yocto-testresults repo is large. I tried a clone here and it is
> taking rather a while, it starts ok but then starts taking a long time
> with a lot of data transfer (over a GB so far) :(.
> 
> Can we check the tags with something like ls-remote and then only fetch
> what we need to compare against?

For reference the clone took 49 minutes and 12.45GiB so we will need to
find a different approach.

Cheers,

Richard

Re: [yocto] [autobuilder][PATCH 0/2] generate regression reports against proper releases

[alexis.lothore]

Hi Richard,

On 13/01/2023 19:01, Richard Purdie wrote:
> On Fri, 2023-01-13 at 16:06 +0100, Alexis Lothoré via
> lists.yoctoproject.org wrote:
>> - I also have some uncommitted python unit tests on "previous version
>>   computation", but current scripts directory does not allow proper 
>> unit tests
>> integration. If appropriate, I could submit some minor reworks to make 
>> it viable
>> for testing and submit those tests
> 
> I'm glad you mentioned this! Yes, I think it is important we add some
> tests for this. We can rework these scripts as needed as the project is
> their main user at this point.
Good, I will then add some minor rework to allow introducing unit tests 
alongside
the version computation in the next revision
> 
>> - This new regression checking introduced the need to clone a wider 
>> testresults
>>   history, which is (very) slow on my machine, but I do not now how 
>> slow it is
>> on autobuilder. Could it be an issue for the delivery process ? Does 
>> it need to
>> be smarter on this point ?
> 
> I think it will be slow on the autobuilder and we're going to have to
> do something different to make this work well. My clone still hasn't
> finished and is at 5.75GB of data now (15 mins to get the last 4.5GB)
> and still going.

OK. I observed a quite similar behaviour while pulling manually the 
whole repository,
but I thought it might be linked to network issue on my side. I will add 
a smarter way
based on your suggestion (git ls-remote)

>> - If appropriate, new patch could follow to allow regression reports 
>> generation
>> against multiple revisions : for example we could check for a 
>> milestone build
>> against previous milestone AND previous major release
> 
> I think that is the direction we need to go so yes please!

Noted. If it is OK for you, I will keep this last point for another set 
of patches, once this
initial work is approved and merged.

Regards,

-- 
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com