[yocto-autobuilder-helper][dunfell 00/23] Patch review

[yocto-autobuilder-helper][dunfell 00/23] Patch review

[Steve Sakoman]

Please review the following patches for dunfell and have comments back by end of day Friday.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1998

with the exception of an autobuilder intermittent issue for qemuppc, which passed on
subsequent retest:

https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/3217

The following changes since commit ef52b284e8cbe90c18fdab6a0d6fa8095a2c4ed9:

  send-qa-email: Save the QA email in case it doesn't reach the mailing lists. (2021-02-23 10:24:14 +0000)

are available in the Git repository at:

  git://git.yoctoproject.org/yocto-autobuilder-helper contrib/sakoman
  http://git.yoctoproject.org/cgit.cgi/yocto-autobuilder-helper/log/?h=contrib/sakoman

Richard Purdie (22):
  scripts: Add runqemu-renice.c for renicing runqemu
  scripts/generate-testresult-index: Update after 'posttrigger' renaming
    broke the index generation
  scripts/generate-testresult-index: Ensure backwards compatibility with
    older layout
  scripts/generate-testresult-index.py: Ensure we're not always
    rerunning resulttool
  scripts/generate-testresult-index: Improve index to list test reports,
    ptest and buildperf separately
  scripts/generate-testresult-index: Reorder buildhistory to improve
    display
  scripts/generate-testresult-index.py: Use bulma css to improve the
    look of the index
  scripts/run-config: Don't execute steps that don't exist!
  scripts/run-config: Ensure stepnum has a value when there are no steps
  scripts/run-config: If target is present default to 1 step
  run-config: Adapt to two pass execution
  scripts/run-config: Improve logfile naming
  scripts/run-config: Ensure logging to both logfile and stdout
  config.json/run-config: Add human readable descriptions of steps
  scripts/run-config: Remove redundant boilerplate json
  scripts/shared-repo-unpack: Add flush call to update the output more
    regularly before buildtools
  config.json/run-config: Add support for shortnames and descriptions
  config.json: Unbreak qa-extras locked sigs test
  config.json: Add further descriptions
  config.json: Use buildtools tarball on debian9
  scripts/run-config: Disable output buffering
  config.json: Split reproduciblity tests into their own target

Ross Burton (1):
  config: build and test SDKs when using package_deb

 config.json                          | 102 ++++++++++--
 scripts/generate-testresult-index.py |  83 +++++++---
 scripts/run-config                   | 235 +++++++++++++++++++--------
 scripts/runqemu-renice.c             |  44 +++++
 scripts/shared-repo-unpack           |   1 +
 scripts/utils.py                     |   5 +-
 6 files changed, 367 insertions(+), 103 deletions(-)
 create mode 100644 scripts/runqemu-renice.c

-- 
2.25.1

[yocto-autobuilder-helper][dunfell 01/23] scripts: Add runqemu-renice.c for renicing runqemu

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 838be1a00c0383b63d1ab60aa991919404b82655)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 scripts/runqemu-renice.c | 44 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)
 create mode 100644 scripts/runqemu-renice.c

diff --git a/scripts/runqemu-renice.c b/scripts/runqemu-renice.c
new file mode 100644
index 0000000..9003e35
--- /dev/null
+++ b/scripts/runqemu-renice.c
@@ -0,0 +1,44 @@
+/*
+ * Copyright (C) 2020 Richard Purdie
+ *
+ * SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Needs sudo setcap 'cap_sys_nice=ep' renice
+ */
+
+#include <sys/time.h>
+#include <sys/resource.h>
+#include <sys/syscall.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+enum {
+    IOPRIO_WHO_PROCESS = 1,
+};
+
+#define IOPRIO_CLASS_SHIFT 13
+#define IOPRIO_PRIO_MASK ((1UL << IOPRIO_CLASS_SHIFT) - 1)
+#define IOPRIO_PRIO_CLASS(mask) ((mask) >> IOPRIO_CLASS_SHIFT)
+#define IOPRIO_PRIO_DATA(mask) ((mask) & IOPRIO_PRIO_MASK)
+#define IOPRIO_PRIO_VALUE(class, data) (((class) << IOPRIO_CLASS_SHIFT) | data)
+
+int main(int argc, char *argv[])
+{
+    int pid, rc;
+    if (argc != 2) {
+        printf("Please specify only the process PID to adjust\n");
+        exit(1);
+    }
+    pid = atoi(argv[1]);
+    rc = setpriority(PRIO_PROCESS, pid, -5);
+    if (rc != 0) {
+        printf("setpriority failed: %d\n", rc);
+        exit(1);
+    }
+    rc = syscall(__NR_ioprio_set, IOPRIO_WHO_PROCESS, pid, IOPRIO_PRIO_VALUE(2, 0));
+    if (rc != 0) {
+        printf("ioprio_set failed: %d\n", rc);
+        exit(1);
+    }
+}
-- 
2.25.1

Re: [yocto] [yocto-autobuilder-helper][dunfell 01/23] scripts: Add runqemu-renice.c for renicing runqemu

[Richard Purdie]

On Wed, 2021-03-24 at 14:39 -1000, Steve Sakoman wrote:
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> (cherry picked from commit 838be1a00c0383b63d1ab60aa991919404b82655)
> Signed-off-by: Steve Sakoman <steve@sakoman.com>
> ---
>  scripts/runqemu-renice.c | 44 ++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 44 insertions(+)
>  create mode 100644 scripts/runqemu-renice.c

This is another one which is only relevant to master, it isn't used from the
older branches.

[Basically the autobuilder has some code which it pulls from the helper
but uses it from the master branch, it isn't release specific. This is 
the janitor, index code and renice offhand, there may be other bits I forget]

Cheers,

Richard

Re: [yocto] [yocto-autobuilder-helper][dunfell 01/23] scripts: Add runqemu-renice.c for renicing runqemu

[Steve Sakoman]

On Thu, Mar 25, 2021 at 4:20 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Wed, 2021-03-24 at 14:39 -1000, Steve Sakoman wrote:
> > From: Richard Purdie <richard.purdie@linuxfoundation.org>
> >
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > (cherry picked from commit 838be1a00c0383b63d1ab60aa991919404b82655)
> > Signed-off-by: Steve Sakoman <steve@sakoman.com>
> > ---
> >  scripts/runqemu-renice.c | 44 ++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 44 insertions(+)
> >  create mode 100644 scripts/runqemu-renice.c
>
> This is another one which is only relevant to master, it isn't used from the
> older branches.

OK, removed!

Steve

> [Basically the autobuilder has some code which it pulls from the helper
> but uses it from the master branch, it isn't release specific. This is
> the janitor, index code and renice offhand, there may be other bits I forget]
>
> Cheers,
>
> Richard
>

[yocto-autobuilder-helper][dunfell 02/23] scripts/generate-testresult-index: Update after 'posttrigger' renaming broke the index generation

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6d6c45cbda5fb19c40a532b1f5c9fad39a354df6)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 scripts/generate-testresult-index.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/generate-testresult-index.py b/scripts/generate-testresult-index.py
index 827395e..d372f30 100755
--- a/scripts/generate-testresult-index.py
+++ b/scripts/generate-testresult-index.py
@@ -83,9 +83,9 @@ for build in sorted(os.listdir(path), key=keygen, reverse=True):
     reldir = "./" + build + "/testresults/"
     btype = "other"
     files = os.listdir(buildpath)
-    if os.path.exists(buildpath + "/a-full"):
+    if os.path.exists(buildpath + "/a-full-posttrigger"):
         btype = "full"
-    elif os.path.exists(buildpath + "/a-quick"):
+    elif os.path.exists(buildpath + "/a-quick-posttrigger"):
         btype = "quick"
     elif len(files) == 1:
         btype = files[0]
-- 
2.25.1

[yocto-autobuilder-helper][dunfell 03/23] scripts/generate-testresult-index: Ensure backwards compatibility with older layout

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 83c2052781a9fd587fbab06702eae31951d08dec)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 scripts/generate-testresult-index.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/scripts/generate-testresult-index.py b/scripts/generate-testresult-index.py
index d372f30..7839cc0 100755
--- a/scripts/generate-testresult-index.py
+++ b/scripts/generate-testresult-index.py
@@ -83,9 +83,11 @@ for build in sorted(os.listdir(path), key=keygen, reverse=True):
     reldir = "./" + build + "/testresults/"
     btype = "other"
     files = os.listdir(buildpath)
-    if os.path.exists(buildpath + "/a-full-posttrigger"):
+    if os.path.exists(buildpath + "/a-full-posttrigger") or \
+            os.path.exists(buildpath + "/a-full"):
         btype = "full"
-    elif os.path.exists(buildpath + "/a-quick-posttrigger"):
+    elif os.path.exists(buildpath + "/a-quick-posttrigger") or \
+            os.path.exists(buildpath + "/a-quick"):
         btype = "quick"
     elif len(files) == 1:
         btype = files[0]
-- 
2.25.1

[yocto-autobuilder-helper][dunfell 04/23] scripts/generate-testresult-index.py: Ensure we're not always rerunning resulttool

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 098de71cbdfcb9d0cd24f604cf049a7a596b8513)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 scripts/generate-testresult-index.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/generate-testresult-index.py b/scripts/generate-testresult-index.py
index 7839cc0..52be26b 100755
--- a/scripts/generate-testresult-index.py
+++ b/scripts/generate-testresult-index.py
@@ -121,6 +121,9 @@ for build in sorted(os.listdir(path), key=keygen, reverse=True):
                     if logs:
                         continue
                     subprocess.check_call(["resulttool", "log", f, "--dump-ptest", f])
+                    # Ensure we don't rerun every time with a dummy log
+                    with open(f + "/resulttool-done.log", "a+") as tf:
+                        tf.write("\n")
 
 t = Template(index_templpate)
 with open(os.path.join(path, "index.html"), 'w') as f:
-- 
2.25.1

[yocto-autobuilder-helper][dunfell 05/23] scripts/generate-testresult-index: Improve index to list test reports, ptest and buildperf separately

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9f50544d000a2ee3678f7baef67473eaf51518a5)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 scripts/generate-testresult-index.py | 38 +++++++++++++++++++++++-----
 1 file changed, 31 insertions(+), 7 deletions(-)

diff --git a/scripts/generate-testresult-index.py b/scripts/generate-testresult-index.py
index 52be26b..8381d5b 100755
--- a/scripts/generate-testresult-index.py
+++ b/scripts/generate-testresult-index.py
@@ -20,8 +20,10 @@ index_templpate = """
   <td>Build</td>     
   <td>Type</td>
   <td>Branch</td>
-  <td>Report</td>
+  <td>Test Results Report</td>
   <td>Buildhistory</td>
+  <td>Performance Reports</td>
+  <td>ptest Logs</td>
 </tr>
 {% for entry in entries %}
 <tr>
@@ -34,6 +36,16 @@ index_templpate = """
      <a href="{{bh[0]}}">{{bh[1]}}</a>
    {% endfor %}
    </td>
+   <td>
+   {% for perfrep in entry[6] %}
+     <a href="{{perfrep[0]}}">{{perfrep[1]}}</a>
+   {% endfor %}
+   </td>
+   <td>
+   {% for ptest in entry[7] %}
+     <a href="{{ptest[0]}}">{{ptest[1]}}</a>
+   {% endfor %}
+   </td>
 </tr>
 {% endfor %}
 </table>
@@ -81,6 +93,7 @@ for build in sorted(os.listdir(path), key=keygen, reverse=True):
         # No test results
         continue
     reldir = "./" + build + "/testresults/"
+
     btype = "other"
     files = os.listdir(buildpath)
     if os.path.exists(buildpath + "/a-full-posttrigger") or \
@@ -91,14 +104,25 @@ for build in sorted(os.listdir(path), key=keygen, reverse=True):
         btype = "quick"
     elif len(files) == 1:
         btype = files[0]
+
     testreport = ""
     if os.path.exists(buildpath + "/testresult-report.txt"):
         testreport = reldir + "testresult-report.txt"
-    elif btype.startswith("buildperf-"):
-        try:
-            testreport = reldir + btype + "/" + os.path.basename(glob.glob(buildpath + "/" + btype + "/*.html")[0])
-        except IndexError:
-            pass
+
+    ptestlogs = []
+    ptestseen = []
+    for p in glob.glob(buildpath + "/*-ptest/*.log"):
+        if p.endswith("resulttool-done.log"):
+            continue
+        buildname = os.path.basename(os.path.dirname(p))
+        if buildname not in ptestseen:
+            ptestlogs.append((reldir + "/" + buildname + "/", buildname.replace("-ptest","")))
+            ptestseen.append(buildname)
+
+    perfreports = []
+    for p in glob.glob(buildpath + "/buildperf*/*.html"):
+        perfname = os.path.basename(os.path.dirname(p))
+        perfreports.append((reldir + "/" + perfname + "/" + os.path.basename(p), perfname.replace("buildperf-","")))
 
     buildhistory = []
     if os.path.exists(buildpath + "/qemux86-64/buildhistory.txt"):
@@ -109,7 +133,7 @@ for build in sorted(os.listdir(path), key=keygen, reverse=True):
 
     branch = get_build_branch(buildpath)
 
-    entries.append((build, reldir, btype, testreport, branch, buildhistory))
+    entries.append((build, reldir, btype, testreport, branch, buildhistory, perfreports, ptestlogs))
 
     # Also ensure we have saved out log data for ptest runs to aid debugging
     if "ptest" in btype or btype in ["full", "quick"]:
-- 
2.25.1

[yocto-autobuilder-helper][dunfell 06/23] scripts/generate-testresult-index: Reorder buildhistory to improve display

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Also, use the artefacts directory in the main link, not direct to testresults.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ab485e89bd535aae17a5cb137e81345fa6ad6c3a)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 scripts/generate-testresult-index.py | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/scripts/generate-testresult-index.py b/scripts/generate-testresult-index.py
index 8381d5b..c3fea84 100755
--- a/scripts/generate-testresult-index.py
+++ b/scripts/generate-testresult-index.py
@@ -21,9 +21,9 @@ index_templpate = """
   <td>Type</td>
   <td>Branch</td>
   <td>Test Results Report</td>
-  <td>Buildhistory</td>
   <td>Performance Reports</td>
   <td>ptest Logs</td>
+  <td>Buildhistory</td>
 </tr>
 {% for entry in entries %}
 <tr>
@@ -32,11 +32,6 @@ index_templpate = """
    <td>{% if entry[4] %} {{entry[4]}}{% endif %}</td>
    <td> {% if entry[3] %}<a href="{{entry[3]}}">Report</a>{% endif %} </td>
    <td>
-   {% for bh in entry[5] %}
-     <a href="{{bh[0]}}">{{bh[1]}}</a>
-   {% endfor %}
-   </td>
-   <td>
    {% for perfrep in entry[6] %}
      <a href="{{perfrep[0]}}">{{perfrep[1]}}</a>
    {% endfor %}
@@ -46,6 +41,11 @@ index_templpate = """
      <a href="{{ptest[0]}}">{{ptest[1]}}</a>
    {% endfor %}
    </td>
+   <td>
+   {% for bh in entry[5] %}
+     <a href="{{bh[0]}}">{{bh[1]}}</a>
+   {% endfor %}
+   </td>
 </tr>
 {% endfor %}
 </table>
@@ -92,7 +92,7 @@ for build in sorted(os.listdir(path), key=keygen, reverse=True):
     if not os.path.exists(buildpath):
         # No test results
         continue
-    reldir = "./" + build + "/testresults/"
+    reldir = "./" + build + "/"
 
     btype = "other"
     files = os.listdir(buildpath)
@@ -107,7 +107,7 @@ for build in sorted(os.listdir(path), key=keygen, reverse=True):
 
     testreport = ""
     if os.path.exists(buildpath + "/testresult-report.txt"):
-        testreport = reldir + "testresult-report.txt"
+        testreport = reldir + "testresults/testresult-report.txt"
 
     ptestlogs = []
     ptestseen = []
@@ -116,20 +116,20 @@ for build in sorted(os.listdir(path), key=keygen, reverse=True):
             continue
         buildname = os.path.basename(os.path.dirname(p))
         if buildname not in ptestseen:
-            ptestlogs.append((reldir + "/" + buildname + "/", buildname.replace("-ptest","")))
+            ptestlogs.append((reldir + "testresults/" + buildname + "/", buildname.replace("-ptest","")))
             ptestseen.append(buildname)
 
     perfreports = []
     for p in glob.glob(buildpath + "/buildperf*/*.html"):
         perfname = os.path.basename(os.path.dirname(p))
-        perfreports.append((reldir + "/" + perfname + "/" + os.path.basename(p), perfname.replace("buildperf-","")))
+        perfreports.append((reldir + "testresults/" + perfname + "/" + os.path.basename(p), perfname.replace("buildperf-","")))
 
     buildhistory = []
     if os.path.exists(buildpath + "/qemux86-64/buildhistory.txt"):
-        buildhistory.append((reldir + "/qemux86-64/buildhistory.txt", "qemux86-64"))
+        buildhistory.append((reldir + "testresults/qemux86-64/buildhistory.txt", "qemux86-64"))
 
     if os.path.exists(buildpath + "/qemuarm/buildhistory.txt"):
-        buildhistory.append((reldir + "/qemuarm/buildhistory.txt", "qemuarm"))
+        buildhistory.append((reldir + "testresults/qemuarm/buildhistory.txt", "qemuarm"))
 
     branch = get_build_branch(buildpath)
 
-- 
2.25.1

[yocto-autobuilder-helper][dunfell 07/23] scripts/generate-testresult-index.py: Use bulma css to improve the look of the index

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 4b8eab92ee1f68ec8cd680c62e40b17006fa6efc)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 scripts/generate-testresult-index.py | 34 ++++++++++++++++++++--------
 1 file changed, 24 insertions(+), 10 deletions(-)

diff --git a/scripts/generate-testresult-index.py b/scripts/generate-testresult-index.py
index c3fea84..d9d577e 100755
--- a/scripts/generate-testresult-index.py
+++ b/scripts/generate-testresult-index.py
@@ -13,18 +13,29 @@ import subprocess
 from jinja2 import Template
 
 index_templpate = """
-<h1>Index of autobuilder test results</h1>
-    
-<table>
+<!DOCTYPE html>
+<html>
+<head>
+  <meta charset="utf-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1">
+  <title>Index of autobuilder test results</title>
+  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bulma@0.9.1/css/bulma.min.css">
+</head>
+<body>
+ 
+<table class="table is-narrow is-striped">
+<thead>
 <tr>
-  <td>Build</td>     
-  <td>Type</td>
-  <td>Branch</td>
-  <td>Test Results Report</td>
-  <td>Performance Reports</td>
-  <td>ptest Logs</td>
-  <td>Buildhistory</td>
+  <th>Build</th>
+  <th>Type</th>
+  <th>Branch</th>
+  <th>Test Results Report</th>
+  <th>Performance Reports</th>
+  <th>ptest Logs</th>
+  <th>Buildhistory</th>
 </tr>
+</thead>
+<tdata>
 {% for entry in entries %}
 <tr>
    <td><a href="{{entry[1]}}">{{entry[0]}}</a></td>
@@ -48,7 +59,10 @@ index_templpate = """
    </td>
 </tr>
 {% endfor %}
+</tdata>
 </table>
+</body>
+</html>
 """
 
 def parse_args(argv=None):
-- 
2.25.1

Re: [yocto] [yocto-autobuilder-helper][dunfell 07/23] scripts/generate-testresult-index.py: Use bulma css to improve the look of the index

[Richard Purdie]

On Wed, 2021-03-24 at 14:39 -1000, Steve Sakoman wrote:
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> (cherry picked from commit 4b8eab92ee1f68ec8cd680c62e40b17006fa6efc)
> Signed-off-by: Steve Sakoman <steve@sakoman.com>
> ---
>  scripts/generate-testresult-index.py | 34 ++++++++++++++++++++--------
>  1 file changed, 24 insertions(+), 10 deletions(-)

I'm torn on this and the other generate-testresult-index patches. These
are only ever run from the master branch so in the context of dunfell,
they don't matter. Equally, they therefore don't hurt anything...

Cheers,

Richard

Re: [yocto] [yocto-autobuilder-helper][dunfell 07/23] scripts/generate-testresult-index.py: Use bulma css to improve the look of the index

[Steve Sakoman]

On Thu, Mar 25, 2021 at 4:04 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Wed, 2021-03-24 at 14:39 -1000, Steve Sakoman wrote:
> > From: Richard Purdie <richard.purdie@linuxfoundation.org>
> >
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > (cherry picked from commit 4b8eab92ee1f68ec8cd680c62e40b17006fa6efc)
> > Signed-off-by: Steve Sakoman <steve@sakoman.com>
> > ---
> >  scripts/generate-testresult-index.py | 34 ++++++++++++++++++++--------
> >  1 file changed, 24 insertions(+), 10 deletions(-)
>
> I'm torn on this and the other generate-testresult-index patches. These
> are only ever run from the master branch so in the context of dunfell,
> they don't matter. Equally, they therefore don't hurt anything...

I wasn't really sure whether these ran in dunfell, thanks for
confirming that they don't.

I'll drop the generate-test-result-index patches from the pull request too.

Steve

[yocto-autobuilder-helper][dunfell 08/23] scripts/run-config: Don't execute steps that don't exist!

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 290e1bc2ee18d5fa88aca84125fb6691db3db5f9)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 scripts/run-config | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/run-config b/scripts/run-config
index ce6072c..ff56fbe 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -81,7 +81,7 @@ elif args.build_type == "full":
         ourconfig["HELPERSTMACHTARGS"] = "-a -t machine -t toolchain-user"
 
 # Find out the number of steps this target has
-maxsteps = 1
+maxsteps = 0
 if args.target in ourconfig['overrides']:
     for v in ourconfig['overrides'][args.target]:
         if v.startswith("step"):
-- 
2.25.1

[yocto-autobuilder-helper][dunfell 09/23] scripts/run-config: Ensure stepnum has a value when there are no steps

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit afb6c5a88773d4da4a8dfd88f19654ca585efc95)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 scripts/run-config | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/run-config b/scripts/run-config
index ff56fbe..e600bf9 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -82,6 +82,7 @@ elif args.build_type == "full":
 
 # Find out the number of steps this target has
 maxsteps = 0
+stepnum = 0
 if args.target in ourconfig['overrides']:
     for v in ourconfig['overrides'][args.target]:
         if v.startswith("step"):
-- 
2.25.1

[yocto-autobuilder-helper][dunfell 10/23] scripts/run-config: If target is present default to 1 step

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e183db413f3b67e0d45a2a9a697aa36b6c90601f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 scripts/run-config | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/run-config b/scripts/run-config
index e600bf9..0b663df 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -84,6 +84,7 @@ elif args.build_type == "full":
 maxsteps = 0
 stepnum = 0
 if args.target in ourconfig['overrides']:
+    maxsteps = 1
     for v in ourconfig['overrides'][args.target]:
         if v.startswith("step"):
             n = int(v[4:])
-- 
2.25.1

[yocto-autobuilder-helper][dunfell 11/23] run-config: Adapt to two pass execution

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 23d65680f8019bccc3fce20381dfcf49f265f601)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 scripts/run-config | 171 +++++++++++++++++++++++++++++++++------------
 scripts/utils.py   |   5 +-
 2 files changed, 130 insertions(+), 46 deletions(-)

diff --git a/scripts/run-config b/scripts/run-config
index 0b663df..116dd49 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -52,6 +52,19 @@ parser.add_argument('--workername',
                     action='store',
                     default=None,
                     help="the name of the worker the build is running on")
+parser.add_argument('-j', '--json-outputfile',
+                    action='store',
+                    default="",
+                    help="the file to store json information about the build in")
+parser.add_argument('--stepname',
+                    action='store',
+                    default=None,
+                    help="the name of the step to run")
+parser.add_argument('--phase',
+                    action='store',
+                    default=None,
+                    help="the phase of the step to run")
+
 
 
 args = parser.parse_args()
@@ -94,7 +107,19 @@ if args.target in ourconfig['overrides']:
 
 hp.printheader("Target task %s has %d steps" % (args.target, maxsteps))
 
-utils.setup_buildtools_tarball(ourconfig, args.workername, args.builddir + "/../buildtools")
+jcfg = False
+if args.json_outputfile:
+    jsonconfig = []
+    jcfg = True
+
+if jcfg:
+    buildtools = utils.setup_buildtools_tarball(ourconfig, args.workername, None, checkonly=True)
+    if buildtools:
+        jsonconfig.append({"name" : "buildtools", "description" : "Extract and setup buildtools tarball", "phase" : "init"})
+else:
+    utils.setup_buildtools_tarball(ourconfig, args.workername, args.builddir + "/../buildtools")
+    if args.phase == "init" and args.stepname == "buildtools":
+        sys.exit(0)
 
 logconfig = args.builddir + "/../bitbake/contrib/autobuilderlog.json"
 print("Using BB_LOGCONFIG=%s" % logconfig)
@@ -181,70 +206,126 @@ def runcmd(cmd, *args, **kwargs):
 
 bh_path, remoterepo, remotebranch, baseremotebranch = utils.getbuildhistoryconfig(ourconfig, args.builddir, args.target, args.reponame, args.branchname, 1)
 if bh_path:
-    runcmd([os.path.join(scriptsdir, "buildhistory-init"), bh_path, remoterepo, remotebranch, baseremotebranch])
-
-for stepnum in range(1, maxsteps + 1):
+    if jcfg:
+        jsonconfig.append({"name" : "buildhistory-init", "description" : "Initialize buildhistory", "phase" : "init"})
+if args.phase == "init" and args.stepname == "buildhistory-init":
+    if bh_path:
+        runcmd([os.path.join(scriptsdir, "buildhistory-init"), bh_path, remoterepo, remotebranch, baseremotebranch])
+    sys.exit(0)
+
+def handle_stepnum(stepnum):
     # Add any layers specified
     layers = utils.getconfiglist("ADDLAYER", ourconfig, args.target, stepnum)
-    for layer in layers:
-        bitbakecmd(args.builddir, "bitbake-layers add-layer %s" % layer, report, stepnum, 'a')
+    if jcfg:
+        if layers:
+            jsonconfig.append({"name" : "add-layers", "description" : "Adding layers %s" % str(layers), "phase" : str(stepnum)})
+    elif args.stepname == "add-layers":
+        for layer in layers:
+            bitbakecmd(args.builddir, "bitbake-layers add-layer %s" % layer, report, stepnum, 'a')
 
     flush()
+
     # Generate the configuration files needed for this step
     if utils.getconfigvar("WRITECONFIG", ourconfig, args.target, stepnum):
-        runcmd([scriptsdir + "/setup-config", args.target, str(stepnum - 1), args.builddir, args.branchname, args.reponame, "-s", args.sstateprefix, "-b", args.buildappsrcrev])
+        if jcfg:
+            jsonconfig.append({"name" : "write-config", "description" : "Writing configuration files", "phase" : str(stepnum)})
+        elif args.stepname == "write-config":
+            runcmd([scriptsdir + "/setup-config", args.target, str(stepnum - 1), args.builddir, args.branchname, args.reponame, "-s", args.sstateprefix, "-b", args.buildappsrcrev])
 
     # Execute the targets for this configuration
     targets = utils.getconfigvar("BBTARGETS", ourconfig, args.target, stepnum)
     if targets:
-        hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, targets))
-        bitbakecmd(args.builddir, "bitbake %s -k" % targets, report, stepnum, 'b')
+        if jcfg:
+            jsonconfig.append({"name" : "build-targets", "description" : "Building targets %s" % str(targets), "phase" : str(stepnum)})
+        elif args.stepname == "build-targets":
+            hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, targets))
+            bitbakecmd(args.builddir, "bitbake %s -k" % targets, report, stepnum, 'b')
 
     # Execute the sanity targets for this configuration
     sanitytargets = utils.getconfigvar("SANITYTARGETS", ourconfig, args.target, stepnum)
     if sanitytargets:
-        hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, sanitytargets))
-        bitbakecmd(args.builddir, "%s/checkvnc; DISPLAY=:1 bitbake %s -k" % (scriptsdir, sanitytargets), report, stepnum, 'c')
+        if jcfg:
+            jsonconfig.append({"name" : "test-targets", "description" : "Running OEQA test targets %s" % str(sanitytargets), "phase" : str(stepnum)})
+        elif args.stepname == "test-targets":
+            hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, sanitytargets))
+            bitbakecmd(args.builddir, "%s/checkvnc; DISPLAY=:1 bitbake %s -k" % (scriptsdir, sanitytargets), report, stepnum, 'c')
 
     # Run any extra commands specified
     cmds = utils.getconfiglist("EXTRACMDS", ourconfig, args.target, stepnum)
-    for cmd in cmds:
-        hp.printheader("Step %s/%s: Running command %s" % (stepnum, maxsteps, cmd))
-        bitbakecmd(args.builddir, cmd, report, stepnum, 'd')
+    if jcfg:
+        if cmds:
+            jsonconfig.append({"name" : "cmds", "description" : "Running bitbake environment commands %s" % str(cmds), "phase" : str(stepnum)})
+    elif args.stepname == "cmds":
+        for cmd in cmds:
+            hp.printheader("Step %s/%s: Running command %s" % (stepnum, maxsteps, cmd))
+            bitbakecmd(args.builddir, cmd, report, stepnum, 'd')
+
     cmds = utils.getconfiglist("EXTRAPLAINCMDS", ourconfig, args.target, stepnum)
-    for cmd in cmds:
-        hp.printheader("Step %s/%s: Running 'plain' command %s" % (stepnum, maxsteps, cmd))
-        bitbakecmd(args.builddir, cmd, report, stepnum, 'd', oeenv=False)
-
-    # Remove any layers we added in a reverse order
-    for layer in reversed(layers):
-        bitbakecmd(args.builddir, "bitbake-layers remove-layer %s" % layer, report, stepnum, 'a')
-
-if args.publish_dir:
-    hp.printheader("Running publish artefacts")
-    runcmd([scriptsdir + "/publish-artefacts", args.builddir, args.publish_dir, args.target])
-
-if args.results_dir:
-    hp.printheader("Running results collection")
-    runcmd([scriptsdir + "/collect-results", args.builddir, args.results_dir, args.target])
-
-if args.build_url and utils.getconfigvar("SENDERRORS", ourconfig, args.target, stepnum):
-    hp.printheader("Sending any error reports")
-    runcmd([scriptsdir + "/upload-error-reports", args.builddir, args.build_url])
-
-if args.builddir and os.path.exists(args.builddir):
-    # Clean up our build directory if things were successful and we're not publishing anything
-    # (keep published builds around for longer just in case we need them)
-    if not finalret and not args.publish_dir:
-        runcmd([scriptsdir + "/../janitor/clobberdir", args.builddir])
-    else:
-        # Rename any completed build directory so that other builds can't reference paths within it
+    if jcfg:
+        if cmds:
+            jsonconfig.append({"name" : "plain-cmds", "description" : "Running commands %s" % str(cmds), "phase" : str(stepnum)})
+    elif args.stepname == "plain-cmds":
+        for cmd in cmds:
+            hp.printheader("Step %s/%s: Running 'plain' command %s" % (stepnum, maxsteps, cmd))
+            bitbakecmd(args.builddir, cmd, report, stepnum, 'd', oeenv=False)
+
+    if jcfg:
+        if layers:
+            jsonconfig.append({"name" : "remove-layers", "description" : "Removing layers %s" % str(layers), "phase" : str(stepnum)})
+    elif args.stepname == "remove-layers":
+        # Remove any layers we added in a reverse order
+        for layer in reversed(layers):
+            bitbakecmd(args.builddir, "bitbake-layers remove-layer %s" % layer, report, stepnum, 'a')
+
+    if not jcfg:
+        sys.exit(finalret)
+
+if jcfg:
+    for stepnum in range(1, maxsteps + 1):
+        handle_stepnum(stepnum)
+else:
+    try:
+        stepnum = int(args.phase)
+    except ValueError:
+        stepnum = None
+
+    if stepnum is not None:
+        handle_stepnum(stepnum)
+
+
+if jcfg:
+    jsonconfig.append({"name" : "publish", "description" : "Publishing artefacts", "phase" : "finish"})
+elif args.phase == "finish" and args.stepname == "publish":
+    if args.publish_dir:
+        hp.printheader("Running publish artefacts")
+        runcmd([scriptsdir + "/publish-artefacts", args.builddir, args.publish_dir, args.target])
+    sys.exit(0)
+
+if jcfg:
+    jsonconfig.append({"name" : "collect-results", "description" : "Collecting result files", "phase" : "finish"})
+elif args.phase == "finish" and args.stepname == "collect-results":
+    if args.results_dir:
+        hp.printheader("Running results collection")
+        runcmd([scriptsdir + "/collect-results", args.builddir, args.results_dir, args.target])
+    sys.exit(0)
+
+if jcfg:
+    jsonconfig.append({"name" : "send-errors", "description" : "Sending error reports", "phase" : "finish"})
+elif args.phase == "finish" and args.stepname == "send-errors":
+    if args.build_url and utils.getconfigvar("SENDERRORS", ourconfig, args.target, stepnum):
+        hp.printheader("Sending any error reports")
+        runcmd([scriptsdir + "/upload-error-reports", args.builddir, args.build_url])
+    sys.exit(0)
+
+if jcfg:
+    jsonconfig.append({"name" : "builddir-cleanup", "description" : "Cleaning up build directory", "phase" : "finish"})
+elif args.phase == "finish" and args.stepname == "builddir-cleanup":
+    if args.builddir and os.path.exists(args.builddir):
         runcmd(["mv", args.builddir, args.builddir + "-renamed"])
 
-if finalret:
-    hp.printheader("There were %s failures" % finalret)
-    hp.printheader("Failures in logfiles: %s" % " ".join(errorlogs))
-    sys.exit(1)
+if args.json_outputfile:
+    with open(args.json_outputfile, "w") as f:
+        json.dump(jsonconfig, f, indent=4, sort_keys=True)
 
 sys.exit(0)
 
diff --git a/scripts/utils.py b/scripts/utils.py
index c7eb6c7..bf1d989 100644
--- a/scripts/utils.py
+++ b/scripts/utils.py
@@ -415,7 +415,7 @@ def enable_buildtools_tarball(btdir):
                 if line in os.environ:
                     del os.environ[line]
 
-def setup_buildtools_tarball(ourconfig, workername, btdir):
+def setup_buildtools_tarball(ourconfig, workername, btdir, checkonly=False):
     bttarball = None
     if "buildtools" in ourconfig and workername:
         btcfg = getconfig("buildtools", ourconfig)
@@ -424,6 +424,9 @@ def setup_buildtools_tarball(ourconfig, workername, btdir):
                 bttarball = btcfg[entry]
                 break
 
+    if checkonly:
+        return bttarball
+
     btenv = None
     if bttarball:
         sha256 = None
-- 
2.25.1

[yocto-autobuilder-helper][dunfell 12/23] scripts/run-config: Improve logfile naming

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 4a4c888f6618c3a7273c6dfe30b640e75e2b0de8)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 scripts/run-config | 28 ++++++++++++----------------
 1 file changed, 12 insertions(+), 16 deletions(-)

diff --git a/scripts/run-config b/scripts/run-config
index 116dd49..25a4296 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -131,11 +131,8 @@ def flush():
     sys.stdout.flush()
     sys.stderr.flush()
 
-lognum = 0
-def logname(path, stepnum, logsuffix):
-    global lognum
-    lognum += 1
-    return path + "/command.log.%s%s" % (stepnum, logsuffix)
+def logname(path, stepnum, stepname):
+    return path + "/command-%s-%s.log" % (stepnum, stepname)
 
 utils.mkdir(args.builddir)
 
@@ -146,10 +143,10 @@ utils.mkdir(errordir)
 
 errorlogs = set()
 
-def bitbakecmd(builddir, cmd, report, stepnum, logsuffix, oeenv=True):
+def bitbakecmd(builddir, cmd, report, stepnum, stepname, oeenv=True):
     global finalret
     flush()
-    log = logname(builddir, stepnum, logsuffix)
+    log = logname(builddir, stepnum, stepname)
     errordir = utils.errorreportdir(builddir)
     try:
         numreports = len(os.listdir(errordir))
@@ -176,10 +173,9 @@ def bitbakecmd(builddir, cmd, report, stepnum, logsuffix, oeenv=True):
 
     with subprocess.Popen(cmd, shell=True, cwd=builddir + "/..", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1) as p, open(log, 'ab') as f:
         for line in p.stdout:
-            if not args.quietlogging:
-                sys.stdout.buffer.write(line)
-                sys.stdout.flush()
+            sys.stdout.buffer.write(line)
             f.write(line)
+            sys.stdout.flush()
             f.flush()
         ret = p.wait()
     if ret:
@@ -221,7 +217,7 @@ def handle_stepnum(stepnum):
             jsonconfig.append({"name" : "add-layers", "description" : "Adding layers %s" % str(layers), "phase" : str(stepnum)})
     elif args.stepname == "add-layers":
         for layer in layers:
-            bitbakecmd(args.builddir, "bitbake-layers add-layer %s" % layer, report, stepnum, 'a')
+            bitbakecmd(args.builddir, "bitbake-layers add-layer %s" % layer, report, stepnum, args.stepname)
 
     flush()
 
@@ -239,7 +235,7 @@ def handle_stepnum(stepnum):
             jsonconfig.append({"name" : "build-targets", "description" : "Building targets %s" % str(targets), "phase" : str(stepnum)})
         elif args.stepname == "build-targets":
             hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, targets))
-            bitbakecmd(args.builddir, "bitbake %s -k" % targets, report, stepnum, 'b')
+            bitbakecmd(args.builddir, "bitbake %s -k" % targets, report, stepnum, args.stepname)
 
     # Execute the sanity targets for this configuration
     sanitytargets = utils.getconfigvar("SANITYTARGETS", ourconfig, args.target, stepnum)
@@ -248,7 +244,7 @@ def handle_stepnum(stepnum):
             jsonconfig.append({"name" : "test-targets", "description" : "Running OEQA test targets %s" % str(sanitytargets), "phase" : str(stepnum)})
         elif args.stepname == "test-targets":
             hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, sanitytargets))
-            bitbakecmd(args.builddir, "%s/checkvnc; DISPLAY=:1 bitbake %s -k" % (scriptsdir, sanitytargets), report, stepnum, 'c')
+            bitbakecmd(args.builddir, "%s/checkvnc; DISPLAY=:1 bitbake %s -k" % (scriptsdir, sanitytargets), report, stepnum, args.stepname)
 
     # Run any extra commands specified
     cmds = utils.getconfiglist("EXTRACMDS", ourconfig, args.target, stepnum)
@@ -258,7 +254,7 @@ def handle_stepnum(stepnum):
     elif args.stepname == "cmds":
         for cmd in cmds:
             hp.printheader("Step %s/%s: Running command %s" % (stepnum, maxsteps, cmd))
-            bitbakecmd(args.builddir, cmd, report, stepnum, 'd')
+            bitbakecmd(args.builddir, cmd, report, stepnum, args.stepname)
 
     cmds = utils.getconfiglist("EXTRAPLAINCMDS", ourconfig, args.target, stepnum)
     if jcfg:
@@ -267,7 +263,7 @@ def handle_stepnum(stepnum):
     elif args.stepname == "plain-cmds":
         for cmd in cmds:
             hp.printheader("Step %s/%s: Running 'plain' command %s" % (stepnum, maxsteps, cmd))
-            bitbakecmd(args.builddir, cmd, report, stepnum, 'd', oeenv=False)
+            bitbakecmd(args.builddir, cmd, report, stepnum, args.stepname, oeenv=False)
 
     if jcfg:
         if layers:
@@ -275,7 +271,7 @@ def handle_stepnum(stepnum):
     elif args.stepname == "remove-layers":
         # Remove any layers we added in a reverse order
         for layer in reversed(layers):
-            bitbakecmd(args.builddir, "bitbake-layers remove-layer %s" % layer, report, stepnum, 'a')
+            bitbakecmd(args.builddir, "bitbake-layers remove-layer %s" % layer, report, stepnum, args.stepname)
 
     if not jcfg:
         sys.exit(finalret)
-- 
2.25.1

[yocto-autobuilder-helper][dunfell 13/23] scripts/run-config: Ensure logging to both logfile and stdout

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b1bc4d64c2d0a7e61aea154635996b6b4a4d04c2)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 scripts/run-config | 27 ++++++++++++++++-----------
 1 file changed, 16 insertions(+), 11 deletions(-)

diff --git a/scripts/run-config b/scripts/run-config
index 25a4296..05c0579 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -153,6 +153,10 @@ def bitbakecmd(builddir, cmd, report, stepnum, stepname, oeenv=True):
     except FileNotFoundError:
         numreports = 0
 
+    def writelog(msg, a, b):
+        a.write(msg)
+        b.write(msg)
+
     if oeenv:
         cmd = ". ./oe-init-build-env; %s" % cmd
 
@@ -160,21 +164,22 @@ def bitbakecmd(builddir, cmd, report, stepnum, stepname, oeenv=True):
         print("Would run '%s'" % cmd)
         return
 
-    print("Running '%s' with output to %s" % (cmd, log))
-    flush()
+    with open(log, "a") as outf:
+        writelog("Running '%s' with output to %s\n" % (cmd, log), outf, sys.stdout)
 
-    autoconf = builddir + "/conf/auto.conf"
-    if os.path.exists(autoconf):
-        with open(autoconf, "r") as inf, open(log, "a") as outf:
-            outf.write("auto.conf settings:\n")
-            for line in inf.readlines():
-                outf.write(line)
-            outf.write("\n")
+        autoconf = builddir + "/conf/auto.conf"
+        if os.path.exists(autoconf):
+            with open(autoconf, "r") as inf, open(log, "a") as outf:
+                writelog("auto.conf settings:\n", outf, sys.stdout)
+                for line in inf.readlines():
+                    writelog(line, outf, sys.stdout)
+                writelog("\n", outf, sys.stdout)
+
+    flush()
 
     with subprocess.Popen(cmd, shell=True, cwd=builddir + "/..", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1) as p, open(log, 'ab') as f:
         for line in p.stdout:
-            sys.stdout.buffer.write(line)
-            f.write(line)
+            writelog(line, f, sys.stdout.buffer)
             sys.stdout.flush()
             f.flush()
         ret = p.wait()
-- 
2.25.1

[yocto-autobuilder-helper][dunfell 14/23] config.json/run-config: Add human readable descriptions of steps

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ce63e8f024834a670cea660c61be632191aed9b9)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 config.json        |  6 ++++++
 scripts/run-config | 30 +++++++++++++++++-------------
 2 files changed, 23 insertions(+), 13 deletions(-)

diff --git a/config.json b/config.json
index b67ef03..1fcc45d 100644
--- a/config.json
+++ b/config.json
@@ -453,6 +453,7 @@
             "MACHINE" : "qemux86-64",
             "SDKMACHINE" : "x86_64",
             "step1" : {
+                "description" : "x86_64 32bit multilib image with rpm",
                 "BBTARGETS" : "lib32-core-image-minimal",
                 "SANITYTARGETS" : "lib32-core-image-minimal:do_testimage",
                 "extravars" : [
@@ -463,6 +464,7 @@
                 ]
             },
             "step2" : {
+                "description" : "x86_64 32bit multilib image with ipk",
                 "PACKAGE_CLASSES" : "package_ipk",
                 "BBTARGETS" : "lib32-core-image-minimal",
                 "SANITYTARGETS" : "lib32-core-image-minimal:do_testimage",
@@ -474,6 +476,7 @@
                 ]
             },
             "step3" : {
+                "description" : "x86_64 64bit image and 32 bit multilibs with rpm",
                 "BBTARGETS" : "core-image-sato",
                 "SANITYTARGETS" : "core-image-sato:do_testimage",
                 "extravars" : [
@@ -486,6 +489,7 @@
                 ]
             },
             "step4" : {
+                "description" : "x86_64 64bit image and 32 bit multilibs with ipk",
                 "PACKAGE_CLASSES" : "package_ipk",
                 "BBTARGETS" : "core-image-sato",
                 "SANITYTARGETS" : "core-image-sato:do_testimage",
@@ -499,6 +503,7 @@
                 ]
             },
             "step5" : {
+                "description" : "x86 building 64bit multilib image",
                 "MACHINE" : "qemux86",
                 "SDKMACHINE" : "i686",
                 "BBTARGETS" : "lib64-core-image-sato lib64-core-image-sato-sdk",
@@ -509,6 +514,7 @@
                 ]
             },
             "step6" : {
+                "description" : "mips64 image using n32 as default",
                 "MACHINE" : "qemumips64",
                 "BBTARGETS" : "core-image-minimal core-image-minimal:do_populate_sdk",
                 "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-minimal:do_testsdk",
diff --git a/scripts/run-config b/scripts/run-config
index 05c0579..89506f5 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -115,7 +115,7 @@ if args.json_outputfile:
 if jcfg:
     buildtools = utils.setup_buildtools_tarball(ourconfig, args.workername, None, checkonly=True)
     if buildtools:
-        jsonconfig.append({"name" : "buildtools", "description" : "Extract and setup buildtools tarball", "phase" : "init"})
+        jsonconfig.append({"name" : "buildtools", "bbname" : "Extract and setup buildtools tarball", "phase" : "init"})
 else:
     utils.setup_buildtools_tarball(ourconfig, args.workername, args.builddir + "/../buildtools")
     if args.phase == "init" and args.stepname == "buildtools":
@@ -208,18 +208,22 @@ def runcmd(cmd, *args, **kwargs):
 bh_path, remoterepo, remotebranch, baseremotebranch = utils.getbuildhistoryconfig(ourconfig, args.builddir, args.target, args.reponame, args.branchname, 1)
 if bh_path:
     if jcfg:
-        jsonconfig.append({"name" : "buildhistory-init", "description" : "Initialize buildhistory", "phase" : "init"})
+        jsonconfig.append({"name" : "buildhistory-init", "bbname" : "Initialize buildhistory", "phase" : "init"})
 if args.phase == "init" and args.stepname == "buildhistory-init":
     if bh_path:
         runcmd([os.path.join(scriptsdir, "buildhistory-init"), bh_path, remoterepo, remotebranch, baseremotebranch])
     sys.exit(0)
 
 def handle_stepnum(stepnum):
+    desc = utils.getconfigvar("description", ourconfig, args.target, stepnum)
+    if desc:
+        desc = desc + ": "
+
     # Add any layers specified
     layers = utils.getconfiglist("ADDLAYER", ourconfig, args.target, stepnum)
     if jcfg:
         if layers:
-            jsonconfig.append({"name" : "add-layers", "description" : "Adding layers %s" % str(layers), "phase" : str(stepnum)})
+            jsonconfig.append({"name" : "add-layers", "bbname" : "%sAdding layers %s" % (desc, str(layers)), "phase" : str(stepnum)})
     elif args.stepname == "add-layers":
         for layer in layers:
             bitbakecmd(args.builddir, "bitbake-layers add-layer %s" % layer, report, stepnum, args.stepname)
@@ -229,7 +233,7 @@ def handle_stepnum(stepnum):
     # Generate the configuration files needed for this step
     if utils.getconfigvar("WRITECONFIG", ourconfig, args.target, stepnum):
         if jcfg:
-            jsonconfig.append({"name" : "write-config", "description" : "Writing configuration files", "phase" : str(stepnum)})
+            jsonconfig.append({"name" : "write-config", "bbname" : "%sWriting configuration files" % desc, "phase" : str(stepnum)})
         elif args.stepname == "write-config":
             runcmd([scriptsdir + "/setup-config", args.target, str(stepnum - 1), args.builddir, args.branchname, args.reponame, "-s", args.sstateprefix, "-b", args.buildappsrcrev])
 
@@ -237,7 +241,7 @@ def handle_stepnum(stepnum):
     targets = utils.getconfigvar("BBTARGETS", ourconfig, args.target, stepnum)
     if targets:
         if jcfg:
-            jsonconfig.append({"name" : "build-targets", "description" : "Building targets %s" % str(targets), "phase" : str(stepnum)})
+            jsonconfig.append({"name" : "build-targets", "bbname" : "%sBuilding targets %s" % (desc, str(targets)), "phase" : str(stepnum)})
         elif args.stepname == "build-targets":
             hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, targets))
             bitbakecmd(args.builddir, "bitbake %s -k" % targets, report, stepnum, args.stepname)
@@ -246,7 +250,7 @@ def handle_stepnum(stepnum):
     sanitytargets = utils.getconfigvar("SANITYTARGETS", ourconfig, args.target, stepnum)
     if sanitytargets:
         if jcfg:
-            jsonconfig.append({"name" : "test-targets", "description" : "Running OEQA test targets %s" % str(sanitytargets), "phase" : str(stepnum)})
+            jsonconfig.append({"name" : "test-targets", "bbname" : "%sRunning OEQA test targets %s" % (desc, str(sanitytargets)), "phase" : str(stepnum)})
         elif args.stepname == "test-targets":
             hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, sanitytargets))
             bitbakecmd(args.builddir, "%s/checkvnc; DISPLAY=:1 bitbake %s -k" % (scriptsdir, sanitytargets), report, stepnum, args.stepname)
@@ -255,7 +259,7 @@ def handle_stepnum(stepnum):
     cmds = utils.getconfiglist("EXTRACMDS", ourconfig, args.target, stepnum)
     if jcfg:
         if cmds:
-            jsonconfig.append({"name" : "cmds", "description" : "Running bitbake environment commands %s" % str(cmds), "phase" : str(stepnum)})
+            jsonconfig.append({"name" : "cmds", "bbname" : "%sRunning bitbake environment commands %s" % (desc, str(cmds)), "phase" : str(stepnum)})
     elif args.stepname == "cmds":
         for cmd in cmds:
             hp.printheader("Step %s/%s: Running command %s" % (stepnum, maxsteps, cmd))
@@ -264,7 +268,7 @@ def handle_stepnum(stepnum):
     cmds = utils.getconfiglist("EXTRAPLAINCMDS", ourconfig, args.target, stepnum)
     if jcfg:
         if cmds:
-            jsonconfig.append({"name" : "plain-cmds", "description" : "Running commands %s" % str(cmds), "phase" : str(stepnum)})
+            jsonconfig.append({"name" : "plain-cmds", "bbname" : "%sRunning commands %s" % (desc, str(cmds)), "phase" : str(stepnum)})
     elif args.stepname == "plain-cmds":
         for cmd in cmds:
             hp.printheader("Step %s/%s: Running 'plain' command %s" % (stepnum, maxsteps, cmd))
@@ -272,7 +276,7 @@ def handle_stepnum(stepnum):
 
     if jcfg:
         if layers:
-            jsonconfig.append({"name" : "remove-layers", "description" : "Removing layers %s" % str(layers), "phase" : str(stepnum)})
+            jsonconfig.append({"name" : "remove-layers", "bbname" : "%sRemoving layers %s" % (desc, str(layers)), "phase" : str(stepnum)})
     elif args.stepname == "remove-layers":
         # Remove any layers we added in a reverse order
         for layer in reversed(layers):
@@ -295,7 +299,7 @@ else:
 
 
 if jcfg:
-    jsonconfig.append({"name" : "publish", "description" : "Publishing artefacts", "phase" : "finish"})
+    jsonconfig.append({"name" : "publish", "bbname" : "Publishing artefacts", "phase" : "finish"})
 elif args.phase == "finish" and args.stepname == "publish":
     if args.publish_dir:
         hp.printheader("Running publish artefacts")
@@ -303,7 +307,7 @@ elif args.phase == "finish" and args.stepname == "publish":
     sys.exit(0)
 
 if jcfg:
-    jsonconfig.append({"name" : "collect-results", "description" : "Collecting result files", "phase" : "finish"})
+    jsonconfig.append({"name" : "collect-results", "bbname" : "Collecting result files", "phase" : "finish"})
 elif args.phase == "finish" and args.stepname == "collect-results":
     if args.results_dir:
         hp.printheader("Running results collection")
@@ -311,7 +315,7 @@ elif args.phase == "finish" and args.stepname == "collect-results":
     sys.exit(0)
 
 if jcfg:
-    jsonconfig.append({"name" : "send-errors", "description" : "Sending error reports", "phase" : "finish"})
+    jsonconfig.append({"name" : "send-errors", "bbname" : "Sending error reports", "phase" : "finish"})
 elif args.phase == "finish" and args.stepname == "send-errors":
     if args.build_url and utils.getconfigvar("SENDERRORS", ourconfig, args.target, stepnum):
         hp.printheader("Sending any error reports")
@@ -319,7 +323,7 @@ elif args.phase == "finish" and args.stepname == "send-errors":
     sys.exit(0)
 
 if jcfg:
-    jsonconfig.append({"name" : "builddir-cleanup", "description" : "Cleaning up build directory", "phase" : "finish"})
+    jsonconfig.append({"name" : "builddir-cleanup", "bbname" : "Cleaning up build directory", "phase" : "finish"})
 elif args.phase == "finish" and args.stepname == "builddir-cleanup":
     if args.builddir and os.path.exists(args.builddir):
         runcmd(["mv", args.builddir, args.builddir + "-renamed"])
-- 
2.25.1

[yocto-autobuilder-helper][dunfell 15/23] scripts/run-config: Remove redundant boilerplate json

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2bb48042438f3154bbfa6fbc7f2c7556bfa7c762)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 scripts/run-config | 31 +++++++++++++++++--------------
 1 file changed, 17 insertions(+), 14 deletions(-)

diff --git a/scripts/run-config b/scripts/run-config
index 89506f5..58ce364 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -112,10 +112,13 @@ if args.json_outputfile:
     jsonconfig = []
     jcfg = True
 
+def addentry(name, description, phase):
+    jsonconfig.append({"name" : name, "bbname" : description[:45], "phase" : phase, "description" : description})
+
 if jcfg:
     buildtools = utils.setup_buildtools_tarball(ourconfig, args.workername, None, checkonly=True)
     if buildtools:
-        jsonconfig.append({"name" : "buildtools", "bbname" : "Extract and setup buildtools tarball", "phase" : "init"})
+        addentry("buildtools", "Extract and setup buildtools tarball", "init")
 else:
     utils.setup_buildtools_tarball(ourconfig, args.workername, args.builddir + "/../buildtools")
     if args.phase == "init" and args.stepname == "buildtools":
@@ -208,14 +211,14 @@ def runcmd(cmd, *args, **kwargs):
 bh_path, remoterepo, remotebranch, baseremotebranch = utils.getbuildhistoryconfig(ourconfig, args.builddir, args.target, args.reponame, args.branchname, 1)
 if bh_path:
     if jcfg:
-        jsonconfig.append({"name" : "buildhistory-init", "bbname" : "Initialize buildhistory", "phase" : "init"})
+        addentry("buildhistory-init", "Initialize buildhistory", "init")
 if args.phase == "init" and args.stepname == "buildhistory-init":
     if bh_path:
         runcmd([os.path.join(scriptsdir, "buildhistory-init"), bh_path, remoterepo, remotebranch, baseremotebranch])
     sys.exit(0)
 
 def handle_stepnum(stepnum):
-    desc = utils.getconfigvar("description", ourconfig, args.target, stepnum)
+    desc = utils.getconfigvar("description", ourconfig, args.target, stepnum) or ""
     if desc:
         desc = desc + ": "
 
@@ -223,7 +226,7 @@ def handle_stepnum(stepnum):
     layers = utils.getconfiglist("ADDLAYER", ourconfig, args.target, stepnum)
     if jcfg:
         if layers:
-            jsonconfig.append({"name" : "add-layers", "bbname" : "%sAdding layers %s" % (desc, str(layers)), "phase" : str(stepnum)})
+            addentry("add-layers", "%sAdding layers %s" % (desc, str(layers)), str(stepnum))
     elif args.stepname == "add-layers":
         for layer in layers:
             bitbakecmd(args.builddir, "bitbake-layers add-layer %s" % layer, report, stepnum, args.stepname)
@@ -233,7 +236,7 @@ def handle_stepnum(stepnum):
     # Generate the configuration files needed for this step
     if utils.getconfigvar("WRITECONFIG", ourconfig, args.target, stepnum):
         if jcfg:
-            jsonconfig.append({"name" : "write-config", "bbname" : "%sWriting configuration files" % desc, "phase" : str(stepnum)})
+            addentry("write-config", "%sWriting configuration files" % desc, str(stepnum))
         elif args.stepname == "write-config":
             runcmd([scriptsdir + "/setup-config", args.target, str(stepnum - 1), args.builddir, args.branchname, args.reponame, "-s", args.sstateprefix, "-b", args.buildappsrcrev])
 
@@ -241,7 +244,7 @@ def handle_stepnum(stepnum):
     targets = utils.getconfigvar("BBTARGETS", ourconfig, args.target, stepnum)
     if targets:
         if jcfg:
-            jsonconfig.append({"name" : "build-targets", "bbname" : "%sBuilding targets %s" % (desc, str(targets)), "phase" : str(stepnum)})
+            addentry("build-targets", "%sBuilding targets %s" % (desc, str(targets)), str(stepnum))
         elif args.stepname == "build-targets":
             hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, targets))
             bitbakecmd(args.builddir, "bitbake %s -k" % targets, report, stepnum, args.stepname)
@@ -250,7 +253,7 @@ def handle_stepnum(stepnum):
     sanitytargets = utils.getconfigvar("SANITYTARGETS", ourconfig, args.target, stepnum)
     if sanitytargets:
         if jcfg:
-            jsonconfig.append({"name" : "test-targets", "bbname" : "%sRunning OEQA test targets %s" % (desc, str(sanitytargets)), "phase" : str(stepnum)})
+            addentry("test-targets", "%sRunning OEQA test targets %s" % (desc, str(sanitytargets)), str(stepnum))
         elif args.stepname == "test-targets":
             hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, sanitytargets))
             bitbakecmd(args.builddir, "%s/checkvnc; DISPLAY=:1 bitbake %s -k" % (scriptsdir, sanitytargets), report, stepnum, args.stepname)
@@ -259,7 +262,7 @@ def handle_stepnum(stepnum):
     cmds = utils.getconfiglist("EXTRACMDS", ourconfig, args.target, stepnum)
     if jcfg:
         if cmds:
-            jsonconfig.append({"name" : "cmds", "bbname" : "%sRunning bitbake environment commands %s" % (desc, str(cmds)), "phase" : str(stepnum)})
+            addentry("cmds", "%sRunning bitbake environment commands %s" % (desc, str(cmds)), str(stepnum))
     elif args.stepname == "cmds":
         for cmd in cmds:
             hp.printheader("Step %s/%s: Running command %s" % (stepnum, maxsteps, cmd))
@@ -268,7 +271,7 @@ def handle_stepnum(stepnum):
     cmds = utils.getconfiglist("EXTRAPLAINCMDS", ourconfig, args.target, stepnum)
     if jcfg:
         if cmds:
-            jsonconfig.append({"name" : "plain-cmds", "bbname" : "%sRunning commands %s" % (desc, str(cmds)), "phase" : str(stepnum)})
+            addentry("plain-cmds", "%sRunning commands %s" % (desc, str(cmds)), str(stepnum))
     elif args.stepname == "plain-cmds":
         for cmd in cmds:
             hp.printheader("Step %s/%s: Running 'plain' command %s" % (stepnum, maxsteps, cmd))
@@ -276,7 +279,7 @@ def handle_stepnum(stepnum):
 
     if jcfg:
         if layers:
-            jsonconfig.append({"name" : "remove-layers", "bbname" : "%sRemoving layers %s" % (desc, str(layers)), "phase" : str(stepnum)})
+            addentry("remove-layers", "%sRemoving layers %s" % (desc, str(layers)), str(stepnum))
     elif args.stepname == "remove-layers":
         # Remove any layers we added in a reverse order
         for layer in reversed(layers):
@@ -299,7 +302,7 @@ else:
 
 
 if jcfg:
-    jsonconfig.append({"name" : "publish", "bbname" : "Publishing artefacts", "phase" : "finish"})
+    addentry("publish", "Publishing artefacts", "finish")
 elif args.phase == "finish" and args.stepname == "publish":
     if args.publish_dir:
         hp.printheader("Running publish artefacts")
@@ -307,7 +310,7 @@ elif args.phase == "finish" and args.stepname == "publish":
     sys.exit(0)
 
 if jcfg:
-    jsonconfig.append({"name" : "collect-results", "bbname" : "Collecting result files", "phase" : "finish"})
+    addentry("collect-results", "Collecting result files", "finish")
 elif args.phase == "finish" and args.stepname == "collect-results":
     if args.results_dir:
         hp.printheader("Running results collection")
@@ -315,7 +318,7 @@ elif args.phase == "finish" and args.stepname == "collect-results":
     sys.exit(0)
 
 if jcfg:
-    jsonconfig.append({"name" : "send-errors", "bbname" : "Sending error reports", "phase" : "finish"})
+    addentry("send-errors", "Sending error reports", "finish")
 elif args.phase == "finish" and args.stepname == "send-errors":
     if args.build_url and utils.getconfigvar("SENDERRORS", ourconfig, args.target, stepnum):
         hp.printheader("Sending any error reports")
@@ -323,7 +326,7 @@ elif args.phase == "finish" and args.stepname == "send-errors":
     sys.exit(0)
 
 if jcfg:
-    jsonconfig.append({"name" : "builddir-cleanup", "bbname" : "Cleaning up build directory", "phase" : "finish"})
+    addentry("builddir-cleanup", "Cleaning up build directory", "finish")
 elif args.phase == "finish" and args.stepname == "builddir-cleanup":
     if args.builddir and os.path.exists(args.builddir):
         runcmd(["mv", args.builddir, args.builddir + "-renamed"])
-- 
2.25.1

[yocto-autobuilder-helper][dunfell 16/23] scripts/shared-repo-unpack: Add flush call to update the output more regularly before buildtools

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6e2825564c0b7b69f56e6e589ec15a1cebdb26d1)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 scripts/shared-repo-unpack | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/shared-repo-unpack b/scripts/shared-repo-unpack
index 7dc250c..f08efa8 100755
--- a/scripts/shared-repo-unpack
+++ b/scripts/shared-repo-unpack
@@ -60,6 +60,7 @@ for repo in sorted(repos.keys()):
         utils.fetchgitrepo(targetsubdir, repo, repos[repo], stashdir)
         if args.publish_dir:
             utils.publishrepo(targetsubdir, repo, args.publish_dir)
+    utils.flush()
 
 utils.setup_buildtools_tarball(ourconfig, args.workername, args.abworkdir + "/buildtools")
 
-- 
2.25.1

[yocto-autobuilder-helper][dunfell 17/23] config.json/run-config: Add support for shortnames and descriptions

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Learn from the previous experiments and add meaninful shortnames and
descriptions to work around the 50 char name limit.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit bceb63fb7952c6ed289733471a0177cfbc365a1e)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 config.json        | 16 +++++++++++-----
 scripts/run-config | 34 +++++++++++++++++++++++-----------
 2 files changed, 34 insertions(+), 16 deletions(-)

diff --git a/config.json b/config.json
index 1fcc45d..4856507 100644
--- a/config.json
+++ b/config.json
@@ -453,7 +453,8 @@
             "MACHINE" : "qemux86-64",
             "SDKMACHINE" : "x86_64",
             "step1" : {
-                "description" : "x86_64 32bit multilib image with rpm",
+                "shortname" : "x86-64 lib32-img rpm",
+                "description" : "qemux86-64 32bit multilib image with rpm",
                 "BBTARGETS" : "lib32-core-image-minimal",
                 "SANITYTARGETS" : "lib32-core-image-minimal:do_testimage",
                 "extravars" : [
@@ -464,7 +465,8 @@
                 ]
             },
             "step2" : {
-                "description" : "x86_64 32bit multilib image with ipk",
+                "shortname" : "x86-64 lib32-img ipk",
+                "description" : "qemux86-64 32bit multilib image with ipk",
                 "PACKAGE_CLASSES" : "package_ipk",
                 "BBTARGETS" : "lib32-core-image-minimal",
                 "SANITYTARGETS" : "lib32-core-image-minimal:do_testimage",
@@ -476,7 +478,8 @@
                 ]
             },
             "step3" : {
-                "description" : "x86_64 64bit image and 32 bit multilibs with rpm",
+                "shortname" : "x86-64 lib32 rpm",
+                "description" : "qemux86-64 64bit image and 32 bit multilibs with rpm",
                 "BBTARGETS" : "core-image-sato",
                 "SANITYTARGETS" : "core-image-sato:do_testimage",
                 "extravars" : [
@@ -489,7 +492,8 @@
                 ]
             },
             "step4" : {
-                "description" : "x86_64 64bit image and 32 bit multilibs with ipk",
+                "shortname" : "x86-64 lib32 ipk",
+                "description" : "qemux86-64 64bit image and 32 bit multilibs with ipk",
                 "PACKAGE_CLASSES" : "package_ipk",
                 "BBTARGETS" : "core-image-sato",
                 "SANITYTARGETS" : "core-image-sato:do_testimage",
@@ -503,6 +507,7 @@
                 ]
             },
             "step5" : {
+                "shortname" : "x86-64 lib64-img",
                 "description" : "x86 building 64bit multilib image",
                 "MACHINE" : "qemux86",
                 "SDKMACHINE" : "i686",
@@ -514,7 +519,8 @@
                 ]
             },
             "step6" : {
-                "description" : "mips64 image using n32 as default",
+                "shortname" : "mip64 n32",
+                "description" : "qemumips64 image using n32 as default",
                 "MACHINE" : "qemumips64",
                 "BBTARGETS" : "core-image-minimal core-image-minimal:do_populate_sdk",
                 "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-minimal:do_testsdk",
diff --git a/scripts/run-config b/scripts/run-config
index 58ce364..aab52c1 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -112,13 +112,26 @@ if args.json_outputfile:
     jsonconfig = []
     jcfg = True
 
+# There is a 50 char limit on "bbname" but buildbot may append "_1", "_2" if multiple steps
+# with the same name exist in a build
 def addentry(name, description, phase):
-    jsonconfig.append({"name" : name, "bbname" : description[:45], "phase" : phase, "description" : description})
+    jsonconfig.append({"name" : name, "bbname" : description[:46], "phase" : phase, "description" : description})
+
+def addstepentry(name, taskdesc, shortname, description, detail, phase):
+    bbname = taskdesc
+    if shortname:
+        bbname = shortname + ": " + taskdesc
+    bbdesc = taskdesc
+    if description:
+        bbdesc = description
+    if detail:
+        bbdesc = bbdesc + ": " + detail
+    jsonconfig.append({"name" : name, "bbname" : bbname[:46], "phase" : phase, "description" : bbdesc})
 
 if jcfg:
     buildtools = utils.setup_buildtools_tarball(ourconfig, args.workername, None, checkonly=True)
     if buildtools:
-        addentry("buildtools", "Extract and setup buildtools tarball", "init")
+        addentry("buildtools", "Setup buildtools tarball", "init")
 else:
     utils.setup_buildtools_tarball(ourconfig, args.workername, args.builddir + "/../buildtools")
     if args.phase == "init" and args.stepname == "buildtools":
@@ -218,15 +231,14 @@ if args.phase == "init" and args.stepname == "buildhistory-init":
     sys.exit(0)
 
 def handle_stepnum(stepnum):
+    shortdesc = utils.getconfigvar("shortname", ourconfig, args.target, stepnum) or ""
     desc = utils.getconfigvar("description", ourconfig, args.target, stepnum) or ""
-    if desc:
-        desc = desc + ": "
 
     # Add any layers specified
     layers = utils.getconfiglist("ADDLAYER", ourconfig, args.target, stepnum)
     if jcfg:
         if layers:
-            addentry("add-layers", "%sAdding layers %s" % (desc, str(layers)), str(stepnum))
+            addstepentry("add-layers", "Add layers", shortdesc, desc, str(layers), str(stepnum))
     elif args.stepname == "add-layers":
         for layer in layers:
             bitbakecmd(args.builddir, "bitbake-layers add-layer %s" % layer, report, stepnum, args.stepname)
@@ -236,7 +248,7 @@ def handle_stepnum(stepnum):
     # Generate the configuration files needed for this step
     if utils.getconfigvar("WRITECONFIG", ourconfig, args.target, stepnum):
         if jcfg:
-            addentry("write-config", "%sWriting configuration files" % desc, str(stepnum))
+            addstepentry("write-config", "Write config", shortdesc, desc, None, str(stepnum))
         elif args.stepname == "write-config":
             runcmd([scriptsdir + "/setup-config", args.target, str(stepnum - 1), args.builddir, args.branchname, args.reponame, "-s", args.sstateprefix, "-b", args.buildappsrcrev])
 
@@ -244,7 +256,7 @@ def handle_stepnum(stepnum):
     targets = utils.getconfigvar("BBTARGETS", ourconfig, args.target, stepnum)
     if targets:
         if jcfg:
-            addentry("build-targets", "%sBuilding targets %s" % (desc, str(targets)), str(stepnum))
+            addstepentry("build-targets", "Build targets", shortdesc, desc, str(targets), str(stepnum))
         elif args.stepname == "build-targets":
             hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, targets))
             bitbakecmd(args.builddir, "bitbake %s -k" % targets, report, stepnum, args.stepname)
@@ -253,7 +265,7 @@ def handle_stepnum(stepnum):
     sanitytargets = utils.getconfigvar("SANITYTARGETS", ourconfig, args.target, stepnum)
     if sanitytargets:
         if jcfg:
-            addentry("test-targets", "%sRunning OEQA test targets %s" % (desc, str(sanitytargets)), str(stepnum))
+            addstepentry("test-targets", "QA targets", shortdesc, desc, str(sanitytargets), str(stepnum))
         elif args.stepname == "test-targets":
             hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, sanitytargets))
             bitbakecmd(args.builddir, "%s/checkvnc; DISPLAY=:1 bitbake %s -k" % (scriptsdir, sanitytargets), report, stepnum, args.stepname)
@@ -262,7 +274,7 @@ def handle_stepnum(stepnum):
     cmds = utils.getconfiglist("EXTRACMDS", ourconfig, args.target, stepnum)
     if jcfg:
         if cmds:
-            addentry("cmds", "%sRunning bitbake environment commands %s" % (desc, str(cmds)), str(stepnum))
+            addstepentry("cmds", "Run cmds", shortdesc, desc, str(cmds), str(stepnum))
     elif args.stepname == "cmds":
         for cmd in cmds:
             hp.printheader("Step %s/%s: Running command %s" % (stepnum, maxsteps, cmd))
@@ -271,7 +283,7 @@ def handle_stepnum(stepnum):
     cmds = utils.getconfiglist("EXTRAPLAINCMDS", ourconfig, args.target, stepnum)
     if jcfg:
         if cmds:
-            addentry("plain-cmds", "%sRunning commands %s" % (desc, str(cmds)), str(stepnum))
+            addstepentry("plain-cmds", "Run cmds", shortdesc, desc, str(cmds), str(stepnum))
     elif args.stepname == "plain-cmds":
         for cmd in cmds:
             hp.printheader("Step %s/%s: Running 'plain' command %s" % (stepnum, maxsteps, cmd))
@@ -279,7 +291,7 @@ def handle_stepnum(stepnum):
 
     if jcfg:
         if layers:
-            addentry("remove-layers", "%sRemoving layers %s" % (desc, str(layers)), str(stepnum))
+            addstepentry("remove-layers", "Remove layers", shortdesc, desc, str(layers), str(stepnum))
     elif args.stepname == "remove-layers":
         # Remove any layers we added in a reverse order
         for layer in reversed(layers):
-- 
2.25.1

[yocto-autobuilder-helper][dunfell 18/23] config.json: Unbreak qa-extras locked sigs test

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

qa-extras and qa-extras2 were split incorrectly, fix this.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 80fe0ab06972c46c82cde29cbdfcdac6e87dde99)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 config.json | 26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/config.json b/config.json
index 4856507..ddf36ae 100644
--- a/config.json
+++ b/config.json
@@ -802,18 +802,18 @@
                     "TMPDIR = '${TOPDIR}/newtmp'",
                     "require ../locked-sigs.inc"
                 ]
-            }
-        },
-        "qa-extras2" : {
-            "MACHINE" : "qemux86-64",
-            "step1" : {
+            },
+            "step7" : {
                 "SDKMACHINE" : "x86_64",
                 "BBTARGETS" : "core-image-sato:do_populate_sdk_ext",
                 "extravars" : [
                     "TMPDIR = '${TOPDIR}/sdktmp'"
                 ]
-            },
-            "step2" : {
+            }
+        },
+        "qa-extras2" : {
+            "MACHINE" : "qemux86-64",
+            "step1" : {
                 "BBTARGETS" : "core-image-sato",
                 "SANITYTARGETS" : "core-image-sato:do_testimage",
                 "extravars" : [
@@ -821,7 +821,7 @@
                     "TEST_SUITES_append = ' logrotate'"
                 ]
             },
-            "step3" : {
+            "step2" : {
                 "BBTARGETS" : "core-image-sato",
                 "SANITYTARGETS" : "core-image-sato:do_testimage",
                 "extravars" : [
@@ -829,7 +829,7 @@
                     "TEST_SUITES_append = ' pam'"
                 ]
             },
-            "step4" : {
+            "step3" : {
                 "BBTARGETS" : "core-image-sato",
                 "SANITYTARGETS" : "core-image-sato:do_testimage",
                 "extravars" : [
@@ -838,7 +838,7 @@
                 ],
                 "ADDLAYER" : ["${BUILDDIR}/../meta-skeleton"]
             },
-            "step5" : {
+            "step4" : {
                 "BBTARGETS" : "core-image-sato",
                 "SANITYTARGETS" : "core-image-sato:do_testimage",
                 "extravars" : [
@@ -847,7 +847,7 @@
                     "TEST_SUITES_append = ' systemd'"
                 ]
             },
-            "step6" : {
+            "step5" : {
                 "BBTARGETS" : "core-image-sato",
                 "SANITYTARGETS" : "core-image-sato:do_testimage",
                 "extravars" : [
@@ -855,7 +855,7 @@
                     "VIRTUAL-RUNTIME_init_manager = 'sysvinit'"
                 ]
             },
-            "step7" : {
+            "step6" : {
                 "BBTARGETS" : "core-image-sato",
                 "SANITYTARGETS" : "core-image-sato:do_testimage",
                 "extravars" : [
@@ -865,7 +865,7 @@
                     "DISTRO_FEATURES_BACKFILL_CONSIDERED = 'sysvinit'"
                 ]
             },
-            "step8" : {
+            "step7" : {
                 "BBTARGETS" : "core-image-sato",
                 "SANITYTARGETS" : "core-image-sato:do_testimage",
                 "extravars" : [
-- 
2.25.1

[yocto-autobuilder-helper][dunfell 19/23] config.json: Add further descriptions

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 19b7456b92c2eb7b2b27f1e378dbc793d068ee3c)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 config.json | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/config.json b/config.json
index ddf36ae..cedcef7 100644
--- a/config.json
+++ b/config.json
@@ -71,6 +71,7 @@
                 "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext"
             },
             "step3" : {
+                "shortname" : "Machine oe-selftest",
                 "BUILDHISTORY" : false,
                 "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"],
                 "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
@@ -128,6 +129,7 @@
                 "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk core-image-sato:do_testsdkext"
             },
             "step2" : {
+                "shortname" : "Machine oe-selftest",
                 "BUILDHISTORY" : false,
                 "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest -a -t machine -j 15"]
             }
@@ -164,13 +166,16 @@
                 "RPM_GPG_SIGN_CHUNK = '1'"
             ],
             "step1" : {
+                "shortname" : "Bitbake Selftest",
                 "EXTRACMDS" : ["bitbake-selftest"]
             },
             "step2" : {
+                "shortname" : "OE Selftest",
                 "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/ DISPLAY=:1 oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror devtool.DevtoolAddTests.test_devtool_add_npm recipetool.RecipetoolTests.test_recipetool_create_npm -T machine -T toolchain-user -T toolchain-system -j 15"],
                 "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
             },
             "step3" : {
+                "shortname" : "Python Linter Report",
                 "EXTRACMDS" : ["if [ `which oe-pylint` ]; then mkdir -p ${HELPERRESULTSDIR}/${HELPERTARGET}; oe-pylint > ${HELPERRESULTSDIR}/${HELPERTARGET}/pylint.log || true; fi"]
             }
         },
@@ -178,6 +183,7 @@
             "SDKMACHINE" : "x86_64",
             "MACHINE" : "qemux86",
             "step1" : {
+                "shortname" : "Sources pre-fetching",
                 "BBTARGETS" : "universe -c fetch -k",
                 "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"],
                 "extravars" : [
@@ -189,6 +195,7 @@
             "SDKMACHINE" : "x86_64",
             "MACHINE" : "qemux86",
             "step1" : {
+                "shortname" : "Source Mirror Selftest",
                 "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest -r buildoptions.SourceMirroring.test_yocto_source_mirror"],
                 "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
             }
@@ -571,6 +578,7 @@
             ],
             "step1" : {
                 "MACHINE" : "qemux86",
+                "shortname" : "qemux86 wic",
                 "BBTARGETS" : "wic-tools core-image-sato",
                 "EXTRACMDS" : [
                     "wic create directdisk -e core-image-sato -o ${BUILDDIR}/tmp/deploy/wic_images/qemux86/directdisk/core-image-sato/",
@@ -580,6 +588,7 @@
             },
             "step2" : {
                 "MACHINE" : "genericx86",
+                "shortname" : "genericx86 wic",
                 "BBTARGETS" : "wic-tools core-image-sato",
                 "EXTRACMDS" : [
                     "wic create directdisk -e core-image-sato -o ${BUILDDIR}/tmp/deploy/wic_images/genericx86/directdisk/core-image-sato/",
@@ -589,6 +598,7 @@
             },
             "step3" : {
                 "MACHINE" : "qemux86-64",
+                "shortname" : "qemux86-64 wic",
                 "BBTARGETS" : "wic-tools core-image-sato",
                 "EXTRACMDS" : [
                     "wic create directdisk -e core-image-sato -o ${BUILDDIR}/tmp/deploy/wic_images/qemux86-64/directdisk/core-image-sato/",
@@ -598,6 +608,7 @@
             },
             "step4" : {
                 "MACHINE" : "genericx86-64",
+                "shortname" : "genericx86-64 wic",
                 "BBTARGETS" : "wic-tools core-image-sato",
                 "EXTRACMDS" : [
                     "wic create directdisk -e core-image-sato -o ${BUILDDIR}/tmp/deploy/wic_images/genericx86-64/directdisk/core-image-sato/",
@@ -613,14 +624,17 @@
             ],
             "step1" : {
                 "SDKMACHINE" : "x86_64",
+                "shortname" : "x86_64 tools",
                 "BBTARGETS" : "buildtools-tarball buildtools-extended-tarball uninative-tarball"
             },
             "step2" : {
                 "SDKMACHINE" : "i686",
+                "shortname" : "i686 tools",
                 "BBTARGETS" : "uninative-tarball"
             },
             "step3" : {
                 "SDKMACHINE" : "aarch64",
+                "shortname" : "aarch64 tools",
                 "BBTARGETS" : "buildtools-tarball buildtools-extended-tarball uninative-tarball"
             }
         },
@@ -635,9 +649,11 @@
                 "SOURCE_MIRROR_URL = 'file://${BASE_SHAREDDIR}/current_sources'"
             ],
             "step1" : {
+                "shortname" : "Universe fetch",
                 "BBTARGETS" : "universe -k -c fetch"
             },
             "step2" : {
+                "shortname" : "BA image build",
                 "BBTARGETS" : "build-appliance-image"
             }
         },
@@ -766,6 +782,7 @@
         "qa-extras" : {
             "MACHINE" : "qemux86-64",
             "step1" : {
+                "shortname" : "Readonly rootfs",
                 "BBTARGETS" : "core-image-minimal",
                 "SANITYTARGETS" : "core-image-minimal:do_testimage",
                 "extravars" : [
@@ -773,6 +790,7 @@
                 ]
             },
             "step2" : {
+                "shortname" : "ROOT_HOME testing",
                 "BBTARGETS" : "core-image-minimal",
                 "SANITYTARGETS" : "core-image-minimal:do_testimage",
                 "extravars" : [
@@ -780,6 +798,7 @@
                 ]
             },
             "step3" : {
+                "shortname" : "Full eSDK type",
                 "SDKMACHINE" : "x86_64",
                 "BBTARGETS" : "core-image-minimal:do_populate_sdk_ext",
                 "extravars" : [
@@ -787,15 +806,18 @@
                 ]
             },
             "step4" : {
+                "shortname" : "Prep locked-sigs test",
                 "SDKMACHINE" : "x86_64",
                 "BBTARGETS" : "core-image-sato core-image-sato:do_populate_sdk_ext"
             },
             "step5" : {
+                "shortname" : "Prep #2 locked-sigs test",
                 "SDKMACHINE" : "x86_64",
                 "BBTARGETS" : "core-image-sato -S none",
                 "EXTRACMDS" : ["${SCRIPTSDIR}/../janitor/clobberdir ${BUILDDIR}/../build/tmp"]
             },
             "step6" : {
+                "shortname" : "Test locked-sigs image",
                 "SDKMACHINE" : "x86_64",
                 "BBTARGETS" : "core-image-sato",
                 "extravars" : [
@@ -804,6 +826,7 @@
                 ]
             },
             "step7" : {
+                "shortname" : "Test locked-sigs eSDK",
                 "SDKMACHINE" : "x86_64",
                 "BBTARGETS" : "core-image-sato:do_populate_sdk_ext",
                 "extravars" : [
@@ -814,6 +837,7 @@
         "qa-extras2" : {
             "MACHINE" : "qemux86-64",
             "step1" : {
+                "shortname" : "Test logrotate",
                 "BBTARGETS" : "core-image-sato",
                 "SANITYTARGETS" : "core-image-sato:do_testimage",
                 "extravars" : [
@@ -830,6 +854,7 @@
                 ]
             },
             "step3" : {
+                "shortname" : "Test skeletoninit",
                 "BBTARGETS" : "core-image-sato",
                 "SANITYTARGETS" : "core-image-sato:do_testimage",
                 "extravars" : [
@@ -839,6 +864,7 @@
                 "ADDLAYER" : ["${BUILDDIR}/../meta-skeleton"]
             },
             "step4" : {
+                "shortname" : "Systemd with sysvinit compat",
                 "BBTARGETS" : "core-image-sato",
                 "SANITYTARGETS" : "core-image-sato:do_testimage",
                 "extravars" : [
@@ -848,6 +874,7 @@
                 ]
             },
             "step5" : {
+                "shortname" : "Sysvinit with systemd",
                 "BBTARGETS" : "core-image-sato",
                 "SANITYTARGETS" : "core-image-sato:do_testimage",
                 "extravars" : [
@@ -856,6 +883,7 @@
                 ]
             },
             "step6" : {
+                "shortname" : "Systemd",
                 "BBTARGETS" : "core-image-sato",
                 "SANITYTARGETS" : "core-image-sato:do_testimage",
                 "extravars" : [
@@ -866,6 +894,7 @@
                 ]
             },
             "step7" : {
+                "shortname" : "Mesa gallium-llvm",
                 "BBTARGETS" : "core-image-sato",
                 "SANITYTARGETS" : "core-image-sato:do_testimage",
                 "extravars" : [
-- 
2.25.1

[yocto-autobuilder-helper][dunfell 20/23] config.json: Use buildtools tarball on debian9

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

debian9 has python 3.5 which doesn't work with new qemu versions. Its
the only remaining 3.5 distro so lets move to a minimum of 3.6.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 62b3f5553e9c215cf94798ae4694ccb97f6e85dd)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 config.json | 1 +
 1 file changed, 1 insertion(+)

diff --git a/config.json b/config.json
index cedcef7..8eb05e6 100644
--- a/config.json
+++ b/config.json
@@ -1006,6 +1006,7 @@
     },
     "buildtools" : {
         "debian8*" : "${BUILDTOOLS_URL}",
+        "debian9*" : "${BUILDTOOLS_URL}",
         "centos7*" : "${BUILDTOOLS_URL}",
         "perf-centos7*" : "${BUILDTOOLS_URL}"
     }
-- 
2.25.1

Re: [yocto] [yocto-autobuilder-helper][dunfell 20/23] config.json: Use buildtools tarball on debian9

[Richard Purdie]

On Wed, 2021-03-24 at 14:39 -1000, Steve Sakoman wrote:
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> 
> debian9 has python 3.5 which doesn't work with new qemu versions. Its
> the only remaining 3.5 distro so lets move to a minimum of 3.6.
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> (cherry picked from commit 62b3f5553e9c215cf94798ae4694ccb97f6e85dd)
> Signed-off-by: Steve Sakoman <steve@sakoman.com>
> ---
>  config.json | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/config.json b/config.json
> index cedcef7..8eb05e6 100644
> --- a/config.json
> +++ b/config.json
> @@ -1006,6 +1006,7 @@
>      },
>      "buildtools" : {
>          "debian8*" : "${BUILDTOOLS_URL}",
> +        "debian9*" : "${BUILDTOOLS_URL}",
>          "centos7*" : "${BUILDTOOLS_URL}",
>          "perf-centos7*" : "${BUILDTOOLS_URL}"
>      }

I suspect you don't need that with dunfell since you're not using such a 
new qemu?

Cheers,

Richard

Re: [yocto] [yocto-autobuilder-helper][dunfell 20/23] config.json: Use buildtools tarball on debian9

[Steve Sakoman]

On Thu, Mar 25, 2021 at 4:02 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Wed, 2021-03-24 at 14:39 -1000, Steve Sakoman wrote:
> > From: Richard Purdie <richard.purdie@linuxfoundation.org>
> >
> > debian9 has python 3.5 which doesn't work with new qemu versions. Its
> > the only remaining 3.5 distro so lets move to a minimum of 3.6.
> >
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > (cherry picked from commit 62b3f5553e9c215cf94798ae4694ccb97f6e85dd)
> > Signed-off-by: Steve Sakoman <steve@sakoman.com>
> > ---
> >  config.json | 1 +
> >  1 file changed, 1 insertion(+)
> >
> > diff --git a/config.json b/config.json
> > index cedcef7..8eb05e6 100644
> > --- a/config.json
> > +++ b/config.json
> > @@ -1006,6 +1006,7 @@
> >      },
> >      "buildtools" : {
> >          "debian8*" : "${BUILDTOOLS_URL}",
> > +        "debian9*" : "${BUILDTOOLS_URL}",
> >          "centos7*" : "${BUILDTOOLS_URL}",
> >          "perf-centos7*" : "${BUILDTOOLS_URL}"
> >      }
>
> I suspect you don't need that with dunfell since you're not using such a
> new qemu?

Yes, indeed.  I'll drop from the pull request.

Steve

[yocto-autobuilder-helper][dunfell 21/23] scripts/run-config: Disable output buffering

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Line buffering (bufsize=1) is unavailable with binary mode so use unbuffered
mode instead. This fixes python runtime warnings.

[YOCTO #14093]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit c21732937c89f7b13a4f8a9a02d7fcb15a4bad2d)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 scripts/run-config | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/run-config b/scripts/run-config
index aab52c1..8ed88cf 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -193,7 +193,7 @@ def bitbakecmd(builddir, cmd, report, stepnum, stepname, oeenv=True):
 
     flush()
 
-    with subprocess.Popen(cmd, shell=True, cwd=builddir + "/..", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1) as p, open(log, 'ab') as f:
+    with subprocess.Popen(cmd, shell=True, cwd=builddir + "/..", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=0) as p, open(log, 'ab') as f:
         for line in p.stdout:
             writelog(line, f, sys.stdout.buffer)
             sys.stdout.flush()
-- 
2.25.1

[yocto-autobuilder-helper][dunfell 22/23] config.json: Split reproduciblity tests into their own target

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 97e0979d6eb0300951445bb2cc5eda315681302e)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 config.json | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)

diff --git a/config.json b/config.json
index 8eb05e6..3ce10c9 100644
--- a/config.json
+++ b/config.json
@@ -171,7 +171,7 @@
             },
             "step2" : {
                 "shortname" : "OE Selftest",
-                "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/ DISPLAY=:1 oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror devtool.DevtoolAddTests.test_devtool_add_npm recipetool.RecipetoolTests.test_recipetool_create_npm -T machine -T toolchain-user -T toolchain-system -j 15"],
+                "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/ DISPLAY=:1 oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror devtool.DevtoolAddTests.test_devtool_add_npm recipetool.RecipetoolTests.test_recipetool_create_npm reproducible -T machine -T toolchain-user -T toolchain-system -j 15"],
                 "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
             },
             "step3" : {
@@ -179,6 +179,16 @@
                 "EXTRACMDS" : ["if [ `which oe-pylint` ]; then mkdir -p ${HELPERRESULTSDIR}/${HELPERTARGET}; oe-pylint > ${HELPERRESULTSDIR}/${HELPERTARGET}/pylint.log || true; fi"]
             }
         },
+        "reproducible" : {
+            "MACHINE" : "qemux86-64",
+            "SDKMACHINE" : "x86_64",
+            "step1" : {
+                "shortname" : "Reproducible Selftest",
+                "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/ DISPLAY=:1 oe-selftest -r reproducible -j 1"],
+                "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
+
+            }
+        },
         "trigger-build" : {
             "SDKMACHINE" : "x86_64",
             "MACHINE" : "qemux86",
@@ -725,6 +735,24 @@
         "oe-selftest-centos" : {
             "TEMPLATE" : "selftest"
         },
+        "reproducible" : {
+            "TEMPLATE" : "reproducible"
+        },
+        "reproducible-ubuntu" : {
+            "TEMPLATE" : "reproducible"
+        },
+        "reproducible-debian" : {
+            "TEMPLATE" : "reproducible"
+        },
+        "reproducible-fedora" : {
+            "TEMPLATE" : "reproducible"
+        },
+        "reproducible-opensuse" : {
+            "TEMPLATE" : "reproducible"
+        },
+        "reproducible-centos" : {
+            "TEMPLATE" : "reproducible"
+        },
         "check-layer" : {
             "NEEDREPOS" : ["poky", "meta-gplv2", "meta-mingw"],
             "step1" : {
-- 
2.25.1

[yocto-autobuilder-helper][dunfell 23/23] config: build and test SDKs when using package_deb

[Steve Sakoman]

From: Ross Burton <ross@burtonini.com>

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b38664d2db940d2ef3238fdf0f2353162e120681)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 config.json | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/config.json b/config.json
index 3ce10c9..ac36a4d 100644
--- a/config.json
+++ b/config.json
@@ -559,8 +559,8 @@
         "pkgman-deb-non-deb" : {
             "MACHINE" : "qemux86",
             "PACKAGE_CLASSES" : "package_deb",
-            "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev",
-            "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage"
+            "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk",
+            "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk"
         },
         "pkgman-non-rpm" : {
             "MACHINE" : "qemux86",
-- 
2.25.1