* [yocto-autobuilder-helper][dunfell V2 01/15] scripts/run-config: Don't execute steps that don't exist!
2021-03-25 16:02 [yocto-autobuilder-helper][dunfell V2 00/15] Patch review V2 Steve Sakoman
@ 2021-03-25 16:02 ` Steve Sakoman
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 02/15] scripts/run-config: Ensure stepnum has a value when there are no steps Steve Sakoman
` (13 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Steve Sakoman @ 2021-03-25 16:02 UTC (permalink / raw)
To: yocto
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 290e1bc2ee18d5fa88aca84125fb6691db3db5f9)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
scripts/run-config | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/run-config b/scripts/run-config
index ce6072c..ff56fbe 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -81,7 +81,7 @@ elif args.build_type == "full":
ourconfig["HELPERSTMACHTARGS"] = "-a -t machine -t toolchain-user"
# Find out the number of steps this target has
-maxsteps = 1
+maxsteps = 0
if args.target in ourconfig['overrides']:
for v in ourconfig['overrides'][args.target]:
if v.startswith("step"):
--
2.25.1
^ permalink raw reply related [flat|nested] 16+ messages in thread* [yocto-autobuilder-helper][dunfell V2 02/15] scripts/run-config: Ensure stepnum has a value when there are no steps
2021-03-25 16:02 [yocto-autobuilder-helper][dunfell V2 00/15] Patch review V2 Steve Sakoman
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 01/15] scripts/run-config: Don't execute steps that don't exist! Steve Sakoman
@ 2021-03-25 16:02 ` Steve Sakoman
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 03/15] scripts/run-config: If target is present default to 1 step Steve Sakoman
` (12 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Steve Sakoman @ 2021-03-25 16:02 UTC (permalink / raw)
To: yocto
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit afb6c5a88773d4da4a8dfd88f19654ca585efc95)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
scripts/run-config | 1 +
1 file changed, 1 insertion(+)
diff --git a/scripts/run-config b/scripts/run-config
index ff56fbe..e600bf9 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -82,6 +82,7 @@ elif args.build_type == "full":
# Find out the number of steps this target has
maxsteps = 0
+stepnum = 0
if args.target in ourconfig['overrides']:
for v in ourconfig['overrides'][args.target]:
if v.startswith("step"):
--
2.25.1
^ permalink raw reply related [flat|nested] 16+ messages in thread* [yocto-autobuilder-helper][dunfell V2 03/15] scripts/run-config: If target is present default to 1 step
2021-03-25 16:02 [yocto-autobuilder-helper][dunfell V2 00/15] Patch review V2 Steve Sakoman
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 01/15] scripts/run-config: Don't execute steps that don't exist! Steve Sakoman
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 02/15] scripts/run-config: Ensure stepnum has a value when there are no steps Steve Sakoman
@ 2021-03-25 16:02 ` Steve Sakoman
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 04/15] run-config: Adapt to two pass execution Steve Sakoman
` (11 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Steve Sakoman @ 2021-03-25 16:02 UTC (permalink / raw)
To: yocto
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e183db413f3b67e0d45a2a9a697aa36b6c90601f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
scripts/run-config | 1 +
1 file changed, 1 insertion(+)
diff --git a/scripts/run-config b/scripts/run-config
index e600bf9..0b663df 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -84,6 +84,7 @@ elif args.build_type == "full":
maxsteps = 0
stepnum = 0
if args.target in ourconfig['overrides']:
+ maxsteps = 1
for v in ourconfig['overrides'][args.target]:
if v.startswith("step"):
n = int(v[4:])
--
2.25.1
^ permalink raw reply related [flat|nested] 16+ messages in thread* [yocto-autobuilder-helper][dunfell V2 04/15] run-config: Adapt to two pass execution
2021-03-25 16:02 [yocto-autobuilder-helper][dunfell V2 00/15] Patch review V2 Steve Sakoman
` (2 preceding siblings ...)
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 03/15] scripts/run-config: If target is present default to 1 step Steve Sakoman
@ 2021-03-25 16:02 ` Steve Sakoman
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 05/15] scripts/run-config: Improve logfile naming Steve Sakoman
` (10 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Steve Sakoman @ 2021-03-25 16:02 UTC (permalink / raw)
To: yocto
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 23d65680f8019bccc3fce20381dfcf49f265f601)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
scripts/run-config | 171 +++++++++++++++++++++++++++++++++------------
scripts/utils.py | 5 +-
2 files changed, 130 insertions(+), 46 deletions(-)
diff --git a/scripts/run-config b/scripts/run-config
index 0b663df..116dd49 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -52,6 +52,19 @@ parser.add_argument('--workername',
action='store',
default=None,
help="the name of the worker the build is running on")
+parser.add_argument('-j', '--json-outputfile',
+ action='store',
+ default="",
+ help="the file to store json information about the build in")
+parser.add_argument('--stepname',
+ action='store',
+ default=None,
+ help="the name of the step to run")
+parser.add_argument('--phase',
+ action='store',
+ default=None,
+ help="the phase of the step to run")
+
args = parser.parse_args()
@@ -94,7 +107,19 @@ if args.target in ourconfig['overrides']:
hp.printheader("Target task %s has %d steps" % (args.target, maxsteps))
-utils.setup_buildtools_tarball(ourconfig, args.workername, args.builddir + "/../buildtools")
+jcfg = False
+if args.json_outputfile:
+ jsonconfig = []
+ jcfg = True
+
+if jcfg:
+ buildtools = utils.setup_buildtools_tarball(ourconfig, args.workername, None, checkonly=True)
+ if buildtools:
+ jsonconfig.append({"name" : "buildtools", "description" : "Extract and setup buildtools tarball", "phase" : "init"})
+else:
+ utils.setup_buildtools_tarball(ourconfig, args.workername, args.builddir + "/../buildtools")
+ if args.phase == "init" and args.stepname == "buildtools":
+ sys.exit(0)
logconfig = args.builddir + "/../bitbake/contrib/autobuilderlog.json"
print("Using BB_LOGCONFIG=%s" % logconfig)
@@ -181,70 +206,126 @@ def runcmd(cmd, *args, **kwargs):
bh_path, remoterepo, remotebranch, baseremotebranch = utils.getbuildhistoryconfig(ourconfig, args.builddir, args.target, args.reponame, args.branchname, 1)
if bh_path:
- runcmd([os.path.join(scriptsdir, "buildhistory-init"), bh_path, remoterepo, remotebranch, baseremotebranch])
-
-for stepnum in range(1, maxsteps + 1):
+ if jcfg:
+ jsonconfig.append({"name" : "buildhistory-init", "description" : "Initialize buildhistory", "phase" : "init"})
+if args.phase == "init" and args.stepname == "buildhistory-init":
+ if bh_path:
+ runcmd([os.path.join(scriptsdir, "buildhistory-init"), bh_path, remoterepo, remotebranch, baseremotebranch])
+ sys.exit(0)
+
+def handle_stepnum(stepnum):
# Add any layers specified
layers = utils.getconfiglist("ADDLAYER", ourconfig, args.target, stepnum)
- for layer in layers:
- bitbakecmd(args.builddir, "bitbake-layers add-layer %s" % layer, report, stepnum, 'a')
+ if jcfg:
+ if layers:
+ jsonconfig.append({"name" : "add-layers", "description" : "Adding layers %s" % str(layers), "phase" : str(stepnum)})
+ elif args.stepname == "add-layers":
+ for layer in layers:
+ bitbakecmd(args.builddir, "bitbake-layers add-layer %s" % layer, report, stepnum, 'a')
flush()
+
# Generate the configuration files needed for this step
if utils.getconfigvar("WRITECONFIG", ourconfig, args.target, stepnum):
- runcmd([scriptsdir + "/setup-config", args.target, str(stepnum - 1), args.builddir, args.branchname, args.reponame, "-s", args.sstateprefix, "-b", args.buildappsrcrev])
+ if jcfg:
+ jsonconfig.append({"name" : "write-config", "description" : "Writing configuration files", "phase" : str(stepnum)})
+ elif args.stepname == "write-config":
+ runcmd([scriptsdir + "/setup-config", args.target, str(stepnum - 1), args.builddir, args.branchname, args.reponame, "-s", args.sstateprefix, "-b", args.buildappsrcrev])
# Execute the targets for this configuration
targets = utils.getconfigvar("BBTARGETS", ourconfig, args.target, stepnum)
if targets:
- hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, targets))
- bitbakecmd(args.builddir, "bitbake %s -k" % targets, report, stepnum, 'b')
+ if jcfg:
+ jsonconfig.append({"name" : "build-targets", "description" : "Building targets %s" % str(targets), "phase" : str(stepnum)})
+ elif args.stepname == "build-targets":
+ hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, targets))
+ bitbakecmd(args.builddir, "bitbake %s -k" % targets, report, stepnum, 'b')
# Execute the sanity targets for this configuration
sanitytargets = utils.getconfigvar("SANITYTARGETS", ourconfig, args.target, stepnum)
if sanitytargets:
- hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, sanitytargets))
- bitbakecmd(args.builddir, "%s/checkvnc; DISPLAY=:1 bitbake %s -k" % (scriptsdir, sanitytargets), report, stepnum, 'c')
+ if jcfg:
+ jsonconfig.append({"name" : "test-targets", "description" : "Running OEQA test targets %s" % str(sanitytargets), "phase" : str(stepnum)})
+ elif args.stepname == "test-targets":
+ hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, sanitytargets))
+ bitbakecmd(args.builddir, "%s/checkvnc; DISPLAY=:1 bitbake %s -k" % (scriptsdir, sanitytargets), report, stepnum, 'c')
# Run any extra commands specified
cmds = utils.getconfiglist("EXTRACMDS", ourconfig, args.target, stepnum)
- for cmd in cmds:
- hp.printheader("Step %s/%s: Running command %s" % (stepnum, maxsteps, cmd))
- bitbakecmd(args.builddir, cmd, report, stepnum, 'd')
+ if jcfg:
+ if cmds:
+ jsonconfig.append({"name" : "cmds", "description" : "Running bitbake environment commands %s" % str(cmds), "phase" : str(stepnum)})
+ elif args.stepname == "cmds":
+ for cmd in cmds:
+ hp.printheader("Step %s/%s: Running command %s" % (stepnum, maxsteps, cmd))
+ bitbakecmd(args.builddir, cmd, report, stepnum, 'd')
+
cmds = utils.getconfiglist("EXTRAPLAINCMDS", ourconfig, args.target, stepnum)
- for cmd in cmds:
- hp.printheader("Step %s/%s: Running 'plain' command %s" % (stepnum, maxsteps, cmd))
- bitbakecmd(args.builddir, cmd, report, stepnum, 'd', oeenv=False)
-
- # Remove any layers we added in a reverse order
- for layer in reversed(layers):
- bitbakecmd(args.builddir, "bitbake-layers remove-layer %s" % layer, report, stepnum, 'a')
-
-if args.publish_dir:
- hp.printheader("Running publish artefacts")
- runcmd([scriptsdir + "/publish-artefacts", args.builddir, args.publish_dir, args.target])
-
-if args.results_dir:
- hp.printheader("Running results collection")
- runcmd([scriptsdir + "/collect-results", args.builddir, args.results_dir, args.target])
-
-if args.build_url and utils.getconfigvar("SENDERRORS", ourconfig, args.target, stepnum):
- hp.printheader("Sending any error reports")
- runcmd([scriptsdir + "/upload-error-reports", args.builddir, args.build_url])
-
-if args.builddir and os.path.exists(args.builddir):
- # Clean up our build directory if things were successful and we're not publishing anything
- # (keep published builds around for longer just in case we need them)
- if not finalret and not args.publish_dir:
- runcmd([scriptsdir + "/../janitor/clobberdir", args.builddir])
- else:
- # Rename any completed build directory so that other builds can't reference paths within it
+ if jcfg:
+ if cmds:
+ jsonconfig.append({"name" : "plain-cmds", "description" : "Running commands %s" % str(cmds), "phase" : str(stepnum)})
+ elif args.stepname == "plain-cmds":
+ for cmd in cmds:
+ hp.printheader("Step %s/%s: Running 'plain' command %s" % (stepnum, maxsteps, cmd))
+ bitbakecmd(args.builddir, cmd, report, stepnum, 'd', oeenv=False)
+
+ if jcfg:
+ if layers:
+ jsonconfig.append({"name" : "remove-layers", "description" : "Removing layers %s" % str(layers), "phase" : str(stepnum)})
+ elif args.stepname == "remove-layers":
+ # Remove any layers we added in a reverse order
+ for layer in reversed(layers):
+ bitbakecmd(args.builddir, "bitbake-layers remove-layer %s" % layer, report, stepnum, 'a')
+
+ if not jcfg:
+ sys.exit(finalret)
+
+if jcfg:
+ for stepnum in range(1, maxsteps + 1):
+ handle_stepnum(stepnum)
+else:
+ try:
+ stepnum = int(args.phase)
+ except ValueError:
+ stepnum = None
+
+ if stepnum is not None:
+ handle_stepnum(stepnum)
+
+
+if jcfg:
+ jsonconfig.append({"name" : "publish", "description" : "Publishing artefacts", "phase" : "finish"})
+elif args.phase == "finish" and args.stepname == "publish":
+ if args.publish_dir:
+ hp.printheader("Running publish artefacts")
+ runcmd([scriptsdir + "/publish-artefacts", args.builddir, args.publish_dir, args.target])
+ sys.exit(0)
+
+if jcfg:
+ jsonconfig.append({"name" : "collect-results", "description" : "Collecting result files", "phase" : "finish"})
+elif args.phase == "finish" and args.stepname == "collect-results":
+ if args.results_dir:
+ hp.printheader("Running results collection")
+ runcmd([scriptsdir + "/collect-results", args.builddir, args.results_dir, args.target])
+ sys.exit(0)
+
+if jcfg:
+ jsonconfig.append({"name" : "send-errors", "description" : "Sending error reports", "phase" : "finish"})
+elif args.phase == "finish" and args.stepname == "send-errors":
+ if args.build_url and utils.getconfigvar("SENDERRORS", ourconfig, args.target, stepnum):
+ hp.printheader("Sending any error reports")
+ runcmd([scriptsdir + "/upload-error-reports", args.builddir, args.build_url])
+ sys.exit(0)
+
+if jcfg:
+ jsonconfig.append({"name" : "builddir-cleanup", "description" : "Cleaning up build directory", "phase" : "finish"})
+elif args.phase == "finish" and args.stepname == "builddir-cleanup":
+ if args.builddir and os.path.exists(args.builddir):
runcmd(["mv", args.builddir, args.builddir + "-renamed"])
-if finalret:
- hp.printheader("There were %s failures" % finalret)
- hp.printheader("Failures in logfiles: %s" % " ".join(errorlogs))
- sys.exit(1)
+if args.json_outputfile:
+ with open(args.json_outputfile, "w") as f:
+ json.dump(jsonconfig, f, indent=4, sort_keys=True)
sys.exit(0)
diff --git a/scripts/utils.py b/scripts/utils.py
index c7eb6c7..bf1d989 100644
--- a/scripts/utils.py
+++ b/scripts/utils.py
@@ -415,7 +415,7 @@ def enable_buildtools_tarball(btdir):
if line in os.environ:
del os.environ[line]
-def setup_buildtools_tarball(ourconfig, workername, btdir):
+def setup_buildtools_tarball(ourconfig, workername, btdir, checkonly=False):
bttarball = None
if "buildtools" in ourconfig and workername:
btcfg = getconfig("buildtools", ourconfig)
@@ -424,6 +424,9 @@ def setup_buildtools_tarball(ourconfig, workername, btdir):
bttarball = btcfg[entry]
break
+ if checkonly:
+ return bttarball
+
btenv = None
if bttarball:
sha256 = None
--
2.25.1
^ permalink raw reply related [flat|nested] 16+ messages in thread* [yocto-autobuilder-helper][dunfell V2 05/15] scripts/run-config: Improve logfile naming
2021-03-25 16:02 [yocto-autobuilder-helper][dunfell V2 00/15] Patch review V2 Steve Sakoman
` (3 preceding siblings ...)
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 04/15] run-config: Adapt to two pass execution Steve Sakoman
@ 2021-03-25 16:02 ` Steve Sakoman
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 06/15] scripts/run-config: Ensure logging to both logfile and stdout Steve Sakoman
` (9 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Steve Sakoman @ 2021-03-25 16:02 UTC (permalink / raw)
To: yocto
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 4a4c888f6618c3a7273c6dfe30b640e75e2b0de8)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
scripts/run-config | 28 ++++++++++++----------------
1 file changed, 12 insertions(+), 16 deletions(-)
diff --git a/scripts/run-config b/scripts/run-config
index 116dd49..25a4296 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -131,11 +131,8 @@ def flush():
sys.stdout.flush()
sys.stderr.flush()
-lognum = 0
-def logname(path, stepnum, logsuffix):
- global lognum
- lognum += 1
- return path + "/command.log.%s%s" % (stepnum, logsuffix)
+def logname(path, stepnum, stepname):
+ return path + "/command-%s-%s.log" % (stepnum, stepname)
utils.mkdir(args.builddir)
@@ -146,10 +143,10 @@ utils.mkdir(errordir)
errorlogs = set()
-def bitbakecmd(builddir, cmd, report, stepnum, logsuffix, oeenv=True):
+def bitbakecmd(builddir, cmd, report, stepnum, stepname, oeenv=True):
global finalret
flush()
- log = logname(builddir, stepnum, logsuffix)
+ log = logname(builddir, stepnum, stepname)
errordir = utils.errorreportdir(builddir)
try:
numreports = len(os.listdir(errordir))
@@ -176,10 +173,9 @@ def bitbakecmd(builddir, cmd, report, stepnum, logsuffix, oeenv=True):
with subprocess.Popen(cmd, shell=True, cwd=builddir + "/..", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1) as p, open(log, 'ab') as f:
for line in p.stdout:
- if not args.quietlogging:
- sys.stdout.buffer.write(line)
- sys.stdout.flush()
+ sys.stdout.buffer.write(line)
f.write(line)
+ sys.stdout.flush()
f.flush()
ret = p.wait()
if ret:
@@ -221,7 +217,7 @@ def handle_stepnum(stepnum):
jsonconfig.append({"name" : "add-layers", "description" : "Adding layers %s" % str(layers), "phase" : str(stepnum)})
elif args.stepname == "add-layers":
for layer in layers:
- bitbakecmd(args.builddir, "bitbake-layers add-layer %s" % layer, report, stepnum, 'a')
+ bitbakecmd(args.builddir, "bitbake-layers add-layer %s" % layer, report, stepnum, args.stepname)
flush()
@@ -239,7 +235,7 @@ def handle_stepnum(stepnum):
jsonconfig.append({"name" : "build-targets", "description" : "Building targets %s" % str(targets), "phase" : str(stepnum)})
elif args.stepname == "build-targets":
hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, targets))
- bitbakecmd(args.builddir, "bitbake %s -k" % targets, report, stepnum, 'b')
+ bitbakecmd(args.builddir, "bitbake %s -k" % targets, report, stepnum, args.stepname)
# Execute the sanity targets for this configuration
sanitytargets = utils.getconfigvar("SANITYTARGETS", ourconfig, args.target, stepnum)
@@ -248,7 +244,7 @@ def handle_stepnum(stepnum):
jsonconfig.append({"name" : "test-targets", "description" : "Running OEQA test targets %s" % str(sanitytargets), "phase" : str(stepnum)})
elif args.stepname == "test-targets":
hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, sanitytargets))
- bitbakecmd(args.builddir, "%s/checkvnc; DISPLAY=:1 bitbake %s -k" % (scriptsdir, sanitytargets), report, stepnum, 'c')
+ bitbakecmd(args.builddir, "%s/checkvnc; DISPLAY=:1 bitbake %s -k" % (scriptsdir, sanitytargets), report, stepnum, args.stepname)
# Run any extra commands specified
cmds = utils.getconfiglist("EXTRACMDS", ourconfig, args.target, stepnum)
@@ -258,7 +254,7 @@ def handle_stepnum(stepnum):
elif args.stepname == "cmds":
for cmd in cmds:
hp.printheader("Step %s/%s: Running command %s" % (stepnum, maxsteps, cmd))
- bitbakecmd(args.builddir, cmd, report, stepnum, 'd')
+ bitbakecmd(args.builddir, cmd, report, stepnum, args.stepname)
cmds = utils.getconfiglist("EXTRAPLAINCMDS", ourconfig, args.target, stepnum)
if jcfg:
@@ -267,7 +263,7 @@ def handle_stepnum(stepnum):
elif args.stepname == "plain-cmds":
for cmd in cmds:
hp.printheader("Step %s/%s: Running 'plain' command %s" % (stepnum, maxsteps, cmd))
- bitbakecmd(args.builddir, cmd, report, stepnum, 'd', oeenv=False)
+ bitbakecmd(args.builddir, cmd, report, stepnum, args.stepname, oeenv=False)
if jcfg:
if layers:
@@ -275,7 +271,7 @@ def handle_stepnum(stepnum):
elif args.stepname == "remove-layers":
# Remove any layers we added in a reverse order
for layer in reversed(layers):
- bitbakecmd(args.builddir, "bitbake-layers remove-layer %s" % layer, report, stepnum, 'a')
+ bitbakecmd(args.builddir, "bitbake-layers remove-layer %s" % layer, report, stepnum, args.stepname)
if not jcfg:
sys.exit(finalret)
--
2.25.1
^ permalink raw reply related [flat|nested] 16+ messages in thread* [yocto-autobuilder-helper][dunfell V2 06/15] scripts/run-config: Ensure logging to both logfile and stdout
2021-03-25 16:02 [yocto-autobuilder-helper][dunfell V2 00/15] Patch review V2 Steve Sakoman
` (4 preceding siblings ...)
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 05/15] scripts/run-config: Improve logfile naming Steve Sakoman
@ 2021-03-25 16:02 ` Steve Sakoman
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 07/15] config.json/run-config: Add human readable descriptions of steps Steve Sakoman
` (8 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Steve Sakoman @ 2021-03-25 16:02 UTC (permalink / raw)
To: yocto
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b1bc4d64c2d0a7e61aea154635996b6b4a4d04c2)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
scripts/run-config | 27 ++++++++++++++++-----------
1 file changed, 16 insertions(+), 11 deletions(-)
diff --git a/scripts/run-config b/scripts/run-config
index 25a4296..05c0579 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -153,6 +153,10 @@ def bitbakecmd(builddir, cmd, report, stepnum, stepname, oeenv=True):
except FileNotFoundError:
numreports = 0
+ def writelog(msg, a, b):
+ a.write(msg)
+ b.write(msg)
+
if oeenv:
cmd = ". ./oe-init-build-env; %s" % cmd
@@ -160,21 +164,22 @@ def bitbakecmd(builddir, cmd, report, stepnum, stepname, oeenv=True):
print("Would run '%s'" % cmd)
return
- print("Running '%s' with output to %s" % (cmd, log))
- flush()
+ with open(log, "a") as outf:
+ writelog("Running '%s' with output to %s\n" % (cmd, log), outf, sys.stdout)
- autoconf = builddir + "/conf/auto.conf"
- if os.path.exists(autoconf):
- with open(autoconf, "r") as inf, open(log, "a") as outf:
- outf.write("auto.conf settings:\n")
- for line in inf.readlines():
- outf.write(line)
- outf.write("\n")
+ autoconf = builddir + "/conf/auto.conf"
+ if os.path.exists(autoconf):
+ with open(autoconf, "r") as inf, open(log, "a") as outf:
+ writelog("auto.conf settings:\n", outf, sys.stdout)
+ for line in inf.readlines():
+ writelog(line, outf, sys.stdout)
+ writelog("\n", outf, sys.stdout)
+
+ flush()
with subprocess.Popen(cmd, shell=True, cwd=builddir + "/..", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1) as p, open(log, 'ab') as f:
for line in p.stdout:
- sys.stdout.buffer.write(line)
- f.write(line)
+ writelog(line, f, sys.stdout.buffer)
sys.stdout.flush()
f.flush()
ret = p.wait()
--
2.25.1
^ permalink raw reply related [flat|nested] 16+ messages in thread* [yocto-autobuilder-helper][dunfell V2 07/15] config.json/run-config: Add human readable descriptions of steps
2021-03-25 16:02 [yocto-autobuilder-helper][dunfell V2 00/15] Patch review V2 Steve Sakoman
` (5 preceding siblings ...)
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 06/15] scripts/run-config: Ensure logging to both logfile and stdout Steve Sakoman
@ 2021-03-25 16:02 ` Steve Sakoman
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 08/15] scripts/run-config: Remove redundant boilerplate json Steve Sakoman
` (7 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Steve Sakoman @ 2021-03-25 16:02 UTC (permalink / raw)
To: yocto
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ce63e8f024834a670cea660c61be632191aed9b9)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
config.json | 6 ++++++
scripts/run-config | 30 +++++++++++++++++-------------
2 files changed, 23 insertions(+), 13 deletions(-)
diff --git a/config.json b/config.json
index b67ef03..1fcc45d 100644
--- a/config.json
+++ b/config.json
@@ -453,6 +453,7 @@
"MACHINE" : "qemux86-64",
"SDKMACHINE" : "x86_64",
"step1" : {
+ "description" : "x86_64 32bit multilib image with rpm",
"BBTARGETS" : "lib32-core-image-minimal",
"SANITYTARGETS" : "lib32-core-image-minimal:do_testimage",
"extravars" : [
@@ -463,6 +464,7 @@
]
},
"step2" : {
+ "description" : "x86_64 32bit multilib image with ipk",
"PACKAGE_CLASSES" : "package_ipk",
"BBTARGETS" : "lib32-core-image-minimal",
"SANITYTARGETS" : "lib32-core-image-minimal:do_testimage",
@@ -474,6 +476,7 @@
]
},
"step3" : {
+ "description" : "x86_64 64bit image and 32 bit multilibs with rpm",
"BBTARGETS" : "core-image-sato",
"SANITYTARGETS" : "core-image-sato:do_testimage",
"extravars" : [
@@ -486,6 +489,7 @@
]
},
"step4" : {
+ "description" : "x86_64 64bit image and 32 bit multilibs with ipk",
"PACKAGE_CLASSES" : "package_ipk",
"BBTARGETS" : "core-image-sato",
"SANITYTARGETS" : "core-image-sato:do_testimage",
@@ -499,6 +503,7 @@
]
},
"step5" : {
+ "description" : "x86 building 64bit multilib image",
"MACHINE" : "qemux86",
"SDKMACHINE" : "i686",
"BBTARGETS" : "lib64-core-image-sato lib64-core-image-sato-sdk",
@@ -509,6 +514,7 @@
]
},
"step6" : {
+ "description" : "mips64 image using n32 as default",
"MACHINE" : "qemumips64",
"BBTARGETS" : "core-image-minimal core-image-minimal:do_populate_sdk",
"SANITYTARGETS" : "core-image-minimal:do_testimage core-image-minimal:do_testsdk",
diff --git a/scripts/run-config b/scripts/run-config
index 05c0579..89506f5 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -115,7 +115,7 @@ if args.json_outputfile:
if jcfg:
buildtools = utils.setup_buildtools_tarball(ourconfig, args.workername, None, checkonly=True)
if buildtools:
- jsonconfig.append({"name" : "buildtools", "description" : "Extract and setup buildtools tarball", "phase" : "init"})
+ jsonconfig.append({"name" : "buildtools", "bbname" : "Extract and setup buildtools tarball", "phase" : "init"})
else:
utils.setup_buildtools_tarball(ourconfig, args.workername, args.builddir + "/../buildtools")
if args.phase == "init" and args.stepname == "buildtools":
@@ -208,18 +208,22 @@ def runcmd(cmd, *args, **kwargs):
bh_path, remoterepo, remotebranch, baseremotebranch = utils.getbuildhistoryconfig(ourconfig, args.builddir, args.target, args.reponame, args.branchname, 1)
if bh_path:
if jcfg:
- jsonconfig.append({"name" : "buildhistory-init", "description" : "Initialize buildhistory", "phase" : "init"})
+ jsonconfig.append({"name" : "buildhistory-init", "bbname" : "Initialize buildhistory", "phase" : "init"})
if args.phase == "init" and args.stepname == "buildhistory-init":
if bh_path:
runcmd([os.path.join(scriptsdir, "buildhistory-init"), bh_path, remoterepo, remotebranch, baseremotebranch])
sys.exit(0)
def handle_stepnum(stepnum):
+ desc = utils.getconfigvar("description", ourconfig, args.target, stepnum)
+ if desc:
+ desc = desc + ": "
+
# Add any layers specified
layers = utils.getconfiglist("ADDLAYER", ourconfig, args.target, stepnum)
if jcfg:
if layers:
- jsonconfig.append({"name" : "add-layers", "description" : "Adding layers %s" % str(layers), "phase" : str(stepnum)})
+ jsonconfig.append({"name" : "add-layers", "bbname" : "%sAdding layers %s" % (desc, str(layers)), "phase" : str(stepnum)})
elif args.stepname == "add-layers":
for layer in layers:
bitbakecmd(args.builddir, "bitbake-layers add-layer %s" % layer, report, stepnum, args.stepname)
@@ -229,7 +233,7 @@ def handle_stepnum(stepnum):
# Generate the configuration files needed for this step
if utils.getconfigvar("WRITECONFIG", ourconfig, args.target, stepnum):
if jcfg:
- jsonconfig.append({"name" : "write-config", "description" : "Writing configuration files", "phase" : str(stepnum)})
+ jsonconfig.append({"name" : "write-config", "bbname" : "%sWriting configuration files" % desc, "phase" : str(stepnum)})
elif args.stepname == "write-config":
runcmd([scriptsdir + "/setup-config", args.target, str(stepnum - 1), args.builddir, args.branchname, args.reponame, "-s", args.sstateprefix, "-b", args.buildappsrcrev])
@@ -237,7 +241,7 @@ def handle_stepnum(stepnum):
targets = utils.getconfigvar("BBTARGETS", ourconfig, args.target, stepnum)
if targets:
if jcfg:
- jsonconfig.append({"name" : "build-targets", "description" : "Building targets %s" % str(targets), "phase" : str(stepnum)})
+ jsonconfig.append({"name" : "build-targets", "bbname" : "%sBuilding targets %s" % (desc, str(targets)), "phase" : str(stepnum)})
elif args.stepname == "build-targets":
hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, targets))
bitbakecmd(args.builddir, "bitbake %s -k" % targets, report, stepnum, args.stepname)
@@ -246,7 +250,7 @@ def handle_stepnum(stepnum):
sanitytargets = utils.getconfigvar("SANITYTARGETS", ourconfig, args.target, stepnum)
if sanitytargets:
if jcfg:
- jsonconfig.append({"name" : "test-targets", "description" : "Running OEQA test targets %s" % str(sanitytargets), "phase" : str(stepnum)})
+ jsonconfig.append({"name" : "test-targets", "bbname" : "%sRunning OEQA test targets %s" % (desc, str(sanitytargets)), "phase" : str(stepnum)})
elif args.stepname == "test-targets":
hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, sanitytargets))
bitbakecmd(args.builddir, "%s/checkvnc; DISPLAY=:1 bitbake %s -k" % (scriptsdir, sanitytargets), report, stepnum, args.stepname)
@@ -255,7 +259,7 @@ def handle_stepnum(stepnum):
cmds = utils.getconfiglist("EXTRACMDS", ourconfig, args.target, stepnum)
if jcfg:
if cmds:
- jsonconfig.append({"name" : "cmds", "description" : "Running bitbake environment commands %s" % str(cmds), "phase" : str(stepnum)})
+ jsonconfig.append({"name" : "cmds", "bbname" : "%sRunning bitbake environment commands %s" % (desc, str(cmds)), "phase" : str(stepnum)})
elif args.stepname == "cmds":
for cmd in cmds:
hp.printheader("Step %s/%s: Running command %s" % (stepnum, maxsteps, cmd))
@@ -264,7 +268,7 @@ def handle_stepnum(stepnum):
cmds = utils.getconfiglist("EXTRAPLAINCMDS", ourconfig, args.target, stepnum)
if jcfg:
if cmds:
- jsonconfig.append({"name" : "plain-cmds", "description" : "Running commands %s" % str(cmds), "phase" : str(stepnum)})
+ jsonconfig.append({"name" : "plain-cmds", "bbname" : "%sRunning commands %s" % (desc, str(cmds)), "phase" : str(stepnum)})
elif args.stepname == "plain-cmds":
for cmd in cmds:
hp.printheader("Step %s/%s: Running 'plain' command %s" % (stepnum, maxsteps, cmd))
@@ -272,7 +276,7 @@ def handle_stepnum(stepnum):
if jcfg:
if layers:
- jsonconfig.append({"name" : "remove-layers", "description" : "Removing layers %s" % str(layers), "phase" : str(stepnum)})
+ jsonconfig.append({"name" : "remove-layers", "bbname" : "%sRemoving layers %s" % (desc, str(layers)), "phase" : str(stepnum)})
elif args.stepname == "remove-layers":
# Remove any layers we added in a reverse order
for layer in reversed(layers):
@@ -295,7 +299,7 @@ else:
if jcfg:
- jsonconfig.append({"name" : "publish", "description" : "Publishing artefacts", "phase" : "finish"})
+ jsonconfig.append({"name" : "publish", "bbname" : "Publishing artefacts", "phase" : "finish"})
elif args.phase == "finish" and args.stepname == "publish":
if args.publish_dir:
hp.printheader("Running publish artefacts")
@@ -303,7 +307,7 @@ elif args.phase == "finish" and args.stepname == "publish":
sys.exit(0)
if jcfg:
- jsonconfig.append({"name" : "collect-results", "description" : "Collecting result files", "phase" : "finish"})
+ jsonconfig.append({"name" : "collect-results", "bbname" : "Collecting result files", "phase" : "finish"})
elif args.phase == "finish" and args.stepname == "collect-results":
if args.results_dir:
hp.printheader("Running results collection")
@@ -311,7 +315,7 @@ elif args.phase == "finish" and args.stepname == "collect-results":
sys.exit(0)
if jcfg:
- jsonconfig.append({"name" : "send-errors", "description" : "Sending error reports", "phase" : "finish"})
+ jsonconfig.append({"name" : "send-errors", "bbname" : "Sending error reports", "phase" : "finish"})
elif args.phase == "finish" and args.stepname == "send-errors":
if args.build_url and utils.getconfigvar("SENDERRORS", ourconfig, args.target, stepnum):
hp.printheader("Sending any error reports")
@@ -319,7 +323,7 @@ elif args.phase == "finish" and args.stepname == "send-errors":
sys.exit(0)
if jcfg:
- jsonconfig.append({"name" : "builddir-cleanup", "description" : "Cleaning up build directory", "phase" : "finish"})
+ jsonconfig.append({"name" : "builddir-cleanup", "bbname" : "Cleaning up build directory", "phase" : "finish"})
elif args.phase == "finish" and args.stepname == "builddir-cleanup":
if args.builddir and os.path.exists(args.builddir):
runcmd(["mv", args.builddir, args.builddir + "-renamed"])
--
2.25.1
^ permalink raw reply related [flat|nested] 16+ messages in thread* [yocto-autobuilder-helper][dunfell V2 08/15] scripts/run-config: Remove redundant boilerplate json
2021-03-25 16:02 [yocto-autobuilder-helper][dunfell V2 00/15] Patch review V2 Steve Sakoman
` (6 preceding siblings ...)
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 07/15] config.json/run-config: Add human readable descriptions of steps Steve Sakoman
@ 2021-03-25 16:02 ` Steve Sakoman
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 09/15] scripts/shared-repo-unpack: Add flush call to update the output more regularly before buildtools Steve Sakoman
` (6 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Steve Sakoman @ 2021-03-25 16:02 UTC (permalink / raw)
To: yocto
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2bb48042438f3154bbfa6fbc7f2c7556bfa7c762)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
scripts/run-config | 31 +++++++++++++++++--------------
1 file changed, 17 insertions(+), 14 deletions(-)
diff --git a/scripts/run-config b/scripts/run-config
index 89506f5..58ce364 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -112,10 +112,13 @@ if args.json_outputfile:
jsonconfig = []
jcfg = True
+def addentry(name, description, phase):
+ jsonconfig.append({"name" : name, "bbname" : description[:45], "phase" : phase, "description" : description})
+
if jcfg:
buildtools = utils.setup_buildtools_tarball(ourconfig, args.workername, None, checkonly=True)
if buildtools:
- jsonconfig.append({"name" : "buildtools", "bbname" : "Extract and setup buildtools tarball", "phase" : "init"})
+ addentry("buildtools", "Extract and setup buildtools tarball", "init")
else:
utils.setup_buildtools_tarball(ourconfig, args.workername, args.builddir + "/../buildtools")
if args.phase == "init" and args.stepname == "buildtools":
@@ -208,14 +211,14 @@ def runcmd(cmd, *args, **kwargs):
bh_path, remoterepo, remotebranch, baseremotebranch = utils.getbuildhistoryconfig(ourconfig, args.builddir, args.target, args.reponame, args.branchname, 1)
if bh_path:
if jcfg:
- jsonconfig.append({"name" : "buildhistory-init", "bbname" : "Initialize buildhistory", "phase" : "init"})
+ addentry("buildhistory-init", "Initialize buildhistory", "init")
if args.phase == "init" and args.stepname == "buildhistory-init":
if bh_path:
runcmd([os.path.join(scriptsdir, "buildhistory-init"), bh_path, remoterepo, remotebranch, baseremotebranch])
sys.exit(0)
def handle_stepnum(stepnum):
- desc = utils.getconfigvar("description", ourconfig, args.target, stepnum)
+ desc = utils.getconfigvar("description", ourconfig, args.target, stepnum) or ""
if desc:
desc = desc + ": "
@@ -223,7 +226,7 @@ def handle_stepnum(stepnum):
layers = utils.getconfiglist("ADDLAYER", ourconfig, args.target, stepnum)
if jcfg:
if layers:
- jsonconfig.append({"name" : "add-layers", "bbname" : "%sAdding layers %s" % (desc, str(layers)), "phase" : str(stepnum)})
+ addentry("add-layers", "%sAdding layers %s" % (desc, str(layers)), str(stepnum))
elif args.stepname == "add-layers":
for layer in layers:
bitbakecmd(args.builddir, "bitbake-layers add-layer %s" % layer, report, stepnum, args.stepname)
@@ -233,7 +236,7 @@ def handle_stepnum(stepnum):
# Generate the configuration files needed for this step
if utils.getconfigvar("WRITECONFIG", ourconfig, args.target, stepnum):
if jcfg:
- jsonconfig.append({"name" : "write-config", "bbname" : "%sWriting configuration files" % desc, "phase" : str(stepnum)})
+ addentry("write-config", "%sWriting configuration files" % desc, str(stepnum))
elif args.stepname == "write-config":
runcmd([scriptsdir + "/setup-config", args.target, str(stepnum - 1), args.builddir, args.branchname, args.reponame, "-s", args.sstateprefix, "-b", args.buildappsrcrev])
@@ -241,7 +244,7 @@ def handle_stepnum(stepnum):
targets = utils.getconfigvar("BBTARGETS", ourconfig, args.target, stepnum)
if targets:
if jcfg:
- jsonconfig.append({"name" : "build-targets", "bbname" : "%sBuilding targets %s" % (desc, str(targets)), "phase" : str(stepnum)})
+ addentry("build-targets", "%sBuilding targets %s" % (desc, str(targets)), str(stepnum))
elif args.stepname == "build-targets":
hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, targets))
bitbakecmd(args.builddir, "bitbake %s -k" % targets, report, stepnum, args.stepname)
@@ -250,7 +253,7 @@ def handle_stepnum(stepnum):
sanitytargets = utils.getconfigvar("SANITYTARGETS", ourconfig, args.target, stepnum)
if sanitytargets:
if jcfg:
- jsonconfig.append({"name" : "test-targets", "bbname" : "%sRunning OEQA test targets %s" % (desc, str(sanitytargets)), "phase" : str(stepnum)})
+ addentry("test-targets", "%sRunning OEQA test targets %s" % (desc, str(sanitytargets)), str(stepnum))
elif args.stepname == "test-targets":
hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, sanitytargets))
bitbakecmd(args.builddir, "%s/checkvnc; DISPLAY=:1 bitbake %s -k" % (scriptsdir, sanitytargets), report, stepnum, args.stepname)
@@ -259,7 +262,7 @@ def handle_stepnum(stepnum):
cmds = utils.getconfiglist("EXTRACMDS", ourconfig, args.target, stepnum)
if jcfg:
if cmds:
- jsonconfig.append({"name" : "cmds", "bbname" : "%sRunning bitbake environment commands %s" % (desc, str(cmds)), "phase" : str(stepnum)})
+ addentry("cmds", "%sRunning bitbake environment commands %s" % (desc, str(cmds)), str(stepnum))
elif args.stepname == "cmds":
for cmd in cmds:
hp.printheader("Step %s/%s: Running command %s" % (stepnum, maxsteps, cmd))
@@ -268,7 +271,7 @@ def handle_stepnum(stepnum):
cmds = utils.getconfiglist("EXTRAPLAINCMDS", ourconfig, args.target, stepnum)
if jcfg:
if cmds:
- jsonconfig.append({"name" : "plain-cmds", "bbname" : "%sRunning commands %s" % (desc, str(cmds)), "phase" : str(stepnum)})
+ addentry("plain-cmds", "%sRunning commands %s" % (desc, str(cmds)), str(stepnum))
elif args.stepname == "plain-cmds":
for cmd in cmds:
hp.printheader("Step %s/%s: Running 'plain' command %s" % (stepnum, maxsteps, cmd))
@@ -276,7 +279,7 @@ def handle_stepnum(stepnum):
if jcfg:
if layers:
- jsonconfig.append({"name" : "remove-layers", "bbname" : "%sRemoving layers %s" % (desc, str(layers)), "phase" : str(stepnum)})
+ addentry("remove-layers", "%sRemoving layers %s" % (desc, str(layers)), str(stepnum))
elif args.stepname == "remove-layers":
# Remove any layers we added in a reverse order
for layer in reversed(layers):
@@ -299,7 +302,7 @@ else:
if jcfg:
- jsonconfig.append({"name" : "publish", "bbname" : "Publishing artefacts", "phase" : "finish"})
+ addentry("publish", "Publishing artefacts", "finish")
elif args.phase == "finish" and args.stepname == "publish":
if args.publish_dir:
hp.printheader("Running publish artefacts")
@@ -307,7 +310,7 @@ elif args.phase == "finish" and args.stepname == "publish":
sys.exit(0)
if jcfg:
- jsonconfig.append({"name" : "collect-results", "bbname" : "Collecting result files", "phase" : "finish"})
+ addentry("collect-results", "Collecting result files", "finish")
elif args.phase == "finish" and args.stepname == "collect-results":
if args.results_dir:
hp.printheader("Running results collection")
@@ -315,7 +318,7 @@ elif args.phase == "finish" and args.stepname == "collect-results":
sys.exit(0)
if jcfg:
- jsonconfig.append({"name" : "send-errors", "bbname" : "Sending error reports", "phase" : "finish"})
+ addentry("send-errors", "Sending error reports", "finish")
elif args.phase == "finish" and args.stepname == "send-errors":
if args.build_url and utils.getconfigvar("SENDERRORS", ourconfig, args.target, stepnum):
hp.printheader("Sending any error reports")
@@ -323,7 +326,7 @@ elif args.phase == "finish" and args.stepname == "send-errors":
sys.exit(0)
if jcfg:
- jsonconfig.append({"name" : "builddir-cleanup", "bbname" : "Cleaning up build directory", "phase" : "finish"})
+ addentry("builddir-cleanup", "Cleaning up build directory", "finish")
elif args.phase == "finish" and args.stepname == "builddir-cleanup":
if args.builddir and os.path.exists(args.builddir):
runcmd(["mv", args.builddir, args.builddir + "-renamed"])
--
2.25.1
^ permalink raw reply related [flat|nested] 16+ messages in thread* [yocto-autobuilder-helper][dunfell V2 09/15] scripts/shared-repo-unpack: Add flush call to update the output more regularly before buildtools
2021-03-25 16:02 [yocto-autobuilder-helper][dunfell V2 00/15] Patch review V2 Steve Sakoman
` (7 preceding siblings ...)
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 08/15] scripts/run-config: Remove redundant boilerplate json Steve Sakoman
@ 2021-03-25 16:02 ` Steve Sakoman
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 10/15] config.json/run-config: Add support for shortnames and descriptions Steve Sakoman
` (5 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Steve Sakoman @ 2021-03-25 16:02 UTC (permalink / raw)
To: yocto
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6e2825564c0b7b69f56e6e589ec15a1cebdb26d1)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
scripts/shared-repo-unpack | 1 +
1 file changed, 1 insertion(+)
diff --git a/scripts/shared-repo-unpack b/scripts/shared-repo-unpack
index 7dc250c..f08efa8 100755
--- a/scripts/shared-repo-unpack
+++ b/scripts/shared-repo-unpack
@@ -60,6 +60,7 @@ for repo in sorted(repos.keys()):
utils.fetchgitrepo(targetsubdir, repo, repos[repo], stashdir)
if args.publish_dir:
utils.publishrepo(targetsubdir, repo, args.publish_dir)
+ utils.flush()
utils.setup_buildtools_tarball(ourconfig, args.workername, args.abworkdir + "/buildtools")
--
2.25.1
^ permalink raw reply related [flat|nested] 16+ messages in thread* [yocto-autobuilder-helper][dunfell V2 10/15] config.json/run-config: Add support for shortnames and descriptions
2021-03-25 16:02 [yocto-autobuilder-helper][dunfell V2 00/15] Patch review V2 Steve Sakoman
` (8 preceding siblings ...)
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 09/15] scripts/shared-repo-unpack: Add flush call to update the output more regularly before buildtools Steve Sakoman
@ 2021-03-25 16:02 ` Steve Sakoman
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 11/15] config.json: Unbreak qa-extras locked sigs test Steve Sakoman
` (4 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Steve Sakoman @ 2021-03-25 16:02 UTC (permalink / raw)
To: yocto
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Learn from the previous experiments and add meaninful shortnames and
descriptions to work around the 50 char name limit.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit bceb63fb7952c6ed289733471a0177cfbc365a1e)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
config.json | 16 +++++++++++-----
scripts/run-config | 34 +++++++++++++++++++++++-----------
2 files changed, 34 insertions(+), 16 deletions(-)
diff --git a/config.json b/config.json
index 1fcc45d..4856507 100644
--- a/config.json
+++ b/config.json
@@ -453,7 +453,8 @@
"MACHINE" : "qemux86-64",
"SDKMACHINE" : "x86_64",
"step1" : {
- "description" : "x86_64 32bit multilib image with rpm",
+ "shortname" : "x86-64 lib32-img rpm",
+ "description" : "qemux86-64 32bit multilib image with rpm",
"BBTARGETS" : "lib32-core-image-minimal",
"SANITYTARGETS" : "lib32-core-image-minimal:do_testimage",
"extravars" : [
@@ -464,7 +465,8 @@
]
},
"step2" : {
- "description" : "x86_64 32bit multilib image with ipk",
+ "shortname" : "x86-64 lib32-img ipk",
+ "description" : "qemux86-64 32bit multilib image with ipk",
"PACKAGE_CLASSES" : "package_ipk",
"BBTARGETS" : "lib32-core-image-minimal",
"SANITYTARGETS" : "lib32-core-image-minimal:do_testimage",
@@ -476,7 +478,8 @@
]
},
"step3" : {
- "description" : "x86_64 64bit image and 32 bit multilibs with rpm",
+ "shortname" : "x86-64 lib32 rpm",
+ "description" : "qemux86-64 64bit image and 32 bit multilibs with rpm",
"BBTARGETS" : "core-image-sato",
"SANITYTARGETS" : "core-image-sato:do_testimage",
"extravars" : [
@@ -489,7 +492,8 @@
]
},
"step4" : {
- "description" : "x86_64 64bit image and 32 bit multilibs with ipk",
+ "shortname" : "x86-64 lib32 ipk",
+ "description" : "qemux86-64 64bit image and 32 bit multilibs with ipk",
"PACKAGE_CLASSES" : "package_ipk",
"BBTARGETS" : "core-image-sato",
"SANITYTARGETS" : "core-image-sato:do_testimage",
@@ -503,6 +507,7 @@
]
},
"step5" : {
+ "shortname" : "x86-64 lib64-img",
"description" : "x86 building 64bit multilib image",
"MACHINE" : "qemux86",
"SDKMACHINE" : "i686",
@@ -514,7 +519,8 @@
]
},
"step6" : {
- "description" : "mips64 image using n32 as default",
+ "shortname" : "mip64 n32",
+ "description" : "qemumips64 image using n32 as default",
"MACHINE" : "qemumips64",
"BBTARGETS" : "core-image-minimal core-image-minimal:do_populate_sdk",
"SANITYTARGETS" : "core-image-minimal:do_testimage core-image-minimal:do_testsdk",
diff --git a/scripts/run-config b/scripts/run-config
index 58ce364..aab52c1 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -112,13 +112,26 @@ if args.json_outputfile:
jsonconfig = []
jcfg = True
+# There is a 50 char limit on "bbname" but buildbot may append "_1", "_2" if multiple steps
+# with the same name exist in a build
def addentry(name, description, phase):
- jsonconfig.append({"name" : name, "bbname" : description[:45], "phase" : phase, "description" : description})
+ jsonconfig.append({"name" : name, "bbname" : description[:46], "phase" : phase, "description" : description})
+
+def addstepentry(name, taskdesc, shortname, description, detail, phase):
+ bbname = taskdesc
+ if shortname:
+ bbname = shortname + ": " + taskdesc
+ bbdesc = taskdesc
+ if description:
+ bbdesc = description
+ if detail:
+ bbdesc = bbdesc + ": " + detail
+ jsonconfig.append({"name" : name, "bbname" : bbname[:46], "phase" : phase, "description" : bbdesc})
if jcfg:
buildtools = utils.setup_buildtools_tarball(ourconfig, args.workername, None, checkonly=True)
if buildtools:
- addentry("buildtools", "Extract and setup buildtools tarball", "init")
+ addentry("buildtools", "Setup buildtools tarball", "init")
else:
utils.setup_buildtools_tarball(ourconfig, args.workername, args.builddir + "/../buildtools")
if args.phase == "init" and args.stepname == "buildtools":
@@ -218,15 +231,14 @@ if args.phase == "init" and args.stepname == "buildhistory-init":
sys.exit(0)
def handle_stepnum(stepnum):
+ shortdesc = utils.getconfigvar("shortname", ourconfig, args.target, stepnum) or ""
desc = utils.getconfigvar("description", ourconfig, args.target, stepnum) or ""
- if desc:
- desc = desc + ": "
# Add any layers specified
layers = utils.getconfiglist("ADDLAYER", ourconfig, args.target, stepnum)
if jcfg:
if layers:
- addentry("add-layers", "%sAdding layers %s" % (desc, str(layers)), str(stepnum))
+ addstepentry("add-layers", "Add layers", shortdesc, desc, str(layers), str(stepnum))
elif args.stepname == "add-layers":
for layer in layers:
bitbakecmd(args.builddir, "bitbake-layers add-layer %s" % layer, report, stepnum, args.stepname)
@@ -236,7 +248,7 @@ def handle_stepnum(stepnum):
# Generate the configuration files needed for this step
if utils.getconfigvar("WRITECONFIG", ourconfig, args.target, stepnum):
if jcfg:
- addentry("write-config", "%sWriting configuration files" % desc, str(stepnum))
+ addstepentry("write-config", "Write config", shortdesc, desc, None, str(stepnum))
elif args.stepname == "write-config":
runcmd([scriptsdir + "/setup-config", args.target, str(stepnum - 1), args.builddir, args.branchname, args.reponame, "-s", args.sstateprefix, "-b", args.buildappsrcrev])
@@ -244,7 +256,7 @@ def handle_stepnum(stepnum):
targets = utils.getconfigvar("BBTARGETS", ourconfig, args.target, stepnum)
if targets:
if jcfg:
- addentry("build-targets", "%sBuilding targets %s" % (desc, str(targets)), str(stepnum))
+ addstepentry("build-targets", "Build targets", shortdesc, desc, str(targets), str(stepnum))
elif args.stepname == "build-targets":
hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, targets))
bitbakecmd(args.builddir, "bitbake %s -k" % targets, report, stepnum, args.stepname)
@@ -253,7 +265,7 @@ def handle_stepnum(stepnum):
sanitytargets = utils.getconfigvar("SANITYTARGETS", ourconfig, args.target, stepnum)
if sanitytargets:
if jcfg:
- addentry("test-targets", "%sRunning OEQA test targets %s" % (desc, str(sanitytargets)), str(stepnum))
+ addstepentry("test-targets", "QA targets", shortdesc, desc, str(sanitytargets), str(stepnum))
elif args.stepname == "test-targets":
hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, sanitytargets))
bitbakecmd(args.builddir, "%s/checkvnc; DISPLAY=:1 bitbake %s -k" % (scriptsdir, sanitytargets), report, stepnum, args.stepname)
@@ -262,7 +274,7 @@ def handle_stepnum(stepnum):
cmds = utils.getconfiglist("EXTRACMDS", ourconfig, args.target, stepnum)
if jcfg:
if cmds:
- addentry("cmds", "%sRunning bitbake environment commands %s" % (desc, str(cmds)), str(stepnum))
+ addstepentry("cmds", "Run cmds", shortdesc, desc, str(cmds), str(stepnum))
elif args.stepname == "cmds":
for cmd in cmds:
hp.printheader("Step %s/%s: Running command %s" % (stepnum, maxsteps, cmd))
@@ -271,7 +283,7 @@ def handle_stepnum(stepnum):
cmds = utils.getconfiglist("EXTRAPLAINCMDS", ourconfig, args.target, stepnum)
if jcfg:
if cmds:
- addentry("plain-cmds", "%sRunning commands %s" % (desc, str(cmds)), str(stepnum))
+ addstepentry("plain-cmds", "Run cmds", shortdesc, desc, str(cmds), str(stepnum))
elif args.stepname == "plain-cmds":
for cmd in cmds:
hp.printheader("Step %s/%s: Running 'plain' command %s" % (stepnum, maxsteps, cmd))
@@ -279,7 +291,7 @@ def handle_stepnum(stepnum):
if jcfg:
if layers:
- addentry("remove-layers", "%sRemoving layers %s" % (desc, str(layers)), str(stepnum))
+ addstepentry("remove-layers", "Remove layers", shortdesc, desc, str(layers), str(stepnum))
elif args.stepname == "remove-layers":
# Remove any layers we added in a reverse order
for layer in reversed(layers):
--
2.25.1
^ permalink raw reply related [flat|nested] 16+ messages in thread* [yocto-autobuilder-helper][dunfell V2 11/15] config.json: Unbreak qa-extras locked sigs test
2021-03-25 16:02 [yocto-autobuilder-helper][dunfell V2 00/15] Patch review V2 Steve Sakoman
` (9 preceding siblings ...)
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 10/15] config.json/run-config: Add support for shortnames and descriptions Steve Sakoman
@ 2021-03-25 16:02 ` Steve Sakoman
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 12/15] config.json: Add further descriptions Steve Sakoman
` (3 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Steve Sakoman @ 2021-03-25 16:02 UTC (permalink / raw)
To: yocto
From: Richard Purdie <richard.purdie@linuxfoundation.org>
qa-extras and qa-extras2 were split incorrectly, fix this.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 80fe0ab06972c46c82cde29cbdfcdac6e87dde99)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
config.json | 26 +++++++++++++-------------
1 file changed, 13 insertions(+), 13 deletions(-)
diff --git a/config.json b/config.json
index 4856507..ddf36ae 100644
--- a/config.json
+++ b/config.json
@@ -802,18 +802,18 @@
"TMPDIR = '${TOPDIR}/newtmp'",
"require ../locked-sigs.inc"
]
- }
- },
- "qa-extras2" : {
- "MACHINE" : "qemux86-64",
- "step1" : {
+ },
+ "step7" : {
"SDKMACHINE" : "x86_64",
"BBTARGETS" : "core-image-sato:do_populate_sdk_ext",
"extravars" : [
"TMPDIR = '${TOPDIR}/sdktmp'"
]
- },
- "step2" : {
+ }
+ },
+ "qa-extras2" : {
+ "MACHINE" : "qemux86-64",
+ "step1" : {
"BBTARGETS" : "core-image-sato",
"SANITYTARGETS" : "core-image-sato:do_testimage",
"extravars" : [
@@ -821,7 +821,7 @@
"TEST_SUITES_append = ' logrotate'"
]
},
- "step3" : {
+ "step2" : {
"BBTARGETS" : "core-image-sato",
"SANITYTARGETS" : "core-image-sato:do_testimage",
"extravars" : [
@@ -829,7 +829,7 @@
"TEST_SUITES_append = ' pam'"
]
},
- "step4" : {
+ "step3" : {
"BBTARGETS" : "core-image-sato",
"SANITYTARGETS" : "core-image-sato:do_testimage",
"extravars" : [
@@ -838,7 +838,7 @@
],
"ADDLAYER" : ["${BUILDDIR}/../meta-skeleton"]
},
- "step5" : {
+ "step4" : {
"BBTARGETS" : "core-image-sato",
"SANITYTARGETS" : "core-image-sato:do_testimage",
"extravars" : [
@@ -847,7 +847,7 @@
"TEST_SUITES_append = ' systemd'"
]
},
- "step6" : {
+ "step5" : {
"BBTARGETS" : "core-image-sato",
"SANITYTARGETS" : "core-image-sato:do_testimage",
"extravars" : [
@@ -855,7 +855,7 @@
"VIRTUAL-RUNTIME_init_manager = 'sysvinit'"
]
},
- "step7" : {
+ "step6" : {
"BBTARGETS" : "core-image-sato",
"SANITYTARGETS" : "core-image-sato:do_testimage",
"extravars" : [
@@ -865,7 +865,7 @@
"DISTRO_FEATURES_BACKFILL_CONSIDERED = 'sysvinit'"
]
},
- "step8" : {
+ "step7" : {
"BBTARGETS" : "core-image-sato",
"SANITYTARGETS" : "core-image-sato:do_testimage",
"extravars" : [
--
2.25.1
^ permalink raw reply related [flat|nested] 16+ messages in thread* [yocto-autobuilder-helper][dunfell V2 12/15] config.json: Add further descriptions
2021-03-25 16:02 [yocto-autobuilder-helper][dunfell V2 00/15] Patch review V2 Steve Sakoman
` (10 preceding siblings ...)
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 11/15] config.json: Unbreak qa-extras locked sigs test Steve Sakoman
@ 2021-03-25 16:02 ` Steve Sakoman
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 13/15] scripts/run-config: Disable output buffering Steve Sakoman
` (2 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Steve Sakoman @ 2021-03-25 16:02 UTC (permalink / raw)
To: yocto
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 19b7456b92c2eb7b2b27f1e378dbc793d068ee3c)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
config.json | 29 +++++++++++++++++++++++++++++
1 file changed, 29 insertions(+)
diff --git a/config.json b/config.json
index ddf36ae..cedcef7 100644
--- a/config.json
+++ b/config.json
@@ -71,6 +71,7 @@
"SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext"
},
"step3" : {
+ "shortname" : "Machine oe-selftest",
"BUILDHISTORY" : false,
"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"],
"ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
@@ -128,6 +129,7 @@
"SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk core-image-sato:do_testsdkext"
},
"step2" : {
+ "shortname" : "Machine oe-selftest",
"BUILDHISTORY" : false,
"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest -a -t machine -j 15"]
}
@@ -164,13 +166,16 @@
"RPM_GPG_SIGN_CHUNK = '1'"
],
"step1" : {
+ "shortname" : "Bitbake Selftest",
"EXTRACMDS" : ["bitbake-selftest"]
},
"step2" : {
+ "shortname" : "OE Selftest",
"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/ DISPLAY=:1 oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror devtool.DevtoolAddTests.test_devtool_add_npm recipetool.RecipetoolTests.test_recipetool_create_npm -T machine -T toolchain-user -T toolchain-system -j 15"],
"ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
},
"step3" : {
+ "shortname" : "Python Linter Report",
"EXTRACMDS" : ["if [ `which oe-pylint` ]; then mkdir -p ${HELPERRESULTSDIR}/${HELPERTARGET}; oe-pylint > ${HELPERRESULTSDIR}/${HELPERTARGET}/pylint.log || true; fi"]
}
},
@@ -178,6 +183,7 @@
"SDKMACHINE" : "x86_64",
"MACHINE" : "qemux86",
"step1" : {
+ "shortname" : "Sources pre-fetching",
"BBTARGETS" : "universe -c fetch -k",
"ADDLAYER" : ["${BUILDDIR}/../meta-selftest"],
"extravars" : [
@@ -189,6 +195,7 @@
"SDKMACHINE" : "x86_64",
"MACHINE" : "qemux86",
"step1" : {
+ "shortname" : "Source Mirror Selftest",
"EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest -r buildoptions.SourceMirroring.test_yocto_source_mirror"],
"ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
}
@@ -571,6 +578,7 @@
],
"step1" : {
"MACHINE" : "qemux86",
+ "shortname" : "qemux86 wic",
"BBTARGETS" : "wic-tools core-image-sato",
"EXTRACMDS" : [
"wic create directdisk -e core-image-sato -o ${BUILDDIR}/tmp/deploy/wic_images/qemux86/directdisk/core-image-sato/",
@@ -580,6 +588,7 @@
},
"step2" : {
"MACHINE" : "genericx86",
+ "shortname" : "genericx86 wic",
"BBTARGETS" : "wic-tools core-image-sato",
"EXTRACMDS" : [
"wic create directdisk -e core-image-sato -o ${BUILDDIR}/tmp/deploy/wic_images/genericx86/directdisk/core-image-sato/",
@@ -589,6 +598,7 @@
},
"step3" : {
"MACHINE" : "qemux86-64",
+ "shortname" : "qemux86-64 wic",
"BBTARGETS" : "wic-tools core-image-sato",
"EXTRACMDS" : [
"wic create directdisk -e core-image-sato -o ${BUILDDIR}/tmp/deploy/wic_images/qemux86-64/directdisk/core-image-sato/",
@@ -598,6 +608,7 @@
},
"step4" : {
"MACHINE" : "genericx86-64",
+ "shortname" : "genericx86-64 wic",
"BBTARGETS" : "wic-tools core-image-sato",
"EXTRACMDS" : [
"wic create directdisk -e core-image-sato -o ${BUILDDIR}/tmp/deploy/wic_images/genericx86-64/directdisk/core-image-sato/",
@@ -613,14 +624,17 @@
],
"step1" : {
"SDKMACHINE" : "x86_64",
+ "shortname" : "x86_64 tools",
"BBTARGETS" : "buildtools-tarball buildtools-extended-tarball uninative-tarball"
},
"step2" : {
"SDKMACHINE" : "i686",
+ "shortname" : "i686 tools",
"BBTARGETS" : "uninative-tarball"
},
"step3" : {
"SDKMACHINE" : "aarch64",
+ "shortname" : "aarch64 tools",
"BBTARGETS" : "buildtools-tarball buildtools-extended-tarball uninative-tarball"
}
},
@@ -635,9 +649,11 @@
"SOURCE_MIRROR_URL = 'file://${BASE_SHAREDDIR}/current_sources'"
],
"step1" : {
+ "shortname" : "Universe fetch",
"BBTARGETS" : "universe -k -c fetch"
},
"step2" : {
+ "shortname" : "BA image build",
"BBTARGETS" : "build-appliance-image"
}
},
@@ -766,6 +782,7 @@
"qa-extras" : {
"MACHINE" : "qemux86-64",
"step1" : {
+ "shortname" : "Readonly rootfs",
"BBTARGETS" : "core-image-minimal",
"SANITYTARGETS" : "core-image-minimal:do_testimage",
"extravars" : [
@@ -773,6 +790,7 @@
]
},
"step2" : {
+ "shortname" : "ROOT_HOME testing",
"BBTARGETS" : "core-image-minimal",
"SANITYTARGETS" : "core-image-minimal:do_testimage",
"extravars" : [
@@ -780,6 +798,7 @@
]
},
"step3" : {
+ "shortname" : "Full eSDK type",
"SDKMACHINE" : "x86_64",
"BBTARGETS" : "core-image-minimal:do_populate_sdk_ext",
"extravars" : [
@@ -787,15 +806,18 @@
]
},
"step4" : {
+ "shortname" : "Prep locked-sigs test",
"SDKMACHINE" : "x86_64",
"BBTARGETS" : "core-image-sato core-image-sato:do_populate_sdk_ext"
},
"step5" : {
+ "shortname" : "Prep #2 locked-sigs test",
"SDKMACHINE" : "x86_64",
"BBTARGETS" : "core-image-sato -S none",
"EXTRACMDS" : ["${SCRIPTSDIR}/../janitor/clobberdir ${BUILDDIR}/../build/tmp"]
},
"step6" : {
+ "shortname" : "Test locked-sigs image",
"SDKMACHINE" : "x86_64",
"BBTARGETS" : "core-image-sato",
"extravars" : [
@@ -804,6 +826,7 @@
]
},
"step7" : {
+ "shortname" : "Test locked-sigs eSDK",
"SDKMACHINE" : "x86_64",
"BBTARGETS" : "core-image-sato:do_populate_sdk_ext",
"extravars" : [
@@ -814,6 +837,7 @@
"qa-extras2" : {
"MACHINE" : "qemux86-64",
"step1" : {
+ "shortname" : "Test logrotate",
"BBTARGETS" : "core-image-sato",
"SANITYTARGETS" : "core-image-sato:do_testimage",
"extravars" : [
@@ -830,6 +854,7 @@
]
},
"step3" : {
+ "shortname" : "Test skeletoninit",
"BBTARGETS" : "core-image-sato",
"SANITYTARGETS" : "core-image-sato:do_testimage",
"extravars" : [
@@ -839,6 +864,7 @@
"ADDLAYER" : ["${BUILDDIR}/../meta-skeleton"]
},
"step4" : {
+ "shortname" : "Systemd with sysvinit compat",
"BBTARGETS" : "core-image-sato",
"SANITYTARGETS" : "core-image-sato:do_testimage",
"extravars" : [
@@ -848,6 +874,7 @@
]
},
"step5" : {
+ "shortname" : "Sysvinit with systemd",
"BBTARGETS" : "core-image-sato",
"SANITYTARGETS" : "core-image-sato:do_testimage",
"extravars" : [
@@ -856,6 +883,7 @@
]
},
"step6" : {
+ "shortname" : "Systemd",
"BBTARGETS" : "core-image-sato",
"SANITYTARGETS" : "core-image-sato:do_testimage",
"extravars" : [
@@ -866,6 +894,7 @@
]
},
"step7" : {
+ "shortname" : "Mesa gallium-llvm",
"BBTARGETS" : "core-image-sato",
"SANITYTARGETS" : "core-image-sato:do_testimage",
"extravars" : [
--
2.25.1
^ permalink raw reply related [flat|nested] 16+ messages in thread* [yocto-autobuilder-helper][dunfell V2 13/15] scripts/run-config: Disable output buffering
2021-03-25 16:02 [yocto-autobuilder-helper][dunfell V2 00/15] Patch review V2 Steve Sakoman
` (11 preceding siblings ...)
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 12/15] config.json: Add further descriptions Steve Sakoman
@ 2021-03-25 16:02 ` Steve Sakoman
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 14/15] config.json: Split reproduciblity tests into their own target Steve Sakoman
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 15/15] config: build and test SDKs when using package_deb Steve Sakoman
14 siblings, 0 replies; 16+ messages in thread
From: Steve Sakoman @ 2021-03-25 16:02 UTC (permalink / raw)
To: yocto
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Line buffering (bufsize=1) is unavailable with binary mode so use unbuffered
mode instead. This fixes python runtime warnings.
[YOCTO #14093]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit c21732937c89f7b13a4f8a9a02d7fcb15a4bad2d)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
scripts/run-config | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/run-config b/scripts/run-config
index aab52c1..8ed88cf 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -193,7 +193,7 @@ def bitbakecmd(builddir, cmd, report, stepnum, stepname, oeenv=True):
flush()
- with subprocess.Popen(cmd, shell=True, cwd=builddir + "/..", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1) as p, open(log, 'ab') as f:
+ with subprocess.Popen(cmd, shell=True, cwd=builddir + "/..", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=0) as p, open(log, 'ab') as f:
for line in p.stdout:
writelog(line, f, sys.stdout.buffer)
sys.stdout.flush()
--
2.25.1
^ permalink raw reply related [flat|nested] 16+ messages in thread* [yocto-autobuilder-helper][dunfell V2 14/15] config.json: Split reproduciblity tests into their own target
2021-03-25 16:02 [yocto-autobuilder-helper][dunfell V2 00/15] Patch review V2 Steve Sakoman
` (12 preceding siblings ...)
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 13/15] scripts/run-config: Disable output buffering Steve Sakoman
@ 2021-03-25 16:02 ` Steve Sakoman
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 15/15] config: build and test SDKs when using package_deb Steve Sakoman
14 siblings, 0 replies; 16+ messages in thread
From: Steve Sakoman @ 2021-03-25 16:02 UTC (permalink / raw)
To: yocto
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 97e0979d6eb0300951445bb2cc5eda315681302e)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
config.json | 30 +++++++++++++++++++++++++++++-
1 file changed, 29 insertions(+), 1 deletion(-)
diff --git a/config.json b/config.json
index cedcef7..0a5068a 100644
--- a/config.json
+++ b/config.json
@@ -171,7 +171,7 @@
},
"step2" : {
"shortname" : "OE Selftest",
- "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/ DISPLAY=:1 oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror devtool.DevtoolAddTests.test_devtool_add_npm recipetool.RecipetoolTests.test_recipetool_create_npm -T machine -T toolchain-user -T toolchain-system -j 15"],
+ "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/ DISPLAY=:1 oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror devtool.DevtoolAddTests.test_devtool_add_npm recipetool.RecipetoolTests.test_recipetool_create_npm reproducible -T machine -T toolchain-user -T toolchain-system -j 15"],
"ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
},
"step3" : {
@@ -179,6 +179,16 @@
"EXTRACMDS" : ["if [ `which oe-pylint` ]; then mkdir -p ${HELPERRESULTSDIR}/${HELPERTARGET}; oe-pylint > ${HELPERRESULTSDIR}/${HELPERTARGET}/pylint.log || true; fi"]
}
},
+ "reproducible" : {
+ "MACHINE" : "qemux86-64",
+ "SDKMACHINE" : "x86_64",
+ "step1" : {
+ "shortname" : "Reproducible Selftest",
+ "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; OEQA_DEBUGGING_SAVED_OUTPUT=${BASE_SHAREDDIR}/pub/repro-fail/ DISPLAY=:1 oe-selftest -r reproducible -j 1"],
+ "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
+
+ }
+ },
"trigger-build" : {
"SDKMACHINE" : "x86_64",
"MACHINE" : "qemux86",
@@ -725,6 +735,24 @@
"oe-selftest-centos" : {
"TEMPLATE" : "selftest"
},
+ "reproducible" : {
+ "TEMPLATE" : "reproducible"
+ },
+ "reproducible-ubuntu" : {
+ "TEMPLATE" : "reproducible"
+ },
+ "reproducible-debian" : {
+ "TEMPLATE" : "reproducible"
+ },
+ "reproducible-fedora" : {
+ "TEMPLATE" : "reproducible"
+ },
+ "reproducible-opensuse" : {
+ "TEMPLATE" : "reproducible"
+ },
+ "reproducible-centos" : {
+ "TEMPLATE" : "reproducible"
+ },
"check-layer" : {
"NEEDREPOS" : ["poky", "meta-gplv2", "meta-mingw"],
"step1" : {
--
2.25.1
^ permalink raw reply related [flat|nested] 16+ messages in thread* [yocto-autobuilder-helper][dunfell V2 15/15] config: build and test SDKs when using package_deb
2021-03-25 16:02 [yocto-autobuilder-helper][dunfell V2 00/15] Patch review V2 Steve Sakoman
` (13 preceding siblings ...)
2021-03-25 16:02 ` [yocto-autobuilder-helper][dunfell V2 14/15] config.json: Split reproduciblity tests into their own target Steve Sakoman
@ 2021-03-25 16:02 ` Steve Sakoman
14 siblings, 0 replies; 16+ messages in thread
From: Steve Sakoman @ 2021-03-25 16:02 UTC (permalink / raw)
To: yocto
From: Ross Burton <ross@burtonini.com>
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b38664d2db940d2ef3238fdf0f2353162e120681)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
config.json | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/config.json b/config.json
index 0a5068a..e77a8fe 100644
--- a/config.json
+++ b/config.json
@@ -559,8 +559,8 @@
"pkgman-deb-non-deb" : {
"MACHINE" : "qemux86",
"PACKAGE_CLASSES" : "package_deb",
- "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev",
- "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage"
+ "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk",
+ "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk"
},
"pkgman-non-rpm" : {
"MACHINE" : "qemux86",
--
2.25.1
^ permalink raw reply related [flat|nested] 16+ messages in thread