[PATCH v3 0/8] yocto-compat-layer: bitbake-diffsigs support + per machine checks + OE-core fixes

[PATCH v3 0/8] yocto-compat-layer: bitbake-diffsigs support + per machine checks + OE-core fixes

[Patrick Ohly]

Manually calling bitbake-diffsigs is cumbersome and doesn't work in a CI
environment, so now the test itself does it for test_signatures.

Regarding the BSP example that I ended up using: it actually was one
of the better BSP layers and only had one problem in a "bitbake world"
build instead of several as in other BSP layers.

All pre-requisites in bitbake and OE-core are merged and no further
work is planned, so this patch series could be merged now after
reviewing the changes.

The OE-core fixes are included here because they were developed in
parallel to enhancing the tool which found them. They probably should
be merged sooner.

V2: - use self.fail() instead of self.assertTrue(False, ...)
    - test_machine_signatures check (work in progress)
V3: - rebased onto master where some yocto-compat-layer enhancements were already merged
    - added OE-core fixes for issues found by the per-machine signature comparison
    - finished test_machine_signatures (shorter output, explanations
      on how to run bitbake-diffsigs, end report with one bitbake-diffsigs invocation)
    - allow "bitbake -S none world" to fail during test_machine_signatures,
      report such failures in a separate test_machine_world

Patrick Ohly (8):
  go-cross: avoid libgcc dependency
  gdb-cross: avoid tune specific paths
  -cross recipes: ignore TARGET_ARCH sstate hash
  yocto-compat-layer: include bitbake-diffsigs output
  yocto-compat-layer: also determine tune flags for each task
  yocto-compat-layer: add --additional-layers
  yocto-compat-layer: test signature differences when setting MACHINE
  yocto-compat-layer: better handling of per-machine world build breakage

 meta/recipes-devtools/binutils/binutils-cross.inc |   3 +-
 meta/recipes-devtools/gcc/gcc-cross.inc           |   3 +-
 meta/recipes-devtools/gdb/gdb-cross.inc           |   5 +-
 meta/recipes-devtools/go/go-cross.inc             |   7 +-
 scripts/lib/compatlayer/__init__.py               |  32 +-
 scripts/lib/compatlayer/cases/bsp.py              | 204 ++++++++++++++-
 scripts/lib/compatlayer/cases/common.py           |  16 +-
 scripts/yocto-compat-layer.py                     |  29 +-
 8 files changed, 288 insertions(+), 11 deletions(-)

base-commit: ec2931852b2a097c9c8cb0d7288f5ca1d79f401c
-- 
git-series 0.9.1

[PATCH v3 1/8] go-cross: avoid libgcc dependency

[Patrick Ohly]

libgcc gets compiled differently depending on the tune flags for the
target. That dependency would make go-cross also tune specific and
prevent sharing it between different machines using the same
architecture.

For example, MACHINE=intel-corei7-64 and MACHINE=qemux86-64 shared the
same go-cross-x86_64, but compiled libgcc differently.

The libgcc dependency gets inherited from go.inc, but does not seem to
be necessary for go-cross (compiling go-helloworld still succeeds).
The dependency is left in go.inc just in case that it is relevant for
the various on-target recipes which inherit that.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 meta/recipes-devtools/go/go-cross.inc | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/recipes-devtools/go/go-cross.inc b/meta/recipes-devtools/go/go-cross.inc
index 68f5efd..9b49738 100644
--- a/meta/recipes-devtools/go/go-cross.inc
+++ b/meta/recipes-devtools/go/go-cross.inc
@@ -4,6 +4,10 @@ DEPENDS += "gcc-cross-${TARGET_ARCH}"
 
 PN = "go-cross-${TARGET_ARCH}"
 
+# go-cross is built once per target architecture, and thus must not
+# depend on the tune-specific libgcc.
+DEPENDS_remove = "libgcc"
+
 FILESEXTRAPATHS =. "${FILE_DIRNAME}/go-cross:"
 
 GOROOT_FINAL = "${libdir}/go"
-- 
git-series 0.9.1

[PATCH v3 2/8] gdb-cross: avoid tune specific paths

[Patrick Ohly]

gdb-cross used to be specific to the tune flags, but isn't
anymore. Therefore it is enough to use TARGET_SYS instead of
TUNE_PKGARCH to create a unique path.

Fixes a sstate signature difference that was found via
yocto-compat-layer.py's test_machine_signatures check. In practice it
probably showed up as unnecessarily rebuilding gdb-cross when
switching between machines like intel-corei7-64 and qemux86-64.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 meta/recipes-devtools/gdb/gdb-cross.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/gdb/gdb-cross.inc b/meta/recipes-devtools/gdb/gdb-cross.inc
index a34223e..d92f31f 100644
--- a/meta/recipes-devtools/gdb/gdb-cross.inc
+++ b/meta/recipes-devtools/gdb/gdb-cross.inc
@@ -24,4 +24,4 @@ BPN = "gdb"
 inherit cross
 inherit gettext
 
-datadir .= "/gdb-${TUNE_PKGARCH}${TARGET_VENDOR}-${TARGET_OS}"
+datadir .= "/gdb-${TARGET_SYS}${TARGET_VENDOR}-${TARGET_OS}"
-- 
git-series 0.9.1

[PATCH v3 3/8] -cross recipes: ignore TARGET_ARCH sstate hash

[Patrick Ohly]

"yocto-compat-layer.py --machines" showed that shared packages like
gcc-cross-powerpc64 have a sstate signature that depends on
TUNEFLAGS. As a result, there are unnecessary rebuilds and potential
conflicts in a multiconfig.

That's due to the way how TARGET_ARCH is set. Richard Purdie suggested
setting TARGET_ARCH[vardepvalue] as fix, which works. It would be
shorter to do that in cross.inc instead of repeating the relevant line
in different recipes, but Richard was concerned about potential
side-effects in other usages of cross.inc.

TARGET_GOARM as used in go.inc is still causing signature differences
for go-cross-powerpc64 and machines b4420qds-64b and p5020ds-64b. This
needs further investigation.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 meta/recipes-devtools/binutils/binutils-cross.inc | 3 +++
 meta/recipes-devtools/gcc/gcc-cross.inc           | 3 +++
 meta/recipes-devtools/gdb/gdb-cross.inc           | 3 +++
 meta/recipes-devtools/go/go-cross.inc             | 3 +++
 4 files changed, 12 insertions(+)

diff --git a/meta/recipes-devtools/binutils/binutils-cross.inc b/meta/recipes-devtools/binutils/binutils-cross.inc
index 5e6f468..02ec891 100644
--- a/meta/recipes-devtools/binutils/binutils-cross.inc
+++ b/meta/recipes-devtools/binutils/binutils-cross.inc
@@ -4,6 +4,9 @@ PROVIDES = "virtual/${TARGET_PREFIX}binutils"
 PN = "binutils-cross-${TARGET_ARCH}"
 BPN = "binutils"
 
+# Ignore how TARGET_ARCH is computed.
+TARGET_ARCH[vardepvalue] = "${TARGET_ARCH}"
+
 INHIBIT_DEFAULT_DEPS = "1"
 INHIBIT_AUTOTOOLS_DEPS = "1"
 
diff --git a/meta/recipes-devtools/gcc/gcc-cross.inc b/meta/recipes-devtools/gcc/gcc-cross.inc
index bb5f7a0..c5c1645 100644
--- a/meta/recipes-devtools/gcc/gcc-cross.inc
+++ b/meta/recipes-devtools/gcc/gcc-cross.inc
@@ -11,6 +11,9 @@ python () {
 
 PN = "gcc-cross-${TARGET_ARCH}"
 
+# Ignore how TARGET_ARCH is computed.
+TARGET_ARCH[vardepvalue] = "${TARGET_ARCH}"
+
 require gcc-configure-common.inc
 
 # While we want the 'gnu' hash style, we explicitly set it to sysv here to
diff --git a/meta/recipes-devtools/gdb/gdb-cross.inc b/meta/recipes-devtools/gdb/gdb-cross.inc
index d92f31f..ebe329f 100644
--- a/meta/recipes-devtools/gdb/gdb-cross.inc
+++ b/meta/recipes-devtools/gdb/gdb-cross.inc
@@ -21,6 +21,9 @@ GDBPROPREFIX = ""
 PN = "gdb-cross-${TARGET_ARCH}"
 BPN = "gdb"
 
+# Ignore how TARGET_ARCH is computed.
+TARGET_ARCH[vardepvalue] = "${TARGET_ARCH}"
+
 inherit cross
 inherit gettext
 
diff --git a/meta/recipes-devtools/go/go-cross.inc b/meta/recipes-devtools/go/go-cross.inc
index 9b49738..8fa09b7 100644
--- a/meta/recipes-devtools/go/go-cross.inc
+++ b/meta/recipes-devtools/go/go-cross.inc
@@ -4,6 +4,9 @@ DEPENDS += "gcc-cross-${TARGET_ARCH}"
 
 PN = "go-cross-${TARGET_ARCH}"
 
+# Ignore how TARGET_ARCH is computed.
+TARGET_ARCH[vardepvalue] = "${TARGET_ARCH}"
+
 # go-cross is built once per target architecture, and thus must not
 # depend on the tune-specific libgcc.
 DEPENDS_remove = "libgcc"
-- 
git-series 0.9.1

[PATCH v3 4/8] yocto-compat-layer: include bitbake-diffsigs output

[Patrick Ohly]

After filtering out potential false positives, it becomes feasible to
include the output of bitbake-diffsigs for those tasks which
definitely have a change.

Depends on bitbake-diffsigs with the "--signature" parameter.

Enhanced output now is:

   AssertionError: False is not true : Layer meta-xxxx changed 120 signatures, initial differences (first hash without, second with layer):
      gstreamer1.0-plugins-base:do_fetch: 76973f19f2e30d282152bdd7e4efe5bb -> e6e7c6fa9f2bd59d7d8d107f7c6ca1ac
         Task dependencies changed from:
         ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch']
         to:
         ['GST_IMX_PATCHES_TO_APPEND', 'PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch']
         basehash changed from d679d30bd1ea41c56e57419b57587f3c to 090a79b45f5fa26d10f9d34e2ed7a1e6
            List of dependencies for variable SRC_URI changed from '{'PV', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]'}' to '{'GST_IMX_PATCHES_TO_APPEND', 'PV', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]'}'
         changed items: {'GST_IMX_PATCHES_TO_APPEND'}
         Dependency on variable GST_IMX_PATCHES_TO_APPEND was added
         Variable SRC_URI value changed:
         "     http://gstreamer.freedesktop.org/src/gst-plugins-base/gst-plugins-base-${PV}.tar.xz     file://get-caps-from-src-pad-when-query-caps.patch     file://0003-ssaparse-enhance-SSA-text-lines-parsing.patch     file://0004-subparse-set-need_segment-after-sink-pad-received-GS.patch     file://encodebin-Need-more-buffers-in-output-queue-for-bett.patch     file://make-gio_unix_2_0-dependency-configurable.patch     file://0001-introspection.m4-prefix-pkgconfig-paths-with-PKG_CON.patch     file://0001-Makefile.am-don-t-hardcode-libtool-name-when-running.patch     file://0002-Makefile.am-prefix-calls-to-pkg-config-with-PKG_CONF.patch     file://0003-riff-add-missing-include-directories-when-calling-in.patch     file://0004-rtsp-drop-incorrect-reference-to-gstreamer-sdp-in-Ma.patch [--] {+${GST_IMX_PATCHES_TO_APPEND}+}"

      pulseaudio:do_install: 6bb6fe23e11a6d5fef9c3a25e73e4f9c -> 3f54ea75673a792e307197cfa6ef2694
         basehash changed from ac4efcfa783bd04a5a98a2c38719aedd to 37679d99623a37c8df955da3a01415a5
         Variable do_install value changed:
         @@ -1,3 +1,7 @@
              autotools_do_install
           	install -d ${D}${sysconfdir}/default/volatiles
          	install -m 0644 ${WORKDIR}/volatiles.04_pulse  ${D}${sysconfdir}/default/volatiles/volatiles.04_pulse
         +    if [ -e "${WORKDIR}/daemon.conf" ] && [ -e "${WORKDIR}/default.pa" ]; then
         +        install -m 0644 ${WORKDIR}/daemon.conf ${D}${sysconfdir}/pulse/daemon.conf
         +        install -m 0644 ${WORKDIR}/default.pa ${D}${sysconfdir}/pulse/default.pa
         +    fi

[YOCTO #11161]

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 scripts/lib/compatlayer/cases/common.py | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/compatlayer/cases/common.py b/scripts/lib/compatlayer/cases/common.py
index b91da9b..a4c2a51 100644
--- a/scripts/lib/compatlayer/cases/common.py
+++ b/scripts/lib/compatlayer/cases/common.py
@@ -75,5 +75,17 @@ class CommonCompatLayer(OECompatLayerTestCase):
             msg.append('Layer %s changed %d signatures, initial differences (first hash without, second with layer):' %
                        (self.tc.layer['name'], len(sig_diff)))
             for diff in sorted(sig_diff_filtered):
+                recipe, taskname = diff[0].rsplit(':', 1)
+                cmd = 'bitbake-diffsigs --task %s %s --signature %s %s' % \
+                      (recipe, taskname, diff[1], diff[2])
                 msg.append('   %s: %s -> %s' % diff)
-            self.assertTrue(False, '\n'.join(msg))
+                msg.append('      %s' % cmd)
+                try:
+                    output = check_command('Determining signature difference failed.',
+                                           cmd).decode('utf-8')
+                except RuntimeError as error:
+                    output = str(error)
+                if output:
+                    msg.extend(['      ' + line for line in output.splitlines()])
+                    msg.append('')
+            self.fail('\n'.join(msg))
-- 
git-series 0.9.1

[PATCH v3 5/8] yocto-compat-layer: also determine tune flags for each task

[Patrick Ohly]

locked-sigs.inc groups tasks according to their tune flags (allarch,
i586, etc.). Also retrieve that information while getting signatures,
it will be needed to determine when setting a machine changes tasks
that aren't machine-specific.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 scripts/lib/compatlayer/__init__.py     |  9 ++++++++-
 scripts/lib/compatlayer/cases/common.py |  2 +-
 scripts/yocto-compat-layer.py           |  2 +-
 3 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/compatlayer/__init__.py b/scripts/lib/compatlayer/__init__.py
index b46527a..6130b85 100644
--- a/scripts/lib/compatlayer/__init__.py
+++ b/scripts/lib/compatlayer/__init__.py
@@ -224,6 +224,7 @@ def get_signatures(builddir, failsafe=False):
     exclude_recipes = ('meta-world-pkgdata',)
 
     sigs = {}
+    tune2tasks = {}
 
     cmd = 'bitbake '
     if failsafe:
@@ -234,9 +235,14 @@ def get_signatures(builddir, failsafe=False):
     sigs_file = os.path.join(builddir, 'locked-sigs.inc')
 
     sig_regex = re.compile("^(?P<task>.*:.*):(?P<hash>.*) .$")
+    tune_regex = re.compile("(^|\s)SIGGEN_LOCKEDSIGS_t-(?P<tune>\S*)\s*=\s*")
+    current_tune = None
     with open(sigs_file, 'r') as f:
         for line in f.readlines():
             line = line.strip()
+            t = tune_regex.search(line)
+            if t:
+                current_tune = t.group('tune')
             s = sig_regex.match(line)
             if s:
                 exclude = False
@@ -249,11 +255,12 @@ def get_signatures(builddir, failsafe=False):
                     continue
 
                 sigs[s.group('task')] = s.group('hash')
+                tune2tasks.setdefault(current_tune, []).append(s.group('task'))
 
     if not sigs:
         raise RuntimeError('Can\'t load signatures from %s' % sigs_file)
 
-    return sigs
+    return (sigs, tune2tasks)
 
 def get_depgraph(targets=['world']):
     '''
diff --git a/scripts/lib/compatlayer/cases/common.py b/scripts/lib/compatlayer/cases/common.py
index a4c2a51..8eeada9 100644
--- a/scripts/lib/compatlayer/cases/common.py
+++ b/scripts/lib/compatlayer/cases/common.py
@@ -33,7 +33,7 @@ class CommonCompatLayer(OECompatLayerTestCase):
 
         # task -> (old signature, new signature)
         sig_diff = {}
-        curr_sigs = get_signatures(self.td['builddir'], failsafe=True)
+        curr_sigs, _ = get_signatures(self.td['builddir'], failsafe=True)
         for task in self.td['sigs']:
             if task in curr_sigs and \
                self.td['sigs'][task] != curr_sigs[task]:
diff --git a/scripts/yocto-compat-layer.py b/scripts/yocto-compat-layer.py
index 22c0c2d..2ebddb6 100755
--- a/scripts/yocto-compat-layer.py
+++ b/scripts/yocto-compat-layer.py
@@ -139,7 +139,7 @@ def main():
         td['bbvars'] = get_bb_vars()
         logger.info('Getting initial signatures ...')
         td['builddir'] = builddir
-        td['sigs'] = get_signatures(td['builddir'])
+        td['sigs'], td['tunetasks'] = get_signatures(td['builddir'])
 
         if not add_layer(bblayersconf, layer, dep_layers, logger):
             logger.info('Skipping %s ???.' % layer['name'])
-- 
git-series 0.9.1

[PATCH v3 6/8] yocto-compat-layer: add --additional-layers

[Patrick Ohly]

The new --addditional-layers parameter takes a list of layer
directories and adds them to the build configuration before starting
testing. The resulting base configuration then more closely matches
a full distro.

This is relevant in two cases:
1. some layers like meta-freescale dynamically enable more recipes
   in their layer.conf depending on which other layers are active,
   so testing only against OE-core might miss problems which occur
   only when also some other layers are active
2. BSP layers might be fine in combination with machines from
   OE-core, but might break in combination to some other machines

As before, test_signatures only warns about signature changes
introduced by the layer which is under testing, and not those changes
introduced by the additional layers.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 scripts/yocto-compat-layer.py | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)

diff --git a/scripts/yocto-compat-layer.py b/scripts/yocto-compat-layer.py
index 2ebddb6..daf3a6a 100755
--- a/scripts/yocto-compat-layer.py
+++ b/scripts/yocto-compat-layer.py
@@ -49,6 +49,8 @@ def main():
             help='File to output log (optional)', action='store')
     parser.add_argument('--dependency', nargs="+",
             help='Layers to process for dependencies', action='store')
+    parser.add_argument('--additional-layers', nargs="+",
+            help='List of additional layers to add during testing', action='store')
     parser.add_argument('-n', '--no-auto', help='Disable auto layer discovery',
             action='store_true')
     parser.add_argument('-d', '--debug', help='Enable debug output',
@@ -82,6 +84,10 @@ def main():
     if not layers:
         logger.error("Fail to detect layers")
         return 1
+    if args.additional_layers:
+        additional_layers = detect_layers(args.additional_layers, args.no_auto)
+    else:
+        additional_layers = []
     if args.dependency:
         dep_layers = detect_layers(args.dependency, args.no_auto)
         dep_layers = dep_layers + layers
@@ -128,13 +134,29 @@ def main():
 
         shutil.copyfile(bblayersconf + '.backup', bblayersconf)
 
-        if not add_layer_dependencies(bblayersconf, layer, dep_layers, logger):
+        missing_dependencies = not add_layer_dependencies(bblayersconf, layer, dep_layers, logger)
+        if not missing_dependencies:
+            for additional_layer in additional_layers:
+                if not add_layer_dependencies(bblayersconf, additional_layer, dep_layers, logger):
+                    missing_dependencies = True
+                    break
+        if not add_layer_dependencies(bblayersconf, layer, dep_layers, logger) or \
+           any(map(lambda additional_layer: not add_layer_dependencies(bblayersconf, additional_layer, dep_layers, logger),
+                   additional_layers)):
             logger.info('Skipping %s due to missing dependencies.' % layer['name'])
             results[layer['name']] = None
             results_status[layer['name']] = 'SKIPPED (Missing dependencies)'
             layers_tested = layers_tested + 1
             continue
 
+        if any(map(lambda additional_layer: not add_layer(bblayersconf, additional_layer, dep_layers, logger),
+                   additional_layers)):
+            logger.info('Skipping %s due to missing additional layers.' % layer['name'])
+            results[layer['name']] = None
+            results_status[layer['name']] = 'SKIPPED (Missing additional layers)'
+            layers_tested = layers_tested + 1
+            continue
+
         logger.info('Getting initial bitbake variables ...')
         td['bbvars'] = get_bb_vars()
         logger.info('Getting initial signatures ...')
-- 
git-series 0.9.1

[PATCH v3 7/8] yocto-compat-layer: test signature differences when setting MACHINE

[Patrick Ohly]

Selecting a machine is only allowed to affect the signature of tasks
that are specific to that machine. In other words, when MACHINE=A and
MACHINE=B share a recipe foo and the output of foo, then both machine
configurations must build foo in exactly the same way. Otherwise it is
not possible to use both machines in the same distribution.

This criteria can only be tested by testing different machines in combination,
i.e. one main layer, potentially several additional BSP layers and an explicit
choice of machines:
yocto-compat-layer --additional-layers .../meta-intel --machines intel-corei7-64 imx6slevk -- .../meta-freescale

The test has already found issues in go-cross (depended on
tune-specific libgcc) and gdb-cross (had a tune-specific path
unnecessarily), so it is also useful to uncover issues that are not
caused by the BSP layer itself.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 scripts/lib/compatlayer/__init__.py  |   7 +-
 scripts/lib/compatlayer/cases/bsp.py | 178 +++++++++++++++++++++++++++-
 scripts/yocto-compat-layer.py        |   3 +-
 3 files changed, 185 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/compatlayer/__init__.py b/scripts/lib/compatlayer/__init__.py
index 6130b85..0d6f4e9 100644
--- a/scripts/lib/compatlayer/__init__.py
+++ b/scripts/lib/compatlayer/__init__.py
@@ -215,7 +215,7 @@ def check_command(error_msg, cmd):
         raise RuntimeError(msg)
     return output
 
-def get_signatures(builddir, failsafe=False):
+def get_signatures(builddir, failsafe=False, machine=None):
     import re
 
     # some recipes needs to be excluded like meta-world-pkgdata
@@ -226,7 +226,10 @@ def get_signatures(builddir, failsafe=False):
     sigs = {}
     tune2tasks = {}
 
-    cmd = 'bitbake '
+    cmd = ''
+    if machine:
+        cmd += 'MACHINE=%s ' % machine
+    cmd += 'bitbake '
     if failsafe:
         cmd += '-k '
     cmd += '-S none world'
diff --git a/scripts/lib/compatlayer/cases/bsp.py b/scripts/lib/compatlayer/cases/bsp.py
index 5d9bf93..b205568 100644
--- a/scripts/lib/compatlayer/cases/bsp.py
+++ b/scripts/lib/compatlayer/cases/bsp.py
@@ -3,7 +3,7 @@
 
 import unittest
 
-from compatlayer import LayerType
+from compatlayer import LayerType, get_signatures, check_command
 from compatlayer.case import OECompatLayerTestCase
 
 class BSPCompatLayer(OECompatLayerTestCase):
@@ -24,3 +24,179 @@ class BSPCompatLayer(OECompatLayerTestCase):
         self.assertEqual(self.td['bbvars']['MACHINE'], machine,
                 msg="Layer %s modified machine %s -> %s" % \
                     (self.tc.layer['name'], self.td['bbvars']['MACHINE'], machine))
+
+    def test_machine_signatures(self):
+        '''
+        Selecting a machine may only affect the signature of tasks that are specific
+        to that machine. In other words, when MACHINE=A and MACHINE=B share a recipe
+        foo and the output of foo, then both machine configurations must build foo
+        in exactly the same way. Otherwise it is not possible to use both machines
+        in the same distribution.
+
+        This criteria can only be tested by testing different machines in combination,
+        i.e. one main layer, potentially several additional BSP layers and an explicit
+        choice of machines:
+        yocto-compat-layer --additional-layers .../meta-intel --machines intel-corei7-64 imx6slevk -- .../meta-freescale
+        '''
+
+        if not self.td['machines']:
+            self.skipTest('No machines set with --machines.')
+
+        # Collect signatures for all machines that we are testing
+        # and merge that into a hash:
+        # tune -> task -> signature -> list of machines with that combination
+        #
+        # It is an error if any tune/task pair has more than one signature,
+        # because that implies that the machines that caused those different
+        # signatures do not agree on how to execute the task.
+        tunes = {}
+        # Preserve ordering of machines as chosen by the user.
+        for machine in self.td['machines']:
+            curr_sigs, tune2tasks = get_signatures(self.td['builddir'], failsafe=True, machine=machine)
+            # Invert the tune -> [tasks] mapping.
+            tasks2tune = {}
+            for tune, tasks in tune2tasks.items():
+                for task in tasks:
+                    tasks2tune[task] = tune
+            for task, sighash in curr_sigs.items():
+                tunes.setdefault(tasks2tune[task], {}).setdefault(task, {}).setdefault(sighash, []).append(machine)
+
+        msg = []
+        pruned = 0
+        last_line_key = None
+        # task order as determined by completion scheduler ("merged task list"),
+        # edited a bit to add tasks not active in the build config that was used
+        taskname_list = ( 'do_fetch',
+                          'do_unpack',
+                          'do_kernel_checkout',
+                          'do_validate_branches',
+                          'do_kernel_metadata',
+                          'do_patch',
+                          'do_preconfigure',
+                          'do_kernel_configme',
+                          'do_generate_toolchain_file',
+                          'do_pam_sanity',
+                          'do_prepare_recipe_sysroot',
+                          'do_configure',
+                          'do_kernel_configcheck',
+                          'do_kernel_version_sanity_check',
+                          'do_configure_ptest_base',
+                          'do_compile',
+                          'do_shared_workdir',
+                          'do_kernel_link_images',
+                          'do_compile_kernelmodules',
+                          'do_strip',
+                          'do_sizecheck',
+                          'do_uboot_mkimage',
+                          'do_gcc_stash_builddir',
+                          'do_compile_ptest_base',
+                          'do_install',
+                          'do_multilib_install',
+                          'do_extra_symlinks',
+                          'do_stash_locale',
+                          'do_poststash_install_cleanup',
+                          'do_install_ptest_base',
+                          'do_package',
+                          'do_packagedata',
+                          'do_populate_sysroot',
+                          'do_bundle_initramfs',
+                          'do_populate_lic',
+                          'do_generate_rmc_db',
+                          'do_deploy',
+                          'do_package_write_ipk',
+                          'do_package_write_rpm',
+                          'do_package_write_deb',
+                          'do_rootfs',
+                          'do_write_qemuboot_conf',
+                          'do_image',
+                          'do_image_cpio',
+                          'do_write_wks_template',
+                          'do_rootfs_wicenv',
+                          'do_image_wic',
+                          'do_image_complete',
+                          'do_image_qa',
+                          'do_deploy_files',
+                          'do_package_qa',
+                          'do_rm_work',
+                          'do_rm_work_all',
+                          'do_build_sysroot',
+                          'do_build')
+        taskname_order = dict([(task, index) for index, task in enumerate(taskname_list) ])
+        def task_key(task):
+            pn, taskname = task.rsplit(':', 1)
+            return (pn, taskname_order.get(taskname, 0), taskname)
+
+        for tune in sorted(tunes.keys()):
+            tasks = tunes[tune]
+            # As for test_signatures it would be nicer to sort tasks
+            # by dependencies here, but that is harder because we have
+            # to report on tasks from different machines, which might
+            # have different dependencies. We resort to pruning the
+            # output by reporting only one task per recipe if the set
+            # of machines matches.
+            #
+            # "bitbake-diffsigs -t -s" is intelligent enough to print
+            # diffs recursively, so often it does not matter that much
+            # if we don't pick the underlying difference
+            # here. However, sometimes recursion fails
+            # (https://bugzilla.yoctoproject.org/show_bug.cgi?id=6428).
+            #
+            # To mitigate that a bit, we use a hard-coded ordering of
+            # tasks that represents how they normally run and prefer
+            # to print the ones that run first.
+            for task in sorted(tasks.keys(), key=task_key):
+                signatures = tasks[task]
+                # do_build can be ignored: it is know to have
+                # different signatures in some cases, for example in
+                # the allarch ca-certificates due to RDEPENDS=openssl.
+                # That particular dependency is whitelisted via
+                # SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, but still shows up
+                # in the sstate signature hash because filtering it
+                # out would be hard and running do_build multiple
+                # times doesn't really matter.
+                if len(signatures.keys()) > 1 and \
+                   not task.endswith(':do_build'):
+                    # Error!
+                    #
+                    # Sort signatures by machines, because the hex values don't mean anything.
+                    # => all-arch adwaita-icon-theme:do_build: 1234... (beaglebone, qemux86) != abcdf... (qemux86-64)
+                    #
+                    # Skip the line if it is covered already by the predecessor (same pn, same sets of machines).
+                    pn, taskname = task.rsplit(':', 1)
+                    next_line_key = (pn, sorted(signatures.values()))
+                    if next_line_key != last_line_key:
+                        line = '   %s %s: ' % (tune, task)
+                        line += ' != '.join(['%s (%s)' % (signature, ', '.join([m for m in signatures[signature]])) for
+                                             signature in sorted(signatures.keys(), key=lambda s: signatures[s])])
+                        last_line_key = next_line_key
+                        msg.append(line)
+                        # Randomly pick two mismatched signatures and remember how to invoke
+                        # bitbake-diffsigs for them.
+                        iterator = iter(signatures.items())
+                        a = next(iterator)
+                        b = next(iterator)
+                        diffsig_machines = '(%s) != (%s)' % (', '.join(a[1]), ', '.join(b[1]))
+                        diffsig_params = '-t %s %s -s %s %s' % (pn, taskname, a[0], b[0])
+                    else:
+                        pruned += 1
+
+        if msg:
+            msg.insert(0, 'The machines have conflicting signatures for some shared tasks:')
+            if pruned > 0:
+                msg.append('')
+                msg.append('%d tasks where not listed because some other task of the recipe already differed.' % pruned)
+                msg.append('It is likely that differences from different recipes also have the same root cause.')
+            msg.append('')
+            # Explain how to investigate...
+            msg.append('To investigate, run bitbake-diffsigs -t recipename taskname -s fromsig tosig.')
+            cmd = 'bitbake-diffsigs %s' % diffsig_params
+            msg.append('Example: %s in the last line' % diffsig_machines)
+            msg.append('Command: %s' % cmd)
+            # ... and actually do it automatically for that example, but without aborting
+            # when that fails.
+            try:
+                output = check_command('Comparing signatures failed.', cmd).decode('utf-8')
+            except RuntimeError as ex:
+                output = str(ex)
+            msg.extend(['   ' + line for line in output.splitlines()])
+            self.fail('\n'.join(msg))
diff --git a/scripts/yocto-compat-layer.py b/scripts/yocto-compat-layer.py
index daf3a6a..0d5700b 100755
--- a/scripts/yocto-compat-layer.py
+++ b/scripts/yocto-compat-layer.py
@@ -49,6 +49,8 @@ def main():
             help='File to output log (optional)', action='store')
     parser.add_argument('--dependency', nargs="+",
             help='Layers to process for dependencies', action='store')
+    parser.add_argument('--machines', nargs="+",
+            help='List of MACHINEs to be used during testing', action='store')
     parser.add_argument('--additional-layers', nargs="+",
             help='List of additional layers to add during testing', action='store')
     parser.add_argument('-n', '--no-auto', help='Disable auto layer discovery',
@@ -162,6 +164,7 @@ def main():
         logger.info('Getting initial signatures ...')
         td['builddir'] = builddir
         td['sigs'], td['tunetasks'] = get_signatures(td['builddir'])
+        td['machines'] = args.machines
 
         if not add_layer(bblayersconf, layer, dep_layers, logger):
             logger.info('Skipping %s ???.' % layer['name'])
-- 
git-series 0.9.1

[PATCH v3 8/8] yocto-compat-layer: better handling of per-machine world build breakage

[Patrick Ohly]

It is fairly common that BSP layers enable recipes when choosing
machines from that layer without checking whether the recipe actually
builds in the current distro. That breaks "bitbake world", retrieving
signatures and thus the test_machine_signatures test.

It's better to let that test continue with the signatures that can be
retrieved and report the broken world build separately. Right now, the
new test_machine_world iterates over all machines. More elegant and
useful in combination with a (currently missing) selection of which
tests to run would be generate one test instance per machine. But that
is not straightforward and has to wait.

The "-k" argument alone was not enough to proceed despite failures,
because bitbake then still returns a non-zero exit code. The existance
of the output file is taken as sign that the bitbake execution managed
was not fatally broken.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 scripts/lib/compatlayer/__init__.py  | 16 ++++++++++++++--
 scripts/lib/compatlayer/cases/bsp.py | 26 ++++++++++++++++++++++++++
 2 files changed, 40 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/compatlayer/__init__.py b/scripts/lib/compatlayer/__init__.py
index 0d6f4e9..e35f8c0 100644
--- a/scripts/lib/compatlayer/__init__.py
+++ b/scripts/lib/compatlayer/__init__.py
@@ -233,9 +233,21 @@ def get_signatures(builddir, failsafe=False, machine=None):
     if failsafe:
         cmd += '-k '
     cmd += '-S none world'
-    check_command('Generating signatures failed. This might be due to some parse error and/or general layer incompatibilities.',
-                  cmd)
     sigs_file = os.path.join(builddir, 'locked-sigs.inc')
+    if os.path.exists(sigs_file):
+        os.unlink(sigs_file)
+    try:
+        check_command('Generating signatures failed. This might be due to some parse error and/or general layer incompatibilities.',
+                      cmd)
+    except RuntimeError as ex:
+        if failsafe and os.path.exists(sigs_file):
+            # Ignore the error here. Most likely some recipes active
+            # in a world build lack some dependencies. There is a
+            # separate test_machine_world_build which exposes the
+            # failure.
+            pass
+        else:
+            raise
 
     sig_regex = re.compile("^(?P<task>.*:.*):(?P<hash>.*) .$")
     tune_regex = re.compile("(^|\s)SIGGEN_LOCKEDSIGS_t-(?P<tune>\S*)\s*=\s*")
diff --git a/scripts/lib/compatlayer/cases/bsp.py b/scripts/lib/compatlayer/cases/bsp.py
index b205568..d07c96e 100644
--- a/scripts/lib/compatlayer/cases/bsp.py
+++ b/scripts/lib/compatlayer/cases/bsp.py
@@ -25,6 +25,32 @@ class BSPCompatLayer(OECompatLayerTestCase):
                 msg="Layer %s modified machine %s -> %s" % \
                     (self.tc.layer['name'], self.td['bbvars']['MACHINE'], machine))
 
+
+    def test_machine_world(self):
+        '''
+        "bitbake world" is expected to work regardless which machine is selected.
+        BSP layers sometimes break that by enabling a recipe for a certain machine
+        without checking whether that recipe actually can be built in the current
+        distro configuration (for example, OpenGL might not enabled).
+
+        This test iterates over all machines. It would be nicer to instantiate
+        it once per machine. It merely checks for errors during parse
+        time. It does not actually attempt to build anything.
+        '''
+
+        if not self.td['machines']:
+            self.skipTest('No machines set with --machines.')
+        msg = []
+        for machine in self.td['machines']:
+            # In contrast to test_machine_signatures() below, errors are fatal here.
+            try:
+                get_signatures(self.td['builddir'], failsafe=False, machine=machine)
+            except RuntimeError as ex:
+                msg.append(str(ex))
+        if msg:
+            msg.insert(0, 'The following machines broke a world build:')
+            self.fail('\n'.join(msg))
+
     def test_machine_signatures(self):
         '''
         Selecting a machine may only affect the signature of tasks that are specific
-- 
git-series 0.9.1

Re: [PATCH v3 2/8] gdb-cross: avoid tune specific paths

[Mark Hatle]

On 4/11/17 9:56 AM, Patrick Ohly wrote:
> gdb-cross used to be specific to the tune flags, but isn't
> anymore. Therefore it is enough to use TARGET_SYS instead of
> TUNE_PKGARCH to create a unique path.

Are you sure about this.  On non-intel architectures, it used to be VERY common
that the specific instruction set for a process was programmed into gdb.

Sometimes the user could (after loading gdb) change the instructions, but this
was an aweful user experience.

Good example of this would be Power, e500, etc.  The instruction sets are not
actually compatible with each other, so GDB needs to know the specific
instruction in order to work.

--Mark

> Fixes a sstate signature difference that was found via
> yocto-compat-layer.py's test_machine_signatures check. In practice it
> probably showed up as unnecessarily rebuilding gdb-cross when
> switching between machines like intel-corei7-64 and qemux86-64.
> 
> Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
> ---
>  meta/recipes-devtools/gdb/gdb-cross.inc | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/recipes-devtools/gdb/gdb-cross.inc b/meta/recipes-devtools/gdb/gdb-cross.inc
> index a34223e..d92f31f 100644
> --- a/meta/recipes-devtools/gdb/gdb-cross.inc
> +++ b/meta/recipes-devtools/gdb/gdb-cross.inc
> @@ -24,4 +24,4 @@ BPN = "gdb"
>  inherit cross
>  inherit gettext
>  
> -datadir .= "/gdb-${TUNE_PKGARCH}${TARGET_VENDOR}-${TARGET_OS}"
> +datadir .= "/gdb-${TARGET_SYS}${TARGET_VENDOR}-${TARGET_OS}"
> 

Re: [PATCH v3 2/8] gdb-cross: avoid tune specific paths

[Patrick Ohly]

On Tue, 2017-04-11 at 10:16 -0500, Mark Hatle wrote:
> On 4/11/17 9:56 AM, Patrick Ohly wrote:
> > gdb-cross used to be specific to the tune flags, but isn't
> > anymore. Therefore it is enough to use TARGET_SYS instead of
> > TUNE_PKGARCH to create a unique path.
> 
> Are you sure about this.

It's what Richard told me and he proposed this fix (forgot to mention
that), so ultimately this is a question for him.

>   On non-intel architectures, it used to be VERY common
> that the specific instruction set for a process was programmed into gdb.

I'm not seeing anything in gdb-cross where that happens. The TUNEFLAGS
dependency was just for the datadir, not for anything else that might
configure the instruction set.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH v3 2/8] gdb-cross: avoid tune specific paths

[Khem Raj]

On Tue, Apr 11, 2017 at 8:16 AM, Mark Hatle <mark.hatle@windriver.com> wrote:
> On 4/11/17 9:56 AM, Patrick Ohly wrote:
>> gdb-cross used to be specific to the tune flags, but isn't
>> anymore. Therefore it is enough to use TARGET_SYS instead of
>> TUNE_PKGARCH to create a unique path.
>
> Are you sure about this.  On non-intel architectures, it used to be VERY common
> that the specific instruction set for a process was programmed into gdb.
>
> Sometimes the user could (after loading gdb) change the instructions, but this
> was an aweful user experience.
>
> Good example of this would be Power, e500, etc.  The instruction sets are not
> actually compatible with each other, so GDB needs to know the specific
> instruction in order to work.

if its mainly disassembly and opcodes if we compile bfd to support
them then gdb should be able to use it.  you can use something like

(gdb) set processor

to tune it to specific one

>
> --Mark
>
>> Fixes a sstate signature difference that was found via
>> yocto-compat-layer.py's test_machine_signatures check. In practice it
>> probably showed up as unnecessarily rebuilding gdb-cross when
>> switching between machines like intel-corei7-64 and qemux86-64.
>>
>> Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
>> ---
>>  meta/recipes-devtools/gdb/gdb-cross.inc | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/meta/recipes-devtools/gdb/gdb-cross.inc b/meta/recipes-devtools/gdb/gdb-cross.inc
>> index a34223e..d92f31f 100644
>> --- a/meta/recipes-devtools/gdb/gdb-cross.inc
>> +++ b/meta/recipes-devtools/gdb/gdb-cross.inc
>> @@ -24,4 +24,4 @@ BPN = "gdb"
>>  inherit cross
>>  inherit gettext
>>
>> -datadir .= "/gdb-${TUNE_PKGARCH}${TARGET_VENDOR}-${TARGET_OS}"
>> +datadir .= "/gdb-${TARGET_SYS}${TARGET_VENDOR}-${TARGET_OS}"
>>
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH v3 2/8] gdb-cross: avoid tune specific paths

[Richard Purdie]

On Tue, 2017-04-11 at 10:16 -0500, Mark Hatle wrote:
> On 4/11/17 9:56 AM, Patrick Ohly wrote:
> > 
> > gdb-cross used to be specific to the tune flags, but isn't
> > anymore. Therefore it is enough to use TARGET_SYS instead of
> > TUNE_PKGARCH to create a unique path.
> Are you sure about this.  On non-intel architectures, it used to be
> VERY common
> that the specific instruction set for a process was programmed into
> gdb.
> 
> Sometimes the user could (after loading gdb) change the instructions,
> but this
> was an aweful user experience.
> 
> Good example of this would be Power, e500, etc.  The instruction sets
> are not
> actually compatible with each other, so GDB needs to know the
> specific
> instruction in order to work.

This may or may not be an issue with gdb however I'd note that we only
build one gdb-cross regardless, we don't configure it with any target
tune specifics and the gdb we build for the sdk is built similarly. So
if it is an issue, its not one we've had reported to us and we don't
configure gdb in any target specific way so this patch should be
safe...

Cheers,

Richard

Re: [PATCH v3 2/8] gdb-cross: avoid tune specific paths

[Mark Hatle]

On 4/11/17 12:19 PM, Richard Purdie wrote:
> On Tue, 2017-04-11 at 10:16 -0500, Mark Hatle wrote:
>> On 4/11/17 9:56 AM, Patrick Ohly wrote:
>>>
>>> gdb-cross used to be specific to the tune flags, but isn't
>>> anymore. Therefore it is enough to use TARGET_SYS instead of
>>> TUNE_PKGARCH to create a unique path.
>> Are you sure about this.  On non-intel architectures, it used to be
>> VERY common
>> that the specific instruction set for a process was programmed into
>> gdb.
>>
>> Sometimes the user could (after loading gdb) change the instructions,
>> but this
>> was an aweful user experience.
>>
>> Good example of this would be Power, e500, etc.  The instruction sets
>> are not
>> actually compatible with each other, so GDB needs to know the
>> specific
>> instruction in order to work.
> 
> This may or may not be an issue with gdb however I'd note that we only
> build one gdb-cross regardless, we don't configure it with any target
> tune specifics and the gdb we build for the sdk is built similarly. So
> if it is an issue, its not one we've had reported to us and we don't
> configure gdb in any target specific way so this patch should be
> safe...

As long as the set processor is docuemnted (gdb docs, not OE/YP) and works.  I'm
fine with this explanation.

--Mark

> Cheers,
> 
> Richard
> 

Re: [PATCH v3 3/8] -cross recipes: ignore TARGET_ARCH sstate hash

[Richard Purdie]

I'd tweak the subject line to recipes/*-cross:

On Tue, 2017-04-11 at 16:56 +0200, Patrick Ohly wrote:
> "yocto-compat-layer.py --machines" showed that shared packages like
> gcc-cross-powerpc64 have a sstate signature that depends on
> TUNEFLAGS. As a result, there are unnecessary rebuilds and potential
> conflicts in a multiconfig.
> 
> That's due to the way how TARGET_ARCH is set. Richard Purdie
> suggested
> setting TARGET_ARCH[vardepvalue] as fix, which works. It would be
> shorter to do that in cross.inc instead of repeating the relevant
> line
> in different recipes, but Richard was concerned about potential
> side-effects in other usages of cross.inc.

s/cross.inc/cross.bbclass/g

Cheers,

Richard

Re: [PATCH v3 3/8] -cross recipes: ignore TARGET_ARCH sstate hash

[Patrick Ohly]

On Tue, 2017-04-11 at 19:25 +0100, Richard Purdie wrote:
> I'd tweak the subject line to recipes/*-cross:
> 
> On Tue, 2017-04-11 at 16:56 +0200, Patrick Ohly wrote:
> > "yocto-compat-layer.py --machines" showed that shared packages like
> > gcc-cross-powerpc64 have a sstate signature that depends on
> > TUNEFLAGS. As a result, there are unnecessary rebuilds and potential
> > conflicts in a multiconfig.
> > 
> > That's due to the way how TARGET_ARCH is set. Richard Purdie
> > suggested
> > setting TARGET_ARCH[vardepvalue] as fix, which works. It would be
> > shorter to do that in cross.inc instead of repeating the relevant
> > line
> > in different recipes, but Richard was concerned about potential
> > side-effects in other usages of cross.inc.
> 
> s/cross.inc/cross.bbclass/g

Okay, will do both as part of V4.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.