[PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Aníbal Limón]

The yocto-compat-layer script serves as a tool to validate the alignament
of a layer with YP Compatible Layers Programme [1], is based on an RFC
sent to the ML to enable automatic testing of layers [2] that wants to
be YP Compatible.

The tool takes an layer (or set of layers) via command line option -l
and detects what kind of layer is distro, machine or software and then
executes a  set of tests against the layer in order to validate the
compatibility.

The tests currently implemented are:

common.test_readme: Test if a README file exists in the layer and isn't
    empty.
common.test_parse: Test for execute bitbake -p without errors.
common.test_show_environment: Test for execute bitbake -e without errors.
common.test_signatures: Test executed in BSP and DISTRO layers to review
    doesn't comes with recipes that changes the signatures.

bsp.test_bsp_defines_machines: Test if a BSP layers has machines
    configurations.
bsp.test_bsp_no_set_machine: Test the BSP layer to doesn't set
    machine at adding layer.

distro.test_distro_defines_distros: Test if a DISTRO layers has distro
    configurations.
distro.test_distro_no_set_distro: Test the DISTRO layer to doesn't set
    distro at adding layer.

Example of usage:

$ source oe-init-build-env
$ yocto-compat-layer.py LAYER_DIR

[YOCTO #10596]

[1] https://www.yoctoproject.org/webform/yocto-project-compatible-registration
[2] https://lists.yoctoproject.org/pipermail/yocto-ab/2016-October/001801.html

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 scripts/lib/compatlayer/__init__.py       | 163 ++++++++++++++++++++++++++++++
 scripts/lib/compatlayer/case.py           |   7 ++
 scripts/lib/compatlayer/cases/__init__.py |   0
 scripts/lib/compatlayer/cases/bsp.py      |  26 +++++
 scripts/lib/compatlayer/cases/common.py   |  66 ++++++++++++
 scripts/lib/compatlayer/cases/distro.py   |  26 +++++
 scripts/lib/compatlayer/context.py        |  14 +++
 scripts/yocto-compat-layer.py             | 153 ++++++++++++++++++++++++++++
 8 files changed, 455 insertions(+)
 create mode 100644 scripts/lib/compatlayer/__init__.py
 create mode 100644 scripts/lib/compatlayer/case.py
 create mode 100644 scripts/lib/compatlayer/cases/__init__.py
 create mode 100644 scripts/lib/compatlayer/cases/bsp.py
 create mode 100644 scripts/lib/compatlayer/cases/common.py
 create mode 100644 scripts/lib/compatlayer/cases/distro.py
 create mode 100644 scripts/lib/compatlayer/context.py
 create mode 100755 scripts/yocto-compat-layer.py

diff --git a/scripts/lib/compatlayer/__init__.py b/scripts/lib/compatlayer/__init__.py
new file mode 100644
index 0000000..b3a166a
--- /dev/null
+++ b/scripts/lib/compatlayer/__init__.py
@@ -0,0 +1,163 @@
+# Yocto Project compatibility layer tool
+#
+# Copyright (C) 2017 Intel Corporation
+# Released under the MIT license (see COPYING.MIT)
+
+import os
+from enum import Enum
+
+class LayerType(Enum):
+    BSP = 0
+    DISTRO = 1
+    SOFTWARE = 2
+    ERROR_NO_LAYER_CONF = 98
+    ERROR_BSP_DISTRO = 99
+
+def _get_configurations(path):
+    configs = []
+
+    for f in os.listdir(path):
+        file_path = os.path.join(path, f)
+        if os.path.isfile(file_path) and f.endswith('.conf'):
+            configs.append(f[:-5]) # strip .conf
+    return configs
+
+def _get_layer_collections(layer_path, lconf=None, data=None):
+    import bb.parse
+    import bb.data
+
+    if lconf is None:
+        lconf = os.path.join(layer_path, 'conf', 'layer.conf')
+
+    if data is None:
+        ldata = bb.data.init()
+        bb.parse.init_parser(ldata)
+    else:
+        ldata = data.createCopy()
+
+    ldata.setVar('LAYERDIR', layer_path)
+    try:
+        ldata = bb.parse.handle(lconf, ldata, include=True)
+    except BaseException as exc:
+        raise LayerError(exc)
+    ldata.expandVarref('LAYERDIR')
+
+    collections = (ldata.getVar('BBFILE_COLLECTIONS', True) or '').split()
+    if not collections:
+        name = os.path.basename(layer_path)
+        collections = [name]
+
+    collections = {c: {} for c in collections}
+    for name in collections:
+        priority = ldata.getVar('BBFILE_PRIORITY_%s' % name, True)
+        pattern = ldata.getVar('BBFILE_PATTERN_%s' % name, True)
+        depends = ldata.getVar('LAYERDEPENDS_%s' % name, True)
+        collections[name]['priority'] = priority
+        collections[name]['pattern'] = pattern
+        collections[name]['depends'] = depends
+
+    return collections
+
+def _detect_layer(layer_path):
+    """
+        Scans layer directory to detect what type of layer
+        is BSP, Distro or Software.
+
+        Returns a dictionary with layer name, type and path.
+    """
+
+    layer = {}
+    layer_name = os.path.basename(layer_path)
+
+    layer['name'] = layer_name
+    layer['path'] = layer_path
+    layer['conf'] = {}
+
+    if not os.path.isfile(os.path.join(layer_path, 'conf', 'layer.conf')):
+        layer['type'] = LayerType.ERROR_NO_LAYER_CONF
+        return layer
+
+    machine_conf = os.path.join(layer_path, 'conf', 'machine')
+    distro_conf = os.path.join(layer_path, 'conf', 'distro')
+
+    is_bsp = False
+    is_distro = False
+
+    if os.path.isdir(machine_conf):
+        machines = _get_configurations(machine_conf)
+        if machines:
+            is_bsp = True
+
+    if os.path.isdir(distro_conf):
+        distros = _get_configurations(distro_conf)
+        if distros:
+            is_distro = True
+
+    if is_bsp and is_distro:
+        layer['type'] = LayerType.ERROR_BSP_DISTRO
+    elif is_bsp:
+        layer['type'] = LayerType.BSP
+        layer['conf']['machines'] = machines
+    elif is_distro:
+        layer['type'] = LayerType.DISTRO
+        layer['conf']['distros'] = distros
+    else:
+        layer['type'] = LayerType.SOFTWARE
+
+    layer['collections'] = _get_layer_collections(layer['path'])
+
+    return layer
+
+def detect_layers(layer_directories):
+    layers = []
+
+    for directory in layer_directories:
+        if directory[-1] == '/':
+            directory = directory[0:-1]
+
+        for root, dirs, files in os.walk(directory):
+            dir_name = os.path.basename(root)
+            conf_dir = os.path.join(root, 'conf')
+            if dir_name.startswith('meta-') and os.path.isdir(conf_dir):
+                layer = _detect_layer(root)
+                if layer:
+                    layers.append(layer)
+
+    return layers
+
+def add_layer(bblayersconf, layer):
+    with open(bblayersconf, 'a+') as f:
+        f.write("\nBBLAYERS += \"%s\"\n" % layer['path'])
+
+def get_signatures(builddir, failsafe=False):
+    import subprocess
+    import re
+
+    sigs = {}
+
+    try:
+        cmd = 'bitbake '
+        if failsafe:
+            cmd += '-k '
+        cmd += '-S none world'
+        output = subprocess.check_output(cmd, shell=True,
+                stderr=subprocess.PIPE)
+    except subprocess.CalledProcessError as e:
+        import traceback
+        exc = traceback.format_exc()
+        msg = '%s\n%s\n' % (exc, e.output.decode('utf-8'))
+        raise RuntimeError(msg)
+    sigs_file = os.path.join(builddir, 'locked-sigs.inc')
+
+    sig_regex = re.compile("^(?P<task>.*:.*):(?P<hash>.*) .$")
+    with open(sigs_file, 'r') as f:
+        for line in f.readlines():
+            line = line.strip()
+            s = sig_regex.match(line)
+            if s:
+                sigs[s.group('task')] = s.group('hash')
+
+    if not sigs:
+        raise RuntimeError('Can\'t load signatures from %s' % sigs_file)
+
+    return sigs
diff --git a/scripts/lib/compatlayer/case.py b/scripts/lib/compatlayer/case.py
new file mode 100644
index 0000000..54ce78a
--- /dev/null
+++ b/scripts/lib/compatlayer/case.py
@@ -0,0 +1,7 @@
+# Copyright (C) 2017 Intel Corporation
+# Released under the MIT license (see COPYING.MIT)
+
+from oeqa.core.case import OETestCase
+
+class OECompatLayerTestCase(OETestCase):
+    pass
diff --git a/scripts/lib/compatlayer/cases/__init__.py b/scripts/lib/compatlayer/cases/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/scripts/lib/compatlayer/cases/bsp.py b/scripts/lib/compatlayer/cases/bsp.py
new file mode 100644
index 0000000..5d9bf93
--- /dev/null
+++ b/scripts/lib/compatlayer/cases/bsp.py
@@ -0,0 +1,26 @@
+# Copyright (C) 2017 Intel Corporation
+# Released under the MIT license (see COPYING.MIT)
+
+import unittest
+
+from compatlayer import LayerType
+from compatlayer.case import OECompatLayerTestCase
+
+class BSPCompatLayer(OECompatLayerTestCase):
+    @classmethod
+    def setUpClass(self):
+        if self.tc.layer['type'] != LayerType.BSP:
+            raise unittest.SkipTest("BSPCompatLayer: Layer %s isn't BSP one." %\
+                self.tc.layer['name'])
+
+    def test_bsp_defines_machines(self):
+        self.assertTrue(self.tc.layer['conf']['machines'], 
+                "Layer is BSP but doesn't defines machines.")
+
+    def test_bsp_no_set_machine(self):
+        from oeqa.utils.commands import get_bb_var
+
+        machine = get_bb_var('MACHINE')
+        self.assertEqual(self.td['bbvars']['MACHINE'], machine,
+                msg="Layer %s modified machine %s -> %s" % \
+                    (self.tc.layer['name'], self.td['bbvars']['MACHINE'], machine))
diff --git a/scripts/lib/compatlayer/cases/common.py b/scripts/lib/compatlayer/cases/common.py
new file mode 100644
index 0000000..4d328ec
--- /dev/null
+++ b/scripts/lib/compatlayer/cases/common.py
@@ -0,0 +1,66 @@
+# Copyright (C) 2017 Intel Corporation
+# Released under the MIT license (see COPYING.MIT)
+
+import os
+import subprocess
+import unittest
+from compatlayer import get_signatures, LayerType
+from compatlayer.case import OECompatLayerTestCase
+
+class CommonCompatLayer(OECompatLayerTestCase):
+    def test_readme(self):
+        readme_file = os.path.join(self.tc.layer['path'], 'README')
+        self.assertTrue(os.path.isfile(readme_file),
+                msg="Layer doesn't contains README file.")
+
+        data = ''
+        with open(readme_file, 'r') as f:
+            data = f.read()
+        self.assertTrue(data,
+                msg="Layer contains README file but is empty.")
+
+    def test_parse(self):
+        try:
+            output = subprocess.check_output('bitbake -p', shell=True,
+                    stderr=subprocess.PIPE)
+        except subprocess.CalledProcessError as e:
+            import traceback
+            exc = traceback.format_exc()
+            msg = 'Layer %s failed to parse.\n%s\n%s\n' % (self.tc.layer['name'],
+                    exc, e.output.decode('utf-8'))
+            raise RuntimeError(msg)
+
+    def test_show_environment(self):
+        try:
+            output = subprocess.check_output('bitbake -e', shell=True,
+                    stderr=subprocess.PIPE)
+        except subprocess.CalledProcessError as e:
+            import traceback
+            exc = traceback.format_exc()
+            msg = 'Layer %s failed to show environment.\n%s\n%s\n' % \
+                    (self.tc.layer['name'], exc, e.output.decode('utf-8'))
+            raise RuntimeError(msg)
+
+    def test_signatures(self):
+        if self.tc.layer['type'] == LayerType.SOFTWARE:
+            raise unittest.SkipTest("Layer %s isn't BSP or DISTRO one." \
+                     % self.tc.layer['name'])
+
+        sig_diff = {}
+
+        curr_sigs = get_signatures(self.td['builddir'], failsafe=True)
+        for task in self.td['sigs']:
+            if task not in curr_sigs:
+                continue
+
+            if self.td['sigs'][task] != curr_sigs[task]:
+                sig_diff[task] = '%s -> %s' % \
+                        (self.td['sigs'][task], curr_sigs[task])
+
+        detail = ''
+        if sig_diff:
+            for task in sig_diff:
+                detail += "%s changed %s\n" % (task, sig_diff[task])
+        self.assertFalse(bool(sig_diff), "Layer %s changed signatures.\n%s" % \
+                (self.tc.layer['name'], detail))
+
diff --git a/scripts/lib/compatlayer/cases/distro.py b/scripts/lib/compatlayer/cases/distro.py
new file mode 100644
index 0000000..523acc1
--- /dev/null
+++ b/scripts/lib/compatlayer/cases/distro.py
@@ -0,0 +1,26 @@
+# Copyright (C) 2017 Intel Corporation
+# Released under the MIT license (see COPYING.MIT)
+
+import unittest
+
+from compatlayer import LayerType
+from compatlayer.case import OECompatLayerTestCase
+
+class DistroCompatLayer(OECompatLayerTestCase):
+    @classmethod
+    def setUpClass(self):
+        if self.tc.layer['type'] != LayerType.DISTRO:
+            raise unittest.SkipTest("DistroCompatLayer: Layer %s isn't Distro one." %\
+                self.tc.layer['name'])
+
+    def test_distro_defines_distros(self):
+        self.assertTrue(self.tc.layer['conf']['distros'], 
+                "Layer is BSP but doesn't defines machines.")
+
+    def test_distro_no_set_distros(self):
+        from oeqa.utils.commands import get_bb_var
+
+        distro = get_bb_var('DISTRO')
+        self.assertEqual(self.td['bbvars']['DISTRO'], distro,
+                msg="Layer %s modified distro %s -> %s" % \
+                    (self.tc.layer['name'], self.td['bbvars']['DISTRO'], distro))
diff --git a/scripts/lib/compatlayer/context.py b/scripts/lib/compatlayer/context.py
new file mode 100644
index 0000000..4932238
--- /dev/null
+++ b/scripts/lib/compatlayer/context.py
@@ -0,0 +1,14 @@
+# Copyright (C) 2017 Intel Corporation
+# Released under the MIT license (see COPYING.MIT)
+
+import os
+import sys
+import glob
+import re
+
+from oeqa.core.context import OETestContext
+
+class CompatLayerTestContext(OETestContext):
+    def __init__(self, td=None, logger=None, layer=None):
+        super(CompatLayerTestContext, self).__init__(td, logger)
+        self.layer = layer
diff --git a/scripts/yocto-compat-layer.py b/scripts/yocto-compat-layer.py
new file mode 100755
index 0000000..09dc5bf
--- /dev/null
+++ b/scripts/yocto-compat-layer.py
@@ -0,0 +1,153 @@
+#!/usr/bin/env python3
+
+# Yocto Project compatibility layer tool
+#
+# Copyright (C) 2017 Intel Corporation
+# Released under the MIT license (see COPYING.MIT)
+
+import os
+import sys
+import argparse
+import logging
+import time
+import signal
+import shutil
+import collections
+
+scripts_path = os.path.dirname(os.path.realpath(__file__))
+lib_path = scripts_path + '/lib'
+sys.path = sys.path + [lib_path]
+import scriptutils
+import scriptpath
+scriptpath.add_oe_lib_path()
+scriptpath.add_bitbake_lib_path()
+
+from compatlayer import LayerType, detect_layers, add_layer, get_signatures
+from oeqa.utils.commands import get_bb_vars
+
+PROGNAME = 'yocto-compat-layer'
+DEFAULT_OUTPUT_LOG = '%s-%s.log' % (PROGNAME,
+        time.strftime("%Y%m%d%H%M%S"))
+OUTPUT_LOG_LINK = "%s.log" % PROGNAME
+CASES_PATHS = [os.path.join(os.path.abspath(os.path.dirname(__file__)),
+                'lib', 'compatlayer', 'cases')]
+logger = scriptutils.logger_create(PROGNAME)
+
+def test_layer_compatibility(td, layer):
+    from compatlayer.context import CompatLayerTestContext
+    logger.info("Starting to analyze: %s" % layer['name'])
+    logger.info("----------------------------------------------------------------------")
+
+    tc = CompatLayerTestContext(td=td, logger=logger, layer=layer)
+    tc.loadTests(CASES_PATHS)
+    return tc.runTests()
+
+def main():
+    parser = argparse.ArgumentParser(
+            description="Yocto Project compatibility layer tool",
+            add_help=False)
+    parser.add_argument('layers', metavar='LAYER_DIR', nargs='+',
+            help='Layer to test compatibility with Yocto Project')
+    parser.add_argument('-o', '--output-log',
+            help='Output log default: %s' % DEFAULT_OUTPUT_LOG,
+            action='store', default=DEFAULT_OUTPUT_LOG)
+
+    parser.add_argument('-d', '--debug', help='Enable debug output',
+            action='store_true')
+    parser.add_argument('-q', '--quiet', help='Print only errors',
+            action='store_true')
+
+    parser.add_argument('-h', '--help', action='help',
+            default=argparse.SUPPRESS,
+            help='show this help message and exit')
+
+    args = parser.parse_args()
+
+    fh = logging.FileHandler(args.output_log)
+    fh.setFormatter(logging.Formatter("%(levelname)s: %(message)s"))
+    logger.addHandler(fh)
+    if args.debug:
+        logger.setLevel(logging.DEBUG)
+    elif args.quiet:
+        logger.setLevel(logging.ERROR)
+    if os.path.exists(OUTPUT_LOG_LINK):
+        os.unlink(OUTPUT_LOG_LINK)
+    os.symlink(args.output_log, OUTPUT_LOG_LINK)
+
+    if not 'BUILDDIR' in os.environ:
+        logger.error("You must source the environment before run this script.")
+        logger.error("$ source oe-init-build-env")
+        return 1
+    builddir = os.environ['BUILDDIR']
+    bblayersconf = os.path.join(builddir, 'conf', 'bblayers.conf')
+
+    layers = detect_layers(args.layers)
+    if not layers:
+        logger.error("Fail to detect layers")
+        return 1
+
+    logger.info("Detected layers:")
+    for layer in layers:
+        if layer['type'] == LayerType.ERROR_BSP_DISTRO:
+            logger.error("%s: Can't be DISTRO and BSP type at the same time."\
+                     " The conf/distro and conf/machine folders was found."\
+                     % layer['name'])
+            layers.remove(layer)
+        elif layer['type'] == LayerType.ERROR_NO_LAYER_CONF:
+            logger.error("%s: Don't have conf/layer.conf file."\
+                     % layer['name'])
+            layers.remove(layer)
+        else:
+            logger.info("%s: %s, %s" % (layer['name'], layer['type'],
+                layer['path']))
+    if not layers:
+        return 1
+
+    shutil.copyfile(bblayersconf, bblayersconf + '.backup')
+    def cleanup_bblayers(signum, frame):
+        shutil.copyfile(bblayersconf + '.backup', bblayersconf)
+        os.unlink(bblayersconf + '.backup')
+    signal.signal(signal.SIGTERM, cleanup_bblayers)
+    signal.signal(signal.SIGINT, cleanup_bblayers)
+
+    td = {}
+    results = collections.OrderedDict()
+
+    logger.info('')
+    logger.info('Getting initial bitbake variables ...')
+    td['bbvars'] = get_bb_vars()
+    logger.info('Getting initial signatures ...')
+    td['builddir'] = builddir
+    td['sigs'] = get_signatures(td['builddir'])
+    logger.info('')
+
+    for layer in layers:
+        if layer['type'] == LayerType.ERROR_NO_LAYER_CONF or \
+                layer['type'] == LayerType.ERROR_BSP_DISTRO:
+            continue
+
+        shutil.copyfile(bblayersconf + '.backup', bblayersconf)
+
+        add_layer(bblayersconf, layer)
+        result = test_layer_compatibility(td, layer)
+        results[layer['name']] = result
+
+    logger.info('')
+    logger.info('Summary of results:')
+    logger.info('')
+    for layer_name in results:
+        logger.info('%s ... %s' % (layer_name, 'PASS' if \
+                results[layer_name].wasSuccessful() else 'FAIL'))
+
+    cleanup_bblayers(None, None)
+
+    return 0
+
+if __name__ == '__main__':
+    try:
+        ret =  main()
+    except Exception:
+        ret = 1
+        import traceback
+        traceback.print_exc()
+    sys.exit(ret)
-- 
2.1.4

Re: [OE-core] [PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Patrick Ohly]

On Mon, 2017-02-20 at 15:12 -0600, Aníbal Limón wrote:
> common.test_signatures: Test executed in BSP and DISTRO layers to review
>     doesn't comes with recipes that changes the signatures.

I have a question about the goal for this test: is it meant to detect
layers which incorrectly change the signatures of allarch recipes or
recipes which share the same tune flags with other machines?

Let's take MACHINE=edison as an example.

Modifying allarch creates a conflict with basically all other machines
in a distro. Example for something not allowed:
VOLATILE_BINDS_append_pn-volatile-binds_edison = " /var/volatile/foo /var/foo \n"

This can be detected by comparing against OE-core, but only when testing
with MACHINE=edison.

More difficult to detect is modifying recipes with the same tune flags,
which is the majority of the recipes. MACHINE=edison and
MACHINE=intel-core2-32 both compile for the same target architecture, so
something like this is incorrect:
do_install_append_pn-base-files_edison () {
    echo "Built for Edison" >>${D}${sysconfdir}/motd
}

This can only be detected when testing with both MACHINE=edison and
MACHINE=intel-core2-32 - at least I think MACHINE=qemux86 uses different
tune flags (haven't checked).

My point is, the test probably needs to be extended to run with a set of
machines, and that set of machines must be broad enough to cover a
variety of common tune flags.

The corresponding selftest, test_sstate_sametune_samesigs in
sstatetests.py, has the same limitation of its scope, i.e. doesn't
actually test with real machine definitions.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Patrick Ohly]

On Mon, 2017-02-20 at 15:12 -0600, Aníbal Limón wrote:
> common.test_signatures: Test executed in BSP and DISTRO layers to review
>     doesn't comes with recipes that changes the signatures.

I have a question about the goal for this test: is it meant to detect
layers which incorrectly change the signatures of allarch recipes or
recipes which share the same tune flags with other machines?

Let's take MACHINE=edison as an example.

Modifying allarch creates a conflict with basically all other machines
in a distro. Example for something not allowed:
VOLATILE_BINDS_append_pn-volatile-binds_edison = " /var/volatile/foo /var/foo \n"

This can be detected by comparing against OE-core, but only when testing
with MACHINE=edison.

More difficult to detect is modifying recipes with the same tune flags,
which is the majority of the recipes. MACHINE=edison and
MACHINE=intel-core2-32 both compile for the same target architecture, so
something like this is incorrect:
do_install_append_pn-base-files_edison () {
    echo "Built for Edison" >>${D}${sysconfdir}/motd
}

This can only be detected when testing with both MACHINE=edison and
MACHINE=intel-core2-32 - at least I think MACHINE=qemux86 uses different
tune flags (haven't checked).

My point is, the test probably needs to be extended to run with a set of
machines, and that set of machines must be broad enough to cover a
variety of common tune flags.

The corresponding selftest, test_sstate_sametune_samesigs in
sstatetests.py, has the same limitation of its scope, i.e. doesn't
actually test with real machine definitions.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [OE-core] [PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Aníbal Limón]

[-- Attachment #1: Type: text/plain, Size: 2089 bytes --]



On 02/28/2017 02:09 PM, Patrick Ohly wrote:
> On Mon, 2017-02-20 at 15:12 -0600, Aníbal Limón wrote:
>> common.test_signatures: Test executed in BSP and DISTRO layers to review
>>     doesn't comes with recipes that changes the signatures.
> 
> I have a question about the goal for this test: is it meant to detect
> layers which incorrectly change the signatures of allarch recipes or
> recipes which share the same tune flags with other machines?

The requirement is DISTRO and BSP layers aren't allowed to automatically
change the MACHINE or DISTRO variable that causes the signatures to change.

> 
> Let's take MACHINE=edison as an example.
> 
> Modifying allarch creates a conflict with basically all other machines
> in a distro. Example for something not allowed:
> VOLATILE_BINDS_append_pn-volatile-binds_edison = " /var/volatile/foo /var/foo \n"
> 
> This can be detected by comparing against OE-core, but only when testing
> with MACHINE=edison.
> 
> More difficult to detect is modifying recipes with the same tune flags,
> which is the majority of the recipes. MACHINE=edison and
> MACHINE=intel-core2-32 both compile for the same target architecture, so
> something like this is incorrect:
> do_install_append_pn-base-files_edison () {
>     echo "Built for Edison" >>${D}${sysconfdir}/motd
> }
> 
> This can only be detected when testing with both MACHINE=edison and
> MACHINE=intel-core2-32 - at least I think MACHINE=qemux86 uses different
> tune flags (haven't checked).
> 
> My point is, the test probably needs to be extended to run with a set of
> machines, and that set of machines must be broad enough to cover a
> variety of common tune flags.

It is expected to change recipe signatures when the machine change, this
leaves me other question. what signatures are expected to change when
set a different MACHINE?

	Anibal

> 
> The corresponding selftest, test_sstate_sametune_samesigs in
> sstatetests.py, has the same limitation of its scope, i.e. doesn't
> actually test with real machine definitions.
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Aníbal Limón]

[-- Attachment #1: Type: text/plain, Size: 2089 bytes --]



On 02/28/2017 02:09 PM, Patrick Ohly wrote:
> On Mon, 2017-02-20 at 15:12 -0600, Aníbal Limón wrote:
>> common.test_signatures: Test executed in BSP and DISTRO layers to review
>>     doesn't comes with recipes that changes the signatures.
> 
> I have a question about the goal for this test: is it meant to detect
> layers which incorrectly change the signatures of allarch recipes or
> recipes which share the same tune flags with other machines?

The requirement is DISTRO and BSP layers aren't allowed to automatically
change the MACHINE or DISTRO variable that causes the signatures to change.

> 
> Let's take MACHINE=edison as an example.
> 
> Modifying allarch creates a conflict with basically all other machines
> in a distro. Example for something not allowed:
> VOLATILE_BINDS_append_pn-volatile-binds_edison = " /var/volatile/foo /var/foo \n"
> 
> This can be detected by comparing against OE-core, but only when testing
> with MACHINE=edison.
> 
> More difficult to detect is modifying recipes with the same tune flags,
> which is the majority of the recipes. MACHINE=edison and
> MACHINE=intel-core2-32 both compile for the same target architecture, so
> something like this is incorrect:
> do_install_append_pn-base-files_edison () {
>     echo "Built for Edison" >>${D}${sysconfdir}/motd
> }
> 
> This can only be detected when testing with both MACHINE=edison and
> MACHINE=intel-core2-32 - at least I think MACHINE=qemux86 uses different
> tune flags (haven't checked).
> 
> My point is, the test probably needs to be extended to run with a set of
> machines, and that set of machines must be broad enough to cover a
> variety of common tune flags.

It is expected to change recipe signatures when the machine change, this
leaves me other question. what signatures are expected to change when
set a different MACHINE?

	Anibal

> 
> The corresponding selftest, test_sstate_sametune_samesigs in
> sstatetests.py, has the same limitation of its scope, i.e. doesn't
> actually test with real machine definitions.
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [OE-core] [PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Patrick Ohly]

On Tue, 2017-02-28 at 14:33 -0600, Aníbal Limón wrote:
> 
> On 02/28/2017 02:09 PM, Patrick Ohly wrote:
> > On Mon, 2017-02-20 at 15:12 -0600, Aníbal Limón wrote:
> >> common.test_signatures: Test executed in BSP and DISTRO layers to review
> >>     doesn't comes with recipes that changes the signatures.
> > 
> > I have a question about the goal for this test: is it meant to detect
> > layers which incorrectly change the signatures of allarch recipes or
> > recipes which share the same tune flags with other machines?
> 
> The requirement is DISTRO and BSP layers aren't allowed to automatically
> change the MACHINE or DISTRO variable that causes the signatures to change.

I do not fully understand this explanation. Can you give an example for
the kind of misbehavior what the test is meant to catch?

> > Let's take MACHINE=edison as an example.
> > 
> > Modifying allarch creates a conflict with basically all other machines
> > in a distro. Example for something not allowed:
> > VOLATILE_BINDS_append_pn-volatile-binds_edison = " /var/volatile/foo /var/foo \n"
> > 
> > This can be detected by comparing against OE-core, but only when testing
> > with MACHINE=edison.
> > 
> > More difficult to detect is modifying recipes with the same tune flags,
> > which is the majority of the recipes. MACHINE=edison and
> > MACHINE=intel-core2-32 both compile for the same target architecture, so
> > something like this is incorrect:
> > do_install_append_pn-base-files_edison () {
> >     echo "Built for Edison" >>${D}${sysconfdir}/motd
> > }
> > 
> > This can only be detected when testing with both MACHINE=edison and
> > MACHINE=intel-core2-32 - at least I think MACHINE=qemux86 uses different
> > tune flags (haven't checked).
> > 
> > My point is, the test probably needs to be extended to run with a set of
> > machines, and that set of machines must be broad enough to cover a
> > variety of common tune flags.
> 
> It is expected to change recipe signatures when the machine change, this
> leaves me other question. what signatures are expected to change when
> set a different MACHINE?

Everything that is machine-specific may change, for example image
recipes and kernel (although even that is a slight simplification, as
meta-intel intentionally shares kernels between different MACHINE
configs).

The real-world test is: can two BSP layers be combined in a single
distro so that one can build first for one MACHINE, then the other (or,
when using multiconfig, even in a single build)?  If any shared package
changes as result of changing the MACHINE, then that won't work.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Patrick Ohly]

On Tue, 2017-02-28 at 14:33 -0600, Aníbal Limón wrote:
> 
> On 02/28/2017 02:09 PM, Patrick Ohly wrote:
> > On Mon, 2017-02-20 at 15:12 -0600, Aníbal Limón wrote:
> >> common.test_signatures: Test executed in BSP and DISTRO layers to review
> >>     doesn't comes with recipes that changes the signatures.
> > 
> > I have a question about the goal for this test: is it meant to detect
> > layers which incorrectly change the signatures of allarch recipes or
> > recipes which share the same tune flags with other machines?
> 
> The requirement is DISTRO and BSP layers aren't allowed to automatically
> change the MACHINE or DISTRO variable that causes the signatures to change.

I do not fully understand this explanation. Can you give an example for
the kind of misbehavior what the test is meant to catch?

> > Let's take MACHINE=edison as an example.
> > 
> > Modifying allarch creates a conflict with basically all other machines
> > in a distro. Example for something not allowed:
> > VOLATILE_BINDS_append_pn-volatile-binds_edison = " /var/volatile/foo /var/foo \n"
> > 
> > This can be detected by comparing against OE-core, but only when testing
> > with MACHINE=edison.
> > 
> > More difficult to detect is modifying recipes with the same tune flags,
> > which is the majority of the recipes. MACHINE=edison and
> > MACHINE=intel-core2-32 both compile for the same target architecture, so
> > something like this is incorrect:
> > do_install_append_pn-base-files_edison () {
> >     echo "Built for Edison" >>${D}${sysconfdir}/motd
> > }
> > 
> > This can only be detected when testing with both MACHINE=edison and
> > MACHINE=intel-core2-32 - at least I think MACHINE=qemux86 uses different
> > tune flags (haven't checked).
> > 
> > My point is, the test probably needs to be extended to run with a set of
> > machines, and that set of machines must be broad enough to cover a
> > variety of common tune flags.
> 
> It is expected to change recipe signatures when the machine change, this
> leaves me other question. what signatures are expected to change when
> set a different MACHINE?

Everything that is machine-specific may change, for example image
recipes and kernel (although even that is a slight simplification, as
meta-intel intentionally shares kernels between different MACHINE
configs).

The real-world test is: can two BSP layers be combined in a single
distro so that one can build first for one MACHINE, then the other (or,
when using multiconfig, even in a single build)?  If any shared package
changes as result of changing the MACHINE, then that won't work.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [OE-core] [PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Richard Purdie]

On Tue, 2017-02-28 at 21:09 +0100, Patrick Ohly wrote:
> On Mon, 2017-02-20 at 15:12 -0600, Aníbal Limón wrote:
> > 
> > common.test_signatures: Test executed in BSP and DISTRO layers to
> > review
> >     doesn't comes with recipes that changes the signatures.
> I have a question about the goal for this test: is it meant to detect
> layers which incorrectly change the signatures of allarch recipes or
> recipes which share the same tune flags with other machines?
> 
> Let's take MACHINE=edison as an example.

The test is not for these things. Its using the sstate signatures for
something different compared to those other tests.

The idea is that if you have a set of layers and generate the
signatures for world, then you add say a BSP layer but do not select
that MACHINE, the signatures should remain unchanged.

Cheers,

Richard

Re: [PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Richard Purdie]

On Tue, 2017-02-28 at 21:09 +0100, Patrick Ohly wrote:
> On Mon, 2017-02-20 at 15:12 -0600, Aníbal Limón wrote:
> > 
> > common.test_signatures: Test executed in BSP and DISTRO layers to
> > review
> >     doesn't comes with recipes that changes the signatures.
> I have a question about the goal for this test: is it meant to detect
> layers which incorrectly change the signatures of allarch recipes or
> recipes which share the same tune flags with other machines?
> 
> Let's take MACHINE=edison as an example.

The test is not for these things. Its using the sstate signatures for
something different compared to those other tests.

The idea is that if you have a set of layers and generate the
signatures for world, then you add say a BSP layer but do not select
that MACHINE, the signatures should remain unchanged.

Cheers,

Richard

Re: [OE-core] [PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Patrick Ohly]

On Wed, 2017-03-01 at 04:00 +0000, Richard Purdie wrote:
> On Tue, 2017-02-28 at 21:09 +0100, Patrick Ohly wrote:
> > On Mon, 2017-02-20 at 15:12 -0600, Aníbal Limón wrote:
> > > 
> > > common.test_signatures: Test executed in BSP and DISTRO layers to
> > > review
> > >     doesn't comes with recipes that changes the signatures.
> > I have a question about the goal for this test: is it meant to detect
> > layers which incorrectly change the signatures of allarch recipes or
> > recipes which share the same tune flags with other machines?
> > 
> > Let's take MACHINE=edison as an example.
> 
> The test is not for these things. Its using the sstate signatures for
> something different compared to those other tests.
> 
> The idea is that if you have a set of layers and generate the
> signatures for world, then you add say a BSP layer but do not select
> that MACHINE, the signatures should remain unchanged.

That's useful too, of course.

Is the "build single distro for different machines" scenario that I
described part of the Yocto Compliance 2.0? Should there be tests for
it?

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Patrick Ohly]

On Wed, 2017-03-01 at 04:00 +0000, Richard Purdie wrote:
> On Tue, 2017-02-28 at 21:09 +0100, Patrick Ohly wrote:
> > On Mon, 2017-02-20 at 15:12 -0600, Aníbal Limón wrote:
> > > 
> > > common.test_signatures: Test executed in BSP and DISTRO layers to
> > > review
> > >     doesn't comes with recipes that changes the signatures.
> > I have a question about the goal for this test: is it meant to detect
> > layers which incorrectly change the signatures of allarch recipes or
> > recipes which share the same tune flags with other machines?
> > 
> > Let's take MACHINE=edison as an example.
> 
> The test is not for these things. Its using the sstate signatures for
> something different compared to those other tests.
> 
> The idea is that if you have a set of layers and generate the
> signatures for world, then you add say a BSP layer but do not select
> that MACHINE, the signatures should remain unchanged.

That's useful too, of course.

Is the "build single distro for different machines" scenario that I
described part of the Yocto Compliance 2.0? Should there be tests for
it?

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [OE-core] [PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Richard Purdie]

On Wed, 2017-03-01 at 08:10 +0100, Patrick Ohly wrote:
> On Wed, 2017-03-01 at 04:00 +0000, Richard Purdie wrote:
> > 
> > On Tue, 2017-02-28 at 21:09 +0100, Patrick Ohly wrote:
> > > 
> > > On Mon, 2017-02-20 at 15:12 -0600, Aníbal Limón wrote:
> > > > 
> > > > 
> > > > common.test_signatures: Test executed in BSP and DISTRO layers
> > > > to
> > > > review
> > > >     doesn't comes with recipes that changes the signatures.
> > > I have a question about the goal for this test: is it meant to
> > > detect
> > > layers which incorrectly change the signatures of allarch recipes
> > > or
> > > recipes which share the same tune flags with other machines?
> > > 
> > > Let's take MACHINE=edison as an example.
> > The test is not for these things. Its using the sstate signatures
> > for
> > something different compared to those other tests.
> > 
> > The idea is that if you have a set of layers and generate the
> > signatures for world, then you add say a BSP layer but do not
> > select
> > that MACHINE, the signatures should remain unchanged.
> That's useful too, of course.
> 
> Is the "build single distro for different machines" scenario that I
> described part of the Yocto Compliance 2.0? Should there be tests for
> it?

Right now its not but I'd consider it. The question is can we write an
easy generic test for it, and also clearly phrase the criteria in the
list of compliance questions with a binary yes/no answer?

Cheers,

Richard

Re: [PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Richard Purdie]

On Wed, 2017-03-01 at 08:10 +0100, Patrick Ohly wrote:
> On Wed, 2017-03-01 at 04:00 +0000, Richard Purdie wrote:
> > 
> > On Tue, 2017-02-28 at 21:09 +0100, Patrick Ohly wrote:
> > > 
> > > On Mon, 2017-02-20 at 15:12 -0600, Aníbal Limón wrote:
> > > > 
> > > > 
> > > > common.test_signatures: Test executed in BSP and DISTRO layers
> > > > to
> > > > review
> > > >     doesn't comes with recipes that changes the signatures.
> > > I have a question about the goal for this test: is it meant to
> > > detect
> > > layers which incorrectly change the signatures of allarch recipes
> > > or
> > > recipes which share the same tune flags with other machines?
> > > 
> > > Let's take MACHINE=edison as an example.
> > The test is not for these things. Its using the sstate signatures
> > for
> > something different compared to those other tests.
> > 
> > The idea is that if you have a set of layers and generate the
> > signatures for world, then you add say a BSP layer but do not
> > select
> > that MACHINE, the signatures should remain unchanged.
> That's useful too, of course.
> 
> Is the "build single distro for different machines" scenario that I
> described part of the Yocto Compliance 2.0? Should there be tests for
> it?

Right now its not but I'd consider it. The question is can we write an
easy generic test for it, and also clearly phrase the criteria in the
list of compliance questions with a binary yes/no answer?

Cheers,

Richard

Re: [OE-core] [PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Patrick Ohly]

On Wed, 2017-03-01 at 15:12 +0000, Richard Purdie wrote:
> On Wed, 2017-03-01 at 08:10 +0100, Patrick Ohly wrote:
> > Is the "build single distro for different machines" scenario that I
> > described part of the Yocto Compliance 2.0? Should there be tests for
> > it?
> 
> Right now its not

Okay, so the goal is a bit less ambitious than I had thought. I wonder
whether that's useful, because at least the problems Ostro and AGL (at
least as far as I understood it from lurking on their mailing list) had
only happened when trying to combine multiple BSP layers *and* actually
using the different machines in the same distro.

> but I'd consider it.

At least I'd find that useful - not sure about others ;-}

>  The question is can we write an
> easy generic test for it,

It's a bit more complicated than the existing tests, but I think it is
doable.

> and also clearly phrase the criteria in the
> list of compliance questions with a binary yes/no answer?

Does the BSP layer only modify machine-specific packages and only when
the MACHINE(s) defined by the BSP layer are selected? [yes/no]

The "only when" part is covered by the existing tests (because they keep
MACHINE constant). The missing part is comparing different MACHINE
sstamps.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Patrick Ohly]

On Wed, 2017-03-01 at 15:12 +0000, Richard Purdie wrote:
> On Wed, 2017-03-01 at 08:10 +0100, Patrick Ohly wrote:
> > Is the "build single distro for different machines" scenario that I
> > described part of the Yocto Compliance 2.0? Should there be tests for
> > it?
> 
> Right now its not

Okay, so the goal is a bit less ambitious than I had thought. I wonder
whether that's useful, because at least the problems Ostro and AGL (at
least as far as I understood it from lurking on their mailing list) had
only happened when trying to combine multiple BSP layers *and* actually
using the different machines in the same distro.

> but I'd consider it.

At least I'd find that useful - not sure about others ;-}

>  The question is can we write an
> easy generic test for it,

It's a bit more complicated than the existing tests, but I think it is
doable.

> and also clearly phrase the criteria in the
> list of compliance questions with a binary yes/no answer?

Does the BSP layer only modify machine-specific packages and only when
the MACHINE(s) defined by the BSP layer are selected? [yes/no]

The "only when" part is covered by the existing tests (because they keep
MACHINE constant). The missing part is comparing different MACHINE
sstamps.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [OE-core] [PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Richard Purdie]

On Wed, 2017-03-01 at 16:51 +0100, Patrick Ohly wrote:
> On Wed, 2017-03-01 at 15:12 +0000, Richard Purdie wrote:
> > 
> > On Wed, 2017-03-01 at 08:10 +0100, Patrick Ohly wrote:
> > > 
> > > Is the "build single distro for different machines" scenario that
> > > I
> > > described part of the Yocto Compliance 2.0? Should there be tests
> > > for
> > > it?
> > Right now its not
> Okay, so the goal is a bit less ambitious than I had thought. I
> wonder
> whether that's useful, because at least the problems Ostro and AGL
> (at
> least as far as I understood it from lurking on their mailing list)
> had
> only happened when trying to combine multiple BSP layers *and*
> actually
> using the different machines in the same distro.
> 
> > 
> > but I'd consider it.
> At least I'd find that useful - not sure about others ;-}

I do like the idea, I'm also mindful of walking before running...

> > 
> >  The question is can we write an
> > easy generic test for it,
> It's a bit more complicated than the existing tests, but I think it
> is
> doable.
> 
> > 
> > and also clearly phrase the criteria in the
> > list of compliance questions with a binary yes/no answer?
> Does the BSP layer only modify machine-specific packages and only
> when
> the MACHINE(s) defined by the BSP layer are selected? [yes/no]
> 
> The "only when" part is covered by the existing tests (because they
> keep
> MACHINE constant). The missing part is comparing different MACHINE
> sstamps.

That seems reasonable, unless the layer in question applying for
compatibility is not a BSP layer but thats a minor detail.

I'm open to more details on what the test would look like.

Cheers,

Richard

Re: [PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Richard Purdie]

On Wed, 2017-03-01 at 16:51 +0100, Patrick Ohly wrote:
> On Wed, 2017-03-01 at 15:12 +0000, Richard Purdie wrote:
> > 
> > On Wed, 2017-03-01 at 08:10 +0100, Patrick Ohly wrote:
> > > 
> > > Is the "build single distro for different machines" scenario that
> > > I
> > > described part of the Yocto Compliance 2.0? Should there be tests
> > > for
> > > it?
> > Right now its not
> Okay, so the goal is a bit less ambitious than I had thought. I
> wonder
> whether that's useful, because at least the problems Ostro and AGL
> (at
> least as far as I understood it from lurking on their mailing list)
> had
> only happened when trying to combine multiple BSP layers *and*
> actually
> using the different machines in the same distro.
> 
> > 
> > but I'd consider it.
> At least I'd find that useful - not sure about others ;-}

I do like the idea, I'm also mindful of walking before running...

> > 
> >  The question is can we write an
> > easy generic test for it,
> It's a bit more complicated than the existing tests, but I think it
> is
> doable.
> 
> > 
> > and also clearly phrase the criteria in the
> > list of compliance questions with a binary yes/no answer?
> Does the BSP layer only modify machine-specific packages and only
> when
> the MACHINE(s) defined by the BSP layer are selected? [yes/no]
> 
> The "only when" part is covered by the existing tests (because they
> keep
> MACHINE constant). The missing part is comparing different MACHINE
> sstamps.

That seems reasonable, unless the layer in question applying for
compatibility is not a BSP layer but thats a minor detail.

I'm open to more details on what the test would look like.

Cheers,

Richard

Re: [OE-core] [PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Patrick Ohly]

On Wed, 2017-03-01 at 16:01 +0000, Richard Purdie wrote:
> On Wed, 2017-03-01 at 16:51 +0100, Patrick Ohly wrote:
> > On Wed, 2017-03-01 at 15:12 +0000, Richard Purdie wrote:
> > > 
> > > On Wed, 2017-03-01 at 08:10 +0100, Patrick Ohly wrote:
> > > > 
> > > > Is the "build single distro for different machines" scenario that
> > > > I
> > > > described part of the Yocto Compliance 2.0? Should there be tests
> > > > for
> > > > it?
> > > Right now its not
> > Okay, so the goal is a bit less ambitious than I had thought. I
> > wonder
> > whether that's useful, because at least the problems Ostro and AGL
> > (at
> > least as far as I understood it from lurking on their mailing list)
> > had
> > only happened when trying to combine multiple BSP layers *and*
> > actually
> > using the different machines in the same distro.
> > 
> > > 
> > > but I'd consider it.
> > At least I'd find that useful - not sure about others ;-}
> 
> I do like the idea, I'm also mindful of walking before running...

But bumping the requirements in the Yocto Compliance often will irritate
people, because they will have to redo the compliance testing more
often.

> > >  The question is can we write an
> > > easy generic test for it,
> > It's a bit more complicated than the existing tests, but I think it
> > is
> > doable.
> > 
> > > 
> > > and also clearly phrase the criteria in the
> > > list of compliance questions with a binary yes/no answer?
> > Does the BSP layer only modify machine-specific packages and only
> > when
> > the MACHINE(s) defined by the BSP layer are selected? [yes/no]
> > 
> > The "only when" part is covered by the existing tests (because they
> > keep
> > MACHINE constant). The missing part is comparing different MACHINE
> > sstamps.
> 
> That seems reasonable, unless the layer in question applying for
> compatibility is not a BSP layer but thats a minor detail.
> 
> I'm open to more details on what the test would look like.

I guess I now have the AR to write such a test? ;-}

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Patrick Ohly]

On Wed, 2017-03-01 at 16:01 +0000, Richard Purdie wrote:
> On Wed, 2017-03-01 at 16:51 +0100, Patrick Ohly wrote:
> > On Wed, 2017-03-01 at 15:12 +0000, Richard Purdie wrote:
> > > 
> > > On Wed, 2017-03-01 at 08:10 +0100, Patrick Ohly wrote:
> > > > 
> > > > Is the "build single distro for different machines" scenario that
> > > > I
> > > > described part of the Yocto Compliance 2.0? Should there be tests
> > > > for
> > > > it?
> > > Right now its not
> > Okay, so the goal is a bit less ambitious than I had thought. I
> > wonder
> > whether that's useful, because at least the problems Ostro and AGL
> > (at
> > least as far as I understood it from lurking on their mailing list)
> > had
> > only happened when trying to combine multiple BSP layers *and*
> > actually
> > using the different machines in the same distro.
> > 
> > > 
> > > but I'd consider it.
> > At least I'd find that useful - not sure about others ;-}
> 
> I do like the idea, I'm also mindful of walking before running...

But bumping the requirements in the Yocto Compliance often will irritate
people, because they will have to redo the compliance testing more
often.

> > >  The question is can we write an
> > > easy generic test for it,
> > It's a bit more complicated than the existing tests, but I think it
> > is
> > doable.
> > 
> > > 
> > > and also clearly phrase the criteria in the
> > > list of compliance questions with a binary yes/no answer?
> > Does the BSP layer only modify machine-specific packages and only
> > when
> > the MACHINE(s) defined by the BSP layer are selected? [yes/no]
> > 
> > The "only when" part is covered by the existing tests (because they
> > keep
> > MACHINE constant). The missing part is comparing different MACHINE
> > sstamps.
> 
> That seems reasonable, unless the layer in question applying for
> compatibility is not a BSP layer but thats a minor detail.
> 
> I'm open to more details on what the test would look like.

I guess I now have the AR to write such a test? ;-}

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [OE-core] [PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Patrick Ohly]

On Mon, 2017-02-20 at 15:12 -0600, Aníbal Limón wrote:
> +    try:
> +        ldata = bb.parse.handle(lconf, ldata, include=True)
> +    except BaseException as exc:
> +        raise LayerError(exc)

Anibal, where is this LayerError class in compatlayer/__init__.py
supposed to come from? It doesn't get imported, leading to:

  File
"/fast/work/intel-iot-refkit/openembedded-core/scripts/lib/compatlayer/__init__.py", line 45, in _get_layer_collections
    raise LayerError(exc)
NameError: name 'LayerError' is not defined

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Patrick Ohly]

On Mon, 2017-02-20 at 15:12 -0600, Aníbal Limón wrote:
> +    try:
> +        ldata = bb.parse.handle(lconf, ldata, include=True)
> +    except BaseException as exc:
> +        raise LayerError(exc)

Anibal, where is this LayerError class in compatlayer/__init__.py
supposed to come from? It doesn't get imported, leading to:

  File
"/fast/work/intel-iot-refkit/openembedded-core/scripts/lib/compatlayer/__init__.py", line 45, in _get_layer_collections
    raise LayerError(exc)
NameError: name 'LayerError' is not defined

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [OE-core] [PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Aníbal Limón]

[-- Attachment #1: Type: text/plain, Size: 959 bytes --]



On 05/08/2017 08:36 AM, Patrick Ohly wrote:
> On Mon, 2017-02-20 at 15:12 -0600, Aníbal Limón wrote:
>> +    try:
>> +        ldata = bb.parse.handle(lconf, ldata, include=True)
>> +    except BaseException as exc:
>> +        raise LayerError(exc)
> 
> Anibal, where is this LayerError class in compatlayer/__init__.py
> supposed to come from? It doesn't get imported, leading to:
> 
>   File
> "/fast/work/intel-iot-refkit/openembedded-core/scripts/lib/compatlayer/__init__.py", line 45, in _get_layer_collections
>     raise LayerError(exc)
> NameError: name 'LayerError' is not defined

This code is based on kergoth ones from oe-tests-scripts [1], i forget
to include the definition sorry for that but only extends exception to
determine if there is a problem on parsing the layer conf.

	Anibal


[1]
https://github.com/kergoth/oe-test-scripts/blob/726fef692c7464932c2ded0fab22a875df78a31f/bb-determine-layers#L111

> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCHv2] yocto-compat-layer.py: Add script to YP Compatible Layer validation

[Aníbal Limón]

[-- Attachment #1: Type: text/plain, Size: 959 bytes --]



On 05/08/2017 08:36 AM, Patrick Ohly wrote:
> On Mon, 2017-02-20 at 15:12 -0600, Aníbal Limón wrote:
>> +    try:
>> +        ldata = bb.parse.handle(lconf, ldata, include=True)
>> +    except BaseException as exc:
>> +        raise LayerError(exc)
> 
> Anibal, where is this LayerError class in compatlayer/__init__.py
> supposed to come from? It doesn't get imported, leading to:
> 
>   File
> "/fast/work/intel-iot-refkit/openembedded-core/scripts/lib/compatlayer/__init__.py", line 45, in _get_layer_collections
>     raise LayerError(exc)
> NameError: name 'LayerError' is not defined

This code is based on kergoth ones from oe-tests-scripts [1], i forget
to include the definition sorry for that but only extends exception to
determine if there is a problem on parsing the layer conf.

	Anibal


[1]
https://github.com/kergoth/oe-test-scripts/blob/726fef692c7464932c2ded0fab22a875df78a31f/bb-determine-layers#L111

> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]