[PATCH 0/4] yocto-compat-layer.py updates

[PATCH 0/4] yocto-compat-layer.py updates

[Mark Hatle]

The changes in this series fix a number of issues with the yocto-compat-layer
script.

- When a layer is skipped due to missing dependencies, it was difficult to
  see this without inspecting the whole log.

- There needs to be a way to pass in a list of layers that can be used
  for dependencies, but should not be evaluated in this pass.  The overall
  evaluation is very time consuming and having to parse/validate 15
  dependent layers to see if the layer you are working on passes is not
  useful!

- The dependency processing was not working properly.  It was only processing
  dependencies at one level.  Move this to a recursive process so we're
  sure that all needed dependencies are present.

- Change how the initial signatures are stored to be compared to the layers
  signatures for distro/BSP checks.  Before the signatures were stored very
  early (before any dependencies were calculated), causing the dependencies
  for a BSP layer to trigger a failure in signature validation.

Mark Hatle (4):
  yocto-compat-layer.py: Add status for skipped items
  yocto-compat-layer.py: Add --dependency argument
  compatlayer/__init__.py: Allow add_layer to process recursive deps
  yocto-compat-layer.py: Fix the signature validation

 scripts/lib/compatlayer/__init__.py | 47 ++++++++++++++++++++++++++++++-------
 scripts/yocto-compat-layer.py       | 47 +++++++++++++++++++++++++++----------
 2 files changed, 73 insertions(+), 21 deletions(-)

-- 
1.8.3.1

[PATCH 1/4] yocto-compat-layer.py: Add status for skipped items

[Mark Hatle]

If items were skipped because the dependencies could not be found, we
want to record this was skipped so we can display it later.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 scripts/yocto-compat-layer.py | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/scripts/yocto-compat-layer.py b/scripts/yocto-compat-layer.py
index 9e74033..d13d8c0 100755
--- a/scripts/yocto-compat-layer.py
+++ b/scripts/yocto-compat-layer.py
@@ -107,6 +107,7 @@ def main():
 
     td = {}
     results = collections.OrderedDict()
+    results_status = collections.OrderedDict()
 
     logger.info('')
     logger.info('Getting initial bitbake variables ...')
@@ -125,19 +126,22 @@ def main():
         shutil.copyfile(bblayersconf + '.backup', bblayersconf)
 
         if not add_layer(bblayersconf, layer, layers, logger):
+            results[layer['name']] = None
+            results_status[layer['name']] = 'SKIPPED (Missing dependencies)'
+            layers_tested = layers_tested + 1
             continue
 
         result = test_layer_compatibility(td, layer)
         results[layer['name']] = result
+        results_status[layer['name']] = 'PASS' if results[layer['name']].wasSuccessful() else 'FAIL'
         layers_tested = layers_tested + 1
 
     if layers_tested:
         logger.info('')
         logger.info('Summary of results:')
         logger.info('')
-        for layer_name in results:
-            logger.info('%s ... %s' % (layer_name, 'PASS' if \
-                    results[layer_name].wasSuccessful() else 'FAIL'))
+        for layer_name in results_status:
+            logger.info('%s ... %s' % (layer_name, results_status[layer_name]))
 
     cleanup_bblayers(None, None)
 
-- 
1.8.3.1

[PATCH 2/4] yocto-compat-layer.py: Add --dependency argument

[Mark Hatle]

When processing a large number of items, there are times that it would be
nice to be able to pass in a series of layers that can be used as dependencies
for the layer that is being scanned.  This avoids the significant overhead
of processing all of the layers to compatibility.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 scripts/yocto-compat-layer.py | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/scripts/yocto-compat-layer.py b/scripts/yocto-compat-layer.py
index d13d8c0..f8a1ac7 100755
--- a/scripts/yocto-compat-layer.py
+++ b/scripts/yocto-compat-layer.py
@@ -47,6 +47,8 @@ def main():
             help='Layer to test compatibility with Yocto Project')
     parser.add_argument('-o', '--output-log',
             help='File to output log (optional)', action='store')
+    parser.add_argument('--dependency', nargs="+",
+            help='Layers to process for dependencies', action='store')
     parser.add_argument('-n', '--no-auto', help='Disable auto layer discovery',
             action='store_true')
     parser.add_argument('-d', '--debug', help='Enable debug output',
@@ -80,6 +82,11 @@ def main():
     if not layers:
         logger.error("Fail to detect layers")
         return 1
+    if args.dependency:
+        dep_layers = detect_layers(args.dependency, args.no_auto)
+        dep_layers = dep_layers + layers
+    else:
+        dep_layers = layers
 
     logger.info("Detected layers:")
     for layer in layers:
@@ -125,7 +132,8 @@ def main():
 
         shutil.copyfile(bblayersconf + '.backup', bblayersconf)
 
-        if not add_layer(bblayersconf, layer, layers, logger):
+        if not add_layer(bblayersconf, layer, dep_layers, logger):
+            logger.info('Skipping %s due to missing dependencies.' % layer['name'])
             results[layer['name']] = None
             results_status[layer['name']] = 'SKIPPED (Missing dependencies)'
             layers_tested = layers_tested + 1
-- 
1.8.3.1

[PATCH 3/4] compatlayer/__init__.py: Allow add_layer to process recursive deps

[Mark Hatle]

When processing a layer for dependencies, you have to process the layer
itself, it's dependencies, the dependencies dependencies and so forth until
all items have been processed.

i.e.:  LayerA requires LayerB requires LayerC requires layerD

The end result should be LayerB, LayerC and LayerD are all dependencies of
LayerA.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 scripts/lib/compatlayer/__init__.py | 40 ++++++++++++++++++++++++++++++++-----
 1 file changed, 35 insertions(+), 5 deletions(-)

diff --git a/scripts/lib/compatlayer/__init__.py b/scripts/lib/compatlayer/__init__.py
index 58f94b7..24edc88 100644
--- a/scripts/lib/compatlayer/__init__.py
+++ b/scripts/lib/compatlayer/__init__.py
@@ -142,10 +142,9 @@ def _find_layer_depends(depend, layers):
 def add_layer(bblayersconf, layer, layers, logger):
     logger.info('Adding layer %s' % layer['name'])
 
-    for collection in layer['collections']:
-        depends = layer['collections'][collection]['depends']
-        if not depends:
-            continue
+    def recurse_dependencies(depends, layer, layers, logger, ret = []):
+        logger.debug('Processing dependencies %s for layer %s.' % \
+                    (depends, layer['name']))
 
         for depend in depends.split():
             # core (oe-core) is suppose to be provided
@@ -156,8 +155,39 @@ def add_layer(bblayersconf, layer, layers, logger):
             if not layer_depend:
                 logger.error('Layer %s depends on %s and isn\'t found.' % \
                         (layer['name'], depend))
-                return False
+                ret = None
+                continue
+
+            # We keep processing, even if ret is None, this allows us to report
+            # multiple errors at once
+            if ret is not None and layer_depend not in ret:
+                ret.append(layer_depend)
+
+            # Recursively process...
+            if 'collections' not in layer_depend:
+                continue
+
+            for collection in layer_depend['collections']:
+                collect_deps = layer_depend['collections'][collection]['depends']
+                if not collect_deps:
+                    continue
+                ret = recurse_dependencies(collect_deps, layer_depend, layers, logger, ret)
+
+        return ret
 
+    layer_depends = []
+    for collection in layer['collections']:
+        depends = layer['collections'][collection]['depends']
+        if not depends:
+            continue
+
+        layer_depends = recurse_dependencies(depends, layer, layers, logger, layer_depends)
+
+    # Note: [] (empty) is allowed, None is not!
+    if layer_depends is None:
+        return False
+    else:
+        for layer_depend in layer_depends:
             logger.info('Adding layer dependency %s' % layer_depend['name'])
             with open(bblayersconf, 'a+') as f:
                 f.write("\nBBLAYERS += \"%s\"\n" % layer_depend['path'])
-- 
1.8.3.1

[PATCH 4/4] yocto-compat-layer.py: Fix the signature validation

[Mark Hatle]

The initial signatures need to be collected -after- the dependency layers have
been added to the system.  Otherwise changes that happen due to dependencies,
outside of the layer being scanned, will show up as signature problems.

The add_layer function was split into two pieces so that we can process
the dependencies first, and then add the layer itself for the comparison.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 scripts/lib/compatlayer/__init__.py |  7 ++++---
 scripts/yocto-compat-layer.py       | 29 +++++++++++++++++++----------
 2 files changed, 23 insertions(+), 13 deletions(-)

diff --git a/scripts/lib/compatlayer/__init__.py b/scripts/lib/compatlayer/__init__.py
index 24edc88..06c84f4 100644
--- a/scripts/lib/compatlayer/__init__.py
+++ b/scripts/lib/compatlayer/__init__.py
@@ -139,9 +139,7 @@ def _find_layer_depends(depend, layers):
                 return layer
     return None
 
-def add_layer(bblayersconf, layer, layers, logger):
-    logger.info('Adding layer %s' % layer['name'])
-
+def add_layer_dependencies(bblayersconf, layer, layers, logger):
     def recurse_dependencies(depends, layer, layers, logger, ret = []):
         logger.debug('Processing dependencies %s for layer %s.' % \
                     (depends, layer['name']))
@@ -191,7 +189,10 @@ def add_layer(bblayersconf, layer, layers, logger):
             logger.info('Adding layer dependency %s' % layer_depend['name'])
             with open(bblayersconf, 'a+') as f:
                 f.write("\nBBLAYERS += \"%s\"\n" % layer_depend['path'])
+    return True
 
+def add_layer(bblayersconf, layer, layers, logger):
+    logger.info('Adding layer %s' % layer['name'])
     with open(bblayersconf, 'a+') as f:
         f.write("\nBBLAYERS += \"%s\"\n" % layer['path'])
 
diff --git a/scripts/yocto-compat-layer.py b/scripts/yocto-compat-layer.py
index f8a1ac7..22c0c2d 100755
--- a/scripts/yocto-compat-layer.py
+++ b/scripts/yocto-compat-layer.py
@@ -22,7 +22,7 @@ import scriptpath
 scriptpath.add_oe_lib_path()
 scriptpath.add_bitbake_lib_path()
 
-from compatlayer import LayerType, detect_layers, add_layer, get_signatures
+from compatlayer import LayerType, detect_layers, add_layer, add_layer_dependencies, get_signatures
 from oeqa.utils.commands import get_bb_vars
 
 PROGNAME = 'yocto-compat-layer'
@@ -116,29 +116,38 @@ def main():
     results = collections.OrderedDict()
     results_status = collections.OrderedDict()
 
-    logger.info('')
-    logger.info('Getting initial bitbake variables ...')
-    td['bbvars'] = get_bb_vars()
-    logger.info('Getting initial signatures ...')
-    td['builddir'] = builddir
-    td['sigs'] = get_signatures(td['builddir'])
-    logger.info('')
-
     layers_tested = 0
     for layer in layers:
         if layer['type'] == LayerType.ERROR_NO_LAYER_CONF or \
                 layer['type'] == LayerType.ERROR_BSP_DISTRO:
             continue
 
+        logger.info('')
+        logger.info("Setting up for %s(%s), %s" % (layer['name'], layer['type'],
+            layer['path']))
+
         shutil.copyfile(bblayersconf + '.backup', bblayersconf)
 
-        if not add_layer(bblayersconf, layer, dep_layers, logger):
+        if not add_layer_dependencies(bblayersconf, layer, dep_layers, logger):
             logger.info('Skipping %s due to missing dependencies.' % layer['name'])
             results[layer['name']] = None
             results_status[layer['name']] = 'SKIPPED (Missing dependencies)'
             layers_tested = layers_tested + 1
             continue
 
+        logger.info('Getting initial bitbake variables ...')
+        td['bbvars'] = get_bb_vars()
+        logger.info('Getting initial signatures ...')
+        td['builddir'] = builddir
+        td['sigs'] = get_signatures(td['builddir'])
+
+        if not add_layer(bblayersconf, layer, dep_layers, logger):
+            logger.info('Skipping %s ???.' % layer['name'])
+            results[layer['name']] = None
+            results_status[layer['name']] = 'SKIPPED (Unknown)'
+            layers_tested = layers_tested + 1
+            continue
+
         result = test_layer_compatibility(td, layer)
         results[layer['name']] = result
         results_status[layer['name']] = 'PASS' if results[layer['name']].wasSuccessful() else 'FAIL'
-- 
1.8.3.1

Re: [PATCH 0/4] yocto-compat-layer.py updates

[Aníbal Limón]

[-- Attachment #1: Type: text/plain, Size: 1651 bytes --]

Acked-by: Aníbal Limón <anibal.limon@linux.intel.com>

On 03/30/2017 08:30 PM, Mark Hatle wrote:
> The changes in this series fix a number of issues with the yocto-compat-layer
> script.
> 
> - When a layer is skipped due to missing dependencies, it was difficult to
>   see this without inspecting the whole log.
> 
> - There needs to be a way to pass in a list of layers that can be used
>   for dependencies, but should not be evaluated in this pass.  The overall
>   evaluation is very time consuming and having to parse/validate 15
>   dependent layers to see if the layer you are working on passes is not
>   useful!
> 
> - The dependency processing was not working properly.  It was only processing
>   dependencies at one level.  Move this to a recursive process so we're
>   sure that all needed dependencies are present.
> 
> - Change how the initial signatures are stored to be compared to the layers
>   signatures for distro/BSP checks.  Before the signatures were stored very
>   early (before any dependencies were calculated), causing the dependencies
>   for a BSP layer to trigger a failure in signature validation.
> 
> Mark Hatle (4):
>   yocto-compat-layer.py: Add status for skipped items
>   yocto-compat-layer.py: Add --dependency argument
>   compatlayer/__init__.py: Allow add_layer to process recursive deps
>   yocto-compat-layer.py: Fix the signature validation
> 
>  scripts/lib/compatlayer/__init__.py | 47 ++++++++++++++++++++++++++++++-------
>  scripts/yocto-compat-layer.py       | 47 +++++++++++++++++++++++++++----------
>  2 files changed, 73 insertions(+), 21 deletions(-)
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]