* [PATCH 0/5] Implement a new remote layer module
@ 2017-07-28 15:37 Mark Hatle
2017-07-28 15:37 ` [PATCH 1/5] lib/layers: Initial layer and layer index implementeation Mark Hatle
` (4 more replies)
0 siblings, 5 replies; 20+ messages in thread
From: Mark Hatle @ 2017-07-28 15:37 UTC (permalink / raw)
To: bitbake-devel; +Cc: toaster
The code for this work is available at:
git://git.openembedde.org/poky-contrib mgh/bitbake-setup
A new module 'layers' is being added to bitbake. This module contains
two parts that work together. A 'layerindex' piece and a 'manager' piece.
The layerindex piece is designed to connect to layerindex content, such as
layers.openembedded.org, pull down layerindex information and provide basic
functionality for using the index. It is capable of managing information
from multiple layerindex sources and resolving dependencies across indexes.
The connection to the external items is handled through plugins. This permits
us to implement both a restapi format (connect to the web), but also an
internal 'cooker' implementation. The cooker implementation will allow us to
create a mini-layerindex from the currently parsed system contents.
The internal data representation always uses the web style restapi, but is
wrapped with a number of layerindex provided object classes.
The layerindex provides object classes for each exported layerindex item, such
as Branch, LayerItem, LayerBranches, LayerDependency, etc... Each of these
objects contain various accessor functions that simply return the data or
further process the data to make it easier to access related objects. For
example a LayerBranch object refers to both a Branch-Id and a LayerItem-Id.
There are accessor functions designed to return back the Branch or LayerItem
without the user needing to know how to process the data structures.
The manager provides connection information to get to cooker instances for
some plugins, such as the layerindex's cooker plugin. It also can interpret
the basic BBLAYERS and COLLECTIONS information to determine what the user
already has. (The layerindex cooker plugin relies on this behavior.)
However, it's primary purpose is to manage downloads of layers, based on
layerindex generated dependency information. This download mechanism also
uses a plugin architecture. Currently only a 'fetch2' plugin has been
implemented. (I expect in the future additional plugins for 'git-repo', and
others to be implemented as needed.)
The download portion of the manager takes a 4 step approach:
* setup(dependencies) - takes a dependency list of what to download/install
* fetch() - fetch the layers
* unpack() - unpack the fetched layers
* update_bblayers() - update bblayers based on the previous activities
The code includes a set of bitbake-selftests for the described functionality.
Some items are not testable, such as the cooker integration, as cooker is not
available and the test data may be too limited to be useful.
The bitbake-layers has been updated to use these new modules for the
layerindex-fetch and layerindex-show-depends information.
The 'lsupdate.py' for the toaster has also been updated to use this item.
However, the longer term expectation is Toaster will be modified to no longer
handle an internal 'snapshot' of the layer index, but will use this new
functionality instead of process layer information. That work will begin once
these items have been reviewed and agreed upon by the community.
Once toaster has been updated, the intention is to create a new
'bitbake-setup' program, that will have similar functionality to the
'wr-lx-setup' project on github (https://github.com/Wind-River/wr-lx-setup).
Mark Hatle (4):
lib/layers: Initial layer and layer index implementeation
bitbake-selftest: Add layers module tests
lib/bblayers: Add support for the new layer modules
toaster/orm/management/commands/lsupdates.py: Use new layerindex
module
Paul Eggleton (1):
bitbake-layers: disable parsing for layerindex commands
bin/bitbake-selftest | 5 +-
lib/bblayers/layerindex.py | 317 +++-----
lib/layers/__init__.py | 0
lib/layers/layerindex/__init__.py | 974 +++++++++++++++++++++++
lib/layers/layerindex/common.py | 146 ++++
lib/layers/layerindex/cooker.py | 223 ++++++
lib/layers/layerindex/restapi.py | 375 +++++++++
lib/layers/manager/__init__.py | 253 ++++++
lib/layers/manager/common.py | 60 ++
lib/layers/manager/fetch2.py | 210 +++++
lib/layers/tests/__init__.py | 0
lib/layers/tests/layerindex.py | 372 +++++++++
lib/layers/tests/manager.py | 155 ++++
lib/toaster/orm/management/commands/lsupdates.py | 215 ++---
14 files changed, 2954 insertions(+), 351 deletions(-)
create mode 100644 lib/layers/__init__.py
create mode 100644 lib/layers/layerindex/__init__.py
create mode 100644 lib/layers/layerindex/common.py
create mode 100644 lib/layers/layerindex/cooker.py
create mode 100644 lib/layers/layerindex/restapi.py
create mode 100644 lib/layers/manager/__init__.py
create mode 100644 lib/layers/manager/common.py
create mode 100644 lib/layers/manager/fetch2.py
create mode 100644 lib/layers/tests/__init__.py
create mode 100644 lib/layers/tests/layerindex.py
create mode 100644 lib/layers/tests/manager.py
--
1.8.3.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH 1/5] lib/layers: Initial layer and layer index implementeation
2017-07-28 15:37 [PATCH 0/5] Implement a new remote layer module Mark Hatle
@ 2017-07-28 15:37 ` Mark Hatle
2017-08-03 18:05 ` Paul Eggleton
2017-07-28 15:37 ` [PATCH 2/5] bitbake-selftest: Add layers module tests Mark Hatle
` (3 subsequent siblings)
4 siblings, 1 reply; 20+ messages in thread
From: Mark Hatle @ 2017-07-28 15:37 UTC (permalink / raw)
To: bitbake-devel; +Cc: toaster
This module provides two components, layerindex, which is used to talk a
layerindex, such as layers.openembedded.org.
The other module is the 'manager'. This module will handle downloading,
re-writing the bblayers to match the downloads and other related layer
management tasks.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
lib/layers/__init__.py | 0
lib/layers/layerindex/__init__.py | 962 ++++++++++++++++++++++++++++++++++++++
lib/layers/layerindex/common.py | 146 ++++++
lib/layers/layerindex/cooker.py | 223 +++++++++
lib/layers/layerindex/restapi.py | 348 ++++++++++++++
lib/layers/manager/__init__.py | 242 ++++++++++
lib/layers/manager/common.py | 60 +++
lib/layers/manager/fetch2.py | 210 +++++++++
8 files changed, 2191 insertions(+)
create mode 100644 lib/layers/__init__.py
create mode 100644 lib/layers/layerindex/__init__.py
create mode 100644 lib/layers/layerindex/common.py
create mode 100644 lib/layers/layerindex/cooker.py
create mode 100644 lib/layers/layerindex/restapi.py
create mode 100644 lib/layers/manager/__init__.py
create mode 100644 lib/layers/manager/common.py
create mode 100644 lib/layers/manager/fetch2.py
diff --git a/lib/layers/__init__.py b/lib/layers/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/lib/layers/layerindex/__init__.py b/lib/layers/layerindex/__init__.py
new file mode 100644
index 0000000..0c7c483
--- /dev/null
+++ b/lib/layers/layerindex/__init__.py
@@ -0,0 +1,962 @@
+# Copyright (C) 2016-2017 Wind River Systems, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+# See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import datetime
+
+import logging
+import imp
+
+import bb.fetch2
+
+from collections import OrderedDict
+
+logger = logging.getLogger('BitBake.layers.layerindex')
+
+class LayerIndex():
+ def __init__(self, d):
+ if d:
+ self.data = d
+ else:
+ import bb.data
+ self.data = bb.data.init()
+ # We need to use the fetcher to parse the URL
+ # it requires DL_DIR to be set
+ self.data.setVar('DL_DIR', os.getcwd())
+
+ self.lindex = []
+
+ self.plugins = []
+
+ import bb.utils
+ bb.utils.load_plugins(logger, self.plugins, os.path.dirname(__file__))
+ for plugin in self.plugins:
+ if hasattr(plugin, 'init'):
+ plugin.init(self)
+
+ def __add__(self, other):
+ newIndex = LayerIndex(self.data)
+
+ if self.__class__ != newIndex.__class__ or \
+ other.__class__ != newIndex.__class__:
+ raise TypeException("Can not add different types.")
+
+ for lindexEnt in self.lindex:
+ newIndex.lindex.append(lindexEnt)
+
+ for lindexEnt in other.lindex:
+ newIndex.lindex.append(lindexEnt)
+
+ return newIndex
+
+ def _get_plugin(self, type):
+ for plugin in self.plugins:
+ if hasattr(plugin, 'plugin_type'):
+ plugintype = plugin.plugin_type()
+ logger.debug(1, "Looking for IndexPlugin - %s ? %s" % (plugintype, type))
+ if plugintype and plugintype == type:
+ return plugin
+ return None
+
+ loadRecipes = 1
+ def load_layerindex(self, indexURIs, reload=False, load='layerDependencies recipes machines distros'):
+ """Load the layerindex.
+
+indexURIs- This may be one or more indexes (white space seperated).
+
+reload - If reload is True, then any previously loaded indexes will be forgotten.
+
+load - Ability to NOT load certain elements for performance. White space seperated list
+ of optional things to load. (branches, layerItems and layerBranches is always
+ loaded.) Note: the plugins are permitted to ignore this and load everything.
+
+The format of the indexURI:
+
+ <url>;type=<type>;branch=<branch>;cache=<cache>;desc=<description>
+
+ Note: the 'branch' parameter if set can select multiple branches by using
+ 'OR', such as 'branch=masterORmortyORpyro'. However, many operations on look
+ at the -first- branch specified!
+
+ The cache value may be undefined, in this case a network failure will
+ result in an error, otherwise the system will look for a file of the cache
+ name and load that instead.
+
+ For example:
+
+ http://layers.openembedded.org/layerindex/api/;type=restapi;branch=master;desc=OpenEmbedded%20Layer%20Index
+ file://conf/bblayers.conf;type=internal
+
+restapi is either a web url or a local file or a local directory with one
+or more .json file in it in the restapi format
+
+internal refers to any layers loaded as part of a project conf/bblayers.conf
+"""
+ if reload:
+ self.lindex = []
+
+ logger.debug(1, 'Loading: %s' % indexURIs)
+
+ for url in indexURIs.split():
+ ud = bb.fetch2.FetchData(url, self.data)
+
+ if 'type' not in ud.parm:
+ raise bb.fetch2.MissingParameterError('type', url)
+
+ plugin = self._get_plugin(ud.parm['type'])
+
+ if not plugin:
+ raise NotImplementedError("%s: type %s is not available" % (url, ud.parm['type']))
+
+ # TODO: Implement 'cache', for when the network is not available
+ lindexEnt = plugin.load_index(ud, load)
+
+ if 'CONFIG' not in lindexEnt:
+ raise Exception('Internal Error: Missing configuration data in index %s' % url)
+
+ # Mark CONFIG data as something we've added...
+ lindexEnt['CONFIG']['local'] = []
+ lindexEnt['CONFIG']['local'].append('CONFIG')
+
+ if 'branches' not in lindexEnt:
+ raise Exception('Internal Error: No branches defined in index %s' % url)
+
+ # Create quick lookup layerBranches_layerId_branchId table
+ if 'layerBranches' in lindexEnt:
+ # Create associated quick lookup indexes
+ lindexEnt['layerBranches_layerId_branchId'] = {}
+ for layerBranchId in lindexEnt['layerBranches']:
+ obj = lindexEnt['layerBranches'][layerBranchId]
+ lindexEnt['layerBranches_layerId_branchId']["%s:%s" % (obj.get_layer_id(), obj.get_branch_id())] = obj
+ # Mark layerBranches_layerId_branchId as something we added
+ lindexEnt['CONFIG']['local'].append('layerBranches_layerId_branchId')
+
+ # Create quick lookup layerDependencies_layerBranchId table
+ if 'layerDependencies' in lindexEnt:
+ # Create associated quick lookup indexes
+ lindexEnt['layerDependencies_layerBranchId'] = {}
+ for layerDependencyId in lindexEnt['layerDependencies']:
+ obj = lindexEnt['layerDependencies'][layerDependencyId]
+ if obj.get_layerbranch_id() not in lindexEnt['layerDependencies_layerBranchId']:
+ lindexEnt['layerDependencies_layerBranchId'][obj.get_layerbranch_id()] = [obj]
+ else:
+ lindexEnt['layerDependencies_layerBranchId'][obj.get_layerbranch_id()].append(obj)
+ # Mark layerDependencies_layerBranchId as something we added
+ lindexEnt['CONFIG']['local'].append('layerDependencies_layerBranchId')
+
+ # Create quick lookup layerUrls
+ if 'layerBranches' in lindexEnt:
+ # Create associated quick lookup indexes
+ lindexEnt['layerUrls'] = {}
+ for layerBranchId in lindexEnt['layerBranches']:
+ obj = lindexEnt['layerBranches'][layerBranchId]
+ vcs_url = obj.get_layer().get_vcs_url()
+ if vcs_url not in lindexEnt['layerUrls']:
+ lindexEnt['layerUrls'][vcs_url] = [obj]
+ else:
+ # We insert this if there is no subdir, we know it's the parent
+ if not obj.get_vcs_subdir():
+ lindexEnt['layerUrls'][vcs_url].insert(0, obj)
+ else:
+ lindexEnt['layerUrls'][vcs_url].append(obj)
+ # Mark layerUrls as something we added
+ lindexEnt['CONFIG']['local'].append('layerUrls')
+
+ self.lindex.append(lindexEnt)
+
+ def store_layerindex(self, indexURI, lindex=None):
+ """Store a layerindex
+
+Typically this will be used to create a local cache file of a remote index.
+
+ file://<path>;type=<type>;branch=<branch>
+
+We can write out in either the restapi or django formats. The split option
+will write out the individual elements split by layer and related components.
+"""
+ if not lindex:
+ logger.warning('No index to write, nothing to do.')
+ return
+
+ ud = bb.fetch2.FetchData(indexURI, self.data)
+
+ if 'type' not in ud.parm:
+ raise bb.fetch2.MissingParameterError('type', indexURI)
+
+ plugin = self._get_plugin(ud.parm['type'])
+
+ if not plugin:
+ raise NotImplementedError("%s: type %s is not available" % (url, ud.parm['type']))
+
+ lindexEnt = plugin.store_index(ud, lindex)
+
+
+ def get_json_query(self, query):
+ """Return a query in restapi format
+
+This is a compatibility function. It will acts like the web restapi query
+and return back the information related to a specific query. It can be used
+but other components of the system that would rather deal with restapi
+style queries then the regular functions in this class.
+
+Note: only select queries are supported. This will have to be expanded
+to support additional queries.
+
+This function will merge multiple databases together to return a single
+coherent 'superset' result, when more then one index has been loaded.
+"""
+
+ # TODO Implement get_json_query
+ raise Exception("get_json_query: not Implemented!")
+
+ def is_empty(self):
+ """Return True or False if the index has any usable data.
+
+We check the lindex entries to see if they have a branch set, as well as
+layerBranches set. If not, they are effectively blank."""
+
+ found = False
+ for lindex in self.lindex:
+ if 'branches' in lindex and 'layerBranches' in lindex and \
+ lindex['branches'] and lindex['layerBranches']:
+ found = True
+ break
+ return not found
+
+
+ def find_vcs_url(self, vcs_url, branch=None):
+ """Return the first layerBranch with the given vcs_url
+
+If a branch has not been specified, we will iterate over the branches in
+the default configuration until the first vcs_url/branch match."""
+
+ for lindex in self.lindex:
+ logger.debug(1, ' searching %s' % lindex['CONFIG']['DESCRIPTION'])
+ layerBranch = self._find_vcs_url(lindex, vcs_url, branch)
+ if layerBranch:
+ return layerBranch
+ return None
+
+ def _find_vcs_url(self, lindex, vcs_url, branch=None):
+ if 'branches' not in lindex or 'layerBranches' not in lindex:
+ return None
+
+ if vcs_url in lindex['layerUrls']:
+ for layerBranch in lindex['layerUrls'][vcs_url]:
+ if branch and branch == layerBranch.get_branch().get_name():
+ return layerBranch
+ if not branch:
+ return layerBranch
+
+ return None
+
+
+ def find_collection(self, collection, version=None, branch=None):
+ """Return the first layerBranch with the given collection name
+
+If a branch has not been specified, we will iterate over the branches in
+the default configuration until the first colelction/branch match."""
+
+ logger.debug(1, 'find_collection: %s (%s) %s' % (collection, version, branch))
+
+ for lindex in self.lindex:
+ logger.debug(1, ' searching %s' % lindex['CONFIG']['DESCRIPTION'])
+ layerBranch = self._find_collection(lindex, collection, version, branch)
+ if layerBranch:
+ return layerBranch
+ else:
+ logger.debug(1, 'Collection %s (%s) not found for branch (%s)' % (collection, version, branch))
+ return None
+
+ def _find_collection(self, lindex, collection, version=None, branch=None):
+ if 'branches' not in lindex or 'layerBranches' not in lindex:
+ return None
+
+ def find_branch_layerItem(branch, collection, version):
+ for branchId in lindex['branches']:
+ if branch == lindex['branches'][branchId].get_name():
+ break
+ else:
+ return None
+
+ for layerBranchId in lindex['layerBranches']:
+ if branchId == lindex['layerBranches'][layerBranchId].get_branch_id() and \
+ collection == lindex['layerBranches'][layerBranchId].get_collection():
+ if not version or version == lindex['layerBranches'][layerBranchId].get_version():
+ return lindex['layerBranches'][layerBranchId]
+
+ return None
+
+ if branch:
+ layerBranch = find_branch_layerItem(branch, collection, version)
+ return layerBranch
+
+ # No branch, so we have to scan the branches in order...
+ # Use the config order if we have it...
+ if 'CONFIG' in lindex and 'BRANCH' in lindex['CONFIG']:
+ for branch in lindex['CONFIG']['BRANCH'].split('OR'):
+ layerBranch = find_branch_layerItem(branch, collection, version)
+ if layerBranch:
+ return layerBranch
+
+ # ...use the index order if we don't...
+ else:
+ for branchId in lindex['branches']:
+ branch = lindex['branches'][branchId].get_name()
+ layerBranch = get_branch_layerItem(branch, collection, version)
+ if layerBranch:
+ return layerBranch
+
+ return None
+
+
+ def get_layerbranch(self, name, branch=None):
+ """Return the layerBranch item for a given name and branch
+
+If a branch has not been specified, we will iterate over the branches in
+the default configuration until the first name/branch match."""
+
+ for lindex in self.lindex:
+ layerBranch = self._get_layerbranch(lindex, name, branch)
+ if layerBranch:
+ return layerBranch
+ return None
+
+ def _get_layerbranch(self, lindex, name, branch=None):
+ if 'branches' not in lindex or 'layerItems' not in lindex:
+ logger.debug(1, 'No branches or no layerItems in lindex %s' % (lindex['CONFIG']['DESCRIPTION']))
+ return None
+
+ def get_branch_layerItem(branch, name):
+ for branchId in lindex['branches']:
+ if branch == lindex['branches'][branchId].get_name():
+ break
+ else:
+ return None
+
+ for layerItemId in lindex['layerItems']:
+ if name == lindex['layerItems'][layerItemId].get_name():
+ break
+ else:
+ return None
+
+ key = "%s:%s" % (layerItemId, branchId)
+ if key in lindex['layerBranches_layerId_branchId']:
+ return lindex['layerBranches_layerId_branchId'][key]
+ return None
+
+ if branch:
+ layerBranch = get_branch_layerItem(branch, name)
+ return layerBranch
+
+ # No branch, so we have to scan the branches in order...
+ # Use the config order if we have it...
+ if 'CONFIG' in lindex and 'BRANCH' in lindex['CONFIG']:
+ for branch in lindex['CONFIG']['BRANCH'].split('OR'):
+ layerBranch = get_branch_layerItem(branch, name)
+ if layerBranch:
+ return layerBranch
+
+ # ...use the index order if we don't...
+ else:
+ for branchId in lindex['branches']:
+ branch = lindex['branches'][branchId].get_name()
+ layerBranch = get_branch_layerItem(branch, name)
+ if layerBranch:
+ return layerBranch
+ return None
+
+ def get_dependencies(self, names=None, layerBranches=None, ignores=None):
+ """Return a tuple of all dependencies and invalid items.
+
+The dependency scanning happens with a depth-first approach, so the returned
+dependencies should be in the best order to define a bblayers.
+
+names - a space deliminated list of layerItem names.
+Branches are resolved in the order of the specified index's load. Subsequent
+branch resolution is on the same branch.
+
+layerBranches - a list of layerBranches to resolve dependencies
+Branches are the same as the passed in layerBranch.
+
+ignores - a list of layer names to ignore
+
+Return value: (dependencies, invalid)
+
+dependencies is an orderedDict, with the key being the layer name.
+The value is a list with the first ([0]) being the layerBranch, and subsequent
+items being the layerDependency entries that caused this to be added.
+
+invalid is just a list of dependencies that were not found.
+"""
+ invalid = []
+
+ if not layerBranches:
+ layerBranches = []
+
+ if names:
+ for name in names.split():
+ if name in ignores:
+ continue
+
+ # Since we don't have a branch, we have to just find the first
+ # layerBranch with that name...
+ for lindex in self.lindex:
+ layerBranch = self._get_layerbranch(lindex, name)
+ if not layerBranch:
+ # Not in this index, hopefully it's in another...
+ continue
+
+ if layerBranch not in layerBranches:
+ layerBranches.append(layerBranch)
+ break
+ else:
+ logger.warning("Layer %s not found. Marked as invalid." % name)
+ invalid.append(name)
+ layerBranch = None
+
+ # Format is required['name'] = [ layer_branch, dependency1, dependency2, ..., dependencyN ]
+ dependencies = OrderedDict()
+ (dependencies, invalid) = self._get_dependencies(layerBranches, ignores, dependencies, invalid)
+
+ for layerBranch in layerBranches:
+ if layerBranch.get_layer().get_name() not in dependencies:
+ dependencies[layerBranch.get_layer().get_name()] = [layerBranch]
+
+ return (dependencies, invalid)
+
+
+ def _get_dependencies(self, layerBranches, ignores, dependencies, invalid):
+ for layerBranch in layerBranches:
+ name = layerBranch.get_layer().get_name()
+ # Do we ignore it?
+ if name in ignores:
+ continue
+
+ if 'layerDependencies_layerBranchId' not in layerBranch.index:
+ raise Exception('Missing layerDepedencies_layerBranchId cache! %s' % layerBranch.index['CONFIG']['DESCRIPTION'])
+
+ # Get a list of dependencies and then recursively process them
+ if layerBranch.get_id() in layerBranch.index['layerDependencies_layerBranchId']:
+ for layerDependency in layerBranch.index['layerDependencies_layerBranchId'][layerBranch.get_id()]:
+ depLayerBranch = layerDependency.get_dependency_layerBranch()
+
+ # Do we need to resolve across indexes?
+ if depLayerBranch.index != self.lindex[0]:
+ rdepLayerBranch = self.find_collection(
+ collection=depLayerBranch.get_collection(),
+ version=depLayerBranch.get_version()
+ )
+ if rdepLayerBranch != depLayerBranch:
+ logger.debug(1, 'Replaced %s:%s:%s with %s:%s:%s' % \
+ (depLayerBranch.index['CONFIG']['DESCRIPTION'],
+ depLayerBranch.get_branch().get_name(),
+ depLayerBranch.get_layer().get_name(),
+ rdepLayerBranch.index['CONFIG']['DESCRIPTION'],
+ rdepLayerBranch.get_branch().get_name(),
+ rdepLayerBranch.get_layer().get_name()))
+ depLayerBranch = rdepLayerBranch
+
+ # Is this dependency on the list to be ignored?
+ if depLayerBranch.get_layer().get_name() in ignores:
+ continue
+
+ # Previously found dependencies have been processed, as
+ # have their dependencies...
+ if depLayerBranch.get_layer().get_name() not in dependencies:
+ (dependencies, invalid) = self._get_dependencies([depLayerBranch], ignores, dependencies, invalid)
+
+ if depLayerBranch.get_layer().get_name() not in dependencies:
+ dependencies[depLayerBranch.get_layer().get_name()] = [depLayerBranch, layerDependency]
+ else:
+ if layerDependency not in dependencies[depLayerBranch.get_layer().get_name()]:
+ dependencies[depLayerBranch.get_layer().get_name()].append(layerDependency)
+
+ return (dependencies, invalid)
+
+
+ def list_obj(self, object):
+ """Print via the plain logger object information
+
+This function is used to implement debugging and provide the user info.
+"""
+ for lix in self.lindex:
+ if object not in lix:
+ continue
+
+ logger.plain ('')
+ logger.plain('Index: %s' % lix['CONFIG']['DESCRIPTION'])
+
+ output = []
+
+ if object == 'branches':
+ logger.plain ('%s %s %s' % ('{:26}'.format('branch'), '{:34}'.format('description'), '{:22}'.format('bitbake branch')))
+ logger.plain ('{:-^80}'.format(""))
+ for branchId in lix['branches']:
+ output.append('%s %s %s' % (
+ '{:26}'.format(lix['branches'][branchId].get_name()),
+ '{:34}'.format(lix['branches'][branchId].get_short_description()),
+ '{:22}'.format(lix['branches'][branchId].get_bitbake_branch())
+ ))
+ for line in sorted(output):
+ logger.plain (line)
+
+ continue
+
+ if object == 'layerItems':
+ logger.plain ('%s %s' % ('{:26}'.format('layer'), '{:34}'.format('description')))
+ logger.plain ('{:-^80}'.format(""))
+ for layerId in lix['layerItems']:
+ output.append('%s %s' % (
+ '{:26}'.format(lix['layerItems'][layerId].get_name()),
+ '{:34}'.format(lix['layerItems'][layerId].get_summary())
+ ))
+ for line in sorted(output):
+ logger.plain (line)
+
+ continue
+
+ if object == 'layerBranches':
+ logger.plain ('%s %s %s' % ('{:26}'.format('layer'), '{:34}'.format('description'), '{:19}'.format('collection:version')))
+ logger.plain ('{:-^80}'.format(""))
+ for layerBranchId in lix['layerBranches']:
+ output.append('%s %s %s' % (
+ '{:26}'.format(lix['layerBranches'][layerBranchId].get_layer().get_name()),
+ '{:34}'.format(lix['layerBranches'][layerBranchId].get_layer().get_summary()),
+ '{:19}'.format("%s:%s" %
+ (lix['layerBranches'][layerBranchId].get_collection(),
+ lix['layerBranches'][layerBranchId].get_version())
+ )
+ ))
+ for line in sorted(output):
+ logger.plain (line)
+
+ continue
+
+ if object == 'layerDependencies':
+ logger.plain ('%s %s %s %s' % ('{:19}'.format('branch'), '{:26}'.format('layer'), '{:11}'.format('dependency'), '{:26}'.format('layer')))
+ logger.plain ('{:-^80}'.format(""))
+ for layerDependency in lix['layerDependencies']:
+ if not lix['layerDependencies'][layerDependency].get_dependency_layerBranch():
+ continue
+
+ output.append('%s %s %s %s' % (
+ '{:19}'.format(lix['layerDependencies'][layerDependency].get_layerbranch().get_branch().get_name()),
+ '{:26}'.format(lix['layerDependencies'][layerDependency].get_layerbranch().get_layer().get_name()),
+ '{:11}'.format('requires' if lix['layerDependencies'][layerDependency].is_required() else 'recommends'),
+ '{:26}'.format(lix['layerDependencies'][layerDependency].get_dependency_layerBranch().get_layer().get_name())
+ ))
+ for line in sorted(output):
+ logger.plain (line)
+
+ continue
+
+ if object == 'recipes':
+ logger.plain ('%s %s %s' % ('{:20}'.format('recipe'), '{:10}'.format('version'), 'layer'))
+ logger.plain ('{:-^80}'.format(""))
+ output = []
+ for recipe in lix['recipes']:
+ output.append('%s %s %s' % (
+ '{:30}'.format(lix['recipes'][recipe].get_pn()),
+ '{:30}'.format(lix['recipes'][recipe].get_pv()),
+ lix['recipes'][recipe].get_layer().get_name()
+ ))
+ for line in sorted(output):
+ logger.plain (line)
+
+ continue
+
+ if object == 'machines':
+ logger.plain ('%s %s %s' % ('{:24}'.format('machine'), '{:34}'.format('description'), '{:19}'.format('layer')))
+ logger.plain ('{:-^80}'.format(""))
+ for machine in lix['machines']:
+ output.append('%s %s %s' % (
+ '{:24}'.format(lix['machines'][machine].get_name()),
+ ('{:34}'.format(lix['machines'][machine].get_description()))[:34],
+ '{:19}'.format(lix['machines'][machine].get_layerbranch().get_layer().get_name() )
+ ))
+ for line in sorted(output):
+ logger.plain (line)
+
+ continue
+
+ if object == 'distros':
+ logger.plain ('%s %s %s' % ('{:24}'.format('distro'), '{:34}'.format('description'), '{:19}'.format('layer')))
+ logger.plain ('{:-^80}'.format(""))
+ for distro in lix['distros']:
+ output.append('%s %s %s' % (
+ '{:24}'.format(lix['distros'][distro].get_name()),
+ ('{:34}'.format(lix['distros'][distro].get_description()))[:34],
+ '{:19}'.format(lix['distros'][distro].get_layerbranch().get_layer().get_name() )
+ ))
+ for line in sorted(output):
+ logger.plain (line)
+
+ continue
+
+ logger.plain ('')
+
+# Define enough of the layer index types so we can easily resolve them...
+# It is up to the loaders to create the classes from the raw data
+class LayerIndexItem():
+ def __init__(self, index, data):
+ self.index = index
+ self.data = data
+
+ def __eq__(self, other):
+ if self.__class__ != other.__class__:
+ return False
+ res=(self.data == other.data)
+ logger.debug(2, 'Compare objects: %s ? %s : %s' % (self.get_id(), other.get_id(), res))
+ return res
+
+ def define_data(self, id):
+ self.data = {}
+ self.data['id'] = id
+
+ def get_id(self):
+ return self.data['id']
+
+
+class Branch(LayerIndexItem):
+ def define_data(self, id, name, bitbake_branch,
+ short_description=None, sort_priority=1,
+ updates_enabled=True, updated=None,
+ update_environment=None):
+ self.data = {}
+ self.data['id'] = id
+ self.data['name'] = name
+ self.data['bitbake_branch'] = bitbake_branch
+ self.data['short_description'] = short_description or name
+ self.data['sort_priority'] = sort_priority
+ self.data['updates_enabled'] = updates_enabled
+ self.data['updated'] = updated or datetime.datetime.today().isoformat()
+ self.data['update_environment'] = update_environment
+
+ def get_name(self):
+ return self.data['name']
+
+ def get_short_description(self):
+ return self.data['short_description'].strip()
+
+ def get_bitbake_branch(self):
+ return self.data['bitbake_branch'] or self.get_name()
+
+
+class LayerItem(LayerIndexItem):
+ def define_data(self, id, name, status='P',
+ layer_type='A', summary=None,
+ description=None,
+ vcs_url=None, vcs_web_url=None,
+ vcs_web_tree_base_url=None,
+ vcs_web_file_base_url=None,
+ usage_url=None,
+ mailing_list_url=None,
+ index_preference=1,
+ classic=False,
+ updated=None):
+ self.data = {}
+ self.data['id'] = id
+ self.data['name'] = name
+ self.data['status'] = status
+ self.data['layer_type'] = layer_type
+ self.data['summary'] = summary or name
+ self.data['description'] = description or summary or name
+ self.data['vcs_url'] = vcs_url
+ self.data['vcs_web_url'] = vcs_web_url
+ self.data['vcs_web_tree_base_url'] = vcs_web_tree_base_url
+ self.data['vcs_web_file_base_url'] = vcs_web_file_base_url
+ self.data['index_preference'] = index_preference
+ self.data['classic'] = classic
+ self.data['updated'] = updated or datetime.datetime.today().isoformat()
+
+ def get_name(self):
+ return self.data['name']
+
+ def get_summary(self):
+ return self.data['summary']
+
+ def get_description(self):
+ return self.data['description'].strip()
+
+ def get_vcs_url(self):
+ return self.data['vcs_url']
+
+ def get_vcs_web_url(self):
+ return self.data['vcs_web_url']
+
+ def get_vcs_web_tree_base_url(self):
+ return self.data['vcs_web_tree_base_url']
+
+ def get_vcs_web_file_base_url(self):
+ return self.data['vcs_web_file_base_url']
+
+ def get_updated(self):
+ return self.data['updated']
+
+class LayerBranch(LayerIndexItem):
+ def define_data(self, id, collection, version, layer, branch,
+ vcs_subdir="", vcs_last_fetch=None,
+ vcs_last_rev=None, vcs_last_commit=None,
+ actual_branch="",
+ updated=None):
+ self.data = {}
+ self.data['id'] = id
+ self.data['collection'] = collection
+ self.data['version'] = version
+ self.data['layer'] = layer
+ self.data['branch'] = branch
+ self.data['vcs_subdir'] = vcs_subdir
+ self.data['vcs_last_fetch'] = vcs_last_fetch
+ self.data['vcs_last_rev'] = vcs_last_rev
+ self.data['vcs_last_commit'] = vcs_last_commit
+ self.data['actual_branch'] = actual_branch
+ self.data['updated'] = updated or datetime.datetime.today().isoformat()
+
+ def get_collection(self):
+ return self.data['collection']
+
+ def get_version(self):
+ return self.data['version']
+
+ def get_vcs_subdir(self):
+ return self.data['vcs_subdir']
+
+ def get_actual_branch(self):
+ return self.data['actual_branch'] or self.get_branch().get_name()
+
+ def get_updated(self):
+ return self.data['updated']
+
+ def get_layer_id(self):
+ return self.data['layer']
+
+ def get_branch_id(self):
+ return self.data['branch']
+
+ def get_layer(self):
+ layerItem = None
+ try:
+ layerItem = self.index['layerItems'][self.get_layer_id()]
+ except KeyError:
+ logger.error('Unable to find layerItems in index')
+ except IndexError:
+ logger.error('Unable to find layerId %s' % self.get_layer_id())
+ return layerItem
+
+ def get_branch(self):
+ branch = None
+ try:
+ branch = self.index['branches'][self.get_branch_id()]
+ except KeyError:
+ logger.error('Unable to find branches in index: %s' % self.index.keys())
+ except IndexError:
+ logger.error('Unable to find branchId %s' % self.get_branch_id())
+ return branch
+
+
+class LayerIndexItem_LayerBranch(LayerIndexItem):
+ def get_layerbranch_id(self):
+ return self.data['layerbranch']
+
+ def get_layerbranch(self):
+ layerBranch = None
+ try:
+ layerBranch = self.index['layerBranches'][self.get_layerbranch_id()]
+ except KeyError:
+ logger.error('Unable to find layerBranches in index')
+ except IndexError:
+ logger.error('Unable to find layerBranchId %s' % self.get_layerbranch_id())
+ return layerBranch
+
+ def get_layer_id(self):
+ layerBranch = self.get_layerbranch()
+ if layerBranch:
+ return layerBranch.get_layer_id()
+ return None
+
+ def get_layer(self):
+ layerBranch = self.get_layerbranch()
+ if layerBranch:
+ return layerBranch.get_layer()
+ return None
+
+class LayerDependency(LayerIndexItem_LayerBranch):
+ def define_data(self, id, layerbranch, dependency, required=True):
+ self.data = {}
+ self.data['id'] = id
+ self.data['layerbranch'] = layerbranch
+ self.data['dependency'] = dependency
+ self.data['required'] = required
+
+ def is_required(self):
+ return self.data['required']
+
+ def get_dependency_id(self):
+ return self.data['dependency']
+
+ def get_dependency_layer(self):
+ layerItem = None
+ try:
+ layerItem = self.index['layerItems'][self.get_dependency_id()]
+ except KeyError:
+ logger.error('Unable to find layerItems in index')
+ except IndexError:
+ logger.error('Unable to find layerId %s' % self.get_dependency_id())
+ return layerItem
+
+ def get_dependency_layerBranch(self):
+ layerBranch = None
+ try:
+ layerId = self.get_dependency_id()
+ branchId = self.get_layerbranch().get_branch_id()
+ layerBranch = self.index['layerBranches_layerId_branchId']["%s:%s" % (layerId, branchId)]
+ except KeyError:
+ logger.error('Unable to find layerBranches_layerId_branchId in index')
+ except IndexError:
+ logger.error("LayerBranch not found layerId %s -- BranchId %s" % (layerId, branchId))
+
+ return layerBranch
+
+
+class Recipe(LayerIndexItem_LayerBranch):
+ def define_data(self, id,
+ filename, filepath, pn, pv,
+ summary, description, section, license,
+ homepage, bugtracker, provides, bbclassextend,
+ inherits, blacklisted, layerbranch, updated=None):
+ self.data = {}
+ self.data['id'] = id
+ self.data['filename'] = filename
+ self.data['filepath'] = filepath
+ self.data['pn'] = pn
+ self.data['pv'] = pv
+ self.data['summary'] = summary
+ self.data['description'] = description
+ self.data['section'] = section
+ self.data['license'] = license
+ self.data['homepage'] = homepage
+ self.data['bugtracker'] = bugtracker
+ self.data['provides'] = provides
+ self.data['bbclassextend'] = bbclassextend
+ self.data['inherits'] = inherits
+ self.data['updated'] = updated or datetime.datetime.today().isoformat()
+ self.data['blacklisted'] = blacklisted
+ self.data['layerbranch'] = layerbranch
+
+ def get_filename(self):
+ return self.data['filename']
+
+ def get_filepath(self):
+ return self.data['filepath']
+
+ def get_fullpath(self):
+ return os.path.join(self.data['filepath'], self.data['filename'])
+
+ def get_summary(self):
+ return self.data['summary']
+
+ def get_description(self):
+ return self.data['description'].strip()
+
+ def get_section(self):
+ return self.data['section']
+
+ def get_pn(self):
+ return self.data['pn']
+
+ def get_pv(self):
+ return self.data['pv']
+
+ def get_license(self):
+ return self.data['license']
+
+ def get_homepage(self):
+ return self.data['homepage']
+
+ def get_bugtracker(self):
+ return self.data['bugtracker']
+
+ def get_provides(self):
+ return self.data['provides']
+
+ def get_updated(self):
+ return self.data['updated']
+
+ def get_inherits(self):
+ if 'inherits' not in self.data:
+ # Older indexes may not have this, so emulate it
+ if '-image-' in self.get_pn():
+ return 'image'
+ return self.data['inherits']
+
+
+class Machine(LayerIndexItem_LayerBranch):
+ def define_data(self, id,
+ name, description, layerbranch,
+ updated=None):
+ self.data = {}
+ self.data['id'] = id
+ self.data['name'] = name
+ self.data['description'] = description
+ self.data['layerbranch'] = layerbranch
+ self.data['updated'] = updated or datetime.datetime.today().isoformat()
+
+ def get_name(self):
+ return self.data['name']
+
+ def get_description(self):
+ return self.data['description'].strip()
+
+ def get_updated(self):
+ return self.data['updated']
+
+class Distro(LayerIndexItem_LayerBranch):
+ def define_data(self, id,
+ name, description, layerbranch,
+ updated=None):
+ self.data = {}
+ self.data['id'] = id
+ self.data['name'] = name
+ self.data['description'] = description
+ self.data['layerbranch'] = layerbranch
+ self.data['updated'] = updated or datetime.datetime.today().isoformat()
+
+ def get_name(self):
+ return self.data['name']
+
+ def get_description(self):
+ return self.data['description'].strip()
+
+ def get_updated(self):
+ return self.data['updated']
+
+# When performing certain actions, we may need to sort the data.
+# This will allow us to keep it consistent from run to run.
+def sort_entry(item):
+ newitem = item
+ try:
+ if type(newitem) == type(dict()):
+ newitem = OrderedDict(sorted(newitem.items(), key=lambda t: t[0]))
+ for index in newitem:
+ newitem[index] = sort_entry(newitem[index])
+ elif type(newitem) == type(list()):
+ newitem.sort(key=lambda obj: obj['id'])
+ for index, _ in enumerate(newitem):
+ newitem[index] = sort_entry(newitem[index])
+ except:
+ logger.error('Sort failed for item %s' % type(item))
+ pass
+
+ return newitem
diff --git a/lib/layers/layerindex/common.py b/lib/layers/layerindex/common.py
new file mode 100644
index 0000000..eb0cd75
--- /dev/null
+++ b/lib/layers/layerindex/common.py
@@ -0,0 +1,146 @@
+# Copyright (C) 2016-2017 Wind River Systems, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+# See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import argparse
+import logging
+import os
+import bb.msg
+
+logger = logging.getLogger('BitBake.layerindex.common')
+
+class LayerIndexError(Exception):
+ """LayerIndex loading error"""
+ def __init__(self, message):
+ self.msg = message
+ Exception.__init__(self, message)
+
+ def __str__(self):
+ return self.msg
+
+class IndexPlugin():
+ def __init__(self):
+ self.type = None
+
+ def init(self, lindex):
+ self.lindex = lindex
+
+ def plugin_type(self):
+ return self.type
+
+ def load_index(self, uri):
+ raise NotImplementedError('load_index is not implemented')
+
+ def store_index(self, uri):
+ raise NotImplementedError('store_index is not implemented')
+
+# Fetch something from a specific URL. This is specifically designed to
+# fetch data from a layer index or related element. It should NOT be
+# used to fetch recipe contents or similar.
+#
+# TODO: Handle BB_NO_NETWORK or allowed hosts, etc.
+#
+def fetch_url(url, username=None, password=None, debuglevel=0):
+ assert url is not None
+
+ import urllib
+ from urllib.request import urlopen, Request
+ from urllib.parse import urlparse
+
+ up = urlparse(url)
+
+ if username:
+ logger.debug(1, "Configuring authentication for %s..." % url)
+ password_mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()
+ password_mgr.add_password(None, "%s://%s" % (up.scheme, up.netloc), username, password)
+ handler = urllib.request.HTTPBasicAuthHandler(password_mgr)
+ opener = urllib.request.build_opener(handler, urllib.request.HTTPSHandler(debuglevel=debuglevel))
+ else:
+ opener = urllib.request.build_opener(urllib.request.HTTPSHandler(debuglevel=debuglevel))
+
+ urllib.request.install_opener(opener)
+
+ logger.debug(1, "Fetching %s (%s)..." % (url, ["without authentication", "with authentication"][not not username]))
+
+ try:
+ res = urlopen(Request(url, headers={'User-Agent': 'Mozilla/5.0 (bitbake/lib/layerindex)'}, unverifiable=True))
+ except urllib.error.HTTPError as e:
+ logger.debug(1, "HTTP Error: %s: %s" % (e.code, e.reason))
+ logger.debug(1, " Requested: %s" % (url))
+ logger.debug(1, " Actual: %s" % (e.geturl()))
+
+ if e.code == 404:
+ logger.debug(1, "Request not found.")
+ raise bb.fetch2.FetchError(e)
+ else:
+ logger.debug(1, "Headers:\n%s" % (e.headers))
+ raise bb.fetch2.FetchError(e)
+ except OSError as e:
+ error = 0
+ reason = ""
+
+ # Process base OSError first...
+ if hasattr(e, 'errno'):
+ error = e.errno
+ reason = e.strerror
+
+ # Process gaierror (socket error) subclass if available.
+ if hasattr(e, 'reason') and hasattr(e.reason, 'errno') and hasattr(e.reason, 'strerror'):
+ error = e.reason.errno
+ reason = e.reason.strerror
+ if error == -2:
+ raise bb.fetch2.FetchError(e)
+
+ if error and error != 0:
+ raise bb.fetch2.FetchError("Unable to fetch %s due to exception: [Error %s] %s" % (url, error, reason))
+ else:
+ raise bb.fetch2.FetchError("Unable to fetch %s due to OSError exception: %s" % (url, e))
+
+ finally:
+ logger.debug(1, "...fetching %s (%s), done." % (url, ["without authentication", "with authentication"][not not username]))
+
+ return res
+
+# Add a raw object of type lType to lindex[lname]
+def add_raw_element(lName, lType, rawObjs, lindex):
+ if lName not in rawObjs:
+ logger.debug(1, '%s not in loaded index' % lName)
+ return lindex
+
+ if lName not in lindex:
+ lindex[lName] = {}
+
+ for entry in rawObjs[lName]:
+ obj = lType(lindex, entry)
+ if obj.get_id() in lindex[lName]:
+ if lindex[lName][obj.get_id()] == obj:
+ continue
+ raise Exception('Conflict adding object %s(%s)' % (lName, obj.get_id()))
+ lindex[lName][obj.get_id()] = obj
+
+ return lindex
+
+# Add a layer index object to lindex[lName]
+def add_element(lName, Objs, lindex):
+ if lName not in lindex:
+ lindex[lName] = {}
+
+ for obj in Objs:
+ if obj.get_id() in lindex[lName]:
+ if lindex[lName][obj.get_id()] == obj:
+ continue
+ raise Exception('Conflict adding object %s(%s)' % (lName, obj.get_id()))
+ lindex[lName][obj.get_id()] = obj
+
+ return lindex
diff --git a/lib/layers/layerindex/cooker.py b/lib/layers/layerindex/cooker.py
new file mode 100644
index 0000000..229f668
--- /dev/null
+++ b/lib/layers/layerindex/cooker.py
@@ -0,0 +1,223 @@
+# Copyright (C) 2016-2017 Wind River Systems, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+# See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import logging
+import json
+
+from collections import OrderedDict, defaultdict
+
+from urllib.parse import unquote
+
+import layers.layerindex
+
+from layers.layerindex.common import IndexPlugin
+from layers.layerindex.common import LayerIndexError
+from layers.layerindex.common import add_element
+
+from layers.manager import _get_manager
+
+logger = logging.getLogger('BitBake.layerindex.cooker')
+
+import bb.utils
+
+def plugin_init(plugins):
+ return CookerPlugin()
+
+class CookerPlugin(IndexPlugin):
+ def __init__(self):
+ self.type = "cooker"
+ self.server_connection = None
+ self.ui_module = None
+ self.server = None
+
+ def load_index(self, ud, load):
+ """
+ Fetches layer information from a build configuration.
+
+ The return value is a dictionary containing API,
+ layer, branch, dependency, recipe, machine, distro, information.
+
+ ud path is ignored.
+ """
+
+ if ud.type != 'file':
+ raise bb.fetch2.FetchError('%s is not a supported protocol, only file, http and https are support.')
+
+ manager = _get_manager()
+ if not manager:
+ raise Exception('layer manager object has not been setup!')
+
+ localdata = self.lindex.data.createCopy()
+ # If the URL passed in branches, then we fake it...
+ if 'branch' in ud.parm:
+ localdata.setVar('LAYERSERIES_CORENAMES', ' '.join(ud.parm['branch'].split('OR')))
+
+ lindex = manager.load_bblayers(localdata)
+
+ lindex['CONFIG'] = {}
+ lindex['CONFIG']['TYPE'] = self.type
+ lindex['CONFIG']['URL'] = ud.url
+
+ if 'desc' in ud.parm:
+ lindex['CONFIG']['DESCRIPTION'] = unquote(ud.parm['desc'])
+ else:
+ lindex['CONFIG']['DESCRIPTION'] = ud.path
+
+ if 'cache' in ud.parm:
+ lindex['CONFIG']['CACHE'] = ud.parm['cache']
+
+ if 'branch' in ud.parm:
+ lindex['CONFIG']['BRANCH'] = ud.parm['branch']
+ else:
+ lindex['CONFIG']['BRANCH'] = localdata.getVar('LAYERSERIES_CORENAMES') or "HEAD"
+
+ # ("layerDependencies", layerindex.LayerDependency)
+ layerDependencyId = 0
+ if "layerDependencies" in load.split():
+ lindex['layerDependencies'] = {}
+ for layerBranchId in lindex['layerBranches']:
+ branchName = lindex['layerBranches'][layerBranchId].get_branch().get_name()
+ collection = lindex['layerBranches'][layerBranchId].get_collection()
+
+ def add_dependency(layerDependencyId, lindex, deps, required):
+ try:
+ depDict = bb.utils.explode_dep_versions2(deps)
+ except bb.utils.VersionStringException as vse:
+ bb.fatal('Error parsing LAYERDEPENDS_%s: %s' % (c, str(vse)))
+
+ for dep, oplist in list(depDict.items()):
+ # We need to search ourselves, so use the _ version...
+ depLayerBranch = self.lindex._find_collection(lindex, dep, branch=branchName)
+ if not depLayerBranch:
+ # Missing dependency?!
+ logger.error('Missing dependency %s (%s)' % (dep, branchName))
+ continue
+
+ # We assume that the oplist matches...
+ layerDependencyId += 1
+ layerDependency = layers.layerindex.LayerDependency(lindex, None)
+ layerDependency.define_data(id=layerDependencyId,
+ required=required, layerbranch=layerBranchId,
+ dependency=depLayerBranch.get_layer_id())
+
+ logger.debug(1, '%s requires %s' % (layerDependency.get_layer().get_name(), layerDependency.get_dependency_layer().get_name()))
+ lindex = add_element("layerDependencies", [layerDependency], lindex)
+
+ return layerDependencyId
+
+ deps = localdata.getVar("LAYERDEPENDS_%s" % collection)
+ if deps:
+ layerDependencyId = add_dependency(layerDependencyId, lindex, deps, True)
+
+ deps = localdata.getVar("LAYERRECOMMENDS_%s" % collection)
+ if deps:
+ layerDependencyId = add_dependency(layerDependencyId, lindex, deps, False)
+
+ # Need to load recipes here
+ recipeId = 0
+ if False and 'recipes' in load.split(): ## TODO NOT IMPLEMENTED
+ lindex['recipes'] = {}
+
+ ret = self.ui_module.main(self.server_connection.connection, self.server_connection.events, config_params)
+
+ all_versions = self._run_command('allProviders')
+
+ all_versions_list = defaultdict(list, all_versions)
+ for pn in all_versions_list:
+ for ((pe, pv, pr), fpath) in all_versions_list[pn]:
+ realfn = bb.cache.virtualfn2realfn(fpath)
+
+ filepath = os.path.dirname(realfn[0])
+ filename = os.path.basename(realfn[0])
+
+ # This is all HORRIBLY slow, and likely unnecessary
+ #dscon = self._run_command('parseRecipeFile', fpath, False, [])
+ #connector = myDataStoreConnector(self, dscon.dsindex)
+ #recipe_data = bb.data.init()
+ #recipe_data.setVar('_remote_data', connector)
+
+ #summary = recipe_data.getVar('SUMMARY')
+ #description = recipe_data.getVar('DESCRIPTION')
+ #section = recipe_data.getVar('SECTION')
+ #license = recipe_data.getVar('LICENSE')
+ #homepage = recipe_data.getVar('HOMEPAGE')
+ #bugtracker = recipe_data.getVar('BUGTRACKER')
+ #provides = recipe_data.getVar('PROVIDES')
+
+ layer = bb.utils.get_file_layer(realfn[0], self.config_data)
+
+ depBranchId = collection_layerbranch[layer]
+
+ recipeId += 1
+ recipe = layerindex.Recipe(lindex, None)
+ recipe.define_data(id=recipeId,
+ filename=filename, filepath=filepath,
+ pn=pn, pv=pv,
+ summary=pn, description=pn, section='?',
+ license='?', homepage='?', bugtracker='?',
+ provides='?', bbclassextend='?', inherits='?',
+ blacklisted='?', layerbranch=depBranchId)
+
+ lindex = addElement("recipes", [recipe], lindex)
+
+ # ("machines", layerindex.Machine)
+ machineId = 0
+ if 'machines' in load.split():
+ lindex['machines'] = {}
+
+ for layerBranchId in lindex['layerBranches']:
+ # load_bblayers uses the description to cache the actual path...
+ machine_path = lindex['layerBranches'][layerBranchId].getDescription()
+ machine_path = os.path.join(machine_path, 'conf/machine')
+ if os.path.isdir(machine_path):
+ for (dirpath, _, filenames) in os.walk(machine_path):
+ # Ignore subdirs...
+ if not dirpath.endswith('conf/machine'):
+ continue
+ for fname in filenames:
+ if fname.endswith('.conf'):
+ machineId += 1
+ machine = layers.layerindex.Machine(lindex, None)
+ machine.define_data(id=machineId, name=fname[:-5],
+ description=fname[:-5],
+ layerbranch=collection_layerbranch[entry])
+
+ lindex = add_element("machines", [machine], lindex)
+
+ # ("distros", layerindex.Distro)
+ distroId = 0
+ if 'distros' in load.split():
+ lindex['distros'] = {}
+
+ for layerBranchId in lindex['layerBranches']:
+ # load_bblayers uses the description to cache the actual path...
+ distro_path = lindex['layerBranches'][layerBranchId].getDescription()
+ distro_path = os.path.join(distro_path, 'conf/distro')
+ if os.path.isdir(distro_path):
+ for (dirpath, _, filenames) in os.walk(distro_path):
+ # Ignore subdirs...
+ if not dirpath.endswith('conf/distro'):
+ continue
+ for fname in filenames:
+ if fname.endswith('.conf'):
+ distroId += 1
+ distro = layers.layerindex.Distro(lindex, None)
+ distro.define_data(id=distroId, name=fname[:-5],
+ description=fname[:-5],
+ layerbranch=collection_layerbranch[entry])
+
+ lindex = add_element("distros", [distro], lindex)
+
+ return lindex
diff --git a/lib/layers/layerindex/restapi.py b/lib/layers/layerindex/restapi.py
new file mode 100644
index 0000000..d6d29a4
--- /dev/null
+++ b/lib/layers/layerindex/restapi.py
@@ -0,0 +1,348 @@
+# Copyright (C) 2016-2017 Wind River Systems, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+# See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import logging
+import bb.fetch2
+import json
+from urllib.parse import unquote
+
+import layers.layerindex
+
+from layers.layerindex.common import IndexPlugin
+from layers.layerindex.common import fetch_url
+from layers.layerindex.common import LayerIndexError
+from layers.layerindex.common import add_raw_element
+
+logger = logging.getLogger('BitBake.layers.layerindex.restapi')
+
+def plugin_init(plugins):
+ return RestApiPlugin()
+
+class RestApiPlugin(IndexPlugin):
+ def __init__(self):
+ self.type = "restapi"
+
+ def load_index(self, ud, load):
+ """
+ Fetches layer information from a local or remote layer index.
+
+ The return value is a dictionary containing API,
+ layer, branch, dependency, recipe, machine, distro, information.
+
+ url is the url to the rest api of the layer index, such as:
+ http://layers.openembedded.org/layerindex/api/
+
+ Or a local file...
+ """
+
+ if ud.type == 'file':
+ return self.load_index_file(ud, load)
+
+ if ud.type == 'http' or ud.type == 'https':
+ return self.load_index_web(ud, load)
+
+ raise bb.fetch2.FetchError('%s is not a supported protocol, only file, http and https are support.')
+
+
+ def load_index_file(self, ud, load):
+ """
+ Fetches layer information from a local file or directory.
+ The return value is a dictionary containing API,
+ layer, branch, dependency, recipe, machine, distro,
+ and template information.
+
+ ud is the parsed url to the local file or directory.
+ """
+ if not os.path.exists(ud.path):
+ raise FileNotFoundError(ud.path)
+
+ lindex = {}
+
+ lindex['CONFIG'] = {}
+ lindex['CONFIG']['TYPE'] = self.type
+ lindex['CONFIG']['URL'] = ud.url
+
+ if 'desc' in ud.parm:
+ lindex['CONFIG']['DESCRIPTION'] = unquote(ud.parm['desc'])
+ else:
+ lindex['CONFIG']['DESCRIPTION'] = ud.path
+
+ if 'cache' in ud.parm:
+ lindex['CONFIG']['CACHE'] = ud.parm['cache']
+
+ branches = None
+ if 'branch' in ud.parm:
+ branches = ud.parm['branch']
+ lindex['CONFIG']['BRANCH'] = branches
+
+
+ def load_cache(path, lindex, branches=None):
+ logger.debug(1, 'Loading json file %s' % path)
+ pindex = json.load(open(path, 'rt', encoding='utf-8'))
+
+ # Filter the branches on loaded files...
+ newpBranch = []
+ if branches:
+ for branch in (branches or "").split('OR'):
+ if 'branches' in pindex:
+ for br in pindex['branches']:
+ if br['name'] == branch:
+ newpBranch.append(br)
+ else:
+ if 'branches' in pindex:
+ newpBranch = pindex['branches']
+
+ if newpBranch:
+ lindex = add_raw_element('branches', layers.layerindex.Branch, { 'branches' : newpBranch }, lindex)
+ else:
+ logger.debug(1, 'No matching branchs (%s) in index file(s)' % branches)
+ # No matching branches.. return nothing...
+ return
+
+ for (lName, lType) in [("layerItems", layers.layerindex.LayerItem),
+ ("layerBranches", layers.layerindex.LayerBranch),
+ ("layerDependencies", layers.layerindex.LayerDependency),
+ ("recipes", layers.layerindex.Recipe),
+ ("machines", layers.layerindex.Machine),
+ ("distros", layers.layerindex.Distro)]:
+ if lName in pindex:
+ lindex = add_raw_element(lName, lType, pindex, lindex)
+
+
+ if not os.path.isdir(ud.path):
+ load_cache(ud.path, lindex, branches)
+ return lindex
+
+ logger.debug(1, 'Loading from dir %s...' % (ud.path))
+ for (dirpath, _, filenames) in os.walk(ud.path):
+ for filename in filenames:
+ if not filename.endswith('.json'):
+ continue
+ fpath = os.path.join(dirpath, filename)
+ load_cache(fpath, lindex, branches)
+
+ return lindex
+
+
+ def load_index_web(self, ud, load):
+ """
+ Fetches layer information from a remote layer index.
+ The return value is a dictionary containing API,
+ layer, branch, dependency, recipe, machine, distro,
+ and template information.
+
+ ud is the parsed url to the rest api of the layer index, such as:
+ http://layers.openembedded.org/layerindex/api/
+ """
+
+ def _get_json_response(apiurl=None, username=None, password=None, retry=True):
+ assert apiurl is not None
+
+ logger.debug(1, "fetching %s" % apiurl)
+
+ res = fetch_url(apiurl, username=username, password=password)
+
+ try:
+ parsed = json.loads(res.read().decode('utf-8'))
+ except ConnectionResetError:
+ if retry:
+ logger.debug(1, "%s: Connection reset by peer. Retrying..." % url)
+ parsed = _get_json_response(apiurl=apiurl, username=username, password=password, retry=False)
+ logger.debug(1, "%s: retry successful.")
+ else:
+ raise bb.fetch2.FetchError('%s: Connection reset by peer. Is there a firewall blocking your connection?' % apiurl)
+
+ return parsed
+
+ lindex = {}
+
+ lindex['CONFIG'] = {}
+ lindex['CONFIG']['TYPE'] = self.type
+ lindex['CONFIG']['URL'] = ud.url
+
+ if 'desc' in ud.parm:
+ lindex['CONFIG']['DESCRIPTION'] = unquote(ud.parm['desc'])
+ else:
+ lindex['CONFIG']['DESCRIPTION'] = ud.host
+
+ if 'cache' in ud.parm:
+ lindex['CONFIG']['CACHE'] = ud.parm['cache']
+
+ if 'branch' in ud.parm:
+ lindex['CONFIG']['BRANCH'] = ud.parm['branch']
+
+ try:
+ lindex['apilinks'] = _get_json_response(bb.fetch2.encodeurl( (ud.type, ud.host, ud.path, None, None, None) ),
+ username=ud.user, password=ud.pswd)
+ except Exception as e:
+ raise LayerIndexError("Unable to load layer index %s: %s" % (ud.url, e))
+
+ branches = None
+ if 'branch' in ud.parm and ud.parm['branch']:
+ branches = ud.parm['branch']
+
+
+ # Local raw index set...
+ pindex = {}
+
+ # Load the branches element
+ filter = ""
+ if branches:
+ filter = "?filter=name:%s" % branches
+
+ logger.debug(1, "Loading %s from %s" % ('branches', lindex['apilinks']['branches']))
+ pindex['branches'] = _get_json_response(lindex['apilinks']['branches'] + filter,
+ username=ud.user, password=ud.pswd)
+ if not pindex['branches']:
+ logger.debug(1, "No valid branches (%s) found at url %s." % (branches or "*", ud.url))
+ return lindex
+ lindex = add_raw_element("branches", layers.layerindex.Branch, pindex, lindex)
+
+
+ # Load all of the layerItems (these can not be easily filtered)
+ logger.debug(1, "Loading %s from %s" % ('layerItems', lindex['apilinks']['layerItems']))
+ pindex['layerItems'] = _get_json_response(lindex['apilinks']['layerItems'],
+ username=ud.user, password=ud.pswd)
+ if not pindex['layerItems']:
+ logger.debug(1, "No layers were found at url %s." % (ud.url))
+ return lindex
+ lindex = add_raw_element("layerItems", layers.layerindex.LayerItem, pindex, lindex)
+
+
+ # From this point on load the contents for each branch. Otherwise we
+ # could run into a timeout.
+ for branch in lindex['branches']:
+ filter = "?filter=branch__name:%s" % lindex['branches'][branch].get_name()
+
+ logger.debug(1, "Loading %s from %s" % ('layerBranches', lindex['apilinks']['layerBranches']))
+ pindex['layerBranches'] = _get_json_response(lindex['apilinks']['layerBranches'] + filter,
+ username=ud.user, password=ud.pswd)
+ if not pindex['layerBranches']:
+ logger.debug(1, "No valid layer branches (%s) found at url %s." % (branches or "*", ud.url))
+ return lindex
+ lindex = add_raw_element("layerBranches", layers.layerindex.LayerBranch, pindex, lindex)
+
+
+ # Load the rest, they all have a similar format
+ filter = "?filter=layerbranch__branch__name:%s" % lindex['branches'][branch].get_name()
+ for (lName, lType) in [("layerDependencies", layers.layerindex.LayerDependency),
+ ("recipes", layers.layerindex.Recipe),
+ ("machines", layers.layerindex.Machine),
+ ("distros", layers.layerindex.Distro)]:
+ if lName not in load.split():
+ continue
+ logger.debug(1, "Loading %s from %s" % (lName, lindex['apilinks'][lName]))
+ pindex[lName] = _get_json_response(lindex['apilinks'][lName] + filter,
+ username=ud.user, password=ud.pswd)
+ lindex = add_raw_element(lName, lType, pindex, lindex)
+
+
+ return lindex
+
+ def store_index(self, ud, lindex):
+ """
+ Store layer information into a local file/dir.
+
+ The return value is a dictionary containing API,
+ layer, branch, dependency, recipe, machine, distro, information.
+
+ ud is a parsed url to a directory or file. If the path is a
+ directory, we will split the files into one file per layer.
+ If the path is to a file (exists or not) the entire DB will be
+ dumped into that one file.
+ """
+
+ if ud.type != 'file':
+ raise NotImplementedError('Writing to anything but a file url is not implemented: %s' % ud.url)
+
+ # Write out to a single file, we have to sort the entries as we write
+ if not os.path.isdir(ud.path):
+ pindex = {}
+ for entry in lindex:
+ # Check for either locally added item or apilinks to ignore
+ if entry in lindex['CONFIG']['local'] or \
+ entry == 'apilinks':
+ continue
+ pindex[entry] = []
+ for objId in lindex[entry]:
+ pindex[entry].append(lindex[entry][objId].data)
+
+ bb.debug(1, 'Writing index to %s' % ud.path)
+ json.dump(layers.layerindex.sort_entry(pindex), open(ud.path, 'wt'), indent=4)
+ return
+
+ # Write out to a directory one file per layerBranch
+ try:
+ layerBranches = lindex['layerBranches']
+ except KeyError:
+ logger.error('No layerBranches to write.')
+ return
+
+ for layerBranchId in layerBranches:
+ pindex = {}
+
+ def filter_item(layerBranchId, objects):
+ filtered = []
+ for obj in lindex[objects]:
+ try:
+ if lindex[objects][obj].get_layerbranch_id() == layerBranchId:
+ filtered.append(lindex[objects][obj].data)
+ except AttributeError:
+ logger.debug(1, 'No obj.get_layerbranch_id(): %s' % objects)
+ # No simple filter method, just include it...
+ try:
+ filtered.append(lindex[objects][obj].data)
+ except AttributeError:
+ logger.debug(1, 'No obj.data: %s %s' % (objects, type(obj)))
+ filtered.append(obj)
+ return filtered
+
+ for entry in lindex:
+ # Skip local items, apilinks and items already processed
+ if entry in lindex['CONFIG']['local'] or \
+ entry == 'apilinks' or \
+ entry == 'branches' or \
+ entry == 'layerBranches' or \
+ entry == 'layerItems':
+ continue
+ pindex[entry] = filter_item(layerBranchId, entry)
+
+ # Add the layer we're processing as the first one...
+ pindex['branches'] = [layerBranches[layerBranchId].get_branch().data]
+ pindex['layerItems'] = [layerBranches[layerBranchId].get_layer().data]
+ pindex['layerBranches'] = [layerBranches[layerBranchId].data]
+
+ # We also need to include the layerbranch for any dependencies...
+ for layerDep in pindex['layerDependencies']:
+ layerDependency = layers.layerindex.LayerDependency(lindex, layerDep)
+
+ layerItem = layerDependency.get_dependency_layer()
+ layerBranch = layerDependency.get_dependency_layerBranch()
+
+ # We need to avoid duplicates...
+ if layerItem.data not in pindex['layerItems']:
+ pindex['layerItems'].append(layerItem.data)
+
+ if layerBranch.data not in pindex['layerBranches']:
+ pindex['layerBranches'].append(layerBranch.data)
+
+ # apply mirroring adjustments here....
+
+ fname = lindex['CONFIG']['DESCRIPTION'] + '__' + pindex['branches'][0]['name'] + '__' + pindex['layerItems'][0]['name']
+ fname = fname.translate(str.maketrans('/ ', '__'))
+ fpath = os.path.join(ud.path, fname)
+
+ bb.debug(1, 'Writing index to %s' % fpath + '.json')
+ json.dump(layers.layerindex.sort_entry(pindex), open(fpath + '.json', 'wt'), indent=4)
diff --git a/lib/layers/manager/__init__.py b/lib/layers/manager/__init__.py
new file mode 100644
index 0000000..3145924
--- /dev/null
+++ b/lib/layers/manager/__init__.py
@@ -0,0 +1,242 @@
+# Copyright (C) 2017 Wind River Systems, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+# See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import logging
+
+import layers.layerindex
+
+import tempfile
+
+import shutil
+
+logger = logging.getLogger('BitBake.layers.manager')
+
+class LayerManager():
+ def __init__(self, d, cooker):
+ _set_manager(self)
+
+ self.data = d
+ self.cooker = cooker
+
+ self.local_index = None # What is in the bblayers.conf
+ self.layers = None # The layers we want to setup (get_dependency format)
+ self.ignore = None # Specific items to ignore on fetch/unpack
+
+ self.plugins = []
+ bb.utils.load_plugins(logger, self.plugins, os.path.dirname(__file__))
+ for plugin in self.plugins:
+ if hasattr(plugin, 'init'):
+ plugin.init(self)
+
+ def get_plugin(self, type):
+ for plugin in self.plugins:
+ if hasattr(plugin, 'plugin_type'):
+ plugintype = plugin.plugin_type()
+ logger.debug(1, "Looking for LayerManagerPlugin - %s ? %s" % (plugintype, type))
+ if plugintype and plugintype == type:
+ return plugin
+ return None
+
+ def _run_command(self, command, path, default=None):
+ try:
+ result, _ = bb.process.run(command, cwd=path)
+ result = result.strip()
+ except bb.process.ExecutionError:
+ result = default
+ return result
+
+ def get_bitbake_info(self):
+ """Return a tuple of bitbake information"""
+
+ # Our path SHOULD be .../bitbake/lib/layers/manager/__init__.py
+ bb_path = os.path.dirname(__file__) # .../bitbake/lib/layers/manager/__init__.py
+ bb_path = os.path.dirname(bb_path) # .../bitbake/lib/layers/manager
+ bb_path = os.path.dirname(bb_path) # .../bitbake/lib/layers
+ bb_path = os.path.dirname(bb_path) # .../bitbake/lib
+ bb_path = os.path.dirname(bb_path) # .../bitbake
+ bb_path = self._run_command('git rev-parse --show-toplevel', os.path.dirname(__file__), default=bb_path)
+ bb_branch = self._run_command('git rev-parse --abbrev-ref HEAD', bb_path, default="<unknown>")
+ bb_rev = self._run_command('git rev-parse HEAD', bb_path, default="<unknown>")
+ for remotes in self._run_command('git remote -v', bb_path, default="").split("\n"):
+ remote = remotes.split("\t")[1].split(" ")[0]
+ if "(fetch)" == remotes.split("\t")[1].split(" ")[1]:
+ bb_remote = _handle_git_remote(remote)
+ break
+ else:
+ bb_remote = _handle_git_remote(bb_path)
+
+ return (bb_remote, bb_branch, bb_rev, bb_path)
+
+ def load_bblayers(self, d=None):
+ """Load the BBLAYERS and related collection information"""
+ if d is None:
+ d = self.data
+
+ default_branches = d.getVar('LAYERSERIES_CORENAMES') or "HEAD"
+
+ collections = d.getVar('BBFILE_COLLECTIONS')
+ layerconfs = d.varhistory.get_variable_items_files('BBFILE_COLLECTIONS', d)
+ bbfile_collections = {layer: os.path.dirname(os.path.dirname(path)) for layer, path in layerconfs.items()}
+
+ bblayers = d.getVar('BBLAYERS').split()
+
+ index = {}
+
+ branchId = 0
+ index['branches'] = {}
+
+ layerItemId = 0
+ index['layerItems'] = {}
+
+ layerBranchId = 0
+ index['layerBranches'] = {}
+
+ (_, bb_branch, _, _) = self.get_bitbake_info()
+
+ for branch in default_branches.split():
+ branchId += 1
+ index['branches'][branchId] = layers.layerindex.Branch(index, None)
+ index['branches'][branchId].define_data(branchId, branch, bb_branch)
+
+ for entry in collections.split():
+ layername = d.getVar('BBLAYERS_LAYERINDEX_NAME_%s' % entry) or os.path.basename(bbfile_collections[entry])
+
+ layerpath = bbfile_collections[entry]
+ layersubdir = ""
+
+ layerbasepath = self._run_command('git rev-parse --show-toplevel', layerpath, default=layerpath)
+ if os.path.abspath(layerpath) != os.path.abspath(layerbasepath):
+ layersubdir = os.path.abspath(layerpath)[len(layerbasepath) + 1:]
+
+ layerversion = d.getVar('LAYERVERSION_%s' % entry) or ""
+
+ layerbranch = self._run_command('git rev-parse --abbrev-ref HEAD', layerpath, default="<unknown>")
+ layerrev = self._run_command('git rev-parse HEAD', layerpath, default="<unknown>")
+
+ for remotes in self._run_command('git remote -v', layerpath, default="").split("\n"):
+ remote = remotes.split("\t")[1].split(" ")[0]
+ if "(fetch)" == remotes.split("\t")[1].split(" ")[1]:
+ layerurl = _handle_git_remote(remote)
+ break
+ else:
+ layerurl = _handle_git_remote(layerpath)
+
+ layerItemId += 1
+ index['layerItems'][layerItemId] = layers.layerindex.LayerItem(index, None)
+ index['layerItems'][layerItemId].define_data(layerItemId, layername, description=layerpath, vcs_url=layerurl)
+
+ for branchId in index['branches']:
+ layerBranchId += 1
+ index['layerBranches'][layerBranchId] = layers.layerindex.LayerBranch(index, None)
+ index['layerBranches'][layerBranchId].define_data(layerBranchId, entry, layerversion, layerItemId, branchId,
+ vcs_subdir=layersubdir, vcs_last_rev= layerrev, actual_branch=layerbranch)
+
+ return index
+
+ def get_clone_base_directory(self):
+ return self.data.getVar('BBLAYERS_FETCH_DIR')
+
+ # You are not allowed to have two of the same url, but different branches
+ def get_clone_directory(self, url):
+ baseDir = get_clone_base_directory()
+ if not baseDir:
+ return None
+ repo = os.path.basename(url)
+ return os.path.join(baseDir, repo)
+
+ def setup(self, layers, ignore=None):
+ """Setup the data structures for fetch and unpack and update bblayers.conf
+
+layers - format returned by LayerIndex.getDependencies
+ignore - a text string with a space deliminated list of layerItem names to ignore when downloading."""
+
+ self.local_index = self.load_bblayers()
+ self.layers = layers
+ self.ignore = (ignore or "").split()
+
+ self.index_fetcher = self.data.getVar('BBLAYERS_FETCHER_TYPE') or 'fetch2'
+
+ plugin = self.get_plugin(self.index_fetcher)
+ if not plugin:
+ raise NotImplementedError("layer manager plugin %s is not available" % index_fetcher)
+
+ plugin.setup()
+
+
+ def fetch(self):
+ """Fetch the layers from setup"""
+
+ plugin = self.get_plugin(self.index_fetcher)
+ if not plugin:
+ raise NotImplementedError("layer manager plugin %s is not available" % index_fetcher)
+
+ plugin.fetch()
+
+
+ def unpack(self):
+ """unpack the layers from fetch"""
+
+ plugin = self.get_plugin(self.index_fetcher)
+ if not plugin:
+ raise NotImplementedError("layer manager plugin %s is not available" % index_fetcher)
+
+ plugin.unpack()
+
+
+ def update_bblayers(self):
+ """Update the bblayers.conf file"""
+
+ plugin = self.get_plugin(self.index_fetcher)
+ if not plugin:
+ raise NotImplementedError("layer manager plugin %s is not available" % index_fetcher)
+
+ layerdirs = plugin.get_new_layers()
+
+ topdir = self.data.getVar('TOPDIR')
+ bblayers_conf = os.path.join(topdir, 'conf', 'bblayers.conf')
+ if not os.path.exists(bblayers_conf):
+ raise Exception('Unable to find bblayers.conf: %s' % bblayers_conf)
+
+ # Back up bblayers.conf to tempdir before we add layers
+ tempdir = tempfile.mkdtemp()
+ backup = tempdir + "/bblayers.conf.bak"
+ shutil.copy2(bblayers_conf, backup)
+
+ try:
+ notadded, _ = bb.utils.edit_bblayers_conf(bblayers_conf, layerdirs, None)
+ except Exception as e:
+ shutil.copy2(backup, bblayers_conf)
+ raise e
+ finally:
+ # Remove the back up copy of bblayers.conf
+ shutil.rmtree(tempdir)
+
+def _set_manager(manager):
+ global _manager
+ _manager = manager
+
+def _get_manager():
+ global _manager
+ return _manager
+
+def _handle_git_remote(remote):
+ if "://" not in remote:
+ if ':' in remote:
+ # This is assumed to be ssh
+ remote = "ssh://" + remote
+ else:
+ # This is assumed to be a file path
+ remote = "file://" + remote
+ return remote
diff --git a/lib/layers/manager/common.py b/lib/layers/manager/common.py
new file mode 100644
index 0000000..35f318d
--- /dev/null
+++ b/lib/layers/manager/common.py
@@ -0,0 +1,60 @@
+# Copyright (C) 2017 Wind River Systems, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+# See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import logging
+import os
+import bb.msg
+
+logger = logging.getLogger('BitBake.layerindex.common')
+
+class LayerManagerError(Exception):
+ """LayerManager error"""
+ def __init__(self, message):
+ self.msg = message
+ Exception.__init__(self, message)
+
+ def __str__(self):
+ return self.msg
+
+class DownloadPlugin():
+ def __init__(self, manager):
+ self.type = None
+
+ def init(self, manager):
+ self.manager = manager
+ self.data = self.manager.data
+
+ def plugin_type(self):
+ return self.type
+
+ def setup(self, layers, ignore):
+ """Setup the download"""
+
+ raise NotImplementedError('setup is not implemented')
+
+ def fetch(self):
+ """Fetch the layers from setup"""
+
+ raise NotImplementedError('fetch is not implemented')
+
+ def unpack(self):
+ """Fetch the layers from setup"""
+
+ raise NotImplementedError('fetch is not implemented')
+
+ def get_new_layers(self):
+ """Return a list of layers that we've unpacked"""
+
+ raise NotImplementedError('get_new_layers is not implemented')
diff --git a/lib/layers/manager/fetch2.py b/lib/layers/manager/fetch2.py
new file mode 100644
index 0000000..894a15d
--- /dev/null
+++ b/lib/layers/manager/fetch2.py
@@ -0,0 +1,210 @@
+# Copyright (C) 2017 Wind River Systems, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+# See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+from layers.manager.common import DownloadPlugin
+
+import logging
+import bb.fetch2
+
+import layers.layerindex
+
+logger = logging.getLogger('BitBake.layers.manager.fetch2')
+
+def plugin_init(plugins):
+ return Fetch2Plugin()
+
+class Fetch2Plugin(DownloadPlugin):
+ def __init__(self):
+ self.type = "fetch2"
+
+ # Dictionary of names and their replacements
+ self.rename = {}
+
+ def setup(self):
+ if not self.manager:
+ raise Exception('plugin was not initialized properly.')
+
+ # Reset this...
+ self.rename = {}
+
+ local_index = self.manager.local_index
+ layers = self.manager.layers
+ ignore = self.manager.ignore
+
+ def gen_src_uri(uri, branch):
+ if "://" not in uri:
+ type = 'file'
+ path = uri
+ else:
+ type, path = uri.split('://', 1)
+ return 'git://%s;protocol=%s;branch=%s;rev=%s' % (path, type, branch, branch)
+
+ # Format:
+ # url[<vcs_url>] = [ <src_uri>, layer_name1, layer_name2, ... layer_name3 ]
+ local_urls = {}
+ for layerBranchId in local_index['layerBranches']:
+ layerBranch = local_index['layerBranches'][layerBranchId]
+ url = layerBranch.get_layer().get_vcs_url()
+ if url not in local_urls:
+ local_urls[url] = [gen_src_uri(url, layerBranch.get_branch().get_name())]
+ if layerBranch.get_layer().get_name() not in local_urls[url]:
+ local_urls[url].append(layerBranch.get_layer().get_name())
+
+ remote_urls = {}
+ for deplayerbranch in layers:
+ layerBranch = layers[deplayerbranch][0]
+ url = layerBranch.get_layer().get_vcs_url()
+ if url not in local_urls:
+ if url not in remote_urls:
+ remote_urls[url] = [gen_src_uri(url, layerBranch.get_branch().get_name())]
+ if layerBranch.get_layer().get_name() not in remote_urls[url]:
+ remote_urls[url].append(layerBranch.get_layer().get_name())
+
+ self.local_urls = local_urls
+ self.remote_urls = remote_urls
+
+ #self.debug()
+
+ # define defaults here... and verify we're not going to break something!
+
+ fetchdir = self.manager.data.getVar('BBLAYERS_FETCH_DIR')
+
+ src_uri = ""
+ for url in remote_urls:
+ original = ondisk = os.path.join(fetchdir, os.path.basename(url).rstrip('.git'))
+ while os.path.exists(ondisk):
+ # Add '_' until it's unique
+ ondisk += "_"
+ if original != ondisk:
+ print('Rename destination from %s to %s' % (original, ondisk))
+ self.rename[original] = ondisk
+
+ src_uri += " " + remote_urls[url][0] + ";destsuffix=%s" % ondisk
+
+ self.src_uri = src_uri.strip()
+
+
+ def fetch(self):
+ src_uri = self.src_uri
+ if not src_uri:
+ # Nothing to fetch
+ self.fetcher = None
+ return
+
+ logger.plain('Fetching...')
+
+ remote_urls = self.remote_urls
+ localdata = self.manager.data.createCopy()
+
+ fetchdir = localdata.getVar('BBLAYERS_FETCH_DIR')
+
+ localdata.setVar('SRC_URI', src_uri)
+
+ localdata.delVar('MIRRORS')
+ localdata.delVar('PREMIRRORS')
+ mirrors = localdata.getVar('BBLAYERS_MIRRORS')
+ if mirrors:
+ localdata.setVar('PREMIRRORS', mirrors)
+
+ dldir = localdata.getVar('BBLAYERS_DL_DIR')
+ if not dldir:
+ dldir = os.path.join(fetchdir, '_layers')
+ localdata.setVar('DL_DIR', dldir)
+ localdata.setVar('FILESPATH', dldir)
+
+ if localdata.getVar('BB_NO_NETWORK') == '1' and localdata.getVar('BBLAYERS_ALLOW_NETWORK'):
+ localdata.delVar('BB_NO_NETWORK')
+
+ self.fetcher = bb.fetch2.Fetch(src_uri.split(), localdata, cache=False)
+ self.fetcher.download()
+
+ def unpack(self):
+ if not self.fetcher:
+ # Nothing to unpack
+ return
+
+ logger.plain('Unpacking...')
+ fetchdir = self.manager.data.getVar('BBLAYERS_FETCH_DIR')
+ self.fetcher.unpack(fetchdir)
+
+ def get_new_layers(self):
+ layers = self.manager.layers
+ remote_urls = self.remote_urls
+
+ fetchdir = self.manager.data.getVar('BBLAYERS_FETCH_DIR')
+
+ new_layers = []
+
+ local_layers = []
+ local_index = self.manager.local_index
+ for layerBranchId in local_index['layerBranches']:
+ layerBranch = local_index['layerBranches'][layerBranchId]
+ local_layers.append(layerBranch.get_layer().get_name())
+
+ for deplayerbranch in layers:
+ layerBranch = layers[deplayerbranch][0]
+ if layerBranch.get_layer().get_name() in local_layers:
+ # We already have it
+ continue
+
+ scm_path = os.path.join(fetchdir,
+ os.path.basename(layerBranch.get_layer().get_vcs_url())).rstrip('.git')
+ if scm_path in self.rename:
+ scm_path = self.rename[scm_path]
+
+ path = os.path.join(scm_path,
+ layerBranch.get_vcs_subdir() or "")
+
+ if not os.path.isdir(path):
+ raise Exception('Expected layer path %s does not exist.' % path)
+ continue
+
+ new_layers.append(path)
+
+ return new_layers
+
+
+ def debug(self):
+ #### Debugging
+ layers = self.manager.layers
+ remote_urls = self.remote_urls
+
+ logger.plain("%s %s %s" % ("Layer".ljust(24), "Git repository (branch)".ljust(54), "Subdirectory"))
+ logger.plain('=' * 105)
+
+ for deplayerbranch in layers:
+ layerBranch = layers[deplayerbranch][0]
+ layerDeps = layers[deplayerbranch][1:]
+
+ requiredby = []
+ recommendedby = []
+ for dep in layerDeps:
+ if dep.is_required():
+ requiredby.append(dep.get_layer().get_name())
+ else:
+ recommendedby.append(dep.get_layer().get_name())
+
+ required = False
+ if (not requiredby and not recommendedby) or requiredby:
+ required = True
+
+ logger.plain('%s%s %s %s' % (
+ [' ', '+'][layerBranch.get_layer().get_vcs_url() in remote_urls],
+ layerBranch.get_layer().get_name().ljust(24),
+ ("%s (%s)" % (layerBranch.get_layer().get_vcs_url(),
+ layerBranch.get_actual_branch())).ljust(55),
+ layerBranch.get_vcs_subdir()
+ ))
+ #### Debugging
--
1.8.3.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 2/5] bitbake-selftest: Add layers module tests
2017-07-28 15:37 [PATCH 0/5] Implement a new remote layer module Mark Hatle
2017-07-28 15:37 ` [PATCH 1/5] lib/layers: Initial layer and layer index implementeation Mark Hatle
@ 2017-07-28 15:37 ` Mark Hatle
2017-07-28 15:37 ` [PATCH 3/5] lib/bblayers: Add support for the new layer modules Mark Hatle
` (2 subsequent siblings)
4 siblings, 0 replies; 20+ messages in thread
From: Mark Hatle @ 2017-07-28 15:37 UTC (permalink / raw)
To: bitbake-devel; +Cc: toaster
The tests cover both the layers/layerindex and the layers/manager.
Both sets of tests included network connection components. They honor the
BB_SKIP_NETTESTS='yes' if networking is not available, to prevent test
failures.
Tests Implemented:
layers/layerindex:
- Branch, LayerItem, LayerBranch, LayerDependency, Recipe, Machine and
Distro objects
- LayerIndex setup using the layers.openembedded.org restapi
- LayerIndex storing and retrieving from a file
- LayerIndex verify dependency resolution ordering
layers/manager:
- manager basic functionality
- manager and fetch2 plugin setup/fetch/unpack of a remote repository
- fetch2 plugin generating a list of downloaded layers
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
bin/bitbake-selftest | 5 +-
lib/layers/tests/__init__.py | 0
lib/layers/tests/layerindex.py | 372 +++++++++++++++++++++++++++++++++++++++++
lib/layers/tests/manager.py | 155 +++++++++++++++++
4 files changed, 531 insertions(+), 1 deletion(-)
create mode 100644 lib/layers/tests/__init__.py
create mode 100644 lib/layers/tests/layerindex.py
create mode 100644 lib/layers/tests/manager.py
diff --git a/bin/bitbake-selftest b/bin/bitbake-selftest
index afe1603..9ad42ab 100755
--- a/bin/bitbake-selftest
+++ b/bin/bitbake-selftest
@@ -22,6 +22,7 @@ sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib
import unittest
try:
import bb
+ import layers
except RuntimeError as exc:
sys.exit(str(exc))
@@ -31,7 +32,9 @@ tests = ["bb.tests.codeparser",
"bb.tests.event",
"bb.tests.fetch",
"bb.tests.parse",
- "bb.tests.utils"]
+ "bb.tests.utils",
+ "layers.tests.layerindex",
+ "layers.tests.manager"]
for t in tests:
t = '.'.join(t.split('.')[:3])
diff --git a/lib/layers/tests/__init__.py b/lib/layers/tests/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/lib/layers/tests/layerindex.py b/lib/layers/tests/layerindex.py
new file mode 100644
index 0000000..c01bdb3
--- /dev/null
+++ b/lib/layers/tests/layerindex.py
@@ -0,0 +1,372 @@
+# Copyright (C) 2017 Wind River Systems, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+# See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import unittest
+import tempfile
+import os
+import bb
+
+from layers import layerindex
+
+import logging
+
+class LayersTest(unittest.TestCase):
+
+ def setUp(self):
+ self.origdir = os.getcwd()
+ self.d = bb.data.init()
+ self.tempdir = tempfile.mkdtemp()
+ self.dldir = os.path.join(self.tempdir, "download")
+ os.mkdir(self.dldir)
+ self.d.setVar("DL_DIR", self.dldir)
+ self.unpackdir = os.path.join(self.tempdir, "unpacked")
+ os.mkdir(self.unpackdir)
+ persistdir = os.path.join(self.tempdir, "persistdata")
+ self.d.setVar("PERSISTENT_DIR", persistdir)
+ self.logger = logging.getLogger("BitBake")
+
+ def tearDown(self):
+ os.chdir(self.origdir)
+ if os.environ.get("BB_TMPDIR_NOCLEAN") == "yes":
+ print("Not cleaning up %s. Please remove manually." % self.tempdir)
+ else:
+ bb.utils.prunedir(self.tempdir)
+
+class LayerObjectTest(LayersTest):
+ def setUp(self):
+ from layers.layerindex import Branch, LayerItem, LayerBranch, LayerDependency, Recipe, Machine, Distro
+
+ LayersTest.setUp(self)
+
+ self.index = {}
+
+ branchId = 0
+ layerItemId = 0
+ layerBranchId = 0
+ layerDependencyId = 0
+ recipeId = 0
+ machineId = 0
+ distroId = 0
+
+ self.index['branches'] = {}
+ self.index['layerItems'] = {}
+ self.index['layerBranches'] = {}
+ self.index['layerDependencies'] = {}
+ self.index['recipes'] = {}
+ self.index['machines'] = {}
+ self.index['distros'] = {}
+
+ branchId += 1
+ self.index['branches'][branchId] = Branch(self.index, None)
+ self.index['branches'][branchId].define_data(branchId,
+ 'test_branch', 'bb_test_branch')
+
+ layerItemId +=1
+ self.index['layerItems'][layerItemId] = LayerItem(self.index, None)
+ self.index['layerItems'][layerItemId].define_data(layerItemId, 'test_layerItem',
+ vcs_url='git://git_test_url/test_layerItem')
+
+ layerBranchId +=1
+ self.index['layerBranches'][layerBranchId] = LayerBranch(self.index, None)
+ self.index['layerBranches'][layerBranchId].define_data(layerBranchId,
+ 'test_collection', '99', layerItemId,
+ branchId)
+
+ recipeId += 1
+ self.index['recipes'][recipeId] = Recipe(self.index, None)
+ self.index['recipes'][recipeId].define_data(recipeId, 'test_git.bb',
+ 'recipes-test', 'test', 'git',
+ layerBranchId)
+
+ machineId += 1
+ self.index['machines'][machineId] = Machine(self.index, None)
+ self.index['machines'][machineId].define_data(machineId,
+ 'test_machine', 'test_machine',
+ layerBranchId)
+
+ distroId += 1
+ self.index['distros'][distroId] = Distro(self.index, None)
+ self.index['distros'][distroId].define_data(distroId,
+ 'test_distro', 'test_distro',
+ layerBranchId)
+
+ layerItemId +=1
+ self.index['layerItems'][layerItemId] = LayerItem(self.index, None)
+ self.index['layerItems'][layerItemId].define_data(layerItemId, 'test_layerItem 2',
+ vcs_url='git://git_test_url/test_layerItem')
+
+ layerBranchId +=1
+ self.index['layerBranches'][layerBranchId] = LayerBranch(self.index, None)
+ self.index['layerBranches'][layerBranchId].define_data(layerBranchId,
+ 'test_collection_2', '72', layerItemId,
+ branchId, actual_branch='some_other_branch')
+
+ layerDependencyId += 1
+ self.index['layerDependencies'][layerDependencyId] = LayerDependency(self.index, None)
+ self.index['layerDependencies'][layerDependencyId].define_data(layerDependencyId,
+ layerBranchId, 1)
+
+ layerDependencyId += 1
+ self.index['layerDependencies'][layerDependencyId] = LayerDependency(self.index, None)
+ self.index['layerDependencies'][layerDependencyId].define_data(layerDependencyId,
+ layerBranchId, 1, required=False)
+
+ def test_branch(self):
+ branch = self.index['branches'][1]
+ self.assertEqual(branch.get_id(), 1)
+ self.assertEqual(branch.get_name(), 'test_branch')
+ self.assertEqual(branch.get_short_description(), 'test_branch')
+ self.assertEqual(branch.get_bitbake_branch(), 'bb_test_branch')
+
+ def test_layerItem(self):
+ layerItem = self.index['layerItems'][1]
+ self.assertEqual(layerItem.get_id(), 1)
+ self.assertEqual(layerItem.get_name(), 'test_layerItem')
+ self.assertEqual(layerItem.get_summary(), 'test_layerItem')
+ self.assertEqual(layerItem.get_description(), 'test_layerItem')
+ self.assertEqual(layerItem.get_vcs_url(), 'git://git_test_url/test_layerItem')
+ self.assertEqual(layerItem.get_vcs_web_url(), None)
+ self.assertEqual(layerItem.get_vcs_web_tree_base_url(), None)
+ self.assertEqual(layerItem.get_vcs_web_file_base_url(), None)
+ self.assertTrue(layerItem.get_updated() != None)
+
+ layerItem = self.index['layerItems'][2]
+ self.assertEqual(layerItem.get_id(), 2)
+ self.assertEqual(layerItem.get_name(), 'test_layerItem 2')
+ self.assertEqual(layerItem.get_summary(), 'test_layerItem 2')
+ self.assertEqual(layerItem.get_description(), 'test_layerItem 2')
+ self.assertEqual(layerItem.get_vcs_url(), 'git://git_test_url/test_layerItem')
+ self.assertEqual(layerItem.get_vcs_web_url(), None)
+ self.assertEqual(layerItem.get_vcs_web_tree_base_url(), None)
+ self.assertEqual(layerItem.get_vcs_web_file_base_url(), None)
+ self.assertTrue(layerItem.get_updated() != None)
+
+ def test_layerBranch(self):
+ layerBranch = self.index['layerBranches'][1]
+ self.assertEqual(layerBranch.get_id(), 1)
+ self.assertEqual(layerBranch.get_collection(), 'test_collection')
+ self.assertEqual(layerBranch.get_version(), '99')
+ self.assertEqual(layerBranch.get_vcs_subdir(), '')
+ self.assertEqual(layerBranch.get_actual_branch(), 'test_branch')
+ self.assertTrue(layerBranch.get_updated() != None)
+ self.assertEqual(layerBranch.get_layer_id(), 1)
+ self.assertEqual(layerBranch.get_branch_id(), 1)
+ self.assertEqual(layerBranch.get_layer(), self.index['layerItems'][1])
+ self.assertEqual(layerBranch.get_branch(), self.index['branches'][1])
+
+ layerBranch = self.index['layerBranches'][2]
+ self.assertEqual(layerBranch.get_id(), 2)
+ self.assertEqual(layerBranch.get_collection(), 'test_collection_2')
+ self.assertEqual(layerBranch.get_version(), '72')
+ self.assertEqual(layerBranch.get_vcs_subdir(), '')
+ self.assertEqual(layerBranch.get_actual_branch(), 'some_other_branch')
+ self.assertTrue(layerBranch.get_updated() != None)
+ self.assertEqual(layerBranch.get_layer_id(), 2)
+ self.assertEqual(layerBranch.get_branch_id(), 1)
+ self.assertEqual(layerBranch.get_layer(), self.index['layerItems'][2])
+ self.assertEqual(layerBranch.get_branch(), self.index['branches'][1])
+
+ def test_layerDependency(self):
+ layerDependency = self.index['layerDependencies'][1]
+ self.assertEqual(layerDependency.get_id(), 1)
+ self.assertEqual(layerDependency.get_layerbranch_id(), 2)
+ self.assertEqual(layerDependency.get_layerbranch(), self.index['layerBranches'][2])
+ self.assertEqual(layerDependency.get_layer_id(), 2)
+ self.assertEqual(layerDependency.get_layer(), self.index['layerItems'][2])
+ self.assertTrue(layerDependency.is_required())
+ self.assertEqual(layerDependency.get_dependency_id(), 1)
+ self.assertEqual(layerDependency.get_dependency_layer(), self.index['layerItems'][1])
+ self.assertEqual(layerDependency.get_dependency_layerBranch(), self.index['layerBranches'][1])
+
+ # Previous check used the fall back method.. now use the faster method
+ # Create quick lookup layerBranches_layerId_branchId table
+ if 'layerBranches' in self.index:
+ # Create associated quick lookup indexes
+ self.index['layerBranches_layerId_branchId'] = {}
+ for layerBranchId in self.index['layerBranches']:
+ obj = self.index['layerBranches'][layerBranchId]
+ self.index['layerBranches_layerId_branchId']["%s:%s" % (obj.get_layer_id(), obj.get_branch_id())] = obj
+
+ layerDependency = self.index['layerDependencies'][2]
+ self.assertEqual(layerDependency.get_id(), 2)
+ self.assertEqual(layerDependency.get_layerbranch_id(), 2)
+ self.assertEqual(layerDependency.get_layerbranch(), self.index['layerBranches'][2])
+ self.assertEqual(layerDependency.get_layer_id(), 2)
+ self.assertEqual(layerDependency.get_layer(), self.index['layerItems'][2])
+ self.assertFalse(layerDependency.is_required())
+ self.assertEqual(layerDependency.get_dependency_id(), 1)
+ self.assertEqual(layerDependency.get_dependency_layer(), self.index['layerItems'][1])
+ self.assertEqual(layerDependency.get_dependency_layerBranch(), self.index['layerBranches'][1])
+
+ def test_recipe(self):
+ recipe = self.index['recipes'][1]
+ self.assertEqual(recipe.get_id(), 1)
+ self.assertEqual(recipe.get_layerbranch_id(), 1)
+ self.assertEqual(recipe.get_layerbranch(), self.index['layerBranches'][1])
+ self.assertEqual(recipe.get_layer_id(), 1)
+ self.assertEqual(recipe.get_layer(), self.index['layerItems'][1])
+ self.assertEqual(recipe.get_filename(), 'test_git.bb')
+ self.assertEqual(recipe.get_filepath(), 'recipes-test')
+ self.assertEqual(recipe.get_fullpath(), 'recipes-test/test_git.bb')
+ self.assertEqual(recipe.get_summary(), "")
+ self.assertEqual(recipe.get_description(), "")
+ self.assertEqual(recipe.get_section(), "")
+ self.assertEqual(recipe.get_pn(), 'test')
+ self.assertEqual(recipe.get_pv(), 'git')
+ self.assertEqual(recipe.get_license(), "")
+ self.assertEqual(recipe.get_homepage(), "")
+ self.assertEqual(recipe.get_bugtracker(), "")
+ self.assertEqual(recipe.get_provides(), "")
+ self.assertTrue(recipe.get_updated() != None)
+ self.assertEqual(recipe.get_inherits(), "")
+
+ def test_machine(self):
+ machine = self.index['machines'][1]
+ self.assertEqual(machine.get_id(), 1)
+ self.assertEqual(machine.get_layerbranch_id(), 1)
+ self.assertEqual(machine.get_layerbranch(), self.index['layerBranches'][1])
+ self.assertEqual(machine.get_layer_id(), 1)
+ self.assertEqual(machine.get_layer(), self.index['layerItems'][1])
+ self.assertEqual(machine.get_name(), 'test_machine')
+ self.assertEqual(machine.get_description(), 'test_machine')
+ self.assertTrue(machine.get_updated() != None)
+
+ def test_distro(self):
+ distro = self.index['distros'][1]
+ self.assertEqual(distro.get_id(), 1)
+ self.assertEqual(distro.get_layerbranch_id(), 1)
+ self.assertEqual(distro.get_layerbranch(), self.index['layerBranches'][1])
+ self.assertEqual(distro.get_layer_id(), 1)
+ self.assertEqual(distro.get_layer(), self.index['layerItems'][1])
+ self.assertEqual(distro.get_name(), 'test_distro')
+ self.assertEqual(distro.get_description(), 'test_distro')
+ self.assertTrue(distro.get_updated() != None)
+
+
+class LayerIndexWebRestApiTest(LayersTest):
+
+ if os.environ.get("BB_SKIP_NETTESTS") == "yes":
+ print("Unset BB_SKIP_NETTESTS to run network tests")
+ else:
+ def setUp(self):
+ LayersTest.setUp(self)
+ self.lindex = layerindex.LayerIndex(self.d)
+ self.lindex.load_layerindex('http://layers.openembedded.org/layerindex/api/;type=restapi;branch=morty', load='layerDependencies')
+
+ def test_layerindex_is_empty(self):
+ self.assertFalse(self.lindex.is_empty())
+
+ def test_layerindex_store_file(self):
+ self.lindex.store_layerindex('file://%s/file.json;type=restapi' % self.tempdir, self.lindex.lindex[0])
+
+ self.assertTrue(os.path.isfile('%s/file.json' % self.tempdir))
+
+ reload = layerindex.LayerIndex(self.d)
+ reload.load_layerindex('file://%s/file.json;type=restapi' % self.tempdir)
+
+ self.assertFalse(reload.is_empty())
+
+ # Calculate layerItems in original index that should NOT be in reload
+ layerItemNames = []
+ for itemId in self.lindex.lindex[0]['layerItems']:
+ layerItemNames.append(self.lindex.lindex[0]['layerItems'][itemId].get_name())
+
+ for layerBranchId in self.lindex.lindex[0]['layerBranches']:
+ layerItemNames.remove(self.lindex.lindex[0]['layerBranches'][layerBranchId].get_layer().get_name())
+
+ for itemId in reload.lindex[0]['layerItems']:
+ self.assertFalse(reload.lindex[0]['layerItems'][itemId].get_name() in layerItemNames)
+
+ # Compare the original to what we wrote...
+ for type in self.lindex.lindex[0]:
+ if type == 'apilinks' or \
+ type == 'layerItems' or \
+ type in self.lindex.lindex[0]['CONFIG']['local']:
+ continue
+ for id in self.lindex.lindex[0][type]:
+ self.logger.debug(1, "type %s" % (type))
+
+ self.assertTrue(id in reload.lindex[0][type])
+
+ self.logger.debug(1, "%s ? %s" % (self.lindex.lindex[0][type][id], reload.lindex[0][type][id]))
+
+ self.assertEqual(self.lindex.lindex[0][type][id], reload.lindex[0][type][id])
+
+ def test_layerindex_store_split(self):
+ self.lindex.store_layerindex('file://%s;type=restapi' % self.tempdir, self.lindex.lindex[0])
+
+ reload = layerindex.LayerIndex(self.d)
+ reload.load_layerindex('file://%s;type=restapi' % self.tempdir)
+
+ self.assertFalse(reload.is_empty())
+
+ for type in self.lindex.lindex[0]:
+ if type == 'apilinks' or \
+ type == 'layerItems' or \
+ type in self.lindex.lindex[0]['CONFIG']['local']:
+ continue
+ for id in self.lindex.lindex[0][type]:
+ self.logger.debug(1, "type %s" % (type))
+
+ self.assertTrue(id in reload.lindex[0][type])
+
+ self.logger.debug(1, "%s ? %s" % (self.lindex.lindex[0][type][id], reload.lindex[0][type][id]))
+
+ self.assertEqual(self.lindex.lindex[0][type][id], reload.lindex[0][type][id])
+
+ def test_dependency_resolution(self):
+ # Verify depth first searching...
+ (dependencies, invalidnames) = self.lindex.get_dependencies(names='meta-python')
+
+ first = True
+ for deplayerbranch in dependencies:
+ layerBranch = dependencies[deplayerbranch][0]
+ layerDeps = dependencies[deplayerbranch][1:]
+
+ if not first:
+ continue
+
+ first = False
+
+ # Top of the deps should be openembedded-core, since everything depends on it.
+ self.assertEquals(layerBranch.get_layer().get_name(), "openembedded-core")
+
+ # meta-python should cause an openembedded-core dependency, if not assert!
+ for dep in layerDeps:
+ if dep.get_layer().get_name() == 'meta-python':
+ break
+ else:
+ self.assetTrue(False)
+
+ # Only check the first element...
+ break
+ else:
+ if first:
+ # Empty list, this is bad.
+ self.assertTrue(False)
+
+ # Last dep should be the requested item
+ layerBranch = dependencies[deplayerbranch][0]
+ self.assertEquals(layerBranch.get_layer().get_name(), "meta-python")
+
+ def test_find_collection(self):
+ result = self.lindex.find_collection('core')
+
+ self.assertTrue(result != None)
+
+ def test_get_layerbranch(self):
+ result = self.lindex.get_layerbranch('openembedded-core')
+
+ self.assertTrue(result != None)
diff --git a/lib/layers/tests/manager.py b/lib/layers/tests/manager.py
new file mode 100644
index 0000000..ccdc44a
--- /dev/null
+++ b/lib/layers/tests/manager.py
@@ -0,0 +1,155 @@
+# Copyright (C) 2017 Wind River Systems, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+# See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import unittest
+import tempfile
+import os
+import bb
+
+from layers import manager
+
+import logging
+
+class LayersTest(unittest.TestCase):
+
+ def setUp(self):
+ self.origdir = os.getcwd()
+ self.d = bb.data.init()
+ self.tempdir = tempfile.mkdtemp()
+ self.dldir = os.path.join(self.tempdir, "download")
+ os.mkdir(self.dldir)
+ self.d.setVar("DL_DIR", self.dldir)
+ self.unpackdir = os.path.join(self.tempdir, "unpacked")
+ self.d.setVar("BBLAYERS_FETCH_DIR", self.unpackdir)
+ os.mkdir(self.unpackdir)
+ persistdir = os.path.join(self.tempdir, "persistdata")
+ self.d.setVar("PERSISTENT_DIR", persistdir)
+ self.logger = logging.getLogger("BitBake")
+ self.d.setVar('BBLAYERS', '')
+
+ def tearDown(self):
+ os.chdir(self.origdir)
+ if os.environ.get("BB_TMPDIR_NOCLEAN") == "yes":
+ print("Not cleaning up %s. Please remove manually." % self.tempdir)
+ else:
+ bb.utils.prunedir(self.tempdir)
+
+class LayerManagerTest(LayersTest):
+ def setUp(self):
+ LayersTest.setUp(self)
+
+ self.manager = manager.LayerManager(self.d, None)
+
+ def test_get_bitbake_info(self):
+ (bb_remote, bb_branch, bb_rev, bb_path) = self.manager.get_bitbake_info()
+
+ self.assertTrue("://" in bb_remote)
+
+ us = os.path.dirname(__file__) # bitbake/lib/layers/tests
+ us = os.path.dirname(us) # bitbake/lib/layers
+ us = os.path.dirname(us) # bitbake/lib
+ us = os.path.dirname(us) # bitbake
+
+ self.assertEqual(bb_path, us)
+
+ def test_load_bblayers(self):
+
+ self.d.setVar('LAYERSERIES_CORENAMES', 'under_test')
+
+ self.d.setVar('BBFILE_COLLECTIONS', 'test1')
+ self.d.appendVar('BBFILE_COLLECTIONS', ' test2')
+ self.d.appendVar('BBFILE_COLLECTIONS', ' test3')
+
+ self.d.setVar('BBLAYERS', '%s/test1_layer %s/test2_layer %s/test3_layer'
+ % (self.tempdir, self.tempdir, self.tempdir))
+
+ self.d.setVar('BBLAYERS_LAYERINDEX_NAME_test1', 'oe-test-layer')
+
+ self.d.setVar('LAYERVERSION_test1', '1')
+ self.d.setVar('LAYERVERSION_test2', '2')
+ self.d.setVar('LAYERVERSION_test3', '3')
+
+ index = self.manager.load_bblayers()
+
+ self.assertEqual(index['branches'][1].get_name(), 'under_test')
+
+ layerBranch = index['layerBranches'][1]
+ self.assertEqual(layerBranch.get_collection(), 'test1')
+ self.assertEqual(layerBranch.get_version(), '1')
+ self.assertEqual(layerBranch.get_layer().get_name(), 'oe-test-layer')
+
+ layerBranch = index['layerBranches'][2]
+ self.assertEqual(layerBranch.get_collection(), 'test2')
+ self.assertEqual(layerBranch.get_version(), '2')
+ self.assertEqual(layerBranch.get_layer().get_name(), 'test2')
+
+ layerBranch = index['layerBranches'][3]
+ self.assertEqual(layerBranch.get_collection(), 'test3')
+ self.assertEqual(layerBranch.get_version(), '3')
+ self.assertEqual(layerBranch.get_layer().get_name(), 'test3')
+
+ def test_clone_directory(self):
+ self.assertEqual(self.manager.get_clone_directory('git://foo/foobar'),
+ os.path.join(self.unpackdir, 'foobar'))
+
+ if os.environ.get("BB_SKIP_NETTESTS") == "yes":
+ print("Unset BB_SKIP_NETTESTS to run network tests")
+ else:
+ def test_fetch2(self):
+ from collections import OrderedDict
+
+ from layers.layerindex import Branch, LayerItem, LayerBranch
+
+ branchId = 0
+ layerItemId = 0
+ layerBranchId = 0
+
+ index = {}
+ index['branches'] = {}
+ index['layerItems'] = {}
+ index['layerBranches'] = {}
+
+ branchId += 1
+ index['branches'][branchId] = Branch(index, None)
+ index['branches'][branchId].define_data(branchId,
+ 'master', 'master')
+
+ layerItemId +=1
+ index['layerItems'][layerItemId] = LayerItem(index, None)
+ index['layerItems'][layerItemId].define_data(layerItemId, 'meta-gplv2',
+ vcs_url='git://git.yoctoproject.org/meta-gplv2')
+
+ layerBranchId +=1
+ index['layerBranches'][layerBranchId] = LayerBranch(index, None)
+ index['layerBranches'][layerBranchId].define_data(layerBranchId,
+ 'gplv2', '1', layerItemId, branchId)
+
+
+ dependencies = OrderedDict()
+ layerBranch = index['layerBranches'][layerBranchId]
+ dependencies[layerBranch.get_layer().get_name()] = [layerBranch]
+
+ self.d.setVar('BBLAYERS_FETCHER_TYPE', 'fetch2')
+ self.manager.setup(dependencies)
+ self.manager.fetch()
+ self.manager.unpack()
+
+ fetchplugin = self.manager.get_plugin(self.manager.index_fetcher)
+ newlayers = fetchplugin.get_new_layers()
+
+ self.assertFalse(not newlayers)
+
+ self.assertTrue(os.path.isdir(newlayers[0]))
+ self.assertTrue(os.path.isfile(os.path.join(newlayers[0],'conf/layer.conf')))
--
1.8.3.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 3/5] lib/bblayers: Add support for the new layer modules
2017-07-28 15:37 [PATCH 0/5] Implement a new remote layer module Mark Hatle
2017-07-28 15:37 ` [PATCH 1/5] lib/layers: Initial layer and layer index implementeation Mark Hatle
2017-07-28 15:37 ` [PATCH 2/5] bitbake-selftest: Add layers module tests Mark Hatle
@ 2017-07-28 15:37 ` Mark Hatle
2017-07-28 16:04 ` Mark Hatle
2017-07-28 15:37 ` [PATCH 4/5] toaster/orm/management/commands/lsupdates.py: Use new layerindex module Mark Hatle
2017-07-28 15:37 ` [PATCH 5/5] bitbake-layers: disable parsing for layerindex commands Mark Hatle
4 siblings, 1 reply; 20+ messages in thread
From: Mark Hatle @ 2017-07-28 15:37 UTC (permalink / raw)
To: bitbake-devel; +Cc: toaster
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
lib/bblayers/layerindex.py | 313 +++++++++++---------------------------
lib/layers/layerindex/__init__.py | 28 +++-
lib/layers/layerindex/cooker.py | 2 +-
lib/layers/layerindex/restapi.py | 95 +++++++-----
lib/layers/manager/__init__.py | 59 ++++---
5 files changed, 210 insertions(+), 287 deletions(-)
diff --git a/lib/bblayers/layerindex.py b/lib/bblayers/layerindex.py
index 506c110..d1dac0b 100644
--- a/lib/bblayers/layerindex.py
+++ b/lib/bblayers/layerindex.py
@@ -1,10 +1,10 @@
+import layers.manager
+import layers.layerindex
+
import argparse
-import http.client
-import json
import logging
import os
import subprocess
-import urllib.parse
from bblayers.action import ActionPlugin
@@ -21,235 +21,108 @@ class LayerIndexPlugin(ActionPlugin):
This class inherits ActionPlugin to get do_add_layer.
"""
- def get_json_data(self, apiurl):
- proxy_settings = os.environ.get("http_proxy", None)
- conn = None
- _parsedurl = urllib.parse.urlparse(apiurl)
- path = _parsedurl.path
- query = _parsedurl.query
-
- def parse_url(url):
- parsedurl = urllib.parse.urlparse(url)
- if parsedurl.netloc[0] == '[':
- host, port = parsedurl.netloc[1:].split(']', 1)
- if ':' in port:
- port = port.rsplit(':', 1)[1]
- else:
- port = None
- else:
- if parsedurl.netloc.count(':') == 1:
- (host, port) = parsedurl.netloc.split(":")
- else:
- host = parsedurl.netloc
- port = None
- return (host, 80 if port is None else int(port))
-
- if proxy_settings is None:
- host, port = parse_url(apiurl)
- conn = http.client.HTTPConnection(host, port)
- conn.request("GET", path + "?" + query)
- else:
- host, port = parse_url(proxy_settings)
- conn = http.client.HTTPConnection(host, port)
- conn.request("GET", apiurl)
-
- r = conn.getresponse()
- if r.status != 200:
- raise Exception("Failed to read " + path + ": %d %s" % (r.status, r.reason))
- return json.loads(r.read().decode())
-
- def get_layer_deps(self, layername, layeritems, layerbranches, layerdependencies, branchnum, selfname=False):
- def layeritems_info_id(items_name, layeritems):
- litems_id = None
- for li in layeritems:
- if li['name'] == items_name:
- litems_id = li['id']
- break
- return litems_id
-
- def layerbranches_info(items_id, layerbranches):
- lbranch = {}
- for lb in layerbranches:
- if lb['layer'] == items_id and lb['branch'] == branchnum:
- lbranch['id'] = lb['id']
- lbranch['vcs_subdir'] = lb['vcs_subdir']
- break
- return lbranch
-
- def layerdependencies_info(lb_id, layerdependencies):
- ld_deps = []
- for ld in layerdependencies:
- if ld['layerbranch'] == lb_id and not ld['dependency'] in ld_deps:
- ld_deps.append(ld['dependency'])
- if not ld_deps:
- logger.error("The dependency of layerDependencies is not found.")
- return ld_deps
-
- def layeritems_info_name_subdir(items_id, layeritems):
- litems = {}
- for li in layeritems:
- if li['id'] == items_id:
- litems['vcs_url'] = li['vcs_url']
- litems['name'] = li['name']
- break
- return litems
-
- if selfname:
- selfid = layeritems_info_id(layername, layeritems)
- lbinfo = layerbranches_info(selfid, layerbranches)
- if lbinfo:
- selfsubdir = lbinfo['vcs_subdir']
- else:
- logger.error("%s is not found in the specified branch" % layername)
- return
- selfurl = layeritems_info_name_subdir(selfid, layeritems)['vcs_url']
- if selfurl:
- return selfurl, selfsubdir
- else:
- logger.error("Cannot get layer %s git repo and subdir" % layername)
- return
- ldict = {}
- itemsid = layeritems_info_id(layername, layeritems)
- if not itemsid:
- return layername, None
- lbid = layerbranches_info(itemsid, layerbranches)
- if lbid:
- lbid = layerbranches_info(itemsid, layerbranches)['id']
- else:
- logger.error("%s is not found in the specified branch" % layername)
- return None, None
- for dependency in layerdependencies_info(lbid, layerdependencies):
- lname = layeritems_info_name_subdir(dependency, layeritems)['name']
- lurl = layeritems_info_name_subdir(dependency, layeritems)['vcs_url']
- lsubdir = layerbranches_info(dependency, layerbranches)['vcs_subdir']
- ldict[lname] = lurl, lsubdir
- return None, ldict
-
- def get_fetch_layer(self, fetchdir, url, subdir, fetch_layer):
- layername = self.get_layer_name(url)
- if os.path.splitext(layername)[1] == '.git':
- layername = os.path.splitext(layername)[0]
- repodir = os.path.join(fetchdir, layername)
- layerdir = os.path.join(repodir, subdir)
- if not os.path.exists(repodir):
- if fetch_layer:
- result = subprocess.call('git clone %s %s' % (url, repodir), shell = True)
- if result:
- logger.error("Failed to download %s" % url)
- return None, None
- else:
- return layername, layerdir
- else:
- logger.plain("Repository %s needs to be fetched" % url)
- return layername, layerdir
- elif os.path.exists(layerdir):
- return layername, layerdir
- else:
- logger.error("%s is not in %s" % (url, subdir))
- return None, None
-
def do_layerindex_fetch(self, args):
"""Fetches a layer from a layer index along with its dependent layers, and adds them to conf/bblayers.conf.
"""
- apiurl = self.tinfoil.config_data.getVar('BBLAYERS_LAYERINDEX_URL')
- if not apiurl:
+
+ def _construct_url(baseurl, branch):
+ if baseurl[-1] != '/':
+ baseurl += '/'
+ baseurl += "api/"
+ baseurl += ";type=restapi"
+
+ if branch:
+ baseurl += ";branch=%s" % branch
+
+ return baseurl
+
+
+ # General URL to use based on standard setting
+ indexurl = self.tinfoil.config_data.getVar('BBLAYERS_LAYERINDEX_URL')
+
+ if not indexurl:
logger.error("Cannot get BBLAYERS_LAYERINDEX_URL")
return 1
- else:
- if apiurl[-1] != '/':
- apiurl += '/'
- apiurl += "api/"
- apilinks = self.get_json_data(apiurl)
- branches = self.get_json_data(apilinks['branches'])
-
- branchnum = 0
- for branch in branches:
- if branch['name'] == args.branch:
- branchnum = branch['id']
- break
- if branchnum == 0:
- validbranches = ', '.join([branch['name'] for branch in branches])
- logger.error('Invalid layer branch name "%s". Valid branches: %s' % (args.branch, validbranches))
+
+ layerManager = layers.manager.LayerManager(self.tinfoil.config_data, self.tinfoil.cooker)
+
+ remoteIndex = layers.layerindex.LayerIndex(self.tinfoil.config_data)
+
+ # Set the default...
+ branch = self.tinfoil.config_data.getVar('LAYERSERIES_CORENAMES') or 'master'
+ if args.branch:
+ branch = args.branch
+
+ logger.debug(1, 'Trying branch %s' % branch)
+ try:
+ remoteIndex.load_layerindex(_construct_url(indexurl, branch))
+ except Exception:
+ if branch == args.branch:
+ logger.error('Branch %s is not available' % branch)
+ return 1
+ logger.debug(1, 'Falling back to branch master')
+ remoteIndex.load_layerindex(_construct_url(indexurl, 'master'), reload=True)
+
+ if remoteIndex.is_empty():
return 1
- ignore_layers = []
- for collection in self.tinfoil.config_data.getVar('BBFILE_COLLECTIONS').split():
- lname = self.tinfoil.config_data.getVar('BBLAYERS_LAYERINDEX_NAME_%s' % collection)
- if lname:
- ignore_layers.append(lname)
+ # If we want the display to include already downloaded
+ # keep the following line, otherwise comment out.
+ cookerIndex = layers.layerindex.LayerIndex(self.tinfoil.config_data)
+ cookerIndex.load_layerindex('file://local;type=cooker', load='layerDependencies')
+
+ lIndex = cookerIndex + remoteIndex
+ ignore_layers = []
if args.ignore:
ignore_layers.extend(args.ignore.split(','))
- layeritems = self.get_json_data(apilinks['layerItems'])
- layerbranches = self.get_json_data(apilinks['layerBranches'])
- layerdependencies = self.get_json_data(apilinks['layerDependencies'])
- invaluenames = []
- repourls = {}
- printlayers = []
-
- def query_dependencies(layers, layeritems, layerbranches, layerdependencies, branchnum):
- depslayer = []
- for layername in layers:
- invaluename, layerdict = self.get_layer_deps(layername, layeritems, layerbranches, layerdependencies, branchnum)
- if layerdict:
- repourls[layername] = self.get_layer_deps(layername, layeritems, layerbranches, layerdependencies, branchnum, selfname=True)
- for layer in layerdict:
- if not layer in ignore_layers:
- depslayer.append(layer)
- printlayers.append((layername, layer, layerdict[layer][0], layerdict[layer][1]))
- if not layer in ignore_layers and not layer in repourls:
- repourls[layer] = (layerdict[layer][0], layerdict[layer][1])
- if invaluename and not invaluename in invaluenames:
- invaluenames.append(invaluename)
- return depslayer
-
- depslayers = query_dependencies(args.layername, layeritems, layerbranches, layerdependencies, branchnum)
- while depslayers:
- depslayer = query_dependencies(depslayers, layeritems, layerbranches, layerdependencies, branchnum)
- depslayers = depslayer
- if invaluenames:
- for invaluename in invaluenames:
+ layernames = ' '.join(args.layername)
+ (dependencies, invalidnames) = lIndex.get_dependencies(names=layernames, ignores=ignore_layers)
+
+ if invalidnames:
+ for invaluename in invalidnames:
logger.error('Layer "%s" not found in layer index' % invaluename)
return 1
- logger.plain("%s %s %s %s" % ("Layer".ljust(19), "Required by".ljust(19), "Git repository".ljust(54), "Subdirectory"))
- logger.plain('=' * 115)
- for layername in args.layername:
- layerurl = repourls[layername]
- logger.plain("%s %s %s %s" % (layername.ljust(20), '-'.ljust(20), layerurl[0].ljust(55), layerurl[1]))
- printedlayers = []
- for layer, dependency, gitrepo, subdirectory in printlayers:
- if dependency in printedlayers:
- continue
- logger.plain("%s %s %s %s" % (dependency.ljust(20), layer.ljust(20), gitrepo.ljust(55), subdirectory))
- printedlayers.append(dependency)
-
- if repourls:
- fetchdir = self.tinfoil.config_data.getVar('BBLAYERS_FETCH_DIR')
- if not fetchdir:
- logger.error("Cannot get BBLAYERS_FETCH_DIR")
- return 1
- if not os.path.exists(fetchdir):
- os.makedirs(fetchdir)
- addlayers = []
- for repourl, subdir in repourls.values():
- name, layerdir = self.get_fetch_layer(fetchdir, repourl, subdir, not args.show_only)
- if not name:
- # Error already shown
- return 1
- addlayers.append((subdir, name, layerdir))
- if not args.show_only:
- for subdir, name, layerdir in set(addlayers):
- if os.path.exists(layerdir):
- if subdir:
- logger.plain("Adding layer \"%s\" to conf/bblayers.conf" % subdir)
- else:
- logger.plain("Adding layer \"%s\" to conf/bblayers.conf" % name)
- localargs = argparse.Namespace()
- localargs.layerdir = layerdir
- self.do_add_layer(localargs)
+ logger.plain("%s %s %s" % ("Layer".ljust(49), "Git repository (branch)".ljust(54), "Subdirectory"))
+ logger.plain('=' * 125)
+
+ for deplayerbranch in dependencies:
+ layerBranch = dependencies[deplayerbranch][0]
+
+ # This is the local content, uncomment to hide local
+ # layers from the display.
+ #if layerBranch.index['CONFIG']['TYPE'] == 'cooker':
+ # continue
+
+ layerDeps = dependencies[deplayerbranch][1:]
+
+ requiredby = []
+ recommendedby = []
+ for dep in layerDeps:
+ if dep.is_required():
+ requiredby.append(dep.get_layer().get_name())
else:
- break
+ recommendedby.append(dep.get_layer().get_name())
+
+ logger.plain('%s %s %s' % (("%s:%s:%s" %
+ (layerBranch.index['CONFIG']['DESCRIPTION'],
+ layerBranch.get_branch().get_name(),
+ layerBranch.get_layer().get_name())).ljust(50),
+ ("%s (%s)" % (layerBranch.get_layer().get_vcs_url(),
+ layerBranch.get_actual_branch())).ljust(55),
+ layerBranch.get_vcs_subdir()
+ ))
+ if requiredby:
+ logger.plain(' required by: %s' % ' '.join(requiredby))
+ if recommendedby:
+ logger.plain(' recommended by: %s' % ' '.join(recommendedby))
+
+ if args.show_only != True:
+ layerManager.setup(dependencies)
+ layerManager.fetch()
+ layerManager.unpack()
+ layerManager.update_bblayers()
def do_layerindex_show_depends(self, args):
"""Find layer dependencies from layer index.
@@ -261,10 +134,10 @@ class LayerIndexPlugin(ActionPlugin):
def register_commands(self, sp):
parser_layerindex_fetch = self.add_command(sp, 'layerindex-fetch', self.do_layerindex_fetch)
parser_layerindex_fetch.add_argument('-n', '--show-only', help='show dependencies and do nothing else', action='store_true')
- parser_layerindex_fetch.add_argument('-b', '--branch', help='branch name to fetch (default %(default)s)', default='master')
+ parser_layerindex_fetch.add_argument('-b', '--branch', help='branch name to fetch (default %(default)s)')
parser_layerindex_fetch.add_argument('-i', '--ignore', help='assume the specified layers do not need to be fetched/added (separate multiple layers with commas, no spaces)', metavar='LAYER')
parser_layerindex_fetch.add_argument('layername', nargs='+', help='layer to fetch')
parser_layerindex_show_depends = self.add_command(sp, 'layerindex-show-depends', self.do_layerindex_show_depends)
- parser_layerindex_show_depends.add_argument('-b', '--branch', help='branch name to fetch (default %(default)s)', default='master')
+ parser_layerindex_show_depends.add_argument('-b', '--branch', help='branch name to fetch (default %(default)s)')
parser_layerindex_show_depends.add_argument('layername', nargs='+', help='layer to query')
diff --git a/lib/layers/layerindex/__init__.py b/lib/layers/layerindex/__init__.py
index 0c7c483..beb7df9 100644
--- a/lib/layers/layerindex/__init__.py
+++ b/lib/layers/layerindex/__init__.py
@@ -407,7 +407,7 @@ invalid is just a list of dependencies that were not found.
if names:
for name in names.split():
- if name in ignores:
+ if ignores and name in ignores:
continue
# Since we don't have a branch, we have to just find the first
@@ -441,7 +441,7 @@ invalid is just a list of dependencies that were not found.
for layerBranch in layerBranches:
name = layerBranch.get_layer().get_name()
# Do we ignore it?
- if name in ignores:
+ if ignores and name in ignores:
continue
if 'layerDependencies_layerBranchId' not in layerBranch.index:
@@ -469,7 +469,7 @@ invalid is just a list of dependencies that were not found.
depLayerBranch = rdepLayerBranch
# Is this dependency on the list to be ignored?
- if depLayerBranch.get_layer().get_name() in ignores:
+ if ignores and depLayerBranch.get_layer().get_name() in ignores:
continue
# Previously found dependencies have been processed, as
@@ -823,7 +823,19 @@ class LayerDependency(LayerIndexItem_LayerBranch):
branchId = self.get_layerbranch().get_branch_id()
layerBranch = self.index['layerBranches_layerId_branchId']["%s:%s" % (layerId, branchId)]
except KeyError:
- logger.error('Unable to find layerBranches_layerId_branchId in index')
+ logger.warning('Unable to find layerBranches_layerId_branchId in index')
+
+ # We don't have a quick lookup index, doing it the slower way...
+ layerId = self.get_dependency_id()
+ branchId = self.get_layerbranch().get_branch_id()
+ for layerBranchId in self.index['layerBranches']:
+ layerBranch = self.index['layerBranches'][layerBranchId]
+ if layerBranch.get_layer_id() == layerId and \
+ layerBranch.get_branch_id() == branchId:
+ break
+ else:
+ logger.error("LayerBranch not found layerId %s -- BranchId %s" % (layerId, branchId))
+ layerBranch = None
except IndexError:
logger.error("LayerBranch not found layerId %s -- BranchId %s" % (layerId, branchId))
@@ -832,10 +844,10 @@ class LayerDependency(LayerIndexItem_LayerBranch):
class Recipe(LayerIndexItem_LayerBranch):
def define_data(self, id,
- filename, filepath, pn, pv,
- summary, description, section, license,
- homepage, bugtracker, provides, bbclassextend,
- inherits, blacklisted, layerbranch, updated=None):
+ filename, filepath, pn, pv, layerbranch,
+ summary="", description="", section="", license="",
+ homepage="", bugtracker="", provides="", bbclassextend="",
+ inherits="", blacklisted="", updated=None):
self.data = {}
self.data['id'] = id
self.data['filename'] = filename
diff --git a/lib/layers/layerindex/cooker.py b/lib/layers/layerindex/cooker.py
index 229f668..76a8739 100644
--- a/lib/layers/layerindex/cooker.py
+++ b/lib/layers/layerindex/cooker.py
@@ -125,7 +125,7 @@ class CookerPlugin(IndexPlugin):
if deps:
layerDependencyId = add_dependency(layerDependencyId, lindex, deps, False)
- # Need to load recipes here
+ # Need to load recipes here (requires cooker access)
recipeId = 0
if False and 'recipes' in load.split(): ## TODO NOT IMPLEMENTED
lindex['recipes'] = {}
diff --git a/lib/layers/layerindex/restapi.py b/lib/layers/layerindex/restapi.py
index d6d29a4..4a2e9d9 100644
--- a/lib/layers/layerindex/restapi.py
+++ b/lib/layers/layerindex/restapi.py
@@ -90,7 +90,8 @@ class RestApiPlugin(IndexPlugin):
def load_cache(path, lindex, branches=None):
logger.debug(1, 'Loading json file %s' % path)
- pindex = json.load(open(path, 'rt', encoding='utf-8'))
+ with open(path, 'rt', encoding='utf-8') as f:
+ pindex = json.load(f)
# Filter the branches on loaded files...
newpBranch = []
@@ -267,47 +268,72 @@ class RestApiPlugin(IndexPlugin):
if ud.type != 'file':
raise NotImplementedError('Writing to anything but a file url is not implemented: %s' % ud.url)
- # Write out to a single file, we have to sort the entries as we write
- if not os.path.isdir(ud.path):
- pindex = {}
- for entry in lindex:
- # Check for either locally added item or apilinks to ignore
- if entry in lindex['CONFIG']['local'] or \
- entry == 'apilinks':
- continue
- pindex[entry] = []
- for objId in lindex[entry]:
- pindex[entry].append(lindex[entry][objId].data)
-
- bb.debug(1, 'Writing index to %s' % ud.path)
- json.dump(layers.layerindex.sort_entry(pindex), open(ud.path, 'wt'), indent=4)
- return
-
- # Write out to a directory one file per layerBranch
try:
layerBranches = lindex['layerBranches']
except KeyError:
logger.error('No layerBranches to write.')
return
- for layerBranchId in layerBranches:
- pindex = {}
- def filter_item(layerBranchId, objects):
- filtered = []
- for obj in lindex[objects]:
+ def filter_item(layerBranchId, objects):
+ filtered = []
+ for obj in lindex[objects]:
+ try:
+ if lindex[objects][obj].get_layerbranch_id() == layerBranchId:
+ filtered.append(lindex[objects][obj].data)
+ except AttributeError:
+ logger.debug(1, 'No obj.get_layerbranch_id(): %s' % objects)
+ # No simple filter method, just include it...
try:
- if lindex[objects][obj].get_layerbranch_id() == layerBranchId:
- filtered.append(lindex[objects][obj].data)
+ filtered.append(lindex[objects][obj].data)
except AttributeError:
- logger.debug(1, 'No obj.get_layerbranch_id(): %s' % objects)
- # No simple filter method, just include it...
- try:
- filtered.append(lindex[objects][obj].data)
- except AttributeError:
- logger.debug(1, 'No obj.data: %s %s' % (objects, type(obj)))
- filtered.append(obj)
- return filtered
+ logger.debug(1, 'No obj.data: %s %s' % (objects, type(obj)))
+ filtered.append(obj)
+ return filtered
+
+
+ # Write out to a single file.
+ # Filter out unnecessary items, then sort as we write for determinism
+ if not os.path.isdir(ud.path):
+ pindex = {}
+
+ pindex['branches'] = []
+ pindex['layerItems'] = []
+ pindex['layerBranches'] = []
+
+ for layerBranchId in layerBranches:
+ if layerBranches[layerBranchId].get_branch().data not in pindex['branches']:
+ pindex['branches'].append(layerBranches[layerBranchId].get_branch().data)
+
+ if layerBranches[layerBranchId].get_layer().data not in pindex['layerItems']:
+ pindex['layerItems'].append(layerBranches[layerBranchId].get_layer().data)
+
+ if layerBranches[layerBranchId].data not in pindex['layerBranches']:
+ pindex['layerBranches'].append(layerBranches[layerBranchId].data)
+
+ for entry in lindex:
+ # Skip local items, apilinks and items already processed
+ if entry in lindex['CONFIG']['local'] or \
+ entry == 'apilinks' or \
+ entry == 'branches' or \
+ entry == 'layerBranches' or \
+ entry == 'layerItems':
+ continue
+ if entry not in pindex:
+ pindex[entry] = []
+ pindex[entry].extend(filter_item(layerBranchId, entry))
+
+ bb.debug(1, 'Writing index to %s' % ud.path)
+ with open(ud.path, 'wt') as f:
+ json.dump(layers.layerindex.sort_entry(pindex), f, indent=4)
+ return
+
+
+ # Write out to a directory one file per layerBranch
+ # Prepare all layer related items, to create a minimal file.
+ # We have to sort the entries as we write so they are deterministic
+ for layerBranchId in layerBranches:
+ pindex = {}
for entry in lindex:
# Skip local items, apilinks and items already processed
@@ -345,4 +371,5 @@ class RestApiPlugin(IndexPlugin):
fpath = os.path.join(ud.path, fname)
bb.debug(1, 'Writing index to %s' % fpath + '.json')
- json.dump(layers.layerindex.sort_entry(pindex), open(fpath + '.json', 'wt'), indent=4)
+ with open(fpath + '.json', 'wt') as f:
+ json.dump(layers.layerindex.sort_entry(pindex), f, indent=4)
diff --git a/lib/layers/manager/__init__.py b/lib/layers/manager/__init__.py
index 3145924..f0ef646 100644
--- a/lib/layers/manager/__init__.py
+++ b/lib/layers/manager/__init__.py
@@ -28,6 +28,9 @@ class LayerManager():
_set_manager(self)
self.data = d
+ # Cooker isn't currently used by this module, but may be referenced
+ # by other layer modules or plugins. This is a single convienent
+ # place to define it.
self.cooker = cooker
self.local_index = None # What is in the bblayers.conf
@@ -86,12 +89,6 @@ class LayerManager():
default_branches = d.getVar('LAYERSERIES_CORENAMES') or "HEAD"
- collections = d.getVar('BBFILE_COLLECTIONS')
- layerconfs = d.varhistory.get_variable_items_files('BBFILE_COLLECTIONS', d)
- bbfile_collections = {layer: os.path.dirname(os.path.dirname(path)) for layer, path in layerconfs.items()}
-
- bblayers = d.getVar('BBLAYERS').split()
-
index = {}
branchId = 0
@@ -103,6 +100,16 @@ class LayerManager():
layerBranchId = 0
index['layerBranches'] = {}
+ bblayers = d.getVar('BBLAYERS').split()
+
+ if not bblayers:
+ # It's blank! Nothing to process...
+ return index
+
+ collections = d.getVar('BBFILE_COLLECTIONS')
+ layerconfs = d.varhistory.get_variable_items_files('BBFILE_COLLECTIONS', d)
+ bbfile_collections = {layer: os.path.dirname(os.path.dirname(path)) for layer, path in layerconfs.items()}
+
(_, bb_branch, _, _) = self.get_bitbake_info()
for branch in default_branches.split():
@@ -111,27 +118,31 @@ class LayerManager():
index['branches'][branchId].define_data(branchId, branch, bb_branch)
for entry in collections.split():
- layername = d.getVar('BBLAYERS_LAYERINDEX_NAME_%s' % entry) or os.path.basename(bbfile_collections[entry])
+ layerpath = entry
+ if entry in bbfile_collections:
+ layerpath = bbfile_collections[entry]
- layerpath = bbfile_collections[entry]
- layersubdir = ""
+ layername = d.getVar('BBLAYERS_LAYERINDEX_NAME_%s' % entry) or os.path.basename(layerpath)
+ layerversion = d.getVar('LAYERVERSION_%s' % entry) or ""
+ layerurl = _handle_git_remote(layerpath)
- layerbasepath = self._run_command('git rev-parse --show-toplevel', layerpath, default=layerpath)
- if os.path.abspath(layerpath) != os.path.abspath(layerbasepath):
- layersubdir = os.path.abspath(layerpath)[len(layerbasepath) + 1:]
+ layersubdir = ""
+ layerrev = "<unknown>"
+ layerbranch = "<unknown>"
- layerversion = d.getVar('LAYERVERSION_%s' % entry) or ""
+ if os.path.isdir(layerpath):
+ layerbasepath = self._run_command('git rev-parse --show-toplevel', layerpath, default=layerpath)
+ if os.path.abspath(layerpath) != os.path.abspath(layerbasepath):
+ layersubdir = os.path.abspath(layerpath)[len(layerbasepath) + 1:]
- layerbranch = self._run_command('git rev-parse --abbrev-ref HEAD', layerpath, default="<unknown>")
- layerrev = self._run_command('git rev-parse HEAD', layerpath, default="<unknown>")
+ layerbranch = self._run_command('git rev-parse --abbrev-ref HEAD', layerpath, default="<unknown>")
+ layerrev = self._run_command('git rev-parse HEAD', layerpath, default="<unknown>")
- for remotes in self._run_command('git remote -v', layerpath, default="").split("\n"):
- remote = remotes.split("\t")[1].split(" ")[0]
- if "(fetch)" == remotes.split("\t")[1].split(" ")[1]:
- layerurl = _handle_git_remote(remote)
- break
- else:
- layerurl = _handle_git_remote(layerpath)
+ for remotes in self._run_command('git remote -v', layerpath, default="").split("\n"):
+ remote = remotes.split("\t")[1].split(" ")[0]
+ if "(fetch)" == remotes.split("\t")[1].split(" ")[1]:
+ layerurl = _handle_git_remote(remote)
+ break
layerItemId += 1
index['layerItems'][layerItemId] = layers.layerindex.LayerItem(index, None)
@@ -141,7 +152,7 @@ class LayerManager():
layerBranchId += 1
index['layerBranches'][layerBranchId] = layers.layerindex.LayerBranch(index, None)
index['layerBranches'][layerBranchId].define_data(layerBranchId, entry, layerversion, layerItemId, branchId,
- vcs_subdir=layersubdir, vcs_last_rev= layerrev, actual_branch=layerbranch)
+ vcs_subdir=layersubdir, vcs_last_rev=layerrev, actual_branch=layerbranch)
return index
@@ -150,7 +161,7 @@ class LayerManager():
# You are not allowed to have two of the same url, but different branches
def get_clone_directory(self, url):
- baseDir = get_clone_base_directory()
+ baseDir = self.get_clone_base_directory()
if not baseDir:
return None
repo = os.path.basename(url)
--
1.8.3.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 4/5] toaster/orm/management/commands/lsupdates.py: Use new layerindex module
2017-07-28 15:37 [PATCH 0/5] Implement a new remote layer module Mark Hatle
` (2 preceding siblings ...)
2017-07-28 15:37 ` [PATCH 3/5] lib/bblayers: Add support for the new layer modules Mark Hatle
@ 2017-07-28 15:37 ` Mark Hatle
2017-08-03 18:06 ` Paul Eggleton
2017-07-28 15:37 ` [PATCH 5/5] bitbake-layers: disable parsing for layerindex commands Mark Hatle
4 siblings, 1 reply; 20+ messages in thread
From: Mark Hatle @ 2017-07-28 15:37 UTC (permalink / raw)
To: bitbake-devel; +Cc: toaster
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
lib/toaster/orm/management/commands/lsupdates.py | 215 +++++++++--------------
1 file changed, 87 insertions(+), 128 deletions(-)
diff --git a/lib/toaster/orm/management/commands/lsupdates.py b/lib/toaster/orm/management/commands/lsupdates.py
index 0b0d4ff..055ad78 100644
--- a/lib/toaster/orm/management/commands/lsupdates.py
+++ b/lib/toaster/orm/management/commands/lsupdates.py
@@ -28,13 +28,16 @@ from orm.models import Distro
import os
import sys
-import json
import logging
import threading
import time
logger = logging.getLogger("toaster")
-DEFAULT_LAYERINDEX_SERVER = "http://layers.openembedded.org/layerindex/api/"
+DEFAULT_LAYERINDEX_SERVER = "http://layers.openembedded.org/layerindex/api/;type=restapi"
+
+# load Bitbake components
+path = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
+sys.path.insert(0, path)
class Spinner(threading.Thread):
@@ -83,43 +86,6 @@ class Command(BaseCommand):
self.apiurl = DEFAULT_LAYERINDEX_SERVER
assert self.apiurl is not None
- try:
- from urllib.request import urlopen, URLError
- from urllib.parse import urlparse
- except ImportError:
- from urllib2 import urlopen, URLError
- from urlparse import urlparse
-
- proxy_settings = os.environ.get("http_proxy", None)
-
- def _get_json_response(apiurl=DEFAULT_LAYERINDEX_SERVER):
- http_progress = Spinner()
- http_progress.start()
-
- _parsedurl = urlparse(apiurl)
- path = _parsedurl.path
-
- # logger.debug("Fetching %s", apiurl)
- try:
- res = urlopen(apiurl)
- except URLError as e:
- raise Exception("Failed to read %s: %s" % (path, e.reason))
-
- parsed = json.loads(res.read().decode('utf-8'))
-
- http_progress.stop()
- return parsed
-
- # verify we can get the basic api
- try:
- apilinks = _get_json_response()
- except Exception as e:
- import traceback
- if proxy_settings is not None:
- logger.info("EE: Using proxy %s" % proxy_settings)
- logger.warning("EE: could not connect to %s, skipping update:"
- "%s\n%s" % (self.apiurl, e, traceback.format_exc()))
- return
# update branches; only those that we already have names listed in the
# Releases table
@@ -128,110 +94,115 @@ class Command(BaseCommand):
if len(whitelist_branch_names) == 0:
raise Exception("Failed to make list of branches to fetch")
+ self.apiurl += ";branch=%s" % "OR".join(whitelist_branch_names)
+
+ http_progress = Spinner()
+
logger.info("Fetching metadata releases for %s",
" ".join(whitelist_branch_names))
- branches_info = _get_json_response(apilinks['branches'] +
- "?filter=name:%s"
- % "OR".join(whitelist_branch_names))
+
+ import layers.layerindex
+
+ layerindex = layers.layerindex.LayerIndex(None)
+
+ http_progress.start()
+ layerindex.load_layerindex(self.apiurl)
+ http_progress.stop()
+
+ # We know we're only processing one entry, so we reference it here
+ # (this is cheating...)
+ lindex = layerindex.lindex[0]
# Map the layer index branches to toaster releases
li_branch_id_to_toaster_release = {}
- total = len(branches_info)
- for i, branch in enumerate(branches_info):
- li_branch_id_to_toaster_release[branch['id']] = \
- Release.objects.get(name=branch['name'])
+ logger.info("Processing branches")
+
+ total = len(lindex['branches'])
+ for i, branchId in enumerate(lindex['branches']):
+ li_branch_id_to_toaster_release[branchId] = \
+ Release.objects.get(name=lindex['branches'][branchId].get_name())
self.mini_progress("Releases", i, total)
# keep a track of the layerindex (li) id mappings so that
# layer_versions can be created for these layers later on
li_layer_id_to_toaster_layer_id = {}
- logger.info("Fetching layers")
-
- layers_info = _get_json_response(apilinks['layerItems'])
+ logger.info("Processing layers")
- total = len(layers_info)
- for i, li in enumerate(layers_info):
+ total = len(lindex['layerItems'])
+ for i, liId in enumerate(lindex['layerItems']):
try:
- l, created = Layer.objects.get_or_create(name=li['name'])
- l.up_date = li['updated']
- l.summary = li['summary']
- l.description = li['description']
+ l, created = Layer.objects.get_or_create(name=lindex['layerItems'][liId].get_name())
+ l.up_date = lindex['layerItems'][liId].get_updated()
+ l.summary = lindex['layerItems'][liId].get_summary()
+ l.description = lindex['layerItems'][liId].get_description()
if created:
- l.vcs_url = li['vcs_url']
- l.vcs_web_url = li['vcs_web_url']
- l.vcs_web_tree_base_url = li['vcs_web_tree_base_url']
- l.vcs_web_file_base_url = li['vcs_web_file_base_url']
+ l.vcs_url = lindex['layerItems'][liId].get_vcs_url()
+ l.vcs_web_url = lindex['layerItems'][liId].get_vcs_web_url()
+ l.vcs_web_tree_base_url = lindex['layerItems'][liId].get_vcs_web_tree_base_url()
+ l.vcs_web_file_base_url = lindex['layerItems'][liId].get_vcs_web_file_base_url()
l.save()
except Layer.MultipleObjectsReturned:
logger.info("Skipped %s as we found multiple layers and "
"don't know which to update" %
li['name'])
- li_layer_id_to_toaster_layer_id[li['id']] = l.pk
+ li_layer_id_to_toaster_layer_id[liId] = l.pk
self.mini_progress("layers", i, total)
# update layer_versions
- logger.info("Fetching layer versions")
- layerbranches_info = _get_json_response(
- apilinks['layerBranches'] + "?filter=branch__name:%s" %
- "OR".join(whitelist_branch_names))
+ logger.info("Processing layer branches")
# Map Layer index layer_branch object id to
# layer_version toaster object id
li_layer_branch_id_to_toaster_lv_id = {}
- total = len(layerbranches_info)
- for i, lbi in enumerate(layerbranches_info):
+ total = len(lindex['layerBranches'])
+ for i, lbiId in enumerate(lindex['layerBranches']):
# release as defined by toaster map to layerindex branch
- release = li_branch_id_to_toaster_release[lbi['branch']]
+ release = li_branch_id_to_toaster_release[lindex['layerBranches'][lbiId].get_branch_id()]
try:
lv, created = Layer_Version.objects.get_or_create(
layer=Layer.objects.get(
- pk=li_layer_id_to_toaster_layer_id[lbi['layer']]),
+ pk=li_layer_id_to_toaster_layer_id[lindex['layerBranches'][lbiId].get_layer_id()]),
release=release
)
except KeyError:
logger.warning(
"No such layerindex layer referenced by layerbranch %d" %
- lbi['layer'])
+ lindex['layerBranches'][lbiId].get_layer_id())
continue
if created:
- lv.release = li_branch_id_to_toaster_release[lbi['branch']]
- lv.up_date = lbi['updated']
- lv.commit = lbi['actual_branch']
- lv.dirpath = lbi['vcs_subdir']
+ lv.release = li_branch_id_to_toaster_release[lindex['layerBranches'][lbiId].get_branch_id()]
+ lv.up_date = lindex['layerBranches'][lbiId].get_updated()
+ lv.commit = lindex['layerBranches'][lbiId].get_actual_branch()
+ lv.dirpath = lindex['layerBranches'][lbiId].get_vcs_subdir()
lv.save()
- li_layer_branch_id_to_toaster_lv_id[lbi['id']] =\
+ li_layer_branch_id_to_toaster_lv_id[lindex['layerBranches'][lbiId].get_id()] =\
lv.pk
self.mini_progress("layer versions", i, total)
- logger.info("Fetching layer version dependencies")
- # update layer dependencies
- layerdependencies_info = _get_json_response(
- apilinks['layerDependencies'] +
- "?filter=layerbranch__branch__name:%s" %
- "OR".join(whitelist_branch_names))
+ logger.info("Processing layer dependencies")
dependlist = {}
- for ldi in layerdependencies_info:
+ for ldiId in lindex['layerDependencies']:
try:
lv = Layer_Version.objects.get(
- pk=li_layer_branch_id_to_toaster_lv_id[ldi['layerbranch']])
+ pk=li_layer_branch_id_to_toaster_lv_id[lindex['layerDependencies'][ldiId].get_layerbranch_id()])
except Layer_Version.DoesNotExist as e:
continue
if lv not in dependlist:
dependlist[lv] = []
try:
- layer_id = li_layer_id_to_toaster_layer_id[ldi['dependency']]
+ layer_id = li_layer_id_to_toaster_layer_id[lindex['layerDependencies'][ldiId].get_dependency_id()]
dependlist[lv].append(
Layer_Version.objects.get(layer__pk=layer_id,
@@ -240,7 +211,7 @@ class Command(BaseCommand):
except Layer_Version.DoesNotExist:
logger.warning("Cannot find layer version (ls:%s),"
"up_id:%s lv:%s" %
- (self, ldi['dependency'], lv))
+ (self, lindex['layerDependencies'][ldiId].get_dependency_id(), lv))
total = len(dependlist)
for i, lv in enumerate(dependlist):
@@ -251,73 +222,61 @@ class Command(BaseCommand):
self.mini_progress("Layer version dependencies", i, total)
# update Distros
- logger.info("Fetching distro information")
- distros_info = _get_json_response(
- apilinks['distros'] + "?filter=layerbranch__branch__name:%s" %
- "OR".join(whitelist_branch_names))
+ logger.info("Processing distro information")
- total = len(distros_info)
- for i, di in enumerate(distros_info):
+ total = len(lindex['distros'])
+ for i, diId in enumerate(lindex['distros']):
distro, created = Distro.objects.get_or_create(
- name=di['name'],
+ name=lindex['distros'][diId].get_name(),
layer_version=Layer_Version.objects.get(
- pk=li_layer_branch_id_to_toaster_lv_id[di['layerbranch']]))
- distro.up_date = di['updated']
- distro.name = di['name']
- distro.description = di['description']
+ pk=li_layer_branch_id_to_toaster_lv_id[lindex['distros'][diId].get_layerbranch_id()]))
+ distro.up_date = lindex['distros'][diId].get_updated()
+ distro.name = lindex['distros'][diId].get_name()
+ distro.description = lindex['distros'][diId].get_description()
distro.save()
self.mini_progress("distros", i, total)
# update machines
- logger.info("Fetching machine information")
- machines_info = _get_json_response(
- apilinks['machines'] + "?filter=layerbranch__branch__name:%s" %
- "OR".join(whitelist_branch_names))
+ logger.info("Processing machine information")
- total = len(machines_info)
- for i, mi in enumerate(machines_info):
+ total = len(lindex['machines'])
+ for i, miId in enumerate(lindex['machines']):
mo, created = Machine.objects.get_or_create(
- name=mi['name'],
+ name=lindex['machines'][miId].get_name(),
layer_version=Layer_Version.objects.get(
- pk=li_layer_branch_id_to_toaster_lv_id[mi['layerbranch']]))
- mo.up_date = mi['updated']
- mo.name = mi['name']
- mo.description = mi['description']
+ pk=li_layer_branch_id_to_toaster_lv_id[lindex['machines'][miId].get_layerbranch_id()]))
+ mo.up_date = lindex['machines'][miId].get_updated()
+ mo.name = lindex['machines'][miId].get_name()
+ mo.description = lindex['machines'][miId].get_description()
mo.save()
self.mini_progress("machines", i, total)
# update recipes; paginate by layer version / layer branch
- logger.info("Fetching recipe information")
- recipes_info = _get_json_response(
- apilinks['recipes'] + "?filter=layerbranch__branch__name:%s" %
- "OR".join(whitelist_branch_names))
+ logger.info("Processing recipe information")
- total = len(recipes_info)
- for i, ri in enumerate(recipes_info):
+ total = len(lindex['recipes'])
+ for i, riId in enumerate(lindex['recipes']):
try:
- lv_id = li_layer_branch_id_to_toaster_lv_id[ri['layerbranch']]
+ lv_id = li_layer_branch_id_to_toaster_lv_id[lindex['recipes'][riId].get_layerbranch_id()]
lv = Layer_Version.objects.get(pk=lv_id)
ro, created = Recipe.objects.get_or_create(
layer_version=lv,
- name=ri['pn']
+ name=lindex['recipes'][riId].get_pn()
)
ro.layer_version = lv
- ro.up_date = ri['updated']
- ro.name = ri['pn']
- ro.version = ri['pv']
- ro.summary = ri['summary']
- ro.description = ri['description']
- ro.section = ri['section']
- ro.license = ri['license']
- ro.homepage = ri['homepage']
- ro.bugtracker = ri['bugtracker']
- ro.file_path = ri['filepath'] + "/" + ri['filename']
- if 'inherits' in ri:
- ro.is_image = 'image' in ri['inherits'].split()
- else: # workaround for old style layer index
- ro.is_image = "-image-" in ri['pn']
+ ro.up_date = lindex['recipes'][riId].get_updated()
+ ro.name = lindex['recipes'][riId].get_pn()
+ ro.version = lindex['recipes'][riId].get_pv()
+ ro.summary = lindex['recipes'][riId].get_summary()
+ ro.description = lindex['recipes'][riId].get_description()
+ ro.section = lindex['recipes'][riId].get_section()
+ ro.license = lindex['recipes'][riId].get_license()
+ ro.homepage = lindex['recipes'][riId].get_homepage()
+ ro.bugtracker = lindex['recipes'][riId].get_bugtracker()
+ ro.file_path = lindex['recipes'][riId].get_filepath() + "/" + lindex['recipes'][riId].get_filename()
+ ro.is_image = 'image' in lindex['recipes'][riId].get_inherits().split()
ro.save()
except Exception as e:
logger.warning("Failed saving recipe %s", e)
--
1.8.3.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 5/5] bitbake-layers: disable parsing for layerindex commands
2017-07-28 15:37 [PATCH 0/5] Implement a new remote layer module Mark Hatle
` (3 preceding siblings ...)
2017-07-28 15:37 ` [PATCH 4/5] toaster/orm/management/commands/lsupdates.py: Use new layerindex module Mark Hatle
@ 2017-07-28 15:37 ` Mark Hatle
4 siblings, 0 replies; 20+ messages in thread
From: Mark Hatle @ 2017-07-28 15:37 UTC (permalink / raw)
To: bitbake-devel; +Cc: toaster
From: Paul Eggleton <paul.eggleton@linux.intel.com>
These don't need to access recipe information, so let's not waste the
user's time parsing all recipes.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
lib/bblayers/layerindex.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/bblayers/layerindex.py b/lib/bblayers/layerindex.py
index d1dac0b..153b0d9 100644
--- a/lib/bblayers/layerindex.py
+++ b/lib/bblayers/layerindex.py
@@ -132,12 +132,12 @@ class LayerIndexPlugin(ActionPlugin):
self.do_layerindex_fetch(args)
def register_commands(self, sp):
- parser_layerindex_fetch = self.add_command(sp, 'layerindex-fetch', self.do_layerindex_fetch)
+ parser_layerindex_fetch = self.add_command(sp, 'layerindex-fetch', self.do_layerindex_fetch, parserecipes=False)
parser_layerindex_fetch.add_argument('-n', '--show-only', help='show dependencies and do nothing else', action='store_true')
parser_layerindex_fetch.add_argument('-b', '--branch', help='branch name to fetch (default %(default)s)')
parser_layerindex_fetch.add_argument('-i', '--ignore', help='assume the specified layers do not need to be fetched/added (separate multiple layers with commas, no spaces)', metavar='LAYER')
parser_layerindex_fetch.add_argument('layername', nargs='+', help='layer to fetch')
- parser_layerindex_show_depends = self.add_command(sp, 'layerindex-show-depends', self.do_layerindex_show_depends)
+ parser_layerindex_show_depends = self.add_command(sp, 'layerindex-show-depends', self.do_layerindex_show_depends, parserecipes=False)
parser_layerindex_show_depends.add_argument('-b', '--branch', help='branch name to fetch (default %(default)s)')
parser_layerindex_show_depends.add_argument('layername', nargs='+', help='layer to query')
--
1.8.3.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [bitbake-devel] [PATCH 3/5] lib/bblayers: Add support for the new layer modules
2017-07-28 15:37 ` [PATCH 3/5] lib/bblayers: Add support for the new layer modules Mark Hatle
@ 2017-07-28 16:04 ` Mark Hatle
0 siblings, 0 replies; 20+ messages in thread
From: Mark Hatle @ 2017-07-28 16:04 UTC (permalink / raw)
To: bitbake-devel; +Cc: toaster
On 7/28/17 10:37 AM, Mark Hatle wrote:
> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
> ---
> lib/bblayers/layerindex.py | 313 +++++++++++---------------------------
> lib/layers/layerindex/__init__.py | 28 +++-
> lib/layers/layerindex/cooker.py | 2 +-
> lib/layers/layerindex/restapi.py | 95 +++++++-----
> lib/layers/manager/__init__.py | 59 ++++---
> 5 files changed, 210 insertions(+), 287 deletions(-)
Argh, I just realized a bunch of fixes got merge to this patch, they should have
been merged to 1/5.
I've fixed this in the git tree.
--Mark
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 3/5] lib/bblayers: Add support for the new layer modules
@ 2017-07-28 16:04 ` Mark Hatle
0 siblings, 0 replies; 20+ messages in thread
From: Mark Hatle @ 2017-07-28 16:04 UTC (permalink / raw)
To: bitbake-devel; +Cc: toaster
On 7/28/17 10:37 AM, Mark Hatle wrote:
> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
> ---
> lib/bblayers/layerindex.py | 313 +++++++++++---------------------------
> lib/layers/layerindex/__init__.py | 28 +++-
> lib/layers/layerindex/cooker.py | 2 +-
> lib/layers/layerindex/restapi.py | 95 +++++++-----
> lib/layers/manager/__init__.py | 59 ++++---
> 5 files changed, 210 insertions(+), 287 deletions(-)
Argh, I just realized a bunch of fixes got merge to this patch, they should have
been merged to 1/5.
I've fixed this in the git tree.
--Mark
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [bitbake-devel] [PATCH 1/5] lib/layers: Initial layer and layer index implementeation
2017-07-28 15:37 ` [PATCH 1/5] lib/layers: Initial layer and layer index implementeation Mark Hatle
@ 2017-08-03 18:05 ` Paul Eggleton
0 siblings, 0 replies; 20+ messages in thread
From: Paul Eggleton @ 2017-08-03 18:05 UTC (permalink / raw)
To: Mark Hatle; +Cc: bitbake-devel, toaster
On Friday, 28 July 2017 5:37:18 PM CEST Mark Hatle wrote:
> This module provides two components, layerindex, which is used to talk a
> layerindex, such as layers.openembedded.org.
>
> The other module is the 'manager'. This module will handle downloading,
> re-writing the bblayers to match the downloads and other related layer
> management tasks.
It's great to have a solid internal API for interacting with the layer index
to replace the less than optimal earlier implementation in bitbake-layers -
thanks for sorting this out. The only comments I'd make:
1) Can you rename the fetch2 plugin to something like bbfetch or bitbakefetch
or something like that. I'd rather we didn't proliferate the fetch2 name any
further - IIRC Richard has talked about renaming the module back to fetch at
some point.
2) cooker.py has a "## TODO NOT IMPLEMENTED" comment that if still valid needs
to be expanded upon - what's actually missing? There's also a moderate size
block of commented-out code just below that, since this is new can we drop
that if it's genuinely not needed?
3) Can we have a default type for layerindex URLs of "restapi" so we don't
need to specify this everywhere in the common case? This would also mean we
wouldn't have to change DEFAULT_LAYERINDEX_SERVER in the toaster code.
4) The "layers" naming of the module under lib is a little confusing given
that we already have a bblayers module under lib/, and one might incorrectly
assume that this was used for all layer handling. I don't actually yet have a
naming scheme / structure I like better though :(
5) "OR" to split the branches is a very awkward. Can we use , (comma) instead?
6) I'm not massively thrilled at duplicating the layer index data structures
in bitbake. It does make things easier on the bitbake side, and the
alternative of making use of the original django models in bitbake isn't
practical, but it's just a bit unfortunate in terms of future maintenance.
Having said that we made the same choice in Toaster, and the layer index
models don't change that often. (I guess I'm just highlighting this so we know
what we're getting into.)
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/5] lib/layers: Initial layer and layer index implementeation
@ 2017-08-03 18:05 ` Paul Eggleton
0 siblings, 0 replies; 20+ messages in thread
From: Paul Eggleton @ 2017-08-03 18:05 UTC (permalink / raw)
To: Mark Hatle; +Cc: bitbake-devel, toaster
On Friday, 28 July 2017 5:37:18 PM CEST Mark Hatle wrote:
> This module provides two components, layerindex, which is used to talk a
> layerindex, such as layers.openembedded.org.
>
> The other module is the 'manager'. This module will handle downloading,
> re-writing the bblayers to match the downloads and other related layer
> management tasks.
It's great to have a solid internal API for interacting with the layer index
to replace the less than optimal earlier implementation in bitbake-layers -
thanks for sorting this out. The only comments I'd make:
1) Can you rename the fetch2 plugin to something like bbfetch or bitbakefetch
or something like that. I'd rather we didn't proliferate the fetch2 name any
further - IIRC Richard has talked about renaming the module back to fetch at
some point.
2) cooker.py has a "## TODO NOT IMPLEMENTED" comment that if still valid needs
to be expanded upon - what's actually missing? There's also a moderate size
block of commented-out code just below that, since this is new can we drop
that if it's genuinely not needed?
3) Can we have a default type for layerindex URLs of "restapi" so we don't
need to specify this everywhere in the common case? This would also mean we
wouldn't have to change DEFAULT_LAYERINDEX_SERVER in the toaster code.
4) The "layers" naming of the module under lib is a little confusing given
that we already have a bblayers module under lib/, and one might incorrectly
assume that this was used for all layer handling. I don't actually yet have a
naming scheme / structure I like better though :(
5) "OR" to split the branches is a very awkward. Can we use , (comma) instead?
6) I'm not massively thrilled at duplicating the layer index data structures
in bitbake. It does make things easier on the bitbake side, and the
alternative of making use of the original django models in bitbake isn't
practical, but it's just a bit unfortunate in terms of future maintenance.
Having said that we made the same choice in Toaster, and the layer index
models don't change that often. (I guess I'm just highlighting this so we know
what we're getting into.)
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [bitbake-devel] [PATCH 4/5] toaster/orm/management/commands/lsupdates.py: Use new layerindex module
2017-07-28 15:37 ` [PATCH 4/5] toaster/orm/management/commands/lsupdates.py: Use new layerindex module Mark Hatle
@ 2017-08-03 18:06 ` Paul Eggleton
0 siblings, 0 replies; 20+ messages in thread
From: Paul Eggleton @ 2017-08-03 18:06 UTC (permalink / raw)
To: Mark Hatle; +Cc: bitbake-devel, toaster
On Friday, 28 July 2017 5:37:21 PM CEST Mark Hatle wrote:
> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Missing a commit message - and you wrote the book on proper commit messages :p
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 4/5] toaster/orm/management/commands/lsupdates.py: Use new layerindex module
@ 2017-08-03 18:06 ` Paul Eggleton
0 siblings, 0 replies; 20+ messages in thread
From: Paul Eggleton @ 2017-08-03 18:06 UTC (permalink / raw)
To: Mark Hatle; +Cc: bitbake-devel, toaster
On Friday, 28 July 2017 5:37:21 PM CEST Mark Hatle wrote:
> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Missing a commit message - and you wrote the book on proper commit messages :p
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [bitbake-devel] [PATCH 1/5] lib/layers: Initial layer and layer index implementeation
2017-08-03 18:05 ` Paul Eggleton
@ 2017-08-03 18:17 ` Mark Hatle
-1 siblings, 0 replies; 20+ messages in thread
From: Mark Hatle @ 2017-08-03 18:17 UTC (permalink / raw)
To: Paul Eggleton; +Cc: bitbake-devel, toaster
On 8/3/17 1:05 PM, Paul Eggleton wrote:
> On Friday, 28 July 2017 5:37:18 PM CEST Mark Hatle wrote:
>> This module provides two components, layerindex, which is used to talk a
>> layerindex, such as layers.openembedded.org.
>>
>> The other module is the 'manager'. This module will handle downloading,
>> re-writing the bblayers to match the downloads and other related layer
>> management tasks.
>
> It's great to have a solid internal API for interacting with the layer index
> to replace the less than optimal earlier implementation in bitbake-layers -
> thanks for sorting this out. The only comments I'd make:
>
> 1) Can you rename the fetch2 plugin to something like bbfetch or bitbakefetch
> or something like that. I'd rather we didn't proliferate the fetch2 name any
> further - IIRC Richard has talked about renaming the module back to fetch at
> some point.
Yes, I can easily do that. The name used is the name of the fetch plugin, so it
is partially use visible.
> 2) cooker.py has a "## TODO NOT IMPLEMENTED" comment that if still valid needs
> to be expanded upon - what's actually missing? There's also a moderate size
> block of commented-out code just below that, since this is new can we drop
> that if it's genuinely not needed?
The block immediately below it is what needs to be implemented. This is the
loading of recipe data from the cooker store (or even having to parse recipes).
As part of the toaster work, I expect to implement it. I left it there
specifically as a guide to the implementer on what is likely needed -- even
though I know the commented out parts do not work as-is.
(Currently no recipe specific behavior is loaded from the currently running system.)
> 3) Can we have a default type for layerindex URLs of "restapi" so we don't
> need to specify this everywhere in the common case? This would also mean we
> wouldn't have to change DEFAULT_LAYERINDEX_SERVER in the toaster code.
Yes, I can make it the default unless a type is defined. The toaster code will
be changing to match up with the bblayers code. I'd rather not specify the same
thing in multiple places. But I won't be making that change until the toaster
work is in progress.
> 4) The "layers" naming of the module under lib is a little confusing given
> that we already have a bblayers module under lib/, and one might incorrectly
> assume that this was used for all layer handling. I don't actually yet have a
> naming scheme / structure I like better though :(
Ya, I couldn't think of one. Initially I had called it layerindex, but the
manager (which current does downloading) confused it.. so I changed it.
> 5) "OR" to split the branches is a very awkward. Can we use , (comma) instead?
The implementation was designed to match the current format of the restapi URL
format. But as long as it can be represented in a URI, anything should be fine.
> 6) I'm not massively thrilled at duplicating the layer index data structures
> in bitbake. It does make things easier on the bitbake side, and the
> alternative of making use of the original django models in bitbake isn't
> practical, but it's just a bit unfortunate in terms of future maintenance.
> Having said that we made the same choice in Toaster, and the layer index
> models don't change that often. (I guess I'm just highlighting this so we know
> what we're getting into.)
Yes.. The internal 'data' side of things rarely changes -- but by having the
class we can handle any changes that do happen in the future more easily -- and
as you noted it does simplify some of the django like features.
--Mark
> Cheers,
> Paul
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/5] lib/layers: Initial layer and layer index implementeation
@ 2017-08-03 18:17 ` Mark Hatle
0 siblings, 0 replies; 20+ messages in thread
From: Mark Hatle @ 2017-08-03 18:17 UTC (permalink / raw)
To: Paul Eggleton; +Cc: bitbake-devel, toaster
On 8/3/17 1:05 PM, Paul Eggleton wrote:
> On Friday, 28 July 2017 5:37:18 PM CEST Mark Hatle wrote:
>> This module provides two components, layerindex, which is used to talk a
>> layerindex, such as layers.openembedded.org.
>>
>> The other module is the 'manager'. This module will handle downloading,
>> re-writing the bblayers to match the downloads and other related layer
>> management tasks.
>
> It's great to have a solid internal API for interacting with the layer index
> to replace the less than optimal earlier implementation in bitbake-layers -
> thanks for sorting this out. The only comments I'd make:
>
> 1) Can you rename the fetch2 plugin to something like bbfetch or bitbakefetch
> or something like that. I'd rather we didn't proliferate the fetch2 name any
> further - IIRC Richard has talked about renaming the module back to fetch at
> some point.
Yes, I can easily do that. The name used is the name of the fetch plugin, so it
is partially use visible.
> 2) cooker.py has a "## TODO NOT IMPLEMENTED" comment that if still valid needs
> to be expanded upon - what's actually missing? There's also a moderate size
> block of commented-out code just below that, since this is new can we drop
> that if it's genuinely not needed?
The block immediately below it is what needs to be implemented. This is the
loading of recipe data from the cooker store (or even having to parse recipes).
As part of the toaster work, I expect to implement it. I left it there
specifically as a guide to the implementer on what is likely needed -- even
though I know the commented out parts do not work as-is.
(Currently no recipe specific behavior is loaded from the currently running system.)
> 3) Can we have a default type for layerindex URLs of "restapi" so we don't
> need to specify this everywhere in the common case? This would also mean we
> wouldn't have to change DEFAULT_LAYERINDEX_SERVER in the toaster code.
Yes, I can make it the default unless a type is defined. The toaster code will
be changing to match up with the bblayers code. I'd rather not specify the same
thing in multiple places. But I won't be making that change until the toaster
work is in progress.
> 4) The "layers" naming of the module under lib is a little confusing given
> that we already have a bblayers module under lib/, and one might incorrectly
> assume that this was used for all layer handling. I don't actually yet have a
> naming scheme / structure I like better though :(
Ya, I couldn't think of one. Initially I had called it layerindex, but the
manager (which current does downloading) confused it.. so I changed it.
> 5) "OR" to split the branches is a very awkward. Can we use , (comma) instead?
The implementation was designed to match the current format of the restapi URL
format. But as long as it can be represented in a URI, anything should be fine.
> 6) I'm not massively thrilled at duplicating the layer index data structures
> in bitbake. It does make things easier on the bitbake side, and the
> alternative of making use of the original django models in bitbake isn't
> practical, but it's just a bit unfortunate in terms of future maintenance.
> Having said that we made the same choice in Toaster, and the layer index
> models don't change that often. (I guess I'm just highlighting this so we know
> what we're getting into.)
Yes.. The internal 'data' side of things rarely changes -- but by having the
class we can handle any changes that do happen in the future more easily -- and
as you noted it does simplify some of the django like features.
--Mark
> Cheers,
> Paul
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [bitbake-devel] [PATCH 4/5] toaster/orm/management/commands/lsupdates.py: Use new layerindex module
2017-08-03 18:06 ` Paul Eggleton
@ 2017-08-03 18:18 ` Mark Hatle
-1 siblings, 0 replies; 20+ messages in thread
From: Mark Hatle @ 2017-08-03 18:18 UTC (permalink / raw)
To: Paul Eggleton; +Cc: bitbake-devel, toaster
On 8/3/17 1:06 PM, Paul Eggleton wrote:
> On Friday, 28 July 2017 5:37:21 PM CEST Mark Hatle wrote:
>> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
>
> Missing a commit message - and you wrote the book on proper commit messages :p
In this case, what is wrong with the message. The change doesn't do anything
but use the new layer index module. Functionality remains the same.
I can duplicate it in the commit message, but this isn't a place where I see any
added value.
--Mark
> Cheers,
> Paul
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 4/5] toaster/orm/management/commands/lsupdates.py: Use new layerindex module
@ 2017-08-03 18:18 ` Mark Hatle
0 siblings, 0 replies; 20+ messages in thread
From: Mark Hatle @ 2017-08-03 18:18 UTC (permalink / raw)
To: Paul Eggleton; +Cc: bitbake-devel, toaster
On 8/3/17 1:06 PM, Paul Eggleton wrote:
> On Friday, 28 July 2017 5:37:21 PM CEST Mark Hatle wrote:
>> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
>
> Missing a commit message - and you wrote the book on proper commit messages :p
In this case, what is wrong with the message. The change doesn't do anything
but use the new layer index module. Functionality remains the same.
I can duplicate it in the commit message, but this isn't a place where I see any
added value.
--Mark
> Cheers,
> Paul
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [bitbake-devel] [PATCH 4/5] toaster/orm/management/commands/lsupdates.py: Use new layerindex module
2017-08-03 18:18 ` Mark Hatle
@ 2017-08-03 22:15 ` Paul Eggleton
-1 siblings, 0 replies; 20+ messages in thread
From: Paul Eggleton @ 2017-08-03 22:15 UTC (permalink / raw)
To: Mark Hatle; +Cc: bitbake-devel, toaster
On Thursday, 3 August 2017 8:18:03 PM CEST Mark Hatle wrote:
> On 8/3/17 1:06 PM, Paul Eggleton wrote:
> > On Friday, 28 July 2017 5:37:21 PM CEST Mark Hatle wrote:
> >> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
> >
> > Missing a commit message - and you wrote the book on proper commit
> > messages :p
>
> In this case, what is wrong with the message. The change doesn't do
> anything but use the new layer index module. Functionality remains the
> same.
Wouldn't you at minimum want to state exactly that last bit then? That isn't
clear without carefully looking through the patch.
Same for the other commit, except there what is displayed by bitbake-layers
layerindex-fetch has changed a little bit.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 4/5] toaster/orm/management/commands/lsupdates.py: Use new layerindex module
@ 2017-08-03 22:15 ` Paul Eggleton
0 siblings, 0 replies; 20+ messages in thread
From: Paul Eggleton @ 2017-08-03 22:15 UTC (permalink / raw)
To: Mark Hatle; +Cc: bitbake-devel, toaster
On Thursday, 3 August 2017 8:18:03 PM CEST Mark Hatle wrote:
> On 8/3/17 1:06 PM, Paul Eggleton wrote:
> > On Friday, 28 July 2017 5:37:21 PM CEST Mark Hatle wrote:
> >> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
> >
> > Missing a commit message - and you wrote the book on proper commit
> > messages :p
>
> In this case, what is wrong with the message. The change doesn't do
> anything but use the new layer index module. Functionality remains the
> same.
Wouldn't you at minimum want to state exactly that last bit then? That isn't
clear without carefully looking through the patch.
Same for the other commit, except there what is displayed by bitbake-layers
layerindex-fetch has changed a little bit.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [bitbake-devel] [PATCH 4/5] toaster/orm/management/commands/lsupdates.py: Use new layerindex module
2017-08-03 22:15 ` Paul Eggleton
@ 2017-08-03 22:40 ` Mark Hatle
-1 siblings, 0 replies; 20+ messages in thread
From: Mark Hatle @ 2017-08-03 22:40 UTC (permalink / raw)
To: Paul Eggleton; +Cc: bitbake-devel, toaster
On 8/3/17 5:15 PM, Paul Eggleton wrote:
> On Thursday, 3 August 2017 8:18:03 PM CEST Mark Hatle wrote:
>> On 8/3/17 1:06 PM, Paul Eggleton wrote:
>>> On Friday, 28 July 2017 5:37:21 PM CEST Mark Hatle wrote:
>>>> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
>>>
>>> Missing a commit message - and you wrote the book on proper commit
>>> messages :p
>>
>> In this case, what is wrong with the message. The change doesn't do
>> anything but use the new layer index module. Functionality remains the
>> same.
>
> Wouldn't you at minimum want to state exactly that last bit then? That isn't
> clear without carefully looking through the patch.
>
> Same for the other commit, except there what is displayed by bitbake-layers
> layerindex-fetch has changed a little bit.
Ok, I can do that.
--Mark
> Cheers,
> Paul
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 4/5] toaster/orm/management/commands/lsupdates.py: Use new layerindex module
@ 2017-08-03 22:40 ` Mark Hatle
0 siblings, 0 replies; 20+ messages in thread
From: Mark Hatle @ 2017-08-03 22:40 UTC (permalink / raw)
To: Paul Eggleton; +Cc: bitbake-devel, toaster
On 8/3/17 5:15 PM, Paul Eggleton wrote:
> On Thursday, 3 August 2017 8:18:03 PM CEST Mark Hatle wrote:
>> On 8/3/17 1:06 PM, Paul Eggleton wrote:
>>> On Friday, 28 July 2017 5:37:21 PM CEST Mark Hatle wrote:
>>>> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
>>>
>>> Missing a commit message - and you wrote the book on proper commit
>>> messages :p
>>
>> In this case, what is wrong with the message. The change doesn't do
>> anything but use the new layer index module. Functionality remains the
>> same.
>
> Wouldn't you at minimum want to state exactly that last bit then? That isn't
> clear without carefully looking through the patch.
>
> Same for the other commit, except there what is displayed by bitbake-layers
> layerindex-fetch has changed a little bit.
Ok, I can do that.
--Mark
> Cheers,
> Paul
>
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2017-08-03 22:40 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-07-28 15:37 [PATCH 0/5] Implement a new remote layer module Mark Hatle
2017-07-28 15:37 ` [PATCH 1/5] lib/layers: Initial layer and layer index implementeation Mark Hatle
2017-08-03 18:05 ` [bitbake-devel] " Paul Eggleton
2017-08-03 18:05 ` Paul Eggleton
2017-08-03 18:17 ` [bitbake-devel] " Mark Hatle
2017-08-03 18:17 ` Mark Hatle
2017-07-28 15:37 ` [PATCH 2/5] bitbake-selftest: Add layers module tests Mark Hatle
2017-07-28 15:37 ` [PATCH 3/5] lib/bblayers: Add support for the new layer modules Mark Hatle
2017-07-28 16:04 ` [bitbake-devel] " Mark Hatle
2017-07-28 16:04 ` Mark Hatle
2017-07-28 15:37 ` [PATCH 4/5] toaster/orm/management/commands/lsupdates.py: Use new layerindex module Mark Hatle
2017-08-03 18:06 ` [bitbake-devel] " Paul Eggleton
2017-08-03 18:06 ` Paul Eggleton
2017-08-03 18:18 ` [bitbake-devel] " Mark Hatle
2017-08-03 18:18 ` Mark Hatle
2017-08-03 22:15 ` [bitbake-devel] " Paul Eggleton
2017-08-03 22:15 ` Paul Eggleton
2017-08-03 22:40 ` [bitbake-devel] " Mark Hatle
2017-08-03 22:40 ` Mark Hatle
2017-07-28 15:37 ` [PATCH 5/5] bitbake-layers: disable parsing for layerindex commands Mark Hatle
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.