* [layerindex-web][PATCH 01/10] import_layer: Add --actual-branch option
2016-09-26 18:25 [layerindex-web][PATCH 00/10] Add Distribution, dependency and recommends detection, and import_project Liam R. Howlett
@ 2016-09-26 18:25 ` Liam R. Howlett
2016-09-26 18:25 ` [layerindex-web][PATCH 02/10] layerindex/tools/import_layer.py: Sanitize layer name Liam R. Howlett
` (8 subsequent siblings)
9 siblings, 0 replies; 19+ messages in thread
From: Liam R. Howlett @ 2016-09-26 18:25 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
Allow users to set actual-branch from the command line import of layers.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
---
layerindex/tools/import_layer.py | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/layerindex/tools/import_layer.py b/layerindex/tools/import_layer.py
index 0521e1c..9b5da22 100755
--- a/layerindex/tools/import_layer.py
+++ b/layerindex/tools/import_layer.py
@@ -200,6 +200,9 @@ def main():
parser.add_option("-q", "--quiet",
help = "Hide all output except error messages",
action="store_const", const=logging.ERROR, dest="loglevel")
+ parser.add_option("-a", "--actual-branch",
+ help = "Set actual branch",
+ action="store", dest="actual_branch")
options, args = parser.parse_args(sys.argv)
@@ -273,10 +276,13 @@ def main():
logger.error("Fetch failed: %s" % str(e))
sys.exit(1)
- actual_branch = ''
+ actual_branch = 'master'
+ if (options.actual_branch):
+ actual_branch = options.actual_branch
try:
- out = utils.runcmd("git checkout origin/master", repodir, logger=logger)
+ out = utils.runcmd("git checkout origin/%s" % actual_branch, repodir, logger=logger)
except subprocess.CalledProcessError:
+ actual_branch = None
branches = utils.runcmd("git branch -r", repodir, logger=logger)
for line in branches.splitlines():
if 'origin/HEAD ->' in line:
--
1.9.1
^ permalink raw reply related [flat|nested] 19+ messages in thread* [layerindex-web][PATCH 02/10] layerindex/tools/import_layer.py: Sanitize layer name.
2016-09-26 18:25 [layerindex-web][PATCH 00/10] Add Distribution, dependency and recommends detection, and import_project Liam R. Howlett
2016-09-26 18:25 ` [layerindex-web][PATCH 01/10] import_layer: Add --actual-branch option Liam R. Howlett
@ 2016-09-26 18:25 ` Liam R. Howlett
2016-10-03 22:47 ` Paul Eggleton
2016-09-26 18:25 ` [layerindex-web][PATCH 03/10] layerindex/tools/import_layer.py: Avoid failing if there is any layer to add Liam R. Howlett
` (7 subsequent siblings)
9 siblings, 1 reply; 19+ messages in thread
From: Liam R. Howlett @ 2016-09-26 18:25 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
Django will produce a cryptic error message if layers are added with
invalid names. Sanitize the layer names when trying to add them.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
---
layerindex/tools/import_layer.py | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/layerindex/tools/import_layer.py b/layerindex/tools/import_layer.py
index 9b5da22..8d9e8cb 100755
--- a/layerindex/tools/import_layer.py
+++ b/layerindex/tools/import_layer.py
@@ -181,6 +181,8 @@ def get_github_layerinfo(layer_url, username = None, password = None):
def main():
+ valid_layer_name = re.compile('[-\w]+$')
+
parser = optparse.OptionParser(
usage = """
%prog [options] <url> [name]""")
@@ -222,6 +224,10 @@ def main():
if layer_name.endswith('.git'):
layer_name = layer_name[:-4]
+ if not valid_layer_name.match(layer_name):
+ logger.error('Invlaid layer name "%s" - Layer name can only include letters, numbers and dashes.', layer_name)
+ sys.exit(1)
+
if options.github_auth:
if not ':' in options.github_auth:
logger.error('--github-auth value must be specified as username:password')
--
1.9.1
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [layerindex-web][PATCH 02/10] layerindex/tools/import_layer.py: Sanitize layer name.
2016-09-26 18:25 ` [layerindex-web][PATCH 02/10] layerindex/tools/import_layer.py: Sanitize layer name Liam R. Howlett
@ 2016-10-03 22:47 ` Paul Eggleton
0 siblings, 0 replies; 19+ messages in thread
From: Paul Eggleton @ 2016-10-03 22:47 UTC (permalink / raw)
To: Liam R. Howlett; +Cc: yocto
On Mon, 26 Sep 2016 14:25:30 Liam R. Howlett wrote:
> Django will produce a cryptic error message if layers are added with
> invalid names. Sanitize the layer names when trying to add them.
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
> ---
> layerindex/tools/import_layer.py | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/layerindex/tools/import_layer.py
> b/layerindex/tools/import_layer.py index 9b5da22..8d9e8cb 100755
> --- a/layerindex/tools/import_layer.py
> +++ b/layerindex/tools/import_layer.py
> @@ -181,6 +181,8 @@ def get_github_layerinfo(layer_url, username = None,
> password = None):
>
>
> def main():
> + valid_layer_name = re.compile('[-\w]+$')
> +
> parser = optparse.OptionParser(
> usage = """
> %prog [options] <url> [name]""")
> @@ -222,6 +224,10 @@ def main():
> if layer_name.endswith('.git'):
> layer_name = layer_name[:-4]
>
> + if not valid_layer_name.match(layer_name):
> + logger.error('Invlaid layer name "%s" - Layer name can only
> include letters, numbers and dashes.', layer_name)
> + sys.exit(1)
Typo "Invlaid", and the indenting is out here too.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 19+ messages in thread
* [layerindex-web][PATCH 03/10] layerindex/tools/import_layer.py: Avoid failing if there is any layer to add.
2016-09-26 18:25 [layerindex-web][PATCH 00/10] Add Distribution, dependency and recommends detection, and import_project Liam R. Howlett
2016-09-26 18:25 ` [layerindex-web][PATCH 01/10] import_layer: Add --actual-branch option Liam R. Howlett
2016-09-26 18:25 ` [layerindex-web][PATCH 02/10] layerindex/tools/import_layer.py: Sanitize layer name Liam R. Howlett
@ 2016-09-26 18:25 ` Liam R. Howlett
2016-10-03 22:53 ` Paul Eggleton
2016-09-26 18:25 ` [layerindex-web][PATCH 04/10] layerindex/utils: Update runcmd to decode binary strings to strings Liam R. Howlett
` (6 subsequent siblings)
9 siblings, 1 reply; 19+ messages in thread
From: Liam R. Howlett @ 2016-09-26 18:25 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
Subdirectories are scanned when adding layers. If any of the
subdirectories or root directory layers already exist in the database,
then the addition fails. This changes that behaviour to report the
failure as a warning and remove it from the list. That way, if a repo
has a new layer added it can be rescanned without issue. Layers being
rescanned are checked against the vcs_url to ensure there is not a name
collision. A name collision without the same vcs_url will still produce
a hard failure.
Note that multiple layers with the same vcs_url are supported in the
error reporting even though this should never happen.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
---
layerindex/tools/import_layer.py | 21 +++++++++++++++++++--
1 file changed, 19 insertions(+), 2 deletions(-)
diff --git a/layerindex/tools/import_layer.py b/layerindex/tools/import_layer.py
index 8d9e8cb..21b31f5 100755
--- a/layerindex/tools/import_layer.py
+++ b/layerindex/tools/import_layer.py
@@ -334,8 +334,21 @@ def main():
else:
subdir = ''
if LayerItem.objects.filter(name=layer.name).exists():
- logger.error('A layer named "%s" already exists in the database' % layer_name)
- sys.exit(1)
+ if LayerItem.objects.filter(name=layer.name).exclude(vcs_url=layer.vcs_url).exists():
+ conflict_list = LayerItem.objects.filter(name=layer.name).exclude(vcs_url=layer.vcs_url)
+ cln = ', '
+ conflict_list_urls = []
+ for conflict in conflict_list:
+ conflict_list_urls.append(conflict.vcs_url)
+ cln = cln.join(conflict_list_urls)
+ logger.error('A layer named "%s" already exists in the database. Possible name collision with %s.vcs_url = %s' % (layer.name, layer.name, cln))
+ sys.exit(1)
+ else:
+ logger.info('The layer named "%s" already exists in the database. Skipping this layer with same vcs_url' % layer.name)
+ layer_paths = [x for x in layer_paths if x != layerdir]
+ continue
+
+
logger.info('Creating layer %s' % layer.name)
# Guess layer type
@@ -411,6 +424,10 @@ def main():
layer.save()
+ if not layer_paths:
+ logger.error('No layers added.')
+ sys.exit(1);
+
if options.dryrun:
raise DryRunRollbackException()
except DryRunRollbackException:
--
1.9.1
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [layerindex-web][PATCH 03/10] layerindex/tools/import_layer.py: Avoid failing if there is any layer to add.
2016-09-26 18:25 ` [layerindex-web][PATCH 03/10] layerindex/tools/import_layer.py: Avoid failing if there is any layer to add Liam R. Howlett
@ 2016-10-03 22:53 ` Paul Eggleton
0 siblings, 0 replies; 19+ messages in thread
From: Paul Eggleton @ 2016-10-03 22:53 UTC (permalink / raw)
To: Liam R. Howlett; +Cc: yocto
On Mon, 26 Sep 2016 14:25:31 Liam R. Howlett wrote:
> Subdirectories are scanned when adding layers. If any of the
> subdirectories or root directory layers already exist in the database,
> then the addition fails. This changes that behaviour to report the
> failure as a warning and remove it from the list. That way, if a repo
> has a new layer added it can be rescanned without issue. Layers being
> rescanned are checked against the vcs_url to ensure there is not a name
> collision. A name collision without the same vcs_url will still produce
> a hard failure.
>
> Note that multiple layers with the same vcs_url are supported in the
> error reporting even though this should never happen.
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
> ---
> layerindex/tools/import_layer.py | 21 +++++++++++++++++++--
> 1 file changed, 19 insertions(+), 2 deletions(-)
>
> diff --git a/layerindex/tools/import_layer.py
> b/layerindex/tools/import_layer.py index 8d9e8cb..21b31f5 100755
> --- a/layerindex/tools/import_layer.py
> +++ b/layerindex/tools/import_layer.py
> @@ -334,8 +334,21 @@ def main():
> else:
> subdir = ''
> if LayerItem.objects.filter(name=layer.name).exists():
> - logger.error('A layer named "%s" already exists in the
> database' % layer_name)
> - sys.exit(1)
> + if
> LayerItem.objects.filter(name=layer.name).exclude(vcs_url=layer.vcs_url).ex
> ists():
> + conflict_list =
> LayerItem.objects.filter(name=layer.name).exclude(vcs_url=layer.vcs_url)
> + cln = ', '
> + conflict_list_urls = []
> + for conflict in conflict_list:
> + conflict_list_urls.append(conflict.vcs_url)
> + cln = cln.join(conflict_list_urls)
Not a huge deal, but is there any particular reason you didn't omit the
initial assignment and do this instead:
cln = ', '.join(conflict_list_urls)
>
> + if not layer_paths:
> + logger.error('No layers added.')
> + sys.exit(1);
> +
Indenting is out here (should always be four spaces for a block indent).
Also typo in the shortlog - "if there is no layer to add" and strictly
speaking there shouldn't be a period at the end.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 19+ messages in thread
* [layerindex-web][PATCH 04/10] layerindex/utils: Update runcmd to decode binary strings to strings.
2016-09-26 18:25 [layerindex-web][PATCH 00/10] Add Distribution, dependency and recommends detection, and import_project Liam R. Howlett
` (2 preceding siblings ...)
2016-09-26 18:25 ` [layerindex-web][PATCH 03/10] layerindex/tools/import_layer.py: Avoid failing if there is any layer to add Liam R. Howlett
@ 2016-09-26 18:25 ` Liam R. Howlett
2016-09-26 18:25 ` [layerindex-web][PATCH 05/10] layerindex: Add distribution to web interface and model Liam R. Howlett
` (5 subsequent siblings)
9 siblings, 0 replies; 19+ messages in thread
From: Liam R. Howlett @ 2016-09-26 18:25 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
Convert binary strings to strings and strip leading/trailing whitespace
prior to returning errors and output.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
---
layerindex/utils.py | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/layerindex/utils.py b/layerindex/utils.py
index 50268e0..23b81f5 100644
--- a/layerindex/utils.py
+++ b/layerindex/utils.py
@@ -32,7 +32,8 @@ def runcmd(cmd, destdir=None, printerr=True, logger=None):
execute command, raise CalledProcessError if fail
return output if succeed
"""
- #logger.debug("run cmd '%s' in %s" % (cmd, os.getcwd() if destdir is None else destdir))
+ if logger:
+ logger.debug("run cmd '%s' in %s" % (cmd, os.getcwd() if destdir is None else destdir))
out = tempfile.TemporaryFile()
try:
subprocess.check_call(cmd, stdout=out, stderr=out, cwd=destdir, shell=True)
@@ -40,6 +41,7 @@ def runcmd(cmd, destdir=None, printerr=True, logger=None):
out.seek(0)
if printerr:
output = out.read()
+ output = output.decode('ascii').strip()
if logger:
logger.error("%s" % output)
else:
@@ -49,7 +51,9 @@ def runcmd(cmd, destdir=None, printerr=True, logger=None):
out.seek(0)
output = out.read()
- #logger.debug("output: %s" % output.rstrip() )
+ output = output.decode('ascii').strip()
+ if logger:
+ logger.debug("output: %s" % output.rstrip() )
return output
def setup_django():
--
1.9.1
^ permalink raw reply related [flat|nested] 19+ messages in thread* [layerindex-web][PATCH 05/10] layerindex: Add distribution to web interface and model.
2016-09-26 18:25 [layerindex-web][PATCH 00/10] Add Distribution, dependency and recommends detection, and import_project Liam R. Howlett
` (3 preceding siblings ...)
2016-09-26 18:25 ` [layerindex-web][PATCH 04/10] layerindex/utils: Update runcmd to decode binary strings to strings Liam R. Howlett
@ 2016-09-26 18:25 ` Liam R. Howlett
2016-10-03 22:59 ` Paul Eggleton
2016-09-26 18:25 ` [layerindex-web][PATCH 06/10] layerindex/tools/import_project: Add import_project Liam R. Howlett
` (4 subsequent siblings)
9 siblings, 1 reply; 19+ messages in thread
From: Liam R. Howlett @ 2016-09-26 18:25 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
Add the distributions to the index. This looks a lot like the machines
and allows users to search for a particular distribution.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
---
layerindex/admin.py | 11 +++++
layerindex/models.py | 14 ++++++
layerindex/recipeparse.py | 5 +++
layerindex/restviews.py | 10 ++++-
layerindex/tools/import_classic.py | 2 +
layerindex/update_layer.py | 46 +++++++++++++++++++-
layerindex/urls.py | 3 ++
layerindex/urls_branch.py | 6 ++-
layerindex/views.py | 29 ++++++++++++-
templates/layerindex/detail.html | 23 ++++++++++
templates/layerindex/distributions.html | 76 +++++++++++++++++++++++++++++++++
templates/layerindex/layers.html | 1 +
templates/layerindex/machines.html | 1 +
templates/layerindex/recipes.html | 1 +
14 files changed, 224 insertions(+), 4 deletions(-)
create mode 100644 templates/layerindex/distributions.html
diff --git a/layerindex/admin.py b/layerindex/admin.py
index accb954..8164644 100644
--- a/layerindex/admin.py
+++ b/layerindex/admin.py
@@ -75,6 +75,16 @@ class MachineAdmin(admin.ModelAdmin):
def has_delete_permission(self, request, obj=None):
return False
+class DistributionAdmin(admin.ModelAdmin):
+ search_fields = ['name']
+ list_filter = ['layerbranch__layer__name', 'layerbranch__branch__name']
+ readonly_fields = Distribution._meta.get_all_field_names()
+ def has_add_permission(self, request, obj=None):
+ return False
+ def has_delete_permission(self, request, obj=None):
+ return False
+
+
class BBAppendAdmin(admin.ModelAdmin):
search_fields = ['filename']
list_filter = ['layerbranch__layer__name', 'layerbranch__branch__name']
@@ -111,6 +121,7 @@ admin.site.register(LayerNote, LayerNoteAdmin)
admin.site.register(Recipe, RecipeAdmin)
admin.site.register(RecipeFileDependency)
admin.site.register(Machine, MachineAdmin)
+admin.site.register(Distribution, DistributionAdmin)
admin.site.register(BBAppend, BBAppendAdmin)
admin.site.register(BBClass, BBClassAdmin)
admin.site.register(RecipeChangeset, RecipeChangesetAdmin)
diff --git a/layerindex/models.py b/layerindex/models.py
index 3fd16d4..6aec030 100644
--- a/layerindex/models.py
+++ b/layerindex/models.py
@@ -361,6 +361,20 @@ class Machine(models.Model):
def __str__(self):
return '%s (%s)' % (self.name, self.layerbranch.layer.name)
+class Distribution(models.Model):
+ layerbranch = models.ForeignKey(LayerBranch)
+ name = models.CharField(max_length=255)
+ description = models.CharField(max_length=255)
+
+ updated = models.DateTimeField(auto_now=True)
+
+ def vcs_web_url(self):
+ url = self.layerbranch.file_url(os.path.join('conf/distro/%s.conf' % self.name))
+ return url or ''
+
+ def __str__(self):
+ return '%s (%s)' % (self.name, self.layerbranch.layer.name)
+
class BBAppend(models.Model):
layerbranch = models.ForeignKey(LayerBranch)
diff --git a/layerindex/recipeparse.py b/layerindex/recipeparse.py
index 7f995ec..61d6fd4 100644
--- a/layerindex/recipeparse.py
+++ b/layerindex/recipeparse.py
@@ -152,6 +152,7 @@ def get_var_files(fn, varlist, d):
return varfiles
machine_conf_re = re.compile(r'conf/machine/([^/.]*).conf$')
+distro_conf_re = re.compile(r'conf/distro/([^/.]*).conf$')
bbclass_re = re.compile(r'classes/([^/.]*).bbclass$')
def detect_file_type(path, subdir_start):
typename = None
@@ -171,6 +172,10 @@ def detect_file_type(path, subdir_start):
if res:
typename = 'bbclass'
return (typename, None, res.group(1))
+ res = distro_conf_re.match(subpath)
+ if res:
+ typename = 'distribution'
+ return (typename, None, res.group(1))
if typename == 'recipe' or typename == 'bbappend':
if subdir_start:
diff --git a/layerindex/restviews.py b/layerindex/restviews.py
index b33d3d1..1efe11a 100644
--- a/layerindex/restviews.py
+++ b/layerindex/restviews.py
@@ -1,4 +1,4 @@
-from layerindex.models import Branch, LayerItem, LayerNote, LayerBranch, LayerDependency, Recipe, Machine
+from layerindex.models import Branch, LayerItem, LayerNote, LayerBranch, LayerDependency, Recipe, Machine, Distribution
from rest_framework import viewsets, serializers
from layerindex.querysethelper import params_to_queryset, get_search_tuple
@@ -56,3 +56,11 @@ class MachineSerializer(serializers.ModelSerializer):
class MachineViewSet(ParametricSearchableModelViewSet):
queryset = Machine.objects.all()
serializer_class = MachineSerializer
+
+class DistributionSerializer(serializers.ModelSerializer):
+ class Meta:
+ model = Distribution
+
+class DistributionViewSet(ParametricSearchableModelViewSet):
+ queryset = Distribution.objects.all()
+ serializer_class = DistributionSerializer
diff --git a/layerindex/tools/import_classic.py b/layerindex/tools/import_classic.py
index 45ccaa9..7d26ef6 100755
--- a/layerindex/tools/import_classic.py
+++ b/layerindex/tools/import_classic.py
@@ -160,6 +160,7 @@ def main():
layerdir_start = os.path.normpath(oeclassicpath) + os.sep
layerrecipes = Recipe.objects.filter(layerbranch=layerbranch)
layermachines = Machine.objects.filter(layerbranch=layerbranch)
+ layerdistributions = Distribution.objects.filter(layerbranch=layerbranch)
layerappends = BBAppend.objects.filter(layerbranch=layerbranch)
layerclasses = BBClass.objects.filter(layerbranch=layerbranch)
@@ -172,6 +173,7 @@ def main():
layerrecipes.delete()
layermachines.delete()
+ layerdistributions.delete()
layerappends.delete()
layerclasses.delete()
for root, dirs, files in os.walk(oeclassicpath):
diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index 13b508f..a937000 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -115,6 +115,19 @@ def update_machine_conf_file(path, machine):
break
machine.description = desc
+def update_distribution_conf_file(path, distribution):
+ logger.debug('Updating distribution %s' % path)
+ desc = ""
+ with open(path, 'r') as f:
+ for line in f:
+ if line.startswith('#@NAME:'):
+ desc = line[7:].strip()
+ if line.startswith('#@DESCRIPTION:'):
+ desc = line[14:].strip()
+ desc = re.sub(r'Distribtuion configuration for( running)*( an)*( the)*', '', desc)
+ break
+ distribution.description = desc
+
def main():
if LooseVersion(git.__version__) < '0.3.1':
logger.error("Version of GitPython is too old, please install GitPython (python-git) 0.3.1 or later in order to use this script")
@@ -161,7 +174,7 @@ def main():
utils.setup_django()
import settings
- from layerindex.models import LayerItem, LayerBranch, Recipe, RecipeFileDependency, Machine, BBAppend, BBClass
+ from layerindex.models import LayerItem, LayerBranch, Recipe, RecipeFileDependency, Machine, Distribution, BBAppend, BBClass
from django.db import transaction
logger.setLevel(options.loglevel)
@@ -269,6 +282,7 @@ def main():
layerdir_start = os.path.normpath(layerdir) + os.sep
layerrecipes = Recipe.objects.filter(layerbranch=layerbranch)
layermachines = Machine.objects.filter(layerbranch=layerbranch)
+ layerdistributions = Distribution.objects.filter(layerbranch=layerbranch)
layerappends = BBAppend.objects.filter(layerbranch=layerbranch)
layerclasses = BBClass.objects.filter(layerbranch=layerbranch)
if layerbranch.vcs_last_rev != topcommit.hexsha or options.reload:
@@ -384,6 +398,15 @@ def main():
else:
logger.warn("Renamed machine %s could not be found" % oldpath)
other_adds.append(diffitem)
+ elif oldtypename == 'distribution':
+ results = layerdistributions.filter(name=oldfilename)
+ if len(results):
+ logger.debug("Rename distribution %s to %s" % (results[0], newfilename))
+ results[0].name = newfilename
+ results[0].save()
+ else:
+ logger.warn("Renamed distribution %s could not be found" % oldpath)
+ other_adds.append(diffitem)
elif oldtypename == 'bbclass':
results = layerclasses.filter(name=oldfilename)
if len(results):
@@ -422,6 +445,8 @@ def main():
layerappends.filter(filepath=filepath).filter(filename=filename).delete()
elif typename == 'machine':
layermachines.filter(name=filename).delete()
+ elif typename == 'distribution':
+ layerdistributions.filter(name=filename).delete()
elif typename == 'bbclass':
layerclasses.filter(name=filename).delete()
@@ -452,6 +477,12 @@ def main():
machine.name = filename
update_machine_conf_file(os.path.join(repodir, path), machine)
machine.save()
+ elif typename == 'distribution':
+ distribution = Distribution()
+ distribution.layerbranch = layerbranch
+ distribution.name = filename
+ update_distribution_conf_file(os.path.join(repodir, path), distribution)
+ distribution.save()
elif typename == 'bbclass':
bbclass = BBClass()
bbclass.layerbranch = layerbranch
@@ -483,6 +514,12 @@ def main():
machine = results[0]
update_machine_conf_file(os.path.join(repodir, path), machine)
machine.save()
+ elif typename == 'distribuiton':
+ results = layerdistribuitons.filter(name=filename)
+ if results:
+ distribuiton = results[0]
+ update_distribuiton_conf_file(os.path.join(repodir, path), distribuiton)
+ distribuiton.save()
deps = RecipeFileDependency.objects.filter(layerbranch=layerbranch).filter(path=path)
for dep in deps:
@@ -523,6 +560,7 @@ def main():
layerrecipe_fns.append(fullpath)
layermachines.delete()
+ layerdistributions.delete()
layerappends.delete()
layerclasses.delete()
for root, dirs, files in os.walk(layerdir):
@@ -550,6 +588,12 @@ def main():
machine.name = filename
update_machine_conf_file(fullpath, machine)
machine.save()
+ elif typename == 'distribution':
+ distribution = Distribution()
+ distribution.layerbranch = layerbranch
+ distribution.name = filename
+ update_distribution_conf_file(fullpath, distribution)
+ distribution.save()
elif typename == 'bbclass':
bbclass = BBClass()
bbclass.layerbranch = layerbranch
diff --git a/layerindex/urls.py b/layerindex/urls.py
index cbd29ad..836543c 100644
--- a/layerindex/urls.py
+++ b/layerindex/urls.py
@@ -21,6 +21,7 @@ router.register(r'layerBranches', restviews.LayerBranchViewSet)
router.register(r'layerDependencies', restviews.LayerDependencyViewSet)
router.register(r'recipes', restviews.RecipeViewSet)
router.register(r'machines', restviews.MachineViewSet)
+router.register(r'distributions', restviews.DistributionViewSet)
urlpatterns = patterns('',
url(r'^$',
@@ -37,6 +38,8 @@ urlpatterns = patterns('',
RedirectView.as_view(url=reverse_lazy('recipe_search', args=('master',)), permanent=False)),
url(r'^machines/$',
RedirectView.as_view(url=reverse_lazy('machine_search', args=('master',)), permanent=False)),
+ url(r'^distributions/$',
+ RedirectView.as_view(url=reverse_lazy('distribution_search', args=('master',)), permanent=False)),
url(r'^submit/$', edit_layer_view, {'template_name': 'layerindex/submitlayer.html'}, name="submit_layer"),
url(r'^submit/thanks$',
diff --git a/layerindex/urls_branch.py b/layerindex/urls_branch.py
index 3313290..ba37c15 100644
--- a/layerindex/urls_branch.py
+++ b/layerindex/urls_branch.py
@@ -7,7 +7,7 @@
from django.conf.urls import *
from django.views.defaults import page_not_found
from django.core.urlresolvers import reverse_lazy
-from layerindex.views import LayerListView, RecipeSearchView, MachineSearchView, PlainTextListView, LayerDetailView, edit_layer_view, delete_layer_view, edit_layernote_view, delete_layernote_view, RedirectParamsView, DuplicatesView
+from layerindex.views import LayerListView, RecipeSearchView, MachineSearchView, DistributionSearchView, PlainTextListView, LayerDetailView, edit_layer_view, delete_layer_view, edit_layernote_view, delete_layernote_view, RedirectParamsView, DuplicatesView
urlpatterns = patterns('',
url(r'^$',
@@ -28,6 +28,10 @@ urlpatterns = patterns('',
MachineSearchView.as_view(
template_name='layerindex/machines.html'),
name='machine_search'),
+ url(r'^distributions/$',
+ DistributionSearchView.as_view(
+ template_name='layerindex/distributions.html'),
+ name='distribution_search'),
url(r'^edit/(?P<slug>[-\w]+)/$', edit_layer_view, {'template_name': 'layerindex/editlayer.html'}, name="edit_layer"),
url(r'^duplicates/$',
DuplicatesView.as_view(
diff --git a/layerindex/views.py b/layerindex/views.py
index 3f9525d..24ebad4 100644
--- a/layerindex/views.py
+++ b/layerindex/views.py
@@ -10,7 +10,7 @@ from django.http import HttpResponse, HttpResponseRedirect, HttpResponseForbidde
from django.core.urlresolvers import reverse, reverse_lazy, resolve
from django.core.exceptions import PermissionDenied
from django.template import RequestContext
-from layerindex.models import Branch, LayerItem, LayerMaintainer, LayerBranch, LayerDependency, LayerNote, Recipe, Machine, BBClass, BBAppend, RecipeChange, RecipeChangeset, ClassicRecipe
+from layerindex.models import Branch, LayerItem, LayerMaintainer, LayerBranch, LayerDependency, LayerNote, Recipe, Machine, Distribution, BBClass, BBAppend, RecipeChange, RecipeChangeset, ClassicRecipe
from datetime import datetime
from itertools import chain
from django.views.generic import TemplateView, DetailView, ListView
@@ -326,6 +326,7 @@ class LayerDetailView(DetailView):
if layerbranch:
context['layerbranch'] = layerbranch
context['machines'] = layerbranch.machine_set.order_by('name')
+ context['distributions'] = layerbranch.distribution_set.order_by('name')
context['appends'] = layerbranch.bbappend_set.order_by('filename')
context['classes'] = layerbranch.bbclass_set.order_by('name')
context['url_branch'] = self.kwargs['branch']
@@ -596,6 +597,32 @@ class MachineSearchView(ListView):
return context
+class DistributionSearchView(ListView):
+ context_object_name = 'distribution_list'
+ paginate_by = 50
+
+ def get_queryset(self):
+ _check_url_branch(self.kwargs)
+ query_string = self.request.GET.get('q', '')
+ init_qs = Distribution.objects.filter(layerbranch__branch__name=self.kwargs['branch'])
+ if query_string.strip():
+ entry_query = simplesearch.get_query(query_string, ['name', 'description'])
+ return init_qs.filter(entry_query).order_by('name', 'layerbranch__layer')
+
+ if 'q' in self.request.GET:
+ return init_qs.order_by('name', 'layerbranch__layer')
+
+ # Be consistent with RecipeSearchView
+ return Distribution.objects.none()
+
+ def get_context_data(self, **kwargs):
+ context = super(DistributionSearchView, self).get_context_data(**kwargs)
+ context['search_keyword'] = self.request.GET.get('q', '')
+ context['url_branch'] = self.kwargs['branch']
+ context['this_url_name'] = resolve(self.request.path_info).url_name
+ return context
+
+
class PlainTextListView(ListView):
def render_to_response(self, context):
"Returns a plain text response rendering of the template"
diff --git a/templates/layerindex/detail.html b/templates/layerindex/detail.html
index d0d11c0..f83b1f2 100644
--- a/templates/layerindex/detail.html
+++ b/templates/layerindex/detail.html
@@ -168,6 +168,9 @@
{% if classes.count > 0 %}
<li><a href="#classes" data-toggle="tab">Classes</a></li>
{% endif %}
+ {% if distributions.count > 0 %}
+ <li><a href="#distributions" data-toggle="tab">Distributions</a></li>
+ {% endif %}
</ul>
<div class="tab-content">
@@ -265,6 +268,26 @@
</table>
</div>
{% endif %}
+ {% if distributions.count > 0 %}
+ <div class="tab-pane" id="distributions">
+ <div class="navbar">
+ <div class="navbar-inner">
+ <a class="brand pull-left">{{ layeritem.name }} distributions</a>
+ </div>
+ </div>
+
+ <table class="table table-bordered">
+ <tbody>
+ {% for distribution in distributions %}
+ <tr>
+ <td><a href="{{ distribution.vcs_web_url }}">{{ distribution.name }}</a></td>
+ <td>{{ distribution.description }}</td>
+ </tr>
+ {% endfor %}
+ </tbody>
+ </table>
+ </div>
+ {% endif %}
</div>
diff --git a/templates/layerindex/distributions.html b/templates/layerindex/distributions.html
new file mode 100644
index 0000000..f9d7755
--- /dev/null
+++ b/templates/layerindex/distributions.html
@@ -0,0 +1,76 @@
+{% extends "base_toplevel.html" %}
+{% load i18n %}
+
+{% comment %}
+
+ layerindex-web - distribution index page template
+
+ Copyright (C) 2013 Intel Corporation
+ Copyright (C) 2016 Wind River Systems
+ Licensed under the MIT license, see COPYING.MIT for details
+
+{% endcomment %}
+
+
+<!--
+{% block title_append %} - distributions{% endblock %}
+-->
+
+{% block navs %}
+{% autoescape on %}
+ <li><a href="{% url 'layer_list' url_branch %}">Layers</a></li>
+ <li><a href="{% url 'recipe_search' url_branch %}">Recipes</a></li>
+ <li><a href="{% url 'machine_search' url_branch %}">Machines</a></li>
+ <li class="active"><a href="{% url 'distribution_search' url_branch %}">Distributions</a></li>
+{% endautoescape %}
+{% endblock %}
+
+
+{% block content_inner %}
+{% autoescape on %}
+
+
+ <div class="row-fluid">
+ <div class="input-append">
+ <form id="filter-form" action="{% url 'distribution_search' url_branch %}" method="get">
+ <input type="text" class="input-xxlarge" id="appendedInputButtons" placeholder="Search distributions" name="q" value="{{ search_keyword }}" />
+ <button class="btn" type="submit">search</button>
+ </form>
+ </div>
+ </div>
+
+{% if distribution_list %}
+ <table class="table table-striped table-bordered distributionstable">
+ <thead>
+ <tr>
+ <th>Distribution name</th>
+ <th class="span9">Description</th>
+ <th>Layer</th>
+ </tr>
+ </thead>
+
+ <tbody>
+ {% for distribution in distribution_list %}
+ <tr>
+ <td><a href="{{ distribution.vcs_web_url }}">{{ distribution.name }}</a></td>
+ <td>{{ distribution.description }}</td>
+ <td><a href="{% url 'layer_item' url_branch distribution.layerbranch.layer.name %}">{{ distribution.layerbranch.layer.name }}</a></td>
+ </tr>
+ {% endfor %}
+ </tbody>
+ </table>
+
+ {% if is_paginated %}
+ {% load pagination %}
+ {% pagination page_obj %}
+ {% endif %}
+{% else %}
+ {% if search_keyword %}
+ <p>No matching distributions in database.</p>
+ {% endif %}
+{% endif %}
+
+
+{% endautoescape %}
+
+{% endblock %}
diff --git a/templates/layerindex/layers.html b/templates/layerindex/layers.html
index cfc4ebd..1e647b3 100644
--- a/templates/layerindex/layers.html
+++ b/templates/layerindex/layers.html
@@ -21,6 +21,7 @@
<li class="active"><a href="{% url 'layer_list' url_branch %}">Layers</a></li>
<li><a href="{% url 'recipe_search' url_branch %}">Recipes</a></li>
<li><a href="{% url 'machine_search' url_branch %}">Machines</a></li>
+ <li><a href="{% url 'distribution_search' url_branch %}">Distributions</a></li>
{% endautoescape %}
{% endblock %}
diff --git a/templates/layerindex/machines.html b/templates/layerindex/machines.html
index e31433c..867d2cc 100644
--- a/templates/layerindex/machines.html
+++ b/templates/layerindex/machines.html
@@ -20,6 +20,7 @@
<li><a href="{% url 'layer_list' url_branch %}">Layers</a></li>
<li><a href="{% url 'recipe_search' url_branch %}">Recipes</a></li>
<li class="active"><a href="{% url 'machine_search' url_branch %}">Machines</a></li>
+ <li><a href="{% url 'distribution_search' url_branch %}">Distributions</a></li>
{% endautoescape %}
{% endblock %}
diff --git a/templates/layerindex/recipes.html b/templates/layerindex/recipes.html
index 74f3bb4..81e081c 100644
--- a/templates/layerindex/recipes.html
+++ b/templates/layerindex/recipes.html
@@ -20,6 +20,7 @@
<li><a href="{% url 'layer_list' url_branch %}">Layers</a></li>
<li class="active"><a href="{% url 'recipe_search' url_branch %}">Recipes</a></li>
<li><a href="{% url 'machine_search' url_branch %}">Machines</a></li>
+ <li><a href="{% url 'distribution_search' url_branch %}">Distributions</a></li>
{% endautoescape %}
{% endblock %}
--
1.9.1
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [layerindex-web][PATCH 05/10] layerindex: Add distribution to web interface and model.
2016-09-26 18:25 ` [layerindex-web][PATCH 05/10] layerindex: Add distribution to web interface and model Liam R. Howlett
@ 2016-10-03 22:59 ` Paul Eggleton
0 siblings, 0 replies; 19+ messages in thread
From: Paul Eggleton @ 2016-10-03 22:59 UTC (permalink / raw)
To: Liam R. Howlett; +Cc: yocto
On Mon, 26 Sep 2016 14:25:33 Liam R. Howlett wrote:
> Add the distributions to the index. This looks a lot like the machines
> and allows users to search for a particular distribution.
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
> ---
> layerindex/admin.py | 11 +++++
> layerindex/models.py | 14 ++++++
> layerindex/recipeparse.py | 5 +++
> layerindex/restviews.py | 10 ++++-
> layerindex/tools/import_classic.py | 2 +
> layerindex/update_layer.py | 46 +++++++++++++++++++-
> layerindex/urls.py | 3 ++
> layerindex/urls_branch.py | 6 ++-
> layerindex/views.py | 29 ++++++++++++-
> templates/layerindex/detail.html | 23 ++++++++++
> templates/layerindex/distributions.html | 76
Sorry, I know it's probably going to be a pain, I'd really like to be using
the term "distro" here rather than "distribution" - that would match up with
our existing variable and path naming, and at the same time will resolve a few
typos in this patch (e.g. "distribuiton").
Also, any change to the structure needs to be accompanied by a migration to
handle existing databases. Since you're working with Django 1.8 you'd be
creating its migrations rather than South's, and we had to start with a clean
slate for that so I can see perhaps why you missed them.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 19+ messages in thread
* [layerindex-web][PATCH 06/10] layerindex/tools/import_project: Add import_project
2016-09-26 18:25 [layerindex-web][PATCH 00/10] Add Distribution, dependency and recommends detection, and import_project Liam R. Howlett
` (4 preceding siblings ...)
2016-09-26 18:25 ` [layerindex-web][PATCH 05/10] layerindex: Add distribution to web interface and model Liam R. Howlett
@ 2016-09-26 18:25 ` Liam R. Howlett
2016-10-03 22:46 ` Paul Eggleton
2016-09-26 18:25 ` [layerindex-web][PATCH 07/10] layerindex: Detect dependencies from layer.conf files Liam R. Howlett
` (3 subsequent siblings)
9 siblings, 1 reply; 19+ messages in thread
From: Liam R. Howlett @ 2016-09-26 18:25 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
import_project will scan through a project and find any layer and add it
to the database by calling import_layer on each layer. This differs
from import_layer as it tires to figure out the remote url and uses the
subdirectory (if one exists) as the name.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
---
layerindex/tools/import_layer.py | 7 ++
layerindex/tools/import_project.py | 176 +++++++++++++++++++++++++++++++++++++
2 files changed, 183 insertions(+)
create mode 100755 layerindex/tools/import_project.py
diff --git a/layerindex/tools/import_layer.py b/layerindex/tools/import_layer.py
index 21b31f5..f84fd85 100755
--- a/layerindex/tools/import_layer.py
+++ b/layerindex/tools/import_layer.py
@@ -360,6 +360,13 @@ def main():
layerbranch = LayerBranch()
layerbranch.layer = layer
layerbranch.branch = master_branch
+ layerbranch.save()
+
+ if layer.name != settings.CORE_LAYER_NAME:
+ layerconfparser = LayerConfParse(logger=logger)
+ config_data = layerconfparser.parse_layer(layerdir)
+ layerconfparser.shutdown()
+ utils.set_layerbranch_collection_version(layerbranch, config_data, logger=logger)
if layerdir != repodir:
layerbranch.vcs_subdir = subdir
if actual_branch:
diff --git a/layerindex/tools/import_project.py b/layerindex/tools/import_project.py
new file mode 100755
index 0000000..0b91c08
--- /dev/null
+++ b/layerindex/tools/import_project.py
@@ -0,0 +1,176 @@
+#!/usr/bin/python3
+
+# Import a project into the database.
+# This will scan through the directories in a project and find any layer and
+# call import_layer.
+#
+#
+# Copyright (C) 2016 Wind River Systems
+# Author: Liam R. Howlett <liam.howlett@windriver.com>
+#
+# Licensed under the MIT license, see COPYING.MIT for details
+
+from git import Repo
+from urllib.parse import urlparse
+import logging
+import optparse
+import os, fnmatch
+import sys
+
+sys.path.insert(0, os.path.realpath(os.path.join(os.path.dirname(__file__), '..')))
+
+import import_layer
+import update
+
+import utils
+
+
+
+
+class ImportProject:
+ logger = utils.logger_create('ProjectIndexImport')
+
+ def find_layers(self, path):
+ self.logger.debug("finding layer..");
+ result = []
+ for root, dirs, files in os.walk(path, followlinks=True):
+ for name in fnmatch.filter(files, 'layer.conf'):
+ if not root.endswith('conf'):
+ continue
+
+ self.logger.debug("Found %s" % root)
+ result.append(root)
+ return result
+
+
+ def main(self):
+ parser = optparse.OptionParser(
+ usage = """
+ %prog [options] [directory]""")
+
+ parser.add_option("-d", "--debug",
+ help = "Enable debug output",
+ action="store_const", const=logging.DEBUG, dest="loglevel", default=logging.INFO)
+ parser.add_option("-n", "--dry-run",
+ help = "Don't write any data back to the database",
+ action="store_true", dest="dryrun")
+
+ self.options, args = parser.parse_args(sys.argv)
+
+ self.logger.setLevel(self.options.loglevel)
+
+ if len(args) == 1:
+ print("Please provide a directory.");
+ sys.exit(1)
+
+ install_dir = args[1]
+ lc_list = self.find_layers(install_dir)
+ self.add_core(lc_list)
+
+ for layer in lc_list:
+ self.add_layer(layer)
+
+ def add_layer(self, layer):
+ self.logger.debug("Processing layer %s" % layer);
+ try:
+ git_dir = utils.runcmd("git rev-parse --show-toplevel", destdir=layer, logger=self.logger)
+ except Exception as e:
+ self.logger.error("Cannot get root dir for layer %s: %s - Skipping." % (layer, str(e)))
+ return 1
+
+ self.logger.error("git_dir = %s" % git_dir);
+ repo = Repo(git_dir)
+ actual_branch = repo.active_branch.name
+
+
+ layer_name = layer.split('/')[-2]
+
+ layer_subdir = None
+ if os.path.basename(git_dir) != layer_name:
+ layer_subdir = layer_name
+
+ layer_name = self.get_layer_name(layer)
+
+ for i in [1, 2, 3]:
+ try:
+ git_url = utils.runcmd("git config --get remote.origin.url", destdir=git_dir, logger=self.logger)
+ except Exception as e:
+ self.logger.info("Cannot get remote.origin.url for git dir %s: %s" % (git_dir, str(e)))
+
+ if not os.path.exists(git_url):
+ # Assume this is remote.
+ self.logger.debug("Found git url = %s" % git_url)
+ break;
+ self.logger.debug("Iterating to find git url into %s" % git_dir)
+ git_dir = git_url
+
+ cmd = ['import_layer.py']
+ if self.options.loglevel == logging.DEBUG:
+ cmd.append("-d")
+ if layer_subdir:
+ cmd.append("-s")
+ cmd.append(layer_subdir)
+
+ if actual_branch:
+ cmd.append("-a")
+ cmd.append(actual_branch)
+ cmd.append(git_url)
+ cmd.append(layer_name)
+ prefix = "Calling"
+
+ if self.options.dryrun:
+ prefix = "Would Call"
+
+
+ self.logger.info("%s import_layer.main with %s for dir %s" % (prefix, str(cmd), layer))
+ sys.argv = cmd
+ if not self.options.dryrun:
+ try:
+ import_layer.main();
+ except SystemExit:
+ return 1
+ return 0
+
+ def get_layer_name(self, layerconfdir):
+ from layerconfparse import LayerConfParse
+
+ layer_name = layerconfdir.split('/')[-2]
+ self.logger.debug('getting layer %s' % layerconfdir)
+ layer_conf = os.path.join(layerconfdir, 'layer.conf')
+ if os.path.isfile(layer_conf):
+ with open(layer_conf) as conf:
+ for line in conf:
+ if 'BBLAYERS_LAYERINDEX_NAME' in line:
+ layer_name = line.split('=')[1].strip(' "\n')
+ return layer_name
+
+ def add_core(self, layers):
+ utils.setup_django()
+ import settings
+ for layer in layers:
+ layer_name = self.get_layer_name(layer)
+ if layer_name == settings.CORE_LAYER_NAME:
+ if self.add_layer(layer):
+ self.logger.info('Failed to add core layer\n')
+ self.update()
+ layers.remove(layer)
+
+ def update(self):
+ update_py = os.path.realpath(os.path.join(os.path.dirname(__file__), '../update.py'))
+ cmd = [update_py]
+ if self.options.loglevel == logging.DEBUG:
+ cmd.append("-d")
+ sys.argv = cmd
+ self.logger.info("update")
+ if not self.options.dryrun:
+ try:
+ update.main();
+ except SystemExit:
+ return 1
+
+ return 0
+
+
+if __name__ == "__main__":
+ x = ImportProject()
+ x.main()
--
1.9.1
^ permalink raw reply related [flat|nested] 19+ messages in thread* [layerindex-web][PATCH 07/10] layerindex: Detect dependencies from layer.conf files
2016-09-26 18:25 [layerindex-web][PATCH 00/10] Add Distribution, dependency and recommends detection, and import_project Liam R. Howlett
` (5 preceding siblings ...)
2016-09-26 18:25 ` [layerindex-web][PATCH 06/10] layerindex/tools/import_project: Add import_project Liam R. Howlett
@ 2016-09-26 18:25 ` Liam R. Howlett
2016-10-03 22:54 ` Paul Eggleton
2016-09-26 18:25 ` [layerindex-web][PATCH 08/10] layerindex: Add collection and version to layerbranch Liam R. Howlett
` (2 subsequent siblings)
9 siblings, 1 reply; 19+ messages in thread
From: Liam R. Howlett @ 2016-09-26 18:25 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
Read dependencies from layer.conf and try to create the LayerDependency
entry by looking up the correct database object. Dependencies are found
by layer name only - no collection support. layer.conf parsing is
handled by the bitbake code.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
---
layerindex/layerconfparse.py | 39 ++++++++++++++
layerindex/models.py | 4 ++
layerindex/recipeparse.py | 31 ++---------
layerindex/tools/import_layer.py | 8 +++
layerindex/update.py | 20 +++++++-
layerindex/utils.py | 108 +++++++++++++++++++++++++++++++++++++++
6 files changed, 181 insertions(+), 29 deletions(-)
create mode 100644 layerindex/layerconfparse.py
diff --git a/layerindex/layerconfparse.py b/layerindex/layerconfparse.py
new file mode 100644
index 0000000..19b390c
--- /dev/null
+++ b/layerindex/layerconfparse.py
@@ -0,0 +1,39 @@
+# Utility functions for parsing layer.conf using bitbake within layerindex-web
+#
+# Copyright (C) 2016 Wind River Systems
+# Author: Liam R. Howlett <liam.howlett@windriver.com>
+#
+# Licensed under the MIT license, see COPYING.MIT for details
+#
+
+import sys
+import os
+import os.path
+import utils
+import tempfile
+import re
+
+class LayerConfParse:
+ def __init__(self, enable_tracking=False, logger=None, bitbakepath=None):
+ import settings
+ self.logger = logger
+
+ if not bitbakepath:
+ fetchdir = settings.LAYER_FETCH_DIR
+ bitbakepath = os.path.join(fetchdir, 'bitbake')
+ self.bbpath = bitbakepath
+
+ # Set up BBPATH.
+ os.environ['BBPATH'] = str("%s" % self.bbpath)
+ self.tinfoil = utils.setup_tinfoil(self.bbpath, enable_tracking)
+ self.config_data_copy = bb.data.createCopy(self.tinfoil.config_data)
+
+
+ def parse_layer(self, layerdir):
+ utils.parse_layer_conf(layerdir, self.config_data_copy)
+ return self.config_data_copy
+
+ def shutdown(self):
+ self.tinfoil.shutdown()
+
+
diff --git a/layerindex/models.py b/layerindex/models.py
index 6aec030..2db8818 100644
--- a/layerindex/models.py
+++ b/layerindex/models.py
@@ -209,6 +209,9 @@ class LayerBranch(models.Model):
return "%s: %s" % (self.layer.name, self.branch.name)
+ def get_required(self):
+ return self.dependencies_set.filter(required=True)
+
class LayerMaintainer(models.Model):
MAINTAINER_STATUS_CHOICES = (
('A', 'Active'),
@@ -230,6 +233,7 @@ class LayerMaintainer(models.Model):
class LayerDependency(models.Model):
layerbranch = models.ForeignKey(LayerBranch, related_name='dependencies_set')
dependency = models.ForeignKey(LayerItem, related_name='dependents_set')
+ required = models.BooleanField(default=True)
class Meta:
verbose_name_plural = "Layer dependencies"
diff --git a/layerindex/recipeparse.py b/layerindex/recipeparse.py
index 61d6fd4..8a63117 100644
--- a/layerindex/recipeparse.py
+++ b/layerindex/recipeparse.py
@@ -20,31 +20,6 @@ class RecipeParseError(Exception):
def __str__(self):
return self.msg
-def _setup_tinfoil(bitbakepath, enable_tracking):
- sys.path.insert(0, bitbakepath + '/lib')
- import bb.tinfoil
- import bb.cooker
- import bb.data
- try:
- tinfoil = bb.tinfoil.Tinfoil(tracking=enable_tracking)
- except TypeError:
- # old API
- tinfoil = bb.tinfoil.Tinfoil()
- if enable_tracking:
- tinfoil.cooker.enableDataTracking()
- tinfoil.prepare(config_only = True)
-
- return tinfoil
-
-def _parse_layer_conf(layerdir, data):
- data.setVar('LAYERDIR', str(layerdir))
- if hasattr(bb, "cookerdata"):
- # Newer BitBake
- data = bb.cookerdata.parse_config_file(os.path.join(layerdir, "conf", "layer.conf"), data)
- else:
- # Older BitBake (1.18 and below)
- data = bb.cooker._parse(os.path.join(layerdir, "conf", "layer.conf"), data)
- data.expandVarref('LAYERDIR')
def init_parser(settings, branch, bitbakepath, enable_tracking=False, nocheckout=False, classic=False, logger=None):
@@ -97,7 +72,7 @@ def init_parser(settings, branch, bitbakepath, enable_tracking=False, nocheckout
tempdir = tempfile.mkdtemp(dir=settings.TEMP_BASE_DIR)
os.chdir(tempdir)
- tinfoil = _setup_tinfoil(bitbakepath, enable_tracking)
+ tinfoil = utils.setup_tinfoil(bitbakepath, enable_tracking)
# Ensure TMPDIR exists (or insane.bbclass will blow up trying to write to the QA log)
oe_tmpdir = tinfoil.config_data.getVar('TMPDIR', True)
@@ -125,7 +100,7 @@ def setup_layer(config_data, fetchdir, layerdir, layer, layerbranch):
# or across layers, but also because custom variable values might be
# set in layer.conf.
config_data_copy = bb.data.createCopy(config_data)
- _parse_layer_conf(layerdir, config_data_copy)
+ utils.parse_layer_conf(layerdir, config_data_copy)
for dep in layerbranch.dependencies_set.all():
depurldir = dep.dependency.get_fetch_dir()
deprepodir = os.path.join(fetchdir, depurldir)
@@ -133,7 +108,7 @@ def setup_layer(config_data, fetchdir, layerdir, layer, layerbranch):
if not deplayerbranch:
raise RecipeParseError('Dependency %s of layer %s does not have branch record for branch %s' % (dep.dependency.name, layer.name, layerbranch.branch.name))
deplayerdir = os.path.join(deprepodir, deplayerbranch.vcs_subdir)
- _parse_layer_conf(deplayerdir, config_data_copy)
+ utils.parse_layer_conf(deplayerdir, config_data_copy)
config_data_copy.delVar('LAYERDIR')
return config_data_copy
diff --git a/layerindex/tools/import_layer.py b/layerindex/tools/import_layer.py
index f84fd85..0a13d21 100755
--- a/layerindex/tools/import_layer.py
+++ b/layerindex/tools/import_layer.py
@@ -19,6 +19,7 @@ import glob
import utils
import logging
import subprocess
+from layerconfparse import LayerConfParse
class DryRunRollbackException(Exception):
pass
@@ -375,11 +376,18 @@ def main():
if layer.name != settings.CORE_LAYER_NAME:
if not core_layer:
core_layer = utils.get_layer(settings.CORE_LAYER_NAME)
+
if core_layer:
+ logger.debug('Adding dep %s to %s' % (core_layer.name, layer.name))
layerdep = LayerDependency()
layerdep.layerbranch = layerbranch
layerdep.dependency = core_layer
layerdep.save()
+ layerconfparser = LayerConfParse(logger=logger)
+ config_data = layerconfparser.parse_layer(layerdir)
+ layerconfparser.shutdown()
+ utils.add_dependencies(layerbranch, config_data, logger=logger)
+
# Get some extra meta-information
readme_files = glob.glob(os.path.join(layerdir, 'README*'))
diff --git a/layerindex/update.py b/layerindex/update.py
index 423eb53..ecd2380 100755
--- a/layerindex/update.py
+++ b/layerindex/update.py
@@ -16,6 +16,8 @@ import subprocess
import signal
from distutils.version import LooseVersion
import utils
+from layerconfparse import LayerConfParse
+
import warnings
warnings.filterwarnings("ignore", category=DeprecationWarning)
@@ -92,7 +94,7 @@ def main():
utils.setup_django()
import settings
- from layerindex.models import Branch, LayerItem
+ from layerindex.models import Branch, LayerItem, LayerDependency
logger.setLevel(options.loglevel)
@@ -201,6 +203,22 @@ def main():
# Interrupted by user, break out of loop
break
+ # Once indexed, then conf/layer.conf dependencies should be reevaluated.
+ layerconfparser = LayerConfParse(logger=logger, bitbakepath=bitbakepath)
+ for branch in branches:
+ for layer in layerquery:
+ urldir = layer.get_fetch_dir()
+ repodir = os.path.join(fetchdir, urldir)
+
+ layerbranch = layer.get_layerbranch(branch)
+ if layerbranch.vcs_subdir:
+ repodir = os.path.join(repodir, layerbranch.vcs_subdir)
+ config_data = layerconfparser.parse_layer(repodir)
+ utils.add_dependencies(layerbranch, config_data, logger=logger)
+
+ layerconfparser.shutdown()
+
+
finally:
utils.unlock_file(lockfile)
diff --git a/layerindex/utils.py b/layerindex/utils.py
index 23b81f5..f82f8c7 100644
--- a/layerindex/utils.py
+++ b/layerindex/utils.py
@@ -27,6 +27,114 @@ def get_layer(layername):
return res[0]
return None
+def get_dependency_layer(depname, version_str=None, logger=None):
+ from layerindex.models import LayerItem, LayerBranch
+
+ # Get any LayerBranch with a layer that has a name that matches the depname
+ res = list(LayerBranch.objects.filter(layer__name=depname))
+
+ # Nothing found, return.
+ if len(res) == 0:
+ return None
+
+ # If there is no version constraint, return the first one found.
+ if not version_str:
+ return res[0].layer
+
+ (operator, dep_version) = version_str.split()
+ for layerbranch in res:
+ layer_ver = layerbranch.version
+
+ # If there is no version in the found layer, then don't use this layer.
+ if not layer_ver:
+ continue
+
+ try:
+ success = bb.utils.vercmp_string_op(layer_ver, version_str, operator)
+ except bb.utils.VersionStringException as vse:
+ raise vse
+
+ if success:
+ return layerbranch.layer
+
+ return None
+
+def add_dependencies(layerbranch, config_data, logger=None):
+ _add_dependency("LAYERDEPENDS", 'dependency', layerbranch, config_data, logger)
+
+def _add_dependency(var, name, layerbranch, config_data, logger=None):
+ from layerindex.models import LayerBranch, LayerDependency
+
+ layer_name = layerbranch.layer.name
+ var_name = layer_name
+
+ dep_list = config_data.getVar("%s_%s" % (var, var_name), True)
+
+ if not dep_list:
+ return
+
+ try:
+ dep_dict = bb.utils.explode_dep_versions2(dep_list)
+ except bb.utils.VersionStringException as vse:
+ logger.debug('Error parsing %s_%s for %s\n%s' % (var, var_name, layer_name, str(vse)))
+ return
+
+ for dep, ver_list in list(dep_dict.items()):
+ ver_str = None
+ if ver_list:
+ ver_str = ver_list[0]
+
+ try:
+ dep_layer = get_dependency_layer(dep, ver_str, logger)
+ except bb.utils.VersionStringException as vse:
+ if logger:
+ logger.error('Error getting %s %s for %s\n%s' %(name, dep. layer_name, str(vse)))
+ continue
+
+ if not dep_layer:
+ if logger:
+ logger.error('Cannot resolve %s %s (version %s) for %s' % (name, dep, ver_str, layer_name))
+ continue
+
+ # Skip existing entries.
+ existing = list(LayerDependency.objects.filter(layerbranch=layerbranch).filter(dependency=dep_layer))
+ if existing:
+ logger.debug('Skipping %s - already a dependency for %s' % (dep, layer_name))
+ continue
+
+ if logger:
+ logger.debug('Adding %s %s to %s' % (name, dep_layer.name, layer_name))
+ layerdep = LayerDependency()
+ layerdep.layerbranch = layerbranch
+ layerdep.dependency = dep_layer
+ layerdep.save()
+
+def setup_tinfoil(bitbakepath, enable_tracking):
+ sys.path.insert(0, bitbakepath + '/lib')
+ import bb.tinfoil
+ import bb.cooker
+ import bb.data
+ try:
+ tinfoil = bb.tinfoil.Tinfoil(tracking=enable_tracking)
+ except TypeError:
+ # old API
+ tinfoil = bb.tinfoil.Tinfoil()
+ if enable_tracking:
+ tinfoil.cooker.enableDataTracking()
+ tinfoil.prepare(config_only = True)
+
+ return tinfoil
+
+def parse_layer_conf(layerdir, data):
+ data.setVar('LAYERDIR', str(layerdir))
+ if hasattr(bb, "cookerdata"):
+ # Newer BitBake
+ data = bb.cookerdata.parse_config_file(os.path.join(layerdir, "conf", "layer.conf"), data)
+ else:
+ # Older BitBake (1.18 and below)
+ data = bb.cooker._parse(os.path.join(layerdir, "conf", "layer.conf"), data)
+ data.expandVarref('LAYERDIR')
+
def runcmd(cmd, destdir=None, printerr=True, logger=None):
"""
execute command, raise CalledProcessError if fail
--
1.9.1
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [layerindex-web][PATCH 07/10] layerindex: Detect dependencies from layer.conf files
2016-09-26 18:25 ` [layerindex-web][PATCH 07/10] layerindex: Detect dependencies from layer.conf files Liam R. Howlett
@ 2016-10-03 22:54 ` Paul Eggleton
0 siblings, 0 replies; 19+ messages in thread
From: Paul Eggleton @ 2016-10-03 22:54 UTC (permalink / raw)
To: Liam R. Howlett; +Cc: yocto
On Mon, 26 Sep 2016 14:25:35 Liam R. Howlett wrote:
> Read dependencies from layer.conf and try to create the LayerDependency
> entry by looking up the correct database object. Dependencies are found
> by layer name only - no collection support. layer.conf parsing is
> handled by the bitbake code.
I see you did some refactoring here to move some of the code needed to parse
the layer conf to the utils module. Generally I do that kind of thing in a
separate patch, although I won't insist upon it - however if you do do it as
part of this patch then it needs to at least be mentioned as part of the
commit message.
> diff --git a/layerindex/models.py b/layerindex/models.py
> index 6aec030..2db8818 100644
> --- a/layerindex/models.py
> +++ b/layerindex/models.py
> @@ -209,6 +209,9 @@ class LayerBranch(models.Model):
> return "%s: %s" % (self.layer.name, self.branch.name)
>
>
> + def get_required(self):
> + return self.dependencies_set.filter(required=True)
> +
> class LayerMaintainer(models.Model):
> MAINTAINER_STATUS_CHOICES = (
> ('A', 'Active'),
> @@ -230,6 +233,7 @@ class LayerMaintainer(models.Model):
> class LayerDependency(models.Model):
> layerbranch = models.ForeignKey(LayerBranch,
> related_name='dependencies_set') dependency = models.ForeignKey(LayerItem,
> related_name='dependents_set')
> + required = models.BooleanField(default=True)
>
> class Meta:
> verbose_name_plural = "Layer dependencies"
This "required" field doesn't seem to be used in this commit - can you move
these two changes to be part of the "recommends" support patch where they are
used?
> --- a/layerindex/tools/import_layer.py
> +++ b/layerindex/tools/import_layer.py
> @@ -19,6 +19,7 @@ import glob
> import utils
> import logging
> import subprocess
> +from layerconfparse import LayerConfParse
>
> class DryRunRollbackException(Exception):
> pass
> @@ -375,11 +376,18 @@ def main():
> if layer.name != settings.CORE_LAYER_NAME:
> if not core_layer:
> core_layer =
> utils.get_layer(settings.CORE_LAYER_NAME)
> +
> if core_layer:
> + logger.debug('Adding dep %s to %s' %
> (core_layer.name, layer.name)) layerdep = LayerDependency()
> layerdep.layerbranch = layerbranch
> layerdep.dependency = core_layer
> layerdep.save()
> + layerconfparser = LayerConfParse(logger=logger)
> + config_data = layerconfparser.parse_layer(layerdir)
> + layerconfparser.shutdown()
> + utils.add_dependencies(layerbranch, config_data,
> logger=logger)
> +
Use try..finally here to ensure shutdown() gets run.
> # Get some extra meta-information
> readme_files = glob.glob(os.path.join(layerdir, 'README*'))
> diff --git a/layerindex/update.py b/layerindex/update.py
> index 423eb53..ecd2380 100755
> --- a/layerindex/update.py
> +++ b/layerindex/update.py
> @@ -16,6 +16,8 @@ import subprocess
> import signal
> from distutils.version import LooseVersion
> import utils
> +from layerconfparse import LayerConfParse
> +
>
> import warnings
> warnings.filterwarnings("ignore", category=DeprecationWarning)
> @@ -92,7 +94,7 @@ def main():
>
> utils.setup_django()
> import settings
> - from layerindex.models import Branch, LayerItem
> + from layerindex.models import Branch, LayerItem, LayerDependency
>
> logger.setLevel(options.loglevel)
>
> @@ -201,6 +203,22 @@ def main():
> # Interrupted by user, break out of loop
> break
>
> + # Once indexed, then conf/layer.conf dependencies should be
> reevaluated.
Shouldn't this be triggered solely when layer.conf changes rather than every
time?
> + layerconfparser = LayerConfParse(logger=logger,
> bitbakepath=bitbakepath)
> + for branch in branches:
> + for layer in layerquery:
> + urldir = layer.get_fetch_dir()
> + repodir = os.path.join(fetchdir, urldir)
> +
> + layerbranch = layer.get_layerbranch(branch)
> + if layerbranch.vcs_subdir:
> + repodir = os.path.join(repodir, layerbranch.vcs_subdir)
> + config_data = layerconfparser.parse_layer(repodir)
> + utils.add_dependencies(layerbranch, config_data,
> logger=logger)
> +
> + layerconfparser.shutdown()
Use try..finally here as well.
> diff --git a/layerindex/utils.py b/layerindex/utils.py
> index 23b81f5..f82f8c7 100644
> --- a/layerindex/utils.py
> +++ b/layerindex/utils.py
> @@ -27,6 +27,114 @@ def get_layer(layername):
> return res[0]
> return None
>
> +def get_dependency_layer(depname, version_str=None, logger=None):
> + from layerindex.models import LayerItem, LayerBranch
> +
> + # Get any LayerBranch with a layer that has a name that matches the
> depname
> + res = list(LayerBranch.objects.filter(layer__name=depname))
> +
> + # Nothing found, return.
> + if len(res) == 0:
> + return None
This is more simply written as:
if res:
return None
> +
> + # If there is no version constraint, return the first one found.
> + if not version_str:
> + return res[0].layer
> +
> + (operator, dep_version) = version_str.split()
> + for layerbranch in res:
> + layer_ver = layerbranch.version
> +
> + # If there is no version in the found layer, then don't use this
> layer.
> + if not layer_ver:
> + continue
> +
> + try:
> + success = bb.utils.vercmp_string_op(layer_ver, version_str,
> operator)
> + except bb.utils.VersionStringException as vse:
> + raise vse
> +
> + if success:
> + return layerbranch.layer
> +
> + return None
> +
> +def add_dependencies(layerbranch, config_data, logger=None):
> + _add_dependency("LAYERDEPENDS", 'dependency', layerbranch, config_data,
> logger)
> +
> +def _add_dependency(var, name, layerbranch, config_data, logger=None):
> + from layerindex.models import LayerBranch, LayerDependency
> +
> + layer_name = layerbranch.layer.name
> + var_name = layer_name
> +
> + dep_list = config_data.getVar("%s_%s" % (var, var_name), True)
> +
> + if not dep_list:
> + return
Indenting is out here.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 19+ messages in thread
* [layerindex-web][PATCH 08/10] layerindex: Add collection and version to layerbranch
2016-09-26 18:25 [layerindex-web][PATCH 00/10] Add Distribution, dependency and recommends detection, and import_project Liam R. Howlett
` (6 preceding siblings ...)
2016-09-26 18:25 ` [layerindex-web][PATCH 07/10] layerindex: Detect dependencies from layer.conf files Liam R. Howlett
@ 2016-09-26 18:25 ` Liam R. Howlett
2016-09-27 20:53 ` Mark Hatle
2016-10-03 22:54 ` Paul Eggleton
2016-09-26 18:25 ` [layerindex-web][PATCH 09/10] layerindexer: Add layer recommends support Liam R. Howlett
2016-09-26 18:25 ` [layerindex-web][PATCH 10/10] recipeparse: remove unnecessary else statement Liam R. Howlett
9 siblings, 2 replies; 19+ messages in thread
From: Liam R. Howlett @ 2016-09-26 18:25 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
Collection and version will be pulled from the layer.conf if it exists
and dependencies will be resolved by first checking for layers with the
dependency name and then checking for collections. It is necessary to
shutdown tinfoil to avoid bitbake complaining about multiple instances.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
---
layerindex/models.py | 2 ++
layerindex/tools/import_layer.py | 3 ---
layerindex/update.py | 1 +
layerindex/update_layer.py | 25 +++++++++++++++++++------
layerindex/utils.py | 16 ++++++++++++++--
5 files changed, 36 insertions(+), 11 deletions(-)
diff --git a/layerindex/models.py b/layerindex/models.py
index 2db8818..dcccb1f 100644
--- a/layerindex/models.py
+++ b/layerindex/models.py
@@ -130,6 +130,8 @@ class LayerItem(models.Model):
class LayerBranch(models.Model):
layer = models.ForeignKey(LayerItem)
branch = models.ForeignKey(Branch)
+ collection = models.CharField('Layer Collection', max_length=40, unique=True, null=True, help_text='Name of the layer that could be used in the list of dependencies - must be unique and can only contain letters, numbers and dashes')
+ version = models.CharField('Layer Version', max_length=10, null=True, help_text='The layer version for this particular branch.')
vcs_subdir = models.CharField('Repository subdirectory', max_length=40, blank=True, help_text='Subdirectory within the repository where the layer is located, if not in the root (usually only used if the repository contains more than one layer)')
vcs_last_fetch = models.DateTimeField('Last successful fetch', blank=True, null=True)
vcs_last_rev = models.CharField('Last revision fetched', max_length=80, blank=True)
diff --git a/layerindex/tools/import_layer.py b/layerindex/tools/import_layer.py
index 0a13d21..21dd802 100755
--- a/layerindex/tools/import_layer.py
+++ b/layerindex/tools/import_layer.py
@@ -383,9 +383,6 @@ def main():
layerdep.layerbranch = layerbranch
layerdep.dependency = core_layer
layerdep.save()
- layerconfparser = LayerConfParse(logger=logger)
- config_data = layerconfparser.parse_layer(layerdir)
- layerconfparser.shutdown()
utils.add_dependencies(layerbranch, config_data, logger=logger)
diff --git a/layerindex/update.py b/layerindex/update.py
index ecd2380..e15caf6 100755
--- a/layerindex/update.py
+++ b/layerindex/update.py
@@ -213,6 +213,7 @@ def main():
layerbranch = layer.get_layerbranch(branch)
if layerbranch.vcs_subdir:
repodir = os.path.join(repodir, layerbranch.vcs_subdir)
+
config_data = layerconfparser.parse_layer(repodir)
utils.add_dependencies(layerbranch, config_data, logger=logger)
diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index a937000..22c0dab 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -196,16 +196,18 @@ def main():
except recipeparse.RecipeParseError as e:
logger.error(str(e))
sys.exit(1)
+ config_data = bb.data.createCopy(tinfoil.config_data)
+ tinfoil.shutdown()
# Clear the default value of SUMMARY so that we can use DESCRIPTION instead if it hasn't been set
- tinfoil.config_data.setVar('SUMMARY', '')
+ config_data.setVar('SUMMARY', '')
# Clear the default value of DESCRIPTION so that we can see where it's not set
- tinfoil.config_data.setVar('DESCRIPTION', '')
+ config_data.setVar('DESCRIPTION', '')
# Clear the default value of HOMEPAGE ('unknown')
- tinfoil.config_data.setVar('HOMEPAGE', '')
+ config_data.setVar('HOMEPAGE', '')
# Set a blank value for LICENSE so that it doesn't cause the parser to die (e.g. with meta-ti -
# why won't they just fix that?!)
- tinfoil.config_data.setVar('LICENSE', '')
+ config_data.setVar('LICENSE', '')
try:
with transaction.atomic():
@@ -244,7 +246,7 @@ def main():
layerbranch = LayerBranch()
layerbranch.layer = layer
layerbranch.branch = branch
- layerbranch_source = layer.get_layerbranch('master')
+
if not layerbranch_source:
layerbranch_source = layer.get_layerbranch(None)
if layerbranch_source:
@@ -262,6 +264,9 @@ def main():
dep.layerbranch = layerbranch
dep.save()
+ layerbranch_source = layer.get_layerbranch('master')
+
+
if layerbranch.vcs_subdir and not options.nocheckout:
# Find latest commit in subdirectory
# A bit odd to do it this way but apparently there's no other way in the GitPython API
@@ -280,6 +285,14 @@ def main():
layerdir = os.path.join(repodir, layerbranch.vcs_subdir)
layerdir_start = os.path.normpath(layerdir) + os.sep
+
+ from layerconfparse import LayerConfParse
+ layerconfparser = LayerConfParse(logger=logger)
+ layer_config_data = layerconfparser.parse_layer(layerdir)
+ layerconfparser.shutdown()
+ utils.set_layerbranch_collection_version(layerbranch, layer_config_data, logger=logger)
+ layerbranch.save()
+
layerrecipes = Recipe.objects.filter(layerbranch=layerbranch)
layermachines = Machine.objects.filter(layerbranch=layerbranch)
layerdistributions = Distribution.objects.filter(layerbranch=layerbranch)
@@ -305,7 +318,7 @@ def main():
logger.info("Collecting data for layer %s on branch %s" % (layer.name, branchdesc))
try:
- config_data_copy = recipeparse.setup_layer(tinfoil.config_data, fetchdir, layerdir, layer, layerbranch)
+ config_data_copy = recipeparse.setup_layer(config_data, fetchdir, layerdir, layer, layerbranch)
except recipeparse.RecipeParseError as e:
logger.error(str(e))
sys.exit(1)
diff --git a/layerindex/utils.py b/layerindex/utils.py
index f82f8c7..2b49d64 100644
--- a/layerindex/utils.py
+++ b/layerindex/utils.py
@@ -30,8 +30,10 @@ def get_layer(layername):
def get_dependency_layer(depname, version_str=None, logger=None):
from layerindex.models import LayerItem, LayerBranch
- # Get any LayerBranch with a layer that has a name that matches the depname
- res = list(LayerBranch.objects.filter(layer__name=depname))
+ # Get any LayerBranch with a layer that has a name that matches depmod, or
+ # a LayerBranch that has the collection name depmod.
+ res = list(LayerBranch.objects.filter(layer__name=depname)) + \
+ list(LayerBranch.objects.filter(collection=depname))
# Nothing found, return.
if len(res) == 0:
@@ -104,11 +106,21 @@ def _add_dependency(var, name, layerbranch, config_data, logger=None):
if logger:
logger.debug('Adding %s %s to %s' % (name, dep_layer.name, layer_name))
+
layerdep = LayerDependency()
layerdep.layerbranch = layerbranch
layerdep.dependency = dep_layer
layerdep.save()
+def set_layerbranch_collection_version(layerbranch, config_data, logger=None):
+
+ layerbranch.collection = config_data.getVar('BBFILE_COLLECTIONS', True)
+ ver_str = "LAYERVERSION_"
+ if layerbranch.collection:
+ layerbranch.collection = layerbranch.collection.strip()
+ ver_str += layerbranch.collection
+ layerbranch.version = config_data.getVar(ver_str, True)
+
def setup_tinfoil(bitbakepath, enable_tracking):
sys.path.insert(0, bitbakepath + '/lib')
import bb.tinfoil
--
1.9.1
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [layerindex-web][PATCH 08/10] layerindex: Add collection and version to layerbranch
2016-09-26 18:25 ` [layerindex-web][PATCH 08/10] layerindex: Add collection and version to layerbranch Liam R. Howlett
@ 2016-09-27 20:53 ` Mark Hatle
2016-10-03 22:54 ` Paul Eggleton
1 sibling, 0 replies; 19+ messages in thread
From: Mark Hatle @ 2016-09-27 20:53 UTC (permalink / raw)
To: Liam R. Howlett, yocto; +Cc: paul.eggleton
On 9/26/16 1:25 PM, Liam R. Howlett wrote:
> Collection and version will be pulled from the layer.conf if it exists
> and dependencies will be resolved by first checking for layers with the
> dependency name and then checking for collections. It is necessary to
> shutdown tinfoil to avoid bitbake complaining about multiple instances.
I think I found a bug in this patch. The collection/version apparently needs to
be 'unique' in the layerbranch.
So if you have two layerbranch entries, associated with different branches (say
master and krogoth) you will get an error that they both have the same
collection name.
(This happened when I tried to use the admin interface to set the values.)
I will look into a solution for this problem.
--Mark
> Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
> ---
> layerindex/models.py | 2 ++
> layerindex/tools/import_layer.py | 3 ---
> layerindex/update.py | 1 +
> layerindex/update_layer.py | 25 +++++++++++++++++++------
> layerindex/utils.py | 16 ++++++++++++++--
> 5 files changed, 36 insertions(+), 11 deletions(-)
>
> diff --git a/layerindex/models.py b/layerindex/models.py
> index 2db8818..dcccb1f 100644
> --- a/layerindex/models.py
> +++ b/layerindex/models.py
> @@ -130,6 +130,8 @@ class LayerItem(models.Model):
> class LayerBranch(models.Model):
> layer = models.ForeignKey(LayerItem)
> branch = models.ForeignKey(Branch)
> + collection = models.CharField('Layer Collection', max_length=40, unique=True, null=True, help_text='Name of the layer that could be used in the list of dependencies - must be unique and can only contain letters, numbers and dashes')
> + version = models.CharField('Layer Version', max_length=10, null=True, help_text='The layer version for this particular branch.')
> vcs_subdir = models.CharField('Repository subdirectory', max_length=40, blank=True, help_text='Subdirectory within the repository where the layer is located, if not in the root (usually only used if the repository contains more than one layer)')
> vcs_last_fetch = models.DateTimeField('Last successful fetch', blank=True, null=True)
> vcs_last_rev = models.CharField('Last revision fetched', max_length=80, blank=True)
> diff --git a/layerindex/tools/import_layer.py b/layerindex/tools/import_layer.py
> index 0a13d21..21dd802 100755
> --- a/layerindex/tools/import_layer.py
> +++ b/layerindex/tools/import_layer.py
> @@ -383,9 +383,6 @@ def main():
> layerdep.layerbranch = layerbranch
> layerdep.dependency = core_layer
> layerdep.save()
> - layerconfparser = LayerConfParse(logger=logger)
> - config_data = layerconfparser.parse_layer(layerdir)
> - layerconfparser.shutdown()
> utils.add_dependencies(layerbranch, config_data, logger=logger)
>
>
> diff --git a/layerindex/update.py b/layerindex/update.py
> index ecd2380..e15caf6 100755
> --- a/layerindex/update.py
> +++ b/layerindex/update.py
> @@ -213,6 +213,7 @@ def main():
> layerbranch = layer.get_layerbranch(branch)
> if layerbranch.vcs_subdir:
> repodir = os.path.join(repodir, layerbranch.vcs_subdir)
> +
> config_data = layerconfparser.parse_layer(repodir)
> utils.add_dependencies(layerbranch, config_data, logger=logger)
>
> diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
> index a937000..22c0dab 100644
> --- a/layerindex/update_layer.py
> +++ b/layerindex/update_layer.py
> @@ -196,16 +196,18 @@ def main():
> except recipeparse.RecipeParseError as e:
> logger.error(str(e))
> sys.exit(1)
> + config_data = bb.data.createCopy(tinfoil.config_data)
> + tinfoil.shutdown()
>
> # Clear the default value of SUMMARY so that we can use DESCRIPTION instead if it hasn't been set
> - tinfoil.config_data.setVar('SUMMARY', '')
> + config_data.setVar('SUMMARY', '')
> # Clear the default value of DESCRIPTION so that we can see where it's not set
> - tinfoil.config_data.setVar('DESCRIPTION', '')
> + config_data.setVar('DESCRIPTION', '')
> # Clear the default value of HOMEPAGE ('unknown')
> - tinfoil.config_data.setVar('HOMEPAGE', '')
> + config_data.setVar('HOMEPAGE', '')
> # Set a blank value for LICENSE so that it doesn't cause the parser to die (e.g. with meta-ti -
> # why won't they just fix that?!)
> - tinfoil.config_data.setVar('LICENSE', '')
> + config_data.setVar('LICENSE', '')
>
> try:
> with transaction.atomic():
> @@ -244,7 +246,7 @@ def main():
> layerbranch = LayerBranch()
> layerbranch.layer = layer
> layerbranch.branch = branch
> - layerbranch_source = layer.get_layerbranch('master')
> +
> if not layerbranch_source:
> layerbranch_source = layer.get_layerbranch(None)
> if layerbranch_source:
> @@ -262,6 +264,9 @@ def main():
> dep.layerbranch = layerbranch
> dep.save()
>
> + layerbranch_source = layer.get_layerbranch('master')
> +
> +
> if layerbranch.vcs_subdir and not options.nocheckout:
> # Find latest commit in subdirectory
> # A bit odd to do it this way but apparently there's no other way in the GitPython API
> @@ -280,6 +285,14 @@ def main():
>
> layerdir = os.path.join(repodir, layerbranch.vcs_subdir)
> layerdir_start = os.path.normpath(layerdir) + os.sep
> +
> + from layerconfparse import LayerConfParse
> + layerconfparser = LayerConfParse(logger=logger)
> + layer_config_data = layerconfparser.parse_layer(layerdir)
> + layerconfparser.shutdown()
> + utils.set_layerbranch_collection_version(layerbranch, layer_config_data, logger=logger)
> + layerbranch.save()
> +
> layerrecipes = Recipe.objects.filter(layerbranch=layerbranch)
> layermachines = Machine.objects.filter(layerbranch=layerbranch)
> layerdistributions = Distribution.objects.filter(layerbranch=layerbranch)
> @@ -305,7 +318,7 @@ def main():
> logger.info("Collecting data for layer %s on branch %s" % (layer.name, branchdesc))
>
> try:
> - config_data_copy = recipeparse.setup_layer(tinfoil.config_data, fetchdir, layerdir, layer, layerbranch)
> + config_data_copy = recipeparse.setup_layer(config_data, fetchdir, layerdir, layer, layerbranch)
> except recipeparse.RecipeParseError as e:
> logger.error(str(e))
> sys.exit(1)
> diff --git a/layerindex/utils.py b/layerindex/utils.py
> index f82f8c7..2b49d64 100644
> --- a/layerindex/utils.py
> +++ b/layerindex/utils.py
> @@ -30,8 +30,10 @@ def get_layer(layername):
> def get_dependency_layer(depname, version_str=None, logger=None):
> from layerindex.models import LayerItem, LayerBranch
>
> - # Get any LayerBranch with a layer that has a name that matches the depname
> - res = list(LayerBranch.objects.filter(layer__name=depname))
> + # Get any LayerBranch with a layer that has a name that matches depmod, or
> + # a LayerBranch that has the collection name depmod.
> + res = list(LayerBranch.objects.filter(layer__name=depname)) + \
> + list(LayerBranch.objects.filter(collection=depname))
>
> # Nothing found, return.
> if len(res) == 0:
> @@ -104,11 +106,21 @@ def _add_dependency(var, name, layerbranch, config_data, logger=None):
>
> if logger:
> logger.debug('Adding %s %s to %s' % (name, dep_layer.name, layer_name))
> +
> layerdep = LayerDependency()
> layerdep.layerbranch = layerbranch
> layerdep.dependency = dep_layer
> layerdep.save()
>
> +def set_layerbranch_collection_version(layerbranch, config_data, logger=None):
> +
> + layerbranch.collection = config_data.getVar('BBFILE_COLLECTIONS', True)
> + ver_str = "LAYERVERSION_"
> + if layerbranch.collection:
> + layerbranch.collection = layerbranch.collection.strip()
> + ver_str += layerbranch.collection
> + layerbranch.version = config_data.getVar(ver_str, True)
> +
> def setup_tinfoil(bitbakepath, enable_tracking):
> sys.path.insert(0, bitbakepath + '/lib')
> import bb.tinfoil
>
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [layerindex-web][PATCH 08/10] layerindex: Add collection and version to layerbranch
2016-09-26 18:25 ` [layerindex-web][PATCH 08/10] layerindex: Add collection and version to layerbranch Liam R. Howlett
2016-09-27 20:53 ` Mark Hatle
@ 2016-10-03 22:54 ` Paul Eggleton
2016-10-04 13:51 ` Liam R. Howlett
1 sibling, 1 reply; 19+ messages in thread
From: Paul Eggleton @ 2016-10-03 22:54 UTC (permalink / raw)
To: Liam R. Howlett; +Cc: yocto
On Mon, 26 Sep 2016 14:25:36 Liam R. Howlett wrote:
> Collection and version will be pulled from the layer.conf if it exists
> and dependencies will be resolved by first checking for layers with the
> dependency name and then checking for collections. It is necessary to
> shutdown tinfoil to avoid bitbake complaining about multiple instances.
I'm not sure I understand why though, and there's not an explanation given
here. What you're doing as a workaround only works by chance and I suspect it
will break horribly when we do the tinfoil2 refactor in the next release (it's
not expected that you'd have datastores still around and be parsing recipes
when tinfoil has been shut down), so we *really* need to try to avoid shutting
down like this. Can you explain what you saw?
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [layerindex-web][PATCH 08/10] layerindex: Add collection and version to layerbranch
2016-10-03 22:54 ` Paul Eggleton
@ 2016-10-04 13:51 ` Liam R. Howlett
0 siblings, 0 replies; 19+ messages in thread
From: Liam R. Howlett @ 2016-10-04 13:51 UTC (permalink / raw)
To: Paul Eggleton; +Cc: yocto
* Paul Eggleton <paul.eggleton@linux.intel.com> [161003 18:54]:
> On Mon, 26 Sep 2016 14:25:36 Liam R. Howlett wrote:
> > Collection and version will be pulled from the layer.conf if it exists
> > and dependencies will be resolved by first checking for layers with the
> > dependency name and then checking for collections. It is necessary to
> > shutdown tinfoil to avoid bitbake complaining about multiple instances.
>
> I'm not sure I understand why though, and there's not an explanation given
> here. What you're doing as a workaround only works by chance and I suspect it
> will break horribly when we do the tinfoil2 refactor in the next release (it's
> not expected that you'd have datastores still around and be parsing recipes
> when tinfoil has been shut down), so we *really* need to try to avoid shutting
> down like this. Can you explain what you saw?
I received messages about having multiple copies of bitbake running if I
did not shut down tinfoil. I also hit `too many files opened` in the
update.py loop. I have since reworked this patch to pass in the tinfoil
instance to layerconfparser in this patch set. I was almost done v2
when I received your comments. I also noticed that I had shut down
tinfoil before it should have been shut down - in v2, I had moved my
shutdown call next to sys.exit(0). I feel much better about the way I
handle this in v2.
>
> Cheers,
> Paul
>
> --
>
> Paul Eggleton
> Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 19+ messages in thread
* [layerindex-web][PATCH 09/10] layerindexer: Add layer recommends support
2016-09-26 18:25 [layerindex-web][PATCH 00/10] Add Distribution, dependency and recommends detection, and import_project Liam R. Howlett
` (7 preceding siblings ...)
2016-09-26 18:25 ` [layerindex-web][PATCH 08/10] layerindex: Add collection and version to layerbranch Liam R. Howlett
@ 2016-09-26 18:25 ` Liam R. Howlett
2016-09-26 18:25 ` [layerindex-web][PATCH 10/10] recipeparse: remove unnecessary else statement Liam R. Howlett
9 siblings, 0 replies; 19+ messages in thread
From: Liam R. Howlett @ 2016-09-26 18:25 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
Parse layer.conf and add dependencies that are not required from
LAYERRECOMMENDS_<name>. Update the layerindex/template to support
recommends. Uses bitbake parsing code & checks versions.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
---
layerindex/models.py | 4 ++++
layerindex/tools/import_layer.py | 2 +-
layerindex/update.py | 1 +
layerindex/utils.py | 10 +++++++++-
templates/layerindex/detail.html | 32 +++++++++++++++++++++-----------
5 files changed, 36 insertions(+), 13 deletions(-)
diff --git a/layerindex/models.py b/layerindex/models.py
index dcccb1f..7286462 100644
--- a/layerindex/models.py
+++ b/layerindex/models.py
@@ -214,6 +214,10 @@ class LayerBranch(models.Model):
def get_required(self):
return self.dependencies_set.filter(required=True)
+ def get_recommends(self):
+ return self.dependencies_set.filter(required=False)
+
+
class LayerMaintainer(models.Model):
MAINTAINER_STATUS_CHOICES = (
('A', 'Active'),
diff --git a/layerindex/tools/import_layer.py b/layerindex/tools/import_layer.py
index 21dd802..33b7085 100755
--- a/layerindex/tools/import_layer.py
+++ b/layerindex/tools/import_layer.py
@@ -384,7 +384,7 @@ def main():
layerdep.dependency = core_layer
layerdep.save()
utils.add_dependencies(layerbranch, config_data, logger=logger)
-
+ utils.add_recommends(layerbranch, config_data, logger=logger)
# Get some extra meta-information
readme_files = glob.glob(os.path.join(layerdir, 'README*'))
diff --git a/layerindex/update.py b/layerindex/update.py
index e15caf6..39af03c 100755
--- a/layerindex/update.py
+++ b/layerindex/update.py
@@ -216,6 +216,7 @@ def main():
config_data = layerconfparser.parse_layer(repodir)
utils.add_dependencies(layerbranch, config_data, logger=logger)
+ utils.add_recommends(layerbranch, config_data, logger=logger)
layerconfparser.shutdown()
diff --git a/layerindex/utils.py b/layerindex/utils.py
index 2b49d64..a5cd2f6 100644
--- a/layerindex/utils.py
+++ b/layerindex/utils.py
@@ -64,16 +64,23 @@ def get_dependency_layer(depname, version_str=None, logger=None):
def add_dependencies(layerbranch, config_data, logger=None):
_add_dependency("LAYERDEPENDS", 'dependency', layerbranch, config_data, logger)
+def add_recommends(layerbranch, config_data, logger=None):
+ _add_dependency("LAYERRECOMMENDS", 'recommends', layerbranch, config_data, logger)
+
def _add_dependency(var, name, layerbranch, config_data, logger=None):
from layerindex.models import LayerBranch, LayerDependency
layer_name = layerbranch.layer.name
var_name = layer_name
+ if layerbranch.collection:
+ var_name = layerbranch.collection
+
+
dep_list = config_data.getVar("%s_%s" % (var, var_name), True)
if not dep_list:
- return
+ return
try:
dep_dict = bb.utils.explode_dep_versions2(dep_list)
@@ -93,6 +100,7 @@ def _add_dependency(var, name, layerbranch, config_data, logger=None):
logger.error('Error getting %s %s for %s\n%s' %(name, dep. layer_name, str(vse)))
continue
+ # No layer found.
if not dep_layer:
if logger:
logger.error('Cannot resolve %s %s (version %s) for %s' % (name, dep, ver_str, layer_name))
diff --git a/templates/layerindex/detail.html b/templates/layerindex/detail.html
index f83b1f2..e3967c5 100644
--- a/templates/layerindex/detail.html
+++ b/templates/layerindex/detail.html
@@ -139,17 +139,27 @@
<div class="span4 pull-right description">
{% if layerbranch.dependencies_set.count > 0 %}
- <div class="well dependency-well">
- <h3>Dependencies </h3>
- <p>The {{ layeritem.name }} layer depends upon:</p>
- <ul>
- {% for dep in layerbranch.dependencies_set.all %}
- <li><a href="{% url 'layer_item' url_branch dep.dependency.name %}">{{ dep.dependency.name }}</a></li>
- {% endfor %}
- </ul>
- </div> <!-- end of well -->
- {% endif %}
-
+ <div class="well dependency-well">
+ {% if layerbranch.get_required.count > 0 %}
+ <h3>Dependencies </h3>
+ <p>The {{ layeritem.name }} layer depends upon:</p>
+ <ul>
+ {% for dep in layerbranch.get_required %}
+ <li><a href="{% url 'layer_item' url_branch dep.dependency.name %}">{{ dep.dependency.name }}</a></li>
+ {% endfor %}
+ </ul>
+ {% endif %} <!-- end of layerbranch.get_required.count -->
+ {% if layerbranch.get_recommends.count > 0 %}
+ <h3>Recommends </h3>
+ <p>The {{ layeritem.name }} layer recommends:</p>
+ <ul>
+ {% for rec in layerbranch.get_recommends %}
+ <li><a href="{% url 'layer_item' url_branch rec.dependency.name %}">{{ rec.dependency.name }}</a></li>
+ {% endfor %}
+ </ul>
+ {% endif %} <!-- end of layerbranch.get_recommends.count -->
+ </div> <!-- end of well -->
+ {% endif %} <!-- end of layerbranch.dependencies_set.count -->
</div> <!-- end of span4 -->
</div> <!-- end of row-fluid -->
</div> <!-- end of container-fluid -->
--
1.9.1
^ permalink raw reply related [flat|nested] 19+ messages in thread* [layerindex-web][PATCH 10/10] recipeparse: remove unnecessary else statement.
2016-09-26 18:25 [layerindex-web][PATCH 00/10] Add Distribution, dependency and recommends detection, and import_project Liam R. Howlett
` (8 preceding siblings ...)
2016-09-26 18:25 ` [layerindex-web][PATCH 09/10] layerindexer: Add layer recommends support Liam R. Howlett
@ 2016-09-26 18:25 ` Liam R. Howlett
9 siblings, 0 replies; 19+ messages in thread
From: Liam R. Howlett @ 2016-09-26 18:25 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
Code clean up.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
---
layerindex/recipeparse.py | 17 ++++++++---------
1 file changed, 8 insertions(+), 9 deletions(-)
diff --git a/layerindex/recipeparse.py b/layerindex/recipeparse.py
index 8a63117..8cf75a1 100644
--- a/layerindex/recipeparse.py
+++ b/layerindex/recipeparse.py
@@ -142,15 +142,14 @@ def detect_file_type(path, subdir_start):
if res:
typename = 'machine'
return (typename, None, res.group(1))
- else:
- res = bbclass_re.match(subpath)
- if res:
- typename = 'bbclass'
- return (typename, None, res.group(1))
- res = distro_conf_re.match(subpath)
- if res:
- typename = 'distribution'
- return (typename, None, res.group(1))
+ res = bbclass_re.match(subpath)
+ if res:
+ typename = 'bbclass'
+ return (typename, None, res.group(1))
+ res = distro_conf_re.match(subpath)
+ if res:
+ typename = 'distribution'
+ return (typename, None, res.group(1))
if typename == 'recipe' or typename == 'bbappend':
if subdir_start:
--
1.9.1
^ permalink raw reply related [flat|nested] 19+ messages in thread