* [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project
@ 2016-10-07 15:57 Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 01/12] import_layer: Add --actual-branch option Liam R. Howlett
` (12 more replies)
0 siblings, 13 replies; 25+ messages in thread
From: Liam R. Howlett @ 2016-10-07 15:57 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
This set of patches adds a number of features to the layerindex code:
- Adds the ability to set the actual-branch to import_layer.
- Adds distro to the database and web interface.
- Adds collection & version information to layerbranch.
- Uses layer name and collection name to determine dependencies and
recommends.
- Adds import_project to simplify pulling all layers and openembedded-core
into a layer index.
Liam R. Howlett (11):
import_layer: Add --actual-branch option
layerindex/tools/import_layer.py: Sanitize layer name.
layerindex/tools/import_layer.py: Avoid failing if there is any layer
to add.
layerindex/utils: Update runcmd to decode binary strings to strings.
layerindex: Add distro to web interface and model.
layerindex/tools/import_project: Add import_project
layerindex/recipeparse.py: refactor setup_tinfoil,
checkout_layer_branch, parse_layer_conf to utils.py
layerindex: Detect dependencies from layer.conf files
layerindex: Add collection and version to layerbranch
layerindexer: Add layer recommends support
recipeparse: remove unnecessary else statement.
Mark Hatle (1):
layerindex/update_layer.py: Preserve the recipedependency files
layerindex/admin.py | 11 +++
layerindex/bulkchange.py | 4 +-
layerindex/layerconfparse.py | 50 +++++++++++
layerindex/models.py | 23 +++++
layerindex/recipeparse.py | 53 +++--------
layerindex/restviews.py | 10 ++-
layerindex/tools/import_classic.py | 2 +
layerindex/tools/import_layer.py | 47 +++++++++-
layerindex/tools/import_project.py | 180 +++++++++++++++++++++++++++++++++++++
layerindex/update.py | 42 ++++++++-
layerindex/update_layer.py | 87 ++++++++++++++++--
layerindex/urls.py | 3 +
layerindex/urls_branch.py | 6 +-
layerindex/utils.py | 159 +++++++++++++++++++++++++++++++-
layerindex/views.py | 29 +++++-
templates/layerindex/detail.html | 55 +++++++++---
templates/layerindex/distros.html | 76 ++++++++++++++++
templates/layerindex/layers.html | 1 +
templates/layerindex/machines.html | 1 +
templates/layerindex/recipes.html | 1 +
20 files changed, 767 insertions(+), 73 deletions(-)
create mode 100644 layerindex/layerconfparse.py
create mode 100755 layerindex/tools/import_project.py
create mode 100644 templates/layerindex/distros.html
--
1.9.1
^ permalink raw reply [flat|nested] 25+ messages in thread
* [layerindex-web][PATCH v2 01/12] import_layer: Add --actual-branch option
2016-10-07 15:57 [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project Liam R. Howlett
@ 2016-10-07 15:57 ` Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 02/12] layerindex/tools/import_layer.py: Sanitize layer name Liam R. Howlett
` (11 subsequent siblings)
12 siblings, 0 replies; 25+ messages in thread
From: Liam R. Howlett @ 2016-10-07 15:57 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
Allow users to set actual-branch from the command line import of layers.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
---
layerindex/tools/import_layer.py | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/layerindex/tools/import_layer.py b/layerindex/tools/import_layer.py
index 0521e1c..9b5da22 100755
--- a/layerindex/tools/import_layer.py
+++ b/layerindex/tools/import_layer.py
@@ -200,6 +200,9 @@ def main():
parser.add_option("-q", "--quiet",
help = "Hide all output except error messages",
action="store_const", const=logging.ERROR, dest="loglevel")
+ parser.add_option("-a", "--actual-branch",
+ help = "Set actual branch",
+ action="store", dest="actual_branch")
options, args = parser.parse_args(sys.argv)
@@ -273,10 +276,13 @@ def main():
logger.error("Fetch failed: %s" % str(e))
sys.exit(1)
- actual_branch = ''
+ actual_branch = 'master'
+ if (options.actual_branch):
+ actual_branch = options.actual_branch
try:
- out = utils.runcmd("git checkout origin/master", repodir, logger=logger)
+ out = utils.runcmd("git checkout origin/%s" % actual_branch, repodir, logger=logger)
except subprocess.CalledProcessError:
+ actual_branch = None
branches = utils.runcmd("git branch -r", repodir, logger=logger)
for line in branches.splitlines():
if 'origin/HEAD ->' in line:
--
1.9.1
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [layerindex-web][PATCH v2 02/12] layerindex/tools/import_layer.py: Sanitize layer name.
2016-10-07 15:57 [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 01/12] import_layer: Add --actual-branch option Liam R. Howlett
@ 2016-10-07 15:57 ` Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 03/12] layerindex/tools/import_layer.py: Avoid failing if there is any layer to add Liam R. Howlett
` (10 subsequent siblings)
12 siblings, 0 replies; 25+ messages in thread
From: Liam R. Howlett @ 2016-10-07 15:57 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
Django will produce a cryptic error message if layers are added with
invalid names. Sanitize the layer names when trying to add them.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
---
layerindex/tools/import_layer.py | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/layerindex/tools/import_layer.py b/layerindex/tools/import_layer.py
index 9b5da22..fefef0c 100755
--- a/layerindex/tools/import_layer.py
+++ b/layerindex/tools/import_layer.py
@@ -181,6 +181,8 @@ def get_github_layerinfo(layer_url, username = None, password = None):
def main():
+ valid_layer_name = re.compile('[-\w]+$')
+
parser = optparse.OptionParser(
usage = """
%prog [options] <url> [name]""")
@@ -222,6 +224,10 @@ def main():
if layer_name.endswith('.git'):
layer_name = layer_name[:-4]
+ if not valid_layer_name.match(layer_name):
+ logger.error('Invalid layer name "%s" - Layer name can only include letters, numbers and dashes.', layer_name)
+ sys.exit(1)
+
if options.github_auth:
if not ':' in options.github_auth:
logger.error('--github-auth value must be specified as username:password')
--
1.9.1
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [layerindex-web][PATCH v2 03/12] layerindex/tools/import_layer.py: Avoid failing if there is any layer to add.
2016-10-07 15:57 [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 01/12] import_layer: Add --actual-branch option Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 02/12] layerindex/tools/import_layer.py: Sanitize layer name Liam R. Howlett
@ 2016-10-07 15:57 ` Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 04/12] layerindex/utils: Update runcmd to decode binary strings to strings Liam R. Howlett
` (9 subsequent siblings)
12 siblings, 0 replies; 25+ messages in thread
From: Liam R. Howlett @ 2016-10-07 15:57 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
Subdirectories are scanned when adding layers. If any of the
subdirectories or root directory layers already exist in the database,
then the addition fails. This changes that behaviour to report the
failure as a warning and remove it from the list. That way, if a repo
has a new layer added it can be rescanned without issue. Layers being
rescanned are checked against the vcs_url to ensure there is not a name
collision. A name collision without the same vcs_url will still produce
a hard failure.
Note that multiple layers with the same vcs_url are supported in the
error reporting even though this should never happen.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
---
layerindex/tools/import_layer.py | 20 ++++++++++++++++++--
1 file changed, 18 insertions(+), 2 deletions(-)
diff --git a/layerindex/tools/import_layer.py b/layerindex/tools/import_layer.py
index fefef0c..184c5cc 100755
--- a/layerindex/tools/import_layer.py
+++ b/layerindex/tools/import_layer.py
@@ -334,8 +334,20 @@ def main():
else:
subdir = ''
if LayerItem.objects.filter(name=layer.name).exists():
- logger.error('A layer named "%s" already exists in the database' % layer_name)
- sys.exit(1)
+ if LayerItem.objects.filter(name=layer.name).exclude(vcs_url=layer.vcs_url).exists():
+ conflict_list = LayerItem.objects.filter(name=layer.name).exclude(vcs_url=layer.vcs_url)
+ conflict_list_urls = []
+ for conflict in conflict_list:
+ conflict_list_urls.append(conflict.vcs_url)
+ cln = ', '.join(conflict_list_urls)
+ logger.error('A layer named "%s" already exists in the database. Possible name collision with %s.vcs_url = %s' % (layer.name, layer.name, cln))
+ sys.exit(1)
+ else:
+ logger.info('The layer named "%s" already exists in the database. Skipping this layer with same vcs_url' % layer.name)
+ layer_paths = [x for x in layer_paths if x != layerdir]
+ continue
+
+
logger.info('Creating layer %s' % layer.name)
# Guess layer type
@@ -411,6 +423,10 @@ def main():
layer.save()
+ if not layer_paths:
+ logger.error('No layers added.')
+ sys.exit(1);
+
if options.dryrun:
raise DryRunRollbackException()
except DryRunRollbackException:
--
1.9.1
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [layerindex-web][PATCH v2 04/12] layerindex/utils: Update runcmd to decode binary strings to strings.
2016-10-07 15:57 [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project Liam R. Howlett
` (2 preceding siblings ...)
2016-10-07 15:57 ` [layerindex-web][PATCH v2 03/12] layerindex/tools/import_layer.py: Avoid failing if there is any layer to add Liam R. Howlett
@ 2016-10-07 15:57 ` Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 05/12] layerindex: Add distro to web interface and model Liam R. Howlett
` (8 subsequent siblings)
12 siblings, 0 replies; 25+ messages in thread
From: Liam R. Howlett @ 2016-10-07 15:57 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
Convert binary strings to strings and strip leading/trailing whitespace
prior to returning errors and output.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
---
layerindex/utils.py | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/layerindex/utils.py b/layerindex/utils.py
index 50268e0..23b81f5 100644
--- a/layerindex/utils.py
+++ b/layerindex/utils.py
@@ -32,7 +32,8 @@ def runcmd(cmd, destdir=None, printerr=True, logger=None):
execute command, raise CalledProcessError if fail
return output if succeed
"""
- #logger.debug("run cmd '%s' in %s" % (cmd, os.getcwd() if destdir is None else destdir))
+ if logger:
+ logger.debug("run cmd '%s' in %s" % (cmd, os.getcwd() if destdir is None else destdir))
out = tempfile.TemporaryFile()
try:
subprocess.check_call(cmd, stdout=out, stderr=out, cwd=destdir, shell=True)
@@ -40,6 +41,7 @@ def runcmd(cmd, destdir=None, printerr=True, logger=None):
out.seek(0)
if printerr:
output = out.read()
+ output = output.decode('ascii').strip()
if logger:
logger.error("%s" % output)
else:
@@ -49,7 +51,9 @@ def runcmd(cmd, destdir=None, printerr=True, logger=None):
out.seek(0)
output = out.read()
- #logger.debug("output: %s" % output.rstrip() )
+ output = output.decode('ascii').strip()
+ if logger:
+ logger.debug("output: %s" % output.rstrip() )
return output
def setup_django():
--
1.9.1
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [layerindex-web][PATCH v2 05/12] layerindex: Add distro to web interface and model.
2016-10-07 15:57 [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project Liam R. Howlett
` (3 preceding siblings ...)
2016-10-07 15:57 ` [layerindex-web][PATCH v2 04/12] layerindex/utils: Update runcmd to decode binary strings to strings Liam R. Howlett
@ 2016-10-07 15:57 ` Liam R. Howlett
2016-10-10 1:20 ` Paul Eggleton
2016-10-07 15:57 ` [layerindex-web][PATCH v2 06/12] layerindex/tools/import_project: Add import_project Liam R. Howlett
` (7 subsequent siblings)
12 siblings, 1 reply; 25+ messages in thread
From: Liam R. Howlett @ 2016-10-07 15:57 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
Add the distros to the index. This looks a lot like the machines
and allows users to search for a particular distro.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
---
layerindex/admin.py | 11 ++++++
layerindex/models.py | 14 +++++++
layerindex/recipeparse.py | 5 +++
layerindex/restviews.py | 10 ++++-
layerindex/tools/import_classic.py | 2 +
layerindex/update_layer.py | 46 ++++++++++++++++++++++-
layerindex/urls.py | 3 ++
layerindex/urls_branch.py | 6 ++-
layerindex/views.py | 29 ++++++++++++++-
templates/layerindex/detail.html | 23 ++++++++++++
templates/layerindex/distros.html | 76 ++++++++++++++++++++++++++++++++++++++
templates/layerindex/layers.html | 1 +
templates/layerindex/machines.html | 1 +
templates/layerindex/recipes.html | 1 +
14 files changed, 224 insertions(+), 4 deletions(-)
create mode 100644 templates/layerindex/distros.html
diff --git a/layerindex/admin.py b/layerindex/admin.py
index accb954..7339ce1 100644
--- a/layerindex/admin.py
+++ b/layerindex/admin.py
@@ -75,6 +75,16 @@ class MachineAdmin(admin.ModelAdmin):
def has_delete_permission(self, request, obj=None):
return False
+class DistroAdmin(admin.ModelAdmin):
+ search_fields = ['name']
+ list_filter = ['layerbranch__layer__name', 'layerbranch__branch__name']
+ readonly_fields = Distro._meta.get_all_field_names()
+ def has_add_permission(self, request, obj=None):
+ return False
+ def has_delete_permission(self, request, obj=None):
+ return False
+
+
class BBAppendAdmin(admin.ModelAdmin):
search_fields = ['filename']
list_filter = ['layerbranch__layer__name', 'layerbranch__branch__name']
@@ -111,6 +121,7 @@ admin.site.register(LayerNote, LayerNoteAdmin)
admin.site.register(Recipe, RecipeAdmin)
admin.site.register(RecipeFileDependency)
admin.site.register(Machine, MachineAdmin)
+admin.site.register(Distro, DistroAdmin)
admin.site.register(BBAppend, BBAppendAdmin)
admin.site.register(BBClass, BBClassAdmin)
admin.site.register(RecipeChangeset, RecipeChangesetAdmin)
diff --git a/layerindex/models.py b/layerindex/models.py
index 3fd16d4..2343ba7 100644
--- a/layerindex/models.py
+++ b/layerindex/models.py
@@ -361,6 +361,20 @@ class Machine(models.Model):
def __str__(self):
return '%s (%s)' % (self.name, self.layerbranch.layer.name)
+class Distro(models.Model):
+ layerbranch = models.ForeignKey(LayerBranch)
+ name = models.CharField(max_length=255)
+ description = models.CharField(max_length=255)
+
+ updated = models.DateTimeField(auto_now=True)
+
+ def vcs_web_url(self):
+ url = self.layerbranch.file_url(os.path.join('conf/distro/%s.conf' % self.name))
+ return url or ''
+
+ def __str__(self):
+ return '%s (%s)' % (self.name, self.layerbranch.layer.name)
+
class BBAppend(models.Model):
layerbranch = models.ForeignKey(LayerBranch)
diff --git a/layerindex/recipeparse.py b/layerindex/recipeparse.py
index 7f995ec..a9cecfc 100644
--- a/layerindex/recipeparse.py
+++ b/layerindex/recipeparse.py
@@ -152,6 +152,7 @@ def get_var_files(fn, varlist, d):
return varfiles
machine_conf_re = re.compile(r'conf/machine/([^/.]*).conf$')
+distro_conf_re = re.compile(r'conf/distro/([^/.]*).conf$')
bbclass_re = re.compile(r'classes/([^/.]*).bbclass$')
def detect_file_type(path, subdir_start):
typename = None
@@ -171,6 +172,10 @@ def detect_file_type(path, subdir_start):
if res:
typename = 'bbclass'
return (typename, None, res.group(1))
+ res = distro_conf_re.match(subpath)
+ if res:
+ typename = 'distro'
+ return (typename, None, res.group(1))
if typename == 'recipe' or typename == 'bbappend':
if subdir_start:
diff --git a/layerindex/restviews.py b/layerindex/restviews.py
index b33d3d1..57f1552 100644
--- a/layerindex/restviews.py
+++ b/layerindex/restviews.py
@@ -1,4 +1,4 @@
-from layerindex.models import Branch, LayerItem, LayerNote, LayerBranch, LayerDependency, Recipe, Machine
+from layerindex.models import Branch, LayerItem, LayerNote, LayerBranch, LayerDependency, Recipe, Machine, Distro
from rest_framework import viewsets, serializers
from layerindex.querysethelper import params_to_queryset, get_search_tuple
@@ -56,3 +56,11 @@ class MachineSerializer(serializers.ModelSerializer):
class MachineViewSet(ParametricSearchableModelViewSet):
queryset = Machine.objects.all()
serializer_class = MachineSerializer
+
+class DistroSerializer(serializers.ModelSerializer):
+ class Meta:
+ model = Distro
+
+class DistroViewSet(ParametricSearchableModelViewSet):
+ queryset = Distro.objects.all()
+ serializer_class = DistroSerializer
diff --git a/layerindex/tools/import_classic.py b/layerindex/tools/import_classic.py
index 45ccaa9..80c49af 100755
--- a/layerindex/tools/import_classic.py
+++ b/layerindex/tools/import_classic.py
@@ -160,6 +160,7 @@ def main():
layerdir_start = os.path.normpath(oeclassicpath) + os.sep
layerrecipes = Recipe.objects.filter(layerbranch=layerbranch)
layermachines = Machine.objects.filter(layerbranch=layerbranch)
+ layerdistros = Distro.objects.filter(layerbranch=layerbranch)
layerappends = BBAppend.objects.filter(layerbranch=layerbranch)
layerclasses = BBClass.objects.filter(layerbranch=layerbranch)
@@ -172,6 +173,7 @@ def main():
layerrecipes.delete()
layermachines.delete()
+ layerdistros.delete()
layerappends.delete()
layerclasses.delete()
for root, dirs, files in os.walk(oeclassicpath):
diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index 13b508f..3f50547 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -115,6 +115,19 @@ def update_machine_conf_file(path, machine):
break
machine.description = desc
+def update_distro_conf_file(path, distro):
+ logger.debug('Updating distro %s' % path)
+ desc = ""
+ with open(path, 'r') as f:
+ for line in f:
+ if line.startswith('#@NAME:'):
+ desc = line[7:].strip()
+ if line.startswith('#@DESCRIPTION:'):
+ desc = line[14:].strip()
+ desc = re.sub(r'Distribtuion configuration for( running)*( an)*( the)*', '', desc)
+ break
+ distro.description = desc
+
def main():
if LooseVersion(git.__version__) < '0.3.1':
logger.error("Version of GitPython is too old, please install GitPython (python-git) 0.3.1 or later in order to use this script")
@@ -161,7 +174,7 @@ def main():
utils.setup_django()
import settings
- from layerindex.models import LayerItem, LayerBranch, Recipe, RecipeFileDependency, Machine, BBAppend, BBClass
+ from layerindex.models import LayerItem, LayerBranch, Recipe, RecipeFileDependency, Machine, Distro, BBAppend, BBClass
from django.db import transaction
logger.setLevel(options.loglevel)
@@ -269,6 +282,7 @@ def main():
layerdir_start = os.path.normpath(layerdir) + os.sep
layerrecipes = Recipe.objects.filter(layerbranch=layerbranch)
layermachines = Machine.objects.filter(layerbranch=layerbranch)
+ layerdistros = Distro.objects.filter(layerbranch=layerbranch)
layerappends = BBAppend.objects.filter(layerbranch=layerbranch)
layerclasses = BBClass.objects.filter(layerbranch=layerbranch)
if layerbranch.vcs_last_rev != topcommit.hexsha or options.reload:
@@ -384,6 +398,15 @@ def main():
else:
logger.warn("Renamed machine %s could not be found" % oldpath)
other_adds.append(diffitem)
+ elif oldtypename == 'distro':
+ results = layerdistros.filter(name=oldfilename)
+ if len(results):
+ logger.debug("Rename distro %s to %s" % (results[0], newfilename))
+ results[0].name = newfilename
+ results[0].save()
+ else:
+ logger.warn("Renamed distro %s could not be found" % oldpath)
+ other_adds.append(diffitem)
elif oldtypename == 'bbclass':
results = layerclasses.filter(name=oldfilename)
if len(results):
@@ -422,6 +445,8 @@ def main():
layerappends.filter(filepath=filepath).filter(filename=filename).delete()
elif typename == 'machine':
layermachines.filter(name=filename).delete()
+ elif typename == 'distro':
+ layerdistros.filter(name=filename).delete()
elif typename == 'bbclass':
layerclasses.filter(name=filename).delete()
@@ -452,6 +477,12 @@ def main():
machine.name = filename
update_machine_conf_file(os.path.join(repodir, path), machine)
machine.save()
+ elif typename == 'distro':
+ distro = Distro()
+ distro.layerbranch = layerbranch
+ distro.name = filename
+ update_distro_conf_file(os.path.join(repodir, path), distro)
+ distro.save()
elif typename == 'bbclass':
bbclass = BBClass()
bbclass.layerbranch = layerbranch
@@ -483,6 +514,12 @@ def main():
machine = results[0]
update_machine_conf_file(os.path.join(repodir, path), machine)
machine.save()
+ elif typename == 'distro':
+ results = layerdistros.filter(name=filename)
+ if results:
+ distro = results[0]
+ update_distro_conf_file(os.path.join(repodir, path), distro)
+ distro.save()
deps = RecipeFileDependency.objects.filter(layerbranch=layerbranch).filter(path=path)
for dep in deps:
@@ -523,6 +560,7 @@ def main():
layerrecipe_fns.append(fullpath)
layermachines.delete()
+ layerdistros.delete()
layerappends.delete()
layerclasses.delete()
for root, dirs, files in os.walk(layerdir):
@@ -550,6 +588,12 @@ def main():
machine.name = filename
update_machine_conf_file(fullpath, machine)
machine.save()
+ elif typename == 'distro':
+ distro = Distro()
+ distro.layerbranch = layerbranch
+ distro.name = filename
+ update_distro_conf_file(fullpath, distro)
+ distro.save()
elif typename == 'bbclass':
bbclass = BBClass()
bbclass.layerbranch = layerbranch
diff --git a/layerindex/urls.py b/layerindex/urls.py
index cbd29ad..e37db24 100644
--- a/layerindex/urls.py
+++ b/layerindex/urls.py
@@ -21,6 +21,7 @@ router.register(r'layerBranches', restviews.LayerBranchViewSet)
router.register(r'layerDependencies', restviews.LayerDependencyViewSet)
router.register(r'recipes', restviews.RecipeViewSet)
router.register(r'machines', restviews.MachineViewSet)
+router.register(r'distros', restviews.DistroViewSet)
urlpatterns = patterns('',
url(r'^$',
@@ -37,6 +38,8 @@ urlpatterns = patterns('',
RedirectView.as_view(url=reverse_lazy('recipe_search', args=('master',)), permanent=False)),
url(r'^machines/$',
RedirectView.as_view(url=reverse_lazy('machine_search', args=('master',)), permanent=False)),
+ url(r'^distros/$',
+ RedirectView.as_view(url=reverse_lazy('distro_search', args=('master',)), permanent=False)),
url(r'^submit/$', edit_layer_view, {'template_name': 'layerindex/submitlayer.html'}, name="submit_layer"),
url(r'^submit/thanks$',
diff --git a/layerindex/urls_branch.py b/layerindex/urls_branch.py
index 3313290..dbfb8a1 100644
--- a/layerindex/urls_branch.py
+++ b/layerindex/urls_branch.py
@@ -7,7 +7,7 @@
from django.conf.urls import *
from django.views.defaults import page_not_found
from django.core.urlresolvers import reverse_lazy
-from layerindex.views import LayerListView, RecipeSearchView, MachineSearchView, PlainTextListView, LayerDetailView, edit_layer_view, delete_layer_view, edit_layernote_view, delete_layernote_view, RedirectParamsView, DuplicatesView
+from layerindex.views import LayerListView, RecipeSearchView, MachineSearchView, DistroSearchView, PlainTextListView, LayerDetailView, edit_layer_view, delete_layer_view, edit_layernote_view, delete_layernote_view, RedirectParamsView, DuplicatesView
urlpatterns = patterns('',
url(r'^$',
@@ -28,6 +28,10 @@ urlpatterns = patterns('',
MachineSearchView.as_view(
template_name='layerindex/machines.html'),
name='machine_search'),
+ url(r'^distros/$',
+ DistroSearchView.as_view(
+ template_name='layerindex/distros.html'),
+ name='distro_search'),
url(r'^edit/(?P<slug>[-\w]+)/$', edit_layer_view, {'template_name': 'layerindex/editlayer.html'}, name="edit_layer"),
url(r'^duplicates/$',
DuplicatesView.as_view(
diff --git a/layerindex/views.py b/layerindex/views.py
index 3f9525d..06045ae 100644
--- a/layerindex/views.py
+++ b/layerindex/views.py
@@ -10,7 +10,7 @@ from django.http import HttpResponse, HttpResponseRedirect, HttpResponseForbidde
from django.core.urlresolvers import reverse, reverse_lazy, resolve
from django.core.exceptions import PermissionDenied
from django.template import RequestContext
-from layerindex.models import Branch, LayerItem, LayerMaintainer, LayerBranch, LayerDependency, LayerNote, Recipe, Machine, BBClass, BBAppend, RecipeChange, RecipeChangeset, ClassicRecipe
+from layerindex.models import Branch, LayerItem, LayerMaintainer, LayerBranch, LayerDependency, LayerNote, Recipe, Machine, Distro, BBClass, BBAppend, RecipeChange, RecipeChangeset, ClassicRecipe
from datetime import datetime
from itertools import chain
from django.views.generic import TemplateView, DetailView, ListView
@@ -326,6 +326,7 @@ class LayerDetailView(DetailView):
if layerbranch:
context['layerbranch'] = layerbranch
context['machines'] = layerbranch.machine_set.order_by('name')
+ context['distros'] = layerbranch.distro_set.order_by('name')
context['appends'] = layerbranch.bbappend_set.order_by('filename')
context['classes'] = layerbranch.bbclass_set.order_by('name')
context['url_branch'] = self.kwargs['branch']
@@ -596,6 +597,32 @@ class MachineSearchView(ListView):
return context
+class DistroSearchView(ListView):
+ context_object_name = 'distro_list'
+ paginate_by = 50
+
+ def get_queryset(self):
+ _check_url_branch(self.kwargs)
+ query_string = self.request.GET.get('q', '')
+ init_qs = Distro.objects.filter(layerbranch__branch__name=self.kwargs['branch'])
+ if query_string.strip():
+ entry_query = simplesearch.get_query(query_string, ['name', 'description'])
+ return init_qs.filter(entry_query).order_by('name', 'layerbranch__layer')
+
+ if 'q' in self.request.GET:
+ return init_qs.order_by('name', 'layerbranch__layer')
+
+ # Be consistent with RecipeSearchView
+ return Distro.objects.none()
+
+ def get_context_data(self, **kwargs):
+ context = super(DistroSearchView, self).get_context_data(**kwargs)
+ context['search_keyword'] = self.request.GET.get('q', '')
+ context['url_branch'] = self.kwargs['branch']
+ context['this_url_name'] = resolve(self.request.path_info).url_name
+ return context
+
+
class PlainTextListView(ListView):
def render_to_response(self, context):
"Returns a plain text response rendering of the template"
diff --git a/templates/layerindex/detail.html b/templates/layerindex/detail.html
index d0d11c0..4bd7ecf 100644
--- a/templates/layerindex/detail.html
+++ b/templates/layerindex/detail.html
@@ -168,6 +168,9 @@
{% if classes.count > 0 %}
<li><a href="#classes" data-toggle="tab">Classes</a></li>
{% endif %}
+ {% if distros.count > 0 %}
+ <li><a href="#distros" data-toggle="tab">Distros</a></li>
+ {% endif %}
</ul>
<div class="tab-content">
@@ -265,6 +268,26 @@
</table>
</div>
{% endif %}
+ {% if distros.count > 0 %}
+ <div class="tab-pane" id="distros">
+ <div class="navbar">
+ <div class="navbar-inner">
+ <a class="brand pull-left">{{ layeritem.name }} distros</a>
+ </div>
+ </div>
+
+ <table class="table table-bordered">
+ <tbody>
+ {% for distro in distros %}
+ <tr>
+ <td><a href="{{ distro.vcs_web_url }}">{{ distro.name }}</a></td>
+ <td>{{ distro.description }}</td>
+ </tr>
+ {% endfor %}
+ </tbody>
+ </table>
+ </div>
+ {% endif %}
</div>
diff --git a/templates/layerindex/distros.html b/templates/layerindex/distros.html
new file mode 100644
index 0000000..7085584
--- /dev/null
+++ b/templates/layerindex/distros.html
@@ -0,0 +1,76 @@
+{% extends "base_toplevel.html" %}
+{% load i18n %}
+
+{% comment %}
+
+ layerindex-web - distro index page template
+
+ Copyright (C) 2013 Intel Corporation
+ Copyright (C) 2016 Wind River Systems
+ Licensed under the MIT license, see COPYING.MIT for details
+
+{% endcomment %}
+
+
+<!--
+{% block title_append %} - distros{% endblock %}
+-->
+
+{% block navs %}
+{% autoescape on %}
+ <li><a href="{% url 'layer_list' url_branch %}">Layers</a></li>
+ <li><a href="{% url 'recipe_search' url_branch %}">Recipes</a></li>
+ <li><a href="{% url 'machine_search' url_branch %}">Machines</a></li>
+ <li class="active"><a href="{% url 'distro_search' url_branch %}">Distros</a></li>
+{% endautoescape %}
+{% endblock %}
+
+
+{% block content_inner %}
+{% autoescape on %}
+
+
+ <div class="row-fluid">
+ <div class="input-append">
+ <form id="filter-form" action="{% url 'distro_search' url_branch %}" method="get">
+ <input type="text" class="input-xxlarge" id="appendedInputButtons" placeholder="Search distros" name="q" value="{{ search_keyword }}" />
+ <button class="btn" type="submit">search</button>
+ </form>
+ </div>
+ </div>
+
+{% if distro_list %}
+ <table class="table table-striped table-bordered distrostable">
+ <thead>
+ <tr>
+ <th>Distro name</th>
+ <th class="span9">Description</th>
+ <th>Layer</th>
+ </tr>
+ </thead>
+
+ <tbody>
+ {% for distro in distro_list %}
+ <tr>
+ <td><a href="{{ distro.vcs_web_url }}">{{ distro.name }}</a></td>
+ <td>{{ distro.description }}</td>
+ <td><a href="{% url 'layer_item' url_branch distro.layerbranch.layer.name %}">{{ distro.layerbranch.layer.name }}</a></td>
+ </tr>
+ {% endfor %}
+ </tbody>
+ </table>
+
+ {% if is_paginated %}
+ {% load pagination %}
+ {% pagination page_obj %}
+ {% endif %}
+{% else %}
+ {% if search_keyword %}
+ <p>No matching distros in database.</p>
+ {% endif %}
+{% endif %}
+
+
+{% endautoescape %}
+
+{% endblock %}
diff --git a/templates/layerindex/layers.html b/templates/layerindex/layers.html
index cfc4ebd..b11ff2f 100644
--- a/templates/layerindex/layers.html
+++ b/templates/layerindex/layers.html
@@ -21,6 +21,7 @@
<li class="active"><a href="{% url 'layer_list' url_branch %}">Layers</a></li>
<li><a href="{% url 'recipe_search' url_branch %}">Recipes</a></li>
<li><a href="{% url 'machine_search' url_branch %}">Machines</a></li>
+ <li><a href="{% url 'distro_search' url_branch %}">Distros</a></li>
{% endautoescape %}
{% endblock %}
diff --git a/templates/layerindex/machines.html b/templates/layerindex/machines.html
index e31433c..2a9f947 100644
--- a/templates/layerindex/machines.html
+++ b/templates/layerindex/machines.html
@@ -20,6 +20,7 @@
<li><a href="{% url 'layer_list' url_branch %}">Layers</a></li>
<li><a href="{% url 'recipe_search' url_branch %}">Recipes</a></li>
<li class="active"><a href="{% url 'machine_search' url_branch %}">Machines</a></li>
+ <li><a href="{% url 'distro_search' url_branch %}">Distros</a></li>
{% endautoescape %}
{% endblock %}
diff --git a/templates/layerindex/recipes.html b/templates/layerindex/recipes.html
index 74f3bb4..60a2667 100644
--- a/templates/layerindex/recipes.html
+++ b/templates/layerindex/recipes.html
@@ -20,6 +20,7 @@
<li><a href="{% url 'layer_list' url_branch %}">Layers</a></li>
<li class="active"><a href="{% url 'recipe_search' url_branch %}">Recipes</a></li>
<li><a href="{% url 'machine_search' url_branch %}">Machines</a></li>
+ <li><a href="{% url 'distro_search' url_branch %}">Distros</a></li>
{% endautoescape %}
{% endblock %}
--
1.9.1
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [layerindex-web][PATCH v2 06/12] layerindex/tools/import_project: Add import_project
2016-10-07 15:57 [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project Liam R. Howlett
` (4 preceding siblings ...)
2016-10-07 15:57 ` [layerindex-web][PATCH v2 05/12] layerindex: Add distro to web interface and model Liam R. Howlett
@ 2016-10-07 15:57 ` Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 07/12] layerindex/recipeparse.py: refactor setup_tinfoil, checkout_layer_branch, parse_layer_conf to utils.py Liam R. Howlett
` (6 subsequent siblings)
12 siblings, 0 replies; 25+ messages in thread
From: Liam R. Howlett @ 2016-10-07 15:57 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
import_project will scan through a project and find any layer and add it
to the database by calling import_layer on each layer. This differs
from import_layer as it tires to figure out the remote url and uses the
subdirectory (if one exists) as the name.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
---
layerindex/tools/import_project.py | 180 +++++++++++++++++++++++++++++++++++++
1 file changed, 180 insertions(+)
create mode 100755 layerindex/tools/import_project.py
diff --git a/layerindex/tools/import_project.py b/layerindex/tools/import_project.py
new file mode 100755
index 0000000..511282d
--- /dev/null
+++ b/layerindex/tools/import_project.py
@@ -0,0 +1,180 @@
+#!/usr/bin/python3
+
+# Import a project into the database.
+# This will scan through the directories in a project and find any layer and
+# call import_layer.
+#
+#
+# Copyright (C) 2016 Wind River Systems
+# Author: Liam R. Howlett <liam.howlett@windriver.com>
+#
+# Licensed under the MIT license, see COPYING.MIT for details
+
+from git import Repo
+from urllib.parse import urlparse
+import logging
+import optparse
+import os, fnmatch
+import sys
+
+sys.path.insert(0, os.path.realpath(os.path.join(os.path.dirname(__file__), '..')))
+
+import import_layer
+import update
+
+import utils
+
+
+
+
+class ImportProject:
+ logger = utils.logger_create('ProjectIndexImport')
+
+ def find_layers(self, path):
+ self.logger.debug("finding layer..");
+ result = []
+ for root, dirs, files in os.walk(path, followlinks=True):
+ for name in fnmatch.filter(files, 'layer.conf'):
+ if not root.endswith('conf'):
+ continue
+
+ self.logger.debug("Found %s" % root)
+ result.append(root)
+ return result
+
+
+ def main(self):
+ parser = optparse.OptionParser(
+ usage = """
+ %prog [options] [directory]""")
+
+ parser.add_option("-d", "--debug",
+ help = "Enable debug output",
+ action="store_const", const=logging.DEBUG, dest="loglevel", default=logging.INFO)
+ parser.add_option("-n", "--dry-run",
+ help = "Don't write any data back to the database",
+ action="store_true", dest="dryrun")
+
+ self.options, args = parser.parse_args(sys.argv)
+
+ self.logger.setLevel(self.options.loglevel)
+
+ if len(args) == 1:
+ print("Please provide a directory.");
+ sys.exit(1)
+
+ install_dir = args[1]
+ lc_list = self.find_layers(install_dir)
+ core_layer = self.add_core(lc_list)
+ if core_layer:
+ lc_list.remove(core_layer)
+
+
+
+ for layer in lc_list:
+ self.add_layer(layer)
+
+ def add_layer(self, layer):
+ self.logger.debug("Processing layer %s" % layer);
+ try:
+ git_dir = utils.runcmd("git rev-parse --show-toplevel", destdir=layer, logger=self.logger)
+ except Exception as e:
+ self.logger.error("Cannot get root dir for layer %s: %s - Skipping." % (layer, str(e)))
+ return 1
+
+ repo = Repo(git_dir)
+ actual_branch = repo.active_branch.name
+
+
+ layer_name = layer.split('/')[-2]
+
+ layer_subdir = None
+ if os.path.basename(git_dir) != layer_name:
+ layer_subdir = layer_name
+
+ layer_name = self.get_layer_name(layer)
+
+ for i in [1, 2, 3]:
+ try:
+ git_url = utils.runcmd("git config --get remote.origin.url", destdir=git_dir, logger=self.logger)
+ except Exception as e:
+ self.logger.info("Cannot get remote.origin.url for git dir %s: %s" % (git_dir, str(e)))
+
+ if not os.path.exists(git_url):
+ # Assume this is remote.
+ self.logger.debug("Found git url = %s" % git_url)
+ break;
+ self.logger.debug("Iterating to find git url into %s" % git_dir)
+ git_dir = git_url
+
+ cmd = ['import_layer.py']
+ if self.options.loglevel == logging.DEBUG:
+ cmd.append("-d")
+ if layer_subdir:
+ cmd.append("-s")
+ cmd.append(layer_subdir)
+
+ if actual_branch:
+ cmd.append("-a")
+ cmd.append(actual_branch)
+ cmd.append(git_url)
+ cmd.append(layer_name)
+ prefix = "Calling"
+
+ if self.options.dryrun:
+ prefix = "Would Call"
+
+
+ self.logger.info("%s import_layer.main with %s for dir %s" % (prefix, str(cmd), layer))
+ sys.argv = cmd
+ if not self.options.dryrun:
+ try:
+ import_layer.main();
+ except SystemExit as see:
+ return see.code
+ return 0
+
+ def get_layer_name(self, layerconfdir):
+ layer_name = layerconfdir.split('/')[-2]
+ self.logger.debug('getting layer %s' % layerconfdir)
+ layer_conf = os.path.join(layerconfdir, 'layer.conf')
+ if os.path.isfile(layer_conf):
+ with open(layer_conf) as conf:
+ for line in conf:
+ if 'BBLAYERS_LAYERINDEX_NAME' in line:
+ layer_name = line.split('=')[1].strip(' "\n')
+ return layer_name
+
+ def add_core(self, layers):
+ utils.setup_django()
+ core = None
+ import settings
+ for layer in layers:
+ layer_name = self.get_layer_name(layer)
+ if layer_name == settings.CORE_LAYER_NAME:
+ if self.add_layer(layer):
+ self.logger.info('Failed to add core layer\n')
+ core = layer
+ self.update()
+ break
+ return core
+
+ def update(self):
+ update_py = os.path.realpath(os.path.join(os.path.dirname(__file__), '../update.py'))
+ cmd = [update_py]
+ if self.options.loglevel == logging.DEBUG:
+ cmd.append("-d")
+ sys.argv = cmd
+ self.logger.info("update")
+ if not self.options.dryrun:
+ try:
+ update.main();
+ except SystemExit:
+ return 1
+
+ return 0
+
+
+if __name__ == "__main__":
+ x = ImportProject()
+ x.main()
--
1.9.1
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [layerindex-web][PATCH v2 07/12] layerindex/recipeparse.py: refactor setup_tinfoil, checkout_layer_branch, parse_layer_conf to utils.py
2016-10-07 15:57 [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project Liam R. Howlett
` (5 preceding siblings ...)
2016-10-07 15:57 ` [layerindex-web][PATCH v2 06/12] layerindex/tools/import_project: Add import_project Liam R. Howlett
@ 2016-10-07 15:57 ` Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 08/12] layerindex: Detect dependencies from layer.conf files Liam R. Howlett
` (5 subsequent siblings)
12 siblings, 0 replies; 25+ messages in thread
From: Liam R. Howlett @ 2016-10-07 15:57 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
Move functions to utils to be used by other classes.
_
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
---
layerindex/bulkchange.py | 4 ++--
layerindex/recipeparse.py | 39 +++-----------------------------------
layerindex/utils.py | 48 +++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 53 insertions(+), 38 deletions(-)
diff --git a/layerindex/bulkchange.py b/layerindex/bulkchange.py
index 8102571..ac9079f 100644
--- a/layerindex/bulkchange.py
+++ b/layerindex/bulkchange.py
@@ -42,9 +42,9 @@ def generate_patches(tinfoil, fetchdir, changeset, outputdir):
patchname = "%s.patch" % layer.name
patches.append(patchname)
layerfetchdir = os.path.join(fetchdir, layer.get_fetch_dir())
- recipeparse.checkout_layer_branch(layerbranch, layerfetchdir)
+ utils.checkout_layer_branch(layerbranch, layerfetchdir)
layerdir = os.path.join(layerfetchdir, layerbranch.vcs_subdir)
- config_data_copy = recipeparse.setup_layer(tinfoil.config_data, fetchdir, layerdir, layer, layerbranch)
+ config_data_copy = utils.setup_layer(tinfoil.config_data, fetchdir, layerdir, layer, layerbranch)
if outfile:
outfile.close()
outfile = open(os.path.join(tmpoutdir, patchname), 'w')
diff --git a/layerindex/recipeparse.py b/layerindex/recipeparse.py
index a9cecfc..91a083b 100644
--- a/layerindex/recipeparse.py
+++ b/layerindex/recipeparse.py
@@ -20,31 +20,6 @@ class RecipeParseError(Exception):
def __str__(self):
return self.msg
-def _setup_tinfoil(bitbakepath, enable_tracking):
- sys.path.insert(0, bitbakepath + '/lib')
- import bb.tinfoil
- import bb.cooker
- import bb.data
- try:
- tinfoil = bb.tinfoil.Tinfoil(tracking=enable_tracking)
- except TypeError:
- # old API
- tinfoil = bb.tinfoil.Tinfoil()
- if enable_tracking:
- tinfoil.cooker.enableDataTracking()
- tinfoil.prepare(config_only = True)
-
- return tinfoil
-
-def _parse_layer_conf(layerdir, data):
- data.setVar('LAYERDIR', str(layerdir))
- if hasattr(bb, "cookerdata"):
- # Newer BitBake
- data = bb.cookerdata.parse_config_file(os.path.join(layerdir, "conf", "layer.conf"), data)
- else:
- # Older BitBake (1.18 and below)
- data = bb.cooker._parse(os.path.join(layerdir, "conf", "layer.conf"), data)
- data.expandVarref('LAYERDIR')
def init_parser(settings, branch, bitbakepath, enable_tracking=False, nocheckout=False, classic=False, logger=None):
@@ -97,7 +72,7 @@ def init_parser(settings, branch, bitbakepath, enable_tracking=False, nocheckout
tempdir = tempfile.mkdtemp(dir=settings.TEMP_BASE_DIR)
os.chdir(tempdir)
- tinfoil = _setup_tinfoil(bitbakepath, enable_tracking)
+ tinfoil = utils.setup_tinfoil(bitbakepath, enable_tracking)
# Ensure TMPDIR exists (or insane.bbclass will blow up trying to write to the QA log)
oe_tmpdir = tinfoil.config_data.getVar('TMPDIR', True)
@@ -110,14 +85,6 @@ def init_parser(settings, branch, bitbakepath, enable_tracking=False, nocheckout
return (tinfoil, tempdir)
-def checkout_layer_branch(layerbranch, repodir, logger=None):
- if layerbranch.actual_branch:
- branchname = layerbranch.actual_branch
- else:
- branchname = layerbranch.branch.name
- out = utils.runcmd("git checkout origin/%s" % branchname, repodir, logger=logger)
- out = utils.runcmd("git clean -f -x", repodir, logger=logger)
-
def setup_layer(config_data, fetchdir, layerdir, layer, layerbranch):
# Parse layer.conf files for this layer and its dependencies
# This is necessary not just because BBPATH needs to be set in order
@@ -125,7 +92,7 @@ def setup_layer(config_data, fetchdir, layerdir, layer, layerbranch):
# or across layers, but also because custom variable values might be
# set in layer.conf.
config_data_copy = bb.data.createCopy(config_data)
- _parse_layer_conf(layerdir, config_data_copy)
+ utils.parse_layer_conf(layerdir, config_data_copy)
for dep in layerbranch.dependencies_set.all():
depurldir = dep.dependency.get_fetch_dir()
deprepodir = os.path.join(fetchdir, depurldir)
@@ -133,7 +100,7 @@ def setup_layer(config_data, fetchdir, layerdir, layer, layerbranch):
if not deplayerbranch:
raise RecipeParseError('Dependency %s of layer %s does not have branch record for branch %s' % (dep.dependency.name, layer.name, layerbranch.branch.name))
deplayerdir = os.path.join(deprepodir, deplayerbranch.vcs_subdir)
- _parse_layer_conf(deplayerdir, config_data_copy)
+ utils.parse_layer_conf(deplayerdir, config_data_copy)
config_data_copy.delVar('LAYERDIR')
return config_data_copy
diff --git a/layerindex/utils.py b/layerindex/utils.py
index 23b81f5..b634ce6 100644
--- a/layerindex/utils.py
+++ b/layerindex/utils.py
@@ -27,6 +27,54 @@ def get_layer(layername):
return res[0]
return None
+def setup_tinfoil(bitbakepath, enable_tracking):
+ sys.path.insert(0, bitbakepath + '/lib')
+ import bb.tinfoil
+ import bb.cooker
+ import bb.data
+ try:
+ tinfoil = bb.tinfoil.Tinfoil(tracking=enable_tracking)
+ except TypeError:
+ # old API
+ tinfoil = bb.tinfoil.Tinfoil()
+ if enable_tracking:
+ tinfoil.cooker.enableDataTracking()
+ tinfoil.prepare(config_only = True)
+
+ return tinfoil
+
+def checkout_layer_branch(layerbranch, repodir, logger=None):
+
+ branchname = layerbranch.branch.name
+ if layerbranch.actual_branch:
+ branchname = layerbranch.actual_branch
+
+ out = runcmd("git checkout origin/%s" % branchname, repodir, logger=logger)
+ out = runcmd("git clean -f -x", repodir, logger=logger)
+
+def is_layer_valid(layerdir):
+ conf_file = os.path.join(layerdir, "conf", "layer.conf")
+ if not os.path.isfile(conf_file):
+ return False
+ return True
+
+def parse_layer_conf(layerdir, data, logger=None):
+ conf_file = os.path.join(layerdir, "conf", "layer.conf")
+
+ if not is_layer_valid(layerdir):
+ if logger:
+ logger.error("Cannot find layer.conf: %s"% conf_file)
+ return
+
+ data.setVar('LAYERDIR', str(layerdir))
+ if hasattr(bb, "cookerdata"):
+ # Newer BitBake
+ data = bb.cookerdata.parse_config_file(conf_file, data)
+ else:
+ # Older BitBake (1.18 and below)
+ data = bb.cooker._parse(conf_file, data)
+ data.expandVarref('LAYERDIR')
+
def runcmd(cmd, destdir=None, printerr=True, logger=None):
"""
execute command, raise CalledProcessError if fail
--
1.9.1
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [layerindex-web][PATCH v2 08/12] layerindex: Detect dependencies from layer.conf files
2016-10-07 15:57 [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project Liam R. Howlett
` (6 preceding siblings ...)
2016-10-07 15:57 ` [layerindex-web][PATCH v2 07/12] layerindex/recipeparse.py: refactor setup_tinfoil, checkout_layer_branch, parse_layer_conf to utils.py Liam R. Howlett
@ 2016-10-07 15:57 ` Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 09/12] layerindex: Add collection and version to layerbranch Liam R. Howlett
` (4 subsequent siblings)
12 siblings, 0 replies; 25+ messages in thread
From: Liam R. Howlett @ 2016-10-07 15:57 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
Read dependencies from layer.conf and try to create the LayerDependency
entry by looking up the correct database object. Dependencies are found
by layer name only - no collection support. layer.conf parsing is
handled by the bitbake code.
Once all layers are added, the dependencies have to be rechecked in case
the layers are not added in order.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
---
layerindex/layerconfparse.py | 50 ++++++++++++++++++++++++
layerindex/tools/import_layer.py | 11 ++++++
layerindex/update.py | 41 +++++++++++++++++++-
layerindex/update_layer.py | 20 ++++++----
layerindex/utils.py | 82 ++++++++++++++++++++++++++++++++++++++++
5 files changed, 196 insertions(+), 8 deletions(-)
create mode 100644 layerindex/layerconfparse.py
diff --git a/layerindex/layerconfparse.py b/layerindex/layerconfparse.py
new file mode 100644
index 0000000..3a453bd
--- /dev/null
+++ b/layerindex/layerconfparse.py
@@ -0,0 +1,50 @@
+# Utility functions for parsing layer.conf using bitbake within layerindex-web
+#
+# Copyright (C) 2016 Wind River Systems
+# Author: Liam R. Howlett <liam.howlett@windriver.com>
+#
+# Licensed under the MIT license, see COPYING.MIT for details
+#
+
+import sys
+import os
+import os.path
+import utils
+import tempfile
+import re
+
+class LayerConfParse:
+ def __init__(self, enable_tracking=False, logger=None, bitbakepath=None, tinfoil=None):
+ import settings
+ self.logger = logger
+
+ if not bitbakepath:
+ fetchdir = settings.LAYER_FETCH_DIR
+ bitbakepath = os.path.join(fetchdir, 'bitbake')
+ self.bbpath = bitbakepath
+
+ # Set up BBPATH.
+ os.environ['BBPATH'] = str("%s" % self.bbpath)
+ self.tinfoil = tinfoil
+
+ if not self.tinfoil:
+ self.tinfoil = utils.setup_tinfoil(self.bbpath, enable_tracking)
+
+ self.config_data_copy = bb.data.createCopy(self.tinfoil.config_data)
+
+ def parse_layer(self, layerbranch, layerdir):
+
+ utils.checkout_layer_branch(layerbranch, layerdir, self.logger)
+
+
+ # This is not a valid layer, parsing will cause exception.
+ if not utils.is_layer_valid(layerdir):
+ return None
+
+ utils.parse_layer_conf(layerdir, self.config_data_copy, logger=self.logger)
+ return self.config_data_copy
+
+ def shutdown(self):
+ self.tinfoil.shutdown()
+
+
diff --git a/layerindex/tools/import_layer.py b/layerindex/tools/import_layer.py
index 184c5cc..1daaeb2 100755
--- a/layerindex/tools/import_layer.py
+++ b/layerindex/tools/import_layer.py
@@ -19,6 +19,7 @@ import glob
import utils
import logging
import subprocess
+from layerconfparse import LayerConfParse
class DryRunRollbackException(Exception):
pass
@@ -367,11 +368,21 @@ def main():
if layer.name != settings.CORE_LAYER_NAME:
if not core_layer:
core_layer = utils.get_layer(settings.CORE_LAYER_NAME)
+
if core_layer:
+ logger.debug('Adding dep %s to %s' % (core_layer.name, layer.name))
layerdep = LayerDependency()
layerdep.layerbranch = layerbranch
layerdep.dependency = core_layer
layerdep.save()
+ try:
+ layerconfparser = LayerConfParse(logger=logger)
+ config_data = layerconfparser.parse_layer(layerbranch, layerdir)
+ finally:
+ layerconfparser.shutdown()
+ if config_data:
+ utils.add_dependencies(layerbranch, config_data, logger=logger)
+
# Get some extra meta-information
readme_files = glob.glob(os.path.join(layerdir, 'README*'))
diff --git a/layerindex/update.py b/layerindex/update.py
index 423eb53..8684d36 100755
--- a/layerindex/update.py
+++ b/layerindex/update.py
@@ -16,6 +16,7 @@ import subprocess
import signal
from distutils.version import LooseVersion
import utils
+from layerconfparse import LayerConfParse
import warnings
warnings.filterwarnings("ignore", category=DeprecationWarning)
@@ -92,7 +93,7 @@ def main():
utils.setup_django()
import settings
- from layerindex.models import Branch, LayerItem
+ from layerindex.models import Branch, LayerItem, LayerDependency
logger.setLevel(options.loglevel)
@@ -169,6 +170,7 @@ def main():
# We now do this by calling out to a separate script; doing otherwise turned out to be
# unreliable due to leaking memory (we're using bitbake internals in a manner in which
# they never get used during normal operation).
+ last_rev = {}
for branch in branches:
for layer in layerquery:
if layer.vcs_url in failedrepos:
@@ -178,6 +180,10 @@ def main():
repodir = os.path.join(fetchdir, urldir)
branchobj = utils.get_branch(branch)
+ layerbranch = layer.get_layerbranch(branch)
+ if layerbranch:
+ last_rev[layerbranch] = layerbranch.vcs_last_rev
+
if branchobj.update_environment:
cmdprefix = branchobj.update_environment.get_command()
else:
@@ -201,6 +207,39 @@ def main():
# Interrupted by user, break out of loop
break
+ # Since update_layer may not be called in the correct order to have the
+ # dependencies created before trying to link them, we now have to loop
+ # back through all the branches and layers and try to link in the
+ # dependencies that may have been missed. Note that creating the
+ # dependencies is a best-effort and continues if they are not found.
+ for branch in branches:
+ try:
+ layerconfparser = LayerConfParse(logger=logger, bitbakepath=bitbakepath)
+ for layer in layerquery:
+
+ layerbranch = layer.get_layerbranch(branch)
+ # Skip layers that did not change.
+ if layerbranch and last_rev[layerbranch] == layerbranch.vcs_last_rev:
+ continue
+
+ urldir = layer.get_fetch_dir()
+ repodir = os.path.join(fetchdir, urldir)
+
+ layerbranch = layer.get_layerbranch(branch)
+ if not layerbranch:
+ continue
+
+ config_data = layerconfparser.parse_layer(layerbranch, repodir)
+ if not config_data:
+ logger.debug("Layer %s does not appear to have branch %s" % (layer.name, branch))
+ continue
+
+ utils.add_dependencies(layerbranch, config_data, logger=logger)
+ finally:
+ layerconfparser.shutdown()
+
+
+
finally:
utils.unlock_file(lockfile)
diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index 3f50547..addeb4c 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -196,7 +196,6 @@ def main():
except recipeparse.RecipeParseError as e:
logger.error(str(e))
sys.exit(1)
-
# Clear the default value of SUMMARY so that we can use DESCRIPTION instead if it hasn't been set
tinfoil.config_data.setVar('SUMMARY', '')
# Clear the default value of DESCRIPTION so that we can see where it's not set
@@ -244,7 +243,7 @@ def main():
layerbranch = LayerBranch()
layerbranch.layer = layer
layerbranch.branch = branch
- layerbranch_source = layer.get_layerbranch('master')
+ layerbranch_source = layer.get_layerbranch(branch)
if not layerbranch_source:
layerbranch_source = layer.get_layerbranch(None)
if layerbranch_source:
@@ -256,11 +255,6 @@ def main():
maintainer.id = None
maintainer.layerbranch = layerbranch
maintainer.save()
- for dep in layerbranch_source.dependencies_set.all():
- dep.pk = None
- dep.id = None
- dep.layerbranch = layerbranch
- dep.save()
if layerbranch.vcs_subdir and not options.nocheckout:
# Find latest commit in subdirectory
@@ -280,6 +274,17 @@ def main():
layerdir = os.path.join(repodir, layerbranch.vcs_subdir)
layerdir_start = os.path.normpath(layerdir) + os.sep
+
+ from layerconfparse import LayerConfParse
+ layerconfparser = LayerConfParse(logger=logger, tinfoil=tinfoil)
+ layer_config_data = layerconfparser.parse_layer(layerbranch, layerdir)
+ if not layer_config_data:
+ logger.info("Skipping update of layer %s for branch %s - conf/layer.conf may have parse issues" % (layer.name, branchdesc))
+ layerconfparser.shutdown()
+ sys.exit(1)
+ utils.add_dependencies(layerbranch, layer_config_data, logger=logger)
+ layerbranch.save()
+
layerrecipes = Recipe.objects.filter(layerbranch=layerbranch)
layermachines = Machine.objects.filter(layerbranch=layerbranch)
layerdistros = Distro.objects.filter(layerbranch=layerbranch)
@@ -655,6 +660,7 @@ def main():
import traceback
traceback.print_exc()
+ tinfoil.shutdown()
shutil.rmtree(tempdir)
sys.exit(0)
diff --git a/layerindex/utils.py b/layerindex/utils.py
index b634ce6..018478e 100644
--- a/layerindex/utils.py
+++ b/layerindex/utils.py
@@ -27,6 +27,88 @@ def get_layer(layername):
return res[0]
return None
+def get_dependency_layer(depname, version_str=None, logger=None):
+ from layerindex.models import LayerItem, LayerBranch
+
+ # Get any LayerBranch with a layer that has a name that matches the depname
+ res = list(LayerBranch.objects.filter(layer__name=depname))
+
+ # Nothing found, return.
+ if not res:
+ return None
+
+ # If there is no version constraint, return the first one found.
+ if not version_str:
+ return res[0].layer
+
+ (operator, dep_version) = version_str.split()
+ for layerbranch in res:
+ layer_ver = layerbranch.version
+
+ # If there is no version in the found layer, then don't use this layer.
+ if not layer_ver:
+ continue
+
+ try:
+ success = bb.utils.vercmp_string_op(layer_ver, version_str, operator)
+ except bb.utils.VersionStringException as vse:
+ raise vse
+
+ if success:
+ return layerbranch.layer
+
+ return None
+
+def add_dependencies(layerbranch, config_data, logger=None):
+ _add_dependency("LAYERDEPENDS", 'dependency', layerbranch, config_data, logger)
+
+def _add_dependency(var, name, layerbranch, config_data, logger=None):
+ from layerindex.models import LayerBranch, LayerDependency
+
+ layer_name = layerbranch.layer.name
+ var_name = layer_name
+
+ dep_list = config_data.getVar("%s_%s" % (var, var_name), True)
+
+ if not dep_list:
+ return
+
+ try:
+ dep_dict = bb.utils.explode_dep_versions2(dep_list)
+ except bb.utils.VersionStringException as vse:
+ logger.debug('Error parsing %s_%s for %s\n%s' % (var, var_name, layer_name, str(vse)))
+ return
+
+ for dep, ver_list in list(dep_dict.items()):
+ ver_str = None
+ if ver_list:
+ ver_str = ver_list[0]
+
+ try:
+ dep_layer = get_dependency_layer(dep, ver_str, logger)
+ except bb.utils.VersionStringException as vse:
+ if logger:
+ logger.error('Error getting %s %s for %s\n%s' %(name, dep. layer_name, str(vse)))
+ continue
+
+ if not dep_layer:
+ if logger:
+ logger.error('Cannot resolve %s %s (version %s) for %s' % (name, dep, ver_str, layer_name))
+ continue
+
+ # Skip existing entries.
+ existing = list(LayerDependency.objects.filter(layerbranch=layerbranch).filter(dependency=dep_layer))
+ if existing:
+ logger.debug('Skipping %s - already a dependency for %s' % (dep, layer_name))
+ continue
+
+ if logger:
+ logger.debug('Adding %s %s to %s' % (name, dep_layer.name, layer_name))
+ layerdep = LayerDependency()
+ layerdep.layerbranch = layerbranch
+ layerdep.dependency = dep_layer
+ layerdep.save()
+
def setup_tinfoil(bitbakepath, enable_tracking):
sys.path.insert(0, bitbakepath + '/lib')
import bb.tinfoil
--
1.9.1
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [layerindex-web][PATCH v2 09/12] layerindex: Add collection and version to layerbranch
2016-10-07 15:57 [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project Liam R. Howlett
` (7 preceding siblings ...)
2016-10-07 15:57 ` [layerindex-web][PATCH v2 08/12] layerindex: Detect dependencies from layer.conf files Liam R. Howlett
@ 2016-10-07 15:57 ` Liam R. Howlett
2016-10-10 1:32 ` Paul Eggleton
2016-10-07 15:57 ` [layerindex-web][PATCH v2 10/12] layerindexer: Add layer recommends support Liam R. Howlett
` (3 subsequent siblings)
12 siblings, 1 reply; 25+ messages in thread
From: Liam R. Howlett @ 2016-10-07 15:57 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
Collection and version will be pulled from the layer.conf if it exists
and dependencies will be resolved by first checking for layers with the
dependency name and then checking for collections. It is necessary to
shutdown tinfoil to avoid bitbake complaining about multiple instances.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
---
layerindex/models.py | 2 ++
layerindex/update_layer.py | 1 +
layerindex/utils.py | 20 ++++++++++++++++++--
3 files changed, 21 insertions(+), 2 deletions(-)
diff --git a/layerindex/models.py b/layerindex/models.py
index 2343ba7..cbfb244 100644
--- a/layerindex/models.py
+++ b/layerindex/models.py
@@ -130,6 +130,8 @@ class LayerItem(models.Model):
class LayerBranch(models.Model):
layer = models.ForeignKey(LayerItem)
branch = models.ForeignKey(Branch)
+ collection = models.CharField('Layer Collection', max_length=40, null=True, help_text='Name of the layer that could be used in the list of dependencies - can only contain letters, numbers and dashes')
+ version = models.CharField('Layer Version', max_length=10, null=True, blank=True, help_text='The layer version for this particular branch.')
vcs_subdir = models.CharField('Repository subdirectory', max_length=40, blank=True, help_text='Subdirectory within the repository where the layer is located, if not in the root (usually only used if the repository contains more than one layer)')
vcs_last_fetch = models.DateTimeField('Last successful fetch', blank=True, null=True)
vcs_last_rev = models.CharField('Last revision fetched', max_length=80, blank=True)
diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index addeb4c..40d1f16 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -283,6 +283,7 @@ def main():
layerconfparser.shutdown()
sys.exit(1)
utils.add_dependencies(layerbranch, layer_config_data, logger=logger)
+ utils.set_layerbranch_collection_version(layerbranch, layer_config_data, logger=logger)
layerbranch.save()
layerrecipes = Recipe.objects.filter(layerbranch=layerbranch)
diff --git a/layerindex/utils.py b/layerindex/utils.py
index 018478e..170d202 100644
--- a/layerindex/utils.py
+++ b/layerindex/utils.py
@@ -30,8 +30,10 @@ def get_layer(layername):
def get_dependency_layer(depname, version_str=None, logger=None):
from layerindex.models import LayerItem, LayerBranch
- # Get any LayerBranch with a layer that has a name that matches the depname
- res = list(LayerBranch.objects.filter(layer__name=depname))
+ # Get any LayerBranch with a layer that has a name that matches depmod, or
+ # a LayerBranch that has the collection name depmod.
+ res = list(LayerBranch.objects.filter(layer__name=depname)) + \
+ list(LayerBranch.objects.filter(collection=depname))
# Nothing found, return.
if not res:
@@ -68,6 +70,10 @@ def _add_dependency(var, name, layerbranch, config_data, logger=None):
layer_name = layerbranch.layer.name
var_name = layer_name
+ if layerbranch.collection:
+ var_name = layerbranch.collection
+
+
dep_list = config_data.getVar("%s_%s" % (var, var_name), True)
if not dep_list:
@@ -104,11 +110,21 @@ def _add_dependency(var, name, layerbranch, config_data, logger=None):
if logger:
logger.debug('Adding %s %s to %s' % (name, dep_layer.name, layer_name))
+
layerdep = LayerDependency()
layerdep.layerbranch = layerbranch
layerdep.dependency = dep_layer
layerdep.save()
+def set_layerbranch_collection_version(layerbranch, config_data, logger=None):
+
+ layerbranch.collection = config_data.getVar('BBFILE_COLLECTIONS', True)
+ ver_str = "LAYERVERSION_"
+ if layerbranch.collection:
+ layerbranch.collection = layerbranch.collection.strip()
+ ver_str += layerbranch.collection
+ layerbranch.version = config_data.getVar(ver_str, True)
+
def setup_tinfoil(bitbakepath, enable_tracking):
sys.path.insert(0, bitbakepath + '/lib')
import bb.tinfoil
--
1.9.1
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [layerindex-web][PATCH v2 10/12] layerindexer: Add layer recommends support
2016-10-07 15:57 [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project Liam R. Howlett
` (8 preceding siblings ...)
2016-10-07 15:57 ` [layerindex-web][PATCH v2 09/12] layerindex: Add collection and version to layerbranch Liam R. Howlett
@ 2016-10-07 15:57 ` Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 11/12] recipeparse: remove unnecessary else statement Liam R. Howlett
` (2 subsequent siblings)
12 siblings, 0 replies; 25+ messages in thread
From: Liam R. Howlett @ 2016-10-07 15:57 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
Parse layer.conf and add dependencies that are not required from
LAYERRECOMMENDS_<name>. Update the layerindex/template to support
recommends. Uses bitbake parsing code & checks versions.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
---
layerindex/models.py | 7 +++++++
layerindex/tools/import_layer.py | 2 +-
layerindex/update.py | 1 +
layerindex/update_layer.py | 1 +
layerindex/utils.py | 9 +++++++--
templates/layerindex/detail.html | 32 +++++++++++++++++++++-----------
6 files changed, 38 insertions(+), 14 deletions(-)
diff --git a/layerindex/models.py b/layerindex/models.py
index cbfb244..869d4a3 100644
--- a/layerindex/models.py
+++ b/layerindex/models.py
@@ -211,6 +211,12 @@ class LayerBranch(models.Model):
return "%s: %s" % (self.layer.name, self.branch.name)
+ def get_required(self):
+ return self.dependencies_set.filter(required=True)
+
+ def get_recommends(self):
+ return self.dependencies_set.filter(required=False)
+
class LayerMaintainer(models.Model):
MAINTAINER_STATUS_CHOICES = (
('A', 'Active'),
@@ -232,6 +238,7 @@ class LayerMaintainer(models.Model):
class LayerDependency(models.Model):
layerbranch = models.ForeignKey(LayerBranch, related_name='dependencies_set')
dependency = models.ForeignKey(LayerItem, related_name='dependents_set')
+ required = models.BooleanField(default=True)
class Meta:
verbose_name_plural = "Layer dependencies"
diff --git a/layerindex/tools/import_layer.py b/layerindex/tools/import_layer.py
index 1daaeb2..377a0f0 100755
--- a/layerindex/tools/import_layer.py
+++ b/layerindex/tools/import_layer.py
@@ -382,7 +382,7 @@ def main():
layerconfparser.shutdown()
if config_data:
utils.add_dependencies(layerbranch, config_data, logger=logger)
-
+ utils.add_recommends(layerbranch, config_data, logger=logger)
# Get some extra meta-information
readme_files = glob.glob(os.path.join(layerdir, 'README*'))
diff --git a/layerindex/update.py b/layerindex/update.py
index 8684d36..6f93099 100755
--- a/layerindex/update.py
+++ b/layerindex/update.py
@@ -235,6 +235,7 @@ def main():
continue
utils.add_dependencies(layerbranch, config_data, logger=logger)
+ utils.add_recommends(layerbranch, config_data, logger=logger)
finally:
layerconfparser.shutdown()
diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index 40d1f16..27d4ebe 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -283,6 +283,7 @@ def main():
layerconfparser.shutdown()
sys.exit(1)
utils.add_dependencies(layerbranch, layer_config_data, logger=logger)
+ utils.add_recommends(layerbranch, layer_config_data, logger=logger)
utils.set_layerbranch_collection_version(layerbranch, layer_config_data, logger=logger)
layerbranch.save()
diff --git a/layerindex/utils.py b/layerindex/utils.py
index 170d202..484d3fe 100644
--- a/layerindex/utils.py
+++ b/layerindex/utils.py
@@ -62,9 +62,12 @@ def get_dependency_layer(depname, version_str=None, logger=None):
return None
def add_dependencies(layerbranch, config_data, logger=None):
- _add_dependency("LAYERDEPENDS", 'dependency', layerbranch, config_data, logger)
+ _add_dependency("LAYERDEPENDS", 'dependency', layerbranch, config_data, logger=logger)
-def _add_dependency(var, name, layerbranch, config_data, logger=None):
+def add_recommends(layerbranch, config_data, logger=None):
+ _add_dependency("LAYERRECOMMENDS", 'recommends', layerbranch, config_data, logger=logger, required=False)
+
+def _add_dependency(var, name, layerbranch, config_data, logger=None, required=True):
from layerindex.models import LayerBranch, LayerDependency
layer_name = layerbranch.layer.name
@@ -97,6 +100,7 @@ def _add_dependency(var, name, layerbranch, config_data, logger=None):
logger.error('Error getting %s %s for %s\n%s' %(name, dep. layer_name, str(vse)))
continue
+ # No layer found.
if not dep_layer:
if logger:
logger.error('Cannot resolve %s %s (version %s) for %s' % (name, dep, ver_str, layer_name))
@@ -114,6 +118,7 @@ def _add_dependency(var, name, layerbranch, config_data, logger=None):
layerdep = LayerDependency()
layerdep.layerbranch = layerbranch
layerdep.dependency = dep_layer
+ layerdep.required = required
layerdep.save()
def set_layerbranch_collection_version(layerbranch, config_data, logger=None):
diff --git a/templates/layerindex/detail.html b/templates/layerindex/detail.html
index 4bd7ecf..9d3ee05 100644
--- a/templates/layerindex/detail.html
+++ b/templates/layerindex/detail.html
@@ -139,17 +139,27 @@
<div class="span4 pull-right description">
{% if layerbranch.dependencies_set.count > 0 %}
- <div class="well dependency-well">
- <h3>Dependencies </h3>
- <p>The {{ layeritem.name }} layer depends upon:</p>
- <ul>
- {% for dep in layerbranch.dependencies_set.all %}
- <li><a href="{% url 'layer_item' url_branch dep.dependency.name %}">{{ dep.dependency.name }}</a></li>
- {% endfor %}
- </ul>
- </div> <!-- end of well -->
- {% endif %}
-
+ <div class="well dependency-well">
+ {% if layerbranch.get_required.count > 0 %}
+ <h3>Dependencies </h3>
+ <p>The {{ layeritem.name }} layer depends upon:</p>
+ <ul>
+ {% for dep in layerbranch.get_required %}
+ <li><a href="{% url 'layer_item' url_branch dep.dependency.name %}">{{ dep.dependency.name }}</a></li>
+ {% endfor %}
+ </ul>
+ {% endif %} <!-- end of layerbranch.get_required.count -->
+ {% if layerbranch.get_recommends.count > 0 %}
+ <h3>Recommends </h3>
+ <p>The {{ layeritem.name }} layer recommends:</p>
+ <ul>
+ {% for rec in layerbranch.get_recommends %}
+ <li><a href="{% url 'layer_item' url_branch rec.dependency.name %}">{{ rec.dependency.name }}</a></li>
+ {% endfor %}
+ </ul>
+ {% endif %} <!-- end of layerbranch.get_recommends.count -->
+ </div> <!-- end of well -->
+ {% endif %} <!-- end of layerbranch.dependencies_set.count -->
</div> <!-- end of span4 -->
</div> <!-- end of row-fluid -->
</div> <!-- end of container-fluid -->
--
1.9.1
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [layerindex-web][PATCH v2 11/12] recipeparse: remove unnecessary else statement.
2016-10-07 15:57 [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project Liam R. Howlett
` (9 preceding siblings ...)
2016-10-07 15:57 ` [layerindex-web][PATCH v2 10/12] layerindexer: Add layer recommends support Liam R. Howlett
@ 2016-10-07 15:57 ` Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 12/12] layerindex/update_layer.py: Preserve the recipedependency files Liam R. Howlett
2016-10-07 18:20 ` [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project Mark Hatle
12 siblings, 0 replies; 25+ messages in thread
From: Liam R. Howlett @ 2016-10-07 15:57 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
Code clean up.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
---
layerindex/recipeparse.py | 17 ++++++++---------
1 file changed, 8 insertions(+), 9 deletions(-)
diff --git a/layerindex/recipeparse.py b/layerindex/recipeparse.py
index 91a083b..dd85bc3 100644
--- a/layerindex/recipeparse.py
+++ b/layerindex/recipeparse.py
@@ -134,15 +134,14 @@ def detect_file_type(path, subdir_start):
if res:
typename = 'machine'
return (typename, None, res.group(1))
- else:
- res = bbclass_re.match(subpath)
- if res:
- typename = 'bbclass'
- return (typename, None, res.group(1))
- res = distro_conf_re.match(subpath)
- if res:
- typename = 'distro'
- return (typename, None, res.group(1))
+ res = bbclass_re.match(subpath)
+ if res:
+ typename = 'bbclass'
+ return (typename, None, res.group(1))
+ res = distro_conf_re.match(subpath)
+ if res:
+ typename = 'distro'
+ return (typename, None, res.group(1))
if typename == 'recipe' or typename == 'bbappend':
if subdir_start:
--
1.9.1
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [layerindex-web][PATCH v2 12/12] layerindex/update_layer.py: Preserve the recipedependency files
2016-10-07 15:57 [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project Liam R. Howlett
` (10 preceding siblings ...)
2016-10-07 15:57 ` [layerindex-web][PATCH v2 11/12] recipeparse: remove unnecessary else statement Liam R. Howlett
@ 2016-10-07 15:57 ` Liam R. Howlett
2016-10-07 18:20 ` [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project Mark Hatle
12 siblings, 0 replies; 25+ messages in thread
From: Liam R. Howlett @ 2016-10-07 15:57 UTC (permalink / raw)
To: yocto; +Cc: paul.eggleton
From: Mark Hatle <mark.hatle@windriver.com>
In order to keep primary keys from constantly changing, preverse the
exist keys as much as possible.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
---
layerindex/update_layer.py | 19 ++++++++++++++++++-
1 file changed, 18 insertions(+), 1 deletion(-)
diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index 27d4ebe..ec40806 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -88,13 +88,30 @@ def update_recipe_file(tinfoil, data, path, recipe, layerdir_start, repodir):
if depstr.startswith(layerdir_start) and not depstr.endswith('/conf/layer.conf'):
filedeps.append(os.path.relpath(depstr, repodir))
from layerindex.models import RecipeFileDependency
- RecipeFileDependency.objects.filter(recipe=recipe).delete()
+
+ recipedeps_delete = []
+
+ recipedeps = RecipeFileDependency.objects.filter(recipe=recipe)
+
+ for values in recipedeps.values('path'):
+ if 'path' in values:
+ recipedeps_delete.append(values['path'])
+
for filedep in filedeps:
+ if filedep in recipedeps_delete:
+ recipedeps_delete.remove(filedep)
+ continue
+ # New item, add it...
recipedep = RecipeFileDependency()
recipedep.layerbranch = recipe.layerbranch
recipedep.recipe = recipe
recipedep.path = filedep
recipedep.save()
+
+ for filedep in recipedeps_delete:
+ logger.debug('%s: removing %s' % (recipe.layerbranch, filedep))
+ recipedeps.filter(path=filedep).delete()
+
except KeyboardInterrupt:
raise
except BaseException as e:
--
1.9.1
^ permalink raw reply related [flat|nested] 25+ messages in thread
* Re: [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project
2016-10-07 15:57 [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project Liam R. Howlett
` (11 preceding siblings ...)
2016-10-07 15:57 ` [layerindex-web][PATCH v2 12/12] layerindex/update_layer.py: Preserve the recipedependency files Liam R. Howlett
@ 2016-10-07 18:20 ` Mark Hatle
2016-10-10 0:54 ` Paul Eggleton
12 siblings, 1 reply; 25+ messages in thread
From: Mark Hatle @ 2016-10-07 18:20 UTC (permalink / raw)
To: Liam R. Howlett, yocto; +Cc: paul.eggleton
FYI, I have made sure these are re-based on top of paule/django18 and pushed to:
git://git.yoctoproject.org/layerindex-web mhatle/django18
BTW for me upgrading from paule/django18 to this version. I only had to run
'manage.py syncdb'.
This created the new database and added the missing field for the recommend.
I then ran update.py --forcereload and everything showed up as expected.
--Mark
On 10/7/16 10:57 AM, Liam R. Howlett wrote:
> This set of patches adds a number of features to the layerindex code:
> - Adds the ability to set the actual-branch to import_layer.
> - Adds distro to the database and web interface.
> - Adds collection & version information to layerbranch.
> - Uses layer name and collection name to determine dependencies and
> recommends.
> - Adds import_project to simplify pulling all layers and openembedded-core
> into a layer index.
>
> Liam R. Howlett (11):
> import_layer: Add --actual-branch option
> layerindex/tools/import_layer.py: Sanitize layer name.
> layerindex/tools/import_layer.py: Avoid failing if there is any layer
> to add.
> layerindex/utils: Update runcmd to decode binary strings to strings.
> layerindex: Add distro to web interface and model.
> layerindex/tools/import_project: Add import_project
> layerindex/recipeparse.py: refactor setup_tinfoil,
> checkout_layer_branch, parse_layer_conf to utils.py
> layerindex: Detect dependencies from layer.conf files
> layerindex: Add collection and version to layerbranch
> layerindexer: Add layer recommends support
> recipeparse: remove unnecessary else statement.
>
> Mark Hatle (1):
> layerindex/update_layer.py: Preserve the recipedependency files
>
> layerindex/admin.py | 11 +++
> layerindex/bulkchange.py | 4 +-
> layerindex/layerconfparse.py | 50 +++++++++++
> layerindex/models.py | 23 +++++
> layerindex/recipeparse.py | 53 +++--------
> layerindex/restviews.py | 10 ++-
> layerindex/tools/import_classic.py | 2 +
> layerindex/tools/import_layer.py | 47 +++++++++-
> layerindex/tools/import_project.py | 180 +++++++++++++++++++++++++++++++++++++
> layerindex/update.py | 42 ++++++++-
> layerindex/update_layer.py | 87 ++++++++++++++++--
> layerindex/urls.py | 3 +
> layerindex/urls_branch.py | 6 +-
> layerindex/utils.py | 159 +++++++++++++++++++++++++++++++-
> layerindex/views.py | 29 +++++-
> templates/layerindex/detail.html | 55 +++++++++---
> templates/layerindex/distros.html | 76 ++++++++++++++++
> templates/layerindex/layers.html | 1 +
> templates/layerindex/machines.html | 1 +
> templates/layerindex/recipes.html | 1 +
> 20 files changed, 767 insertions(+), 73 deletions(-)
> create mode 100644 layerindex/layerconfparse.py
> create mode 100755 layerindex/tools/import_project.py
> create mode 100644 templates/layerindex/distros.html
>
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project
2016-10-07 18:20 ` [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project Mark Hatle
@ 2016-10-10 0:54 ` Paul Eggleton
2016-10-10 4:48 ` Hatle, Mark
0 siblings, 1 reply; 25+ messages in thread
From: Paul Eggleton @ 2016-10-10 0:54 UTC (permalink / raw)
To: Mark Hatle; +Cc: Liam R. Howlett, yocto
On Fri, 07 Oct 2016 13:20:50 Mark Hatle wrote:
> FYI, I have made sure these are re-based on top of paule/django18 and pushed
> to:
>
> git://git.yoctoproject.org/layerindex-web mhatle/django18
>
> BTW for me upgrading from paule/django18 to this version. I only had to run
> 'manage.py syncdb'.
>
> This created the new database and added the missing field for the recommend.
I'm not sure how that worked there but it doesn't work here. If I take a
database that works on paule/django18, run syncdb then open the admin
interface and click on "Layer branches" I get:
no such column: layerindex_layerbranch.collection
I'm afraid we really do need to create migrations whenever we change the
models.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [layerindex-web][PATCH v2 05/12] layerindex: Add distro to web interface and model.
2016-10-07 15:57 ` [layerindex-web][PATCH v2 05/12] layerindex: Add distro to web interface and model Liam R. Howlett
@ 2016-10-10 1:20 ` Paul Eggleton
2016-10-11 8:14 ` Mark Hatle
0 siblings, 1 reply; 25+ messages in thread
From: Paul Eggleton @ 2016-10-10 1:20 UTC (permalink / raw)
To: Liam R. Howlett; +Cc: yocto
This looks good, just a few notes about this bit:
On Fri, 07 Oct 2016 11:57:14 Liam R. Howlett wrote:
> +def update_distro_conf_file(path, distro):
> + logger.debug('Updating distro %s' % path)
> + desc = ""
> + with open(path, 'r') as f:
> + for line in f:
> + if line.startswith('#@NAME:'):
> + desc = line[7:].strip()
> + if line.startswith('#@DESCRIPTION:'):
> + desc = line[14:].strip()
> + desc = re.sub(r'Distribtuion configuration for( running)*( an)*( the)*', '', desc)
Aside from the typo, that last line isn't needed for distros.
Also, unlike machines, the readable distro name actually gets set in a proper
variable (DISTRO_NAME) and though some distro conf files do still use the
meta-comments, a lot don't use them at all. Technically being a variable,
DISTRO_NAME could be set in an inc file rather than the distro conf file, and
often it is, so we should really parse to get it rather than scraping it.
(I'm happy if we sort this out as a later improvement, I just wanted to make
a note of it now.)
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [layerindex-web][PATCH v2 09/12] layerindex: Add collection and version to layerbranch
2016-10-07 15:57 ` [layerindex-web][PATCH v2 09/12] layerindex: Add collection and version to layerbranch Liam R. Howlett
@ 2016-10-10 1:32 ` Paul Eggleton
2016-10-11 8:15 ` Mark Hatle
0 siblings, 1 reply; 25+ messages in thread
From: Paul Eggleton @ 2016-10-10 1:32 UTC (permalink / raw)
To: Liam R. Howlett; +Cc: yocto
On Fri, 07 Oct 2016 11:57:18 Liam R. Howlett wrote:
> Collection and version will be pulled from the layer.conf if it exists
> and dependencies will be resolved by first checking for layers with the
> dependency name and then checking for collections. It is necessary to
> shutdown tinfoil to avoid bitbake complaining about multiple instances.
That last bit of the commit message is now invalid in v2.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project
2016-10-10 0:54 ` Paul Eggleton
@ 2016-10-10 4:48 ` Hatle, Mark
2016-10-10 10:37 ` Paul Eggleton
0 siblings, 1 reply; 25+ messages in thread
From: Hatle, Mark @ 2016-10-10 4:48 UTC (permalink / raw)
To: Paul Eggleton; +Cc: Howlett, Liam, yocto
> On Oct 10, 2016, at 2:54 AM, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:
>
>> On Fri, 07 Oct 2016 13:20:50 Mark Hatle wrote:
>> FYI, I have made sure these are re-based on top of paule/django18 and pushed
>> to:
>>
>> git://git.yoctoproject.org/layerindex-web mhatle/django18
>>
>> BTW for me upgrading from paule/django18 to this version. I only had to run
>> 'manage.py syncdb'.
>>
>> This created the new database and added the missing field for the recommend.
>
> I'm not sure how that worked there but it doesn't work here. If I take a
> database that works on paule/django18, run syncdb then open the admin
> interface and click on "Layer branches" I get:
>
We found a severe lack of instructions for creating migrations. There were instructions for adding and deleting full DBs but not changing them. We will need pointers or help to resolve this.
> no such column: layerindex_layerbranch.collection
I never got that, but I'm also running SQLite. May be the difference.
> I'm afraid we really do need to create migrations whenever we change the
> models.
Based on your prior comments we failed to find ANY reasonable examples, docs or instructions on doing this. So we are stuck without help.
> Cheers,
> Paul
>
> --
>
> Paul Eggleton
> Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project
2016-10-10 4:48 ` Hatle, Mark
@ 2016-10-10 10:37 ` Paul Eggleton
2016-10-10 10:47 ` Mark Hatle
0 siblings, 1 reply; 25+ messages in thread
From: Paul Eggleton @ 2016-10-10 10:37 UTC (permalink / raw)
To: Hatle, Mark; +Cc: Howlett, Liam, yocto
On Mon, 10 Oct 2016 04:48:41 Hatle, Mark wrote:
> > On Oct 10, 2016, at 2:54 AM, Paul Eggleton <paul.eggleton@linux.intel.com>
> > wrote:
> >> On Fri, 07 Oct 2016 13:20:50 Mark Hatle wrote:
> >> FYI, I have made sure these are re-based on top of paule/django18 and
> >> pushed to:
> >>
> >> git://git.yoctoproject.org/layerindex-web mhatle/django18
> >>
> >> BTW for me upgrading from paule/django18 to this version. I only had to
> >> run 'manage.py syncdb'.
> >>
> >> This created the new database and added the missing field for the
> >> recommend.>
> > I'm not sure how that worked there but it doesn't work here. If I take a
> > database that works on paule/django18, run syncdb then open the admin
> > interface and click on "Layer branches" I get:
>
> We found a severe lack of instructions for creating migrations. There were
> instructions for adding and deleting full DBs but not changing them. We
> will need pointers or help to resolve this.
>
> > no such column: layerindex_layerbranch.collection
>
> I never got that, but I'm also running SQLite. May be the difference.
I'm using sqlite here also.
> > I'm afraid we really do need to create migrations whenever we change the
> > models.
>
> Based on your prior comments we failed to find ANY reasonable examples, docs
> or instructions on doing this. So we are stuck without help.
I'm happy to help directly if needed, but did you come across this already?
https://docs.djangoproject.com/en/1.8/topics/migrations/
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project
2016-10-10 10:37 ` Paul Eggleton
@ 2016-10-10 10:47 ` Mark Hatle
2016-10-10 20:19 ` Paul Eggleton
0 siblings, 1 reply; 25+ messages in thread
From: Mark Hatle @ 2016-10-10 10:47 UTC (permalink / raw)
To: Paul Eggleton; +Cc: Howlett, Liam, yocto
On 10/10/16 12:37 PM, Paul Eggleton wrote:
> On Mon, 10 Oct 2016 04:48:41 Hatle, Mark wrote:
>>> On Oct 10, 2016, at 2:54 AM, Paul Eggleton <paul.eggleton@linux.intel.com>
>>> wrote:
>>>> On Fri, 07 Oct 2016 13:20:50 Mark Hatle wrote:
>>>> FYI, I have made sure these are re-based on top of paule/django18 and
>>>> pushed to:
>>>>
>>>> git://git.yoctoproject.org/layerindex-web mhatle/django18
>>>>
>>>> BTW for me upgrading from paule/django18 to this version. I only had to
>>>> run 'manage.py syncdb'.
>>>>
>>>> This created the new database and added the missing field for the
>>>> recommend.>
>>> I'm not sure how that worked there but it doesn't work here. If I take a
>>> database that works on paule/django18, run syncdb then open the admin
>>> interface and click on "Layer branches" I get:
>>
>> We found a severe lack of instructions for creating migrations. There were
>> instructions for adding and deleting full DBs but not changing them. We
>> will need pointers or help to resolve this.
>>
>>> no such column: layerindex_layerbranch.collection
>>
>> I never got that, but I'm also running SQLite. May be the difference.
>
> I'm using sqlite here also.
>
>>> I'm afraid we really do need to create migrations whenever we change the
>>> models.
>>
>> Based on your prior comments we failed to find ANY reasonable examples, docs
>> or instructions on doing this. So we are stuck without help.
>
> I'm happy to help directly if needed, but did you come across this already?
>
> https://docs.djangoproject.com/en/1.8/topics/migrations/
Unfortunately that is the exactly document I was looking at and could not figure
it out.
I can the various commands, makemigrations and related.. and the system simply
told me "no migrations necessary".. and didn't generate any.
So we were never able to figure out what a migration should look like or how to
generate it.
If you (or someone else) can help us figure out how to create a migration, I
suspect after the first one --- future ones will be much easier to generate.
--Mark
> Cheers,
> Paul
>
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project
2016-10-10 10:47 ` Mark Hatle
@ 2016-10-10 20:19 ` Paul Eggleton
2016-10-10 20:24 ` Paul Eggleton
0 siblings, 1 reply; 25+ messages in thread
From: Paul Eggleton @ 2016-10-10 20:19 UTC (permalink / raw)
To: Mark Hatle; +Cc: Howlett, Liam, yocto
On Mon, 10 Oct 2016 12:47:11 Mark Hatle wrote:
> On 10/10/16 12:37 PM, Paul Eggleton wrote:
> > On Mon, 10 Oct 2016 04:48:41 Hatle, Mark wrote:
> >>> On Oct 10, 2016, at 2:54 AM, Paul Eggleton
> >>> <paul.eggleton@linux.intel.com>
> >>>
> >>> wrote:
> >>>> On Fri, 07 Oct 2016 13:20:50 Mark Hatle wrote:
> >>>> FYI, I have made sure these are re-based on top of paule/django18 and
> >>>> pushed to:
> >>>>
> >>>> git://git.yoctoproject.org/layerindex-web mhatle/django18
> >>>>
> >>>> BTW for me upgrading from paule/django18 to this version. I only had
> >>>> to
> >>>> run 'manage.py syncdb'.
> >>>>
> >>>> This created the new database and added the missing field for the
> >>>> recommend.>
> >>>
> >>> I'm not sure how that worked there but it doesn't work here. If I take a
> >>> database that works on paule/django18, run syncdb then open the admin
> >>
> >>> interface and click on "Layer branches" I get:
> >> We found a severe lack of instructions for creating migrations. There
> >> were
> >> instructions for adding and deleting full DBs but not changing them. We
> >> will need pointers or help to resolve this.
> >>
> >>> no such column: layerindex_layerbranch.collection
> >>
> >> I never got that, but I'm also running SQLite. May be the difference.
> >
> > I'm using sqlite here also.
> >
> >>> I'm afraid we really do need to create migrations whenever we change the
> >>> models.
> >>
> >> Based on your prior comments we failed to find ANY reasonable examples,
> >> docs or instructions on doing this. So we are stuck without help.
> >
> > I'm happy to help directly if needed, but did you come across this
> > already?
> >
> > https://docs.djangoproject.com/en/1.8/topics/migrations/
>
> Unfortunately that is the exactly document I was looking at and could not
> figure it out.
>
> I can the various commands, makemigrations and related.. and the system
> simply told me "no migrations necessary".. and didn't generate any.
>
> So we were never able to figure out what a migration should look like or how
> to generate it.
>
> If you (or someone else) can help us figure out how to create a migration, I
> suspect after the first one --- future ones will be much easier to
> generate.
OK, so here's a step-by-step procedure - I suspect you'll need to stash and
unstash your config changes along the way as well:
1) Ensure you are using a database created at the place you want to start,
i.e. in this case paule/django18.
2) Determine which commits make changes to the models. This is one easy way:
$ git log -p paule/django18..mhatle/django18 layerindex/models.py
3) We don't have an initial migration, so we need to create that first in a
single commit before all of the others, so to do that we'll direct the rebase
to edit the top commit on paule/django18 but instead of modifying that we
insert a commit after it. We also want to edit each commit found in step 1
that made changes so we can add the migrations to them.
$ git checkout mhatle/django18
$ git rebase -i paule/django18^
(now set the command to "e" on the oldest commit, in this case 6f8201fe, and
do the same for all the commits you found in step 1, then save & quit the
editor)
4) Create the initial migration:
$ python3 manage.py makemigrations layerindex
Migrations for 'layerindex':
0001_initial.py:
- Create model BBAppend
- Create model BBClass
- Create model Branch
- Create model LayerBranch
- Create model LayerDependency
- Create model LayerItem
- Create model LayerMaintainer
- Create model LayerNote
- Create model Machine
- Create model PythonEnvironment
- Create model Recipe
- Create model RecipeChange
- Create model RecipeChangeset
- Create model RecipeFileDependency
- Create model ClassicRecipe
- Add field recipe to recipefiledependency
- Add field changeset to recipechange
- Add field recipe to recipechange
- Add field layerbranch to recipe
- Add field dependency to layerdependency
- Add field layerbranch to layerdependency
- Add field layer to layerbranch
- Add field update_environment to branch
- Add field layerbranch to bbclass
- Add field layerbranch to bbappend
- Add field cover_layerbranch to classicrecipe
5) Apply the migrations to the database - this time we make it "fake" because
the tables already exist:
$ python manage.py migrate layerindex --fake-initial
6) Create the initial migrations commit. (We won't need to do this ever again)
$ git add layerindex/migrations/
$ git commit
7) Continue the rebase:
$ git rebase --continue
8) Create the migration for the changes in this commit:
$ python manage.py makemigrations layerindex
Migrations for 'layerindex':
0002_distro.py:
- Create model Distro
9) Apply the migrations (*not* fake this time):
$ python manage.py migrate layerindex
10) Amend the migrations to the current commit:
$ git add layerindex/migrations/
$ git commit --amend
11) Continue the rebase:
$ git rebase --continue
12) Repeat steps 8-11 until the rebase finishes.
That should do it. It gets more complicated if you rename columns, or
introduce columns that need initial data beyond a simple default, split models
etc. but in this case when you're simply adding or removing columns it will
take care of creating the migrations for you as long as you know the steps
(and I admit it's a bit awkward if you've never done it before - I have the
advantage of having done this already, not exactly with Django's migrations
but with South and that was pretty similar).
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project
2016-10-10 20:19 ` Paul Eggleton
@ 2016-10-10 20:24 ` Paul Eggleton
2016-10-11 8:16 ` Mark Hatle
0 siblings, 1 reply; 25+ messages in thread
From: Paul Eggleton @ 2016-10-10 20:24 UTC (permalink / raw)
To: Mark Hatle; +Cc: yocto, Howlett, Liam
On Tue, 11 Oct 2016 09:19:15 Paul Eggleton wrote:
> On Mon, 10 Oct 2016 12:47:11 Mark Hatle wrote:
> > On 10/10/16 12:37 PM, Paul Eggleton wrote:
> > > On Mon, 10 Oct 2016 04:48:41 Hatle, Mark wrote:
> > >>> On Oct 10, 2016, at 2:54 AM, Paul Eggleton
> > >>> <paul.eggleton@linux.intel.com>
> > >>>
> > >>> wrote:
> > >>>> On Fri, 07 Oct 2016 13:20:50 Mark Hatle wrote:
> > >>>> FYI, I have made sure these are re-based on top of paule/django18 and
> > >>>> pushed to:
> > >>>>
> > >>>> git://git.yoctoproject.org/layerindex-web mhatle/django18
> > >>>>
> > >>>> BTW for me upgrading from paule/django18 to this version. I only had
> > >>>> to
> > >>>> run 'manage.py syncdb'.
> > >>>>
> > >>>> This created the new database and added the missing field for the
> > >>>> recommend.>
> > >>>
> > >>> I'm not sure how that worked there but it doesn't work here. If I take
> > >>> a
> > >>> database that works on paule/django18, run syncdb then open the admin
> > >>
> > >>> interface and click on "Layer branches" I get:
> > >> We found a severe lack of instructions for creating migrations. There
> > >> were
> > >> instructions for adding and deleting full DBs but not changing them.
> > >> We
> > >> will need pointers or help to resolve this.
> > >>
> > >>> no such column: layerindex_layerbranch.collection
> > >>
> > >> I never got that, but I'm also running SQLite. May be the difference.
> > >
> > > I'm using sqlite here also.
> > >
> > >>> I'm afraid we really do need to create migrations whenever we change
> > >>> the
> > >>> models.
> > >>
> > >> Based on your prior comments we failed to find ANY reasonable examples,
> > >> docs or instructions on doing this. So we are stuck without help.
> > >
> > > I'm happy to help directly if needed, but did you come across this
> > > already?
> > >
> > > https://docs.djangoproject.com/en/1.8/topics/migrations/
> >
> > Unfortunately that is the exactly document I was looking at and could not
> > figure it out.
> >
> > I can the various commands, makemigrations and related.. and the system
> > simply told me "no migrations necessary".. and didn't generate any.
> >
> > So we were never able to figure out what a migration should look like or
> > how to generate it.
> >
> > If you (or someone else) can help us figure out how to create a migration,
> > I suspect after the first one --- future ones will be much easier to
> > generate.
>
> OK, so here's a step-by-step procedure - I suspect you'll need to stash and
> unstash your config changes along the way as well:
>
> 1) Ensure you are using a database created at the place you want to start,
> i.e. in this case paule/django18.
>
> 2) Determine which commits make changes to the models. This is one easy way:
>
> $ git log -p paule/django18..mhatle/django18 layerindex/models.py
>
> 3) We don't have an initial migration, so we need to create that first in a
> single commit before all of the others, so to do that we'll direct the
> rebase to edit the top commit on paule/django18 but instead of modifying
> that we insert a commit after it. We also want to edit each commit found in
> step 1 that made changes so we can add the migrations to them.
>
> $ git checkout mhatle/django18
> $ git rebase -i paule/django18^
> (now set the command to "e" on the oldest commit, in this case 6f8201fe, and
> do the same for all the commits you found in step 1, then save & quit the
> editor)
>
> 4) Create the initial migration:
>
> $ python3 manage.py makemigrations layerindex
> Migrations for 'layerindex':
> 0001_initial.py:
> - Create model BBAppend
> - Create model BBClass
> - Create model Branch
> - Create model LayerBranch
> - Create model LayerDependency
> - Create model LayerItem
> - Create model LayerMaintainer
> - Create model LayerNote
> - Create model Machine
> - Create model PythonEnvironment
> - Create model Recipe
> - Create model RecipeChange
> - Create model RecipeChangeset
> - Create model RecipeFileDependency
> - Create model ClassicRecipe
> - Add field recipe to recipefiledependency
> - Add field changeset to recipechange
> - Add field recipe to recipechange
> - Add field layerbranch to recipe
> - Add field dependency to layerdependency
> - Add field layerbranch to layerdependency
> - Add field layer to layerbranch
> - Add field update_environment to branch
> - Add field layerbranch to bbclass
> - Add field layerbranch to bbappend
> - Add field cover_layerbranch to classicrecipe
>
> 5) Apply the migrations to the database - this time we make it "fake"
> because the tables already exist:
>
> $ python manage.py migrate layerindex --fake-initial
>
> 6) Create the initial migrations commit. (We won't need to do this ever
> again)
>
> $ git add layerindex/migrations/
> $ git commit
>
> 7) Continue the rebase:
>
> $ git rebase --continue
>
> 8) Create the migration for the changes in this commit:
>
> $ python manage.py makemigrations layerindex
> Migrations for 'layerindex':
> 0002_distro.py:
> - Create model Distro
>
> 9) Apply the migrations (*not* fake this time):
>
> $ python manage.py migrate layerindex
>
> 10) Amend the migrations to the current commit:
>
> $ git add layerindex/migrations/
> $ git commit --amend
>
> 11) Continue the rebase:
>
> $ git rebase --continue
>
> 12) Repeat steps 8-11 until the rebase finishes.
>
> That should do it. It gets more complicated if you rename columns, or
> introduce columns that need initial data beyond a simple default, split
> models etc. but in this case when you're simply adding or removing columns
> it will take care of creating the migrations for you as long as you know
> the steps (and I admit it's a bit awkward if you've never done it before -
> I have the advantage of having done this already, not exactly with Django's
> migrations but with South and that was pretty similar).
Heh, I knew I would muck something up - where you see "python" in the above,
read "python3". (In my virtualenv-based setup it doesn't make any difference,
but that may not be true for yours.)
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [layerindex-web][PATCH v2 05/12] layerindex: Add distro to web interface and model.
2016-10-10 1:20 ` Paul Eggleton
@ 2016-10-11 8:14 ` Mark Hatle
0 siblings, 0 replies; 25+ messages in thread
From: Mark Hatle @ 2016-10-11 8:14 UTC (permalink / raw)
To: Paul Eggleton, Liam R. Howlett; +Cc: yocto
On 10/10/16 3:20 AM, Paul Eggleton wrote:
> This looks good, just a few notes about this bit:
>
> On Fri, 07 Oct 2016 11:57:14 Liam R. Howlett wrote:
>> +def update_distro_conf_file(path, distro):
>> + logger.debug('Updating distro %s' % path)
>> + desc = ""
>> + with open(path, 'r') as f:
>> + for line in f:
>> + if line.startswith('#@NAME:'):
>> + desc = line[7:].strip()
>> + if line.startswith('#@DESCRIPTION:'):
>> + desc = line[14:].strip()
>> + desc = re.sub(r'Distribtuion configuration for( running)*( an)*( the)*', '', desc)
>
> Aside from the typo, that last line isn't needed for distros.
>
> Also, unlike machines, the readable distro name actually gets set in a proper
> variable (DISTRO_NAME) and though some distro conf files do still use the
> meta-comments, a lot don't use them at all. Technically being a variable,
> DISTRO_NAME could be set in an inc file rather than the distro conf file, and
> often it is, so we should really parse to get it rather than scraping it.
>
> (I'm happy if we sort this out as a later improvement, I just wanted to make
> a note of it now.)
I have fixed the typo in the current mhatle/django18 layer.
> Cheers,
> Paul
>
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [layerindex-web][PATCH v2 09/12] layerindex: Add collection and version to layerbranch
2016-10-10 1:32 ` Paul Eggleton
@ 2016-10-11 8:15 ` Mark Hatle
0 siblings, 0 replies; 25+ messages in thread
From: Mark Hatle @ 2016-10-11 8:15 UTC (permalink / raw)
To: Paul Eggleton, Liam R. Howlett; +Cc: yocto
On 10/10/16 3:32 AM, Paul Eggleton wrote:
> On Fri, 07 Oct 2016 11:57:18 Liam R. Howlett wrote:
>> Collection and version will be pulled from the layer.conf if it exists
>> and dependencies will be resolved by first checking for layers with the
>> dependency name and then checking for collections. It is necessary to
>> shutdown tinfoil to avoid bitbake complaining about multiple instances.
>
> That last bit of the commit message is now invalid in v2.
I have removed the errant comment (last line) from the commit message in the
current 'mhatle/django18' branch.
> Cheers,
> Paul
>
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project
2016-10-10 20:24 ` Paul Eggleton
@ 2016-10-11 8:16 ` Mark Hatle
0 siblings, 0 replies; 25+ messages in thread
From: Mark Hatle @ 2016-10-11 8:16 UTC (permalink / raw)
To: Paul Eggleton; +Cc: yocto, Howlett, Liam
On 10/10/16 10:24 PM, Paul Eggleton wrote:
> On Tue, 11 Oct 2016 09:19:15 Paul Eggleton wrote:
>> On Mon, 10 Oct 2016 12:47:11 Mark Hatle wrote:
>>> On 10/10/16 12:37 PM, Paul Eggleton wrote:
>>>> On Mon, 10 Oct 2016 04:48:41 Hatle, Mark wrote:
>>>>>> On Oct 10, 2016, at 2:54 AM, Paul Eggleton
>>>>>> <paul.eggleton@linux.intel.com>
>>>>>>
>>>>>> wrote:
>>>>>>> On Fri, 07 Oct 2016 13:20:50 Mark Hatle wrote:
>>>>>>> FYI, I have made sure these are re-based on top of paule/django18 and
>>>>>>> pushed to:
>>>>>>>
>>>>>>> git://git.yoctoproject.org/layerindex-web mhatle/django18
>>>>>>>
>>>>>>> BTW for me upgrading from paule/django18 to this version. I only had
>>>>>>> to
>>>>>>> run 'manage.py syncdb'.
>>>>>>>
>>>>>>> This created the new database and added the missing field for the
>>>>>>> recommend.>
>>>>>>
>>>>>> I'm not sure how that worked there but it doesn't work here. If I take
>>>>>> a
>>>>>> database that works on paule/django18, run syncdb then open the admin
>>>>>
>>>>>> interface and click on "Layer branches" I get:
>>>>> We found a severe lack of instructions for creating migrations. There
>>>>> were
>>>>> instructions for adding and deleting full DBs but not changing them.
>>>>> We
>>>>> will need pointers or help to resolve this.
>>>>>
>>>>>> no such column: layerindex_layerbranch.collection
>>>>>
>>>>> I never got that, but I'm also running SQLite. May be the difference.
>>>>
>>>> I'm using sqlite here also.
>>>>
>>>>>> I'm afraid we really do need to create migrations whenever we change
>>>>>> the
>>>>>> models.
>>>>>
>>>>> Based on your prior comments we failed to find ANY reasonable examples,
>>>>> docs or instructions on doing this. So we are stuck without help.
>>>>
>>>> I'm happy to help directly if needed, but did you come across this
>>>> already?
>>>>
>>>> https://docs.djangoproject.com/en/1.8/topics/migrations/
>>>
>>> Unfortunately that is the exactly document I was looking at and could not
>>> figure it out.
>>>
>>> I can the various commands, makemigrations and related.. and the system
>>> simply told me "no migrations necessary".. and didn't generate any.
>>>
>>> So we were never able to figure out what a migration should look like or
>>> how to generate it.
>>>
>>> If you (or someone else) can help us figure out how to create a migration,
>>> I suspect after the first one --- future ones will be much easier to
>>> generate.
>>
>> OK, so here's a step-by-step procedure - I suspect you'll need to stash and
>> unstash your config changes along the way as well:
>>
>> 1) Ensure you are using a database created at the place you want to start,
>> i.e. in this case paule/django18.
>>
>> 2) Determine which commits make changes to the models. This is one easy way:
>>
>> $ git log -p paule/django18..mhatle/django18 layerindex/models.py
>>
>> 3) We don't have an initial migration, so we need to create that first in a
>> single commit before all of the others, so to do that we'll direct the
>> rebase to edit the top commit on paule/django18 but instead of modifying
>> that we insert a commit after it. We also want to edit each commit found in
>> step 1 that made changes so we can add the migrations to them.
>>
>> $ git checkout mhatle/django18
>> $ git rebase -i paule/django18^
>> (now set the command to "e" on the oldest commit, in this case 6f8201fe, and
>> do the same for all the commits you found in step 1, then save & quit the
>> editor)
>>
>> 4) Create the initial migration:
>>
>> $ python3 manage.py makemigrations layerindex
>> Migrations for 'layerindex':
>> 0001_initial.py:
>> - Create model BBAppend
>> - Create model BBClass
>> - Create model Branch
>> - Create model LayerBranch
>> - Create model LayerDependency
>> - Create model LayerItem
>> - Create model LayerMaintainer
>> - Create model LayerNote
>> - Create model Machine
>> - Create model PythonEnvironment
>> - Create model Recipe
>> - Create model RecipeChange
>> - Create model RecipeChangeset
>> - Create model RecipeFileDependency
>> - Create model ClassicRecipe
>> - Add field recipe to recipefiledependency
>> - Add field changeset to recipechange
>> - Add field recipe to recipechange
>> - Add field layerbranch to recipe
>> - Add field dependency to layerdependency
>> - Add field layerbranch to layerdependency
>> - Add field layer to layerbranch
>> - Add field update_environment to branch
>> - Add field layerbranch to bbclass
>> - Add field layerbranch to bbappend
>> - Add field cover_layerbranch to classicrecipe
>>
>> 5) Apply the migrations to the database - this time we make it "fake"
>> because the tables already exist:
>>
>> $ python manage.py migrate layerindex --fake-initial
>>
>> 6) Create the initial migrations commit. (We won't need to do this ever
>> again)
>>
>> $ git add layerindex/migrations/
>> $ git commit
>>
>> 7) Continue the rebase:
>>
>> $ git rebase --continue
>>
>> 8) Create the migration for the changes in this commit:
>>
>> $ python manage.py makemigrations layerindex
>> Migrations for 'layerindex':
>> 0002_distro.py:
>> - Create model Distro
>>
>> 9) Apply the migrations (*not* fake this time):
>>
>> $ python manage.py migrate layerindex
>>
>> 10) Amend the migrations to the current commit:
>>
>> $ git add layerindex/migrations/
>> $ git commit --amend
>>
>> 11) Continue the rebase:
>>
>> $ git rebase --continue
>>
>> 12) Repeat steps 8-11 until the rebase finishes.
>>
>> That should do it. It gets more complicated if you rename columns, or
>> introduce columns that need initial data beyond a simple default, split
>> models etc. but in this case when you're simply adding or removing columns
>> it will take care of creating the migrations for you as long as you know
>> the steps (and I admit it's a bit awkward if you've never done it before -
>> I have the advantage of having done this already, not exactly with Django's
>> migrations but with South and that was pretty similar).
>
> Heh, I knew I would muck something up - where you see "python" in the above,
> read "python3". (In my virtualenv-based setup it doesn't make any difference,
> but that may not be true for yours.)
No problem. The steps above did work for me. Migrations have now been added to
the mhatle/django18 branch.
I think what we were missing was the setup of the initial migration, step #4
above. Without doing that you get messages like no migrations necessary and
such -- very confusing.
Thanks for the help, let me know if there is anything else that needs to be
adjusted.
--Mark
> Cheers,
> Paul
>
^ permalink raw reply [flat|nested] 25+ messages in thread
end of thread, other threads:[~2016-10-11 8:16 UTC | newest]
Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-10-07 15:57 [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 01/12] import_layer: Add --actual-branch option Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 02/12] layerindex/tools/import_layer.py: Sanitize layer name Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 03/12] layerindex/tools/import_layer.py: Avoid failing if there is any layer to add Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 04/12] layerindex/utils: Update runcmd to decode binary strings to strings Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 05/12] layerindex: Add distro to web interface and model Liam R. Howlett
2016-10-10 1:20 ` Paul Eggleton
2016-10-11 8:14 ` Mark Hatle
2016-10-07 15:57 ` [layerindex-web][PATCH v2 06/12] layerindex/tools/import_project: Add import_project Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 07/12] layerindex/recipeparse.py: refactor setup_tinfoil, checkout_layer_branch, parse_layer_conf to utils.py Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 08/12] layerindex: Detect dependencies from layer.conf files Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 09/12] layerindex: Add collection and version to layerbranch Liam R. Howlett
2016-10-10 1:32 ` Paul Eggleton
2016-10-11 8:15 ` Mark Hatle
2016-10-07 15:57 ` [layerindex-web][PATCH v2 10/12] layerindexer: Add layer recommends support Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 11/12] recipeparse: remove unnecessary else statement Liam R. Howlett
2016-10-07 15:57 ` [layerindex-web][PATCH v2 12/12] layerindex/update_layer.py: Preserve the recipedependency files Liam R. Howlett
2016-10-07 18:20 ` [layerindex-web][PATCH v2 00/12] Add Distro, dependency and recommends detection, and import_project Mark Hatle
2016-10-10 0:54 ` Paul Eggleton
2016-10-10 4:48 ` Hatle, Mark
2016-10-10 10:37 ` Paul Eggleton
2016-10-10 10:47 ` Mark Hatle
2016-10-10 20:19 ` Paul Eggleton
2016-10-10 20:24 ` Paul Eggleton
2016-10-11 8:16 ` Mark Hatle
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.