[PATCH 00/18] Patches for upstream 22.11.16

[PATCH 00/18] Patches for upstream 22.11.16

[Michael Wood]

Patches for upstream submission. This is a patchset for all the current patches waiting to be upstreamed.

Tests completed:
- https://travis-ci.org/toastertester/toaster-next/builds/178028675
- Run command line build of binutils-native
- Run local build of core-image-minimal
- Run master project build of core-image-minimal
- Build customised core-image-minimal

Branch available at poky-contrib michaelw/toaster/patch_queue_22_11_16
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=michaelw/toaster/patch_queue_22_11_16


Michael Wood (14):
  toaster: runbuilds Write the pidfile in python rather than shell
    script
  toaster: tests Add management command tests
  toaster: Add an example production settings file
  toaster: layerindex updater Take into account layers being predefined
  toaster: orm/fixtures Add the master release and correct morty release
  toaster: settings fixture Set default release to master
  toaster: customrecipejs Consume click event on 'a' link if disabled
  toaster: buildinfohelper toaster-custom-images layer
  toaster: orm models Project class Fix pyflake errors
  toaster: orm models Handle CustomImageRecipe BRLayer here
  toaster: bldcontrol Move CustomImageRecipe file creation into own
    function
  toaster: buildinfohelper Simplify layer event to toaster layer
    function
  toaster: buildinfohelper fix _get_layer_version_for_dependency
  toaster: buildinfohelper Clarify log message for build history

Reyna, David (2):
  toaster: orm gen_layerdeps Protect against circular Layer dependencies
  toaster: tablejs Fix missing close square bracket

Sujith H (2):
  toaster: localhostbecontroller accept custom init script for build
  toaster: localhostbecontroller write toaster layers for project to
    toaster-bblayers.conf

 bitbake/bin/toaster                                |   2 +-
 bitbake/lib/bb/ui/buildinfohelper.py               | 112 +++++++--------
 .../toaster/bldcontrol/localhostbecontroller.py    | 160 +++++++++++----------
 .../bldcontrol/management/commands/runbuilds.py    |  21 ++-
 bitbake/lib/toaster/orm/fixtures/oe-core.xml       |  23 ++-
 bitbake/lib/toaster/orm/fixtures/poky.xml          |  82 ++++++++++-
 bitbake/lib/toaster/orm/fixtures/settings.xml      |   2 +-
 .../toaster/orm/management/commands/lsupdates.py   |  61 +++-----
 bitbake/lib/toaster/orm/models.py                  |  94 +++++++-----
 bitbake/lib/toaster/tests/commands/__init__.py     |   0
 .../lib/toaster/tests/commands/test_loaddata.py    |  61 ++++++++
 .../lib/toaster/tests/commands/test_lsupdates.py   |  45 ++++++
 .../lib/toaster/tests/commands/test_runbuilds.py   |  88 ++++++++++++
 bitbake/lib/toaster/toastergui/api.py              |  16 ++-
 .../toaster/toastergui/static/js/customrecipe.js   |   6 +
 bitbake/lib/toaster/toastergui/static/js/table.js  |   2 +-
 .../toastermain/settings_production_example.py     |  58 ++++++++
 17 files changed, 597 insertions(+), 236 deletions(-)
 create mode 100644 bitbake/lib/toaster/tests/commands/__init__.py
 create mode 100644 bitbake/lib/toaster/tests/commands/test_loaddata.py
 create mode 100644 bitbake/lib/toaster/tests/commands/test_lsupdates.py
 create mode 100644 bitbake/lib/toaster/tests/commands/test_runbuilds.py
 create mode 100644 bitbake/lib/toaster/toastermain/settings_production_example.py

-- 
2.7.4

[PATCH 01/18] toaster: runbuilds Write the pidfile in python rather than shell script

[Michael Wood]

Write the pid file out in the start up of this management command. This
ensures this has happened instead of relying on the shell command having
been run which may or may not be the case. This also makes it simpler for
testing.

Couple of clean ups of runbuilds as identified by pyflake

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/bin/toaster                                 |  2 +-
 .../bldcontrol/management/commands/runbuilds.py     | 21 ++++++++++++++-------
 .../toaster/orm/management/commands/lsupdates.py    |  1 -
 3 files changed, 15 insertions(+), 9 deletions(-)

diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster
index f92d38e..e0aac1a 100755
--- a/bitbake/bin/toaster
+++ b/bitbake/bin/toaster
@@ -254,7 +254,7 @@ case $CMD in
             return 4
         fi
         export BITBAKE_UI='toasterui'
-        $MANAGE runbuilds & echo $! >${BUILDDIR}/.runbuilds.pid
+        $MANAGE runbuilds &
         # set fail safe stop system on terminal exit
         trap stop_system SIGHUP
         echo "Successful ${CMD}."
diff --git a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
index 7f7a5a9..df11f9d 100644
--- a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
+++ b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
@@ -11,9 +11,11 @@ from orm.models import Build, LogMessage, Target
 import logging
 import traceback
 import signal
+import os
 
 logger = logging.getLogger("toaster")
 
+
 class Command(NoArgsCommand):
     args = ""
     help = "Schedules and executes build requests as possible. "\
@@ -50,7 +52,7 @@ class Command(NoArgsCommand):
                 logger.debug("runbuilds: No build env")
                 return
 
-            logger.info("runbuilds: starting build %s, environment %s" % \
+            logger.info("runbuilds: starting build %s, environment %s" %
                         (br, bec.be))
 
             # let the build request know where it is being executed
@@ -80,7 +82,7 @@ class Command(NoArgsCommand):
 
     def archive(self):
         for br in BuildRequest.objects.filter(state=BuildRequest.REQ_ARCHIVE):
-            if br.build == None:
+            if br.build is None:
                 br.state = BuildRequest.REQ_FAILED
             else:
                 br.state = BuildRequest.REQ_COMPLETED
@@ -99,10 +101,10 @@ class Command(NoArgsCommand):
             Q(updated__lt=timezone.now() - timedelta(seconds=30))
         ).update(lock=BuildEnvironment.LOCK_FREE)
 
-
         # update all Builds that were in progress and failed to start
-        for br in BuildRequest.objects.filter(state=BuildRequest.REQ_FAILED,
-                                              build__outcome=Build.IN_PROGRESS):
+        for br in BuildRequest.objects.filter(
+                state=BuildRequest.REQ_FAILED,
+                build__outcome=Build.IN_PROGRESS):
             # transpose the launch errors in ToasterExceptions
             br.build.outcome = Build.FAILED
             for brerror in br.brerror_set.all():
@@ -117,7 +119,6 @@ class Command(NoArgsCommand):
             br.environment.lock = BuildEnvironment.LOCK_FREE
             br.environment.save()
 
-
         # update all BuildRequests without a build created
         for br in BuildRequest.objects.filter(build=None):
             br.build = Build.objects.create(project=br.project,
@@ -144,7 +145,7 @@ class Command(NoArgsCommand):
 
         # Make sure the LOCK is removed for builds which have been fully
         # cancelled
-        for br in BuildRequest.objects.filter(\
+        for br in BuildRequest.objects.filter(
                       Q(build__outcome=Build.CANCELLED) &
                       Q(state=BuildRequest.REQ_CANCELLING) &
                       ~Q(environment=None)):
@@ -168,6 +169,12 @@ class Command(NoArgsCommand):
             logger.warn("runbuilds: schedule exception %s" % str(e))
 
     def handle_noargs(self, **options):
+        pidfile_path = os.path.join(os.environ.get("BUILDDIR", "."),
+                                    ".runbuilds.pid")
+
+        with open(pidfile_path, 'w') as pidfile:
+            pidfile.write("%s" % os.getpid())
+
         self.runbuild()
 
         signal.signal(signal.SIGUSR1, lambda sig, frame: None)
diff --git a/bitbake/lib/toaster/orm/management/commands/lsupdates.py b/bitbake/lib/toaster/orm/management/commands/lsupdates.py
index 8ff120e..5ada1ac 100644
--- a/bitbake/lib/toaster/orm/management/commands/lsupdates.py
+++ b/bitbake/lib/toaster/orm/management/commands/lsupdates.py
@@ -90,7 +90,6 @@ class Command(NoArgsCommand):
             from urlparse import urlparse
 
         proxy_settings = os.environ.get("http_proxy", None)
-        oe_core_layer = 'openembedded-core'
 
         def _get_json_response(apiurl=DEFAULT_LAYERINDEX_SERVER):
             http_progress = Spinner()
-- 
2.7.4

[PATCH 02/18] toaster: tests Add management command tests

[Michael Wood]

Add some simple sanity tests for the management commands that we use for
Toaster.

Can be executed with ./manage.py test tests.commands

For faster execution use the test settings and keepdb flag:
DJANGO_SETTINGS_MODULE=toastermain.settings_test ./manage.py test
tests.commands --keepdb

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/toaster/tests/commands/__init__.py     |  0
 .../lib/toaster/tests/commands/test_loaddata.py    | 61 +++++++++++++++
 .../lib/toaster/tests/commands/test_lsupdates.py   | 45 +++++++++++
 .../lib/toaster/tests/commands/test_runbuilds.py   | 88 ++++++++++++++++++++++
 4 files changed, 194 insertions(+)
 create mode 100644 bitbake/lib/toaster/tests/commands/__init__.py
 create mode 100644 bitbake/lib/toaster/tests/commands/test_loaddata.py
 create mode 100644 bitbake/lib/toaster/tests/commands/test_lsupdates.py
 create mode 100644 bitbake/lib/toaster/tests/commands/test_runbuilds.py

diff --git a/bitbake/lib/toaster/tests/commands/__init__.py b/bitbake/lib/toaster/tests/commands/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/bitbake/lib/toaster/tests/commands/test_loaddata.py b/bitbake/lib/toaster/tests/commands/test_loaddata.py
new file mode 100644
index 0000000..951f6ff
--- /dev/null
+++ b/bitbake/lib/toaster/tests/commands/test_loaddata.py
@@ -0,0 +1,61 @@
+#! /usr/bin/env python
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# BitBake Toaster Implementation
+#
+# Copyright (C) 2016 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+from django.test import TestCase
+from django.core import management
+
+from orm.models import Layer_Version, Layer, Release, ToasterSetting
+
+
+class TestLoadDataFixtures(TestCase):
+    """ Test loading our 3 provided fixtures """
+    def test_run_loaddata_poky_command(self):
+        management.call_command('loaddata', 'poky')
+
+        num_releases = Release.objects.count()
+
+        self.assertTrue(
+            Layer_Version.objects.filter(
+                layer__name="meta-poky").count() == num_releases,
+            "Loaded poky fixture but don't have a meta-poky for all releases"
+            " defined")
+
+    def test_run_loaddata_oecore_command(self):
+        management.call_command('loaddata', 'oe-core')
+
+        # We only have the one layer for oe-core setup
+        self.assertTrue(
+            Layer.objects.filter(name="openembedded-core").count() > 0,
+            "Loaded oe-core fixture but still have no openemebedded-core"
+            " layer")
+
+    def test_run_loaddata_settings_command(self):
+        management.call_command('loaddata', 'settings')
+
+        self.assertTrue(
+            ToasterSetting.objects.filter(name="DEFAULT_RELEASE").count() > 0,
+            "Loaded settings but have no DEFAULT_RELEASE")
+
+        self.assertTrue(
+            ToasterSetting.objects.filter(
+                name__startswith="DEFCONF").count() > 0,
+            "Loaded settings but have no DEFCONF (default project "
+            "configuration values)")
diff --git a/bitbake/lib/toaster/tests/commands/test_lsupdates.py b/bitbake/lib/toaster/tests/commands/test_lsupdates.py
new file mode 100644
index 0000000..49897a4
--- /dev/null
+++ b/bitbake/lib/toaster/tests/commands/test_lsupdates.py
@@ -0,0 +1,45 @@
+#! /usr/bin/env python
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# BitBake Toaster Implementation
+#
+# Copyright (C) 2016 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+from django.test import TestCase
+from django.core import management
+
+from orm.models import Layer_Version, Machine, Recipe
+
+
+class TestLayerIndexUpdater(TestCase):
+    def test_run_lsupdates_command(self):
+        # Load some release information for us to fetch from the layer index
+        management.call_command('loaddata', 'poky')
+
+        old_layers_count = Layer_Version.objects.count()
+        old_recipes_count = Recipe.objects.count()
+        old_machines_count = Machine.objects.count()
+
+        # Now fetch the metadata from the layer index
+        management.call_command('lsupdates')
+
+        self.assertTrue(Layer_Version.objects.count() > old_layers_count,
+                        "lsupdates ran but we still have no more layers!")
+        self.assertTrue(Recipe.objects.count() > old_recipes_count,
+                        "lsupdates ran but we still have no more Recipes!")
+        self.assertTrue(Machine.objects.count() > old_machines_count,
+                        "lsupdates ran but we still have no more Machines!")
diff --git a/bitbake/lib/toaster/tests/commands/test_runbuilds.py b/bitbake/lib/toaster/tests/commands/test_runbuilds.py
new file mode 100644
index 0000000..3e63483
--- /dev/null
+++ b/bitbake/lib/toaster/tests/commands/test_runbuilds.py
@@ -0,0 +1,88 @@
+#! /usr/bin/env python
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# BitBake Toaster Implementation
+#
+# Copyright (C) 2016 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import os
+
+from django.test import TestCase
+from django.core import management
+
+from orm.models import signal_runbuilds
+
+import threading
+import time
+import subprocess
+import signal
+
+
+class KillRunbuilds(threading.Thread):
+    """ Kill the runbuilds process after an amount of time """
+    def __init__(self, *args, **kwargs):
+        super(KillRunbuilds, self).__init__(*args, **kwargs)
+        self.setDaemon(True)
+
+    def run(self):
+        time.sleep(5)
+        signal_runbuilds()
+        time.sleep(1)
+
+        pidfile_path = os.path.join(os.environ.get("BUILDDIR", "."),
+                                    ".runbuilds.pid")
+
+        with open(pidfile_path) as pidfile:
+            pid = pidfile.read()
+            os.kill(int(pid), signal.SIGTERM)
+
+
+class TestCommands(TestCase):
+    """ Sanity test that runbuilds executes OK """
+
+    def setUp(self):
+        os.environ.setdefault("DJANGO_SETTINGS_MODULE",
+                              "toastermain.settings_test")
+        os.environ.setdefault("BUILDDIR",
+                              "/tmp/")
+
+        # Setup a real database if needed for runbuilds process
+        # to connect to
+        management.call_command('migrate')
+
+    def test_runbuilds_command(self):
+        kill_runbuilds = KillRunbuilds()
+        kill_runbuilds.start()
+
+        manage_py = os.path.join(
+            os.path.dirname(os.path.abspath(__file__)),
+            os.pardir,
+            os.pardir,
+            "manage.py")
+
+        command = "%s runbuilds" % manage_py
+
+        process = subprocess.Popen(command,
+                                   shell=True,
+                                   stdout=subprocess.PIPE,
+                                   stderr=subprocess.PIPE)
+
+        (out, err) = process.communicate()
+        process.wait()
+
+        self.assertNotEqual(process.returncode, 1,
+                            "Runbuilds returned an error %s" % err)
-- 
2.7.4

[PATCH 03/18] toaster: Add an example production settings file

[Michael Wood]

Add an example settings that can be used for the basis of the production
instance of Toaster.

[YOCTO #10581]

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 .../toastermain/settings_production_example.py     | 58 ++++++++++++++++++++++
 1 file changed, 58 insertions(+)
 create mode 100644 bitbake/lib/toaster/toastermain/settings_production_example.py

diff --git a/bitbake/lib/toaster/toastermain/settings_production_example.py b/bitbake/lib/toaster/toastermain/settings_production_example.py
new file mode 100644
index 0000000..61a2888
--- /dev/null
+++ b/bitbake/lib/toaster/toastermain/settings_production_example.py
@@ -0,0 +1,58 @@
+#
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# BitBake Toaster Implementation
+#
+# Copyright (C) 2016        Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+# See Django documentation for more information about deployment
+# https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/
+
+# Toaster production settings example overlay
+# To use this copy this example to "settings_production.py" and set in your
+# environment DJANGO_SETTINGS_MODULE=toastermain.settings_production
+# This can be permanently set in a new .wsgi file
+
+from toastermain.settings import *  # NOQA
+
+# Set this value!
+SECRET_KEY = None
+
+# Switch off any debugging
+DEBUG = True
+TEMPLATE_DEBUG = DEBUG
+
+DATABASES = {
+     'default': {
+         'ENGINE': 'django.db.backends.mysql',
+         'NAME': 'toaster_data',
+         'USER': 'toaster',
+         'PASSWORD': 'yourpasswordhere',
+         'HOST': '127.0.0.1',
+         'PORT': '3306',
+     }
+}
+
+# Location where static files will be placed by "manage.py collectstatic"
+STATIC_ROOT = '/var/www/static-toaster/'
+
+# URL prefix for static files.
+STATIC_URL = '/static-toaster/'
+
+# Hosts that Django will serve
+# https://docs.djangoproject.com/en/1.8/ref/settings/#std:setting-ALLOWED_HOSTS
+ALLOWED_HOSTS = ['toaster-example.example.com']
-- 
2.7.4

[PATCH 04/18] toaster: layerindex updater Take into account layers being predefined

[Michael Wood]

As we can now provide layer definitions through fixtures we need to be
more clever how we update the metadata in the database to avoid
duplicate metadata being created. To do this we make more effort to
match existing data in the database and update only the fields which
will be better provided by the layer index.

This removes the need for us to special case layers which are provided
as part of poky such as openembedded-core or meta-poky which exist on
the layerindex but with different git urls.

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 .../toaster/orm/management/commands/lsupdates.py   | 60 +++++++---------------
 1 file changed, 19 insertions(+), 41 deletions(-)

diff --git a/bitbake/lib/toaster/orm/management/commands/lsupdates.py b/bitbake/lib/toaster/orm/management/commands/lsupdates.py
index 5ada1ac..68c6c42 100644
--- a/bitbake/lib/toaster/orm/management/commands/lsupdates.py
+++ b/bitbake/lib/toaster/orm/management/commands/lsupdates.py
@@ -153,41 +153,17 @@ class Command(NoArgsCommand):
 
         total = len(layers_info)
         for i, li in enumerate(layers_info):
-            # Special case for the openembedded-core layer
-            if li['name'] == oe_core_layer:
-                try:
-                    # If we have an existing openembedded-core for example
-                    # from the toasterconf.json augment the info using the
-                    # layerindex rather than duplicate it
-                    oe_core_l = Layer.objects.get(name=oe_core_layer)
-                    # Take ownership of the layer as now coming from the
-                    # layerindex
-                    oe_core_l.summary = li['summary']
-                    oe_core_l.description = li['description']
-                    oe_core_l.vcs_web_url = li['vcs_web_url']
-                    oe_core_l.vcs_web_tree_base_url = \
-                        li['vcs_web_tree_base_url']
-                    oe_core_l.vcs_web_file_base_url = \
-                        li['vcs_web_file_base_url']
-
-                    oe_core_l.save()
-                    li_layer_id_to_toaster_layer_id[li['id']] = oe_core_l.pk
-                    self.mini_progress("layers", i, total)
-                    continue
-
-                except Layer.DoesNotExist:
-                    pass
-
             try:
-                l, created = Layer.objects.get_or_create(name=li['name'],
-                                                         vcs_url=li['vcs_url'])
+                l, created = Layer.objects.get_or_create(name=li['name'])
                 l.up_date = li['updated']
-                l.vcs_url = li['vcs_url']
-                l.vcs_web_url = li['vcs_web_url']
-                l.vcs_web_tree_base_url = li['vcs_web_tree_base_url']
-                l.vcs_web_file_base_url = li['vcs_web_file_base_url']
                 l.summary = li['summary']
                 l.description = li['description']
+
+                if created:
+                    l.vcs_url = li['vcs_url']
+                    l.vcs_web_url = li['vcs_web_url']
+                    l.vcs_web_tree_base_url = li['vcs_web_tree_base_url']
+                    l.vcs_web_file_base_url = li['vcs_web_file_base_url']
                 l.save()
             except Layer.MultipleObjectsReturned:
                 logger.info("Skipped %s as we found multiple layers and "
@@ -210,12 +186,14 @@ class Command(NoArgsCommand):
 
         total = len(layerbranches_info)
         for i, lbi in enumerate(layerbranches_info):
+            # release as defined by toaster map to layerindex branch
+            release = li_branch_id_to_toaster_release[lbi['branch']]
 
             try:
                 lv, created = Layer_Version.objects.get_or_create(
-                    layer_source=LayerSource.TYPE_LAYERINDEX,
                     layer=Layer.objects.get(
-                        pk=li_layer_id_to_toaster_layer_id[lbi['layer']])
+                        pk=li_layer_id_to_toaster_layer_id[lbi['layer']]),
+                    release=release
                 )
             except KeyError:
                 logger.warning(
@@ -223,11 +201,12 @@ class Command(NoArgsCommand):
                     lbi['layer'])
                 continue
 
-            lv.release = li_branch_id_to_toaster_release[lbi['branch']]
-            lv.up_date = lbi['updated']
-            lv.commit = lbi['actual_branch']
-            lv.dirpath = lbi['vcs_subdir']
-            lv.save()
+            if created:
+                lv.release = li_branch_id_to_toaster_release[lbi['branch']]
+                lv.up_date = lbi['updated']
+                lv.commit = lbi['actual_branch']
+                lv.dirpath = lbi['vcs_subdir']
+                lv.save()
 
             li_layer_branch_id_to_toaster_lv_id[lbi['id']] =\
                 lv.pk
@@ -254,9 +233,8 @@ class Command(NoArgsCommand):
                 layer_id = li_layer_id_to_toaster_layer_id[ldi['dependency']]
 
                 dependlist[lv].append(
-                    Layer_Version.objects.get(
-                        layer_source=LayerSource.TYPE_LAYERINDEX,
-                        layer__pk=layer_id))
+                    Layer_Version.objects.get(layer__pk=layer_id,
+                                              release=lv.release))
 
             except Layer_Version.DoesNotExist:
                 logger.warning("Cannot find layer version (ls:%s),"
-- 
2.7.4

[PATCH 05/18] toaster: orm/fixtures Add the master release and correct morty release

[Michael Wood]

Add the master release option to base your project on and correct the
morty release so that for poky based setups we use the poky provided
version of the layer rather than checking out the layer from its own git
repository.

[YOCTO #10497]

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/toaster/orm/fixtures/oe-core.xml | 23 +++++++-
 bitbake/lib/toaster/orm/fixtures/poky.xml    | 82 +++++++++++++++++++++++++---
 2 files changed, 96 insertions(+), 9 deletions(-)

diff --git a/bitbake/lib/toaster/orm/fixtures/oe-core.xml b/bitbake/lib/toaster/orm/fixtures/oe-core.xml
index a6c834f..4d3d185 100644
--- a/bitbake/lib/toaster/orm/fixtures/oe-core.xml
+++ b/bitbake/lib/toaster/orm/fixtures/oe-core.xml
@@ -14,6 +14,13 @@
   </object>
   <object model="orm.bitbakeversion" pk="2">
     <field type="CharField" name="name">HEAD</field>
+    <field type="CharField" name="giturl">git://git.openembedded.org/bitbake</field>
+    <field type="CharField" name="branch">HEAD</field>
+  </object>
+  <object model="orm.bitbakeversion" pk="3">
+    <field type="CharField" name="name">master</field>
+    <field type="CharField" name="giturl">git://git.openembedded.org/bitbake</field>
+    <field type="CharField" name="branch">master</field>
   </object>
 
   <!-- Releases available -->
@@ -31,18 +38,30 @@
     <field type="CharField" name="branch_name">HEAD</field>
     <field type="TextField" name="helptext">Toaster will run your builds with the version of OpenEmbedded that you have cloned or downloaded to your computer.</field>
   </object>
+  <object model="orm.release" pk="3">
+    <field type="CharField" name="name">master</field>
+    <field type="CharField" name="description">OpenEmbedded core master</field>
+    <field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">3</field>
+    <field type="CharField" name="branch_name">master</field>
+    <field type="TextField" name="helptext">Toaster will run your builds using the tip of the &lt;a href=\"http://cgit.openembedded.org/openembedded-core/log/\"&gt;OpenEmbedded master&lt;/a&gt; branch.</field>
+  </object>
 
   <!-- Default layers for each release -->
   <object model="orm.releasedefaultlayer" pk="1">
     <field rel="ManyToOneRel" to="orm.release" name="release">1</field>
     <field type="CharField" name="layer_name">openembedded-core</field>
   </object>
-  <object model="orm.releasedefaultlayer" pk="4">
+  <object model="orm.releasedefaultlayer" pk="2">
     <field rel="ManyToOneRel" to="orm.release" name="release">2</field>
     <field type="CharField" name="layer_name">openembedded-core</field>
   </object>
+  <object model="orm.releasedefaultlayer" pk="3">
+    <field rel="ManyToOneRel" to="orm.release" name="release">3</field>
+    <field type="CharField" name="layer_name">openembedded-core</field>
+  </object>
+
 
-  <!-- TYPE_LOCAL = 0 Layers for the Local release -->
+  <!-- Layer for the Local release -->
   <object model="orm.layer" pk="1">
     <field type="CharField" name="name">openembedded-core</field>
     <field type="CharField" name="vcs_url">git://git.openembedded.org/openembedded-core</field>
diff --git a/bitbake/lib/toaster/orm/fixtures/poky.xml b/bitbake/lib/toaster/orm/fixtures/poky.xml
index c192baa..dcc7aaf 100644
--- a/bitbake/lib/toaster/orm/fixtures/poky.xml
+++ b/bitbake/lib/toaster/orm/fixtures/poky.xml
@@ -19,6 +19,13 @@
     <field type="CharField" name="branch">HEAD</field>
     <field type="CharField" name="dirpath">bitbake</field>
   </object>
+  <object model="orm.bitbakeversion" pk="3">
+    <field type="CharField" name="name">master</field>
+    <field type="CharField" name="giturl">git://git.yoctoproject.org/poky</field>
+    <field type="CharField" name="branch">master</field>
+    <field type="CharField" name="dirpath">bitbake</field>
+  </object>
+
 
   <!-- Releases available -->
   <object model="orm.release" pk="1">
@@ -35,8 +42,15 @@
     <field type="CharField" name="branch_name">HEAD</field>
     <field type="TextField" name="helptext">Toaster will run your builds with the version of the Yocto Project you have cloned or downloaded to your computer.</field>
   </object>
+  <object model="orm.release" pk="3">
+    <field type="CharField" name="name">master</field>
+    <field type="CharField" name="description">Yocto Project master</field>
+    <field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">3</field>
+    <field type="CharField" name="branch_name">master</field>
+    <field type="TextField" name="helptext">Toaster will run your builds using the tip of the &lt;a href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/"&gt;Yocto Project Master branch&lt;/a&gt;.</field>
+  </object>
 
-  <!-- Default layers for each release -->
+  <!-- Default project layers for each release -->
   <object model="orm.releasedefaultlayer" pk="1">
     <field rel="ManyToOneRel" to="orm.release" name="release">1</field>
     <field type="CharField" name="layer_name">openembedded-core</field>
@@ -61,9 +75,23 @@
     <field rel="ManyToOneRel" to="orm.release" name="release">2</field>
     <field type="CharField" name="layer_name">meta-yocto-bsp</field>
   </object>
+  <object model="orm.releasedefaultlayer" pk="7">
+    <field rel="ManyToOneRel" to="orm.release" name="release">3</field>
+    <field type="CharField" name="layer_name">openembedded-core</field>
+  </object>
+  <object model="orm.releasedefaultlayer" pk="8">
+    <field rel="ManyToOneRel" to="orm.release" name="release">3</field>
+    <field type="CharField" name="layer_name">meta-poky</field>
+  </object>
+  <object model="orm.releasedefaultlayer" pk="9">
+    <field rel="ManyToOneRel" to="orm.release" name="release">3</field>
+    <field type="CharField" name="layer_name">meta-yocto-bsp</field>
+  </object>
 
-  <!-- Layers for the Local release
-       layersource TYPE_LOCAL = 0
+  <!-- Default layers provided by poky
+       openembedded-core
+       meta-poky
+       meta-yocto-bsp
   -->
   <object model="orm.layer" pk="1">
     <field type="CharField" name="name">openembedded-core</field>
@@ -73,19 +101,39 @@
   <object model="orm.layer_version" pk="1">
     <field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
     <field type="IntegerField" name="layer_source">0</field>
+    <field rel="ManyToOneRel" to="orm.release" name="release">1</field>
+    <field type="CharField" name="branch">morty</field>
+    <field type="CharField" name="dirpath">meta</field>
+  </object>
+  <object model="orm.layer_version" pk="2">
+    <field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
+    <field type="IntegerField" name="layer_source">0</field>
     <field rel="ManyToOneRel" to="orm.release" name="release">2</field>
     <field type="CharField" name="branch">HEAD</field>
     <field type="CharField" name="commit">HEAD</field>
     <field type="CharField" name="dirpath">meta</field>
   </object>
-
+  <object model="orm.layer_version" pk="3">
+    <field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
+    <field type="IntegerField" name="layer_source">0</field>
+    <field rel="ManyToOneRel" to="orm.release" name="release">3</field>
+    <field type="CharField" name="branch">master</field>
+    <field type="CharField" name="dirpath">meta</field>
+  </object>
 
   <object model="orm.layer" pk="2">
     <field type="CharField" name="name">meta-poky</field>
     <field type="CharField" name="layer_index_url"></field>
     <field type="CharField" name="vcs_url">git://git.yoctoproject.org/poky</field>
   </object>
-  <object model="orm.layer_version" pk="2">
+  <object model="orm.layer_version" pk="4">
+    <field rel="ManyToOneRel" to="orm.layer" name="layer">2</field>
+    <field type="IntegerField" name="layer_source">0</field>
+    <field rel="ManyToOneRel" to="orm.release" name="release">1</field>
+    <field type="CharField" name="branch">morty</field>
+    <field type="CharField" name="dirpath">meta-poky</field>
+  </object>
+  <object model="orm.layer_version" pk="5">
     <field rel="ManyToOneRel" to="orm.layer" name="layer">2</field>
     <field type="IntegerField" name="layer_source">0</field>
     <field rel="ManyToOneRel" to="orm.release" name="release">2</field>
@@ -93,14 +141,27 @@
     <field type="CharField" name="commit">HEAD</field>
     <field type="CharField" name="dirpath">meta-poky</field>
   </object>
-
+  <object model="orm.layer_version" pk="6">
+    <field rel="ManyToOneRel" to="orm.layer" name="layer">2</field>
+    <field type="IntegerField" name="layer_source">0</field>
+    <field rel="ManyToOneRel" to="orm.release" name="release">3</field>
+    <field type="CharField" name="branch">master</field>
+    <field type="CharField" name="dirpath">meta-poky</field>
+  </object>
 
   <object model="orm.layer" pk="3">
     <field type="CharField" name="name">meta-yocto-bsp</field>
     <field type="CharField" name="layer_index_url"></field>
     <field type="CharField" name="vcs_url">git://git.yoctoproject.org/poky</field>
   </object>
-  <object model="orm.layer_version" pk="3">
+  <object model="orm.layer_version" pk="7">
+    <field rel="ManyToOneRel" to="orm.layer" name="layer">3</field>
+    <field type="IntegerField" name="layer_source">0</field>
+    <field rel="ManyToOneRel" to="orm.release" name="release">1</field>
+    <field type="CharField" name="branch">morty</field>
+    <field type="CharField" name="dirpath">meta-yocto-bsp</field>
+  </object>
+  <object model="orm.layer_version" pk="8">
     <field rel="ManyToOneRel" to="orm.layer" name="layer">3</field>
     <field type="IntegerField" name="layer_source">0</field>
     <field rel="ManyToOneRel" to="orm.release" name="release">2</field>
@@ -108,4 +169,11 @@
     <field type="CharField" name="commit">HEAD</field>
     <field type="CharField" name="dirpath">meta-yocto-bsp</field>
   </object>
+  <object model="orm.layer_version" pk="9">
+    <field rel="ManyToOneRel" to="orm.layer" name="layer">3</field>
+    <field type="IntegerField" name="layer_source">0</field>
+    <field rel="ManyToOneRel" to="orm.release" name="release">3</field>
+    <field type="CharField" name="branch">master</field>
+    <field type="CharField" name="dirpath">meta-yocto-bsp</field>
+  </object>
 </django-objects>
-- 
2.7.4

[PATCH 06/18] toaster: settings fixture Set default release to master

[Michael Wood]

Now that morty has been released we now set the DEFAULT_RELEASE back to
master.

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/toaster/orm/fixtures/settings.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bitbake/lib/toaster/orm/fixtures/settings.xml b/bitbake/lib/toaster/orm/fixtures/settings.xml
index ee6a202..78c0fdc 100644
--- a/bitbake/lib/toaster/orm/fixtures/settings.xml
+++ b/bitbake/lib/toaster/orm/fixtures/settings.xml
@@ -4,7 +4,7 @@
   <!-- pk=1 is DISTRO -->
   <object model="orm.toastersetting" pk="2">
     <field type="CharField" name="name">DEFAULT_RELEASE</field>
-    <field type="CharField" name="value">morty</field>
+    <field type="CharField" name="value">master</field>
   </object>
   <object model="orm.toastersetting" pk="3">
     <field type="CharField" name="name">DEFCONF_PACKAGE_CLASSES</field>
-- 
2.7.4

[PATCH 07/18] toaster: customrecipejs Consume click event on 'a' link if disabled

[Michael Wood]

Consume the click event on the download recipe link if it's disabled. To
prevent the link from sending user to an error page.
See http://getbootstrap.com/css/#forms-disabled-fieldsets and a link
caveat.

[YOCTO #10151]

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/toaster/toastergui/static/js/customrecipe.js | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/bitbake/lib/toaster/toastergui/static/js/customrecipe.js b/bitbake/lib/toaster/toastergui/static/js/customrecipe.js
index 9ea9602..8b1c190 100644
--- a/bitbake/lib/toaster/toastergui/static/js/customrecipe.js
+++ b/bitbake/lib/toaster/toastergui/static/js/customrecipe.js
@@ -312,5 +312,11 @@ function customRecipePageInit(ctx) {
     });
   });
 
+ /* Stop the download link from working if it is in disabled state
+  * http://getbootstrap.com/css/#forms-disabled-fieldsets
+  */
+ $("a[disabled=disabled]").click(function(e){
+   e.preventDefault();
+ });
 
 }
-- 
2.7.4

[PATCH 08/18] toaster: orm gen_layerdeps Protect against circular Layer dependencies

[Michael Wood]

From: "Reyna, David" <david.reyna@windriver.com>

Limit the recursion (to say 20 levels) when processing layer dependencies
so that circular dependecies do not cause infinite decent and an
out-of-memory failure. The duplicate found layers are already immediately
filtered in the code.

[YOCTO #10630]

Signed-off-by: David Reyna <David.Reyna@windriver.com>
---
 bitbake/lib/toaster/orm/models.py | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/bitbake/lib/toaster/orm/models.py b/bitbake/lib/toaster/orm/models.py
index 4f8510c..b24e9c5 100644
--- a/bitbake/lib/toaster/orm/models.py
+++ b/bitbake/lib/toaster/orm/models.py
@@ -1478,17 +1478,22 @@ class Layer_Version(models.Model):
 
     def get_alldeps(self, project_id):
         """Get full list of unique layer dependencies."""
-        def gen_layerdeps(lver, project):
+        def gen_layerdeps(lver, project, depth):
+            if depth == 0:
+                return
             for ldep in lver.dependencies.all():
                 yield ldep.depends_on
                 # get next level of deps recursively calling gen_layerdeps
-                for subdep in gen_layerdeps(ldep.depends_on, project):
+                for subdep in gen_layerdeps(ldep.depends_on, project, depth-1):
                     yield subdep
 
         project = Project.objects.get(pk=project_id)
         result = []
-        projectlvers = [player.layercommit for player in project.projectlayer_set.all()]
-        for dep in gen_layerdeps(self, project):
+        projectlvers = [player.layercommit for player in
+                        project.projectlayer_set.all()]
+        # protect against infinite layer dependency loops
+        maxdepth = 20
+        for dep in gen_layerdeps(self, project, maxdepth):
             # filter out duplicates and layers already belonging to the project
             if dep not in result + projectlvers:
                 result.append(dep)
-- 
2.7.4

[PATCH 09/18] toaster: tablejs Fix missing close square bracket

[Michael Wood]

From: "Reyna, David" <david.reyna@windriver.com>

There is a missing close square bracket.

[YOCTO #10631]

Signed-off-by: David Reyna <David.Reyna@windriver.com>
---
 bitbake/lib/toaster/toastergui/static/js/table.js | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bitbake/lib/toaster/toastergui/static/js/table.js b/bitbake/lib/toaster/toastergui/static/js/table.js
index 176ce57..aaa3e1f 100644
--- a/bitbake/lib/toaster/toastergui/static/js/table.js
+++ b/bitbake/lib/toaster/toastergui/static/js/table.js
@@ -706,7 +706,7 @@ function tableInit(ctx){
             if (action) {
               // Setup the current selected filter; default to 'all' if
               // no current filter selected
-              var radioInput = action.find('input[name="filter"]');
+              var radioInput = action.find('input[name]="filter"]');
               if ((tableParams.filter &&
                   tableParams.filter === radioInput.val()) ||
                   filterActionData.action_name == 'all') {
-- 
2.7.4

[PATCH 10/18] toaster: localhostbecontroller accept custom init script for build

[Michael Wood]

From: Sujith H <sujith.h@gmail.com>

When passed variable CUSTOM_BUILD_INIT_SCRIPT to toaster
setting, it would be nice to use it. Else toaster
can use oe-init script. This gives an oppurtunity to
use customized build init scritps.

Signed-off-by: Sujith H <sujith.h@gmail.com>
---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index e5f7c98..3896a82 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -27,7 +27,7 @@ import shutil
 from django.db import transaction
 from django.db.models import Q
 from bldcontrol.models import BuildEnvironment, BRLayer, BRVariable, BRTarget, BRBitbake
-from orm.models import CustomImageRecipe, Layer, Layer_Version, ProjectLayer
+from orm.models import CustomImageRecipe, Layer, Layer_Version, ProjectLayer, ToasterSetting
 import subprocess
 
 from toastermain import settings
@@ -277,7 +277,12 @@ class LocalhostBEController(BuildEnvironmentController):
         builddir = '%s-toaster-%d' % (self.be.builddir, bitbake.req.project.id)
         oe_init = os.path.join(self.pokydirname, 'oe-init-build-env')
         # init build environment
-        self._shellcmd("bash -c 'source %s %s'" % (oe_init, builddir),
+        try:
+            custom_script = ToasterSetting.objects.get(name="CUSTOM_BUILD_INIT_SCRIPT").value
+            custom_script = custom_script.replace("%BUILDDIR%" ,builddir)
+            self._shellcmd("bash -c 'source %s'" % (custom_script))
+        except ToasterSetting.DoesNotExist:
+            self._shellcmd("bash -c 'source %s %s'" % (oe_init, builddir),
                        self.be.sourcedir)
 
         # update bblayers.conf
-- 
2.7.4

[PATCH 11/18] toaster: localhostbecontroller write toaster layers for project to toaster-bblayers.conf

[Michael Wood]

From: Sujith H <sujith.h@gmail.com>

Instead of updating conf/bblayers, here we update toaster-bblayers.conf
file. So extra effort to update bblayers.conf can be removed safely.

Signed-off-by: Sujith H <sujith.h@gmail.com>
---
 .../lib/toaster/bldcontrol/localhostbecontroller.py    | 18 ++++--------------
 1 file changed, 4 insertions(+), 14 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 3896a82..11335ac 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -286,19 +286,8 @@ class LocalhostBEController(BuildEnvironmentController):
                        self.be.sourcedir)
 
         # update bblayers.conf
-        bblconfpath = os.path.join(builddir, "conf/bblayers.conf")
-        conflines = open(bblconfpath, "r").readlines()
-        skip = False
+        bblconfpath = os.path.join(builddir, "conf/toaster-bblayers.conf")
         with open(bblconfpath, 'w') as bblayers:
-            for line in conflines:
-                if line.startswith("# line added by toaster"):
-                    skip = True
-                    continue
-                if skip:
-                    skip = False
-                else:
-                    bblayers.write(line)
-
             bblayers.write('# line added by toaster build control\n'
                            'BBLAYERS = "%s"' % ' '.join(layers))
 
@@ -311,9 +300,10 @@ class LocalhostBEController(BuildEnvironmentController):
 
         # run bitbake server from the clone
         bitbake = os.path.join(self.pokydirname, 'bitbake', 'bin', 'bitbake')
-        self._shellcmd('bash -c \"source %s %s; BITBAKE_UI="knotty" %s --read %s '
+        toasterlayers = os.path.join(builddir,"conf/toaster-bblayers.conf")
+        self._shellcmd('bash -c \"source %s %s; BITBAKE_UI="knotty" %s --read %s --read %s '
                        '--server-only -t xmlrpc -B 0.0.0.0:0\"' % (oe_init,
-                       builddir, bitbake, confpath), self.be.sourcedir)
+                       builddir, bitbake, confpath, toasterlayers), self.be.sourcedir)
 
         # read port number from bitbake.lock
         self.be.bbport = ""
-- 
2.7.4

[PATCH 12/18] toaster: buildinfohelper toaster-custom-images layer

[Michael Wood]

This fixes the unidentified layers issue by making the
toaster-custom-images layer a local layer. By doing this we also fix the
git assumptions made for the local layers which stop recipes and other
meta data being associated with them. This also removed some of the
special casing previously needed when we didn't have the concept of a
local (non git) layer.

Also rename created flag var to a have a different var for each returned
value so that the same value isn't used multiple times.

[YOCTO #10220]

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py               |  7 +-----
 .../toaster/bldcontrol/localhostbecontroller.py    | 25 ++++++++++++++++------
 bitbake/lib/toaster/orm/models.py                  |  3 ++-
 bitbake/lib/toaster/toastergui/api.py              | 16 ++++++++------
 4 files changed, 31 insertions(+), 20 deletions(-)

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index 3ddcb2a..e96e934 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -42,7 +42,7 @@ from orm.models import Variable, VariableHistory
 from orm.models import Package, Package_File, Target_Installed_Package, Target_File
 from orm.models import Task_Dependency, Package_Dependency
 from orm.models import Recipe_Dependency, Provides
-from orm.models import Project, CustomImagePackage, CustomImageRecipe
+from orm.models import Project, CustomImagePackage
 from orm.models import signal_runbuilds
 
 from bldcontrol.models import BuildEnvironment, BuildRequest
@@ -361,11 +361,6 @@ class ORMWrapper(object):
 
     def get_update_layer_version_object(self, build_obj, layer_obj, layer_version_information):
         if isinstance(layer_obj, Layer_Version):
-            # Special case the toaster-custom-images layer which is created
-            # on the fly so don't update the values which may cause the layer
-            # to be duplicated on a future get_or_create
-            if layer_obj.layer.name == CustomImageRecipe.LAYER_NAME:
-                return layer_obj
             # We already found our layer version for this build so just
             # update it with the new build information
             logger.debug("We found our layer from toaster")
diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 11335ac..18870d9 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -228,13 +228,22 @@ class LocalhostBEController(BuildEnvironmentController):
             br_layer_base_recipe = layers.get(
                 layer_version=customrecipe.base_recipe.layer_version)
 
-            br_layer_base_dirpath = \
-                    os.path.join(self.be.sourcedir,
-                                 self.getGitCloneDirectory(
-                                     br_layer_base_recipe.giturl,
-                                     br_layer_base_recipe.commit),
-                                 customrecipe.base_recipe.layer_version.dirpath
-                                )
+            # If the layer is one that we've cloned we know where it lives
+            if br_layer_base_recipe.giturl and br_layer_base_recipe.commit:
+                layer_path = self.getGitCloneDirectory(
+                    br_layer_base_recipe.giturl,
+                    br_layer_base_recipe.commit)
+            # Otherwise it's a local layer
+            elif br_layer_base_recipe.local_source_dir:
+                layer_path = br_layer_base_recipe.local_source_dir
+            else:
+                logger.error("Unable to workout the dir path for the custom"
+                             " image recipe")
+
+            br_layer_base_dirpath = os.path.join(
+                self.be.sourcedir,
+                layer_path,
+                customrecipe.base_recipe.layer_version.dirpath)
 
             customrecipe.base_recipe.layer_version.dirpath = \
                          br_layer_base_dirpath
@@ -249,6 +258,8 @@ class LocalhostBEController(BuildEnvironmentController):
 
             # Update the layer and recipe objects
             customrecipe.layer_version.dirpath = layerpath
+            customrecipe.layer_version.layer.local_source_dir = layerpath
+            customrecipe.layer_version.layer.save()
             customrecipe.layer_version.save()
 
             customrecipe.file_path = recipe_path
diff --git a/bitbake/lib/toaster/orm/models.py b/bitbake/lib/toaster/orm/models.py
index b24e9c5..fd15fbe 100644
--- a/bitbake/lib/toaster/orm/models.py
+++ b/bitbake/lib/toaster/orm/models.py
@@ -1636,7 +1636,8 @@ class CustomImageRecipe(Recipe):
         if base_recipe_path:
             base_recipe = open(base_recipe_path, 'r').read()
         else:
-            raise IOError("Based on recipe file not found")
+            raise IOError("Based on recipe file not found: %s" %
+                          base_recipe_path)
 
         # Add a special case for when the recipe we have based a custom image
         # recipe on requires another recipe.
diff --git a/bitbake/lib/toaster/toastergui/api.py b/bitbake/lib/toaster/toastergui/api.py
index ae1f150..4851047 100644
--- a/bitbake/lib/toaster/toastergui/api.py
+++ b/bitbake/lib/toaster/toastergui/api.py
@@ -291,10 +291,13 @@ class XhrCustomRecipe(View):
                 return error_response("recipe-already-exists")
 
             # create layer 'Custom layer' and verion if needed
-            layer = Layer.objects.get_or_create(
+            layer, l_created = Layer.objects.get_or_create(
                 name=CustomImageRecipe.LAYER_NAME,
-                summary="Layer for custom recipes",
-                vcs_url="file:///toaster_created_layer")[0]
+                summary="Layer for custom recipes")
+
+            if l_created:
+                layer.local_source_dir = "toaster_created_layer"
+                layer.save()
 
             # Check if we have a layer version already
             # We don't use get_or_create here because the dirpath will change
@@ -303,9 +306,10 @@ class XhrCustomRecipe(View):
                                                 Q(layer=layer) &
                                                 Q(build=None)).last()
             if lver is None:
-                lver, created = Layer_Version.objects.get_or_create(
+                lver, lv_created = Layer_Version.objects.get_or_create(
                     project=params['project'],
                     layer=layer,
+                    layer_source=LayerSource.TYPE_LOCAL,
                     dirpath="toaster_created_layer")
 
             # Add a dependency on our layer to the base recipe's layer
@@ -319,7 +323,7 @@ class XhrCustomRecipe(View):
                                                optional=False)
 
             # Create the actual recipe
-            recipe, created = CustomImageRecipe.objects.get_or_create(
+            recipe, r_created = CustomImageRecipe.objects.get_or_create(
                 name=request.POST["name"],
                 base_recipe=params["base"],
                 project=params["project"],
@@ -329,7 +333,7 @@ class XhrCustomRecipe(View):
             # If we created the object then setup these fields. They may get
             # overwritten later on and cause the get_or_create to create a
             # duplicate if they've changed.
-            if created:
+            if r_created:
                 recipe.file_path = request.POST["name"]
                 recipe.license = "MIT"
                 recipe.version = "0.1"
-- 
2.7.4

[PATCH 13/18] toaster: orm models Project class Fix pyflake errors

[Michael Wood]

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/toaster/orm/models.py | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)

diff --git a/bitbake/lib/toaster/orm/models.py b/bitbake/lib/toaster/orm/models.py
index fd15fbe..394c886 100644
--- a/bitbake/lib/toaster/orm/models.py
+++ b/bitbake/lib/toaster/orm/models.py
@@ -38,6 +38,7 @@ import re
 import itertools
 from signal import SIGUSR1
 
+
 import logging
 logger = logging.getLogger("toaster")
 
@@ -178,24 +179,27 @@ class ProjectManager(models.Manager):
         else:
             return projects[0]
 
+
 class Project(models.Model):
-    search_allowed_fields = ['name', 'short_description', 'release__name', 'release__branch_name']
+    search_allowed_fields = ['name', 'short_description', 'release__name',
+                             'release__branch_name']
     name = models.CharField(max_length=100)
     short_description = models.CharField(max_length=50, blank=True)
     bitbake_version = models.ForeignKey('BitbakeVersion', null=True)
-    release     = models.ForeignKey("Release", null=True)
-    created     = models.DateTimeField(auto_now_add = True)
-    updated     = models.DateTimeField(auto_now = True)
+    release = models.ForeignKey("Release", null=True)
+    created = models.DateTimeField(auto_now_add=True)
+    updated = models.DateTimeField(auto_now=True)
     # This is a horrible hack; since Toaster has no "User" model available when
     # running in interactive mode, we can't reference the field here directly
-    # Instead, we keep a possible null reference to the User id, as not to force
+    # Instead, we keep a possible null reference to the User id,
+    # as not to force
     # hard links to possibly missing models
-    user_id     = models.IntegerField(null = True)
-    objects     = ProjectManager()
+    user_id = models.IntegerField(null=True)
+    objects = ProjectManager()
 
     # set to True for the project which is the default container
     # for builds initiated by the command line etc.
-    is_default  = models.BooleanField(default = False)
+    is_default= models.BooleanField(default=False)
 
     def __unicode__(self):
         return "%s (Release %s, BBV %s)" % (self.name, self.release, self.bitbake_version)
-- 
2.7.4

[PATCH 14/18] toaster: orm models Handle CustomImageRecipe BRLayer here

[Michael Wood]

The schedule_build function on the project object is where the BRLayers
are created for the build. Instead of creating the BRLayer for the
CustomImageRecipe in the localhostbbcontroller create it here so that
all that mechanism is in one place.

Also fix a number of pyflake errors.

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/toaster/orm/models.py | 58 +++++++++++++++++++++++++--------------
 1 file changed, 37 insertions(+), 21 deletions(-)

diff --git a/bitbake/lib/toaster/orm/models.py b/bitbake/lib/toaster/orm/models.py
index 394c886..8816fe0 100644
--- a/bitbake/lib/toaster/orm/models.py
+++ b/bitbake/lib/toaster/orm/models.py
@@ -337,20 +337,45 @@ class Project(models.Model):
 
         return queryset
 
-
     def schedule_build(self):
-        from bldcontrol.models import BuildRequest, BRTarget, BRLayer, BRVariable, BRBitbake
-        br = BuildRequest.objects.create(project = self)
+
+        from bldcontrol.models import BuildRequest, BRTarget, BRLayer
+        from bldcontrol.models import BRBitbake, BRVariable
+
         try:
+            now = timezone.now()
+            build = Build.objects.create(project=self,
+                                         completed_on=now,
+                                         started_on=now)
+
+            br = BuildRequest.objects.create(project=self,
+                                             state=BuildRequest.REQ_QUEUED,
+                                             build=build)
+            BRBitbake.objects.create(req=br,
+                                     giturl=self.bitbake_version.giturl,
+                                     commit=self.bitbake_version.branch,
+                                     dirpath=self.bitbake_version.dirpath)
 
-            BRBitbake.objects.create(req = br,
-                giturl = self.bitbake_version.giturl,
-                commit = self.bitbake_version.branch,
-                dirpath = self.bitbake_version.dirpath)
+            for t in self.projecttarget_set.all():
+                BRTarget.objects.create(req=br, target=t.target, task=t.task)
+                Target.objects.create(build=br.build, target=t.target,
+                                      task=t.task)
+                # If we're about to build a custom image recipe make sure
+                # that layer is currently in the project before we create the
+                # BRLayer objects
+                customrecipe = CustomImageRecipe.objects.filter(
+                    name=t.target,
+                    project=self).first()
+                if customrecipe:
+                    ProjectLayer.objects.get_or_create(
+                        project=self,
+                        layercommit=customrecipe.layer_version,
+                        optional=False)
 
             for l in self.projectlayer_set.all().order_by("pk"):
                 commit = l.layercommit.get_vcs_reference()
-                print("ii Building layer ", l.layercommit.layer.name, " at vcs point ", commit)
+                logger.debug("Adding layer to build %s" %
+                             l.layercommit.layer.name)
                 BRLayer.objects.create(
                     req=br,
                     name=l.layercommit.layer.name,
@@ -361,25 +386,16 @@ class Project(models.Model):
                     local_source_dir=l.layercommit.layer.local_source_dir
                 )
 
-            br.state = BuildRequest.REQ_QUEUED
-            now = timezone.now()
-            br.build = Build.objects.create(project = self,
-                                completed_on=now,
-                                started_on=now,
-                                )
-            for t in self.projecttarget_set.all():
-                BRTarget.objects.create(req = br, target = t.target, task = t.task)
-                Target.objects.create(build = br.build, target = t.target, task = t.task)
-
             for v in self.projectvariable_set.all():
-                BRVariable.objects.create(req = br, name = v.name, value = v.value)
-
+                BRVariable.objects.create(req=br, name=v.name, value=v.value)
 
             try:
-                br.build.machine = self.projectvariable_set.get(name = 'MACHINE').value
+                br.build.machine = self.projectvariable_set.get(
+                    name='MACHINE').value
                 br.build.save()
             except ProjectVariable.DoesNotExist:
                 pass
+
             br.save()
             signal_runbuilds()
 
-- 
2.7.4

[PATCH 15/18] toaster: bldcontrol Move CustomImageRecipe file creation into own function

[Michael Wood]

Move the custom image file creation (i.e. create the layer file
structure, conf and recipe file) into it's own function and remove the
creation of the BRLayer as this is done at schedule_build just like all
the other layers.

Fix a bug where the toaster-custom-images layer was always being appened
to the layer list if the directory exists.

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 .../toaster/bldcontrol/localhostbecontroller.py    | 144 +++++++++++----------
 1 file changed, 75 insertions(+), 69 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 18870d9..62e62fe 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -201,82 +201,88 @@ class LocalhostBEController(BuildEnvironmentController):
         logger.debug("localhostbecontroller: current layer list %s " % pformat(layerlist))
 
         # 5. create custom layer and add custom recipes to it
-        layerpath = os.path.join(self.be.builddir,
-                                 CustomImageRecipe.LAYER_NAME)
         for target in targets:
             try:
-                customrecipe = CustomImageRecipe.objects.get(name=target.target,
-                                                             project=bitbake.req.project)
+                customrecipe = CustomImageRecipe.objects.get(
+                    name=target.target,
+                    project=bitbake.req.project)
+
+                custom_layer_path = self.setup_custom_image_recipe(
+                    customrecipe, layers)
+
+                if os.path.isdir(custom_layer_path):
+                    layerlist.append(custom_layer_path)
+
             except CustomImageRecipe.DoesNotExist:
-                continue # not a custom recipe, skip
-
-            # create directory structure
-            for name in ("conf", "recipes"):
-                path = os.path.join(layerpath, name)
-                if not os.path.isdir(path):
-                    os.makedirs(path)
-
-            # create layer.oonf
-            config = os.path.join(layerpath, "conf", "layer.conf")
-            if not os.path.isfile(config):
-                with open(config, "w") as conf:
-                    conf.write('BBPATH .= ":${LAYERDIR}"\nBBFILES += "${LAYERDIR}/recipes/*.bb"\n')
-
-            # Update the Layer_Version dirpath that has our base_recipe in
-            # to be able to read the base recipe to then  generate the
-            # custom recipe.
-            br_layer_base_recipe = layers.get(
-                layer_version=customrecipe.base_recipe.layer_version)
-
-            # If the layer is one that we've cloned we know where it lives
-            if br_layer_base_recipe.giturl and br_layer_base_recipe.commit:
-                layer_path = self.getGitCloneDirectory(
-                    br_layer_base_recipe.giturl,
-                    br_layer_base_recipe.commit)
-            # Otherwise it's a local layer
-            elif br_layer_base_recipe.local_source_dir:
-                layer_path = br_layer_base_recipe.local_source_dir
-            else:
-                logger.error("Unable to workout the dir path for the custom"
-                             " image recipe")
-
-            br_layer_base_dirpath = os.path.join(
-                self.be.sourcedir,
-                layer_path,
-                customrecipe.base_recipe.layer_version.dirpath)
-
-            customrecipe.base_recipe.layer_version.dirpath = \
-                         br_layer_base_dirpath
-
-            customrecipe.base_recipe.layer_version.save()
-
-            # create recipe
-            recipe_path = \
-                    os.path.join(layerpath, "recipes", "%s.bb" % target.target)
-            with open(recipe_path, "w") as recipef:
-                recipef.write(customrecipe.generate_recipe_file_contents())
-
-            # Update the layer and recipe objects
-            customrecipe.layer_version.dirpath = layerpath
-            customrecipe.layer_version.layer.local_source_dir = layerpath
-            customrecipe.layer_version.layer.save()
-            customrecipe.layer_version.save()
-
-            customrecipe.file_path = recipe_path
-            customrecipe.save()
-
-            # create *Layer* objects needed for build machinery to work
-            BRLayer.objects.get_or_create(req=target.req,
-                                          name=layer.name,
-                                          dirpath=layerpath,
-                                          giturl="file://%s" % layerpath)
-        if os.path.isdir(layerpath):
-            layerlist.append(layerpath)
+                continue  # not a custom recipe, skip
 
-        self.islayerset = True
         layerlist.extend(nongitlayerlist)
+        logger.debug("\n\nset layers gives this list \n %s" % ''.join(layerlist))
+        self.islayerset = True
         return layerlist
 
+    def setup_custom_image_recipe(self, customrecipe, layers):
+        """ Set up toaster-custom-images layer and recipe files """
+        layerpath = os.path.join(self.be.builddir,
+                                 CustomImageRecipe.LAYER_NAME)
+
+        # create directory structure
+        for name in ("conf", "recipes"):
+            path = os.path.join(layerpath, name)
+            if not os.path.isdir(path):
+                os.makedirs(path)
+
+        # create layer.conf
+        config = os.path.join(layerpath, "conf", "layer.conf")
+        if not os.path.isfile(config):
+            with open(config, "w") as conf:
+                conf.write('BBPATH .= ":${LAYERDIR}"\nBBFILES += "${LAYERDIR}/recipes/*.bb"\n')
+
+        # Update the Layer_Version dirpath that has our base_recipe in
+        # to be able to read the base recipe to then  generate the
+        # custom recipe.
+        br_layer_base_recipe = layers.get(
+            layer_version=customrecipe.base_recipe.layer_version)
+
+        # If the layer is one that we've cloned we know where it lives
+        if br_layer_base_recipe.giturl and br_layer_base_recipe.commit:
+            layer_path = self.getGitCloneDirectory(
+                br_layer_base_recipe.giturl,
+                br_layer_base_recipe.commit)
+        # Otherwise it's a local layer
+        elif br_layer_base_recipe.local_source_dir:
+            layer_path = br_layer_base_recipe.local_source_dir
+        else:
+            logger.error("Unable to workout the dir path for the custom"
+                         " image recipe")
+
+        br_layer_base_dirpath = os.path.join(
+            self.be.sourcedir,
+            layer_path,
+            customrecipe.base_recipe.layer_version.dirpath)
+
+        customrecipe.base_recipe.layer_version.dirpath = br_layer_base_dirpath
+
+        customrecipe.base_recipe.layer_version.save()
+
+        # create recipe
+        recipe_path = os.path.join(layerpath, "recipes", "%s.bb" %
+                                   customrecipe.name)
+        with open(recipe_path, "w") as recipef:
+            recipef.write(customrecipe.generate_recipe_file_contents())
+
+        # Update the layer and recipe objects
+        customrecipe.layer_version.dirpath = layerpath
+        customrecipe.layer_version.layer.local_source_dir = layerpath
+        customrecipe.layer_version.layer.save()
+        customrecipe.layer_version.save()
+
+        customrecipe.file_path = recipe_path
+        customrecipe.save()
+
+        return layerpath
+
+
     def readServerLogFile(self):
         return open(os.path.join(self.be.builddir, "toaster_server.log"), "r").read()
 
-- 
2.7.4

[PATCH 16/18] toaster: buildinfohelper Simplify layer event to toaster layer function

[Michael Wood]

Simplify the layer event information to layer version object in toaster
function. Previously this attempted many different methods of trying to
obtain the correct layer from toaster by manipulating the data from the
event or the data from the known layers to try and match them together.

We speed up and simplify this process by making better use of django's
orm methods and by working down the most likely matching methods in order
of accuracy.

[YOCTO #10220]

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py | 69 +++++++++++++++---------------------
 1 file changed, 28 insertions(+), 41 deletions(-)

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index e96e934..43a1411 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -46,6 +46,8 @@ from orm.models import Project, CustomImagePackage
 from orm.models import signal_runbuilds
 
 from bldcontrol.models import BuildEnvironment, BuildRequest
+from bldcontrol.models import BRLayer
+from bldcontrol import bbcontroller
 
 from bb.msg import BBLogFormatter as formatter
 from django.db import models
@@ -436,48 +438,33 @@ class ORMWrapper(object):
         else:
             br_id, be_id = brbe.split(":")
 
-            # find layer by checkout path;
-            from bldcontrol import bbcontroller
-            bc = bbcontroller.getBuildEnvironmentController(pk = be_id)
-
-            # we might have a race condition here, as the project layers may change between the build trigger and the actual build execution
-            # but we can only match on the layer name, so the worst thing can happen is a mis-identification of the layer, not a total failure
-
-            # note that this is different
-            buildrequest = BuildRequest.objects.get(pk = br_id)
-            for brl in buildrequest.brlayer_set.all():
-                if brl.local_source_dir:
-                    localdirname = os.path.join(brl.local_source_dir,
-                                                brl.dirpath)
-                else:
-                    localdirname = os.path.join(bc.getGitCloneDirectory(brl.giturl, brl.commit), brl.dirpath)
-                # we get a relative path, unless running in HEAD mode where the path is absolute
-                if not localdirname.startswith("/"):
-                    localdirname = os.path.join(bc.be.sourcedir, localdirname)
-                #logger.debug(1, "Localdirname %s lcal_path %s" % (localdirname, layer_information['local_path']))
-                if localdirname.startswith(layer_information['local_path']):
-                  # If the build request came from toaster this field
-                  # should contain the information from the layer_version
-                  # That created this build request.
-                    if brl.layer_version:
-                        return brl.layer_version
-
-                # This might be a local layer (i.e. no git info) so try
-                # matching local_source_dir
-                if brl.local_source_dir and brl.local_source_dir == layer_information["local_path"]:
-                    return brl.layer_version
-
-                    # we matched the BRLayer, but we need the layer_version that generated this BR; reverse of the Project.schedule_build()
-                    #logger.debug(1, "Matched %s to BRlayer %s" % (pformat(layer_information["local_path"]), localdirname))
-
-                    for pl in buildrequest.project.projectlayer_set.filter(layercommit__layer__name = brl.name):
-                        if pl.layercommit.layer.vcs_url == brl.giturl :
-                            layer = pl.layercommit.layer
-                            layer.save()
-                            return layer
-
-            raise NotExisting("Unidentified layer %s" % pformat(layer_information))
+            # Find the layer version by matching the layer event information
+            # against the metadata we have in Toaster
 
+            try:
+                br_layer = BRLayer.objects.get(req=br_id,
+                                               name=layer_information['name'])
+                return br_layer.layer_version
+            except (BRLayer.MultipleObjectsReturned, BRLayer.DoesNotExist):
+                # There are multiple of the same layer name or the name
+                # hasn't been determined by the toaster.bbclass layer
+                # so let's filter by the local_path
+                bc = bbcontroller.getBuildEnvironmentController(pk=be_id)
+                for br_layer in BRLayer.objects.filter(req=br_id):
+                    if br_layer.giturl and \
+                       layer_information['local_path'].endswith(
+                           bc.getGitCloneDirectory(br_layer.giturl,
+                                                   br_layer.commit)):
+                            return br_layer.layer_version
+
+                    if br_layer.local_source_dir == \
+                            layer_information['local_path']:
+                        return br_layer.layer_version
+
+        # We've reached the end of our search and couldn't find the layer
+        # we can continue but some data may be missing
+        raise NotExisting("Unidentified layer %s" %
+                          pformat(layer_information))
 
     def save_target_file_information(self, build_obj, target_obj, filedata):
         assert isinstance(build_obj, Build)
-- 
2.7.4

[PATCH 17/18] toaster: buildinfohelper fix _get_layer_version_for_dependency

[Michael Wood]

This function is simplified by not trying to handle replacing the regex
and just compiling and using it for matching.

- Fix typo in logger output with undefined variable
- Fix pyflake errors

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py | 32 ++++++++++++++++++++------------
 1 file changed, 20 insertions(+), 12 deletions(-)

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index 43a1411..63c779e 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -965,9 +965,10 @@ class BuildInfoHelper(object):
         return task_information
 
     def _get_layer_version_for_dependency(self, pathRE):
-        """ Returns the layer in the toaster db that has a full regex match to the pathRE.
-        pathRE - the layer path passed as a regex in the event. It is created in
-          cooker.py as a collection for the layer priorities.
+        """ Returns the layer in the toaster db that has a full regex
+        match to the pathRE. pathRE - the layer path passed as a regex in the
+        event. It is created in cooker.py as a collection for the layer
+        priorities.
         """
         self._ensure_build()
 
@@ -975,19 +976,26 @@ class BuildInfoHelper(object):
             assert isinstance(layer_version, Layer_Version)
             return len(layer_version.local_path)
 
-        # we don't care if we match the trailing slashes
-        p = re.compile(re.sub("/[^/]*?$","",pathRE))
-        # Heuristics: we always match recipe to the deepest layer path in the discovered layers
-        for lvo in sorted(self.orm_wrapper.layer_version_objects, reverse=True, key=_sort_longest_path):
-            if p.fullmatch(lvo.local_path):
+        # Our paths don't append a trailing slash
+        if pathRE.endswith("/"):
+            pathRE = pathRE[:-1]
+
+        p = re.compile(pathRE)
+        # Heuristics: we always match recipe to the deepest layer path in
+        # the discovered layers
+        for lvo in sorted(self.orm_wrapper.layer_version_objects,
+                          reverse=True, key=_sort_longest_path):
+            if p.fullmatch(os.path.abspath(lvo.local_path)):
                 return lvo
             if lvo.layer.local_source_dir:
-                if p.fullmatch(lvo.layer.local_source_dir):
+                if p.fullmatch(os.path.abspath(lvo.layer.local_source_dir)):
                     return lvo
-        #if we get here, we didn't read layers correctly; dump whatever information we have on the error log
-        logger.warning("Could not match layer dependency for path %s : %s", path, self.orm_wrapper.layer_version_objects)
-
 
+        # if we get here, we didn't read layers correctly;
+        # dump whatever information we have on the error log
+        logger.warning("Could not match layer dependency for path %s : %s",
+                       pathRE,
+                       self.orm_wrapper.layer_version_objects)
 
     def _get_layer_version_for_path(self, path):
         self._ensure_build()
-- 
2.7.4

[PATCH 18/18] toaster: buildinfohelper Clarify log message for build history

[Michael Wood]

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index 63c779e..5ed150d 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -381,8 +381,8 @@ class ORMWrapper(object):
                 local_path=layer_version_information['local_path'],
             )
 
-            logger.info("created new historical layer version %d",
-                        layer_copy.pk)
+            logger.debug("Created new layer version %s for build history",
+                         layer_copy.layer.name)
 
             self.layer_version_built.append(layer_copy)
 
-- 
2.7.4

Re: [PATCH 00/18] Patches for upstream 22.11.16

[Michael Wood]

Should have said - if there are no objections I will send this patch set 
upstream after Toaster meeting on 23rd Nov

Thanks,

Michael

On 22/11/16 17:55, Michael Wood wrote:
> Patches for upstream submission. This is a patchset for all the current patches waiting to be upstreamed.
>
> Tests completed:
> - https://travis-ci.org/toastertester/toaster-next/builds/178028675
> - Run command line build of binutils-native
> - Run local build of core-image-minimal
> - Run master project build of core-image-minimal
> - Build customised core-image-minimal
>
> Branch available at poky-contrib michaelw/toaster/patch_queue_22_11_16
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=michaelw/toaster/patch_queue_22_11_16
>
>
> Michael Wood (14):
>    toaster: runbuilds Write the pidfile in python rather than shell
>      script
>    toaster: tests Add management command tests
>    toaster: Add an example production settings file
>    toaster: layerindex updater Take into account layers being predefined
>    toaster: orm/fixtures Add the master release and correct morty release
>    toaster: settings fixture Set default release to master
>    toaster: customrecipejs Consume click event on 'a' link if disabled
>    toaster: buildinfohelper toaster-custom-images layer
>    toaster: orm models Project class Fix pyflake errors
>    toaster: orm models Handle CustomImageRecipe BRLayer here
>    toaster: bldcontrol Move CustomImageRecipe file creation into own
>      function
>    toaster: buildinfohelper Simplify layer event to toaster layer
>      function
>    toaster: buildinfohelper fix _get_layer_version_for_dependency
>    toaster: buildinfohelper Clarify log message for build history
>
> Reyna, David (2):
>    toaster: orm gen_layerdeps Protect against circular Layer dependencies
>    toaster: tablejs Fix missing close square bracket
>
> Sujith H (2):
>    toaster: localhostbecontroller accept custom init script for build
>    toaster: localhostbecontroller write toaster layers for project to
>      toaster-bblayers.conf
>
>   bitbake/bin/toaster                                |   2 +-
>   bitbake/lib/bb/ui/buildinfohelper.py               | 112 +++++++--------
>   .../toaster/bldcontrol/localhostbecontroller.py    | 160 +++++++++++----------
>   .../bldcontrol/management/commands/runbuilds.py    |  21 ++-
>   bitbake/lib/toaster/orm/fixtures/oe-core.xml       |  23 ++-
>   bitbake/lib/toaster/orm/fixtures/poky.xml          |  82 ++++++++++-
>   bitbake/lib/toaster/orm/fixtures/settings.xml      |   2 +-
>   .../toaster/orm/management/commands/lsupdates.py   |  61 +++-----
>   bitbake/lib/toaster/orm/models.py                  |  94 +++++++-----
>   bitbake/lib/toaster/tests/commands/__init__.py     |   0
>   .../lib/toaster/tests/commands/test_loaddata.py    |  61 ++++++++
>   .../lib/toaster/tests/commands/test_lsupdates.py   |  45 ++++++
>   .../lib/toaster/tests/commands/test_runbuilds.py   |  88 ++++++++++++
>   bitbake/lib/toaster/toastergui/api.py              |  16 ++-
>   .../toaster/toastergui/static/js/customrecipe.js   |   6 +
>   bitbake/lib/toaster/toastergui/static/js/table.js  |   2 +-
>   .../toastermain/settings_production_example.py     |  58 ++++++++
>   17 files changed, 597 insertions(+), 236 deletions(-)
>   create mode 100644 bitbake/lib/toaster/tests/commands/__init__.py
>   create mode 100644 bitbake/lib/toaster/tests/commands/test_loaddata.py
>   create mode 100644 bitbake/lib/toaster/tests/commands/test_lsupdates.py
>   create mode 100644 bitbake/lib/toaster/tests/commands/test_runbuilds.py
>   create mode 100644 bitbake/lib/toaster/toastermain/settings_production_example.py
>