[sumo][PATCH 1/8] toaster: allow pokydirname to be evaluated when all layers are local

[sumo][PATCH 1/8] toaster: allow pokydirname to be evaluated when all layers are local

[Awais Belal]

Toaster depends on pokydirname for identifying the location of
the oe-init-build-env script and there might be other purposes
in the future. The problem with current approach is that it
only checks/sets the variable with git based repos whereas
toaster provides mechanism to allow having layers that are all
locally available the evaluation of the variable fails in such
scenarios so we use a more flexible mechanism in this case and
try to locate poky in the local layers as well if not already
set.

Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
---
 .../lib/toaster/bldcontrol/localhostbecontroller.py    | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 16c7c80..f960a38 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -217,9 +217,21 @@ class LocalhostBEController(BuildEnvironmentController):
         self.setCloneStatus(bitbake,'complete',clone_total,clone_count)
         logger.debug("localhostbecontroller: current layer list %s " % pformat(layerlist))
 
-        if self.pokydirname is None and os.path.exists(os.path.join(self.be.sourcedir, "oe-init-build-env")):
-            logger.debug("localhostbecontroller: selected poky dir name %s" % self.be.sourcedir)
-            self.pokydirname = self.be.sourcedir
+        # Resolve self.pokydirname if not resolved yet, consider the scenario
+        # where all layers are local, that's the else clause
+        if self.pokydirname is None:
+            if os.path.exists(os.path.join(self.be.sourcedir, "oe-init-build-env")):
+                logger.debug("localhostbecontroller: selected poky dir name %s" % self.be.sourcedir)
+                self.pokydirname = self.be.sourcedir
+            else:
+                # Alternatively, scan local layers for relative "oe-init-build-env" location
+                for layer in layers:
+                    if os.path.exists(os.path.join(layer.layer_version.layer.local_source_dir,"..","oe-init-build-env")):
+                        logger.debug("localhostbecontroller, setting pokydirname to %s" % (layer.layer_version.layer.local_source_dir))
+                        self.pokydirname = os.path.join(layer.layer_version.layer.local_source_dir,"..")
+                        break
+                else:
+                    logger.error("pokydirname is not set, you will run into trouble!")
 
         # 5. create custom layer and add custom recipes to it
         for target in targets:
-- 
2.7.4

[sumo][PATCH 2/8] toaster: use a more flexible way to find bitbake

[Awais Belal]

The current mechanism for finding the bitbake binary
assumes a directory structure which is identical to
poky where oe-core's meta and bitbake directories are
at the same level. There can be a case where bitbake
is used from elsewhere and in such cases the above
mentioned assumption fails to hold whereas this is
totally allowed by the oe-init-build-env script which
can take bitbakedir as an argument.
So a better approach would be to use bitbake from
PATH while keeping the older mechanism in place so
it can be removed after tests are done in various
environments. This makes more sense as toaster has
also been launched from the same bitbake instance
that is the one in PATH.

Signed-off-by: Awais Belal <awais_belal@mentor.com>
---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index f960a38..69a75ff 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -351,8 +351,19 @@ class LocalhostBEController(BuildEnvironmentController):
         # clean the Toaster to build environment
         env_clean = 'unset BBPATH;' # clean BBPATH for <= YP-2.4.0
 
-        # run bitbake server from the clone
+        # run bitbake server from the clone if available
+        # otherwise pick it from the PATH
         bitbake = os.path.join(self.pokydirname, 'bitbake', 'bin', 'bitbake')
+        if not os.path.exists(bitbake):
+            logger.info("Bitbake not available under %s, will try to use it from PATH" %
+                        self.pokydirname)
+            for path in os.environ["PATH"].split(os.pathsep):
+                if os.path.exists(os.path.join(path, 'bitbake')):
+                    bitbake = os.path.join(path, 'bitbake')
+                    break
+            else:
+                logger.error("Looks like Bitbake is not available, please fix your environment")
+
         toasterlayers = os.path.join(builddir,"conf/toaster-bblayers.conf")
         self._shellcmd('%s bash -c \"source %s %s; BITBAKE_UI="knotty" %s --read %s --read %s '
                        '--server-only -B 0.0.0.0:0\"' % (env_clean, oe_init,
-- 
2.7.4

[sumo][PATCH 3/8] toaster: allow TOASTER_DIR to be overridden from cmdline

[Awais Belal]

TOASTER_DIR is used for higher level toaster artifacts
such the SQL DB and creating toaster internal build
directories for projects. Prior to this change it was
evaluated as `dirname $BUILDDIR` and user had no control
over it. This change allows to override this variable
from the command line for more flexibility. The variable
defaults to its original setting if the optional argument
is not passed.

Signed-off-by: Awais Belal <awais_belal@mentor.com>
---
 bitbake/bin/toaster | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster
index ed365ee..2a484e5 100755
--- a/bitbake/bin/toaster
+++ b/bitbake/bin/toaster
@@ -18,11 +18,12 @@
 # along with this program. If not, see http://www.gnu.org/licenses/.
 
 HELP="
-Usage: source toaster start|stop [webport=<address:port>] [noweb] [nobuild]
+Usage: source toaster start|stop [webport=<address:port>] [noweb] [nobuild] [toasterdir=<absolute_path>]
     Optional arguments:
         [nobuild] Setup the environment for capturing builds with toaster but disable managed builds
         [noweb] Setup the environment for capturing builds with toaster but don't start the web server
         [webport] Set the development server (default: localhost:8000)
+        [toasterdir] Set the path to be used as TOASTER_DIR, should be absolute (default: BUILDDIR/../)
 "
 
 custom_extention()
@@ -186,6 +187,7 @@ unset OE_ROOT
 WEBSERVER=1
 export TOASTER_BUILDSERVER=1
 ADDR_PORT="localhost:8000"
+TOASTERDIR=`dirname $BUILDDIR`
 unset CMD
 for param in $*; do
     case $param in
@@ -211,6 +213,9 @@ for param in $*; do
                 ADDR_PORT="localhost:$PORT"
             fi
     ;;
+    toasterdir=*)
+            TOASTERDIR="${param#*=}"
+    ;;
     --help)
             echo "$HELP"
             return 0
@@ -241,7 +246,7 @@ fi
 # 2) the build dir (in build)
 # 3) the sqlite db if that is being used.
 # 4) pid's we need to clean up on exit/shutdown
-export TOASTER_DIR=`dirname $BUILDDIR`
+export TOASTER_DIR=$TOASTERDIR
 export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE TOASTER_DIR"
 
 # Determine the action. If specified by arguments, fine, if not, toggle it
-- 
2.7.4

[sumo][PATCH 4/8] toaster/widgets.py: avoid divide by zero issues

[Awais Belal]

There can be cases where the variables being used
to divide in these expressions can be zero. e.g.
a setup consisting of only local repos will have
repos_to_clone=0 and will generate a divide by
zero scenario.
Fix this by checking the divisor in such cases.

Signed-off-by: Awais Belal <awais_belal@mentor.com>
---
 bitbake/lib/toaster/toastergui/widgets.py | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/bitbake/lib/toaster/toastergui/widgets.py b/bitbake/lib/toaster/toastergui/widgets.py
index a1792d9..b2a4599 100644
--- a/bitbake/lib/toaster/toastergui/widgets.py
+++ b/bitbake/lib/toaster/toastergui/widgets.py
@@ -511,13 +511,15 @@ class MostRecentBuildsView(View):
                 buildrequest_id = build_obj.buildrequest.pk
             build['buildrequest_id'] = buildrequest_id
 
-            build['recipes_parsed_percentage'] = \
-                int((build_obj.recipes_parsed /
-                     build_obj.recipes_to_parse) * 100)
-
-            build['repos_cloned_percentage'] = \
-                int((build_obj.repos_cloned /
-                     build_obj.repos_to_clone) * 100)
+            if build_obj.recipes_to_parse > 0:
+                build['recipes_parsed_percentage'] = \
+                    int((build_obj.recipes_parsed /
+                         build_obj.recipes_to_parse) * 100)
+
+            if build_obj.repos_to_clone > 0:
+                build['repos_cloned_percentage'] = \
+                    int((build_obj.repos_cloned /
+                         build_obj.repos_to_clone) * 100)
 
             tasks_complete_percentage = 0
             if build_obj.outcome in (Build.SUCCEEDED, Build.FAILED):
-- 
2.7.4

[sumo][PATCH 5/8] toastergui/newproject.html: fix release divs

[Awais Belal]

The release drop down divs were not being closed
appropriately which showed adverse reactions on
the UI that aligned the "Create project" button
with the left edge of the screen without any
margins. This fixes these divs which inturn
aligns the button appropriately.

Signed-off-by: Awais Belal <awais_belal@mentor.com>
---
 bitbake/lib/toaster/toastergui/templates/newproject.html | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/toaster/toastergui/templates/newproject.html b/bitbake/lib/toaster/toastergui/templates/newproject.html
index acb614e..bd03bb5 100644
--- a/bitbake/lib/toaster/toastergui/templates/newproject.html
+++ b/bitbake/lib/toaster/toastergui/templates/newproject.html
@@ -54,12 +54,12 @@
                     <span class="help-block">{{release.helptext|safe}}</span>
                   </div>
                 {% endfor %}
+                </div>
+              </div>
             {% else %}
               <input type="hidden" name="projectversion" value="{{releases.0.id}}"/>
             {% endif %}
                 </div>
-              </div>
-            </fieldset>
         {% endif %}
             <div class="top-air">
               <input type="submit" id="create-project-button" class="btn btn-primary btn-lg" value="Create project"/>
-- 
2.7.4

[sumo][PATCH 6/8] toaster/checksettings: allow CUSTOM_XML_ONLY setting through env

[Awais Belal]

This change allows the CUSTOM_XML_ONLY toaster setting to be
provided through the environment so the user can do this without
mingling with the settings.xml in scenarios where modifying
settings.xml is not achieveable.

Signed-off-by: Awais Belal <awais_belal@mentor.com>
---
 bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py b/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
index 823c6f1..14298d9 100644
--- a/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
+++ b/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
@@ -74,8 +74,9 @@ class Command(BaseCommand):
                         print("Loading default settings")
                         call_command("loaddata", "settings")
                         template_conf = os.environ.get("TEMPLATECONF", "")
+                        custom_xml_only = os.environ.get("CUSTOM_XML_ONLY")
 
-                        if ToasterSetting.objects.filter(name='CUSTOM_XML_ONLY').count() > 0:
+                        if ToasterSetting.objects.filter(name='CUSTOM_XML_ONLY').count() > 0 or (not custom_xml_only == None):
                             # only use the custom settings
                             pass
                         elif "poky" in template_conf:
-- 
2.7.4

[sumo][PATCH 7/8] toaster/models.py: allow local paths for custom recipe's base

[Awais Belal]

In a case where the layer source is local only and the recipe
is not yet built we can search for the path with layer's
local_source_dir and if available that should be used rather
than just skipping the scenario.

Signed-off-by: Awais Belal <awais_belal@mentor.com>
---
 bitbake/lib/toaster/orm/models.py | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/bitbake/lib/toaster/orm/models.py b/bitbake/lib/toaster/orm/models.py
index 3a7dff8..4b77e8f 100644
--- a/bitbake/lib/toaster/orm/models.py
+++ b/bitbake/lib/toaster/orm/models.py
@@ -1663,6 +1663,9 @@ class CustomImageRecipe(Recipe):
 
         path_schema_two = self.base_recipe.file_path
 
+        path_schema_three = "%s/%s" % (self.base_recipe.layer_version.layer.local_source_dir,
+                                     self.base_recipe.file_path)
+
         if os.path.exists(path_schema_one):
             return path_schema_one
 
@@ -1670,6 +1673,10 @@ class CustomImageRecipe(Recipe):
         if os.path.exists(path_schema_two):
             return path_schema_two
 
+        # Or a local path if all layers are local
+        if os.path.exists(path_schema_three):
+            return path_schema_three
+
         return None
 
     def generate_recipe_file_contents(self):
-- 
2.7.4

[sumo][PATCH 8/8] toaster/layerdetails.js: don't hide local layer info

[Awais Belal]

The local layer info (provided through custom fixtures) should
not be hidden and it is better to handle it in the same manner
as an imported layer otherwise the layer path and dependency
info is not shown. The layer editing fields are handled in the
html side of things appropriately so this does not harm that
implementation.

Signed-off-by: Awais Belal <awais_belal@mentor.com>
---
 bitbake/lib/toaster/toastergui/static/js/layerdetails.js | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/toaster/toastergui/static/js/layerdetails.js b/bitbake/lib/toaster/toastergui/static/js/layerdetails.js
index 9ead393..933b65b 100644
--- a/bitbake/lib/toaster/toastergui/static/js/layerdetails.js
+++ b/bitbake/lib/toaster/toastergui/static/js/layerdetails.js
@@ -359,7 +359,8 @@ function layerDetailsPageInit (ctx) {
     if ($(this).is("dt")) {
       var dd = $(this).next("dd");
       if (!dd.children("form:visible")|| !dd.find(".current-value").html()){
-        if (ctx.layerVersion.layer_source == ctx.layerSourceTypes.TYPE_IMPORTED){
+        if (ctx.layerVersion.layer_source == ctx.layerSourceTypes.TYPE_IMPORTED ||
+            ctx.layerVersion.layer_source == ctx.layerSourceTypes.TYPE_LOCAL) {
         /* There's no current value and the layer is editable
          * so show the "Not set" and hide the delete icon
          */
-- 
2.7.4