[PATCH 0/5] remove startup questions v3

[PATCH 0/5] remove startup questions v3

[brian avery]

This version gets rid of some bashisms that made the startup script fail
if dash was the default shell.

-b

The following changes since commit 6d78e09708ecb268ca10809ef6e8c9ea709d3ec4:

  bitbake: toaster: display warnings for bad "IMAGE_FSTYPES" values (2015-10-06 16:46:52 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib bavery/toaster/startup_simplification_8217_v3
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=bavery/toaster/startup_simplification_8217_v3

brian avery (5):
  toaster: removed superuser question from startup
  toaster: remove layer and build dir interactive questions
  toaster: check for configuration file and exit if not found
  toaster: get rid of interactivity in bldcontrol
  toaster: remove bashisms so script works in dash as well

 bitbake/bin/toaster                                |  39 ++++--
 .../management/commands/checksettings.py           | 138 ++++++---------------
 2 files changed, 69 insertions(+), 108 deletions(-)

--
1.9.1

[PATCH 1/5] toaster: removed superuser question from startup

[brian avery]

added comment explaining how to create one later

Signed-off-by: brian avery <avery.brian@gmail.com>
---
 bitbake/bin/toaster | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster
index ac27826..d56d525 100755
--- a/bitbake/bin/toaster
+++ b/bitbake/bin/toaster
@@ -54,11 +54,10 @@ webserverStartAll()
     fi
 
     retval=0
-    if [ "$TOASTER_MANAGED" '=' '1' ]; then
-        python $BBBASEDIR/lib/toaster/manage.py syncdb || retval=1
-    else
-        python $BBBASEDIR/lib/toaster/manage.py syncdb --noinput || retval=1
-    fi
+    # you can always add a superuser later via
+    # python bitbake/lib/toaster/manage.py python manage.py createsuperuser --username=<ME>
+    python $BBBASEDIR/lib/toaster/manage.py syncdb --noinput || retval=1
+
     python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=2
     if [ $retval -eq 1 ]; then
         echo "Failed db sync, stopping system start" 1>&2
-- 
1.9.1

[PATCH 2/5] toaster: remove layer and build dir interactive questions

[brian avery]

added TOASTER_CONF env var for toaasterconf.json

added TOASTER_DIR env var for working dir

added bugfix so WEB_PORT env variable is honored

Signed-off-by: brian avery <avery.brian@gmail.com>
---
 bitbake/bin/toaster | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster
index d56d525..ff3123c 100755
--- a/bitbake/bin/toaster
+++ b/bitbake/bin/toaster
@@ -162,7 +162,30 @@ RUNNING=0
 NOTOASTERUI=0
 WEBSERVER=1
 TOASTER_BRBE=""
-WEB_PORT="8000"
+if [ "$WEB_PORT" == "" ]; then
+    WEB_PORT="8000"
+fi
+# this is the configuraton file we are using for toaster
+# note default is assuming yocto. Override this if you are
+# running in a pure OE environment and use the toasterconf.json
+# in meta/conf/toasterconf.json
+# note: for future there are a number of relative path assumptions
+# in the local layers that currently prevent using an arbitrary
+# toasterconf.json
+if [ "$TOASTER_CONF" == "" ]; then
+    BIN_DIR=$(dirname "${BASH_SOURCE[0]}")
+    export TOASTER_CONF="$BIN_DIR/../../meta-yocto/conf/toasterconf.json"
+fi
+# this defines the dir toaster will use for
+# 1) clones of layers (in _toaster_clones )
+# 2) the build dir (in build)
+# 3) the sqlite db if that is being used.
+# 4) pid's we need to clean up on exit/shutdown
+# note: for future. in order to make this an arbitrary directory, we need to
+# make sure that the toaster.sqlite file doesn't default to `pwd` like it currently does.
+export TOASTER_DIR=`pwd`
+
+
 NOBROWSER=0
 
 for param in $*; do
-- 
1.9.1

[PATCH 3/5] toaster: check for configuration file and exit if not found

[brian avery]

[YOCTO #8217]

Signed-off-by: brian avery <avery.brian@gmail.com>
---
 bitbake/bin/toaster | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster
index ff3123c..bfedfd3 100755
--- a/bitbake/bin/toaster
+++ b/bitbake/bin/toaster
@@ -174,7 +174,11 @@ fi
 # toasterconf.json
 if [ "$TOASTER_CONF" == "" ]; then
     BIN_DIR=$(dirname "${BASH_SOURCE[0]}")
-    export TOASTER_CONF="$BIN_DIR/../../meta-yocto/conf/toasterconf.json"
+    export TOASTER_CONF="$BIN_DIR/../../meta-yocto/conf/toasterconf.json"    
+fi
+if [ ! -f $TOASTER_CONF ]; then
+    echo "$TOASTER_CONF configuration file not found, exiting..."
+    exit -1
 fi
 # this defines the dir toaster will use for
 # 1) clones of layers (in _toaster_clones )
-- 
1.9.1

[PATCH 4/5] toaster: get rid of interactivity in bldcontrol

[brian avery]

removed layer dir questions and base off TOASTER_DIR

removed build dir questions and base off TOASTER_DIR

base configuration file off of TOASTER_CONF

fixed some pylint issues

[YOCTO #8217]

Signed-off-by: brian avery <avery.brian@gmail.com>
---
 bitbake/bin/toaster                                |   2 +-
 .../management/commands/checksettings.py           | 138 ++++++---------------
 2 files changed, 38 insertions(+), 102 deletions(-)

diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster
index bfedfd3..f1aee62 100755
--- a/bitbake/bin/toaster
+++ b/bitbake/bin/toaster
@@ -174,7 +174,7 @@ fi
 # toasterconf.json
 if [ "$TOASTER_CONF" == "" ]; then
     BIN_DIR=$(dirname "${BASH_SOURCE[0]}")
-    export TOASTER_CONF="$BIN_DIR/../../meta-yocto/conf/toasterconf.json"    
+    export TOASTER_CONF="$BIN_DIR/../../meta-yocto/conf/toasterconf.json"
 fi
 if [ ! -f $TOASTER_CONF ]; then
     echo "$TOASTER_CONF configuration file not found, exiting..."
diff --git a/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py b/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
index b2c573c..5e70437 100644
--- a/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
+++ b/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
@@ -4,7 +4,7 @@ from bldcontrol.bbcontroller import getBuildEnvironmentController, ShellCmdExcep
 from bldcontrol.models import BuildRequest, BuildEnvironment, BRError
 from orm.models import ToasterSetting, Build
 import os
-import sys, traceback
+import traceback
 
 def DN(path):
     if path is None:
@@ -21,7 +21,7 @@ class Command(NoArgsCommand):
         super(Command, self).__init__(*args, **kwargs)
         self.guesspath = DN(DN(DN(DN(DN(DN(DN(__file__)))))))
 
-    def _find_first_path_for_file(self, startdirectory, filename, level = 0):
+    def _find_first_path_for_file(self, startdirectory, filename, level=0):
         if level < 0:
             return None
         dirs = []
@@ -38,7 +38,7 @@ class Command(NoArgsCommand):
                 return ret
         return None
 
-    def _recursive_list_directories(self, startdirectory, level = 0):
+    def _recursive_list_directories(self, startdirectory, level=0):
         if level < 0:
             return []
         dirs = []
@@ -50,49 +50,23 @@ class Command(NoArgsCommand):
         except OSError:
             pass
         for j in dirs:
-                dirs = dirs + self._recursive_list_directories(j, level - 1)
+            dirs = dirs + self._recursive_list_directories(j, level - 1)
         return dirs
 
 
-    def _get_suggested_sourcedir(self, be):
-        if be.betype != BuildEnvironment.TYPE_LOCAL:
-            return ""
-        return DN(DN(DN(self._find_first_path_for_file(self.guesspath, "toasterconf.json", 4))))
-
-    def _get_suggested_builddir(self, be):
-        if be.betype != BuildEnvironment.TYPE_LOCAL:
-            return ""
-        return DN(self._find_first_path_for_file(DN(self.guesspath), "bblayers.conf", 4))
-
     def _verify_build_environment(self):
-        # refuse to start if we have no build environments
-        while BuildEnvironment.objects.count() == 0:
-            print(" !! No build environments found. Toaster needs at least one build environment in order to be able to run builds.\n" +
-                "You can manually define build environments in the database table bldcontrol_buildenvironment.\n" +
-                "Or Toaster can define a simple localhost-based build environment for you.")
-
-            i = raw_input(" --  Do you want to create a basic localhost build environment ? (Y/n) ");
-            if not len(i) or i.startswith("y") or i.startswith("Y"):
-                BuildEnvironment.objects.create(pk = 1, betype = 0)
-            else:
-                raise Exception("Toaster cannot start without build environments. Aborting.")
-
+        # provide a local build env. This will be extended later to include non local
+        if BuildEnvironment.objects.count() == 0:
+            BuildEnvironment.objects.create(betype=BuildEnvironment.TYPE_LOCAL)
 
         # we make sure we have builddir and sourcedir for all defined build envionments
         for be in BuildEnvironment.objects.all():
             be.needs_import = False
             def _verify_be():
                 is_changed = False
-                print("\nVerifying the build environment. If the local build environment is not properly configured, you will be asked to configure it.")
 
                 def _update_sourcedir():
-                    suggesteddir = self._get_suggested_sourcedir(be)
-                    if len(suggesteddir) > 0:
-                        be.sourcedir = raw_input("This is the directory Toaster uses to check out the source code of the layers you will build. Toaster will create new clones of the layers, so existing content in the chosen directory will not be changed.\nToaster suggests you use \"%s\" as your layers checkout directory. If you select this directory, a layer like \"meta-intel\" will end up in \"%s/meta-intel\".\nPress Enter to select \"%s\" or type the full path to a different directory. If you provide your own directory, it must be a parent of the cloned directory for the sources you are using to run Toaster: " % (suggesteddir, suggesteddir, suggesteddir))
-                    else:
-                        be.sourcedir = raw_input("Toaster needs to know in which directory it should check out the source code of the layers you will build. The directory should be a parent of the cloned directory for the sources you are using to run Toaster. Toaster will create new clones of the layers, so existing content in the chosen directory will not be changed.\nType the full path to the directory (for example: \"%s\": " % os.environ.get('HOME', '/tmp/'))
-                    if len(be.sourcedir) == 0 and len(suggesteddir) > 0:
-                        be.sourcedir = suggesteddir
+                    be.sourcedir = os.environ.get('TOASTER_DIR')
                     return True
 
                 if len(be.sourcedir) == 0:
@@ -103,23 +77,13 @@ class Command(NoArgsCommand):
                     print "\n -- Validation: The layers checkout directory must be set to an absolute path."
                     is_changed = _update_sourcedir()
 
-                if not be.sourcedir in DN(__file__):
-                    print "\n -- Validation: The layers checkout directory must be a parent of the current checkout."
-                    is_changed = _update_sourcedir()
-
                 if is_changed:
                     if be.betype == BuildEnvironment.TYPE_LOCAL:
                         be.needs_import = True
                     return True
 
                 def _update_builddir():
-                    suggesteddir = self._get_suggested_builddir(be)
-                    if len(suggesteddir) > 0:
-                        be.builddir = raw_input("Toaster needs to know where your build directory is located.\nThe build directory is where all the artifacts created by your builds will be stored. Toaster suggests \"%s\".\nPress Enter to select \"%s\" or type the full path to a different directory: " % (suggesteddir, suggesteddir))
-                    else:
-                        be.builddir = raw_input("Toaster needs to know where is your build directory.\nThe build directory is where all the artifacts created by your builds will be stored. Type the full path to the directory (for example: \" %s/build\")" % os.environ.get('HOME','/tmp/'))
-                    if len(be.builddir) == 0 and len(suggesteddir) > 0:
-                        be.builddir = suggesteddir
+                    be.builddir = os.environ.get('TOASTER_DIR')+"/build"
                     return True
 
                 if len(be.builddir) == 0:
@@ -138,79 +102,51 @@ class Command(NoArgsCommand):
 
 
                 if be.needs_import:
-                    print "\nToaster can use a SINGLE predefined configuration file to set up default project settings and layer information sources.\n"
-
-                    # find configuration files
-                    config_files = []
-                    for dirname in self._recursive_list_directories(be.sourcedir,2):
-                        if os.path.exists(os.path.join(dirname, ".templateconf")):
-                            import subprocess
-                            proc = subprocess.Popen('bash -c ". '+os.path.join(dirname, ".templateconf")+'; echo \"\$TEMPLATECONF\""', shell=True, stdout=subprocess.PIPE)
-                            conffilepath, stderroroutput = proc.communicate()
-                            proc.wait()
-                            if proc.returncode != 0:
-                                raise Exception("Failed to source TEMPLATECONF: %s" % stderroroutput)
-
-                            conffilepath = os.path.join(conffilepath.strip(), "toasterconf.json")
-                            candidatefilepath = os.path.join(dirname, conffilepath)
-                            if "toaster_cloned" in candidatefilepath:
-                                continue
-                            if os.path.exists(candidatefilepath):
-                                config_files.append(candidatefilepath)
-
-                    if len(config_files) > 0:
-                        print "Toaster will list now the configuration files that it found. Select the number to use the desired configuration file."
-                        for cf in config_files:
-                            print "  [%d] - %s" % (config_files.index(cf) + 1, cf)
-                        print "\n  [0] - Exit without importing any file"
-                        try:
-                                i = raw_input("\nEnter your option: ")
-                                if len(i) and (int(i) - 1 >= 0 and int(i) - 1 < len(config_files)):
-                                    print "\nImporting file: %s" % config_files[int(i)-1]
-                                    from loadconf import Command as LoadConfigCommand
-
-                                    LoadConfigCommand()._import_layer_config(config_files[int(i)-1])
-                                    # we run lsupdates after config update
-                                    print "\nLayer configuration imported. Updating information from the layer sources, please wait.\nYou can re-update any time later by running bitbake/lib/toaster/manage.py lsupdates"
-                                    from django.core.management import call_command
-                                    call_command("lsupdates")
-
-                                    # we don't look for any other config files
-                                    return is_changed
-                        except Exception as e:
-                            print "Failure while trying to import the toaster config file: %s" % e
-                            traceback.print_exc(e)
-                    else:
-                        print "\nToaster could not find a configuration file. You need to configure Toaster manually using the web interface, or create a configuration file and use\n  bitbake/lib/toaster/managepy.py loadconf [filename]\n command to load it. You can use https://wiki.yoctoproject.org/wiki/File:Toasterconf.json.txt.patch as a starting point."
-
-
-
+                    try:
+                        config_file = os.environ.get('TOASTER_CONF')
+                        print "\nImporting file: %s" % config_file
+                        from loadconf import Command as LoadConfigCommand
+
+                        LoadConfigCommand()._import_layer_config(config_file)
+                        # we run lsupdates after config update
+                        print "\nLayer configuration imported. Updating information from the layer sources, please wait.\nYou can re-update any time later by running bitbake/lib/toaster/manage.py lsupdates"
+                        from django.core.management import call_command
+                        call_command("lsupdates")
+
+                        # we don't look for any other config files
+                        return is_changed
+                    except Exception as e:
+                        print "Failure while trying to import the toaster config file %s: %s" %\
+                            (config_file, e)
+                        traceback.print_exc(e)
 
                 return is_changed
 
-            while (_verify_be()):
+            while _verify_be():
                 pass
         return 0
 
     def _verify_default_settings(self):
         # verify that default settings are there
-        if ToasterSetting.objects.filter(name = 'DEFAULT_RELEASE').count() != 1:
-            ToasterSetting.objects.filter(name = 'DEFAULT_RELEASE').delete()
-            ToasterSetting.objects.get_or_create(name = 'DEFAULT_RELEASE', value = '')
+        if ToasterSetting.objects.filter(name='DEFAULT_RELEASE').count() != 1:
+            ToasterSetting.objects.filter(name='DEFAULT_RELEASE').delete()
+            ToasterSetting.objects.get_or_create(name='DEFAULT_RELEASE', value='')
         return 0
 
     def _verify_builds_in_progress(self):
         # we are just starting up. we must not have any builds in progress, or build environments taken
-        for b in BuildRequest.objects.filter(state = BuildRequest.REQ_INPROGRESS):
-            BRError.objects.create(req = b, errtype = "toaster", errmsg = "Toaster found this build IN PROGRESS while Toaster started up. This is an inconsistent state, and the build was marked as failed")
+        for b in BuildRequest.objects.filter(state=BuildRequest.REQ_INPROGRESS):
+            BRError.objects.create(req=b, errtype="toaster",
+                                   errmsg=
+                                   "Toaster found this build IN PROGRESS while Toaster started up. This is an inconsistent state, and the build was marked as failed")
 
-        BuildRequest.objects.filter(state = BuildRequest.REQ_INPROGRESS).update(state = BuildRequest.REQ_FAILED)
+        BuildRequest.objects.filter(state=BuildRequest.REQ_INPROGRESS).update(state=BuildRequest.REQ_FAILED)
 
-        BuildEnvironment.objects.update(lock = BuildEnvironment.LOCK_FREE)
+        BuildEnvironment.objects.update(lock=BuildEnvironment.LOCK_FREE)
 
         # also mark "In Progress builds as failures"
         from django.utils import timezone
-        Build.objects.filter(outcome = Build.IN_PROGRESS).update(outcome = Build.FAILED, completed_on = timezone.now())
+        Build.objects.filter(outcome=Build.IN_PROGRESS).update(outcome=Build.FAILED, completed_on=timezone.now())
 
         return 0
 
-- 
1.9.1

[PATCH 5/5] toaster: remove bashisms so script works in dash as well

[brian avery]

[YOCTO #8217]

Signed-off-by: brian avery <avery.brian@gmail.com>
---
 bitbake/bin/toaster | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster
index f1aee62..bc439e6 100755
--- a/bitbake/bin/toaster
+++ b/bitbake/bin/toaster
@@ -162,7 +162,7 @@ RUNNING=0
 NOTOASTERUI=0
 WEBSERVER=1
 TOASTER_BRBE=""
-if [ "$WEB_PORT" == "" ]; then
+if [ "$WEB_PORT" = "" ]; then
     WEB_PORT="8000"
 fi
 # this is the configuraton file we are using for toaster
@@ -172,13 +172,13 @@ fi
 # note: for future there are a number of relative path assumptions
 # in the local layers that currently prevent using an arbitrary
 # toasterconf.json
-if [ "$TOASTER_CONF" == "" ]; then
-    BIN_DIR=$(dirname "${BASH_SOURCE[0]}")
+if [ "$TOASTER_CONF" = "" ]; then
+    BIN_DIR=$(dirname -- "$0")
     export TOASTER_CONF="$BIN_DIR/../../meta-yocto/conf/toasterconf.json"
 fi
 if [ ! -f $TOASTER_CONF ]; then
     echo "$TOASTER_CONF configuration file not found, exiting..."
-    exit -1
+    exit 1;
 fi
 # this defines the dir toaster will use for
 # 1) clones of layers (in _toaster_clones )
@@ -375,4 +375,3 @@ case $CMD in
         echo "Successful ${CMD}."
     ;;
 esac
-
-- 
1.9.1

Re: [PATCH 0/5] remove startup questions v3

[Barros Pena, Belen]

On 07/10/2015 01:38, "toaster-bounces@yoctoproject.org on behalf of brian
avery" <toaster-bounces@yoctoproject.org on behalf of
avery.brian@gmail.com> wrote:

>This version gets rid of some bashisms that made the startup script fail
>if dash was the default shell.
>
>-b
>
>The following changes since commit
>6d78e09708ecb268ca10809ef6e8c9ea709d3ec4:
>
>  bitbake: toaster: display warnings for bad "IMAGE_FSTYPES" values
>(2015-10-06 16:46:52 +0100)
>
>are available in the git repository at:
>
>  git://git.yoctoproject.org/poky-contrib
>bavery/toaster/startup_simplification_8217_v3
>  
>http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=bavery/toaster/st
>artup_simplification_8217_v3

This works for me now. Thanks!

Belén

>
>brian avery (5):
>  toaster: removed superuser question from startup
>  toaster: remove layer and build dir interactive questions
>  toaster: check for configuration file and exit if not found
>  toaster: get rid of interactivity in bldcontrol
>  toaster: remove bashisms so script works in dash as well
>
> bitbake/bin/toaster                                |  39 ++++--
> .../management/commands/checksettings.py           | 138
>++++++---------------
> 2 files changed, 69 insertions(+), 108 deletions(-)
>
>--
>1.9.1
>-- 
>_______________________________________________
>toaster mailing list
>toaster@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/toaster

Re: [PATCH 0/5] remove startup questions v3

[Ed Bartosh]

Submitted upstream.

On Tue, Oct 06, 2015 at 05:38:20PM -0700, brian avery wrote:
> This version gets rid of some bashisms that made the startup script fail
> if dash was the default shell.
> 
> -b
> 
> The following changes since commit 6d78e09708ecb268ca10809ef6e8c9ea709d3ec4:
> 
>   bitbake: toaster: display warnings for bad "IMAGE_FSTYPES" values (2015-10-06 16:46:52 +0100)
> 
> are available in the git repository at:
> 
>   git://git.yoctoproject.org/poky-contrib bavery/toaster/startup_simplification_8217_v3
>   http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=bavery/toaster/startup_simplification_8217_v3
> 
> brian avery (5):
>   toaster: removed superuser question from startup
>   toaster: remove layer and build dir interactive questions
>   toaster: check for configuration file and exit if not found
>   toaster: get rid of interactivity in bldcontrol
>   toaster: remove bashisms so script works in dash as well
> 
>  bitbake/bin/toaster                                |  39 ++++--
>  .../management/commands/checksettings.py           | 138 ++++++---------------
>  2 files changed, 69 insertions(+), 108 deletions(-)
> 
> --
> 1.9.1
> -- 
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster

-- 
--
Regards,
Ed

[PATCH 1/5] toaster: removed superuser question from startup

[Ed Bartosh]

From: brian avery <avery.brian@gmail.com>

added comment explaining how to create one later

Signed-off-by: brian avery <avery.brian@gmail.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bin/toaster | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/bin/toaster b/bin/toaster
index ac27826..d56d525 100755
--- a/bin/toaster
+++ b/bin/toaster
@@ -54,11 +54,10 @@ webserverStartAll()
     fi
 
     retval=0
-    if [ "$TOASTER_MANAGED" '=' '1' ]; then
-        python $BBBASEDIR/lib/toaster/manage.py syncdb || retval=1
-    else
-        python $BBBASEDIR/lib/toaster/manage.py syncdb --noinput || retval=1
-    fi
+    # you can always add a superuser later via
+    # python bitbake/lib/toaster/manage.py python manage.py createsuperuser --username=<ME>
+    python $BBBASEDIR/lib/toaster/manage.py syncdb --noinput || retval=1
+
     python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=2
     if [ $retval -eq 1 ]; then
         echo "Failed db sync, stopping system start" 1>&2
-- 
2.1.4