All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v5 00/19] per project build directory
@ 2016-03-16 12:05 Ed Bartosh
  2016-03-16 12:05 ` [PATCH v5 01/19] toaster: don't start bitbake server Ed Bartosh
                   ` (19 more replies)
  0 siblings, 20 replies; 23+ messages in thread
From: Ed Bartosh @ 2016-03-16 12:05 UTC (permalink / raw)
  To: toaster

Hi,

Default build directory is not going to be used for project builds anymore.
It can be used only for command line builds. Toaster will use build-toaster-<project id>
directories for project builds.

This is how to test this:
 - run toaster
 - run command line build: build zlib
 - check if it's shown in Toaster UI
 - create project for Yocto master
 - trigger project build from UI
 - wait until it succeeds
 - create project for Jethro
 - trigger project build from UI
 - wait until it succeds
 - check that 2 build directories build-toaster-<project id> exist and contain
   correct conf/toaster.conf and conf/bblayers.conf

Changes in v2: stop bitbake server after the build
Changes in v3: fixed '/bin/sh: 1: source: not found' dash issue
               used project id in build directory name
               added 'INHERIT+='toaster buildhistory' to conf/local.conf on toaster start
Changes in v4: fixed builds on Ubuntu caused by dash being default shell
Changes in v5: fixed nasty bug in toasterui causing it to stuck in event loop

The following changes since commit 01ab10ddaf21e34d8b3578975d1494cf0cfa4ef7:

  toaster.bbclass: show packages that were setscened into existence too (2016-03-10 15:58:34 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/toaster/project-build-dir-7880
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/project-build-dir-7880

Ed Bartosh (18):
  toaster: don't start bitbake server
  toaster: get rid of noui option
  toaster: set BITBAKE_UI environment variable
  toasterui: add brbe parameter to buildinfohelper
  uievent: improve BBUIEventQueue code
  buildinfohelper: improve handling of providermap
  toasterui: fix brbe reporting
  toaster: remove startBBServer API
  toaster: remove release API
  toaster: add brbe parameter to triggerBuild
  toaster: modified setLayers API
  toaster: reimplement triggerBuild
  toaster: add new parameter to _shellcmd
  toaster: stop bitbake server after the build
  toaster: update conf/local.conf
  toaster: fix jethro build
  toaster: use bash explicitly
  toasterui: shutdown on BuildCompleted event

Elliot Smith (1):
  toasterui: detect build run start correctly on Jethro

 bitbake/bin/toaster                                |  79 ++---------
 bitbake/lib/bb/ui/buildinfohelper.py               |  48 +++----
 bitbake/lib/bb/ui/toasterui.py                     |  48 ++++---
 bitbake/lib/bb/ui/uievent.py                       |  10 +-
 bitbake/lib/toaster/bldcontrol/bbcontroller.py     |  33 -----
 .../toaster/bldcontrol/localhostbecontroller.py    | 146 ++++++++++-----------
 .../bldcontrol/management/commands/runbuilds.py    |   6 +-
 7 files changed, 147 insertions(+), 223 deletions(-)

--
Regards,
Ed


^ permalink raw reply	[flat|nested] 23+ messages in thread
* [PATCH v5 01/19] toaster: don't start bitbake server
  2016-03-16 12:05 [PATCH v5 00/19] per project build directory Ed Bartosh
@ 2016-03-16 12:05 ` Ed Bartosh
  2016-03-16 12:05 ` [PATCH v5 02/19] toaster: get rid of noui option Ed Bartosh
                   ` (18 subsequent siblings)
  19 siblings, 0 replies; 23+ messages in thread
From: Ed Bartosh @ 2016-03-16 12:05 UTC (permalink / raw)
  To: toaster

From now on toaster script will not run bitbake server.
It will be started by runbuilds and stopped after the build.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/bin/toaster | 67 +++++------------------------------------------------
 1 file changed, 6 insertions(+), 61 deletions(-)

diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster
index d409554..6a85ae2 100755
--- a/bitbake/bin/toaster
+++ b/bitbake/bin/toaster
@@ -17,7 +17,7 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see http://www.gnu.org/licenses/.
 
-# Usage: source toaster [start|stop|restart-bitbake]
+# Usage: source toaster [start|stop]
 #                       [webport=<port>] [noui] [noweb]
 
 # Helper function to kill a background toaster development server
@@ -95,7 +95,6 @@ stop_system()
         kill `cat ${BUILDDIR}/.toasterui.pid` 2>/dev/null
         rm ${BUILDDIR}/.toasterui.pid
     fi
-    stop_bitbake
     webserverKillAll
     # unset exported variables
     unset DATABASE_URL
@@ -106,31 +105,6 @@ stop_system()
     INSTOPSYSTEM=0
 }
 
-start_bitbake() {
-    unset BBSERVER
-    bitbake --read conf/toaster.conf --server-only -t xmlrpc -B 0.0.0.0:0
-    if [ $? -ne 0 ]; then
-        echo "Bitbake server start failed"
-        return 1
-    fi
-    export BBSERVER=0.0.0.0:-1
-    export DATABASE_URL=`$MANAGE get-dburl`
-    # we start the TOASTERUI only if not inhibited
-    if [ $NOTOASTERUI -eq 0 ]; then
-        bitbake --observe-only -u toasterui --remote-server=$BBSERVER -t xmlrpc \
-            >>${BUILDDIR}/toaster_ui.log 2>&1 \
-            & echo $! >${BUILDDIR}/.toasterui.pid
-    fi
-    return 0
-}
-
-stop_bitbake() {
-    BBSERVER=0.0.0.0:-1 bitbake -m
-    unset BBSERVER
-    # force stop any misbehaving bitbake server
-    lsof -t bitbake.lock | awk '{print $2}' | grep "[0-9]\+" | xargs -n1 -r kill
-}
-
 verify_prereq() {
     # Verify Django version
     reqfile=$(python -c "import os; print os.path.realpath('$BBBASEDIR/toaster-requirements.txt')")
@@ -229,15 +203,6 @@ if [ `basename \"$0\"` = `basename \"${TOASTER}\"` ]; then
     return 1
 fi
 
-if [ "$1" = 'restart-bitbake' ] ; then
-    stop_bitbake
-    sleep 1
-    start_bitbake
-    rc=$?
-    sleep 3
-    return $rc
-fi
-
 verify_prereq || return 1
 
 # We make sure we're running in the current shell and in a good environment
@@ -281,16 +246,6 @@ case $CMD in
              $MANAGE checksocket "0.0.0.0:$WEB_PORT" || return 1
         fi
 
-        # Make sure it's safe to start by checking bitbake lock
-        if [ -e $BUILDDIR/bitbake.lock ]; then
-            python -c "import fcntl; fcntl.flock(open(\"$BUILDDIR/bitbake.lock\"), fcntl.LOCK_EX|fcntl.LOCK_NB)" 2>/dev/null
-            if [ $? -ne 0 ] ; then
-                echo "Error: bitbake lock state error. File locks show that the system is on." 1>&2
-                echo "Please wait for the current build to finish, stop and then start the system again." 1>&2
-                return 3
-            fi
-        fi
-
         # kill Toaster web server if it's alive
         if [ -e $BUILDDIR/.toastermain.pid ] && kill -0 `cat $BUILDDIR/.toastermain.pid`; then
             echo "Warning: bitbake appears to be dead, but the Toaster web server is running." 1>&2
@@ -308,22 +263,12 @@ case $CMD in
             echo "Failed ${CMD}."
             return 4
         fi
-        start_bitbake
-        if [ $? -eq 0 ]; then
-            $MANAGE runbuilds & echo $! >${BUILDDIR}/.runbuilds.pid
-            # set fail safe stop system on terminal exit
-            trap stop_system SIGHUP
-            echo "Successful ${CMD}."
-            return 0
-        else
-            # failed start, do stop
-            stop_system
-            echo "Failed ${CMD}."
-            return 1
-        fi
-        # stop system on terminal exit
-        set -o monitor
+        export DATABASE_URL=`$MANAGE get-dburl`
+        $MANAGE runbuilds & echo $! >${BUILDDIR}/.runbuilds.pid
+        # set fail safe stop system on terminal exit
         trap stop_system SIGHUP
+        echo "Successful ${CMD}."
+        return 0
     ;;
     stop )
         stop_system
-- 
2.1.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v5 02/19] toaster: get rid of noui option
  2016-03-16 12:05 [PATCH v5 00/19] per project build directory Ed Bartosh
  2016-03-16 12:05 ` [PATCH v5 01/19] toaster: don't start bitbake server Ed Bartosh
@ 2016-03-16 12:05 ` Ed Bartosh
  2016-03-16 12:05 ` [PATCH v5 03/19] toaster: set BITBAKE_UI environment variable Ed Bartosh
                   ` (17 subsequent siblings)
  19 siblings, 0 replies; 23+ messages in thread
From: Ed Bartosh @ 2016-03-16 12:05 UTC (permalink / raw)
  To: toaster

noui command line option doesn't makes sense anymore as toaster doesn't
run bitbake. It should be safe to to remove it.

The purpose of this option was to skip running bitbake observer process.
This was never used before as it's not possible to run toaster build
without running observer.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/bin/toaster | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster
index 6a85ae2..1e6e690 100755
--- a/bitbake/bin/toaster
+++ b/bitbake/bin/toaster
@@ -175,15 +175,11 @@ fi
 # make sure that the toaster.sqlite file doesn't default to `pwd` like it currently does.
 export TOASTER_DIR=`pwd`
 
-NOTOASTERUI=0
 WEBSERVER=1
 WEB_PORT="8000"
 unset CMD
 for param in $*; do
     case $param in
-    noui )
-            NOTOASTERUI=1
-    ;;
     noweb )
             WEBSERVER=0
     ;;
-- 
2.1.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v5 03/19] toaster: set BITBAKE_UI environment variable
  2016-03-16 12:05 [PATCH v5 00/19] per project build directory Ed Bartosh
  2016-03-16 12:05 ` [PATCH v5 01/19] toaster: don't start bitbake server Ed Bartosh
  2016-03-16 12:05 ` [PATCH v5 02/19] toaster: get rid of noui option Ed Bartosh
@ 2016-03-16 12:05 ` Ed Bartosh
  2016-03-16 12:05 ` [PATCH v5 04/19] toasterui: add brbe parameter to buildinfohelper Ed Bartosh
                   ` (16 subsequent siblings)
  19 siblings, 0 replies; 23+ messages in thread
From: Ed Bartosh @ 2016-03-16 12:05 UTC (permalink / raw)
  To: toaster

Set BITBAKE_UI variable to 'toastergui' for command line builds
to use toasterui as a default ui module for bitbake.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/bin/toaster | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster
index 1e6e690..5cc77ed 100755
--- a/bitbake/bin/toaster
+++ b/bitbake/bin/toaster
@@ -100,6 +100,7 @@ stop_system()
     unset DATABASE_URL
     unset TOASTER_CONF
     unset TOASTER_DIR
+    unset BITBAKE_UI
     trap - SIGHUP
     #trap - SIGCHLD
     INSTOPSYSTEM=0
@@ -259,6 +260,7 @@ case $CMD in
             echo "Failed ${CMD}."
             return 4
         fi
+        export BITBAKE_UI='toasterui'
         export DATABASE_URL=`$MANAGE get-dburl`
         $MANAGE runbuilds & echo $! >${BUILDDIR}/.runbuilds.pid
         # set fail safe stop system on terminal exit
-- 
2.1.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v5 04/19] toasterui: add brbe parameter to buildinfohelper
  2016-03-16 12:05 [PATCH v5 00/19] per project build directory Ed Bartosh
                   ` (2 preceding siblings ...)
  2016-03-16 12:05 ` [PATCH v5 03/19] toaster: set BITBAKE_UI environment variable Ed Bartosh
@ 2016-03-16 12:05 ` Ed Bartosh
  2016-03-16 12:05 ` [PATCH v5 05/19] uievent: improve BBUIEventQueue code Ed Bartosh
                   ` (15 subsequent siblings)
  19 siblings, 0 replies; 23+ messages in thread
From: Ed Bartosh @ 2016-03-16 12:05 UTC (permalink / raw)
  To: toaster

In current toaster code BRBE(build request:build environment) value
is passed from toaster to buildinfohelper through the 'SetBRBE' event.

Passing it through environment variable is easier as it doesn't
involve rpc communication between toaster and bitbake server.
It also eliminates the need in running bitbake observer process.

Added parameter 'brbe' to BuildInfoHelper.__init__
Used environment variable TOASTER_BRBE to set brbe for
buildinfohelper object.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py | 4 ++--
 bitbake/lib/bb/ui/toasterui.py       | 3 ++-
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index 7805086..81fd5de 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -837,7 +837,7 @@ class BuildInfoHelper(object):
     # pylint: disable=bad-continuation
     # we do not follow the python conventions for continuation indentation due to long lines here
 
-    def __init__(self, server, has_build_history = False):
+    def __init__(self, server, has_build_history = False, brbe = None):
         self.internal_state = {}
         self.internal_state['taskdata'] = {}
         self.internal_state['targets'] = []
@@ -853,7 +853,7 @@ class BuildInfoHelper(object):
 
         # this is set for Toaster-triggered builds by localhostbecontroller
         # via toasterui
-        self.brbe = None
+        self.brbe = brbe
 
         self.project = None
 
diff --git a/bitbake/lib/bb/ui/toasterui.py b/bitbake/lib/bb/ui/toasterui.py
index ee40110..d1f058f 100644
--- a/bitbake/lib/bb/ui/toasterui.py
+++ b/bitbake/lib/bb/ui/toasterui.py
@@ -197,7 +197,8 @@ def main(server, eventHandler, params):
     taskfailures = []
     first = True
 
-    buildinfohelper = BuildInfoHelper(server, build_history_enabled)
+    buildinfohelper = BuildInfoHelper(server, build_history_enabled,
+                                      os.getenv('TOASTER_BRBE'))
 
     # write our own log files into bitbake's log directory;
     # we're only interested in the path to the parent directory of
-- 
2.1.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v5 05/19] uievent: improve BBUIEventQueue code
  2016-03-16 12:05 [PATCH v5 00/19] per project build directory Ed Bartosh
                   ` (3 preceding siblings ...)
  2016-03-16 12:05 ` [PATCH v5 04/19] toasterui: add brbe parameter to buildinfohelper Ed Bartosh
@ 2016-03-16 12:05 ` Ed Bartosh
  2016-03-16 12:05 ` [PATCH v5 06/19] buildinfohelper: improve handling of providermap Ed Bartosh
                   ` (14 subsequent siblings)
  19 siblings, 0 replies; 23+ messages in thread
From: Ed Bartosh @ 2016-03-16 12:05 UTC (permalink / raw)
  To: toaster

Return value of self.BBServer.registerEventHandler differs between
jethro and master. To be able to build jethro toaster should be
able to communicate with jethro bitbake server i.e. it must work
with both old and new registerEventHandler call.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/ui/uievent.py | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/bb/ui/uievent.py b/bitbake/lib/bb/ui/uievent.py
index 6b479bf..df093c5 100644
--- a/bitbake/lib/bb/ui/uievent.py
+++ b/bitbake/lib/bb/ui/uievent.py
@@ -24,7 +24,7 @@ server and queue them for the UI to process. This process must be used to avoid
 client/server deadlocks.
 """
 
-import socket, threading, pickle
+import socket, threading, pickle, collections
 from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler
 
 class BBUIEventQueue:
@@ -51,7 +51,13 @@ class BBUIEventQueue:
         # giving up
 
         for count_tries in range(5):
-            self.EventHandle, error = self.BBServer.registerEventHandler(self.host, self.port)
+            ret = self.BBServer.registerEventHandler(self.host, self.port)
+
+            if isinstance(ret, collections.Iterable):
+                self.EventHandle, error = ret
+            else:
+                self.EventHandle = ret
+                error = ""
 
             if self.EventHandle != None:
                 break
-- 
2.1.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v5 06/19] buildinfohelper: improve handling of providermap
  2016-03-16 12:05 [PATCH v5 00/19] per project build directory Ed Bartosh
                   ` (4 preceding siblings ...)
  2016-03-16 12:05 ` [PATCH v5 05/19] uievent: improve BBUIEventQueue code Ed Bartosh
@ 2016-03-16 12:05 ` Ed Bartosh
  2016-03-16 12:05 ` [PATCH v5 07/19] toasterui: fix brbe reporting Ed Bartosh
                   ` (13 subsequent siblings)
  19 siblings, 0 replies; 23+ messages in thread
From: Ed Bartosh @ 2016-03-16 12:05 UTC (permalink / raw)
  To: toaster

DepTreeGenerated event doesn't contain 'providermap' data in jethro.
Modified buildinfohelper to handle events without this data. This
should make it possible to handle jethro events coming from jethro
bitbake server by the latest buildinfohelper.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index 81fd5de..0e5f2cf 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -1230,7 +1230,6 @@ class BuildInfoHelper(object):
         assert 'layer-priorities' in event._depgraph
         assert 'pn' in event._depgraph
         assert 'tdepends' in event._depgraph
-        assert 'providermap' in event._depgraph
 
         errormsg = ""
 
@@ -1313,7 +1312,7 @@ class BuildInfoHelper(object):
                 if dep in assume_provided:
                     continue
                 via = None
-                if dep in event._depgraph['providermap']:
+                if 'providermap' in event._depgraph and dep in event._depgraph['providermap']:
                     deprecipe = event._depgraph['providermap'][dep][0]
                     dependency = self.internal_state['recipes'][deprecipe]
                     via = Provides.objects.get_or_create(name=dep,
-- 
2.1.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v5 07/19] toasterui: fix brbe reporting
  2016-03-16 12:05 [PATCH v5 00/19] per project build directory Ed Bartosh
                   ` (5 preceding siblings ...)
  2016-03-16 12:05 ` [PATCH v5 06/19] buildinfohelper: improve handling of providermap Ed Bartosh
@ 2016-03-16 12:05 ` Ed Bartosh
  2016-03-16 12:05 ` [PATCH v5 08/19] toaster: remove startBBServer API Ed Bartosh
                   ` (12 subsequent siblings)
  19 siblings, 0 replies; 23+ messages in thread
From: Ed Bartosh @ 2016-03-16 12:05 UTC (permalink / raw)
  To: toaster

buildinfohelper.brbe is lost when buildinfohelper is closed.
This causes incorrect report of brbe when build is done.

Saved brbe attribute before closing buildinfohelper and used
it to report correct brbe.

Got rid of useless and confusing 'ToasterUI build done 1'
log message.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/ui/toasterui.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/bb/ui/toasterui.py b/bitbake/lib/bb/ui/toasterui.py
index d1f058f..b4d0a39 100644
--- a/bitbake/lib/bb/ui/toasterui.py
+++ b/bitbake/lib/bb/ui/toasterui.py
@@ -374,9 +374,9 @@ def main(server, eventHandler, params):
 
                 # update the build info helper on BuildCompleted, not on CommandXXX
                 buildinfohelper.update_build_information(event, errors, warnings, taskfailures)
+
+                brbe = buildinfohelper.brbe
                 buildinfohelper.close(errorcode)
-                # mark the log output; controllers may kill the toasterUI after seeing this log
-                logger.info("ToasterUI build done 1, brbe: %s", buildinfohelper.brbe )
 
                 # we start a new build info
                 logger.debug("ToasterUI prepared for new build")
@@ -385,7 +385,7 @@ def main(server, eventHandler, params):
                 taskfailures = []
                 buildinfohelper = BuildInfoHelper(server, build_history_enabled)
 
-                logger.info("ToasterUI build done 2")
+                logger.info("ToasterUI build done, brbe: %s", brbe)
                 continue
 
             if isinstance(event, (bb.command.CommandCompleted,
-- 
2.1.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v5 08/19] toaster: remove startBBServer API
  2016-03-16 12:05 [PATCH v5 00/19] per project build directory Ed Bartosh
                   ` (6 preceding siblings ...)
  2016-03-16 12:05 ` [PATCH v5 07/19] toasterui: fix brbe reporting Ed Bartosh
@ 2016-03-16 12:05 ` Ed Bartosh
  2016-03-16 12:05 ` [PATCH v5 09/19] toaster: remove release API Ed Bartosh
                   ` (11 subsequent siblings)
  19 siblings, 0 replies; 23+ messages in thread
From: Ed Bartosh @ 2016-03-16 12:05 UTC (permalink / raw)
  To: toaster

We still will have to run bitbake server, but it will be done
different way and the code will be in triggerBuild function.

Removed startBBServer API from BuildEnvironmentController and
LocalhostBEController classes.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/bbcontroller.py     |  9 ------
 .../toaster/bldcontrol/localhostbecontroller.py    | 32 ----------------------
 2 files changed, 41 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/bbcontroller.py b/bitbake/lib/toaster/bldcontrol/bbcontroller.py
index f40103c..535a398 100644
--- a/bitbake/lib/toaster/bldcontrol/bbcontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/bbcontroller.py
@@ -127,14 +127,6 @@ class BuildEnvironmentController(object):
         bblayerconffile.write("# line added by toaster build control\nBBLAYERS = \"" + " ".join(layerlist) + "\"")
         bblayerconffile.close()
 
-    def startBBServer(self):
-        """ Starts a  BB server with Toaster toasterui set up to record the builds, an no controlling UI.
-            After this method executes, self.be bbaddress/bbport MUST point to a running and free server,
-            and the bbstate MUST be  updated to "started".
-        """
-        raise NotImplementedError("FIXME: Must override in order to actually start the BB server")
-
-
     def setLayers(self, bitbake, ls):
         """ Checks-out bitbake executor and layers from git repositories.
             Sets the layer variables in the config file, after validating local layer paths.
@@ -151,7 +143,6 @@ class BuildEnvironmentController(object):
             starts if needed; or reconnects to the server if we can
         """
         if not self.connection:
-            self.startBBServer()
             self.be.lock = BuildEnvironment.LOCK_RUNNING
             self.be.save()
 
diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index efd82c3..9027291 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -70,38 +70,6 @@ class LocalhostBEController(BuildEnvironmentController):
             logger.debug("localhostbecontroller: shellcmd success")
             return out
 
-    def startBBServer(self):
-        assert self.pokydirname and os.path.exists(self.pokydirname)
-        assert self.islayerset
-
-        # find our own toasterui listener/bitbake
-        from toaster.bldcontrol.management.commands.loadconf import _reduce_canon_path
-
-        toaster = _reduce_canon_path(os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../../bin/toaster"))
-        assert os.path.exists(toaster) and os.path.isfile(toaster)
-
-        # restart bitbake server and toastergui observer
-        self._shellcmd("bash -c 'source %s restart-bitbake'" % toaster, self.be.builddir)
-        logger.debug("localhostbecontroller: restarted bitbake server")
-
-        # read port number from bitbake.lock
-        self.be.bbport = ""
-        bblock = os.path.join(self.be.builddir, 'bitbake.lock')
-        if os.path.exists(bblock):
-            with open(bblock) as fplock:
-                for line in fplock:
-                    if ":" in line:
-                        self.be.bbport = line.split(":")[-1].strip()
-                        logger.debug("localhostbecontroller: bitbake port %s", self.be.bbport)
-                        break
-
-        if not self.be.bbport:
-            raise BuildSetupException("localhostbecontroller: can't read bitbake port from %s" % bblock)
-
-        self.be.bbaddress = "localhost"
-        self.be.bbstate = BuildEnvironment.SERVER_STARTED
-        self.be.save()
-
     def getGitCloneDirectory(self, url, branch):
         """Construct unique clone directory name out of url and branch."""
         if branch != "HEAD":
-- 
2.1.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v5 09/19] toaster: remove release API
  2016-03-16 12:05 [PATCH v5 00/19] per project build directory Ed Bartosh
                   ` (7 preceding siblings ...)
  2016-03-16 12:05 ` [PATCH v5 08/19] toaster: remove startBBServer API Ed Bartosh
@ 2016-03-16 12:05 ` Ed Bartosh
  2016-03-16 12:05 ` [PATCH v5 10/19] toaster: add brbe parameter to triggerBuild Ed Bartosh
                   ` (10 subsequent siblings)
  19 siblings, 0 replies; 23+ messages in thread
From: Ed Bartosh @ 2016-03-16 12:05 UTC (permalink / raw)
  To: toaster

This API is quite dangerous as it removes build directory.
It's not used anywhere and most probably will not be used in future
as toaster is going to have one build directory per project.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/bbcontroller.py          | 6 ------
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 6 ------
 2 files changed, 12 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/bbcontroller.py b/bitbake/lib/toaster/bldcontrol/bbcontroller.py
index 535a398..2ca2661 100644
--- a/bitbake/lib/toaster/bldcontrol/bbcontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/bbcontroller.py
@@ -163,12 +163,6 @@ class BuildEnvironmentController(object):
         """
         raise NotImplementedError("Must return the REST URL of the artifact")
 
-    def release(self):
-        """ This stops the server and releases any resources. After this point, all resources
-            are un-available for further reference
-        """
-        raise NotImplementedError("Must override BE release")
-
     def triggerBuild(self, bitbake, layers, variables, targets):
         raise NotImplementedError("Must override BE release")
 
diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 9027291..007f3be 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -255,12 +255,6 @@ class LocalhostBEController(BuildEnvironmentController):
     def readServerLogFile(self):
         return open(os.path.join(self.be.builddir, "toaster_server.log"), "r").read()
 
-    def release(self):
-        assert self.be.sourcedir and os.path.exists(self.be.builddir)
-        import shutil
-        shutil.rmtree(os.path.join(self.be.sourcedir, "build"))
-        assert not os.path.exists(self.be.builddir)
-
 
     def triggerBuild(self, bitbake, layers, variables, targets):
         # set up the build environment with the needed layers
-- 
2.1.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v5 10/19] toaster: add brbe parameter to triggerBuild
  2016-03-16 12:05 [PATCH v5 00/19] per project build directory Ed Bartosh
                   ` (8 preceding siblings ...)
  2016-03-16 12:05 ` [PATCH v5 09/19] toaster: remove release API Ed Bartosh
@ 2016-03-16 12:05 ` Ed Bartosh
  2016-03-16 12:05 ` [PATCH v5 11/19] toaster: modified setLayers API Ed Bartosh
                   ` (9 subsequent siblings)
  19 siblings, 0 replies; 23+ messages in thread
From: Ed Bartosh @ 2016-03-16 12:05 UTC (permalink / raw)
  To: toaster

Called triggerBuild with brbe parameter instead of adding TOASTER_BRBE
variable to the database and fetching it in triggerBuild.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py         | 2 +-
 bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py | 6 ++----
 2 files changed, 3 insertions(+), 5 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 007f3be..6f2ca33 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -256,7 +256,7 @@ class LocalhostBEController(BuildEnvironmentController):
         return open(os.path.join(self.be.builddir, "toaster_server.log"), "r").read()
 
 
-    def triggerBuild(self, bitbake, layers, variables, targets):
+    def triggerBuild(self, bitbake, layers, variables, targets, brbe):
         # set up the build environment with the needed layers
         self.setLayers(bitbake, layers, targets)
 
diff --git a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
index 0bd5d08..8ba836e 100644
--- a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
+++ b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
@@ -49,15 +49,13 @@ class Command(NoArgsCommand):
 
             logger.debug("runbuilds: starting build %s, environment %s" % (str(br).decode('utf-8'), bec.be))
 
-            # write the build identification variable
-            BRVariable.objects.create(req = br, name="TOASTER_BRBE", value="%d:%d" % (br.pk, bec.be.pk))
-
             # let the build request know where it is being executed
             br.environment = bec.be
             br.save()
 
             # this triggers an async build
-            bec.triggerBuild(br.brbitbake, br.brlayer_set.all(), br.brvariable_set.all(), br.brtarget_set.all())
+            bec.triggerBuild(br.brbitbake, br.brlayer_set.all(), br.brvariable_set.all(),
+                             br.brtarget_set.all(), "%d:%d" % (br.pk, bec.be.pk))
 
         except Exception as e:
             logger.error("runbuilds: Error launching build %s" % e)
-- 
2.1.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v5 11/19] toaster: modified setLayers API
  2016-03-16 12:05 [PATCH v5 00/19] per project build directory Ed Bartosh
                   ` (9 preceding siblings ...)
  2016-03-16 12:05 ` [PATCH v5 10/19] toaster: add brbe parameter to triggerBuild Ed Bartosh
@ 2016-03-16 12:05 ` Ed Bartosh
  2016-03-16 12:05 ` [PATCH v5 12/19] toaster: reimplement triggerBuild Ed Bartosh
                   ` (8 subsequent siblings)
  19 siblings, 0 replies; 23+ messages in thread
From: Ed Bartosh @ 2016-03-16 12:05 UTC (permalink / raw)
  To: toaster

Removed updating of bblayers.conf. It will be done in runBuild method.
Changed return value: return list of layers.
Removed _updateBBLayers method.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/bbcontroller.py         | 18 ------------------
 .../lib/toaster/bldcontrol/localhostbecontroller.py    |  9 +--------
 2 files changed, 1 insertion(+), 26 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/bbcontroller.py b/bitbake/lib/toaster/bldcontrol/bbcontroller.py
index 2ca2661..058e490 100644
--- a/bitbake/lib/toaster/bldcontrol/bbcontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/bbcontroller.py
@@ -109,24 +109,6 @@ class BuildEnvironmentController(object):
         self.be = be
         self.connection = None
 
-    @staticmethod
-    def _updateBBLayers(bblayerconf, layerlist):
-        conflines = open(bblayerconf, "r").readlines()
-
-        bblayerconffile = open(bblayerconf, "w")
-        skip = 0
-        for i in xrange(len(conflines)):
-            if skip > 0:
-                skip =- 1
-                continue
-            if conflines[i].startswith("# line added by toaster"):
-                skip = 1
-            else:
-                bblayerconffile.write(conflines[i])
-
-        bblayerconffile.write("# line added by toaster build control\nBBLAYERS = \"" + " ".join(layerlist) + "\"")
-        bblayerconffile.close()
-
     def setLayers(self, bitbake, ls):
         """ Checks-out bitbake executor and layers from git repositories.
             Sets the layer variables in the config file, after validating local layer paths.
diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 6f2ca33..833c685 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -179,11 +179,6 @@ class LocalhostBEController(BuildEnvironmentController):
 
         logger.debug("localhostbecontroller: current layer list %s " % pformat(layerlist))
 
-        # 4. update the bblayers.conf
-        bblayerconf = os.path.join(self.be.builddir, "conf/bblayers.conf")
-        if not os.path.exists(bblayerconf):
-            raise BuildSetupException("BE is not consistent: bblayers.conf file missing at %s" % bblayerconf)
-
         # 5. create custom layer and add custom recipes to it
         layerpath = os.path.join(self.be.sourcedir, "_meta-toaster-custom")
         if os.path.isdir(layerpath):
@@ -247,10 +242,8 @@ class LocalhostBEController(BuildEnvironmentController):
         if os.path.isdir(layerpath):
             layerlist.append(layerpath)
 
-        BuildEnvironmentController._updateBBLayers(bblayerconf, layerlist)
-
         self.islayerset = True
-        return True
+        return layerlist
 
     def readServerLogFile(self):
         return open(os.path.join(self.be.builddir, "toaster_server.log"), "r").read()
-- 
2.1.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v5 12/19] toaster: reimplement triggerBuild
  2016-03-16 12:05 [PATCH v5 00/19] per project build directory Ed Bartosh
                   ` (10 preceding siblings ...)
  2016-03-16 12:05 ` [PATCH v5 11/19] toaster: modified setLayers API Ed Bartosh
@ 2016-03-16 12:05 ` Ed Bartosh
  2016-03-16 12:05 ` [PATCH v5 13/19] toaster: add new parameter to _shellcmd Ed Bartosh
                   ` (7 subsequent siblings)
  19 siblings, 0 replies; 23+ messages in thread
From: Ed Bartosh @ 2016-03-16 12:05 UTC (permalink / raw)
  To: toaster

Reimplemented triggerBuild method to support one build directory
per project:
 - start bitbake server from the cloned repository
 - don't run observer
 - run bitbake build directly instead of triggering it
   through xmlrpc

[YOCTO #7880]
[YOCTO #9058]
[YOCTO #8958]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 .../toaster/bldcontrol/localhostbecontroller.py    | 100 +++++++++++++--------
 1 file changed, 65 insertions(+), 35 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 833c685..4894b1b 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -250,44 +250,74 @@ class LocalhostBEController(BuildEnvironmentController):
 
 
     def triggerBuild(self, bitbake, layers, variables, targets, brbe):
-        # set up the build environment with the needed layers
-        self.setLayers(bitbake, layers, targets)
+        layers = self.setLayers(bitbake, layers, targets)
+
+        # init build environment from the clone
+        builddir = '%s-toaster-%d' % (self.be.builddir, bitbake.req.project.id)
+        oe_init = os.path.join(self.pokydirname, 'oe-init-build-env')
+        # init build environment
+        self._shellcmd('source %s %s' % (oe_init, builddir), self.be.sourcedir)
+
+        # update bblayers.conf
+        bblconfpath = os.path.join(builddir, "conf/bblayers.conf")
+        conflines = open(bblconfpath, "r").readlines()
+        skip = False
+        with open(bblconfpath, 'w') as bblayers:
+            for line in conflines:
+                if line.startswith("# line added by toaster"):
+                    skip = True
+                    continue
+                if skip:
+                    skip = False
+                else:
+                    bblayers.write(line)
+
+            bblayers.write('# line added by toaster build control\n'
+                           'BBLAYERS = "%s"' % ' '.join(layers))
 
         # write configuration file
-        filepath = os.path.join(self.be.builddir, "conf/toaster.conf")
-        with open(filepath, 'w') as conf:
+        confpath = os.path.join(builddir, 'conf/toaster.conf')
+        with open(confpath, 'w') as conf:
             for var in variables:
                 conf.write('%s="%s"\n' % (var.name, var.value))
             conf.write('INHERIT+="toaster buildhistory"')
 
-        # get the bb server running with the build req id and build env id
-        bbctrl = self.getBBController()
-
-        # set variables; TOASTER_BRBE is not set on the server, as this
-        # causes events from command-line builds to be attached to the last
-        # Toaster-triggered build; instead, TOASTER_BRBE is fired as an event so
-        # that toasterui can set it on the buildinfohelper;
-        # see https://bugzilla.yoctoproject.org/show_bug.cgi?id=9021
-        for var in variables:
-            if var.name == 'TOASTER_BRBE':
-                bbctrl.triggerEvent('bb.event.MetadataEvent("SetBRBE", "%s")' \
-                                     % var.value)
-            else:
-                bbctrl.setVariable(var.name, var.value)
-
-        # Add 'toaster' and 'buildhistory' to INHERIT variable
-        inherit = {item.strip() for item in bbctrl.getVariable('INHERIT').split()}
-        inherit = inherit.union(["toaster", "buildhistory"])
-        bbctrl.setVariable('INHERIT', ' '.join(inherit))
-
-        # trigger the build command
-        task = reduce(lambda x, y: x if len(y)== 0 else y, map(lambda y: y.task, targets))
-        if len(task) == 0:
-            task = None
-
-        bbctrl.build(list(map(lambda x:x.target, targets)), task)
-
-        logger.debug("localhostbecontroller: Build launched, exiting. Follow build logs at %s/toaster_ui.log" % self.be.builddir)
-
-        # disconnect from the server
-        bbctrl.disconnect()
+        # run bitbake server from the clone
+        bitbake = os.path.join(self.pokydirname, 'bitbake', 'bin', 'bitbake')
+        self._shellcmd('source %s %s; BITBAKE_UI="" %s --read %s '
+                       '--server-only -t xmlrpc -B 0.0.0.0:0' % (oe_init, builddir,
+                       bitbake, confpath), self.be.sourcedir)
+
+        # read port number from bitbake.lock
+        self.be.bbport = ""
+        bblock = os.path.join(builddir, 'bitbake.lock')
+        with open(bblock) as fplock:
+            for line in fplock:
+                if ":" in line:
+                    self.be.bbport = line.split(":")[-1].strip()
+                    logger.debug("localhostbecontroller: bitbake port %s", self.be.bbport)
+                    break
+
+        if not self.be.bbport:
+            raise BuildSetupException("localhostbecontroller: can't read bitbake port from %s" % bblock)
+
+        self.be.bbaddress = "localhost"
+        self.be.bbstate = BuildEnvironment.SERVER_STARTED
+        self.be.lock = BuildEnvironment.LOCK_RUNNING
+        self.be.save()
+
+        bbtargets = ''
+        for target in targets:
+            task = target.task or 'do_build'
+            if not task.startswith('do_'):
+                task = 'do_' + task
+            bbtargets += '%s:%s ' % (target.target, task)
+
+        # run build with local bitbake
+        log = os.path.join(builddir, 'toaster_ui.log')
+        self._shellcmd('TOASTER_BRBE="%s" BBSERVER="0.0.0.0:-1" '
+                       '../bitbake/bin/bitbake %s -u toasterui '
+                       '>>%s 2>&1 &' % (brbe, bbtargets, log), builddir)
+
+        logger.debug('localhostbecontroller: Build launched, exiting. '
+                     'Follow build logs at %s' % log)
-- 
2.1.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v5 13/19] toaster: add new parameter to _shellcmd
  2016-03-16 12:05 [PATCH v5 00/19] per project build directory Ed Bartosh
                   ` (11 preceding siblings ...)
  2016-03-16 12:05 ` [PATCH v5 12/19] toaster: reimplement triggerBuild Ed Bartosh
@ 2016-03-16 12:05 ` Ed Bartosh
  2016-03-16 12:05 ` [PATCH v5 14/19] toaster: stop bitbake server after the build Ed Bartosh
                   ` (6 subsequent siblings)
  19 siblings, 0 replies; 23+ messages in thread
From: Ed Bartosh @ 2016-03-16 12:05 UTC (permalink / raw)
  To: toaster

Added 'nowait' parameter to _shellcmd method to support
running chain of commands in a subshell.

This is going to be used to stop bitbake server after
the build.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 4894b1b..5905842 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -51,12 +51,14 @@ class LocalhostBEController(BuildEnvironmentController):
         self.pokydirname = None
         self.islayerset = False
 
-    def _shellcmd(self, command, cwd = None):
+    def _shellcmd(self, command, cwd=None, nowait=False):
         if cwd is None:
             cwd = self.be.sourcedir
 
         logger.debug("lbc_shellcmmd: (%s) %s" % (cwd, command))
         p = subprocess.Popen(command, cwd = cwd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        if nowait:
+            return
         (out,err) = p.communicate()
         p.wait()
         if p.returncode:
-- 
2.1.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v5 14/19] toaster: stop bitbake server after the build
  2016-03-16 12:05 [PATCH v5 00/19] per project build directory Ed Bartosh
                   ` (12 preceding siblings ...)
  2016-03-16 12:05 ` [PATCH v5 13/19] toaster: add new parameter to _shellcmd Ed Bartosh
@ 2016-03-16 12:05 ` Ed Bartosh
  2016-03-16 12:05 ` [PATCH v5 15/19] toaster: update conf/local.conf Ed Bartosh
                   ` (5 subsequent siblings)
  19 siblings, 0 replies; 23+ messages in thread
From: Ed Bartosh @ 2016-03-16 12:05 UTC (permalink / raw)
  To: toaster

Bitbake server is used only during the build. There is no need
in keeping server running after the build.

Running bitbake -m in the subshell after the build should stop
the server.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 5905842..ad6fed2 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -315,11 +315,13 @@ class LocalhostBEController(BuildEnvironmentController):
                 task = 'do_' + task
             bbtargets += '%s:%s ' % (target.target, task)
 
-        # run build with local bitbake
+        # run build with local bitbake. stop the server after the build.
         log = os.path.join(builddir, 'toaster_ui.log')
-        self._shellcmd('TOASTER_BRBE="%s" BBSERVER="0.0.0.0:-1" '
-                       '../bitbake/bin/bitbake %s -u toasterui '
-                       '>>%s 2>&1 &' % (brbe, bbtargets, log), builddir)
+        self._shellcmd(['(TOASTER_BRBE="%s" BBSERVER="0.0.0.0:-1" '
+                        '../bitbake/bin/bitbake %s -u toasterui >>%s 2>&1;'
+                        'BITBAKE_UI="" BBSERVER=0.0.0.0:-1 %s -m)&' \
+                        % (brbe, bbtargets, log, bitbake)], builddir,
+                        nowait=True)
 
         logger.debug('localhostbecontroller: Build launched, exiting. '
                      'Follow build logs at %s' % log)
-- 
2.1.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v5 15/19] toaster: update conf/local.conf
  2016-03-16 12:05 [PATCH v5 00/19] per project build directory Ed Bartosh
                   ` (13 preceding siblings ...)
  2016-03-16 12:05 ` [PATCH v5 14/19] toaster: stop bitbake server after the build Ed Bartosh
@ 2016-03-16 12:05 ` Ed Bartosh
  2016-03-16 12:05 ` [PATCH v5 16/19] toaster: fix jethro build Ed Bartosh
                   ` (4 subsequent siblings)
  19 siblings, 0 replies; 23+ messages in thread
From: Ed Bartosh @ 2016-03-16 12:05 UTC (permalink / raw)
  To: toaster

Added 'INHERIT+="toaster buildhistory"' line to the conf/local conf
when Toaster starts. It should make commandline builds to provide
all required information to Toaster backend.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/bin/toaster | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster
index 5cc77ed..dfaff2e 100755
--- a/bitbake/bin/toaster
+++ b/bitbake/bin/toaster
@@ -252,9 +252,9 @@ case $CMD in
         fi
 
         # Create configuration file
-        conf=${BUILDDIR}/conf/toaster.conf
-        echo "# Created by toaster start script" > $conf
-        echo "INHERIT+=\"toaster buildhistory\"" >> $conf
+        conf=${BUILDDIR}/conf/local.conf
+        line='INHERIT+="toaster buildhistory"'
+        grep -q "$line" $conf || echo $line >> $conf
 
         if [ $WEBSERVER -gt 0 ] && ! webserverStartAll; then
             echo "Failed ${CMD}."
-- 
2.1.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v5 16/19] toaster: fix jethro build
  2016-03-16 12:05 [PATCH v5 00/19] per project build directory Ed Bartosh
                   ` (14 preceding siblings ...)
  2016-03-16 12:05 ` [PATCH v5 15/19] toaster: update conf/local.conf Ed Bartosh
@ 2016-03-16 12:05 ` Ed Bartosh
  2016-03-16 12:05 ` [PATCH v5 17/19] toaster: use bash explicitly Ed Bartosh
                   ` (3 subsequent siblings)
  19 siblings, 0 replies; 23+ messages in thread
From: Ed Bartosh @ 2016-03-16 12:05 UTC (permalink / raw)
  To: toaster

The keys 'started', 'ended', 'cpu_time_user', 'disk_io_read' and
'disk_io_write' were added to the event recently, so they don't
exist in the events generated by bitbake server from older releases.

Checking if task_to_update structure has these keys before using
them should fix build of older releases.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index 0e5f2cf..67826a5 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -238,14 +238,16 @@ class ORMWrapper(object):
             recipe__name = recipe_name
         )
 
-        task_to_update.started = self._timestamp_to_datetime(task_stats['started'])
-        task_to_update.ended = self._timestamp_to_datetime(task_stats['ended'])
-        task_to_update.elapsed_time = (task_stats['ended'] - task_stats['started'])
-        task_to_update.cpu_time_user = task_stats['cpu_time_user']
-        task_to_update.cpu_time_system = task_stats['cpu_time_system']
-        task_to_update.disk_io_read = task_stats['disk_io_read']
-        task_to_update.disk_io_write = task_stats['disk_io_write']
-        task_to_update.disk_io = task_stats['disk_io_read'] + task_stats['disk_io_write']
+        if 'started' in task_stats and 'ended' in task_stats:
+            task_to_update.started = self._timestamp_to_datetime(task_stats['started'])
+            task_to_update.ended = self._timestamp_to_datetime(task_stats['ended'])
+            task_to_update.elapsed_time = (task_stats['ended'] - task_stats['started'])
+        task_to_update.cpu_time_user = task_stats.get('cpu_time_user')
+        task_to_update.cpu_time_system = task_stats.get('cpu_time_system')
+        if 'disk_io_read' in task_stats and 'disk_io_write' in task_stats:
+            task_to_update.disk_io_read = task_stats['disk_io_read']
+            task_to_update.disk_io_write = task_stats['disk_io_write']
+            task_to_update.disk_io = task_stats['disk_io_read'] + task_stats['disk_io_write']
 
         task_to_update.save()
 
-- 
2.1.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v5 17/19] toaster: use bash explicitly
  2016-03-16 12:05 [PATCH v5 00/19] per project build directory Ed Bartosh
                   ` (15 preceding siblings ...)
  2016-03-16 12:05 ` [PATCH v5 16/19] toaster: fix jethro build Ed Bartosh
@ 2016-03-16 12:05 ` Ed Bartosh
  2016-03-16 12:05 ` [PATCH v5 18/19] toasterui: detect build run start correctly on Jethro Ed Bartosh
                   ` (2 subsequent siblings)
  19 siblings, 0 replies; 23+ messages in thread
From: Ed Bartosh @ 2016-03-16 12:05 UTC (permalink / raw)
  To: toaster

Toaster can't run builds on Ubuntu as default /bin/sh points
to dash there. The reason is that oe-init-build-env can't
be sourced under dash for various reasons. It can be fixed
or work arounded, but it wouldn't fix builds for older releases.
Explicitly using bash to start builds should fix the issue.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index ad6fed2..480f945 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -258,7 +258,8 @@ class LocalhostBEController(BuildEnvironmentController):
         builddir = '%s-toaster-%d' % (self.be.builddir, bitbake.req.project.id)
         oe_init = os.path.join(self.pokydirname, 'oe-init-build-env')
         # init build environment
-        self._shellcmd('source %s %s' % (oe_init, builddir), self.be.sourcedir)
+        self._shellcmd("bash -c 'source %s %s'" % (oe_init, builddir),
+                       self.be.sourcedir)
 
         # update bblayers.conf
         bblconfpath = os.path.join(builddir, "conf/bblayers.conf")
@@ -286,9 +287,9 @@ class LocalhostBEController(BuildEnvironmentController):
 
         # run bitbake server from the clone
         bitbake = os.path.join(self.pokydirname, 'bitbake', 'bin', 'bitbake')
-        self._shellcmd('source %s %s; BITBAKE_UI="" %s --read %s '
-                       '--server-only -t xmlrpc -B 0.0.0.0:0' % (oe_init, builddir,
-                       bitbake, confpath), self.be.sourcedir)
+        self._shellcmd('bash -c \"source %s %s; BITBAKE_UI="" %s --read %s '
+                       '--server-only -t xmlrpc -B 0.0.0.0:0\"' % (oe_init,
+                       builddir, bitbake, confpath), self.be.sourcedir)
 
         # read port number from bitbake.lock
         self.be.bbport = ""
@@ -317,9 +318,9 @@ class LocalhostBEController(BuildEnvironmentController):
 
         # run build with local bitbake. stop the server after the build.
         log = os.path.join(builddir, 'toaster_ui.log')
-        self._shellcmd(['(TOASTER_BRBE="%s" BBSERVER="0.0.0.0:-1" '
+        self._shellcmd(['bash -c \"(TOASTER_BRBE="%s" BBSERVER="0.0.0.0:-1" '
                         '../bitbake/bin/bitbake %s -u toasterui >>%s 2>&1;'
-                        'BITBAKE_UI="" BBSERVER=0.0.0.0:-1 %s -m)&' \
+                        'BITBAKE_UI="" BBSERVER=0.0.0.0:-1 %s -m)&\"' \
                         % (brbe, bbtargets, log, bitbake)], builddir,
                         nowait=True)
 
-- 
2.1.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v5 18/19] toasterui: detect build run start correctly on Jethro
  2016-03-16 12:05 [PATCH v5 00/19] per project build directory Ed Bartosh
                   ` (16 preceding siblings ...)
  2016-03-16 12:05 ` [PATCH v5 17/19] toaster: use bash explicitly Ed Bartosh
@ 2016-03-16 12:05 ` Ed Bartosh
  2016-03-16 12:05 ` [PATCH v5 19/19] toasterui: shutdown on BuildCompleted event Ed Bartosh
  2016-03-17 17:20 ` [PATCH v5 00/19] per project build directory Barros Pena, Belen
  19 siblings, 0 replies; 23+ messages in thread
From: Ed Bartosh @ 2016-03-16 12:05 UTC (permalink / raw)
  To: toaster

From: Elliot Smith <elliot.smith@intel.com>

We currently use the TargetsAcquired event fired by the XMLRPC
server to mark the start of a build run from the command line.
At this point, we create the Build object in buildinfohelper,
then append targets etc. to it as the run continues.

However, on Jethro, this event isn't fired. This means that we
can't support builds with the Jethro release correctly, as
a Build object is never created.

Add a condition so that we can create a Build object when building
with Jethro by falling back to the BuildStarted event. Jethro
builds which fail early (bad target, bad MACHINE, bad DISTRO etc.)
will not be captured (as we need the TargetsAcquired event for
that), but other builds which succeed or fail later will be
captured.

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py | 23 +++++++++++++----------
 bitbake/lib/bb/ui/toasterui.py       | 26 +++++++++++++++++++-------
 2 files changed, 32 insertions(+), 17 deletions(-)

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index 67826a5..455caaf 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -970,24 +970,27 @@ class BuildInfoHelper(object):
         build_information = self._get_build_information(build_log_path)
         self.internal_state['build'] = self.orm_wrapper.create_build_object(build_information, self.brbe)
 
-    def store_targets(self, event):
+    def store_targets(self, targets):
         """
         store targets for the current build, if that build was started from
         the command line; targets for non-cli builds are irrelevant, as we
         create them from the BuildRequest anyway
 
-        event: a TargetsAcquired event with a task property (e.g. "build")
-        and a targetsList property (e.g. ["zlib", "dropbear"])
+        targets: a list of targets for the build, e.g.
+        ["zlib:build", "dropbear:build"]
         """
-        if self.internal_state['build'].project.is_default:
-            targets = map(lambda target: target + ':' + event.task, event.targetsList)
 
-            target_information = {
-              'targets': targets,
-              'build': self.internal_state['build']
-            }
+        # non-command-line builds have their targets stored when the build
+        # request is created
+        if not self.internal_state['build'].project.is_default:
+            return
+
+        target_information = {
+          'targets': targets,
+          'build': self.internal_state['build']
+        }
 
-            self.internal_state['targets'] = self.orm_wrapper.get_or_create_targets(target_information)
+        self.internal_state['targets'] = self.orm_wrapper.get_or_create_targets(target_information)
 
     def update_build(self, event):
         """
diff --git a/bitbake/lib/bb/ui/toasterui.py b/bitbake/lib/bb/ui/toasterui.py
index b4d0a39..8270767 100644
--- a/bitbake/lib/bb/ui/toasterui.py
+++ b/bitbake/lib/bb/ui/toasterui.py
@@ -233,18 +233,30 @@ def main(server, eventHandler, params):
             # pylint: disable=protected-access
             # the code will look into the protected variables of the event; no easy way around this
 
-            # start of build: this event is fired just before the buildTargets()
-            # command is invoked on the XMLRPC server
-            if isinstance(event, bb.event.TargetsAcquired):
+            # start of build: TargetsAcquired event is fired just before the
+            # buildTargets() command is invoked on the XMLRPC server; if
+            # we are on Jethro, we don't get TargetsAcquired, so treat
+            # BuildStarted as the start of the build instead
+            targets_acquired = isinstance(event, bb.event.TargetsAcquired)
+            build_started = isinstance(event, bb.event.BuildStarted)
+
+            if targets_acquired or build_started:
                 if not (build_log and build_log_file_path):
                     build_log, build_log_file_path = _open_build_log(log_dir)
-                buildinfohelper.store_new_build(build_log_file_path)
-                buildinfohelper.store_targets(event)
-                continue
+                    buildinfohelper.store_new_build(build_log_file_path)
+
+                    # BuildStarted signals the start of the build on Toaster 2.0,
+                    # TargetsAcquired signals it on Toaster 2.1+
+                    if build_started:
+                        targets = event._pkgs
+                    else:
+                        targets = map(lambda target: target + ':' + event.task, event.targetsList)
+
+                    buildinfohelper.store_targets(targets)
 
             # when the build proper starts, we extract information about
             # any layers and config data
-            if isinstance(event, bb.event.BuildStarted):
+            if build_started:
                 buildinfohelper.update_build(event)
                 continue
 
-- 
2.1.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v5 19/19] toasterui: shutdown on BuildCompleted event
  2016-03-16 12:05 [PATCH v5 00/19] per project build directory Ed Bartosh
                   ` (17 preceding siblings ...)
  2016-03-16 12:05 ` [PATCH v5 18/19] toasterui: detect build run start correctly on Jethro Ed Bartosh
@ 2016-03-16 12:05 ` Ed Bartosh
  2016-03-17 17:20 ` [PATCH v5 00/19] per project build directory Barros Pena, Belen
  19 siblings, 0 replies; 23+ messages in thread
From: Ed Bartosh @ 2016-03-16 12:05 UTC (permalink / raw)
  To: toaster

toasterui exits event loop on one of the following events:
CommandCompleted, CommandFailed or CommandExit.
Unfortunately none of them come from bitbake when build fails.
This is normai if toasterui runs in observer mode. However, if it's
in build mode this causes toasterui to stuck in the infinite loop
waiting for new events.

The only event we can rely on is BuildCompleted as it always
comes from bitbake unlike 3 above mentioned events.

Modified the code to always shutdown toasterui in build mode
on BuildCompleted event.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/ui/toasterui.py | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/bitbake/lib/bb/ui/toasterui.py b/bitbake/lib/bb/ui/toasterui.py
index 8270767..48a5138 100644
--- a/bitbake/lib/bb/ui/toasterui.py
+++ b/bitbake/lib/bb/ui/toasterui.py
@@ -391,11 +391,14 @@ def main(server, eventHandler, params):
                 buildinfohelper.close(errorcode)
 
                 # we start a new build info
-                logger.debug("ToasterUI prepared for new build")
-                errors = 0
-                warnings = 0
-                taskfailures = []
-                buildinfohelper = BuildInfoHelper(server, build_history_enabled)
+                if params.observe_only:
+                    logger.debug("ToasterUI prepared for new build")
+                    errors = 0
+                    warnings = 0
+                    taskfailures = []
+                    buildinfohelper = BuildInfoHelper(server, build_history_enabled)
+                else:
+                    main.shutdown = 1
 
                 logger.info("ToasterUI build done, brbe: %s", brbe)
                 continue
-- 
2.1.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread
* Re: [PATCH v5 00/19] per project build directory
  2016-03-17 17:20 ` [PATCH v5 00/19] per project build directory Barros Pena, Belen
@ 2016-03-17 16:22   ` Ed Bartosh
  2016-03-18 17:03     ` Barros Pena, Belen
  0 siblings, 1 reply; 23+ messages in thread
From: Ed Bartosh @ 2016-03-17 16:22 UTC (permalink / raw)
  To: Barros Pena, Belen; +Cc: Avery, Brian, toaster

Hi Belen,

Thank you for testing!

See my comments below.

On Thu, Mar 17, 2016 at 05:20:59PM +0000, Barros Pena, Belen wrote:
> On 16/03/2016 12:05, "toaster-bounces@yoctoproject.org on behalf of Ed
> Bartosh" <toaster-bounces@yoctoproject.org on behalf of
> ed.bartosh@linux.intel.com> wrote:
> 
> >Hi,
> >
> >Default build directory is not going to be used for project builds
> >anymore.
> >It can be used only for command line builds. Toaster will use
> >build-toaster-<project id>
> >directories for project builds.
> 
> FWIW, this seems to be working for me. I have built master, jethro, local
> and from cli. 
> 
> Some things to keep in mind, once the series is upstream:
> 
> 1. Projects should be sharing sstate and downloads directories by default.
> Right not they don't, and my testing because of that was painfully slow (I
> had to run a clean build every time). I am not sure if Sujith's patches
> are enough for this
> (https://lists.yoctoproject.org/pipermail/toaster/2016-February/003832.html
> ) or if something else needs to be done. Ed: would you be able to have a
> look to Sujith's branch once you are done with this
>
Sure, I'm going to look at it.

What should be default values for DL_DIR and SSTATE_DIR?
Would something like $(TOPDIR)/../downloads and $(TOPDIR)/../sstate-cache be ok?

> 2. No toaster_ui.log file is created inside the /build directory, which is
> where cli builds are run. This might be expected, though.
>
Are you sure?
They should be created in build-toaster-<project id>/ directories.

> 3. The _meta-toaster-custom layer for the custom images should be created
> inside each project build directory: it should not be shared across
> projects (right now it is, since it is in the source root directory)
>
Makes sense to me. I'll try to do it after looking at Sujith patchset.

Regards,
Ed

> Cheers
> 
> Belén
> 
> >
> >This is how to test this:
> > - run toaster
> > - run command line build: build zlib
> > - check if it's shown in Toaster UI
> > - create project for Yocto master
> > - trigger project build from UI
> > - wait until it succeeds
> > - create project for Jethro
> > - trigger project build from UI
> > - wait until it succeds
> > - check that 2 build directories build-toaster-<project id> exist and
> >contain
> >   correct conf/toaster.conf and conf/bblayers.conf
> >
> >Changes in v2: stop bitbake server after the build
> >Changes in v3: fixed '/bin/sh: 1: source: not found' dash issue
> >               used project id in build directory name
> >               added 'INHERIT+='toaster buildhistory' to conf/local.conf
> >on toaster start
> >Changes in v4: fixed builds on Ubuntu caused by dash being default shell
> >Changes in v5: fixed nasty bug in toasterui causing it to stuck in event
> >loop
> >
> >The following changes since commit
> >01ab10ddaf21e34d8b3578975d1494cf0cfa4ef7:
> >
> >  toaster.bbclass: show packages that were setscened into existence too
> >(2016-03-10 15:58:34 +0000)
> >
> >are available in the git repository at:
> >
> >  git://git.yoctoproject.org/poky-contrib
> >ed/toaster/project-build-dir-7880
> >  
> >http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/projec
> >t-build-dir-7880
> >
> >Ed Bartosh (18):
> >  toaster: don't start bitbake server
> >  toaster: get rid of noui option
> >  toaster: set BITBAKE_UI environment variable
> >  toasterui: add brbe parameter to buildinfohelper
> >  uievent: improve BBUIEventQueue code
> >  buildinfohelper: improve handling of providermap
> >  toasterui: fix brbe reporting
> >  toaster: remove startBBServer API
> >  toaster: remove release API
> >  toaster: add brbe parameter to triggerBuild
> >  toaster: modified setLayers API
> >  toaster: reimplement triggerBuild
> >  toaster: add new parameter to _shellcmd
> >  toaster: stop bitbake server after the build
> >  toaster: update conf/local.conf
> >  toaster: fix jethro build
> >  toaster: use bash explicitly
> >  toasterui: shutdown on BuildCompleted event
> >
> >Elliot Smith (1):
> >  toasterui: detect build run start correctly on Jethro
> >
> > bitbake/bin/toaster                                |  79 ++---------
> > bitbake/lib/bb/ui/buildinfohelper.py               |  48 +++----
> > bitbake/lib/bb/ui/toasterui.py                     |  48 ++++---
> > bitbake/lib/bb/ui/uievent.py                       |  10 +-
> > bitbake/lib/toaster/bldcontrol/bbcontroller.py     |  33 -----
> > .../toaster/bldcontrol/localhostbecontroller.py    | 146
> >++++++++++-----------
> > .../bldcontrol/management/commands/runbuilds.py    |   6 +-
> > 7 files changed, 147 insertions(+), 223 deletions(-)
> >
> >--
> >Regards,
> >Ed
> >-- 
> >_______________________________________________
> >toaster mailing list
> >toaster@yoctoproject.org
> >https://lists.yoctoproject.org/listinfo/toaster
> 

-- 
--
Regards,
Ed


^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH v5 00/19] per project build directory
  2016-03-16 12:05 [PATCH v5 00/19] per project build directory Ed Bartosh
                   ` (18 preceding siblings ...)
  2016-03-16 12:05 ` [PATCH v5 19/19] toasterui: shutdown on BuildCompleted event Ed Bartosh
@ 2016-03-17 17:20 ` Barros Pena, Belen
  2016-03-17 16:22   ` Ed Bartosh
  19 siblings, 1 reply; 23+ messages in thread
From: Barros Pena, Belen @ 2016-03-17 17:20 UTC (permalink / raw)
  To: Ed Bartosh, toaster; +Cc: Avery, Brian

On 16/03/2016 12:05, "toaster-bounces@yoctoproject.org on behalf of Ed
Bartosh" <toaster-bounces@yoctoproject.org on behalf of
ed.bartosh@linux.intel.com> wrote:

>Hi,
>
>Default build directory is not going to be used for project builds
>anymore.
>It can be used only for command line builds. Toaster will use
>build-toaster-<project id>
>directories for project builds.

FWIW, this seems to be working for me. I have built master, jethro, local
and from cli. 

Some things to keep in mind, once the series is upstream:

1. Projects should be sharing sstate and downloads directories by default.
Right not they don't, and my testing because of that was painfully slow (I
had to run a clean build every time). I am not sure if Sujith's patches
are enough for this
(https://lists.yoctoproject.org/pipermail/toaster/2016-February/003832.html
) or if something else needs to be done. Ed: would you be able to have a
look to Sujith's branch once you are done with this

2. No toaster_ui.log file is created inside the /build directory, which is
where cli builds are run. This might be expected, though.

3. The _meta-toaster-custom layer for the custom images should be created
inside each project build directory: it should not be shared across
projects (right now it is, since it is in the source root directory)

Cheers

Belén

>
>This is how to test this:
> - run toaster
> - run command line build: build zlib
> - check if it's shown in Toaster UI
> - create project for Yocto master
> - trigger project build from UI
> - wait until it succeeds
> - create project for Jethro
> - trigger project build from UI
> - wait until it succeds
> - check that 2 build directories build-toaster-<project id> exist and
>contain
>   correct conf/toaster.conf and conf/bblayers.conf
>
>Changes in v2: stop bitbake server after the build
>Changes in v3: fixed '/bin/sh: 1: source: not found' dash issue
>               used project id in build directory name
>               added 'INHERIT+='toaster buildhistory' to conf/local.conf
>on toaster start
>Changes in v4: fixed builds on Ubuntu caused by dash being default shell
>Changes in v5: fixed nasty bug in toasterui causing it to stuck in event
>loop
>
>The following changes since commit
>01ab10ddaf21e34d8b3578975d1494cf0cfa4ef7:
>
>  toaster.bbclass: show packages that were setscened into existence too
>(2016-03-10 15:58:34 +0000)
>
>are available in the git repository at:
>
>  git://git.yoctoproject.org/poky-contrib
>ed/toaster/project-build-dir-7880
>  
>http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/projec
>t-build-dir-7880
>
>Ed Bartosh (18):
>  toaster: don't start bitbake server
>  toaster: get rid of noui option
>  toaster: set BITBAKE_UI environment variable
>  toasterui: add brbe parameter to buildinfohelper
>  uievent: improve BBUIEventQueue code
>  buildinfohelper: improve handling of providermap
>  toasterui: fix brbe reporting
>  toaster: remove startBBServer API
>  toaster: remove release API
>  toaster: add brbe parameter to triggerBuild
>  toaster: modified setLayers API
>  toaster: reimplement triggerBuild
>  toaster: add new parameter to _shellcmd
>  toaster: stop bitbake server after the build
>  toaster: update conf/local.conf
>  toaster: fix jethro build
>  toaster: use bash explicitly
>  toasterui: shutdown on BuildCompleted event
>
>Elliot Smith (1):
>  toasterui: detect build run start correctly on Jethro
>
> bitbake/bin/toaster                                |  79 ++---------
> bitbake/lib/bb/ui/buildinfohelper.py               |  48 +++----
> bitbake/lib/bb/ui/toasterui.py                     |  48 ++++---
> bitbake/lib/bb/ui/uievent.py                       |  10 +-
> bitbake/lib/toaster/bldcontrol/bbcontroller.py     |  33 -----
> .../toaster/bldcontrol/localhostbecontroller.py    | 146
>++++++++++-----------
> .../bldcontrol/management/commands/runbuilds.py    |   6 +-
> 7 files changed, 147 insertions(+), 223 deletions(-)
>
>--
>Regards,
>Ed
>-- 
>_______________________________________________
>toaster mailing list
>toaster@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/toaster



^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH v5 00/19] per project build directory
  2016-03-17 16:22   ` Ed Bartosh
@ 2016-03-18 17:03     ` Barros Pena, Belen
  0 siblings, 0 replies; 23+ messages in thread
From: Barros Pena, Belen @ 2016-03-18 17:03 UTC (permalink / raw)
  To: ed.bartosh; +Cc: Avery, Brian, toaster



On 17/03/2016 16:22, "Ed Bartosh" <ed.bartosh@linux.intel.com> wrote:

>Hi Belen,
>
>Thank you for testing!
>
>See my comments below.
>
>On Thu, Mar 17, 2016 at 05:20:59PM +0000, Barros Pena, Belen wrote:
>> On 16/03/2016 12:05, "toaster-bounces@yoctoproject.org on behalf of Ed
>> Bartosh" <toaster-bounces@yoctoproject.org on behalf of
>> ed.bartosh@linux.intel.com> wrote:
>> 
>> >Hi,
>> >
>> >Default build directory is not going to be used for project builds
>> >anymore.
>> >It can be used only for command line builds. Toaster will use
>> >build-toaster-<project id>
>> >directories for project builds.
>> 
>> FWIW, this seems to be working for me. I have built master, jethro,
>>local
>> and from cli. 
>> 
>> Some things to keep in mind, once the series is upstream:
>> 
>> 1. Projects should be sharing sstate and downloads directories by
>>default.
>> Right not they don't, and my testing because of that was painfully slow
>>(I
>> had to run a clean build every time). I am not sure if Sujith's patches
>> are enough for this
>> 
>>(https://lists.yoctoproject.org/pipermail/toaster/2016-February/003832.ht
>>ml
>> ) or if something else needs to be done. Ed: would you be able to have a
>> look to Sujith's branch once you are done with this
>>
>Sure, I'm going to look at it.
>
>What should be default values for DL_DIR and SSTATE_DIR?
>Would something like $(TOPDIR)/../downloads and $(TOPDIR)/../sstate-cache
>be ok?

That looks ok to me.

>
>> 2. No toaster_ui.log file is created inside the /build directory, which
>>is
>> where cli builds are run. This might be expected, though.
>>
>Are you sure?
>They should be created in build-toaster-<project id>/ directories.

Yes, you have them inside the build-toaster-* directories, but not in the
/build directory (where cli builds happen). But as I said, that might be
expected. 

>
>> 3. The _meta-toaster-custom layer for the custom images should be
>>created
>> inside each project build directory: it should not be shared across
>> projects (right now it is, since it is in the source root directory)
>>
>Makes sense to me. I'll try to do it after looking at Sujith patchset.
>
>Regards,
>Ed
>
>> Cheers
>> 
>> Belén
>> 
>> >
>> >This is how to test this:
>> > - run toaster
>> > - run command line build: build zlib
>> > - check if it's shown in Toaster UI
>> > - create project for Yocto master
>> > - trigger project build from UI
>> > - wait until it succeeds
>> > - create project for Jethro
>> > - trigger project build from UI
>> > - wait until it succeds
>> > - check that 2 build directories build-toaster-<project id> exist and
>> >contain
>> >   correct conf/toaster.conf and conf/bblayers.conf
>> >
>> >Changes in v2: stop bitbake server after the build
>> >Changes in v3: fixed '/bin/sh: 1: source: not found' dash issue
>> >               used project id in build directory name
>> >               added 'INHERIT+='toaster buildhistory' to
>>conf/local.conf
>> >on toaster start
>> >Changes in v4: fixed builds on Ubuntu caused by dash being default
>>shell
>> >Changes in v5: fixed nasty bug in toasterui causing it to stuck in
>>event
>> >loop
>> >
>> >The following changes since commit
>> >01ab10ddaf21e34d8b3578975d1494cf0cfa4ef7:
>> >
>> >  toaster.bbclass: show packages that were setscened into existence too
>> >(2016-03-10 15:58:34 +0000)
>> >
>> >are available in the git repository at:
>> >
>> >  git://git.yoctoproject.org/poky-contrib
>> >ed/toaster/project-build-dir-7880
>> >  
>> 
>>>http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/proj
>>>ec
>> >t-build-dir-7880
>> >
>> >Ed Bartosh (18):
>> >  toaster: don't start bitbake server
>> >  toaster: get rid of noui option
>> >  toaster: set BITBAKE_UI environment variable
>> >  toasterui: add brbe parameter to buildinfohelper
>> >  uievent: improve BBUIEventQueue code
>> >  buildinfohelper: improve handling of providermap
>> >  toasterui: fix brbe reporting
>> >  toaster: remove startBBServer API
>> >  toaster: remove release API
>> >  toaster: add brbe parameter to triggerBuild
>> >  toaster: modified setLayers API
>> >  toaster: reimplement triggerBuild
>> >  toaster: add new parameter to _shellcmd
>> >  toaster: stop bitbake server after the build
>> >  toaster: update conf/local.conf
>> >  toaster: fix jethro build
>> >  toaster: use bash explicitly
>> >  toasterui: shutdown on BuildCompleted event
>> >
>> >Elliot Smith (1):
>> >  toasterui: detect build run start correctly on Jethro
>> >
>> > bitbake/bin/toaster                                |  79 ++---------
>> > bitbake/lib/bb/ui/buildinfohelper.py               |  48 +++----
>> > bitbake/lib/bb/ui/toasterui.py                     |  48 ++++---
>> > bitbake/lib/bb/ui/uievent.py                       |  10 +-
>> > bitbake/lib/toaster/bldcontrol/bbcontroller.py     |  33 -----
>> > .../toaster/bldcontrol/localhostbecontroller.py    | 146
>> >++++++++++-----------
>> > .../bldcontrol/management/commands/runbuilds.py    |   6 +-
>> > 7 files changed, 147 insertions(+), 223 deletions(-)
>> >
>> >--
>> >Regards,
>> >Ed
>> >-- 
>> >_______________________________________________
>> >toaster mailing list
>> >toaster@yoctoproject.org
>> >https://lists.yoctoproject.org/listinfo/toaster
>> 
>
>-- 
>--
>Regards,
>Ed



^ permalink raw reply	[flat|nested] 23+ messages in thread
end of thread, other threads:[~2016-03-18 17:04 UTC | newest]

Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-03-16 12:05 [PATCH v5 00/19] per project build directory Ed Bartosh
2016-03-16 12:05 ` [PATCH v5 01/19] toaster: don't start bitbake server Ed Bartosh
2016-03-16 12:05 ` [PATCH v5 02/19] toaster: get rid of noui option Ed Bartosh
2016-03-16 12:05 ` [PATCH v5 03/19] toaster: set BITBAKE_UI environment variable Ed Bartosh
2016-03-16 12:05 ` [PATCH v5 04/19] toasterui: add brbe parameter to buildinfohelper Ed Bartosh
2016-03-16 12:05 ` [PATCH v5 05/19] uievent: improve BBUIEventQueue code Ed Bartosh
2016-03-16 12:05 ` [PATCH v5 06/19] buildinfohelper: improve handling of providermap Ed Bartosh
2016-03-16 12:05 ` [PATCH v5 07/19] toasterui: fix brbe reporting Ed Bartosh
2016-03-16 12:05 ` [PATCH v5 08/19] toaster: remove startBBServer API Ed Bartosh
2016-03-16 12:05 ` [PATCH v5 09/19] toaster: remove release API Ed Bartosh
2016-03-16 12:05 ` [PATCH v5 10/19] toaster: add brbe parameter to triggerBuild Ed Bartosh
2016-03-16 12:05 ` [PATCH v5 11/19] toaster: modified setLayers API Ed Bartosh
2016-03-16 12:05 ` [PATCH v5 12/19] toaster: reimplement triggerBuild Ed Bartosh
2016-03-16 12:05 ` [PATCH v5 13/19] toaster: add new parameter to _shellcmd Ed Bartosh
2016-03-16 12:05 ` [PATCH v5 14/19] toaster: stop bitbake server after the build Ed Bartosh
2016-03-16 12:05 ` [PATCH v5 15/19] toaster: update conf/local.conf Ed Bartosh
2016-03-16 12:05 ` [PATCH v5 16/19] toaster: fix jethro build Ed Bartosh
2016-03-16 12:05 ` [PATCH v5 17/19] toaster: use bash explicitly Ed Bartosh
2016-03-16 12:05 ` [PATCH v5 18/19] toasterui: detect build run start correctly on Jethro Ed Bartosh
2016-03-16 12:05 ` [PATCH v5 19/19] toasterui: shutdown on BuildCompleted event Ed Bartosh
2016-03-17 17:20 ` [PATCH v5 00/19] per project build directory Barros Pena, Belen
2016-03-17 16:22   ` Ed Bartosh
2016-03-18 17:03     ` Barros Pena, Belen

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.