[PATCH 0/7] Fix for #8918: Change the way we handle queued builds

[PATCH 0/7] Fix for #8918: Change the way we handle queued builds

[Ed Bartosh]

Hi,

This patchset introduced a bit lighter way of handling builds.
Instead of polling database every second we process builds only when runbuilds
receives SIGUSR1. The signal is sent when build is either scheduled, canceled or
finished. A little bit of code cleanup is also included into this patchset.

Test instructions:
 - Create multiple builds in UI. Check if they'll be built sequentially
 - Create multiple builds in UI, cancel one. Check if the next one starts building.
 - Repeat first two test cases when command line build is run. Only one UI build
   should be run in parallel with command line build.
 - Test any other combinations of running and cancelling builds to make sure
   there are no regressions.

The following changes since commit 75927f8f6809b9b4696fc438b240bc7a3db561e2:

  toaster: don't kill all runserver processes (2016-09-05 15:32:25 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/toaster/nopoll-queued-builds-8918
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/nopoll-queued-builds-8918

Ed Bartosh (7):
  toaster: implement signal_runbuilds function
  toaster: notify runbuilds when build status changes
  runbuilds: process builds on SIGUSR1
  runbuilds: process builds on start
  runbuilds: code cleanup - whitespaces, long lines
  runbuilds: code cleanup - remove unused imports
  toaster: unlock BuildEnvirnoment when build is done

 bitbake/lib/bb/ui/buildinfohelper.py               | 12 ++++--
 .../bldcontrol/management/commands/runbuilds.py    | 49 ++++++++++------------
 bitbake/lib/toaster/orm/models.py                  | 10 ++++-
 3 files changed, 41 insertions(+), 30 deletions(-)

--
Regards,
Ed

[PATCH 1/7] toaster: implement signal_runbuilds function

[Ed Bartosh]

This function reads pid of runbuilds process from
BUILDDIR/.runbuilds.pid and sends SIGUSR1 to it. signal_runbuilds
function will be used in Toaster code to notify runbuilds when
build is scheduled, finished or cancelled.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/orm/models.py | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/toaster/orm/models.py b/bitbake/lib/toaster/orm/models.py
index 8ee0c87..38d014a 100644
--- a/bitbake/lib/toaster/orm/models.py
+++ b/bitbake/lib/toaster/orm/models.py
@@ -33,9 +33,10 @@ from django.conf import settings
 import django.db.models.signals
 
 import sys
-import os.path
+import os
 import re
 import itertools
+from signal import SIGUSR1
 
 import logging
 logger = logging.getLogger("toaster")
@@ -1736,6 +1737,11 @@ def invalidate_cache(**kwargs):
     except Exception as e:
       logger.warning("Problem with cache backend: Failed to clear cache: %s" % e)
 
+def signal_runbuilds():
+    """Send SIGUSR1 to runbuilds process"""
+    with open(os.path.join(os.getenv('BUILDDIR'), '.runbuilds.pid')) as pidf:
+        os.kill(int(pidf.read()), SIGUSR1)
+
 django.db.models.signals.post_save.connect(invalidate_cache)
 django.db.models.signals.post_delete.connect(invalidate_cache)
 django.db.models.signals.m2m_changed.connect(invalidate_cache)
-- 
2.1.4

[PATCH 2/7] toaster: notify runbuilds when build status changes

[Ed Bartosh]

Called signal_runbuilds API when build is scheduled, cancelled or
finished to notify runbuilds process about builds status change.

[YOCTO #8918]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py | 4 ++++
 bitbake/lib/toaster/orm/models.py    | 2 ++
 2 files changed, 6 insertions(+)

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index 96166dc..c93ee94 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -43,6 +43,7 @@ from orm.models import Package, Package_File, Target_Installed_Package, Target_F
 from orm.models import Task_Dependency, Package_Dependency
 from orm.models import Recipe_Dependency, Provides
 from orm.models import Project, CustomImagePackage, CustomImageRecipe
+from orm.models import signal_runbuilds
 
 from bldcontrol.models import BuildEnvironment, BuildRequest
 
@@ -234,6 +235,7 @@ class ORMWrapper(object):
         build.completed_on = timezone.now()
         build.outcome = outcome
         build.save()
+        signal_runbuilds()
 
     def update_target_set_license_manifest(self, target, license_manifest_path):
         target.license_manifest_path = license_manifest_path
@@ -1354,6 +1356,7 @@ class BuildInfoHelper(object):
         self._ensure_build()
         self.internal_state['build'].outcome = Build.CANCELLED
         self.internal_state['build'].save()
+        signal_runbuilds()
 
     def store_dependency_information(self, event):
         assert '_depgraph' in vars(event)
@@ -1540,6 +1543,7 @@ class BuildInfoHelper(object):
         else:
             br.state = BuildRequest.REQ_FAILED
         br.save()
+        signal_runbuilds()
 
     def store_log_error(self, text):
         mockevent = MockEvent()
diff --git a/bitbake/lib/toaster/orm/models.py b/bitbake/lib/toaster/orm/models.py
index 38d014a..a7de57c 100644
--- a/bitbake/lib/toaster/orm/models.py
+++ b/bitbake/lib/toaster/orm/models.py
@@ -377,6 +377,8 @@ class Project(models.Model):
             except ProjectVariable.DoesNotExist:
                 pass
             br.save()
+            signal_runbuilds()
+
         except Exception:
             # revert the build request creation since we're not done cleanly
             br.delete()
-- 
2.1.4

[PATCH 3/7] runbuilds: process builds on SIGUSR1

[Ed Bartosh]

Run main processing function 'runbuild' only if SIGUSR1 is
received. This signal is sent by Toaster when build status
is changed (either started, cancelled or finished).

This should stop continuous database polling as run_builds function
will be called only when needed, i.e. after build status is changed.

[YOCTO #8918]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
index a703770..61a520c 100644
--- a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
+++ b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
@@ -11,9 +11,9 @@ from orm.models import Build, ToasterSetting, LogMessage, Target
 
 import os
 import logging
-import time
 import sys
 import traceback
+import signal
 
 logger = logging.getLogger("toaster")
 
@@ -175,6 +175,8 @@ class Command(NoArgsCommand):
             logger.warn("runbuilds: schedule exception %s" % str(e))
 
     def handle_noargs(self, **options):
+        signal.signal(signal.SIGUSR1, lambda sig, frame: None)
+
         while True:
+            signal.pause()
             self.runbuild()
-            time.sleep(1)
-- 
2.1.4

[PATCH 4/7] runbuilds: process builds on start

[Ed Bartosh]

If Toaster is stopped incorrectly there could be some
build requests and builds in incorrect state left from the previous run.
Running main processing function on start should take care of those.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
index 61a520c..7df81dc 100644
--- a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
+++ b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
@@ -175,6 +175,8 @@ class Command(NoArgsCommand):
             logger.warn("runbuilds: schedule exception %s" % str(e))
 
     def handle_noargs(self, **options):
+        self.runbuild()
+
         signal.signal(signal.SIGUSR1, lambda sig, frame: None)
 
         while True:
-- 
2.1.4

[PATCH 5/7] runbuilds: code cleanup - whitespaces, long lines

[Ed Bartosh]

Fixed following pylint warnings:
 C0330(bad-continuation): Wrong hanging indentation before block.
 C0326(bad-whitespace): No space allowed around keyword argument assignment
 C0326(bad-whitespace): Exactly one space required before assignment
 C0301(line-too-long): Line too long

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 .../bldcontrol/management/commands/runbuilds.py    | 34 ++++++++++------------
 1 file changed, 15 insertions(+), 19 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
index 7df81dc..3421726 100644
--- a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
+++ b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
@@ -18,14 +18,13 @@ import signal
 logger = logging.getLogger("toaster")
 
 class Command(NoArgsCommand):
-    args    = ""
-    help    = "Schedules and executes build requests as possible."
-    "Does not return (interrupt with Ctrl-C)"
-
+    args = ""
+    help = "Schedules and executes build requests as possible. "\
+           "Does not return (interrupt with Ctrl-C)"
 
     @transaction.atomic
     def _selectBuildEnvironment(self):
-        bec = getBuildEnvironmentController(lock = BuildEnvironment.LOCK_FREE)
+        bec = getBuildEnvironmentController(lock=BuildEnvironment.LOCK_FREE)
         bec.be.lock = BuildEnvironment.LOCK_LOCK
         bec.be.save()
         return bec
@@ -75,17 +74,15 @@ class Command(NoArgsCommand):
             else:
                 errmsg = str(e)
 
-            BRError.objects.create(req = br,
-                    errtype = str(type(e)),
-                    errmsg = errmsg,
-                    traceback = traceback.format_exc())
+            BRError.objects.create(req=br, errtype=str(type(e)), errmsg=errmsg,
+                                   traceback=traceback.format_exc())
             br.state = BuildRequest.REQ_FAILED
             br.save()
             bec.be.lock = BuildEnvironment.LOCK_FREE
             bec.be.save()
 
     def archive(self):
-        for br in BuildRequest.objects.filter(state = BuildRequest.REQ_ARCHIVE):
+        for br in BuildRequest.objects.filter(state=BuildRequest.REQ_ARCHIVE):
             if br.build == None:
                 br.state = BuildRequest.REQ_FAILED
             else:
@@ -102,14 +99,13 @@ class Command(NoArgsCommand):
                                        BuildRequest.REQ_COMPLETED,
                                        BuildRequest.REQ_CANCELLING]) &
             Q(lock=BuildEnvironment.LOCK_LOCK) &
-            Q(updated__lt=timezone.now() - timedelta(seconds = 30))
+            Q(updated__lt=timezone.now() - timedelta(seconds=30))
         ).update(lock=BuildEnvironment.LOCK_FREE)
 
 
         # update all Builds that were in progress and failed to start
-        for br in BuildRequest.objects.filter(
-            state=BuildRequest.REQ_FAILED,
-            build__outcome=Build.IN_PROGRESS):
+        for br in BuildRequest.objects.filter(state=BuildRequest.REQ_FAILED,
+                                              build__outcome=Build.IN_PROGRESS):
             # transpose the launch errors in ToasterExceptions
             br.build.outcome = Build.FAILED
             for brerror in br.brerror_set.all():
@@ -126,7 +122,7 @@ class Command(NoArgsCommand):
 
 
         # update all BuildRequests without a build created
-        for br in BuildRequest.objects.filter(build = None):
+        for br in BuildRequest.objects.filter(build=None):
             br.build = Build.objects.create(project=br.project,
                                             completed_on=br.updated,
                                             started_on=br.created)
@@ -151,10 +147,10 @@ class Command(NoArgsCommand):
 
         # Make sure the LOCK is removed for builds which have been fully
         # cancelled
-        for br in BuildRequest.objects.filter(
-            Q(build__outcome=Build.CANCELLED) &
-            Q(state=BuildRequest.REQ_CANCELLING) &
-            ~Q(environment=None)):
+        for br in BuildRequest.objects.filter(\
+                      Q(build__outcome=Build.CANCELLED) &
+                      Q(state=BuildRequest.REQ_CANCELLING) &
+                      ~Q(environment=None)):
             br.environment.lock = BuildEnvironment.LOCK_FREE
             br.environment.save()
 
-- 
2.1.4

[PATCH 6/7] runbuilds: code cleanup - remove unused imports

[Ed Bartosh]

Fixed pylint warning:  W0611(unused-import): Unused import

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
index 3421726..7f7a5a9 100644
--- a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
+++ b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
@@ -1,17 +1,14 @@
-from django.core.management.base import NoArgsCommand, CommandError
+from django.core.management.base import NoArgsCommand
 from django.db import transaction
 from django.db.models import Q
 
 from bldcontrol.bbcontroller import getBuildEnvironmentController
-from bldcontrol.bbcontroller import ShellCmdException, BuildSetupException
 from bldcontrol.models import BuildRequest, BuildEnvironment
 from bldcontrol.models import BRError, BRVariable
 
-from orm.models import Build, ToasterSetting, LogMessage, Target
+from orm.models import Build, LogMessage, Target
 
-import os
 import logging
-import sys
 import traceback
 import signal
 
-- 
2.1.4

[PATCH 7/7] toaster: unlock BuildEnvirnoment when build is done

[Ed Bartosh]

There is no need to lock build environment before changing
build status as this operation is very fast. However, there
is a need to unlock it after changing build status.

Explicitly unlocked BuildEnvironment after build reaches
final status SUCCEEDED, FAILED or CANCELLED. This should
allow runbuilds process to pickup next build faster.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index c93ee94..2246960 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -1524,9 +1524,7 @@ class BuildInfoHelper(object):
             return
 
         br_id, be_id = self.brbe.split(":")
-        be = BuildEnvironment.objects.get(pk = be_id)
-        be.lock = BuildEnvironment.LOCK_LOCK
-        be.save()
+
         br = BuildRequest.objects.get(pk = br_id)
 
         # if we're 'done' because we got cancelled update the build outcome
@@ -1543,6 +1541,10 @@ class BuildInfoHelper(object):
         else:
             br.state = BuildRequest.REQ_FAILED
         br.save()
+
+        be = BuildEnvironment.objects.get(pk = be_id)
+        be.lock = BuildEnvironment.LOCK_FREE
+        be.save()
         signal_runbuilds()
 
     def store_log_error(self, text):
-- 
2.1.4

Re: [PATCH 7/7] toaster: unlock BuildEnvirnoment when build is done

[Michael Wood]

Thanks - send to bitbake-devel and applied to toaster-next

On 13/09/16 14:21, Ed Bartosh wrote:
> There is no need to lock build environment before changing
> build status as this operation is very fast. However, there
> is a need to unlock it after changing build status.
>
> Explicitly unlocked BuildEnvironment after build reaches
> final status SUCCEEDED, FAILED or CANCELLED. This should
> allow runbuilds process to pickup next build faster.
>
> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> ---
>   bitbake/lib/bb/ui/buildinfohelper.py | 8 +++++---
>   1 file changed, 5 insertions(+), 3 deletions(-)
>
> diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
> index c93ee94..2246960 100644
> --- a/bitbake/lib/bb/ui/buildinfohelper.py
> +++ b/bitbake/lib/bb/ui/buildinfohelper.py
> @@ -1524,9 +1524,7 @@ class BuildInfoHelper(object):
>               return
>   
>           br_id, be_id = self.brbe.split(":")
> -        be = BuildEnvironment.objects.get(pk = be_id)
> -        be.lock = BuildEnvironment.LOCK_LOCK
> -        be.save()
> +
>           br = BuildRequest.objects.get(pk = br_id)
>   
>           # if we're 'done' because we got cancelled update the build outcome
> @@ -1543,6 +1541,10 @@ class BuildInfoHelper(object):
>           else:
>               br.state = BuildRequest.REQ_FAILED
>           br.save()
> +
> +        be = BuildEnvironment.objects.get(pk = be_id)
> +        be.lock = BuildEnvironment.LOCK_FREE
> +        be.save()
>           signal_runbuilds()
>   
>       def store_log_error(self, text):