[AB PATCH 00/22] Consolidated patch set.

[AB PATCH 00/22] Consolidated patch set.

[Elizabeth Flanagan]

This patch set is currently running on dev-ab-master.pdx.yoctodev.org:8010 for
testing purposes. 

The main bits of functionality here:
- fixes up some of the last issues with yoctogit.py that have cropped up.
- fixes how we use patchExists within yoctogit.py
- reimpliments some old release functionality that we lost with the UI changes.
- Cleans up some yoctogit.py code
- supports the name change of core-image-basic
- gives us some logging with the buildslave-janitor
- adds a new buildset to run builds on target

The following changes since commit ff48d4619a6910aa37473fd323d1865d8a32eb0c:

  BuildSet.py: Indent error causes triggers with no nowait to fail (2014-03-19 08:47:16 -0700)

are available in the git repository at:

  git://git.yoctoproject.org/yocto-autobuilder eflanagan/yocto-autobuilder-dev
  http://git.yoctoproject.org/cgit.cgi//log/?h=eflanagan/yocto-autobuilder-dev

Beth Flanagan (10):
  yoctogit.py: Use self.srcdir, not local srcdir construct.
  yoctogit.py: movecopy should only move 'build'
  yoctogit.py: DRYing up some code
  yoctogit.py: Fix timeouts.
  yoctogit.py: Set workdir for the class
  yoctogit.py: Set workdir within dovccmd
  yoctogit.py: change _srccopy fromdir to it's own construction
  PublishArtifacts.py: Exclude genericx86-64 pre-1.5
  SetDest.py: Use new UI variables.
  yoctogit.py: sync on mv

Elizabeth Flanagan (9):
  yoctogit.py: Fix cp construction from source to build
  BuildImages.py: core-image-basic name change support
  yoctogit.py: movecopy doesn't care if it's updateable
  yoctogit.py: movecopy needs an incremental callback
  CheckOutLayers.py: Remove unset Props
  yoctogit.py: pathExists returns a defered value.
  CheckOutLayers.py: Remove srcdir
  yoctogit.py: Stop passing in srcdir.
  yoctogit.py: Check for mirror on incremental builds

Richard Purdie (2):
  yocto-start-autobuilder: Make sure janitor logging goes to a log file
  buildslave-janitor: Add in try/except handling to make calls robust
    against failure

Stefan Stanacar (1):
  buildsets: add a new buildset to run builds on target

 bin/buildslave-janitor                             |  15 +-
 .../nightly-qa-targetbuilds.conf                   |  26 +++
 .../site-packages/autobuilder/BuildSet.py          |   2 -
 .../autobuilder/buildsteps/BuildImages.py          |   9 ++
 .../autobuilder/buildsteps/CheckOutLayers.py       |   9 +-
 .../autobuilder/buildsteps/PublishArtifacts.py     |  15 +-
 .../autobuilder/buildsteps/SetDest.py              |   7 +-
 .../buildbot/steps/source/yoctogit.py              | 177 ++++++++++-----------
 yocto-start-autobuilder                            |   2 +-
 9 files changed, 145 insertions(+), 117 deletions(-)
 create mode 100644 buildset-config.master/nightly-qa-targetbuilds.conf

-- 
1.8.1.2

[AB PATCH 01/22] buildsets: add a new buildset to run builds on target

[Elizabeth Flanagan]

From: Stefan Stanacar <stefanx.stanacar@intel.com>

Builds core-image-sato-sdk for each qemu arch and
runs tests building sudoku/cvs/iptables.
These do take a while, so they shouldn't be added to nightly!

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 .../nightly-qa-targetbuilds.conf                   | 26 ++++++++++++++++++++++
 1 file changed, 26 insertions(+)
 create mode 100644 buildset-config.master/nightly-qa-targetbuilds.conf

diff --git a/buildset-config.master/nightly-qa-targetbuilds.conf b/buildset-config.master/nightly-qa-targetbuilds.conf
new file mode 100644
index 0000000..a95db78
--- /dev/null
+++ b/buildset-config.master/nightly-qa-targetbuilds.conf
@@ -0,0 +1,26 @@
+[nightly-qa-targetbuilds]
+builders: 'builder1'
+repos: [{'poky':
+            {'repourl':'git://git.yoctoproject.org/poky',
+             'layerversion':{'core':'meta', 'yoctobsp':'meta-yocto-bsp'},
+             'branch':'master'}}]
+steps: [{'SetDest':{}},
+        {'CheckOutLayers': {}},
+        {'RunPreamble': {}},
+        {'GetDistroVersion' : {'distro': 'poky'}},
+        {'CreateAutoConf': {'machine': 'qemux86-64', 'SDKMACHINE' : 'x86_64', 'distro': 'poky'}},
+        {'CreateBBLayersConf': {'buildprovider' : 'yocto'}},
+        {'BuildImages': {'images': 'core-image-sato-sdk'}},
+        {'RunSanityTests': {'images': 'core-image-sato-sdk', 'suites' : 'ping ssh buildsudoku buildcvs buildiptables'}},
+        {'CreateAutoConf': {'machine': 'qemux86', 'SDKMACHINE' : 'x86_64', 'distro': 'poky'}},
+        {'BuildImages': {'images': 'core-image-sato-sdk'}},
+        {'RunSanityTests': {'images': 'core-image-sato-sdk', 'suites' : 'ping ssh buildsudoku buildcvs buildiptables'}},
+        {'CreateAutoConf': {'machine': 'qemuarm', 'SDKMACHINE' : 'x86_64', 'distro': 'poky'}},
+        {'BuildImages': {'images': 'core-image-sato-sdk'}},
+        {'RunSanityTests': {'images': 'core-image-sato-sdk', 'suites' : 'ping ssh buildsudoku buildcvs buildiptables'}},
+        {'CreateAutoConf': {'machine': 'qemuppc', 'SDKMACHINE' : 'x86_64', 'distro': 'poky'}},
+        {'BuildImages': {'images': 'core-image-sato-sdk'}},
+        {'RunSanityTests': {'images': 'core-image-sato-sdk', 'suites' : 'ping ssh buildsudoku buildcvs buildiptables'}},
+        {'CreateAutoConf': {'machine': 'qemumips', 'SDKMACHINE' : 'x86_64', 'distro': 'poky'}},
+        {'BuildImages': {'images': 'core-image-sato-sdk'}},
+        {'RunSanityTests': {'images': 'core-image-sato-sdk', 'suites' : 'ping ssh buildsudoku buildcvs buildiptables'}}]
-- 
1.8.1.2

[AB PATCH 02/22] yoctogit.py: Fix cp construction from source to build

[Elizabeth Flanagan]

This commit removes the mkdir from movecopy as it is not needed.
It also changes the workdir in barecopy and movecopy to
the source dir, as we can't alwas be sure that the destination
dir even exists.

This also ensures that we are in fact constructing our cp string
correctly, by striping any trailing "/" from self.repourl

As repourl is used to construct the cp from source to build, that
trailing slash, if it exists, causes the directory structure to
not be correct (e.g. cp -R ./foo/ ./bar and cp -R ./foo ./bar
don't do the same thing, obviously)

Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
---
 .../buildbot/steps/source/yoctogit.py              | 35 +++++++---------------
 1 file changed, 11 insertions(+), 24 deletions(-)

diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index 26599e7..487e286 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -121,7 +121,7 @@ class YoctoGit(Source):
         self.branch    = branch
         self.method    = method
         self.prog  = progress
-        self.repourl   = repourl
+        self.repourl   = repourl.rstrip('/')
         self.retryFetch = retryFetch
         self.submodules = submodules
         self.shallow   = shallow
@@ -255,13 +255,11 @@ class YoctoGit(Source):
         if "poky" in self.layername or \
            "oecore" in self.layername or \
            "eclipse" in self.layername:
-            todir = "."
-            self.workdir = "build"
+            todir = "build"
+            self.workdir = 'source/'+self.repourl
         else:
-            todir = "./" + self.layername
-            self.workdir = "build/" + self.layername
-
-
+            todir = "build/" + self.layername
+            self.workdir = 'source/'+self.repourl
         d.addCallback(lambda _: self.incremental())
         def copy(_):
             cmd = buildstep.RemoteCommand('cpdir',
@@ -278,7 +276,7 @@ class YoctoGit(Source):
             d.addCallback(lambda _: evaluateCommand(cmd))
             return d
         d.addCallback(copy)
-        
+       
         def resetWorkdir(_):
             if "poky" in self.layername or \
                "oecore" in self.layername or \
@@ -316,26 +314,15 @@ class YoctoGit(Source):
                 log.msg("Source step failed while running command %s" % cmd)
                 raise buildstep.BuildStepFailed()
         d.addCallback(lambda _: evaluateCommand(cmd))
-
-        if self.layername == "poky" or self.layername == "oecore":
+        if "poky" in self.layername or \
+           "oecore" in self.layername or \
+           "eclipse" in self.layername:
             todir = "build"
+            self.workdir = 'source/'+self.repourl.rstrip('/')
         else:
             todir = "build/" + self.layername
+            self.workdir = 'source/'+self.repourl
 
-        def mkdir(_):
-            cmd = buildstep.RemoteCommand('mkdir', {'dir': 'build',
-                                                    'logEnviron': self.logEnviron,
-                                                    'timeout': self.timeout,})
-            cmd.useLog(self.stdio_log, False)
-            d = self.runCommand(cmd)
-            def evaluateCommand(cmd):
-                if cmd.didFail():
-                    log.msg("Source step failed while running command %s" % cmd)
-                    raise buildstep.BuildStepFailed()
-            d.addCallback(lambda _: evaluateCommand(cmd))
-            return d
-        d.addCallback(mkdir)
-        d.addCallback(lambda _: self.incremental())
         def copy(_):
             cmd = buildstep.RemoteCommand('cpdir',
                                           {'fromdir': srcdir,
-- 
1.8.1.2

[AB PATCH 03/22] BuildImages.py: core-image-basic name change support

[Elizabeth Flanagan]

Supports http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=b7f1cca517bbd4191828c6bae32e0c5041f1ff19

Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
---
 .../site-packages/autobuilder/buildsteps/BuildImages.py          | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py b/lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py
index 2dc72dd..5f1b549 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/BuildImages.py
@@ -34,7 +34,16 @@ class BuildImages(ShellCommand):
 
     def start(self):
         self.layerversion_yoctobsp = self.getProperty("layerversion_yoctobsp")
+        self.layerversion_core = self.getProperty("layerversion_core")
         self.machine = self.getProperty("MACHINE")
+        # core-image-basic rename
+        # See: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=b7f1cca517bbd4191828c6bae32e0c5041f1ff19
+        # I hate making people change their configs, so support both.
+        if self.layerversion_core < "4":
+            self.images=self.images.replace("core-image-full-cmdline", "core-image-basic")
+        elif self.layerversion_core > "3":
+            self.images=self.images.replace("core-image-basic", "core-image-full-cmdline")
+
         if self.layerversion_yoctobsp is not None and int(self.layerversion_yoctobsp) < 2 and self.machine is not None and self.machine == "genericx86-64":
             self.command = "echo 'Skipping Step.'"
         else:
-- 
1.8.1.2

[AB PATCH 04/22] yoctogit.py: movecopy doesn't care if it's updateable

[Elizabeth Flanagan]

movecopy really shouldn't care if it's updatable or not. It should
have an incremental callback in it to take care of that (it doesn't
though, which is part of the problem!)

Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
---
 .../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py          | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index 487e286..25e5308 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -184,7 +184,7 @@ class YoctoGit(Source):
         elif self.method == 'barecopy':
             yield self.barecopy()
             return
-        elif updatable and self.method == 'movecopy':
+        elif self.method == 'movecopy':
             yield self.movecopy()
             return
 
-- 
1.8.1.2

[AB PATCH 05/22] yoctogit.py: movecopy needs an incremental callback

[Elizabeth Flanagan]

We need to do an incremental before we copy or else, we're
essentially copying nothing.

Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
---
 .../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py           | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index 25e5308..080f5b2 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -322,6 +322,7 @@ class YoctoGit(Source):
         else:
             todir = "build/" + self.layername
             self.workdir = 'source/'+self.repourl
+        d.addCallback(lambda _: self.incremental())
 
         def copy(_):
             cmd = buildstep.RemoteCommand('cpdir',
-- 
1.8.1.2

[AB PATCH 06/22] CheckOutLayers.py: Remove unset Props

[Elizabeth Flanagan]

We no longer set these trigger* props, so let's remove them.

Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
---
 lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py b/lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py
index 90b90c5..5e95189 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py
@@ -82,10 +82,6 @@ class CheckOutLayers(YoctoGit):
            self.getProperty('commit_resolvedhead_'+self.layername) is not None:
             log.msg("Using Resolved HEAD")
             self.commit = self.getProperty('commit_resolvedhead_'+self.layername)
-        elif self.getProperty('commit_trigger_'+self.layername+"_"+self.scheduler):
-            self.commit = self.getProperty('commit_trigger_'+self.layername+"_"+self.scheduler)
-        elif self.getProperty('commit_triggerer_'+self.layername+"_"+self.scheduler) :
-            self.commit = self.getProperty('commit_triggerer_'+self.layername+"_"+self.scheduler)
         elif self.getProperty('commit_'+self.layername):
             self.commit = self.getProperty('commit_'+self.layername)
         else:
-- 
1.8.1.2

[AB PATCH 07/22] yoctogit.py: pathExists returns a defered value.

[Elizabeth Flanagan]

We can't use pathExists this way to figure out if the path actually
exists as it will always evalutate to true as it returns a generator
object and not T/F.

In order to actually figure out if the mirrordir + repourl exists,
we need to call a function that returns the deferred generator
object outside of any of fetch/movecopy/barecopy functions (which
also return defereds and you can't yield a defered in a defered)

So, yield the generator object at the beginning, and save everyone
the headache.

Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
---
 .../buildbot/steps/source/yoctogit.py                     | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index 080f5b2..4c89ceb 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -177,6 +177,7 @@ class YoctoGit(Source):
     @defer.inlineCallbacks
     def full(self):
         updatable = yield self._sourcedirIsUpdatable()
+        self.mirrorexists = yield self._mirrorExists()
         
         if self.method == 'clobber':
             yield self.clobber()
@@ -248,9 +249,9 @@ class YoctoGit(Source):
         d.addCallback(lambda _: evaluateCommand(cmd))
         if self.mirrordir:
             mirror = self.mirrordir + "/" + self.repourl.replace("git://", "") + "/"
-            if self.pathExists(mirror):
+            if self.mirrorexists:
                 srcdir=mirror
-            else:
+            else: 
                 srcdir='source/'+self.repourl
         if "poky" in self.layername or \
            "oecore" in self.layername or \
@@ -493,8 +494,10 @@ class YoctoGit(Source):
         mirror = None
         if self.mirrordir:
             mirror = self.mirrordir + "/" + self.repourl.replace("git://", "") + "/"
-            if self.pathExists(mirror):
-                  command = ['clone'] + ["-s", "-n"] + [mirror, '.']
+            if self.mirrorexists:
+                command = ['clone'] + ["-s", "-n"] + [mirror, '.']
+            else: 
+                command = ['clone'] + ["-s", "-n"] + [self.repourl, '.']
 
         #Fix references
         if self.prog:
@@ -560,6 +563,10 @@ class YoctoGit(Source):
     def _sourcedirIsUpdatable(self):
         return self.pathExists(self.build.path_module.join(self.workdir, '.git'))
 
+    def _mirrorExists(self):
+        mirror = self.mirrordir + "/" + self.repourl.replace("git://", "") + "/"
+        return self.pathExists(self.build.path_module.join(mirror, '.git'))
+
     def _updateSubmodule(self, _):
         if self.submodules:
             return self._dovccmd(['submodule', 'update', '--recursive'])
-- 
1.8.1.2

[AB PATCH 08/22] CheckOutLayers.py: Remove srcdir

[Elizabeth Flanagan]

We're removing srcdir from yoctogit.py. It causes way too many
issues.

Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
---
 lib/python2.7/site-packages/autobuilder/BuildSet.py                  | 2 --
 lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py | 5 ++---
 2 files changed, 2 insertions(+), 5 deletions(-)

diff --git a/lib/python2.7/site-packages/autobuilder/BuildSet.py b/lib/python2.7/site-packages/autobuilder/BuildSet.py
index ab7b0bb..4bcda66 100644
--- a/lib/python2.7/site-packages/autobuilder/BuildSet.py
+++ b/lib/python2.7/site-packages/autobuilder/BuildSet.py
@@ -73,7 +73,6 @@ class BuildSet():
                             mirrordir=None
                             method = 'clobber'
                             mode='full'
-                            srcdir = 'source'
                             if str(os.environ.get('OPTIMIZED_GIT_CLONE')) == "True":
                                 if 'poky' in layername or 'oecore' in layername or 'eclipse' in layername:
                                     method='movecopy'
@@ -93,7 +92,6 @@ class BuildSet():
                             func = getattr(m, step)
                             factoryFN(func(locals()['f'+name],
                                       layername=layername,
-                                      srcdir=srcdir,
                                       mode=mode,
                                       scheduler=name,
                                       method=method,
diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py b/lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py
index 5e95189..029b523 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/CheckOutLayers.py
@@ -35,12 +35,11 @@ from buildbot.process import buildstep
 
 class CheckOutLayers(YoctoGit):
     def __init__(self, factory, scheduler=None, layername=None, mode='full',  storedir=None, mirrordir=None,
-                method='clobber', submodules=False, shallow=False, srcdir='source', workdir='build',
+                method='clobber', submodules=False, shallow=False, workdir='build',
                 timeout=100000, progress=True, retryFetch=True, clobberOnFailure=False,
                 getDescription=True, argdict=None,
                  **kwargs):
         self.scheduler = scheduler
-        self.srcdir=srcdir
         self.workdir=workdir
         self.branch = ""
         self.commit = ""
@@ -63,7 +62,7 @@ class CheckOutLayers(YoctoGit):
         for k, v in argdict.iteritems():
             setattr(self, k, v)
         YoctoGit.__init__(self, repourl=self.repourl, branch=self.branch, mode=self.mode,
-                 method=self.method, storedir=self.storedir, mirrordir=mirrordir, submodules=self.submodules, srcdir=self.srcdir+"/"+self.repourl,
+                 method=self.method, storedir=self.storedir, mirrordir=mirrordir, submodules=self.submodules,
                  workdir=self.workdir, shallow=self.shallow, progress=self.progress,
                  retryFetch=self.retryFetch, clobberOnFailure=self.clobberOnFailure,
                  getDescription=self.getDescription, layername=self.layername, **kwargs)
-- 
1.8.1.2

[AB PATCH 09/22] yoctogit.py: Stop passing in srcdir.

[Elizabeth Flanagan]

Don't pass in srcdir. Set it depending on mirrordir/method as a
class level property.

Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
---
 .../buildbot/steps/source/yoctogit.py                | 20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)

diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index 4c89ceb..62e3a02 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -63,7 +63,7 @@ class YoctoGit(Source):
     def __init__(self, repourl=None, branch='HEAD', mode='incremental',
                  method=None, submodules=False, shallow=False, progress=False,
                  retryFetch=False, clobberOnFailure=False, getDescription=False, 
-                 workdir='build', layername=None, srcdir='sources', storedir="/tmp/junk", mirrordir=None,
+                 workdir='build', layername=None, storedir="/tmp/junk", mirrordir=None,
                  config=None, **kwargs):
         """
         @type  repourl: string
@@ -98,9 +98,6 @@ class YoctoGit(Source):
         @type  getDescription: boolean or dict
         @param getDescription: Use 'git describe' to describe the fetched revision
 
-        @type srcdir: string
-        @param srcdir: Used to pass in where git clones should occur
-
         @type workdir: string
         @param workdir: Used to indicate the destination of the copy of srcdir
 
@@ -125,7 +122,6 @@ class YoctoGit(Source):
         self.retryFetch = retryFetch
         self.submodules = submodules
         self.shallow   = shallow
-        self.srcdir  = srcdir
         self.workdir = workdir
         self.storedir = storedir
         self.mirrordir = mirrordir
@@ -135,6 +131,20 @@ class YoctoGit(Source):
         self.mode = mode
         self.getDescription = getDescription
         self.config = config
+        # We no longer pass srcdir in. There are way too many ways to hang
+        # yourself by doing that.
+        if self.mirrordir:
+            if self.method in ['barecopy', 'movecopy']:
+                self.srcdir='source/'+self.repourl
+            else:
+                if "poky" in self.layername or \
+                   "oecore" in self.layername or \
+                   "eclipse" in self.layername:
+                    self.srcdir = "build"
+                else:
+                    self.srcdir = 'build/'+self.layername
+        else:
+            self.srcdir = 'source'
         Source.__init__(self, **kwargs)
 
         if self.mode not in ['incremental', 'full']:
-- 
1.8.1.2

[AB PATCH 10/22] yoctogit.py: Check for mirror on incremental builds

[Elizabeth Flanagan]

We're assuming people are getting to incremental from full.
This is a bad assumption. So, check for mirror in both.

Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
---
 .../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py           | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index 62e3a02..fcbc2cb 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -212,6 +212,7 @@ class YoctoGit(Source):
     @defer.inlineCallbacks
     def incremental(self):
         updatable = yield self._sourcedirIsUpdatable()
+        self.mirrorexists = yield self._mirrorExists()
         
         # if not updateable, do a full checkout
         if not updatable:
-- 
1.8.1.2

[AB PATCH 11/22] yoctogit.py: Use self.srcdir, not local srcdir construct.

[Elizabeth Flanagan]

From: Beth Flanagan <elizabeth.flanagan@intel.com>

Since we're now constructing the objects srcdir property, there
is no need to use a local srcdir prop.

Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
---
 .../buildbot/steps/source/yoctogit.py                        | 12 ++----------
 1 file changed, 2 insertions(+), 10 deletions(-)

diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index fcbc2cb..1924612 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -258,12 +258,6 @@ class YoctoGit(Source):
                 log.msg("Source step failed while running command %s" % cmd)
                 raise buildstep.BuildStepFailed()
         d.addCallback(lambda _: evaluateCommand(cmd))
-        if self.mirrordir:
-            mirror = self.mirrordir + "/" + self.repourl.replace("git://", "") + "/"
-            if self.mirrorexists:
-                srcdir=mirror
-            else: 
-                srcdir='source/'+self.repourl
         if "poky" in self.layername or \
            "oecore" in self.layername or \
            "eclipse" in self.layername:
@@ -275,7 +269,7 @@ class YoctoGit(Source):
         d.addCallback(lambda _: self.incremental())
         def copy(_):
             cmd = buildstep.RemoteCommand('cpdir',
-                                          {'fromdir': srcdir,
+                                          {'fromdir': self.srcdir,
                                            'todir': todir,
                                            'logEnviron': self.logEnviron,
                                            'timeout': self.timeout,})
@@ -311,8 +305,6 @@ class YoctoGit(Source):
         # method probably won't work correctly crossing filesystems, so utilize 
         # movecopy with caution.
 
-        srcdir='source/'+self.repourl
-
         cmd = buildstep.RemoteCommand('mv', {'fromdir': self.workdir,
                                              'todir': self.storedir + '/' + self.getProperty("DEST") + '/' + 
                                                         str(int(time.time())) + '-'  + 
@@ -338,7 +330,7 @@ class YoctoGit(Source):
 
         def copy(_):
             cmd = buildstep.RemoteCommand('cpdir',
-                                          {'fromdir': srcdir,
+                                          {'fromdir': self.srcdir,
                                            'todir': todir,
                                            'logEnviron': self.logEnviron,
                                            'timeout': self.timeout,})
-- 
1.8.1.2

[AB PATCH 12/22] yoctogit.py: movecopy should only move 'build'

[Elizabeth Flanagan]

From: Beth Flanagan <elizabeth.flanagan@intel.com>

Pin what movecopy moves to be 'build' and not workdir. This keeps
people from doing stupid stuff like passing in a weird workdir.

Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
---
 .../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py        | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index 1924612..66bfb51 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -305,7 +305,7 @@ class YoctoGit(Source):
         # method probably won't work correctly crossing filesystems, so utilize 
         # movecopy with caution.
 
-        cmd = buildstep.RemoteCommand('mv', {'fromdir': self.workdir,
+        cmd = buildstep.RemoteCommand('mv', {'fromdir': 'build',
                                              'todir': self.storedir + '/' + self.getProperty("DEST") + '/' + 
                                                         str(int(time.time())) + '-'  + 
                                                         str(random.randrange(100, 100000, 2)) + 
@@ -322,10 +322,8 @@ class YoctoGit(Source):
            "oecore" in self.layername or \
            "eclipse" in self.layername:
             todir = "build"
-            self.workdir = 'source/'+self.repourl.rstrip('/')
         else:
             todir = "build/" + self.layername
-            self.workdir = 'source/'+self.repourl
         d.addCallback(lambda _: self.incremental())
 
         def copy(_):
-- 
1.8.1.2

[AB PATCH 13/22] yoctogit.py: DRYing up some code

[Elizabeth Flanagan]

From: Beth Flanagan <elizabeth.flanagan@intel.com>

Remove duplicate code from movecopy and barecopy and add it to
a new callback _srccopy.

Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
---
 .../buildbot/steps/source/yoctogit.py              | 73 +++++++++-------------
 1 file changed, 30 insertions(+), 43 deletions(-)

diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index 66bfb51..c5dbf05 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -194,9 +194,11 @@ class YoctoGit(Source):
             return
         elif self.method == 'barecopy':
             yield self.barecopy()
+            yield self._srccopy()
             return
         elif self.method == 'movecopy':
             yield self.movecopy()
+            yield self._srccopy()
             return
 
         if not updatable:
@@ -267,32 +269,6 @@ class YoctoGit(Source):
             todir = "build/" + self.layername
             self.workdir = 'source/'+self.repourl
         d.addCallback(lambda _: self.incremental())
-        def copy(_):
-            cmd = buildstep.RemoteCommand('cpdir',
-                                          {'fromdir': self.srcdir,
-                                           'todir': todir,
-                                           'logEnviron': self.logEnviron,
-                                           'timeout': self.timeout,})
-            cmd.useLog(self.stdio_log, False)
-            d = self.runCommand(cmd)
-            def evaluateCommand(cmd):
-                if cmd.didFail():
-                    log.msg("Source step failed while running command %s" % cmd)
-                    raise buildstep.BuildStepFailed()
-            d.addCallback(lambda _: evaluateCommand(cmd))
-            return d
-        d.addCallback(copy)
-       
-        def resetWorkdir(_):
-            if "poky" in self.layername or \
-               "oecore" in self.layername or \
-               "eclipse" in self.layername:
-                self.workdir = 'build'
-            else:
-                self.workdir = "build/" + self.layername
-            return 0
-
-        d.addCallback(resetWorkdir)
         return d
 
     def movecopy(self):
@@ -325,22 +301,35 @@ class YoctoGit(Source):
         else:
             todir = "build/" + self.layername
         d.addCallback(lambda _: self.incremental())
+        return d
 
-        def copy(_):
-            cmd = buildstep.RemoteCommand('cpdir',
-                                          {'fromdir': self.srcdir,
-                                           'todir': todir,
-                                           'logEnviron': self.logEnviron,
-                                           'timeout': self.timeout,})
-            cmd.useLog(self.stdio_log, False)
-            d = self.runCommand(cmd)
-            def evaluateCommand(cmd):
-                if cmd.didFail():
-                    log.msg("Source step failed while running command %s" % cmd)
-                    raise buildstep.BuildStepFailed()
-            d.addCallback(lambda _: evaluateCommand(cmd))
-            return d
-        d.addCallback(copy)
+    # Buildbot's standard copy has been removed from yoctogit. 
+    # We use _srccopy from either movecopy or barecopy as a callback
+    # this uses self.srcdir to copy to the build dir.
+    
+    def _srccopy(self):
+        if "poky" in self.layername or \
+           "oecore" in self.layername or \
+           "eclipse" in self.layername:
+            todir = "build"
+        else:
+            todir = "build/" + self.layername
+
+        cmd = buildstep.RemoteCommand('cpdir',
+                                      {'fromdir': self.srcdir,
+                                       'todir': todir,
+                                       'logEnviron': self.logEnviron,
+                                       'timeout': self.timeout,})
+        cmd.useLog(self.stdio_log, False)
+        d = self.runCommand(cmd)
+
+        def evaluateCommand(cmd):
+            if cmd.didFail():
+                log.msg("Source step failed while running command %s" % cmd)
+                raise buildstep.BuildStepFailed()
+
+        d.addCallback(lambda _: evaluateCommand(cmd))
+        return d
 
         def resetWorkdir(_):
             if "poky" in self.layername or \
@@ -353,8 +342,6 @@ class YoctoGit(Source):
         d.addCallback(resetWorkdir)
         return d
 
-    # Buildbot's standard copy has been removed from yoctogit. 
-
     def finish(self, res):
         d = defer.succeed(res)
         def _gotResults(results):
-- 
1.8.1.2

[AB PATCH 14/22] yoctogit.py: Fix timeouts.

[Elizabeth Flanagan]

From: Beth Flanagan <elizabeth.flanagan@intel.com>

We should really never need this big of a timeout, but
poky-contrib (despite my continual kvetching) can be a
large clone, or an autobuilder can have a large build
directory and the mv across filesystems *which you should
never do!* can go slow.

So let's set this sufficently high enough that we don't
have to have failures from a timeout, but sufficently low
enough that if things go south we don't wait forever to
know about it.

Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
---
 .../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py        | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index c5dbf05..98a2fba 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -131,6 +131,7 @@ class YoctoGit(Source):
         self.mode = mode
         self.getDescription = getDescription
         self.config = config
+        self.timeout = 10*60
         # We no longer pass srcdir in. There are way too many ways to hang
         # yourself by doing that.
         if self.mirrordir:
@@ -286,7 +287,8 @@ class YoctoGit(Source):
                                                         str(int(time.time())) + '-'  + 
                                                         str(random.randrange(100, 100000, 2)) + 
                                                         self.layername,
-                                              'logEnviron':self.logEnviron,})
+                                              'logEnviron':self.logEnviron,
+                                              'timeout': self.timeout,})
         cmd.useLog(self.stdio_log, False)
         d = self.runCommand(cmd)
         def evaluateCommand(cmd):
-- 
1.8.1.2

[AB PATCH 15/22] yoctogit.py: Set workdir for the class

[Elizabeth Flanagan]

From: Beth Flanagan <elizabeth.flanagan@intel.com>

Stop utilizing resetWorkdir, it just confuses things and is really
not needed.
Just set self.workdir for the class. This ignores the self.workdir passed
in (which I intend on removing soon.)

Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
---
 .../buildbot/steps/source/yoctogit.py              | 31 ++++------------------
 1 file changed, 5 insertions(+), 26 deletions(-)

diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index 98a2fba..c739d7d 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -146,6 +146,11 @@ class YoctoGit(Source):
                     self.srcdir = 'build/'+self.layername
         else:
             self.srcdir = 'source'
+        if self.method in ['barecopy', 'movecopy']:
+            self.workdir='source/'+self.repourl
+        else:
+            self.workdir = 'build'
+
         Source.__init__(self, **kwargs)
 
         if self.mode not in ['incremental', 'full']:
@@ -261,14 +266,6 @@ class YoctoGit(Source):
                 log.msg("Source step failed while running command %s" % cmd)
                 raise buildstep.BuildStepFailed()
         d.addCallback(lambda _: evaluateCommand(cmd))
-        if "poky" in self.layername or \
-           "oecore" in self.layername or \
-           "eclipse" in self.layername:
-            todir = "build"
-            self.workdir = 'source/'+self.repourl
-        else:
-            todir = "build/" + self.layername
-            self.workdir = 'source/'+self.repourl
         d.addCallback(lambda _: self.incremental())
         return d
 
@@ -296,12 +293,6 @@ class YoctoGit(Source):
                 log.msg("Source step failed while running command %s" % cmd)
                 raise buildstep.BuildStepFailed()
         d.addCallback(lambda _: evaluateCommand(cmd))
-        if "poky" in self.layername or \
-           "oecore" in self.layername or \
-           "eclipse" in self.layername:
-            todir = "build"
-        else:
-            todir = "build/" + self.layername
         d.addCallback(lambda _: self.incremental())
         return d
 
@@ -329,21 +320,9 @@ class YoctoGit(Source):
             if cmd.didFail():
                 log.msg("Source step failed while running command %s" % cmd)
                 raise buildstep.BuildStepFailed()
-
         d.addCallback(lambda _: evaluateCommand(cmd))
         return d
 
-        def resetWorkdir(_):
-            if "poky" in self.layername or \
-               "oecore" in self.layername or \
-               "eclipse" in self.layername:
-                self.workdir = 'build.'
-            else:
-                self.workdir = "build/" + self.layername
-            return 0
-        d.addCallback(resetWorkdir)
-        return d
-
     def finish(self, res):
         d = defer.succeed(res)
         def _gotResults(results):
-- 
1.8.1.2

[AB PATCH 16/22] yoctogit.py: Set workdir within dovccmd

[Elizabeth Flanagan]

From: Beth Flanagan <elizabeth.flanagan@intel.com>

Ensure we're in the correct place by setting workdir in dovccmd.

Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
---
 .../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py        | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index c739d7d..6990abe 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -376,6 +376,10 @@ class YoctoGit(Source):
                 full_command.append('-c')
                 full_command.append('%s=%s' % (name, value))
         full_command.extend(command)
+        if self.method in ['barecopy', 'movecopy']:
+            self.workdir='source/'+self.repourl
+        else:
+            self.workdir = 'build'
         cmd = buildstep.RemoteShellCommand(self.workdir,
                                            full_command,
                                            env=self.env,
-- 
1.8.1.2

[AB PATCH 17/22] yoctogit.py: change _srccopy fromdir to it's own construction

[Elizabeth Flanagan]

From: Beth Flanagan <elizabeth.flanagan@intel.com>

fromdir was using self.srcdir, but lets be a bit safe and use
a variable we construct in the method.

Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
---
 .../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py         | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index 6990abe..4e1a334 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -307,9 +307,10 @@ class YoctoGit(Source):
             todir = "build"
         else:
             todir = "build/" + self.layername
+        fromdir='source/'+self.repourl
 
         cmd = buildstep.RemoteCommand('cpdir',
-                                      {'fromdir': self.srcdir,
+                                      {'fromdir': fromdir,
                                        'todir': todir,
                                        'logEnviron': self.logEnviron,
                                        'timeout': self.timeout,})
-- 
1.8.1.2

[AB PATCH 18/22] PublishArtifacts.py: Exclude genericx86-64 pre-1.5

[Elizabeth Flanagan]

From: Beth Flanagan <elizabeth.flanagan@intel.com>

We don't build it so we shouldn't publish it.

Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
---
 .../autobuilder/buildsteps/PublishArtifacts.py            | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/PublishArtifacts.py b/lib/python2.7/site-packages/autobuilder/buildsteps/PublishArtifacts.py
index e23ecbf..ef9ad81 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/PublishArtifacts.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/PublishArtifacts.py
@@ -169,11 +169,16 @@ class PublishArtifacts(ShellCommand):
                     command=command+"echo 'Skipping copy of " + artifact + ".'"
                 else:
                     artifact_name, deploy_image_dir = self.getDeployNames(artifact, buildername)
-
-                    command=command+"mkdir -p " + DEST + "/"+ MACHINE_PUBLISH_DIR +"/" + artifact_name + ";"
-                    command=command+"cp -R --no-dereference --preserve=links " + \
-                                     deploy_image_dir + \
-                                     "/*"+artifact+"* " + DEST + "/" + MACHINE_PUBLISH_DIR +"/" + artifact_name + ";"
+                    self.layerversion_yoctobsp = self.getProperty("layerversion_yoctobsp")
+                    if self.layerversion_yoctobsp is not None and \
+                       int(self.layerversion_yoctobsp) < 2 and \
+                       "genericx86-64" in artifacts:
+                        command = command+"echo 'Skipping copy of genericx86-64.'; "
+                    else:
+                        command=command+"mkdir -p " + DEST + "/"+ MACHINE_PUBLISH_DIR +"/" + artifact_name + ";"
+                        command=command+"cp -R --no-dereference --preserve=links " + \
+                                         deploy_image_dir + \
+                                         "/*"+artifact+"* " + DEST + "/" + MACHINE_PUBLISH_DIR +"/" + artifact_name + ";"
 
 
             self.command = command
-- 
1.8.1.2

[AB PATCH 19/22] SetDest.py: Use new UI variables.

[Elizabeth Flanagan]

From: Beth Flanagan <elizabeth.flanagan@intel.com>

How we name releases is a lot more complex than the old UI could
handle. This alligns us a bit more with what we actually do.

Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
---
 lib/python2.7/site-packages/autobuilder/buildsteps/SetDest.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/lib/python2.7/site-packages/autobuilder/buildsteps/SetDest.py b/lib/python2.7/site-packages/autobuilder/buildsteps/SetDest.py
index cf901cb..8c1edc5 100644
--- a/lib/python2.7/site-packages/autobuilder/buildsteps/SetDest.py
+++ b/lib/python2.7/site-packages/autobuilder/buildsteps/SetDest.py
@@ -49,8 +49,11 @@ class SetDest(ShellCommand):
         else:
             buildername=self.getProperty("buildername")
             self.workdir=os.path.join(self.slaveworkdir, buildername)
-            if str(self.getProperty("custom_release_me_" + buildername)) == "True":
-                rel_name = self.getProperty("custom_release_name_"  + buildername)
+            if str(self.getProperty("custom_release_me")) == "True":
+                poky_name = self.getProperty("custom_poky_name")
+                poky_number = self.getProperty("custom_poky_number")
+                yocto_number = self.getProperty("custom_yocto_number")
+                rel_name = 'yocto-'+ yocto_number
                 if not rel_name or rel_name == "":
                     rel_name == self.getProperty(commit-description)
                 DEST=os.path.normpath(os.path.join(
-- 
1.8.1.2

[AB PATCH 20/22] yocto-start-autobuilder: Make sure janitor logging goes to a log file

[Elizabeth Flanagan]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Currently logging from the janitor gets lost. Ensure it gets to a log
file instead including anything to stderr.
---
 yocto-start-autobuilder | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/yocto-start-autobuilder b/yocto-start-autobuilder
index b34a60d..74e3a13 100755
--- a/yocto-start-autobuilder
+++ b/yocto-start-autobuilder
@@ -79,7 +79,7 @@ if os.path.isfile(os.path.join(AB_BASE, ".setupdone")):
         os.chdir(AB_BASE)
         if os.environ["OPTIMIZED_GIT_CLONE"] == "True":
             os.chdir(AB_BASE)
-            subprocess.Popen("python bin/buildslave-janitor " + os.path.join(AB_BASE, "config/autobuilder.conf"), 
+            subprocess.Popen("python bin/buildslave-janitor " + os.path.join(AB_BASE, "config/autobuilder.conf") + ">> yocto-slave/janitor.log 2>&1", 
                            shell=True, stdin=None, 
                            stdout=None, stderr=None, 
                            close_fds=True)
-- 
1.8.1.2

[AB PATCH 21/22] buildslave-janitor: Add in try/except handling to make calls robust against failure

[Elizabeth Flanagan]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

There is a suspicion that the command can fail causing the janitor trash
removal process to exit. This adds in some exception handling to deal
with this. It include a timeout to stop it entering nasty loops.

This combined with the previous patch should let us figure out what
error conditions its hitting.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 bin/buildslave-janitor | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/bin/buildslave-janitor b/bin/buildslave-janitor
index 153f18d..7495ce9 100755
--- a/bin/buildslave-janitor
+++ b/bin/buildslave-janitor
@@ -66,11 +66,16 @@ def trash_processor(trashdir):
         print("Not prepared to use a trashdir of /")
         return
     while True:
-        files = os.listdir(trashdir)
-        if files:
-            os.system("ionice -c 3 rm %s -rf" % trashdir)
-        else:
-            time.sleep(30*60) # 30 minutes
+        try:
+            files = os.listdir(trashdir)
+            if files:
+                os.system("ionice -c 3 rm %s -rf" % trashdir)
+            else:
+                time.sleep(30*60) # 30 minutes
+        except Exception as e:
+            print("Exception %s in trash cleaner" % str(e))
+            time.sleep(60) # 1 minute timeout to prevent crazy looping
+            pass
     return
 
 def mirror_processor(mirrordir):
-- 
1.8.1.2

[AB PATCH 22/22] yoctogit.py: sync on mv

[Elizabeth Flanagan]

From: Beth Flanagan <elizabeth.flanagan@intel.com>

On systems with very high load sometimes the system still thinks the
build dir is there. I've only seen this once, but a sync here ensures
that the system knows that build has been moved.

Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
---
 .../buildbot/steps/source/yoctogit.py                  | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
index 4e1a334..c6b8258 100644
--- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
+++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
@@ -204,6 +204,7 @@ class YoctoGit(Source):
             return
         elif self.method == 'movecopy':
             yield self.movecopy()
+            yield self._sync()
             yield self._srccopy()
             return
 
@@ -269,6 +270,23 @@ class YoctoGit(Source):
         d.addCallback(lambda _: self.incremental())
         return d
 
+    def _sync(self):
+        cmd = buildstep.RemoteShellCommand(self.workdir,
+                                           'sync',
+                                           env=self.env,
+                                           logEnviron=self.logEnviron,
+                                           timeout=self.timeout)
+        cmd.useLog(self.stdio_log, False)
+        log.msg("syncing filesystem")
+        d = self.runCommand(cmd)
+        def evaluateCommand(cmd):
+            if cmd.didFail():
+                log.msg("Source step failed while running command %s" % cmd)
+                raise buildstep.BuildStepFailed()
+        d.addCallback(lambda _: evaluateCommand(cmd))
+        return d
+
+
     def movecopy(self):
         # This relies on an external method of cleaning up the todir, like a cron job
         # it can be rather disk space intensive.
-- 
1.8.1.2

Re: [AB PATCH 03/22] BuildImages.py: core-image-basic name change support

[Burton, Ross]

On 19 March 2014 20:51, Elizabeth Flanagan <elizabeth.flanagan@intel.com> wrote:
> +        if self.layerversion_core < "4":
> +            self.images=self.images.replace("core-image-full-cmdline", "core-image-basic")
> +        elif self.layerversion_core > "3":
> +            self.images=self.images.replace("core-image-basic", "core-image-full-cmdline")

I had to read this logic carefully to make sure I wasn't missing any
nuances, why can't the elif be a simple "else"?

Ross

Re: [AB PATCH 05/22] yoctogit.py: movecopy needs an incremental callback

[Burton, Ross]

On 19 March 2014 20:51, Elizabeth Flanagan <elizabeth.flanagan@intel.com> wrote:
> We need to do an incremental before we copy or else, we're
> essentially copying nothing.
>
> Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
> ---
>  .../buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py           | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
> index 25e5308..080f5b2 100644
> --- a/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
> +++ b/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/steps/source/yoctogit.py
> @@ -322,6 +322,7 @@ class YoctoGit(Source):
>          else:
>              todir = "build/" + self.layername
>              self.workdir = 'source/'+self.repourl
> +        d.addCallback(lambda _: self.incremental())
>
>          def copy(_):
>              cmd = buildstep.RemoteCommand('cpdir',
> --
> 1.8.1.2

This is fixing a bug in 2/22 so should be squashed into that commit.

Ross

Re: [AB PATCH 02/22] yoctogit.py: Fix cp construction from source to build

[Burton, Ross]

On 19 March 2014 20:51, Elizabeth Flanagan <elizabeth.flanagan@intel.com> wrote:
> +        if "poky" in self.layername or \
> +           "oecore" in self.layername or \
> +           "eclipse" in self.layername:

Do you actually need to do substring comparisons (in vs is) there?  I
suspect not, so I prefer this idiom as it involves less repetition:

if self.layername in ('poky', 'oecore', 'eclipse')

Ross

Re: [AB PATCH 02/22] yoctogit.py: Fix cp construction from source to build

[Flanagan, Elizabeth]

On Thu, Mar 20, 2014 at 7:29 AM, Burton, Ross <ross.burton@intel.com> wrote:
> On 19 March 2014 20:51, Elizabeth Flanagan <elizabeth.flanagan@intel.com> wrote:
>> +        if "poky" in self.layername or \
>> +           "oecore" in self.layername or \
>> +           "eclipse" in self.layername:
>
> Do you actually need to do substring comparisons (in vs is) there?  I
> suspect not, so I prefer this idiom as it involves less repetition:
>
> if self.layername in ('poky', 'oecore', 'eclipse')

In this case, yeah, we need a substring comparision, as we need to
support the layer being named eclipse-poky-<branch-name> and
poky/poky-contrib. We need to eventually figure out a better, more
standardized way of indicating the base layer, but until we do and
propagate that change out (as it changes configs), doing it this way
both supports the way people have come to understand what "base" is.

-b

>
> Ross
> ---------------------------------------------------------------------
> Intel Corporation (UK) Limited
> Registered No. 1134945 (England)
> Registered Office: Pipers Way, Swindon SN3 1RJ
> VAT No: 860 2173 47
>
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.



-- 
Elizabeth Flanagan
Yocto Project
Build and Release

Re: [AB PATCH 22/22] yoctogit.py: sync on mv

[Burton, Ross]

On 19 March 2014 20:51, Elizabeth Flanagan <elizabeth.flanagan@intel.com> wrote:
> On systems with very high load sometimes the system still thinks the
> build dir is there. I've only seen this once, but a sync here ensures
> that the system knows that build has been moved.

That smells like a race in the AB code instead of the kernel being
confused over the state of the file system, and the best a sync can do
is reduce the window of failure at the cost of a potentially very
heavy IO burst.

I'd say drop this and if the problem occurs again, attempt to diagnose
where the race is.

Ross