[autobuilder][PATCH 1/4] README: update filenames and locations

[autobuilder][PATCH 1/4] README: update filenames and locations

[Joe MacDonald]

The main README points at conf/autobuilder.conf and conf/yoctoAB.conf,
both of which aren't the current places where the autobuilder looks for
these files anymore.

README-NEW-AUTOBUILDER has a bit of whitespace damage, in the text and in
the examples, so clean that up to make it more paste-friendly.

Signed-off-by: Joe MacDonald <joe_macdonald@mentor.com>
---
 README                 |  6 +++---
 README-NEW-AUTOBUILDER | 14 +++++++-------
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/README b/README
index 94dae28..06b9cbc 100644
--- a/README
+++ b/README
@@ -97,13 +97,13 @@ Setting up build history requires the following steps:
 Files used for Yocto-Autobuilder Configuration
 ==============================================
 
-Name: conf/autobuilder.conf
+Name: config/autobuilder.conf
 Purpose:
 This file is used to set autobuilder wide parameters, like where various build
 artifacts are published to, DL_DIR, SSTATE_DIR or if build artifacts should be
 published (needed for production autobuilders, not needed for desktop builders)
 
-Name: conf/yoctoAB.conf
+Name: buildset-config/yoctoAB.conf
 Purpose:
 This is the main yocto autobuilder config. Documentation for this and associated
-format is in README_CONFIG
+format is in README-NEW-AUTOBUILDER
diff --git a/README-NEW-AUTOBUILDER b/README-NEW-AUTOBUILDER
index 97bbe61..73be208 100644
--- a/README-NEW-AUTOBUILDER
+++ b/README-NEW-AUTOBUILDER
@@ -170,12 +170,12 @@ scheduler: A list of dicts. Each item defines a scheduler associated with this
                     This defaults to '*', meaning every month.
                dayOfWeek: The day of the week in which to start the build, with
                     Monday = 1.  This defaults to '*', meaning every day.
-	       hour: The hour of the day in which to start the build, with 
+               hour: The hour of the day in which to start the build, with
                     12:00 AM = 0.  This must be set by the user.
-	       minute: The minute of the hour in which to start the build, with
+               minute: The minute of the hour in which to start the build, with
                     0 to 59 as valid values.  This must be set by the user.
 
-	  SingleBranchScheduler properties:
+       SingleBranchScheduler properties:
                repository: the repository to attach the scheduler to; this
                     is the repo name from the 'repos' section; the branch which
                     the scheduler is attached to matches that in the repo
@@ -185,7 +185,7 @@ scheduler: A list of dicts. Each item defines a scheduler associated with this
                     triggering build to allow for changes to settle.  Defaults
                     to 60 seconds.
 
-	       change-user: the user name which the remote hook will use; the
+               change-user: the user name which the remote hook will use; the
                     default value is the repository name as per the 'repository'
                     property.
 
@@ -209,7 +209,7 @@ scheduler: A list of dicts. Each item defines a scheduler associated with this
            Extra Notes on SingleBranchScheduler setup
            ------------------------------------------
 
-	   There is one SBS scheduler for each repository in the buildset, i.e.,
+           There is one SBS scheduler for each repository in the buildset, i.e.,
            if your buildset contains more repositories, and you want build
            triggered by changes to any of them, you need to define a scheduler
            for each. This also means that you can mix and match, e.g., only
@@ -272,8 +272,8 @@ steps: [{'SetDest':{}},
         {'PublishArtifacts': {'artifacts': ['qemux86', 'atom-pc']}}]
 scheduler: [{'dev-branch-scheduler' :
                 {'type':'SingleBranchScheduler',
-		 'repository':'poky',
-		 'stable-timer':30,
+                 'repository':'poky',
+                 'stable-timer':30,
                  'change-password':'secret_change_password'}}]
 
 Adding Buildsteps
-- 
1.9.1

[autobuilder][PATCH 2/4] gitignore: add janitor log and additional config files

[Joe MacDonald]

The janitor log for workers was missing from gitignore, so add it.

Every deployment is expected to make their own changes to buildset-config,
so add the whole directory to the ignore list.

Similarly, every deployment is likely to make their own changes to
config/autobuilder.conf, so ignore changes to it as well.

Signed-off-by: Joe MacDonald <joe_macdonald@mentor.com>
---
 .gitignore | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/.gitignore b/.gitignore
index f1f9e4e..cb595bb 100644
--- a/.gitignore
+++ b/.gitignore
@@ -4,6 +4,11 @@
 *.so
 *.pyc
 
+# Configuration hunks, changed by anyone using AB #
+###################################################
+buildset-config
+config/autobuilder.conf
+
 # Everything else #
 ###################
 *.log.*
@@ -22,6 +27,7 @@ yocto-controller/twistd.log*
 yocto-controller/buildbot.tac
 yocto-worker/build-appliance/build(newcommits)
 yocto-worker/buildbot.tac
+yocto-worker/janitor.log
 yocto-worker/nightly-x86/.buildbot-sourcedata-YnVpbGQ=
 yocto-worker/nightly/build(newcommits)
 yocto-worker/twistd.hostname
-- 
1.9.1

[autobuilder][PATCH 3/4] gitignore: add worker-init

[Joe MacDonald]

yocto-autobuilder-setup modifies bin/worker-init on every new deployment,
so we don't want to consider worker-init for tracking by default.

Signed-off-by: Joe MacDonald <joe_macdonald@mentor.com>
---
 .gitignore | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/.gitignore b/.gitignore
index cb595bb..a4b31c6 100644
--- a/.gitignore
+++ b/.gitignore
@@ -33,3 +33,8 @@ yocto-worker/nightly/build(newcommits)
 yocto-worker/twistd.hostname
 yocto-worker/twistd.log*
 yocto-worker/twistd.pid
+
+# This gets updated on every yocto-setup-autobuilder so should be ignored #
+# for nearly everyone.                                                    #
+###########################################################################
+bin/worker-init
-- 
1.9.1

[autobuilder][PATCH 4/4] janitor: clean up janitor logging

[Joe MacDonald]

If OPTIMIZED_GIT_CLONE was set, the worker janitor was start up and
depending on the system, produce an error of the form:

/bin/sh: line 1: 11093 Killed                  python bin/buildworker-janitor [path-to--autobuilder]/config/autobuilder.conf >> yocto-worker/janitor.log 2>&1

By removing the shell-redirect from the Popen() call and instead sending
all logs directly to a regular log file, the error goes to the log where
it belongs.  Since this happens regularly when the host is both a control
and a build node, this change also makes it clearer to the user that
something bad has not necessarily happened the first time they try to
start up the autobuilder.

At the same time, since the logs tend to not have much in them by way of
context, adding a timestamp to each new start of the janitor makes it much
easier when looking over the old logs to see if errors are current or long
in the past.

Signed-off-by: Joe MacDonald <joe_macdonald@mentor.com>
---
 yocto-start-autobuilder | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/yocto-start-autobuilder b/yocto-start-autobuilder
index 7078163..ee597df 100755
--- a/yocto-start-autobuilder
+++ b/yocto-start-autobuilder
@@ -20,7 +20,7 @@
 # along with this program; if not, write to the Free Software
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 
-import os, sys, optparse, subprocess
+import os, sys, optparse, subprocess, datetime
 from socket import gethostname
 import ConfigParser
 
@@ -79,10 +79,15 @@ if os.path.isfile(os.path.join(AB_BASE, ".setupdone")):
         os.chdir(AB_BASE)
         if os.environ["OPTIMIZED_GIT_CLONE"] == "True":
             os.chdir(AB_BASE)
-            subprocess.Popen("python bin/buildworker-janitor " + os.path.join(AB_BASE, "config/autobuilder.conf") + ">> yocto-worker/janitor.log 2>&1", 
+            janitor_log = open('yocto-worker/janitor.log', 'a')
+            janitor_log.write('[ janitor started: %s ]' % datetime.datetime.now())
+            janitor_log.write('\n')
+            subprocess.Popen("python bin/buildworker-janitor " + os.path.join(AB_BASE, "config/autobuilder.conf"), 
                            shell=True, stdin=None, 
-                           stdout=None, stderr=None, 
+                           stdout=janitor_log,
+                           stderr=janitor_log, 
                            close_fds=True)
+            janitor_log.close()
 else:
     print "You have not sourced ./yocto-autobuilder-setup. Please do so first!"
     print ""
-- 
1.9.1

Re: [autobuilder][PATCH 1/4] README: update filenames and locations

[Flanagan, Elizabeth]

Pulled to master. Production this evening. Thanks!

On 17 February 2015 at 21:41, Joe MacDonald <joe_macdonald@mentor.com> wrote:
> The main README points at conf/autobuilder.conf and conf/yoctoAB.conf,
> both of which aren't the current places where the autobuilder looks for
> these files anymore.
>
> README-NEW-AUTOBUILDER has a bit of whitespace damage, in the text and in
> the examples, so clean that up to make it more paste-friendly.
>
> Signed-off-by: Joe MacDonald <joe_macdonald@mentor.com>
> ---
>  README                 |  6 +++---
>  README-NEW-AUTOBUILDER | 14 +++++++-------
>  2 files changed, 10 insertions(+), 10 deletions(-)
>
> diff --git a/README b/README
> index 94dae28..06b9cbc 100644
> --- a/README
> +++ b/README
> @@ -97,13 +97,13 @@ Setting up build history requires the following steps:
>  Files used for Yocto-Autobuilder Configuration
>  ==============================================
>
> -Name: conf/autobuilder.conf
> +Name: config/autobuilder.conf
>  Purpose:
>  This file is used to set autobuilder wide parameters, like where various build
>  artifacts are published to, DL_DIR, SSTATE_DIR or if build artifacts should be
>  published (needed for production autobuilders, not needed for desktop builders)
>
> -Name: conf/yoctoAB.conf
> +Name: buildset-config/yoctoAB.conf
>  Purpose:
>  This is the main yocto autobuilder config. Documentation for this and associated
> -format is in README_CONFIG
> +format is in README-NEW-AUTOBUILDER
> diff --git a/README-NEW-AUTOBUILDER b/README-NEW-AUTOBUILDER
> index 97bbe61..73be208 100644
> --- a/README-NEW-AUTOBUILDER
> +++ b/README-NEW-AUTOBUILDER
> @@ -170,12 +170,12 @@ scheduler: A list of dicts. Each item defines a scheduler associated with this
>                      This defaults to '*', meaning every month.
>                 dayOfWeek: The day of the week in which to start the build, with
>                      Monday = 1.  This defaults to '*', meaning every day.
> -              hour: The hour of the day in which to start the build, with
> +               hour: The hour of the day in which to start the build, with
>                      12:00 AM = 0.  This must be set by the user.
> -              minute: The minute of the hour in which to start the build, with
> +               minute: The minute of the hour in which to start the build, with
>                      0 to 59 as valid values.  This must be set by the user.
>
> -         SingleBranchScheduler properties:
> +       SingleBranchScheduler properties:
>                 repository: the repository to attach the scheduler to; this
>                      is the repo name from the 'repos' section; the branch which
>                      the scheduler is attached to matches that in the repo
> @@ -185,7 +185,7 @@ scheduler: A list of dicts. Each item defines a scheduler associated with this
>                      triggering build to allow for changes to settle.  Defaults
>                      to 60 seconds.
>
> -              change-user: the user name which the remote hook will use; the
> +               change-user: the user name which the remote hook will use; the
>                      default value is the repository name as per the 'repository'
>                      property.
>
> @@ -209,7 +209,7 @@ scheduler: A list of dicts. Each item defines a scheduler associated with this
>             Extra Notes on SingleBranchScheduler setup
>             ------------------------------------------
>
> -          There is one SBS scheduler for each repository in the buildset, i.e.,
> +           There is one SBS scheduler for each repository in the buildset, i.e.,
>             if your buildset contains more repositories, and you want build
>             triggered by changes to any of them, you need to define a scheduler
>             for each. This also means that you can mix and match, e.g., only
> @@ -272,8 +272,8 @@ steps: [{'SetDest':{}},
>          {'PublishArtifacts': {'artifacts': ['qemux86', 'atom-pc']}}]
>  scheduler: [{'dev-branch-scheduler' :
>                  {'type':'SingleBranchScheduler',
> -                'repository':'poky',
> -                'stable-timer':30,
> +                 'repository':'poky',
> +                 'stable-timer':30,
>                   'change-password':'secret_change_password'}}]
>
>  Adding Buildsteps
> --
> 1.9.1
>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto



-- 
Elizabeth Flanagan
Yocto Project
Build and Release

Re: [autobuilder][PATCH 1/4] README: update filenames and locations

[Joe MacDonald]

[-- Attachment #1: Type: text/plain, Size: 5397 bytes --]

[Re: [yocto] [autobuilder][PATCH 1/4] README: update filenames and locations] On 15.02.19 (Thu 16:54) Flanagan, Elizabeth wrote:

> Pulled to master. Production this evening. Thanks!

Cool, thanks.  If you haven't already, please drop my 'templates' patch
from consideration, I figure I'll send out the completed version,
including external configuration support, for consideration soon.  It
works well for me and I don't know that the half-step is really worth
it.

FWIW, it'll still be two separate patches, if you opt to pick the first
and not the second, I'll just include both in a single thread.

-J.

> 
> On 17 February 2015 at 21:41, Joe MacDonald <joe_macdonald@mentor.com> wrote:
> > The main README points at conf/autobuilder.conf and conf/yoctoAB.conf,
> > both of which aren't the current places where the autobuilder looks for
> > these files anymore.
> >
> > README-NEW-AUTOBUILDER has a bit of whitespace damage, in the text and in
> > the examples, so clean that up to make it more paste-friendly.
> >
> > Signed-off-by: Joe MacDonald <joe_macdonald@mentor.com>
> > ---
> >  README                 |  6 +++---
> >  README-NEW-AUTOBUILDER | 14 +++++++-------
> >  2 files changed, 10 insertions(+), 10 deletions(-)
> >
> > diff --git a/README b/README
> > index 94dae28..06b9cbc 100644
> > --- a/README
> > +++ b/README
> > @@ -97,13 +97,13 @@ Setting up build history requires the following steps:
> >  Files used for Yocto-Autobuilder Configuration
> >  ==============================================
> >
> > -Name: conf/autobuilder.conf
> > +Name: config/autobuilder.conf
> >  Purpose:
> >  This file is used to set autobuilder wide parameters, like where various build
> >  artifacts are published to, DL_DIR, SSTATE_DIR or if build artifacts should be
> >  published (needed for production autobuilders, not needed for desktop builders)
> >
> > -Name: conf/yoctoAB.conf
> > +Name: buildset-config/yoctoAB.conf
> >  Purpose:
> >  This is the main yocto autobuilder config. Documentation for this and associated
> > -format is in README_CONFIG
> > +format is in README-NEW-AUTOBUILDER
> > diff --git a/README-NEW-AUTOBUILDER b/README-NEW-AUTOBUILDER
> > index 97bbe61..73be208 100644
> > --- a/README-NEW-AUTOBUILDER
> > +++ b/README-NEW-AUTOBUILDER
> > @@ -170,12 +170,12 @@ scheduler: A list of dicts. Each item defines a scheduler associated with this
> >                      This defaults to '*', meaning every month.
> >                 dayOfWeek: The day of the week in which to start the build, with
> >                      Monday = 1.  This defaults to '*', meaning every day.
> > -              hour: The hour of the day in which to start the build, with
> > +               hour: The hour of the day in which to start the build, with
> >                      12:00 AM = 0.  This must be set by the user.
> > -              minute: The minute of the hour in which to start the build, with
> > +               minute: The minute of the hour in which to start the build, with
> >                      0 to 59 as valid values.  This must be set by the user.
> >
> > -         SingleBranchScheduler properties:
> > +       SingleBranchScheduler properties:
> >                 repository: the repository to attach the scheduler to; this
> >                      is the repo name from the 'repos' section; the branch which
> >                      the scheduler is attached to matches that in the repo
> > @@ -185,7 +185,7 @@ scheduler: A list of dicts. Each item defines a scheduler associated with this
> >                      triggering build to allow for changes to settle.  Defaults
> >                      to 60 seconds.
> >
> > -              change-user: the user name which the remote hook will use; the
> > +               change-user: the user name which the remote hook will use; the
> >                      default value is the repository name as per the 'repository'
> >                      property.
> >
> > @@ -209,7 +209,7 @@ scheduler: A list of dicts. Each item defines a scheduler associated with this
> >             Extra Notes on SingleBranchScheduler setup
> >             ------------------------------------------
> >
> > -          There is one SBS scheduler for each repository in the buildset, i.e.,
> > +           There is one SBS scheduler for each repository in the buildset, i.e.,
> >             if your buildset contains more repositories, and you want build
> >             triggered by changes to any of them, you need to define a scheduler
> >             for each. This also means that you can mix and match, e.g., only
> > @@ -272,8 +272,8 @@ steps: [{'SetDest':{}},
> >          {'PublishArtifacts': {'artifacts': ['qemux86', 'atom-pc']}}]
> >  scheduler: [{'dev-branch-scheduler' :
> >                  {'type':'SingleBranchScheduler',
> > -                'repository':'poky',
> > -                'stable-timer':30,
> > +                 'repository':'poky',
> > +                 'stable-timer':30,
> >                   'change-password':'secret_change_password'}}]
> >
> >  Adding Buildsteps
> > --
> > 1.9.1
> >
> > --
> > _______________________________________________
> > yocto mailing list
> > yocto@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/yocto
> 
> 
> 
-- 
-Joe MacDonald.
:wq

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 501 bytes --]

Re: [autobuilder][PATCH 1/4] README: update filenames and locations

[Flanagan, Elizabeth]

On 19 February 2015 at 19:30, Joe MacDonald <Joe_MacDonald@mentor.com> wrote:
> [Re: [yocto] [autobuilder][PATCH 1/4] README: update filenames and locations] On 15.02.19 (Thu 16:54) Flanagan, Elizabeth wrote:
>
>> Pulled to master. Production this evening. Thanks!
>
> Cool, thanks.  If you haven't already, please drop my 'templates' patch
> from consideration, I figure I'll send out the completed version,
> including external configuration support, for consideration soon.  It
> works well for me and I don't know that the half-step is really worth
> it.
>

I haven't gotten that far in my email backlog yet, so I'll just wait
for the resend.

> FWIW, it'll still be two separate patches, if you opt to pick the first
> and not the second, I'll just include both in a single thread.

Please do.

>
> -J.
>
>>
>> On 17 February 2015 at 21:41, Joe MacDonald <joe_macdonald@mentor.com> wrote:
>> > The main README points at conf/autobuilder.conf and conf/yoctoAB.conf,
>> > both of which aren't the current places where the autobuilder looks for
>> > these files anymore.
>> >
>> > README-NEW-AUTOBUILDER has a bit of whitespace damage, in the text and in
>> > the examples, so clean that up to make it more paste-friendly.
>> >
>> > Signed-off-by: Joe MacDonald <joe_macdonald@mentor.com>
>> > ---
>> >  README                 |  6 +++---
>> >  README-NEW-AUTOBUILDER | 14 +++++++-------
>> >  2 files changed, 10 insertions(+), 10 deletions(-)
>> >
>> > diff --git a/README b/README
>> > index 94dae28..06b9cbc 100644
>> > --- a/README
>> > +++ b/README
>> > @@ -97,13 +97,13 @@ Setting up build history requires the following steps:
>> >  Files used for Yocto-Autobuilder Configuration
>> >  ==============================================
>> >
>> > -Name: conf/autobuilder.conf
>> > +Name: config/autobuilder.conf
>> >  Purpose:
>> >  This file is used to set autobuilder wide parameters, like where various build
>> >  artifacts are published to, DL_DIR, SSTATE_DIR or if build artifacts should be
>> >  published (needed for production autobuilders, not needed for desktop builders)
>> >
>> > -Name: conf/yoctoAB.conf
>> > +Name: buildset-config/yoctoAB.conf
>> >  Purpose:
>> >  This is the main yocto autobuilder config. Documentation for this and associated
>> > -format is in README_CONFIG
>> > +format is in README-NEW-AUTOBUILDER
>> > diff --git a/README-NEW-AUTOBUILDER b/README-NEW-AUTOBUILDER
>> > index 97bbe61..73be208 100644
>> > --- a/README-NEW-AUTOBUILDER
>> > +++ b/README-NEW-AUTOBUILDER
>> > @@ -170,12 +170,12 @@ scheduler: A list of dicts. Each item defines a scheduler associated with this
>> >                      This defaults to '*', meaning every month.
>> >                 dayOfWeek: The day of the week in which to start the build, with
>> >                      Monday = 1.  This defaults to '*', meaning every day.
>> > -              hour: The hour of the day in which to start the build, with
>> > +               hour: The hour of the day in which to start the build, with
>> >                      12:00 AM = 0.  This must be set by the user.
>> > -              minute: The minute of the hour in which to start the build, with
>> > +               minute: The minute of the hour in which to start the build, with
>> >                      0 to 59 as valid values.  This must be set by the user.
>> >
>> > -         SingleBranchScheduler properties:
>> > +       SingleBranchScheduler properties:
>> >                 repository: the repository to attach the scheduler to; this
>> >                      is the repo name from the 'repos' section; the branch which
>> >                      the scheduler is attached to matches that in the repo
>> > @@ -185,7 +185,7 @@ scheduler: A list of dicts. Each item defines a scheduler associated with this
>> >                      triggering build to allow for changes to settle.  Defaults
>> >                      to 60 seconds.
>> >
>> > -              change-user: the user name which the remote hook will use; the
>> > +               change-user: the user name which the remote hook will use; the
>> >                      default value is the repository name as per the 'repository'
>> >                      property.
>> >
>> > @@ -209,7 +209,7 @@ scheduler: A list of dicts. Each item defines a scheduler associated with this
>> >             Extra Notes on SingleBranchScheduler setup
>> >             ------------------------------------------
>> >
>> > -          There is one SBS scheduler for each repository in the buildset, i.e.,
>> > +           There is one SBS scheduler for each repository in the buildset, i.e.,
>> >             if your buildset contains more repositories, and you want build
>> >             triggered by changes to any of them, you need to define a scheduler
>> >             for each. This also means that you can mix and match, e.g., only
>> > @@ -272,8 +272,8 @@ steps: [{'SetDest':{}},
>> >          {'PublishArtifacts': {'artifacts': ['qemux86', 'atom-pc']}}]
>> >  scheduler: [{'dev-branch-scheduler' :
>> >                  {'type':'SingleBranchScheduler',
>> > -                'repository':'poky',
>> > -                'stable-timer':30,
>> > +                 'repository':'poky',
>> > +                 'stable-timer':30,
>> >                   'change-password':'secret_change_password'}}]
>> >
>> >  Adding Buildsteps
>> > --
>> > 1.9.1
>> >
>> > --
>> > _______________________________________________
>> > yocto mailing list
>> > yocto@yoctoproject.org
>> > https://lists.yoctoproject.org/listinfo/yocto
>>
>>
>>
> --
> -Joe MacDonald.
> :wq



-- 
Elizabeth Flanagan
Yocto Project
Build and Release