* [U-Boot] [PATCH v4 0/13] Add some missing buildman features and deprecate MAKEALL
@ 2014-08-05 14:46 Simon Glass
2014-08-05 14:46 ` [U-Boot] [PATCH v4 01/13] buildman: Fix a few typos Simon Glass
` (12 more replies)
0 siblings, 13 replies; 42+ messages in thread
From: Simon Glass @ 2014-08-05 14:46 UTC (permalink / raw)
To: u-boot
Buildman has been around for a little over a year and is used by a fair
number of U-Boot developers. However quite a few people still use MAKEALL.
Buildman was intended to replace MAKEALL, so perhaps now is a good time to
start that process.
The reasons to deprecate MAKEALL are:
- We don't want to maintain two build systems
- Buildman is typically faster
- Buildman has a lot more features
This series adds a few features to buildman to fill some gaps, adds some
information into the README on how to migrate from MAKEALL, and adds a
deprecation message to MAKEALL.
Changes in v4:
- Fix missing import in builder.py
- Fix typo in commit message
- Add new patch to remove unused non-incremental build method code
- Add new patch to allow the config file to be specified
- Add new patch to show a message when there are no errors
- Add new patch to search for *cc instead of *gcc
- Add new path to add a few more toolchain examples to the README
Changes in v3:
- Add new patch to build current source tree
- Add new patch to move BuilderThread code to its own file
- Add new patch to sort command line options
- Add new patch to refactor output options
- Add new patch to add a verbose option
Changes in v2:
- Add new patch to fix existing typos
- Minor changes to the text
Simon Glass (13):
buildman: Fix a few typos
buildman: Add some notes about moving from MAKEALL
buildman: Allow building of current source tree
buildman: Move BuilderThread code to its own file
buildman: Sort command line options
buildman: Refactor output options
buildman: Add verbose option to display errors as they happen
buildman: Remove unused non-incremental build method code
buildman: Add an option to specify the buildman config file
buildman: Add a message indicating there are no errors
buildman: Search for *cc instead of *gcc for the compiler
buildman: Add a few more toolchain examples to the README
RFC: Deprecate MAKEALL
MAKEALL | 4 +
tools/buildman/README | 150 +++++++++--
tools/buildman/builder.py | 544 ++++++----------------------------------
tools/buildman/builderthread.py | 434 ++++++++++++++++++++++++++++++++
tools/buildman/buildman.py | 16 +-
tools/buildman/control.py | 101 +++++---
tools/buildman/test.py | 1 -
tools/buildman/toolchain.py | 2 +-
8 files changed, 718 insertions(+), 534 deletions(-)
create mode 100644 tools/buildman/builderthread.py
--
2.0.0.526.g5318336
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 01/13] buildman: Fix a few typos
2014-08-05 14:46 [U-Boot] [PATCH v4 0/13] Add some missing buildman features and deprecate MAKEALL Simon Glass
@ 2014-08-05 14:46 ` Simon Glass
2014-08-05 14:46 ` [U-Boot] [PATCH v4 02/13] buildman: Add some notes about moving from MAKEALL Simon Glass
` (11 subsequent siblings)
12 siblings, 0 replies; 42+ messages in thread
From: Simon Glass @ 2014-08-05 14:46 UTC (permalink / raw)
To: u-boot
There are several typos in the README - fix them.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v4: None
Changes in v3: None
Changes in v2:
- Add new patch to fix existing typos
tools/buildman/README | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/tools/buildman/README b/tools/buildman/README
index c30c1d4..a5d181c 100644
--- a/tools/buildman/README
+++ b/tools/buildman/README
@@ -71,8 +71,8 @@ directory. It clones this repository into a copy for each thread, and the
threads do not affect the state of your git repository. Any checkouts done
by the thread affect only the working directory for that thread.
-Buildman automatically selects the correct toolchain for each board. You
-must supply suitable toolchains, but buildman takes care of selecting the
+Buildman automatically selects the correct tool chain for each board. You
+must supply suitable tool chains, but buildman takes care of selecting the
right one.
Buildman always builds a branch, and always builds the upstream commit as
@@ -287,7 +287,7 @@ If it can't detect the upstream branch, try checking out the branch, and
doing something like 'git branch --set-upstream <branch> upstream/master'
or something similar.
-As an exmmple:
+As an example:
Dry run, so not doing much. But I would do this:
@@ -339,7 +339,7 @@ Building 18 commits for 1059 boards (4 threads, 1 job per thread)
528 36 124 /19062 1:13:30 : SIMPC8313_SP
This means that it is building 19062 board/commit combinations. So far it
-has managed to succesfully build 528. Another 36 have built with warnings,
+has managed to successfully build 528. Another 36 have built with warnings,
and 124 more didn't build at all. Buildman expects to complete the process
in an hour and 15 minutes. Use this time to buy a faster computer.
@@ -413,7 +413,7 @@ again.
At commit 16, the error moves - you can see that the old error at line 120
is fixed, but there is a new one at line 126. This is probably only because
-we added some code and moved the broken line futher down the file.
+we added some code and moved the broken line father down the file.
If many boards have the same error, then -e will display the error only
once. This makes the output as concise as possible.
@@ -491,7 +491,7 @@ You can also use -d to see a detailed size breakdown for each board. This
list is sorted in order from largest growth to largest reduction.
It is possible to go a little further with the -B option (--bloat). This
-shows where U-Boot has bloted, breaking the size change down to the function
+shows where U-Boot has bloated, breaking the size change down to the function
level. Example output is below:
$ ./tools/buildman/buildman -b us-mem4 -sSdB
--
2.0.0.526.g5318336
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 02/13] buildman: Add some notes about moving from MAKEALL
2014-08-05 14:46 [U-Boot] [PATCH v4 0/13] Add some missing buildman features and deprecate MAKEALL Simon Glass
2014-08-05 14:46 ` [U-Boot] [PATCH v4 01/13] buildman: Fix a few typos Simon Glass
@ 2014-08-05 14:46 ` Simon Glass
2014-08-05 22:34 ` York Sun
2014-08-05 14:46 ` [U-Boot] [PATCH v4 03/13] buildman: Allow building of current source tree Simon Glass
` (10 subsequent siblings)
12 siblings, 1 reply; 42+ messages in thread
From: Simon Glass @ 2014-08-05 14:46 UTC (permalink / raw)
To: u-boot
For those used to MAKEALL, buildman seems strange. Add some notes to ease
the transition.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v4: None
Changes in v3: None
Changes in v2:
- Minor changes to the text
tools/buildman/README | 92 +++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 92 insertions(+)
diff --git a/tools/buildman/README b/tools/buildman/README
index a5d181c..1c919af 100644
--- a/tools/buildman/README
+++ b/tools/buildman/README
@@ -3,6 +3,8 @@
# SPDX-License-Identifier: GPL-2.0+
#
+(Please read 'How to change from MAKEALL' if you are used to that tool)
+
What is this?
=============
@@ -663,6 +665,96 @@ Other options
Buildman has various other command line options. Try --help to see them.
+How to change from MAKEALL
+==========================
+
+Buildman includes most of the features of MAKEALL and is generally faster
+and easier to use. In particular it builds entire branches: if a particular
+commit introduces an error in a particular board, buildman can easily show
+you this, even if a later commit fixes that error.
+
+The reasons to deprecate MAKEALL are:
+- We don't want to maintain two build systems
+- Buildman is typically faster
+- Buildman has a lot more features
+
+But still, many people will be sad to lose MAKEALL. If you are used to
+MAKEALL, here are a few pointers.
+
+First you need to set up your tool chains - see the 'Setting up' section
+for details. Once you have your required toolchain(s) detected then you are
+ready to go.
+
+Buildman works on entire branches, so the normal use is:
+
+ ./tools/buildman/buildman -b <branch_name> <list of things to build>
+
+followed by (afterwards, or perhaps concurrently in another terminal):
+
+ ./tools/buildman/buildman -b <branch_name> -s <list of things to build>
+
+to see the results of the build. Rather than showing you all the output,
+buildman just shows a summary, with red indicating that a commit introduced
+an error and green indicating that a commit fixed an error. Use the -e
+flag to see the full errors.
+
+You don't need to stick around on that branch while buildman is running. It
+checks out its own copy of the source code, so you can change branches,
+add commits, etc. without affecting the build in progress.
+
+The <list of things to build> can include board names, architectures or the
+like. There are no flags to disambiguate since ambiguities are rare. Using
+the examples from MAKEALL:
+
+Examples:
+ - build all Power Architecture boards:
+ MAKEALL -a powerpc
+ MAKEALL --arch powerpc
+ MAKEALL powerpc
+ ** buildman -b <branch> powerpc
+ - build all PowerPC boards manufactured by vendor "esd":
+ MAKEALL -a powerpc -v esd
+ ** buildman -b <branch> esd
+ - build all PowerPC boards manufactured either by "keymile" or "siemens":
+ MAKEALL -a powerpc -v keymile -v siemens
+ ** buildman -b <branch> keymile siemens
+ - build all Freescale boards with MPC83xx CPUs, plus all 4xx boards:
+ MAKEALL -c mpc83xx -v freescale 4xx
+ ** buildman -b <branch> mpc83xx freescale 4xx
+
+Buildman automatically tries to use all the CPUs in your machine. If you
+are building a lot of boards it will use one thread for every CPU core
+it detects in your machine. This is like MAKEALL's BUILD_NBUILDS option.
+You can use the -T flag to change the number of threads. If you are only
+building a few boards, buildman will automatically run make with the -j
+flag to increase the number of concurrent make tasks. It isn't normally
+that helpful to fiddle with this option, but if you use the BUILD_NCPUS
+option in MAKEALL then -j is the equivalent in buildman.
+
+Buildman puts its output in ../<branch_name> by default but you can change
+this with the -o option. Buildman normally does out-of-tree builds: use -i
+to disable that if you really want to. But be careful that once you have
+used -i you pollute buildman's copies of the source tree, and you will need
+to remove the build directory (normally ../<branch_name>) to run buildman
+in normal mode (without -i).
+
+Buildman doesn't keep the output result normally, but use the -k option to
+do this.
+
+Please read 'Theory of Operation' a few times as it will make a lot of
+things clearer.
+
+Some options you might like are:
+
+ -B shows which functions are growing/shrinking in which commit - great
+ for finding code bloat.
+ -S shows image sizes for each commit (just an overall summary)
+ -u shows boards that you haven't built yet
+ --step 0 will build just the upstream commit and the last commit of your
+ branch. This is often a quick sanity check that your branch doesn't
+ break anything. But note this does not check bisectability!
+
+
TODO
====
--
2.0.0.526.g5318336
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 03/13] buildman: Allow building of current source tree
2014-08-05 14:46 [U-Boot] [PATCH v4 0/13] Add some missing buildman features and deprecate MAKEALL Simon Glass
2014-08-05 14:46 ` [U-Boot] [PATCH v4 01/13] buildman: Fix a few typos Simon Glass
2014-08-05 14:46 ` [U-Boot] [PATCH v4 02/13] buildman: Add some notes about moving from MAKEALL Simon Glass
@ 2014-08-05 14:46 ` Simon Glass
2014-08-05 18:54 ` Tom Rini
2014-08-05 14:46 ` [U-Boot] [PATCH v4 04/13] buildman: Move BuilderThread code to its own file Simon Glass
` (9 subsequent siblings)
12 siblings, 1 reply; 42+ messages in thread
From: Simon Glass @ 2014-08-05 14:46 UTC (permalink / raw)
To: u-boot
Originally buildman had some support for building the current source tree.
However this was dropped before it was submitted, as part of the effort to
make it faster when building entire branches.
Reinstate this support. If no -b option is given, buildman will build the
current source tree.
Reported-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v4: None
Changes in v3:
- Add new patch to build current source tree
Changes in v2: None
tools/buildman/builder.py | 71 +++++++++++++++++++++++++-----------
tools/buildman/control.py | 91 ++++++++++++++++++++++++++++++-----------------
2 files changed, 108 insertions(+), 54 deletions(-)
diff --git a/tools/buildman/builder.py b/tools/buildman/builder.py
index 48408ff..d2b72d5 100644
--- a/tools/buildman/builder.py
+++ b/tools/buildman/builder.py
@@ -248,21 +248,34 @@ class BuilderThread(threading.Thread):
if self.toolchain:
# Checkout the right commit
- if commit_upto is not None:
+ if self.builder.commits:
commit = self.builder.commits[commit_upto]
if self.builder.checkout:
git_dir = os.path.join(work_dir, '.git')
gitutil.Checkout(commit.hash, git_dir, work_dir,
force=True)
else:
- commit = self.builder.commit # Ick, fix this for BuildCommits()
+ commit = 'current'
# Set up the environment and command line
env = self.toolchain.MakeEnvironment()
Mkdir(out_dir)
args = []
+ cwd = work_dir
if not self.builder.in_tree:
- args.append('O=build')
+ if commit_upto is None:
+ # In this case we are building in the original source
+ # directory (i.e. the current directory where buildman
+ # is invoked. The output directory is set to this
+ # thread's selected work directory.
+ #
+ # Symlinks can confuse U-Boot's Makefile since
+ # we may use '..' in our path, so remove them.
+ work_dir = os.path.realpath(work_dir)
+ args.append('O=%s/build' % work_dir)
+ cwd = None
+ else:
+ args.append('O=build')
args.append('-s')
if self.builder.num_jobs is not None:
args.extend(['-j', str(self.builder.num_jobs)])
@@ -272,14 +285,14 @@ class BuilderThread(threading.Thread):
# If we need to reconfigure, do that now
if do_config:
- result = self.Make(commit, brd, 'distclean', work_dir,
+ result = self.Make(commit, brd, 'distclean', cwd,
'distclean', *args, env=env)
- result = self.Make(commit, brd, 'config', work_dir,
+ result = self.Make(commit, brd, 'config', cwd,
*(args + config_args), env=env)
config_out = result.combined
do_config = False # No need to configure next time
if result.return_code == 0:
- result = self.Make(commit, brd, 'build', work_dir, *args,
+ result = self.Make(commit, brd, 'build', cwd, *args,
env=env)
result.stdout = config_out + result.stdout
else:
@@ -478,8 +491,10 @@ class BuilderThread(threading.Thread):
self.builder.out_queue.put(result)
else:
# Just build the currently checked-out build
- result = self.RunCommit(None, True)
- result.commit_upto = self.builder.upto
+ result, request_config = self.RunCommit(None, brd, work_dir, True,
+ True, self.builder.force_build_failures)
+ result.commit_upto = 0
+ self._WriteResult(result, job.keep_outputs)
self.builder.out_queue.put(result)
def run(self):
@@ -491,12 +506,16 @@ class BuilderThread(threading.Thread):
alive = True
while True:
job = self.builder.queue.get()
+ if self.builder.active and alive:
+ self.RunJob(job)
+ '''
try:
if self.builder.active and alive:
self.RunJob(job)
except Exception as err:
alive = False
print err
+ '''
self.builder.queue.task_done()
@@ -763,10 +782,13 @@ class Builder:
Args:
commit_upto: Commit number to use (0..self.count-1)
"""
- commit = self.commits[commit_upto]
- subject = commit.subject.translate(trans_valid_chars)
- commit_dir = ('%02d_of_%02d_g%s_%s' % (commit_upto + 1,
- self.commit_count, commit.hash, subject[:20]))
+ if self.commits:
+ commit = self.commits[commit_upto]
+ subject = commit.subject.translate(trans_valid_chars)
+ commit_dir = ('%02d_of_%02d_g%s_%s' % (commit_upto + 1,
+ self.commit_count, commit.hash, subject[:20]))
+ else:
+ commit_dir = 'current'
output_dir = os.path.join(self.base_dir, commit_dir)
return output_dir
@@ -1308,14 +1330,18 @@ class Builder:
show_detail: Show detail for each board
show_bloat: Show detail for each function
"""
- self.commit_count = len(commits)
+ self.commit_count = len(commits) if commits else 1
self.commits = commits
self.ResetResultSummary(board_selected)
for commit_upto in range(0, self.commit_count, self._step):
board_dict, err_lines = self.GetResultSummary(board_selected,
commit_upto, read_func_sizes=show_bloat)
- msg = '%02d: %s' % (commit_upto + 1, commits[commit_upto].subject)
+ if commits:
+ msg = '%02d: %s' % (commit_upto + 1,
+ commits[commit_upto].subject)
+ else:
+ msg = 'current'
print self.col.Color(self.col.BLUE, msg)
self.PrintResultSummary(board_selected, board_dict,
err_lines if show_errors else [], show_sizes, show_detail,
@@ -1330,7 +1356,7 @@ class Builder:
commits: Selected commits to build
"""
# First work out how many commits we will build
- count = (len(commits) + self._step - 1) / self._step
+ count = (self.commit_count + self._step - 1) / self._step
self.count = len(board_selected) * count
self.upto = self.warned = self.fail = 0
self._timestamps = collections.deque()
@@ -1377,13 +1403,14 @@ class Builder:
"""
return os.path.join(self._working_dir, '%02d' % thread_num)
- def _PrepareThread(self, thread_num):
+ def _PrepareThread(self, thread_num, setup_git):
"""Prepare the working directory for a thread.
This clones or fetches the repo into the thread's work directory.
Args:
thread_num: Thread number (0, 1, ...)
+ setup_git: True to set up a git repo clone
"""
thread_dir = self.GetThreadDir(thread_num)
Mkdir(thread_dir)
@@ -1392,7 +1419,7 @@ class Builder:
# Clone the repo if it doesn't already exist
# TODO(sjg at chromium): Perhaps some git hackery to symlink instead, so
# we have a private index but uses the origin repo's contents?
- if self.git_dir:
+ if setup_git and self.git_dir:
src_dir = os.path.abspath(self.git_dir)
if os.path.exists(git_dir):
gitutil.Fetch(git_dir, thread_dir)
@@ -1400,17 +1427,18 @@ class Builder:
print 'Cloning repo for thread %d' % thread_num
gitutil.Clone(src_dir, thread_dir)
- def _PrepareWorkingSpace(self, max_threads):
+ def _PrepareWorkingSpace(self, max_threads, setup_git):
"""Prepare the working directory for use.
Set up the git repo for each thread.
Args:
max_threads: Maximum number of threads we expect to need.
+ setup_git: True to set up a git repo clone
"""
Mkdir(self._working_dir)
for thread in range(max_threads):
- self._PrepareThread(thread)
+ self._PrepareThread(thread, setup_git)
def _PrepareOutputSpace(self):
"""Get the output directories ready to receive files.
@@ -1437,12 +1465,13 @@ class Builder:
show_errors: True to show summarised error/warning info
keep_outputs: True to save build output files
"""
- self.commit_count = len(commits)
+ self.commit_count = len(commits) if commits else 1
self.commits = commits
self.ResetResultSummary(board_selected)
Mkdir(self.base_dir)
- self._PrepareWorkingSpace(min(self.num_threads, len(board_selected)))
+ self._PrepareWorkingSpace(min(self.num_threads, len(board_selected)),
+ commits is not None)
self._PrepareOutputSpace()
self.SetupBuild(board_selected, commits)
self.ProcessResult(None)
diff --git a/tools/buildman/control.py b/tools/buildman/control.py
index 75b6498..12a9699 100644
--- a/tools/buildman/control.py
+++ b/tools/buildman/control.py
@@ -21,15 +21,20 @@ def GetPlural(count):
"""Returns a plural 's' if count is not 1"""
return 's' if count != 1 else ''
-def GetActionSummary(is_summary, count, selected, options):
+def GetActionSummary(is_summary, commits, selected, options):
"""Return a string summarising the intended action.
Returns:
Summary string.
"""
- count = (count + options.step - 1) / options.step
- str = '%s %d commit%s for %d boards' % (
- 'Summary of' if is_summary else 'Building', count, GetPlural(count),
+ if commits:
+ count = len(commits)
+ count = (count + options.step - 1) / options.step
+ commit_str = '%d commit%s' % (count, GetPlural(count))
+ else:
+ commit_str = 'current source'
+ str = '%s %s for %d boards' % (
+ 'Summary of' if is_summary else 'Building', commit_str,
len(selected))
str += ' (%d thread%s, %d job%s per thread)' % (options.threads,
GetPlural(options.threads), options.jobs, GetPlural(options.jobs))
@@ -53,13 +58,18 @@ def ShowActions(series, why_selected, boards_selected, builder, options):
col = terminal.Color()
print 'Dry run, so not doing much. But I would do this:'
print
- print GetActionSummary(False, len(series.commits), boards_selected,
+ if series:
+ commits = series.commits
+ else:
+ commits = None
+ print GetActionSummary(False, commits, boards_selected,
options)
print 'Build directory: %s' % builder.base_dir
- for upto in range(0, len(series.commits), options.step):
- commit = series.commits[upto]
- print ' ', col.Color(col.YELLOW, commit.hash, bright=False),
- print commit.subject
+ if commits:
+ for upto in range(0, len(series.commits), options.step):
+ commit = series.commits[upto]
+ print ' ', col.Color(col.YELLOW, commit.hash, bright=False),
+ print commit.subject
print
for arg in why_selected:
if arg != 'all':
@@ -93,15 +103,16 @@ def DoBuildman(options, args):
count = options.count
if count == -1:
if not options.branch:
- str = 'Please use -b to specify a branch to build'
- print col.Color(col.RED, str)
- sys.exit(1)
- count = gitutil.CountCommitsInBranch(options.git_dir, options.branch)
- if count is None:
- str = "Branch '%s' not found or has no upstream" % options.branch
- print col.Color(col.RED, str)
- sys.exit(1)
- count += 1 # Build upstream commit also
+ count = 1
+ else:
+ count = gitutil.CountCommitsInBranch(options.git_dir,
+ options.branch)
+ if count is None:
+ str = ("Branch '%s' not found or has no upstream" %
+ options.branch)
+ print col.Color(col.RED, str)
+ sys.exit(1)
+ count += 1 # Build upstream commit also
if not count:
str = ("No commits found to process in branch '%s': "
@@ -132,17 +143,21 @@ def DoBuildman(options, args):
# upstream/master~..branch but that isn't possible if upstream/master is
# a merge commit (it will list all the commits that form part of the
# merge)
- range_expr = gitutil.GetRangeInBranch(options.git_dir, options.branch)
- upstream_commit = gitutil.GetUpstream(options.git_dir, options.branch)
- series = patchstream.GetMetaDataForList(upstream_commit, options.git_dir,
- 1)
- # Conflicting tags are not a problem for buildman, since it does not use
- # them. For example, Series-version is not useful for buildman. On the
- # other hand conflicting tags will cause an error. So allow later tags
- # to overwrite earlier ones.
- series.allow_overwrite = True
- series = patchstream.GetMetaDataForList(range_expr, options.git_dir, None,
- series)
+ if options.branch:
+ range_expr = gitutil.GetRangeInBranch(options.git_dir, options.branch)
+ upstream_commit = gitutil.GetUpstream(options.git_dir, options.branch)
+ series = patchstream.GetMetaDataForList(upstream_commit,
+ options.git_dir, 1)
+
+ # Conflicting tags are not a problem for buildman, since it does not
+ # use them. For example, Series-version is not useful for buildman. On
+ # the other hand conflicting tags will cause an error. So allow later
+ # tags to overwrite earlier ones.
+ series.allow_overwrite = True
+ series = patchstream.GetMetaDataForList(range_expr, options.git_dir, None,
+ series)
+ else:
+ series = None
# By default we have one thread per CPU. But if there are not enough jobs
# we can have fewer threads and use a high '-j' value for make.
@@ -162,7 +177,11 @@ def DoBuildman(options, args):
sys.exit(1)
# Create a new builder with the selected options
- output_dir = os.path.join(options.output_dir, options.branch)
+ if options.branch:
+ dirname = options.branch
+ else:
+ dirname = 'current'
+ output_dir = os.path.join(options.output_dir, dirname)
builder = Builder(toolchains, output_dir, options.git_dir,
options.threads, options.jobs, gnu_make=gnu_make, checkout=True,
show_unknown=options.show_unknown, step=options.step)
@@ -180,15 +199,21 @@ def DoBuildman(options, args):
# Work out which boards to build
board_selected = boards.GetSelectedDict()
- print GetActionSummary(options.summary, count, board_selected, options)
+ if series:
+ commits = series.commits
+ else:
+ commits = None
+
+ print GetActionSummary(options.summary, commits, board_selected,
+ options)
if options.summary:
# We can't show function sizes without board details at present
if options.show_bloat:
options.show_detail = True
- builder.ShowSummary(series.commits, board_selected,
+ builder.ShowSummary(commits, board_selected,
options.show_errors, options.show_sizes,
options.show_detail, options.show_bloat)
else:
- builder.BuildBoards(series.commits, board_selected,
+ builder.BuildBoards(commits, board_selected,
options.show_errors, options.keep_outputs)
--
2.0.0.526.g5318336
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 04/13] buildman: Move BuilderThread code to its own file
2014-08-05 14:46 [U-Boot] [PATCH v4 0/13] Add some missing buildman features and deprecate MAKEALL Simon Glass
` (2 preceding siblings ...)
2014-08-05 14:46 ` [U-Boot] [PATCH v4 03/13] buildman: Allow building of current source tree Simon Glass
@ 2014-08-05 14:46 ` Simon Glass
2014-08-05 14:46 ` [U-Boot] [PATCH v4 05/13] buildman: Sort command line options Simon Glass
` (8 subsequent siblings)
12 siblings, 0 replies; 42+ messages in thread
From: Simon Glass @ 2014-08-05 14:46 UTC (permalink / raw)
To: u-boot
The builder.py file is getting too long, so split out some code.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v4:
- Fix missing import in builder.py
Changes in v3:
- Add new patch to move BuilderThread code to its own file
Changes in v2: None
tools/buildman/builder.py | 439 +---------------------------------------
tools/buildman/builderthread.py | 434 +++++++++++++++++++++++++++++++++++++++
2 files changed, 442 insertions(+), 431 deletions(-)
create mode 100644 tools/buildman/builderthread.py
diff --git a/tools/buildman/builder.py b/tools/buildman/builder.py
index d2b72d5..e2da0eb 100644
--- a/tools/buildman/builder.py
+++ b/tools/buildman/builder.py
@@ -6,7 +6,6 @@
#
import collections
-import errno
from datetime import datetime, timedelta
import glob
import os
@@ -15,9 +14,9 @@ import Queue
import shutil
import string
import sys
-import threading
import time
+import builderthread
import command
import gitutil
import terminal
@@ -97,428 +96,6 @@ OUTCOME_OK, OUTCOME_WARNING, OUTCOME_ERROR, OUTCOME_UNKNOWN = range(4)
trans_valid_chars = string.maketrans("/: ", "---")
-def Mkdir(dirname):
- """Make a directory if it doesn't already exist.
-
- Args:
- dirname: Directory to create
- """
- try:
- os.mkdir(dirname)
- except OSError as err:
- if err.errno == errno.EEXIST:
- pass
- else:
- raise
-
-class BuilderJob:
- """Holds information about a job to be performed by a thread
-
- Members:
- board: Board object to build
- commits: List of commit options to build.
- """
- def __init__(self):
- self.board = None
- self.commits = []
-
-
-class ResultThread(threading.Thread):
- """This thread processes results from builder threads.
-
- It simply passes the results on to the builder. There is only one
- result thread, and this helps to serialise the build output.
- """
- def __init__(self, builder):
- """Set up a new result thread
-
- Args:
- builder: Builder which will be sent each result
- """
- threading.Thread.__init__(self)
- self.builder = builder
-
- def run(self):
- """Called to start up the result thread.
-
- We collect the next result job and pass it on to the build.
- """
- while True:
- result = self.builder.out_queue.get()
- self.builder.ProcessResult(result)
- self.builder.out_queue.task_done()
-
-
-class BuilderThread(threading.Thread):
- """This thread builds U-Boot for a particular board.
-
- An input queue provides each new job. We run 'make' to build U-Boot
- and then pass the results on to the output queue.
-
- Members:
- builder: The builder which contains information we might need
- thread_num: Our thread number (0-n-1), used to decide on a
- temporary directory
- """
- def __init__(self, builder, thread_num):
- """Set up a new builder thread"""
- threading.Thread.__init__(self)
- self.builder = builder
- self.thread_num = thread_num
-
- def Make(self, commit, brd, stage, cwd, *args, **kwargs):
- """Run 'make' on a particular commit and board.
-
- The source code will already be checked out, so the 'commit'
- argument is only for information.
-
- Args:
- commit: Commit object that is being built
- brd: Board object that is being built
- stage: Stage of the build. Valid stages are:
- distclean - can be called to clean source
- config - called to configure for a board
- build - the main make invocation - it does the build
- args: A list of arguments to pass to 'make'
- kwargs: A list of keyword arguments to pass to command.RunPipe()
-
- Returns:
- CommandResult object
- """
- return self.builder.do_make(commit, brd, stage, cwd, *args,
- **kwargs)
-
- def RunCommit(self, commit_upto, brd, work_dir, do_config, force_build,
- force_build_failures):
- """Build a particular commit.
-
- If the build is already done, and we are not forcing a build, we skip
- the build and just return the previously-saved results.
-
- Args:
- commit_upto: Commit number to build (0...n-1)
- brd: Board object to build
- work_dir: Directory to which the source will be checked out
- do_config: True to run a make <board>_defconfig on the source
- force_build: Force a build even if one was previously done
- force_build_failures: Force a bulid if the previous result showed
- failure
-
- Returns:
- tuple containing:
- - CommandResult object containing the results of the build
- - boolean indicating whether 'make config' is still needed
- """
- # Create a default result - it will be overwritte by the call to
- # self.Make() below, in the event that we do a build.
- result = command.CommandResult()
- result.return_code = 0
- if self.builder.in_tree:
- out_dir = work_dir
- else:
- out_dir = os.path.join(work_dir, 'build')
-
- # Check if the job was already completed last time
- done_file = self.builder.GetDoneFile(commit_upto, brd.target)
- result.already_done = os.path.exists(done_file)
- will_build = (force_build or force_build_failures or
- not result.already_done)
- if result.already_done and will_build:
- # Get the return code from that build and use it
- with open(done_file, 'r') as fd:
- result.return_code = int(fd.readline())
- err_file = self.builder.GetErrFile(commit_upto, brd.target)
- if os.path.exists(err_file) and os.stat(err_file).st_size:
- result.stderr = 'bad'
- elif not force_build:
- # The build passed, so no need to build it again
- will_build = False
-
- if will_build:
- # We are going to have to build it. First, get a toolchain
- if not self.toolchain:
- try:
- self.toolchain = self.builder.toolchains.Select(brd.arch)
- except ValueError as err:
- result.return_code = 10
- result.stdout = ''
- result.stderr = str(err)
- # TODO(sjg at chromium.org): This gets swallowed, but needs
- # to be reported.
-
- if self.toolchain:
- # Checkout the right commit
- if self.builder.commits:
- commit = self.builder.commits[commit_upto]
- if self.builder.checkout:
- git_dir = os.path.join(work_dir, '.git')
- gitutil.Checkout(commit.hash, git_dir, work_dir,
- force=True)
- else:
- commit = 'current'
-
- # Set up the environment and command line
- env = self.toolchain.MakeEnvironment()
- Mkdir(out_dir)
- args = []
- cwd = work_dir
- if not self.builder.in_tree:
- if commit_upto is None:
- # In this case we are building in the original source
- # directory (i.e. the current directory where buildman
- # is invoked. The output directory is set to this
- # thread's selected work directory.
- #
- # Symlinks can confuse U-Boot's Makefile since
- # we may use '..' in our path, so remove them.
- work_dir = os.path.realpath(work_dir)
- args.append('O=%s/build' % work_dir)
- cwd = None
- else:
- args.append('O=build')
- args.append('-s')
- if self.builder.num_jobs is not None:
- args.extend(['-j', str(self.builder.num_jobs)])
- config_args = ['%s_defconfig' % brd.target]
- config_out = ''
- args.extend(self.builder.toolchains.GetMakeArguments(brd))
-
- # If we need to reconfigure, do that now
- if do_config:
- result = self.Make(commit, brd, 'distclean', cwd,
- 'distclean', *args, env=env)
- result = self.Make(commit, brd, 'config', cwd,
- *(args + config_args), env=env)
- config_out = result.combined
- do_config = False # No need to configure next time
- if result.return_code == 0:
- result = self.Make(commit, brd, 'build', cwd, *args,
- env=env)
- result.stdout = config_out + result.stdout
- else:
- result.return_code = 1
- result.stderr = 'No tool chain for %s\n' % brd.arch
- result.already_done = False
-
- result.toolchain = self.toolchain
- result.brd = brd
- result.commit_upto = commit_upto
- result.out_dir = out_dir
- return result, do_config
-
- def _WriteResult(self, result, keep_outputs):
- """Write a built result to the output directory.
-
- Args:
- result: CommandResult object containing result to write
- keep_outputs: True to store the output binaries, False
- to delete them
- """
- # Fatal error
- if result.return_code < 0:
- return
-
- # Aborted?
- if result.stderr and 'No child processes' in result.stderr:
- return
-
- if result.already_done:
- return
-
- # Write the output and stderr
- output_dir = self.builder._GetOutputDir(result.commit_upto)
- Mkdir(output_dir)
- build_dir = self.builder.GetBuildDir(result.commit_upto,
- result.brd.target)
- Mkdir(build_dir)
-
- outfile = os.path.join(build_dir, 'log')
- with open(outfile, 'w') as fd:
- if result.stdout:
- fd.write(result.stdout)
-
- errfile = self.builder.GetErrFile(result.commit_upto,
- result.brd.target)
- if result.stderr:
- with open(errfile, 'w') as fd:
- fd.write(result.stderr)
- elif os.path.exists(errfile):
- os.remove(errfile)
-
- if result.toolchain:
- # Write the build result and toolchain information.
- done_file = self.builder.GetDoneFile(result.commit_upto,
- result.brd.target)
- with open(done_file, 'w') as fd:
- fd.write('%s' % result.return_code)
- with open(os.path.join(build_dir, 'toolchain'), 'w') as fd:
- print >>fd, 'gcc', result.toolchain.gcc
- print >>fd, 'path', result.toolchain.path
- print >>fd, 'cross', result.toolchain.cross
- print >>fd, 'arch', result.toolchain.arch
- fd.write('%s' % result.return_code)
-
- with open(os.path.join(build_dir, 'toolchain'), 'w') as fd:
- print >>fd, 'gcc', result.toolchain.gcc
- print >>fd, 'path', result.toolchain.path
-
- # Write out the image and function size information and an objdump
- env = result.toolchain.MakeEnvironment()
- lines = []
- for fname in ['u-boot', 'spl/u-boot-spl']:
- cmd = ['%snm' % self.toolchain.cross, '--size-sort', fname]
- nm_result = command.RunPipe([cmd], capture=True,
- capture_stderr=True, cwd=result.out_dir,
- raise_on_error=False, env=env)
- if nm_result.stdout:
- nm = self.builder.GetFuncSizesFile(result.commit_upto,
- result.brd.target, fname)
- with open(nm, 'w') as fd:
- print >>fd, nm_result.stdout,
-
- cmd = ['%sobjdump' % self.toolchain.cross, '-h', fname]
- dump_result = command.RunPipe([cmd], capture=True,
- capture_stderr=True, cwd=result.out_dir,
- raise_on_error=False, env=env)
- rodata_size = ''
- if dump_result.stdout:
- objdump = self.builder.GetObjdumpFile(result.commit_upto,
- result.brd.target, fname)
- with open(objdump, 'w') as fd:
- print >>fd, dump_result.stdout,
- for line in dump_result.stdout.splitlines():
- fields = line.split()
- if len(fields) > 5 and fields[1] == '.rodata':
- rodata_size = fields[2]
-
- cmd = ['%ssize' % self.toolchain.cross, fname]
- size_result = command.RunPipe([cmd], capture=True,
- capture_stderr=True, cwd=result.out_dir,
- raise_on_error=False, env=env)
- if size_result.stdout:
- lines.append(size_result.stdout.splitlines()[1] + ' ' +
- rodata_size)
-
- # Write out the image sizes file. This is similar to the output
- # of binutil's 'size' utility, but it omits the header line and
- # adds an additional hex value at the end of each line for the
- # rodata size
- if len(lines):
- sizes = self.builder.GetSizesFile(result.commit_upto,
- result.brd.target)
- with open(sizes, 'w') as fd:
- print >>fd, '\n'.join(lines)
-
- # Now write the actual build output
- if keep_outputs:
- patterns = ['u-boot', '*.bin', 'u-boot.dtb', '*.map',
- 'include/autoconf.mk', 'spl/u-boot-spl',
- 'spl/u-boot-spl.bin']
- for pattern in patterns:
- file_list = glob.glob(os.path.join(result.out_dir, pattern))
- for fname in file_list:
- shutil.copy(fname, build_dir)
-
-
- def RunJob(self, job):
- """Run a single job
-
- A job consists of a building a list of commits for a particular board.
-
- Args:
- job: Job to build
- """
- brd = job.board
- work_dir = self.builder.GetThreadDir(self.thread_num)
- self.toolchain = None
- if job.commits:
- # Run 'make board_defconfig' on the first commit
- do_config = True
- commit_upto = 0
- force_build = False
- for commit_upto in range(0, len(job.commits), job.step):
- result, request_config = self.RunCommit(commit_upto, brd,
- work_dir, do_config,
- force_build or self.builder.force_build,
- self.builder.force_build_failures)
- failed = result.return_code or result.stderr
- did_config = do_config
- if failed and not do_config:
- # If our incremental build failed, try building again
- # with a reconfig.
- if self.builder.force_config_on_failure:
- result, request_config = self.RunCommit(commit_upto,
- brd, work_dir, True, True, False)
- did_config = True
- if not self.builder.force_reconfig:
- do_config = request_config
-
- # If we built that commit, then config is done. But if we got
- # an warning, reconfig next time to force it to build the same
- # files that created warnings this time. Otherwise an
- # incremental build may not build the same file, and we will
- # think that the warning has gone away.
- # We could avoid this by using -Werror everywhere...
- # For errors, the problem doesn't happen, since presumably
- # the build stopped and didn't generate output, so will retry
- # that file next time. So we could detect warnings and deal
- # with them specially here. For now, we just reconfigure if
- # anything goes work.
- # Of course this is substantially slower if there are build
- # errors/warnings (e.g. 2-3x slower even if only 10% of builds
- # have problems).
- if (failed and not result.already_done and not did_config and
- self.builder.force_config_on_failure):
- # If this build failed, try the next one with a
- # reconfigure.
- # Sometimes if the board_config.h file changes it can mess
- # with dependencies, and we get:
- # make: *** No rule to make target `include/autoconf.mk',
- # needed by `depend'.
- do_config = True
- force_build = True
- else:
- force_build = False
- if self.builder.force_config_on_failure:
- if failed:
- do_config = True
- result.commit_upto = commit_upto
- if result.return_code < 0:
- raise ValueError('Interrupt')
-
- # We have the build results, so output the result
- self._WriteResult(result, job.keep_outputs)
- self.builder.out_queue.put(result)
- else:
- # Just build the currently checked-out build
- result, request_config = self.RunCommit(None, brd, work_dir, True,
- True, self.builder.force_build_failures)
- result.commit_upto = 0
- self._WriteResult(result, job.keep_outputs)
- self.builder.out_queue.put(result)
-
- def run(self):
- """Our thread's run function
-
- This thread picks a job from the queue, runs it, and then goes to the
- next job.
- """
- alive = True
- while True:
- job = self.builder.queue.get()
- if self.builder.active and alive:
- self.RunJob(job)
- '''
- try:
- if self.builder.active and alive:
- self.RunJob(job)
- except Exception as err:
- alive = False
- print err
- '''
- self.builder.queue.task_done()
-
-
class Builder:
"""Class for building U-Boot for a particular commit.
@@ -639,13 +216,13 @@ class Builder:
self.queue = Queue.Queue()
self.out_queue = Queue.Queue()
for i in range(self.num_threads):
- t = BuilderThread(self, i)
+ t = builderthread.BuilderThread(self, i)
t.setDaemon(True)
t.start()
self.threads.append(t)
self.last_line_len = 0
- t = ResultThread(self)
+ t = builderthread.ResultThread(self)
t.setDaemon(True)
t.start()
self.threads.append(t)
@@ -1385,7 +962,7 @@ class Builder:
for self.commit_upto in range(self.commit_count):
self.SelectCommit(commits[self.commit_upto])
self.SelectOutputDir()
- Mkdir(self.output_dir)
+ builderthread.Mkdir(self.output_dir)
self.BuildBoardsForCommit(board_selected, keep_outputs)
board_dict, err_lines = self.GetResultSummary()
@@ -1413,7 +990,7 @@ class Builder:
setup_git: True to set up a git repo clone
"""
thread_dir = self.GetThreadDir(thread_num)
- Mkdir(thread_dir)
+ builderthread.Mkdir(thread_dir)
git_dir = os.path.join(thread_dir, '.git')
# Clone the repo if it doesn't already exist
@@ -1436,7 +1013,7 @@ class Builder:
max_threads: Maximum number of threads we expect to need.
setup_git: True to set up a git repo clone
"""
- Mkdir(self._working_dir)
+ builderthread.Mkdir(self._working_dir)
for thread in range(max_threads):
self._PrepareThread(thread, setup_git)
@@ -1469,7 +1046,7 @@ class Builder:
self.commits = commits
self.ResetResultSummary(board_selected)
- Mkdir(self.base_dir)
+ builderthread.Mkdir(self.base_dir)
self._PrepareWorkingSpace(min(self.num_threads, len(board_selected)),
commits is not None)
self._PrepareOutputSpace()
@@ -1478,7 +1055,7 @@ class Builder:
# Create jobs to build all commits for each board
for brd in board_selected.itervalues():
- job = BuilderJob()
+ job = builderthread.BuilderJob()
job.board = brd
job.commits = commits
job.keep_outputs = keep_outputs
diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py
new file mode 100644
index 0000000..32297e2
--- /dev/null
+++ b/tools/buildman/builderthread.py
@@ -0,0 +1,434 @@
+# Copyright (c) 2014 Google, Inc
+#
+# SPDX-License-Identifier: GPL-2.0+
+#
+
+import errno
+import glob
+import os
+import shutil
+import threading
+
+import command
+import gitutil
+
+def Mkdir(dirname):
+ """Make a directory if it doesn't already exist.
+
+ Args:
+ dirname: Directory to create
+ """
+ try:
+ os.mkdir(dirname)
+ except OSError as err:
+ if err.errno == errno.EEXIST:
+ pass
+ else:
+ raise
+
+class BuilderJob:
+ """Holds information about a job to be performed by a thread
+
+ Members:
+ board: Board object to build
+ commits: List of commit options to build.
+ """
+ def __init__(self):
+ self.board = None
+ self.commits = []
+
+
+class ResultThread(threading.Thread):
+ """This thread processes results from builder threads.
+
+ It simply passes the results on to the builder. There is only one
+ result thread, and this helps to serialise the build output.
+ """
+ def __init__(self, builder):
+ """Set up a new result thread
+
+ Args:
+ builder: Builder which will be sent each result
+ """
+ threading.Thread.__init__(self)
+ self.builder = builder
+
+ def run(self):
+ """Called to start up the result thread.
+
+ We collect the next result job and pass it on to the build.
+ """
+ while True:
+ result = self.builder.out_queue.get()
+ self.builder.ProcessResult(result)
+ self.builder.out_queue.task_done()
+
+
+class BuilderThread(threading.Thread):
+ """This thread builds U-Boot for a particular board.
+
+ An input queue provides each new job. We run 'make' to build U-Boot
+ and then pass the results on to the output queue.
+
+ Members:
+ builder: The builder which contains information we might need
+ thread_num: Our thread number (0-n-1), used to decide on a
+ temporary directory
+ """
+ def __init__(self, builder, thread_num):
+ """Set up a new builder thread"""
+ threading.Thread.__init__(self)
+ self.builder = builder
+ self.thread_num = thread_num
+
+ def Make(self, commit, brd, stage, cwd, *args, **kwargs):
+ """Run 'make' on a particular commit and board.
+
+ The source code will already be checked out, so the 'commit'
+ argument is only for information.
+
+ Args:
+ commit: Commit object that is being built
+ brd: Board object that is being built
+ stage: Stage of the build. Valid stages are:
+ distclean - can be called to clean source
+ config - called to configure for a board
+ build - the main make invocation - it does the build
+ args: A list of arguments to pass to 'make'
+ kwargs: A list of keyword arguments to pass to command.RunPipe()
+
+ Returns:
+ CommandResult object
+ """
+ return self.builder.do_make(commit, brd, stage, cwd, *args,
+ **kwargs)
+
+ def RunCommit(self, commit_upto, brd, work_dir, do_config, force_build,
+ force_build_failures):
+ """Build a particular commit.
+
+ If the build is already done, and we are not forcing a build, we skip
+ the build and just return the previously-saved results.
+
+ Args:
+ commit_upto: Commit number to build (0...n-1)
+ brd: Board object to build
+ work_dir: Directory to which the source will be checked out
+ do_config: True to run a make <board>_defconfig on the source
+ force_build: Force a build even if one was previously done
+ force_build_failures: Force a bulid if the previous result showed
+ failure
+
+ Returns:
+ tuple containing:
+ - CommandResult object containing the results of the build
+ - boolean indicating whether 'make config' is still needed
+ """
+ # Create a default result - it will be overwritte by the call to
+ # self.Make() below, in the event that we do a build.
+ result = command.CommandResult()
+ result.return_code = 0
+ if self.builder.in_tree:
+ out_dir = work_dir
+ else:
+ out_dir = os.path.join(work_dir, 'build')
+
+ # Check if the job was already completed last time
+ done_file = self.builder.GetDoneFile(commit_upto, brd.target)
+ result.already_done = os.path.exists(done_file)
+ will_build = (force_build or force_build_failures or
+ not result.already_done)
+ if result.already_done and will_build:
+ # Get the return code from that build and use it
+ with open(done_file, 'r') as fd:
+ result.return_code = int(fd.readline())
+ err_file = self.builder.GetErrFile(commit_upto, brd.target)
+ if os.path.exists(err_file) and os.stat(err_file).st_size:
+ result.stderr = 'bad'
+ elif not force_build:
+ # The build passed, so no need to build it again
+ will_build = False
+
+ if will_build:
+ # We are going to have to build it. First, get a toolchain
+ if not self.toolchain:
+ try:
+ self.toolchain = self.builder.toolchains.Select(brd.arch)
+ except ValueError as err:
+ result.return_code = 10
+ result.stdout = ''
+ result.stderr = str(err)
+ # TODO(sjg at chromium.org): This gets swallowed, but needs
+ # to be reported.
+
+ if self.toolchain:
+ # Checkout the right commit
+ if self.builder.commits:
+ commit = self.builder.commits[commit_upto]
+ if self.builder.checkout:
+ git_dir = os.path.join(work_dir, '.git')
+ gitutil.Checkout(commit.hash, git_dir, work_dir,
+ force=True)
+ else:
+ commit = 'current'
+
+ # Set up the environment and command line
+ env = self.toolchain.MakeEnvironment()
+ Mkdir(out_dir)
+ args = []
+ cwd = work_dir
+ if not self.builder.in_tree:
+ if commit_upto is None:
+ # In this case we are building in the original source
+ # directory (i.e. the current directory where buildman
+ # is invoked. The output directory is set to this
+ # thread's selected work directory.
+ #
+ # Symlinks can confuse U-Boot's Makefile since
+ # we may use '..' in our path, so remove them.
+ work_dir = os.path.realpath(work_dir)
+ args.append('O=%s/build' % work_dir)
+ cwd = None
+ else:
+ args.append('O=build')
+ args.append('-s')
+ if self.builder.num_jobs is not None:
+ args.extend(['-j', str(self.builder.num_jobs)])
+ config_args = ['%s_defconfig' % brd.target]
+ config_out = ''
+ args.extend(self.builder.toolchains.GetMakeArguments(brd))
+
+ # If we need to reconfigure, do that now
+ if do_config:
+ result = self.Make(commit, brd, 'distclean', cwd,
+ 'distclean', *args, env=env)
+ result = self.Make(commit, brd, 'config', cwd,
+ *(args + config_args), env=env)
+ config_out = result.combined
+ do_config = False # No need to configure next time
+ if result.return_code == 0:
+ result = self.Make(commit, brd, 'build', cwd, *args,
+ env=env)
+ result.stdout = config_out + result.stdout
+ else:
+ result.return_code = 1
+ result.stderr = 'No tool chain for %s\n' % brd.arch
+ result.already_done = False
+
+ result.toolchain = self.toolchain
+ result.brd = brd
+ result.commit_upto = commit_upto
+ result.out_dir = out_dir
+ return result, do_config
+
+ def _WriteResult(self, result, keep_outputs):
+ """Write a built result to the output directory.
+
+ Args:
+ result: CommandResult object containing result to write
+ keep_outputs: True to store the output binaries, False
+ to delete them
+ """
+ # Fatal error
+ if result.return_code < 0:
+ return
+
+ # Aborted?
+ if result.stderr and 'No child processes' in result.stderr:
+ return
+
+ if result.already_done:
+ return
+
+ # Write the output and stderr
+ output_dir = self.builder._GetOutputDir(result.commit_upto)
+ Mkdir(output_dir)
+ build_dir = self.builder.GetBuildDir(result.commit_upto,
+ result.brd.target)
+ Mkdir(build_dir)
+
+ outfile = os.path.join(build_dir, 'log')
+ with open(outfile, 'w') as fd:
+ if result.stdout:
+ fd.write(result.stdout)
+
+ errfile = self.builder.GetErrFile(result.commit_upto,
+ result.brd.target)
+ if result.stderr:
+ with open(errfile, 'w') as fd:
+ fd.write(result.stderr)
+ elif os.path.exists(errfile):
+ os.remove(errfile)
+
+ if result.toolchain:
+ # Write the build result and toolchain information.
+ done_file = self.builder.GetDoneFile(result.commit_upto,
+ result.brd.target)
+ with open(done_file, 'w') as fd:
+ fd.write('%s' % result.return_code)
+ with open(os.path.join(build_dir, 'toolchain'), 'w') as fd:
+ print >>fd, 'gcc', result.toolchain.gcc
+ print >>fd, 'path', result.toolchain.path
+ print >>fd, 'cross', result.toolchain.cross
+ print >>fd, 'arch', result.toolchain.arch
+ fd.write('%s' % result.return_code)
+
+ with open(os.path.join(build_dir, 'toolchain'), 'w') as fd:
+ print >>fd, 'gcc', result.toolchain.gcc
+ print >>fd, 'path', result.toolchain.path
+
+ # Write out the image and function size information and an objdump
+ env = result.toolchain.MakeEnvironment()
+ lines = []
+ for fname in ['u-boot', 'spl/u-boot-spl']:
+ cmd = ['%snm' % self.toolchain.cross, '--size-sort', fname]
+ nm_result = command.RunPipe([cmd], capture=True,
+ capture_stderr=True, cwd=result.out_dir,
+ raise_on_error=False, env=env)
+ if nm_result.stdout:
+ nm = self.builder.GetFuncSizesFile(result.commit_upto,
+ result.brd.target, fname)
+ with open(nm, 'w') as fd:
+ print >>fd, nm_result.stdout,
+
+ cmd = ['%sobjdump' % self.toolchain.cross, '-h', fname]
+ dump_result = command.RunPipe([cmd], capture=True,
+ capture_stderr=True, cwd=result.out_dir,
+ raise_on_error=False, env=env)
+ rodata_size = ''
+ if dump_result.stdout:
+ objdump = self.builder.GetObjdumpFile(result.commit_upto,
+ result.brd.target, fname)
+ with open(objdump, 'w') as fd:
+ print >>fd, dump_result.stdout,
+ for line in dump_result.stdout.splitlines():
+ fields = line.split()
+ if len(fields) > 5 and fields[1] == '.rodata':
+ rodata_size = fields[2]
+
+ cmd = ['%ssize' % self.toolchain.cross, fname]
+ size_result = command.RunPipe([cmd], capture=True,
+ capture_stderr=True, cwd=result.out_dir,
+ raise_on_error=False, env=env)
+ if size_result.stdout:
+ lines.append(size_result.stdout.splitlines()[1] + ' ' +
+ rodata_size)
+
+ # Write out the image sizes file. This is similar to the output
+ # of binutil's 'size' utility, but it omits the header line and
+ # adds an additional hex value at the end of each line for the
+ # rodata size
+ if len(lines):
+ sizes = self.builder.GetSizesFile(result.commit_upto,
+ result.brd.target)
+ with open(sizes, 'w') as fd:
+ print >>fd, '\n'.join(lines)
+
+ # Now write the actual build output
+ if keep_outputs:
+ patterns = ['u-boot', '*.bin', 'u-boot.dtb', '*.map',
+ 'include/autoconf.mk', 'spl/u-boot-spl',
+ 'spl/u-boot-spl.bin']
+ for pattern in patterns:
+ file_list = glob.glob(os.path.join(result.out_dir, pattern))
+ for fname in file_list:
+ shutil.copy(fname, build_dir)
+
+
+ def RunJob(self, job):
+ """Run a single job
+
+ A job consists of a building a list of commits for a particular board.
+
+ Args:
+ job: Job to build
+ """
+ brd = job.board
+ work_dir = self.builder.GetThreadDir(self.thread_num)
+ self.toolchain = None
+ if job.commits:
+ # Run 'make board_defconfig' on the first commit
+ do_config = True
+ commit_upto = 0
+ force_build = False
+ for commit_upto in range(0, len(job.commits), job.step):
+ result, request_config = self.RunCommit(commit_upto, brd,
+ work_dir, do_config,
+ force_build or self.builder.force_build,
+ self.builder.force_build_failures)
+ failed = result.return_code or result.stderr
+ did_config = do_config
+ if failed and not do_config:
+ # If our incremental build failed, try building again
+ # with a reconfig.
+ if self.builder.force_config_on_failure:
+ result, request_config = self.RunCommit(commit_upto,
+ brd, work_dir, True, True, False)
+ did_config = True
+ if not self.builder.force_reconfig:
+ do_config = request_config
+
+ # If we built that commit, then config is done. But if we got
+ # an warning, reconfig next time to force it to build the same
+ # files that created warnings this time. Otherwise an
+ # incremental build may not build the same file, and we will
+ # think that the warning has gone away.
+ # We could avoid this by using -Werror everywhere...
+ # For errors, the problem doesn't happen, since presumably
+ # the build stopped and didn't generate output, so will retry
+ # that file next time. So we could detect warnings and deal
+ # with them specially here. For now, we just reconfigure if
+ # anything goes work.
+ # Of course this is substantially slower if there are build
+ # errors/warnings (e.g. 2-3x slower even if only 10% of builds
+ # have problems).
+ if (failed and not result.already_done and not did_config and
+ self.builder.force_config_on_failure):
+ # If this build failed, try the next one with a
+ # reconfigure.
+ # Sometimes if the board_config.h file changes it can mess
+ # with dependencies, and we get:
+ # make: *** No rule to make target `include/autoconf.mk',
+ # needed by `depend'.
+ do_config = True
+ force_build = True
+ else:
+ force_build = False
+ if self.builder.force_config_on_failure:
+ if failed:
+ do_config = True
+ result.commit_upto = commit_upto
+ if result.return_code < 0:
+ raise ValueError('Interrupt')
+
+ # We have the build results, so output the result
+ self._WriteResult(result, job.keep_outputs)
+ self.builder.out_queue.put(result)
+ else:
+ # Just build the currently checked-out build
+ result, request_config = self.RunCommit(None, brd, work_dir, True,
+ True, self.builder.force_build_failures)
+ result.commit_upto = 0
+ self._WriteResult(result, job.keep_outputs)
+ self.builder.out_queue.put(result)
+
+ def run(self):
+ """Our thread's run function
+
+ This thread picks a job from the queue, runs it, and then goes to the
+ next job.
+ """
+ alive = True
+ while True:
+ job = self.builder.queue.get()
+ if self.builder.active and alive:
+ self.RunJob(job)
+ '''
+ try:
+ if self.builder.active and alive:
+ self.RunJob(job)
+ except Exception as err:
+ alive = False
+ print err
+ '''
+ self.builder.queue.task_done()
--
2.0.0.526.g5318336
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 05/13] buildman: Sort command line options
2014-08-05 14:46 [U-Boot] [PATCH v4 0/13] Add some missing buildman features and deprecate MAKEALL Simon Glass
` (3 preceding siblings ...)
2014-08-05 14:46 ` [U-Boot] [PATCH v4 04/13] buildman: Move BuilderThread code to its own file Simon Glass
@ 2014-08-05 14:46 ` Simon Glass
2014-08-05 14:46 ` [U-Boot] [PATCH v4 06/13] buildman: Refactor output options Simon Glass
` (7 subsequent siblings)
12 siblings, 0 replies; 42+ messages in thread
From: Simon Glass @ 2014-08-05 14:46 UTC (permalink / raw)
To: u-boot
These options have got slightly out of order. Fix them.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v4:
- Fix typo in commit message
Changes in v3:
- Add new patch to sort command line options
Changes in v2: None
tools/buildman/buildman.py | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/tools/buildman/buildman.py b/tools/buildman/buildman.py
index 42847ac..da6025a 100755
--- a/tools/buildman/buildman.py
+++ b/tools/buildman/buildman.py
@@ -70,6 +70,9 @@ parser.add_option('-c', '--count', dest='count', type='int',
parser.add_option('-C', '--force-reconfig', dest='force_reconfig',
action='store_true', default=False,
help='Reconfigure for every commit (disable incremental build)')
+parser.add_option('-d', '--detail', dest='show_detail',
+ action='store_true', default=False,
+ help='Show detailed information for each board in summary')
parser.add_option('-e', '--show_errors', action='store_true',
default=False, help='Show errors and warnings')
parser.add_option('-f', '--force-build', dest='force_build',
@@ -78,9 +81,6 @@ parser.add_option('-f', '--force-build', dest='force_build',
parser.add_option('-F', '--force-build-failures', dest='force_build_failures',
action='store_true', default=False,
help='Force build of previously-failed build')
-parser.add_option('-d', '--detail', dest='show_detail',
- action='store_true', default=False,
- help='Show detailed information for each board in summary')
parser.add_option('-g', '--git', type='string',
help='Git repo containing branch to build', default='.')
parser.add_option('-H', '--full-help', action='store_true', dest='full_help',
@@ -96,6 +96,9 @@ parser.add_option('--list-tool-chains', action='store_true', default=False,
help='List available tool chains')
parser.add_option('-n', '--dry-run', action='store_true', dest='dry_run',
default=False, help="Do a try run (describe actions, but no nothing)")
+parser.add_option('-o', '--output-dir', type='string',
+ dest='output_dir', default='..',
+ help='Directory where all builds happen and buildman has its workspace (default is ../)')
parser.add_option('-Q', '--quick', action='store_true',
default=False, help='Do a rough build, with limited warning resolution')
parser.add_option('-s', '--summary', action='store_true',
@@ -110,9 +113,6 @@ parser.add_option('-T', '--threads', type='int',
default=None, help='Number of builder threads to use')
parser.add_option('-u', '--show_unknown', action='store_true',
default=False, help='Show boards with unknown build result')
-parser.add_option('-o', '--output-dir', type='string',
- dest='output_dir', default='..',
- help='Directory where all builds happen and buildman has its workspace (default is ../)')
parser.usage = """buildman -b <branch> [options]
--
2.0.0.526.g5318336
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 06/13] buildman: Refactor output options
2014-08-05 14:46 [U-Boot] [PATCH v4 0/13] Add some missing buildman features and deprecate MAKEALL Simon Glass
` (4 preceding siblings ...)
2014-08-05 14:46 ` [U-Boot] [PATCH v4 05/13] buildman: Sort command line options Simon Glass
@ 2014-08-05 14:46 ` Simon Glass
2014-08-05 14:46 ` [U-Boot] [PATCH v4 07/13] buildman: Add verbose option to display errors as they happen Simon Glass
` (6 subsequent siblings)
12 siblings, 0 replies; 42+ messages in thread
From: Simon Glass @ 2014-08-05 14:46 UTC (permalink / raw)
To: u-boot
We need the output options to be available in several places. It's a pain
to pass them into each function. Make them properties of the builder and
add a single function to set them up. At the same time, add a function which
produces summary output using these options.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v4: None
Changes in v3:
- Add new patch to refactor output options
Changes in v2: None
tools/buildman/builder.py | 46 +++++++++++++++++++++++++++-------------------
tools/buildman/control.py | 8 ++++----
2 files changed, 31 insertions(+), 23 deletions(-)
diff --git a/tools/buildman/builder.py b/tools/buildman/builder.py
index e2da0eb..f3d0435 100644
--- a/tools/buildman/builder.py
+++ b/tools/buildman/builder.py
@@ -235,6 +235,20 @@ class Builder:
for t in self.threads:
del t
+ def SetDisplayOptions(self, show_errors, show_sizes, show_detail,
+ show_bloat):
+ """Setup display options for the builder.
+
+ show_errors: True to show summarised error/warning info
+ show_sizes: Show size deltas
+ show_detail: Show detail for each board
+ show_bloat: Show detail for each function
+ """
+ self._show_errors = show_errors
+ self._show_sizes = show_sizes
+ self._show_detail = show_detail
+ self._show_bloat = show_bloat
+
def _AddTimestamp(self):
"""Add a new timestamp to the list and record the build period.
@@ -891,9 +905,18 @@ class Builder:
print "Boards not built (%d): %s" % (len(not_built),
', '.join(not_built))
+ def ProduceResultSummary(self, commit_upto, commits, board_selected):
+ board_dict, err_lines = self.GetResultSummary(board_selected,
+ commit_upto, read_func_sizes=self._show_bloat)
+ if commits:
+ msg = '%02d: %s' % (commit_upto + 1,
+ commits[commit_upto].subject)
+ print self.col.Color(self.col.BLUE, msg)
+ self.PrintResultSummary(board_selected, board_dict,
+ err_lines if self._show_errors else [],
+ self._show_sizes, self._show_detail, self._show_bloat)
- def ShowSummary(self, commits, board_selected, show_errors, show_sizes,
- show_detail, show_bloat):
+ def ShowSummary(self, commits, board_selected):
"""Show a build summary for U-Boot for a given board list.
Reset the result summary, then repeatedly call GetResultSummary on
@@ -902,27 +925,13 @@ class Builder:
Args:
commit: Commit objects to summarise
board_selected: Dict containing boards to summarise
- show_errors: Show errors that occured
- show_sizes: Show size deltas
- show_detail: Show detail for each board
- show_bloat: Show detail for each function
"""
self.commit_count = len(commits) if commits else 1
self.commits = commits
self.ResetResultSummary(board_selected)
for commit_upto in range(0, self.commit_count, self._step):
- board_dict, err_lines = self.GetResultSummary(board_selected,
- commit_upto, read_func_sizes=show_bloat)
- if commits:
- msg = '%02d: %s' % (commit_upto + 1,
- commits[commit_upto].subject)
- else:
- msg = 'current'
- print self.col.Color(self.col.BLUE, msg)
- self.PrintResultSummary(board_selected, board_dict,
- err_lines if show_errors else [], show_sizes, show_detail,
- show_bloat)
+ self.ProduceResultSummary(commit_upto, commits, board_selected)
def SetupBuild(self, board_selected, commits):
@@ -1032,14 +1041,13 @@ class Builder:
if dirname not in dir_list:
shutil.rmtree(dirname)
- def BuildBoards(self, commits, board_selected, show_errors, keep_outputs):
+ def BuildBoards(self, commits, board_selected, keep_outputs):
"""Build all commits for a list of boards
Args:
commits: List of commits to be build, each a Commit object
boards_selected: Dict of selected boards, key is target name,
value is Board object
- show_errors: True to show summarised error/warning info
keep_outputs: True to save build output files
"""
self.commit_count = len(commits) if commits else 1
diff --git a/tools/buildman/control.py b/tools/buildman/control.py
index 12a9699..d8fa74b 100644
--- a/tools/buildman/control.py
+++ b/tools/buildman/control.py
@@ -207,13 +207,13 @@ def DoBuildman(options, args):
print GetActionSummary(options.summary, commits, board_selected,
options)
+ builder.SetDisplayOptions(options.show_errors, options.show_sizes,
+ options.show_detail, options.show_bloat)
if options.summary:
# We can't show function sizes without board details at present
if options.show_bloat:
options.show_detail = True
- builder.ShowSummary(commits, board_selected,
- options.show_errors, options.show_sizes,
- options.show_detail, options.show_bloat)
+ builder.ShowSummary(commits, board_selected)
else:
builder.BuildBoards(commits, board_selected,
- options.show_errors, options.keep_outputs)
+ options.keep_outputs)
--
2.0.0.526.g5318336
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 07/13] buildman: Add verbose option to display errors as they happen
2014-08-05 14:46 [U-Boot] [PATCH v4 0/13] Add some missing buildman features and deprecate MAKEALL Simon Glass
` (5 preceding siblings ...)
2014-08-05 14:46 ` [U-Boot] [PATCH v4 06/13] buildman: Refactor output options Simon Glass
@ 2014-08-05 14:46 ` Simon Glass
2014-08-05 14:46 ` [U-Boot] [PATCH v4 08/13] buildman: Remove unused non-incremental build method code Simon Glass
` (5 subsequent siblings)
12 siblings, 0 replies; 42+ messages in thread
From: Simon Glass @ 2014-08-05 14:46 UTC (permalink / raw)
To: u-boot
Normally buildman operates in two passes - one to do the build and another
to summarise the errors. Add a verbose option (-v) to display build problems
as they happen. With -e also given, this will display errors too.
When building the current source tree (rather than a list of commits in a
branch), both -v and -e are enabled automatically.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v4: None
Changes in v3:
- Add new patch to add a verbose option
Changes in v2: None
tools/buildman/README | 46 ++++++++++++++++++++++++++++++++++++----------
tools/buildman/builder.py | 16 +++++++++++++---
tools/buildman/buildman.py | 2 ++
tools/buildman/control.py | 4 +++-
4 files changed, 54 insertions(+), 14 deletions(-)
diff --git a/tools/buildman/README b/tools/buildman/README
index 1c919af..6ba24c0 100644
--- a/tools/buildman/README
+++ b/tools/buildman/README
@@ -41,9 +41,10 @@ Theory of Operation
Buildman is a builder. It is not make, although it runs make. It does not
produce any useful output on the terminal while building, except for
-progress information. All the output (errors, warnings and binaries if you
-are ask for them) is stored in output directories, which you can look at
-while the build is progressing, or when it is finished.
+progress information (except with -v, see below). All the output (errors,
+warnings and binaries if you are ask for them) is stored in output
+directories, which you can look at while the build is progressing, or when
+it is finished.
Buildman produces a concise summary of which boards succeeded and failed.
It shows which commit introduced which board failure using a simple
@@ -77,12 +78,17 @@ Buildman automatically selects the correct tool chain for each board. You
must supply suitable tool chains, but buildman takes care of selecting the
right one.
-Buildman always builds a branch, and always builds the upstream commit as
-well, for comparison. It cannot build individual commits at present, unless
-(maybe) you point it at an empty branch. Put all your commits in a branch,
-set the branch's upstream to a valid value, and all will be well. Otherwise
-buildman will perform random actions. Use -n to check what the random
-actions might be.
+Buildman generally builds a branch (with the -b flag), and in this case
+builds the upstream commit as well, for comparison. It cannot build
+individual commits at present, unless (maybe) you point it at an empty
+branch. Put all your commits in a branch, set the branch's upstream to a
+valid value, and all will be well. Otherwise buildman will perform random
+actions. Use -n to check what the random actions might be.
+
+If you just want to build the current source tree, leave off the -b flag.
+This will display results and errors as they happen. You can still look
+at them later using -s. Note that buildman will assume that the source
+has changed, and will build all specified boards in this case.
Buildman is optimised for building many commits at once, for many boards.
On multi-core machines, Buildman is fast because it uses most of the
@@ -659,6 +665,15 @@ It is expected that any variables added are dealt with in U-Boot's
config.mk file and documented in the README.
+Quick Sanity Check
+==================
+
+If you have made changes and want to do a quick sanity check of the
+currently-checked-out source, run buildman without the -b flag. This will
+build the selected boards and display build status and errors as it runs
+(i.e. -v amd -e are enabled automatically).
+
+
Other options
=============
@@ -685,7 +700,15 @@ First you need to set up your tool chains - see the 'Setting up' section
for details. Once you have your required toolchain(s) detected then you are
ready to go.
-Buildman works on entire branches, so the normal use is:
+To build the current source tree, run buildman without a -b flag:
+
+ ./tools/buildman/buildman <list of things to build>
+
+This will build the current source tree for the given boards and display
+the results and errors.
+
+However buildman usually works on entire branches, and for that you must
+specify a board flag:
./tools/buildman/buildman -b <branch_name> <list of things to build>
@@ -698,6 +721,9 @@ buildman just shows a summary, with red indicating that a commit introduced
an error and green indicating that a commit fixed an error. Use the -e
flag to see the full errors.
+If you really want to see build results as they happen, use -v when doing a
+build (and -e if you want to see errors as well).
+
You don't need to stick around on that branch while buildman is running. It
checks out its own copy of the source code, so you can change branches,
add commits, etc. without affecting the build in progress.
diff --git a/tools/buildman/builder.py b/tools/buildman/builder.py
index f3d0435..af8f3f3 100644
--- a/tools/buildman/builder.py
+++ b/tools/buildman/builder.py
@@ -321,7 +321,8 @@ class Builder:
"""Process the result of a build, showing progress information
Args:
- result: A CommandResult object
+ result: A CommandResult object, which indicates the result for
+ a single build
"""
col = terminal.Color()
if result:
@@ -339,6 +340,13 @@ class Builder:
self.warned += 1
if result.already_done:
self.already_done += 1
+ if self._verbose:
+ print '\r',
+ self.ClearLine(0)
+ boards_selected = {target : result.brd}
+ self.ResetResultSummary(boards_selected)
+ self.ProduceResultSummary(result.commit_upto, self.commits,
+ boards_selected)
else:
target = '(starting)'
@@ -362,7 +370,7 @@ class Builder:
name += target
print line + name,
- length = 13 + len(name)
+ length = 14 + len(name)
self.ClearLine(length)
def _GetOutputDir(self, commit_upto):
@@ -1041,7 +1049,7 @@ class Builder:
if dirname not in dir_list:
shutil.rmtree(dirname)
- def BuildBoards(self, commits, board_selected, keep_outputs):
+ def BuildBoards(self, commits, board_selected, keep_outputs, verbose):
"""Build all commits for a list of boards
Args:
@@ -1049,9 +1057,11 @@ class Builder:
boards_selected: Dict of selected boards, key is target name,
value is Board object
keep_outputs: True to save build output files
+ verbose: Display build results as they are completed
"""
self.commit_count = len(commits) if commits else 1
self.commits = commits
+ self._verbose = verbose
self.ResetResultSummary(board_selected)
builderthread.Mkdir(self.base_dir)
diff --git a/tools/buildman/buildman.py b/tools/buildman/buildman.py
index da6025a..f32557f 100755
--- a/tools/buildman/buildman.py
+++ b/tools/buildman/buildman.py
@@ -113,6 +113,8 @@ parser.add_option('-T', '--threads', type='int',
default=None, help='Number of builder threads to use')
parser.add_option('-u', '--show_unknown', action='store_true',
default=False, help='Show boards with unknown build result')
+parser.add_option('-v', '--verbose', action='store_true',
+ default=False, help='Show build results while the build progresses')
parser.usage = """buildman -b <branch> [options]
diff --git a/tools/buildman/control.py b/tools/buildman/control.py
index d8fa74b..cc8593f 100644
--- a/tools/buildman/control.py
+++ b/tools/buildman/control.py
@@ -158,6 +158,8 @@ def DoBuildman(options, args):
series)
else:
series = None
+ options.verbose = True
+ options.show_errors = True
# By default we have one thread per CPU. But if there are not enough jobs
# we can have fewer threads and use a high '-j' value for make.
@@ -216,4 +218,4 @@ def DoBuildman(options, args):
builder.ShowSummary(commits, board_selected)
else:
builder.BuildBoards(commits, board_selected,
- options.keep_outputs)
+ options.keep_outputs, options.verbose)
--
2.0.0.526.g5318336
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 08/13] buildman: Remove unused non-incremental build method code
2014-08-05 14:46 [U-Boot] [PATCH v4 0/13] Add some missing buildman features and deprecate MAKEALL Simon Glass
` (6 preceding siblings ...)
2014-08-05 14:46 ` [U-Boot] [PATCH v4 07/13] buildman: Add verbose option to display errors as they happen Simon Glass
@ 2014-08-05 14:46 ` Simon Glass
2014-08-05 14:46 ` [U-Boot] [PATCH v4 09/13] buildman: Add an option to specify the buildman config file Simon Glass
` (4 subsequent siblings)
12 siblings, 0 replies; 42+ messages in thread
From: Simon Glass @ 2014-08-05 14:46 UTC (permalink / raw)
To: u-boot
The non-incremental build method is no longer used, so remove it.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v4:
- Add new patch to remove unused non-incremental build method code
Changes in v3: None
Changes in v2: None
tools/buildman/builder.py | 34 ----------------------------------
tools/buildman/test.py | 1 -
2 files changed, 35 deletions(-)
diff --git a/tools/buildman/builder.py b/tools/buildman/builder.py
index af8f3f3..f148b93 100644
--- a/tools/buildman/builder.py
+++ b/tools/buildman/builder.py
@@ -955,40 +955,6 @@ class Builder:
self.upto = self.warned = self.fail = 0
self._timestamps = collections.deque()
- def BuildBoardsForCommit(self, board_selected, keep_outputs):
- """Build all boards for a single commit"""
- self.SetupBuild(board_selected)
- self.count = len(board_selected)
- for brd in board_selected.itervalues():
- job = BuilderJob()
- job.board = brd
- job.commits = None
- job.keep_outputs = keep_outputs
- self.queue.put(brd)
-
- self.queue.join()
- self.out_queue.join()
- print
- self.ClearLine(0)
-
- def BuildCommits(self, commits, board_selected, show_errors, keep_outputs):
- """Build all boards for all commits (non-incremental)"""
- self.commit_count = len(commits)
-
- self.ResetResultSummary(board_selected)
- for self.commit_upto in range(self.commit_count):
- self.SelectCommit(commits[self.commit_upto])
- self.SelectOutputDir()
- builderthread.Mkdir(self.output_dir)
-
- self.BuildBoardsForCommit(board_selected, keep_outputs)
- board_dict, err_lines = self.GetResultSummary()
- self.PrintResultSummary(board_selected, board_dict,
- err_lines if show_errors else [])
-
- if self.already_done:
- print '%d builds already done' % self.already_done
-
def GetThreadDir(self, thread_num):
"""Get the directory path to the working dir for a thread.
diff --git a/tools/buildman/test.py b/tools/buildman/test.py
index 068784a..6f96c18 100644
--- a/tools/buildman/test.py
+++ b/tools/buildman/test.py
@@ -136,7 +136,6 @@ class TestBuild(unittest.TestCase):
build.do_make = self.Make
board_selected = self.boards.GetSelectedDict()
- #build.BuildCommits(self.commits, board_selected, False)
build.BuildBoards(self.commits, board_selected, False, False)
build.ShowSummary(self.commits, board_selected, True, False,
False, False)
--
2.0.0.526.g5318336
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 09/13] buildman: Add an option to specify the buildman config file
2014-08-05 14:46 [U-Boot] [PATCH v4 0/13] Add some missing buildman features and deprecate MAKEALL Simon Glass
` (7 preceding siblings ...)
2014-08-05 14:46 ` [U-Boot] [PATCH v4 08/13] buildman: Remove unused non-incremental build method code Simon Glass
@ 2014-08-05 14:46 ` Simon Glass
2014-08-05 14:47 ` [U-Boot] [PATCH v4 10/13] buildman: Add a message indicating there are no errors Simon Glass
` (3 subsequent siblings)
12 siblings, 0 replies; 42+ messages in thread
From: Simon Glass @ 2014-08-05 14:46 UTC (permalink / raw)
To: u-boot
Add a new --config-file option (-G) to specify a different configuration
file from the default ~/.buildman.
Reported-by: Tom Rini <trini@ti.com>
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v4:
- Add new patch to allow the config file to be specified
Changes in v3: None
Changes in v2: None
tools/buildman/buildman.py | 2 ++
tools/buildman/control.py | 2 +-
2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/tools/buildman/buildman.py b/tools/buildman/buildman.py
index f32557f..6ca8dc6 100755
--- a/tools/buildman/buildman.py
+++ b/tools/buildman/buildman.py
@@ -83,6 +83,8 @@ parser.add_option('-F', '--force-build-failures', dest='force_build_failures',
help='Force build of previously-failed build')
parser.add_option('-g', '--git', type='string',
help='Git repo containing branch to build', default='.')
+parser.add_option('-G', '--config-file', type='string',
+ help='Path to buildman config file', default='')
parser.add_option('-H', '--full-help', action='store_true', dest='full_help',
default=False, help='Display the README file')
parser.add_option('-i', '--in-tree', dest='in_tree',
diff --git a/tools/buildman/control.py b/tools/buildman/control.py
index cc8593f..0c9e2cb 100644
--- a/tools/buildman/control.py
+++ b/tools/buildman/control.py
@@ -86,7 +86,7 @@ def DoBuildman(options, args):
"""
gitutil.Setup()
- bsettings.Setup()
+ bsettings.Setup(options.config_file)
options.git_dir = os.path.join(options.git, '.git')
toolchains = toolchain.Toolchains()
--
2.0.0.526.g5318336
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 10/13] buildman: Add a message indicating there are no errors
2014-08-05 14:46 [U-Boot] [PATCH v4 0/13] Add some missing buildman features and deprecate MAKEALL Simon Glass
` (8 preceding siblings ...)
2014-08-05 14:46 ` [U-Boot] [PATCH v4 09/13] buildman: Add an option to specify the buildman config file Simon Glass
@ 2014-08-05 14:47 ` Simon Glass
2014-08-05 14:47 ` [U-Boot] [PATCH v4 11/13] buildman: Search for *cc instead of *gcc for the compiler Simon Glass
` (2 subsequent siblings)
12 siblings, 0 replies; 42+ messages in thread
From: Simon Glass @ 2014-08-05 14:47 UTC (permalink / raw)
To: u-boot
If buildman finds no problems it prints nothing. This can be a bit confusing,
so add a message that all is well.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v4:
- Add new patch to show a message when there are no errors
Changes in v3: None
Changes in v2: None
tools/buildman/builder.py | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/tools/buildman/builder.py b/tools/buildman/builder.py
index f148b93..e33f31e 100644
--- a/tools/buildman/builder.py
+++ b/tools/buildman/builder.py
@@ -891,10 +891,13 @@ class Builder:
self.col.MAGENTA)
for arch, target_list in arch_list.iteritems():
print '%10s: %s' % (arch, target_list)
+ self._error_lines += 1
if better_err:
print self.col.Color(self.col.GREEN, '\n'.join(better_err))
+ self._error_lines += 1
if worse_err:
print self.col.Color(self.col.RED, '\n'.join(worse_err))
+ self._error_lines += 1
if show_sizes:
self.PrintSizeSummary(board_selected, board_dict, show_detail,
@@ -937,9 +940,12 @@ class Builder:
self.commit_count = len(commits) if commits else 1
self.commits = commits
self.ResetResultSummary(board_selected)
+ self._error_lines = 0
for commit_upto in range(0, self.commit_count, self._step):
self.ProduceResultSummary(commit_upto, commits, board_selected)
+ if not self._error_lines:
+ print self.col.Color(self.col.GREEN, '(no errors to report)')
def SetupBuild(self, board_selected, commits):
--
2.0.0.526.g5318336
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 11/13] buildman: Search for *cc instead of *gcc for the compiler
2014-08-05 14:46 [U-Boot] [PATCH v4 0/13] Add some missing buildman features and deprecate MAKEALL Simon Glass
` (9 preceding siblings ...)
2014-08-05 14:47 ` [U-Boot] [PATCH v4 10/13] buildman: Add a message indicating there are no errors Simon Glass
@ 2014-08-05 14:47 ` Simon Glass
2014-08-05 17:41 ` Jeroen Hofstee
2014-08-05 14:47 ` [U-Boot] [PATCH v4 12/13] buildman: Add a few more toolchain examples to the README Simon Glass
2014-08-05 14:47 ` [U-Boot] [PATCH v4 13/13] RFC: Deprecate MAKEALL Simon Glass
12 siblings, 1 reply; 42+ messages in thread
From: Simon Glass @ 2014-08-05 14:47 UTC (permalink / raw)
To: u-boot
Compiler names normally end with cc, but not always gcc. So update the
detection logic.
Reported-by: Jeroen Hofstee <jeroen@myspectrum.nl>
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v4:
- Add new patch to search for *cc instead of *gcc
Changes in v3: None
Changes in v2: None
tools/buildman/toolchain.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/buildman/toolchain.py b/tools/buildman/toolchain.py
index 1b9771f..e4fcf1b 100644
--- a/tools/buildman/toolchain.py
+++ b/tools/buildman/toolchain.py
@@ -146,7 +146,7 @@ class Toolchains:
for subdir in ['.', 'bin', 'usr/bin']:
dirname = os.path.join(path, subdir)
if verbose: print " - looking in '%s'" % dirname
- for fname in glob.glob(dirname + '/*gcc'):
+ for fname in glob.glob(dirname + '/*cc'):
if verbose: print " - found '%s'" % fname
self.Add(fname, True, verbose)
--
2.0.0.526.g5318336
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 12/13] buildman: Add a few more toolchain examples to the README
2014-08-05 14:46 [U-Boot] [PATCH v4 0/13] Add some missing buildman features and deprecate MAKEALL Simon Glass
` (10 preceding siblings ...)
2014-08-05 14:47 ` [U-Boot] [PATCH v4 11/13] buildman: Search for *cc instead of *gcc for the compiler Simon Glass
@ 2014-08-05 14:47 ` Simon Glass
2014-08-05 14:47 ` [U-Boot] [PATCH v4 13/13] RFC: Deprecate MAKEALL Simon Glass
12 siblings, 0 replies; 42+ messages in thread
From: Simon Glass @ 2014-08-05 14:47 UTC (permalink / raw)
To: u-boot
The current README is a bit sparse in this area, so add a few more
examples.
Suggested-by: Tom Rini <trini@ti.com>
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v4:
- Add new path to add a few more toolchain examples to the README
Changes in v3: None
Changes in v2: None
tools/buildman/README | 2 ++
1 file changed, 2 insertions(+)
diff --git a/tools/buildman/README b/tools/buildman/README
index 6ba24c0..aaf0a10 100644
--- a/tools/buildman/README
+++ b/tools/buildman/README
@@ -134,6 +134,8 @@ example:
root: /
rest: /toolchains/*
eldk: /opt/eldk-4.2
+arm: /opt/linaro/gcc-linaro-arm-linux-gnueabihf-4.8-2013.08_linux
+aarch64: /opt/linaro/gcc-linaro-aarch64-none-elf-4.8-2013.10_linux
[toolchain-alias]
x86: i386
--
2.0.0.526.g5318336
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 13/13] RFC: Deprecate MAKEALL
2014-08-05 14:46 [U-Boot] [PATCH v4 0/13] Add some missing buildman features and deprecate MAKEALL Simon Glass
` (11 preceding siblings ...)
2014-08-05 14:47 ` [U-Boot] [PATCH v4 12/13] buildman: Add a few more toolchain examples to the README Simon Glass
@ 2014-08-05 14:47 ` Simon Glass
2014-08-05 16:43 ` York Sun
12 siblings, 1 reply; 42+ messages in thread
From: Simon Glass @ 2014-08-05 14:47 UTC (permalink / raw)
To: u-boot
Since buildman now includes most of the features of MAKEALL it is probably
time to talk about deprecating MAKEALL.
Comments welcome.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
Changes in v4: None
Changes in v3: None
Changes in v2: None
MAKEALL | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/MAKEALL b/MAKEALL
index 929fe88..dbbf74b 100755
--- a/MAKEALL
+++ b/MAKEALL
@@ -60,6 +60,10 @@ usage()
exit ${ret}
}
+echo "** Note: MAKEALL is deprecated - please use buildman instead"
+echo "** See tools/buildman/README for details"
+echo
+
SHORT_OPTS="ha:c:v:s:b:lmMCnr"
LONG_OPTS="help,arch:,cpu:,vendor:,soc:,board:,list,maintainers,mails,check,continue,rebuild-errors"
--
2.0.0.526.g5318336
^ permalink raw reply related [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 13/13] RFC: Deprecate MAKEALL
2014-08-05 14:47 ` [U-Boot] [PATCH v4 13/13] RFC: Deprecate MAKEALL Simon Glass
@ 2014-08-05 16:43 ` York Sun
2014-08-05 18:41 ` Simon Glass
0 siblings, 1 reply; 42+ messages in thread
From: York Sun @ 2014-08-05 16:43 UTC (permalink / raw)
To: u-boot
On 08/05/2014 07:47 AM, Simon Glass wrote:
> Since buildman now includes most of the features of MAKEALL it is probably
> time to talk about deprecating MAKEALL.
>
> Comments welcome.
>
Simon,
I know buildman has been out for a while. I just rely too much on MAKEALL
automation and am reluctant to try buildman.
Reading buildman/README, it seems buildman always builds a branch and its
upstream commit. I am hoping you can help me to understand how to use buildman
in my environment with gerrit and Jenkins. For every patch (internal
development), we use gerrit to conduct review and Jenkins to test. If you are
not familiar with either, the simplest way to understand is a script will run on
every commit. In the script I use MAKEALL to build all concerned (hundreds)
targets for _this_ commit. The dependency is maintained by gerrit. The result is
fed back to gerrit to show the author (and reviewers) if a failure happens.
If using buildman and the upstream commit is always built, a great amount of
time will be consumed with no benefit. If you see a better way to use buildman,
I can give it a try.
Regards,
York
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 11/13] buildman: Search for *cc instead of *gcc for the compiler
2014-08-05 14:47 ` [U-Boot] [PATCH v4 11/13] buildman: Search for *cc instead of *gcc for the compiler Simon Glass
@ 2014-08-05 17:41 ` Jeroen Hofstee
2014-08-05 18:25 ` Simon Glass
0 siblings, 1 reply; 42+ messages in thread
From: Jeroen Hofstee @ 2014-08-05 17:41 UTC (permalink / raw)
To: u-boot
Hello Simon,
On 05-08-14 16:47, Simon Glass wrote:
> Compiler names normally end with cc, but not always gcc. So update the
> detection logic.
>
> Reported-by: Jeroen Hofstee <jeroen@myspectrum.nl>
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
>
> Changes in v4:
> - Add new patch to search for *cc instead of *gcc
>
> Changes in v3: None
> Changes in v2: None
>
> tools/buildman/toolchain.py | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tools/buildman/toolchain.py b/tools/buildman/toolchain.py
> index 1b9771f..e4fcf1b 100644
> --- a/tools/buildman/toolchain.py
> +++ b/tools/buildman/toolchain.py
> @@ -146,7 +146,7 @@ class Toolchains:
> for subdir in ['.', 'bin', 'usr/bin']:
> dirname = os.path.join(path, subdir)
> if verbose: print " - looking in '%s'" % dirname
> - for fname in glob.glob(dirname + '/*gcc'):
> + for fname in glob.glob(dirname + '/*cc'):
> if verbose: print " - found '%s'" % fname
> self.Add(fname, True, verbose)
>
I don't have the time to test this now, but this might have
funny side affects. Since cc itself is commonly a symlink to
gcc / clang etc, it will find the same compiler twice. Not sure
if this is handled correctly.
In my hacky attempt to support this I also stored the actual
compiler name, since it is concatenated with gcc somewhere
else if I recall correctly.
Anyway, unless you are sure this works, I don't mind dropping it,
for the time being. There are some more issues for e.g. FreeBSD,
gcc will get installed in /usr/local/bin and carries a version number
like gcc47 etc. clang on linux is just clang and there is no cc
version etc.
Regards,
Jeroen
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 11/13] buildman: Search for *cc instead of *gcc for the compiler
2014-08-05 17:41 ` Jeroen Hofstee
@ 2014-08-05 18:25 ` Simon Glass
0 siblings, 0 replies; 42+ messages in thread
From: Simon Glass @ 2014-08-05 18:25 UTC (permalink / raw)
To: u-boot
Hi Jeroen,
On 5 August 2014 11:41, Jeroen Hofstee <jeroen@myspectrum.nl> wrote:
> Hello Simon,
>
>
> On 05-08-14 16:47, Simon Glass wrote:
>>
>> Compiler names normally end with cc, but not always gcc. So update the
>> detection logic.
>>
>> Reported-by: Jeroen Hofstee <jeroen@myspectrum.nl>
>> Signed-off-by: Simon Glass <sjg@chromium.org>
>> ---
>>
>> Changes in v4:
>> - Add new patch to search for *cc instead of *gcc
>>
>> Changes in v3: None
>> Changes in v2: None
>>
>> tools/buildman/toolchain.py | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/tools/buildman/toolchain.py b/tools/buildman/toolchain.py
>> index 1b9771f..e4fcf1b 100644
>> --- a/tools/buildman/toolchain.py
>> +++ b/tools/buildman/toolchain.py
>> @@ -146,7 +146,7 @@ class Toolchains:
>> for subdir in ['.', 'bin', 'usr/bin']:
>> dirname = os.path.join(path, subdir)
>> if verbose: print " - looking in '%s'" % dirname
>> - for fname in glob.glob(dirname + '/*gcc'):
>> + for fname in glob.glob(dirname + '/*cc'):
>> if verbose: print " - found '%s'" % fname
>> self.Add(fname, True, verbose)
>>
>
>
> I don't have the time to test this now, but this might have
> funny side affects. Since cc itself is commonly a symlink to
> gcc / clang etc, it will find the same compiler twice. Not sure
> if this is handled correctly.
>
> In my hacky attempt to support this I also stored the actual
> compiler name, since it is concatenated with gcc somewhere
> else if I recall correctly.
>
> Anyway, unless you are sure this works, I don't mind dropping it,
> for the time being. There are some more issues for e.g. FreeBSD,
> gcc will get installed in /usr/local/bin and carries a version number
> like gcc47 etc. clang on linux is just clang and there is no cc
> version etc.
OK let's drop it for now until we are clear what is required.
Regards,
Simon
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 13/13] RFC: Deprecate MAKEALL
2014-08-05 16:43 ` York Sun
@ 2014-08-05 18:41 ` Simon Glass
2014-08-05 18:48 ` York Sun
0 siblings, 1 reply; 42+ messages in thread
From: Simon Glass @ 2014-08-05 18:41 UTC (permalink / raw)
To: u-boot
Hi York,
On 5 August 2014 10:43, York Sun <yorksun@freescale.com> wrote:
> On 08/05/2014 07:47 AM, Simon Glass wrote:
>> Since buildman now includes most of the features of MAKEALL it is probably
>> time to talk about deprecating MAKEALL.
>>
>> Comments welcome.
>>
>
> Simon,
>
> I know buildman has been out for a while. I just rely too much on MAKEALL
> automation and am reluctant to try buildman.
>
> Reading buildman/README, it seems buildman always builds a branch and its
> upstream commit. I am hoping you can help me to understand how to use buildman
> in my environment with gerrit and Jenkins. For every patch (internal
> development), we use gerrit to conduct review and Jenkins to test. If you are
> not familiar with either, the simplest way to understand is a script will run on
> every commit. In the script I use MAKEALL to build all concerned (hundreds)
> targets for _this_ commit. The dependency is maintained by gerrit. The result is
> fed back to gerrit to show the author (and reviewers) if a failure happens.
>
> If using buildman and the upstream commit is always built, a great amount of
> time will be consumed with no benefit. If you see a better way to use buildman,
> I can give it a try.
With the v3 or v4 series you can omit the -b option and it will build
the currently checked-out commit for the selected boards. It might be
useful in that the problems are stored in files as well as displayed
on the command line.
So perhaps you could try that?
BTW where does this information get published?
Regards,
Simon
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 13/13] RFC: Deprecate MAKEALL
2014-08-05 18:41 ` Simon Glass
@ 2014-08-05 18:48 ` York Sun
2014-08-05 18:52 ` Simon Glass
0 siblings, 1 reply; 42+ messages in thread
From: York Sun @ 2014-08-05 18:48 UTC (permalink / raw)
To: u-boot
On 08/05/2014 11:41 AM, Simon Glass wrote:
> Hi York,
>
> On 5 August 2014 10:43, York Sun <yorksun@freescale.com> wrote:
>> On 08/05/2014 07:47 AM, Simon Glass wrote:
>>> Since buildman now includes most of the features of MAKEALL it is probably
>>> time to talk about deprecating MAKEALL.
>>>
>>> Comments welcome.
>>>
>>
>> Simon,
>>
>> I know buildman has been out for a while. I just rely too much on MAKEALL
>> automation and am reluctant to try buildman.
>>
>> Reading buildman/README, it seems buildman always builds a branch and its
>> upstream commit. I am hoping you can help me to understand how to use buildman
>> in my environment with gerrit and Jenkins. For every patch (internal
>> development), we use gerrit to conduct review and Jenkins to test. If you are
>> not familiar with either, the simplest way to understand is a script will run on
>> every commit. In the script I use MAKEALL to build all concerned (hundreds)
>> targets for _this_ commit. The dependency is maintained by gerrit. The result is
>> fed back to gerrit to show the author (and reviewers) if a failure happens.
>>
>> If using buildman and the upstream commit is always built, a great amount of
>> time will be consumed with no benefit. If you see a better way to use buildman,
>> I can give it a try.
>
> With the v3 or v4 series you can omit the -b option and it will build
> the currently checked-out commit for the selected boards. It might be
> useful in that the problems are stored in files as well as displayed
> on the command line.
I use the .ERR file from MAKEALL. It would be nice to have similar log, in
individual files.
>
> So perhaps you could try that?
I would like to try that.
>
> BTW where does this information get published?
>
When we use gerrit and Jenkins, the build log is stored in Jenkins log, and
emails were sent with the failure log.
York
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 13/13] RFC: Deprecate MAKEALL
2014-08-05 18:48 ` York Sun
@ 2014-08-05 18:52 ` Simon Glass
2014-08-05 18:55 ` York Sun
0 siblings, 1 reply; 42+ messages in thread
From: Simon Glass @ 2014-08-05 18:52 UTC (permalink / raw)
To: u-boot
Hi York,
On 5 August 2014 12:48, York Sun <yorksun@freescale.com> wrote:
> On 08/05/2014 11:41 AM, Simon Glass wrote:
>> Hi York,
>>
>> On 5 August 2014 10:43, York Sun <yorksun@freescale.com> wrote:
>>> On 08/05/2014 07:47 AM, Simon Glass wrote:
>>>> Since buildman now includes most of the features of MAKEALL it is probably
>>>> time to talk about deprecating MAKEALL.
>>>>
>>>> Comments welcome.
>>>>
>>>
>>> Simon,
>>>
>>> I know buildman has been out for a while. I just rely too much on MAKEALL
>>> automation and am reluctant to try buildman.
>>>
>>> Reading buildman/README, it seems buildman always builds a branch and its
>>> upstream commit. I am hoping you can help me to understand how to use buildman
>>> in my environment with gerrit and Jenkins. For every patch (internal
>>> development), we use gerrit to conduct review and Jenkins to test. If you are
>>> not familiar with either, the simplest way to understand is a script will run on
>>> every commit. In the script I use MAKEALL to build all concerned (hundreds)
>>> targets for _this_ commit. The dependency is maintained by gerrit. The result is
>>> fed back to gerrit to show the author (and reviewers) if a failure happens.
>>>
>>> If using buildman and the upstream commit is always built, a great amount of
>>> time will be consumed with no benefit. If you see a better way to use buildman,
>>> I can give it a try.
>>
>> With the v3 or v4 series you can omit the -b option and it will build
>> the currently checked-out commit for the selected boards. It might be
>> useful in that the problems are stored in files as well as displayed
>> on the command line.
>
> I use the .ERR file from MAKEALL. It would be nice to have similar log, in
> individual files.
It should be similar, and you get one file for each commit / board combination.
>
>>
>> So perhaps you could try that?
>
> I would like to try that.
>
>>
>> BTW where does this information get published?
>>
>
> When we use gerrit and Jenkins, the build log is stored in Jenkins log, and
> emails were sent with the failure log.
OK, I don't think I've send such an email. Do I have to do anything to
get them to come to me?
Regards,
Simon
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 03/13] buildman: Allow building of current source tree
2014-08-05 14:46 ` [U-Boot] [PATCH v4 03/13] buildman: Allow building of current source tree Simon Glass
@ 2014-08-05 18:54 ` Tom Rini
2014-08-05 18:58 ` Simon Glass
0 siblings, 1 reply; 42+ messages in thread
From: Tom Rini @ 2014-08-05 18:54 UTC (permalink / raw)
To: u-boot
On Tue, Aug 05, 2014 at 08:46:53AM -0600, Simon Glass wrote:
> Originally buildman had some support for building the current source tree.
> However this was dropped before it was submitted, as part of the effort to
> make it faster when building entire branches.
>
> Reinstate this support. If no -b option is given, buildman will build the
> current source tree.
>
> Reported-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
> Signed-off-by: Simon Glass <sjg@chromium.org>
I think this is the main use case of MAKEALL so lets also update the
relevant part of the README to cover it. Thanks!
--
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20140805/93ee5ba6/attachment.pgp>
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 13/13] RFC: Deprecate MAKEALL
2014-08-05 18:52 ` Simon Glass
@ 2014-08-05 18:55 ` York Sun
2014-08-05 18:59 ` Simon Glass
0 siblings, 1 reply; 42+ messages in thread
From: York Sun @ 2014-08-05 18:55 UTC (permalink / raw)
To: u-boot
On 08/05/2014 11:52 AM, Simon Glass wrote:
>>>
>>> BTW where does this information get published?
>>>
>>
>> When we use gerrit and Jenkins, the build log is stored in Jenkins log, and
>> emails were sent with the failure log.
>
> OK, I don't think I've send such an email. Do I have to do anything to
> get them to come to me?
>
The emails were sent from Jenkins, by my script collecting information from .ERR
files.
Maybe I didn't understand your question. What do you mean "where does this
information get published"?
York
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 03/13] buildman: Allow building of current source tree
2014-08-05 18:54 ` Tom Rini
@ 2014-08-05 18:58 ` Simon Glass
2014-08-05 19:01 ` Tom Rini
0 siblings, 1 reply; 42+ messages in thread
From: Simon Glass @ 2014-08-05 18:58 UTC (permalink / raw)
To: u-boot
Hi Tom,
On 5 August 2014 12:54, Tom Rini <trini@ti.com> wrote:
> On Tue, Aug 05, 2014 at 08:46:53AM -0600, Simon Glass wrote:
>
>> Originally buildman had some support for building the current source tree.
>> However this was dropped before it was submitted, as part of the effort to
>> make it faster when building entire branches.
>>
>> Reinstate this support. If no -b option is given, buildman will build the
>> current source tree.
>>
>> Reported-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
>> Signed-off-by: Simon Glass <sjg@chromium.org>
>
> I think this is the main use case of MAKEALL so lets also update the
> relevant part of the README to cover it. Thanks!
I did that in the patch that adds -v, since it makes things consistent
between the two ways of running buildman. See the patch 'buildman: Add
verbose option to display errors as they happen'.
Regards,
Simon
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 13/13] RFC: Deprecate MAKEALL
2014-08-05 18:55 ` York Sun
@ 2014-08-05 18:59 ` Simon Glass
2014-08-05 19:01 ` York Sun
0 siblings, 1 reply; 42+ messages in thread
From: Simon Glass @ 2014-08-05 18:59 UTC (permalink / raw)
To: u-boot
Hi York,
On 5 August 2014 12:55, York Sun <yorksun@freescale.com> wrote:
> On 08/05/2014 11:52 AM, Simon Glass wrote:
>>>>
>>>> BTW where does this information get published?
>>>>
>>>
>>> When we use gerrit and Jenkins, the build log is stored in Jenkins log, and
>>> emails were sent with the failure log.
>>
>> OK, I don't think I've send such an email. Do I have to do anything to
>> get them to come to me?
>>
> The emails were sent from Jenkins, by my script collecting information from .ERR
> files.
>
> Maybe I didn't understand your question. What do you mean "where does this
> information get published"?
I just mean that I don't think I have received an email yet, and don't
know about the service, so am wondering how to get an email.
Regards,
Simon
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 03/13] buildman: Allow building of current source tree
2014-08-05 18:58 ` Simon Glass
@ 2014-08-05 19:01 ` Tom Rini
0 siblings, 0 replies; 42+ messages in thread
From: Tom Rini @ 2014-08-05 19:01 UTC (permalink / raw)
To: u-boot
On Tue, Aug 05, 2014 at 12:58:55PM -0600, Simon Glass wrote:
> Hi Tom,
>
> On 5 August 2014 12:54, Tom Rini <trini@ti.com> wrote:
> > On Tue, Aug 05, 2014 at 08:46:53AM -0600, Simon Glass wrote:
> >
> >> Originally buildman had some support for building the current source tree.
> >> However this was dropped before it was submitted, as part of the effort to
> >> make it faster when building entire branches.
> >>
> >> Reinstate this support. If no -b option is given, buildman will build the
> >> current source tree.
> >>
> >> Reported-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
> >> Signed-off-by: Simon Glass <sjg@chromium.org>
> >
> > I think this is the main use case of MAKEALL so lets also update the
> > relevant part of the README to cover it. Thanks!
>
> I did that in the patch that adds -v, since it makes things consistent
> between the two ways of running buildman. See the patch 'buildman: Add
> verbose option to display errors as they happen'.
Ah, OK, thanks.
--
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20140805/b8d3c07d/attachment.pgp>
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 13/13] RFC: Deprecate MAKEALL
2014-08-05 18:59 ` Simon Glass
@ 2014-08-05 19:01 ` York Sun
2014-08-05 19:10 ` Tom Rini
0 siblings, 1 reply; 42+ messages in thread
From: York Sun @ 2014-08-05 19:01 UTC (permalink / raw)
To: u-boot
On 08/05/2014 11:59 AM, Simon Glass wrote:
> Hi York,
>
> On 5 August 2014 12:55, York Sun <yorksun@freescale.com> wrote:
>> On 08/05/2014 11:52 AM, Simon Glass wrote:
>>>>>
>>>>> BTW where does this information get published?
>>>>>
>>>>
>>>> When we use gerrit and Jenkins, the build log is stored in Jenkins log, and
>>>> emails were sent with the failure log.
>>>
>>> OK, I don't think I've send such an email. Do I have to do anything to
>>> get them to come to me?
>>>
>> The emails were sent from Jenkins, by my script collecting information from .ERR
>> files.
>>
>> Maybe I didn't understand your question. What do you mean "where does this
>> information get published"?
>
> I just mean that I don't think I have received an email yet, and don't
> know about the service, so am wondering how to get an email.
>
Simon,
Sorry I misled you. The emails are sent by my scripts under Jenkins (A tool, see
http://jenkins-ci.org/). It is not within u-boot.
York
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 13/13] RFC: Deprecate MAKEALL
2014-08-05 19:01 ` York Sun
@ 2014-08-05 19:10 ` Tom Rini
2014-08-05 22:06 ` York Sun
0 siblings, 1 reply; 42+ messages in thread
From: Tom Rini @ 2014-08-05 19:10 UTC (permalink / raw)
To: u-boot
On Tue, Aug 05, 2014 at 12:01:33PM -0700, York Sun wrote:
> On 08/05/2014 11:59 AM, Simon Glass wrote:
> > Hi York,
> >
> > On 5 August 2014 12:55, York Sun <yorksun@freescale.com> wrote:
> >> On 08/05/2014 11:52 AM, Simon Glass wrote:
> >>>>>
> >>>>> BTW where does this information get published?
> >>>>>
> >>>>
> >>>> When we use gerrit and Jenkins, the build log is stored in Jenkins log, and
> >>>> emails were sent with the failure log.
> >>>
> >>> OK, I don't think I've send such an email. Do I have to do anything to
> >>> get them to come to me?
> >>>
> >> The emails were sent from Jenkins, by my script collecting information from .ERR
> >> files.
> >>
> >> Maybe I didn't understand your question. What do you mean "where does this
> >> information get published"?
> >
> > I just mean that I don't think I have received an email yet, and don't
> > know about the service, so am wondering how to get an email.
> >
>
> Simon,
>
> Sorry I misled you. The emails are sent by my scripts under Jenkins (A
> tool, see http://jenkins-ci.org/). It is not within u-boot.
Would you be able to post your jobs / scripts somewhere? I've toyed
with a personal setup and I know Marek has run one before as well.
Thanks!
--
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20140805/8b9d4b6c/attachment.pgp>
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 13/13] RFC: Deprecate MAKEALL
2014-08-05 19:10 ` Tom Rini
@ 2014-08-05 22:06 ` York Sun
2014-08-08 21:12 ` Tom Rini
0 siblings, 1 reply; 42+ messages in thread
From: York Sun @ 2014-08-05 22:06 UTC (permalink / raw)
To: u-boot
On 08/05/2014 12:10 PM, Tom Rini wrote:
> On Tue, Aug 05, 2014 at 12:01:33PM -0700, York Sun wrote:
>> On 08/05/2014 11:59 AM, Simon Glass wrote:
>>> Hi York,
>>>
>>> On 5 August 2014 12:55, York Sun <yorksun@freescale.com> wrote:
>>>> On 08/05/2014 11:52 AM, Simon Glass wrote:
>>>>>>>
>>>>>>> BTW where does this information get published?
>>>>>>>
>>>>>>
>>>>>> When we use gerrit and Jenkins, the build log is stored in Jenkins log, and
>>>>>> emails were sent with the failure log.
>>>>>
>>>>> OK, I don't think I've send such an email. Do I have to do anything to
>>>>> get them to come to me?
>>>>>
>>>> The emails were sent from Jenkins, by my script collecting information from .ERR
>>>> files.
>>>>
>>>> Maybe I didn't understand your question. What do you mean "where does this
>>>> information get published"?
>>>
>>> I just mean that I don't think I have received an email yet, and don't
>>> know about the service, so am wondering how to get an email.
>>>
>>
>> Simon,
>>
>> Sorry I misled you. The emails are sent by my scripts under Jenkins (A
>> tool, see http://jenkins-ci.org/). It is not within u-boot.
>
> Would you be able to post your jobs / scripts somewhere? I've toyed
> with a personal setup and I know Marek has run one before as well.
> Thanks!
>
Tom,
My script is long and messy. This is the part sending emails.
if ls LOG/*.ERR &>/dev/null; then
error_msg="'Build failed. Log was sent to $GERRIT_PATCHSET_UPLOADER_EMAIL.'"
ssh gerrit.hostname.masked gerrit review --message $error_msg
$GERRIT_PATCHSET_REVISION
/bin/mail -s "Build failed $GERRIT_CHANGE_URL" $GERRIT_PATCHSET_UPLOADER_EMAIL
-c $admin_email < build.result
exit 1
else
echo "Passed"
exit 0
fi
York
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 02/13] buildman: Add some notes about moving from MAKEALL
2014-08-05 14:46 ` [U-Boot] [PATCH v4 02/13] buildman: Add some notes about moving from MAKEALL Simon Glass
@ 2014-08-05 22:34 ` York Sun
2014-08-05 23:07 ` Simon Glass
0 siblings, 1 reply; 42+ messages in thread
From: York Sun @ 2014-08-05 22:34 UTC (permalink / raw)
To: u-boot
On 08/05/2014 07:46 AM, Simon Glass wrote:
> + - build all Freescale boards with MPC83xx CPUs, plus all 4xx boards:
> + MAKEALL -c mpc83xx -v freescale 4xx
> + ** buildman -b <branch> mpc83xx freescale 4xx
> +
This is not very clear to me. Is the condition "AND", or "OR"? When I do
"MAKEALL -c mpc83xx -v freescale", I want to build all Freescale boards with
MPC83xx. It is "AND". But with the buildman command, it is "OR". Examples below
$ ./tools/buildman/buildman -n -b master mpc83xx
No section: 'make-flags'
Could not find ./boards.cfg
Generating boards.cfg ... (jobs: 4)
1177/1177 [=======================================================>]
Dry run, so not doing much. But I would do this:
Building 14 commits for 50 boards (4 threads, 1 job per thread)
Build directory: ../denx_master
25b4adbb include: remove CONFIG_SPL/CONFIG_TPL definition in config headers
35c84002 buildman: Fix a few typos
b308e4a8 buildman: Add some notes about moving from MAKEALL
cd739429 buildman: Allow building of current source tree
5ace7165 buildman: Move BuilderThread code to its own file
d095b480 buildman: Sort command line options
5bf27712 buildman: Refactor output options
461f4050 buildman: Add verbose option to display errors as they happen
65f46549 buildman: Remove unused non-incremental build method code
0723a71a buildman: Add an option to specify the buildman config file
b2b89c32 buildman: Add a message indicating there are no errors
eedb4545 buildman: Search for *cc instead of *gcc for the compiler
fe99029f buildman: Add a few more toolchain examples to the README
c976629f RFC: Deprecate MAKEALL
mpc83xx : 50 boards
Total boards to build for each commit: 50
$ ./tools/buildman/buildman -n -b master mpc83xx freescale
No section: 'make-flags'
Dry run, so not doing much. But I would do this:
Building 14 commits for 364 boards (4 threads, 1 job per thread)
Build directory: ../denx_master
25b4adbb include: remove CONFIG_SPL/CONFIG_TPL definition in config headers
35c84002 buildman: Fix a few typos
b308e4a8 buildman: Add some notes about moving from MAKEALL
cd739429 buildman: Allow building of current source tree
5ace7165 buildman: Move BuilderThread code to its own file
d095b480 buildman: Sort command line options
5bf27712 buildman: Refactor output options
461f4050 buildman: Add verbose option to display errors as they happen
65f46549 buildman: Remove unused non-incremental build method code
0723a71a buildman: Add an option to specify the buildman config file
b2b89c32 buildman: Add a message indicating there are no errors
eedb4545 buildman: Search for *cc instead of *gcc for the compiler
fe99029f buildman: Add a few more toolchain examples to the README
c976629f RFC: Deprecate MAKEALL
mpc83xx : 50 boards
freescale : 314 boards
Total boards to build for each commit: 364
This is not the desired target list.
Beside, buildman still needs boards.cfg. It takes long to generate this file.
Not too long, but if my other tools clean the working directory for each commit,
this accumulates to long time.
York
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 02/13] buildman: Add some notes about moving from MAKEALL
2014-08-05 22:34 ` York Sun
@ 2014-08-05 23:07 ` Simon Glass
2014-08-05 23:21 ` York Sun
0 siblings, 1 reply; 42+ messages in thread
From: Simon Glass @ 2014-08-05 23:07 UTC (permalink / raw)
To: u-boot
Hi York,
On 5 August 2014 16:34, York Sun <yorksun@freescale.com> wrote:
> On 08/05/2014 07:46 AM, Simon Glass wrote:
>> + - build all Freescale boards with MPC83xx CPUs, plus all 4xx boards:
>> + MAKEALL -c mpc83xx -v freescale 4xx
>> + ** buildman -b <branch> mpc83xx freescale 4xx
>> +
>
> This is not very clear to me. Is the condition "AND", or "OR"? When I do
> "MAKEALL -c mpc83xx -v freescale", I want to build all Freescale boards with
> MPC83xx. It is "AND". But with the buildman command, it is "OR". Examples below
>
> $ ./tools/buildman/buildman -n -b master mpc83xx
> No section: 'make-flags'
> Could not find ./boards.cfg
> Generating boards.cfg ... (jobs: 4)
> 1177/1177 [=======================================================>]
> Dry run, so not doing much. But I would do this:
>
> Building 14 commits for 50 boards (4 threads, 1 job per thread)
> Build directory: ../denx_master
> 25b4adbb include: remove CONFIG_SPL/CONFIG_TPL definition in config headers
> 35c84002 buildman: Fix a few typos
> b308e4a8 buildman: Add some notes about moving from MAKEALL
> cd739429 buildman: Allow building of current source tree
> 5ace7165 buildman: Move BuilderThread code to its own file
> d095b480 buildman: Sort command line options
> 5bf27712 buildman: Refactor output options
> 461f4050 buildman: Add verbose option to display errors as they happen
> 65f46549 buildman: Remove unused non-incremental build method code
> 0723a71a buildman: Add an option to specify the buildman config file
> b2b89c32 buildman: Add a message indicating there are no errors
> eedb4545 buildman: Search for *cc instead of *gcc for the compiler
> fe99029f buildman: Add a few more toolchain examples to the README
> c976629f RFC: Deprecate MAKEALL
>
> mpc83xx : 50 boards
> Total boards to build for each commit: 50
>
> $ ./tools/buildman/buildman -n -b master mpc83xx freescale
> No section: 'make-flags'
> Dry run, so not doing much. But I would do this:
>
> Building 14 commits for 364 boards (4 threads, 1 job per thread)
> Build directory: ../denx_master
> 25b4adbb include: remove CONFIG_SPL/CONFIG_TPL definition in config headers
> 35c84002 buildman: Fix a few typos
> b308e4a8 buildman: Add some notes about moving from MAKEALL
> cd739429 buildman: Allow building of current source tree
> 5ace7165 buildman: Move BuilderThread code to its own file
> d095b480 buildman: Sort command line options
> 5bf27712 buildman: Refactor output options
> 461f4050 buildman: Add verbose option to display errors as they happen
> 65f46549 buildman: Remove unused non-incremental build method code
> 0723a71a buildman: Add an option to specify the buildman config file
> b2b89c32 buildman: Add a message indicating there are no errors
> eedb4545 buildman: Search for *cc instead of *gcc for the compiler
> fe99029f buildman: Add a few more toolchain examples to the README
> c976629f RFC: Deprecate MAKEALL
>
> mpc83xx : 50 boards
> freescale : 314 boards
> Total boards to build for each commit: 364
>
> This is not the desired target list.
OK I see.
But in this case why not just leave off the 'freescale'?
>
> Beside, buildman still needs boards.cfg. It takes long to generate this file.
So does MAKEALL, right?
> Not too long, but if my other tools clean the working directory for each commit,
> this accumulates to long time.
Can you expand on at a little please? I'm not sure what this refers to.
Regards,
Simon
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 02/13] buildman: Add some notes about moving from MAKEALL
2014-08-05 23:07 ` Simon Glass
@ 2014-08-05 23:21 ` York Sun
2014-08-06 2:15 ` Simon Glass
0 siblings, 1 reply; 42+ messages in thread
From: York Sun @ 2014-08-05 23:21 UTC (permalink / raw)
To: u-boot
On 08/05/2014 04:07 PM, Simon Glass wrote:
> Hi York,
>
> On 5 August 2014 16:34, York Sun <yorksun@freescale.com> wrote:
>> On 08/05/2014 07:46 AM, Simon Glass wrote:
>>> + - build all Freescale boards with MPC83xx CPUs, plus all 4xx boards:
>>> + MAKEALL -c mpc83xx -v freescale 4xx
>>> + ** buildman -b <branch> mpc83xx freescale 4xx
>>> +
>>
>> This is not very clear to me. Is the condition "AND", or "OR"? When I do
>> "MAKEALL -c mpc83xx -v freescale", I want to build all Freescale boards with
>> MPC83xx. It is "AND". But with the buildman command, it is "OR". Examples below
>>
>> $ ./tools/buildman/buildman -n -b master mpc83xx
>> No section: 'make-flags'
>> Could not find ./boards.cfg
>> Generating boards.cfg ... (jobs: 4)
>> 1177/1177 [=======================================================>]
>> Dry run, so not doing much. But I would do this:
>>
>> Building 14 commits for 50 boards (4 threads, 1 job per thread)
>> Build directory: ../denx_master
>> 25b4adbb include: remove CONFIG_SPL/CONFIG_TPL definition in config headers
>> 35c84002 buildman: Fix a few typos
>> b308e4a8 buildman: Add some notes about moving from MAKEALL
>> cd739429 buildman: Allow building of current source tree
>> 5ace7165 buildman: Move BuilderThread code to its own file
>> d095b480 buildman: Sort command line options
>> 5bf27712 buildman: Refactor output options
>> 461f4050 buildman: Add verbose option to display errors as they happen
>> 65f46549 buildman: Remove unused non-incremental build method code
>> 0723a71a buildman: Add an option to specify the buildman config file
>> b2b89c32 buildman: Add a message indicating there are no errors
>> eedb4545 buildman: Search for *cc instead of *gcc for the compiler
>> fe99029f buildman: Add a few more toolchain examples to the README
>> c976629f RFC: Deprecate MAKEALL
>>
>> mpc83xx : 50 boards
>> Total boards to build for each commit: 50
>>
>> $ ./tools/buildman/buildman -n -b master mpc83xx freescale
>> No section: 'make-flags'
>> Dry run, so not doing much. But I would do this:
>>
>> Building 14 commits for 364 boards (4 threads, 1 job per thread)
>> Build directory: ../denx_master
>> 25b4adbb include: remove CONFIG_SPL/CONFIG_TPL definition in config headers
>> 35c84002 buildman: Fix a few typos
>> b308e4a8 buildman: Add some notes about moving from MAKEALL
>> cd739429 buildman: Allow building of current source tree
>> 5ace7165 buildman: Move BuilderThread code to its own file
>> d095b480 buildman: Sort command line options
>> 5bf27712 buildman: Refactor output options
>> 461f4050 buildman: Add verbose option to display errors as they happen
>> 65f46549 buildman: Remove unused non-incremental build method code
>> 0723a71a buildman: Add an option to specify the buildman config file
>> b2b89c32 buildman: Add a message indicating there are no errors
>> eedb4545 buildman: Search for *cc instead of *gcc for the compiler
>> fe99029f buildman: Add a few more toolchain examples to the README
>> c976629f RFC: Deprecate MAKEALL
>>
>> mpc83xx : 50 boards
>> freescale : 314 boards
>> Total boards to build for each commit: 364
>>
>> This is not the desired target list.
>
> OK I see.
>
> But in this case why not just leave off the 'freescale'?
This is just an example. What if I chose "-a arm" and "-v freescale". ARM has
300+ targets, but only 20+ are for Freescale. I could save time by building a
lot less platforms.
The point here is the "OR" logic.
>
>>
>> Beside, buildman still needs boards.cfg. It takes long to generate this file.
>
> So does MAKEALL, right?
Yes. MAKEALL also generates the boards.cfg.
>
>> Not too long, but if my other tools clean the working directory for each commit,
>> this accumulates to long time.
>
> Can you expand on at a little please? I'm not sure what this refers to.
>
Again it is our internal tools of choice. We use Gerrit and Jenkins. Each commit
triggers a build on Jenkins. Right now I use MAKEALL to build the concerned
targets. Before each build, the working directory is checkout out to that
particular commit and cleaned. So if each build needs to generate the
boards.cfg, a lot of time will be consumed if I have many commits in the queue.
But generating boards.cfg is not caused by your patch. I just point out what I saw.
York
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 02/13] buildman: Add some notes about moving from MAKEALL
2014-08-05 23:21 ` York Sun
@ 2014-08-06 2:15 ` Simon Glass
2014-08-06 2:53 ` York Sun
0 siblings, 1 reply; 42+ messages in thread
From: Simon Glass @ 2014-08-06 2:15 UTC (permalink / raw)
To: u-boot
Hi York,
On 5 August 2014 17:21, York Sun <yorksun@freescale.com> wrote:
> On 08/05/2014 04:07 PM, Simon Glass wrote:
>> Hi York,
>>
>> On 5 August 2014 16:34, York Sun <yorksun@freescale.com> wrote:
>>> On 08/05/2014 07:46 AM, Simon Glass wrote:
>>>> + - build all Freescale boards with MPC83xx CPUs, plus all 4xx boards:
>>>> + MAKEALL -c mpc83xx -v freescale 4xx
>>>> + ** buildman -b <branch> mpc83xx freescale 4xx
>>>> +
>>>
>>> This is not very clear to me. Is the condition "AND", or "OR"? When I do
>>> "MAKEALL -c mpc83xx -v freescale", I want to build all Freescale boards with
>>> MPC83xx. It is "AND". But with the buildman command, it is "OR". Examples below
>>>
>>> $ ./tools/buildman/buildman -n -b master mpc83xx
>>> No section: 'make-flags'
>>> Could not find ./boards.cfg
>>> Generating boards.cfg ... (jobs: 4)
>>> 1177/1177 [=======================================================>]
>>> Dry run, so not doing much. But I would do this:
>>>
>>> Building 14 commits for 50 boards (4 threads, 1 job per thread)
>>> Build directory: ../denx_master
>>> 25b4adbb include: remove CONFIG_SPL/CONFIG_TPL definition in config headers
>>> 35c84002 buildman: Fix a few typos
>>> b308e4a8 buildman: Add some notes about moving from MAKEALL
>>> cd739429 buildman: Allow building of current source tree
>>> 5ace7165 buildman: Move BuilderThread code to its own file
>>> d095b480 buildman: Sort command line options
>>> 5bf27712 buildman: Refactor output options
>>> 461f4050 buildman: Add verbose option to display errors as they happen
>>> 65f46549 buildman: Remove unused non-incremental build method code
>>> 0723a71a buildman: Add an option to specify the buildman config file
>>> b2b89c32 buildman: Add a message indicating there are no errors
>>> eedb4545 buildman: Search for *cc instead of *gcc for the compiler
>>> fe99029f buildman: Add a few more toolchain examples to the README
>>> c976629f RFC: Deprecate MAKEALL
>>>
>>> mpc83xx : 50 boards
>>> Total boards to build for each commit: 50
>>>
>>> $ ./tools/buildman/buildman -n -b master mpc83xx freescale
>>> No section: 'make-flags'
>>> Dry run, so not doing much. But I would do this:
>>>
>>> Building 14 commits for 364 boards (4 threads, 1 job per thread)
>>> Build directory: ../denx_master
>>> 25b4adbb include: remove CONFIG_SPL/CONFIG_TPL definition in config headers
>>> 35c84002 buildman: Fix a few typos
>>> b308e4a8 buildman: Add some notes about moving from MAKEALL
>>> cd739429 buildman: Allow building of current source tree
>>> 5ace7165 buildman: Move BuilderThread code to its own file
>>> d095b480 buildman: Sort command line options
>>> 5bf27712 buildman: Refactor output options
>>> 461f4050 buildman: Add verbose option to display errors as they happen
>>> 65f46549 buildman: Remove unused non-incremental build method code
>>> 0723a71a buildman: Add an option to specify the buildman config file
>>> b2b89c32 buildman: Add a message indicating there are no errors
>>> eedb4545 buildman: Search for *cc instead of *gcc for the compiler
>>> fe99029f buildman: Add a few more toolchain examples to the README
>>> c976629f RFC: Deprecate MAKEALL
>>>
>>> mpc83xx : 50 boards
>>> freescale : 314 boards
>>> Total boards to build for each commit: 364
>>>
>>> This is not the desired target list.
>>
>> OK I see.
>>
>> But in this case why not just leave off the 'freescale'?
>
> This is just an example. What if I chose "-a arm" and "-v freescale". ARM has
> 300+ targets, but only 20+ are for Freescale. I could save time by building a
> lot less platforms.
>
> The point here is the "OR" logic.
I suppose you could use mx6 or similar, but I take your point.
So what could we do here? Perhaps add a --vendor flag to limit to a
particular vendor? Would that be enough?
>
>>
>>>
>>> Beside, buildman still needs boards.cfg. It takes long to generate this file.
>>
>> So does MAKEALL, right?
>
> Yes. MAKEALL also generates the boards.cfg.
>
>>
>>> Not too long, but if my other tools clean the working directory for each commit,
>>> this accumulates to long time.
>>
>> Can you expand on at a little please? I'm not sure what this refers to.
>>
> Again it is our internal tools of choice. We use Gerrit and Jenkins. Each commit
> triggers a build on Jenkins. Right now I use MAKEALL to build the concerned
> targets. Before each build, the working directory is checkout out to that
> particular commit and cleaned. So if each build needs to generate the
> boards.cfg, a lot of time will be consumed if I have many commits in the queue.
I don't think it works like that. Once you have a boards.cfg you don't
need to regenerate it. Granted if a patch adds a new board there will
be confusion.
>
> But generating boards.cfg is not caused by your patch. I just point out what I saw.
Regards,
Simon
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 02/13] buildman: Add some notes about moving from MAKEALL
2014-08-06 2:15 ` Simon Glass
@ 2014-08-06 2:53 ` York Sun
2014-08-06 14:20 ` Simon Glass
0 siblings, 1 reply; 42+ messages in thread
From: York Sun @ 2014-08-06 2:53 UTC (permalink / raw)
To: u-boot
On 8/5/14 7:15 PM, "Simon Glass" <sjg@chromium.org> wrote:
>>>
>>> But in this case why not just leave off the 'freescale'?
>>
>> This is just an example. What if I chose "-a arm" and "-v freescale".
>>ARM has
>> 300+ targets, but only 20+ are for Freescale. I could save time by
>>building a
>> lot less platforms.
>>
>> The point here is the "OR" logic.
>
>I suppose you could use mx6 or similar, but I take your point.
>
>So what could we do here? Perhaps add a --vendor flag to limit to a
>particular vendor? Would that be enough?
With the ability to build targets for more than one arch, I will be
tempted to use syntax like this
buildman (powerpc & freescale) (arm & freescale) aarch64
buildman mpc85xx mpc83xx mpc86xx (arm & freescale) aarch64
buildman powerpc (arm & freescale) aarch64
I can see myself using these commands if they can be supported.
>
>>
>>>
>>>>
>>>> Beside, buildman still needs boards.cfg. It takes long to generate
>>>>this file.
>>>
>>> So does MAKEALL, right?
>>
>> Yes. MAKEALL also generates the boards.cfg.
>>
>>>
>>>> Not too long, but if my other tools clean the working directory for
>>>>each commit,
>>>> this accumulates to long time.
>>>
>>> Can you expand on at a little please? I'm not sure what this refers to.
>>>
>> Again it is our internal tools of choice. We use Gerrit and Jenkins.
>>Each commit
>> triggers a build on Jenkins. Right now I use MAKEALL to build the
>>concerned
>> targets. Before each build, the working directory is checkout out to
>>that
>> particular commit and cleaned. So if each build needs to generate the
>> boards.cfg, a lot of time will be consumed if I have many commits in
>>the queue.
>
>I don't think it works like that. Once you have a boards.cfg you don't
>need to regenerate it. Granted if a patch adds a new board there will
>be confusion.
I use "git clean -xfqd" after checkout every commit to make sure I have a
clean working directory. Each commit can potentially add/remove a board.
York
>
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 02/13] buildman: Add some notes about moving from MAKEALL
2014-08-06 2:53 ` York Sun
@ 2014-08-06 14:20 ` Simon Glass
2014-08-06 15:06 ` Tom Rini
0 siblings, 1 reply; 42+ messages in thread
From: Simon Glass @ 2014-08-06 14:20 UTC (permalink / raw)
To: u-boot
Hi York,
On 5 August 2014 20:53, York Sun <yorksun@freescale.com> wrote:
>
>
> On 8/5/14 7:15 PM, "Simon Glass" <sjg@chromium.org> wrote:
>>>>
>>>> But in this case why not just leave off the 'freescale'?
>>>
>>> This is just an example. What if I chose "-a arm" and "-v freescale".
>>>ARM has
>>> 300+ targets, but only 20+ are for Freescale. I could save time by
>>>building a
>>> lot less platforms.
>>>
>>> The point here is the "OR" logic.
>>
>>I suppose you could use mx6 or similar, but I take your point.
>>
>>So what could we do here? Perhaps add a --vendor flag to limit to a
>>particular vendor? Would that be enough?
>
> With the ability to build targets for more than one arch, I will be
> tempted to use syntax like this
>
> buildman (powerpc & freescale) (arm & freescale) aarch64
>
> buildman mpc85xx mpc83xx mpc86xx (arm & freescale) aarch64
>
> buildman powerpc (arm & freescale) aarch64
>
> I can see myself using these commands if they can be supported.
Gosh that sounds like a lot of complexity. Can you explain what you
would want these commands to do?
>
>>
>>>
>>>>
>>>>>
>>>>> Beside, buildman still needs boards.cfg. It takes long to generate
>>>>>this file.
>>>>
>>>> So does MAKEALL, right?
>>>
>>> Yes. MAKEALL also generates the boards.cfg.
>>>
>>>>
>>>>> Not too long, but if my other tools clean the working directory for
>>>>>each commit,
>>>>> this accumulates to long time.
>>>>
>>>> Can you expand on at a little please? I'm not sure what this refers to.
>>>>
>>> Again it is our internal tools of choice. We use Gerrit and Jenkins.
>>>Each commit
>>> triggers a build on Jenkins. Right now I use MAKEALL to build the
>>>concerned
>>> targets. Before each build, the working directory is checkout out to
>>>that
>>> particular commit and cleaned. So if each build needs to generate the
>>> boards.cfg, a lot of time will be consumed if I have many commits in
>>>the queue.
>>
>>I don't think it works like that. Once you have a boards.cfg you don't
>>need to regenerate it. Granted if a patch adds a new board there will
>>be confusion.
>
> I use "git clean -xfqd" after checkout every commit to make sure I have a
> clean working directory. Each commit can potentially add/remove a board.
Then again, you will have the same problem with MAKEALL so it is more
a Kconfig/process question than a buildman one.
Regards,
Simon
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 02/13] buildman: Add some notes about moving from MAKEALL
2014-08-06 14:20 ` Simon Glass
@ 2014-08-06 15:06 ` Tom Rini
2014-08-07 12:12 ` Simon Glass
0 siblings, 1 reply; 42+ messages in thread
From: Tom Rini @ 2014-08-06 15:06 UTC (permalink / raw)
To: u-boot
On Wed, Aug 06, 2014 at 08:20:47AM -0600, Simon Glass wrote:
> Hi York,
>
> On 5 August 2014 20:53, York Sun <yorksun@freescale.com> wrote:
> >
> >
> > On 8/5/14 7:15 PM, "Simon Glass" <sjg@chromium.org> wrote:
> >>>>
> >>>> But in this case why not just leave off the 'freescale'?
> >>>
> >>> This is just an example. What if I chose "-a arm" and "-v freescale".
> >>>ARM has
> >>> 300+ targets, but only 20+ are for Freescale. I could save time by
> >>>building a
> >>> lot less platforms.
> >>>
> >>> The point here is the "OR" logic.
> >>
> >>I suppose you could use mx6 or similar, but I take your point.
> >>
> >>So what could we do here? Perhaps add a --vendor flag to limit to a
> >>particular vendor? Would that be enough?
> >
> > With the ability to build targets for more than one arch, I will be
> > tempted to use syntax like this
> >
> > buildman (powerpc & freescale) (arm & freescale) aarch64
Spaces outside of parens are implicit ORs here, so all
powerpc+freescale, all arm+freescale and all aarch64. In the fullness
of time the last one might become all freescale+aarch64. Bonus points
if we can easily write 'buildman ... ((powerpc|arm|aarch64) & freescale)'
Thinking out loud, the problem is today we have | for OR but we don't
have an AND symbol. I can do:
$ buildman 'arm|powerpc|aarch64'
Today and get what I want. But I can't:
$ buildman '(powerpc&freescale)|(arm&freescale)'
--
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20140806/0ef81488/attachment.pgp>
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 02/13] buildman: Add some notes about moving from MAKEALL
2014-08-06 15:06 ` Tom Rini
@ 2014-08-07 12:12 ` Simon Glass
2014-08-07 13:14 ` Masahiro Yamada
0 siblings, 1 reply; 42+ messages in thread
From: Simon Glass @ 2014-08-07 12:12 UTC (permalink / raw)
To: u-boot
Hi Tom, York,
On 6 August 2014 09:06, Tom Rini <trini@ti.com> wrote:
> On Wed, Aug 06, 2014 at 08:20:47AM -0600, Simon Glass wrote:
>> Hi York,
>>
>> On 5 August 2014 20:53, York Sun <yorksun@freescale.com> wrote:
>> >
>> >
>> > On 8/5/14 7:15 PM, "Simon Glass" <sjg@chromium.org> wrote:
>> >>>>
>> >>>> But in this case why not just leave off the 'freescale'?
>> >>>
>> >>> This is just an example. What if I chose "-a arm" and "-v freescale".
>> >>>ARM has
>> >>> 300+ targets, but only 20+ are for Freescale. I could save time by
>> >>>building a
>> >>> lot less platforms.
>> >>>
>> >>> The point here is the "OR" logic.
>> >>
>> >>I suppose you could use mx6 or similar, but I take your point.
>> >>
>> >>So what could we do here? Perhaps add a --vendor flag to limit to a
>> >>particular vendor? Would that be enough?
>> >
>> > With the ability to build targets for more than one arch, I will be
>> > tempted to use syntax like this
>> >
>> > buildman (powerpc & freescale) (arm & freescale) aarch64
>
> Spaces outside of parens are implicit ORs here, so all
> powerpc+freescale, all arm+freescale and all aarch64. In the fullness
> of time the last one might become all freescale+aarch64. Bonus points
> if we can easily write 'buildman ... ((powerpc|arm|aarch64) & freescale)'
>
> Thinking out loud, the problem is today we have | for OR but we don't
> have an AND symbol. I can do:
> $ buildman 'arm|powerpc|aarch64'
> Today and get what I want. But I can't:
> $ buildman '(powerpc&freescale)|(arm&freescale)'
OK I'll have a crack at this.
Regards,
Simon
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 02/13] buildman: Add some notes about moving from MAKEALL
2014-08-07 12:12 ` Simon Glass
@ 2014-08-07 13:14 ` Masahiro Yamada
2014-08-08 10:59 ` Simon Glass
0 siblings, 1 reply; 42+ messages in thread
From: Masahiro Yamada @ 2014-08-07 13:14 UTC (permalink / raw)
To: u-boot
Hi.
On Thu, 7 Aug 2014 06:12:18 -0600
Simon Glass <sjg@chromium.org> wrote:
> Hi Tom, York,
>
> On 6 August 2014 09:06, Tom Rini <trini@ti.com> wrote:
> > On Wed, Aug 06, 2014 at 08:20:47AM -0600, Simon Glass wrote:
> >> Hi York,
> >>
> >> On 5 August 2014 20:53, York Sun <yorksun@freescale.com> wrote:
> >> >
> >> >
> >> > On 8/5/14 7:15 PM, "Simon Glass" <sjg@chromium.org> wrote:
> >> >>>>
> >> >>>> But in this case why not just leave off the 'freescale'?
> >> >>>
> >> >>> This is just an example. What if I chose "-a arm" and "-v freescale".
> >> >>>ARM has
> >> >>> 300+ targets, but only 20+ are for Freescale. I could save time by
> >> >>>building a
> >> >>> lot less platforms.
> >> >>>
> >> >>> The point here is the "OR" logic.
> >> >>
> >> >>I suppose you could use mx6 or similar, but I take your point.
> >> >>
> >> >>So what could we do here? Perhaps add a --vendor flag to limit to a
> >> >>particular vendor? Would that be enough?
> >> >
> >> > With the ability to build targets for more than one arch, I will be
> >> > tempted to use syntax like this
> >> >
> >> > buildman (powerpc & freescale) (arm & freescale) aarch64
> >
> > Spaces outside of parens are implicit ORs here, so all
> > powerpc+freescale, all arm+freescale and all aarch64. In the fullness
> > of time the last one might become all freescale+aarch64. Bonus points
> > if we can easily write 'buildman ... ((powerpc|arm|aarch64) & freescale)'
> >
> > Thinking out loud, the problem is today we have | for OR but we don't
> > have an AND symbol. I can do:
> > $ buildman 'arm|powerpc|aarch64'
> > Today and get what I want. But I can't:
> > $ buildman '(powerpc&freescale)|(arm&freescale)'
>
> OK I'll have a crack at this.
>
I am afraid the requirement for buildman is getting too complicated.
> > $ buildman '(powerpc&freescale)|(arm&freescale)'
This feature completely depends on boards.cfg
but we want to completely break with it, don't we?
Grepping Kconfig for searching Freescale boards
is almost impossible, I think.
Maybe it is time to change our mind.
But I have no idea for the replacement.
Best Regards
Masahiro Yamada
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 02/13] buildman: Add some notes about moving from MAKEALL
2014-08-07 13:14 ` Masahiro Yamada
@ 2014-08-08 10:59 ` Simon Glass
0 siblings, 0 replies; 42+ messages in thread
From: Simon Glass @ 2014-08-08 10:59 UTC (permalink / raw)
To: u-boot
Hi Masahiro,
On 7 August 2014 07:14, Masahiro Yamada <yamada.m@jp.panasonic.com> wrote:
> Hi.
>
>
>
> On Thu, 7 Aug 2014 06:12:18 -0600
> Simon Glass <sjg@chromium.org> wrote:
>
>> Hi Tom, York,
>>
>> On 6 August 2014 09:06, Tom Rini <trini@ti.com> wrote:
>> > On Wed, Aug 06, 2014 at 08:20:47AM -0600, Simon Glass wrote:
>> >> Hi York,
>> >>
>> >> On 5 August 2014 20:53, York Sun <yorksun@freescale.com> wrote:
>> >> >
>> >> >
>> >> > On 8/5/14 7:15 PM, "Simon Glass" <sjg@chromium.org> wrote:
>> >> >>>>
>> >> >>>> But in this case why not just leave off the 'freescale'?
>> >> >>>
>> >> >>> This is just an example. What if I chose "-a arm" and "-v freescale".
>> >> >>>ARM has
>> >> >>> 300+ targets, but only 20+ are for Freescale. I could save time by
>> >> >>>building a
>> >> >>> lot less platforms.
>> >> >>>
>> >> >>> The point here is the "OR" logic.
>> >> >>
>> >> >>I suppose you could use mx6 or similar, but I take your point.
>> >> >>
>> >> >>So what could we do here? Perhaps add a --vendor flag to limit to a
>> >> >>particular vendor? Would that be enough?
>> >> >
>> >> > With the ability to build targets for more than one arch, I will be
>> >> > tempted to use syntax like this
>> >> >
>> >> > buildman (powerpc & freescale) (arm & freescale) aarch64
>> >
>> > Spaces outside of parens are implicit ORs here, so all
>> > powerpc+freescale, all arm+freescale and all aarch64. In the fullness
>> > of time the last one might become all freescale+aarch64. Bonus points
>> > if we can easily write 'buildman ... ((powerpc|arm|aarch64) & freescale)'
>> >
>> > Thinking out loud, the problem is today we have | for OR but we don't
>> > have an AND symbol. I can do:
>> > $ buildman 'arm|powerpc|aarch64'
>> > Today and get what I want. But I can't:
>> > $ buildman '(powerpc&freescale)|(arm&freescale)'
>>
>> OK I'll have a crack at this.
>>
>
> I am afraid the requirement for buildman is getting too complicated.
>
>
>> > $ buildman '(powerpc&freescale)|(arm&freescale)'
>
> This feature completely depends on boards.cfg
> but we want to completely break with it, don't we?
>
> Grepping Kconfig for searching Freescale boards
> is almost impossible, I think.
Actually buildman requires this information regardless, so it knows
what boards to build. It needs a list of all boards even if there are
no conditions.
>
> Maybe it is time to change our mind.
> But I have no idea for the replacement.
Let's continue in the other thread.
Regards,
Simon
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 13/13] RFC: Deprecate MAKEALL
2014-08-05 22:06 ` York Sun
@ 2014-08-08 21:12 ` Tom Rini
2014-08-08 21:19 ` York Sun
0 siblings, 1 reply; 42+ messages in thread
From: Tom Rini @ 2014-08-08 21:12 UTC (permalink / raw)
To: u-boot
On Tue, Aug 05, 2014 at 03:06:55PM -0700, York Sun wrote:
> On 08/05/2014 12:10 PM, Tom Rini wrote:
> > On Tue, Aug 05, 2014 at 12:01:33PM -0700, York Sun wrote:
> >> On 08/05/2014 11:59 AM, Simon Glass wrote:
> >>> Hi York,
> >>>
> >>> On 5 August 2014 12:55, York Sun <yorksun@freescale.com> wrote:
> >>>> On 08/05/2014 11:52 AM, Simon Glass wrote:
> >>>>>>>
> >>>>>>> BTW where does this information get published?
> >>>>>>>
> >>>>>>
> >>>>>> When we use gerrit and Jenkins, the build log is stored in Jenkins log, and
> >>>>>> emails were sent with the failure log.
> >>>>>
> >>>>> OK, I don't think I've send such an email. Do I have to do anything to
> >>>>> get them to come to me?
> >>>>>
> >>>> The emails were sent from Jenkins, by my script collecting information from .ERR
> >>>> files.
> >>>>
> >>>> Maybe I didn't understand your question. What do you mean "where does this
> >>>> information get published"?
> >>>
> >>> I just mean that I don't think I have received an email yet, and don't
> >>> know about the service, so am wondering how to get an email.
> >>>
> >>
> >> Simon,
> >>
> >> Sorry I misled you. The emails are sent by my scripts under Jenkins (A
> >> tool, see http://jenkins-ci.org/). It is not within u-boot.
> >
> > Would you be able to post your jobs / scripts somewhere? I've toyed
> > with a personal setup and I know Marek has run one before as well.
> > Thanks!
> >
> Tom,
>
> My script is long and messy. This is the part sending emails.
Oh I'm sure, but maybe we can help get everyone a better script. For
example, you've said the way your job goes currently is every commit
gets a test. But.. that's not what you need exactly it sounds like. It
sounds like you need a given gerrit issue, which can have a few commits
to it, to be bisectable and compile tested. buildman will handle this
part for you, if you let it.
--
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20140808/c7c2bd60/attachment.pgp>
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 13/13] RFC: Deprecate MAKEALL
2014-08-08 21:12 ` Tom Rini
@ 2014-08-08 21:19 ` York Sun
2014-08-08 21:30 ` Tom Rini
0 siblings, 1 reply; 42+ messages in thread
From: York Sun @ 2014-08-08 21:19 UTC (permalink / raw)
To: u-boot
On 08/08/2014 02:12 PM, Tom Rini wrote:
> On Tue, Aug 05, 2014 at 03:06:55PM -0700, York Sun wrote:
>> On 08/05/2014 12:10 PM, Tom Rini wrote:
>>> On Tue, Aug 05, 2014 at 12:01:33PM -0700, York Sun wrote:
>>>> On 08/05/2014 11:59 AM, Simon Glass wrote:
>>>>> Hi York,
>>>>>
>>>>> On 5 August 2014 12:55, York Sun <yorksun@freescale.com> wrote:
>>>>>> On 08/05/2014 11:52 AM, Simon Glass wrote:
>>>>>>>>>
>>>>>>>>> BTW where does this information get published?
>>>>>>>>>
>>>>>>>>
>>>>>>>> When we use gerrit and Jenkins, the build log is stored in Jenkins log, and
>>>>>>>> emails were sent with the failure log.
>>>>>>>
>>>>>>> OK, I don't think I've send such an email. Do I have to do anything to
>>>>>>> get them to come to me?
>>>>>>>
>>>>>> The emails were sent from Jenkins, by my script collecting information from .ERR
>>>>>> files.
>>>>>>
>>>>>> Maybe I didn't understand your question. What do you mean "where does this
>>>>>> information get published"?
>>>>>
>>>>> I just mean that I don't think I have received an email yet, and don't
>>>>> know about the service, so am wondering how to get an email.
>>>>>
>>>>
>>>> Simon,
>>>>
>>>> Sorry I misled you. The emails are sent by my scripts under Jenkins (A
>>>> tool, see http://jenkins-ci.org/). It is not within u-boot.
>>>
>>> Would you be able to post your jobs / scripts somewhere? I've toyed
>>> with a personal setup and I know Marek has run one before as well.
>>> Thanks!
>>>
>> Tom,
>>
>> My script is long and messy. This is the part sending emails.
>
> Oh I'm sure, but maybe we can help get everyone a better script. For
> example, you've said the way your job goes currently is every commit
> gets a test. But.. that's not what you need exactly it sounds like. It
> sounds like you need a given gerrit issue, which can have a few commits
> to it, to be bisectable and compile tested. buildman will handle this
> part for you, if you let it.
>
Tom,
I agree buildman has improvements comparing with MAKEALL. I spent time to make
Gerrit, Jenkins and MAKEALL work together. It is not easy as one would expect.
For now, if I can make builmand to build the last commit, I can replace MAKEALL
with buildman. But that seems not working.
York
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 13/13] RFC: Deprecate MAKEALL
2014-08-08 21:19 ` York Sun
@ 2014-08-08 21:30 ` Tom Rini
2014-08-08 21:36 ` York Sun
0 siblings, 1 reply; 42+ messages in thread
From: Tom Rini @ 2014-08-08 21:30 UTC (permalink / raw)
To: u-boot
On Fri, Aug 08, 2014 at 02:19:34PM -0700, York Sun wrote:
> On 08/08/2014 02:12 PM, Tom Rini wrote:
[snip]
> > Oh I'm sure, but maybe we can help get everyone a better script. For
> > example, you've said the way your job goes currently is every commit
> > gets a test. But.. that's not what you need exactly it sounds like. It
> > sounds like you need a given gerrit issue, which can have a few commits
> > to it, to be bisectable and compile tested. buildman will handle this
> > part for you, if you let it.
> >
> Tom,
>
> I agree buildman has improvements comparing with MAKEALL. I spent time
> to make Gerrit, Jenkins and MAKEALL work together. It is not easy as
> one would expect. For now, if I can make builmand to build the last
> commit, I can replace MAKEALL with buildman. But that seems not
> working.
I believe you, I've spent a bunch of time inside of Jenkins myself (and
then with IBM's LSF compute setup) and I'm doing some of the same
hoop-jumping now too. But I'm hoping maybe we can all take this as a
chance to improve the various ad-hoc builder setups many of us have.
--
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20140808/958d041e/attachment.pgp>
^ permalink raw reply [flat|nested] 42+ messages in thread
* [U-Boot] [PATCH v4 13/13] RFC: Deprecate MAKEALL
2014-08-08 21:30 ` Tom Rini
@ 2014-08-08 21:36 ` York Sun
0 siblings, 0 replies; 42+ messages in thread
From: York Sun @ 2014-08-08 21:36 UTC (permalink / raw)
To: u-boot
On 08/08/2014 02:30 PM, Tom Rini wrote:
> On Fri, Aug 08, 2014 at 02:19:34PM -0700, York Sun wrote:
>> On 08/08/2014 02:12 PM, Tom Rini wrote:
> [snip]
>>> Oh I'm sure, but maybe we can help get everyone a better script. For
>>> example, you've said the way your job goes currently is every commit
>>> gets a test. But.. that's not what you need exactly it sounds like. It
>>> sounds like you need a given gerrit issue, which can have a few commits
>>> to it, to be bisectable and compile tested. buildman will handle this
>>> part for you, if you let it.
>>>
>> Tom,
>>
>> I agree buildman has improvements comparing with MAKEALL. I spent time
>> to make Gerrit, Jenkins and MAKEALL work together. It is not easy as
>> one would expect. For now, if I can make builmand to build the last
>> commit, I can replace MAKEALL with buildman. But that seems not
>> working.
>
> I believe you, I've spent a bunch of time inside of Jenkins myself (and
> then with IBM's LSF compute setup) and I'm doing some of the same
> hoop-jumping now too. But I'm hoping maybe we can all take this as a
> chance to improve the various ad-hoc builder setups many of us have.
>
Since we use Gerrit for internal review, each commit (proposed patch) triggers a
Jenkins event. I can change the script so no build is triggered, but I need to
explorer if buildman servers better than MAKEALL in this case. Because Jenkins
has no knowledge of the number of patches pushed, I don't think buildman has
enough information to find the "top" and start the build.
But anyway, it shouldn't stop us from moving to buildman. I just need the basic
feature available before removing MAKEALL.
York
^ permalink raw reply [flat|nested] 42+ messages in thread
end of thread, other threads:[~2014-08-08 21:36 UTC | newest]
Thread overview: 42+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-08-05 14:46 [U-Boot] [PATCH v4 0/13] Add some missing buildman features and deprecate MAKEALL Simon Glass
2014-08-05 14:46 ` [U-Boot] [PATCH v4 01/13] buildman: Fix a few typos Simon Glass
2014-08-05 14:46 ` [U-Boot] [PATCH v4 02/13] buildman: Add some notes about moving from MAKEALL Simon Glass
2014-08-05 22:34 ` York Sun
2014-08-05 23:07 ` Simon Glass
2014-08-05 23:21 ` York Sun
2014-08-06 2:15 ` Simon Glass
2014-08-06 2:53 ` York Sun
2014-08-06 14:20 ` Simon Glass
2014-08-06 15:06 ` Tom Rini
2014-08-07 12:12 ` Simon Glass
2014-08-07 13:14 ` Masahiro Yamada
2014-08-08 10:59 ` Simon Glass
2014-08-05 14:46 ` [U-Boot] [PATCH v4 03/13] buildman: Allow building of current source tree Simon Glass
2014-08-05 18:54 ` Tom Rini
2014-08-05 18:58 ` Simon Glass
2014-08-05 19:01 ` Tom Rini
2014-08-05 14:46 ` [U-Boot] [PATCH v4 04/13] buildman: Move BuilderThread code to its own file Simon Glass
2014-08-05 14:46 ` [U-Boot] [PATCH v4 05/13] buildman: Sort command line options Simon Glass
2014-08-05 14:46 ` [U-Boot] [PATCH v4 06/13] buildman: Refactor output options Simon Glass
2014-08-05 14:46 ` [U-Boot] [PATCH v4 07/13] buildman: Add verbose option to display errors as they happen Simon Glass
2014-08-05 14:46 ` [U-Boot] [PATCH v4 08/13] buildman: Remove unused non-incremental build method code Simon Glass
2014-08-05 14:46 ` [U-Boot] [PATCH v4 09/13] buildman: Add an option to specify the buildman config file Simon Glass
2014-08-05 14:47 ` [U-Boot] [PATCH v4 10/13] buildman: Add a message indicating there are no errors Simon Glass
2014-08-05 14:47 ` [U-Boot] [PATCH v4 11/13] buildman: Search for *cc instead of *gcc for the compiler Simon Glass
2014-08-05 17:41 ` Jeroen Hofstee
2014-08-05 18:25 ` Simon Glass
2014-08-05 14:47 ` [U-Boot] [PATCH v4 12/13] buildman: Add a few more toolchain examples to the README Simon Glass
2014-08-05 14:47 ` [U-Boot] [PATCH v4 13/13] RFC: Deprecate MAKEALL Simon Glass
2014-08-05 16:43 ` York Sun
2014-08-05 18:41 ` Simon Glass
2014-08-05 18:48 ` York Sun
2014-08-05 18:52 ` Simon Glass
2014-08-05 18:55 ` York Sun
2014-08-05 18:59 ` Simon Glass
2014-08-05 19:01 ` York Sun
2014-08-05 19:10 ` Tom Rini
2014-08-05 22:06 ` York Sun
2014-08-08 21:12 ` Tom Rini
2014-08-08 21:19 ` York Sun
2014-08-08 21:30 ` Tom Rini
2014-08-08 21:36 ` York Sun
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.