* [PATCH 01/19] buildman: Drop -I option
2022-07-12 1:03 [PATCH 00/19] buildman: Integration of boards.cfg file Simon Glass
@ 2022-07-12 1:03 ` Simon Glass
2022-07-12 1:03 ` [PATCH 02/19] buildman: Support running from an IDE Simon Glass
` (18 subsequent siblings)
19 siblings, 0 replies; 31+ messages in thread
From: Simon Glass @ 2022-07-12 1:03 UTC (permalink / raw)
To: U-Boot Mailing List; +Cc: Tom Rini, Simon Glass
This has been deprecated with a notice that it will be removed after April
2021. Drop it now.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
tools/buildman/cmdline.py | 3 ---
tools/buildman/control.py | 3 ---
2 files changed, 6 deletions(-)
diff --git a/tools/buildman/cmdline.py b/tools/buildman/cmdline.py
index 8586bdf3b58..ddcb1ec5fcd 100644
--- a/tools/buildman/cmdline.py
+++ b/tools/buildman/cmdline.py
@@ -59,9 +59,6 @@ def ParseArgs():
parser.add_option('-i', '--in-tree', dest='in_tree',
action='store_true', default=False,
help='Build in the source tree instead of a separate directory')
- # -I will be removed after April 2021
- parser.add_option('-I', '--incremental', action='store_true',
- default=False, help='Deprecated, does nothing. See -m')
parser.add_option('-j', '--jobs', dest='jobs', type='int',
default=None, help='Number of jobs to run at once (passed to make)')
parser.add_option('-k', '--keep-outputs', action='store_true',
diff --git a/tools/buildman/control.py b/tools/buildman/control.py
index 8f4810bc3ef..beade62408b 100644
--- a/tools/buildman/control.py
+++ b/tools/buildman/control.py
@@ -176,9 +176,6 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None,
print()
return 0
- if options.incremental:
- print(col.build(col.RED,
- 'Warning: -I has been removed. See documentation'))
if not options.output_dir:
if options.work_in_output:
sys.exit(col.build(col.RED, '-w requires that you specify -o'))
--
2.37.0.144.g8ac04bfd2-goog
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 02/19] buildman: Support running from an IDE
2022-07-12 1:03 [PATCH 00/19] buildman: Integration of boards.cfg file Simon Glass
2022-07-12 1:03 ` [PATCH 01/19] buildman: Drop -I option Simon Glass
@ 2022-07-12 1:03 ` Simon Glass
2022-07-12 1:03 ` [PATCH 03/19] buildman: Avoid using board as a variable Simon Glass
` (17 subsequent siblings)
19 siblings, 0 replies; 31+ messages in thread
From: Simon Glass @ 2022-07-12 1:03 UTC (permalink / raw)
To: U-Boot Mailing List; +Cc: Tom Rini, Simon Glass
Add a flag to allow buildman to behave properly for use from an IDE. This
shows error/warning output on stderr and drops all summary and progress
information.
This should normally only be used when building a single board.
Fix up a confusing comment for GetResultSummary() while we are here, since
we want to use the Outcome object to access the unprocessed error lines
from the build.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
tools/buildman/README | 15 +++++++++
tools/buildman/builder.py | 68 ++++++++++++++++++++++++---------------
tools/buildman/cmdline.py | 2 ++
tools/buildman/control.py | 7 ++--
4 files changed, 63 insertions(+), 29 deletions(-)
diff --git a/tools/buildman/README b/tools/buildman/README
index 49438cb909d..c67a074cb50 100644
--- a/tools/buildman/README
+++ b/tools/buildman/README
@@ -1092,6 +1092,21 @@ This will write the full build into /tmp/build including object files. You must
specify the output directory with -o when using -w.
+Support for IDEs (Integrated Development Environments)
+======================================================
+
+Normally buildman summarises the output and shows information indicating the
+meaning of each line of output. For example a '+' symbol appears at the start of
+each error line. Also, buildman prints information about what it is about to do,
+along with a summary at the end.
+
+When using buildman from an IDE, it is helpful to drop this behaviour. Use the
+-I/--ide option for that. You might find -W helpful also so that warnings do
+not cause the build to fail:
+
+ buildman -o /tmp/build --board sandbox -wWI
+
+
Changing the configuration
==========================
diff --git a/tools/buildman/builder.py b/tools/buildman/builder.py
index aa2ffe16f6c..078ddf070d3 100644
--- a/tools/buildman/builder.py
+++ b/tools/buildman/builder.py
@@ -213,6 +213,8 @@ class Builder:
threading is not being used
_terminated: Thread was terminated due to an error
_restarting_config: True if 'Restart config' is detected in output
+ _ide: Produce output suitable for an Integrated Development Environment,
+ i.e. dont emit progress information and put errors/warnings on stderr
"""
class Outcome:
"""Records a build outcome for a single make invocation
@@ -325,6 +327,7 @@ class Builder:
self.config_filenames = BASE_CONFIG_FILENAMES
self.work_in_output = work_in_output
self.adjust_cfg = adjust_cfg
+ self._ide = False
if not self.squash_config_y:
self.config_filenames += EXTRA_CONFIG_FILENAMES
@@ -382,7 +385,7 @@ class Builder:
show_detail=False, show_bloat=False,
list_error_boards=False, show_config=False,
show_environment=False, filter_dtb_warnings=False,
- filter_migration_warnings=False):
+ filter_migration_warnings=False, ide=False):
"""Setup display options for the builder.
Args:
@@ -397,6 +400,8 @@ class Builder:
compiler
filter_migration_warnings: Filter out any warnings about migrating
a board to driver model
+ ide: Create output that can be parsed by an IDE. There is no '+' prefix on
+ error lines and output on stderr stays on stderr.
"""
self._show_errors = show_errors
self._show_sizes = show_sizes
@@ -407,6 +412,7 @@ class Builder:
self._show_environment = show_environment
self._filter_dtb_warnings = filter_dtb_warnings
self._filter_migration_warnings = filter_migration_warnings
+ self._ide = ide
def _AddTimestamp(self):
"""Add a new timestamp to the list and record the build period.
@@ -535,8 +541,9 @@ class Builder:
line += '%s : ' % self._complete_delay
line += target
- terminal.print_clear()
- tprint(line, newline=False, limit_to_line=True)
+ if not self._ide:
+ terminal.print_clear()
+ tprint(line, newline=False, limit_to_line=True)
def _GetOutputDir(self, commit_upto):
"""Get the name of the output directory for a commit number
@@ -834,8 +841,9 @@ class Builder:
Returns:
Tuple:
- Dict containing boards which passed building this commit.
- keyed by board.target
+ Dict containing boards which built this commit:
+ key: board.target
+ value: Builder.Outcome object
List containing a summary of error lines
Dict keyed by error line, containing a list of the Board
objects with that error
@@ -1369,8 +1377,14 @@ class Builder:
better_warn, worse_warn = _CalcErrorDelta(self._base_warn_lines,
self._base_warn_line_boards, warn_lines, warn_line_boards, 'w')
+ # For the IDE mode, print out all the output
+ if self._ide:
+ outcome = board_dict[target]
+ for line in outcome.err_lines:
+ sys.stderr.write(line)
+
# Display results by arch
- if any((ok_boards, warn_boards, err_boards, unknown_boards, new_boards,
+ elif any((ok_boards, warn_boards, err_boards, unknown_boards, new_boards,
worse_err, better_err, worse_warn, better_warn)):
arch_list = {}
self.AddOutcome(board_selected, arch_list, ok_boards, '',
@@ -1746,7 +1760,8 @@ class Builder:
self._PrepareWorkingSpace(min(self.num_threads, len(board_selected)),
commits is not None)
self._PrepareOutputSpace()
- tprint('\rStarting build...', newline=False)
+ if not self._ide:
+ tprint('\rStarting build...', newline=False)
self.SetupBuild(board_selected, commits)
self.ProcessResult(None)
self.thread_exceptions = []
@@ -1773,24 +1788,25 @@ class Builder:
# Wait until we have processed all output
self.out_queue.join()
- tprint()
-
- msg = 'Completed: %d total built' % self.count
- if self.already_done:
- msg += ' (%d previously' % self.already_done
- if self.already_done != self.count:
- msg += ', %d newly' % (self.count - self.already_done)
- msg += ')'
- duration = datetime.now() - self._start_time
- if duration > timedelta(microseconds=1000000):
- if duration.microseconds >= 500000:
- duration = duration + timedelta(seconds=1)
- duration = duration - timedelta(microseconds=duration.microseconds)
- rate = float(self.count) / duration.total_seconds()
- msg += ', duration %s, rate %1.2f' % (duration, rate)
- tprint(msg)
- if self.thread_exceptions:
- tprint('Failed: %d thread exceptions' % len(self.thread_exceptions),
- colour=self.col.RED)
+ if not self._ide:
+ tprint()
+
+ msg = 'Completed: %d total built' % self.count
+ if self.already_done:
+ msg += ' (%d previously' % self.already_done
+ if self.already_done != self.count:
+ msg += ', %d newly' % (self.count - self.already_done)
+ msg += ')'
+ duration = datetime.now() - self._start_time
+ if duration > timedelta(microseconds=1000000):
+ if duration.microseconds >= 500000:
+ duration = duration + timedelta(seconds=1)
+ duration = duration - timedelta(microseconds=duration.microseconds)
+ rate = float(self.count) / duration.total_seconds()
+ msg += ', duration %s, rate %1.2f' % (duration, rate)
+ tprint(msg)
+ if self.thread_exceptions:
+ tprint('Failed: %d thread exceptions' % len(self.thread_exceptions),
+ colour=self.col.RED)
return (self.fail, self.warned, self.thread_exceptions)
diff --git a/tools/buildman/cmdline.py b/tools/buildman/cmdline.py
index ddcb1ec5fcd..cef2068c983 100644
--- a/tools/buildman/cmdline.py
+++ b/tools/buildman/cmdline.py
@@ -59,6 +59,8 @@ def ParseArgs():
parser.add_option('-i', '--in-tree', dest='in_tree',
action='store_true', default=False,
help='Build in the source tree instead of a separate directory')
+ parser.add_option('-I', '--ide', action='store_true', default=False,
+ help='Create build output that can be parsed by an IDE')
parser.add_option('-j', '--jobs', dest='jobs', type='int',
default=None, help='Number of jobs to run at once (passed to make)')
parser.add_option('-k', '--keep-outputs', action='store_true',
diff --git a/tools/buildman/control.py b/tools/buildman/control.py
index beade62408b..68dca97413f 100644
--- a/tools/buildman/control.py
+++ b/tools/buildman/control.py
@@ -359,8 +359,9 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None,
else:
commits = None
- tprint(GetActionSummary(options.summary, commits, board_selected,
- options))
+ if not options.ide:
+ tprint(GetActionSummary(options.summary, commits, board_selected,
+ options))
# We can't show function sizes without board details at present
if options.show_bloat:
@@ -369,7 +370,7 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None,
options.show_errors, options.show_sizes, options.show_detail,
options.show_bloat, options.list_error_boards, options.show_config,
options.show_environment, options.filter_dtb_warnings,
- options.filter_migration_warnings)
+ options.filter_migration_warnings, options.ide)
if options.summary:
builder.ShowSummary(commits, board_selected)
else:
--
2.37.0.144.g8ac04bfd2-goog
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 03/19] buildman: Avoid using board as a variable
2022-07-12 1:03 [PATCH 00/19] buildman: Integration of boards.cfg file Simon Glass
2022-07-12 1:03 ` [PATCH 01/19] buildman: Drop -I option Simon Glass
2022-07-12 1:03 ` [PATCH 02/19] buildman: Support running from an IDE Simon Glass
@ 2022-07-12 1:03 ` Simon Glass
2022-07-12 1:03 ` [PATCH 04/19] buildman: Fix use of 'boards' in func_test Simon Glass
` (16 subsequent siblings)
19 siblings, 0 replies; 31+ messages in thread
From: Simon Glass @ 2022-07-12 1:03 UTC (permalink / raw)
To: U-Boot Mailing List; +Cc: Tom Rini, Simon Glass
We have a module called 'board'. Sometimes buildman uses 'brd' as an
instance variable but sometimes it uses 'board', which is confusing and
can mess with the module handling. Update the code to use 'brd'
consistently, making it easier for tools to determine when the module
is being referenced.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
tools/buildman/board.py | 40 ++++++++++++++---------------
tools/buildman/builder.py | 45 ++++++++++++++++-----------------
tools/buildman/builderthread.py | 6 ++---
tools/buildman/func_test.py | 8 +++---
tools/buildman/test.py | 4 +--
tools/buildman/toolchain.py | 8 +++---
6 files changed, 55 insertions(+), 56 deletions(-)
diff --git a/tools/buildman/board.py b/tools/buildman/board.py
index 447aaabea86..08771b3f158 100644
--- a/tools/buildman/board.py
+++ b/tools/buildman/board.py
@@ -103,15 +103,15 @@ class Boards:
# Use a simple list here, sinc OrderedDict requires Python 2.7
self._boards = []
- def AddBoard(self, board):
+ def AddBoard(self, brd):
"""Add a new board to the list.
The board's target member must not already exist in the board list.
Args:
- board: board to add
+ brd: board to add
"""
- self._boards.append(board)
+ self._boards.append(brd)
def ReadBoards(self, fname):
"""Read a list of boards from a board file.
@@ -136,8 +136,8 @@ class Boards:
if len(fields) > 8:
fields = fields[:8]
- board = Board(*fields)
- self.AddBoard(board)
+ brd = Board(*fields)
+ self.AddBoard(brd)
def GetList(self):
@@ -157,8 +157,8 @@ class Boards:
value is board
"""
board_dict = OrderedDict()
- for board in self._boards:
- board_dict[board.target] = board
+ for brd in self._boards:
+ board_dict[brd.target] = brd
return board_dict
def GetSelectedDict(self):
@@ -168,9 +168,9 @@ class Boards:
List of Board objects that are marked selected
"""
board_dict = OrderedDict()
- for board in self._boards:
- if board.build_it:
- board_dict[board.target] = board
+ for brd in self._boards:
+ if brd.build_it:
+ board_dict[brd.target] = brd
return board_dict
def GetSelected(self):
@@ -179,7 +179,7 @@ class Boards:
Returns:
List of Board objects that are marked selected
"""
- return [board for board in self._boards if board.build_it]
+ return [brd for brd in self._boards if brd.build_it]
def GetSelectedNames(self):
"""Return a list of selected boards
@@ -187,7 +187,7 @@ class Boards:
Returns:
List of board names that are marked selected
"""
- return [board.target for board in self._boards if board.build_it]
+ return [brd.target for brd in self._boards if brd.build_it]
def _BuildTerms(self, args):
"""Convert command line arguments to a list of terms.
@@ -273,34 +273,34 @@ class Boards:
exclude_list.append(Expr(expr))
found = []
- for board in self._boards:
+ for brd in self._boards:
matching_term = None
build_it = False
if terms:
match = False
for term in terms:
- if term.Matches(board.props):
+ if term.Matches(brd.props):
matching_term = str(term)
build_it = True
break
elif boards:
- if board.target in boards:
+ if brd.target in boards:
build_it = True
- found.append(board.target)
+ found.append(brd.target)
else:
build_it = True
# Check that it is not specifically excluded
for expr in exclude_list:
- if expr.Matches(board.props):
+ if expr.Matches(brd.props):
build_it = False
break
if build_it:
- board.build_it = True
+ brd.build_it = True
if matching_term:
- result[matching_term].append(board.target)
- result['all'].append(board.target)
+ result[matching_term].append(brd.target)
+ result['all'].append(brd.target)
if boards:
remaining = set(boards) - set(found)
diff --git a/tools/buildman/builder.py b/tools/buildman/builder.py
index 078ddf070d3..462266b4b84 100644
--- a/tools/buildman/builder.py
+++ b/tools/buildman/builder.py
@@ -875,11 +875,11 @@ class Builder:
config = {}
environment = {}
- for board in boards_selected.values():
- outcome = self.GetBuildOutcome(commit_upto, board.target,
+ for brd in boards_selected.values():
+ outcome = self.GetBuildOutcome(commit_upto, brd.target,
read_func_sizes, read_config,
read_environment)
- board_dict[board.target] = outcome
+ board_dict[brd.target] = outcome
last_func = None
last_was_warning = False
for line in outcome.err_lines:
@@ -894,29 +894,29 @@ class Builder:
if is_warning or (last_was_warning and is_note):
if last_func:
AddLine(warn_lines_summary, warn_lines_boards,
- last_func, board)
+ last_func, brd)
AddLine(warn_lines_summary, warn_lines_boards,
- line, board)
+ line, brd)
else:
if last_func:
AddLine(err_lines_summary, err_lines_boards,
- last_func, board)
+ last_func, brd)
AddLine(err_lines_summary, err_lines_boards,
- line, board)
+ line, brd)
last_was_warning = is_warning
last_func = None
- tconfig = Config(self.config_filenames, board.target)
+ tconfig = Config(self.config_filenames, brd.target)
for fname in self.config_filenames:
if outcome.config:
for key, value in outcome.config[fname].items():
tconfig.Add(fname, key, value)
- config[board.target] = tconfig
+ config[brd.target] = tconfig
- tenvironment = Environment(board.target)
+ tenvironment = Environment(brd.target)
if outcome.environment:
for key, value in outcome.environment.items():
tenvironment.Add(key, value)
- environment[board.target] = tenvironment
+ environment[brd.target] = tenvironment
return (board_dict, err_lines_summary, err_lines_boards,
warn_lines_summary, warn_lines_boards, config, environment)
@@ -971,9 +971,8 @@ class Builder:
board.target
"""
self._base_board_dict = {}
- for board in board_selected:
- self._base_board_dict[board] = Builder.Outcome(0, [], [], {}, {},
- {})
+ for brd in board_selected:
+ self._base_board_dict[brd] = Builder.Outcome(0, [], [], {}, {}, {})
self._base_err_lines = []
self._base_warn_lines = []
self._base_err_line_boards = {}
@@ -1220,10 +1219,10 @@ class Builder:
boards = []
board_set = set()
if self._list_error_boards:
- for board in line_boards[line]:
- if not board in board_set:
- boards.append(board)
- board_set.add(board)
+ for brd in line_boards[line]:
+ if not brd in board_set:
+ boards.append(brd)
+ board_set.add(brd)
return boards
def _CalcErrorDelta(base_lines, base_line_boards, lines, line_boards,
@@ -1328,7 +1327,7 @@ class Builder:
out_list = []
for line in err_lines:
boards = ''
- names = [board.target for board in line.boards]
+ names = [brd.target for brd in line.boards]
board_str = ' '.join(names) if names else ''
if board_str:
out = self.col.build(colour, line.char + '(')
@@ -1549,9 +1548,9 @@ class Builder:
# Get a list of boards that did not get built, if needed
not_built = []
- for board in board_selected:
- if not board in board_dict:
- not_built.append(board)
+ for brd in board_selected:
+ if not brd in board_dict:
+ not_built.append(brd)
if not_built:
tprint("Boards not built (%d): %s" % (len(not_built),
', '.join(not_built)))
@@ -1768,7 +1767,7 @@ class Builder:
# Create jobs to build all commits for each board
for brd in board_selected.values():
job = builderthread.BuilderJob()
- job.board = brd
+ job.brd = brd
job.commits = commits
job.keep_outputs = keep_outputs
job.work_in_output = self.work_in_output
diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py
index 7522ff62de6..6240e08c767 100644
--- a/tools/buildman/builderthread.py
+++ b/tools/buildman/builderthread.py
@@ -40,7 +40,7 @@ class BuilderJob:
"""Holds information about a job to be performed by a thread
Members:
- board: Board object to build
+ brd: Board object to build
commits: List of Commit objects to build
keep_outputs: True to save build output files
step: 1 to process every commit, n to process every nth commit
@@ -48,7 +48,7 @@ class BuilderJob:
don't write to a separate output directory.
"""
def __init__(self):
- self.board = None
+ self.brd = None
self.commits = []
self.keep_outputs = False
self.step = 1
@@ -491,7 +491,7 @@ class BuilderThread(threading.Thread):
Returns:
List of Result objects
"""
- brd = job.board
+ brd = job.brd
work_dir = self.builder.GetThreadDir(self.thread_num)
self.toolchain = None
if job.commits:
diff --git a/tools/buildman/func_test.py b/tools/buildman/func_test.py
index fbf6706644b..0265286908d 100644
--- a/tools/buildman/func_test.py
+++ b/tools/buildman/func_test.py
@@ -476,12 +476,12 @@ class TestFunctional(unittest.TestCase):
self.assertEqual(ret_code, 100)
for commit in range(self._commits):
- for board in self._boards.GetList():
- if board.arch != 'sandbox':
- errfile = self._builder.GetErrFile(commit, board.target)
+ for brd in self._boards.GetList():
+ if brd.arch != 'sandbox':
+ errfile = self._builder.GetErrFile(commit, brd.target)
fd = open(errfile)
self.assertEqual(fd.readlines(),
- ['No tool chain for %s\n' % board.arch])
+ ['No tool chain for %s\n' % brd.arch])
fd.close()
def testBranch(self):
diff --git a/tools/buildman/test.py b/tools/buildman/test.py
index 27287438eea..62ad25db397 100644
--- a/tools/buildman/test.py
+++ b/tools/buildman/test.py
@@ -184,8 +184,8 @@ class TestBuild(unittest.TestCase):
# TODO(sjg@chromium.org): If plus is '', we shouldn't need this
expect += ' ' + col.build(expected_colour, plus)
expect += ' '
- for board in boards:
- expect += col.build(expected_colour, ' %s' % board)
+ for brd in boards:
+ expect += col.build(expected_colour, ' %s' % brd)
self.assertEqual(text, expect)
def _SetupTest(self, echo_lines=False, threads=1, **kwdisplay_args):
diff --git a/tools/buildman/toolchain.py b/tools/buildman/toolchain.py
index 46a4e5ed409..fea40ba2151 100644
--- a/tools/buildman/toolchain.py
+++ b/tools/buildman/toolchain.py
@@ -441,7 +441,7 @@ class Toolchains:
args = args[:m.start(0)] + value + args[m.end(0):]
return args
- def GetMakeArguments(self, board):
+ def GetMakeArguments(self, brd):
"""Returns 'make' arguments for a given board
The flags are in a section called 'make-flags'. Flags are named
@@ -462,13 +462,13 @@ class Toolchains:
A special 'target' variable is set to the board target.
Args:
- board: Board object for the board to check.
+ brd: Board object for the board to check.
Returns:
'make' flags for that board, or '' if none
"""
- self._make_flags['target'] = board.target
+ self._make_flags['target'] = brd.target
arg_str = self.ResolveReferences(self._make_flags,
- self._make_flags.get(board.target, ''))
+ self._make_flags.get(brd.target, ''))
args = re.findall("(?:\".*?\"|\S)+", arg_str)
i = 0
while i < len(args):
--
2.37.0.144.g8ac04bfd2-goog
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 04/19] buildman: Fix use of 'boards' in func_test
2022-07-12 1:03 [PATCH 00/19] buildman: Integration of boards.cfg file Simon Glass
` (2 preceding siblings ...)
2022-07-12 1:03 ` [PATCH 03/19] buildman: Avoid using board as a variable Simon Glass
@ 2022-07-12 1:03 ` Simon Glass
2022-07-12 1:03 ` [PATCH 05/19] buildman: Fix use of 'boards' in test Simon Glass
` (15 subsequent siblings)
19 siblings, 0 replies; 31+ messages in thread
From: Simon Glass @ 2022-07-12 1:03 UTC (permalink / raw)
To: U-Boot Mailing List; +Cc: Tom Rini, Simon Glass
We want to create a module called 'boards' so avoid use of this variable
name in this module. Change the global to be capitalised, as required by
Python style.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
tools/buildman/func_test.py | 38 ++++++++++++++++++-------------------
1 file changed, 19 insertions(+), 19 deletions(-)
diff --git a/tools/buildman/func_test.py b/tools/buildman/func_test.py
index 0265286908d..812dc924c8d 100644
--- a/tools/buildman/func_test.py
+++ b/tools/buildman/func_test.py
@@ -35,7 +35,7 @@ chromeos_daisy=VBOOT=${chroot}/build/daisy/usr ${vboot}
chromeos_peach=VBOOT=${chroot}/build/peach_pit/usr ${vboot}
'''
-boards = [
+BOARDS = [
['Active', 'arm', 'armv7', '', 'Tester', 'ARM Board 1', 'board0', ''],
['Active', 'arm', 'armv7', '', 'Tester', 'ARM Board 2', 'board1', ''],
['Active', 'powerpc', 'powerpc', '', 'Tester', 'PowerPC board 1', 'board2', ''],
@@ -188,13 +188,13 @@ class TestFunctional(unittest.TestCase):
self._toolchains.Add('arm-gcc', test=False)
self._toolchains.Add('powerpc-gcc', test=False)
self._boards = board.Boards()
- for brd in boards:
+ for brd in BOARDS:
self._boards.AddBoard(board.Board(*brd))
# Directories where the source been cloned
self._clone_dirs = []
self._commits = len(commit_shortlog.splitlines()) + 1
- self._total_builds = self._commits * len(boards)
+ self._total_builds = self._commits * len(BOARDS)
# Number of calls to make
self._make_calls = 0
@@ -220,13 +220,13 @@ class TestFunctional(unittest.TestCase):
return command.run_pipe([[self._buildman_pathname] + list(args)],
capture=True, capture_stderr=True)
- def _RunControl(self, *args, boards=None, clean_dir=False,
+ def _RunControl(self, *args, brds=None, clean_dir=False,
test_thread_exceptions=False):
"""Run buildman
Args:
args: List of arguments to pass
- boards:
+ brds: Boards object
clean_dir: Used for tests only, indicates that the existing output_dir
should be removed before starting the build
test_thread_exceptions: Uses for tests only, True to make the threads
@@ -239,7 +239,7 @@ class TestFunctional(unittest.TestCase):
sys.argv = [sys.argv[0]] + list(args)
options, args = cmdline.ParseArgs()
result = control.DoBuildman(options, args, toolchains=self._toolchains,
- make_func=self._HandleMake, boards=boards or self._boards,
+ make_func=self._HandleMake, boards=brds or self._boards,
clean_dir=clean_dir,
test_thread_exceptions=test_thread_exceptions)
self._builder = control.builder
@@ -451,7 +451,7 @@ class TestFunctional(unittest.TestCase):
self.setupToolchains();
self._RunControl('-o', self._output_dir)
lines = terminal.get_print_test_lines()
- self.assertIn('Building current source for %d boards' % len(boards),
+ self.assertIn('Building current source for %d boards' % len(BOARDS),
lines[0].text)
def testBadBranch(self):
@@ -467,7 +467,7 @@ class TestFunctional(unittest.TestCase):
# Buildman always builds the upstream commit as well
self.assertIn('Building %d commits for %d boards' %
- (self._commits, len(boards)), lines[0].text)
+ (self._commits, len(BOARDS)), lines[0].text)
self.assertEqual(self._builder.count, self._total_builds)
# Only sandbox should succeed, the others don't have toolchains
@@ -493,17 +493,17 @@ class TestFunctional(unittest.TestCase):
def testCount(self):
"""Test building a specific number of commitst"""
self._RunControl('-b', TEST_BRANCH, '-c2', '-o', self._output_dir)
- self.assertEqual(self._builder.count, 2 * len(boards))
+ self.assertEqual(self._builder.count, 2 * len(BOARDS))
self.assertEqual(self._builder.fail, 0)
# Each board has a config, and then one make per commit
- self.assertEqual(self._make_calls, len(boards) * (1 + 2))
+ self.assertEqual(self._make_calls, len(BOARDS) * (1 + 2))
def testIncremental(self):
"""Test building a branch twice - the second time should do nothing"""
self._RunControl('-b', TEST_BRANCH, '-o', self._output_dir)
# Each board has a mrproper, config, and then one make per commit
- self.assertEqual(self._make_calls, len(boards) * (self._commits + 1))
+ self.assertEqual(self._make_calls, len(BOARDS) * (self._commits + 1))
self._make_calls = 0
self._RunControl('-b', TEST_BRANCH, '-o', self._output_dir, clean_dir=False)
self.assertEqual(self._make_calls, 0)
@@ -516,19 +516,19 @@ class TestFunctional(unittest.TestCase):
self._make_calls = 0
self._RunControl('-b', TEST_BRANCH, '-f', '-o', self._output_dir, clean_dir=False)
# Each board has a config and one make per commit
- self.assertEqual(self._make_calls, len(boards) * (self._commits + 1))
+ self.assertEqual(self._make_calls, len(BOARDS) * (self._commits + 1))
def testForceReconfigure(self):
"""The -f flag should force a rebuild"""
self._RunControl('-b', TEST_BRANCH, '-C', '-o', self._output_dir)
# Each commit has a config and make
- self.assertEqual(self._make_calls, len(boards) * self._commits * 2)
+ self.assertEqual(self._make_calls, len(BOARDS) * self._commits * 2)
def testMrproper(self):
"""The -f flag should force a rebuild"""
self._RunControl('-b', TEST_BRANCH, '-m', '-o', self._output_dir)
# Each board has a mkproper, config and then one make per commit
- self.assertEqual(self._make_calls, len(boards) * (self._commits + 2))
+ self.assertEqual(self._make_calls, len(BOARDS) * (self._commits + 2))
def testErrors(self):
"""Test handling of build errors"""
@@ -581,9 +581,9 @@ class TestFunctional(unittest.TestCase):
def testWorkInOutput(self):
"""Test the -w option which should write directly to the output dir"""
board_list = board.Boards()
- board_list.AddBoard(board.Board(*boards[0]))
+ board_list.AddBoard(board.Board(*BOARDS[0]))
self._RunControl('-o', self._output_dir, '-w', clean_dir=False,
- boards=board_list)
+ brds=board_list)
self.assertTrue(
os.path.exists(os.path.join(self._output_dir, 'u-boot')))
self.assertTrue(
@@ -600,14 +600,14 @@ class TestFunctional(unittest.TestCase):
os.path.exists(os.path.join(self._output_dir, 'u-boot')))
board_list = board.Boards()
- board_list.AddBoard(board.Board(*boards[0]))
+ board_list.AddBoard(board.Board(*BOARDS[0]))
with self.assertRaises(SystemExit) as e:
self._RunControl('-b', self._test_branch, '-o', self._output_dir,
- '-w', clean_dir=False, boards=board_list)
+ '-w', clean_dir=False, brds=board_list)
self.assertIn("single commit", str(e.exception))
board_list = board.Boards()
- board_list.AddBoard(board.Board(*boards[0]))
+ board_list.AddBoard(board.Board(*BOARDS[0]))
with self.assertRaises(SystemExit) as e:
self._RunControl('-w', clean_dir=False)
self.assertIn("specify -o", str(e.exception))
--
2.37.0.144.g8ac04bfd2-goog
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 05/19] buildman: Fix use of 'boards' in test
2022-07-12 1:03 [PATCH 00/19] buildman: Integration of boards.cfg file Simon Glass
` (3 preceding siblings ...)
2022-07-12 1:03 ` [PATCH 04/19] buildman: Fix use of 'boards' in func_test Simon Glass
@ 2022-07-12 1:03 ` Simon Glass
2022-07-12 1:04 ` [PATCH 06/19] buildman: Drop use of 'boards' in control Simon Glass
` (14 subsequent siblings)
19 siblings, 0 replies; 31+ messages in thread
From: Simon Glass @ 2022-07-12 1:03 UTC (permalink / raw)
To: U-Boot Mailing List; +Cc: Tom Rini, Simon Glass
We want to create a module called 'boards' so avoid use of this variable
name in this module. Change the global to be capitalised, as required by
Python style.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
tools/buildman/test.py | 42 +++++++++++++++++++++---------------------
1 file changed, 21 insertions(+), 21 deletions(-)
diff --git a/tools/buildman/test.py b/tools/buildman/test.py
index 62ad25db397..6c09cb7fad5 100644
--- a/tools/buildman/test.py
+++ b/tools/buildman/test.py
@@ -94,7 +94,7 @@ commits = [
[errors[4]]],
]
-boards = [
+BOARDS = [
['Active', 'arm', 'armv7', '', 'Tester', 'ARM Board 1', 'board0', ''],
['Active', 'arm', 'armv7', '', 'Tester', 'ARM Board 2', 'board1', ''],
['Active', 'powerpc', 'powerpc', '', 'Tester', 'PowerPC board 1', 'board2', ''],
@@ -131,10 +131,10 @@ class TestBuild(unittest.TestCase):
self.commits.append(comm)
# Set up boards to build
- self.boards = board.Boards()
- for brd in boards:
- self.boards.AddBoard(board.Board(*brd))
- self.boards.SelectBoards([])
+ self.brds = board.Boards()
+ for brd in BOARDS:
+ self.brds.AddBoard(board.Board(*brd))
+ self.brds.SelectBoards([])
# Add some test settings
bsettings.Setup(None)
@@ -176,7 +176,7 @@ class TestBuild(unittest.TestCase):
result.combined = result.stdout + result.stderr
return result
- def assertSummary(self, text, arch, plus, boards, outcome=OUTCOME_ERR):
+ def assertSummary(self, text, arch, plus, brds, outcome=OUTCOME_ERR):
col = self._col
expected_colour = (col.GREEN if outcome == OUTCOME_OK else
col.YELLOW if outcome == OUTCOME_WARN else col.RED)
@@ -184,7 +184,7 @@ class TestBuild(unittest.TestCase):
# TODO(sjg@chromium.org): If plus is '', we shouldn't need this
expect += ' ' + col.build(expected_colour, plus)
expect += ' '
- for brd in boards:
+ for brd in brds:
expect += col.build(expected_colour, ' %s' % brd)
self.assertEqual(text, expect)
@@ -203,7 +203,7 @@ class TestBuild(unittest.TestCase):
build = builder.Builder(self.toolchains, self.base_dir, None, threads,
2, checkout=False, show_unknown=False)
build.do_make = self.Make
- board_selected = self.boards.GetSelectedDict()
+ board_selected = self.brds.GetSelectedDict()
# Build the boards for the pre-defined commits and warnings/errors
# associated with each. This calls our Make() to inject the fake output.
@@ -217,7 +217,7 @@ class TestBuild(unittest.TestCase):
# We should get two starting messages, an update for every commit built
# and a summary message
- self.assertEqual(count, len(commits) * len(boards) + 3)
+ self.assertEqual(count, len(commits) * len(BOARDS) + 3)
build.SetDisplayOptions(**kwdisplay_args);
build.ShowSummary(self.commits, board_selected)
if echo_lines:
@@ -236,7 +236,7 @@ class TestBuild(unittest.TestCase):
filter_dtb_warnings: Adjust the check for output produced with the
--filter-dtb-warnings flag
"""
- def add_line_prefix(prefix, boards, error_str, colour):
+ def add_line_prefix(prefix, brds, error_str, colour):
"""Add a prefix to each line of a string
The training \n in error_str is removed before processing
@@ -253,9 +253,9 @@ class TestBuild(unittest.TestCase):
lines = error_str.strip().splitlines()
new_lines = []
for line in lines:
- if boards:
+ if brds:
expect = self._col.build(colour, prefix + '(')
- expect += self._col.build(self._col.MAGENTA, boards,
+ expect += self._col.build(self._col.MAGENTA, brds,
bright=False)
expect += self._col.build(colour, ') %s' % line)
else:
@@ -468,18 +468,18 @@ class TestBuild(unittest.TestCase):
def testBoardSingle(self):
"""Test single board selection"""
- self.assertEqual(self.boards.SelectBoards(['sandbox']),
+ self.assertEqual(self.brds.SelectBoards(['sandbox']),
({'all': ['board4'], 'sandbox': ['board4']}, []))
def testBoardArch(self):
"""Test single board selection"""
- self.assertEqual(self.boards.SelectBoards(['arm']),
+ self.assertEqual(self.brds.SelectBoards(['arm']),
({'all': ['board0', 'board1'],
'arm': ['board0', 'board1']}, []))
def testBoardArchSingle(self):
"""Test single board selection"""
- self.assertEqual(self.boards.SelectBoards(['arm sandbox']),
+ self.assertEqual(self.brds.SelectBoards(['arm sandbox']),
({'sandbox': ['board4'],
'all': ['board0', 'board1', 'board4'],
'arm': ['board0', 'board1']}, []))
@@ -487,20 +487,20 @@ class TestBuild(unittest.TestCase):
def testBoardArchSingleMultiWord(self):
"""Test single board selection"""
- self.assertEqual(self.boards.SelectBoards(['arm', 'sandbox']),
+ self.assertEqual(self.brds.SelectBoards(['arm', 'sandbox']),
({'sandbox': ['board4'],
'all': ['board0', 'board1', 'board4'],
'arm': ['board0', 'board1']}, []))
def testBoardSingleAnd(self):
"""Test single board selection"""
- self.assertEqual(self.boards.SelectBoards(['Tester & arm']),
+ self.assertEqual(self.brds.SelectBoards(['Tester & arm']),
({'Tester&arm': ['board0', 'board1'],
'all': ['board0', 'board1']}, []))
def testBoardTwoAnd(self):
"""Test single board selection"""
- self.assertEqual(self.boards.SelectBoards(['Tester', '&', 'arm',
+ self.assertEqual(self.brds.SelectBoards(['Tester', '&', 'arm',
'Tester' '&', 'powerpc',
'sandbox']),
({'sandbox': ['board4'],
@@ -511,19 +511,19 @@ class TestBuild(unittest.TestCase):
def testBoardAll(self):
"""Test single board selection"""
- self.assertEqual(self.boards.SelectBoards([]),
+ self.assertEqual(self.brds.SelectBoards([]),
({'all': ['board0', 'board1', 'board2', 'board3',
'board4']}, []))
def testBoardRegularExpression(self):
"""Test single board selection"""
- self.assertEqual(self.boards.SelectBoards(['T.*r&^Po']),
+ self.assertEqual(self.brds.SelectBoards(['T.*r&^Po']),
({'all': ['board2', 'board3'],
'T.*r&^Po': ['board2', 'board3']}, []))
def testBoardDuplicate(self):
"""Test single board selection"""
- self.assertEqual(self.boards.SelectBoards(['sandbox sandbox',
+ self.assertEqual(self.brds.SelectBoards(['sandbox sandbox',
'sandbox']),
({'all': ['board4'], 'sandbox': ['board4']}, []))
def CheckDirs(self, build, dirname):
--
2.37.0.144.g8ac04bfd2-goog
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 06/19] buildman: Drop use of 'boards' in control
2022-07-12 1:03 [PATCH 00/19] buildman: Integration of boards.cfg file Simon Glass
` (4 preceding siblings ...)
2022-07-12 1:03 ` [PATCH 05/19] buildman: Fix use of 'boards' in test Simon Glass
@ 2022-07-12 1:04 ` Simon Glass
2022-07-12 1:04 ` [PATCH 07/19] buildman: Drop use of 'board' in board module Simon Glass
` (13 subsequent siblings)
19 siblings, 0 replies; 31+ messages in thread
From: Simon Glass @ 2022-07-12 1:04 UTC (permalink / raw)
To: U-Boot Mailing List; +Cc: Tom Rini, Simon Glass
Use brds instead so that we can reserve 'boards' for a module name.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
tools/buildman/builder.py | 11 +++++------
tools/buildman/control.py | 26 +++++++++++++-------------
tools/buildman/func_test.py | 2 +-
3 files changed, 19 insertions(+), 20 deletions(-)
diff --git a/tools/buildman/builder.py b/tools/buildman/builder.py
index 462266b4b84..33f9373b8f8 100644
--- a/tools/buildman/builder.py
+++ b/tools/buildman/builder.py
@@ -108,7 +108,7 @@ u-boot/ source directory
boards: List of Board objects which have line in the error/warning output
errline: The text of the error line
"""
-ErrLine = collections.namedtuple('ErrLine', 'char,boards,errline')
+ErrLine = collections.namedtuple('ErrLine', 'char,brds,errline')
# Possible build outcomes
OUTCOME_OK, OUTCOME_WARNING, OUTCOME_ERROR, OUTCOME_UNKNOWN = list(range(4))
@@ -1216,14 +1216,14 @@ class Builder:
List of boards with that error line, or [] if the user has not
requested such a list
"""
- boards = []
+ brds = []
board_set = set()
if self._list_error_boards:
for brd in line_boards[line]:
if not brd in board_set:
- boards.append(brd)
+ brds.append(brd)
board_set.add(brd)
- return boards
+ return brds
def _CalcErrorDelta(base_lines, base_line_boards, lines, line_boards,
char):
@@ -1326,8 +1326,7 @@ class Builder:
if err_lines:
out_list = []
for line in err_lines:
- boards = ''
- names = [brd.target for brd in line.boards]
+ names = [brd.target for brd in line.brds]
board_str = ' '.join(names) if names else ''
if board_str:
out = self.col.build(colour, line.char + '(')
diff --git a/tools/buildman/control.py b/tools/buildman/control.py
index 68dca97413f..c4dfc2af571 100644
--- a/tools/buildman/control.py
+++ b/tools/buildman/control.py
@@ -87,7 +87,7 @@ def ShowActions(series, why_selected, boards_selected, builder, options,
for warning in board_warnings:
print(col.build(col.YELLOW, warning))
-def ShowToolchainPrefix(boards, toolchains):
+def ShowToolchainPrefix(brds, toolchains):
"""Show information about a the tool chain used by one or more boards
The function checks that all boards use the same toolchain, then prints
@@ -100,9 +100,9 @@ def ShowToolchainPrefix(boards, toolchains):
Return:
None on success, string error message otherwise
"""
- boards = boards.GetSelectedDict()
+ board_selected = brds.GetSelectedDict()
tc_set = set()
- for brd in boards.values():
+ for brd in board_selected.values():
tc_set.add(toolchains.Select(brd.arch))
if len(tc_set) != 1:
return 'Supplied boards must share one toolchain'
@@ -111,7 +111,7 @@ def ShowToolchainPrefix(boards, toolchains):
print(tc.GetEnvArgs(toolchain.VAR_CROSS_COMPILE))
return None
-def DoBuildman(options, args, toolchains=None, make_func=None, boards=None,
+def DoBuildman(options, args, toolchains=None, make_func=None, brds=None,
clean_dir=False, test_thread_exceptions=False):
"""The main control code for buildman
@@ -124,7 +124,7 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None,
to execute 'make'. If this is None, the normal function
will be used, which calls the 'make' tool with suitable
arguments. This setting is useful for tests.
- board: Boards() object to use, containing a list of available
+ brds: Boards() object to use, containing a list of available
boards. If this is None it will be created and scanned.
clean_dir: Used for tests only, indicates that the existing output_dir
should be removed before starting the build
@@ -182,7 +182,7 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None,
options.output_dir = '..'
# Work out what subset of the boards we are building
- if not boards:
+ if not brds:
if not os.path.exists(options.output_dir):
os.makedirs(options.output_dir)
board_file = os.path.join(options.output_dir, 'boards.cfg')
@@ -197,8 +197,8 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None,
if status != 0:
sys.exit("Failed to generate boards.cfg")
- boards = board.Boards()
- boards.ReadBoards(board_file)
+ brds = board.Boards()
+ brds.ReadBoards(board_file)
exclude = []
if options.exclude:
@@ -211,14 +211,14 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None,
requested_boards += b.split(',')
else:
requested_boards = None
- why_selected, board_warnings = boards.SelectBoards(args, exclude,
- requested_boards)
- selected = boards.GetSelected()
+ why_selected, board_warnings = brds.SelectBoards(args, exclude,
+ requested_boards)
+ selected = brds.GetSelected()
if not len(selected):
sys.exit(col.build(col.RED, 'No matching boards found'))
if options.print_prefix:
- err = ShowToolchainPrefix(boards, toolchains)
+ err = ShowToolchainPrefix(brds, toolchains)
if err:
sys.exit(col.build(col.RED, err))
return 0
@@ -349,7 +349,7 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None,
builder.in_tree = options.in_tree
# Work out which boards to build
- board_selected = boards.GetSelectedDict()
+ board_selected = brds.GetSelectedDict()
if series:
commits = series.commits
diff --git a/tools/buildman/func_test.py b/tools/buildman/func_test.py
index 812dc924c8d..23627f3b0f2 100644
--- a/tools/buildman/func_test.py
+++ b/tools/buildman/func_test.py
@@ -239,7 +239,7 @@ class TestFunctional(unittest.TestCase):
sys.argv = [sys.argv[0]] + list(args)
options, args = cmdline.ParseArgs()
result = control.DoBuildman(options, args, toolchains=self._toolchains,
- make_func=self._HandleMake, boards=brds or self._boards,
+ make_func=self._HandleMake, brds=brds or self._boards,
clean_dir=clean_dir,
test_thread_exceptions=test_thread_exceptions)
self._builder = control.builder
--
2.37.0.144.g8ac04bfd2-goog
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 07/19] buildman: Drop use of 'board' in board module
2022-07-12 1:03 [PATCH 00/19] buildman: Integration of boards.cfg file Simon Glass
` (5 preceding siblings ...)
2022-07-12 1:04 ` [PATCH 06/19] buildman: Drop use of 'boards' in control Simon Glass
@ 2022-07-12 1:04 ` Simon Glass
2022-07-12 1:04 ` [PATCH 08/19] buildman: Convert camel case in board.py Simon Glass
` (12 subsequent siblings)
19 siblings, 0 replies; 31+ messages in thread
From: Simon Glass @ 2022-07-12 1:04 UTC (permalink / raw)
To: U-Boot Mailing List; +Cc: Tom Rini, Simon Glass
Use brds instead so that we can reserve 'boards' and 'board' as module
names.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
tools/buildman/board.py | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/tools/buildman/board.py b/tools/buildman/board.py
index 08771b3f158..8de71e487ee 100644
--- a/tools/buildman/board.py
+++ b/tools/buildman/board.py
@@ -116,7 +116,7 @@ class Boards:
def ReadBoards(self, fname):
"""Read a list of boards from a board file.
- Create a board object for each and add it to our _boards list.
+ Create a Board object for each and add it to our _boards list.
Args:
fname: Filename of boards.cfg file
@@ -238,21 +238,21 @@ class Boards:
terms.append(term)
return terms
- def SelectBoards(self, args, exclude=[], boards=None):
+ def SelectBoards(self, args, exclude=[], brds=None):
"""Mark boards selected based on args
Normally either boards (an explicit list of boards) or args (a list of
terms to match against) is used. It is possible to specify both, in
which case they are additive.
- If boards and args are both empty, all boards are selected.
+ If brds and args are both empty, all boards are selected.
Args:
args: List of strings specifying boards to include, either named,
or by their target, architecture, cpu, vendor or soc. If
empty, all boards are selected.
exclude: List of boards to exclude, regardless of 'args'
- boards: List of boards to build
+ brds: List of boards to build
Returns:
Tuple
@@ -283,8 +283,8 @@ class Boards:
matching_term = str(term)
build_it = True
break
- elif boards:
- if brd.target in boards:
+ elif brds:
+ if brd.target in brds:
build_it = True
found.append(brd.target)
else:
@@ -302,8 +302,8 @@ class Boards:
result[matching_term].append(brd.target)
result['all'].append(brd.target)
- if boards:
- remaining = set(boards) - set(found)
+ if brds:
+ remaining = set(brds) - set(found)
if remaining:
warnings.append('Boards not found: %s\n' % ', '.join(remaining))
--
2.37.0.144.g8ac04bfd2-goog
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 08/19] buildman: Convert camel case in board.py
2022-07-12 1:03 [PATCH 00/19] buildman: Integration of boards.cfg file Simon Glass
` (6 preceding siblings ...)
2022-07-12 1:04 ` [PATCH 07/19] buildman: Drop use of 'board' in board module Simon Glass
@ 2022-07-12 1:04 ` Simon Glass
2022-07-12 1:04 ` [PATCH 09/19] buildman: Split out Boards into its own file Simon Glass
` (11 subsequent siblings)
19 siblings, 0 replies; 31+ messages in thread
From: Simon Glass @ 2022-07-12 1:04 UTC (permalink / raw)
To: U-Boot Mailing List; +Cc: Tom Rini, Simon Glass
Convert this file to snake case and update all files which use it.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
tools/buildman/board.py | 38 ++++++++++++++++++-------------------
tools/buildman/control.py | 12 ++++++------
tools/buildman/func_test.py | 10 +++++-----
tools/buildman/test.py | 24 +++++++++++------------
4 files changed, 42 insertions(+), 42 deletions(-)
diff --git a/tools/buildman/board.py b/tools/buildman/board.py
index 8de71e487ee..ebb9d6f67dc 100644
--- a/tools/buildman/board.py
+++ b/tools/buildman/board.py
@@ -16,7 +16,7 @@ class Expr:
self._expr = expr
self._re = re.compile(expr)
- def Matches(self, props):
+ def matches(self, props):
"""Check if any of the properties match the regular expression.
Args:
@@ -42,7 +42,7 @@ class Term:
self._expr_list = []
self._board_count = 0
- def AddExpr(self, expr):
+ def add_expr(self, expr):
"""Add an Expr object to the list to check.
Args:
@@ -55,7 +55,7 @@ class Term:
"""Return some sort of useful string describing the term"""
return '&'.join([str(expr) for expr in self._expr_list])
- def Matches(self, props):
+ def matches(self, props):
"""Check if any of the properties match this term
Each of the expressions in the term is checked. All must match.
@@ -66,7 +66,7 @@ class Term:
True if all of the expressions in the Term match, else False
"""
for expr in self._expr_list:
- if not expr.Matches(props):
+ if not expr.matches(props):
return False
return True
@@ -103,7 +103,7 @@ class Boards:
# Use a simple list here, sinc OrderedDict requires Python 2.7
self._boards = []
- def AddBoard(self, brd):
+ def add_board(self, brd):
"""Add a new board to the list.
The board's target member must not already exist in the board list.
@@ -113,7 +113,7 @@ class Boards:
"""
self._boards.append(brd)
- def ReadBoards(self, fname):
+ def read_boards(self, fname):
"""Read a list of boards from a board file.
Create a Board object for each and add it to our _boards list.
@@ -137,10 +137,10 @@ class Boards:
fields = fields[:8]
brd = Board(*fields)
- self.AddBoard(brd)
+ self.add_board(brd)
- def GetList(self):
+ def get_list(self):
"""Return a list of available boards.
Returns:
@@ -148,7 +148,7 @@ class Boards:
"""
return self._boards
- def GetDict(self):
+ def get_dict(self):
"""Build a dictionary containing all the boards.
Returns:
@@ -161,7 +161,7 @@ class Boards:
board_dict[brd.target] = brd
return board_dict
- def GetSelectedDict(self):
+ def get_selected_dict(self):
"""Return a dictionary containing the selected boards
Returns:
@@ -173,7 +173,7 @@ class Boards:
board_dict[brd.target] = brd
return board_dict
- def GetSelected(self):
+ def get_selected(self):
"""Return a list of selected boards
Returns:
@@ -181,7 +181,7 @@ class Boards:
"""
return [brd for brd in self._boards if brd.build_it]
- def GetSelectedNames(self):
+ def get_selected_names(self):
"""Return a list of selected boards
Returns:
@@ -189,7 +189,7 @@ class Boards:
"""
return [brd.target for brd in self._boards if brd.build_it]
- def _BuildTerms(self, args):
+ def _build_terms(self, args):
"""Convert command line arguments to a list of terms.
This deals with parsing of the arguments. It handles the '&'
@@ -227,18 +227,18 @@ class Boards:
if sym == '&':
oper = sym
elif oper:
- term.AddExpr(sym)
+ term.add_expr(sym)
oper = None
else:
if term:
terms.append(term)
term = Term()
- term.AddExpr(sym)
+ term.add_expr(sym)
if term:
terms.append(term)
return terms
- def SelectBoards(self, args, exclude=[], brds=None):
+ def select_boards(self, args, exclude=[], brds=None):
"""Mark boards selected based on args
Normally either boards (an explicit list of boards) or args (a list of
@@ -262,7 +262,7 @@ class Boards:
"""
result = OrderedDict()
warnings = []
- terms = self._BuildTerms(args)
+ terms = self._build_terms(args)
result['all'] = []
for term in terms:
@@ -279,7 +279,7 @@ class Boards:
if terms:
match = False
for term in terms:
- if term.Matches(brd.props):
+ if term.matches(brd.props):
matching_term = str(term)
build_it = True
break
@@ -292,7 +292,7 @@ class Boards:
# Check that it is not specifically excluded
for expr in exclude_list:
- if expr.Matches(brd.props):
+ if expr.matches(brd.props):
build_it = False
break
diff --git a/tools/buildman/control.py b/tools/buildman/control.py
index c4dfc2af571..a5c1c2e51c6 100644
--- a/tools/buildman/control.py
+++ b/tools/buildman/control.py
@@ -100,7 +100,7 @@ def ShowToolchainPrefix(brds, toolchains):
Return:
None on success, string error message otherwise
"""
- board_selected = brds.GetSelectedDict()
+ board_selected = brds.get_selected_dict()
tc_set = set()
for brd in board_selected.values():
tc_set.add(toolchains.Select(brd.arch))
@@ -198,7 +198,7 @@ def DoBuildman(options, args, toolchains=None, make_func=None, brds=None,
sys.exit("Failed to generate boards.cfg")
brds = board.Boards()
- brds.ReadBoards(board_file)
+ brds.read_boards(board_file)
exclude = []
if options.exclude:
@@ -211,9 +211,9 @@ def DoBuildman(options, args, toolchains=None, make_func=None, brds=None,
requested_boards += b.split(',')
else:
requested_boards = None
- why_selected, board_warnings = brds.SelectBoards(args, exclude,
- requested_boards)
- selected = brds.GetSelected()
+ why_selected, board_warnings = brds.select_boards(args, exclude,
+ requested_boards)
+ selected = brds.get_selected()
if not len(selected):
sys.exit(col.build(col.RED, 'No matching boards found'))
@@ -349,7 +349,7 @@ def DoBuildman(options, args, toolchains=None, make_func=None, brds=None,
builder.in_tree = options.in_tree
# Work out which boards to build
- board_selected = brds.GetSelectedDict()
+ board_selected = brds.get_selected_dict()
if series:
commits = series.commits
diff --git a/tools/buildman/func_test.py b/tools/buildman/func_test.py
index 23627f3b0f2..6806ea7fbe6 100644
--- a/tools/buildman/func_test.py
+++ b/tools/buildman/func_test.py
@@ -189,7 +189,7 @@ class TestFunctional(unittest.TestCase):
self._toolchains.Add('powerpc-gcc', test=False)
self._boards = board.Boards()
for brd in BOARDS:
- self._boards.AddBoard(board.Board(*brd))
+ self._boards.add_board(board.Board(*brd))
# Directories where the source been cloned
self._clone_dirs = []
@@ -476,7 +476,7 @@ class TestFunctional(unittest.TestCase):
self.assertEqual(ret_code, 100)
for commit in range(self._commits):
- for brd in self._boards.GetList():
+ for brd in self._boards.get_list():
if brd.arch != 'sandbox':
errfile = self._builder.GetErrFile(commit, brd.target)
fd = open(errfile)
@@ -581,7 +581,7 @@ class TestFunctional(unittest.TestCase):
def testWorkInOutput(self):
"""Test the -w option which should write directly to the output dir"""
board_list = board.Boards()
- board_list.AddBoard(board.Board(*BOARDS[0]))
+ board_list.add_board(board.Board(*BOARDS[0]))
self._RunControl('-o', self._output_dir, '-w', clean_dir=False,
brds=board_list)
self.assertTrue(
@@ -600,14 +600,14 @@ class TestFunctional(unittest.TestCase):
os.path.exists(os.path.join(self._output_dir, 'u-boot')))
board_list = board.Boards()
- board_list.AddBoard(board.Board(*BOARDS[0]))
+ board_list.add_board(board.Board(*BOARDS[0]))
with self.assertRaises(SystemExit) as e:
self._RunControl('-b', self._test_branch, '-o', self._output_dir,
'-w', clean_dir=False, brds=board_list)
self.assertIn("single commit", str(e.exception))
board_list = board.Boards()
- board_list.AddBoard(board.Board(*BOARDS[0]))
+ board_list.add_board(board.Board(*BOARDS[0]))
with self.assertRaises(SystemExit) as e:
self._RunControl('-w', clean_dir=False)
self.assertIn("specify -o", str(e.exception))
diff --git a/tools/buildman/test.py b/tools/buildman/test.py
index 6c09cb7fad5..d7306fb4dfa 100644
--- a/tools/buildman/test.py
+++ b/tools/buildman/test.py
@@ -133,8 +133,8 @@ class TestBuild(unittest.TestCase):
# Set up boards to build
self.brds = board.Boards()
for brd in BOARDS:
- self.brds.AddBoard(board.Board(*brd))
- self.brds.SelectBoards([])
+ self.brds.add_board(board.Board(*brd))
+ self.brds.select_boards([])
# Add some test settings
bsettings.Setup(None)
@@ -203,7 +203,7 @@ class TestBuild(unittest.TestCase):
build = builder.Builder(self.toolchains, self.base_dir, None, threads,
2, checkout=False, show_unknown=False)
build.do_make = self.Make
- board_selected = self.brds.GetSelectedDict()
+ board_selected = self.brds.get_selected_dict()
# Build the boards for the pre-defined commits and warnings/errors
# associated with each. This calls our Make() to inject the fake output.
@@ -468,18 +468,18 @@ class TestBuild(unittest.TestCase):
def testBoardSingle(self):
"""Test single board selection"""
- self.assertEqual(self.brds.SelectBoards(['sandbox']),
+ self.assertEqual(self.brds.select_boards(['sandbox']),
({'all': ['board4'], 'sandbox': ['board4']}, []))
def testBoardArch(self):
"""Test single board selection"""
- self.assertEqual(self.brds.SelectBoards(['arm']),
+ self.assertEqual(self.brds.select_boards(['arm']),
({'all': ['board0', 'board1'],
'arm': ['board0', 'board1']}, []))
def testBoardArchSingle(self):
"""Test single board selection"""
- self.assertEqual(self.brds.SelectBoards(['arm sandbox']),
+ self.assertEqual(self.brds.select_boards(['arm sandbox']),
({'sandbox': ['board4'],
'all': ['board0', 'board1', 'board4'],
'arm': ['board0', 'board1']}, []))
@@ -487,20 +487,20 @@ class TestBuild(unittest.TestCase):
def testBoardArchSingleMultiWord(self):
"""Test single board selection"""
- self.assertEqual(self.brds.SelectBoards(['arm', 'sandbox']),
+ self.assertEqual(self.brds.select_boards(['arm', 'sandbox']),
({'sandbox': ['board4'],
'all': ['board0', 'board1', 'board4'],
'arm': ['board0', 'board1']}, []))
def testBoardSingleAnd(self):
"""Test single board selection"""
- self.assertEqual(self.brds.SelectBoards(['Tester & arm']),
+ self.assertEqual(self.brds.select_boards(['Tester & arm']),
({'Tester&arm': ['board0', 'board1'],
'all': ['board0', 'board1']}, []))
def testBoardTwoAnd(self):
"""Test single board selection"""
- self.assertEqual(self.brds.SelectBoards(['Tester', '&', 'arm',
+ self.assertEqual(self.brds.select_boards(['Tester', '&', 'arm',
'Tester' '&', 'powerpc',
'sandbox']),
({'sandbox': ['board4'],
@@ -511,19 +511,19 @@ class TestBuild(unittest.TestCase):
def testBoardAll(self):
"""Test single board selection"""
- self.assertEqual(self.brds.SelectBoards([]),
+ self.assertEqual(self.brds.select_boards([]),
({'all': ['board0', 'board1', 'board2', 'board3',
'board4']}, []))
def testBoardRegularExpression(self):
"""Test single board selection"""
- self.assertEqual(self.brds.SelectBoards(['T.*r&^Po']),
+ self.assertEqual(self.brds.select_boards(['T.*r&^Po']),
({'all': ['board2', 'board3'],
'T.*r&^Po': ['board2', 'board3']}, []))
def testBoardDuplicate(self):
"""Test single board selection"""
- self.assertEqual(self.brds.SelectBoards(['sandbox sandbox',
+ self.assertEqual(self.brds.select_boards(['sandbox sandbox',
'sandbox']),
({'all': ['board4'], 'sandbox': ['board4']}, []))
def CheckDirs(self, build, dirname):
--
2.37.0.144.g8ac04bfd2-goog
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 09/19] buildman: Split out Boards into its own file
2022-07-12 1:03 [PATCH 00/19] buildman: Integration of boards.cfg file Simon Glass
` (7 preceding siblings ...)
2022-07-12 1:04 ` [PATCH 08/19] buildman: Convert camel case in board.py Simon Glass
@ 2022-07-12 1:04 ` Simon Glass
2022-07-12 1:04 ` [PATCH 10/19] buildman: Incorporate the genboardscfg.py tool Simon Glass
` (10 subsequent siblings)
19 siblings, 0 replies; 31+ messages in thread
From: Simon Glass @ 2022-07-12 1:04 UTC (permalink / raw)
To: U-Boot Mailing List; +Cc: Tom Rini, Simon Glass
Use a separate file for the Boards class so that its name matches the
module name.
Fix up the function names to match the pylint style and fix some other
warnings.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
tools/buildman/board.py | 281 +---------------------------------
tools/buildman/boards.py | 290 ++++++++++++++++++++++++++++++++++++
tools/buildman/control.py | 4 +-
tools/buildman/func_test.py | 11 +-
tools/buildman/test.py | 3 +-
5 files changed, 301 insertions(+), 288 deletions(-)
create mode 100644 tools/buildman/boards.py
diff --git a/tools/buildman/board.py b/tools/buildman/board.py
index ebb9d6f67dc..3268b39e356 100644
--- a/tools/buildman/board.py
+++ b/tools/buildman/board.py
@@ -1,74 +1,8 @@
# SPDX-License-Identifier: GPL-2.0+
# Copyright (c) 2012 The Chromium OS Authors.
-from collections import OrderedDict
-import re
-class Expr:
- """A single regular expression for matching boards to build"""
-
- def __init__(self, expr):
- """Set up a new Expr object.
-
- Args:
- expr: String cotaining regular expression to store
- """
- self._expr = expr
- self._re = re.compile(expr)
-
- def matches(self, props):
- """Check if any of the properties match the regular expression.
-
- Args:
- props: List of properties to check
- Returns:
- True if any of the properties match the regular expression
- """
- for prop in props:
- if self._re.match(prop):
- return True
- return False
-
- def __str__(self):
- return self._expr
-
-class Term:
- """A list of expressions each of which must match with properties.
-
- This provides a list of 'AND' expressions, meaning that each must
- match the board properties for that board to be built.
- """
- def __init__(self):
- self._expr_list = []
- self._board_count = 0
-
- def add_expr(self, expr):
- """Add an Expr object to the list to check.
-
- Args:
- expr: New Expr object to add to the list of those that must
- match for a board to be built.
- """
- self._expr_list.append(Expr(expr))
-
- def __str__(self):
- """Return some sort of useful string describing the term"""
- return '&'.join([str(expr) for expr in self._expr_list])
-
- def matches(self, props):
- """Check if any of the properties match this term
-
- Each of the expressions in the term is checked. All must match.
-
- Args:
- props: List of properties to check
- Returns:
- True if all of the expressions in the Term match, else False
- """
- for expr in self._expr_list:
- if not expr.matches(props):
- return False
- return True
+"""A single board which can be selected and built"""
class Board:
"""A particular board that we can build"""
@@ -95,216 +29,3 @@ class Board:
self.props = [self.target, self.arch, self.cpu, self.board_name,
self.vendor, self.soc, self.options]
self.build_it = False
-
-
-class Boards:
- """Manage a list of boards."""
- def __init__(self):
- # Use a simple list here, sinc OrderedDict requires Python 2.7
- self._boards = []
-
- def add_board(self, brd):
- """Add a new board to the list.
-
- The board's target member must not already exist in the board list.
-
- Args:
- brd: board to add
- """
- self._boards.append(brd)
-
- def read_boards(self, fname):
- """Read a list of boards from a board file.
-
- Create a Board object for each and add it to our _boards list.
-
- Args:
- fname: Filename of boards.cfg file
- """
- with open(fname, 'r', encoding='utf-8') as fd:
- for line in fd:
- if line[0] == '#':
- continue
- fields = line.split()
- if not fields:
- continue
- for upto in range(len(fields)):
- if fields[upto] == '-':
- fields[upto] = ''
- while len(fields) < 8:
- fields.append('')
- if len(fields) > 8:
- fields = fields[:8]
-
- brd = Board(*fields)
- self.add_board(brd)
-
-
- def get_list(self):
- """Return a list of available boards.
-
- Returns:
- List of Board objects
- """
- return self._boards
-
- def get_dict(self):
- """Build a dictionary containing all the boards.
-
- Returns:
- Dictionary:
- key is board.target
- value is board
- """
- board_dict = OrderedDict()
- for brd in self._boards:
- board_dict[brd.target] = brd
- return board_dict
-
- def get_selected_dict(self):
- """Return a dictionary containing the selected boards
-
- Returns:
- List of Board objects that are marked selected
- """
- board_dict = OrderedDict()
- for brd in self._boards:
- if brd.build_it:
- board_dict[brd.target] = brd
- return board_dict
-
- def get_selected(self):
- """Return a list of selected boards
-
- Returns:
- List of Board objects that are marked selected
- """
- return [brd for brd in self._boards if brd.build_it]
-
- def get_selected_names(self):
- """Return a list of selected boards
-
- Returns:
- List of board names that are marked selected
- """
- return [brd.target for brd in self._boards if brd.build_it]
-
- def _build_terms(self, args):
- """Convert command line arguments to a list of terms.
-
- This deals with parsing of the arguments. It handles the '&'
- operator, which joins several expressions into a single Term.
-
- For example:
- ['arm & freescale sandbox', 'tegra']
-
- will produce 3 Terms containing expressions as follows:
- arm, freescale
- sandbox
- tegra
-
- The first Term has two expressions, both of which must match for
- a board to be selected.
-
- Args:
- args: List of command line arguments
- Returns:
- A list of Term objects
- """
- syms = []
- for arg in args:
- for word in arg.split():
- sym_build = []
- for term in word.split('&'):
- if term:
- sym_build.append(term)
- sym_build.append('&')
- syms += sym_build[:-1]
- terms = []
- term = None
- oper = None
- for sym in syms:
- if sym == '&':
- oper = sym
- elif oper:
- term.add_expr(sym)
- oper = None
- else:
- if term:
- terms.append(term)
- term = Term()
- term.add_expr(sym)
- if term:
- terms.append(term)
- return terms
-
- def select_boards(self, args, exclude=[], brds=None):
- """Mark boards selected based on args
-
- Normally either boards (an explicit list of boards) or args (a list of
- terms to match against) is used. It is possible to specify both, in
- which case they are additive.
-
- If brds and args are both empty, all boards are selected.
-
- Args:
- args: List of strings specifying boards to include, either named,
- or by their target, architecture, cpu, vendor or soc. If
- empty, all boards are selected.
- exclude: List of boards to exclude, regardless of 'args'
- brds: List of boards to build
-
- Returns:
- Tuple
- Dictionary which holds the list of boards which were selected
- due to each argument, arranged by argument.
- List of errors found
- """
- result = OrderedDict()
- warnings = []
- terms = self._build_terms(args)
-
- result['all'] = []
- for term in terms:
- result[str(term)] = []
-
- exclude_list = []
- for expr in exclude:
- exclude_list.append(Expr(expr))
-
- found = []
- for brd in self._boards:
- matching_term = None
- build_it = False
- if terms:
- match = False
- for term in terms:
- if term.matches(brd.props):
- matching_term = str(term)
- build_it = True
- break
- elif brds:
- if brd.target in brds:
- build_it = True
- found.append(brd.target)
- else:
- build_it = True
-
- # Check that it is not specifically excluded
- for expr in exclude_list:
- if expr.matches(brd.props):
- build_it = False
- break
-
- if build_it:
- brd.build_it = True
- if matching_term:
- result[matching_term].append(brd.target)
- result['all'].append(brd.target)
-
- if brds:
- remaining = set(brds) - set(found)
- if remaining:
- warnings.append('Boards not found: %s\n' % ', '.join(remaining))
-
- return result, warnings
diff --git a/tools/buildman/boards.py b/tools/buildman/boards.py
new file mode 100644
index 00000000000..ec143f9e0f5
--- /dev/null
+++ b/tools/buildman/boards.py
@@ -0,0 +1,290 @@
+# SPDX-License-Identifier: GPL-2.0+
+# Copyright (c) 2012 The Chromium OS Authors.
+
+"""Maintains a list of boards and allows them to be selected"""
+
+from collections import OrderedDict
+import re
+
+from buildman import board
+
+
+class Expr:
+ """A single regular expression for matching boards to build"""
+
+ def __init__(self, expr):
+ """Set up a new Expr object.
+
+ Args:
+ expr: String cotaining regular expression to store
+ """
+ self._expr = expr
+ self._re = re.compile(expr)
+
+ def matches(self, props):
+ """Check if any of the properties match the regular expression.
+
+ Args:
+ props: List of properties to check
+ Returns:
+ True if any of the properties match the regular expression
+ """
+ for prop in props:
+ if self._re.match(prop):
+ return True
+ return False
+
+ def __str__(self):
+ return self._expr
+
+class Term:
+ """A list of expressions each of which must match with properties.
+
+ This provides a list of 'AND' expressions, meaning that each must
+ match the board properties for that board to be built.
+ """
+ def __init__(self):
+ self._expr_list = []
+ self._board_count = 0
+
+ def add_expr(self, expr):
+ """Add an Expr object to the list to check.
+
+ Args:
+ expr: New Expr object to add to the list of those that must
+ match for a board to be built.
+ """
+ self._expr_list.append(Expr(expr))
+
+ def __str__(self):
+ """Return some sort of useful string describing the term"""
+ return '&'.join([str(expr) for expr in self._expr_list])
+
+ def matches(self, props):
+ """Check if any of the properties match this term
+
+ Each of the expressions in the term is checked. All must match.
+
+ Args:
+ props: List of properties to check
+ Returns:
+ True if all of the expressions in the Term match, else False
+ """
+ for expr in self._expr_list:
+ if not expr.matches(props):
+ return False
+ return True
+
+
+class Boards:
+ """Manage a list of boards."""
+ def __init__(self):
+ # Use a simple list here, sinc OrderedDict requires Python 2.7
+ self._boards = []
+
+ def add_board(self, brd):
+ """Add a new board to the list.
+
+ The board's target member must not already exist in the board list.
+
+ Args:
+ brd: board to add
+ """
+ self._boards.append(brd)
+
+ def read_boards(self, fname):
+ """Read a list of boards from a board file.
+
+ Create a Board object for each and add it to our _boards list.
+
+ Args:
+ fname: Filename of boards.cfg file
+ """
+ with open(fname, 'r', encoding='utf-8') as inf:
+ for line in inf:
+ if line[0] == '#':
+ continue
+ fields = line.split()
+ if not fields:
+ continue
+ for upto, field in enumerate(fields):
+ if field == '-':
+ fields[upto] = ''
+ while len(fields) < 8:
+ fields.append('')
+ if len(fields) > 8:
+ fields = fields[:8]
+
+ brd = board.Board(*fields)
+ self.add_board(brd)
+
+
+ def get_list(self):
+ """Return a list of available boards.
+
+ Returns:
+ List of Board objects
+ """
+ return self._boards
+
+ def get_dict(self):
+ """Build a dictionary containing all the boards.
+
+ Returns:
+ Dictionary:
+ key is board.target
+ value is board
+ """
+ board_dict = OrderedDict()
+ for brd in self._boards:
+ board_dict[brd.target] = brd
+ return board_dict
+
+ def get_selected_dict(self):
+ """Return a dictionary containing the selected boards
+
+ Returns:
+ List of Board objects that are marked selected
+ """
+ board_dict = OrderedDict()
+ for brd in self._boards:
+ if brd.build_it:
+ board_dict[brd.target] = brd
+ return board_dict
+
+ def get_selected(self):
+ """Return a list of selected boards
+
+ Returns:
+ List of Board objects that are marked selected
+ """
+ return [brd for brd in self._boards if brd.build_it]
+
+ def get_selected_names(self):
+ """Return a list of selected boards
+
+ Returns:
+ List of board names that are marked selected
+ """
+ return [brd.target for brd in self._boards if brd.build_it]
+
+ @classmethod
+ def _build_terms(cls, args):
+ """Convert command line arguments to a list of terms.
+
+ This deals with parsing of the arguments. It handles the '&'
+ operator, which joins several expressions into a single Term.
+
+ For example:
+ ['arm & freescale sandbox', 'tegra']
+
+ will produce 3 Terms containing expressions as follows:
+ arm, freescale
+ sandbox
+ tegra
+
+ The first Term has two expressions, both of which must match for
+ a board to be selected.
+
+ Args:
+ args: List of command line arguments
+ Returns:
+ A list of Term objects
+ """
+ syms = []
+ for arg in args:
+ for word in arg.split():
+ sym_build = []
+ for term in word.split('&'):
+ if term:
+ sym_build.append(term)
+ sym_build.append('&')
+ syms += sym_build[:-1]
+ terms = []
+ term = None
+ oper = None
+ for sym in syms:
+ if sym == '&':
+ oper = sym
+ elif oper:
+ term.add_expr(sym)
+ oper = None
+ else:
+ if term:
+ terms.append(term)
+ term = Term()
+ term.add_expr(sym)
+ if term:
+ terms.append(term)
+ return terms
+
+ def select_boards(self, args, exclude=None, brds=None):
+ """Mark boards selected based on args
+
+ Normally either boards (an explicit list of boards) or args (a list of
+ terms to match against) is used. It is possible to specify both, in
+ which case they are additive.
+
+ If brds and args are both empty, all boards are selected.
+
+ Args:
+ args: List of strings specifying boards to include, either named,
+ or by their target, architecture, cpu, vendor or soc. If
+ empty, all boards are selected.
+ exclude: List of boards to exclude, regardless of 'args'
+ brds: List of boards to build
+
+ Returns:
+ Tuple
+ Dictionary which holds the list of boards which were selected
+ due to each argument, arranged by argument.
+ List of errors found
+ """
+ result = OrderedDict()
+ warnings = []
+ terms = self._build_terms(args)
+
+ result['all'] = []
+ for term in terms:
+ result[str(term)] = []
+
+ exclude_list = []
+ if exclude:
+ for expr in exclude:
+ exclude_list.append(Expr(expr))
+
+ found = []
+ for brd in self._boards:
+ matching_term = None
+ build_it = False
+ if terms:
+ for term in terms:
+ if term.matches(brd.props):
+ matching_term = str(term)
+ build_it = True
+ break
+ elif brds:
+ if brd.target in brds:
+ build_it = True
+ found.append(brd.target)
+ else:
+ build_it = True
+
+ # Check that it is not specifically excluded
+ for expr in exclude_list:
+ if expr.matches(brd.props):
+ build_it = False
+ break
+
+ if build_it:
+ brd.build_it = True
+ if matching_term:
+ result[matching_term].append(brd.target)
+ result['all'].append(brd.target)
+
+ if brds:
+ remaining = set(brds) - set(found)
+ if remaining:
+ warnings.append(f"Boards not found: {', '.join(remaining)}\n")
+
+ return result, warnings
diff --git a/tools/buildman/control.py b/tools/buildman/control.py
index a5c1c2e51c6..8d3e781d51a 100644
--- a/tools/buildman/control.py
+++ b/tools/buildman/control.py
@@ -8,7 +8,7 @@ import shutil
import subprocess
import sys
-from buildman import board
+from buildman import boards
from buildman import bsettings
from buildman import cfgutil
from buildman import toolchain
@@ -197,7 +197,7 @@ def DoBuildman(options, args, toolchains=None, make_func=None, brds=None,
if status != 0:
sys.exit("Failed to generate boards.cfg")
- brds = board.Boards()
+ brds = boards.Boards()
brds.read_boards(board_file)
exclude = []
diff --git a/tools/buildman/func_test.py b/tools/buildman/func_test.py
index 6806ea7fbe6..f12e9966349 100644
--- a/tools/buildman/func_test.py
+++ b/tools/buildman/func_test.py
@@ -9,6 +9,7 @@ import tempfile
import unittest
from buildman import board
+from buildman import boards
from buildman import bsettings
from buildman import cmdline
from buildman import control
@@ -187,7 +188,7 @@ class TestFunctional(unittest.TestCase):
self.setupToolchains()
self._toolchains.Add('arm-gcc', test=False)
self._toolchains.Add('powerpc-gcc', test=False)
- self._boards = board.Boards()
+ self._boards = boards.Boards()
for brd in BOARDS:
self._boards.add_board(board.Board(*brd))
@@ -442,7 +443,7 @@ class TestFunctional(unittest.TestCase):
def testNoBoards(self):
"""Test that buildman aborts when there are no boards"""
- self._boards = board.Boards()
+ self._boards = boards.Boards()
with self.assertRaises(SystemExit):
self._RunControl()
@@ -580,7 +581,7 @@ class TestFunctional(unittest.TestCase):
def testWorkInOutput(self):
"""Test the -w option which should write directly to the output dir"""
- board_list = board.Boards()
+ board_list = boards.Boards()
board_list.add_board(board.Board(*BOARDS[0]))
self._RunControl('-o', self._output_dir, '-w', clean_dir=False,
brds=board_list)
@@ -599,14 +600,14 @@ class TestFunctional(unittest.TestCase):
self.assertFalse(
os.path.exists(os.path.join(self._output_dir, 'u-boot')))
- board_list = board.Boards()
+ board_list = boards.Boards()
board_list.add_board(board.Board(*BOARDS[0]))
with self.assertRaises(SystemExit) as e:
self._RunControl('-b', self._test_branch, '-o', self._output_dir,
'-w', clean_dir=False, brds=board_list)
self.assertIn("single commit", str(e.exception))
- board_list = board.Boards()
+ board_list = boards.Boards()
board_list.add_board(board.Board(*BOARDS[0]))
with self.assertRaises(SystemExit) as e:
self._RunControl('-w', clean_dir=False)
diff --git a/tools/buildman/test.py b/tools/buildman/test.py
index d7306fb4dfa..daf5467503e 100644
--- a/tools/buildman/test.py
+++ b/tools/buildman/test.py
@@ -10,6 +10,7 @@ import time
import unittest
from buildman import board
+from buildman import boards
from buildman import bsettings
from buildman import builder
from buildman import cfgutil
@@ -131,7 +132,7 @@ class TestBuild(unittest.TestCase):
self.commits.append(comm)
# Set up boards to build
- self.brds = board.Boards()
+ self.brds = boards.Boards()
for brd in BOARDS:
self.brds.add_board(board.Board(*brd))
self.brds.select_boards([])
--
2.37.0.144.g8ac04bfd2-goog
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 10/19] buildman: Incorporate the genboardscfg.py tool
2022-07-12 1:03 [PATCH 00/19] buildman: Integration of boards.cfg file Simon Glass
` (8 preceding siblings ...)
2022-07-12 1:04 ` [PATCH 09/19] buildman: Split out Boards into its own file Simon Glass
@ 2022-07-12 1:04 ` Simon Glass
2022-07-12 21:38 ` Tom Rini
2022-07-12 1:04 ` [PATCH 11/19] buildman: Tidy up pylint problems in boards module Simon Glass
` (9 subsequent siblings)
19 siblings, 1 reply; 31+ messages in thread
From: Simon Glass @ 2022-07-12 1:04 UTC (permalink / raw)
To: U-Boot Mailing List; +Cc: Tom Rini, Simon Glass
Bring this tool into buildman, so we don't have to run it separately. The
board.cfg file is still produced as part of the build, to save time when
doing another build in the same working directory. If it is out of date
with respect to the Kconfig, it is updated.
Time to regenerate on a recent single-thread machine is 4.6s (1.3s on a
32-thread machine), so we do need some sort of cache if we want buildman
to be useful on incremental builds. We could use Python's pickle format
but:
- it seems useful to allow boards.cfg to be regenerated, at least for a
while, in case other tools use it
- it is possible to grep the file easily, e.g. to find boards which use
a particular SoC (similar to 'buildman -nv <soc>'
Signed-off-by: Simon Glass <sjg@chromium.org>
Suggested-by: Tom Rini <trini@konsulko.com>
---
tools/buildman/README | 21 +-
tools/buildman/boards.py | 402 ++++++++++++++++++++++++++++++++++++++
tools/buildman/cmdline.py | 2 +
tools/buildman/control.py | 16 +-
4 files changed, 426 insertions(+), 15 deletions(-)
diff --git a/tools/buildman/README b/tools/buildman/README
index c67a074cb50..0666bc7d564 100644
--- a/tools/buildman/README
+++ b/tools/buildman/README
@@ -128,10 +128,10 @@ Selecting which boards to build
===============================
Buildman lets you build all boards, or a subset. Specify the subset by passing
-command-line arguments that list the desired board name, architecture name,
-SOC name, or anything else in the boards.cfg file. Multiple arguments are
-allowed. Each argument will be interpreted as a regular expression, so
-behaviour is a superset of exact or substring matching. Examples are:
+command-line arguments that list the desired build target, architecture,
+CPU, board name, vendor, SoC or options. Multiple arguments are allowed. Each
+argument will be interpreted as a regular expression, so behaviour is a superset
+of exact or substring matching. Examples are:
* 'tegra20' All boards with a Tegra20 SoC
* 'tegra' All boards with any Tegra Soc (Tegra20, Tegra30, Tegra114...)
@@ -1054,7 +1054,6 @@ between one commit and the next.
For example:
$ buildman -b squash brppt1 -sU
-boards.cfg is up to date. Nothing to do.
Summary of 2 commits for 3 boards (3 threads, 3 jobs per thread)
01: Migrate bootlimit to Kconfig
02: Squashed commit of the following:
@@ -1309,6 +1308,18 @@ Some options you might like are:
break anything. But note this does not check bisectability!
+Using boards.cfg
+================
+
+This file is no-longer needed by buildman but it is still generated in the
+working directory. This helps avoid a delay on every build, since scanning all
+the Kconfig files takes a few seconds. Use the -R flag to force regeneration
+of the file - in that case buildman exits after writing the file.
+
+You should use 'buildman -nv <criteria>' instead of greoing the boards.cfg file,
+since it may be dropped altogether in future.
+
+
TODO
====
diff --git a/tools/buildman/boards.py b/tools/buildman/boards.py
index ec143f9e0f5..c18914253e4 100644
--- a/tools/buildman/boards.py
+++ b/tools/buildman/boards.py
@@ -1,12 +1,93 @@
# SPDX-License-Identifier: GPL-2.0+
# Copyright (c) 2012 The Chromium OS Authors.
+# Author: Simon Glass <sjg@chromium.org>
+# Author: Masahiro Yamada <yamada.m@jp.panasonic.com>
"""Maintains a list of boards and allows them to be selected"""
from collections import OrderedDict
+import errno
+import fnmatch
+import glob
+import multiprocessing
+import os
import re
+import sys
+import tempfile
+import time
from buildman import board
+from buildman import kconfiglib
+
+
+### constant variables ###
+OUTPUT_FILE = 'boards.cfg'
+CONFIG_DIR = 'configs'
+SLEEP_TIME = 0.03
+COMMENT_BLOCK = '''#
+# List of boards
+# Automatically generated by %s: don't edit
+#
+# Status, Arch, CPU, SoC, Vendor, Board, Target, Options, Maintainers
+
+''' % __file__
+
+
+def try_remove(f):
+ """Remove a file ignoring 'No such file or directory' error."""
+ try:
+ os.remove(f)
+ except OSError as exception:
+ # Ignore 'No such file or directory' error
+ if exception.errno != errno.ENOENT:
+ raise
+
+
+def output_is_new(output):
+ """Check if the output file is up to date.
+
+ Returns:
+ True if the given output file exists and is newer than any of
+ *_defconfig, MAINTAINERS and Kconfig*. False otherwise.
+ """
+ try:
+ ctime = os.path.getctime(output)
+ except OSError as exception:
+ if exception.errno == errno.ENOENT:
+ # return False on 'No such file or directory' error
+ return False
+ else:
+ raise
+
+ for (dirpath, dirnames, filenames) in os.walk(CONFIG_DIR):
+ for filename in fnmatch.filter(filenames, '*_defconfig'):
+ if fnmatch.fnmatch(filename, '.*'):
+ continue
+ filepath = os.path.join(dirpath, filename)
+ if ctime < os.path.getctime(filepath):
+ return False
+
+ for (dirpath, dirnames, filenames) in os.walk('.'):
+ for filename in filenames:
+ if (fnmatch.fnmatch(filename, '*~') or
+ not fnmatch.fnmatch(filename, 'Kconfig*') and
+ not filename == 'MAINTAINERS'):
+ continue
+ filepath = os.path.join(dirpath, filename)
+ if ctime < os.path.getctime(filepath):
+ return False
+
+ # Detect a board that has been removed since the current board database
+ # was generated
+ with open(output, encoding="utf-8") as f:
+ for line in f:
+ if line[0] == '#' or line == '\n':
+ continue
+ defconfig = line.split()[6] + '_defconfig'
+ if not os.path.exists(os.path.join(CONFIG_DIR, defconfig)):
+ return False
+
+ return True
class Expr:
@@ -76,6 +157,190 @@ class Term:
return True
+class KconfigScanner:
+
+ """Kconfig scanner."""
+
+ ### constant variable only used in this class ###
+ _SYMBOL_TABLE = {
+ 'arch' : 'SYS_ARCH',
+ 'cpu' : 'SYS_CPU',
+ 'soc' : 'SYS_SOC',
+ 'vendor' : 'SYS_VENDOR',
+ 'board' : 'SYS_BOARD',
+ 'config' : 'SYS_CONFIG_NAME',
+ 'options' : 'SYS_EXTRA_OPTIONS'
+ }
+
+ def __init__(self):
+ """Scan all the Kconfig files and create a Kconfig object."""
+ # Define environment variables referenced from Kconfig
+ os.environ['srctree'] = os.getcwd()
+ os.environ['UBOOTVERSION'] = 'dummy'
+ os.environ['KCONFIG_OBJDIR'] = ''
+ self._conf = kconfiglib.Kconfig(warn=False)
+
+ def __del__(self):
+ """Delete a leftover temporary file before exit.
+
+ The scan() method of this class creates a temporay file and deletes
+ it on success. If scan() method throws an exception on the way,
+ the temporary file might be left over. In that case, it should be
+ deleted in this destructor.
+ """
+ if hasattr(self, '_tmpfile') and self._tmpfile:
+ try_remove(self._tmpfile)
+
+ def scan(self, defconfig):
+ """Load a defconfig file to obtain board parameters.
+
+ Arguments:
+ defconfig: path to the defconfig file to be processed
+
+ Returns:
+ A dictionary of board parameters. It has a form of:
+ {
+ 'arch': <arch_name>,
+ 'cpu': <cpu_name>,
+ 'soc': <soc_name>,
+ 'vendor': <vendor_name>,
+ 'board': <board_name>,
+ 'target': <target_name>,
+ 'config': <config_header_name>,
+ 'options': <extra_options>
+ }
+ """
+ # strip special prefixes and save it in a temporary file
+ fd, self._tmpfile = tempfile.mkstemp()
+ with os.fdopen(fd, 'w') as f:
+ for line in open(defconfig):
+ colon = line.find(':CONFIG_')
+ if colon == -1:
+ f.write(line)
+ else:
+ f.write(line[colon + 1:])
+
+ self._conf.load_config(self._tmpfile)
+ try_remove(self._tmpfile)
+ self._tmpfile = None
+
+ params = {}
+
+ # Get the value of CONFIG_SYS_ARCH, CONFIG_SYS_CPU, ... etc.
+ # Set '-' if the value is empty.
+ for key, symbol in list(self._SYMBOL_TABLE.items()):
+ value = self._conf.syms.get(symbol).str_value
+ if value:
+ params[key] = value
+ else:
+ params[key] = '-'
+
+ defconfig = os.path.basename(defconfig)
+ params['target'], match, rear = defconfig.partition('_defconfig')
+ assert match and not rear, '%s : invalid defconfig' % defconfig
+
+ # fix-up for aarch64
+ if params['arch'] == 'arm' and params['cpu'] == 'armv8':
+ params['arch'] = 'aarch64'
+
+ # fix-up options field. It should have the form:
+ # <config name>[:comma separated config options]
+ if params['options'] != '-':
+ params['options'] = params['config'] + ':' + \
+ params['options'].replace(r'\"', '"')
+ elif params['config'] != params['target']:
+ params['options'] = params['config']
+
+ return params
+
+
+class MaintainersDatabase:
+
+ """The database of board status and maintainers."""
+
+ def __init__(self):
+ """Create an empty database."""
+ self.database = {}
+
+ def get_status(self, target):
+ """Return the status of the given board.
+
+ The board status is generally either 'Active' or 'Orphan'.
+ Display a warning message and return '-' if status information
+ is not found.
+
+ Returns:
+ 'Active', 'Orphan' or '-'.
+ """
+ if not target in self.database:
+ print("WARNING: no status info for '%s'" % target, file=sys.stderr)
+ return '-'
+
+ tmp = self.database[target][0]
+ if tmp.startswith('Maintained'):
+ return 'Active'
+ elif tmp.startswith('Supported'):
+ return 'Active'
+ elif tmp.startswith('Orphan'):
+ return 'Orphan'
+ else:
+ print(("WARNING: %s: unknown status for '%s'" %
+ (tmp, target)), file=sys.stderr)
+ return '-'
+
+ def get_maintainers(self, target):
+ """Return the maintainers of the given board.
+
+ Returns:
+ Maintainers of the board. If the board has two or more maintainers,
+ they are separated with colons.
+ """
+ if not target in self.database:
+ print("WARNING: no maintainers for '%s'" % target, file=sys.stderr)
+ return ''
+
+ return ':'.join(self.database[target][1])
+
+ def parse_file(self, file):
+ """Parse a MAINTAINERS file.
+
+ Parse a MAINTAINERS file and accumulates board status and
+ maintainers information.
+
+ Arguments:
+ file: MAINTAINERS file to be parsed
+ """
+ targets = []
+ maintainers = []
+ status = '-'
+ for line in open(file, encoding="utf-8"):
+ # Check also commented maintainers
+ if line[:3] == '#M:':
+ line = line[1:]
+ tag, rest = line[:2], line[2:].strip()
+ if tag == 'M:':
+ maintainers.append(rest)
+ elif tag == 'F:':
+ # expand wildcard and filter by 'configs/*_defconfig'
+ for f in glob.glob(rest):
+ front, match, rear = f.partition('configs/')
+ if not front and match:
+ front, match, rear = rear.rpartition('_defconfig')
+ if match and not rear:
+ targets.append(front)
+ elif tag == 'S:':
+ status = rest
+ elif line == '\n':
+ for target in targets:
+ self.database[target] = (status, maintainers)
+ targets = []
+ maintainers = []
+ status = '-'
+ if targets:
+ for target in targets:
+ self.database[target] = (status, maintainers)
+
+
class Boards:
"""Manage a list of boards."""
def __init__(self):
@@ -288,3 +553,140 @@ class Boards:
warnings.append(f"Boards not found: {', '.join(remaining)}\n")
return result, warnings
+
+ def scan_defconfigs_for_multiprocess(self, queue, defconfigs):
+ """Scan defconfig files and queue their board parameters
+
+ This function is intended to be passed to
+ multiprocessing.Process() constructor.
+
+ Arguments:
+ queue: An instance of multiprocessing.Queue().
+ The resulting board parameters are written into it.
+ defconfigs: A sequence of defconfig files to be scanned.
+ """
+ kconf_scanner = KconfigScanner()
+ for defconfig in defconfigs:
+ queue.put(kconf_scanner.scan(defconfig))
+
+ def read_queues(self, queues, params_list):
+ """Read the queues and append the data to the paramers list"""
+ for q in queues:
+ while not q.empty():
+ params_list.append(q.get())
+
+ def scan_defconfigs(self, jobs=1):
+ """Collect board parameters for all defconfig files.
+
+ This function invokes multiple processes for faster processing.
+
+ Arguments:
+ jobs: The number of jobs to run simultaneously
+ """
+ all_defconfigs = []
+ for (dirpath, dirnames, filenames) in os.walk(CONFIG_DIR):
+ for filename in fnmatch.filter(filenames, '*_defconfig'):
+ if fnmatch.fnmatch(filename, '.*'):
+ continue
+ all_defconfigs.append(os.path.join(dirpath, filename))
+
+ total_boards = len(all_defconfigs)
+ processes = []
+ queues = []
+ for i in range(jobs):
+ defconfigs = all_defconfigs[total_boards * i // jobs :
+ total_boards * (i + 1) // jobs]
+ q = multiprocessing.Queue(maxsize=-1)
+ p = multiprocessing.Process(
+ target=self.scan_defconfigs_for_multiprocess,
+ args=(q, defconfigs))
+ p.start()
+ processes.append(p)
+ queues.append(q)
+
+ # The resulting data should be accumulated to this list
+ params_list = []
+
+ # Data in the queues should be retrieved preriodically.
+ # Otherwise, the queues would become full and subprocesses would get stuck.
+ while any([p.is_alive() for p in processes]):
+ self.read_queues(queues, params_list)
+ # sleep for a while until the queues are filled
+ time.sleep(SLEEP_TIME)
+
+ # Joining subprocesses just in case
+ # (All subprocesses should already have been finished)
+ for p in processes:
+ p.join()
+
+ # retrieve leftover data
+ self.read_queues(queues, params_list)
+
+ return params_list
+
+ def insert_maintainers_info(self, params_list):
+ """Add Status and Maintainers information to the board parameters list.
+
+ Arguments:
+ params_list: A list of the board parameters
+ """
+ database = MaintainersDatabase()
+ for (dirpath, dirnames, filenames) in os.walk('.'):
+ if 'MAINTAINERS' in filenames:
+ database.parse_file(os.path.join(dirpath, 'MAINTAINERS'))
+
+ for i, params in enumerate(params_list):
+ target = params['target']
+ params['status'] = database.get_status(target)
+ params['maintainers'] = database.get_maintainers(target)
+ params_list[i] = params
+
+ def format_and_output(self, params_list, output):
+ """Write board parameters into a file.
+
+ Columnate the board parameters, sort lines alphabetically,
+ and then write them to a file.
+
+ Arguments:
+ params_list: The list of board parameters
+ output: The path to the output file
+ """
+ FIELDS = ('status', 'arch', 'cpu', 'soc', 'vendor', 'board', 'target',
+ 'options', 'maintainers')
+
+ # First, decide the width of each column
+ max_length = dict([ (f, 0) for f in FIELDS])
+ for params in params_list:
+ for f in FIELDS:
+ max_length[f] = max(max_length[f], len(params[f]))
+
+ output_lines = []
+ for params in params_list:
+ line = ''
+ for f in FIELDS:
+ # insert two spaces between fields like column -t would
+ line += ' ' + params[f].ljust(max_length[f])
+ output_lines.append(line.strip())
+
+ # ignore case when sorting
+ output_lines.sort(key=str.lower)
+
+ with open(output, 'w', encoding="utf-8") as f:
+ f.write(COMMENT_BLOCK + '\n'.join(output_lines) + '\n')
+
+ def ensure_board_list(self, output, jobs=1, force=False, quiet=False):
+ """Generate a board database file if needed.
+
+ Arguments:
+ output: The name of the output file
+ jobs: The number of jobs to run simultaneously
+ force: Force to generate the output even if it is new
+ quiet: True to avoid printing a message if nothing needs doing
+ """
+ if not force and output_is_new(output):
+ if not quiet:
+ print("%s is up to date. Nothing to do." % output)
+ return
+ params_list = self.scan_defconfigs(jobs)
+ self.insert_maintainers_info(params_list)
+ self.format_and_output(params_list, output)
diff --git a/tools/buildman/cmdline.py b/tools/buildman/cmdline.py
index cef2068c983..b29c1eb5ee7 100644
--- a/tools/buildman/cmdline.py
+++ b/tools/buildman/cmdline.py
@@ -89,6 +89,8 @@ def ParseArgs():
default=False, help="Use full toolchain path in CROSS_COMPILE")
parser.add_option('-P', '--per-board-out-dir', action='store_true',
default=False, help="Use an O= (output) directory per board rather than per thread")
+ parser.add_option('-R', '--regen-board-list', action='store_true',
+ help='Force regeneration of the list of boards, like the old boards.cfg file')
parser.add_option('-s', '--summary', action='store_true',
default=False, help='Show a build summary')
parser.add_option('-S', '--show-sizes', action='store_true',
diff --git a/tools/buildman/control.py b/tools/buildman/control.py
index 8d3e781d51a..79ce2f6978a 100644
--- a/tools/buildman/control.py
+++ b/tools/buildman/control.py
@@ -186,18 +186,14 @@ def DoBuildman(options, args, toolchains=None, make_func=None, brds=None,
if not os.path.exists(options.output_dir):
os.makedirs(options.output_dir)
board_file = os.path.join(options.output_dir, 'boards.cfg')
- our_path = os.path.dirname(os.path.realpath(__file__))
- genboardscfg = os.path.join(our_path, '../genboardscfg.py')
- if not os.path.exists(genboardscfg):
- genboardscfg = os.path.join(options.git, 'tools/genboardscfg.py')
- status = subprocess.call([genboardscfg, '-q', '-o', board_file])
- if status != 0:
- # Older versions don't support -q
- status = subprocess.call([genboardscfg, '-o', board_file])
- if status != 0:
- sys.exit("Failed to generate boards.cfg")
brds = boards.Boards()
+ brds.ensure_board_list(board_file,
+ options.threads or multiprocessing.cpu_count(),
+ force=options.regen_board_list,
+ quiet=not options.verbose)
+ if options.regen_board_list:
+ return 0
brds.read_boards(board_file)
exclude = []
--
2.37.0.144.g8ac04bfd2-goog
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [PATCH 10/19] buildman: Incorporate the genboardscfg.py tool
2022-07-12 1:04 ` [PATCH 10/19] buildman: Incorporate the genboardscfg.py tool Simon Glass
@ 2022-07-12 21:38 ` Tom Rini
2022-07-13 15:28 ` Simon Glass
0 siblings, 1 reply; 31+ messages in thread
From: Tom Rini @ 2022-07-12 21:38 UTC (permalink / raw)
To: Simon Glass; +Cc: U-Boot Mailing List
[-- Attachment #1: Type: text/plain, Size: 1243 bytes --]
On Mon, Jul 11, 2022 at 07:04:04PM -0600, Simon Glass wrote:
> Bring this tool into buildman, so we don't have to run it separately. The
> board.cfg file is still produced as part of the build, to save time when
> doing another build in the same working directory. If it is out of date
> with respect to the Kconfig, it is updated.
>
> Time to regenerate on a recent single-thread machine is 4.6s (1.3s on a
> 32-thread machine), so we do need some sort of cache if we want buildman
> to be useful on incremental builds. We could use Python's pickle format
> but:
>
> - it seems useful to allow boards.cfg to be regenerated, at least for a
> while, in case other tools use it
> - it is possible to grep the file easily, e.g. to find boards which use
> a particular SoC (similar to 'buildman -nv <soc>'
While I don't think other tools still use boards.cfg, this will make it
easier to find out that I'm wrong. Perhaps once the CONFIG to Kconfig
migration is done we can move to just pickle'ing the data or similar
since I find the main use of what was in boards.cfg can be figured out
with some other git grep'ing, and in turn that's mainly for me when
trying to convert stuff. Thanks for doing this.
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 10/19] buildman: Incorporate the genboardscfg.py tool
2022-07-12 21:38 ` Tom Rini
@ 2022-07-13 15:28 ` Simon Glass
2022-07-13 18:21 ` Tom Rini
0 siblings, 1 reply; 31+ messages in thread
From: Simon Glass @ 2022-07-13 15:28 UTC (permalink / raw)
To: Tom Rini; +Cc: U-Boot Mailing List
Hi Tom,
On Tue, 12 Jul 2022 at 15:38, Tom Rini <trini@konsulko.com> wrote:
>
> On Mon, Jul 11, 2022 at 07:04:04PM -0600, Simon Glass wrote:
> > Bring this tool into buildman, so we don't have to run it separately. The
> > board.cfg file is still produced as part of the build, to save time when
> > doing another build in the same working directory. If it is out of date
> > with respect to the Kconfig, it is updated.
> >
> > Time to regenerate on a recent single-thread machine is 4.6s (1.3s on a
> > 32-thread machine), so we do need some sort of cache if we want buildman
> > to be useful on incremental builds. We could use Python's pickle format
> > but:
> >
> > - it seems useful to allow boards.cfg to be regenerated, at least for a
> > while, in case other tools use it
> > - it is possible to grep the file easily, e.g. to find boards which use
> > a particular SoC (similar to 'buildman -nv <soc>'
>
> While I don't think other tools still use boards.cfg, this will make it
> easier to find out that I'm wrong. Perhaps once the CONFIG to Kconfig
> migration is done we can move to just pickle'ing the data or similar
> since I find the main use of what was in boards.cfg can be figured out
> with some other git grep'ing, and in turn that's mainly for me when
> trying to convert stuff. Thanks for doing this.
Yes. I'm excited to hear that Kconfig migration might be done - any
forecast as to when?
One thing we could to is provide an option for buildman to spit out
the various fields that go into boards.cfg
Regards,
SImon
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 10/19] buildman: Incorporate the genboardscfg.py tool
2022-07-13 15:28 ` Simon Glass
@ 2022-07-13 18:21 ` Tom Rini
2022-07-14 10:21 ` Simon Glass
0 siblings, 1 reply; 31+ messages in thread
From: Tom Rini @ 2022-07-13 18:21 UTC (permalink / raw)
To: Simon Glass; +Cc: U-Boot Mailing List
[-- Attachment #1: Type: text/plain, Size: 2288 bytes --]
On Wed, Jul 13, 2022 at 09:28:06AM -0600, Simon Glass wrote:
> Hi Tom,
>
> On Tue, 12 Jul 2022 at 15:38, Tom Rini <trini@konsulko.com> wrote:
> >
> > On Mon, Jul 11, 2022 at 07:04:04PM -0600, Simon Glass wrote:
> > > Bring this tool into buildman, so we don't have to run it separately. The
> > > board.cfg file is still produced as part of the build, to save time when
> > > doing another build in the same working directory. If it is out of date
> > > with respect to the Kconfig, it is updated.
> > >
> > > Time to regenerate on a recent single-thread machine is 4.6s (1.3s on a
> > > 32-thread machine), so we do need some sort of cache if we want buildman
> > > to be useful on incremental builds. We could use Python's pickle format
> > > but:
> > >
> > > - it seems useful to allow boards.cfg to be regenerated, at least for a
> > > while, in case other tools use it
> > > - it is possible to grep the file easily, e.g. to find boards which use
> > > a particular SoC (similar to 'buildman -nv <soc>'
> >
> > While I don't think other tools still use boards.cfg, this will make it
> > easier to find out that I'm wrong. Perhaps once the CONFIG to Kconfig
> > migration is done we can move to just pickle'ing the data or similar
> > since I find the main use of what was in boards.cfg can be figured out
> > with some other git grep'ing, and in turn that's mainly for me when
> > trying to convert stuff. Thanks for doing this.
>
> Yes. I'm excited to hear that Kconfig migration might be done - any
> forecast as to when?
Not yet. I'm auditing CONFIG_SYS_* now, with a notion to move
everything that's not really configurable just out of CONFIG namespace
as the starting point. That'll drop us down to ~500 to migrate, which
feels a bit less daunting.
> One thing we could to is provide an option for buildman to spit out
> the various fields that go into boards.cfg
Right. So I might not have said this before, but one reason I wanted
buildman to natively know kconfiglib and have everything was that while
we can do a lot of good matching on what to build, it would be amazingly
good to be able to say "build every platform with NVME_PCI set" (and if
it's not too hard hex/int options with a specific value).
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 10/19] buildman: Incorporate the genboardscfg.py tool
2022-07-13 18:21 ` Tom Rini
@ 2022-07-14 10:21 ` Simon Glass
2022-07-18 12:11 ` Tom Rini
0 siblings, 1 reply; 31+ messages in thread
From: Simon Glass @ 2022-07-14 10:21 UTC (permalink / raw)
To: Tom Rini; +Cc: U-Boot Mailing List
Hi Tom,
On Wed, 13 Jul 2022 at 12:21, Tom Rini <trini@konsulko.com> wrote:
>
> On Wed, Jul 13, 2022 at 09:28:06AM -0600, Simon Glass wrote:
> > Hi Tom,
> >
> > On Tue, 12 Jul 2022 at 15:38, Tom Rini <trini@konsulko.com> wrote:
> > >
> > > On Mon, Jul 11, 2022 at 07:04:04PM -0600, Simon Glass wrote:
> > > > Bring this tool into buildman, so we don't have to run it separately. The
> > > > board.cfg file is still produced as part of the build, to save time when
> > > > doing another build in the same working directory. If it is out of date
> > > > with respect to the Kconfig, it is updated.
> > > >
> > > > Time to regenerate on a recent single-thread machine is 4.6s (1.3s on a
> > > > 32-thread machine), so we do need some sort of cache if we want buildman
> > > > to be useful on incremental builds. We could use Python's pickle format
> > > > but:
> > > >
> > > > - it seems useful to allow boards.cfg to be regenerated, at least for a
> > > > while, in case other tools use it
> > > > - it is possible to grep the file easily, e.g. to find boards which use
> > > > a particular SoC (similar to 'buildman -nv <soc>'
> > >
> > > While I don't think other tools still use boards.cfg, this will make it
> > > easier to find out that I'm wrong. Perhaps once the CONFIG to Kconfig
> > > migration is done we can move to just pickle'ing the data or similar
> > > since I find the main use of what was in boards.cfg can be figured out
> > > with some other git grep'ing, and in turn that's mainly for me when
> > > trying to convert stuff. Thanks for doing this.
> >
> > Yes. I'm excited to hear that Kconfig migration might be done - any
> > forecast as to when?
>
> Not yet. I'm auditing CONFIG_SYS_* now, with a notion to move
> everything that's not really configurable just out of CONFIG namespace
> as the starting point. That'll drop us down to ~500 to migrate, which
> feels a bit less daunting.
>
> > One thing we could to is provide an option for buildman to spit out
> > the various fields that go into boards.cfg
>
> Right. So I might not have said this before, but one reason I wanted
> buildman to natively know kconfiglib and have everything was that while
> we can do a lot of good matching on what to build, it would be amazingly
> good to be able to say "build every platform with NVME_PCI set" (and if
> it's not too hard hex/int options with a specific value).
Ah OK. At present moveconfig has the functionality to list the boards
that have particular options (-b and -f). It is expensive to build the
database though - over a minute on a 32-thread machine. So we would
have to cache it. Also just about any change would invalidate the
cache and I'm not sure if it possible to detect which changes have no
effect on which cache entries...
Regards,
Simon
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 10/19] buildman: Incorporate the genboardscfg.py tool
2022-07-14 10:21 ` Simon Glass
@ 2022-07-18 12:11 ` Tom Rini
2022-07-20 15:01 ` Simon Glass
0 siblings, 1 reply; 31+ messages in thread
From: Tom Rini @ 2022-07-18 12:11 UTC (permalink / raw)
To: Simon Glass; +Cc: U-Boot Mailing List
[-- Attachment #1: Type: text/plain, Size: 3263 bytes --]
On Thu, Jul 14, 2022 at 04:21:57AM -0600, Simon Glass wrote:
> Hi Tom,
>
> On Wed, 13 Jul 2022 at 12:21, Tom Rini <trini@konsulko.com> wrote:
> >
> > On Wed, Jul 13, 2022 at 09:28:06AM -0600, Simon Glass wrote:
> > > Hi Tom,
> > >
> > > On Tue, 12 Jul 2022 at 15:38, Tom Rini <trini@konsulko.com> wrote:
> > > >
> > > > On Mon, Jul 11, 2022 at 07:04:04PM -0600, Simon Glass wrote:
> > > > > Bring this tool into buildman, so we don't have to run it separately. The
> > > > > board.cfg file is still produced as part of the build, to save time when
> > > > > doing another build in the same working directory. If it is out of date
> > > > > with respect to the Kconfig, it is updated.
> > > > >
> > > > > Time to regenerate on a recent single-thread machine is 4.6s (1.3s on a
> > > > > 32-thread machine), so we do need some sort of cache if we want buildman
> > > > > to be useful on incremental builds. We could use Python's pickle format
> > > > > but:
> > > > >
> > > > > - it seems useful to allow boards.cfg to be regenerated, at least for a
> > > > > while, in case other tools use it
> > > > > - it is possible to grep the file easily, e.g. to find boards which use
> > > > > a particular SoC (similar to 'buildman -nv <soc>'
> > > >
> > > > While I don't think other tools still use boards.cfg, this will make it
> > > > easier to find out that I'm wrong. Perhaps once the CONFIG to Kconfig
> > > > migration is done we can move to just pickle'ing the data or similar
> > > > since I find the main use of what was in boards.cfg can be figured out
> > > > with some other git grep'ing, and in turn that's mainly for me when
> > > > trying to convert stuff. Thanks for doing this.
> > >
> > > Yes. I'm excited to hear that Kconfig migration might be done - any
> > > forecast as to when?
> >
> > Not yet. I'm auditing CONFIG_SYS_* now, with a notion to move
> > everything that's not really configurable just out of CONFIG namespace
> > as the starting point. That'll drop us down to ~500 to migrate, which
> > feels a bit less daunting.
> >
> > > One thing we could to is provide an option for buildman to spit out
> > > the various fields that go into boards.cfg
> >
> > Right. So I might not have said this before, but one reason I wanted
> > buildman to natively know kconfiglib and have everything was that while
> > we can do a lot of good matching on what to build, it would be amazingly
> > good to be able to say "build every platform with NVME_PCI set" (and if
> > it's not too hard hex/int options with a specific value).
>
> Ah OK. At present moveconfig has the functionality to list the boards
> that have particular options (-b and -f). It is expensive to build the
> database though - over a minute on a 32-thread machine. So we would
> have to cache it. Also just about any change would invalidate the
> cache and I'm not sure if it possible to detect which changes have no
> effect on which cache entries...
Ah, maybe it will take some more thinking about then. Maybe an
"advanced" match option, and also seeing how to have Azure generate the db
in one job and pass it as an artifact to every other job in the world
build stage. Not an immediate need.
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 10/19] buildman: Incorporate the genboardscfg.py tool
2022-07-18 12:11 ` Tom Rini
@ 2022-07-20 15:01 ` Simon Glass
2022-07-20 17:15 ` Tom Rini
0 siblings, 1 reply; 31+ messages in thread
From: Simon Glass @ 2022-07-20 15:01 UTC (permalink / raw)
To: Tom Rini; +Cc: U-Boot Mailing List
Hi Tom,
On Mon, 18 Jul 2022 at 06:11, Tom Rini <trini@konsulko.com> wrote:
>
> On Thu, Jul 14, 2022 at 04:21:57AM -0600, Simon Glass wrote:
> > Hi Tom,
> >
> > On Wed, 13 Jul 2022 at 12:21, Tom Rini <trini@konsulko.com> wrote:
> > >
> > > On Wed, Jul 13, 2022 at 09:28:06AM -0600, Simon Glass wrote:
> > > > Hi Tom,
> > > >
> > > > On Tue, 12 Jul 2022 at 15:38, Tom Rini <trini@konsulko.com> wrote:
> > > > >
> > > > > On Mon, Jul 11, 2022 at 07:04:04PM -0600, Simon Glass wrote:
> > > > > > Bring this tool into buildman, so we don't have to run it separately. The
> > > > > > board.cfg file is still produced as part of the build, to save time when
> > > > > > doing another build in the same working directory. If it is out of date
> > > > > > with respect to the Kconfig, it is updated.
> > > > > >
> > > > > > Time to regenerate on a recent single-thread machine is 4.6s (1.3s on a
> > > > > > 32-thread machine), so we do need some sort of cache if we want buildman
> > > > > > to be useful on incremental builds. We could use Python's pickle format
> > > > > > but:
> > > > > >
> > > > > > - it seems useful to allow boards.cfg to be regenerated, at least for a
> > > > > > while, in case other tools use it
> > > > > > - it is possible to grep the file easily, e.g. to find boards which use
> > > > > > a particular SoC (similar to 'buildman -nv <soc>'
> > > > >
> > > > > While I don't think other tools still use boards.cfg, this will make it
> > > > > easier to find out that I'm wrong. Perhaps once the CONFIG to Kconfig
> > > > > migration is done we can move to just pickle'ing the data or similar
> > > > > since I find the main use of what was in boards.cfg can be figured out
> > > > > with some other git grep'ing, and in turn that's mainly for me when
> > > > > trying to convert stuff. Thanks for doing this.
> > > >
> > > > Yes. I'm excited to hear that Kconfig migration might be done - any
> > > > forecast as to when?
> > >
> > > Not yet. I'm auditing CONFIG_SYS_* now, with a notion to move
> > > everything that's not really configurable just out of CONFIG namespace
> > > as the starting point. That'll drop us down to ~500 to migrate, which
> > > feels a bit less daunting.
> > >
> > > > One thing we could to is provide an option for buildman to spit out
> > > > the various fields that go into boards.cfg
> > >
> > > Right. So I might not have said this before, but one reason I wanted
> > > buildman to natively know kconfiglib and have everything was that while
> > > we can do a lot of good matching on what to build, it would be amazingly
> > > good to be able to say "build every platform with NVME_PCI set" (and if
> > > it's not too hard hex/int options with a specific value).
> >
> > Ah OK. At present moveconfig has the functionality to list the boards
> > that have particular options (-b and -f). It is expensive to build the
> > database though - over a minute on a 32-thread machine. So we would
> > have to cache it. Also just about any change would invalidate the
> > cache and I'm not sure if it possible to detect which changes have no
> > effect on which cache entries...
>
> Ah, maybe it will take some more thinking about then. Maybe an
> "advanced" match option, and also seeing how to have Azure generate the db
> in one job and pass it as an artifact to every other job in the world
> build stage. Not an immediate need.
Well I suppose having that logic in moveconfig doesn't make a lot of
sense. So we could move it to buildman and have a way of creating the
database, as you say. But bear in mind that every commit being built
has the potential to change the Kconfig. It may be possible to hash
the files or detect changes using timestamps.
Regards,
Simon
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 10/19] buildman: Incorporate the genboardscfg.py tool
2022-07-20 15:01 ` Simon Glass
@ 2022-07-20 17:15 ` Tom Rini
2022-07-22 8:59 ` Simon Glass
0 siblings, 1 reply; 31+ messages in thread
From: Tom Rini @ 2022-07-20 17:15 UTC (permalink / raw)
To: Simon Glass; +Cc: U-Boot Mailing List
[-- Attachment #1: Type: text/plain, Size: 4448 bytes --]
On Wed, Jul 20, 2022 at 09:01:04AM -0600, Simon Glass wrote:
> Hi Tom,
>
> On Mon, 18 Jul 2022 at 06:11, Tom Rini <trini@konsulko.com> wrote:
> >
> > On Thu, Jul 14, 2022 at 04:21:57AM -0600, Simon Glass wrote:
> > > Hi Tom,
> > >
> > > On Wed, 13 Jul 2022 at 12:21, Tom Rini <trini@konsulko.com> wrote:
> > > >
> > > > On Wed, Jul 13, 2022 at 09:28:06AM -0600, Simon Glass wrote:
> > > > > Hi Tom,
> > > > >
> > > > > On Tue, 12 Jul 2022 at 15:38, Tom Rini <trini@konsulko.com> wrote:
> > > > > >
> > > > > > On Mon, Jul 11, 2022 at 07:04:04PM -0600, Simon Glass wrote:
> > > > > > > Bring this tool into buildman, so we don't have to run it separately. The
> > > > > > > board.cfg file is still produced as part of the build, to save time when
> > > > > > > doing another build in the same working directory. If it is out of date
> > > > > > > with respect to the Kconfig, it is updated.
> > > > > > >
> > > > > > > Time to regenerate on a recent single-thread machine is 4.6s (1.3s on a
> > > > > > > 32-thread machine), so we do need some sort of cache if we want buildman
> > > > > > > to be useful on incremental builds. We could use Python's pickle format
> > > > > > > but:
> > > > > > >
> > > > > > > - it seems useful to allow boards.cfg to be regenerated, at least for a
> > > > > > > while, in case other tools use it
> > > > > > > - it is possible to grep the file easily, e.g. to find boards which use
> > > > > > > a particular SoC (similar to 'buildman -nv <soc>'
> > > > > >
> > > > > > While I don't think other tools still use boards.cfg, this will make it
> > > > > > easier to find out that I'm wrong. Perhaps once the CONFIG to Kconfig
> > > > > > migration is done we can move to just pickle'ing the data or similar
> > > > > > since I find the main use of what was in boards.cfg can be figured out
> > > > > > with some other git grep'ing, and in turn that's mainly for me when
> > > > > > trying to convert stuff. Thanks for doing this.
> > > > >
> > > > > Yes. I'm excited to hear that Kconfig migration might be done - any
> > > > > forecast as to when?
> > > >
> > > > Not yet. I'm auditing CONFIG_SYS_* now, with a notion to move
> > > > everything that's not really configurable just out of CONFIG namespace
> > > > as the starting point. That'll drop us down to ~500 to migrate, which
> > > > feels a bit less daunting.
> > > >
> > > > > One thing we could to is provide an option for buildman to spit out
> > > > > the various fields that go into boards.cfg
> > > >
> > > > Right. So I might not have said this before, but one reason I wanted
> > > > buildman to natively know kconfiglib and have everything was that while
> > > > we can do a lot of good matching on what to build, it would be amazingly
> > > > good to be able to say "build every platform with NVME_PCI set" (and if
> > > > it's not too hard hex/int options with a specific value).
> > >
> > > Ah OK. At present moveconfig has the functionality to list the boards
> > > that have particular options (-b and -f). It is expensive to build the
> > > database though - over a minute on a 32-thread machine. So we would
> > > have to cache it. Also just about any change would invalidate the
> > > cache and I'm not sure if it possible to detect which changes have no
> > > effect on which cache entries...
> >
> > Ah, maybe it will take some more thinking about then. Maybe an
> > "advanced" match option, and also seeing how to have Azure generate the db
> > in one job and pass it as an artifact to every other job in the world
> > build stage. Not an immediate need.
>
> Well I suppose having that logic in moveconfig doesn't make a lot of
> sense. So we could move it to buildman and have a way of creating the
> database, as you say. But bear in mind that every commit being built
> has the potential to change the Kconfig. It may be possible to hash
> the files or detect changes using timestamps.
Yeah, we can revisit this later on down the line. I don't think (and we
don't do it today either, fwiw) we need to ensure the list of boards to
build is right for each step of the commit, and might even be
counter-productive, at least for my use case. Build the list of boards
on top of tree, tell me how it changed over N commits (which commit
caused the size change) is how I use things. Then the CI case is only
for top of tree anyhow.
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH 10/19] buildman: Incorporate the genboardscfg.py tool
2022-07-20 17:15 ` Tom Rini
@ 2022-07-22 8:59 ` Simon Glass
0 siblings, 0 replies; 31+ messages in thread
From: Simon Glass @ 2022-07-22 8:59 UTC (permalink / raw)
To: Tom Rini; +Cc: U-Boot Mailing List
Hi Tom,
On Wed, 20 Jul 2022 at 11:16, Tom Rini <trini@konsulko.com> wrote:
>
> On Wed, Jul 20, 2022 at 09:01:04AM -0600, Simon Glass wrote:
> > Hi Tom,
> >
> > On Mon, 18 Jul 2022 at 06:11, Tom Rini <trini@konsulko.com> wrote:
> > >
> > > On Thu, Jul 14, 2022 at 04:21:57AM -0600, Simon Glass wrote:
> > > > Hi Tom,
> > > >
> > > > On Wed, 13 Jul 2022 at 12:21, Tom Rini <trini@konsulko.com> wrote:
> > > > >
> > > > > On Wed, Jul 13, 2022 at 09:28:06AM -0600, Simon Glass wrote:
> > > > > > Hi Tom,
> > > > > >
> > > > > > On Tue, 12 Jul 2022 at 15:38, Tom Rini <trini@konsulko.com> wrote:
> > > > > > >
> > > > > > > On Mon, Jul 11, 2022 at 07:04:04PM -0600, Simon Glass wrote:
> > > > > > > > Bring this tool into buildman, so we don't have to run it separately. The
> > > > > > > > board.cfg file is still produced as part of the build, to save time when
> > > > > > > > doing another build in the same working directory. If it is out of date
> > > > > > > > with respect to the Kconfig, it is updated.
> > > > > > > >
> > > > > > > > Time to regenerate on a recent single-thread machine is 4.6s (1.3s on a
> > > > > > > > 32-thread machine), so we do need some sort of cache if we want buildman
> > > > > > > > to be useful on incremental builds. We could use Python's pickle format
> > > > > > > > but:
> > > > > > > >
> > > > > > > > - it seems useful to allow boards.cfg to be regenerated, at least for a
> > > > > > > > while, in case other tools use it
> > > > > > > > - it is possible to grep the file easily, e.g. to find boards which use
> > > > > > > > a particular SoC (similar to 'buildman -nv <soc>'
> > > > > > >
> > > > > > > While I don't think other tools still use boards.cfg, this will make it
> > > > > > > easier to find out that I'm wrong. Perhaps once the CONFIG to Kconfig
> > > > > > > migration is done we can move to just pickle'ing the data or similar
> > > > > > > since I find the main use of what was in boards.cfg can be figured out
> > > > > > > with some other git grep'ing, and in turn that's mainly for me when
> > > > > > > trying to convert stuff. Thanks for doing this.
> > > > > >
> > > > > > Yes. I'm excited to hear that Kconfig migration might be done - any
> > > > > > forecast as to when?
> > > > >
> > > > > Not yet. I'm auditing CONFIG_SYS_* now, with a notion to move
> > > > > everything that's not really configurable just out of CONFIG namespace
> > > > > as the starting point. That'll drop us down to ~500 to migrate, which
> > > > > feels a bit less daunting.
> > > > >
> > > > > > One thing we could to is provide an option for buildman to spit out
> > > > > > the various fields that go into boards.cfg
> > > > >
> > > > > Right. So I might not have said this before, but one reason I wanted
> > > > > buildman to natively know kconfiglib and have everything was that while
> > > > > we can do a lot of good matching on what to build, it would be amazingly
> > > > > good to be able to say "build every platform with NVME_PCI set" (and if
> > > > > it's not too hard hex/int options with a specific value).
> > > >
> > > > Ah OK. At present moveconfig has the functionality to list the boards
> > > > that have particular options (-b and -f). It is expensive to build the
> > > > database though - over a minute on a 32-thread machine. So we would
> > > > have to cache it. Also just about any change would invalidate the
> > > > cache and I'm not sure if it possible to detect which changes have no
> > > > effect on which cache entries...
> > >
> > > Ah, maybe it will take some more thinking about then. Maybe an
> > > "advanced" match option, and also seeing how to have Azure generate the db
> > > in one job and pass it as an artifact to every other job in the world
> > > build stage. Not an immediate need.
> >
> > Well I suppose having that logic in moveconfig doesn't make a lot of
> > sense. So we could move it to buildman and have a way of creating the
> > database, as you say. But bear in mind that every commit being built
> > has the potential to change the Kconfig. It may be possible to hash
> > the files or detect changes using timestamps.
>
> Yeah, we can revisit this later on down the line. I don't think (and we
> don't do it today either, fwiw) we need to ensure the list of boards to
> build is right for each step of the commit, and might even be
> counter-productive, at least for my use case. Build the list of boards
> on top of tree, tell me how it changed over N commits (which commit
> caused the size change) is how I use things. Then the CI case is only
> for top of tree anyhow.
OK, yes, actually that's the decision I came to for the original board
selection, i.e. the board selection comes from the current commits.
Regards,
Simon
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH 11/19] buildman: Tidy up pylint problems in boards module
2022-07-12 1:03 [PATCH 00/19] buildman: Integration of boards.cfg file Simon Glass
` (9 preceding siblings ...)
2022-07-12 1:04 ` [PATCH 10/19] buildman: Incorporate the genboardscfg.py tool Simon Glass
@ 2022-07-12 1:04 ` Simon Glass
2022-07-12 1:04 ` [PATCH 12/19] buildman: Replace the Options column with config name Simon Glass
` (8 subsequent siblings)
19 siblings, 0 replies; 31+ messages in thread
From: Simon Glass @ 2022-07-12 1:04 UTC (permalink / raw)
To: U-Boot Mailing List; +Cc: Tom Rini, Simon Glass
Fix all the pylint warnings. Also tidy up the comments so that they show
type information, as required.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
tools/buildman/boards.py | 352 +++++++++++++++++++++++----------------
1 file changed, 204 insertions(+), 148 deletions(-)
diff --git a/tools/buildman/boards.py b/tools/buildman/boards.py
index c18914253e4..e16f3268ab1 100644
--- a/tools/buildman/boards.py
+++ b/tools/buildman/boards.py
@@ -24,19 +24,26 @@ from buildman import kconfiglib
OUTPUT_FILE = 'boards.cfg'
CONFIG_DIR = 'configs'
SLEEP_TIME = 0.03
-COMMENT_BLOCK = '''#
+COMMENT_BLOCK = f'''#
# List of boards
-# Automatically generated by %s: don't edit
+# Automatically generated by {__file__}: don't edit
#
# Status, Arch, CPU, SoC, Vendor, Board, Target, Options, Maintainers
-''' % __file__
+'''
-def try_remove(f):
- """Remove a file ignoring 'No such file or directory' error."""
+def try_remove(fname):
+ """Remove a file ignoring 'No such file or directory' error.
+
+ Args:
+ fname (str): Filename to remove
+
+ Raises:
+ OSError: output file exists but could not be removed
+ """
try:
- os.remove(f)
+ os.remove(fname)
except OSError as exception:
# Ignore 'No such file or directory' error
if exception.errno != errno.ENOENT:
@@ -46,20 +53,30 @@ def try_remove(f):
def output_is_new(output):
"""Check if the output file is up to date.
+ Looks at defconfig and Kconfig files to make sure none is newer than the
+ output file. Also ensures that the boards.cfg does not mention any removed
+ boards.
+
+ Args:
+ output (str): Filename to check
+
Returns:
- True if the given output file exists and is newer than any of
- *_defconfig, MAINTAINERS and Kconfig*. False otherwise.
+ True if the given output file exists and is newer than any of
+ *_defconfig, MAINTAINERS and Kconfig*. False otherwise.
+
+ Raises:
+ OSError: output file exists but could not be opened
"""
+ # pylint: disable=too-many-branches
try:
ctime = os.path.getctime(output)
except OSError as exception:
if exception.errno == errno.ENOENT:
# return False on 'No such file or directory' error
return False
- else:
- raise
+ raise
- for (dirpath, dirnames, filenames) in os.walk(CONFIG_DIR):
+ for (dirpath, _, filenames) in os.walk(CONFIG_DIR):
for filename in fnmatch.filter(filenames, '*_defconfig'):
if fnmatch.fnmatch(filename, '.*'):
continue
@@ -67,7 +84,7 @@ def output_is_new(output):
if ctime < os.path.getctime(filepath):
return False
- for (dirpath, dirnames, filenames) in os.walk('.'):
+ for (dirpath, _, filenames) in os.walk('.'):
for filename in filenames:
if (fnmatch.fnmatch(filename, '*~') or
not fnmatch.fnmatch(filename, 'Kconfig*') and
@@ -79,8 +96,8 @@ def output_is_new(output):
# Detect a board that has been removed since the current board database
# was generated
- with open(output, encoding="utf-8") as f:
- for line in f:
+ with open(output, encoding="utf-8") as inf:
+ for line in inf:
if line[0] == '#' or line == '\n':
continue
defconfig = line.split()[6] + '_defconfig'
@@ -97,7 +114,7 @@ class Expr:
"""Set up a new Expr object.
Args:
- expr: String cotaining regular expression to store
+ expr (str): String cotaining regular expression to store
"""
self._expr = expr
self._re = re.compile(expr)
@@ -106,7 +123,7 @@ class Expr:
"""Check if any of the properties match the regular expression.
Args:
- props: List of properties to check
+ props (list of str): List of properties to check
Returns:
True if any of the properties match the regular expression
"""
@@ -132,7 +149,7 @@ class Term:
"""Add an Expr object to the list to check.
Args:
- expr: New Expr object to add to the list of those that must
+ expr (Expr): New Expr object to add to the list of those that must
match for a board to be built.
"""
self._expr_list.append(Expr(expr))
@@ -147,7 +164,7 @@ class Term:
Each of the expressions in the term is checked. All must match.
Args:
- props: List of properties to check
+ props (list of str): List of properties to check
Returns:
True if all of the expressions in the Term match, else False
"""
@@ -178,6 +195,7 @@ class KconfigScanner:
os.environ['srctree'] = os.getcwd()
os.environ['UBOOTVERSION'] = 'dummy'
os.environ['KCONFIG_OBJDIR'] = ''
+ self._tmpfile = None
self._conf = kconfiglib.Kconfig(warn=False)
def __del__(self):
@@ -188,37 +206,38 @@ class KconfigScanner:
the temporary file might be left over. In that case, it should be
deleted in this destructor.
"""
- if hasattr(self, '_tmpfile') and self._tmpfile:
+ if self._tmpfile:
try_remove(self._tmpfile)
def scan(self, defconfig):
"""Load a defconfig file to obtain board parameters.
- Arguments:
- defconfig: path to the defconfig file to be processed
+ Args:
+ defconfig (str): path to the defconfig file to be processed
Returns:
- A dictionary of board parameters. It has a form of:
- {
- 'arch': <arch_name>,
- 'cpu': <cpu_name>,
- 'soc': <soc_name>,
- 'vendor': <vendor_name>,
- 'board': <board_name>,
- 'target': <target_name>,
- 'config': <config_header_name>,
- 'options': <extra_options>
- }
+ Dictionary of board parameters. It has a form:
+ {
+ 'arch': <arch_name>,
+ 'cpu': <cpu_name>,
+ 'soc': <soc_name>,
+ 'vendor': <vendor_name>,
+ 'board': <board_name>,
+ 'target': <target_name>,
+ 'config': <config_header_name>,
+ 'options': <extra_options>
+ }
"""
# strip special prefixes and save it in a temporary file
- fd, self._tmpfile = tempfile.mkstemp()
- with os.fdopen(fd, 'w') as f:
- for line in open(defconfig):
- colon = line.find(':CONFIG_')
- if colon == -1:
- f.write(line)
- else:
- f.write(line[colon + 1:])
+ outfd, self._tmpfile = tempfile.mkstemp()
+ with os.fdopen(outfd, 'w') as outf:
+ with open(defconfig, encoding='utf-8') as inf:
+ for line in inf:
+ colon = line.find(':CONFIG_')
+ if colon == -1:
+ outf.write(line)
+ else:
+ outf.write(line[colon + 1:])
self._conf.load_config(self._tmpfile)
try_remove(self._tmpfile)
@@ -237,7 +256,7 @@ class KconfigScanner:
defconfig = os.path.basename(defconfig)
params['target'], match, rear = defconfig.partition('_defconfig')
- assert match and not rear, '%s : invalid defconfig' % defconfig
+ assert match and not rear, f'{defconfig} : invalid defconfig'
# fix-up for aarch64
if params['arch'] == 'arm' and params['cpu'] == 'armv8':
@@ -256,7 +275,15 @@ class KconfigScanner:
class MaintainersDatabase:
- """The database of board status and maintainers."""
+ """The database of board status and maintainers.
+
+ Properties:
+ database: dict:
+ key: Board-target name (e.g. 'snow')
+ value: tuple:
+ str: Board status (e.g. 'Active')
+ str: List of maintainers, separated by :
+ """
def __init__(self):
"""Create an empty database."""
@@ -269,73 +296,78 @@ class MaintainersDatabase:
Display a warning message and return '-' if status information
is not found.
+ Args:
+ target (str): Build-target name
+
Returns:
- 'Active', 'Orphan' or '-'.
+ str: 'Active', 'Orphan' or '-'.
"""
if not target in self.database:
- print("WARNING: no status info for '%s'" % target, file=sys.stderr)
+ print(f"WARNING: no status info for '{target}'", file=sys.stderr)
return '-'
tmp = self.database[target][0]
if tmp.startswith('Maintained'):
return 'Active'
- elif tmp.startswith('Supported'):
+ if tmp.startswith('Supported'):
return 'Active'
- elif tmp.startswith('Orphan'):
+ if tmp.startswith('Orphan'):
return 'Orphan'
- else:
- print(("WARNING: %s: unknown status for '%s'" %
- (tmp, target)), file=sys.stderr)
- return '-'
+ print(f"WARNING: {tmp}: unknown status for '{target}'", file=sys.stderr)
+ return '-'
def get_maintainers(self, target):
"""Return the maintainers of the given board.
+ Args:
+ target (str): Build-target name
+
Returns:
- Maintainers of the board. If the board has two or more maintainers,
- they are separated with colons.
+ str: Maintainers of the board. If the board has two or more
+ maintainers, they are separated with colons.
"""
if not target in self.database:
- print("WARNING: no maintainers for '%s'" % target, file=sys.stderr)
+ print(f"WARNING: no maintainers for '{target}'", file=sys.stderr)
return ''
return ':'.join(self.database[target][1])
- def parse_file(self, file):
+ def parse_file(self, fname):
"""Parse a MAINTAINERS file.
- Parse a MAINTAINERS file and accumulates board status and
- maintainers information.
+ Parse a MAINTAINERS file and accumulate board status and maintainers
+ information in the self.database dict.
- Arguments:
- file: MAINTAINERS file to be parsed
+ Args:
+ fname (str): MAINTAINERS file to be parsed
"""
targets = []
maintainers = []
status = '-'
- for line in open(file, encoding="utf-8"):
- # Check also commented maintainers
- if line[:3] == '#M:':
- line = line[1:]
- tag, rest = line[:2], line[2:].strip()
- if tag == 'M:':
- maintainers.append(rest)
- elif tag == 'F:':
- # expand wildcard and filter by 'configs/*_defconfig'
- for f in glob.glob(rest):
- front, match, rear = f.partition('configs/')
- if not front and match:
- front, match, rear = rear.rpartition('_defconfig')
- if match and not rear:
- targets.append(front)
- elif tag == 'S:':
- status = rest
- elif line == '\n':
- for target in targets:
- self.database[target] = (status, maintainers)
- targets = []
- maintainers = []
- status = '-'
+ with open(fname, encoding="utf-8") as inf:
+ for line in inf:
+ # Check also commented maintainers
+ if line[:3] == '#M:':
+ line = line[1:]
+ tag, rest = line[:2], line[2:].strip()
+ if tag == 'M:':
+ maintainers.append(rest)
+ elif tag == 'F:':
+ # expand wildcard and filter by 'configs/*_defconfig'
+ for item in glob.glob(rest):
+ front, match, rear = item.partition('configs/')
+ if not front and match:
+ front, match, rear = rear.rpartition('_defconfig')
+ if match and not rear:
+ targets.append(front)
+ elif tag == 'S:':
+ status = rest
+ elif line == '\n':
+ for target in targets:
+ self.database[target] = (status, maintainers)
+ targets = []
+ maintainers = []
+ status = '-'
if targets:
for target in targets:
self.database[target] = (status, maintainers)
@@ -353,7 +385,7 @@ class Boards:
The board's target member must not already exist in the board list.
Args:
- brd: board to add
+ brd (Board): board to add
"""
self._boards.append(brd)
@@ -363,7 +395,7 @@ class Boards:
Create a Board object for each and add it to our _boards list.
Args:
- fname: Filename of boards.cfg file
+ fname (str): Filename of boards.cfg file
"""
with open(fname, 'r', encoding='utf-8') as inf:
for line in inf:
@@ -452,9 +484,10 @@ class Boards:
a board to be selected.
Args:
- args: List of command line arguments
+ args (list of str): List of command line arguments
+
Returns:
- A list of Term objects
+ list of Term: A list of Term objects
"""
syms = []
for arg in args:
@@ -493,11 +526,12 @@ class Boards:
If brds and args are both empty, all boards are selected.
Args:
- args: List of strings specifying boards to include, either named,
- or by their target, architecture, cpu, vendor or soc. If
- empty, all boards are selected.
- exclude: List of boards to exclude, regardless of 'args'
- brds: List of boards to build
+ args (list of str): List of strings specifying boards to include,
+ either named, or by their target, architecture, cpu, vendor or
+ soc. If empty, all boards are selected.
+ exclude (list of str): List of boards to exclude, regardless of
+ 'args', or None for none
+ brds (list of Board): List of boards to build, or None/[] for all
Returns:
Tuple
@@ -505,21 +539,21 @@ class Boards:
due to each argument, arranged by argument.
List of errors found
"""
- result = OrderedDict()
- warnings = []
- terms = self._build_terms(args)
+ def _check_board(brd):
+ """Check whether to include or exclude a board
- result['all'] = []
- for term in terms:
- result[str(term)] = []
+ Checks the various terms and decide whether to build it or not (the
+ 'build_it' variable).
- exclude_list = []
- if exclude:
- for expr in exclude:
- exclude_list.append(Expr(expr))
+ If it is built, add the board to the result[term] list so we know
+ which term caused it to be built. Add it to result['all'] also.
- found = []
- for brd in self._boards:
+ Keep a list of boards we found in 'found', so we can report boards
+ which appear in self._boards but not in brds.
+
+ Args:
+ brd (Board): Board to check
+ """
matching_term = None
build_it = False
if terms:
@@ -547,6 +581,23 @@ class Boards:
result[matching_term].append(brd.target)
result['all'].append(brd.target)
+ result = OrderedDict()
+ warnings = []
+ terms = self._build_terms(args)
+
+ result['all'] = []
+ for term in terms:
+ result[str(term)] = []
+
+ exclude_list = []
+ if exclude:
+ for expr in exclude:
+ exclude_list.append(Expr(expr))
+
+ found = []
+ for brd in self._boards:
+ _check_board(brd)
+
if brds:
remaining = set(brds) - set(found)
if remaining:
@@ -554,37 +605,40 @@ class Boards:
return result, warnings
- def scan_defconfigs_for_multiprocess(self, queue, defconfigs):
+ @classmethod
+ def scan_defconfigs_for_multiprocess(cls, queue, defconfigs):
"""Scan defconfig files and queue their board parameters
- This function is intended to be passed to
- multiprocessing.Process() constructor.
+ This function is intended to be passed to multiprocessing.Process()
+ constructor.
- Arguments:
- queue: An instance of multiprocessing.Queue().
- The resulting board parameters are written into it.
- defconfigs: A sequence of defconfig files to be scanned.
+ Args:
+ queue (multiprocessing.Queue): The resulting board parameters are
+ written into this.
+ defconfigs (sequence of str): A sequence of defconfig files to be
+ scanned.
"""
kconf_scanner = KconfigScanner()
for defconfig in defconfigs:
queue.put(kconf_scanner.scan(defconfig))
- def read_queues(self, queues, params_list):
+ @classmethod
+ def read_queues(cls, queues, params_list):
"""Read the queues and append the data to the paramers list"""
- for q in queues:
- while not q.empty():
- params_list.append(q.get())
+ for que in queues:
+ while not que.empty():
+ params_list.append(que.get())
def scan_defconfigs(self, jobs=1):
"""Collect board parameters for all defconfig files.
This function invokes multiple processes for faster processing.
- Arguments:
- jobs: The number of jobs to run simultaneously
+ Args:
+ jobs (int): The number of jobs to run simultaneously
"""
all_defconfigs = []
- for (dirpath, dirnames, filenames) in os.walk(CONFIG_DIR):
+ for (dirpath, _, filenames) in os.walk(CONFIG_DIR):
for filename in fnmatch.filter(filenames, '*_defconfig'):
if fnmatch.fnmatch(filename, '.*'):
continue
@@ -596,42 +650,43 @@ class Boards:
for i in range(jobs):
defconfigs = all_defconfigs[total_boards * i // jobs :
total_boards * (i + 1) // jobs]
- q = multiprocessing.Queue(maxsize=-1)
- p = multiprocessing.Process(
+ que = multiprocessing.Queue(maxsize=-1)
+ proc = multiprocessing.Process(
target=self.scan_defconfigs_for_multiprocess,
- args=(q, defconfigs))
- p.start()
- processes.append(p)
- queues.append(q)
+ args=(que, defconfigs))
+ proc.start()
+ processes.append(proc)
+ queues.append(que)
# The resulting data should be accumulated to this list
params_list = []
# Data in the queues should be retrieved preriodically.
# Otherwise, the queues would become full and subprocesses would get stuck.
- while any([p.is_alive() for p in processes]):
+ while any(p.is_alive() for p in processes):
self.read_queues(queues, params_list)
# sleep for a while until the queues are filled
time.sleep(SLEEP_TIME)
# Joining subprocesses just in case
# (All subprocesses should already have been finished)
- for p in processes:
- p.join()
+ for proc in processes:
+ proc.join()
# retrieve leftover data
self.read_queues(queues, params_list)
return params_list
- def insert_maintainers_info(self, params_list):
+ @classmethod
+ def insert_maintainers_info(cls, params_list):
"""Add Status and Maintainers information to the board parameters list.
- Arguments:
- params_list: A list of the board parameters
+ Args:
+ params_list (list of dict): A list of the board parameters
"""
database = MaintainersDatabase()
- for (dirpath, dirnames, filenames) in os.walk('.'):
+ for (dirpath, _, filenames) in os.walk('.'):
if 'MAINTAINERS' in filenames:
database.parse_file(os.path.join(dirpath, 'MAINTAINERS'))
@@ -641,51 +696,52 @@ class Boards:
params['maintainers'] = database.get_maintainers(target)
params_list[i] = params
- def format_and_output(self, params_list, output):
+ @classmethod
+ def format_and_output(cls, params_list, output):
"""Write board parameters into a file.
Columnate the board parameters, sort lines alphabetically,
and then write them to a file.
- Arguments:
- params_list: The list of board parameters
- output: The path to the output file
+ Args:
+ params_list (list of dict): The list of board parameters
+ output (str): The path to the output file
"""
- FIELDS = ('status', 'arch', 'cpu', 'soc', 'vendor', 'board', 'target',
- 'options', 'maintainers')
+ fields = ('status', 'arch', 'cpu', 'soc', 'vendor', 'board', 'target',
+ 'options', 'maintainers')
# First, decide the width of each column
- max_length = dict([ (f, 0) for f in FIELDS])
+ max_length = {f: 0 for f in fields}
for params in params_list:
- for f in FIELDS:
- max_length[f] = max(max_length[f], len(params[f]))
+ for field in fields:
+ max_length[field] = max(max_length[field], len(params[field]))
output_lines = []
for params in params_list:
line = ''
- for f in FIELDS:
+ for field in fields:
# insert two spaces between fields like column -t would
- line += ' ' + params[f].ljust(max_length[f])
+ line += ' ' + params[field].ljust(max_length[field])
output_lines.append(line.strip())
# ignore case when sorting
output_lines.sort(key=str.lower)
- with open(output, 'w', encoding="utf-8") as f:
- f.write(COMMENT_BLOCK + '\n'.join(output_lines) + '\n')
+ with open(output, 'w', encoding="utf-8") as outf:
+ outf.write(COMMENT_BLOCK + '\n'.join(output_lines) + '\n')
def ensure_board_list(self, output, jobs=1, force=False, quiet=False):
"""Generate a board database file if needed.
- Arguments:
- output: The name of the output file
- jobs: The number of jobs to run simultaneously
- force: Force to generate the output even if it is new
- quiet: True to avoid printing a message if nothing needs doing
+ Args:
+ output (str): The name of the output file
+ jobs (int): The number of jobs to run simultaneously
+ force (bool): Force to generate the output even if it is new
+ quiet (bool): True to avoid printing a message if nothing needs doing
"""
if not force and output_is_new(output):
if not quiet:
- print("%s is up to date. Nothing to do." % output)
+ print(f'{output} is up to date. Nothing to do.')
return
params_list = self.scan_defconfigs(jobs)
self.insert_maintainers_info(params_list)
--
2.37.0.144.g8ac04bfd2-goog
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 12/19] buildman: Replace the Options column with config name
2022-07-12 1:03 [PATCH 00/19] buildman: Integration of boards.cfg file Simon Glass
` (10 preceding siblings ...)
2022-07-12 1:04 ` [PATCH 11/19] buildman: Tidy up pylint problems in boards module Simon Glass
@ 2022-07-12 1:04 ` Simon Glass
2022-07-12 1:04 ` [PATCH 13/19] Revert "Revert "global: Remove CONFIG_SYS_EXTRA_OPTIONS support"" Simon Glass
` (7 subsequent siblings)
19 siblings, 0 replies; 31+ messages in thread
From: Simon Glass @ 2022-07-12 1:04 UTC (permalink / raw)
To: U-Boot Mailing List; +Cc: Tom Rini, Simon Glass
This appears in boards.cfg but we want to remove it. Drop support for
generating it and reading it. Detect an old boards.cfg file that has
this field and regenerate it, to avoid problems.
Instead, add the config name in that place. This fixes a subtle bug in
the generation code, since it uses 'target' for the config name and then
overwrites the value in scan() by setting params['target'] to the name
of the defconfig. The defconfig name is not the same as the
SYS_CONFIG_NAME variable.
With this change, we still have the config name and it can be searched
by buildman, e.g. with:
buildman -nv sun5i
Signed-off-by: Simon Glass <sjg@chromium.org>
Reported-by: Tom Rini <trini@konsulko.com>
---
tools/buildman/board.py | 8 ++++----
tools/buildman/boards.py | 19 ++++++-------------
2 files changed, 10 insertions(+), 17 deletions(-)
diff --git a/tools/buildman/board.py b/tools/buildman/board.py
index 3268b39e356..8ef905b8ce1 100644
--- a/tools/buildman/board.py
+++ b/tools/buildman/board.py
@@ -6,7 +6,7 @@
class Board:
"""A particular board that we can build"""
- def __init__(self, status, arch, cpu, soc, vendor, board_name, target, options):
+ def __init__(self, status, arch, cpu, soc, vendor, board_name, target, cfg_name):
"""Create a new board type.
Args:
@@ -17,7 +17,7 @@ class Board:
vendor: Name of vendor (e.g. armltd)
board_name: Name of board (e.g. integrator)
target: Target name (use make <target>_defconfig to configure)
- options: board-specific options (e.g. integratorcp:CM1136)
+ cfg_name: Config name
"""
self.target = target
self.arch = arch
@@ -25,7 +25,7 @@ class Board:
self.board_name = board_name
self.vendor = vendor
self.soc = soc
- self.options = options
+ self.cfg_name = cfg_name
self.props = [self.target, self.arch, self.cpu, self.board_name,
- self.vendor, self.soc, self.options]
+ self.vendor, self.soc, self.cfg_name]
self.build_it = False
diff --git a/tools/buildman/boards.py b/tools/buildman/boards.py
index e16f3268ab1..8832e40cd5d 100644
--- a/tools/buildman/boards.py
+++ b/tools/buildman/boards.py
@@ -28,7 +28,7 @@ COMMENT_BLOCK = f'''#
# List of boards
# Automatically generated by {__file__}: don't edit
#
-# Status, Arch, CPU, SoC, Vendor, Board, Target, Options, Maintainers
+# Status, Arch, CPU, SoC, Vendor, Board, Target, Config, Maintainers
'''
@@ -98,6 +98,8 @@ def output_is_new(output):
# was generated
with open(output, encoding="utf-8") as inf:
for line in inf:
+ if 'Options,' in line:
+ return False
if line[0] == '#' or line == '\n':
continue
defconfig = line.split()[6] + '_defconfig'
@@ -186,7 +188,7 @@ class KconfigScanner:
'vendor' : 'SYS_VENDOR',
'board' : 'SYS_BOARD',
'config' : 'SYS_CONFIG_NAME',
- 'options' : 'SYS_EXTRA_OPTIONS'
+ # 'target' is added later
}
def __init__(self):
@@ -216,7 +218,7 @@ class KconfigScanner:
defconfig (str): path to the defconfig file to be processed
Returns:
- Dictionary of board parameters. It has a form:
+ A dictionary of board parameters. It has a form of:
{
'arch': <arch_name>,
'cpu': <cpu_name>,
@@ -225,7 +227,6 @@ class KconfigScanner:
'board': <board_name>,
'target': <target_name>,
'config': <config_header_name>,
- 'options': <extra_options>
}
"""
# strip special prefixes and save it in a temporary file
@@ -262,14 +263,6 @@ class KconfigScanner:
if params['arch'] == 'arm' and params['cpu'] == 'armv8':
params['arch'] = 'aarch64'
- # fix-up options field. It should have the form:
- # <config name>[:comma separated config options]
- if params['options'] != '-':
- params['options'] = params['config'] + ':' + \
- params['options'].replace(r'\"', '"')
- elif params['config'] != params['target']:
- params['options'] = params['config']
-
return params
@@ -708,7 +701,7 @@ class Boards:
output (str): The path to the output file
"""
fields = ('status', 'arch', 'cpu', 'soc', 'vendor', 'board', 'target',
- 'options', 'maintainers')
+ 'config', 'maintainers')
# First, decide the width of each column
max_length = {f: 0 for f in fields}
--
2.37.0.144.g8ac04bfd2-goog
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 13/19] Revert "Revert "global: Remove CONFIG_SYS_EXTRA_OPTIONS support""
2022-07-12 1:03 [PATCH 00/19] buildman: Integration of boards.cfg file Simon Glass
` (11 preceding siblings ...)
2022-07-12 1:04 ` [PATCH 12/19] buildman: Replace the Options column with config name Simon Glass
@ 2022-07-12 1:04 ` Simon Glass
2022-07-12 21:38 ` Tom Rini
2022-07-12 1:04 ` [PATCH 14/19] buildman: Return an error if there are maintainer warnings Simon Glass
` (6 subsequent siblings)
19 siblings, 1 reply; 31+ messages in thread
From: Simon Glass @ 2022-07-12 1:04 UTC (permalink / raw)
To: U-Boot Mailing List
Cc: Tom Rini, Simon Glass, Alexandru Gagniuc, Alper Nebi Yasak,
Angelo Dureghello, Heinrich Schuchardt, Patrick Delaunay,
Steffen Jaeckel
This is not needed now that CONFIG_SYS_TARGET_NAME is correctly determined
when scanning Kconfig.
This reverts commit 25b8acee2ea11a9edc100c42a61f5d6187eb6167.
Signed-off-by: Simon Glass <sjg@chromium.org>
Suggested-by: Tom Rini <trini@konsulko.com>
---
boot/Kconfig | 13 --------
doc/README.kconfig | 7 ----
doc/develop/moveconfig.rst | 3 +-
scripts/Makefile.autoconf | 4 ---
scripts/build-whitelist.sh | 23 ++------------
tools/moveconfig.py | 65 --------------------------------------
6 files changed, 4 insertions(+), 111 deletions(-)
diff --git a/boot/Kconfig b/boot/Kconfig
index 17438b566d5..37880af5519 100644
--- a/boot/Kconfig
+++ b/boot/Kconfig
@@ -452,19 +452,6 @@ config OF_STDOUT_VIA_ALIAS
incorrect when used with device tree as this option does not
exist / should not be used.
-config SYS_EXTRA_OPTIONS
- string "Extra Options (DEPRECATED)"
- help
- The old configuration infrastructure (= mkconfig + boards.cfg)
- provided the extra options field. If you have something like
- "HAS_BAR,BAZ=64", the optional options
- #define CONFIG_HAS
- #define CONFIG_BAZ 64
- will be defined in include/config.h.
- This option was prepared for the smooth migration from the old
- configuration to Kconfig. Since this option will be removed sometime,
- new boards should not use this option.
-
config HAVE_SYS_TEXT_BASE
bool
depends on !NIOS2 && !XTENSA
diff --git a/doc/README.kconfig b/doc/README.kconfig
index 0689f66c2cd..808cf56e59c 100644
--- a/doc/README.kconfig
+++ b/doc/README.kconfig
@@ -99,7 +99,6 @@ Kconfig. Each field of boards.cfg was converted as follows:
Vendor -> CONFIG_SYS_VENDOR defined by Kconfig
Board -> CONFIG_SYS_BOARD defined by Kconfig
Target -> File name of defconfig (configs/<target>_defconfig)
- Options -> CONFIG_SYS_EXTRA_OPTIONS defined by Kconfig
Maintainers -> "M:" entry of MAINTAINERS
@@ -140,12 +139,6 @@ When removing an obsolete board, the following steps are generally needed:
TODO
----
-- The option field of boards.cfg, which was used for the pre-Kconfig
- configuration, moved to CONFIG_SYS_EXTRA_OPTIONS verbatim now.
- Board maintainers are expected to implement proper Kconfig options
- and switch over to them. Eventually CONFIG_SYS_EXTRA_OPTIONS will go away.
- CONFIG_SYS_EXTRA_OPTIONS should not be used for new boards.
-
- In the pre-Kconfig, a single board had multiple entries in the boards.cfg
file with differences in the option fields. The corresponding defconfig
files were auto-generated when switching to Kconfig. Now we have too many
diff --git a/doc/develop/moveconfig.rst b/doc/develop/moveconfig.rst
index 2f53ea52b71..bfb7aff3582 100644
--- a/doc/develop/moveconfig.rst
+++ b/doc/develop/moveconfig.rst
@@ -295,8 +295,7 @@ Available options
-y, --yes
Instead of prompting, automatically go ahead with all operations. This
- includes cleaning up headers, CONFIG_SYS_EXTRA_OPTIONS, the config whitelist
- and the README.
+ includes cleaning up headers, the config whitelist and the README.
To see the complete list of supported options, run::
diff --git a/scripts/Makefile.autoconf b/scripts/Makefile.autoconf
index 3fa4d50f1ea..5a4a1489553 100644
--- a/scripts/Makefile.autoconf
+++ b/scripts/Makefile.autoconf
@@ -112,10 +112,6 @@ vpl/include/autoconf.mk: vpl/u-boot.cfg
# Prior to Kconfig, it was generated by mkconfig. Now it is created here.
define filechk_config_h
(echo "/* Automatically generated - do not edit */"; \
- for i in $$(echo $(CONFIG_SYS_EXTRA_OPTIONS) | sed 's/,/ /g'); do \
- echo \#define CONFIG_$$i \
- | sed '/=/ {s/=/ /;q; } ; { s/$$/ 1/; }'; \
- done; \
echo \#define CONFIG_BOARDDIR board/$(if $(VENDOR),$(VENDOR)/)$(BOARD);\
echo \#include \<config_uncmd_spl.h\>; \
echo \#include \<configs/$(CONFIG_SYS_CONFIG_NAME).h\>; \
diff --git a/scripts/build-whitelist.sh b/scripts/build-whitelist.sh
index 6feb9b67cf5..37630c0271c 100755
--- a/scripts/build-whitelist.sh
+++ b/scripts/build-whitelist.sh
@@ -10,30 +10,13 @@
#
export LC_ALL=C LC_COLLATE=C
-# There are two independent greps. The first pulls out the component parts
-# of CONFIG_SYS_EXTRA_OPTIONS. An example is:
+# Looks for the rest of the CONFIG options, but exclude those in Kconfig and
+# defconfig files.
#
-# SUN7I_GMAC,AHCI,SATAPWR=SUNXI_GPB(8)
-#
-# We want this to produce:
-# CONFIG_SUN7I_GMAC
-# CONFIG_AHCI
-# CONFIG_SATAPWR
-#
-# The second looks for the rest of the CONFIG options, but excludes those in
-# Kconfig and defconfig files.
-#
-(
-git grep CONFIG_SYS_EXTRA_OPTIONS |sed -n \
- 's/.*CONFIG_SYS_EXTRA_OPTIONS="\(.*\)"/\1/ p' \
- | tr , '\n' \
- | sed 's/ *\([A-Za-z0-9_]*\).*/CONFIG_\1/'
-
git grep CONFIG_ | \
egrep -vi "(Kconfig:|defconfig:|README|\.py|\.pl:)" \
| tr ' \t' '\n\n' \
- | sed -n 's/^\(CONFIG_[A-Za-z0-9_]*\).*/\1/p'
-) \
+ | sed -n 's/^\(CONFIG_[A-Za-z0-9_]*\).*/\1/p' \
|sort |uniq >scripts/config_whitelist.txt.tmp1;
# Finally, we need a list of the valid Kconfig options to exclude these from
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 84bc875fff8..09617a07f91 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -443,70 +443,6 @@ def cleanup_headers(configs, args):
cleanup_one_header(header_path, patterns, args)
cleanup_empty_blocks(header_path, args)
-def cleanup_one_extra_option(defconfig_path, configs, args):
- """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in one defconfig file.
-
- Args:
- defconfig_path: path to the cleaned defconfig file.
- configs: A list of CONFIGs to remove.
- args (Namespace): program arguments
- """
-
- start = 'CONFIG_SYS_EXTRA_OPTIONS="'
- end = '"'
-
- lines = read_file(defconfig_path)
-
- for i, line in enumerate(lines):
- if line.startswith(start) and line.endswith(end):
- break
- else:
- # CONFIG_SYS_EXTRA_OPTIONS was not found in this defconfig
- return
-
- old_tokens = line[len(start):-len(end)].split(',')
- new_tokens = []
-
- for token in old_tokens:
- pos = token.find('=')
- if not (token[:pos] if pos >= 0 else token) in configs:
- new_tokens.append(token)
-
- if new_tokens == old_tokens:
- return
-
- tolines = copy.copy(lines)
-
- if new_tokens:
- tolines[i] = start + ','.join(new_tokens) + end
- else:
- tolines.pop(i)
-
- show_diff(lines, tolines, defconfig_path, args.color)
-
- if args.dry_run:
- return
-
- write_file(defconfig_path, tolines)
-
-def cleanup_extra_options(configs, args):
- """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in defconfig files.
-
- Args:
- configs: A list of CONFIGs to remove.
- args (Namespace): program arguments
- """
- if not confirm(args, 'Clean up CONFIG_SYS_EXTRA_OPTIONS?'):
- return
-
- configs = [ config[len('CONFIG_'):] for config in configs ]
-
- defconfigs = get_all_defconfigs()
-
- for defconfig in defconfigs:
- cleanup_one_extra_option(os.path.join('configs', defconfig), configs,
- args)
-
def cleanup_whitelist(configs, args):
"""Delete config whitelist entries
@@ -1803,7 +1739,6 @@ doc/develop/moveconfig.rst for documentation.'''
if configs:
cleanup_headers(configs, args)
- cleanup_extra_options(configs, args)
cleanup_whitelist(configs, args)
cleanup_readme(configs, args)
--
2.37.0.144.g8ac04bfd2-goog
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [PATCH 13/19] Revert "Revert "global: Remove CONFIG_SYS_EXTRA_OPTIONS support""
2022-07-12 1:04 ` [PATCH 13/19] Revert "Revert "global: Remove CONFIG_SYS_EXTRA_OPTIONS support"" Simon Glass
@ 2022-07-12 21:38 ` Tom Rini
0 siblings, 0 replies; 31+ messages in thread
From: Tom Rini @ 2022-07-12 21:38 UTC (permalink / raw)
To: Simon Glass
Cc: U-Boot Mailing List, Alexandru Gagniuc, Alper Nebi Yasak,
Angelo Dureghello, Heinrich Schuchardt, Patrick Delaunay,
Steffen Jaeckel
[-- Attachment #1: Type: text/plain, Size: 395 bytes --]
On Mon, Jul 11, 2022 at 07:04:07PM -0600, Simon Glass wrote:
> This is not needed now that CONFIG_SYS_TARGET_NAME is correctly determined
> when scanning Kconfig.
>
> This reverts commit 25b8acee2ea11a9edc100c42a61f5d6187eb6167.
>
> Signed-off-by: Simon Glass <sjg@chromium.org>
> Suggested-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Tom Rini <trini@konsulko.com>
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH 14/19] buildman: Return an error if there are maintainer warnings
2022-07-12 1:03 [PATCH 00/19] buildman: Integration of boards.cfg file Simon Glass
` (12 preceding siblings ...)
2022-07-12 1:04 ` [PATCH 13/19] Revert "Revert "global: Remove CONFIG_SYS_EXTRA_OPTIONS support"" Simon Glass
@ 2022-07-12 1:04 ` Simon Glass
2022-07-12 1:04 ` [PATCH 15/19] gitlab/azure: Use buildman instead of genboardscfg Simon Glass
` (5 subsequent siblings)
19 siblings, 0 replies; 31+ messages in thread
From: Simon Glass @ 2022-07-12 1:04 UTC (permalink / raw)
To: U-Boot Mailing List; +Cc: Tom Rini, Simon Glass
Detect warnings about missing maintain info and return result code 2 in
that case.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
tools/buildman/README | 3 ++-
tools/buildman/boards.py | 22 +++++++++++++++++-----
tools/buildman/control.py | 10 +++++-----
3 files changed, 24 insertions(+), 11 deletions(-)
diff --git a/tools/buildman/README b/tools/buildman/README
index 0666bc7d564..3ba08d0dace 100644
--- a/tools/buildman/README
+++ b/tools/buildman/README
@@ -1314,7 +1314,8 @@ Using boards.cfg
This file is no-longer needed by buildman but it is still generated in the
working directory. This helps avoid a delay on every build, since scanning all
the Kconfig files takes a few seconds. Use the -R flag to force regeneration
-of the file - in that case buildman exits after writing the file.
+of the file - in that case buildman exits after writing the file. with exit code
+2 if there was an error in the maintainer files.
You should use 'buildman -nv <criteria>' instead of greoing the boards.cfg file,
since it may be dropped altogether in future.
diff --git a/tools/buildman/boards.py b/tools/buildman/boards.py
index 8832e40cd5d..b30b344bc8e 100644
--- a/tools/buildman/boards.py
+++ b/tools/buildman/boards.py
@@ -276,11 +276,13 @@ class MaintainersDatabase:
value: tuple:
str: Board status (e.g. 'Active')
str: List of maintainers, separated by :
+ warnings (list of str): List of warnings due to missing status, etc.
"""
def __init__(self):
"""Create an empty database."""
self.database = {}
+ self.warnings = []
def get_status(self, target):
"""Return the status of the given board.
@@ -296,7 +298,7 @@ class MaintainersDatabase:
str: 'Active', 'Orphan' or '-'.
"""
if not target in self.database:
- print(f"WARNING: no status info for '{target}'", file=sys.stderr)
+ self.warnings.append(f"WARNING: no status info for '{target}'")
return '-'
tmp = self.database[target][0]
@@ -306,7 +308,7 @@ class MaintainersDatabase:
return 'Active'
if tmp.startswith('Orphan'):
return 'Orphan'
- print(f"WARNING: {tmp}: unknown status for '{target}'", file=sys.stderr)
+ self.warnings.append(f"WARNING: {tmp}: unknown status for '{target}'")
return '-'
def get_maintainers(self, target):
@@ -320,7 +322,7 @@ class MaintainersDatabase:
maintainers, they are separated with colons.
"""
if not target in self.database:
- print(f"WARNING: no maintainers for '{target}'", file=sys.stderr)
+ self.warnings.append(f"WARNING: no maintainers for '{target}'")
return ''
return ':'.join(self.database[target][1])
@@ -677,6 +679,9 @@ class Boards:
Args:
params_list (list of dict): A list of the board parameters
+
+ Returns:
+ list of str: List of warnings collected due to missing status, etc.
"""
database = MaintainersDatabase()
for (dirpath, _, filenames) in os.walk('.'):
@@ -688,6 +693,7 @@ class Boards:
params['status'] = database.get_status(target)
params['maintainers'] = database.get_maintainers(target)
params_list[i] = params
+ return database.warnings
@classmethod
def format_and_output(cls, params_list, output):
@@ -731,11 +737,17 @@ class Boards:
jobs (int): The number of jobs to run simultaneously
force (bool): Force to generate the output even if it is new
quiet (bool): True to avoid printing a message if nothing needs doing
+
+ Returns:
+ bool: True if all is well, False if there were warnings
"""
if not force and output_is_new(output):
if not quiet:
print(f'{output} is up to date. Nothing to do.')
- return
+ return True
params_list = self.scan_defconfigs(jobs)
- self.insert_maintainers_info(params_list)
+ warnings = self.insert_maintainers_info(params_list)
+ for warn in warnings:
+ print(warn, file=sys.stderr)
self.format_and_output(params_list, output)
+ return not warnings
diff --git a/tools/buildman/control.py b/tools/buildman/control.py
index 79ce2f6978a..0c75466fbd3 100644
--- a/tools/buildman/control.py
+++ b/tools/buildman/control.py
@@ -188,12 +188,12 @@ def DoBuildman(options, args, toolchains=None, make_func=None, brds=None,
board_file = os.path.join(options.output_dir, 'boards.cfg')
brds = boards.Boards()
- brds.ensure_board_list(board_file,
- options.threads or multiprocessing.cpu_count(),
- force=options.regen_board_list,
- quiet=not options.verbose)
+ ok = brds.ensure_board_list(board_file,
+ options.threads or multiprocessing.cpu_count(),
+ force=options.regen_board_list,
+ quiet=not options.verbose)
if options.regen_board_list:
- return 0
+ return 0 if ok else 2
brds.read_boards(board_file)
exclude = []
--
2.37.0.144.g8ac04bfd2-goog
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 15/19] gitlab/azure: Use buildman instead of genboardscfg
2022-07-12 1:03 [PATCH 00/19] buildman: Integration of boards.cfg file Simon Glass
` (13 preceding siblings ...)
2022-07-12 1:04 ` [PATCH 14/19] buildman: Return an error if there are maintainer warnings Simon Glass
@ 2022-07-12 1:04 ` Simon Glass
2022-07-12 12:06 ` Tom Rini
2022-07-12 1:04 ` [PATCH 16/19] Drop genboardscfg.py Simon Glass
` (4 subsequent siblings)
19 siblings, 1 reply; 31+ messages in thread
From: Simon Glass @ 2022-07-12 1:04 UTC (permalink / raw)
To: U-Boot Mailing List; +Cc: Tom Rini, Simon Glass, AKASHI Takahiro, Bin Meng
Use the equivalent buildman functionality to check maintainer info.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
.azure-pipelines.yml | 2 +-
.gitlab-ci.yml | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml
index bc2b437bd99..36ca3cb4626 100644
--- a/.azure-pipelines.yml
+++ b/.azure-pipelines.yml
@@ -140,7 +140,7 @@ stages:
options: $(container_option)
steps:
- script: |
- if [ `./tools/genboardscfg.py -f 2>&1 | wc -l` -ne 0 ]; then exit 1; fi
+ ./tools/buildman/buildman -R
- job: tools_only
displayName: 'Ensure host tools build'
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index f9cd4175079..a516c5d4ee0 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -180,7 +180,7 @@ sloccount:
Check for configs without MAINTAINERS entry:
stage: testsuites
script:
- - if [ `./tools/genboardscfg.py -f 2>&1 | wc -l` -ne 0 ]; then exit 1; fi
+ - ./tools/buildman/buildman -R
# Ensure host tools build
Build tools-only:
--
2.37.0.144.g8ac04bfd2-goog
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 16/19] Drop genboardscfg.py
2022-07-12 1:03 [PATCH 00/19] buildman: Integration of boards.cfg file Simon Glass
` (14 preceding siblings ...)
2022-07-12 1:04 ` [PATCH 15/19] gitlab/azure: Use buildman instead of genboardscfg Simon Glass
@ 2022-07-12 1:04 ` Simon Glass
2022-07-12 1:04 ` [PATCH 17/19] buildman: Allow lines without a symbol Simon Glass
` (3 subsequent siblings)
19 siblings, 0 replies; 31+ messages in thread
From: Simon Glass @ 2022-07-12 1:04 UTC (permalink / raw)
To: U-Boot Mailing List
Cc: Tom Rini, Simon Glass, Andre Przywara, Heinrich Schuchardt, Jan Kiszka
Now that buildman can generate this with the -R option, drop the script.
Signed-off-by: Simon Glass <sjg@chromium.org>
Suggested-by: Tom Rini <trini@konsulko.com>
---
scripts/pylint.base | 1 -
tools/genboardscfg.py | 444 ------------------------------------------
2 files changed, 445 deletions(-)
delete mode 100755 tools/genboardscfg.py
diff --git a/scripts/pylint.base b/scripts/pylint.base
index 9ebebf47ab9..dd6360e6256 100644
--- a/scripts/pylint.base
+++ b/scripts/pylint.base
@@ -201,7 +201,6 @@ tools_dtoc_test_fdt 6.88
tools_dtoc_test_src_scan 9.43
tools_efivar 6.71
tools_endian-swap 9.29
-tools_genboardscfg 7.95
tools_microcode-tool 7.25
tools_moveconfig 8.34
tools_patman___init__ 0.00
diff --git a/tools/genboardscfg.py b/tools/genboardscfg.py
deleted file mode 100755
index 07bf681d1d9..00000000000
--- a/tools/genboardscfg.py
+++ /dev/null
@@ -1,444 +0,0 @@
-#!/usr/bin/env python3
-# SPDX-License-Identifier: GPL-2.0+
-#
-# Author: Masahiro Yamada <yamada.m@jp.panasonic.com>
-#
-
-"""
-Converter from Kconfig and MAINTAINERS to a board database.
-
-Run 'tools/genboardscfg.py' to create a board database.
-
-Run 'tools/genboardscfg.py -h' for available options.
-"""
-
-import errno
-import fnmatch
-import glob
-import multiprocessing
-import optparse
-import os
-import sys
-import tempfile
-import time
-
-from buildman import kconfiglib
-
-### constant variables ###
-OUTPUT_FILE = 'boards.cfg'
-CONFIG_DIR = 'configs'
-SLEEP_TIME = 0.03
-COMMENT_BLOCK = '''#
-# List of boards
-# Automatically generated by %s: don't edit
-#
-# Status, Arch, CPU, SoC, Vendor, Board, Target, Options, Maintainers
-
-''' % __file__
-
-### helper functions ###
-def try_remove(f):
- """Remove a file ignoring 'No such file or directory' error."""
- try:
- os.remove(f)
- except OSError as exception:
- # Ignore 'No such file or directory' error
- if exception.errno != errno.ENOENT:
- raise
-
-def check_top_directory():
- """Exit if we are not at the top of source directory."""
- for f in ('README', 'Licenses'):
- if not os.path.exists(f):
- sys.exit('Please run at the top of source directory.')
-
-def output_is_new(output):
- """Check if the output file is up to date.
-
- Returns:
- True if the given output file exists and is newer than any of
- *_defconfig, MAINTAINERS and Kconfig*. False otherwise.
- """
- try:
- ctime = os.path.getctime(output)
- except OSError as exception:
- if exception.errno == errno.ENOENT:
- # return False on 'No such file or directory' error
- return False
- else:
- raise
-
- for (dirpath, dirnames, filenames) in os.walk(CONFIG_DIR):
- for filename in fnmatch.filter(filenames, '*_defconfig'):
- if fnmatch.fnmatch(filename, '.*'):
- continue
- filepath = os.path.join(dirpath, filename)
- if ctime < os.path.getctime(filepath):
- return False
-
- for (dirpath, dirnames, filenames) in os.walk('.'):
- for filename in filenames:
- if (fnmatch.fnmatch(filename, '*~') or
- not fnmatch.fnmatch(filename, 'Kconfig*') and
- not filename == 'MAINTAINERS'):
- continue
- filepath = os.path.join(dirpath, filename)
- if ctime < os.path.getctime(filepath):
- return False
-
- # Detect a board that has been removed since the current board database
- # was generated
- with open(output, encoding="utf-8") as f:
- for line in f:
- if line[0] == '#' or line == '\n':
- continue
- defconfig = line.split()[6] + '_defconfig'
- if not os.path.exists(os.path.join(CONFIG_DIR, defconfig)):
- return False
-
- return True
-
-### classes ###
-class KconfigScanner:
-
- """Kconfig scanner."""
-
- ### constant variable only used in this class ###
- _SYMBOL_TABLE = {
- 'arch' : 'SYS_ARCH',
- 'cpu' : 'SYS_CPU',
- 'soc' : 'SYS_SOC',
- 'vendor' : 'SYS_VENDOR',
- 'board' : 'SYS_BOARD',
- 'config' : 'SYS_CONFIG_NAME',
- 'options' : 'SYS_EXTRA_OPTIONS'
- }
-
- def __init__(self):
- """Scan all the Kconfig files and create a Kconfig object."""
- # Define environment variables referenced from Kconfig
- os.environ['srctree'] = os.getcwd()
- os.environ['UBOOTVERSION'] = 'dummy'
- os.environ['KCONFIG_OBJDIR'] = ''
- self._conf = kconfiglib.Kconfig(warn=False)
-
- def __del__(self):
- """Delete a leftover temporary file before exit.
-
- The scan() method of this class creates a temporay file and deletes
- it on success. If scan() method throws an exception on the way,
- the temporary file might be left over. In that case, it should be
- deleted in this destructor.
- """
- if hasattr(self, '_tmpfile') and self._tmpfile:
- try_remove(self._tmpfile)
-
- def scan(self, defconfig):
- """Load a defconfig file to obtain board parameters.
-
- Arguments:
- defconfig: path to the defconfig file to be processed
-
- Returns:
- A dictionary of board parameters. It has a form of:
- {
- 'arch': <arch_name>,
- 'cpu': <cpu_name>,
- 'soc': <soc_name>,
- 'vendor': <vendor_name>,
- 'board': <board_name>,
- 'target': <target_name>,
- 'config': <config_header_name>,
- 'options': <extra_options>
- }
- """
- # strip special prefixes and save it in a temporary file
- fd, self._tmpfile = tempfile.mkstemp()
- with os.fdopen(fd, 'w') as f:
- for line in open(defconfig):
- colon = line.find(':CONFIG_')
- if colon == -1:
- f.write(line)
- else:
- f.write(line[colon + 1:])
-
- self._conf.load_config(self._tmpfile)
- try_remove(self._tmpfile)
- self._tmpfile = None
-
- params = {}
-
- # Get the value of CONFIG_SYS_ARCH, CONFIG_SYS_CPU, ... etc.
- # Set '-' if the value is empty.
- for key, symbol in list(self._SYMBOL_TABLE.items()):
- value = self._conf.syms.get(symbol).str_value
- if value:
- params[key] = value
- else:
- params[key] = '-'
-
- defconfig = os.path.basename(defconfig)
- params['target'], match, rear = defconfig.partition('_defconfig')
- assert match and not rear, '%s : invalid defconfig' % defconfig
-
- # fix-up for aarch64
- if params['arch'] == 'arm' and params['cpu'] == 'armv8':
- params['arch'] = 'aarch64'
-
- # fix-up options field. It should have the form:
- # <config name>[:comma separated config options]
- if params['options'] != '-':
- params['options'] = params['config'] + ':' + \
- params['options'].replace(r'\"', '"')
- elif params['config'] != params['target']:
- params['options'] = params['config']
-
- return params
-
-def scan_defconfigs_for_multiprocess(queue, defconfigs):
- """Scan defconfig files and queue their board parameters
-
- This function is intended to be passed to
- multiprocessing.Process() constructor.
-
- Arguments:
- queue: An instance of multiprocessing.Queue().
- The resulting board parameters are written into it.
- defconfigs: A sequence of defconfig files to be scanned.
- """
- kconf_scanner = KconfigScanner()
- for defconfig in defconfigs:
- queue.put(kconf_scanner.scan(defconfig))
-
-def read_queues(queues, params_list):
- """Read the queues and append the data to the paramers list"""
- for q in queues:
- while not q.empty():
- params_list.append(q.get())
-
-def scan_defconfigs(jobs=1):
- """Collect board parameters for all defconfig files.
-
- This function invokes multiple processes for faster processing.
-
- Arguments:
- jobs: The number of jobs to run simultaneously
- """
- all_defconfigs = []
- for (dirpath, dirnames, filenames) in os.walk(CONFIG_DIR):
- for filename in fnmatch.filter(filenames, '*_defconfig'):
- if fnmatch.fnmatch(filename, '.*'):
- continue
- all_defconfigs.append(os.path.join(dirpath, filename))
-
- total_boards = len(all_defconfigs)
- processes = []
- queues = []
- for i in range(jobs):
- defconfigs = all_defconfigs[total_boards * i // jobs :
- total_boards * (i + 1) // jobs]
- q = multiprocessing.Queue(maxsize=-1)
- p = multiprocessing.Process(target=scan_defconfigs_for_multiprocess,
- args=(q, defconfigs))
- p.start()
- processes.append(p)
- queues.append(q)
-
- # The resulting data should be accumulated to this list
- params_list = []
-
- # Data in the queues should be retrieved preriodically.
- # Otherwise, the queues would become full and subprocesses would get stuck.
- while any([p.is_alive() for p in processes]):
- read_queues(queues, params_list)
- # sleep for a while until the queues are filled
- time.sleep(SLEEP_TIME)
-
- # Joining subprocesses just in case
- # (All subprocesses should already have been finished)
- for p in processes:
- p.join()
-
- # retrieve leftover data
- read_queues(queues, params_list)
-
- return params_list
-
-class MaintainersDatabase:
-
- """The database of board status and maintainers."""
-
- def __init__(self):
- """Create an empty database."""
- self.database = {}
-
- def get_status(self, target):
- """Return the status of the given board.
-
- The board status is generally either 'Active' or 'Orphan'.
- Display a warning message and return '-' if status information
- is not found.
-
- Returns:
- 'Active', 'Orphan' or '-'.
- """
- if not target in self.database:
- print("WARNING: no status info for '%s'" % target, file=sys.stderr)
- return '-'
-
- tmp = self.database[target][0]
- if tmp.startswith('Maintained'):
- return 'Active'
- elif tmp.startswith('Supported'):
- return 'Active'
- elif tmp.startswith('Orphan'):
- return 'Orphan'
- else:
- print(("WARNING: %s: unknown status for '%s'" %
- (tmp, target)), file=sys.stderr)
- return '-'
-
- def get_maintainers(self, target):
- """Return the maintainers of the given board.
-
- Returns:
- Maintainers of the board. If the board has two or more maintainers,
- they are separated with colons.
- """
- if not target in self.database:
- print("WARNING: no maintainers for '%s'" % target, file=sys.stderr)
- return ''
-
- return ':'.join(self.database[target][1])
-
- def parse_file(self, file):
- """Parse a MAINTAINERS file.
-
- Parse a MAINTAINERS file and accumulates board status and
- maintainers information.
-
- Arguments:
- file: MAINTAINERS file to be parsed
- """
- targets = []
- maintainers = []
- status = '-'
- for line in open(file, encoding="utf-8"):
- # Check also commented maintainers
- if line[:3] == '#M:':
- line = line[1:]
- tag, rest = line[:2], line[2:].strip()
- if tag == 'M:':
- maintainers.append(rest)
- elif tag == 'F:':
- # expand wildcard and filter by 'configs/*_defconfig'
- for f in glob.glob(rest):
- front, match, rear = f.partition('configs/')
- if not front and match:
- front, match, rear = rear.rpartition('_defconfig')
- if match and not rear:
- targets.append(front)
- elif tag == 'S:':
- status = rest
- elif line == '\n':
- for target in targets:
- self.database[target] = (status, maintainers)
- targets = []
- maintainers = []
- status = '-'
- if targets:
- for target in targets:
- self.database[target] = (status, maintainers)
-
-def insert_maintainers_info(params_list):
- """Add Status and Maintainers information to the board parameters list.
-
- Arguments:
- params_list: A list of the board parameters
- """
- database = MaintainersDatabase()
- for (dirpath, dirnames, filenames) in os.walk('.'):
- if 'MAINTAINERS' in filenames:
- database.parse_file(os.path.join(dirpath, 'MAINTAINERS'))
-
- for i, params in enumerate(params_list):
- target = params['target']
- params['status'] = database.get_status(target)
- params['maintainers'] = database.get_maintainers(target)
- params_list[i] = params
-
-def format_and_output(params_list, output):
- """Write board parameters into a file.
-
- Columnate the board parameters, sort lines alphabetically,
- and then write them to a file.
-
- Arguments:
- params_list: The list of board parameters
- output: The path to the output file
- """
- FIELDS = ('status', 'arch', 'cpu', 'soc', 'vendor', 'board', 'target',
- 'options', 'maintainers')
-
- # First, decide the width of each column
- max_length = dict([ (f, 0) for f in FIELDS])
- for params in params_list:
- for f in FIELDS:
- max_length[f] = max(max_length[f], len(params[f]))
-
- output_lines = []
- for params in params_list:
- line = ''
- for f in FIELDS:
- # insert two spaces between fields like column -t would
- line += ' ' + params[f].ljust(max_length[f])
- output_lines.append(line.strip())
-
- # ignore case when sorting
- output_lines.sort(key=str.lower)
-
- with open(output, 'w', encoding="utf-8") as f:
- f.write(COMMENT_BLOCK + '\n'.join(output_lines) + '\n')
-
-def gen_boards_cfg(output, jobs=1, force=False, quiet=False):
- """Generate a board database file.
-
- Arguments:
- output: The name of the output file
- jobs: The number of jobs to run simultaneously
- force: Force to generate the output even if it is new
- quiet: True to avoid printing a message if nothing needs doing
- """
- check_top_directory()
-
- if not force and output_is_new(output):
- if not quiet:
- print("%s is up to date. Nothing to do." % output)
- sys.exit(0)
-
- params_list = scan_defconfigs(jobs)
- insert_maintainers_info(params_list)
- format_and_output(params_list, output)
-
-def main():
- try:
- cpu_count = multiprocessing.cpu_count()
- except NotImplementedError:
- cpu_count = 1
-
- parser = optparse.OptionParser()
- # Add options here
- parser.add_option('-f', '--force', action="store_true", default=False,
- help='regenerate the output even if it is new')
- parser.add_option('-j', '--jobs', type='int', default=min(cpu_count, 240),
- help='the number of jobs to run simultaneously')
- parser.add_option('-o', '--output', default=OUTPUT_FILE,
- help='output file [default=%s]' % OUTPUT_FILE)
- parser.add_option('-q', '--quiet', action="store_true", help='run silently')
- (options, args) = parser.parse_args()
-
- gen_boards_cfg(options.output, jobs=options.jobs, force=options.force,
- quiet=options.quiet)
-
-if __name__ == '__main__':
- main()
--
2.37.0.144.g8ac04bfd2-goog
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 17/19] buildman: Allow lines without a symbol
2022-07-12 1:03 [PATCH 00/19] buildman: Integration of boards.cfg file Simon Glass
` (15 preceding siblings ...)
2022-07-12 1:04 ` [PATCH 16/19] Drop genboardscfg.py Simon Glass
@ 2022-07-12 1:04 ` Simon Glass
2022-07-12 1:04 ` [PATCH 18/19] buildman: Drop a Python 2.7 comment Simon Glass
` (2 subsequent siblings)
19 siblings, 0 replies; 31+ messages in thread
From: Simon Glass @ 2022-07-12 1:04 UTC (permalink / raw)
To: U-Boot Mailing List; +Cc: Tom Rini, Simon Glass
The 'nm' tool can produce lines without a symbol, for example:
00000004 t
Silently skip these and anything else without three fields. Drop the
warning since there is nothing the user can do about it.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reported-by: Tom Rini <trini@konsulko.com>
---
tools/buildman/builder.py | 20 +++++++++-----------
1 file changed, 9 insertions(+), 11 deletions(-)
diff --git a/tools/buildman/builder.py b/tools/buildman/builder.py
index 33f9373b8f8..76252b90792 100644
--- a/tools/buildman/builder.py
+++ b/tools/buildman/builder.py
@@ -669,17 +669,15 @@ class Builder:
"""
sym = {}
for line in fd.readlines():
- try:
- if line.strip():
- size, type, name = line[:-1].split()
- except:
- tprint("Invalid line in file '%s': '%s'" % (fname, line[:-1]))
- continue
- if type in 'tTdDbB':
- # function names begin with '.' on 64-bit powerpc
- if '.' in name[1:]:
- name = 'static.' + name.split('.')[0]
- sym[name] = sym.get(name, 0) + int(size, 16)
+ line = line.strip()
+ parts = line.split()
+ if line and len(parts) == 3:
+ size, type, name = line.split()
+ if type in 'tTdDbB':
+ # function names begin with '.' on 64-bit powerpc
+ if '.' in name[1:]:
+ name = 'static.' + name.split('.')[0]
+ sym[name] = sym.get(name, 0) + int(size, 16)
return sym
def _ProcessConfig(self, fname):
--
2.37.0.144.g8ac04bfd2-goog
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 18/19] buildman: Drop a Python 2.7 comment
2022-07-12 1:03 [PATCH 00/19] buildman: Integration of boards.cfg file Simon Glass
` (16 preceding siblings ...)
2022-07-12 1:04 ` [PATCH 17/19] buildman: Allow lines without a symbol Simon Glass
@ 2022-07-12 1:04 ` Simon Glass
2022-07-12 1:04 ` [PATCH 19/19] buildman: Drop a TODO that is done Simon Glass
2022-08-05 17:24 ` [PATCH 00/19] buildman: Integration of boards.cfg file Tom Rini
19 siblings, 0 replies; 31+ messages in thread
From: Simon Glass @ 2022-07-12 1:04 UTC (permalink / raw)
To: U-Boot Mailing List; +Cc: Tom Rini, Simon Glass
This is well out of date, but it is still reasonable to use a list. Drop
the comment.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
tools/buildman/boards.py | 1 -
1 file changed, 1 deletion(-)
diff --git a/tools/buildman/boards.py b/tools/buildman/boards.py
index b30b344bc8e..8a0971aa407 100644
--- a/tools/buildman/boards.py
+++ b/tools/buildman/boards.py
@@ -371,7 +371,6 @@ class MaintainersDatabase:
class Boards:
"""Manage a list of boards."""
def __init__(self):
- # Use a simple list here, sinc OrderedDict requires Python 2.7
self._boards = []
def add_board(self, brd):
--
2.37.0.144.g8ac04bfd2-goog
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH 19/19] buildman: Drop a TODO that is done
2022-07-12 1:03 [PATCH 00/19] buildman: Integration of boards.cfg file Simon Glass
` (17 preceding siblings ...)
2022-07-12 1:04 ` [PATCH 18/19] buildman: Drop a Python 2.7 comment Simon Glass
@ 2022-07-12 1:04 ` Simon Glass
2022-08-05 17:24 ` [PATCH 00/19] buildman: Integration of boards.cfg file Tom Rini
19 siblings, 0 replies; 31+ messages in thread
From: Simon Glass @ 2022-07-12 1:04 UTC (permalink / raw)
To: U-Boot Mailing List; +Cc: Tom Rini, Simon Glass
Buildman now uses worktrees when available, instead of doing a full clone.
This was done in this commit:
76de29fc4f buildman: Use git worktrees instead of git clones when possible
Drop the TODO.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
tools/buildman/README | 3 ---
1 file changed, 3 deletions(-)
diff --git a/tools/buildman/README b/tools/buildman/README
index 3ba08d0dace..a8357a804b3 100644
--- a/tools/buildman/README
+++ b/tools/buildman/README
@@ -1331,9 +1331,6 @@ scope for more though, e.g.:
- 'hunting' for problems, perhaps by building a few boards for each arch, or
checking commits for changed files and building only boards which use those
files
-- using the same git repo for all threads instead of cloning it. Currently
- it uses about 500MB per thread, so on a 64-thread machine this is 32GB for
- the build.
Credits
--
2.37.0.144.g8ac04bfd2-goog
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [PATCH 00/19] buildman: Integration of boards.cfg file
2022-07-12 1:03 [PATCH 00/19] buildman: Integration of boards.cfg file Simon Glass
` (18 preceding siblings ...)
2022-07-12 1:04 ` [PATCH 19/19] buildman: Drop a TODO that is done Simon Glass
@ 2022-08-05 17:24 ` Tom Rini
19 siblings, 0 replies; 31+ messages in thread
From: Tom Rini @ 2022-08-05 17:24 UTC (permalink / raw)
To: U-Boot Mailing List, Simon Glass
Cc: Alper Nebi Yasak, Andre Przywara, Patrick Delaunay, Bin Meng,
Steffen Jaeckel, Angelo Dureghello, Heinrich Schuchardt,
AKASHI Takahiro, Alexandru Gagniuc, Jan Kiszka
On Mon, 11 Jul 2022 19:03:54 -0600, Simon Glass wrote:
> This series drops the need for the genboardscfg.py script, so that the
> boards.cfg file is produced (and consumed) entirely within buildman. The
> file is not entirely removed since it does have some uses and we need some
> sort of cache for the information. The genboardscfg.py script is
> effectively incorporated in buildman.
>
> It also improves operation from an IDE with a new -I option and fixes up
> some of the pylint warnings in buildman.
>
> [...]
Applied to u-boot/u-boot.git/master, thanks!
--
Tom
^ permalink raw reply [flat|nested] 31+ messages in thread