* [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups
@ 2016-05-19 6:51 Masahiro Yamada
2016-05-19 6:51 ` [U-Boot] [PATCH 01/21] tools: moveconfig: fix --dry-run option Masahiro Yamada
` (21 more replies)
0 siblings, 22 replies; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-19 6:51 UTC (permalink / raw)
To: u-boot
Masahiro Yamada (21):
tools: moveconfig: fix --dry-run option
tools: moveconfig: rename update_defconfig() to update_dotconfig()
tools: moveconfig: remove redundant else: after sys.exit()
tools: moveconfig: check directory relocation before compilers
tools: moveconfig: check compilers before starting defconfig walk
tools: moveconfig: exit with error message for not clean directory
tools: moveconfig: increment number of processed files monotonically
tools: moveconfig: do not rely on type and default value given by
users
tools: moveconfig: drop code for handling type and default value
tools: moveconfig: allow to give CONFIG names as argument directly
tools: moveconfig: add --undef option to move CONFIGs with default y
tools: moveconfig: compute file paths just once
tools: moveconfig: move log output code out of Kconfig Parser class
tools: moveconfig: display log atomically in more readable format
tools: moveconfig: refactor code to go back to idle state
tools: moveconfig: skip savedefconfig if .config was not updated
tools: moveconfig: display log when savedefconfig occurs
tools: moveconfig: report when CONFIGs are removed by savedefconfig
tools: moveconfig: report when defconfig is updated
tools: moveconfig: add --force-sync option
tools: moveconfig: allow to run without any CONFIG specified
scripts/Makefile.autoconf | 3 +-
tools/moveconfig.py | 621 +++++++++++++++++++++++++---------------------
2 files changed, 339 insertions(+), 285 deletions(-)
--
1.9.1
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 01/21] tools: moveconfig: fix --dry-run option
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
@ 2016-05-19 6:51 ` Masahiro Yamada
2016-05-24 14:27 ` Joe Hershberger
2016-05-19 6:51 ` [U-Boot] [PATCH 02/21] tools: moveconfig: rename update_defconfig() to update_dotconfig() Masahiro Yamada
` (20 subsequent siblings)
21 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-19 6:51 UTC (permalink / raw)
To: u-boot
Since commit 96464badc794 ("moveconfig: Always run savedefconfig on
the moved config"), --dry-run option is broken.
The --dry-run option prevents the .config from being modified,
but defconfig files might be updated by "make savedefconfig"
regardless of the --dry-run option.
Move the "if not self.options.dry_run" conditional to the correct
place.
Fixes 96464badc794 ("moveconfig: Always run savedefconfig on the moved config")
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---
tools/moveconfig.py | 15 +++++++--------
1 file changed, 7 insertions(+), 8 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 68631b7..fd98e41 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -511,11 +511,10 @@ class KconfigParser:
# Print log in one shot to not mix up logs from different threads.
print log,
- if not self.options.dry_run:
- with open(dotconfig_path, 'a') as f:
- for (action, value) in results:
- if action == ACTION_MOVE:
- f.write(value + '\n')
+ with open(dotconfig_path, 'a') as f:
+ for (action, value) in results:
+ if action == ACTION_MOVE:
+ f.write(value + '\n')
os.remove(os.path.join(self.build_dir, 'include', 'config', 'auto.conf'))
os.remove(autoconf_path)
@@ -647,9 +646,9 @@ class Slot:
return False
if self.state == STATE_SAVEDEFCONFIG:
- defconfig_path = os.path.join(self.build_dir, 'defconfig')
- shutil.move(defconfig_path,
- os.path.join('configs', self.defconfig))
+ if not self.options.dry_run:
+ shutil.move(os.path.join(self.build_dir, 'defconfig'),
+ os.path.join('configs', self.defconfig))
self.state = STATE_IDLE
return True
--
1.9.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 02/21] tools: moveconfig: rename update_defconfig() to update_dotconfig()
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
2016-05-19 6:51 ` [U-Boot] [PATCH 01/21] tools: moveconfig: fix --dry-run option Masahiro Yamada
@ 2016-05-19 6:51 ` Masahiro Yamada
2016-05-24 14:27 ` Joe Hershberger
2016-05-19 6:51 ` [U-Boot] [PATCH 03/21] tools: moveconfig: remove redundant else: after sys.exit() Masahiro Yamada
` (19 subsequent siblings)
21 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-19 6:51 UTC (permalink / raw)
To: u-boot
Commit 96464badc794 ("moveconfig: Always run savedefconfig on the
moved config") changed how defconfig files were updated.
Since then, the function update_defconfig() does not modify defconfig
files at all (instead, they are updated by "make savedefconfig").
The name update_dotconfig() is a better fit for this function.
Also, update the comment block to match the actual behavior.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---
tools/moveconfig.py | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index fd98e41..9029287 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -459,13 +459,13 @@ class KconfigParser:
return (action, value)
- def update_defconfig(self, defconfig):
- """Parse files for the config options and update the defconfig.
+ def update_dotconfig(self, defconfig):
+ """Parse files for the config options and update the .config.
This function parses the given defconfig, the generated .config
and include/autoconf.mk searching the target options.
- Move the config option(s) to the defconfig or do nothing if unneeded.
- Also, display the log to show what happened to this defconfig.
+ Move the config option(s) to the .config as needed.
+ Also, display the log to show what happened to the .config.
Arguments:
defconfig: defconfig name.
@@ -632,7 +632,7 @@ class Slot:
return True
if self.state == STATE_AUTOCONF:
- self.parser.update_defconfig(self.defconfig)
+ self.parser.update_dotconfig(self.defconfig)
print ' %d defconfigs out of %d\r' % (self.num + 1, self.total),
sys.stdout.flush()
--
1.9.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 03/21] tools: moveconfig: remove redundant else: after sys.exit()
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
2016-05-19 6:51 ` [U-Boot] [PATCH 01/21] tools: moveconfig: fix --dry-run option Masahiro Yamada
2016-05-19 6:51 ` [U-Boot] [PATCH 02/21] tools: moveconfig: rename update_defconfig() to update_dotconfig() Masahiro Yamada
@ 2016-05-19 6:51 ` Masahiro Yamada
2016-05-24 14:27 ` Joe Hershberger
2016-05-19 6:51 ` [U-Boot] [PATCH 04/21] tools: moveconfig: check directory relocation before compilers Masahiro Yamada
` (18 subsequent siblings)
21 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-19 6:51 UTC (permalink / raw)
To: u-boot
Nesting by "else:" is not generally useful after such statements
as return, break, and sys.exit(), etc.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---
tools/moveconfig.py | 12 +++++-------
1 file changed, 5 insertions(+), 7 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 9029287..1332bd2 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -623,13 +623,11 @@ class Slot:
COLOR_LIGHT_CYAN, errout)
if self.options.exit_on_error:
sys.exit("Exit on error.")
- else:
- # If --exit-on-error flag is not set,
- # skip this board and continue.
- # Record the failed board.
- self.failed_boards.append(self.defconfig)
- self.state = STATE_IDLE
- return True
+ # If --exit-on-error flag is not set, skip this board and continue.
+ # Record the failed board.
+ self.failed_boards.append(self.defconfig)
+ self.state = STATE_IDLE
+ return True
if self.state == STATE_AUTOCONF:
self.parser.update_dotconfig(self.defconfig)
--
1.9.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 04/21] tools: moveconfig: check directory relocation before compilers
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
` (2 preceding siblings ...)
2016-05-19 6:51 ` [U-Boot] [PATCH 03/21] tools: moveconfig: remove redundant else: after sys.exit() Masahiro Yamada
@ 2016-05-19 6:51 ` Masahiro Yamada
2016-05-24 14:27 ` Joe Hershberger
2016-05-19 6:51 ` [U-Boot] [PATCH 05/21] tools: moveconfig: check compilers before starting defconfig walk Masahiro Yamada
` (17 subsequent siblings)
21 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-19 6:51 UTC (permalink / raw)
To: u-boot
We must make sure this tool is run from the top of source directory
before calling update_cross_compile(). Otherwise, the following
exception is thrown:
Traceback (most recent call last):
File "./moveconfig.py", line 918, in <module>
main()
File "./moveconfig.py", line 908, in main
update_cross_compile()
File "./moveconfig.py", line 292, in update_cross_compile
for arch in os.listdir('arch'):
OSError: [Errno 2] No such file or directory: 'arch'
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---
tools/moveconfig.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 1332bd2..ce8245a 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -905,10 +905,10 @@ def main():
config_attrs = parse_recipe(args[0])
- update_cross_compile()
-
check_top_directory()
+ update_cross_compile()
+
if not options.cleanup_headers_only:
move_config(config_attrs, options)
--
1.9.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 05/21] tools: moveconfig: check compilers before starting defconfig walk
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
` (3 preceding siblings ...)
2016-05-19 6:51 ` [U-Boot] [PATCH 04/21] tools: moveconfig: check directory relocation before compilers Masahiro Yamada
@ 2016-05-19 6:51 ` Masahiro Yamada
2016-05-24 14:27 ` Joe Hershberger
2016-05-19 6:51 ` [U-Boot] [PATCH 06/21] tools: moveconfig: exit with error message for not clean directory Masahiro Yamada
` (16 subsequent siblings)
21 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-19 6:51 UTC (permalink / raw)
To: u-boot
Since commit 25400090b1e2 ("moveconfig: Print a message for
missing compiler"), this tool parses error messages generated by
"make include/config/auto.conf" every time an error occurs in order
to detect missing compiler.
Instead of that, we can look for compilers in the PATH environment
only once before starting the defconfig walk. If specified compilers
are missing, "make include/config/auto.conf" will apparently fail
for corresponding architectures. So, the tool can just report
"Compiler is missing." and skip those boards.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---
tools/moveconfig.py | 71 ++++++++++++++++++++++++++++++++++++-----------------
1 file changed, 48 insertions(+), 23 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index ce8245a..7e916c2 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -82,7 +82,12 @@ It looks like one of the followings:
This config option was not found in the config header.
Nothing to do.
- - Failed to process. Skip.
+ - Compiler is missing. Do nothing.
+ The compiler specified for this architecture was not found
+ in your PATH environment.
+ (If -e option is passed, the tool exits immediately.)
+
+ - Failed to process.
An error occurred during processing this defconfig. Skipped.
(If -e option is passed, the tool exits immediately on error.)
@@ -272,7 +277,7 @@ def log_msg(color_enabled, color, defconfig, msg):
return defconfig[:-len('_defconfig')].ljust(37) + ': ' + \
color_text(color_enabled, color, msg) + '\n'
-def update_cross_compile():
+def update_cross_compile(color_enabled):
"""Update per-arch CROSS_COMPILE via environment variables
The default CROSS_COMPILE values are available
@@ -286,6 +291,9 @@ def update_cross_compile():
export CROSS_COMPILE_ARM=...
export CROSS_COMPILE_POWERPC=...
+
+ Then, this function checks if specified compilers really exist in your
+ PATH environment.
"""
archs = []
@@ -299,8 +307,20 @@ def update_cross_compile():
for arch in archs:
env = 'CROSS_COMPILE_' + arch.upper()
cross_compile = os.environ.get(env)
- if cross_compile:
- CROSS_COMPILE[arch] = cross_compile
+ if not cross_compile:
+ cross_compile = CROSS_COMPILE.get(arch, '')
+
+ for path in os.environ["PATH"].split(os.pathsep):
+ gcc_path = os.path.join(path, cross_compile + 'gcc')
+ if os.path.isfile(gcc_path) and os.access(gcc_path, os.X_OK):
+ break
+ else:
+ print >> sys.stderr, color_text(color_enabled, COLOR_YELLOW,
+ 'warning: %sgcc: not found in PATH. %s architecture boards will be skipped'
+ % (cross_compile, arch))
+ cross_compile = None
+
+ CROSS_COMPILE[arch] = cross_compile
def cleanup_one_header(header_path, patterns, dry_run):
"""Clean regex-matched lines away from a file.
@@ -387,6 +407,10 @@ class KconfigParser:
Returns:
A string storing the compiler prefix for the architecture.
+ Return a NULL string for architectures that do not require
+ compiler prefix (Sandbox and native build is the case).
+ Return None if the specified compiler is missing in your PATH.
+ Caller should distinguish '' and None.
"""
arch = ''
cpu = ''
@@ -400,13 +424,14 @@ class KconfigParser:
if m:
cpu = m.group(1)
- assert arch, 'Error: arch is not defined in %s' % defconfig
+ if not arch:
+ return None
# fix-up for aarch64
if arch == 'arm' and cpu == 'armv8':
arch = 'aarch64'
- return CROSS_COMPILE.get(arch, '')
+ return CROSS_COMPILE.get(arch, None)
def parse_one_config(self, config_attr, defconfig_lines, autoconf_lines):
"""Parse .config, defconfig, include/autoconf.mk for one config.
@@ -606,21 +631,12 @@ class Slot:
return False
if self.ps.poll() != 0:
- errmsg = 'Failed to process.'
- errout = self.ps.stderr.read()
- if errout.find('gcc: command not found') != -1:
- errmsg = 'Compiler not found ('
- errmsg += color_text(self.options.color, COLOR_YELLOW,
- self.cross_compile)
- errmsg += color_text(self.options.color, COLOR_LIGHT_RED,
- ')')
- print >> sys.stderr, log_msg(self.options.color,
- COLOR_LIGHT_RED,
- self.defconfig,
- errmsg),
+ print >> sys.stderr, log_msg(self.options.color, COLOR_LIGHT_RED,
+ self.defconfig, "Failed to process."),
if self.options.verbose:
print >> sys.stderr, color_text(self.options.color,
- COLOR_LIGHT_CYAN, errout)
+ COLOR_LIGHT_CYAN,
+ self.ps.stderr.read())
if self.options.exit_on_error:
sys.exit("Exit on error.")
# If --exit-on-error flag is not set, skip this board and continue.
@@ -651,15 +667,24 @@ class Slot:
return True
self.cross_compile = self.parser.get_cross_compile()
+ if self.cross_compile is None:
+ print >> sys.stderr, log_msg(self.options.color, COLOR_YELLOW,
+ self.defconfig,
+ "Compiler is missing. Do nothing."),
+ if self.options.exit_on_error:
+ sys.exit("Exit on error.")
+ # If --exit-on-error flag is not set, skip this board and continue.
+ # Record the failed board.
+ self.failed_boards.append(self.defconfig)
+ self.state = STATE_IDLE
+ return True
+
cmd = list(self.make_cmd)
if self.cross_compile:
cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
cmd.append('include/config/auto.conf')
- """This will be screen-scraped, so be sure the expected text will be
- returned consistently on every machine by setting LANG=C"""
self.ps = subprocess.Popen(cmd, stdout=self.devnull,
- env=dict(os.environ, LANG='C'),
stderr=subprocess.PIPE)
self.state = STATE_AUTOCONF
return False
@@ -907,7 +932,7 @@ def main():
check_top_directory()
- update_cross_compile()
+ update_cross_compile(options.color)
if not options.cleanup_headers_only:
move_config(config_attrs, options)
--
1.9.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 06/21] tools: moveconfig: exit with error message for not clean directory
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
` (4 preceding siblings ...)
2016-05-19 6:51 ` [U-Boot] [PATCH 05/21] tools: moveconfig: check compilers before starting defconfig walk Masahiro Yamada
@ 2016-05-19 6:51 ` Masahiro Yamada
2016-05-24 14:28 ` Joe Hershberger
2016-05-19 6:51 ` [U-Boot] [PATCH 07/21] tools: moveconfig: increment number of processed files monotonically Masahiro Yamada
` (15 subsequent siblings)
21 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-19 6:51 UTC (permalink / raw)
To: u-boot
When the source tree is not clean, this tool raises an exception
with a message like follows:
Traceback (most recent call last):
File "tools/moveconfig.py", line 939, in <module>
main()
File "tools/moveconfig.py", line 934, in main
move_config(config_attrs, options)
File "tools/moveconfig.py", line 808, in move_config
while not slots.available():
File "tools/moveconfig.py", line 733, in available
if slot.poll():
File "tools/moveconfig.py", line 645, in poll
self.parser.update_dotconfig(self.defconfig)
File "tools/moveconfig.py", line 503, in update_dotconfig
with open(autoconf_path) as f:
IOError: [Errno 2] No such file or directory: '/tmp/tmpDtzCgl/include/autoconf.mk'
This does not explain clearly what is wrong. Show an appropriate
error message "srctree is not clean, please run 'make mrproper'"
in such a situation.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---
tools/moveconfig.py | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 7e916c2..3da4726 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -252,6 +252,12 @@ def check_top_directory():
if not os.path.exists(f):
sys.exit('Please run at the top of source directory.')
+def check_clean_directory():
+ """Exit if the source tree is not clean."""
+ for f in ('.config', 'include/config'):
+ if os.path.exists(f):
+ sys.exit("srctree is not clean, please run 'make mrproper'")
+
def get_make_cmd():
"""Get the command name of GNU Make.
@@ -932,6 +938,8 @@ def main():
check_top_directory()
+ check_clean_directory()
+
update_cross_compile(options.color)
if not options.cleanup_headers_only:
--
1.9.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 07/21] tools: moveconfig: increment number of processed files monotonically
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
` (5 preceding siblings ...)
2016-05-19 6:51 ` [U-Boot] [PATCH 06/21] tools: moveconfig: exit with error message for not clean directory Masahiro Yamada
@ 2016-05-19 6:51 ` Masahiro Yamada
2016-05-24 14:28 ` Joe Hershberger
2016-05-19 6:51 ` [U-Boot] [PATCH 08/21] tools: moveconfig: do not rely on type and default value given by users Masahiro Yamada
` (14 subsequent siblings)
21 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-19 6:51 UTC (permalink / raw)
To: u-boot
Currently, the progress " * defconfigs out of 1133" does not increase
monotonically.
Moreover, the number of processed defconfigs does not match the total
number of defconfigs when this tool finishes, like:
1132 defconfigs out of 1133
Clean up headers? [y/n]:
It looks like the task was not completed, and some users might feel
upset about it.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---
tools/moveconfig.py | 65 ++++++++++++++++++++++++++++++++++++++++-------------
1 file changed, 49 insertions(+), 16 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 3da4726..5e78266 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -388,6 +388,29 @@ def cleanup_headers(config_attrs, dry_run):
patterns, dry_run)
### classes ###
+class Progress:
+
+ """Progress Indicator"""
+
+ def __init__(self, total):
+ """Create a new progress indicator.
+
+ Arguments:
+ total: A number of defconfig files to process.
+ """
+ self.current = 0
+ self.total = total
+
+ def inc(self):
+ """Increment the number of processed defconfig files."""
+
+ self.current += 1
+
+ def show(self):
+ """Display the progress."""
+ print ' %d defconfigs out of %d\r' % (self.current, self.total),
+ sys.stdout.flush()
+
class KconfigParser:
"""A parser of .config and include/autoconf.mk."""
@@ -395,17 +418,19 @@ class KconfigParser:
re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"')
re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"')
- def __init__(self, config_attrs, options, build_dir):
+ def __init__(self, config_attrs, options, progress, build_dir):
"""Create a new parser.
Arguments:
config_attrs: A list of dictionaris, each of them includes the name,
the type, and the default value of the target config.
options: option flags.
+ progress: A progress indicator
build_dir: Build directory.
"""
self.config_attrs = config_attrs
self.options = options
+ self.progress = progress
self.build_dir = build_dir
def get_cross_compile(self):
@@ -541,6 +566,7 @@ class KconfigParser:
# Some threads are running in parallel.
# Print log in one shot to not mix up logs from different threads.
print log,
+ self.progress.show()
with open(dotconfig_path, 'a') as f:
for (action, value) in results:
@@ -559,21 +585,24 @@ class Slot:
for faster processing.
"""
- def __init__(self, config_attrs, options, devnull, make_cmd):
+ def __init__(self, config_attrs, options, progress, devnull, make_cmd):
"""Create a new process slot.
Arguments:
config_attrs: A list of dictionaris, each of them includes the name,
the type, and the default value of the target config.
options: option flags.
+ progress: A progress indicator.
devnull: A file object of '/dev/null'.
make_cmd: command name of GNU Make.
"""
self.options = options
+ self.progress = progress
self.build_dir = tempfile.mkdtemp()
self.devnull = devnull
self.make_cmd = (make_cmd, 'O=' + self.build_dir)
- self.parser = KconfigParser(config_attrs, options, self.build_dir)
+ self.parser = KconfigParser(config_attrs, options, progress,
+ self.build_dir)
self.state = STATE_IDLE
self.failed_boards = []
@@ -592,7 +621,7 @@ class Slot:
pass
shutil.rmtree(self.build_dir)
- def add(self, defconfig, num, total):
+ def add(self, defconfig):
"""Assign a new subprocess for defconfig and add it to the slot.
If the slot is vacant, create a new subprocess for processing the
@@ -613,8 +642,6 @@ class Slot:
stderr=subprocess.PIPE)
self.defconfig = defconfig
self.state = STATE_DEFCONFIG
- self.num = num
- self.total = total
return True
def poll(self):
@@ -643,6 +670,8 @@ class Slot:
print >> sys.stderr, color_text(self.options.color,
COLOR_LIGHT_CYAN,
self.ps.stderr.read())
+ self.progress.inc()
+ self.progress.show()
if self.options.exit_on_error:
sys.exit("Exit on error.")
# If --exit-on-error flag is not set, skip this board and continue.
@@ -654,9 +683,6 @@ class Slot:
if self.state == STATE_AUTOCONF:
self.parser.update_dotconfig(self.defconfig)
- print ' %d defconfigs out of %d\r' % (self.num + 1, self.total),
- sys.stdout.flush()
-
"""Save off the defconfig in a consistent way"""
cmd = list(self.make_cmd)
cmd.append('savedefconfig')
@@ -669,6 +695,7 @@ class Slot:
if not self.options.dry_run:
shutil.move(os.path.join(self.build_dir, 'defconfig'),
os.path.join('configs', self.defconfig))
+ self.progress.inc()
self.state = STATE_IDLE
return True
@@ -677,6 +704,8 @@ class Slot:
print >> sys.stderr, log_msg(self.options.color, COLOR_YELLOW,
self.defconfig,
"Compiler is missing. Do nothing."),
+ self.progress.inc()
+ self.progress.show()
if self.options.exit_on_error:
sys.exit("Exit on error.")
# If --exit-on-error flag is not set, skip this board and continue.
@@ -704,22 +733,24 @@ class Slots:
"""Controller of the array of subprocess slots."""
- def __init__(self, config_attrs, options):
+ def __init__(self, config_attrs, options, progress):
"""Create a new slots controller.
Arguments:
config_attrs: A list of dictionaris containing the name, the type,
and the default value of the target CONFIG.
options: option flags.
+ progress: A progress indicator.
"""
self.options = options
self.slots = []
devnull = get_devnull()
make_cmd = get_make_cmd()
for i in range(options.jobs):
- self.slots.append(Slot(config_attrs, options, devnull, make_cmd))
+ self.slots.append(Slot(config_attrs, options, progress, devnull,
+ make_cmd))
- def add(self, defconfig, num, total):
+ def add(self, defconfig):
"""Add a new subprocess if a vacant slot is found.
Arguments:
@@ -729,7 +760,7 @@ class Slots:
Return True on success or False on failure
"""
for slot in self.slots:
- if slot.add(defconfig, num, total):
+ if slot.add(defconfig):
return True
return False
@@ -808,13 +839,14 @@ def move_config(config_attrs, options):
for filename in fnmatch.filter(filenames, '*_defconfig'):
defconfigs.append(os.path.join(dirpath, filename))
- slots = Slots(config_attrs, options)
+ progress = Progress(len(defconfigs))
+ slots = Slots(config_attrs, options, progress)
# Main loop to process defconfig files:
# Add a new subprocess into a vacant slot.
# Sleep if there is no available slot.
- for i, defconfig in enumerate(defconfigs):
- while not slots.add(defconfig, i, len(defconfigs)):
+ for defconfig in defconfigs:
+ while not slots.add(defconfig):
while not slots.available():
# No available slot: sleep for a while
time.sleep(SLEEP_TIME)
@@ -823,6 +855,7 @@ def move_config(config_attrs, options):
while not slots.empty():
time.sleep(SLEEP_TIME)
+ progress.show()
print ''
slots.show_failed_boards()
--
1.9.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 08/21] tools: moveconfig: do not rely on type and default value given by users
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
` (6 preceding siblings ...)
2016-05-19 6:51 ` [U-Boot] [PATCH 07/21] tools: moveconfig: increment number of processed files monotonically Masahiro Yamada
@ 2016-05-19 6:51 ` Masahiro Yamada
2016-05-24 14:27 ` Joe Hershberger
2016-05-19 6:51 ` [U-Boot] [PATCH 09/21] tools: moveconfig: drop code for handling type and default value Masahiro Yamada
` (13 subsequent siblings)
21 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-19 6:51 UTC (permalink / raw)
To: u-boot
Commit 96464badc794 ("moveconfig: Always run savedefconfig on the
moved config") changed the work flow of this tool a lot from the
original intention when this tool was designed first.
Since then, before running this tool, users must edit the Kconfig to
add the menu entries for the configs they are moving. It means users
had already specified the type and the default value for each CONFIG
via the Kconfig entries. Nevertheless, users are still required to
dictate the same type and the default value in each line of the input
file. So, my idea here is to deprecate the latter.
Before moving forward with it, there is one thing worth mentioning;
since the savedefconfig re-sync was introduced, there is a case this
tool can not handle correctly.
Let's say we are moving CONFIG_FOO from board headers to Kconfig,
and we want to make it "default y" option. First, we are supposed
to create an entry:
config FOO
bool "foo"
default y
CONFIG_FOO=y will appear in the .config for every processed defconfig.
It will be duplicated to the include/autoconf.mk as well unless the
board explicitly #undef's it. Therefore, the defconfig without
"#define CONFIG_FOO" or "#undef CONFIG_FOO" that should be converted
to "unset" would be misconverted to "=y".
Anyway, this has been a problem for a while. Yet, I am not inclined
to revert that commit because the "create a Kconfig entry beforehand,
and sync with savedefconfig" approach has been successful in practical
use cases. I will come back to this problem with a different solution
in a few commits later.
For other use cases, I see no reason to require redundant dictation
in the input file of this tool. Instead, the tool can know the types
and default values by parsing the .config file.
This commit changes the tool to use the CONFIG names, but ignore the
types and default values given by the input file.
This commit also fixes one bug. Before this commit, the tool could
not move a integer-typed CONFIG with value 1.
For example, assume we are moving CONFIG_CONS_INDEX. Please note
this is a integer type option.
Many board headers define this CONFIG as 1.
#define CONFIG_CONS_INDEX 1
It will be converted to
CONFIG_CONS_INDEX=y
and moved to include/autoconf.mk, by the tools/scripts/define2mk.sed.
It will cause "make savedefconfig" to fail due to the type conflict.
This commit takes care of it by detecting the type and converting the
CONFIG value correctly.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---
tools/moveconfig.py | 88 ++++++++++++++++++++++++++---------------------------
1 file changed, 43 insertions(+), 45 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 5e78266..ee0afbd 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -70,13 +70,17 @@ It looks like one of the followings:
- Move 'CONFIG_... '
This config option was moved to the defconfig
- - Default value 'CONFIG_...'. Do nothing.
- The value of this option is the same as default.
- We do not have to add it to the defconfig.
-
- - 'CONFIG_...' already exists in Kconfig. Do nothing.
- This config option is already defined in Kconfig.
- We do not need/want to touch it.
+ - CONFIG_... is not defined in Kconfig. Do nothing.
+ The entry for this CONFIG was not found in Kconfig.
+ There are two common cases:
+ - You forgot to create an entry for the CONFIG before running
+ this tool, or made a typo in a CONFIG passed to this tool.
+ - The entry was hidden due to unmet 'depends on'.
+ This is correct behavior.
+
+ - 'CONFIG_...' is the same as the define in Kconfig. Do nothing.
+ The define in the config header matched the one in Kconfig.
+ We do not need to touch it.
- Undefined. Do nothing.
This config option was not found in the config header.
@@ -216,9 +220,8 @@ STATE_AUTOCONF = 2
STATE_SAVEDEFCONFIG = 3
ACTION_MOVE = 0
-ACTION_DEFAULT_VALUE = 1
-ACTION_ALREADY_EXIST = 2
-ACTION_UNDEFINED = 3
+ACTION_NO_ENTRY = 1
+ACTION_NO_CHANGE = 2
COLOR_BLACK = '0;30'
COLOR_RED = '0;31'
@@ -464,7 +467,7 @@ class KconfigParser:
return CROSS_COMPILE.get(arch, None)
- def parse_one_config(self, config_attr, defconfig_lines, autoconf_lines):
+ def parse_one_config(self, config_attr, dotconfig_lines, autoconf_lines):
"""Parse .config, defconfig, include/autoconf.mk for one config.
This function looks for the config options in the lines from
@@ -474,7 +477,7 @@ class KconfigParser:
Arguments:
config_attr: A dictionary including the name, the type,
and the default value of the target config.
- defconfig_lines: lines from the original defconfig file.
+ dotconfig_lines: lines from the .config file.
autoconf_lines: lines from the include/autoconf.mk file.
Returns:
@@ -484,42 +487,40 @@ class KconfigParser:
config = config_attr['config']
not_set = '# %s is not set' % config
- if config_attr['type'] in ('bool', 'tristate') and \
- config_attr['default'] == 'n':
- default = not_set
- else:
- default = config + '=' + config_attr['default']
-
- for line in defconfig_lines:
+ for line in dotconfig_lines:
line = line.rstrip()
if line.startswith(config + '=') or line == not_set:
- return (ACTION_ALREADY_EXIST, line)
-
- if config_attr['type'] in ('bool', 'tristate'):
- value = not_set
+ old_val = line
+ break
else:
- value = '(undefined)'
+ return (ACTION_NO_ENTRY, config)
for line in autoconf_lines:
line = line.rstrip()
if line.startswith(config + '='):
- value = line
+ new_val = line
break
-
- if value == default:
- action = ACTION_DEFAULT_VALUE
- elif value == '(undefined)':
- action = ACTION_UNDEFINED
else:
- action = ACTION_MOVE
+ new_val = not_set
+
+ if old_val == new_val:
+ return (ACTION_NO_CHANGE, new_val)
+
+ # If this CONFIG is neither bool nor trisate
+ if old_val[-2:] != '=y' and old_val[-2:] != '=m' and old_val != not_set:
+ # tools/scripts/define2mk.sed changes '1' to 'y'.
+ # This is a problem if the CONFIG is int type.
+ # Check the type in Kconfig and handle it correctly.
+ if new_val[-2:] == '=y':
+ new_val = new_val[:-1] + '1'
- return (action, value)
+ return (ACTION_MOVE, new_val)
def update_dotconfig(self, defconfig):
"""Parse files for the config options and update the .config.
- This function parses the given defconfig, the generated .config
- and include/autoconf.mk searching the target options.
+ This function parses the generated .config and include/autoconf.mk
+ searching the target options.
Move the config option(s) to the .config as needed.
Also, display the log to show what happened to the .config.
@@ -527,19 +528,18 @@ class KconfigParser:
defconfig: defconfig name.
"""
- defconfig_path = os.path.join('configs', defconfig)
dotconfig_path = os.path.join(self.build_dir, '.config')
autoconf_path = os.path.join(self.build_dir, 'include', 'autoconf.mk')
results = []
- with open(defconfig_path) as f:
- defconfig_lines = f.readlines()
+ with open(dotconfig_path) as f:
+ dotconfig_lines = f.readlines()
with open(autoconf_path) as f:
autoconf_lines = f.readlines()
for config_attr in self.config_attrs:
- result = self.parse_one_config(config_attr, defconfig_lines,
+ result = self.parse_one_config(config_attr, dotconfig_lines,
autoconf_lines)
results.append(result)
@@ -549,15 +549,13 @@ class KconfigParser:
if action == ACTION_MOVE:
actlog = "Move '%s'" % value
log_color = COLOR_LIGHT_GREEN
- elif action == ACTION_DEFAULT_VALUE:
- actlog = "Default value '%s'. Do nothing." % value
+ elif action == ACTION_NO_ENTRY:
+ actlog = "%s is not defined in Kconfig. Do nothing." % value
log_color = COLOR_LIGHT_BLUE
- elif action == ACTION_ALREADY_EXIST:
- actlog = "'%s' already defined in Kconfig. Do nothing." % value
+ elif action == ACTION_NO_CHANGE:
+ actlog = "'%s' is the same as the define in Kconfig. Do nothing." \
+ % value
log_color = COLOR_LIGHT_PURPLE
- elif action == ACTION_UNDEFINED:
- actlog = "Undefined. Do nothing."
- log_color = COLOR_DARK_GRAY
else:
sys.exit("Internal Error. This should not happen.")
--
1.9.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 09/21] tools: moveconfig: drop code for handling type and default value
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
` (7 preceding siblings ...)
2016-05-19 6:51 ` [U-Boot] [PATCH 08/21] tools: moveconfig: do not rely on type and default value given by users Masahiro Yamada
@ 2016-05-19 6:51 ` Masahiro Yamada
2016-05-24 14:38 ` Joe Hershberger
2016-05-19 6:51 ` [U-Boot] [PATCH 10/21] tools: moveconfig: allow to give CONFIG names as argument directly Masahiro Yamada
` (12 subsequent siblings)
21 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-19 6:51 UTC (permalink / raw)
To: u-boot
Now types and defalut values given by the input file are just
ignored. Delete unnecessary code.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---
tools/moveconfig.py | 105 +++++++++++++++-------------------------------------
1 file changed, 29 insertions(+), 76 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index ee0afbd..c452c7a 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -360,12 +360,11 @@ def cleanup_one_header(header_path, patterns, dry_run):
if not i in matched:
f.write(line)
-def cleanup_headers(config_attrs, dry_run):
+def cleanup_headers(configs, dry_run):
"""Delete config defines from board headers.
Arguments:
- config_attrs: A list of dictionaris, each of them includes the name,
- the type, and the default value of the target config.
+ configs: A list of CONFIGs to remove.
dry_run: make no changes, but still display log.
"""
while True:
@@ -378,8 +377,7 @@ def cleanup_headers(config_attrs, dry_run):
return
patterns = []
- for config_attr in config_attrs:
- config = config_attr['config']
+ for config in configs:
patterns.append(re.compile(r'#\s*define\s+%s\W' % config))
patterns.append(re.compile(r'#\s*undef\s+%s\W' % config))
@@ -421,17 +419,16 @@ class KconfigParser:
re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"')
re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"')
- def __init__(self, config_attrs, options, progress, build_dir):
+ def __init__(self, configs, options, progress, build_dir):
"""Create a new parser.
Arguments:
- config_attrs: A list of dictionaris, each of them includes the name,
- the type, and the default value of the target config.
+ configs: A list of CONFIGs to move.
options: option flags.
progress: A progress indicator
build_dir: Build directory.
"""
- self.config_attrs = config_attrs
+ self.configs = configs
self.options = options
self.progress = progress
self.build_dir = build_dir
@@ -467,7 +464,7 @@ class KconfigParser:
return CROSS_COMPILE.get(arch, None)
- def parse_one_config(self, config_attr, dotconfig_lines, autoconf_lines):
+ def parse_one_config(self, config, dotconfig_lines, autoconf_lines):
"""Parse .config, defconfig, include/autoconf.mk for one config.
This function looks for the config options in the lines from
@@ -475,8 +472,7 @@ class KconfigParser:
which action should be taken for this defconfig.
Arguments:
- config_attr: A dictionary including the name, the type,
- and the default value of the target config.
+ config: CONFIG name to parse.
dotconfig_lines: lines from the .config file.
autoconf_lines: lines from the include/autoconf.mk file.
@@ -484,7 +480,6 @@ class KconfigParser:
A tupple of the action for this defconfig and the line
matched for the config.
"""
- config = config_attr['config']
not_set = '# %s is not set' % config
for line in dotconfig_lines:
@@ -538,8 +533,8 @@ class KconfigParser:
with open(autoconf_path) as f:
autoconf_lines = f.readlines()
- for config_attr in self.config_attrs:
- result = self.parse_one_config(config_attr, dotconfig_lines,
+ for config in self.configs:
+ result = self.parse_one_config(config, dotconfig_lines,
autoconf_lines)
results.append(result)
@@ -583,12 +578,11 @@ class Slot:
for faster processing.
"""
- def __init__(self, config_attrs, options, progress, devnull, make_cmd):
+ def __init__(self, configs, options, progress, devnull, make_cmd):
"""Create a new process slot.
Arguments:
- config_attrs: A list of dictionaris, each of them includes the name,
- the type, and the default value of the target config.
+ configs: A list of CONFIGs to move.
options: option flags.
progress: A progress indicator.
devnull: A file object of '/dev/null'.
@@ -599,8 +593,7 @@ class Slot:
self.build_dir = tempfile.mkdtemp()
self.devnull = devnull
self.make_cmd = (make_cmd, 'O=' + self.build_dir)
- self.parser = KconfigParser(config_attrs, options, progress,
- self.build_dir)
+ self.parser = KconfigParser(configs, options, progress, self.build_dir)
self.state = STATE_IDLE
self.failed_boards = []
@@ -731,12 +724,11 @@ class Slots:
"""Controller of the array of subprocess slots."""
- def __init__(self, config_attrs, options, progress):
+ def __init__(self, configs, options, progress):
"""Create a new slots controller.
Arguments:
- config_attrs: A list of dictionaris containing the name, the type,
- and the default value of the target CONFIG.
+ configs: A list of CONFIGs to move.
options: option flags.
progress: A progress indicator.
"""
@@ -745,7 +737,7 @@ class Slots:
devnull = get_devnull()
make_cmd = get_make_cmd()
for i in range(options.jobs):
- self.slots.append(Slot(config_attrs, options, progress, devnull,
+ self.slots.append(Slot(configs, options, progress, devnull,
make_cmd))
def add(self, defconfig):
@@ -803,23 +795,18 @@ class Slots:
for board in failed_boards:
f.write(board + '\n')
-def move_config(config_attrs, options):
+def move_config(configs, options):
"""Move config options to defconfig files.
Arguments:
- config_attrs: A list of dictionaris, each of them includes the name,
- the type, and the default value of the target config.
+ configs: A list of CONFIGs to move.
options: option flags
"""
- if len(config_attrs) == 0:
+ if len(configs) == 0:
print 'Nothing to do. exit.'
sys.exit(0)
- print 'Move the following CONFIG options (jobs: %d)' % options.jobs
- for config_attr in config_attrs:
- print ' %s (type: %s, default: %s)' % (config_attr['config'],
- config_attr['type'],
- config_attr['default'])
+ print 'Move %s (jobs: %d)' % (', '.join(configs), options.jobs)
if options.defconfigs:
defconfigs = [line.strip() for line in open(options.defconfigs)]
@@ -838,7 +825,7 @@ def move_config(config_attrs, options):
defconfigs.append(os.path.join(dirpath, filename))
progress = Progress(len(defconfigs))
- slots = Slots(config_attrs, options, progress)
+ slots = Slots(configs, options, progress)
# Main loop to process defconfig files:
# Add a new subprocess into a vacant slot.
@@ -862,7 +849,7 @@ def bad_recipe(filename, linenum, msg):
sys.exit("%s: line %d: error : " % (filename, linenum) + msg)
def parse_recipe(filename):
- """Parse the recipe file and retrieve the config attributes.
+ """Parse the recipe file and retrieve CONFIGs to move.
This function parses the given recipe file and gets the name,
the type, and the default value of the target config options.
@@ -870,10 +857,9 @@ def parse_recipe(filename):
Arguments:
filename: path to file to be parsed.
Returns:
- A list of dictionaris, each of them includes the name,
- the type, and the default value of the target config.
+ A list of CONFIGs to move.
"""
- config_attrs = []
+ configs = []
linenum = 1
for line in open(filename):
@@ -889,43 +875,10 @@ def parse_recipe(filename):
if not config.startswith('CONFIG_'):
config = 'CONFIG_' + config
- # sanity check of default values
- if type == 'bool':
- if not default in ('y', 'n'):
- bad_recipe(filename, linenum,
- "default for bool type must be either y or n")
- elif type == 'tristate':
- if not default in ('y', 'm', 'n'):
- bad_recipe(filename, linenum,
- "default for tristate type must be y, m, or n")
- elif type == 'string':
- if default[0] != '"' or default[-1] != '"':
- bad_recipe(filename, linenum,
- "default for string type must be surrounded by double-quotations")
- elif type == 'int':
- try:
- int(default)
- except:
- bad_recipe(filename, linenum,
- "type is int, but default value is not decimal")
- elif type == 'hex':
- if len(default) < 2 or default[:2] != '0x':
- bad_recipe(filename, linenum,
- "default for hex type must be prefixed with 0x")
- try:
- int(default, 16)
- except:
- bad_recipe(filename, linenum,
- "type is hex, but default value is not hexadecimal")
- else:
- bad_recipe(filename, linenum,
- "unsupported type '%s'. type must be one of bool, tristate, string, int, hex"
- % type)
-
- config_attrs.append({'config': config, 'type': type, 'default': default})
+ configs.append(config)
linenum += 1
- return config_attrs
+ return configs
def main():
try:
@@ -965,7 +918,7 @@ def main():
parser.print_usage()
sys.exit(1)
- config_attrs = parse_recipe(args[0])
+ configs = parse_recipe(args[0])
check_top_directory()
@@ -974,9 +927,9 @@ def main():
update_cross_compile(options.color)
if not options.cleanup_headers_only:
- move_config(config_attrs, options)
+ move_config(configs, options)
- cleanup_headers(config_attrs, options.dry_run)
+ cleanup_headers(configs, options.dry_run)
if __name__ == '__main__':
main()
--
1.9.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 10/21] tools: moveconfig: allow to give CONFIG names as argument directly
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
` (8 preceding siblings ...)
2016-05-19 6:51 ` [U-Boot] [PATCH 09/21] tools: moveconfig: drop code for handling type and default value Masahiro Yamada
@ 2016-05-19 6:51 ` Masahiro Yamada
2016-05-24 14:38 ` Joe Hershberger
2016-05-19 6:51 ` [U-Boot] [PATCH 11/21] tools: moveconfig: add --undef option to move CONFIGs with default y Masahiro Yamada
` (11 subsequent siblings)
21 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-19 6:51 UTC (permalink / raw)
To: u-boot
We still pass the input file with CONFIG name, type, default value
in each line, but the last two fields are just ignored by the tool.
So, let's deprecate the input file and allow users to give CONFIG
names directly from the command line. The types and default values
are automatically detected and handled nicely by the tool.
Going forward, we can use this tool more easily like:
tools/moveconfig.py CONFIG_FOO CONFIG_BAR
Update the documentation and fix some typos I noticed while I was
working on.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---
tools/moveconfig.py | 108 ++++++++++------------------------------------------
1 file changed, 20 insertions(+), 88 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index c452c7a..7ff6e71 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -17,44 +17,16 @@ This tool intends to help this tremendous work.
Usage
-----
-This tool takes one input file. (let's say 'recipe' file here.)
-The recipe describes the list of config options you want to move.
-Each line takes the form:
-<config_name> <type> <default>
-(the fields must be separated with whitespaces.)
-
-<config_name> is the name of config option.
-
-<type> is the type of the option. It must be one of bool, tristate,
-string, int, and hex.
-
-<default> is the default value of the option. It must be appropriate
-value corresponding to the option type. It must be either y or n for
-the bool type. Tristate options can also take m (although U-Boot has
-not supported the module feature).
-
-You can add two or more lines in the recipe file, so you can move
-multiple options at once.
-
-Let's say, for example, you want to move CONFIG_CMD_USB and
-CONFIG_SYS_TEXT_BASE.
-
-The type should be bool, hex, respectively. So, the recipe file
-should look like this:
-
- $ cat recipe
- CONFIG_CMD_USB bool n
- CONFIG_SYS_TEXT_BASE hex 0x00000000
-
-Next you must edit the Kconfig to add the menu entries for the configs
+First, you must edit the Kconfig to add the menu entries for the configs
you are moving.
-And then run this tool giving the file name of the recipe
+And then run this tool giving CONFIG names you want to move.
+For example, if you want to move CONFIG_CMD_USB and CONFIG_SYS_TEXT_BASE,
+simply type as follows:
- $ tools/moveconfig.py recipe
+ $ tools/moveconfig.py CONFIG_CMD_USB CONFIG_SYS_TEXT_BASE
-The tool walks through all the defconfig files to move the config
-options specified by the recipe file.
+The tool walks through all the defconfig files and move the given CONFIGs.
The log is also displayed on the terminal.
@@ -103,19 +75,19 @@ It just uses the regex method, so you should not rely on it.
Just in case, please do 'git diff' to see what happened.
-How does it works?
-------------------
+How does it work?
+-----------------
This tool runs configuration and builds include/autoconf.mk for every
defconfig. The config options defined in Kconfig appear in the .config
file (unless they are hidden because of unmet dependency.)
On the other hand, the config options defined by board headers are seen
in include/autoconf.mk. The tool looks for the specified options in both
-of them to decide the appropriate action for the options. If the option
-is found in the .config or the value is the same as the specified default,
-the option does not need to be touched. If the option is found in
-include/autoconf.mk, but not in the .config, and the value is different
-from the default, the tools adds the option to the defconfig.
+of them to decide the appropriate action for the options. If the given
+config option is found in the .config, but its value does not match the
+one from the board header, the config option in the .config is replaced
+with the define in the board header. Then, the .config is synced by
+"make savedefconfig" and the defconfig is updated with it.
For faster processing, this tool handles multi-threading. It creates
separate build directories where the out-of-tree build is run. The
@@ -148,7 +120,7 @@ Available options
Specify a file containing a list of defconfigs to move
-n, --dry-run
- Peform a trial run that does not make any changes. It is useful to
+ Perform a trial run that does not make any changes. It is useful to
see what is going to happen before one actually runs it.
-e, --exit-on-error
@@ -844,42 +816,6 @@ def move_config(configs, options):
print ''
slots.show_failed_boards()
-def bad_recipe(filename, linenum, msg):
- """Print error message with the file name and the line number and exit."""
- sys.exit("%s: line %d: error : " % (filename, linenum) + msg)
-
-def parse_recipe(filename):
- """Parse the recipe file and retrieve CONFIGs to move.
-
- This function parses the given recipe file and gets the name,
- the type, and the default value of the target config options.
-
- Arguments:
- filename: path to file to be parsed.
- Returns:
- A list of CONFIGs to move.
- """
- configs = []
- linenum = 1
-
- for line in open(filename):
- tokens = line.split()
- if len(tokens) != 3:
- bad_recipe(filename, linenum,
- "%d fields in this line. Each line must contain 3 fields"
- % len(tokens))
-
- (config, type, default) = tokens
-
- # prefix the option name with CONFIG_ if missing
- if not config.startswith('CONFIG_'):
- config = 'CONFIG_' + config
-
- configs.append(config)
- linenum += 1
-
- return configs
-
def main():
try:
cpu_count = multiprocessing.cpu_count()
@@ -904,21 +840,17 @@ def main():
help='the number of jobs to run simultaneously')
parser.add_option('-v', '--verbose', action='store_true', default=False,
help='show any build errors as boards are built')
- parser.usage += ' recipe_file\n\n' + \
- 'The recipe_file should describe config options you want to move.\n' + \
- 'Each line should contain config_name, type, default_value\n\n' + \
- 'Example:\n' + \
- 'CONFIG_FOO bool n\n' + \
- 'CONFIG_BAR int 100\n' + \
- 'CONFIG_BAZ string "hello"\n'
+ parser.usage += ' CONFIG ...'
- (options, args) = parser.parse_args()
+ (options, configs) = parser.parse_args()
- if len(args) != 1:
+ if len(configs) == 0:
parser.print_usage()
sys.exit(1)
- configs = parse_recipe(args[0])
+ # prefix the option name with CONFIG_ if missing
+ configs = [ config if config.startswith('CONFIG_') else 'CONFIG_' + config
+ for config in configs ]
check_top_directory()
--
1.9.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 11/21] tools: moveconfig: add --undef option to move CONFIGs with default y
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
` (9 preceding siblings ...)
2016-05-19 6:51 ` [U-Boot] [PATCH 10/21] tools: moveconfig: allow to give CONFIG names as argument directly Masahiro Yamada
@ 2016-05-19 6:51 ` Masahiro Yamada
2016-05-24 14:38 ` Joe Hershberger
2016-05-19 6:52 ` [U-Boot] [PATCH 12/21] tools: moveconfig: compute file paths just once Masahiro Yamada
` (10 subsequent siblings)
21 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-19 6:51 UTC (permalink / raw)
To: u-boot
Since 96464badc794 ("moveconfig: Always run savedefconfig on the
moved config"), this tool can not correctly move bool configs with
the default value y.
The reason is like follows:
We are supposed to add the config entries in Kconfig for the options
we are moving before running this tool. Otherwise, the moved options
would all disappear during the "make savedefconfig" stage.
Let's say we want to move CONFIG_FOO to Kconfig, making it an option
with "default y". The first thing we need to do is to create an entry
like follows:
config FOO
bool "foo"
default y
So, CONFIG_FOO=y will be set in the .config for every defconfig.
Commit 7740f653e6b3 ("moveconfig: Ignore duplicate configs when
moving") introduced KCONFIG_IGNORE_DUPLICATES to fix the false
negative (= boards that should be converted to "=y" are misconverted
to "not set") problem. With that commit, the CONFIGs in the .config
are now duplicated to include/autoconf.mk unless they are defined to
a different value or #undef'ed in the board header. It causes false
positive problem this time; boards without #define/#undef CONFIG_FOO
in their header should be converted to "not set", but misconverted
to "=y" actually.
This commit intends to handle such cases correctly. Anyway, we need
to create a Kconfig entry beforehand to persevere savedefconfig.
But, we do not want to propagate CONFIGs coming from "default y"
to the include/autoconf.mk. To achieve it, the new option --undef
is here!
We can move bool option with default y in this way:
1. create an entry in Kconfig
2. run "tools/moveconfig.py --undef=CONFIG_FOO CONFIG_FOO"
This option can appear multiple times in the command line, so that
we can move multiple options at a time.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---
scripts/Makefile.autoconf | 3 ++-
tools/moveconfig.py | 39 +++++++++++++++++++++++++++++++++++++++
2 files changed, 41 insertions(+), 1 deletion(-)
diff --git a/scripts/Makefile.autoconf b/scripts/Makefile.autoconf
index 01a739d..57c6e9f 100644
--- a/scripts/Makefile.autoconf
+++ b/scripts/Makefile.autoconf
@@ -39,7 +39,8 @@ UBOOTINCLUDE := \
-include $(srctree)/include/linux/kconfig.h
c_flags := $(KBUILD_CFLAGS) $(KBUILD_CPPFLAGS) $(PLATFORM_CPPFLAGS) \
- $(UBOOTINCLUDE) $(NOSTDINC_FLAGS)
+ $(UBOOTINCLUDE) $(NOSTDINC_FLAGS) \
+ $(KBUILD_MOVECONFIG_FLAGS)
quiet_cmd_autoconf_dep = GEN $@
cmd_autoconf_dep = $(CC) -x c -DDO_DEPS_ONLY -M -MP $(c_flags) \
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 7ff6e71..b509f49 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -74,6 +74,27 @@ from the config headers (include/configs/*.h).
It just uses the regex method, so you should not rely on it.
Just in case, please do 'git diff' to see what happened.
+How to move bool option with 'default y'?
+-----------------------------------------
+
+Moving bool options with 'default y' is a bit tricky. In general use
+of this tool, CONFIGs set in the .config are all duplicated to the
+include/autoconf.mk, that is, CONFIGs with 'default y' Kconfig entry
+will squelch the defines that came from the board header. In order to
+know whether the board header defined those CONFIGs or not, you must
+stop the target CONFIGs from being propagated to the include/autoconf.mk.
+The option -U (--undef) can be used to do this.
+
+For example, if you want to move CONFIG_FOO and make it an option with
+'default y', you need to do create an entry in Kconfig
+
+ config FOO
+ bool "foo"
+ default y
+
+and then run
+
+ $ tools/moveconfig.py -U CONFIG_FOO CONFIG_FOO
How does it work?
-----------------
@@ -134,6 +155,10 @@ Available options
Specify the number of threads to run simultaneously. If not specified,
the number of threads is the same as the number of CPU cores.
+ -U, --undef
+ Undefine the given CONFIG option when generating include/autoconf.mk.
+ This is generally useful to move a boolean option with "default y".
+
-v, --verbose
Show any build errors as boards are built
@@ -569,6 +594,13 @@ class Slot:
self.state = STATE_IDLE
self.failed_boards = []
+ if options.undef:
+ undef_path = os.path.join(self.build_dir, 'undef.h')
+ with open(undef_path, 'w') as f:
+ for config in options.undef:
+ f.write('#undef %s\n' % config)
+ self.make_cmd += ("KBUILD_MOVECONFIG_FLAGS=-include %s" % undef_path, )
+
def __del__(self):
"""Delete the working directory
@@ -838,6 +870,9 @@ def main():
help='only cleanup the headers')
parser.add_option('-j', '--jobs', type='int', default=cpu_count,
help='the number of jobs to run simultaneously')
+ parser.add_option('-U', '--undef', action='append', default=[],
+ help='macro to undefine for autoconf processing' + \
+ ' (can be given multiple times)')
parser.add_option('-v', '--verbose', action='store_true', default=False,
help='show any build errors as boards are built')
parser.usage += ' CONFIG ...'
@@ -852,6 +887,10 @@ def main():
configs = [ config if config.startswith('CONFIG_') else 'CONFIG_' + config
for config in configs ]
+ # likewise for options.undef
+ options.undef = [ config if config.startswith('CONFIG_')
+ else 'CONFIG_' + config for config in options.undef ]
+
check_top_directory()
check_clean_directory()
--
1.9.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 12/21] tools: moveconfig: compute file paths just once
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
` (10 preceding siblings ...)
2016-05-19 6:51 ` [U-Boot] [PATCH 11/21] tools: moveconfig: add --undef option to move CONFIGs with default y Masahiro Yamada
@ 2016-05-19 6:52 ` Masahiro Yamada
2016-05-24 14:54 ` Joe Hershberger
2016-05-19 6:52 ` [U-Boot] [PATCH 13/21] tools: moveconfig: move log output code out of Kconfig Parser class Masahiro Yamada
` (9 subsequent siblings)
21 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-19 6:52 UTC (permalink / raw)
To: u-boot
The paths to .config, include/autoconf.mk, include/config/auto.conf
are not changed during the defconfig walk. Compute them only once
when a new class instance is created.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---
tools/moveconfig.py | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index b509f49..9f38a08 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -428,7 +428,10 @@ class KconfigParser:
self.configs = configs
self.options = options
self.progress = progress
- self.build_dir = build_dir
+ self.dotconfig = os.path.join(build_dir, '.config')
+ self.autoconf = os.path.join(build_dir, 'include', 'autoconf.mk')
+ self.config_autoconf = os.path.join(build_dir, 'include', 'config',
+ 'auto.conf')
def get_cross_compile(self):
"""Parse .config file and return CROSS_COMPILE.
@@ -442,8 +445,7 @@ class KconfigParser:
"""
arch = ''
cpu = ''
- dotconfig = os.path.join(self.build_dir, '.config')
- for line in open(dotconfig):
+ for line in open(self.dotconfig):
m = self.re_arch.match(line)
if m:
arch = m.group(1)
@@ -520,14 +522,12 @@ class KconfigParser:
defconfig: defconfig name.
"""
- dotconfig_path = os.path.join(self.build_dir, '.config')
- autoconf_path = os.path.join(self.build_dir, 'include', 'autoconf.mk')
results = []
- with open(dotconfig_path) as f:
+ with open(self.dotconfig) as f:
dotconfig_lines = f.readlines()
- with open(autoconf_path) as f:
+ with open(self.autoconf) as f:
autoconf_lines = f.readlines()
for config in self.configs:
@@ -558,13 +558,13 @@ class KconfigParser:
print log,
self.progress.show()
- with open(dotconfig_path, 'a') as f:
+ with open(self.dotconfig, 'a') as f:
for (action, value) in results:
if action == ACTION_MOVE:
f.write(value + '\n')
- os.remove(os.path.join(self.build_dir, 'include', 'config', 'auto.conf'))
- os.remove(autoconf_path)
+ os.remove(self.config_autoconf)
+ os.remove(self.autoconf)
class Slot:
--
1.9.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 13/21] tools: moveconfig: move log output code out of Kconfig Parser class
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
` (11 preceding siblings ...)
2016-05-19 6:52 ` [U-Boot] [PATCH 12/21] tools: moveconfig: compute file paths just once Masahiro Yamada
@ 2016-05-19 6:52 ` Masahiro Yamada
2016-05-24 14:54 ` Joe Hershberger
2016-05-19 6:52 ` [U-Boot] [PATCH 14/21] tools: moveconfig: display log atomically in more readable format Masahiro Yamada
` (8 subsequent siblings)
21 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-19 6:52 UTC (permalink / raw)
To: u-boot
This will help further improvement/clean-up.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---
tools/moveconfig.py | 24 ++++++++++++------------
1 file changed, 12 insertions(+), 12 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 9f38a08..96ada0d 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -416,18 +416,16 @@ class KconfigParser:
re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"')
re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"')
- def __init__(self, configs, options, progress, build_dir):
+ def __init__(self, configs, options, build_dir):
"""Create a new parser.
Arguments:
configs: A list of CONFIGs to move.
options: option flags.
- progress: A progress indicator
build_dir: Build directory.
"""
self.configs = configs
self.options = options
- self.progress = progress
self.dotconfig = os.path.join(build_dir, '.config')
self.autoconf = os.path.join(build_dir, 'include', 'autoconf.mk')
self.config_autoconf = os.path.join(build_dir, 'include', 'config',
@@ -516,10 +514,12 @@ class KconfigParser:
This function parses the generated .config and include/autoconf.mk
searching the target options.
Move the config option(s) to the .config as needed.
- Also, display the log to show what happened to the .config.
Arguments:
defconfig: defconfig name.
+
+ Returns:
+ Return log string
"""
results = []
@@ -553,11 +553,6 @@ class KconfigParser:
log += log_msg(self.options.color, log_color, defconfig, actlog)
- # Some threads are running in parallel.
- # Print log in one shot to not mix up logs from different threads.
- print log,
- self.progress.show()
-
with open(self.dotconfig, 'a') as f:
for (action, value) in results:
if action == ACTION_MOVE:
@@ -566,6 +561,8 @@ class KconfigParser:
os.remove(self.config_autoconf)
os.remove(self.autoconf)
+ return log
+
class Slot:
"""A slot to store a subprocess.
@@ -590,7 +587,7 @@ class Slot:
self.build_dir = tempfile.mkdtemp()
self.devnull = devnull
self.make_cmd = (make_cmd, 'O=' + self.build_dir)
- self.parser = KconfigParser(configs, options, progress, self.build_dir)
+ self.parser = KconfigParser(configs, options, self.build_dir)
self.state = STATE_IDLE
self.failed_boards = []
@@ -676,7 +673,7 @@ class Slot:
return True
if self.state == STATE_AUTOCONF:
- self.parser.update_dotconfig(self.defconfig)
+ self.log = self.parser.update_dotconfig(self.defconfig)
"""Save off the defconfig in a consistent way"""
cmd = list(self.make_cmd)
@@ -690,7 +687,11 @@ class Slot:
if not self.options.dry_run:
shutil.move(os.path.join(self.build_dir, 'defconfig'),
os.path.join('configs', self.defconfig))
+ # Some threads are running in parallel.
+ # Print log in one shot to not mix up logs from different threads.
+ print self.log,
self.progress.inc()
+ self.progress.show()
self.state = STATE_IDLE
return True
@@ -844,7 +845,6 @@ def move_config(configs, options):
while not slots.empty():
time.sleep(SLEEP_TIME)
- progress.show()
print ''
slots.show_failed_boards()
--
1.9.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 14/21] tools: moveconfig: display log atomically in more readable format
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
` (12 preceding siblings ...)
2016-05-19 6:52 ` [U-Boot] [PATCH 13/21] tools: moveconfig: move log output code out of Kconfig Parser class Masahiro Yamada
@ 2016-05-19 6:52 ` Masahiro Yamada
2016-05-24 14:54 ` Joe Hershberger
2016-05-19 6:52 ` [U-Boot] [PATCH 15/21] tools: moveconfig: refactor code to go back to idle state Masahiro Yamada
` (7 subsequent siblings)
21 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-19 6:52 UTC (permalink / raw)
To: u-boot
Before this commit, the log was displayed in the format:
<defconfig_name> : <action1>
<defconfig_name> : <action2>
<defconfig_name> : <action3>
When we move multiple CONFIGs at the same time, we see as many
<defconfig_name> strings as actions for every defconfig, which is
redundant information.
Moreover, since normal log and error log are displayed separately,
Messages from different threads could be mixed, like this:
<foo> : <action1>
<foo> : <action2>
<bar> : <action1>
<bar> : <action2>
<foo> : <error_log>
This commit makes sure to call "print" once a defconfig, which
enables atomic logging for each defconfig. It also makes it
possible to refactor the log format as follows:
<foo_defconfig>
<action1>
<action2>
<error_log>
<bar_defconfig>
<action1>
<action2>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---
tools/moveconfig.py | 73 ++++++++++++++++++++++++++++++-----------------------
1 file changed, 42 insertions(+), 31 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 96ada0d..cf4004f 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -30,13 +30,17 @@ The tool walks through all the defconfig files and move the given CONFIGs.
The log is also displayed on the terminal.
-Each line is printed in the format
-<defconfig_name> : <action>
+The log is printed for each defconfig as follows:
-<defconfig_name> is the name of the defconfig
-(without the suffix _defconfig).
+<defconfig_name>
+ <action1>
+ <action2>
+ <action3>
+ ...
-<action> shows what the tool did for that defconfig.
+<defconfig_name> is the name of the defconfig.
+
+<action*> shows what the tool did for that defconfig.
It looks like one of the followings:
- Move 'CONFIG_... '
@@ -274,15 +278,13 @@ def get_make_cmd():
def color_text(color_enabled, color, string):
"""Return colored string."""
if color_enabled:
- return '\033[' + color + 'm' + string + '\033[0m'
+ # LF should not be surrounded by the escape sequence.
+ # Otherwise, additional whitespace or line-feed might be printed.
+ return '\n'.join([ '\033[' + color + 'm' + s + '\033[0m' if s else ''
+ for s in string.split('\n') ])
else:
return string
-def log_msg(color_enabled, color, defconfig, msg):
- """Return the formated line for the log."""
- return defconfig[:-len('_defconfig')].ljust(37) + ': ' + \
- color_text(color_enabled, color, msg) + '\n'
-
def update_cross_compile(color_enabled):
"""Update per-arch CROSS_COMPILE via environment variables
@@ -508,7 +510,7 @@ class KconfigParser:
return (ACTION_MOVE, new_val)
- def update_dotconfig(self, defconfig):
+ def update_dotconfig(self):
"""Parse files for the config options and update the .config.
This function parses the generated .config and include/autoconf.mk
@@ -551,7 +553,7 @@ class KconfigParser:
else:
sys.exit("Internal Error. This should not happen.")
- log += log_msg(self.options.color, log_color, defconfig, actlog)
+ log += color_text(self.options.color, log_color, actlog) + '\n'
with open(self.dotconfig, 'a') as f:
for (action, value) in results:
@@ -634,6 +636,7 @@ class Slot:
stderr=subprocess.PIPE)
self.defconfig = defconfig
self.state = STATE_DEFCONFIG
+ self.log = ''
return True
def poll(self):
@@ -656,14 +659,12 @@ class Slot:
return False
if self.ps.poll() != 0:
- print >> sys.stderr, log_msg(self.options.color, COLOR_LIGHT_RED,
- self.defconfig, "Failed to process."),
+ self.log += color_text(self.options.color, COLOR_LIGHT_RED,
+ "Failed to process.\n")
if self.options.verbose:
- print >> sys.stderr, color_text(self.options.color,
- COLOR_LIGHT_CYAN,
- self.ps.stderr.read())
- self.progress.inc()
- self.progress.show()
+ self.log += color_text(self.options.color, COLOR_LIGHT_CYAN,
+ self.ps.stderr.read())
+ self.show_log(sys.stderr)
if self.options.exit_on_error:
sys.exit("Exit on error.")
# If --exit-on-error flag is not set, skip this board and continue.
@@ -673,7 +674,7 @@ class Slot:
return True
if self.state == STATE_AUTOCONF:
- self.log = self.parser.update_dotconfig(self.defconfig)
+ self.log += self.parser.update_dotconfig()
"""Save off the defconfig in a consistent way"""
cmd = list(self.make_cmd)
@@ -687,21 +688,15 @@ class Slot:
if not self.options.dry_run:
shutil.move(os.path.join(self.build_dir, 'defconfig'),
os.path.join('configs', self.defconfig))
- # Some threads are running in parallel.
- # Print log in one shot to not mix up logs from different threads.
- print self.log,
- self.progress.inc()
- self.progress.show()
+ self.show_log()
self.state = STATE_IDLE
return True
self.cross_compile = self.parser.get_cross_compile()
if self.cross_compile is None:
- print >> sys.stderr, log_msg(self.options.color, COLOR_YELLOW,
- self.defconfig,
- "Compiler is missing. Do nothing."),
- self.progress.inc()
- self.progress.show()
+ self.log += color_text(self.options.color, COLOR_YELLOW,
+ "Compiler is missing. Do nothing.\n")
+ self.show_log(sys.stderr)
if self.options.exit_on_error:
sys.exit("Exit on error.")
# If --exit-on-error flag is not set, skip this board and continue.
@@ -720,6 +715,22 @@ class Slot:
self.state = STATE_AUTOCONF
return False
+ def show_log(self, file=sys.stdout):
+ """Display log along with progress.
+
+ Arguments:
+ file: A file object to which the log string is sent.
+ """
+ # output at least 30 characters to hide the "* defconfigs out of *".
+ log = self.defconfig.ljust(30) + '\n'
+
+ log += '\n'.join([ ' ' + s for s in self.log.split('\n') ])
+ # Some threads are running in parallel.
+ # Print log atomically to not mix up logs from different threads.
+ print >> file, log
+ self.progress.inc()
+ self.progress.show()
+
def get_failed_boards(self):
"""Returns a list of failed boards (defconfigs) in this slot.
"""
--
1.9.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 15/21] tools: moveconfig: refactor code to go back to idle state
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
` (13 preceding siblings ...)
2016-05-19 6:52 ` [U-Boot] [PATCH 14/21] tools: moveconfig: display log atomically in more readable format Masahiro Yamada
@ 2016-05-19 6:52 ` Masahiro Yamada
2016-05-24 14:54 ` Joe Hershberger
2016-05-19 6:52 ` [U-Boot] [PATCH 16/21] tools: moveconfig: skip savedefconfig if .config was not updated Masahiro Yamada
` (6 subsequent siblings)
21 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-19 6:52 UTC (permalink / raw)
To: u-boot
Move similar code to finish() function.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---
tools/moveconfig.py | 37 +++++++++++++++++--------------------
1 file changed, 17 insertions(+), 20 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index cf4004f..da7120d 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -664,13 +664,7 @@ class Slot:
if self.options.verbose:
self.log += color_text(self.options.color, COLOR_LIGHT_CYAN,
self.ps.stderr.read())
- self.show_log(sys.stderr)
- if self.options.exit_on_error:
- sys.exit("Exit on error.")
- # If --exit-on-error flag is not set, skip this board and continue.
- # Record the failed board.
- self.failed_boards.append(self.defconfig)
- self.state = STATE_IDLE
+ self.finish(False)
return True
if self.state == STATE_AUTOCONF:
@@ -688,21 +682,14 @@ class Slot:
if not self.options.dry_run:
shutil.move(os.path.join(self.build_dir, 'defconfig'),
os.path.join('configs', self.defconfig))
- self.show_log()
- self.state = STATE_IDLE
+ self.finish(True)
return True
self.cross_compile = self.parser.get_cross_compile()
if self.cross_compile is None:
self.log += color_text(self.options.color, COLOR_YELLOW,
"Compiler is missing. Do nothing.\n")
- self.show_log(sys.stderr)
- if self.options.exit_on_error:
- sys.exit("Exit on error.")
- # If --exit-on-error flag is not set, skip this board and continue.
- # Record the failed board.
- self.failed_boards.append(self.defconfig)
- self.state = STATE_IDLE
+ self.finish(False)
return True
cmd = list(self.make_cmd)
@@ -715,11 +702,12 @@ class Slot:
self.state = STATE_AUTOCONF
return False
- def show_log(self, file=sys.stdout):
- """Display log along with progress.
+ def finish(self, success):
+ """Display log along with progress and go to the idle state.
Arguments:
- file: A file object to which the log string is sent.
+ success: Should be True when the defconfig was processed
+ successfully, or False when it fails.
"""
# output at least 30 characters to hide the "* defconfigs out of *".
log = self.defconfig.ljust(30) + '\n'
@@ -727,9 +715,18 @@ class Slot:
log += '\n'.join([ ' ' + s for s in self.log.split('\n') ])
# Some threads are running in parallel.
# Print log atomically to not mix up logs from different threads.
- print >> file, log
+ print >> (sys.stdout if success else sys.stderr), log
+
+ if not success:
+ if self.options.exit_on_error:
+ sys.exit("Exit on error.")
+ # If --exit-on-error flag is not set, skip this board and continue.
+ # Record the failed board.
+ self.failed_boards.append(self.defconfig)
+
self.progress.inc()
self.progress.show()
+ self.state = STATE_IDLE
def get_failed_boards(self):
"""Returns a list of failed boards (defconfigs) in this slot.
--
1.9.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 16/21] tools: moveconfig: skip savedefconfig if .config was not updated
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
` (14 preceding siblings ...)
2016-05-19 6:52 ` [U-Boot] [PATCH 15/21] tools: moveconfig: refactor code to go back to idle state Masahiro Yamada
@ 2016-05-19 6:52 ` Masahiro Yamada
2016-05-24 14:55 ` Joe Hershberger
2016-05-19 6:52 ` [U-Boot] [PATCH 17/21] tools: moveconfig: display log when savedefconfig occurs Masahiro Yamada
` (5 subsequent siblings)
21 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-19 6:52 UTC (permalink / raw)
To: u-boot
If no CONFIG option is moved to the .config, no need to sync the
defconfig file. This accelerates the processing by skipping
unneeded "make savedefconfig".
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---
tools/moveconfig.py | 21 ++++++++++++++++-----
1 file changed, 16 insertions(+), 5 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index da7120d..cb26b14 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -521,10 +521,13 @@ class KconfigParser:
defconfig: defconfig name.
Returns:
- Return log string
+ Return a tuple of (updated flag, log string).
+ The "updated flag" is True if the .config was updated, False
+ otherwise. The "log string" shows what happend to the .config.
"""
results = []
+ updated = False
with open(self.dotconfig) as f:
dotconfig_lines = f.readlines()
@@ -559,11 +562,12 @@ class KconfigParser:
for (action, value) in results:
if action == ACTION_MOVE:
f.write(value + '\n')
+ updated = True
os.remove(self.config_autoconf)
os.remove(self.autoconf)
- return log
+ return (updated, log)
class Slot:
@@ -646,8 +650,11 @@ class Slot:
If the configuration is successfully finished, assign a new
subprocess to build include/autoconf.mk.
If include/autoconf.mk is generated, invoke the parser to
- parse the .config and the include/autoconf.mk, and then set the
- slot back to the idle state.
+ parse the .config and the include/autoconf.mk, moving
+ config options to the .config as needed.
+ If the .config was updated, run "make savedefconfig" to sync
+ it, update the original defconfig, and then set the slot back
+ to the idle state.
Returns:
Return True if the subprocess is terminated, False otherwise
@@ -668,8 +675,12 @@ class Slot:
return True
if self.state == STATE_AUTOCONF:
- self.log += self.parser.update_dotconfig()
+ (updated, log) = self.parser.update_dotconfig()
+ self.log += log
+ if not updated:
+ self.finish(True)
+ return True
"""Save off the defconfig in a consistent way"""
cmd = list(self.make_cmd)
cmd.append('savedefconfig')
--
1.9.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 17/21] tools: moveconfig: display log when savedefconfig occurs
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
` (15 preceding siblings ...)
2016-05-19 6:52 ` [U-Boot] [PATCH 16/21] tools: moveconfig: skip savedefconfig if .config was not updated Masahiro Yamada
@ 2016-05-19 6:52 ` Masahiro Yamada
2016-05-24 14:55 ` Joe Hershberger
2016-05-19 6:52 ` [U-Boot] [PATCH 18/21] tools: moveconfig: report when CONFIGs are removed by savedefconfig Masahiro Yamada
` (4 subsequent siblings)
21 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-19 6:52 UTC (permalink / raw)
To: u-boot
Now, "make savedefconfig" does not always happen. Display the log
when it happens.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---
tools/moveconfig.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index cb26b14..80542c5 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -681,7 +681,8 @@ class Slot:
if not updated:
self.finish(True)
return True
- """Save off the defconfig in a consistent way"""
+ self.log += color_text(self.options.color, COLOR_LIGHT_GREEN,
+ "Syncing by savedefconfig...\n")
cmd = list(self.make_cmd)
cmd.append('savedefconfig')
self.ps = subprocess.Popen(cmd, stdout=self.devnull,
--
1.9.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 18/21] tools: moveconfig: report when CONFIGs are removed by savedefconfig
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
` (16 preceding siblings ...)
2016-05-19 6:52 ` [U-Boot] [PATCH 17/21] tools: moveconfig: display log when savedefconfig occurs Masahiro Yamada
@ 2016-05-19 6:52 ` Masahiro Yamada
2016-05-24 14:55 ` Joe Hershberger
2016-05-19 6:52 ` [U-Boot] [PATCH 19/21] tools: moveconfig: report when defconfig is updated Masahiro Yamada
` (3 subsequent siblings)
21 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-19 6:52 UTC (permalink / raw)
To: u-boot
This is a rare case, but there is still possibility that some CONFIG
is moved to the .config, but it is removed by "make savedefconfig".
(For example, it happens when the specified CONFIG has no prompt in
the Kconfig entry, i.e. it is not user-configurable.)
It might be an unexpected case. So, display the log in this case
(in yellow color to gain user's attention if --color option is given).
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---
tools/moveconfig.py | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 80542c5..9c73b30 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -432,6 +432,7 @@ class KconfigParser:
self.autoconf = os.path.join(build_dir, 'include', 'autoconf.mk')
self.config_autoconf = os.path.join(build_dir, 'include', 'config',
'auto.conf')
+ self.defconfig = os.path.join(build_dir, 'defconfig')
def get_cross_compile(self):
"""Parse .config file and return CROSS_COMPILE.
@@ -564,11 +565,35 @@ class KconfigParser:
f.write(value + '\n')
updated = True
+ self.results = results
os.remove(self.config_autoconf)
os.remove(self.autoconf)
return (updated, log)
+ def check_defconfig(self):
+ """Check the defconfig after savedefconfig
+
+ Returns:
+ Return additional log if moved CONFIGs were removed again by
+ 'make savedefconfig'.
+ """
+
+ log = ''
+
+ with open(self.defconfig) as f:
+ defconfig_lines = f.readlines()
+
+ for (action, value) in self.results:
+ if action != ACTION_MOVE:
+ continue
+ if not value + '\n' in defconfig_lines:
+ log += color_text(self.options.color, COLOR_YELLOW,
+ "'%s' was removed by savedefconfig.\n" %
+ value)
+
+ return log
+
class Slot:
"""A slot to store a subprocess.
@@ -691,6 +716,7 @@ class Slot:
return False
if self.state == STATE_SAVEDEFCONFIG:
+ self.log += self.parser.check_defconfig()
if not self.options.dry_run:
shutil.move(os.path.join(self.build_dir, 'defconfig'),
os.path.join('configs', self.defconfig))
--
1.9.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 19/21] tools: moveconfig: report when defconfig is updated
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
` (17 preceding siblings ...)
2016-05-19 6:52 ` [U-Boot] [PATCH 18/21] tools: moveconfig: report when CONFIGs are removed by savedefconfig Masahiro Yamada
@ 2016-05-19 6:52 ` Masahiro Yamada
2016-05-24 14:55 ` Joe Hershberger
2016-05-19 6:52 ` [U-Boot] [PATCH 20/21] tools: moveconfig: add --force-sync option Masahiro Yamada
` (2 subsequent siblings)
21 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-19 6:52 UTC (permalink / raw)
To: u-boot
There are various factors that determine if the given defconfig is
updated, and it is probably what users are more interested in.
Show the log when the defconfig is updated. Also, copy the file
only when the file content was really updated to avoid changing
the time stamp needlessly.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---
tools/moveconfig.py | 14 +++++++++++---
1 file changed, 11 insertions(+), 3 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 9c73b30..65cc5b3 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -172,6 +172,7 @@ To see the complete list of supported options, run
"""
+import filecmp
import fnmatch
import multiprocessing
import optparse
@@ -717,9 +718,16 @@ class Slot:
if self.state == STATE_SAVEDEFCONFIG:
self.log += self.parser.check_defconfig()
- if not self.options.dry_run:
- shutil.move(os.path.join(self.build_dir, 'defconfig'),
- os.path.join('configs', self.defconfig))
+ orig_defconfig = os.path.join('configs', self.defconfig)
+ new_defconfig = os.path.join(self.build_dir, 'defconfig')
+ updated = not filecmp.cmp(orig_defconfig, new_defconfig)
+
+ if updated:
+ self.log += color_text(self.options.color, COLOR_LIGHT_GREEN,
+ "defconfig was updated.\n")
+
+ if not self.options.dry_run and updated:
+ shutil.move(new_defconfig, orig_defconfig)
self.finish(True)
return True
--
1.9.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 20/21] tools: moveconfig: add --force-sync option
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
` (18 preceding siblings ...)
2016-05-19 6:52 ` [U-Boot] [PATCH 19/21] tools: moveconfig: report when defconfig is updated Masahiro Yamada
@ 2016-05-19 6:52 ` Masahiro Yamada
2016-05-24 14:55 ` Joe Hershberger
2016-05-19 6:52 ` [U-Boot] [PATCH 21/21] tools: moveconfig: allow to run without any CONFIG specified Masahiro Yamada
2016-05-23 22:13 ` [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Tom Rini
21 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-19 6:52 UTC (permalink / raw)
To: u-boot
Now, this tools invokes "make savedefconfig" only when it needs to
do so, but there might be cases where a user wants the tool to do
savedefconfig forcibly, for example, some defconfigs were already
out of sync and the user wants to fix it as well.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---
tools/moveconfig.py | 17 ++++++++++++++---
1 file changed, 14 insertions(+), 3 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 65cc5b3..acc3dbb 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -152,6 +152,11 @@ Available options
Exit immediately if Make exits with a non-zero status while processing
a defconfig file.
+ -s, --force-sync
+ Do "make savedefconfig" forcibly for all the defconfig files.
+ If not specified, "make savedefconfig" only occurs for cases
+ where at least one CONFIG was moved.
+
-H, --headers-only
Only cleanup the headers; skip the defconfig processing
@@ -704,11 +709,15 @@ class Slot:
(updated, log) = self.parser.update_dotconfig()
self.log += log
- if not updated:
+ if not self.options.force_sync and not updated:
self.finish(True)
return True
- self.log += color_text(self.options.color, COLOR_LIGHT_GREEN,
- "Syncing by savedefconfig...\n")
+ if updated:
+ self.log += color_text(self.options.color, COLOR_LIGHT_GREEN,
+ "Syncing by savedefconfig...\n")
+ else:
+ self.log += "Syncing by savedefconfig (forced by option)...\n"
+
cmd = list(self.make_cmd)
cmd.append('savedefconfig')
self.ps = subprocess.Popen(cmd, stdout=self.devnull,
@@ -919,6 +928,8 @@ def main():
parser.add_option('-e', '--exit-on-error', action='store_true',
default=False,
help='exit immediately on any error')
+ parser.add_option('-s', '--force-sync', action='store_true', default=False,
+ help='force sync by savedefconfig')
parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
action='store_true', default=False,
help='only cleanup the headers')
--
1.9.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 21/21] tools: moveconfig: allow to run without any CONFIG specified
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
` (19 preceding siblings ...)
2016-05-19 6:52 ` [U-Boot] [PATCH 20/21] tools: moveconfig: add --force-sync option Masahiro Yamada
@ 2016-05-19 6:52 ` Masahiro Yamada
2016-05-24 14:55 ` Joe Hershberger
2016-05-23 22:13 ` [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Tom Rini
21 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-19 6:52 UTC (permalink / raw)
To: u-boot
I found "tools/moveconfig -s" might be useful for defconfig re-sync.
I could optimize it for re-sync if I wanted, but I do not want to
make the code complex for this feature.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---
tools/moveconfig.py | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index acc3dbb..a9f7c50 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -871,10 +871,13 @@ def move_config(configs, options):
options: option flags
"""
if len(configs) == 0:
- print 'Nothing to do. exit.'
- sys.exit(0)
-
- print 'Move %s (jobs: %d)' % (', '.join(configs), options.jobs)
+ if options.force_sync:
+ print 'No CONFIG is specified. You are probably syncing defconfigs.',
+ else:
+ print 'Neigher CONFIG nor --force-sync is specified. Nothing will happen.',
+ else:
+ print 'Move ' + ', '.join(configs),
+ print '(jobs: %d)\n' % options.jobs
if options.defconfigs:
defconfigs = [line.strip() for line in open(options.defconfigs)]
@@ -944,7 +947,7 @@ def main():
(options, configs) = parser.parse_args()
- if len(configs) == 0:
+ if len(configs) == 0 and not options.force_sync:
parser.print_usage()
sys.exit(1)
--
1.9.1
^ permalink raw reply related [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
` (20 preceding siblings ...)
2016-05-19 6:52 ` [U-Boot] [PATCH 21/21] tools: moveconfig: allow to run without any CONFIG specified Masahiro Yamada
@ 2016-05-23 22:13 ` Tom Rini
2016-05-24 1:27 ` Masahiro Yamada
21 siblings, 1 reply; 50+ messages in thread
From: Tom Rini @ 2016-05-23 22:13 UTC (permalink / raw)
To: u-boot
On Thu, May 19, 2016 at 03:51:48PM +0900, Masahiro Yamada wrote:
> Masahiro Yamada (21):
> tools: moveconfig: fix --dry-run option
> tools: moveconfig: rename update_defconfig() to update_dotconfig()
> tools: moveconfig: remove redundant else: after sys.exit()
> tools: moveconfig: check directory relocation before compilers
> tools: moveconfig: check compilers before starting defconfig walk
> tools: moveconfig: exit with error message for not clean directory
> tools: moveconfig: increment number of processed files monotonically
> tools: moveconfig: do not rely on type and default value given by
> users
> tools: moveconfig: drop code for handling type and default value
> tools: moveconfig: allow to give CONFIG names as argument directly
> tools: moveconfig: add --undef option to move CONFIGs with default y
> tools: moveconfig: compute file paths just once
> tools: moveconfig: move log output code out of Kconfig Parser class
> tools: moveconfig: display log atomically in more readable format
> tools: moveconfig: refactor code to go back to idle state
> tools: moveconfig: skip savedefconfig if .config was not updated
> tools: moveconfig: display log when savedefconfig occurs
> tools: moveconfig: report when CONFIGs are removed by savedefconfig
> tools: moveconfig: report when defconfig is updated
> tools: moveconfig: add --force-sync option
> tools: moveconfig: allow to run without any CONFIG specified
>
> scripts/Makefile.autoconf | 3 +-
> tools/moveconfig.py | 621 +++++++++++++++++++++++++---------------------
> 2 files changed, 339 insertions(+), 285 deletions(-)
Assuming no comments come up, please feel free to throw this into a
branch in the uniphier repository for me to pick up, thanks!
--
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20160523/3b323a1b/attachment.sig>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups
2016-05-23 22:13 ` [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Tom Rini
@ 2016-05-24 1:27 ` Masahiro Yamada
2016-06-12 22:30 ` Masahiro Yamada
0 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-24 1:27 UTC (permalink / raw)
To: u-boot
2016-05-24 7:13 GMT+09:00 Tom Rini <trini@konsulko.com>:
> On Thu, May 19, 2016 at 03:51:48PM +0900, Masahiro Yamada wrote:
>
>> Masahiro Yamada (21):
>> tools: moveconfig: fix --dry-run option
>> tools: moveconfig: rename update_defconfig() to update_dotconfig()
>> tools: moveconfig: remove redundant else: after sys.exit()
>> tools: moveconfig: check directory relocation before compilers
>> tools: moveconfig: check compilers before starting defconfig walk
>> tools: moveconfig: exit with error message for not clean directory
>> tools: moveconfig: increment number of processed files monotonically
>> tools: moveconfig: do not rely on type and default value given by
>> users
>> tools: moveconfig: drop code for handling type and default value
>> tools: moveconfig: allow to give CONFIG names as argument directly
>> tools: moveconfig: add --undef option to move CONFIGs with default y
>> tools: moveconfig: compute file paths just once
>> tools: moveconfig: move log output code out of Kconfig Parser class
>> tools: moveconfig: display log atomically in more readable format
>> tools: moveconfig: refactor code to go back to idle state
>> tools: moveconfig: skip savedefconfig if .config was not updated
>> tools: moveconfig: display log when savedefconfig occurs
>> tools: moveconfig: report when CONFIGs are removed by savedefconfig
>> tools: moveconfig: report when defconfig is updated
>> tools: moveconfig: add --force-sync option
>> tools: moveconfig: allow to run without any CONFIG specified
>>
>> scripts/Makefile.autoconf | 3 +-
>> tools/moveconfig.py | 621 +++++++++++++++++++++++++---------------------
>> 2 files changed, 339 insertions(+), 285 deletions(-)
>
> Assuming no comments come up, please feel free to throw this into a
> branch in the uniphier repository for me to pick up, thanks!
>
OK, I will wait a few days more in case somebody
is interested in this series,
then apply it.
--
Best Regards
Masahiro Yamada
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 08/21] tools: moveconfig: do not rely on type and default value given by users
2016-05-19 6:51 ` [U-Boot] [PATCH 08/21] tools: moveconfig: do not rely on type and default value given by users Masahiro Yamada
@ 2016-05-24 14:27 ` Joe Hershberger
2016-06-12 22:20 ` Masahiro Yamada
0 siblings, 1 reply; 50+ messages in thread
From: Joe Hershberger @ 2016-05-24 14:27 UTC (permalink / raw)
To: u-boot
On Thu, May 19, 2016 at 1:51 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Commit 96464badc794 ("moveconfig: Always run savedefconfig on the
> moved config") changed the work flow of this tool a lot from the
> original intention when this tool was designed first.
>
> Since then, before running this tool, users must edit the Kconfig to
> add the menu entries for the configs they are moving. It means users
> had already specified the type and the default value for each CONFIG
> via the Kconfig entries. Nevertheless, users are still required to
> dictate the same type and the default value in each line of the input
> file. So, my idea here is to deprecate the latter.
>
> Before moving forward with it, there is one thing worth mentioning;
> since the savedefconfig re-sync was introduced, there is a case this
> tool can not handle correctly.
>
> Let's say we are moving CONFIG_FOO from board headers to Kconfig,
> and we want to make it "default y" option. First, we are supposed
> to create an entry:
>
> config FOO
> bool "foo"
> default y
>
> CONFIG_FOO=y will appear in the .config for every processed defconfig.
> It will be duplicated to the include/autoconf.mk as well unless the
> board explicitly #undef's it. Therefore, the defconfig without
> "#define CONFIG_FOO" or "#undef CONFIG_FOO" that should be converted
> to "unset" would be misconverted to "=y".
I solved this in a patch I sent long ago, but it was never applied.
> Anyway, this has been a problem for a while. Yet, I am not inclined
> to revert that commit because the "create a Kconfig entry beforehand,
> and sync with savedefconfig" approach has been successful in practical
> use cases. I will come back to this problem with a different solution
> in a few commits later.
>
> For other use cases, I see no reason to require redundant dictation
> in the input file of this tool. Instead, the tool can know the types
> and default values by parsing the .config file.
>
> This commit changes the tool to use the CONFIG names, but ignore the
> types and default values given by the input file.
>
> This commit also fixes one bug. Before this commit, the tool could
> not move a integer-typed CONFIG with value 1.
>
> For example, assume we are moving CONFIG_CONS_INDEX. Please note
> this is a integer type option.
>
> Many board headers define this CONFIG as 1.
>
> #define CONFIG_CONS_INDEX 1
>
> It will be converted to
>
> CONFIG_CONS_INDEX=y
>
> and moved to include/autoconf.mk, by the tools/scripts/define2mk.sed.
> It will cause "make savedefconfig" to fail due to the type conflict.
>
> This commit takes care of it by detecting the type and converting the
> CONFIG value correctly.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 01/21] tools: moveconfig: fix --dry-run option
2016-05-19 6:51 ` [U-Boot] [PATCH 01/21] tools: moveconfig: fix --dry-run option Masahiro Yamada
@ 2016-05-24 14:27 ` Joe Hershberger
0 siblings, 0 replies; 50+ messages in thread
From: Joe Hershberger @ 2016-05-24 14:27 UTC (permalink / raw)
To: u-boot
On Thu, May 19, 2016 at 1:51 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Since commit 96464badc794 ("moveconfig: Always run savedefconfig on
> the moved config"), --dry-run option is broken.
>
> The --dry-run option prevents the .config from being modified,
> but defconfig files might be updated by "make savedefconfig"
> regardless of the --dry-run option.
>
> Move the "if not self.options.dry_run" conditional to the correct
> place.
>
> Fixes 96464badc794 ("moveconfig: Always run savedefconfig on the moved config")
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 02/21] tools: moveconfig: rename update_defconfig() to update_dotconfig()
2016-05-19 6:51 ` [U-Boot] [PATCH 02/21] tools: moveconfig: rename update_defconfig() to update_dotconfig() Masahiro Yamada
@ 2016-05-24 14:27 ` Joe Hershberger
0 siblings, 0 replies; 50+ messages in thread
From: Joe Hershberger @ 2016-05-24 14:27 UTC (permalink / raw)
To: u-boot
On Thu, May 19, 2016 at 1:51 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Commit 96464badc794 ("moveconfig: Always run savedefconfig on the
> moved config") changed how defconfig files were updated.
>
> Since then, the function update_defconfig() does not modify defconfig
> files at all (instead, they are updated by "make savedefconfig").
>
> The name update_dotconfig() is a better fit for this function.
> Also, update the comment block to match the actual behavior.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 03/21] tools: moveconfig: remove redundant else: after sys.exit()
2016-05-19 6:51 ` [U-Boot] [PATCH 03/21] tools: moveconfig: remove redundant else: after sys.exit() Masahiro Yamada
@ 2016-05-24 14:27 ` Joe Hershberger
0 siblings, 0 replies; 50+ messages in thread
From: Joe Hershberger @ 2016-05-24 14:27 UTC (permalink / raw)
To: u-boot
On Thu, May 19, 2016 at 1:51 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Nesting by "else:" is not generally useful after such statements
> as return, break, and sys.exit(), etc.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 04/21] tools: moveconfig: check directory relocation before compilers
2016-05-19 6:51 ` [U-Boot] [PATCH 04/21] tools: moveconfig: check directory relocation before compilers Masahiro Yamada
@ 2016-05-24 14:27 ` Joe Hershberger
0 siblings, 0 replies; 50+ messages in thread
From: Joe Hershberger @ 2016-05-24 14:27 UTC (permalink / raw)
To: u-boot
On Thu, May 19, 2016 at 1:51 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> We must make sure this tool is run from the top of source directory
> before calling update_cross_compile(). Otherwise, the following
> exception is thrown:
>
> Traceback (most recent call last):
> File "./moveconfig.py", line 918, in <module>
> main()
> File "./moveconfig.py", line 908, in main
> update_cross_compile()
> File "./moveconfig.py", line 292, in update_cross_compile
> for arch in os.listdir('arch'):
> OSError: [Errno 2] No such file or directory: 'arch'
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 05/21] tools: moveconfig: check compilers before starting defconfig walk
2016-05-19 6:51 ` [U-Boot] [PATCH 05/21] tools: moveconfig: check compilers before starting defconfig walk Masahiro Yamada
@ 2016-05-24 14:27 ` Joe Hershberger
0 siblings, 0 replies; 50+ messages in thread
From: Joe Hershberger @ 2016-05-24 14:27 UTC (permalink / raw)
To: u-boot
On Thu, May 19, 2016 at 1:51 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Since commit 25400090b1e2 ("moveconfig: Print a message for
> missing compiler"), this tool parses error messages generated by
> "make include/config/auto.conf" every time an error occurs in order
> to detect missing compiler.
>
> Instead of that, we can look for compilers in the PATH environment
> only once before starting the defconfig walk. If specified compilers
> are missing, "make include/config/auto.conf" will apparently fail
> for corresponding architectures. So, the tool can just report
> "Compiler is missing." and skip those boards.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 06/21] tools: moveconfig: exit with error message for not clean directory
2016-05-19 6:51 ` [U-Boot] [PATCH 06/21] tools: moveconfig: exit with error message for not clean directory Masahiro Yamada
@ 2016-05-24 14:28 ` Joe Hershberger
0 siblings, 0 replies; 50+ messages in thread
From: Joe Hershberger @ 2016-05-24 14:28 UTC (permalink / raw)
To: u-boot
On Thu, May 19, 2016 at 1:51 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> When the source tree is not clean, this tool raises an exception
> with a message like follows:
>
> Traceback (most recent call last):
> File "tools/moveconfig.py", line 939, in <module>
> main()
> File "tools/moveconfig.py", line 934, in main
> move_config(config_attrs, options)
> File "tools/moveconfig.py", line 808, in move_config
> while not slots.available():
> File "tools/moveconfig.py", line 733, in available
> if slot.poll():
> File "tools/moveconfig.py", line 645, in poll
> self.parser.update_dotconfig(self.defconfig)
> File "tools/moveconfig.py", line 503, in update_dotconfig
> with open(autoconf_path) as f:
> IOError: [Errno 2] No such file or directory: '/tmp/tmpDtzCgl/include/autoconf.mk'
>
> This does not explain clearly what is wrong. Show an appropriate
> error message "srctree is not clean, please run 'make mrproper'"
> in such a situation.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 07/21] tools: moveconfig: increment number of processed files monotonically
2016-05-19 6:51 ` [U-Boot] [PATCH 07/21] tools: moveconfig: increment number of processed files monotonically Masahiro Yamada
@ 2016-05-24 14:28 ` Joe Hershberger
0 siblings, 0 replies; 50+ messages in thread
From: Joe Hershberger @ 2016-05-24 14:28 UTC (permalink / raw)
To: u-boot
On Thu, May 19, 2016 at 1:51 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Currently, the progress " * defconfigs out of 1133" does not increase
> monotonically.
>
> Moreover, the number of processed defconfigs does not match the total
> number of defconfigs when this tool finishes, like:
>
> 1132 defconfigs out of 1133
> Clean up headers? [y/n]:
>
> It looks like the task was not completed, and some users might feel
> upset about it.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 11/21] tools: moveconfig: add --undef option to move CONFIGs with default y
2016-05-19 6:51 ` [U-Boot] [PATCH 11/21] tools: moveconfig: add --undef option to move CONFIGs with default y Masahiro Yamada
@ 2016-05-24 14:38 ` Joe Hershberger
2016-05-24 23:16 ` Masahiro Yamada
0 siblings, 1 reply; 50+ messages in thread
From: Joe Hershberger @ 2016-05-24 14:38 UTC (permalink / raw)
To: u-boot
On Thu, May 19, 2016 at 1:51 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Since 96464badc794 ("moveconfig: Always run savedefconfig on the
> moved config"), this tool can not correctly move bool configs with
> the default value y.
>
> The reason is like follows:
>
> We are supposed to add the config entries in Kconfig for the options
> we are moving before running this tool. Otherwise, the moved options
> would all disappear during the "make savedefconfig" stage.
>
> Let's say we want to move CONFIG_FOO to Kconfig, making it an option
> with "default y". The first thing we need to do is to create an entry
> like follows:
>
> config FOO
> bool "foo"
> default y
>
> So, CONFIG_FOO=y will be set in the .config for every defconfig.
>
> Commit 7740f653e6b3 ("moveconfig: Ignore duplicate configs when
> moving") introduced KCONFIG_IGNORE_DUPLICATES to fix the false
> negative (= boards that should be converted to "=y" are misconverted
> to "not set") problem. With that commit, the CONFIGs in the .config
> are now duplicated to include/autoconf.mk unless they are defined to
> a different value or #undef'ed in the board header. It causes false
> positive problem this time; boards without #define/#undef CONFIG_FOO
> in their header should be converted to "not set", but misconverted
> to "=y" actually.
>
> This commit intends to handle such cases correctly. Anyway, we need
> to create a Kconfig entry beforehand to persevere savedefconfig.
> But, we do not want to propagate CONFIGs coming from "default y"
> to the include/autoconf.mk. To achieve it, the new option --undef
> is here!
>
> We can move bool option with default y in this way:
> 1. create an entry in Kconfig
> 2. run "tools/moveconfig.py --undef=CONFIG_FOO CONFIG_FOO"
This seems tedious to use. The way I solved this didn't require any
extra specification of what was expected to happen.
http://lists.denx.de/pipermail/u-boot/2015-June/215724.html
> This option can appear multiple times in the command line, so that
> we can move multiple options at a time.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 09/21] tools: moveconfig: drop code for handling type and default value
2016-05-19 6:51 ` [U-Boot] [PATCH 09/21] tools: moveconfig: drop code for handling type and default value Masahiro Yamada
@ 2016-05-24 14:38 ` Joe Hershberger
0 siblings, 0 replies; 50+ messages in thread
From: Joe Hershberger @ 2016-05-24 14:38 UTC (permalink / raw)
To: u-boot
On Thu, May 19, 2016 at 1:51 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Now types and defalut values given by the input file are just
> ignored. Delete unnecessary code.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 10/21] tools: moveconfig: allow to give CONFIG names as argument directly
2016-05-19 6:51 ` [U-Boot] [PATCH 10/21] tools: moveconfig: allow to give CONFIG names as argument directly Masahiro Yamada
@ 2016-05-24 14:38 ` Joe Hershberger
0 siblings, 0 replies; 50+ messages in thread
From: Joe Hershberger @ 2016-05-24 14:38 UTC (permalink / raw)
To: u-boot
On Thu, May 19, 2016 at 1:51 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> We still pass the input file with CONFIG name, type, default value
> in each line, but the last two fields are just ignored by the tool.
> So, let's deprecate the input file and allow users to give CONFIG
> names directly from the command line. The types and default values
> are automatically detected and handled nicely by the tool.
>
> Going forward, we can use this tool more easily like:
>
> tools/moveconfig.py CONFIG_FOO CONFIG_BAR
>
> Update the documentation and fix some typos I noticed while I was
> working on.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 12/21] tools: moveconfig: compute file paths just once
2016-05-19 6:52 ` [U-Boot] [PATCH 12/21] tools: moveconfig: compute file paths just once Masahiro Yamada
@ 2016-05-24 14:54 ` Joe Hershberger
0 siblings, 0 replies; 50+ messages in thread
From: Joe Hershberger @ 2016-05-24 14:54 UTC (permalink / raw)
To: u-boot
On Thu, May 19, 2016 at 1:52 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> The paths to .config, include/autoconf.mk, include/config/auto.conf
> are not changed during the defconfig walk. Compute them only once
> when a new class instance is created.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 13/21] tools: moveconfig: move log output code out of Kconfig Parser class
2016-05-19 6:52 ` [U-Boot] [PATCH 13/21] tools: moveconfig: move log output code out of Kconfig Parser class Masahiro Yamada
@ 2016-05-24 14:54 ` Joe Hershberger
0 siblings, 0 replies; 50+ messages in thread
From: Joe Hershberger @ 2016-05-24 14:54 UTC (permalink / raw)
To: u-boot
On Thu, May 19, 2016 at 1:52 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> This will help further improvement/clean-up.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 14/21] tools: moveconfig: display log atomically in more readable format
2016-05-19 6:52 ` [U-Boot] [PATCH 14/21] tools: moveconfig: display log atomically in more readable format Masahiro Yamada
@ 2016-05-24 14:54 ` Joe Hershberger
0 siblings, 0 replies; 50+ messages in thread
From: Joe Hershberger @ 2016-05-24 14:54 UTC (permalink / raw)
To: u-boot
On Thu, May 19, 2016 at 1:52 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Before this commit, the log was displayed in the format:
>
> <defconfig_name> : <action1>
> <defconfig_name> : <action2>
> <defconfig_name> : <action3>
>
> When we move multiple CONFIGs at the same time, we see as many
> <defconfig_name> strings as actions for every defconfig, which is
> redundant information.
>
> Moreover, since normal log and error log are displayed separately,
> Messages from different threads could be mixed, like this:
>
> <foo> : <action1>
> <foo> : <action2>
> <bar> : <action1>
> <bar> : <action2>
> <foo> : <error_log>
>
> This commit makes sure to call "print" once a defconfig, which
> enables atomic logging for each defconfig. It also makes it
> possible to refactor the log format as follows:
>
> <foo_defconfig>
> <action1>
> <action2>
> <error_log>
>
> <bar_defconfig>
> <action1>
> <action2>
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Much better!
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 15/21] tools: moveconfig: refactor code to go back to idle state
2016-05-19 6:52 ` [U-Boot] [PATCH 15/21] tools: moveconfig: refactor code to go back to idle state Masahiro Yamada
@ 2016-05-24 14:54 ` Joe Hershberger
0 siblings, 0 replies; 50+ messages in thread
From: Joe Hershberger @ 2016-05-24 14:54 UTC (permalink / raw)
To: u-boot
On Thu, May 19, 2016 at 1:52 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Move similar code to finish() function.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 16/21] tools: moveconfig: skip savedefconfig if .config was not updated
2016-05-19 6:52 ` [U-Boot] [PATCH 16/21] tools: moveconfig: skip savedefconfig if .config was not updated Masahiro Yamada
@ 2016-05-24 14:55 ` Joe Hershberger
0 siblings, 0 replies; 50+ messages in thread
From: Joe Hershberger @ 2016-05-24 14:55 UTC (permalink / raw)
To: u-boot
On Thu, May 19, 2016 at 1:52 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> If no CONFIG option is moved to the .config, no need to sync the
> defconfig file. This accelerates the processing by skipping
> unneeded "make savedefconfig".
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 17/21] tools: moveconfig: display log when savedefconfig occurs
2016-05-19 6:52 ` [U-Boot] [PATCH 17/21] tools: moveconfig: display log when savedefconfig occurs Masahiro Yamada
@ 2016-05-24 14:55 ` Joe Hershberger
0 siblings, 0 replies; 50+ messages in thread
From: Joe Hershberger @ 2016-05-24 14:55 UTC (permalink / raw)
To: u-boot
On Thu, May 19, 2016 at 1:52 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Now, "make savedefconfig" does not always happen. Display the log
> when it happens.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 18/21] tools: moveconfig: report when CONFIGs are removed by savedefconfig
2016-05-19 6:52 ` [U-Boot] [PATCH 18/21] tools: moveconfig: report when CONFIGs are removed by savedefconfig Masahiro Yamada
@ 2016-05-24 14:55 ` Joe Hershberger
0 siblings, 0 replies; 50+ messages in thread
From: Joe Hershberger @ 2016-05-24 14:55 UTC (permalink / raw)
To: u-boot
On Thu, May 19, 2016 at 1:52 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> This is a rare case, but there is still possibility that some CONFIG
> is moved to the .config, but it is removed by "make savedefconfig".
> (For example, it happens when the specified CONFIG has no prompt in
> the Kconfig entry, i.e. it is not user-configurable.)
>
> It might be an unexpected case. So, display the log in this case
> (in yellow color to gain user's attention if --color option is given).
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 19/21] tools: moveconfig: report when defconfig is updated
2016-05-19 6:52 ` [U-Boot] [PATCH 19/21] tools: moveconfig: report when defconfig is updated Masahiro Yamada
@ 2016-05-24 14:55 ` Joe Hershberger
0 siblings, 0 replies; 50+ messages in thread
From: Joe Hershberger @ 2016-05-24 14:55 UTC (permalink / raw)
To: u-boot
On Thu, May 19, 2016 at 1:52 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> There are various factors that determine if the given defconfig is
> updated, and it is probably what users are more interested in.
>
> Show the log when the defconfig is updated. Also, copy the file
> only when the file content was really updated to avoid changing
> the time stamp needlessly.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 20/21] tools: moveconfig: add --force-sync option
2016-05-19 6:52 ` [U-Boot] [PATCH 20/21] tools: moveconfig: add --force-sync option Masahiro Yamada
@ 2016-05-24 14:55 ` Joe Hershberger
0 siblings, 0 replies; 50+ messages in thread
From: Joe Hershberger @ 2016-05-24 14:55 UTC (permalink / raw)
To: u-boot
On Thu, May 19, 2016 at 1:52 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Now, this tools invokes "make savedefconfig" only when it needs to
> do so, but there might be cases where a user wants the tool to do
> savedefconfig forcibly, for example, some defconfigs were already
> out of sync and the user wants to fix it as well.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 21/21] tools: moveconfig: allow to run without any CONFIG specified
2016-05-19 6:52 ` [U-Boot] [PATCH 21/21] tools: moveconfig: allow to run without any CONFIG specified Masahiro Yamada
@ 2016-05-24 14:55 ` Joe Hershberger
0 siblings, 0 replies; 50+ messages in thread
From: Joe Hershberger @ 2016-05-24 14:55 UTC (permalink / raw)
To: u-boot
On Thu, May 19, 2016 at 1:52 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> I found "tools/moveconfig -s" might be useful for defconfig re-sync.
> I could optimize it for re-sync if I wanted, but I do not want to
> make the code complex for this feature.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 11/21] tools: moveconfig: add --undef option to move CONFIGs with default y
2016-05-24 14:38 ` Joe Hershberger
@ 2016-05-24 23:16 ` Masahiro Yamada
2016-05-25 15:00 ` Joe Hershberger
0 siblings, 1 reply; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-24 23:16 UTC (permalink / raw)
To: u-boot
Hi Joe,
2016-05-24 23:38 GMT+09:00 Joe Hershberger <joe.hershberger@gmail.com>:
>> We can move bool option with default y in this way:
>> 1. create an entry in Kconfig
>> 2. run "tools/moveconfig.py --undef=CONFIG_FOO CONFIG_FOO"
>
> This seems tedious to use. The way I solved this didn't require any
> extra specification of what was expected to happen.
>
> http://lists.denx.de/pipermail/u-boot/2015-June/215724.html
>
I recalled it now.
If I apply this series without 11/21,
do you like to send v2 of yours on top of this series?
I still do not like to borrow a function from patman, though.
--
Best Regards
Masahiro Yamada
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 11/21] tools: moveconfig: add --undef option to move CONFIGs with default y
2016-05-24 23:16 ` Masahiro Yamada
@ 2016-05-25 15:00 ` Joe Hershberger
2016-05-25 15:32 ` Masahiro Yamada
0 siblings, 1 reply; 50+ messages in thread
From: Joe Hershberger @ 2016-05-25 15:00 UTC (permalink / raw)
To: u-boot
On Tue, May 24, 2016 at 6:16 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Hi Joe,
>
> 2016-05-24 23:38 GMT+09:00 Joe Hershberger <joe.hershberger@gmail.com>:
>
>>> We can move bool option with default y in this way:
>>> 1. create an entry in Kconfig
>>> 2. run "tools/moveconfig.py --undef=CONFIG_FOO CONFIG_FOO"
>>
>> This seems tedious to use. The way I solved this didn't require any
>> extra specification of what was expected to happen.
>>
>> http://lists.denx.de/pipermail/u-boot/2015-June/215724.html
>>
>
> I recalled it now.
>
> If I apply this series without 11/21,
> do you like to send v2 of yours on top of this series?
Sure.
> I still do not like to borrow a function from patman, though.
OK, I'll rework it.
-Joe
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 11/21] tools: moveconfig: add --undef option to move CONFIGs with default y
2016-05-25 15:00 ` Joe Hershberger
@ 2016-05-25 15:32 ` Masahiro Yamada
0 siblings, 0 replies; 50+ messages in thread
From: Masahiro Yamada @ 2016-05-25 15:32 UTC (permalink / raw)
To: u-boot
2016-05-26 0:00 GMT+09:00 Joe Hershberger <joe.hershberger@gmail.com>:
> On Tue, May 24, 2016 at 6:16 PM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> Hi Joe,
>>
>> 2016-05-24 23:38 GMT+09:00 Joe Hershberger <joe.hershberger@gmail.com>:
>>
>>>> We can move bool option with default y in this way:
>>>> 1. create an entry in Kconfig
>>>> 2. run "tools/moveconfig.py --undef=CONFIG_FOO CONFIG_FOO"
>>>
>>> This seems tedious to use. The way I solved this didn't require any
>>> extra specification of what was expected to happen.
>>>
>>> http://lists.denx.de/pipermail/u-boot/2015-June/215724.html
>>>
>>
>> I recalled it now.
>>
>> If I apply this series without 11/21,
>> do you like to send v2 of yours on top of this series?
>
> Sure.
>
>> I still do not like to borrow a function from patman, though.
>
> OK, I'll rework it.
>
This series should be applied onto u-boot/master
cleanly without 11/21.
--
Best Regards
Masahiro Yamada
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 08/21] tools: moveconfig: do not rely on type and default value given by users
2016-05-24 14:27 ` Joe Hershberger
@ 2016-06-12 22:20 ` Masahiro Yamada
0 siblings, 0 replies; 50+ messages in thread
From: Masahiro Yamada @ 2016-06-12 22:20 UTC (permalink / raw)
To: u-boot
2016-05-24 23:27 GMT+09:00 Joe Hershberger <joe.hershberger@gmail.com>:
> On Thu, May 19, 2016 at 1:51 AM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> Commit 96464badc794 ("moveconfig: Always run savedefconfig on the
>> moved config") changed the work flow of this tool a lot from the
>> original intention when this tool was designed first.
>>
>> Since then, before running this tool, users must edit the Kconfig to
>> add the menu entries for the configs they are moving. It means users
>> had already specified the type and the default value for each CONFIG
>> via the Kconfig entries. Nevertheless, users are still required to
>> dictate the same type and the default value in each line of the input
>> file. So, my idea here is to deprecate the latter.
>>
>> Before moving forward with it, there is one thing worth mentioning;
>> since the savedefconfig re-sync was introduced, there is a case this
>> tool can not handle correctly.
>>
>> Let's say we are moving CONFIG_FOO from board headers to Kconfig,
>> and we want to make it "default y" option. First, we are supposed
>> to create an entry:
>>
>> config FOO
>> bool "foo"
>> default y
>>
>> CONFIG_FOO=y will appear in the .config for every processed defconfig.
>> It will be duplicated to the include/autoconf.mk as well unless the
>> board explicitly #undef's it. Therefore, the defconfig without
>> "#define CONFIG_FOO" or "#undef CONFIG_FOO" that should be converted
>> to "unset" would be misconverted to "=y".
>
> I solved this in a patch I sent long ago, but it was never applied.
Apology.
I rephrased this git-log.
I mentioned that you were already trying to fix this problem a year ago.
--
Best Regards
Masahiro Yamada
^ permalink raw reply [flat|nested] 50+ messages in thread
* [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups
2016-05-24 1:27 ` Masahiro Yamada
@ 2016-06-12 22:30 ` Masahiro Yamada
0 siblings, 0 replies; 50+ messages in thread
From: Masahiro Yamada @ 2016-06-12 22:30 UTC (permalink / raw)
To: u-boot
2016-05-24 10:27 GMT+09:00 Masahiro Yamada <yamada.masahiro@socionext.com>:
> 2016-05-24 7:13 GMT+09:00 Tom Rini <trini@konsulko.com>:
>> On Thu, May 19, 2016 at 03:51:48PM +0900, Masahiro Yamada wrote:
>>
>>> Masahiro Yamada (21):
>>> tools: moveconfig: fix --dry-run option
>>> tools: moveconfig: rename update_defconfig() to update_dotconfig()
>>> tools: moveconfig: remove redundant else: after sys.exit()
>>> tools: moveconfig: check directory relocation before compilers
>>> tools: moveconfig: check compilers before starting defconfig walk
>>> tools: moveconfig: exit with error message for not clean directory
>>> tools: moveconfig: increment number of processed files monotonically
>>> tools: moveconfig: do not rely on type and default value given by
>>> users
>>> tools: moveconfig: drop code for handling type and default value
>>> tools: moveconfig: allow to give CONFIG names as argument directly
>>> tools: moveconfig: add --undef option to move CONFIGs with default y
>>> tools: moveconfig: compute file paths just once
>>> tools: moveconfig: move log output code out of Kconfig Parser class
>>> tools: moveconfig: display log atomically in more readable format
>>> tools: moveconfig: refactor code to go back to idle state
>>> tools: moveconfig: skip savedefconfig if .config was not updated
>>> tools: moveconfig: display log when savedefconfig occurs
>>> tools: moveconfig: report when CONFIGs are removed by savedefconfig
>>> tools: moveconfig: report when defconfig is updated
>>> tools: moveconfig: add --force-sync option
>>> tools: moveconfig: allow to run without any CONFIG specified
Series applied except 11/21 "tools: moveconfig: add --undef option to
move CONFIGs with default y"
It was dropped because Joe's idea is cooler.
Some minor updates:
6/21: rephrase "srctree" -> "source tree"
8/21: rephrase git-log to mention that Joe sent a patch
to fix the problem a year ago.
21/21: fix a typo pointed out by Joe.
call cleanup_headers() only when at least one config is given
--
Best Regards
Masahiro Yamada
^ permalink raw reply [flat|nested] 50+ messages in thread
end of thread, other threads:[~2016-06-12 22:30 UTC | newest]
Thread overview: 50+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-05-19 6:51 [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Masahiro Yamada
2016-05-19 6:51 ` [U-Boot] [PATCH 01/21] tools: moveconfig: fix --dry-run option Masahiro Yamada
2016-05-24 14:27 ` Joe Hershberger
2016-05-19 6:51 ` [U-Boot] [PATCH 02/21] tools: moveconfig: rename update_defconfig() to update_dotconfig() Masahiro Yamada
2016-05-24 14:27 ` Joe Hershberger
2016-05-19 6:51 ` [U-Boot] [PATCH 03/21] tools: moveconfig: remove redundant else: after sys.exit() Masahiro Yamada
2016-05-24 14:27 ` Joe Hershberger
2016-05-19 6:51 ` [U-Boot] [PATCH 04/21] tools: moveconfig: check directory relocation before compilers Masahiro Yamada
2016-05-24 14:27 ` Joe Hershberger
2016-05-19 6:51 ` [U-Boot] [PATCH 05/21] tools: moveconfig: check compilers before starting defconfig walk Masahiro Yamada
2016-05-24 14:27 ` Joe Hershberger
2016-05-19 6:51 ` [U-Boot] [PATCH 06/21] tools: moveconfig: exit with error message for not clean directory Masahiro Yamada
2016-05-24 14:28 ` Joe Hershberger
2016-05-19 6:51 ` [U-Boot] [PATCH 07/21] tools: moveconfig: increment number of processed files monotonically Masahiro Yamada
2016-05-24 14:28 ` Joe Hershberger
2016-05-19 6:51 ` [U-Boot] [PATCH 08/21] tools: moveconfig: do not rely on type and default value given by users Masahiro Yamada
2016-05-24 14:27 ` Joe Hershberger
2016-06-12 22:20 ` Masahiro Yamada
2016-05-19 6:51 ` [U-Boot] [PATCH 09/21] tools: moveconfig: drop code for handling type and default value Masahiro Yamada
2016-05-24 14:38 ` Joe Hershberger
2016-05-19 6:51 ` [U-Boot] [PATCH 10/21] tools: moveconfig: allow to give CONFIG names as argument directly Masahiro Yamada
2016-05-24 14:38 ` Joe Hershberger
2016-05-19 6:51 ` [U-Boot] [PATCH 11/21] tools: moveconfig: add --undef option to move CONFIGs with default y Masahiro Yamada
2016-05-24 14:38 ` Joe Hershberger
2016-05-24 23:16 ` Masahiro Yamada
2016-05-25 15:00 ` Joe Hershberger
2016-05-25 15:32 ` Masahiro Yamada
2016-05-19 6:52 ` [U-Boot] [PATCH 12/21] tools: moveconfig: compute file paths just once Masahiro Yamada
2016-05-24 14:54 ` Joe Hershberger
2016-05-19 6:52 ` [U-Boot] [PATCH 13/21] tools: moveconfig: move log output code out of Kconfig Parser class Masahiro Yamada
2016-05-24 14:54 ` Joe Hershberger
2016-05-19 6:52 ` [U-Boot] [PATCH 14/21] tools: moveconfig: display log atomically in more readable format Masahiro Yamada
2016-05-24 14:54 ` Joe Hershberger
2016-05-19 6:52 ` [U-Boot] [PATCH 15/21] tools: moveconfig: refactor code to go back to idle state Masahiro Yamada
2016-05-24 14:54 ` Joe Hershberger
2016-05-19 6:52 ` [U-Boot] [PATCH 16/21] tools: moveconfig: skip savedefconfig if .config was not updated Masahiro Yamada
2016-05-24 14:55 ` Joe Hershberger
2016-05-19 6:52 ` [U-Boot] [PATCH 17/21] tools: moveconfig: display log when savedefconfig occurs Masahiro Yamada
2016-05-24 14:55 ` Joe Hershberger
2016-05-19 6:52 ` [U-Boot] [PATCH 18/21] tools: moveconfig: report when CONFIGs are removed by savedefconfig Masahiro Yamada
2016-05-24 14:55 ` Joe Hershberger
2016-05-19 6:52 ` [U-Boot] [PATCH 19/21] tools: moveconfig: report when defconfig is updated Masahiro Yamada
2016-05-24 14:55 ` Joe Hershberger
2016-05-19 6:52 ` [U-Boot] [PATCH 20/21] tools: moveconfig: add --force-sync option Masahiro Yamada
2016-05-24 14:55 ` Joe Hershberger
2016-05-19 6:52 ` [U-Boot] [PATCH 21/21] tools: moveconfig: allow to run without any CONFIG specified Masahiro Yamada
2016-05-24 14:55 ` Joe Hershberger
2016-05-23 22:13 ` [U-Boot] [PATCH 00/21] tools: moveconfig: many fixes, improvements, code clean-ups Tom Rini
2016-05-24 1:27 ` Masahiro Yamada
2016-06-12 22:30 ` Masahiro Yamada
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.