[U-Boot] [PATCH] Don't apply: tools: add a tool to move automatically CONFIGs from headers to defconfigs

[U-Boot] [PATCH] Don't apply: tools: add a tool to move automatically CONFIGs from headers to defconfigs

[Masahiro Yamada]

This tool can move CONFIG macros from C headers (include/configs/*.h)
to defconfigs (configs/*_defconfig) all over the boards.

There are tons of CONFIGs in U-boot and moving them by hand is
absolutely tool painful.  I wrote this script for my local use,
so this patch might not clean enough to be applied to the code base.
It might contain some bugs.

Please do not apply this patch!

This patch is here because Simon requested me to share it.
See the discussion in here:
http://lists.denx.de/pipermail/u-boot/2015-January/201556.html

Usage:

[1] Please run "git status" and make sure your git repository is clean.

[2] Describe the CONFIG option you want move in the file "~/.moveconfig"

  Say, you want to move CONFIG_CMD_NAND, for example.
  In this case, the content of "~/.moveconfig" should be like this:

  $ cat .moveconfig
  CONFIG_CMD_NAND bool n y

  - The first field is the name of the CONFIG.
  - The second field is the type of the CONFIG such as "bool", "int", "hex", etc.
  - The third value is the default value.
    The value that is same as the default is omitted in each defconfig.
    If the type of the CONFIG is bool, the default value must be either 'y' or 'n'.
  - The forth field shows whether the CONFIG depends on CONFIG_SPL_BUILD.
    For example, CONFIG_CMD_* makes sense only on the main-image.
    (depends on !SPL_BUILD)  In this case, specify 'y' in the forth fields.
    If the CONFIG should appear also on SPL, specify 'n' here.

[4] Run "tools/moveconfig.py"

  The log will be displayed like this

  $ tools/moveconfig.py
  Moving CONFIG_CMD_NAND (type: bool, default: n, no_spl: y) ...  (jobs: 8)
  ms7750se                             : (default)
  davinci_sonata                       : (default)
  tk71                                 : y
  db-mv784mp-gp                        : (default)
  cm_t3517                             : y
  P2041RDB_SDCARD                      : y
        ...

  The left-hand side field is the board[defconfig] name and the colon
  is followed by the value of the CONFIG.

  At the last stage, you will be asked like this:

  Clean up CONFIG_CMD_NAND in headers? [y/n]:

  If you say 'y' here, the defines in C headers will be removed.

Enjoy!

Signed-off-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Cc: Simon Glass <sjg@chromium.org>
Cc: Alexey Brodkin <abrodkin@synopsys.com>
---

 tools/moveconfig.py | 439 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 439 insertions(+)
 create mode 100755 tools/moveconfig.py

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
new file mode 100755
index 0000000..cb7f7ea
--- /dev/null
+++ b/tools/moveconfig.py
@@ -0,0 +1,439 @@
+#!/usr/bin/env python
+#
+# Author: Masahiro Yamada <yamada.m@jp.panasonic.com>
+#
+# SPDX-License-Identifier:	GPL-2.0+
+#
+
+"""
+Move the specified config option to Kconfig
+"""
+
+import errno
+import fnmatch
+import multiprocessing
+import optparse
+import os
+import re
+import shutil
+import subprocess
+import sys
+import tempfile
+import time
+
+SHOW_GNU_MAKE = 'scripts/show-gnu-make'
+SLEEP_TIME=0.03
+
+CROSS_COMPILE = {
+    'aarch64': 'aarch64-linux-',
+    'arm': 'arm-unknown-linux-gnueabi-',
+    'avr32': 'avr32-linux-',
+    'blackfin': 'bfin-elf-',
+    'm68k': 'm68k-linux-',
+    'microblaze': 'microblaze-linux-',
+    'mips': 'mips-linux-',
+    'nds32': 'nds32le-linux-',
+    'nios2': 'nios2-linux-',
+    'openrisc': 'or32-linux-',
+    'powerpc': 'powerpc-linux-',
+    'sh': 'sh4-gentoo-linux-gnu-',
+    'sparc': 'sparc-linux-',
+    'x86': 'i386-linux-'
+}
+
+STATE_IDLE = 0
+STATE_DEFCONFIG = 1
+STATE_SILENTOLDCONFIG = 2
+
+### helper functions ###
+def get_devnull():
+    """Get the file object of '/dev/null' device."""
+    try:
+        devnull = subprocess.DEVNULL # py3k
+    except AttributeError:
+        devnull = open(os.devnull, 'wb')
+    return devnull
+
+def check_top_directory():
+    """Exit if we are not at the top of source directory."""
+    for f in ('README', 'Licenses'):
+        if not os.path.exists(f):
+            sys.exit('Please run at the top of source directory.')
+
+def get_make_cmd():
+    """Get the command name of GNU Make."""
+    process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE)
+    ret = process.communicate()
+    if process.returncode:
+        sys.exit('GNU Make not found')
+    return ret[0].rstrip()
+
+def cleanup_header(header_path, patterns):
+    with open(header_path) as f:
+        lines = f.readlines()
+
+    matched = []
+    for i, line in enumerate(lines):
+        for pattern in patterns:
+            m = pattern.search(line)
+            if m:
+                print '%s: %s: %s' % (header_path, i + 1, line),
+                matched.append(i)
+                break
+
+    if not matched:
+        return
+
+    with open(header_path, 'w') as f:
+        for i, line in enumerate(lines):
+            if not i in matched:
+                f.write(line)
+
+def cleanup_headers(config):
+    while True:
+        choice = raw_input('Clean up %s in headers? [y/n]: ' % config).lower()
+        print choice
+        if choice == 'y' or choice == 'n':
+            break
+
+    if choice != 'y':
+        return
+
+    patterns = []
+    patterns.append(re.compile(r'#\s*define\s+%s\W' % config))
+    patterns.append(re.compile(r'#\s*undef\s+%s\W' % config))
+
+    for (dirpath, dirnames, filenames) in os.walk('include'):
+        for filename in filenames:
+            if not fnmatch.fnmatch(filename, '*~'):
+                cleanup_header(os.path.join(dirpath, filename), patterns)
+
+### classes ###
+class KconfigParser:
+
+    """A parser of .config file.
+
+    Each line of the output should have the form of:
+    Status, Arch, CPU, SoC, Vendor, Board, Target, Options, Maintainers
+    Most of them are extracted from .config file.
+    MAINTAINERS files are also consulted for Status and Maintainers fields.
+    """
+
+    re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"')
+    re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"')
+    PREFIX = { '.': '+', 'spl': 'S', 'tpl': 'T' }
+
+    def __init__(self, build_dir, config_attr):
+        """Create a new .config perser.
+
+        Arguments:
+          build_dir: Build directory where .config is located
+          output: File object which the result is written to
+        """
+        self.build_dir = build_dir
+        self.config = config_attr['config']
+        self.type = config_attr['type']
+        self.default = config_attr['default']
+        if config_attr['no_spl_support'] == 'y':
+            self.no_spl_support = True
+        else:
+            self.no_spl_support = False
+        self.re = re.compile(r'%s=(.*)' % self.config)
+
+    def get_cross_compile(self):
+        """Parse .config file and return CROSS_COMPILE
+
+        Arguments:
+          defconfig: Board (defconfig) name
+        """
+        arch = ''
+        cpu = ''
+        dotconfig = os.path.join(self.build_dir, '.config')
+        for line in open(dotconfig):
+            m = self.re_arch.match(line)
+            if m:
+                arch = m.group(1)
+                continue
+            m = self.re_cpu.match(line)
+            if m:
+                cpu = m.group(1)
+
+        assert arch, 'Error: arch is not defined in %s' % defconfig
+
+        # fix-up for aarch64
+        if arch == 'arm' and cpu == 'armv8':
+            arch = 'aarch64'
+
+        return CROSS_COMPILE.get(arch, '')
+
+    def update_defconfig(self, defconfig):
+        values = []
+        output_lines = []
+        prefixes = {}
+
+        if self.no_spl_support:
+            images = ('.')
+        else:
+            images = ('.', 'spl', 'tpl')
+
+
+        for img in images:
+            autoconf = os.path.join(self.build_dir, img, 'include',
+                                    'autoconf.mk')
+
+            if not os.path.exists(autoconf):
+                values.append('')
+                continue
+
+            if self.type == 'bool':
+                value = 'n'
+            else:
+                value = '(undef)'
+
+            for line in open(autoconf):
+                m = self.re.match(line)
+                if m:
+                    value = m.group(1)
+                    break
+
+            if value == self.default:
+                value = '(default)'
+
+            values.append(value)
+            os.remove(autoconf)
+
+            if value == '(undef)' or value == '(default)':
+                continue
+
+            if self.type == 'bool' and value == 'n':
+                output_line = '# %s is not set' % (self.config)
+            else:
+                output_line = '%s=%s' % (self.config, value)
+            if output_line in output_lines:
+                prefixes[output_line] += self.PREFIX[img]
+            else:
+                output_lines.append(output_line)
+                prefixes[output_line] = self.PREFIX[img]
+
+        output = defconfig[:-len('_defconfig')].ljust(37) + ': '
+        for value in values:
+            output += value.ljust(12)
+
+        print output.strip()
+
+        with open(os.path.join('configs', defconfig), 'a') as f:
+            for line in output_lines:
+                if prefixes[line] != '+':
+                    line = prefixes[line] + ':' + line
+                f.write(line + '\n')
+
+class Slot:
+
+    """A slot to store a subprocess.
+
+    Each instance of this class handles one subprocess.
+    This class is useful to control multiple processes
+    for faster processing.
+    """
+
+    def __init__(self, config_attr, devnull, make_cmd):
+        """Create a new slot.
+
+        Arguments:
+          output: File object which the result is written to
+        """
+        self.build_dir = tempfile.mkdtemp()
+        self.devnull = devnull
+        self.make_cmd = (make_cmd, 'O=' + self.build_dir)
+        self.parser = KconfigParser(self.build_dir, config_attr)
+        self.state = STATE_IDLE
+
+    def __del__(self):
+        """Delete the working directory"""
+        if self.state != STATE_IDLE:
+            while self.ps.poll() == None:
+                pass
+        shutil.rmtree(self.build_dir)
+
+    def add(self, defconfig):
+        """Add a new subprocess to the slot.
+
+        Fails if the slot is occupied, that is, the current subprocess
+        is still running.
+
+        Arguments:
+          defconfig: Board (defconfig) name
+
+        Returns:
+          Return True on success or False on fail
+        """
+        if self.state != STATE_IDLE:
+            return False
+        cmd = list(self.make_cmd)
+        cmd.append(defconfig)
+        self.ps = subprocess.Popen(cmd, stdout=self.devnull)
+        self.defconfig = defconfig
+        self.state = STATE_DEFCONFIG
+        return True
+
+    def poll(self):
+        """Check if the subprocess is running and invoke the .config
+        parser if the subprocess is terminated.
+
+        Returns:
+          Return True if the subprocess is terminated, False otherwise
+        """
+        if self.state == STATE_IDLE:
+            return True
+
+        if self.ps.poll() == None:
+            return False
+
+        if self.ps.poll() != 0:
+            sys.exit("failed to process '%s'" % self.defconfig)
+
+        if self.state == STATE_SILENTOLDCONFIG:
+            self.parser.update_defconfig(self.defconfig)
+            self.state = STATE_IDLE
+            return True
+
+        cross_compile = self.parser.get_cross_compile()
+        cmd = list(self.make_cmd)
+        if cross_compile:
+            cmd.append('CROSS_COMPILE=%s' % cross_compile)
+        cmd.append('silentoldconfig')
+        self.ps = subprocess.Popen(cmd, stdout=self.devnull)
+        self.state = STATE_SILENTOLDCONFIG
+        return False
+
+class Slots:
+
+    """Controller of the array of subprocess slots."""
+
+    def __init__(self, config_attr, jobs):
+        """Create a new slots controller.
+
+        Arguments:
+          jobs: A number of slots to instantiate
+        """
+        self.slots = []
+        devnull = get_devnull()
+        make_cmd = get_make_cmd()
+        for i in range(jobs):
+            self.slots.append(Slot(config_attr, devnull, make_cmd))
+
+    def add(self, defconfig):
+        """Add a new subprocess if a vacant slot is available.
+
+        Arguments:
+          defconfig: Board (defconfig) name
+
+        Returns:
+          Return True on success or False on fail
+        """
+        for slot in self.slots:
+            if slot.add(defconfig):
+                return True
+        return False
+
+    def available(self):
+        """Check if there is a vacant slot.
+
+        Returns:
+          Return True if a vacant slot is found, False if all slots are full
+        """
+        for slot in self.slots:
+            if slot.poll():
+                return True
+        return False
+
+    def empty(self):
+        """Check if all slots are vacant.
+
+        Returns:
+          Return True if all slots are vacant, False if at least one slot
+          is running
+        """
+        ret = True
+        for slot in self.slots:
+            if not slot.poll():
+                ret = False
+        return ret
+
+def move_config(config_attr, jobs=1):
+    check_top_directory()
+    print 'Moving %s (type: %s, default: %s, no_spl: %s) ...  (jobs: %d)' % (
+        config_attr['config'],
+        config_attr['type'],
+        config_attr['default'],
+        config_attr['no_spl_support'],
+        jobs)
+
+    # All the defconfig files to be processed
+    defconfigs = []
+    for (dirpath, dirnames, filenames) in os.walk('configs'):
+        dirpath = dirpath[len('configs') + 1:]
+        for filename in fnmatch.filter(filenames, '*_defconfig'):
+            if fnmatch.fnmatch(filename, '.*'):
+                continue
+            defconfigs.append(os.path.join(dirpath, filename))
+
+    slots = Slots(config_attr, jobs)
+
+    # Main loop to process defconfig files:
+    #  Add a new subprocess into a vacant slot.
+    #  Sleep if there is no available slot.
+    for defconfig in defconfigs:
+        while not slots.add(defconfig):
+            while not slots.available():
+                # No available slot: sleep for a while
+                time.sleep(SLEEP_TIME)
+
+    # wait until all the subprocesses finish
+    while not slots.empty():
+        time.sleep(SLEEP_TIME)
+
+    cleanup_headers(config_attr['config'])
+
+def main():
+    try:
+        cpu_count = multiprocessing.cpu_count()
+    except NotImplementedError:
+        cpu_count = 1
+
+    parser = optparse.OptionParser()
+    # Add options here
+    parser.add_option('-j', '--jobs', type='int', default=cpu_count,
+                      help='the number of jobs to run simultaneously')
+    parser.usage += ' config type default no_spl_support'
+    (options, args) = parser.parse_args()
+
+    args_key = ('config', 'type', 'default', 'no_spl_support')
+    config_attr = {}
+
+    if len(args) >= len(args_key):
+        saved_attr = ''
+        for i, key in enumerate(args_key):
+            config_attr[key] = args[i]
+            saved_attr = saved_attr + ' %s' % args[i]
+        with open('.moveconfig', 'w') as f:
+            f.write(saved_attr)
+    elif os.path.exists('.moveconfig'):
+        f = open('.moveconfig')
+        try:
+            saved_attr = f.readline().split()
+            for i, key in enumerate(args_key):
+                config_attr[key] = saved_attr[i]
+        except:
+            sys.exit('%s: broken format' % '.moveconfig')
+    else:
+        parser.print_usage()
+        sys.exit(1)
+
+    if not config_attr['config'].startswith('CONFIG_'):
+        config_attr['config'] = 'CONFIG_' + config_attr['config']
+
+    move_config(config_attr, options.jobs)
+
+if __name__ == '__main__':
+    main()
-- 
1.9.1

[U-Boot] [PATCH] Don't apply: tools: add a tool to move automatically CONFIGs from headers to defconfigs

[Simon Glass]

Hi Masahiro,

On 19 January 2015 at 05:12, Masahiro Yamada <yamada.m@jp.panasonic.com> wrote:
> This tool can move CONFIG macros from C headers (include/configs/*.h)
> to defconfigs (configs/*_defconfig) all over the boards.
>
> There are tons of CONFIGs in U-boot and moving them by hand is
> absolutely tool painful.  I wrote this script for my local use,
> so this patch might not clean enough to be applied to the code base.
> It might contain some bugs.
>
> Please do not apply this patch!
>
> This patch is here because Simon requested me to share it.
> See the discussion in here:
> http://lists.denx.de/pipermail/u-boot/2015-January/201556.html
>
> Usage:
>
> [1] Please run "git status" and make sure your git repository is clean.
>
> [2] Describe the CONFIG option you want move in the file "~/.moveconfig"
>
>   Say, you want to move CONFIG_CMD_NAND, for example.
>   In this case, the content of "~/.moveconfig" should be like this:
>
>   $ cat .moveconfig
>   CONFIG_CMD_NAND bool n y
>
>   - The first field is the name of the CONFIG.
>   - The second field is the type of the CONFIG such as "bool", "int", "hex", etc.
>   - The third value is the default value.
>     The value that is same as the default is omitted in each defconfig.
>     If the type of the CONFIG is bool, the default value must be either 'y' or 'n'.
>   - The forth field shows whether the CONFIG depends on CONFIG_SPL_BUILD.
>     For example, CONFIG_CMD_* makes sense only on the main-image.
>     (depends on !SPL_BUILD)  In this case, specify 'y' in the forth fields.
>     If the CONFIG should appear also on SPL, specify 'n' here.
>
> [4] Run "tools/moveconfig.py"
>
>   The log will be displayed like this
>
>   $ tools/moveconfig.py
>   Moving CONFIG_CMD_NAND (type: bool, default: n, no_spl: y) ...  (jobs: 8)
>   ms7750se                             : (default)
>   davinci_sonata                       : (default)
>   tk71                                 : y
>   db-mv784mp-gp                        : (default)
>   cm_t3517                             : y
>   P2041RDB_SDCARD                      : y
>         ...
>
>   The left-hand side field is the board[defconfig] name and the colon
>   is followed by the value of the CONFIG.
>
>   At the last stage, you will be asked like this:
>
>   Clean up CONFIG_CMD_NAND in headers? [y/n]:
>
>   If you say 'y' here, the defines in C headers will be removed.
>
> Enjoy!

Just a note to say that I used this tool to handle the BOOTSTAGE
patch. It worked very nicely, thank you.!

I don't think it would take much to get this into shape for applying.
What do you think?

>
> Signed-off-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
> Cc: Simon Glass <sjg@chromium.org>
> Cc: Alexey Brodkin <abrodkin@synopsys.com>
> ---
>
>  tools/moveconfig.py | 439 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 439 insertions(+)
>  create mode 100755 tools/moveconfig.py
>

Regards,
Simon

[U-Boot] [PATCH] Don't apply: tools: add a tool to move automatically CONFIGs from headers to defconfigs

[Masahiro Yamada]

Hi Simon,
Sorry for my late reply.

2015-03-04 9:18 GMT+09:00 Simon Glass <sjg@chromium.org>:
> Hi Masahiro,
>
> On 19 January 2015 at 05:12, Masahiro Yamada <yamada.m@jp.panasonic.com> wrote:
>> This tool can move CONFIG macros from C headers (include/configs/*.h)
>> to defconfigs (configs/*_defconfig) all over the boards.
>>
>> There are tons of CONFIGs in U-boot and moving them by hand is
>> absolutely tool painful.  I wrote this script for my local use,
>> so this patch might not clean enough to be applied to the code base.
>> It might contain some bugs.
>>
>> Please do not apply this patch!
>>
>> This patch is here because Simon requested me to share it.
>> See the discussion in here:
>> http://lists.denx.de/pipermail/u-boot/2015-January/201556.html
>>
>> Usage:
>>
>> [1] Please run "git status" and make sure your git repository is clean.
>>
>> [2] Describe the CONFIG option you want move in the file "~/.moveconfig"
>>
>>   Say, you want to move CONFIG_CMD_NAND, for example.
>>   In this case, the content of "~/.moveconfig" should be like this:
>>
>>   $ cat .moveconfig
>>   CONFIG_CMD_NAND bool n y
>>
>>   - The first field is the name of the CONFIG.
>>   - The second field is the type of the CONFIG such as "bool", "int", "hex", etc.
>>   - The third value is the default value.
>>     The value that is same as the default is omitted in each defconfig.
>>     If the type of the CONFIG is bool, the default value must be either 'y' or 'n'.
>>   - The forth field shows whether the CONFIG depends on CONFIG_SPL_BUILD.
>>     For example, CONFIG_CMD_* makes sense only on the main-image.
>>     (depends on !SPL_BUILD)  In this case, specify 'y' in the forth fields.
>>     If the CONFIG should appear also on SPL, specify 'n' here.
>>
>> [4] Run "tools/moveconfig.py"
>>
>>   The log will be displayed like this
>>
>>   $ tools/moveconfig.py
>>   Moving CONFIG_CMD_NAND (type: bool, default: n, no_spl: y) ...  (jobs: 8)
>>   ms7750se                             : (default)
>>   davinci_sonata                       : (default)
>>   tk71                                 : y
>>   db-mv784mp-gp                        : (default)
>>   cm_t3517                             : y
>>   P2041RDB_SDCARD                      : y
>>         ...
>>
>>   The left-hand side field is the board[defconfig] name and the colon
>>   is followed by the value of the CONFIG.
>>
>>   At the last stage, you will be asked like this:
>>
>>   Clean up CONFIG_CMD_NAND in headers? [y/n]:
>>
>>   If you say 'y' here, the defines in C headers will be removed.
>>
>> Enjoy!
>
> Just a note to say that I used this tool to handle the BOOTSTAGE
> patch. It worked very nicely, thank you.!
>
> I don't think it would take much to get this into shape for applying.
> What do you think?

OK, I will do it.


Best Regards
Masahiro Yamada

[U-Boot] [PATCH 1/5] moveconfig: Actually build autoconf.mk

[Joe Hershberger]

The existing target won't actually build the needed .mk file that is
expected by the rest of the script. It seems that silentoldconfig does
not actually cause this to be built any longer.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
---
This patch is based on:
 "Don't apply: tools: add a tool to move automatically CONFIGs from headers to defconfigs"
from the list.

 tools/moveconfig.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index cb7f7ea..30dc4f6 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -301,7 +301,7 @@ class Slot:
         cmd = list(self.make_cmd)
         if cross_compile:
             cmd.append('CROSS_COMPILE=%s' % cross_compile)
-        cmd.append('silentoldconfig')
+        cmd.append('include/autoconf.mk')
         self.ps = subprocess.Popen(cmd, stdout=self.devnull)
         self.state = STATE_SILENTOLDCONFIG
         return False
-- 
1.7.11.5

[U-Boot] [PATCH 2/5] moveconfig: Add a mapping for the arc cross-compiler

[Joe Hershberger]

The arc architecture is supported by U-Boot, so add a mapping here
for it as well.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
---

 tools/moveconfig.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 30dc4f6..c81f32c 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -26,6 +26,7 @@ SLEEP_TIME=0.03
 
 CROSS_COMPILE = {
     'aarch64': 'aarch64-linux-',
+    'arc': 'arc-linux-',
     'arm': 'arm-unknown-linux-gnueabi-',
     'avr32': 'avr32-linux-',
     'blackfin': 'bfin-elf-',
-- 
1.7.11.5

[U-Boot] [PATCH 3/5] moveconfig: Continue moving even if one board fails

[Joe Hershberger]

Some compilers are hard to come by or have so few boards they are not
worth messing with for this tool. Provide a list that need manual
intervention and continue moving the bulk of boards.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
---

 tools/moveconfig.py | 29 +++++++++++++++++++++++++++--
 1 file changed, 27 insertions(+), 2 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index c81f32c..ab53476 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -17,6 +17,7 @@ import os
 import re
 import shutil
 import subprocess
+from subprocess import PIPE
 import sys
 import tempfile
 import time
@@ -277,6 +278,13 @@ class Slot:
         self.state = STATE_DEFCONFIG
         return True
 
+    def defconfig_error(self, errmsg):
+        output = self.defconfig[:-len('_defconfig')].ljust(37) + ': '
+        print output + errmsg
+
+        """Save a list of targets that have to be checked by hand"""
+        open('moveconfig.failed', 'a+').write("%s\n" % self.defconfig)
+
     def poll(self):
         """Check if the subprocess is running and invoke the .config
         parser if the subprocess is terminated.
@@ -291,7 +299,13 @@ class Slot:
             return False
 
         if self.ps.poll() != 0:
-            sys.exit("failed to process '%s'" % self.defconfig)
+            errmsg = 'ERROR - build error'
+            errout = self.ps.stderr.read()
+            if errout.find('gcc: command not found') != -1:
+                errmsg = 'ERROR - compiler not found'
+            self.defconfig_error(errmsg)
+            self.state = STATE_IDLE
+            return True
 
         if self.state == STATE_SILENTOLDCONFIG:
             self.parser.update_defconfig(self.defconfig)
@@ -303,7 +317,7 @@ class Slot:
         if cross_compile:
             cmd.append('CROSS_COMPILE=%s' % cross_compile)
         cmd.append('include/autoconf.mk')
-        self.ps = subprocess.Popen(cmd, stdout=self.devnull)
+        self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE)
         self.state = STATE_SILENTOLDCONFIG
         return False
 
@@ -363,6 +377,11 @@ class Slots:
 
 def move_config(config_attr, jobs=1):
     check_top_directory()
+
+    """Clean up any previous log of failed moves"""
+    if os.path.exists('moveconfig.failed'):
+        os.remove('moveconfig.failed')
+
     print 'Moving %s (type: %s, default: %s, no_spl: %s) ...  (jobs: %d)' % (
         config_attr['config'],
         config_attr['type'],
@@ -396,6 +415,12 @@ def move_config(config_attr, jobs=1):
 
     cleanup_headers(config_attr['config'])
 
+    if os.path.exists('moveconfig.failed'):
+        print '!!!  Some boards were not processed; move the config manually.'
+        print '!!!  The list of failed boards are saved in moveconfig.failed'
+        print
+        print open('moveconfig.failed', 'r').read()
+
 def main():
     try:
         cpu_count = multiprocessing.cpu_count()
-- 
1.7.11.5

[U-Boot] [PATCH 4/5] moveconfig: Error if missing the include/autoconf.mk

[Joe Hershberger]

The main image autoconf.mk certainly had better exist. I'm not sure
about SPL and TPL, but at least this one. Check for this and error if
missing.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
---

 tools/moveconfig.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index ab53476..7635186 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -184,6 +184,8 @@ class KconfigParser:
                                     'autoconf.mk')
 
             if not os.path.exists(autoconf):
+                if img == '.':
+                    return True
                 values.append('')
                 continue
 
@@ -228,6 +230,7 @@ class KconfigParser:
                 if prefixes[line] != '+':
                     line = prefixes[line] + ':' + line
                 f.write(line + '\n')
+        return False
 
 class Slot:
 
@@ -308,7 +311,8 @@ class Slot:
             return True
 
         if self.state == STATE_SILENTOLDCONFIG:
-            self.parser.update_defconfig(self.defconfig)
+            if self.parser.update_defconfig(self.defconfig):
+                self.defconfig_error('ERROR - autoconf.mk not found')
             self.state = STATE_IDLE
             return True
 
-- 
1.7.11.5

[U-Boot] [PATCH 5/5] moveconfig: Always run savedefconfig on the moved config

[Joe Hershberger]

This will ensure that the order of the defconfig entries will always
match that of the Kconfig files. After one slightly painful (but
still early in the process) pass over all boards, this should keep
the defconfigs clean from here on.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
---

 tools/moveconfig.py | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 7635186..02ec976 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -46,6 +46,7 @@ CROSS_COMPILE = {
 STATE_IDLE = 0
 STATE_DEFCONFIG = 1
 STATE_SILENTOLDCONFIG = 2
+STATE_SAVEDEFCONFIG = 3
 
 ### helper functions ###
 def get_devnull():
@@ -150,8 +151,8 @@ class KconfigParser:
         """
         arch = ''
         cpu = ''
-        dotconfig = os.path.join(self.build_dir, '.config')
-        for line in open(dotconfig):
+        self.dotconfig = os.path.join(self.build_dir, '.config')
+        for line in open(self.dotconfig):
             m = self.re_arch.match(line)
             if m:
                 arch = m.group(1)
@@ -225,7 +226,7 @@ class KconfigParser:
 
         print output.strip()
 
-        with open(os.path.join('configs', defconfig), 'a') as f:
+        with open(os.path.join(self.dotconfig), 'a') as f:
             for line in output_lines:
                 if prefixes[line] != '+':
                     line = prefixes[line] + ':' + line
@@ -313,6 +314,23 @@ class Slot:
         if self.state == STATE_SILENTOLDCONFIG:
             if self.parser.update_defconfig(self.defconfig):
                 self.defconfig_error('ERROR - autoconf.mk not found')
+                self.state = STATE_IDLE
+                return True
+
+            """Save off the defconfig in a consistent way"""
+            cmd = list(self.make_cmd)
+            cmd.append('savedefconfig')
+            self.ps = subprocess.Popen(cmd, stdout=self.devnull,
+                                       stderr=self.devnull)
+            self.state = STATE_SAVEDEFCONFIG
+            return False
+
+        if self.state == STATE_SAVEDEFCONFIG:
+            defconfig_path = os.path.join(self.build_dir, 'defconfig')
+            if not os.path.exists(defconfig_path):
+                self.defconfig_error('ERROR - defconfig not updated')
+            shutil.move(defconfig_path,
+                        os.path.join('configs', self.defconfig))
             self.state = STATE_IDLE
             return True
 
-- 
1.7.11.5

[U-Boot] [PATCH v2 1/7] moveconfig: Actually build autoconf.mk

[Joe Hershberger]

The existing target won't actually build the needed .mk file that is
expected by the rest of the script. It seems that silentoldconfig does
not actually cause this to be built any longer.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
---
This patch is based on:
 "Don't apply: tools: add a tool to move automatically CONFIGs from headers to defconfigs"
from the list.

Changes in v2: None

 tools/moveconfig.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index cb7f7ea..30dc4f6 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -301,7 +301,7 @@ class Slot:
         cmd = list(self.make_cmd)
         if cross_compile:
             cmd.append('CROSS_COMPILE=%s' % cross_compile)
-        cmd.append('silentoldconfig')
+        cmd.append('include/autoconf.mk')
         self.ps = subprocess.Popen(cmd, stdout=self.devnull)
         self.state = STATE_SILENTOLDCONFIG
         return False
-- 
1.7.11.5

[U-Boot] [PATCH v2 2/7] moveconfig: Add a mapping for the arc cross-compiler

[Joe Hershberger]

The arc architecture is supported by U-Boot, so add a mapping here
for it as well.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
---

Changes in v2: None

 tools/moveconfig.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 30dc4f6..c81f32c 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -26,6 +26,7 @@ SLEEP_TIME=0.03
 
 CROSS_COMPILE = {
     'aarch64': 'aarch64-linux-',
+    'arc': 'arc-linux-',
     'arm': 'arm-unknown-linux-gnueabi-',
     'avr32': 'avr32-linux-',
     'blackfin': 'bfin-elf-',
-- 
1.7.11.5

[U-Boot] [PATCH v2 2/7] moveconfig: Add a mapping for the arc cross-compiler

[Masahiro Yamada]

Hi Joe.

2015-05-12 2:23 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
> The arc architecture is supported by U-Boot, so add a mapping here
> for it as well.
>
> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
> ---
>
> Changes in v2: None
>
>  tools/moveconfig.py | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
> index 30dc4f6..c81f32c 100755
> --- a/tools/moveconfig.py
> +++ b/tools/moveconfig.py
> @@ -26,6 +26,7 @@ SLEEP_TIME=0.03
>
>  CROSS_COMPILE = {
>      'aarch64': 'aarch64-linux-',
> +    'arc': 'arc-linux-',
>      'arm': 'arm-unknown-linux-gnueabi-',
>      'avr32': 'avr32-linux-',
>      'blackfin': 'bfin-elf-',


IIRC, I intentionally did not add it
because the default ARC CROSS_COMPILE is defined by arch/arc/config.mk
(Looks like ARC wants to choose little/big endian variants
depending on the configuration, although it might not be an issue for
this tool.)

Better to override it?


-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH v2 2/7] moveconfig: Add a mapping for the arc cross-compiler

[Joe Hershberger]

Hi Masahiro-san,

On Tue, May 12, 2015 at 9:14 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Hi Joe.
>
> 2015-05-12 2:23 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
>> The arc architecture is supported by U-Boot, so add a mapping here
>> for it as well.
>>
>> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>> ---
>>
>> Changes in v2: None
>>
>>  tools/moveconfig.py | 1 +
>>  1 file changed, 1 insertion(+)
>>
>> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
>> index 30dc4f6..c81f32c 100755
>> --- a/tools/moveconfig.py
>> +++ b/tools/moveconfig.py
>> @@ -26,6 +26,7 @@ SLEEP_TIME=0.03
>>
>>  CROSS_COMPILE = {
>>      'aarch64': 'aarch64-linux-',
>> +    'arc': 'arc-linux-',
>>      'arm': 'arm-unknown-linux-gnueabi-',
>>      'avr32': 'avr32-linux-',
>>      'blackfin': 'bfin-elf-',
>
>
> IIRC, I intentionally did not add it
> because the default ARC CROSS_COMPILE is defined by arch/arc/config.mk
> (Looks like ARC wants to choose little/big endian variants
> depending on the configuration, although it might not be an issue for
> this tool.)

Thanks for pointing that out.

> Better to override it?

I am successfully building, at least to get the config output, without
switching to a big-endian compiler. I can build the other compiler too
if you prefer not to map this.

Cheers,
-Joe

[U-Boot] [PATCH v2 3/7] moveconfig: Continue moving even if one board fails

[Joe Hershberger]

Some compilers are hard to come by or have so few boards they are not
worth messing with for this tool. Provide a list that need manual
intervention and continue moving the bulk of boards.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>

---

Changes in v2:
-Print which compiler is missing

 tools/moveconfig.py | 35 ++++++++++++++++++++++++++++++-----
 1 file changed, 30 insertions(+), 5 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index c81f32c..315f4be 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -17,6 +17,7 @@ import os
 import re
 import shutil
 import subprocess
+from subprocess import PIPE
 import sys
 import tempfile
 import time
@@ -277,6 +278,13 @@ class Slot:
         self.state = STATE_DEFCONFIG
         return True
 
+    def defconfig_error(self, errmsg):
+        output = self.defconfig[:-len('_defconfig')].ljust(37) + ': '
+        print output + errmsg
+
+        """Save a list of targets that have to be checked by hand"""
+        open('moveconfig.failed', 'a+').write("%s\n" % self.defconfig)
+
     def poll(self):
         """Check if the subprocess is running and invoke the .config
         parser if the subprocess is terminated.
@@ -291,19 +299,25 @@ class Slot:
             return False
 
         if self.ps.poll() != 0:
-            sys.exit("failed to process '%s'" % self.defconfig)
+            errmsg = 'ERROR - build error'
+            errout = self.ps.stderr.read()
+            if errout.find('gcc: command not found') != -1:
+                errmsg = 'ERROR - compiler not found (%s)' % self.cross_compile
+            self.defconfig_error(errmsg)
+            self.state = STATE_IDLE
+            return True
 
         if self.state == STATE_SILENTOLDCONFIG:
             self.parser.update_defconfig(self.defconfig)
             self.state = STATE_IDLE
             return True
 
-        cross_compile = self.parser.get_cross_compile()
+        self.cross_compile = self.parser.get_cross_compile()
         cmd = list(self.make_cmd)
-        if cross_compile:
-            cmd.append('CROSS_COMPILE=%s' % cross_compile)
+        if self.cross_compile:
+            cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
         cmd.append('include/autoconf.mk')
-        self.ps = subprocess.Popen(cmd, stdout=self.devnull)
+        self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE)
         self.state = STATE_SILENTOLDCONFIG
         return False
 
@@ -363,6 +377,7 @@ class Slots:
 
 def move_config(config_attr, jobs=1):
     check_top_directory()
+
     print 'Moving %s (type: %s, default: %s, no_spl: %s) ...  (jobs: %d)' % (
         config_attr['config'],
         config_attr['type'],
@@ -379,6 +394,10 @@ def move_config(config_attr, jobs=1):
                 continue
             defconfigs.append(os.path.join(dirpath, filename))
 
+    """Clean up any previous log of failed moves"""
+    if os.path.exists('moveconfig.failed'):
+        os.remove('moveconfig.failed')
+
     slots = Slots(config_attr, jobs)
 
     # Main loop to process defconfig files:
@@ -396,6 +415,12 @@ def move_config(config_attr, jobs=1):
 
     cleanup_headers(config_attr['config'])
 
+    if os.path.exists('moveconfig.failed'):
+        print '!!!  Some boards were not processed; move the config manually.'
+        print '!!!  The list of failed boards are saved in moveconfig.failed'
+        print
+        print open('moveconfig.failed', 'r').read()
+
 def main():
     try:
         cpu_count = multiprocessing.cpu_count()
-- 
1.7.11.5

[U-Boot] [PATCH v2 4/7] moveconfig: Error if missing the include/autoconf.mk

[Joe Hershberger]

The main image autoconf.mk certainly had better exist. I'm not sure
about SPL and TPL, but at least this one. Check for this and error if
missing.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
---

Changes in v2: None

 tools/moveconfig.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 315f4be..2e5f2ce 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -184,6 +184,8 @@ class KconfigParser:
                                     'autoconf.mk')
 
             if not os.path.exists(autoconf):
+                if img == '.':
+                    return True
                 values.append('')
                 continue
 
@@ -228,6 +230,7 @@ class KconfigParser:
                 if prefixes[line] != '+':
                     line = prefixes[line] + ':' + line
                 f.write(line + '\n')
+        return False
 
 class Slot:
 
@@ -308,7 +311,8 @@ class Slot:
             return True
 
         if self.state == STATE_SILENTOLDCONFIG:
-            self.parser.update_defconfig(self.defconfig)
+            if self.parser.update_defconfig(self.defconfig):
+                self.defconfig_error('ERROR - autoconf.mk not found')
             self.state = STATE_IDLE
             return True
 
-- 
1.7.11.5

[U-Boot] [PATCH v2 5/7] moveconfig: Always run savedefconfig on the moved config

[Joe Hershberger]

This will ensure that the order of the defconfig entries will always
match that of the Kconfig files. After one slightly painful (but
still early in the process) pass over all boards, this should keep
the defconfigs clean from here on.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
---

Changes in v2: None

 tools/moveconfig.py | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 2e5f2ce..5572be9 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -46,6 +46,7 @@ CROSS_COMPILE = {
 STATE_IDLE = 0
 STATE_DEFCONFIG = 1
 STATE_SILENTOLDCONFIG = 2
+STATE_SAVEDEFCONFIG = 3
 
 ### helper functions ###
 def get_devnull():
@@ -150,8 +151,8 @@ class KconfigParser:
         """
         arch = ''
         cpu = ''
-        dotconfig = os.path.join(self.build_dir, '.config')
-        for line in open(dotconfig):
+        self.dotconfig = os.path.join(self.build_dir, '.config')
+        for line in open(self.dotconfig):
             m = self.re_arch.match(line)
             if m:
                 arch = m.group(1)
@@ -225,7 +226,7 @@ class KconfigParser:
 
         print output.strip()
 
-        with open(os.path.join('configs', defconfig), 'a') as f:
+        with open(os.path.join(self.dotconfig), 'a') as f:
             for line in output_lines:
                 if prefixes[line] != '+':
                     line = prefixes[line] + ':' + line
@@ -313,6 +314,23 @@ class Slot:
         if self.state == STATE_SILENTOLDCONFIG:
             if self.parser.update_defconfig(self.defconfig):
                 self.defconfig_error('ERROR - autoconf.mk not found')
+                self.state = STATE_IDLE
+                return True
+
+            """Save off the defconfig in a consistent way"""
+            cmd = list(self.make_cmd)
+            cmd.append('savedefconfig')
+            self.ps = subprocess.Popen(cmd, stdout=self.devnull,
+                                       stderr=self.devnull)
+            self.state = STATE_SAVEDEFCONFIG
+            return False
+
+        if self.state == STATE_SAVEDEFCONFIG:
+            defconfig_path = os.path.join(self.build_dir, 'defconfig')
+            if not os.path.exists(defconfig_path):
+                self.defconfig_error('ERROR - defconfig not updated')
+            shutil.move(defconfig_path,
+                        os.path.join('configs', self.defconfig))
             self.state = STATE_IDLE
             return True
 
-- 
1.7.11.5

[U-Boot] [PATCH v2 6/7] moveconfig: Add a parameter to accept a list to build

[Joe Hershberger]

This is helpful to re-attempt to move failed boards from a previous run
without starting over.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
---

Changes in v2: None

 tools/moveconfig.py | 31 ++++++++++++++++++-------------
 1 file changed, 18 insertions(+), 13 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 5572be9..ad9dbac 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -397,7 +397,7 @@ class Slots:
                 ret = False
         return ret
 
-def move_config(config_attr, jobs=1):
+def move_config(config_attr, options):
     check_top_directory()
 
     print 'Moving %s (type: %s, default: %s, no_spl: %s) ...  (jobs: %d)' % (
@@ -405,22 +405,25 @@ def move_config(config_attr, jobs=1):
         config_attr['type'],
         config_attr['default'],
         config_attr['no_spl_support'],
-        jobs)
-
-    # All the defconfig files to be processed
-    defconfigs = []
-    for (dirpath, dirnames, filenames) in os.walk('configs'):
-        dirpath = dirpath[len('configs') + 1:]
-        for filename in fnmatch.filter(filenames, '*_defconfig'):
-            if fnmatch.fnmatch(filename, '.*'):
-                continue
-            defconfigs.append(os.path.join(dirpath, filename))
+        options.jobs)
+
+    if options.defconfigs:
+        defconfigs = [line.strip() for line in open(options.defconfigs, 'r')]
+    else:
+        # All the defconfig files to be processed
+        defconfigs = []
+        for (dirpath, dirnames, filenames) in os.walk('configs'):
+            dirpath = dirpath[len('configs') + 1:]
+            for filename in fnmatch.filter(filenames, '*_defconfig'):
+                if fnmatch.fnmatch(filename, '.*'):
+                    continue
+                defconfigs.append(os.path.join(dirpath, filename))
 
     """Clean up any previous log of failed moves"""
     if os.path.exists('moveconfig.failed'):
         os.remove('moveconfig.failed')
 
-    slots = Slots(config_attr, jobs)
+    slots = Slots(config_attr, options.jobs)
 
     # Main loop to process defconfig files:
     #  Add a new subprocess into a vacant slot.
@@ -453,6 +456,8 @@ def main():
     # Add options here
     parser.add_option('-j', '--jobs', type='int', default=cpu_count,
                       help='the number of jobs to run simultaneously')
+    parser.add_option('-d', '--defconfigs', type='string',
+                      help='a file containing a list of defconfigs to move')
     parser.usage += ' config type default no_spl_support'
     (options, args) = parser.parse_args()
 
@@ -481,7 +486,7 @@ def main():
     if not config_attr['config'].startswith('CONFIG_'):
         config_attr['config'] = 'CONFIG_' + config_attr['config']
 
-    move_config(config_attr, options.jobs)
+    move_config(config_attr, options)
 
 if __name__ == '__main__':
     main()
-- 
1.7.11.5

[U-Boot] [PATCH v2 7/7] moveconfig: Add a switch to enable printing errors

[Joe Hershberger]

In some cases the build for the autoconf breaks. This outputs the errors
following the status so that action can be taken without building again
manually.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
---

Changes in v2: None

 tools/moveconfig.py | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index ad9dbac..b5c3157 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -242,7 +242,7 @@ class Slot:
     for faster processing.
     """
 
-    def __init__(self, config_attr, devnull, make_cmd):
+    def __init__(self, config_attr, devnull, make_cmd, options):
         """Create a new slot.
 
         Arguments:
@@ -251,6 +251,7 @@ class Slot:
         self.build_dir = tempfile.mkdtemp()
         self.devnull = devnull
         self.make_cmd = (make_cmd, 'O=' + self.build_dir)
+        self.options = options
         self.parser = KconfigParser(self.build_dir, config_attr)
         self.state = STATE_IDLE
 
@@ -308,6 +309,8 @@ class Slot:
             if errout.find('gcc: command not found') != -1:
                 errmsg = 'ERROR - compiler not found (%s)' % self.cross_compile
             self.defconfig_error(errmsg)
+            if self.options.verbose:
+                print errout
             self.state = STATE_IDLE
             return True
 
@@ -347,7 +350,7 @@ class Slots:
 
     """Controller of the array of subprocess slots."""
 
-    def __init__(self, config_attr, jobs):
+    def __init__(self, config_attr, options):
         """Create a new slots controller.
 
         Arguments:
@@ -356,8 +359,8 @@ class Slots:
         self.slots = []
         devnull = get_devnull()
         make_cmd = get_make_cmd()
-        for i in range(jobs):
-            self.slots.append(Slot(config_attr, devnull, make_cmd))
+        for i in range(options.jobs):
+            self.slots.append(Slot(config_attr, devnull, make_cmd, options))
 
     def add(self, defconfig):
         """Add a new subprocess if a vacant slot is available.
@@ -423,7 +426,7 @@ def move_config(config_attr, options):
     if os.path.exists('moveconfig.failed'):
         os.remove('moveconfig.failed')
 
-    slots = Slots(config_attr, options.jobs)
+    slots = Slots(config_attr, options)
 
     # Main loop to process defconfig files:
     #  Add a new subprocess into a vacant slot.
@@ -458,6 +461,9 @@ def main():
                       help='the number of jobs to run simultaneously')
     parser.add_option('-d', '--defconfigs', type='string',
                       help='a file containing a list of defconfigs to move')
+    parser.add_option('-v', '--verbose', dest='verbose',
+                      action='store_true', default=False,
+                      help='show any build errors as boards are build')
     parser.usage += ' config type default no_spl_support'
     (options, args) = parser.parse_args()
 
-- 
1.7.11.5

[U-Boot] [PATCH v3 01/10] moveconfig: Actually build autoconf.mk

[Joe Hershberger]

The existing target won't actually build the needed .mk file that is
expected by the rest of the script. It seems that silentoldconfig does
not actually cause this to be built any longer.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
---
This patch is based on:
 "Don't apply: tools: add a tool to move automatically CONFIGs from headers to defconfigs"
from the list.

Also dropped the patch that adds an arc cross-tool mapping.

Changes in v3: None
Changes in v2: None

 tools/moveconfig.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index cb7f7ea..30dc4f6 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -301,7 +301,7 @@ class Slot:
         cmd = list(self.make_cmd)
         if cross_compile:
             cmd.append('CROSS_COMPILE=%s' % cross_compile)
-        cmd.append('silentoldconfig')
+        cmd.append('include/autoconf.mk')
         self.ps = subprocess.Popen(cmd, stdout=self.devnull)
         self.state = STATE_SILENTOLDCONFIG
         return False
-- 
1.7.11.5

[U-Boot] [PATCH v3 02/10] moveconfig: Continue moving even if one board fails

[Joe Hershberger]

Some compilers are hard to come by or have so few boards they are not
worth messing with for this tool. Provide a list that need manual
intervention and continue moving the bulk of boards.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>

---

Changes in v3: None
Changes in v2:
-Print which compiler is missing

 tools/moveconfig.py | 35 ++++++++++++++++++++++++++++++-----
 1 file changed, 30 insertions(+), 5 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 30dc4f6..d9ae859 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -17,6 +17,7 @@ import os
 import re
 import shutil
 import subprocess
+from subprocess import PIPE
 import sys
 import tempfile
 import time
@@ -276,6 +277,13 @@ class Slot:
         self.state = STATE_DEFCONFIG
         return True
 
+    def defconfig_error(self, errmsg):
+        output = self.defconfig[:-len('_defconfig')].ljust(37) + ': '
+        print output + errmsg
+
+        """Save a list of targets that have to be checked by hand"""
+        open('moveconfig.failed', 'a+').write("%s\n" % self.defconfig)
+
     def poll(self):
         """Check if the subprocess is running and invoke the .config
         parser if the subprocess is terminated.
@@ -290,19 +298,25 @@ class Slot:
             return False
 
         if self.ps.poll() != 0:
-            sys.exit("failed to process '%s'" % self.defconfig)
+            errmsg = 'ERROR - build error'
+            errout = self.ps.stderr.read()
+            if errout.find('gcc: command not found') != -1:
+                errmsg = 'ERROR - compiler not found (%s)' % self.cross_compile
+            self.defconfig_error(errmsg)
+            self.state = STATE_IDLE
+            return True
 
         if self.state == STATE_SILENTOLDCONFIG:
             self.parser.update_defconfig(self.defconfig)
             self.state = STATE_IDLE
             return True
 
-        cross_compile = self.parser.get_cross_compile()
+        self.cross_compile = self.parser.get_cross_compile()
         cmd = list(self.make_cmd)
-        if cross_compile:
-            cmd.append('CROSS_COMPILE=%s' % cross_compile)
+        if self.cross_compile:
+            cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
         cmd.append('include/autoconf.mk')
-        self.ps = subprocess.Popen(cmd, stdout=self.devnull)
+        self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE)
         self.state = STATE_SILENTOLDCONFIG
         return False
 
@@ -362,6 +376,7 @@ class Slots:
 
 def move_config(config_attr, jobs=1):
     check_top_directory()
+
     print 'Moving %s (type: %s, default: %s, no_spl: %s) ...  (jobs: %d)' % (
         config_attr['config'],
         config_attr['type'],
@@ -378,6 +393,10 @@ def move_config(config_attr, jobs=1):
                 continue
             defconfigs.append(os.path.join(dirpath, filename))
 
+    """Clean up any previous log of failed moves"""
+    if os.path.exists('moveconfig.failed'):
+        os.remove('moveconfig.failed')
+
     slots = Slots(config_attr, jobs)
 
     # Main loop to process defconfig files:
@@ -395,6 +414,12 @@ def move_config(config_attr, jobs=1):
 
     cleanup_headers(config_attr['config'])
 
+    if os.path.exists('moveconfig.failed'):
+        print '!!!  Some boards were not processed; move the config manually.'
+        print '!!!  The list of failed boards are saved in moveconfig.failed'
+        print
+        print open('moveconfig.failed', 'r').read()
+
 def main():
     try:
         cpu_count = multiprocessing.cpu_count()
-- 
1.7.11.5

[U-Boot] [PATCH v3 02/10] moveconfig: Continue moving even if one board fails

[Masahiro Yamada]

2015-05-14 7:28 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
> Some compilers are hard to come by or have so few boards they are not
> worth messing with for this tool. Provide a list that need manual
> intervention and continue moving the bulk of boards.
>
> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>
> ---
>
> Changes in v3: None
> Changes in v2:
> -Print which compiler is missing
>
>  tools/moveconfig.py | 35 ++++++++++++++++++++++++++++++-----
>  1 file changed, 30 insertions(+), 5 deletions(-)
>
> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
> index 30dc4f6..d9ae859 100755
> --- a/tools/moveconfig.py
> +++ b/tools/moveconfig.py
> @@ -17,6 +17,7 @@ import os
>  import re
>  import shutil
>  import subprocess
> +from subprocess import PIPE
>  import sys
>  import tempfile
>  import time
> @@ -276,6 +277,13 @@ class Slot:
>          self.state = STATE_DEFCONFIG
>          return True
>
> +    def defconfig_error(self, errmsg):
> +        output = self.defconfig[:-len('_defconfig')].ljust(37) + ': '
> +        print output + errmsg
> +
> +        """Save a list of targets that have to be checked by hand"""
> +        open('moveconfig.failed', 'a+').write("%s\n" % self.defconfig)
> +
>      def poll(self):
>          """Check if the subprocess is running and invoke the .config
>          parser if the subprocess is terminated.
> @@ -290,19 +298,25 @@ class Slot:
>              return False
>
>          if self.ps.poll() != 0:
> -            sys.exit("failed to process '%s'" % self.defconfig)
> +            errmsg = 'ERROR - build error'
> +            errout = self.ps.stderr.read()
> +            if errout.find('gcc: command not found') != -1:
> +                errmsg = 'ERROR - compiler not found (%s)' % self.cross_compile
> +            self.defconfig_error(errmsg)
> +            self.state = STATE_IDLE
> +            return True
>
>          if self.state == STATE_SILENTOLDCONFIG:
>              self.parser.update_defconfig(self.defconfig)
>              self.state = STATE_IDLE
>              return True
>
> -        cross_compile = self.parser.get_cross_compile()
> +        self.cross_compile = self.parser.get_cross_compile()
>          cmd = list(self.make_cmd)
> -        if cross_compile:
> -            cmd.append('CROSS_COMPILE=%s' % cross_compile)
> +        if self.cross_compile:
> +            cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
>          cmd.append('include/autoconf.mk')
> -        self.ps = subprocess.Popen(cmd, stdout=self.devnull)
> +        self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE)
>          self.state = STATE_SILENTOLDCONFIG
>          return False
>
> @@ -362,6 +376,7 @@ class Slots:
>
>  def move_config(config_attr, jobs=1):
>      check_top_directory()
> +
>      print 'Moving %s (type: %s, default: %s, no_spl: %s) ...  (jobs: %d)' % (
>          config_attr['config'],
>          config_attr['type'],
> @@ -378,6 +393,10 @@ def move_config(config_attr, jobs=1):
>                  continue
>              defconfigs.append(os.path.join(dirpath, filename))
>
> +    """Clean up any previous log of failed moves"""
> +    if os.path.exists('moveconfig.failed'):
> +        os.remove('moveconfig.failed')
> +
>      slots = Slots(config_attr, jobs)
>
>      # Main loop to process defconfig files:
> @@ -395,6 +414,12 @@ def move_config(config_attr, jobs=1):
>
>      cleanup_headers(config_attr['config'])
>
> +    if os.path.exists('moveconfig.failed'):
> +        print '!!!  Some boards were not processed; move the config manually.'
> +        print '!!!  The list of failed boards are saved in moveconfig.failed'
> +        print
> +        print open('moveconfig.failed', 'r').read()
> +
>  def main():
>      try:
>          cpu_count = multiprocessing.cpu_count()



I could successfully convert all the defconfigs with my toolchains,
but I thought this feature would be helpful.

I merged this feature in my new version with your signed-off-by
although the implementation is a bit different.



-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH v3 03/10] moveconfig: Error if missing the include/autoconf.mk

[Joe Hershberger]

The main image autoconf.mk certainly had better exist or this tool won't
function at all. Check for this and error if missing.

The SPL and TPL don't even exist as separate .config files any more.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
---

Changes in v3: None
Changes in v2: None

 tools/moveconfig.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index d9ae859..b4ee0e3 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -183,6 +183,8 @@ class KconfigParser:
                                     'autoconf.mk')
 
             if not os.path.exists(autoconf):
+                if img == '.':
+                    return True
                 values.append('')
                 continue
 
@@ -227,6 +229,7 @@ class KconfigParser:
                 if prefixes[line] != '+':
                     line = prefixes[line] + ':' + line
                 f.write(line + '\n')
+        return False
 
 class Slot:
 
@@ -307,7 +310,8 @@ class Slot:
             return True
 
         if self.state == STATE_SILENTOLDCONFIG:
-            self.parser.update_defconfig(self.defconfig)
+            if self.parser.update_defconfig(self.defconfig):
+                self.defconfig_error('ERROR - autoconf.mk not found')
             self.state = STATE_IDLE
             return True
 
-- 
1.7.11.5

[U-Boot] [PATCH v3 04/10] moveconfig: Always run savedefconfig on the moved config

[Joe Hershberger]

This will ensure that the order of the defconfig entries will always
match that of the Kconfig files. After one slightly painful (but
still early in the process) pass over all boards, this should keep
the defconfigs clean from here on.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
---

Changes in v3: None
Changes in v2: None

 tools/moveconfig.py | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index b4ee0e3..f2651a9 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -45,6 +45,7 @@ CROSS_COMPILE = {
 STATE_IDLE = 0
 STATE_DEFCONFIG = 1
 STATE_SILENTOLDCONFIG = 2
+STATE_SAVEDEFCONFIG = 3
 
 ### helper functions ###
 def get_devnull():
@@ -149,8 +150,8 @@ class KconfigParser:
         """
         arch = ''
         cpu = ''
-        dotconfig = os.path.join(self.build_dir, '.config')
-        for line in open(dotconfig):
+        self.dotconfig = os.path.join(self.build_dir, '.config')
+        for line in open(self.dotconfig):
             m = self.re_arch.match(line)
             if m:
                 arch = m.group(1)
@@ -224,7 +225,7 @@ class KconfigParser:
 
         print output.strip()
 
-        with open(os.path.join('configs', defconfig), 'a') as f:
+        with open(os.path.join(self.dotconfig), 'a') as f:
             for line in output_lines:
                 if prefixes[line] != '+':
                     line = prefixes[line] + ':' + line
@@ -312,6 +313,23 @@ class Slot:
         if self.state == STATE_SILENTOLDCONFIG:
             if self.parser.update_defconfig(self.defconfig):
                 self.defconfig_error('ERROR - autoconf.mk not found')
+                self.state = STATE_IDLE
+                return True
+
+            """Save off the defconfig in a consistent way"""
+            cmd = list(self.make_cmd)
+            cmd.append('savedefconfig')
+            self.ps = subprocess.Popen(cmd, stdout=self.devnull,
+                                       stderr=self.devnull)
+            self.state = STATE_SAVEDEFCONFIG
+            return False
+
+        if self.state == STATE_SAVEDEFCONFIG:
+            defconfig_path = os.path.join(self.build_dir, 'defconfig')
+            if not os.path.exists(defconfig_path):
+                self.defconfig_error('ERROR - defconfig not updated')
+            shutil.move(defconfig_path,
+                        os.path.join('configs', self.defconfig))
             self.state = STATE_IDLE
             return True
 
-- 
1.7.11.5

[U-Boot] [PATCH v3 04/10] moveconfig: Always run savedefconfig on the moved config

[Masahiro Yamada]

2015-05-14 7:28 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
> This will ensure that the order of the defconfig entries will always
> match that of the Kconfig files. After one slightly painful (but
> still early in the process) pass over all boards, this should keep
> the defconfigs clean from here on.
>
> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>


I a bit hesitate to always enable this feature
because savedefconfig might make git-diff  noisier.

Is it possible to make it optional?
-s, --savedefconfig  or --no-savedefconfig?



-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH v3 04/10] moveconfig: Always run savedefconfig on the moved config

[Joe Hershberger]

Hi Masahiro-san.

On Thu, May 14, 2015 at 10:15 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> 2015-05-14 7:28 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
>> This will ensure that the order of the defconfig entries will always
>> match that of the Kconfig files. After one slightly painful (but
>> still early in the process) pass over all boards, this should keep
>> the defconfigs clean from here on.
>>
>> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>
>
> I a bit hesitate to always enable this feature
> because savedefconfig might make git-diff  noisier.

That should never be the case. Its consistent use should always lead
to the least noisy diff.

> Is it possible to make it optional?
> -s, --savedefconfig  or --no-savedefconfig?

It is possible, but I recommend against it. Its inconsistent use would
lead to noisy diffs.

Cheers,
-Joe

[U-Boot] [PATCH v3 05/10] moveconfig: Add a parameter to accept a list to build

[Joe Hershberger]

This is helpful to re-attempt to move failed boards from a previous run
without starting over.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>

---

Changes in v3:
-Fixed command line options order (alphabetize)

Changes in v2:
-New for version 2

 tools/moveconfig.py | 31 ++++++++++++++++++-------------
 1 file changed, 18 insertions(+), 13 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index f2651a9..2688ad1 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -396,7 +396,7 @@ class Slots:
                 ret = False
         return ret
 
-def move_config(config_attr, jobs=1):
+def move_config(config_attr, options):
     check_top_directory()
 
     print 'Moving %s (type: %s, default: %s, no_spl: %s) ...  (jobs: %d)' % (
@@ -404,22 +404,25 @@ def move_config(config_attr, jobs=1):
         config_attr['type'],
         config_attr['default'],
         config_attr['no_spl_support'],
-        jobs)
-
-    # All the defconfig files to be processed
-    defconfigs = []
-    for (dirpath, dirnames, filenames) in os.walk('configs'):
-        dirpath = dirpath[len('configs') + 1:]
-        for filename in fnmatch.filter(filenames, '*_defconfig'):
-            if fnmatch.fnmatch(filename, '.*'):
-                continue
-            defconfigs.append(os.path.join(dirpath, filename))
+        options.jobs)
+
+    if options.defconfigs:
+        defconfigs = [line.strip() for line in open(options.defconfigs, 'r')]
+    else:
+        # All the defconfig files to be processed
+        defconfigs = []
+        for (dirpath, dirnames, filenames) in os.walk('configs'):
+            dirpath = dirpath[len('configs') + 1:]
+            for filename in fnmatch.filter(filenames, '*_defconfig'):
+                if fnmatch.fnmatch(filename, '.*'):
+                    continue
+                defconfigs.append(os.path.join(dirpath, filename))
 
     """Clean up any previous log of failed moves"""
     if os.path.exists('moveconfig.failed'):
         os.remove('moveconfig.failed')
 
-    slots = Slots(config_attr, jobs)
+    slots = Slots(config_attr, options.jobs)
 
     # Main loop to process defconfig files:
     #  Add a new subprocess into a vacant slot.
@@ -450,6 +453,8 @@ def main():
 
     parser = optparse.OptionParser()
     # Add options here
+    parser.add_option('-d', '--defconfigs', type='string',
+                      help='a file containing a list of defconfigs to move')
     parser.add_option('-j', '--jobs', type='int', default=cpu_count,
                       help='the number of jobs to run simultaneously')
     parser.usage += ' config type default no_spl_support'
@@ -480,7 +485,7 @@ def main():
     if not config_attr['config'].startswith('CONFIG_'):
         config_attr['config'] = 'CONFIG_' + config_attr['config']
 
-    move_config(config_attr, options.jobs)
+    move_config(config_attr, options)
 
 if __name__ == '__main__':
     main()
-- 
1.7.11.5

[U-Boot] [PATCH v3 05/10] moveconfig: Add a parameter to accept a list to build

[Masahiro Yamada]

2015-05-14 7:28 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
> This is helpful to re-attempt to move failed boards from a previous run
> without starting over.
>
> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>

I like this idea, but I have just submitted a new version.

Would you mind rebasing this patch onto mine, or shall I do it?


-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH v3 06/10] moveconfig: Add a switch to enable printing errors

[Joe Hershberger]

In some cases the build for the autoconf breaks. This outputs the errors
following the status so that action can be taken without building again
manually.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>

---

Changes in v3: None
Changes in v2:
-New for version 2

 tools/moveconfig.py | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 2688ad1..9135e1d 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -241,7 +241,7 @@ class Slot:
     for faster processing.
     """
 
-    def __init__(self, config_attr, devnull, make_cmd):
+    def __init__(self, config_attr, devnull, make_cmd, options):
         """Create a new slot.
 
         Arguments:
@@ -250,6 +250,7 @@ class Slot:
         self.build_dir = tempfile.mkdtemp()
         self.devnull = devnull
         self.make_cmd = (make_cmd, 'O=' + self.build_dir)
+        self.options = options
         self.parser = KconfigParser(self.build_dir, config_attr)
         self.state = STATE_IDLE
 
@@ -307,6 +308,8 @@ class Slot:
             if errout.find('gcc: command not found') != -1:
                 errmsg = 'ERROR - compiler not found (%s)' % self.cross_compile
             self.defconfig_error(errmsg)
+            if self.options.verbose:
+                print errout
             self.state = STATE_IDLE
             return True
 
@@ -346,7 +349,7 @@ class Slots:
 
     """Controller of the array of subprocess slots."""
 
-    def __init__(self, config_attr, jobs):
+    def __init__(self, config_attr, options):
         """Create a new slots controller.
 
         Arguments:
@@ -355,8 +358,8 @@ class Slots:
         self.slots = []
         devnull = get_devnull()
         make_cmd = get_make_cmd()
-        for i in range(jobs):
-            self.slots.append(Slot(config_attr, devnull, make_cmd))
+        for i in range(options.jobs):
+            self.slots.append(Slot(config_attr, devnull, make_cmd, options))
 
     def add(self, defconfig):
         """Add a new subprocess if a vacant slot is available.
@@ -422,7 +425,7 @@ def move_config(config_attr, options):
     if os.path.exists('moveconfig.failed'):
         os.remove('moveconfig.failed')
 
-    slots = Slots(config_attr, options.jobs)
+    slots = Slots(config_attr, options)
 
     # Main loop to process defconfig files:
     #  Add a new subprocess into a vacant slot.
@@ -457,6 +460,9 @@ def main():
                       help='a file containing a list of defconfigs to move')
     parser.add_option('-j', '--jobs', type='int', default=cpu_count,
                       help='the number of jobs to run simultaneously')
+    parser.add_option('-v', '--verbose', dest='verbose',
+                      action='store_true', default=False,
+                      help='show any build errors as boards are build')
     parser.usage += ' config type default no_spl_support'
     (options, args) = parser.parse_args()
 
-- 
1.7.11.5

[U-Boot] [PATCH v3 07/10] moveconfig: Ignore duplicate configs when moving

[Joe Hershberger]

When moving configs, it is important to know what was defined in the
config header even if it duplicates the configs coming from Kconfig.

tools/moveconfig.py now defines KCONFIG_IGNORE_DUPLICATES to prevent the
filtering from happening and selecting wrong values for the defconfig.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>

---

Changes in v3:
-New for version 3

Changes in v2: None

 scripts/Makefile.autoconf | 3 ++-
 tools/moveconfig.py       | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/Makefile.autoconf b/scripts/Makefile.autoconf
index f054081..36bfa17 100644
--- a/scripts/Makefile.autoconf
+++ b/scripts/Makefile.autoconf
@@ -58,7 +58,8 @@ quiet_cmd_autoconf = GEN     $@
 	$(CPP) $(c_flags) $2 -DDO_DEPS_ONLY -dM $(srctree)/include/common.h > $@.tmp && { \
 		sed -n -f $(srctree)/tools/scripts/define2mk.sed $@.tmp |		\
 		while read line; do							\
-			if ! grep -q "$${line%=*}=" include/config/auto.conf; then	\
+			if [ -n "${KCONFIG_IGNORE_DUPLICATES}" ] ||			\
+			   ! grep -q "$${line%=*}=" include/config/auto.conf; then	\
 				echo "$$line";						\
 			fi								\
 		done > $@;								\
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 9135e1d..97ff597 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -340,6 +340,7 @@ class Slot:
         cmd = list(self.make_cmd)
         if self.cross_compile:
             cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
+        cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
         cmd.append('include/autoconf.mk')
         self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE)
         self.state = STATE_SILENTOLDCONFIG
-- 
1.7.11.5

[U-Boot] [PATCH v3 07/10] moveconfig: Ignore duplicate configs when moving

[Masahiro Yamada]

2015-05-14 7:28 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
> When moving configs, it is important to know what was defined in the
> config header even if it duplicates the configs coming from Kconfig.
>
> tools/moveconfig.py now defines KCONFIG_IGNORE_DUPLICATES to prevent the
> filtering from happening and selecting wrong values for the defconfig.
>
> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>
> ---
>
> Changes in v3:
> -New for version 3
>
> Changes in v2: None
>
>  scripts/Makefile.autoconf | 3 ++-
>  tools/moveconfig.py       | 1 +
>  2 files changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/scripts/Makefile.autoconf b/scripts/Makefile.autoconf
> index f054081..36bfa17 100644
> --- a/scripts/Makefile.autoconf
> +++ b/scripts/Makefile.autoconf
> @@ -58,7 +58,8 @@ quiet_cmd_autoconf = GEN     $@
>         $(CPP) $(c_flags) $2 -DDO_DEPS_ONLY -dM $(srctree)/include/common.h > $@.tmp && { \
>                 sed -n -f $(srctree)/tools/scripts/define2mk.sed $@.tmp |               \
>                 while read line; do                                                     \
> -                       if ! grep -q "$${line%=*}=" include/config/auto.conf; then      \
> +                       if [ -n "${KCONFIG_IGNORE_DUPLICATES}" ] ||                     \
> +                          ! grep -q "$${line%=*}=" include/config/auto.conf; then      \
>                                 echo "$$line";                                          \
>                         fi                                                              \
>                 done > $@;                                                              \
> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
> index 9135e1d..97ff597 100755
> --- a/tools/moveconfig.py
> +++ b/tools/moveconfig.py
> @@ -340,6 +340,7 @@ class Slot:
>          cmd = list(self.make_cmd)
>          if self.cross_compile:
>              cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
> +        cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
>          cmd.append('include/autoconf.mk')
>          self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE)
>          self.state = STATE_SILENTOLDCONFIG


This patch is not clear to me.

Isn't it the same concept as u-boot.cfg
instroduced by commit 741e58e0fc8.


-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH v3 07/10] moveconfig: Ignore duplicate configs when moving

[Joe Hershberger]

Hi Masahiro-san,

On Thu, May 14, 2015 at 10:36 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> 2015-05-14 7:28 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
>> When moving configs, it is important to know what was defined in the
>> config header even if it duplicates the configs coming from Kconfig.
>>
>> tools/moveconfig.py now defines KCONFIG_IGNORE_DUPLICATES to prevent the
>> filtering from happening and selecting wrong values for the defconfig.
>>
>> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>>
>> ---
>>
>> Changes in v3:
>> -New for version 3
>>
>> Changes in v2: None
>>
>>  scripts/Makefile.autoconf | 3 ++-
>>  tools/moveconfig.py       | 1 +
>>  2 files changed, 3 insertions(+), 1 deletion(-)
>>
>> diff --git a/scripts/Makefile.autoconf b/scripts/Makefile.autoconf
>> index f054081..36bfa17 100644
>> --- a/scripts/Makefile.autoconf
>> +++ b/scripts/Makefile.autoconf
>> @@ -58,7 +58,8 @@ quiet_cmd_autoconf = GEN     $@
>>         $(CPP) $(c_flags) $2 -DDO_DEPS_ONLY -dM $(srctree)/include/common.h > $@.tmp && { \
>>                 sed -n -f $(srctree)/tools/scripts/define2mk.sed $@.tmp |               \
>>                 while read line; do                                                     \
>> -                       if ! grep -q "$${line%=*}=" include/config/auto.conf; then      \
>> +                       if [ -n "${KCONFIG_IGNORE_DUPLICATES}" ] ||                     \
>> +                          ! grep -q "$${line%=*}=" include/config/auto.conf; then      \
>>                                 echo "$$line";                                          \
>>                         fi                                                              \
>>                 done > $@;                                                              \
>> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
>> index 9135e1d..97ff597 100755
>> --- a/tools/moveconfig.py
>> +++ b/tools/moveconfig.py
>> @@ -340,6 +340,7 @@ class Slot:
>>          cmd = list(self.make_cmd)
>>          if self.cross_compile:
>>              cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
>> +        cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
>>          cmd.append('include/autoconf.mk')
>>          self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE)
>>          self.state = STATE_SILENTOLDCONFIG
>
>
> This patch is not clear to me.

This was specifically a problem when moving a config that defaulted to
true in the Kconfig. That meant that it would be "duplicate removed"
from the include since it was already true (by default) in the
Kconfig. So it had the effect of disabling that config in every board.
This stops the filtering in the case where we are trying to move
configs instead of trying to build the board in this mixed config
system environment.

> Isn't it the same concept as u-boot.cfg
> instroduced by commit 741e58e0fc8.

I don't believe it is the same thing because this will still include
the configs enabled through the config header, but (I think) not those
from Kconfig.  And anyway, we are evaluating autoconf.mk, not
u-boot.cfg.

Cheers,
-Joe

[U-Boot] [PATCH v3 07/10] moveconfig: Ignore duplicate configs when moving

[Masahiro Yamada]

Hi Joe,

2015-05-15 3:02 GMT+09:00 Joe Hershberger <joe.hershberger@gmail.com>:
> Hi Masahiro-san,
>
> On Thu, May 14, 2015 at 10:36 AM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> 2015-05-14 7:28 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
>>> When moving configs, it is important to know what was defined in the
>>> config header even if it duplicates the configs coming from Kconfig.
>>>
>>> tools/moveconfig.py now defines KCONFIG_IGNORE_DUPLICATES to prevent the
>>> filtering from happening and selecting wrong values for the defconfig.
>>>
>>> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>>>
>>> ---
>>>
>>> Changes in v3:
>>> -New for version 3
>>>
>>> Changes in v2: None
>>>
>>>  scripts/Makefile.autoconf | 3 ++-
>>>  tools/moveconfig.py       | 1 +
>>>  2 files changed, 3 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/scripts/Makefile.autoconf b/scripts/Makefile.autoconf
>>> index f054081..36bfa17 100644
>>> --- a/scripts/Makefile.autoconf
>>> +++ b/scripts/Makefile.autoconf
>>> @@ -58,7 +58,8 @@ quiet_cmd_autoconf = GEN     $@
>>>         $(CPP) $(c_flags) $2 -DDO_DEPS_ONLY -dM $(srctree)/include/common.h > $@.tmp && { \
>>>                 sed -n -f $(srctree)/tools/scripts/define2mk.sed $@.tmp |               \
>>>                 while read line; do                                                     \
>>> -                       if ! grep -q "$${line%=*}=" include/config/auto.conf; then      \
>>> +                       if [ -n "${KCONFIG_IGNORE_DUPLICATES}" ] ||                     \
>>> +                          ! grep -q "$${line%=*}=" include/config/auto.conf; then      \
>>>                                 echo "$$line";                                          \
>>>                         fi                                                              \
>>>                 done > $@;                                                              \
>>> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
>>> index 9135e1d..97ff597 100755
>>> --- a/tools/moveconfig.py
>>> +++ b/tools/moveconfig.py
>>> @@ -340,6 +340,7 @@ class Slot:
>>>          cmd = list(self.make_cmd)
>>>          if self.cross_compile:
>>>              cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
>>> +        cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
>>>          cmd.append('include/autoconf.mk')
>>>          self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE)
>>>          self.state = STATE_SILENTOLDCONFIG
>>
>>
>> This patch is not clear to me.
>
> This was specifically a problem when moving a config that defaulted to
> true in the Kconfig. That meant that it would be "duplicate removed"
> from the include since it was already true (by default) in the
> Kconfig. So it had the effect of disabling that config in every board.
> This stops the filtering in the case where we are trying to move
> configs instead of trying to build the board in this mixed config
> system environment.


Uh, make sense.

I was almost forgetting how the "don't apply" version was implemented.

My new one parses .config, defconfig as well as autoconf.mk,
so please check if we still need to this hack.



>> Isn't it the same concept as u-boot.cfg
>> instroduced by commit 741e58e0fc8.
>
> I don't believe it is the same thing because this will still include
> the configs enabled through the config header, but (I think) not those
> from Kconfig.  And anyway, we are evaluating autoconf.mk, not
> u-boot.cfg.

Now, I understood your intention.

u-boot.cfg is not useful for this tool,
although u-boot.cfg also includes configs from Kconfig.


-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH v3 07/10] moveconfig: Ignore duplicate configs when moving

[Joe Hershberger]

Hi Masahiro-san,

On Fri, May 15, 2015 at 12:10 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Hi Joe,
>
> 2015-05-15 3:02 GMT+09:00 Joe Hershberger <joe.hershberger@gmail.com>:
>> Hi Masahiro-san,
>>
>> On Thu, May 14, 2015 at 10:36 AM, Masahiro Yamada
>> <yamada.masahiro@socionext.com> wrote:
>>> 2015-05-14 7:28 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
>>>> When moving configs, it is important to know what was defined in the
>>>> config header even if it duplicates the configs coming from Kconfig.
>>>>
>>>> tools/moveconfig.py now defines KCONFIG_IGNORE_DUPLICATES to prevent the
>>>> filtering from happening and selecting wrong values for the defconfig.
>>>>
>>>> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>>>>
>>>> ---
>>>>
>>>> Changes in v3:
>>>> -New for version 3
>>>>
>>>> Changes in v2: None
>>>>
>>>>  scripts/Makefile.autoconf | 3 ++-
>>>>  tools/moveconfig.py       | 1 +
>>>>  2 files changed, 3 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/scripts/Makefile.autoconf b/scripts/Makefile.autoconf
>>>> index f054081..36bfa17 100644
>>>> --- a/scripts/Makefile.autoconf
>>>> +++ b/scripts/Makefile.autoconf
>>>> @@ -58,7 +58,8 @@ quiet_cmd_autoconf = GEN     $@
>>>>         $(CPP) $(c_flags) $2 -DDO_DEPS_ONLY -dM $(srctree)/include/common.h > $@.tmp && { \
>>>>                 sed -n -f $(srctree)/tools/scripts/define2mk.sed $@.tmp |               \
>>>>                 while read line; do                                                     \
>>>> -                       if ! grep -q "$${line%=*}=" include/config/auto.conf; then      \
>>>> +                       if [ -n "${KCONFIG_IGNORE_DUPLICATES}" ] ||                     \
>>>> +                          ! grep -q "$${line%=*}=" include/config/auto.conf; then      \
>>>>                                 echo "$$line";                                          \
>>>>                         fi                                                              \
>>>>                 done > $@;                                                              \
>>>> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
>>>> index 9135e1d..97ff597 100755
>>>> --- a/tools/moveconfig.py
>>>> +++ b/tools/moveconfig.py
>>>> @@ -340,6 +340,7 @@ class Slot:
>>>>          cmd = list(self.make_cmd)
>>>>          if self.cross_compile:
>>>>              cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
>>>> +        cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
>>>>          cmd.append('include/autoconf.mk')
>>>>          self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE)
>>>>          self.state = STATE_SILENTOLDCONFIG
>>>
>>>
>>> This patch is not clear to me.
>>
>> This was specifically a problem when moving a config that defaulted to
>> true in the Kconfig. That meant that it would be "duplicate removed"
>> from the include since it was already true (by default) in the
>> Kconfig. So it had the effect of disabling that config in every board.
>> This stops the filtering in the case where we are trying to move
>> configs instead of trying to build the board in this mixed config
>> system environment.
>
>
> Uh, make sense.
>
> I was almost forgetting how the "don't apply" version was implemented.
>
> My new one parses .config, defconfig as well as autoconf.mk,
> so please check if we still need to this hack.

I checked it out. The same problem exists and the patch is still needed.

Also, I don't believe it is valid to check the .config for things to
skip. Just because there is a mention of something does not mean it is
the correct value. This is specifically the case for using
savedefconfig. For savedefconfig to work properly, the user must add
the entry to the Kconfig menu first, then run moveconfig.py. You seem
to be expecting that users run moveconfig.py before editing the
Kconfigs.

8<snip>8

Cheers,
-Joe

[U-Boot] [PATCH v3 08/10] moveconfig: Handle moving multiple configs at once

[Joe Hershberger]

Moving configs is a fairly slow process since each board config must
pass through a compiler to evaluate the configs. Also, many configs
are related in a way that makes sense to atomically move.

Add support to tools/moveconfig.py to read multiple lines from the
.moveconfig file and create a parser for each. After compiling the
configs, simply run all the parsers on the autoconf.mk to find the
state of each of those configs.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>

---

Changes in v3:
-New for version 3

Changes in v2: None

 tools/moveconfig.py | 75 ++++++++++++++++++++++++++++++-----------------------
 1 file changed, 42 insertions(+), 33 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 97ff597..798f717 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -91,9 +91,9 @@ def cleanup_header(header_path, patterns):
             if not i in matched:
                 f.write(line)
 
-def cleanup_headers(config):
+def cleanup_headers(config_attrs):
     while True:
-        choice = raw_input('Clean up %s in headers? [y/n]: ' % config).lower()
+        choice = raw_input('Clean up headers? [y/n]: ').lower()
         print choice
         if choice == 'y' or choice == 'n':
             break
@@ -102,8 +102,9 @@ def cleanup_headers(config):
         return
 
     patterns = []
-    patterns.append(re.compile(r'#\s*define\s+%s\W' % config))
-    patterns.append(re.compile(r'#\s*undef\s+%s\W' % config))
+    for config_attr in config_attrs:
+        patterns.append(re.compile(r'#\s*define\s+%s\W' % config_attr['config']))
+        patterns.append(re.compile(r'#\s*undef\s+%s\W' % config_attr['config']))
 
     for (dirpath, dirnames, filenames) in os.walk('include'):
         for filename in filenames:
@@ -126,7 +127,7 @@ class KconfigParser:
     PREFIX = { '.': '+', 'spl': 'S', 'tpl': 'T' }
 
     def __init__(self, build_dir, config_attr):
-        """Create a new .config perser.
+        """Create a new .config parser.
 
         Arguments:
           build_dir: Build directory where .config is located
@@ -141,6 +142,7 @@ class KconfigParser:
         else:
             self.no_spl_support = False
         self.re = re.compile(r'%s=(.*)' % self.config)
+        self.dotconfig = os.path.join(self.build_dir, '.config')
 
     def get_cross_compile(self):
         """Parse .config file and return CROSS_COMPILE
@@ -150,7 +152,6 @@ class KconfigParser:
         """
         arch = ''
         cpu = ''
-        self.dotconfig = os.path.join(self.build_dir, '.config')
         for line in open(self.dotconfig):
             m = self.re_arch.match(line)
             if m:
@@ -204,7 +205,6 @@ class KconfigParser:
                 value = '(default)'
 
             values.append(value)
-            os.remove(autoconf)
 
             if value == '(undef)' or value == '(default)':
                 continue
@@ -220,6 +220,7 @@ class KconfigParser:
                 prefixes[output_line] = self.PREFIX[img]
 
         output = defconfig[:-len('_defconfig')].ljust(37) + ': '
+        output += self.config.replace('CONFIG_','').ljust(18) + ': '
         for value in values:
             output += value.ljust(12)
 
@@ -241,7 +242,7 @@ class Slot:
     for faster processing.
     """
 
-    def __init__(self, config_attr, devnull, make_cmd, options):
+    def __init__(self, config_attrs, devnull, make_cmd, options):
         """Create a new slot.
 
         Arguments:
@@ -251,7 +252,9 @@ class Slot:
         self.devnull = devnull
         self.make_cmd = (make_cmd, 'O=' + self.build_dir)
         self.options = options
-        self.parser = KconfigParser(self.build_dir, config_attr)
+        self.parsers = []
+        for config_attr in config_attrs:
+            self.parsers.append(KconfigParser(self.build_dir, config_attr))
         self.state = STATE_IDLE
 
     def __del__(self):
@@ -314,10 +317,11 @@ class Slot:
             return True
 
         if self.state == STATE_SILENTOLDCONFIG:
-            if self.parser.update_defconfig(self.defconfig):
-                self.defconfig_error('ERROR - autoconf.mk not found')
-                self.state = STATE_IDLE
-                return True
+            for parser in self.parsers:
+                if parser.update_defconfig(self.defconfig):
+                    self.defconfig_error('ERROR - autoconf.mk not found')
+                    self.state = STATE_IDLE
+                    return True
 
             """Save off the defconfig in a consistent way"""
             cmd = list(self.make_cmd)
@@ -336,7 +340,7 @@ class Slot:
             self.state = STATE_IDLE
             return True
 
-        self.cross_compile = self.parser.get_cross_compile()
+        self.cross_compile = self.parsers[0].get_cross_compile()
         cmd = list(self.make_cmd)
         if self.cross_compile:
             cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
@@ -350,7 +354,7 @@ class Slots:
 
     """Controller of the array of subprocess slots."""
 
-    def __init__(self, config_attr, options):
+    def __init__(self, config_attrs, options):
         """Create a new slots controller.
 
         Arguments:
@@ -360,7 +364,7 @@ class Slots:
         devnull = get_devnull()
         make_cmd = get_make_cmd()
         for i in range(options.jobs):
-            self.slots.append(Slot(config_attr, devnull, make_cmd, options))
+            self.slots.append(Slot(config_attrs, devnull, make_cmd, options))
 
     def add(self, defconfig):
         """Add a new subprocess if a vacant slot is available.
@@ -400,15 +404,16 @@ class Slots:
                 ret = False
         return ret
 
-def move_config(config_attr, options):
+def move_config(config_attrs, options):
     check_top_directory()
 
-    print 'Moving %s (type: %s, default: %s, no_spl: %s) ...  (jobs: %d)' % (
-        config_attr['config'],
-        config_attr['type'],
-        config_attr['default'],
-        config_attr['no_spl_support'],
-        options.jobs)
+    for config_attr in config_attrs:
+        print 'Moving %s (type: %s, default: %s, no_spl: %s)' % (
+            config_attr['config'],
+            config_attr['type'],
+            config_attr['default'],
+            config_attr['no_spl_support'])
+    print '%d jobs...' % options.jobs
 
     if options.defconfigs:
         defconfigs = [line.strip() for line in open(options.defconfigs, 'r')]
@@ -426,7 +431,7 @@ def move_config(config_attr, options):
     if os.path.exists('moveconfig.failed'):
         os.remove('moveconfig.failed')
 
-    slots = Slots(config_attr, options)
+    slots = Slots(config_attrs, options)
 
     # Main loop to process defconfig files:
     #  Add a new subprocess into a vacant slot.
@@ -441,7 +446,7 @@ def move_config(config_attr, options):
     while not slots.empty():
         time.sleep(SLEEP_TIME)
 
-    cleanup_headers(config_attr['config'])
+    cleanup_headers(config_attrs)
 
     if os.path.exists('moveconfig.failed'):
         print '!!!  Some boards were not processed; move the config manually.'
@@ -468,31 +473,35 @@ def main():
     (options, args) = parser.parse_args()
 
     args_key = ('config', 'type', 'default', 'no_spl_support')
-    config_attr = {}
+    config_attrs = []
 
     if len(args) >= len(args_key):
         saved_attr = ''
         for i, key in enumerate(args_key):
-            config_attr[key] = args[i]
+            config_attrs.append({})
+            config_attrs[0][key] = args[i]
             saved_attr = saved_attr + ' %s' % args[i]
         with open('.moveconfig', 'w') as f:
             f.write(saved_attr)
     elif os.path.exists('.moveconfig'):
         f = open('.moveconfig')
         try:
-            saved_attr = f.readline().split()
-            for i, key in enumerate(args_key):
-                config_attr[key] = saved_attr[i]
+            for j, line in enumerate(f):
+                config_attrs.append({})
+                saved_attr = line.split()
+                for i, key in enumerate(args_key):
+                    config_attrs[j][key] = saved_attr[i]
         except:
             sys.exit('%s: broken format' % '.moveconfig')
     else:
         parser.print_usage()
         sys.exit(1)
 
-    if not config_attr['config'].startswith('CONFIG_'):
-        config_attr['config'] = 'CONFIG_' + config_attr['config']
+    for config_attr in config_attrs:
+        if not config_attr['config'].startswith('CONFIG_'):
+            config_attr['config'] = 'CONFIG_' + config_attr['config']
 
-    move_config(config_attr, options)
+    move_config(config_attrs, options)
 
 if __name__ == '__main__':
     main()
-- 
1.7.11.5

[U-Boot] [PATCH v3 08/10] moveconfig: Handle moving multiple configs at once

[Masahiro Yamada]

Hi Joe,

2015-05-14 7:28 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
> Moving configs is a fairly slow process since each board config must
> pass through a compiler to evaluate the configs. Also, many configs
> are related in a way that makes sense to atomically move.
>
> Add support to tools/moveconfig.py to read multiple lines from the
> .moveconfig file and create a parser for each. After compiling the
> configs, simply run all the parsers on the autoconf.mk to find the
> state of each of those configs.
>
> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>

Sorry for leaving this scripts for such a long time.
(I have been busy in the Linux side lately.)
causing two people to develope a similar feature.

I thought the same thing.
My idea was to stop giving config attributes from the command arguments.
Instead, the name of the input file is given as the argument.


-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH v3 08/10] moveconfig: Handle moving multiple configs at once

[Joe Hershberger]

Hi Masahiro-san,

On Thu, May 14, 2015 at 9:37 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Hi Joe,
>
> 2015-05-14 7:28 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
>> Moving configs is a fairly slow process since each board config must
>> pass through a compiler to evaluate the configs. Also, many configs
>> are related in a way that makes sense to atomically move.
>>
>> Add support to tools/moveconfig.py to read multiple lines from the
>> .moveconfig file and create a parser for each. After compiling the
>> configs, simply run all the parsers on the autoconf.mk to find the
>> state of each of those configs.
>>
>> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>>
>
> Sorry for leaving this scripts for such a long time.
> (I have been busy in the Linux side lately.)
> causing two people to develope a similar feature.

Not too big a deal. Hopefully this will be in the tree soon so we can
stop managing this in every branch.

> I thought the same thing.
> My idea was to stop giving config attributes from the command arguments.
> Instead, the name of the input file is given as the argument.

That's fine. I was just trying to maintain your original features.

Cheers,
-Joe

[U-Boot] [PATCH v3 09/10] moveconfig: Print status about the processed defconfigs

[Joe Hershberger]

This gives a basic idea about progress.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>

---

Changes in v3:
-New for version 3

Changes in v2: None

 tools/moveconfig.py | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 798f717..bb087d4 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -93,7 +93,7 @@ def cleanup_header(header_path, patterns):
 
 def cleanup_headers(config_attrs):
     while True:
-        choice = raw_input('Clean up headers? [y/n]: ').lower()
+        choice = raw_input('\nClean up headers? [y/n]: ').lower()
         print choice
         if choice == 'y' or choice == 'n':
             break
@@ -264,7 +264,7 @@ class Slot:
                 pass
         shutil.rmtree(self.build_dir)
 
-    def add(self, defconfig):
+    def add(self, defconfig, num, total):
         """Add a new subprocess to the slot.
 
         Fails if the slot is occupied, that is, the current subprocess
@@ -283,6 +283,8 @@ class Slot:
         self.ps = subprocess.Popen(cmd, stdout=self.devnull)
         self.defconfig = defconfig
         self.state = STATE_DEFCONFIG
+        self.num = num
+        self.total = total
         return True
 
     def defconfig_error(self, errmsg):
@@ -323,6 +325,9 @@ class Slot:
                     self.state = STATE_IDLE
                     return True
 
+            print ' %d defconfigs out of %d\r' % (self.num + 1, self.total),
+            sys.stdout.flush()
+
             """Save off the defconfig in a consistent way"""
             cmd = list(self.make_cmd)
             cmd.append('savedefconfig')
@@ -366,7 +371,7 @@ class Slots:
         for i in range(options.jobs):
             self.slots.append(Slot(config_attrs, devnull, make_cmd, options))
 
-    def add(self, defconfig):
+    def add(self, defconfig, num, total):
         """Add a new subprocess if a vacant slot is available.
 
         Arguments:
@@ -376,7 +381,7 @@ class Slots:
           Return True on success or False on fail
         """
         for slot in self.slots:
-            if slot.add(defconfig):
+            if slot.add(defconfig, num, total):
                 return True
         return False
 
@@ -436,8 +441,8 @@ def move_config(config_attrs, options):
     # Main loop to process defconfig files:
     #  Add a new subprocess into a vacant slot.
     #  Sleep if there is no available slot.
-    for defconfig in defconfigs:
-        while not slots.add(defconfig):
+    for i, defconfig in enumerate(defconfigs):
+        while not slots.add(defconfig, i, len(defconfigs)):
             while not slots.available():
                 # No available slot: sleep for a while
                 time.sleep(SLEEP_TIME)
-- 
1.7.11.5

[U-Boot] [PATCH v3 10/10] moveconfig: Add a switch to only cleanup headers

[Joe Hershberger]

In some case you may want to only cleanup the headers. Make it possible
without waiting for all boards to compile.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>

---

Changes in v3:
-New for version 3

Changes in v2: None

 tools/moveconfig.py | 83 ++++++++++++++++++++++++++++-------------------------
 1 file changed, 44 insertions(+), 39 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index bb087d4..d63f47f 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -412,48 +412,50 @@ class Slots:
 def move_config(config_attrs, options):
     check_top_directory()
 
-    for config_attr in config_attrs:
-        print 'Moving %s (type: %s, default: %s, no_spl: %s)' % (
-            config_attr['config'],
-            config_attr['type'],
-            config_attr['default'],
-            config_attr['no_spl_support'])
-    print '%d jobs...' % options.jobs
-
-    if options.defconfigs:
-        defconfigs = [line.strip() for line in open(options.defconfigs, 'r')]
-    else:
-        # All the defconfig files to be processed
-        defconfigs = []
-        for (dirpath, dirnames, filenames) in os.walk('configs'):
-            dirpath = dirpath[len('configs') + 1:]
-            for filename in fnmatch.filter(filenames, '*_defconfig'):
-                if fnmatch.fnmatch(filename, '.*'):
-                    continue
-                defconfigs.append(os.path.join(dirpath, filename))
-
-    """Clean up any previous log of failed moves"""
-    if os.path.exists('moveconfig.failed'):
-        os.remove('moveconfig.failed')
-
-    slots = Slots(config_attrs, options)
-
-    # Main loop to process defconfig files:
-    #  Add a new subprocess into a vacant slot.
-    #  Sleep if there is no available slot.
-    for i, defconfig in enumerate(defconfigs):
-        while not slots.add(defconfig, i, len(defconfigs)):
-            while not slots.available():
-                # No available slot: sleep for a while
-                time.sleep(SLEEP_TIME)
-
-    # wait until all the subprocesses finish
-    while not slots.empty():
-        time.sleep(SLEEP_TIME)
+    if not options.clean_only:
+        for config_attr in config_attrs:
+            print 'Moving %s (type: %s, default: %s, no_spl: %s)' % (
+                config_attr['config'],
+                config_attr['type'],
+                config_attr['default'],
+                config_attr['no_spl_support'])
+        print '%d jobs...' % options.jobs
+
+        if options.defconfigs:
+            defconfigs = [line.strip() for line in
+                open(options.defconfigs, 'r')]
+        else:
+            # All the defconfig files to be processed
+            defconfigs = []
+            for (dirpath, dirnames, filenames) in os.walk('configs'):
+                dirpath = dirpath[len('configs') + 1:]
+                for filename in fnmatch.filter(filenames, '*_defconfig'):
+                    if fnmatch.fnmatch(filename, '.*'):
+                        continue
+                    defconfigs.append(os.path.join(dirpath, filename))
+
+        """Clean up any previous log of failed moves"""
+        if os.path.exists('moveconfig.failed'):
+            os.remove('moveconfig.failed')
+
+        slots = Slots(config_attrs, options)
+
+        # Main loop to process defconfig files:
+        #  Add a new subprocess into a vacant slot.
+        #  Sleep if there is no available slot.
+        for i, defconfig in enumerate(defconfigs):
+            while not slots.add(defconfig, i, len(defconfigs)):
+                while not slots.available():
+                    # No available slot: sleep for a while
+                    time.sleep(SLEEP_TIME)
+
+        # wait until all the subprocesses finish
+        while not slots.empty():
+            time.sleep(SLEEP_TIME)
 
     cleanup_headers(config_attrs)
 
-    if os.path.exists('moveconfig.failed'):
+    if (not options.clean_only) & os.path.exists('moveconfig.failed'):
         print '!!!  Some boards were not processed; move the config manually.'
         print '!!!  The list of failed boards are saved in moveconfig.failed'
         print
@@ -467,6 +469,9 @@ def main():
 
     parser = optparse.OptionParser()
     # Add options here
+    parser.add_option('-c', '--clean-only', dest='clean_only',
+                      action='store_true', default=False,
+                      help='only clean the headers')
     parser.add_option('-d', '--defconfigs', type='string',
                       help='a file containing a list of defconfigs to move')
     parser.add_option('-j', '--jobs', type='int', default=cpu_count,
-- 
1.7.11.5

[U-Boot] [PATCH v3 10/10] moveconfig: Add a switch to only cleanup headers

[Masahiro Yamada]

2015-05-14 7:28 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
> In some case you may want to only cleanup the headers. Make it possible
> without waiting for all boards to compile.
>
> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>
> ---
>
> Changes in v3:
> -New for version 3
>
> Changes in v2: None
>
>  tools/moveconfig.py | 83 ++++++++++++++++++++++++++++-------------------------
>  1 file changed, 44 insertions(+), 39 deletions(-)
>
> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
> index bb087d4..d63f47f 100755
> --- a/tools/moveconfig.py
> +++ b/tools/moveconfig.py
> @@ -412,48 +412,50 @@ class Slots:
>  def move_config(config_attrs, options):
>      check_top_directory()
>
> -    for config_attr in config_attrs:
> -        print 'Moving %s (type: %s, default: %s, no_spl: %s)' % (
> -            config_attr['config'],
> -            config_attr['type'],
> -            config_attr['default'],
> -            config_attr['no_spl_support'])
> -    print '%d jobs...' % options.jobs
> -
> -    if options.defconfigs:
> -        defconfigs = [line.strip() for line in open(options.defconfigs, 'r')]
> -    else:
> -        # All the defconfig files to be processed
> -        defconfigs = []
> -        for (dirpath, dirnames, filenames) in os.walk('configs'):
> -            dirpath = dirpath[len('configs') + 1:]
> -            for filename in fnmatch.filter(filenames, '*_defconfig'):
> -                if fnmatch.fnmatch(filename, '.*'):
> -                    continue
> -                defconfigs.append(os.path.join(dirpath, filename))
> -
> -    """Clean up any previous log of failed moves"""
> -    if os.path.exists('moveconfig.failed'):
> -        os.remove('moveconfig.failed')
> -
> -    slots = Slots(config_attrs, options)
> -
> -    # Main loop to process defconfig files:
> -    #  Add a new subprocess into a vacant slot.
> -    #  Sleep if there is no available slot.
> -    for i, defconfig in enumerate(defconfigs):
> -        while not slots.add(defconfig, i, len(defconfigs)):
> -            while not slots.available():
> -                # No available slot: sleep for a while
> -                time.sleep(SLEEP_TIME)
> -
> -    # wait until all the subprocesses finish
> -    while not slots.empty():
> -        time.sleep(SLEEP_TIME)
> +    if not options.clean_only:
> +        for config_attr in config_attrs:
> +            print 'Moving %s (type: %s, default: %s, no_spl: %s)' % (
> +                config_attr['config'],
> +                config_attr['type'],
> +                config_attr['default'],
> +                config_attr['no_spl_support'])
> +        print '%d jobs...' % options.jobs
> +
> +        if options.defconfigs:
> +            defconfigs = [line.strip() for line in
> +                open(options.defconfigs, 'r')]
> +        else:
> +            # All the defconfig files to be processed
> +            defconfigs = []
> +            for (dirpath, dirnames, filenames) in os.walk('configs'):
> +                dirpath = dirpath[len('configs') + 1:]
> +                for filename in fnmatch.filter(filenames, '*_defconfig'):
> +                    if fnmatch.fnmatch(filename, '.*'):
> +                        continue
> +                    defconfigs.append(os.path.join(dirpath, filename))
> +
> +        """Clean up any previous log of failed moves"""
> +        if os.path.exists('moveconfig.failed'):
> +            os.remove('moveconfig.failed')
> +
> +        slots = Slots(config_attrs, options)
> +
> +        # Main loop to process defconfig files:
> +        #  Add a new subprocess into a vacant slot.
> +        #  Sleep if there is no available slot.
> +        for i, defconfig in enumerate(defconfigs):
> +            while not slots.add(defconfig, i, len(defconfigs)):
> +                while not slots.available():
> +                    # No available slot: sleep for a while
> +                    time.sleep(SLEEP_TIME)
> +
> +        # wait until all the subprocesses finish
> +        while not slots.empty():
> +            time.sleep(SLEEP_TIME)
>
>      cleanup_headers(config_attrs)
>
> -    if os.path.exists('moveconfig.failed'):
> +    if (not options.clean_only) & os.path.exists('moveconfig.failed'):
>          print '!!!  Some boards were not processed; move the config manually.'
>          print '!!!  The list of failed boards are saved in moveconfig.failed'
>          print


I am OK with this feature, but this 'if' statement ranges over many code lines.

Perhaps, moving the cleanup_headers to the main function is simpler?


def main():
    [snip]

    if not options.clean_only:
          move_config(config_attrs, options)

    cleanup_headers(config_attrs)





-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH v3 10/10] moveconfig: Add a switch to only cleanup headers

[Joe Hershberger]

Hi Masahiro-san,

On Thu, May 14, 2015 at 9:51 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> 2015-05-14 7:28 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
>> In some case you may want to only cleanup the headers. Make it possible
>> without waiting for all boards to compile.
>>
>> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>>
>> ---
>>
>> Changes in v3:
>> -New for version 3
>>
>> Changes in v2: None
>>
>>  tools/moveconfig.py | 83 ++++++++++++++++++++++++++++-------------------------
>>  1 file changed, 44 insertions(+), 39 deletions(-)
>>
>> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
>> index bb087d4..d63f47f 100755
>> --- a/tools/moveconfig.py
>> +++ b/tools/moveconfig.py
>> @@ -412,48 +412,50 @@ class Slots:
>>  def move_config(config_attrs, options):
>>      check_top_directory()
>>
>> -    for config_attr in config_attrs:
>> -        print 'Moving %s (type: %s, default: %s, no_spl: %s)' % (
>> -            config_attr['config'],
>> -            config_attr['type'],
>> -            config_attr['default'],
>> -            config_attr['no_spl_support'])
>> -    print '%d jobs...' % options.jobs
>> -
>> -    if options.defconfigs:
>> -        defconfigs = [line.strip() for line in open(options.defconfigs, 'r')]
>> -    else:
>> -        # All the defconfig files to be processed
>> -        defconfigs = []
>> -        for (dirpath, dirnames, filenames) in os.walk('configs'):
>> -            dirpath = dirpath[len('configs') + 1:]
>> -            for filename in fnmatch.filter(filenames, '*_defconfig'):
>> -                if fnmatch.fnmatch(filename, '.*'):
>> -                    continue
>> -                defconfigs.append(os.path.join(dirpath, filename))
>> -
>> -    """Clean up any previous log of failed moves"""
>> -    if os.path.exists('moveconfig.failed'):
>> -        os.remove('moveconfig.failed')
>> -
>> -    slots = Slots(config_attrs, options)
>> -
>> -    # Main loop to process defconfig files:
>> -    #  Add a new subprocess into a vacant slot.
>> -    #  Sleep if there is no available slot.
>> -    for i, defconfig in enumerate(defconfigs):
>> -        while not slots.add(defconfig, i, len(defconfigs)):
>> -            while not slots.available():
>> -                # No available slot: sleep for a while
>> -                time.sleep(SLEEP_TIME)
>> -
>> -    # wait until all the subprocesses finish
>> -    while not slots.empty():
>> -        time.sleep(SLEEP_TIME)
>> +    if not options.clean_only:
>> +        for config_attr in config_attrs:
>> +            print 'Moving %s (type: %s, default: %s, no_spl: %s)' % (
>> +                config_attr['config'],
>> +                config_attr['type'],
>> +                config_attr['default'],
>> +                config_attr['no_spl_support'])
>> +        print '%d jobs...' % options.jobs
>> +
>> +        if options.defconfigs:
>> +            defconfigs = [line.strip() for line in
>> +                open(options.defconfigs, 'r')]
>> +        else:
>> +            # All the defconfig files to be processed
>> +            defconfigs = []
>> +            for (dirpath, dirnames, filenames) in os.walk('configs'):
>> +                dirpath = dirpath[len('configs') + 1:]
>> +                for filename in fnmatch.filter(filenames, '*_defconfig'):
>> +                    if fnmatch.fnmatch(filename, '.*'):
>> +                        continue
>> +                    defconfigs.append(os.path.join(dirpath, filename))
>> +
>> +        """Clean up any previous log of failed moves"""
>> +        if os.path.exists('moveconfig.failed'):
>> +            os.remove('moveconfig.failed')
>> +
>> +        slots = Slots(config_attrs, options)
>> +
>> +        # Main loop to process defconfig files:
>> +        #  Add a new subprocess into a vacant slot.
>> +        #  Sleep if there is no available slot.
>> +        for i, defconfig in enumerate(defconfigs):
>> +            while not slots.add(defconfig, i, len(defconfigs)):
>> +                while not slots.available():
>> +                    # No available slot: sleep for a while
>> +                    time.sleep(SLEEP_TIME)
>> +
>> +        # wait until all the subprocesses finish
>> +        while not slots.empty():
>> +            time.sleep(SLEEP_TIME)
>>
>>      cleanup_headers(config_attrs)
>>
>> -    if os.path.exists('moveconfig.failed'):
>> +    if (not options.clean_only) & os.path.exists('moveconfig.failed'):
>>          print '!!!  Some boards were not processed; move the config manually.'
>>          print '!!!  The list of failed boards are saved in moveconfig.failed'
>>          print
>
>
> I am OK with this feature, but this 'if' statement ranges over many code lines.
>
> Perhaps, moving the cleanup_headers to the main function is simpler?
>
>
> def main():
>     [snip]
>
>     if not options.clean_only:
>           move_config(config_attrs, options)
>
>     cleanup_headers(config_attrs)

OK.

-Joe

[U-Boot] [PATCH v3 01/10] moveconfig: Actually build autoconf.mk

[Masahiro Yamada]

2015-05-14 7:28 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
> The existing target won't actually build the needed .mk file that is
> expected by the rest of the script. It seems that silentoldconfig does
> not actually cause this to be built any longer.
>
> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
> ---
> This patch is based on:
>  "Don't apply: tools: add a tool to move automatically CONFIGs from headers to defconfigs"
> from the list.
>
> Also dropped the patch that adds an arc cross-tool mapping.
>
> Changes in v3: None
> Changes in v2: None
>
>  tools/moveconfig.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
> index cb7f7ea..30dc4f6 100755
> --- a/tools/moveconfig.py
> +++ b/tools/moveconfig.py
> @@ -301,7 +301,7 @@ class Slot:
>          cmd = list(self.make_cmd)
>          if cross_compile:
>              cmd.append('CROSS_COMPILE=%s' % cross_compile)
> -        cmd.append('silentoldconfig')
> +        cmd.append('include/autoconf.mk')
>          self.ps = subprocess.Popen(cmd, stdout=self.devnull)
>          self.state = STATE_SILENTOLDCONFIG
>          return False

Yup, the 'Don't apply' version is not working since I removed
scripts/multiconfig.sh.

I had locally modified this tool though I tweaked a bit differently.


-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH v4 01/10] moveconfig: Always run savedefconfig on the moved config

[Joe Hershberger]

This will ensure that the order of the defconfig entries will always
match that of the Kconfig files. After one slightly painful (but
still early in the process) pass over all boards, this should keep
the defconfigs clean from here on.

Users must edit the Kconfig first to add the menu entries and then run
moveconfig.py to update the defconfig files and the include configs.

As such, moveconfig.py cannot compare against the '.config' contents.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
---
This is based on https://patchwork.ozlabs.org/patch/472591/

Changes in v4:
-Rebased series on Masahiro's v2

Changes in v3: None
Changes in v2: None

 tools/moveconfig.py | 35 ++++++++++++++++++++++++++---------
 1 file changed, 26 insertions(+), 9 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index c39ea95..544f6af 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -46,6 +46,9 @@ should look like this:
   CONFIG_CMD_USB bool n
   CONFIG_SYS_TEXT_BASE hex 0x00000000
 
+Next you must edit the Kconfig to add the menu entries for the configs
+you are moving.
+
 And then run this tool giving the file name of the recipe
 
   $ tools/moveconfig.py recipe
@@ -192,6 +195,7 @@ CROSS_COMPILE = {
 STATE_IDLE = 0
 STATE_DEFCONFIG = 1
 STATE_AUTOCONF = 2
+STATE_SAVEDEFCONFIG = 3
 
 ACTION_MOVE = 0
 ACTION_DEFAULT_VALUE = 1
@@ -390,8 +394,7 @@ class KconfigParser:
 
         return CROSS_COMPILE.get(arch, '')
 
-    def parse_one_config(self, config_attr, defconfig_lines,
-                         dotconfig_lines, autoconf_lines):
+    def parse_one_config(self, config_attr, defconfig_lines, autoconf_lines):
         """Parse .config, defconfig, include/autoconf.mk for one config.
 
         This function looks for the config options in the lines from
@@ -402,7 +405,6 @@ class KconfigParser:
           config_attr: A dictionary including the name, the type,
                        and the default value of the target config.
           defconfig_lines: lines from the original defconfig file.
-          dotconfig_lines: lines from the .config file.
           autoconf_lines: lines from the include/autoconf.mk file.
 
         Returns:
@@ -418,7 +420,7 @@ class KconfigParser:
         else:
             default = config + '=' + config_attr['default']
 
-        for line in defconfig_lines + dotconfig_lines:
+        for line in defconfig_lines:
             line = line.rstrip()
             if line.startswith(config + '=') or line == not_set:
                 return (ACTION_ALREADY_EXIST, line)
@@ -463,15 +465,12 @@ class KconfigParser:
         with open(defconfig_path) as f:
             defconfig_lines = f.readlines()
 
-        with open(dotconfig_path) as f:
-            dotconfig_lines = f.readlines()
-
         with open(autoconf_path) as f:
             autoconf_lines = f.readlines()
 
         for config_attr in self.config_attrs:
             result = self.parse_one_config(config_attr, defconfig_lines,
-                                           dotconfig_lines, autoconf_lines)
+                                           autoconf_lines)
             results.append(result)
 
         log = ''
@@ -499,7 +498,7 @@ class KconfigParser:
         print log,
 
         if not self.options.dry_run:
-            with open(defconfig_path, 'a') as f:
+            with open(dotconfig_path, 'a') as f:
                 for (action, value) in results:
                     if action == ACTION_MOVE:
                         f.write(value + '\n')
@@ -608,6 +607,24 @@ class Slot:
 
         if self.state == STATE_AUTOCONF:
             self.parser.update_defconfig(self.defconfig)
+
+            """Save off the defconfig in a consistent way"""
+            cmd = list(self.make_cmd)
+            cmd.append('savedefconfig')
+            self.ps = subprocess.Popen(cmd, stdout=self.devnull,
+                                       stderr=self.devnull)
+            self.state = STATE_SAVEDEFCONFIG
+            return False
+
+        if self.state == STATE_SAVEDEFCONFIG:
+            defconfig_path = os.path.join(self.build_dir, 'defconfig')
+            if not os.path.exists(defconfig_path):
+                print >> sys.stderr, log_msg(self.options.color,
+                                             COLOR_LIGHT_RED,
+                                             self.defconfig,
+                                             'The defconfig was not updated')
+            shutil.move(defconfig_path,
+                        os.path.join('configs', self.defconfig))
             self.state = STATE_IDLE
             return True
 
-- 
1.7.11.5

[U-Boot] [PATCH v4 02/10] moveconfig: Ignore duplicate configs when moving

[Joe Hershberger]

When moving configs, it is important to know what was defined in the
config header even if it duplicates the configs coming from Kconfig.

This is specifically needed for the case where a config is set to
default 'y' in the Kconfig. This would previously cause the actual value
from the include config to be filtered out, and moveconfig.py would
think that it was 'n'... This means that the value that should be 'y'
is now (in every defconfig) set to 'not set'.

tools/moveconfig.py now defines KCONFIG_IGNORE_DUPLICATES to prevent the
filtering from happening and selecting wrong values for the defconfig.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>

---

Changes in v4: None
Changes in v3:
-New for version 3

Changes in v2: None

 scripts/Makefile.autoconf | 3 ++-
 tools/moveconfig.py       | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/Makefile.autoconf b/scripts/Makefile.autoconf
index f054081..36bfa17 100644
--- a/scripts/Makefile.autoconf
+++ b/scripts/Makefile.autoconf
@@ -58,7 +58,8 @@ quiet_cmd_autoconf = GEN     $@
 	$(CPP) $(c_flags) $2 -DDO_DEPS_ONLY -dM $(srctree)/include/common.h > $@.tmp && { \
 		sed -n -f $(srctree)/tools/scripts/define2mk.sed $@.tmp |		\
 		while read line; do							\
-			if ! grep -q "$${line%=*}=" include/config/auto.conf; then	\
+			if [ -n "${KCONFIG_IGNORE_DUPLICATES}" ] ||			\
+			   ! grep -q "$${line%=*}=" include/config/auto.conf; then	\
 				echo "$$line";						\
 			fi								\
 		done > $@;								\
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 544f6af..d3009de 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -632,6 +632,7 @@ class Slot:
         cmd = list(self.make_cmd)
         if cross_compile:
             cmd.append('CROSS_COMPILE=%s' % cross_compile)
+        cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
         cmd.append('include/config/auto.conf')
         self.ps = subprocess.Popen(cmd, stdout=self.devnull)
         self.state = STATE_AUTOCONF
-- 
1.7.11.5

[U-Boot] [PATCH v4 02/10] moveconfig: Ignore duplicate configs when moving

[Masahiro Yamada]

2015-05-16 6:40 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
> When moving configs, it is important to know what was defined in the
> config header even if it duplicates the configs coming from Kconfig.
>
> This is specifically needed for the case where a config is set to
> default 'y' in the Kconfig. This would previously cause the actual value
> from the include config to be filtered out, and moveconfig.py would
> think that it was 'n'... This means that the value that should be 'y'
> is now (in every defconfig) set to 'not set'.
>
> tools/moveconfig.py now defines KCONFIG_IGNORE_DUPLICATES to prevent the
> filtering from happening and selecting wrong values for the defconfig.
>
> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>
> ---
>
> Changes in v4: None
> Changes in v3:
> -New for version 3
>
> Changes in v2: None
>
>  scripts/Makefile.autoconf | 3 ++-
>  tools/moveconfig.py       | 1 +
>  2 files changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/scripts/Makefile.autoconf b/scripts/Makefile.autoconf
> index f054081..36bfa17 100644
> --- a/scripts/Makefile.autoconf
> +++ b/scripts/Makefile.autoconf
> @@ -58,7 +58,8 @@ quiet_cmd_autoconf = GEN     $@
>         $(CPP) $(c_flags) $2 -DDO_DEPS_ONLY -dM $(srctree)/include/common.h > $@.tmp && { \
>                 sed -n -f $(srctree)/tools/scripts/define2mk.sed $@.tmp |               \
>                 while read line; do                                                     \
> -                       if ! grep -q "$${line%=*}=" include/config/auto.conf; then      \
> +                       if [ -n "${KCONFIG_IGNORE_DUPLICATES}" ] ||                     \
> +                          ! grep -q "$${line%=*}=" include/config/auto.conf; then      \
>                                 echo "$$line";                                          \
>                         fi                                                              \
>                 done > $@;                                                              \
> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
> index 544f6af..d3009de 100755
> --- a/tools/moveconfig.py
> +++ b/tools/moveconfig.py
> @@ -632,6 +632,7 @@ class Slot:
>          cmd = list(self.make_cmd)
>          if cross_compile:
>              cmd.append('CROSS_COMPILE=%s' % cross_compile)
> +        cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
>          cmd.append('include/config/auto.conf')
>          self.ps = subprocess.Popen(cmd, stdout=self.devnull)
>          self.state = STATE_AUTOCONF
> --


Now, I fully understood your intention.
This comes from our work-flow differences.

Because I edited the Kconfig after moving configs that are default to y,
so I did not notice this necessity.

Anyway, we should do it beforehand for savedefconfig.

So, this looks good to me.

Acked-by: Masahiro Yamada <yamada.masahiro@socionext.com>


-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH v4 03/10] moveconfig: Add a parameter to accept a list to build

[Joe Hershberger]

This is helpful to re-attempt to move failed boards from a previous run
without starting over.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>

---

Changes in v4: None
Changes in v3:
-Fixed command line options order (alphabetize)

Changes in v2:
-New for version 2

 tools/moveconfig.py | 26 ++++++++++++++++++++------
 1 file changed, 20 insertions(+), 6 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index d3009de..3f31e81 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -135,6 +135,9 @@ Available options
    Surround each portion of the log with escape sequences to display it
    in color on the terminal.
 
+ -d, --defconfigs
+  Specify a file containing a list of defconfigs to move
+
  -n, --dry-run
    Peform a trial run that does not make any changes.  It is useful to
    see what is going to happen before one actually runs it.
@@ -734,12 +737,21 @@ def move_config(config_attrs, options):
                                                 config_attr['type'],
                                                 config_attr['default'])
 
-    # All the defconfig files to be processed
-    defconfigs = []
-    for (dirpath, dirnames, filenames) in os.walk('configs'):
-        dirpath = dirpath[len('configs') + 1:]
-        for filename in fnmatch.filter(filenames, '*_defconfig'):
-            defconfigs.append(os.path.join(dirpath, filename))
+    if options.defconfigs:
+        defconfigs = [line.strip() for line in open(options.defconfigs, 'r')]
+        for i, defconfig in enumerate(defconfigs):
+            if not defconfig.endswith('_defconfig'):
+                defconfigs[i] = defconfig + '_defconfig'
+            if not os.path.exists(os.path.join('configs', defconfigs[i])):
+                sys.exit('%s - defconfig does not exist. Stopping.' %
+                         defconfigs[i])
+    else:
+        # All the defconfig files to be processed
+        defconfigs = []
+        for (dirpath, dirnames, filenames) in os.walk('configs'):
+            dirpath = dirpath[len('configs') + 1:]
+            for filename in fnmatch.filter(filenames, '*_defconfig'):
+                defconfigs.append(os.path.join(dirpath, filename))
 
     slots = Slots(config_attrs, options)
 
@@ -840,6 +852,8 @@ def main():
     # Add options here
     parser.add_option('-c', '--color', action='store_true', default=False,
                       help='display the log in color')
+    parser.add_option('-d', '--defconfigs', type='string',
+                      help='a file containing a list of defconfigs to move')
     parser.add_option('-n', '--dry-run', action='store_true', default=False,
                       help='perform a trial run (show log with no changes)')
     parser.add_option('-e', '--exit-on-error', action='store_true',
-- 
1.7.11.5

[U-Boot] [PATCH v4 03/10] moveconfig: Add a parameter to accept a list to build

[Masahiro Yamada]

2015-05-16 6:40 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
> This is helpful to re-attempt to move failed boards from a previous run
> without starting over.
>
> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>
> ---
>
> Changes in v4: None
> Changes in v3:
> -Fixed command line options order (alphabetize)
>
> Changes in v2:
> -New for version 2
>
>  tools/moveconfig.py | 26 ++++++++++++++++++++------
>  1 file changed, 20 insertions(+), 6 deletions(-)
>
> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
> index d3009de..3f31e81 100755
> --- a/tools/moveconfig.py
> +++ b/tools/moveconfig.py
> @@ -135,6 +135,9 @@ Available options
>     Surround each portion of the log with escape sequences to display it
>     in color on the terminal.
>
> + -d, --defconfigs
> +  Specify a file containing a list of defconfigs to move
> +
>   -n, --dry-run
>     Peform a trial run that does not make any changes.  It is useful to
>     see what is going to happen before one actually runs it.
> @@ -734,12 +737,21 @@ def move_config(config_attrs, options):
>                                                  config_attr['type'],
>                                                  config_attr['default'])
>
> -    # All the defconfig files to be processed
> -    defconfigs = []
> -    for (dirpath, dirnames, filenames) in os.walk('configs'):
> -        dirpath = dirpath[len('configs') + 1:]
> -        for filename in fnmatch.filter(filenames, '*_defconfig'):
> -            defconfigs.append(os.path.join(dirpath, filename))
> +    if options.defconfigs:
> +        defconfigs = [line.strip() for line in open(options.defconfigs, 'r')]
> +        for i, defconfig in enumerate(defconfigs):
> +            if not defconfig.endswith('_defconfig'):
> +                defconfigs[i] = defconfig + '_defconfig'
> +            if not os.path.exists(os.path.join('configs', defconfigs[i])):
> +                sys.exit('%s - defconfig does not exist. Stopping.' %
> +                         defconfigs[i])


'r' is redundant for open() because read-mode is default.

moveconfig.failed always prefixes _defconfig,
so we need not to do so again, I think. (or will we make this file by hand?)

Then, we can drop enumarate and write the error message within 80 columns.

    if options.defconfigs:
        defconfigs = [line.strip() for line in open(options.defconfigs)]
        for defconfig in defconfigs:
            if not os.path.exists(os.path.join('configs', defconfig)):
                sys.exit('%s: defconfig does not exist. Stopping.' % defconfig)



>      slots = Slots(config_attrs, options)
>
> @@ -840,6 +852,8 @@ def main():
>      # Add options here
>      parser.add_option('-c', '--color', action='store_true', default=False,
>                        help='display the log in color')
> +    parser.add_option('-d', '--defconfigs', type='string',
> +                      help='a file containing a list of defconfigs to move')
>      parser.add_option('-n', '--dry-run', action='store_true', default=False,
>                        help='perform a trial run (show log with no changes)')
>      parser.add_option('-e', '--exit-on-error', action='store_true',





-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH v4 03/10] moveconfig: Add a parameter to accept a list to build

[Joe Hershberger]

Hi Masahiro-san,

On Mon, May 18, 2015 at 11:33 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> 2015-05-16 6:40 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
>> This is helpful to re-attempt to move failed boards from a previous run
>> without starting over.
>>
>> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>>
>> ---
>>
>> Changes in v4: None
>> Changes in v3:
>> -Fixed command line options order (alphabetize)
>>
>> Changes in v2:
>> -New for version 2
>>
>>  tools/moveconfig.py | 26 ++++++++++++++++++++------
>>  1 file changed, 20 insertions(+), 6 deletions(-)
>>
>> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
>> index d3009de..3f31e81 100755
>> --- a/tools/moveconfig.py
>> +++ b/tools/moveconfig.py
>> @@ -135,6 +135,9 @@ Available options
>>     Surround each portion of the log with escape sequences to display it
>>     in color on the terminal.
>>
>> + -d, --defconfigs
>> +  Specify a file containing a list of defconfigs to move
>> +
>>   -n, --dry-run
>>     Peform a trial run that does not make any changes.  It is useful to
>>     see what is going to happen before one actually runs it.
>> @@ -734,12 +737,21 @@ def move_config(config_attrs, options):
>>                                                  config_attr['type'],
>>                                                  config_attr['default'])
>>
>> -    # All the defconfig files to be processed
>> -    defconfigs = []
>> -    for (dirpath, dirnames, filenames) in os.walk('configs'):
>> -        dirpath = dirpath[len('configs') + 1:]
>> -        for filename in fnmatch.filter(filenames, '*_defconfig'):
>> -            defconfigs.append(os.path.join(dirpath, filename))
>> +    if options.defconfigs:
>> +        defconfigs = [line.strip() for line in open(options.defconfigs, 'r')]
>> +        for i, defconfig in enumerate(defconfigs):
>> +            if not defconfig.endswith('_defconfig'):
>> +                defconfigs[i] = defconfig + '_defconfig'
>> +            if not os.path.exists(os.path.join('configs', defconfigs[i])):
>> +                sys.exit('%s - defconfig does not exist. Stopping.' %
>> +                         defconfigs[i])
>
>
> 'r' is redundant for open() because read-mode is default.

OK.

> moveconfig.failed always prefixes _defconfig,
> so we need not to do so again, I think. (or will we make this file by hand?)

I have done both moveconfig.failed as well as a hand-written file for
testing certain boards.  That's why I added both the append
'_defconfig' as well as the check for it actually existing.

> Then, we can drop enumarate and write the error message within 80 columns.
>
>     if options.defconfigs:
>         defconfigs = [line.strip() for line in open(options.defconfigs)]
>         for defconfig in defconfigs:
>             if not os.path.exists(os.path.join('configs', defconfig)):
>                 sys.exit('%s: defconfig does not exist. Stopping.' % defconfig)

I think we should keep the check for endswith('_defconfig').

>>      slots = Slots(config_attrs, options)
>>
>> @@ -840,6 +852,8 @@ def main():
>>      # Add options here
>>      parser.add_option('-c', '--color', action='store_true', default=False,
>>                        help='display the log in color')
>> +    parser.add_option('-d', '--defconfigs', type='string',
>> +                      help='a file containing a list of defconfigs to move')
>>      parser.add_option('-n', '--dry-run', action='store_true', default=False,
>>                        help='perform a trial run (show log with no changes)')
>>      parser.add_option('-e', '--exit-on-error', action='store_true',

Cheers,
-Joe

[U-Boot] [PATCH v4 03/10] moveconfig: Add a parameter to accept a list to build

[Masahiro Yamada]

2015-05-20 2:58 GMT+09:00 Joe Hershberger <joe.hershberger@gmail.com>:
> Hi Masahiro-san,
>
> On Mon, May 18, 2015 at 11:33 PM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> 2015-05-16 6:40 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
>>> This is helpful to re-attempt to move failed boards from a previous run
>>> without starting over.
>>>
>>> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>>>
>>> ---
>>>
>>> Changes in v4: None
>>> Changes in v3:
>>> -Fixed command line options order (alphabetize)
>>>
>>> Changes in v2:
>>> -New for version 2
>>>
>>>  tools/moveconfig.py | 26 ++++++++++++++++++++------
>>>  1 file changed, 20 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
>>> index d3009de..3f31e81 100755
>>> --- a/tools/moveconfig.py
>>> +++ b/tools/moveconfig.py
>>> @@ -135,6 +135,9 @@ Available options
>>>     Surround each portion of the log with escape sequences to display it
>>>     in color on the terminal.
>>>
>>> + -d, --defconfigs
>>> +  Specify a file containing a list of defconfigs to move
>>> +
>>>   -n, --dry-run
>>>     Peform a trial run that does not make any changes.  It is useful to
>>>     see what is going to happen before one actually runs it.
>>> @@ -734,12 +737,21 @@ def move_config(config_attrs, options):
>>>                                                  config_attr['type'],
>>>                                                  config_attr['default'])
>>>
>>> -    # All the defconfig files to be processed
>>> -    defconfigs = []
>>> -    for (dirpath, dirnames, filenames) in os.walk('configs'):
>>> -        dirpath = dirpath[len('configs') + 1:]
>>> -        for filename in fnmatch.filter(filenames, '*_defconfig'):
>>> -            defconfigs.append(os.path.join(dirpath, filename))
>>> +    if options.defconfigs:
>>> +        defconfigs = [line.strip() for line in open(options.defconfigs, 'r')]
>>> +        for i, defconfig in enumerate(defconfigs):
>>> +            if not defconfig.endswith('_defconfig'):
>>> +                defconfigs[i] = defconfig + '_defconfig'
>>> +            if not os.path.exists(os.path.join('configs', defconfigs[i])):
>>> +                sys.exit('%s - defconfig does not exist. Stopping.' %
>>> +                         defconfigs[i])
>>
>>
>> 'r' is redundant for open() because read-mode is default.
>
> OK.
>
>> moveconfig.failed always prefixes _defconfig,
>> so we need not to do so again, I think. (or will we make this file by hand?)
>
> I have done both moveconfig.failed as well as a hand-written file for
> testing certain boards.  That's why I added both the append
> '_defconfig' as well as the check for it actually existing.
>
>> Then, we can drop enumarate and write the error message within 80 columns.
>>
>>     if options.defconfigs:
>>         defconfigs = [line.strip() for line in open(options.defconfigs)]
>>         for defconfig in defconfigs:
>>             if not os.path.exists(os.path.join('configs', defconfig)):
>>                 sys.exit('%s: defconfig does not exist. Stopping.' % defconfig)
>
> I think we should keep the check for endswith('_defconfig').


OK, then.  Let's keep it.



-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH v4 04/10] moveconfig: Add a switch to only cleanup headers

[Joe Hershberger]

In some case you may want to only cleanup the headers. Make it possible
without waiting for all boards to compile.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>

---

Changes in v4: None
Changes in v3:
-New for version 3

Changes in v2: None

 tools/moveconfig.py | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 3f31e81..b6db058 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -146,6 +146,9 @@ Available options
    Exit immediately if Make exits with a non-zero status while processing
    a defconfig file.
 
+ -H, --headers-only
+   Only cleanup the headers; skip the defconfig processing
+
  -j, --jobs
    Specify the number of threads to run simultaneously.  If not specified,
    the number of threads is the same as the number of CPU cores.
@@ -770,8 +773,6 @@ def move_config(config_attrs, options):
 
     slots.show_failed_boards()
 
-    cleanup_headers(config_attrs, options.dry_run)
-
 def bad_recipe(filename, linenum, msg):
     """Print error message with the file name and the line number and exit."""
     sys.exit("%s: line %d: error : " % (filename, linenum) + msg)
@@ -859,6 +860,9 @@ def main():
     parser.add_option('-e', '--exit-on-error', action='store_true',
                       default=False,
                       help='exit immediately on any error')
+    parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
+                      action='store_true', default=False,
+                      help='only cleanup the headers')
     parser.add_option('-j', '--jobs', type='int', default=cpu_count,
                       help='the number of jobs to run simultaneously')
     parser.usage += ' recipe_file\n\n' + \
@@ -879,7 +883,10 @@ def main():
 
     update_cross_compile()
 
-    move_config(config_attrs, options)
+    if not options.cleanup_headers_only:
+        move_config(config_attrs, options)
+
+    cleanup_headers(config_attrs, options.dry_run)
 
 if __name__ == '__main__':
     main()
-- 
1.7.11.5

[U-Boot] [PATCH v4 04/10] moveconfig: Add a switch to only cleanup headers

[Masahiro Yamada]

Hi Joe,

2015-05-16 6:40 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
> In some case you may want to only cleanup the headers. Make it possible
> without waiting for all boards to compile.
>
> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>
> ---
>
> Changes in v4: None
> Changes in v3:
> -New for version 3
>
> Changes in v2: None
>
>  tools/moveconfig.py | 13 ++++++++++---
>  1 file changed, 10 insertions(+), 3 deletions(-)
>
> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
> index 3f31e81..b6db058 100755
> --- a/tools/moveconfig.py
> +++ b/tools/moveconfig.py
> @@ -146,6 +146,9 @@ Available options
>     Exit immediately if Make exits with a non-zero status while processing
>     a defconfig file.
>
> + -H, --headers-only
> +   Only cleanup the headers; skip the defconfig processing
> +
>   -j, --jobs
>     Specify the number of threads to run simultaneously.  If not specified,
>     the number of threads is the same as the number of CPU cores.
> @@ -770,8 +773,6 @@ def move_config(config_attrs, options):
>
>      slots.show_failed_boards()
>
> -    cleanup_headers(config_attrs, options.dry_run)
> -
>  def bad_recipe(filename, linenum, msg):
>      """Print error message with the file name and the line number and exit."""
>      sys.exit("%s: line %d: error : " % (filename, linenum) + msg)
> @@ -859,6 +860,9 @@ def main():
>      parser.add_option('-e', '--exit-on-error', action='store_true',
>                        default=False,
>                        help='exit immediately on any error')
> +    parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
> +                      action='store_true', default=False,
> +                      help='only cleanup the headers')
>      parser.add_option('-j', '--jobs', type='int', default=cpu_count,
>                        help='the number of jobs to run simultaneously')
>      parser.usage += ' recipe_file\n\n' + \
> @@ -879,7 +883,10 @@ def main():
>
>      update_cross_compile()
>
> -    move_config(config_attrs, options)
> +    if not options.cleanup_headers_only:
> +        move_config(config_attrs, options)
> +
> +    cleanup_headers(config_attrs, options.dry_run)
>
>  if __name__ == '__main__':
>      main()


Could you also move the check_top_directory() call
to main() function, above the move_config call.

We should make sure we are at the top directory
also for cleaning headers.



-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH v4 04/10] moveconfig: Add a switch to only cleanup headers

[Joe Hershberger]

Hi Masahiro-san,

On Mon, May 18, 2015 at 9:03 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Hi Joe,
>
> 2015-05-16 6:40 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
>> In some case you may want to only cleanup the headers. Make it possible
>> without waiting for all boards to compile.
>>
>> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>>
>> ---
>>
>> Changes in v4: None
>> Changes in v3:
>> -New for version 3
>>
>> Changes in v2: None
>>
>>  tools/moveconfig.py | 13 ++++++++++---
>>  1 file changed, 10 insertions(+), 3 deletions(-)
>>
>> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
>> index 3f31e81..b6db058 100755
>> --- a/tools/moveconfig.py
>> +++ b/tools/moveconfig.py
>> @@ -146,6 +146,9 @@ Available options
>>     Exit immediately if Make exits with a non-zero status while processing
>>     a defconfig file.
>>
>> + -H, --headers-only
>> +   Only cleanup the headers; skip the defconfig processing
>> +
>>   -j, --jobs
>>     Specify the number of threads to run simultaneously.  If not specified,
>>     the number of threads is the same as the number of CPU cores.
>> @@ -770,8 +773,6 @@ def move_config(config_attrs, options):
>>
>>      slots.show_failed_boards()
>>
>> -    cleanup_headers(config_attrs, options.dry_run)
>> -
>>  def bad_recipe(filename, linenum, msg):
>>      """Print error message with the file name and the line number and exit."""
>>      sys.exit("%s: line %d: error : " % (filename, linenum) + msg)
>> @@ -859,6 +860,9 @@ def main():
>>      parser.add_option('-e', '--exit-on-error', action='store_true',
>>                        default=False,
>>                        help='exit immediately on any error')
>> +    parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
>> +                      action='store_true', default=False,
>> +                      help='only cleanup the headers')
>>      parser.add_option('-j', '--jobs', type='int', default=cpu_count,
>>                        help='the number of jobs to run simultaneously')
>>      parser.usage += ' recipe_file\n\n' + \
>> @@ -879,7 +883,10 @@ def main():
>>
>>      update_cross_compile()
>>
>> -    move_config(config_attrs, options)
>> +    if not options.cleanup_headers_only:
>> +        move_config(config_attrs, options)
>> +
>> +    cleanup_headers(config_attrs, options.dry_run)
>>
>>  if __name__ == '__main__':
>>      main()
>
>
> Could you also move the check_top_directory() call
> to main() function, above the move_config call.
>
> We should make sure we are at the top directory
> also for cleaning headers.

Yes... moving it.

-Joe

[U-Boot] [PATCH v4 05/10] moveconfig: Cleanup headers in arch and board

[Joe Hershberger]

Some config.h files live in arch and board directories. They will need
to be cleaned up as well, so run the same filters there.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>

---

Changes in v4:
-New for version 4

Changes in v3: None
Changes in v2: None

 tools/moveconfig.py | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index b6db058..25cee21 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -352,6 +352,16 @@ def cleanup_headers(config_attrs, dry_run):
             if not fnmatch.fnmatch(filename, '*~'):
                 cleanup_one_header(os.path.join(dirpath, filename), patterns,
                                    dry_run)
+    for (dirpath, dirnames, filenames) in os.walk('arch'):
+        for filename in filenames:
+            if not fnmatch.fnmatch(filename, '*~'):
+                cleanup_one_header(os.path.join(dirpath, filename), patterns,
+                                   dry_run)
+    for (dirpath, dirnames, filenames) in os.walk('board'):
+        for filename in filenames:
+            if not fnmatch.fnmatch(filename, '*~'):
+                cleanup_one_header(os.path.join(dirpath, filename), patterns,
+                                   dry_run)
 
 ### classes ###
 class KconfigParser:
-- 
1.7.11.5

[U-Boot] [PATCH v4 05/10] moveconfig: Cleanup headers in arch and board

[Masahiro Yamada]

Hi Joe,


2015-05-16 6:40 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
> Some config.h files live in arch and board directories. They will need
> to be cleaned up as well, so run the same filters there.
>
> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>
> ---
>
> Changes in v4:
> -New for version 4
>
> Changes in v3: None
> Changes in v2: None
>
>  tools/moveconfig.py | 10 ++++++++++
>  1 file changed, 10 insertions(+)
>
> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
> index b6db058..25cee21 100755
> --- a/tools/moveconfig.py
> +++ b/tools/moveconfig.py
> @@ -352,6 +352,16 @@ def cleanup_headers(config_attrs, dry_run):
>              if not fnmatch.fnmatch(filename, '*~'):
>                  cleanup_one_header(os.path.join(dirpath, filename), patterns,
>                                     dry_run)
> +    for (dirpath, dirnames, filenames) in os.walk('arch'):
> +        for filename in filenames:
> +            if not fnmatch.fnmatch(filename, '*~'):
> +                cleanup_one_header(os.path.join(dirpath, filename), patterns,
> +                                   dry_run)
> +    for (dirpath, dirnames, filenames) in os.walk('board'):
> +        for filename in filenames:
> +            if not fnmatch.fnmatch(filename, '*~'):
> +                cleanup_one_header(os.path.join(dirpath, filename), patterns,
> +                                   dry_run)


To reduce code duplication, can we write like this or something?

    for dir in 'include', 'arch', 'board':
        for (dirpath, dirnames, filenames) in os.walk(dir):
            for filename in filenames:
                if not fnmatch.fnmatch(filename, '*~'):
                    cleanup_one_header(os.path.join(dirpath, filename),
                                       patterns, dry_run)



-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH v4 05/10] moveconfig: Cleanup headers in arch and board

[Joe Hershberger]

On Mon, May 18, 2015 at 8:41 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Hi Joe,
>
>
> 2015-05-16 6:40 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
>> Some config.h files live in arch and board directories. They will need
>> to be cleaned up as well, so run the same filters there.
>>
>> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>>
>> ---
>>
>> Changes in v4:
>> -New for version 4
>>
>> Changes in v3: None
>> Changes in v2: None
>>
>>  tools/moveconfig.py | 10 ++++++++++
>>  1 file changed, 10 insertions(+)
>>
>> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
>> index b6db058..25cee21 100755
>> --- a/tools/moveconfig.py
>> +++ b/tools/moveconfig.py
>> @@ -352,6 +352,16 @@ def cleanup_headers(config_attrs, dry_run):
>>              if not fnmatch.fnmatch(filename, '*~'):
>>                  cleanup_one_header(os.path.join(dirpath, filename), patterns,
>>                                     dry_run)
>> +    for (dirpath, dirnames, filenames) in os.walk('arch'):
>> +        for filename in filenames:
>> +            if not fnmatch.fnmatch(filename, '*~'):
>> +                cleanup_one_header(os.path.join(dirpath, filename), patterns,
>> +                                   dry_run)
>> +    for (dirpath, dirnames, filenames) in os.walk('board'):
>> +        for filename in filenames:
>> +            if not fnmatch.fnmatch(filename, '*~'):
>> +                cleanup_one_header(os.path.join(dirpath, filename), patterns,
>> +                                   dry_run)
>
>
> To reduce code duplication, can we write like this or something?
>
>     for dir in 'include', 'arch', 'board':
>         for (dirpath, dirnames, filenames) in os.walk(dir):
>             for filename in filenames:
>                 if not fnmatch.fnmatch(filename, '*~'):
>                     cleanup_one_header(os.path.join(dirpath, filename),
>                                        patterns, dry_run)

OK.

[U-Boot] [PATCH v4 06/10] moveconfig: Remove probable debug print

[Joe Hershberger]

This print seems to be redundant and unformatted compared to the next
few lines, so remove it.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
---

Changes in v4: None
Changes in v3: None
Changes in v2: None

 tools/moveconfig.py | 1 -
 1 file changed, 1 deletion(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 25cee21..15b0f2b 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -725,7 +725,6 @@ class Slots:
         if len(failed_boards) > 0:
             msg = [ "The following boards were not processed due to error:" ]
             msg += failed_boards
-            print msg
             for line in msg:
                 print >> sys.stderr, color_text(self.options.color,
                                                 COLOR_LIGHT_RED, line)
-- 
1.7.11.5

[U-Boot] [PATCH v4 06/10] moveconfig: Remove probable debug print

[Masahiro Yamada]

Hi Joe

2015-05-16 6:40 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
> This print seems to be redundant and unformatted compared to the next
> few lines, so remove it.
>
> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
> ---
>
> Changes in v4: None
> Changes in v3: None
> Changes in v2: None
>
>  tools/moveconfig.py | 1 -
>  1 file changed, 1 deletion(-)
>
> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
> index 25cee21..15b0f2b 100755
> --- a/tools/moveconfig.py
> +++ b/tools/moveconfig.py
> @@ -725,7 +725,6 @@ class Slots:
>          if len(failed_boards) > 0:
>              msg = [ "The following boards were not processed due to error:" ]
>              msg += failed_boards
> -            print msg
>              for line in msg:
>                  print >> sys.stderr, color_text(self.options.color,
>                                                  COLOR_LIGHT_RED, line)
> --

Oops, my mistake.

Acked-by: Masahiro Yamada <yamada.masahiro@socionext.com>


(Or, because it is a simply bug fix, you can comment on my base patch.)


-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH v4 06/10] moveconfig: Remove probable debug print

[Joe Hershberger]

Hi Masahiro-san,

On Mon, May 18, 2015 at 9:10 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Hi Joe
>
> 2015-05-16 6:40 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
>> This print seems to be redundant and unformatted compared to the next
>> few lines, so remove it.
>>
>> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>> ---
>>
>> Changes in v4: None
>> Changes in v3: None
>> Changes in v2: None
>>
>>  tools/moveconfig.py | 1 -
>>  1 file changed, 1 deletion(-)
>>
>> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
>> index 25cee21..15b0f2b 100755
>> --- a/tools/moveconfig.py
>> +++ b/tools/moveconfig.py
>> @@ -725,7 +725,6 @@ class Slots:
>>          if len(failed_boards) > 0:
>>              msg = [ "The following boards were not processed due to error:" ]
>>              msg += failed_boards
>> -            print msg
>>              for line in msg:
>>                  print >> sys.stderr, color_text(self.options.color,
>>                                                  COLOR_LIGHT_RED, line)
>> --
>
> Oops, my mistake.
>
> Acked-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>
>
> (Or, because it is a simply bug fix, you can comment on my base patch.)

I'm dropping this patch and sent a reply to your original patch (not
applied to master yet).

-Joe

[U-Boot] [PATCH v4 07/10] moveconfig: Output a list of failed boards

[Joe Hershberger]

If boards fail, output that list to a file so that it can easily be
passed back into moveconfig.py using the -d option.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
---

Changes in v4: None
Changes in v3: None
Changes in v2: None

 tools/moveconfig.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 15b0f2b..9e923da 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -728,6 +728,10 @@ class Slots:
             for line in msg:
                 print >> sys.stderr, color_text(self.options.color,
                                                 COLOR_LIGHT_RED, line)
+            ffail = open('moveconfig.failed', 'w')
+            for failed_board in failed_boards:
+                ffail.write("%s\n" % failed_board)
+            ffail.close()
 
 def move_config(config_attrs, options):
     """Move config options to defconfig files.
-- 
1.7.11.5

[U-Boot] [PATCH v4 07/10] moveconfig: Output a list of failed boards

[Masahiro Yamada]

2015-05-16 6:40 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
> If boards fail, output that list to a file so that it can easily be
> passed back into moveconfig.py using the -d option.
>
> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
> ---
>
> Changes in v4: None
> Changes in v3: None
> Changes in v2: None
>
>  tools/moveconfig.py | 4 ++++
>  1 file changed, 4 insertions(+)
>
> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
> index 15b0f2b..9e923da 100755
> --- a/tools/moveconfig.py
> +++ b/tools/moveconfig.py
> @@ -728,6 +728,10 @@ class Slots:
>              for line in msg:
>                  print >> sys.stderr, color_text(self.options.color,
>                                                  COLOR_LIGHT_RED, line)
> +            ffail = open('moveconfig.failed', 'w')
> +            for failed_board in failed_boards:
> +                ffail.write("%s\n" % failed_board)
> +            ffail.close()
>
>  def move_config(config_attrs, options):
>      """Move config options to defconfig files.


If you use with ... as ..., it will automatically close
the file on exit.

    with open('moveconfig.failed', 'w') as f:
    for board in failed_boards:
        f.write(board + '\n')





-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH v4 07/10] moveconfig: Output a list of failed boards

[Joe Hershberger]

Hi Masahiro-san,

On Mon, May 18, 2015 at 10:12 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> 2015-05-16 6:40 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
>> If boards fail, output that list to a file so that it can easily be
>> passed back into moveconfig.py using the -d option.
>>
>> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>> ---
>>
>> Changes in v4: None
>> Changes in v3: None
>> Changes in v2: None
>>
>>  tools/moveconfig.py | 4 ++++
>>  1 file changed, 4 insertions(+)
>>
>> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
>> index 15b0f2b..9e923da 100755
>> --- a/tools/moveconfig.py
>> +++ b/tools/moveconfig.py
>> @@ -728,6 +728,10 @@ class Slots:
>>              for line in msg:
>>                  print >> sys.stderr, color_text(self.options.color,
>>                                                  COLOR_LIGHT_RED, line)
>> +            ffail = open('moveconfig.failed', 'w')
>> +            for failed_board in failed_boards:
>> +                ffail.write("%s\n" % failed_board)
>> +            ffail.close()
>>
>>  def move_config(config_attrs, options):
>>      """Move config options to defconfig files.
>
>
> If you use with ... as ..., it will automatically close
> the file on exit.
>
>     with open('moveconfig.failed', 'w') as f:
>     for board in failed_boards:
>         f.write(board + '\n')

OK.

-Joe

[U-Boot] [PATCH v4 08/10] moveconfig: Print a message for missing compiler

[Joe Hershberger]

A common case for failed builds is a missing compiler. Print a message
for that case to tell the user concisely which compiler was expected
that was not found.

This patch also has the effect of not printing build errors any longer.
The next patch will add a switch to optionally bring that back.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
---

Changes in v4: None
Changes in v3: None
Changes in v2: None

 tools/moveconfig.py | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 9e923da..f986f55 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -166,6 +166,7 @@ import os
 import re
 import shutil
 import subprocess
+from subprocess import PIPE
 import sys
 import tempfile
 import time
@@ -606,11 +607,14 @@ class Slot:
             return False
 
         if self.ps.poll() != 0:
-
+            errmsg = 'Failed to process.'
+            errout = self.ps.stderr.read()
+            if errout.find('gcc: command not found') != -1:
+                errmsg = 'Compiler not found (%s)' % self.cross_compile
             print >> sys.stderr, log_msg(self.options.color,
                                          COLOR_LIGHT_RED,
                                          self.defconfig,
-                                         "failed to process.")
+                                         errmsg),
             if self.options.exit_on_error:
                 sys.exit("Exit on error.")
             else:
@@ -644,13 +648,13 @@ class Slot:
             self.state = STATE_IDLE
             return True
 
-        cross_compile = self.parser.get_cross_compile()
+        self.cross_compile = self.parser.get_cross_compile()
         cmd = list(self.make_cmd)
-        if cross_compile:
-            cmd.append('CROSS_COMPILE=%s' % cross_compile)
+        if self.cross_compile:
+            cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
         cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
         cmd.append('include/config/auto.conf')
-        self.ps = subprocess.Popen(cmd, stdout=self.devnull)
+        self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE)
         self.state = STATE_AUTOCONF
         return False
 
-- 
1.7.11.5

[U-Boot] [PATCH v4 08/10] moveconfig: Print a message for missing compiler

[Masahiro Yamada]

2015-05-16 6:40 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
> A common case for failed builds is a missing compiler. Print a message
> for that case to tell the user concisely which compiler was expected
> that was not found.
>
> This patch also has the effect of not printing build errors any longer.
> The next patch will add a switch to optionally bring that back.
>
> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
> ---
>
> Changes in v4: None
> Changes in v3: None
> Changes in v2: None
>
>  tools/moveconfig.py | 16 ++++++++++------
>  1 file changed, 10 insertions(+), 6 deletions(-)
>
> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
> index 9e923da..f986f55 100755
> --- a/tools/moveconfig.py
> +++ b/tools/moveconfig.py
> @@ -166,6 +166,7 @@ import os
>  import re
>  import shutil
>  import subprocess
> +from subprocess import PIPE

Personally I do not prefer from ... import
because it disturbs the naming space.

Could you use subprocess.PIPE instead?


>  import sys
>  import tempfile
>  import time
> @@ -606,11 +607,14 @@ class Slot:
>              return False
>
>          if self.ps.poll() != 0:
> -
> +            errmsg = 'Failed to process.'
> +            errout = self.ps.stderr.read()

This throws exception if "make *_defconfig" or "make savedefconfig" fail.



Traceback (most recent call last):
  File "tools/moveconfig.py", line 924, in <module>
    main()
  File "tools/moveconfig.py", line 919, in main
    move_config(config_attrs, options)
  File "tools/moveconfig.py", line 794, in move_config
    while not slots.available():
  File "tools/moveconfig.py", line 717, in available
    if slot.poll():
  File "tools/moveconfig.py", line 616, in poll
    errout = self.ps.stderr.read()
AttributeError: 'NoneType' object has no attribute 'read'



Seems better to add PIPE for all the call of subprocess.Popen()





> +            if errout.find('gcc: command not found') != -1:
> +                errmsg = 'Compiler not found (%s)' % self.cross_compile

If you do this, should the locale be changed?

Without LANG=C, "command not found" is displayed in Japanese on my computer.


If --verbose is given, we will be able to know the cause of erorr.
"missing compiler" is a special case error?



>              print >> sys.stderr, log_msg(self.options.color,
>                                           COLOR_LIGHT_RED,
>                                           self.defconfig,
> -                                         "failed to process.")
> +                                         errmsg),
>              if self.options.exit_on_error:
>                  sys.exit("Exit on error.")
>              else:
> @@ -644,13 +648,13 @@ class Slot:
>              self.state = STATE_IDLE
>              return True
>
> -        cross_compile = self.parser.get_cross_compile()
> +        self.cross_compile = self.parser.get_cross_compile()
>          cmd = list(self.make_cmd)
> -        if cross_compile:
> -            cmd.append('CROSS_COMPILE=%s' % cross_compile)
> +        if self.cross_compile:
> +            cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
>          cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
>          cmd.append('include/config/auto.conf')
> -        self.ps = subprocess.Popen(cmd, stdout=self.devnull)
> +        self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE)
>          self.state = STATE_AUTOCONF
>          return False
>

-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH v4 08/10] moveconfig: Print a message for missing compiler

[Joe Hershberger]

Hi Masahiro-san,

On Mon, May 18, 2015 at 10:23 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> 2015-05-16 6:40 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
>> A common case for failed builds is a missing compiler. Print a message
>> for that case to tell the user concisely which compiler was expected
>> that was not found.
>>
>> This patch also has the effect of not printing build errors any longer.
>> The next patch will add a switch to optionally bring that back.
>>
>> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>> ---
>>
>> Changes in v4: None
>> Changes in v3: None
>> Changes in v2: None
>>
>>  tools/moveconfig.py | 16 ++++++++++------
>>  1 file changed, 10 insertions(+), 6 deletions(-)
>>
>> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
>> index 9e923da..f986f55 100755
>> --- a/tools/moveconfig.py
>> +++ b/tools/moveconfig.py
>> @@ -166,6 +166,7 @@ import os
>>  import re
>>  import shutil
>>  import subprocess
>> +from subprocess import PIPE
>
> Personally I do not prefer from ... import
> because it disturbs the naming space.
>
> Could you use subprocess.PIPE instead?

OK.

>>  import sys
>>  import tempfile
>>  import time
>> @@ -606,11 +607,14 @@ class Slot:
>>              return False
>>
>>          if self.ps.poll() != 0:
>> -
>> +            errmsg = 'Failed to process.'
>> +            errout = self.ps.stderr.read()
>
> This throws exception if "make *_defconfig" or "make savedefconfig" fail.
>
>
>
> Traceback (most recent call last):
>   File "tools/moveconfig.py", line 924, in <module>
>     main()
>   File "tools/moveconfig.py", line 919, in main
>     move_config(config_attrs, options)
>   File "tools/moveconfig.py", line 794, in move_config
>     while not slots.available():
>   File "tools/moveconfig.py", line 717, in available
>     if slot.poll():
>   File "tools/moveconfig.py", line 616, in poll
>     errout = self.ps.stderr.read()
> AttributeError: 'NoneType' object has no attribute 'read'
>
>
>
> Seems better to add PIPE for all the call of subprocess.Popen()

OK

>> +            if errout.find('gcc: command not found') != -1:
>> +                errmsg = 'Compiler not found (%s)' % self.cross_compile
>
> If you do this, should the locale be changed?
>
> Without LANG=C, "command not found" is displayed in Japanese on my computer.
>
>
> If --verbose is given, we will be able to know the cause of erorr.
> "missing compiler" is a special case error?

That's true, but at least for my use-case before I spent several days
getting all tool-chains working, it was nice to know what the build
was trying to use for a cross tool-chain in a concise one-liner
instead of parsing a bunch of error prints. That's part of why I added
the -v flag. It's also helpful (naturally) in getting the compilers
all working.

>>              print >> sys.stderr, log_msg(self.options.color,
>>                                           COLOR_LIGHT_RED,
>>                                           self.defconfig,
>> -                                         "failed to process.")
>> +                                         errmsg),
>>              if self.options.exit_on_error:
>>                  sys.exit("Exit on error.")
>>              else:
>> @@ -644,13 +648,13 @@ class Slot:
>>              self.state = STATE_IDLE
>>              return True
>>
>> -        cross_compile = self.parser.get_cross_compile()
>> +        self.cross_compile = self.parser.get_cross_compile()
>>          cmd = list(self.make_cmd)
>> -        if cross_compile:
>> -            cmd.append('CROSS_COMPILE=%s' % cross_compile)
>> +        if self.cross_compile:
>> +            cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
>>          cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
>>          cmd.append('include/config/auto.conf')
>> -        self.ps = subprocess.Popen(cmd, stdout=self.devnull)
>> +        self.ps = subprocess.Popen(cmd, stdout=self.devnull, stderr=PIPE)
>>          self.state = STATE_AUTOCONF
>>          return False
>>
>
> --
> Best Regards
> Masahiro Yamada
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> http://lists.denx.de/mailman/listinfo/u-boot

[U-Boot] [PATCH v4 08/10] moveconfig: Print a message for missing compiler

[Masahiro Yamada]

2015-05-20 2:51 GMT+09:00 Joe Hershberger <joe.hershberger@gmail.com>:
> Hi Masahiro-san,
>
> On Mon, May 18, 2015 at 10:23 PM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> 2015-05-16 6:40 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
>>> A common case for failed builds is a missing compiler. Print a message
>>> for that case to tell the user concisely which compiler was expected
>>> that was not found.
>>>
>>> This patch also has the effect of not printing build errors any longer.
>>> The next patch will add a switch to optionally bring that back.
>>>
>>> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>>> ---
>>>
>>> Changes in v4: None
>>> Changes in v3: None
>>> Changes in v2: None
>>>
>>>  tools/moveconfig.py | 16 ++++++++++------
>>>  1 file changed, 10 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
>>> index 9e923da..f986f55 100755
>>> --- a/tools/moveconfig.py
>>> +++ b/tools/moveconfig.py
>>> @@ -166,6 +166,7 @@ import os
>>>  import re
>>>  import shutil
>>>  import subprocess
>>> +from subprocess import PIPE
>>
>> Personally I do not prefer from ... import
>> because it disturbs the naming space.
>>
>> Could you use subprocess.PIPE instead?
>
> OK.
>
>>>  import sys
>>>  import tempfile
>>>  import time
>>> @@ -606,11 +607,14 @@ class Slot:
>>>              return False
>>>
>>>          if self.ps.poll() != 0:
>>> -
>>> +            errmsg = 'Failed to process.'
>>> +            errout = self.ps.stderr.read()
>>
>> This throws exception if "make *_defconfig" or "make savedefconfig" fail.
>>
>>
>>
>> Traceback (most recent call last):
>>   File "tools/moveconfig.py", line 924, in <module>
>>     main()
>>   File "tools/moveconfig.py", line 919, in main
>>     move_config(config_attrs, options)
>>   File "tools/moveconfig.py", line 794, in move_config
>>     while not slots.available():
>>   File "tools/moveconfig.py", line 717, in available
>>     if slot.poll():
>>   File "tools/moveconfig.py", line 616, in poll
>>     errout = self.ps.stderr.read()
>> AttributeError: 'NoneType' object has no attribute 'read'
>>
>>
>>
>> Seems better to add PIPE for all the call of subprocess.Popen()
>
> OK
>
>>> +            if errout.find('gcc: command not found') != -1:
>>> +                errmsg = 'Compiler not found (%s)' % self.cross_compile
>>
>> If you do this, should the locale be changed?
>>
>> Without LANG=C, "command not found" is displayed in Japanese on my computer.
>>
>>
>> If --verbose is given, we will be able to know the cause of erorr.
>> "missing compiler" is a special case error?
>
> That's true, but at least for my use-case before I spent several days
> getting all tool-chains working, it was nice to know what the build
> was trying to use for a cross tool-chain in a concise one-liner
> instead of parsing a bunch of error prints. That's part of why I added
> the -v flag. It's also helpful (naturally) in getting the compilers
> all working.

OK.
You pass LANG=C in v5, so it works fine for me.




-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH v4 09/10] moveconfig: Add a switch to enable printing errors

[Joe Hershberger]

In some cases the build for the autoconf breaks. This outputs the errors
following the status so that action can be taken without building again
manually.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>

---

Changes in v4: None
Changes in v3: None
Changes in v2:
-New for version 2

 tools/moveconfig.py | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index f986f55..685b47b 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -153,6 +153,9 @@ Available options
    Specify the number of threads to run simultaneously.  If not specified,
    the number of threads is the same as the number of CPU cores.
 
+ -v, --verbose
+   Show any build errors as boards are built
+
 To see the complete list of supported options, run
 
   $ tools/moveconfig.py -h
@@ -615,6 +618,9 @@ class Slot:
                                          COLOR_LIGHT_RED,
                                          self.defconfig,
                                          errmsg),
+            if self.options.verbose:
+                print >> sys.stderr, color_text(self.options.color,
+                                                COLOR_LIGHT_CYAN, errout)
             if self.options.exit_on_error:
                 sys.exit("Exit on error.")
             else:
@@ -882,6 +888,9 @@ def main():
                       help='only cleanup the headers')
     parser.add_option('-j', '--jobs', type='int', default=cpu_count,
                       help='the number of jobs to run simultaneously')
+    parser.add_option('-v', '--verbose', dest='verbose',
+                      action='store_true', default=False,
+                      help='show any build errors as boards are built')
     parser.usage += ' recipe_file\n\n' + \
                     'The recipe_file should describe config options you want to move.\n' + \
                     'Each line should contain config_name, type, default_value\n\n' + \
-- 
1.7.11.5

[U-Boot] [PATCH v4 09/10] moveconfig: Add a switch to enable printing errors

[Masahiro Yamada]

2015-05-16 6:40 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
> In some cases the build for the autoconf breaks. This outputs the errors
> following the status so that action can be taken without building again
> manually.
>
> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>
> ---
>
> Changes in v4: None
> Changes in v3: None
> Changes in v2:
> -New for version 2
>
>  tools/moveconfig.py | 9 +++++++++
>  1 file changed, 9 insertions(+)
>
> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
> index f986f55..685b47b 100755
> --- a/tools/moveconfig.py
> +++ b/tools/moveconfig.py
> @@ -153,6 +153,9 @@ Available options
>     Specify the number of threads to run simultaneously.  If not specified,
>     the number of threads is the same as the number of CPU cores.
>
> + -v, --verbose
> +   Show any build errors as boards are built
> +
>  To see the complete list of supported options, run
>
>    $ tools/moveconfig.py -h
> @@ -615,6 +618,9 @@ class Slot:
>                                           COLOR_LIGHT_RED,
>                                           self.defconfig,
>                                           errmsg),
> +            if self.options.verbose:
> +                print >> sys.stderr, color_text(self.options.color,
> +                                                COLOR_LIGHT_CYAN, errout)
>              if self.options.exit_on_error:
>                  sys.exit("Exit on error.")
>              else:
> @@ -882,6 +888,9 @@ def main():
>                        help='only cleanup the headers')
>      parser.add_option('-j', '--jobs', type='int', default=cpu_count,
>                        help='the number of jobs to run simultaneously')
> +    parser.add_option('-v', '--verbose', dest='verbose',
> +                      action='store_true', default=False,
> +                      help='show any build errors as boards are built')

dest='verbose' is redundant.

The destination is chosen based on the long option name.


Apart from that, I think this is a nice improvement.


-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH v4 09/10] moveconfig: Add a switch to enable printing errors

[Joe Hershberger]

Hi Masahiro-san,

On Mon, May 18, 2015 at 10:25 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> 2015-05-16 6:40 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
>> In some cases the build for the autoconf breaks. This outputs the errors
>> following the status so that action can be taken without building again
>> manually.
>>
>> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>>
>> ---
>>
>> Changes in v4: None
>> Changes in v3: None
>> Changes in v2:
>> -New for version 2
>>
>>  tools/moveconfig.py | 9 +++++++++
>>  1 file changed, 9 insertions(+)
>>
>> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
>> index f986f55..685b47b 100755
>> --- a/tools/moveconfig.py
>> +++ b/tools/moveconfig.py
>> @@ -153,6 +153,9 @@ Available options
>>     Specify the number of threads to run simultaneously.  If not specified,
>>     the number of threads is the same as the number of CPU cores.
>>
>> + -v, --verbose
>> +   Show any build errors as boards are built
>> +
>>  To see the complete list of supported options, run
>>
>>    $ tools/moveconfig.py -h
>> @@ -615,6 +618,9 @@ class Slot:
>>                                           COLOR_LIGHT_RED,
>>                                           self.defconfig,
>>                                           errmsg),
>> +            if self.options.verbose:
>> +                print >> sys.stderr, color_text(self.options.color,
>> +                                                COLOR_LIGHT_CYAN, errout)
>>              if self.options.exit_on_error:
>>                  sys.exit("Exit on error.")
>>              else:
>> @@ -882,6 +888,9 @@ def main():
>>                        help='only cleanup the headers')
>>      parser.add_option('-j', '--jobs', type='int', default=cpu_count,
>>                        help='the number of jobs to run simultaneously')
>> +    parser.add_option('-v', '--verbose', dest='verbose',
>> +                      action='store_true', default=False,
>> +                      help='show any build errors as boards are built')
>
> dest='verbose' is redundant.
>
> The destination is chosen based on the long option name.

OK. Dropped.

> Apart from that, I think this is a nice improvement.
>
>
> --
> Best Regards
> Masahiro Yamada
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> http://lists.denx.de/mailman/listinfo/u-boot

[U-Boot] [PATCH v4 10/10] moveconfig: Print status about the processed defconfigs

[Joe Hershberger]

This gives a basic idea about progress.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>

---

Changes in v4: None
Changes in v3:
-New for version 3

Changes in v2: None

 tools/moveconfig.py | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 685b47b..fca0197 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -568,7 +568,7 @@ class Slot:
                 pass
         shutil.rmtree(self.build_dir)
 
-    def add(self, defconfig):
+    def add(self, defconfig, num, total):
         """Assign a new subprocess for defconfig and add it to the slot.
 
         If the slot is vacant, create a new subprocess for processing the
@@ -588,6 +588,8 @@ class Slot:
         self.ps = subprocess.Popen(cmd, stdout=self.devnull)
         self.defconfig = defconfig
         self.state = STATE_DEFCONFIG
+        self.num = num
+        self.total = total
         return True
 
     def poll(self):
@@ -634,6 +636,9 @@ class Slot:
         if self.state == STATE_AUTOCONF:
             self.parser.update_defconfig(self.defconfig)
 
+            print ' %d defconfigs out of %d\r' % (self.num + 1, self.total),
+            sys.stdout.flush()
+
             """Save off the defconfig in a consistent way"""
             cmd = list(self.make_cmd)
             cmd.append('savedefconfig')
@@ -688,7 +693,7 @@ class Slots:
         for i in range(options.jobs):
             self.slots.append(Slot(config_attrs, options, devnull, make_cmd))
 
-    def add(self, defconfig):
+    def add(self, defconfig, num, total):
         """Add a new subprocess if a vacant slot is found.
 
         Arguments:
@@ -698,7 +703,7 @@ class Slots:
           Return True on success or False on failure
         """
         for slot in self.slots:
-            if slot.add(defconfig):
+            if slot.add(defconfig, num, total):
                 return True
         return False
 
@@ -784,8 +789,8 @@ def move_config(config_attrs, options):
     # Main loop to process defconfig files:
     #  Add a new subprocess into a vacant slot.
     #  Sleep if there is no available slot.
-    for defconfig in defconfigs:
-        while not slots.add(defconfig):
+    for i, defconfig in enumerate(defconfigs):
+        while not slots.add(defconfig, i, len(defconfigs)):
             while not slots.available():
                 # No available slot: sleep for a while
                 time.sleep(SLEEP_TIME)
@@ -794,6 +799,7 @@ def move_config(config_attrs, options):
     while not slots.empty():
         time.sleep(SLEEP_TIME)
 
+    print ''
     slots.show_failed_boards()
 
 def bad_recipe(filename, linenum, msg):
-- 
1.7.11.5

[U-Boot] [PATCH v4 10/10] moveconfig: Print status about the processed defconfigs

[Masahiro Yamada]

2015-05-16 6:40 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
> This gives a basic idea about progress.
>
> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>

Good idea!


-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH v4 01/10] moveconfig: Always run savedefconfig on the moved config

[Masahiro Yamada]

Hi Joe,

2015-05-16 6:40 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
> This will ensure that the order of the defconfig entries will always
> match that of the Kconfig files. After one slightly painful (but
> still early in the process) pass over all boards, this should keep
> the defconfigs clean from here on.
>
> Users must edit the Kconfig first to add the menu entries and then run
> moveconfig.py to update the defconfig files and the include configs.
>
> As such, moveconfig.py cannot compare against the '.config' contents.
>
> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>


This feature expects the defconfigs are all clean.
Otherwise, savedefconfig would make "git diff" noisier.

It is safer to use "make menuconfig && make savedefconfig"
for adding new options, but some people still try to
edit the defconfig directly...

Perhaps, should do the global cleanup periodically?


> ---
> This is based on https://patchwork.ozlabs.org/patch/472591/
>
> Changes in v4:
> -Rebased series on Masahiro's v2
>
> Changes in v3: None
> Changes in v2: None
>
>  tools/moveconfig.py | 35 ++++++++++++++++++++++++++---------
>  1 file changed, 26 insertions(+), 9 deletions(-)
>
> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
> index c39ea95..544f6af 100755
> --- a/tools/moveconfig.py
> +++ b/tools/moveconfig.py
> @@ -46,6 +46,9 @@ should look like this:
>    CONFIG_CMD_USB bool n
>    CONFIG_SYS_TEXT_BASE hex 0x00000000
>
> +Next you must edit the Kconfig to add the menu entries for the configs
> +you are moving.
> +
>  And then run this tool giving the file name of the recipe

Uh, I was doing in a different work-flow.
(I edit the Kconfig after I move CONFIGs to defconfigs).

But, the Kconfig must be edited beforehand
to do savedefconfig.

So, I am OK with this change.



>    $ tools/moveconfig.py recipe
> @@ -192,6 +195,7 @@ CROSS_COMPILE = {
>  STATE_IDLE = 0
>  STATE_DEFCONFIG = 1
>  STATE_AUTOCONF = 2
> +STATE_SAVEDEFCONFIG = 3
>
>  ACTION_MOVE = 0
>  ACTION_DEFAULT_VALUE = 1
> @@ -390,8 +394,7 @@ class KconfigParser:
>
>          return CROSS_COMPILE.get(arch, '')
>
> -    def parse_one_config(self, config_attr, defconfig_lines,
> -                         dotconfig_lines, autoconf_lines):
> +    def parse_one_config(self, config_attr, defconfig_lines, autoconf_lines):
>          """Parse .config, defconfig, include/autoconf.mk for one config.
>
>          This function looks for the config options in the lines from
> @@ -402,7 +405,6 @@ class KconfigParser:
>            config_attr: A dictionary including the name, the type,
>                         and the default value of the target config.
>            defconfig_lines: lines from the original defconfig file.
> -          dotconfig_lines: lines from the .config file.
>            autoconf_lines: lines from the include/autoconf.mk file.
>
>          Returns:
> @@ -418,7 +420,7 @@ class KconfigParser:
>          else:
>              default = config + '=' + config_attr['default']
>
> -        for line in defconfig_lines + dotconfig_lines:
> +        for line in defconfig_lines:
>              line = line.rstrip()
>              if line.startswith(config + '=') or line == not_set:
>                  return (ACTION_ALREADY_EXIST, line)
> @@ -463,15 +465,12 @@ class KconfigParser:
>          with open(defconfig_path) as f:
>              defconfig_lines = f.readlines()
>
> -        with open(dotconfig_path) as f:
> -            dotconfig_lines = f.readlines()
> -
>          with open(autoconf_path) as f:
>              autoconf_lines = f.readlines()
>
>          for config_attr in self.config_attrs:
>              result = self.parse_one_config(config_attr, defconfig_lines,
> -                                           dotconfig_lines, autoconf_lines)
> +                                           autoconf_lines)
>              results.append(result)
>
>          log = ''


With the change of the work-flow above,
we need not parse the .config, so this seems OK.



> @@ -499,7 +498,7 @@ class KconfigParser:
>          print log,
>
>          if not self.options.dry_run:
> -            with open(defconfig_path, 'a') as f:
> +            with open(dotconfig_path, 'a') as f:
>                  for (action, value) in results:
>                      if action == ACTION_MOVE:
>                          f.write(value + '\n')
> @@ -608,6 +607,24 @@ class Slot:
>
>          if self.state == STATE_AUTOCONF:
>              self.parser.update_defconfig(self.defconfig)
> +
> +            """Save off the defconfig in a consistent way"""
> +            cmd = list(self.make_cmd)
> +            cmd.append('savedefconfig')
> +            self.ps = subprocess.Popen(cmd, stdout=self.devnull,
> +                                       stderr=self.devnull)
> +            self.state = STATE_SAVEDEFCONFIG
> +            return False
> +
> +        if self.state == STATE_SAVEDEFCONFIG:
> +            defconfig_path = os.path.join(self.build_dir, 'defconfig')
> +            if not os.path.exists(defconfig_path):
> +                print >> sys.stderr, log_msg(self.options.color,
> +                                             COLOR_LIGHT_RED,
> +                                             self.defconfig,
> +                                             'The defconfig was not updated')

Is this warning message reachable?

The missing defconfig means "make savedefconfig" failed.

That case has been already caught by 'Failed to process.' above, I think.


> +            shutil.move(defconfig_path,
> +                        os.path.join('configs', self.defconfig))
>              self.state = STATE_IDLE
>              return True





-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH v4 01/10] moveconfig: Always run savedefconfig on the moved config

[Joe Hershberger]

Hi Masahiro-san,

On Mon, May 18, 2015 at 8:58 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Hi Joe,
>
> 2015-05-16 6:40 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
>> This will ensure that the order of the defconfig entries will always
>> match that of the Kconfig files. After one slightly painful (but
>> still early in the process) pass over all boards, this should keep
>> the defconfigs clean from here on.
>>
>> Users must edit the Kconfig first to add the menu entries and then run
>> moveconfig.py to update the defconfig files and the include configs.
>>
>> As such, moveconfig.py cannot compare against the '.config' contents.
>>
>> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>
>
> This feature expects the defconfigs are all clean.
> Otherwise, savedefconfig would make "git diff" noisier.
>
> It is safer to use "make menuconfig && make savedefconfig"
> for adding new options, but some people still try to
> edit the defconfig directly...
>
> Perhaps, should do the global cleanup periodically?
>
>
>> ---
>> This is based on https://patchwork.ozlabs.org/patch/472591/
>>
>> Changes in v4:
>> -Rebased series on Masahiro's v2
>>
>> Changes in v3: None
>> Changes in v2: None
>>
>>  tools/moveconfig.py | 35 ++++++++++++++++++++++++++---------
>>  1 file changed, 26 insertions(+), 9 deletions(-)
>>
>> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
>> index c39ea95..544f6af 100755
>> --- a/tools/moveconfig.py
>> +++ b/tools/moveconfig.py
>> @@ -46,6 +46,9 @@ should look like this:
>>    CONFIG_CMD_USB bool n
>>    CONFIG_SYS_TEXT_BASE hex 0x00000000
>>
>> +Next you must edit the Kconfig to add the menu entries for the configs
>> +you are moving.
>> +
>>  And then run this tool giving the file name of the recipe
>
> Uh, I was doing in a different work-flow.
> (I edit the Kconfig after I move CONFIGs to defconfigs).
>
> But, the Kconfig must be edited beforehand
> to do savedefconfig.
>
> So, I am OK with this change.
>
>
>
>>    $ tools/moveconfig.py recipe
>> @@ -192,6 +195,7 @@ CROSS_COMPILE = {
>>  STATE_IDLE = 0
>>  STATE_DEFCONFIG = 1
>>  STATE_AUTOCONF = 2
>> +STATE_SAVEDEFCONFIG = 3
>>
>>  ACTION_MOVE = 0
>>  ACTION_DEFAULT_VALUE = 1
>> @@ -390,8 +394,7 @@ class KconfigParser:
>>
>>          return CROSS_COMPILE.get(arch, '')
>>
>> -    def parse_one_config(self, config_attr, defconfig_lines,
>> -                         dotconfig_lines, autoconf_lines):
>> +    def parse_one_config(self, config_attr, defconfig_lines, autoconf_lines):
>>          """Parse .config, defconfig, include/autoconf.mk for one config.
>>
>>          This function looks for the config options in the lines from
>> @@ -402,7 +405,6 @@ class KconfigParser:
>>            config_attr: A dictionary including the name, the type,
>>                         and the default value of the target config.
>>            defconfig_lines: lines from the original defconfig file.
>> -          dotconfig_lines: lines from the .config file.
>>            autoconf_lines: lines from the include/autoconf.mk file.
>>
>>          Returns:
>> @@ -418,7 +420,7 @@ class KconfigParser:
>>          else:
>>              default = config + '=' + config_attr['default']
>>
>> -        for line in defconfig_lines + dotconfig_lines:
>> +        for line in defconfig_lines:
>>              line = line.rstrip()
>>              if line.startswith(config + '=') or line == not_set:
>>                  return (ACTION_ALREADY_EXIST, line)
>> @@ -463,15 +465,12 @@ class KconfigParser:
>>          with open(defconfig_path) as f:
>>              defconfig_lines = f.readlines()
>>
>> -        with open(dotconfig_path) as f:
>> -            dotconfig_lines = f.readlines()
>> -
>>          with open(autoconf_path) as f:
>>              autoconf_lines = f.readlines()
>>
>>          for config_attr in self.config_attrs:
>>              result = self.parse_one_config(config_attr, defconfig_lines,
>> -                                           dotconfig_lines, autoconf_lines)
>> +                                           autoconf_lines)
>>              results.append(result)
>>
>>          log = ''
>
>
> With the change of the work-flow above,
> we need not parse the .config, so this seems OK.
>
>
>
>> @@ -499,7 +498,7 @@ class KconfigParser:
>>          print log,
>>
>>          if not self.options.dry_run:
>> -            with open(defconfig_path, 'a') as f:
>> +            with open(dotconfig_path, 'a') as f:
>>                  for (action, value) in results:
>>                      if action == ACTION_MOVE:
>>                          f.write(value + '\n')
>> @@ -608,6 +607,24 @@ class Slot:
>>
>>          if self.state == STATE_AUTOCONF:
>>              self.parser.update_defconfig(self.defconfig)
>> +
>> +            """Save off the defconfig in a consistent way"""
>> +            cmd = list(self.make_cmd)
>> +            cmd.append('savedefconfig')
>> +            self.ps = subprocess.Popen(cmd, stdout=self.devnull,
>> +                                       stderr=self.devnull)
>> +            self.state = STATE_SAVEDEFCONFIG
>> +            return False
>> +
>> +        if self.state == STATE_SAVEDEFCONFIG:
>> +            defconfig_path = os.path.join(self.build_dir, 'defconfig')
>> +            if not os.path.exists(defconfig_path):
>> +                print >> sys.stderr, log_msg(self.options.color,
>> +                                             COLOR_LIGHT_RED,
>> +                                             self.defconfig,
>> +                                             'The defconfig was not updated')
>
> Is this warning message reachable?
>
> The missing defconfig means "make savedefconfig" failed.
>
> That case has been already caught by 'Failed to process.' above, I think.

OK... dropping that warning.

>> +            shutil.move(defconfig_path,
>> +                        os.path.join('configs', self.defconfig))
>>              self.state = STATE_IDLE
>>              return True
>
>
>
>
>
> --
> Best Regards
> Masahiro Yamada
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> http://lists.denx.de/mailman/listinfo/u-boot

[U-Boot] [PATCH v4 01/10] moveconfig: Always run savedefconfig on the moved config

[Joe Hershberger]

Hi Masahiro-san,

On Mon, May 18, 2015 at 8:58 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Hi Joe,
>
> 2015-05-16 6:40 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
>> This will ensure that the order of the defconfig entries will always
>> match that of the Kconfig files. After one slightly painful (but
>> still early in the process) pass over all boards, this should keep
>> the defconfigs clean from here on.
>>
>> Users must edit the Kconfig first to add the menu entries and then run
>> moveconfig.py to update the defconfig files and the include configs.
>>
>> As such, moveconfig.py cannot compare against the '.config' contents.
>>
>> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
>
>
> This feature expects the defconfigs are all clean.
> Otherwise, savedefconfig would make "git diff" noisier.
>
> It is safer to use "make menuconfig && make savedefconfig"
> for adding new options, but some people still try to
> edit the defconfig directly...
>
> Perhaps, should do the global cleanup periodically?

I agree that until people (both contributors and custodians) get used
to this as the best way, we will likely need a few global clean-ups,
but I think with the addition of this tool to main-line the instance
of this will likely drop sharply. Moving configs is a hassle, so with
a tool available I doubt many people will do it manually. That's why
it is important that the tool do it the cleanest way. Hopefully for
the case where a config is changed on a board, the custodian will
notice if it is just added to the bottom of the file. It seems to work
out for Linux.

Cheers,
-Joe

[U-Boot] [PATCH v5 1/9] moveconfig: Always run savedefconfig on the moved config

[Joe Hershberger]

This will ensure that the order of the defconfig entries will always
match that of the Kconfig files. After one slightly painful (but
still early in the process) pass over all boards, this should keep
the defconfigs clean from here on.

Users must edit the Kconfig first to add the menu entries and then run
moveconfig.py to update the defconfig files and the include configs.

As such, moveconfig.py cannot compare against the '.config' contents.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
---
This is based on https://patchwork.ozlabs.org/patch/472591/

Changes in v5:
-Removed warning that may never be reached

Changes in v4:
-Rebased series on Masahiro's v2

Changes in v3: None
Changes in v2: None

 tools/moveconfig.py | 30 +++++++++++++++++++++---------
 1 file changed, 21 insertions(+), 9 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index c39ea95..c9984b4 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -46,6 +46,9 @@ should look like this:
   CONFIG_CMD_USB bool n
   CONFIG_SYS_TEXT_BASE hex 0x00000000
 
+Next you must edit the Kconfig to add the menu entries for the configs
+you are moving.
+
 And then run this tool giving the file name of the recipe
 
   $ tools/moveconfig.py recipe
@@ -192,6 +195,7 @@ CROSS_COMPILE = {
 STATE_IDLE = 0
 STATE_DEFCONFIG = 1
 STATE_AUTOCONF = 2
+STATE_SAVEDEFCONFIG = 3
 
 ACTION_MOVE = 0
 ACTION_DEFAULT_VALUE = 1
@@ -390,8 +394,7 @@ class KconfigParser:
 
         return CROSS_COMPILE.get(arch, '')
 
-    def parse_one_config(self, config_attr, defconfig_lines,
-                         dotconfig_lines, autoconf_lines):
+    def parse_one_config(self, config_attr, defconfig_lines, autoconf_lines):
         """Parse .config, defconfig, include/autoconf.mk for one config.
 
         This function looks for the config options in the lines from
@@ -402,7 +405,6 @@ class KconfigParser:
           config_attr: A dictionary including the name, the type,
                        and the default value of the target config.
           defconfig_lines: lines from the original defconfig file.
-          dotconfig_lines: lines from the .config file.
           autoconf_lines: lines from the include/autoconf.mk file.
 
         Returns:
@@ -418,7 +420,7 @@ class KconfigParser:
         else:
             default = config + '=' + config_attr['default']
 
-        for line in defconfig_lines + dotconfig_lines:
+        for line in defconfig_lines:
             line = line.rstrip()
             if line.startswith(config + '=') or line == not_set:
                 return (ACTION_ALREADY_EXIST, line)
@@ -463,15 +465,12 @@ class KconfigParser:
         with open(defconfig_path) as f:
             defconfig_lines = f.readlines()
 
-        with open(dotconfig_path) as f:
-            dotconfig_lines = f.readlines()
-
         with open(autoconf_path) as f:
             autoconf_lines = f.readlines()
 
         for config_attr in self.config_attrs:
             result = self.parse_one_config(config_attr, defconfig_lines,
-                                           dotconfig_lines, autoconf_lines)
+                                           autoconf_lines)
             results.append(result)
 
         log = ''
@@ -499,7 +498,7 @@ class KconfigParser:
         print log,
 
         if not self.options.dry_run:
-            with open(defconfig_path, 'a') as f:
+            with open(dotconfig_path, 'a') as f:
                 for (action, value) in results:
                     if action == ACTION_MOVE:
                         f.write(value + '\n')
@@ -608,6 +607,19 @@ class Slot:
 
         if self.state == STATE_AUTOCONF:
             self.parser.update_defconfig(self.defconfig)
+
+            """Save off the defconfig in a consistent way"""
+            cmd = list(self.make_cmd)
+            cmd.append('savedefconfig')
+            self.ps = subprocess.Popen(cmd, stdout=self.devnull,
+                                       stderr=self.devnull)
+            self.state = STATE_SAVEDEFCONFIG
+            return False
+
+        if self.state == STATE_SAVEDEFCONFIG:
+            defconfig_path = os.path.join(self.build_dir, 'defconfig')
+            shutil.move(defconfig_path,
+                        os.path.join('configs', self.defconfig))
             self.state = STATE_IDLE
             return True
 
-- 
1.7.11.5

[U-Boot] [PATCH v5 2/9] moveconfig: Ignore duplicate configs when moving

[Joe Hershberger]

When moving configs, it is important to know what was defined in the
config header even if it duplicates the configs coming from Kconfig.

This is specifically needed for the case where a config is set to
default 'y' in the Kconfig. This would previously cause the actual value
from the include config to be filtered out, and moveconfig.py would
think that it was 'n'... This means that the value that should be 'y'
is now (in every defconfig) set to 'not set'.

tools/moveconfig.py now defines KCONFIG_IGNORE_DUPLICATES to prevent the
filtering from happening and selecting wrong values for the defconfig.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
Acked-by: Masahiro Yamada <yamada.masahiro@socionext.com>

---

Changes in v5: None
Changes in v4: None
Changes in v3:
-New for version 3

Changes in v2: None

 scripts/Makefile.autoconf | 3 ++-
 tools/moveconfig.py       | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/Makefile.autoconf b/scripts/Makefile.autoconf
index f054081..36bfa17 100644
--- a/scripts/Makefile.autoconf
+++ b/scripts/Makefile.autoconf
@@ -58,7 +58,8 @@ quiet_cmd_autoconf = GEN     $@
 	$(CPP) $(c_flags) $2 -DDO_DEPS_ONLY -dM $(srctree)/include/common.h > $@.tmp && { \
 		sed -n -f $(srctree)/tools/scripts/define2mk.sed $@.tmp |		\
 		while read line; do							\
-			if ! grep -q "$${line%=*}=" include/config/auto.conf; then	\
+			if [ -n "${KCONFIG_IGNORE_DUPLICATES}" ] ||			\
+			   ! grep -q "$${line%=*}=" include/config/auto.conf; then	\
 				echo "$$line";						\
 			fi								\
 		done > $@;								\
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index c9984b4..dd9434d 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -627,6 +627,7 @@ class Slot:
         cmd = list(self.make_cmd)
         if cross_compile:
             cmd.append('CROSS_COMPILE=%s' % cross_compile)
+        cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
         cmd.append('include/config/auto.conf')
         self.ps = subprocess.Popen(cmd, stdout=self.devnull)
         self.state = STATE_AUTOCONF
-- 
1.7.11.5

[U-Boot] [PATCH v5 3/9] moveconfig: Add a parameter to accept a list to build

[Joe Hershberger]

This is helpful to re-attempt to move failed boards from a previous run
without starting over.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>

---

Changes in v5:
-Remove default 'r' mode on open

Changes in v4: None
Changes in v3:
-Fixed command line options order (alphabetize)

Changes in v2:
-New for version 2

 tools/moveconfig.py | 26 ++++++++++++++++++++------
 1 file changed, 20 insertions(+), 6 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index dd9434d..e93548c 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -135,6 +135,9 @@ Available options
    Surround each portion of the log with escape sequences to display it
    in color on the terminal.
 
+ -d, --defconfigs
+  Specify a file containing a list of defconfigs to move
+
  -n, --dry-run
    Peform a trial run that does not make any changes.  It is useful to
    see what is going to happen before one actually runs it.
@@ -729,12 +732,21 @@ def move_config(config_attrs, options):
                                                 config_attr['type'],
                                                 config_attr['default'])
 
-    # All the defconfig files to be processed
-    defconfigs = []
-    for (dirpath, dirnames, filenames) in os.walk('configs'):
-        dirpath = dirpath[len('configs') + 1:]
-        for filename in fnmatch.filter(filenames, '*_defconfig'):
-            defconfigs.append(os.path.join(dirpath, filename))
+    if options.defconfigs:
+        defconfigs = [line.strip() for line in open(options.defconfigs)]
+        for i, defconfig in enumerate(defconfigs):
+            if not defconfig.endswith('_defconfig'):
+                defconfigs[i] = defconfig + '_defconfig'
+            if not os.path.exists(os.path.join('configs', defconfigs[i])):
+                sys.exit('%s - defconfig does not exist. Stopping.' %
+                         defconfigs[i])
+    else:
+        # All the defconfig files to be processed
+        defconfigs = []
+        for (dirpath, dirnames, filenames) in os.walk('configs'):
+            dirpath = dirpath[len('configs') + 1:]
+            for filename in fnmatch.filter(filenames, '*_defconfig'):
+                defconfigs.append(os.path.join(dirpath, filename))
 
     slots = Slots(config_attrs, options)
 
@@ -835,6 +847,8 @@ def main():
     # Add options here
     parser.add_option('-c', '--color', action='store_true', default=False,
                       help='display the log in color')
+    parser.add_option('-d', '--defconfigs', type='string',
+                      help='a file containing a list of defconfigs to move')
     parser.add_option('-n', '--dry-run', action='store_true', default=False,
                       help='perform a trial run (show log with no changes)')
     parser.add_option('-e', '--exit-on-error', action='store_true',
-- 
1.7.11.5

[U-Boot] [PATCH v5 4/9] moveconfig: Add a switch to only cleanup headers

[Joe Hershberger]

In some case you may want to only cleanup the headers. Make it possible
without waiting for all boards to compile.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>

---

Changes in v5:
-Move check_top_directory to main

Changes in v4: None
Changes in v3:
-New for version 3

Changes in v2: None

 tools/moveconfig.py | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index e93548c..561cd9a 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -146,6 +146,9 @@ Available options
    Exit immediately if Make exits with a non-zero status while processing
    a defconfig file.
 
+ -H, --headers-only
+   Only cleanup the headers; skip the defconfig processing
+
  -j, --jobs
    Specify the number of threads to run simultaneously.  If not specified,
    the number of threads is the same as the number of CPU cores.
@@ -720,8 +723,6 @@ def move_config(config_attrs, options):
                     the type, and the default value of the target config.
       options: option flags
     """
-    check_top_directory()
-
     if len(config_attrs) == 0:
         print 'Nothing to do. exit.'
         sys.exit(0)
@@ -765,8 +766,6 @@ def move_config(config_attrs, options):
 
     slots.show_failed_boards()
 
-    cleanup_headers(config_attrs, options.dry_run)
-
 def bad_recipe(filename, linenum, msg):
     """Print error message with the file name and the line number and exit."""
     sys.exit("%s: line %d: error : " % (filename, linenum) + msg)
@@ -854,6 +853,9 @@ def main():
     parser.add_option('-e', '--exit-on-error', action='store_true',
                       default=False,
                       help='exit immediately on any error')
+    parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
+                      action='store_true', default=False,
+                      help='only cleanup the headers')
     parser.add_option('-j', '--jobs', type='int', default=cpu_count,
                       help='the number of jobs to run simultaneously')
     parser.usage += ' recipe_file\n\n' + \
@@ -874,7 +876,12 @@ def main():
 
     update_cross_compile()
 
-    move_config(config_attrs, options)
+    check_top_directory()
+
+    if not options.cleanup_headers_only:
+        move_config(config_attrs, options)
+
+    cleanup_headers(config_attrs, options.dry_run)
 
 if __name__ == '__main__':
     main()
-- 
1.7.11.5

[U-Boot] [PATCH v5 5/9] moveconfig: Cleanup headers in arch and board

[Joe Hershberger]

Some config.h files live in arch and board directories. They will need
to be cleaned up as well, so run the same filters there.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>

---

Changes in v5:
-Consolidate code to clean up dirs

Changes in v4:
-New for version 4

Changes in v3: None
Changes in v2: None

 tools/moveconfig.py | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 561cd9a..f38dac1 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -347,11 +347,12 @@ def cleanup_headers(config_attrs, dry_run):
         patterns.append(re.compile(r'#\s*define\s+%s\W' % config))
         patterns.append(re.compile(r'#\s*undef\s+%s\W' % config))
 
-    for (dirpath, dirnames, filenames) in os.walk('include'):
-        for filename in filenames:
-            if not fnmatch.fnmatch(filename, '*~'):
-                cleanup_one_header(os.path.join(dirpath, filename), patterns,
-                                   dry_run)
+    for dir in 'include', 'arch', 'board':
+        for (dirpath, dirnames, filenames) in os.walk(dir):
+            for filename in filenames:
+                if not fnmatch.fnmatch(filename, '*~'):
+                    cleanup_one_header(os.path.join(dirpath, filename),
+                                       patterns, dry_run)
 
 ### classes ###
 class KconfigParser:
-- 
1.7.11.5

[U-Boot] [PATCH v5 6/9] moveconfig: Output a list of failed boards

[Joe Hershberger]

If boards fail, output that list to a file so that it can easily be
passed back into moveconfig.py using the -d option.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>

---

Changes in v5:
-Changed file handling to use with/as

Changes in v4: None
Changes in v3: None
Changes in v2: None

 tools/moveconfig.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index f38dac1..8b8eed6 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -716,6 +716,10 @@ class Slots:
                 print >> sys.stderr, color_text(self.options.color,
                                                 COLOR_LIGHT_RED, line)
 
+            with open('moveconfig.failed', 'w') as f:
+                for board in failed_boards:
+                    f.write(board + '\n')
+
 def move_config(config_attrs, options):
     """Move config options to defconfig files.
 
-- 
1.7.11.5

[U-Boot] [PATCH v5 7/9] moveconfig: Print a message for missing compiler

[Joe Hershberger]

A common case for failed builds is a missing compiler. Print a message
for that case to tell the user concisely which compiler was expected
that was not found.

This patch also has the effect of not printing build errors any longer.
The next patch will add a switch to optionally bring that back.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>

---

Changes in v5:
-Use scoped PIPE instead of importing it
-Set LANG=C for calling the compiler when planning to screen-scrape
-Use PIPE for all Popen
-Make missing compilers yellow

Changes in v4: None
Changes in v3: None
Changes in v2: None

 tools/moveconfig.py | 28 ++++++++++++++++++++--------
 1 file changed, 20 insertions(+), 8 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 8b8eed6..2fa98e3 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -572,7 +572,8 @@ class Slot:
             return False
         cmd = list(self.make_cmd)
         cmd.append(defconfig)
-        self.ps = subprocess.Popen(cmd, stdout=self.devnull)
+        self.ps = subprocess.Popen(cmd, stdout=self.devnull,
+                                   stderr=subprocess.PIPE)
         self.defconfig = defconfig
         self.state = STATE_DEFCONFIG
         return True
@@ -597,11 +598,18 @@ class Slot:
             return False
 
         if self.ps.poll() != 0:
-
+            errmsg = 'Failed to process.'
+            errout = self.ps.stderr.read()
+            if errout.find('gcc: command not found') != -1:
+                errmsg = 'Compiler not found ('
+                errmsg += color_text(self.options.color, COLOR_YELLOW,
+                                     self.cross_compile)
+                errmsg += color_text(self.options.color, COLOR_LIGHT_RED,
+                                     ')')
             print >> sys.stderr, log_msg(self.options.color,
                                          COLOR_LIGHT_RED,
                                          self.defconfig,
-                                         "failed to process.")
+                                         errmsg),
             if self.options.exit_on_error:
                 sys.exit("Exit on error.")
             else:
@@ -619,7 +627,7 @@ class Slot:
             cmd = list(self.make_cmd)
             cmd.append('savedefconfig')
             self.ps = subprocess.Popen(cmd, stdout=self.devnull,
-                                       stderr=self.devnull)
+                                       stderr=subprocess.PIPE)
             self.state = STATE_SAVEDEFCONFIG
             return False
 
@@ -630,13 +638,17 @@ class Slot:
             self.state = STATE_IDLE
             return True
 
-        cross_compile = self.parser.get_cross_compile()
+        self.cross_compile = self.parser.get_cross_compile()
         cmd = list(self.make_cmd)
-        if cross_compile:
-            cmd.append('CROSS_COMPILE=%s' % cross_compile)
+        if self.cross_compile:
+            cmd.append('CROSS_COMPILE=%s' % self.cross_compile)
         cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
         cmd.append('include/config/auto.conf')
-        self.ps = subprocess.Popen(cmd, stdout=self.devnull)
+        """This will be screen-scraped, so be sure the expected text will be
+        returned consistently on every machine by setting LANG=C"""
+        self.ps = subprocess.Popen(cmd, stdout=self.devnull,
+                                   env=dict(os.environ, LANG='C'),
+                                   stderr=subprocess.PIPE)
         self.state = STATE_AUTOCONF
         return False
 
-- 
1.7.11.5

[U-Boot] [PATCH v5 8/9] moveconfig: Add a switch to enable printing errors

[Joe Hershberger]

In some cases the build for the autoconf breaks. This outputs the errors
following the status so that action can be taken without building again
manually.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>

---

Changes in v5:
-Remove redundant destination

Changes in v4: None
Changes in v3: None
Changes in v2:
-New for version 2

 tools/moveconfig.py | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 2fa98e3..a6f1853 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -153,6 +153,9 @@ Available options
    Specify the number of threads to run simultaneously.  If not specified,
    the number of threads is the same as the number of CPU cores.
 
+ -v, --verbose
+   Show any build errors as boards are built
+
 To see the complete list of supported options, run
 
   $ tools/moveconfig.py -h
@@ -610,6 +613,9 @@ class Slot:
                                          COLOR_LIGHT_RED,
                                          self.defconfig,
                                          errmsg),
+            if self.options.verbose:
+                print >> sys.stderr, color_text(self.options.color,
+                                                COLOR_LIGHT_CYAN, errout)
             if self.options.exit_on_error:
                 sys.exit("Exit on error.")
             else:
@@ -875,6 +881,8 @@ def main():
                       help='only cleanup the headers')
     parser.add_option('-j', '--jobs', type='int', default=cpu_count,
                       help='the number of jobs to run simultaneously')
+    parser.add_option('-v', '--verbose', action='store_true', default=False,
+                      help='show any build errors as boards are built')
     parser.usage += ' recipe_file\n\n' + \
                     'The recipe_file should describe config options you want to move.\n' + \
                     'Each line should contain config_name, type, default_value\n\n' + \
-- 
1.7.11.5

[U-Boot] [PATCH v5 9/9] moveconfig: Print status about the processed defconfigs

[Joe Hershberger]

This gives a basic idea about progress.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>

---

Changes in v5: None
Changes in v4: None
Changes in v3:
-New for version 3

Changes in v2: None

 tools/moveconfig.py | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index a6f1853..a6e7c22 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -558,7 +558,7 @@ class Slot:
                 pass
         shutil.rmtree(self.build_dir)
 
-    def add(self, defconfig):
+    def add(self, defconfig, num, total):
         """Assign a new subprocess for defconfig and add it to the slot.
 
         If the slot is vacant, create a new subprocess for processing the
@@ -579,6 +579,8 @@ class Slot:
                                    stderr=subprocess.PIPE)
         self.defconfig = defconfig
         self.state = STATE_DEFCONFIG
+        self.num = num
+        self.total = total
         return True
 
     def poll(self):
@@ -629,6 +631,9 @@ class Slot:
         if self.state == STATE_AUTOCONF:
             self.parser.update_defconfig(self.defconfig)
 
+            print ' %d defconfigs out of %d\r' % (self.num + 1, self.total),
+            sys.stdout.flush()
+
             """Save off the defconfig in a consistent way"""
             cmd = list(self.make_cmd)
             cmd.append('savedefconfig')
@@ -682,7 +687,7 @@ class Slots:
         for i in range(options.jobs):
             self.slots.append(Slot(config_attrs, options, devnull, make_cmd))
 
-    def add(self, defconfig):
+    def add(self, defconfig, num, total):
         """Add a new subprocess if a vacant slot is found.
 
         Arguments:
@@ -692,7 +697,7 @@ class Slots:
           Return True on success or False on failure
         """
         for slot in self.slots:
-            if slot.add(defconfig):
+            if slot.add(defconfig, num, total):
                 return True
         return False
 
@@ -777,8 +782,8 @@ def move_config(config_attrs, options):
     # Main loop to process defconfig files:
     #  Add a new subprocess into a vacant slot.
     #  Sleep if there is no available slot.
-    for defconfig in defconfigs:
-        while not slots.add(defconfig):
+    for i, defconfig in enumerate(defconfigs):
+        while not slots.add(defconfig, i, len(defconfigs)):
             while not slots.available():
                 # No available slot: sleep for a while
                 time.sleep(SLEEP_TIME)
@@ -787,6 +792,7 @@ def move_config(config_attrs, options):
     while not slots.empty():
         time.sleep(SLEEP_TIME)
 
+    print ''
     slots.show_failed_boards()
 
 def bad_recipe(filename, linenum, msg):
-- 
1.7.11.5

[U-Boot] [PATCH v5 1/9] moveconfig: Always run savedefconfig on the moved config

[Masahiro Yamada]

Hi Joe,



2015-05-20 3:21 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
> This will ensure that the order of the defconfig entries will always
> match that of the Kconfig files. After one slightly painful (but
> still early in the process) pass over all boards, this should keep
> the defconfigs clean from here on.
>
> Users must edit the Kconfig first to add the menu entries and then run
> moveconfig.py to update the defconfig files and the include configs.
>
> As such, moveconfig.py cannot compare against the '.config' contents.
>
> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
> ---
> This is based on https://patchwork.ozlabs.org/patch/472591/
>
> Changes in v5:
> -Removed warning that may never be reached


The whole series of v5 looks good to me.


Tom,
looks like this series is delegated to me.

Shall I apply my base patch and Joe's great improvements and then send
a pull-req?
Or would you do it?



-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH v5 1/9] moveconfig: Always run savedefconfig on the moved config

[Masahiro Yamada]

2015-05-20 3:21 GMT+09:00 Joe Hershberger <joe.hershberger@ni.com>:
> This will ensure that the order of the defconfig entries will always
> match that of the Kconfig files. After one slightly painful (but
> still early in the process) pass over all boards, this should keep
> the defconfigs clean from here on.
>
> Users must edit the Kconfig first to add the menu entries and then run
> moveconfig.py to update the defconfig files and the include configs.
>
> As such, moveconfig.py cannot compare against the '.config' contents.
>
> Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
> ---
> This is based on https://patchwork.ozlabs.org/patch/472591/
>
> Changes in v5:
> -Removed warning that may never be reached


The series, applied to u-boot-uniphier/misc with my Acked-by.
Thanks!


-- 
Best Regards
Masahiro Yamada