[U-Boot] [RFC PATCH v3 0/3] python tools to inspect configs

[U-Boot] [RFC PATCH v3 0/3] python tools to inspect configs

[Jean-Jacques Hiblot]

This series introduce 2 new python tools that helps getting an overview
of the configuration options.
First one is fairly simple and is used to locate deconfigs based on several
criteria: 'arch', 'soc', 'cpu', 'vendor', 'board', 'defconfig name',
'maintainer' and 'status'. All the parameters use regexp.
ex: Show all defconfigs of platforms built around imx25 or imx27 or imxs
$ tools/find_defconfigs.py --soc 'mx(25|27|s)'

The 2nd tool is used to produce a CSV file that summarizes the usage of
config options by a set of defconfigs. Useful to check what platforms might
need modifications when working on a particular option.
ex:
Get the TI platforms that enable CONFIG_DM_I2C_COMPAT either in u-boot or
the SPL
$ tools/configs2csv.py -X  CONFIG_DM_I2C_COMPAT --spl --u-boot --vendor ti \
  --discard-empty

limitations:
- must be executed at the root of the source tree
- the source tree must be clean (make mrproper)
- only supports CSV format. visualization in terminal sould be added.
  But LibreOffice calc is better suited with all its ordering/
  filtering capabilities.

Changes in v3:
- Add more comments to describe the classes and functions
- Stylistics changes

Changes in v2:
- basically rewrote the whole thing
- use tools/moveconfig.py to generate the database of configs
- use tools/find_defconfigs.py to get the list of defconfigs off interest
- removed diff with .config. tools/moveconfig.py does a better job

Jean-Jacques Hiblot (3):
  tools: moveconfig: Add an option to build a fuller database of options
  tools: Add a tool to get a list of defconfigs based on filters
  tools: Add a tool to get an overview of the usage of CONFIG options

 tools/configs2csv.py     | 427 +++++++++++++++++++++++++++++++++++++++++++++++
 tools/find_defconfigs.py | 204 ++++++++++++++++++++++
 tools/moveconfig.py      |  56 ++++++-
 3 files changed, 680 insertions(+), 7 deletions(-)
 create mode 100755 tools/configs2csv.py
 create mode 100755 tools/find_defconfigs.py

-- 
2.7.4

[U-Boot] [RFC PATCH v3 1/3] tools: moveconfig: Add an option to build a fuller database of options

[Jean-Jacques Hiblot]

"moveconfig -b" will build a database of config options based on the
content of include/config/auto.conf that reflects the .config

Add a new option '-B' that does essentially the same, except that it uses
the content of u-boot.cfg, spl/u-boot.cfg and tpl/u-boot.cfg.
This allows to get the options from .config AND the headers for all the
possible binary types (u-boot, SPL and TPL)

Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
---

Changes in v3: None
Changes in v2: New

 tools/moveconfig.py | 56 ++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 49 insertions(+), 7 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index caa81ac..85c2b8b 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -1151,9 +1151,17 @@ class Slot:
                                    self.ps.stderr.read())
         self.finish(False)
 
+    def remove_cfg_files(self):
+        for base in ['.', 'spl', 'tpl']:
+            try:
+                os.remove(os.path.join(self.build_dir, base, "u-boot.cfg"))
+            except:
+                pass
+
     def do_defconfig(self):
         """Run 'make <board>_defconfig' to create the .config file."""
-
+        # first remove old cfg files that may have been produced earlier
+        self.remove_cfg_files()
         cmd = list(self.make_cmd)
         cmd.append(self.defconfig)
         self.ps = subprocess.Popen(cmd, stdout=self.devnull,
@@ -1182,15 +1190,43 @@ class Slot:
                                    cwd=self.current_src_dir)
         self.state = STATE_AUTOCONF
 
+
     def do_build_db(self):
         """Add the board to the database"""
-        configs = {}
-        with open(os.path.join(self.build_dir, AUTO_CONF_PATH)) as fd:
-            for line in fd.readlines():
+        def conf_to_dic(filename):
+            configs = {}
+            if not os.path.isfile(filename):
+                return None
+
+            try:
+                if filename.endswith("u-boot.cfg"):
+                        with tempfile.NamedTemporaryFile(delete = True) as fd:
+                            lines = subprocess.check_call("sed -n -f tools/scripts/define2mk.sed {}".format(filename).split(), stdout = fd)
+                            fd.seek(0)
+                            lines = fd.readlines()
+                else:
+                        with open(filename, "r") as fd:
+                            lines = fd.readlines()
+            except:
+                return None
+
+            for line in lines:
                 if line.startswith('CONFIG'):
                     config, value = line.split('=', 1)
                     configs[config] = value.rstrip()
-        self.db_queue.put([self.defconfig, configs])
+            return configs
+
+        list_of_conf = []
+        if self.options.build_full_db:
+            list_of_conf.append((self.defconfig,"u-boot.cfg"))
+            list_of_conf.append(("{} SPL".format(self.defconfig),"spl/u-boot.cfg"))
+            list_of_conf.append(("{} TPL".format(self.defconfig),"tpl/u-boot.cfg"))
+        else:
+            list_of_conf.append((self.defconfig,AUTO_CONF_PATH))
+        for name,conf_file in list_of_conf:
+            configs = conf_to_dic(os.path.join(self.build_dir, conf_file))
+            if configs:
+                self.db_queue.put([name, configs])
         self.finish(True)
 
     def do_savedefconfig(self):
@@ -1770,7 +1806,9 @@ def main():
                       help="don't show options which are already marked as "
                       'implying others')
     parser.add_option('-b', '--build-db', action='store_true', default=False,
-                      help='build a CONFIG database')
+                      help='build a CONFIG database based only on auto.conf')
+    parser.add_option('-B', '--build-full-db', action='store_true', default=False,
+                      help='build a CONFIG database based only on u-boot.cfg (and also for SPL and TPL)')
     parser.add_option('-c', '--color', action='store_true', default=False,
                       help='display the log in color')
     parser.add_option('-C', '--commit', action='store_true', default=False,
@@ -1807,6 +1845,9 @@ def main():
 
     (options, configs) = parser.parse_args()
 
+    if options.build_full_db:
+        options.build_db = True
+
     if len(configs) == 0 and not any((options.force_sync, options.build_db,
                                       options.imply)):
         parser.print_usage()
@@ -1875,7 +1916,8 @@ def main():
 
     if options.build_db:
         with open(CONFIG_DATABASE, 'w') as fd:
-            for defconfig, configs in config_db.iteritems():
+            for defconfig in sorted(config_db.keys()):
+                configs = config_db[defconfig]
                 fd.write('%s\n' % defconfig)
                 for config in sorted(configs.keys()):
                     fd.write('   %s=%s\n' % (config, configs[config]))
-- 
2.7.4

[U-Boot] [RFC PATCH v3 2/3] tools: Add a tool to get a list of defconfigs based on filters

[Jean-Jacques Hiblot]

The possible filters are "arch", "vendor", "soc", "cpu" and "arch".

The list of all the defconfigs is read from boards.cfg. If this file
doesn't exist, then tools/genboardscfg.py is called to generate it.

Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
---

Changes in v3:
- Add more comments to describe the classes and functions
- Capitalize the Class names

Changes in v2: New

 tools/find_defconfigs.py | 204 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 204 insertions(+)
 create mode 100755 tools/find_defconfigs.py

diff --git a/tools/find_defconfigs.py b/tools/find_defconfigs.py
new file mode 100755
index 0000000..aac5212
--- /dev/null
+++ b/tools/find_defconfigs.py
@@ -0,0 +1,204 @@
+#!/usr/bin/env python
+# SPDX-License-Identifier: GPL-2.0+
+#
+# Author: JJ Hiblot <jjhiblot@ti.com>
+#
+
+"""
+Output a list of defconfig-matching criteria.
+
+The possible criteria are soc, vendor, arch, cpu, board and defconfig name.
+The criteria are expressed as regexp, allowing for complex selection.
+
+How does it work?
+-----------------
+
+This tools uses the boards.cfg file produced by tools/genboardscfg.py
+It reads the file to get a list of all the defconfigs and the information
+about the soc, vendor etc. for each of them.
+Then it walks this list and outputs the defconfigs for which the info match
+the regexp passed to the program.
+
+examples:
+---------
+
+1) Get the list of defconfigs for boards built around omap5, omap4 and k3, not built by TI
+
+$ tools/find_defconfigs.py  --soc 'omap[45]|k3' --vendor '(?!ti)'
+kc1_defconfig
+duovero_defconfig
+cl-som-am57x_defconfig
+cm_t54_defconfig
+
+2) Same list but iwth more details on the items that were used as filters
+
+$  tools/find_defconfigs.py  --soc 'omap[45]|k3' --vendor '(?!ti)' --show-details
+kc1_defconfig | omap4 | amazon
+duovero_defconfig | omap4 | gumstix
+cl-som-am57x_defconfig | omap5 | compulab
+cm_t54_defconfig | omap5 | compulab
+
+"""
+
+import argparse
+import os
+import re
+
+
+class Board:
+
+    """A placeholder for properties read from boards.cfg"""
+
+    def __init__(self, status, arch, cpu, soc,
+                 vendor, board, target, options, maintainer):
+        self.status = status
+        self.arch = arch
+        self.cpu = cpu
+        self.soc = soc
+        self.vendor = vendor
+        self.board = board
+        self.target = target
+        self.defconfig = "{}_defconfig".format(target)
+        self.options = options
+        self.maintainer = maintainer
+
+    def show(self, sep=' | ', props=None):
+        """Show the properties of a board
+
+        Args:
+            sep: a separator insertyed between the properties for readability
+            props: the list of properties we want to display
+        """
+        if not props:
+            print(
+                sep.join([self.defconfig,
+                          self.vendor,
+                          self.arch,
+                          self.cpu,
+                          self.soc,
+                          self.board,
+                          self.status,
+                          self.maintainer]))
+        else:
+            print(sep.join([getattr(self, prop) for prop in props]))
+
+    def match(self, rules):
+        """ match a board against a set of rules
+
+        Args:
+            rules: a set of rules. a rule is an object that offers a 'match(str)'
+            method, usually a compiled regexp.
+        Returns:
+            True if the board matches all of the rules, False otherwise
+        """
+        for prop, r in rules:
+            val = getattr(self, prop)
+            if not val or val == "-":
+                return False
+            if not r.match(val):
+                return False
+        return True
+
+
+def get_all_boards():
+    """ extract a list of boards from 'boards.cfg'
+    If boards.cfg does not exit, it is generated with tools/genboardscfg.py
+    boards.cfg is parsed to create a list of Board objects
+
+    Returns:
+        A list of Board objects that have been created based on the information
+    found in boards.cfg
+    """
+
+    result = []
+    if not os.path.isfile("boards.cfg"):
+        os.system('tools/genboardscfg.py')
+
+    with open('boards.cfg', 'r') as f:
+        for l in f.readlines():
+            if not l or l[0] == "#":
+                continue
+            props = l.strip().split(None, 8)
+            if not props:
+                continue
+            if len(props) < 9:
+                props.extend(["-"] * (9 - len(props)))
+            result.append(Board(*props))
+    return result
+
+
+def get_default_options():
+    return ["board", "soc", "vendor", "arch", "cpu", "target"]
+
+
+def update_parser_with_default_options(parser):
+    """ Augment a command line parser with a list of options commonly used to
+    filter the targets (vendor, arch, cpu, soc, board, target)
+
+        Args:
+            parser: The parser to augment
+    """
+    parser.add_argument('-i', '--ignore-case', action="store_true")
+    parser.add_argument("--soc",
+                        help="regexp to filter on SoC. ex: 'omap[45]' to inspect omap5 and omap5 targets")
+    parser.add_argument("--vendor", help="regexp to filter on Vendor.")
+    parser.add_argument("--arch", help="regexp to filter on Arch")
+    parser.add_argument("--cpu", help="regexp to filter on CPU")
+    parser.add_argument("--board", help="regexp to filter on Board")
+    parser.add_argument("--target",
+                        help="regexp to filter on Target (defconfig filename without the '_defconfig' suffix)")
+
+
+def get_matching_boards(args, fields=get_default_options()):
+    """ Construct a list of boards matching criteria defined in args and
+    listed in fields
+
+    Args:
+        args: an object that has some members such as 'soc','cpu' etc.
+        The values of those members are strings that will be used to compile
+        a regexp object. It must have a boolean member named "ignore_case".
+        fields: a list of names of the members we are interested in.
+
+   Returns:
+       A list of Board objects matching the given criteria
+   """
+
+    # compile a list of regexp used to filter the targets
+    boards = []
+    rules = []
+    for f in fields:
+        arg = getattr(args, f)
+        if arg:
+            rules.append((f, re.compile("\\b{}\\b".format(arg),
+                         re.IGNORECASE if args.ignore_case else 0)))
+
+    # get a list of boards matching the rules
+    for b in get_all_boards():
+        if b.match(rules):
+            boards.append(b)
+    return boards
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Show CONFIG options usage")
+    update_parser_with_default_options(parser)
+    parser.add_argument("--maintainer", help="regexp to filter on maintainer.")
+    parser.add_argument("--status", help="regexp to filter on status.")
+    parser.add_argument("--show-details", help="show fields used as filter",
+                        action="store_true")
+    parser.add_argument("--show-all", help="show all fields",
+                        action="store_true")
+    args = parser.parse_args()
+    fields = get_default_options() + ["status", "maintainer"]
+
+    for b in get_matching_boards(args, fields):
+        if args.show_details:
+            props = ["defconfig"] + [f for f in fields if getattr(args, f)]
+            b.show(props=props)
+        elif args.show_all:
+            b.show()
+        else:
+            print(b.defconfig)
+
+if __name__ == '__main__':
+    main()
-- 
2.7.4

[U-Boot] [RFC PATCH v3 3/3] tools: Add a tool to get an overview of the usage of CONFIG options

[Jean-Jacques Hiblot]

configs2csv.py is tool that allow to check how some options are used for a
particular subset of platforms.
The purpose is to identify the targets that are actually using one or more
options of interest.
For example, it can tell what targets are still using CONFIG_DM_I2_COMPAT.
It relies on the config database produced by tools/moveconfig.py.
If the database doesn't exist, it will build it for the restricted set of
the selected platforms. Once the database is built, it is much faster than
greping the configs directory and more accurate as it relies on the
information found in u-boot.cfg instead of defconfigs.
It possible to look for options in the u-boot, the SPL or the TPL
configurations. It can also perform diffs between those configurations.

usage: configs2csv.py [-h] [-X] [--u-boot] [--spl] [--tpl] [--diff]
                      [--rebuild-db] [-j JOBS] [-o OUTPUT] [--no-header]
                      [--discard-empty] [-i] [--soc SOC] [--vendor VENDOR]
                      [--arch ARCH] [--cpu CPU] [--board BOARD]
                      [--target TARGET]
                      OPTION [OPTION ...]

all filtering parameters (OPTION, vendor, arch, ...) accept regexp.
ex: configs2csv.py .*DM_I2C.* --soc 'omap[2345]|k3' will match
CONFIG_DM_I2C and CONFIG_DM_I2C_COMPAT and look for it only for targets
using the omap2, omap3, omap4, omap5 or k3 SOCs.

Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>

---

Changes in v3:
- stylistics changes
- Add more comments to describe classes and functions

Changes in v2:
- basically rewrote the whole thing
- use tools/moveconfig.py to generate the database of configs
- use tools/find_defconfigs.py to get the list of defconfigs off interest
- removed diff with .config. tools/moveconfig.py does a better job

 tools/configs2csv.py | 427 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 427 insertions(+)
 create mode 100755 tools/configs2csv.py

diff --git a/tools/configs2csv.py b/tools/configs2csv.py
new file mode 100755
index 0000000..e2c4d53
--- /dev/null
+++ b/tools/configs2csv.py
@@ -0,0 +1,427 @@
+#!/usr/bin/env python
+# SPDX-License-Identifier: GPL-2.0+
+#
+# Author: JJ Hiblot <jjhiblot@ti.com>
+#
+
+"""
+scan the configuration of specified targets (ie defconfigs) and outputs a
+summary in a csv file.
+Useful tool to check what platform is using a particular set of options.
+
+
+How does it work?
+-----------------
+
+This tools uses the config database produced by tools/moveconfig.py (called
+with option -B to get all the configs: SPl, TPL and u-boot). If the database
+is not present, it will build it. A rebuild can be forced with the option
+'--rebuild-db'.
+
+The list of the targets of interest can be specified by a set of filter (soc,
+vendor, defconfig name, ..). All those filters are actually regexp, allowing
+for complex selection. The selection process is done by
+tools/find_defconfigs.py
+ex: --soc omap[23] --vendor 'ti|compulab' will inspect the omap2 and omap3
+platforms from TI and compulab
+
+
+examples:
+---------
+
+
+1) Get an overview of the usage of CONFIG_DM, CONFIG_SPL_DM, and DM/I2C related
+   options for platforms with omap5 or k3 SOC in u-boot and in SPL
+
+$ tools/configs2csv.py CONFIG_SPL_DM CONFIG_DM CONFIG_DM_I2C.* --vendor ti \
+       --soc 'omap5|k3' -X --u-boot --spl  -o dummy.csv
+
+vendor   soc            defconfig            type     CONFIG_DM  CONFIG_DM_I2C  CONFIG_DM_I2C_COMPAT  CONFIG_SPL_DM
+ti      omap5     am57xx_evm_defconfig        SPL         X                                               X
+ti      omap5     am57xx_evm_defconfig        u-boot      X         X                    X                X
+ti      omap5     am57xx_hs_evm_defconfig     SPL         X                                               X
+ti      omap5     am57xx_hs_evm_defconfig     u-boot      X         X                    X                X
+ti      k3        am65x_evm_a53_defconfig     SPL         X                                               X
+ti      k3        am65x_evm_a53_defconfig     u-boot      X                                               X
+ti      omap5     dra7xx_evm_defconfig        SPL         X                                               X
+ti      omap5     dra7xx_evm_defconfig        u-boot      X         X                    X                X
+ti      omap5     dra7xx_hs_evm_defconfig     SPL         X                                               X
+ti      omap5     dra7xx_hs_evm_defconfig     u-boot      X         X                    X                X
+ti      omap5     omap5_uevm_defconfig        SPL
+ti      omap5     omap5_uevm_defconfig        u-boot
+
+
+This shows quickly that DM is not supported at all for omap5_uevm, that
+only am65x_evm_a53 in not using DM_I2C in u-boot, and finally that DM_I2C is
+not enabled in the SPL for any platform although SPL_DM is.
+Also all the other platforms that enabled DM_I2C, also enabled
+CONFIG_DM_I2C_COMPAT.
+
+
+2) Check differences in config between SPL, TPL and u-boot (--diff option)
+
+Some platforms may disable/enable stuff in the configuration header files if
+in SPl. This makes it hard to know the usage of a variable by just looking at
+the .config. This is specially true for DM stuff.
+
+$ tools/configs2csv.py CONFIG\(_SPL\)?_DM_.*  --vendor ti \
+  --soc 'omap5|k3' --diff --spl --u-boot > dummy.csv
+
+vendor    soc        defconfig             CONFIG_DM_I2C CONFIG_DM_I2C_COMPAT  CONFIG_DM_STDIO  CONFIG_DM_WARN
+ti        omap5    am57xx_evm_defconfig       u-boot         u-boot                 u-boot           u-boot
+ti        omap5    am57xx_hs_evm_defconfig    u-boot         u-boot                 u-boot           u-boot
+ti        k3       am65x_evm_a53_defconfig                                          u-boot           u-boot
+ti        omap5    dra7xx_evm_defconfig       u-boot         u-boot                 u-boot           u-boot
+ti        omap5    dra7xx_hs_evm_defconfig    u-boot         u-boot                 u-boot           u-boot
+
+This shows that k3 has no real config diff between SPl and u-boot. whereas am57
+and dra7 have different settings for DM_I2C and DM_I2C_COMPAT
+
+"""
+
+import argparse
+import csv
+import os
+import re
+import sys
+from collections import namedtuple
+from itertools import combinations
+
+import find_defconfigs
+
+CONFIG_DATABASE = 'moveconfig.db'
+target = namedtuple("target", ["defconfig", "binary_type"])
+
+
+class TargetsDb:
+
+    """ TargetsDb is an object that store a collection of targets and their
+    CONFIG options
+    A target is identified by its defconfig and its binary type. ex:
+    (omap3_evm_defconfig,SPL)
+    The main purpose of this object is to output a CSV file that describes all
+    the targets.
+    There is also the possibility to create a "diff" TargetsDb from a
+    TargetsDb : this new TargetsDb contains a summary of the differences between
+    the targets built with the same defconfig (diff between the configs of SPL,
+    TPL and u-boot).
+    """
+
+    def __init__(self):
+        self.targets = dict()
+
+    def add_target(self, target):
+        self.targets[target] = dict()
+
+    def add_option(self, target, option, value):
+        self.targets[target][option] = value
+
+    def add_options(self, target, dic):
+        self.targets[target].update(dic)
+
+    def output_csv(
+            self, output, show_X=False, header=True, left_columns=None, discard_empty_rows=False):
+        """ write a CSV into a file. columns are the CONFIG options, rows are the targets
+        To make results easier to compare, the columns and rows are sorted alphabetically.
+
+        Args:
+            output: the file where to write the data
+            show_X: replace the value of the CONFIG option with a X is the option is set
+            header: output the CSV header (first row with the description of the columns)
+            left_columns: function to produce a description of the targets in the left columns
+            discard_empty_rows: If True, a row that has no CONFIG option set is not included in the CSV
+        """
+        all_options = set()
+        if len(self.targets) == 0:
+            return
+
+        if discard_empty_rows:
+            dic = {k: self.targets[k] for k in self.targets if self.targets[k]}
+        else:
+            dic = self.targets.copy()
+        for target in dic.keys():
+            for option in dic[target].keys():
+                all_options.add(option)
+                if show_X:
+                    dic[target][option] = "X"
+            if left_columns:
+                left_columns(target, dic, header=False)
+
+        columns = []
+        if left_columns:
+            columns.extend(left_columns(None, header=True))
+        columns.extend(sorted(all_options))
+
+        writer = csv.DictWriter(output, fieldnames=columns,
+                                lineterminator='\n')
+        if header:
+            writer.writeheader()
+        for target in sorted(dic.keys()):
+            writer.writerow(dic[target])
+
+    def diff_one_defconfig(self, defconfig):
+        """ This function creates a dictionary of the differences between the
+        binaries os a single target.
+        For example, for "dra7xx_evm_defconfig" it will compute the diffence
+        between the options used to compile u-boot and the SPL (not the TPL
+        because this platform doesn't have it).
+        The return value looks as follow: { 'CONFIG_DM_I2C: "u-boot",
+        CONFIG_SPL_BUILD:"SPL", CONFIG_DUMMY_SPI_FREQ: "diff" }.
+
+        The algorithm can probably be optimized, but I didn't care enough.
+        algo is:
+        - return immediately is there is only one binary type (u-boot)
+        - create a dic that is merge of the dic for all the binary types
+        - for each binary type, compare its dic to the merged_dic. If it is
+        different then break. It means that at least one option is different.
+        - if no difference has been found, then return
+        - at this point, we know that there is at least one diff. For each
+        binary types and for all options used for this binary type, check if it
+        is in the merged dic and, if so, if its value is the same. update our
+        return dic with the proper description.
+
+        Returns:
+            dict:
+                key: CONFIG option
+                value: a string that can be empty if there is no diff, or
+                "diff" if a difference exist between all the configs, or
+                a combination of "u-boot", "spl", "tpl" if the config does
+                not exit in all the binaries
+        """
+
+        diffs = dict()
+        diff_found = False
+
+        # get all binary types (spl, TPL, u-boot) generated by this defconfig
+        all_binary_types = sorted(
+            set([t.binary_type for t in self.targets.keys() if t.defconfig == defconfig]))
+
+        # If there is only one type of binary, no need to do a diff
+        if len(all_binary_types) <= 1:
+            return None
+
+        # create a dict with all options:values used by all binaries
+        merged_dic = dict()
+        for bin_type in all_binary_types:
+            merged_dic.update(self.targets[target(defconfig, bin_type)])
+
+        # check if all binaries have the same options (should be the case for
+        # most of the defconfigs)
+        for bin_type in all_binary_types:
+            if self.targets[target(defconfig, bin_type)] != merged_dic:
+                diff_found = True
+                break
+        if not diff_found:
+            return None
+
+        # at this point, we know that there are some options that differ
+        # between binaries (either not present or different)
+
+        # Get a list (actually a set) of the options that are different
+        differing_keys = set()
+        for bin_type in all_binary_types:
+            dic = self.targets[target(defconfig, bin_type)]
+            for opt, value in merged_dic.items():
+                if dic.get(opt, None) != value:
+                    differing_keys.add(opt)
+
+        # create a dictionary that summarize the differences
+        for bin_type in all_binary_types:
+            dic = self.targets[target(defconfig, bin_type)]
+            for opt in differing_keys:
+                dic_value = dic.get(opt, None)
+                merged_value = merged_dic.get(opt, None)
+                previous = diffs.get(opt, None)
+                if dic_value:
+                    if dic_value != merged_value:
+                        diffs[opt] = "diff"
+                    elif previous != "diff":
+                        diffs[opt] = ' / '.join(
+                            [previous, bin_type]) if previous else bin_type
+
+        return diffs
+
+    def diff(self):
+        """ create a new db that contains the differences between the binaries
+        for all the defconfig in the db
+
+        Returns:
+            a TargetsDb object that contains for each defconfig a dict
+            describing the differences between the SPL, TPL and u-boot
+            configurations.
+        """
+        diff_db = TargetsDb()
+        # get a list of all the defconfigs
+        all_defconfigs = set([t.defconfig for t in self.targets.keys()])
+        # for every defconfig of the list, get a dictionary of the differences.
+        # if the dictionary is not empty, add it the new db
+        for defconfig in all_defconfigs:
+            diff_dic = self.diff_one_defconfig(defconfig)
+            if diff_dic:
+                diff_db.add_target(target(defconfig, None))
+                diff_db.add_options(target(defconfig, None), diff_dic)
+        return diff_db
+
+
+def read_db(boards, binary_types, option_filter):
+    """Create a new TargetsDb object based on the content of moveconfig.db
+    and the filters passed in parameters
+
+    Args:
+        boards: a list of defconfig we want to add to the db
+        binary_types: a list of binary types (SPL, TPL, u-boot) we want to add
+        to the db
+        option_filter: a function to keep only some CONFIG options.
+
+    Returns:
+        A TargetsDb with the targets we are interested in (based on boards
+        and binary_types).
+"""
+
+    defconfig = ""
+    _db = TargetsDb()
+
+    # Read in the database
+    with open(CONFIG_DATABASE) as fd:
+        for line in fd.readlines():
+            line = line.rstrip()
+            if not line:  # Separator between defconfigs.
+                # We do not really care. We detect a new config by the absence
+                # of ' 'at the beginning of the line
+                pass
+            elif line[0] == ' ':  # CONFIG_xxx line
+                if t and option_filter(line):
+                    config, value = line.strip().split('=', 1)
+                    _db.add_option(t, config, value)
+            else:  # New defconfig
+                infos = line.split()
+                defconfig = infos[0]
+                try:
+                    binary_type = infos[1]
+                except:
+                    binary_type = "u-boot"
+                if binary_type in binary_types and defconfig in boards:
+                    t = target(defconfig, binary_type)
+                    _db.add_target(t)
+                else:
+                    t = None
+    return _db
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Show CONFIG options usage")
+    parser.add_argument("options", metavar='OPTION', type=str, nargs='+',
+                        help="regexp to filter on options.\
+        ex: CONFIG_DM_I2C_COMPAT or '.*DM_MMC.*'")
+    parser.add_argument(
+        "-X", help="show a X instead of the value of the option",
+                        action="store_true")
+    parser.add_argument("--u-boot", help="parse the u-boot configs",
+                        action="store_true")
+    parser.add_argument("--spl", help="parse the SPL configs",
+                        action="store_true")
+    parser.add_argument("--tpl", help="parse the TPL configs",
+                        action="store_true")
+    parser.add_argument("--diff",
+                        help="show only the options that differs between the selected configs (SPL, TPL, u-boot)",
+                        action="store_true")
+    parser.add_argument("--rebuild-db",
+                        help="Force a rebuild of the config database",
+                        action="store_true")
+    parser.add_argument('-j', '--jobs',
+                        help='the number of jobs to run simultaneously')
+    parser.add_argument('-o', '--output',
+                        help='The output CSV filename. uses stdout if not specified')
+    parser.add_argument('--no-header', help='Do not put the header at the top',
+                        action="store_true")
+    parser.add_argument('--discard-empty', action="store_true",
+                        help='Discard the empty rows (defconfigs that do not enable@least one option)')
+
+    find_defconfigs.update_parser_with_default_options(parser)
+    args = parser.parse_args()
+
+    # generate db file if needed or requested
+    # The job of generating the db is actually done by tools/moveconfig.py
+    # (called with -B)
+    if args.rebuild_db or not os.path.isfile(CONFIG_DATABASE):
+        find_defconfig_args = ["--{} '{}'".format(f, getattr(args, f))
+                               for f in find_defconfigs.get_default_options()
+                               if getattr(args, f)]
+        if args.jobs:
+            jobs_option = "-j {}".format(args.jobs)
+        else:
+            jobs_option = ""
+
+        rc = os.system(
+            "tools/find_defconfigs.py {} | tools/moveconfig.py -B {} -d - 1>&2 "
+            .format(" ".join(find_defconfig_args), jobs_option))
+        if rc:
+            sys.exit(1)
+
+    # get a list of defconfigs matching the rules
+    targets = [t for t in find_defconfigs.get_matching_boards(args)]
+    defconfigs = [t.defconfig for t in targets]
+
+    # create a list of binary types we are interested in
+    binary_types = []
+    if args.spl:
+        binary_types.append("SPL")
+    if args.tpl:
+        binary_types.append("TPL")
+    if args.u_boot or not binary_types:
+        binary_types.append("u-boot")
+
+    # define a function used to filter on the options
+    rules = [re.compile("   {}=".format(cfg_opt))
+             for cfg_opt in args.options]
+
+    def match_any_rule(line):
+        for r in rules:
+            if r.match(line):
+                return True
+        return False
+
+    # read the database
+    db = read_db(defconfigs, binary_types, match_any_rule)
+
+    target_dict = {}
+    for t in targets:
+        target_dict[t.defconfig] = t
+
+    def populate_left_columns(target=None, dic=None, header=True):
+        if header:
+            return ["vendor", "soc", "defconfig", "type"]
+        else:
+            dic[target]["vendor"] = target_dict[target.defconfig].vendor
+            dic[target]["soc"] = target_dict[target.defconfig].soc
+            dic[target]["defconfig"] = target.defconfig
+            dic[target]["type"] = target.binary_type
+
+    def populate_left_columns_diff(target=None, dic=None, header=True):
+        if header:
+            return ["vendor", "soc", "defconfig"]
+        else:
+            dic[target]["vendor"] = target_dict[target.defconfig].vendor
+            dic[target]["soc"] = target_dict[target.defconfig].soc
+            dic[target]["defconfig"] = target.defconfig
+
+    if args.output:
+        out = open(args.output, "w")
+    else:
+        out = sys.stdout
+
+    if args.diff:
+        db.diff().output_csv(output=out, show_X=False,
+                             header=not args.no_header,
+                             discard_empty_rows=args.discard_empty,
+                             left_columns=populate_left_columns_diff)
+    else:
+        db.output_csv(output=out, show_X=args.X, header=not args.no_header,
+                      discard_empty_rows=args.discard_empty,
+                      left_columns=populate_left_columns)
+
+    if out != sys.stdout:
+        out.close()
+
+if __name__ == '__main__':
+    main()
-- 
2.7.4

[U-Boot] [RFC PATCH v3 1/3] tools: moveconfig: Add an option to build a fuller database of options

[Simon Glass]

On 26 October 2018 at 05:14, Jean-Jacques Hiblot <jjhiblot@ti.com> wrote:
> "moveconfig -b" will build a database of config options based on the
> content of include/config/auto.conf that reflects the .config
>
> Add a new option '-B' that does essentially the same, except that it uses
> the content of u-boot.cfg, spl/u-boot.cfg and tpl/u-boot.cfg.
> This allows to get the options from .config AND the headers for all the
> possible binary types (u-boot, SPL and TPL)
>
> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
> Reviewed-by: Simon Glass <sjg@chromium.org>
> ---
>
> Changes in v3: None
> Changes in v2: New
>
>  tools/moveconfig.py | 56 ++++++++++++++++++++++++++++++++++++++++++++++-------
>  1 file changed, 49 insertions(+), 7 deletions(-)


Reviewed-by: Simon Glass <sjg@chromium.org>

[U-Boot] [RFC PATCH v3 2/3] tools: Add a tool to get a list of defconfigs based on filters

[Simon Glass]

On 26 October 2018 at 05:14, Jean-Jacques Hiblot <jjhiblot@ti.com> wrote:
> The possible filters are "arch", "vendor", "soc", "cpu" and "arch".
>
> The list of all the defconfigs is read from boards.cfg. If this file
> doesn't exist, then tools/genboardscfg.py is called to generate it.
>
> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
> ---
>
> Changes in v3:
> - Add more comments to describe the classes and functions
> - Capitalize the Class names
>
> Changes in v2: New
>
>  tools/find_defconfigs.py | 204 +++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 204 insertions(+)
>  create mode 100755 tools/find_defconfigs.py

Reviewed-by: Simon Glass <sjg@chromium.org>

[U-Boot] [RFC PATCH v3 3/3] tools: Add a tool to get an overview of the usage of CONFIG options

[Simon Glass]

On 26 October 2018 at 05:14, Jean-Jacques Hiblot <jjhiblot@ti.com> wrote:
> configs2csv.py is tool that allow to check how some options are used for a
> particular subset of platforms.
> The purpose is to identify the targets that are actually using one or more
> options of interest.
> For example, it can tell what targets are still using CONFIG_DM_I2_COMPAT.
> It relies on the config database produced by tools/moveconfig.py.
> If the database doesn't exist, it will build it for the restricted set of
> the selected platforms. Once the database is built, it is much faster than
> greping the configs directory and more accurate as it relies on the
> information found in u-boot.cfg instead of defconfigs.
> It possible to look for options in the u-boot, the SPL or the TPL
> configurations. It can also perform diffs between those configurations.
>
> usage: configs2csv.py [-h] [-X] [--u-boot] [--spl] [--tpl] [--diff]
>                       [--rebuild-db] [-j JOBS] [-o OUTPUT] [--no-header]
>                       [--discard-empty] [-i] [--soc SOC] [--vendor VENDOR]
>                       [--arch ARCH] [--cpu CPU] [--board BOARD]
>                       [--target TARGET]
>                       OPTION [OPTION ...]
>
> all filtering parameters (OPTION, vendor, arch, ...) accept regexp.
> ex: configs2csv.py .*DM_I2C.* --soc 'omap[2345]|k3' will match
> CONFIG_DM_I2C and CONFIG_DM_I2C_COMPAT and look for it only for targets
> using the omap2, omap3, omap4, omap5 or k3 SOCs.
>
> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
>
> ---
>
> Changes in v3:
> - stylistics changes
> - Add more comments to describe classes and functions
>
> Changes in v2:
> - basically rewrote the whole thing
> - use tools/moveconfig.py to generate the database of configs
> - use tools/find_defconfigs.py to get the list of defconfigs off interest
> - removed diff with .config. tools/moveconfig.py does a better job
>
>  tools/configs2csv.py | 427 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 427 insertions(+)
>  create mode 100755 tools/configs2csv.py

Reviewed-by: Simon Glass <sjg@chromium.org>