* [U-Boot] [RFC PATCH v2 0/3] python tools to inspect configs
@ 2018-10-03 13:53 Jean-Jacques Hiblot
2018-10-03 13:53 ` [U-Boot] [RFC PATCH v2 1/3] tools: moveconfig: Add an option to build a fuller database of options Jean-Jacques Hiblot
` (2 more replies)
0 siblings, 3 replies; 10+ messages in thread
From: Jean-Jacques Hiblot @ 2018-10-03 13:53 UTC (permalink / raw)
To: u-boot
This series introduce 2 new python tools that helps getting an overview
of the configuration options.
First one is fairly simple and is used to locate deconfigs based on several
criteria: 'arch', 'soc', 'cpu', 'vendor', 'board', 'defconfig name',
'maintainer' and 'status'. All the parameters use regexp.
ex: Show all defconfigs of platforms built around imx25 or imx27 or imxs
$ tools/find_defconfigs.py --soc 'mx(25|27|s)'
The 2nd tool is used to produce a CSV file that summarizes the usage of
config options by a set of defconfigs. Useful to check what platforms might
need modifications when working on a particular option.
ex:
Get the TI platforms that enable CONFIG_DM_I2C_COMPAT either in u-boot or
the SPL
$ tools/configs2csv.py -X CONFIG_DM_I2C_COMPAT --spl --u-boot --vendor ti \
--discard-empty
limitations:
- must be executed at the root of the source tree
- the source tree must be clean (make mrproper)
- only supports CSV format. visualization in terminal sould be added.
But LibreOffice calc is better suited with all its ordering/
filtering capabilities.
Changes in v2:
- basically rewrote the whole thing
- use tools/moveconfig.py to generate the database of configs
- use tools/find_defconfigs.py to get the list of defconfigs off interest
- removed diff with .config. tools/moveconfig.py does a better job
Jean-Jacques Hiblot (3):
tools: moveconfig: Add an option to build a fuller database of options
tools: Add a tool to get a list of defconfigs based on filters
tools: Add a tool to get an overview of the usage of CONFIG options
tools/configs2csv.py | 387 +++++++++++++++++++++++++++++++++++++++++++++++
tools/find_defconfigs.py | 167 ++++++++++++++++++++
tools/moveconfig.py | 56 ++++++-
3 files changed, 603 insertions(+), 7 deletions(-)
create mode 100755 tools/configs2csv.py
create mode 100755 tools/find_defconfigs.py
--
2.7.4
^ permalink raw reply [flat|nested] 10+ messages in thread
* [U-Boot] [RFC PATCH v2 1/3] tools: moveconfig: Add an option to build a fuller database of options
2018-10-03 13:53 [U-Boot] [RFC PATCH v2 0/3] python tools to inspect configs Jean-Jacques Hiblot
@ 2018-10-03 13:53 ` Jean-Jacques Hiblot
2018-10-09 16:20 ` Simon Glass
2018-10-03 13:53 ` [U-Boot] [RFC PATCH v2 2/3] tools: Add a tool to get a list of defconfigs based on filters Jean-Jacques Hiblot
2018-10-03 13:53 ` [U-Boot] [RFC PATCH v2 3/3] tools: Add a tool to get an overview of the usage of CONFIG options Jean-Jacques Hiblot
2 siblings, 1 reply; 10+ messages in thread
From: Jean-Jacques Hiblot @ 2018-10-03 13:53 UTC (permalink / raw)
To: u-boot
"moveconfig -b" will build a database of config options based on the
content of include/config/auto.conf that reflects the .config
Add a new option '-B' that does essentially the same, except that it uses
the content of u-boot.cfg, spl/u-boot.cfg and tpl/u-boot.cfg.
This allows to get the options from .config AND the headers for all the
possible binary types (u-boot, SPL and TPL)
Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
---
Changes in v2: None
tools/moveconfig.py | 56 ++++++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 49 insertions(+), 7 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index caa81ac..85c2b8b 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -1151,9 +1151,17 @@ class Slot:
self.ps.stderr.read())
self.finish(False)
+ def remove_cfg_files(self):
+ for base in ['.', 'spl', 'tpl']:
+ try:
+ os.remove(os.path.join(self.build_dir, base, "u-boot.cfg"))
+ except:
+ pass
+
def do_defconfig(self):
"""Run 'make <board>_defconfig' to create the .config file."""
-
+ # first remove old cfg files that may have been produced earlier
+ self.remove_cfg_files()
cmd = list(self.make_cmd)
cmd.append(self.defconfig)
self.ps = subprocess.Popen(cmd, stdout=self.devnull,
@@ -1182,15 +1190,43 @@ class Slot:
cwd=self.current_src_dir)
self.state = STATE_AUTOCONF
+
def do_build_db(self):
"""Add the board to the database"""
- configs = {}
- with open(os.path.join(self.build_dir, AUTO_CONF_PATH)) as fd:
- for line in fd.readlines():
+ def conf_to_dic(filename):
+ configs = {}
+ if not os.path.isfile(filename):
+ return None
+
+ try:
+ if filename.endswith("u-boot.cfg"):
+ with tempfile.NamedTemporaryFile(delete = True) as fd:
+ lines = subprocess.check_call("sed -n -f tools/scripts/define2mk.sed {}".format(filename).split(), stdout = fd)
+ fd.seek(0)
+ lines = fd.readlines()
+ else:
+ with open(filename, "r") as fd:
+ lines = fd.readlines()
+ except:
+ return None
+
+ for line in lines:
if line.startswith('CONFIG'):
config, value = line.split('=', 1)
configs[config] = value.rstrip()
- self.db_queue.put([self.defconfig, configs])
+ return configs
+
+ list_of_conf = []
+ if self.options.build_full_db:
+ list_of_conf.append((self.defconfig,"u-boot.cfg"))
+ list_of_conf.append(("{} SPL".format(self.defconfig),"spl/u-boot.cfg"))
+ list_of_conf.append(("{} TPL".format(self.defconfig),"tpl/u-boot.cfg"))
+ else:
+ list_of_conf.append((self.defconfig,AUTO_CONF_PATH))
+ for name,conf_file in list_of_conf:
+ configs = conf_to_dic(os.path.join(self.build_dir, conf_file))
+ if configs:
+ self.db_queue.put([name, configs])
self.finish(True)
def do_savedefconfig(self):
@@ -1770,7 +1806,9 @@ def main():
help="don't show options which are already marked as "
'implying others')
parser.add_option('-b', '--build-db', action='store_true', default=False,
- help='build a CONFIG database')
+ help='build a CONFIG database based only on auto.conf')
+ parser.add_option('-B', '--build-full-db', action='store_true', default=False,
+ help='build a CONFIG database based only on u-boot.cfg (and also for SPL and TPL)')
parser.add_option('-c', '--color', action='store_true', default=False,
help='display the log in color')
parser.add_option('-C', '--commit', action='store_true', default=False,
@@ -1807,6 +1845,9 @@ def main():
(options, configs) = parser.parse_args()
+ if options.build_full_db:
+ options.build_db = True
+
if len(configs) == 0 and not any((options.force_sync, options.build_db,
options.imply)):
parser.print_usage()
@@ -1875,7 +1916,8 @@ def main():
if options.build_db:
with open(CONFIG_DATABASE, 'w') as fd:
- for defconfig, configs in config_db.iteritems():
+ for defconfig in sorted(config_db.keys()):
+ configs = config_db[defconfig]
fd.write('%s\n' % defconfig)
for config in sorted(configs.keys()):
fd.write(' %s=%s\n' % (config, configs[config]))
--
2.7.4
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [U-Boot] [RFC PATCH v2 2/3] tools: Add a tool to get a list of defconfigs based on filters
2018-10-03 13:53 [U-Boot] [RFC PATCH v2 0/3] python tools to inspect configs Jean-Jacques Hiblot
2018-10-03 13:53 ` [U-Boot] [RFC PATCH v2 1/3] tools: moveconfig: Add an option to build a fuller database of options Jean-Jacques Hiblot
@ 2018-10-03 13:53 ` Jean-Jacques Hiblot
2018-10-09 16:20 ` Simon Glass
2018-10-03 13:53 ` [U-Boot] [RFC PATCH v2 3/3] tools: Add a tool to get an overview of the usage of CONFIG options Jean-Jacques Hiblot
2 siblings, 1 reply; 10+ messages in thread
From: Jean-Jacques Hiblot @ 2018-10-03 13:53 UTC (permalink / raw)
To: u-boot
The possible filters are "arch", "vendor", "soc", "cpu" and "arch".
The list of all the defconfigs is read from boards.cfg. If this file
doesn't exist, then tools/genboardscfg.py is called to generate it.
Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
---
Changes in v2: None
tools/find_defconfigs.py | 167 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 167 insertions(+)
create mode 100755 tools/find_defconfigs.py
diff --git a/tools/find_defconfigs.py b/tools/find_defconfigs.py
new file mode 100755
index 0000000..9d68cef
--- /dev/null
+++ b/tools/find_defconfigs.py
@@ -0,0 +1,167 @@
+#!/usr/bin/env python
+# SPDX-License-Identifier: GPL-2.0+
+#
+# Author: JJ Hiblot <jjhiblot@ti.com>
+#
+
+"""
+Output a list of defconfig matching criteria.
+
+The possible criteria are soc, vendor, arch, cpu, board and defconfig name.
+The criteria are expressed as regexp, allowing for complex selection.
+
+How does it work?
+-----------------
+
+This tools uses the boards.cfg file produced by tools/genboardscfg.py
+It reads the file to get a list of all the defconfigs and the information
+about the soc, vendor etc. for each of them.
+Then it walks this list and outputs the defconfigs for which the info match
+the regexp passed to the program.
+
+examples:
+---------
+
+1) Get the list of defconfigs for boards built around omap5, omap4 and k3, not built by TI
+
+$ tools/find_defconfigs.py --soc 'omap[45]|k3' --vendor '(?!ti)'
+kc1_defconfig
+duovero_defconfig
+cl-som-am57x_defconfig
+cm_t54_defconfig
+
+2) Same list but iwth more details on the items that were used as filters
+
+$ tools/find_defconfigs.py --soc 'omap[45]|k3' --vendor '(?!ti)' --show-details
+kc1_defconfig | omap4 | amazon
+duovero_defconfig | omap4 | gumstix
+cl-som-am57x_defconfig | omap5 | compulab
+cm_t54_defconfig | omap5 | compulab
+
+
+"""
+
+import re
+import os
+import argparse
+
+
+class board:
+
+ def __init__(self, status, arch, cpu, soc,
+ vendor, board, target, options, maintainer):
+ self.status = status
+ self.arch = arch
+ self.cpu = cpu
+ self.soc = soc
+ self.vendor = vendor
+ self.board = board
+ self.target = target
+ self.defconfig = "{}_defconfig".format(target)
+ self.options = options
+ self.maintainer = maintainer
+
+ def show(self, sep=' | ', props=None):
+ if not props:
+ print(
+ sep.join([self.defconfig,
+ self.vendor,
+ self.arch,
+ self.cpu,
+ self.soc,
+ self.board,
+ self.status,
+ self.maintainer]))
+ else:
+ print(sep.join([self.defconfig] + [getattr(self, prop) for prop in props]))
+
+ def cleanup(self):
+ """ remove the directory in which the cfg files have been built """
+ shutil.rmtree(self.temp_dir)
+
+ def match(self, rules):
+ """ return True if the board match all the criteria """
+ for prop, r in rules:
+ val = getattr(self, prop)
+ if not val or val == "-":
+ return False
+ if not r.match(val):
+ return False
+ return True
+
+
+def get_all_boards():
+ """ extract a list of boards from 'boards.cfg' """
+ result = []
+ if not os.path.isfile("boards.cfg"):
+ os.system('tools/genboardscfg.py')
+
+ with open('boards.cfg', 'r') as f:
+ for l in f.readlines():
+ if not l or l[0] == "#":
+ continue
+ props = l.strip().split(None, 8)
+ if not props:
+ continue
+ if len(props) < 9:
+ props.extend(["-"] * (9 - len(props)))
+ result.append(board(*props))
+ return result
+
+
+def get_default_options():
+ return ["board", "soc", "vendor", "arch", "cpu", "target"]
+
+
+def update_parser_with_default_options(parser):
+ parser.add_argument('-i', '--ignore-case', action="store_true")
+ parser.add_argument("--soc",
+ help="regexp to filter on SoC. ex: 'omap[45]' to inspect omap5 and omap5 targets")
+ parser.add_argument("--vendor", help="regexp to filter on Vendor.")
+ parser.add_argument("--arch", help="regexp to filter on Arch")
+ parser.add_argument("--cpu", help="regexp to filter on CPU")
+ parser.add_argument("--board", help="regexp to filter on Board")
+ parser.add_argument("--target",
+ help="regexp to filter on Target (defconfig filename without the '_defconfig' suffix)")
+
+
+def get_matching_boards(args, fields=get_default_options()):
+ # compile a list of regexp used to filter the targets
+ boards = []
+ rules = []
+ for f in fields:
+ arg = getattr(args, f)
+ if arg:
+ rules.append((f, re.compile("\\b{}\\b".format(arg),
+ re.IGNORECASE if args.ignore_case else 0)))
+
+ # get a list of boards matching the rules
+ for b in get_all_boards():
+ if b.match(rules):
+ boards.append(b)
+ return boards
+
+
+def main():
+ parser = argparse.ArgumentParser(description="Show CONFIG options usage")
+ update_parser_with_default_options(parser)
+ parser.add_argument("--maintainer", help="regexp to filter on maintainer.")
+ parser.add_argument("--status", help="regexp to filter on status.")
+ parser.add_argument("--show-details", help="show fields used as filter",
+ action="store_true")
+ parser.add_argument("--show-all", help="show all fields",
+ action="store_true")
+ args = parser.parse_args()
+ fields = get_default_options() + ["status", "maintainer"]
+
+ for b in get_matching_boards(args, fields):
+ if args.show_details:
+ props = [f for f in fields if getattr(args, f)]
+ b.show(props=props)
+ elif args.show_all:
+ b.show()
+ else:
+ print(b.defconfig)
+
+if __name__ == '__main__':
+ main()
--
2.7.4
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [U-Boot] [RFC PATCH v2 3/3] tools: Add a tool to get an overview of the usage of CONFIG options
2018-10-03 13:53 [U-Boot] [RFC PATCH v2 0/3] python tools to inspect configs Jean-Jacques Hiblot
2018-10-03 13:53 ` [U-Boot] [RFC PATCH v2 1/3] tools: moveconfig: Add an option to build a fuller database of options Jean-Jacques Hiblot
2018-10-03 13:53 ` [U-Boot] [RFC PATCH v2 2/3] tools: Add a tool to get a list of defconfigs based on filters Jean-Jacques Hiblot
@ 2018-10-03 13:53 ` Jean-Jacques Hiblot
2018-10-09 16:20 ` Simon Glass
2 siblings, 1 reply; 10+ messages in thread
From: Jean-Jacques Hiblot @ 2018-10-03 13:53 UTC (permalink / raw)
To: u-boot
configs2csv.py is tool that allow to check how some options are used for a
particular subset of platforms.
The purpose is to identify the targets that are actually using one or more
options of interest.
For example, it can tell what targets are still using CONFIG_DM_I2_COMPAT.
It relies on the config database produced by tools/moveconfig.py.
If the database doesn't exist, it will build it for the restricted set of
the selected platforms. Once the database is built, it is much faster than
greping the configs directory and more accurate as it relies on the
information found in u-boot.cfg instead of defconfigs.
It possible to look for options in the u-boot, the SPL or the TPL
configurations. It can also perform diffs between those configurations.
usage: configs2csv.py [-h] [-X] [--u-boot] [--spl] [--tpl] [--diff]
[--rebuild-db] [-j JOBS] [-o OUTPUT] [--no-header]
[--discard-empty] [-i] [--soc SOC] [--vendor VENDOR]
[--arch ARCH] [--cpu CPU] [--board BOARD]
[--target TARGET]
OPTION [OPTION ...]
all filtering parameters (OPTION, vendor, arch, ...) accept regexp.
ex: configs2csv.py .*DM_I2C.* --soc 'omap[2345]|k3' will match
CONFIG_DM_I2C and CONFIG_DM_I2C_COMPAT and look for it only for targets
using the omap2, omap3, omap4, omap5 or k3 SOCs.
Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
---
Changes in v2:
- basically rewrote the whole thing
- use tools/moveconfig.py to generate the database of configs
- use tools/find_defconfigs.py to get the list of defconfigs off interest
- removed diff with .config. tools/moveconfig.py does a better job
tools/configs2csv.py | 387 +++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 387 insertions(+)
create mode 100755 tools/configs2csv.py
diff --git a/tools/configs2csv.py b/tools/configs2csv.py
new file mode 100755
index 0000000..70b6602
--- /dev/null
+++ b/tools/configs2csv.py
@@ -0,0 +1,387 @@
+#!/usr/bin/env python
+# SPDX-License-Identifier: GPL-2.0+
+#
+# Author: JJ Hiblot <jjhiblot@ti.com>
+#
+
+"""
+scan the configuration of specified targets (ie defconfigs) and outputs a
+summary in a csv file.
+Useful tool to check what platform is using a particular set of options.
+
+
+How does it work?
+-----------------
+
+This tools uses the config database produced by tools/moveconfig.py (called
+with option -B to get all the configs: SPl, TPL and u-boot). If the database
+is not present, it will build it. A rebuild can be forced with the option
+'--rebuild-db'.
+
+The list of the targets of interest can be specified by a set of filter (soc,
+vendor, defconfig name, ..). All those filters are actually regexp, allowing
+for complex selection. The selection process is done by
+tools/find_defconfigs.py
+ex: --soc omap[23] --vendor 'ti|compulab' will inspect the omap2 and omap3
+platforms from TI and compulab
+
+
+examples:
+---------
+
+
+1) Get an overview of the usage of CONFIG_DM, CONFIG_SPL_DM, and DM/I2C related
+ options for platforms with omap5 or k3 SOC in u-boot and in SPL
+
+$ tools/configs2csv.py CONFIG_SPL_DM CONFIG_DM CONFIG_DM_I2C.* --vendor ti \
+ --soc 'omap5|k3' -X --u-boot --spl -o dummy.csv
+
+vendor soc defconfig type CONFIG_DM CONFIG_DM_I2C CONFIG_DM_I2C_COMPAT CONFIG_SPL_DM
+ti omap5 am57xx_evm_defconfig SPL X X
+ti omap5 am57xx_evm_defconfig u-boot X X X X
+ti omap5 am57xx_hs_evm_defconfig SPL X X
+ti omap5 am57xx_hs_evm_defconfig u-boot X X X X
+ti k3 am65x_evm_a53_defconfig SPL X X
+ti k3 am65x_evm_a53_defconfig u-boot X X
+ti omap5 dra7xx_evm_defconfig SPL X X
+ti omap5 dra7xx_evm_defconfig u-boot X X X X
+ti omap5 dra7xx_hs_evm_defconfig SPL X X
+ti omap5 dra7xx_hs_evm_defconfig u-boot X X X X
+ti omap5 omap5_uevm_defconfig SPL
+ti omap5 omap5_uevm_defconfig u-boot
+
+
+This shows quickly that DM is not supported at all for omap5_uevm, that
+only am65x_evm_a53 in not using DM_I2C in u-boot, and finally that DM_I2C is
+not enabled in the SPL for any platform although SPL_DM is.
+Also all the other platforms that enabled DM_I2C, also enabled
+CONFIG_DM_I2C_COMPAT.
+
+
+2) Check differences in config between SPL, TPL and u-boot (--diff option)
+
+Some platforms may disable/enable stuff in the configuration header files if
+in SPl. This makes it hard to know the usage of a variable by just looking at
+the .config. This is specially true for DM stuff.
+
+$ tools/configs2csv.py CONFIG\(_SPL\)?_DM_.* --vendor ti \
+ --soc 'omap5|k3' --diff --spl --u-boot > dummy.csv
+
+vendor soc defconfig CONFIG_DM_I2C CONFIG_DM_I2C_COMPAT CONFIG_DM_STDIO CONFIG_DM_WARN
+ti omap5 am57xx_evm_defconfig u-boot u-boot u-boot u-boot
+ti omap5 am57xx_hs_evm_defconfig u-boot u-boot u-boot u-boot
+ti k3 am65x_evm_a53_defconfig u-boot u-boot
+ti omap5 dra7xx_evm_defconfig u-boot u-boot u-boot u-boot
+ti omap5 dra7xx_hs_evm_defconfig u-boot u-boot u-boot u-boot
+
+This shows that k3 has no real config diff between SPl and u-boot. whereas am57
+and dra7 have different settings for DM_I2C and DM_I2C_COMPAT
+
+"""
+
+import argparse
+import csv
+import os
+import re
+import sys
+from collections import namedtuple
+from itertools import combinations
+
+import find_defconfigs
+
+CONFIG_DATABASE = 'moveconfig.db'
+target = namedtuple("target", ["defconfig", "binary_type"])
+
+
+class db:
+
+ """ db is an object that store a collection of targets
+ a target is identified buy its defconfig and its binary type. ex:
+ (omap3_evm_defconfig,SPL)
+ The main purpose of this object is to output a CSV file that describes all
+ the targets.
+ There is also the possibility to create a "diff" db from a db. This new db
+ contains a summary of the differences between target of same defconfig.
+ """
+
+ def __init__(self):
+ self.targets = dict()
+
+ def add_target(self, target):
+ self.targets[target] = dict()
+
+ def add_option(self, target, option, value):
+ self.targets[target][option] = value
+
+ def add_options(self, target, dic):
+ self.targets[target].update(dic)
+
+ def output_csv(
+ self, output, show_X=False, header=True, left_columns=None, discard_empty_rows=False):
+ all_options = set()
+ if len(self.targets) == 0:
+ return
+
+ if discard_empty_rows:
+ dic = {k: self.targets[k] for k in self.targets if self.targets[k]}
+ else:
+ dic = self.targets.copy()
+ for target in dic.keys():
+ for option in dic[target].keys():
+ all_options.add(option)
+ if show_X:
+ dic[target][option] = "X"
+ if left_columns:
+ left_columns(target, dic, header=False)
+
+ columns = []
+ if left_columns:
+ columns.extend(left_columns(None, header=True))
+ columns.extend(sorted(all_options))
+
+ writer = csv.DictWriter(output, fieldnames=columns,
+ lineterminator='\n')
+ if header:
+ writer.writeheader()
+ for target in sorted(dic.keys()):
+ writer.writerow(dic[target])
+
+ def diff_one_defconfig(self, defconfig):
+ """ This function creates a dictionary of the differences between the
+ binaries os a single target.
+ For example, for "dra7xx_evm_defconfig" it will compute the diffence
+ between the options used to compile u-boot and the SPL (not the TPL
+ because this platform doesn't have it).
+ The return value looks as follow: { 'CONFIG_DM_I2C: "u-boot",
+ CONFIG_SPL_BUILD:"SPL", CONFIG_DUMMY_SPI_FREQ: "diff" }.
+
+ The algorithm can probably be optimized, but I didn't care enough.
+ algo is:
+ - return immediately is there is only one binary type (u-boot)
+ - create a dic that is merge of the dic for all the binary types
+ - for each binary type, compare its dic to the merged_dic. If it is
+ different then break. It means that at least one option is different.
+ - if no difference has been found, then return
+ - at this point, we know that there is at least one diff. For each
+ binary types and for all options used for this binary type, check if it
+ is in the merged dic and, if so, if its value is the same. update our
+ return dic with the proper description.
+
+ """
+
+ diffs = dict()
+ diff_found = False
+
+ # get all binary types (spl, TPL, u-boot) generated by this defconfig
+ all_binary_types = sorted(
+ set([t.binary_type for t in self.targets.keys() if t.defconfig == defconfig]))
+
+ # If there is only one type of binary, no need to do a diff
+ if len(all_binary_types) <= 1:
+ return None
+
+ # create a dict with all options:values used by all binaries
+ merged_dic = dict()
+ for bin_type in all_binary_types:
+ merged_dic.update(self.targets[target(defconfig, bin_type)])
+
+ # check if all binaries have the same options (should be the case for
+ # most of the defconfigs)
+ for bin_type in all_binary_types:
+ if self.targets[target(defconfig, bin_type)] != merged_dic:
+ diff_found = True
+ break
+ if not diff_found:
+ return None
+
+ # at this point, we know that there are some options that differ
+ # between binaries (either not present or different)
+
+ # Get a list (actually a set) of the options that are different
+ differing_keys = set()
+ for bin_type in all_binary_types:
+ dic = self.targets[target(defconfig, bin_type)]
+ for opt, value in merged_dic.items():
+ if dic.get(opt, None) != value:
+ differing_keys.add(opt)
+
+ # create a dictionary that summarize the differences
+ for bin_type in all_binary_types:
+ dic = self.targets[target(defconfig, bin_type)]
+ for opt in differing_keys:
+ dic_value = dic.get(opt, None)
+ merged_value = merged_dic.get(opt, None)
+ previous = diffs.get(opt, None)
+ if dic_value:
+ if dic_value != merged_value:
+ diffs[opt] = "diff"
+ elif previous != "diff":
+ diffs[opt] = ' / '.join(
+ [previous, bin_type]) if previous else bin_type
+
+ return diffs
+
+ def diff(self):
+ """ create a new db that contains the differences between the binaries
+ for all the targets in the db """
+ diff_db = db()
+ # get a list of all the targets
+ all_defconfigs = set([t.defconfig for t in self.targets.keys()])
+ # for every target of the list, get a dictionary of the difference.
+ # if the dictionary is not empty, add it the new db
+ for defconfig in all_defconfigs:
+ diff_dic = self.diff_one_defconfig(defconfig)
+ if diff_dic:
+ diff_db.add_target(target(defconfig, None))
+ diff_db.add_options(target(defconfig, None), diff_dic)
+ return diff_db
+
+
+def read_db(boards, option_filter, binary_types):
+ defconfig = ""
+ _db = db()
+
+ # Read in the database
+ with open(CONFIG_DATABASE) as fd:
+ for line in fd.readlines():
+ line = line.rstrip()
+ if not line: # Separator between defconfigs.
+ # We do not really care. We detect a new config by the absence
+ # of ' 'at the beginning of the line
+ pass
+ elif line[0] == ' ': # CONFIG_xxx line
+ if t and option_filter(line):
+ config, value = line.strip().split('=', 1)
+ _db.add_option(t, config, value)
+ else: # New defconfig
+ infos = line.split()
+ defconfig = infos[0]
+ try:
+ binary_type = infos[1]
+ except:
+ binary_type = "u-boot"
+ if binary_type in binary_types and defconfig in boards:
+ t = target(defconfig, binary_type)
+ _db.add_target(t)
+ else:
+ t = None
+ return _db
+
+
+def main():
+ parser = argparse.ArgumentParser(description="Show CONFIG options usage")
+ parser.add_argument("options", metavar='OPTION', type=str, nargs='+',
+ help="regexp to filter on options.\
+ ex: CONFIG_DM_I2C_COMPAT or '.*DM_MMC.*'")
+ parser.add_argument(
+ "-X", help="show a X instead of the value of the option",
+ action="store_true")
+ parser.add_argument("--u-boot", help="parse the u-boot configs",
+ action="store_true")
+ parser.add_argument("--spl", help="parse the SPL configs",
+ action="store_true")
+ parser.add_argument("--tpl", help="parse the TPL configs",
+ action="store_true")
+ parser.add_argument("--diff",
+ help="show only the options that differs between the selected configs (SPL, TPL, u-boot)",
+ action="store_true")
+ parser.add_argument("--rebuild-db",
+ help="Force a rebuild of the config database",
+ action="store_true")
+ parser.add_argument('-j', '--jobs',
+ help='the number of jobs to run simultaneously')
+ parser.add_argument('-o', '--output',
+ help='The output CSV filename. uses stdout if not specified')
+ parser.add_argument('--no-header', help='Do not put the header at the top',
+ action="store_true")
+ parser.add_argument('--discard-empty', action="store_true",
+ help='Discard the empty rows (defconfigs that do not enable@least one option)')
+
+ find_defconfigs.update_parser_with_default_options(parser)
+ args = parser.parse_args()
+
+ # generate db file if needed or requested
+ # The job of generating the db is actually done by tools/moveconfig.py
+ # (called with -B)
+ if args.rebuild_db or not os.path.isfile(CONFIG_DATABASE):
+ find_defconfig_args = ["--{} '{}'".format(f, getattr(args, f))
+ for f in find_defconfigs.get_default_options()
+ if getattr(args, f)]
+ if args.jobs:
+ jobs_option = "-j {}".format(args.jobs)
+ else:
+ jobs_option = ""
+
+ rc = os.system(
+ "tools/find_defconfigs.py {} | tools/moveconfig.py -B {} -d - 1>&2 "
+ .format(" ".join(find_defconfig_args), jobs_option))
+ if rc:
+ sys.exit(1)
+
+ # get a list of defconfigs matching the rules
+ targets = [t for t in find_defconfigs.get_matching_boards(args)]
+ defconfigs = [t.defconfig for t in targets]
+
+ # create a list of binary types we are interested in
+ binary_types = []
+ if args.spl:
+ binary_types.append("SPL")
+ if args.tpl:
+ binary_types.append("TPL")
+ if args.u_boot or not binary_types:
+ binary_types.append("u-boot")
+
+ # define a function used to filter on the options
+ rules = [re.compile(" {}=".format(cfg_opt))
+ for cfg_opt in args.options]
+
+ def match_any_rule(line):
+ for r in rules:
+ if r.match(line):
+ return True
+ return False
+
+ # read the database
+ db = read_db(defconfigs, match_any_rule, binary_types)
+
+ target_dict = {}
+ for t in targets:
+ target_dict[t.defconfig] = t
+
+ def populate_left_columns(target=None, dic=None, header=True):
+ if header:
+ return ["vendor", "soc", "defconfig", "type"]
+ else:
+ dic[target]["vendor"] = target_dict[target.defconfig].vendor
+ dic[target]["soc"] = target_dict[target.defconfig].soc
+ dic[target]["defconfig"] = target.defconfig
+ dic[target]["type"] = target.binary_type
+
+ def populate_left_columns_diff(target=None, dic=None, header=True):
+ if header:
+ return ["vendor", "soc", "defconfig"]
+ else:
+ dic[target]["vendor"] = target_dict[target.defconfig].vendor
+ dic[target]["soc"] = target_dict[target.defconfig].soc
+ dic[target]["defconfig"] = target.defconfig
+
+ if args.output:
+ out = open(args.output, "w")
+ else:
+ out = sys.stdout
+
+ if args.diff:
+ db.diff().output_csv(output=out, show_X=False,
+ header=not args.no_header,
+ discard_empty_rows=args.discard_empty,
+ left_columns=populate_left_columns_diff)
+ else:
+ db.output_csv(output=out, show_X=args.X, header=not args.no_header,
+ discard_empty_rows=args.discard_empty,
+ left_columns=populate_left_columns)
+
+ if out != sys.stdout:
+ out.close()
+
+if __name__ == '__main__':
+ main()
--
2.7.4
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [U-Boot] [RFC PATCH v2 3/3] tools: Add a tool to get an overview of the usage of CONFIG options
2018-10-03 13:53 ` [U-Boot] [RFC PATCH v2 3/3] tools: Add a tool to get an overview of the usage of CONFIG options Jean-Jacques Hiblot
@ 2018-10-09 16:20 ` Simon Glass
2018-10-18 11:38 ` Jean-Jacques Hiblot
0 siblings, 1 reply; 10+ messages in thread
From: Simon Glass @ 2018-10-09 16:20 UTC (permalink / raw)
To: u-boot
Hi Jean-Jacques,
On 3 October 2018 at 07:53, Jean-Jacques Hiblot <jjhiblot@ti.com> wrote:
> configs2csv.py is tool that allow to check how some options are used for a
> particular subset of platforms.
> The purpose is to identify the targets that are actually using one or more
> options of interest.
> For example, it can tell what targets are still using CONFIG_DM_I2_COMPAT.
> It relies on the config database produced by tools/moveconfig.py.
> If the database doesn't exist, it will build it for the restricted set of
> the selected platforms. Once the database is built, it is much faster than
> greping the configs directory and more accurate as it relies on the
> information found in u-boot.cfg instead of defconfigs.
> It possible to look for options in the u-boot, the SPL or the TPL
> configurations. It can also perform diffs between those configurations.
>
> usage: configs2csv.py [-h] [-X] [--u-boot] [--spl] [--tpl] [--diff]
> [--rebuild-db] [-j JOBS] [-o OUTPUT] [--no-header]
> [--discard-empty] [-i] [--soc SOC] [--vendor VENDOR]
> [--arch ARCH] [--cpu CPU] [--board BOARD]
> [--target TARGET]
> OPTION [OPTION ...]
>
> all filtering parameters (OPTION, vendor, arch, ...) accept regexp.
> ex: configs2csv.py .*DM_I2C.* --soc 'omap[2345]|k3' will match
> CONFIG_DM_I2C and CONFIG_DM_I2C_COMPAT and look for it only for targets
> using the omap2, omap3, omap4, omap5 or k3 SOCs.
>
> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
>
> ---
>
> Changes in v2:
> - basically rewrote the whole thing
> - use tools/moveconfig.py to generate the database of configs
> - use tools/find_defconfigs.py to get the list of defconfigs off interest
> - removed diff with .config. tools/moveconfig.py does a better job
>
> tools/configs2csv.py | 387 +++++++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 387 insertions(+)
> create mode 100755 tools/configs2csv.py
This looks like a useful tool and a big improvement on the movconfig
starting point. Style comments below.
>
> diff --git a/tools/configs2csv.py b/tools/configs2csv.py
> new file mode 100755
> index 0000000..70b6602
> --- /dev/null
> +++ b/tools/configs2csv.py
> @@ -0,0 +1,387 @@
> +#!/usr/bin/env python
> +# SPDX-License-Identifier: GPL-2.0+
> +#
> +# Author: JJ Hiblot <jjhiblot@ti.com>
> +#
> +
> +"""
> +scan the configuration of specified targets (ie defconfigs) and outputs a
> +summary in a csv file.
> +Useful tool to check what platform is using a particular set of options.
> +
> +
> +How does it work?
> +-----------------
> +
> +This tools uses the config database produced by tools/moveconfig.py (called
> +with option -B to get all the configs: SPl, TPL and u-boot). If the database
> +is not present, it will build it. A rebuild can be forced with the option
> +'--rebuild-db'.
> +
> +The list of the targets of interest can be specified by a set of filter (soc,
> +vendor, defconfig name, ..). All those filters are actually regexp, allowing
> +for complex selection. The selection process is done by
> +tools/find_defconfigs.py
> +ex: --soc omap[23] --vendor 'ti|compulab' will inspect the omap2 and omap3
> +platforms from TI and compulab
> +
> +
> +examples:
> +---------
> +
> +
> +1) Get an overview of the usage of CONFIG_DM, CONFIG_SPL_DM, and DM/I2C related
> + options for platforms with omap5 or k3 SOC in u-boot and in SPL
> +
> +$ tools/configs2csv.py CONFIG_SPL_DM CONFIG_DM CONFIG_DM_I2C.* --vendor ti \
> + --soc 'omap5|k3' -X --u-boot --spl -o dummy.csv
> +
Is this output below showing the contents of the ,csv file in a non-CVS format?
> +vendor soc defconfig type CONFIG_DM CONFIG_DM_I2C CONFIG_DM_I2C_COMPAT CONFIG_SPL_DM
> +ti omap5 am57xx_evm_defconfig SPL X X
> +ti omap5 am57xx_evm_defconfig u-boot X X X X
> +ti omap5 am57xx_hs_evm_defconfig SPL X X
> +ti omap5 am57xx_hs_evm_defconfig u-boot X X X X
> +ti k3 am65x_evm_a53_defconfig SPL X X
> +ti k3 am65x_evm_a53_defconfig u-boot X X
> +ti omap5 dra7xx_evm_defconfig SPL X X
> +ti omap5 dra7xx_evm_defconfig u-boot X X X X
> +ti omap5 dra7xx_hs_evm_defconfig SPL X X
> +ti omap5 dra7xx_hs_evm_defconfig u-boot X X X X
> +ti omap5 omap5_uevm_defconfig SPL
> +ti omap5 omap5_uevm_defconfig u-boot
> +
> +
> +This shows quickly that DM is not supported at all for omap5_uevm, that
> +only am65x_evm_a53 in not using DM_I2C in u-boot, and finally that DM_I2C is
> +not enabled in the SPL for any platform although SPL_DM is.
> +Also all the other platforms that enabled DM_I2C, also enabled
> +CONFIG_DM_I2C_COMPAT.
> +
> +
> +2) Check differences in config between SPL, TPL and u-boot (--diff option)
> +
> +Some platforms may disable/enable stuff in the configuration header files if
> +in SPl. This makes it hard to know the usage of a variable by just looking at
> +the .config. This is specially true for DM stuff.
> +
> +$ tools/configs2csv.py CONFIG\(_SPL\)?_DM_.* --vendor ti \
> + --soc 'omap5|k3' --diff --spl --u-boot > dummy.csv
> +
> +vendor soc defconfig CONFIG_DM_I2C CONFIG_DM_I2C_COMPAT CONFIG_DM_STDIO CONFIG_DM_WARN
> +ti omap5 am57xx_evm_defconfig u-boot u-boot u-boot u-boot
> +ti omap5 am57xx_hs_evm_defconfig u-boot u-boot u-boot u-boot
> +ti k3 am65x_evm_a53_defconfig u-boot u-boot
> +ti omap5 dra7xx_evm_defconfig u-boot u-boot u-boot u-boot
> +ti omap5 dra7xx_hs_evm_defconfig u-boot u-boot u-boot u-boot
> +
> +This shows that k3 has no real config diff between SPl and u-boot. whereas am57
> +and dra7 have different settings for DM_I2C and DM_I2C_COMPAT
> +
> +"""
> +
> +import argparse
> +import csv
> +import os
> +import re
> +import sys
> +from collections import namedtuple
> +from itertools import combinations
> +
> +import find_defconfigs
> +
> +CONFIG_DATABASE = 'moveconfig.db'
> +target = namedtuple("target", ["defconfig", "binary_type"])
> +
> +
> +class db:
Please capitalise the class name. Also how about ConfigDb or something
a little longer than db? Below you actually use db as a variable name.
> +
> + """ db is an object that store a collection of targets
> + a target is identified buy its defconfig and its binary type. ex:
> + (omap3_evm_defconfig,SPL)
> + The main purpose of this object is to output a CSV file that describes all
> + the targets.
> + There is also the possibility to create a "diff" db from a db. This new db
> + contains a summary of the differences between target of same defconfig.
> + """
> +
> + def __init__(self):
> + self.targets = dict()
> +
> + def add_target(self, target):
> + self.targets[target] = dict()
> +
> + def add_option(self, target, option, value):
> + self.targets[target][option] = value
> +
> + def add_options(self, target, dic):
> + self.targets[target].update(dic)
> +
> + def output_csv(
> + self, output, show_X=False, header=True, left_columns=None, discard_empty_rows=False):
Please add function comments with Args and Returns (if you have
argument and return value) along with what the function does.
> + all_options = set()
> + if len(self.targets) == 0:
> + return
> +
> + if discard_empty_rows:
> + dic = {k: self.targets[k] for k in self.targets if self.targets[k]}
> + else:
> + dic = self.targets.copy()
> + for target in dic.keys():
> + for option in dic[target].keys():
> + all_options.add(option)
> + if show_X:
> + dic[target][option] = "X"
> + if left_columns:
> + left_columns(target, dic, header=False)
> +
> + columns = []
> + if left_columns:
> + columns.extend(left_columns(None, header=True))
> + columns.extend(sorted(all_options))
> +
> + writer = csv.DictWriter(output, fieldnames=columns,
> + lineterminator='\n')
> + if header:
> + writer.writeheader()
> + for target in sorted(dic.keys()):
> + writer.writerow(dic[target])
> +
> + def diff_one_defconfig(self, defconfig):
> + """ This function creates a dictionary of the differences between the
> + binaries os a single target.
> + For example, for "dra7xx_evm_defconfig" it will compute the diffence
> + between the options used to compile u-boot and the SPL (not the TPL
> + because this platform doesn't have it).
> + The return value looks as follow: { 'CONFIG_DM_I2C: "u-boot",
> + CONFIG_SPL_BUILD:"SPL", CONFIG_DUMMY_SPI_FREQ: "diff" }.
> +
> + The algorithm can probably be optimized, but I didn't care enough.
> + algo is:
> + - return immediately is there is only one binary type (u-boot)
> + - create a dic that is merge of the dic for all the binary types
> + - for each binary type, compare its dic to the merged_dic. If it is
> + different then break. It means that at least one option is different.
> + - if no difference has been found, then return
> + - at this point, we know that there is at least one diff. For each
> + binary types and for all options used for this binary type, check if it
> + is in the merged dic and, if so, if its value is the same. update our
> + return dic with the proper description.
> +
Drop blank line before """
Returns:
dict:
key: ...
value: ...
See buildman,, etc. for examples.
> + """
> +
> + diffs = dict()
> + diff_found = False
> +
> + # get all binary types (spl, TPL, u-boot) generated by this defconfig
> + all_binary_types = sorted(
> + set([t.binary_type for t in self.targets.keys() if t.defconfig == defconfig]))
> +
> + # If there is only one type of binary, no need to do a diff
> + if len(all_binary_types) <= 1:
> + return None
> +
> + # create a dict with all options:values used by all binaries
> + merged_dic = dict()
> + for bin_type in all_binary_types:
> + merged_dic.update(self.targets[target(defconfig, bin_type)])
> +
> + # check if all binaries have the same options (should be the case for
> + # most of the defconfigs)
> + for bin_type in all_binary_types:
> + if self.targets[target(defconfig, bin_type)] != merged_dic:
> + diff_found = True
> + break
> + if not diff_found:
> + return None
> +
> + # at this point, we know that there are some options that differ
> + # between binaries (either not present or different)
> +
> + # Get a list (actually a set) of the options that are different
> + differing_keys = set()
> + for bin_type in all_binary_types:
> + dic = self.targets[target(defconfig, bin_type)]
> + for opt, value in merged_dic.items():
> + if dic.get(opt, None) != value:
> + differing_keys.add(opt)
> +
> + # create a dictionary that summarize the differences
> + for bin_type in all_binary_types:
> + dic = self.targets[target(defconfig, bin_type)]
> + for opt in differing_keys:
> + dic_value = dic.get(opt, None)
> + merged_value = merged_dic.get(opt, None)
> + previous = diffs.get(opt, None)
> + if dic_value:
> + if dic_value != merged_value:
> + diffs[opt] = "diff"
> + elif previous != "diff":
> + diffs[opt] = ' / '.join(
> + [previous, bin_type]) if previous else bin_type
> +
> + return diffs
> +
> + def diff(self):
> + """ create a new db that contains the differences between the binaries
> + for all the targets in the db """
""" goes on its own line. Looks like this needs Returns comment.
> + diff_db = db()
> + # get a list of all the targets
> + all_defconfigs = set([t.defconfig for t in self.targets.keys()])
> + # for every target of the list, get a dictionary of the difference.
> + # if the dictionary is not empty, add it the new db
> + for defconfig in all_defconfigs:
> + diff_dic = self.diff_one_defconfig(defconfig)
> + if diff_dic:
> + diff_db.add_target(target(defconfig, None))
> + diff_db.add_options(target(defconfig, None), diff_dic)
> + return diff_db
> +
> +
> +def read_db(boards, option_filter, binary_types):
Needs function comment again.
> + defconfig = ""
> + _db = db()
> +
> + # Read in the database
> + with open(CONFIG_DATABASE) as fd:
> + for line in fd.readlines():
> + line = line.rstrip()
> + if not line: # Separator between defconfigs.
> + # We do not really care. We detect a new config by the absence
> + # of ' 'at the beginning of the line
> + pass
> + elif line[0] == ' ': # CONFIG_xxx line
> + if t and option_filter(line):
> + config, value = line.strip().split('=', 1)
> + _db.add_option(t, config, value)
> + else: # New defconfig
> + infos = line.split()
> + defconfig = infos[0]
> + try:
> + binary_type = infos[1]
> + except:
> + binary_type = "u-boot"
> + if binary_type in binary_types and defconfig in boards:
> + t = target(defconfig, binary_type)
> + _db.add_target(t)
> + else:
> + t = None
> + return _db
> +
> +
> +def main():
> + parser = argparse.ArgumentParser(description="Show CONFIG options usage")
> + parser.add_argument("options", metavar='OPTION', type=str, nargs='+',
> + help="regexp to filter on options.\
> + ex: CONFIG_DM_I2C_COMPAT or '.*DM_MMC.*'")
> + parser.add_argument(
> + "-X", help="show a X instead of the value of the option",
Please capitalise the help, e.g. 'Show a X'
> + action="store_true")
> + parser.add_argument("--u-boot", help="parse the u-boot configs",
> + action="store_true")
> + parser.add_argument("--spl", help="parse the SPL configs",
> + action="store_true")
> + parser.add_argument("--tpl", help="parse the TPL configs",
> + action="store_true")
> + parser.add_argument("--diff",
> + help="show only the options that differs between the selected configs (SPL, TPL, u-boot)",
> + action="store_true")
> + parser.add_argument("--rebuild-db",
> + help="Force a rebuild of the config database",
> + action="store_true")
> + parser.add_argument('-j', '--jobs',
> + help='the number of jobs to run simultaneously')
> + parser.add_argument('-o', '--output',
> + help='The output CSV filename. uses stdout if not specified')
> + parser.add_argument('--no-header', help='Do not put the header at the top',
> + action="store_true")
> + parser.add_argument('--discard-empty', action="store_true",
> + help='Discard the empty rows (defconfigs that do not enable at least one option)')
> +
> + find_defconfigs.update_parser_with_default_options(parser)
> + args = parser.parse_args()
> +
> + # generate db file if needed or requested
> + # The job of generating the db is actually done by tools/moveconfig.py
> + # (called with -B)
> + if args.rebuild_db or not os.path.isfile(CONFIG_DATABASE):
> + find_defconfig_args = ["--{} '{}'".format(f, getattr(args, f))
> + for f in find_defconfigs.get_default_options()
> + if getattr(args, f)]
> + if args.jobs:
> + jobs_option = "-j {}".format(args.jobs)
> + else:
> + jobs_option = ""
> +
> + rc = os.system(
> + "tools/find_defconfigs.py {} | tools/moveconfig.py -B {} -d - 1>&2 "
> + .format(" ".join(find_defconfig_args), jobs_option))
> + if rc:
> + sys.exit(1)
This is fine, but I wonder why you don't import this module and call
it directly? It could return a dict, perhaps?
> +
> + # get a list of defconfigs matching the rules
> + targets = [t for t in find_defconfigs.get_matching_boards(args)]
> + defconfigs = [t.defconfig for t in targets]
> +
> + # create a list of binary types we are interested in
> + binary_types = []
> + if args.spl:
> + binary_types.append("SPL")
> + if args.tpl:
> + binary_types.append("TPL")
> + if args.u_boot or not binary_types:
> + binary_types.append("u-boot")
> +
> + # define a function used to filter on the options
> + rules = [re.compile(" {}=".format(cfg_opt))
> + for cfg_opt in args.options]
> +
> + def match_any_rule(line):
> + for r in rules:
> + if r.match(line):
> + return True
> + return False
> +
> + # read the database
> + db = read_db(defconfigs, match_any_rule, binary_types)
> +
> + target_dict = {}
> + for t in targets:
> + target_dict[t.defconfig] = t
> +
> + def populate_left_columns(target=None, dic=None, header=True):
> + if header:
> + return ["vendor", "soc", "defconfig", "type"]
> + else:
> + dic[target]["vendor"] = target_dict[target.defconfig].vendor
> + dic[target]["soc"] = target_dict[target.defconfig].soc
> + dic[target]["defconfig"] = target.defconfig
> + dic[target]["type"] = target.binary_type
> +
> + def populate_left_columns_diff(target=None, dic=None, header=True):
> + if header:
> + return ["vendor", "soc", "defconfig"]
> + else:
> + dic[target]["vendor"] = target_dict[target.defconfig].vendor
> + dic[target]["soc"] = target_dict[target.defconfig].soc
> + dic[target]["defconfig"] = target.defconfig
> +
> + if args.output:
> + out = open(args.output, "w")
> + else:
> + out = sys.stdout
> +
> + if args.diff:
> + db.diff().output_csv(output=out, show_X=False,
> + header=not args.no_header,
> + discard_empty_rows=args.discard_empty,
> + left_columns=populate_left_columns_diff)
> + else:
> + db.output_csv(output=out, show_X=args.X, header=not args.no_header,
> + discard_empty_rows=args.discard_empty,
> + left_columns=populate_left_columns)
> +
> + if out != sys.stdout:
> + out.close()
> +
> +if __name__ == '__main__':
> + main()
> --
> 2.7.4
>
Regards,
Simon
^ permalink raw reply [flat|nested] 10+ messages in thread
* [U-Boot] [RFC PATCH v2 2/3] tools: Add a tool to get a list of defconfigs based on filters
2018-10-03 13:53 ` [U-Boot] [RFC PATCH v2 2/3] tools: Add a tool to get a list of defconfigs based on filters Jean-Jacques Hiblot
@ 2018-10-09 16:20 ` Simon Glass
2018-10-18 12:03 ` Jean-Jacques Hiblot
2018-10-18 14:36 ` Jean-Jacques Hiblot
0 siblings, 2 replies; 10+ messages in thread
From: Simon Glass @ 2018-10-09 16:20 UTC (permalink / raw)
To: u-boot
Hi Jean-Jacques,
On 3 October 2018 at 07:53, Jean-Jacques Hiblot <jjhiblot@ti.com> wrote:
> The possible filters are "arch", "vendor", "soc", "cpu" and "arch".
>
> The list of all the defconfigs is read from boards.cfg. If this file
> doesn't exist, then tools/genboardscfg.py is called to generate it.
>
> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
> ---
>
> Changes in v2: None
>
> tools/find_defconfigs.py | 167 +++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 167 insertions(+)
> create mode 100755 tools/find_defconfigs.py
This looks good, but I have some style comments below.
Also it seems to do a similar thing to tools/buildman/board.py. Should
we replace that impl with what you have here? It looks more flexible
that what buildman currently provides.
>
> diff --git a/tools/find_defconfigs.py b/tools/find_defconfigs.py
> new file mode 100755
> index 0000000..9d68cef
> --- /dev/null
> +++ b/tools/find_defconfigs.py
> @@ -0,0 +1,167 @@
> +#!/usr/bin/env python
> +# SPDX-License-Identifier: GPL-2.0+
> +#
> +# Author: JJ Hiblot <jjhiblot@ti.com>
> +#
> +
> +"""
> +Output a list of defconfig matching criteria.
I think you mean defconfig-matching?
> +
> +The possible criteria are soc, vendor, arch, cpu, board and defconfig name.
> +The criteria are expressed as regexp, allowing for complex selection.
> +
> +How does it work?
> +-----------------
> +
> +This tools uses the boards.cfg file produced by tools/genboardscfg.py
> +It reads the file to get a list of all the defconfigs and the information
> +about the soc, vendor etc. for each of them.
> +Then it walks this list and outputs the defconfigs for which the info match
> +the regexp passed to the program.
> +
> +examples:
> +---------
> +
> +1) Get the list of defconfigs for boards built around omap5, omap4 and k3, not built by TI
> +
> +$ tools/find_defconfigs.py --soc 'omap[45]|k3' --vendor '(?!ti)'
> +kc1_defconfig
> +duovero_defconfig
> +cl-som-am57x_defconfig
> +cm_t54_defconfig
> +
> +2) Same list but iwth more details on the items that were used as filters
> +
> +$ tools/find_defconfigs.py --soc 'omap[45]|k3' --vendor '(?!ti)' --show-details
> +kc1_defconfig | omap4 | amazon
> +duovero_defconfig | omap4 | gumstix
> +cl-som-am57x_defconfig | omap5 | compulab
> +cm_t54_defconfig | omap5 | compulab
> +
> +
> +"""
> +
> +import re
> +import os
> +import argparse
Please sort these
> +
> +
> +class board:
> +
Need a class comment here, also use Board since it is a class name
> + def __init__(self, status, arch, cpu, soc,
> + vendor, board, target, options, maintainer):
> + self.status = status
> + self.arch = arch
> + self.cpu = cpu
> + self.soc = soc
> + self.vendor = vendor
> + self.board = board
> + self.target = target
> + self.defconfig = "{}_defconfig".format(target)
> + self.options = options
> + self.maintainer = maintainer
> +
> + def show(self, sep=' | ', props=None):
Function comment (see other tools for style). Need to document args
and any return value.
> + if not props:
> + print(
> + sep.join([self.defconfig,
> + self.vendor,
> + self.arch,
> + self.cpu,
> + self.soc,
> + self.board,
> + self.status,
> + self.maintainer]))
> + else:
> + print(sep.join([self.defconfig] + [getattr(self, prop) for prop in props]))
Does this need to import print_function from __future__ for Python 2?
> +
> + def cleanup(self):
> + """ remove the directory in which the cfg files have been built """
Please use comment style from other tools. Same below.
> + shutil.rmtree(self.temp_dir)
> +
> + def match(self, rules):
> + """ return True if the board match all the criteria """
> + for prop, r in rules:
> + val = getattr(self, prop)
> + if not val or val == "-":
> + return False
> + if not r.match(val):
> + return False
> + return True
> +
> +
> +def get_all_boards():
> + """ extract a list of boards from 'boards.cfg' """
> + result = []
> + if not os.path.isfile("boards.cfg"):
> + os.system('tools/genboardscfg.py')
> +
> + with open('boards.cfg', 'r') as f:
> + for l in f.readlines():
> + if not l or l[0] == "#":
> + continue
> + props = l.strip().split(None, 8)
> + if not props:
> + continue
> + if len(props) < 9:
> + props.extend(["-"] * (9 - len(props)))
> + result.append(board(*props))
> + return result
> +
> +
> +def get_default_options():
> + return ["board", "soc", "vendor", "arch", "cpu", "target"]
> +
> +
> +def update_parser_with_default_options(parser):
> + parser.add_argument('-i', '--ignore-case', action="store_true")
> + parser.add_argument("--soc",
> + help="regexp to filter on SoC. ex: 'omap[45]' to inspect omap5 and omap5 targets")
> + parser.add_argument("--vendor", help="regexp to filter on Vendor.")
> + parser.add_argument("--arch", help="regexp to filter on Arch")
> + parser.add_argument("--cpu", help="regexp to filter on CPU")
> + parser.add_argument("--board", help="regexp to filter on Board")
> + parser.add_argument("--target",
> + help="regexp to filter on Target (defconfig filename without the '_defconfig' suffix)")
> +
> +
> +def get_matching_boards(args, fields=get_default_options()):
> + # compile a list of regexp used to filter the targets
> + boards = []
> + rules = []
> + for f in fields:
> + arg = getattr(args, f)
> + if arg:
> + rules.append((f, re.compile("\\b{}\\b".format(arg),
> + re.IGNORECASE if args.ignore_case else 0)))
> +
> + # get a list of boards matching the rules
> + for b in get_all_boards():
> + if b.match(rules):
> + boards.append(b)
> + return boards
> +
> +
> +def main():
> + parser = argparse.ArgumentParser(description="Show CONFIG options usage")
> + update_parser_with_default_options(parser)
> + parser.add_argument("--maintainer", help="regexp to filter on maintainer.")
> + parser.add_argument("--status", help="regexp to filter on status.")
> + parser.add_argument("--show-details", help="show fields used as filter",
> + action="store_true")
> + parser.add_argument("--show-all", help="show all fields",
> + action="store_true")
> + args = parser.parse_args()
> + fields = get_default_options() + ["status", "maintainer"]
> +
> + for b in get_matching_boards(args, fields):
> + if args.show_details:
> + props = [f for f in fields if getattr(args, f)]
> + b.show(props=props)
> + elif args.show_all:
> + b.show()
> + else:
> + print(b.defconfig)
> +
> +if __name__ == '__main__':
> + main()
> --
> 2.7.4
>
Regards,
Simon
^ permalink raw reply [flat|nested] 10+ messages in thread
* [U-Boot] [RFC PATCH v2 1/3] tools: moveconfig: Add an option to build a fuller database of options
2018-10-03 13:53 ` [U-Boot] [RFC PATCH v2 1/3] tools: moveconfig: Add an option to build a fuller database of options Jean-Jacques Hiblot
@ 2018-10-09 16:20 ` Simon Glass
0 siblings, 0 replies; 10+ messages in thread
From: Simon Glass @ 2018-10-09 16:20 UTC (permalink / raw)
To: u-boot
On 3 October 2018 at 07:53, Jean-Jacques Hiblot <jjhiblot@ti.com> wrote:
> "moveconfig -b" will build a database of config options based on the
> content of include/config/auto.conf that reflects the .config
>
> Add a new option '-B' that does essentially the same, except that it uses
> the content of u-boot.cfg, spl/u-boot.cfg and tpl/u-boot.cfg.
> This allows to get the options from .config AND the headers for all the
> possible binary types (u-boot, SPL and TPL)
>
> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
> ---
>
> Changes in v2: None
Really?
>
> tools/moveconfig.py | 56 ++++++++++++++++++++++++++++++++++++++++++++++-------
> 1 file changed, 49 insertions(+), 7 deletions(-)
Reviewed-by: Simon Glass <sjg@chromium.org>
^ permalink raw reply [flat|nested] 10+ messages in thread
* [U-Boot] [RFC PATCH v2 3/3] tools: Add a tool to get an overview of the usage of CONFIG options
2018-10-09 16:20 ` Simon Glass
@ 2018-10-18 11:38 ` Jean-Jacques Hiblot
0 siblings, 0 replies; 10+ messages in thread
From: Jean-Jacques Hiblot @ 2018-10-18 11:38 UTC (permalink / raw)
To: u-boot
Simon,
On 09/10/2018 18:20, Simon Glass wrote:
> Hi Jean-Jacques,
>
> On 3 October 2018 at 07:53, Jean-Jacques Hiblot <jjhiblot@ti.com> wrote:
>> configs2csv.py is tool that allow to check how some options are used for a
>> particular subset of platforms.
>> The purpose is to identify the targets that are actually using one or more
>> options of interest.
>> For example, it can tell what targets are still using CONFIG_DM_I2_COMPAT.
>> It relies on the config database produced by tools/moveconfig.py.
>> If the database doesn't exist, it will build it for the restricted set of
>> the selected platforms. Once the database is built, it is much faster than
>> greping the configs directory and more accurate as it relies on the
>> information found in u-boot.cfg instead of defconfigs.
>> It possible to look for options in the u-boot, the SPL or the TPL
>> configurations. It can also perform diffs between those configurations.
>>
>> usage: configs2csv.py [-h] [-X] [--u-boot] [--spl] [--tpl] [--diff]
>> [--rebuild-db] [-j JOBS] [-o OUTPUT] [--no-header]
>> [--discard-empty] [-i] [--soc SOC] [--vendor VENDOR]
>> [--arch ARCH] [--cpu CPU] [--board BOARD]
>> [--target TARGET]
>> OPTION [OPTION ...]
>>
>> all filtering parameters (OPTION, vendor, arch, ...) accept regexp.
>> ex: configs2csv.py .*DM_I2C.* --soc 'omap[2345]|k3' will match
>> CONFIG_DM_I2C and CONFIG_DM_I2C_COMPAT and look for it only for targets
>> using the omap2, omap3, omap4, omap5 or k3 SOCs.
>>
>> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
>>
>> ---
>>
>> Changes in v2:
>> - basically rewrote the whole thing
>> - use tools/moveconfig.py to generate the database of configs
>> - use tools/find_defconfigs.py to get the list of defconfigs off interest
>> - removed diff with .config. tools/moveconfig.py does a better job
>>
>> tools/configs2csv.py | 387 +++++++++++++++++++++++++++++++++++++++++++++++++++
>> 1 file changed, 387 insertions(+)
>> create mode 100755 tools/configs2csv.py
> This looks like a useful tool and a big improvement on the movconfig
> starting point. Style comments below.
Thanks for the review. I'll send a v3 taking care of the comments in a
couple of days.
>
>> diff --git a/tools/configs2csv.py b/tools/configs2csv.py
>> new file mode 100755
>> index 0000000..70b6602
>> --- /dev/null
>> +++ b/tools/configs2csv.py
>> @@ -0,0 +1,387 @@
>> +#!/usr/bin/env python
>> +# SPDX-License-Identifier: GPL-2.0+
>> +#
>> +# Author: JJ Hiblot <jjhiblot@ti.com>
>> +#
>> +
>> +"""
>> +scan the configuration of specified targets (ie defconfigs) and outputs a
>> +summary in a csv file.
>> +Useful tool to check what platform is using a particular set of options.
>> +
>> +
>> +How does it work?
>> +-----------------
>> +
>> +This tools uses the config database produced by tools/moveconfig.py (called
>> +with option -B to get all the configs: SPl, TPL and u-boot). If the database
>> +is not present, it will build it. A rebuild can be forced with the option
>> +'--rebuild-db'.
>> +
>> +The list of the targets of interest can be specified by a set of filter (soc,
>> +vendor, defconfig name, ..). All those filters are actually regexp, allowing
>> +for complex selection. The selection process is done by
>> +tools/find_defconfigs.py
>> +ex: --soc omap[23] --vendor 'ti|compulab' will inspect the omap2 and omap3
>> +platforms from TI and compulab
>> +
>> +
>> +examples:
>> +---------
>> +
>> +
>> +1) Get an overview of the usage of CONFIG_DM, CONFIG_SPL_DM, and DM/I2C related
>> + options for platforms with omap5 or k3 SOC in u-boot and in SPL
>> +
>> +$ tools/configs2csv.py CONFIG_SPL_DM CONFIG_DM CONFIG_DM_I2C.* --vendor ti \
>> + --soc 'omap5|k3' -X --u-boot --spl -o dummy.csv
>> +
> Is this output below showing the contents of the ,csv file in a non-CVS format?
Yes. This represents what we would see in a spreadsheet viewer.
The csv output is not easily readable.
>> +vendor soc defconfig type CONFIG_DM CONFIG_DM_I2C CONFIG_DM_I2C_COMPAT CONFIG_SPL_DM
>> +ti omap5 am57xx_evm_defconfig SPL X X
>> +ti omap5 am57xx_evm_defconfig u-boot X X X X
>> +ti omap5 am57xx_hs_evm_defconfig SPL X X
>> +ti omap5 am57xx_hs_evm_defconfig u-boot X X X X
>> +ti k3 am65x_evm_a53_defconfig SPL X X
>> +ti k3 am65x_evm_a53_defconfig u-boot X X
>> +ti omap5 dra7xx_evm_defconfig SPL X X
>> +ti omap5 dra7xx_evm_defconfig u-boot X X X X
>> +ti omap5 dra7xx_hs_evm_defconfig SPL X X
>> +ti omap5 dra7xx_hs_evm_defconfig u-boot X X X X
>> +ti omap5 omap5_uevm_defconfig SPL
>> +ti omap5 omap5_uevm_defconfig u-boot
>> +
>> +
>> +This shows quickly that DM is not supported at all for omap5_uevm, that
>> +only am65x_evm_a53 in not using DM_I2C in u-boot, and finally that DM_I2C is
>> +not enabled in the SPL for any platform although SPL_DM is.
>> +Also all the other platforms that enabled DM_I2C, also enabled
>> +CONFIG_DM_I2C_COMPAT.
>> +
>> +
>> +2) Check differences in config between SPL, TPL and u-boot (--diff option)
>> +
>> +Some platforms may disable/enable stuff in the configuration header files if
>> +in SPl. This makes it hard to know the usage of a variable by just looking at
>> +the .config. This is specially true for DM stuff.
>> +
>> +$ tools/configs2csv.py CONFIG\(_SPL\)?_DM_.* --vendor ti \
>> + --soc 'omap5|k3' --diff --spl --u-boot > dummy.csv
>> +
>> +vendor soc defconfig CONFIG_DM_I2C CONFIG_DM_I2C_COMPAT CONFIG_DM_STDIO CONFIG_DM_WARN
>> +ti omap5 am57xx_evm_defconfig u-boot u-boot u-boot u-boot
>> +ti omap5 am57xx_hs_evm_defconfig u-boot u-boot u-boot u-boot
>> +ti k3 am65x_evm_a53_defconfig u-boot u-boot
>> +ti omap5 dra7xx_evm_defconfig u-boot u-boot u-boot u-boot
>> +ti omap5 dra7xx_hs_evm_defconfig u-boot u-boot u-boot u-boot
>> +
>> +This shows that k3 has no real config diff between SPl and u-boot. whereas am57
>> +and dra7 have different settings for DM_I2C and DM_I2C_COMPAT
>> +
>> +"""
>> +
>> +import argparse
>> +import csv
>> +import os
>> +import re
>> +import sys
>> +from collections import namedtuple
>> +from itertools import combinations
>> +
>> +import find_defconfigs
>> +
>> +CONFIG_DATABASE = 'moveconfig.db'
>> +target = namedtuple("target", ["defconfig", "binary_type"])
>> +
>> +
>> +class db:
> Please capitalise the class name. Also how about ConfigDb or something
> a little longer than db? Below you actually use db as a variable name.
>
>> +
>> + """ db is an object that store a collection of targets
>> + a target is identified buy its defconfig and its binary type. ex:
>> + (omap3_evm_defconfig,SPL)
>> + The main purpose of this object is to output a CSV file that describes all
>> + the targets.
>> + There is also the possibility to create a "diff" db from a db. This new db
>> + contains a summary of the differences between target of same defconfig.
>> + """
>> +
>> + def __init__(self):
>> + self.targets = dict()
>> +
>> + def add_target(self, target):
>> + self.targets[target] = dict()
>> +
>> + def add_option(self, target, option, value):
>> + self.targets[target][option] = value
>> +
>> + def add_options(self, target, dic):
>> + self.targets[target].update(dic)
>> +
>> + def output_csv(
>> + self, output, show_X=False, header=True, left_columns=None, discard_empty_rows=False):
> Please add function comments with Args and Returns (if you have
> argument and return value) along with what the function does.
>
>> + all_options = set()
>> + if len(self.targets) == 0:
>> + return
>> +
>> + if discard_empty_rows:
>> + dic = {k: self.targets[k] for k in self.targets if self.targets[k]}
>> + else:
>> + dic = self.targets.copy()
>> + for target in dic.keys():
>> + for option in dic[target].keys():
>> + all_options.add(option)
>> + if show_X:
>> + dic[target][option] = "X"
>> + if left_columns:
>> + left_columns(target, dic, header=False)
>> +
>> + columns = []
>> + if left_columns:
>> + columns.extend(left_columns(None, header=True))
>> + columns.extend(sorted(all_options))
>> +
>> + writer = csv.DictWriter(output, fieldnames=columns,
>> + lineterminator='\n')
>> + if header:
>> + writer.writeheader()
>> + for target in sorted(dic.keys()):
>> + writer.writerow(dic[target])
>> +
>> + def diff_one_defconfig(self, defconfig):
>> + """ This function creates a dictionary of the differences between the
>> + binaries os a single target.
>> + For example, for "dra7xx_evm_defconfig" it will compute the diffence
>> + between the options used to compile u-boot and the SPL (not the TPL
>> + because this platform doesn't have it).
>> + The return value looks as follow: { 'CONFIG_DM_I2C: "u-boot",
>> + CONFIG_SPL_BUILD:"SPL", CONFIG_DUMMY_SPI_FREQ: "diff" }.
>> +
>> + The algorithm can probably be optimized, but I didn't care enough.
>> + algo is:
>> + - return immediately is there is only one binary type (u-boot)
>> + - create a dic that is merge of the dic for all the binary types
>> + - for each binary type, compare its dic to the merged_dic. If it is
>> + different then break. It means that at least one option is different.
>> + - if no difference has been found, then return
>> + - at this point, we know that there is at least one diff. For each
>> + binary types and for all options used for this binary type, check if it
>> + is in the merged dic and, if so, if its value is the same. update our
>> + return dic with the proper description.
>> +
> Drop blank line before """
>
> Returns:
> dict:
> key: ...
> value: ...
>
> See buildman,, etc. for examples.
>
>> + """
>> +
>> + diffs = dict()
>> + diff_found = False
>> +
>> + # get all binary types (spl, TPL, u-boot) generated by this defconfig
>> + all_binary_types = sorted(
>> + set([t.binary_type for t in self.targets.keys() if t.defconfig == defconfig]))
>> +
>> + # If there is only one type of binary, no need to do a diff
>> + if len(all_binary_types) <= 1:
>> + return None
>> +
>> + # create a dict with all options:values used by all binaries
>> + merged_dic = dict()
>> + for bin_type in all_binary_types:
>> + merged_dic.update(self.targets[target(defconfig, bin_type)])
>> +
>> + # check if all binaries have the same options (should be the case for
>> + # most of the defconfigs)
>> + for bin_type in all_binary_types:
>> + if self.targets[target(defconfig, bin_type)] != merged_dic:
>> + diff_found = True
>> + break
>> + if not diff_found:
>> + return None
>> +
>> + # at this point, we know that there are some options that differ
>> + # between binaries (either not present or different)
>> +
>> + # Get a list (actually a set) of the options that are different
>> + differing_keys = set()
>> + for bin_type in all_binary_types:
>> + dic = self.targets[target(defconfig, bin_type)]
>> + for opt, value in merged_dic.items():
>> + if dic.get(opt, None) != value:
>> + differing_keys.add(opt)
>> +
>> + # create a dictionary that summarize the differences
>> + for bin_type in all_binary_types:
>> + dic = self.targets[target(defconfig, bin_type)]
>> + for opt in differing_keys:
>> + dic_value = dic.get(opt, None)
>> + merged_value = merged_dic.get(opt, None)
>> + previous = diffs.get(opt, None)
>> + if dic_value:
>> + if dic_value != merged_value:
>> + diffs[opt] = "diff"
>> + elif previous != "diff":
>> + diffs[opt] = ' / '.join(
>> + [previous, bin_type]) if previous else bin_type
>> +
>> + return diffs
>> +
>> + def diff(self):
>> + """ create a new db that contains the differences between the binaries
>> + for all the targets in the db """
> """ goes on its own line. Looks like this needs Returns comment.
>
>> + diff_db = db()
>> + # get a list of all the targets
>> + all_defconfigs = set([t.defconfig for t in self.targets.keys()])
>> + # for every target of the list, get a dictionary of the difference.
>> + # if the dictionary is not empty, add it the new db
>> + for defconfig in all_defconfigs:
>> + diff_dic = self.diff_one_defconfig(defconfig)
>> + if diff_dic:
>> + diff_db.add_target(target(defconfig, None))
>> + diff_db.add_options(target(defconfig, None), diff_dic)
>> + return diff_db
>> +
>> +
>> +def read_db(boards, option_filter, binary_types):
> Needs function comment again.
>
>> + defconfig = ""
>> + _db = db()
>> +
>> + # Read in the database
>> + with open(CONFIG_DATABASE) as fd:
>> + for line in fd.readlines():
>> + line = line.rstrip()
>> + if not line: # Separator between defconfigs.
>> + # We do not really care. We detect a new config by the absence
>> + # of ' 'at the beginning of the line
>> + pass
>> + elif line[0] == ' ': # CONFIG_xxx line
>> + if t and option_filter(line):
>> + config, value = line.strip().split('=', 1)
>> + _db.add_option(t, config, value)
>> + else: # New defconfig
>> + infos = line.split()
>> + defconfig = infos[0]
>> + try:
>> + binary_type = infos[1]
>> + except:
>> + binary_type = "u-boot"
>> + if binary_type in binary_types and defconfig in boards:
>> + t = target(defconfig, binary_type)
>> + _db.add_target(t)
>> + else:
>> + t = None
>> + return _db
>> +
>> +
>> +def main():
>> + parser = argparse.ArgumentParser(description="Show CONFIG options usage")
>> + parser.add_argument("options", metavar='OPTION', type=str, nargs='+',
>> + help="regexp to filter on options.\
>> + ex: CONFIG_DM_I2C_COMPAT or '.*DM_MMC.*'")
>> + parser.add_argument(
>> + "-X", help="show a X instead of the value of the option",
> Please capitalise the help, e.g. 'Show a X'
>
>> + action="store_true")
>> + parser.add_argument("--u-boot", help="parse the u-boot configs",
>> + action="store_true")
>> + parser.add_argument("--spl", help="parse the SPL configs",
>> + action="store_true")
>> + parser.add_argument("--tpl", help="parse the TPL configs",
>> + action="store_true")
>> + parser.add_argument("--diff",
>> + help="show only the options that differs between the selected configs (SPL, TPL, u-boot)",
>> + action="store_true")
>> + parser.add_argument("--rebuild-db",
>> + help="Force a rebuild of the config database",
>> + action="store_true")
>> + parser.add_argument('-j', '--jobs',
>> + help='the number of jobs to run simultaneously')
>> + parser.add_argument('-o', '--output',
>> + help='The output CSV filename. uses stdout if not specified')
>> + parser.add_argument('--no-header', help='Do not put the header at the top',
>> + action="store_true")
>> + parser.add_argument('--discard-empty', action="store_true",
>> + help='Discard the empty rows (defconfigs that do not enable at least one option)')
>> +
>> + find_defconfigs.update_parser_with_default_options(parser)
>> + args = parser.parse_args()
>> +
>> + # generate db file if needed or requested
>> + # The job of generating the db is actually done by tools/moveconfig.py
>> + # (called with -B)
>> + if args.rebuild_db or not os.path.isfile(CONFIG_DATABASE):
>> + find_defconfig_args = ["--{} '{}'".format(f, getattr(args, f))
>> + for f in find_defconfigs.get_default_options()
>> + if getattr(args, f)]
>> + if args.jobs:
>> + jobs_option = "-j {}".format(args.jobs)
>> + else:
>> + jobs_option = ""
>> +
>> + rc = os.system(
>> + "tools/find_defconfigs.py {} | tools/moveconfig.py -B {} -d - 1>&2 "
>> + .format(" ".join(find_defconfig_args), jobs_option))
>> + if rc:
>> + sys.exit(1)
> This is fine, but I wonder why you don't import this module and call
> it directly? It could return a dict, perhaps?
I tried at first but ended up with a lot of changes in moveconfig.py.
moveconfig.py has evolved and grown in complexity overtime. It could be
split into small independent programs using a generic core to build and
parse the configs. I am not sure that it is worth it though.
>> +
>> + # get a list of defconfigs matching the rules
>> + targets = [t for t in find_defconfigs.get_matching_boards(args)]
>> + defconfigs = [t.defconfig for t in targets]
>> +
>> + # create a list of binary types we are interested in
>> + binary_types = []
>> + if args.spl:
>> + binary_types.append("SPL")
>> + if args.tpl:
>> + binary_types.append("TPL")
>> + if args.u_boot or not binary_types:
>> + binary_types.append("u-boot")
>> +
>> + # define a function used to filter on the options
>> + rules = [re.compile(" {}=".format(cfg_opt))
>> + for cfg_opt in args.options]
>> +
>> + def match_any_rule(line):
>> + for r in rules:
>> + if r.match(line):
>> + return True
>> + return False
>> +
>> + # read the database
>> + db = read_db(defconfigs, match_any_rule, binary_types)
>> +
>> + target_dict = {}
>> + for t in targets:
>> + target_dict[t.defconfig] = t
>> +
>> + def populate_left_columns(target=None, dic=None, header=True):
>> + if header:
>> + return ["vendor", "soc", "defconfig", "type"]
>> + else:
>> + dic[target]["vendor"] = target_dict[target.defconfig].vendor
>> + dic[target]["soc"] = target_dict[target.defconfig].soc
>> + dic[target]["defconfig"] = target.defconfig
>> + dic[target]["type"] = target.binary_type
>> +
>> + def populate_left_columns_diff(target=None, dic=None, header=True):
>> + if header:
>> + return ["vendor", "soc", "defconfig"]
>> + else:
>> + dic[target]["vendor"] = target_dict[target.defconfig].vendor
>> + dic[target]["soc"] = target_dict[target.defconfig].soc
>> + dic[target]["defconfig"] = target.defconfig
>> +
>> + if args.output:
>> + out = open(args.output, "w")
>> + else:
>> + out = sys.stdout
>> +
>> + if args.diff:
>> + db.diff().output_csv(output=out, show_X=False,
>> + header=not args.no_header,
>> + discard_empty_rows=args.discard_empty,
>> + left_columns=populate_left_columns_diff)
>> + else:
>> + db.output_csv(output=out, show_X=args.X, header=not args.no_header,
>> + discard_empty_rows=args.discard_empty,
>> + left_columns=populate_left_columns)
>> +
>> + if out != sys.stdout:
>> + out.close()
>> +
>> +if __name__ == '__main__':
>> + main()
>> --
>> 2.7.4
>>
> Regards,
> Simon
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* [U-Boot] [RFC PATCH v2 2/3] tools: Add a tool to get a list of defconfigs based on filters
2018-10-09 16:20 ` Simon Glass
@ 2018-10-18 12:03 ` Jean-Jacques Hiblot
2018-10-18 14:36 ` Jean-Jacques Hiblot
1 sibling, 0 replies; 10+ messages in thread
From: Jean-Jacques Hiblot @ 2018-10-18 12:03 UTC (permalink / raw)
To: u-boot
On 09/10/2018 18:20, Simon Glass wrote:
> Hi Jean-Jacques,
>
> On 3 October 2018 at 07:53, Jean-Jacques Hiblot <jjhiblot@ti.com> wrote:
>> The possible filters are "arch", "vendor", "soc", "cpu" and "arch".
>>
>> The list of all the defconfigs is read from boards.cfg. If this file
>> doesn't exist, then tools/genboardscfg.py is called to generate it.
>>
>> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
>> ---
>>
>> Changes in v2: None
>>
>> tools/find_defconfigs.py | 167 +++++++++++++++++++++++++++++++++++++++++++++++
>> 1 file changed, 167 insertions(+)
>> create mode 100755 tools/find_defconfigs.py
> This looks good, but I have some style comments below.
>
> Also it seems to do a similar thing to tools/buildman/board.py. Should
> we replace that impl with what you have here? It looks more flexible
> that what buildman currently provides.
It is very similar indeed and they certainly overlap. When time permits,
I'll look at the possibility to re-use this code in buildman.
>
>> diff --git a/tools/find_defconfigs.py b/tools/find_defconfigs.py
>> new file mode 100755
>> index 0000000..9d68cef
>> --- /dev/null
>> +++ b/tools/find_defconfigs.py
>> @@ -0,0 +1,167 @@
>> +#!/usr/bin/env python
>> +# SPDX-License-Identifier: GPL-2.0+
>> +#
>> +# Author: JJ Hiblot <jjhiblot@ti.com>
>> +#
>> +
>> +"""
>> +Output a list of defconfig matching criteria.
> I think you mean defconfig-matching?
>
>> +
>> +The possible criteria are soc, vendor, arch, cpu, board and defconfig name.
>> +The criteria are expressed as regexp, allowing for complex selection.
>> +
>> +How does it work?
>> +-----------------
>> +
>> +This tools uses the boards.cfg file produced by tools/genboardscfg.py
>> +It reads the file to get a list of all the defconfigs and the information
>> +about the soc, vendor etc. for each of them.
>> +Then it walks this list and outputs the defconfigs for which the info match
>> +the regexp passed to the program.
>> +
>> +examples:
>> +---------
>> +
>> +1) Get the list of defconfigs for boards built around omap5, omap4 and k3, not built by TI
>> +
>> +$ tools/find_defconfigs.py --soc 'omap[45]|k3' --vendor '(?!ti)'
>> +kc1_defconfig
>> +duovero_defconfig
>> +cl-som-am57x_defconfig
>> +cm_t54_defconfig
>> +
>> +2) Same list but iwth more details on the items that were used as filters
>> +
>> +$ tools/find_defconfigs.py --soc 'omap[45]|k3' --vendor '(?!ti)' --show-details
>> +kc1_defconfig | omap4 | amazon
>> +duovero_defconfig | omap4 | gumstix
>> +cl-som-am57x_defconfig | omap5 | compulab
>> +cm_t54_defconfig | omap5 | compulab
>> +
>> +
>> +"""
>> +
>> +import re
>> +import os
>> +import argparse
> Please sort these
>
>> +
>> +
>> +class board:
>> +
> Need a class comment here, also use Board since it is a class name
>
>> + def __init__(self, status, arch, cpu, soc,
>> + vendor, board, target, options, maintainer):
>> + self.status = status
>> + self.arch = arch
>> + self.cpu = cpu
>> + self.soc = soc
>> + self.vendor = vendor
>> + self.board = board
>> + self.target = target
>> + self.defconfig = "{}_defconfig".format(target)
>> + self.options = options
>> + self.maintainer = maintainer
>> +
>> + def show(self, sep=' | ', props=None):
> Function comment (see other tools for style). Need to document args
> and any return value.
>
>> + if not props:
>> + print(
>> + sep.join([self.defconfig,
>> + self.vendor,
>> + self.arch,
>> + self.cpu,
>> + self.soc,
>> + self.board,
>> + self.status,
>> + self.maintainer]))
>> + else:
>> + print(sep.join([self.defconfig] + [getattr(self, prop) for prop in props]))
> Does this need to import print_function from __future__ for Python 2?
>
>> +
>> + def cleanup(self):
>> + """ remove the directory in which the cfg files have been built """
> Please use comment style from other tools. Same below.
>
>> + shutil.rmtree(self.temp_dir)
>> +
>> + def match(self, rules):
>> + """ return True if the board match all the criteria """
>> + for prop, r in rules:
>> + val = getattr(self, prop)
>> + if not val or val == "-":
>> + return False
>> + if not r.match(val):
>> + return False
>> + return True
>> +
>> +
>> +def get_all_boards():
>> + """ extract a list of boards from 'boards.cfg' """
>> + result = []
>> + if not os.path.isfile("boards.cfg"):
>> + os.system('tools/genboardscfg.py')
>> +
>> + with open('boards.cfg', 'r') as f:
>> + for l in f.readlines():
>> + if not l or l[0] == "#":
>> + continue
>> + props = l.strip().split(None, 8)
>> + if not props:
>> + continue
>> + if len(props) < 9:
>> + props.extend(["-"] * (9 - len(props)))
>> + result.append(board(*props))
>> + return result
>> +
>> +
>> +def get_default_options():
>> + return ["board", "soc", "vendor", "arch", "cpu", "target"]
>> +
>> +
>> +def update_parser_with_default_options(parser):
>> + parser.add_argument('-i', '--ignore-case', action="store_true")
>> + parser.add_argument("--soc",
>> + help="regexp to filter on SoC. ex: 'omap[45]' to inspect omap5 and omap5 targets")
>> + parser.add_argument("--vendor", help="regexp to filter on Vendor.")
>> + parser.add_argument("--arch", help="regexp to filter on Arch")
>> + parser.add_argument("--cpu", help="regexp to filter on CPU")
>> + parser.add_argument("--board", help="regexp to filter on Board")
>> + parser.add_argument("--target",
>> + help="regexp to filter on Target (defconfig filename without the '_defconfig' suffix)")
>> +
>> +
>> +def get_matching_boards(args, fields=get_default_options()):
>> + # compile a list of regexp used to filter the targets
>> + boards = []
>> + rules = []
>> + for f in fields:
>> + arg = getattr(args, f)
>> + if arg:
>> + rules.append((f, re.compile("\\b{}\\b".format(arg),
>> + re.IGNORECASE if args.ignore_case else 0)))
>> +
>> + # get a list of boards matching the rules
>> + for b in get_all_boards():
>> + if b.match(rules):
>> + boards.append(b)
>> + return boards
>> +
>> +
>> +def main():
>> + parser = argparse.ArgumentParser(description="Show CONFIG options usage")
>> + update_parser_with_default_options(parser)
>> + parser.add_argument("--maintainer", help="regexp to filter on maintainer.")
>> + parser.add_argument("--status", help="regexp to filter on status.")
>> + parser.add_argument("--show-details", help="show fields used as filter",
>> + action="store_true")
>> + parser.add_argument("--show-all", help="show all fields",
>> + action="store_true")
>> + args = parser.parse_args()
>> + fields = get_default_options() + ["status", "maintainer"]
>> +
>> + for b in get_matching_boards(args, fields):
>> + if args.show_details:
>> + props = [f for f in fields if getattr(args, f)]
>> + b.show(props=props)
>> + elif args.show_all:
>> + b.show()
>> + else:
>> + print(b.defconfig)
>> +
>> +if __name__ == '__main__':
>> + main()
>> --
>> 2.7.4
>>
> Regards,
> Simon
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* [U-Boot] [RFC PATCH v2 2/3] tools: Add a tool to get a list of defconfigs based on filters
2018-10-09 16:20 ` Simon Glass
2018-10-18 12:03 ` Jean-Jacques Hiblot
@ 2018-10-18 14:36 ` Jean-Jacques Hiblot
1 sibling, 0 replies; 10+ messages in thread
From: Jean-Jacques Hiblot @ 2018-10-18 14:36 UTC (permalink / raw)
To: u-boot
On 09/10/2018 18:20, Simon Glass wrote:
> Hi Jean-Jacques,
>
> On 3 October 2018 at 07:53, Jean-Jacques Hiblot <jjhiblot@ti.com> wrote:
>> The possible filters are "arch", "vendor", "soc", "cpu" and "arch".
>>
>> The list of all the defconfigs is read from boards.cfg. If this file
>> doesn't exist, then tools/genboardscfg.py is called to generate it.
>>
>> Signed-off-by: Jean-Jacques Hiblot <jjhiblot@ti.com>
>> ---
>>
>> Changes in v2: None
>>
>> tools/find_defconfigs.py | 167 +++++++++++++++++++++++++++++++++++++++++++++++
>> 1 file changed, 167 insertions(+)
>> create mode 100755 tools/find_defconfigs.py
> This looks good, but I have some style comments below.
>
> Also it seems to do a similar thing to tools/buildman/board.py. Should
> we replace that impl with what you have here? It looks more flexible
> that what buildman currently provides.
>
>> diff --git a/tools/find_defconfigs.py b/tools/find_defconfigs.py
>> new file mode 100755
>> index 0000000..9d68cef
>> --- /dev/null
>> +++ b/tools/find_defconfigs.py
>> @@ -0,0 +1,167 @@
>> +#!/usr/bin/env python
>> +# SPDX-License-Identifier: GPL-2.0+
>> +#
>> +# Author: JJ Hiblot <jjhiblot@ti.com>
>> +#
>> +
>> +"""
>> +Output a list of defconfig matching criteria.
> I think you mean defconfig-matching?
>
>> +
>> +The possible criteria are soc, vendor, arch, cpu, board and defconfig name.
>> +The criteria are expressed as regexp, allowing for complex selection.
>> +
>> +How does it work?
>> +-----------------
>> +
>> +This tools uses the boards.cfg file produced by tools/genboardscfg.py
>> +It reads the file to get a list of all the defconfigs and the information
>> +about the soc, vendor etc. for each of them.
>> +Then it walks this list and outputs the defconfigs for which the info match
>> +the regexp passed to the program.
>> +
>> +examples:
>> +---------
>> +
>> +1) Get the list of defconfigs for boards built around omap5, omap4 and k3, not built by TI
>> +
>> +$ tools/find_defconfigs.py --soc 'omap[45]|k3' --vendor '(?!ti)'
>> +kc1_defconfig
>> +duovero_defconfig
>> +cl-som-am57x_defconfig
>> +cm_t54_defconfig
>> +
>> +2) Same list but iwth more details on the items that were used as filters
>> +
>> +$ tools/find_defconfigs.py --soc 'omap[45]|k3' --vendor '(?!ti)' --show-details
>> +kc1_defconfig | omap4 | amazon
>> +duovero_defconfig | omap4 | gumstix
>> +cl-som-am57x_defconfig | omap5 | compulab
>> +cm_t54_defconfig | omap5 | compulab
>> +
>> +
>> +"""
>> +
>> +import re
>> +import os
>> +import argparse
> Please sort these
>
>> +
>> +
>> +class board:
>> +
> Need a class comment here, also use Board since it is a class name
>
>> + def __init__(self, status, arch, cpu, soc,
>> + vendor, board, target, options, maintainer):
>> + self.status = status
>> + self.arch = arch
>> + self.cpu = cpu
>> + self.soc = soc
>> + self.vendor = vendor
>> + self.board = board
>> + self.target = target
>> + self.defconfig = "{}_defconfig".format(target)
>> + self.options = options
>> + self.maintainer = maintainer
>> +
>> + def show(self, sep=' | ', props=None):
> Function comment (see other tools for style). Need to document args
> and any return value.
>
>> + if not props:
>> + print(
>> + sep.join([self.defconfig,
>> + self.vendor,
>> + self.arch,
>> + self.cpu,
>> + self.soc,
>> + self.board,
>> + self.status,
>> + self.maintainer]))
>> + else:
>> + print(sep.join([self.defconfig] + [getattr(self, prop) for prop in props]))
> Does this need to import print_function from __future__ for Python 2?
No it doesn't. You can run this program as-is with python2 or python3.
>
>> +
>> + def cleanup(self):
>> + """ remove the directory in which the cfg files have been built """
> Please use comment style from other tools. Same below.
>
>> + shutil.rmtree(self.temp_dir)
>> +
>> + def match(self, rules):
>> + """ return True if the board match all the criteria """
>> + for prop, r in rules:
>> + val = getattr(self, prop)
>> + if not val or val == "-":
>> + return False
>> + if not r.match(val):
>> + return False
>> + return True
>> +
>> +
>> +def get_all_boards():
>> + """ extract a list of boards from 'boards.cfg' """
>> + result = []
>> + if not os.path.isfile("boards.cfg"):
>> + os.system('tools/genboardscfg.py')
>> +
>> + with open('boards.cfg', 'r') as f:
>> + for l in f.readlines():
>> + if not l or l[0] == "#":
>> + continue
>> + props = l.strip().split(None, 8)
>> + if not props:
>> + continue
>> + if len(props) < 9:
>> + props.extend(["-"] * (9 - len(props)))
>> + result.append(board(*props))
>> + return result
>> +
>> +
>> +def get_default_options():
>> + return ["board", "soc", "vendor", "arch", "cpu", "target"]
>> +
>> +
>> +def update_parser_with_default_options(parser):
>> + parser.add_argument('-i', '--ignore-case', action="store_true")
>> + parser.add_argument("--soc",
>> + help="regexp to filter on SoC. ex: 'omap[45]' to inspect omap5 and omap5 targets")
>> + parser.add_argument("--vendor", help="regexp to filter on Vendor.")
>> + parser.add_argument("--arch", help="regexp to filter on Arch")
>> + parser.add_argument("--cpu", help="regexp to filter on CPU")
>> + parser.add_argument("--board", help="regexp to filter on Board")
>> + parser.add_argument("--target",
>> + help="regexp to filter on Target (defconfig filename without the '_defconfig' suffix)")
>> +
>> +
>> +def get_matching_boards(args, fields=get_default_options()):
>> + # compile a list of regexp used to filter the targets
>> + boards = []
>> + rules = []
>> + for f in fields:
>> + arg = getattr(args, f)
>> + if arg:
>> + rules.append((f, re.compile("\\b{}\\b".format(arg),
>> + re.IGNORECASE if args.ignore_case else 0)))
>> +
>> + # get a list of boards matching the rules
>> + for b in get_all_boards():
>> + if b.match(rules):
>> + boards.append(b)
>> + return boards
>> +
>> +
>> +def main():
>> + parser = argparse.ArgumentParser(description="Show CONFIG options usage")
>> + update_parser_with_default_options(parser)
>> + parser.add_argument("--maintainer", help="regexp to filter on maintainer.")
>> + parser.add_argument("--status", help="regexp to filter on status.")
>> + parser.add_argument("--show-details", help="show fields used as filter",
>> + action="store_true")
>> + parser.add_argument("--show-all", help="show all fields",
>> + action="store_true")
>> + args = parser.parse_args()
>> + fields = get_default_options() + ["status", "maintainer"]
>> +
>> + for b in get_matching_boards(args, fields):
>> + if args.show_details:
>> + props = [f for f in fields if getattr(args, f)]
>> + b.show(props=props)
>> + elif args.show_all:
>> + b.show()
>> + else:
>> + print(b.defconfig)
>> +
>> +if __name__ == '__main__':
>> + main()
>> --
>> 2.7.4
>>
> Regards,
> Simon
>
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2018-10-18 14:36 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-10-03 13:53 [U-Boot] [RFC PATCH v2 0/3] python tools to inspect configs Jean-Jacques Hiblot
2018-10-03 13:53 ` [U-Boot] [RFC PATCH v2 1/3] tools: moveconfig: Add an option to build a fuller database of options Jean-Jacques Hiblot
2018-10-09 16:20 ` Simon Glass
2018-10-03 13:53 ` [U-Boot] [RFC PATCH v2 2/3] tools: Add a tool to get a list of defconfigs based on filters Jean-Jacques Hiblot
2018-10-09 16:20 ` Simon Glass
2018-10-18 12:03 ` Jean-Jacques Hiblot
2018-10-18 14:36 ` Jean-Jacques Hiblot
2018-10-03 13:53 ` [U-Boot] [RFC PATCH v2 3/3] tools: Add a tool to get an overview of the usage of CONFIG options Jean-Jacques Hiblot
2018-10-09 16:20 ` Simon Glass
2018-10-18 11:38 ` Jean-Jacques Hiblot
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.