[PATCH 1/5] moveconfig: Correct operation of the 'imply' feature

[PATCH 1/5] moveconfig: Correct operation of the 'imply' feature

[Simon Glass]

This doesn't work anymore, since the Kconfig update. The script has no
tests so we did not notice. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 tools/moveconfig.py | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 373b395fda4..8d059be039d 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -1263,7 +1263,7 @@ def find_kconfig_rules(kconf, config, imply_config):
     sym = kconf.syms.get(imply_config)
     if sym:
         for sel, cond in (sym.selects + sym.implies):
-            if sel == config:
+            if sel.name == config:
                 return sym
     return None
 
@@ -1291,7 +1291,8 @@ def check_imply_rule(kconf, config, imply_config):
     nodes = sym.nodes
     if len(nodes) != 1:
         return '%d locations' % len(nodes)
-    fname, linenum = nodes[0].filename, nodes[0].linern
+    node = nodes[0]
+    fname, linenum = node.filename, node.linenr
     cwd = os.getcwd()
     if cwd and fname.startswith(cwd):
         fname = fname[len(cwd) + 1:]
@@ -1382,7 +1383,7 @@ def do_imply_config(config_list, add_imply, imply_flags, skip_added,
     """
     kconf = KconfigScanner().conf if check_kconfig else None
     if add_imply and add_imply != 'all':
-        add_imply = add_imply.split()
+        add_imply = add_imply.split(',')
 
     # key is defconfig name, value is dict of (CONFIG_xxx, value)
     config_db = {}
@@ -1414,7 +1415,7 @@ def do_imply_config(config_list, add_imply, imply_flags, skip_added,
             else:  # New defconfig
                 defconfig = line
 
-    # Work through each target config option in tern, independently
+    # Work through each target config option in turn, independently
     for config in config_list:
         defconfigs = defconfig_db.get(config)
         if not defconfigs:
-- 
2.34.1.173.g76aa8bc2d0-goog

[PATCH 2/5] moveconfig: Read the database in a separate function

[Simon Glass]

Move this code out into a function so it can be used elsewhere.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 tools/moveconfig.py | 78 ++++++++++++++++++++++++++++-----------------
 1 file changed, 49 insertions(+), 29 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 8d059be039d..4ad892e2c0c 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -1339,6 +1339,54 @@ IMPLY_FLAGS = {
         'Allow Kconfig options outside arch/ and /board/ to imply'],
 };
 
+
+def read_database():
+    """Read in the config database
+
+    Returns:
+        tuple:
+            set of all config options seen (each a str)
+            set of all defconfigs seen (each a str)
+            dict of configs for each defconfig:
+                key: defconfig name, e.g. "MPC8548CDS_legacy_defconfig"
+                value: dict:
+                    key: CONFIG option
+                    value: Value of option
+            dict of defconfigs for each config:
+                key: CONFIG option
+                value: set of boards using that option
+
+    """
+    configs = {}
+
+    # key is defconfig name, value is dict of (CONFIG_xxx, value)
+    config_db = {}
+
+    # Set of all config options we have seen
+    all_configs = set()
+
+    # Set of all defconfigs we have seen
+    all_defconfigs = set()
+
+    defconfig_db = collections.defaultdict(set)
+    with open(CONFIG_DATABASE) as fd:
+        for line in fd.readlines():
+            line = line.rstrip()
+            if not line:  # Separator between defconfigs
+                config_db[defconfig] = configs
+                all_defconfigs.add(defconfig)
+                configs = {}
+            elif line[0] == ' ':  # CONFIG line
+                config, value = line.strip().split('=', 1)
+                configs[config] = value
+                defconfig_db[config].add(defconfig)
+                all_configs.add(config)
+            else:  # New defconfig
+                defconfig = line
+
+    return all_configs, all_defconfigs, config_db, defconfig_db
+
+
 def do_imply_config(config_list, add_imply, imply_flags, skip_added,
                     check_kconfig=True, find_superset=False):
     """Find CONFIG options which imply those in the list
@@ -1385,35 +1433,7 @@ def do_imply_config(config_list, add_imply, imply_flags, skip_added,
     if add_imply and add_imply != 'all':
         add_imply = add_imply.split(',')
 
-    # key is defconfig name, value is dict of (CONFIG_xxx, value)
-    config_db = {}
-
-    # Holds a dict containing the set of defconfigs that contain each config
-    # key is config, value is set of defconfigs using that config
-    defconfig_db = collections.defaultdict(set)
-
-    # Set of all config options we have seen
-    all_configs = set()
-
-    # Set of all defconfigs we have seen
-    all_defconfigs = set()
-
-    # Read in the database
-    configs = {}
-    with open(CONFIG_DATABASE) as fd:
-        for line in fd.readlines():
-            line = line.rstrip()
-            if not line:  # Separator between defconfigs
-                config_db[defconfig] = configs
-                all_defconfigs.add(defconfig)
-                configs = {}
-            elif line[0] == ' ':  # CONFIG line
-                config, value = line.strip().split('=', 1)
-                configs[config] = value
-                defconfig_db[config].add(defconfig)
-                all_configs.add(config)
-            else:  # New defconfig
-                defconfig = line
+    all_configs, all_defconfigs, config_db, defconfig_db = read_database()
 
     # Work through each target config option in turn, independently
     for config in config_list:
-- 
2.34.1.173.g76aa8bc2d0-goog

[PATCH 3/5] moveconfig: Sort the options

[Simon Glass]

Put the options in sorted order by their short name so it is easier to
find an option.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 tools/moveconfig.py | 22 ++++++++++++----------
 1 file changed, 12 insertions(+), 10 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 4ad892e2c0c..369589727cc 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -1591,26 +1591,28 @@ def main():
                       help='a file containing a list of defconfigs to move, '
                       "one per line (for example 'snow_defconfig') "
                       "or '-' to read from stdin")
-    parser.add_option('-i', '--imply', action='store_true', default=False,
-                      help='find options which imply others')
-    parser.add_option('-I', '--imply-flags', type='string', default='',
-                      help="control the -i option ('help' for help")
-    parser.add_option('-n', '--dry-run', action='store_true', default=False,
-                      help='perform a trial run (show log with no changes)')
     parser.add_option('-e', '--exit-on-error', action='store_true',
                       default=False,
                       help='exit immediately on any error')
-    parser.add_option('-s', '--force-sync', action='store_true', default=False,
-                      help='force sync by savedefconfig')
-    parser.add_option('-S', '--spl', action='store_true', default=False,
-                      help='parse config options defined for SPL build')
     parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
                       action='store_true', default=False,
                       help='only cleanup the headers')
+    parser.add_option('-i', '--imply', action='store_true', default=False,
+                      help='find options which imply others')
+    parser.add_option('-I', '--imply-flags', type='string', default='',
+                      help="control the -i option ('help' for help")
     parser.add_option('-j', '--jobs', type='int', default=cpu_count,
                       help='the number of jobs to run simultaneously')
+    parser.add_option('-n', '--dry-run', action='store_true', default=False,
+                      help='perform a trial run (show log with no changes)')
     parser.add_option('-r', '--git-ref', type='string',
                       help='the git ref to clone for building the autoconf.mk')
+    parser.add_option('-s', '--force-sync', action='store_true', default=False,
+                      help='force sync by savedefconfig')
+    parser.add_option('-S', '--spl', action='store_true', default=False,
+                      help='parse config options defined for SPL build')
+    parser.add_option('-t', '--test', action='store_true', default=False,
+                      help='run unit tests')
     parser.add_option('-y', '--yes', action='store_true', default=False,
                       help="respond 'yes' to any prompts")
     parser.add_option('-v', '--verbose', action='store_true', default=False,
-- 
2.34.1.173.g76aa8bc2d0-goog

[PATCH 4/5] moveconfig: Allow adding unit tests

[Simon Glass]

Add a -t option to run unit tests in this program. So far, there is none.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 tools/moveconfig.py | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 369589727cc..71a7736ca63 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -14,6 +14,7 @@ import asteval
 import collections
 import copy
 import difflib
+import doctest
 import filecmp
 import fnmatch
 import glob
@@ -28,6 +29,7 @@ import sys
 import tempfile
 import threading
 import time
+import unittest
 
 from buildman import bsettings
 from buildman import kconfiglib
@@ -1621,6 +1623,13 @@ def main():
 
     (options, configs) = parser.parse_args()
 
+    if options.test:
+        sys.argv = [sys.argv[0]]
+        fail, count = doctest.testmod()
+        if fail:
+            return 1
+        unittest.main()
+
     if len(configs) == 0 and not any((options.force_sync, options.build_db,
                                       options.imply)):
         parser.print_usage()
-- 
2.34.1.173.g76aa8bc2d0-goog

[PATCH 5/5] moveconfig: Allow querying board configuration

[Simon Glass]

It is useful to be able to find out which boards define a particular
option, or combination of options. This is not as easy as grepping the
defconfig files since many options are implied by others.

Add a -f option to the moveconfig tool to permit this. Update the
documentation to cover this, including a better title for the doc page.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

 doc/develop/moveconfig.rst | 25 ++++++++++-
 tools/moveconfig.py        | 86 ++++++++++++++++++++++++++++++++++++--
 2 files changed, 105 insertions(+), 6 deletions(-)

diff --git a/doc/develop/moveconfig.rst b/doc/develop/moveconfig.rst
index dcd4d927e40..2f53ea52b71 100644
--- a/doc/develop/moveconfig.rst
+++ b/doc/develop/moveconfig.rst
@@ -1,7 +1,7 @@
 .. SPDX-License-Identifier: GPL-2.0+
 
-moveconfig
-==========
+moveconfig - Migrating and querying CONFIG options
+==================================================
 
 Since Kconfig was introduced to U-Boot, we have worked on moving
 config options from headers to Kconfig (defconfig).
@@ -129,6 +129,24 @@ To process CONFIG_CMD_FPGAD only for a subset of configs based on path match::
        ./tools/moveconfig.py -Cy CONFIG_CMD_FPGAD -d -
 
 
+Finding boards with particular CONFIG combinations
+--------------------------------------------------
+
+You can use `moveconfig.py` to figure out which boards have a CONFIG enabled, or
+which do not. To use it, first build a database::
+
+    ./tools/moveconfig.py -b
+
+Then you can run queries using the `-f` flag followed by a list of CONFIG terms.
+Each term is CONFIG name, with or without a tilde (~) prefix. The tool searches
+for boards which match the CONFIG name, or do not match if tilde is used. For
+example, to find boards which enabled CONFIG_SCSI but not CONFIG_BLK::
+
+    tools/moveconfig.py -f SCSI ~BLK
+    3 matches
+    pg_wcom_seli8_defconfig highbank_defconfig pg_wcom_expu1_defconfig
+
+
 Finding implied CONFIGs
 -----------------------
 
@@ -235,6 +253,9 @@ Available options
   Specify a file containing a list of defconfigs to move.  The defconfig
   files can be given with shell-style wildcards. Use '-' to read from stdin.
 
+ -f, --find
+   Find boards with a given config combination
+
  -n, --dry-run
    Perform a trial run that does not make any changes.  It is useful to
    see what is going to happen before one actually runs it.
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 71a7736ca63..a86c07caa6e 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -1569,6 +1569,79 @@ def do_imply_config(config_list, add_imply, imply_flags, skip_added,
                 add_imply_rule(config[CONFIG_LEN:], fname, linenum)
 
 
+def do_find_config(config_list):
+    """Find boards with a given combination of CONFIGs
+
+    Params:
+        config_list: List of CONFIG options to check (each a string consisting
+            of a config option, with or without a CONFIG_ prefix. If an option
+            is preceded by a tilde (~) then it must be false, otherwise it must
+            be true)
+    """
+    all_configs, all_defconfigs, config_db, defconfig_db = read_database()
+
+    # Get the whitelist
+    with open('scripts/config_whitelist.txt') as inf:
+        adhoc_configs = set(inf.read().splitlines())
+
+    # Start with all defconfigs
+    out = all_defconfigs
+
+    # Work through each config in turn
+    adhoc = []
+    for item in config_list:
+        # Get the real config name and whether we want this config or not
+        cfg = item
+        want = True
+        if cfg[0] == '~':
+            want = False
+            cfg = cfg[1:]
+
+        if cfg in adhoc_configs:
+            adhoc.append(cfg)
+            continue
+
+        # Search everything that is still in the running. If it has a config
+        # that we want, or doesn't have one that we don't, add it into the
+        # running for the next stage
+        in_list = out
+        out = set()
+        for defc in in_list:
+            has_cfg = cfg in config_db[defc]
+            if has_cfg == want:
+                out.add(defc)
+    if adhoc:
+        print(f"Error: Not in Kconfig: %s" % ' '.join(adhoc))
+    else:
+        print(f'{len(out)} matches')
+        print(' '.join(out))
+
+
+def prefix_config(cfg):
+    """Prefix a config with CONFIG_ if needed
+
+    This handles ~ operator, which indicates that the CONFIG should be disabled
+
+    >>> prefix_config('FRED')
+    'CONFIG_FRED'
+    >>> prefix_config('CONFIG_FRED')
+    'CONFIG_FRED'
+    >>> prefix_config('~FRED')
+    '~CONFIG_FRED'
+    >>> prefix_config('~CONFIG_FRED')
+    '~CONFIG_FRED'
+    >>> prefix_config('A123')
+    'CONFIG_A123'
+    """
+    op = ''
+    if cfg[0] == '~':
+        op = cfg[0]
+        cfg = cfg[1:]
+    if not cfg.startswith('CONFIG_'):
+        cfg = 'CONFIG_' + cfg
+    return op + cfg
+
+
 def main():
     try:
         cpu_count = multiprocessing.cpu_count()
@@ -1596,6 +1669,8 @@ def main():
     parser.add_option('-e', '--exit-on-error', action='store_true',
                       default=False,
                       help='exit immediately on any error')
+    parser.add_option('-f', '--find', action='store_true', default=False,
+                      help='Find boards with a given config combination')
     parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
                       action='store_true', default=False,
                       help='only cleanup the headers')
@@ -1631,13 +1706,12 @@ def main():
         unittest.main()
 
     if len(configs) == 0 and not any((options.force_sync, options.build_db,
-                                      options.imply)):
+                                      options.imply, options.find)):
         parser.print_usage()
         sys.exit(1)
 
     # prefix the option name with CONFIG_ if missing
-    configs = [ config if config.startswith('CONFIG_') else 'CONFIG_' + config
-                for config in configs ]
+    configs = [prefix_config(cfg) for cfg in configs]
 
     check_top_directory()
 
@@ -1663,6 +1737,10 @@ def main():
                         options.skip_added)
         return
 
+    if options.find:
+        do_find_config(configs)
+        return
+
     config_db = {}
     db_queue = queue.Queue()
     t = DatabaseThread(config_db, db_queue)
@@ -1705,4 +1783,4 @@ def main():
                 fd.write('\n')
 
 if __name__ == '__main__':
-    main()
+    sys.exit(main())
-- 
2.34.1.173.g76aa8bc2d0-goog

Re: [PATCH 1/5] moveconfig: Correct operation of the 'imply' feature

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 279 bytes --]

On Sat, Dec 18, 2021 at 08:09:42AM -0700, Simon Glass wrote:

> This doesn't work anymore, since the Kconfig update. The script has no
> tests so we did not notice. Fix it.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>

Applied to u-boot/master, thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH 2/5] moveconfig: Read the database in a separate function

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 233 bytes --]

On Sat, Dec 18, 2021 at 08:09:43AM -0700, Simon Glass wrote:

> Move this code out into a function so it can be used elsewhere.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>

Applied to u-boot/master, thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH 3/5] moveconfig: Sort the options

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 259 bytes --]

On Sat, Dec 18, 2021 at 08:09:44AM -0700, Simon Glass wrote:

> Put the options in sorted order by their short name so it is easier to
> find an option.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>

Applied to u-boot/master, thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH 4/5] moveconfig: Allow adding unit tests

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 243 bytes --]

On Sat, Dec 18, 2021 at 08:09:45AM -0700, Simon Glass wrote:

> Add a -t option to run unit tests in this program. So far, there is none.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>

Applied to u-boot/master, thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH 5/5] moveconfig: Allow querying board configuration

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 521 bytes --]

On Sat, Dec 18, 2021 at 08:09:46AM -0700, Simon Glass wrote:

> It is useful to be able to find out which boards define a particular
> option, or combination of options. This is not as easy as grepping the
> defconfig files since many options are implied by others.
> 
> Add a -f option to the moveconfig tool to permit this. Update the
> documentation to cover this, including a better title for the doc page.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>

Applied to u-boot/master, thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]