[U-Boot] [PATCH 1/2] tools: moveconfig: support wildcards in --defconfigs file

[U-Boot] [PATCH 1/2] tools: moveconfig: support wildcards in --defconfigs file

[Masahiro Yamada]

Supporting shell-style wildcards for the --defconfigs option will be
useful to run the moveconfig tool against a specific platform.  For
example, "uniphier*" in the file passed by --defconfigs option will
be expanded to defconfig files that start with "uniphier".  This is
easier than listing out all the UniPhier defconfig files.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 tools/moveconfig.py | 23 +++++++++++++++--------
 1 file changed, 15 insertions(+), 8 deletions(-)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 87e2bb2..67cf5f0 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -127,7 +127,8 @@ Available options
    standard commit message is used which may need to be edited.
 
  -d, --defconfigs
-  Specify a file containing a list of defconfigs to move
+  Specify a file containing a list of defconfigs to move.  The defconfig
+  files can be given with shell-style wildcards.
 
  -n, --dry-run
    Perform a trial run that does not make any changes.  It is useful to
@@ -180,6 +181,7 @@ import copy
 import difflib
 import filecmp
 import fnmatch
+import glob
 import multiprocessing
 import optparse
 import os
@@ -284,6 +286,17 @@ def get_make_cmd():
         sys.exit('GNU Make not found')
     return ret[0].rstrip()
 
+def get_matched_defconfigs(defconfigs_file):
+    """Get all the defconfig files that match the patterns in a file."""
+    # use a set rather than a list in order to avoid double-counting
+    defconfigs = set()
+    for line in open(defconfigs_file):
+        line = os.path.join('configs', line.strip())
+        defconfigs |= set(glob.glob(line))
+        defconfigs |= set(glob.glob(line + '_defconfig'))
+
+    return [ defconfig[len('configs') + 1:]  for defconfig in defconfigs ]
+
 def get_all_defconfigs():
     """Get all the defconfig files under the configs/ directory."""
     defconfigs = []
@@ -1204,13 +1217,7 @@ def move_config(configs, options):
         reference_src_dir = None
 
     if options.defconfigs:
-        defconfigs = [line.strip() for line in open(options.defconfigs)]
-        for i, defconfig in enumerate(defconfigs):
-            if not defconfig.endswith('_defconfig'):
-                defconfigs[i] = defconfig + '_defconfig'
-            if not os.path.exists(os.path.join('configs', defconfigs[i])):
-                sys.exit('%s - defconfig does not exist. Stopping.' %
-                         defconfigs[i])
+        defconfigs = get_matched_defconfigs(options.defconfigs)
     else:
         defconfigs = get_all_defconfigs()
 
-- 
1.9.1

[U-Boot] [PATCH 2/2] tools: moveconfig: process defconfig files in alphabetical order

[Masahiro Yamada]

This will make the log more readable.  The log is not deterministic
in case of parallel processing, though.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 tools/moveconfig.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 67cf5f0..49cb1e2 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -1221,6 +1221,8 @@ def move_config(configs, options):
     else:
         defconfigs = get_all_defconfigs()
 
+    defconfigs.sort()
+
     progress = Progress(len(defconfigs))
     slots = Slots(configs, options, progress, reference_src_dir)
 
-- 
1.9.1

[U-Boot] [PATCH 2/2] tools: moveconfig: process defconfig files in alphabetical order

[Tom Rini]

On Mon, Oct 17, 2016 at 02:17:23AM +0900, Masahiro Yamada wrote:
> This will make the log more readable.  The log is not deterministic
> in case of parallel processing, though.
> 
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

I'm honestly ambivalent about this since single threaded is basically
never the case, right?

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20161016/d380d513/attachment.sig>

[U-Boot] [PATCH 1/2] tools: moveconfig: support wildcards in --defconfigs file

[Tom Rini]

On Mon, Oct 17, 2016 at 02:17:22AM +0900, Masahiro Yamada wrote:

> Supporting shell-style wildcards for the --defconfigs option will be
> useful to run the moveconfig tool against a specific platform.  For
> example, "uniphier*" in the file passed by --defconfigs option will
> be expanded to defconfig files that start with "uniphier".  This is
> easier than listing out all the UniPhier defconfig files.
> 
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
> 
>  tools/moveconfig.py | 23 +++++++++++++++--------
>  1 file changed, 15 insertions(+), 8 deletions(-)
> 
> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
> index 87e2bb2..67cf5f0 100755
> --- a/tools/moveconfig.py
> +++ b/tools/moveconfig.py
> @@ -127,7 +127,8 @@ Available options
>     standard commit message is used which may need to be edited.
>  
>   -d, --defconfigs
> -  Specify a file containing a list of defconfigs to move
> +  Specify a file containing a list of defconfigs to move.  The defconfig
> +  files can be given with shell-style wildcards.
>  
>   -n, --dry-run
>     Perform a trial run that does not make any changes.  It is useful to
> @@ -180,6 +181,7 @@ import copy
>  import difflib
>  import filecmp
>  import fnmatch
> +import glob
>  import multiprocessing
>  import optparse
>  import os
> @@ -284,6 +286,17 @@ def get_make_cmd():
>          sys.exit('GNU Make not found')
>      return ret[0].rstrip()
>  
> +def get_matched_defconfigs(defconfigs_file):
> +    """Get all the defconfig files that match the patterns in a file."""
> +    # use a set rather than a list in order to avoid double-counting
> +    defconfigs = set()
> +    for line in open(defconfigs_file):
> +        line = os.path.join('configs', line.strip())
> +        defconfigs |= set(glob.glob(line))
> +        defconfigs |= set(glob.glob(line + '_defconfig'))
> +
> +    return [ defconfig[len('configs') + 1:]  for defconfig in defconfigs ]
> +
>  def get_all_defconfigs():
>      """Get all the defconfig files under the configs/ directory."""
>      defconfigs = []
> @@ -1204,13 +1217,7 @@ def move_config(configs, options):
>          reference_src_dir = None
>  
>      if options.defconfigs:
> -        defconfigs = [line.strip() for line in open(options.defconfigs)]
> -        for i, defconfig in enumerate(defconfigs):
> -            if not defconfig.endswith('_defconfig'):
> -                defconfigs[i] = defconfig + '_defconfig'
> -            if not os.path.exists(os.path.join('configs', defconfigs[i])):
> -                sys.exit('%s - defconfig does not exist. Stopping.' %
> -                         defconfigs[i])
> +        defconfigs = get_matched_defconfigs(options.defconfigs)

I don't see error handling in the case of files (and now globs) not
existing now, is this just handled by other parts of the code?  Thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20161016/c0865ed5/attachment.sig>

[U-Boot] [PATCH 1/2] tools: moveconfig: support wildcards in --defconfigs file

[Masahiro Yamada]

2016-10-17 5:15 GMT+09:00 Tom Rini <trini@konsulko.com>:
> On Mon, Oct 17, 2016 at 02:17:22AM +0900, Masahiro Yamada wrote:
>
>> Supporting shell-style wildcards for the --defconfigs option will be
>> useful to run the moveconfig tool against a specific platform.  For
>> example, "uniphier*" in the file passed by --defconfigs option will
>> be expanded to defconfig files that start with "uniphier".  This is
>> easier than listing out all the UniPhier defconfig files.
>>
>> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>> ---
>>
>>  tools/moveconfig.py | 23 +++++++++++++++--------
>>  1 file changed, 15 insertions(+), 8 deletions(-)
>>
>> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
>> index 87e2bb2..67cf5f0 100755
>> --- a/tools/moveconfig.py
>> +++ b/tools/moveconfig.py
>> @@ -127,7 +127,8 @@ Available options
>>     standard commit message is used which may need to be edited.
>>
>>   -d, --defconfigs
>> -  Specify a file containing a list of defconfigs to move
>> +  Specify a file containing a list of defconfigs to move.  The defconfig
>> +  files can be given with shell-style wildcards.
>>
>>   -n, --dry-run
>>     Perform a trial run that does not make any changes.  It is useful to
>> @@ -180,6 +181,7 @@ import copy
>>  import difflib
>>  import filecmp
>>  import fnmatch
>> +import glob
>>  import multiprocessing
>>  import optparse
>>  import os
>> @@ -284,6 +286,17 @@ def get_make_cmd():
>>          sys.exit('GNU Make not found')
>>      return ret[0].rstrip()
>>
>> +def get_matched_defconfigs(defconfigs_file):
>> +    """Get all the defconfig files that match the patterns in a file."""
>> +    # use a set rather than a list in order to avoid double-counting
>> +    defconfigs = set()
>> +    for line in open(defconfigs_file):
>> +        line = os.path.join('configs', line.strip())
>> +        defconfigs |= set(glob.glob(line))
>> +        defconfigs |= set(glob.glob(line + '_defconfig'))
>> +
>> +    return [ defconfig[len('configs') + 1:]  for defconfig in defconfigs ]
>> +
>>  def get_all_defconfigs():
>>      """Get all the defconfig files under the configs/ directory."""
>>      defconfigs = []
>> @@ -1204,13 +1217,7 @@ def move_config(configs, options):
>>          reference_src_dir = None
>>
>>      if options.defconfigs:
>> -        defconfigs = [line.strip() for line in open(options.defconfigs)]
>> -        for i, defconfig in enumerate(defconfigs):
>> -            if not defconfig.endswith('_defconfig'):
>> -                defconfigs[i] = defconfig + '_defconfig'
>> -            if not os.path.exists(os.path.join('configs', defconfigs[i])):
>> -                sys.exit('%s - defconfig does not exist. Stopping.' %
>> -                         defconfigs[i])
>> +        defconfigs = get_matched_defconfigs(options.defconfigs)
>
> I don't see error handling in the case of files (and now globs) not
> existing now, is this just handled by other parts of the code?  Thanks!


No, it will be removed.

With this patch, each line will be a pattern
that may match multiple defconfigs,
or that may match nothing.

Is it important to check users' possible typos?

[U-Boot] [PATCH 2/2] tools: moveconfig: process defconfig files in alphabetical order

[Masahiro Yamada]

Hi Tom,

2016-10-17 5:12 GMT+09:00 Tom Rini <trini@konsulko.com>:
> On Mon, Oct 17, 2016 at 02:17:23AM +0900, Masahiro Yamada wrote:
>> This will make the log more readable.  The log is not deterministic
>> in case of parallel processing, though.
>>
>> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>
> I'm honestly ambivalent about this since single threaded is basically
> never the case, right?


Right.
If you want to diff two logs, you need to pass -j1 option.

But, my main motivation is not that.
You will be able to find the log of boards you are interested in.

For example, when some boards are listed in "suspicious results"
at the end of conversion, you may want to check the log
to see what happened.  If the log is sorted *almost* alphabetically,
it will be easier to jump there.



-- 
Best Regards
Masahiro Yamada

[U-Boot] [PATCH 1/2] tools: moveconfig: support wildcards in --defconfigs file

[Tom Rini]

On Wed, Oct 19, 2016 at 01:14:58AM +0900, Masahiro Yamada wrote:
> 2016-10-17 5:15 GMT+09:00 Tom Rini <trini@konsulko.com>:
> > On Mon, Oct 17, 2016 at 02:17:22AM +0900, Masahiro Yamada wrote:
> >
> >> Supporting shell-style wildcards for the --defconfigs option will be
> >> useful to run the moveconfig tool against a specific platform.  For
> >> example, "uniphier*" in the file passed by --defconfigs option will
> >> be expanded to defconfig files that start with "uniphier".  This is
> >> easier than listing out all the UniPhier defconfig files.
> >>
> >> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> >> ---
> >>
> >>  tools/moveconfig.py | 23 +++++++++++++++--------
> >>  1 file changed, 15 insertions(+), 8 deletions(-)
> >>
> >> diff --git a/tools/moveconfig.py b/tools/moveconfig.py
> >> index 87e2bb2..67cf5f0 100755
> >> --- a/tools/moveconfig.py
> >> +++ b/tools/moveconfig.py
> >> @@ -127,7 +127,8 @@ Available options
> >>     standard commit message is used which may need to be edited.
> >>
> >>   -d, --defconfigs
> >> -  Specify a file containing a list of defconfigs to move
> >> +  Specify a file containing a list of defconfigs to move.  The defconfig
> >> +  files can be given with shell-style wildcards.
> >>
> >>   -n, --dry-run
> >>     Perform a trial run that does not make any changes.  It is useful to
> >> @@ -180,6 +181,7 @@ import copy
> >>  import difflib
> >>  import filecmp
> >>  import fnmatch
> >> +import glob
> >>  import multiprocessing
> >>  import optparse
> >>  import os
> >> @@ -284,6 +286,17 @@ def get_make_cmd():
> >>          sys.exit('GNU Make not found')
> >>      return ret[0].rstrip()
> >>
> >> +def get_matched_defconfigs(defconfigs_file):
> >> +    """Get all the defconfig files that match the patterns in a file."""
> >> +    # use a set rather than a list in order to avoid double-counting
> >> +    defconfigs = set()
> >> +    for line in open(defconfigs_file):
> >> +        line = os.path.join('configs', line.strip())
> >> +        defconfigs |= set(glob.glob(line))
> >> +        defconfigs |= set(glob.glob(line + '_defconfig'))
> >> +
> >> +    return [ defconfig[len('configs') + 1:]  for defconfig in defconfigs ]
> >> +
> >>  def get_all_defconfigs():
> >>      """Get all the defconfig files under the configs/ directory."""
> >>      defconfigs = []
> >> @@ -1204,13 +1217,7 @@ def move_config(configs, options):
> >>          reference_src_dir = None
> >>
> >>      if options.defconfigs:
> >> -        defconfigs = [line.strip() for line in open(options.defconfigs)]
> >> -        for i, defconfig in enumerate(defconfigs):
> >> -            if not defconfig.endswith('_defconfig'):
> >> -                defconfigs[i] = defconfig + '_defconfig'
> >> -            if not os.path.exists(os.path.join('configs', defconfigs[i])):
> >> -                sys.exit('%s - defconfig does not exist. Stopping.' %
> >> -                         defconfigs[i])
> >> +        defconfigs = get_matched_defconfigs(options.defconfigs)
> >
> > I don't see error handling in the case of files (and now globs) not
> > existing now, is this just handled by other parts of the code?  Thanks!
> 
> 
> No, it will be removed.
> 
> With this patch, each line will be a pattern
> that may match multiple defconfigs,
> or that may match nothing.

Ah, OK.

> Is it important to check users' possible typos?

Well, will a list of invalid config names say something useful, or fail
silently or fail in unexpected ways?

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20161018/ad00a201/attachment.sig>

[U-Boot] [PATCH 2/2] tools: moveconfig: process defconfig files in alphabetical order

[Tom Rini]

On Wed, Oct 19, 2016 at 01:23:24AM +0900, Masahiro Yamada wrote:
> Hi Tom,
> 
> 2016-10-17 5:12 GMT+09:00 Tom Rini <trini@konsulko.com>:
> > On Mon, Oct 17, 2016 at 02:17:23AM +0900, Masahiro Yamada wrote:
> >> This will make the log more readable.  The log is not deterministic
> >> in case of parallel processing, though.
> >>
> >> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> >
> > I'm honestly ambivalent about this since single threaded is basically
> > never the case, right?
> 
> 
> Right.
> If you want to diff two logs, you need to pass -j1 option.
> 
> But, my main motivation is not that.
> You will be able to find the log of boards you are interested in.
> 
> For example, when some boards are listed in "suspicious results"
> at the end of conversion, you may want to check the log
> to see what happened.  If the log is sorted *almost* alphabetically,
> it will be easier to jump there.

I'm still ambivalent.  I 'less' the log and search doesn't care about
order.  And I try and fix one broken board at a time since it's often
the case that N boards will have the same form of thinko.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20161018/c0110126/attachment.sig>

[U-Boot] [PATCH 1/2] tools: moveconfig: support wildcards in --defconfigs file

[Joe Hershberger]

On Tue, Oct 18, 2016 at 2:23 PM, Tom Rini <trini@konsulko.com> wrote:
> On Wed, Oct 19, 2016 at 01:14:58AM +0900, Masahiro Yamada wrote:
>> 2016-10-17 5:15 GMT+09:00 Tom Rini <trini@konsulko.com>:
>> > On Mon, Oct 17, 2016 at 02:17:22AM +0900, Masahiro Yamada wrote:
>> > I don't see error handling in the case of files (and now globs) not
>> > existing now, is this just handled by other parts of the code?  Thanks!
>>
>>
>> No, it will be removed.
>>
>> With this patch, each line will be a pattern
>> that may match multiple defconfigs,
>> or that may match nothing.
>
> Ah, OK.
>
>> Is it important to check users' possible typos?
>
> Well, will a list of invalid config names say something useful, or fail
> silently or fail in unexpected ways?

It seems like it would be good to warn if a glob (or otherwise)
results in an empty set, because that it probably not what was
intended.

-Joe

[U-Boot] [PATCH 2/2] tools: moveconfig: process defconfig files in alphabetical order

[Masahiro Yamada]

2016-10-19 4:25 GMT+09:00 Tom Rini <trini@konsulko.com>:
> On Wed, Oct 19, 2016 at 01:23:24AM +0900, Masahiro Yamada wrote:
>> Hi Tom,
>>
>> 2016-10-17 5:12 GMT+09:00 Tom Rini <trini@konsulko.com>:
>> > On Mon, Oct 17, 2016 at 02:17:23AM +0900, Masahiro Yamada wrote:
>> >> This will make the log more readable.  The log is not deterministic
>> >> in case of parallel processing, though.
>> >>
>> >> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>> >
>> > I'm honestly ambivalent about this since single threaded is basically
>> > never the case, right?
>>
>>
>> Right.
>> If you want to diff two logs, you need to pass -j1 option.
>>
>> But, my main motivation is not that.
>> You will be able to find the log of boards you are interested in.
>>
>> For example, when some boards are listed in "suspicious results"
>> at the end of conversion, you may want to check the log
>> to see what happened.  If the log is sorted *almost* alphabetically,
>> it will be easier to jump there.
>
> I'm still ambivalent.  I 'less' the log and search doesn't care about
> order.  And I try and fix one broken board at a time since it's often
> the case that N boards will have the same form of thinko.

OK, fine.

Please drop this patch.

I sent v2 only for 1/2.



-- 
Best Regards
Masahiro Yamada