[PATCH] kbuild: avoid regex RS for POSIX awk

[PATCH] kbuild: avoid regex RS for POSIX awk

[Kevin Locke]

In 22f26f21774f8 awk was added to deduplicate *.mod files.  The awk
invocation passes -v RS='( |\n)' to match a space or newline character
as the record separator.  Unfortunately, POSIX states[1]

> If RS contains more than one character, the results are unspecified.

Some implementations (such as the One True Awk[2] used by the BSDs) do
not treat RS as a regular expression.  When awk does not support regex
RS, build failures such as the following are produced (first error using
allmodconfig):

      CC [M]  arch/x86/events/intel/uncore.o
      CC [M]  arch/x86/events/intel/uncore_nhmex.o
      CC [M]  arch/x86/events/intel/uncore_snb.o
      CC [M]  arch/x86/events/intel/uncore_snbep.o
      CC [M]  arch/x86/events/intel/uncore_discovery.o
      LD [M]  arch/x86/events/intel/intel-uncore.o
    ld: cannot find uncore_nhmex.o: No such file or directory
    ld: cannot find uncore_snb.o: No such file or directory
    ld: cannot find uncore_snbep.o: No such file or directory
    ld: cannot find uncore_discovery.o: No such file or directory
    make[3]: *** [scripts/Makefile.build:422: arch/x86/events/intel/intel-uncore.o] Error 1
    make[2]: *** [scripts/Makefile.build:487: arch/x86/events/intel] Error 2
    make[1]: *** [scripts/Makefile.build:487: arch/x86/events] Error 2
    make: *** [Makefile:1839: arch/x86] Error 2

To avoid this, use printf(1) to produce a newline between each object
path, instead of the space produced by echo(1), so that the default RS
can be used by awk.

[1]: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/awk.html
[2]: https://github.com/onetrueawk/awk

Fixes: 22f26f21774f ("kbuild: get rid of duplication in *.mod files")
Signed-off-by: Kevin Locke <kevin@kevinlocke.name>
---
 scripts/Makefile.build | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index 1f01ac65c0cd..cac070aee791 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -251,8 +251,8 @@ $(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
 
 # To make this rule robust against "Argument list too long" error,
 # ensure to add $(obj)/ prefix by a shell command.
-cmd_mod = echo $(call real-search, $*.o, .o, -objs -y -m) | \
-	$(AWK) -v RS='( |\n)' '!x[$$0]++ { print("$(obj)/"$$0) }' > $@
+cmd_mod = printf '%s\n' $(call real-search, $*.o, .o, -objs -y -m) | \
+	$(AWK) '!x[$$0]++ { print("$(obj)/"$$0) }' > $@
 
 $(obj)/%.mod: FORCE
 	$(call if_changed,mod)
-- 
2.35.1

Re: [PATCH] kbuild: avoid regex RS for POSIX awk

[Masahiro Yamada]

On Tue, Jun 7, 2022 at 11:43 AM Kevin Locke <kevin@kevinlocke.name> wrote:
>
> In 22f26f21774f8 awk was added to deduplicate *.mod files.  The awk
> invocation passes -v RS='( |\n)' to match a space or newline character
> as the record separator.  Unfortunately, POSIX states[1]
>
> > If RS contains more than one character, the results are unspecified.
>
> Some implementations (such as the One True Awk[2] used by the BSDs) do
> not treat RS as a regular expression.  When awk does not support regex
> RS, build failures such as the following are produced (first error using
> allmodconfig):
>
>       CC [M]  arch/x86/events/intel/uncore.o
>       CC [M]  arch/x86/events/intel/uncore_nhmex.o
>       CC [M]  arch/x86/events/intel/uncore_snb.o
>       CC [M]  arch/x86/events/intel/uncore_snbep.o
>       CC [M]  arch/x86/events/intel/uncore_discovery.o
>       LD [M]  arch/x86/events/intel/intel-uncore.o
>     ld: cannot find uncore_nhmex.o: No such file or directory
>     ld: cannot find uncore_snb.o: No such file or directory
>     ld: cannot find uncore_snbep.o: No such file or directory
>     ld: cannot find uncore_discovery.o: No such file or directory
>     make[3]: *** [scripts/Makefile.build:422: arch/x86/events/intel/intel-uncore.o] Error 1
>     make[2]: *** [scripts/Makefile.build:487: arch/x86/events/intel] Error 2
>     make[1]: *** [scripts/Makefile.build:487: arch/x86/events] Error 2
>     make: *** [Makefile:1839: arch/x86] Error 2
>
> To avoid this, use printf(1) to produce a newline between each object
> path, instead of the space produced by echo(1), so that the default RS
> can be used by awk.
>
> [1]: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/awk.html
> [2]: https://github.com/onetrueawk/awk
>
> Fixes: 22f26f21774f ("kbuild: get rid of duplication in *.mod files")
> Signed-off-by: Kevin Locke <kevin@kevinlocke.name>
> ---

Portable and clean solution!

Applied to linux-kbuild/fixes. Thanks.





-- 
Best Regards
Masahiro Yamada

RE: [PATCH] kbuild: avoid regex RS for POSIX awk

[David Laight]

From: Kevin Locke
> Sent: 07 June 2022 03:43
> 
> In 22f26f21774f8 awk was added to deduplicate *.mod files.

Can't this be done with gmake's $(sort) function?

$(sort list)

    Sorts the words of list in lexical order, removing duplicate words.
    The output is a list of words separated by single spaces.

...
>  # To make this rule robust against "Argument list too long" error,
>  # ensure to add $(obj)/ prefix by a shell command.
> -cmd_mod = echo $(call real-search, $*.o, .o, -objs -y -m) | \
> -	$(AWK) -v RS='( |\n)' '!x[$$0]++ { print("$(obj)/"$$0) }' > $@
> +cmd_mod = printf '%s\n' $(call real-search, $*.o, .o, -objs -y -m) | \
> +	$(AWK) '!x[$$0]++ { print("$(obj)/"$$0) }' > $@

I think the above only works because 'printf' is (usually) a
shell builtin - so the kernel's argv[] limit doesn't apply.
So the comment isn't really right.

But I think:

cmd_mod = $(addprefix $(obj)/,$(sort $(call real-search, $*.o, .o, -objs -y -m))) >$@

will have the required effect.
Without forking and execing multiple processes.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Re: [PATCH] kbuild: avoid regex RS for POSIX awk

[Masahiro Yamada]

On Wed, Jun 8, 2022 at 4:43 PM David Laight <David.Laight@aculab.com> wrote:
>
> From: Kevin Locke
> > Sent: 07 June 2022 03:43
> >
> > In 22f26f21774f8 awk was added to deduplicate *.mod files.
>
> Can't this be done with gmake's $(sort) function?
>
> $(sort list)
>
>     Sorts the words of list in lexical order, removing duplicate words.
>     The output is a list of words separated by single spaces.


$(sort ...) does two things,
  - sort the list alphabetically
  - deduplicate the elements in the list

I want to do only deduplication
without changing the order.




>
> ...
> >  # To make this rule robust against "Argument list too long" error,
> >  # ensure to add $(obj)/ prefix by a shell command.
> > -cmd_mod = echo $(call real-search, $*.o, .o, -objs -y -m) | \
> > -     $(AWK) -v RS='( |\n)' '!x[$$0]++ { print("$(obj)/"$$0) }' > $@
> > +cmd_mod = printf '%s\n' $(call real-search, $*.o, .o, -objs -y -m) | \
> > +     $(AWK) '!x[$$0]++ { print("$(obj)/"$$0) }' > $@
>
> I think the above only works because 'printf' is (usually) a
> shell builtin - so the kernel's argv[] limit doesn't apply.
> So the comment isn't really right.

Is there any difference if 'printf' were not built-in?



Right, for bash and dash, yes, 'printf' is built-in,
and we do not need to be worried about
"Argument list too long", but
I am not sure if we are able to cover all the systems.



> But I think:
>
> cmd_mod = $(addprefix $(obj)/,$(sort $(call real-search, $*.o, .o, -objs -y -m))) >$@
>
> will have the required effect.


I think 'echo' is missing here.
As I noted above, I do not want to change the order.



> Without forking and execing multiple processes.
>
>         David
>
> -
> Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
> Registration No: 1397386 (Wales)
>


-- 
Best Regards
Masahiro Yamada

Re: [PATCH] kbuild: avoid regex RS for POSIX awk

[Sedat Dilek]

On Tue, Jun 7, 2022 at 7:31 AM Kevin Locke <kevin@kevinlocke.name> wrote:
>
> In 22f26f21774f8 awk was added to deduplicate *.mod files.  The awk
> invocation passes -v RS='( |\n)' to match a space or newline character
> as the record separator.  Unfortunately, POSIX states[1]
>
> > If RS contains more than one character, the results are unspecified.
>
> Some implementations (such as the One True Awk[2] used by the BSDs) do
> not treat RS as a regular expression.  When awk does not support regex
> RS, build failures such as the following are produced (first error using
> allmodconfig):
>
>       CC [M]  arch/x86/events/intel/uncore.o
>       CC [M]  arch/x86/events/intel/uncore_nhmex.o
>       CC [M]  arch/x86/events/intel/uncore_snb.o
>       CC [M]  arch/x86/events/intel/uncore_snbep.o
>       CC [M]  arch/x86/events/intel/uncore_discovery.o
>       LD [M]  arch/x86/events/intel/intel-uncore.o
>     ld: cannot find uncore_nhmex.o: No such file or directory
>     ld: cannot find uncore_snb.o: No such file or directory
>     ld: cannot find uncore_snbep.o: No such file or directory
>     ld: cannot find uncore_discovery.o: No such file or directory
>     make[3]: *** [scripts/Makefile.build:422: arch/x86/events/intel/intel-uncore.o] Error 1
>     make[2]: *** [scripts/Makefile.build:487: arch/x86/events/intel] Error 2
>     make[1]: *** [scripts/Makefile.build:487: arch/x86/events] Error 2
>     make: *** [Makefile:1839: arch/x86] Error 2
>
> To avoid this, use printf(1) to produce a newline between each object
> path, instead of the space produced by echo(1), so that the default RS
> can be used by awk.
>
> [1]: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/awk.html
> [2]: https://github.com/onetrueawk/awk
>
> Fixes: 22f26f21774f ("kbuild: get rid of duplication in *.mod files")
> Signed-off-by: Kevin Locke <kevin@kevinlocke.name>

Tested-by: Sedat Dilek <sedat.dilek@gmail.com> # LLVM-14 (x86-64)

-Sedat-

> ---
>  scripts/Makefile.build | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/Makefile.build b/scripts/Makefile.build
> index 1f01ac65c0cd..cac070aee791 100644
> --- a/scripts/Makefile.build
> +++ b/scripts/Makefile.build
> @@ -251,8 +251,8 @@ $(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
>
>  # To make this rule robust against "Argument list too long" error,
>  # ensure to add $(obj)/ prefix by a shell command.
> -cmd_mod = echo $(call real-search, $*.o, .o, -objs -y -m) | \
> -       $(AWK) -v RS='( |\n)' '!x[$$0]++ { print("$(obj)/"$$0) }' > $@
> +cmd_mod = printf '%s\n' $(call real-search, $*.o, .o, -objs -y -m) | \
> +       $(AWK) '!x[$$0]++ { print("$(obj)/"$$0) }' > $@
>
>  $(obj)/%.mod: FORCE
>         $(call if_changed,mod)
> --
> 2.35.1
>