[PATCH v2 1/2] kbuild: pass jobserver to cmd_ld_vmlinux.o

[PATCH v2 1/2] kbuild: pass jobserver to cmd_ld_vmlinux.o

[Jiri Slaby]

Until the link-vmlinux.sh split (cf. the commit below), the linker was
run with jobserver set in MAKEFLAGS. After the split, the command in
Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
is lost.

Restore it as linkers working in parallel (esp. the LTO ones) make a use
of it. In this case, it is gcc producing this warning otherwise:
lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’

Cc: Sedat Dilek <sedat.dilek@gmail.com>
Cc: Masahiro Yamada <masahiroy@kernel.org>
Cc: Michal Marek <michal.lkml@markovi.net>
Cc: Nick Desaulniers <ndesaulniers@google.com>
Cc: Signed-off-by: Martin Liska <mliska@suse.cz>
Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
Signed-off-by: Jiri Slaby <jslaby@suse.cz>
---

Notes:
    [v2] only commit message updated

 scripts/Makefile.vmlinux_o | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/Makefile.vmlinux_o b/scripts/Makefile.vmlinux_o
index 84019814f33f..886db0659d9c 100644
--- a/scripts/Makefile.vmlinux_o
+++ b/scripts/Makefile.vmlinux_o
@@ -65,7 +65,7 @@ define rule_ld_vmlinux.o
 endef
 
 vmlinux.o: $(initcalls-lds) $(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS) FORCE
-	$(call if_changed_rule,ld_vmlinux.o)
+	+$(call if_changed_rule,ld_vmlinux.o)
 
 targets += vmlinux.o
 
-- 
2.37.1

[PATCH v2 2/2] kbuild: lto: preserve MAKEFLAGS for module linking

[Jiri Slaby]

From: Martin Liska <mliska@suse.cz>

Prefix command in makefile run in order to preserve access to jobserver.
This is needed for gcc at least.

Fixes this warning:
lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’

Cc: Sedat Dilek <sedat.dilek@gmail.com>
Cc: Masahiro Yamada <masahiroy@kernel.org>
Cc: Michal Marek <michal.lkml@markovi.net>
Cc: Nick Desaulniers <ndesaulniers@google.com>
Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
Signed-off-by: Martin Liska <mliska@suse.cz>
Signed-off-by: Jiri Slaby <jslaby@suse.cz>
---

Notes:
    [v2] this is new in v2

 scripts/Makefile.build | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index bb63374fb055..493f3c4e8461 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -247,8 +247,8 @@ endef
 
 # Built-in and composite module parts
 $(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
-	$(call if_changed_rule,cc_o_c)
-	$(call cmd,force_checksrc)
+	+$(call if_changed_rule,cc_o_c)
+	+$(call cmd,force_checksrc)
 
 # To make this rule robust against "Argument list too long" error,
 # ensure to add $(obj)/ prefix by a shell command.
@@ -457,7 +457,7 @@ endef
 $(multi-obj-m): objtool-enabled := $(delay-objtool)
 $(multi-obj-m): part-of-module := y
 $(multi-obj-m): %.o: %.mod FORCE
-	$(call if_changed_rule,ld_multi_m)
+	+$(call if_changed_rule,ld_multi_m)
 $(call multi_depend, $(multi-obj-m), .o, -objs -y -m)
 
 targets := $(filter-out $(PHONY), $(targets))
-- 
2.37.1

Re: [PATCH v2 1/2] kbuild: pass jobserver to cmd_ld_vmlinux.o

[Masahiro Yamada]

On Tue, Jul 19, 2022 at 5:41 PM Jiri Slaby <jslaby@suse.cz> wrote:
>
> Until the link-vmlinux.sh split (cf. the commit below), the linker was
> run with jobserver set in MAKEFLAGS. After the split, the command in
> Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> is lost.
>
> Restore it as linkers working in parallel (esp. the LTO ones) make a use
> of it. In this case, it is gcc producing this warning otherwise:
> lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’
>
> Cc: Sedat Dilek <sedat.dilek@gmail.com>
> Cc: Masahiro Yamada <masahiroy@kernel.org>
> Cc: Michal Marek <michal.lkml@markovi.net>
> Cc: Nick Desaulniers <ndesaulniers@google.com>
> Cc: Signed-off-by: Martin Liska <mliska@suse.cz>
> Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
> Signed-off-by: Jiri Slaby <jslaby@suse.cz>
> ---
>
> Notes:
>     [v2] only commit message updated


Did you read my previous email?

https://lore.kernel.org/linux-kbuild/CAK7LNAS46hrVh3FNiHHL8_Hqp85VgWnsHyjQ4t-TFH_H344NOg@mail.gmail.com/






>
>  scripts/Makefile.vmlinux_o | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/scripts/Makefile.vmlinux_o b/scripts/Makefile.vmlinux_o
> index 84019814f33f..886db0659d9c 100644
> --- a/scripts/Makefile.vmlinux_o
> +++ b/scripts/Makefile.vmlinux_o
> @@ -65,7 +65,7 @@ define rule_ld_vmlinux.o
>  endef
>
>  vmlinux.o: $(initcalls-lds) $(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS) FORCE
> -       $(call if_changed_rule,ld_vmlinux.o)
> +       +$(call if_changed_rule,ld_vmlinux.o)
>
>  targets += vmlinux.o
>
> --
> 2.37.1
>


-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2 2/2] kbuild: lto: preserve MAKEFLAGS for module linking

[Masahiro Yamada]

On Tue, Jul 19, 2022 at 5:41 PM Jiri Slaby <jslaby@suse.cz> wrote:
>
> From: Martin Liska <mliska@suse.cz>
>
> Prefix command in makefile run in order to preserve access to jobserver.
> This is needed for gcc at least.
>
> Fixes this warning:
> lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’


What is lto-wrapper ?

>
> Cc: Sedat Dilek <sedat.dilek@gmail.com>
> Cc: Masahiro Yamada <masahiroy@kernel.org>
> Cc: Michal Marek <michal.lkml@markovi.net>
> Cc: Nick Desaulniers <ndesaulniers@google.com>
> Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
> Signed-off-by: Martin Liska <mliska@suse.cz>
> Signed-off-by: Jiri Slaby <jslaby@suse.cz>
> ---
>
> Notes:
>     [v2] this is new in v2
>
>  scripts/Makefile.build | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/scripts/Makefile.build b/scripts/Makefile.build
> index bb63374fb055..493f3c4e8461 100644
> --- a/scripts/Makefile.build
> +++ b/scripts/Makefile.build
> @@ -247,8 +247,8 @@ endef
>
>  # Built-in and composite module parts
>  $(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
> -       $(call if_changed_rule,cc_o_c)
> -       $(call cmd,force_checksrc)
> +       +$(call if_changed_rule,cc_o_c)
> +       +$(call cmd,force_checksrc)


cmd_force_checksrc invokes sparse.
How is it related to GCC?




>
>  # To make this rule robust against "Argument list too long" error,
>  # ensure to add $(obj)/ prefix by a shell command.
> @@ -457,7 +457,7 @@ endef
>  $(multi-obj-m): objtool-enabled := $(delay-objtool)
>  $(multi-obj-m): part-of-module := y
>  $(multi-obj-m): %.o: %.mod FORCE
> -       $(call if_changed_rule,ld_multi_m)
> +       +$(call if_changed_rule,ld_multi_m)
>  $(call multi_depend, $(multi-obj-m), .o, -objs -y -m)


cmd_ld_multi_m invokes $(LD) directly.
How is it related to gcc?






>
>  targets := $(filter-out $(PHONY), $(targets))
> --
> 2.37.1
>


-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2 1/2] kbuild: pass jobserver to cmd_ld_vmlinux.o

[Jiri Slaby]

On 19. 07. 22, 11:15, Masahiro Yamada wrote:
> On Tue, Jul 19, 2022 at 5:41 PM Jiri Slaby <jslaby@suse.cz> wrote:
>>
>> Until the link-vmlinux.sh split (cf. the commit below), the linker was
>> run with jobserver set in MAKEFLAGS. After the split, the command in
>> Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
>> is lost.
>>
>> Restore it as linkers working in parallel (esp. the LTO ones) make a use
>> of it. In this case, it is gcc producing this warning otherwise:
>> lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’
>>
>> Cc: Sedat Dilek <sedat.dilek@gmail.com>
>> Cc: Masahiro Yamada <masahiroy@kernel.org>
>> Cc: Michal Marek <michal.lkml@markovi.net>
>> Cc: Nick Desaulniers <ndesaulniers@google.com>
>> Cc: Signed-off-by: Martin Liska <mliska@suse.cz>
>> Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
>> Signed-off-by: Jiri Slaby <jslaby@suse.cz>
>> ---
>>
>> Notes:
>>      [v2] only commit message updated
> 
> 
> Did you read my previous email?
> 
> https://lore.kernel.org/linux-kbuild/CAK7LNAS46hrVh3FNiHHL8_Hqp85VgWnsHyjQ4t-TFH_H344NOg@mail.gmail.com/

Huh, I neither received this email nor the linked one to my inbox. 
@suse.cz somehow filters mails from you :/. Let me figure out (and use 
kernel.org account for now). (And get back to your comments too.)

>>   scripts/Makefile.vmlinux_o | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/scripts/Makefile.vmlinux_o b/scripts/Makefile.vmlinux_o
>> index 84019814f33f..886db0659d9c 100644
>> --- a/scripts/Makefile.vmlinux_o
>> +++ b/scripts/Makefile.vmlinux_o
>> @@ -65,7 +65,7 @@ define rule_ld_vmlinux.o
>>   endef
>>
>>   vmlinux.o: $(initcalls-lds) $(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS) FORCE
>> -       $(call if_changed_rule,ld_vmlinux.o)
>> +       +$(call if_changed_rule,ld_vmlinux.o)
>>
>>   targets += vmlinux.o
>>
>> --
>> 2.37.1
>>
> 
> 

-- 
js

Re: [PATCH v2 2/2] kbuild: lto: preserve MAKEFLAGS for module linking

[Jiri Slaby]

On 19. 07. 22, 11:15, Masahiro Yamada wrote:
> On Tue, Jul 19, 2022 at 5:41 PM Jiri Slaby <jslaby@suse.cz> wrote:
>>
>> From: Martin Liska <mliska@suse.cz>
>>
>> Prefix command in makefile run in order to preserve access to jobserver.
>> This is needed for gcc at least.
>>
>> Fixes this warning:
>> lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’
> 
> 
> What is lto-wrapper ?

lto-wrapper is a part of gcc:
https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/lto-wrapper.cc

And is invoked during the link of LTO objects (also to link in parallel 
if jobserver is available).

I am leaving the unanswered questions below for Martin to answer.

>> Cc: Sedat Dilek <sedat.dilek@gmail.com>
>> Cc: Masahiro Yamada <masahiroy@kernel.org>
>> Cc: Michal Marek <michal.lkml@markovi.net>
>> Cc: Nick Desaulniers <ndesaulniers@google.com>
>> Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
>> Signed-off-by: Martin Liska <mliska@suse.cz>
>> Signed-off-by: Jiri Slaby <jslaby@suse.cz>
>> ---
>>
>> Notes:
>>      [v2] this is new in v2
>>
>>   scripts/Makefile.build | 6 +++---
>>   1 file changed, 3 insertions(+), 3 deletions(-)
>>
>> diff --git a/scripts/Makefile.build b/scripts/Makefile.build
>> index bb63374fb055..493f3c4e8461 100644
>> --- a/scripts/Makefile.build
>> +++ b/scripts/Makefile.build
>> @@ -247,8 +247,8 @@ endef
>>
>>   # Built-in and composite module parts
>>   $(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
>> -       $(call if_changed_rule,cc_o_c)
>> -       $(call cmd,force_checksrc)
>> +       +$(call if_changed_rule,cc_o_c)
>> +       +$(call cmd,force_checksrc)
> 
> 
> cmd_force_checksrc invokes sparse.
> How is it related to GCC?
> 
> 
> 
> 
>>
>>   # To make this rule robust against "Argument list too long" error,
>>   # ensure to add $(obj)/ prefix by a shell command.
>> @@ -457,7 +457,7 @@ endef
>>   $(multi-obj-m): objtool-enabled := $(delay-objtool)
>>   $(multi-obj-m): part-of-module := y
>>   $(multi-obj-m): %.o: %.mod FORCE
>> -       $(call if_changed_rule,ld_multi_m)
>> +       +$(call if_changed_rule,ld_multi_m)
>>   $(call multi_depend, $(multi-obj-m), .o, -objs -y -m)
> 
> 
> cmd_ld_multi_m invokes $(LD) directly.
> How is it related to gcc?



-- 
js
suse labs

Re: [PATCH v2 2/2] kbuild: lto: preserve MAKEFLAGS for module linking

[Martin Liška]

On 9/20/22 10:06, Jiri Slaby wrote:
> On 19. 07. 22, 11:15, Masahiro Yamada wrote:
>> On Tue, Jul 19, 2022 at 5:41 PM Jiri Slaby <jslaby@suse.cz> wrote:
>>>
>>> From: Martin Liska <mliska@suse.cz>
>>>
>>> Prefix command in makefile run in order to preserve access to jobserver.
>>> This is needed for gcc at least.
>>>
>>> Fixes this warning:
>>> lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’
>>
>>
>> What is lto-wrapper ?
> 
> lto-wrapper is a part of gcc:
> https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/lto-wrapper.cc
> 
> And is invoked during the link of LTO objects (also to link in parallel if jobserver is available).

Yes, it's a driver that is invoked by Linker when it decides to use LTO mode in GCC compiler.
Later on the lto-wrapper connects to make's jobserver and runs in parallel. Note a recent [1]
make implemented named pipes when it comes to communication with jobserver. And thus the '+'
won't be needed in the future, but we need it for now.

Cheers,
Martin

[1] https://git.savannah.gnu.org/cgit/make.git/commit/?id=7ad2593b2d2bb5b9332f4444d8bf93ac6f958bc6

> 
> I am leaving the unanswered questions below for Martin to answer.
> 
>>> Cc: Sedat Dilek <sedat.dilek@gmail.com>
>>> Cc: Masahiro Yamada <masahiroy@kernel.org>
>>> Cc: Michal Marek <michal.lkml@markovi.net>
>>> Cc: Nick Desaulniers <ndesaulniers@google.com>
>>> Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
>>> Signed-off-by: Martin Liska <mliska@suse.cz>
>>> Signed-off-by: Jiri Slaby <jslaby@suse.cz>
>>> ---
>>>
>>> Notes:
>>>      [v2] this is new in v2
>>>
>>>   scripts/Makefile.build | 6 +++---
>>>   1 file changed, 3 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/scripts/Makefile.build b/scripts/Makefile.build
>>> index bb63374fb055..493f3c4e8461 100644
>>> --- a/scripts/Makefile.build
>>> +++ b/scripts/Makefile.build
>>> @@ -247,8 +247,8 @@ endef
>>>
>>>   # Built-in and composite module parts
>>>   $(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
>>> -       $(call if_changed_rule,cc_o_c)
>>> -       $(call cmd,force_checksrc)
>>> +       +$(call if_changed_rule,cc_o_c)
>>> +       +$(call cmd,force_checksrc)
>>
>>
>> cmd_force_checksrc invokes sparse.
>> How is it related to GCC?
>>
>>
>>
>>
>>>
>>>   # To make this rule robust against "Argument list too long" error,
>>>   # ensure to add $(obj)/ prefix by a shell command.
>>> @@ -457,7 +457,7 @@ endef
>>>   $(multi-obj-m): objtool-enabled := $(delay-objtool)
>>>   $(multi-obj-m): part-of-module := y
>>>   $(multi-obj-m): %.o: %.mod FORCE
>>> -       $(call if_changed_rule,ld_multi_m)
>>> +       +$(call if_changed_rule,ld_multi_m)
>>>   $(call multi_depend, $(multi-obj-m), .o, -objs -y -m)
>>
>>
>> cmd_ld_multi_m invokes $(LD) directly.
>> How is it related to gcc?
> 
> 
>