[PATCH] kbuild: doc: describe proper script invocation

[PATCH] kbuild: doc: describe proper script invocation

[Lukas Bulwahn]

During an investigation to fix up the execute bits of scripts in the kernel
repository, Andrew Morton and Kees Cook pointed out that the execute bit
should not matter, and that build scripts cannot rely on that. Kees could
not point to any documentation, though.

Masahiro Yamada explained the convention of setting execute bits to make it
easier for manual script invocation.

Provide some basic documentation how the build shall invoke scripts, such
that the execute bits do not matter, and acknowledge that execute bits
are useful nonetheless.

This serves as reference for further clean-up patches in the future.

Link: https://lore.kernel.org/lkml/20200830174409.c24c3f67addcce0cea9a9d4c@linux-foundation.org/
Link: https://lore.kernel.org/lkml/202008271102.FEB906C88@keescook/
Link: https://lore.kernel.org/linux-kbuild/CAK7LNAQdrvMkDA6ApDJCGr+5db8SiPo=G+p8EiOvnnGvEN80gA@mail.gmail.com/

Suggested-by: Andrew Morton <akpm@linux-foundation.org>
Suggested-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Lukas Bulwahn <lukas.bulwahn@gmail.com>
---
RFC v1 -> RFC v2:
explain why execute bits are still convenient.

Kees, Andrew, please ack.

Masahiro-san, I have taken your feedback into account. Please pick this small
documentation update into your kbuild tree.

Ujjwal Kumar, a potential future mentee, will follow up with further fixes to
the build scripts.

 Documentation/kbuild/makefiles.rst | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/Documentation/kbuild/makefiles.rst b/Documentation/kbuild/makefiles.rst
index 58d513a0fa95..bd3e1baf58be 100644
--- a/Documentation/kbuild/makefiles.rst
+++ b/Documentation/kbuild/makefiles.rst
@@ -21,6 +21,7 @@ This document describes the Linux kernel Makefiles.
 	   --- 3.10 Special Rules
 	   --- 3.11 $(CC) support functions
 	   --- 3.12 $(LD) support functions
+	   --- 3.13 Script Invocation
 
 	=== 4 Host Program support
 	   --- 4.1 Simple Host Program
@@ -605,6 +606,25 @@ more details, with real examples.
 		#Makefile
 		LDFLAGS_vmlinux += $(call ld-option, -X)
 
+3.13 Script invocation
+----------------------
+
+	Make rules may invoke scripts to build the kernel. The rules shall
+	always provide the appropriate interpreter to execute the script. They
+	shall not rely on the execute bits being set, and shall not invoke the
+	script directly. For the convenience of manual script invocation, such
+	as invoking ./scripts/checkpatch.pl, it is recommended to set execute
+	bits on the scripts nonetheless.
+
+	Kbuild provides variables $(CONFIG_SHELL), $(AWK), $(PERL),
+	$(PYTHON) and $(PYTHON3) to refer to interpreters for the respective
+	scripts.
+
+	Example::
+
+		#Makefile
+		cmd_depmod = $(CONFIG_SHELL) $(srctree)/scripts/depmod.sh $(DEPMOD) \
+			     $(KERNELRELEASE)
 
 4 Host Program support
 ======================
-- 
2.17.1

Re: [PATCH] kbuild: doc: describe proper script invocation

[Masahiro Yamada]

On Thu, Oct 1, 2020 at 4:57 PM Lukas Bulwahn <lukas.bulwahn@gmail.com> wrote:
>
> During an investigation to fix up the execute bits of scripts in the kernel
> repository, Andrew Morton and Kees Cook pointed out that the execute bit
> should not matter, and that build scripts cannot rely on that. Kees could
> not point to any documentation, though.
>
> Masahiro Yamada explained the convention of setting execute bits to make it
> easier for manual script invocation.
>
> Provide some basic documentation how the build shall invoke scripts, such
> that the execute bits do not matter, and acknowledge that execute bits
> are useful nonetheless.
>
> This serves as reference for further clean-up patches in the future.
>
> Link: https://lore.kernel.org/lkml/20200830174409.c24c3f67addcce0cea9a9d4c@linux-foundation.org/
> Link: https://lore.kernel.org/lkml/202008271102.FEB906C88@keescook/
> Link: https://lore.kernel.org/linux-kbuild/CAK7LNAQdrvMkDA6ApDJCGr+5db8SiPo=G+p8EiOvnnGvEN80gA@mail.gmail.com/
>
> Suggested-by: Andrew Morton <akpm@linux-foundation.org>
> Suggested-by: Kees Cook <keescook@chromium.org>
> Signed-off-by: Lukas Bulwahn <lukas.bulwahn@gmail.com>


Applied to linux-kbuild.
Thanks.


> ---
> RFC v1 -> RFC v2:
> explain why execute bits are still convenient.
>
> Kees, Andrew, please ack.
>
> Masahiro-san, I have taken your feedback into account. Please pick this small
> documentation update into your kbuild tree.
>
> Ujjwal Kumar, a potential future mentee, will follow up with further fixes to
> the build scripts.
>
>  Documentation/kbuild/makefiles.rst | 20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
>
> diff --git a/Documentation/kbuild/makefiles.rst b/Documentation/kbuild/makefiles.rst
> index 58d513a0fa95..bd3e1baf58be 100644
> --- a/Documentation/kbuild/makefiles.rst
> +++ b/Documentation/kbuild/makefiles.rst
> @@ -21,6 +21,7 @@ This document describes the Linux kernel Makefiles.
>            --- 3.10 Special Rules
>            --- 3.11 $(CC) support functions
>            --- 3.12 $(LD) support functions
> +          --- 3.13 Script Invocation
>
>         === 4 Host Program support
>            --- 4.1 Simple Host Program
> @@ -605,6 +606,25 @@ more details, with real examples.
>                 #Makefile
>                 LDFLAGS_vmlinux += $(call ld-option, -X)
>
> +3.13 Script invocation
> +----------------------
> +
> +       Make rules may invoke scripts to build the kernel. The rules shall
> +       always provide the appropriate interpreter to execute the script. They
> +       shall not rely on the execute bits being set, and shall not invoke the
> +       script directly. For the convenience of manual script invocation, such
> +       as invoking ./scripts/checkpatch.pl, it is recommended to set execute
> +       bits on the scripts nonetheless.
> +
> +       Kbuild provides variables $(CONFIG_SHELL), $(AWK), $(PERL),
> +       $(PYTHON) and $(PYTHON3) to refer to interpreters for the respective
> +       scripts.
> +
> +       Example::
> +
> +               #Makefile
> +               cmd_depmod = $(CONFIG_SHELL) $(srctree)/scripts/depmod.sh $(DEPMOD) \
> +                            $(KERNELRELEASE)
>
>  4 Host Program support
>  ======================
> --
> 2.17.1
>


-- 
Best Regards
Masahiro Yamada

Re: [PATCH] kbuild: doc: describe proper script invocation

[Masahiro Yamada]

On Tue, Oct 13, 2020 at 11:31 PM Masahiro Yamada <masahiroy@kernel.org> wrote:
>
> On Thu, Oct 1, 2020 at 4:57 PM Lukas Bulwahn <lukas.bulwahn@gmail.com> wrote:
> >
> > During an investigation to fix up the execute bits of scripts in the kernel
> > repository, Andrew Morton and Kees Cook pointed out that the execute bit
> > should not matter, and that build scripts cannot rely on that. Kees could
> > not point to any documentation, though.
> >
> > Masahiro Yamada explained the convention of setting execute bits to make it
> > easier for manual script invocation.
> >
> > Provide some basic documentation how the build shall invoke scripts, such
> > that the execute bits do not matter, and acknowledge that execute bits
> > are useful nonetheless.
> >
> > This serves as reference for further clean-up patches in the future.
> >
> > Link: https://lore.kernel.org/lkml/20200830174409.c24c3f67addcce0cea9a9d4c@linux-foundation.org/
> > Link: https://lore.kernel.org/lkml/202008271102.FEB906C88@keescook/
> > Link: https://lore.kernel.org/linux-kbuild/CAK7LNAQdrvMkDA6ApDJCGr+5db8SiPo=G+p8EiOvnnGvEN80gA@mail.gmail.com/
> >
> > Suggested-by: Andrew Morton <akpm@linux-foundation.org>
> > Suggested-by: Kees Cook <keescook@chromium.org>
> > Signed-off-by: Lukas Bulwahn <lukas.bulwahn@gmail.com>
>
>
> Applied to linux-kbuild.
> Thanks.
>

No, not really.

I will re-think if this is the right thing to do.




--
Best Regards
Masahiro Yamada

Re: [PATCH] kbuild: doc: describe proper script invocation

[Lukas Bulwahn]

On Wed, 14 Oct 2020, Masahiro Yamada wrote:

> On Tue, Oct 13, 2020 at 11:31 PM Masahiro Yamada <masahiroy@kernel.org> wrote:
> >
> > On Thu, Oct 1, 2020 at 4:57 PM Lukas Bulwahn <lukas.bulwahn@gmail.com> wrote:
> > >
> > > During an investigation to fix up the execute bits of scripts in the kernel
> > > repository, Andrew Morton and Kees Cook pointed out that the execute bit
> > > should not matter, and that build scripts cannot rely on that. Kees could
> > > not point to any documentation, though.
> > >
> > > Masahiro Yamada explained the convention of setting execute bits to make it
> > > easier for manual script invocation.
> > >
> > > Provide some basic documentation how the build shall invoke scripts, such
> > > that the execute bits do not matter, and acknowledge that execute bits
> > > are useful nonetheless.
> > >
> > > This serves as reference for further clean-up patches in the future.
> > >
> > > Link: https://lore.kernel.org/lkml/20200830174409.c24c3f67addcce0cea9a9d4c@linux-foundation.org/
> > > Link: https://lore.kernel.org/lkml/202008271102.FEB906C88@keescook/
> > > Link: https://lore.kernel.org/linux-kbuild/CAK7LNAQdrvMkDA6ApDJCGr+5db8SiPo=G+p8EiOvnnGvEN80gA@mail.gmail.com/
> > >
> > > Suggested-by: Andrew Morton <akpm@linux-foundation.org>
> > > Suggested-by: Kees Cook <keescook@chromium.org>
> > > Signed-off-by: Lukas Bulwahn <lukas.bulwahn@gmail.com>
> >
> >
> > Applied to linux-kbuild.
> > Thanks.
> >
> 
> No, not really.
> 
> I will re-think if this is the right thing to do.
> 
>

Masahiro-san, Andrew Morton has already picked this patch in his branch; 
so, if we do not shout loudly, it will be merged. And we can follow up 
with a solid improvement for rc2.

I am happy to rework the documentation and help guide and review the 
follow-up patches that have been sent by Ujjwal.

Please provide some specific feedback what is wrong and why; I think the 
basic request not to rely on the execute bit from Andrew and Kees is still 
valid. Maybe we just need to be more specific on the rules to follow in 
Makefiles and have multiple examples for the different cases to consider.

Thanks,

Lukas