[PATCH 1/3] kbuild: refactor Makefile.asm-generic

[PATCH 1/3] kbuild: refactor Makefile.asm-generic

[Masahiro Yamada]

 - Use conventional $(MAKE) $(asm-generic)=<dir> style
   for directory descending

 - Remove unneeded FORCE since "all" is a phony target

 - Remove unneeded "_dummy :=" assignment

 - Skip $(shell mkdir ...) when headers exist in the directory

 - Misc cleanups

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 Makefile                     |  8 ++++----
 scripts/Makefile.asm-generic | 37 +++++++++++++++++++------------------
 2 files changed, 23 insertions(+), 22 deletions(-)

diff --git a/Makefile b/Makefile
index ab38eb2..3af0379 100644
--- a/Makefile
+++ b/Makefile
@@ -1122,13 +1122,13 @@ prepare0: archprepare
 prepare: prepare0 prepare-objtool
 
 # Support for using generic headers in asm-generic
+asm-generic := -f $(srctree)/scripts/Makefile.asm-generic obj
+
 PHONY += asm-generic uapi-asm-generic
 asm-generic: uapi-asm-generic
-	$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.asm-generic \
-	            src=asm obj=arch/$(SRCARCH)/include/generated/asm
+	$(Q)$(MAKE) $(asm-generic)=arch/$(SRCARCH)/include/generated/asm
 uapi-asm-generic:
-	$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.asm-generic \
-	            src=uapi/asm obj=arch/$(SRCARCH)/include/generated/uapi/asm
+	$(Q)$(MAKE) $(asm-generic)=arch/$(SRCARCH)/include/generated/uapi/asm
 
 PHONY += prepare-objtool
 prepare-objtool: $(objtool_target)
diff --git a/scripts/Makefile.asm-generic b/scripts/Makefile.asm-generic
index 32ad8e9..760323e 100644
--- a/scripts/Makefile.asm-generic
+++ b/scripts/Makefile.asm-generic
@@ -2,41 +2,42 @@
 # include/asm-generic contains a lot of files that are used
 # verbatim by several architectures.
 #
-# This Makefile reads the file arch/$(SRCARCH)/include/$(src)/Kbuild
+# This Makefile reads the file arch/$(SRCARCH)/include/(uapi/)/asm/Kbuild
 # and for each file listed in this file with generic-y creates
-# a small wrapper file in $(obj) (arch/$(SRCARCH)/include/generated/$(src))
+# a small wrapper file in arch/$(SRCARCH)/include/generated/(uapi/)/asm.
 
 PHONY := all
 all:
 
-kbuild-file := $(srctree)/arch/$(SRCARCH)/include/$(src)/Kbuild
--include $(kbuild-file)
+src := $(subst /generated,,$(obj))
+-include $(src)/Kbuild
 
 include scripts/Kbuild.include
 
-# Create output directory if not already present
-_dummy := $(shell [ -d $(obj) ] || mkdir -p $(obj))
+generic-y   := $(addprefix $(obj)/, $(generic-y))
+generated-y := $(addprefix $(obj)/, $(generated-y))
 
-# Stale wrappers when the corresponding files are removed from generic-y
-# need removing.
-generated-y   := $(generic-y) $(generated-y)
-all-files     := $(patsubst %, $(obj)/%, $(generated-y))
-old-headers   := $(wildcard $(obj)/*.h)
-unwanted      := $(filter-out $(all-files),$(old-headers))
+# Remove stale wrappers when the corresponding files are removed from generic-y
+old-headers := $(wildcard $(obj)/*.h)
+unwanted    := $(filter-out $(generic-y) $(generated-y),$(old-headers))
 
 quiet_cmd_wrap = WRAP    $@
-cmd_wrap = echo "\#include <asm-generic/$*.h>" >$@
+      cmd_wrap = echo "\#include <asm-generic/$*.h>" > $@
 
 quiet_cmd_remove = REMOVE  $(unwanted)
-cmd_remove = rm -f $(unwanted)
+      cmd_remove = rm -f $(unwanted)
 
-all: $(patsubst %, $(obj)/%, $(generic-y)) FORCE
-	$(if $(unwanted),$(call cmd,remove),)
+all: $(generic-y)
+	$(if $(unwanted),$(call cmd,remove))
 	@:
 
 $(obj)/%.h:
 	$(call cmd,wrap)
 
-PHONY += FORCE
+# Create output directory. Skip it if at least one old header exists
+# since we know the output directory already exists.
+ifeq ($(old-headers),)
+$(shell mkdir -p $(obj))
+endif
+
 .PHONY: $(PHONY)
-FORCE: ;
-- 
2.7.4

[PATCH 2/3] kbuild: generate asm-generic wrappers if mandatory headers are missing

[Masahiro Yamada]

Some time ago, Sam pointed out a certain degree of overwrap between
generic-y and mandatory-y. (https://lkml.org/lkml/2017/7/10/121)

I a bit tweaked the meaning of mandatory-y; now it defines the minimum
set of ASM headers that all architectures must have.

If arch does not have specific implementation of a mandatory header,
Kbuild will let it fallback to the asm-generic one by automatically
generating a wrapper. This will allow to drop lots of redundant
generic-y defines.

Previously, "mandatory" was used in the context of UAPI, but I guess
this can be extended to kernel space ASM headers.

Suggested-by: Sam Ravnborg <sam@ravnborg.org>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 Documentation/kbuild/makefiles.txt | 9 ++++++---
 scripts/Makefile.asm-generic       | 3 +++
 scripts/Makefile.headersinst       | 7 -------
 3 files changed, 9 insertions(+), 10 deletions(-)

diff --git a/Documentation/kbuild/makefiles.txt b/Documentation/kbuild/makefiles.txt
index 8da26c6..bf28c47 100644
--- a/Documentation/kbuild/makefiles.txt
+++ b/Documentation/kbuild/makefiles.txt
@@ -1296,9 +1296,12 @@ See subsequent chapter for the syntax of the Kbuild file.
 
 --- 7.4 mandatory-y
 
-	mandatory-y is essentially used by include/uapi/asm-generic/Kbuild.asm
-	to define the minimum set of headers that must be exported in
-	include/asm.
+	mandatory-y is essentially used by include/(uapi/)asm-generic/Kbuild.asm
+	to define the minimum set of ASM headers that all architectures must have.
+
+	This works like optional generic-y. If a mandatory header is missing
+	in arch/$(ARCH)/include/(uapi/)/asm, Kbuild will automatically generate
+	a wrapper of the asm-generic one.
 
 	The convention is to list one subdir per line and
 	preferably in alphabetic order.
diff --git a/scripts/Makefile.asm-generic b/scripts/Makefile.asm-generic
index 760323e..c7cd23d 100644
--- a/scripts/Makefile.asm-generic
+++ b/scripts/Makefile.asm-generic
@@ -14,6 +14,9 @@ src := $(subst /generated,,$(obj))
 
 include scripts/Kbuild.include
 
+# If arch does not implement mandatory headers, fallback to asm-generic ones.
+generic-y   += $(foreach f, $(mandatory-y), $(if $(wildcard $(srctree)/$(src)/$(f)),,$(f)))
+
 generic-y   := $(addprefix $(obj)/, $(generic-y))
 generated-y := $(addprefix $(obj)/, $(generated-y))
 
diff --git a/scripts/Makefile.headersinst b/scripts/Makefile.headersinst
index d5e1314..0d4a96d 100644
--- a/scripts/Makefile.headersinst
+++ b/scripts/Makefile.headersinst
@@ -57,13 +57,6 @@ check-file    := $(installdir)/.check
 all-files     := $(header-files) $(genhdr-files)
 output-files  := $(addprefix $(installdir)/, $(all-files))
 
-ifneq ($(mandatory-y),)
-missing       := $(filter-out $(all-files),$(mandatory-y))
-ifneq ($(missing),)
-$(error Some mandatory headers ($(missing)) are missing in $(obj))
-endif
-endif
-
 # Work out what needs to be removed
 oldheaders    := $(patsubst $(installdir)/%,%,$(wildcard $(installdir)/*.h))
 unwanted      := $(filter-out $(all-files),$(oldheaders))
-- 
2.7.4

Re: [PATCH 2/3] kbuild: generate asm-generic wrappers if mandatory headers are missing

[Christoph Hellwig]

On Wed, Dec 05, 2018 at 08:28:05PM +0900, Masahiro Yamada wrote:
> Some time ago, Sam pointed out a certain degree of overwrap between
> generic-y and mandatory-y. (https://lkml.org/lkml/2017/7/10/121)
> 
> I a bit tweaked the meaning of mandatory-y; now it defines the minimum
> set of ASM headers that all architectures must have.
> 
> If arch does not have specific implementation of a mandatory header,
> Kbuild will let it fallback to the asm-generic one by automatically
> generating a wrapper. This will allow to drop lots of redundant
> generic-y defines.
> 
> Previously, "mandatory" was used in the context of UAPI, but I guess
> this can be extended to kernel space ASM headers.

How useful is it to keep the generic-y behavior around at all vs making
everything useful mandatory?

Re: [PATCH 2/3] kbuild: generate asm-generic wrappers if mandatory headers are missing

[Sam Ravnborg]

On Wed, Dec 05, 2018 at 08:28:05PM +0900, Masahiro Yamada wrote:
> Some time ago, Sam pointed out a certain degree of overwrap between
> generic-y and mandatory-y. (https://lkml.org/lkml/2017/7/10/121)
> 
> I a bit tweaked the meaning of mandatory-y; now it defines the minimum
> set of ASM headers that all architectures must have.
> 
> If arch does not have specific implementation of a mandatory header,
> Kbuild will let it fallback to the asm-generic one by automatically
> generating a wrapper. This will allow to drop lots of redundant
> generic-y defines.
> 
> Previously, "mandatory" was used in the context of UAPI, but I guess
> this can be extended to kernel space ASM headers.
> 
> Suggested-by: Sam Ravnborg <sam@ravnborg.org>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Nice work!

For the full series:
Acked-by: Sam Ravnborg <sam@ravnborg.org>

Have you considered to warn if generic-y contains a header listed
in mandatory-y - to prevent that they sneak back in.
And to catch when we lift a header from available to mandatory.

	Sam

Re: [PATCH 2/3] kbuild: generate asm-generic wrappers if mandatory headers are missing

[Masahiro Yamada]

Hi Christoph,


On Fri, Dec 7, 2018 at 12:04 AM Christoph Hellwig <hch@infradead.org> wrote:
>
> On Wed, Dec 05, 2018 at 08:28:05PM +0900, Masahiro Yamada wrote:
> > Some time ago, Sam pointed out a certain degree of overwrap between
> > generic-y and mandatory-y. (https://lkml.org/lkml/2017/7/10/121)
> >
> > I a bit tweaked the meaning of mandatory-y; now it defines the minimum
> > set of ASM headers that all architectures must have.
> >
> > If arch does not have specific implementation of a mandatory header,
> > Kbuild will let it fallback to the asm-generic one by automatically
> > generating a wrapper. This will allow to drop lots of redundant
> > generic-y defines.
> >
> > Previously, "mandatory" was used in the context of UAPI, but I guess
> > this can be extended to kernel space ASM headers.
>
> How useful is it to keep the generic-y behavior around at all vs making
> everything useful mandatory?


What I can tell is not all architectures
support kvm_para.h, ucontext.h

I guess they will stay as arch-specific generic-y,
but I am not an expert in this area.


kvm_para.h is missing csky, nds32, riscv.

ucontext.h is missing in alpha, arm, m68k, parisc, sparc, xtensa



bpf_perf_event.h could be promoted to mandatory-y ?
All architectures have it.


-- 
Best Regards
Masahiro Yamada

Re: [PATCH 2/3] kbuild: generate asm-generic wrappers if mandatory headers are missing

[Masahiro Yamada]

Hi Sam,


On Fri, Dec 7, 2018 at 3:06 AM Sam Ravnborg <sam@ravnborg.org> wrote:
>
> On Wed, Dec 05, 2018 at 08:28:05PM +0900, Masahiro Yamada wrote:
> > Some time ago, Sam pointed out a certain degree of overwrap between
> > generic-y and mandatory-y. (https://lkml.org/lkml/2017/7/10/121)
> >
> > I a bit tweaked the meaning of mandatory-y; now it defines the minimum
> > set of ASM headers that all architectures must have.
> >
> > If arch does not have specific implementation of a mandatory header,
> > Kbuild will let it fallback to the asm-generic one by automatically
> > generating a wrapper. This will allow to drop lots of redundant
> > generic-y defines.
> >
> > Previously, "mandatory" was used in the context of UAPI, but I guess
> > this can be extended to kernel space ASM headers.
> >
> > Suggested-by: Sam Ravnborg <sam@ravnborg.org>
> > Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>
> Nice work!
>
> For the full series:
> Acked-by: Sam Ravnborg <sam@ravnborg.org>
>
> Have you considered to warn if generic-y contains a header listed
> in mandatory-y - to prevent that they sneak back in.
> And to catch when we lift a header from available to mandatory.


Yes, I also thought of this,
and probably we should do it.





--
Best Regards
Masahiro Yamada

Re: [PATCH 2/3] kbuild: generate asm-generic wrappers if mandatory headers are missing

[Palmer Dabbelt]

On Tue, 11 Dec 2018 04:59:26 PST (-0800), yamada.masahiro@socionext.com wrote:
> Hi Christoph,
>
>
> On Fri, Dec 7, 2018 at 12:04 AM Christoph Hellwig <hch@infradead.org> wrote:
>>
>> On Wed, Dec 05, 2018 at 08:28:05PM +0900, Masahiro Yamada wrote:
>> > Some time ago, Sam pointed out a certain degree of overwrap between
>> > generic-y and mandatory-y. (https://lkml.org/lkml/2017/7/10/121)
>> >
>> > I a bit tweaked the meaning of mandatory-y; now it defines the minimum
>> > set of ASM headers that all architectures must have.
>> >
>> > If arch does not have specific implementation of a mandatory header,
>> > Kbuild will let it fallback to the asm-generic one by automatically
>> > generating a wrapper. This will allow to drop lots of redundant
>> > generic-y defines.
>> >
>> > Previously, "mandatory" was used in the context of UAPI, but I guess
>> > this can be extended to kernel space ASM headers.
>>
>> How useful is it to keep the generic-y behavior around at all vs making
>> everything useful mandatory?
>
>
> What I can tell is not all architectures
> support kvm_para.h, ucontext.h
>
> I guess they will stay as arch-specific generic-y,
> but I am not an expert in this area.
>
>
> kvm_para.h is missing csky, nds32, riscv.

It looks like RISC-V missed it and everyone else copied us.  I don't see any 
reason why the generic version wouldn't work on RISC-V, as it just has failures 
for all the calls.

> ucontext.h is missing in alpha, arm, m68k, parisc, sparc, xtensa
>
>
>
> bpf_perf_event.h could be promoted to mandatory-y ?
> All architectures have it.

Re: [PATCH 1/3] kbuild: refactor Makefile.asm-generic

[Masahiro Yamada]

On Wed, Dec 5, 2018 at 8:29 PM Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
>
>  - Use conventional $(MAKE) $(asm-generic)=<dir> style
>    for directory descending
>
>  - Remove unneeded FORCE since "all" is a phony target
>
>  - Remove unneeded "_dummy :=" assignment
>
>  - Skip $(shell mkdir ...) when headers exist in the directory
>
>  - Misc cleanups
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

1/3 applied to linux-kbuild.

I will send v2 for the rest in the series later.



-- 
Best Regards
Masahiro Yamada