[RFC 0/2] Kbuild: Support nested composite objects

[RFC 0/2] Kbuild: Support nested composite objects

[Elliot Berman]

This series was developed after discussion in https://lkml.org/lkml/2021/1/19/850

The motivation for this series is an out-of-tree module which contains a large
number of source files. This causes Kbuild to exceed the maximum command line
argument length when linking the files. Proposal here permits composite objects
to contain other composite objects. This allows the driver to split linking into
several steps and avoid the maximum command line length error.

Kbuild composite objects only supports one level of composite objects.
That is, a composite object may only be composed of real compilable
source files.

As a simple example, the following Kbuild description is now supported:

bar-a-y := a/bar0.o a/bar1.o
bar-b-y := b/bar2.o b/bar3.o

foo-objs := bar-a.o bar-b.o

obj-m += foo.o

Add such support by recursively searching for composite objects and
listing them in $(multi-used-*) and $(real-obj-*).

Elliot Berman (2):
  Kbuild: Make composite object searching more generic
  Kbuild: Support nested composite objects

 scripts/Makefile.lib | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

[RFC 1/2] Kbuild: Make composite object searching more generic

[Elliot Berman]

Reduce repeated logic around expanding composite objects in preparation
for later commit to support nested composite objects.

Signed-off-by: Elliot Berman <eberman@codeaurora.org>
---
 scripts/Makefile.lib | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 213677a5..93e4f10 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -56,15 +56,19 @@ else
 obj-y		:= $(filter-out %/, $(obj-y))
 endif
 
+# Expand $(foo-objs) $(foo-y) by calling $(call suffix-search,foo.o,-objs -y)
+suffix-search = $(foreach s,$(2),$($(1:.o=$s)))
 # If $(foo-objs), $(foo-y), $(foo-m), or $(foo-) exists, foo.o is a composite object
-multi-used-y := $(sort $(foreach m,$(obj-y), $(if $(strip $($(m:.o=-objs)) $($(m:.o=-y)) $($(m:.o=-))), $(m))))
-multi-used-m := $(sort $(foreach m,$(obj-m), $(if $(strip $($(m:.o=-objs)) $($(m:.o=-y)) $($(m:.o=-m)) $($(m:.o=-))), $(m))))
+multi-search = $(sort $(foreach m,$(1), $(if $(strip $(call suffix-search,$(m),$(2) -)), $(m))))
+multi-used-y := $(call multi-search,$(obj-y),-objs -y)
+multi-used-m := $(call multi-search,$(obj-m),-objs -y -m)
 multi-used   := $(multi-used-y) $(multi-used-m)
 
 # Replace multi-part objects by their individual parts,
 # including built-in.a from subdirectories
-real-obj-y := $(foreach m, $(obj-y), $(if $(strip $($(m:.o=-objs)) $($(m:.o=-y)) $($(m:.o=-))),$($(m:.o=-objs)) $($(m:.o=-y)),$(m)))
-real-obj-m := $(foreach m, $(obj-m), $(if $(strip $($(m:.o=-objs)) $($(m:.o=-y)) $($(m:.o=-m)) $($(m:.o=-))),$($(m:.o=-objs)) $($(m:.o=-y)) $($(m:.o=-m)),$(m)))
+real-search = $(foreach m,$(1), $(if $(strip $(call suffix-search,$(m),$(2) -)),$(call suffix-search,$(m),$(2)),$(m)))
+real-obj-y := $(call real-search, $(obj-y),-objs -y)
+real-obj-m := $(call real-search, $(obj-m),-objs -y -m)
 
 always-y += $(always-m)
 
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: [RFC 1/2] Kbuild: Make composite object searching more generic

[Masahiro Yamada]

On Sat, Jan 23, 2021 at 4:27 AM Elliot Berman <eberman@codeaurora.org> wrote:
>
> Reduce repeated logic around expanding composite objects in preparation
> for later commit to support nested composite objects.
>
> Signed-off-by: Elliot Berman <eberman@codeaurora.org>

This patch is a nice clean-up
regardless of your main motivation in 2/2.


I removed the "in preparation ..." part.

Applied to linux-kbuild. Thanks.




> ---
>  scripts/Makefile.lib | 12 ++++++++----
>  1 file changed, 8 insertions(+), 4 deletions(-)
>
> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> index 213677a5..93e4f10 100644
> --- a/scripts/Makefile.lib
> +++ b/scripts/Makefile.lib
> @@ -56,15 +56,19 @@ else
>  obj-y          := $(filter-out %/, $(obj-y))
>  endif
>
> +# Expand $(foo-objs) $(foo-y) by calling $(call suffix-search,foo.o,-objs -y)
> +suffix-search = $(foreach s,$(2),$($(1:.o=$s)))
>  # If $(foo-objs), $(foo-y), $(foo-m), or $(foo-) exists, foo.o is a composite object
> -multi-used-y := $(sort $(foreach m,$(obj-y), $(if $(strip $($(m:.o=-objs)) $($(m:.o=-y)) $($(m:.o=-))), $(m))))
> -multi-used-m := $(sort $(foreach m,$(obj-m), $(if $(strip $($(m:.o=-objs)) $($(m:.o=-y)) $($(m:.o=-m)) $($(m:.o=-))), $(m))))
> +multi-search = $(sort $(foreach m,$(1), $(if $(strip $(call suffix-search,$(m),$(2) -)), $(m))))
> +multi-used-y := $(call multi-search,$(obj-y),-objs -y)
> +multi-used-m := $(call multi-search,$(obj-m),-objs -y -m)
>  multi-used   := $(multi-used-y) $(multi-used-m)
>
>  # Replace multi-part objects by their individual parts,
>  # including built-in.a from subdirectories
> -real-obj-y := $(foreach m, $(obj-y), $(if $(strip $($(m:.o=-objs)) $($(m:.o=-y)) $($(m:.o=-))),$($(m:.o=-objs)) $($(m:.o=-y)),$(m)))
> -real-obj-m := $(foreach m, $(obj-m), $(if $(strip $($(m:.o=-objs)) $($(m:.o=-y)) $($(m:.o=-m)) $($(m:.o=-))),$($(m:.o=-objs)) $($(m:.o=-y)) $($(m:.o=-m)),$(m)))
> +real-search = $(foreach m,$(1), $(if $(strip $(call suffix-search,$(m),$(2) -)),$(call suffix-search,$(m),$(2)),$(m)))
> +real-obj-y := $(call real-search, $(obj-y),-objs -y)
> +real-obj-m := $(call real-search, $(obj-m),-objs -y -m)
>
>  always-y += $(always-m)
>
> --
> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
> a Linux Foundation Collaborative Project
>


--
Best Regards
Masahiro Yamada

[RFC 2/2] Kbuild: Support nested composite objects

[Elliot Berman]

Kbuild composite objects only supports one level of composite objects.
That is, a composite object may only be composed of real compilable
source files.

As a simple example, the following Kbuild description is now supported:

bar-a-y := a/bar0.o a/bar1.o
bar-b-y := b/bar2.o b/bar3.o

foo-objs := bar-a.o bar-b.o

obj-m += foo.o

Add such support by recursively searching for composite objects and
listing them in $(multi-used-*) and $(real-obj-*).

Signed-off-by: Elliot Berman <eberman@codeaurora.org>
---
 scripts/Makefile.lib | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 93e4f10..5118204 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -59,14 +59,22 @@ endif
 # Expand $(foo-objs) $(foo-y) by calling $(call suffix-search,foo.o,-objs -y)
 suffix-search = $(foreach s,$(2),$($(1:.o=$s)))
 # If $(foo-objs), $(foo-y), $(foo-m), or $(foo-) exists, foo.o is a composite object
-multi-search = $(sort $(foreach m,$(1), $(if $(strip $(call suffix-search,$(m),$(2) -)), $(m))))
+# Do this recursively to find nested composite objects. For backwards compatibility,
+# foo-y may contain foo.o bar.o . We shouldn't recurse further in this case.
+multi-search = $(sort $(foreach m,$(1),$(if $(strip $(call suffix-search,$(m),$(2) -)),\
+	$(m) $(call multi-search,$(filter-out $(m),$(call suffix-search,$(m),$(2))),$(2)))))
 multi-used-y := $(call multi-search,$(obj-y),-objs -y)
 multi-used-m := $(call multi-search,$(obj-m),-objs -y -m)
 multi-used   := $(multi-used-y) $(multi-used-m)
 
 # Replace multi-part objects by their individual parts,
 # including built-in.a from subdirectories
-real-search = $(foreach m,$(1), $(if $(strip $(call suffix-search,$(m),$(2) -)),$(call suffix-search,$(m),$(2)),$(m)))
+# Recursively search for real files. For backwards compatibility,
+# foo-y may contain foo.o bar.o . foo.o in this context is a real object, and
+# shouldn't be recursed into.
+real-search = $(foreach m,$(1), $(if $(strip $(call suffix-search,$(m),$(2) -)), \
+	$(filter $(m),$(call suffix-search,$(m),$(2))) $(call real-search,$(filter-out $(m),$(call suffix-search,$(m),$(2))),$(2)),\
+	$(m)))
 real-obj-y := $(call real-search, $(obj-y),-objs -y)
 real-obj-m := $(call real-search, $(obj-m),-objs -y -m)
 
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: [RFC 0/2] Kbuild: Support nested composite objects

[Masahiro Yamada]

On Sat, Jan 23, 2021 at 4:27 AM Elliot Berman <eberman@codeaurora.org> wrote:
>
> This series was developed after discussion in https://lkml.org/lkml/2021/1/19/850
>
> The motivation for this series is an out-of-tree module which contains a large
> number of source files. This causes Kbuild to exceed the maximum command line
> argument length when linking the files. Proposal here permits composite objects
> to contain other composite objects. This allows the driver to split linking into
> several steps and avoid the maximum command line length error.


External modules often get the cold shoulder.

For example,
https://lore.kernel.org/patchwork/patch/1175556/#1372233


This problem has not been observed in the upstream tree.

I do not see a good reason to complicate the build
infrastructure for some external modules.






> Kbuild composite objects only supports one level of composite objects.
> That is, a composite object may only be composed of real compilable
> source files.
>
> As a simple example, the following Kbuild description is now supported:
>
> bar-a-y := a/bar0.o a/bar1.o
> bar-b-y := b/bar2.o b/bar3.o
>
> foo-objs := bar-a.o bar-b.o
>
> obj-m += foo.o
>
> Add such support by recursively searching for composite objects and
> listing them in $(multi-used-*) and $(real-obj-*).
>
> Elliot Berman (2):
>   Kbuild: Make composite object searching more generic
>   Kbuild: Support nested composite objects
>
>  scripts/Makefile.lib | 20 ++++++++++++++++----
>  1 file changed, 16 insertions(+), 4 deletions(-)
>
> --
> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
> a Linux Foundation Collaborative Project
>


-- 
Best Regards
Masahiro Yamada

Re: [RFC 0/2] Kbuild: Support nested composite objects

[Christoph Hellwig]

On Fri, Jan 22, 2021 at 11:27:16AM -0800, Elliot Berman wrote:
> This series was developed after discussion in https://lkml.org/lkml/2021/1/19/850
> 
> The motivation for this series is an out-of-tree module which contains a large
> number of source files. This causes Kbuild to exceed the maximum command line
> argument length when linking the files. Proposal here permits composite objects

Please don't even try to bloat the kernel build system for this.
Thanks!

Re: [RFC 0/2] Kbuild: Support nested composite objects

[Trent Piepho]

On Friday, January 22, 2021 11:27:16 AM PDT Elliot Berman wrote:
> The motivation for this series is an out-of-tree module which contains a
> large number of source files. This causes Kbuild to exceed the maximum
> command line argument length when linking the files. Proposal here

I think I am familiar with this module.

In addition to exceeding the argument length when linking, it will fail to 
compile due to 169 include directories being added to c_flags.  Depending on 
root path, this can also overflow the single argument limit of 128 kB.  
Perhaps you have not triggered this yet?  These patches here will not help 
with this issue.  But see below for correct and incorrect fixes.

The fundamental problem is that the kernel build system is designed to be 
recursive, with one makefile per directory.  This module has a single Kbuild 
of over 3000 lines for the entire tree.

Some of this is from this driver not using multiple modules for what is a 
giant driver (337 MB .ko file).  It should be split into multiple modules, if 
only to manage complexity by creating public vs private interfaces through 
choosing what symbols to export or not.  It would avoid merge conflicts on 
the single shared Kbuild file used tree-wide.  Potential name collisions 
across the 100+ include directories could be avoided by not having a shared 
set of include flags for the entire tree.

But some I think is also a limitation of kbuild.  Giant monolithic Kbuilds 
do not work well, but this is ok, because you are supposed to use a 
recursive make design with a Kbuild per directory.  But only non-module code 
may be split across directories.  This important design feature is not 
extended to modules.  A module must be be contained in one directory or 
monolithic tree.  Why should non-module code be allowed the complexity of a 
recursive build but code for a module is not?

I did split this driver.  I'm not aware of any way for me to send patches to 
Qualcomm.   I'd have to maintain this change myself and that would be very 
difficult.

So I worked around it without needing to modify the kernel build system and 
with a minimal patch to Qualcomm's code for me to maintain.

1st, use @file to inject the include directories.  I didn't test well, but I 
think this does not break kbuild's ability to detect changes in command 
lines.  

-ccflags-y += $(INCS)
+ccflags-y += @$(obj)/includes
+$(obj)/includes: FORCE
+       $(call if_changed,include_flag_file)
+
+quiet_cmd_include_flag_file = I-FLAGS $@
+cmd_include_flag_file = echo "$(INCS)" > $@
+
+# To make all objects depend on the file that contains the flags
+$(addprefix $(obj)/,$(OBJS)): $(obj)/includes

The last line might be a bit hacky.  I couldn't find any proper mechanism to 
manually add a dependency to object files.  It seems like auto dependency 
discovered headers are the only ones supported.

Next, we must deal with ~550 object files being linked at once.  The subject 
of this patch series.

-$(MODNAME)-y := $(OBJS)
+$(MODNAME)-y := blob1.o blob2.o blob3.o
+blob1-objs := $(wordlist 1, 200, $(OBJS))
+blob2-objs := $(wordlist 201, 400, $(OBJS))
+blob3-objs := $(wordlist 401, 700, $(OBJS))
+# Multiple levels of composite objects don't really work.  There is no rule
+# to create blobX.o unless we add them to the obj-m list, even though
+# wlan-y lists them and will be used to create wlan.o from them.  We'll get
+# blobX.ko modules we don't actually want by doing this.
+obj-$(CONFIG_QCA_CLD_WLAN) += blob1.o blob2.o blob3.o

The last line here is a total hack, but it does work.  By added blob1.ko as 
a module to be built, a rule to build blob1.o is also created, which will 
then be used by wlan.o's dependency on blob1.o