[PATCH v2 0/6] kbuild: optimize output directory creation

[PATCH v2 0/6] kbuild: optimize output directory creation

[Masahiro Yamada]

I looked into the build scripts, focusing on "mkdir" optimization.

With this series, I succeeded in speeding up the incremental build
with O= option.

The following is the result of "time make O=foo",
where "foo" is the output directory that has already been built.

Before:

real    0m8.322s
user    0m4.324s
sys     0m1.220s

After:

real    0m6.989s
user    0m4.168s
sys     0m1.080s

4/6 gave the biggest impact.



Masahiro Yamada (6):
  kbuild: create directory for make cache only when necessary
  kbuild: remove redundant $(wildcard ...) for cmd_files calculation
  kbuild: filter-out PHONY targets from "targets"
  kbuild: create object directories simpler and faster
  kbuild: optimize object directory creation for incremental build
  kbuild: remove redundant mkdir from ./Kbuild

 Kbuild                       |  2 --
 Makefile                     |  3 +--
 arch/x86/entry/vdso/Makefile |  4 ----
 scripts/Kbuild.include       | 13 +++++++++----
 scripts/Makefile.build       | 25 +++++++++++++------------
 scripts/Makefile.headersinst |  3 +--
 scripts/Makefile.host        | 12 ------------
 scripts/Makefile.lib         |  5 -----
 scripts/Makefile.modpost     |  3 +--
 9 files changed, 25 insertions(+), 45 deletions(-)

-- 
2.7.4

[PATCH v2 1/6] kbuild: create directory for make cache only when necessary

[Masahiro Yamada]

Currently, the existence of $(dir $(make-cache)) is always checked,
and created if it is missing.

We can avoid unnecessary system calls by some tricks.

[1] If KBUILD_SRC is unset, we are building in the source tree.
    The output directory checks can be entirely skipped.
[2] If at least one cache data is found, it means the cache file
    was included.  Obviously its directory exists.  Skip "mkdir -p".
[3] If Makefile does not contain any call of __run-and-store, it will
    not create a cache file.  No need to create its directory.
[4] The "mkdir -p" should be only invoked by the first call of
    __run-and-store

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Douglas Anderson <dianders@chromium.org>
---

Changes in v2:
  - Use boolean flag "create-cache-dir" for clarification
  - Fix a typo in git-log:  Obiously -> Obviously

 scripts/Kbuild.include | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/scripts/Kbuild.include b/scripts/Kbuild.include
index be1c9d6..065324a 100644
--- a/scripts/Kbuild.include
+++ b/scripts/Kbuild.include
@@ -99,18 +99,19 @@ cc-cross-prefix =  \
 
 # Include values from last time
 make-cache := $(if $(KBUILD_EXTMOD),$(KBUILD_EXTMOD)/,$(if $(obj),$(obj)/)).cache.mk
-ifeq ($(wildcard $(dir $(make-cache))),)
-$(shell mkdir -p '$(dir $(make-cache))')
-endif
 $(make-cache): ;
 -include $(make-cache)
 
+cached-data := $(filter __cached_%, $(.VARIABLES))
+
 # If cache exceeds 1000 lines, shrink it down to 500.
-ifneq ($(word 1000,$(filter __cached_%, $(.VARIABLES))),)
+ifneq ($(word 1000,$(cached-data)),)
 $(shell tail -n 500 $(make-cache) > $(make-cache).tmp; \
 	mv $(make-cache).tmp $(make-cache))
 endif
 
+create-cache-dir := $(if $(KBUILD_SRC),$(if $(cache-data),,1))
+
 # Usage: $(call __sanitize-opt,Hello=Hola$(comma)Goodbye Adios)
 #
 # Convert all '$', ')', '(', '\', '=', ' ', ',', ':' to '_'
@@ -136,6 +137,10 @@ __sanitize-opt = $(subst $$,_,$(subst $(right_paren),_,$(subst $(left_paren),_,$
 define __run-and-store
 ifeq ($(origin $(1)),undefined)
   $$(eval $(1) := $$(shell $$(2)))
+ifeq ($(create-cache-dir),1)
+  $$(shell mkdir -p $(dir $(make-cache)))
+  $$(eval create-cache-dir :=)
+endif
   $$(shell echo '$(1) := $$($(1))' >> $(make-cache))
 endif
 endef
-- 
2.7.4

[PATCH v2 2/6] kbuild: remove redundant $(wildcard ...) for cmd_files calculation

[Masahiro Yamada]

I do not see any reason why $(wildcard ...) needs to be called twice
for computing cmd_files.  Remove the first one.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v2:
  - Fix "$$" in scripts/Makefile.headersinst per Douglas

 Makefile                     | 3 +--
 scripts/Makefile.build       | 3 +--
 scripts/Makefile.headersinst | 3 +--
 scripts/Makefile.modpost     | 3 +--
 4 files changed, 4 insertions(+), 8 deletions(-)

diff --git a/Makefile b/Makefile
index a7476e6..58dd245 100644
--- a/Makefile
+++ b/Makefile
@@ -1693,8 +1693,7 @@ cmd_crmodverdir = $(Q)mkdir -p $(MODVERDIR) \
 
 # read all saved command lines
 
-targets := $(wildcard $(sort $(targets)))
-cmd_files := $(wildcard .*.cmd $(foreach f,$(targets),$(dir $(f)).$(notdir $(f)).cmd))
+cmd_files := $(wildcard .*.cmd $(foreach f,$(sort $(targets)),$(dir $(f)).$(notdir $(f)).cmd))
 
 ifneq ($(cmd_files),)
   $(cmd_files): ;	# Do not try to update included dependency files
diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index 061d0c3..62d5314 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -583,8 +583,7 @@ FORCE:
 # optimization, we don't need to read them if the target does not
 # exist, we will rebuild anyway in that case.
 
-targets := $(wildcard $(sort $(targets)))
-cmd_files := $(wildcard $(foreach f,$(targets),$(dir $(f)).$(notdir $(f)).cmd))
+cmd_files := $(wildcard $(foreach f,$(sort $(targets)),$(dir $(f)).$(notdir $(f)).cmd))
 
 ifneq ($(cmd_files),)
   include $(cmd_files)
diff --git a/scripts/Makefile.headersinst b/scripts/Makefile.headersinst
index 5692d7a..c6fb2b7 100644
--- a/scripts/Makefile.headersinst
+++ b/scripts/Makefile.headersinst
@@ -114,9 +114,8 @@ $(check-file): scripts/headers_check.pl $(output-files) FORCE
 
 endif
 
-targets := $(wildcard $(sort $(targets)))
 cmd_files := $(wildcard \
-             $(foreach f,$(targets),$(dir $(f)).$(notdir $(f)).cmd))
+             $(foreach f,$(sort $(targets)),$(dir $(f)).$(notdir $(f)).cmd))
 
 ifneq ($(cmd_files),)
 	include $(cmd_files)
diff --git a/scripts/Makefile.modpost b/scripts/Makefile.modpost
index 16923ba..cf125c1 100644
--- a/scripts/Makefile.modpost
+++ b/scripts/Makefile.modpost
@@ -143,8 +143,7 @@ FORCE:
 # optimization, we don't need to read them if the target does not
 # exist, we will rebuild anyway in that case.
 
-targets := $(wildcard $(sort $(targets)))
-cmd_files := $(wildcard $(foreach f,$(targets),$(dir $(f)).$(notdir $(f)).cmd))
+cmd_files := $(wildcard $(foreach f,$(sort $(targets)),$(dir $(f)).$(notdir $(f)).cmd))
 
 ifneq ($(cmd_files),)
   include $(cmd_files)
-- 
2.7.4

[PATCH v2 3/6] kbuild: filter-out PHONY targets from "targets"

[Masahiro Yamada]

The variable "targets" contains object paths for which existing .*.cmd
files should be included.

scripts/Makefile.build automatically adds $(MAKECMDGOALS) to "targets"
as follows:

  targets += $(extra-y) $(MAKECMDGOALS) $(always)

The $(MAKECMDGOALS) is a PHONY target in several places.  PHONY targets
never create .*.cmd files.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v2:
  - Newly added

 scripts/Makefile.build | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index 62d5314..6f60377 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -562,7 +562,7 @@ $(multi-used-m): FORCE
 $(call multi_depend, $(multi-used-m), .o, -objs -y -m)
 
 targets += $(multi-used-y) $(multi-used-m)
-
+targets := $(filter-out $(PHONY), $(targets))
 
 # Descending
 # ---------------------------------------------------------------------------
-- 
2.7.4

[PATCH v2 4/6] kbuild: create object directories simpler and faster

[Masahiro Yamada]

For the out-of-tree build, scripts/Makefile.build creates output
directories, but this operation is not efficient.

scripts/Makefile.lib calculates obj-dirs as follows:

  obj-dirs := $(dir $(multi-objs) $(obj-y))

Please notice $(sort ...) is not used here.  Usually the result is
as many "./" as objects here.

For a lot of duplicated paths, the following command is invoked.

  _dummy := $(foreach d,$(obj-dirs), $(shell [ -d $(d) ] || mkdir -p $(d)))

Then, the costly shell command is run over and over again.

I see many points for optimization:

[1] Use $(sort ...) to cut down duplicated paths before passing them
    to system call
[2] Use single $(shell ...) instead of repeating it with $(foreach ...)
    This will reduce forking.
[3] We can calculate obj-dirs more simply.  Most of objects are already
    accumulated in $(targets).  So, $(dir $(targets)) is fine and more
    comprehensive.

I also removed ugly code in arch/x86/entry/vdso/Makefile.  This is now
really unnecessary.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Ingo Molnar <mingo@kernel.org>
Tested-by: Douglas Anderson <dianders@chromium.org>
---

Changes in v2:
  - Complete "host-objdirs" removal
  - Remove (obj) from mkdir argument per Cao

 arch/x86/entry/vdso/Makefile |  4 ----
 scripts/Makefile.build       | 15 ++++++---------
 scripts/Makefile.host        | 12 ------------
 scripts/Makefile.lib         |  5 -----
 4 files changed, 6 insertions(+), 30 deletions(-)

diff --git a/arch/x86/entry/vdso/Makefile b/arch/x86/entry/vdso/Makefile
index d540966..f8e3d85 100644
--- a/arch/x86/entry/vdso/Makefile
+++ b/arch/x86/entry/vdso/Makefile
@@ -129,10 +129,6 @@ $(obj)/vdsox32.so.dbg: $(src)/vdsox32.lds $(vobjx32s) FORCE
 CPPFLAGS_vdso32.lds = $(CPPFLAGS_vdso.lds)
 VDSO_LDFLAGS_vdso32.lds = -m32 -Wl,-m,elf_i386 -Wl,-soname=linux-gate.so.1
 
-# This makes sure the $(obj) subdirectory exists even though vdso32/
-# is not a kbuild sub-make subdirectory.
-override obj-dirs = $(dir $(obj)) $(obj)/vdso32/
-
 targets += vdso32/vdso32.lds
 targets += vdso32/note.o vdso32/system_call.o vdso32/sigreturn.o
 targets += vdso32/vclock_gettime.o
diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index 6f60377..496ecd8 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -64,15 +64,6 @@ ifneq ($(hostprogs-y)$(hostprogs-m)$(hostlibs-y)$(hostlibs-m)$(hostcxxlibs-y)$(h
 include scripts/Makefile.host
 endif
 
-ifneq ($(KBUILD_SRC),)
-# Create output directory if not already present
-_dummy := $(shell [ -d $(obj) ] || mkdir -p $(obj))
-
-# Create directories for object files if directory does not exist
-# Needed when obj-y := dir/file.o syntax is used
-_dummy := $(foreach d,$(obj-dirs), $(shell [ -d $(d) ] || mkdir -p $(d)))
-endif
-
 ifndef obj
 $(warning kbuild: Makefile.build is included improperly)
 endif
@@ -589,6 +580,12 @@ ifneq ($(cmd_files),)
   include $(cmd_files)
 endif
 
+ifneq ($(KBUILD_SRC),)
+# Create directories for object files if they do not exist
+obj-dirs := $(sort $(obj) $(patsubst %/,%, $(dir $(targets))))
+$(shell mkdir -p $(obj-dirs))
+endif
+
 # Declare the contents of the .PHONY variable as phony.  We keep that
 # information in a variable se we can use it in if_changed and friends.
 
diff --git a/scripts/Makefile.host b/scripts/Makefile.host
index 9cfd5c8..a5e0383 100644
--- a/scripts/Makefile.host
+++ b/scripts/Makefile.host
@@ -48,15 +48,6 @@ host-cxxobjs	:= $(sort $(foreach m,$(host-cxxmulti),$($(m)-cxxobjs)))
 host-cshobjs	:= $(sort $(foreach m,$(host-cshlib),$($(m:.so=-objs))))
 host-cxxshobjs	:= $(sort $(foreach m,$(host-cxxshlib),$($(m:.so=-objs))))
 
-# output directory for programs/.o files
-# hostprogs-y := tools/build may have been specified.
-# Retrieve also directory of .o files from prog-objs or prog-cxxobjs notation
-host-objdirs := $(dir $(__hostprogs) $(host-cobjs) $(host-cxxobjs))
-
-host-objdirs := $(strip $(sort $(filter-out ./,$(host-objdirs))))
-
-
-__hostprogs     := $(addprefix $(obj)/,$(__hostprogs))
 host-csingle	:= $(addprefix $(obj)/,$(host-csingle))
 host-cmulti	:= $(addprefix $(obj)/,$(host-cmulti))
 host-cobjs	:= $(addprefix $(obj)/,$(host-cobjs))
@@ -66,9 +57,6 @@ host-cshlib	:= $(addprefix $(obj)/,$(host-cshlib))
 host-cxxshlib	:= $(addprefix $(obj)/,$(host-cxxshlib))
 host-cshobjs	:= $(addprefix $(obj)/,$(host-cshobjs))
 host-cxxshobjs	:= $(addprefix $(obj)/,$(host-cxxshobjs))
-host-objdirs    := $(addprefix $(obj)/,$(host-objdirs))
-
-obj-dirs += $(host-objdirs)
 
 #####
 # Handle options to gcc. Support building with separate output directory
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 4d88ad7..5fbc46d 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -50,15 +50,11 @@ single-used-m := $(sort $(filter-out $(multi-used-m),$(obj-m)))
 # objects depend on those (obviously)
 multi-objs-y := $(foreach m, $(multi-used-y), $($(m:.o=-objs)) $($(m:.o=-y)))
 multi-objs-m := $(foreach m, $(multi-used-m), $($(m:.o=-objs)) $($(m:.o=-y)))
-multi-objs   := $(multi-objs-y) $(multi-objs-m)
 
 # $(subdir-obj-y) is the list of objects in $(obj-y) which uses dir/ to
 # tell kbuild to descend
 subdir-obj-y := $(filter %/built-in.o, $(obj-y))
 
-# $(obj-dirs) is a list of directories that contain object files
-obj-dirs := $(dir $(multi-objs) $(obj-y))
-
 # Replace multi-part objects by their individual parts, look at local dir only
 real-objs-y := $(foreach m, $(filter-out $(subdir-obj-y), $(obj-y)), $(if $(strip $($(m:.o=-objs)) $($(m:.o=-y))),$($(m:.o=-objs)) $($(m:.o=-y)),$(m))) $(extra-y)
 real-objs-m := $(foreach m, $(obj-m), $(if $(strip $($(m:.o=-objs)) $($(m:.o=-y)) $($(m:.o=-m))),$($(m:.o=-objs)) $($(m:.o=-y)) $($(m:.o=-m)),$(m)))
@@ -81,7 +77,6 @@ multi-used-m	:= $(addprefix $(obj)/,$(multi-used-m))
 multi-objs-y	:= $(addprefix $(obj)/,$(multi-objs-y))
 multi-objs-m	:= $(addprefix $(obj)/,$(multi-objs-m))
 subdir-ym	:= $(addprefix $(obj)/,$(subdir-ym))
-obj-dirs	:= $(addprefix $(obj)/,$(obj-dirs))
 
 # These flags are needed for modversions and compiling, so we define them here
 # $(modname_flags) defines KBUILD_MODNAME as the name of the module it will
-- 
2.7.4

[PATCH v2 5/6] kbuild: optimize object directory creation for incremental build

[Masahiro Yamada]

The previous commit largely optimized the object directory creation.
We can optimize it more for incremental build.

There are already *.cmd files in the output directory.  The existing
*.cmd files have been picked up by $(wildcard ...).  Obviously,
directories containing them exist too, so we can skip "mkdir -p".

With this, Kbuild runs almost zero "mkdir -p" in incremental building.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v2:
  - Remove "." from the filter-out list

 scripts/Makefile.build | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index 496ecd8..8624924 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -583,8 +583,13 @@ endif
 ifneq ($(KBUILD_SRC),)
 # Create directories for object files if they do not exist
 obj-dirs := $(sort $(obj) $(patsubst %/,%, $(dir $(targets))))
+# If cmd_files exist, their directories apparently exist.  Skip mkdir.
+exist-dirs := $(sort $(patsubst %/,%, $(dir $(cmd_files))))
+obj-dirs := $(strip $(filter-out $(exist-dirs), $(obj-dirs)))
+ifneq ($(obj-dirs),)
 $(shell mkdir -p $(obj-dirs))
 endif
+endif
 
 # Declare the contents of the .PHONY variable as phony.  We keep that
 # information in a variable se we can use it in if_changed and friends.
-- 
2.7.4

[PATCH v2 6/6] kbuild: remove redundant mkdir from ./Kbuild

[Masahiro Yamada]

These two targets are added to "targets".  Their directories are
automatically created.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v2: None

 Kbuild | 2 --
 1 file changed, 2 deletions(-)

diff --git a/Kbuild b/Kbuild
index 94c7527..f1997d8 100644
--- a/Kbuild
+++ b/Kbuild
@@ -17,7 +17,6 @@ targets := kernel/bounds.s
 
 # We use internal kbuild rules to avoid the "is up to date" message from make
 kernel/bounds.s: kernel/bounds.c FORCE
-	$(Q)mkdir -p $(dir $@)
 	$(call if_changed_dep,cc_s_c)
 
 $(obj)/$(bounds-file): kernel/bounds.s FORCE
@@ -53,7 +52,6 @@ targets += arch/$(SRCARCH)/kernel/asm-offsets.s
 # We use internal kbuild rules to avoid the "is up to date" message from make
 arch/$(SRCARCH)/kernel/asm-offsets.s: arch/$(SRCARCH)/kernel/asm-offsets.c \
                                       $(obj)/$(timeconst-file) $(obj)/$(bounds-file) FORCE
-	$(Q)mkdir -p $(dir $@)
 	$(call if_changed_dep,cc_s_c)
 
 $(obj)/$(offsets-file): arch/$(SRCARCH)/kernel/asm-offsets.s FORCE
-- 
2.7.4

Re: [PATCH v2 0/6] kbuild: optimize output directory creation

[Masahiro Yamada]

2017-11-13 19:29 GMT+09:00 Masahiro Yamada <yamada.masahiro@socionext.com>:
>
> I looked into the build scripts, focusing on "mkdir" optimization.
>
> With this series, I succeeded in speeding up the incremental build
> with O= option.
>
> The following is the result of "time make O=foo",
> where "foo" is the output directory that has already been built.
>
> Before:
>
> real    0m8.322s
> user    0m4.324s
> sys     0m1.220s
>
> After:
>
> real    0m6.989s
> user    0m4.168s
> sys     0m1.080s
>
> 4/6 gave the biggest impact.
>
>


Series, applied to linux-kbuild/kbuild.

-- 
Best Regards
Masahiro Yamada