From: Anthony PERARD <anthony.perard@citrix.com>
To: <xen-devel@lists.xenproject.org>
Cc: Stefano Stabellini <sstabellini@kernel.org>,
Julien Grall <julien@xen.org>, Wei Liu <wl@xen.org>,
Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>,
George Dunlap <George.Dunlap@eu.citrix.com>,
Andrew Cooper <andrew.cooper3@citrix.com>,
Ian Jackson <ian.jackson@eu.citrix.com>,
Jan Beulich <jbeulich@suse.com>,
Anthony PERARD <anthony.perard@citrix.com>
Subject: [Xen-devel] [XEN PATCH v3 16/23] xen/build: introduce if_changed and if_changed_rule
Date: Wed, 26 Feb 2020 11:33:48 +0000 [thread overview]
Message-ID: <20200226113355.2532224-17-anthony.perard@citrix.com> (raw)
In-Reply-To: <20200226113355.2532224-1-anthony.perard@citrix.com>
The if_changed macro from Linux can record the command used to build a
target then compare it on rebuild. Thus if a command has changed, for
example due to introducing new flags in CFLAGS or due to using a
different compiler, the target will be rebuilt.
if_changed_rule checks dependencies like if_changed, but execute
rule_$(1) instead of cmd_$(1) when the command is different. A rule_
macro can call more than one cmd_ macro. One of the cmd_ macro in a
rule need to be call using a macro that record the command line, so
cmd_and_record is introduced. It is similar to cmd_and_fixup from
Linux but without a call to fixdep which we don't have yet. (We will
later replace cmd_and_record by cmd_and_fixup.)
Example of a rule_ macro:
define rule_cc_o_c
$(call cmd_and_record,cc_o_o)
$(call cmd,objcopy)
endef
This needs one of the call to use cmd_and_record, otherwise no .*.cmd
file will be created, and the target will keep been rebuilt.
In order for if_changed to works correctly, we need to load the .%.cmd
files that the macro generates, this is done by adding targets in to
the $(targets) variable. We use intermediate_targets to add %.init.o
dependency %.o to target since there aren't in obj-y.
We also add $(MAKECMDGOALS) to targets so that when running for
example `make common/memory.i`, make will load the associated .%.cmd
dependency file.
Beside the if_changed*, we import the machinery used for a "beautify
output". The important one is when running make with V=2 which help to
debug the makefiles by printing why a target is been rebuilt, via the
$(echo-why) macro.
if_changed and if_changed_rule aren't used yet.
Most of this code is copied from Linux v5.4.
Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
---
.gitignore | 1 +
xen/Makefile | 53 +++++++++++++++++-
xen/Rules.mk | 33 +++++++++++-
xen/scripts/Kbuild.include | 107 +++++++++++++++++++++++++++++++++++++
4 files changed, 192 insertions(+), 2 deletions(-)
diff --git a/.gitignore b/.gitignore
index 4ca679ddbc9a..c73f9f480780 100644
--- a/.gitignore
+++ b/.gitignore
@@ -6,6 +6,7 @@
*.o
*.d
*.d2
+.*.cmd
*.opic
*.a
*.so
diff --git a/xen/Makefile b/xen/Makefile
index da017dc29d36..fbd087e6f360 100644
--- a/xen/Makefile
+++ b/xen/Makefile
@@ -52,7 +52,57 @@ dist: install
ifeq ($(root-make-done),)
# section to run before calling Rules.mk, but only once.
+
+# Beautify output
+# ---------------------------------------------------------------------------
+#
+# Normally, we echo the whole command before executing it. By making
+# that echo $($(quiet)$(cmd)), we now have the possibility to set
+# $(quiet) to choose other forms of output instead, e.g.
+#
+# quiet_cmd_cc_o_c = Compiling $(RELDIR)/$@
+# cmd_cc_o_c = $(CC) $(c_flags) -c -o $@ $<
+#
+# If $(quiet) is empty, the whole command will be printed.
+# If it is set to "quiet_", only the short version will be printed.
+# If it is set to "silent_", nothing will be printed at all, since
+# the variable $(silent_cmd_cc_o_c) doesn't exist.
+#
+# A simple variant is to prefix commands with $(Q) - that's useful
+# for commands that shall be hidden in non-verbose mode.
#
+# $(Q)ln $@ :<
+#
+# If KBUILD_VERBOSE equals 0 then the above command will be hidden.
+# If KBUILD_VERBOSE equals 1 then the above command is displayed.
+#
+# To put more focus on warnings, be less verbose as default
+# Use 'make V=1' to see the full commands
+
+ifeq ("$(origin V)", "command line")
+ KBUILD_VERBOSE = $(V)
+endif
+ifndef KBUILD_VERBOSE
+ KBUILD_VERBOSE = 0
+endif
+
+ifeq ($(KBUILD_VERBOSE),1)
+ quiet =
+ Q =
+else
+ quiet=quiet_
+ Q = @
+endif
+
+# If the user is running make -s (silent mode), suppress echoing of
+# commands
+
+ifneq ($(findstring s,$(filter-out --%,$(MAKEFLAGS))),)
+ quiet=silent_
+endif
+
+export quiet Q KBUILD_VERBOSE
+
# To make sure we do not include .config for any of the *config targets
# catch them early, and hand them over to tools/kconfig/Makefile
@@ -258,7 +308,8 @@ _clean: delete-unfresh-files
$(MAKE) $(clean) arch/x86
$(MAKE) $(clean) test
$(MAKE) -f $(BASEDIR)/tools/kconfig/Makefile.kconfig ARCH=$(ARCH) SRCARCH=$(SRCARCH) clean
- find . \( -name "*.o" -o -name ".*.d" -o -name ".*.d2" -o -name "*.gcno" \) -exec rm -f {} \;
+ find . \( -name "*.o" -o -name ".*.d" -o -name ".*.d2" \
+ -o -name "*.gcno" -o -name ".*.cmd" \) -exec rm -f {} \;
rm -f include/asm $(TARGET) $(TARGET).gz $(TARGET).efi $(TARGET).efi.map $(TARGET)-syms $(TARGET)-syms.map *~ core
rm -f include/asm-*/asm-offsets.h
rm -f .banner
diff --git a/xen/Rules.mk b/xen/Rules.mk
index f1311c45a372..8807a2e21c94 100644
--- a/xen/Rules.mk
+++ b/xen/Rules.mk
@@ -38,6 +38,7 @@ ALL_OBJS-y += $(BASEDIR)/arch/$(TARGET_ARCH)/built_in.o
ALL_OBJS-$(CONFIG_CRYPTO) += $(BASEDIR)/crypto/built_in.o
# Initialise some variables
+targets :=
CFLAGS-y :=
AFLAGS-y :=
@@ -65,6 +66,10 @@ $(foreach o,$(filter-out %/,$(obj-y) $(obj-bin-y) $(extra-y)),$(eval $(call gend
subdir-y := $(subdir-y) $(filter %/, $(obj-y))
obj-y := $(patsubst %/, %/built_in.o, $(obj-y))
+# $(subdir-obj-y) is the list of objects in $(obj-y) which uses dir/ to
+# tell kbuild to descend
+subdir-obj-y := $(filter %/built_in.o, $(obj-y))
+
$(filter %.init.o,$(obj-y) $(obj-bin-y) $(extra-y)): CFLAGS-y += -DINIT_SECTIONS_ONLY
ifeq ($(CONFIG_COVERAGE),y)
@@ -120,6 +125,10 @@ else
endif
endif
+targets += built_in.o
+targets += $(filter-out $(subdir-obj-y), $(obj-y)) $(extra-y)
+targets += $(MAKECMDGOALS)
+
built_in_bin.o: $(obj-bin-y) $(extra-y)
ifeq ($(obj-bin-y),)
$(CC) $(a_flags) -c -x assembler /dev/null -o $@
@@ -128,7 +137,7 @@ else
endif
# Force execution of pattern rules (for which PHONY cannot be directly used).
-.PHONY: FORCE
+PHONY += FORCE
FORCE:
%/built_in.o: FORCE
@@ -176,4 +185,26 @@ $(filter %.init.o,$(obj-y) $(obj-bin-y) $(extra-y)): %.init.o: %.o Makefile
%.s: %.S Makefile
$(CPP) $(filter-out -Wa$(comma)%,$(a_flags)) $< -o $@
+# Add intermediate targets:
+# When building objects with specific suffix patterns, add intermediate
+# targets that the final targets are derived from.
+intermediate_targets = $(foreach sfx, $(2), \
+ $(patsubst %$(strip $(1)),%$(sfx), \
+ $(filter %$(strip $(1)), $(targets))))
+# %.init.o <- %.o
+targets += $(call intermediate_targets, .init.o, .o)
+
-include $(DEPS_INCLUDE)
+
+# Read all saved command lines and dependencies for the $(targets) we
+# may be building above, using $(if_changed{,_dep}). As an
+# optimization, we don't need to read them if the target does not
+# exist, we will rebuild anyway in that case.
+
+existing-targets := $(wildcard $(sort $(targets)))
+
+-include $(foreach f,$(existing-targets),$(dir $(f)).$(notdir $(f)).cmd)
+
+# Declare the contents of the PHONY variable as phony. We keep that
+# information in a variable so we can use it in if_changed and friends.
+.PHONY: $(PHONY)
diff --git a/xen/scripts/Kbuild.include b/xen/scripts/Kbuild.include
index 14bd4e110b45..f24d664db5ff 100644
--- a/xen/scripts/Kbuild.include
+++ b/xen/scripts/Kbuild.include
@@ -2,11 +2,30 @@
####
# kbuild: Generic definitions
+# Convenient variables
+squote := '
+empty :=
+space := $(empty) $(empty)
+space_escape := _-_SPACE_-_
+pound := \#
+
+###
+# Name of target with a '.' as filename prefix. foo/bar.o => foo/.bar.o
+dot-target = $(@D)/.$(@F)
+
###
# dependencies
DEPS = .*.d
DEPS_INCLUDE = $(addsuffix .d2, $(basename $(wildcard $(DEPS))))
+###
+# real prerequisites without phony targets
+real-prereqs = $(filter-out $(PHONY), $^)
+
+###
+# Escape single quote for use in echo statements
+escsq = $(subst $(squote),'\$(squote)',$1)
+
# as-insn: Check whether assembler supports an instruction.
# Usage: cflags-y += $(call as-insn,CC FLAGS,"insn",option-yes,option-no)
as-insn = $(if $(shell echo 'void _(void) { asm volatile ( $(2) ); }' \
@@ -32,3 +51,91 @@ cc-ifversion = $(shell [ $(CONFIG_GCC_VERSION)0 $(1) $(2)000 ] && echo $(3) || e
# Usage:
# $(MAKE) $(clean) dir
clean := -f $(BASEDIR)/scripts/Makefile.clean clean -C
+
+# echo command.
+# Short version is used, if $(quiet) equals `quiet_', otherwise full one.
+echo-cmd = $(if $($(quiet)cmd_$(1)),\
+ echo ' $(call escsq,$($(quiet)cmd_$(1)))$(echo-why)';)
+
+# printing commands
+cmd = @set -e; $(echo-cmd) $(cmd_$(1))
+
+###
+# if_changed - execute command if any prerequisite is newer than
+# target, or command line has changed
+# if_changed_rule - as if_changed but execute rule instead
+
+ifneq ($(KBUILD_NOCMDDEP),1)
+# Check if both commands are the same including their order. Result is empty
+# string if equal. User may override this check using make KBUILD_NOCMDDEP=1
+cmd-check = $(filter-out $(subst $(space),$(space_escape),$(strip $(cmd_$@))), \
+ $(subst $(space),$(space_escape),$(strip $(cmd_$1))))
+else
+cmd-check = $(if $(strip $(cmd_$@)),,1)
+endif
+
+# Replace >$< with >$$< to preserve $ when reloading the .cmd file
+# (needed for make)
+# Replace >#< with >$(pound)< to avoid starting a comment in the .cmd file
+# (needed for make)
+# Replace >'< with >'\''< to be able to enclose the whole string in '...'
+# (needed for the shell)
+make-cmd = $(call escsq,$(subst $(pound),$$(pound),$(subst $$,$$$$,$(cmd_$(1)))))
+
+# Find any prerequisites that is newer than target or that does not exist.
+# PHONY targets skipped in both cases.
+any-prereq = $(filter-out $(PHONY),$?)$(filter-out $(PHONY) $(wildcard $^),$^)
+
+# Execute command if command has changed or prerequisite(s) are updated.
+if_changed = $(if $(any-prereq)$(cmd-check), \
+ $(cmd); \
+ printf '%s\n' 'cmd_$@ := $(make-cmd)' > $(dot-target).cmd, @:)
+
+# Usage: $(call if_changed_rule,foo)
+# Will check if $(cmd_foo) or any of the prerequisites changed,
+# and if so will execute $(rule_foo).
+if_changed_rule = $(if $(any-prereq)$(cmd-check),$(rule_$(1)),@:)
+
+cmd_and_record = \
+ $(cmd); \
+ printf '%s\n' 'cmd_$@ := $(make-cmd)' > $(dot-target).cmd
+
+###
+# why - tell why a target got built
+# enabled by make V=2
+# Output (listed in the order they are checked):
+# (1) - due to target is PHONY
+# (2) - due to target missing
+# (3) - due to: file1.h file2.h
+# (4) - due to command line change
+# (5) - due to missing .cmd file
+# (6) - due to target not in $(targets)
+# (1) PHONY targets are always build
+# (2) No target, so we better build it
+# (3) Prerequisite is newer than target
+# (4) The command line stored in the file named dir/.target.cmd
+# differed from actual command line. This happens when compiler
+# options changes
+# (5) No dir/.target.cmd file (used to store command line)
+# (6) No dir/.target.cmd file and target not listed in $(targets)
+# This is a good hint that there is a bug in the kbuild file
+ifeq ($(KBUILD_VERBOSE),2)
+why = \
+ $(if $(filter $@, $(PHONY)),- due to target is PHONY, \
+ $(if $(wildcard $@), \
+ $(if $(any-prereq),- due to: $(any-prereq), \
+ $(if $(cmd-check), \
+ $(if $(cmd_$@),- due to command line change, \
+ $(if $(filter $@, $(targets)), \
+ - due to missing .cmd file, \
+ - due to $(notdir $@) not in $$(targets) \
+ ) \
+ ) \
+ ) \
+ ), \
+ - due to target missing \
+ ) \
+ )
+
+echo-why = $(call escsq, $(strip $(why)))
+endif
--
Anthony PERARD
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
next prev parent reply other threads:[~2020-02-26 11:41 UTC|newest]
Thread overview: 76+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-02-26 11:33 [Xen-devel] [XEN PATCH v3 00/23] xen: Build system improvements Anthony PERARD
2020-02-26 11:33 ` [Xen-devel] [XEN PATCH v3 01/23] xen/include: remove include of Config.mk Anthony PERARD
2020-02-26 11:33 ` [Xen-devel] [XEN PATCH v3 02/23] Makefile: Fix install-tests Anthony PERARD
2020-02-26 11:33 ` [Xen-devel] [XEN PATCH v3 03/23] xen/build: Remove confusing comment on the %.s:%.S rule Anthony PERARD
2020-02-26 11:53 ` Wei Liu
2020-02-26 11:33 ` [Xen-devel] [XEN PATCH v3 04/23] xen/build: remove use of AFLAGS-y Anthony PERARD
2020-02-26 12:58 ` Jan Beulich
2020-02-26 11:33 ` [Xen-devel] [XEN PATCH v3 05/23] xen/build: Allow to test clang .include without asm symlink Anthony PERARD
2020-02-27 9:05 ` Roger Pau Monné
2020-02-27 9:22 ` Jan Beulich
2020-02-26 11:33 ` [Xen-devel] [XEN PATCH v3 06/23] xen/build: Fix section-renaming of libfdt and libelf Anthony PERARD
2020-02-27 9:38 ` Jan Beulich
2020-02-26 11:33 ` [Xen-devel] [XEN PATCH v3 07/23] xen/build: Use obj-y += subdir/ instead of subdir-y Anthony PERARD
2020-02-27 9:22 ` Roger Pau Monné
2020-02-27 9:43 ` Jan Beulich
2020-03-04 14:14 ` Jan Beulich
2020-03-05 9:24 ` Jan Beulich
2020-03-05 13:42 ` Andrew Cooper
2020-03-05 15:02 ` Julien Grall
2020-03-05 15:59 ` Anthony PERARD
2020-03-05 16:31 ` Julien Grall
2020-03-09 6:46 ` Tian, Kevin
2020-02-26 11:33 ` [Xen-devel] [XEN PATCH v3 08/23] xen/build: use $(clean) shorthand for clean targets Anthony PERARD
2020-02-26 11:33 ` [Xen-devel] [XEN PATCH v3 09/23] xen/build: extract clean target from Rules.mk Anthony PERARD
2020-03-04 14:13 ` Jan Beulich
2020-02-26 11:33 ` [Xen-devel] [XEN PATCH v3 10/23] xen/build: run targets csopes, tags, .. without Rules.mk Anthony PERARD
2020-03-04 14:17 ` Jan Beulich
2020-02-26 11:33 ` [Xen-devel] [XEN PATCH v3 11/23] xen/build: make tests in test/ directly Anthony PERARD
2020-02-26 11:33 ` [Xen-devel] [XEN PATCH v3 12/23] xen/build: Move as-option-add to xen/ Anthony PERARD
2020-03-04 14:17 ` Jan Beulich
2020-02-26 11:33 ` [Xen-devel] [XEN PATCH v3 13/23] xen/build: include include/config/auto.conf in main Makefile Anthony PERARD
2020-03-04 14:29 ` Jan Beulich
2020-03-10 17:10 ` Anthony PERARD
2020-03-11 9:26 ` Jan Beulich
2020-02-26 11:33 ` [Xen-devel] [XEN PATCH v3 14/23] xen/build: use new $(c_flags) and $(a_flags) instead of $(CFLAGS) Anthony PERARD
2020-02-27 10:22 ` Roger Pau Monné
2020-03-10 17:55 ` Anthony PERARD
2020-03-04 14:42 ` Jan Beulich
2020-03-10 17:43 ` Anthony PERARD
2020-02-26 11:33 ` [Xen-devel] [XEN PATCH v3 15/23] xen/build: have the root Makefile generates the CFLAGS Anthony PERARD
2020-02-27 11:05 ` Roger Pau Monné
2020-03-17 18:05 ` Anthony PERARD
2020-03-19 16:24 ` Anthony PERARD
2020-03-23 15:11 ` Roger Pau Monné
2020-03-24 17:11 ` [Xen-devel] " Anthony PERARD
2020-03-04 15:00 ` Jan Beulich
2020-03-17 18:35 ` Anthony PERARD
2020-03-18 10:20 ` Jan Beulich
2020-02-26 11:33 ` Anthony PERARD [this message]
2020-03-04 15:45 ` [Xen-devel] [XEN PATCH v3 16/23] xen/build: introduce if_changed and if_changed_rule Jan Beulich
2020-03-18 10:44 ` Anthony PERARD
2020-03-18 11:14 ` Jan Beulich
2020-02-26 11:33 ` [Xen-devel] [XEN PATCH v3 17/23] xen/build: Start using if_changed Anthony PERARD
2020-02-27 13:09 ` Roger Pau Monné
2020-03-04 16:00 ` Jan Beulich
2020-03-18 10:52 ` Anthony PERARD
2020-02-26 11:33 ` [Xen-devel] [XEN PATCH v3 18/23] xen/build: use if_changed on built_in.o Anthony PERARD
2020-03-04 16:03 ` Jan Beulich
2020-03-18 10:55 ` Anthony PERARD
2020-02-26 11:33 ` [Xen-devel] [XEN PATCH v3 19/23] xen/build: Use if_changed_rules with %.o:%.c targets Anthony PERARD
2020-03-04 16:09 ` Jan Beulich
2020-03-18 11:14 ` Anthony PERARD
2020-02-26 11:33 ` [Xen-devel] [XEN PATCH v3 20/23] xen/build: factorise generation of the linker scripts Anthony PERARD
2020-02-27 13:14 ` Roger Pau Monné
2020-03-05 11:07 ` Jan Beulich
2020-03-05 11:05 ` Jan Beulich
2020-03-18 11:59 ` Anthony PERARD
2020-02-26 11:33 ` [Xen-devel] [XEN PATCH v3 21/23] xen/build: Use if_changed for prelink*.o Anthony PERARD
2020-02-27 13:16 ` Roger Pau Monné
2020-03-04 16:12 ` Jan Beulich
2020-02-26 11:33 ` [Xen-devel] [XEN PATCH v3 22/23] xen, symbols: rework file symbols selection Anthony PERARD
2020-03-05 14:44 ` Jan Beulich
2020-02-26 11:33 ` [Xen-devel] [XEN PATCH v3 23/23] xen/build: use if_changed to build guest_%.o Anthony PERARD
2020-03-05 15:12 ` Jan Beulich
2020-02-27 21:17 ` [Xen-devel] [XEN PATCH v3 00/23] xen: Build system improvements Stewart Hildebrand
2020-03-06 10:12 ` Anthony PERARD
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200226113355.2532224-17-anthony.perard@citrix.com \
--to=anthony.perard@citrix.com \
--cc=George.Dunlap@eu.citrix.com \
--cc=andrew.cooper3@citrix.com \
--cc=ian.jackson@eu.citrix.com \
--cc=jbeulich@suse.com \
--cc=julien@xen.org \
--cc=konrad.wilk@oracle.com \
--cc=sstabellini@kernel.org \
--cc=wl@xen.org \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).