All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v7 0/6] Reproducible build
       [not found] <drivers/net/vmxnet3/vmxnet3_rxtx.c.rej>
@ 2017-09-05 20:51 ` luca.boccassi
  2017-09-05 20:51   ` [PATCH v7 1/6] mk: sort list of shared objects in linker script luca.boccassi
                     ` (8 more replies)
  0 siblings, 9 replies; 23+ messages in thread
From: luca.boccassi @ 2017-09-05 20:51 UTC (permalink / raw)
  To: dev; +Cc: ferruh.yigit, Luca Boccassi

From: Luca Boccassi <luca.boccassi@gmail.com>

In the past couple of years a concerted effort among almost all Linux
distros has been striving toward achieving reproducible builds. [1]
This involves changes to the toolchain, new tools and CI systems. [2]

v1 fixed the documentation, examples and linker script generation.
v2 fixes all problems, which were caused by unstable order of headers
inclusion, source files listing and object file listing when passing
them to the compiler.
DPDK's build, at least with the default configuration, is fully
reproducible with this patch series as tested by the Reproducible
Builds developers experimental toolchain. [3]

v3 restores the first patch, which was eaten by git send-email.

v4 drops the patch that reorders rebuilds, and adds a patch to make
the inclusion of headers deterministic with regards to GCC embedding
the full file path when expading __FILE__ and when writing the
directory listing in the DWARF objects.
It also drops the first 2 patches which have already been merged.

v5 adds the -I$(SRCDIR) workaround to librte_eal linuxapp's and
librte_gro's Makefiles.

v6 fixes copypasta added in v5 - librte-gro's Makefile should have used
$(SRCDIR) rather than $(SRCDIR)/include.

v7 dropped the big patch to change the CFLAGS in all libraries Makefiles
and instead uses a much simpler approach suggested by Ferruh (thanks!),
which consists in installing the library public headers symlinks before
building. Although the drawback is that the DWARF directory listing
will always include the user configured build output directory, it is
a much smaller patch that will be rendered obsolete by the move to Meson
anyway.

[1] https://reproducible-builds.org/
[2] https://reproducible-builds.org/tools/
[3] https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain#Us

Luca Boccassi (6):
  mk: sort list of shared objects in linker script
  mk: sort list of files in examples.dox
  mk: sort headers before wildcard inclusion
  mk: sort source files before passing them to the compiler
  mk: sort object files when building deps lists
  mk: install symlinks before build step

 drivers/net/cxgbe/Makefile                                 | 2 +-
 drivers/net/e1000/Makefile                                 | 2 +-
 drivers/net/fm10k/Makefile                                 | 2 +-
 drivers/net/i40e/Makefile                                  | 2 +-
 drivers/net/ixgbe/Makefile                                 | 2 +-
 drivers/net/qede/Makefile                                  | 2 +-
 drivers/net/sfc/Makefile                                   | 2 +-
 drivers/net/thunderx/Makefile                              | 2 +-
 examples/ip_pipeline/Makefile                              | 2 +-
 examples/multi_process/client_server_mp/mp_server/Makefile | 2 +-
 examples/server_node_efd/server/Makefile                   | 2 +-
 lib/librte_eal/common/Makefile                             | 2 +-
 mk/rte.app.mk                                              | 4 ++--
 mk/rte.combinedlib.mk                                      | 2 +-
 mk/rte.hostapp.mk                                          | 4 ++--
 mk/rte.lib.mk                                              | 7 ++++++-
 mk/rte.sdkdoc.mk                                           | 2 +-
 mk/rte.shared.mk                                           | 4 ++--
 18 files changed, 26 insertions(+), 21 deletions(-)

-- 
2.11.0

^ permalink raw reply	[flat|nested] 23+ messages in thread
* [PATCH v7 1/6] mk: sort list of shared objects in linker script
  2017-09-05 20:51 ` [PATCH v7 0/6] Reproducible build luca.boccassi
@ 2017-09-05 20:51   ` luca.boccassi
  2017-09-05 20:51   ` [PATCH v7 2/6] mk: sort list of files in examples.dox luca.boccassi
                     ` (7 subsequent siblings)
  8 siblings, 0 replies; 23+ messages in thread
From: luca.boccassi @ 2017-09-05 20:51 UTC (permalink / raw)
  To: dev; +Cc: ferruh.yigit, Luca Boccassi

From: Luca Boccassi <luca.boccassi@gmail.com>

The output of wildcard might not be stable and depend on the
filesystem and other factors.
This means the content libdpdk.so linker script might change between
builds from the same sources.
Run the list through sort to ensure reproducibility.

Signed-off-by: Luca Boccassi <luca.boccassi@gmail.com>
---
 mk/rte.combinedlib.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mk/rte.combinedlib.mk b/mk/rte.combinedlib.mk
index 449358b33..2ab7ee8a1 100644
--- a/mk/rte.combinedlib.mk
+++ b/mk/rte.combinedlib.mk
@@ -42,7 +42,7 @@ endif
 RTE_LIBNAME := dpdk
 COMBINEDLIB := lib$(RTE_LIBNAME)$(EXT)
 
-LIBS := $(filter-out $(COMBINEDLIB), $(notdir $(wildcard $(RTE_OUTPUT)/lib/*$(EXT))))
+LIBS := $(filter-out $(COMBINEDLIB), $(sort $(notdir $(wildcard $(RTE_OUTPUT)/lib/*$(EXT)))))
 
 all: FORCE
 	$(Q)echo "GROUP ( $(LIBS) )" > $(RTE_OUTPUT)/lib/$(COMBINEDLIB)
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v7 2/6] mk: sort list of files in examples.dox
  2017-09-05 20:51 ` [PATCH v7 0/6] Reproducible build luca.boccassi
  2017-09-05 20:51   ` [PATCH v7 1/6] mk: sort list of shared objects in linker script luca.boccassi
@ 2017-09-05 20:51   ` luca.boccassi
  2017-09-05 20:51   ` [PATCH v7 3/6] mk: sort headers before wildcard inclusion luca.boccassi
                     ` (6 subsequent siblings)
  8 siblings, 0 replies; 23+ messages in thread
From: luca.boccassi @ 2017-09-05 20:51 UTC (permalink / raw)
  To: dev; +Cc: ferruh.yigit, Luca Boccassi

From: Luca Boccassi <luca.boccassi@gmail.com>

The result of find might not be stable depending on external
conditions.
Pipe it through LC_ALL=C sort to ensure reproducible results when
generating examples.dox.

Signed-off-by: Luca Boccassi <luca.boccassi@gmail.com>
---
 mk/rte.sdkdoc.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mk/rte.sdkdoc.mk b/mk/rte.sdkdoc.mk
index c0eaa3502..de31b78cf 100644
--- a/mk/rte.sdkdoc.mk
+++ b/mk/rte.sdkdoc.mk
@@ -93,7 +93,7 @@ $(API_EXAMPLES): api-html-clean
 	$(Q)mkdir -p $(@D)
 	@printf '/**\n' > $(API_EXAMPLES)
 	@printf '@page examples DPDK Example Programs\n\n' >> $(API_EXAMPLES)
-	@find examples -type f -name '*.c' -printf '@example %p\n' >> $(API_EXAMPLES)
+	@find examples -type f -name '*.c' -printf '@example %p\n' | LC_ALL=C sort >> $(API_EXAMPLES)
 	@printf '*/\n' >> $(API_EXAMPLES)
 
 guides-pdf-clean: guides-pdf-img-clean
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v7 3/6] mk: sort headers before wildcard inclusion
  2017-09-05 20:51 ` [PATCH v7 0/6] Reproducible build luca.boccassi
  2017-09-05 20:51   ` [PATCH v7 1/6] mk: sort list of shared objects in linker script luca.boccassi
  2017-09-05 20:51   ` [PATCH v7 2/6] mk: sort list of files in examples.dox luca.boccassi
@ 2017-09-05 20:51   ` luca.boccassi
  2017-09-05 20:51   ` [PATCH v7 4/6] mk: sort source files before passing them to the compiler luca.boccassi
                     ` (5 subsequent siblings)
  8 siblings, 0 replies; 23+ messages in thread
From: luca.boccassi @ 2017-09-05 20:51 UTC (permalink / raw)
  To: dev; +Cc: ferruh.yigit, Luca Boccassi

From: Luca Boccassi <luca.boccassi@gmail.com>

In order to achieve fully reproducible builds, always use the same
inclusion order for headers in the Makefiles.

Signed-off-by: Luca Boccassi <luca.boccassi@gmail.com>
---
 examples/ip_pipeline/Makefile                              | 2 +-
 examples/multi_process/client_server_mp/mp_server/Makefile | 2 +-
 examples/server_node_efd/server/Makefile                   | 2 +-
 lib/librte_eal/common/Makefile                             | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/examples/ip_pipeline/Makefile b/examples/ip_pipeline/Makefile
index dc7e0ddd7..12ce0a1d5 100644
--- a/examples/ip_pipeline/Makefile
+++ b/examples/ip_pipeline/Makefile
@@ -43,7 +43,7 @@ APP = ip_pipeline
 
 VPATH += $(SRCDIR)/pipeline
 
-INC += $(wildcard *.h) $(wildcard pipeline/*.h)
+INC += $(sort $(wildcard *.h)) $(sort $(wildcard pipeline/*.h))
 
 # all source are stored in SRCS-y
 SRCS-$(CONFIG_RTE_LIBRTE_PIPELINE) := main.c
diff --git a/examples/multi_process/client_server_mp/mp_server/Makefile b/examples/multi_process/client_server_mp/mp_server/Makefile
index 5552999b5..160c17b68 100644
--- a/examples/multi_process/client_server_mp/mp_server/Makefile
+++ b/examples/multi_process/client_server_mp/mp_server/Makefile
@@ -49,7 +49,7 @@ APP = mp_server
 # all source are stored in SRCS-y
 SRCS-y := main.c init.c args.c
 
-INC := $(wildcard *.h)
+INC := $(sort $(wildcard *.h))
 
 CFLAGS += $(WERROR_FLAGS) -O3
 CFLAGS += -I$(SRCDIR)/../shared
diff --git a/examples/server_node_efd/server/Makefile b/examples/server_node_efd/server/Makefile
index a2f2f361b..9f1fe2894 100644
--- a/examples/server_node_efd/server/Makefile
+++ b/examples/server_node_efd/server/Makefile
@@ -49,7 +49,7 @@ APP = server
 # all source are stored in SRCS-y
 SRCS-y := main.c init.c args.c
 
-INC := $(wildcard *.h)
+INC := $(sort $(wildcard *.h))
 
 CFLAGS += $(WERROR_FLAGS) -O3
 CFLAGS += -I$(SRCDIR)/../shared
diff --git a/lib/librte_eal/common/Makefile b/lib/librte_eal/common/Makefile
index e8fd67a27..4e6baaa72 100644
--- a/lib/librte_eal/common/Makefile
+++ b/lib/librte_eal/common/Makefile
@@ -49,7 +49,7 @@ GENERIC_INC += rte_vect.h rte_pause.h rte_io.h
 
 # defined in mk/arch/$(RTE_ARCH)/rte.vars.mk
 ARCH_DIR ?= $(RTE_ARCH)
-ARCH_INC := $(notdir $(wildcard $(RTE_SDK)/lib/librte_eal/common/include/arch/$(ARCH_DIR)/*.h))
+ARCH_INC := $(sort $(notdir $(wildcard $(RTE_SDK)/lib/librte_eal/common/include/arch/$(ARCH_DIR)/*.h)))
 
 SYMLINK-$(CONFIG_RTE_LIBRTE_EAL)-include := $(addprefix include/,$(INC))
 SYMLINK-$(CONFIG_RTE_LIBRTE_EAL)-include += \
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v7 4/6] mk: sort source files before passing them to the compiler
  2017-09-05 20:51 ` [PATCH v7 0/6] Reproducible build luca.boccassi
                     ` (2 preceding siblings ...)
  2017-09-05 20:51   ` [PATCH v7 3/6] mk: sort headers before wildcard inclusion luca.boccassi
@ 2017-09-05 20:51   ` luca.boccassi
  2017-09-05 20:51   ` [PATCH v7 5/6] mk: sort object files when building deps lists luca.boccassi
                     ` (4 subsequent siblings)
  8 siblings, 0 replies; 23+ messages in thread
From: luca.boccassi @ 2017-09-05 20:51 UTC (permalink / raw)
  To: dev; +Cc: ferruh.yigit, Luca Boccassi

From: Luca Boccassi <luca.boccassi@gmail.com>

In order to achieve reproducible builds, always use the same
order when listing files for compilation.

Signed-off-by: Luca Boccassi <luca.boccassi@gmail.com>
---
 drivers/net/cxgbe/Makefile    | 2 +-
 drivers/net/e1000/Makefile    | 2 +-
 drivers/net/fm10k/Makefile    | 2 +-
 drivers/net/i40e/Makefile     | 2 +-
 drivers/net/ixgbe/Makefile    | 2 +-
 drivers/net/qede/Makefile     | 2 +-
 drivers/net/sfc/Makefile      | 2 +-
 drivers/net/thunderx/Makefile | 2 +-
 8 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/drivers/net/cxgbe/Makefile b/drivers/net/cxgbe/Makefile
index 7cef6279c..b4666b5af 100644
--- a/drivers/net/cxgbe/Makefile
+++ b/drivers/net/cxgbe/Makefile
@@ -67,7 +67,7 @@ endif
 # Add extra flags for base driver files (also known as shared code)
 # to disable warnings in them
 #
-BASE_DRIVER_OBJS=$(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c)))
+BASE_DRIVER_OBJS=$(sort $(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c))))
 $(foreach obj, $(BASE_DRIVER_OBJS), $(eval CFLAGS_$(obj)+=$(CFLAGS_BASE_DRIVER)))
 
 VPATH += $(SRCDIR)/base
diff --git a/drivers/net/e1000/Makefile b/drivers/net/e1000/Makefile
index ffdf36d37..59d96bca1 100644
--- a/drivers/net/e1000/Makefile
+++ b/drivers/net/e1000/Makefile
@@ -68,7 +68,7 @@ endif
 # Add extra flags for base driver files (also known as shared code)
 # to disable warnings in them
 #
-BASE_DRIVER_OBJS=$(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c)))
+BASE_DRIVER_OBJS=$(sort $(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c))))
 $(foreach obj, $(BASE_DRIVER_OBJS), $(eval CFLAGS_$(obj)+=$(CFLAGS_BASE_DRIVER)))
 
 VPATH += $(SRCDIR)/base
diff --git a/drivers/net/fm10k/Makefile b/drivers/net/fm10k/Makefile
index e0024f052..0bc124eb1 100644
--- a/drivers/net/fm10k/Makefile
+++ b/drivers/net/fm10k/Makefile
@@ -80,7 +80,7 @@ endif
 #
 # Add extra flags for base driver source files to disable warnings in them
 #
-BASE_DRIVER_OBJS=$(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c)))
+BASE_DRIVER_OBJS=$(sort $(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c))))
 $(foreach obj, $(BASE_DRIVER_OBJS), $(eval CFLAGS_$(obj)+=$(CFLAGS_BASE_DRIVER)))
 
 VPATH += $(SRCDIR)/base
diff --git a/drivers/net/i40e/Makefile b/drivers/net/i40e/Makefile
index 55c79a60a..8040a890e 100644
--- a/drivers/net/i40e/Makefile
+++ b/drivers/net/i40e/Makefile
@@ -78,7 +78,7 @@ endif
 
 CFLAGS_i40e_lan_hmc.o += -Wno-error
 endif
-OBJS_BASE_DRIVER=$(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c)))
+OBJS_BASE_DRIVER=$(sort $(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c))))
 $(foreach obj, $(OBJS_BASE_DRIVER), $(eval CFLAGS_$(obj)+=$(CFLAGS_BASE_DRIVER)))
 
 VPATH += $(SRCDIR)/base
diff --git a/drivers/net/ixgbe/Makefile b/drivers/net/ixgbe/Makefile
index 5e57cb353..d47ce24aa 100644
--- a/drivers/net/ixgbe/Makefile
+++ b/drivers/net/ixgbe/Makefile
@@ -87,7 +87,7 @@ endif
 # Add extra flags for base driver files (also known as shared code)
 # to disable warnings in them
 #
-BASE_DRIVER_OBJS=$(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c)))
+BASE_DRIVER_OBJS=$(sort $(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c))))
 $(foreach obj, $(BASE_DRIVER_OBJS), $(eval CFLAGS_$(obj)+=$(CFLAGS_BASE_DRIVER)))
 
 VPATH += $(SRCDIR)/base
diff --git a/drivers/net/qede/Makefile b/drivers/net/qede/Makefile
index f03441d9a..83ff95474 100644
--- a/drivers/net/qede/Makefile
+++ b/drivers/net/qede/Makefile
@@ -78,7 +78,7 @@ endif
 # to disable warnings in them
 #
 #
-BASE_DRIVER_OBJS=$(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c)))
+BASE_DRIVER_OBJS=$(sort $(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c))))
 $(foreach obj, $(BASE_DRIVER_OBJS), $(eval CFLAGS_$(obj)+=$(CFLAGS_BASE_DRIVER)))
 
 VPATH += $(SRCDIR)/base
diff --git a/drivers/net/sfc/Makefile b/drivers/net/sfc/Makefile
index 57aa963ba..8cfd14d45 100644
--- a/drivers/net/sfc/Makefile
+++ b/drivers/net/sfc/Makefile
@@ -71,7 +71,7 @@ endif
 # List of base driver object files for which
 # special CFLAGS above should be applied
 #
-BASE_DRIVER_OBJS=$(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c)))
+BASE_DRIVER_OBJS=$(sort $(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c))))
 $(foreach obj, $(BASE_DRIVER_OBJS), \
   $(eval CFLAGS_$(obj)+=$(CFLAGS_BASE_DRIVER)))
 
diff --git a/drivers/net/thunderx/Makefile b/drivers/net/thunderx/Makefile
index 915ae945a..a2b6caae2 100644
--- a/drivers/net/thunderx/Makefile
+++ b/drivers/net/thunderx/Makefile
@@ -45,7 +45,7 @@ EXPORT_MAP := rte_pmd_thunderx_nicvf_version.map
 
 LIBABIVER := 1
 
-OBJS_BASE_DRIVER=$(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c)))
+OBJS_BASE_DRIVER=$(sort $(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c))))
 $(foreach obj, $(OBJS_BASE_DRIVER), $(eval CFLAGS_$(obj)+=$(CFLAGS_BASE_DRIVER)))
 
 VPATH += $(SRCDIR)/base
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v7 5/6] mk: sort object files when building deps lists
  2017-09-05 20:51 ` [PATCH v7 0/6] Reproducible build luca.boccassi
                     ` (3 preceding siblings ...)
  2017-09-05 20:51   ` [PATCH v7 4/6] mk: sort source files before passing them to the compiler luca.boccassi
@ 2017-09-05 20:51   ` luca.boccassi
  2017-09-05 20:51   ` [PATCH v7 6/6] mk: install symlinks before build step luca.boccassi
                     ` (3 subsequent siblings)
  8 siblings, 0 replies; 23+ messages in thread
From: luca.boccassi @ 2017-09-05 20:51 UTC (permalink / raw)
  To: dev; +Cc: ferruh.yigit, Luca Boccassi

From: Luca Boccassi <luca.boccassi@gmail.com>

In order to achieve reproducible builds, always use the same
order when listing object files to build dependencies lists.

Signed-off-by: Luca Boccassi <luca.boccassi@gmail.com>
---
 mk/rte.app.mk     | 4 ++--
 mk/rte.hostapp.mk | 4 ++--
 mk/rte.shared.mk  | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index c25fdd9f5..c5b8c1ead 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -270,8 +270,8 @@ LDLIBS_NAMES += $(patsubst -Wl$(comma)-l%,lib%.a,$(filter -Wl$(comma)-l%,$(LDLIB
 
 # list of found libraries files (useful for deps). If not found, the
 # library is silently ignored and dep won't be checked
-LDLIBS_FILES := $(wildcard $(foreach dir,$(LDLIBS_PATH),\
-	$(addprefix $(dir)/,$(LDLIBS_NAMES))))
+LDLIBS_FILES := $(sort $(wildcard $(foreach dir,$(LDLIBS_PATH),\
+	$(addprefix $(dir)/,$(LDLIBS_NAMES)))))
 
 #
 # Compile executable file if needed
diff --git a/mk/rte.hostapp.mk b/mk/rte.hostapp.mk
index 5cb4909cb..f58173c31 100644
--- a/mk/rte.hostapp.mk
+++ b/mk/rte.hostapp.mk
@@ -69,9 +69,9 @@ O_TO_EXE_DO = @set -e; \
 -include .$(HOSTAPP).cmd
 
 # list of .a files that are linked to this application
-LDLIBS_FILES := $(wildcard \
+LDLIBS_FILES := $(sort $(wildcard \
 	$(addprefix $(RTE_OUTPUT)/lib/, \
-	$(patsubst -l%,lib%.a,$(filter -l%,$(LDLIBS)))))
+	$(patsubst -l%,lib%.a,$(filter -l%,$(LDLIBS))))))
 
 #
 # Compile executable file if needed
diff --git a/mk/rte.shared.mk b/mk/rte.shared.mk
index 87ccf0ba4..4e680bc03 100644
--- a/mk/rte.shared.mk
+++ b/mk/rte.shared.mk
@@ -85,8 +85,8 @@ LDLIBS_NAMES += $(patsubst -Wl$(comma)-l%,lib%.a,$(filter -Wl$(comma)-l%,$(LDLIB
 
 # list of found libraries files (useful for deps). If not found, the
 # library is silently ignored and dep won't be checked
-LDLIBS_FILES := $(wildcard $(foreach dir,$(LDLIBS_PATH),\
-	$(addprefix $(dir)/,$(LDLIBS_NAMES))))
+LDLIBS_FILES := $(sort $(wildcard $(foreach dir,$(LDLIBS_PATH),\
+	$(addprefix $(dir)/,$(LDLIBS_NAMES)))))
 
 #
 # Archive objects in .so file if needed
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v7 6/6] mk: install symlinks before build step
  2017-09-05 20:51 ` [PATCH v7 0/6] Reproducible build luca.boccassi
                     ` (4 preceding siblings ...)
  2017-09-05 20:51   ` [PATCH v7 5/6] mk: sort object files when building deps lists luca.boccassi
@ 2017-09-05 20:51   ` luca.boccassi
  2017-10-12  2:58     ` Thomas Monjalon
  2017-09-05 20:56   ` [PATCH v7 0/6] Reproducible build Luca Boccassi
                     ` (2 subsequent siblings)
  8 siblings, 1 reply; 23+ messages in thread
From: luca.boccassi @ 2017-09-05 20:51 UTC (permalink / raw)
  To: dev; +Cc: ferruh.yigit, Luca Boccassi

From: Luca Boccassi <luca.boccassi@gmail.com>

A race condition can happen during parallel builds, where a header
might be installed in RTE_OUT/include before CFLAGS is recursively
expanded. This causes GCC to sometimes pick the header path as
SRCDIR/... and sometimes as RTE_OUT/include/... making the build
unreproducible, as the full path is used for the expansion of
__FILE__ and in the DWARF directory listing.

Installing all symlinks before all builds solves the problem. It is
still suboptimal, as the (fixed) path recorded in the DWARF dir
listing will include the user-configurable build output directory,
and thus will result in a different binary between different users
despite all other conditions being equal, but it is a simpler
approach that will anyway be obsolete once the build system is
switched to Meson.

Suggested-by: Ferruh Yigit <ferruh.yigit@intel.com>
Signed-off-by: Luca Boccassi <luca.boccassi@gmail.com>
---
 mk/rte.lib.mk | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
index 13115d146..643da47da 100644
--- a/mk/rte.lib.mk
+++ b/mk/rte.lib.mk
@@ -59,14 +59,19 @@ endif
 
 
 _BUILD = $(LIB)
-_INSTALL = $(INSTALL-FILES-y) $(SYMLINK-FILES-y) $(RTE_OUTPUT)/lib/$(LIB)
+PREINSTALL = $(SYMLINK-FILES-y)
+_INSTALL = $(INSTALL-FILES-y) $(RTE_OUTPUT)/lib/$(LIB)
 _CLEAN = doclean
 
 .PHONY: all
 all: install
 
 .PHONY: install
+ifeq ($(SYMLINK-FILES-y),)
 install: build _postinstall
+else
+install: _preinstall build _postinstall
+endif
 
 _postinstall: build
 
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* Re: [PATCH v7 0/6] Reproducible build
  2017-09-05 20:51 ` [PATCH v7 0/6] Reproducible build luca.boccassi
                     ` (5 preceding siblings ...)
  2017-09-05 20:51   ` [PATCH v7 6/6] mk: install symlinks before build step luca.boccassi
@ 2017-09-05 20:56   ` Luca Boccassi
  2017-09-05 21:00   ` Stephen Hemminger
  2017-10-12 13:15   ` [PATCH v8 " luca.boccassi
  8 siblings, 0 replies; 23+ messages in thread
From: Luca Boccassi @ 2017-09-05 20:56 UTC (permalink / raw)
  To: dev

On Tue, 2017-09-05 at 21:51 +0100, luca.boccassi@gmail.com wrote:
> From: Luca Boccassi <luca.boccassi@gmail.com>
> 
> In the past couple of years a concerted effort among almost all Linux
> distros has been striving toward achieving reproducible builds. [1]
> This involves changes to the toolchain, new tools and CI systems. [2]
> 
> v1 fixed the documentation, examples and linker script generation.
> v2 fixes all problems, which were caused by unstable order of headers
> inclusion, source files listing and object file listing when passing
> them to the compiler.
> DPDK's build, at least with the default configuration, is fully
> reproducible with this patch series as tested by the Reproducible
> Builds developers experimental toolchain. [3]
> 
> v3 restores the first patch, which was eaten by git send-email.
> 
> v4 drops the patch that reorders rebuilds, and adds a patch to make
> the inclusion of headers deterministic with regards to GCC embedding
> the full file path when expading __FILE__ and when writing the
> directory listing in the DWARF objects.
> It also drops the first 2 patches which have already been merged.
> 
> v5 adds the -I$(SRCDIR) workaround to librte_eal linuxapp's and
> librte_gro's Makefiles.
> 
> v6 fixes copypasta added in v5 - librte-gro's Makefile should have
> used
> $(SRCDIR) rather than $(SRCDIR)/include.
> 
> v7 dropped the big patch to change the CFLAGS in all libraries
> Makefiles
> and instead uses a much simpler approach suggested by Ferruh
> (thanks!),
> which consists in installing the library public headers symlinks
> before
> building. Although the drawback is that the DWARF directory listing
> will always include the user configured build output directory, it is
> a much smaller patch that will be rendered obsolete by the move to
> Meson
> anyway.
> 
> [1] https://reproducible-builds.org/
> [2] https://reproducible-builds.org/tools/
> [3] https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain#
> Us
> 
> Luca Boccassi (6):
>   mk: sort list of shared objects in linker script
>   mk: sort list of files in examples.dox
>   mk: sort headers before wildcard inclusion
>   mk: sort source files before passing them to the compiler
>   mk: sort object files when building deps lists
>   mk: install symlinks before build step
> 
>  drivers/net/cxgbe/Makefile                                 | 2 +-
>  drivers/net/e1000/Makefile                                 | 2 +-
>  drivers/net/fm10k/Makefile                                 | 2 +-
>  drivers/net/i40e/Makefile                                  | 2 +-
>  drivers/net/ixgbe/Makefile                                 | 2 +-
>  drivers/net/qede/Makefile                                  | 2 +-
>  drivers/net/sfc/Makefile                                   | 2 +-
>  drivers/net/thunderx/Makefile                              | 2 +-
>  examples/ip_pipeline/Makefile                              | 2 +-
>  examples/multi_process/client_server_mp/mp_server/Makefile | 2 +-
>  examples/server_node_efd/server/Makefile                   | 2 +-
>  lib/librte_eal/common/Makefile                             | 2 +-
>  mk/rte.app.mk                                              | 4 ++--
>  mk/rte.combinedlib.mk                                      | 2 +-
>  mk/rte.hostapp.mk                                          | 4 ++--
>  mk/rte.lib.mk                                              | 7
> ++++++-
>  mk/rte.sdkdoc.mk                                           | 2 +-
>  mk/rte.shared.mk                                           | 4 ++--
>  18 files changed, 26 insertions(+), 21 deletions(-)

Sigh, made a copypasta in the reply-to (lol).

Original thread:

http://dpdk.org/ml/archives/dev/2017-August/073000.html

-- 
Kind regards,
Luca Boccassi

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH v7 0/6] Reproducible build
  2017-09-05 20:51 ` [PATCH v7 0/6] Reproducible build luca.boccassi
                     ` (6 preceding siblings ...)
  2017-09-05 20:56   ` [PATCH v7 0/6] Reproducible build Luca Boccassi
@ 2017-09-05 21:00   ` Stephen Hemminger
  2017-09-05 21:15     ` Luca Boccassi
  2017-10-12 13:15   ` [PATCH v8 " luca.boccassi
  8 siblings, 1 reply; 23+ messages in thread
From: Stephen Hemminger @ 2017-09-05 21:00 UTC (permalink / raw)
  To: luca.boccassi; +Cc: dev, ferruh.yigit

On Tue,  5 Sep 2017 21:51:53 +0100
luca.boccassi@gmail.com wrote:

> From: Luca Boccassi <luca.boccassi@gmail.com>
> 
> In the past couple of years a concerted effort among almost all Linux
> distros has been striving toward achieving reproducible builds. [1]
> This involves changes to the toolchain, new tools and CI systems. [2]
> 
> v1 fixed the documentation, examples and linker script generation.
> v2 fixes all problems, which were caused by unstable order of headers
> inclusion, source files listing and object file listing when passing
> them to the compiler.
> DPDK's build, at least with the default configuration, is fully
> reproducible with this patch series as tested by the Reproducible
> Builds developers experimental toolchain. [3]
> 
> v3 restores the first patch, which was eaten by git send-email.
> 
> v4 drops the patch that reorders rebuilds, and adds a patch to make
> the inclusion of headers deterministic with regards to GCC embedding
> the full file path when expading __FILE__ and when writing the
> directory listing in the DWARF objects.
> It also drops the first 2 patches which have already been merged.
> 
> v5 adds the -I$(SRCDIR) workaround to librte_eal linuxapp's and
> librte_gro's Makefiles.
> 
> v6 fixes copypasta added in v5 - librte-gro's Makefile should have used
> $(SRCDIR) rather than $(SRCDIR)/include.
> 
> v7 dropped the big patch to change the CFLAGS in all libraries Makefiles
> and instead uses a much simpler approach suggested by Ferruh (thanks!),
> which consists in installing the library public headers symlinks before
> building. Although the drawback is that the DWARF directory listing
> will always include the user configured build output directory, it is
> a much smaller patch that will be rendered obsolete by the move to Meson
> anyway.
> 
> [1] https://reproducible-builds.org/
> [2] https://reproducible-builds.org/tools/
> [3] https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain#Us
> 
> Luca Boccassi (6):
>   mk: sort list of shared objects in linker script
>   mk: sort list of files in examples.dox
>   mk: sort headers before wildcard inclusion
>   mk: sort source files before passing them to the compiler
>   mk: sort object files when building deps lists
>   mk: install symlinks before build step
> 
>  drivers/net/cxgbe/Makefile                                 | 2 +-
>  drivers/net/e1000/Makefile                                 | 2 +-
>  drivers/net/fm10k/Makefile                                 | 2 +-
>  drivers/net/i40e/Makefile                                  | 2 +-
>  drivers/net/ixgbe/Makefile                                 | 2 +-
>  drivers/net/qede/Makefile                                  | 2 +-
>  drivers/net/sfc/Makefile                                   | 2 +-
>  drivers/net/thunderx/Makefile                              | 2 +-
>  examples/ip_pipeline/Makefile                              | 2 +-
>  examples/multi_process/client_server_mp/mp_server/Makefile | 2 +-
>  examples/server_node_efd/server/Makefile                   | 2 +-
>  lib/librte_eal/common/Makefile                             | 2 +-
>  mk/rte.app.mk                                              | 4 ++--
>  mk/rte.combinedlib.mk                                      | 2 +-
>  mk/rte.hostapp.mk                                          | 4 ++--
>  mk/rte.lib.mk                                              | 7 ++++++-
>  mk/rte.sdkdoc.mk                                           | 2 +-
>  mk/rte.shared.mk                                           | 4 ++--
>  18 files changed, 26 insertions(+), 21 deletions(-)
> 

How does this interact with propose new build process.
Hate to have do everything twice.

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH v7 0/6] Reproducible build
  2017-09-05 21:00   ` Stephen Hemminger
@ 2017-09-05 21:15     ` Luca Boccassi
  2017-09-06  8:46       ` Bruce Richardson
  0 siblings, 1 reply; 23+ messages in thread
From: Luca Boccassi @ 2017-09-05 21:15 UTC (permalink / raw)
  To: Stephen Hemminger; +Cc: dev, ferruh.yigit

On Tue, 2017-09-05 at 14:00 -0700, Stephen Hemminger wrote:
> On Tue,  5 Sep 2017 21:51:53 +0100
> luca.boccassi@gmail.com wrote:
> 
> > From: Luca Boccassi <luca.boccassi@gmail.com>
> > 
> > In the past couple of years a concerted effort among almost all
> > Linux
> > distros has been striving toward achieving reproducible builds. [1]
> > This involves changes to the toolchain, new tools and CI systems.
> > [2]
> > 
> > v1 fixed the documentation, examples and linker script generation.
> > v2 fixes all problems, which were caused by unstable order of
> > headers
> > inclusion, source files listing and object file listing when
> > passing
> > them to the compiler.
> > DPDK's build, at least with the default configuration, is fully
> > reproducible with this patch series as tested by the Reproducible
> > Builds developers experimental toolchain. [3]
> > 
> > v3 restores the first patch, which was eaten by git send-email.
> > 
> > v4 drops the patch that reorders rebuilds, and adds a patch to make
> > the inclusion of headers deterministic with regards to GCC
> > embedding
> > the full file path when expading __FILE__ and when writing the
> > directory listing in the DWARF objects.
> > It also drops the first 2 patches which have already been merged.
> > 
> > v5 adds the -I$(SRCDIR) workaround to librte_eal linuxapp's and
> > librte_gro's Makefiles.
> > 
> > v6 fixes copypasta added in v5 - librte-gro's Makefile should have
> > used
> > $(SRCDIR) rather than $(SRCDIR)/include.
> > 
> > v7 dropped the big patch to change the CFLAGS in all libraries
> > Makefiles
> > and instead uses a much simpler approach suggested by Ferruh
> > (thanks!),
> > which consists in installing the library public headers symlinks
> > before
> > building. Although the drawback is that the DWARF directory listing
> > will always include the user configured build output directory, it
> > is
> > a much smaller patch that will be rendered obsolete by the move to
> > Meson
> > anyway.
> > 
> > [1] https://reproducible-builds.org/
> > [2] https://reproducible-builds.org/tools/
> > [3] https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchai
> > n#Us
> > 
> > Luca Boccassi (6):
> >   mk: sort list of shared objects in linker script
> >   mk: sort list of files in examples.dox
> >   mk: sort headers before wildcard inclusion
> >   mk: sort source files before passing them to the compiler
> >   mk: sort object files when building deps lists
> >   mk: install symlinks before build step
> > 
> >  drivers/net/cxgbe/Makefile                                 | 2 +-
> >  drivers/net/e1000/Makefile                                 | 2 +-
> >  drivers/net/fm10k/Makefile                                 | 2 +-
> >  drivers/net/i40e/Makefile                                  | 2 +-
> >  drivers/net/ixgbe/Makefile                                 | 2 +-
> >  drivers/net/qede/Makefile                                  | 2 +-
> >  drivers/net/sfc/Makefile                                   | 2 +-
> >  drivers/net/thunderx/Makefile                              | 2 +-
> >  examples/ip_pipeline/Makefile                              | 2 +-
> >  examples/multi_process/client_server_mp/mp_server/Makefile | 2 +-
> >  examples/server_node_efd/server/Makefile                   | 2 +-
> >  lib/librte_eal/common/Makefile                             | 2 +-
> >  mk/rte.app.mk                                              | 4 ++-
> > -
> >  mk/rte.combinedlib.mk                                      | 2 +-
> >  mk/rte.hostapp.mk                                          | 4 ++-
> > -
> >  mk/rte.lib.mk                                              | 7
> > ++++++-
> >  mk/rte.sdkdoc.mk                                           | 2 +-
> >  mk/rte.shared.mk                                           | 4 ++-
> > -
> >  18 files changed, 26 insertions(+), 21 deletions(-)
> > 
> 
> How does this interact with propose new build process.
> Hate to have do everything twice.

Most of these patches, mainly for the custom bits that discover the
source/headers files on the system (all the wildcards, etc) will not be
necessary any longer.

The documentation build is not there yet in the patchset, so I'll check
once that's available.

But I'm still testing the Meson patches, I had issues integrating it in
the package builds as Meson support is still quite new in debhelper, I
hope to finish that up next week and give feedback to Bruce.

The reproducible build testing infra I've got is picky and eats only
deb packages so until I get that up and running I cannot test the
reproducibility using Meson unfortunately.

-- 
Kind regards,
Luca Boccassi

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH v7 0/6] Reproducible build
  2017-09-05 21:15     ` Luca Boccassi
@ 2017-09-06  8:46       ` Bruce Richardson
  2017-09-06  9:10         ` Luca Boccassi
  0 siblings, 1 reply; 23+ messages in thread
From: Bruce Richardson @ 2017-09-06  8:46 UTC (permalink / raw)
  To: Luca Boccassi; +Cc: Stephen Hemminger, dev, ferruh.yigit

On Tue, Sep 05, 2017 at 10:15:29PM +0100, Luca Boccassi wrote:
> On Tue, 2017-09-05 at 14:00 -0700, Stephen Hemminger wrote:
> > On Tue,  5 Sep 2017 21:51:53 +0100
> > luca.boccassi@gmail.com wrote:
> > 
> > > From: Luca Boccassi <luca.boccassi@gmail.com>
> > > 
> > > In the past couple of years a concerted effort among almost all
> > > Linux
> > > distros has been striving toward achieving reproducible builds. [1]
> > > This involves changes to the toolchain, new tools and CI systems.
> > > [2]
> > > 
> > > v1 fixed the documentation, examples and linker script generation.
> > > v2 fixes all problems, which were caused by unstable order of
> > > headers
> > > inclusion, source files listing and object file listing when
> > > passing
> > > them to the compiler.
> > > DPDK's build, at least with the default configuration, is fully
> > > reproducible with this patch series as tested by the Reproducible
> > > Builds developers experimental toolchain. [3]
> > > 
> > > v3 restores the first patch, which was eaten by git send-email.
> > > 
> > > v4 drops the patch that reorders rebuilds, and adds a patch to make
> > > the inclusion of headers deterministic with regards to GCC
> > > embedding
> > > the full file path when expading __FILE__ and when writing the
> > > directory listing in the DWARF objects.
> > > It also drops the first 2 patches which have already been merged.
> > > 
> > > v5 adds the -I$(SRCDIR) workaround to librte_eal linuxapp's and
> > > librte_gro's Makefiles.
> > > 
> > > v6 fixes copypasta added in v5 - librte-gro's Makefile should have
> > > used
> > > $(SRCDIR) rather than $(SRCDIR)/include.
> > > 
> > > v7 dropped the big patch to change the CFLAGS in all libraries
> > > Makefiles
> > > and instead uses a much simpler approach suggested by Ferruh
> > > (thanks!),
> > > which consists in installing the library public headers symlinks
> > > before
> > > building. Although the drawback is that the DWARF directory listing
> > > will always include the user configured build output directory, it
> > > is
> > > a much smaller patch that will be rendered obsolete by the move to
> > > Meson
> > > anyway.
> > > 
> > > [1] https://reproducible-builds.org/
> > > [2] https://reproducible-builds.org/tools/
> > > [3] https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchai
> > > n#Us
> > > 
> > > Luca Boccassi (6):
> > >   mk: sort list of shared objects in linker script
> > >   mk: sort list of files in examples.dox
> > >   mk: sort headers before wildcard inclusion
> > >   mk: sort source files before passing them to the compiler
> > >   mk: sort object files when building deps lists
> > >   mk: install symlinks before build step
> > > 
> > >  drivers/net/cxgbe/Makefile                                 | 2 +-
> > >  drivers/net/e1000/Makefile                                 | 2 +-
> > >  drivers/net/fm10k/Makefile                                 | 2 +-
> > >  drivers/net/i40e/Makefile                                  | 2 +-
> > >  drivers/net/ixgbe/Makefile                                 | 2 +-
> > >  drivers/net/qede/Makefile                                  | 2 +-
> > >  drivers/net/sfc/Makefile                                   | 2 +-
> > >  drivers/net/thunderx/Makefile                              | 2 +-
> > >  examples/ip_pipeline/Makefile                              | 2 +-
> > >  examples/multi_process/client_server_mp/mp_server/Makefile | 2 +-
> > >  examples/server_node_efd/server/Makefile                   | 2 +-
> > >  lib/librte_eal/common/Makefile                             | 2 +-
> > >  mk/rte.app.mk                                              | 4 ++-
> > > -
> > >  mk/rte.combinedlib.mk                                      | 2 +-
> > >  mk/rte.hostapp.mk                                          | 4 ++-
> > > -
> > >  mk/rte.lib.mk                                              | 7
> > > ++++++-
> > >  mk/rte.sdkdoc.mk                                           | 2 +-
> > >  mk/rte.shared.mk                                           | 4 ++-
> > > -
> > >  18 files changed, 26 insertions(+), 21 deletions(-)
> > > 
> > 
> > How does this interact with propose new build process.
> > Hate to have do everything twice.
> 
> Most of these patches, mainly for the custom bits that discover the
> source/headers files on the system (all the wildcards, etc) will not be
> necessary any longer.
> 
> The documentation build is not there yet in the patchset, so I'll check
> once that's available.
> 
> But I'm still testing the Meson patches, I had issues integrating it in
> the package builds as Meson support is still quite new in debhelper, I
> hope to finish that up next week and give feedback to Bruce.
> 
> The reproducible build testing infra I've got is picky and eats only
> deb packages so until I get that up and running I cannot test the
> reproducibility using Meson unfortunately.
> 
> -- 
It's also safer to assume that any replacement build system is going to
take a while longer to complete, so it's as well to consider these
changes for integration, since they are not that large of patches.

Also, looking forward to your feedback on the meson build patches. I was
going to do a v2 with minor updates, but I'll hold off until next week
when I get your comments.

/Bruce

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH v7 0/6] Reproducible build
  2017-09-06  8:46       ` Bruce Richardson
@ 2017-09-06  9:10         ` Luca Boccassi
  2017-09-06 10:45           ` Bruce Richardson
  0 siblings, 1 reply; 23+ messages in thread
From: Luca Boccassi @ 2017-09-06  9:10 UTC (permalink / raw)
  To: Bruce Richardson; +Cc: Stephen Hemminger, dev, ferruh.yigit

On Wed, 2017-09-06 at 09:46 +0100, Bruce Richardson wrote:
> On Tue, Sep 05, 2017 at 10:15:29PM +0100, Luca Boccassi wrote:
> > On Tue, 2017-09-05 at 14:00 -0700, Stephen Hemminger wrote:
> > > On Tue,  5 Sep 2017 21:51:53 +0100
> > > luca.boccassi@gmail.com wrote:
> > > 
> > > > From: Luca Boccassi <luca.boccassi@gmail.com>
> > > > 
> > > > In the past couple of years a concerted effort among almost all
> > > > Linux
> > > > distros has been striving toward achieving reproducible builds.
> > > > [1]
> > > > This involves changes to the toolchain, new tools and CI
> > > > systems.
> > > > [2]
> > > > 
> > > > v1 fixed the documentation, examples and linker script
> > > > generation.
> > > > v2 fixes all problems, which were caused by unstable order of
> > > > headers
> > > > inclusion, source files listing and object file listing when
> > > > passing
> > > > them to the compiler.
> > > > DPDK's build, at least with the default configuration, is fully
> > > > reproducible with this patch series as tested by the
> > > > Reproducible
> > > > Builds developers experimental toolchain. [3]
> > > > 
> > > > v3 restores the first patch, which was eaten by git send-email.
> > > > 
> > > > v4 drops the patch that reorders rebuilds, and adds a patch to
> > > > make
> > > > the inclusion of headers deterministic with regards to GCC
> > > > embedding
> > > > the full file path when expading __FILE__ and when writing the
> > > > directory listing in the DWARF objects.
> > > > It also drops the first 2 patches which have already been
> > > > merged.
> > > > 
> > > > v5 adds the -I$(SRCDIR) workaround to librte_eal linuxapp's and
> > > > librte_gro's Makefiles.
> > > > 
> > > > v6 fixes copypasta added in v5 - librte-gro's Makefile should
> > > > have
> > > > used
> > > > $(SRCDIR) rather than $(SRCDIR)/include.
> > > > 
> > > > v7 dropped the big patch to change the CFLAGS in all libraries
> > > > Makefiles
> > > > and instead uses a much simpler approach suggested by Ferruh
> > > > (thanks!),
> > > > which consists in installing the library public headers
> > > > symlinks
> > > > before
> > > > building. Although the drawback is that the DWARF directory
> > > > listing
> > > > will always include the user configured build output directory,
> > > > it
> > > > is
> > > > a much smaller patch that will be rendered obsolete by the move
> > > > to
> > > > Meson
> > > > anyway.
> > > > 
> > > > [1] https://reproducible-builds.org/
> > > > [2] https://reproducible-builds.org/tools/
> > > > [3] https://wiki.debian.org/ReproducibleBuilds/ExperimentalTool
> > > > chai
> > > > n#Us
> > > > 
> > > > Luca Boccassi (6):
> > > >   mk: sort list of shared objects in linker script
> > > >   mk: sort list of files in examples.dox
> > > >   mk: sort headers before wildcard inclusion
> > > >   mk: sort source files before passing them to the compiler
> > > >   mk: sort object files when building deps lists
> > > >   mk: install symlinks before build step
> > > > 
> > > >  drivers/net/cxgbe/Makefile                                 | 2
> > > > +-
> > > >  drivers/net/e1000/Makefile                                 | 2
> > > > +-
> > > >  drivers/net/fm10k/Makefile                                 | 2
> > > > +-
> > > >  drivers/net/i40e/Makefile                                  | 2
> > > > +-
> > > >  drivers/net/ixgbe/Makefile                                 | 2
> > > > +-
> > > >  drivers/net/qede/Makefile                                  | 2
> > > > +-
> > > >  drivers/net/sfc/Makefile                                   | 2
> > > > +-
> > > >  drivers/net/thunderx/Makefile                              | 2
> > > > +-
> > > >  examples/ip_pipeline/Makefile                              | 2
> > > > +-
> > > >  examples/multi_process/client_server_mp/mp_server/Makefile | 2
> > > > +-
> > > >  examples/server_node_efd/server/Makefile                   | 2
> > > > +-
> > > >  lib/librte_eal/common/Makefile                             | 2
> > > > +-
> > > >  mk/rte.app.mk                                              | 4
> > > > ++-
> > > > -
> > > >  mk/rte.combinedlib.mk                                      | 2
> > > > +-
> > > >  mk/rte.hostapp.mk                                          | 4
> > > > ++-
> > > > -
> > > >  mk/rte.lib.mk                                              | 7
> > > > ++++++-
> > > >  mk/rte.sdkdoc.mk                                           | 2
> > > > +-
> > > >  mk/rte.shared.mk                                           | 4
> > > > ++-
> > > > -
> > > >  18 files changed, 26 insertions(+), 21 deletions(-)
> > > > 
> > > 
> > > How does this interact with propose new build process.
> > > Hate to have do everything twice.
> > 
> > Most of these patches, mainly for the custom bits that discover the
> > source/headers files on the system (all the wildcards, etc) will
> > not be
> > necessary any longer.
> > 
> > The documentation build is not there yet in the patchset, so I'll
> > check
> > once that's available.
> > 
> > But I'm still testing the Meson patches, I had issues integrating
> > it in
> > the package builds as Meson support is still quite new in
> > debhelper, I
> > hope to finish that up next week and give feedback to Bruce.
> > 
> > The reproducible build testing infra I've got is picky and eats
> > only
> > deb packages so until I get that up and running I cannot test the
> > reproducibility using Meson unfortunately.
> > 
> > -- 
> 
> It's also safer to assume that any replacement build system is going
> to
> take a while longer to complete, so it's as well to consider these
> changes for integration, since they are not that large of patches.

I agree :-)

> Also, looking forward to your feedback on the meson build patches. I
> was
> going to do a v2 with minor updates, but I'll hold off until next
> week
> when I get your comments.
> 
> /Bruce

Ok, but please bear in mind that I'll be travelling from today to
Monday, so it will probably be end of the week. So if needed feel free
to send v2 already. Sorry for the delay!

-- 
Kind regards,
Luca Boccassi

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH v7 0/6] Reproducible build
  2017-09-06  9:10         ` Luca Boccassi
@ 2017-09-06 10:45           ` Bruce Richardson
  0 siblings, 0 replies; 23+ messages in thread
From: Bruce Richardson @ 2017-09-06 10:45 UTC (permalink / raw)
  To: Luca Boccassi; +Cc: Stephen Hemminger, dev, ferruh.yigit

On Wed, Sep 06, 2017 at 10:10:34AM +0100, Luca Boccassi wrote:
> On Wed, 2017-09-06 at 09:46 +0100, Bruce Richardson wrote:
> > On Tue, Sep 05, 2017 at 10:15:29PM +0100, Luca Boccassi wrote:
> > > On Tue, 2017-09-05 at 14:00 -0700, Stephen Hemminger wrote:
> > > > On Tue,  5 Sep 2017 21:51:53 +0100
> > > > luca.boccassi@gmail.com wrote:
> > > > 
> > > > > From: Luca Boccassi <luca.boccassi@gmail.com>
> > > > > 
> > > > > In the past couple of years a concerted effort among almost all
> > > > > Linux
> > > > > distros has been striving toward achieving reproducible builds.
> > > > > [1]
> > > > > This involves changes to the toolchain, new tools and CI
> > > > > systems.
> > > > > [2]
> > > > > 
> > > > > v1 fixed the documentation, examples and linker script
> > > > > generation.
> > > > > v2 fixes all problems, which were caused by unstable order of
> > > > > headers
> > > > > inclusion, source files listing and object file listing when
> > > > > passing
> > > > > them to the compiler.
> > > > > DPDK's build, at least with the default configuration, is fully
> > > > > reproducible with this patch series as tested by the
> > > > > Reproducible
> > > > > Builds developers experimental toolchain. [3]
> > > > > 
> > > > > v3 restores the first patch, which was eaten by git send-email.
> > > > > 
> > > > > v4 drops the patch that reorders rebuilds, and adds a patch to
> > > > > make
> > > > > the inclusion of headers deterministic with regards to GCC
> > > > > embedding
> > > > > the full file path when expading __FILE__ and when writing the
> > > > > directory listing in the DWARF objects.
> > > > > It also drops the first 2 patches which have already been
> > > > > merged.
> > > > > 
> > > > > v5 adds the -I$(SRCDIR) workaround to librte_eal linuxapp's and
> > > > > librte_gro's Makefiles.
> > > > > 
> > > > > v6 fixes copypasta added in v5 - librte-gro's Makefile should
> > > > > have
> > > > > used
> > > > > $(SRCDIR) rather than $(SRCDIR)/include.
> > > > > 
> > > > > v7 dropped the big patch to change the CFLAGS in all libraries
> > > > > Makefiles
> > > > > and instead uses a much simpler approach suggested by Ferruh
> > > > > (thanks!),
> > > > > which consists in installing the library public headers
> > > > > symlinks
> > > > > before
> > > > > building. Although the drawback is that the DWARF directory
> > > > > listing
> > > > > will always include the user configured build output directory,
> > > > > it
> > > > > is
> > > > > a much smaller patch that will be rendered obsolete by the move
> > > > > to
> > > > > Meson
> > > > > anyway.
> > > > > 
> > > > > [1] https://reproducible-builds.org/
> > > > > [2] https://reproducible-builds.org/tools/
> > > > > [3] https://wiki.debian.org/ReproducibleBuilds/ExperimentalTool
> > > > > chai
> > > > > n#Us
> > > > > 
> > > > > Luca Boccassi (6):
> > > > >   mk: sort list of shared objects in linker script
> > > > >   mk: sort list of files in examples.dox
> > > > >   mk: sort headers before wildcard inclusion
> > > > >   mk: sort source files before passing them to the compiler
> > > > >   mk: sort object files when building deps lists
> > > > >   mk: install symlinks before build step
> > > > > 
> > > > >  drivers/net/cxgbe/Makefile                                 | 2
> > > > > +-
> > > > >  drivers/net/e1000/Makefile                                 | 2
> > > > > +-
> > > > >  drivers/net/fm10k/Makefile                                 | 2
> > > > > +-
> > > > >  drivers/net/i40e/Makefile                                  | 2
> > > > > +-
> > > > >  drivers/net/ixgbe/Makefile                                 | 2
> > > > > +-
> > > > >  drivers/net/qede/Makefile                                  | 2
> > > > > +-
> > > > >  drivers/net/sfc/Makefile                                   | 2
> > > > > +-
> > > > >  drivers/net/thunderx/Makefile                              | 2
> > > > > +-
> > > > >  examples/ip_pipeline/Makefile                              | 2
> > > > > +-
> > > > >  examples/multi_process/client_server_mp/mp_server/Makefile | 2
> > > > > +-
> > > > >  examples/server_node_efd/server/Makefile                   | 2
> > > > > +-
> > > > >  lib/librte_eal/common/Makefile                             | 2
> > > > > +-
> > > > >  mk/rte.app.mk                                              | 4
> > > > > ++-
> > > > > -
> > > > >  mk/rte.combinedlib.mk                                      | 2
> > > > > +-
> > > > >  mk/rte.hostapp.mk                                          | 4
> > > > > ++-
> > > > > -
> > > > >  mk/rte.lib.mk                                              | 7
> > > > > ++++++-
> > > > >  mk/rte.sdkdoc.mk                                           | 2
> > > > > +-
> > > > >  mk/rte.shared.mk                                           | 4
> > > > > ++-
> > > > > -
> > > > >  18 files changed, 26 insertions(+), 21 deletions(-)
> > > > > 
> > > > 
> > > > How does this interact with propose new build process.
> > > > Hate to have do everything twice.
> > > 
> > > Most of these patches, mainly for the custom bits that discover the
> > > source/headers files on the system (all the wildcards, etc) will
> > > not be
> > > necessary any longer.
> > > 
> > > The documentation build is not there yet in the patchset, so I'll
> > > check
> > > once that's available.
> > > 
> > > But I'm still testing the Meson patches, I had issues integrating
> > > it in
> > > the package builds as Meson support is still quite new in
> > > debhelper, I
> > > hope to finish that up next week and give feedback to Bruce.
> > > 
> > > The reproducible build testing infra I've got is picky and eats
> > > only
> > > deb packages so until I get that up and running I cannot test the
> > > reproducibility using Meson unfortunately.
> > > 
> > > -- 
> > 
> > It's also safer to assume that any replacement build system is going
> > to
> > take a while longer to complete, so it's as well to consider these
> > changes for integration, since they are not that large of patches.
> 
> I agree :-)
> 
> > Also, looking forward to your feedback on the meson build patches. I
> > was
> > going to do a v2 with minor updates, but I'll hold off until next
> > week
> > when I get your comments.
> > 
> > /Bruce
> 
> Ok, but please bear in mind that I'll be travelling from today to
> Monday, so it will probably be end of the week. So if needed feel free
> to send v2 already. Sorry for the delay!
> 
Ok, thanks for the info. I'll see how it goes and how many other changes
I might have to make. The only major issue right now is the one that you
already flagged about removing the check for a Kconfig file in the kernel.

/Bruce

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH v7 6/6] mk: install symlinks before build step
  2017-09-05 20:51   ` [PATCH v7 6/6] mk: install symlinks before build step luca.boccassi
@ 2017-10-12  2:58     ` Thomas Monjalon
  2017-10-12 12:19       ` Luca Boccassi
  0 siblings, 1 reply; 23+ messages in thread
From: Thomas Monjalon @ 2017-10-12  2:58 UTC (permalink / raw)
  To: luca.boccassi; +Cc: dev, ferruh.yigit

05/09/2017 22:51, luca.boccassi@gmail.com:
>  .PHONY: install
> +ifeq ($(SYMLINK-FILES-y),)
>  install: build _postinstall
> +else
> +install: _preinstall build _postinstall
> +endif

I think you cannot be sure that "_preinstall" will be finished
before the "build" target starts in parallel make.
Am I missing something?

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH v7 6/6] mk: install symlinks before build step
  2017-10-12  2:58     ` Thomas Monjalon
@ 2017-10-12 12:19       ` Luca Boccassi
  0 siblings, 0 replies; 23+ messages in thread
From: Luca Boccassi @ 2017-10-12 12:19 UTC (permalink / raw)
  To: Thomas Monjalon; +Cc: dev, ferruh.yigit

On Thu, 2017-10-12 at 04:58 +0200, Thomas Monjalon wrote:
> 05/09/2017 22:51, luca.boccassi@gmail.com:
> >  .PHONY: install
> > +ifeq ($(SYMLINK-FILES-y),)
> >  install: build _postinstall
> > +else
> > +install: _preinstall build _postinstall
> > +endif
> 
> I think you cannot be sure that "_preinstall" will be finished
> before the "build" target starts in parallel make.
> Am I missing something?

Yes, you are right, good catch. I'll add a dependency:

build: _preinstall

Which should take care of it, retest and send a new version.

-- 
Kind regards,
Luca Boccassi

^ permalink raw reply	[flat|nested] 23+ messages in thread
* [PATCH v8 0/6] Reproducible build
  2017-09-05 20:51 ` [PATCH v7 0/6] Reproducible build luca.boccassi
                     ` (7 preceding siblings ...)
  2017-09-05 21:00   ` Stephen Hemminger
@ 2017-10-12 13:15   ` luca.boccassi
  2017-10-12 13:15     ` [PATCH v8 1/6] mk: sort list of shared objects in linker script luca.boccassi
                       ` (6 more replies)
  8 siblings, 7 replies; 23+ messages in thread
From: luca.boccassi @ 2017-10-12 13:15 UTC (permalink / raw)
  To: dev; +Cc: ferruh.yigit, thomas

From: Luca Boccassi <bluca@debian.org>

In the past couple of years a concerted effort among almost all Linux
distros has been striving toward achieving reproducible builds. [1]
This involves changes to the toolchain, new tools and CI systems. [2]

v1 fixed the documentation, examples and linker script generation.
v2 fixes all problems, which were caused by unstable order of headers
inclusion, source files listing and object file listing when passing
them to the compiler.
DPDK's build, at least with the default configuration, is fully
reproducible with this patch series as tested by the Reproducible
Builds developers experimental toolchain. [3]

v3 restores the first patch, which was eaten by git send-email.

v4 drops the patch that reorders rebuilds, and adds a patch to make
the inclusion of headers deterministic with regards to GCC embedding
the full file path when expading __FILE__ and when writing the
directory listing in the DWARF objects.
It also drops the first 2 patches which have already been merged.

v5 adds the -I$(SRCDIR) workaround to librte_eal linuxapp's and
librte_gro's Makefiles.

v6 fixes copypasta added in v5 - librte-gro's Makefile should have used
$(SRCDIR) rather than $(SRCDIR)/include.

v7 dropped the big patch to change the CFLAGS in all libraries Makefiles
and instead uses a much simpler approach suggested by Ferruh (thanks!),
which consists in installing the library public headers symlinks before
building. Although the drawback is that the DWARF directory listing
will always include the user configured build output directory, it is
a much smaller patch that will be rendered obsolete by the move to Meson
anyway.

v8 fixes a missing dependency problem in rte.lib.mk from the build target
to the _preinstall target, which could have caused race conditions in
parallel builds. Spotted by Thomas. Thanks!

Luca Boccassi (6):
  mk: sort list of shared objects in linker script
  mk: sort list of files in examples.dox
  mk: sort headers before wildcard inclusion
  mk: sort source files before passing them to the compiler
  mk: sort object files when building deps lists
  mk: install symlinks before build step

 drivers/net/cxgbe/Makefile                                 | 2 +-
 drivers/net/e1000/Makefile                                 | 2 +-
 drivers/net/fm10k/Makefile                                 | 2 +-
 drivers/net/i40e/Makefile                                  | 2 +-
 drivers/net/ixgbe/Makefile                                 | 2 +-
 drivers/net/qede/Makefile                                  | 2 +-
 drivers/net/sfc/Makefile                                   | 2 +-
 drivers/net/thunderx/Makefile                              | 2 +-
 examples/ip_pipeline/Makefile                              | 2 +-
 examples/multi_process/client_server_mp/mp_server/Makefile | 2 +-
 examples/server_node_efd/server/Makefile                   | 2 +-
 lib/librte_eal/common/Makefile                             | 2 +-
 mk/rte.app.mk                                              | 4 ++--
 mk/rte.combinedlib.mk                                      | 2 +-
 mk/rte.hostapp.mk                                          | 4 ++--
 mk/rte.lib.mk                                              | 8 +++++++-
 mk/rte.sdkdoc.mk                                           | 2 +-
 mk/rte.shared.mk                                           | 4 ++--
 18 files changed, 27 insertions(+), 21 deletions(-)

-- 
2.11.0

^ permalink raw reply	[flat|nested] 23+ messages in thread
* [PATCH v8 1/6] mk: sort list of shared objects in linker script
  2017-10-12 13:15   ` [PATCH v8 " luca.boccassi
@ 2017-10-12 13:15     ` luca.boccassi
  2017-10-12 13:15     ` [PATCH v8 2/6] mk: sort list of files in examples.dox luca.boccassi
                       ` (5 subsequent siblings)
  6 siblings, 0 replies; 23+ messages in thread
From: luca.boccassi @ 2017-10-12 13:15 UTC (permalink / raw)
  To: dev; +Cc: ferruh.yigit, thomas, Luca Boccassi

From: Luca Boccassi <luca.boccassi@gmail.com>

The output of wildcard might not be stable and depend on the
filesystem and other factors.
This means the content libdpdk.so linker script might change between
builds from the same sources.
Run the list through sort to ensure reproducibility.

Signed-off-by: Luca Boccassi <luca.boccassi@gmail.com>
---
 mk/rte.combinedlib.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mk/rte.combinedlib.mk b/mk/rte.combinedlib.mk
index 449358b33..2ab7ee8a1 100644
--- a/mk/rte.combinedlib.mk
+++ b/mk/rte.combinedlib.mk
@@ -42,7 +42,7 @@ endif
 RTE_LIBNAME := dpdk
 COMBINEDLIB := lib$(RTE_LIBNAME)$(EXT)
 
-LIBS := $(filter-out $(COMBINEDLIB), $(notdir $(wildcard $(RTE_OUTPUT)/lib/*$(EXT))))
+LIBS := $(filter-out $(COMBINEDLIB), $(sort $(notdir $(wildcard $(RTE_OUTPUT)/lib/*$(EXT)))))
 
 all: FORCE
 	$(Q)echo "GROUP ( $(LIBS) )" > $(RTE_OUTPUT)/lib/$(COMBINEDLIB)
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v8 2/6] mk: sort list of files in examples.dox
  2017-10-12 13:15   ` [PATCH v8 " luca.boccassi
  2017-10-12 13:15     ` [PATCH v8 1/6] mk: sort list of shared objects in linker script luca.boccassi
@ 2017-10-12 13:15     ` luca.boccassi
  2017-10-12 13:15     ` [PATCH v8 3/6] mk: sort headers before wildcard inclusion luca.boccassi
                       ` (4 subsequent siblings)
  6 siblings, 0 replies; 23+ messages in thread
From: luca.boccassi @ 2017-10-12 13:15 UTC (permalink / raw)
  To: dev; +Cc: ferruh.yigit, thomas, Luca Boccassi

From: Luca Boccassi <luca.boccassi@gmail.com>

The result of find might not be stable depending on external
conditions.
Pipe it through LC_ALL=C sort to ensure reproducible results when
generating examples.dox.

Signed-off-by: Luca Boccassi <luca.boccassi@gmail.com>
---
 mk/rte.sdkdoc.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mk/rte.sdkdoc.mk b/mk/rte.sdkdoc.mk
index c0eaa3502..de31b78cf 100644
--- a/mk/rte.sdkdoc.mk
+++ b/mk/rte.sdkdoc.mk
@@ -93,7 +93,7 @@ $(API_EXAMPLES): api-html-clean
 	$(Q)mkdir -p $(@D)
 	@printf '/**\n' > $(API_EXAMPLES)
 	@printf '@page examples DPDK Example Programs\n\n' >> $(API_EXAMPLES)
-	@find examples -type f -name '*.c' -printf '@example %p\n' >> $(API_EXAMPLES)
+	@find examples -type f -name '*.c' -printf '@example %p\n' | LC_ALL=C sort >> $(API_EXAMPLES)
 	@printf '*/\n' >> $(API_EXAMPLES)
 
 guides-pdf-clean: guides-pdf-img-clean
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v8 3/6] mk: sort headers before wildcard inclusion
  2017-10-12 13:15   ` [PATCH v8 " luca.boccassi
  2017-10-12 13:15     ` [PATCH v8 1/6] mk: sort list of shared objects in linker script luca.boccassi
  2017-10-12 13:15     ` [PATCH v8 2/6] mk: sort list of files in examples.dox luca.boccassi
@ 2017-10-12 13:15     ` luca.boccassi
  2017-10-12 13:15     ` [PATCH v8 4/6] mk: sort source files before passing them to the compiler luca.boccassi
                       ` (3 subsequent siblings)
  6 siblings, 0 replies; 23+ messages in thread
From: luca.boccassi @ 2017-10-12 13:15 UTC (permalink / raw)
  To: dev; +Cc: ferruh.yigit, thomas, Luca Boccassi

From: Luca Boccassi <luca.boccassi@gmail.com>

In order to achieve fully reproducible builds, always use the same
inclusion order for headers in the Makefiles.

Signed-off-by: Luca Boccassi <luca.boccassi@gmail.com>
---
 examples/ip_pipeline/Makefile                              | 2 +-
 examples/multi_process/client_server_mp/mp_server/Makefile | 2 +-
 examples/server_node_efd/server/Makefile                   | 2 +-
 lib/librte_eal/common/Makefile                             | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/examples/ip_pipeline/Makefile b/examples/ip_pipeline/Makefile
index dc7e0ddd7..12ce0a1d5 100644
--- a/examples/ip_pipeline/Makefile
+++ b/examples/ip_pipeline/Makefile
@@ -43,7 +43,7 @@ APP = ip_pipeline
 
 VPATH += $(SRCDIR)/pipeline
 
-INC += $(wildcard *.h) $(wildcard pipeline/*.h)
+INC += $(sort $(wildcard *.h)) $(sort $(wildcard pipeline/*.h))
 
 # all source are stored in SRCS-y
 SRCS-$(CONFIG_RTE_LIBRTE_PIPELINE) := main.c
diff --git a/examples/multi_process/client_server_mp/mp_server/Makefile b/examples/multi_process/client_server_mp/mp_server/Makefile
index 5552999b5..160c17b68 100644
--- a/examples/multi_process/client_server_mp/mp_server/Makefile
+++ b/examples/multi_process/client_server_mp/mp_server/Makefile
@@ -49,7 +49,7 @@ APP = mp_server
 # all source are stored in SRCS-y
 SRCS-y := main.c init.c args.c
 
-INC := $(wildcard *.h)
+INC := $(sort $(wildcard *.h))
 
 CFLAGS += $(WERROR_FLAGS) -O3
 CFLAGS += -I$(SRCDIR)/../shared
diff --git a/examples/server_node_efd/server/Makefile b/examples/server_node_efd/server/Makefile
index a2f2f361b..9f1fe2894 100644
--- a/examples/server_node_efd/server/Makefile
+++ b/examples/server_node_efd/server/Makefile
@@ -49,7 +49,7 @@ APP = server
 # all source are stored in SRCS-y
 SRCS-y := main.c init.c args.c
 
-INC := $(wildcard *.h)
+INC := $(sort $(wildcard *.h))
 
 CFLAGS += $(WERROR_FLAGS) -O3
 CFLAGS += -I$(SRCDIR)/../shared
diff --git a/lib/librte_eal/common/Makefile b/lib/librte_eal/common/Makefile
index e8fd67a27..4e6baaa72 100644
--- a/lib/librte_eal/common/Makefile
+++ b/lib/librte_eal/common/Makefile
@@ -49,7 +49,7 @@ GENERIC_INC += rte_vect.h rte_pause.h rte_io.h
 
 # defined in mk/arch/$(RTE_ARCH)/rte.vars.mk
 ARCH_DIR ?= $(RTE_ARCH)
-ARCH_INC := $(notdir $(wildcard $(RTE_SDK)/lib/librte_eal/common/include/arch/$(ARCH_DIR)/*.h))
+ARCH_INC := $(sort $(notdir $(wildcard $(RTE_SDK)/lib/librte_eal/common/include/arch/$(ARCH_DIR)/*.h)))
 
 SYMLINK-$(CONFIG_RTE_LIBRTE_EAL)-include := $(addprefix include/,$(INC))
 SYMLINK-$(CONFIG_RTE_LIBRTE_EAL)-include += \
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v8 4/6] mk: sort source files before passing them to the compiler
  2017-10-12 13:15   ` [PATCH v8 " luca.boccassi
                       ` (2 preceding siblings ...)
  2017-10-12 13:15     ` [PATCH v8 3/6] mk: sort headers before wildcard inclusion luca.boccassi
@ 2017-10-12 13:15     ` luca.boccassi
  2017-10-12 13:15     ` [PATCH v8 5/6] mk: sort object files when building deps lists luca.boccassi
                       ` (2 subsequent siblings)
  6 siblings, 0 replies; 23+ messages in thread
From: luca.boccassi @ 2017-10-12 13:15 UTC (permalink / raw)
  To: dev; +Cc: ferruh.yigit, thomas, Luca Boccassi

From: Luca Boccassi <luca.boccassi@gmail.com>

In order to achieve reproducible builds, always use the same
order when listing files for compilation.

Signed-off-by: Luca Boccassi <luca.boccassi@gmail.com>
---
 drivers/net/cxgbe/Makefile    | 2 +-
 drivers/net/e1000/Makefile    | 2 +-
 drivers/net/fm10k/Makefile    | 2 +-
 drivers/net/i40e/Makefile     | 2 +-
 drivers/net/ixgbe/Makefile    | 2 +-
 drivers/net/qede/Makefile     | 2 +-
 drivers/net/sfc/Makefile      | 2 +-
 drivers/net/thunderx/Makefile | 2 +-
 8 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/drivers/net/cxgbe/Makefile b/drivers/net/cxgbe/Makefile
index 7cef6279c..b4666b5af 100644
--- a/drivers/net/cxgbe/Makefile
+++ b/drivers/net/cxgbe/Makefile
@@ -67,7 +67,7 @@ endif
 # Add extra flags for base driver files (also known as shared code)
 # to disable warnings in them
 #
-BASE_DRIVER_OBJS=$(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c)))
+BASE_DRIVER_OBJS=$(sort $(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c))))
 $(foreach obj, $(BASE_DRIVER_OBJS), $(eval CFLAGS_$(obj)+=$(CFLAGS_BASE_DRIVER)))
 
 VPATH += $(SRCDIR)/base
diff --git a/drivers/net/e1000/Makefile b/drivers/net/e1000/Makefile
index ffdf36d37..59d96bca1 100644
--- a/drivers/net/e1000/Makefile
+++ b/drivers/net/e1000/Makefile
@@ -68,7 +68,7 @@ endif
 # Add extra flags for base driver files (also known as shared code)
 # to disable warnings in them
 #
-BASE_DRIVER_OBJS=$(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c)))
+BASE_DRIVER_OBJS=$(sort $(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c))))
 $(foreach obj, $(BASE_DRIVER_OBJS), $(eval CFLAGS_$(obj)+=$(CFLAGS_BASE_DRIVER)))
 
 VPATH += $(SRCDIR)/base
diff --git a/drivers/net/fm10k/Makefile b/drivers/net/fm10k/Makefile
index e0024f052..0bc124eb1 100644
--- a/drivers/net/fm10k/Makefile
+++ b/drivers/net/fm10k/Makefile
@@ -80,7 +80,7 @@ endif
 #
 # Add extra flags for base driver source files to disable warnings in them
 #
-BASE_DRIVER_OBJS=$(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c)))
+BASE_DRIVER_OBJS=$(sort $(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c))))
 $(foreach obj, $(BASE_DRIVER_OBJS), $(eval CFLAGS_$(obj)+=$(CFLAGS_BASE_DRIVER)))
 
 VPATH += $(SRCDIR)/base
diff --git a/drivers/net/i40e/Makefile b/drivers/net/i40e/Makefile
index 1290d7f32..ddd1b1253 100644
--- a/drivers/net/i40e/Makefile
+++ b/drivers/net/i40e/Makefile
@@ -78,7 +78,7 @@ endif
 
 CFLAGS_i40e_lan_hmc.o += -Wno-error
 endif
-OBJS_BASE_DRIVER=$(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c)))
+OBJS_BASE_DRIVER=$(sort $(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c))))
 $(foreach obj, $(OBJS_BASE_DRIVER), $(eval CFLAGS_$(obj)+=$(CFLAGS_BASE_DRIVER)))
 
 VPATH += $(SRCDIR)/base
diff --git a/drivers/net/ixgbe/Makefile b/drivers/net/ixgbe/Makefile
index 18ad4feca..95c806d26 100644
--- a/drivers/net/ixgbe/Makefile
+++ b/drivers/net/ixgbe/Makefile
@@ -87,7 +87,7 @@ endif
 # Add extra flags for base driver files (also known as shared code)
 # to disable warnings in them
 #
-BASE_DRIVER_OBJS=$(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c)))
+BASE_DRIVER_OBJS=$(sort $(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c))))
 $(foreach obj, $(BASE_DRIVER_OBJS), $(eval CFLAGS_$(obj)+=$(CFLAGS_BASE_DRIVER)))
 
 VPATH += $(SRCDIR)/base
diff --git a/drivers/net/qede/Makefile b/drivers/net/qede/Makefile
index f03441d9a..83ff95474 100644
--- a/drivers/net/qede/Makefile
+++ b/drivers/net/qede/Makefile
@@ -78,7 +78,7 @@ endif
 # to disable warnings in them
 #
 #
-BASE_DRIVER_OBJS=$(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c)))
+BASE_DRIVER_OBJS=$(sort $(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c))))
 $(foreach obj, $(BASE_DRIVER_OBJS), $(eval CFLAGS_$(obj)+=$(CFLAGS_BASE_DRIVER)))
 
 VPATH += $(SRCDIR)/base
diff --git a/drivers/net/sfc/Makefile b/drivers/net/sfc/Makefile
index 0adb786b8..429eb4352 100644
--- a/drivers/net/sfc/Makefile
+++ b/drivers/net/sfc/Makefile
@@ -74,7 +74,7 @@ endif
 # List of base driver object files for which
 # special CFLAGS above should be applied
 #
-BASE_DRIVER_OBJS=$(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c)))
+BASE_DRIVER_OBJS=$(sort $(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c))))
 $(foreach obj, $(BASE_DRIVER_OBJS), \
   $(eval CFLAGS_$(obj)+=$(CFLAGS_BASE_DRIVER)))
 
diff --git a/drivers/net/thunderx/Makefile b/drivers/net/thunderx/Makefile
index 915ae945a..a2b6caae2 100644
--- a/drivers/net/thunderx/Makefile
+++ b/drivers/net/thunderx/Makefile
@@ -45,7 +45,7 @@ EXPORT_MAP := rte_pmd_thunderx_nicvf_version.map
 
 LIBABIVER := 1
 
-OBJS_BASE_DRIVER=$(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c)))
+OBJS_BASE_DRIVER=$(sort $(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c))))
 $(foreach obj, $(OBJS_BASE_DRIVER), $(eval CFLAGS_$(obj)+=$(CFLAGS_BASE_DRIVER)))
 
 VPATH += $(SRCDIR)/base
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v8 5/6] mk: sort object files when building deps lists
  2017-10-12 13:15   ` [PATCH v8 " luca.boccassi
                       ` (3 preceding siblings ...)
  2017-10-12 13:15     ` [PATCH v8 4/6] mk: sort source files before passing them to the compiler luca.boccassi
@ 2017-10-12 13:15     ` luca.boccassi
  2017-10-12 13:15     ` [PATCH v8 6/6] mk: install symlinks before build step luca.boccassi
  2017-10-12 20:31     ` [PATCH v8 0/6] Reproducible build Thomas Monjalon
  6 siblings, 0 replies; 23+ messages in thread
From: luca.boccassi @ 2017-10-12 13:15 UTC (permalink / raw)
  To: dev; +Cc: ferruh.yigit, thomas, Luca Boccassi

From: Luca Boccassi <luca.boccassi@gmail.com>

In order to achieve reproducible builds, always use the same
order when listing object files to build dependencies lists.

Signed-off-by: Luca Boccassi <luca.boccassi@gmail.com>
---
 mk/rte.app.mk     | 4 ++--
 mk/rte.hostapp.mk | 4 ++--
 mk/rte.shared.mk  | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index 0b8f6126b..0038a8725 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -283,8 +283,8 @@ LDLIBS_NAMES += $(patsubst -Wl$(comma)-l%,lib%.a,$(filter -Wl$(comma)-l%,$(LDLIB
 
 # list of found libraries files (useful for deps). If not found, the
 # library is silently ignored and dep won't be checked
-LDLIBS_FILES := $(wildcard $(foreach dir,$(LDLIBS_PATH),\
-	$(addprefix $(dir)/,$(LDLIBS_NAMES))))
+LDLIBS_FILES := $(sort $(wildcard $(foreach dir,$(LDLIBS_PATH),\
+	$(addprefix $(dir)/,$(LDLIBS_NAMES)))))
 
 #
 # Compile executable file if needed
diff --git a/mk/rte.hostapp.mk b/mk/rte.hostapp.mk
index 5cb4909cb..f58173c31 100644
--- a/mk/rte.hostapp.mk
+++ b/mk/rte.hostapp.mk
@@ -69,9 +69,9 @@ O_TO_EXE_DO = @set -e; \
 -include .$(HOSTAPP).cmd
 
 # list of .a files that are linked to this application
-LDLIBS_FILES := $(wildcard \
+LDLIBS_FILES := $(sort $(wildcard \
 	$(addprefix $(RTE_OUTPUT)/lib/, \
-	$(patsubst -l%,lib%.a,$(filter -l%,$(LDLIBS)))))
+	$(patsubst -l%,lib%.a,$(filter -l%,$(LDLIBS))))))
 
 #
 # Compile executable file if needed
diff --git a/mk/rte.shared.mk b/mk/rte.shared.mk
index 87ccf0ba4..4e680bc03 100644
--- a/mk/rte.shared.mk
+++ b/mk/rte.shared.mk
@@ -85,8 +85,8 @@ LDLIBS_NAMES += $(patsubst -Wl$(comma)-l%,lib%.a,$(filter -Wl$(comma)-l%,$(LDLIB
 
 # list of found libraries files (useful for deps). If not found, the
 # library is silently ignored and dep won't be checked
-LDLIBS_FILES := $(wildcard $(foreach dir,$(LDLIBS_PATH),\
-	$(addprefix $(dir)/,$(LDLIBS_NAMES))))
+LDLIBS_FILES := $(sort $(wildcard $(foreach dir,$(LDLIBS_PATH),\
+	$(addprefix $(dir)/,$(LDLIBS_NAMES)))))
 
 #
 # Archive objects in .so file if needed
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH v8 6/6] mk: install symlinks before build step
  2017-10-12 13:15   ` [PATCH v8 " luca.boccassi
                       ` (4 preceding siblings ...)
  2017-10-12 13:15     ` [PATCH v8 5/6] mk: sort object files when building deps lists luca.boccassi
@ 2017-10-12 13:15     ` luca.boccassi
  2017-10-12 20:31     ` [PATCH v8 0/6] Reproducible build Thomas Monjalon
  6 siblings, 0 replies; 23+ messages in thread
From: luca.boccassi @ 2017-10-12 13:15 UTC (permalink / raw)
  To: dev; +Cc: ferruh.yigit, thomas, Luca Boccassi

From: Luca Boccassi <luca.boccassi@gmail.com>

A race condition can happen during parallel builds, where a header
might be installed in RTE_OUT/include before CFLAGS is recursively
expanded. This causes GCC to sometimes pick the header path as
SRCDIR/... and sometimes as RTE_OUT/include/... making the build
unreproducible, as the full path is used for the expansion of
__FILE__ and in the DWARF directory listing.

Installing all symlinks before all builds solves the problem. It is
still suboptimal, as the (fixed) path recorded in the DWARF dir
listing will include the user-configurable build output directory,
and thus will result in a different binary between different users
despite all other conditions being equal, but it is a simpler
approach that will anyway be obsolete once the build system is
switched to Meson.

Suggested-by: Ferruh Yigit <ferruh.yigit@intel.com>
Signed-off-by: Luca Boccassi <luca.boccassi@gmail.com>
---
 mk/rte.lib.mk | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
index 13115d146..4767ce4f2 100644
--- a/mk/rte.lib.mk
+++ b/mk/rte.lib.mk
@@ -59,14 +59,20 @@ endif
 
 
 _BUILD = $(LIB)
-_INSTALL = $(INSTALL-FILES-y) $(SYMLINK-FILES-y) $(RTE_OUTPUT)/lib/$(LIB)
+PREINSTALL = $(SYMLINK-FILES-y)
+_INSTALL = $(INSTALL-FILES-y) $(RTE_OUTPUT)/lib/$(LIB)
 _CLEAN = doclean
 
 .PHONY: all
 all: install
 
 .PHONY: install
+ifeq ($(SYMLINK-FILES-y),)
 install: build _postinstall
+else
+install: _preinstall build _postinstall
+build: _preinstall
+endif
 
 _postinstall: build
 
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* Re: [PATCH v8 0/6] Reproducible build
  2017-10-12 13:15   ` [PATCH v8 " luca.boccassi
                       ` (5 preceding siblings ...)
  2017-10-12 13:15     ` [PATCH v8 6/6] mk: install symlinks before build step luca.boccassi
@ 2017-10-12 20:31     ` Thomas Monjalon
  6 siblings, 0 replies; 23+ messages in thread
From: Thomas Monjalon @ 2017-10-12 20:31 UTC (permalink / raw)
  To: luca.boccassi; +Cc: dev, ferruh.yigit

> Luca Boccassi (6):
>   mk: sort list of shared objects in linker script
>   mk: sort list of files in examples.dox
>   mk: sort headers before wildcard inclusion
>   mk: sort source files before passing them to the compiler
>   mk: sort object files when building deps lists
>   mk: install symlinks before build step

Applied, thanks

^ permalink raw reply	[flat|nested] 23+ messages in thread
end of thread, other threads:[~2017-10-12 20:31 UTC | newest]

Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <drivers/net/vmxnet3/vmxnet3_rxtx.c.rej>
2017-09-05 20:51 ` [PATCH v7 0/6] Reproducible build luca.boccassi
2017-09-05 20:51   ` [PATCH v7 1/6] mk: sort list of shared objects in linker script luca.boccassi
2017-09-05 20:51   ` [PATCH v7 2/6] mk: sort list of files in examples.dox luca.boccassi
2017-09-05 20:51   ` [PATCH v7 3/6] mk: sort headers before wildcard inclusion luca.boccassi
2017-09-05 20:51   ` [PATCH v7 4/6] mk: sort source files before passing them to the compiler luca.boccassi
2017-09-05 20:51   ` [PATCH v7 5/6] mk: sort object files when building deps lists luca.boccassi
2017-09-05 20:51   ` [PATCH v7 6/6] mk: install symlinks before build step luca.boccassi
2017-10-12  2:58     ` Thomas Monjalon
2017-10-12 12:19       ` Luca Boccassi
2017-09-05 20:56   ` [PATCH v7 0/6] Reproducible build Luca Boccassi
2017-09-05 21:00   ` Stephen Hemminger
2017-09-05 21:15     ` Luca Boccassi
2017-09-06  8:46       ` Bruce Richardson
2017-09-06  9:10         ` Luca Boccassi
2017-09-06 10:45           ` Bruce Richardson
2017-10-12 13:15   ` [PATCH v8 " luca.boccassi
2017-10-12 13:15     ` [PATCH v8 1/6] mk: sort list of shared objects in linker script luca.boccassi
2017-10-12 13:15     ` [PATCH v8 2/6] mk: sort list of files in examples.dox luca.boccassi
2017-10-12 13:15     ` [PATCH v8 3/6] mk: sort headers before wildcard inclusion luca.boccassi
2017-10-12 13:15     ` [PATCH v8 4/6] mk: sort source files before passing them to the compiler luca.boccassi
2017-10-12 13:15     ` [PATCH v8 5/6] mk: sort object files when building deps lists luca.boccassi
2017-10-12 13:15     ` [PATCH v8 6/6] mk: install symlinks before build step luca.boccassi
2017-10-12 20:31     ` [PATCH v8 0/6] Reproducible build Thomas Monjalon

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.