bpf.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH bpf] selftests/bpf: make directory prerequisites order-only
@ 2019-07-12 13:56 Ilya Leoshkevich
  2019-07-12 19:57 ` Andrii Nakryiko
  2019-07-15 22:19 ` Daniel Borkmann
  0 siblings, 2 replies; 6+ messages in thread
From: Ilya Leoshkevich @ 2019-07-12 13:56 UTC (permalink / raw)
  To: bpf, netdev; +Cc: gor, heiko.carstens, Ilya Leoshkevich

When directories are used as prerequisites in Makefiles, they can cause
a lot of unnecessary rebuilds, because a directory is considered changed
whenever a file in this directory is added, removed or modified.

If the only thing a target is interested in is the existence of the
directory it depends on, which is the case for selftests/bpf, this
directory should be specified as an order-only prerequisite: it would
still be created in case it does not exist, but it would not trigger a
rebuild of a target in case it's considered changed.

Signed-off-by: Ilya Leoshkevich <iii@linux.ibm.com>
---
 tools/testing/selftests/bpf/Makefile | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 277d8605e340..0e003fb6641b 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -183,12 +183,12 @@ TEST_CUSTOM_PROGS += $(ALU32_BUILD_DIR)/test_progs_32
 $(ALU32_BUILD_DIR):
 	mkdir -p $@
 
-$(ALU32_BUILD_DIR)/urandom_read: $(OUTPUT)/urandom_read
+$(ALU32_BUILD_DIR)/urandom_read: $(OUTPUT)/urandom_read | $(ALU32_BUILD_DIR)
 	cp $< $@
 
 $(ALU32_BUILD_DIR)/test_progs_32: test_progs.c $(OUTPUT)/libbpf.a\
-						$(ALU32_BUILD_DIR) \
-						$(ALU32_BUILD_DIR)/urandom_read
+						$(ALU32_BUILD_DIR)/urandom_read \
+						| $(ALU32_BUILD_DIR)
 	$(CC) $(TEST_PROGS_CFLAGS) $(CFLAGS) \
 		-o $(ALU32_BUILD_DIR)/test_progs_32 \
 		test_progs.c test_stub.c trace_helpers.c prog_tests/*.c \
@@ -197,8 +197,8 @@ $(ALU32_BUILD_DIR)/test_progs_32: test_progs.c $(OUTPUT)/libbpf.a\
 $(ALU32_BUILD_DIR)/test_progs_32: $(PROG_TESTS_H)
 $(ALU32_BUILD_DIR)/test_progs_32: prog_tests/*.c
 
-$(ALU32_BUILD_DIR)/%.o: progs/%.c $(ALU32_BUILD_DIR) \
-					$(ALU32_BUILD_DIR)/test_progs_32
+$(ALU32_BUILD_DIR)/%.o: progs/%.c $(ALU32_BUILD_DIR)/test_progs_32 \
+					| $(ALU32_BUILD_DIR)
 	($(CLANG) $(CLANG_FLAGS) -O2 -target bpf -emit-llvm -c $< -o - || \
 		echo "clang failed") | \
 	$(LLC) -march=bpf -mattr=+alu32 -mcpu=$(CPU) $(LLC_FLAGS) \
@@ -236,7 +236,7 @@ $(PROG_TESTS_DIR):
 	mkdir -p $@
 
 PROG_TESTS_FILES := $(wildcard prog_tests/*.c)
-$(PROG_TESTS_H): $(PROG_TESTS_DIR) $(PROG_TESTS_FILES)
+$(PROG_TESTS_H): $(PROG_TESTS_FILES) | $(PROG_TESTS_DIR)
 	$(shell ( cd prog_tests/; \
 		  echo '/* Generated header, do not edit */'; \
 		  echo '#ifdef DECLARE'; \
@@ -257,7 +257,7 @@ MAP_TESTS_H := $(MAP_TESTS_DIR)/tests.h
 test_maps.c: $(MAP_TESTS_H)
 $(OUTPUT)/test_maps: CFLAGS += $(TEST_MAPS_CFLAGS)
 MAP_TESTS_FILES := $(wildcard map_tests/*.c)
-$(MAP_TESTS_H): $(MAP_TESTS_DIR) $(MAP_TESTS_FILES)
+$(MAP_TESTS_H): $(MAP_TESTS_FILES) | $(MAP_TESTS_DIR)
 	$(shell ( cd map_tests/; \
 		  echo '/* Generated header, do not edit */'; \
 		  echo '#ifdef DECLARE'; \
@@ -279,7 +279,7 @@ $(VERIFIER_TESTS_DIR):
 	mkdir -p $@
 
 VERIFIER_TEST_FILES := $(wildcard verifier/*.c)
-$(OUTPUT)/verifier/tests.h: $(VERIFIER_TESTS_DIR) $(VERIFIER_TEST_FILES)
+$(OUTPUT)/verifier/tests.h: $(VERIFIER_TEST_FILES) | $(VERIFIER_TESTS_DIR)
 	$(shell ( cd verifier/; \
 		  echo '/* Generated header, do not edit */'; \
 		  echo '#ifdef FILL_ARRAY'; \
-- 
2.21.0


^ permalink raw reply related	[flat|nested] 6+ messages in thread

* Re: [PATCH bpf] selftests/bpf: make directory prerequisites order-only
  2019-07-12 13:56 [PATCH bpf] selftests/bpf: make directory prerequisites order-only Ilya Leoshkevich
@ 2019-07-12 19:57 ` Andrii Nakryiko
  2019-07-15 22:19 ` Daniel Borkmann
  1 sibling, 0 replies; 6+ messages in thread
From: Andrii Nakryiko @ 2019-07-12 19:57 UTC (permalink / raw)
  To: Ilya Leoshkevich; +Cc: bpf, Networking, gor, heiko.carstens

On Fri, Jul 12, 2019 at 6:57 AM Ilya Leoshkevich <iii@linux.ibm.com> wrote:
>
> When directories are used as prerequisites in Makefiles, they can cause
> a lot of unnecessary rebuilds, because a directory is considered changed
> whenever a file in this directory is added, removed or modified.
>
> If the only thing a target is interested in is the existence of the
> directory it depends on, which is the case for selftests/bpf, this
> directory should be specified as an order-only prerequisite: it would
> still be created in case it does not exist, but it would not trigger a
> rebuild of a target in case it's considered changed.
>
> Signed-off-by: Ilya Leoshkevich <iii@linux.ibm.com>
> ---

Thanks!

Acked-by: Andrii Nakryiko <andriin@fb.com>


>  tools/testing/selftests/bpf/Makefile | 16 ++++++++--------
>  1 file changed, 8 insertions(+), 8 deletions(-)
>

[...]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH bpf] selftests/bpf: make directory prerequisites order-only
  2019-07-12 13:56 [PATCH bpf] selftests/bpf: make directory prerequisites order-only Ilya Leoshkevich
  2019-07-12 19:57 ` Andrii Nakryiko
@ 2019-07-15 22:19 ` Daniel Borkmann
  2019-07-16 17:49   ` Alexei Starovoitov
  1 sibling, 1 reply; 6+ messages in thread
From: Daniel Borkmann @ 2019-07-15 22:19 UTC (permalink / raw)
  To: Ilya Leoshkevich, bpf, netdev; +Cc: gor, heiko.carstens

On 7/12/19 3:56 PM, Ilya Leoshkevich wrote:
> When directories are used as prerequisites in Makefiles, they can cause
> a lot of unnecessary rebuilds, because a directory is considered changed
> whenever a file in this directory is added, removed or modified.
> 
> If the only thing a target is interested in is the existence of the
> directory it depends on, which is the case for selftests/bpf, this
> directory should be specified as an order-only prerequisite: it would
> still be created in case it does not exist, but it would not trigger a
> rebuild of a target in case it's considered changed.
> 
> Signed-off-by: Ilya Leoshkevich <iii@linux.ibm.com>

Applied, thanks!

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH bpf] selftests/bpf: make directory prerequisites order-only
  2019-07-15 22:19 ` Daniel Borkmann
@ 2019-07-16 17:49   ` Alexei Starovoitov
  2019-07-16 19:39     ` Andrii Nakryiko
  2019-07-17  9:10     ` Ilya Leoshkevich
  0 siblings, 2 replies; 6+ messages in thread
From: Alexei Starovoitov @ 2019-07-16 17:49 UTC (permalink / raw)
  To: Daniel Borkmann
  Cc: Ilya Leoshkevich, bpf, Network Development, gor, Heiko Carstens

On Mon, Jul 15, 2019 at 3:22 PM Daniel Borkmann <daniel@iogearbox.net> wrote:
>
> On 7/12/19 3:56 PM, Ilya Leoshkevich wrote:
> > When directories are used as prerequisites in Makefiles, they can cause
> > a lot of unnecessary rebuilds, because a directory is considered changed
> > whenever a file in this directory is added, removed or modified.
> >
> > If the only thing a target is interested in is the existence of the
> > directory it depends on, which is the case for selftests/bpf, this
> > directory should be specified as an order-only prerequisite: it would
> > still be created in case it does not exist, but it would not trigger a
> > rebuild of a target in case it's considered changed.
> >
> > Signed-off-by: Ilya Leoshkevich <iii@linux.ibm.com>
>
> Applied, thanks!

Hi Ilya,

this commit breaks map_tests.
To reproduce:
rm map_tests/tests.h
make
tests.h will not be regenerated.
Please provide a fix asap.
We cannot ship bpf tree with such failure.

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH bpf] selftests/bpf: make directory prerequisites order-only
  2019-07-16 17:49   ` Alexei Starovoitov
@ 2019-07-16 19:39     ` Andrii Nakryiko
  2019-07-17  9:10     ` Ilya Leoshkevich
  1 sibling, 0 replies; 6+ messages in thread
From: Andrii Nakryiko @ 2019-07-16 19:39 UTC (permalink / raw)
  To: Alexei Starovoitov
  Cc: Daniel Borkmann, Ilya Leoshkevich, bpf, Network Development, gor,
	Heiko Carstens

On Tue, Jul 16, 2019 at 10:49 AM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Mon, Jul 15, 2019 at 3:22 PM Daniel Borkmann <daniel@iogearbox.net> wrote:
> >
> > On 7/12/19 3:56 PM, Ilya Leoshkevich wrote:
> > > When directories are used as prerequisites in Makefiles, they can cause
> > > a lot of unnecessary rebuilds, because a directory is considered changed
> > > whenever a file in this directory is added, removed or modified.
> > >
> > > If the only thing a target is interested in is the existence of the
> > > directory it depends on, which is the case for selftests/bpf, this
> > > directory should be specified as an order-only prerequisite: it would
> > > still be created in case it does not exist, but it would not trigger a
> > > rebuild of a target in case it's considered changed.
> > >
> > > Signed-off-by: Ilya Leoshkevich <iii@linux.ibm.com>
> >
> > Applied, thanks!
>
> Hi Ilya,
>
> this commit breaks map_tests.

This change just exposed existing problem with Makefile. Sent out fix.

> To reproduce:
> rm map_tests/tests.h
> make
> tests.h will not be regenerated.
> Please provide a fix asap.
> We cannot ship bpf tree with such failure.

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH bpf] selftests/bpf: make directory prerequisites order-only
  2019-07-16 17:49   ` Alexei Starovoitov
  2019-07-16 19:39     ` Andrii Nakryiko
@ 2019-07-17  9:10     ` Ilya Leoshkevich
  1 sibling, 0 replies; 6+ messages in thread
From: Ilya Leoshkevich @ 2019-07-17  9:10 UTC (permalink / raw)
  To: Alexei Starovoitov
  Cc: Daniel Borkmann, bpf, Network Development, Vasily Gorbik,
	Heiko Carstens, Andrii Nakryiko

> Am 16.07.2019 um 19:49 schrieb Alexei Starovoitov <alexei.starovoitov@gmail.com>:
> 
> On Mon, Jul 15, 2019 at 3:22 PM Daniel Borkmann <daniel@iogearbox.net> wrote:
>> 
>> On 7/12/19 3:56 PM, Ilya Leoshkevich wrote:
>>> When directories are used as prerequisites in Makefiles, they can cause
>>> a lot of unnecessary rebuilds, because a directory is considered changed
>>> whenever a file in this directory is added, removed or modified.
>>> 
>>> If the only thing a target is interested in is the existence of the
>>> directory it depends on, which is the case for selftests/bpf, this
>>> directory should be specified as an order-only prerequisite: it would
>>> still be created in case it does not exist, but it would not trigger a
>>> rebuild of a target in case it's considered changed.
>>> 
>>> Signed-off-by: Ilya Leoshkevich <iii@linux.ibm.com>
>> 
>> Applied, thanks!
> 
> Hi Ilya,
> 
> this commit breaks map_tests.
> To reproduce:
> rm map_tests/tests.h
> make
> tests.h will not be regenerated.
> Please provide a fix asap.
> We cannot ship bpf tree with such failure.

Hi Alexei,

Sorry about this! I actually had the following in my local tree:

diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index f1f2b82b8fb8..95795cf5805c 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -231,7 +231,7 @@ ifeq ($(DWARF2BTF),y)
 endif

 PROG_TESTS_H := $(OUTPUT)/prog_tests/tests.h
-test_progs.c: $(PROG_TESTS_H)
+$(OUTPUT)/test_progs: $(PROG_TESTS_H)
 $(OUTPUT)/test_progs: CFLAGS += $(TEST_PROGS_CFLAGS)
 $(OUTPUT)/test_progs: prog_tests/*.c

@@ -258,7 +258,7 @@ MAP_TESTS_DIR = $(OUTPUT)/map_tests
 $(MAP_TESTS_DIR):
 <------>mkdir -p $@
 MAP_TESTS_H := $(MAP_TESTS_DIR)/tests.h
-test_maps.c: $(MAP_TESTS_H)
+$(OUTPUT)/test_maps: $(MAP_TESTS_H)
 $(OUTPUT)/test_maps: CFLAGS += $(TEST_MAPS_CFLAGS)
 MAP_TESTS_FILES := $(wildcard map_tests/*.c)
 $(MAP_TESTS_H): $(MAP_TESTS_FILES) | $(MAP_TESTS_DIR)
@@ -275,7 +275,7 @@ $(MAP_TESTS_H): $(MAP_TESTS_FILES) | $(MAP_TESTS_DIR)
 <------><------> ) > $(MAP_TESTS_H))

 VERIFIER_TESTS_H := $(OUTPUT)/verifier/tests.h
-test_verifier.c: $(VERIFIER_TESTS_H)
+$(OUTPUT)/test_verifier: $(VERIFIER_TESTS_H)
 $(OUTPUT)/test_verifier: CFLAGS += $(TEST_VERIFIER_CFLAGS)

 VERIFIER_TESTS_DIR = $(OUTPUT)/verifier

but did not realise that this is a pre-requisite for my directories change.
I should have tested it separately, then I would have noticed.

Andrii,
Thanks for helping out and providing the fix!

Best regards,
Ilya

^ permalink raw reply related	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2019-07-17  9:10 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-07-12 13:56 [PATCH bpf] selftests/bpf: make directory prerequisites order-only Ilya Leoshkevich
2019-07-12 19:57 ` Andrii Nakryiko
2019-07-15 22:19 ` Daniel Borkmann
2019-07-16 17:49   ` Alexei Starovoitov
2019-07-16 19:39     ` Andrii Nakryiko
2019-07-17  9:10     ` Ilya Leoshkevich

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).