BPF Archive on lore.kernel.org
 help / color / Atom feed
* [PATCH bpf-next] tools build: propagate build failures from tools/build/Makefile.build
@ 2020-07-31  2:42 Andrii Nakryiko
  2020-08-02  3:33 ` Alexei Starovoitov
                   ` (2 more replies)
  0 siblings, 3 replies; 6+ messages in thread
From: Andrii Nakryiko @ 2020-07-31  2:42 UTC (permalink / raw)
  To: bpf, netdev, ast, daniel
  Cc: andrii.nakryiko, kernel-team, Andrii Nakryiko, Jiri Olsa,
	Arnaldo Carvalho de Melo

The '&&' command seems to have a bad effect when $(cmd_$(1)) exits with
non-zero effect: the command failure is masked (despite `set -e`) and all but
the first command of $(dep-cmd) is executed (successfully, as they are mostly
printfs), thus overall returning 0 in the end.

This means in practice that despite compilation errors, tools's build Makefile
will return success. We see this very reliably with libbpf's Makefile, which
doesn't get compilation error propagated properly. This in turns causes issues
with selftests build, as well as bpftool and other projects that rely on
building libbpf.

The fix is simple: don't use &&. Given `set -e`, we don't need to chain
commands with &&. The shell will exit on first failure, giving desired
behavior and propagating error properly.

Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Fixes: 275e2d95591e ("tools build: Move dependency copy into function")
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
---

I'm sending this against bpf-next tree, given libbpf is affected enough for me
to debug this fun problem that no one seemed to notice (or care, at least) in
almost 5 years. If there is a better kernel tree, please let me know.

 tools/build/Build.include | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tools/build/Build.include b/tools/build/Build.include
index 9ec01f4454f9..585486e40995 100644
--- a/tools/build/Build.include
+++ b/tools/build/Build.include
@@ -74,7 +74,8 @@ dep-cmd = $(if $(wildcard $(fixdep)),
 #                   dependencies in the cmd file
 if_changed_dep = $(if $(strip $(any-prereq) $(arg-check)),         \
                   @set -e;                                         \
-                  $(echo-cmd) $(cmd_$(1)) && $(dep-cmd))
+                  $(echo-cmd) $(cmd_$(1));                         \
+                  $(dep-cmd))
 
 # if_changed      - execute command if any prerequisite is newer than
 #                   target, or command line has changed
-- 
2.24.1


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH bpf-next] tools build: propagate build failures from tools/build/Makefile.build
  2020-07-31  2:42 [PATCH bpf-next] tools build: propagate build failures from tools/build/Makefile.build Andrii Nakryiko
@ 2020-08-02  3:33 ` Alexei Starovoitov
  2020-08-02 16:11 ` Jiri Olsa
  2020-08-03 14:18 ` Daniel Borkmann
  2 siblings, 0 replies; 6+ messages in thread
From: Alexei Starovoitov @ 2020-08-02  3:33 UTC (permalink / raw)
  To: Andrii Nakryiko
  Cc: bpf, Network Development, Alexei Starovoitov, Daniel Borkmann,
	Andrii Nakryiko, Kernel Team, Jiri Olsa,
	Arnaldo Carvalho de Melo, Masahiro Yamada

On Thu, Jul 30, 2020 at 7:44 PM Andrii Nakryiko <andriin@fb.com> wrote:
>
> The '&&' command seems to have a bad effect when $(cmd_$(1)) exits with
> non-zero effect: the command failure is masked (despite `set -e`) and all but
> the first command of $(dep-cmd) is executed (successfully, as they are mostly
> printfs), thus overall returning 0 in the end.
>
> This means in practice that despite compilation errors, tools's build Makefile
> will return success. We see this very reliably with libbpf's Makefile, which
> doesn't get compilation error propagated properly. This in turns causes issues
> with selftests build, as well as bpftool and other projects that rely on
> building libbpf.
>
> The fix is simple: don't use &&. Given `set -e`, we don't need to chain
> commands with &&. The shell will exit on first failure, giving desired
> behavior and propagating error properly.
>
> Cc: Jiri Olsa <jolsa@kernel.org>
> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> Fixes: 275e2d95591e ("tools build: Move dependency copy into function")
> Signed-off-by: Andrii Nakryiko <andriin@fb.com>

Jiri, Arnaldo,
could you please review and ack?

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH bpf-next] tools build: propagate build failures from tools/build/Makefile.build
  2020-07-31  2:42 [PATCH bpf-next] tools build: propagate build failures from tools/build/Makefile.build Andrii Nakryiko
  2020-08-02  3:33 ` Alexei Starovoitov
@ 2020-08-02 16:11 ` Jiri Olsa
  2020-08-02 18:22   ` Andrii Nakryiko
  2020-08-03 14:18 ` Daniel Borkmann
  2 siblings, 1 reply; 6+ messages in thread
From: Jiri Olsa @ 2020-08-02 16:11 UTC (permalink / raw)
  To: Andrii Nakryiko
  Cc: bpf, netdev, ast, daniel, andrii.nakryiko, kernel-team,
	Jiri Olsa, Arnaldo Carvalho de Melo

On Thu, Jul 30, 2020 at 07:42:44PM -0700, Andrii Nakryiko wrote:
> The '&&' command seems to have a bad effect when $(cmd_$(1)) exits with
> non-zero effect: the command failure is masked (despite `set -e`) and all but
> the first command of $(dep-cmd) is executed (successfully, as they are mostly
> printfs), thus overall returning 0 in the end.

nice, thanks for digging into this,
any idea why is the failure masked?

Acked-by: Jiri Olsa <jolsa@redhat.com>

jirka

> 
> This means in practice that despite compilation errors, tools's build Makefile
> will return success. We see this very reliably with libbpf's Makefile, which
> doesn't get compilation error propagated properly. This in turns causes issues
> with selftests build, as well as bpftool and other projects that rely on
> building libbpf.
> 
> The fix is simple: don't use &&. Given `set -e`, we don't need to chain
> commands with &&. The shell will exit on first failure, giving desired
> behavior and propagating error properly.
> 
> Cc: Jiri Olsa <jolsa@kernel.org>
> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> Fixes: 275e2d95591e ("tools build: Move dependency copy into function")
> Signed-off-by: Andrii Nakryiko <andriin@fb.com>
> ---
> 
> I'm sending this against bpf-next tree, given libbpf is affected enough for me
> to debug this fun problem that no one seemed to notice (or care, at least) in
> almost 5 years. If there is a better kernel tree, please let me know.
> 
>  tools/build/Build.include | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/build/Build.include b/tools/build/Build.include
> index 9ec01f4454f9..585486e40995 100644
> --- a/tools/build/Build.include
> +++ b/tools/build/Build.include
> @@ -74,7 +74,8 @@ dep-cmd = $(if $(wildcard $(fixdep)),
>  #                   dependencies in the cmd file
>  if_changed_dep = $(if $(strip $(any-prereq) $(arg-check)),         \
>                    @set -e;                                         \
> -                  $(echo-cmd) $(cmd_$(1)) && $(dep-cmd))
> +                  $(echo-cmd) $(cmd_$(1));                         \
> +                  $(dep-cmd))
>  
>  # if_changed      - execute command if any prerequisite is newer than
>  #                   target, or command line has changed
> -- 
> 2.24.1
> 


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH bpf-next] tools build: propagate build failures from tools/build/Makefile.build
  2020-08-02 16:11 ` Jiri Olsa
@ 2020-08-02 18:22   ` Andrii Nakryiko
  2020-08-02 21:51     ` Jiri Olsa
  0 siblings, 1 reply; 6+ messages in thread
From: Andrii Nakryiko @ 2020-08-02 18:22 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: Andrii Nakryiko, bpf, Networking, Alexei Starovoitov,
	Daniel Borkmann, Kernel Team, Jiri Olsa,
	Arnaldo Carvalho de Melo

On Sun, Aug 2, 2020 at 9:11 AM Jiri Olsa <jolsa@redhat.com> wrote:
>
> On Thu, Jul 30, 2020 at 07:42:44PM -0700, Andrii Nakryiko wrote:
> > The '&&' command seems to have a bad effect when $(cmd_$(1)) exits with
> > non-zero effect: the command failure is masked (despite `set -e`) and all but
> > the first command of $(dep-cmd) is executed (successfully, as they are mostly
> > printfs), thus overall returning 0 in the end.
>
> nice, thanks for digging into this,
> any idea why is the failure masked?

Two things.

1. In make, assume you have command f = a in one function and g = b; c
in another. If you write f && g, you end up with (a && b); c, right?

2. Try this shell script:

set -ex
false && true
true

It will return success. It won't execute the first true command, as
expected, but won't terminate the shell as you'd expect from set -e.

So basically, having a "logical operator" in a sequence of commands
negates the effect of `set -e`. Intuitively I'd expect that from ||,
but seems like && does that as well. if [] has similar effect -- any
failing command in an if check doesn't trigger an early termination of
a script.

>
> Acked-by: Jiri Olsa <jolsa@redhat.com>
>
> jirka
>
> >
> > This means in practice that despite compilation errors, tools's build Makefile
> > will return success. We see this very reliably with libbpf's Makefile, which
> > doesn't get compilation error propagated properly. This in turns causes issues
> > with selftests build, as well as bpftool and other projects that rely on
> > building libbpf.
> >
> > The fix is simple: don't use &&. Given `set -e`, we don't need to chain
> > commands with &&. The shell will exit on first failure, giving desired
> > behavior and propagating error properly.
> >
> > Cc: Jiri Olsa <jolsa@kernel.org>
> > Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> > Fixes: 275e2d95591e ("tools build: Move dependency copy into function")
> > Signed-off-by: Andrii Nakryiko <andriin@fb.com>
> > ---
> >
> > I'm sending this against bpf-next tree, given libbpf is affected enough for me
> > to debug this fun problem that no one seemed to notice (or care, at least) in
> > almost 5 years. If there is a better kernel tree, please let me know.
> >
> >  tools/build/Build.include | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/tools/build/Build.include b/tools/build/Build.include
> > index 9ec01f4454f9..585486e40995 100644
> > --- a/tools/build/Build.include
> > +++ b/tools/build/Build.include
> > @@ -74,7 +74,8 @@ dep-cmd = $(if $(wildcard $(fixdep)),
> >  #                   dependencies in the cmd file
> >  if_changed_dep = $(if $(strip $(any-prereq) $(arg-check)),         \
> >                    @set -e;                                         \
> > -                  $(echo-cmd) $(cmd_$(1)) && $(dep-cmd))
> > +                  $(echo-cmd) $(cmd_$(1));                         \
> > +                  $(dep-cmd))
> >
> >  # if_changed      - execute command if any prerequisite is newer than
> >  #                   target, or command line has changed
> > --
> > 2.24.1
> >
>

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH bpf-next] tools build: propagate build failures from tools/build/Makefile.build
  2020-08-02 18:22   ` Andrii Nakryiko
@ 2020-08-02 21:51     ` Jiri Olsa
  0 siblings, 0 replies; 6+ messages in thread
From: Jiri Olsa @ 2020-08-02 21:51 UTC (permalink / raw)
  To: Andrii Nakryiko
  Cc: Andrii Nakryiko, bpf, Networking, Alexei Starovoitov,
	Daniel Borkmann, Kernel Team, Jiri Olsa,
	Arnaldo Carvalho de Melo

On Sun, Aug 02, 2020 at 11:22:07AM -0700, Andrii Nakryiko wrote:
> On Sun, Aug 2, 2020 at 9:11 AM Jiri Olsa <jolsa@redhat.com> wrote:
> >
> > On Thu, Jul 30, 2020 at 07:42:44PM -0700, Andrii Nakryiko wrote:
> > > The '&&' command seems to have a bad effect when $(cmd_$(1)) exits with
> > > non-zero effect: the command failure is masked (despite `set -e`) and all but
> > > the first command of $(dep-cmd) is executed (successfully, as they are mostly
> > > printfs), thus overall returning 0 in the end.
> >
> > nice, thanks for digging into this,
> > any idea why is the failure masked?
> 
> Two things.
> 
> 1. In make, assume you have command f = a in one function and g = b; c
> in another. If you write f && g, you end up with (a && b); c, right?
> 
> 2. Try this shell script:
> 
> set -ex
> false && true
> true
> 
> It will return success. It won't execute the first true command, as
> expected, but won't terminate the shell as you'd expect from set -e.
> 
> So basically, having a "logical operator" in a sequence of commands
> negates the effect of `set -e`. Intuitively I'd expect that from ||,
> but seems like && does that as well. if [] has similar effect -- any
> failing command in an if check doesn't trigger an early termination of
> a script.

nice, thanks for explanation

jirka

> 
> >
> > Acked-by: Jiri Olsa <jolsa@redhat.com>
> >
> > jirka
> >
> > >
> > > This means in practice that despite compilation errors, tools's build Makefile
> > > will return success. We see this very reliably with libbpf's Makefile, which
> > > doesn't get compilation error propagated properly. This in turns causes issues
> > > with selftests build, as well as bpftool and other projects that rely on
> > > building libbpf.
> > >
> > > The fix is simple: don't use &&. Given `set -e`, we don't need to chain
> > > commands with &&. The shell will exit on first failure, giving desired
> > > behavior and propagating error properly.
> > >
> > > Cc: Jiri Olsa <jolsa@kernel.org>
> > > Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> > > Fixes: 275e2d95591e ("tools build: Move dependency copy into function")
> > > Signed-off-by: Andrii Nakryiko <andriin@fb.com>
> > > ---
> > >
> > > I'm sending this against bpf-next tree, given libbpf is affected enough for me
> > > to debug this fun problem that no one seemed to notice (or care, at least) in
> > > almost 5 years. If there is a better kernel tree, please let me know.
> > >
> > >  tools/build/Build.include | 3 ++-
> > >  1 file changed, 2 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/tools/build/Build.include b/tools/build/Build.include
> > > index 9ec01f4454f9..585486e40995 100644
> > > --- a/tools/build/Build.include
> > > +++ b/tools/build/Build.include
> > > @@ -74,7 +74,8 @@ dep-cmd = $(if $(wildcard $(fixdep)),
> > >  #                   dependencies in the cmd file
> > >  if_changed_dep = $(if $(strip $(any-prereq) $(arg-check)),         \
> > >                    @set -e;                                         \
> > > -                  $(echo-cmd) $(cmd_$(1)) && $(dep-cmd))
> > > +                  $(echo-cmd) $(cmd_$(1));                         \
> > > +                  $(dep-cmd))
> > >
> > >  # if_changed      - execute command if any prerequisite is newer than
> > >  #                   target, or command line has changed
> > > --
> > > 2.24.1
> > >
> >
> 


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH bpf-next] tools build: propagate build failures from tools/build/Makefile.build
  2020-07-31  2:42 [PATCH bpf-next] tools build: propagate build failures from tools/build/Makefile.build Andrii Nakryiko
  2020-08-02  3:33 ` Alexei Starovoitov
  2020-08-02 16:11 ` Jiri Olsa
@ 2020-08-03 14:18 ` Daniel Borkmann
  2 siblings, 0 replies; 6+ messages in thread
From: Daniel Borkmann @ 2020-08-03 14:18 UTC (permalink / raw)
  To: Andrii Nakryiko, bpf, netdev, ast
  Cc: andrii.nakryiko, kernel-team, Jiri Olsa, Arnaldo Carvalho de Melo

On 7/31/20 4:42 AM, Andrii Nakryiko wrote:
> The '&&' command seems to have a bad effect when $(cmd_$(1)) exits with
> non-zero effect: the command failure is masked (despite `set -e`) and all but
> the first command of $(dep-cmd) is executed (successfully, as they are mostly
> printfs), thus overall returning 0 in the end.
> 
> This means in practice that despite compilation errors, tools's build Makefile
> will return success. We see this very reliably with libbpf's Makefile, which
> doesn't get compilation error propagated properly. This in turns causes issues
> with selftests build, as well as bpftool and other projects that rely on
> building libbpf.
> 
> The fix is simple: don't use &&. Given `set -e`, we don't need to chain
> commands with &&. The shell will exit on first failure, giving desired
> behavior and propagating error properly.
> 
> Cc: Jiri Olsa <jolsa@kernel.org>
> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> Fixes: 275e2d95591e ("tools build: Move dependency copy into function")
> Signed-off-by: Andrii Nakryiko <andriin@fb.com>

Applied, thanks!

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, back to index

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-31  2:42 [PATCH bpf-next] tools build: propagate build failures from tools/build/Makefile.build Andrii Nakryiko
2020-08-02  3:33 ` Alexei Starovoitov
2020-08-02 16:11 ` Jiri Olsa
2020-08-02 18:22   ` Andrii Nakryiko
2020-08-02 21:51     ` Jiri Olsa
2020-08-03 14:18 ` Daniel Borkmann

BPF Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/bpf/0 bpf/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 bpf bpf/ https://lore.kernel.org/bpf \
		bpf@vger.kernel.org
	public-inbox-index bpf

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.bpf


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git