BPF Archive on lore.kernel.org
 help / color / Atom feed
* [PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs
@ 2019-12-12  1:35 Andrii Nakryiko
  2019-12-13 21:04 ` Alexei Starovoitov
                   ` (2 more replies)
  0 siblings, 3 replies; 9+ messages in thread
From: Andrii Nakryiko @ 2019-12-12  1:35 UTC (permalink / raw)
  To: bpf, netdev, ast, daniel; +Cc: andrii.nakryiko, kernel-team, Andrii Nakryiko

This patch set fixes perf_buffer__new() behavior on systems which have some of
the CPUs offline/missing (due to difference between "possible" and "online"
sets). perf_buffer will create per-CPU buffer and open/attach to corresponding
perf_event only on CPUs present and online at the moment of perf_buffer
creation. Without this logic, perf_buffer creation has no chances of
succeeding on such systems, preventing valid and correct BPF applications from
starting.

Andrii Nakryiko (4):
  libbpf: extract and generalize CPU mask parsing logic
  selftests/bpf: add CPU mask parsing tests
  libbpf: don't attach perf_buffer to offline/missing CPUs
  selftests/bpf: fix perf_buffer test on systems w/ offline CPUs

 tools/lib/bpf/libbpf.c                        | 157 ++++++++++++------
 tools/lib/bpf/libbpf_internal.h               |   2 +
 .../selftests/bpf/prog_tests/cpu_mask.c       |  78 +++++++++
 .../selftests/bpf/prog_tests/perf_buffer.c    |  29 +++-
 4 files changed, 213 insertions(+), 53 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/cpu_mask.c

-- 
2.17.1


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs
  2019-12-12  1:35 [PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs Andrii Nakryiko
@ 2019-12-13 21:04 ` Alexei Starovoitov
  2019-12-16 14:44 ` Daniel Borkmann
  2020-02-09 17:18 ` Naresh Kamboju
  2 siblings, 0 replies; 9+ messages in thread
From: Alexei Starovoitov @ 2019-12-13 21:04 UTC (permalink / raw)
  To: Andrii Nakryiko
  Cc: bpf, Network Development, Alexei Starovoitov, Daniel Borkmann,
	Andrii Nakryiko, Kernel Team

On Wed, Dec 11, 2019 at 5:35 PM Andrii Nakryiko <andriin@fb.com> wrote:
>
> This patch set fixes perf_buffer__new() behavior on systems which have some of
> the CPUs offline/missing (due to difference between "possible" and "online"
> sets). perf_buffer will create per-CPU buffer and open/attach to corresponding
> perf_event only on CPUs present and online at the moment of perf_buffer
> creation. Without this logic, perf_buffer creation has no chances of
> succeeding on such systems, preventing valid and correct BPF applications from
> starting.

Applied. Thanks

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs
  2019-12-12  1:35 [PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs Andrii Nakryiko
  2019-12-13 21:04 ` Alexei Starovoitov
@ 2019-12-16 14:44 ` Daniel Borkmann
  2019-12-16 17:59   ` Andrii Nakryiko
  2020-02-09 17:18 ` Naresh Kamboju
  2 siblings, 1 reply; 9+ messages in thread
From: Daniel Borkmann @ 2019-12-16 14:44 UTC (permalink / raw)
  To: Andrii Nakryiko; +Cc: bpf, netdev, ast, andrii.nakryiko, kernel-team

On Wed, Dec 11, 2019 at 05:35:20PM -0800, Andrii Nakryiko wrote:
> This patch set fixes perf_buffer__new() behavior on systems which have some of
> the CPUs offline/missing (due to difference between "possible" and "online"
> sets). perf_buffer will create per-CPU buffer and open/attach to corresponding
> perf_event only on CPUs present and online at the moment of perf_buffer
> creation. Without this logic, perf_buffer creation has no chances of
> succeeding on such systems, preventing valid and correct BPF applications from
> starting.

Once CPU goes back online and processes BPF events, any attempt to push into
perf RB via bpf_perf_event_output() with flag BPF_F_CURRENT_CPU would silently
get discarded. Should rather perf API be fixed instead of plain skipping as done
here to at least allow creation of ring buffer for BPF to avoid such case?

> Andrii Nakryiko (4):
>   libbpf: extract and generalize CPU mask parsing logic
>   selftests/bpf: add CPU mask parsing tests
>   libbpf: don't attach perf_buffer to offline/missing CPUs
>   selftests/bpf: fix perf_buffer test on systems w/ offline CPUs
> 
>  tools/lib/bpf/libbpf.c                        | 157 ++++++++++++------
>  tools/lib/bpf/libbpf_internal.h               |   2 +
>  .../selftests/bpf/prog_tests/cpu_mask.c       |  78 +++++++++
>  .../selftests/bpf/prog_tests/perf_buffer.c    |  29 +++-
>  4 files changed, 213 insertions(+), 53 deletions(-)
>  create mode 100644 tools/testing/selftests/bpf/prog_tests/cpu_mask.c
> 
> -- 
> 2.17.1
> 

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs
  2019-12-16 14:44 ` Daniel Borkmann
@ 2019-12-16 17:59   ` Andrii Nakryiko
  2019-12-17 13:00     ` Daniel Borkmann
  0 siblings, 1 reply; 9+ messages in thread
From: Andrii Nakryiko @ 2019-12-16 17:59 UTC (permalink / raw)
  To: Daniel Borkmann
  Cc: Andrii Nakryiko, bpf, Networking, Alexei Starovoitov, Kernel Team

On Mon, Dec 16, 2019 at 6:44 AM Daniel Borkmann <daniel@iogearbox.net> wrote:
>
> On Wed, Dec 11, 2019 at 05:35:20PM -0800, Andrii Nakryiko wrote:
> > This patch set fixes perf_buffer__new() behavior on systems which have some of
> > the CPUs offline/missing (due to difference between "possible" and "online"
> > sets). perf_buffer will create per-CPU buffer and open/attach to corresponding
> > perf_event only on CPUs present and online at the moment of perf_buffer
> > creation. Without this logic, perf_buffer creation has no chances of
> > succeeding on such systems, preventing valid and correct BPF applications from
> > starting.
>
> Once CPU goes back online and processes BPF events, any attempt to push into
> perf RB via bpf_perf_event_output() with flag BPF_F_CURRENT_CPU would silently

bpf_perf_event_output() will return error code in such case, so it's
not exactly undetectable by application.


> get discarded. Should rather perf API be fixed instead of plain skipping as done
> here to at least allow creation of ring buffer for BPF to avoid such case?

Can you elaborate on what perf API fix you have in mind? Do you mean
for perf to allow attaching ring buffer to offline CPU or something
else?

>
> > Andrii Nakryiko (4):
> >   libbpf: extract and generalize CPU mask parsing logic
> >   selftests/bpf: add CPU mask parsing tests
> >   libbpf: don't attach perf_buffer to offline/missing CPUs
> >   selftests/bpf: fix perf_buffer test on systems w/ offline CPUs
> >
> >  tools/lib/bpf/libbpf.c                        | 157 ++++++++++++------
> >  tools/lib/bpf/libbpf_internal.h               |   2 +
> >  .../selftests/bpf/prog_tests/cpu_mask.c       |  78 +++++++++
> >  .../selftests/bpf/prog_tests/perf_buffer.c    |  29 +++-
> >  4 files changed, 213 insertions(+), 53 deletions(-)
> >  create mode 100644 tools/testing/selftests/bpf/prog_tests/cpu_mask.c
> >
> > --
> > 2.17.1
> >

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs
  2019-12-16 17:59   ` Andrii Nakryiko
@ 2019-12-17 13:00     ` Daniel Borkmann
  2019-12-20 17:46       ` Andrii Nakryiko
  0 siblings, 1 reply; 9+ messages in thread
From: Daniel Borkmann @ 2019-12-17 13:00 UTC (permalink / raw)
  To: Andrii Nakryiko
  Cc: Andrii Nakryiko, bpf, Networking, Alexei Starovoitov, Kernel Team

On 12/16/19 6:59 PM, Andrii Nakryiko wrote:
> On Mon, Dec 16, 2019 at 6:44 AM Daniel Borkmann <daniel@iogearbox.net> wrote:
>> On Wed, Dec 11, 2019 at 05:35:20PM -0800, Andrii Nakryiko wrote:
>>> This patch set fixes perf_buffer__new() behavior on systems which have some of
>>> the CPUs offline/missing (due to difference between "possible" and "online"
>>> sets). perf_buffer will create per-CPU buffer and open/attach to corresponding
>>> perf_event only on CPUs present and online at the moment of perf_buffer
>>> creation. Without this logic, perf_buffer creation has no chances of
>>> succeeding on such systems, preventing valid and correct BPF applications from
>>> starting.
>>
>> Once CPU goes back online and processes BPF events, any attempt to push into
>> perf RB via bpf_perf_event_output() with flag BPF_F_CURRENT_CPU would silently
> 
> bpf_perf_event_output() will return error code in such case, so it's
> not exactly undetectable by application.

Yeah, true, given there would be no element in the perf map at that slot, the
program would receive -ENOENT and we could account for missed events via per
CPU map or such.

>> get discarded. Should rather perf API be fixed instead of plain skipping as done
>> here to at least allow creation of ring buffer for BPF to avoid such case?
> 
> Can you elaborate on what perf API fix you have in mind? Do you mean
> for perf to allow attaching ring buffer to offline CPU or something
> else?

Yes, was wondering about the former, meaning, possibility to attach ring buffer
to offline CPU.

>>> Andrii Nakryiko (4):
>>>    libbpf: extract and generalize CPU mask parsing logic
>>>    selftests/bpf: add CPU mask parsing tests
>>>    libbpf: don't attach perf_buffer to offline/missing CPUs
>>>    selftests/bpf: fix perf_buffer test on systems w/ offline CPUs
>>>
>>>   tools/lib/bpf/libbpf.c                        | 157 ++++++++++++------
>>>   tools/lib/bpf/libbpf_internal.h               |   2 +
>>>   .../selftests/bpf/prog_tests/cpu_mask.c       |  78 +++++++++
>>>   .../selftests/bpf/prog_tests/perf_buffer.c    |  29 +++-
>>>   4 files changed, 213 insertions(+), 53 deletions(-)
>>>   create mode 100644 tools/testing/selftests/bpf/prog_tests/cpu_mask.c
>>>
>>> --
>>> 2.17.1
>>>


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs
  2019-12-17 13:00     ` Daniel Borkmann
@ 2019-12-20 17:46       ` Andrii Nakryiko
  0 siblings, 0 replies; 9+ messages in thread
From: Andrii Nakryiko @ 2019-12-20 17:46 UTC (permalink / raw)
  To: Daniel Borkmann
  Cc: Andrii Nakryiko, bpf, Networking, Alexei Starovoitov, Kernel Team

On Tue, Dec 17, 2019 at 5:00 AM Daniel Borkmann <daniel@iogearbox.net> wrote:
>
> On 12/16/19 6:59 PM, Andrii Nakryiko wrote:
> > On Mon, Dec 16, 2019 at 6:44 AM Daniel Borkmann <daniel@iogearbox.net> wrote:
> >> On Wed, Dec 11, 2019 at 05:35:20PM -0800, Andrii Nakryiko wrote:
> >>> This patch set fixes perf_buffer__new() behavior on systems which have some of
> >>> the CPUs offline/missing (due to difference between "possible" and "online"
> >>> sets). perf_buffer will create per-CPU buffer and open/attach to corresponding
> >>> perf_event only on CPUs present and online at the moment of perf_buffer
> >>> creation. Without this logic, perf_buffer creation has no chances of
> >>> succeeding on such systems, preventing valid and correct BPF applications from
> >>> starting.
> >>
> >> Once CPU goes back online and processes BPF events, any attempt to push into
> >> perf RB via bpf_perf_event_output() with flag BPF_F_CURRENT_CPU would silently
> >
> > bpf_perf_event_output() will return error code in such case, so it's
> > not exactly undetectable by application.
>
> Yeah, true, given there would be no element in the perf map at that slot, the
> program would receive -ENOENT and we could account for missed events via per
> CPU map or such.
>
> >> get discarded. Should rather perf API be fixed instead of plain skipping as done
> >> here to at least allow creation of ring buffer for BPF to avoid such case?
> >
> > Can you elaborate on what perf API fix you have in mind? Do you mean
> > for perf to allow attaching ring buffer to offline CPU or something
> > else?
>
> Yes, was wondering about the former, meaning, possibility to attach ring buffer
> to offline CPU.

This sounds like a more heavy-weight fix, I'll put it on backburner
for now and will look at perf code when I get a chance to see if/how
it's possible.

>
> >>> Andrii Nakryiko (4):
> >>>    libbpf: extract and generalize CPU mask parsing logic
> >>>    selftests/bpf: add CPU mask parsing tests
> >>>    libbpf: don't attach perf_buffer to offline/missing CPUs
> >>>    selftests/bpf: fix perf_buffer test on systems w/ offline CPUs
> >>>
> >>>   tools/lib/bpf/libbpf.c                        | 157 ++++++++++++------
> >>>   tools/lib/bpf/libbpf_internal.h               |   2 +
> >>>   .../selftests/bpf/prog_tests/cpu_mask.c       |  78 +++++++++
> >>>   .../selftests/bpf/prog_tests/perf_buffer.c    |  29 +++-
> >>>   4 files changed, 213 insertions(+), 53 deletions(-)
> >>>   create mode 100644 tools/testing/selftests/bpf/prog_tests/cpu_mask.c
> >>>
> >>> --
> >>> 2.17.1
> >>>
>

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs
  2019-12-12  1:35 [PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs Andrii Nakryiko
  2019-12-13 21:04 ` Alexei Starovoitov
  2019-12-16 14:44 ` Daniel Borkmann
@ 2020-02-09 17:18 ` Naresh Kamboju
  2020-02-09 18:32   ` Andrii Nakryiko
  2 siblings, 1 reply; 9+ messages in thread
From: Naresh Kamboju @ 2020-02-09 17:18 UTC (permalink / raw)
  To: Andrii Nakryiko, Greg Kroah-Hartman, Sasha Levin
  Cc: bpf, Netdev, ast, Daniel Borkmann, Andrii Nakryiko, Kernel Team,
	linux- stable, lkft-triage, Arnaldo Carvalho de Melo, Leo Yan

On Thu, 12 Dec 2019 at 07:05, Andrii Nakryiko <andriin@fb.com> wrote:
>
> This patch set fixes perf_buffer__new() behavior on systems which have some of
> the CPUs offline/missing (due to difference between "possible" and "online"
> sets). perf_buffer will create per-CPU buffer and open/attach to corresponding
> perf_event only on CPUs present and online at the moment of perf_buffer
> creation. Without this logic, perf_buffer creation has no chances of
> succeeding on such systems, preventing valid and correct BPF applications from
> starting.
>
> Andrii Nakryiko (4):
>   libbpf: extract and generalize CPU mask parsing logic
>   selftests/bpf: add CPU mask parsing tests
>   libbpf: don't attach perf_buffer to offline/missing CPUs

perf build failed on stable-rc 5.5 branch.

libbpf.c: In function '__perf_buffer__new':
libbpf.c:6159:8: error: implicit declaration of function
'parse_cpu_mask_file'; did you mean 'parse_uint_from_file'?
[-Werror=implicit-function-declaration]
  err = parse_cpu_mask_file(online_cpus_file, &online, &n);
        ^~~~~~~~~~~~~~~~~~~
        parse_uint_from_file
libbpf.c:6159:8: error: nested extern declaration of
'parse_cpu_mask_file' [-Werror=nested-externs]

build log,
https://ci.linaro.org/view/lkft/job/openembedded-lkft-linux-stable-rc-5.5/DISTRO=lkft,MACHINE=hikey,label=docker-lkft/11/console

-- 
Linaro LKFT
https://lkft.linaro.org

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs
  2020-02-09 17:18 ` Naresh Kamboju
@ 2020-02-09 18:32   ` Andrii Nakryiko
  2020-02-09 21:03     ` Greg Kroah-Hartman
  0 siblings, 1 reply; 9+ messages in thread
From: Andrii Nakryiko @ 2020-02-09 18:32 UTC (permalink / raw)
  To: Naresh Kamboju
  Cc: Andrii Nakryiko, Greg Kroah-Hartman, Sasha Levin, bpf, Netdev,
	Alexei Starovoitov, Daniel Borkmann, Kernel Team, linux- stable,
	lkft-triage, Arnaldo Carvalho de Melo, Leo Yan

On Sun, Feb 9, 2020 at 9:18 AM Naresh Kamboju <naresh.kamboju@linaro.org> wrote:
>
> On Thu, 12 Dec 2019 at 07:05, Andrii Nakryiko <andriin@fb.com> wrote:
> >
> > This patch set fixes perf_buffer__new() behavior on systems which have some of
> > the CPUs offline/missing (due to difference between "possible" and "online"
> > sets). perf_buffer will create per-CPU buffer and open/attach to corresponding
> > perf_event only on CPUs present and online at the moment of perf_buffer
> > creation. Without this logic, perf_buffer creation has no chances of
> > succeeding on such systems, preventing valid and correct BPF applications from
> > starting.
> >
> > Andrii Nakryiko (4):
> >   libbpf: extract and generalize CPU mask parsing logic
> >   selftests/bpf: add CPU mask parsing tests
> >   libbpf: don't attach perf_buffer to offline/missing CPUs
>
> perf build failed on stable-rc 5.5 branch.
>
> libbpf.c: In function '__perf_buffer__new':
> libbpf.c:6159:8: error: implicit declaration of function
> 'parse_cpu_mask_file'; did you mean 'parse_uint_from_file'?
> [-Werror=implicit-function-declaration]
>   err = parse_cpu_mask_file(online_cpus_file, &online, &n);
>         ^~~~~~~~~~~~~~~~~~~
>         parse_uint_from_file
> libbpf.c:6159:8: error: nested extern declaration of
> 'parse_cpu_mask_file' [-Werror=nested-externs]
>
> build log,
> https://ci.linaro.org/view/lkft/job/openembedded-lkft-linux-stable-rc-5.5/DISTRO=lkft,MACHINE=hikey,label=docker-lkft/11/console
>

Thanks for reporting!

These changes depend on commit 6803ee25f0ea ("libbpf: Extract and
generalize CPU mask parsing logic"), which weren't backported to
stable. Greg, can you please pull that one as well? Thanks!

> --
> Linaro LKFT
> https://lkft.linaro.org

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs
  2020-02-09 18:32   ` Andrii Nakryiko
@ 2020-02-09 21:03     ` Greg Kroah-Hartman
  0 siblings, 0 replies; 9+ messages in thread
From: Greg Kroah-Hartman @ 2020-02-09 21:03 UTC (permalink / raw)
  To: Andrii Nakryiko
  Cc: Naresh Kamboju, Andrii Nakryiko, Sasha Levin, bpf, Netdev,
	Alexei Starovoitov, Daniel Borkmann, Kernel Team, linux- stable,
	lkft-triage, Arnaldo Carvalho de Melo, Leo Yan

On Sun, Feb 09, 2020 at 10:32:43AM -0800, Andrii Nakryiko wrote:
> On Sun, Feb 9, 2020 at 9:18 AM Naresh Kamboju <naresh.kamboju@linaro.org> wrote:
> >
> > On Thu, 12 Dec 2019 at 07:05, Andrii Nakryiko <andriin@fb.com> wrote:
> > >
> > > This patch set fixes perf_buffer__new() behavior on systems which have some of
> > > the CPUs offline/missing (due to difference between "possible" and "online"
> > > sets). perf_buffer will create per-CPU buffer and open/attach to corresponding
> > > perf_event only on CPUs present and online at the moment of perf_buffer
> > > creation. Without this logic, perf_buffer creation has no chances of
> > > succeeding on such systems, preventing valid and correct BPF applications from
> > > starting.
> > >
> > > Andrii Nakryiko (4):
> > >   libbpf: extract and generalize CPU mask parsing logic
> > >   selftests/bpf: add CPU mask parsing tests
> > >   libbpf: don't attach perf_buffer to offline/missing CPUs
> >
> > perf build failed on stable-rc 5.5 branch.
> >
> > libbpf.c: In function '__perf_buffer__new':
> > libbpf.c:6159:8: error: implicit declaration of function
> > 'parse_cpu_mask_file'; did you mean 'parse_uint_from_file'?
> > [-Werror=implicit-function-declaration]
> >   err = parse_cpu_mask_file(online_cpus_file, &online, &n);
> >         ^~~~~~~~~~~~~~~~~~~
> >         parse_uint_from_file
> > libbpf.c:6159:8: error: nested extern declaration of
> > 'parse_cpu_mask_file' [-Werror=nested-externs]
> >
> > build log,
> > https://ci.linaro.org/view/lkft/job/openembedded-lkft-linux-stable-rc-5.5/DISTRO=lkft,MACHINE=hikey,label=docker-lkft/11/console
> >
> 
> Thanks for reporting!
> 
> These changes depend on commit 6803ee25f0ea ("libbpf: Extract and
> generalize CPU mask parsing logic"), which weren't backported to
> stable. Greg, can you please pull that one as well? Thanks!

Now applied, thanks!

greg k-h

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, back to index

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-12-12  1:35 [PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs Andrii Nakryiko
2019-12-13 21:04 ` Alexei Starovoitov
2019-12-16 14:44 ` Daniel Borkmann
2019-12-16 17:59   ` Andrii Nakryiko
2019-12-17 13:00     ` Daniel Borkmann
2019-12-20 17:46       ` Andrii Nakryiko
2020-02-09 17:18 ` Naresh Kamboju
2020-02-09 18:32   ` Andrii Nakryiko
2020-02-09 21:03     ` Greg Kroah-Hartman

BPF Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/bpf/0 bpf/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 bpf bpf/ https://lore.kernel.org/bpf \
		bpf@vger.kernel.org
	public-inbox-index bpf

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.bpf


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git