[PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs

[PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs

[Andrii Nakryiko]

This patch set fixes perf_buffer__new() behavior on systems which have some of
the CPUs offline/missing (due to difference between "possible" and "online"
sets). perf_buffer will create per-CPU buffer and open/attach to corresponding
perf_event only on CPUs present and online at the moment of perf_buffer
creation. Without this logic, perf_buffer creation has no chances of
succeeding on such systems, preventing valid and correct BPF applications from
starting.

Andrii Nakryiko (4):
  libbpf: extract and generalize CPU mask parsing logic
  selftests/bpf: add CPU mask parsing tests
  libbpf: don't attach perf_buffer to offline/missing CPUs
  selftests/bpf: fix perf_buffer test on systems w/ offline CPUs

 tools/lib/bpf/libbpf.c                        | 157 ++++++++++++------
 tools/lib/bpf/libbpf_internal.h               |   2 +
 .../selftests/bpf/prog_tests/cpu_mask.c       |  78 +++++++++
 .../selftests/bpf/prog_tests/perf_buffer.c    |  29 +++-
 4 files changed, 213 insertions(+), 53 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/cpu_mask.c

-- 
2.17.1

Re: [PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs

[Alexei Starovoitov]

On Wed, Dec 11, 2019 at 5:35 PM Andrii Nakryiko <andriin@fb.com> wrote:
>
> This patch set fixes perf_buffer__new() behavior on systems which have some of
> the CPUs offline/missing (due to difference between "possible" and "online"
> sets). perf_buffer will create per-CPU buffer and open/attach to corresponding
> perf_event only on CPUs present and online at the moment of perf_buffer
> creation. Without this logic, perf_buffer creation has no chances of
> succeeding on such systems, preventing valid and correct BPF applications from
> starting.

Applied. Thanks

Re: [PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs

[Daniel Borkmann]

On Wed, Dec 11, 2019 at 05:35:20PM -0800, Andrii Nakryiko wrote:
> This patch set fixes perf_buffer__new() behavior on systems which have some of
> the CPUs offline/missing (due to difference between "possible" and "online"
> sets). perf_buffer will create per-CPU buffer and open/attach to corresponding
> perf_event only on CPUs present and online at the moment of perf_buffer
> creation. Without this logic, perf_buffer creation has no chances of
> succeeding on such systems, preventing valid and correct BPF applications from
> starting.

Once CPU goes back online and processes BPF events, any attempt to push into
perf RB via bpf_perf_event_output() with flag BPF_F_CURRENT_CPU would silently
get discarded. Should rather perf API be fixed instead of plain skipping as done
here to at least allow creation of ring buffer for BPF to avoid such case?

> Andrii Nakryiko (4):
>   libbpf: extract and generalize CPU mask parsing logic
>   selftests/bpf: add CPU mask parsing tests
>   libbpf: don't attach perf_buffer to offline/missing CPUs
>   selftests/bpf: fix perf_buffer test on systems w/ offline CPUs
> 
>  tools/lib/bpf/libbpf.c                        | 157 ++++++++++++------
>  tools/lib/bpf/libbpf_internal.h               |   2 +
>  .../selftests/bpf/prog_tests/cpu_mask.c       |  78 +++++++++
>  .../selftests/bpf/prog_tests/perf_buffer.c    |  29 +++-
>  4 files changed, 213 insertions(+), 53 deletions(-)
>  create mode 100644 tools/testing/selftests/bpf/prog_tests/cpu_mask.c
> 
> -- 
> 2.17.1
> 

Re: [PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs

[Andrii Nakryiko]

On Mon, Dec 16, 2019 at 6:44 AM Daniel Borkmann <daniel@iogearbox.net> wrote:
>
> On Wed, Dec 11, 2019 at 05:35:20PM -0800, Andrii Nakryiko wrote:
> > This patch set fixes perf_buffer__new() behavior on systems which have some of
> > the CPUs offline/missing (due to difference between "possible" and "online"
> > sets). perf_buffer will create per-CPU buffer and open/attach to corresponding
> > perf_event only on CPUs present and online at the moment of perf_buffer
> > creation. Without this logic, perf_buffer creation has no chances of
> > succeeding on such systems, preventing valid and correct BPF applications from
> > starting.
>
> Once CPU goes back online and processes BPF events, any attempt to push into
> perf RB via bpf_perf_event_output() with flag BPF_F_CURRENT_CPU would silently

bpf_perf_event_output() will return error code in such case, so it's
not exactly undetectable by application.


> get discarded. Should rather perf API be fixed instead of plain skipping as done
> here to at least allow creation of ring buffer for BPF to avoid such case?

Can you elaborate on what perf API fix you have in mind? Do you mean
for perf to allow attaching ring buffer to offline CPU or something
else?

>
> > Andrii Nakryiko (4):
> >   libbpf: extract and generalize CPU mask parsing logic
> >   selftests/bpf: add CPU mask parsing tests
> >   libbpf: don't attach perf_buffer to offline/missing CPUs
> >   selftests/bpf: fix perf_buffer test on systems w/ offline CPUs
> >
> >  tools/lib/bpf/libbpf.c                        | 157 ++++++++++++------
> >  tools/lib/bpf/libbpf_internal.h               |   2 +
> >  .../selftests/bpf/prog_tests/cpu_mask.c       |  78 +++++++++
> >  .../selftests/bpf/prog_tests/perf_buffer.c    |  29 +++-
> >  4 files changed, 213 insertions(+), 53 deletions(-)
> >  create mode 100644 tools/testing/selftests/bpf/prog_tests/cpu_mask.c
> >
> > --
> > 2.17.1
> >

Re: [PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs

[Daniel Borkmann]

On 12/16/19 6:59 PM, Andrii Nakryiko wrote:
> On Mon, Dec 16, 2019 at 6:44 AM Daniel Borkmann <daniel@iogearbox.net> wrote:
>> On Wed, Dec 11, 2019 at 05:35:20PM -0800, Andrii Nakryiko wrote:
>>> This patch set fixes perf_buffer__new() behavior on systems which have some of
>>> the CPUs offline/missing (due to difference between "possible" and "online"
>>> sets). perf_buffer will create per-CPU buffer and open/attach to corresponding
>>> perf_event only on CPUs present and online at the moment of perf_buffer
>>> creation. Without this logic, perf_buffer creation has no chances of
>>> succeeding on such systems, preventing valid and correct BPF applications from
>>> starting.
>>
>> Once CPU goes back online and processes BPF events, any attempt to push into
>> perf RB via bpf_perf_event_output() with flag BPF_F_CURRENT_CPU would silently
> 
> bpf_perf_event_output() will return error code in such case, so it's
> not exactly undetectable by application.

Yeah, true, given there would be no element in the perf map at that slot, the
program would receive -ENOENT and we could account for missed events via per
CPU map or such.

>> get discarded. Should rather perf API be fixed instead of plain skipping as done
>> here to at least allow creation of ring buffer for BPF to avoid such case?
> 
> Can you elaborate on what perf API fix you have in mind? Do you mean
> for perf to allow attaching ring buffer to offline CPU or something
> else?

Yes, was wondering about the former, meaning, possibility to attach ring buffer
to offline CPU.

>>> Andrii Nakryiko (4):
>>>    libbpf: extract and generalize CPU mask parsing logic
>>>    selftests/bpf: add CPU mask parsing tests
>>>    libbpf: don't attach perf_buffer to offline/missing CPUs
>>>    selftests/bpf: fix perf_buffer test on systems w/ offline CPUs
>>>
>>>   tools/lib/bpf/libbpf.c                        | 157 ++++++++++++------
>>>   tools/lib/bpf/libbpf_internal.h               |   2 +
>>>   .../selftests/bpf/prog_tests/cpu_mask.c       |  78 +++++++++
>>>   .../selftests/bpf/prog_tests/perf_buffer.c    |  29 +++-
>>>   4 files changed, 213 insertions(+), 53 deletions(-)
>>>   create mode 100644 tools/testing/selftests/bpf/prog_tests/cpu_mask.c
>>>
>>> --
>>> 2.17.1
>>>

Re: [PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs

[Andrii Nakryiko]

On Tue, Dec 17, 2019 at 5:00 AM Daniel Borkmann <daniel@iogearbox.net> wrote:
>
> On 12/16/19 6:59 PM, Andrii Nakryiko wrote:
> > On Mon, Dec 16, 2019 at 6:44 AM Daniel Borkmann <daniel@iogearbox.net> wrote:
> >> On Wed, Dec 11, 2019 at 05:35:20PM -0800, Andrii Nakryiko wrote:
> >>> This patch set fixes perf_buffer__new() behavior on systems which have some of
> >>> the CPUs offline/missing (due to difference between "possible" and "online"
> >>> sets). perf_buffer will create per-CPU buffer and open/attach to corresponding
> >>> perf_event only on CPUs present and online at the moment of perf_buffer
> >>> creation. Without this logic, perf_buffer creation has no chances of
> >>> succeeding on such systems, preventing valid and correct BPF applications from
> >>> starting.
> >>
> >> Once CPU goes back online and processes BPF events, any attempt to push into
> >> perf RB via bpf_perf_event_output() with flag BPF_F_CURRENT_CPU would silently
> >
> > bpf_perf_event_output() will return error code in such case, so it's
> > not exactly undetectable by application.
>
> Yeah, true, given there would be no element in the perf map at that slot, the
> program would receive -ENOENT and we could account for missed events via per
> CPU map or such.
>
> >> get discarded. Should rather perf API be fixed instead of plain skipping as done
> >> here to at least allow creation of ring buffer for BPF to avoid such case?
> >
> > Can you elaborate on what perf API fix you have in mind? Do you mean
> > for perf to allow attaching ring buffer to offline CPU or something
> > else?
>
> Yes, was wondering about the former, meaning, possibility to attach ring buffer
> to offline CPU.

This sounds like a more heavy-weight fix, I'll put it on backburner
for now and will look at perf code when I get a chance to see if/how
it's possible.

>
> >>> Andrii Nakryiko (4):
> >>>    libbpf: extract and generalize CPU mask parsing logic
> >>>    selftests/bpf: add CPU mask parsing tests
> >>>    libbpf: don't attach perf_buffer to offline/missing CPUs
> >>>    selftests/bpf: fix perf_buffer test on systems w/ offline CPUs
> >>>
> >>>   tools/lib/bpf/libbpf.c                        | 157 ++++++++++++------
> >>>   tools/lib/bpf/libbpf_internal.h               |   2 +
> >>>   .../selftests/bpf/prog_tests/cpu_mask.c       |  78 +++++++++
> >>>   .../selftests/bpf/prog_tests/perf_buffer.c    |  29 +++-
> >>>   4 files changed, 213 insertions(+), 53 deletions(-)
> >>>   create mode 100644 tools/testing/selftests/bpf/prog_tests/cpu_mask.c
> >>>
> >>> --
> >>> 2.17.1
> >>>
>

Re: [PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs

[Naresh Kamboju]

On Thu, 12 Dec 2019 at 07:05, Andrii Nakryiko <andriin@fb.com> wrote:
>
> This patch set fixes perf_buffer__new() behavior on systems which have some of
> the CPUs offline/missing (due to difference between "possible" and "online"
> sets). perf_buffer will create per-CPU buffer and open/attach to corresponding
> perf_event only on CPUs present and online at the moment of perf_buffer
> creation. Without this logic, perf_buffer creation has no chances of
> succeeding on such systems, preventing valid and correct BPF applications from
> starting.
>
> Andrii Nakryiko (4):
>   libbpf: extract and generalize CPU mask parsing logic
>   selftests/bpf: add CPU mask parsing tests
>   libbpf: don't attach perf_buffer to offline/missing CPUs

perf build failed on stable-rc 5.5 branch.

libbpf.c: In function '__perf_buffer__new':
libbpf.c:6159:8: error: implicit declaration of function
'parse_cpu_mask_file'; did you mean 'parse_uint_from_file'?
[-Werror=implicit-function-declaration]
  err = parse_cpu_mask_file(online_cpus_file, &online, &n);
        ^~~~~~~~~~~~~~~~~~~
        parse_uint_from_file
libbpf.c:6159:8: error: nested extern declaration of
'parse_cpu_mask_file' [-Werror=nested-externs]

build log,
https://ci.linaro.org/view/lkft/job/openembedded-lkft-linux-stable-rc-5.5/DISTRO=lkft,MACHINE=hikey,label=docker-lkft/11/console

-- 
Linaro LKFT
https://lkft.linaro.org

Re: [PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs

[Andrii Nakryiko]

On Sun, Feb 9, 2020 at 9:18 AM Naresh Kamboju <naresh.kamboju@linaro.org> wrote:
>
> On Thu, 12 Dec 2019 at 07:05, Andrii Nakryiko <andriin@fb.com> wrote:
> >
> > This patch set fixes perf_buffer__new() behavior on systems which have some of
> > the CPUs offline/missing (due to difference between "possible" and "online"
> > sets). perf_buffer will create per-CPU buffer and open/attach to corresponding
> > perf_event only on CPUs present and online at the moment of perf_buffer
> > creation. Without this logic, perf_buffer creation has no chances of
> > succeeding on such systems, preventing valid and correct BPF applications from
> > starting.
> >
> > Andrii Nakryiko (4):
> >   libbpf: extract and generalize CPU mask parsing logic
> >   selftests/bpf: add CPU mask parsing tests
> >   libbpf: don't attach perf_buffer to offline/missing CPUs
>
> perf build failed on stable-rc 5.5 branch.
>
> libbpf.c: In function '__perf_buffer__new':
> libbpf.c:6159:8: error: implicit declaration of function
> 'parse_cpu_mask_file'; did you mean 'parse_uint_from_file'?
> [-Werror=implicit-function-declaration]
>   err = parse_cpu_mask_file(online_cpus_file, &online, &n);
>         ^~~~~~~~~~~~~~~~~~~
>         parse_uint_from_file
> libbpf.c:6159:8: error: nested extern declaration of
> 'parse_cpu_mask_file' [-Werror=nested-externs]
>
> build log,
> https://ci.linaro.org/view/lkft/job/openembedded-lkft-linux-stable-rc-5.5/DISTRO=lkft,MACHINE=hikey,label=docker-lkft/11/console
>

Thanks for reporting!

These changes depend on commit 6803ee25f0ea ("libbpf: Extract and
generalize CPU mask parsing logic"), which weren't backported to
stable. Greg, can you please pull that one as well? Thanks!

> --
> Linaro LKFT
> https://lkft.linaro.org

Re: [PATCH bpf-next 0/4] Fix perf_buffer creation on systems with offline CPUs

[Greg Kroah-Hartman]

On Sun, Feb 09, 2020 at 10:32:43AM -0800, Andrii Nakryiko wrote:
> On Sun, Feb 9, 2020 at 9:18 AM Naresh Kamboju <naresh.kamboju@linaro.org> wrote:
> >
> > On Thu, 12 Dec 2019 at 07:05, Andrii Nakryiko <andriin@fb.com> wrote:
> > >
> > > This patch set fixes perf_buffer__new() behavior on systems which have some of
> > > the CPUs offline/missing (due to difference between "possible" and "online"
> > > sets). perf_buffer will create per-CPU buffer and open/attach to corresponding
> > > perf_event only on CPUs present and online at the moment of perf_buffer
> > > creation. Without this logic, perf_buffer creation has no chances of
> > > succeeding on such systems, preventing valid and correct BPF applications from
> > > starting.
> > >
> > > Andrii Nakryiko (4):
> > >   libbpf: extract and generalize CPU mask parsing logic
> > >   selftests/bpf: add CPU mask parsing tests
> > >   libbpf: don't attach perf_buffer to offline/missing CPUs
> >
> > perf build failed on stable-rc 5.5 branch.
> >
> > libbpf.c: In function '__perf_buffer__new':
> > libbpf.c:6159:8: error: implicit declaration of function
> > 'parse_cpu_mask_file'; did you mean 'parse_uint_from_file'?
> > [-Werror=implicit-function-declaration]
> >   err = parse_cpu_mask_file(online_cpus_file, &online, &n);
> >         ^~~~~~~~~~~~~~~~~~~
> >         parse_uint_from_file
> > libbpf.c:6159:8: error: nested extern declaration of
> > 'parse_cpu_mask_file' [-Werror=nested-externs]
> >
> > build log,
> > https://ci.linaro.org/view/lkft/job/openembedded-lkft-linux-stable-rc-5.5/DISTRO=lkft,MACHINE=hikey,label=docker-lkft/11/console
> >
> 
> Thanks for reporting!
> 
> These changes depend on commit 6803ee25f0ea ("libbpf: Extract and
> generalize CPU mask parsing logic"), which weren't backported to
> stable. Greg, can you please pull that one as well? Thanks!

Now applied, thanks!

greg k-h