Invalid event (cycles:pp) in per-thread mode, enable system wide with '-a'.

Invalid event (cycles:pp) in per-thread mode, enable system wide with '-a'.

[Nick Desaulniers]

So I recently moved from a dual-xeon box to a zen 2 based threadripper
workstation.

My usual incantation for measuring profiles for compile time isn't working:

$ perf record -e cycles:pp --freq=128 --call-graph lbr -- make LLVM=1 -j$(nproc)
Error:
Invalid event (cycles:pp) in per-thread mode, enable system wide with '-a'.

I've already set /proc/sys/kernel/perf_event_paranoid and
/proc/sys/kernel/kptr_restrict to 0.

I remember hearing rumblings about issues with zen 2, LBR, vs zen 3.
Is this a known issue, or am I holding it wrong?
-- 
Thanks,
~Nick Desaulniers

Re: Invalid event (cycles:pp) in per-thread mode, enable system wide with '-a'.

[Slade Watkins]

Hey Nick,

> On Sep 29, 2022, at 5:54 PM, Nick Desaulniers <ndesaulniers@google.com> wrote:
> 
> I remember hearing rumblings about issues with zen 2, LBR, vs zen 3.
> Is this a known issue, or am I holding it wrong?

Hm… I also remember this. I have a Zen 2 based system that I can do testing on, so I will do so when I’m able.

If I discover something of note, I’ll get back to you.

Cheers,
-srw

Re: Invalid event (cycles:pp) in per-thread mode, enable system wide with '-a'.

[Ian Rogers]

On Thu, Sep 29, 2022 at 3:10 PM Slade Watkins <srw@sladewatkins.net> wrote:
>
> Hey Nick,
>
> > On Sep 29, 2022, at 5:54 PM, Nick Desaulniers <ndesaulniers@google.com> wrote:
> >
> > I remember hearing rumblings about issues with zen 2, LBR, vs zen 3.
> > Is this a known issue, or am I holding it wrong?
>
> Hm… I also remember this. I have a Zen 2 based system that I can do testing on, so I will do so when I’m able.
>
> If I discover something of note, I’ll get back to you.
>
> Cheers,
> -srw
>

LBR isn't yet supported for Zen but is coming:
https://lore.kernel.org/lkml/166155216401.401.5809694678609694438.tip-bot2@tip-bot2/
I'd recommend frame-pointers.

+Ravi who may be able to say if there are any issues with the precise
sampling on AMD.

Thanks,
Ian

Re: Invalid event (cycles:pp) in per-thread mode, enable system wide with '-a'.

[Namhyung Kim]

Hello,

On Thu, Sep 29, 2022 at 8:49 PM Ian Rogers <irogers@google.com> wrote:
>
> On Thu, Sep 29, 2022 at 3:10 PM Slade Watkins <srw@sladewatkins.net> wrote:
> >
> > Hey Nick,
> >
> > > On Sep 29, 2022, at 5:54 PM, Nick Desaulniers <ndesaulniers@google.com> wrote:
> > >
> > > I remember hearing rumblings about issues with zen 2, LBR, vs zen 3.
> > > Is this a known issue, or am I holding it wrong?
> >
> > Hm… I also remember this. I have a Zen 2 based system that I can do testing on, so I will do so when I’m able.
> >
> > If I discover something of note, I’ll get back to you.
> >
> > Cheers,
> > -srw
> >
>
> LBR isn't yet supported for Zen but is coming:
> https://lore.kernel.org/lkml/166155216401.401.5809694678609694438.tip-bot2@tip-bot2/
> I'd recommend frame-pointers.
>
> +Ravi who may be able to say if there are any issues with the precise
> sampling on AMD.

Afaik cvcles:pp will use IBS but it doesn't support per-task profiling
since it has no task context.  Ravi is working on it..

Thanks,
Namhyung

Re: Invalid event (cycles:pp) in per-thread mode, enable system wide with '-a'.

[Ravi Bangoria]

On 30-Sep-22 9:56 AM, Namhyung Kim wrote:
> Hello,
> 
> On Thu, Sep 29, 2022 at 8:49 PM Ian Rogers <irogers@google.com> wrote:
>>
>> On Thu, Sep 29, 2022 at 3:10 PM Slade Watkins <srw@sladewatkins.net> wrote:
>>>
>>> Hey Nick,
>>>
>>>> On Sep 29, 2022, at 5:54 PM, Nick Desaulniers <ndesaulniers@google.com> wrote:
>>>>
>>>> I remember hearing rumblings about issues with zen 2, LBR, vs zen 3.
>>>> Is this a known issue, or am I holding it wrong?
>>>
>>> Hm… I also remember this. I have a Zen 2 based system that I can do testing on, so I will do so when I’m able.
>>>
>>> If I discover something of note, I’ll get back to you.
>>>
>>> Cheers,
>>> -srw
>>>
>>
>> LBR isn't yet supported for Zen but is coming:
>> https://lore.kernel.org/lkml/166155216401.401.5809694678609694438.tip-bot2@tip-bot2/
>> I'd recommend frame-pointers.
>>
>> +Ravi who may be able to say if there are any issues with the precise
>> sampling on AMD.
> 
> Afaik cvcles:pp will use IBS but it doesn't support per-task profiling
> since it has no task context.  Ravi is working on it..

Right.
https://lore.kernel.org/lkml/20220829113347.295-1-ravi.bangoria@amd.com

Thanks,
Ravi

Re: Invalid event (cycles:pp) in per-thread mode, enable system wide with '-a'.

[Nick Desaulniers]

+ Stephane, Kees, Sandipan, Bill, ClangBuiltLinux mailing list, Yonghong
https://www.spinics.net/lists/linux-perf-users/msg23103.html
starts the thread, for context.

On Thu, Sep 29, 2022 at 9:32 PM Ravi Bangoria <ravi.bangoria@amd.com> wrote:
>
> On 30-Sep-22 9:56 AM, Namhyung Kim wrote:
> > Hello,
> >
> > On Thu, Sep 29, 2022 at 8:49 PM Ian Rogers <irogers@google.com> wrote:
> >>
> >> On Thu, Sep 29, 2022 at 3:10 PM Slade Watkins <srw@sladewatkins.net> wrote:
> >>>
> >>> Hey Nick,
> >>>
> >>>> On Sep 29, 2022, at 5:54 PM, Nick Desaulniers <ndesaulniers@google.com> wrote:
> >>>>
> >>>> I remember hearing rumblings about issues with zen 2, LBR, vs zen 3.
> >>>> Is this a known issue, or am I holding it wrong?
> >>>
> >>> Hm… I also remember this. I have a Zen 2 based system that I can do testing on, so I will do so when I’m able.
> >>>
> >>> If I discover something of note, I’ll get back to you.
> >>>
> >>> Cheers,
> >>> -srw
> >>>
> >>
> >> LBR isn't yet supported for Zen but is coming:
> >> https://lore.kernel.org/lkml/166155216401.401.5809694678609694438.tip-bot2@tip-bot2/
> >> I'd recommend frame-pointers.

Having to recompile is less than ideal for my workflow.  I have added a note to
https://github.com/ClangBuiltLinux/profiling/tree/main/perf#errors
Please let me know how I might improve the documentation.

> >>
> >> +Ravi who may be able to say if there are any issues with the precise
> >> sampling on AMD.
> >
> > Afaik cvcles:pp will use IBS but it doesn't support per-task profiling
> > since it has no task context.  Ravi is working on it..
>
> Right.
> https://lore.kernel.org/lkml/20220829113347.295-1-ravi.bangoria@amd.com

Cool, thanks for working on this Ravi.

I'm not sure yet whether I may replace the kernel on my corporate
provided workstation, so I'm not sure yet I can help test that patch.

Can you confirm that
$ perf record -e cycles:pp --freq=128 --call-graph lbr -- <command to profile>

works with just that patch applied? Or is there more work required?
What is the status of that patch?

For context, we had difficulty upstreaming support for instrumentation
based profile guided optimizations in the Linux kernel.
https://lore.kernel.org/lkml/CAHk-=whqCT0BeqBQhW8D-YoLLgp_eFY=8Y=9ieREM5xx0ef08w@mail.gmail.com/
We'd like to be able to use either instrumentation or sampling to
optimize our builds.  The major barrier to sample based approaches are
architecture / micro architecture issues with sample based profile
data collection, and bitrot of data processing utilities.
https://github.com/google/autofdo/issues/144
-- 
Thanks,
~Nick Desaulniers

Re: Invalid event (cycles:pp) in per-thread mode, enable system wide with '-a'.

[Stephane Eranian]

On Wed, Oct 5, 2022 at 2:56 PM Nick Desaulniers <ndesaulniers@google.com> wrote:
>
> + Stephane, Kees, Sandipan, Bill, ClangBuiltLinux mailing list, Yonghong
> https://www.spinics.net/lists/linux-perf-users/msg23103.html
> starts the thread, for context.
>
> On Thu, Sep 29, 2022 at 9:32 PM Ravi Bangoria <ravi.bangoria@amd.com> wrote:
> >
> > On 30-Sep-22 9:56 AM, Namhyung Kim wrote:
> > > Hello,
> > >
> > > On Thu, Sep 29, 2022 at 8:49 PM Ian Rogers <irogers@google.com> wrote:
> > >>
> > >> On Thu, Sep 29, 2022 at 3:10 PM Slade Watkins <srw@sladewatkins.net> wrote:
> > >>>
> > >>> Hey Nick,
> > >>>
> > >>>> On Sep 29, 2022, at 5:54 PM, Nick Desaulniers <ndesaulniers@google.com> wrote:
> > >>>>
> > >>>> I remember hearing rumblings about issues with zen 2, LBR, vs zen 3.
> > >>>> Is this a known issue, or am I holding it wrong?
> > >>>
> > >>> Hm… I also remember this. I have a Zen 2 based system that I can do testing on, so I will do so when I’m able.
> > >>>
> > >>> If I discover something of note, I’ll get back to you.
> > >>>
> > >>> Cheers,
> > >>> -srw
> > >>>
> > >>
> > >> LBR isn't yet supported for Zen but is coming:
> > >> https://lore.kernel.org/lkml/166155216401.401.5809694678609694438.tip-bot2@tip-bot2/
> > >> I'd recommend frame-pointers.
>
> Having to recompile is less than ideal for my workflow.  I have added a note to
> https://github.com/ClangBuiltLinux/profiling/tree/main/perf#errors
> Please let me know how I might improve the documentation.
>
> > >>
> > >> +Ravi who may be able to say if there are any issues with the precise
> > >> sampling on AMD.
> > >
> > > Afaik cvcles:pp will use IBS but it doesn't support per-task profiling
> > > since it has no task context.  Ravi is working on it..
> >
> > Right.
> > https://lore.kernel.org/lkml/20220829113347.295-1-ravi.bangoria@amd.com
>
> Cool, thanks for working on this Ravi.
>
> I'm not sure yet whether I may replace the kernel on my corporate
> provided workstation, so I'm not sure yet I can help test that patch.
>
> Can you confirm that
> $ perf record -e cycles:pp --freq=128 --call-graph lbr -- <command to profile>
>
> works with just that patch applied? Or is there more work required?
> What is the status of that patch?
>
> For context, we had difficulty upstreaming support for instrumentation
> based profile guided optimizations in the Linux kernel.
> https://lore.kernel.org/lkml/CAHk-=whqCT0BeqBQhW8D-YoLLgp_eFY=8Y=9ieREM5xx0ef08w@mail.gmail.com/
> We'd like to be able to use either instrumentation or sampling to
> optimize our builds.  The major barrier to sample based approaches are
> architecture / micro architecture issues with sample based profile
> data collection, and bitrot of data processing utilities.
> https://github.com/google/autofdo/issues/144

On existing AMD Zen2, Zen3 the following cmdline:
$ perf record -e cycles:pp --freq=128 --call-graph lbr -- <command to profile>

does not work. I see two reasons:

1. cycles:pp is likely converted into IBS op in cycle mode.
    Current kernels do not support IBS in per-thread mode.
    This is purely a kernel limitation

2. call-graph lbr is not supported on AMD because they do
   not have LBR and therefore no LBR callstack mode

The best way to get what you want here today on AMD Zen2 and Zen3:

   $ perf record -e cycles --freq=128 -g -- <command to profile>

On AMD Zen3, there is a precursor to LBR with Branch Sampling (BRS),
and you can use it to sample taken branches but not for callstacks. I
mention the cmdline here for reference:

$ perf record -e cpu/branch-brs/ -c 1000037  -b  -- <command to profile>

Note that AMD Zen3 BRS is enough to get the autoFDO usage of an
LBR working as per the cmdline above.

Hope this helps.

Re: Invalid event (cycles:pp) in per-thread mode, enable system wide with '-a'.

[Ravi Bangoria]

+cc: PeterZ

>>>>> +Ravi who may be able to say if there are any issues with the precise
>>>>> sampling on AMD.
>>>>
>>>> Afaik cvcles:pp will use IBS but it doesn't support per-task profiling
>>>> since it has no task context.  Ravi is working on it..
>>>
>>> Right.
>>> https://lore.kernel.org/lkml/20220829113347.295-1-ravi.bangoria@amd.com
>>
>> Cool, thanks for working on this Ravi.
>>
>> I'm not sure yet whether I may replace the kernel on my corporate
>> provided workstation, so I'm not sure yet I can help test that patch.
>>
>> Can you confirm that
>> $ perf record -e cycles:pp --freq=128 --call-graph lbr -- <command to profile>
>>
>> works with just that patch applied? Or is there more work required?
>> What is the status of that patch?
>>
>> For context, we had difficulty upstreaming support for instrumentation
>> based profile guided optimizations in the Linux kernel.
>> https://lore.kernel.org/lkml/CAHk-=whqCT0BeqBQhW8D-YoLLgp_eFY=8Y=9ieREM5xx0ef08w@mail.gmail.com/
>> We'd like to be able to use either instrumentation or sampling to
>> optimize our builds.  The major barrier to sample based approaches are
>> architecture / micro architecture issues with sample based profile
>> data collection, and bitrot of data processing utilities.
>> https://github.com/google/autofdo/issues/144
> 
> On existing AMD Zen2, Zen3 the following cmdline:
> $ perf record -e cycles:pp --freq=128 --call-graph lbr -- <command to profile>
> 
> does not work. I see two reasons:
> 
> 1. cycles:pp is likely converted into IBS op in cycle mode.
>     Current kernels do not support IBS in per-thread mode.
>     This is purely a kernel limitation

Right, it's purely a kernel limitation. And below simple patch on top
of event-context rewrite patch[1] should be sufficient to make cycles:pp
working in per-process mode on AMD Zen.

---
diff --git a/arch/x86/events/amd/ibs.c b/arch/x86/events/amd/ibs.c
index c251bc44c088..de01b5d27e40 100644
--- a/arch/x86/events/amd/ibs.c
+++ b/arch/x86/events/amd/ibs.c
@@ -665,7 +665,7 @@ static struct perf_ibs perf_ibs_fetch = {
 
 static struct perf_ibs perf_ibs_op = {
 	.pmu = {
-		.task_ctx_nr	= perf_invalid_context,
+		.task_ctx_nr	= perf_hw_context,
 
 		.event_init	= perf_ibs_init,
 		.add		= perf_ibs_add,
---

[1]: https://lore.kernel.org/lkml/20220829113347.295-1-ravi.bangoria@amd.com

Re: Invalid event (cycles:pp) in per-thread mode, enable system wide with '-a'.

[Nick Desaulniers]

On Thu, Oct 6, 2022 at 8:56 PM Ravi Bangoria <ravi.bangoria@amd.com> wrote:
>
> +cc: PeterZ
>
> >>>>> +Ravi who may be able to say if there are any issues with the precise
> >>>>> sampling on AMD.
> >>>>
> >>>> Afaik cvcles:pp will use IBS but it doesn't support per-task profiling
> >>>> since it has no task context.  Ravi is working on it..
> >>>
> >>> Right.
> >>> https://lore.kernel.org/lkml/20220829113347.295-1-ravi.bangoria@amd.com
> >>
> >> Cool, thanks for working on this Ravi.
> >>
> >> I'm not sure yet whether I may replace the kernel on my corporate
> >> provided workstation, so I'm not sure yet I can help test that patch.
> >>
> >> Can you confirm that
> >> $ perf record -e cycles:pp --freq=128 --call-graph lbr -- <command to profile>
> >>
> >> works with just that patch applied? Or is there more work required?
> >> What is the status of that patch?
> >>
> >> For context, we had difficulty upstreaming support for instrumentation
> >> based profile guided optimizations in the Linux kernel.
> >> https://lore.kernel.org/lkml/CAHk-=whqCT0BeqBQhW8D-YoLLgp_eFY=8Y=9ieREM5xx0ef08w@mail.gmail.com/
> >> We'd like to be able to use either instrumentation or sampling to
> >> optimize our builds.  The major barrier to sample based approaches are
> >> architecture / micro architecture issues with sample based profile
> >> data collection, and bitrot of data processing utilities.
> >> https://github.com/google/autofdo/issues/144
> >
> > On existing AMD Zen2, Zen3 the following cmdline:
> > $ perf record -e cycles:pp --freq=128 --call-graph lbr -- <command to profile>
> >
> > does not work. I see two reasons:
> >
> > 1. cycles:pp is likely converted into IBS op in cycle mode.
> >     Current kernels do not support IBS in per-thread mode.
> >     This is purely a kernel limitation
>
> Right, it's purely a kernel limitation. And below simple patch on top
> of event-context rewrite patch[1] should be sufficient to make cycles:pp
> working in per-process mode on AMD Zen.
>
> ---
> diff --git a/arch/x86/events/amd/ibs.c b/arch/x86/events/amd/ibs.c
> index c251bc44c088..de01b5d27e40 100644
> --- a/arch/x86/events/amd/ibs.c
> +++ b/arch/x86/events/amd/ibs.c
> @@ -665,7 +665,7 @@ static struct perf_ibs perf_ibs_fetch = {
>
>  static struct perf_ibs perf_ibs_op = {
>         .pmu = {
> -               .task_ctx_nr    = perf_invalid_context,
> +               .task_ctx_nr    = perf_hw_context,
>
>                 .event_init     = perf_ibs_init,
>                 .add            = perf_ibs_add,
> ---
>
> [1]: https://lore.kernel.org/lkml/20220829113347.295-1-ravi.bangoria@amd.com

Hi Ravi,
I didn't see the above diff in
https://lore.kernel.org/lkml/20221008062424.313-1-ravi.bangoria@amd.com/
Was there another distinct patch you were going to send for the above?

-- 
Thanks,
~Nick Desaulniers

Re: Invalid event (cycles:pp) in per-thread mode, enable system wide with '-a'.

[Nick Desaulniers]

On Wed, Oct 5, 2022 at 3:50 PM Stephane Eranian <eranian@google.com> wrote:
>
> On Wed, Oct 5, 2022 at 2:56 PM Nick Desaulniers <ndesaulniers@google.com> wrote:
> >
> > + Stephane, Kees, Sandipan, Bill, ClangBuiltLinux mailing list, Yonghong
> > https://www.spinics.net/lists/linux-perf-users/msg23103.html
> > starts the thread, for context.
> >
> > On Thu, Sep 29, 2022 at 9:32 PM Ravi Bangoria <ravi.bangoria@amd.com> wrote:
> > >
> > > On 30-Sep-22 9:56 AM, Namhyung Kim wrote:
> > > > Hello,
> > > >
> > > > On Thu, Sep 29, 2022 at 8:49 PM Ian Rogers <irogers@google.com> wrote:
> > > >>
> > > >> On Thu, Sep 29, 2022 at 3:10 PM Slade Watkins <srw@sladewatkins.net> wrote:
> > > >>>
> > > >>> Hey Nick,
> > > >>>
> > > >>>> On Sep 29, 2022, at 5:54 PM, Nick Desaulniers <ndesaulniers@google.com> wrote:
> > > >>>>
> > > >>>> I remember hearing rumblings about issues with zen 2, LBR, vs zen 3.
> > > >>>> Is this a known issue, or am I holding it wrong?
> > > >>>
> > > >>> Hm… I also remember this. I have a Zen 2 based system that I can do testing on, so I will do so when I’m able.
> > > >>>
> > > >>> If I discover something of note, I’ll get back to you.
> > > >>>
> > > >>> Cheers,
> > > >>> -srw
> > > >>>
> > > >>
> > > >> LBR isn't yet supported for Zen but is coming:
> > > >> https://lore.kernel.org/lkml/166155216401.401.5809694678609694438.tip-bot2@tip-bot2/
> > > >> I'd recommend frame-pointers.
> >
> > Having to recompile is less than ideal for my workflow.  I have added a note to
> > https://github.com/ClangBuiltLinux/profiling/tree/main/perf#errors
> > Please let me know how I might improve the documentation.
> >
> > > >>
> > > >> +Ravi who may be able to say if there are any issues with the precise
> > > >> sampling on AMD.
> > > >
> > > > Afaik cvcles:pp will use IBS but it doesn't support per-task profiling
> > > > since it has no task context.  Ravi is working on it..
> > >
> > > Right.
> > > https://lore.kernel.org/lkml/20220829113347.295-1-ravi.bangoria@amd.com
> >
> > Cool, thanks for working on this Ravi.
> >
> > I'm not sure yet whether I may replace the kernel on my corporate
> > provided workstation, so I'm not sure yet I can help test that patch.
> >
> > Can you confirm that
> > $ perf record -e cycles:pp --freq=128 --call-graph lbr -- <command to profile>
> >
> > works with just that patch applied? Or is there more work required?
> > What is the status of that patch?
> >
> > For context, we had difficulty upstreaming support for instrumentation
> > based profile guided optimizations in the Linux kernel.
> > https://lore.kernel.org/lkml/CAHk-=whqCT0BeqBQhW8D-YoLLgp_eFY=8Y=9ieREM5xx0ef08w@mail.gmail.com/
> > We'd like to be able to use either instrumentation or sampling to
> > optimize our builds.  The major barrier to sample based approaches are
> > architecture / micro architecture issues with sample based profile
> > data collection, and bitrot of data processing utilities.
> > https://github.com/google/autofdo/issues/144
>
> On existing AMD Zen2, Zen3 the following cmdline:
> $ perf record -e cycles:pp --freq=128 --call-graph lbr -- <command to profile>
>
> does not work. I see two reasons:
>
> 1. cycles:pp is likely converted into IBS op in cycle mode.
>     Current kernels do not support IBS in per-thread mode.
>     This is purely a kernel limitation

Sounds like Ravi has a diff that might work?
https://lore.kernel.org/linux-perf-users/85822c3c-2254-52cc-e6b1-9c89adb63771@amd.com/

>
> 2. call-graph lbr is not supported on AMD because they do
>    not have LBR and therefore no LBR callstack mode
>
> The best way to get what you want here today on AMD Zen2 and Zen3:
>
>    $ perf record -e cycles --freq=128 -g -- <command to profile>

So without recompiling to explicitly re-enable frame pointers, code
I'm looking to profile is built with -gmlt, so I get symbols, but I
can't observe callers.

Sounds like I'd need to rebuild with -fno-omit-frame-pointers which is
more painful than my prior LBR based workflow.

>
> On AMD Zen3, there is a precursor to LBR with Branch Sampling (BRS),
> and you can use it to sample taken branches but not for callstacks. I
> mention the cmdline here for reference:
>
> $ perf record -e cpu/branch-brs/ -c 1000037  -b  -- <command to profile>
>
> Note that AMD Zen3 BRS is enough to get the autoFDO usage of an
> LBR working as per the cmdline above.

Interesting, but I'm stuck with a Zen2 box for a couple years now
(corporate workstation).  This pretty much blows up all of the
profiling work I was doing, and any hope I had of contributing towards
building the kernel with AutoFDO profiles until this works on my box
with the kernel it has.

>
> Hope this helps.



-- 
Thanks,
~Nick Desaulniers

Re: Invalid event (cycles:pp) in per-thread mode, enable system wide with '-a'.

[Ravi Bangoria]

On 12-Oct-22 3:02 AM, Nick Desaulniers wrote:
> On Thu, Oct 6, 2022 at 8:56 PM Ravi Bangoria <ravi.bangoria@amd.com> wrote:
>>
>> +cc: PeterZ
>>
>>>>>>> +Ravi who may be able to say if there are any issues with the precise
>>>>>>> sampling on AMD.
>>>>>>
>>>>>> Afaik cvcles:pp will use IBS but it doesn't support per-task profiling
>>>>>> since it has no task context.  Ravi is working on it..
>>>>>
>>>>> Right.
>>>>> https://lore.kernel.org/lkml/20220829113347.295-1-ravi.bangoria@amd.com
>>>>
>>>> Cool, thanks for working on this Ravi.
>>>>
>>>> I'm not sure yet whether I may replace the kernel on my corporate
>>>> provided workstation, so I'm not sure yet I can help test that patch.
>>>>
>>>> Can you confirm that
>>>> $ perf record -e cycles:pp --freq=128 --call-graph lbr -- <command to profile>
>>>>
>>>> works with just that patch applied? Or is there more work required?
>>>> What is the status of that patch?
>>>>
>>>> For context, we had difficulty upstreaming support for instrumentation
>>>> based profile guided optimizations in the Linux kernel.
>>>> https://lore.kernel.org/lkml/CAHk-=whqCT0BeqBQhW8D-YoLLgp_eFY=8Y=9ieREM5xx0ef08w@mail.gmail.com/
>>>> We'd like to be able to use either instrumentation or sampling to
>>>> optimize our builds.  The major barrier to sample based approaches are
>>>> architecture / micro architecture issues with sample based profile
>>>> data collection, and bitrot of data processing utilities.
>>>> https://github.com/google/autofdo/issues/144
>>>
>>> On existing AMD Zen2, Zen3 the following cmdline:
>>> $ perf record -e cycles:pp --freq=128 --call-graph lbr -- <command to profile>
>>>
>>> does not work. I see two reasons:
>>>
>>> 1. cycles:pp is likely converted into IBS op in cycle mode.
>>>     Current kernels do not support IBS in per-thread mode.
>>>     This is purely a kernel limitation
>>
>> Right, it's purely a kernel limitation. And below simple patch on top
>> of event-context rewrite patch[1] should be sufficient to make cycles:pp
>> working in per-process mode on AMD Zen.
>>
>> ---
>> diff --git a/arch/x86/events/amd/ibs.c b/arch/x86/events/amd/ibs.c
>> index c251bc44c088..de01b5d27e40 100644
>> --- a/arch/x86/events/amd/ibs.c
>> +++ b/arch/x86/events/amd/ibs.c
>> @@ -665,7 +665,7 @@ static struct perf_ibs perf_ibs_fetch = {
>>
>>  static struct perf_ibs perf_ibs_op = {
>>         .pmu = {
>> -               .task_ctx_nr    = perf_invalid_context,
>> +               .task_ctx_nr    = perf_hw_context,
>>
>>                 .event_init     = perf_ibs_init,
>>                 .add            = perf_ibs_add,
>> ---
>>
>> [1]: https://lore.kernel.org/lkml/20220829113347.295-1-ravi.bangoria@amd.com
> 
> Hi Ravi,
> I didn't see the above diff in
> https://lore.kernel.org/lkml/20221008062424.313-1-ravi.bangoria@amd.com/
> Was there another distinct patch you were going to send for the above?

Yes Nick. I was planning to send it once the rewrite stuff goes in.

Thanks,
Ravi

Re: Invalid event (cycles:pp) in per-thread mode, enable system wide with '-a'.

[Ravi Bangoria]

On 12-Oct-22 9:36 AM, Ravi Bangoria wrote:
> On 12-Oct-22 3:02 AM, Nick Desaulniers wrote:
>> On Thu, Oct 6, 2022 at 8:56 PM Ravi Bangoria <ravi.bangoria@amd.com> wrote:
>>>
>>> +cc: PeterZ
>>>
>>>>>>>> +Ravi who may be able to say if there are any issues with the precise
>>>>>>>> sampling on AMD.
>>>>>>>
>>>>>>> Afaik cvcles:pp will use IBS but it doesn't support per-task profiling
>>>>>>> since it has no task context.  Ravi is working on it..
>>>>>>
>>>>>> Right.
>>>>>> https://lore.kernel.org/lkml/20220829113347.295-1-ravi.bangoria@amd.com
>>>>>
>>>>> Cool, thanks for working on this Ravi.
>>>>>
>>>>> I'm not sure yet whether I may replace the kernel on my corporate
>>>>> provided workstation, so I'm not sure yet I can help test that patch.
>>>>>
>>>>> Can you confirm that
>>>>> $ perf record -e cycles:pp --freq=128 --call-graph lbr -- <command to profile>
>>>>>
>>>>> works with just that patch applied? Or is there more work required?
>>>>> What is the status of that patch?
>>>>>
>>>>> For context, we had difficulty upstreaming support for instrumentation
>>>>> based profile guided optimizations in the Linux kernel.
>>>>> https://lore.kernel.org/lkml/CAHk-=whqCT0BeqBQhW8D-YoLLgp_eFY=8Y=9ieREM5xx0ef08w@mail.gmail.com/
>>>>> We'd like to be able to use either instrumentation or sampling to
>>>>> optimize our builds.  The major barrier to sample based approaches are
>>>>> architecture / micro architecture issues with sample based profile
>>>>> data collection, and bitrot of data processing utilities.
>>>>> https://github.com/google/autofdo/issues/144
>>>>
>>>> On existing AMD Zen2, Zen3 the following cmdline:
>>>> $ perf record -e cycles:pp --freq=128 --call-graph lbr -- <command to profile>
>>>>
>>>> does not work. I see two reasons:
>>>>
>>>> 1. cycles:pp is likely converted into IBS op in cycle mode.
>>>>     Current kernels do not support IBS in per-thread mode.
>>>>     This is purely a kernel limitation
>>>
>>> Right, it's purely a kernel limitation. And below simple patch on top
>>> of event-context rewrite patch[1] should be sufficient to make cycles:pp
>>> working in per-process mode on AMD Zen.
>>>
>>> ---
>>> diff --git a/arch/x86/events/amd/ibs.c b/arch/x86/events/amd/ibs.c
>>> index c251bc44c088..de01b5d27e40 100644
>>> --- a/arch/x86/events/amd/ibs.c
>>> +++ b/arch/x86/events/amd/ibs.c
>>> @@ -665,7 +665,7 @@ static struct perf_ibs perf_ibs_fetch = {
>>>
>>>  static struct perf_ibs perf_ibs_op = {
>>>         .pmu = {
>>> -               .task_ctx_nr    = perf_invalid_context,
>>> +               .task_ctx_nr    = perf_hw_context,
>>>
>>>                 .event_init     = perf_ibs_init,
>>>                 .add            = perf_ibs_add,
>>> ---
>>>
>>> [1]: https://lore.kernel.org/lkml/20220829113347.295-1-ravi.bangoria@amd.com
>>
>> Hi Ravi,
>> I didn't see the above diff in
>> https://lore.kernel.org/lkml/20221008062424.313-1-ravi.bangoria@amd.com/
>> Was there another distinct patch you were going to send for the above?
> 
> Yes Nick. I was planning to send it once the rewrite stuff goes in.

Hi Nick,

Since you have practical use case, would it be possible to run your workflow
with perf rewrite and IBS patches applied? It will help us in finding/fixing
more bugs and upstreaming these changes.

Thanks,
Ravi

Re: Invalid event (cycles:pp) in per-thread mode, enable system wide with '-a'.

[Nick Desaulniers]

On Tue, Oct 11, 2022 at 10:05 PM Ravi Bangoria <ravi.bangoria@amd.com> wrote:
>
> On 12-Oct-22 9:36 AM, Ravi Bangoria wrote:
> > On 12-Oct-22 3:02 AM, Nick Desaulniers wrote:
> >> On Thu, Oct 6, 2022 at 8:56 PM Ravi Bangoria <ravi.bangoria@amd.com> wrote:
> >>>
> >>> +cc: PeterZ
> >>>
> >>>>>>>> +Ravi who may be able to say if there are any issues with the precise
> >>>>>>>> sampling on AMD.
> >>>>>>>
> >>>>>>> Afaik cvcles:pp will use IBS but it doesn't support per-task profiling
> >>>>>>> since it has no task context.  Ravi is working on it..
> >>>>>>
> >>>>>> Right.
> >>>>>> https://lore.kernel.org/lkml/20220829113347.295-1-ravi.bangoria@amd.com
> >>>>>
> >>>>> Cool, thanks for working on this Ravi.
> >>>>>
> >>>>> I'm not sure yet whether I may replace the kernel on my corporate
> >>>>> provided workstation, so I'm not sure yet I can help test that patch.
> >>>>>
> >>>>> Can you confirm that
> >>>>> $ perf record -e cycles:pp --freq=128 --call-graph lbr -- <command to profile>
> >>>>>
> >>>>> works with just that patch applied? Or is there more work required?
> >>>>> What is the status of that patch?
> >>>>>
> >>>>> For context, we had difficulty upstreaming support for instrumentation
> >>>>> based profile guided optimizations in the Linux kernel.
> >>>>> https://lore.kernel.org/lkml/CAHk-=whqCT0BeqBQhW8D-YoLLgp_eFY=8Y=9ieREM5xx0ef08w@mail.gmail.com/
> >>>>> We'd like to be able to use either instrumentation or sampling to
> >>>>> optimize our builds.  The major barrier to sample based approaches are
> >>>>> architecture / micro architecture issues with sample based profile
> >>>>> data collection, and bitrot of data processing utilities.
> >>>>> https://github.com/google/autofdo/issues/144
> >>>>
> >>>> On existing AMD Zen2, Zen3 the following cmdline:
> >>>> $ perf record -e cycles:pp --freq=128 --call-graph lbr -- <command to profile>
> >>>>
> >>>> does not work. I see two reasons:
> >>>>
> >>>> 1. cycles:pp is likely converted into IBS op in cycle mode.
> >>>>     Current kernels do not support IBS in per-thread mode.
> >>>>     This is purely a kernel limitation
> >>>
> >>> Right, it's purely a kernel limitation. And below simple patch on top
> >>> of event-context rewrite patch[1] should be sufficient to make cycles:pp
> >>> working in per-process mode on AMD Zen.
> >>>
> >>> ---
> >>> diff --git a/arch/x86/events/amd/ibs.c b/arch/x86/events/amd/ibs.c
> >>> index c251bc44c088..de01b5d27e40 100644
> >>> --- a/arch/x86/events/amd/ibs.c
> >>> +++ b/arch/x86/events/amd/ibs.c
> >>> @@ -665,7 +665,7 @@ static struct perf_ibs perf_ibs_fetch = {
> >>>
> >>>  static struct perf_ibs perf_ibs_op = {
> >>>         .pmu = {
> >>> -               .task_ctx_nr    = perf_invalid_context,
> >>> +               .task_ctx_nr    = perf_hw_context,
> >>>
> >>>                 .event_init     = perf_ibs_init,
> >>>                 .add            = perf_ibs_add,
> >>> ---
> >>>
> >>> [1]: https://lore.kernel.org/lkml/20220829113347.295-1-ravi.bangoria@amd.com
> >>
> >> Hi Ravi,
> >> I didn't see the above diff in
> >> https://lore.kernel.org/lkml/20221008062424.313-1-ravi.bangoria@amd.com/
> >> Was there another distinct patch you were going to send for the above?
> >
> > Yes Nick. I was planning to send it once the rewrite stuff goes in.
>
> Hi Nick,
>
> Since you have practical use case, would it be possible to run your workflow
> with perf rewrite and IBS patches applied? It will help us in finding/fixing
> more bugs and upstreaming these changes.

Hi Ravi,
Sorry, I'm not able to load a custom kernel image on my employer
provided workstation, and I never got approval to expense hardware for
testing this otherwise.

Was there ever any update on this? I'm on 6.1.25 now and still cant run
$ perf record -e cycles:pp --call-graph lbr <any command to profile>
$ cat /proc/cpuinfo
...
model name      : AMD Ryzen Threadripper PRO 3995WX 64-Cores
...
-- 
Thanks,
~Nick Desaulniers

Re: Invalid event (cycles:pp) in per-thread mode, enable system wide with '-a'.

[Namhyung Kim]

Hi Nick,

On Fri, Jun 23, 2023 at 9:23 AM Nick Desaulniers
<ndesaulniers@google.com> wrote:
>
> On Tue, Oct 11, 2022 at 10:05 PM Ravi Bangoria <ravi.bangoria@amd.com> wrote:
> >
> > On 12-Oct-22 9:36 AM, Ravi Bangoria wrote:
> > > On 12-Oct-22 3:02 AM, Nick Desaulniers wrote:
> > >> On Thu, Oct 6, 2022 at 8:56 PM Ravi Bangoria <ravi.bangoria@amd.com> wrote:
> > >>>
> > >>> +cc: PeterZ
> > >>>
> > >>>>>>>> +Ravi who may be able to say if there are any issues with the precise
> > >>>>>>>> sampling on AMD.
> > >>>>>>>
> > >>>>>>> Afaik cvcles:pp will use IBS but it doesn't support per-task profiling
> > >>>>>>> since it has no task context.  Ravi is working on it..
> > >>>>>>
> > >>>>>> Right.
> > >>>>>> https://lore.kernel.org/lkml/20220829113347.295-1-ravi.bangoria@amd.com
> > >>>>>
> > >>>>> Cool, thanks for working on this Ravi.
> > >>>>>
> > >>>>> I'm not sure yet whether I may replace the kernel on my corporate
> > >>>>> provided workstation, so I'm not sure yet I can help test that patch.
> > >>>>>
> > >>>>> Can you confirm that
> > >>>>> $ perf record -e cycles:pp --freq=128 --call-graph lbr -- <command to profile>
> > >>>>>
> > >>>>> works with just that patch applied? Or is there more work required?
> > >>>>> What is the status of that patch?
> > >>>>>
> > >>>>> For context, we had difficulty upstreaming support for instrumentation
> > >>>>> based profile guided optimizations in the Linux kernel.
> > >>>>> https://lore.kernel.org/lkml/CAHk-=whqCT0BeqBQhW8D-YoLLgp_eFY=8Y=9ieREM5xx0ef08w@mail.gmail.com/
> > >>>>> We'd like to be able to use either instrumentation or sampling to
> > >>>>> optimize our builds.  The major barrier to sample based approaches are
> > >>>>> architecture / micro architecture issues with sample based profile
> > >>>>> data collection, and bitrot of data processing utilities.
> > >>>>> https://github.com/google/autofdo/issues/144
> > >>>>
> > >>>> On existing AMD Zen2, Zen3 the following cmdline:
> > >>>> $ perf record -e cycles:pp --freq=128 --call-graph lbr -- <command to profile>
> > >>>>
> > >>>> does not work. I see two reasons:
> > >>>>
> > >>>> 1. cycles:pp is likely converted into IBS op in cycle mode.
> > >>>>     Current kernels do not support IBS in per-thread mode.
> > >>>>     This is purely a kernel limitation
> > >>>
> > >>> Right, it's purely a kernel limitation. And below simple patch on top
> > >>> of event-context rewrite patch[1] should be sufficient to make cycles:pp
> > >>> working in per-process mode on AMD Zen.
> > >>>
> > >>> ---
> > >>> diff --git a/arch/x86/events/amd/ibs.c b/arch/x86/events/amd/ibs.c
> > >>> index c251bc44c088..de01b5d27e40 100644
> > >>> --- a/arch/x86/events/amd/ibs.c
> > >>> +++ b/arch/x86/events/amd/ibs.c
> > >>> @@ -665,7 +665,7 @@ static struct perf_ibs perf_ibs_fetch = {
> > >>>
> > >>>  static struct perf_ibs perf_ibs_op = {
> > >>>         .pmu = {
> > >>> -               .task_ctx_nr    = perf_invalid_context,
> > >>> +               .task_ctx_nr    = perf_hw_context,
> > >>>
> > >>>                 .event_init     = perf_ibs_init,
> > >>>                 .add            = perf_ibs_add,
> > >>> ---
> > >>>
> > >>> [1]: https://lore.kernel.org/lkml/20220829113347.295-1-ravi.bangoria@amd.com
> > >>
> > >> Hi Ravi,
> > >> I didn't see the above diff in
> > >> https://lore.kernel.org/lkml/20221008062424.313-1-ravi.bangoria@amd.com/
> > >> Was there another distinct patch you were going to send for the above?
> > >
> > > Yes Nick. I was planning to send it once the rewrite stuff goes in.
> >
> > Hi Nick,
> >
> > Since you have practical use case, would it be possible to run your workflow
> > with perf rewrite and IBS patches applied? It will help us in finding/fixing
> > more bugs and upstreaming these changes.
>
> Hi Ravi,
> Sorry, I'm not able to load a custom kernel image on my employer
> provided workstation, and I never got approval to expense hardware for
> testing this otherwise.
>
> Was there ever any update on this? I'm on 6.1.25 now and still cant run
> $ perf record -e cycles:pp --call-graph lbr <any command to profile>
> $ cat /proc/cpuinfo
> ...
> model name      : AMD Ryzen Threadripper PRO 3995WX 64-Cores
> ...

The commit 30093056f7b2 ("perf/amd/ibs: Make IBS a core pmu") in v6.2.

  $ git name-rev --tags --refs=v[2-6].* 30093056f7b2
  30093056f7b2 v6.2-rc1~176^2~16

https://git.kernel.org/torvalds/c/30093056f7b2

Thanks,
Namhyung

Re: Invalid event (cycles:pp) in per-thread mode, enable system wide with '-a'.

[Ravi Bangoria]

Hi Nick,

On 23-Jun-23 9:53 PM, Nick Desaulniers wrote:
> On Tue, Oct 11, 2022 at 10:05 PM Ravi Bangoria <ravi.bangoria@amd.com> wrote:
>>
>> On 12-Oct-22 9:36 AM, Ravi Bangoria wrote:
>>> On 12-Oct-22 3:02 AM, Nick Desaulniers wrote:
>>>> On Thu, Oct 6, 2022 at 8:56 PM Ravi Bangoria <ravi.bangoria@amd.com> wrote:
>>>>>
>>>>> +cc: PeterZ
>>>>>
>>>>>>>>>> +Ravi who may be able to say if there are any issues with the precise
>>>>>>>>>> sampling on AMD.
>>>>>>>>>
>>>>>>>>> Afaik cvcles:pp will use IBS but it doesn't support per-task profiling
>>>>>>>>> since it has no task context.  Ravi is working on it..
>>>>>>>>
>>>>>>>> Right.
>>>>>>>> https://lore.kernel.org/lkml/20220829113347.295-1-ravi.bangoria@amd.com
>>>>>>>
>>>>>>> Cool, thanks for working on this Ravi.
>>>>>>>
>>>>>>> I'm not sure yet whether I may replace the kernel on my corporate
>>>>>>> provided workstation, so I'm not sure yet I can help test that patch.
>>>>>>>
>>>>>>> Can you confirm that
>>>>>>> $ perf record -e cycles:pp --freq=128 --call-graph lbr -- <command to profile>
>>>>>>>
>>>>>>> works with just that patch applied? Or is there more work required?
>>>>>>> What is the status of that patch?
>>>>>>>
>>>>>>> For context, we had difficulty upstreaming support for instrumentation
>>>>>>> based profile guided optimizations in the Linux kernel.
>>>>>>> https://lore.kernel.org/lkml/CAHk-=whqCT0BeqBQhW8D-YoLLgp_eFY=8Y=9ieREM5xx0ef08w@mail.gmail.com/
>>>>>>> We'd like to be able to use either instrumentation or sampling to
>>>>>>> optimize our builds.  The major barrier to sample based approaches are
>>>>>>> architecture / micro architecture issues with sample based profile
>>>>>>> data collection, and bitrot of data processing utilities.
>>>>>>> https://github.com/google/autofdo/issues/144
>>>>>>
>>>>>> On existing AMD Zen2, Zen3 the following cmdline:
>>>>>> $ perf record -e cycles:pp --freq=128 --call-graph lbr -- <command to profile>
>>>>>>
>>>>>> does not work. I see two reasons:
>>>>>>
>>>>>> 1. cycles:pp is likely converted into IBS op in cycle mode.
>>>>>>     Current kernels do not support IBS in per-thread mode.
>>>>>>     This is purely a kernel limitation
>>>>>
>>>>> Right, it's purely a kernel limitation. And below simple patch on top
>>>>> of event-context rewrite patch[1] should be sufficient to make cycles:pp
>>>>> working in per-process mode on AMD Zen.
>>>>>
>>>>> ---
>>>>> diff --git a/arch/x86/events/amd/ibs.c b/arch/x86/events/amd/ibs.c
>>>>> index c251bc44c088..de01b5d27e40 100644
>>>>> --- a/arch/x86/events/amd/ibs.c
>>>>> +++ b/arch/x86/events/amd/ibs.c
>>>>> @@ -665,7 +665,7 @@ static struct perf_ibs perf_ibs_fetch = {
>>>>>
>>>>>  static struct perf_ibs perf_ibs_op = {
>>>>>         .pmu = {
>>>>> -               .task_ctx_nr    = perf_invalid_context,
>>>>> +               .task_ctx_nr    = perf_hw_context,
>>>>>
>>>>>                 .event_init     = perf_ibs_init,
>>>>>                 .add            = perf_ibs_add,
>>>>> ---
>>>>>
>>>>> [1]: https://lore.kernel.org/lkml/20220829113347.295-1-ravi.bangoria@amd.com
>>>>
>>>> Hi Ravi,
>>>> I didn't see the above diff in
>>>> https://lore.kernel.org/lkml/20221008062424.313-1-ravi.bangoria@amd.com/
>>>> Was there another distinct patch you were going to send for the above?
>>>
>>> Yes Nick. I was planning to send it once the rewrite stuff goes in.
>>
>> Hi Nick,
>>
>> Since you have practical use case, would it be possible to run your workflow
>> with perf rewrite and IBS patches applied? It will help us in finding/fixing
>> more bugs and upstreaming these changes.
> 
> Hi Ravi,
> Sorry, I'm not able to load a custom kernel image on my employer
> provided workstation, and I never got approval to expense hardware for
> testing this otherwise.
> 
> Was there ever any update on this? I'm on 6.1.25 now and still cant run
> $ perf record -e cycles:pp --call-graph lbr <any command to profile>

Per-process precise sampling on AMD platforms should work from 6.2-rc1
onward. However, --call-graph=lbr is not supported on AMD (hw limitation).

Thanks,
Ravi

Re: Invalid event (cycles:pp) in per-thread mode, enable system wide with '-a'.

[Nick Desaulniers]

On Sun, Jun 25, 2023 at 10:45 PM Ravi Bangoria <ravi.bangoria@amd.com> wrote:
>
> Hi Nick,
>
> On 23-Jun-23 9:53 PM, Nick Desaulniers wrote:
> >>> On 12-Oct-22 3:02 AM, Nick Desaulniers wrote:
> > Hi Ravi,
> > Sorry, I'm not able to load a custom kernel image on my employer
> > provided workstation, and I never got approval to expense hardware for
> > testing this otherwise.
> >
> > Was there ever any update on this? I'm on 6.1.25 now and still cant run
> > $ perf record -e cycles:pp --call-graph lbr <any command to profile>
>
> Per-process precise sampling on AMD platforms should work from 6.2-rc1
> onward.

Ok, I can wait for my employer to ship 6.2 on our workstations.

> However, --call-graph=lbr is not supported on AMD (hw limitation).

On any AMD uarches? Is there an equivalent? LBR encoding is compact
which makes working it much faster than DWARF or stack frame
unwinding.
-- 
Thanks,
~Nick Desaulniers

Re: Invalid event (cycles:pp) in per-thread mode, enable system wide with '-a'.

[Ravi Bangoria]

On 11-Jul-23 2:52 AM, Nick Desaulniers wrote:
> On Sun, Jun 25, 2023 at 10:45 PM Ravi Bangoria <ravi.bangoria@amd.com> wrote:
>>
>> Hi Nick,
>>
>> On 23-Jun-23 9:53 PM, Nick Desaulniers wrote:
>>>>> On 12-Oct-22 3:02 AM, Nick Desaulniers wrote:
>>> Hi Ravi,
>>> Sorry, I'm not able to load a custom kernel image on my employer
>>> provided workstation, and I never got approval to expense hardware for
>>> testing this otherwise.
>>>
>>> Was there ever any update on this? I'm on 6.1.25 now and still cant run
>>> $ perf record -e cycles:pp --call-graph lbr <any command to profile>
>>
>> Per-process precise sampling on AMD platforms should work from 6.2-rc1
>> onward.
> 
> Ok, I can wait for my employer to ship 6.2 on our workstations.
> 
>> However, --call-graph=lbr is not supported on AMD (hw limitation).
> 
> On any AMD uarches? Is there an equivalent? LBR encoding is compact
> which makes working it much faster than DWARF or stack frame
> unwinding.

I understand that LBR call-stack is the fastest option but unfortunately
none of the current AMD uarch supports it.

Thanks,
Ravi