Linux Foundation accepted for Google Summer-of-Code 2021

Linux Foundation accepted for Google Summer-of-Code 2021

[Ian Rogers]

Hi,

I'm happy to share that the Linux Foundation has been accepted into
the Google Summer-of-Code 2021 and we are looking for students
interested in profiling, perf, etc. to apply to do projects.

The main details on the Linux foundation application are here:
https://wiki.linuxfoundation.org/gsoc/google-summer-code-2021

Details on potential projects with perf here:
https://wiki.linuxfoundation.org/gsoc/2021-gsoc-perf

Please help to share, especially to students.

Additional project ideas, mentors and help is also appreciated.

Thanks,
Ian

Re: Linux Foundation accepted for Google Summer-of-Code 2021

[Milian Wolff]

[-- Attachment #1: Type: text/plain, Size: 1495 bytes --]

On Mittwoch, 10. März 2021 17:32:25 CET Ian Rogers wrote:
> Hi,
> 
> I'm happy to share that the Linux Foundation has been accepted into
> the Google Summer-of-Code 2021 and we are looking for students
> interested in profiling, perf, etc. to apply to do projects.
> 
> The main details on the Linux foundation application are here:
> https://wiki.linuxfoundation.org/gsoc/google-summer-code-2021
> 
> Details on potential projects with perf here:
> https://wiki.linuxfoundation.org/gsoc/2021-gsoc-perf
> 
> Please help to share, especially to students.
> 
> Additional project ideas, mentors and help is also appreciated.

As the maintainer of hotspot [1], I'd be happy to discuss how we could reuse 
the prior art in providing a GUI around `perf report` and more. Considering 
that hotspot is already a pretty successfull project, I wonder why we have to 
reinvent the wheel to provide a gtk-based GUI - is it really just the 
framework choice - Qt vs. GTK? Or are there other reasons?

[1]: https://github.com/KDAB/hotspot/

Generally, I would love to see a proper library provided by perf that could be 
used to do the `perf report` and/or `perf annotate` data processing.

As I said - I'm happy to discuss this topic in more detail with anyone 
interested in this topic.

Cheers

-- 
Milian Wolff | milian.wolff@kdab.com | Senior Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5272 bytes --]

Re: Linux Foundation accepted for Google Summer-of-Code 2021

[Ian Rogers]

Hi Milian,

Thanks for the email and hotspot is a very interesting project!

On Sat, Mar 13, 2021 at 8:25 AM Milian Wolff <milian.wolff@kdab.com> wrote:
>
> On Mittwoch, 10. März 2021 17:32:25 CET Ian Rogers wrote:
> > Hi,
> >
> > I'm happy to share that the Linux Foundation has been accepted into
> > the Google Summer-of-Code 2021 and we are looking for students
> > interested in profiling, perf, etc. to apply to do projects.
> >
> > The main details on the Linux foundation application are here:
> > https://wiki.linuxfoundation.org/gsoc/google-summer-code-2021
> >
> > Details on potential projects with perf here:
> > https://wiki.linuxfoundation.org/gsoc/2021-gsoc-perf
> >
> > Please help to share, especially to students.
> >
> > Additional project ideas, mentors and help is also appreciated.
>
> As the maintainer of hotspot [1], I'd be happy to discuss how we could reuse
> the prior art in providing a GUI around `perf report` and more. Considering
> that hotspot is already a pretty successfull project, I wonder why we have to
> reinvent the wheel to provide a gtk-based GUI - is it really just the
> framework choice - Qt vs. GTK? Or are there other reasons?

There are a few issues:
 - there's an observation that the gtk support in perf was bitrotting.
Having a student project in this area would hopefully improve this
situation.
 - commands like perf report even in text mode are slow, single
threaded, etc. Again, an area where a student project can help out.
 - wrt UI there is also pprof (https://github.com/google/pprof) which
I think a web UI is interesting beyond just Qt, GTK, etc. Having the
code be within the perf tool means that testing can be automated, it
is easy for shared contributions, the kernel and user APIs aren't
separated by projects..

> [1]: https://github.com/KDAB/hotspot/
>
> Generally, I would love to see a proper library provided by perf that could be
> used to do the `perf report` and/or `perf annotate` data processing.

There is a library interface developing:
https://github.com/torvalds/linux/tree/master/tools/lib/perf

but it is mainly focussed on say 'perf record' than 'perf report'.
Perhaps you could mentor a project to refactor the report APIs, which
could be integrated with hotspot? If so, we can update the GSoC wiki.

It'd be nice to refactor things like event parsing into a library, in
part to make unit and fuzz testing easier. The event sorting is a big
source of perf's slowness and something that should be reasonable to
parallelize, an issue here is a lack of a thread pool and other
abstractions. These dependencies across projects can be something that
is tricky.

> As I said - I'm happy to discuss this topic in more detail with anyone
> interested in this topic.

I'm interested! There is an IRC channel #perf see www.oftc.net, but
I'd suggest the mailing list is better for discussion. I know that BPF
is doing regular office hours. I could imagine that similar for perf
would be useful, but I also know that maintainers are busy folks.

Thanks!
Ian

> Cheers
>
> --
> Milian Wolff | milian.wolff@kdab.com | Senior Software Engineer
> KDAB (Deutschland) GmbH, a KDAB Group company
> Tel: +49-30-521325470
> KDAB - The Qt, C++ and OpenGL Experts

Re: Linux Foundation accepted for Google Summer-of-Code 2021

[Milian Wolff]

[-- Attachment #1: Type: text/plain, Size: 6122 bytes --]

On Sonntag, 14. März 2021 18:29:09 CET Ian Rogers wrote:
> Hi Milian,
> 
> Thanks for the email and hotspot is a very interesting project!
> 
> On Sat, Mar 13, 2021 at 8:25 AM Milian Wolff <milian.wolff@kdab.com> wrote:
> > On Mittwoch, 10. März 2021 17:32:25 CET Ian Rogers wrote:
> > > Hi,
> > > 
> > > I'm happy to share that the Linux Foundation has been accepted into
> > > the Google Summer-of-Code 2021 and we are looking for students
> > > interested in profiling, perf, etc. to apply to do projects.
> > > 
> > > The main details on the Linux foundation application are here:
> > > https://wiki.linuxfoundation.org/gsoc/google-summer-code-2021
> > > 
> > > Details on potential projects with perf here:
> > > https://wiki.linuxfoundation.org/gsoc/2021-gsoc-perf
> > > 
> > > Please help to share, especially to students.
> > > 
> > > Additional project ideas, mentors and help is also appreciated.
> > 
> > As the maintainer of hotspot [1], I'd be happy to discuss how we could
> > reuse the prior art in providing a GUI around `perf report` and more.
> > Considering that hotspot is already a pretty successfull project, I
> > wonder why we have to reinvent the wheel to provide a gtk-based GUI - is
> > it really just the framework choice - Qt vs. GTK? Or are there other
> > reasons?
> 
> There are a few issues:
>  - there's an observation that the gtk support in perf was bitrotting.
> Having a student project in this area would hopefully improve this
> situation.

Agreed.

>  - commands like perf report even in text mode are slow, single
> threaded, etc. Again, an area where a student project can help out.

Again, agreed. That is actually something that I've spent a considerable 
amount of time on. So far, I mostly concentrated on better caching - something 
which has had very significant effects on the performance of hotspot for some 
perf.data workloads. See e.g.: https://www.kdab.com/hotspot-version-1-3-0-released/

Threading wise, in hotspot, we have basically two threads - one in the 
background process perfparser which does unwinding and symbolication. Then 
another one in the hotspot GUI process, which does the data aggregation for 
the various views. So far, I shied away from trying to further parallelize the 
former part, as unwinding and symbolication are both heavily dependend on the 
order in which events are processed. I.e. we have to make sure to process mmap 
events and the like before we process a sampling event. Additionally, making 
caching efficient is much easier in a singlethreaded context, which was so far 
much more important to get right.

>  - wrt UI there is also pprof (https://github.com/google/pprof) which
> I think a web UI is interesting beyond just Qt, GTK, etc. Having the
> code be within the perf tool means that testing can be automated, it
> is easy for shared contributions, the kernel and user APIs aren't
> separated by projects..

Yes, nowadays there are tons of pretty good UI options out there - esp. in the 
web context. I have not used pprof, but heard a lot of praise for the Mozilla 
profiler too: https://profiler.firefox.com/

And then there's catapult which is integrated into chromium or Perfetto which 
is afaik used for Android? The list probably goes on and on ;-)

But in the end it would be extremely useful if perf could directly create a 
pre-processed, compressed and portable result file with some command. That 
could in turn use some easy-to-use binary format (CBOR, protobof, whatever) 
which could then be parsed by the GUI instead of it having to reinvent the 
wheel there.

> > Generally, I would love to see a proper library provided by perf that
> > could be used to do the `perf report` and/or `perf annotate` data
> > processing.
>
> There is a library interface developing:
> https://github.com/torvalds/linux/tree/master/tools/lib/perf

Ah, that's very interesting - that didn't exist back when we started 
perfparser/hotspot. 
 
> But it is mainly focussed on say 'perf record' than 'perf report'.
> Perhaps you could mentor a project to refactor the report APIs, which
> could be integrated with hotspot? If so, we can update the GSoC wiki.

While I'm honored by the invitation, I'm afraid that I don't have the capacity 
to fulfil this role. I've attended GSOC myself in the past, and even mentored 
a couple of times, so I know that doing a good job there as a mentor requires 
more time than I can spare currently :(

That said, I am very willing to be available for discussion as an interested 
party.

> It'd be nice to refactor things like event parsing into a library, in
> part to make unit and fuzz testing easier. The event sorting is a big
> source of perf's slowness and something that should be reasonable to
> parallelize, an issue here is a lack of a thread pool and other
> abstractions. These dependencies across projects can be something that
> is tricky.

In our case, sorting isn't the bottleneck at all. But it does make 
parallelization harder as I mentioned above. Unwinding and symbolication is 
usually what's much much slower compared to just sorting a couple of events 
in-memory.

Regarding missing abstractions: I'm a C++ developer myself, and having easy 
access to more abstractions like hash maps and more has helped tremendously in 
improving the performance of our perf.data parser too. So yes, I again totally 
agree with you in that regards.

> > As I said - I'm happy to discuss this topic in more detail with anyone
> > interested in this topic.
> 
> I'm interested! There is an IRC channel #perf see www.oftc.net, but
> I'd suggest the mailing list is better for discussion. I know that BPF
> is doing regular office hours. I could imagine that similar for perf
> would be useful, but I also know that maintainers are busy folks.

I've joined that channel now.

Cheers
-- 
Milian Wolff | milian.wolff@kdab.com | Senior Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5272 bytes --]