* [RFC] Adding Support for Firefox's Gecko Profile Format @ 2023-05-24 19:25 Anup Sharma 2023-05-25 20:52 ` Namhyung Kim 0 siblings, 1 reply; 21+ messages in thread From: Anup Sharma @ 2023-05-24 19:25 UTC (permalink / raw) To: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Mark Rutland, Alexander Shishkin, Jiri Olsa, Namhyung Kim, Ian Rogers, Adrian Hunter, linux-perf-users, linux-kernel Hello everyone, I'm happy to share that I'll be working on adding support for Firefox's Gecko profile format. This format is utilized by the Firefox profiler, which is a powerful tool for performance analysis and debugging. By enhancing the perf data command to generate perf.data files in the Gecko profile format, it will allow us to leverage the capabilities of the Firefox profiler for visualizing and analyzing the performance data. As a starter task, I have created a test for testing the perf data JSON converter command. I'm also looking for a few more starter tasks related to this project. I would greatly appreciate your advice and guidance. In my effort to identify any existing bugs, I have enabled flags like 'fsanitize=address' to detect potential issues but have not found any :). Additionally, I am running perf data commands to ensure that all use cases are handled properly. I have one question regarding the installation process. Typically, I navigate to the perf directory, run 'make', and then cp 'perf' to '/usr/bin'. However, I noticed that by default, perf is installed in the '~/bin/perf' directory. Could someone please clarify why this is the case? Furthermore, I would like to know how all of you compile the perf tree. Thank you for your support and I'm looking forward to collaborating with you on this project! Regards, Anup ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] Adding Support for Firefox's Gecko Profile Format 2023-05-24 19:25 [RFC] Adding Support for Firefox's Gecko Profile Format Anup Sharma @ 2023-05-25 20:52 ` Namhyung Kim 2023-05-26 15:17 ` Ian Rogers 0 siblings, 1 reply; 21+ messages in thread From: Namhyung Kim @ 2023-05-25 20:52 UTC (permalink / raw) To: Anup Sharma Cc: Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Mark Rutland, Alexander Shishkin, Jiri Olsa, Ian Rogers, Adrian Hunter, linux-perf-users, linux-kernel Hi Anup, On Wed, May 24, 2023 at 12:25 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > Hello everyone, > > I'm happy to share that I'll be working on adding support for Firefox's Gecko profile format. > This format is utilized by the Firefox profiler, which is a powerful tool for performance analysis > and debugging. By enhancing the perf data command to generate perf.data files in the Gecko > profile format, it will allow us to leverage the capabilities of the Firefox profiler for visualizing > and analyzing the performance data. > > As a starter task, I have created a test for testing the perf data JSON converter command. > I'm also looking for a few more starter tasks related to this project. I would greatly appreciate > your advice and guidance. > > In my effort to identify any existing bugs, I have enabled flags like 'fsanitize=address' to detect > potential issues but have not found any :). Additionally, I am running perf data commands to ensure > that all use cases are handled properly. Great, good to know it works well with asan for the basic use cases at least. > > I have one question regarding the installation process. Typically, I navigate to the perf directory, > run 'make', and then cp 'perf' to '/usr/bin'. However, I noticed that by default, perf is installed in > the '~/bin/perf' directory. Could someone please clarify why this is the case? Furthermore, I would > like to know how all of you compile the perf tree. I guess $HOME is the default prefix unless you set it to other, then make install will put the binary there. You can make sure if your PATH contains the ~/bin and use it. But it's also possible you can run the perf without installing. I have a symlink in my tmp directory to point to the recent build of the binary and use it always. :) To build, you can either 'cd tools/perf; make' or 'make -C tools/perf' in the top level linux source tree. I also pass "BUILD_BPF_SKEL=1" to enable BPF. Thanks, Namhyung > > Thank you for your support and I'm looking forward to collaborating with you on this project! ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] Adding Support for Firefox's Gecko Profile Format 2023-05-25 20:52 ` Namhyung Kim @ 2023-05-26 15:17 ` Ian Rogers 2023-05-26 19:49 ` Anup Sharma 0 siblings, 1 reply; 21+ messages in thread From: Ian Rogers @ 2023-05-26 15:17 UTC (permalink / raw) To: Namhyung Kim Cc: Anup Sharma, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Mark Rutland, Alexander Shishkin, Jiri Olsa, Adrian Hunter, linux-perf-users, linux-kernel On Thu, May 25, 2023 at 1:53 PM Namhyung Kim <namhyung@kernel.org> wrote: > > Hi Anup, > > On Wed, May 24, 2023 at 12:25 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > > > Hello everyone, > > > > I'm happy to share that I'll be working on adding support for Firefox's Gecko profile format. > > This format is utilized by the Firefox profiler, which is a powerful tool for performance analysis > > and debugging. By enhancing the perf data command to generate perf.data files in the Gecko > > profile format, it will allow us to leverage the capabilities of the Firefox profiler for visualizing > > and analyzing the performance data. > > > > As a starter task, I have created a test for testing the perf data JSON converter command. > > I'm also looking for a few more starter tasks related to this project. I would greatly appreciate > > your advice and guidance. > > > > In my effort to identify any existing bugs, I have enabled flags like 'fsanitize=address' to detect > > potential issues but have not found any :). Additionally, I am running perf data commands to ensure > > that all use cases are handled properly. > > Great, good to know it works well with asan for the basic use cases at least. > > > > > I have one question regarding the installation process. Typically, I navigate to the perf directory, > > run 'make', and then cp 'perf' to '/usr/bin'. However, I noticed that by default, perf is installed in > > the '~/bin/perf' directory. Could someone please clarify why this is the case? Furthermore, I would > > like to know how all of you compile the perf tree. > > I guess $HOME is the default prefix unless you set it to other, then > make install will put > the binary there. You can make sure if your PATH contains the ~/bin and use it. > > But it's also possible you can run the perf without installing. I > have a symlink in > my tmp directory to point to the recent build of the binary and use it > always. :) > To build, you can either 'cd tools/perf; make' or 'make -C tools/perf' > in the top > level linux source tree. I also pass "BUILD_BPF_SKEL=1" to enable BPF. > > Thanks, > Namhyung I quite often test with address sanitizer, I do this by passing to make: DEBUG=1 EXTRA_CFLAGS="-O0 -g -fno-omit-frame- pointer -fsanitize=address" NO_LIBTRACEEVENT=1 The libtraceevent exclusion is to avoid false address sanitizer warnings in libtraceevent (it wasn't compiled with address sanitizer). The other flags are to make the code easier to debug. A good place to start for a description of the build flags is Makefile.perf: https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/Makefile.perf?h=perf-tools-next There's also some description here: https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/Documentation/Build.txt?h=perf-tools-next Perhaps you can suggest improvements :-) Thanks, Ian > > > > Thank you for your support and I'm looking forward to collaborating with you on this project! ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] Adding Support for Firefox's Gecko Profile Format 2023-05-26 15:17 ` Ian Rogers @ 2023-05-26 19:49 ` Anup Sharma 2023-06-01 20:49 ` Anup Sharma 0 siblings, 1 reply; 21+ messages in thread From: Anup Sharma @ 2023-05-26 19:49 UTC (permalink / raw) To: Ian Rogers Cc: Namhyung Kim, Anup Sharma, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Mark Rutland, Alexander Shishkin, Jiri Olsa, Adrian Hunter, linux-perf-users, linux-kernel On Fri, May 26, 2023 at 08:17:44AM -0700, Ian Rogers wrote: > On Thu, May 25, 2023 at 1:53 PM Namhyung Kim <namhyung@kernel.org> wrote: > > > > Hi Anup, > > > > On Wed, May 24, 2023 at 12:25 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > > > > > Hello everyone, > > > > > > I'm happy to share that I'll be working on adding support for Firefox's Gecko profile format. > > > This format is utilized by the Firefox profiler, which is a powerful tool for performance analysis > > > and debugging. By enhancing the perf data command to generate perf.data files in the Gecko > > > profile format, it will allow us to leverage the capabilities of the Firefox profiler for visualizing > > > and analyzing the performance data. > > > > > > As a starter task, I have created a test for testing the perf data JSON converter command. > > > I'm also looking for a few more starter tasks related to this project. I would greatly appreciate > > > your advice and guidance. > > > > > > In my effort to identify any existing bugs, I have enabled flags like 'fsanitize=address' to detect > > > potential issues but have not found any :). Additionally, I am running perf data commands to ensure > > > that all use cases are handled properly. > > > > Great, good to know it works well with asan for the basic use cases at least. > > > > > > > > I have one question regarding the installation process. Typically, I navigate to the perf directory, > > > run 'make', and then cp 'perf' to '/usr/bin'. However, I noticed that by default, perf is installed in > > > the '~/bin/perf' directory. Could someone please clarify why this is the case? Furthermore, I would > > > like to know how all of you compile the perf tree. > > > > I guess $HOME is the default prefix unless you set it to other, then > > make install will put > > the binary there. You can make sure if your PATH contains the ~/bin and use it. > > > > But it's also possible you can run the perf without installing. I > > have a symlink in > > my tmp directory to point to the recent build of the binary and use it > > always. :) > > To build, you can either 'cd tools/perf; make' or 'make -C tools/perf' > > in the top > > level linux source tree. I also pass "BUILD_BPF_SKEL=1" to enable BPF. > > > > Thanks, > > Namhyung > > I quite often test with address sanitizer, I do this by passing to make: > DEBUG=1 EXTRA_CFLAGS="-O0 -g -fno-omit-frame- pointer > -fsanitize=address" NO_LIBTRACEEVENT=1 > > The libtraceevent exclusion is to avoid false address sanitizer > warnings in libtraceevent (it wasn't compiled with address sanitizer). > The other flags are to make the code easier to debug. A good place to > start for a description of the build flags is Makefile.perf: > https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/Makefile.perf?h=perf-tools-next > > There's also some description here: > https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/Documentation/Build.txt?h=perf-tools-next > > Perhaps you can suggest improvements :-) > Thanks Namhyung and Ian for your suggestions. I'll try them out and let you know if I have any questions. > Thanks, > Ian > > > > > > > Thank you for your support and I'm looking forward to collaborating with you on this project! ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] Adding Support for Firefox's Gecko Profile Format 2023-05-26 19:49 ` Anup Sharma @ 2023-06-01 20:49 ` Anup Sharma 2023-06-05 21:47 ` Anup Sharma 0 siblings, 1 reply; 21+ messages in thread From: Anup Sharma @ 2023-06-01 20:49 UTC (permalink / raw) To: linux-perf-users Cc: Ian Rogers, Namhyung Kim, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Mark Rutland, Alexander Shishkin, Jiri Olsa, Adrian Hunter, linux-kernel On Sat, May 27, 2023 at 01:19:47AM +0530, Anup Sharma wrote: > On Fri, May 26, 2023 at 08:17:44AM -0700, Ian Rogers wrote: > > On Thu, May 25, 2023 at 1:53 PM Namhyung Kim <namhyung@kernel.org> wrote: > > > > > > Hi Anup, > > > > > > On Wed, May 24, 2023 at 12:25 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > > > > > > > Hello everyone, > > > > > > > > I'm happy to share that I'll be working on adding support for Firefox's Gecko profile format. > > > > This format is utilized by the Firefox profiler, which is a powerful tool for performance analysis > > > > and debugging. By enhancing the perf data command to generate perf.data files in the Gecko > > > > profile format, it will allow us to leverage the capabilities of the Firefox profiler for visualizing > > > > and analyzing the performance data. > > > > > > > > As a starter task, I have created a test for testing the perf data JSON converter command. > > > > I'm also looking for a few more starter tasks related to this project. I would greatly appreciate > > > > your advice and guidance. > > > > > > > > In my effort to identify any existing bugs, I have enabled flags like 'fsanitize=address' to detect > > > > potential issues but have not found any :). Additionally, I am running perf data commands to ensure > > > > that all use cases are handled properly. > > > > > > Great, good to know it works well with asan for the basic use cases at least. > > > > > > > > > > > I have one question regarding the installation process. Typically, I navigate to the perf directory, > > > > run 'make', and then cp 'perf' to '/usr/bin'. However, I noticed that by default, perf is installed in > > > > the '~/bin/perf' directory. Could someone please clarify why this is the case? Furthermore, I would > > > > like to know how all of you compile the perf tree. > > > > > > I guess $HOME is the default prefix unless you set it to other, then > > > make install will put > > > the binary there. You can make sure if your PATH contains the ~/bin and use it. > > > > > > But it's also possible you can run the perf without installing. I > > > have a symlink in > > > my tmp directory to point to the recent build of the binary and use it > > > always. :) > > > To build, you can either 'cd tools/perf; make' or 'make -C tools/perf' > > > in the top > > > level linux source tree. I also pass "BUILD_BPF_SKEL=1" to enable BPF. > > > > > > Thanks, > > > Namhyung > > > > I quite often test with address sanitizer, I do this by passing to make: > > DEBUG=1 EXTRA_CFLAGS="-O0 -g -fno-omit-frame- pointer > > -fsanitize=address" NO_LIBTRACEEVENT=1 > > > > The libtraceevent exclusion is to avoid false address sanitizer > > warnings in libtraceevent (it wasn't compiled with address sanitizer). > > The other flags are to make the code easier to debug. A good place to > > start for a description of the build flags is Makefile.perf: > > https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/Makefile.perf?h=perf-tools-next > > > > There's also some description here: > > https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/Documentation/Build.txt?h=perf-tools-next > > > > Perhaps you can suggest improvements :-) > > > > Thanks Namhyung and Ian for your suggestions. I'll try them out and let you know if I have any questions. > Hello all, Thank you for participating in today's office hours and helping me clarify my doubts. One topic that emerged from our discussion is the choice of programming language for writing the converter. I am considering using Python as it offers convenient libraries for JSON manipulation. However, I need to investigate whether Python is enabled by default in most of the Linux distributions that ship with perf, as its absence could pose a potential obstacle. Additionally, two profile formats are available: Gecko profile format and Processed profile format[1]. Upon discussing on Firefox Profiler matrix channel, they recommended opting for the Processed format, as it will be supported in future releases as well. Therefore, I intend to begin by working with the Processed format and evaluate the results. If any of you have suggestions regarding the choice of format, I would greatly appreciate your input. Later on, I will address concerns I have regarding the file-loading process. Once again, thank you all for your time. Thanks, Anup [1] https://github.com/firefox-devtools/profiler/blob/main/docs-developer/custom-importer.md > > Thanks, > > Ian > > > > > > > > > > Thank you for your support and I'm looking forward to collaborating with you on this project! ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] Adding Support for Firefox's Gecko Profile Format 2023-06-01 20:49 ` Anup Sharma @ 2023-06-05 21:47 ` Anup Sharma 2023-06-08 6:58 ` Ian Rogers 0 siblings, 1 reply; 21+ messages in thread From: Anup Sharma @ 2023-06-05 21:47 UTC (permalink / raw) To: Anup Sharma Cc: linux-perf-users, Ian Rogers, Namhyung Kim, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Mark Rutland, Alexander Shishkin, Jiri Olsa, Adrian Hunter, linux-kernel On Fri, Jun 02, 2023 at 02:19:30AM +0530, Anup Sharma wrote: > On Sat, May 27, 2023 at 01:19:47AM +0530, Anup Sharma wrote: > > On Fri, May 26, 2023 at 08:17:44AM -0700, Ian Rogers wrote: > > > On Thu, May 25, 2023 at 1:53 PM Namhyung Kim <namhyung@kernel.org> wrote: > > > > > > > > Hi Anup, > > > > > > > > On Wed, May 24, 2023 at 12:25 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > > > > > > > > > Hello everyone, > > > > > > > > > > I'm happy to share that I'll be working on adding support for Firefox's Gecko profile format. > > > > > This format is utilized by the Firefox profiler, which is a powerful tool for performance analysis > > > > > and debugging. By enhancing the perf data command to generate perf.data files in the Gecko > > > > > profile format, it will allow us to leverage the capabilities of the Firefox profiler for visualizing > > > > > and analyzing the performance data. > > > > > > > > > > As a starter task, I have created a test for testing the perf data JSON converter command. > > > > > I'm also looking for a few more starter tasks related to this project. I would greatly appreciate > > > > > your advice and guidance. > > > > > > > > > > In my effort to identify any existing bugs, I have enabled flags like 'fsanitize=address' to detect > > > > > potential issues but have not found any :). Additionally, I am running perf data commands to ensure > > > > > that all use cases are handled properly. > > > > > > > > Great, good to know it works well with asan for the basic use cases at least. > > > > > > > > > > > > > > I have one question regarding the installation process. Typically, I navigate to the perf directory, > > > > > run 'make', and then cp 'perf' to '/usr/bin'. However, I noticed that by default, perf is installed in > > > > > the '~/bin/perf' directory. Could someone please clarify why this is the case? Furthermore, I would > > > > > like to know how all of you compile the perf tree. > > > > > > > > I guess $HOME is the default prefix unless you set it to other, then > > > > make install will put > > > > the binary there. You can make sure if your PATH contains the ~/bin and use it. > > > > > > > > But it's also possible you can run the perf without installing. I > > > > have a symlink in > > > > my tmp directory to point to the recent build of the binary and use it > > > > always. :) > > > > To build, you can either 'cd tools/perf; make' or 'make -C tools/perf' > > > > in the top > > > > level linux source tree. I also pass "BUILD_BPF_SKEL=1" to enable BPF. > > > > > > > > Thanks, > > > > Namhyung > > > > > > I quite often test with address sanitizer, I do this by passing to make: > > > DEBUG=1 EXTRA_CFLAGS="-O0 -g -fno-omit-frame- pointer > > > -fsanitize=address" NO_LIBTRACEEVENT=1 > > > > > > The libtraceevent exclusion is to avoid false address sanitizer > > > warnings in libtraceevent (it wasn't compiled with address sanitizer). > > > The other flags are to make the code easier to debug. A good place to > > > start for a description of the build flags is Makefile.perf: > > > https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/Makefile.perf?h=perf-tools-next > > > > > > There's also some description here: > > > https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/Documentation/Build.txt?h=perf-tools-next > > > > > > Perhaps you can suggest improvements :-) > > > > > > > Thanks Namhyung and Ian for your suggestions. I'll try them out and let you know if I have any questions. > > > > Hello all, > > Thank you for participating in today's office hours and helping me clarify my doubts. > One topic that emerged from our discussion is the choice of programming language > for writing the converter. I am considering using Python as it offers convenient > libraries for JSON manipulation. However, I need to investigate whether Python is > enabled by default in most of the Linux distributions that ship with perf, as its > absence could pose a potential obstacle. > > Additionally, two profile formats are available: Gecko profile format and Processed profile format[1]. > Upon discussing on Firefox Profiler matrix channel, they recommended opting for the Processed format, > as it will be supported in future releases as well. Therefore, I intend to begin by working with the > Processed format and evaluate the results. If any of you have suggestions regarding the choice of > format, I would greatly appreciate your input. > > Later on, I will address concerns I have regarding the file-loading process. > Once again, thank you all for your time. I wanted to provide an update on my exploration of various tools and methods to generate a profiler format. In my experimentation, I considered following command as a reference. perf record -F 99 -g -- perf test -w noploop The command generated perf.data file has been used with three different tools to generate the required format and uploaded them to profiler.firefox.com. Interestingly, I noticed distinct call trees in each of the three cases. I find myself in a state of confusion regarding which result to consider as a reference. Here are the outcomes: 1. Result obtained using perf script: https://share.firefox.dev/3qxEt7F 2. Result obtained using samply:https://share.firefox.dev/3OZsha2 3. Result obtained using simpleperf gecko_profile_generator: https://share.firefox.dev/45Q2BTe I would greatly appreciate any guidance or suggestions in this matter. > Thanks, > Anup > > [1] https://github.com/firefox-devtools/profiler/blob/main/docs-developer/custom-importer.md > > > > Thanks, > > > Ian > > > > > > > > > > > > > Thank you for your support and I'm looking forward to collaborating with you on this project! ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] Adding Support for Firefox's Gecko Profile Format 2023-06-05 21:47 ` Anup Sharma @ 2023-06-08 6:58 ` Ian Rogers 2023-06-08 12:53 ` Anup Sharma 0 siblings, 1 reply; 21+ messages in thread From: Ian Rogers @ 2023-06-08 6:58 UTC (permalink / raw) To: Anup Sharma Cc: linux-perf-users, Namhyung Kim, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Mark Rutland, Alexander Shishkin, Jiri Olsa, Adrian Hunter, linux-kernel On Mon, Jun 5, 2023 at 2:47 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > On Fri, Jun 02, 2023 at 02:19:30AM +0530, Anup Sharma wrote: > > On Sat, May 27, 2023 at 01:19:47AM +0530, Anup Sharma wrote: > > > On Fri, May 26, 2023 at 08:17:44AM -0700, Ian Rogers wrote: > > > > On Thu, May 25, 2023 at 1:53 PM Namhyung Kim <namhyung@kernel.org> wrote: > > > > > > > > > > Hi Anup, > > > > > > > > > > On Wed, May 24, 2023 at 12:25 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > I'm happy to share that I'll be working on adding support for Firefox's Gecko profile format. > > > > > > This format is utilized by the Firefox profiler, which is a powerful tool for performance analysis > > > > > > and debugging. By enhancing the perf data command to generate perf.data files in the Gecko > > > > > > profile format, it will allow us to leverage the capabilities of the Firefox profiler for visualizing > > > > > > and analyzing the performance data. > > > > > > > > > > > > As a starter task, I have created a test for testing the perf data JSON converter command. > > > > > > I'm also looking for a few more starter tasks related to this project. I would greatly appreciate > > > > > > your advice and guidance. > > > > > > > > > > > > In my effort to identify any existing bugs, I have enabled flags like 'fsanitize=address' to detect > > > > > > potential issues but have not found any :). Additionally, I am running perf data commands to ensure > > > > > > that all use cases are handled properly. > > > > > > > > > > Great, good to know it works well with asan for the basic use cases at least. > > > > > > > > > > > > > > > > > I have one question regarding the installation process. Typically, I navigate to the perf directory, > > > > > > run 'make', and then cp 'perf' to '/usr/bin'. However, I noticed that by default, perf is installed in > > > > > > the '~/bin/perf' directory. Could someone please clarify why this is the case? Furthermore, I would > > > > > > like to know how all of you compile the perf tree. > > > > > > > > > > I guess $HOME is the default prefix unless you set it to other, then > > > > > make install will put > > > > > the binary there. You can make sure if your PATH contains the ~/bin and use it. > > > > > > > > > > But it's also possible you can run the perf without installing. I > > > > > have a symlink in > > > > > my tmp directory to point to the recent build of the binary and use it > > > > > always. :) > > > > > To build, you can either 'cd tools/perf; make' or 'make -C tools/perf' > > > > > in the top > > > > > level linux source tree. I also pass "BUILD_BPF_SKEL=1" to enable BPF. > > > > > > > > > > Thanks, > > > > > Namhyung > > > > > > > > I quite often test with address sanitizer, I do this by passing to make: > > > > DEBUG=1 EXTRA_CFLAGS="-O0 -g -fno-omit-frame- pointer > > > > -fsanitize=address" NO_LIBTRACEEVENT=1 > > > > > > > > The libtraceevent exclusion is to avoid false address sanitizer > > > > warnings in libtraceevent (it wasn't compiled with address sanitizer). > > > > The other flags are to make the code easier to debug. A good place to > > > > start for a description of the build flags is Makefile.perf: > > > > https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/Makefile.perf?h=perf-tools-next > > > > > > > > There's also some description here: > > > > https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/Documentation/Build.txt?h=perf-tools-next > > > > > > > > Perhaps you can suggest improvements :-) > > > > > > > > > > Thanks Namhyung and Ian for your suggestions. I'll try them out and let you know if I have any questions. > > > > > > > Hello all, > > > > Thank you for participating in today's office hours and helping me clarify my doubts. > > One topic that emerged from our discussion is the choice of programming language > > for writing the converter. I am considering using Python as it offers convenient > > libraries for JSON manipulation. However, I need to investigate whether Python is > > enabled by default in most of the Linux distributions that ship with perf, as its > > absence could pose a potential obstacle. > > > > Additionally, two profile formats are available: Gecko profile format and Processed profile format[1]. > > Upon discussing on Firefox Profiler matrix channel, they recommended opting for the Processed format, > > as it will be supported in future releases as well. Therefore, I intend to begin by working with the > > Processed format and evaluate the results. If any of you have suggestions regarding the choice of > > format, I would greatly appreciate your input. > > > > Later on, I will address concerns I have regarding the file-loading process. > > Once again, thank you all for your time. > > I wanted to provide an update on my exploration of various tools > and methods to generate a profiler format. In my experimentation, I > considered following command as a reference. > > perf record -F 99 -g -- perf test -w noploop > > The command generated perf.data file has been used with three > different tools to generate the required format and uploaded them to > profiler.firefox.com. Interestingly, I noticed distinct call trees in > each of the three cases. I find myself in a state of confusion regarding > which result to consider as a reference. Here are the outcomes: > > 1. Result obtained using perf script: https://share.firefox.dev/3qxEt7F > 2. Result obtained using samply:https://share.firefox.dev/3OZsha2 > 3. Result obtained using simpleperf gecko_profile_generator: https://share.firefox.dev/45Q2BTe > > I would greatly appreciate any guidance or suggestions in this matter. Hi Anup, of the 3 outcomes the perf script output looks best. Is the reason using the processed profile format? For the other two the major issue appears to be symbolization. Presumably with the first perf script is helping out. I'm not sure why the simpleperf stacks are truncated. Sorry to not be of more use, presumably you've looked into this more and we can discuss this in the office hours tomorrow. Thanks, Ian > > Thanks, > > Anup > > > > [1] https://github.com/firefox-devtools/profiler/blob/main/docs-developer/custom-importer.md > > > > > > Thanks, > > > > Ian > > > > > > > > > > > > > > > > Thank you for your support and I'm looking forward to collaborating with you on this project! ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] Adding Support for Firefox's Gecko Profile Format 2023-06-08 6:58 ` Ian Rogers @ 2023-06-08 12:53 ` Anup Sharma 2023-06-14 20:14 ` Anup Sharma 0 siblings, 1 reply; 21+ messages in thread From: Anup Sharma @ 2023-06-08 12:53 UTC (permalink / raw) To: Ian Rogers Cc: Anup Sharma, linux-perf-users, Namhyung Kim, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Mark Rutland, Alexander Shishkin, Jiri Olsa, Adrian Hunter, linux-kernel On Wed, Jun 07, 2023 at 11:58:57PM -0700, Ian Rogers wrote: > On Mon, Jun 5, 2023 at 2:47 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > > > On Fri, Jun 02, 2023 at 02:19:30AM +0530, Anup Sharma wrote: > > > On Sat, May 27, 2023 at 01:19:47AM +0530, Anup Sharma wrote: > > > > On Fri, May 26, 2023 at 08:17:44AM -0700, Ian Rogers wrote: > > > > > On Thu, May 25, 2023 at 1:53 PM Namhyung Kim <namhyung@kernel.org> wrote: > > > > > > > > > > > > Hi Anup, > > > > > > > > > > > > On Wed, May 24, 2023 at 12:25 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > > > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > > > I'm happy to share that I'll be working on adding support for Firefox's Gecko profile format. > > > > > > > This format is utilized by the Firefox profiler, which is a powerful tool for performance analysis > > > > > > > and debugging. By enhancing the perf data command to generate perf.data files in the Gecko > > > > > > > profile format, it will allow us to leverage the capabilities of the Firefox profiler for visualizing > > > > > > > and analyzing the performance data. > > > > > > > > > > > > > > As a starter task, I have created a test for testing the perf data JSON converter command. > > > > > > > I'm also looking for a few more starter tasks related to this project. I would greatly appreciate > > > > > > > your advice and guidance. > > > > > > > > > > > > > > In my effort to identify any existing bugs, I have enabled flags like 'fsanitize=address' to detect > > > > > > > potential issues but have not found any :). Additionally, I am running perf data commands to ensure > > > > > > > that all use cases are handled properly. > > > > > > > > > > > > Great, good to know it works well with asan for the basic use cases at least. > > > > > > > > > > > > > > > > > > > > I have one question regarding the installation process. Typically, I navigate to the perf directory, > > > > > > > run 'make', and then cp 'perf' to '/usr/bin'. However, I noticed that by default, perf is installed in > > > > > > > the '~/bin/perf' directory. Could someone please clarify why this is the case? Furthermore, I would > > > > > > > like to know how all of you compile the perf tree. > > > > > > > > > > > > I guess $HOME is the default prefix unless you set it to other, then > > > > > > make install will put > > > > > > the binary there. You can make sure if your PATH contains the ~/bin and use it. > > > > > > > > > > > > But it's also possible you can run the perf without installing. I > > > > > > have a symlink in > > > > > > my tmp directory to point to the recent build of the binary and use it > > > > > > always. :) > > > > > > To build, you can either 'cd tools/perf; make' or 'make -C tools/perf' > > > > > > in the top > > > > > > level linux source tree. I also pass "BUILD_BPF_SKEL=1" to enable BPF. > > > > > > > > > > > > Thanks, > > > > > > Namhyung > > > > > > > > > > I quite often test with address sanitizer, I do this by passing to make: > > > > > DEBUG=1 EXTRA_CFLAGS="-O0 -g -fno-omit-frame- pointer > > > > > -fsanitize=address" NO_LIBTRACEEVENT=1 > > > > > > > > > > The libtraceevent exclusion is to avoid false address sanitizer > > > > > warnings in libtraceevent (it wasn't compiled with address sanitizer). > > > > > The other flags are to make the code easier to debug. A good place to > > > > > start for a description of the build flags is Makefile.perf: > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/Makefile.perf?h=perf-tools-next > > > > > > > > > > There's also some description here: > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/Documentation/Build.txt?h=perf-tools-next > > > > > > > > > > Perhaps you can suggest improvements :-) > > > > > > > > > > > > > Thanks Namhyung and Ian for your suggestions. I'll try them out and let you know if I have any questions. > > > > > > > > > > Hello all, > > > > > > Thank you for participating in today's office hours and helping me clarify my doubts. > > > One topic that emerged from our discussion is the choice of programming language > > > for writing the converter. I am considering using Python as it offers convenient > > > libraries for JSON manipulation. However, I need to investigate whether Python is > > > enabled by default in most of the Linux distributions that ship with perf, as its > > > absence could pose a potential obstacle. > > > > > > Additionally, two profile formats are available: Gecko profile format and Processed profile format[1]. > > > Upon discussing on Firefox Profiler matrix channel, they recommended opting for the Processed format, > > > as it will be supported in future releases as well. Therefore, I intend to begin by working with the > > > Processed format and evaluate the results. If any of you have suggestions regarding the choice of > > > format, I would greatly appreciate your input. > > > > > > Later on, I will address concerns I have regarding the file-loading process. > > > Once again, thank you all for your time. > > > > I wanted to provide an update on my exploration of various tools > > and methods to generate a profiler format. In my experimentation, I > > considered following command as a reference. > > > > perf record -F 99 -g -- perf test -w noploop > > > > The command generated perf.data file has been used with three > > different tools to generate the required format and uploaded them to > > profiler.firefox.com. Interestingly, I noticed distinct call trees in > > each of the three cases. I find myself in a state of confusion regarding > > which result to consider as a reference. Here are the outcomes: > > > > 1. Result obtained using perf script: https://share.firefox.dev/3qxEt7F > > 2. Result obtained using samply:https://share.firefox.dev/3OZsha2 > > 3. Result obtained using simpleperf gecko_profile_generator: https://share.firefox.dev/45Q2BTe > > > > I would greatly appreciate any guidance or suggestions in this matter. > > Hi Anup, > > of the 3 outcomes the perf script output looks best. Is the reason > using the processed profile format? For the other two the major issue > appears to be symbolization. Presumably with the first perf script is > helping out. I'm not sure why the simpleperf stacks are truncated. > Sorry to not be of more use, presumably you've looked into this more > and we can discuss this in the office hours tomorrow. The issue does not appear to be related to the processed or gecko profile format. Despite specifying the path to the debug symbols using the -symfs option, the simpleperf gecko_profile_generator script is unable to detect the debug symbols in my system. I have confirmed that the permissions are correct. Here is the basic output: root@yoga:/home/anup/perf/simpleperf/scripts# ./gecko_profile_generator.py -i perf.data | gzip > profiler.gz python3 W 06-08 18:13:27 160392 160392 dso.cpp:446] /usr/lib/x86_64-linux-gnu/ ld-linux-x86-64.so.2 doesn't contain symbol table python3 W 06-08 18:13:27 160392 160392 dso.cpp:446] /usr/lib/x86_64-linux-gnu/ libc.so.6 doesn't contain symbol table > > Thanks, > Ian > > > > Thanks, > > > Anup > > > > > > [1] https://github.com/firefox-devtools/profiler/blob/main/docs-developer/custom-importer.md > > > > > > > > Thanks, > > > > > Ian > > > > > > > > > > > > > > > > > > > Thank you for your support and I'm looking forward to collaborating with you on this project! ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] Adding Support for Firefox's Gecko Profile Format 2023-06-08 12:53 ` Anup Sharma @ 2023-06-14 20:14 ` Anup Sharma 2023-06-14 20:17 ` Ian Rogers 0 siblings, 1 reply; 21+ messages in thread From: Anup Sharma @ 2023-06-14 20:14 UTC (permalink / raw) To: Anup Sharma Cc: Ian Rogers, linux-perf-users, Namhyung Kim, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Mark Rutland, Alexander Shishkin, Jiri Olsa, Adrian Hunter, linux-kernel On Thu, Jun 08, 2023 at 06:23:49PM +0530, Anup Sharma wrote: > On Wed, Jun 07, 2023 at 11:58:57PM -0700, Ian Rogers wrote: > > On Mon, Jun 5, 2023 at 2:47 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > > > > > On Fri, Jun 02, 2023 at 02:19:30AM +0530, Anup Sharma wrote: > > > > On Sat, May 27, 2023 at 01:19:47AM +0530, Anup Sharma wrote: > > > > > On Fri, May 26, 2023 at 08:17:44AM -0700, Ian Rogers wrote: > > > > > > On Thu, May 25, 2023 at 1:53 PM Namhyung Kim <namhyung@kernel.org> wrote: > > > > > > > > > > > > > > Hi Anup, > > > > > > > > > > > > > > On Wed, May 24, 2023 at 12:25 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > > > > > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > > > > > I'm happy to share that I'll be working on adding support for Firefox's Gecko profile format. > > > > > > > > This format is utilized by the Firefox profiler, which is a powerful tool for performance analysis > > > > > > > > and debugging. By enhancing the perf data command to generate perf.data files in the Gecko > > > > > > > > profile format, it will allow us to leverage the capabilities of the Firefox profiler for visualizing > > > > > > > > and analyzing the performance data. > > > > > > > > > > > > > > > > As a starter task, I have created a test for testing the perf data JSON converter command. > > > > > > > > I'm also looking for a few more starter tasks related to this project. I would greatly appreciate > > > > > > > > your advice and guidance. > > > > > > > > > > > > > > > > In my effort to identify any existing bugs, I have enabled flags like 'fsanitize=address' to detect > > > > > > > > potential issues but have not found any :). Additionally, I am running perf data commands to ensure > > > > > > > > that all use cases are handled properly. > > > > > > > > > > > > > > Great, good to know it works well with asan for the basic use cases at least. > > > > > > > > > > > > > > > > > > > > > > > I have one question regarding the installation process. Typically, I navigate to the perf directory, > > > > > > > > run 'make', and then cp 'perf' to '/usr/bin'. However, I noticed that by default, perf is installed in > > > > > > > > the '~/bin/perf' directory. Could someone please clarify why this is the case? Furthermore, I would > > > > > > > > like to know how all of you compile the perf tree. > > > > > > > > > > > > > > I guess $HOME is the default prefix unless you set it to other, then > > > > > > > make install will put > > > > > > > the binary there. You can make sure if your PATH contains the ~/bin and use it. > > > > > > > > > > > > > > But it's also possible you can run the perf without installing. I > > > > > > > have a symlink in > > > > > > > my tmp directory to point to the recent build of the binary and use it > > > > > > > always. :) > > > > > > > To build, you can either 'cd tools/perf; make' or 'make -C tools/perf' > > > > > > > in the top > > > > > > > level linux source tree. I also pass "BUILD_BPF_SKEL=1" to enable BPF. > > > > > > > > > > > > > > Thanks, > > > > > > > Namhyung > > > > > > > > > > > > I quite often test with address sanitizer, I do this by passing to make: > > > > > > DEBUG=1 EXTRA_CFLAGS="-O0 -g -fno-omit-frame- pointer > > > > > > -fsanitize=address" NO_LIBTRACEEVENT=1 > > > > > > > > > > > > The libtraceevent exclusion is to avoid false address sanitizer > > > > > > warnings in libtraceevent (it wasn't compiled with address sanitizer). > > > > > > The other flags are to make the code easier to debug. A good place to > > > > > > start for a description of the build flags is Makefile.perf: > > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/Makefile.perf?h=perf-tools-next > > > > > > > > > > > > There's also some description here: > > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/Documentation/Build.txt?h=perf-tools-next > > > > > > > > > > > > Perhaps you can suggest improvements :-) > > > > > > > > > > > > > > > > Thanks Namhyung and Ian for your suggestions. I'll try them out and let you know if I have any questions. > > > > > > > > > > > > > Hello all, > > > > > > > > Thank you for participating in today's office hours and helping me clarify my doubts. > > > > One topic that emerged from our discussion is the choice of programming language > > > > for writing the converter. I am considering using Python as it offers convenient > > > > libraries for JSON manipulation. However, I need to investigate whether Python is > > > > enabled by default in most of the Linux distributions that ship with perf, as its > > > > absence could pose a potential obstacle. > > > > > > > > Additionally, two profile formats are available: Gecko profile format and Processed profile format[1]. > > > > Upon discussing on Firefox Profiler matrix channel, they recommended opting for the Processed format, > > > > as it will be supported in future releases as well. Therefore, I intend to begin by working with the > > > > Processed format and evaluate the results. If any of you have suggestions regarding the choice of > > > > format, I would greatly appreciate your input. > > > > > > > > Later on, I will address concerns I have regarding the file-loading process. > > > > Once again, thank you all for your time. > > > > > > I wanted to provide an update on my exploration of various tools > > > and methods to generate a profiler format. In my experimentation, I > > > considered following command as a reference. > > > > > > perf record -F 99 -g -- perf test -w noploop > > > > > > The command generated perf.data file has been used with three > > > different tools to generate the required format and uploaded them to > > > profiler.firefox.com. Interestingly, I noticed distinct call trees in > > > each of the three cases. I find myself in a state of confusion regarding > > > which result to consider as a reference. Here are the outcomes: > > > > > > 1. Result obtained using perf script: https://share.firefox.dev/3qxEt7F > > > 2. Result obtained using samply:https://share.firefox.dev/3OZsha2 > > > 3. Result obtained using simpleperf gecko_profile_generator: https://share.firefox.dev/45Q2BTe > > > > > > I would greatly appreciate any guidance or suggestions in this matter. > > > > Hi Anup, > > > > of the 3 outcomes the perf script output looks best. Is the reason > > using the processed profile format? For the other two the major issue > > appears to be symbolization. Presumably with the first perf script is > > helping out. I'm not sure why the simpleperf stacks are truncated. > > Sorry to not be of more use, presumably you've looked into this more > > and we can discuss this in the office hours tomorrow. > > The issue does not appear to be related to the processed or gecko profile format. > Despite specifying the path to the debug symbols using the -symfs option, the > simpleperf gecko_profile_generator script is unable to detect the debug symbols > in my system. I have confirmed that the permissions are correct. > > Here is the basic output: > > root@yoga:/home/anup/perf/simpleperf/scripts# ./gecko_profile_generator.py -i > perf.data | gzip > profiler.gz > python3 W 06-08 18:13:27 160392 160392 dso.cpp:446] /usr/lib/x86_64-linux-gnu/ > ld-linux-x86-64.so.2 doesn't contain symbol table > python3 W 06-08 18:13:27 160392 160392 dso.cpp:446] /usr/lib/x86_64-linux-gnu/ > libc.so.6 doesn't contain symbol table > Hi All, I wanted to take a moment to provide you with an update on the progress of our Firefox Gecko converter work. While I must emphasize that the code I'm sharing is not the final version, I wanted to share the advancements I have made thus far. This script can generate a JSON format from the output of the "perf script" command. I attempted to load this JSON file into profile.firefox.com, and although it successfully loaded, the call tree are not visible. I'm certain this issue is related to the format of the JSON file or if there is another underlying cause. I will continue investigating to determine the cause of this problem. ----------------------->cut<--------------------------- diff --git a/tools/perf/scripts/python/gecko_converter.py b/tools/perf/scripts/python/gecko_converter.py new file mode 100644 index 000000000000..7340ea4bd84e --- /dev/null +++ b/tools/perf/scripts/python/gecko_converter.py @@ -0,0 +1,236 @@ +#!/usr/env/bin python3 +import re +import sys +import json +from functools import reduce + +def isPerfScriptFormat(profile): + if profile.startswith('{'): + return False + + firstLine = profile[:profile.index('\n')] + return bool(re.match(r'^\S.*?\s+(?:\d+/)?\d+\s+(?:\d+\d+\s+)?[\d.]+:', firstLine)) + +CATEGORIES = [ +{'name': 'User', 'color': 'yellow', 'subcategories': ['Other']}, +{'name': 'Kernel', 'color': 'orange', 'subcategories': ['Other']} +] +USER_CATEGORY_INDEX = 0 +KERNEL_CATEGORY_INDEX = 1 + +def convertPerfScriptProfile(profile): + def _createtread(name, pid, tid): + markers = { + 'schema': { + 'name': 0, + 'startTime': 1, + 'endTime': 2, + 'phase': 3, + 'category': 4, + 'data': 5, + }, + 'data': [], + } + samples = { + 'schema': { + 'stack': 0, + 'time': 1, + 'responsiveness': 2, + }, + 'data': [], + } + frameTable = { + 'schema': { + 'location': 0, + 'relevantForJS': 1, + 'innerWindowID': 2, + 'implementation': 3, + 'optimizations': 4, + 'line': 5, + 'column': 6, + 'category': 7, + 'subcategory': 8, + }, + 'data': [], + } + stackTable = { + 'schema': { + 'prefix': 0, + 'frame': 1, + }, + 'data': [], + } + stringTable = [] + + stackMap = dict() + def get_or_create_stack(frame, prefix): + key = f"{frame}" if prefix is None else f"{frame},{prefix}" + stack = stackMap.get(key) + if stack is None: + stack = len(stackTable['data']) + stackTable['data'].append([prefix, frame]) + stackMap[key] = stack + return stack + + frameMap = dict() + def get_or_create_frame(frameString): + frame = frameMap.get(frameString) + if frame is None: + frame = len(frameTable['data']) + location = len(stringTable) + stringTable.append(frameString) + # print('batman', frame, frameString) + + category = KERNEL_CATEGORY_INDEX if frameString.find('kallsyms') != -1 or frameString.find('/vmlinux') != 1 or frameString.endswith('.ko)') else USER_CATEGORY_INDEX + implementation = None + optimizations = None + line = None + relevantForJS = False + subcategory = None + innerWindowID = 0 + column = None + + frameTable['data'].append([ + location, + relevantForJS, + innerWindowID, + implementation, + optimizations, + line, + column, + category, + subcategory, + ]) + frameMap[frameString] = frame + return frame + + def addSample(threadName, stackArray, time): + nonlocal name + if name != threadName: + name = threadName + stack = reduce(lambda prefix, stackFrame: get_or_create_stack(get_or_create_frame(stackFrame), prefix), stackArray) + responsiveness = 0 + samples['data'].append([stack, time, responsiveness]) + + def finish(): + return { + "tid": tid, + "pid": pid, + "name": name, + "markers": markers, + "samples": samples, + "frameTable": frameTable, + "stackTable": stackTable, + "stringTable": stringTable, + "registerTime": 0, + "unregisterTime": None, + "processType": 'default' + } + + return { + "addSample": addSample, + "finish": finish + } + + threadMap = dict() + def _addThreadSample(pid, tid, threadName, time_stamp, stack): + thread = threadMap.get(tid) + if not thread: + thread = _createtread(pid, tid, threadName) + threadMap[tid] = thread + thread['addSample'](threadName, stack, time_stamp) + + lines = profile.split('\n') + + line_index = 0 + start_time = 0 + while line_index < len(lines): + line = lines[line_index] + line_index += 1 + if line == '' or line.startswith('#'): + continue + + sample_start_line = line + + sample_start_match = re.match(r'^(.*)\s+([\d.]+):', sample_start_line) + if not sample_start_match: + print(f'Could not parse line as the start of a sample in the "perf script" profile format: "{sample_start_line}"') + continue + + before_time_stamp = sample_start_match[1] + time_stamp = float(sample_start_match[2]) * 1000 + + threadNamePidAndTidMatch = re.match(r'^(.*)\s+(?:(\d+)\/)?(\d+)\b', before_time_stamp) + + if not threadNamePidAndTidMatch: + print('Could not parse line as the start of a sample in the "perf script" profile format: "%s"' % sampleStartLine) + continue + + threadName = threadNamePidAndTidMatch[1].strip() + pid = int(threadNamePidAndTidMatch[2] or 0) + tid = int(threadNamePidAndTidMatch[3] or 0) + startTime = 0 #workarround + if startTime == 0: + startTime = time_stamp + stack = [] + while line_index < len(lines): + stackFrameLine = lines[line_index] + line_index += 1 + if stackFrameLine.strip() == '': + break + stackFrameMatch = re.match(r'^\s*(\w+)\s*(.+) \(([^)]*)\)', stackFrameLine) + if stackFrameMatch: + rawFunc = stackFrameMatch[2] + mod = stackFrameMatch[3] + rawFunc = re.sub(r'\+0x[\da-f]+$', '', rawFunc) + + if rawFunc.startswith('('): + continue # skip process names + + if mod: + rawFunc += f' (in {mod})' + + stack.append(rawFunc) + if len(stack) != 0: + stack.reverse() + _addThreadSample(pid, tid, threadName, time_stamp, stack) + + thread_array = list(map(lambda thread: thread['finish'](), threadMap.values())) + + for thread in thread_array: + key = thread['samples']['schema']['time'] + thread['samples']['data'].sort(key=lambda data : float(data[key])) + + return { + 'meta': { + 'interval': 1, + 'processType': 0, + 'product': 'Firefox', + 'stackwalk': 1, + 'debug': 0, + 'gcpoison': 0, + 'asyncstack': 1, + 'startTime': startTime, + 'shutdownTime': None, + 'version': 24, + 'presymbolicated': True, + 'categories': CATEGORIES, + 'markerSchema': [] + }, + 'libs': [], + 'threads': thread_array, + 'processes': [], + 'pausedRanges': [] +} + +def main(): +# inputFile = input('Enter input file name: ') + with open('test.txt') as f: + profile = f.read() + isPerfScript = isPerfScriptFormat(profile) + output = convertPerfScriptProfile(profile) + json.dump(output, sys.stdout, indent=2) + # print('isPerfScript: {}'.format(isPerfScript)) + +if __name__ == '__main__': + main() ------------------------>cut<--------------------------- Thanks, Anup > > > > Thanks, > > Ian > > > > > > Thanks, > > > > Anup > > > > > > > > [1] https://github.com/firefox-devtools/profiler/blob/main/docs-developer/custom-importer.md > > > > > > > > > > Thanks, > > > > > > Ian > > > > > > > > > > > > > > > > > > > > > > Thank you for your support and I'm looking forward to collaborating with you on this project! ^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [RFC] Adding Support for Firefox's Gecko Profile Format 2023-06-14 20:14 ` Anup Sharma @ 2023-06-14 20:17 ` Ian Rogers 2023-06-28 20:11 ` Anup Sharma 0 siblings, 1 reply; 21+ messages in thread From: Ian Rogers @ 2023-06-14 20:17 UTC (permalink / raw) To: Anup Sharma Cc: linux-perf-users, Namhyung Kim, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Mark Rutland, Alexander Shishkin, Jiri Olsa, Adrian Hunter, linux-kernel On Wed, Jun 14, 2023 at 1:14 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > On Thu, Jun 08, 2023 at 06:23:49PM +0530, Anup Sharma wrote: > > On Wed, Jun 07, 2023 at 11:58:57PM -0700, Ian Rogers wrote: > > > On Mon, Jun 5, 2023 at 2:47 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > > > > > > > On Fri, Jun 02, 2023 at 02:19:30AM +0530, Anup Sharma wrote: > > > > > On Sat, May 27, 2023 at 01:19:47AM +0530, Anup Sharma wrote: > > > > > > On Fri, May 26, 2023 at 08:17:44AM -0700, Ian Rogers wrote: > > > > > > > On Thu, May 25, 2023 at 1:53 PM Namhyung Kim <namhyung@kernel.org> wrote: > > > > > > > > > > > > > > > > Hi Anup, > > > > > > > > > > > > > > > > On Wed, May 24, 2023 at 12:25 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > > > > > > > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > > > > > > > I'm happy to share that I'll be working on adding support for Firefox's Gecko profile format. > > > > > > > > > This format is utilized by the Firefox profiler, which is a powerful tool for performance analysis > > > > > > > > > and debugging. By enhancing the perf data command to generate perf.data files in the Gecko > > > > > > > > > profile format, it will allow us to leverage the capabilities of the Firefox profiler for visualizing > > > > > > > > > and analyzing the performance data. > > > > > > > > > > > > > > > > > > As a starter task, I have created a test for testing the perf data JSON converter command. > > > > > > > > > I'm also looking for a few more starter tasks related to this project. I would greatly appreciate > > > > > > > > > your advice and guidance. > > > > > > > > > > > > > > > > > > In my effort to identify any existing bugs, I have enabled flags like 'fsanitize=address' to detect > > > > > > > > > potential issues but have not found any :). Additionally, I am running perf data commands to ensure > > > > > > > > > that all use cases are handled properly. > > > > > > > > > > > > > > > > Great, good to know it works well with asan for the basic use cases at least. > > > > > > > > > > > > > > > > > > > > > > > > > > I have one question regarding the installation process. Typically, I navigate to the perf directory, > > > > > > > > > run 'make', and then cp 'perf' to '/usr/bin'. However, I noticed that by default, perf is installed in > > > > > > > > > the '~/bin/perf' directory. Could someone please clarify why this is the case? Furthermore, I would > > > > > > > > > like to know how all of you compile the perf tree. > > > > > > > > > > > > > > > > I guess $HOME is the default prefix unless you set it to other, then > > > > > > > > make install will put > > > > > > > > the binary there. You can make sure if your PATH contains the ~/bin and use it. > > > > > > > > > > > > > > > > But it's also possible you can run the perf without installing. I > > > > > > > > have a symlink in > > > > > > > > my tmp directory to point to the recent build of the binary and use it > > > > > > > > always. :) > > > > > > > > To build, you can either 'cd tools/perf; make' or 'make -C tools/perf' > > > > > > > > in the top > > > > > > > > level linux source tree. I also pass "BUILD_BPF_SKEL=1" to enable BPF. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Namhyung > > > > > > > > > > > > > > I quite often test with address sanitizer, I do this by passing to make: > > > > > > > DEBUG=1 EXTRA_CFLAGS="-O0 -g -fno-omit-frame- pointer > > > > > > > -fsanitize=address" NO_LIBTRACEEVENT=1 > > > > > > > > > > > > > > The libtraceevent exclusion is to avoid false address sanitizer > > > > > > > warnings in libtraceevent (it wasn't compiled with address sanitizer). > > > > > > > The other flags are to make the code easier to debug. A good place to > > > > > > > start for a description of the build flags is Makefile.perf: > > > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/Makefile.perf?h=perf-tools-next > > > > > > > > > > > > > > There's also some description here: > > > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/Documentation/Build.txt?h=perf-tools-next > > > > > > > > > > > > > > Perhaps you can suggest improvements :-) > > > > > > > > > > > > > > > > > > > Thanks Namhyung and Ian for your suggestions. I'll try them out and let you know if I have any questions. > > > > > > > > > > > > > > > > Hello all, > > > > > > > > > > Thank you for participating in today's office hours and helping me clarify my doubts. > > > > > One topic that emerged from our discussion is the choice of programming language > > > > > for writing the converter. I am considering using Python as it offers convenient > > > > > libraries for JSON manipulation. However, I need to investigate whether Python is > > > > > enabled by default in most of the Linux distributions that ship with perf, as its > > > > > absence could pose a potential obstacle. > > > > > > > > > > Additionally, two profile formats are available: Gecko profile format and Processed profile format[1]. > > > > > Upon discussing on Firefox Profiler matrix channel, they recommended opting for the Processed format, > > > > > as it will be supported in future releases as well. Therefore, I intend to begin by working with the > > > > > Processed format and evaluate the results. If any of you have suggestions regarding the choice of > > > > > format, I would greatly appreciate your input. > > > > > > > > > > Later on, I will address concerns I have regarding the file-loading process. > > > > > Once again, thank you all for your time. > > > > > > > > I wanted to provide an update on my exploration of various tools > > > > and methods to generate a profiler format. In my experimentation, I > > > > considered following command as a reference. > > > > > > > > perf record -F 99 -g -- perf test -w noploop > > > > > > > > The command generated perf.data file has been used with three > > > > different tools to generate the required format and uploaded them to > > > > profiler.firefox.com. Interestingly, I noticed distinct call trees in > > > > each of the three cases. I find myself in a state of confusion regarding > > > > which result to consider as a reference. Here are the outcomes: > > > > > > > > 1. Result obtained using perf script: https://share.firefox.dev/3qxEt7F > > > > 2. Result obtained using samply:https://share.firefox.dev/3OZsha2 > > > > 3. Result obtained using simpleperf gecko_profile_generator: https://share.firefox.dev/45Q2BTe > > > > > > > > I would greatly appreciate any guidance or suggestions in this matter. > > > > > > Hi Anup, > > > > > > of the 3 outcomes the perf script output looks best. Is the reason > > > using the processed profile format? For the other two the major issue > > > appears to be symbolization. Presumably with the first perf script is > > > helping out. I'm not sure why the simpleperf stacks are truncated. > > > Sorry to not be of more use, presumably you've looked into this more > > > and we can discuss this in the office hours tomorrow. > > > > The issue does not appear to be related to the processed or gecko profile format. > > Despite specifying the path to the debug symbols using the -symfs option, the > > simpleperf gecko_profile_generator script is unable to detect the debug symbols > > in my system. I have confirmed that the permissions are correct. > > > > Here is the basic output: > > > > root@yoga:/home/anup/perf/simpleperf/scripts# ./gecko_profile_generator.py -i > > perf.data | gzip > profiler.gz > > python3 W 06-08 18:13:27 160392 160392 dso.cpp:446] /usr/lib/x86_64-linux-gnu/ > > ld-linux-x86-64.so.2 doesn't contain symbol table > > python3 W 06-08 18:13:27 160392 160392 dso.cpp:446] /usr/lib/x86_64-linux-gnu/ > > libc.so.6 doesn't contain symbol table > > > > Hi All, > > I wanted to take a moment to provide you with an update on the progress > of our Firefox Gecko converter work. While I must emphasize that the code > I'm sharing is not the final version, I wanted to share the advancements > I have made thus far. > > This script can generate a JSON format from the output of the "perf script" command. > I attempted to load this JSON file into profile.firefox.com, and although it > successfully loaded, the call tree are not visible. I'm certain this issue > is related to the format of the JSON file or if there is another underlying > cause. I will continue investigating to determine the cause of this problem. Great Anup, progress can be frustrating slow at first but it is a good milestone to be generating output and having firefox profiler consume it. You can open up the JavaScript console for the firefox profiler and it will give some debug output. This should hopefully help you make progress. Thanks, Ian > ----------------------->cut<--------------------------- > diff --git a/tools/perf/scripts/python/gecko_converter.py b/tools/perf/scripts/python/gecko_converter.py > new file mode 100644 > index 000000000000..7340ea4bd84e > --- /dev/null > +++ b/tools/perf/scripts/python/gecko_converter.py > @@ -0,0 +1,236 @@ > +#!/usr/env/bin python3 > +import re > +import sys > +import json > +from functools import reduce > + > +def isPerfScriptFormat(profile): > + if profile.startswith('{'): > + return False > + > + firstLine = profile[:profile.index('\n')] > + return bool(re.match(r'^\S.*?\s+(?:\d+/)?\d+\s+(?:\d+\d+\s+)?[\d.]+:', firstLine)) > + > +CATEGORIES = [ > +{'name': 'User', 'color': 'yellow', 'subcategories': ['Other']}, > +{'name': 'Kernel', 'color': 'orange', 'subcategories': ['Other']} > +] > +USER_CATEGORY_INDEX = 0 > +KERNEL_CATEGORY_INDEX = 1 > + > +def convertPerfScriptProfile(profile): > + def _createtread(name, pid, tid): > + markers = { > + 'schema': { > + 'name': 0, > + 'startTime': 1, > + 'endTime': 2, > + 'phase': 3, > + 'category': 4, > + 'data': 5, > + }, > + 'data': [], > + } > + samples = { > + 'schema': { > + 'stack': 0, > + 'time': 1, > + 'responsiveness': 2, > + }, > + 'data': [], > + } > + frameTable = { > + 'schema': { > + 'location': 0, > + 'relevantForJS': 1, > + 'innerWindowID': 2, > + 'implementation': 3, > + 'optimizations': 4, > + 'line': 5, > + 'column': 6, > + 'category': 7, > + 'subcategory': 8, > + }, > + 'data': [], > + } > + stackTable = { > + 'schema': { > + 'prefix': 0, > + 'frame': 1, > + }, > + 'data': [], > + } > + stringTable = [] > + > + stackMap = dict() > + def get_or_create_stack(frame, prefix): > + key = f"{frame}" if prefix is None else f"{frame},{prefix}" > + stack = stackMap.get(key) > + if stack is None: > + stack = len(stackTable['data']) > + stackTable['data'].append([prefix, frame]) > + stackMap[key] = stack > + return stack > + > + frameMap = dict() > + def get_or_create_frame(frameString): > + frame = frameMap.get(frameString) > + if frame is None: > + frame = len(frameTable['data']) > + location = len(stringTable) > + stringTable.append(frameString) > + # print('batman', frame, frameString) > + > + category = KERNEL_CATEGORY_INDEX if frameString.find('kallsyms') != -1 or frameString.find('/vmlinux') != 1 or frameString.endswith('.ko)') else USER_CATEGORY_INDEX > + implementation = None > + optimizations = None > + line = None > + relevantForJS = False > + subcategory = None > + innerWindowID = 0 > + column = None > + > + frameTable['data'].append([ > + location, > + relevantForJS, > + innerWindowID, > + implementation, > + optimizations, > + line, > + column, > + category, > + subcategory, > + ]) > + frameMap[frameString] = frame > + return frame > + > + def addSample(threadName, stackArray, time): > + nonlocal name > + if name != threadName: > + name = threadName > + stack = reduce(lambda prefix, stackFrame: get_or_create_stack(get_or_create_frame(stackFrame), prefix), stackArray) > + responsiveness = 0 > + samples['data'].append([stack, time, responsiveness]) > + > + def finish(): > + return { > + "tid": tid, > + "pid": pid, > + "name": name, > + "markers": markers, > + "samples": samples, > + "frameTable": frameTable, > + "stackTable": stackTable, > + "stringTable": stringTable, > + "registerTime": 0, > + "unregisterTime": None, > + "processType": 'default' > + } > + > + return { > + "addSample": addSample, > + "finish": finish > + } > + > + threadMap = dict() > + def _addThreadSample(pid, tid, threadName, time_stamp, stack): > + thread = threadMap.get(tid) > + if not thread: > + thread = _createtread(pid, tid, threadName) > + threadMap[tid] = thread > + thread['addSample'](threadName, stack, time_stamp) > + > + lines = profile.split('\n') > + > + line_index = 0 > + start_time = 0 > + while line_index < len(lines): > + line = lines[line_index] > + line_index += 1 > + if line == '' or line.startswith('#'): > + continue > + > + sample_start_line = line > + > + sample_start_match = re.match(r'^(.*)\s+([\d.]+):', sample_start_line) > + if not sample_start_match: > + print(f'Could not parse line as the start of a sample in the "perf script" profile format: "{sample_start_line}"') > + continue > + > + before_time_stamp = sample_start_match[1] > + time_stamp = float(sample_start_match[2]) * 1000 > + > + threadNamePidAndTidMatch = re.match(r'^(.*)\s+(?:(\d+)\/)?(\d+)\b', before_time_stamp) > + > + if not threadNamePidAndTidMatch: > + print('Could not parse line as the start of a sample in the "perf script" profile format: "%s"' % sampleStartLine) > + continue > + > + threadName = threadNamePidAndTidMatch[1].strip() > + pid = int(threadNamePidAndTidMatch[2] or 0) > + tid = int(threadNamePidAndTidMatch[3] or 0) > + startTime = 0 #workarround > + if startTime == 0: > + startTime = time_stamp > + stack = [] > + while line_index < len(lines): > + stackFrameLine = lines[line_index] > + line_index += 1 > + if stackFrameLine.strip() == '': > + break > + stackFrameMatch = re.match(r'^\s*(\w+)\s*(.+) \(([^)]*)\)', stackFrameLine) > + if stackFrameMatch: > + rawFunc = stackFrameMatch[2] > + mod = stackFrameMatch[3] > + rawFunc = re.sub(r'\+0x[\da-f]+$', '', rawFunc) > + > + if rawFunc.startswith('('): > + continue # skip process names > + > + if mod: > + rawFunc += f' (in {mod})' > + > + stack.append(rawFunc) > + if len(stack) != 0: > + stack.reverse() > + _addThreadSample(pid, tid, threadName, time_stamp, stack) > + > + thread_array = list(map(lambda thread: thread['finish'](), threadMap.values())) > + > + for thread in thread_array: > + key = thread['samples']['schema']['time'] > + thread['samples']['data'].sort(key=lambda data : float(data[key])) > + > + return { > + 'meta': { > + 'interval': 1, > + 'processType': 0, > + 'product': 'Firefox', > + 'stackwalk': 1, > + 'debug': 0, > + 'gcpoison': 0, > + 'asyncstack': 1, > + 'startTime': startTime, > + 'shutdownTime': None, > + 'version': 24, > + 'presymbolicated': True, > + 'categories': CATEGORIES, > + 'markerSchema': [] > + }, > + 'libs': [], > + 'threads': thread_array, > + 'processes': [], > + 'pausedRanges': [] > +} > + > +def main(): > +# inputFile = input('Enter input file name: ') > + with open('test.txt') as f: > + profile = f.read() > + isPerfScript = isPerfScriptFormat(profile) > + output = convertPerfScriptProfile(profile) > + json.dump(output, sys.stdout, indent=2) > + # print('isPerfScript: {}'.format(isPerfScript)) > + > +if __name__ == '__main__': > + main() > ------------------------>cut<--------------------------- > > Thanks, > Anup > > > > > > > Thanks, > > > Ian > > > > > > > > Thanks, > > > > > Anup > > > > > > > > > > [1] https://github.com/firefox-devtools/profiler/blob/main/docs-developer/custom-importer.md > > > > > > > > > > > > Thanks, > > > > > > > Ian > > > > > > > > > > > > > > > > > > > > > > > > > Thank you for your support and I'm looking forward to collaborating with you on this project! ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] Adding Support for Firefox's Gecko Profile Format 2023-06-14 20:17 ` Ian Rogers @ 2023-06-28 20:11 ` Anup Sharma 2023-06-29 6:26 ` Namhyung Kim 0 siblings, 1 reply; 21+ messages in thread From: Anup Sharma @ 2023-06-28 20:11 UTC (permalink / raw) To: Ian Rogers Cc: Anup Sharma, linux-perf-users, Namhyung Kim, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Mark Rutland, Alexander Shishkin, Jiri Olsa, Adrian Hunter, linux-kernel On Wed, Jun 14, 2023 at 01:17:52PM -0700, Ian Rogers wrote: > On Wed, Jun 14, 2023 at 1:14 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > > > On Thu, Jun 08, 2023 at 06:23:49PM +0530, Anup Sharma wrote: > > > On Wed, Jun 07, 2023 at 11:58:57PM -0700, Ian Rogers wrote: > > > > On Mon, Jun 5, 2023 at 2:47 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > > > > > > > > > On Fri, Jun 02, 2023 at 02:19:30AM +0530, Anup Sharma wrote: > > > > > > On Sat, May 27, 2023 at 01:19:47AM +0530, Anup Sharma wrote: > > > > > > > On Fri, May 26, 2023 at 08:17:44AM -0700, Ian Rogers wrote: > > > > > > > > On Thu, May 25, 2023 at 1:53 PM Namhyung Kim <namhyung@kernel.org> wrote: > > > > > > > > > > > > > > > > > > Hi Anup, > > > > > > > > > > > > > > > > > > On Wed, May 24, 2023 at 12:25 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > > > > > > > > > I'm happy to share that I'll be working on adding support for Firefox's Gecko profile format. > > > > > > > > > > This format is utilized by the Firefox profiler, which is a powerful tool for performance analysis > > > > > > > > > > and debugging. By enhancing the perf data command to generate perf.data files in the Gecko > > > > > > > > > > profile format, it will allow us to leverage the capabilities of the Firefox profiler for visualizing > > > > > > > > > > and analyzing the performance data. > > > > > > > > > > > > > > > > > > > > As a starter task, I have created a test for testing the perf data JSON converter command. > > > > > > > > > > I'm also looking for a few more starter tasks related to this project. I would greatly appreciate > > > > > > > > > > your advice and guidance. > > > > > > > > > > > > > > > > > > > > In my effort to identify any existing bugs, I have enabled flags like 'fsanitize=address' to detect > > > > > > > > > > potential issues but have not found any :). Additionally, I am running perf data commands to ensure > > > > > > > > > > that all use cases are handled properly. > > > > > > > > > > > > > > > > > > Great, good to know it works well with asan for the basic use cases at least. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have one question regarding the installation process. Typically, I navigate to the perf directory, > > > > > > > > > > run 'make', and then cp 'perf' to '/usr/bin'. However, I noticed that by default, perf is installed in > > > > > > > > > > the '~/bin/perf' directory. Could someone please clarify why this is the case? Furthermore, I would > > > > > > > > > > like to know how all of you compile the perf tree. > > > > > > > > > > > > > > > > > > I guess $HOME is the default prefix unless you set it to other, then > > > > > > > > > make install will put > > > > > > > > > the binary there. You can make sure if your PATH contains the ~/bin and use it. > > > > > > > > > > > > > > > > > > But it's also possible you can run the perf without installing. I > > > > > > > > > have a symlink in > > > > > > > > > my tmp directory to point to the recent build of the binary and use it > > > > > > > > > always. :) > > > > > > > > > To build, you can either 'cd tools/perf; make' or 'make -C tools/perf' > > > > > > > > > in the top > > > > > > > > > level linux source tree. I also pass "BUILD_BPF_SKEL=1" to enable BPF. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Namhyung > > > > > > > > > > > > > > > > I quite often test with address sanitizer, I do this by passing to make: > > > > > > > > DEBUG=1 EXTRA_CFLAGS="-O0 -g -fno-omit-frame- pointer > > > > > > > > -fsanitize=address" NO_LIBTRACEEVENT=1 > > > > > > > > > > > > > > > > The libtraceevent exclusion is to avoid false address sanitizer > > > > > > > > warnings in libtraceevent (it wasn't compiled with address sanitizer). > > > > > > > > The other flags are to make the code easier to debug. A good place to > > > > > > > > start for a description of the build flags is Makefile.perf: > > > > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/Makefile.perf?h=perf-tools-next > > > > > > > > > > > > > > > > There's also some description here: > > > > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/tree/tools/perf/Documentation/Build.txt?h=perf-tools-next > > > > > > > > > > > > > > > > Perhaps you can suggest improvements :-) > > > > > > > > > > > > > > > > > > > > > > Thanks Namhyung and Ian for your suggestions. I'll try them out and let you know if I have any questions. > > > > > > > > > > > > > > > > > > > Hello all, > > > > > > > > > > > > Thank you for participating in today's office hours and helping me clarify my doubts. > > > > > > One topic that emerged from our discussion is the choice of programming language > > > > > > for writing the converter. I am considering using Python as it offers convenient > > > > > > libraries for JSON manipulation. However, I need to investigate whether Python is > > > > > > enabled by default in most of the Linux distributions that ship with perf, as its > > > > > > absence could pose a potential obstacle. > > > > > > > > > > > > Additionally, two profile formats are available: Gecko profile format and Processed profile format[1]. > > > > > > Upon discussing on Firefox Profiler matrix channel, they recommended opting for the Processed format, > > > > > > as it will be supported in future releases as well. Therefore, I intend to begin by working with the > > > > > > Processed format and evaluate the results. If any of you have suggestions regarding the choice of > > > > > > format, I would greatly appreciate your input. > > > > > > > > > > > > Later on, I will address concerns I have regarding the file-loading process. > > > > > > Once again, thank you all for your time. > > > > > > > > > > I wanted to provide an update on my exploration of various tools > > > > > and methods to generate a profiler format. In my experimentation, I > > > > > considered following command as a reference. > > > > > > > > > > perf record -F 99 -g -- perf test -w noploop > > > > > > > > > > The command generated perf.data file has been used with three > > > > > different tools to generate the required format and uploaded them to > > > > > profiler.firefox.com. Interestingly, I noticed distinct call trees in > > > > > each of the three cases. I find myself in a state of confusion regarding > > > > > which result to consider as a reference. Here are the outcomes: > > > > > > > > > > 1. Result obtained using perf script: https://share.firefox.dev/3qxEt7F > > > > > 2. Result obtained using samply:https://share.firefox.dev/3OZsha2 > > > > > 3. Result obtained using simpleperf gecko_profile_generator: https://share.firefox.dev/45Q2BTe > > > > > > > > > > I would greatly appreciate any guidance or suggestions in this matter. > > > > > > > > Hi Anup, > > > > > > > > of the 3 outcomes the perf script output looks best. Is the reason > > > > using the processed profile format? For the other two the major issue > > > > appears to be symbolization. Presumably with the first perf script is > > > > helping out. I'm not sure why the simpleperf stacks are truncated. > > > > Sorry to not be of more use, presumably you've looked into this more > > > > and we can discuss this in the office hours tomorrow. > > > > > > The issue does not appear to be related to the processed or gecko profile format. > > > Despite specifying the path to the debug symbols using the -symfs option, the > > > simpleperf gecko_profile_generator script is unable to detect the debug symbols > > > in my system. I have confirmed that the permissions are correct. > > > > > > Here is the basic output: > > > > > > root@yoga:/home/anup/perf/simpleperf/scripts# ./gecko_profile_generator.py -i > > > perf.data | gzip > profiler.gz > > > python3 W 06-08 18:13:27 160392 160392 dso.cpp:446] /usr/lib/x86_64-linux-gnu/ > > > ld-linux-x86-64.so.2 doesn't contain symbol table > > > python3 W 06-08 18:13:27 160392 160392 dso.cpp:446] /usr/lib/x86_64-linux-gnu/ > > > libc.so.6 doesn't contain symbol table > > > > > > > Hi All, > > > > I wanted to take a moment to provide you with an update on the progress > > of our Firefox Gecko converter work. While I must emphasize that the code > > I'm sharing is not the final version, I wanted to share the advancements > > I have made thus far. > > > > This script can generate a JSON format from the output of the "perf script" command. > > I attempted to load this JSON file into profile.firefox.com, and although it > > successfully loaded, the call tree are not visible. I'm certain this issue > > is related to the format of the JSON file or if there is another underlying > > cause. I will continue investigating to determine the cause of this problem. > > Great Anup, progress can be frustrating slow at first but it is a good > milestone to be generating output and having firefox profiler consume > it. You can open up the JavaScript console for the firefox profiler > and it will give some debug output. This should hopefully help you > make progress. This week I tried playing with perf-script-python, the first challenge was figuring out how to parse the data coming from the process_events. Understanding the structure and extracting the necessary information proved to be more complex than anticipated. This required me to spend a significant amount of time researching and experimenting with different parsing techniques. The second challenge revolved around the usage of event hooks provided with the perf script python event handlers. I found myself deliberating between two approaches. The first approach involved creating custom functions that would be called using the event handlers. These functions would then save the data in an organized format within globally created data structures. The alternative approach was to write the entire logic inside the event handlers themselves. Additionally, I contemplated whether it would be more suitable to handle the creation of a Gecko format for JSON and the profile format within the same script or to separate them into different scripts. I will discuss this points during tomorrow's office hour. However, I have gained a deeper understanding of the problem at hand and will use this knowledge to make more informed decisions and progress more effectively in the coming weeks. > Thanks, > Ian > > > ----------------------->cut<--------------------------- > > diff --git a/tools/perf/scripts/python/gecko_converter.py b/tools/perf/scripts/python/gecko_converter.py > > new file mode 100644 > > index 000000000000..7340ea4bd84e > > --- /dev/null > > +++ b/tools/perf/scripts/python/gecko_converter.py > > @@ -0,0 +1,236 @@ > > +#!/usr/env/bin python3 > > +import re > > +import sys > > +import json > > +from functools import reduce > > + > > +def isPerfScriptFormat(profile): > > + if profile.startswith('{'): > > + return False > > + > > + firstLine = profile[:profile.index('\n')] > > + return bool(re.match(r'^\S.*?\s+(?:\d+/)?\d+\s+(?:\d+\d+\s+)?[\d.]+:', firstLine)) > > + > > +CATEGORIES = [ > > +{'name': 'User', 'color': 'yellow', 'subcategories': ['Other']}, > > +{'name': 'Kernel', 'color': 'orange', 'subcategories': ['Other']} > > +] > > +USER_CATEGORY_INDEX = 0 > > +KERNEL_CATEGORY_INDEX = 1 > > + > > +def convertPerfScriptProfile(profile): > > + def _createtread(name, pid, tid): > > + markers = { > > + 'schema': { > > + 'name': 0, > > + 'startTime': 1, > > + 'endTime': 2, > > + 'phase': 3, > > + 'category': 4, > > + 'data': 5, > > + }, > > + 'data': [], > > + } > > + samples = { > > + 'schema': { > > + 'stack': 0, > > + 'time': 1, > > + 'responsiveness': 2, > > + }, > > + 'data': [], > > + } > > + frameTable = { > > + 'schema': { > > + 'location': 0, > > + 'relevantForJS': 1, > > + 'innerWindowID': 2, > > + 'implementation': 3, > > + 'optimizations': 4, > > + 'line': 5, > > + 'column': 6, > > + 'category': 7, > > + 'subcategory': 8, > > + }, > > + 'data': [], > > + } > > + stackTable = { > > + 'schema': { > > + 'prefix': 0, > > + 'frame': 1, > > + }, > > + 'data': [], > > + } > > + stringTable = [] > > + > > + stackMap = dict() > > + def get_or_create_stack(frame, prefix): > > + key = f"{frame}" if prefix is None else f"{frame},{prefix}" > > + stack = stackMap.get(key) > > + if stack is None: > > + stack = len(stackTable['data']) > > + stackTable['data'].append([prefix, frame]) > > + stackMap[key] = stack > > + return stack > > + > > + frameMap = dict() > > + def get_or_create_frame(frameString): > > + frame = frameMap.get(frameString) > > + if frame is None: > > + frame = len(frameTable['data']) > > + location = len(stringTable) > > + stringTable.append(frameString) > > + # print('batman', frame, frameString) > > + > > + category = KERNEL_CATEGORY_INDEX if frameString.find('kallsyms') != -1 or frameString.find('/vmlinux') != 1 or frameString.endswith('.ko)') else USER_CATEGORY_INDEX > > + implementation = None > > + optimizations = None > > + line = None > > + relevantForJS = False > > + subcategory = None > > + innerWindowID = 0 > > + column = None > > + > > + frameTable['data'].append([ > > + location, > > + relevantForJS, > > + innerWindowID, > > + implementation, > > + optimizations, > > + line, > > + column, > > + category, > > + subcategory, > > + ]) > > + frameMap[frameString] = frame > > + return frame > > + > > + def addSample(threadName, stackArray, time): > > + nonlocal name > > + if name != threadName: > > + name = threadName > > + stack = reduce(lambda prefix, stackFrame: get_or_create_stack(get_or_create_frame(stackFrame), prefix), stackArray) > > + responsiveness = 0 > > + samples['data'].append([stack, time, responsiveness]) > > + > > + def finish(): > > + return { > > + "tid": tid, > > + "pid": pid, > > + "name": name, > > + "markers": markers, > > + "samples": samples, > > + "frameTable": frameTable, > > + "stackTable": stackTable, > > + "stringTable": stringTable, > > + "registerTime": 0, > > + "unregisterTime": None, > > + "processType": 'default' > > + } > > + > > + return { > > + "addSample": addSample, > > + "finish": finish > > + } > > + > > + threadMap = dict() > > + def _addThreadSample(pid, tid, threadName, time_stamp, stack): > > + thread = threadMap.get(tid) > > + if not thread: > > + thread = _createtread(pid, tid, threadName) > > + threadMap[tid] = thread > > + thread['addSample'](threadName, stack, time_stamp) > > + > > + lines = profile.split('\n') > > + > > + line_index = 0 > > + start_time = 0 > > + while line_index < len(lines): > > + line = lines[line_index] > > + line_index += 1 > > + if line == '' or line.startswith('#'): > > + continue > > + > > + sample_start_line = line > > + > > + sample_start_match = re.match(r'^(.*)\s+([\d.]+):', sample_start_line) > > + if not sample_start_match: > > + print(f'Could not parse line as the start of a sample in the "perf script" profile format: "{sample_start_line}"') > > + continue > > + > > + before_time_stamp = sample_start_match[1] > > + time_stamp = float(sample_start_match[2]) * 1000 > > + > > + threadNamePidAndTidMatch = re.match(r'^(.*)\s+(?:(\d+)\/)?(\d+)\b', before_time_stamp) > > + > > + if not threadNamePidAndTidMatch: > > + print('Could not parse line as the start of a sample in the "perf script" profile format: "%s"' % sampleStartLine) > > + continue > > + > > + threadName = threadNamePidAndTidMatch[1].strip() > > + pid = int(threadNamePidAndTidMatch[2] or 0) > > + tid = int(threadNamePidAndTidMatch[3] or 0) > > + startTime = 0 #workarround > > + if startTime == 0: > > + startTime = time_stamp > > + stack = [] > > + while line_index < len(lines): > > + stackFrameLine = lines[line_index] > > + line_index += 1 > > + if stackFrameLine.strip() == '': > > + break > > + stackFrameMatch = re.match(r'^\s*(\w+)\s*(.+) \(([^)]*)\)', stackFrameLine) > > + if stackFrameMatch: > > + rawFunc = stackFrameMatch[2] > > + mod = stackFrameMatch[3] > > + rawFunc = re.sub(r'\+0x[\da-f]+$', '', rawFunc) > > + > > + if rawFunc.startswith('('): > > + continue # skip process names > > + > > + if mod: > > + rawFunc += f' (in {mod})' > > + > > + stack.append(rawFunc) > > + if len(stack) != 0: > > + stack.reverse() > > + _addThreadSample(pid, tid, threadName, time_stamp, stack) > > + > > + thread_array = list(map(lambda thread: thread['finish'](), threadMap.values())) > > + > > + for thread in thread_array: > > + key = thread['samples']['schema']['time'] > > + thread['samples']['data'].sort(key=lambda data : float(data[key])) > > + > > + return { > > + 'meta': { > > + 'interval': 1, > > + 'processType': 0, > > + 'product': 'Firefox', > > + 'stackwalk': 1, > > + 'debug': 0, > > + 'gcpoison': 0, > > + 'asyncstack': 1, > > + 'startTime': startTime, > > + 'shutdownTime': None, > > + 'version': 24, > > + 'presymbolicated': True, > > + 'categories': CATEGORIES, > > + 'markerSchema': [] > > + }, > > + 'libs': [], > > + 'threads': thread_array, > > + 'processes': [], > > + 'pausedRanges': [] > > +} > > + > > +def main(): > > +# inputFile = input('Enter input file name: ') > > + with open('test.txt') as f: > > + profile = f.read() > > + isPerfScript = isPerfScriptFormat(profile) > > + output = convertPerfScriptProfile(profile) > > + json.dump(output, sys.stdout, indent=2) > > + # print('isPerfScript: {}'.format(isPerfScript)) > > + > > +if __name__ == '__main__': > > + main() > > ------------------------>cut<--------------------------- > > > > Thanks, > > Anup > > > > > > > > > > Thanks, > > > > Ian > > > > > > > > > > Thanks, > > > > > > Anup > > > > > > > > > > > > [1] https://github.com/firefox-devtools/profiler/blob/main/docs-developer/custom-importer.md > > > > > > > > > > > > > > Thanks, > > > > > > > > Ian > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you for your support and I'm looking forward to collaborating with you on this project! ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] Adding Support for Firefox's Gecko Profile Format 2023-06-28 20:11 ` Anup Sharma @ 2023-06-29 6:26 ` Namhyung Kim 2023-06-29 6:35 ` Adrian Hunter 0 siblings, 1 reply; 21+ messages in thread From: Namhyung Kim @ 2023-06-29 6:26 UTC (permalink / raw) To: Anup Sharma Cc: Ian Rogers, linux-perf-users, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Mark Rutland, Alexander Shishkin, Jiri Olsa, Adrian Hunter, linux-kernel Hi Anup, On Wed, Jun 28, 2023 at 1:11 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > On Wed, Jun 14, 2023 at 01:17:52PM -0700, Ian Rogers wrote: > > On Wed, Jun 14, 2023 at 1:14 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > > > > > On Thu, Jun 08, 2023 at 06:23:49PM +0530, Anup Sharma wrote: > > > I wanted to take a moment to provide you with an update on the progress > > > of our Firefox Gecko converter work. While I must emphasize that the code > > > I'm sharing is not the final version, I wanted to share the advancements > > > I have made thus far. > > > > > > This script can generate a JSON format from the output of the "perf script" command. > > > I attempted to load this JSON file into profile.firefox.com, and although it > > > successfully loaded, the call tree are not visible. I'm certain this issue > > > is related to the format of the JSON file or if there is another underlying > > > cause. I will continue investigating to determine the cause of this problem. > > > > Great Anup, progress can be frustrating slow at first but it is a good > > milestone to be generating output and having firefox profiler consume > > it. You can open up the JavaScript console for the firefox profiler > > and it will give some debug output. This should hopefully help you > > make progress. > > This week I tried playing with perf-script-python, the first challenge was > figuring out how to parse the data coming from the process_events. > Understanding the structure and extracting the necessary information > proved to be more complex than anticipated. This required me to spend > a significant amount of time researching and experimenting with different > parsing techniques. I'm not sure what kind of parsing you meant. IIRC the sample data is in a dict and all the information should be available there. Maybe we missed some new sample data types though. > > The second challenge revolved around the usage of event hooks provided with the perf script > python event handlers. I found myself deliberating between two approaches. The first > approach involved creating custom functions that would be called using the event > handlers. These functions would then save the data in an organized format within > globally created data structures. The alternative approach was to write the entire > logic inside the event handlers themselves. > > Additionally, I contemplated whether it would be more suitable to handle the creation of > a Gecko format for JSON and the profile format within the same script or to separate > them into different scripts. > > I will discuss this points during tomorrow's office hour. > > However, I have gained a deeper understanding of the problem at hand and will use this > knowledge to make more informed decisions and progress more effectively in the coming weeks. Sounds like you did a good amount of research on how perf script works. We can talk about the details in the meeting. Thanks, Namhyung ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] Adding Support for Firefox's Gecko Profile Format 2023-06-29 6:26 ` Namhyung Kim @ 2023-06-29 6:35 ` Adrian Hunter 2023-07-04 14:38 ` Anup Sharma 0 siblings, 1 reply; 21+ messages in thread From: Adrian Hunter @ 2023-06-29 6:35 UTC (permalink / raw) To: Namhyung Kim, Anup Sharma Cc: Ian Rogers, linux-perf-users, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Mark Rutland, Alexander Shishkin, Jiri Olsa, linux-kernel On 29/06/23 09:26, Namhyung Kim wrote: > Hi Anup, > > On Wed, Jun 28, 2023 at 1:11 PM Anup Sharma <anupnewsmail@gmail.com> wrote: >> >> On Wed, Jun 14, 2023 at 01:17:52PM -0700, Ian Rogers wrote: >>> On Wed, Jun 14, 2023 at 1:14 PM Anup Sharma <anupnewsmail@gmail.com> wrote: >>>> >>>> On Thu, Jun 08, 2023 at 06:23:49PM +0530, Anup Sharma wrote: >>>> I wanted to take a moment to provide you with an update on the progress >>>> of our Firefox Gecko converter work. While I must emphasize that the code >>>> I'm sharing is not the final version, I wanted to share the advancements >>>> I have made thus far. >>>> >>>> This script can generate a JSON format from the output of the "perf script" command. >>>> I attempted to load this JSON file into profile.firefox.com, and although it >>>> successfully loaded, the call tree are not visible. I'm certain this issue >>>> is related to the format of the JSON file or if there is another underlying >>>> cause. I will continue investigating to determine the cause of this problem. >>> >>> Great Anup, progress can be frustrating slow at first but it is a good >>> milestone to be generating output and having firefox profiler consume >>> it. You can open up the JavaScript console for the firefox profiler >>> and it will give some debug output. This should hopefully help you >>> make progress. >> >> This week I tried playing with perf-script-python, the first challenge was >> figuring out how to parse the data coming from the process_events. >> Understanding the structure and extracting the necessary information >> proved to be more complex than anticipated. This required me to spend >> a significant amount of time researching and experimenting with different >> parsing techniques. > > I'm not sure what kind of parsing you meant. IIRC the sample data is > in a dict and all the information should be available there. Maybe we > missed some new sample data types though. Most data is there. There are existing scripts in tools/perf/scripts/python/ for reference. There is also the dlfilter API: https://perf.wiki.kernel.org/index.php/Latest_Manual_Page_of_perf-dlfilter.1 > >> >> The second challenge revolved around the usage of event hooks provided with the perf script >> python event handlers. I found myself deliberating between two approaches. The first >> approach involved creating custom functions that would be called using the event >> handlers. These functions would then save the data in an organized format within >> globally created data structures. The alternative approach was to write the entire >> logic inside the event handlers themselves. >> >> Additionally, I contemplated whether it would be more suitable to handle the creation of >> a Gecko format for JSON and the profile format within the same script or to separate >> them into different scripts. >> >> I will discuss this points during tomorrow's office hour. >> >> However, I have gained a deeper understanding of the problem at hand and will use this >> knowledge to make more informed decisions and progress more effectively in the coming weeks. > > Sounds like you did a good amount of research on how perf script > works. We can talk about the details in the meeting. > > Thanks, > Namhyung ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] Adding Support for Firefox's Gecko Profile Format 2023-06-29 6:35 ` Adrian Hunter @ 2023-07-04 14:38 ` Anup Sharma 2023-07-04 17:52 ` Anup Sharma 0 siblings, 1 reply; 21+ messages in thread From: Anup Sharma @ 2023-07-04 14:38 UTC (permalink / raw) To: Adrian Hunter Cc: Namhyung Kim, Anup Sharma, Ian Rogers, linux-perf-users, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Mark Rutland, Alexander Shishkin, Jiri Olsa, linux-kernel On Thu, Jun 29, 2023 at 09:35:22AM +0300, Adrian Hunter wrote: > On 29/06/23 09:26, Namhyung Kim wrote: > > Hi Anup, > > > > On Wed, Jun 28, 2023 at 1:11 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > >> > >> On Wed, Jun 14, 2023 at 01:17:52PM -0700, Ian Rogers wrote: > >>> On Wed, Jun 14, 2023 at 1:14 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > >>>> > >>>> On Thu, Jun 08, 2023 at 06:23:49PM +0530, Anup Sharma wrote: > >>>> I wanted to take a moment to provide you with an update on the progress > >>>> of our Firefox Gecko converter work. While I must emphasize that the code > >>>> I'm sharing is not the final version, I wanted to share the advancements > >>>> I have made thus far. > >>>> > >>>> This script can generate a JSON format from the output of the "perf script" command. > >>>> I attempted to load this JSON file into profile.firefox.com, and although it > >>>> successfully loaded, the call tree are not visible. I'm certain this issue > >>>> is related to the format of the JSON file or if there is another underlying > >>>> cause. I will continue investigating to determine the cause of this problem. > >>> > >>> Great Anup, progress can be frustrating slow at first but it is a good > >>> milestone to be generating output and having firefox profiler consume > >>> it. You can open up the JavaScript console for the firefox profiler > >>> and it will give some debug output. This should hopefully help you > >>> make progress. > >> > >> This week I tried playing with perf-script-python, the first challenge was > >> figuring out how to parse the data coming from the process_events. > >> Understanding the structure and extracting the necessary information > >> proved to be more complex than anticipated. This required me to spend > >> a significant amount of time researching and experimenting with different > >> parsing techniques. > > > > I'm not sure what kind of parsing you meant. IIRC the sample data is > > in a dict and all the information should be available there. Maybe we > > missed some new sample data types though. > > Most data is there. There are existing scripts in > tools/perf/scripts/python/ for reference. > > There is also the dlfilter API: > > https://perf.wiki.kernel.org/index.php/Latest_Manual_Page_of_perf-dlfilter.1 > Hello everyone, I hope you're all doing well. I have been working on the using process_events function and have made progress in developing the converter script. Currently, I am in the testing phase. However, I have encountered a problem after performing some recent compilations. I am no longer receiving the complete callchains in the output as I used to. For Example, when I ran the command perf script -F +pid, the output would include detailed information like the following example: perf-exec 132554/132554 171854.674582: 356 cycles:P: ffffffff8fab3fc6 arch_static_branch+0x6 (inlined) ffffffff8fab3fc6 static_key_false+0x6 (inlined) ffffffff8fab3fc6 native_write_msr+0x6 (/lib/modules/6.4.0-rc1/build/vmlinux) ffffffff8fa12ca5 intel_pmu_enable_all+0x15 (/lib/modules/6.4.0-rc1/build/vmlinux) However, in my current situation, the output is limited to the following: perf 108107/108107 [000] 67650.031827: 1 cycles:P: ffffffff83ab3fc4 native_write_msr+0x4 (/lib/modules/6.4.0-rc1/build/vmlinux) perf 108107/108107 [000] 67650.031832: 1 cycles:P: ffffffff83ab3fc4 native_write_msr+0x4 (/lib/modules/6.4.0-rc1/build/vmlinux) It seems to be issue in perf record. I would appreciate any suggestions or assistance in resolving this issue. Thank you all for your help. Regards, Anup > > > >> > >> The second challenge revolved around the usage of event hooks provided with the perf script > >> python event handlers. I found myself deliberating between two approaches. The first > >> approach involved creating custom functions that would be called using the event > >> handlers. These functions would then save the data in an organized format within > >> globally created data structures. The alternative approach was to write the entire > >> logic inside the event handlers themselves. > >> > >> Additionally, I contemplated whether it would be more suitable to handle the creation of > >> a Gecko format for JSON and the profile format within the same script or to separate > >> them into different scripts. > >> > >> I will discuss this points during tomorrow's office hour. > >> > >> However, I have gained a deeper understanding of the problem at hand and will use this > >> knowledge to make more informed decisions and progress more effectively in the coming weeks. > > > > Sounds like you did a good amount of research on how perf script > > works. We can talk about the details in the meeting. > > > > Thanks, > > Namhyung > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] Adding Support for Firefox's Gecko Profile Format 2023-07-04 14:38 ` Anup Sharma @ 2023-07-04 17:52 ` Anup Sharma 2023-07-26 18:04 ` Anup Sharma 0 siblings, 1 reply; 21+ messages in thread From: Anup Sharma @ 2023-07-04 17:52 UTC (permalink / raw) To: Anup Sharma Cc: Adrian Hunter, Namhyung Kim, Ian Rogers, linux-perf-users, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Mark Rutland, Alexander Shishkin, Jiri Olsa, linux-kernel On Tue, Jul 04, 2023 at 08:08:11PM +0530, Anup Sharma wrote: > On Thu, Jun 29, 2023 at 09:35:22AM +0300, Adrian Hunter wrote: > > On 29/06/23 09:26, Namhyung Kim wrote: > > > Hi Anup, > > > > > > On Wed, Jun 28, 2023 at 1:11 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > >> > > >> On Wed, Jun 14, 2023 at 01:17:52PM -0700, Ian Rogers wrote: > > >>> On Wed, Jun 14, 2023 at 1:14 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > >>>> > > >>>> On Thu, Jun 08, 2023 at 06:23:49PM +0530, Anup Sharma wrote: > > >>>> I wanted to take a moment to provide you with an update on the progress > > >>>> of our Firefox Gecko converter work. While I must emphasize that the code > > >>>> I'm sharing is not the final version, I wanted to share the advancements > > >>>> I have made thus far. > > >>>> > > >>>> This script can generate a JSON format from the output of the "perf script" command. > > >>>> I attempted to load this JSON file into profile.firefox.com, and although it > > >>>> successfully loaded, the call tree are not visible. I'm certain this issue > > >>>> is related to the format of the JSON file or if there is another underlying > > >>>> cause. I will continue investigating to determine the cause of this problem. > > >>> > > >>> Great Anup, progress can be frustrating slow at first but it is a good > > >>> milestone to be generating output and having firefox profiler consume > > >>> it. You can open up the JavaScript console for the firefox profiler > > >>> and it will give some debug output. This should hopefully help you > > >>> make progress. > > >> > > >> This week I tried playing with perf-script-python, the first challenge was > > >> figuring out how to parse the data coming from the process_events. > > >> Understanding the structure and extracting the necessary information > > >> proved to be more complex than anticipated. This required me to spend > > >> a significant amount of time researching and experimenting with different > > >> parsing techniques. > > > > > > I'm not sure what kind of parsing you meant. IIRC the sample data is > > > in a dict and all the information should be available there. Maybe we > > > missed some new sample data types though. > > > > Most data is there. There are existing scripts in > > tools/perf/scripts/python/ for reference. > > > > There is also the dlfilter API: > > > > https://perf.wiki.kernel.org/index.php/Latest_Manual_Page_of_perf-dlfilter.1 > > > Hello everyone, > > I hope you're all doing well. I have been working on the using process_events function and have made > progress in developing the converter script. Currently, I am in the testing phase. However, I > have encountered a problem after performing some recent compilations. I am no longer receiving > the complete callchains in the output as I used to. > For Example, when I ran the command perf script -F +pid, the output would include detailed > information like the following example: > > perf-exec 132554/132554 171854.674582: 356 cycles:P: > ffffffff8fab3fc6 arch_static_branch+0x6 (inlined) > ffffffff8fab3fc6 static_key_false+0x6 (inlined) > ffffffff8fab3fc6 native_write_msr+0x6 (/lib/modules/6.4.0-rc1/build/vmlinux) > ffffffff8fa12ca5 intel_pmu_enable_all+0x15 (/lib/modules/6.4.0-rc1/build/vmlinux) > > However, in my current situation, the output is limited to the following: > > perf 108107/108107 [000] 67650.031827: 1 cycles:P: > ffffffff83ab3fc4 native_write_msr+0x4 (/lib/modules/6.4.0-rc1/build/vmlinux) > perf 108107/108107 [000] 67650.031832: 1 cycles:P: > ffffffff83ab3fc4 native_write_msr+0x4 (/lib/modules/6.4.0-rc1/build/vmlinux) > > It seems to be issue in perf record. I would appreciate any suggestions or assistance > in resolving this issue. Thank you all for your help. My apologies, I realized that I had mistakenly not included the '-g' option, and it is now working fine." > Regards, > Anup > > > > > >> > > >> The second challenge revolved around the usage of event hooks provided with the perf script > > >> python event handlers. I found myself deliberating between two approaches. The first > > >> approach involved creating custom functions that would be called using the event > > >> handlers. These functions would then save the data in an organized format within > > >> globally created data structures. The alternative approach was to write the entire > > >> logic inside the event handlers themselves. > > >> > > >> Additionally, I contemplated whether it would be more suitable to handle the creation of > > >> a Gecko format for JSON and the profile format within the same script or to separate > > >> them into different scripts. > > >> > > >> I will discuss this points during tomorrow's office hour. > > >> > > >> However, I have gained a deeper understanding of the problem at hand and will use this > > >> knowledge to make more informed decisions and progress more effectively in the coming weeks. > > > > > > Sounds like you did a good amount of research on how perf script > > > works. We can talk about the details in the meeting. > > > > > > Thanks, > > > Namhyung > > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] Adding Support for Firefox's Gecko Profile Format 2023-07-04 17:52 ` Anup Sharma @ 2023-07-26 18:04 ` Anup Sharma 2023-07-26 19:22 ` Arnaldo Carvalho de Melo 0 siblings, 1 reply; 21+ messages in thread From: Anup Sharma @ 2023-07-26 18:04 UTC (permalink / raw) To: Anup Sharma Cc: Adrian Hunter, Namhyung Kim, Ian Rogers, linux-perf-users, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo, Mark Rutland, Alexander Shishkin, Jiri Olsa, linux-kernel On Tue, Jul 04, 2023 at 11:22:14PM +0530, Anup Sharma wrote: > On Tue, Jul 04, 2023 at 08:08:11PM +0530, Anup Sharma wrote: > > On Thu, Jun 29, 2023 at 09:35:22AM +0300, Adrian Hunter wrote: > > > On 29/06/23 09:26, Namhyung Kim wrote: > > > > Hi Anup, > > > > > > > > On Wed, Jun 28, 2023 at 1:11 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > > >> > > > >> On Wed, Jun 14, 2023 at 01:17:52PM -0700, Ian Rogers wrote: > > > >>> On Wed, Jun 14, 2023 at 1:14 PM Anup Sharma <anupnewsmail@gmail.com> wrote: > > > >>>> > > > >>>> On Thu, Jun 08, 2023 at 06:23:49PM +0530, Anup Sharma wrote: > > > >>>> I wanted to take a moment to provide you with an update on the progress > > > >>>> of our Firefox Gecko converter work. While I must emphasize that the code > > > >>>> I'm sharing is not the final version, I wanted to share the advancements > > > >>>> I have made thus far. > > > >>>> > > > >>>> This script can generate a JSON format from the output of the "perf script" command. > > > >>>> I attempted to load this JSON file into profile.firefox.com, and although it > > > >>>> successfully loaded, the call tree are not visible. I'm certain this issue > > > >>>> is related to the format of the JSON file or if there is another underlying > > > >>>> cause. I will continue investigating to determine the cause of this problem. > > > >>> > > > >>> Great Anup, progress can be frustrating slow at first but it is a good > > > >>> milestone to be generating output and having firefox profiler consume > > > >>> it. You can open up the JavaScript console for the firefox profiler > > > >>> and it will give some debug output. This should hopefully help you > > > >>> make progress. > > > >> > > > >> This week I tried playing with perf-script-python, the first challenge was > > > >> figuring out how to parse the data coming from the process_events. > > > >> Understanding the structure and extracting the necessary information > > > >> proved to be more complex than anticipated. This required me to spend > > > >> a significant amount of time researching and experimenting with different > > > >> parsing techniques. > > > > > > > > I'm not sure what kind of parsing you meant. IIRC the sample data is > > > > in a dict and all the information should be available there. Maybe we > > > > missed some new sample data types though. > > > > > > Most data is there. There are existing scripts in > > > tools/perf/scripts/python/ for reference. > > > > > > There is also the dlfilter API: > > > > > > https://perf.wiki.kernel.org/index.php/Latest_Manual_Page_of_perf-dlfilter.1 > > > > > Hello everyone, > > > > I hope you're all doing well. I have been working on the using process_events function and have made > > progress in developing the converter script. Currently, I am in the testing phase. However, I > > have encountered a problem after performing some recent compilations. I am no longer receiving > > the complete callchains in the output as I used to. > > For Example, when I ran the command perf script -F +pid, the output would include detailed > > information like the following example: > > > > perf-exec 132554/132554 171854.674582: 356 cycles:P: > > ffffffff8fab3fc6 arch_static_branch+0x6 (inlined) > > ffffffff8fab3fc6 static_key_false+0x6 (inlined) > > ffffffff8fab3fc6 native_write_msr+0x6 (/lib/modules/6.4.0-rc1/build/vmlinux) > > ffffffff8fa12ca5 intel_pmu_enable_all+0x15 (/lib/modules/6.4.0-rc1/build/vmlinux) > > > > However, in my current situation, the output is limited to the following: > > > > perf 108107/108107 [000] 67650.031827: 1 cycles:P: > > ffffffff83ab3fc4 native_write_msr+0x4 (/lib/modules/6.4.0-rc1/build/vmlinux) > > perf 108107/108107 [000] 67650.031832: 1 cycles:P: > > ffffffff83ab3fc4 native_write_msr+0x4 (/lib/modules/6.4.0-rc1/build/vmlinux) > > > > It seems to be issue in perf record. I would appreciate any suggestions or assistance > > in resolving this issue. Thank you all for your help. > > My apologies, I realized that I had mistakenly not included the '-g' option, > and it is now working fine." Dear all, I want to share a recent development in our project that I have been working on this week. I have successfully implemented a new feature that allows users to directly open the browser and visualize the output of perf.data on Firefox Profiler. To enhance user experience, I have made sure this option is enabled by default. Additionally, I've also added a handy command line argument, "--save-only," which allows users to save the output file as needed. This enhancement replaces the previous standard output method, providing a much-improved experience. Moreover, Everything is processed locally, and no data is sent to any server. During this implementation, I had the opportunity to delve into several exciting topics, including threading, HTTP server, and CORS mechanisms. The learning process has been both challenging and rewarding. Currently, I am in the process of performing some clean-up tasks. Once that is complete, I will be sharing the next version of the series, which includes this fantastic new feature, with all of you very soon. Thank you for your ongoing support and encouragement. Thanks, Anup > > Regards, > > Anup > > > > > > > >> > > > >> The second challenge revolved around the usage of event hooks provided with the perf script > > > >> python event handlers. I found myself deliberating between two approaches. The first > > > >> approach involved creating custom functions that would be called using the event > > > >> handlers. These functions would then save the data in an organized format within > > > >> globally created data structures. The alternative approach was to write the entire > > > >> logic inside the event handlers themselves. > > > >> > > > >> Additionally, I contemplated whether it would be more suitable to handle the creation of > > > >> a Gecko format for JSON and the profile format within the same script or to separate > > > >> them into different scripts. > > > >> > > > >> I will discuss this points during tomorrow's office hour. > > > >> > > > >> However, I have gained a deeper understanding of the problem at hand and will use this > > > >> knowledge to make more informed decisions and progress more effectively in the coming weeks. > > > > > > > > Sounds like you did a good amount of research on how perf script > > > > works. We can talk about the details in the meeting. > > > > > > > > Thanks, > > > > Namhyung > > > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] Adding Support for Firefox's Gecko Profile Format 2023-07-26 18:04 ` Anup Sharma @ 2023-07-26 19:22 ` Arnaldo Carvalho de Melo 2023-08-02 18:59 ` Anup Sharma 0 siblings, 1 reply; 21+ messages in thread From: Arnaldo Carvalho de Melo @ 2023-07-26 19:22 UTC (permalink / raw) To: Anup Sharma Cc: Adrian Hunter, Namhyung Kim, Ian Rogers, linux-perf-users, Peter Zijlstra, Ingo Molnar, Mark Rutland, Alexander Shishkin, Jiri Olsa, linux-kernel Em Wed, Jul 26, 2023 at 11:34:17PM +0530, Anup Sharma escreveu: > Dear all, > > I want to share a recent development in our project that > I have been working on this week. I have successfully > implemented a new feature that allows users to directly > open the browser and visualize the output of perf.data > on Firefox Profiler. Excellent news! > To enhance user experience, I have made sure this option > is enabled by default. Additionally, I've also added a > handy command line argument, "--save-only," which allows > users to save the output file as needed. This enhancement > replaces the previous standard output method, providing a > much-improved experience. > > Moreover, Everything is processed locally, and no data is > sent to any server. That addresses the worry about sending possibly sensitive data over the network by default, great. > During this implementation, I had the opportunity to delve > into several exciting topics, including threading, HTTP server, > and CORS mechanisms. The learning process has been both > challenging and rewarding. Cool that you're so excited learning all this. > Currently, I am in the process of performing some clean-up tasks. > Once that is complete, I will be sharing the next version of the > series, which includes this fantastic new feature, with all of > you very soon. > > Thank you for your ongoing support and encouragement. Keep it up! - Arnaldo > Thanks, > Anup > > > > Regards, > > > Anup > > > > > > > > > >> > > > > >> The second challenge revolved around the usage of event hooks provided with the perf script > > > > >> python event handlers. I found myself deliberating between two approaches. The first > > > > >> approach involved creating custom functions that would be called using the event > > > > >> handlers. These functions would then save the data in an organized format within > > > > >> globally created data structures. The alternative approach was to write the entire > > > > >> logic inside the event handlers themselves. > > > > >> > > > > >> Additionally, I contemplated whether it would be more suitable to handle the creation of > > > > >> a Gecko format for JSON and the profile format within the same script or to separate > > > > >> them into different scripts. > > > > >> > > > > >> I will discuss this points during tomorrow's office hour. > > > > >> > > > > >> However, I have gained a deeper understanding of the problem at hand and will use this > > > > >> knowledge to make more informed decisions and progress more effectively in the coming weeks. > > > > > > > > > > Sounds like you did a good amount of research on how perf script > > > > > works. We can talk about the details in the meeting. > > > > > > > > > > Thanks, > > > > > Namhyung > > > > -- - Arnaldo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] Adding Support for Firefox's Gecko Profile Format 2023-07-26 19:22 ` Arnaldo Carvalho de Melo @ 2023-08-02 18:59 ` Anup Sharma 2023-08-09 20:01 ` Anup Sharma 0 siblings, 1 reply; 21+ messages in thread From: Anup Sharma @ 2023-08-02 18:59 UTC (permalink / raw) To: Arnaldo Carvalho de Melo Cc: Anup Sharma, Adrian Hunter, Namhyung Kim, Ian Rogers, linux-perf-users, Peter Zijlstra, Ingo Molnar, Mark Rutland, Alexander Shishkin, Jiri Olsa, linux-kernel On Wed, Jul 26, 2023 at 04:22:51PM -0300, Arnaldo Carvalho de Melo wrote: > Em Wed, Jul 26, 2023 at 11:34:17PM +0530, Anup Sharma escreveu: > > Dear all, > > > > I want to share a recent development in our project that > > I have been working on this week. I have successfully > > implemented a new feature that allows users to directly > > open the browser and visualize the output of perf.data > > on Firefox Profiler. > > Excellent news! > > > To enhance user experience, I have made sure this option > > is enabled by default. Additionally, I've also added a > > handy command line argument, "--save-only," which allows > > users to save the output file as needed. This enhancement > > replaces the previous standard output method, providing a > > much-improved experience. > > > > Moreover, Everything is processed locally, and no data is > > sent to any server. > > That addresses the worry about sending possibly sensitive data over the > network by default, great. > > > During this implementation, I had the opportunity to delve > > into several exciting topics, including threading, HTTP server, > > and CORS mechanisms. The learning process has been both > > challenging and rewarding. > > Cool that you're so excited learning all this. > > > Currently, I am in the process of performing some clean-up tasks. > > Once that is complete, I will be sharing the next version of the > > series, which includes this fantastic new feature, with all of > > you very soon. > > > > Thank you for your ongoing support and encouragement. > > Keep it up! This week, I'm working on documenting the gecko converter for our wiki. I've dedicated a significant amount of time to think about the content and how to approach it. So far, I've written sections covering the introduction to Firefox Profiler and the formatting guidelines(not published yet). However, I've left the usage part blank because I'm uncertain about how to enable gecko.py arguments when executing the "perf script gecko" command. I've looked at examples like flamegraph-report and sctop-report, but I'm still confused. I attempted to enable the script to take the gecko.py command example "perf script report gecko --save-only=profiler.json," but this caused issues when running "perf script gecko -a sleep 3" again. This has left me feeling rather puzzled, and I would appreciate discussing this during our office hour, or any help here. Due to this blocker, I haven't been able to send the updated patch for test_gecko yet. However, I hope I'll find a solution soon and overcome this obstacle. > - Arnaldo > > > Thanks, > > Anup > > > > > > Regards, > > > > Anup > > > > > > > > > > > >> > > > > > >> The second challenge revolved around the usage of event hooks provided with the perf script > > > > > >> python event handlers. I found myself deliberating between two approaches. The first > > > > > >> approach involved creating custom functions that would be called using the event > > > > > >> handlers. These functions would then save the data in an organized format within > > > > > >> globally created data structures. The alternative approach was to write the entire > > > > > >> logic inside the event handlers themselves. > > > > > >> > > > > > >> Additionally, I contemplated whether it would be more suitable to handle the creation of > > > > > >> a Gecko format for JSON and the profile format within the same script or to separate > > > > > >> them into different scripts. > > > > > >> > > > > > >> I will discuss this points during tomorrow's office hour. > > > > > >> > > > > > >> However, I have gained a deeper understanding of the problem at hand and will use this > > > > > >> knowledge to make more informed decisions and progress more effectively in the coming weeks. > > > > > > > > > > > > Sounds like you did a good amount of research on how perf script > > > > > > works. We can talk about the details in the meeting. > > > > > > > > > > > > Thanks, > > > > > > Namhyung > > > > > > > -- > > - Arnaldo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] Adding Support for Firefox's Gecko Profile Format 2023-08-02 18:59 ` Anup Sharma @ 2023-08-09 20:01 ` Anup Sharma 2023-08-16 17:10 ` Anup Sharma 0 siblings, 1 reply; 21+ messages in thread From: Anup Sharma @ 2023-08-09 20:01 UTC (permalink / raw) To: Arnaldo Carvalho de Melo, Namhyung Kim, Ian Rogers Cc: Adrian Hunter, linux-perf-users, Peter Zijlstra, Ingo Molnar, Mark Rutland, Alexander Shishkin, Jiri Olsa, linux-kernel, anupnewsmail On Thu, Aug 03, 2023 at 12:29:51AM +0530, Anup Sharma wrote: > On Wed, Jul 26, 2023 at 04:22:51PM -0300, Arnaldo Carvalho de Melo wrote: > > Em Wed, Jul 26, 2023 at 11:34:17PM +0530, Anup Sharma escreveu: > > > Dear all, > > > > > > I want to share a recent development in our project that > > > I have been working on this week. I have successfully > > > implemented a new feature that allows users to directly > > > open the browser and visualize the output of perf.data > > > on Firefox Profiler. > > > > Excellent news! > > > > > To enhance user experience, I have made sure this option > > > is enabled by default. Additionally, I've also added a > > > handy command line argument, "--save-only," which allows > > > users to save the output file as needed. This enhancement > > > replaces the previous standard output method, providing a > > > much-improved experience. > > > > > > Moreover, Everything is processed locally, and no data is > > > sent to any server. > > > > That addresses the worry about sending possibly sensitive data over the > > network by default, great. > > > > > During this implementation, I had the opportunity to delve > > > into several exciting topics, including threading, HTTP server, > > > and CORS mechanisms. The learning process has been both > > > challenging and rewarding. > > > > Cool that you're so excited learning all this. > > > > > Currently, I am in the process of performing some clean-up tasks. > > > Once that is complete, I will be sharing the next version of the > > > series, which includes this fantastic new feature, with all of > > > you very soon. > > > > > > Thank you for your ongoing support and encouragement. > > > > Keep it up! > > This week, I'm working on documenting the gecko converter for our wiki. > I've dedicated a significant amount of time to think about the content > and how to approach it. So far, I've written sections covering the > introduction to Firefox Profiler and the formatting guidelines(not > published yet). > > However, I've left the usage part blank because I'm uncertain about how > to enable gecko.py arguments when executing the "perf script gecko" command. > I've looked at examples like flamegraph-report and sctop-report, but I'm > still confused. I attempted to enable the script to take the gecko.py > command example "perf script report gecko --save-only=profiler.json," > but this caused issues when running "perf script gecko -a sleep 3" again. > This has left me feeling rather puzzled, and I would appreciate discussing > this during our office hour, or any help here. > > Due to this blocker, I haven't been able to send the updated patch > for test_gecko yet. However, I hope I'll find a solution soon and overcome > this obstacle. Hello All, As discussed during last office hour, I was little occupied this week, so I haven't been able to make much progress on my TODO list. Nonetheless, I've resent a server patch and test script patch for your review. If I can get it reviewed it would be great to make further progress. I'm also working on writing documentation for our wiki. I've already written a bit, and you can see it here: https://perf.wiki.kernel.org/index.php/Tutorial#Firefox_Profiler. If you have any thoughts or suggestions, I'd appreciate hearing them. I'll add more details once I've sorted out some issues I'm having with certain commands. This week, I do not have a specific agenda. Hence, if we opt to skip this week's office hour, it wouldn't pose any inconvenience. Nevertheless, I am available if there are matters you wish to discuss. Here's what I have in mind to work on next: 1. Making sure that gecko script args work with commands, like (perf script gecko --save-only=profile.json) or (perf script gecko --kernel-color=green),which aren't working right now. 2. Trying out "perf record -p firefox" or using it with a specific process ID to see if the current script supports these recording commands. 3. Adding page fault and one more event in stackTable in gecko script. 4. Making the documentation better. Thanks, Anup > > - Arnaldo > > > > > Thanks, > > > Anup > > > > > > > > Regards, > > > > > Anup > > > > > > > > > > > > > >> > > > > > > >> The second challenge revolved around the usage of event hooks provided with the perf script > > > > > > >> python event handlers. I found myself deliberating between two approaches. The first > > > > > > >> approach involved creating custom functions that would be called using the event > > > > > > >> handlers. These functions would then save the data in an organized format within > > > > > > >> globally created data structures. The alternative approach was to write the entire > > > > > > >> logic inside the event handlers themselves. > > > > > > >> > > > > > > >> Additionally, I contemplated whether it would be more suitable to handle the creation of > > > > > > >> a Gecko format for JSON and the profile format within the same script or to separate > > > > > > >> them into different scripts. > > > > > > >> > > > > > > >> I will discuss this points during tomorrow's office hour. > > > > > > >> > > > > > > >> However, I have gained a deeper understanding of the problem at hand and will use this > > > > > > >> knowledge to make more informed decisions and progress more effectively in the coming weeks. > > > > > > > > > > > > > > Sounds like you did a good amount of research on how perf script > > > > > > > works. We can talk about the details in the meeting. > > > > > > > > > > > > > > Thanks, > > > > > > > Namhyung > > > > > > > > > > -- > > > > - Arnaldo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] Adding Support for Firefox's Gecko Profile Format 2023-08-09 20:01 ` Anup Sharma @ 2023-08-16 17:10 ` Anup Sharma 2023-09-18 3:39 ` Anup Sharma 0 siblings, 1 reply; 21+ messages in thread From: Anup Sharma @ 2023-08-16 17:10 UTC (permalink / raw) To: Anup Sharma Cc: Arnaldo Carvalho de Melo, Namhyung Kim, Ian Rogers, Adrian Hunter, linux-perf-users, Peter Zijlstra, Ingo Molnar, Mark Rutland, Alexander Shishkin, Jiri Olsa, linux-kernel On Thu, Aug 10, 2023 at 01:31:09AM +0530, Anup Sharma wrote: > On Thu, Aug 03, 2023 at 12:29:51AM +0530, Anup Sharma wrote: > > On Wed, Jul 26, 2023 at 04:22:51PM -0300, Arnaldo Carvalho de Melo wrote: > > > Em Wed, Jul 26, 2023 at 11:34:17PM +0530, Anup Sharma escreveu: > > > > Dear all, > > > > > > > > I want to share a recent development in our project that > > > > I have been working on this week. I have successfully > > > > implemented a new feature that allows users to directly > > > > open the browser and visualize the output of perf.data > > > > on Firefox Profiler. > > > > > > Excellent news! > > > > > > > To enhance user experience, I have made sure this option > > > > is enabled by default. Additionally, I've also added a > > > > handy command line argument, "--save-only," which allows > > > > users to save the output file as needed. This enhancement > > > > replaces the previous standard output method, providing a > > > > much-improved experience. > > > > > > > > Moreover, Everything is processed locally, and no data is > > > > sent to any server. > > > > > > That addresses the worry about sending possibly sensitive data over the > > > network by default, great. > > > > > > > During this implementation, I had the opportunity to delve > > > > into several exciting topics, including threading, HTTP server, > > > > and CORS mechanisms. The learning process has been both > > > > challenging and rewarding. > > > > > > Cool that you're so excited learning all this. > > > > > > > Currently, I am in the process of performing some clean-up tasks. > > > > Once that is complete, I will be sharing the next version of the > > > > series, which includes this fantastic new feature, with all of > > > > you very soon. > > > > > > > > Thank you for your ongoing support and encouragement. > > > > > > Keep it up! > > > > This week, I'm working on documenting the gecko converter for our wiki. > > I've dedicated a significant amount of time to think about the content > > and how to approach it. So far, I've written sections covering the > > introduction to Firefox Profiler and the formatting guidelines(not > > published yet). > > > > However, I've left the usage part blank because I'm uncertain about how > > to enable gecko.py arguments when executing the "perf script gecko" command. > > I've looked at examples like flamegraph-report and sctop-report, but I'm > > still confused. I attempted to enable the script to take the gecko.py > > command example "perf script report gecko --save-only=profiler.json," > > but this caused issues when running "perf script gecko -a sleep 3" again. > > This has left me feeling rather puzzled, and I would appreciate discussing > > this during our office hour, or any help here. > > > > Due to this blocker, I haven't been able to send the updated patch > > for test_gecko yet. However, I hope I'll find a solution soon and overcome > > this obstacle. > > Hello All, > As discussed during last office hour, I was little occupied this week, so > I haven't been able to make much progress on my TODO list. Nonetheless, > I've resent a server patch and test script patch for your review. If I can > get it reviewed it would be great to make further progress. > > I'm also working on writing documentation for our wiki. I've already written a > bit, and you can see it here: https://perf.wiki.kernel.org/index.php/Tutorial#Firefox_Profiler. > If you have any thoughts or suggestions, I'd appreciate hearing them. I'll add > more details once I've sorted out some issues I'm having with certain commands. > > This week, I do not have a specific agenda. Hence, if we opt to skip this week's > office hour, it wouldn't pose any inconvenience. Nevertheless, I am available if > there are matters you wish to discuss. > > Here's what I have in mind to work on next: > > 1. Making sure that gecko script args work with commands, like > (perf script gecko --save-only=profile.json) or (perf script gecko --kernel-color=green),which > aren't working right now. I have done this part and sent the patch for review. https://lore.kernel.org/all/ZNf7W+EIrrCSHZN0@yoga/ > 2. Trying out "perf record -p firefox" or using it with a specific process ID to > see if the current script supports these recording commands. This is working with the existing script. I've tested it and documented it as well. https://perf.wiki.kernel.org/index.php/Tutorial#Firefox_Profiler > 3. Adding page fault and one more event in stackTable in gecko script. I have few doubt regarding this will discuss in office hour. > 4. Making the documentation better. I am trying to make it better, but I'm not sure if it's good enough. Here's the link: https://perf.wiki.kernel.org/index.php/Tutorial#Firefox_Profiler I also need to understand on how to prepare the final report for GSoC and planning to record a 5 min tutorial video on how to use the script. Moreover, I need to make the script run as a root user. This are few TODOs I have in mind. Please let me know if you have any suggestions. > Thanks, > Anup > > > > - Arnaldo > > > > > > > Thanks, > > > > Anup > > > > > > > > > > Regards, > > > > > > Anup > > > > > > > > > > > > > > > >> > > > > > > > >> The second challenge revolved around the usage of event hooks provided with the perf script > > > > > > > >> python event handlers. I found myself deliberating between two approaches. The first > > > > > > > >> approach involved creating custom functions that would be called using the event > > > > > > > >> handlers. These functions would then save the data in an organized format within > > > > > > > >> globally created data structures. The alternative approach was to write the entire > > > > > > > >> logic inside the event handlers themselves. > > > > > > > >> > > > > > > > >> Additionally, I contemplated whether it would be more suitable to handle the creation of > > > > > > > >> a Gecko format for JSON and the profile format within the same script or to separate > > > > > > > >> them into different scripts. > > > > > > > >> > > > > > > > >> I will discuss this points during tomorrow's office hour. > > > > > > > >> > > > > > > > >> However, I have gained a deeper understanding of the problem at hand and will use this > > > > > > > >> knowledge to make more informed decisions and progress more effectively in the coming weeks. > > > > > > > > > > > > > > > > Sounds like you did a good amount of research on how perf script > > > > > > > > works. We can talk about the details in the meeting. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Namhyung > > > > > > > > > > > > > -- > > > > > > - Arnaldo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] Adding Support for Firefox's Gecko Profile Format 2023-08-16 17:10 ` Anup Sharma @ 2023-09-18 3:39 ` Anup Sharma 0 siblings, 0 replies; 21+ messages in thread From: Anup Sharma @ 2023-09-18 3:39 UTC (permalink / raw) To: Anup Sharma Cc: Arnaldo Carvalho de Melo, Namhyung Kim, Ian Rogers, Adrian Hunter, linux-perf-users, Peter Zijlstra, Ingo Molnar, Mark Rutland, Alexander Shishkin, Jiri Olsa, linux-kernel On Wed, Aug 16, 2023 at 10:40:07PM +0530, Anup Sharma wrote: > On Thu, Aug 10, 2023 at 01:31:09AM +0530, Anup Sharma wrote: > > On Thu, Aug 03, 2023 at 12:29:51AM +0530, Anup Sharma wrote: > > > On Wed, Jul 26, 2023 at 04:22:51PM -0300, Arnaldo Carvalho de Melo wrote: > > > > Em Wed, Jul 26, 2023 at 11:34:17PM +0530, Anup Sharma escreveu: > > > > > Dear all, > > > > > > > > > > I want to share a recent development in our project that > > > > > I have been working on this week. I have successfully > > > > > implemented a new feature that allows users to directly > > > > > open the browser and visualize the output of perf.data > > > > > on Firefox Profiler. > > > > > > > > Excellent news! > > > > > > > > > To enhance user experience, I have made sure this option > > > > > is enabled by default. Additionally, I've also added a > > > > > handy command line argument, "--save-only," which allows > > > > > users to save the output file as needed. This enhancement > > > > > replaces the previous standard output method, providing a > > > > > much-improved experience. > > > > > > > > > > Moreover, Everything is processed locally, and no data is > > > > > sent to any server. > > > > > > > > That addresses the worry about sending possibly sensitive data over the > > > > network by default, great. > > > > > > > > > During this implementation, I had the opportunity to delve > > > > > into several exciting topics, including threading, HTTP server, > > > > > and CORS mechanisms. The learning process has been both > > > > > challenging and rewarding. > > > > > > > > Cool that you're so excited learning all this. > > > > > > > > > Currently, I am in the process of performing some clean-up tasks. > > > > > Once that is complete, I will be sharing the next version of the > > > > > series, which includes this fantastic new feature, with all of > > > > > you very soon. > > > > > > > > > > Thank you for your ongoing support and encouragement. > > > > > > > > Keep it up! > > > > > > This week, I'm working on documenting the gecko converter for our wiki. > > > I've dedicated a significant amount of time to think about the content > > > and how to approach it. So far, I've written sections covering the > > > introduction to Firefox Profiler and the formatting guidelines(not > > > published yet). > > > > > > However, I've left the usage part blank because I'm uncertain about how > > > to enable gecko.py arguments when executing the "perf script gecko" command. > > > I've looked at examples like flamegraph-report and sctop-report, but I'm > > > still confused. I attempted to enable the script to take the gecko.py > > > command example "perf script report gecko --save-only=profiler.json," > > > but this caused issues when running "perf script gecko -a sleep 3" again. > > > This has left me feeling rather puzzled, and I would appreciate discussing > > > this during our office hour, or any help here. > > > > > > Due to this blocker, I haven't been able to send the updated patch > > > for test_gecko yet. However, I hope I'll find a solution soon and overcome > > > this obstacle. > > > > Hello All, > > As discussed during last office hour, I was little occupied this week, so > > I haven't been able to make much progress on my TODO list. Nonetheless, > > I've resent a server patch and test script patch for your review. If I can > > get it reviewed it would be great to make further progress. > > > > I'm also working on writing documentation for our wiki. I've already written a > > bit, and you can see it here: https://perf.wiki.kernel.org/index.php/Tutorial#Firefox_Profiler. > > If you have any thoughts or suggestions, I'd appreciate hearing them. I'll add > > more details once I've sorted out some issues I'm having with certain commands. > > > > This week, I do not have a specific agenda. Hence, if we opt to skip this week's > > office hour, it wouldn't pose any inconvenience. Nevertheless, I am available if > > there are matters you wish to discuss. > > > > Here's what I have in mind to work on next: > > > > 1. Making sure that gecko script args work with commands, like > > (perf script gecko --save-only=profile.json) or (perf script gecko --kernel-color=green),which > > aren't working right now. > > I have done this part and sent the patch for review. > https://lore.kernel.org/all/ZNf7W+EIrrCSHZN0@yoga/ > > > 2. Trying out "perf record -p firefox" or using it with a specific process ID to > > see if the current script supports these recording commands. > > This is working with the existing script. I've tested it and documented it as well. > https://perf.wiki.kernel.org/index.php/Tutorial#Firefox_Profiler > > > 3. Adding page fault and one more event in stackTable in gecko script. > > I have few doubt regarding this will discuss in office hour. > > > 4. Making the documentation better. > > I am trying to make it better, but I'm not sure if it's good enough. Here's the link: > https://perf.wiki.kernel.org/index.php/Tutorial#Firefox_Profiler > > I also need to understand on how to prepare the final report for GSoC and > planning to record a 5 min tutorial video on how to use the script. > > Moreover, I need to make the script run as a root user. > > This are few TODOs I have in mind. Please let me know if you have any suggestions. I'm thrilled to share that I've received an offer from Nutanix Technologies for the MTS-1 Kernel Development role, and I couldn't be more excited about it. I want to express my gratitude for all your support. I'm currently on vacation, but I'll be back in a few days to continue making contributions. > > Thanks, > > Anup > > > > > > - Arnaldo > > > > > > > > > Thanks, > > > > > Anup > > > > > > > > > > > > Regards, > > > > > > > Anup > > > > > > > > > > > > > > > > > >> > > > > > > > > >> The second challenge revolved around the usage of event hooks provided with the perf script > > > > > > > > >> python event handlers. I found myself deliberating between two approaches. The first > > > > > > > > >> approach involved creating custom functions that would be called using the event > > > > > > > > >> handlers. These functions would then save the data in an organized format within > > > > > > > > >> globally created data structures. The alternative approach was to write the entire > > > > > > > > >> logic inside the event handlers themselves. > > > > > > > > >> > > > > > > > > >> Additionally, I contemplated whether it would be more suitable to handle the creation of > > > > > > > > >> a Gecko format for JSON and the profile format within the same script or to separate > > > > > > > > >> them into different scripts. > > > > > > > > >> > > > > > > > > >> I will discuss this points during tomorrow's office hour. > > > > > > > > >> > > > > > > > > >> However, I have gained a deeper understanding of the problem at hand and will use this > > > > > > > > >> knowledge to make more informed decisions and progress more effectively in the coming weeks. > > > > > > > > > > > > > > > > > > Sounds like you did a good amount of research on how perf script > > > > > > > > > works. We can talk about the details in the meeting. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Namhyung > > > > > > > > > > > > > > > > -- > > > > > > > > - Arnaldo ^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2023-09-18 3:40 UTC | newest] Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-05-24 19:25 [RFC] Adding Support for Firefox's Gecko Profile Format Anup Sharma 2023-05-25 20:52 ` Namhyung Kim 2023-05-26 15:17 ` Ian Rogers 2023-05-26 19:49 ` Anup Sharma 2023-06-01 20:49 ` Anup Sharma 2023-06-05 21:47 ` Anup Sharma 2023-06-08 6:58 ` Ian Rogers 2023-06-08 12:53 ` Anup Sharma 2023-06-14 20:14 ` Anup Sharma 2023-06-14 20:17 ` Ian Rogers 2023-06-28 20:11 ` Anup Sharma 2023-06-29 6:26 ` Namhyung Kim 2023-06-29 6:35 ` Adrian Hunter 2023-07-04 14:38 ` Anup Sharma 2023-07-04 17:52 ` Anup Sharma 2023-07-26 18:04 ` Anup Sharma 2023-07-26 19:22 ` Arnaldo Carvalho de Melo 2023-08-02 18:59 ` Anup Sharma 2023-08-09 20:01 ` Anup Sharma 2023-08-16 17:10 ` Anup Sharma 2023-09-18 3:39 ` Anup Sharma
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).