From: Andrew Cooper <andrew.cooper3@citrix.com>
To: Steven Rostedt <rostedt@goodmis.org>,
Giuseppe Eletto <giuseppe.eletto@edu.unito.it>
Cc: <linux-trace-devel@vger.kernel.org>,
<xen-devel@lists.xenproject.org>,
Dario Faggioli <dfaggioli@suse.com>,
Enrico Bini <enrico.bini@unito.it>
Subject: Re: A KernelShark plugin for Xen traces analysis
Date: Wed, 14 Apr 2021 11:07:33 +0100 [thread overview]
Message-ID: <094c4b3f-3988-c51f-3a69-cfbc8d6a45bf@citrix.com> (raw)
In-Reply-To: <20210413114614.4971caff@gandalf.local.home>
On 13/04/2021 16:46, Steven Rostedt wrote:
> Hi Giuseppe,
>
> On Tue, 13 Apr 2021 16:28:36 +0200
> Giuseppe Eletto <giuseppe.eletto@edu.unito.it> wrote:
>
>> Hello,
>> I want to share with you a new plugin developed by me, under the
>> supervision of Dario Faggioli, which allows the new version of KernelShark
>> (the v2-beta) to open and view the Xen traces created using the "xentrace" tool.
>>
>> In fact, KernelShark is a well known tool for graphical visualization
>> Linux kernel traces, obtained via "ftrace" and "trace-cmd". Anyway thanks
>> to its modular architecture, it is now possible to implement plugins which
>> open and display traces with arbitrary format, for example, as in in
>> this case, traces of the Xen hypervisor.
> I'm guessing you have trace events coming from Xen itself?
>
>
>> For more information on how to build the plugin and/or
>> to view the source code I leave the repository below:
>> https://github.com/giuseppe998e/kernelshark-xentrace-plugin
>>
>>
>> In short:
>>
>> $ sudo apt install git build-essential libjson-c-dev
>> $ git clone --recurse-submodules
>> https://github.com/giuseppe998e/kernelshark-xentrace-plugin.git
>> $ cd kernelshark-xentrace-plugin/
>> $ make
>>
>> $ export XEN_CPUHZ=3G # Sets the CPU frequency ((G)hz/(M)hz/(K)hz/hz)
>> $ kernelshark -p out/ks-xentrace.so trace.xen
>>
>>
>> You will need the development version of KernelShark, available here:
>> https://git.kernel.org/pub/scm/utils/trace-cmd/kernel-shark.git
> This will soon be the main repository, as we are going to deprecate the
> version in the trace-cmd.git repo soon. And because libtracecmd 1.0 has
> already been released.
>
>
>> A screenshot of the plugin in action is available here:
>> https://github.com/giuseppe998e/kernelshark-xentrace-plugin/raw/master/.github/img/ks-xentrace.png
>>
>> I'm happy to receive whatever feedback you may have about it,
>> and to answer any question.
>>
> Thanks for doing this. What would be nice is to have the xen traces along
> side the linux tracing. Perhaps we can update trace-cmd agent to work with
> Xen as well. Does xen implement vsock or some other way to communicate
> between the guests and the Dom0 kernel? If not, we should add one. The you
> could do the following:
>
> 1. On each guest, run as root: trace-cmd agent --xen
> 2. On Dom0 run: trace-cmd record -e (events on Dom0) \
> --xen (commands to do tracing in Xen HV) \
> -A <guest-name1> -e (events on guest)
>
> And then you would get a trace.dat file for Dom0 and the guest, and also
> have a trace file for Xen (however that is done). And then on KernelShark,
> we have a KVM plugin in development that does this. But you can do the same
> with Xen.
>
> For KVM, we have:
>
> 1. On each guest: trace-cmd agent
> 2. On the host: trace-cmd record -e kvm -e sched -e irq \
> -A guest-name -e all
> The above produces trace.dat for the host trace, and
> trace-<guest-name>.dat for the guest.
> 3. kernelshark trace.dat -a trace-Fedora21.dat
>
> (I have a guest called Fedora21).
>
> http://rostedt.org/private/kernelshark-kvm.png
>
> Where you can see the kvm hypervisor task KVM-2356 is the host task running
> the guest VCPU 0, and you see the switch between the two.
>
> Perhaps we can do something like that with Xen as well. The plugin is still
> in the works, but should be published soon. And when it is, you could use
> that as a template for Xen.
A possibly tangential question. Where does KernelShark's idea of CPUs
(i.e. real logical threads) come from?
In a Xen system, dom0 is just a VM, and particularly on larger servers,
may not be as many vcpus as the system has logical threads.
This causes major problems for `perf` support under Xen, which assumes
that the kernel's idea of CPUs matches that of the system.
When rendering a trace including Xen data, Xen can provide the real
system CPUs, and dom0 wants to be rendered as a VM under Xen, similar to
trace-Fedora21 in your screenshot above. (Obviously, if you're doing
nested virt, things need to start nesting.)
~Andrew
next prev parent reply other threads:[~2021-04-14 10:07 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-13 14:28 A KernelShark plugin for Xen traces analysis Giuseppe Eletto
2021-04-13 15:33 ` Andrew Cooper
2021-04-14 17:31 ` Dario Faggioli
2021-04-14 18:11 ` Andrew Cooper
2021-04-14 19:07 ` A KernelShark plugin for Xen traces analysis Steven Rostedt
2021-04-15 0:50 ` Dario Faggioli
2021-04-15 13:29 ` Steven Rostedt
2021-04-14 21:51 ` A KernelShark plugin for Xen traces analysis Dario Faggioli
2021-04-13 15:46 ` A KernelShark plugin for Xen traces analysis Steven Rostedt
2021-04-14 10:07 ` Andrew Cooper [this message]
2021-04-14 13:43 ` Steven Rostedt
2021-04-14 20:05 ` Andrew Cooper
2021-04-15 0:41 ` Dario Faggioli
2021-04-15 0:13 ` Dario Faggioli
2021-04-14 22:11 ` Dario Faggioli
2021-04-14 22:25 ` Steven Rostedt
2021-04-14 9:25 ` A KernelShark plugin for Xen traces analysis Yordan Karadzhov (VMware)
2021-04-14 17:46 ` Dario Faggioli
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=094c4b3f-3988-c51f-3a69-cfbc8d6a45bf@citrix.com \
--to=andrew.cooper3@citrix.com \
--cc=dfaggioli@suse.com \
--cc=enrico.bini@unito.it \
--cc=giuseppe.eletto@edu.unito.it \
--cc=linux-trace-devel@vger.kernel.org \
--cc=rostedt@goodmis.org \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).