A KernelShark plugin for Xen traces analysis ​

A KernelShark plugin for Xen traces analysis ​

[Giuseppe Eletto]

Hello,
I want to share with you a new plugin developed by me, under the
supervision of Dario Faggioli, which allows the new version of KernelShark
(the v2-beta) to open and view the Xen traces created using the "xentrace" tool.

In fact, KernelShark is a well known tool for graphical visualization
Linux kernel traces, obtained via "ftrace" and "trace-cmd". Anyway thanks
to its modular architecture, it is now possible to implement plugins which
open and display traces with arbitrary format, for example, as in in
this case, traces of the Xen hypervisor.

For more information on how to build the plugin and/or
to view the source code I leave the repository below:
https://github.com/giuseppe998e/kernelshark-xentrace-plugin


In short:

$ sudo apt install git build-essential libjson-c-dev
$ git clone --recurse-submodules
https://github.com/giuseppe998e/kernelshark-xentrace-plugin.git
$ cd kernelshark-xentrace-plugin/
$ make

$ export XEN_CPUHZ=3G # Sets the CPU frequency ((G)hz/(M)hz/(K)hz/hz)
$ kernelshark -p out/ks-xentrace.so trace.xen


You will need the development version of KernelShark, available here:
https://git.kernel.org/pub/scm/utils/trace-cmd/kernel-shark.git

A screenshot of the plugin in action is available here:
https://github.com/giuseppe998e/kernelshark-xentrace-plugin/raw/master/.github/img/ks-xentrace.png

I'm happy to receive whatever feedback you may have about it,
and to answer any question.

Kind regards,
Giuseppe Eletto.

-- 
------------------------



Indirizzo istituzionale di posta elettronica 
degli studenti e dei laureati dell'Università degli Studi di TorinoOfficial 
University of Turin email address for students and graduates 

Re: A KernelShark plugin for Xen traces analysis ​

[Andrew Cooper]

On 13/04/2021 15:28, Giuseppe Eletto wrote:
> Hello,
> I want to share with you a new plugin developed by me, under the
> supervision of Dario Faggioli, which allows the new version of KernelShark
> (the v2-beta) to open and view the Xen traces created using the "xentrace" tool.
>
> In fact, KernelShark is a well known tool for graphical visualization
> Linux kernel traces, obtained via "ftrace" and "trace-cmd". Anyway thanks
> to its modular architecture, it is now possible to implement plugins which
> open and display traces with arbitrary format, for example, as in in
> this case, traces of the Xen hypervisor.
>
> For more information on how to build the plugin and/or
> to view the source code I leave the repository below:
> https://github.com/giuseppe998e/kernelshark-xentrace-plugin
>
>
> In short:
>
> $ sudo apt install git build-essential libjson-c-dev
> $ git clone --recurse-submodules
> https://github.com/giuseppe998e/kernelshark-xentrace-plugin.git
> $ cd kernelshark-xentrace-plugin/
> $ make
>
> $ export XEN_CPUHZ=3G # Sets the CPU frequency ((G)hz/(M)hz/(K)hz/hz)
> $ kernelshark -p out/ks-xentrace.so trace.xen
>
>
> You will need the development version of KernelShark, available here:
> https://git.kernel.org/pub/scm/utils/trace-cmd/kernel-shark.git
>
> A screenshot of the plugin in action is available here:
> https://github.com/giuseppe998e/kernelshark-xentrace-plugin/raw/master/.github/img/ks-xentrace.png
>
> I'm happy to receive whatever feedback you may have about it,
> and to answer any question.

Very nice.

A couple of questions.  Which Xen libraries do you currently use map the
frames?

There is a bug which prevents the mapping being usable in a PVH Dom0,
and I was proposing fixing it by switching to the Acquire Resource
interfaces.  A bonus of doing this is that it can be implemented on
exclusively stable hypercall interfaces, meaning that the plugin no
longer needs recompiling when you change the hypervisor.

To others on xen-devel, do we have firm statement on whether the trace
format itself is stable or not?  I think we've be somewhat nondescript
on this point in the past.

For the screenshot, there are a lot of examples where you have a
dom:vcpu pair, and a number rendered in hex.  Throughout the hypervisor,
we're standardising on d$v$ as a format, and e.g. d[IDLE]v$ for some of
the magic domid's (0x7ff0 ... 0x7fff).

~Andrew

Re: A KernelShark plugin for Xen traces analysis

[Steven Rostedt]

Hi Giuseppe,

On Tue, 13 Apr 2021 16:28:36 +0200
Giuseppe Eletto <giuseppe.eletto@edu.unito.it> wrote:

> Hello,
> I want to share with you a new plugin developed by me, under the
> supervision of Dario Faggioli, which allows the new version of KernelShark
> (the v2-beta) to open and view the Xen traces created using the "xentrace" tool.
> 
> In fact, KernelShark is a well known tool for graphical visualization
> Linux kernel traces, obtained via "ftrace" and "trace-cmd". Anyway thanks
> to its modular architecture, it is now possible to implement plugins which
> open and display traces with arbitrary format, for example, as in in
> this case, traces of the Xen hypervisor.

I'm guessing you have trace events coming from Xen itself?


> 
> For more information on how to build the plugin and/or
> to view the source code I leave the repository below:
> https://github.com/giuseppe998e/kernelshark-xentrace-plugin
> 
> 
> In short:
> 
> $ sudo apt install git build-essential libjson-c-dev
> $ git clone --recurse-submodules
> https://github.com/giuseppe998e/kernelshark-xentrace-plugin.git
> $ cd kernelshark-xentrace-plugin/
> $ make
> 
> $ export XEN_CPUHZ=3G # Sets the CPU frequency ((G)hz/(M)hz/(K)hz/hz)
> $ kernelshark -p out/ks-xentrace.so trace.xen
> 
> 
> You will need the development version of KernelShark, available here:
> https://git.kernel.org/pub/scm/utils/trace-cmd/kernel-shark.git

This will soon be the main repository, as we are going to deprecate the
version in the trace-cmd.git repo soon. And because libtracecmd 1.0 has
already been released.


> 
> A screenshot of the plugin in action is available here:
> https://github.com/giuseppe998e/kernelshark-xentrace-plugin/raw/master/.github/img/ks-xentrace.png
> 
> I'm happy to receive whatever feedback you may have about it,
> and to answer any question.
> 

Thanks for doing this. What would be nice is to have the xen traces along
side the linux tracing. Perhaps we can update trace-cmd agent to work with
Xen as well. Does xen implement vsock or some other way to communicate
between the guests and the Dom0 kernel? If not, we should add one. The you
could do the following:

 1. On each guest, run as root: trace-cmd agent --xen
 2. On Dom0 run: trace-cmd record -e (events on Dom0) \
     --xen (commands to do tracing in Xen HV) \
     -A <guest-name1> -e (events on guest)

And then you would get a trace.dat file for Dom0 and the guest, and also
have a trace file for Xen (however that is done). And then on KernelShark,
we have a KVM plugin in development that does this. But you can do the same
with Xen.

For KVM, we have:

 1. On each guest: trace-cmd agent
 2. On the host: trace-cmd record -e kvm -e sched -e irq \
      -A guest-name -e all
    The above produces trace.dat for the host trace, and 
     trace-<guest-name>.dat for the guest.
 3. kernelshark trace.dat -a trace-Fedora21.dat

(I have a guest called Fedora21).

  http://rostedt.org/private/kernelshark-kvm.png

Where you can see the kvm hypervisor task KVM-2356 is the host task running
the guest VCPU 0, and you see the switch between the two.

Perhaps we can do something like that with Xen as well. The plugin is still
in the works, but should be published soon. And when it is, you could use
that as a template for Xen.

-- Steve

Re: A KernelShark plugin for Xen traces analysis ​

[Yordan Karadzhov (VMware)]

Hi Giuseppe,

It is great to see such progress in the development of the Xen plugin.

Can you share with us what are your plans for continuing this work. Is 
this a first prototype of the plugin, or it is an almost final version?

I was also thinking that maybe you can prepare a short tutorial on 
writing KernelShark plugins for data inputs. You have the best 
experience for this task, since you are the first to develop an external 
plugin. Ideally, this can even turn into a talk that you can give at 
some open source event.
What do you think?

Best,
Yordan


On 13.04.21 г. 17:28, Giuseppe Eletto wrote:
> Hello,
> I want to share with you a new plugin developed by me, under the
> supervision of Dario Faggioli, which allows the new version of KernelShark
> (the v2-beta) to open and view the Xen traces created using the "xentrace" tool.
> 
> In fact, KernelShark is a well known tool for graphical visualization
> Linux kernel traces, obtained via "ftrace" and "trace-cmd". Anyway thanks
> to its modular architecture, it is now possible to implement plugins which
> open and display traces with arbitrary format, for example, as in in
> this case, traces of the Xen hypervisor.
> 
> For more information on how to build the plugin and/or
> to view the source code I leave the repository below:
> https://github.com/giuseppe998e/kernelshark-xentrace-plugin
> 
> 
> In short:
> 
> $ sudo apt install git build-essential libjson-c-dev
> $ git clone --recurse-submodules
> https://github.com/giuseppe998e/kernelshark-xentrace-plugin.git
> $ cd kernelshark-xentrace-plugin/
> $ make
> 
> $ export XEN_CPUHZ=3G # Sets the CPU frequency ((G)hz/(M)hz/(K)hz/hz)
> $ kernelshark -p out/ks-xentrace.so trace.xen
> 
> 
> You will need the development version of KernelShark, available here:
> https://git.kernel.org/pub/scm/utils/trace-cmd/kernel-shark.git
> 
> A screenshot of the plugin in action is available here:
> https://github.com/giuseppe998e/kernelshark-xentrace-plugin/raw/master/.github/img/ks-xentrace.png
> 
> I'm happy to receive whatever feedback you may have about it,
> and to answer any question.
> 
> Kind regards,
> Giuseppe Eletto.
> 

Re: A KernelShark plugin for Xen traces analysis

[Andrew Cooper]

On 13/04/2021 16:46, Steven Rostedt wrote:
> Hi Giuseppe,
>
> On Tue, 13 Apr 2021 16:28:36 +0200
> Giuseppe Eletto <giuseppe.eletto@edu.unito.it> wrote:
>
>> Hello,
>> I want to share with you a new plugin developed by me, under the
>> supervision of Dario Faggioli, which allows the new version of KernelShark
>> (the v2-beta) to open and view the Xen traces created using the "xentrace" tool.
>>
>> In fact, KernelShark is a well known tool for graphical visualization
>> Linux kernel traces, obtained via "ftrace" and "trace-cmd". Anyway thanks
>> to its modular architecture, it is now possible to implement plugins which
>> open and display traces with arbitrary format, for example, as in in
>> this case, traces of the Xen hypervisor.
> I'm guessing you have trace events coming from Xen itself?
>
>
>> For more information on how to build the plugin and/or
>> to view the source code I leave the repository below:
>> https://github.com/giuseppe998e/kernelshark-xentrace-plugin
>>
>>
>> In short:
>>
>> $ sudo apt install git build-essential libjson-c-dev
>> $ git clone --recurse-submodules
>> https://github.com/giuseppe998e/kernelshark-xentrace-plugin.git
>> $ cd kernelshark-xentrace-plugin/
>> $ make
>>
>> $ export XEN_CPUHZ=3G # Sets the CPU frequency ((G)hz/(M)hz/(K)hz/hz)
>> $ kernelshark -p out/ks-xentrace.so trace.xen
>>
>>
>> You will need the development version of KernelShark, available here:
>> https://git.kernel.org/pub/scm/utils/trace-cmd/kernel-shark.git
> This will soon be the main repository, as we are going to deprecate the
> version in the trace-cmd.git repo soon. And because libtracecmd 1.0 has
> already been released.
>
>
>> A screenshot of the plugin in action is available here:
>> https://github.com/giuseppe998e/kernelshark-xentrace-plugin/raw/master/.github/img/ks-xentrace.png
>>
>> I'm happy to receive whatever feedback you may have about it,
>> and to answer any question.
>>
> Thanks for doing this. What would be nice is to have the xen traces along
> side the linux tracing. Perhaps we can update trace-cmd agent to work with
> Xen as well. Does xen implement vsock or some other way to communicate
> between the guests and the Dom0 kernel? If not, we should add one. The you
> could do the following:
>
>  1. On each guest, run as root: trace-cmd agent --xen
>  2. On Dom0 run: trace-cmd record -e (events on Dom0) \
>      --xen (commands to do tracing in Xen HV) \
>      -A <guest-name1> -e (events on guest)
>
> And then you would get a trace.dat file for Dom0 and the guest, and also
> have a trace file for Xen (however that is done). And then on KernelShark,
> we have a KVM plugin in development that does this. But you can do the same
> with Xen.
>
> For KVM, we have:
>
>  1. On each guest: trace-cmd agent
>  2. On the host: trace-cmd record -e kvm -e sched -e irq \
>       -A guest-name -e all
>     The above produces trace.dat for the host trace, and 
>      trace-<guest-name>.dat for the guest.
>  3. kernelshark trace.dat -a trace-Fedora21.dat
>
> (I have a guest called Fedora21).
>
>   http://rostedt.org/private/kernelshark-kvm.png
>
> Where you can see the kvm hypervisor task KVM-2356 is the host task running
> the guest VCPU 0, and you see the switch between the two.
>
> Perhaps we can do something like that with Xen as well. The plugin is still
> in the works, but should be published soon. And when it is, you could use
> that as a template for Xen.

A possibly tangential question.  Where does KernelShark's idea of CPUs
(i.e. real logical threads) come from?

In a Xen system, dom0 is just a VM, and particularly on larger servers,
may not be as many vcpus as the system has logical threads.

This causes major problems for `perf` support under Xen, which assumes
that the kernel's idea of CPUs matches that of the system.

When rendering a trace including Xen data, Xen can provide the real
system CPUs, and dom0 wants to be rendered as a VM under Xen, similar to
trace-Fedora21 in your screenshot above.  (Obviously, if you're doing
nested virt, things need to start nesting.)

~Andrew

Re: A KernelShark plugin for Xen traces analysis

[Steven Rostedt]

On Wed, 14 Apr 2021 11:07:33 +0100
Andrew Cooper <andrew.cooper3@citrix.com> wrote:

> On 13/04/2021 16:46, Steven Rostedt wrote:
> > Hi Giuseppe,
> >
> > On Tue, 13 Apr 2021 16:28:36 +0200
> > Giuseppe Eletto <giuseppe.eletto@edu.unito.it> wrote:
> >  
> >> Hello,
> >> I want to share with you a new plugin developed by me, under the
> >> supervision of Dario Faggioli, which allows the new version of KernelShark
> >> (the v2-beta) to open and view the Xen traces created using the "xentrace" tool.
> >>
> >> In fact, KernelShark is a well known tool for graphical visualization
> >> Linux kernel traces, obtained via "ftrace" and "trace-cmd". Anyway thanks
> >> to its modular architecture, it is now possible to implement plugins which
> >> open and display traces with arbitrary format, for example, as in in
> >> this case, traces of the Xen hypervisor.  
> > I'm guessing you have trace events coming from Xen itself?
> >
> >  
> >> For more information on how to build the plugin and/or
> >> to view the source code I leave the repository below:
> >> https://github.com/giuseppe998e/kernelshark-xentrace-plugin
> >>
> >>
> >> In short:
> >>
> >> $ sudo apt install git build-essential libjson-c-dev
> >> $ git clone --recurse-submodules
> >> https://github.com/giuseppe998e/kernelshark-xentrace-plugin.git
> >> $ cd kernelshark-xentrace-plugin/
> >> $ make
> >>
> >> $ export XEN_CPUHZ=3G # Sets the CPU frequency ((G)hz/(M)hz/(K)hz/hz)
> >> $ kernelshark -p out/ks-xentrace.so trace.xen
> >>
> >>
> >> You will need the development version of KernelShark, available here:
> >> https://git.kernel.org/pub/scm/utils/trace-cmd/kernel-shark.git  
> > This will soon be the main repository, as we are going to deprecate the
> > version in the trace-cmd.git repo soon. And because libtracecmd 1.0 has
> > already been released.
> >
> >  
> >> A screenshot of the plugin in action is available here:
> >> https://github.com/giuseppe998e/kernelshark-xentrace-plugin/raw/master/.github/img/ks-xentrace.png
> >>
> >> I'm happy to receive whatever feedback you may have about it,
> >> and to answer any question.
> >>  
> > Thanks for doing this. What would be nice is to have the xen traces along
> > side the linux tracing. Perhaps we can update trace-cmd agent to work with
> > Xen as well. Does xen implement vsock or some other way to communicate
> > between the guests and the Dom0 kernel? If not, we should add one. The you
> > could do the following:
> >
> >  1. On each guest, run as root: trace-cmd agent --xen
> >  2. On Dom0 run: trace-cmd record -e (events on Dom0) \
> >      --xen (commands to do tracing in Xen HV) \
> >      -A <guest-name1> -e (events on guest)
> >
> > And then you would get a trace.dat file for Dom0 and the guest, and also
> > have a trace file for Xen (however that is done). And then on KernelShark,
> > we have a KVM plugin in development that does this. But you can do the same
> > with Xen.
> >
> > For KVM, we have:
> >
> >  1. On each guest: trace-cmd agent
> >  2. On the host: trace-cmd record -e kvm -e sched -e irq \
> >       -A guest-name -e all
> >     The above produces trace.dat for the host trace, and 
> >      trace-<guest-name>.dat for the guest.
> >  3. kernelshark trace.dat -a trace-Fedora21.dat
> >
> > (I have a guest called Fedora21).
> >
> >   http://rostedt.org/private/kernelshark-kvm.png
> >
> > Where you can see the kvm hypervisor task KVM-2356 is the host task running
> > the guest VCPU 0, and you see the switch between the two.
> >
> > Perhaps we can do something like that with Xen as well. The plugin is still
> > in the works, but should be published soon. And when it is, you could use
> > that as a template for Xen.  
> 
> A possibly tangential question.  Where does KernelShark's idea of CPUs
> (i.e. real logical threads) come from?

It comes from the plugin that parses the file.

KernelShark can take multiple inputs. We have a plugin internally at VMware
that can read trace data from ESX and produces CPUs and Tasks on top of
them. In ESX tasks are actually called "Worlds".

> 
> In a Xen system, dom0 is just a VM, and particularly on larger servers,
> may not be as many vcpus as the system has logical threads.

Yes, I have some idea of how Xen works. When I was hired at Red Hat in
2006, my first job was to work on Xen. I even ported "logdev" (the
predecessor to ftrace) to the Xen hypervisor).

> 
> This causes major problems for `perf` support under Xen, which assumes
> that the kernel's idea of CPUs matches that of the system.

Things are different with KernelShark.

> 
> When rendering a trace including Xen data, Xen can provide the real
> system CPUs, and dom0 wants to be rendered as a VM under Xen, similar to
> trace-Fedora21 in your screenshot above.  (Obviously, if you're doing
> nested virt, things need to start nesting.)

Right.

What I would envision how this would work, is that you would produce a
set of tracing files. One for each guest (including Dom0), and one for the
Xen hypervisor itself. The trick is to have a way to synchronize the time
stamps. What we just did with KVM is to have all the tracing record the
CPUs TSC, including the shift and multiplier that the CPU might change for
the guests. Then we have a way to convert the TSC to nanoseconds. This way
all tracing data has the same clock. It's somewhat complicated to get
right, and requires access to how the guests clocks are modified by the CPU.

For KVM, each machine has a unique id and is stored in the trace.dat files.
We have the host store a mapping of what thread represents which guest VCPU
(virtual CPU). Then the "-a" option tells KernelShark to append the
tracing data as a dependency. I would imagine we can have something like
this:

 kernelshark xen.dat -a trace-dom0.dat -a trace-guest1.dat -a trace-guest2.dat

The Xen plugin would then need to read the how the threads in xen.dat map
to the virtual CPUs of each of the guest files. Which would give you the
layering.

To give you a better idea about the CPU mappings, before I enabled the KVM
combo plugin (that maps the VCPUS with the host threads), kernelshark comes
up like this:

  http://rostedt.org/private/kernel-shark-kvm2.png

Where the first 8 plots represent the host CPUs and the next two represent
the guest CPUs.

Finally, we have been thinking about nesting VMs. It's been in our thoughts
from the beginning, and we worked to make sure that the design can be
extended to handle it. We haven't done anything with nested VM mappings, as
we are trying to sort out all the issues with just mapping a single guest to
host. ;-)  But yes, when we do support nested VMs, We could show the host
thread, the guest-host thread, then the top guest CPU stacked on top of
each other.

As everything is done via plugins, the options are limitless ;-)

-- Steve

Re: A KernelShark plugin for Xen traces analysis ​

[Dario Faggioli]

[-- Attachment #1: Type: text/plain, Size: 1966 bytes --]

On Tue, 2021-04-13 at 16:33 +0100, Andrew Cooper wrote:
> On 13/04/2021 15:28, Giuseppe Eletto wrote:
> > 
> > You will need the development version of KernelShark, available
> > here:
> > https://git.kernel.org/pub/scm/utils/trace-cmd/kernel-shark.git
> > 
> > A screenshot of the plugin in action is available here:
> > https://github.com/giuseppe998e/kernelshark-xentrace-plugin/raw/master/.github/img/ks-xentrace.png
> > 
> > I'm happy to receive whatever feedback you may have about it,
> > and to answer any question.
> 
> Very nice.
> 
> A couple of questions.  Which Xen libraries do you currently use map
> the
> frames?
>
Err... If I understood the question none, as the plugin loads and
parses a file, as it is produced by `xentrace`. :-)

But maybe I didn't understand the question?


> For the screenshot, there are a lot of examples where you have a
> dom:vcpu pair, and a number rendered in hex.  Throughout the
> hypervisor,
> we're standardising on d$v$ as a format, and e.g. d[IDLE]v$ for some
> of
> the magic domid's (0x7ff0 ... 0x7fff).
> 
Yes, the content of the "info" column is currently a bit "raw". I
believe it should be made more similar to what `xenalyze --dump-all`
looks like, rather than to what xentrace_format` does (just to make and
example that people that have used these two tools can understand).

This is just due to the fact that we wanted to let the Xen and
KernelShark communities know about this work as soon as Giuseppe got it
working properly and reliably, to gather any kind of feedback, decide
where this should live, in the long run, (in Xen? In KS? In its own
project?), etc. :-)

Thanks and Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: A KernelShark plugin for Xen traces analysis ​

[Dario Faggioli]

[-- Attachment #1: Type: text/plain, Size: 1966 bytes --]

On Tue, 2021-04-13 at 16:33 +0100, Andrew Cooper wrote:
> On 13/04/2021 15:28, Giuseppe Eletto wrote:
> > 
> > You will need the development version of KernelShark, available
> > here:
> > https://git.kernel.org/pub/scm/utils/trace-cmd/kernel-shark.git
> > 
> > A screenshot of the plugin in action is available here:
> > https://github.com/giuseppe998e/kernelshark-xentrace-plugin/raw/master/.github/img/ks-xentrace.png
> > 
> > I'm happy to receive whatever feedback you may have about it,
> > and to answer any question.
> 
> Very nice.
> 
> A couple of questions.  Which Xen libraries do you currently use map
> the
> frames?
>
Err... If I understood the question none, as the plugin loads and
parses a file, as it is produced by `xentrace`. :-)

But maybe I didn't understand the question?


> For the screenshot, there are a lot of examples where you have a
> dom:vcpu pair, and a number rendered in hex.  Throughout the
> hypervisor,
> we're standardising on d$v$ as a format, and e.g. d[IDLE]v$ for some
> of
> the magic domid's (0x7ff0 ... 0x7fff).
> 
Yes, the content of the "info" column is currently a bit "raw". I
believe it should be made more similar to what `xenalyze --dump-all`
looks like, rather than to what xentrace_format` does (just to make and
example that people that have used these two tools can understand).

This is just due to the fact that we wanted to let the Xen and
KernelShark communities know about this work as soon as Giuseppe got it
working properly and reliably, to gather any kind of feedback, decide
where this should live, in the long run, (in Xen? In KS? In its own
project?), etc. :-)

Thanks and Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: A KernelShark plugin for Xen traces analysis ​

[Dario Faggioli]

[-- Attachment #1: Type: text/plain, Size: 1229 bytes --]

On Wed, 2021-04-14 at 12:25 +0300, Yordan Karadzhov (VMware) wrote:
> It is great to see such progress in the development of the Xen
> plugin.
> 
> Can you share with us what are your plans for continuing this work.
> Is 
> this a first prototype of the plugin, or it is an almost final
> version?
> 
I'm only going to say that the plugin is working well with the traces
that we've thrown at it so far, so I'd say it's past the prototype
phase. :-P

As I was telling to Andrew on xen-devel, there is indeed some room for
improvement, e.g., the content of the 'Info' column is a bit "raw", and
can be made less cryptic and hence more useful.

To me (but this is just my personal opinion) a rather interesting thing
is whether something like this plugin should live in its own project or
will it be better be integrated in either Xen or KS sources...

And I'll now leave any further comments to Giuseppe himself. :-)

Thanks and Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: A KernelShark plugin for Xen traces analysis ​

[Dario Faggioli]

[-- Attachment #1: Type: text/plain, Size: 1229 bytes --]

On Wed, 2021-04-14 at 12:25 +0300, Yordan Karadzhov (VMware) wrote:
> It is great to see such progress in the development of the Xen
> plugin.
> 
> Can you share with us what are your plans for continuing this work.
> Is 
> this a first prototype of the plugin, or it is an almost final
> version?
> 
I'm only going to say that the plugin is working well with the traces
that we've thrown at it so far, so I'd say it's past the prototype
phase. :-P

As I was telling to Andrew on xen-devel, there is indeed some room for
improvement, e.g., the content of the 'Info' column is a bit "raw", and
can be made less cryptic and hence more useful.

To me (but this is just my personal opinion) a rather interesting thing
is whether something like this plugin should live in its own project or
will it be better be integrated in either Xen or KS sources...

And I'll now leave any further comments to Giuseppe himself. :-)

Thanks and Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: A KernelShark plugin for Xen traces analysis ​

[Andrew Cooper]

On 14/04/2021 18:31, Dario Faggioli wrote:
> On Tue, 2021-04-13 at 16:33 +0100, Andrew Cooper wrote:
>> On 13/04/2021 15:28, Giuseppe Eletto wrote:
>>> You will need the development version of KernelShark, available
>>> here:
>>> https://git.kernel.org/pub/scm/utils/trace-cmd/kernel-shark.git
>>>
>>> A screenshot of the plugin in action is available here:
>>> https://github.com/giuseppe998e/kernelshark-xentrace-plugin/raw/master/.github/img/ks-xentrace.png
>>>
>>> I'm happy to receive whatever feedback you may have about it,
>>> and to answer any question.
>> Very nice.
>>
>> A couple of questions.  Which Xen libraries do you currently use map
>> the
>> frames?
>>
> Err... If I understood the question none, as the plugin loads and
> parses a file, as it is produced by `xentrace`. :-)
>
> But maybe I didn't understand the question?

Ah no - that answer's my question.  I'd blindly assumed that the plugin
was talking directly to Xen to obtain the tracebuffer.

>> For the screenshot, there are a lot of examples where you have a
>> dom:vcpu pair, and a number rendered in hex.  Throughout the
>> hypervisor,
>> we're standardising on d$v$ as a format, and e.g. d[IDLE]v$ for some
>> of
>> the magic domid's (0x7ff0 ... 0x7fff).
>>
> Yes, the content of the "info" column is currently a bit "raw". I
> believe it should be made more similar to what `xenalyze --dump-all`
> looks like, rather than to what xentrace_format` does (just to make and
> example that people that have used these two tools can understand).
>
> This is just due to the fact that we wanted to let the Xen and
> KernelShark communities know about this work as soon as Giuseppe got it
> working properly and reliably, to gather any kind of feedback, decide
> where this should live, in the long run, (in Xen? In KS? In its own
> project?), etc. :-)

Where the plugin (ought to) live depends heavily on whether we consider
the trace format a stable ABI or not.

~Andrew

Re: A KernelShark plugin for Xen traces analysis

[Steven Rostedt]

On Wed, 14 Apr 2021 19:11:19 +0100
Andrew Cooper <andrew.cooper3@citrix.com> wrote:

> Where the plugin (ought to) live depends heavily on whether we consider
> the trace format a stable ABI or not.

Agreed. Like the VMware plugin to handle ESX traces. It's internal and not
published as the API is not stable.

But if it ever becomes stable, and you would like it to live with
KernelShark, we are looking to have a place to store third party plugins.

We are working to make sure that the API for KernelShark plugins remains
stable, so your plugins should always work too.

-- Steve

Re: A KernelShark plugin for Xen traces analysis

[Andrew Cooper]

On 14/04/2021 14:43, Steven Rostedt wrote:
>> This causes major problems for `perf` support under Xen, which assumes
>> that the kernel's idea of CPUs matches that of the system.
> Things are different with KernelShark.

That is very encouraging to hear.

>> When rendering a trace including Xen data, Xen can provide the real
>> system CPUs, and dom0 wants to be rendered as a VM under Xen, similar to
>> trace-Fedora21 in your screenshot above.  (Obviously, if you're doing
>> nested virt, things need to start nesting.)
> Right.
>
> What I would envision how this would work, is that you would produce a
> set of tracing files. One for each guest (including Dom0), and one for the
> Xen hypervisor itself. The trick is to have a way to synchronize the time
> stamps. What we just did with KVM is to have all the tracing record the
> CPUs TSC, including the shift and multiplier that the CPU might change for
> the guests. Then we have a way to convert the TSC to nanoseconds. This way
> all tracing data has the same clock. It's somewhat complicated to get
> right, and requires access to how the guests clocks are modified by the CPU.

Hmm.  In the past, I have had success by modifying Xen to refuse any
shift/scale settings, at which point VMs and the hypervisor have
directly-comparable raw TSC values.

Xen certainly has enough information to describe what TSC rate/epoch
each guest is seeing, but I doubt any of this is coherently exposed at
the moment.

> For KVM, each machine has a unique id and is stored in the trace.dat files.
> We have the host store a mapping of what thread represents which guest VCPU
> (virtual CPU). Then the "-a" option tells KernelShark to append the
> tracing data as a dependency. I would imagine we can have something like
> this:
>
>  kernelshark xen.dat -a trace-dom0.dat -a trace-guest1.dat -a trace-guest2.dat
>
> The Xen plugin would then need to read the how the threads in xen.dat map
> to the virtual CPUs of each of the guest files. Which would give you the
> layering.

Looks good.  I suspect we might need to do a little work on Xen's trace
data to make this mesh together nicely.  In particular, Xen doesn't have
a terribly good scheme on unique IDs for "a VM".

We've got domain ID's which are Xen's unique instances of a running
"thing", but they change across VM reboot/migrate/etc.  I suspect we
have some atomicity problems with unique identification information and
VM-fork too.

There is a UUID field but we leave that entirely up to the toolstack to
manage.  (A good test for naive toolstack code comes on the a localhost
live migrate, because suddenly the toolstack is presented with one
logical VM (=> one UUID) and two concurrent domid's.)


I'll try to have a play with the plugin in some copious free time, but
this work does look exciting.

~Andrew

Re: A KernelShark plugin for Xen traces analysis ​

[Dario Faggioli]

[-- Attachment #1: Type: text/plain, Size: 1026 bytes --]

On Wed, 2021-04-14 at 19:11 +0100, Andrew Cooper wrote:
> On 14/04/2021 18:31, Dario Faggioli wrote:
> > > A couple of questions.  Which Xen libraries do you currently use
> > > map
> > > the
> > > frames?
> > > 
> > Err... If I understood the question none, as the plugin loads and
> > parses a file, as it is produced by `xentrace`. :-)
> > 
> > But maybe I didn't understand the question?
> 
> Ah no - that answer's my question.  I'd blindly assumed that the
> plugin
> was talking directly to Xen to obtain the tracebuffer.
> 
Right. No, KernelShark, for Linux, "just" reads trace-data file
produced by trace-cmd. So we adopted the same model and made it "just"
read a trace-data file, in our case produced by xentrace.

Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: A KernelShark plugin for Xen traces analysis ​

[Dario Faggioli]

[-- Attachment #1: Type: text/plain, Size: 1026 bytes --]

On Wed, 2021-04-14 at 19:11 +0100, Andrew Cooper wrote:
> On 14/04/2021 18:31, Dario Faggioli wrote:
> > > A couple of questions.  Which Xen libraries do you currently use
> > > map
> > > the
> > > frames?
> > > 
> > Err... If I understood the question none, as the plugin loads and
> > parses a file, as it is produced by `xentrace`. :-)
> > 
> > But maybe I didn't understand the question?
> 
> Ah no - that answer's my question.  I'd blindly assumed that the
> plugin
> was talking directly to Xen to obtain the tracebuffer.
> 
Right. No, KernelShark, for Linux, "just" reads trace-data file
produced by trace-cmd. So we adopted the same model and made it "just"
read a trace-data file, in our case produced by xentrace.

Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: A KernelShark plugin for Xen traces analysis

[Dario Faggioli]

[-- Attachment #1: Type: text/plain, Size: 3129 bytes --]

On Tue, 2021-04-13 at 11:46 -0400, Steven Rostedt wrote:
> On Tue, 13 Apr 2021 16:28:36 +0200
> Giuseppe Eletto <giuseppe.eletto@edu.unito.it> wrote:
> > 
> > In fact, KernelShark is a well known tool for graphical
> > visualization
> > Linux kernel traces, obtained via "ftrace" and "trace-cmd". Anyway
> > thanks
> > to its modular architecture, it is now possible to implement
> > plugins which
> > open and display traces with arbitrary format, for example, as in
> > in
> > this case, traces of the Xen hypervisor.
> 
> I'm guessing you have trace events coming from Xen itself?
> 
Yes, basically, we can say that a Xen system has "its own trace-cmd".
It's called `xentrace`, you run it from Dom0 and you get a (binary)
file which contains a bunch of events.

Not that differently from a trace-cmd's "trace.dat" file, but the
events in there comes from tracepoints within the hypervisor (which, of
course, use a different tracing mechanism than ftrace).

> 
> > A screenshot of the plugin in action is available here:
> > https://github.com/giuseppe998e/kernelshark-xentrace-plugin/raw/master/.github/img/ks-xentrace.png
> > 
> > I'm happy to receive whatever feedback you may have about it,
> > and to answer any question.
> > 
> 
> Thanks for doing this. What would be nice is to have the xen traces
> along
> side the linux tracing. 
>
Indeed! :-P

> Perhaps we can update trace-cmd agent to work with
> Xen as well. Does xen implement vsock or some other way to
> communicate
> between the guests and the Dom0 kernel? 
>
Not vsock, AFAIK. But we probably can use something else/come up with
something new.

>  1. On each guest, run as root: trace-cmd agent --xen
>  2. On Dom0 run: trace-cmd record -e (events on Dom0) \
>      --xen (commands to do tracing in Xen HV) \
>      -A <guest-name1> -e (events on guest)
> 
> And then you would get a trace.dat file for Dom0 and the guest, and
> also
> have a trace file for Xen (however that is done). 
>
Yep, and the implementation of the `--xen (commands to do tracing in
Xen HV)` part, can just "call-in" to xentrace (somehow), and we'll get
the trace.xen file that then can be interpreted with this plugin.

> And then on KernelShark,
> we have a KVM plugin in development that does this. But you can do
> the same
> with Xen.
> 
I think that one of the trickiest aspects would be synchronizing the
timestamps in the 3 traces.

*I guess* that the dom0 trace and the guest traces could at least use
the PTP algorithm that is currently implemented in the trace-cmd
patches (but not KVM specific one). For synch'ing the Xen trace with
them, well, I don't really know... We'd have to think about it. :-P

> Perhaps we can do something like that with Xen as well. 
>
Would be awesome, IMO. :-)

Thanks and Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: A KernelShark plugin for Xen traces analysis

[Dario Faggioli]

[-- Attachment #1: Type: text/plain, Size: 3129 bytes --]

On Tue, 2021-04-13 at 11:46 -0400, Steven Rostedt wrote:
> On Tue, 13 Apr 2021 16:28:36 +0200
> Giuseppe Eletto <giuseppe.eletto@edu.unito.it> wrote:
> > 
> > In fact, KernelShark is a well known tool for graphical
> > visualization
> > Linux kernel traces, obtained via "ftrace" and "trace-cmd". Anyway
> > thanks
> > to its modular architecture, it is now possible to implement
> > plugins which
> > open and display traces with arbitrary format, for example, as in
> > in
> > this case, traces of the Xen hypervisor.
> 
> I'm guessing you have trace events coming from Xen itself?
> 
Yes, basically, we can say that a Xen system has "its own trace-cmd".
It's called `xentrace`, you run it from Dom0 and you get a (binary)
file which contains a bunch of events.

Not that differently from a trace-cmd's "trace.dat" file, but the
events in there comes from tracepoints within the hypervisor (which, of
course, use a different tracing mechanism than ftrace).

> 
> > A screenshot of the plugin in action is available here:
> > https://github.com/giuseppe998e/kernelshark-xentrace-plugin/raw/master/.github/img/ks-xentrace.png
> > 
> > I'm happy to receive whatever feedback you may have about it,
> > and to answer any question.
> > 
> 
> Thanks for doing this. What would be nice is to have the xen traces
> along
> side the linux tracing. 
>
Indeed! :-P

> Perhaps we can update trace-cmd agent to work with
> Xen as well. Does xen implement vsock or some other way to
> communicate
> between the guests and the Dom0 kernel? 
>
Not vsock, AFAIK. But we probably can use something else/come up with
something new.

>  1. On each guest, run as root: trace-cmd agent --xen
>  2. On Dom0 run: trace-cmd record -e (events on Dom0) \
>      --xen (commands to do tracing in Xen HV) \
>      -A <guest-name1> -e (events on guest)
> 
> And then you would get a trace.dat file for Dom0 and the guest, and
> also
> have a trace file for Xen (however that is done). 
>
Yep, and the implementation of the `--xen (commands to do tracing in
Xen HV)` part, can just "call-in" to xentrace (somehow), and we'll get
the trace.xen file that then can be interpreted with this plugin.

> And then on KernelShark,
> we have a KVM plugin in development that does this. But you can do
> the same
> with Xen.
> 
I think that one of the trickiest aspects would be synchronizing the
timestamps in the 3 traces.

*I guess* that the dom0 trace and the guest traces could at least use
the PTP algorithm that is currently implemented in the trace-cmd
patches (but not KVM specific one). For synch'ing the Xen trace with
them, well, I don't really know... We'd have to think about it. :-P

> Perhaps we can do something like that with Xen as well. 
>
Would be awesome, IMO. :-)

Thanks and Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: A KernelShark plugin for Xen traces analysis

[Steven Rostedt]

On Thu, 15 Apr 2021 00:11:32 +0200
Dario Faggioli <dfaggioli@suse.com> wrote:

 
> Yes, basically, we can say that a Xen system has "its own trace-cmd".
> It's called `xentrace`, you run it from Dom0 and you get a (binary)
> file which contains a bunch of events.
> 
> Not that differently from a trace-cmd's "trace.dat" file, but the
> events in there comes from tracepoints within the hypervisor (which, of
> course, use a different tracing mechanism than ftrace).

Right, that's exactly what the ESX trace did as well.

> > Perhaps we can update trace-cmd agent to work with
> > Xen as well. Does xen implement vsock or some other way to
> > communicate
> > between the guests and the Dom0 kernel? 
> >  
> Not vsock, AFAIK. But we probably can use something else/come up with
> something new.
> 

Yeah, we would like to have trace-cmd agent work with more than just vsock.
Heck, we could just use networking as well. It's just a bit more overhead.


> > And then on KernelShark,
> > we have a KVM plugin in development that does this. But you can do
> > the same
> > with Xen.
> >   
> I think that one of the trickiest aspects would be synchronizing the
> timestamps in the 3 traces.
> 
> *I guess* that the dom0 trace and the guest traces could at least use
> the PTP algorithm that is currently implemented in the trace-cmd
> patches (but not KVM specific one). For synch'ing the Xen trace with
> them, well, I don't really know... We'd have to think about it. :-P

Really, TSC is the way to go. All you would need to do is to have a way to
map all the TSCs together. Assuming the xen trace has a unmodified TSC, and
you can retrieve all the multipliers and shifts used for each guest, you
then will have a synchronized TSC. Then only one guest or the xen HV needs
to calculate the TSC to nanoseconds, and then have all use that. Probably
would need to be the xen HV as it would be the one without a modified TSC.

-- Steve

Re: A KernelShark plugin for Xen traces analysis

[Steven Rostedt]

On Thu, 15 Apr 2021 00:11:32 +0200
Dario Faggioli <dfaggioli@suse.com> wrote:

 
> Yes, basically, we can say that a Xen system has "its own trace-cmd".
> It's called `xentrace`, you run it from Dom0 and you get a (binary)
> file which contains a bunch of events.
> 
> Not that differently from a trace-cmd's "trace.dat" file, but the
> events in there comes from tracepoints within the hypervisor (which, of
> course, use a different tracing mechanism than ftrace).

Right, that's exactly what the ESX trace did as well.

> > Perhaps we can update trace-cmd agent to work with
> > Xen as well. Does xen implement vsock or some other way to
> > communicate
> > between the guests and the Dom0 kernel? 
> >  
> Not vsock, AFAIK. But we probably can use something else/come up with
> something new.
> 

Yeah, we would like to have trace-cmd agent work with more than just vsock.
Heck, we could just use networking as well. It's just a bit more overhead.


> > And then on KernelShark,
> > we have a KVM plugin in development that does this. But you can do
> > the same
> > with Xen.
> >   
> I think that one of the trickiest aspects would be synchronizing the
> timestamps in the 3 traces.
> 
> *I guess* that the dom0 trace and the guest traces could at least use
> the PTP algorithm that is currently implemented in the trace-cmd
> patches (but not KVM specific one). For synch'ing the Xen trace with
> them, well, I don't really know... We'd have to think about it. :-P

Really, TSC is the way to go. All you would need to do is to have a way to
map all the TSCs together. Assuming the xen trace has a unmodified TSC, and
you can retrieve all the multipliers and shifts used for each guest, you
then will have a synchronized TSC. Then only one guest or the xen HV needs
to calculate the TSC to nanoseconds, and then have all use that. Probably
would need to be the xen HV as it would be the one without a modified TSC.

-- Steve

Re: A KernelShark plugin for Xen traces analysis

[Dario Faggioli]

[-- Attachment #1: Type: text/plain, Size: 5239 bytes --]

On Wed, 2021-04-14 at 11:07 +0100, Andrew Cooper wrote:
> On 13/04/2021 16:46, Steven Rostedt wrote:
> In a Xen system, dom0 is just a VM, and particularly on larger servers,
> may not be as many vcpus as the system has logical threads.
> 
> This causes major problems for `perf` support under Xen, which assumes
> that the kernel's idea of CPUs matches that of the system.
> 
Yeah, but we wouldn't need anything like that, in this case.

> When rendering a trace including Xen data, Xen can provide the real
> system CPUs, and dom0 wants to be rendered as a VM under Xen, similar
> to
> trace-Fedora21 in your screenshot above.  (Obviously, if you're doing
> nested virt, things need to start nesting.)
> 
Exactly!

So, basically, events captured with trace-cmd on Dom0 will end up in a,
say, trace-dom0.dat file. It will contain ftrace event records, each
one with a timestamp and the ID of the CPU where it occurred. Of
course, in this case, that ID would be the ID of one of the Dom0 vCPUs
(seen from inside Dom0 itself).

And the same for all the trace files collected inside any other guest.

In KVM, vCPUs are Linux task. In fact, in Steve's screenshot [1], you
can see a task called 'CPU 0/KVM', with PID 2356. Note, however, that
with the KVM-Combo view active, there's a special 'CPU 0/KVM-2356' plot
in the graph. That shows _when_ that particular task, which is vCPU 0
of the KVM VM, runs. It's special because it has a 'CPU 0' plot "glued"
to it. In that plot we see the tasks (and the events) that were running
inside the VM, on vCPU 0.

[1] http://rostedt.org/private/kernelshark-kvm.png

So, if task A run from time t1 to time t2 on vCPU 0 of the Fedora21 KVM
guest, which in turn was running on physical CPU 3, we will see a non-
white (magenta, in the screenshot) box between t1 and t2 in the 'CPU
0/KVM-2356'. And we will see a non-white box on the 'CPU 0' plot which
is "glued" to 'CPU 0/KVM-2356' (it will be of whatever color KS decides
decide to use for task A, say, blue).

If at time t2 task B starts to run on vCPU 0 of Fedora21, and runs
until t3, we'll still see the magenta box on the 'CPU 0/KVM-2356',
between t2 and t3, while on the 'CPU 0' plot glued to it, we'll see a
box of whatever color is chosen for B (say, yellow).

Note that you don't see, from just this plots, _where_ task 2356 is
running, as in on which physical CPU. That will be visible in the CPU
plots, which do not appear in Steve's screenshot. Basically, if 2356
(i.e., the task corresponding to vCPU 0 of the Fedora21 VM) was running
on physical CPU 3 between t1 and t3, we'll see a magenta box, between
t1 and t3, on the plot called 'CPU 3 (and this is a "single plot",
i.e., one that does not have any other plot "glued" to it, e.g., like
the ones that are visible in Giuseppe's screenshot).

In the case of Xen, it's not that different. vCPUs are modeled and
considered by the plugin as if they where Linux tasks. In fact, you see
in Giuseppe's screenshot [2] that the 'Task' column contains strings
such as "d0/v5" or "idle/v4".

[2]
https://raw.githubusercontent.com/giuseppe998e/kernelshark-xentrace-plugin/master/.github/img/ks-xentrace.png

We needed to come up with a PID, as KS not only wants to print it in
the relevant column, but it uses it for associating colors to tasks
(i.e., colors to vCPUs, in our case). Currently, the value stored in
that PID field is computed as a combination of domain ID and vCPU ID.

If d0v5 runs on CPU 6 from t1 to t3, we'll see a, let's say, green box
between t1 and t3 on the CPU plot CPU 6. And this is something that
Giuseppe's plugin can already tell (show! :-D) us.

If task A runs on vCPU 5 of Dom0 from t1 to t2 and that task B runs
there from t2 to t3 there will be events about that in our
trace-dom0.dat file (collected via trace-cmd), and the CPU-ID of these
events will be 5.

A "Xen-Combo" plugin will have to have a way to know that events from
the trace-dom0.dat file which have a CPU-IDs equal to 5 should be
matched with task 'd0/v5' from the trace.xen file (collected via
xentrace).

It will therefore draw, in addition to the CPU plots that are already
there in the current version of the plugin (what plots one wants to
visualize is configurable, BTW) some special plots, one for each vCPU
running on the system, including the ones of dom0 (or, at least, one
for each vCPUs of each domain for which we have a trace file).

Among these special plots, there will be a 'd0/v5' plot which will have
a green box between t1 and t3. And "glued" on top of it, there will be
a 'CPU 5' plot which will have, say, a blue box between t1 and t2 and a
yellow box between t2 and t3.

As for the KVM case, the CPU plots will tell you on what physical CPU
d0v5 runs. While the special 'CPU 5' plot that is "glued" to the
'd0/v5' one will tell you what tasks run on vCPU 5 of Dom0, at any
given point in time.

Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: A KernelShark plugin for Xen traces analysis

[Dario Faggioli]

[-- Attachment #1: Type: text/plain, Size: 5239 bytes --]

On Wed, 2021-04-14 at 11:07 +0100, Andrew Cooper wrote:
> On 13/04/2021 16:46, Steven Rostedt wrote:
> In a Xen system, dom0 is just a VM, and particularly on larger servers,
> may not be as many vcpus as the system has logical threads.
> 
> This causes major problems for `perf` support under Xen, which assumes
> that the kernel's idea of CPUs matches that of the system.
> 
Yeah, but we wouldn't need anything like that, in this case.

> When rendering a trace including Xen data, Xen can provide the real
> system CPUs, and dom0 wants to be rendered as a VM under Xen, similar
> to
> trace-Fedora21 in your screenshot above.  (Obviously, if you're doing
> nested virt, things need to start nesting.)
> 
Exactly!

So, basically, events captured with trace-cmd on Dom0 will end up in a,
say, trace-dom0.dat file. It will contain ftrace event records, each
one with a timestamp and the ID of the CPU where it occurred. Of
course, in this case, that ID would be the ID of one of the Dom0 vCPUs
(seen from inside Dom0 itself).

And the same for all the trace files collected inside any other guest.

In KVM, vCPUs are Linux task. In fact, in Steve's screenshot [1], you
can see a task called 'CPU 0/KVM', with PID 2356. Note, however, that
with the KVM-Combo view active, there's a special 'CPU 0/KVM-2356' plot
in the graph. That shows _when_ that particular task, which is vCPU 0
of the KVM VM, runs. It's special because it has a 'CPU 0' plot "glued"
to it. In that plot we see the tasks (and the events) that were running
inside the VM, on vCPU 0.

[1] http://rostedt.org/private/kernelshark-kvm.png

So, if task A run from time t1 to time t2 on vCPU 0 of the Fedora21 KVM
guest, which in turn was running on physical CPU 3, we will see a non-
white (magenta, in the screenshot) box between t1 and t2 in the 'CPU
0/KVM-2356'. And we will see a non-white box on the 'CPU 0' plot which
is "glued" to 'CPU 0/KVM-2356' (it will be of whatever color KS decides
decide to use for task A, say, blue).

If at time t2 task B starts to run on vCPU 0 of Fedora21, and runs
until t3, we'll still see the magenta box on the 'CPU 0/KVM-2356',
between t2 and t3, while on the 'CPU 0' plot glued to it, we'll see a
box of whatever color is chosen for B (say, yellow).

Note that you don't see, from just this plots, _where_ task 2356 is
running, as in on which physical CPU. That will be visible in the CPU
plots, which do not appear in Steve's screenshot. Basically, if 2356
(i.e., the task corresponding to vCPU 0 of the Fedora21 VM) was running
on physical CPU 3 between t1 and t3, we'll see a magenta box, between
t1 and t3, on the plot called 'CPU 3 (and this is a "single plot",
i.e., one that does not have any other plot "glued" to it, e.g., like
the ones that are visible in Giuseppe's screenshot).

In the case of Xen, it's not that different. vCPUs are modeled and
considered by the plugin as if they where Linux tasks. In fact, you see
in Giuseppe's screenshot [2] that the 'Task' column contains strings
such as "d0/v5" or "idle/v4".

[2]
https://raw.githubusercontent.com/giuseppe998e/kernelshark-xentrace-plugin/master/.github/img/ks-xentrace.png

We needed to come up with a PID, as KS not only wants to print it in
the relevant column, but it uses it for associating colors to tasks
(i.e., colors to vCPUs, in our case). Currently, the value stored in
that PID field is computed as a combination of domain ID and vCPU ID.

If d0v5 runs on CPU 6 from t1 to t3, we'll see a, let's say, green box
between t1 and t3 on the CPU plot CPU 6. And this is something that
Giuseppe's plugin can already tell (show! :-D) us.

If task A runs on vCPU 5 of Dom0 from t1 to t2 and that task B runs
there from t2 to t3 there will be events about that in our
trace-dom0.dat file (collected via trace-cmd), and the CPU-ID of these
events will be 5.

A "Xen-Combo" plugin will have to have a way to know that events from
the trace-dom0.dat file which have a CPU-IDs equal to 5 should be
matched with task 'd0/v5' from the trace.xen file (collected via
xentrace).

It will therefore draw, in addition to the CPU plots that are already
there in the current version of the plugin (what plots one wants to
visualize is configurable, BTW) some special plots, one for each vCPU
running on the system, including the ones of dom0 (or, at least, one
for each vCPUs of each domain for which we have a trace file).

Among these special plots, there will be a 'd0/v5' plot which will have
a green box between t1 and t3. And "glued" on top of it, there will be
a 'CPU 5' plot which will have, say, a blue box between t1 and t2 and a
yellow box between t2 and t3.

As for the KVM case, the CPU plots will tell you on what physical CPU
d0v5 runs. While the special 'CPU 5' plot that is "glued" to the
'd0/v5' one will tell you what tasks run on vCPU 5 of Dom0, at any
given point in time.

Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: A KernelShark plugin for Xen traces analysis

[Dario Faggioli]

[-- Attachment #1: Type: text/plain, Size: 2514 bytes --]

On Wed, 2021-04-14 at 21:05 +0100, Andrew Cooper wrote:
> On 14/04/2021 14:43, Steven Rostedt wrote:
> > 
> > What I would envision how this would work, is that you would
> > produce a
> > set of tracing files. One for each guest (including Dom0), and one
> > for the
> > Xen hypervisor itself. The trick is to have a way to synchronize
> > the time
> > stamps. What we just did with KVM is to have all the tracing record
> > the
> > CPUs TSC, including the shift and multiplier that the CPU might
> > change for
> > the guests. Then we have a way to convert the TSC to nanoseconds.
> > This way
> > all tracing data has the same clock. It's somewhat complicated to
> > get
> > right, and requires access to how the guests clocks are modified by
> > the CPU.
> 
> Xen certainly has enough information to describe what TSC rate/epoch
> each guest is seeing, but I doubt any of this is coherently exposed
> at
> the moment.
> 
Yeah. In KVM its in, e.g.:

/sys/kernel/debug/kvm/85276-13/vcpu0/tsc-offset

Where 85276-13 is a combination of the PID of the main QEMU process and
one of the KVM file descriptors.

> > The Xen plugin would then need to read the how the threads in
> > xen.dat map
> > to the virtual CPUs of each of the guest files. Which would give
> > you the
> > layering.
> 
> Looks good.  I suspect we might need to do a little work on Xen's
> trace
> data to make this mesh together nicely.  In particular, Xen doesn't
> have
> a terribly good scheme on unique IDs for "a VM".
> 
True. However, Linux recycles PIDs too, doesn't it?

I mean, if PID 1234 is running when I start tracing and it dies and,
after a while, another task with PID 1234 is started KernelShark will
probably treat and display them as if they were the same task? Or
wouldn't it?

Currently, the Xen plugin combines the domain ID and the vCPU ID to
come up with a PID, that then will be used for assigning a color to
each vCPU.

This is pretty much arbitrary and can be changed, but it of course has
to be something that we can extract from from the trace event records
or, at least, from the trace file (e.g., as Andrew said, we do have
per-VM UUIDs, but they're currently not there).

Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: A KernelShark plugin for Xen traces analysis

[Dario Faggioli]

[-- Attachment #1: Type: text/plain, Size: 2514 bytes --]

On Wed, 2021-04-14 at 21:05 +0100, Andrew Cooper wrote:
> On 14/04/2021 14:43, Steven Rostedt wrote:
> > 
> > What I would envision how this would work, is that you would
> > produce a
> > set of tracing files. One for each guest (including Dom0), and one
> > for the
> > Xen hypervisor itself. The trick is to have a way to synchronize
> > the time
> > stamps. What we just did with KVM is to have all the tracing record
> > the
> > CPUs TSC, including the shift and multiplier that the CPU might
> > change for
> > the guests. Then we have a way to convert the TSC to nanoseconds.
> > This way
> > all tracing data has the same clock. It's somewhat complicated to
> > get
> > right, and requires access to how the guests clocks are modified by
> > the CPU.
> 
> Xen certainly has enough information to describe what TSC rate/epoch
> each guest is seeing, but I doubt any of this is coherently exposed
> at
> the moment.
> 
Yeah. In KVM its in, e.g.:

/sys/kernel/debug/kvm/85276-13/vcpu0/tsc-offset

Where 85276-13 is a combination of the PID of the main QEMU process and
one of the KVM file descriptors.

> > The Xen plugin would then need to read the how the threads in
> > xen.dat map
> > to the virtual CPUs of each of the guest files. Which would give
> > you the
> > layering.
> 
> Looks good.  I suspect we might need to do a little work on Xen's
> trace
> data to make this mesh together nicely.  In particular, Xen doesn't
> have
> a terribly good scheme on unique IDs for "a VM".
> 
True. However, Linux recycles PIDs too, doesn't it?

I mean, if PID 1234 is running when I start tracing and it dies and,
after a while, another task with PID 1234 is started KernelShark will
probably treat and display them as if they were the same task? Or
wouldn't it?

Currently, the Xen plugin combines the domain ID and the vCPU ID to
come up with a PID, that then will be used for assigning a color to
each vCPU.

This is pretty much arbitrary and can be changed, but it of course has
to be something that we can extract from from the trace event records
or, at least, from the trace file (e.g., as Andrew said, we do have
per-VM UUIDs, but they're currently not there).

Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: A KernelShark plugin for Xen traces analysis

[Dario Faggioli]

[-- Attachment #1: Type: text/plain, Size: 1489 bytes --]

On Wed, 2021-04-14 at 15:07 -0400, Steven Rostedt wrote:
> On Wed, 14 Apr 2021 19:11:19 +0100
> Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> 
> > Where the plugin (ought to) live depends heavily on whether we
> > consider
> > the trace format a stable ABI or not.
> 
> Agreed. Like the VMware plugin to handle ESX traces. It's internal
> and not
> published as the API is not stable.
> 
Mmm... Does this imply that Linux's tracepoints should be considered a
stable ABI then? :-D :-D :-D

> But if it ever becomes stable, and you would like it to live with
> KernelShark, we are looking to have a place to store third party
> plugins.
> 
Sure. TBH, either Xen or KernelShark main or plugin repositories would
be fine for me.

Which doesn't mean we should choose randomly, as clearly each solution
has pros and cons that needs to be evaluated.

I'm just saying that we would prefer the plugin to end up in one of
those places, rather than remaining its own project. And of course
we're up for maintaining it, wherever it lands. :-)

> We are working to make sure that the API for KernelShark plugins
> remains
> stable, so your plugins should always work too.
> 
Great!

Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: A KernelShark plugin for Xen traces analysis

[Dario Faggioli]

[-- Attachment #1: Type: text/plain, Size: 1489 bytes --]

On Wed, 2021-04-14 at 15:07 -0400, Steven Rostedt wrote:
> On Wed, 14 Apr 2021 19:11:19 +0100
> Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> 
> > Where the plugin (ought to) live depends heavily on whether we
> > consider
> > the trace format a stable ABI or not.
> 
> Agreed. Like the VMware plugin to handle ESX traces. It's internal
> and not
> published as the API is not stable.
> 
Mmm... Does this imply that Linux's tracepoints should be considered a
stable ABI then? :-D :-D :-D

> But if it ever becomes stable, and you would like it to live with
> KernelShark, we are looking to have a place to store third party
> plugins.
> 
Sure. TBH, either Xen or KernelShark main or plugin repositories would
be fine for me.

Which doesn't mean we should choose randomly, as clearly each solution
has pros and cons that needs to be evaluated.

I'm just saying that we would prefer the plugin to end up in one of
those places, rather than remaining its own project. And of course
we're up for maintaining it, wherever it lands. :-)

> We are working to make sure that the API for KernelShark plugins
> remains
> stable, so your plugins should always work too.
> 
Great!

Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: A KernelShark plugin for Xen traces analysis

[Steven Rostedt]

On Thu, 15 Apr 2021 02:50:53 +0200
Dario Faggioli <dfaggioli@suse.com> wrote:

> On Wed, 2021-04-14 at 15:07 -0400, Steven Rostedt wrote:
> > On Wed, 14 Apr 2021 19:11:19 +0100
> > Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> >   
> > > Where the plugin (ought to) live depends heavily on whether we
> > > consider
> > > the trace format a stable ABI or not.  
> > 
> > Agreed. Like the VMware plugin to handle ESX traces. It's internal
> > and not
> > published as the API is not stable.
> >   
> Mmm... Does this imply that Linux's tracepoints should be considered a
> stable ABI then? :-D :-D :-D

Some already are. Like the sched_switch tracepoint. That's one of the
reasons Peter Zijlstra now hates exported tracepoints.

> 
> > But if it ever becomes stable, and you would like it to live with
> > KernelShark, we are looking to have a place to store third party
> > plugins.
> >   
> Sure. TBH, either Xen or KernelShark main or plugin repositories would
> be fine for me.
> 
> Which doesn't mean we should choose randomly, as clearly each solution
> has pros and cons that needs to be evaluated.
> 
> I'm just saying that we would prefer the plugin to end up in one of
> those places, rather than remaining its own project. And of course
> we're up for maintaining it, wherever it lands. :-)

Like I said, we can have a third party repository within the KernelShark
repo (or along side of it). As a claws-mail user, I like their method. They
have a bunch of plugins you can add that they have in their repo, but those
plugins are maintained by different people.

-- Steve


> 
> > We are working to make sure that the API for KernelShark plugins
> > remains
> > stable, so your plugins should always work too.
> >   
> Great!
> 
> Regards

Re: A KernelShark plugin for Xen traces analysis

[Steven Rostedt]

On Thu, 15 Apr 2021 02:50:53 +0200
Dario Faggioli <dfaggioli@suse.com> wrote:

> On Wed, 2021-04-14 at 15:07 -0400, Steven Rostedt wrote:
> > On Wed, 14 Apr 2021 19:11:19 +0100
> > Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> >   
> > > Where the plugin (ought to) live depends heavily on whether we
> > > consider
> > > the trace format a stable ABI or not.  
> > 
> > Agreed. Like the VMware plugin to handle ESX traces. It's internal
> > and not
> > published as the API is not stable.
> >   
> Mmm... Does this imply that Linux's tracepoints should be considered a
> stable ABI then? :-D :-D :-D

Some already are. Like the sched_switch tracepoint. That's one of the
reasons Peter Zijlstra now hates exported tracepoints.

> 
> > But if it ever becomes stable, and you would like it to live with
> > KernelShark, we are looking to have a place to store third party
> > plugins.
> >   
> Sure. TBH, either Xen or KernelShark main or plugin repositories would
> be fine for me.
> 
> Which doesn't mean we should choose randomly, as clearly each solution
> has pros and cons that needs to be evaluated.
> 
> I'm just saying that we would prefer the plugin to end up in one of
> those places, rather than remaining its own project. And of course
> we're up for maintaining it, wherever it lands. :-)

Like I said, we can have a third party repository within the KernelShark
repo (or along side of it). As a claws-mail user, I like their method. They
have a bunch of plugins you can add that they have in their repo, but those
plugins are maintained by different people.

-- Steve


> 
> > We are working to make sure that the API for KernelShark plugins
> > remains
> > stable, so your plugins should always work too.
> >   
> Great!
> 
> Regards

Re: A KernelShark plugin for Xen traces analysis ​

[Giuseppe Eletto]

[-- Attachment #1: Type: text/plain, Size: 3829 bytes --]

Hi,

As Dario already said, the plugin seems to work well with all the traces we
fed to it.
The largest trace was nearly 500MB, but still it had no trouble showing it.

The only minor troubles I encountered writing the plugin is with the
different data
types between KShark and XenTrace: for example regarding the "event_id"
field of
the "struct kshark_entry" structure which in KShark has type "int16_t"
while XenTrace
has events of the type "uint32_t" at 28 bits.
But since there are functions like "get_event_id" these differences can be
bypassed without seemingly creating problems.

So yes, the plugin is still a prototype. The main purpose so far was to
create
something that works: in fact, as mr. @Andrew_Cooper pointed out, the
couple
"dom:vcpu" is still displayed as a HEX number without any kind of
formatting.
This is because at the moment I have copied the same formatting as the
"xentrace_format" tool.
So a future plan for updating this plugin could be to improve its output.

As for the tutorial, I wouldn't mind trying, but right now I'm busy writing
the thesis for the graduation...

Kind regards,
Giuseppe Eletto.

Il giorno mer 14 apr 2021 alle ore 11:26 Yordan Karadzhov (VMware) <
y.karadz@gmail.com> ha scritto:

> Hi Giuseppe,
>
> It is great to see such progress in the development of the Xen plugin.
>
> Can you share with us what are your plans for continuing this work. Is
> this a first prototype of the plugin, or it is an almost final version?
>
> I was also thinking that maybe you can prepare a short tutorial on
> writing KernelShark plugins for data inputs. You have the best
> experience for this task, since you are the first to develop an external
> plugin. Ideally, this can even turn into a talk that you can give at
> some open source event.
> What do you think?
>
> Best,
> Yordan
>
>
> On 13.04.21 г. 17:28, Giuseppe Eletto wrote:
> > Hello,
> > I want to share with you a new plugin developed by me, under the
> > supervision of Dario Faggioli, which allows the new version of
> KernelShark
> > (the v2-beta) to open and view the Xen traces created using the
> "xentrace" tool.
> >
> > In fact, KernelShark is a well known tool for graphical visualization
> > Linux kernel traces, obtained via "ftrace" and "trace-cmd". Anyway thanks
> > to its modular architecture, it is now possible to implement plugins
> which
> > open and display traces with arbitrary format, for example, as in in
> > this case, traces of the Xen hypervisor.
> >
> > For more information on how to build the plugin and/or
> > to view the source code I leave the repository below:
> > https://github.com/giuseppe998e/kernelshark-xentrace-plugin
> >
> >
> > In short:
> >
> > $ sudo apt install git build-essential libjson-c-dev
> > $ git clone --recurse-submodules
> > https://github.com/giuseppe998e/kernelshark-xentrace-plugin.git
> > $ cd kernelshark-xentrace-plugin/
> > $ make
> >
> > $ export XEN_CPUHZ=3G # Sets the CPU frequency ((G)hz/(M)hz/(K)hz/hz)
> > $ kernelshark -p out/ks-xentrace.so trace.xen
> >
> >
> > You will need the development version of KernelShark, available here:
> > https://git.kernel.org/pub/scm/utils/trace-cmd/kernel-shark.git
> >
> > A screenshot of the plugin in action is available here:
> >
> https://github.com/giuseppe998e/kernelshark-xentrace-plugin/raw/master/.github/img/ks-xentrace.png
> >
> > I'm happy to receive whatever feedback you may have about it,
> > and to answer any question.
> >
> > Kind regards,
> > Giuseppe Eletto.
> >
>

-- 
------------------------



Indirizzo istituzionale di posta elettronica 
degli studenti e dei laureati dell'Università degli Studi di TorinoOfficial 
University of Turin email address for students and graduates 

[-- Attachment #2: Type: text/html, Size: 5565 bytes --]