Re: Instructions for clock sync for tracing host/guest

[Tzvetomir Stoyanov <tz.stoyanov@gmail.com>]

Hi Joel,

On Fri, Apr 23, 2021 at 1:50 AM Joel Fernandes <joelaf@google.com> wrote:
>
> Looks like my trace.dat files bounced from goodmis.org so I uploaded them here:
> https://drive.google.com/file/d/16wGsVo4PJ0kYQGAy195dfBW0RHSWVkx0/view?usp=sharing
>
> ---------- Forwarded message ---------
> From: Joel Fernandes <joelaf@google.com>
> Date: Thu, Apr 22, 2021 at 6:46 PM
> Subject: Re: Instructions for clock sync for tracing host/guest
> To: Steven Rostedt <rostedt@goodmis.org>, Yordan Karadzhov (VMware)
> <y.karadz@gmail.com>
> Cc: Tzvetomir Stoyanov <tz.stoyanov@gmail.com>
>
>
> On Thu, Apr 22, 2021 at 5:12 PM Joel Fernandes <joelaf@google.com> wrote:
> >
> > On Thu, Apr 22, 2021 at 4:03 PM Steven Rostedt <rostedt@goodmis.org> wrote:
> > >
> > > On Thu, 22 Apr 2021 15:49:21 -0400
> > > Joel Fernandes <joelaf@google.com> wrote:
> > >
> > > > On Thu, Apr 22, 2021 at 3:48 PM Steven Rostedt <rostedt@goodmis.org> wrote:
> > > > >
> > > > > On Thu, 22 Apr 2021 15:43:37 -0400
> > > > > Joel Fernandes <joelaf@google.com> wrote:
> > > > >
> > > > > > >  # trace-cmd record -e irq -e sched -e kvm -A test@32:823 -e sched
> > > > > >
> > > > > > OMG, that worked, thank you Steve and Tzvetomir !!!!!
> > > > >
> > > > > Technically, you can probably leave off "-e irq", but I do find it rather
> > > > > useful. But then, I add "-e irq" to the guest as well. It lets me see how
> > > > > interrupts transpire from the host to the guest.
> > > >
> > > > Got it thanks for the tip. Will report back any issues I see.
> > >
> > > And you know you can pull this up into KernelShark as well, right?
> > >
> > > Checkout the development version from Yordan's repository:
> > >
> > >
> > >  $ git clone https://github.com/yordan-karadzhov/kernel-shark.git
> > >  $ cd kernel-shark
> > >  $ git checkout origin/yordan_devel
> > >  $ cmake .
> > >  $ make
> > >  $ sudo make install
> > >
> > > Then run kernelshark with:
> > >
> > >  $ kernelshark trace-host.dat -a trace-guest.dat
> > >
> > > (obviously, using the actual names of the trace.dat files for the host and
> > > the guest)
> > >
> > > Then you can select: Plots -> KVM Combo Plots
> > >
> > > Then select the "all" box, and then apply. Then you get something like this:
> > >
> > >   http://rostedt.org/private/ks-host-guest.png
> > >
> > > Where you see how the tasks mapping to the host and guest are aligned.
> > >
> > > That is, if everything works fine.
> >
> > Yes, I sort of knew KernelShark had this support, but so glad you sent
> > me more details on how to do it, I'll try it out. I am also wondering
> > how the bars of the vCPUs will look like because you can have vCPU
> > threads migrated to different physical CPUs. I will go try it out and
> > see what comes up :-)
>
> Hi all, Just wanted to report my progress today:
>
> When I try to open the guest+host trace with the yordan_devel branch and command
> kernelshark ~/vm-host-trace/trace.dat -a ~/vm-host-trace/trace-test.dat
>
> I get the following in the console: https://pastebin.com/raw/EmbsyuB8
>
> And the GUI looks like the attached picture, I don't see any mention
> of KVM like Steve sees in
> http://rostedt.org/private/ks-host-guest.png . What am I missing?
>
> I also zipped and attached my trace files to this email. Could you
> take a look at them?

Looks like there is a gap in our implementation, affecting your use
case. We rely too much on the quemu guest information that we gather,
which is missing on your setup. As this information is not mandatory,
the implementation should not rely on it. What we need:
 - PID of the process, running the guest VM. In case of KVM, we use
this PID to get the KVM guest TSC clock parameters, needed for better
host and guest trace timestamps synchronization. Without this PID, KVM
cannot be used for timestamps synchronization. The logic should fail
back to a PTP-like algorithm, which is more generic, works in all
environments, but is less accurate than KVM logic.
 - PIDs of each thread, running a guest virtual CPU. This is not
required for the tracing, but for better trace visualisation in
KernelShark. It helps to map the host task to a vCPU and to visualise
them together.

Currently we collect that information from quemu, is there a way to
get it from crosvm ? If yes, a crosvm support can be implemented in
trace-cmd. But as I said, it is not mandatory to have it for the
trace, I can send you a patch next week addressing this gap in the
implementation.

Thanks for testing this code!

>
> My kernelshark HEAD commit is: 070d657 ("kernel-shark: Add KVMCombo
> plugin") . Let me know if I should be building some other commit sha.
>
> Adding Yordan to this email as well.
>
> Thanks a lot!
> -Joel



-- 
Tzvetomir (Ceco) Stoyanov
VMware Open Source Technology Center