Re: Accuracy of traces sync [Was: Re: [PATCH] Fix `make -jN trace-cmd gui`]

[Steven Rostedt <rostedt@goodmis.org>]

On Fri, 20 Nov 2020 14:43:21 +0100
Dario Faggioli <dfaggioli@suse.com> wrote:

> On Fri, 2020-11-20 at 14:32 +0200, Yordan Karadzhov (VMware) wrote:
> > On 20.11.20 г. 14:06 ч., Dario Faggioli wrote:  
> > > >   
> > > Yep, sure! I followed the talks you've been giving about it at
> > > events, and I even played a little with what you have here:
> > > 
> > > https://github.com/yordan-karadzhov/kernel-shark-2.alpha
> > > 
> > > The host-guest tracing part, as I think you can guess.  
> > 
> > Ciao Dario,
> >  
> Hey!
> 
> > I am very happy to hear that. The timestamp synchronization patches
> > for 
> > trace-cmd are almost ready to go upstream.
> >   
> Yes, I tried those patches!
> 
> In fact, now that I have you here, do you mind if I change the subject
> and ask a quick question about them?
> 
> So, you often say that "the accuracy of the synchronization protocol is
> XX ms". Now, I guess that means that an event in the guest and the

Note, we are usually microsecond (us) apart, not millisecond (ms) ;-)

> corresponding event in the host (or vice versa) are XX ms apart. And
> that's even after the synchronization of the two traces, is that right?

At plumbers we talked with Thomas Gleixner and he suggested ideas of how to
get to the actual shifts used in the hardware that should give us exact
timestamp offsets. We are currently working on that. But in the mean time,
the P2P is giving us somewhere between 5 and 10 us accuracy. And that's
simply because the jitter of the vsock connection (which is used for the
synchronization at start and end of the traces) has a 5 to 10 us jitter,
and it's not possible to get a more accurate than the medium that is being
used.

> 
> Question is, how do you measure that? Sure, I can look manually for an
> occurrence of the pattern that I described above: i.e., an event in the
> guest, then the corresponding one in the host and compute the
> difference between the timestamps.

You mean, how we measure the accuracy? It's usually done by seeing when we
have events from the guest showing up when we should be in the host (it's
like seeing events from userspace when you are in the kernel).

> 
> But do you have a way to do so automatically, or with a script/program,
> etc?

We have talked about having something scan to find cases where the guest
event happens in kernel and do some post processing shifting, but haven't
gotten there yet. If the hardware settings can work, then there will be no
need to do so.

-- Steve

> 
> I saw that the series included a patch which was meant at debugging and
> profiling PTP, but even with that one applied and having it generate
> the graphs, I have not been able to get that info without manual
> inspection.
> 
> > And we are currently reviewing a beta version of KernelShark_2 that
> > will 
> > include the guest/host visualization.
> > As Steven said: stay tuned!
> >   
> You bet I will. :-P
> 
> Thanks and Regards