Re: Accuracy of traces sync [Was: Re: [PATCH] Fix `make -jN trace-cmd gui`]

[Tzvetomir Stoyanov <tz.stoyanov@gmail.com>]

On Thu, Nov 26, 2020 at 2:15 AM Dario Faggioli <dfaggioli@suse.com> wrote:
>
> On Fri, 2020-11-20 at 09:08 -0500, Steven Rostedt wrote:
> > On Fri, 20 Nov 2020 14:43:21 +0100
> > Dario Faggioli <dfaggioli@suse.com> wrote:
> > >
> > > So, you often say that "the accuracy of the synchronization
> > > protocol is
> > > XX ms". Now, I guess that means that an event in the guest and the
> >
> > Note, we are usually microsecond (us) apart, not millisecond (ms) ;-)
> >
> Ah, yes, sure... And sorry about that! I know it us, I'm not sure how I
> ended up writing ms. That would be quite terrible indeed! :-D
>
> > > corresponding event in the host (or vice versa) are XX ms apart.
> > > And
> > > that's even after the synchronization of the two traces, is that
> > > right?
> >
> > At plumbers we talked with Thomas Gleixner and he suggested ideas of
> > how to
> > get to the actual shifts used in the hardware that should give us
> > exact
> > timestamp offsets. We are currently working on that.
> >
> Yes, I remember that, I attended the BoF.
>
> > But in the mean time,
> > the P2P is giving us somewhere between 5 and 10 us accuracy. And
> > that's
> > simply because the jitter of the vsock connection (which is used for
> > the
> > synchronization at start and end of the traces) has a 5 to 10 us
> > jitter,
> > and it's not possible to get a more accurate than the medium that is
> > being
> > used.
> >
> Yes, with a student that I was helping with his thesis, we applied one
> debug patch to trace-cmd that you have on this list, and we tried the
> different synchronization strategies, frequency, etc.
>
> > > Question is, how do you measure that? Sure, I can look manually for
> > > an
> > > occurrence of the pattern that I described above: i.e., an event in
> > > the
> > > guest, then the corresponding one in the host and compute the
> > > difference between the timestamps.
> >
> > You mean, how we measure the accuracy? It's usually done by seeing
> > when we
> > have events from the guest showing up when we should be in the host
> > (it's
> > like seeing events from userspace when you are in the kernel).
> >
> Ok, makes sense. I need to try it first hand to make sure I've properly
> understood it, though. I'll collect some more tracing and looks for
> situations like these.
>
> Thanks!
>
> > > But do you have a way to do so automatically, or with a
> > > script/program,
> > > etc?
> >
> > We have talked about having something scan to find cases where the
> > guest
> > event happens in kernel and do some post processing shifting, but
> > haven't
> > gotten there yet.
> >
> Yep, as said, I was thinking at it as a way to measure how accurately
> the traces are synched, but indeed once one has it, it can even use it
> to actually synch them better.

Hi Dario
There is one approach to measure the accuracy of the synchronisation.
May be you remember the suggestions proposed in the BoF - KVM exposes
the clock offset and scaling factor in the /proc FS. In the last
version of the time
sync patch set, v25, I've implemented such logic - when x86-tsc is used for
trace clock source, the offset is read from the KVM debug FS. In theory this
should bring the best synchronisation, but I still see some guest events in
wrong order according to the host events.
I use the offset exposed from KVM as a reference - run PTP algorithm using
the x86-tsc trace clock and compare the calculated offset with the one from
the KVM debug FS.
We still need to find a way to improve the synchronisation accuracy for cases
when other hypervisors and trace clocks are used.


>
> But I understand how it's rather tricky.
>
> > If the hardware settings can work, then there will be no
> > need to do so.
> >
> Indeed. Well, perhaps it could still be useful, as a test/check whether
> things are working? :-)
>
> Regards
> --
> Dario Faggioli, Ph.D
> http://about.me/dario.faggioli
> Virtualization Software Engineer
> SUSE Labs, SUSE https://www.suse.com/
> -------------------------------------------------------------------
> <<This happens because _I_ choose it to happen!>> (Raistlin Majere)



--
Tzvetomir (Ceco) Stoyanov
VMware Open Source Technology Center