Re: [patch 3/3] PTP: add kvm PTP driver

[Radim Krcmar <rkrcmar@redhat.com>]

2017-01-18 13:46+0100, Paolo Bonzini:
> 
> 
> On 18/01/2017 13:24, Marcelo Tosatti wrote:
> > On Wed, Jan 18, 2017 at 10:17:38AM -0200, Marcelo Tosatti wrote:
> >> On Tue, Jan 17, 2017 at 04:36:21PM +0100, Radim Krcmar wrote:
> >>> 2017-01-17 09:30-0200, Marcelo Tosatti:
> >>>> On Tue, Jan 17, 2017 at 09:03:27AM +0100, Miroslav Lichvar wrote:
> >>>>> Users of the PTP_SYS_OFFSET ioctl assume that (ts[0]+ts[2])/2
> >>>>> corresponds to ts[1], (ts[2]+ts[4])/2 corresponds to ts[3], and so on.
> >>>>>
> >>>>>                     ts[1]     ts[3]
> >>>>> Host time    ---------+---------+........
> >>>>>                       |         |
> >>>>>                       |         |
> >>>>> Guest time   ----+---------+---------+......
> >>>>>                 ts[0]    ts[2]     ts[4]
> >>>
> >>> KVM PTP delay moves host ts[i] to be close to guest ts[i+1] and makes
> >>> the offset very consistent, so the graph would look like:
> >>>
> >>>                         ts[1]     ts[3]
> >>> Host time    -------------+---------+........
> >>>                           |         |
> >>>                           |         |
> >>> Guest time   ----+---------+---------+......
> >>>                 ts[0]    ts[2]     ts[4]
> >>>
> >>> which doesn't sound good if users assume that the host reading is in the
> >>> middle -- the guest time would be ahead of the host time.
> >>
> >> Testcase: run a guest and a loop sending SIGUSR1 to vcpu0 (emulating
> >> intense interrupts). Follows results:
> >>
> >> Without TSC delta calculation:
> >> =============================
> >>
> >> #* PHC0                          0   3   377     2    -99ns[ +206ns] +/-  116ns
> >> #* PHC0                          0   3   377     8   +202ns[ +249ns] +/-  111ns
> >> #* PHC0                          0   3   377     8   -213ns[ +683ns] +/-   88ns
> >> #* PHC0                          0   3   377     6    +77ns[ +319ns] +/-   56ns
> >> #* PHC0                          0   3   377     4   -771ns[-1029ns] +/-   93ns
> >> #* PHC0                          0   3   377    10    -49ns[  -58ns] +/-  121ns
> >> #* PHC0                          0   3   377     9   +562ns[ +703ns] +/-  107ns
> >> #* PHC0                          0   3   377     6     -2ns[   -3ns] +/-   94ns
> >> #* PHC0                          0   3   377     4   +451ns[ +494ns] +/-  138ns
> >> #* PHC0                          0   3   377    11    -67ns[  -74ns] +/-  113ns
> >> #* PHC0                          0   3   377     8   +244ns[ +264ns] +/-  119ns
> >> #* PHC0                          0   3   377     7   -696ns[ -890ns] +/-   89ns
> >> #* PHC0                          0   3   377     4   +468ns[ +560ns] +/-  110ns
> >> #* PHC0                          0   3   377    11   -310ns[ -430ns] +/-   72ns
> >> #* PHC0                          0   3   377     9   +189ns[ +298ns] +/-   54ns
> >> #* PHC0                          0   3   377     7   +594ns[ +473ns] +/-   96ns
> >> #* PHC0                          0   3   377     5   +151ns[ +280ns] +/-   71ns
> >> #* PHC0                          0   3   377    10   -590ns[ -696ns] +/-   94ns
> >> #* PHC0                          0   3   377     8   +415ns[ +526ns] +/-   74ns
> >> #* PHC0                          0   3   377     6  +1381ns[+1469ns] +/-  101ns
> >> #* PHC0                          0   3   377     4   +571ns[+1304ns] +/-   54ns
> >> #* PHC0                          0   3   377     8     -5ns[  +71ns] +/-  139ns
> >> #* PHC0                          0   3   377     7   -247ns[ -502ns] +/-   69ns
> >> #* PHC0                          0   3   377     5   -283ns[ +879ns] +/-   73ns
> >> #* PHC0                          0   3   377     3   +148ns[ -109ns] +/-   61ns
> >>
> >> With TSC delta calculation:
> >> ============================
> >>
> >> #* PHC0                          0   3   377     7   +379ns[ +432ns] +/-   53ns
> >> #* PHC0                          0   3   377     9   +106ns[ +420ns] +/-   42ns
> >> #* PHC0                          0   3   377     7    -58ns[ -136ns] +/-   62ns
> >> #* PHC0                          0   3   377    12    +93ns[  -38ns] +/-   64ns
> >> #* PHC0                          0   3   377     8    +84ns[ +107ns] +/-   69ns
> >> #* PHC0                          0   3   377     3    -76ns[ -103ns] +/-   52ns
> >> #* PHC0                          0   3   377     7    +52ns[  +63ns] +/-   50ns
> >> #* PHC0                          0   3   377    11    +29ns[  +31ns] +/-   70ns
> >> #* PHC0                          0   3   377     7    -47ns[  -56ns] +/-   42ns
> >> #* PHC0                          0   3   377    10    -35ns[  -42ns] +/-   33ns
> >> #* PHC0                          0   3   377     7    -32ns[  -34ns] +/-   42ns
> >> #* PHC0                          0   3   377    11   -172ns[ -173ns] +/-  118ns
> >> #* PHC0                          0   3   377     6    +65ns[  +76ns] +/-   23ns
> >> #* PHC0                          0   3   377     9    +18ns[  +23ns] +/-   37ns
> >> #* PHC0                          0   3   377     6    +41ns[  -60ns] +/-   30ns
> >> #* PHC0                          0   3   377    10    +39ns[ +183ns] +/-   42ns
> >> #* PHC0                          0   3   377     6    +50ns[ +102ns] +/-   86ns
> >> #* PHC0                          0   3   377    11    +50ns[  +75ns] +/-   52ns
> >> #* PHC0                          0   3   377     6    +50ns[ +116ns] +/-  100ns
> >> #* PHC0                          0   3   377    10    +46ns[  +65ns] +/-   79ns
> >> #* PHC0                          0   3   377     7    -38ns[  -51ns] +/-   29ns
> >> #* PHC0                          0   3   377    10    -11ns[  -12ns] +/-   32ns
> >> #* PHC0                          0   3   377     7    -31ns[  -32ns] +/-   99ns
> >> #* PHC0                          0   3   377    10   +222ns[ +238ns] +/-   58ns
> >> #* PHC0                          0   3   377     6   +185ns[ +207ns] +/-   39ns
> >> #* PHC0                          0   3   377    10   -392ns[ -394ns] +/-  118ns
> >> #* PHC0                          0   3   377     6     -9ns[  -50ns] +/-   35ns
> >> #* PHC0                          0   3   377    10   -346ns[ -355ns] +/-  111ns
> >>
> >>
> >> Do you still want to drop it in favour of simplicity?
> > 
> > This is the output of "chronyc sources". See section "Time sources"
> > of https://chrony.tuxfamily.org/doc/2.4/chronyc.html.
> 
> It's just that it's not obvious why you get better results with biased
> host timestamps.  What makes the biased host timestamp more precise?

I think that TSC offsetting is more precise (the difference between two
samples smaller), but less accurate (the difference between guest and
host clock is bigger).
And chronyc can't tell how much off from the host clock our guest clock
is (that would be measured how I wrote above).

The main reason why it is more precise is because the TSC offset erases
the effect of many latency sources that can happen before or after the
host time read.

The precision with TSC offsetting is almost only affected by the
duration between two guest time measurements, because the host time is
shifted to be very close to the latter guest time measurement.

Without TSC offsetting, the host time will be closer to the middle
between two guest reads (= better accuracy), but the moment of the tsc
read on the host has more variables -- host interrupts and whatnot,
which will result in lower precision.

> I'd rather use PTP_SYS_OFFSET_PRECISE instead, but unfortunately chrony
> does not support it---but I would still prefer you to support
> PTP_SYS_OFFSET_PRECISE as well.

Oh, I didn't expect that userspace can't use PTP_SYS_OFFSET_PRECISE ...