xen-devel.lists.xenproject.org archive mirror
 help / color / mirror / Atom feed
From: "Jan Beulich" <JBeulich@suse.com>
To: xen-devel <xen-devel@lists.xenproject.org>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>,
	Dario Faggioli <dario.faggioli@citrix.com>,
	Joao Martins <joao.m.martins@oracle.com>
Subject: [PATCH 0/8] x86/time: improve cross-CPU clock monotonicity (and more)
Date: Wed, 15 Jun 2016 03:50:11 -0600	[thread overview]
Message-ID: <576140F302000078000F52FE@prv-mh.provo.novell.com> (raw)

1: x86/time: adjust local system time initialization
2: x86: also generate assembler usable equates for synthesized features
3: x86/time: introduce and use rdtsc_ordered()
4: x86/time: calibrate TSC against platform timer
5: x86/time: correctly honor late clearing of TSC related feature flags
6: x86/time: support 32-bit wide ACPI PM timer
7: x86/time: fold recurring code
8: x86/time: group time stamps into a structure

The first patch was submitted on its own before; I'd like it to be
looked at in the context of this entire series though, and hence I'm
including it here again.

Expectations towards the improvements of what is now patch 4
didn't get fulfilled (it brought down the skew only be a factor of
two in my measurements), but to "level" that improvements by
patch 3 were significantly larger than expected (hence the two
got switched in order, albeit I can't say whether the improvements
by 3 would also be as prominent without patch 4, as I simply never
measured 3 alone): I've observed the TSC value read without the
ordering fence to be up to several hundred clocks off compared to
the fenced variant.

I'd like to note though that even with the full series in place I'm
still seeing some skew, and measurement over about 24h suggests
that what is left now is actually increasing over time (which
obviously is bad). What looks particularly odd to me is that over
longer periods of time maximum skew between hyperthreads
stays relatively stable on a certain core, but eventually another
core may "overtake"). So far I did not have a good idea of what
might be causing this (short of suspecting constant TSC not really
being all that constant).

Another observed oddity is that with this debugging code

@@ -1259,6 +1327,14 @@ time_calibration_rendezvous_tail(const s
     c->local_tsc    = rdtsc_ordered();
     c->local_stime  = get_s_time_fixed(c->local_tsc);
     c->master_stime = r->master_stime;
+{//temp
+ s64 delta = rdtsc_ordered() - c->local_tsc;
+ static s64 delta_max = 100;
+ if(delta > delta_max) {
+  delta_max = delta;
+  printk("CPU%02u: delta=%ld\n", smp_processor_id(), delta);
+ }
+}
 
     raise_softirq(TIME_CALIBRATE_SOFTIRQ);
 }

added on top of the series deltas of beyond 20,000 get logged.
Afaict (I admit I didn't check) interrupts are off here, so I can't
see where such spikes would result from. Is a watchdog NMI a
possible explanation for this? If so, what other bad effects on
time calculations might result from such events? And how would
one explain spikes of both around 13,000 and 22,000 with this
(I'd expect individual watchdog NMIs to always take about the
same time)?

Signed-off-by: Jan Beulich <jbeulich@suse.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

             reply	other threads:[~2016-06-15  9:50 UTC|newest]

Thread overview: 39+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-06-15  9:50 Jan Beulich [this message]
2016-06-15 10:26 ` [PATCH 1/8] x86/time: improve cross-CPU clock monotonicity (and more) Jan Beulich
2016-06-15 10:32   ` Jan Beulich
2016-06-15 22:51   ` Joao Martins
2016-06-16  8:27     ` Jan Beulich
2016-06-16 20:27       ` Joao Martins
2016-06-17  7:32         ` Jan Beulich
2016-06-21 12:05           ` Joao Martins
2016-06-21 12:28             ` Jan Beulich
2016-06-21 13:57               ` Joao Martins
2016-08-02 19:30   ` Andrew Cooper
2016-06-15 10:26 ` [PATCH 2/8] x86: also generate assembler usable equates for synthesized features Jan Beulich
2016-06-20 12:50   ` Andrew Cooper
2016-06-15 10:27 ` [PATCH 3/8] x86/time: introduce and use rdtsc_ordered() Jan Beulich
2016-06-20 12:59   ` Andrew Cooper
2016-06-20 13:06     ` Jan Beulich
2016-06-20 13:07       ` Andrew Cooper
2016-07-11 11:39     ` Dario Faggioli
2016-06-15 10:28 ` [PATCH 4/8] x86/time: calibrate TSC against platform timer Jan Beulich
2016-06-20 14:20   ` Andrew Cooper
2016-06-20 15:19     ` Jan Beulich
2016-08-02 19:25       ` Andrew Cooper
2016-08-03  9:32         ` Jan Beulich
2016-06-15 10:28 ` [PATCH 5/8] x86/time: correctly honor late clearing of TSC related feature flags Jan Beulich
2016-06-20 14:32   ` Andrew Cooper
2016-06-20 15:20     ` Jan Beulich
2016-07-04 15:39       ` Andrew Cooper
2016-07-04 15:53         ` Jan Beulich
2016-08-02 19:08           ` Andrew Cooper
2016-08-03  9:43             ` Jan Beulich
2016-08-31 13:42               ` Andrew Cooper
2016-08-31 14:03                 ` Jan Beulich
2016-06-15 10:29 ` [PATCH 6/8] x86/time: support 32-bit wide ACPI PM timer Jan Beulich
2016-07-04 15:40   ` Andrew Cooper
2016-06-15 10:30 ` [PATCH 7/8] x86/time: fold recurring code Jan Beulich
2016-07-04 15:43   ` Andrew Cooper
2016-06-15 10:30 ` [PATCH 8/8] x86/time: group time stamps into a structure Jan Beulich
2016-07-04 15:57   ` Andrew Cooper
2016-07-01  7:44 ` Ping: [PATCH 0/8] x86/time: improve cross-CPU clock monotonicity (and more) Jan Beulich

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=576140F302000078000F52FE@prv-mh.provo.novell.com \
    --to=jbeulich@suse.com \
    --cc=andrew.cooper3@citrix.com \
    --cc=dario.faggioli@citrix.com \
    --cc=joao.m.martins@oracle.com \
    --cc=xen-devel@lists.xenproject.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).