Dual core Athlons and unsynced TSCs

Dual core Athlons and unsynced TSCs

[Lee Revell]

It's been known for quite some time that the TSCs are not synced between
cores on Athlon X2 machines and this screws up the kernel's timekeeping,
as it still uses the TSC as the default time source on these machines.

This problem still seems to be present in the latest kernels.  What is
the plan to fix it?  Is the fix simply to make the kernel use the ACPI
PM timer by default on Athlon X2?

Lee

RE: Dual core Athlons and unsynced TSCs

[Roger Heflin]

 

> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org 
> [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Lee Revell
> Sent: Thursday, January 12, 2006 4:18 PM
> To: linux-kernel
> Subject: Dual core Athlons and unsynced TSCs
> 
> It's been known for quite some time that the TSCs are not 
> synced between cores on Athlon X2 machines and this screws up 
> the kernel's timekeeping, as it still uses the TSC as the 
> default time source on these machines.
> 
> This problem still seems to be present in the latest kernels. 
>  What is the plan to fix it?  Is the fix simply to make the 
> kernel use the ACPI PM timer by default on Athlon X2?


Do we know if this also affects dual-core opterons?

The symptoms are that the clocks run at 2x the speed, correct?

                               Roger

RE: Dual core Athlons and unsynced TSCs

[Steven Rostedt]

On Fri, 2006-01-13 at 09:10 -0600, Roger Heflin wrote:
>  
> > -----Original Message-----
> > From: linux-kernel-owner@vger.kernel.org 
> > [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Lee Revell
> > Sent: Thursday, January 12, 2006 4:18 PM
> > To: linux-kernel
> > Subject: Dual core Athlons and unsynced TSCs
> > 
> > It's been known for quite some time that the TSCs are not 
> > synced between cores on Athlon X2 machines and this screws up 
> > the kernel's timekeeping, as it still uses the TSC as the 
> > default time source on these machines.
> > 
> > This problem still seems to be present in the latest kernels. 
> >  What is the plan to fix it?  Is the fix simply to make the 
> > kernel use the ACPI PM timer by default on Athlon X2?
> 
> 
> Do we know if this also affects dual-core opterons?
> 
> The symptoms are that the clocks run at 2x the speed, correct?

No, worse.  The monotonic clock can go backwards.  The tscs of the CPUs
are not in sync when one slows down due to idle.  So if you read from
two different CPUs, you may get the second read have an earlier time
than the first.  Breaks the rule of what a monotonic clock is.

-- Steve

RE: Dual core Athlons and unsynced TSCs

[Lee Revell]

On Fri, 2006-01-13 at 11:04 -0500, Steven Rostedt wrote:
> On Fri, 2006-01-13 at 09:10 -0600, Roger Heflin wrote:
> >  
> > > -----Original Message-----
> > > From: linux-kernel-owner@vger.kernel.org 
> > > [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Lee Revell
> > > Sent: Thursday, January 12, 2006 4:18 PM
> > > To: linux-kernel
> > > Subject: Dual core Athlons and unsynced TSCs
> > > 
> > > It's been known for quite some time that the TSCs are not 
> > > synced between cores on Athlon X2 machines and this screws up 
> > > the kernel's timekeeping, as it still uses the TSC as the 
> > > default time source on these machines.
> > > 
> > > This problem still seems to be present in the latest kernels. 
> > >  What is the plan to fix it?  Is the fix simply to make the 
> > > kernel use the ACPI PM timer by default on Athlon X2?
> > 
> > 
> > Do we know if this also affects dual-core opterons?
> > 
> > The symptoms are that the clocks run at 2x the speed, correct?
> 
> No, worse.  The monotonic clock can go backwards.  The tscs of the CPUs
> are not in sync when one slows down due to idle.  So if you read from
> two different CPUs, you may get the second read have an earlier time
> than the first.  Breaks the rule of what a monotonic clock is.
> 

Steve,

I don't have hardware to test this, can you confirm that the only
workaround needed is to boot with "clock=pmtmr"?

Lee

RE: Dual core Athlons and unsynced TSCs

[Steven Rostedt]

On Fri, 13 Jan 2006, Lee Revell wrote:

> I don't have hardware to test this, can you confirm that the only
> workaround needed is to boot with "clock=pmtmr"?

For which kernel?

-- Steve

RE: Dual core Athlons and unsynced TSCs

[Lee Revell]

On Fri, 2006-01-13 at 12:52 -0500, Steven Rostedt wrote:
> On Fri, 13 Jan 2006, Lee Revell wrote:
> 
> > I don't have hardware to test this, can you confirm that the only
> > workaround needed is to boot with "clock=pmtmr"?
> 
> For which kernel?

I believe the problem exists on all kernels, it should not matter.

Lee

Re: Dual core Athlons and unsynced TSCs

[Lee Revell]

On Fri, 2006-01-13 at 10:06 -0800, thockin@hockin.org wrote:
> On Thu, Jan 12, 2006 at 05:17:39PM -0500, Lee Revell wrote:
> > It's been known for quite some time that the TSCs are not synced between
> > cores on Athlon X2 machines and this screws up the kernel's timekeeping,
> > as it still uses the TSC as the default time source on these machines.
> > 
> > This problem still seems to be present in the latest kernels.  What is
> > the plan to fix it?  Is the fix simply to make the kernel use the ACPI
> > PM timer by default on Athlon X2?
> 
> If your BIOS has an ACPI "HPET" table, then the kernel can use the HPET
> timer instead.  It doesn't solve the problem for apps or other kernel bits
> that use rdtsc directly, but there are other fixes for that floating
> around which will hopefully get consideration when they're mature.
> 

The apps can be converted to use clock_gettime(), but that does not help
if the kernel can't keep reasonable time on these machines. 

But if we just use "clock=pmtmr" by default on these machines the TSC is
not a problem right?

Lee

Re: Dual core Athlons and unsynced TSCs

[thockin]

On Thu, Jan 12, 2006 at 05:17:39PM -0500, Lee Revell wrote:
> It's been known for quite some time that the TSCs are not synced between
> cores on Athlon X2 machines and this screws up the kernel's timekeeping,
> as it still uses the TSC as the default time source on these machines.
> 
> This problem still seems to be present in the latest kernels.  What is
> the plan to fix it?  Is the fix simply to make the kernel use the ACPI
> PM timer by default on Athlon X2?

If your BIOS has an ACPI "HPET" table, then the kernel can use the HPET
timer instead.  It doesn't solve the problem for apps or other kernel bits
that use rdtsc directly, but there are other fixes for that floating
around which will hopefully get consideration when they're mature.

Re: Dual core Athlons and unsynced TSCs

[Lee Revell]

On Fri, 2006-01-13 at 10:16 -0800, thockin@hockin.org wrote:
> On Fri, Jan 13, 2006 at 12:58:36PM -0500, Lee Revell wrote:
> > > If your BIOS has an ACPI "HPET" table, then the kernel can use the HPET
> > > timer instead.  It doesn't solve the problem for apps or other kernel bits
> > > that use rdtsc directly, but there are other fixes for that floating
> > > around which will hopefully get consideration when they're mature.
> > 
> > The apps can be converted to use clock_gettime(), but that does not help
> > if the kernel can't keep reasonable time on these machines. 
> 
> Some apps/users need higher resolution and lower overhead that only rdtsc
> can offer currently.
> 

But obviously if the TSC gives wildly inaccurate results, it cannot be
used no matter how low the overhead.

> > But if we just use "clock=pmtmr" by default on these machines the TSC is
> > not a problem right?
> 
> I never tried it with pmtimer, we had HPET available.  Empirically TSC did
> not work (and we had a simple test case to show how bad it could get).
> HPET made that go away for users of gettimeofday().
> 
> We're exploring rdtsc-compatible solutions.
> 

Since timekeeping on these machines has always been completely broken,
shouldn't the default time source have been changed to the PM timer or
HPET as soon as the problem was known?

Lee

Re: Dual core Athlons and unsynced TSCs

[thockin]

On Fri, Jan 13, 2006 at 12:58:36PM -0500, Lee Revell wrote:
> > If your BIOS has an ACPI "HPET" table, then the kernel can use the HPET
> > timer instead.  It doesn't solve the problem for apps or other kernel bits
> > that use rdtsc directly, but there are other fixes for that floating
> > around which will hopefully get consideration when they're mature.
> 
> The apps can be converted to use clock_gettime(), but that does not help
> if the kernel can't keep reasonable time on these machines. 

Some apps/users need higher resolution and lower overhead that only rdtsc
can offer currently.

> But if we just use "clock=pmtmr" by default on these machines the TSC is
> not a problem right?

I never tried it with pmtimer, we had HPET available.  Empirically TSC did
not work (and we had a simple test case to show how bad it could get).
HPET made that go away for users of gettimeofday().

We're exploring rdtsc-compatible solutions.

RE: Dual core Athlons and unsynced TSCs

[Steven Rostedt]

On Fri, 13 Jan 2006, Lee Revell wrote:

> On Fri, 2006-01-13 at 12:52 -0500, Steven Rostedt wrote:
> > On Fri, 13 Jan 2006, Lee Revell wrote:
> >
> > > I don't have hardware to test this, can you confirm that the only
> > > workaround needed is to boot with "clock=pmtmr"?
> >
> > For which kernel?
>
> I believe the problem exists on all kernels, it should not matter.


Well, I just booted up (and am currently running) 2.6.15-rt4-sr2 and I
used "clock=pmtmr".  I haven't got any

read_tsc: ACK! TSC went backward! Unsynced TSCs?


If that's what you want to know?

But the latency traces are still screwed up.


-- Steve

RE: Dual core Athlons and unsynced TSCs

[Lee Revell]

On Fri, 2006-01-13 at 13:41 -0500, Steven Rostedt wrote:
> On Fri, 13 Jan 2006, Lee Revell wrote:
> 
> > On Fri, 2006-01-13 at 12:52 -0500, Steven Rostedt wrote:
> > > On Fri, 13 Jan 2006, Lee Revell wrote:
> > >
> > > > I don't have hardware to test this, can you confirm that the only
> > > > workaround needed is to boot with "clock=pmtmr"?
> > >
> > > For which kernel?
> >
> > I believe the problem exists on all kernels, it should not matter.
> 
> 
> Well, I just booted up (and am currently running) 2.6.15-rt4-sr2 and I
> used "clock=pmtmr".  I haven't got any
> 
> read_tsc: ACK! TSC went backward! Unsynced TSCs?
> 
> 
> If that's what you want to know?

I want to know if clock_gettime(CLOCK_MONOTONIC, *ts) is actually
guaranteed to be monotonic on these machines AKA do we break POSIX
compliance or not.

Lee

Re: Dual core Athlons and unsynced TSCs

[thockin]

On Fri, Jan 13, 2006 at 01:09:51PM -0500, Lee Revell wrote:
> > Some apps/users need higher resolution and lower overhead that only rdtsc
> > can offer currently.
> 
> But obviously if the TSC gives wildly inaccurate results, it cannot be
> used no matter how low the overhead.

unless we can re-sync the TSCs often enough that apps don't notice.

> > I never tried it with pmtimer, we had HPET available.  Empirically TSC did
> > not work (and we had a simple test case to show how bad it could get).
> > HPET made that go away for users of gettimeofday().
> > 
> > We're exploring rdtsc-compatible solutions.
> > 
> 
> Since timekeeping on these machines has always been completely broken,
> shouldn't the default time source have been changed to the PM timer or
> HPET as soon as the problem was known?

If you have HPET, the kernel will prefer that.  I don't know enough about
the PMtimer overhead or resolution to say for sure.

Re: Dual core Athlons and unsynced TSCs

[Sven-Thorsten Dietrich]

On Fri, 2006-01-13 at 10:55 -0800, thockin@hockin.org wrote:
> On Fri, Jan 13, 2006 at 01:09:51PM -0500, Lee Revell wrote:
> > > Some apps/users need higher resolution and lower overhead that only rdtsc
> > > can offer currently.
> > 
> > But obviously if the TSC gives wildly inaccurate results, it cannot be
> > used no matter how low the overhead.
> 
> unless we can re-sync the TSCs often enough that apps don't notice.
> 

You'd have to quantify that somehow, in terms of the max drift rate
(ppm), and the max resolution available (< tsc frequency).  

Either that, or track an offset, and use one TSC as truth, and update
the correction factor for the other TSCs as often as needed, maybe?

This is kind of analogous to the "drift" NTP calculates against a
free-running oscillator. 

So you'd be pushing that functionality deeper into the OS-core.

Dave Mills had that "hardpps" stuff in there for a while, it might be a
starting point.

Just some thoughts for now... 

Sven

> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
-- 
***********************************
Sven-Thorsten Dietrich
Real-Time Software Architect
MontaVista Software, Inc.
1237 East Arques Ave.
Sunnyvale, CA 94085

Phone: 408.992.4515
Fax: 408.328.9204

http://www.mvista.com
Platform To Innovate
*********************************** 

Re: Dual core Athlons and unsynced TSCs

[Lee Revell]

On Fri, 2006-01-13 at 10:56 -0800, Sven-Thorsten Dietrich wrote:
> On Fri, 2006-01-13 at 10:55 -0800, thockin@hockin.org wrote:
> > On Fri, Jan 13, 2006 at 01:09:51PM -0500, Lee Revell wrote:
> > > > Some apps/users need higher resolution and lower overhead that only rdtsc
> > > > can offer currently.
> > > 
> > > But obviously if the TSC gives wildly inaccurate results, it cannot be
> > > used no matter how low the overhead.
> > 
> > unless we can re-sync the TSCs often enough that apps don't notice.
> > 
> 
> You'd have to quantify that somehow, in terms of the max drift rate
> (ppm), and the max resolution available (< tsc frequency).  
> 
> Either that, or track an offset, and use one TSC as truth, and update
> the correction factor for the other TSCs as often as needed, maybe?
> 
> This is kind of analogous to the "drift" NTP calculates against a
> free-running oscillator. 
> 
> So you'd be pushing that functionality deeper into the OS-core.
> 
> Dave Mills had that "hardpps" stuff in there for a while, it might be a
> starting point.
> 
> Just some thoughts for now... 
> 

It kind of makes you wonder what in the heck AMD were thinking, whether
they realized that this design decision would cause so many problems at
the OS level (it's broken at least Linux and Solaris).  Maybe Windows
keeps time in a way that was unaffected by this?

Lee

Re: Dual core Athlons and unsynced TSCs

[Sven-Thorsten Dietrich]

On Fri, 2006-01-13 at 14:00 -0500, Lee Revell wrote:
> On Fri, 2006-01-13 at 10:56 -0800, Sven-Thorsten Dietrich wrote:
> > On Fri, 2006-01-13 at 10:55 -0800, thockin@hockin.org wrote:
> > > On Fri, Jan 13, 2006 at 01:09:51PM -0500, Lee Revell wrote:
> > > > > Some apps/users need higher resolution and lower overhead that only rdtsc
> > > > > can offer currently.
> > > > 
> > > > But obviously if the TSC gives wildly inaccurate results, it cannot be
> > > > used no matter how low the overhead.
> > > 
> > > unless we can re-sync the TSCs often enough that apps don't notice.
> > > 
> > 
> > You'd have to quantify that somehow, in terms of the max drift rate
> > (ppm), and the max resolution available (< tsc frequency).  
> > 
> > Either that, or track an offset, and use one TSC as truth, and update
> > the correction factor for the other TSCs as often as needed, maybe?
> > 
> > This is kind of analogous to the "drift" NTP calculates against a
> > free-running oscillator. 
> > 
> > So you'd be pushing that functionality deeper into the OS-core.
> > 
> > Dave Mills had that "hardpps" stuff in there for a while, it might be a
> > starting point.
> > 
> > Just some thoughts for now... 
> > 
> 
> It kind of makes you wonder what in the heck AMD were thinking, whether
> they realized that this design decision would cause so many problems at
> the OS level (it's broken at least Linux and Solaris).  Maybe Windows
> keeps time in a way that was unaffected by this?
> 

Last time I looked at Windows, we had to create an RT task to
interpolate between jiffies, b/c Windows was only keeping time at the
10 / 16 (smp) ms resolution. (to get NTP to run)

That was 6 years ago. 

I haven't looked back since, and may I turn to salt or stone if I ever
do.

> Lee
> 
-- 
***********************************
Sven-Thorsten Dietrich
Real-Time Software Architect
MontaVista Software, Inc.
1237 East Arques Ave.
Sunnyvale, CA 94085

Phone: 408.992.4515
Fax: 408.328.9204

http://www.mvista.com
Platform To Innovate
*********************************** 

Re: Dual core Athlons and unsynced TSCs

[Xavier Bestel]

Le vendredi 13 janvier 2006 à 10:56 -0800, Sven-Thorsten Dietrich a
écrit :
> On Fri, 2006-01-13 at 10:55 -0800, thockin@hockin.org wrote:
> > On Fri, Jan 13, 2006 at 01:09:51PM -0500, Lee Revell wrote:
> > > > Some apps/users need higher resolution and lower overhead that only rdtsc
> > > > can offer currently.
> > > 
> > > But obviously if the TSC gives wildly inaccurate results, it cannot be
> > > used no matter how low the overhead.
> > 
> > unless we can re-sync the TSCs often enough that apps don't notice.
> > 
> 
> You'd have to quantify that somehow, in terms of the max drift rate
> (ppm), and the max resolution available (< tsc frequency).  
> 
> Either that, or track an offset, and use one TSC as truth, and update
> the correction factor for the other TSCs as often as needed, maybe?

As often as needed being each time a thread changes CPU ?

Re: Dual core Athlons and unsynced TSCs

[thockin]

On Fri, Jan 13, 2006 at 10:56:13AM -0800, Sven-Thorsten Dietrich wrote:
> On Fri, 2006-01-13 at 10:55 -0800, thockin@hockin.org wrote:
> > unless we can re-sync the TSCs often enough that apps don't notice.
> 
> You'd have to quantify that somehow, in terms of the max drift rate
> (ppm), and the max resolution available (< tsc frequency).  
> 
> Either that, or track an offset, and use one TSC as truth, and update
> the correction factor for the other TSCs as often as needed, maybe?
> 
> This is kind of analogous to the "drift" NTP calculates against a
> free-running oscillator. 
> 
> So you'd be pushing that functionality deeper into the OS-core.
> 
> Dave Mills had that "hardpps" stuff in there for a while, it might be a
> starting point.
> 
> Just some thoughts for now... 

There's some chatter here about a solution involving a lazy sync of TSCs
to the HPET (or other) whenever an app calls rdtsc after a potentially
unsyncing event.

For example, 'hlt' will initiate C1 which may cause clock ramping (and TSC
skew).  We can trap rdtsc after a hlt and re-sync the TSCs to some truly
monotonic source, like HPET.

I don't have all the details, some problems remain, and the work is not
quite done yet, but it looks promising.

Even if we eventually get synced TSCs, it's too little too late.
basically, anything in-kernel that uses rdtsc is bound to break, and any
app that uses rdtsc had better be using CPU affinity.

Tim

RE: Dual core Athlons and unsynced TSCs

[Steven Rostedt]

On Fri, 2006-01-13 at 13:55 -0500, Lee Revell wrote:

> > 
> > If that's what you want to know?
> 
> I want to know if clock_gettime(CLOCK_MONOTONIC, *ts) is actually
> guaranteed to be monotonic on these machines AKA do we break POSIX
> compliance or not.

Nope it doesn't work :-(

I ran this test:
 http://www.kihontech.com/tests/rt/monotonic.c

And got this:

$ ./monotonic
main program pid=8477
hello from thread 0!
hello from thread 1!
hello from thread 2!
hello from thread 3!
hello from thread 4!
Failed! prev: 6279.925610302   current: 6279.874615349

$ cat /proc/cmdline
root=/dev/md0 ro console=ttyS0,115200 console=tty0 nmi_watchdog=2 lapic earlyprintk=ttyS0,115200 clock=pmtmr

$ uname -r
2.6.15-rt4-sr2

(I also added Ingo, Thomas, and John in the CC)

I'll reboot to vanilla 2.6.15 and see if that is broken too (just to
make sure)

-- Steve

RE: Dual core Athlons and unsynced TSCs

[Steven Rostedt]

On Fri, 2006-01-13 at 15:09 -0500, Steven Rostedt wrote:

> Nope it doesn't work :-(
> 
> I ran this test:
>  http://www.kihontech.com/tests/rt/monotonic.c
> 
[...]
> 
> I'll reboot to vanilla 2.6.15 and see if that is broken too (just to
> make sure)
> 

Failed on 2.6.15 too:

$ ./monotonic
main program pid=6724
hello from thread 0!
hello from thread 4!
Failed! prev: 316.952599248   current: 316.951899248
hello from thread 3!
hello from thread 2!
hello from thread 1!

$ uname -r
2.6.15
$ cat /proc/cmdline
root=/dev/md0 ro console=tty0 nmi_watchdog=2 lapic clock=pmtmr

:-(

-- Steve

RE: Dual core Athlons and unsynced TSCs

[Lee Revell]

On Fri, 2006-01-13 at 15:26 -0500, Steven Rostedt wrote:
> On Fri, 2006-01-13 at 15:09 -0500, Steven Rostedt wrote:
> 
> > Nope it doesn't work :-(
> > 
> > I ran this test:
> >  http://www.kihontech.com/tests/rt/monotonic.c
> > 
> [...]
> > 
> > I'll reboot to vanilla 2.6.15 and see if that is broken too (just to
> > make sure)
> > 
> 
> Failed on 2.6.15 too:

That's what I suspected based on reports from several JACK users (JACK
used to use rdtsc directly, it now uses clock_gettime).

Hmm, it looks like the kernel uses the TSC left and right, even if
clock=pmtmr is used.  There's even an rdtscll in pmtimer_mark_offset().

Wow, what a mess.  Thanks, AMD...

Lee

RE: Dual core Athlons and unsynced TSCs

[Lee Revell]

On Fri, 2006-01-13 at 15:26 -0500, Steven Rostedt wrote:
> On Fri, 2006-01-13 at 15:09 -0500, Steven Rostedt wrote:
> 
> > Nope it doesn't work :-(
> > 
> > I ran this test:
> >  http://www.kihontech.com/tests/rt/monotonic.c
> > 
> [...]
> > 
> > I'll reboot to vanilla 2.6.15 and see if that is broken too (just to
> > make sure)
> > 
> 
> Failed on 2.6.15 too:
> 

Ugh.  In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC
unconditionally.

Lee

RE: Dual core Athlons and unsynced TSCs

[Thomas Gleixner]

On Fri, 2006-01-13 at 15:43 -0500, Lee Revell wrote:

> Ugh.  In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC
> unconditionally.

Can you try 

http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch

please ?

	tglx

RE: Dual core Athlons and unsynced TSCs

[Lee Revell]

On Fri, 2006-01-13 at 21:46 +0100, Thomas Gleixner wrote:
> On Fri, 2006-01-13 at 15:43 -0500, Lee Revell wrote:
> 
> > Ugh.  In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC
> > unconditionally.
> 
> Can you try 
> 
> http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch
> 
> please ?

I can try it (well I can pass it on to someone who has the hardware),
but that's a huge patch, not likely to be mergeable in the 2.6.16
timeframe.  Surely there has to be a way this can be fixed for 2.6.16?

Lee

RE: Dual core Athlons and unsynced TSCs

[Thomas Gleixner]

On Fri, 2006-01-13 at 15:48 -0500, Lee Revell wrote:
> On Fri, 2006-01-13 at 21:46 +0100, Thomas Gleixner wrote:
> > On Fri, 2006-01-13 at 15:43 -0500, Lee Revell wrote:
> > 
> > > Ugh.  In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC
> > > unconditionally.
> > 
> > Can you try 
> > 
> > http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch
> > 
> > please ?
> 
> I can try it (well I can pass it on to someone who has the hardware),
> but that's a huge patch, not likely to be mergeable in the 2.6.16
> timeframe.  Surely there has to be a way this can be fixed for 2.6.16?

Parts of the patch are in mainline (-gitX), but those alone will not fix
the problem AFAICS

	tglx

RE: Dual core Athlons and unsynced TSCs

[Steven Rostedt]

On Fri, 13 Jan 2006, Lee Revell wrote:

> On Fri, 2006-01-13 at 21:46 +0100, Thomas Gleixner wrote:
> > On Fri, 2006-01-13 at 15:43 -0500, Lee Revell wrote:
> >
> > > Ugh.  In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC
> > > unconditionally.
> >
> > Can you try
> >
> > http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch
> >
> > please ?
>
> I can try it (well I can pass it on to someone who has the hardware),
> but that's a huge patch, not likely to be mergeable in the 2.6.16
> timeframe.  Surely there has to be a way this can be fixed for 2.6.16?

I'll try it.

If it works, at least it might give us a clue to what to do.

-- Steve

Re: Dual core Athlons and unsynced TSCs

[Steven Rostedt]

On Fri, 2006-01-13 at 11:32 -0800, thockin@hockin.org wrote:
> On Fri, Jan 13, 2006 at 10:56:13AM -0800, Sven-Thorsten Dietrich wrote:
> > On Fri, 2006-01-13 at 10:55 -0800, thockin@hockin.org wrote:
> > > unless we can re-sync the TSCs often enough that apps don't notice.
> > 
> > You'd have to quantify that somehow, in terms of the max drift rate
> > (ppm), and the max resolution available (< tsc frequency).  
> > 
> > Either that, or track an offset, and use one TSC as truth, and update
> > the correction factor for the other TSCs as often as needed, maybe?
> > 
> > This is kind of analogous to the "drift" NTP calculates against a
> > free-running oscillator. 
> > 
> > So you'd be pushing that functionality deeper into the OS-core.
> > 
> > Dave Mills had that "hardpps" stuff in there for a while, it might be a
> > starting point.
> > 
> > Just some thoughts for now... 
> 
> There's some chatter here about a solution involving a lazy sync of TSCs
> to the HPET (or other) whenever an app calls rdtsc after a potentially
> unsyncing event.
> 
> For example, 'hlt' will initiate C1 which may cause clock ramping (and TSC
> skew).  We can trap rdtsc after a hlt and re-sync the TSCs to some truly
> monotonic source, like HPET.
> 
> I don't have all the details, some problems remain, and the work is not
> quite done yet, but it looks promising.
> 
> Even if we eventually get synced TSCs, it's too little too late.
> basically, anything in-kernel that uses rdtsc is bound to break, and any
> app that uses rdtsc had better be using CPU affinity.

I know this might be somewhat of an overhead, but what about resyncing
on wakeup of the idle function?  Isn't that where they fall apart?

-- Steve

RE: Dual core Athlons and unsynced TSCs

[Steven Rostedt]

On Fri, 2006-01-13 at 21:46 +0100, Thomas Gleixner wrote:
> On Fri, 2006-01-13 at 15:43 -0500, Lee Revell wrote:
> 
> > Ugh.  In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC
> > unconditionally.
> 
> Can you try 
> 
> http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch
> 
> please ?
> 
> 	tglx
> 
> 

Hmm, it doesn't compile :(

-- Steve

rostedt@gandalf:~/work/kernels/linux-2.6.15-hrt2$ make
amdmake
  CHK     include/linux/version.h
  CC      arch/x86_64/kernel/asm-offsets.s
In file included from include/linux/timex.h:61,
                 from include/linux/sched.h:11,
                 from arch/x86_64/kernel/asm-offsets.c:7:
include/asm/timex.h: In function 'get_cycles_sync':
include/asm/timex.h:36: warning: implicit declaration of function 'alternative_io'
include/asm/timex.h:37: error: called object '"=a"' is not a function
include/asm/timex.h:37: error: called object '"0"' is not a function
include/asm/timex.h:37: error: syntax error before ':' token
make[1]: *** [arch/x86_64/kernel/asm-offsets.s] Error 1
make: *** [prepare0] Error 2

RE: Dual core Athlons and unsynced TSCs

[Thomas Gleixner]

On Fri, 2006-01-13 at 16:05 -0500, Steven Rostedt wrote:
> On Fri, 2006-01-13 at 21:46 +0100, Thomas Gleixner wrote:
> > On Fri, 2006-01-13 at 15:43 -0500, Lee Revell wrote:
> > 
> > > Ugh.  In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC
> > > unconditionally.
> > 
> > Can you try 
> > 
> > http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch
> > 
> > please ?
> > 
> > 	tglx
> > 
> > 
> 
> Hmm, it doesn't compile :(

Grmbl, I check this tomorrow. Getting late here

	tglx

RE: Dual core Athlons and unsynced TSCs

[Steven Rostedt]

On Fri, 13 Jan 2006, Thomas Gleixner wrote:

> On Fri, 2006-01-13 at 16:05 -0500, Steven Rostedt wrote:
> > On Fri, 2006-01-13 at 21:46 +0100, Thomas Gleixner wrote:
> > > On Fri, 2006-01-13 at 15:43 -0500, Lee Revell wrote:
> > >
> > > > Ugh.  In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC
> > > > unconditionally.
> > >
> > > Can you try
> > >
> > > http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch
> > >
> > > please ?
> > >
> > > 	tglx
> > >
> > >
> >
> > Hmm, it doesn't compile :(
>
> Grmbl, I check this tomorrow. Getting late here
>

OK, I'll probably wont work on it till Monday then (wifes rules ;)

-- Steve

Re: Dual core Athlons and unsynced TSCs

[David Lang]

On Fri, 13 Jan 2006, Lee Revell wrote:

> On Fri, 2006-01-13 at 10:56 -0800, Sven-Thorsten Dietrich wrote:
>> On Fri, 2006-01-13 at 10:55 -0800, thockin@hockin.org wrote:
>>> On Fri, Jan 13, 2006 at 01:09:51PM -0500, Lee Revell wrote:
>>>>> Some apps/users need higher resolution and lower overhead that only rdtsc
>>>>> can offer currently.
>>>>
>>>> But obviously if the TSC gives wildly inaccurate results, it cannot be
>>>> used no matter how low the overhead.
>>>
>>> unless we can re-sync the TSCs often enough that apps don't notice.
>>>
>>
>> You'd have to quantify that somehow, in terms of the max drift rate
>> (ppm), and the max resolution available (< tsc frequency).
>>
>> Either that, or track an offset, and use one TSC as truth, and update
>> the correction factor for the other TSCs as often as needed, maybe?
>>
>> This is kind of analogous to the "drift" NTP calculates against a
>> free-running oscillator.
>>
>> So you'd be pushing that functionality deeper into the OS-core.
>>
>> Dave Mills had that "hardpps" stuff in there for a while, it might be a
>> starting point.
>>
>> Just some thoughts for now...
>>
>
> It kind of makes you wonder what in the heck AMD were thinking, whether
> they realized that this design decision would cause so many problems at
> the OS level (it's broken at least Linux and Solaris).  Maybe Windows
> keeps time in a way that was unaffected by this?

Lee, the last time I saw this discussion I thought it was identified that 
the multiple cores (or IIRC the multiple chips in a SMP motherboard) would 
only get out of sync when power management calls were made (hlt or 
switching the c-state). IIRC the workaround that was posted then was to 
just disable these in the kernel build.

if this is the case then I could easily see the hardware engineers 
thinking that if the software did a sleep thing then it was up to the 
software to re-sync the TSC clocks when the wakeup takes place.

David Lang

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare

Re: Dual core Athlons and unsynced TSCs

[thockin]

On Fri, Jan 13, 2006 at 01:18:35PM -0800, David Lang wrote:
> Lee, the last time I saw this discussion I thought it was identified that 
> the multiple cores (or IIRC the multiple chips in a SMP motherboard) would 
> only get out of sync when power management calls were made (hlt or 
> switching the c-state). IIRC the workaround that was posted then was to 
> just disable these in the kernel build.

not using 'hlt' when idling means that you spend 10s of Watts more power
on mostly idle systems.

Re: Dual core Athlons and unsynced TSCs

[David Lang]

On Fri, 13 Jan 2006 thockin@hockin.org wrote:

> On Fri, Jan 13, 2006 at 01:18:35PM -0800, David Lang wrote:
>> Lee, the last time I saw this discussion I thought it was identified that
>> the multiple cores (or IIRC the multiple chips in a SMP motherboard) would
>> only get out of sync when power management calls were made (hlt or
>> switching the c-state). IIRC the workaround that was posted then was to
>> just disable these in the kernel build.
>
> not using 'hlt' when idling means that you spend 10s of Watts more power
> on mostly idle systems.

true, but for people who need better time accruacy then the other 
workaround this may be very acceptable.

David Lang

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare

Re: Dual core Athlons and unsynced TSCs

[Sven-Thorsten Dietrich]

On Fri, 2006-01-13 at 14:05 -0800, David Lang wrote:
> On Fri, 13 Jan 2006 thockin@hockin.org wrote:
> 
> > On Fri, Jan 13, 2006 at 01:18:35PM -0800, David Lang wrote:
> >> Lee, the last time I saw this discussion I thought it was identified that
> >> the multiple cores (or IIRC the multiple chips in a SMP motherboard) would
> >> only get out of sync when power management calls were made (hlt or
> >> switching the c-state). IIRC the workaround that was posted then was to
> >> just disable these in the kernel build.
> >
> > not using 'hlt' when idling means that you spend 10s of Watts more power
> > on mostly idle systems.
> 
> true, but for people who need better time accruacy then the other 
> workaround this may be very acceptable.
> 

1/4 KW / day for time synchronisation. 

The power company would love that.

> David Lang
> 
-- 
***********************************
Sven-Thorsten Dietrich
Real-Time Software Architect
MontaVista Software, Inc.
1237 East Arques Ave.
Sunnyvale, CA 94085

Phone: 408.992.4515
Fax: 408.328.9204

http://www.mvista.com
Platform To Innovate
*********************************** 

Re: Dual core Athlons and unsynced TSCs

[thockin]

On Fri, Jan 13, 2006 at 02:05:23PM -0800, David Lang wrote:
> On Fri, 13 Jan 2006 thockin@hockin.org wrote:
> 
> >On Fri, Jan 13, 2006 at 01:18:35PM -0800, David Lang wrote:
> >>Lee, the last time I saw this discussion I thought it was identified that
> >>the multiple cores (or IIRC the multiple chips in a SMP motherboard) would
> >>only get out of sync when power management calls were made (hlt or
> >>switching the c-state). IIRC the workaround that was posted then was to
> >>just disable these in the kernel build.
> >
> >not using 'hlt' when idling means that you spend 10s of Watts more power
> >on mostly idle systems.
> 
> true, but for people who need better time accruacy then the other 
> workaround this may be very acceptable.

sure, if power doesn't matter use idle=poll => problem solved.

Re: Dual core Athlons and unsynced TSCs

[David Lang]

On Fri, 13 Jan 2006, Sven-Thorsten Dietrich wrote:

> On Fri, 2006-01-13 at 14:05 -0800, David Lang wrote:
>> On Fri, 13 Jan 2006 thockin@hockin.org wrote:
>>
>>> On Fri, Jan 13, 2006 at 01:18:35PM -0800, David Lang wrote:
>>>> Lee, the last time I saw this discussion I thought it was identified that
>>>> the multiple cores (or IIRC the multiple chips in a SMP motherboard) would
>>>> only get out of sync when power management calls were made (hlt or
>>>> switching the c-state). IIRC the workaround that was posted then was to
>>>> just disable these in the kernel build.
>>>
>>> not using 'hlt' when idling means that you spend 10s of Watts more power
>>> on mostly idle systems.
>>
>> true, but for people who need better time accruacy then the other
>> workaround this may be very acceptable.
>>
>
> 1/4 KW / day for time synchronisation.
>
> The power company would love that.

more precisely 1/4 KW Hour / day

$0.01 - $0.02/day (I had to lookup the current rates)

they probably won't notice.

David Lang

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare

RE: Dual core Athlons and unsynced TSCs

[john stultz]

[-- Attachment #1: Type: text/plain, Size: 1301 bytes --]

On Fri, 2006-01-13 at 15:09 -0500, Steven Rostedt wrote:
> On Fri, 2006-01-13 at 13:55 -0500, Lee Revell wrote:
> 
> > > 
> > > If that's what you want to know?
> > 
> > I want to know if clock_gettime(CLOCK_MONOTONIC, *ts) is actually
> > guaranteed to be monotonic on these machines AKA do we break POSIX
> > compliance or not.
> 
> Nope it doesn't work :-(
> 
> I ran this test:
>  http://www.kihontech.com/tests/rt/monotonic.c
> 
> And got this:
> 
> $ ./monotonic
> main program pid=8477
> hello from thread 0!
> hello from thread 1!
> hello from thread 2!
> hello from thread 3!
> hello from thread 4!
> Failed! prev: 6279.925610302   current: 6279.874615349

Not sure, but I think your test is broken.

While it serializes the checking and exchange of current and last, it
doesn't serialize the actual calling of clock_gettime(). 

Thus you can get something like:

last=0
	A:				B:
1:	now = clock_gettime()
2:					now = clock_gettime()
3:					atomic {
						test(now, last) [2>0]
						last = now
					}
4:	atomic {
		test(now, last) [1>2]
		FAIL!
	}


Attached is a similar testcase that I've been using myself that avoids
this issue (although I just converted it from gettimeofday to
clock_gettime, so there may still be an issue). Let me know if you have
any comments on it.

thanks
-john


[-- Attachment #2: threadtest.c --]
[-- Type: text/x-csrc, Size: 2890 bytes --]

/* threadtest.c 
 *		by: john stultz (johnstul@us.ibm.com)
 *		(C) Copyright IBM 2004, 2005, 2006
 *		Licensed under the GPL
 */
#include <stdio.h>
#include <sys/time.h>
#include <pthread.h>


/* serializes shared list access */
pthread_mutex_t list_lock = PTHREAD_MUTEX_INITIALIZER;
/* serializes console output */
pthread_mutex_t print_lock = PTHREAD_MUTEX_INITIALIZER;


#define MAX_THREADS 128
#define LISTSIZE 128


int done = 0;

struct timespec global_list[LISTSIZE];
int listcount = 0;


void checklist(struct timespec* list, int size)
{
	int i,j;
	struct timespec *a,*b;

	static int dbgct = 0;
	if(!(dbgct++%7000)){
		printf(".");
		fflush(0);
	}

	/* scan the list */
	for(i=0; i < size-1; i++){
		a = &list[i];
		b = &list[i+1];
		
		/* look for any time inconsistencies */
		if((b->tv_sec <= a->tv_sec)&&
			(b->tv_nsec < a->tv_nsec)){

			/* flag other threads */
			done = 1;

			/*serialize printing to avoid junky output*/
			pthread_mutex_lock(&print_lock);

			/* dump the list */
			printf("\n");
			for(j=0; j< size; j++){
				if(j == i)
					printf("---------------\n");
				printf("%lu:%lu\n", list[j].tv_sec,list[j].tv_nsec);
				if(j == i+1)
					printf("---------------\n");
			}
			printf("TEST FAILED\n");

			pthread_mutex_unlock(&print_lock);


		}
	}
}

/* The Shared Thread shares a global list
 * that each thread fills while holding the lock.
 * This stresses clock syncronization across cpus. 
 */
void* shared_thread(void* arg)
{
	while(!done){
		/* protect the list */
		pthread_mutex_lock(&list_lock);
		
		/* see if we're ready to check the list */
		if(listcount >= LISTSIZE){
			checklist(global_list, LISTSIZE);
			listcount = 0;
		}
		clock_gettime(CLOCK_MONOTONIC, &global_list[listcount++]);
				
		pthread_mutex_unlock(&list_lock);
	}
}


/* Each independent thread fills in its own
 * list. This stresses clock_gettime() lock contention. 
 */
void* independent_thread(void* arg)
{
	struct timespec my_list[LISTSIZE];
	int count;

	while(!done){
		/* fill the list */
		for(count=0; count < LISTSIZE; count++)
			clock_gettime(CLOCK_MONOTONIC, &my_list[count]);
		checklist(my_list, LISTSIZE);
	}
}


int main(int argc, void** argv)
{
	int thread_count = 1, i;
	pthread_t pth[MAX_THREADS];
	void* ret;
	void* (*thread)(void*) = shared_thread;
	
	/* pull the thread count */
	if(argc > 1)
		thread_count = strtol(argv[1],0,0);
	if(thread_count > MAX_THREADS)
		thread_count = MAX_THREADS;

	/* check if we're running independent threads */	
	if(argc == 3 && !strncmp(argv[2],"indie", 5)){
		thread = independent_thread;
		printf("using independent threads\n");
	}
	
	
	system("date");
	printf("spawning %i threads\n", thread_count,0,0);

	/* spawn */
	for(i=0; i < thread_count; i++)
		pthread_create(&pth[i], 0, thread, 0);
	
	/* wait */
	for(i=0; i< thread_count; i++)
		pthread_join(pth[i],&ret);

	system("date");	
	
	/* die */
	return 0;
}

Re: Dual core Athlons and unsynced TSCs

[Andreas Steinmetz]

David Lang wrote:
> On Fri, 13 Jan 2006, Sven-Thorsten Dietrich wrote:
> 
>> On Fri, 2006-01-13 at 14:05 -0800, David Lang wrote:
>>
>>> On Fri, 13 Jan 2006 thockin@hockin.org wrote:
>>>
>>>> On Fri, Jan 13, 2006 at 01:18:35PM -0800, David Lang wrote:
>>>>
>>>>> Lee, the last time I saw this discussion I thought it was
>>>>> identified that
>>>>> the multiple cores (or IIRC the multiple chips in a SMP
>>>>> motherboard) would
>>>>> only get out of sync when power management calls were made (hlt or
>>>>> switching the c-state). IIRC the workaround that was posted then
>>>>> was to
>>>>> just disable these in the kernel build.
>>>>
>>>>
>>>> not using 'hlt' when idling means that you spend 10s of Watts more
>>>> power
>>>> on mostly idle systems.
>>>
>>>
>>> true, but for people who need better time accruacy then the other
>>> workaround this may be very acceptable.
>>>
>>
>> 1/4 KW / day for time synchronisation.
>>
>> The power company would love that.
> 
> 
> more precisely 1/4 KW Hour / day
> 
> $0.01 - $0.02/day (I had to lookup the current rates)
> 
> they probably won't notice.
> 

Well, wait until there's AMD based dual core x86_64 laptops out there
(this email being written on a single core x86_64 one). I can already
see the faces of the unhappy future owners being told "use idle=poll"
when on battery and anyway going deaf by fan noise.

(/me ducks and runs)
-- 
Andreas Steinmetz                       SPAMmers use robotrap@domdv.de

Re: Dual core Athlons and unsynced TSCs

[David Lang]

On Sat, 14 Jan 2006, Andreas Steinmetz wrote:

> David Lang wrote:
>> On Fri, 13 Jan 2006, Sven-Thorsten Dietrich wrote:
>>
>>> On Fri, 2006-01-13 at 14:05 -0800, David Lang wrote:
>>>
>>>> On Fri, 13 Jan 2006 thockin@hockin.org wrote:
>>>>
>>>>> On Fri, Jan 13, 2006 at 01:18:35PM -0800, David Lang wrote:
>>>>>
>>>>>> Lee, the last time I saw this discussion I thought it was
>>>>>> identified that
>>>>>> the multiple cores (or IIRC the multiple chips in a SMP
>>>>>> motherboard) would
>>>>>> only get out of sync when power management calls were made (hlt or
>>>>>> switching the c-state). IIRC the workaround that was posted then
>>>>>> was to
>>>>>> just disable these in the kernel build.
>>>>>
>>>>>
>>>>> not using 'hlt' when idling means that you spend 10s of Watts more
>>>>> power
>>>>> on mostly idle systems.
>>>>
>>>>
>>>> true, but for people who need better time accruacy then the other
>>>> workaround this may be very acceptable.
>>>>
>>>
>>> 1/4 KW / day for time synchronisation.
>>>
>>> The power company would love that.
>>
>>
>> more precisely 1/4 KW Hour / day
>>
>> $0.01 - $0.02/day (I had to lookup the current rates)
>>
>> they probably won't notice.
>>
>
> Well, wait until there's AMD based dual core x86_64 laptops out there
> (this email being written on a single core x86_64 one). I can already
> see the faces of the unhappy future owners being told "use idle=poll"
> when on battery and anyway going deaf by fan noise.
>
> (/me ducks and runs)

I'm not saying it's the right answer, but it's one of two workarounds 
currently available.

idle=poll causes increased power useage

timer source change (mentioned earlier in this thread) limits timer 
precision

neither of these are fixes, but by understanding the different costs 
people can choose the work around they want to use while waiting for a 
better fix.

David Lang

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare

RE: Dual core Athlons and unsynced TSCs

[Steven Rostedt]

On Fri, 2006-01-13 at 16:23 -0800, john stultz wrote:

> > Failed! prev: 6279.925610302   current: 6279.874615349
> 
> Not sure, but I think your test is broken.

Yep and it is now fixed:

http://www.kihontech.com/tests/rt/monotonic.c


> 
> Attached is a similar testcase that I've been using myself that avoids
> this issue (although I just converted it from gettimeofday to
> clock_gettime, so there may still be an issue). Let me know if you have
> any comments on it.

Thanks, I'll add that to my list of tests too.

Oh and 2.6.15 passed as well (with clock=pmtmr)

-- Steve

Re: Dual core Athlons and unsynced TSCs

[thockin]

On Sat, Jan 14, 2006 at 01:41:43AM +0100, Andreas Steinmetz wrote:
> > more precisely 1/4 KW Hour / day
> > 
> > $0.01 - $0.02/day (I had to lookup the current rates)
> > 
> > they probably won't notice.
> 
> Well, wait until there's AMD based dual core x86_64 laptops out there
> (this email being written on a single core x86_64 one). I can already
> see the faces of the unhappy future owners being told "use idle=poll"
> when on battery and anyway going deaf by fan noise.

Yeah, and then multiply that cost by a few tens of thousands of servers.

idle=poll is a hack at best.

RE: Dual core Athlons and unsynced TSCs

[Lee Revell]

On Fri, 2006-01-13 at 20:10 -0500, Steven Rostedt wrote:
> 
> Thanks, I'll add that to my list of tests too.
> 
> Oh and 2.6.15 passed as well (with clock=pmtmr) 

It really seems like it would fail if you gave it enough time due to the
rdtsc in monotonic_clock()...

Lee

Re: Dual core Athlons and unsynced TSCs

[Andreas Steinmetz]

David Lang wrote:
[snip]
> I'm not saying it's the right answer, but it's one of two workarounds
> currently available.
> 
> idle=poll causes increased power useage
> 
> timer source change (mentioned earlier in this thread) limits timer
> precision
> 
> neither of these are fixes, but by understanding the different costs
> people can choose the work around they want to use while waiting for a
> better fix.

The problem being that some of us use their laptops for audio work too.
And then high battery usage, noisy fans or lack of high res timers will
be really bad.

Simple example:
I do final mastering work using my laptop and Ardour/Jack/JAMin out of
house in a place with a good stage audio system (Mackie mixer, 2KW
Dynacord Amp/Syrincs S3) and a sufficient listening space to get a
proper bass mix. I run on battery in this case to avoid any kind of
audio interference (ground loops, etc). Now thinking of a dual core
laptop...
-- 
Andreas Steinmetz                       SPAMmers use robotrap@domdv.de

RE: Dual core Athlons and unsynced TSCs

[john stultz]

On Fri, 2006-01-13 at 20:14 -0500, Lee Revell wrote:
> On Fri, 2006-01-13 at 20:10 -0500, Steven Rostedt wrote:
> > 
> > Thanks, I'll add that to my list of tests too.
> > 
> > Oh and 2.6.15 passed as well (with clock=pmtmr) 
> 
> It really seems like it would fail if you gave it enough time due to the
> rdtsc in monotonic_clock()...

Currently monotonic_clock()is only used by the hangcheck-timer, and is
not used by gettimeofday/clock_gettime (even w/ CLOCK_MONOTONIC). 

So there may still be an issue there w/ the hangcheck-timer(for x86-64,
on i386 the acpi pm timer can be used for monotonic_clock), but its
doesn't affect the time related userland interfaces.

thanks
-john

RE: Dual core Athlons and unsynced TSCs

[Lee Revell]

On Fri, 2006-01-13 at 17:23 -0800, john stultz wrote:
> On Fri, 2006-01-13 at 20:14 -0500, Lee Revell wrote:
> > On Fri, 2006-01-13 at 20:10 -0500, Steven Rostedt wrote:
> > > 
> > > Thanks, I'll add that to my list of tests too.
> > > 
> > > Oh and 2.6.15 passed as well (with clock=pmtmr) 
> > 
> > It really seems like it would fail if you gave it enough time due to the
> > rdtsc in monotonic_clock()...
> 
> Currently monotonic_clock()is only used by the hangcheck-timer, and is
> not used by gettimeofday/clock_gettime (even w/ CLOCK_MONOTONIC). 
> 
> So there may still be an issue there w/ the hangcheck-timer(for x86-64,
> on i386 the acpi pm timer can be used for monotonic_clock), but its
> doesn't affect the time related userland interfaces.

OK so the last question is how do we make sure the kernel uses the
clock=pmtmr behavior by default on those machines?

Lee

RE: Dual core Athlons and unsynced TSCs

[john stultz]

On Fri, 2006-01-13 at 20:27 -0500, Lee Revell wrote:
> On Fri, 2006-01-13 at 17:23 -0800, john stultz wrote:
> > On Fri, 2006-01-13 at 20:14 -0500, Lee Revell wrote:
> > > On Fri, 2006-01-13 at 20:10 -0500, Steven Rostedt wrote:
> > > > 
> > > > Thanks, I'll add that to my list of tests too.
> > > > 
> > > > Oh and 2.6.15 passed as well (with clock=pmtmr) 
> > > 
> > > It really seems like it would fail if you gave it enough time due to the
> > > rdtsc in monotonic_clock()...
> > 
> > Currently monotonic_clock()is only used by the hangcheck-timer, and is
> > not used by gettimeofday/clock_gettime (even w/ CLOCK_MONOTONIC). 
> > 
> > So there may still be an issue there w/ the hangcheck-timer(for x86-64,
> > on i386 the acpi pm timer can be used for monotonic_clock), but its
> > doesn't affect the time related userland interfaces.
> 
> OK so the last question is how do we make sure the kernel uses the
> clock=pmtmr behavior by default on those machines?

This is as I understand it:

With 2.6.15 on x86-64:
	If available, alternate timesources (HPET, ACPI PM) will be used if
available on AMD SMP systems. (clock= is i386 only)

With 2.6.15 on i386:
	If CONFIG_X86_PM_TIMER is enabled, and available it is the preferred
clocksource over the TSC.  Some distros have changed this priority
causing the TSC to be preferred. In these cases clock=pmtmr is needed.

How's that?
-john

RE: Dual core Athlons and unsynced TSCs

[Steven Rostedt]

On Fri, 13 Jan 2006, john stultz wrote:

>
> This is as I understand it:
>
> With 2.6.15 on x86-64:
> 	If available, alternate timesources (HPET, ACPI PM) will be used if
> available on AMD SMP systems. (clock= is i386 only)

Hmm, should I boot without the clock= to prove this?

-- Steve

>
> With 2.6.15 on i386:
> 	If CONFIG_X86_PM_TIMER is enabled, and available it is the preferred
> clocksource over the TSC.  Some distros have changed this priority
> causing the TSC to be preferred. In these cases clock=pmtmr is needed.
>
> How's that?
> -john
>
>
>

RE: Dual core Athlons and unsynced TSCs

[john stultz]

On Fri, 2006-01-13 at 20:43 -0500, Steven Rostedt wrote:
> On Fri, 13 Jan 2006, john stultz wrote:
> 
> >
> > This is as I understand it:
> >
> > With 2.6.15 on x86-64:
> > 	If available, alternate timesources (HPET, ACPI PM) will be used if
> > available on AMD SMP systems. (clock= is i386 only)
> 
> Hmm, should I boot without the clock= to prove this?

Feel free. That or grep the x86-64 time.c code.

Look for:
	time.c: Using PM based timekeeping.

To verify the timesource selection.

thanks
-john

Re: Dual core Athlons and unsynced TSCs

[thockin]

On Sat, Jan 14, 2006 at 02:21:23AM +0100, Andreas Steinmetz wrote:
> > neither of these are fixes, but by understanding the different costs
> > people can choose the work around they want to use while waiting for a
> > better fix.
> 
> The problem being that some of us use their laptops for audio work too.
> And then high battery usage, noisy fans or lack of high res timers will
> be really bad.
> 
> Simple example:
> I do final mastering work using my laptop and Ardour/Jack/JAMin out of
> house in a place with a good stage audio system (Mackie mixer, 2KW
> Dynacord Amp/Syrincs S3) and a sufficient listening space to get a
> proper bass mix. I run on battery in this case to avoid any kind of
> audio interference (ground loops, etc). Now thinking of a dual core
> laptop...

Fixes are coming, just need to sort out the fidlly bits.

RE: Dual core Athlons and unsynced TSCs

[Steven Rostedt]

On Fri, 13 Jan 2006, john stultz wrote:

> On Fri, 2006-01-13 at 20:43 -0500, Steven Rostedt wrote:
> > On Fri, 13 Jan 2006, john stultz wrote:
> >
> > >
> > > This is as I understand it:
> > >
> > > With 2.6.15 on x86-64:
> > > 	If available, alternate timesources (HPET, ACPI PM) will be used if
> > > available on AMD SMP systems. (clock= is i386 only)
> >
> > Hmm, should I boot without the clock= to prove this?
>
> Feel free. That or grep the x86-64 time.c code.
>
> Look for:
> 	time.c: Using PM based timekeeping.
>
> To verify the timesource selection.

I'm too busy trying to install linux on my powerbook to look at code at
the moment :-)

I just rebooted and ran the test on 2.6.15 without the clock= option and
it passed my test.  I'll wait a while and run it again to see if I get any
other drift too prove it more.

-- Steve

RE: Dual core Athlons and unsynced TSCs

[Steven Rostedt]

On Fri, 2006-01-13 at 17:50 -0800, john stultz wrote:
> On Fri, 2006-01-13 at 20:43 -0500, Steven Rostedt wrote:
> > On Fri, 13 Jan 2006, john stultz wrote:
> > 
> > >
> > > This is as I understand it:
> > >
> > > With 2.6.15 on x86-64:
> > > 	If available, alternate timesources (HPET, ACPI PM) will be used if
> > > available on AMD SMP systems. (clock= is i386 only)
> > 
> > Hmm, should I boot without the clock= to prove this?
> 
> Feel free. That or grep the x86-64 time.c code.
> 
> Look for:
> 	time.c: Using PM based timekeeping.
> 
> To verify the timesource selection.

So much for not looking at code! 

I uploaded yet another version of 
http://www.kihontech.com/tests/rt/monotonic.c
that now also tests the tsc too, to see if it goes backwards.  If it
does, then it prints (only once) that fact and continues testing the
gettime function.

So this proves that the tsc does go back and the gettime still is
working.


$ ./monotonic
main program pid=7137
hello from thread 0!
hello from thread 4!
last tsc is 4389431102233  this tsc is 4389429636684
hello from thread 1!
hello from thread 2!
hello from thread 3!

-- Steve

RE: Dual core Athlons and unsynced TSCs

[Christoph Lameter]

On Fri, 13 Jan 2006, Lee Revell wrote:

> I want to know if clock_gettime(CLOCK_MONOTONIC, *ts) is actually
> guaranteed to be monotonic on these machines AKA do we break POSIX
> compliance or not.

Yes you do. Other arches have had the same issues.... See IA64 code.

Re: Dual core Athlons and unsynced TSCs

[Zan Lynx]

[-- Attachment #1: Type: text/plain, Size: 1136 bytes --]

On Fri, 2006-01-13 at 17:04 -0800, David Lang wrote: 
> On Sat, 14 Jan 2006, Andreas Steinmetz wrote:
> 
> > David Lang wrote:
> > Well, wait until there's AMD based dual core x86_64 laptops out there
> > (this email being written on a single core x86_64 one). I can already
> > see the faces of the unhappy future owners being told "use idle=poll"
> > when on battery and anyway going deaf by fan noise.
> >
> > (/me ducks and runs)
> 
> I'm not saying it's the right answer, but it's one of two workarounds 
> currently available.
> 
> idle=poll causes increased power useage
> 
> timer source change (mentioned earlier in this thread) limits timer 
> precision
> 
> neither of these are fixes, but by understanding the different costs 
> people can choose the work around they want to use while waiting for a 
> better fix.

A laptop user could also bind a process to a single CPU, and use the
scaling min/max values to lock CPU speed to a single value.  The TSC may
still stop during HLT, but software must be handling that already.

Wouldn't that provide an accurate TSC?
-- 
Zan Lynx <zlynx@acm.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Dual core Athlons and unsynced TSCs

[thockin]

On Sun, Jan 15, 2006 at 01:52:44AM -0700, Zan Lynx wrote:
> A laptop user could also bind a process to a single CPU, and use the
> scaling min/max values to lock CPU speed to a single value.  The TSC may
> still stop during HLT, but software must be handling that already.
> 
> Wouldn't that provide an accurate TSC?

monotonic but not linear.  Also remember that the OS will use rdtsc here
and there, and you can't affine the OS :)

Re: Dual core Athlons and unsynced TSCs

[Lee Revell]

On Sun, 2006-01-15 at 08:25 -0800, thockin@hockin.org wrote:
> On Sun, Jan 15, 2006 at 01:52:44AM -0700, Zan Lynx wrote:
> > A laptop user could also bind a process to a single CPU, and use the
> > scaling min/max values to lock CPU speed to a single value.  The TSC may
> > still stop during HLT, but software must be handling that already.
> > 
> > Wouldn't that provide an accurate TSC?
> 
> monotonic but not linear.  Also remember that the OS will use rdtsc here
> and there, and you can't affine the OS :)
> 

So the options are either to fix the TSC handling on these systems (by
resyncing the TSCs when exiting from HLT), or eliminate the use of rdtsc
by the OS?

Lee

Re: Dual core Athlons and unsynced TSCs

[thockin]

On Sun, Jan 15, 2006 at 11:33:36AM -0500, Lee Revell wrote:
> On Sun, 2006-01-15 at 08:25 -0800, thockin@hockin.org wrote:
> > On Sun, Jan 15, 2006 at 01:52:44AM -0700, Zan Lynx wrote:
> > > A laptop user could also bind a process to a single CPU, and use the
> > > scaling min/max values to lock CPU speed to a single value.  The TSC may
> > > still stop during HLT, but software must be handling that already.
> > > 
> > > Wouldn't that provide an accurate TSC?
> > 
> > monotonic but not linear.  Also remember that the OS will use rdtsc here
> > and there, and you can't affine the OS :)
> > 
> 
> So the options are either to fix the TSC handling on these systems (by
> resyncing the TSCs when exiting from HLT), or eliminate the use of rdtsc
> by the OS?

Or both.  You can trap rdtsc users in userland, you have to manually audit
kernel users.

It should be abolished or properly wrapped in kernel code, as soon as a
resync path is available.

Re: Dual core Athlons and unsynced TSCs

[Lee Revell]

On Sun, 2006-01-15 at 10:21 -0800, thockin@hockin.org wrote:
> On Sun, Jan 15, 2006 at 11:33:36AM -0500, Lee Revell wrote:
> > On Sun, 2006-01-15 at 08:25 -0800, thockin@hockin.org wrote:
> > > On Sun, Jan 15, 2006 at 01:52:44AM -0700, Zan Lynx wrote:
> > > > A laptop user could also bind a process to a single CPU, and use the
> > > > scaling min/max values to lock CPU speed to a single value.  The TSC may
> > > > still stop during HLT, but software must be handling that already.
> > > > 
> > > > Wouldn't that provide an accurate TSC?
> > > 
> > > monotonic but not linear.  Also remember that the OS will use rdtsc here
> > > and there, and you can't affine the OS :)
> > > 
> > 
> > So the options are either to fix the TSC handling on these systems (by
> > resyncing the TSCs when exiting from HLT), or eliminate the use of rdtsc
> > by the OS?
> 
> Or both.  You can trap rdtsc users in userland, you have to manually audit
> kernel users.
> 
> It should be abolished or properly wrapped in kernel code, as soon as a
> resync path is available.
> 

For the purposes of this discussion I was not considering direct use of
rdtsc from userland for timing, I've accepted the arguments that this is
a lost cause with today's hardware (although I maintain it was viable in
practice on previous generations of hardware).  Besides, rdtsc is
useless from userspace if the kernel traps it, because the whole point
of using it over gettimeofday() was that it was dirt cheap.

I'm content to make sure the kernel's internal timekeeping is solid.

Lee

Re: Dual core Athlons and unsynced TSCs

[thockin]

On Sun, Jan 15, 2006 at 01:29:36PM -0500, Lee Revell wrote:
> > Or both.  You can trap rdtsc users in userland, you have to manually audit
> > kernel users.
> > 
> > It should be abolished or properly wrapped in kernel code, as soon as a
> > resync path is available.
> 
> For the purposes of this discussion I was not considering direct use of
> rdtsc from userland for timing, I've accepted the arguments that this is
> a lost cause with today's hardware (although I maintain it was viable in
> practice on previous generations of hardware).  Besides, rdtsc is
> useless from userspace if the kernel traps it, because the whole point
> of using it over gettimeofday() was that it was dirt cheap.
> 
> I'm content to make sure the kernel's internal timekeeping is solid.

You only need to trap rdtsc if you know that some unsyncronizing event has
happened.  E.g. 'hlt' or some other sleep state.  If we sync every time we
hlt, then we don't need to trap.  But we can maybe be lazier about it.

The sync operation itself may not be cheap, and it may be useful to skip
it if possible.

Re: Dual core Athlons and unsynced TSCs

[Andi Kleen]

john stultz <johnstul@us.ibm.com> writes:
> 
> With 2.6.15 on x86-64:
> 	If available, alternate timesources (HPET, ACPI PM) will be used if
> available on AMD SMP systems. (clock= is i386 only)

It would be good if it worked on x86-64 too - simply to unconfuse people.
It's somewhere on my todo list, but patches welcome.
 
> With 2.6.15 on i386:
> 	If CONFIG_X86_PM_TIMER is enabled, and available it is the preferred
> clocksource over the TSC.  Some distros have changed this priority
> causing the TSC to be preferred. In these cases clock=pmtmr is needed.

One problem is that it is not obvious enough to people that
CONFIG_X86_PM_TIMER is really needed for correct timing on many system
and they just don't enable it. That is why we got so many bogus reports.
 
I just changed it on x86-64 to be dependent on EMBEDDED, on by
default. i386 probably should do this change too.

-Andi

RE: Dual core Athlons and unsynced TSCs

[Thomas Gleixner]

On Fri, 2006-01-13 at 16:17 -0500, Steven Rostedt wrote:
> > >
> > > Hmm, it doesn't compile :(
> >
> > Grmbl, I check this tomorrow. Getting late here
> >
> 
> OK, I'll probably wont work on it till Monday then (wifes rules ;)

Can you try this one please ?

http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt3.patch


	tglx

RE: Dual core Athlons and unsynced TSCs

[Steven Rostedt]

On Mon, 2006-01-16 at 10:52 +0100, Thomas Gleixner wrote:
> On Fri, 2006-01-13 at 16:17 -0500, Steven Rostedt wrote:
> > > >
> > > > Hmm, it doesn't compile :(
> > >
> > > Grmbl, I check this tomorrow. Getting late here
> > >
> > 
> > OK, I'll probably wont work on it till Monday then (wifes rules ;)
> 
> Can you try this one please ?
> 
> http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt3.patch

It compiled.

I'll boot into it now.

-- Steve

Re: Dual core Athlons and unsynced TSCs

[Robert Hancock]

Lee Revell wrote:
>>>>But obviously if the TSC gives wildly inaccurate results, it cannot be
>>>>used no matter how low the overhead.
>>>
>>>unless we can re-sync the TSCs often enough that apps don't notice.
>>>
>>
>>You'd have to quantify that somehow, in terms of the max drift rate
>>(ppm), and the max resolution available (< tsc frequency).  
>>
>>Either that, or track an offset, and use one TSC as truth, and update
>>the correction factor for the other TSCs as often as needed, maybe?
>>
>>This is kind of analogous to the "drift" NTP calculates against a
>>free-running oscillator. 
>>
>>So you'd be pushing that functionality deeper into the OS-core.
>>
>>Dave Mills had that "hardpps" stuff in there for a while, it might be a
>>starting point.
>>
>>Just some thoughts for now... 
> 
> It kind of makes you wonder what in the heck AMD were thinking, whether
> they realized that this design decision would cause so many problems at
> the OS level (it's broken at least Linux and Solaris).  Maybe Windows
> keeps time in a way that was unaffected by this?

Sounds to me like they are doing something like what was being mentioned 
above:

http://support.microsoft.com/kb/896256/en-us

"When TSC does not increment monotonically, system components that use 
the kernel KeQueryPerformanceCounter function may not work correctly. To 
address this problem, Microsoft makes it possible for the ACPI Power 
Management Timer to be used as the operating system timer that supports 
the kernel KeQueryPerformanceCounter function. However, some programs 
may directly access the TSC by bypassing the Windows timer APIs. The 
multiple-processor Hardware Abstraction Layer (HAL) makes sure that the 
TSC registers on all processors on a multiple-processor computer remain 
closely synchronized. Therefore, access by system software that may be 
directed to different processors does not return different results."

Also, Microsoft's docs for QueryPerformanceCounter specify that 
different results on different CPUs will only occur if there are "bugs 
in the basic input/output system (BIOS) or the hardware abstraction 
layer (HAL)" and recommends that threads using this function set their 
affinity to run on one processor only.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/