* Dual core Athlons and unsynced TSCs @ 2006-01-12 22:17 Lee Revell 2006-01-13 15:10 ` Roger Heflin 2006-01-13 18:06 ` thockin 0 siblings, 2 replies; 62+ messages in thread From: Lee Revell @ 2006-01-12 22:17 UTC (permalink / raw) To: linux-kernel It's been known for quite some time that the TSCs are not synced between cores on Athlon X2 machines and this screws up the kernel's timekeeping, as it still uses the TSC as the default time source on these machines. This problem still seems to be present in the latest kernels. What is the plan to fix it? Is the fix simply to make the kernel use the ACPI PM timer by default on Athlon X2? Lee ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-12 22:17 Dual core Athlons and unsynced TSCs Lee Revell @ 2006-01-13 15:10 ` Roger Heflin 2006-01-13 16:04 ` Steven Rostedt 2006-01-13 18:06 ` thockin 1 sibling, 1 reply; 62+ messages in thread From: Roger Heflin @ 2006-01-13 15:10 UTC (permalink / raw) To: 'Lee Revell', 'linux-kernel' > -----Original Message----- > From: linux-kernel-owner@vger.kernel.org > [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Lee Revell > Sent: Thursday, January 12, 2006 4:18 PM > To: linux-kernel > Subject: Dual core Athlons and unsynced TSCs > > It's been known for quite some time that the TSCs are not > synced between cores on Athlon X2 machines and this screws up > the kernel's timekeeping, as it still uses the TSC as the > default time source on these machines. > > This problem still seems to be present in the latest kernels. > What is the plan to fix it? Is the fix simply to make the > kernel use the ACPI PM timer by default on Athlon X2? Do we know if this also affects dual-core opterons? The symptoms are that the clocks run at 2x the speed, correct? Roger ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-13 15:10 ` Roger Heflin @ 2006-01-13 16:04 ` Steven Rostedt 2006-01-13 17:47 ` Lee Revell 0 siblings, 1 reply; 62+ messages in thread From: Steven Rostedt @ 2006-01-13 16:04 UTC (permalink / raw) To: Roger Heflin; +Cc: 'linux-kernel', 'Lee Revell' On Fri, 2006-01-13 at 09:10 -0600, Roger Heflin wrote: > > > -----Original Message----- > > From: linux-kernel-owner@vger.kernel.org > > [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Lee Revell > > Sent: Thursday, January 12, 2006 4:18 PM > > To: linux-kernel > > Subject: Dual core Athlons and unsynced TSCs > > > > It's been known for quite some time that the TSCs are not > > synced between cores on Athlon X2 machines and this screws up > > the kernel's timekeeping, as it still uses the TSC as the > > default time source on these machines. > > > > This problem still seems to be present in the latest kernels. > > What is the plan to fix it? Is the fix simply to make the > > kernel use the ACPI PM timer by default on Athlon X2? > > > Do we know if this also affects dual-core opterons? > > The symptoms are that the clocks run at 2x the speed, correct? No, worse. The monotonic clock can go backwards. The tscs of the CPUs are not in sync when one slows down due to idle. So if you read from two different CPUs, you may get the second read have an earlier time than the first. Breaks the rule of what a monotonic clock is. -- Steve ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-13 16:04 ` Steven Rostedt @ 2006-01-13 17:47 ` Lee Revell 2006-01-13 17:52 ` Steven Rostedt 0 siblings, 1 reply; 62+ messages in thread From: Lee Revell @ 2006-01-13 17:47 UTC (permalink / raw) To: Steven Rostedt; +Cc: Roger Heflin, 'linux-kernel' On Fri, 2006-01-13 at 11:04 -0500, Steven Rostedt wrote: > On Fri, 2006-01-13 at 09:10 -0600, Roger Heflin wrote: > > > > > -----Original Message----- > > > From: linux-kernel-owner@vger.kernel.org > > > [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Lee Revell > > > Sent: Thursday, January 12, 2006 4:18 PM > > > To: linux-kernel > > > Subject: Dual core Athlons and unsynced TSCs > > > > > > It's been known for quite some time that the TSCs are not > > > synced between cores on Athlon X2 machines and this screws up > > > the kernel's timekeeping, as it still uses the TSC as the > > > default time source on these machines. > > > > > > This problem still seems to be present in the latest kernels. > > > What is the plan to fix it? Is the fix simply to make the > > > kernel use the ACPI PM timer by default on Athlon X2? > > > > > > Do we know if this also affects dual-core opterons? > > > > The symptoms are that the clocks run at 2x the speed, correct? > > No, worse. The monotonic clock can go backwards. The tscs of the CPUs > are not in sync when one slows down due to idle. So if you read from > two different CPUs, you may get the second read have an earlier time > than the first. Breaks the rule of what a monotonic clock is. > Steve, I don't have hardware to test this, can you confirm that the only workaround needed is to boot with "clock=pmtmr"? Lee ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-13 17:47 ` Lee Revell @ 2006-01-13 17:52 ` Steven Rostedt 2006-01-13 17:54 ` Lee Revell 0 siblings, 1 reply; 62+ messages in thread From: Steven Rostedt @ 2006-01-13 17:52 UTC (permalink / raw) To: Lee Revell; +Cc: Roger Heflin, 'linux-kernel' On Fri, 13 Jan 2006, Lee Revell wrote: > I don't have hardware to test this, can you confirm that the only > workaround needed is to boot with "clock=pmtmr"? For which kernel? -- Steve ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-13 17:52 ` Steven Rostedt @ 2006-01-13 17:54 ` Lee Revell 2006-01-13 18:41 ` Steven Rostedt 0 siblings, 1 reply; 62+ messages in thread From: Lee Revell @ 2006-01-13 17:54 UTC (permalink / raw) To: Steven Rostedt; +Cc: Roger Heflin, 'linux-kernel' On Fri, 2006-01-13 at 12:52 -0500, Steven Rostedt wrote: > On Fri, 13 Jan 2006, Lee Revell wrote: > > > I don't have hardware to test this, can you confirm that the only > > workaround needed is to boot with "clock=pmtmr"? > > For which kernel? I believe the problem exists on all kernels, it should not matter. Lee ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-13 17:54 ` Lee Revell @ 2006-01-13 18:41 ` Steven Rostedt 2006-01-13 18:55 ` Lee Revell 0 siblings, 1 reply; 62+ messages in thread From: Steven Rostedt @ 2006-01-13 18:41 UTC (permalink / raw) To: Lee Revell; +Cc: Roger Heflin, 'linux-kernel' On Fri, 13 Jan 2006, Lee Revell wrote: > On Fri, 2006-01-13 at 12:52 -0500, Steven Rostedt wrote: > > On Fri, 13 Jan 2006, Lee Revell wrote: > > > > > I don't have hardware to test this, can you confirm that the only > > > workaround needed is to boot with "clock=pmtmr"? > > > > For which kernel? > > I believe the problem exists on all kernels, it should not matter. Well, I just booted up (and am currently running) 2.6.15-rt4-sr2 and I used "clock=pmtmr". I haven't got any read_tsc: ACK! TSC went backward! Unsynced TSCs? If that's what you want to know? But the latency traces are still screwed up. -- Steve ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-13 18:41 ` Steven Rostedt @ 2006-01-13 18:55 ` Lee Revell 2006-01-13 20:09 ` Steven Rostedt 2006-01-14 6:54 ` Christoph Lameter 0 siblings, 2 replies; 62+ messages in thread From: Lee Revell @ 2006-01-13 18:55 UTC (permalink / raw) To: Steven Rostedt; +Cc: Roger Heflin, 'linux-kernel' On Fri, 2006-01-13 at 13:41 -0500, Steven Rostedt wrote: > On Fri, 13 Jan 2006, Lee Revell wrote: > > > On Fri, 2006-01-13 at 12:52 -0500, Steven Rostedt wrote: > > > On Fri, 13 Jan 2006, Lee Revell wrote: > > > > > > > I don't have hardware to test this, can you confirm that the only > > > > workaround needed is to boot with "clock=pmtmr"? > > > > > > For which kernel? > > > > I believe the problem exists on all kernels, it should not matter. > > > Well, I just booted up (and am currently running) 2.6.15-rt4-sr2 and I > used "clock=pmtmr". I haven't got any > > read_tsc: ACK! TSC went backward! Unsynced TSCs? > > > If that's what you want to know? I want to know if clock_gettime(CLOCK_MONOTONIC, *ts) is actually guaranteed to be monotonic on these machines AKA do we break POSIX compliance or not. Lee ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-13 18:55 ` Lee Revell @ 2006-01-13 20:09 ` Steven Rostedt 2006-01-13 20:26 ` Steven Rostedt 2006-01-14 0:23 ` john stultz 2006-01-14 6:54 ` Christoph Lameter 1 sibling, 2 replies; 62+ messages in thread From: Steven Rostedt @ 2006-01-13 20:09 UTC (permalink / raw) To: Lee Revell Cc: john stultz, Thomas Gleixner, Ingo Molnar, Roger Heflin, 'linux-kernel' On Fri, 2006-01-13 at 13:55 -0500, Lee Revell wrote: > > > > If that's what you want to know? > > I want to know if clock_gettime(CLOCK_MONOTONIC, *ts) is actually > guaranteed to be monotonic on these machines AKA do we break POSIX > compliance or not. Nope it doesn't work :-( I ran this test: http://www.kihontech.com/tests/rt/monotonic.c And got this: $ ./monotonic main program pid=8477 hello from thread 0! hello from thread 1! hello from thread 2! hello from thread 3! hello from thread 4! Failed! prev: 6279.925610302 current: 6279.874615349 $ cat /proc/cmdline root=/dev/md0 ro console=ttyS0,115200 console=tty0 nmi_watchdog=2 lapic earlyprintk=ttyS0,115200 clock=pmtmr $ uname -r 2.6.15-rt4-sr2 (I also added Ingo, Thomas, and John in the CC) I'll reboot to vanilla 2.6.15 and see if that is broken too (just to make sure) -- Steve ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-13 20:09 ` Steven Rostedt @ 2006-01-13 20:26 ` Steven Rostedt 2006-01-13 20:40 ` Lee Revell 2006-01-13 20:43 ` Lee Revell 2006-01-14 0:23 ` john stultz 1 sibling, 2 replies; 62+ messages in thread From: Steven Rostedt @ 2006-01-13 20:26 UTC (permalink / raw) To: Lee Revell Cc: 'linux-kernel', Roger Heflin, Ingo Molnar, Thomas Gleixner, john stultz On Fri, 2006-01-13 at 15:09 -0500, Steven Rostedt wrote: > Nope it doesn't work :-( > > I ran this test: > http://www.kihontech.com/tests/rt/monotonic.c > [...] > > I'll reboot to vanilla 2.6.15 and see if that is broken too (just to > make sure) > Failed on 2.6.15 too: $ ./monotonic main program pid=6724 hello from thread 0! hello from thread 4! Failed! prev: 316.952599248 current: 316.951899248 hello from thread 3! hello from thread 2! hello from thread 1! $ uname -r 2.6.15 $ cat /proc/cmdline root=/dev/md0 ro console=tty0 nmi_watchdog=2 lapic clock=pmtmr :-( -- Steve ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-13 20:26 ` Steven Rostedt @ 2006-01-13 20:40 ` Lee Revell 2006-01-13 20:43 ` Lee Revell 1 sibling, 0 replies; 62+ messages in thread From: Lee Revell @ 2006-01-13 20:40 UTC (permalink / raw) To: Steven Rostedt Cc: 'linux-kernel', Roger Heflin, Ingo Molnar, Thomas Gleixner, john stultz On Fri, 2006-01-13 at 15:26 -0500, Steven Rostedt wrote: > On Fri, 2006-01-13 at 15:09 -0500, Steven Rostedt wrote: > > > Nope it doesn't work :-( > > > > I ran this test: > > http://www.kihontech.com/tests/rt/monotonic.c > > > [...] > > > > I'll reboot to vanilla 2.6.15 and see if that is broken too (just to > > make sure) > > > > Failed on 2.6.15 too: That's what I suspected based on reports from several JACK users (JACK used to use rdtsc directly, it now uses clock_gettime). Hmm, it looks like the kernel uses the TSC left and right, even if clock=pmtmr is used. There's even an rdtscll in pmtimer_mark_offset(). Wow, what a mess. Thanks, AMD... Lee ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-13 20:26 ` Steven Rostedt 2006-01-13 20:40 ` Lee Revell @ 2006-01-13 20:43 ` Lee Revell 2006-01-13 20:46 ` Thomas Gleixner 1 sibling, 1 reply; 62+ messages in thread From: Lee Revell @ 2006-01-13 20:43 UTC (permalink / raw) To: Steven Rostedt Cc: 'linux-kernel', Roger Heflin, Ingo Molnar, Thomas Gleixner, john stultz On Fri, 2006-01-13 at 15:26 -0500, Steven Rostedt wrote: > On Fri, 2006-01-13 at 15:09 -0500, Steven Rostedt wrote: > > > Nope it doesn't work :-( > > > > I ran this test: > > http://www.kihontech.com/tests/rt/monotonic.c > > > [...] > > > > I'll reboot to vanilla 2.6.15 and see if that is broken too (just to > > make sure) > > > > Failed on 2.6.15 too: > Ugh. In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC unconditionally. Lee ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-13 20:43 ` Lee Revell @ 2006-01-13 20:46 ` Thomas Gleixner 2006-01-13 20:48 ` Lee Revell 2006-01-13 21:05 ` Steven Rostedt 0 siblings, 2 replies; 62+ messages in thread From: Thomas Gleixner @ 2006-01-13 20:46 UTC (permalink / raw) To: Lee Revell Cc: Steven Rostedt, 'linux-kernel', Roger Heflin, Ingo Molnar, john stultz On Fri, 2006-01-13 at 15:43 -0500, Lee Revell wrote: > Ugh. In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC > unconditionally. Can you try http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch please ? tglx ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-13 20:46 ` Thomas Gleixner @ 2006-01-13 20:48 ` Lee Revell 2006-01-13 20:51 ` Thomas Gleixner 2006-01-13 20:52 ` Steven Rostedt 2006-01-13 21:05 ` Steven Rostedt 1 sibling, 2 replies; 62+ messages in thread From: Lee Revell @ 2006-01-13 20:48 UTC (permalink / raw) To: tglx Cc: Steven Rostedt, 'linux-kernel', Roger Heflin, Ingo Molnar, john stultz On Fri, 2006-01-13 at 21:46 +0100, Thomas Gleixner wrote: > On Fri, 2006-01-13 at 15:43 -0500, Lee Revell wrote: > > > Ugh. In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC > > unconditionally. > > Can you try > > http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch > > please ? I can try it (well I can pass it on to someone who has the hardware), but that's a huge patch, not likely to be mergeable in the 2.6.16 timeframe. Surely there has to be a way this can be fixed for 2.6.16? Lee ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-13 20:48 ` Lee Revell @ 2006-01-13 20:51 ` Thomas Gleixner 2006-01-13 20:52 ` Steven Rostedt 1 sibling, 0 replies; 62+ messages in thread From: Thomas Gleixner @ 2006-01-13 20:51 UTC (permalink / raw) To: Lee Revell Cc: Steven Rostedt, 'linux-kernel', Roger Heflin, Ingo Molnar, john stultz On Fri, 2006-01-13 at 15:48 -0500, Lee Revell wrote: > On Fri, 2006-01-13 at 21:46 +0100, Thomas Gleixner wrote: > > On Fri, 2006-01-13 at 15:43 -0500, Lee Revell wrote: > > > > > Ugh. In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC > > > unconditionally. > > > > Can you try > > > > http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch > > > > please ? > > I can try it (well I can pass it on to someone who has the hardware), > but that's a huge patch, not likely to be mergeable in the 2.6.16 > timeframe. Surely there has to be a way this can be fixed for 2.6.16? Parts of the patch are in mainline (-gitX), but those alone will not fix the problem AFAICS tglx ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-13 20:48 ` Lee Revell 2006-01-13 20:51 ` Thomas Gleixner @ 2006-01-13 20:52 ` Steven Rostedt 1 sibling, 0 replies; 62+ messages in thread From: Steven Rostedt @ 2006-01-13 20:52 UTC (permalink / raw) To: Lee Revell Cc: tglx, 'linux-kernel', Roger Heflin, Ingo Molnar, john stultz On Fri, 13 Jan 2006, Lee Revell wrote: > On Fri, 2006-01-13 at 21:46 +0100, Thomas Gleixner wrote: > > On Fri, 2006-01-13 at 15:43 -0500, Lee Revell wrote: > > > > > Ugh. In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC > > > unconditionally. > > > > Can you try > > > > http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch > > > > please ? > > I can try it (well I can pass it on to someone who has the hardware), > but that's a huge patch, not likely to be mergeable in the 2.6.16 > timeframe. Surely there has to be a way this can be fixed for 2.6.16? I'll try it. If it works, at least it might give us a clue to what to do. -- Steve ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-13 20:46 ` Thomas Gleixner 2006-01-13 20:48 ` Lee Revell @ 2006-01-13 21:05 ` Steven Rostedt 2006-01-13 21:10 ` Thomas Gleixner 1 sibling, 1 reply; 62+ messages in thread From: Steven Rostedt @ 2006-01-13 21:05 UTC (permalink / raw) To: tglx Cc: Lee Revell, 'linux-kernel', Roger Heflin, Ingo Molnar, john stultz On Fri, 2006-01-13 at 21:46 +0100, Thomas Gleixner wrote: > On Fri, 2006-01-13 at 15:43 -0500, Lee Revell wrote: > > > Ugh. In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC > > unconditionally. > > Can you try > > http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch > > please ? > > tglx > > Hmm, it doesn't compile :( -- Steve rostedt@gandalf:~/work/kernels/linux-2.6.15-hrt2$ make amdmake CHK include/linux/version.h CC arch/x86_64/kernel/asm-offsets.s In file included from include/linux/timex.h:61, from include/linux/sched.h:11, from arch/x86_64/kernel/asm-offsets.c:7: include/asm/timex.h: In function 'get_cycles_sync': include/asm/timex.h:36: warning: implicit declaration of function 'alternative_io' include/asm/timex.h:37: error: called object '"=a"' is not a function include/asm/timex.h:37: error: called object '"0"' is not a function include/asm/timex.h:37: error: syntax error before ':' token make[1]: *** [arch/x86_64/kernel/asm-offsets.s] Error 1 make: *** [prepare0] Error 2 ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-13 21:05 ` Steven Rostedt @ 2006-01-13 21:10 ` Thomas Gleixner 2006-01-13 21:17 ` Steven Rostedt 0 siblings, 1 reply; 62+ messages in thread From: Thomas Gleixner @ 2006-01-13 21:10 UTC (permalink / raw) To: Steven Rostedt Cc: Lee Revell, 'linux-kernel', Roger Heflin, Ingo Molnar, john stultz On Fri, 2006-01-13 at 16:05 -0500, Steven Rostedt wrote: > On Fri, 2006-01-13 at 21:46 +0100, Thomas Gleixner wrote: > > On Fri, 2006-01-13 at 15:43 -0500, Lee Revell wrote: > > > > > Ugh. In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC > > > unconditionally. > > > > Can you try > > > > http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch > > > > please ? > > > > tglx > > > > > > Hmm, it doesn't compile :( Grmbl, I check this tomorrow. Getting late here tglx ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-13 21:10 ` Thomas Gleixner @ 2006-01-13 21:17 ` Steven Rostedt 2006-01-16 9:52 ` Thomas Gleixner 0 siblings, 1 reply; 62+ messages in thread From: Steven Rostedt @ 2006-01-13 21:17 UTC (permalink / raw) To: Thomas Gleixner Cc: Lee Revell, 'linux-kernel', Roger Heflin, Ingo Molnar, john stultz On Fri, 13 Jan 2006, Thomas Gleixner wrote: > On Fri, 2006-01-13 at 16:05 -0500, Steven Rostedt wrote: > > On Fri, 2006-01-13 at 21:46 +0100, Thomas Gleixner wrote: > > > On Fri, 2006-01-13 at 15:43 -0500, Lee Revell wrote: > > > > > > > Ugh. In arch/x86_64/kernel/time.c monotonic_clock() uses the TSC > > > > unconditionally. > > > > > > Can you try > > > > > > http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt2.patch > > > > > > please ? > > > > > > tglx > > > > > > > > > > Hmm, it doesn't compile :( > > Grmbl, I check this tomorrow. Getting late here > OK, I'll probably wont work on it till Monday then (wifes rules ;) -- Steve ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-13 21:17 ` Steven Rostedt @ 2006-01-16 9:52 ` Thomas Gleixner 2006-01-16 15:55 ` Steven Rostedt 0 siblings, 1 reply; 62+ messages in thread From: Thomas Gleixner @ 2006-01-16 9:52 UTC (permalink / raw) To: Steven Rostedt Cc: Lee Revell, 'linux-kernel', Roger Heflin, Ingo Molnar, john stultz On Fri, 2006-01-13 at 16:17 -0500, Steven Rostedt wrote: > > > > > > Hmm, it doesn't compile :( > > > > Grmbl, I check this tomorrow. Getting late here > > > > OK, I'll probably wont work on it till Monday then (wifes rules ;) Can you try this one please ? http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt3.patch tglx ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-16 9:52 ` Thomas Gleixner @ 2006-01-16 15:55 ` Steven Rostedt 0 siblings, 0 replies; 62+ messages in thread From: Steven Rostedt @ 2006-01-16 15:55 UTC (permalink / raw) To: tglx Cc: Lee Revell, 'linux-kernel', Roger Heflin, Ingo Molnar, john stultz On Mon, 2006-01-16 at 10:52 +0100, Thomas Gleixner wrote: > On Fri, 2006-01-13 at 16:17 -0500, Steven Rostedt wrote: > > > > > > > > Hmm, it doesn't compile :( > > > > > > Grmbl, I check this tomorrow. Getting late here > > > > > > > OK, I'll probably wont work on it till Monday then (wifes rules ;) > > Can you try this one please ? > > http://tglx.de/projects/hrtimers/2.6.15/patch-2.6.15-hrt3.patch It compiled. I'll boot into it now. -- Steve ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-13 20:09 ` Steven Rostedt 2006-01-13 20:26 ` Steven Rostedt @ 2006-01-14 0:23 ` john stultz 2006-01-14 1:10 ` Steven Rostedt 1 sibling, 1 reply; 62+ messages in thread From: john stultz @ 2006-01-14 0:23 UTC (permalink / raw) To: Steven Rostedt Cc: Lee Revell, Thomas Gleixner, Ingo Molnar, Roger Heflin, 'linux-kernel' [-- Attachment #1: Type: text/plain, Size: 1301 bytes --] On Fri, 2006-01-13 at 15:09 -0500, Steven Rostedt wrote: > On Fri, 2006-01-13 at 13:55 -0500, Lee Revell wrote: > > > > > > > If that's what you want to know? > > > > I want to know if clock_gettime(CLOCK_MONOTONIC, *ts) is actually > > guaranteed to be monotonic on these machines AKA do we break POSIX > > compliance or not. > > Nope it doesn't work :-( > > I ran this test: > http://www.kihontech.com/tests/rt/monotonic.c > > And got this: > > $ ./monotonic > main program pid=8477 > hello from thread 0! > hello from thread 1! > hello from thread 2! > hello from thread 3! > hello from thread 4! > Failed! prev: 6279.925610302 current: 6279.874615349 Not sure, but I think your test is broken. While it serializes the checking and exchange of current and last, it doesn't serialize the actual calling of clock_gettime(). Thus you can get something like: last=0 A: B: 1: now = clock_gettime() 2: now = clock_gettime() 3: atomic { test(now, last) [2>0] last = now } 4: atomic { test(now, last) [1>2] FAIL! } Attached is a similar testcase that I've been using myself that avoids this issue (although I just converted it from gettimeofday to clock_gettime, so there may still be an issue). Let me know if you have any comments on it. thanks -john [-- Attachment #2: threadtest.c --] [-- Type: text/x-csrc, Size: 2890 bytes --] /* threadtest.c * by: john stultz (johnstul@us.ibm.com) * (C) Copyright IBM 2004, 2005, 2006 * Licensed under the GPL */ #include <stdio.h> #include <sys/time.h> #include <pthread.h> /* serializes shared list access */ pthread_mutex_t list_lock = PTHREAD_MUTEX_INITIALIZER; /* serializes console output */ pthread_mutex_t print_lock = PTHREAD_MUTEX_INITIALIZER; #define MAX_THREADS 128 #define LISTSIZE 128 int done = 0; struct timespec global_list[LISTSIZE]; int listcount = 0; void checklist(struct timespec* list, int size) { int i,j; struct timespec *a,*b; static int dbgct = 0; if(!(dbgct++%7000)){ printf("."); fflush(0); } /* scan the list */ for(i=0; i < size-1; i++){ a = &list[i]; b = &list[i+1]; /* look for any time inconsistencies */ if((b->tv_sec <= a->tv_sec)&& (b->tv_nsec < a->tv_nsec)){ /* flag other threads */ done = 1; /*serialize printing to avoid junky output*/ pthread_mutex_lock(&print_lock); /* dump the list */ printf("\n"); for(j=0; j< size; j++){ if(j == i) printf("---------------\n"); printf("%lu:%lu\n", list[j].tv_sec,list[j].tv_nsec); if(j == i+1) printf("---------------\n"); } printf("TEST FAILED\n"); pthread_mutex_unlock(&print_lock); } } } /* The Shared Thread shares a global list * that each thread fills while holding the lock. * This stresses clock syncronization across cpus. */ void* shared_thread(void* arg) { while(!done){ /* protect the list */ pthread_mutex_lock(&list_lock); /* see if we're ready to check the list */ if(listcount >= LISTSIZE){ checklist(global_list, LISTSIZE); listcount = 0; } clock_gettime(CLOCK_MONOTONIC, &global_list[listcount++]); pthread_mutex_unlock(&list_lock); } } /* Each independent thread fills in its own * list. This stresses clock_gettime() lock contention. */ void* independent_thread(void* arg) { struct timespec my_list[LISTSIZE]; int count; while(!done){ /* fill the list */ for(count=0; count < LISTSIZE; count++) clock_gettime(CLOCK_MONOTONIC, &my_list[count]); checklist(my_list, LISTSIZE); } } int main(int argc, void** argv) { int thread_count = 1, i; pthread_t pth[MAX_THREADS]; void* ret; void* (*thread)(void*) = shared_thread; /* pull the thread count */ if(argc > 1) thread_count = strtol(argv[1],0,0); if(thread_count > MAX_THREADS) thread_count = MAX_THREADS; /* check if we're running independent threads */ if(argc == 3 && !strncmp(argv[2],"indie", 5)){ thread = independent_thread; printf("using independent threads\n"); } system("date"); printf("spawning %i threads\n", thread_count,0,0); /* spawn */ for(i=0; i < thread_count; i++) pthread_create(&pth[i], 0, thread, 0); /* wait */ for(i=0; i< thread_count; i++) pthread_join(pth[i],&ret); system("date"); /* die */ return 0; } ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-14 0:23 ` john stultz @ 2006-01-14 1:10 ` Steven Rostedt 2006-01-14 1:14 ` Lee Revell 0 siblings, 1 reply; 62+ messages in thread From: Steven Rostedt @ 2006-01-14 1:10 UTC (permalink / raw) To: john stultz Cc: Lee Revell, Thomas Gleixner, Ingo Molnar, Roger Heflin, 'linux-kernel' On Fri, 2006-01-13 at 16:23 -0800, john stultz wrote: > > Failed! prev: 6279.925610302 current: 6279.874615349 > > Not sure, but I think your test is broken. Yep and it is now fixed: http://www.kihontech.com/tests/rt/monotonic.c > > Attached is a similar testcase that I've been using myself that avoids > this issue (although I just converted it from gettimeofday to > clock_gettime, so there may still be an issue). Let me know if you have > any comments on it. Thanks, I'll add that to my list of tests too. Oh and 2.6.15 passed as well (with clock=pmtmr) -- Steve ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-14 1:10 ` Steven Rostedt @ 2006-01-14 1:14 ` Lee Revell 2006-01-14 1:23 ` john stultz 0 siblings, 1 reply; 62+ messages in thread From: Lee Revell @ 2006-01-14 1:14 UTC (permalink / raw) To: Steven Rostedt Cc: john stultz, Thomas Gleixner, Ingo Molnar, Roger Heflin, 'linux-kernel' On Fri, 2006-01-13 at 20:10 -0500, Steven Rostedt wrote: > > Thanks, I'll add that to my list of tests too. > > Oh and 2.6.15 passed as well (with clock=pmtmr) It really seems like it would fail if you gave it enough time due to the rdtsc in monotonic_clock()... Lee ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-14 1:14 ` Lee Revell @ 2006-01-14 1:23 ` john stultz 2006-01-14 1:27 ` Lee Revell 0 siblings, 1 reply; 62+ messages in thread From: john stultz @ 2006-01-14 1:23 UTC (permalink / raw) To: Lee Revell Cc: Steven Rostedt, Thomas Gleixner, Ingo Molnar, Roger Heflin, 'linux-kernel' On Fri, 2006-01-13 at 20:14 -0500, Lee Revell wrote: > On Fri, 2006-01-13 at 20:10 -0500, Steven Rostedt wrote: > > > > Thanks, I'll add that to my list of tests too. > > > > Oh and 2.6.15 passed as well (with clock=pmtmr) > > It really seems like it would fail if you gave it enough time due to the > rdtsc in monotonic_clock()... Currently monotonic_clock()is only used by the hangcheck-timer, and is not used by gettimeofday/clock_gettime (even w/ CLOCK_MONOTONIC). So there may still be an issue there w/ the hangcheck-timer(for x86-64, on i386 the acpi pm timer can be used for monotonic_clock), but its doesn't affect the time related userland interfaces. thanks -john ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-14 1:23 ` john stultz @ 2006-01-14 1:27 ` Lee Revell 2006-01-14 1:39 ` john stultz 0 siblings, 1 reply; 62+ messages in thread From: Lee Revell @ 2006-01-14 1:27 UTC (permalink / raw) To: john stultz Cc: Steven Rostedt, Thomas Gleixner, Ingo Molnar, Roger Heflin, 'linux-kernel' On Fri, 2006-01-13 at 17:23 -0800, john stultz wrote: > On Fri, 2006-01-13 at 20:14 -0500, Lee Revell wrote: > > On Fri, 2006-01-13 at 20:10 -0500, Steven Rostedt wrote: > > > > > > Thanks, I'll add that to my list of tests too. > > > > > > Oh and 2.6.15 passed as well (with clock=pmtmr) > > > > It really seems like it would fail if you gave it enough time due to the > > rdtsc in monotonic_clock()... > > Currently monotonic_clock()is only used by the hangcheck-timer, and is > not used by gettimeofday/clock_gettime (even w/ CLOCK_MONOTONIC). > > So there may still be an issue there w/ the hangcheck-timer(for x86-64, > on i386 the acpi pm timer can be used for monotonic_clock), but its > doesn't affect the time related userland interfaces. OK so the last question is how do we make sure the kernel uses the clock=pmtmr behavior by default on those machines? Lee ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-14 1:27 ` Lee Revell @ 2006-01-14 1:39 ` john stultz 2006-01-14 1:43 ` Steven Rostedt 2006-01-16 1:19 ` Andi Kleen 0 siblings, 2 replies; 62+ messages in thread From: john stultz @ 2006-01-14 1:39 UTC (permalink / raw) To: Lee Revell Cc: Steven Rostedt, Thomas Gleixner, Ingo Molnar, Roger Heflin, 'linux-kernel' On Fri, 2006-01-13 at 20:27 -0500, Lee Revell wrote: > On Fri, 2006-01-13 at 17:23 -0800, john stultz wrote: > > On Fri, 2006-01-13 at 20:14 -0500, Lee Revell wrote: > > > On Fri, 2006-01-13 at 20:10 -0500, Steven Rostedt wrote: > > > > > > > > Thanks, I'll add that to my list of tests too. > > > > > > > > Oh and 2.6.15 passed as well (with clock=pmtmr) > > > > > > It really seems like it would fail if you gave it enough time due to the > > > rdtsc in monotonic_clock()... > > > > Currently monotonic_clock()is only used by the hangcheck-timer, and is > > not used by gettimeofday/clock_gettime (even w/ CLOCK_MONOTONIC). > > > > So there may still be an issue there w/ the hangcheck-timer(for x86-64, > > on i386 the acpi pm timer can be used for monotonic_clock), but its > > doesn't affect the time related userland interfaces. > > OK so the last question is how do we make sure the kernel uses the > clock=pmtmr behavior by default on those machines? This is as I understand it: With 2.6.15 on x86-64: If available, alternate timesources (HPET, ACPI PM) will be used if available on AMD SMP systems. (clock= is i386 only) With 2.6.15 on i386: If CONFIG_X86_PM_TIMER is enabled, and available it is the preferred clocksource over the TSC. Some distros have changed this priority causing the TSC to be preferred. In these cases clock=pmtmr is needed. How's that? -john ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-14 1:39 ` john stultz @ 2006-01-14 1:43 ` Steven Rostedt 2006-01-14 1:50 ` john stultz 2006-01-16 1:19 ` Andi Kleen 1 sibling, 1 reply; 62+ messages in thread From: Steven Rostedt @ 2006-01-14 1:43 UTC (permalink / raw) To: john stultz Cc: Lee Revell, Thomas Gleixner, Ingo Molnar, Roger Heflin, 'linux-kernel' On Fri, 13 Jan 2006, john stultz wrote: > > This is as I understand it: > > With 2.6.15 on x86-64: > If available, alternate timesources (HPET, ACPI PM) will be used if > available on AMD SMP systems. (clock= is i386 only) Hmm, should I boot without the clock= to prove this? -- Steve > > With 2.6.15 on i386: > If CONFIG_X86_PM_TIMER is enabled, and available it is the preferred > clocksource over the TSC. Some distros have changed this priority > causing the TSC to be preferred. In these cases clock=pmtmr is needed. > > How's that? > -john > > > ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-14 1:43 ` Steven Rostedt @ 2006-01-14 1:50 ` john stultz 2006-01-14 2:15 ` Steven Rostedt 2006-01-14 2:34 ` Steven Rostedt 0 siblings, 2 replies; 62+ messages in thread From: john stultz @ 2006-01-14 1:50 UTC (permalink / raw) To: Steven Rostedt Cc: Lee Revell, Thomas Gleixner, Ingo Molnar, Roger Heflin, 'linux-kernel' On Fri, 2006-01-13 at 20:43 -0500, Steven Rostedt wrote: > On Fri, 13 Jan 2006, john stultz wrote: > > > > > This is as I understand it: > > > > With 2.6.15 on x86-64: > > If available, alternate timesources (HPET, ACPI PM) will be used if > > available on AMD SMP systems. (clock= is i386 only) > > Hmm, should I boot without the clock= to prove this? Feel free. That or grep the x86-64 time.c code. Look for: time.c: Using PM based timekeeping. To verify the timesource selection. thanks -john ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-14 1:50 ` john stultz @ 2006-01-14 2:15 ` Steven Rostedt 2006-01-14 2:34 ` Steven Rostedt 1 sibling, 0 replies; 62+ messages in thread From: Steven Rostedt @ 2006-01-14 2:15 UTC (permalink / raw) To: john stultz Cc: Lee Revell, Thomas Gleixner, Ingo Molnar, Roger Heflin, 'linux-kernel' On Fri, 13 Jan 2006, john stultz wrote: > On Fri, 2006-01-13 at 20:43 -0500, Steven Rostedt wrote: > > On Fri, 13 Jan 2006, john stultz wrote: > > > > > > > > This is as I understand it: > > > > > > With 2.6.15 on x86-64: > > > If available, alternate timesources (HPET, ACPI PM) will be used if > > > available on AMD SMP systems. (clock= is i386 only) > > > > Hmm, should I boot without the clock= to prove this? > > Feel free. That or grep the x86-64 time.c code. > > Look for: > time.c: Using PM based timekeeping. > > To verify the timesource selection. I'm too busy trying to install linux on my powerbook to look at code at the moment :-) I just rebooted and ran the test on 2.6.15 without the clock= option and it passed my test. I'll wait a while and run it again to see if I get any other drift too prove it more. -- Steve ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-14 1:50 ` john stultz 2006-01-14 2:15 ` Steven Rostedt @ 2006-01-14 2:34 ` Steven Rostedt 1 sibling, 0 replies; 62+ messages in thread From: Steven Rostedt @ 2006-01-14 2:34 UTC (permalink / raw) To: john stultz Cc: Lee Revell, Thomas Gleixner, Ingo Molnar, Roger Heflin, 'linux-kernel' On Fri, 2006-01-13 at 17:50 -0800, john stultz wrote: > On Fri, 2006-01-13 at 20:43 -0500, Steven Rostedt wrote: > > On Fri, 13 Jan 2006, john stultz wrote: > > > > > > > > This is as I understand it: > > > > > > With 2.6.15 on x86-64: > > > If available, alternate timesources (HPET, ACPI PM) will be used if > > > available on AMD SMP systems. (clock= is i386 only) > > > > Hmm, should I boot without the clock= to prove this? > > Feel free. That or grep the x86-64 time.c code. > > Look for: > time.c: Using PM based timekeeping. > > To verify the timesource selection. So much for not looking at code! I uploaded yet another version of http://www.kihontech.com/tests/rt/monotonic.c that now also tests the tsc too, to see if it goes backwards. If it does, then it prints (only once) that fact and continues testing the gettime function. So this proves that the tsc does go back and the gettime still is working. $ ./monotonic main program pid=7137 hello from thread 0! hello from thread 4! last tsc is 4389431102233 this tsc is 4389429636684 hello from thread 1! hello from thread 2! hello from thread 3! -- Steve ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-14 1:39 ` john stultz 2006-01-14 1:43 ` Steven Rostedt @ 2006-01-16 1:19 ` Andi Kleen 1 sibling, 0 replies; 62+ messages in thread From: Andi Kleen @ 2006-01-16 1:19 UTC (permalink / raw) To: john stultz Cc: Steven Rostedt, Thomas Gleixner, Ingo Molnar, Roger Heflin, 'linux-kernel' john stultz <johnstul@us.ibm.com> writes: > > With 2.6.15 on x86-64: > If available, alternate timesources (HPET, ACPI PM) will be used if > available on AMD SMP systems. (clock= is i386 only) It would be good if it worked on x86-64 too - simply to unconfuse people. It's somewhere on my todo list, but patches welcome. > With 2.6.15 on i386: > If CONFIG_X86_PM_TIMER is enabled, and available it is the preferred > clocksource over the TSC. Some distros have changed this priority > causing the TSC to be preferred. In these cases clock=pmtmr is needed. One problem is that it is not obvious enough to people that CONFIG_X86_PM_TIMER is really needed for correct timing on many system and they just don't enable it. That is why we got so many bogus reports. I just changed it on x86-64 to be dependent on EMBEDDED, on by default. i386 probably should do this change too. -Andi ^ permalink raw reply [flat|nested] 62+ messages in thread
* RE: Dual core Athlons and unsynced TSCs 2006-01-13 18:55 ` Lee Revell 2006-01-13 20:09 ` Steven Rostedt @ 2006-01-14 6:54 ` Christoph Lameter 1 sibling, 0 replies; 62+ messages in thread From: Christoph Lameter @ 2006-01-14 6:54 UTC (permalink / raw) To: Lee Revell; +Cc: Steven Rostedt, Roger Heflin, 'linux-kernel' On Fri, 13 Jan 2006, Lee Revell wrote: > I want to know if clock_gettime(CLOCK_MONOTONIC, *ts) is actually > guaranteed to be monotonic on these machines AKA do we break POSIX > compliance or not. Yes you do. Other arches have had the same issues.... See IA64 code. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-12 22:17 Dual core Athlons and unsynced TSCs Lee Revell 2006-01-13 15:10 ` Roger Heflin @ 2006-01-13 18:06 ` thockin 2006-01-13 17:58 ` Lee Revell 1 sibling, 1 reply; 62+ messages in thread From: thockin @ 2006-01-13 18:06 UTC (permalink / raw) To: Lee Revell; +Cc: linux-kernel On Thu, Jan 12, 2006 at 05:17:39PM -0500, Lee Revell wrote: > It's been known for quite some time that the TSCs are not synced between > cores on Athlon X2 machines and this screws up the kernel's timekeeping, > as it still uses the TSC as the default time source on these machines. > > This problem still seems to be present in the latest kernels. What is > the plan to fix it? Is the fix simply to make the kernel use the ACPI > PM timer by default on Athlon X2? If your BIOS has an ACPI "HPET" table, then the kernel can use the HPET timer instead. It doesn't solve the problem for apps or other kernel bits that use rdtsc directly, but there are other fixes for that floating around which will hopefully get consideration when they're mature. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-13 18:06 ` thockin @ 2006-01-13 17:58 ` Lee Revell 2006-01-13 18:16 ` thockin 0 siblings, 1 reply; 62+ messages in thread From: Lee Revell @ 2006-01-13 17:58 UTC (permalink / raw) To: thockin; +Cc: linux-kernel On Fri, 2006-01-13 at 10:06 -0800, thockin@hockin.org wrote: > On Thu, Jan 12, 2006 at 05:17:39PM -0500, Lee Revell wrote: > > It's been known for quite some time that the TSCs are not synced between > > cores on Athlon X2 machines and this screws up the kernel's timekeeping, > > as it still uses the TSC as the default time source on these machines. > > > > This problem still seems to be present in the latest kernels. What is > > the plan to fix it? Is the fix simply to make the kernel use the ACPI > > PM timer by default on Athlon X2? > > If your BIOS has an ACPI "HPET" table, then the kernel can use the HPET > timer instead. It doesn't solve the problem for apps or other kernel bits > that use rdtsc directly, but there are other fixes for that floating > around which will hopefully get consideration when they're mature. > The apps can be converted to use clock_gettime(), but that does not help if the kernel can't keep reasonable time on these machines. But if we just use "clock=pmtmr" by default on these machines the TSC is not a problem right? Lee ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-13 17:58 ` Lee Revell @ 2006-01-13 18:16 ` thockin 2006-01-13 18:09 ` Lee Revell 0 siblings, 1 reply; 62+ messages in thread From: thockin @ 2006-01-13 18:16 UTC (permalink / raw) To: Lee Revell; +Cc: linux-kernel On Fri, Jan 13, 2006 at 12:58:36PM -0500, Lee Revell wrote: > > If your BIOS has an ACPI "HPET" table, then the kernel can use the HPET > > timer instead. It doesn't solve the problem for apps or other kernel bits > > that use rdtsc directly, but there are other fixes for that floating > > around which will hopefully get consideration when they're mature. > > The apps can be converted to use clock_gettime(), but that does not help > if the kernel can't keep reasonable time on these machines. Some apps/users need higher resolution and lower overhead that only rdtsc can offer currently. > But if we just use "clock=pmtmr" by default on these machines the TSC is > not a problem right? I never tried it with pmtimer, we had HPET available. Empirically TSC did not work (and we had a simple test case to show how bad it could get). HPET made that go away for users of gettimeofday(). We're exploring rdtsc-compatible solutions. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-13 18:16 ` thockin @ 2006-01-13 18:09 ` Lee Revell 2006-01-13 18:55 ` thockin 0 siblings, 1 reply; 62+ messages in thread From: Lee Revell @ 2006-01-13 18:09 UTC (permalink / raw) To: thockin; +Cc: linux-kernel On Fri, 2006-01-13 at 10:16 -0800, thockin@hockin.org wrote: > On Fri, Jan 13, 2006 at 12:58:36PM -0500, Lee Revell wrote: > > > If your BIOS has an ACPI "HPET" table, then the kernel can use the HPET > > > timer instead. It doesn't solve the problem for apps or other kernel bits > > > that use rdtsc directly, but there are other fixes for that floating > > > around which will hopefully get consideration when they're mature. > > > > The apps can be converted to use clock_gettime(), but that does not help > > if the kernel can't keep reasonable time on these machines. > > Some apps/users need higher resolution and lower overhead that only rdtsc > can offer currently. > But obviously if the TSC gives wildly inaccurate results, it cannot be used no matter how low the overhead. > > But if we just use "clock=pmtmr" by default on these machines the TSC is > > not a problem right? > > I never tried it with pmtimer, we had HPET available. Empirically TSC did > not work (and we had a simple test case to show how bad it could get). > HPET made that go away for users of gettimeofday(). > > We're exploring rdtsc-compatible solutions. > Since timekeeping on these machines has always been completely broken, shouldn't the default time source have been changed to the PM timer or HPET as soon as the problem was known? Lee ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-13 18:09 ` Lee Revell @ 2006-01-13 18:55 ` thockin 2006-01-13 18:56 ` Sven-Thorsten Dietrich 0 siblings, 1 reply; 62+ messages in thread From: thockin @ 2006-01-13 18:55 UTC (permalink / raw) To: Lee Revell; +Cc: linux-kernel On Fri, Jan 13, 2006 at 01:09:51PM -0500, Lee Revell wrote: > > Some apps/users need higher resolution and lower overhead that only rdtsc > > can offer currently. > > But obviously if the TSC gives wildly inaccurate results, it cannot be > used no matter how low the overhead. unless we can re-sync the TSCs often enough that apps don't notice. > > I never tried it with pmtimer, we had HPET available. Empirically TSC did > > not work (and we had a simple test case to show how bad it could get). > > HPET made that go away for users of gettimeofday(). > > > > We're exploring rdtsc-compatible solutions. > > > > Since timekeeping on these machines has always been completely broken, > shouldn't the default time source have been changed to the PM timer or > HPET as soon as the problem was known? If you have HPET, the kernel will prefer that. I don't know enough about the PMtimer overhead or resolution to say for sure. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-13 18:55 ` thockin @ 2006-01-13 18:56 ` Sven-Thorsten Dietrich 2006-01-13 19:00 ` Lee Revell ` (2 more replies) 0 siblings, 3 replies; 62+ messages in thread From: Sven-Thorsten Dietrich @ 2006-01-13 18:56 UTC (permalink / raw) To: thockin; +Cc: Lee Revell, linux-kernel On Fri, 2006-01-13 at 10:55 -0800, thockin@hockin.org wrote: > On Fri, Jan 13, 2006 at 01:09:51PM -0500, Lee Revell wrote: > > > Some apps/users need higher resolution and lower overhead that only rdtsc > > > can offer currently. > > > > But obviously if the TSC gives wildly inaccurate results, it cannot be > > used no matter how low the overhead. > > unless we can re-sync the TSCs often enough that apps don't notice. > You'd have to quantify that somehow, in terms of the max drift rate (ppm), and the max resolution available (< tsc frequency). Either that, or track an offset, and use one TSC as truth, and update the correction factor for the other TSCs as often as needed, maybe? This is kind of analogous to the "drift" NTP calculates against a free-running oscillator. So you'd be pushing that functionality deeper into the OS-core. Dave Mills had that "hardpps" stuff in there for a while, it might be a starting point. Just some thoughts for now... Sven > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ -- *********************************** Sven-Thorsten Dietrich Real-Time Software Architect MontaVista Software, Inc. 1237 East Arques Ave. Sunnyvale, CA 94085 Phone: 408.992.4515 Fax: 408.328.9204 http://www.mvista.com Platform To Innovate *********************************** ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-13 18:56 ` Sven-Thorsten Dietrich @ 2006-01-13 19:00 ` Lee Revell 2006-01-13 19:07 ` Sven-Thorsten Dietrich 2006-01-13 21:18 ` David Lang 2006-01-13 19:13 ` Xavier Bestel 2006-01-13 19:32 ` thockin 2 siblings, 2 replies; 62+ messages in thread From: Lee Revell @ 2006-01-13 19:00 UTC (permalink / raw) To: Sven-Thorsten Dietrich; +Cc: thockin, linux-kernel On Fri, 2006-01-13 at 10:56 -0800, Sven-Thorsten Dietrich wrote: > On Fri, 2006-01-13 at 10:55 -0800, thockin@hockin.org wrote: > > On Fri, Jan 13, 2006 at 01:09:51PM -0500, Lee Revell wrote: > > > > Some apps/users need higher resolution and lower overhead that only rdtsc > > > > can offer currently. > > > > > > But obviously if the TSC gives wildly inaccurate results, it cannot be > > > used no matter how low the overhead. > > > > unless we can re-sync the TSCs often enough that apps don't notice. > > > > You'd have to quantify that somehow, in terms of the max drift rate > (ppm), and the max resolution available (< tsc frequency). > > Either that, or track an offset, and use one TSC as truth, and update > the correction factor for the other TSCs as often as needed, maybe? > > This is kind of analogous to the "drift" NTP calculates against a > free-running oscillator. > > So you'd be pushing that functionality deeper into the OS-core. > > Dave Mills had that "hardpps" stuff in there for a while, it might be a > starting point. > > Just some thoughts for now... > It kind of makes you wonder what in the heck AMD were thinking, whether they realized that this design decision would cause so many problems at the OS level (it's broken at least Linux and Solaris). Maybe Windows keeps time in a way that was unaffected by this? Lee ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-13 19:00 ` Lee Revell @ 2006-01-13 19:07 ` Sven-Thorsten Dietrich 2006-01-13 21:18 ` David Lang 1 sibling, 0 replies; 62+ messages in thread From: Sven-Thorsten Dietrich @ 2006-01-13 19:07 UTC (permalink / raw) To: Lee Revell; +Cc: thockin, linux-kernel On Fri, 2006-01-13 at 14:00 -0500, Lee Revell wrote: > On Fri, 2006-01-13 at 10:56 -0800, Sven-Thorsten Dietrich wrote: > > On Fri, 2006-01-13 at 10:55 -0800, thockin@hockin.org wrote: > > > On Fri, Jan 13, 2006 at 01:09:51PM -0500, Lee Revell wrote: > > > > > Some apps/users need higher resolution and lower overhead that only rdtsc > > > > > can offer currently. > > > > > > > > But obviously if the TSC gives wildly inaccurate results, it cannot be > > > > used no matter how low the overhead. > > > > > > unless we can re-sync the TSCs often enough that apps don't notice. > > > > > > > You'd have to quantify that somehow, in terms of the max drift rate > > (ppm), and the max resolution available (< tsc frequency). > > > > Either that, or track an offset, and use one TSC as truth, and update > > the correction factor for the other TSCs as often as needed, maybe? > > > > This is kind of analogous to the "drift" NTP calculates against a > > free-running oscillator. > > > > So you'd be pushing that functionality deeper into the OS-core. > > > > Dave Mills had that "hardpps" stuff in there for a while, it might be a > > starting point. > > > > Just some thoughts for now... > > > > It kind of makes you wonder what in the heck AMD were thinking, whether > they realized that this design decision would cause so many problems at > the OS level (it's broken at least Linux and Solaris). Maybe Windows > keeps time in a way that was unaffected by this? > Last time I looked at Windows, we had to create an RT task to interpolate between jiffies, b/c Windows was only keeping time at the 10 / 16 (smp) ms resolution. (to get NTP to run) That was 6 years ago. I haven't looked back since, and may I turn to salt or stone if I ever do. > Lee > -- *********************************** Sven-Thorsten Dietrich Real-Time Software Architect MontaVista Software, Inc. 1237 East Arques Ave. Sunnyvale, CA 94085 Phone: 408.992.4515 Fax: 408.328.9204 http://www.mvista.com Platform To Innovate *********************************** ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-13 19:00 ` Lee Revell 2006-01-13 19:07 ` Sven-Thorsten Dietrich @ 2006-01-13 21:18 ` David Lang 2006-01-13 21:56 ` thockin 1 sibling, 1 reply; 62+ messages in thread From: David Lang @ 2006-01-13 21:18 UTC (permalink / raw) To: Lee Revell; +Cc: Sven-Thorsten Dietrich, thockin, linux-kernel On Fri, 13 Jan 2006, Lee Revell wrote: > On Fri, 2006-01-13 at 10:56 -0800, Sven-Thorsten Dietrich wrote: >> On Fri, 2006-01-13 at 10:55 -0800, thockin@hockin.org wrote: >>> On Fri, Jan 13, 2006 at 01:09:51PM -0500, Lee Revell wrote: >>>>> Some apps/users need higher resolution and lower overhead that only rdtsc >>>>> can offer currently. >>>> >>>> But obviously if the TSC gives wildly inaccurate results, it cannot be >>>> used no matter how low the overhead. >>> >>> unless we can re-sync the TSCs often enough that apps don't notice. >>> >> >> You'd have to quantify that somehow, in terms of the max drift rate >> (ppm), and the max resolution available (< tsc frequency). >> >> Either that, or track an offset, and use one TSC as truth, and update >> the correction factor for the other TSCs as often as needed, maybe? >> >> This is kind of analogous to the "drift" NTP calculates against a >> free-running oscillator. >> >> So you'd be pushing that functionality deeper into the OS-core. >> >> Dave Mills had that "hardpps" stuff in there for a while, it might be a >> starting point. >> >> Just some thoughts for now... >> > > It kind of makes you wonder what in the heck AMD were thinking, whether > they realized that this design decision would cause so many problems at > the OS level (it's broken at least Linux and Solaris). Maybe Windows > keeps time in a way that was unaffected by this? Lee, the last time I saw this discussion I thought it was identified that the multiple cores (or IIRC the multiple chips in a SMP motherboard) would only get out of sync when power management calls were made (hlt or switching the c-state). IIRC the workaround that was posted then was to just disable these in the kernel build. if this is the case then I could easily see the hardware engineers thinking that if the software did a sleep thing then it was up to the software to re-sync the TSC clocks when the wakeup takes place. David Lang -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-13 21:18 ` David Lang @ 2006-01-13 21:56 ` thockin 2006-01-13 22:05 ` David Lang 0 siblings, 1 reply; 62+ messages in thread From: thockin @ 2006-01-13 21:56 UTC (permalink / raw) To: David Lang; +Cc: Lee Revell, Sven-Thorsten Dietrich, linux-kernel On Fri, Jan 13, 2006 at 01:18:35PM -0800, David Lang wrote: > Lee, the last time I saw this discussion I thought it was identified that > the multiple cores (or IIRC the multiple chips in a SMP motherboard) would > only get out of sync when power management calls were made (hlt or > switching the c-state). IIRC the workaround that was posted then was to > just disable these in the kernel build. not using 'hlt' when idling means that you spend 10s of Watts more power on mostly idle systems. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-13 21:56 ` thockin @ 2006-01-13 22:05 ` David Lang 2006-01-13 22:18 ` Sven-Thorsten Dietrich 2006-01-13 22:23 ` thockin 0 siblings, 2 replies; 62+ messages in thread From: David Lang @ 2006-01-13 22:05 UTC (permalink / raw) To: thockin; +Cc: Lee Revell, Sven-Thorsten Dietrich, linux-kernel On Fri, 13 Jan 2006 thockin@hockin.org wrote: > On Fri, Jan 13, 2006 at 01:18:35PM -0800, David Lang wrote: >> Lee, the last time I saw this discussion I thought it was identified that >> the multiple cores (or IIRC the multiple chips in a SMP motherboard) would >> only get out of sync when power management calls were made (hlt or >> switching the c-state). IIRC the workaround that was posted then was to >> just disable these in the kernel build. > > not using 'hlt' when idling means that you spend 10s of Watts more power > on mostly idle systems. true, but for people who need better time accruacy then the other workaround this may be very acceptable. David Lang -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-13 22:05 ` David Lang @ 2006-01-13 22:18 ` Sven-Thorsten Dietrich 2006-01-13 22:49 ` David Lang 2006-01-13 22:23 ` thockin 1 sibling, 1 reply; 62+ messages in thread From: Sven-Thorsten Dietrich @ 2006-01-13 22:18 UTC (permalink / raw) To: David Lang; +Cc: thockin, Lee Revell, linux-kernel On Fri, 2006-01-13 at 14:05 -0800, David Lang wrote: > On Fri, 13 Jan 2006 thockin@hockin.org wrote: > > > On Fri, Jan 13, 2006 at 01:18:35PM -0800, David Lang wrote: > >> Lee, the last time I saw this discussion I thought it was identified that > >> the multiple cores (or IIRC the multiple chips in a SMP motherboard) would > >> only get out of sync when power management calls were made (hlt or > >> switching the c-state). IIRC the workaround that was posted then was to > >> just disable these in the kernel build. > > > > not using 'hlt' when idling means that you spend 10s of Watts more power > > on mostly idle systems. > > true, but for people who need better time accruacy then the other > workaround this may be very acceptable. > 1/4 KW / day for time synchronisation. The power company would love that. > David Lang > -- *********************************** Sven-Thorsten Dietrich Real-Time Software Architect MontaVista Software, Inc. 1237 East Arques Ave. Sunnyvale, CA 94085 Phone: 408.992.4515 Fax: 408.328.9204 http://www.mvista.com Platform To Innovate *********************************** ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-13 22:18 ` Sven-Thorsten Dietrich @ 2006-01-13 22:49 ` David Lang 2006-01-14 0:41 ` Andreas Steinmetz 0 siblings, 1 reply; 62+ messages in thread From: David Lang @ 2006-01-13 22:49 UTC (permalink / raw) To: Sven-Thorsten Dietrich; +Cc: thockin, Lee Revell, linux-kernel On Fri, 13 Jan 2006, Sven-Thorsten Dietrich wrote: > On Fri, 2006-01-13 at 14:05 -0800, David Lang wrote: >> On Fri, 13 Jan 2006 thockin@hockin.org wrote: >> >>> On Fri, Jan 13, 2006 at 01:18:35PM -0800, David Lang wrote: >>>> Lee, the last time I saw this discussion I thought it was identified that >>>> the multiple cores (or IIRC the multiple chips in a SMP motherboard) would >>>> only get out of sync when power management calls were made (hlt or >>>> switching the c-state). IIRC the workaround that was posted then was to >>>> just disable these in the kernel build. >>> >>> not using 'hlt' when idling means that you spend 10s of Watts more power >>> on mostly idle systems. >> >> true, but for people who need better time accruacy then the other >> workaround this may be very acceptable. >> > > 1/4 KW / day for time synchronisation. > > The power company would love that. more precisely 1/4 KW Hour / day $0.01 - $0.02/day (I had to lookup the current rates) they probably won't notice. David Lang -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-13 22:49 ` David Lang @ 2006-01-14 0:41 ` Andreas Steinmetz 2006-01-14 1:04 ` David Lang 2006-01-14 1:13 ` thockin 0 siblings, 2 replies; 62+ messages in thread From: Andreas Steinmetz @ 2006-01-14 0:41 UTC (permalink / raw) To: David Lang; +Cc: Sven-Thorsten Dietrich, thockin, Lee Revell, linux-kernel David Lang wrote: > On Fri, 13 Jan 2006, Sven-Thorsten Dietrich wrote: > >> On Fri, 2006-01-13 at 14:05 -0800, David Lang wrote: >> >>> On Fri, 13 Jan 2006 thockin@hockin.org wrote: >>> >>>> On Fri, Jan 13, 2006 at 01:18:35PM -0800, David Lang wrote: >>>> >>>>> Lee, the last time I saw this discussion I thought it was >>>>> identified that >>>>> the multiple cores (or IIRC the multiple chips in a SMP >>>>> motherboard) would >>>>> only get out of sync when power management calls were made (hlt or >>>>> switching the c-state). IIRC the workaround that was posted then >>>>> was to >>>>> just disable these in the kernel build. >>>> >>>> >>>> not using 'hlt' when idling means that you spend 10s of Watts more >>>> power >>>> on mostly idle systems. >>> >>> >>> true, but for people who need better time accruacy then the other >>> workaround this may be very acceptable. >>> >> >> 1/4 KW / day for time synchronisation. >> >> The power company would love that. > > > more precisely 1/4 KW Hour / day > > $0.01 - $0.02/day (I had to lookup the current rates) > > they probably won't notice. > Well, wait until there's AMD based dual core x86_64 laptops out there (this email being written on a single core x86_64 one). I can already see the faces of the unhappy future owners being told "use idle=poll" when on battery and anyway going deaf by fan noise. (/me ducks and runs) -- Andreas Steinmetz SPAMmers use robotrap@domdv.de ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-14 0:41 ` Andreas Steinmetz @ 2006-01-14 1:04 ` David Lang 2006-01-14 1:21 ` Andreas Steinmetz 2006-01-15 8:52 ` Zan Lynx 2006-01-14 1:13 ` thockin 1 sibling, 2 replies; 62+ messages in thread From: David Lang @ 2006-01-14 1:04 UTC (permalink / raw) To: Andreas Steinmetz Cc: Sven-Thorsten Dietrich, thockin, Lee Revell, linux-kernel On Sat, 14 Jan 2006, Andreas Steinmetz wrote: > David Lang wrote: >> On Fri, 13 Jan 2006, Sven-Thorsten Dietrich wrote: >> >>> On Fri, 2006-01-13 at 14:05 -0800, David Lang wrote: >>> >>>> On Fri, 13 Jan 2006 thockin@hockin.org wrote: >>>> >>>>> On Fri, Jan 13, 2006 at 01:18:35PM -0800, David Lang wrote: >>>>> >>>>>> Lee, the last time I saw this discussion I thought it was >>>>>> identified that >>>>>> the multiple cores (or IIRC the multiple chips in a SMP >>>>>> motherboard) would >>>>>> only get out of sync when power management calls were made (hlt or >>>>>> switching the c-state). IIRC the workaround that was posted then >>>>>> was to >>>>>> just disable these in the kernel build. >>>>> >>>>> >>>>> not using 'hlt' when idling means that you spend 10s of Watts more >>>>> power >>>>> on mostly idle systems. >>>> >>>> >>>> true, but for people who need better time accruacy then the other >>>> workaround this may be very acceptable. >>>> >>> >>> 1/4 KW / day for time synchronisation. >>> >>> The power company would love that. >> >> >> more precisely 1/4 KW Hour / day >> >> $0.01 - $0.02/day (I had to lookup the current rates) >> >> they probably won't notice. >> > > Well, wait until there's AMD based dual core x86_64 laptops out there > (this email being written on a single core x86_64 one). I can already > see the faces of the unhappy future owners being told "use idle=poll" > when on battery and anyway going deaf by fan noise. > > (/me ducks and runs) I'm not saying it's the right answer, but it's one of two workarounds currently available. idle=poll causes increased power useage timer source change (mentioned earlier in this thread) limits timer precision neither of these are fixes, but by understanding the different costs people can choose the work around they want to use while waiting for a better fix. David Lang -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-14 1:04 ` David Lang @ 2006-01-14 1:21 ` Andreas Steinmetz 2006-01-14 1:51 ` thockin 2006-01-15 8:52 ` Zan Lynx 1 sibling, 1 reply; 62+ messages in thread From: Andreas Steinmetz @ 2006-01-14 1:21 UTC (permalink / raw) To: David Lang; +Cc: Sven-Thorsten Dietrich, thockin, Lee Revell, linux-kernel David Lang wrote: [snip] > I'm not saying it's the right answer, but it's one of two workarounds > currently available. > > idle=poll causes increased power useage > > timer source change (mentioned earlier in this thread) limits timer > precision > > neither of these are fixes, but by understanding the different costs > people can choose the work around they want to use while waiting for a > better fix. The problem being that some of us use their laptops for audio work too. And then high battery usage, noisy fans or lack of high res timers will be really bad. Simple example: I do final mastering work using my laptop and Ardour/Jack/JAMin out of house in a place with a good stage audio system (Mackie mixer, 2KW Dynacord Amp/Syrincs S3) and a sufficient listening space to get a proper bass mix. I run on battery in this case to avoid any kind of audio interference (ground loops, etc). Now thinking of a dual core laptop... -- Andreas Steinmetz SPAMmers use robotrap@domdv.de ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-14 1:21 ` Andreas Steinmetz @ 2006-01-14 1:51 ` thockin 0 siblings, 0 replies; 62+ messages in thread From: thockin @ 2006-01-14 1:51 UTC (permalink / raw) To: Andreas Steinmetz Cc: David Lang, Sven-Thorsten Dietrich, Lee Revell, linux-kernel On Sat, Jan 14, 2006 at 02:21:23AM +0100, Andreas Steinmetz wrote: > > neither of these are fixes, but by understanding the different costs > > people can choose the work around they want to use while waiting for a > > better fix. > > The problem being that some of us use their laptops for audio work too. > And then high battery usage, noisy fans or lack of high res timers will > be really bad. > > Simple example: > I do final mastering work using my laptop and Ardour/Jack/JAMin out of > house in a place with a good stage audio system (Mackie mixer, 2KW > Dynacord Amp/Syrincs S3) and a sufficient listening space to get a > proper bass mix. I run on battery in this case to avoid any kind of > audio interference (ground loops, etc). Now thinking of a dual core > laptop... Fixes are coming, just need to sort out the fidlly bits. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-14 1:04 ` David Lang 2006-01-14 1:21 ` Andreas Steinmetz @ 2006-01-15 8:52 ` Zan Lynx 2006-01-15 16:25 ` thockin 1 sibling, 1 reply; 62+ messages in thread From: Zan Lynx @ 2006-01-15 8:52 UTC (permalink / raw) To: David Lang Cc: Andreas Steinmetz, Sven-Thorsten Dietrich, thockin, Lee Revell, linux-kernel [-- Attachment #1: Type: text/plain, Size: 1136 bytes --] On Fri, 2006-01-13 at 17:04 -0800, David Lang wrote: > On Sat, 14 Jan 2006, Andreas Steinmetz wrote: > > > David Lang wrote: > > Well, wait until there's AMD based dual core x86_64 laptops out there > > (this email being written on a single core x86_64 one). I can already > > see the faces of the unhappy future owners being told "use idle=poll" > > when on battery and anyway going deaf by fan noise. > > > > (/me ducks and runs) > > I'm not saying it's the right answer, but it's one of two workarounds > currently available. > > idle=poll causes increased power useage > > timer source change (mentioned earlier in this thread) limits timer > precision > > neither of these are fixes, but by understanding the different costs > people can choose the work around they want to use while waiting for a > better fix. A laptop user could also bind a process to a single CPU, and use the scaling min/max values to lock CPU speed to a single value. The TSC may still stop during HLT, but software must be handling that already. Wouldn't that provide an accurate TSC? -- Zan Lynx <zlynx@acm.org> [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-15 8:52 ` Zan Lynx @ 2006-01-15 16:25 ` thockin 2006-01-15 16:33 ` Lee Revell 0 siblings, 1 reply; 62+ messages in thread From: thockin @ 2006-01-15 16:25 UTC (permalink / raw) To: Zan Lynx Cc: David Lang, Andreas Steinmetz, Sven-Thorsten Dietrich, Lee Revell, linux-kernel On Sun, Jan 15, 2006 at 01:52:44AM -0700, Zan Lynx wrote: > A laptop user could also bind a process to a single CPU, and use the > scaling min/max values to lock CPU speed to a single value. The TSC may > still stop during HLT, but software must be handling that already. > > Wouldn't that provide an accurate TSC? monotonic but not linear. Also remember that the OS will use rdtsc here and there, and you can't affine the OS :) ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-15 16:25 ` thockin @ 2006-01-15 16:33 ` Lee Revell 2006-01-15 18:21 ` thockin 0 siblings, 1 reply; 62+ messages in thread From: Lee Revell @ 2006-01-15 16:33 UTC (permalink / raw) To: thockin Cc: Zan Lynx, David Lang, Andreas Steinmetz, Sven-Thorsten Dietrich, linux-kernel On Sun, 2006-01-15 at 08:25 -0800, thockin@hockin.org wrote: > On Sun, Jan 15, 2006 at 01:52:44AM -0700, Zan Lynx wrote: > > A laptop user could also bind a process to a single CPU, and use the > > scaling min/max values to lock CPU speed to a single value. The TSC may > > still stop during HLT, but software must be handling that already. > > > > Wouldn't that provide an accurate TSC? > > monotonic but not linear. Also remember that the OS will use rdtsc here > and there, and you can't affine the OS :) > So the options are either to fix the TSC handling on these systems (by resyncing the TSCs when exiting from HLT), or eliminate the use of rdtsc by the OS? Lee ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-15 16:33 ` Lee Revell @ 2006-01-15 18:21 ` thockin 2006-01-15 18:29 ` Lee Revell 0 siblings, 1 reply; 62+ messages in thread From: thockin @ 2006-01-15 18:21 UTC (permalink / raw) To: Lee Revell Cc: Zan Lynx, David Lang, Andreas Steinmetz, Sven-Thorsten Dietrich, linux-kernel On Sun, Jan 15, 2006 at 11:33:36AM -0500, Lee Revell wrote: > On Sun, 2006-01-15 at 08:25 -0800, thockin@hockin.org wrote: > > On Sun, Jan 15, 2006 at 01:52:44AM -0700, Zan Lynx wrote: > > > A laptop user could also bind a process to a single CPU, and use the > > > scaling min/max values to lock CPU speed to a single value. The TSC may > > > still stop during HLT, but software must be handling that already. > > > > > > Wouldn't that provide an accurate TSC? > > > > monotonic but not linear. Also remember that the OS will use rdtsc here > > and there, and you can't affine the OS :) > > > > So the options are either to fix the TSC handling on these systems (by > resyncing the TSCs when exiting from HLT), or eliminate the use of rdtsc > by the OS? Or both. You can trap rdtsc users in userland, you have to manually audit kernel users. It should be abolished or properly wrapped in kernel code, as soon as a resync path is available. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-15 18:21 ` thockin @ 2006-01-15 18:29 ` Lee Revell 2006-01-15 18:49 ` thockin 0 siblings, 1 reply; 62+ messages in thread From: Lee Revell @ 2006-01-15 18:29 UTC (permalink / raw) To: thockin Cc: Zan Lynx, David Lang, Andreas Steinmetz, Sven-Thorsten Dietrich, linux-kernel On Sun, 2006-01-15 at 10:21 -0800, thockin@hockin.org wrote: > On Sun, Jan 15, 2006 at 11:33:36AM -0500, Lee Revell wrote: > > On Sun, 2006-01-15 at 08:25 -0800, thockin@hockin.org wrote: > > > On Sun, Jan 15, 2006 at 01:52:44AM -0700, Zan Lynx wrote: > > > > A laptop user could also bind a process to a single CPU, and use the > > > > scaling min/max values to lock CPU speed to a single value. The TSC may > > > > still stop during HLT, but software must be handling that already. > > > > > > > > Wouldn't that provide an accurate TSC? > > > > > > monotonic but not linear. Also remember that the OS will use rdtsc here > > > and there, and you can't affine the OS :) > > > > > > > So the options are either to fix the TSC handling on these systems (by > > resyncing the TSCs when exiting from HLT), or eliminate the use of rdtsc > > by the OS? > > Or both. You can trap rdtsc users in userland, you have to manually audit > kernel users. > > It should be abolished or properly wrapped in kernel code, as soon as a > resync path is available. > For the purposes of this discussion I was not considering direct use of rdtsc from userland for timing, I've accepted the arguments that this is a lost cause with today's hardware (although I maintain it was viable in practice on previous generations of hardware). Besides, rdtsc is useless from userspace if the kernel traps it, because the whole point of using it over gettimeofday() was that it was dirt cheap. I'm content to make sure the kernel's internal timekeeping is solid. Lee ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-15 18:29 ` Lee Revell @ 2006-01-15 18:49 ` thockin 0 siblings, 0 replies; 62+ messages in thread From: thockin @ 2006-01-15 18:49 UTC (permalink / raw) To: Lee Revell Cc: Zan Lynx, David Lang, Andreas Steinmetz, Sven-Thorsten Dietrich, linux-kernel On Sun, Jan 15, 2006 at 01:29:36PM -0500, Lee Revell wrote: > > Or both. You can trap rdtsc users in userland, you have to manually audit > > kernel users. > > > > It should be abolished or properly wrapped in kernel code, as soon as a > > resync path is available. > > For the purposes of this discussion I was not considering direct use of > rdtsc from userland for timing, I've accepted the arguments that this is > a lost cause with today's hardware (although I maintain it was viable in > practice on previous generations of hardware). Besides, rdtsc is > useless from userspace if the kernel traps it, because the whole point > of using it over gettimeofday() was that it was dirt cheap. > > I'm content to make sure the kernel's internal timekeeping is solid. You only need to trap rdtsc if you know that some unsyncronizing event has happened. E.g. 'hlt' or some other sleep state. If we sync every time we hlt, then we don't need to trap. But we can maybe be lazier about it. The sync operation itself may not be cheap, and it may be useful to skip it if possible. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-14 0:41 ` Andreas Steinmetz 2006-01-14 1:04 ` David Lang @ 2006-01-14 1:13 ` thockin 1 sibling, 0 replies; 62+ messages in thread From: thockin @ 2006-01-14 1:13 UTC (permalink / raw) To: Andreas Steinmetz Cc: David Lang, Sven-Thorsten Dietrich, Lee Revell, linux-kernel On Sat, Jan 14, 2006 at 01:41:43AM +0100, Andreas Steinmetz wrote: > > more precisely 1/4 KW Hour / day > > > > $0.01 - $0.02/day (I had to lookup the current rates) > > > > they probably won't notice. > > Well, wait until there's AMD based dual core x86_64 laptops out there > (this email being written on a single core x86_64 one). I can already > see the faces of the unhappy future owners being told "use idle=poll" > when on battery and anyway going deaf by fan noise. Yeah, and then multiply that cost by a few tens of thousands of servers. idle=poll is a hack at best. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-13 22:05 ` David Lang 2006-01-13 22:18 ` Sven-Thorsten Dietrich @ 2006-01-13 22:23 ` thockin 1 sibling, 0 replies; 62+ messages in thread From: thockin @ 2006-01-13 22:23 UTC (permalink / raw) To: David Lang; +Cc: Lee Revell, Sven-Thorsten Dietrich, linux-kernel On Fri, Jan 13, 2006 at 02:05:23PM -0800, David Lang wrote: > On Fri, 13 Jan 2006 thockin@hockin.org wrote: > > >On Fri, Jan 13, 2006 at 01:18:35PM -0800, David Lang wrote: > >>Lee, the last time I saw this discussion I thought it was identified that > >>the multiple cores (or IIRC the multiple chips in a SMP motherboard) would > >>only get out of sync when power management calls were made (hlt or > >>switching the c-state). IIRC the workaround that was posted then was to > >>just disable these in the kernel build. > > > >not using 'hlt' when idling means that you spend 10s of Watts more power > >on mostly idle systems. > > true, but for people who need better time accruacy then the other > workaround this may be very acceptable. sure, if power doesn't matter use idle=poll => problem solved. ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-13 18:56 ` Sven-Thorsten Dietrich 2006-01-13 19:00 ` Lee Revell @ 2006-01-13 19:13 ` Xavier Bestel 2006-01-13 19:32 ` thockin 2 siblings, 0 replies; 62+ messages in thread From: Xavier Bestel @ 2006-01-13 19:13 UTC (permalink / raw) To: Sven-Thorsten Dietrich; +Cc: thockin, Lee Revell, linux-kernel Le vendredi 13 janvier 2006 à 10:56 -0800, Sven-Thorsten Dietrich a écrit : > On Fri, 2006-01-13 at 10:55 -0800, thockin@hockin.org wrote: > > On Fri, Jan 13, 2006 at 01:09:51PM -0500, Lee Revell wrote: > > > > Some apps/users need higher resolution and lower overhead that only rdtsc > > > > can offer currently. > > > > > > But obviously if the TSC gives wildly inaccurate results, it cannot be > > > used no matter how low the overhead. > > > > unless we can re-sync the TSCs often enough that apps don't notice. > > > > You'd have to quantify that somehow, in terms of the max drift rate > (ppm), and the max resolution available (< tsc frequency). > > Either that, or track an offset, and use one TSC as truth, and update > the correction factor for the other TSCs as often as needed, maybe? As often as needed being each time a thread changes CPU ? ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-13 18:56 ` Sven-Thorsten Dietrich 2006-01-13 19:00 ` Lee Revell 2006-01-13 19:13 ` Xavier Bestel @ 2006-01-13 19:32 ` thockin 2006-01-13 20:58 ` Steven Rostedt 2 siblings, 1 reply; 62+ messages in thread From: thockin @ 2006-01-13 19:32 UTC (permalink / raw) To: Sven-Thorsten Dietrich; +Cc: Lee Revell, linux-kernel On Fri, Jan 13, 2006 at 10:56:13AM -0800, Sven-Thorsten Dietrich wrote: > On Fri, 2006-01-13 at 10:55 -0800, thockin@hockin.org wrote: > > unless we can re-sync the TSCs often enough that apps don't notice. > > You'd have to quantify that somehow, in terms of the max drift rate > (ppm), and the max resolution available (< tsc frequency). > > Either that, or track an offset, and use one TSC as truth, and update > the correction factor for the other TSCs as often as needed, maybe? > > This is kind of analogous to the "drift" NTP calculates against a > free-running oscillator. > > So you'd be pushing that functionality deeper into the OS-core. > > Dave Mills had that "hardpps" stuff in there for a while, it might be a > starting point. > > Just some thoughts for now... There's some chatter here about a solution involving a lazy sync of TSCs to the HPET (or other) whenever an app calls rdtsc after a potentially unsyncing event. For example, 'hlt' will initiate C1 which may cause clock ramping (and TSC skew). We can trap rdtsc after a hlt and re-sync the TSCs to some truly monotonic source, like HPET. I don't have all the details, some problems remain, and the work is not quite done yet, but it looks promising. Even if we eventually get synced TSCs, it's too little too late. basically, anything in-kernel that uses rdtsc is bound to break, and any app that uses rdtsc had better be using CPU affinity. Tim ^ permalink raw reply [flat|nested] 62+ messages in thread
* Re: Dual core Athlons and unsynced TSCs 2006-01-13 19:32 ` thockin @ 2006-01-13 20:58 ` Steven Rostedt 0 siblings, 0 replies; 62+ messages in thread From: Steven Rostedt @ 2006-01-13 20:58 UTC (permalink / raw) To: thockin; +Cc: linux-kernel, Lee Revell, Sven-Thorsten Dietrich On Fri, 2006-01-13 at 11:32 -0800, thockin@hockin.org wrote: > On Fri, Jan 13, 2006 at 10:56:13AM -0800, Sven-Thorsten Dietrich wrote: > > On Fri, 2006-01-13 at 10:55 -0800, thockin@hockin.org wrote: > > > unless we can re-sync the TSCs often enough that apps don't notice. > > > > You'd have to quantify that somehow, in terms of the max drift rate > > (ppm), and the max resolution available (< tsc frequency). > > > > Either that, or track an offset, and use one TSC as truth, and update > > the correction factor for the other TSCs as often as needed, maybe? > > > > This is kind of analogous to the "drift" NTP calculates against a > > free-running oscillator. > > > > So you'd be pushing that functionality deeper into the OS-core. > > > > Dave Mills had that "hardpps" stuff in there for a while, it might be a > > starting point. > > > > Just some thoughts for now... > > There's some chatter here about a solution involving a lazy sync of TSCs > to the HPET (or other) whenever an app calls rdtsc after a potentially > unsyncing event. > > For example, 'hlt' will initiate C1 which may cause clock ramping (and TSC > skew). We can trap rdtsc after a hlt and re-sync the TSCs to some truly > monotonic source, like HPET. > > I don't have all the details, some problems remain, and the work is not > quite done yet, but it looks promising. > > Even if we eventually get synced TSCs, it's too little too late. > basically, anything in-kernel that uses rdtsc is bound to break, and any > app that uses rdtsc had better be using CPU affinity. I know this might be somewhat of an overhead, but what about resyncing on wakeup of the idle function? Isn't that where they fall apart? -- Steve ^ permalink raw reply [flat|nested] 62+ messages in thread
[parent not found: <5ujmj-1UQ-511@gated-at.bofh.it>]
[parent not found: <5uBnF-2SG-11@gated-at.bofh.it>]
[parent not found: <5uBnF-2SG-9@gated-at.bofh.it>]
[parent not found: <5uBxi-3iM-21@gated-at.bofh.it>]
[parent not found: <5uBGY-3ul-21@gated-at.bofh.it>]
[parent not found: <5uCa5-443-45@gated-at.bofh.it>]
[parent not found: <5uCjF-4fW-15@gated-at.bofh.it>]
[parent not found: <5uCtj-4Fi-15@gated-at.bofh.it>]
* Re: Dual core Athlons and unsynced TSCs [not found] ` <5uCtj-4Fi-15@gated-at.bofh.it> @ 2006-01-14 0:24 ` Robert Hancock 0 siblings, 0 replies; 62+ messages in thread From: Robert Hancock @ 2006-01-14 0:24 UTC (permalink / raw) To: linux-kernel Lee Revell wrote: >>>>But obviously if the TSC gives wildly inaccurate results, it cannot be >>>>used no matter how low the overhead. >>> >>>unless we can re-sync the TSCs often enough that apps don't notice. >>> >> >>You'd have to quantify that somehow, in terms of the max drift rate >>(ppm), and the max resolution available (< tsc frequency). >> >>Either that, or track an offset, and use one TSC as truth, and update >>the correction factor for the other TSCs as often as needed, maybe? >> >>This is kind of analogous to the "drift" NTP calculates against a >>free-running oscillator. >> >>So you'd be pushing that functionality deeper into the OS-core. >> >>Dave Mills had that "hardpps" stuff in there for a while, it might be a >>starting point. >> >>Just some thoughts for now... > > It kind of makes you wonder what in the heck AMD were thinking, whether > they realized that this design decision would cause so many problems at > the OS level (it's broken at least Linux and Solaris). Maybe Windows > keeps time in a way that was unaffected by this? Sounds to me like they are doing something like what was being mentioned above: http://support.microsoft.com/kb/896256/en-us "When TSC does not increment monotonically, system components that use the kernel KeQueryPerformanceCounter function may not work correctly. To address this problem, Microsoft makes it possible for the ACPI Power Management Timer to be used as the operating system timer that supports the kernel KeQueryPerformanceCounter function. However, some programs may directly access the TSC by bypassing the Windows timer APIs. The multiple-processor Hardware Abstraction Layer (HAL) makes sure that the TSC registers on all processors on a multiple-processor computer remain closely synchronized. Therefore, access by system software that may be directed to different processors does not return different results." Also, Microsoft's docs for QueryPerformanceCounter specify that different results on different CPUs will only occur if there are "bugs in the basic input/output system (BIOS) or the hardware abstraction layer (HAL)" and recommends that threads using this function set their affinity to run on one processor only. -- Robert Hancock Saskatoon, SK, Canada To email, remove "nospam" from hancockr@nospamshaw.ca Home Page: http://www.roberthancock.com/ ^ permalink raw reply [flat|nested] 62+ messages in thread
end of thread, other threads:[~2006-01-16 15:56 UTC | newest] Thread overview: 62+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2006-01-12 22:17 Dual core Athlons and unsynced TSCs Lee Revell 2006-01-13 15:10 ` Roger Heflin 2006-01-13 16:04 ` Steven Rostedt 2006-01-13 17:47 ` Lee Revell 2006-01-13 17:52 ` Steven Rostedt 2006-01-13 17:54 ` Lee Revell 2006-01-13 18:41 ` Steven Rostedt 2006-01-13 18:55 ` Lee Revell 2006-01-13 20:09 ` Steven Rostedt 2006-01-13 20:26 ` Steven Rostedt 2006-01-13 20:40 ` Lee Revell 2006-01-13 20:43 ` Lee Revell 2006-01-13 20:46 ` Thomas Gleixner 2006-01-13 20:48 ` Lee Revell 2006-01-13 20:51 ` Thomas Gleixner 2006-01-13 20:52 ` Steven Rostedt 2006-01-13 21:05 ` Steven Rostedt 2006-01-13 21:10 ` Thomas Gleixner 2006-01-13 21:17 ` Steven Rostedt 2006-01-16 9:52 ` Thomas Gleixner 2006-01-16 15:55 ` Steven Rostedt 2006-01-14 0:23 ` john stultz 2006-01-14 1:10 ` Steven Rostedt 2006-01-14 1:14 ` Lee Revell 2006-01-14 1:23 ` john stultz 2006-01-14 1:27 ` Lee Revell 2006-01-14 1:39 ` john stultz 2006-01-14 1:43 ` Steven Rostedt 2006-01-14 1:50 ` john stultz 2006-01-14 2:15 ` Steven Rostedt 2006-01-14 2:34 ` Steven Rostedt 2006-01-16 1:19 ` Andi Kleen 2006-01-14 6:54 ` Christoph Lameter 2006-01-13 18:06 ` thockin 2006-01-13 17:58 ` Lee Revell 2006-01-13 18:16 ` thockin 2006-01-13 18:09 ` Lee Revell 2006-01-13 18:55 ` thockin 2006-01-13 18:56 ` Sven-Thorsten Dietrich 2006-01-13 19:00 ` Lee Revell 2006-01-13 19:07 ` Sven-Thorsten Dietrich 2006-01-13 21:18 ` David Lang 2006-01-13 21:56 ` thockin 2006-01-13 22:05 ` David Lang 2006-01-13 22:18 ` Sven-Thorsten Dietrich 2006-01-13 22:49 ` David Lang 2006-01-14 0:41 ` Andreas Steinmetz 2006-01-14 1:04 ` David Lang 2006-01-14 1:21 ` Andreas Steinmetz 2006-01-14 1:51 ` thockin 2006-01-15 8:52 ` Zan Lynx 2006-01-15 16:25 ` thockin 2006-01-15 16:33 ` Lee Revell 2006-01-15 18:21 ` thockin 2006-01-15 18:29 ` Lee Revell 2006-01-15 18:49 ` thockin 2006-01-14 1:13 ` thockin 2006-01-13 22:23 ` thockin 2006-01-13 19:13 ` Xavier Bestel 2006-01-13 19:32 ` thockin 2006-01-13 20:58 ` Steven Rostedt [not found] <5ujmj-1UQ-511@gated-at.bofh.it> [not found] ` <5uBnF-2SG-11@gated-at.bofh.it> [not found] ` <5uBnF-2SG-9@gated-at.bofh.it> [not found] ` <5uBxi-3iM-21@gated-at.bofh.it> [not found] ` <5uBGY-3ul-21@gated-at.bofh.it> [not found] ` <5uCa5-443-45@gated-at.bofh.it> [not found] ` <5uCjF-4fW-15@gated-at.bofh.it> [not found] ` <5uCtj-4Fi-15@gated-at.bofh.it> 2006-01-14 0:24 ` Robert Hancock
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).