2.6.15-rt2 x86_64 SMP instability

2.6.15-rt2 x86_64 SMP instability

[John Rigg]

I've just compiled 2.6.15-rt2 on x86_64 SMP (dual Opteron) and it's giving
a lot of weird instabilities. If I start jackd (this is an audio workstation)
with realtime privileges from an xterm I get a lot of spurious xruns. When I
first start it, moving the mouse makes the xruns scroll off the screen.
That stops for a couple of minutes, during which there's a slow but
steady stream of xruns. After two or three minutes the xruns suddenly
scroll off the screen too quickly to read, and keep going until the jack
watchdog timer kills jackd (the latter is usually caused by the two
wordclock-locked sound cards losing sync with each other, which
shouldn't happen). Sometimes X locks up shortly after this and it needs
a hard reboot.

None of the above happens with a non-rt kernel, and I've had the same
thing with 2.6.15-rt1 and every 2.6.15-rcx-rtx kernel I tried.

Here's an excerpt from dmesg that may shed some light on this:

<snip>
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
Time: tsc clocksource has been installed.
check_monotonic_clock: monotonic inconsistency detected!
	from         26cbf3ad (650900397) to         260079b7 (637565367).
softirq-timer/1/13[CPU#1]: BUG in check_monotonic_clock at kernel/time/timeofday.c:160

Call Trace:<ffffffff801361e2>{__WARN_ON+114} <ffffffff8015141d>{check_monotonic_clock+109}
       <ffffffff80151cfc>{get_realtime_clock+92} <ffffffff8014eed1>{hrtimer_run_queues+49}
       <ffffffff8013fc87>{run_timer_softirq+455} <ffffffff8013b4c0>{ksoftirqd+304}
       <ffffffff8013b390>{ksoftirqd+0} <ffffffff8013b390>{ksoftirqd+0}
       <ffffffff8014c009>{kthread+217} <ffffffff80131258>{schedule_tail+136}
       <ffffffff8010f076>{child_rip+8} <ffffffff8014bf30>{kthread+0}
       <ffffffff8010f06e>{child_rip+0} 
read_tsc: ACK! TSC went backward! Unsynced TSCs?
Linux agpgart interface v0.101 (c) Dave Jones
PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
...
...
VP_IDE: chipset revision 6
VP_IDE: not 100% native mode: will probe irqs later
VP_IDE: VIA vt8237 (rev 00) IDE UDMA133 controller on pci0000:00:0f.0
    ide0: BM-DMA at 0xd000-0xd007check_periodic_interval: Long interval! 158008629.
		Something may be blocking interrupts.
, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0xd008-0xd00f, BIOS settings: hdc:pio, hdd:DMA
Probing IDE interface ide0...
hda: Maxtor 6K040L0, ATA DISK drive
</snip>

Is there any more info I can supply to help debug this?

John

Re: 2.6.15-rt2 x86_64 SMP instability

[Steven Rostedt]

On Sun, 8 Jan 2006, John Rigg wrote:

> I've just compiled 2.6.15-rt2 on x86_64 SMP (dual Opteron) and it's giving
> a lot of weird instabilities. If I start jackd (this is an audio workstation)
> with realtime privileges from an xterm I get a lot of spurious xruns. When I
> first start it, moving the mouse makes the xruns scroll off the screen.
> That stops for a couple of minutes, during which there's a slow but
> steady stream of xruns. After two or three minutes the xruns suddenly
> scroll off the screen too quickly to read, and keep going until the jack
> watchdog timer kills jackd (the latter is usually caused by the two
> wordclock-locked sound cards losing sync with each other, which
> shouldn't happen). Sometimes X locks up shortly after this and it needs
> a hard reboot.
>
> None of the above happens with a non-rt kernel, and I've had the same
> thing with 2.6.15-rt1 and every 2.6.15-rcx-rtx kernel I tried.
>
> Here's an excerpt from dmesg that may shed some light on this:
>
> <snip>
> io scheduler noop registered
> io scheduler anticipatory registered
> io scheduler deadline registered
> io scheduler cfq registered
> Time: tsc clocksource has been installed.
> check_monotonic_clock: monotonic inconsistency detected!
> 	from         26cbf3ad (650900397) to         260079b7 (637565367).
> softirq-timer/1/13[CPU#1]: BUG in check_monotonic_clock at kernel/time/timeofday.c:160
>
> Call Trace:<ffffffff801361e2>{__WARN_ON+114} <ffffffff8015141d>{check_monotonic_clock+109}
>        <ffffffff80151cfc>{get_realtime_clock+92} <ffffffff8014eed1>{hrtimer_run_queues+49}
>        <ffffffff8013fc87>{run_timer_softirq+455} <ffffffff8013b4c0>{ksoftirqd+304}
>        <ffffffff8013b390>{ksoftirqd+0} <ffffffff8013b390>{ksoftirqd+0}
>        <ffffffff8014c009>{kthread+217} <ffffffff80131258>{schedule_tail+136}
>        <ffffffff8010f076>{child_rip+8} <ffffffff8014bf30>{kthread+0}
>        <ffffffff8010f06e>{child_rip+0}

Yep, this is a known issue, with the x86_64 SMP.  The timestamp counter
does not run in sync with each cpu, so the timing gets all screwed up.
If you want to fix this, boot with the command line option idle=poll.
But, unfortunately, this means that the cpu will waste energy even when
it's not doing anything.

I'm looking into ways to fix this for my main machine which is also a
x86_64 SMP.

-- Steve

Re: 2.6.15-rt2 x86_64 SMP instability

[Lee Revell]

On Sun, 2006-01-08 at 19:26 -0500, Steven Rostedt wrote:
> Yep, this is a known issue, with the x86_64 SMP.  The timestamp counter
> does not run in sync with each cpu, so the timing gets all screwed up.
> If you want to fix this, boot with the command line option idle=poll.
> But, unfortunately, this means that the cpu will waste energy even when
> it's not doing anything.
> 
> I'm looking into ways to fix this for my main machine which is also a
> x86_64 SMP.
> 

I thought this could be worked around by using the ACPI PM timer?

Lee