Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Mark_H_Johnson]

>I've uploaded -V0.6.5 to the usual place:
>
>  http://redhat.com/~mingo/realtime-preempt/

Hmm. I was in the middle of a V0.6.4 build when I saw this message.
I let it run but the build stopped with the following error:

  CC [M]  drivers/scsi/qla2xxx/qla_os.o
  CC [M]  drivers/usb/media/stv680.o
drivers/scsi/qla2xxx/qla_os.c: In function `qla2x00_do_dpc':
drivers/scsi/qla2xxx/qla_os.c:3193: warning: implicit declaration of
function `DECLARE_MUTEX_LOCKED'
drivers/scsi/qla2xxx/qla_os.c:3193: error: `sem' undeclared (first use in
this function)
drivers/scsi/qla2xxx/qla_os.c:3193: error: (Each undeclared identifier is
reported only once
drivers/scsi/qla2xxx/qla_os.c:3193: error: for each function it appears
in.)
drivers/scsi/qla2xxx/qla_os.c:3194: warning: ISO C90 forbids mixed
declarations and code

Based on my quick review of the updated patch, I don't see a fix for this
so I'll remove it from .config (and step up to V0.6.5) but if you get a
chance to fix this, please do.
  --Mark

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Mark_H_Johnson@raytheon.com <Mark_H_Johnson@raytheon.com> wrote:

> Hmm. I was in the middle of a V0.6.4 build when I saw this message.
> I let it run but the build stopped with the following error:
> 
>   CC [M]  drivers/scsi/qla2xxx/qla_os.o
>   CC [M]  drivers/usb/media/stv680.o
> drivers/scsi/qla2xxx/qla_os.c: In function `qla2x00_do_dpc':
> drivers/scsi/qla2xxx/qla_os.c:3193: warning: implicit declaration of
> function `DECLARE_MUTEX_LOCKED'

the patch below should fix it, will be in the next release.

	Ingo

--- linux/drivers/scsi/qla2xxx/qla_os.c.orig
+++ linux/drivers/scsi/qla2xxx/qla_os.c
@@ -3190,7 +3190,7 @@ qla2x00_free_sp_pool( scsi_qla_host_t *h
 static int
 qla2x00_do_dpc(void *data)
 {
-	DECLARE_MUTEX_LOCKED(sem);
+	DECLARE_MUTEX(sem);
 	scsi_qla_host_t *ha;
 	fc_port_t	*fcport;
 	os_lun_t        *q;
@@ -3204,6 +3204,8 @@ qla2x00_do_dpc(void *data)
 	int t;
 	os_tgt_t *tq;
 
+	down(&sem);
+
 	ha = (scsi_qla_host_t *)data;
 
 	lock_kernel();

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Paul Davis <paul@linuxaudiosystems.com> wrote:

> >poll() is quite complex and with a good number of locks in the path the
> >maximum latency increases accordingly.
> 
> how can poll(2) be more complex than read/write? if it is, it
> shouldn't be ;)

poll() is fundamentally more complex: it has to watch multiple channels,
while read()/write() has to watch only a single event channel. The fact
that read()/write() also has to do some actual IO makes little
difference to complexity, as poll() already has to do most of the
locking read()/write() has to do, to figure out that it _could_ do the
read()/write().

> >btw., couldnt jackd use a separate input and output thread (of identical
> >priority), to be purely read()/write() based? This method should also
> >solve the priority problems of poll(): the thread woken up later will do
> >the work later. (hence the _earlier_ interrupt source will be handled
> >first.) With poll() how do you tell which fd needs attention first, if
> >both are set?
> 
> we don't really care which one needs attention "first". [...]

well, order of processing can make a difference under a high event load.
Couldnt capture and playback interrupts be separate and differently
timed? I understand your previous points that the audio 'channels' are
highly coupled and cannot be considered separate 'event sources', but is
the same true for all the fds that jackd passes into poll()? Is it true
if multiple cards are used?

the scenario that could trigger problems is that if an event (or group
of events) triggers some processing in the highprio thread, and two more
events arrive, one at the beginning of the previous processing, one at
the end of it. Once the highprio thread calls poll() again, the timing
of the two events has been lost - and jackd could end up processing the
_later_ event.

while this should normally make no difference with low audio loads, i
can very much see this causing problem as the number of cards/events
increases. What is typically the longest amount of time the highprio
thread can spend 'processing' without being actively poll()-ing? I know
it is typically short, but what is roughly the longest amount of time?
(is it the 1.4 msecs displayed in one of Rui's earlier testresults?)

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

> > also, there are no "arbitrary load" latency guarantees with
> > DEADLOCK_DETECTION turned on, since we search the list of all held locks
> > during task-exit time - this can generate pretty bad latencies e.g.
> > during hackbench.
> 
> btw: i see the same build failure as Rui with lock debugging disabled.

(just remove the offending call to show_all_locks())

> Since the lock debugging can screw timings, will this be fixed in [one
> of] the next version[s]?

yeah, i think so. Right now i've increased the complexity of the checks
to root out bugs as clearly there's a stability problem. Once things
have calmed down i think we can remove the 'check all locks at exit
time' portion that is the problematic one.

note that you really need some insane loads with thousands of tasks in
the runqueue (hackbench) to really trigger that kind of overhead. Normal
loads are not supposed to trigger any of this, even with full debugging
turned on.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Bill Huey <bhuey@lnxw.com> wrote:

> The lock chains aren't that deep in Linux so the algorithmic
> complexity is not going to hit some crazy polynomial time unless
> there's some seriously nasty contention at a certain point in the
> kernel (billions of readers for example against a write aquire). But
> when we start to see things like that under pressure is when we need
> to start shortening the need for that/those lock(s) for that/those
> critical section(s) in question.

also note that in the -U series i removed the true 'read' logic from
semaphores. What we have now are single writers only, plus readers
emulated as a writer plus the ability to self-recurse. ('writers' are
not allowed to self-recurse.) This is quite close to the semantic needs
of Linux rwlocks and rwsems and it simplified both locking, deadlock
detection and PI quite significantly.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Michal Schmidt]

Hi,

This is after "modprobe -r uhci-hcd" on -V0.6.5:

BUG: sleeping function called from invalid context rmmod(5093) at 
kernel/mutex.c:30
in_atomic():1 [00000001], irqs_disabled():1
  [<c01060d3>] dump_stack+0x23/0x30 (20)
  [<c011b2e2>] __might_sleep+0xc2/0xe0 (36)
  [<c0134d78>] __mutex_lock+0x38/0x50 (24)
  [<c0134dad>] _mutex_lock+0x1d/0x20 (16)
  [<c01482c4>] do_drain+0x24/0x60 (32)
  [<c0148269>] smp_call_function_all_cpus+0x29/0x60 (28)
  [<c0148327>] drain_cpu_caches+0x27/0x60 (28)
  [<c014837c>] __cache_shrink+0x1c/0x100 (36)
  [<c014855d>] kmem_cache_destroy+0x9d/0x190 (28)
  [<f8a40e24>] uhci_hcd_cleanup+0x24/0x67 [uhci_hcd] (16)
  [<c0138c40>] sys_delete_module+0x120/0x150 (96)
  [<c0105293>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000002 ]
| 2-level deep critical section nesting:
----------------------------------------
.. [<c0148263>] .... smp_call_function_all_cpus+0x23/0x60
.....[<c0148327>] ..   ( <= drain_cpu_caches+0x27/0x60)
.. [<c0136b0d>] .... print_traces+0x1d/0x90
.....[<c01060d3>] ..   ( <= dump_stack+0x23/0x30)


Michal

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Bill Huey]

On Mon, Nov 01, 2004 at 11:51:25PM +0100, Florian Schmidt wrote:
[...lock debugging...]
> ack. understood. i was just asking since i don't have a second machine and
> thus am not really able to help with the deadlock debugging. so i figured i
> could at least do some timing. btw: even with deadlock detection, the
> results for 0.6.5 looked pretty good [in 10 minutes uptime ca.3-4% max
> jitter [30something usecs].  until the deadlock that is [i head three finds
> plus kernel compile at nice -10 running]..

The lock chains aren't that deep in Linux so the algorithmic complexity
is not going to hit some crazy polynomial time unless there's some seriously
nasty contention at a certain point in the kernel (billions of readers for
example against a write aquire). But when we start to see things like that
under pressure is when we need to start shortening the need for that/those
lock(s) for that/those critical section(s) in question.

bill

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

On Mon, 1 Nov 2004 14:40:47 -0800
Bill Huey (hui) <bhuey@lnxw.com> wrote:

> Unlikely, it's got to lock the entire kernel to make sure that things aren't
> changing under it. Getting measurements is useful at this stage, but don't expect
> it to be a finished product any time soon and please keep that in mind. Stability
> should be, if it's not already, the single most important things in this project
> at this phase. Getting numbers now for a specific single application is going to
> be of limit use until the system is stable enough for general characterization.
> 
> Keep doing it, but keep in mind that anything you're going to get at this time
> is going to be very coarse. Don't put too much weight on it.
> 
> That's my take.

ack. understood. i was just asking since i don't have a second machine and
thus am not really able to help with the deadlock debugging. so i figured i
could at least do some timing. btw: even with deadlock detection, the
results for 0.6.5 looked pretty good [in 10 minutes uptime ca.3-4% max
jitter [30something usecs].  until the deadlock that is [i head three finds
plus kernel compile at nice -10 running]..

flo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Bill Huey]

On Mon, Nov 01, 2004 at 11:30:37PM +0100, Florian Schmidt wrote:
> took jackit-devel off cc, since it bounced anyways [too many recipients]
> On Mon, 1 Nov 2004 19:46:15 +0100
> Ingo Molnar <mingo@elte.hu> wrote:
... 
> > 
> > also, there are no "arbitrary load" latency guarantees with
> > DEADLOCK_DETECTION turned on, since we search the list of all held locks
> > during task-exit time - this can generate pretty bad latencies e.g.
> > during hackbench.

> btw: i see the same build failure as Rui with lock debugging disabled. Since
> the lock debugging can screw timings, will this be fixed in [one of] the next
> version[s]?

Unlikely, it's got to lock the entire kernel to make sure that things aren't
changing under it. Getting measurements is useful at this stage, but don't expect
it to be a finished product any time soon and please keep that in mind. Stability
should be, if it's not already, the single most important things in this project
at this phase. Getting numbers now for a specific single application is going to
be of limit use until the system is stable enough for general characterization.

Keep doing it, but keep in mind that anything you're going to get at this time
is going to be very coarse. Don't put too much weight on it.

That's my take.

bill

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

took jackit-devel off cc, since it bounced anyways [too many recipients]

On Mon, 1 Nov 2004 19:46:15 +0100
Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Thomas Gleixner <tglx@linutronix.de> wrote:
> 
> > >   http://redhat.com/~mingo/realtime-preempt/
> > > 
> > > Thomas, can you confirm that this kernel fixes the irqs-off latencies? 
> > > (the priority loop indeed was done with irqs turned off.)
> > 
> > The latencies are still there. I have the feeling it's worse than 0.6.2.
> 
> what is the worst latency you can trigger with Florian's latest
> rtc_wakeup code? (please re-download it, there has been a recent update)

I have fixed one or two more small issues:

1] the cpu cycles/s measurement was done while not yet running SCHED_FIFO.
   changed it so the rt priv is aquired beforehand

2] when one or more irq's were missed, the cycle timestamps for the last wakeup
   do not get used  anymore for neither threshold nor max jitter reporting. 
   It's kind of pointless to calculate jitter for a period with the 
   fundamental requirement that no irq's be missed being violated.

3] the output file format (-o) is now 

num_of_irqs_since_last_wakeup  cycles_count 

and basically looks like this:

1 116817809121123
1 116817810280681
1 116817811456573
1 116817812617197
1 116817813788473
1 116817814948983
1 116817816121533
...

i suppose this data might be easily fed into a histogram producing script or
something..

New download place:

http://www.affenbande.org/~tapas/wiki/index.php?rtc_wakeup

direct link (has changed. i'll remove the old tarball)

http://affenbande.org/~tapas/rtc_wakeup/rtc_wakeup.tgz

Whoever wants to be notified of new versions should mail me a reply
(private), so i can drop em a line in case (lkml is hi traffic enough
already)..

> 
> also, there are no "arbitrary load" latency guarantees with
> DEADLOCK_DETECTION turned on, since we search the list of all held locks
> during task-exit time - this can generate pretty bad latencies e.g.
> during hackbench.

btw: i see the same build failure as Rui with lock debugging disabled. Since
the lock debugging can screw timings, will this be fixed in [one of] the next
version[s]?

flo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Rui Nuno Capela]

Thomas Gleixner wrote:
> Rui Nuno Capela wrote:
>> Ingo Molnar wrote:
>> >
>> > I've uploaded -V0.6.5 to the usual place:
>> >
>> >   http://redhat.com/~mingo/realtime-preempt/
>> >
>>
>> Build error:
>>
>> kernel/built-in.o(.text+0x308a): In function `show_state':
>> : undefined reference to `show_all_locks'
>> kernel/built-in.o(.text+0x30bc): In function `show_state':
>> : undefined reference to `show_all_locks'
>> make: *** [.tmp_vmlinux1] Error 1
>
> You have lock debugging disabled.
>

OK. Applied Thomas' corrective patch and now a debug-stripped RT-V0.6.5
has been built fine on both of my machines.

But I'm afraid to have bad news.

My (now infamous:) jackd-R + 9*fluidsynth test is being abruptly killed by
jack_watchdog. Annoyingly always. On both of my configurations, laptop
(P4/UP) and desktop (P4/SMT), and in any other circunstance, jackd -R
can't survive for long. It just stalls after a while.

To be sincere, this happened once while testing with RT-V0.6.4, but only
after too many successful test runs.

Take care.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Paul Davis]

>poll() is quite complex and with a good number of locks in the path the
>maximum latency increases accordingly.

how can poll(2) be more complex than read/write? if it is, it
shouldn't be ;)

>btw., couldnt jackd use a separate input and output thread (of identical
>priority), to be purely read()/write() based? This method should also
>solve the priority problems of poll(): the thread woken up later will do
>the work later. (hence the _earlier_ interrupt source will be handled
>first.) With poll() how do you tell which fd needs attention first, if
>both are set?

we don't really care which one needs attention "first". the jack process
cycle ends up consuming and producing data - 

      before the ALSA backend tells jackd to run the process cycle,
         it will read the data from the capture fd (though its done
	 using mmap, so there is no read(2) call)

      just before jackd goes back to sleep waiting for the next
         interrupt, the ALSA backend will write data to the playback fd
	 (again, using mmap, so there is no actual write(2) call)

it is important to use mmap - it avoids multiple kernel entries and
data copying. for consumer cards this doesn't matter much, but for
high end multichannel cards, the data copy would be inexcusable.

--p

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Mark_H_Johnson@raytheon.com <Mark_H_Johnson@raytheon.com> wrote:

> Did notice an odd message during the dump of tasks...
> Losing too many ticks!
> TSC cannot be used as a timesource.
> Possible reasons for this are:
>   You're running with Speedstep.   [almost surely not]
>   You don't have DMA enabled for your hard disk (see hdparm), [udma4 should
> be set at this point]
>   Incorrect TSC synchronization on an SMP system (see dmesg). [can't look
> at that, system is hung]
> Falling back to a sane timesource now.

this is just due to the messages going out to the serial console slowly
and with interrupts disabled. You can safely ignore this during the
dump, the system wont get into any much worse state from this. So it's a
pure side-effect.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Thomas Gleixner <tglx@linutronix.de> wrote:

> >   http://redhat.com/~mingo/realtime-preempt/
> > 
> > Thomas, can you confirm that this kernel fixes the irqs-off latencies? 
> > (the priority loop indeed was done with irqs turned off.)
> 
> The latencies are still there. I have the feeling it's worse than 0.6.2.

what is the worst latency you can trigger with Florian's latest
rtc_wakeup code? (please re-download it, there has been a recent update)

also, there are no "arbitrary load" latency guarantees with
DEADLOCK_DETECTION turned on, since we search the list of all held locks
during task-exit time - this can generate pretty bad latencies e.g.
during hackbench.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> This was my conclusion as well.  I have a patch sitting around to add
> this to the emu10k1 ALSA driver, it's quite useful.  It would be nice
> if there were a facility in the kernel to easily identify missed
> interrupts like this or (even better) unbalanced irq disable/enable -
> AFAICT userspace alone cannot reliably distinguish lost interrupts
> from scheduling problems (though you can get a lot of hints).  Paul
> mentioned trying to debug the unbalanced irq disable in his talk at
> ZKM 2003, and said it's hard because the hardware will enable/disable
> interrupts on its own and he could not identify all those places. 
> Ingo, is there an easy way to trace this like we do for unbalanced
> preempt count?

i wrote a cli/sti latency tracer a couple of years ago so it's possible.
Note that an irqs-off condition is near impossible to 'leak' into
userspace code though, since the x86 iret path restores flags to the
previous value. Worst-case the irqs-off condition may leak into
kernelspace, and that can still cause bad effects. X startup/shutdown
can disable interrupts for a long time, was that excluded from your
testing?

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Thomas Gleixner]

On Mon, 2004-11-01 at 12:55 -0500, Lee Revell wrote:
> I think there might even be a _very_ rare errant irq disable in T3 even.
> On my 2-day test runs with 100s of millions of samples, I got 2 or 3
> outliers in each graph.  Jow Gwinn from LKML (thanks Joe) ran a
> statistical analysis that showed these to be independent from the 4 or 5
> underlying exponential distributions (each corresponding to a known or
> suspected nonpreemptible section).  Our theory was that these were very
> rare code paths or race conditions that left IRQs off.  Unfortunately
> this seems impossible to debug unless you have a way to make the machine
> crash dump immediately when it detects the situation.
> 
> Anyway, the clearest way to demonstrate the problem with the -V series
> here is to run the version of Florian's tool that tells you how many
> interrupts were missed.  If I spin my (USB, not sharing irq with
> soundcard) trackball as fast as I can I can get it up to 54 or 55 in a
> row.  If I move it just the right way I can see the humps appear - 2,
> then 15, then 50 then 15 then 2 missed interrupts in a row.  This is at
> 1024Hz - at 2048 I can get it to miss several hundred IRQs, but this
> inevitably locks the machine.
> 
> I suspect the lockups and the latencies are same bug.

Until now there was no output on lockup. No I got one.
The command was: cat /proc/interrupts

Hope that helps

tglx

cat/1488: BUG in down_mutex
at /work/thomas/Linux/2.6.9-mm1-RT/lib/rwsem-generic.c:1059
 [<c01ca9e2>] down_mutex+0x72/0x80 (8)
 [<c012db9a>] __mutex_lock+0x2a/0x40 (36)
 [<c012dbc7>] _mutex_lock+0x17/0x20 (12)
 [<c013fd62>] kfree+0x52/0xf0 (8)
 [<c013fd62>] kfree+0x52/0xf0 (4)
 [<c01774a2>] single_release+0x32/0x40 (32)
 [<c01571ba>] __fput+0x13a/0x150 (20)
 [<c0155899>] filp_close+0x59/0x90 (32)
 [<c015593f>] sys_close+0x6f/0xa0 (24)
 [<c010603b>] syscall_call+0x7/0xb (28)

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Mon, 2004-11-01 at 16:47 +0100, Thomas Gleixner wrote:
> The latencies are still there. I have the feeling it's worse than 0.6.2.
> 
> It's definitely irq-off. I have a card with a controller, which produces
> an 2ms interrupt. The controller busy loops until the second level ack
> is done. The time is measured from raising the irq to the 2nd level ack.

This was my conclusion as well.  I have a patch sitting around to add
this to the emu10k1 ALSA driver, it's quite useful.  It would be nice if
there were a facility in the kernel to easily identify missed interrupts
like this or (even better) unbalanced irq disable/enable - AFAICT
userspace alone cannot reliably distinguish lost interrupts from
scheduling problems (though you can get a lot of hints).  Paul mentioned
trying to debug the unbalanced irq disable in his talk at ZKM 2003, and
said it's hard because the hardware will enable/disable interrupts on
its own and he could not identify all those places.  Ingo, is there an
easy way to trace this like we do for unbalanced preempt count?

I think there might even be a _very_ rare errant irq disable in T3 even.
On my 2-day test runs with 100s of millions of samples, I got 2 or 3
outliers in each graph.  Jow Gwinn from LKML (thanks Joe) ran a
statistical analysis that showed these to be independent from the 4 or 5
underlying exponential distributions (each corresponding to a known or
suspected nonpreemptible section).  Our theory was that these were very
rare code paths or race conditions that left IRQs off.  Unfortunately
this seems impossible to debug unless you have a way to make the machine
crash dump immediately when it detects the situation.

Anyway, the clearest way to demonstrate the problem with the -V series
here is to run the version of Florian's tool that tells you how many
interrupts were missed.  If I spin my (USB, not sharing irq with
soundcard) trackball as fast as I can I can get it up to 54 or 55 in a
row.  If I move it just the right way I can see the humps appear - 2,
then 15, then 50 then 15 then 2 missed interrupts in a row.  This is at
1024Hz - at 2048 I can get it to miss several hundred IRQs, but this
inevitably locks the machine.

I suspect the lockups and the latencies are same bug.

Lee

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[K.R. Foley]

Ingo Molnar wrote:
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> 
>>>removing the poll() lines doesnt seem to impact the quality of the
>>>data, but i still see roughly 50 usecs added to the 'real' latency
>>>that i see in traces.
>>
>>this i think is related to what Thomas observed, that there's a new
>>irqs-off critical section somewhere. (it's in the new priority
>>handling code i think.)
> 
> 
> ah, found it. Only RT tasks were supposed to get special priority
> handling, while in fact all tasks got it - so when Thomas ran hackbench
> (Thomas, you did, right?) it created an O(nr_hackbench) overhead within
> the mutex code ... I've uploaded -V0.6.5 to the usual place:
> 
>   http://redhat.com/~mingo/realtime-preempt/
> 

V0.6.5 built and booted fine on my SMP workstation. However, just a few 
minutes after booting it exhibited behavior like a system does when you 
can't fork any new processses.
The system was responsive.
I could switch between windows.
I could switch from X to a virtual terminal.
I could type in commands but they never return.
At the virtual terminal login I could type the login but never get a 
passwd prompt.
Nothing in the logs during this timeframe.
Reset button was the only way to recover.

kr

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Thomas Gleixner]

On Mon, 2004-11-01 at 17:04 +0000, Rui Nuno Capela wrote:
> Ingo Molnar wrote:
> >
> > I've uploaded -V0.6.5 to the usual place:
> >
> >   http://redhat.com/~mingo/realtime-preempt/
> >
> 
> Build error:
> 
> kernel/built-in.o(.text+0x308a): In function `show_state':
> : undefined reference to `show_all_locks'
> kernel/built-in.o(.text+0x30bc): In function `show_state':
> : undefined reference to `show_all_locks'
> make: *** [.tmp_vmlinux1] Error 1

You have lock debugging disabled.

tglx


--- linux/lib/rwsem-generic.o.c	2004-11-01 18:07:02.000000000 +0100
+++ linux/lib/rwsem-generic.c	2004-11-01 15:49:27.000000000 +0100
@@ -413,6 +413,9 @@
 {
 }
 
+void show_all_locks(void)
+{
+}
 #endif
 
 /*

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Rui Nuno Capela]

Ingo Molnar wrote:
>
> I've uploaded -V0.6.5 to the usual place:
>
>   http://redhat.com/~mingo/realtime-preempt/
>

Build error:

kernel/built-in.o(.text+0x308a): In function `show_state':
: undefined reference to `show_all_locks'
kernel/built-in.o(.text+0x30bc): In function `show_state':
: undefined reference to `show_all_locks'
make: *** [.tmp_vmlinux1] Error 1

Bye.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Rui Nuno Capela]

Ingo Molnar wrote:
>
>
> i've uploaded -V0.6.3 to the usual place:
>
>     http://redhat.com/~mingo/realtime-preempt/
>
> which fixes two issues:
>
>  - priorities of SCHED_OTHER tasks not getting properly managed, hence
>    they could end up 'starving' other SCHED_OTHER tasks pretty
>    indefinitely. This could possibly solve the 'temporary lockup'
>    problem reported by Mark H Johnson.
>
>  - fixed the 'high load average' bug
>
> i dont know whether this will solve the 'hard lockups' reported though.
> It could solve your problem because the 'find /' kept running so it
> wasnt a hard lockup. I'll keep testing and i'll fix any problem i can
> reproduce myself.
>

OK. Just tested in a hurry with RT-V0.6.4, and it passed three times in a
row my jackd-R + 9*fluidsynth tests, without freezing, so I think the
problem was the 'starvation' one.

OTOH, I can say that the results I'm early reading are really an
improvement. Fewer xruns and smaller delays. Nice.

Cheers.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Mark_H_Johnson]

>I've uploaded -V0.6.5 to the usual place:
>
>  http://redhat.com/~mingo/realtime-preempt/

I built with this patch and had some problems with the system
locking up.

First attempt:
 - booted to single user without problem
 - telinit 5 was OK as well
 - logged in. After first window popped up, I did
  su -
to get root access and the system locked up. No display updates
nor any mouse movement. Entering Alt-SysRq-L displayed
  SysRq : Show Regs On All CPUs
and no other messages appeared on the serial console. Attempts
to use other Alt-SysRq keys were ignored, hard reset to reboot.

Second attempt:
 - booted to single user without problem
 - telinit 5 failed after kudzu timed out (had "detected" a
change due to the serial console)

No messages on serial console after the kudzu timeout. This time
Alt-SysRq-L did work. Also did Alt-SysRq-T and -D and will send
the all the serial console messages separately.

Did notice an odd message during the dump of tasks...
Losing too many ticks!
TSC cannot be used as a timesource.
Possible reasons for this are:
  You're running with Speedstep.   [almost surely not]
  You don't have DMA enabled for your hard disk (see hdparm), [udma4 should
be set at this point]
  Incorrect TSC synchronization on an SMP system (see dmesg). [can't look
at that, system is hung]
Falling back to a sane timesource now.

Alas, this "sane timesource" didn't help either, system still
not responding right. Alt-SysRq-S said
  SysRq : Emergency Sync
[without any complete messages]
and Alt-SysRq-B did reboot the system.

--Mark H Johnson
  <mailto:Mark_H_Johnson@raytheon.com>

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Thomas Gleixner]

On Mon, 2004-11-01 at 15:06 +0100, Ingo Molnar wrote:
> ah, found it. Only RT tasks were supposed to get special priority
> handling, while in fact all tasks got it - so when Thomas ran hackbench
> (Thomas, you did, right?) it created an O(nr_hackbench) overhead within
> the mutex code ... I've uploaded -V0.6.5 to the usual place:

Yes, I was running hackbench as usual

>   http://redhat.com/~mingo/realtime-preempt/
> 
> Thomas, can you confirm that this kernel fixes the irqs-off latencies? 
> (the priority loop indeed was done with irqs turned off.)

The latencies are still there. I have the feeling it's worse than 0.6.2.

It's definitely irq-off. I have a card with a controller, which produces
an 2ms interrupt. The controller busy loops until the second level ack
is done. The time is measured from raising the irq to the 2nd level ack.

The irqhandler is using NODELAY and keeps irqs disabled. So the measured
time in the controller is the irqs disabled time + the irq latency
(which is ~7µs). The testrun on 0.6.5 showed latencies up to 600µs
within 10 minutes.

After 15 Mintes the keyboard was dead. 

I'm porting some of the Libertos debugging stuff into 0.6.5, so I can
instrument the problem. 

More later.

tglx

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[K.R. Foley]

Florian Schmidt wrote:
> On Mon, 1 Nov 2004 14:42:35 +0100
> Ingo Molnar <mingo@elte.hu> wrote:
> 
> 
>>* Florian Schmidt <mista.tapas@gmx.net> wrote:
>>
>>
>>>new max. jitter: 4.3% (41 usec)
>>>new max. jitter: 4.9% (47 usec)
>>
>>a couple of conceptual questions: why does rtc_wakeup poll() on
>>/dev/rtc? Shouldnt a read() be enough?
> 
> 
> well, it works like this:
> 
> 
> while(!stopit) {
> 	// returns when data is ready
> 	poll(on /dev/rtc);
> 
> 	// when ready generate the timestamp
> 	cycles = get_cycles();
> 
> 	// now read the data	
> 	read(on /dev/rtc);			
> 
> 	// and now stuff the data (including timestamp) into the ringbuffer
> 
> 	// rinse and repeat
> }
> 
> i get the timestamp before reading because i figured i want to take the
> timestamp as close as possible to data being available. The read() and
> passing the data to the other thread done after the timestamp generation (in
> that ca. 1 - 0.1 ms (1024 - 8192 hz) time window which we have until the
> next irq occurs)
> 
> 
>>i'm seeing some weird traces, which show rtc_wakeup doing this cycle:
>>
>>	[~900 usecs pass]
>>
>>	hardirq 8 comes in, wakes IRQ 8 thread
>>	IRQ 8 thread wakes up rtc_wakeup
>>
>>	rtc_wakeup fast-thread returns from sys_read()
>>	rtc_wakeup fast-thread enters sys_poll() and returns immediately
>>	rtc_wakeup fast-thread enters sys_read() and blocks
> 
> 
> weird. why could poll return immeaditly? Only when data should be available
> right? Ahh, maybe there's less data available than which is needed by
> read(). I suppose i need to check if enough data is available. If not,
> repoll(), then generate the timestamp. Then read(). I had the impression
> that this small amount of data which rtc delivers (4 bytes i think) would
> not be split into smaller parts.
> 
> It never occured to me that poll() could return with incomplete rtc data
> available..
> 
> As i don't know of any way of finding out how much data is available i
> suppose we can just make the poll() a read(). I suppose overhead is
> neglectable right?
> 
Someone please correct me if I'm wrong, but is there any MORE overhead 
associated with a blocking read than with a poll or select? Won't the 
process just go to sleep until there is data available?

> 
>>	rtc_wakeup slow-thread runs and does the calculations.
>>
>>	[repeat]
>>
>>this i think shows that the logic is wrong somewhere and that read()
>>will achieve the blocking. This also means that the sys_read()-return +
>>sys_poll() overhead is added to the 'IRQ wakeup' overhead!
>>
>>removing the poll() lines doesnt seem to impact the quality of the data,
>>but i still see roughly 50 usecs added to the 'real' latency that i see
>>in traces.
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Paul Davis <paul@linuxaudiosystems.com> wrote:

> >
> >* Florian Schmidt <mista.tapas@gmx.net> wrote:
> >
> >> new max. jitter: 4.3% (41 usec)
> >> new max. jitter: 4.9% (47 usec)
> >
> >a couple of conceptual questions: why does rtc_wakeup poll() on
> >/dev/rtc? Shouldnt a read() be enough?
> 
> i suggested to florian that it should model jackd's behaviour as
> closely as possible. because jackd requires duplex operation, using
> just read/write doesn't work.

ok - but i think there should at least be an option to turn the
poll()-ing off. To showcase the best-possible wakeup latency offered by
the kernel :-)

poll() is quite complex and with a good number of locks in the path the
maximum latency increases accordingly.

btw., couldnt jackd use a separate input and output thread (of identical
priority), to be purely read()/write() based? This method should also
solve the priority problems of poll(): the thread woken up later will do
the work later. (hence the _earlier_ interrupt source will be handled
first.) With poll() how do you tell which fd needs attention first, if
both are set?

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

> > i'm seeing some weird traces, which show rtc_wakeup doing this cycle:
> > 
> > 	[~900 usecs pass]
> > 
> > 	hardirq 8 comes in, wakes IRQ 8 thread
> > 	IRQ 8 thread wakes up rtc_wakeup
> > 
> > 	rtc_wakeup fast-thread returns from sys_read()
> > 	rtc_wakeup fast-thread enters sys_poll() and returns immediately
> > 	rtc_wakeup fast-thread enters sys_read() and blocks
> 
> weird. why could poll return immeaditly? Only when data should be
> available right? Ahh, maybe there's less data available than which is
> needed by read(). I suppose i need to check if enough data is
> available. If not, repoll(), then generate the timestamp. Then read().
> I had the impression that this small amount of data which rtc delivers
> (4 bytes i think) would not be split into smaller parts.
> 
> It never occured to me that poll() could return with incomplete rtc
> data available..

this is how it works:

static unsigned int rtc_poll(struct file *file, poll_table *wait)
{
        unsigned long l;

        if (rtc_has_irq == 0)
                return 0;

        poll_wait(file, &rtc_wait, wait);

        spin_lock_irq (&rtc_lock);
        l = rtc_irq_data;
        spin_unlock_irq (&rtc_lock);

        if (l != 0)
                return POLLIN | POLLRDNORM;
        return 0;
}

it seems that it cannot return with incomplete data. rtc_has_irq is a
boot-time-only thing - either the RTC can generate interrupts or it
cannot. rtc_irq_data is updated atomically - either it's full with 4
bytes or it's completely empty.

safest would be to call the read() with O_NONBLOCK and figure out how
often -EAGAIN happens? (O_NONBLOCK is honored by /dev/rtc.)

but i could imagine that you could get into a 'wrong phase' if for
whatever reason an RTC interrupt slips in before the poll()?

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> > removing the poll() lines doesnt seem to impact the quality of the
> > data, but i still see roughly 50 usecs added to the 'real' latency
> > that i see in traces.
> 
> this i think is related to what Thomas observed, that there's a new
> irqs-off critical section somewhere. (it's in the new priority
> handling code i think.)

ah, found it. Only RT tasks were supposed to get special priority
handling, while in fact all tasks got it - so when Thomas ran hackbench
(Thomas, you did, right?) it created an O(nr_hackbench) overhead within
the mutex code ... I've uploaded -V0.6.5 to the usual place:

  http://redhat.com/~mingo/realtime-preempt/

Thomas, can you confirm that this kernel fixes the irqs-off latencies? 
(the priority loop indeed was done with irqs turned off.)

i'm not sure this fix is related to the deadlocks reported though.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

On Mon, 1 Nov 2004 14:42:35 +0100
Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Florian Schmidt <mista.tapas@gmx.net> wrote:
> 
> > new max. jitter: 4.3% (41 usec)
> > new max. jitter: 4.9% (47 usec)
> 
> a couple of conceptual questions: why does rtc_wakeup poll() on
> /dev/rtc? Shouldnt a read() be enough?

well, it works like this:


while(!stopit) {
	// returns when data is ready
	poll(on /dev/rtc);

	// when ready generate the timestamp
	cycles = get_cycles();

	// now read the data	
	read(on /dev/rtc);			

	// and now stuff the data (including timestamp) into the ringbuffer

	// rinse and repeat
}

i get the timestamp before reading because i figured i want to take the
timestamp as close as possible to data being available. The read() and
passing the data to the other thread done after the timestamp generation (in
that ca. 1 - 0.1 ms (1024 - 8192 hz) time window which we have until the
next irq occurs)

> 
> i'm seeing some weird traces, which show rtc_wakeup doing this cycle:
> 
> 	[~900 usecs pass]
> 
> 	hardirq 8 comes in, wakes IRQ 8 thread
> 	IRQ 8 thread wakes up rtc_wakeup
> 
> 	rtc_wakeup fast-thread returns from sys_read()
> 	rtc_wakeup fast-thread enters sys_poll() and returns immediately
> 	rtc_wakeup fast-thread enters sys_read() and blocks

weird. why could poll return immeaditly? Only when data should be available
right? Ahh, maybe there's less data available than which is needed by
read(). I suppose i need to check if enough data is available. If not,
repoll(), then generate the timestamp. Then read(). I had the impression
that this small amount of data which rtc delivers (4 bytes i think) would
not be split into smaller parts.

It never occured to me that poll() could return with incomplete rtc data
available..

As i don't know of any way of finding out how much data is available i
suppose we can just make the poll() a read(). I suppose overhead is
neglectable right?

> 
> 	rtc_wakeup slow-thread runs and does the calculations.
> 
> 	[repeat]
> 
> this i think shows that the logic is wrong somewhere and that read()
> will achieve the blocking. This also means that the sys_read()-return +
> sys_poll() overhead is added to the 'IRQ wakeup' overhead!
> 
> removing the poll() lines doesnt seem to impact the quality of the data,
> but i still see roughly 50 usecs added to the 'real' latency that i see
> in traces.

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Paul Davis]

>
>* Florian Schmidt <mista.tapas@gmx.net> wrote:
>
>> new max. jitter: 4.3% (41 usec)
>> new max. jitter: 4.9% (47 usec)
>
>a couple of conceptual questions: why does rtc_wakeup poll() on
>/dev/rtc? Shouldnt a read() be enough?

i suggested to florian that it should model jackd's behaviour as
closely as possible. because jackd requires duplex operation, using
just read/write doesn't work.

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> removing the poll() lines doesnt seem to impact the quality of the
> data, but i still see roughly 50 usecs added to the 'real' latency
> that i see in traces.

this i think is related to what Thomas observed, that there's a new
irqs-off critical section somewhere. (it's in the new priority handling
code i think.)

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

> well, how would i best check for the presence of the process/thread "IRQ 8"?

pidof 'IRQ 8' seems to work pretty well.

> Although: is rtc always garanteed to be "IRQ 8" or is this only the
> case on ia32 with XT-PIC?

no. But the following command should work even if rtc is not on 
IRQ8:

 chrt -f 99 -p `pidof "IRQ \`ls -d /proc/irq/*/rtc | cut -d/ -f4\`"`

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

> new max. jitter: 4.3% (41 usec)
> new max. jitter: 4.9% (47 usec)

a couple of conceptual questions: why does rtc_wakeup poll() on
/dev/rtc? Shouldnt a read() be enough?

i'm seeing some weird traces, which show rtc_wakeup doing this cycle:

	[~900 usecs pass]

	hardirq 8 comes in, wakes IRQ 8 thread
	IRQ 8 thread wakes up rtc_wakeup

	rtc_wakeup fast-thread returns from sys_read()
	rtc_wakeup fast-thread enters sys_poll() and returns immediately
	rtc_wakeup fast-thread enters sys_read() and blocks

	rtc_wakeup slow-thread runs and does the calculations.

	[repeat]

this i think shows that the logic is wrong somewhere and that read()
will achieve the blocking. This also means that the sys_read()-return +
sys_poll() overhead is added to the 'IRQ wakeup' overhead!

removing the poll() lines doesnt seem to impact the quality of the data,
but i still see roughly 50 usecs added to the 'real' latency that i see
in traces.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

On Mon, 1 Nov 2004 14:15:11 +0100
Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > [...] I am too seeing rtc_wakeup weirdnesses that were not present in
> > earlier -V0.6 kernels.
> 
> this turned out to be a user error - used the wrong script to renice
> IRQ8. Perhaps rtc_wakeup could check for the presence of IRQ 8 and chrt
> it to 99 if it's present? :-|

Hi,

well, how would i best check for the presence of the process/thread "IRQ 8"?
A quick glance through the output of apropos pid didn't show anything. i
suppose in the worst case i'd have to iterate over the list of all processes
and find the one that matches the name. Dunno how to do that either yet. If
it's fairly straightforward, i'll add it.

Although: is rtc always garanteed to be "IRQ 8" or is this only the case on
ia32 with XT-PIC?

?

flo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> [...] I am too seeing rtc_wakeup weirdnesses that were not present in
> earlier -V0.6 kernels.

this turned out to be a user error - used the wrong script to renice
IRQ8. Perhaps rtc_wakeup could check for the presence of IRQ 8 and chrt
it to 99 if it's present? :-|

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Thomas Gleixner <tglx@linutronix.de> wrote:

> I'm doing some tests from my RT environment on 0.6.2. I'm quite sure,
> that interrupts are sporadically disabled for > 200µs. Did you change
> anything relevant to this between 0.6.2 and 0.6.4 ?

hm, i changed the task-exit schedule() to be called with irqs-off and
__schedule() - but that should be fine. I've attached the delta diff -
there's nothing suspicious at first sight. Maybe -V0.6.4 just made some
already existing bug more likely to trigger? I am too seeing rtc_wakeup
weirdnesses that were not present in earlier -V0.6 kernels.

	Ingo

--- linux.old/Makefile	
+++ linux.new/Makefile	
@@ -1,7 +1,7 @@
 VERSION = 2
 PATCHLEVEL = 6
 SUBLEVEL = 9
-EXTRAVERSION = -mm1-RT-V0.6.2
+EXTRAVERSION = -mm1-RT-V0.6.4
 NAME=Zonked Quokka
 
 # *DOCUMENTATION*
--- linux.old/kernel/exit.c	
+++ linux.new/kernel/exit.c	
@@ -840,9 +840,9 @@ asmlinkage NORET_TYPE void do_exit(long 
 #endif
 	check_no_held_locks(tsk);
 	/* PF_DEAD causes final put_task_struct after we schedule. */
-	wmb();
+	local_irq_disable();
 	tsk->flags |= PF_DEAD;
-	schedule();
+	__schedule();
 	BUG();
 	/* Avoid "noreturn function does return".  */
 	for (;;) ;
--- linux.old/kernel/sched.c	
+++ linux.new/kernel/sched.c	
@@ -599,13 +599,11 @@ static inline void enqueue_task_head(str
  *
  * Both properties are important to certain workloads.
  */
-static int effective_prio(task_t *p)
+
+static inline int __effective_prio(task_t *p)
 {
 	int bonus, prio;
 
-	if (rt_task(p))
-		return p->prio;
-
 	bonus = CURRENT_BONUS(p) - MAX_BONUS / 2;
 
 	prio = p->static_prio - bonus;
@@ -616,6 +614,13 @@ static int effective_prio(task_t *p)
 	return prio;
 }
 
+static int effective_prio(task_t *p)
+{
+	if (rt_task(p))
+		return p->prio;
+	return __effective_prio(p);
+}
+
 /*
  * __activate_task - move a task to the runqueue.
  */
@@ -3540,8 +3545,7 @@ int mutex_getprio(task_t *p)
 	if (p->policy != SCHED_NORMAL)
 		return MAX_USER_RT_PRIO-1 - p->rt_priority;
 	else
-//		return effective_prio(p);
-		return p->static_prio;
+		return __effective_prio(p);
 }
 
 /*
@@ -3559,15 +3563,14 @@ void mutex_setprio(task_t *p, int prio)
 
 	rq = task_rq_lock(p, &flags);
 
+	oldprio = p->prio;
 	array = p->array;
 	if (array)
-		deactivate_task(p, rq);
-	oldprio = p->prio;
-
+		dequeue_task(p, array);
 	p->prio = prio;
 
 	if (array) {
-		__activate_task(p, rq);
+		enqueue_task(p, array);
 		/*
 		 * Reschedule if we are currently running on this runqueue and
 		 * our priority decreased, or if we are not currently running on
--- linux.old/kernel/timer.c	
+++ linux.new/kernel/timer.c	
@@ -956,23 +956,25 @@ EXPORT_SYMBOL(xtime_lock);
  */
 static inline void update_times(void)
 {
+	unsigned long ticks = 0;
 	/*
 	 * First test outside the lock for performance reasons:
 	 */
-	if (jiffies - wall_jiffies) {
+	if (jiffies != wall_jiffies) {
 		unsigned long flags;
 
 		write_seqlock_irqsave(&xtime_lock, flags);
-		while (jiffies - wall_jiffies) {
+		while (jiffies != wall_jiffies) {
 			wall_jiffies++;
+			ticks++;
 			update_wall_time(1);
-			calc_load(1);
 			if (seqlock_need_resched(&xtime_lock)) {
 				write_sequnlock_irqrestore(&xtime_lock, flags);
 				cond_resched();
 				write_seqlock_irqsave(&xtime_lock, flags);
 			}
 		}
+		calc_load(ticks);
 		write_sequnlock_irqrestore(&xtime_lock, flags);
 	}
 }

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> i've uploaded -V0.6.3 to the usual place:
> 
>     http://redhat.com/~mingo/realtime-preempt/

hm, make it -V0.6.4. -V0.6.3 had a bug in where the system clock
'stopped' a couple of minutes after bootup, creating weird effects. (it
stopped when jiffies overflowed from negative territory they default to
at bootup, into positive numbers.)

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Thomas Gleixner]

On Mon, 2004-11-01 at 13:37 +0100, Ingo Molnar wrote:
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > i've uploaded -V0.6.3 to the usual place:
> > 
> >     http://redhat.com/~mingo/realtime-preempt/
> 
> hm, make it -V0.6.4. -V0.6.3 had a bug in where the system clock
> 'stopped' a couple of minutes after bootup, creating weird effects. (it
> stopped when jiffies overflowed from negative territory they default to
> at bootup, into positive numbers.)
> 
> 	Ingo

I'm doing some tests from my RT environment on 0.6.2. I'm quite sure,
that interrupts are sporadically disabled for > 200µs. Did you change
anything relevant to this between 0.6.2 and 0.6.4 ?
I also had a freeze of the machine. Sysrq-T was the only thing that
worked.

tglx


                                               sibling
  task             PC      pid father child younger older
init          [c113d6f0]D C03D9540     0     1      0     2
(NOTLB)
c113fe5c 00000086 c113d6f0 c03d9540 00000246 c113d6f0 00000001 c113fea4 
       c012007c c0326060 c113fea4 c01c9988 c113fea4 00001f7c 26c698f9
000002d0 
       c113d8a4 c113e000 c113fe98 0000000b c113fe80 c02d49fb 00000296
c0327ff4 
Call Trace:
 [<c012007c>] __mod_timer+0x11c/0x1e0 (36)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c02d49fb>] schedule+0x3b/0x130 (40)
 [<c02d5678>] schedule_timeout+0x88/0xe0 (36)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (60)
 [<c0120d30>] process_timeout+0x0/0x10 (20)
 [<c0169ec7>] do_select+0x197/0x2f0 (28)
 [<c0169b60>] __pollwait+0x0/0xd0 (84)
 [<c016a33b>] sys_select+0x2db/0x4f0 (32)
 [<c010602b>] syscall_call+0x7/0xb (92)
ksoftirqd/0   [c113d140]R C03D9540     0     2      1             3
(L-TLB)
c7fe3f88 00000046 c113d140 c03d9540 c7fe3f6c c0326370 c7fe3f6c c7fe2000 
       c0120a86 00000000 c0364580 00000000 c7fe2000 000000d5 02053ba5
000002d1 
       c113d2f4 c7fe2000 c7fe2000 00000000 c7fe3fac c02d49fb c011cad0
00000000 
Call Trace:
 [<c0120a86>] run_timer_softirq+0x2a6/0x460 (36)
 [<c02d49fb>] schedule+0x3b/0x130 (52)
 [<c011cad0>] ksoftirqd+0x0/0xe0 (4)
 [<c011cad0>] ksoftirqd+0x0/0xe0 (28)
 [<c011cb96>] ksoftirqd+0xc6/0xe0 (4)
 [<c012ca7a>] kthread+0xaa/0xb0 (20)
 [<c012c9d0>] kthread+0x0/0xb0 (28)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
desched/0     [c113cb90]D C03D9540     0     3      1             4
2 (L-TLB)
c7fe5f88 00000046 c113cb90 c03d9540 c013f0dd c7ff9a40 c1137ee4 c012d597 
       c7ff9a40 c1137ee4 c013f0dd c1137ee4 c135a0d0 00000107 95a45f2d
0000029e 
       c113cd44 c7fe4000 c7fe4000 00000000 c7fe5fac c02d49fb c2d1e7a4
c0116cf3 
Call Trace:
 [<c013f0dd>] kmem_cache_free+0x4d/0xe0 (20)
 [<c012d597>] _mutex_lock+0x17/0x20 (12)
 [<c013f0dd>] kmem_cache_free+0x4d/0xe0 (12)
 [<c02d49fb>] schedule+0x3b/0x130 (44)
 [<c0116cf3>] mmdrop_complete+0x73/0x110 (8)
 [<c0116de0>] desched_thread+0x0/0x90 (24)
 [<c0116e4e>] desched_thread+0x6e/0x90 (4)
 [<c012ca7a>] kthread+0xaa/0xb0 (20)
 [<c012c9d0>] kthread+0x0/0xb0 (28)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
events/0      [c113c5e0]R C03D9570     0     4      1             5
3 (L-TLB)
c7fedf20 00000046 c113d140 c03d9570 000002d1 00000000 00000246 c113c5e0 
       00000246 c113c5e0 015f136e 000002d1 c113d140 000002f7 015f3f88
000002d1 
       c113c794 c7fec000 c7fca740 c7fedf94 c7fedf44 c02d49fb c7fedf94
c7fca77c 
Call Trace:
 [<c02d49fb>] schedule+0x3b/0x130 (88)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (12)
 [<c0127fe8>] worker_thread+0x258/0x280 (24)
 [<c01f8f60>] console_callback+0x0/0xe0 (24)
 [<c0113460>] default_wake_function+0x0/0x30 (28)
 [<c0113460>] default_wake_function+0x0/0x30 (32)
 [<c0127d90>] worker_thread+0x0/0x280 (36)
 [<c012ca7a>] kthread+0xaa/0xb0 (4)
 [<c012c9d0>] kthread+0x0/0xb0 (28)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
khelper       [c113c030]D C03D9540     0     5      1            10
4 (L-TLB)
c7fcdf20 00000046 c113c030 c03d9540 c113c030 00000000 00000246 c113c030 
       00000246 c113c030 00000001 c7fca67c c113d140 00004693 32e90bc6
00000023 
       c113c1e4 c7fcc000 c7fca640 c7fcdf94 c7fcdf44 c02d49fb c7fcdf94
c7fca67c 
Call Trace:
 [<c02d49fb>] schedule+0x3b/0x130 (88)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (12)
 [<c0127fe8>] worker_thread+0x258/0x280 (24)
 [<c0127910>] __call_usermodehelper+0x0/0x70 (24)
 [<c0113460>] default_wake_function+0x0/0x30 (28)
 [<c0113460>] default_wake_function+0x0/0x30 (32)
 [<c0127d90>] worker_thread+0x0/0x280 (36)
 [<c012ca7a>] kthread+0xaa/0xb0 (4)
 [<c012c9d0>] kthread+0x0/0xb0 (28)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
kthread       [c7fd1710]D C03D9540     0    10      1    24      77
5 (L-TLB)
c7fd3f20 00000046 c7fd1710 c03d9540 c7fd1710 00000000 00000246 c7fd1710 
       00000246 c7fd1710 00000001 c7fca57c c7d05770 00001363 36d5ab71
00000023 
       c7fd18c4 c7fd2000 c7fca540 c7fd3f94 c7fd3f44 c02d49fb c7fd3f94
c7fca57c 
Call Trace:
 [<c02d49fb>] schedule+0x3b/0x130 (88)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (12)
 [<c0127fe8>] worker_thread+0x258/0x280 (24)
 [<c012ca80>] keventd_create_kthread+0x0/0x60 (24)
 [<c0113460>] default_wake_function+0x0/0x30 (28)
 [<c0113460>] default_wake_function+0x0/0x30 (32)
 [<c0127d90>] worker_thread+0x0/0x280 (36)
 [<c012ca7a>] kthread+0xaa/0xb0 (4)
 [<c012c9d0>] kthread+0x0/0xb0 (28)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
kblockd/0     [c7fd1160]D C03D9540     0    24     10            75
(L-TLB)
c7fdff20 00000046 c7fd1160 c03d9540 c7fd1160 00000000 00000246 c7fd1160 
       00000246 c7fd1160 00000001 c7fca37c c135ac30 0000228e 9bb8ebd4
00000190 
       c7fd1314 c7fde000 c7fca340 c7fdff94 c7fdff44 c02d49fb c7fdff94
c7fca37c 
Call Trace:
 [<c02d49fb>] schedule+0x3b/0x130 (88)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (12)
 [<c0127fe8>] worker_thread+0x258/0x280 (24)
 [<c0215410>] as_work_handler+0x0/0x60 (24)
 [<c0113460>] default_wake_function+0x0/0x30 (28)
 [<c0113460>] default_wake_function+0x0/0x30 (32)
 [<c0127d90>] worker_thread+0x0/0x280 (36)
 [<c012ca7a>] kthread+0xaa/0xb0 (4)
 [<c012c9d0>] kthread+0x0/0xb0 (28)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
pdflush       [c7fd0bb0]D C03D9540     0    75     10            76
24 (L-TLB)
c7f43f5c 00000046 c7fd0bb0 c03d9540 c013b0c8 c013bd30 00000246 c7fd0bb0 
       00000ccc 00002146 00000000 00000000 c0d28c30 00001fb6 bba27e42
00000190 
       c7fd0d64 c7f42000 c7f43fb0 c7f43fa4 c7f43f80 c02d49fb c0328500
00000001 
Call Trace:
 [<c013b0c8>] background_writeout+0x98/0xc0 (20)
 [<c013bd30>] pdflush+0x0/0x30 (4)
 [<c02d49fb>] schedule+0x3b/0x130 (64)
 [<c013bbff>] __pdflush+0x7f/0x1b0 (12)
 [<c012d5d7>] _mutex_lock_irq+0x17/0x20 (12)
 [<c013bd30>] pdflush+0x0/0x30 (8)
 [<c013bc04>] __pdflush+0x84/0x1b0 (4)
 [<c013bd56>] pdflush+0x26/0x30 (20)
 [<c013bd30>] pdflush+0x0/0x30 (40)
 [<c012ca7a>] kthread+0xaa/0xb0 (4)
 [<c012c9d0>] kthread+0x0/0xb0 (28)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
pdflush       [c7fd0600]D C03D9540     0    76     10            78
75 (L-TLB)
c7f57f5c 00000046 c7fd0600 c03d9540 002a8188 c013bd30 00000246 c7fd0600 
       c0328420 002a8188 0029f8d0 00000000 c113d140 000003b9 2698f358
000002d0 
       c7fd07b4 c7f56000 c7f57fb0 c7f57fa4 c7f57f80 c02d49fb c0328500
00000001 
Call Trace:
 [<c013bd30>] pdflush+0x0/0x30 (24)
 [<c02d49fb>] schedule+0x3b/0x130 (64)
 [<c013bbff>] __pdflush+0x7f/0x1b0 (12)
 [<c012d5d7>] _mutex_lock_irq+0x17/0x20 (12)
 [<c013bd30>] pdflush+0x0/0x30 (8)
 [<c013bc04>] __pdflush+0x84/0x1b0 (4)
 [<c013bd56>] pdflush+0x26/0x30 (20)
 [<c013bd30>] pdflush+0x0/0x30 (40)
 [<c012ca7a>] kthread+0xaa/0xb0 (4)
 [<c012c9d0>] kthread+0x0/0xb0 (28)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
aio/0         [c7f5b730]D C03D9540     0    78     10           664
76 (L-TLB)
c7f5df20 00000046 c7f5b730 c03d9540 00000000 00000000 00000246 c7f5b730 
       00000246 c7f5b730 00000001 c7fca07c c113d6f0 00003100 deb9886e
00000004 
       c7f5b8e4 c7f5c000 c7fca040 c7f5df94 c7f5df44 c02d49fb c7f5df94
c7fca07c 
Call Trace:
 [<c02d49fb>] schedule+0x3b/0x130 (88)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (12)
 [<c0127fe8>] worker_thread+0x258/0x280 (24)
 [<c0113460>] default_wake_function+0x0/0x30 (52)
 [<c0113460>] default_wake_function+0x0/0x30 (32)
 [<c0127d90>] worker_thread+0x0/0x280 (36)
 [<c012ca7a>] kthread+0xaa/0xb0 (4)
 [<c012c9d0>] kthread+0x0/0xb0 (28)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
kswapd0       [c7fd0050]D C03D9540     0    77      1           666
10 (L-TLB)
c7f59f60 00000046 c7fd0050 c03d9540 00000000 00000000 c0327ea0 00000246 
       c7fd0050 00000001 c03282ec c012cf9e c2314700 00000508 d7982974
00000190 
       c7fd0204 c7f58000 c7f59f98 c7f59fc0 c7f59f84 c02d49fb c03282ec
c012cf9e 
Call Trace:
 [<c012cf9e>] prepare_to_wait+0x5e/0x80 (48)
 [<c02d49fb>] schedule+0x3b/0x130 (40)
 [<c012cf9e>] prepare_to_wait+0x5e/0x80 (8)
 [<c0142cb3>] kswapd+0xd3/0xe0 (28)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (32)
 [<c0105f02>] ret_from_fork+0x6/0x14 (20)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (12)
 [<c0142be0>] kswapd+0x0/0xe0 (24)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
IRQ 8         [c7f5b180]D C03D9570     0   664     10           672
78 (L-TLB)
c7f71f80 00000046 c7d04c10 c03d9570 00000009 00000246 00000000 c01fb371 
       c03e8090 0000001d e9bf49d0 00000009 c7d04c10 000043a0 e9bf57a8
00000009 
       c7f5b334 c7f70000 c0398d80 c0398d80 c7f71fa4 c02d49fb 00000008
c0398d80 
Call Trace:
 [<c01fb371>] rtc_interrupt+0xe1/0x130 (32)
 [<c02d49fb>] schedule+0x3b/0x130 (56)
 [<c0133e60>] do_irqd+0x0/0xd0 (32)
 [<c0133f08>] do_irqd+0xa8/0xd0 (4)
 [<c012ca7a>] kthread+0xaa/0xb0 (28)
 [<c012c9d0>] kthread+0x0/0xb0 (28)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
IRQ 12        [c7f5a620]D C03D99E8     0   672     10           686
664 (L-TLB)
c7e0ff80 00000046 c113d6f0 c03d99e8 00000006 c1187400 000000fa 00000000 
       00000000 35fa4d60 0907682f 00000006 c113d6f0 000004f6 090772a7
00000006 
       c7f5a7d4 c7e0e000 c0398f00 c0398f00 c7e0ffa4 c02d49fb 0000000c
c0398f00 
Call Trace:
 [<c02d49fb>] schedule+0x3b/0x130 (88)
 [<c0133e60>] do_irqd+0x0/0xd0 (32)
 [<c0133f08>] do_irqd+0xa8/0xd0 (4)
 [<c012ca7a>] kthread+0xaa/0xb0 (28)
 [<c012c9d0>] kthread+0x0/0xb0 (28)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
IRQ 6         [c7f5a070]D C03D9540     0   686     10           709
672 (L-TLB)
c7e11f80 00000046 c7f5a070 c03d9540 0836a1c0 00000000 c021c610 c021c52b 
       c0303866 00000292 c7e15460 00000000 c113c5e0 0005a201 026a5625
00000005 
       c7f5a224 c7e10000 c0398cc0 c0398cc0 c7e11fa4 c02d49fb 00000006
c0398cc0 
Call Trace:
 [<c021c610>] reset_interrupt+0x0/0x90 (28)
 [<c021c52b>] floppy_interrupt+0x14b/0x1b0 (4)
 [<c02d49fb>] schedule+0x3b/0x130 (56)
 [<c0133e60>] do_irqd+0x0/0xd0 (32)
 [<c0133f08>] do_irqd+0xa8/0xd0 (4)
 [<c012ca7a>] kthread+0xaa/0xb0 (28)
 [<c012c9d0>] kthread+0x0/0xb0 (28)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
kseriod       [c7f5abd0]D C03D9540     0   666      1           778
77 (L-TLB)
c7e0df6c 00000046 c7f5abd0 c03d9540 00000000 00000246 c7f5abd0 00000246 
       c7f5abd0 00000001 c0366b40 c012cf9e 00000286 00008752 01c979f5
00000005 
       c7f5ad84 c7e0c000 c7e0c000 c7e0c000 c7e0df90 c02d49fb c0366b40
c012cf9e 
Call Trace:
 [<c012cf9e>] prepare_to_wait+0x5e/0x80 (48)
 [<c02d49fb>] schedule+0x3b/0x130 (40)
 [<c012cf9e>] prepare_to_wait+0x5e/0x80 (8)
 [<c01fcd3e>] serio_thread+0xce/0x110 (28)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (20)
 [<c0105f02>] ret_from_fork+0x6/0x14 (20)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (12)
 [<c01fcc70>] serio_thread+0x0/0x110 (24)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
IRQ 14        [c7e1d750]T C03D9540     0   709     10           741
686 (L-TLB)
c7e1fd98 00000046 c7e1d750 c03d9540 c03d9570 00000190 c03d9540 c6604330 
       c7e1fd84 c7e1fd94 00000086 c6604330 c6604330 00001fbe 9bc32661
00000190 
       c7e1d904 c7e1e000 00000000 c7e1d750 c7e1fdbc c02d49fb c03d9570
00000000 
Call Trace:
 [<c02d49fb>] schedule+0x3b/0x130 (88)
 [<c02d5c32>] down_write_mutex+0x102/0x1b0 (36)
 [<c013b98d>] test_clear_page_writeback+0x3d/0xe0 (20)
 [<c012d7bc>] _rw_mutex_write_lock_irqsave+0xc/0x10 (56)
 [<c013b98d>] test_clear_page_writeback+0x3d/0xe0 (4)
 [<c01572c0>] end_buffer_async_write+0x70/0x140 (8)
 [<c0134e59>] end_page_writeback+0x29/0x70 (28)
 [<c01572c8>] end_buffer_async_write+0x78/0x140 (20)
 [<c015ab9d>] bio_destructor+0x3d/0x90 (16)
 [<c015ae0d>] bio_put+0x3d/0x80 (24)
 [<c015a409>] end_bio_bh_io_sync+0x39/0x50 (20)
 [<c015bad3>] bio_endio+0x53/0x80 (16)
 [<c020f825>] __end_that_request_first+0x205/0x270 (28)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (16)
 [<c0239368>] __ide_end_request+0x78/0x1c0 (36)
 [<c0239515>] ide_end_request+0x65/0xc0 (36)
 [<c024226b>] ide_dma_intr+0x8b/0xb0 (36)
 [<c02421e0>] ide_dma_intr+0x0/0xb0 (20)
 [<c023b0b1>] ide_intr+0xe1/0x170 (4)
 [<c0133624>] handle_IRQ_event+0x44/0x80 (32)
 [<c0133dc3>] do_hardirq+0x63/0x100 (32)
 [<c0133e60>] do_irqd+0x0/0xd0 (32)
 [<c0133ef9>] do_irqd+0x99/0xd0 (4)
 [<c012ca7a>] kthread+0xaa/0xb0 (28)
 [<c012c9d0>] kthread+0x0/0xb0 (28)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
IRQ 1         [c7e1d1a0]D 00000000     0   741     10           779
709 (L-TLB)
00000015 00000000 00000000 00000000 c01fdff0 c1187200 00000046 00000000 
       00000000 15468ae0 00000001 c7e39de0 00000000 c0133624 00000001
c03e81c0 
       00000000 c0398ae0 00000001 c7e30000 00000000 c0133dc3 00000001
00000000 
Call Trace:
 [<c01fdff0>] i8042_interrupt+0xc0/0x1a0 (20)
 [<c0133624>] handle_IRQ_event+0x44/0x80 (36)
 [<c0133dc3>] do_hardirq+0x63/0x100 (32)
 [<c0133e60>] do_irqd+0x0/0xd0 (32)
 [<c0133ef9>] do_irqd+0x99/0xd0 (4)
 [<c012ca7a>] kthread+0xaa/0xb0 (28)
 [<c012c9d0>] kthread+0x0/0xb0 (28)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
kjournald     [c7e1cbf0]T C03D9540     0   778      1           929
666 (L-TLB)
c7e43d68 00000046 c7e1cbf0 c03d9540 00000000 c212b8c0 00000246 c7e1cbf0 
       c7e1cbf0 00000001 c7f75580 c012cf9e c53879f0 0000068b ae093765
00000191 
       c7e1cda4 c7e42000 c7f75580 c7e43df4 c7e43d8c c02d49fb c7f75414
00000001 
Call Trace:
 [<c012cf9e>] prepare_to_wait+0x5e/0x80 (48)
 [<c02d49fb>] schedule+0x3b/0x130 (40)
 [<c01adf3b>] journal_commit_transaction+0x156b/0x16c0 (12)
 [<c01adf40>] journal_commit_transaction+0x1570/0x16c0 (24)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (88)
 [<c0133624>] handle_IRQ_event+0x44/0x80 (12)
 [<c01cbde2>] __delay+0x12/0x20 (8)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (12)
 [<c02d4671>] __schedule+0x2d1/0x620 (136)
 [<c02d4b45>] preempt_schedule+0x55/0x70 (8)
 [<c02d4b45>] preempt_schedule+0x55/0x70 (72)
 [<c012d08f>] finish_wait+0x4f/0x70 (8)
 [<c01b0bf6>] kjournald+0xf6/0x280 (24)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c01b0bf6>] kjournald+0xf6/0x280 (12)
 [<c01b0c0c>] kjournald+0x10c/0x280 (24)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (40)
 [<c0119910>] exit_notify+0x350/0x980 (12)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (20)
 [<c0105f02>] ret_from_fork+0x6/0x14 (68)
 [<c01b0b00>] kjournald+0x0/0x280 (8)
 [<c01b0ae0>] commit_timeout+0x0/0x10 (8)
 [<c01b0b00>] kjournald+0x0/0x280 (20)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
IRQ 4         [c7e1c640]D C03D9540     0   779     10           994
741 (L-TLB)
c7e8df80 00000046 c7e1c640 c03d9540 c03e8300 00000002 00000000 00000001 
       00000060 c7e4b5e0 00000000 00000000 c682ccf0 00000505 16bd0447
0000000b 
       c7e1c7f4 c7e8c000 c0398c00 c0398c00 c7e8dfa4 c02d49fb 00000004
c0398c00 
Call Trace:
 [<c02d49fb>] schedule+0x3b/0x130 (88)
 [<c0133e60>] do_irqd+0x0/0xd0 (32)
 [<c0133f08>] do_irqd+0xa8/0xd0 (4)
 [<c012ca7a>] kthread+0xaa/0xb0 (28)
 [<c012c9d0>] kthread+0x0/0xb0 (28)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
kjournald     [c135a680]T C03D9540     0   929      1          1135
778 (L-TLB)
c787dcdc 00000046 c135a680 c03d9540 00000246 c1100570 00000246 c135a680 
       00000000 c135a834 c787c000 ffffffff c135a680 000002c4 66b1a4ec
0000019d 
       c135a834 c787c000 00000000 c787dd68 c787dd00 c02d49fb c036f180
00000001 
Call Trace:
 [<c02d49fb>] schedule+0x3b/0x130 (88)
 [<c020d46f>] blk_backing_dev_unplug+0x1f/0x30 (12)
 [<c020d431>] generic_unplug_device+0x21/0x40 (12)
 [<c02d559e>] io_schedule+0xe/0x20 (12)
 [<c0156701>] sync_buffer+0x31/0x50 (8)
 [<c02d57e7>] __wait_on_bit+0x67/0x70 (12)
 [<c01566d0>] sync_buffer+0x0/0x50 (8)
 [<c01566d0>] sync_buffer+0x0/0x50 (16)
 [<c02d5883>] out_of_line_wait_on_bit+0x93/0xa0 (4)
 [<c012d110>] wake_bit_function+0x0/0x60 (28)
 [<c012d110>] wake_bit_function+0x0/0x60 (32)
 [<c01ad908>] journal_commit_transaction+0xf38/0x16c0 (32)
 [<c01cbde2>] __delay+0x12/0x20 (84)
 [<c888d031>] lpptest_irq+0x31/0x50 [lpptest] (8)
 [<c0133624>] handle_IRQ_event+0x44/0x80 (8)
 [<c0133775>] __do_IRQ+0x115/0x160 (32)
 [<c0112f7f>] scheduler_tick+0x1f/0x470 (60)
 [<c01127d8>] recalc_task_prio+0xa8/0x1a0 (12)
 [<c02d4671>] __schedule+0x2d1/0x620 (52)
 [<c02d4b45>] preempt_schedule+0x55/0x70 (8)
 [<c02d4b45>] preempt_schedule+0x55/0x70 (72)
 [<c012d08f>] finish_wait+0x4f/0x70 (8)
 [<c01b0bf6>] kjournald+0xf6/0x280 (24)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c01b0bf6>] kjournald+0xf6/0x280 (12)
 [<c01b0c0c>] kjournald+0x10c/0x280 (24)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (40)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (32)
 [<c0105f02>] ret_from_fork+0x6/0x14 (68)
 [<c01b0b00>] kjournald+0x0/0x280 (8)
 [<c01b0ae0>] commit_timeout+0x0/0x10 (8)
 [<c01b0b00>] kjournald+0x0/0x280 (20)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
IRQ 3         [c135b1e0]D C03D9540     0   994     10          1282
779 (L-TLB)
c1379f80 00000046 c135b1e0 c03d9540 c03e82d0 00000002 00000001 00000001 
       00000060 c7e4bd20 00000000 00000000 c7d04c10 00002227 b5d387fe
00000009 
       c135b394 c1378000 c0398ba0 c0398ba0 c1379fa4 c02d49fb 00000003
c0398ba0 
Call Trace:
 [<c02d49fb>] schedule+0x3b/0x130 (88)
 [<c0133e60>] do_irqd+0x0/0xd0 (32)
 [<c0133f08>] do_irqd+0xa8/0xd0 (4)
 [<c012ca7a>] kthread+0xaa/0xb0 (28)
 [<c012c9d0>] kthread+0x0/0xb0 (28)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
portmap       [c7e1c090]D C03D9540     0  1135      1          1199
929 (NOTLB)
c7cd5eb8 00000086 c7e1c090 c03d9540 c7e1c090 00000282 c0328040 00000282 
       00000000 00000000 00000246 00000246 c7e1c090 000202dc efd174bd
0000000a 
       c7e1c244 c7cd4000 7fffffff c7cd5f68 c7cd5edc c02d49fb c77fb028
c12caa78 
Call Trace:
 [<c02d49fb>] schedule+0x3b/0x130 (88)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (24)
 [<c02d56c4>] schedule_timeout+0xd4/0xe0 (12)
 [<c0292b97>] tcp_poll+0x37/0x190 (12)
 [<c015647b>] fget+0x4b/0x70 (20)
 [<c02674a9>] sock_poll+0x29/0x40 (28)
 [<c016a5ab>] do_pollfd+0x5b/0xa0 (20)
 [<c016a698>] do_poll+0xa8/0xd0 (28)
 [<c016a821>] sys_poll+0x161/0x240 (48)
 [<c0169b60>] __pollwait+0x0/0xd0 (36)
 [<c010602b>] syscall_call+0x7/0xb (32)
syslogd       [c78600f0]T C03D9540     0  1199      1          1206
1135 (NOTLB)
c78afc7c 00000086 c78600f0 c03d9540 00000000 c7f7547c 00000246 c78600f0 
       c78600f0 00000001 c7f7547c c012cf9e 00000282 00008fb6 2ad9eccb
00000239 
       c78602a4 c78ae000 00000000 c78afd18 c78afca0 c02d49fb c7f75414
00000001 
Call Trace:
 [<c012cf9e>] prepare_to_wait+0x5e/0x80 (48)
 [<c02d49fb>] schedule+0x3b/0x130 (40)
 [<c01a9891>] start_this_handle+0x2a1/0x410 (12)
 [<c01a9896>] start_this_handle+0x2a6/0x410 (24)
 [<c013ee9c>] kmem_cache_alloc+0x4c/0xe0 (56)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (16)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (32)
 [<c01a9b1b>] journal_start+0xcb/0xf0 (40)
 [<c0107deb>] do_IRQ+0x2b/0x40 (20)
 [<c019d3e9>] ext3_dirty_inode+0x39/0x90 (16)
 [<c01786a6>] __mark_inode_dirty+0x1e6/0x1f0 (24)
 [<c0136c83>] remove_suid+0x3/0x90 (16)
 [<c0133775>] __do_IRQ+0x115/0x160 (20)
 [<c0171f50>] inode_update_time+0xd0/0xe0 (24)
 [<c01377f0>] generic_file_aio_write_nolock+0x230/0x4c0 (48)
 [<c0137b23>] generic_file_write_nolock+0xa3/0xc0 (100)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (112)
 [<c0105e5d>] do_signal+0xcd/0x130 (16)
 [<c0137e14>] generic_file_writev+0x44/0xb0 (24)
 [<c0155a0c>] do_readv_writev+0x22c/0x280 (44)
 [<c0155360>] do_sync_write+0x0/0xf0 (40)
 [<c01cbde2>] __delay+0x12/0x20 (52)
 [<c888d031>] lpptest_irq+0x31/0x50 [lpptest] (8)
 [<c0155b28>] vfs_writev+0x58/0x70 (24)
 [<c0155c11>] sys_writev+0x51/0x80 (24)
 [<c010602b>] syscall_call+0x7/0xb (40)
klogd         [c7d04c10]D C03D9540     0  1206      1          1214
1199 (NOTLB)
c7d15d08 00000082 c7d04c10 c03d9540 00000060 00000246 c7d04c10 c032829c 
       c0327ff4 000000d0 00000000 00000001 c113d6f0 00089689 bda57400
000002a9 
       c7d04dc4 c7d14000 7fffffff 00000001 c7d15d2c c02d49fb c7d14000
c7d14000 
Call Trace:
 [<c02d49fb>] schedule+0x3b/0x130 (88)
 [<c02d56c4>] schedule_timeout+0xd4/0xe0 (36)
 [<c012d01d>] prepare_to_wait_exclusive+0x5d/0x80 (32)
 [<c02c7a7e>] unix_wait_for_peer+0xae/0xe0 (28)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c02c7a7e>] unix_wait_for_peer+0xae/0xe0 (12)
 [<c02c7aa7>] unix_wait_for_peer+0xd7/0xe0 (24)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (16)
 [<c01cc432>] copy_from_user+0x42/0x70 (12)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (20)
 [<c02c85a2>] unix_dgram_sendmsg+0x282/0x5f0 (40)
 [<c0134fde>] find_get_page+0x3e/0x50 (48)
 [<c0145f51>] do_no_page+0x1b1/0x3b0 (32)
 [<c0266fa6>] sock_aio_write+0xf6/0x120 (20)
 [<c0145f51>] do_no_page+0x1b1/0x3b0 (28)
 [<c0155417>] do_sync_write+0xb7/0xf0 (84)
 [<c01118e9>] do_page_fault+0x409/0x610 (24)
 [<c01118e9>] do_page_fault+0x409/0x610 (4)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (88)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c015554f>] vfs_write+0xff/0x130 (36)
 [<c0155651>] sys_write+0x51/0x80 (44)
 [<c010602b>] syscall_call+0x7/0xb (40)
inetd         [c135b790]D C03D9540     0  1214      1          1218
1206 (NOTLB)
c135de5c 00000082 c135b790 c03d9540 00000292 00000292 c0328040 00000292 
       c0139999 0000000d 00000246 c135b790 c135a0d0 000d1e8d 3dd989aa
0000000a 
       c135b944 c135c000 7fffffff 0000000e c135de80 c02d49fb c012ce1f
c7661ae4 
Call Trace:
 [<c0139999>] buffered_rmqueue+0x89/0xf0 (36)
 [<c02d49fb>] schedule+0x3b/0x130 (52)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (4)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c02d56c4>] schedule_timeout+0xd4/0xe0 (20)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (16)
 [<c0292b97>] tcp_poll+0x37/0x190 (24)
 [<c015647b>] fget+0x4b/0x70 (20)
 [<c02674a9>] sock_poll+0x29/0x40 (28)
 [<c0169ec7>] do_select+0x197/0x2f0 (20)
 [<c0169b60>] __pollwait+0x0/0xd0 (84)
 [<c016a33b>] sys_select+0x2db/0x4f0 (32)
 [<c010602b>] syscall_call+0x7/0xb (92)
lpd           [c78606a0]D C03D9540     0  1218      1          1272
1214 (NOTLB)
c78cde5c 00000086 c78606a0 c03d9540 c10ed420 c0327ff4 c01c9988 c0328040 
       00000001 00000006 00000246 c78606a0 00000286 00085c44 51ea0297
0000000a 
       c7860854 c78cc000 7fffffff 00000007 c78cde80 c02d49fb c012ce1f
c7661364 
Call Trace:
 [<c01c9988>] up_write_mutex+0x38/0xc0 (28)
 [<c02d49fb>] schedule+0x3b/0x130 (60)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (4)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c02d56c4>] schedule_timeout+0xd4/0xe0 (20)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (16)
 [<c0292b97>] tcp_poll+0x37/0x190 (24)
 [<c015647b>] fget+0x4b/0x70 (20)
 [<c02674a9>] sock_poll+0x29/0x40 (28)
 [<c0169ec7>] do_select+0x197/0x2f0 (20)
 [<c0169b60>] __pollwait+0x0/0xd0 (84)
 [<c016a33b>] sys_select+0x2db/0x4f0 (32)
 [<c0148d72>] sys_munmap+0x42/0x50 (68)
 [<c010602b>] syscall_call+0x7/0xb (24)
nfsd          [c76d37f0]D C03D9540     0  1272      1          1273
1218 (L-TLB)
c76e1eb0 00000046 c76d37f0 c03d9540 00000246 c76d37f0 00000001 c76e1ef8 
       c012007c c0326060 c76e1ef8 c01c9988 c76d3240 000046bc a7bad305
0000000a 
       c76d39a4 c76e0000 c76e1eec c76fd278 c76e1ed4 c02d49fb 00000292
c79c1b14 
Call Trace:
 [<c012007c>] __mod_timer+0x11c/0x1e0 (36)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c02d49fb>] schedule+0x3b/0x130 (40)
 [<c02d5678>] schedule_timeout+0x88/0xe0 (36)
 [<c883f907>] svc_recv+0x3a7/0x540 [sunrpc] (60)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c0120d30>] process_timeout+0x0/0x10 (8)
 [<c883f910>] svc_recv+0x3b0/0x540 [sunrpc] (28)
 [<c02d4b45>] preempt_schedule+0x55/0x70 (16)
 [<c0113460>] default_wake_function+0x0/0x30 (20)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c0113460>] default_wake_function+0x0/0x30 (20)
 [<c0123e7d>] sigprocmask+0x6d/0xe0 (16)
 [<c8865359>] nfsd+0x109/0x3c0 [nfsd] (24)
 [<c8865250>] nfsd+0x0/0x3c0 [nfsd] (48)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
nfsd          [c76d3240]D C03D9540     0  1273      1          1274
1272 (L-TLB)
c76e3eb0 00000046 c76d3240 c03d9540 00000246 c76d3240 00000001 c76e3ef8 
       c012007c c0326060 c76e3ef8 c01c9988 c76d2130 00002394 a7baf699
0000000a 
       c76d33f4 c76e2000 c76e3eec c76fd278 c76e3ed4 c02d49fb 00000292
00000000 
Call Trace:
 [<c012007c>] __mod_timer+0x11c/0x1e0 (36)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c02d49fb>] schedule+0x3b/0x130 (40)
 [<c02d5678>] schedule_timeout+0x88/0xe0 (36)
 [<c883f907>] svc_recv+0x3a7/0x540 [sunrpc] (60)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c0120d30>] process_timeout+0x0/0x10 (8)
 [<c883f910>] svc_recv+0x3b0/0x540 [sunrpc] (28)
 [<c02d4b45>] preempt_schedule+0x55/0x70 (16)
 [<c0113460>] default_wake_function+0x0/0x30 (20)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c0113460>] default_wake_function+0x0/0x30 (20)
 [<c0123e7d>] sigprocmask+0x6d/0xe0 (16)
 [<c8865359>] nfsd+0x109/0x3c0 [nfsd] (24)
 [<c8865250>] nfsd+0x0/0x3c0 [nfsd] (48)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
nfsd          [c76d2130]D C03D9540     0  1274      1          1275
1273 (L-TLB)
c6c33eb0 00000046 c76d2130 c03d9540 00000246 c76d2130 00000001 c6c33ef8 
       c012007c c0326060 c6c33ef8 c01c9988 c76d26e0 000020ea a7bb1783
0000000a 
       c76d22e4 c6c32000 c6c33eec c76fd278 c6c33ed4 c02d49fb 00000292
00000000 
Call Trace:
 [<c012007c>] __mod_timer+0x11c/0x1e0 (36)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c02d49fb>] schedule+0x3b/0x130 (40)
 [<c02d5678>] schedule_timeout+0x88/0xe0 (36)
 [<c883f907>] svc_recv+0x3a7/0x540 [sunrpc] (60)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c0120d30>] process_timeout+0x0/0x10 (8)
 [<c883f910>] svc_recv+0x3b0/0x540 [sunrpc] (28)
 [<c02d4b45>] preempt_schedule+0x55/0x70 (16)
 [<c0113460>] default_wake_function+0x0/0x30 (20)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c0113460>] default_wake_function+0x0/0x30 (20)
 [<c0123e7d>] sigprocmask+0x6d/0xe0 (16)
 [<c8865359>] nfsd+0x109/0x3c0 [nfsd] (24)
 [<c8865250>] nfsd+0x0/0x3c0 [nfsd] (48)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
nfsd          [c76d26e0]D C03D9540     0  1275      1          1276
1274 (L-TLB)
c73fbeb0 00000046 c76d26e0 c03d9540 00000246 c76d26e0 00000001 c73fbef8 
       c012007c c0326060 c73fbef8 c01c9988 c76d2c90 000020ea a7bb386d
0000000a 
       c76d2894 c73fa000 c73fbeec c76fd278 c73fbed4 c02d49fb 00000292
00000000 
Call Trace:
 [<c012007c>] __mod_timer+0x11c/0x1e0 (36)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c02d49fb>] schedule+0x3b/0x130 (40)
 [<c02d5678>] schedule_timeout+0x88/0xe0 (36)
 [<c883f907>] svc_recv+0x3a7/0x540 [sunrpc] (60)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c0120d30>] process_timeout+0x0/0x10 (8)
 [<c883f910>] svc_recv+0x3b0/0x540 [sunrpc] (28)
 [<c02d4b45>] preempt_schedule+0x55/0x70 (16)
 [<c0113460>] default_wake_function+0x0/0x30 (20)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c0113460>] default_wake_function+0x0/0x30 (20)
 [<c0123e7d>] sigprocmask+0x6d/0xe0 (16)
 [<c8865359>] nfsd+0x109/0x3c0 [nfsd] (24)
 [<c8865250>] nfsd+0x0/0x3c0 [nfsd] (48)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
nfsd          [c76d2c90]D C03D9540     0  1276      1          1277
1275 (L-TLB)
c73c3eb0 00000046 c76d2c90 c03d9540 00000246 c76d2c90 00000001 c73c3ef8 
       c012007c c0326060 c73c3ef8 c01c9988 c6c4b810 0000221b a7bb5a88
0000000a 
       c76d2e44 c73c2000 c73c3eec c76fd278 c73c3ed4 c02d49fb 00000292
00000000 
Call Trace:
 [<c012007c>] __mod_timer+0x11c/0x1e0 (36)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c02d49fb>] schedule+0x3b/0x130 (40)
 [<c02d5678>] schedule_timeout+0x88/0xe0 (36)
 [<c883f907>] svc_recv+0x3a7/0x540 [sunrpc] (60)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c0120d30>] process_timeout+0x0/0x10 (8)
 [<c883f910>] svc_recv+0x3b0/0x540 [sunrpc] (28)
 [<c02d4b45>] preempt_schedule+0x55/0x70 (16)
 [<c0113460>] default_wake_function+0x0/0x30 (20)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c0113460>] default_wake_function+0x0/0x30 (20)
 [<c0123e7d>] sigprocmask+0x6d/0xe0 (16)
 [<c8865359>] nfsd+0x109/0x3c0 [nfsd] (24)
 [<c8865250>] nfsd+0x0/0x3c0 [nfsd] (48)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
nfsd          [c6c4b810]D C03D9540     0  1277      1          1278
1276 (L-TLB)
c6c4deb0 00000046 c6c4b810 c03d9540 00000246 c6c4b810 00000001 c6c4def8 
       c012007c c0326060 c6c4def8 c01c9988 c6c4b260 0000189c a7bbaf55
0000000a 
       c6c4b9c4 c6c4c000 c6c4deec c76fd278 c6c4ded4 c02d49fb 00000292
00000000 
Call Trace:
 [<c012007c>] __mod_timer+0x11c/0x1e0 (36)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c02d49fb>] schedule+0x3b/0x130 (40)
 [<c02d5678>] schedule_timeout+0x88/0xe0 (36)
 [<c883f907>] svc_recv+0x3a7/0x540 [sunrpc] (60)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c0120d30>] process_timeout+0x0/0x10 (8)
 [<c883f910>] svc_recv+0x3b0/0x540 [sunrpc] (28)
 [<c0113460>] default_wake_function+0x0/0x30 (36)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c0113460>] default_wake_function+0x0/0x30 (20)
 [<c0123e7d>] sigprocmask+0x6d/0xe0 (16)
 [<c8865359>] nfsd+0x109/0x3c0 [nfsd] (24)
 [<c8865250>] nfsd+0x0/0x3c0 [nfsd] (48)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
nfsd          [c6c4b260]D C03D9540     0  1278      1          1279
1277 (L-TLB)
c7113eb0 00000046 c6c4b260 c03d9540 00000246 c6c4b260 00000001 c7113ef8 
       c012007c c0326060 c7113ef8 c01c9988 c7113ef8 000032c2 ab0b7866
0000000a 
       c6c4b414 c7112000 c7113eec c76fd278 c7113ed4 c02d49fb 00000292
c79c1b14 
Call Trace:
 [<c012007c>] __mod_timer+0x11c/0x1e0 (36)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c02d49fb>] schedule+0x3b/0x130 (40)
 [<c02d5678>] schedule_timeout+0x88/0xe0 (36)
 [<c883f907>] svc_recv+0x3a7/0x540 [sunrpc] (60)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c0120d30>] process_timeout+0x0/0x10 (8)
 [<c883f910>] svc_recv+0x3b0/0x540 [sunrpc] (28)
 [<c0113460>] default_wake_function+0x0/0x30 (36)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c0113460>] default_wake_function+0x0/0x30 (20)
 [<c0123e7d>] sigprocmask+0x6d/0xe0 (16)
 [<c8865359>] nfsd+0x109/0x3c0 [nfsd] (24)
 [<c8865250>] nfsd+0x0/0x3c0 [nfsd] (48)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
nfsd          [c6c4acb0]D C03D9540     0  1279      1          1281
1278 (L-TLB)
c6cbbeb0 00000046 c6c4acb0 c03d9540 00000246 c6c4acb0 00000001 c6cbbef8 
       c012007c c0326060 c6cbbef8 c01c9988 c6c4b260 00018d87 ab0b45a4
0000000a 
       c6c4ae64 c6cba000 c6cbbeec c76fd278 c6cbbed4 c02d49fb 00000292
c01c9988 
Call Trace:
 [<c012007c>] __mod_timer+0x11c/0x1e0 (36)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c02d49fb>] schedule+0x3b/0x130 (40)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (8)
 [<c02d5678>] schedule_timeout+0x88/0xe0 (28)
 [<c883f907>] svc_recv+0x3a7/0x540 [sunrpc] (60)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c0120d30>] process_timeout+0x0/0x10 (8)
 [<c883f910>] svc_recv+0x3b0/0x540 [sunrpc] (28)
 [<c0113460>] default_wake_function+0x0/0x30 (36)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c0113460>] default_wake_function+0x0/0x30 (20)
 [<c0123e7d>] sigprocmask+0x6d/0xe0 (16)
 [<c8865359>] nfsd+0x109/0x3c0 [nfsd] (24)
 [<c8865250>] nfsd+0x0/0x3c0 [nfsd] (48)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
lockd         [c6c4a700]D C03D9540     0  1281      1          1285
1279 (L-TLB)
c6c8becc 00000046 c6c4a700 c03d9540 c6c8be8c c6c8be8c c79c0108 00000246 
       c6c4a700 00000000 00000000 00000000 c76d37f0 00005dec a7b980d4
0000000a 
       c6c4a8b4 c6c8a000 7fffffff c76fd178 c6c8bef0 c02d49fb c01c9988
c79c0114 
Call Trace:
 [<c02d49fb>] schedule+0x3b/0x130 (88)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (4)
 [<c02d56c4>] schedule_timeout+0xd4/0xe0 (32)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (36)
 [<c883f907>] svc_recv+0x3a7/0x540 [sunrpc] (24)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c883f907>] svc_recv+0x3a7/0x540 [sunrpc] (12)
 [<c883f910>] svc_recv+0x3b0/0x540 [sunrpc] (24)
 [<c0113460>] default_wake_function+0x0/0x30 (36)
 [<c01ca189>] up+0x69/0xc0 (20)
 [<c0113460>] default_wake_function+0x0/0x30 (12)
 [<c8a0e574>] lockd+0x134/0x2b0 [lockd] (40)
 [<c8a0e440>] lockd+0x0/0x2b0 [lockd] (20)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
rpciod/0      [c6c4a150]D C03D9540     0  1282     10          1471
994 (L-TLB)
c6c65f20 00000046 c6c4a150 c03d9540 0000000a 00000000 00000246 c6c4a150 
       00000246 c6c4a150 00000001 c75295dc c6c4a700 00001be9 a7b922e8
0000000a 
       c6c4a304 c6c64000 c75295a0 c6c65f94 c6c65f44 c02d49fb c6c65f94
c75295dc 
Call Trace:
 [<c02d49fb>] schedule+0x3b/0x130 (88)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (12)
 [<c0127fe8>] worker_thread+0x258/0x280 (24)
 [<c0113460>] default_wake_function+0x0/0x30 (52)
 [<c0113460>] default_wake_function+0x0/0x30 (32)
 [<c0127d90>] worker_thread+0x0/0x280 (36)
 [<c012ca7a>] kthread+0xaa/0xb0 (4)
 [<c012c9d0>] kthread+0x0/0xb0 (28)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
rpc.mountd    [c6f49830]D C03D9540     0  1285      1          1309
1281 (NOTLB)
c6f4be5c 00000082 c6f49830 c03d9540 c10de0e0 c0327ff4 c01c9988 c0328040 
       00000001 00000004 00000246 c6f49830 c135a0d0 00082460 af6c86af
0000000a 
       c6f499e4 c6f4a000 7fffffff 00000005 c6f4be80 c02d49fb c012ce1f
c6d47884 
Call Trace:
 [<c01c9988>] up_write_mutex+0x38/0xc0 (28)
 [<c02d49fb>] schedule+0x3b/0x130 (60)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (4)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c02d56c4>] schedule_timeout+0xd4/0xe0 (20)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (16)
 [<c0292b97>] tcp_poll+0x37/0x190 (24)
 [<c015647b>] fget+0x4b/0x70 (20)
 [<c02674a9>] sock_poll+0x29/0x40 (28)
 [<c0169ec7>] do_select+0x197/0x2f0 (20)
 [<c0169b60>] __pollwait+0x0/0xd0 (84)
 [<c016a33b>] sys_select+0x2db/0x4f0 (32)
 [<c010602b>] syscall_call+0x7/0xb (92)
sshd          [c6f48cd0]D C03D9540     0  1309      1  1893    1315
1285 (NOTLB)
c6f71e5c 00000086 c6f48cd0 c03d9540 00000292 00000292 c0328040 00000292 
       c0139999 00000003 00000246 c6f48cd0 c6988800 0000944a fb0b23a8
0000005a 
       c6f48e84 c6f70000 7fffffff 00000004 c6f71e80 c02d49fb c012ce1f
c6d47604 
Call Trace:
 [<c0139999>] buffered_rmqueue+0x89/0xf0 (36)
 [<c02d49fb>] schedule+0x3b/0x130 (52)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (4)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c02d56c4>] schedule_timeout+0xd4/0xe0 (20)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (16)
 [<c0292b97>] tcp_poll+0x37/0x190 (24)
 [<c015647b>] fget+0x4b/0x70 (20)
 [<c02674a9>] sock_poll+0x29/0x40 (28)
 [<c0169ec7>] do_select+0x197/0x2f0 (20)
 [<c0169b60>] __pollwait+0x0/0xd0 (84)
 [<c016a33b>] sys_select+0x2db/0x4f0 (32)
 [<c02d4ba8>] preempt_schedule_irq+0x48/0x70 (72)
 [<c010602b>] syscall_call+0x7/0xb (20)
rpc.statd     [c6f49280]D C03D9540     0  1315      1          1318
1309 (NOTLB)
c70c7e5c 00000082 c6f49280 c03d9540 c10e0f80 c0327ff4 c01c9988 c0328040 
       00000001 00000006 00000246 c6f49280 00000286 0001de0e efe0198d
0000000a 
       c6f49434 c70c6000 7fffffff 00000007 c70c7e80 c02d49fb c012ce1f
c6fc6b24 
Call Trace:
 [<c01c9988>] up_write_mutex+0x38/0xc0 (28)
 [<c02d49fb>] schedule+0x3b/0x130 (60)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (4)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c02d56c4>] schedule_timeout+0xd4/0xe0 (20)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (16)
 [<c0292b97>] tcp_poll+0x37/0x190 (24)
 [<c015647b>] fget+0x4b/0x70 (20)
 [<c02674a9>] sock_poll+0x29/0x40 (28)
 [<c0169ec7>] do_select+0x197/0x2f0 (20)
 [<c0169b60>] __pollwait+0x0/0xd0 (84)
 [<c016a33b>] sys_select+0x2db/0x4f0 (32)
 [<c010602b>] syscall_call+0x7/0xb (92)
atd           [c6f48720]D C03D9540     0  1318      1          1321
1315 (NOTLB)
c6f97ef8 00000086 c6f48720 c03d9540 00000246 c6f48720 00000001 c6f97f40 
       c012007c c0326060 c6f97f40 c01c9988 c7d040b0 000313e4 f3b1a1b5
0000000a 
       c6f488d4 c6f96000 c6f97f34 000f41a7 c6f97f1c c02d49fb 00000282
00000246 
Call Trace:
 [<c012007c>] __mod_timer+0x11c/0x1e0 (36)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c02d49fb>] schedule+0x3b/0x130 (40)
 [<c02d5678>] schedule_timeout+0x88/0xe0 (36)
 [<c0123e7d>] sigprocmask+0x6d/0xe0 (64)
 [<c0120d30>] process_timeout+0x0/0x10 (16)
 [<c0120e2e>] sys_nanosleep+0xde/0x170 (28)
 [<c010602b>] syscall_call+0x7/0xb (52)
cron          [c6f48170]T C03D9540     0  1321      1          1327
1318 (NOTLB)
c680fce0 00000082 c6f48170 c03d9540 00000246 c1100508 00000246 c6f48170 
       00000000 00000000 c7ffb680 00000000 c553ec50 00036951 3041657d
00000192 
       c6f48324 c680e000 00000000 c680fd6c c680fd04 c02d49fb c036f180
00000001 
Call Trace:
 [<c02d49fb>] schedule+0x3b/0x130 (88)
 [<c020d46f>] blk_backing_dev_unplug+0x1f/0x30 (12)
 [<c020d431>] generic_unplug_device+0x21/0x40 (12)
 [<c02d559e>] io_schedule+0xe/0x20 (12)
 [<c0156701>] sync_buffer+0x31/0x50 (8)
 [<c02d57e7>] __wait_on_bit+0x67/0x70 (12)
 [<c01566d0>] sync_buffer+0x0/0x50 (8)
 [<c01566d0>] sync_buffer+0x0/0x50 (16)
 [<c02d5883>] out_of_line_wait_on_bit+0x93/0xa0 (4)
 [<c012d110>] wake_bit_function+0x0/0x60 (28)
 [<c012d110>] wake_bit_function+0x0/0x60 (32)
 [<c019e846>] ext3_find_entry+0x196/0x440 (32)
 [<c016f267>] d_alloc+0x1b7/0x240 (40)
 [<c016f6f7>] __d_lookup+0x97/0x130 (60)
 [<c019ed71>] ext3_lookup+0x41/0xd0 (40)
 [<c0163955>] real_lookup+0xd5/0x100 (28)
 [<c0163cd6>] do_lookup+0x96/0xb0 (32)
 [<c01644dc>] link_path_walk+0x7ec/0xfc0 (32)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (48)
 [<c0164fd9>] path_lookup+0x99/0x1b0 (36)
 [<c01652b3>] __user_walk+0x33/0x60 (28)
 [<c015f78f>] vfs_stat+0x1f/0x60 (28)
 [<c02d5682>] schedule_timeout+0x92/0xe0 (24)
 [<c015fedb>] sys_stat64+0x1b/0x40 (64)
 [<c0120d30>] process_timeout+0x0/0x10 (16)
 [<c0120e2e>] sys_nanosleep+0xde/0x170 (28)
 [<c010ae4a>] do_gettimeofday+0x1a/0xd0 (8)
 [<c011bd2b>] sys_time+0x1b/0x60 (20)
 [<c010602b>] syscall_call+0x7/0xb (24)
bash          [c7d04660]D C03D9540     0  1327      1 12577    1328
1321 (NOTLB)
c1233ef8 00000086 c7d04660 c03d9540 c13b07ec c7d04704 00000246 c7d04660 
       00000246 c7d04660 00000001 00000086 c012ce1f 0000ebf9 3ccd43c6
0000019c 
       c7d04814 c1232000 c7d04660 fffffe00 c1233f1c c02d49fb c0324d60
00000001 
Call Trace:
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (52)
 [<c02d49fb>] schedule+0x3b/0x130 (36)
 [<c011b42b>] do_wait+0x1bb/0x4f0 (12)
 [<c011b459>] do_wait+0x1e9/0x4f0 (24)
 [<c0113460>] default_wake_function+0x0/0x30 (44)
 [<c0123e7d>] sigprocmask+0x6d/0xe0 (20)
 [<c0113460>] default_wake_function+0x0/0x30 (12)
 [<c01cc3de>] copy_to_user+0x3e/0x50 (12)
 [<c011b82d>] sys_wait4+0x3d/0x50 (28)
 [<c011b865>] sys_waitpid+0x25/0x30 (24)
 [<c010602b>] syscall_call+0x7/0xb (20)
bash          [c7d040b0]D C03D9540     0  1328      1  2302    1329
1327 (NOTLB)
c1243ef8 00000086 c7d040b0 c03d9540 c78f282c c7d04154 00000246 c7d040b0 
       00000246 c7d040b0 00000001 00000086 c4d9f300 000802fc e5d1231c
00000064 
       c7d04264 c1242000 c7d040b0 fffffe00 c1243f1c c02d49fb c0324d60
00000001 
Call Trace:
 [<c02d49fb>] schedule+0x3b/0x130 (88)
 [<c011b42b>] do_wait+0x1bb/0x4f0 (12)
 [<c011b459>] do_wait+0x1e9/0x4f0 (24)
 [<c0113460>] default_wake_function+0x0/0x30 (44)
 [<c0123e7d>] sigprocmask+0x6d/0xe0 (20)
 [<c0113460>] default_wake_function+0x0/0x30 (12)
 [<c01cc3de>] copy_to_user+0x3e/0x50 (12)
 [<c011b82d>] sys_wait4+0x3d/0x50 (28)
 [<c011b865>] sys_waitpid+0x25/0x30 (24)
 [<c010602b>] syscall_call+0x7/0xb (20)
bash          [c135a0d0]T C03D9540     0  1329      1          1330
1328 (NOTLB)
c7847ce0 00000086 c135a0d0 c03d9540 00000246 c1100f98 00000246 c135a0d0 
       00000000 00000000 c7ffb680 00000000 00000246 00016576 95b0edcf
0000029e 
       c135a284 c7846000 00000000 c7847d6c c7847d04 c02d49fb c036f180
00000001 
Call Trace:
 [<c02d49fb>] schedule+0x3b/0x130 (88)
 [<c020d46f>] blk_backing_dev_unplug+0x1f/0x30 (12)
 [<c020d431>] generic_unplug_device+0x21/0x40 (12)
 [<c02d559e>] io_schedule+0xe/0x20 (12)
 [<c0156701>] sync_buffer+0x31/0x50 (8)
 [<c02d57e7>] __wait_on_bit+0x67/0x70 (12)
 [<c01566d0>] sync_buffer+0x0/0x50 (8)
 [<c01566d0>] sync_buffer+0x0/0x50 (16)
 [<c02d5883>] out_of_line_wait_on_bit+0x93/0xa0 (4)
 [<c012d110>] wake_bit_function+0x0/0x60 (28)
 [<c012d110>] wake_bit_function+0x0/0x60 (32)
 [<c019e846>] ext3_find_entry+0x196/0x440 (32)
 [<c016f267>] d_alloc+0x1b7/0x240 (40)
 [<c016f6f7>] __d_lookup+0x97/0x130 (60)
 [<c019ed71>] ext3_lookup+0x41/0xd0 (40)
 [<c0163955>] real_lookup+0xd5/0x100 (28)
 [<c0163cd6>] do_lookup+0x96/0xb0 (32)
 [<c0163e15>] link_path_walk+0x125/0xfc0 (32)
 [<c0164fd9>] path_lookup+0x99/0x1b0 (84)
 [<c01652b3>] __user_walk+0x33/0x60 (28)
 [<c015f78f>] vfs_stat+0x1f/0x60 (28)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (24)
 [<c012d5d7>] _mutex_lock_irq+0x17/0x20 (24)
 [<c01cc3de>] copy_to_user+0x3e/0x50 (20)
 [<c015fedb>] sys_stat64+0x1b/0x40 (20)
 [<c0123f76>] sys_rt_sigprocmask+0x86/0x120 (48)
 [<c010ae4a>] do_gettimeofday+0x1a/0xd0 (4)
 [<c01114e0>] do_page_fault+0x0/0x610 (28)
 [<c01061d5>] error_code+0x2d/0x38 (8)
 [<c010602b>] syscall_call+0x7/0xb (8)
getty         [c682d850]T C03D9540     0  1330      1          1331
1329 (NOTLB)
c682fd10 00000082 c682d850 c03d9540 00000246 c1101410 00000246 c682d850 
       00000000 00000000 c7ffb680 00000000 00000246 0003b5b2 e0ecd1d1
000002a0 
       c682da04 c682e000 00000000 c682fd9c c682fd34 c02d49fb c036f180
00000001 
Call Trace:
 [<c02d49fb>] schedule+0x3b/0x130 (88)
 [<c020d46f>] blk_backing_dev_unplug+0x1f/0x30 (12)
 [<c020d431>] generic_unplug_device+0x21/0x40 (12)
 [<c02d559e>] io_schedule+0xe/0x20 (12)
 [<c0156701>] sync_buffer+0x31/0x50 (8)
 [<c02d57e7>] __wait_on_bit+0x67/0x70 (12)
 [<c01566d0>] sync_buffer+0x0/0x50 (8)
 [<c01566d0>] sync_buffer+0x0/0x50 (16)
 [<c02d5883>] out_of_line_wait_on_bit+0x93/0xa0 (4)
 [<c012d110>] wake_bit_function+0x0/0x60 (28)
 [<c012d110>] wake_bit_function+0x0/0x60 (32)
 [<c019e846>] ext3_find_entry+0x196/0x440 (32)
 [<c016f267>] d_alloc+0x1b7/0x240 (40)
 [<c016f6f7>] __d_lookup+0x97/0x130 (60)
 [<c019ed71>] ext3_lookup+0x41/0xd0 (40)
 [<c0163955>] real_lookup+0xd5/0x100 (28)
 [<c0163cd6>] do_lookup+0x96/0xb0 (32)
 [<c01644dc>] link_path_walk+0x7ec/0xfc0 (32)
 [<c0164fd9>] path_lookup+0x99/0x1b0 (84)
 [<c0160990>] open_exec+0x30/0x100 (28)
 [<c01e7c76>] tty_write+0xb6/0x260 (4)
 [<c013ee9c>] kmem_cache_alloc+0x4c/0xe0 (24)
 [<c013ee9c>] kmem_cache_alloc+0x4c/0xe0 (24)
 [<c013ee9c>] kmem_cache_alloc+0x4c/0xe0 (24)
 [<c0161b13>] do_execve+0x13/0x260 (20)
 [<c0104b36>] sys_execve+0x46/0xb0 (36)
 [<c010602b>] syscall_call+0x7/0xb (32)
getty         [c682d2a0]D C03D9540     0  1331      1          1332
1330 (NOTLB)
c1281e10 00000082 c682d2a0 c03d9540 00000000 00000000 00000286 00000046 
       c682d2a0 c682d2a0 00000001 c0328040 c01391e3 0007b645 53bd158d
0000000b 
       c682d454 c1280000 7fffffff c1281f04 c1281e34 c02d49fb c01ca189
00000286 
Call Trace:
 [<c01391e3>] free_pages_bulk+0x1e3/0x300 (52)
 [<c02d49fb>] schedule+0x3b/0x130 (36)
 [<c01ca189>] up+0x69/0xc0 (4)
 [<c02d56c4>] schedule_timeout+0xd4/0xe0 (32)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (36)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (12)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (24)
 [<c01ed789>] read_chan+0x699/0x7a0 (24)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (20)
 [<c01e6c25>] tty_ldisc_try+0x45/0x60 (12)
 [<c0113460>] default_wake_function+0x0/0x30 (52)
 [<c01c9d6e>] down+0x2e/0x140 (12)
 [<c0113460>] default_wake_function+0x0/0x30 (20)
 [<c01e7b96>] tty_read+0xf6/0x120 (40)
 [<c0155522>] vfs_write+0xd2/0x130 (20)
 [<c01552e8>] vfs_read+0xb8/0x130 (24)
 [<c01555d1>] sys_read+0x51/0x80 (44)
 [<c010602b>] syscall_call+0x7/0xb (40)
getty         [c682ccf0]D C03D9540     0  1332      1         12175
1331 (NOTLB)
c1273e10 00000082 c682ccf0 c03d9540 00000000 00000000 00000286 00000046 
       c682ccf0 c682ccf0 00000001 c0328040 c7d04660 0007cdc6 539abfcc
0000000b 
       c682cea4 c1272000 7fffffff c1273f04 c1273e34 c02d49fb c01ca189
00000286 
Call Trace:
 [<c02d49fb>] schedule+0x3b/0x130 (88)
 [<c01ca189>] up+0x69/0xc0 (4)
 [<c02d56c4>] schedule_timeout+0xd4/0xe0 (32)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (36)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (12)
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (24)
 [<c01ed789>] read_chan+0x699/0x7a0 (24)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (20)
 [<c01e6c25>] tty_ldisc_try+0x45/0x60 (12)
 [<c0113460>] default_wake_function+0x0/0x30 (52)
 [<c01c9d6e>] down+0x2e/0x140 (12)
 [<c0113460>] default_wake_function+0x0/0x30 (20)
 [<c01e7b96>] tty_read+0xf6/0x120 (40)
 [<c0155522>] vfs_write+0xd2/0x130 (20)
 [<c01552e8>] vfs_read+0xb8/0x130 (24)
 [<c01555d1>] sys_read+0x51/0x80 (44)
 [<c010602b>] syscall_call+0x7/0xb (40)
IRQ 5         [c7860c50]R C03D9570     0  1471     10          1486
1282 (L-TLB)
c7a73f80 00000046 c113d140 c03d9570 000002ce c7a73f64 c0112932 c113d140 
       c03d9570 000002ce 8a5ae53c 000002ce c113d140 00000420 8a5ae838
000002ce 
       c7860e04 c7a72000 c0398c60 c0398c60 c7a73fa4 c02d49fb c0398c60
c7a73fa4 
Call Trace:
 [<c0112932>] activate_task+0x62/0x80 (28)
 [<c02d49fb>] schedule+0x3b/0x130 (60)
 [<c0112a83>] wake_up_process+0x23/0x30 (12)
 [<c0133e60>] do_irqd+0x0/0xd0 (20)
 [<c0133f08>] do_irqd+0xa8/0xd0 (4)
 [<c012ca7a>] kthread+0xaa/0xb0 (28)
 [<c012c9d0>] kthread+0x0/0xb0 (28)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
IRQ 7         [c7d051c0]D C03D9540     0  1486     10
1471 (L-TLB)
c6865f9c 00000046 c7d051c0 c03d9540 00000246 c7d051c0 00000001 c6aefea8 
       c0113652 00000296 c6865fc0 c01c9988 c7fd1710 000011a2 36d58ac6
00000023 
       c7d05374 c6864000 c6aefe9c c0398d20 c6865fc0 c02d49fb 00000296
00000001 
Call Trace:
 [<c0113652>] complete+0x52/0x60 (36)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c02d49fb>] schedule+0x3b/0x130 (40)
 [<c0133e60>] do_irqd+0x0/0xd0 (32)
 [<c012ca40>] kthread+0x70/0xb0 (4)
 [<c012c9d0>] kthread+0x0/0xb0 (28)
 [<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
sshd          [c7d05770]D C03D9540     0  1893   1309  1896
(NOTLB)
c6aefcf0 00000082 c7d05770 c03d9540 00000001 c013f0dd c7f50ec0 c7f00ce4 
       c012d597 c7f50ec0 c7f00ce4 c013f0dd c62d67a0 00046315 fff1f1f7
0000005a 
       c7d05924 c6aee000 7fffffff c12cc9a0 c6aefd14 c02d49fb c0159141
00000000 
Call Trace:
 [<c013f0dd>] kmem_cache_free+0x4d/0xe0 (24)
 [<c012d597>] _mutex_lock+0x17/0x20 (12)
 [<c013f0dd>] kmem_cache_free+0x4d/0xe0 (12)
 [<c02d49fb>] schedule+0x3b/0x130 (40)
 [<c0159141>] __block_commit_write+0x91/0xc0 (4)
 [<c019a5b3>] ext3_journal_dirty_data+0x23/0x70 (8)
 [<c02d56c4>] schedule_timeout+0xd4/0xe0 (24)
 [<c012cf9e>] prepare_to_wait+0x5e/0x80 (32)
 [<c02c9112>] unix_stream_data_wait+0xd2/0x120 (28)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c02c9112>] unix_stream_data_wait+0xd2/0x120 (12)
 [<c02c9119>] unix_stream_data_wait+0xd9/0x120 (24)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (20)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (32)
 [<c02c95a2>] unix_stream_recvmsg+0x442/0x4a0 (16)
 [<c02c95b6>] unix_stream_recvmsg+0x456/0x4a0 (24)
 [<c0266e96>] sock_aio_read+0xf6/0x110 (116)
 [<c01551f7>] do_sync_read+0xb7/0xf0 (116)
 [<c012d6fd>] atomic_dec_and_mutex_lock+0x4d/0x70 (68)
 [<c012d6fd>] atomic_dec_and_mutex_lock+0x4d/0x70 (24)
 [<c012d56a>] __mutex_lock+0x2a/0x40 (12)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (12)
 [<c016e423>] dput+0x33/0x2b0 (20)
 [<c015532f>] vfs_read+0xff/0x130 (28)
 [<c01555d1>] sys_read+0x51/0x80 (44)
 [<c010602b>] syscall_call+0x7/0xb (40)
sshd          [c135ac30]D C03D9540     0  1896   1893  1897
(NOTLB)
c5337e5c 00000086 c135ac30 c03d9540 00000000 00000292 c5337eb0 00000246 
       c135ac30 c56fdcc0 00000246 c135ac30 00000000 00012732 7bd4c399
000002a4 
       c135ade4 c5336000 7fffffff 0000000a c5337e80 c02d49fb c01c9988
c0363020 
Call Trace:
 [<c02d49fb>] schedule+0x3b/0x130 (88)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (4)
 [<c011354c>] __wake_up+0x4c/0x60 (12)
 [<c02d56c4>] schedule_timeout+0xd4/0xe0 (20)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c01e6d73>] tty_ldisc_deref+0x43/0xb0 (12)
 [<c01e6d73>] tty_ldisc_deref+0x43/0xb0 (24)
 [<c01e940b>] tty_poll+0x8b/0xc0 (24)
 [<c0169ec7>] do_select+0x197/0x2f0 (36)
 [<c0169b60>] __pollwait+0x0/0xd0 (84)
 [<c016a33b>] sys_select+0x2db/0x4f0 (32)
 [<c010602b>] syscall_call+0x7/0xb (92)
bash          [c4d8ed30]D C03D9540     0  1897   1896  2299
(NOTLB)
c4db5ef8 00000086 c4d8ed30 c03d9540 c64fbc2c c4d8edd4 00000246 c4d8ed30 
       00000246 c4d8ed30 00000001 00000086 c68bc8a0 00075636 3fecedf3
00000061 
       c4d8eee4 c4db4000 c4d8ed30 fffffe00 c4db5f1c c02d49fb c0324d60
00000001 
Call Trace:
 [<c02d49fb>] schedule+0x3b/0x130 (88)
 [<c011b42b>] do_wait+0x1bb/0x4f0 (12)
 [<c011b459>] do_wait+0x1e9/0x4f0 (24)
 [<c0113460>] default_wake_function+0x0/0x30 (44)
 [<c0123e7d>] sigprocmask+0x6d/0xe0 (20)
 [<c0113460>] default_wake_function+0x0/0x30 (12)
 [<c01cc3de>] copy_to_user+0x3e/0x50 (12)
 [<c011b82d>] sys_wait4+0x3d/0x50 (28)
 [<c011b865>] sys_waitpid+0x25/0x30 (24)
 [<c010602b>] syscall_call+0x7/0xb (20)
bash          [c5040130]D C03D9540     0  2299   1897 12578
(NOTLB)
c454def8 00000082 c5040130 c03d9540 c5735c4c c50401d4 00000246 c5040130 
       00000246 c5040130 00000001 00000086 c012ce1f 000066d2 abd7f311
0000019e 
       c50402e4 c454c000 c5040130 fffffe00 c454df1c c02d49fb c0324d60
00000001 
Call Trace:
 [<c012ce1f>] add_wait_queue+0x3f/0x50 (52)
 [<c02d49fb>] schedule+0x3b/0x130 (36)
 [<c011b42b>] do_wait+0x1bb/0x4f0 (12)
 [<c011b459>] do_wait+0x1e9/0x4f0 (24)
 [<c01127d8>] recalc_task_prio+0xa8/0x1a0 (20)
 [<c0113460>] default_wake_function+0x0/0x30 (24)
 [<c0123e7d>] sigprocmask+0x6d/0xe0 (20)
 [<c0113460>] default_wake_function+0x0/0x30 (12)
 [<c01cc3de>] copy_to_user+0x3e/0x50 (12)
 [<c011b82d>] sys_wait4+0x3d/0x50 (28)
 [<c011b865>] sys_waitpid+0x25/0x30 (24)
 [<c010602b>] syscall_call+0x7/0xb (20)
top           [c53fe150]T C03D9540     0  2302   1328
(NOTLB)
c5edbba8 00000082 c53fe150 c03d9540 00000246 00000000 00000246 c53fe150 
       00000000 00000000 c01127d8 42e4e86d c53fecb0 00006272 d78f69cb
00000190 
       c53fe304 c5eda000 c5edbc34 c1100508 c5edbbcc c02d49fb c036f180
00000001 
Call Trace:
 [<c01127d8>] recalc_task_prio+0xa8/0x1a0 (44)
 [<c02d49fb>] schedule+0x3b/0x130 (44)
 [<c020d46f>] blk_backing_dev_unplug+0x1f/0x30 (12)
 [<c020d431>] generic_unplug_device+0x21/0x40 (12)
 [<c02d559e>] io_schedule+0xe/0x20 (12)
 [<c0156701>] sync_buffer+0x31/0x50 (8)
 [<c02d58ee>] __wait_on_bit_lock+0x5e/0x70 (12)
 [<c01566d0>] sync_buffer+0x0/0x50 (8)
 [<c01566d0>] sync_buffer+0x0/0x50 (16)
 [<c02d5993>] out_of_line_wait_on_bit_lock+0x93/0xa0 (4)
 [<c012d110>] wake_bit_function+0x0/0x60 (28)
 [<c012d110>] wake_bit_function+0x0/0x60 (32)
 [<c0156758>] __lock_buffer+0x38/0x40 (32)
 [<c01aa804>] do_get_write_access+0x834/0x850 (12)
 [<c01583cb>] __getblk+0x2b/0x60 (72)
 [<c019c4ba>] ext3_get_inode_loc+0x5a/0x260 (28)
 [<c01aa857>] journal_get_write_access+0x37/0x50 (24)
 [<c019d2d5>] ext3_reserve_inode_write+0x55/0xd0 (28)
 [<c019d37b>] ext3_mark_inode_dirty+0x2b/0x60 (40)
 [<c013d162>] do_page_cache_readahead+0xc2/0x1c0 (28)
 [<c019d43d>] ext3_dirty_inode+0x8d/0x90 (16)
 [<c01786a6>] __mark_inode_dirty+0x1e6/0x1f0 (24)
 [<c013d162>] do_page_cache_readahead+0xc2/0x1c0 (8)
 [<c0171e79>] update_atime+0xd9/0xe0 (52)
 [<c01356ee>] do_generic_mapping_read+0x2ae/0x5a0 (48)
 [<c0135cdb>] __generic_file_aio_read+0x20b/0x240 (116)
 [<c01359e0>] file_read_actor+0x0/0xf0 (24)
 [<c013f0dd>] kmem_cache_free+0x4d/0xe0 (8)
 [<c0135d6b>] generic_file_aio_read+0x5b/0xa0 (52)
 [<c01551f7>] do_sync_read+0xb7/0xf0 (40)
 [<c016d37b>] fcntl_setlk+0x11b/0x2f0 (64)
 [<c012d597>] _mutex_lock+0x17/0x20 (44)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (8)
 [<c015647b>] fget+0x4b/0x70 (24)
 [<c01552e8>] vfs_read+0xb8/0x130 (24)
 [<c0168a77>] do_fcntl+0xf7/0x1b0 (24)
 [<c01555d1>] sys_read+0x51/0x80 (20)
 [<c010602b>] syscall_call+0x7/0xb (40)
grep          [c6604330]T C03D9540     0 12175      1
1332 (NOTLB)
c3289d2c 00000082 c6604330 c03d9540 00000190 c113d140 00032f46 9c11a953 
       00000190 c66044e4 c3288000 ffffffff c62dc880 00000d0c 9c11f78e
00000190 
       c66044e4 c3288000 c3289db0 c1101bc8 c3289d50 c02d49fb c66048c8
c3289db0 
Call Trace:
 [<c02d49fb>] schedule+0x3b/0x130 (88)
 [<c020007b>] uart_set_info+0x29b/0x4d0 (24)
 [<c02d559e>] io_schedule+0xe/0x20 (12)
 [<c01347a5>] sync_page+0x35/0x50 (8)
 [<c02d58ee>] __wait_on_bit_lock+0x5e/0x70 (8)
 [<c0134770>] sync_page+0x0/0x50 (8)
 [<c012d110>] wake_bit_function+0x0/0x60 (4)
 [<c0134f31>] __lock_page+0x91/0xa0 (16)
 [<c012d110>] wake_bit_function+0x0/0x60 (24)
 [<c012d110>] wake_bit_function+0x0/0x60 (32)
 [<c0134fde>] find_get_page+0x3e/0x50 (12)
 [<c013591d>] do_generic_mapping_read+0x4dd/0x5a0 (20)
 [<c0135cdb>] __generic_file_aio_read+0x20b/0x240 (116)
 [<c01359e0>] file_read_actor+0x0/0xf0 (24)
 [<c0135d6b>] generic_file_aio_read+0x5b/0xa0 (60)
 [<c01551f7>] do_sync_read+0xb7/0xf0 (40)
 [<c015fea8>] cp_new_stat64+0xf8/0x110 (24)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (92)
 [<c01552e8>] vfs_read+0xb8/0x130 (48)
 [<c01555d1>] sys_read+0x51/0x80 (44)
 [<c010602b>] syscall_call+0x7/0xb (40)
bash          [c62dc880]T C03D9540     0 12577   1327
(NOTLB)
c5573be0 00000082 c62dc880 c03d9540 00000000 c01c9988 c1100508 00000246 
       c62dc880 00000001 c1101854 c012cf9e c7d04660 000c5ac8 3cc49ca5
0000019c 
       c62dca34 c5572000 c79cc6bc c5573c5c c5573c04 c02d49fb c1101854
c012cf9e 
Call Trace:
 [<c01c9988>] up_write_mutex+0x38/0xc0 (24)
 [<c012cf9e>] prepare_to_wait+0x5e/0x80 (24)
 [<c02d49fb>] schedule+0x3b/0x130 (40)
 [<c012cf9e>] prepare_to_wait+0x5e/0x80 (8)
 [<c01aa61c>] do_get_write_access+0x64c/0x850 (28)
 [<c012d110>] wake_bit_function+0x0/0x60 (60)
 [<c01583cb>] __getblk+0x2b/0x60 (12)
 [<c012d110>] wake_bit_function+0x0/0x60 (20)
 [<c01aa857>] journal_get_write_access+0x37/0x50 (32)
 [<c019d2d5>] ext3_reserve_inode_write+0x55/0xd0 (28)
 [<c019d37b>] ext3_mark_inode_dirty+0x2b/0x60 (40)
 [<c013d162>] do_page_cache_readahead+0xc2/0x1c0 (28)
 [<c019d43d>] ext3_dirty_inode+0x8d/0x90 (16)
 [<c01786a6>] __mark_inode_dirty+0x1e6/0x1f0 (24)
 [<c013d1ca>] do_page_cache_readahead+0x12a/0x1c0 (8)
 [<c0171e79>] update_atime+0xd9/0xe0 (52)
 [<c01356ee>] do_generic_mapping_read+0x2ae/0x5a0 (48)
 [<c0135cdb>] __generic_file_aio_read+0x20b/0x240 (116)
 [<c01359e0>] file_read_actor+0x0/0xf0 (24)
 [<c0135d6b>] generic_file_aio_read+0x5b/0xa0 (60)
 [<c01551f7>] do_sync_read+0xb7/0xf0 (40)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (116)
 [<c01552e8>] vfs_read+0xb8/0x130 (48)
 [<c0160aae>] kernel_read+0x4e/0x60 (44)
 [<c016172a>] prepare_binprm+0xca/0xf0 (36)
 [<c0161c15>] do_execve+0x115/0x260 (32)
 [<c0104b36>] sys_execve+0x46/0xb0 (36)
 [<c010602b>] syscall_call+0x7/0xb (32)
bash          [c47fb8f0]T C03D9540     0 12578   2299
(NOTLB)
c4ae5be0 00000086 c47fb8f0 c03d9540 00000000 c01c9988 c1100508 00000246 
       c47fb8f0 00000001 c1101854 c012cf9e c5040130 00075001 abcf7095
0000019e 
       c47fbaa4 c4ae4000 c79cc6bc c4ae5c5c c4ae5c04 c02d49fb c1101854
c012cf9e 
Call Trace:
 [<c01c9988>] up_write_mutex+0x38/0xc0 (24)
 [<c012cf9e>] prepare_to_wait+0x5e/0x80 (24)
 [<c02d49fb>] schedule+0x3b/0x130 (40)
 [<c012cf9e>] prepare_to_wait+0x5e/0x80 (8)
 [<c01aa61c>] do_get_write_access+0x64c/0x850 (28)
 [<c01a975a>] start_this_handle+0x16a/0x410 (8)
 [<c01c9988>] up_write_mutex+0x38/0xc0 (12)
 [<c012d110>] wake_bit_function+0x0/0x60 (40)
 [<c01583cb>] __getblk+0x2b/0x60 (12)
 [<c012d110>] wake_bit_function+0x0/0x60 (20)
 [<c01aa857>] journal_get_write_access+0x37/0x50 (32)
 [<c019d2d5>] ext3_reserve_inode_write+0x55/0xd0 (28)
 [<c019d37b>] ext3_mark_inode_dirty+0x2b/0x60 (40)
 [<c013d162>] do_page_cache_readahead+0xc2/0x1c0 (28)
 [<c019d43d>] ext3_dirty_inode+0x8d/0x90 (16)
 [<c01786a6>] __mark_inode_dirty+0x1e6/0x1f0 (24)
 [<c013d162>] do_page_cache_readahead+0xc2/0x1c0 (8)
 [<c0171e79>] update_atime+0xd9/0xe0 (52)
 [<c01356ee>] do_generic_mapping_read+0x2ae/0x5a0 (48)
 [<c0135cdb>] __generic_file_aio_read+0x20b/0x240 (116)
 [<c01359e0>] file_read_actor+0x0/0xf0 (24)
 [<c0135d6b>] generic_file_aio_read+0x5b/0xa0 (60)
 [<c01551f7>] do_sync_read+0xb7/0xf0 (40)
 [<c012d0b0>] autoremove_wake_function+0x0/0x60 (116)
 [<c01552e8>] vfs_read+0xb8/0x130 (48)
 [<c0160aae>] kernel_read+0x4e/0x60 (44)
 [<c016172a>] prepare_binprm+0xca/0xf0 (36)
 [<c0161c15>] do_execve+0x115/0x260 (32)
 [<c0104b36>] sys_execve+0x46/0xb0 (36)
 [<c010602b>] syscall_call+0x7/0xb (32)

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

> i got a deadlock though. it was a weird one. mouse and keyboard
> [including sysrq] froze. but the find / i started kept on running in
> an xterm. had to press reset due to lack of second machine..

i've uploaded -V0.6.3 to the usual place:

    http://redhat.com/~mingo/realtime-preempt/

which fixes two issues:

 - priorities of SCHED_OTHER tasks not getting properly managed, hence 
   they could end up 'starving' other SCHED_OTHER tasks pretty
   indefinitely. This could possibly solve the 'temporary lockup'
   problem reported by Mark H Johnson.

 - fixed the 'high load average' bug

i dont know whether this will solve the 'hard lockups' reported though. 
It could solve your problem because the 'find /' kept running so it
wasnt a hard lockup. I'll keep testing and i'll fix any problem i can
reproduce myself.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Magnus Määttä]

Hi Ingo,

On Sunday 31 October 2004 14.40, Ingo Molnar wrote:
> * Ingo Molnar <mingo@elte.hu> wrote:
>
> i've just uploaded V0.6.2 that fixes a console-unblanking-timer
> thinko. This bug was present for quite some time, but this is the
> first time it triggered on my testbox - might be more common on
> others.

I'm trying to boot V0.6.2 on my laptop, which doesn't have a serial 
port, and I got a hardlock on boot. So I setup netconsole, which also 
hardlocks the machine, so I'm back to U9.3 for now..
It's a different hardlock when booting without netconsole. SysRq keys 
doesn't work.

Config:
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
CONFIG_PREEMPT_REALTIME=y
CONFIG_PREEMPT_BKL=y
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_PREEMPT=y
CONFIG_PREEMPT_TIMING=y
CONFIG_PREEMPT_TRACE=y
CONFIG_DEBUG_STACKOVERFLOW=y
CONFIG_DEBUG_STACK_USAGE=y

No USB or PARPORT support in kernel.

Here's the boot log until it hardlock:

Linux version 2.6.9-mm1-RT-V0.6.2 (root@barbara) (gcc version 3.3.3 
(SuSE Linux)) #12 SMP Mon Nov 1 01:57:57 CET 2004
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
 BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000ce000 - 00000000000d0000 (reserved)
 BIOS-e820: 00000000000d8000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000001f6e0000 (usable)
 BIOS-e820: 000000001f6e0000 - 000000001f6eb000 (ACPI data)
 BIOS-e820: 000000001f6eb000 - 000000001f700000 (ACPI NVS)
 BIOS-e820: 000000001f700000 - 0000000020000000 (reserved)
 BIOS-e820: 00000000fec10000 - 00000000fec20000 (reserved)
 BIOS-e820: 00000000ff800000 - 00000000ffc00000 (reserved)
 BIOS-e820: 00000000fffffc00 - 0000000100000000 (reserved)
502MB LOWMEM available.
DMI present.
ACPI: PM-Timer IO Port: 0x1008
Built 1 zonelists
No local APIC present or hardware disabled
Initializing CPU#0
Kernel command line: vga=0x314 resume=/dev/hda9 nmi_watchdog=1 
netconsole=1234@10.0.3.6/eth0,4567@10.0.2.1/00:50:BA:C2:11:34
netconsole: local port 1234
netconsole: local IP 10.0.3.6
netconsole: interface eth0
netconsole: remote port 4567
netconsole: remote IP 10.0.2.1
netconsole: remote ethernet address 00:50:ba:c2:11:34
PID hash table entries: 2048 (order: 11, 32768 bytes)
Detected 1500.066 MHz processor.
Using pmtmr for high-res timesource
Console: colour dummy device 80x25
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Memory: 502708k/514944k available (2931k kernel code, 11848k reserved, 
807k data, 228k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor 
mode... Ok.
Security Framework v1.0.0 initialized
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 1024K
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... OK.
ACPI: IRQ9 SCI: Edge set to Level Trigger.
CPU0: Intel(R) Pentium(R) M processor 1500MHz stepping 05
per-CPU timeslice cutoff: 2924.02 usecs.
task migration cache decay timeout: 3 msecs.
SMP motherboard not detected.
Local APIC not detected. Using dummy APIC emulation.
spawn_desched_task(00000000)
desched cpu_callback 3/00000000
ksoftirqd started up.
softirq RT prio: 24.
desched cpu_callback 2/00000000
Brought up 1 CPUs
desched thread 0 started up.
NET: Registered protocol family 16
PCI: PCI BIOS revision 2.10 entry at 0xfd6c4, last bus=3
PCI: Using configuration type 1
mtrr: v2.0 (20020519)
ACPI: Subsystem revision 20040816
ACPI: Interpreter enabled
ACPI: Using PIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (00:00)
PCI: Probing PCI hardware (bus 00)
PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.1
PCI: Transparent bridge - 0000:00:1e.0
ACPI: PCI Interrupt Link [LNKA] (IRQs 10 *11)
ACPI: PCI Interrupt Link [LNKB] (IRQs *10 11)
ACPI: PCI Interrupt Link [LNKC] (IRQs 10 *11)
ACPI: PCI Interrupt Link [LNKD] (IRQs *11)
ACPI: PCI Interrupt Link [LNKE] (IRQs *10 11)
ACPI: PCI Interrupt Link [LNKF] (IRQs *10 11)
ACPI: PCI Interrupt Link [LNKG] (IRQs *10 11)
ACPI: PCI Interrupt Link [LNKH] (IRQs 10 *11)
ACPI: Embedded Controller [EC] (gpe 28)
Linux Plug and Play Support v0.97 (c) Adam Belay
PCI: Using ACPI for IRQ routing
** PCI interrupts are no longer routed automatically.  If this
** causes a device to stop working, it is probably because the
** driver failed to call pci_enable_device().  As a temporary
** workaround, the "pci=routeirq" argument restores the old
** behavior.  If this argument makes the device work again,
** please email the output of "lspci" to bjorn.helgaas@hp.com
** so I can fix the driver.
Simple Boot Flag at 0x36 set to 0x1
apm: BIOS version 1.2 Flags 0x03 (Driver version 1.16ac)
apm: overridden by ACPI.
audit: initializing netlink socket (disabled)
audit(1099274447.119:0): initialized
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
Initializing Cryptographic API
vesafb: framebuffer at 0xe8000000, mapped to 0xe0080000, using 1875k, 
total 8000k
vesafb: mode is 800x600x16, linelength=1600, pages=7
vesafb: protected mode interface info at 00ff:44f0
vesafb: scrolling: redraw
vesafb: Truecolor: size=0:5:6:5, shift=0:11:5:0
Console: switching to colour frame buffer device 100x37
fb0: VESA VGA frame buffer device
isapnp: Scanning for PnP cards...
isapnp: No Plug & Play device found
Real Time Clock Driver v1.12
ACPI: PS/2 Keyboard Controller [PS2K] at I/O 0x60, 0x64, irq 1
ACPI: PS/2 Mouse Controller [PS2M] at irq 12
i8042.c: Detected active multiplexing controller, rev 1.1.
serio: i8042 AUX0 port at 0x60,0x64 irq 12
serio: i8042 AUX1 port at 0x60,0x64 irq 12
serio: i8042 AUX2 port at 0x60,0x64 irq 12
serio: i8042 AUX3 port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
Serial: 8250/16550 driver $Revision: 1.90 $ 48 ports, IRQ sharing 
enabled
ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10
PCI: setting IRQ 10 as level-triggered
ACPI: PCI interrupt 0000:00:1f.6[B] -> GSI 10 (level, low) -> IRQ 10
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
RAMDISK driver initialized: 16 RAM disks of 64000K size 1024 blocksize
loop: loaded (max 8 devices)
b44.c:v0.95 (Aug 3, 2004)
ACPI: PCI Interrupt Link [LNKE] enabled at IRQ 10
ACPI: PCI interrupt 0000:02:05.0[A] -> GSI 10 (level, low) -> IRQ 10
eth0: Broadcom 4400 10/100BaseT Ethernet 00:0a:e4:27:0f:c2
netconsole: device eth0 not up yet, forcing it
netconsole: carrier detect appears flaky, waiting 10 seconds
b44: eth0: Link is down.
b44: eth0: Link is up at 10 Mbps, half duplex.
b44: eth0: Flow control is off for TX and off for RX.
IRQ#10 thread RT prio: 49.


/ Magnus Määttä

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Michal Schmidt]

Ingo Molnar wrote:
> i've just uploaded V0.6.2 that fixes a console-unblanking-timer thinko. 
> This bug was present for quite some time, but this is the first time it 
> triggered on my testbox - might be more common on others.
> 
> 	Ingo

Hi Ingo,
I finally got to test your patch again. Now I've been running -V0.6.2 
for almost two hours and have not yet encountered a single deadlock.
So it seems that the netfilter deadlock, that I could easily reproduce 
in -V0.4.1, is solved.

There is one strange thing, though:
michich@k4-912b:~$ uptime
  00:09:17 up  1:49,  7 users,  load average: 707.72, 706.40, 682.16

In fact my computer is mostly idle.
ps shows no zombies nor any D-state processes. The system runs fine.

Michal

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[K.R. Foley]

Ingo Molnar wrote:
> * Florian Schmidt <mista.tapas@gmx.net> wrote:
> 
> 
>>On Sat, 30 Oct 2004 17:26:06 -0400
>>Lee Revell <rlrevell@joe-job.com> wrote:
>>
>>
>>>OK this is pretty sweet.  With T3 the jitter never exceeds 7% on an idle
>>>system.  As soon as I start moving the mouse this goes to 7 or 8%.  I
>>>cannot get it to go higher than 10%.  Moving windows around has no
>>>effect, the highest jitter happens when I type or move the mouse really
>>>fast IOW it corresponds to the interrupt rate.
>>>
>>>This is a pretty good baseline for what an xrun-free system would look
>>>like.  Now to test the latest version...
>>
>>Well, 
>>
>>on V0.5.16 i see something like the below output (which is much worse). It
>>seems that missed irq's with rtc show up at the same time as the xruns in
>>jackd do [i ran both jackd and wakeup in parallel].
> 
> 
> ok, could you try the -RT-V0.6.0 patch i've just uploaded? It could i
> believe improve these latencies.
> 
> 	Ingo
> -

I am not able to get V0.6.2 to boot on my SMP system here. There is no 
indication as to why. It just locks. I don't see anything in the log 
that gives an indication as to why so I am only attaching the end of the 
log. If you would like all of it, just let me know. I am going to try 
building it for my SMP system now.

kr

Oct 31 09:28:05 porky vsftpd: vsftpd vsftpd succeeded
Oct 31 09:28:06 porky xinetd[2824]: xinetd Version 2.3.13 started with 
libwrap loadavg options compiled in.
Oct 31 09:28:06 porky xinetd[2824]: Started working: 2 available services
Oct 31 09:28:06 porky sendmail: sendmail startup succeeded
Oct 31 09:28:06 porky sendmail: sm-client startup succeeded
Oct 31 09:28:07 porky gpm[2897]: *** info [startup.c(95)]:
Oct 31 09:28:07 porky gpm[2897]: Started gpm successfully. Entered 
daemon mode.
Oct 31 09:28:07 porky gpm[2897]: *** info [mice.c(1766)]:
Oct 31 09:28:07 porky gpm[2897]: imps2: Auto-detected intellimouse PS/2
Oct 31 09:28:07 porky gpm: gpm startup succeeded
Oct 31 09:28:07 porky crond: crond startup succeeded
Oct 31 09:28:08 porky xfs: xfs startup succeeded
Oct 31 09:28:08 porky anacron: anacron startup succeeded
Oct 31 09:28:09 porky atd: atd startup succeeded
Oct 31 09:28:09 porky readahead: Starting background readahead:
Oct 31 09:28:09 porky rc: Starting readahead:  succeeded
Oct 31 09:28:10 porky messagebus: messagebus startup succeeded

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Rui Nuno Capela]

Ingo Molnar wrote:
>
> ok, could you try the -RT-V0.6.0 patch i've just uploaded? It could i
> believe improve these latencies.
>

Just made a try with RT-V0.6.2 and it locks pretty easily on my P4/UP
laptop while running my jackd-R + 9*fluidsynth workload. These tests are
setup to run for 5 min. Not once it got through, and I've tried several
times now.

Although it's been 100% reproducible, the system lockup varies in tine
from the start of the test. Unfortunately it says nothing through
netconsole, so there's no additional info/dump/trace to give you for
analysis.

Most of the kernel hacking/debugging options are unset (N), except
Magic-SysRq. Hitting on SysRq+T, right after when system hangs, comes out
with absolutely nothing, only this line (via netconsole):

SysRq : Show State

Guess something's got wrong, yet again.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

On Sun, 31 Oct 2004 16:59:13 +0100
Florian Schmidt <mista.tapas@gmx.net> wrote:

> p.s. new rtc_wakeup version uploaded, which shows the percentage converted
> to usecs (always positive, you need the sign?). btw: with V0.6.2 i sometimes
> see jitter > 100% but still no lost irq (/dev/rtc still reports only 1
> delivered irq on next wakeup). I cannot provke lost irqs with -f up to 2048.
> With -f 8192 i do get lost irq's [not amazing though].

another update upped. small bugfix (still depended on jack, should work now
w/o jack installed) plus minor cosmetic corrections. Here's example output
for 2.6.8.1-P9 (under heavy load (multiple finds plus kernel compile plus UI
artistics):

~/source/my_projects/rtc_wakeup$ ./rtc_wakeup 
rtc_wakeup - press ctrl-c to stop
freq:             1024
max # of irqs:    0 (run until stopped)
jitter threshold: 5% (48 usec)
output filename:  /dev/null
rt priority:      90(91)
getting cpu speed
1194913254.155 Hz (1194.913 MHz)
# of cycles for "perfect" period: 1166907 (976 usec)
setting up ringbuffer
setting up consumer thread
setting up /dev/rtc
locking memory
turning irq on, beginning measurement
new max. jitter: 1.6% (15 usec)
new max. jitter: 1.9% (18 usec)
new max. jitter: 2.6% (25 usec)
new max. jitter: 2.9% (28 usec)
new max. jitter: 3.2% (31 usec)
new max. jitter: 3.3% (31 usec)
new max. jitter: 3.3% (32 usec)
new max. jitter: 3.5% (34 usec)
new max. jitter: 3.5% (34 usec)
new max. jitter: 3.7% (35 usec)
new max. jitter: 4.3% (41 usec)
new max. jitter: 4.9% (47 usec)
threshold violated: 5.6% (54usec)
new max. jitter: 5.6% (54 usec)
threshold violated: 5.3% (51usec)
done.
total # of irqs:      168072
missed irqs:          0
threshold violations: 2
max jitter:           5.6% (54 usec)

So basically like lee said, 7% seems to be a normal upper limit for VP
boxes (there might be spikes though). 

V0.6.2 looked good, too, until it locked up again (under heavy load with
rtc_wakeup running) ;) Will build one with debugging enabled tomorrow. 

flo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

On Sun, 31 Oct 2004 16:20:59 +0100
Florian Schmidt <mista.tapas@gmx.net> wrote:

> 
> V0.6.2 works pretty good. max jitter until now 21% [205us]. still fiddling
> with the output formatting for rtc_wakeup.
> 

i got a deadlock though. it was a weird one. mouse and keyboard [including
sysrq] froze. but the find / i started kept on running in an xterm. had to
press reset due to lack of second machine..

flo

p.s. new rtc_wakeup version uploaded, which shows the percentage converted
to usecs (always positive, you need the sign?). btw: with V0.6.2 i sometimes
see jitter > 100% but still no lost irq (/dev/rtc still reports only 1
delivered irq on next wakeup). I cannot provke lost irqs with -f up to 2048.
With -f 8192 i do get lost irq's [not amazing though].

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

On Sun, 31 Oct 2004 14:40:16 +0100
Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > > FWIW with V0.5.16 I had several hard lockups when running Florian's
> > > test app at 2048 Hz.
> > 
> > please check out V0.6.1, i made the semaphore code more robust and
> > more compatible. The V0.6 series could also fix the XFS and reiserfs
> > problems reported.
> 
> i've just uploaded V0.6.2 that fixes a console-unblanking-timer thinko. 
> This bug was present for quite some time, but this is the first time it 
> triggered on my testbox - might be more common on others.

V0.6.2 works pretty good. max jitter until now 21% [205us]. still fiddling
with the output formatting for rtc_wakeup.

flo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

On Sun, 31 Oct 2004 14:22:37 +0100
Ingo Molnar <mingo@elte.hu> wrote:


> cool! May i have a feature request :-) It would be quite useful to have
> an option to see the jitter output in percentage and in microseconds as
> well. I.e. an option to have such a format:
> 
>  late wakeup: +151.3 usecs (14.6%) jitter.

yep, will do so ater trying the 0.6.1 kernel.

> cycles are too large to be human readable, and absolute values are
> harder to read when HZ is not 1024. So to get a 'quick feel' of the
> delays in a system the above line would be the most informative (for me
> that is). Also, the percentage can go back to %.1f i think, instead of
> %.5f or so - it's good to have a decimal point in the percentage but one
> number after it is more than enough.) The more compact the output, the
> fewer useless info, the quicker the human brain can read it. (Obviously
> to generate a nice graph out of it needs a different format.)

well, to generate graphs out of the data one can choose to write the raw
cycle counts into a "history file". Then one can process this raw data to
hearts content. i will update the output routines to truncate the percentage
to %.1f [or the c++ equivalent :)]

> 
> > Here's a typical run (still on V0.5.16, will try V0.6 now):
> 
> please try -V0.6.1, -V0.6 had a pretty stupid bug that could trigger
> frequently.
> 
> > threshold violations: 20
> > max jitter:           58.1228%
> 
> here it would also be useful to have the 'max jitter' in usecs. I.e. in
> the above case it was somewhere around ~570 usecs.

ack.

flo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

On Sun, 31 Oct 2004 13:48:28 +0100
Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > ok, could you try the -RT-V0.6.0 patch i've just uploaded? It could i
> > believe improve these latencies.
> 
> hm, CONFIG_PARPORT_1284 seems broken, and USB too, it locks up during
> bootup. Investigating ...

i will try anyways since i don't use neither USB nor my parallel port :)

also, i have uploaded an overhauled version of the wakeup timer program. It
now deferres all output to another SCHED_FIFO thread [with prio 1 lower than
the main process]. The data is passed via a lockless fifo [ripped from jack
sourcecode]. Also it handles commandline options better and has sensible [?]
defaults:

~/source/my_projects/wakeup$ ./rtc_wakeup -h
usage: wakeup [options]
options:
-f freqency(hz) (default 1024) 
-p realtime prio (default 90(91))
-n max number of interrupts (default 0: run until stopped)
-t jitter threshold (%) (default 5) 
-o history_file (default /dev/null)
-h show help

The "history" file contains three rows: irq #, cycle count at wakeup,
jitter(%). The first few entries are probably off [due to startup stuff, all
reporting during runtime takes only irq's after the third into account, to
avoid the startup mess]..

grab it at http://affenbande.org/~tapas/rtc_wakeup.tgz

Here's a typical run (still on V0.5.16, will try V0.6 now):

~/source/my_projects/rtc_wakeup$ ./rtc_wakeup 
rtc_wakeup - press ctrl-c to stop
freq:             1024
max # of irqs:    0 (run until stopped)
jitter threshold: 5%
output filename:  /dev/null
rt priority:      90(91)
getting cpu speed
1194909286.409 Hz (1194.909 MHz)
# of cycles for "perfect" period: 1166903
setting up ringbuffer
setting up consumer thread
setting up /dev/rtc.
locking memory...
turning irq on, beginning measurement (might take a while).
new max. jitter: 0.268231%
threshold violated: 2523891286957 (26.3406%)
new max. jitter: 26.3406%
threshold violated: 2523892156958 (25.4436%)
threshold violated: 2523896074240 (36.5712%)
new max. jitter: 36.5712%
threshold violated: 2523896831195 (35.1313%)
threshold violated: 2523899757527 (50.2766%)
new max. jitter: 50.2766%
threshold violated: 2523900328433 (51.0751%)
new max. jitter: 51.0751%
threshold violated: 2523903338753 (57.0041%)
new max. jitter: 57.0041%
threshold violated: 2523903827419 (58.1228%)
new max. jitter: 58.1228%
threshold violated: 2523905321471 (28.0357%)
threshold violated: 2523906161154 (28.0417%)
threshold violated: 2523909924647 (22.6959%)
threshold violated: 2523911166386 (6.41322%)
threshold violated: 2523911990002 (29.4186%)
threshold violated: 2523913267054 (9.43943%)
threshold violated: 2523914321799 (9.6116%)
threshold violated: 2523915653813 (14.1495%)
threshold violated: 2523916760657 (5.14687%)
threshold violated: 2523917830826 (8.28981%)
threshold violated: 2523920520433 (29.74%)
threshold violated: 2523921329137 (30.6966%)
done.
total # of irqs:      2754
missed irqs:          0
threshold violations: 20
max jitter:           58.1228%


flo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> > FWIW with V0.5.16 I had several hard lockups when running Florian's
> > test app at 2048 Hz.
> 
> please check out V0.6.1, i made the semaphore code more robust and
> more compatible. The V0.6 series could also fix the XFS and reiserfs
> problems reported.

i've just uploaded V0.6.2 that fixes a console-unblanking-timer thinko. 
This bug was present for quite some time, but this is the first time it 
triggered on my testbox - might be more common on others.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

> also, i have uploaded an overhauled version of the wakeup timer
> program. It now deferres all output to another SCHED_FIFO thread [with
> prio 1 lower than the main process]. The data is passed via a lockless
> fifo [ripped from jack sourcecode]. [...]

it works really great - previously i had problems with 'secondary'
latencies when running the app with HZ=8192, now i get only the primary
message(s).

i also like it that if you run the app it does a very useful type of
measurement right out of box. The simpler you make a validation app for
kernel developers the more they will use it ;-)

[ btw., FC3's gcc produces this (harmless) warning during the build:

 cc1plus: warning: command line option "-Wstrict-prototypes" is valid 
 for Ada/C/ObjC but not for C++ ]

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

> also, i have uploaded an overhauled version of the wakeup timer
> program. It now deferres all output to another SCHED_FIFO thread [with
> prio 1 lower than the main process]. The data is passed via a lockless
> fifo [ripped from jack sourcecode]. Also it handles commandline
> options better and has sensible [?] defaults:
> 
> ~/source/my_projects/wakeup$ ./rtc_wakeup -h
> usage: wakeup [options]
> options:
> -f freqency(hz) (default 1024) 
> -p realtime prio (default 90(91))
> -n max number of interrupts (default 0: run until stopped)
> -t jitter threshold (%) (default 5) 
> -o history_file (default /dev/null)
> -h show help

cool! May i have a feature request :-) It would be quite useful to have
an option to see the jitter output in percentage and in microseconds as
well. I.e. an option to have such a format:

 late wakeup: +151.3 usecs (14.6%) jitter.

cycles are too large to be human readable, and absolute values are
harder to read when HZ is not 1024. So to get a 'quick feel' of the
delays in a system the above line would be the most informative (for me
that is). Also, the percentage can go back to %.1f i think, instead of
%.5f or so - it's good to have a decimal point in the percentage but one
number after it is more than enough.) The more compact the output, the
fewer useless info, the quicker the human brain can read it. (Obviously
to generate a nice graph out of it needs a different format.)

> Here's a typical run (still on V0.5.16, will try V0.6 now):

please try -V0.6.1, -V0.6 had a pretty stupid bug that could trigger
frequently.

> threshold violations: 20
> max jitter:           58.1228%

here it would also be useful to have the 'max jitter' in usecs. I.e. in
the above case it was somewhere around ~570 usecs.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> > > ok, could you try the -RT-V0.6.0 patch i've just uploaded? It could i
> > > believe improve these latencies.
> > 
> > hm, CONFIG_PARPORT_1284 seems broken, and USB too, it locks up during
> > bootup. Investigating ...
> > 
> 
> FWIW with V0.5.16 I had several hard lockups when running Florian's
> test app at 2048 Hz.

please check out V0.6.1, i made the semaphore code more robust and more
compatible. The V0.6 series could also fix the XFS and reiserfs problems
reported.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> > ok, could you try the -RT-V0.6.0 patch i've just uploaded? It could i
> > believe improve these latencies.
> 
> hm, CONFIG_PARPORT_1284 seems broken, and USB too, it locks up during
> bootup. Investigating ...

ok, fixed it and uploaded -V0.6.1.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Sun, 2004-10-31 at 13:48 +0100, Ingo Molnar wrote:
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > ok, could you try the -RT-V0.6.0 patch i've just uploaded? It could i
> > believe improve these latencies.
> 
> hm, CONFIG_PARPORT_1284 seems broken, and USB too, it locks up during
> bootup. Investigating ...
> 

FWIW with V0.5.16 I had several hard lockups when running Florian's test
app at 2048 Hz.

Lee

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> ok, could you try the -RT-V0.6.0 patch i've just uploaded? It could i
> believe improve these latencies.

hm, CONFIG_PARPORT_1284 seems broken, and USB too, it locks up during
bootup. Investigating ...

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> Hmm, maybe the timer interrupt should be 99 and the rest say 50.
> Wouldn't it be bad if you had a fully loaded jackd (DSP load in JACK
> is the proportion of the process cycle to the period time; in a fully
> loaded jack setup the clients are using all the available time) and
> jackd + the soundcard IRQ's RT priorities are higher than the timer
> interrupt?  Seems like you could starve the timer interrupt
> indefinitely.

i misspoke when i said timer interrupt - that one is the lone
non-threaded IRQ so its priority is 100 in essence.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Sun, 2004-10-31 at 13:19 +0100, Ingo Molnar wrote:
> we could do this too. The reason why i picked the current "start at
> SCHED_FIFO prio 49 and decrease it by 1 until it reaches 25, then stay
> constant" logic is that typically the irqs registered first are 'more
> important' - e.g. the timer interrupt.

Hmm, maybe the timer interrupt should be 99 and the rest say 50.
Wouldn't it be bad if you had a fully loaded jackd (DSP load in JACK is
the proportion of the process cycle to the period time; in a fully
loaded jack setup the clients are using all the available time) and
jackd + the soundcard IRQ's RT priorities are higher than the timer
interrupt?  Seems like you could starve the timer interrupt
indefinitely.

In fact, the only IRQ thread that currently _needs_ to be lower prio
than the others is IDE - the others all execute quickly enough to only
cause a problem at _extreme_ latencies that you would never use in the
real world, at least for audio/JACK.  Last time I checked no other
hardirq ran for more than about 50 usecs.  With Jens' patch to move IDE
bh processing into a softirq, I suspect the relative priorities of the
IRQ threads would not matter at all.

Lee

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> Actually this raises an interesting point.  Maybe all IRQ threads
> should get the same RT priority by default, so we get FIFO scheduling
> among IRQ threads.  It seems like this would make it harder for IRQ
> threads to starve each other.  Then we only have to elevate the
> priority of the IRQ thread(s) we are interested in.

we could do this too. The reason why i picked the current "start at
SCHED_FIFO prio 49 and decrease it by 1 until it reaches 25, then stay
constant" logic is that typically the irqs registered first are 'more
important' - e.g. the timer interrupt.

> Another idea is to allow SCHED_FIFO processes of equal priority to
> preempt one another on a LIFO basis.  Wouldn't this be very close to
> the traditional Linux interrupt model, where interrupts can interrupt
> each other and we handle the most recent interrupt first?

well, it's called SCHED_*FIFO* for a reason :-) What might make sense is
a SCHED_LIFO policy. But, i'm not so sure it's the right thing to do:
the best work-queueing model is almost always FIFO, as it gives the best
possible fairness between equals.

Fairness also often translates into better performance, because a system
that 'fluctuates' due to LIFO often underperforms a FIFO one because
when it fluctuates towards lower load it under-utilizes the resources,
and when it fluctuates up it overloads.  LIFO makes sense for anonymous
resources like pages/slabs where LIFO allocation leads to better cache
utilization. But i'd say for non-anonymous workloads it's almost always
a loss.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Sun, 2004-10-31 at 11:00 +0100, Florian Schmidt wrote:
> thanks for the patch (it has a little problem, since it uses prio 99 which
> is always equal or greater than the rtc thread prio. i changed it in my
> local version to accept a parameter).

OK good idea.  As in the JACK case, the relative priorities of the RTC
irq thread and the test program should not matter as these two should
never contend - something is seriously wrong if both are ever runnable
at the same time.

Actually this raises an interesting point.  Maybe all IRQ threads should
get the same RT priority by default, so we get FIFO scheduling among IRQ
threads.  It seems like this would make it harder for IRQ threads to
starve each other.  Then we only have to elevate the priority of the IRQ
thread(s) we are interested in.

Another idea is to allow SCHED_FIFO processes of equal priority to
preempt one another on a LIFO basis.  Wouldn't this be very close to the
traditional Linux interrupt model, where interrupts can interrupt each
other and we handle the most recent interrupt first?

Lee

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

> On Sat, 30 Oct 2004 17:26:06 -0400
> Lee Revell <rlrevell@joe-job.com> wrote:
> 
> > OK this is pretty sweet.  With T3 the jitter never exceeds 7% on an idle
> > system.  As soon as I start moving the mouse this goes to 7 or 8%.  I
> > cannot get it to go higher than 10%.  Moving windows around has no
> > effect, the highest jitter happens when I type or move the mouse really
> > fast IOW it corresponds to the interrupt rate.
> > 
> > This is a pretty good baseline for what an xrun-free system would look
> > like.  Now to test the latest version...
> 
> Well, 
> 
> on V0.5.16 i see something like the below output (which is much worse). It
> seems that missed irq's with rtc show up at the same time as the xruns in
> jackd do [i ran both jackd and wakeup in parallel].

ok, could you try the -RT-V0.6.0 patch i've just uploaded? It could i
believe improve these latencies.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

On Sat, 30 Oct 2004 22:20:24 -0400
Lee Revell <rlrevell@joe-job.com> wrote:

> Here is a patch that causes the number of consecutive missed interrupts
> to be printed, as well as setting RT priority itself instead of relying
> on the rt_wakeup wrapper.
> 
> I have found that moving the mouse quickly will reliably cause many
> interrupts to be missed.

thanks for the patch (it has a little problem, since it uses prio 99 which
is always equal or greater than the rtc thread prio. i changed it in my
local version to accept a parameter). will do some more cleanup of the
program [saner option handling, msec/usec output of the relevant cycle
counts, history file output, ctrl-c handling etc].

U\L in an hour or two.

flo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Sat, 2004-10-30 at 23:13 +0200, Florian Schmidt wrote: 
> On Sat, 30 Oct 2004 16:26:35 -0400
> Lee Revell <rlrevell@joe-job.com> wrote:
> 
> > Take a look at the patch I posted to jackit-devel the other day to
> > calculate the CPU speed (previously we grabbed it from /proc/cpuinfo).
> > I just copied the code from realfeel2.
> > 
> 
> ah, ok.. tarball updated. The third argument is now a percentage. If the
> cycle count difference between two different wakeups differs more than the
> specified percentage from the "perfect" period, then a line is printed to
> the terminal showing by how much percent it differs. 
> 
> flo
> 

Here is a patch that causes the number of consecutive missed interrupts
to be printed, as well as setting RT priority itself instead of relying
on the rt_wakeup wrapper.

I have found that moving the mouse quickly will reliably cause many
interrupts to be missed.

Lee

--- wakeup/wakeup.cc	2004-10-30 17:11:08.000000000 -0400
+++ wakeup2/wakeup.cc	2004-10-30 21:03:21.000000000 -0400
@@ -90,6 +90,22 @@
 }
 // realfeel
 
+int set_realtime_priority(void)
+{
+	struct sched_param schp;
+	/*
+	 * set the process to realtime privs
+	 */
+	memset(&schp, 0, sizeof(schp));
+	schp.sched_priority = sched_get_priority_max(SCHED_FIFO);
+	
+	if (sched_setscheduler(0, SCHED_FIFO, &schp) != 0) {
+		perror("sched_setscheduler");
+		exit(1);
+	}
+
+	return 0;
+}
 
 
 
@@ -181,6 +197,8 @@
     perror("ioctl");
     exit(errno);
   }
+ 
+  set_realtime_priority();
   
   unsigned int irqs = 0;
   unsigned int missed_irqs = 0;
@@ -220,7 +238,7 @@
     irqs += data;
     // std::cout << data << std::endl;
     if (data > 1) {
-      std::cout << "ouch! we missed one ore more irq[s]" << std::endl;
+      std::cout << "ouch! we missed " << data << " or more irq[s]" << std::endl;
       missed_irqs++;
     }
   }

Lee

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Sat, 2004-10-30 at 23:38 +0200, Florian Schmidt wrote:
> While there's no way to deterministically force missed irq's by window
> wiggling [we should make it olympic discipline :)], UI action seems to raise
> the probability somewhat.
> 

OK, I am seeing the exact same results.  They are so close to yours that
I am not going to bother posting them.  UI activity does seem to trigger
missed IRQs - if I switch to my mail window or Mozilla and back to the
test I see that 4 or 5 IRQs in a row got skipped.  So maybe context
switching triggers the bug.

Lee

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

On Sat, 30 Oct 2004 17:26:06 -0400
Lee Revell <rlrevell@joe-job.com> wrote:

> On Sat, 2004-10-30 at 23:13 +0200, Florian Schmidt wrote:
> > ah, ok.. tarball updated. The third argument is now a percentage. If the
> > cycle count difference between two different wakeups differs more than the
> > specified percentage from the "perfect" period, then a line is printed to
> > the terminal showing by how much percent it differs. 
> 
> OK this is pretty sweet.  With T3 the jitter never exceeds 7% on an idle
> system.  As soon as I start moving the mouse this goes to 7 or 8%.  I
> cannot get it to go higher than 10%.  Moving windows around has no
> effect, the highest jitter happens when I type or move the mouse really
> fast IOW it corresponds to the interrupt rate.
> 
> This is a pretty good baseline for what an xrun-free system would look
> like.  Now to test the latest version...

Well, 

on V0.5.16 i see something like the below output (which is much worse). It
seems that missed irq's with rtc show up at the same time as the xruns in
jackd do [i ran both jackd and wakeup in parallel].

While there's no way to deterministically force missed irq's by window
wiggling [we should make it olympic discipline :)], UI action seems to raise
the probability somewhat.

~/source/my_projects/wakeup$ ./rt_wakeup 1024 50000 10
freq: 1024 #: 50000
getting cpu speed
1194.908 MHz
# of cycles for "perfect" period: 1166902
setting up /dev/rtc.
locking memory...
turning irq on, beginning measurement (might take a while).
threshold violated: 1485237 cycles since last wakeup (27.2804%).
threshold violated: 1047469 cycles since last wakeup (10.235%).
threshold violated: 964069 cycles since last wakeup (17.3822%).
ouch! we missed one ore more irq[s]
threshold violated: 4037774 cycles since last wakeup (246.025%).
threshold violated: 620764 cycles since last wakeup (46.8024%).
threshold violated: 10327889 cycles since last wakeup (785.069%).
ouch! we missed one ore more irq[s]
threshold violated: 700832 cycles since last wakeup (39.9408%).
threshold violated: 3148900 cycles since last wakeup (169.851%).
ouch! we missed one ore more irq[s]
threshold violated: 231791 cycles since last wakeup (80.1362%).
threshold violated: 768584 cycles since last wakeup (34.1347%).
ouch! we missed one ore more irq[s]
threshold violated: 3500123 cycles since last wakeup (199.95%).
threshold violated: 2581587 cycles since last wakeup (121.234%).
threshold violated: 920020 cycles since last wakeup (21.157%).
threshold violated: 2430290 cycles since last wakeup (108.269%).
threshold violated: 240850 cycles since last wakeup (79.3599%).
threshold violated: 3614768 cycles since last wakeup (209.775%).
ouch! we missed one ore more irq[s]
threshold violated: 883969 cycles since last wakeup (24.2465%).
threshold violated: 479302 cycles since last wakeup (58.9253%).
threshold violated: 830208 cycles since last wakeup (28.8537%).
threshold violated: 2313154 cycles since last wakeup (98.2304%).
threshold violated: 516698 cycles since last wakeup (55.7205%).
threshold violated: 2545998 cycles since last wakeup (118.184%).
ouch! we missed one ore more irq[s]
threshold violated: 4466286 cycles since last wakeup (282.747%).
ouch! we missed one ore more irq[s]
threshold violated: 565233 cycles since last wakeup (51.5612%).
threshold violated: 184668 cycles since last wakeup (84.1745%).
threshold violated: 2854676 cycles since last wakeup (144.637%).
threshold violated: 3384620 cycles since last wakeup (190.052%).
ouch! we missed one ore more irq[s]
threshold violated: 282832 cycles since last wakeup (75.7621%).
threshold violated: 2741798 cycles since last wakeup (134.964%).
ouch! we missed one ore more irq[s]
threshold violated: 1405657 cycles since last wakeup (20.4606%).
threshold violated: 227030 cycles since last wakeup (80.5442%).
threshold violated: 3476092 cycles since last wakeup (197.891%).
ouch! we missed one ore more irq[s]
threshold violated: 448723 cycles since last wakeup (61.5458%).
threshold violated: 2390327 cycles since last wakeup (104.844%).
ouch! we missed one ore more irq[s]
threshold violated: 1489252 cycles since last wakeup (27.6244%).
threshold violated: 517884 cycles since last wakeup (55.6189%).
threshold violated: 3774860 cycles since last wakeup (223.494%).
ouch! we missed one ore more irq[s]
threshold violated: 1762452 cycles since last wakeup (51.0368%).
threshold violated: 1545052 cycles since last wakeup (32.4063%).
ouch! we missed one ore more irq[s]
threshold violated: 442749 cycles since last wakeup (62.0577%).
threshold violated: 2230841 cycles since last wakeup (91.1764%).
threshold violated: 5825655 cycles since last wakeup (399.241%).
ouch! we missed one ore more irq[s]
threshold violated: 2874721 cycles since last wakeup (146.355%).
ouch! we missed one ore more irq[s]
threshold violated: 647431 cycles since last wakeup (44.5171%).
threshold violated: 126309 cycles since last wakeup (89.1757%).
threshold violated: 5817415 cycles since last wakeup (398.535%).
ouch! we missed one ore more irq[s]
threshold violated: 661937 cycles since last wakeup (43.274%).
threshold violated: 4351747 cycles since last wakeup (272.932%).
ouch! we missed one ore more irq[s]
threshold violated: 442189 cycles since last wakeup (62.1057%).
threshold violated: 115427 cycles since last wakeup (90.1083%).
done.

total # of irqs: 50032. missed irq deadlines: 17

mean cycle difference betweem two wakeups: 1.16779e+06 cycles

min. cycle difference betweem two wakeups: 115427 cycles (#: 44584) 
 diff from mean diff: 1.05237e+06

max. cycle difference betweem two wakeups: 1.03279e+07 cycles (#: 8869) 
 diff from mean diff: 9.1601e+06

mean difference from mean difference: 6458.8 cycles

~/source/my_projects/wakeup$ 

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Sat, 2004-10-30 at 23:13 +0200, Florian Schmidt wrote:
> ah, ok.. tarball updated. The third argument is now a percentage. If the
> cycle count difference between two different wakeups differs more than the
> specified percentage from the "perfect" period, then a line is printed to
> the terminal showing by how much percent it differs. 

OK this is pretty sweet.  With T3 the jitter never exceeds 7% on an idle
system.  As soon as I start moving the mouse this goes to 7 or 8%.  I
cannot get it to go higher than 10%.  Moving windows around has no
effect, the highest jitter happens when I type or move the mouse really
fast IOW it corresponds to the interrupt rate.

This is a pretty good baseline for what an xrun-free system would look
like.  Now to test the latest version...

Lee

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

On Sat, 30 Oct 2004 16:26:35 -0400
Lee Revell <rlrevell@joe-job.com> wrote:

> Take a look at the patch I posted to jackit-devel the other day to
> calculate the CPU speed (previously we grabbed it from /proc/cpuinfo).
> I just copied the code from realfeel2.
> 

ah, ok.. tarball updated. The third argument is now a percentage. If the
cycle count difference between two different wakeups differs more than the
specified percentage from the "perfect" period, then a line is printed to
the terminal showing by how much percent it differs. 

flo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

On Sat, 30 Oct 2004 16:05:14 -0400
Lee Revell <rlrevell@joe-job.com> wrote:

> So maybe the bug is not related to scheduling of SCHED_FIFO threads, but
> that we are missing IRQs.  I think this would explain the choppy
> playback with mplayer (it uses the RTC and does not run SCHED_FIFO).

I wonder about what X11 is doing. Is it maybe doing some locking which is
"out of line" for RP or something? I mean the RP patches touch everything
that accesses hw directly with the exception of X11, right? [/me never
grokked how X fits into linux' driver model anyways]

flo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Sat, 2004-10-30 at 22:15 +0200, Florian Schmidt wrote:
> What's the best way to find out the cycles/s of the cpu? This way the
> input/output could become a little nicer [because then i can calculate
> programatically how long a "perfect" period should be in cycles].

Take a look at the patch I posted to jackit-devel the other day to
calculate the CPU speed (previously we grabbed it from /proc/cpuinfo).
I just copied the code from realfeel2.

Lee

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

On Sat, 30 Oct 2004 15:52:04 -0400
Lee Revell <rlrevell@joe-job.com> wrote:

> Yup there is definitely something not right:

[snip]

> You should modify the program to print something when it sees a big
> miss.  This would make it easier to figure out what kind of system
> activity triggers the problem.

right, i just wanted to avoid doing that from the process that polls itself,
because a std::cout << "ugh!" << std::endl; might already be enough to skew
the following irq's, right?

anyways, this new version [just upped] prints a line when a missed irq was
detected. Also this version understands a third parameter which acts as an
upper threshold. A line is printed when the difference of the cycle count of
two consecutive wakeups is greater than the threshold.

run it once w/o threshold on an idle system to see what a useful thresh
would be.

i use it like this for example:

./rt_wakeup 1024 50000 1200000

What's the best way to find out the cycles/s of the cpu? This way the
input/output could become a little nicer [because then i can calculate
programatically how long a "perfect" period should be in cycles].

flo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Sat, 2004-10-30 at 21:47 +0200, Florian Schmidt wrote:
> Hi, in the meantime i also booted into P9 again and the results differ
> dramatically. Much better in P9. Anyways, i reuploaded the tarball. The
> program tries to detect missed irq's now and counts the total number of
> irq's delivered by /dev/rtc. Since the program does not recover from missed
> irq's the "statistical" data for these runs is useless [except for the
> knowledge of the fact that one or more irq was missed :)]

Here are my results with T3.  As expected, much better.

--

rlrevell@mindpipe:~/cvs/wakeup$ ./rt_wakeup 1024 100000 freq: 1024 #:
100000
setting up /dev/rtc.
locking memory...
turning irq on, beginning measurement (might take a while).
done.

total # of irqs: 100000. missed irq's: 0

mean cycle difference betweem two wakeups: 585875 cycles

min. cycle difference betweem two wakeups: 517708 cycles (#: 27478)
 diff from mean diff: 68166.8

max. cycle difference betweem two wakeups: 656018 cycles (#: 27477)
 diff from mean diff: 70143.2

mean difference from mean difference: 3052.86 cycles

--

So maybe the bug is not related to scheduling of SCHED_FIFO threads, but
that we are missing IRQs.  I think this would explain the choppy
playback with mplayer (it uses the RTC and does not run SCHED_FIFO).

Lee

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

On Sat, 30 Oct 2004 21:37:06 +0200
Ingo Molnar <mingo@elte.hu> wrote:

> > Hi, in the meantime i also booted into P9 again and the results differ
> > dramatically. Much better in P9. Anyways, i reuploaded the tarball.
> > The program tries to detect missed irq's now and counts the total
> > number of irq's delivered by /dev/rtc. Since the program does not
> > recover from missed irq's the "statistical" data for these runs is
> > useless [except for the knowledge of the fact that one or more irq was
> > missed :)]
> 
> just to make sure - you are running this on an UP system, correct?

right. all debugging off [when possible], RP on.

mango:~# cat /proc/cpuinfo 
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 6
model           : 4
model name      : AMD Athlon(tm) Processor
stepping        : 2
cpu MHz         : 1195.144
cache size      : 256 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat pse36 mmx fxsr syscall mmxext 3dnowext 3dnow
bogomips        : 2375.68


flo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Sat, 2004-10-30 at 21:47 +0200, Florian Schmidt wrote:
> Hi, in the meantime i also booted into P9 again and the results differ
> dramatically. Much better in P9. Anyways, i reuploaded the tarball. The
> program tries to detect missed irq's now and counts the total number of
> irq's delivered by /dev/rtc. Since the program does not recover from missed
> irq's the "statistical" data for these runs is useless [except for the
> knowledge of the fact that one or more irq was missed :)]

Yup there is definitely something not right:

rlrevell@mindpipe:~/cvs/wakeup$ ./rt_wakeup 1024 100000
freq: 1024 #: 100000
setting up /dev/rtc.
locking memory...
turning irq on, beginning measurement (might take a while).
done.

total # of irqs: 100065. missed irq's: 4

mean cycle difference betweem two wakeups: 586332 cycles

min. cycle difference betweem two wakeups: 25184 cycles (#: 28766)
 diff from mean diff: 561148

max. cycle difference betweem two wakeups: 2.78432e+07 cycles (#: 98195)
 diff from mean diff: 2.72569e+07

mean difference from mean difference: 3759.45 cycles

You should modify the program to print something when it sees a big
miss.  This would make it easier to figure out what kind of system
activity triggers the problem.

Lee

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

On Sat, 30 Oct 2004 21:17:25 +0200
Ingo Molnar <mingo@elte.hu> wrote:

> > No, this cannot be the whole story, because unless verbose mode is
> > specified, jackd will only prints anything if there is an xrun.  So
> > there is something else going on.
> > 
> > This _really_ feels like a kernel bug.  Are you saying that this could
> > still be a jackd problem, even though T3 works perfectly with the
> > exact same jackd binary?
> 
> i think you are right - there are too many independent indicators
> pointing towards some sort of kernel problem. I'll try Florian's testapp
> and see whether i can see the 'window wiggle' problem.

Hi, in the meantime i also booted into P9 again and the results differ
dramatically. Much better in P9. Anyways, i reuploaded the tarball. The
program tries to detect missed irq's now and counts the total number of
irq's delivered by /dev/rtc. Since the program does not recover from missed
irq's the "statistical" data for these runs is useless [except for the
knowledge of the fact that one or more irq was missed :)]

flo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Sat, 2004-10-30 at 21:17 +0200, Ingo Molnar wrote:
> i think you are right - there are too many independent indicators
> pointing towards some sort of kernel problem. I'll try Florian's testapp
> and see whether i can see the 'window wiggle' problem.


This is kind of weird:

Checking 'hlt' instruction... OK.
spawn_desched_task(00000000)
desched cpu_callback 3/00000000
ksoftirqd started up.
softirq RT prio: 24.
desched cpu_callback 2/00000000
desched thread 0 started up.

But:

rlrevell@mindpipe:~/cvs/wakeup$ chrt -p 2
pid 2's current scheduling policy: SCHED_OTHER
pid 2's current scheduling priority: 0

I do not think I changed ksoftirqd's priority.

Lee

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

> Hi, in the meantime i also booted into P9 again and the results differ
> dramatically. Much better in P9. Anyways, i reuploaded the tarball.
> The program tries to detect missed irq's now and counts the total
> number of irq's delivered by /dev/rtc. Since the program does not
> recover from missed irq's the "statistical" data for these runs is
> useless [except for the knowledge of the fact that one or more irq was
> missed :)]

just to make sure - you are running this on an UP system, correct?

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> No, this cannot be the whole story, because unless verbose mode is
> specified, jackd will only prints anything if there is an xrun.  So
> there is something else going on.
> 
> This _really_ feels like a kernel bug.  Are you saying that this could
> still be a jackd problem, even though T3 works perfectly with the
> exact same jackd binary?

i think you are right - there are too many independent indicators
pointing towards some sort of kernel problem. I'll try Florian's testapp
and see whether i can see the 'window wiggle' problem.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

[-- Attachment #1: Type: text/plain, Size: 2811 bytes --]

On Sat, 30 Oct 2004 15:15:07 +0200
Ingo Molnar <mingo@elte.hu> wrote:

> should have mentioned that in the user-triggered modus you have to do
> the latency measurement in userspace. It is only the trace that is
> correct, for the time being. This trace shows what i'd expect a jackd
> wakeup to look like normally: 13 usecs.
> 
> 	Ingo
> 

Hi,

i thought, why not try to get jack out of the equation first and use a
different irq source. So i wrote a small test app for /dev/rtc to see if i
see the same erratic behaviour [.cc source file atached. find a small
tarball here: http://affenbande.org/~tapas/wakeup.tgz (it has a simple
makefile, plus a script which does the chrt thing to make it SCHED_FIFO)].

This program takes two parameters: the desired freq of the rtc and the
number of irq's to be measured.

Then it polls on the fd of /dev/rtc and takes a cpu cycle count timestamp as
soon as poll returns. At the end of the program i tried to gather some
[useful ???] statistics of the data: 

- the mean difference in cycles between two wakeups

- the max difference in cycles between two wakeups (and how much this
differs from the  mean difference)

- the min difference in cycles between two wakeups (and how much this
differs from the  mean difference)

- the mean difference from the mean difference :)

And alas, wiggling windows screws up the timing on V0.5.6 for this, too.

dunno, if this is any useful to you, but i felt the urge to try it :)

	flo


P.S.: don't forget to make your rtc irq SCHED_FIFO with a high priority, too

here's a sample output with window wiggling in X (rt_wakeup runs chrt -f 90
wakeup, so make rtc at least prio 91):

~/source/my_projects/wakeup$ ./rt_wakeup 1024 6000
freq: 1024 #: 6000
setting up /dev/rtc...
locking memory...
turning irq on, beginning measurement (might take a while)...
...measurement done

mean cycle difference betweem two wakeups: 1.17845e+06 cycles

min. cycle difference betweem two wakeups: 185992 cycles (#: 1563) 
 diff from mean diff: 992461

max. cycle difference betweem two wakeups: 9.73166e+06 cycles (#: 1546) 
 diff from mean diff: 8.5532e+06
                      ^this is bad i suppose :)

mean difference from mean difference: 25279.2 cycles


here's one on a rather idle system:

~/source/my_projects/wakeup$ ./rt_wakeup 1024 6000
freq: 1024 #: 6000
setting up /dev/rtc...
locking memory...
turning irq on, beginning measurement (might take a while)...
...measurement done

mean cycle difference betweem two wakeups: 1.16697e+06 cycles

min. cycle difference betweem two wakeups: 1.1486e+06 cycles (#: 5492) 
 diff from mean diff: 18379.2

max. cycle difference betweem two wakeups: 1.18439e+06 cycles (#: 5491) 
 diff from mean diff: 17417.8

mean difference from mean difference: 1144.82 cycles
~/source/my_projects/wakeup$ 

[-- Attachment #2: wakeup.cc --]
[-- Type: text/x-c++src, Size: 4266 bytes --]

#include <iostream>
#include <sstream>
#include <string>

#include <linux/rtc.h>
#include <sys/ioctl.h>
#include <sys/time.h>
#include <sys/types.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <errno.h>
#include <unistd.h>
#include <poll.h>

#include <cmath>

// this program is partly ripped off the rtc docs in the linux kernel source tree

// the cycle count count ripped from jackd
typedef unsigned long long cycles_t;

extern cycles_t cacheflush_time;

#define rdtscll(val) \
     __asm__ __volatile__("rdtsc" : "=A" (val))

static inline cycles_t get_cycles (void)
{
	unsigned long long ret;

	rdtscll(ret);
	return ret;
}



int main(int argc, char *argv[]) {
  if (argc < 3) {
    std::cout << "usage: wakeup [freqency(hz)] [number of interrupts]" << std::endl;
  }
  
  std::stringstream args;
  args << argv[1] << " " << argv[2];
  
  int freq;
  args >> freq;

  int number;
  args >> number;

  std::cout << "freq: " << freq << " #: " << number << std::endl;

  if (number < 3) {
    std::cout << "number of irq's must be >= 3" << std::endl;
    return(0);
  }

  // an array to store the cycles for each interrupt in.
  cycles_t *cycles = new cycles_t[number];

  std::cout << "setting up /dev/rtc..." << std::endl;

  int fd;
  fd = open("/dev/rtc", O_RDONLY);
  if (fd ==  -1) {
    perror("/dev/rtc");
    exit(errno);
  }
  
  int retval = ioctl(fd, RTC_IRQP_SET, freq);
  if (retval == -1) {
    perror("ioctl");
    exit(errno);
  }

  
  // we poll only on a single descriptor, the /dev/rtc one
  struct pollfd fds[1];
  pollfd pfd;
  pfd.fd = fd;
  pfd.events = POLLRDNORM|POLLRDBAND;
  fds[0] = pfd;


  // we set the timeout to 8 periods
  int timeout;
  timeout = 8*( (int)( (1.0f/(float)freq)*1000.0f ) );

  std::cout << "locking memory..." << std::endl;
  mlockall(MCL_CURRENT);
  // std::cout << "sleeping 1 sec" << std::endl;
  // sleep(1);

  std::cout << "turning irq on, beginning measurement (might take a while)..." << std::endl;
  
  retval = ioctl(fd, RTC_PIE_ON, 0);
  if (retval == -1) {
    perror("ioctl");
    exit(errno);
  }
  
  unsigned long data;
  for (int i = 0; i < number; ++i) {
    // first we poll, until data is available
    retval = poll(fds, 1, timeout);
    if (retval == -1) {
      perror("poll");
      exit(errno);
    }

    // then we take a timestamp;
    cycles[i] = get_cycles();
    // std::cout << "irq!" << std::endl;
    
    // then we read it
    retval = read(fd, &data, sizeof(unsigned long));
    if (retval == -1) {
      perror("read");
      exit(errno);
    }

    // see if the high bytes of the data contains a number of irq > 1. probably wrong,
    data = (data >> 16);
    if (data > 1) {std::cout << "more than 1 irq happened inbetween this and last wakeup" << std::endl;}
  }

  
  std::cout << "...measurement done" << std::endl;

  // first one is often skewed
  double mean_diff = ((double)cycles[number-1] - (double)cycles[1])/(double)(number-2);

  // std::cout << "0: \t" << cycles[0] << std::endl;
  double min_diff, max_diff;
  min_diff = max_diff = (double)(cycles[2] - cycles[1]);

  unsigned int min_diff_i, max_diff_i;
  min_diff_i = max_diff_i = 2;
  
  double mean_diff_diff = 0;

  for (int i = 2; i < number; ++i) {
    double diff = (double)(cycles[i] - cycles[i-1]);
    if (diff < min_diff) {min_diff = diff; min_diff_i = i;}
    if (diff > max_diff) {max_diff = diff; max_diff_i = i;}

    mean_diff_diff += fabs(mean_diff - diff);
    // std::cout << i << ": \t" << cycles[i] << std::endl;
  }
  mean_diff_diff /= (double)(number - 2);
  
  std::cout << std::endl << "mean cycle difference betweem two wakeups: " 
	    << mean_diff 
	    << " cycles" <<  std::endl << std::endl;

  std::cout << "min. cycle difference betweem two wakeups: " 
	    << min_diff << " cycles (#: " << min_diff_i << ") \n diff from mean diff: " 
	    << fabs(min_diff - mean_diff) << std::endl << std::endl;

  std::cout << "max. cycle difference betweem two wakeups: " 
	    << max_diff << " cycles (#: " << max_diff_i << ") \n diff from mean diff: " 
	    << fabs(max_diff - mean_diff) << std::endl << std::endl;
  
  std::cout << "mean difference from mean difference: " 
	    << mean_diff_diff << " cycles" << std::endl;
  
  // return success
  close(fd);
  return 1;
}

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Sat, 2004-10-30 at 13:58 +0200, Ingo Molnar wrote:
> * Lee Revell <rlrevell@joe-job.com> wrote:
> 
> > Here is the dmesg output.  It looks like the problem could be related
> > to jackd's printing from the realtime thread.  But, this has to be the
> > kernel's fault on some level, because with an earlier version I get no
> > xruns.
> 
> with the earlier version these spinlocks were simply disabling
> preemption, while now they will schedule away on contention. If that tty
> lock is held for a long time by a lowprio task then that could delay the
> highprio thread. We are starting to see priority inversion problems. 
> But, the core issue is doing tty printouts - does jackd do that
> periodically, or only as a reaction to an already existing latency?
> 

No, this cannot be the whole story, because unless verbose mode is
specified, jackd will only prints anything if there is an xrun.  So
there is something else going on.

This _really_ feels like a kernel bug.  Are you saying that this could
still be a jackd problem, even though T3 works perfectly with the exact
same jackd binary?

Lee

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Sat, 2004-10-30 at 13:16 +0200, Ingo Molnar wrote:
> * Lee Revell <rlrevell@joe-job.com> wrote:
> 
> >   835 80000000 0.473ms (+0.000ms): finish_task_switch (__schedule)
> >   835 80000000 0.474ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
> >   835 80000000 0.474ms (+0.849ms): (835) ((49))
> >   835 80000000 1.324ms (+0.000ms): do_IRQ (finish_task_switch)
> >   835 80000000 1.324ms (+0.001ms): do_IRQ ((0))
> >   835 80000000 1.325ms (+0.002ms): mask_and_ack_8259A (__do_IRQ)
> 
> this seems to be very similar to the DMA problems Mark H Johnson had. It
> is almost certainly not caused by the kernel. It could in theory be some
> SMM overhead, but the bigger likelyhood is disk DMA.
> 

I find this hard to believe - I _never_ saw that problem on this system
before.  And it works _perfectly_ with T3 - I can run hundreds of
millions of process cycles without a single xrun.

Lee

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

On Sat, 30 Oct 2004 13:28:05 +0200
Ingo Molnar <mingo@elte.hu> wrote:

> > I think there was a patch posted to the JACK mailing list to print
> > from a separate thread, I will look into this.
> 
> that would be the better longterm solution.

afaik jack should ony print from the RT thread when something bad happened
(like an xrun), so in normal operation this shouldn't make problems. but i'm
not sure. maybe paul knows more.

flo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

> and another one:
> 
> preemption latency trace v1.0.7 on 2.6.9-mm1-RT-V0.5.14-rt
> -------------------------------------------------------
>  latency: 225 us, entries: 36 (36)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
>     -----------------
>     | task: jackd/1083, uid:1000 nice:0 policy:1 rt_prio:60
>     -----------------
>  => started at: try_to_wake_up+0x5a/0x110 <c01151ca>
>  => ended at:   finish_task_switch+0x31/0xc0 <c0115691>
> =======>
>   637 80000000 0.000ms (+0.000ms): (39) ((116))
>   637 80000000 0.000ms (+0.000ms): (1083) ((637))
>   637 80000000 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up)
>   637 80000000 0.000ms (+0.000ms): preempt_schedule (__up_write)
>   637 00000000 0.000ms (+0.000ms): preempt_schedule (up_write_mutex)
>   637 80000000 0.000ms (+0.000ms): __schedule (preempt_schedule)
>   637 80000000 0.000ms (+0.000ms): profile_hit (__schedule)
>   637 80000000 0.000ms (+0.001ms): sched_clock (__schedule)
>  1083 80000000 0.002ms (+0.000ms): __switch_to (__schedule)
>  1083 80000000 0.002ms (+0.000ms): (637) ((1083))
>  1083 80000000 0.002ms (+0.000ms): (116) ((39))
>  1083 80000000 0.002ms (+0.000ms): finish_task_switch (__schedule)
>  1083 80000000 0.002ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
>  1083 80000000 0.002ms (+0.000ms): (1083) ((39))
>  1083 00000000 0.003ms (+0.000ms): bad_range (free_pages_bulk)
>  1083 00000000 0.003ms (+0.000ms): bad_range (free_pages_bulk)
>  1083 00000000 0.003ms (+0.000ms): _mutex_unlock_irqrestore (free_pages_bulk)
>  1083 00000000 0.003ms (+0.000ms): up_mutex (free_pages_bulk)
>  1083 00000000 0.004ms (+0.000ms): up_write_mutex (free_pages_bulk)
>  1083 00000000 0.004ms (+0.000ms): __up_write (up_write_mutex)
>  1083 80000000 0.004ms (+0.000ms): wake_up_process_mutex (__up_write)
>  1083 80000000 0.004ms (+0.000ms): try_to_wake_up (wake_up_process_mutex)
>  1083 80000000 0.004ms (+0.000ms): task_rq_lock (try_to_wake_up)
>  1083 80000000 0.004ms (+0.000ms): activate_task (try_to_wake_up)
>  1083 80000000 0.004ms (+0.000ms): sched_clock (activate_task)
>  1083 80000000 0.004ms (+0.000ms): recalc_task_prio (activate_task)
>  1083 80000000 0.004ms (+0.000ms): effective_prio (recalc_task_prio)
>  1083 80000000 0.004ms (+0.000ms): enqueue_task (activate_task)
>  1083 80000000 0.005ms (+0.000ms): (115) ((39))
>  1083 80000000 0.005ms (+0.006ms): (782) ((1083))
>  1083 00000000 0.011ms (+0.000ms): sys_gettimeofday (syscall_call)
>  1083 00000000 0.011ms (+0.000ms): user_trace_stop (sys_gettimeofday)
>  1083 80000000 0.012ms (+0.000ms): user_trace_stop (sys_gettimeofday)
>  1083 80000000 0.012ms (+0.000ms): update_max_trace (user_trace_stop)
>  1083 80000000 0.012ms (+0.000ms): _mmx_memcpy (update_max_trace)
>  1083 80000000 0.013ms (+0.000ms): kernel_fpu_begin (_mmx_memcpy)

should have mentioned that in the user-triggered modus you have to do
the latency measurement in userspace. It is only the trace that is
correct, for the time being. This trace shows what i'd expect a jackd
wakeup to look like normally: 13 usecs.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> Here is the dmesg output.  It looks like the problem could be related
> to jackd's printing from the realtime thread.  But, this has to be the
> kernel's fault on some level, because with an earlier version I get no
> xruns.

with the earlier version these spinlocks were simply disabling
preemption, while now they will schedule away on contention. If that tty
lock is held for a long time by a lowprio task then that could delay the
highprio thread. We are starting to see priority inversion problems. 
But, the core issue is doing tty printouts - does jackd do that
periodically, or only as a reaction to an already existing latency?

> jackd:1846 userspace BUG: scheduling in user-atomic context!
>  [<c01069fc>] dump_stack+0x1c/0x20 (20)
>  [<c0283e60>] schedule+0x70/0x100 (24)
>  [<c0119efa>] do_exit+0x29a/0x500 (24)
>  [<c011a196>] sys_exit+0x16/0x20 (12)
>  [<c0106367>] syscall_call+0x7/0xb (-8124)

this one is interesting - does the jackd highprio thread start new
threads and lets them exit? The above schedule() is the final one of an
exit()-ing thread.

> jackd:1854 userspace BUG: scheduling in user-atomic context!
>  [<c01069fc>] dump_stack+0x1c/0x20 (20)
>  [<c0283e60>] schedule+0x70/0x100 (24)
>  [<c0119efa>] do_exit+0x29a/0x500 (24)

same exit() scenario. That would be pretty much a no-no, a new child
thread inherits the parent's SCHED_FIFO priority and due to
child-runs-first it could delay the parent possibly indefinitely.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Paul Davis <paul@linuxaudiosystems.com> wrote:

> IIRC, Rui was running with -p128, which at 48000kHz is 2600usecs (and
> longer at 44100kHz). If the maximum process cycle was on the order of
> 1500usecs, that leaves nearly 1ms until the next interrupt is due.
> Unless jackd was held up between computing the process cycle time and
> entering poll, it doesn't seem that the process cycle ever gets close
> to the interrupt interval duration.

yeah, i'd agree with this, there must be something else going on.

> So I don't think that delays caused *during* jackd's processing cycle
> are involved here. However, I agree that your suggestion to check for
> this before computing max_delay is probably wise in general.

yep.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> > jackd:1507 userspace BUG: scheduling in user-atomic context!
> >  [<c01069fc>] dump_stack+0x1c/0x20 (20)
> >  [<c02834d0>] schedule+0x70/0x100 (24)
> >  [<c010639b>] work_resched+0x6/0x17 (-8124)

> > jackd:1507 userspace BUG: scheduling in user-atomic context!
> >  [<c01069fc>] dump_stack+0x1c/0x20 (20)
> >  [<c02834d0>] schedule+0x70/0x100 (24)
> >  [<c028465d>] down_write_mutex+0xbd/0x180 (36)
> >  [<c012cbf6>] __mutex_lock+0x36/0x40 (16)
> >  [<c012cc75>] _mutex_lock_irqsave+0x15/0x20 (16)
> >  [<c01f1237>] tty_ldisc_try+0x17/0x50 (20)
> >  [<c01f1287>] tty_ldisc_ref_wait+0x17/0xc0 (88)

> this too i'd consider a false positive, because this particular
> printout happened due to a mutex, and a mutex blockage we can consider
> involuntary blockage. [...]

i've uploaded -V0.5.15 with these two false positives fixed, could you
retry your Jackd workload and see whether any other type of scheduling
gets detected? Please also chrt the watchdog thread to a lower priority
than the main thread's priority.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> Ingo, here are some of my traces with the same settings.  These do
> seem to correspond to the xruns.  Many of them look tty related -
> could the recent changes to the tty layer be responsible?  Possibly
> this has nothing to do with RT preempt, but is an unrelated bug in
> -mm?  The xruns do seem to correspond to display activity such as
> switching tabs in gnome-terminal. 
> 
> jackd:1507 userspace BUG: scheduling in user-atomic context!
>  [<c01069fc>] dump_stack+0x1c/0x20 (20)
>  [<c02834d0>] schedule+0x70/0x100 (24)
>  [<c010639b>] work_resched+0x6/0x17 (-8124)

ok, this shows jackd getting preempted by a higher-prio task. This could
be the watchdog thread - could you chrt the watchdog thread to make sure
jackd itself never gets (legitimately) preempted by any other thread?

also, in my tree i fixed this codepath to not trigger an atomicity
warning, since this is an involuntary rescheduling event.

> jackd:1507 userspace BUG: scheduling in user-atomic context!
>  [<c01069fc>] dump_stack+0x1c/0x20 (20)
>  [<c02834d0>] schedule+0x70/0x100 (24)
>  [<c028465d>] down_write_mutex+0xbd/0x180 (36)
>  [<c012cbf6>] __mutex_lock+0x36/0x40 (16)
>  [<c012cc75>] _mutex_lock_irqsave+0x15/0x20 (16)
>  [<c01f1237>] tty_ldisc_try+0x17/0x50 (20)
>  [<c01f1287>] tty_ldisc_ref_wait+0x17/0xc0 (88)
>  [<c01f21fd>] tty_write+0x7d/0x230 (68)
>  [<c01546ac>] vfs_write+0xbc/0x110 (36)
>  [<c01547b1>] sys_write+0x41/0x70 (44)
>  [<c0106367>] syscall_call+0x7/0xb (-8124)

this too i'd consider a false positive, because this particular printout
happened due to a mutex, and a mutex blockage we can consider
involuntary blockage. But ... tty writes can cause BKL locking.

> We know that jackd prints from the realtime thread, and that in theory
> this could be a problem, in practice it works OK.  Maybe some recent
> changes to the tty layer made this problematic.

i'd suggest to do an atomic_off() whenever jackd prints an xrun warning 
from the highprio thread.

> I think there was a patch posted to the JACK mailing list to print
> from a separate thread, I will look into this.

that would be the better longterm solution.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

>   835 80000000 0.473ms (+0.000ms): finish_task_switch (__schedule)
>   835 80000000 0.474ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
>   835 80000000 0.474ms (+0.849ms): (835) ((49))
>   835 80000000 1.324ms (+0.000ms): do_IRQ (finish_task_switch)
>   835 80000000 1.324ms (+0.001ms): do_IRQ ((0))
>   835 80000000 1.325ms (+0.002ms): mask_and_ack_8259A (__do_IRQ)

this seems to be very similar to the DMA problems Mark H Johnson had. It
is almost certainly not caused by the kernel. It could in theory be some
SMM overhead, but the bigger likelyhood is disk DMA.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Fri, 2004-10-29 at 23:36 -0400, Lee Revell wrote:
> Ingo, here are some of my traces with the same settings.  These do seem
> to correspond to the xruns.  Many of them look tty related - could the
> recent changes to the tty layer be responsible?  Possibly this has
> nothing to do with RT preempt, but is an unrelated bug in -mm?  The
> xruns do seem to correspond to display activity such as switching tabs
> in gnome-terminal. 
> 

OK here is a trace that might explain the xruns:

preemption latency trace v1.0.7 on 2.6.9-mm1-RT-V0.5.14
-------------------------------------------------------
 latency: 476 us, entries: 260 (260)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
    -----------------
    | task: jackd/1726, uid:1000 nice:0 policy:1 rt_prio:10
    -----------------
 => started at: try_to_wake_up+0x4c/0x100 <c01126ec>
 => ended at:   finish_task_switch+0x27/0xb0 <c0112b77>
=======>
    4 80000000 0.000ms (+0.000ms): (49) ((98))
    4 80000000 0.000ms (+0.000ms): (835) ((4))
    4 80000000 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up)
    4 80000000 0.001ms (+0.000ms): preempt_schedule (__do_IRQ)
    4 80000000 0.001ms (+0.000ms): irq_exit (do_IRQ)
    4 80000000 0.002ms (+0.000ms): do_softirq (irq_exit)
    4 80000000 0.002ms (+0.075ms): __do_softirq (do_softirq)
    4 80000000 0.078ms (+0.000ms): do_IRQ (_mmx_memcpy)
    4 80000000 0.078ms (+0.000ms): do_IRQ ((0))
    4 80000000 0.079ms (+0.002ms): mask_and_ack_8259A (__do_IRQ)
    4 80000000 0.081ms (+0.000ms): preempt_schedule (__do_IRQ)
    4 80000000 0.082ms (+0.000ms): redirect_hardirq (__do_IRQ)
    4 80000000 0.082ms (+0.000ms): preempt_schedule (__do_IRQ)
    4 80000000 0.083ms (+0.000ms): handle_IRQ_event (__do_IRQ)
    4 80000000 0.083ms (+0.000ms): timer_interrupt (handle_IRQ_event)
    4 80000000 0.084ms (+0.003ms): mark_offset_tsc (timer_interrupt)
    4 80000000 0.088ms (+0.001ms): preempt_schedule (mark_offset_tsc)
    4 80000000 0.089ms (+0.000ms): preempt_schedule (mark_offset_tsc)
    4 80000000 0.089ms (+0.000ms): do_timer (timer_interrupt)
    4 80000000 0.090ms (+0.000ms): update_process_times (timer_interrupt)
    4 80000000 0.090ms (+0.000ms): update_one_process (update_process_times)
    4 80000000 0.091ms (+0.000ms): run_local_timers (update_process_times)
    4 80000000 0.091ms (+0.000ms): raise_softirq (update_process_times)
    4 80000000 0.091ms (+0.000ms): scheduler_tick (update_process_times)
    4 80000000 0.092ms (+0.001ms): sched_clock (scheduler_tick)
    4 80000000 0.093ms (+0.000ms): preempt_schedule (scheduler_tick)
    4 80000000 0.093ms (+0.000ms): profile_hit (timer_interrupt)
    4 80000000 0.094ms (+0.000ms): preempt_schedule (timer_interrupt)
    4 80000000 0.094ms (+0.000ms): note_interrupt (__do_IRQ)
    4 80000000 0.095ms (+0.000ms): end_8259A_irq (__do_IRQ)
    4 80000000 0.095ms (+0.001ms): enable_8259A_irq (__do_IRQ)
    4 80000000 0.097ms (+0.000ms): preempt_schedule (__do_IRQ)
    4 80000000 0.097ms (+0.000ms): preempt_schedule (__do_IRQ)
    4 80000000 0.098ms (+0.000ms): irq_exit (do_IRQ)
    4 80000000 0.098ms (+0.000ms): do_softirq (irq_exit)
    4 80000000 0.098ms (+0.371ms): __do_softirq (do_softirq)
    4 00000000 0.470ms (+0.000ms): preempt_schedule (_mmx_memcpy)
    4 80000000 0.470ms (+0.000ms): __schedule (preempt_schedule)
    4 80000000 0.470ms (+0.000ms): profile_hit (__schedule)
    4 80000000 0.471ms (+0.001ms): sched_clock (__schedule)
  835 80000000 0.472ms (+0.000ms): __switch_to (__schedule)
  835 80000000 0.473ms (+0.000ms): (4) ((835))
  835 80000000 0.473ms (+0.000ms): (98) ((49))
  835 80000000 0.473ms (+0.000ms): finish_task_switch (__schedule)
  835 80000000 0.474ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
  835 80000000 0.474ms (+0.849ms): (835) ((49))
  835 80000000 1.324ms (+0.000ms): do_IRQ (finish_task_switch)
  835 80000000 1.324ms (+0.001ms): do_IRQ ((0))
  835 80000000 1.325ms (+0.002ms): mask_and_ack_8259A (__do_IRQ)
  835 80000000 1.328ms (+0.000ms): redirect_hardirq (__do_IRQ)
  835 80000000 1.329ms (+0.000ms): handle_IRQ_event (__do_IRQ)
  835 80000000 1.329ms (+0.001ms): timer_interrupt (handle_IRQ_event)
  835 80000000 1.330ms (+0.004ms): mark_offset_tsc (timer_interrupt)
  835 80000000 1.335ms (+0.000ms): do_timer (timer_interrupt)
  835 80000000 1.335ms (+0.000ms): update_process_times (timer_interrupt)
  835 80000000 1.336ms (+0.000ms): update_one_process (update_process_times)
  835 80000000 1.336ms (+0.000ms): run_local_timers (update_process_times)
  835 80000000 1.337ms (+0.000ms): raise_softirq (update_process_times)

Lee

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Sat, 2004-10-30 at 00:31 +0200, Florian Schmidt wrote:
> On Fri, 29 Oct 2004 23:25:45 +0200
> Ingo Molnar <mingo@elte.hu> wrote:
> 
> > > will do so. btw: i think i'm a bit confused right now. What debugging
> > > features should i have enabled for this test?
> > 
> > this particular one (atomicity-checking) is always-enabled if you have
> > the -RT patch applied (it's a really cheap check).
> > 
> > for the 'application-triggered tracing' facility we talked about earlier
> > is only active if LATENCY_TRACING is enabled. In that case to turn the 
> > tracer on, call:
> [snip]
> 
> Ok,
> 
> .config attached,
> 
> running the patched jackd -R -P 60 -d alsa -p64 for a bit and provoking
> xruns via window wiggling :)
> 

Ingo, here are some of my traces with the same settings.  These do seem
to correspond to the xruns.  Many of them look tty related - could the
recent changes to the tty layer be responsible?  Possibly this has
nothing to do with RT preempt, but is an unrelated bug in -mm?  The
xruns do seem to correspond to display activity such as switching tabs
in gnome-terminal. 

jackd:1507 userspace BUG: scheduling in user-atomic context!
 [<c01069fc>] dump_stack+0x1c/0x20 (20)
 [<c02834d0>] schedule+0x70/0x100 (24)
 [<c010639b>] work_resched+0x6/0x17 (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e5d3>] .... print_traces+0x13/0x50
.....[<c01069fc>] ..   ( <= dump_stack+0x1c/0x20)

jackd:1507 userspace BUG: scheduling in user-atomic context!
 [<c01069fc>] dump_stack+0x1c/0x20 (20)
 [<c02834d0>] schedule+0x70/0x100 (24)
 [<c028465d>] down_write_mutex+0xbd/0x180 (36)
 [<c012cbf6>] __mutex_lock+0x36/0x40 (16)
 [<c012cc75>] _mutex_lock_irqsave+0x15/0x20 (16)
 [<c01f1237>] tty_ldisc_try+0x17/0x50 (20)
 [<c01f1287>] tty_ldisc_ref_wait+0x17/0xc0 (88)
 [<c01f21fd>] tty_write+0x7d/0x230 (68)
 [<c01546ac>] vfs_write+0xbc/0x110 (36)
 [<c01547b1>] sys_write+0x41/0x70 (44)
 [<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e5d3>] .... print_traces+0x13/0x50
.....[<c01069fc>] ..   ( <= dump_stack+0x1c/0x20)

jackd:1507 userspace BUG: scheduling in user-atomic context!
 [<c01069fc>] dump_stack+0x1c/0x20 (20)
 [<c02834d0>] schedule+0x70/0x100 (24)
 [<c010639b>] work_resched+0x6/0x17 (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e5d3>] .... print_traces+0x13/0x50
.....[<c01069fc>] ..   ( <= dump_stack+0x1c/0x20)

jackd:1507 userspace BUG: scheduling in user-atomic context!
 [<c01069fc>] dump_stack+0x1c/0x20 (20)
 [<c02834d0>] schedule+0x70/0x100 (24)
 [<c028465d>] down_write_mutex+0xbd/0x180 (36)
 [<c012cbf6>] __mutex_lock+0x36/0x40 (16)
 [<c012cc75>] _mutex_lock_irqsave+0x15/0x20 (16)
 [<c01f1237>] tty_ldisc_try+0x17/0x50 (20)
 [<c01f1287>] tty_ldisc_ref_wait+0x17/0xc0 (88)
 [<c01f21fd>] tty_write+0x7d/0x230 (68)
 [<c01546ac>] vfs_write+0xbc/0x110 (36)
 [<c01547b1>] sys_write+0x41/0x70 (44)
 [<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e5d3>] .... print_traces+0x13/0x50
.....[<c01069fc>] ..   ( <= dump_stack+0x1c/0x20)

jackd:1507 userspace BUG: scheduling in user-atomic context!
 [<c01069fc>] dump_stack+0x1c/0x20 (20)
 [<c02834d0>] schedule+0x70/0x100 (24)
 [<c028449c>] _down_write+0xcc/0x170 (32)
 [<c01131d3>] lock_kernel+0x23/0x30 (16)
 [<c01f22f0>] tty_write+0x170/0x230 (64)
 [<c01546ac>] vfs_write+0xbc/0x110 (36)
 [<c01547b1>] sys_write+0x41/0x70 (44)
 [<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e5d3>] .... print_traces+0x13/0x50
.....[<c01069fc>] ..   ( <= dump_stack+0x1c/0x20)

jackd:1507 userspace BUG: scheduling in user-atomic context!
 [<c01069fc>] dump_stack+0x1c/0x20 (20)
 [<c02834d0>] schedule+0x70/0x100 (24)
 [<c0119cba>] do_exit+0x29a/0x500 (24)
 [<c0119f56>] sys_exit+0x16/0x20 (12)
 [<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e5d3>] .... print_traces+0x13/0x50
.....[<c01069fc>] ..   ( <= dump_stack+0x1c/0x20)


However the longest latency recorded during all these xruns is 40-150 usecs:


preemption latency trace v1.0.7 on 2.6.9-mm1-RT-V0.5.14
-------------------------------------------------------
 latency: 48 us, entries: 26 (26)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
    -----------------
    | task: jackd/1629, uid:1000 nice:0 policy:1 rt_prio:10
    -----------------
 => started at: try_to_wake_up+0x4c/0x100 <c01126ec>
 => ended at:   finish_task_switch+0x27/0xb0 <c0112b77>
=======>
 1101 80000000 0.000ms (+0.000ms): (89) ((116))
 1101 80000000 0.000ms (+0.000ms): (1629) ((1101))
 1101 80000000 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up)
 1101 80000000 0.001ms (+0.000ms): preempt_schedule (__up_write)
 1101 00000000 0.001ms (+0.000ms): preempt_schedule (up_write_mutex)
 1101 80000000 0.002ms (+0.000ms): __schedule (preempt_schedule)
 1101 80000000 0.002ms (+0.000ms): profile_hit (__schedule)
 1101 80000000 0.003ms (+0.002ms): sched_clock (__schedule)
 1629 80000000 0.005ms (+0.000ms): __switch_to (__schedule)
 1629 80000000 0.006ms (+0.000ms): (1101) ((1629))
 1629 80000000 0.006ms (+0.000ms): (116) ((89))
 1629 80000000 0.006ms (+0.000ms): finish_task_switch (__schedule)
 1629 80000000 0.007ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
 1629 80000000 0.007ms (+0.002ms): (1629) ((89))
 1629 00000000 0.010ms (+0.000ms): bad_range (free_pages_bulk)
 1629 00000000 0.010ms (+0.000ms): bad_range (free_pages_bulk)
 1629 00000000 0.011ms (+0.000ms): _mutex_unlock_irqrestore (free_pages_bulk)
 1629 00000000 0.011ms (+0.000ms): up_mutex (free_pages_bulk)
 1629 00000000 0.011ms (+0.000ms): up_write_mutex (free_pages_bulk)
 1629 00000000 0.012ms (+0.004ms): __up_write (up_write_mutex)
 1629 00000000 0.016ms (+0.000ms): sys_gettimeofday (syscall_call)
 1629 00000000 0.016ms (+0.000ms): user_trace_stop (sys_gettimeofday)
 1629 80000000 0.017ms (+0.000ms): user_trace_stop (sys_gettimeofday)
 1629 80000000 0.017ms (+0.000ms): update_max_trace (user_trace_stop)
 1629 80000000 0.018ms (+0.000ms): _mmx_memcpy (update_max_trace)
 1629 80000000 0.018ms (+0.000ms): kernel_fpu_begin (_mmx_memcpy)

We know that jackd prints from the realtime thread, and that in theory
this could be a problem, in practice it works OK.  Maybe some recent
changes to the tty layer made this problematic.

I think there was a patch posted to the JACK mailing list to print from
a separate thread, I will look into this.

Lee  

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Fri, 2004-10-29 at 23:25 +0200, Ingo Molnar wrote:
> this particular one (atomicity-checking) is always-enabled if you have
> the -RT patch applied (it's a really cheap check).

I get these as well.  Basically I think these are the exact same results
Florian had.

jackd:1646 userspace BUG: scheduling in user-atomic context!
 [<c02733eb>] schedule+0x6b/0xf0 (8)
 [<c027431d>] _down_write+0xcd/0x130 (24)
 [<c0111e1e>] lock_kernel+0x1e/0x30 (32)
 [<c01e7ad9>] tty_write+0x189/0x240 (16)
 [<c01ecf20>] write_chan+0x0/0x210 (28)
 [<c014ea4a>] vfs_write+0xba/0x100 (32)
 [<c014eb3d>] sys_write+0x3d/0x70 (36)
 [<c0105d07>] syscall_call+0x7/0xb (40)
jackd:1646 userspace BUG: scheduling in user-atomic context!
 [<c02733eb>] schedule+0x6b/0xf0 (8)
 [<c027431d>] _down_write+0xcd/0x130 (24)
 [<c0111e1e>] lock_kernel+0x1e/0x30 (32)
 [<c01e7ad9>] tty_write+0x189/0x240 (16)
 [<c01ecf20>] write_chan+0x0/0x210 (28)
 [<c014ea4a>] vfs_write+0xba/0x100 (32)
 [<c014eb3d>] sys_write+0x3d/0x70 (36)
 [<c0105d07>] syscall_call+0x7/0xb (40)

A known issue with JACK is that it prints from the realtime thread.  I
think someone might have posted a fix for this recently.

Lee

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Fri, 2004-10-29 at 23:25 +0200, Ingo Molnar wrote:
> this particular one (atomicity-checking) is always-enabled if you have
> the -RT patch applied (it's a really cheap check).

I tried this patch with all DEBUG options disabled, and this reduces the
xruns, but I still get a few.  This is what I get in the logs when
running jackd with your patch:

jackd:1590 userspace BUG: scheduling in user-atomic context!
 [<c02733eb>] schedule+0x6b/0xf0 (8)
 [<c0105d3b>] work_resched+0x6/0x17 (24)
jackd:1590 userspace BUG: scheduling in user-atomic context!
 [<c02733eb>] schedule+0x6b/0xf0 (8)
 [<c0105d3b>] work_resched+0x6/0x17 (24)
jackd:1590 userspace BUG: scheduling in user-atomic context!
 [<c02733eb>] schedule+0x6b/0xf0 (8)
 [<c0105d3b>] work_resched+0x6/0x17 (24)
jackd:1590 userspace BUG: scheduling in user-atomic context!
 [<c02733eb>] schedule+0x6b/0xf0 (8)
 [<c0105d3b>] work_resched+0x6/0x17 (24)
jackd:1590 userspace BUG: scheduling in user-atomic context!
 [<c02733eb>] schedule+0x6b/0xf0 (8)
 [<c0105d3b>] work_resched+0x6/0x17 (24)
jackd:1590 userspace BUG: scheduling in user-atomic context!
 [<c02733eb>] schedule+0x6b/0xf0 (8)
 [<c0105d3b>] work_resched+0x6/0x17 (24)
jackd:1590 userspace BUG: scheduling in user-atomic context!
 [<c02733eb>] schedule+0x6b/0xf0 (8)
 [<c0105d3b>] work_resched+0x6/0x17 (24)
jackd:1590 userspace BUG: scheduling in user-atomic context!
 [<c02733eb>] schedule+0x6b/0xf0 (8)
 [<c0105d3b>] work_resched+0x6/0x17 (24)

Lee

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Paul Davis]

>but i'd also suggest to put in a counter into that branch so that this
>condition doesnt get lost. In fact the Maximum Process Cycle stat from
>Rui:
>
>>>   Maximum Delay . . . . . . . . .    6904       921       721    usecs
>>>   Maximum Process Cycle . . . . .    1449      1469      1590    usecs
>
>seems to suggest that there can be significant processing delays? (if
>Maximum Process Cycle is indeed the time spent from poll_ret to the next
>poll_enter.)

IIRC, Rui was running with -p128, which at 48000kHz is 2600usecs (and
longer at 44100kHz). If the maximum process cycle was on the order of
1500usecs, that leaves nearly 1ms until the next interrupt is
due. Unless jackd was held up between computing the process cycle time
and entering poll, it doesn't seem that the process cycle ever gets
close to the interrupt interval duration.

So I don't think that delays caused *during* jackd's processing cycle
are involved here. However, I agree that your suggestion to check for
this before computing max_delay is probably wise in general.

--p

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Fri, 2004-10-29 at 23:46 +0200, Ingo Molnar wrote: 
> > One more question, what do you recommend the priorities of the IRQ
> > threads be set to?  AIUI for xrun-free operation with JACK, all that
> > is needed is to set the RT priorities of the soundcard IRQ thread
> > highest, followed by the JACK threads, then the other IRQ threads.  Is
> > this correct?
> 
> correct. softirqs are not used by the sound subsystem so there's no
> ksoftirqd dependency.
> 

I don't see much point in latency testing this one yet; I have had to go
for the reset button twice in the past hour.  Web surfing and a kernel
compile kill it pretty quickly.

Lockups aside, it's not necessary to use JACK to demonstrate the
problem, playing AVIs with mplayer will skip.

As a final test I am recompiling with all debug options disabled.

Lee

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Fri, 2004-10-29 at 23:46 +0200, Ingo Molnar wrote:
> * Lee Revell <rlrevell@joe-job.com> wrote:
> 
> > On Fri, 2004-10-29 at 23:25 +0200, Ingo Molnar wrote:
> > > > will do so. btw: i think i'm a bit confused right now. What debugging
> > > > features should i have enabled for this test?
> > > 
> > > this particular one (atomicity-checking) is always-enabled if you have
> > > the -RT patch applied (it's a really cheap check).
> > 
> > One more question, what do you recommend the priorities of the IRQ
> > threads be set to?  AIUI for xrun-free operation with JACK, all that
> > is needed is to set the RT priorities of the soundcard IRQ thread
> > highest, followed by the JACK threads, then the other IRQ threads.  Is
> > this correct?
> 
> correct. softirqs are not used by the sound subsystem so there's no
> ksoftirqd dependency.
> 

OK well I set all IRQ threads to SCHED_OTHER except the soundcard, which
is SCHED_FIFO, and I still get a LOT of xruns, compared to zero xruns
over tens of millions of cycles with T3.

rlrevell@mindpipe:~$ for p in `ps auxww | grep IRQ | grep -v grep | awk
'{print $2}'`; do chrt -p $p ;done
pid 647's current scheduling policy: SCHED_OTHER
pid 647's current scheduling priority: 0
pid 655's current scheduling policy: SCHED_OTHER
pid 655's current scheduling priority: 0
pid 678's current scheduling policy: SCHED_OTHER
pid 678's current scheduling priority: 0
pid 693's current scheduling policy: SCHED_OTHER
pid 693's current scheduling priority: 0
pid 831's current scheduling policy: SCHED_OTHER
pid 831's current scheduling priority: 0
pid 835's current scheduling policy: SCHED_FIFO  <-- soundcard irq
pid 835's current scheduling priority: 90

Here is the dmesg output.  It looks like the problem could be related to
jackd's printing from the realtime thread.  But, this has to be the
kernel's fault on some level, because with an earlier version I get no
xruns.

jackd:1726 userspace BUG: scheduling in user-atomic context!
 [<c01069fc>] dump_stack+0x1c/0x20 (20)
 [<c0283e60>] schedule+0x70/0x100 (24)
 [<c010639b>] work_resched+0x6/0x17 (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e833>] .... print_traces+0x13/0x50
.....[<c01069fc>] ..   ( <= dump_stack+0x1c/0x20)

jackd:1726 userspace BUG: scheduling in user-atomic context!
 [<c01069fc>] dump_stack+0x1c/0x20 (20)
 [<c0283e60>] schedule+0x70/0x100 (24)
 [<c0284e2c>] _down_write+0xcc/0x170 (32)
 [<c0113413>] lock_kernel+0x23/0x30 (16)
 [<c01f2570>] tty_write+0x170/0x230 (64)
 [<c01548ec>] vfs_write+0xbc/0x110 (36)
 [<c01549f1>] sys_write+0x41/0x70 (44)
 [<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e833>] .... print_traces+0x13/0x50
.....[<c01069fc>] ..   ( <= dump_stack+0x1c/0x20)

jackd:1726 userspace BUG: scheduling in user-atomic context!
 [<c01069fc>] dump_stack+0x1c/0x20 (20)
 [<c0283e60>] schedule+0x70/0x100 (24)
 [<c0119efa>] do_exit+0x29a/0x500 (24)
 [<c011a196>] sys_exit+0x16/0x20 (12)
 [<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e833>] .... print_traces+0x13/0x50
.....[<c01069fc>] ..   ( <= dump_stack+0x1c/0x20)

jackd:1731 userspace BUG: scheduling in user-atomic context!
 [<c01069fc>] dump_stack+0x1c/0x20 (20)
 [<c0283e60>] schedule+0x70/0x100 (24)
 [<c0119efa>] do_exit+0x29a/0x500 (24)
 [<c011a196>] sys_exit+0x16/0x20 (12)
 [<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e833>] .... print_traces+0x13/0x50
.....[<c01069fc>] ..   ( <= dump_stack+0x1c/0x20)

jackd:1736 userspace BUG: scheduling in user-atomic context!
 [<c01069fc>] dump_stack+0x1c/0x20 (20)
 [<c0283e60>] schedule+0x70/0x100 (24)
 [<c0119efa>] do_exit+0x29a/0x500 (24)
 [<c011a196>] sys_exit+0x16/0x20 (12)
 [<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e833>] .... print_traces+0x13/0x50
.....[<c01069fc>] ..   ( <= dump_stack+0x1c/0x20)

jackd:1775 userspace BUG: scheduling in user-atomic context!
 [<c01069fc>] dump_stack+0x1c/0x20 (20)
 [<c0283e60>] schedule+0x70/0x100 (24)
 [<c0119efa>] do_exit+0x29a/0x500 (24)
 [<c011a196>] (ksoftirqd/0/2/CPU#0): new 12 us maximum-latency wakeup.
(ksoftirqd/0/2/CPU#0): new 15 us maximum-latency wakeup.
(ksoftirqd/0/2/CPU#0): new 22 us maximum-latency wakeup.
(ksoftirqd/0/2/CPU#0): new 31 us maximum-latency wakeup.
(ksoftirqd/0/2/CPU#0): new 32 us maximum-latency wakeup.
(IRQ 1/693/CPU#0): new 39 us maximum-latency wakeup.
jackd:1787 userspace BUG: scheduling in user-atomic context!
 [<c01069fc>] dump_stack+0x1c/0x20 (20)
 [<c0283e60>] schedule+0x70/0x100 (24)
 [<c0119efa>] do_exit+0x29a/0x500 (24)
 [<c011a196>] sys_exit+0x16/0x20 (12)
 [<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e833>] .... print_traces+0x13/0x50
.....[<c01069fc>] ..   ( <= dump_stack+0x1c/0x20)

(ksoftirqd/0/2/CPU#0): new 42 us maximum-latency wakeup.
(desched/0/3/CPU#0): new 43 us maximum-latency wakeup.
(IRQ 15/678/CPU#0): new 44 us maximum-latency wakeup.
(IRQ 11/831/CPU#0): new 45 us maximum-latency wakeup.
(IRQ 11/831/CPU#0): new 52 us maximum-latency wakeup.
(IRQ 11/831/CPU#0): new 55 us maximum-latency wakeup.
jackd:1846 userspace BUG: scheduling in user-atomic context!
 [<c01069fc>] dump_stack+0x1c/0x20 (20)
 [<c0283e60>] schedule+0x70/0x100 (24)
 [<c0284acb>] schedule_timeout+0xbb/0xc0 (80)
 [<c012f11f>] futex_wait+0x10f/0x190 (168)
 [<c012f406>] do_futex+0x36/0x80 (32)
 [<c012f51a>] sys_futex+0xca/0xe0 (68)
 [<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e833>] .... print_traces+0x13/0x50
.....[<c01069fc>] ..   ( <= dump_stack+0x1c/0x20)

jackd:1846 userspace BUG: scheduling in user-atomic context!
 [<c01069fc>] dump_stack+0x1c/0x20 (20)
 [<c0283e60>] schedule+0x70/0x100 (24)
 [<c0119efa>] do_exit+0x29a/0x500 (24)
 [<c011a196>] sys_exit+0x16/0x20 (12)
 [<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e833>] .... print_traces+0x13/0x50
.....[<c01069fc>] ..   ( <= dump_stack+0x1c/0x20)

jackd:1854 userspace BUG: scheduling in user-atomic context!
 [<c01069fc>] dump_stack+0x1c/0x20 (20)
 [<c0283e60>] schedule+0x70/0x100 (24)
 [<c0119efa>] do_exit+0x29a/0x500 (24)
 [<c011a196>] sys_exit+0x16/0x20 (12)
 [<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e833>] .... print_traces+0x13/0x50
.....[<c01069fc>] ..   ( <= dump_stack+0x1c/0x20)

Lee

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

[-- Attachment #1: Type: text/plain, Size: 7223 bytes --]

On Sat, 30 Oct 2004 00:31:22 +0200
Florian Schmidt <mista.tapas@gmx.net> wrote:

> Ok,
> 
> .config attached,

now it is :)

> Oct 30 00:28:29 mango kernel: (IRQ 3/118/CPU#0): new 205 us maximum-latency wakeup.

i could as well include this one 

preemption latency trace v1.0.7 on 2.6.9-mm1-RT-V0.5.14-rt
-------------------------------------------------------
 latency: 205 us, entries: 36 (36)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
    -----------------
    | task: jackd/1083, uid:1000 nice:0 policy:1 rt_prio:60
    -----------------
 => started at: try_to_wake_up+0x5a/0x110 <c01151ca>
 => ended at:   finish_task_switch+0x31/0xc0 <c0115691>
=======>
  768 80000000 0.000ms (+0.000ms): (39) ((115))
  768 80000000 0.000ms (+0.000ms): (1083) ((768))
  768 80000000 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up)
  768 80000000 0.000ms (+0.000ms): preempt_schedule (__up_write)
  768 00000000 0.000ms (+0.000ms): preempt_schedule (up_write_mutex)
  768 80000000 0.000ms (+0.000ms): __schedule (preempt_schedule)
  768 80000000 0.000ms (+0.000ms): profile_hit (__schedule)
  768 80000000 0.000ms (+0.000ms): sched_clock (__schedule)
 1083 80000000 0.001ms (+0.000ms): __switch_to (__schedule)
 1083 80000000 0.002ms (+0.000ms): (768) ((1083))
 1083 80000000 0.002ms (+0.000ms): (115) ((39))
 1083 80000000 0.002ms (+0.000ms): finish_task_switch (__schedule)
 1083 80000000 0.002ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
 1083 80000000 0.002ms (+0.000ms): (1083) ((39))
 1083 00000000 0.003ms (+0.000ms): bad_range (free_pages_bulk)
 1083 00000000 0.003ms (+0.000ms): bad_range (free_pages_bulk)
 1083 00000000 0.003ms (+0.000ms): _mutex_unlock_irqrestore (free_pages_bulk)
 1083 00000000 0.003ms (+0.000ms): up_mutex (free_pages_bulk)
 1083 00000000 0.003ms (+0.000ms): up_write_mutex (free_pages_bulk)
 1083 00000000 0.003ms (+0.000ms): __up_write (up_write_mutex)
 1083 80000000 0.004ms (+0.000ms): wake_up_process_mutex (__up_write)
 1083 80000000 0.004ms (+0.000ms): try_to_wake_up (wake_up_process_mutex)
 1083 80000000 0.004ms (+0.000ms): task_rq_lock (try_to_wake_up)
 1083 80000000 0.004ms (+0.000ms): activate_task (try_to_wake_up)
 1083 80000000 0.004ms (+0.000ms): sched_clock (activate_task)
 1083 80000000 0.004ms (+0.000ms): recalc_task_prio (activate_task)
 1083 80000000 0.004ms (+0.000ms): effective_prio (recalc_task_prio)
 1083 80000000 0.005ms (+0.000ms): enqueue_task (activate_task)
 1083 80000000 0.005ms (+0.000ms): (115) ((39))
 1083 80000000 0.005ms (+0.004ms): (834) ((1083))
 1083 00000000 0.010ms (+0.000ms): sys_gettimeofday (syscall_call)
 1083 00000000 0.010ms (+0.000ms): user_trace_stop (sys_gettimeofday)
 1083 80000000 0.010ms (+0.000ms): user_trace_stop (sys_gettimeofday)
 1083 80000000 0.010ms (+0.000ms): update_max_trace (user_trace_stop)
 1083 80000000 0.011ms (+0.000ms): _mmx_memcpy (update_max_trace)
 1083 80000000 0.011ms (+0.000ms): kernel_fpu_begin (_mmx_memcpy)


and another one:

preemption latency trace v1.0.7 on 2.6.9-mm1-RT-V0.5.14-rt
-------------------------------------------------------
 latency: 225 us, entries: 36 (36)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
    -----------------
    | task: jackd/1083, uid:1000 nice:0 policy:1 rt_prio:60
    -----------------
 => started at: try_to_wake_up+0x5a/0x110 <c01151ca>
 => ended at:   finish_task_switch+0x31/0xc0 <c0115691>
=======>
  637 80000000 0.000ms (+0.000ms): (39) ((116))
  637 80000000 0.000ms (+0.000ms): (1083) ((637))
  637 80000000 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up)
  637 80000000 0.000ms (+0.000ms): preempt_schedule (__up_write)
  637 00000000 0.000ms (+0.000ms): preempt_schedule (up_write_mutex)
  637 80000000 0.000ms (+0.000ms): __schedule (preempt_schedule)
  637 80000000 0.000ms (+0.000ms): profile_hit (__schedule)
  637 80000000 0.000ms (+0.001ms): sched_clock (__schedule)
 1083 80000000 0.002ms (+0.000ms): __switch_to (__schedule)
 1083 80000000 0.002ms (+0.000ms): (637) ((1083))
 1083 80000000 0.002ms (+0.000ms): (116) ((39))
 1083 80000000 0.002ms (+0.000ms): finish_task_switch (__schedule)
 1083 80000000 0.002ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
 1083 80000000 0.002ms (+0.000ms): (1083) ((39))
 1083 00000000 0.003ms (+0.000ms): bad_range (free_pages_bulk)
 1083 00000000 0.003ms (+0.000ms): bad_range (free_pages_bulk)
 1083 00000000 0.003ms (+0.000ms): _mutex_unlock_irqrestore (free_pages_bulk)
 1083 00000000 0.003ms (+0.000ms): up_mutex (free_pages_bulk)
 1083 00000000 0.004ms (+0.000ms): up_write_mutex (free_pages_bulk)
 1083 00000000 0.004ms (+0.000ms): __up_write (up_write_mutex)
 1083 80000000 0.004ms (+0.000ms): wake_up_process_mutex (__up_write)
 1083 80000000 0.004ms (+0.000ms): try_to_wake_up (wake_up_process_mutex)
 1083 80000000 0.004ms (+0.000ms): task_rq_lock (try_to_wake_up)
 1083 80000000 0.004ms (+0.000ms): activate_task (try_to_wake_up)
 1083 80000000 0.004ms (+0.000ms): sched_clock (activate_task)
 1083 80000000 0.004ms (+0.000ms): recalc_task_prio (activate_task)
 1083 80000000 0.004ms (+0.000ms): effective_prio (recalc_task_prio)
 1083 80000000 0.004ms (+0.000ms): enqueue_task (activate_task)
 1083 80000000 0.005ms (+0.000ms): (115) ((39))
 1083 80000000 0.005ms (+0.006ms): (782) ((1083))
 1083 00000000 0.011ms (+0.000ms): sys_gettimeofday (syscall_call)
 1083 00000000 0.011ms (+0.000ms): user_trace_stop (sys_gettimeofday)
 1083 80000000 0.012ms (+0.000ms): user_trace_stop (sys_gettimeofday)
 1083 80000000 0.012ms (+0.000ms): update_max_trace (user_trace_stop)
 1083 80000000 0.012ms (+0.000ms): _mmx_memcpy (update_max_trace)
 1083 80000000 0.013ms (+0.000ms): kernel_fpu_begin (_mmx_memcpy)


Oct 30 00:48:45 mango kernel: (IRQ 3/118/CPU#0): new 225 us maximum-latency wakeup.
Oct 30 00:49:15 mango kernel: jackd:1083 userspace BUG: scheduling in user-atomic context!
Oct 30 00:49:15 mango kernel:  [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:49:15 mango kernel:  [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:49:15 mango kernel:  [Bad_offset+13/23] down_write_mutex+0xc2/0x1a0 (48)
Oct 30 00:49:15 mango kernel:  [do_posix_clock_monotonic_gettime_parts+83/96] __mutex_lock+0x43/0x50 (24)
Oct 30 00:49:15 mango kernel:  [do_posix_clock_nonanosleep+13/16] _mutex_lock_irqsave+0x1d/0x30 (16)
Oct 30 00:49:15 mango kernel:  [stop_tty+2/144] tty_ldisc_try+0x22/0x60 (24)
Oct 30 00:49:15 mango kernel:  [stop_tty+88/144] tty_ldisc_ref_wait+0x18/0xb0 (84)
Oct 30 00:49:15 mango kernel:  [release_dev+865/1984] tty_write+0x91/0x230 (80)
Oct 30 00:49:15 mango kernel:  [sys_llseek+90/176] vfs_write+0xca/0x140 (44)
Oct 30 00:49:15 mango kernel:  [vfs_read+16/320] sys_write+0x50/0x80 (44)
Oct 30 00:49:15 mango kernel:  [irq_entries_start+3/128] syscall_call+0x7/0xb (-8124)
Oct 30 00:49:15 mango kernel: ---------------------------
Oct 30 00:49:15 mango kernel: | preempt count: 00000001 ]
Oct 30 00:49:15 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:49:15 mango kernel: ----------------------------------------
Oct 30 00:49:15 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:49:15 mango kernel: .....[show_registers+99/464] ..   ( <= dump_stack+0x23/0x30)
Oct 30 00:49:15 mango kernel: 

[-- Attachment #2: .config --]
[-- Type: application/octet-stream, Size: 26233 bytes --]

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.9-mm1-RT-V0.5.14
# Fri Oct 29 23:49:01 2004
#
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y

#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y
CONFIG_CLEAN_COMPILE=y
CONFIG_BROKEN_ON_SMP=y

#
# General setup
#
CONFIG_LOCALVERSION="-rt"
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_BSD_PROCESS_ACCT=y
# CONFIG_BSD_PROCESS_ACCT_V3 is not set
CONFIG_SYSCTL=y
# CONFIG_AUDIT is not set
CONFIG_LOG_BUF_SHIFT=14
# CONFIG_HOTPLUG is not set
CONFIG_KOBJECT_UEVENT=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
# CONFIG_EMBEDDED is not set
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SHMEM=y
CONFIG_CC_ALIGN_FUNCTIONS=0
CONFIG_CC_ALIGN_LABELS=0
CONFIG_CC_ALIGN_LOOPS=0
CONFIG_CC_ALIGN_JUMPS=0
# CONFIG_TINY_SHMEM is not set

#
# Loadable module support
#
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
CONFIG_OBSOLETE_MODPARM=y
CONFIG_MODVERSIONS=y
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_KMOD=y

#
# Processor type and features
#
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_NUMAQ is not set
# CONFIG_X86_SUMMIT is not set
# CONFIG_X86_BIGSMP is not set
# CONFIG_X86_VISWS is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_ES7000 is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
CONFIG_MK7=y
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_X86_GENERIC is not set
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_RWSEM_GENERIC=y
CONFIG_GENERIC_SEMAPHORES=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_USE_3DNOW=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
# CONFIG_SMP is not set
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
CONFIG_PREEMPT_REALTIME=y
CONFIG_PREEMPT_BKL=y
# CONFIG_X86_UP_APIC is not set
CONFIG_X86_TSC=y
# CONFIG_X86_MCE is not set
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
# CONFIG_MICROCODE is not set
# CONFIG_X86_MSR is not set
# CONFIG_X86_CPUID is not set

#
# Firmware Drivers
#
# CONFIG_EDD is not set
CONFIG_NOHIGHMEM=y
# CONFIG_HIGHMEM4G is not set
# CONFIG_HIGHMEM64G is not set
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
CONFIG_HAVE_DEC_LOCK=y

#
# Performance-monitoring counters support
#
# CONFIG_PERFCTR is not set
CONFIG_KERN_PHYS_OFFSET=1
# CONFIG_KEXEC is not set

#
# Power management options (ACPI, APM)
#
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
# CONFIG_SOFTWARE_SUSPEND is not set

#
# ACPI (Advanced Configuration and Power Interface) Support
#
# CONFIG_ACPI is not set
CONFIG_ACPI_BLACKLIST_YEAR=0

#
# APM (Advanced Power Management) BIOS Support
#
CONFIG_APM=y
# CONFIG_APM_IGNORE_USER_SUSPEND is not set
# CONFIG_APM_DO_ENABLE is not set
# CONFIG_APM_CPU_IDLE is not set
# CONFIG_APM_DISPLAY_BLANK is not set
# CONFIG_APM_RTC_IS_GMT is not set
# CONFIG_APM_ALLOW_INTS is not set
CONFIG_APM_REAL_MODE_POWER_OFF=y

#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set

#
# Bus options (PCI, PCMCIA, EISA, MCA, ISA)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
# CONFIG_PCI_LEGACY_PROC is not set
CONFIG_PCI_NAMES=y
# CONFIG_ISA is not set
# CONFIG_MCA is not set
# CONFIG_SCx200 is not set

#
# Executable file formats
#
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_AOUT=y
CONFIG_BINFMT_MISC=y

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
# CONFIG_DEBUG_DRIVER is not set

#
# Memory Technology Devices (MTD)
#
# CONFIG_MTD is not set

#
# Parallel port support
#
# CONFIG_PARPORT is not set

#
# Plug and Play support
#

#
# Block devices
#
CONFIG_BLK_DEV_FD=m
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_NBD=m
# CONFIG_BLK_DEV_SX8 is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_SIZE=4096
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
# CONFIG_LBD is not set
# CONFIG_CDROM_PKTCDVD is not set

#
# ATA/ATAPI/MFM/RLL support
#
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y

#
# Please see Documentation/ide.txt for help/info on IDE drives
#
# CONFIG_BLK_DEV_IDE_SATA is not set
# CONFIG_BLK_DEV_HD_IDE is not set
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
CONFIG_BLK_DEV_IDECD=y
# CONFIG_BLK_DEV_IDETAPE is not set
# CONFIG_BLK_DEV_IDEFLOPPY is not set
CONFIG_BLK_DEV_IDESCSI=m
# CONFIG_IDE_TASK_IOCTL is not set
# CONFIG_IDE_TASKFILE_IO is not set

#
# IDE chipset support/bugfixes
#
CONFIG_IDE_GENERIC=y
# CONFIG_BLK_DEV_CMD640 is not set
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
# CONFIG_BLK_DEV_OFFBOARD is not set
CONFIG_BLK_DEV_GENERIC=y
# CONFIG_BLK_DEV_OPTI621 is not set
# CONFIG_BLK_DEV_RZ1000 is not set
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_IDEDMA_FORCED is not set
CONFIG_IDEDMA_PCI_AUTO=y
# CONFIG_IDEDMA_ONLYDISK is not set
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
# CONFIG_BLK_DEV_AMD74XX is not set
# CONFIG_BLK_DEV_ATIIXP is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_TRIFLEX is not set
# CONFIG_BLK_DEV_CY82C693 is not set
# CONFIG_BLK_DEV_CS5520 is not set
# CONFIG_BLK_DEV_CS5530 is not set
# CONFIG_BLK_DEV_HPT34X is not set
# CONFIG_BLK_DEV_HPT366 is not set
# CONFIG_BLK_DEV_SC1200 is not set
# CONFIG_BLK_DEV_PIIX is not set
# CONFIG_BLK_DEV_NS87415 is not set
# CONFIG_BLK_DEV_PDC202XX_OLD is not set
# CONFIG_BLK_DEV_PDC202XX_NEW is not set
# CONFIG_BLK_DEV_SVWKS is not set
# CONFIG_BLK_DEV_SIIMAGE is not set
CONFIG_BLK_DEV_SIS5513=y
# CONFIG_BLK_DEV_SLC90E66 is not set
# CONFIG_BLK_DEV_TRM290 is not set
# CONFIG_BLK_DEV_VIA82CXXX is not set
# CONFIG_IDE_ARM is not set
CONFIG_BLK_DEV_IDEDMA=y
# CONFIG_IDEDMA_IVB is not set
CONFIG_IDEDMA_AUTO=y
# CONFIG_BLK_DEV_HD is not set

#
# SCSI device support
#
CONFIG_SCSI=m
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=m
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=m
# CONFIG_BLK_DEV_SR_VENDOR is not set
CONFIG_CHR_DEV_SG=m

#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
# CONFIG_SCSI_MULTI_LUN is not set
CONFIG_SCSI_CONSTANTS=y
# CONFIG_SCSI_LOGGING is not set

#
# SCSI Transport Attributes
#
# CONFIG_SCSI_SPI_ATTRS is not set
# CONFIG_SCSI_FC_ATTRS is not set

#
# SCSI low-level drivers
#
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AACRAID is not set
CONFIG_SCSI_AIC7XXX=m
CONFIG_AIC7XXX_CMDS_PER_DEVICE=32
CONFIG_AIC7XXX_RESET_DELAY_MS=15000
# CONFIG_AIC7XXX_DEBUG_ENABLE is not set
CONFIG_AIC7XXX_DEBUG_MASK=0
# CONFIG_AIC7XXX_REG_PRETTY_PRINT is not set
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_SCSI_SATA is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_EATA_PIO is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_QLOGIC_ISP is not set
# CONFIG_SCSI_QLOGIC_FC is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
# CONFIG_SCSI_QLOGIC_1280_1040 is not set
CONFIG_SCSI_QLA2XXX=m
# CONFIG_SCSI_QLA21XX is not set
# CONFIG_SCSI_QLA22XX is not set
# CONFIG_SCSI_QLA2300 is not set
# CONFIG_SCSI_QLA2322 is not set
# CONFIG_SCSI_QLA6312 is not set
# CONFIG_SCSI_QLA6322 is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_NSP32 is not set
# CONFIG_SCSI_DEBUG is not set

#
# Multi-device support (RAID and LVM)
#
# CONFIG_MD is not set

#
# Fusion MPT device support
#
# CONFIG_FUSION is not set

#
# IEEE 1394 (FireWire) support
#
# CONFIG_IEEE1394 is not set

#
# I2O device support
#
# CONFIG_I2O is not set

#
# Networking support
#
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
# CONFIG_PACKET_MMAP is not set
CONFIG_NETLINK_DEV=y
CONFIG_UNIX=y
# CONFIG_NET_KEY is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
# CONFIG_IP_MULTIPLE_TABLES is not set
# CONFIG_IP_ROUTE_MULTIPATH is not set
# CONFIG_IP_ROUTE_VERBOSE is not set
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set
# CONFIG_IP_MROUTE is not set
# CONFIG_ARPD is not set
# CONFIG_SYN_COOKIES is not set
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_TUNNEL is not set

#
# IP: Virtual Server Configuration
#
# CONFIG_IP_VS is not set
# CONFIG_IPV6 is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set

#
# IP: Netfilter Configuration
#
CONFIG_IP_NF_CONNTRACK=m
# CONFIG_IP_NF_CT_ACCT is not set
# CONFIG_IP_NF_CT_PROTO_SCTP is not set
# CONFIG_IP_NF_FTP is not set
# CONFIG_IP_NF_IRC is not set
# CONFIG_IP_NF_TFTP is not set
# CONFIG_IP_NF_AMANDA is not set
# CONFIG_IP_NF_QUEUE is not set
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_LIMIT=m
CONFIG_IP_NF_MATCH_IPRANGE=m
CONFIG_IP_NF_MATCH_MAC=m
CONFIG_IP_NF_MATCH_PKTTYPE=m
CONFIG_IP_NF_MATCH_MARK=m
CONFIG_IP_NF_MATCH_MULTIPORT=m
CONFIG_IP_NF_MATCH_TOS=m
CONFIG_IP_NF_MATCH_RECENT=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_DSCP=m
CONFIG_IP_NF_MATCH_AH_ESP=m
CONFIG_IP_NF_MATCH_LENGTH=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_MATCH_TCPMSS=m
CONFIG_IP_NF_MATCH_HELPER=m
CONFIG_IP_NF_MATCH_STATE=m
CONFIG_IP_NF_MATCH_CONNTRACK=m
CONFIG_IP_NF_MATCH_OWNER=m
CONFIG_IP_NF_MATCH_ADDRTYPE=m
CONFIG_IP_NF_MATCH_REALM=m
# CONFIG_IP_NF_MATCH_SCTP is not set
# CONFIG_IP_NF_MATCH_COMMENT is not set
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_IP_NF_TARGET_TCPMSS=m
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_SAME=m
# CONFIG_IP_NF_NAT_LOCAL is not set
# CONFIG_IP_NF_NAT_SNMP_BASIC is not set
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_TOS=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_DSCP=m
CONFIG_IP_NF_TARGET_MARK=m
CONFIG_IP_NF_TARGET_CLASSIFY=m
# CONFIG_IP_NF_RAW is not set
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m
# CONFIG_IP_NF_COMPAT_IPCHAINS is not set
# CONFIG_IP_NF_COMPAT_IPFWADM is not set

#
# SCTP Configuration (EXPERIMENTAL)
#
# CONFIG_IP_SCTP is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_NET_DIVERT is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_NET_HW_FLOWCONTROL is not set

#
# QoS and/or fair queueing
#
CONFIG_NET_SCHED=y
CONFIG_NET_SCH_CLK_JIFFIES=y
# CONFIG_NET_SCH_CLK_GETTIMEOFDAY is not set
# CONFIG_NET_SCH_CLK_CPU is not set
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
# CONFIG_NET_SCH_HFSC is not set
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
# CONFIG_NET_SCH_NETEM is not set
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_QOS=y
CONFIG_NET_ESTIMATOR=y
CONFIG_NET_CLS=y
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
# CONFIG_CLS_U32_PERF is not set
# CONFIG_NET_CLS_IND is not set
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
# CONFIG_NET_CLS_ACT is not set
CONFIG_NET_CLS_POLICE=y

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_KGDBOE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NETPOLL_RX is not set
# CONFIG_NETPOLL_TRAP is not set
# CONFIG_NET_POLL_CONTROLLER is not set
# CONFIG_HAMRADIO is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
CONFIG_NETDEVICES=y
CONFIG_DUMMY=m
# CONFIG_BONDING is not set
# CONFIG_EQUALIZER is not set
# CONFIG_TUN is not set
# CONFIG_ETHERTAP is not set

#
# ARCnet devices
#
# CONFIG_ARCNET is not set

#
# Ethernet (10 or 100Mbit)
#
CONFIG_NET_ETHERNET=y
CONFIG_MII=m
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
# CONFIG_NET_VENDOR_3COM is not set

#
# Tulip family network device support
#
# CONFIG_NET_TULIP is not set
# CONFIG_HP100 is not set
CONFIG_NET_PCI=y
# CONFIG_PCNET32 is not set
# CONFIG_AMD8111_ETH is not set
# CONFIG_ADAPTEC_STARFIRE is not set
# CONFIG_B44 is not set
# CONFIG_FORCEDETH is not set
# CONFIG_DGRS is not set
# CONFIG_EEPRO100 is not set
# CONFIG_E100 is not set
# CONFIG_FEALNX is not set
# CONFIG_NATSEMI is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_8139CP is not set
# CONFIG_8139TOO is not set
CONFIG_SIS900=m
# CONFIG_EPIC100 is not set
# CONFIG_SUNDANCE is not set
# CONFIG_TLAN is not set
# CONFIG_VIA_RHINE is not set

#
# Ethernet (1000 Mbit)
#
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SK98LIN is not set
# CONFIG_VIA_VELOCITY is not set
# CONFIG_TIGON3 is not set

#
# Ethernet (10000 Mbit)
#
# CONFIG_IXGB is not set
# CONFIG_S2IO is not set

#
# Token Ring devices
#
# CONFIG_TR is not set

#
# Wireless LAN (non-hamradio)
#
# CONFIG_NET_RADIO is not set

#
# Wan interfaces
#
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
CONFIG_PPP=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPPOE=m
# CONFIG_SLIP is not set
# CONFIG_NET_FC is not set
# CONFIG_SHAPER is not set
# CONFIG_NETCONSOLE is not set

#
# ISDN subsystem
#
# CONFIG_ISDN is not set

#
# Telephony Support
#
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_JOYDEV=m
# CONFIG_INPUT_TSDEV is not set
# CONFIG_INPUT_EVDEV is not set
# CONFIG_INPUT_EVBUG is not set

#
# Input I/O drivers
#
CONFIG_GAMEPORT=m
CONFIG_SOUND_GAMEPORT=m
# CONFIG_GAMEPORT_NS558 is not set
# CONFIG_GAMEPORT_L4 is not set
# CONFIG_GAMEPORT_EMU10K1 is not set
# CONFIG_GAMEPORT_VORTEX is not set
# CONFIG_GAMEPORT_FM801 is not set
# CONFIG_GAMEPORT_CS461x is not set
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
# CONFIG_SERIO_SERPORT is not set
# CONFIG_SERIO_CT82C710 is not set
CONFIG_SERIO_PCIPS2=m
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
CONFIG_INPUT_PCSPKR=y
# CONFIG_INPUT_UINPUT is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_SERIAL_NONSTANDARD is not set

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
# CONFIG_SERIAL_8250_CONSOLE is not set
CONFIG_SERIAL_8250_NR_UARTS=4
# CONFIG_SERIAL_8250_EXTENDED is not set

#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256

#
# IPMI
#
# CONFIG_IPMI_HANDLER is not set

#
# Watchdog Cards
#
# CONFIG_WATCHDOG is not set
# CONFIG_HW_RANDOM is not set
CONFIG_NVRAM=y
CONFIG_RTC=y
# CONFIG_DTLK is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_SONYPI is not set

#
# Ftape, the floppy tape device driver
#
# CONFIG_FTAPE is not set
CONFIG_AGP=m
# CONFIG_AGP_ALI is not set
# CONFIG_AGP_ATI is not set
# CONFIG_AGP_AMD is not set
# CONFIG_AGP_AMD64 is not set
# CONFIG_AGP_INTEL is not set
# CONFIG_AGP_INTEL_MCH is not set
# CONFIG_AGP_NVIDIA is not set
CONFIG_AGP_SIS=m
# CONFIG_AGP_SWORKS is not set
# CONFIG_AGP_VIA is not set
# CONFIG_AGP_EFFICEON is not set
# CONFIG_DRM is not set
# CONFIG_MWAVE is not set
# CONFIG_RAW_DRIVER is not set
CONFIG_HANGCHECK_TIMER=m

#
# I2C support
#
CONFIG_I2C=m
CONFIG_I2C_CHARDEV=m

#
# I2C Algorithms
#
CONFIG_I2C_ALGOBIT=m
CONFIG_I2C_ALGOPCF=m
# CONFIG_I2C_ALGOPCA is not set

#
# I2C Hardware Bus support
#
CONFIG_I2C_ALI1535=m
CONFIG_I2C_ALI1563=m
CONFIG_I2C_ALI15X3=m
CONFIG_I2C_AMD756=m
CONFIG_I2C_AMD8111=m
CONFIG_I2C_I801=m
CONFIG_I2C_I810=m
CONFIG_I2C_ISA=m
CONFIG_I2C_NFORCE2=m
CONFIG_I2C_PARPORT_LIGHT=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_PROSAVAGE=m
CONFIG_I2C_SAVAGE4=m
CONFIG_SCx200_ACB=m
CONFIG_I2C_SIS5595=m
CONFIG_I2C_SIS630=m
CONFIG_I2C_SIS96X=m
# CONFIG_I2C_STUB is not set
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m
CONFIG_I2C_VOODOO3=m
# CONFIG_I2C_PCA_ISA is not set

#
# Hardware Sensors Chip support
#
CONFIG_I2C_SENSOR=m
CONFIG_SENSORS_ADM1021=m
CONFIG_SENSORS_ADM1025=m
CONFIG_SENSORS_ADM1031=m
CONFIG_SENSORS_ASB100=m
CONFIG_SENSORS_DS1621=m
CONFIG_SENSORS_FSCHER=m
CONFIG_SENSORS_GL518SM=m
CONFIG_SENSORS_IT87=m
CONFIG_SENSORS_LM75=m
CONFIG_SENSORS_LM77=m
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=m
CONFIG_SENSORS_LM85=m
# CONFIG_SENSORS_LM87 is not set
CONFIG_SENSORS_LM90=m
CONFIG_SENSORS_MAX1619=m
# CONFIG_SENSORS_SMSC47M1 is not set
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83L785TS=m
CONFIG_SENSORS_W83627HF=m

#
# Other I2C Chip support
#
CONFIG_SENSORS_EEPROM=m
CONFIG_SENSORS_PCF8574=m
CONFIG_SENSORS_PCF8591=m
CONFIG_SENSORS_RTC8564=m
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set

#
# Dallas's 1-wire bus
#
# CONFIG_W1 is not set

#
# Misc devices
#
# CONFIG_IBM_ASM is not set

#
# Multimedia devices
#
# CONFIG_VIDEO_DEV is not set

#
# Digital Video Broadcasting Devices
#
# CONFIG_DVB is not set

#
# Graphics support
#
# CONFIG_FB is not set
CONFIG_VIDEO_SELECT=y

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y

#
# Sound
#
CONFIG_SOUND=m

#
# Advanced Linux Sound Architecture
#
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_RTCTIMER=m
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set

#
# Generic devices
#
CONFIG_SND_DUMMY=m
CONFIG_SND_VIRMIDI=m
# CONFIG_SND_MTPAV is not set
# CONFIG_SND_SERIAL_U16550 is not set
# CONFIG_SND_MPU401 is not set

#
# PCI devices
#
CONFIG_SND_AC97_CODEC=m
# CONFIG_SND_ALI5451 is not set
# CONFIG_SND_ATIIXP is not set
# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AZT3328 is not set
# CONFIG_SND_BT87X is not set
CONFIG_SND_CS46XX=m
CONFIG_SND_CS46XX_NEW_DSP=y
# CONFIG_SND_CS4281 is not set
# CONFIG_SND_EMU10K1 is not set
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_HDSP is not set
# CONFIG_SND_TRIDENT is not set
# CONFIG_SND_YMFPCI is not set
# CONFIG_SND_ALS4000 is not set
# CONFIG_SND_CMIPCI is not set
# CONFIG_SND_ENS1370 is not set
# CONFIG_SND_ENS1371 is not set
# CONFIG_SND_ES1938 is not set
# CONFIG_SND_ES1968 is not set
# CONFIG_SND_MAESTRO3 is not set
# CONFIG_SND_FM801 is not set
# CONFIG_SND_ICE1712 is not set
# CONFIG_SND_ICE1724 is not set
CONFIG_SND_INTEL8X0=m
# CONFIG_SND_INTEL8X0M is not set
# CONFIG_SND_SONICVIBES is not set
# CONFIG_SND_VIA82XX is not set
# CONFIG_SND_VX222 is not set

#
# Open Sound System
#
# CONFIG_SOUND_PRIME is not set

#
# USB support
#
# CONFIG_USB is not set
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y

#
# USB Gadget Support
#
# CONFIG_USB_GADGET is not set

#
# File systems
#
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
# CONFIG_EXT2_FS_POSIX_ACL is not set
# CONFIG_EXT2_FS_SECURITY is not set
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
# CONFIG_EXT3_FS_POSIX_ACL is not set
# CONFIG_EXT3_FS_SECURITY is not set
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_FS_MBCACHE=y
# CONFIG_REISER4_FS is not set
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
# CONFIG_XFS_FS is not set
# CONFIG_MINIX_FS is not set
CONFIG_ROMFS_FS=y
# CONFIG_QUOTA is not set
# CONFIG_AUTOFS_FS is not set
CONFIG_AUTOFS4_FS=y

#
# Caches
#
# CONFIG_FSCACHE is not set

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_ZISOFS_FS=y
CONFIG_UDF_FS=y
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_NTFS_FS=m
# CONFIG_NTFS_DEBUG is not set
# CONFIG_NTFS_RW is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_SYSFS=y
# CONFIG_DEVFS_FS is not set
CONFIG_DEVPTS_FS_XATTR=y
# CONFIG_DEVPTS_FS_SECURITY is not set
CONFIG_TMPFS=y
# CONFIG_TMPFS_XATTR is not set
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_RAMFS=y

#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
CONFIG_CRAMFS=m
# CONFIG_VXFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set

#
# Network File Systems
#
# CONFIG_NFS_FS is not set
# CONFIG_NFSD is not set
# CONFIG_EXPORTFS is not set
# CONFIG_SMB_FS is not set
# CONFIG_CIFS is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
CONFIG_AFS_FS=m
CONFIG_RXRPC=m

#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y

#
# Native Language Support
#
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
CONFIG_NLS_CODEPAGE_850=y
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
CONFIG_NLS_CODEPAGE_1250=y
# CONFIG_NLS_CODEPAGE_1251 is not set
# CONFIG_NLS_ASCII is not set
CONFIG_NLS_ISO8859_1=y
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
CONFIG_NLS_ISO8859_15=y
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
# CONFIG_NLS_UTF8 is not set

#
# Profiling support
#
# CONFIG_PROFILING is not set

#
# Kernel hacking
#
CONFIG_DEBUG_KERNEL=y
CONFIG_MAGIC_SYSRQ=y
# CONFIG_SCHEDSTATS is not set
# CONFIG_DEBUG_SLAB is not set
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_PREEMPT=y
CONFIG_PREEMPT_TIMING=y
CONFIG_PREEMPT_TRACE=y
CONFIG_LATENCY_TRACE=y
CONFIG_MCOUNT=y
# CONFIG_RWSEM_DEADLOCK_DETECT is not set
# CONFIG_DEBUG_INFO is not set
CONFIG_FRAME_POINTER=y
CONFIG_EARLY_PRINTK=y
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_KPROBES is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_4KSTACKS is not set
# CONFIG_KGDB is not set

#
# Security options
#
# CONFIG_KEYS is not set
CONFIG_SECURITY=y
# CONFIG_SECURITY_NETWORK is not set
CONFIG_SECURITY_CAPABILITIES=m
# CONFIG_SECURITY_SECLVL is not set
# CONFIG_SECURITY_SELINUX is not set

#
# Cryptographic options
#
CONFIG_CRYPTO=y
# CONFIG_CRYPTO_HMAC is not set
# CONFIG_CRYPTO_NULL is not set
# CONFIG_CRYPTO_MD4 is not set
# CONFIG_CRYPTO_MD5 is not set
# CONFIG_CRYPTO_SHA1 is not set
# CONFIG_CRYPTO_SHA256 is not set
# CONFIG_CRYPTO_SHA512 is not set
# CONFIG_CRYPTO_WP512 is not set
# CONFIG_CRYPTO_DES is not set
# CONFIG_CRYPTO_BLOWFISH is not set
# CONFIG_CRYPTO_TWOFISH is not set
# CONFIG_CRYPTO_SERPENT is not set
# CONFIG_CRYPTO_AES_586 is not set
# CONFIG_CRYPTO_CAST5 is not set
# CONFIG_CRYPTO_CAST6 is not set
# CONFIG_CRYPTO_TEA is not set
# CONFIG_CRYPTO_ARC4 is not set
# CONFIG_CRYPTO_KHAZAD is not set
# CONFIG_CRYPTO_DEFLATE is not set
# CONFIG_CRYPTO_MICHAEL_MIC is not set
# CONFIG_CRYPTO_CRC32C is not set
# CONFIG_CRYPTO_TEST is not set

#
# Library routines
#
CONFIG_CRC_CCITT=m
CONFIG_CRC32=m
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_PC=y

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

On Fri, 29 Oct 2004 23:25:45 +0200
Ingo Molnar <mingo@elte.hu> wrote:

> > will do so. btw: i think i'm a bit confused right now. What debugging
> > features should i have enabled for this test?
> 
> this particular one (atomicity-checking) is always-enabled if you have
> the -RT patch applied (it's a really cheap check).
> 
> for the 'application-triggered tracing' facility we talked about earlier
> is only active if LATENCY_TRACING is enabled. In that case to turn the 
> tracer on, call:
[snip]

Ok,

.config attached,

running the patched jackd -R -P 60 -d alsa -p64 for a bit and provoking
xruns via window wiggling :)

mango:~# cat /proc/sys/kernel/trace_enabled 
2
mango:~# cat /proc/sys/kernel/preempt_max_latency 
75
mango:~# cat /proc/sys/kernel/preempt_thresh      
0

mango:~# cat /proc/latency_trace 
preemption latency trace v1.0.7 on 2.6.9-mm1-RT-V0.5.14-rt
-------------------------------------------------------
 latency: 75 us, entries: 54 (54)   |   [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
    -----------------
    | task: jackd/1083, uid:1000 nice:0 policy:1 rt_prio:60
    -----------------
 => started at: try_to_wake_up+0x5a/0x110 <c01151ca>
 => ended at:   finish_task_switch+0x31/0xc0 <c0115691>
=======>
  834 80000000 0.000ms (+0.000ms): (39) ((115))
  834 80000000 0.000ms (+0.000ms): (1083) ((834))
  834 80000000 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up)
  834 80000000 0.000ms (+0.000ms): preempt_schedule (__up_write)
  834 00000000 0.000ms (+0.000ms): preempt_schedule (up_write_mutex)
  834 80000000 0.000ms (+0.000ms): __schedule (preempt_schedule)
  834 80000000 0.000ms (+0.000ms): profile_hit (__schedule)
  834 80000000 0.000ms (+0.000ms): sched_clock (__schedule)
 1083 80000000 0.001ms (+0.000ms): __switch_to (__schedule)
 1083 80000000 0.001ms (+0.000ms): (834) ((1083))
 1083 80000000 0.001ms (+0.000ms): (115) ((39))
 1083 80000000 0.002ms (+0.000ms): finish_task_switch (__schedule)
 1083 80000000 0.002ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
 1083 80000000 0.002ms (+0.000ms): (1083) ((39))
 1083 00000000 0.003ms (+0.000ms): _mutex_unlock (kfree)
 1083 00000000 0.003ms (+0.000ms): up_mutex (kfree)
 1083 00000000 0.003ms (+0.000ms): up_write_mutex (kfree)
 1083 00000000 0.003ms (+0.000ms): __up_write (up_write_mutex)
 1083 00000000 0.003ms (+0.000ms): poll_freewait (sys_poll)
 1083 00000000 0.003ms (+0.000ms): remove_wait_queue (poll_freewait)
 1083 00000000 0.003ms (+0.000ms): _mutex_lock_irqsave (remove_wait_queue)
 1083 00000000 0.003ms (+0.000ms): __mutex_lock (_mutex_lock_irqsave)
 1083 00000000 0.003ms (+0.000ms): __might_sleep (__mutex_lock)
 1083 00000000 0.004ms (+0.000ms): down_mutex (__mutex_lock)
 1083 00000000 0.004ms (+0.000ms): down_write_mutex (__mutex_lock)
 1083 00000000 0.004ms (+0.000ms): __might_sleep (down_write_mutex)
 1083 00000000 0.004ms (+0.000ms): _mutex_unlock_irqrestore (remove_wait_queue)
 1083 00000000 0.004ms (+0.000ms): up_mutex (remove_wait_queue)
 1083 00000000 0.004ms (+0.000ms): up_write_mutex (remove_wait_queue)
 1083 00000000 0.004ms (+0.000ms): __up_write (up_write_mutex)
 1083 00000000 0.004ms (+0.000ms): fput (poll_freewait)
 1083 00000000 0.005ms (+0.000ms): free_pages (poll_freewait)
 1083 00000000 0.005ms (+0.000ms): __free_pages (poll_freewait)
 1083 00000000 0.005ms (+0.000ms): free_hot_page (poll_freewait)
 1083 00000000 0.006ms (+0.000ms): __free_pages_ok (free_hot_page)
 1083 00000000 0.006ms (+0.000ms): free_pages_bulk (__free_pages_ok)
 1083 00000000 0.006ms (+0.000ms): _mutex_lock_irqsave (free_pages_bulk)
 1083 00000000 0.006ms (+0.000ms): __mutex_lock (_mutex_lock_irqsave)
 1083 00000000 0.007ms (+0.000ms): __might_sleep (__mutex_lock)
 1083 00000000 0.007ms (+0.000ms): down_mutex (__mutex_lock)
 1083 00000000 0.007ms (+0.000ms): down_write_mutex (__mutex_lock)
 1083 00000000 0.007ms (+0.000ms): __might_sleep (down_write_mutex)
 1083 00000000 0.007ms (+0.000ms): bad_range (free_pages_bulk)
 1083 00000000 0.007ms (+0.000ms): bad_range (free_pages_bulk)
 1083 00000000 0.008ms (+0.000ms): _mutex_unlock_irqrestore (free_pages_bulk)
 1083 00000000 0.008ms (+0.000ms): up_mutex (free_pages_bulk)
 1083 00000000 0.008ms (+0.000ms): up_write_mutex (free_pages_bulk)
 1083 00000000 0.008ms (+0.004ms): __up_write (up_write_mutex)
 1083 00000000 0.013ms (+0.000ms): sys_gettimeofday (syscall_call)
 1083 00000000 0.013ms (+0.000ms): user_trace_stop (sys_gettimeofday)
 1083 80000000 0.013ms (+0.000ms): user_trace_stop (sys_gettimeofday)
 1083 80000000 0.013ms (+0.000ms): update_max_trace (user_trace_stop)
 1083 80000000 0.014ms (+0.000ms): _mmx_memcpy (update_max_trace)
 1083 80000000 0.014ms (+0.000ms): kernel_fpu_begin (_mmx_memcpy)



mango:~# chrt -p `pidof "IRQ 3"`
pid 118's current scheduling policy: SCHED_FIFO
pid 118's current scheduling priority: 99

mango:~# ps aux|grep jackd
tapas     1080  4.2  3.6 28244 28220 pts/1   SLl+ 00:22   0:18 jackd -R -P 60 -d alsa -p 64
mango:~# chrt -p 1080
pid 1080's current scheduling policy: SCHED_OTHER
pid 1080's current scheduling priority: 0
mango:~# chrt -p 1081
pid 1081's current scheduling policy: SCHED_OTHER
pid 1081's current scheduling priority: 0
mango:~# chrt -p 1082
pid 1082's current scheduling policy: SCHED_FIFO
pid 1082's current scheduling priority: 70
mango:~# chrt -p 1083
pid 1083's current scheduling policy: SCHED_FIFO
pid 1083's current scheduling priority: 60


and here's some syslog stuff:

Oct 30 00:20:32 mango kernel: Realtime LSM initialized (group 1002, mlock=1)
Oct 30 00:20:35 mango kernel: IRQ#3 thread RT prio: 43.
Oct 30 00:20:38 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:20:38 mango kernel:  [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:20:38 mango kernel:  [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:20:38 mango kernel:  [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:20:38 mango kernel: ---------------------------
Oct 30 00:20:38 mango kernel: | preempt count: 00000001 ]
Oct 30 00:20:38 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:20:38 mango kernel: ----------------------------------------
Oct 30 00:20:38 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:20:38 mango kernel: .....[show_registers+99/464] ..   ( <= dump_stack+0x23/0x30)
Oct 30 00:20:38 mango kernel: 
Oct 30 00:20:47 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:20:47 mango kernel:  [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:20:47 mango kernel:  [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:20:47 mango kernel:  [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:20:47 mango kernel: ---------------------------
Oct 30 00:20:47 mango kernel: | preempt count: 00000001 ]
Oct 30 00:20:47 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:20:47 mango kernel: ----------------------------------------
Oct 30 00:20:47 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:20:47 mango kernel: .....[show_registers+99/464] ..   ( <= dump_stack+0x23/0x30)
Oct 30 00:20:47 mango kernel: 
Oct 30 00:20:51 mango kernel: (IRQ 3/118/CPU#0): new 4 us maximum-latency wakeup.
Oct 30 00:20:51 mango kernel: (bash/867/CPU#0): new 6 us maximum-latency wakeup.
Oct 30 00:20:52 mango kernel: (bash/1035/CPU#0): new 7 us maximum-latency wakeup.
Oct 30 00:20:52 mango kernel: (ksoftirqd/0/2/CPU#0): new 8 us maximum-latency wakeup.
Oct 30 00:20:52 mango kernel: (ksoftirqd/0/2/CPU#0): new 13 us maximum-latency wakeup.
Oct 30 00:21:11 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:11 mango kernel: (IRQ 12/15/CPU#0): new 19 us maximum-latency wakeup.
Oct 30 00:21:11 mango kernel:  [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:11 mango kernel:  [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:11 mango kernel:  [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:11 mango kernel: ---------------------------
Oct 30 00:21:11 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:11 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:11 mango kernel: ----------------------------------------
Oct 30 00:21:11 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:11 mango kernel: .....[show_registers+99/464] ..   ( <= dump_stack+0x23/0x30)
Oct 30 00:21:11 mango kernel: 
Oct 30 00:21:11 mango kernel: (IRQ 12/15/CPU#0): new 75 us maximum-latency wakeup.
Oct 30 00:21:11 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:11 mango kernel:  [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:11 mango kernel:  [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:11 mango kernel:  [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:11 mango kernel: ---------------------------
Oct 30 00:21:11 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:11 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:11 mango kernel: ----------------------------------------
Oct 30 00:21:11 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:11 mango kernel: .....[show_registers+99/464] ..   ( <= dump_stack+0x23/0x30)
Oct 30 00:21:11 mango kernel: 
Oct 30 00:21:11 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:11 mango kernel:  [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:11 mango kernel:  [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:11 mango kernel:  [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:11 mango kernel: ---------------------------
Oct 30 00:21:11 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:11 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:11 mango kernel: ----------------------------------------
Oct 30 00:21:11 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:11 mango kernel: .....[show_registers+99/464] ..   ( <= dump_stack+0x23/0x30)
Oct 30 00:21:11 mango kernel: 
Oct 30 00:21:11 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:11 mango kernel:  [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:11 mango kernel:  [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:11 mango kernel:  [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:11 mango kernel: ---------------------------
Oct 30 00:21:11 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:11 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:11 mango kernel: ----------------------------------------
Oct 30 00:21:11 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:11 mango kernel: .....[show_registers+99/464] ..   ( <= dump_stack+0x23/0x30)
Oct 30 00:21:11 mango kernel: 
Oct 30 00:21:11 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:11 mango kernel:  [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:11 mango kernel:  [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:11 mango kernel:  [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:11 mango kernel: ---------------------------
Oct 30 00:21:11 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:11 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:11 mango kernel: ----------------------------------------
Oct 30 00:21:11 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:11 mango kernel: .....[show_registers+99/464] ..   ( <= dump_stack+0x23/0x30)
Oct 30 00:21:11 mango kernel: 
Oct 30 00:21:11 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:11 mango kernel:  [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:11 mango kernel:  [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:11 mango kernel:  [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:11 mango kernel: ---------------------------
Oct 30 00:21:11 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:11 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:11 mango kernel: ----------------------------------------
Oct 30 00:21:11 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:11 mango kernel: .....[show_registers+99/464] ..   ( <= dump_stack+0x23/0x30)
Oct 30 00:21:11 mango kernel: 
Oct 30 00:21:12 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:12 mango kernel:  [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:12 mango kernel:  [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:12 mango kernel:  [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:12 mango kernel: ---------------------------
Oct 30 00:21:12 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:12 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:12 mango kernel: ----------------------------------------
Oct 30 00:21:12 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:12 mango kernel: .....[show_registers+99/464] ..   ( <= dump_stack+0x23/0x30)
Oct 30 00:21:12 mango kernel: 
Oct 30 00:21:13 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:13 mango kernel:  [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:13 mango kernel:  [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:13 mango kernel:  [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:13 mango kernel: ---------------------------
Oct 30 00:21:13 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:13 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:13 mango kernel: ----------------------------------------
Oct 30 00:21:13 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:13 mango kernel: .....[show_registers+99/464] ..   ( <= dump_stack+0x23/0x30)
Oct 30 00:21:13 mango kernel: 
Oct 30 00:21:31 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:31 mango kernel:  [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:31 mango kernel:  [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:31 mango kernel:  [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:31 mango kernel: ---------------------------
Oct 30 00:21:31 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:31 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:31 mango kernel: ----------------------------------------
Oct 30 00:21:31 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:31 mango kernel: .....[show_registers+99/464] ..   ( <= dump_stack+0x23/0x30)
Oct 30 00:21:31 mango kernel: 
Oct 30 00:21:32 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:32 mango kernel:  [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:32 mango kernel:  [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:32 mango kernel:  [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:32 mango kernel: ---------------------------
Oct 30 00:21:32 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:32 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:32 mango kernel: ----------------------------------------
Oct 30 00:21:32 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:32 mango kernel: .....[show_registers+99/464] ..   ( <= dump_stack+0x23/0x30)
Oct 30 00:21:32 mango kernel: 
Oct 30 00:21:32 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:32 mango kernel:  [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:32 mango kernel:  [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:32 mango kernel:  [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:32 mango kernel: ---------------------------
Oct 30 00:21:32 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:32 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:32 mango kernel: ----------------------------------------
Oct 30 00:21:32 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:32 mango kernel: .....[show_registers+99/464] ..   ( <= dump_stack+0x23/0x30)
Oct 30 00:21:32 mango kernel: 
Oct 30 00:21:32 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:32 mango kernel:  [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:32 mango kernel:  [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:32 mango kernel:  [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:32 mango kernel: ---------------------------
Oct 30 00:21:32 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:32 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:32 mango kernel: ----------------------------------------
Oct 30 00:21:32 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:32 mango kernel: .....[show_registers+99/464] ..   ( <= dump_stack+0x23/0x30)
Oct 30 00:21:32 mango kernel: 
Oct 30 00:21:34 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:34 mango kernel:  [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:34 mango kernel:  [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:34 mango kernel:  [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:34 mango kernel: ---------------------------
Oct 30 00:21:34 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:34 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:34 mango kernel: ----------------------------------------
Oct 30 00:21:34 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:34 mango kernel: .....[show_registers+99/464] ..   ( <= dump_stack+0x23/0x30)
Oct 30 00:21:34 mango kernel: 
Oct 30 00:21:35 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:35 mango kernel:  [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:35 mango kernel:  [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:35 mango kernel:  [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:35 mango kernel: ---------------------------
Oct 30 00:21:35 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:35 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:35 mango kernel: ----------------------------------------
Oct 30 00:21:35 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:35 mango kernel: .....[show_registers+99/464] ..   ( <= dump_stack+0x23/0x30)
Oct 30 00:21:35 mango kernel: 
Oct 30 00:21:37 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:37 mango kernel:  [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:37 mango kernel:  [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:37 mango kernel:  [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:37 mango kernel: ---------------------------
Oct 30 00:21:37 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:37 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:37 mango kernel: ----------------------------------------
Oct 30 00:21:37 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:37 mango kernel: .....[show_registers+99/464] ..   ( <= dump_stack+0x23/0x30)
Oct 30 00:21:37 mango kernel: 
Oct 30 00:22:09 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:22:09 mango kernel:  [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:22:09 mango kernel:  [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:22:09 mango kernel:  [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:22:09 mango kernel: ---------------------------
Oct 30 00:22:09 mango kernel: | preempt count: 00000001 ]
Oct 30 00:22:09 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:22:09 mango kernel: ----------------------------------------
Oct 30 00:22:09 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:22:09 mango kernel: .....[show_registers+99/464] ..   ( <= dump_stack+0x23/0x30)
Oct 30 00:22:09 mango kernel: 
Oct 30 00:22:10 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:22:10 mango kernel:  [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:22:10 mango kernel:  [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:22:10 mango kernel:  [kernel_config_data+5091/6569] io_schedule+0x13/0x20 (8)
Oct 30 00:22:10 mango kernel:  [do_hardirq+104/304] sync_page+0x38/0x50 (12)
Oct 30 00:22:10 mango kernel:  [kernel_config_data+5981/6569] __wait_on_bit_lock+0x5d/0x70 (32)
Oct 30 00:22:10 mango kernel:  [init_irq_proc+137/176] __lock_page+0x89/0xa0 (92)
Oct 30 00:22:10 mango kernel:  [__generic_file_aio_read+402/544] filemap_nopage+0x2b2/0x3e0 (60)
Oct 30 00:22:10 mango kernel:  [do_no_page+74/848] do_no_page+0xba/0x390 (68)
Oct 30 00:22:10 mango kernel:  [handle_mm_fault+207/416] handle_mm_fault+0xef/0x180 (48)
Oct 30 00:22:10 mango kernel:  [get_user_pages+169/912] get_user_pages+0x139/0x380 (60)
Oct 30 00:22:10 mango kernel:  [make_pages_present+108/192] make_pages_present+0x9c/0x110 (56)
Oct 30 00:22:10 mango kernel:  [do_mmap_pgoff+699/1808] do_mmap_pgoff+0x47b/0x700 (88)
Oct 30 00:22:10 mango kernel:  [old_mmap+149/304] old_mmap+0xc5/0xf0 (96)
Oct 30 00:22:10 mango kernel:  [irq_entries_start+3/128] syscall_call+0x7/0xb (-8124)
Oct 30 00:22:10 mango kernel: ---------------------------
Oct 30 00:22:10 mango kernel: | preempt count: 00000001 ]
Oct 30 00:22:10 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:22:10 mango kernel: ----------------------------------------
Oct 30 00:22:10 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:22:10 mango kernel: .....[show_registers+99/464] ..   ( <= dump_stack+0x23/0x30)
Oct 30 00:22:10 mango kernel: 
Oct 30 00:24:40 mango kernel: jackd:1083 userspace BUG: scheduling in user-atomic context!
Oct 30 00:24:40 mango kernel:  [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:24:40 mango kernel:  [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:24:40 mango kernel:  [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:24:40 mango kernel: ---------------------------
Oct 30 00:24:40 mango kernel: | preempt count: 00000001 ]
Oct 30 00:24:40 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:24:40 mango kernel: ----------------------------------------
Oct 30 00:24:40 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:24:40 mango kernel: .....[show_registers+99/464] ..   ( <= dump_stack+0x23/0x30)
Oct 30 00:24:40 mango kernel: 
Oct 30 00:28:29 mango kernel: (IRQ 3/118/CPU#0): new 205 us maximum-latency wakeup.

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Fri, 2004-10-29 at 23:46 +0200, Ingo Molnar wrote:
> correct. softirqs are not used by the sound subsystem so there's no
> ksoftirqd dependency.
> 

I only recommended this because it did seem to make a difference in a
previous version, probably due to debug overhead or something.  It does
not seem like it should.

Lee

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> On Fri, 2004-10-29 at 23:25 +0200, Ingo Molnar wrote:
> > > will do so. btw: i think i'm a bit confused right now. What debugging
> > > features should i have enabled for this test?
> > 
> > this particular one (atomicity-checking) is always-enabled if you have
> > the -RT patch applied (it's a really cheap check).
> 
> One more question, what do you recommend the priorities of the IRQ
> threads be set to?  AIUI for xrun-free operation with JACK, all that
> is needed is to set the RT priorities of the soundcard IRQ thread
> highest, followed by the JACK threads, then the other IRQ threads.  Is
> this correct?

correct. softirqs are not used by the sound subsystem so there's no
ksoftirqd dependency.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Fri, 2004-10-29 at 23:25 +0200, Ingo Molnar wrote:
> > will do so. btw: i think i'm a bit confused right now. What debugging
> > features should i have enabled for this test?
> 
> this particular one (atomicity-checking) is always-enabled if you have
> the -RT patch applied (it's a really cheap check).

One more question, what do you recommend the priorities of the IRQ
threads be set to?  AIUI for xrun-free operation with JACK, all that is
needed is to set the RT priorities of the soundcard IRQ thread highest,
followed by the JACK threads, then the other IRQ threads.  Is this
correct?

Lee  

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

On Fri, 29 Oct 2004 22:42:20 +0200
Ingo Molnar <mingo@elte.hu> wrote:

> > compiles and boots fine. no observable change in xrun behaviour
> > though. 
> 
> do you compile jackd from sources? If yes then could you try the patch
> below? With this added, the kernel will produce a stackdump whenever
> jackd does an 'illegal' sleep.
> 
> Also, could you do a small modification to kernel/sched.c and remove
> this line:
> 
> 		send_sig(SIGUSR1, current, 1);
> 
> just to make it easier to get Jack up and running. (by default an
> atomicity violation triggers a signal to make it easier to debug it in
> userspace, but i suspect there will be alot of such violations so jackd
> would stop all the time.)

[snip]

will do so. btw: i think i'm a bit confused right now. What debugging
features should i have enabled for this test?

flo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

> > do you compile jackd from sources? If yes then could you try the patch
> > below? With this added, the kernel will produce a stackdump whenever
> > jackd does an 'illegal' sleep.
> > 
> > Also, could you do a small modification to kernel/sched.c and remove
> > this line:
> > 
> > 		send_sig(SIGUSR1, current, 1);
> > 
> > just to make it easier to get Jack up and running. (by default an
> > atomicity violation triggers a signal to make it easier to debug it in
> > userspace, but i suspect there will be alot of such violations so jackd
> > would stop all the time.)
> 
> [snip]
> 
> will do so. btw: i think i'm a bit confused right now. What debugging
> features should i have enabled for this test?

this particular one (atomicity-checking) is always-enabled if you have
the -RT patch applied (it's a really cheap check).

for the 'application-triggered tracing' facility we talked about earlier
is only active if LATENCY_TRACING is enabled. In that case to turn the 
tracer on, call:

	gettimeofday(0,1);

and to turn the tracer off and save the current trace into 
/proc/latency_trace, call:

	gettimeofday(0,0);

or apply the patch below - i've added the tracer bits too. I've added a
simple limit: all delays above 2 msec will be saved - you might want to
do a maximum search there or something. And dont forget to:

	echo 2 > /proc/sys/kernel/trace_enabled

to activate the jackd-triggered kernel tracer.

	Ingo

--- jack-audio-connection-kit-0.99.0/drivers/alsa/alsa_driver.c.orig
+++ jack-audio-connection-kit-0.99.0/drivers/alsa/alsa_driver.c
@@ -1161,6 +1161,7 @@ alsa_driver_wait (alsa_driver_t *driver,
 		unsigned int p_timed_out, c_timed_out;
 		unsigned int ci = 0;
 		unsigned int nfds;
+		int ret;
 
 		nfds = 0;
 
@@ -1194,7 +1195,20 @@ alsa_driver_wait (alsa_driver_t *driver,
 
 		poll_enter = jack_get_microseconds ();
 
-		if (poll (driver->pfd, nfds, driver->poll_timeout) < 0) {
+		gettimeofday((void *)1,(void *)0); // atomic off
+
+		gettimeofday((void *)0,(void *)1); // start tracing
+
+		ret = poll (driver->pfd, nfds, driver->poll_timeout);
+
+		poll_ret = jack_get_microseconds ();
+
+		if (poll_ret - poll_enter > 2000)
+			gettimeofday((void *)0,(void *)0); // save trace
+
+		gettimeofday((void *)1,(void *)1); // atomic on
+
+		if (ret < 0) {
 
 			if (errno == EINTR) {
 				printf ("poll interrupt\n");
@@ -1214,8 +1228,6 @@ alsa_driver_wait (alsa_driver_t *driver,
 			
 		}
 
-		poll_ret = jack_get_microseconds ();
-
 		if (extra_fd < 0) {
 			if (driver->poll_next && poll_ret > driver->poll_next) {
 				*delayed_usecs = poll_ret - driver->poll_next;

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Paul Davis <paul@linuxaudiosystems.com> wrote:

>   [ I trimmed the CC: line because several people there are on
>     jackit-devel. ]
> 
> >> compiles and boots fine. no observable change in xrun behaviour though. 
> >
> >ok, so there's something else going on as well - or i missed an ioctl. 
> 
> i really don't think the ioctl's are relevant. 
> 
> consider what will happen if jackd does make a system call that causes
> a major delay (say, because of the BKL). we will get an xrun, yes, but
> this will cause jackd to stop the audio interface and restart.
> max_delay is not affected by this behaviour.

indeed. I'd exclude the ioctls at this point. But:

> as far as i can tell, the number reported by max_delay entirely (or
> almost entirely) represents problems in kernel scheduling, specifically
> with a combination of:
> 
>      a) handling the audio interface interrupt in time.
>      b) marking the relevant jackd thread runnable
>      c) context switching back to the relevant jackd thread
> 
> things that jackd does once its running do not, it appear to me, have
> any impact on max_delay, which is based on the simple observation: 
> 
>    "i was just woken, i expect to be awakened again in N usecs or
>    less.

i dont yet see how this conclusion follows. Here's the poll() code
(simplified):

                poll_enter = jack_get_microseconds ();

                ret = poll (driver->pfd, nfds, driver->poll_timeout);

		[...]

		if (extra_fd < 0) {
			if (driver->poll_next && poll_ret > driver->poll_next) {
				*delayed_usecs = poll_ret - driver->poll_next;
			} 
			driver->poll_last = poll_ret;
			driver->poll_next = poll_ret + driver->period_usecs;
			driver->engine->transport_cycle_start (driver->engine, 
							       poll_ret);
		}

is there a mechanism that ensures that the next poll() will be called
_before_ ->poll_next? Do you get a real hard ALSA xrun in that case or
something similar?

if it's possible to 'silently' overrun the next due interrupt (somewhat,
but not large enough overrun to cause a hard ALSA xrun) then the
processing delay will i believe be accounted as a 'wakeup delay'. In
that case to make the delayed_usecs value truly accurate, i'd at least
add this:

                poll_enter = jack_get_microseconds ();

		if (poll_enter > driver->poll_next) {
			/*
			 * This processing cycle got delayed over
			 * the next due interrupt! Do not account this
			 * as a wakeup delay:
			 */
			driver->poll_next = 0;
		}

but i'd also suggest to put in a counter into that branch so that this
condition doesnt get lost. In fact the Maximum Process Cycle stat from
Rui:

>>   Maximum Delay . . . . . . . . .    6904       921       721    usecs
>>   Maximum Process Cycle . . . . .    1449      1469      1590    usecs

seems to suggest that there can be significant processing delays? (if
Maximum Process Cycle is indeed the time spent from poll_ret to the next
poll_enter.)

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Paul Davis]

  [ I trimmed the CC: line because several people there are on
    jackit-devel. ]

>> compiles and boots fine. no observable change in xrun behaviour though. 
>
>ok, so there's something else going on as well - or i missed an ioctl. 

i really don't think the ioctl's are relevant. 

consider what will happen if jackd does make a system call that causes
a major delay (say, because of the BKL). we will get an xrun, yes, but
this will cause jackd to stop the audio interface and
restart. max_delay is not affected by this behaviour.
 
as far as i can tell, the number reported by max_delay entirely (or
almost entirely) represents problems in kernel scheduling, specifically
with a combination of:

     a) handling the audio interface interrupt in time.
     b) marking the relevant jackd thread runnable
     c) context switching back to the relevant jackd thread

things that jackd does once its running do not, it appear to me, have
any impact on max_delay, which is based on the simple observation: 

   "i was just woken, i expect to be awakened again in N usecs or
   less.

--p

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

> > > fs.h chunk went missing ... uploading -V0.5.14 in a minute.
> > 
> > done.
> 
> compiles and boots fine. no observable change in xrun behaviour
> though. 

do you compile jackd from sources? If yes then could you try the patch
below? With this added, the kernel will produce a stackdump whenever
jackd does an 'illegal' sleep.

Also, could you do a small modification to kernel/sched.c and remove
this line:

		send_sig(SIGUSR1, current, 1);

just to make it easier to get Jack up and running. (by default an
atomicity violation triggers a signal to make it easier to debug it in
userspace, but i suspect there will be alot of such violations so jackd
would stop all the time.)

	Ingo

--- jack-audio-connection-kit-0.99.0/drivers/alsa/alsa_driver.c.orig
+++ jack-audio-connection-kit-0.99.0/drivers/alsa/alsa_driver.c
@@ -1161,6 +1161,7 @@ alsa_driver_wait (alsa_driver_t *driver,
 		unsigned int p_timed_out, c_timed_out;
 		unsigned int ci = 0;
 		unsigned int nfds;
+		int ret;
 
 		nfds = 0;
 
@@ -1194,7 +1195,11 @@ alsa_driver_wait (alsa_driver_t *driver,
 
 		poll_enter = jack_get_microseconds ();
 
-		if (poll (driver->pfd, nfds, driver->poll_timeout) < 0) {
+		gettimeofday((void *)1,(void *)0); // atomic off
+		ret = poll (driver->pfd, nfds, driver->poll_timeout);
+		gettimeofday((void *)1,(void *)1); // atomic on
+
+		if (ret < 0) {
 
 			if (errno == EINTR) {
 				printf ("poll interrupt\n");

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

> > > > fs/ioctl.c: In function `sys_ioctl':
> > > > fs/ioctl.c:75: error: structure has no member named `ioctl_nobkl'
> > > > fs/ioctl.c:76: error: structure has no member named `ioctl_nobkl'
> > > 
> > > fs.h chunk went missing ... uploading -V0.5.14 in a minute.
> > 
> > done.
> 
> compiles and boots fine. no observable change in xrun behaviour though. 

ok, so there's something else going on as well - or i missed an ioctl. 
Even if it's not the BKL, the ioctl/BKL problem would come back later
and haunt us during stresstests anyway so it's not bad that we have a
solution for it.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Thomas Gleixner]

On Fri, 2004-10-29 at 15:33 -0400, Lee Revell wrote:
> On Fri, 2004-10-29 at 10:02 +0200, Ingo Molnar wrote:
> >    I know that Jackd does alot of precautions 
> >    to avoid unintentional scheduling (mlockall, the use of SCHED_FIFO), 
> >    but are you absolutely sure it doesnt happen? This scenario could be 
> >    excluded by measuring the time Jackd calls poll(), and comparing it
> >    to the expected value. [Or is this value already included in the 
> >    stats Rui collected? Maybe the "Maximum Process Cycle" value?]
> 
> Yes, this is already accounted for in the 'Maximum Process Cycle' value.
> This measures the time between returning from poll() and entering it
> again.  I will try to add some instrumentation to jackd and test this
> weekend.  I do agree that it could be a jackd bug; this would not be the
> first time the VP patches exposed bugs in other apps.
> 

Can you check out, whether the memory is getting low when you are doing
this tests ? The VM code has a serious problem, which might be related
to those latencies.

tglx

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Fri, 2004-10-29 at 10:02 +0200, Ingo Molnar wrote:
>    I know that Jackd does alot of precautions 
>    to avoid unintentional scheduling (mlockall, the use of SCHED_FIFO), 
>    but are you absolutely sure it doesnt happen? This scenario could be 
>    excluded by measuring the time Jackd calls poll(), and comparing it
>    to the expected value. [Or is this value already included in the 
>    stats Rui collected? Maybe the "Maximum Process Cycle" value?]

Yes, this is already accounted for in the 'Maximum Process Cycle' value.
This measures the time between returning from poll() and entering it
again.  I will try to add some instrumentation to jackd and test this
weekend.  I do agree that it could be a jackd bug; this would not be the
first time the VP patches exposed bugs in other apps.

Lee

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

On Fri, 29 Oct 2004 19:22:43 +0200
Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > 
> > * Florian Schmidt <mista.tapas@gmx.net> wrote:
> > 
> > >   CC      fs/ioctl.o
> > > fs/ioctl.c: In function `sys_ioctl':
> > > fs/ioctl.c:75: error: structure has no member named `ioctl_nobkl'
> > > fs/ioctl.c:76: error: structure has no member named `ioctl_nobkl'
> > 
> > fs.h chunk went missing ... uploading -V0.5.14 in a minute.
> 
> done.

compiles and boots fine. no observable change in xrun behaviour though. 

flo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

On Fri, 29 Oct 2004 19:09:48 +0200
Ingo Molnar <mingo@elte.hu> wrote:

> ok, i've uploaded -V0.5.12 which has this BKL trick. I hope i got the
> ALSA ioctls right: the ones that go outside the BKL for now are the
> pcm_native playback/capture ones, and rawmidi's ioctl. All the others
> are still BKL users. To recap, it's these ioctls that matter:

doesn't build here:

  CC      fs/exec.o
  CC      fs/pipe.o
  CC      fs/namei.o
  CC      fs/fcntl.o
  CC      fs/ioctl.o
fs/ioctl.c: In function `sys_ioctl':
fs/ioctl.c:75: error: structure has no member named `ioctl_nobkl'
fs/ioctl.c:76: error: structure has no member named `ioctl_nobkl'
make[1]: *** [fs/ioctl.o] Error 1
make: *** [fs] Error 2

flo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Florian Schmidt <mista.tapas@gmx.net> wrote:
> 
> >   CC      fs/ioctl.o
> > fs/ioctl.c: In function `sys_ioctl':
> > fs/ioctl.c:75: error: structure has no member named `ioctl_nobkl'
> > fs/ioctl.c:76: error: structure has no member named `ioctl_nobkl'
> 
> fs.h chunk went missing ... uploading -V0.5.14 in a minute.

done.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

>   CC      fs/ioctl.o
> fs/ioctl.c: In function `sys_ioctl':
> fs/ioctl.c:75: error: structure has no member named `ioctl_nobkl'
> fs/ioctl.c:76: error: structure has no member named `ioctl_nobkl'

fs.h chunk went missing ... uploading -V0.5.14 in a minute.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

On Fri, 29 Oct 2004 18:31:55 +0200
Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > > > the BKL can generate arbitrary latencies. Anything up to 100-200
> > > > milliseconds. Rui, Florian, could you try the quick hack below?
> > > 
> > > sure, with a fully REALTIME_PREEMPTION kernel?
> > 
> > correct, and with the changeall-tree hack in addition. And keep your
> > finger near the reset button, just in case ...
> 
> it wont even boot ...
> 
> let me try some more hacks to make this a little bit safer.

hehe, it even booted for me [kinda]. will build the one where you got xmms
to run. but i will sure as hell hit "get new emails" during the build more
than once ;)

flo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

>  5971  ioctl(7, 0x4143, 0x446b7d3c)      = 0
>  5971  ioctl(7, 0x4140, 0x446b7d3c)      = 0
>  5971  ioctl(7, 0x4142, 0x446b7d3c)      = 0
> 
> which ones are these? Look at the patch for how to change a .ioctl one
> to .ioctl_nobkl (lame solution ...). So if your setup uses any other
> ioctl (sndctl perhaps?) then you should change that one to nobkl too.

note that even if just one of these ioctls is still a BKL user then
-V0.5.12 will show no significant difference. So i've uploaded -V0.5.13
with some more converted. I think this should cover the above ones, but
i'm not 100% sure.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> > > let me try some more hacks to make this a little bit safer.
> > 
> > hehe, it even booted for me [kinda]. will build the one where you got
> > xmms to run. but i will sure as hell hit "get new emails" during the
> > build more than once ;)
> 
> indeed - i'm preparing 5.0.12 with a better way to do this. (the trick
> is to allow the BKL to 'underflow' - this way ALSA can be kept largely
> unmodified.)

ok, i've uploaded -V0.5.12 which has this BKL trick. I hope i got the
ALSA ioctls right: the ones that go outside the BKL for now are the
pcm_native playback/capture ones, and rawmidi's ioctl. All the others
are still BKL users. To recap, it's these ioctls that matter:

 5971  ioctl(7, 0x4143, 0x446b7d3c)      = 0
 5971  ioctl(7, 0x4140, 0x446b7d3c)      = 0
 5971  ioctl(7, 0x4142, 0x446b7d3c)      = 0

which ones are these? Look at the patch for how to change a .ioctl one
to .ioctl_nobkl (lame solution ...). So if your setup uses any other
ioctl (sndctl perhaps?) then you should change that one to nobkl too.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

> > let me try some more hacks to make this a little bit safer.
> 
> hehe, it even booted for me [kinda]. will build the one where you got
> xmms to run. but i will sure as hell hit "get new emails" during the
> build more than once ;)

indeed - i'm preparing 5.0.12 with a better way to do this. (the trick
is to allow the BKL to 'underflow' - this way ALSA can be kept largely
unmodified.)

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> ok, next attempt. Please apply this patch and do the changeall-tree
> thing in the sound/ directory. This should in theory switch the ALSA
> ioctls (and only them) to a BKL-less variant.

ok, new version - the previous one was slighly racy. The one below works
better - it boots on my box and simple audio playback (xmms) works.

	Ingo

--- linux/sound/core/pcm_native.c.orig2	
+++ linux/sound/core/pcm_native.c	
@@ -3317,7 +3317,7 @@ static struct file_operations snd_pcm_f_
 	.open =		snd_pcm_open,
 	.release =	snd_pcm_release,
 	.poll =		snd_pcm_playback_poll,
-	.ioctl =	snd_pcm_playback_ioctl,
+	.ioctl2 =	snd_pcm_playback_ioctl,
 	.mmap =		snd_pcm_mmap,
 	.fasync =	snd_pcm_fasync,
 };
@@ -3329,7 +3329,7 @@ static struct file_operations snd_pcm_f_
 	.open =		snd_pcm_open,
 	.release =	snd_pcm_release,
 	.poll =		snd_pcm_capture_poll,
-	.ioctl =	snd_pcm_capture_ioctl,
+	.ioctl2 =	snd_pcm_capture_ioctl,
 	.mmap =		snd_pcm_mmap,
 	.fasync =	snd_pcm_fasync,
 };
--- linux/sound/pci/ali5451/ali5451.c.orig2	
+++ linux/sound/pci/ali5451/ali5451.c	
@@ -33,6 +33,7 @@
 #include <linux/pci.h>
 #include <linux/slab.h>
 #include <linux/moduleparam.h>
+#include <linux/syscalls.h>
 #include <sound/core.h>
 #include <sound/pcm.h>
 #include <sound/info.h>
@@ -985,11 +986,11 @@ static void snd_ali_update_ptr(ali_t *co
 	pvoice = &codec->synth.voices[channel];
 	runtime = pvoice->substream->runtime;
 
-	udelay(100);
 	spin_lock(&codec->reg_lock);
 
 	if (pvoice->pcm && pvoice->substream) {
 		/* pcm interrupt */
+		sys_gettimeofday((void *)0, (void *)1); // start the tracer
 #ifdef ALI_DEBUG
 		outb((u8)(pvoice->number), ALI_REG(codec, ALI_GC_CIR));
 		temp = inw(ALI_REG(codec, ALI_CSO_ALPHA_FMS + 2));
--- linux/fs/ioctl.c.orig2	
+++ linux/fs/ioctl.c	
@@ -68,6 +68,13 @@ asmlinkage long sys_ioctl(unsigned int f
                 goto out;
         }
 
+	error = -ENOTTY;
+	if (filp->f_op && filp->f_op->ioctl2) {
+		error = filp->f_op->ioctl2(filp->f_dentry->d_inode, filp, cmd, arg);
+		fput(filp);
+		return error;
+	}
+
 	lock_kernel();
 	switch (cmd) {
 		case FIOCLEX:
@@ -132,7 +139,6 @@ asmlinkage long sys_ioctl(unsigned int f
 	}
 	unlock_kernel();
 	fput(filp);
-
 out:
 	return error;
 }
--- linux/include/linux/fs.h.orig2	
+++ linux/include/linux/fs.h	
@@ -993,6 +993,7 @@ struct file_operations {
 	int (*check_flags)(int);
 	int (*dir_notify)(struct file *filp, unsigned long arg);
 	int (*flock) (struct file *, int, struct file_lock *);
+	int (*ioctl2) (struct inode *, struct file *, unsigned int, unsigned long);
 };
 
 struct inode_operations {

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> > correct, and with the changeall-tree hack in addition. And keep your
> > finger near the reset button, just in case ...
> 
> it wont even boot ...
> 
> let me try some more hacks to make this a little bit safer.

ok, next attempt. Please apply this patch and do the changeall-tree
thing in the sound/ directory. This should in theory switch the ALSA
ioctls (and only them) to a BKL-less variant.

	Ingo

--- linux/sound/core/pcm_native.c.orig
+++ linux/sound/core/pcm_native.c	
@@ -3317,7 +3317,7 @@ static struct file_operations snd_pcm_f_
 	.open =		snd_pcm_open,
 	.release =	snd_pcm_release,
 	.poll =		snd_pcm_playback_poll,
-	.ioctl =	snd_pcm_playback_ioctl,
+	.ioctl2 =	snd_pcm_playback_ioctl,
 	.mmap =		snd_pcm_mmap,
 	.fasync =	snd_pcm_fasync,
 };
@@ -3329,7 +3329,7 @@ static struct file_operations snd_pcm_f_
 	.open =		snd_pcm_open,
 	.release =	snd_pcm_release,
 	.poll =		snd_pcm_capture_poll,
-	.ioctl =	snd_pcm_capture_ioctl,
+	.ioctl2 =	snd_pcm_capture_ioctl,
 	.mmap =		snd_pcm_mmap,
 	.fasync =	snd_pcm_fasync,
 };
--- linux/fs/ioctl.c.orig
+++ linux/fs/ioctl.c	
@@ -68,19 +68,26 @@ asmlinkage long sys_ioctl(unsigned int f
                 goto out;
         }
 
-	lock_kernel();
 	switch (cmd) {
 		case FIOCLEX:
+			lock_kernel();
 			set_close_on_exec(fd, 1);
+			unlock_kernel();
 			break;
 
 		case FIONCLEX:
+			lock_kernel();
 			set_close_on_exec(fd, 0);
+			unlock_kernel();
 			break;
 
 		case FIONBIO:
-			if ((error = get_user(on, (int __user *)arg)) != 0)
+			lock_kernel();
+			unlock_kernel();
+			if ((error = get_user(on, (int __user *)arg)) != 0) {
+				unlock_kernel();
 				break;
+			}
 			flag = O_NONBLOCK;
 #ifdef __sparc__
 			/* SunOS compatibility item. */
@@ -91,11 +98,15 @@ asmlinkage long sys_ioctl(unsigned int f
 				filp->f_flags |= flag;
 			else
 				filp->f_flags &= ~flag;
+			unlock_kernel();
 			break;
 
 		case FIOASYNC:
-			if ((error = get_user(on, (int __user *)arg)) != 0)
+			lock_kernel();
+			if ((error = get_user(on, (int __user *)arg)) != 0) {
+				unlock_kernel();
 				break;
+			}
 			flag = on ? FASYNC : 0;
 
 			/* Did FASYNC state change ? */
@@ -104,16 +115,20 @@ asmlinkage long sys_ioctl(unsigned int f
 					error = filp->f_op->fasync(fd, filp, on);
 				else error = -ENOTTY;
 			}
-			if (error != 0)
+			if (error != 0) {
+				unlock_kernel();
 				break;
+			}
 
 			if (on)
 				filp->f_flags |= FASYNC;
 			else
 				filp->f_flags &= ~FASYNC;
+			unlock_kernel();
 			break;
 
 		case FIOQSIZE:
+			lock_kernel();
 			if (S_ISDIR(filp->f_dentry->d_inode->i_mode) ||
 			    S_ISREG(filp->f_dentry->d_inode->i_mode) ||
 			    S_ISLNK(filp->f_dentry->d_inode->i_mode)) {
@@ -122,15 +137,21 @@ asmlinkage long sys_ioctl(unsigned int f
 			}
 			else
 				error = -ENOTTY;
+			unlock_kernel();
 			break;
 		default:
 			error = -ENOTTY;
-			if (S_ISREG(filp->f_dentry->d_inode->i_mode))
+			if (S_ISREG(filp->f_dentry->d_inode->i_mode)) {
+				lock_kernel();
 				error = file_ioctl(filp, cmd, arg);
-			else if (filp->f_op && filp->f_op->ioctl)
+				unlock_kernel();
+			} else if (filp->f_op && filp->f_op->ioctl) {
+				lock_kernel();
 				error = filp->f_op->ioctl(filp->f_dentry->d_inode, filp, cmd, arg);
+				unlock_kernel();
+			} else if (filp->f_op && filp->f_op->ioctl2)
+				error = filp->f_op->ioctl2(filp->f_dentry->d_inode, filp, cmd, arg);
 	}
-	unlock_kernel();
 	fput(filp);
 
 out:
--- linux/include/linux/fs.h.orig
+++ linux/include/linux/fs.h
@@ -993,6 +993,7 @@ struct file_operations {
 	int (*check_flags)(int);
 	int (*dir_notify)(struct file *filp, unsigned long arg);
 	int (*flock) (struct file *, int, struct file_lock *);
+	int (*ioctl2) (struct inode *, struct file *, unsigned int, unsigned long);
 };
 
 struct inode_operations {

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Florian Schmidt]

On Fri, 29 Oct 2004 18:14:33 +0200
Ingo Molnar <mingo@elte.hu> wrote:

> the BKL can generate arbitrary latencies. Anything up to 100-200
> milliseconds. Rui, Florian, could you try the quick hack below?

sure, with a fully REALTIME_PREEMPTION kernel?

flo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> > > the BKL can generate arbitrary latencies. Anything up to 100-200
> > > milliseconds. Rui, Florian, could you try the quick hack below?
> > 
> > sure, with a fully REALTIME_PREEMPTION kernel?
> 
> correct, and with the changeall-tree hack in addition. And keep your
> finger near the reset button, just in case ...

it wont even boot ...

let me try some more hacks to make this a little bit safer.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

> > the BKL can generate arbitrary latencies. Anything up to 100-200
> > milliseconds. Rui, Florian, could you try the quick hack below?
> 
> sure, with a fully REALTIME_PREEMPTION kernel?

correct, and with the changeall-tree hack in addition. And keep your
finger near the reset button, just in case ...

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

[-- Attachment #1: Type: text/plain, Size: 628 bytes --]


* Ingo Molnar <mingo@elte.hu> wrote:

> the BKL can generate arbitrary latencies. Anything up to 100-200
> milliseconds. Rui, Florian, could you try the quick hack below?

hm, this alone probably wont work, because the audio layer does
unlock_kernel() in many places.

could you try another hack in addition to the ioctl.c hack, save the
attached script, and do this in your Linux source tree:

	cd sound
	~/changeall-tree "unlock_kernel()" ""
	~/changeall-tree "lock_kernel()" ""

(in strictly this order.) This gets rid of the BKL from the sound
subsystem in quite drastic ways... Totally untested, so be careful ...

	Ingo


[-- Attachment #2: changeall-tree --]
[-- Type: text/plain, Size: 214 bytes --]

#!/bin/bash

if [ $# -lt "2" ]; then
        echo 'usage: changeall <from> <to>'
	exit -1;
fi

echo "changing $N => $N"

for N in `find . -type f`; do
  echo $N;
  cat $N | sed "s/$1/$2/g" > .tmp; mv .tmp $N;
done

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> ahh ... the ioctls. All ioctls in Linux take the 'big kernel lock'. So
> if jackd calls an ioctl() in the fastpath then that could easily get
> delayed. Also, there are a couple of other syscalls too that touch the
> BKL, so an strace of the 'jackd latency path' would be quite useful.

i straced jackd and it seems to do ioctls:

5971  ioctl(7, 0x4143, 0x446b7d3c)      = 0
5971  ioctl(7, 0x4140, 0x446b7d3c)      = 0
5971  ioctl(7, 0x4142, 0x446b7d3c)      = 0
5971  poll([{fd=7, events=POLLOUT|POLLERR|POLLNVAL, revents=POLLOUT}], 1, 4353) = 1
5971  poll([{fd=7, events=POLLOUT|POLLERR|POLLNVAL, revents=POLLOUT}], 1, 4353)= 1
5971  poll([{fd=7, events=POLLOUT|POLLERR|POLLNVAL, revents=POLLOUT|POLLERR}], 1, 4353) = 1
5971  ioctl(7, 0x806c4120, 0xb77bb960)  = 0
5971  gettimeofday({1099064915, 776665}, NULL) = 0

the BKL can generate arbitrary latencies. Anything up to 100-200
milliseconds. Rui, Florian, could you try the quick hack below?

	Ingo

--- linux/fs/ioctl.c.orig2	
+++ linux/fs/ioctl.c	
@@ -68,19 +68,26 @@ asmlinkage long sys_ioctl(unsigned int f
                 goto out;
         }
 
-	lock_kernel();
 	switch (cmd) {
 		case FIOCLEX:
+			lock_kernel();
 			set_close_on_exec(fd, 1);
+			unlock_kernel();
 			break;
 
 		case FIONCLEX:
+			lock_kernel();
 			set_close_on_exec(fd, 0);
+			unlock_kernel();
 			break;
 
 		case FIONBIO:
-			if ((error = get_user(on, (int __user *)arg)) != 0)
+			lock_kernel();
+			unlock_kernel();
+			if ((error = get_user(on, (int __user *)arg)) != 0) {
+				unlock_kernel();
 				break;
+			}
 			flag = O_NONBLOCK;
 #ifdef __sparc__
 			/* SunOS compatibility item. */
@@ -91,11 +98,15 @@ asmlinkage long sys_ioctl(unsigned int f
 				filp->f_flags |= flag;
 			else
 				filp->f_flags &= ~flag;
+			unlock_kernel();
 			break;
 
 		case FIOASYNC:
-			if ((error = get_user(on, (int __user *)arg)) != 0)
+			lock_kernel();
+			if ((error = get_user(on, (int __user *)arg)) != 0) {
+				unlock_kernel();
 				break;
+			}
 			flag = on ? FASYNC : 0;
 
 			/* Did FASYNC state change ? */
@@ -104,16 +115,20 @@ asmlinkage long sys_ioctl(unsigned int f
 					error = filp->f_op->fasync(fd, filp, on);
 				else error = -ENOTTY;
 			}
-			if (error != 0)
+			if (error != 0) {
+				unlock_kernel();
 				break;
+			}
 
 			if (on)
 				filp->f_flags |= FASYNC;
 			else
 				filp->f_flags &= ~FASYNC;
+			unlock_kernel();
 			break;
 
 		case FIOQSIZE:
+			lock_kernel();
 			if (S_ISDIR(filp->f_dentry->d_inode->i_mode) ||
 			    S_ISREG(filp->f_dentry->d_inode->i_mode) ||
 			    S_ISLNK(filp->f_dentry->d_inode->i_mode)) {
@@ -122,6 +137,7 @@ asmlinkage long sys_ioctl(unsigned int f
 			}
 			else
 				error = -ENOTTY;
+			unlock_kernel();
 			break;
 		default:
 			error = -ENOTTY;
@@ -130,7 +146,6 @@ asmlinkage long sys_ioctl(unsigned int f
 			else if (filp->f_op && filp->f_op->ioctl)
 				error = filp->f_op->ioctl(filp->f_dentry->d_inode, filp, cmd, arg);
 	}
-	unlock_kernel();
 	fput(filp);
 
 out:

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Paul Davis]

>here's Rui's setup:
> 
>| Note: all tests were carried out running jackd -v -dalsa -dhw:0 
>| -r44100 -p128 -n2 -S -P, loaded with 9 (nine) fluidsynth instances,
>| on a P4@2.533Ghz laptop, against the onboard sound device 
>| (snd-ali5451).
>
>i suspect this means there was playback only, no capture, and thus the 
>capture/playback interrupt interaction cannot be the case, right?

ok, he used the -P flag which means playback only. so yes, the ALSA
backend would only have opened the playback stream.

oh, and i must correct what i said before. the ALSA backend will
typically be polling on *two* fd's. ALSA represents playback and
capture as different devices, so a default jackd will open both and
poll both of them. the backend will wait until they are both ready
(ie. the capture device has the correct amount of data, the playback
device has the correct amount of space).

--p

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Paul Davis <paul@linuxaudiosystems.com> wrote:

>      (c) the audio interface doesn't interrupt independently for
>                capture and playback. Rui will need to get back
> 	       to me with details on what type of audio interface
> 	       he is using for me to comment on this. If its
> 	       a consumer device with poor support for full
> 	       duplex operation, then it can happen that
> 	       capture and playback streams are running out
> 	       of sync and this can cause some odd timing.

here's Rui's setup:

| Note: all tests were carried out running jackd -v -dalsa -dhw:0 
| -r44100 -p128 -n2 -S -P, loaded with 9 (nine) fluidsynth instances,
| on a P4@2.533Ghz laptop, against the onboard sound device 
| (snd-ali5451).

i suspect this means there was playback only, no capture, and thus the 
capture/playback interrupt interaction cannot be the case, right?

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Paul Davis <paul@linuxaudiosystems.com> wrote:

> for our purposes, the number of channels is irrelevant. the audio
> interface interrupts when it has and/or needs data. whether that data
> covers 2, 9, 26 or 128 channels isn't particularly important. each
> channel is handled at the same time. for most stereo devices, the data
> for the channels is actually interleaved; for some high end
> multichannel devices, each channel uses a separate memory buffer.
> either way, its really not central.

ok, thanks for the explanations.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Paul Davis]

>after reading some more code i believe the concept is called 'stream' in
>ALSA speak. How many streams did Rui's test utilize? (nine?)

it doesn't matter, actually ...

>if multiple streams were used, then can the handling of one stream delay
>another stream? How are streams prioritized, and are they queued in a
>FIFO or in a LIFO manner? (Also, are such 'streams' directly mapped to
>some hardware concept on the audio card ('channel' or 'voice'?) or am i
>confusing things?)

for our purposes, the number of channels is irrelevant. the audio
interface interrupts when it has and/or needs data. whether that data
covers 2, 9, 26 or 128 channels isn't particularly important. each
channel is handled at the same time. for most stereo devices, the data
for the channels is actually interleaved; for some high end
multichannel devices, each channel uses a separate memory
buffer. either way, its really not central.

\begin{digression}

what may confuse you is the kernel-level ALSA concept of a "stream"
and a "substream". this is not relevant to the issue at hand, but i
may as well explain anyway :)

these are not actually channels at all, but abstractions that ALSA
uses to handle many different audio h/w architectures with a
reasonable level of uniformity. many consumer audio devices these days
have multiple independent audio engines on them. for example, some
cards have analog stereo output, analog surround (so-called "5.1")
output and/or digital S/PDIF output. if they are truly independent
(i.e. they carry different data and can be started and stopped
independently), then ALSA generally represents each device
independently. in the driver architecture, each device is represented
by a "stream" when it is open. audio interfaces with just a single
device has only one stream that can be opened.

in addition, there are a number of audio interfaces that can do h/w
mixing of multiple data flowing into the same device. i have a
consumer card, for example, that can handle 32 different stereo
streams and it mixes them down to a single stereo output. these are
not independent in the sense that they cannot be started/stopped
independently, and so ALSA represents each of these as a
"substream". An audio interface that doesn't do h/w mixing has only a
single substream associated with the stream.

When an application opens an audio interface, it is opening a
substream of a stream. By default, it will get the first (typically
only) substream on the first stream (typically only) of the first
audio interface.  

The term "voice" is a driver-specific one that has been inherited from
the way the makes of certain audio interfaces talk about their hardware.
It corresponds very loosely to ALSA's notion of a substream, but not
precisely. 

\end{digression}

to get back to max_delay .... the important thing here is that
max_delay is measuring the delay between when jackd believes it should
*next* be woken and when it actually is. on the assumption (valid for
rui's test AFAIK) that jackd has gone back to sleep a long time before
the next audio interface interrupt wakes it up, then this delay is
troublesome, because it happens entirely in the kernel.

some things we need to rule out: 

     (a) jackd is not asleep when the next interrupt arrives
               (though this should not cause a delay of 700usecs)
     (b) the audio interface delivered the interrupt on time
     (c) the audio interface doesn't interrupt independently for
               capture and playback. Rui will need to get back
	       to me with details on what type of audio interface
	       he is using for me to comment on this. If its
	       a consumer device with poor support for full
	       duplex operation, then it can happen that
	       capture and playback streams are running out
	       of sync and this can cause some odd timing.

--p

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Paul Davis <paul@linuxaudiosystems.com> wrote:

> >> 		if (poll (driver->pfd, nfds, driver->poll_timeout) < 0) {
> >
> >in Rui's test (9 fluidsynth instances running), what would be the value
> >of nfds - 9 or 1? I.e. is the '9 streams' abstraction provided by jackd,
> >and you map it to a single alsa driver fd over which you poll, or is
> >each stream a separate fd?
> 
> for almost all audio interfaces, there is one fd per hardware device.
> so whether its a 26 channel device or a simple stereo device, there is
> one fd. jackd will be polling a single fd in that call under almost
> all circumstances, and certainly those used in rui's testing.

great - this certainly simplifies the scenario.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Paul Davis]

>> 		if (poll (driver->pfd, nfds, driver->poll_timeout) < 0) {
>
>in Rui's test (9 fluidsynth instances running), what would be the value
>of nfds - 9 or 1? I.e. is the '9 streams' abstraction provided by jackd,
>and you map it to a single alsa driver fd over which you poll, or is
>each stream a separate fd?

for almost all audio interfaces, there is one fd per hardware
device. so whether its a 26 channel device or a simple stereo device,
there is one fd. jackd will be polling a single fd in that call under
almost all circumstances, and certainly those used in rui's testing.

--p

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Paul Davis <paul@linuxaudiosystems.com> wrote:

> 		if (poll (driver->pfd, nfds, driver->poll_timeout) < 0) {

in Rui's test (9 fluidsynth instances running), what would be the value
of nfds - 9 or 1? I.e. is the '9 streams' abstraction provided by jackd,
and you map it to a single alsa driver fd over which you poll, or is
each stream a separate fd?

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

> i don't know what you mean by "channel" information in this context.

after reading some more code i believe the concept is called 'stream' in
ALSA speak. How many streams did Rui's test utilize? (nine?)

if multiple streams were used, then can the handling of one stream delay
another stream? How are streams prioritized, and are they queued in a
FIFO or in a LIFO manner? (Also, are such 'streams' directly mapped to
some hardware concept on the audio card ('channel' or 'voice'?) or am i
confusing things?)

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Paul Davis]

(sorry for the slow turnaround. i am also going offline for the next 6
hours, so this is my last reply till later ...)

>> the "max delay" measurement isn't a reflection of the runtime activity
>> of jackd. its simply a measurement of the delay between when jackd
>> expected to be next woken from poll and when it actually was.
>> 
>> as you noted, jackd generally goes back to sleep in poll typically a
>> long time before the next interrupt is expected. hence any delay in
>> the wakeup is between the interrupt handler and the scheduler getting
>> jackd's main thread back on the processor. i think.
>
>this brings up the next question: does the jackd measurement also
>timestamp the time when it calls poll() - hence detecting in-jackd
>processing delays? If yes, which value is this from Rui's stats? If not
>then it might make sense to add it.

i'm not quite sure what you mean. the timestamps used to measure
max_delay look like this:
	  

                ... set up poll fd struct  ....

		poll_enter = jack_get_microseconds ();

		if (poll (driver->pfd, nfds, driver->poll_timeout) < 0) {

			if (errno == EINTR) {
				printf ("poll interrupt\n");
				// this happens mostly when run
				// under gdb, or when exiting due to a signal
				if (under_gdb) {
					goto again;
				}
				*status = -2;
				return 0;
			}
			
			jack_error ("ALSA: poll call failed (%s)",
				    strerror (errno));
			*status = -3;
			return 0;
			
		}

		poll_ret = jack_get_microseconds ();

		if (extra_fd < 0) {
			if (driver->poll_next && poll_ret > driver->poll_next) {
				*delayed_usecs = poll_ret - driver->poll_next;
			} 
			driver->poll_last = poll_ret;
			driver->poll_next = poll_ret + driver->period_usecs;
			driver->engine->transport_cycle_start (driver->engine, 
							       poll_ret);
		}

you can view this here in context:

   http://jackit.sourceforge.net/cgi-bin/lxr/http/source/drivers/alsa/alsa_driver.c
  
(its at line 1195)

the max_delay value is taken from "delayed_usecs" above. each backend
(the one shown is the ALSA driver) sets this every time its "wait"
function is called; jackd collects the figures and these days manages
them as a set of statistics.

as you can see, poll_enter is available, but we don't use it for much
at all (mostly just some information in the very unusual event that
poll times out).

jack_get_microseconds() uses the cycle timer to compute elapsed time,
there is no system call involved.

--p

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Paul Davis <paul@linuxaudiosystems.com> wrote:

> >my main suspicion is that either the main jackd thread itself calls the
> >kernel where the kernel (unexpectedly for jackd) schedules away for
> >whatever reason, or that the chain of wakeup in the audio path somehow
> >gets violated (i.e. a kernel problem). There's one quick thing we could
> 
> the "max delay" measurement isn't a reflection of the runtime activity
> of jackd. its simply a measurement of the delay between when jackd
> expected to be next woken from poll and when it actually was.
> 
> as you noted, jackd generally goes back to sleep in poll typically a
> long time before the next interrupt is expected. hence any delay in
> the wakeup is between the interrupt handler and the scheduler getting
> jackd's main thread back on the processor. i think.

this brings up the next question: does the jackd measurement also
timestamp the time when it calls poll() - hence detecting in-jackd
processing delays? If yes, which value is this from Rui's stats? If not
then it might make sense to add it.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Paul Davis]

>my main suspicion is that either the main jackd thread itself calls the
>kernel where the kernel (unexpectedly for jackd) schedules away for
>whatever reason, or that the chain of wakeup in the audio path somehow
>gets violated (i.e. a kernel problem). There's one quick thing we could

the "max delay" measurement isn't a reflection of the runtime activity
of jackd. its simply a measurement of the delay between when jackd
expected to be next woken from poll and when it actually was.

as you noted, jackd generally goes back to sleep in poll typically a
long time before the next interrupt is expected. hence any delay in
the wakeup is between the interrupt handler and the scheduler getting
jackd's main thread back on the processor. i think.

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Paul Davis <paul@linuxaudiosystems.com> wrote:

> this is something i haven't mentioned. when running in RT mode, jackd
> also runs a higher prio RT thread called the watchdog. it wakes every
> 5 seconds to check that the "main" thread is not stalled (i.e. a
> client that is stuck in a loop) and kills everything if it is.
> however, this thread's loop is incredibly simple (it checks and resets
> a single int variable and goes back to sleep) that causing a 700usec
> delay would itself seem to me to be indicative of a problem. do you
> agree?

agreed, the watchdog thread should have no measurable impact - it should
have all of its work done well under 10 usecs on a fast box, and not
above 20 usecs even on a slow box. As long as it doesnt log to a file it
should have no impact. It's also easy to exclude this from the list of
things, Rui could perhaps chrt this particular jackd thread to below the
main jack thread's priority. (it shouldnt break the watchdog
functionality too much, unless the main jackd thread itself goes into a
loop which clearly isnt the problem currently.)

my main suspicion is that either the main jackd thread itself calls the
kernel where the kernel (unexpectedly for jackd) schedules away for
whatever reason, or that the chain of wakeup in the audio path somehow
gets violated (i.e. a kernel problem). There's one quick thing we could
try: could you send me an 'strace -p' log of a couple of interrupt
cycles of jackd handling an ordinary audio stream? There's so many ways
the kernel can schedule away, perhaps i can spot it by looking at the
syscalls called.

in fact, does the jackd high-prio thread (excluding the watchdog) call
any syscalls normally, besides the poll() and then the read()/write() on
the audio device fd? (perhaps it does an ioctl too?)

ahh ... the ioctls. All ioctls in Linux take the 'big kernel lock'. So
if jackd calls an ioctl() in the fastpath then that could easily get
delayed. Also, there are a couple of other syscalls too that touch the
BKL, so an strace of the 'jackd latency path' would be quite useful.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Paul Davis]

>but now it should be possible to prove that Jackd itself doesnt
>introduce latencies, via the userspace-atomicity mode feature in -V0.5.5
>:-) If that proves that the latencies dont occur in Jackd (and if it's
>not another, even higher prio thread that causes the delay) then we can
>pass the ball back to the kernel.

this is something i haven't mentioned. when running in RT mode, jackd
also runs a higher prio RT thread called the watchdog. it wakes every
5 seconds to check that the "main" thread is not stalled (i.e. a
client that is stuck in a loop) and kills everything if it
is. however, this thread's loop is incredibly simple (it checks and
resets a single int variable and goes back to sleep) that causing a
700usec delay would itself seem to me to be indicative of a
problem. do you agree?

--p

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> i've implemented this feature and have put it into -RT-V0.5.5 which can
> be downloaded from the usual place:
> 
> 	http://redhat.com/~mingo/realtime-preempt/

there was a typo in it, i've uploaded -RT-V0.5.6 with that fixed.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Thomas Gleixner <tglx@linutronix.de> wrote:

> The sound subsystem uses a lot of sleep_on() variants. We know that
> they are racy. Might this be related ?

i doubt it is related - sleep_on() is mostly racy on SMP and Rui is
doing UP tests. Also, for a sleep_on() race to occur you'd have to race
with the wakeup - but in the Jackd case that is highly unlikely because
it sleeps well before the next interrupt is due. So while a sleep_on()
race could in theory make _existing_ latencies worse, it cannot by
itself introduce the first latency.

but now it should be possible to prove that Jackd itself doesnt
introduce latencies, via the userspace-atomicity mode feature in -V0.5.5
:-) If that proves that the latencies dont occur in Jackd (and if it's
not another, even higher prio thread that causes the delay) then we can
pass the ball back to the kernel.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

[-- Attachment #1: Type: text/plain, Size: 1618 bytes --]


* Ingo Molnar <mingo@elte.hu> wrote:

> Could the kernel help some more in debugging this? E.g. if the Jackd
> SCHED_FIFO thread (after it has started up) can never legitimately
> reschedule except via poll(), i could try to put in a syscall hack in
> where in the Jackd code you could call say gettimeofday(0,3) to turn
> on 'do not reschedule' mode, and call gettimeofday(0,4) to turn it
> off. If Jackd reschedules while in 'do not reschedule' mode then the
> scheduler code will detect this and will print out the offending
> user-space EIP and a stacktrace. [and will turn off 'do-not schedule'
> mode.]

i've implemented this feature and have put it into -RT-V0.5.5 which can
be downloaded from the usual place:

	http://redhat.com/~mingo/realtime-preempt/

i've attached a simple testcase showing how to use it. When running the 
testcode on a -V0.5.5 kernel it gives:

 saturn:~> ./rt-atomic
 testing atomic mode functionality.
 ok, kernel supports atomic mode.
 atomic mode is now off - doing sleep(), should succeed:
 turning atomic mode on.
 doing getppid() syscall - should succeed.
 doing sleep() syscall - should abort!
 User defined signal 1

the kernel sends SIGUSR1 when it detects illegal scheduling. (change the
SIGUSR1 in the patch if you want another signal.)

could someone with more Jackd experience than me add this to the Jackd
code and check whether and where it triggers? I'd suggest to do
something like around the poll() call:

	atomic_off();
	poll();
	atomic_on();

and add an atomic_off() to the Jackd shutdown handler (or any codepath
that is legitimately allowed to schedule).

	Ingo

[-- Attachment #2: rt-atomic.c --]
[-- Type: text/plain, Size: 746 bytes --]


#include <stdio.h>
#include <stdlib.h>
#include <sys/time.h>
#include <unistd.h>

#define atomic_on() \
do { \
	if (gettimeofday((void *)1,(void *)1)) { \
		printf("failed: wrong kernel?\n"); \
		abort(); \
	} \
} while (0)

#define atomic_off()	gettimeofday((void *)1,(void *)0)

int main(void)
{
	printf("testing atomic mode functionality.\n");
	atomic_on();
	atomic_off();
	printf("ok, kernel supports atomic mode.\n");

	printf("atomic mode is now off - doing sleep(), should succeed:\n");
	sleep(1);
	printf("turning atomic mode on.\n");
	atomic_on();
	printf("doing getppid() syscall - should succeed.\n");
	getppid();
	printf("doing sleep() syscall - should abort!\n");
	sleep(1);
	printf("huh? got back and no signal?\n");

	return 0;
}

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Thomas Gleixner]

On Fri, 2004-10-29 at 10:02 +0200, Ingo Molnar wrote:
> there are multiple possibilities of how this ~700 usecs delay occured:
> 
>  - the kernel still has a wakeup bug. But this should be detected by the
>    tracer which measures the time between when the task hits the
>    runqueue and the task gets to execute on the CPU. Also, if there is a
>    critical section in the kernel that is 700 usecs long it would be
>    detected by _another_, independent timing/tracing mechanism that
>    measures critical sections. The likelyhood of both the scheduler
>    _and_ two independent kernel-tracers being buggy in the same way is
>    quite significantly low. (not to mention the user-space amlat tool
>    which seems to agree with the kernel instrumentation.)
> 

The sound subsystem uses a lot of sleep_on() variants. We know that they
are racy. Might this be related ?

tglx

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Ingo Molnar]

* Paul Davis <paul@linuxaudiosystems.com> wrote:

> >how is the 'maximum delay' calculated? Could you put in a tracing hook
> >into jackd whenever such a ~720 usecs maximum is hit? I'd _love_ to see 
> >how such a latency path looks like, it seems a bit long.
> 
> "maximum delay" is computed like this:
> 
> 	 a) we know that jackd should be woken at regular, known
> 	    intervals based on the interrupt of the audio interface.
> 
> 	 b) jackd is (often) blocked in poll(2), waiting for the
> 	    audio interface to become readable+writable.
> 
> 	 c) on return from poll, we check the time we are returning
> 	    and compare it to the last wakeup. the difference should
> 	    be almost exactly the interrupt interval. 
> 
> 	 d) any amount larger than this is a delay caused by
> 	    kernel scheduling. we track the largest such delay.

there are multiple possibilities of how this ~700 usecs delay occured:

 - the kernel still has a wakeup bug. But this should be detected by the
   tracer which measures the time between when the task hits the
   runqueue and the task gets to execute on the CPU. Also, if there is a
   critical section in the kernel that is 700 usecs long it would be
   detected by _another_, independent timing/tracing mechanism that
   measures critical sections. The likelyhood of both the scheduler
   _and_ two independent kernel-tracers being buggy in the same way is
   quite significantly low. (not to mention the user-space amlat tool
   which seems to agree with the kernel instrumentation.)

note that currently the maximum delay reported by the tracer is around
20-30 usecs. It is nowhere near the 700 usecs measured here. Here are
some of the other latency possibilities:

 - ALSA latency bug: Jackd didnt get woken up in time. Or driver bug, it
   somehow didnt do the wakeup. I'd say this has pretty low probability
   as well since it would break for alot of other scenarios as well.

 - poll() latency bug: pretty low probability too. It's simple and does
   the wakeup immediately, and would be noticed by many people if this
   were buggy.

 - Jackd related latency: is the thread that waits in poll() _truly_ the 
   highest-prio task in the system? IIRC there is some other Jackd
   thread around too that has SCHED_FIFO priority 50, couldnt that 
   thread come in occasionally and delay the poll()-ing thread? Or, is 
   it 100% sure that the only processing the poll()-ing thread does is 
   waiting for the next interrupt and immediately and atomically
   handling it? If there is any delay (either due to processing
   overhead or due to unintended scheduling) then Jackd wont hit poll() 
   in time and could miss the next interrupt - and there's nothing the
   kernel can do about this. I know that Jackd does alot of precautions 
   to avoid unintentional scheduling (mlockall, the use of SCHED_FIFO), 
   but are you absolutely sure it doesnt happen? This scenario could be 
   excluded by measuring the time Jackd calls poll(), and comparing it
   to the expected value. [Or is this value already included in the 
   stats Rui collected? Maybe the "Maximum Process Cycle" value?]

but at this point i'd say that the likelyhood that the scheduler caused
this delay is pretty low - all of that codepath is instrumented and i
think we should instrument Jackd a bit more before putting the blame
back on the scheduler. So while ~3 months ago, when i started this
project, the blame was clearly on the kernel and on the scheduler, i'd
now say for the first time that this particular phenomenon is not
directly caused by it :-)

Could the kernel help some more in debugging this? E.g. if the Jackd
SCHED_FIFO thread (after it has started up) can never legitimately
reschedule except via poll(), i could try to put in a syscall hack in
where in the Jackd code you could call say gettimeofday(0,3) to turn on
'do not reschedule' mode, and call gettimeofday(0,4) to turn it off. If
Jackd reschedules while in 'do not reschedule' mode then the scheduler
code will detect this and will print out the offending user-space EIP
and a stacktrace. [and will turn off 'do-not schedule' mode.]

(i could even make it a clean scheduler feature and send a SIGBUS (or
any other alert signal) if this occurs, and implement a new syscall to
control it. I guess this could be useful to most RT applications. This
signal would only be sent if the RT task got scheduled voluntarily (i.e.
not delayed due to some even higher-prio task).)

> we block in poll(2), waiting for the interrupt from the audio
> interface to mark the relevant jackd thread as runnable.

ok, good. Not having signals involved certainly makes this simpler ;)

> i don't know what you mean by "channel" information in this context.

it was just a stupid question based on the ALSA source code, i assumed
that somehow you could have multiple audio streams [the driver called it
'voices'?] per driver, and the same interrupt comes for each stream. If
multiple streams are handled then there must be some sort of index to
find out where stream (or streams) got that particular interrupt, so
that Jackd can refill the proper output stream. I assumed (it seems
incorrectly) that when Rui ran 9 'fluidsynth' instances then Jackd would
open 9 'channels', one for each fludsynth instance?

but i suspect your comment means that there is a single 'output stream'
assembled by Jackd out of the 9 streams coming from the 9 fluidsynth
instances, with no concurrency issues on the driver level and the
SCHED_FIFO Jackd thread is the sole owner of that periodic interrupt,
and that there's no additional per-device information it needs to
associate it with each interrupt to ge to the current audio stream?
(i.e. there is only one output audio stream per device?)

and here i'm specifically concentrating on the scenario that Rui's
testing triggers: 9 fluidsynth instances running, and Jackd handling a
single ali5451 output device.

	Ingo

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Lee Revell]

On Thu, 2004-10-28 at 20:57 -0400, Paul Davis wrote:
> >>   XRUN Rate . . . . . . . . . . .     424         8         4    /hour
> >>   Delay Rate (>spare time)  . . .     496         0         0    /hour
> >>   Delay Rate (>1000 usecs)  . . .     940         8         4    /hour
> >>   Maximum Delay . . . . . . . . .    6904       921       721    usecs
> >>   Maximum Process Cycle . . . . .    1449      1469      1590    usecs
> >>   Average DSP CPU Load  . . . . .      38        39        40    %
> >>   Average Context-Switch Rate . .    7480      8929      9726    /sec
> >
> >looks pretty good, doesnt it?
> 
> yes and no. its troubling that we're using an extra 100usecs of time
> for the max process cycle, if the statistics make that number
> meaningful. and why a 30% increase in the context switch rate? is that
> an artifact or a real behavioural change? the xrun rate is not bad,
> although i'd need to know the period size. 4 clicks per hour would
> actually be unacceptable to most professionals, but this may have been
> with significant loading outside of JACK - i don't know.

I would not take these results too seriously yet, they are comparing one
highly experimental kernel to another.  Neither of these setups claims
to be ready for professional use yet - we are definitely going for zero
xruns, period, this seems to be achievable with most hardware.  I just
left this in to give you some context.

Lee

Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4]

[Paul Davis]

>Can anyone answer Ingo's questions about JACK & the best way to profile
>it?  This is a bit over my head.
>
>Lee
>
>
>-------- Forwarded Message --------
>Subject: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4
>Date: Thu, 28 Oct 2004 21:16:05 +0200
>* Rui Nuno Capela <rncbc@rncbc.org> wrote:
>
>> OK. Here are my early consolidated results. Feel free to comment.
>> 
>>                                     2.6.9     RT-U3   RT-V0.4.3
>>                                   --------- --------- ---------
>>   XRUN Rate . . . . . . . . . . .     424         8         4    /hour
>>   Delay Rate (>spare time)  . . .     496         0         0    /hour
>>   Delay Rate (>1000 usecs)  . . .     940         8         4    /hour
>>   Maximum Delay . . . . . . . . .    6904       921       721    usecs
>>   Maximum Process Cycle . . . . .    1449      1469      1590    usecs
>>   Average DSP CPU Load  . . . . .      38        39        40    %
>>   Average Context-Switch Rate . .    7480      8929      9726    /sec
>
>looks pretty good, doesnt it?

yes and no. its troubling that we're using an extra 100usecs of time
for the max process cycle, if the statistics make that number
meaningful. and why a 30% increase in the context switch rate? is that
an artifact or a real behavioural change? the xrun rate is not bad,
although i'd need to know the period size. 4 clicks per hour would
actually be unacceptable to most professionals, but this may have been
with significant loading outside of JACK - i don't know.

>how is the 'maximum delay' calculated? Could you put in a tracing hook
>into jackd whenever such a ~720 usecs maximum is hit? I'd _love_ to see 
>how such a latency path looks like, it seems a bit long.

"maximum delay" is computed like this:

	 a) we know that jackd should be woken at regular, known
	    intervals based on the interrupt of the audio interface.

	 b) jackd is (often) blocked in poll(2), waiting for the
	    audio interface to become readable+writable.

	 c) on return from poll, we check the time we are returning
	    and compare it to the last wakeup. the difference should
	    be almost exactly the interrupt interval. 

	 d) any amount larger than this is a delay caused by
	    kernel scheduling. we track the largest such delay.

>also, i looked at the sound/pci/ali5451/ali5451.c driver code and it has
>one weird piece of code on line 988:
>
>	udelay(100);

yeah, some ALSA drivers have broken stuff like this in them. its a
definite problem. this is why we know that some audio interfaces are
not able to provide acceptable performance for JACK - that is OK, the
same is true under Win/Mac where some h/w just can't be used for low
latency work.

>plus i've also got questions about how Jackd interfaces with ALSA: does
>it use SIGIO, or some direct driver ioctl? If SIGIO is used then how is
>it done precisely - is an 'RT' queued signal used or ordinary SIGIO? 
>Also, how is the 'channel' information established upon getting a SIGIO,
>is it in the siginfo structure?

we absolutely do not use SIGIO - i am one of the most vocal critics of
this interface even be present. POSIX says that signal handling
context is essentially crippled, so there is very little you can do
from there. in addition, measurements of signal delivery time on Linux
suggest that it is really quite slow. i considered using signals to
wake jack clients, but it was dramatically slower than using a FIFO
plus poll(2).

we block in poll(2), waiting for the interrupt from the audio
interface to mark the relevant jackd thread as runnable.

i don't know what you mean by "channel" information in this context.

--p