Re: Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-19

Re: Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-19

[Andrew Burgess]

Ingo Molnar said:

>Florian Schmidt <mista.tapas@gmx.net> wrote:
>>
>> Hmm, i wonder if there's a way to detect non RT behaviour in jackd
>> clients. I mean AFAIK the only thing allowed for the process callback
>> of on is the FIFO it waits on to be woken, right? Every other sleeping
>> is to be considered a bug. 

>there's such a feature in -RT kernels. If a user process calls:
>	gettimeofday(1,1);
>then the kernel turns 'atomic mode' on. To turn it off, call:
>	gettimeofday(1,0);

>while in atomic-mode, any non-atomic activity (scheduling) will produce
>a kernel message and a SIGUSR2 sent to the offending process (once,
>atomic mode has to be re-enabled again for the next message). Preemption
>by a higher-prio task does not trigger a message/signal.

>If you run the client under gdb you should be able to catch the SIGUSR2
>signal and then you can see the offending code's backtrace via 'bt'.

Might be handy to have the option to send a SIGABRT, then you don't need
to guess which app to run under gdb and the offending code is there in
the core file.

Also, I'm cc-ing jack-devel. This could fit into libjack so no client
mods would be needed I think. After the thread_init_callback is run libjack
could run 'gettimeofday(1,1);' for each client thread. Then if any client
breaks the rules you get a core showing where. 

On further thought, I suppose libjack could install a SIGUSR2 handler and
have that call abort for all the rt client threads. Still no client mods
needed, only an RT-aware libjack.

A big thank you to Ingo and everyone else involved on behalf of all the
linux audio users!

Re: [Jackit-devel] Re: Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-19

[Florian Schmidt]

On Thu, 2 Dec 2004 07:46:20 -0800
Andrew Burgess <aab@cichlid.com> wrote:

> Ingo Molnar said:
> 
> >Florian Schmidt <mista.tapas@gmx.net> wrote:
> >>
> >> Hmm, i wonder if there's a way to detect non RT behaviour in jackd
> >> clients. I mean AFAIK the only thing allowed for the process callback
> >> of on is the FIFO it waits on to be woken, right? Every other sleeping
> >> is to be considered a bug. 
> 
> >there's such a feature in -RT kernels. If a user process calls:
> >	gettimeofday(1,1);
> >then the kernel turns 'atomic mode' on. To turn it off, call:
> >	gettimeofday(1,0);
> 
> >while in atomic-mode, any non-atomic activity (scheduling) will produce
> >a kernel message and a SIGUSR2 sent to the offending process (once,
> >atomic mode has to be re-enabled again for the next message). Preemption
> >by a higher-prio task does not trigger a message/signal.
> 
> >If you run the client under gdb you should be able to catch the SIGUSR2
> >signal and then you can see the offending code's backtrace via 'bt'.
> 
> Might be handy to have the option to send a SIGABRT, then you don't need
> to guess which app to run under gdb and the offending code is there in
> the core file.
> 
> Also, I'm cc-ing jack-devel. This could fit into libjack so no client
> mods would be needed I think. After the thread_init_callback is run libjack
> could run 'gettimeofday(1,1);' for each client thread. Then if any client
> breaks the rules you get a core showing where. 
> 
> On further thought, I suppose libjack could install a SIGUSR2 handler and
> have that call abort for all the rt client threads. Still no client mods
> needed, only an RT-aware libjack.

right. Or instead of aborting jackd might print a debug output (like
"client foo violated RT constraints"). 

But the calls to gettimeofday would need to be done right before and
after every process callback as each client's RT thread does wait on the
FIFO to get woken by jackd. This waiting would appear as RT constraints
violation if the gettimeofday would be done only once per thread
lifetime at thread startup.

> 
> A big thank you to Ingo and everyone else involved on behalf of all the
> linux audio users!

BTW: i suppose pretty much every jack client except for very simple ones
do break the RT constraints.

Flo

Re: [Jackit-devel] Re: Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-19

[Jack O'Quin]

> On Thu, 2 Dec 2004 07:46:20 -0800
> Andrew Burgess <aab@cichlid.com> wrote:
> > On further thought, I suppose libjack could install a SIGUSR2 handler and
> > have that call abort for all the rt client threads. Still no client mods
> > needed, only an RT-aware libjack.

Florian Schmidt <mista.tapas@gmx.net> writes:
> right. Or instead of aborting jackd might print a debug output (like
> "client foo violated RT constraints"). 

Libjack cannot assume the client has no SIGUSR2 handler of its own.

> > A big thank you to Ingo and everyone else involved on behalf of all the
> > linux audio users!

thanks++  :-)

> BTW: i suppose pretty much every jack client except for very simple ones
> do break the RT constraints.

True.  

It would be wonderful to have a reliable mechanism for debugging them.
-- 
  joq

Re: [Jackit-devel] Re: Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-19

[Florian Schmidt]

On 02 Dec 2004 10:26:55 -0600
"Jack O'Quin" <joq@io.com> wrote:

> 
> > On Thu, 2 Dec 2004 07:46:20 -0800
> > Andrew Burgess <aab@cichlid.com> wrote:
> > > On further thought, I suppose libjack could install a SIGUSR2 handler and
> > > have that call abort for all the rt client threads. Still no client mods
> > > needed, only an RT-aware libjack.
> 
> Florian Schmidt <mista.tapas@gmx.net> writes:
> > right. Or instead of aborting jackd might print a debug output (like
> > "client foo violated RT constraints"). 
> 
> Libjack cannot assume the client has no SIGUSR2 handler of its own.

i see..

> It would be wonderful to have a reliable mechanism for debugging them.

I suppose instead of catching the signal the user might just monitor the
syslog. I'm not sure there's printk's triggered by thisalready , but i'm
sure if not, ingo might add them. So a trivial patch for jackd would
probably look like this:

--- libjack/client.c.orig       2004-12-02 17:55:04.000000000 +0100
+++ libjack/client.c    2004-12-02 17:56:23.000000000 +0100
@@ -1238,6 +1238,9 @@
                        if (control->sync_cb)
                                jack_call_sync_client (client);
 
+                       // enable atomicity check for RP kernels
+                       gettimeofday(1,1);
+
                        if (control->process) {
                                if (control->process (control->nframes,
                                                      control->process_arg)
@@ -1247,7 +1250,10 @@
                        } else {
                                control->state = Finished;
                        }
-
+
+                       // disable atomicity check
+                       gettimeofday(0,1);
+
                        if (control->timebase_cb)
                                jack_call_timebase_master (client);
 
flo

Re: [Jackit-devel] Re: Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-19

[Jack O'Quin]

Florian Schmidt <mista.tapas@gmx.net> writes:

> I suppose instead of catching the signal the user might just monitor the
> syslog. I'm not sure there's printk's triggered by thisalready , but i'm
> sure if not, ingo might add them. So a trivial patch for jackd would
> probably look like this:
> 
> --- libjack/client.c.orig       2004-12-02 17:55:04.000000000 +0100
> +++ libjack/client.c    2004-12-02 17:56:23.000000000 +0100
> @@ -1238,6 +1238,9 @@
>                         if (control->sync_cb)
>                                 jack_call_sync_client (client);
>  
> +                       // enable atomicity check for RP kernels
> +                       gettimeofday(1,1);
> +
>                         if (control->process) {
>                                 if (control->process (control->nframes,
>                                                       control->process_arg)
> @@ -1247,7 +1250,10 @@
>                         } else {
>                                 control->state = Finished;
>                         }
> -
> +
> +                       // disable atomicity check
> +                       gettimeofday(0,1);
> +
>                         if (control->timebase_cb)
>                                 jack_call_timebase_master (client);
>  

The sync_cb and timebase_cb callbacks actually need to be RT-safe,
too.  ;-)

Is printk() guaranteed not to wait inside the kernel?  I am not
familiar with its internal implementation.
-- 
  joq

Re: [Jackit-devel] Re: Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-19

[Florian Schmidt]

On Thu, 2 Dec 2004 17:57:56 +0100
Florian Schmidt <mista.tapas@gmx.net> wrote:

> On 02 Dec 2004 10:26:55 -0600
> "Jack O'Quin" <joq@io.com> wrote:
> 
> > 
> > > On Thu, 2 Dec 2004 07:46:20 -0800
> > > Andrew Burgess <aab@cichlid.com> wrote:
> > > > On further thought, I suppose libjack could install a SIGUSR2 handler and
> > > > have that call abort for all the rt client threads. Still no client mods
> > > > needed, only an RT-aware libjack.
> > 
> > Florian Schmidt <mista.tapas@gmx.net> writes:
> > > right. Or instead of aborting jackd might print a debug output (like
> > > "client foo violated RT constraints"). 
> > 
> > Libjack cannot assume the client has no SIGUSR2 handler of its own.
> 
> i see..
> 
> > It would be wonderful to have a reliable mechanism for debugging them.
> 
> I suppose instead of catching the signal the user might just monitor the
> syslog. I'm not sure there's printk's triggered by thisalready , but i'm
> sure if not, ingo might add them. So a trivial patch for jackd would
> probably look like this:
> 
> --- libjack/client.c.orig       2004-12-02 17:55:04.000000000 +0100
> +++ libjack/client.c    2004-12-02 17:56:23.000000000 +0100
> @@ -1238,6 +1238,9 @@
>                         if (control->sync_cb)
>                                 jack_call_sync_client (client);
>  
> +                       // enable atomicity check for RP kernels
> +                       gettimeofday(1,1);
> +
>                         if (control->process) {
>                                 if (control->process (control->nframes,
>                                                       control->process_arg)
> @@ -1247,7 +1250,10 @@
>                         } else {
>                                 control->state = Finished;
>                         }
> -
> +
> +                       // disable atomicity check
> +                       gettimeofday(0,1);
> +
>                         if (control->timebase_cb)
>                                 jack_call_timebase_master (client);
>  

Well, i do get syslog output with this patch like this:

Dec  2 18:01:11 mango kernel: jack_test:22645 userspace BUG: scheduling in user-atomic context!
Dec  2 18:01:11 mango kernel:  [<c02a38b6>] schedule+0x76/0x130 (8)
Dec  2 18:01:11 mango kernel:  [<c02a44c5>] schedule_timeout+0x85/0xe0 (36)
Dec  2 18:01:11 mango kernel:  [<c016677f>] do_pollfd+0x4f/0x90 (48)
Dec  2 18:01:11 mango kernel:  [<c011ceb0>] process_timeout+0x0/0x10 (8)
Dec  2 18:01:11 mango kernel:  [<c016686a>] do_poll+0xaa/0xd0 (20)
Dec  2 18:01:11 mango kernel:  [<c01669e2>] sys_poll+0x152/0x230 (48)
Dec  2 18:01:11 mango kernel:  [<c0165db0>] __pollwait+0x0/0xd0 (36)
Dec  2 18:01:11 mango kernel:  [<c01025cb>] syscall_call+0x7/0xb (32)

even if the client's process callback is a noop (except for returning
0). Hmm, i must have missed something in jackd's source. i thought
control->process() directly calls the clients process callback..

hmm..

Re: [Jackit-devel] Re: Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-19

[Jack O'Quin]

Florian Schmidt <mista.tapas@gmx.net> writes:

> Hmm, i must have missed something in jackd's source. i thought
> control->process() directly calls the clients process callback..

It does.
-- 
  joq

Re: [Jackit-devel] Re: Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-19

[Florian Schmidt]

On 02 Dec 2004 11:32:49 -0600
"Jack O'Quin" <joq@io.com> wrote:

> Florian Schmidt <mista.tapas@gmx.net> writes:
> 
> > Hmm, i must have missed something in jackd's source. i thought
> > control->process() directly calls the clients process callback..
> 
> It does.

Then either i have misunderstood how it works, or the mechanism is still
buggy.. Wrote mail to Ingo. Will report to jackit-devel when i get an
answer.

Flo

Re: [Jackit-devel] Re: Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-19

[Lee Revell]

On Thu, 2004-12-02 at 11:07 -0600, Jack O'Quin wrote:
> Is printk() guaranteed not to wait inside the kernel?  I am not
> familiar with its internal implementation.

Yes.  It just writes to a ring buffer and klogd dumps this to syslog.
So if you really start to spew printk's they don't all make it to the
log but you never get blocked.

The implementation probably looks a lot like a correct solution to fix
the printf-from-RT-context issue in JACK would.

Lee

Re: [Jackit-devel] Re: Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-19

[Jack O'Quin]

Lee Revell <rlrevell@joe-job.com> writes:

> On Thu, 2004-12-02 at 11:07 -0600, Jack O'Quin wrote:
> > Is printk() guaranteed not to wait inside the kernel?  I am not
> > familiar with its internal implementation.
> 
> Yes.  It just writes to a ring buffer and klogd dumps this to syslog.
> So if you really start to spew printk's they don't all make it to the
> log but you never get blocked.
> 
> The implementation probably looks a lot like a correct solution to fix
> the printf-from-RT-context issue in JACK would.

Right.  That's exactly what I have in mind, whenever I find time to
work on it.  :-)

There's some similar code I wrote for JAMin, which we could adapt.
-- 
  joq

Re: [Jackit-devel] Re: Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-19

[Florian Schmidt]

[-- Attachment #1: Type: text/plain, Size: 1907 bytes --]

On Thu, 2 Dec 2004 21:03:18 +0100
Florian Schmidt <mista.tapas@gmx.net> wrote:

> On 02 Dec 2004 11:32:49 -0600
> "Jack O'Quin" <joq@io.com> wrote:
> 
> > Florian Schmidt <mista.tapas@gmx.net> writes:
> > 
> > > Hmm, i must have missed something in jackd's source. i thought
> > > control->process() directly calls the clients process callback..
> > 
> > It does.
> 
> Then either i have misunderstood how it works, or the mechanism is still
> buggy.. Wrote mail to Ingo. Will report to jackit-devel when i get an
> answer.

PEBCAK :) Here's the corrected patch:

--- libjack/client.c.orig	2004-12-02 17:55:04.000000000 +0100
+++ libjack/client.c	2004-12-02 22:04:06.000000000 +0100
@@ -1238,6 +1238,9 @@
 			if (control->sync_cb)
 				jack_call_sync_client (client);
 
+			// enable atomicity check for RP kernels
+			gettimeofday(1,1);
+			
 			if (control->process) {
 				if (control->process (control->nframes,
 						      control->process_arg)
@@ -1247,7 +1250,10 @@
 			} else {
 				control->state = Finished;
 			}
-
+			
+			// disable atomicity check
+			gettimeofday(1,0);
+			
 			if (control->timebase_cb)
 				jack_call_timebase_master (client);
 

seems to work well with my changed jack_test client (this one sleeps in
the 1000th call of the process callback and thuis triggers this trace
and aborts as it doesn't handle SIGUSR2). test client attached.

Dec  2 22:05:10 mango kernel: jack_test:3043 userspace BUG: scheduling in user-atomic context!
Dec  2 22:05:10 mango kernel:  [<c02a38b6>] schedule+0x76/0x130 (8)
Dec  2 22:05:10 mango kernel:  [<c02a44c5>] schedule_timeout+0x85/0xe0 (36)
Dec  2 22:05:10 mango kernel:  [<c01e2f02>] copy_from_user+0x42/0x80 (48)
Dec  2 22:05:10 mango kernel:  [<c011ceb0>] process_timeout+0x0/0x10 (8)
Dec  2 22:05:10 mango kernel:  [<c011cfae>] sys_nanosleep+0xde/0x170 (20)
Dec  2 22:05:10 mango kernel:  [<c01025cb>] syscall_call+0x7/0xb (52)

flo

[-- Attachment #2: jack_test.cc --]
[-- Type: text/x-c++src, Size: 2186 bytes --]

#include <jack/jack.h>
#include <iostream>
#include <sstream>
#include <unistd.h>
#include <signal.h>

jack_client_t *client;
jack_port_t *iport;
jack_port_t *oport;

int wasted_loops = 0;

int sleep_seconds = 1;
int sleep_in_period = 1000;
int counter = 0;

int process(jack_nframes_t frames, void *arg) {
	
	// std::cout << "process callback" << std::endl;
	jack_default_audio_sample_t *ibuf;
	ibuf = (jack_default_audio_sample_t*)jack_port_get_buffer(iport, frames);

	jack_default_audio_sample_t *obuf;
	obuf = (jack_default_audio_sample_t*)jack_port_get_buffer(oport, frames);

	for (jack_nframes_t frame = 0; frame < frames; frame++) {
		for (int i = 0; i < wasted_loops; ++i) {
			// do nothing
		}
		obuf[frame] = ibuf[frame];
	}
	counter++;
	if (counter == sleep_in_period) {
	  sleep(sleep_seconds);
	}

        return 0;
}
void signalled(int sig) {
  std::cout << "signalled" << std::endl;
}


int main(int argc, char *argv[]) {
	// default = 60 seconds
	unsigned int seconds_to_run = 60;
	if (argc > 1) {
		std::stringstream sec_stream;
		sec_stream << argv[1];
		sec_stream >> seconds_to_run;
		if (argc > 2) {
			std::stringstream waste_stream;
			waste_stream << argv[2];
			waste_stream >> wasted_loops;
			std::cout << "wasted loops: " << wasted_loops << std::endl;
		}
	}

	// signal(SIGUSR2, signalled);

	std::cout << "seconds to run: " << seconds_to_run << std::endl;
	
	std::stringstream pid_stream;
	pid_stream << getpid();
	
        std::cout << "client_new" << std::endl;
        client = jack_client_new(pid_stream.str().c_str());

        std::cout << "port_register." << std::endl;
        iport = jack_port_register(client, "in", JACK_DEFAULT_AUDIO_TYPE, JackPortIsInput|JackPortIsTerminal, 0);
	oport = jack_port_register(client, "out", JACK_DEFAULT_AUDIO_TYPE, JackPortIsTerminal|JackPortIsOutput, 0);

        std::cout << "set_process_callback" << std::endl;
        jack_set_process_callback(client, process, 0);

        std::cout << "activate" << std::endl;
        jack_activate(client);

        std::cout << "running" << std::endl;

        // while(1) {sleep(1);};
	sleep(seconds_to_run);

	jack_deactivate(client);
	jack_client_close(client);
}

Re: Real-Time Preemption, -RT-2.6.10-rc2-mm3-V0.7.31-19

[Bill Huey]

On Thu, Dec 02, 2004 at 07:46:20AM -0800, Andrew Burgess wrote:
> A big thank you to Ingo and everyone else involved on behalf of all the
> linux audio users!

This is going to have a lot of ripple effect throughout the Linux
community in that game developers will quite possibly have the ability
to do low latency OpenGL, frame accurate video and other things along
those lines. The previous Linux kernels without these mods couldn't
allow this level of temporal precious for application developers. It's
going to push applications like jackd and others in ways that will
flush out bugs and general techniques that aren't typically apart of
a proper RT applications.

We have the possibility of providing a first class gaming platform and
other things, that Longhorn can't beging to touch if the middleware
falls into place. :)

bill