[Xenomai-core] Troubles with switchtest

[Xenomai-core] Troubles with switchtest

[Jan Kiszka]

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head
(only tested this so far): running it inside my test VM (ie. with
frequent excessive latencies) I get a stalled Linux timer IRQ quite
quickly. System is otherwise still responsive, Xenomai timers are still
being delivered, other Linux IRQs too. switchtest complained about

    "Warning: Linux is compiled to use FPU in kernel-space."

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
same effect.

Seen this before?

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Xenomai-core] Troubles with switchtest

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> Hi Gilles,
> 
> I'm currently facing a nasty effect with switchtest over latest git head
> (only tested this so far): running it inside my test VM (ie. with
> frequent excessive latencies) I get a stalled Linux timer IRQ quite
> quickly. System is otherwise still responsive, Xenomai timers are still
> being delivered, other Linux IRQs too. switchtest complained about
> 
>     "Warning: Linux is compiled to use FPU in kernel-space."
> 
> when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
> 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
> same effect.
> 
> Seen this before?

The warning about Linux being compiled to use FPU in kernel-space means
that you enabled soft RAID or compiled for K7, Geode, or any other
configuration using 3DNow for such simple operations as memcpy. It is
harmless, it simply means that switchtest can not use fpu in kernel-space.

The bug you have is probably the same as the one described here, which I
am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html

Unfortunately, I for one am working on ARM issues and am not available
to debug x86 issues. I think Philippe is busy too...

-- 
                                                 Gilles.

Re: [Xenomai-core] Troubles with switchtest

[Jan Kiszka]

Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>> Hi Gilles,
>>
>> I'm currently facing a nasty effect with switchtest over latest git head
>> (only tested this so far): running it inside my test VM (ie. with
>> frequent excessive latencies) I get a stalled Linux timer IRQ quite
>> quickly. System is otherwise still responsive, Xenomai timers are still
>> being delivered, other Linux IRQs too. switchtest complained about
>>
>>     "Warning: Linux is compiled to use FPU in kernel-space."
>>
>> when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
>> 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
>> same effect.
>>
>> Seen this before?
> 
> The warning about Linux being compiled to use FPU in kernel-space means
> that you enabled soft RAID or compiled for K7, Geode, or any other

RAID is on (ordinary server config).

> configuration using 3DNow for such simple operations as memcpy. It is
> harmless, it simply means that switchtest can not use fpu in kernel-space.
> 
> The bug you have is probably the same as the one described here, which I
> am able to reproduce on my atom:
> https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html
> 
> Unfortunately, I for one am working on ARM issues and am not available
> to debug x86 issues. I think Philippe is busy too...

OK, looks like I got the same flu here.

Philippe, did you find out any more details in the meantime? Then I'm
afraid I have to pick this up.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Xenomai-core] Troubles with switchtest

[Philippe Gerum]

On Wed, 2009-05-13 at 15:18 +0200, Jan Kiszka wrote:
> Gilles Chanteperdrix wrote:
> > Jan Kiszka wrote:
> >> Hi Gilles,
> >>
> >> I'm currently facing a nasty effect with switchtest over latest git head
> >> (only tested this so far): running it inside my test VM (ie. with
> >> frequent excessive latencies) I get a stalled Linux timer IRQ quite
> >> quickly. System is otherwise still responsive, Xenomai timers are still
> >> being delivered, other Linux IRQs too. switchtest complained about
> >>
> >>     "Warning: Linux is compiled to use FPU in kernel-space."
> >>
> >> when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
> >> 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
> >> same effect.
> >>
> >> Seen this before?
> > 
> > The warning about Linux being compiled to use FPU in kernel-space means
> > that you enabled soft RAID or compiled for K7, Geode, or any other
> 
> RAID is on (ordinary server config).
> 
> > configuration using 3DNow for such simple operations as memcpy. It is
> > harmless, it simply means that switchtest can not use fpu in kernel-space.
> > 
> > The bug you have is probably the same as the one described here, which I
> > am able to reproduce on my atom:
> > https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html
> > 
> > Unfortunately, I for one am working on ARM issues and am not available
> > to debug x86 issues. I think Philippe is busy too...
> 
> OK, looks like I got the same flu here.
> 
> Philippe, did you find out any more details in the meantime? Then I'm
> afraid I have to pick this up.

No, I did not resume this task yet. Working from the powerpc side of the
universe here.

> 
> Jan
> 
-- 
Philippe.

Re: [Xenomai-core] Troubles with switchtest

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> Gilles Chanteperdrix wrote:
>> Jan Kiszka wrote:
>>> Hi Gilles,
>>>
>>> I'm currently facing a nasty effect with switchtest over latest git head
>>> (only tested this so far): running it inside my test VM (ie. with
>>> frequent excessive latencies) I get a stalled Linux timer IRQ quite
>>> quickly. System is otherwise still responsive, Xenomai timers are still
>>> being delivered, other Linux IRQs too. switchtest complained about
>>>
>>>     "Warning: Linux is compiled to use FPU in kernel-space."
>>>
>>> when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
>>> 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
>>> same effect.
>>>
>>> Seen this before?
>> The warning about Linux being compiled to use FPU in kernel-space means
>> that you enabled soft RAID or compiled for K7, Geode, or any other
> 
> RAID is on (ordinary server config).

By the way, I wonder how MMX accelerated software raid works on K7,
since the way I understand the code, calls to kernel_fpu_begin() can not
be nested.

If you think they can be nested, then we can make switchtest test fpu in
Linux kernel-space when these options are enabled.

-- 
                                                 Gilles.

[Xenomai-core] [PATCH] Fix host IRQ propagation (was: Troubles with switchtest)

[Jan Kiszka]

Philippe Gerum wrote:
> On Wed, 2009-05-13 at 15:18 +0200, Jan Kiszka wrote:
>> Gilles Chanteperdrix wrote:
>>> Jan Kiszka wrote:
>>>> Hi Gilles,
>>>>
>>>> I'm currently facing a nasty effect with switchtest over latest git head
>>>> (only tested this so far): running it inside my test VM (ie. with
>>>> frequent excessive latencies) I get a stalled Linux timer IRQ quite
>>>> quickly. System is otherwise still responsive, Xenomai timers are still
>>>> being delivered, other Linux IRQs too. switchtest complained about
>>>>
>>>>     "Warning: Linux is compiled to use FPU in kernel-space."
>>>>
>>>> when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
>>>> 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
>>>> same effect.
>>>>
>>>> Seen this before?
>>> The warning about Linux being compiled to use FPU in kernel-space means
>>> that you enabled soft RAID or compiled for K7, Geode, or any other
>> RAID is on (ordinary server config).
>>
>>> configuration using 3DNow for such simple operations as memcpy. It is
>>> harmless, it simply means that switchtest can not use fpu in kernel-space.
>>>
>>> The bug you have is probably the same as the one described here, which I
>>> am able to reproduce on my atom:
>>> https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html
>>>
>>> Unfortunately, I for one am working on ARM issues and am not available
>>> to debug x86 issues. I think Philippe is busy too...
>> OK, looks like I got the same flu here.
>>
>> Philippe, did you find out any more details in the meantime? Then I'm
>> afraid I have to pick this up.
> 
> No, I did not resume this task yet. Working from the powerpc side of the
> universe here.

Hoho, don't think this rain here over x86 would have never made it down
to ARM or PPC land! ;)

Martin, could you check if this helps you, too?

Jan

(as usual, ready to be pulled from 'for-upstream')

--------->

Host IRQs may not only be triggered from non-root domains. But
rthal_propagate_irq's implemenation in I-pipe assumes so, which broke
host tick propagation under certain load scenarios. Besides that,
rthal_schedule_irq_root is the more efficient service for this purpose
anyway.

Signed-off-by: Jan Kiszka <jan.kiszka@domain.hid>
---

 include/asm-generic/hal.h |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/include/asm-generic/hal.h b/include/asm-generic/hal.h
index b37e476..8137856 100644
--- a/include/asm-generic/hal.h
+++ b/include/asm-generic/hal.h
@@ -437,7 +437,7 @@ int rthal_irq_host_release(unsigned irq,
 
 static inline void rthal_irq_host_pend(unsigned irq)
 {
-	rthal_propagate_irq(irq);
+	rthal_schedule_irq_root(irq);
 }
 
 int rthal_apc_alloc(const char *name,

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Xenomai-core] Troubles with switchtest

[Jan Kiszka]

Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>> Gilles Chanteperdrix wrote:
>>> Jan Kiszka wrote:
>>>> Hi Gilles,
>>>>
>>>> I'm currently facing a nasty effect with switchtest over latest git head
>>>> (only tested this so far): running it inside my test VM (ie. with
>>>> frequent excessive latencies) I get a stalled Linux timer IRQ quite
>>>> quickly. System is otherwise still responsive, Xenomai timers are still
>>>> being delivered, other Linux IRQs too. switchtest complained about
>>>>
>>>>     "Warning: Linux is compiled to use FPU in kernel-space."
>>>>
>>>> when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
>>>> 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
>>>> same effect.
>>>>
>>>> Seen this before?
>>> The warning about Linux being compiled to use FPU in kernel-space means
>>> that you enabled soft RAID or compiled for K7, Geode, or any other
>> RAID is on (ordinary server config).
> 
> By the way, I wonder how MMX accelerated software raid works on K7,
> since the way I understand the code, calls to kernel_fpu_begin() can not
> be nested.
> 
> If you think they can be nested, then we can make switchtest test fpu in
> Linux kernel-space when these options are enabled.

Sorry, I haven't looked that close into the in-kernel FPU handling so
far. What users are nested in the standard kernel? Or is it RAID in itself?

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation (was: Troubles with switchtest)

[Philippe Gerum]

On Wed, 2009-05-13 at 17:28 +0200, Jan Kiszka wrote:
> Philippe Gerum wrote:
> > On Wed, 2009-05-13 at 15:18 +0200, Jan Kiszka wrote:
> >> Gilles Chanteperdrix wrote:
> >>> Jan Kiszka wrote:
> >>>> Hi Gilles,
> >>>>
> >>>> I'm currently facing a nasty effect with switchtest over latest git head
> >>>> (only tested this so far): running it inside my test VM (ie. with
> >>>> frequent excessive latencies) I get a stalled Linux timer IRQ quite
> >>>> quickly. System is otherwise still responsive, Xenomai timers are still
> >>>> being delivered, other Linux IRQs too. switchtest complained about
> >>>>
> >>>>     "Warning: Linux is compiled to use FPU in kernel-space."
> >>>>
> >>>> when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
> >>>> 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
> >>>> same effect.
> >>>>
> >>>> Seen this before?
> >>> The warning about Linux being compiled to use FPU in kernel-space means
> >>> that you enabled soft RAID or compiled for K7, Geode, or any other
> >> RAID is on (ordinary server config).
> >>
> >>> configuration using 3DNow for such simple operations as memcpy. It is
> >>> harmless, it simply means that switchtest can not use fpu in kernel-space.
> >>>
> >>> The bug you have is probably the same as the one described here, which I
> >>> am able to reproduce on my atom:
> >>> https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html
> >>>
> >>> Unfortunately, I for one am working on ARM issues and am not available
> >>> to debug x86 issues. I think Philippe is busy too...
> >> OK, looks like I got the same flu here.
> >>
> >> Philippe, did you find out any more details in the meantime? Then I'm
> >> afraid I have to pick this up.
> > 
> > No, I did not resume this task yet. Working from the powerpc side of the
> > universe here.
> 
> Hoho, don't think this rain here over x86 would have never made it down
> to ARM or PPC land! ;)
> 
> Martin, could you check if this helps you, too?
> 
> Jan
> 
> (as usual, ready to be pulled from 'for-upstream')
> 
> --------->
> 
> Host IRQs may not only be triggered from non-root domains.

Are you sure of this? I can't find any spot where this assumption would
be wrong. host_pend() is basically there to relay RT timer ticks and
device IRQs, and this only happens on behalf of the pipeline head. At
least, this is how rthal_irq_host_pend() should be used in any case. If
you did find a spot where this interface is being called from the lower
stage, then this is the root bug to fix.

>  But
> rthal_propagate_irq's implemenation in I-pipe assumes so, which broke
> host tick propagation under certain load scenarios. Besides that,
> rthal_schedule_irq_root is the more efficient service for this purpose
> anyway.

Ack.

> 
> Signed-off-by: Jan Kiszka <jan.kiszka@domain.hid>
> ---
> 
>  include/asm-generic/hal.h |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/include/asm-generic/hal.h b/include/asm-generic/hal.h
> index b37e476..8137856 100644
> --- a/include/asm-generic/hal.h
> +++ b/include/asm-generic/hal.h
> @@ -437,7 +437,7 @@ int rthal_irq_host_release(unsigned irq,
>  
>  static inline void rthal_irq_host_pend(unsigned irq)
>  {
> -	rthal_propagate_irq(irq);
> +	rthal_schedule_irq_root(irq);
>  }
>  
>  int rthal_apc_alloc(const char *name,
> 
-- 
Philippe.

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

[Jan Kiszka]

Philippe Gerum wrote:
> On Wed, 2009-05-13 at 17:28 +0200, Jan Kiszka wrote:
>> Philippe Gerum wrote:
>>> On Wed, 2009-05-13 at 15:18 +0200, Jan Kiszka wrote:
>>>> Gilles Chanteperdrix wrote:
>>>>> Jan Kiszka wrote:
>>>>>> Hi Gilles,
>>>>>>
>>>>>> I'm currently facing a nasty effect with switchtest over latest git head
>>>>>> (only tested this so far): running it inside my test VM (ie. with
>>>>>> frequent excessive latencies) I get a stalled Linux timer IRQ quite
>>>>>> quickly. System is otherwise still responsive, Xenomai timers are still
>>>>>> being delivered, other Linux IRQs too. switchtest complained about
>>>>>>
>>>>>>     "Warning: Linux is compiled to use FPU in kernel-space."
>>>>>>
>>>>>> when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
>>>>>> 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
>>>>>> same effect.
>>>>>>
>>>>>> Seen this before?
>>>>> The warning about Linux being compiled to use FPU in kernel-space means
>>>>> that you enabled soft RAID or compiled for K7, Geode, or any other
>>>> RAID is on (ordinary server config).
>>>>
>>>>> configuration using 3DNow for such simple operations as memcpy. It is
>>>>> harmless, it simply means that switchtest can not use fpu in kernel-space.
>>>>>
>>>>> The bug you have is probably the same as the one described here, which I
>>>>> am able to reproduce on my atom:
>>>>> https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html
>>>>>
>>>>> Unfortunately, I for one am working on ARM issues and am not available
>>>>> to debug x86 issues. I think Philippe is busy too...
>>>> OK, looks like I got the same flu here.
>>>>
>>>> Philippe, did you find out any more details in the meantime? Then I'm
>>>> afraid I have to pick this up.
>>> No, I did not resume this task yet. Working from the powerpc side of the
>>> universe here.
>> Hoho, don't think this rain here over x86 would have never made it down
>> to ARM or PPC land! ;)
>>
>> Martin, could you check if this helps you, too?
>>
>> Jan
>>
>> (as usual, ready to be pulled from 'for-upstream')
>>
>> --------->
>>
>> Host IRQs may not only be triggered from non-root domains.
> 
> Are you sure of this? I can't find any spot where this assumption would
> be wrong. host_pend() is basically there to relay RT timer ticks and
> device IRQs, and this only happens on behalf of the pipeline head. At
> least, this is how rthal_irq_host_pend() should be used in any case. If
> you did find a spot where this interface is being called from the lower
> stage, then this is the root bug to fix.

I haven't studied the I-pipe trace /wrt this in details yet, but I could
imagine that some shadow task is interrupted in primary mode by the
timer IRQ and then leaves the handler in secondary mode due to whatever
events between schedule-out and in at the end of xnintr_clock_handler.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> Philippe Gerum wrote:
>> On Wed, 2009-05-13 at 15:18 +0200, Jan Kiszka wrote:
>>> Gilles Chanteperdrix wrote:
>>>> Jan Kiszka wrote:
>>>>> Hi Gilles,
>>>>>
>>>>> I'm currently facing a nasty effect with switchtest over latest git head
>>>>> (only tested this so far): running it inside my test VM (ie. with
>>>>> frequent excessive latencies) I get a stalled Linux timer IRQ quite
>>>>> quickly. System is otherwise still responsive, Xenomai timers are still
>>>>> being delivered, other Linux IRQs too. switchtest complained about
>>>>>
>>>>>     "Warning: Linux is compiled to use FPU in kernel-space."
>>>>>
>>>>> when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
>>>>> 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
>>>>> same effect.
>>>>>
>>>>> Seen this before?
>>>> The warning about Linux being compiled to use FPU in kernel-space means
>>>> that you enabled soft RAID or compiled for K7, Geode, or any other
>>> RAID is on (ordinary server config).
>>>
>>>> configuration using 3DNow for such simple operations as memcpy. It is
>>>> harmless, it simply means that switchtest can not use fpu in kernel-space.
>>>>
>>>> The bug you have is probably the same as the one described here, which I
>>>> am able to reproduce on my atom:
>>>> https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html
>>>>
>>>> Unfortunately, I for one am working on ARM issues and am not available
>>>> to debug x86 issues. I think Philippe is busy too...
>>> OK, looks like I got the same flu here.
>>>
>>> Philippe, did you find out any more details in the meantime? Then I'm
>>> afraid I have to pick this up.
>> No, I did not resume this task yet. Working from the powerpc side of the
>> universe here.
> 
> Hoho, don't think this rain here over x86 would have never made it down
> to ARM or PPC land! ;)
> 
> Martin, could you check if this helps you, too?
> 
> Jan
> 
> (as usual, ready to be pulled from 'for-upstream')
> 
> --------->
> 
> Host IRQs may not only be triggered from non-root domains. But
> rthal_propagate_irq's implemenation in I-pipe assumes so, which broke
> host tick propagation under certain load scenarios. Besides that,
> rthal_schedule_irq_root is the more efficient service for this purpose
> anyway.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@domain.hid>
> ---
> 
>  include/asm-generic/hal.h |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/include/asm-generic/hal.h b/include/asm-generic/hal.h
> index b37e476..8137856 100644
> --- a/include/asm-generic/hal.h
> +++ b/include/asm-generic/hal.h
> @@ -437,7 +437,7 @@ int rthal_irq_host_release(unsigned irq,
>  
>  static inline void rthal_irq_host_pend(unsigned irq)
>  {
> -	rthal_propagate_irq(irq);
> +	rthal_schedule_irq_root(irq);
>  }
>  
>  int rthal_apc_alloc(const char *name,
> 

Well, I would assume this could fix head, with the newly delayed root
tick irq propagation. But the user reports this bug with 2.4 too...

-- 
					    Gilles.

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

[Jan Kiszka]

Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>> Philippe Gerum wrote:
>>> On Wed, 2009-05-13 at 15:18 +0200, Jan Kiszka wrote:
>>>> Gilles Chanteperdrix wrote:
>>>>> Jan Kiszka wrote:
>>>>>> Hi Gilles,
>>>>>>
>>>>>> I'm currently facing a nasty effect with switchtest over latest git head
>>>>>> (only tested this so far): running it inside my test VM (ie. with
>>>>>> frequent excessive latencies) I get a stalled Linux timer IRQ quite
>>>>>> quickly. System is otherwise still responsive, Xenomai timers are still
>>>>>> being delivered, other Linux IRQs too. switchtest complained about
>>>>>>
>>>>>>     "Warning: Linux is compiled to use FPU in kernel-space."
>>>>>>
>>>>>> when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
>>>>>> 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
>>>>>> same effect.
>>>>>>
>>>>>> Seen this before?
>>>>> The warning about Linux being compiled to use FPU in kernel-space means
>>>>> that you enabled soft RAID or compiled for K7, Geode, or any other
>>>> RAID is on (ordinary server config).
>>>>
>>>>> configuration using 3DNow for such simple operations as memcpy. It is
>>>>> harmless, it simply means that switchtest can not use fpu in kernel-space.
>>>>>
>>>>> The bug you have is probably the same as the one described here, which I
>>>>> am able to reproduce on my atom:
>>>>> https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html
>>>>>
>>>>> Unfortunately, I for one am working on ARM issues and am not available
>>>>> to debug x86 issues. I think Philippe is busy too...
>>>> OK, looks like I got the same flu here.
>>>>
>>>> Philippe, did you find out any more details in the meantime? Then I'm
>>>> afraid I have to pick this up.
>>> No, I did not resume this task yet. Working from the powerpc side of the
>>> universe here.
>> Hoho, don't think this rain here over x86 would have never made it down
>> to ARM or PPC land! ;)
>>
>> Martin, could you check if this helps you, too?
>>
>> Jan
>>
>> (as usual, ready to be pulled from 'for-upstream')
>>
>> --------->
>>
>> Host IRQs may not only be triggered from non-root domains. But
>> rthal_propagate_irq's implemenation in I-pipe assumes so, which broke
>> host tick propagation under certain load scenarios. Besides that,
>> rthal_schedule_irq_root is the more efficient service for this purpose
>> anyway.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@domain.hid>
>> ---
>>
>>  include/asm-generic/hal.h |    2 +-
>>  1 files changed, 1 insertions(+), 1 deletions(-)
>>
>> diff --git a/include/asm-generic/hal.h b/include/asm-generic/hal.h
>> index b37e476..8137856 100644
>> --- a/include/asm-generic/hal.h
>> +++ b/include/asm-generic/hal.h
>> @@ -437,7 +437,7 @@ int rthal_irq_host_release(unsigned irq,
>>  
>>  static inline void rthal_irq_host_pend(unsigned irq)
>>  {
>> -	rthal_propagate_irq(irq);
>> +	rthal_schedule_irq_root(irq);
>>  }
>>  
>>  int rthal_apc_alloc(const char *name,
>>
> 
> Well, I would assume this could fix head, with the newly delayed root
> tick irq propagation. But the user reports this bug with 2.4 too...

According to my instrumentation, a delayed tick was not involved. The
problem was 'propagate' vs. non-root trigger domain. So 2.4 should
suffer from the same issue.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation (was: Troubles with switchtest)

[Martin Shepherd]

On Wed, 13 May 2009, Jan Kiszka wrote:
>...
> Martin, could you check if this helps you, too?

It doesn't appear to help. To check, first I turned on the HPET and PM
timer options, and recompiled the kernel without your patch, to verify
that this reproduced the problem. I then manually applied your patch
to include/asm-generic/xenomai/hal.h in the kernel source tree,
recompiled the kernel, installed etc, and rebooted. However even with
the patch in place, whenever I ran:

  dd if=/dev/zero of=/dev/null count=20000000

then initially top showed that dd was getting very little CPU time
(<10%), then after 30 seconds or so, the system became completely
unresponsive until the dd ended. This is how it acts without the patch
as well. So it doesn't appear that the patch has made any difference
to this problem.

Note that I applied the patch to Xenomai 2.5-rc1 in linux kernel
2.6.29.1. If the patch somehow relies on head, tell me, and I'll
endeavor to set up a new kernel using that.

Martin

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

[Philippe Gerum]

On Wed, 2009-05-13 at 18:10 +0200, Jan Kiszka wrote:
> Philippe Gerum wrote:
> > On Wed, 2009-05-13 at 17:28 +0200, Jan Kiszka wrote:
> >> Philippe Gerum wrote:
> >>> On Wed, 2009-05-13 at 15:18 +0200, Jan Kiszka wrote:
> >>>> Gilles Chanteperdrix wrote:
> >>>>> Jan Kiszka wrote:
> >>>>>> Hi Gilles,
> >>>>>>
> >>>>>> I'm currently facing a nasty effect with switchtest over latest git head
> >>>>>> (only tested this so far): running it inside my test VM (ie. with
> >>>>>> frequent excessive latencies) I get a stalled Linux timer IRQ quite
> >>>>>> quickly. System is otherwise still responsive, Xenomai timers are still
> >>>>>> being delivered, other Linux IRQs too. switchtest complained about
> >>>>>>
> >>>>>>     "Warning: Linux is compiled to use FPU in kernel-space."
> >>>>>>
> >>>>>> when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
> >>>>>> 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
> >>>>>> same effect.
> >>>>>>
> >>>>>> Seen this before?
> >>>>> The warning about Linux being compiled to use FPU in kernel-space means
> >>>>> that you enabled soft RAID or compiled for K7, Geode, or any other
> >>>> RAID is on (ordinary server config).
> >>>>
> >>>>> configuration using 3DNow for such simple operations as memcpy. It is
> >>>>> harmless, it simply means that switchtest can not use fpu in kernel-space.
> >>>>>
> >>>>> The bug you have is probably the same as the one described here, which I
> >>>>> am able to reproduce on my atom:
> >>>>> https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html
> >>>>>
> >>>>> Unfortunately, I for one am working on ARM issues and am not available
> >>>>> to debug x86 issues. I think Philippe is busy too...
> >>>> OK, looks like I got the same flu here.
> >>>>
> >>>> Philippe, did you find out any more details in the meantime? Then I'm
> >>>> afraid I have to pick this up.
> >>> No, I did not resume this task yet. Working from the powerpc side of the
> >>> universe here.
> >> Hoho, don't think this rain here over x86 would have never made it down
> >> to ARM or PPC land! ;)
> >>
> >> Martin, could you check if this helps you, too?
> >>
> >> Jan
> >>
> >> (as usual, ready to be pulled from 'for-upstream')
> >>
> >> --------->
> >>
> >> Host IRQs may not only be triggered from non-root domains.
> > 
> > Are you sure of this? I can't find any spot where this assumption would
> > be wrong. host_pend() is basically there to relay RT timer ticks and
> > device IRQs, and this only happens on behalf of the pipeline head. At
> > least, this is how rthal_irq_host_pend() should be used in any case. If
> > you did find a spot where this interface is being called from the lower
> > stage, then this is the root bug to fix.
> 
> I haven't studied the I-pipe trace /wrt this in details yet, but I could
> imagine that some shadow task is interrupted in primary mode by the
> timer IRQ and then leaves the handler in secondary mode due to whatever
> events between schedule-out and in at the end of xnintr_clock_handler.
> 

You need a thread context to move to secondary, I just can't see how
such scenario would be possible.

> Jan
> 
-- 
Philippe.

Re: [Xenomai-core] Troubles with switchtest

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> Gilles Chanteperdrix wrote:
>> Jan Kiszka wrote:
>>> Gilles Chanteperdrix wrote:
>>>> Jan Kiszka wrote:
>>>>> Hi Gilles,
>>>>>
>>>>> I'm currently facing a nasty effect with switchtest over latest git head
>>>>> (only tested this so far): running it inside my test VM (ie. with
>>>>> frequent excessive latencies) I get a stalled Linux timer IRQ quite
>>>>> quickly. System is otherwise still responsive, Xenomai timers are still
>>>>> being delivered, other Linux IRQs too. switchtest complained about
>>>>>
>>>>>     "Warning: Linux is compiled to use FPU in kernel-space."
>>>>>
>>>>> when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
>>>>> 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
>>>>> same effect.
>>>>>
>>>>> Seen this before?
>>>> The warning about Linux being compiled to use FPU in kernel-space means
>>>> that you enabled soft RAID or compiled for K7, Geode, or any other
>>> RAID is on (ordinary server config).
>> By the way, I wonder how MMX accelerated software raid works on K7,
>> since the way I understand the code, calls to kernel_fpu_begin() can not
>> be nested.
>>
>> If you think they can be nested, then we can make switchtest test fpu in
>> Linux kernel-space when these options are enabled.
> 
> Sorry, I haven't looked that close into the in-kernel FPU handling so
> far. What users are nested in the standard kernel? Or is it RAID in itself?

RAID uses FPU. And on K7, basic things like clearing or copying user
pages, or even large memcpy do use FPU too. So, I would expect that when
enabling both, they could happen to be nested.

-- 
					    Gilles.

Re: [Xenomai-core] Troubles with switchtest

[Jan Kiszka]

Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>> Gilles Chanteperdrix wrote:
>>> Jan Kiszka wrote:
>>>> Gilles Chanteperdrix wrote:
>>>>> Jan Kiszka wrote:
>>>>>> Hi Gilles,
>>>>>>
>>>>>> I'm currently facing a nasty effect with switchtest over latest git head
>>>>>> (only tested this so far): running it inside my test VM (ie. with
>>>>>> frequent excessive latencies) I get a stalled Linux timer IRQ quite
>>>>>> quickly. System is otherwise still responsive, Xenomai timers are still
>>>>>> being delivered, other Linux IRQs too. switchtest complained about
>>>>>>
>>>>>>     "Warning: Linux is compiled to use FPU in kernel-space."
>>>>>>
>>>>>> when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
>>>>>> 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
>>>>>> same effect.
>>>>>>
>>>>>> Seen this before?
>>>>> The warning about Linux being compiled to use FPU in kernel-space means
>>>>> that you enabled soft RAID or compiled for K7, Geode, or any other
>>>> RAID is on (ordinary server config).
>>> By the way, I wonder how MMX accelerated software raid works on K7,
>>> since the way I understand the code, calls to kernel_fpu_begin() can not
>>> be nested.
>>>
>>> If you think they can be nested, then we can make switchtest test fpu in
>>> Linux kernel-space when these options are enabled.
>> Sorry, I haven't looked that close into the in-kernel FPU handling so
>> far. What users are nested in the standard kernel? Or is it RAID in itself?
> 
> RAID uses FPU. And on K7, basic things like clearing or copying user
> pages, or even large memcpy do use FPU too. So, I would expect that when
> enabling both, they could happen to be nested.
> 

I think the trick against /this/ is preempt_disable/enable in
kernel_fpu/begin/end. But that won't work for Xenomai, of course.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Xenomai-core] Troubles with switchtest

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> I think the trick against /this/ is preempt_disable/enable in
> kernel_fpu/begin/end. But that won't work for Xenomai, of course.

Well, that does not prevent an IRQ or a page fault from computing a RAID
cheksum...

-- 
                                                 Gilles.

Re: [Xenomai-core] Troubles with switchtest

[Jan Kiszka]

Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>> I think the trick against /this/ is preempt_disable/enable in
>> kernel_fpu/begin/end. But that won't work for Xenomai, of course.
> 
> Well, that does not prevent an IRQ or a page fault from computing a RAID
> cheksum...

If this is allowed, then... My impression was that FPU access is not
legal from IRQ context.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Xenomai-core] Troubles with switchtest

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> Gilles Chanteperdrix wrote:
>> Jan Kiszka wrote:
>>> I think the trick against /this/ is preempt_disable/enable in
>>> kernel_fpu/begin/end. But that won't work for Xenomai, of course.
>> Well, that does not prevent an IRQ or a page fault from computing a RAID
>> cheksum...
> 
> If this is allowed, then... My impression was that FPU access is not
> legal from IRQ context.

If you have a mapped file on a raid disk and a page fault triggers a
write, I think the RAID routines will be used. But you are right, maybe
there is a test to use the plain integer RAID routines when called from
interrupt context.

-- 
                                                 Gilles.

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

[Jan Kiszka]

Philippe Gerum wrote:
> On Wed, 2009-05-13 at 18:10 +0200, Jan Kiszka wrote:
>> Philippe Gerum wrote:
>>> On Wed, 2009-05-13 at 17:28 +0200, Jan Kiszka wrote:
>>>> Philippe Gerum wrote:
>>>>> On Wed, 2009-05-13 at 15:18 +0200, Jan Kiszka wrote:
>>>>>> Gilles Chanteperdrix wrote:
>>>>>>> Jan Kiszka wrote:
>>>>>>>> Hi Gilles,
>>>>>>>>
>>>>>>>> I'm currently facing a nasty effect with switchtest over latest git head
>>>>>>>> (only tested this so far): running it inside my test VM (ie. with
>>>>>>>> frequent excessive latencies) I get a stalled Linux timer IRQ quite
>>>>>>>> quickly. System is otherwise still responsive, Xenomai timers are still
>>>>>>>> being delivered, other Linux IRQs too. switchtest complained about
>>>>>>>>
>>>>>>>>     "Warning: Linux is compiled to use FPU in kernel-space."
>>>>>>>>
>>>>>>>> when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
>>>>>>>> 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
>>>>>>>> same effect.
>>>>>>>>
>>>>>>>> Seen this before?
>>>>>>> The warning about Linux being compiled to use FPU in kernel-space means
>>>>>>> that you enabled soft RAID or compiled for K7, Geode, or any other
>>>>>> RAID is on (ordinary server config).
>>>>>>
>>>>>>> configuration using 3DNow for such simple operations as memcpy. It is
>>>>>>> harmless, it simply means that switchtest can not use fpu in kernel-space.
>>>>>>>
>>>>>>> The bug you have is probably the same as the one described here, which I
>>>>>>> am able to reproduce on my atom:
>>>>>>> https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html
>>>>>>>
>>>>>>> Unfortunately, I for one am working on ARM issues and am not available
>>>>>>> to debug x86 issues. I think Philippe is busy too...
>>>>>> OK, looks like I got the same flu here.
>>>>>>
>>>>>> Philippe, did you find out any more details in the meantime? Then I'm
>>>>>> afraid I have to pick this up.
>>>>> No, I did not resume this task yet. Working from the powerpc side of the
>>>>> universe here.
>>>> Hoho, don't think this rain here over x86 would have never made it down
>>>> to ARM or PPC land! ;)
>>>>
>>>> Martin, could you check if this helps you, too?
>>>>
>>>> Jan
>>>>
>>>> (as usual, ready to be pulled from 'for-upstream')
>>>>
>>>> --------->
>>>>
>>>> Host IRQs may not only be triggered from non-root domains.
>>> Are you sure of this? I can't find any spot where this assumption would
>>> be wrong. host_pend() is basically there to relay RT timer ticks and
>>> device IRQs, and this only happens on behalf of the pipeline head. At
>>> least, this is how rthal_irq_host_pend() should be used in any case. If
>>> you did find a spot where this interface is being called from the lower
>>> stage, then this is the root bug to fix.
>> I haven't studied the I-pipe trace /wrt this in details yet, but I could
>> imagine that some shadow task is interrupted in primary mode by the
>> timer IRQ and then leaves the handler in secondary mode due to whatever
>> events between schedule-out and in at the end of xnintr_clock_handler.
>>
> 
> You need a thread context to move to secondary, I just can't see how
> such scenario would be possible.

Here is the trace of events:

=> Shadow task starts migration to secondary
=> in xnpod_suspend_thread, nklock is briefly released before
   xnpod_schedule
=> timer IRQ intercepts
=> as the current CPU is marked for reschedule, we enter xnpod_schedule
   before propagating the host tick
=> once the migrating thread comes in again, it will run the
   xnintr_clock_handler tail, i.e. xnarch_relay_tick, already over the
   root domain

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

[Jan Kiszka]

Martin Shepherd wrote:
> On Wed, 13 May 2009, Jan Kiszka wrote:
>> ...
>> Martin, could you check if this helps you, too?
> 
> It doesn't appear to help. To check, first I turned on the HPET and PM
> timer options, and recompiled the kernel without your patch, to verify
> that this reproduced the problem. I then manually applied your patch
> to include/asm-generic/xenomai/hal.h in the kernel source tree,
> recompiled the kernel, installed etc, and rebooted. However even with
> the patch in place, whenever I ran:
> 
>   dd if=/dev/zero of=/dev/null count=20000000
> 
> then initially top showed that dd was getting very little CPU time
> (<10%), then after 30 seconds or so, the system became completely
> unresponsive until the dd ended. This is how it acts without the patch
> as well. So it doesn't appear that the patch has made any difference
> to this problem.

Hmm, so there is no Xenomai task running during this test, just Linux
load? Tried to reproduce with my kernel here (both HPET and PM enabled,
as usual), but no success. Strange.

> Note that I applied the patch to Xenomai 2.5-rc1 in linux kernel
> 2.6.29.1. If the patch somehow relies on head, tell me, and I'll
> endeavor to set up a new kernel using that.

The patch must cleanly apply as is on -rc1, any later changes did not
affect its context.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

[Philippe Gerum]

On Thu, 2009-05-14 at 12:20 +0200, Jan Kiszka wrote:
> Philippe Gerum wrote:
> > On Wed, 2009-05-13 at 18:10 +0200, Jan Kiszka wrote:
> >> Philippe Gerum wrote:
> >>> On Wed, 2009-05-13 at 17:28 +0200, Jan Kiszka wrote:
> >>>> Philippe Gerum wrote:
> >>>>> On Wed, 2009-05-13 at 15:18 +0200, Jan Kiszka wrote:
> >>>>>> Gilles Chanteperdrix wrote:
> >>>>>>> Jan Kiszka wrote:
> >>>>>>>> Hi Gilles,
> >>>>>>>>
> >>>>>>>> I'm currently facing a nasty effect with switchtest over latest git head
> >>>>>>>> (only tested this so far): running it inside my test VM (ie. with
> >>>>>>>> frequent excessive latencies) I get a stalled Linux timer IRQ quite
> >>>>>>>> quickly. System is otherwise still responsive, Xenomai timers are still
> >>>>>>>> being delivered, other Linux IRQs too. switchtest complained about
> >>>>>>>>
> >>>>>>>>     "Warning: Linux is compiled to use FPU in kernel-space."
> >>>>>>>>
> >>>>>>>> when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
> >>>>>>>> 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
> >>>>>>>> same effect.
> >>>>>>>>
> >>>>>>>> Seen this before?
> >>>>>>> The warning about Linux being compiled to use FPU in kernel-space means
> >>>>>>> that you enabled soft RAID or compiled for K7, Geode, or any other
> >>>>>> RAID is on (ordinary server config).
> >>>>>>
> >>>>>>> configuration using 3DNow for such simple operations as memcpy. It is
> >>>>>>> harmless, it simply means that switchtest can not use fpu in kernel-space.
> >>>>>>>
> >>>>>>> The bug you have is probably the same as the one described here, which I
> >>>>>>> am able to reproduce on my atom:
> >>>>>>> https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html
> >>>>>>>
> >>>>>>> Unfortunately, I for one am working on ARM issues and am not available
> >>>>>>> to debug x86 issues. I think Philippe is busy too...
> >>>>>> OK, looks like I got the same flu here.
> >>>>>>
> >>>>>> Philippe, did you find out any more details in the meantime? Then I'm
> >>>>>> afraid I have to pick this up.
> >>>>> No, I did not resume this task yet. Working from the powerpc side of the
> >>>>> universe here.
> >>>> Hoho, don't think this rain here over x86 would have never made it down
> >>>> to ARM or PPC land! ;)
> >>>>
> >>>> Martin, could you check if this helps you, too?
> >>>>
> >>>> Jan
> >>>>
> >>>> (as usual, ready to be pulled from 'for-upstream')
> >>>>
> >>>> --------->
> >>>>
> >>>> Host IRQs may not only be triggered from non-root domains.
> >>> Are you sure of this? I can't find any spot where this assumption would
> >>> be wrong. host_pend() is basically there to relay RT timer ticks and
> >>> device IRQs, and this only happens on behalf of the pipeline head. At
> >>> least, this is how rthal_irq_host_pend() should be used in any case. If
> >>> you did find a spot where this interface is being called from the lower
> >>> stage, then this is the root bug to fix.
> >> I haven't studied the I-pipe trace /wrt this in details yet, but I could
> >> imagine that some shadow task is interrupted in primary mode by the
> >> timer IRQ and then leaves the handler in secondary mode due to whatever
> >> events between schedule-out and in at the end of xnintr_clock_handler.
> >>
> > 
> > You need a thread context to move to secondary, I just can't see how
> > such scenario would be possible.
> 
> Here is the trace of events:
> 
> => Shadow task starts migration to secondary
> => in xnpod_suspend_thread, nklock is briefly released before
>    xnpod_schedule

Which is the root bug. Blame on me; this recent change in -head breaks a
basic rule a lot of code is based on: a self-suspending thread may not
be preempted while scheduling out, i.e. suspension and rescheduling must
be atomically performed. xnshadow_relax() counts on this too.

> => timer IRQ intercepts
> => as the current CPU is marked for reschedule, we enter xnpod_schedule
>    before propagating the host tick
> => once the migrating thread comes in again, it will run the
>    xnintr_clock_handler tail, i.e. xnarch_relay_tick, already over the
>    root domain

Ok, makes sense now. However, this can't happen with 2.4 which has no
such lock release in xnpod_suspend_thread(). So the question is: was the
"lost tick" bug observed also on 2.4, or not?

> 
> Jan
> 
-- 
Philippe.

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

[Jan Kiszka]

Philippe Gerum wrote:
> On Thu, 2009-05-14 at 12:20 +0200, Jan Kiszka wrote:
>> Philippe Gerum wrote:
>>> On Wed, 2009-05-13 at 18:10 +0200, Jan Kiszka wrote:
>>>> Philippe Gerum wrote:
>>>>> On Wed, 2009-05-13 at 17:28 +0200, Jan Kiszka wrote:
>>>>>> Philippe Gerum wrote:
>>>>>>> On Wed, 2009-05-13 at 15:18 +0200, Jan Kiszka wrote:
>>>>>>>> Gilles Chanteperdrix wrote:
>>>>>>>>> Jan Kiszka wrote:
>>>>>>>>>> Hi Gilles,
>>>>>>>>>>
>>>>>>>>>> I'm currently facing a nasty effect with switchtest over latest git head
>>>>>>>>>> (only tested this so far): running it inside my test VM (ie. with
>>>>>>>>>> frequent excessive latencies) I get a stalled Linux timer IRQ quite
>>>>>>>>>> quickly. System is otherwise still responsive, Xenomai timers are still
>>>>>>>>>> being delivered, other Linux IRQs too. switchtest complained about
>>>>>>>>>>
>>>>>>>>>>     "Warning: Linux is compiled to use FPU in kernel-space."
>>>>>>>>>>
>>>>>>>>>> when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
>>>>>>>>>> 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
>>>>>>>>>> same effect.
>>>>>>>>>>
>>>>>>>>>> Seen this before?
>>>>>>>>> The warning about Linux being compiled to use FPU in kernel-space means
>>>>>>>>> that you enabled soft RAID or compiled for K7, Geode, or any other
>>>>>>>> RAID is on (ordinary server config).
>>>>>>>>
>>>>>>>>> configuration using 3DNow for such simple operations as memcpy. It is
>>>>>>>>> harmless, it simply means that switchtest can not use fpu in kernel-space.
>>>>>>>>>
>>>>>>>>> The bug you have is probably the same as the one described here, which I
>>>>>>>>> am able to reproduce on my atom:
>>>>>>>>> https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html
>>>>>>>>>
>>>>>>>>> Unfortunately, I for one am working on ARM issues and am not available
>>>>>>>>> to debug x86 issues. I think Philippe is busy too...
>>>>>>>> OK, looks like I got the same flu here.
>>>>>>>>
>>>>>>>> Philippe, did you find out any more details in the meantime? Then I'm
>>>>>>>> afraid I have to pick this up.
>>>>>>> No, I did not resume this task yet. Working from the powerpc side of the
>>>>>>> universe here.
>>>>>> Hoho, don't think this rain here over x86 would have never made it down
>>>>>> to ARM or PPC land! ;)
>>>>>>
>>>>>> Martin, could you check if this helps you, too?
>>>>>>
>>>>>> Jan
>>>>>>
>>>>>> (as usual, ready to be pulled from 'for-upstream')
>>>>>>
>>>>>> --------->
>>>>>>
>>>>>> Host IRQs may not only be triggered from non-root domains.
>>>>> Are you sure of this? I can't find any spot where this assumption would
>>>>> be wrong. host_pend() is basically there to relay RT timer ticks and
>>>>> device IRQs, and this only happens on behalf of the pipeline head. At
>>>>> least, this is how rthal_irq_host_pend() should be used in any case. If
>>>>> you did find a spot where this interface is being called from the lower
>>>>> stage, then this is the root bug to fix.
>>>> I haven't studied the I-pipe trace /wrt this in details yet, but I could
>>>> imagine that some shadow task is interrupted in primary mode by the
>>>> timer IRQ and then leaves the handler in secondary mode due to whatever
>>>> events between schedule-out and in at the end of xnintr_clock_handler.
>>>>
>>> You need a thread context to move to secondary, I just can't see how
>>> such scenario would be possible.
>> Here is the trace of events:
>>
>> => Shadow task starts migration to secondary
>> => in xnpod_suspend_thread, nklock is briefly released before
>>    xnpod_schedule
> 
> Which is the root bug. Blame on me; this recent change in -head breaks a
> basic rule a lot of code is based on: a self-suspending thread may not
> be preempted while scheduling out, i.e. suspension and rescheduling must
> be atomically performed. xnshadow_relax() counts on this too.

Oh, good that you insisted on this. Will you fix it soon? We are
currently packaging a delivery of 2.5.git, and I would like to see this
hole closed there already.

> 
>> => timer IRQ intercepts
>> => as the current CPU is marked for reschedule, we enter xnpod_schedule
>>    before propagating the host tick
>> => once the migrating thread comes in again, it will run the
>>    xnintr_clock_handler tail, i.e. xnarch_relay_tick, already over the
>>    root domain
> 
> Ok, makes sense now. However, this can't happen with 2.4 which has no
> such lock release in xnpod_suspend_thread(). So the question is: was the
> "lost tick" bug observed also on 2.4, or not?

I haven't tested on 2.4, but Martin anyway reported that his problem is
still unfixed for 2.5 even with my patch.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

[Philippe Gerum]

On Thu, 2009-05-14 at 13:00 +0200, Jan Kiszka wrote:
> Philippe Gerum wrote:
> > On Thu, 2009-05-14 at 12:20 +0200, Jan Kiszka wrote:
> >> Philippe Gerum wrote:
> >>> On Wed, 2009-05-13 at 18:10 +0200, Jan Kiszka wrote:
> >>>> Philippe Gerum wrote:
> >>>>> On Wed, 2009-05-13 at 17:28 +0200, Jan Kiszka wrote:
> >>>>>> Philippe Gerum wrote:
> >>>>>>> On Wed, 2009-05-13 at 15:18 +0200, Jan Kiszka wrote:
> >>>>>>>> Gilles Chanteperdrix wrote:
> >>>>>>>>> Jan Kiszka wrote:
> >>>>>>>>>> Hi Gilles,
> >>>>>>>>>>
> >>>>>>>>>> I'm currently facing a nasty effect with switchtest over latest git head
> >>>>>>>>>> (only tested this so far): running it inside my test VM (ie. with
> >>>>>>>>>> frequent excessive latencies) I get a stalled Linux timer IRQ quite
> >>>>>>>>>> quickly. System is otherwise still responsive, Xenomai timers are still
> >>>>>>>>>> being delivered, other Linux IRQs too. switchtest complained about
> >>>>>>>>>>
> >>>>>>>>>>     "Warning: Linux is compiled to use FPU in kernel-space."
> >>>>>>>>>>
> >>>>>>>>>> when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
> >>>>>>>>>> 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
> >>>>>>>>>> same effect.
> >>>>>>>>>>
> >>>>>>>>>> Seen this before?
> >>>>>>>>> The warning about Linux being compiled to use FPU in kernel-space means
> >>>>>>>>> that you enabled soft RAID or compiled for K7, Geode, or any other
> >>>>>>>> RAID is on (ordinary server config).
> >>>>>>>>
> >>>>>>>>> configuration using 3DNow for such simple operations as memcpy. It is
> >>>>>>>>> harmless, it simply means that switchtest can not use fpu in kernel-space.
> >>>>>>>>>
> >>>>>>>>> The bug you have is probably the same as the one described here, which I
> >>>>>>>>> am able to reproduce on my atom:
> >>>>>>>>> https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html
> >>>>>>>>>
> >>>>>>>>> Unfortunately, I for one am working on ARM issues and am not available
> >>>>>>>>> to debug x86 issues. I think Philippe is busy too...
> >>>>>>>> OK, looks like I got the same flu here.
> >>>>>>>>
> >>>>>>>> Philippe, did you find out any more details in the meantime? Then I'm
> >>>>>>>> afraid I have to pick this up.
> >>>>>>> No, I did not resume this task yet. Working from the powerpc side of the
> >>>>>>> universe here.
> >>>>>> Hoho, don't think this rain here over x86 would have never made it down
> >>>>>> to ARM or PPC land! ;)
> >>>>>>
> >>>>>> Martin, could you check if this helps you, too?
> >>>>>>
> >>>>>> Jan
> >>>>>>
> >>>>>> (as usual, ready to be pulled from 'for-upstream')
> >>>>>>
> >>>>>> --------->
> >>>>>>
> >>>>>> Host IRQs may not only be triggered from non-root domains.
> >>>>> Are you sure of this? I can't find any spot where this assumption would
> >>>>> be wrong. host_pend() is basically there to relay RT timer ticks and
> >>>>> device IRQs, and this only happens on behalf of the pipeline head. At
> >>>>> least, this is how rthal_irq_host_pend() should be used in any case. If
> >>>>> you did find a spot where this interface is being called from the lower
> >>>>> stage, then this is the root bug to fix.
> >>>> I haven't studied the I-pipe trace /wrt this in details yet, but I could
> >>>> imagine that some shadow task is interrupted in primary mode by the
> >>>> timer IRQ and then leaves the handler in secondary mode due to whatever
> >>>> events between schedule-out and in at the end of xnintr_clock_handler.
> >>>>
> >>> You need a thread context to move to secondary, I just can't see how
> >>> such scenario would be possible.
> >> Here is the trace of events:
> >>
> >> => Shadow task starts migration to secondary
> >> => in xnpod_suspend_thread, nklock is briefly released before
> >>    xnpod_schedule
> > 
> > Which is the root bug. Blame on me; this recent change in -head breaks a
> > basic rule a lot of code is based on: a self-suspending thread may not
> > be preempted while scheduling out, i.e. suspension and rescheduling must
> > be atomically performed. xnshadow_relax() counts on this too.
> 
> Oh, good that you insisted on this. Will you fix it soon? We are
> currently packaging a delivery of 2.5.git, and I would like to see this
> hole closed there already.
> 

I just pushed a fix for this.

> > 
> >> => timer IRQ intercepts
> >> => as the current CPU is marked for reschedule, we enter xnpod_schedule
> >>    before propagating the host tick
> >> => once the migrating thread comes in again, it will run the
> >>    xnintr_clock_handler tail, i.e. xnarch_relay_tick, already over the
> >>    root domain
> > 
> > Ok, makes sense now. However, this can't happen with 2.4 which has no
> > such lock release in xnpod_suspend_thread(). So the question is: was the
> > "lost tick" bug observed also on 2.4, or not?
> 
> I haven't tested on 2.4, but Martin anyway reported that his problem is
> still unfixed for 2.5 even with my patch.
> 
> Jan
> 
-- 
Philippe.

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> Martin Shepherd wrote:
>> On Wed, 13 May 2009, Jan Kiszka wrote:
>>> ...
>>> Martin, could you check if this helps you, too?
>> It doesn't appear to help. To check, first I turned on the HPET and PM
>> timer options, and recompiled the kernel without your patch, to verify
>> that this reproduced the problem. I then manually applied your patch
>> to include/asm-generic/xenomai/hal.h in the kernel source tree,
>> recompiled the kernel, installed etc, and rebooted. However even with
>> the patch in place, whenever I ran:
>>
>>   dd if=/dev/zero of=/dev/null count=20000000
>>
>> then initially top showed that dd was getting very little CPU time
>> (<10%), then after 30 seconds or so, the system became completely
>> unresponsive until the dd ended. This is how it acts without the patch
>> as well. So it doesn't appear that the patch has made any difference
>> to this problem.
> 
> Hmm, so there is no Xenomai task running during this test, just Linux
> load? Tried to reproduce with my kernel here (both HPET and PM enabled,
> as usual), but no success. Strange.

Just having HPET and PM_TIMER enabled is not enough: your CPU needs to
actually have an HPET timer or PM timer.

-- 
                                                 Gilles.

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

[Jan Kiszka]

Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>> Martin Shepherd wrote:
>>> On Wed, 13 May 2009, Jan Kiszka wrote:
>>>> ...
>>>> Martin, could you check if this helps you, too?
>>> It doesn't appear to help. To check, first I turned on the HPET and PM
>>> timer options, and recompiled the kernel without your patch, to verify
>>> that this reproduced the problem. I then manually applied your patch
>>> to include/asm-generic/xenomai/hal.h in the kernel source tree,
>>> recompiled the kernel, installed etc, and rebooted. However even with
>>> the patch in place, whenever I ran:
>>>
>>>   dd if=/dev/zero of=/dev/null count=20000000
>>>
>>> then initially top showed that dd was getting very little CPU time
>>> (<10%), then after 30 seconds or so, the system became completely
>>> unresponsive until the dd ended. This is how it acts without the patch
>>> as well. So it doesn't appear that the patch has made any difference
>>> to this problem.
>> Hmm, so there is no Xenomai task running during this test, just Linux
>> load? Tried to reproduce with my kernel here (both HPET and PM enabled,
>> as usual), but no success. Strange.
> 
> Just having HPET and PM_TIMER enabled is not enough: your CPU needs to
> actually have an HPET timer or PM timer.

That's the case for practically any modern system. So I bet the "trick"
is to make Linux _use_ one of them - in favor of the normally better
suited LAPIC. Need to check again what the reasons for this switch could
be. One that comes to my mind is problems of LAPIC vs. power management.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

[Gilles Chanteperdrix]

Philippe Gerum wrote:
> On Thu, 2009-05-14 at 12:20 +0200, Jan Kiszka wrote:
>> Philippe Gerum wrote:
>>> On Wed, 2009-05-13 at 18:10 +0200, Jan Kiszka wrote:
>>>> Philippe Gerum wrote:
>>>>> On Wed, 2009-05-13 at 17:28 +0200, Jan Kiszka wrote:
>>>>>> Philippe Gerum wrote:
>>>>>>> On Wed, 2009-05-13 at 15:18 +0200, Jan Kiszka wrote:
>>>>>>>> Gilles Chanteperdrix wrote:
>>>>>>>>> Jan Kiszka wrote:
>>>>>>>>>> Hi Gilles,
>>>>>>>>>>
>>>>>>>>>> I'm currently facing a nasty effect with switchtest over latest git head
>>>>>>>>>> (only tested this so far): running it inside my test VM (ie. with
>>>>>>>>>> frequent excessive latencies) I get a stalled Linux timer IRQ quite
>>>>>>>>>> quickly. System is otherwise still responsive, Xenomai timers are still
>>>>>>>>>> being delivered, other Linux IRQs too. switchtest complained about
>>>>>>>>>>
>>>>>>>>>>     "Warning: Linux is compiled to use FPU in kernel-space."
>>>>>>>>>>
>>>>>>>>>> when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
>>>>>>>>>> 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
>>>>>>>>>> same effect.
>>>>>>>>>>
>>>>>>>>>> Seen this before?
>>>>>>>>> The warning about Linux being compiled to use FPU in kernel-space means
>>>>>>>>> that you enabled soft RAID or compiled for K7, Geode, or any other
>>>>>>>> RAID is on (ordinary server config).
>>>>>>>>
>>>>>>>>> configuration using 3DNow for such simple operations as memcpy. It is
>>>>>>>>> harmless, it simply means that switchtest can not use fpu in kernel-space.
>>>>>>>>>
>>>>>>>>> The bug you have is probably the same as the one described here, which I
>>>>>>>>> am able to reproduce on my atom:
>>>>>>>>> https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html
>>>>>>>>>
>>>>>>>>> Unfortunately, I for one am working on ARM issues and am not available
>>>>>>>>> to debug x86 issues. I think Philippe is busy too...
>>>>>>>> OK, looks like I got the same flu here.
>>>>>>>>
>>>>>>>> Philippe, did you find out any more details in the meantime? Then I'm
>>>>>>>> afraid I have to pick this up.
>>>>>>> No, I did not resume this task yet. Working from the powerpc side of the
>>>>>>> universe here.
>>>>>> Hoho, don't think this rain here over x86 would have never made it down
>>>>>> to ARM or PPC land! ;)
>>>>>>
>>>>>> Martin, could you check if this helps you, too?
>>>>>>
>>>>>> Jan
>>>>>>
>>>>>> (as usual, ready to be pulled from 'for-upstream')
>>>>>>
>>>>>> --------->
>>>>>>
>>>>>> Host IRQs may not only be triggered from non-root domains.
>>>>> Are you sure of this? I can't find any spot where this assumption would
>>>>> be wrong. host_pend() is basically there to relay RT timer ticks and
>>>>> device IRQs, and this only happens on behalf of the pipeline head. At
>>>>> least, this is how rthal_irq_host_pend() should be used in any case. If
>>>>> you did find a spot where this interface is being called from the lower
>>>>> stage, then this is the root bug to fix.
>>>> I haven't studied the I-pipe trace /wrt this in details yet, but I could
>>>> imagine that some shadow task is interrupted in primary mode by the
>>>> timer IRQ and then leaves the handler in secondary mode due to whatever
>>>> events between schedule-out and in at the end of xnintr_clock_handler.
>>>>
>>> You need a thread context to move to secondary, I just can't see how
>>> such scenario would be possible.
>> Here is the trace of events:
>>
>> => Shadow task starts migration to secondary
>> => in xnpod_suspend_thread, nklock is briefly released before
>>    xnpod_schedule
> 
> Which is the root bug. Blame on me; this recent change in -head breaks a
> basic rule a lot of code is based on: a self-suspending thread may not
> be preempted while scheduling out, i.e. suspension and rescheduling must
> be atomically performed. xnshadow_relax() counts on this too.

Actually, I think the idea was mine in the first place... Maybe we can
specify a special flag to xnpod_suspend_thread to ask fo the atomic
suspension (maybe reuse XNATOMIC ?).

-- 
                                                 Gilles.

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

[Philippe Gerum]

On Thu, 2009-05-14 at 14:52 +0200, Gilles Chanteperdrix wrote:
> Philippe Gerum wrote:
> > On Thu, 2009-05-14 at 12:20 +0200, Jan Kiszka wrote:
> >> Philippe Gerum wrote:
> >>> On Wed, 2009-05-13 at 18:10 +0200, Jan Kiszka wrote:
> >>>> Philippe Gerum wrote:
> >>>>> On Wed, 2009-05-13 at 17:28 +0200, Jan Kiszka wrote:
> >>>>>> Philippe Gerum wrote:
> >>>>>>> On Wed, 2009-05-13 at 15:18 +0200, Jan Kiszka wrote:
> >>>>>>>> Gilles Chanteperdrix wrote:
> >>>>>>>>> Jan Kiszka wrote:
> >>>>>>>>>> Hi Gilles,
> >>>>>>>>>>
> >>>>>>>>>> I'm currently facing a nasty effect with switchtest over latest git head
> >>>>>>>>>> (only tested this so far): running it inside my test VM (ie. with
> >>>>>>>>>> frequent excessive latencies) I get a stalled Linux timer IRQ quite
> >>>>>>>>>> quickly. System is otherwise still responsive, Xenomai timers are still
> >>>>>>>>>> being delivered, other Linux IRQs too. switchtest complained about
> >>>>>>>>>>
> >>>>>>>>>>     "Warning: Linux is compiled to use FPU in kernel-space."
> >>>>>>>>>>
> >>>>>>>>>> when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
> >>>>>>>>>> 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
> >>>>>>>>>> same effect.
> >>>>>>>>>>
> >>>>>>>>>> Seen this before?
> >>>>>>>>> The warning about Linux being compiled to use FPU in kernel-space means
> >>>>>>>>> that you enabled soft RAID or compiled for K7, Geode, or any other
> >>>>>>>> RAID is on (ordinary server config).
> >>>>>>>>
> >>>>>>>>> configuration using 3DNow for such simple operations as memcpy. It is
> >>>>>>>>> harmless, it simply means that switchtest can not use fpu in kernel-space.
> >>>>>>>>>
> >>>>>>>>> The bug you have is probably the same as the one described here, which I
> >>>>>>>>> am able to reproduce on my atom:
> >>>>>>>>> https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html
> >>>>>>>>>
> >>>>>>>>> Unfortunately, I for one am working on ARM issues and am not available
> >>>>>>>>> to debug x86 issues. I think Philippe is busy too...
> >>>>>>>> OK, looks like I got the same flu here.
> >>>>>>>>
> >>>>>>>> Philippe, did you find out any more details in the meantime? Then I'm
> >>>>>>>> afraid I have to pick this up.
> >>>>>>> No, I did not resume this task yet. Working from the powerpc side of the
> >>>>>>> universe here.
> >>>>>> Hoho, don't think this rain here over x86 would have never made it down
> >>>>>> to ARM or PPC land! ;)
> >>>>>>
> >>>>>> Martin, could you check if this helps you, too?
> >>>>>>
> >>>>>> Jan
> >>>>>>
> >>>>>> (as usual, ready to be pulled from 'for-upstream')
> >>>>>>
> >>>>>> --------->
> >>>>>>
> >>>>>> Host IRQs may not only be triggered from non-root domains.
> >>>>> Are you sure of this? I can't find any spot where this assumption would
> >>>>> be wrong. host_pend() is basically there to relay RT timer ticks and
> >>>>> device IRQs, and this only happens on behalf of the pipeline head. At
> >>>>> least, this is how rthal_irq_host_pend() should be used in any case. If
> >>>>> you did find a spot where this interface is being called from the lower
> >>>>> stage, then this is the root bug to fix.
> >>>> I haven't studied the I-pipe trace /wrt this in details yet, but I could
> >>>> imagine that some shadow task is interrupted in primary mode by the
> >>>> timer IRQ and then leaves the handler in secondary mode due to whatever
> >>>> events between schedule-out and in at the end of xnintr_clock_handler.
> >>>>
> >>> You need a thread context to move to secondary, I just can't see how
> >>> such scenario would be possible.
> >> Here is the trace of events:
> >>
> >> => Shadow task starts migration to secondary
> >> => in xnpod_suspend_thread, nklock is briefly released before
> >>    xnpod_schedule
> > 
> > Which is the root bug. Blame on me; this recent change in -head breaks a
> > basic rule a lot of code is based on: a self-suspending thread may not
> > be preempted while scheduling out, i.e. suspension and rescheduling must
> > be atomically performed. xnshadow_relax() counts on this too.
> 
> Actually, I think the idea was mine in the first place... Maybe we can
> specify a special flag to xnpod_suspend_thread to ask fo the atomic
> suspension (maybe reuse XNATOMIC ?).
> 

I don't think so. We really need the basic assumption to hold in any
case, because this is expected by most of the callers, and this
micro-optimization is not worth the risk of introducing a race if
misused.

-- 
Philippe.

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

[Gilles Chanteperdrix]

Philippe Gerum wrote:
> On Thu, 2009-05-14 at 14:52 +0200, Gilles Chanteperdrix wrote:
>> Philippe Gerum wrote:
>>> On Thu, 2009-05-14 at 12:20 +0200, Jan Kiszka wrote:
>>>> Philippe Gerum wrote:
>>>>> On Wed, 2009-05-13 at 18:10 +0200, Jan Kiszka wrote:
>>>>>> Philippe Gerum wrote:
>>>>>>> On Wed, 2009-05-13 at 17:28 +0200, Jan Kiszka wrote:
>>>>>>>> Philippe Gerum wrote:
>>>>>>>>> On Wed, 2009-05-13 at 15:18 +0200, Jan Kiszka wrote:
>>>>>>>>>> Gilles Chanteperdrix wrote:
>>>>>>>>>>> Jan Kiszka wrote:
>>>>>>>>>>>> Hi Gilles,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm currently facing a nasty effect with switchtest over latest git head
>>>>>>>>>>>> (only tested this so far): running it inside my test VM (ie. with
>>>>>>>>>>>> frequent excessive latencies) I get a stalled Linux timer IRQ quite
>>>>>>>>>>>> quickly. System is otherwise still responsive, Xenomai timers are still
>>>>>>>>>>>> being delivered, other Linux IRQs too. switchtest complained about
>>>>>>>>>>>>
>>>>>>>>>>>>     "Warning: Linux is compiled to use FPU in kernel-space."
>>>>>>>>>>>>
>>>>>>>>>>>> when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
>>>>>>>>>>>> 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
>>>>>>>>>>>> same effect.
>>>>>>>>>>>>
>>>>>>>>>>>> Seen this before?
>>>>>>>>>>> The warning about Linux being compiled to use FPU in kernel-space means
>>>>>>>>>>> that you enabled soft RAID or compiled for K7, Geode, or any other
>>>>>>>>>> RAID is on (ordinary server config).
>>>>>>>>>>
>>>>>>>>>>> configuration using 3DNow for such simple operations as memcpy. It is
>>>>>>>>>>> harmless, it simply means that switchtest can not use fpu in kernel-space.
>>>>>>>>>>>
>>>>>>>>>>> The bug you have is probably the same as the one described here, which I
>>>>>>>>>>> am able to reproduce on my atom:
>>>>>>>>>>> https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html
>>>>>>>>>>>
>>>>>>>>>>> Unfortunately, I for one am working on ARM issues and am not available
>>>>>>>>>>> to debug x86 issues. I think Philippe is busy too...
>>>>>>>>>> OK, looks like I got the same flu here.
>>>>>>>>>>
>>>>>>>>>> Philippe, did you find out any more details in the meantime? Then I'm
>>>>>>>>>> afraid I have to pick this up.
>>>>>>>>> No, I did not resume this task yet. Working from the powerpc side of the
>>>>>>>>> universe here.
>>>>>>>> Hoho, don't think this rain here over x86 would have never made it down
>>>>>>>> to ARM or PPC land! ;)
>>>>>>>>
>>>>>>>> Martin, could you check if this helps you, too?
>>>>>>>>
>>>>>>>> Jan
>>>>>>>>
>>>>>>>> (as usual, ready to be pulled from 'for-upstream')
>>>>>>>>
>>>>>>>> --------->
>>>>>>>>
>>>>>>>> Host IRQs may not only be triggered from non-root domains.
>>>>>>> Are you sure of this? I can't find any spot where this assumption would
>>>>>>> be wrong. host_pend() is basically there to relay RT timer ticks and
>>>>>>> device IRQs, and this only happens on behalf of the pipeline head. At
>>>>>>> least, this is how rthal_irq_host_pend() should be used in any case. If
>>>>>>> you did find a spot where this interface is being called from the lower
>>>>>>> stage, then this is the root bug to fix.
>>>>>> I haven't studied the I-pipe trace /wrt this in details yet, but I could
>>>>>> imagine that some shadow task is interrupted in primary mode by the
>>>>>> timer IRQ and then leaves the handler in secondary mode due to whatever
>>>>>> events between schedule-out and in at the end of xnintr_clock_handler.
>>>>>>
>>>>> You need a thread context to move to secondary, I just can't see how
>>>>> such scenario would be possible.
>>>> Here is the trace of events:
>>>>
>>>> => Shadow task starts migration to secondary
>>>> => in xnpod_suspend_thread, nklock is briefly released before
>>>>    xnpod_schedule
>>> Which is the root bug. Blame on me; this recent change in -head breaks a
>>> basic rule a lot of code is based on: a self-suspending thread may not
>>> be preempted while scheduling out, i.e. suspension and rescheduling must
>>> be atomically performed. xnshadow_relax() counts on this too.
>> Actually, I think the idea was mine in the first place... Maybe we can
>> specify a special flag to xnpod_suspend_thread to ask fo the atomic
>> suspension (maybe reuse XNATOMIC ?).
>>
> 
> I don't think so. We really need the basic assumption to hold in any
> case, because this is expected by most of the callers, and this
> micro-optimization is not worth the risk of introducing a race if
> misused.

Well, I tend to disagree. The assumption that the thread is suspended
from the point of view of the scheduler still holds even when the nklock
is released, and it is what callers like rt_cond_wait are expecting. The
assumptions of xnshadow_relax do not seem to me like a common assumption.

To go further, maybe forwarding the tick in xnintr_clock_handler
epilogue is wrong in the first place, precisely because the current
thread where xnintr_clock_handler happens may be scheduled out for a
long time by the clock handler itself.

-- 
                                                 Gilles.

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

[Philippe Gerum]

On Thu, 2009-05-14 at 15:10 +0200, Gilles Chanteperdrix wrote: 
> Philippe Gerum wrote:
> > On Thu, 2009-05-14 at 14:52 +0200, Gilles Chanteperdrix wrote:
> >> Philippe Gerum wrote:
> >>> On Thu, 2009-05-14 at 12:20 +0200, Jan Kiszka wrote:
> >>>> Philippe Gerum wrote:
> >>>>> On Wed, 2009-05-13 at 18:10 +0200, Jan Kiszka wrote:
> >>>>>> Philippe Gerum wrote:
> >>>>>>> On Wed, 2009-05-13 at 17:28 +0200, Jan Kiszka wrote:
> >>>>>>>> Philippe Gerum wrote:
> >>>>>>>>> On Wed, 2009-05-13 at 15:18 +0200, Jan Kiszka wrote:
> >>>>>>>>>> Gilles Chanteperdrix wrote:
> >>>>>>>>>>> Jan Kiszka wrote:
> >>>>>>>>>>>> Hi Gilles,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'm currently facing a nasty effect with switchtest over latest git head
> >>>>>>>>>>>> (only tested this so far): running it inside my test VM (ie. with
> >>>>>>>>>>>> frequent excessive latencies) I get a stalled Linux timer IRQ quite
> >>>>>>>>>>>> quickly. System is otherwise still responsive, Xenomai timers are still
> >>>>>>>>>>>> being delivered, other Linux IRQs too. switchtest complained about
> >>>>>>>>>>>>
> >>>>>>>>>>>>     "Warning: Linux is compiled to use FPU in kernel-space."
> >>>>>>>>>>>>
> >>>>>>>>>>>> when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
> >>>>>>>>>>>> 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
> >>>>>>>>>>>> same effect.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Seen this before?
> >>>>>>>>>>> The warning about Linux being compiled to use FPU in kernel-space means
> >>>>>>>>>>> that you enabled soft RAID or compiled for K7, Geode, or any other
> >>>>>>>>>> RAID is on (ordinary server config).
> >>>>>>>>>>
> >>>>>>>>>>> configuration using 3DNow for such simple operations as memcpy. It is
> >>>>>>>>>>> harmless, it simply means that switchtest can not use fpu in kernel-space.
> >>>>>>>>>>>
> >>>>>>>>>>> The bug you have is probably the same as the one described here, which I
> >>>>>>>>>>> am able to reproduce on my atom:
> >>>>>>>>>>> https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html
> >>>>>>>>>>>
> >>>>>>>>>>> Unfortunately, I for one am working on ARM issues and am not available
> >>>>>>>>>>> to debug x86 issues. I think Philippe is busy too...
> >>>>>>>>>> OK, looks like I got the same flu here.
> >>>>>>>>>>
> >>>>>>>>>> Philippe, did you find out any more details in the meantime? Then I'm
> >>>>>>>>>> afraid I have to pick this up.
> >>>>>>>>> No, I did not resume this task yet. Working from the powerpc side of the
> >>>>>>>>> universe here.
> >>>>>>>> Hoho, don't think this rain here over x86 would have never made it down
> >>>>>>>> to ARM or PPC land! ;)
> >>>>>>>>
> >>>>>>>> Martin, could you check if this helps you, too?
> >>>>>>>>
> >>>>>>>> Jan
> >>>>>>>>
> >>>>>>>> (as usual, ready to be pulled from 'for-upstream')
> >>>>>>>>
> >>>>>>>> --------->
> >>>>>>>>
> >>>>>>>> Host IRQs may not only be triggered from non-root domains.
> >>>>>>> Are you sure of this? I can't find any spot where this assumption would
> >>>>>>> be wrong. host_pend() is basically there to relay RT timer ticks and
> >>>>>>> device IRQs, and this only happens on behalf of the pipeline head. At
> >>>>>>> least, this is how rthal_irq_host_pend() should be used in any case. If
> >>>>>>> you did find a spot where this interface is being called from the lower
> >>>>>>> stage, then this is the root bug to fix.
> >>>>>> I haven't studied the I-pipe trace /wrt this in details yet, but I could
> >>>>>> imagine that some shadow task is interrupted in primary mode by the
> >>>>>> timer IRQ and then leaves the handler in secondary mode due to whatever
> >>>>>> events between schedule-out and in at the end of xnintr_clock_handler.
> >>>>>>
> >>>>> You need a thread context to move to secondary, I just can't see how
> >>>>> such scenario would be possible.
> >>>> Here is the trace of events:
> >>>>
> >>>> => Shadow task starts migration to secondary
> >>>> => in xnpod_suspend_thread, nklock is briefly released before
> >>>>    xnpod_schedule
> >>> Which is the root bug. Blame on me; this recent change in -head breaks a
> >>> basic rule a lot of code is based on: a self-suspending thread may not
> >>> be preempted while scheduling out, i.e. suspension and rescheduling must
> >>> be atomically performed. xnshadow_relax() counts on this too.
> >> Actually, I think the idea was mine in the first place... Maybe we can
> >> specify a special flag to xnpod_suspend_thread to ask fo the atomic
> >> suspension (maybe reuse XNATOMIC ?).
> >>
> > 
> > I don't think so. We really need the basic assumption to hold in any
> > case, because this is expected by most of the callers, and this
> > micro-optimization is not worth the risk of introducing a race if
> > misused.
> 
> Well, I tend to disagree. The assumption that the thread is suspended
> from the point of view of the scheduler still holds even when the nklock
> is released, and it is what callers like rt_cond_wait are expecting. The
> assumptions of xnshadow_relax do not seem to me like a common assumption.
> 

The assumption is that the thread has been suspended _and_ scheduled out
atomically, not only put in a suspended thread, which is quite different
when considered from an interrupt context. I'm worried by the fact that
re-enabling interrupts in the middle of this critical transition breaks
the unspoken rule that sched->curr may not be seen as bearing any block
bit in its status word from anywhere in the code executed from the local
CPU but xnpod_suspend_thread().

Another issue may arise in the SMP case, where xnpod_suspend_thread()
would block a thread running on a remote CPU; in theory, re-enabling
interrupts before the IPIs are sent from xnpod_schedule() - to kick the
remote resched procedure - may introduce an undefined delay due to local
interrupts being processed, albeit some runnable thread may be waiting
for the CPU to be released on the remote CPU. We would then end up with
a local scheduling artefact leaking remotely. I don't quite like this,
in fact.

Admittedly, we could plug each and every subtle hole we find like this
one, introducing XNATOMIC to fix the migration case as well, but my
point is that it may not be worth opening pandora's box just for that
micro-optimization.

> To go further, maybe forwarding the tick in xnintr_clock_handler
> epilogue is wrong in the first place, precisely because the current
> thread where xnintr_clock_handler happens may be scheduled out for a
> long time by the clock handler itself.
> 

It was mostly ok in the early days before RPI came in and with legacy
kernels, since the root thread would always be the last to be scheduled;
it is now wrong because:
- RPI may boost the root thread, so we could pend the timer IRQ long
after the root domain has regained control.
- Linux tickless timing requires that no tick is lost due to the current
IRQ context not returning from xnpod_schedule(), such as when calling
xnpod_delete_thread() on behalf of a kernel-based RT thread, previously
preempted by a timer IRQ. The same goes if xnpod_suspend_thread() is
called upon this thread as well.

So in short, yes, you are definitely right. We would be better off
moving this code close to the deferred tick handling instead, since we
know for sure that the root thread is about to be scheduled in at that
point. I just committed a fix for -head; this is a good candidate for a
backport to 2.4.x.

-- 
Philippe.