AW: Large latencies with 4.4.1-rt6

AW: Large latencies with 4.4.1-rt6

[Martin.Wirth]

> -----Ursprüngliche Nachricht-----
> Von: Thomas Gleixner [mailto:tglx@linutronix.de]
> Gesendet: Dienstag, 1. März 2016 11:08
> An: Wirth, Martin
> Cc: bigeasy@linutronix.de
> Betreff: Re: AW: AW: AW: Large latencies with 4.4.1-rt6
> 
> 
> Can you try to gather a trace:
> 
> cyclictest -Sp99 -m -i200 -f -b500
> 
> You might need to up the interval a bit, but that should give us the reason for
> this hickup.
>

Exactly this is running now since about 1,5 h and no trigger so far - seems to be 
some sort of Heisenbug.

This is with 4.4.3-rt9 which also yesterday needed about 40 Minutes to trigger 
without tracing enabled. If this gives no output within the next 2 hours I will try
an older version (4.4.1-rt5)

Martin

AW: Large latencies with 4.4.1-rt6

[Martin.Wirth]

Hi Josh,
> -----Ursprüngliche Nachricht-----
> Von: Josh Cartwright [mailto:joshc@ni.com]
> Gesendet: Donnerstag, 3. März 2016 16:42
> An: Wirth, Martin
> Cc: tglx@linutronix.de; bigeasy@linutronix.de; linux-rt-
> users@vger.kernel.org
> Betreff: Re: AW: AW: Large latencies with 4.4.1-rt6
> 
> > > cyclictest -Sp99 -m -i200 -f -b500
> 
> Cyclictest should have mentioned which CPU was triggering the break, the
> most valuable traces are from that CPU.
> 

This one came from CPU 0

 
> Do you have more complete traces of what's going on on the other CPUs as
> well?  As far as I can see here, CPU0 is handling a couple interrupts, which
> trigger remote wakeups of irqthreads running on the other CPUs..nothing
> unusual?

To me it looked like the CPU kept IRQs disabled over the 100 µs, but it seems 
that this is a misinterpretation. 

The complete trace, including all CPUs, is about 12 MB (600 KB compressed) and
therefore  much too large for the mailing list. If anyone is interested to get the
complete file I will send it to his e-mail only.

Thanks,

Martin 
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: AW: Large latencies with 4.4.1-rt6

[Sebastian Andrzej Siewior]

On 02/25/2016 05:08 PM, Martin.Wirth@dlr.de wrote:
> Hi Sebastian

Hi Martin,

> I will have access to the machine for longer testing next Monday again. But I remember that v4.4.1-rt5 showed the same long latencies.
> Will try v4.4-rt3 and v4.4-rt2 beginning of next week.

If this would be your first try I would suggest to run hwlatdetect in
case BIOS is calling for attention. However since you don't see this in
v4.1 is looks unlikely. It also might be help to have a log to see what
the other CPUs are doing around that time. Are they busy doing
something or do they pause as well. Also do try some fancy PM things
(but then again why now and not in v4.1 as well).

> Martin

Sebastian

AW: Large latencies with 4.4.1-rt6

[Martin.Wirth]

Hi Sebastian

> -----Ursprüngliche Nachricht-----
> Von: Sebastian Andrzej Siewior [mailto:bigeasy@linutronix.de]
> Gesendet: Donnerstag, 25. Februar 2016 16:30
> An: Wirth, Martin
> Cc: linux-rt-users@vger.kernel.org
> Betreff: Re: Large latencies with 4.4.1-rt6
> 
> You could try the v4.4 line as of:
>  v4.4-rt2
>  v4.4-rt3
>  v4.4.1-rt5
> 

I will have access to the machine for longer testing next Monday again. But I remember that v4.4.1-rt5 showed the same long latencies.
Will try v4.4-rt3 and v4.4-rt2 beginning of next week.

Martin

Re: AW: Large latencies with 4.4.1-rt6

[Tim Sander]

Hi Martin
Am Donnerstag, 25. Februar 2016, 12:47:31 schrieb Martin.Wirth@dlr.de:
> > I have also seen a latency increase with 4.4.1-rtX releases. I have the
> > impression that some stable kernel updates made it worse. Care to test
> > 4.4-
> > rt3 or 4.4-rt4? Had not the chance to pin them down by now.
> 
> The worst case cyclictest latencies (running over serveral hours) with pre
> 4.4.x-rt versions were below 10 µs for an idle system and below 23 µs for a
> kernel compile with make -j4. Since this is a machine with is in lab-use
> during the day I hesitate a bit to try too early 4.4.x version which might
> be instable enough to eat my disk.
Well the 4.4 releases have been pretty stable for me on ARM but your milage 
may by different...

Best regards
Tim
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

AW: Large latencies with 4.4.1-rt6

[Martin.Wirth]

Hi Tim

> -----Ursprüngliche Nachricht-----
> Von: Tim Sander [mailto:tim@krieglstein.org]
> Gesendet: Donnerstag, 25. Februar 2016 13:34
> An: Wirth, Martin
> Cc: linux-rt-users@vger.kernel.org
> Betreff: Re: Large latencies with 4.4.1-rt6
> 
> Hi Martin
> 
> Am Freitag, 19. Februar 2016, 17:02:20 schrieb Martin.Wirth@dlr.de:
> > I today switched from 4.1.15-rt17 to 4.4.1-rt6.
> > Now I'm seeing large latencies of more than 1 ms.
> > They occur sporadic ( every few minutes) and show up for example
> > running
> It would be interesting to know the platform the rt kernel is running on?

This is an intel  i7-4700EQ CPU @ 2.40GHz machine  on an ADLINK cPCI-6530 board.

> I have also seen a latency increase with 4.4.1-rtX releases. I have the
> impression that some stable kernel updates made it worse. Care to test 4.4-
> rt3 or 4.4-rt4? Had not the chance to pin them down by now.
> 
The worst case cyclictest latencies (running over serveral hours) with pre 4.4.x-rt versions 
were below 10 µs for an idle system and below 23 µs for a kernel compile with make -j4.
Since this is a machine with is in lab-use during the day I hesitate a bit to try too 
early 4.4.x version which might be instable enough to eat my disk.

Best regards,

Martin
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html