Re: [Xenomai] x86 Linux with xenomai and i915 graphics stall

Re: [Xenomai] x86 Linux with xenomai and i915 graphics stall

[Daniel M. Drucker, Ph.D.]

Hi Gilles et al.,

> That is unusual, a typical latency on x86 is under 50us (and this is a
> conservative bound). In order to investigate this, enable the I-pipe
> tracer in the kernel configuration and run latency -f. Also set the
> number of backtrace points to a large value, say 1000 or 10000 and post
> the trace here.

Here is the trace before running glxgears. The worst latency is 32us.

https://gist.github.com/raw/4599095/203ce77b2bb8e1b2c797302d33ce1a7b0a18fbac/gistfile1

And here is the trace while running glxgears. The worst latency is 724us.

https://gist.github.com/raw/4599099/27054c46f246999375346ed942f76d0757c4fa0a/gistfile1

Thanks,
Daniel and Andy

Re: [Xenomai] x86 Linux with xenomai and i915 graphics stall

[Gilles Chanteperdrix]

On 01/22/2013 11:23 PM, Daniel M. Drucker, Ph.D. wrote:

> Hi Gilles et al.,
> 
>> That is unusual, a typical latency on x86 is under 50us (and this is a
>> conservative bound). In order to investigate this, enable the I-pipe
>> tracer in the kernel configuration and run latency -f. Also set the
>> number of backtrace points to a large value, say 1000 or 10000 and post
>> the trace here.
> 
> Here is the trace before running glxgears. The worst latency is 32us.
> 
> https://gist.github.com/raw/4599095/203ce77b2bb8e1b2c797302d33ce1a7b0a18fbac/gistfile1
> 
> And here is the trace while running glxgears. The worst latency is 724us.
> 
> https://gist.github.com/raw/4599099/27054c46f246999375346ed942f76d0757c4fa0a/gistfile1


And you are sure that it is not an SMI issue? Could you post your kernel
configuration?

-- 
                                                                Gilles.

Re: [Xenomai] x86 Linux with xenomai and i915 graphics stall

[Daniel M. Drucker, Ph.D.]

>
>
> And you are sure that it is not an SMI issue? Could you post your kernel
> configuration?
>
>
Here it is:
https://gist.github.com/raw/4599242/0eea44377371e4d7213eea28cd33823950afeaca/gistfile1

Daniel

Re: [Xenomai] x86 Linux with xenomai and i915 graphics stall

[Gilles Chanteperdrix]

On 01/22/2013 11:29 PM, Daniel M. Drucker, Ph.D. wrote:

> 
>     And you are sure that it is not an SMI issue? Could you post your kernel
>     configuration?
> 
> 
> Here it
> is: https://gist.github.com/raw/4599242/0eea44377371e4d7213eea28cd33823950afeaca/gistfile1


You can try disabling CONFIG_PM_RUNTIME

What could take long in __xnpod_schedule is locking the spinlock, that
could be because another CPU has the lock, in which case you will be
able to catch the culprit by enabling CONFIG_XENO_OPT_DEBUG_XNLOCK

or because the bus is locked for a long time, you could also try
disabling CONFIG_MTRR or CONFIG_X86_PAT.

Finally, what may happen is an NMI, the perf system uses NMIs and
oprofile as well. You can avoid compiling oprofile, but I am not sure
perf can be disabled, but you can comment out the code enabling the NMI
with the following patch:

diff --git a/arch/x86/kernel/cpu/perf_event.c
b/arch/x86/kernel/cpu/perf_event.c
index bb8e034..a26df66 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1324,6 +1324,8 @@ static int __init init_hw_perf_events(void)
 	struct event_constraint *c;
 	int err;

+	return 0;
+	
 	pr_info("Performance Events: ");

 	switch (boot_cpu_data.x86_vendor) {

-- 
                                                                Gilles.

Re: [Xenomai] x86 Linux with xenomai and i915 graphics stall

[Daniel M. Drucker, Ph.D.]

Hi Gilles et al.,

We tried setting processor type more correctly, turning off CONFIG_M686 and
turning on CONFIG_MCORE2, and also turned off CONFIG_X86_GENERIC.

That change has caused our worst latencies to drop to around 200us, but
those are actually rare - we can only [but reliably] trigger that by using
glxgears with a very large (1200x700 or larger or so) window size. (Other
things we tried had no latency-increasing effects - we tried heavy disk and
network activity, playing youtube videos, etc.)''

/proc/ipipe/trace/frozen, without glxgears running: http://ix.io/46I
with glxgears running: http://ix.io/46J

Log of latency: http://ix.io/46L

And.... here is a video of the problem:
http://www.youtube.com/watch?v=l4rYvhqZql8
 (problem starts at about 0:41)

Can you think of anything else we could try? We've tried enabling SMI
workaround, disabling PM_RUNTIME, disabling MTRR, and disabling X86_PAT,
and none of these have solved this issue.

Thanks so much for your help,
Daniel and Andy
Interactive Motion

Re: [Xenomai] x86 Linux with xenomai and i915 graphics stall

[Gilles Chanteperdrix]

On 01/24/2013 11:22 PM, Daniel M. Drucker, Ph.D. wrote:

> Hi Gilles et al.,
> 
> We tried setting processor type more correctly, turning off CONFIG_M686
> and turning on CONFIG_MCORE2, and also turned off CONFIG_X86_GENERIC.
> 
> That change has caused our worst latencies to drop to around 200us, but
> those are actually rare - we can only [but reliably] trigger that by
> using glxgears with a very large (1200x700 or larger or so) window size.
> (Other things we tried had no latency-increasing effects - we tried
> heavy disk and network activity, playing youtube videos, etc.)''
> 
> /proc/ipipe/trace/frozen, without glxgears running: http://ix.io/46I
> with glxgears running: http://ix.io/46J


It would be interesting to know what happens on other cpus at the time
when this happens. Or could you try forcing glxgears to run on the same
CPU as the latency test?

> 
> Log of latency: http://ix.io/46L
> 
> And.... here is a video of the
> problem: http://www.youtube.com/watch?v=l4rYvhqZql8
>  (problem starts at about 0:41)
> 
> Can you think of anything else we could try? We've tried enabling SMI


Yes, but did you run dmesg | grep SMI to see if your chipset is acutally
supported?

Have you tried to turn off graphic acceleration in X window configuration?

-- 
                                                                Gilles.

Re: [Xenomai] x86 Linux with xenomai and i915 graphics stall

[Gilles Chanteperdrix]

On 01/22/2013 08:32 PM, A User wrote:

> When we run latency, we still get about 10 usec on a quiet system.
> When we run glxgears and make the window size  600x600 pixels, and we
> use the mouse to drag the glxgears window quickly around on the
> desktop, the max latency goes to about 250 usec (with gnome-3
> fallback and compiz).
> 
> So, it looks like we are ok for now with this configuration, unless
> that latency is unusual. glxgears in a window that is not being
> dragged has lower latency.


Hi,

I have been able to spend some time on this issue. It seems the time 
gets spent in the "__gen6_gt_force_wake_get" function.

I modified the function as such:

void __gen6_gt_force_wake_get(struct drm_i915_private *dev_priv)
{
	unsigned long flags;
	int count;
	count = 0;
	while (count++ < 50 && (I915_READ_NOTRACE(FORCEWAKE_ACK) & 1))
		udelay(10);

	ipipe_trace_special(50, native_apic_mem_read(APIC_TMCCT));
	I915_WRITE_NOTRACE(FORCEWAKE, 1);
	ipipe_trace_special(51, native_apic_mem_read(APIC_TMCCT));
	POSTING_READ(FORCEWAKE);
	ipipe_trace_special(52, native_apic_mem_read(APIC_TMCCT));

	count = 0;
	while (count++ < 50 && (I915_READ_NOTRACE(FORCEWAKE_ACK) & 1) == 0)
		udelay(10);
}

And I get the following trace:
(...)
:|  # event   tick@-56    -154	  0.036  xntimer_next_local_shot+0x9d (xntimer_tick_aperiodic+0x154)
(...)
:    #func                 -79	  0.103  __gen6_gt_force_wake_get+0x7 [i915] (i915_read32+0x5f [i915])
:    #(0x32)	0x00000093   -79	  0.738  __gen6_gt_force_wake_get+0x56 [i915] (i915_read32+0x5f [i915])
:    #(0x33)	0x0000008e   -78!  66.706  __gen6_gt_force_wake_get+0x79 [i915] (i915_read32+0x5f [i915])
:|   #begin   0x000000ef   -12	  0.257  apic_timer_interrupt+0x6a (__gen6_gt_force_wake_get+0x84 [i915])

The second hexadecimal value in the trace is the value of the APIC 
current counter register. The APIC timer runs on my machine at 6249609Hz.
This means that:
- before I915_WRITE_NOTRACE(FORCEWAKE, 1), there are
0x93 * 1000000 / 6249609 = 23us remaining before the timer tick.
- after I915_WRITE_NOTRACE(FORCEWAKE, 1), which lasted for 67us, there are
0x8e * 1000000 / 6249609 = 22us remaining before the timer tick.

The only explanation I see for this behaviour, is that the APIC timer clock 
stops during the write to the FORCEWAKE register.

I have tried the usual graphic driver workaround:
set "Option NoAccel" in /etc/X11/xorg.conf, it did not work.

However, installing the debian packet xserver-xorg-video-fbdev, and
replacing "Driver "intel"" with "Driver "fbdev"" does the trick for me.

Regards.

-- 
                                                                Gilles.

Re: [Xenomai] x86 Linux with xenomai and i915 graphics stall

[Gilles Chanteperdrix]

On 01/22/2013 08:32 PM, A User wrote:

> When we run latency, we still get about 10 usec on a quiet system.
> When we run glxgears and make the window size  600x600 pixels, and we
> use the mouse to drag the glxgears window quickly around on the
> desktop, the max latency goes to about 250 usec (with gnome-3
> fallback and compiz).


That is unusual, a typical latency on x86 is under 50us (and this is a
conservative bound). In order to investigate this, enable the I-pipe
tracer in the kernel configuration and run latency -f. Also set the
number of backtrace points to a large value, say 1000 or 10000 and post
the trace here.

> The option CONFIG_PCI_MSI should be disabled
> 
> """
> 
> 
> We will keep CONFIG_PCI_MSI turned on for now, as Gilles suggests.


The danger with CONFIG_PCI_MSI is that you can not mask/disable the
interrupt in a real-time interrupt handler, but as long as you do not do
that, you should be safe.

-- 
                                                                Gilles.

Re: [Xenomai] x86 Linux with xenomai and i915 graphics stall

[A User]

Gilles,

Thank you for your help.

Today we compiled our buggy 3.2.21 Linux (i915 stalls) with the I-pipe tracer turned on.  That had the effect of making the GPU stall bug disappear (or made it less frequent - it's intermittent).  So instead of trying to figure that out, we decided to try running:  3.5.7/2.6.2.

Note, when we tried compiling 3.5.3 and 2.6.2 on January 3, we had some problems, which have since been fixed:
http://www.xenomai.org/pipermail/xenomai/2013-January/027187.html

So today Daniel and I compiled a kernel.org Linux 3.5.7 patched with Xenomai 2.6.2 from git.  We used the same Linux .config as our 3.2.21 that was giving us GPU stall problems.  (We ran make xconfig and accepted the changes that it made.)

Our new Linux 3.5.7 with Xenomai 2.6.2 compiles and runs.
The compile gets warning messages that we will ignore for now.
http://www.xenomai.org/pipermail/xenomai/2013-January/027491.html

We compiled two 3.5.7 kernels, with CONFIG_PCI_MSI turned off and on.

It looks like both kernels behave about the same.  We do not see the GPU stalls now.  (The GPU stalls were intermittent before, so we are not 100% sure they are gone, but we have not seen them yet.)


When we run latency, we still get about 10 usec on a quiet system.  When we run glxgears and make the window size  600x600 pixels, and we use the mouse to drag the glxgears window quickly around on the desktop, the max latency goes to about 250 usec (with gnome-3 fallback and compiz).

So, it looks like we are ok for now with this configuration, unless that latency is unusual.
glxgears in a window that is not being dragged has lower latency.


Please note that the x86 config help file:

http://www.xenomai.org/index.php/Configuring_x86_kernels

says:

"""

Use care when referring to old mail posts, archives, and google results. This page is kept up to date with information on what options you might need to tweak.
...

The option CONFIG_PCI_MSI should be disabled

"""


We will keep CONFIG_PCI_MSI turned on for now, as Gilles suggests.


-Andy and Daniel



----- Original Message -----
From: A User <t02446@yahoo.com>
To: "xenomai@xenomai.org" <xenomai@xenomai.org>
Cc: 
Sent: Monday, January 21, 2013 6:04 PM
Subject: x86 Linux with xenomai and i915 graphics stall

I'm trying to run a Linux/Xenomai system with Ubuntu 12.04 LTS
distro including X Window System on an Intel x86 board.

...

Re: [Xenomai] x86 Linux with xenomai and i915 graphics stall

[Gilles Chanteperdrix]

On 01/22/2013 12:04 AM, A User wrote:

> I'm trying to run a Linux/Xenomai system with Ubuntu 12.04 LTS
> distro including X Window System on an Intel x86 board.
> 
> linux: 3.2.21
> xenomai: 2.6.1
> i-pipe Patch: 3.2.21-x86-1
> 
> When I run with CONFIG_PCI_MSI turned off, I get good Xenomai latency,
> but the X Window System is unusable.  With MSI turned on, graphics is
> solid, but latency jumps to about 450 usec when I run glxgears
> (instead of avg 10 usec, max 50 usec).
> 
> I believe that if I want to use X Window system in a recent x86 system
> with motherboard graphics, the most popular choice (the only choice?)
> is i915.
> 
> Do you have any advice on how I can run with i915 and Xenomai with
> this hardware and software?


That is strange, I have tested xenomai on my workstation which is a core
i5 and also has an i915, I have been playing video and running other
opengl apps (including glxgears) and dit not observe high latencies. I
was using Linux 3.5.7 patched with Xenomai 2.6.2, but I guess it should
not make any difference. My core i5 is a "sandy bridge" though, so
should have an intel graphics HD3000 not 4000.

I would suggest enabling the I-pipe tracer in the kernel configuration
in order to find what is taking so long.

Anyway, turning on CONFIG_PCI_MSI is recommended these days. Also, have
you read the troubleshooting guide about SMIs?

If you can not run an accelerated X server, there is always the solution
to revert to disable the acceleration in the server configuration, as is
documented in the troubleshooting guide as well.

-- 
                                                                Gilles.

[Xenomai] x86 Linux with xenomai and i915 graphics stall

[A User]

I'm trying to run a Linux/Xenomai system with Ubuntu 12.04 LTS
distro including X Window System on an Intel x86 board.

linux: 3.2.21
xenomai: 2.6.1
i-pipe Patch: 3.2.21-x86-1

At present, the board I am testing is a DFI-Itox CR100-CRM with an i5
CPU, it has Intel HD Graphics 4000, and runs the Linux i915 driver.

Less interesting to you, but useful for me, it has mini-PCIe,
a handful of rs232s, and mini-ITX form factor.

Following the Xenomai x86 config advice, I turn off CONFIG_PCI_MSI

http://www.xenomai.org/index.php/Configuring_x86_kernels#Message_Signaled_Interrupts_.28MSI_and_MSI-X.29

When I build, install, and boot, the X Window system is sluggish on
about 1/4 to 1/2 the boots.  When it is sluggish, graphics will freeze
until I wiggle the mouse.  When this happens, I get this in dmesg,
several times per minute:

[drm:i915_hangcheck_ring_idle] ERROR Hangcheck timer elapsed... render ring idle [waiting on 1403, at 1403], missed IRQ?

Googling suggests that when this bug appeared in the Intel driver,
the code involved MSI:

https://patchwork.kernel.org/patch/879522/

+ if (IS_GEN6(dev))
+ /* allow blitter user interrupt to generate a MSI write from
+ the ISR */
+ I915_WRITE(GEN6_BLITTER_HWSTAM,
+ 0xffffffff & ~GEN6_BLITTER_USER_INTERRUPT);

There are a few hits about GPU stalling on Linux kernel and and Intel
graphics mailings lists when you google for "i915 msi gen6" or
i915_hangcheck_ring_idle (mostly from mid-2011).

I have tried various other hacks to try to work around this problem,
there are bug fix discussion threads that suggest turning off:

  /sys/module/i915/parameters/semaphores
  or
  /sys/modules/i915/parameters/i915_enable_rc6

but these did not work for me.

It seems that for me, if the bug appears, it appears immediately at
boot time, and if it does not appear, it does not appear again until reboot,
but I am not fully sure of this.

When I run with CONFIG_PCI_MSI turned off, I get good Xenomai latency,
but the X Window System is unusable.  With MSI turned on, graphics is
solid, but latency jumps to about 450 usec when I run glxgears
(instead of avg 10 usec, max 50 usec).

I believe that if I want to use X Window system in a recent x86 system
with motherboard graphics, the most popular choice (the only choice?)
is i915.

Do you have any advice on how I can run with i915 and Xenomai with
this hardware and software?

-Andy