i915 GPU woes

i915 GPU woes

[Richard Weinberger]

Hi!

On an Intel(R) Celeron(R) J6413 4-core system, I've observed the
following symptoms:

As soon as an X11 application redraws frequently, real-time
performance suffers. In such situations, the i915 IRQ takes between
200 and 1500 microseconds, seemingly blocking the real-time thread for
an equivalent amount of time. Since the i915 IRQ is a) on the Linux
side and b) pinned to a different core than the real-time thread, it
does not seem to be a software issue.

With extensive tracing, I've noticed that both the i915 IRQ and my
real-time thread stall when accessing memory-mapped IO (the real-time
application drivers an Ethernet card in userspace).

It appears that the GPU is doing something that dramatically slows
down MMIO access across all CPU cores. With graphics disabled, the
application runs smoothly.

The following i915 module parameters are already set:
i915.disable_power_well=0 i915.enable_dc=0 i915.enable_guc=2
i915.enable_rc6=0

I'm currently using Xenomai 3.2.4 on Linux kernel version 6.1, though
I don't think this detail is very significant.

Does anyone have suggestions on what else to try?

-- 
Thanks,
//richard

RE: i915 GPU woes

[Chen, Hongzhan]

>-----Original Message-----
>From: Richard Weinberger <richard.weinberger@gmail.com>
>Sent: Monday, April 15, 2024 4:54 AM
>To: Xenomai <xenomai@lists.linux.dev>
>Subject: i915 GPU woes
>
>Hi!
>
>On an Intel(R) Celeron(R) J6413 4-core system, I've observed the
>following symptoms:
>
>As soon as an X11 application redraws frequently, real-time
>performance suffers. In such situations, the i915 IRQ takes between
>200 and 1500 microseconds, seemingly blocking the real-time thread for
>an equivalent amount of time. Since the i915 IRQ is a) on the Linux
>side and b) pinned to a different core than the real-time thread, it
>does not seem to be a software issue.
>
>With extensive tracing, I've noticed that both the i915 IRQ and my
>real-time thread stall when accessing memory-mapped IO (the real-time
>application drivers an Ethernet card in userspace).
>
>It appears that the GPU is doing something that dramatically slows
>down MMIO access across all CPU cores. With graphics disabled, the
>application runs smoothly.
>
>The following i915 module parameters are already set:
>i915.disable_power_well=0 i915.enable_dc=0 i915.enable_guc=2
>i915.enable_rc6=0
>
>I'm currently using Xenomai 3.2.4 on Linux kernel version 6.1, though
>I don't think this detail is very significant.
>
>Does anyone have suggestions on what else to try?

Would you like to try to lock GPU frequency to minimum?
        frequency=200
        echo $frequency > /sys/class/drm/card0/gt_boost_freq_mhz
        echo $frequency > /sys/class/drm/card0/gt_max_freq_mhz
        echo $frequency > /sys/class/drm/card0/gt_min_freq_mhz
        echo $frequency > /sys/class/drm/card0/gt_idle_freq_mhz

Regards

Hongzhan Chen
>
>--
>Thanks,
>//richard

RE: i915 GPU woes

[Chen, Hongzhan]

>-----Original Message-----
>From: Chen, Hongzhan
>Sent: Monday, April 15, 2024 12:28 PM
>To: Richard Weinberger <richard.weinberger@gmail.com>; Xenomai
><xenomai@lists.linux.dev>
>Subject: RE: i915 GPU woes
>
>
>
>>-----Original Message-----
>>From: Richard Weinberger <richard.weinberger@gmail.com>
>>Sent: Monday, April 15, 2024 4:54 AM
>>To: Xenomai <xenomai@lists.linux.dev>
>>Subject: i915 GPU woes
>>
>>Hi!
>>
>>On an Intel(R) Celeron(R) J6413 4-core system, I've observed the
>>following symptoms:
>>
>>As soon as an X11 application redraws frequently, real-time
>>performance suffers. In such situations, the i915 IRQ takes between
>>200 and 1500 microseconds, seemingly blocking the real-time thread for
>>an equivalent amount of time. Since the i915 IRQ is a) on the Linux
>>side and b) pinned to a different core than the real-time thread, it
>>does not seem to be a software issue.
>>
>>With extensive tracing, I've noticed that both the i915 IRQ and my
>>real-time thread stall when accessing memory-mapped IO (the real-time
>>application drivers an Ethernet card in userspace).
>>
>>It appears that the GPU is doing something that dramatically slows
>>down MMIO access across all CPU cores. With graphics disabled, the
>>application runs smoothly.
>>
>>The following i915 module parameters are already set:
>>i915.disable_power_well=0 i915.enable_dc=0 i915.enable_guc=2
>>i915.enable_rc6=0
>>
>>I'm currently using Xenomai 3.2.4 on Linux kernel version 6.1, though
>>I don't think this detail is very significant.
>>
>>Does anyone have suggestions on what else to try?
>
>Would you like to try to lock GPU frequency to minimum?
>        frequency=200
>        echo $frequency > /sys/class/drm/card0/gt_boost_freq_mhz
>        echo $frequency > /sys/class/drm/card0/gt_max_freq_mhz
>        echo $frequency > /sys/class/drm/card0/gt_min_freq_mhz
>        echo $frequency > /sys/class/drm/card0/gt_idle_freq_mhz
>
>Regards

In addition, you may like to try to decrease DVMT Pre-Allocated mem configuration for graphics in BIOS  to a relatively small size like 64M concurrently.

Regards

Hongzhan Chen
>
>Hongzhan Chen
>>
>>--
>>Thanks,
>>//richard

Re: i915 GPU woes

[Richard Weinberger]

Hello Hongzhan Chen,

On Mon, Apr 15, 2024 at 6:38 AM Chen, Hongzhan <hongzhan.chen@intel.com> wrote:
> >Would you like to try to lock GPU frequency to minimum?
> >        frequency=200
> >        echo $frequency > /sys/class/drm/card0/gt_boost_freq_mhz
> >        echo $frequency > /sys/class/drm/card0/gt_max_freq_mhz
> >        echo $frequency > /sys/class/drm/card0/gt_min_freq_mhz
> >        echo $frequency > /sys/class/drm/card0/gt_idle_freq_mhz

My system does not have gt_idle_freq_mhz, but I set all others to 200.
Sadly, the situation did not improve.

> In addition, you may like to try to decrease DVMT Pre-Allocated mem configuration for graphics in BIOS  to a relatively small size like 64M concurrently.

My BIOS has no such setting. Looks I have to talk to the vendor.

What do you think about the i915 module parameters I have configured?

RE: i915 GPU woes

[Chen, Hongzhan]

>-----Original Message-----
>From: Richard Weinberger <richard.weinberger@gmail.com>
>Sent: Tuesday, April 16, 2024 5:46 AM
>To: Chen, Hongzhan <hongzhan.chen@intel.com>
>Cc: Xenomai <xenomai@lists.linux.dev>
>Subject: Re: i915 GPU woes
>
>Hello Hongzhan Chen,
>
>On Mon, Apr 15, 2024 at 6:38 AM Chen, Hongzhan
><hongzhan.chen@intel.com> wrote:
>> >Would you like to try to lock GPU frequency to minimum?
>> >        frequency=200
>> >        echo $frequency > /sys/class/drm/card0/gt_boost_freq_mhz
>> >        echo $frequency > /sys/class/drm/card0/gt_max_freq_mhz
>> >        echo $frequency > /sys/class/drm/card0/gt_min_freq_mhz
>> >        echo $frequency > /sys/class/drm/card0/gt_idle_freq_mhz

>My system does not have gt_idle_freq_mhz, but I set all others to 200.

I also found that that there is no gt_idle_freq_mhz on my 6412 platform. 
Sorry , I do not have 6413 on my side and have not verified it.

>Sadly, the situation did not improve.
>
>> In addition, you may like to try to decrease DVMT Pre-Allocated mem
>configuration for graphics in BIOS  to a relatively small size like 64M
>concurrently.
>
>My BIOS has no such setting. Looks I have to talk to the vendor.

On my 6412 board, I have such options in bios to do configuration. I do not know
if there is anything different from 6413.

>
>What do you think about the i915 module parameters I have configured?

You already set all the parameters we suggest to set for i915 like " i915.disable_power_well=0 
i915.enable_dc=0 i915.enable_rc6=0" which is to prevent integrated graphics from changing power states.
If you already narrow down to graphics MMIO related, I do not know if it may help improve performance to limit 
mem used by graphics via DVMT but there is no such options in your bios.

Regards

Hongzhan Chen

Re: i915 GPU woes

[Richard Weinberger]

On Tue, Apr 16, 2024 at 4:02 AM Chen, Hongzhan <hongzhan.chen@intel.com> wrote:
> >What do you think about the i915 module parameters I have configured?
>
> You already set all the parameters we suggest to set for i915 like " i915.disable_power_well=0
> i915.enable_dc=0 i915.enable_rc6=0" which is to prevent integrated graphics from changing power states.
> If you already narrow down to graphics MMIO related, I do not know if it may help improve performance to limit
> mem used by graphics via DVMT but there is no such options in your bios.

The riddle is more or less solved.

It turned(sic!) out that the problem occurs only on systems with a
rotated screen.
Plus, only when the Intel xorg driver is in use.
By switching to xorg modsetting the problem is gone. Looks like the
modesetting driver
causes the GPU to do the screen rotation differently and the GPU no
longer blocks other
cores.
Further I had to disable frame buffer compression (i915.enable_fbc=0)
to get decent results.

-- 
Thanks,
//richard