oomd with 6.0-rc1 has ridiculously high memory pressure stats wit

oomd with 6.0-rc1 has ridiculously high memory pressure stats wit

[Chris Murphy]

Hi,

Tracking a downstream bug in Fedora Rawhide testing, where 6.0-rc1 has landed, and we're seeing various GNOME components getting kllled off by systemd-oomd, with the stats showing suspiciously high values:

https://bugzilla.redhat.com/show_bug.cgi?id=2119518

e.g.

Killed /user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org/session.slice/org.gnome.Shell-r28gBBs99rhXz5zEmyOJwQ@public.gmane.org due to memory pressure for /user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org being 27925460729.27% > 50.00% for > 20s with reclaim activity

I'm not seeing evidence of high memory pressure in /proc/pressure though, whereas oomd is reporting really high memory pressure and absolute time for it that makes no sense at all:

>>Sep 09 03:01:05 fedora systemd-oomd[604]:                 Pressure: Avg10: 1255260529528.42 Avg60: 325612.68 Avg300: 757127258245.62 Total: 2month 4w 2d 8h 15min 12s

It's been up for about 2 minutes at this point, not 3 months.

Thanks,

Re: oomd with 6.0-rc1 has ridiculously high memory pressure stats wit

[Tejun Heo]

(cc'ing Johannes for visibility)

On Fri, Aug 19, 2022 at 10:51:27PM -0400, Chris Murphy wrote:
> Hi,
> 
> Tracking a downstream bug in Fedora Rawhide testing, where 6.0-rc1 has landed, and we're seeing various GNOME components getting kllled off by systemd-oomd, with the stats showing suspiciously high values:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=2119518
> 
> e.g.
> 
> Killed /user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org/session.slice/org.gnome.Shell-r28gBBs99rhXz5zEmyOJwQ@public.gmane.org due to memory pressure for /user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org being 27925460729.27% > 50.00% for > 20s with reclaim activity
> 
> I'm not seeing evidence of high memory pressure in /proc/pressure though, whereas oomd is reporting really high memory pressure and absolute time for it that makes no sense at all:
> 
> >>Sep 09 03:01:05 fedora systemd-oomd[604]:                 Pressure: Avg10: 1255260529528.42 Avg60: 325612.68 Avg300: 757127258245.62 Total: 2month 4w 2d 8h 15min 12s
> 
> It's been up for about 2 minutes at this point, not 3 months.
> 
> Thanks,
> 
> 
> --
> Chris Murphy Murphy

-- 
tejun

Re: oomd with 6.0-rc1 has ridiculously high memory pressure stats wit

[Chris Murphy]

Another example,  with 6.0-rc2:

# oomctl
Dry Run: no
Swap Used Limit: 90.00%
Default Memory Pressure Limit: 60.00%
Default Memory Pressure Duration: 20s
System Context:
        Memory: Used: 1.7G Total: 3.8G
        Swap: Used: 0B Total: 3.8G
Swap Monitored CGroups:
        Path: /
                Swap Usage: (see System Context)
Memory Pressure Monitored CGroups:
        Path: /user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org
                Memory Pressure Limit: 50.00%
                Pressure: Avg10: 0.00 Avg60: 0.00 Avg300: 0.10 Total: 140y 3month 18h 24min 20s
                Current Memory Usage: 1.5G
                Memory Min: 250.0M
                Memory Low: 0B
                Pgscan: 0
                Last Pgscan: 0
# cat /sys/fs/cgroup/user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org/memory.pressure
some avg10=0.00 avg60=0.00 avg300=0.18 total=8367757640799118
full avg10=0.00 avg60=0.00 avg300=0.07 total=4426019660641432

# grep -R . /sys/fs/cgroup/user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org/
https://drive.google.com/file/d/1Ro6rKnEx1CCapmO3rz6SDysjP1Bs4_Re/view?usp=sharing 

Actual uptime is ~10 minutes.

-- 
Chris Murphy

Re: oomd with 6.0-rc1 has ridiculously high memory pressure stats wit

[Chris Murphy]

Same VM but a different boot:

Excerpts:

/sys/fs/cgroup/user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org/session.slice/gvfs-goa-volume-monitor.service/io.pressure:some avg10=3031575.41 avg60=56713935870.67 avg300=624837039080.83 total=18446621498826359
/sys/fs/cgroup/user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org/session.slice/gvfs-goa-volume-monitor.service/io.pressure:full avg10=3031575.41 avg60=56713935870.80 avg300=624837039080.99 total=16045481047390973

None of that seems possible.

io is also affected:

/sys/fs/cgroup/user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org/session.slice/org.gnome.SettingsDaemon.Smartcard.service/io.pressure:full avg10=0.00 avg60=0.13 avg300=626490311370.87 total=16045481047397307

# oomctl
# grep -R . /sys/fs/cgroup/user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org/
https://drive.google.com/file/d/1JoUxjQ2ribDvn5jmydCWXJdg0daaNScG/view?usp=sharing

We're going to try reverting 5f69a6577bc33d8f6d6bbe02bccdeb357b287f56 and see if it helps.

Re: oomd with 6.0-rc1 has ridiculously high memory pressure stats wit

[Tejun Heo]

On Tue, Aug 23, 2022 at 02:59:29PM -0400, Chris Murphy wrote:
> Same VM but a different boot:
> 
> Excerpts:
> 
> /sys/fs/cgroup/user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org/session.slice/gvfs-goa-volume-monitor.service/io.pressure:some avg10=3031575.41 avg60=56713935870.67 avg300=624837039080.83 total=18446621498826359
> /sys/fs/cgroup/user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org/session.slice/gvfs-goa-volume-monitor.service/io.pressure:full avg10=3031575.41 avg60=56713935870.80 avg300=624837039080.99 total=16045481047390973
> 
> None of that seems possible.
> 
> io is also affected:
> 
> /sys/fs/cgroup/user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org/session.slice/org.gnome.SettingsDaemon.Smartcard.service/io.pressure:full avg10=0.00 avg60=0.13 avg300=626490311370.87 total=16045481047397307
> 
> # oomctl
> # grep -R . /sys/fs/cgroup/user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org/
> https://drive.google.com/file/d/1JoUxjQ2ribDvn5jmydCWXJdg0daaNScG/view?usp=sharing
> 
> We're going to try reverting 5f69a6577bc33d8f6d6bbe02bccdeb357b287f56 and see if it helps.

Can you see whether the following helps?

Thanks.

diff --git a/kernel/sched/psi.c b/kernel/sched/psi.c
index ec66b40bdd40..00d62681ea6a 100644
--- a/kernel/sched/psi.c
+++ b/kernel/sched/psi.c
@@ -957,7 +957,7 @@ int psi_cgroup_alloc(struct cgroup *cgroup)
 	if (static_branch_likely(&psi_disabled))
 		return 0;
 
-	cgroup->psi = kmalloc(sizeof(struct psi_group), GFP_KERNEL);
+	cgroup->psi = kzalloc(sizeof(struct psi_group), GFP_KERNEL);
 	if (!cgroup->psi)
 		return -ENOMEM;
 

Re: oomd with 6.0-rc1 has ridiculously high memory pressure stats wit

[Chris Murphy]

On Tue, Aug 23, 2022, at 4:12 PM, Tejun Heo wrote:
> On Tue, Aug 23, 2022 at 02:59:29PM -0400, Chris Murphy wrote:
>> Same VM but a different boot:
>> 
>> Excerpts:
>> 
>> /sys/fs/cgroup/user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org/session.slice/gvfs-goa-volume-monitor.service/io.pressure:some avg10=3031575.41 avg60=56713935870.67 avg300=624837039080.83 total=18446621498826359
>> /sys/fs/cgroup/user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org/session.slice/gvfs-goa-volume-monitor.service/io.pressure:full avg10=3031575.41 avg60=56713935870.80 avg300=624837039080.99 total=16045481047390973
>> 
>> None of that seems possible.
>> 
>> io is also affected:
>> 
>> /sys/fs/cgroup/user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org/session.slice/org.gnome.SettingsDaemon.Smartcard.service/io.pressure:full avg10=0.00 avg60=0.13 avg300=626490311370.87 total=16045481047397307
>> 
>> # oomctl
>> # grep -R . /sys/fs/cgroup/user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org/
>> https://drive.google.com/file/d/1JoUxjQ2ribDvn5jmydCWXJdg0daaNScG/view?usp=sharing
>> 
>> We're going to try reverting 5f69a6577bc33d8f6d6bbe02bccdeb357b287f56 and see if it helps.
>
> Can you see whether the following helps?
>
> Thanks.
>
> diff --git a/kernel/sched/psi.c b/kernel/sched/psi.c
> index ec66b40bdd40..00d62681ea6a 100644
> --- a/kernel/sched/psi.c
> +++ b/kernel/sched/psi.c
> @@ -957,7 +957,7 @@ int psi_cgroup_alloc(struct cgroup *cgroup)
>  	if (static_branch_likely(&psi_disabled))
>  		return 0;
> 
> -	cgroup->psi = kmalloc(sizeof(struct psi_group), GFP_KERNEL);
> +	cgroup->psi = kzalloc(sizeof(struct psi_group), GFP_KERNEL);
>  	if (!cgroup->psi)
>  		return -ENOMEM;

Looks like it's fixed in limited testing. It'll get put into Rawhide and automated testing today hopefully. Thanks!

-- 
Chris Murphy

Re: oomd with 6.0-rc1 has ridiculously high memory pressure stats wit

[Chris Murphy]

On Wed, Aug 24, 2022, at 7:40 AM, Chris Murphy wrote:
> On Tue, Aug 23, 2022, at 4:12 PM, Tejun Heo wrote:
>> On Tue, Aug 23, 2022 at 02:59:29PM -0400, Chris Murphy wrote:
>>> Same VM but a different boot:
>>> 
>>> Excerpts:
>>> 
>>> /sys/fs/cgroup/user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org/session.slice/gvfs-goa-volume-monitor.service/io.pressure:some avg10=3031575.41 avg60=56713935870.67 avg300=624837039080.83 total=18446621498826359
>>> /sys/fs/cgroup/user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org/session.slice/gvfs-goa-volume-monitor.service/io.pressure:full avg10=3031575.41 avg60=56713935870.80 avg300=624837039080.99 total=16045481047390973
>>> 
>>> None of that seems possible.
>>> 
>>> io is also affected:
>>> 
>>> /sys/fs/cgroup/user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org/session.slice/org.gnome.SettingsDaemon.Smartcard.service/io.pressure:full avg10=0.00 avg60=0.13 avg300=626490311370.87 total=16045481047397307
>>> 
>>> # oomctl
>>> # grep -R . /sys/fs/cgroup/user.slice/user-1000.slice/user-HmGangybm7RTDjBF/Jpztg@public.gmane.org/
>>> https://drive.google.com/file/d/1JoUxjQ2ribDvn5jmydCWXJdg0daaNScG/view?usp=sharing
>>> 
>>> We're going to try reverting 5f69a6577bc33d8f6d6bbe02bccdeb357b287f56 and see if it helps.
>>
>> Can you see whether the following helps?
>>
>> Thanks.
>>
>> diff --git a/kernel/sched/psi.c b/kernel/sched/psi.c
>> index ec66b40bdd40..00d62681ea6a 100644
>> --- a/kernel/sched/psi.c
>> +++ b/kernel/sched/psi.c
>> @@ -957,7 +957,7 @@ int psi_cgroup_alloc(struct cgroup *cgroup)
>>  	if (static_branch_likely(&psi_disabled))
>>  		return 0;
>> 
>> -	cgroup->psi = kmalloc(sizeof(struct psi_group), GFP_KERNEL);
>> +	cgroup->psi = kzalloc(sizeof(struct psi_group), GFP_KERNEL);
>>  	if (!cgroup->psi)
>>  		return -ENOMEM;
>
> Looks like it's fixed in limited testing. It'll get put into Rawhide 
> and automated testing today hopefully. 

Patch has been in Rawhide without any failures, consider it fixed. Thanks!


-- 
Chris Murphy

Re: oomd with 6.0-rc1 has ridiculously high memory pressure stats wit

[Tejun Heo]

On Fri, Aug 26, 2022 at 08:06:15AM -0400, Chris Murphy wrote:
> Patch has been in Rawhide without any failures, consider it fixed. Thanks!

Upstream fix is 2b97cf76289a ("Hao Jia <jiahao.os-EC8Uxl6Npydl57MIdRCFDg@public.gmane.org>") which was
already sitting in my queue by the time you reported the problem. -rc3
should have it fixed. I should have pushed that out earlier. Sorry about
that.

Thanks.

-- 
tejun