From: "Christian König" <christian.koenig@amd.com>
To: "Michal Hocko" <mhocko@suse.com>,
"Christian König" <ckoenig.leichtzumerken@gmail.com>
Cc: linux-media@vger.kernel.org, linux-kernel@vger.kernel.org,
intel-gfx@lists.freedesktop.org, amd-gfx@lists.freedesktop.org,
nouveau@lists.freedesktop.org, linux-tegra@vger.kernel.org,
linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
alexander.deucher@amd.com, daniel@ffwll.ch,
viro@zeniv.linux.org.uk, akpm@linux-foundation.org,
hughd@google.com, andrey.grodzovsky@amd.com
Subject: Re: [PATCH 03/13] mm: shmem: provide oom badness for shmem files
Date: Mon, 13 Jun 2022 13:50:28 +0200 [thread overview]
Message-ID: <b8b9aba5-575e-8a34-e627-79bef4ed7f97@amd.com> (raw)
In-Reply-To: <Yqbq/Q5jz2ou87Jx@dhcp22.suse.cz>
Am 13.06.22 um 09:45 schrieb Michal Hocko:
> On Sat 11-06-22 10:06:18, Christian König wrote:
>> Am 10.06.22 um 16:16 schrieb Michal Hocko:
> [...]
>> I could of course add something to struct page to track which memcg (or
>> process) it was charged against, but extending struct page is most likely a
>> no-go.
> Struct page already maintains is memcg. The one which has charged it and
> it will stay constatnt throughout of the allocation lifetime (cgroup v1
> has a concept of the charge migration but this hasn't been adopted in
> v2).
>
> We have a concept of active_memcg which allows to charge against a
> different memcg than the allocating context. From your example above I
> do not think this is really usable for the described usecase as the X is
> not aware where the request comes from?
Well X/Wayland is aware, but not the underlying kernel drivers.
When X/Wayland would want to forward this information to the kernel we
would need to extend the existing UAPI quite a bit. And that of course
doesn't help us at all with existing desktops.
>> Alternative I could try to track the "owner" of a buffer (e.g. a shmem
>> file), but then it can happen that one processes creates the object and
>> another one is writing to it and actually allocating the memory.
> If you can enforce that the owner is really responsible for the
> allocation then all should be fine. That would require MAP_POPULATE like
> semantic and I suspect this is not really feasible with the existing
> userspace. It would be certainly hard to enforce for bad players.
I've tried this today and the result was: "BUG: Bad rss-counter state
mm:000000008751d9ff type:MM_FILEPAGES val:-571286".
The problem is once more that files are not informed when the process
clones. So what happened is that somebody called fork() with an
mm_struct I've accounted my pages to. The result is just that we messed
up the rss_stats and the the "BUG..." above.
The key difference between normal allocated pages and the resources here
is just that we are not bound to an mm_struct in any way.
I could just potentially add a dummy VMA to the mm_struct, but to be
honest I think that this would just be an absolutely hack.
So I'm running out of ideas how to fix this, except for adding this per
file oom badness like I proposed.
Regards,
Christian.
WARNING: multiple messages have this Message-ID (diff)
From: "Christian König" <christian.koenig@amd.com>
To: "Michal Hocko" <mhocko@suse.com>,
"Christian König" <ckoenig.leichtzumerken@gmail.com>
Cc: andrey.grodzovsky@amd.com, linux-mm@kvack.org,
nouveau@lists.freedesktop.org, intel-gfx@lists.freedesktop.org,
hughd@google.com, linux-kernel@vger.kernel.org,
amd-gfx@lists.freedesktop.org, linux-fsdevel@vger.kernel.org,
viro@zeniv.linux.org.uk, daniel@ffwll.ch,
linux-tegra@vger.kernel.org, alexander.deucher@amd.com,
akpm@linux-foundation.org, linux-media@vger.kernel.org
Subject: Re: [Nouveau] [PATCH 03/13] mm: shmem: provide oom badness for shmem files
Date: Mon, 13 Jun 2022 13:50:28 +0200 [thread overview]
Message-ID: <b8b9aba5-575e-8a34-e627-79bef4ed7f97@amd.com> (raw)
In-Reply-To: <Yqbq/Q5jz2ou87Jx@dhcp22.suse.cz>
Am 13.06.22 um 09:45 schrieb Michal Hocko:
> On Sat 11-06-22 10:06:18, Christian König wrote:
>> Am 10.06.22 um 16:16 schrieb Michal Hocko:
> [...]
>> I could of course add something to struct page to track which memcg (or
>> process) it was charged against, but extending struct page is most likely a
>> no-go.
> Struct page already maintains is memcg. The one which has charged it and
> it will stay constatnt throughout of the allocation lifetime (cgroup v1
> has a concept of the charge migration but this hasn't been adopted in
> v2).
>
> We have a concept of active_memcg which allows to charge against a
> different memcg than the allocating context. From your example above I
> do not think this is really usable for the described usecase as the X is
> not aware where the request comes from?
Well X/Wayland is aware, but not the underlying kernel drivers.
When X/Wayland would want to forward this information to the kernel we
would need to extend the existing UAPI quite a bit. And that of course
doesn't help us at all with existing desktops.
>> Alternative I could try to track the "owner" of a buffer (e.g. a shmem
>> file), but then it can happen that one processes creates the object and
>> another one is writing to it and actually allocating the memory.
> If you can enforce that the owner is really responsible for the
> allocation then all should be fine. That would require MAP_POPULATE like
> semantic and I suspect this is not really feasible with the existing
> userspace. It would be certainly hard to enforce for bad players.
I've tried this today and the result was: "BUG: Bad rss-counter state
mm:000000008751d9ff type:MM_FILEPAGES val:-571286".
The problem is once more that files are not informed when the process
clones. So what happened is that somebody called fork() with an
mm_struct I've accounted my pages to. The result is just that we messed
up the rss_stats and the the "BUG..." above.
The key difference between normal allocated pages and the resources here
is just that we are not bound to an mm_struct in any way.
I could just potentially add a dummy VMA to the mm_struct, but to be
honest I think that this would just be an absolutely hack.
So I'm running out of ideas how to fix this, except for adding this per
file oom badness like I proposed.
Regards,
Christian.
WARNING: multiple messages have this Message-ID (diff)
From: "Christian König" <christian.koenig@amd.com>
To: "Michal Hocko" <mhocko@suse.com>,
"Christian König" <ckoenig.leichtzumerken@gmail.com>
Cc: andrey.grodzovsky@amd.com, linux-mm@kvack.org,
nouveau@lists.freedesktop.org, intel-gfx@lists.freedesktop.org,
hughd@google.com, linux-kernel@vger.kernel.org,
amd-gfx@lists.freedesktop.org, linux-fsdevel@vger.kernel.org,
viro@zeniv.linux.org.uk, linux-tegra@vger.kernel.org,
alexander.deucher@amd.com, akpm@linux-foundation.org,
linux-media@vger.kernel.org
Subject: Re: [Intel-gfx] [PATCH 03/13] mm: shmem: provide oom badness for shmem files
Date: Mon, 13 Jun 2022 13:50:28 +0200 [thread overview]
Message-ID: <b8b9aba5-575e-8a34-e627-79bef4ed7f97@amd.com> (raw)
In-Reply-To: <Yqbq/Q5jz2ou87Jx@dhcp22.suse.cz>
Am 13.06.22 um 09:45 schrieb Michal Hocko:
> On Sat 11-06-22 10:06:18, Christian König wrote:
>> Am 10.06.22 um 16:16 schrieb Michal Hocko:
> [...]
>> I could of course add something to struct page to track which memcg (or
>> process) it was charged against, but extending struct page is most likely a
>> no-go.
> Struct page already maintains is memcg. The one which has charged it and
> it will stay constatnt throughout of the allocation lifetime (cgroup v1
> has a concept of the charge migration but this hasn't been adopted in
> v2).
>
> We have a concept of active_memcg which allows to charge against a
> different memcg than the allocating context. From your example above I
> do not think this is really usable for the described usecase as the X is
> not aware where the request comes from?
Well X/Wayland is aware, but not the underlying kernel drivers.
When X/Wayland would want to forward this information to the kernel we
would need to extend the existing UAPI quite a bit. And that of course
doesn't help us at all with existing desktops.
>> Alternative I could try to track the "owner" of a buffer (e.g. a shmem
>> file), but then it can happen that one processes creates the object and
>> another one is writing to it and actually allocating the memory.
> If you can enforce that the owner is really responsible for the
> allocation then all should be fine. That would require MAP_POPULATE like
> semantic and I suspect this is not really feasible with the existing
> userspace. It would be certainly hard to enforce for bad players.
I've tried this today and the result was: "BUG: Bad rss-counter state
mm:000000008751d9ff type:MM_FILEPAGES val:-571286".
The problem is once more that files are not informed when the process
clones. So what happened is that somebody called fork() with an
mm_struct I've accounted my pages to. The result is just that we messed
up the rss_stats and the the "BUG..." above.
The key difference between normal allocated pages and the resources here
is just that we are not bound to an mm_struct in any way.
I could just potentially add a dummy VMA to the mm_struct, but to be
honest I think that this would just be an absolutely hack.
So I'm running out of ideas how to fix this, except for adding this per
file oom badness like I proposed.
Regards,
Christian.
WARNING: multiple messages have this Message-ID (diff)
From: "Christian König" <christian.koenig@amd.com>
To: "Michal Hocko" <mhocko@suse.com>,
"Christian König" <ckoenig.leichtzumerken@gmail.com>
Cc: andrey.grodzovsky@amd.com, linux-mm@kvack.org,
nouveau@lists.freedesktop.org, intel-gfx@lists.freedesktop.org,
hughd@google.com, linux-kernel@vger.kernel.org,
amd-gfx@lists.freedesktop.org, linux-fsdevel@vger.kernel.org,
viro@zeniv.linux.org.uk, daniel@ffwll.ch,
linux-tegra@vger.kernel.org, alexander.deucher@amd.com,
akpm@linux-foundation.org, linux-media@vger.kernel.org
Subject: Re: [PATCH 03/13] mm: shmem: provide oom badness for shmem files
Date: Mon, 13 Jun 2022 13:50:28 +0200 [thread overview]
Message-ID: <b8b9aba5-575e-8a34-e627-79bef4ed7f97@amd.com> (raw)
In-Reply-To: <Yqbq/Q5jz2ou87Jx@dhcp22.suse.cz>
Am 13.06.22 um 09:45 schrieb Michal Hocko:
> On Sat 11-06-22 10:06:18, Christian König wrote:
>> Am 10.06.22 um 16:16 schrieb Michal Hocko:
> [...]
>> I could of course add something to struct page to track which memcg (or
>> process) it was charged against, but extending struct page is most likely a
>> no-go.
> Struct page already maintains is memcg. The one which has charged it and
> it will stay constatnt throughout of the allocation lifetime (cgroup v1
> has a concept of the charge migration but this hasn't been adopted in
> v2).
>
> We have a concept of active_memcg which allows to charge against a
> different memcg than the allocating context. From your example above I
> do not think this is really usable for the described usecase as the X is
> not aware where the request comes from?
Well X/Wayland is aware, but not the underlying kernel drivers.
When X/Wayland would want to forward this information to the kernel we
would need to extend the existing UAPI quite a bit. And that of course
doesn't help us at all with existing desktops.
>> Alternative I could try to track the "owner" of a buffer (e.g. a shmem
>> file), but then it can happen that one processes creates the object and
>> another one is writing to it and actually allocating the memory.
> If you can enforce that the owner is really responsible for the
> allocation then all should be fine. That would require MAP_POPULATE like
> semantic and I suspect this is not really feasible with the existing
> userspace. It would be certainly hard to enforce for bad players.
I've tried this today and the result was: "BUG: Bad rss-counter state
mm:000000008751d9ff type:MM_FILEPAGES val:-571286".
The problem is once more that files are not informed when the process
clones. So what happened is that somebody called fork() with an
mm_struct I've accounted my pages to. The result is just that we messed
up the rss_stats and the the "BUG..." above.
The key difference between normal allocated pages and the resources here
is just that we are not bound to an mm_struct in any way.
I could just potentially add a dummy VMA to the mm_struct, but to be
honest I think that this would just be an absolutely hack.
So I'm running out of ideas how to fix this, except for adding this per
file oom badness like I proposed.
Regards,
Christian.
next prev parent reply other threads:[~2022-06-13 14:46 UTC|newest]
Thread overview: 145+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-05-31 9:59 Per file OOM badness Christian König
2022-05-31 9:59 ` Christian König
2022-05-31 9:59 ` [Nouveau] " Christian König
2022-05-31 9:59 ` [PATCH 01/13] fs: add OOM badness callback to file_operatrations struct Christian König
2022-05-31 9:59 ` Christian König
2022-05-31 9:59 ` [Nouveau] " Christian König
2022-05-31 9:59 ` [PATCH 02/13] oom: take per file badness into account Christian König
2022-05-31 9:59 ` Christian König
2022-05-31 9:59 ` [Nouveau] " Christian König
2022-05-31 9:59 ` [PATCH 03/13] mm: shmem: provide oom badness for shmem files Christian König
2022-05-31 9:59 ` Christian König
2022-05-31 9:59 ` [Nouveau] " Christian König
2022-06-09 9:18 ` Michal Hocko
2022-06-09 9:18 ` [Nouveau] " Michal Hocko
2022-06-09 9:18 ` Michal Hocko
2022-06-09 9:18 ` [Intel-gfx] " Michal Hocko
2022-06-09 12:16 ` Christian König
2022-06-09 12:16 ` Christian König
2022-06-09 12:16 ` [Intel-gfx] " Christian König
2022-06-09 12:16 ` [Nouveau] " Christian König
2022-06-09 12:57 ` Michal Hocko
2022-06-09 12:57 ` [Nouveau] " Michal Hocko
2022-06-09 12:57 ` Michal Hocko
2022-06-09 12:57 ` [Intel-gfx] " Michal Hocko
2022-06-09 14:10 ` Christian König
2022-06-09 14:10 ` Christian König
2022-06-09 14:10 ` [Nouveau] " Christian König
2022-06-09 14:21 ` Michal Hocko
2022-06-09 14:21 ` [Nouveau] " Michal Hocko
2022-06-09 14:21 ` Michal Hocko
2022-06-09 14:21 ` [Intel-gfx] " Michal Hocko
2022-06-09 14:29 ` Christian König
2022-06-09 14:29 ` Christian König
2022-06-09 14:29 ` [Intel-gfx] " Christian König
2022-06-09 14:29 ` [Nouveau] " Christian König
2022-06-09 15:07 ` Michal Hocko
2022-06-09 15:07 ` [Nouveau] " Michal Hocko
2022-06-09 15:07 ` Michal Hocko
2022-06-09 15:07 ` [Intel-gfx] " Michal Hocko
2022-06-10 10:58 ` Christian König
2022-06-10 10:58 ` Christian König
2022-06-10 10:58 ` [Nouveau] " Christian König
2022-06-10 11:44 ` Michal Hocko
2022-06-10 11:44 ` [Nouveau] " Michal Hocko
2022-06-10 11:44 ` Michal Hocko
2022-06-10 11:44 ` [Intel-gfx] " Michal Hocko
2022-06-10 12:17 ` Christian König
2022-06-10 12:17 ` Christian König
2022-06-10 12:17 ` [Intel-gfx] " Christian König
2022-06-10 12:17 ` [Nouveau] " Christian König
2022-06-10 14:16 ` Michal Hocko
2022-06-10 14:16 ` [Nouveau] " Michal Hocko
2022-06-10 14:16 ` Michal Hocko
2022-06-10 14:16 ` [Intel-gfx] " Michal Hocko
2022-06-11 8:06 ` Christian König
2022-06-11 8:06 ` Christian König
2022-06-11 8:06 ` [Intel-gfx] " Christian König
2022-06-11 8:06 ` [Nouveau] " Christian König
2022-06-13 7:45 ` Michal Hocko
2022-06-13 7:45 ` [Nouveau] " Michal Hocko
2022-06-13 7:45 ` Michal Hocko
2022-06-13 7:45 ` [Intel-gfx] " Michal Hocko
2022-06-13 11:50 ` Christian König [this message]
2022-06-13 11:50 ` Christian König
2022-06-13 11:50 ` [Intel-gfx] " Christian König
2022-06-13 11:50 ` [Nouveau] " Christian König
2022-06-13 12:11 ` Michal Hocko
2022-06-13 12:11 ` [Nouveau] " Michal Hocko
2022-06-13 12:11 ` Michal Hocko
2022-06-13 12:11 ` [Intel-gfx] " Michal Hocko
2022-06-13 12:55 ` [Nouveau] " Christian König
2022-06-13 12:55 ` Christian König
2022-06-13 12:55 ` Christian König
2022-06-13 12:55 ` [Intel-gfx] " Christian König
2022-06-13 14:11 ` Michal Hocko
2022-06-13 14:11 ` [Nouveau] " Michal Hocko
2022-06-13 14:11 ` Michal Hocko
2022-06-13 14:11 ` Michal Hocko
2022-06-15 12:35 ` Christian König
2022-06-15 12:35 ` Christian König
2022-06-15 12:35 ` [Intel-gfx] " Christian König
2022-06-15 12:35 ` [Nouveau] " Christian König
2022-06-15 13:15 ` Michal Hocko
2022-06-15 13:15 ` [Nouveau] " Michal Hocko
2022-06-15 13:15 ` Michal Hocko
2022-06-15 13:15 ` [Intel-gfx] " Michal Hocko
2022-06-15 14:24 ` Christian König
2022-06-15 14:24 ` Christian König
2022-06-15 14:24 ` [Intel-gfx] " Christian König
2022-06-15 14:24 ` [Nouveau] " Christian König
2022-06-13 9:08 ` Michel Dänzer
2022-06-13 9:08 ` [Nouveau] " Michel Dänzer
2022-06-13 9:08 ` [Intel-gfx] " Michel Dänzer
2022-06-13 9:08 ` Michel Dänzer
2022-06-13 9:11 ` Christian König
2022-06-13 9:11 ` Christian König
2022-06-13 9:11 ` [Intel-gfx] " Christian König
2022-06-13 9:11 ` [Nouveau] " Christian König
2022-06-09 15:19 ` Felix Kuehling
2022-06-09 15:19 ` Felix Kuehling
2022-06-09 15:19 ` [Intel-gfx] " Felix Kuehling
2022-06-09 15:19 ` [Nouveau] " Felix Kuehling
2022-06-09 15:22 ` Christian König
2022-06-09 15:22 ` Christian König
2022-06-09 15:22 ` [Intel-gfx] " Christian König
2022-06-09 15:22 ` [Nouveau] " Christian König
2022-06-09 15:54 ` Michal Hocko
2022-06-09 15:54 ` [Nouveau] " Michal Hocko
2022-06-09 15:54 ` Michal Hocko
2022-06-09 15:54 ` [Intel-gfx] " Michal Hocko
2022-05-31 9:59 ` [PATCH 04/13] dma-buf: provide oom badness for DMA-buf files Christian König
2022-05-31 9:59 ` Christian König
2022-05-31 9:59 ` [Nouveau] " Christian König
2022-05-31 9:59 ` [PATCH 05/13] drm/gem: adjust per file OOM badness on handling buffers Christian König
2022-05-31 9:59 ` Christian König
2022-05-31 9:59 ` [Nouveau] " Christian König
2022-05-31 10:00 ` [PATCH 06/13] drm/gma500: use drm_oom_badness Christian König
2022-05-31 10:00 ` Christian König
2022-05-31 10:00 ` [Nouveau] " Christian König
2022-05-31 10:00 ` [PATCH 07/13] drm/amdgpu: Use drm_oom_badness for amdgpu Christian König
2022-05-31 10:00 ` Christian König
2022-05-31 10:00 ` [Nouveau] " Christian König
2022-05-31 10:00 ` [PATCH 08/13] drm/radeon: use drm_oom_badness Christian König
2022-05-31 10:00 ` Christian König
2022-05-31 10:00 ` [Nouveau] " Christian König
2022-05-31 10:00 ` [PATCH 09/13] drm/i915: " Christian König
2022-05-31 10:00 ` Christian König
2022-05-31 10:00 ` [Nouveau] " Christian König
2022-05-31 10:00 ` [PATCH 10/13] drm/nouveau: " Christian König
2022-05-31 10:00 ` Christian König
2022-05-31 10:00 ` [Nouveau] " Christian König
2022-05-31 10:00 ` [PATCH 11/13] drm/omap: " Christian König
2022-05-31 10:00 ` Christian König
2022-05-31 10:00 ` [Nouveau] " Christian König
2022-05-31 10:00 ` [PATCH 12/13] drm/vmwgfx: " Christian König
2022-05-31 10:00 ` Christian König
2022-05-31 10:00 ` [Nouveau] " Christian König
2022-05-31 10:00 ` [PATCH 13/13] drm/tegra: " Christian König
2022-05-31 10:00 ` Christian König
2022-05-31 10:00 ` [Nouveau] " Christian König
2022-05-31 22:00 ` Per file OOM badness Alex Deucher
2022-05-31 22:00 ` Alex Deucher
2022-05-31 22:00 ` [Intel-gfx] " Alex Deucher
2022-05-31 22:00 ` Alex Deucher
2022-05-31 22:00 ` [Nouveau] " Alex Deucher
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=b8b9aba5-575e-8a34-e627-79bef4ed7f97@amd.com \
--to=christian.koenig@amd.com \
--cc=akpm@linux-foundation.org \
--cc=alexander.deucher@amd.com \
--cc=amd-gfx@lists.freedesktop.org \
--cc=andrey.grodzovsky@amd.com \
--cc=ckoenig.leichtzumerken@gmail.com \
--cc=daniel@ffwll.ch \
--cc=hughd@google.com \
--cc=intel-gfx@lists.freedesktop.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-media@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-tegra@vger.kernel.org \
--cc=mhocko@suse.com \
--cc=nouveau@lists.freedesktop.org \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.