From: "Christian König" <ckoenig.leichtzumerken@gmail.com>
To: linux-media@vger.kernel.org, linux-kernel@vger.kernel.org,
intel-gfx@lists.freedesktop.org, amd-gfx@lists.freedesktop.org,
nouveau@lists.freedesktop.org, linux-tegra@vger.kernel.org,
linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
dri-devel@lists.freedesktop.org
Cc: mhocko@suse.com
Subject: [RFC] Per file OOM-badness / RSS once more
Date: Fri, 24 Jun 2022 10:04:30 +0200 [thread overview]
Message-ID: <20220624080444.7619-1-christian.koenig@amd.com> (raw)
Hello everyone,
To summarize the issue I'm trying to address here: Processes can allocate
resources through a file descriptor without being held responsible for it.
I'm not explaining all the details again. See here for a more deeply
description of the problem: https://lwn.net/ml/linux-kernel/20220531100007.174649-1-christian.koenig@amd.com/
With this iteration I'm trying to address a bunch of the comments Michal Hocko
(thanks a lot for that) gave as well as giving some new ideas.
Changes made so far:
1. Renamed the callback into file_rss(). This is at least a start to better
describe what this is all about. I've been going back and forth over the
naming here, if you have any better idea please speak up.
2. Cleanups, e.g. now providing a helper function in the fs layer to sum up
all the pages allocated by the files in a file descriptor table.
3. Using the actual number of allocated pages for the shmem implementation
instead of just the size. I also tried to ignore shmem files which are part
of tmpfs, cause that has a separate accounting/limitation approach.
4. The OOM killer now prints the memory of the killed process including the per
file pages which makes the whole things much more comprehensible.
5. I've added the per file pages to the different reports in RSS in procfs.
This has the interesting effect that tools like top suddenly give a much
more accurate overview of the memory use as well. This of course increases
the overhead of gathering those information quite a bit and I'm not sure how
feasible that is for up-streaming. On the other hand this once more clearly
shows that we need to do something about this issue.
Another rather interesting observation is that multiple subsystems (shmem,
tmpfs, ttm) came up with the same workaround of limiting the memory which can
be allocated through them to 50% of the whole system memory. Unfortunately
that isn't the same 50% and it doesn't apply everywhere, so you can still
easily crash the box.
Ideas and/or comments are really welcome.
Thanks,
Christian.
WARNING: multiple messages have this Message-ID (diff)
From: "Christian König" <ckoenig.leichtzumerken@gmail.com>
To: linux-media@vger.kernel.org, linux-kernel@vger.kernel.org,
intel-gfx@lists.freedesktop.org, amd-gfx@lists.freedesktop.org,
nouveau@lists.freedesktop.org, linux-tegra@vger.kernel.org,
linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
dri-devel@lists.freedesktop.org
Cc: mhocko@suse.com
Subject: [Nouveau] [RFC] Per file OOM-badness / RSS once more
Date: Fri, 24 Jun 2022 10:04:30 +0200 [thread overview]
Message-ID: <20220624080444.7619-1-christian.koenig@amd.com> (raw)
Hello everyone,
To summarize the issue I'm trying to address here: Processes can allocate
resources through a file descriptor without being held responsible for it.
I'm not explaining all the details again. See here for a more deeply
description of the problem: https://lwn.net/ml/linux-kernel/20220531100007.174649-1-christian.koenig@amd.com/
With this iteration I'm trying to address a bunch of the comments Michal Hocko
(thanks a lot for that) gave as well as giving some new ideas.
Changes made so far:
1. Renamed the callback into file_rss(). This is at least a start to better
describe what this is all about. I've been going back and forth over the
naming here, if you have any better idea please speak up.
2. Cleanups, e.g. now providing a helper function in the fs layer to sum up
all the pages allocated by the files in a file descriptor table.
3. Using the actual number of allocated pages for the shmem implementation
instead of just the size. I also tried to ignore shmem files which are part
of tmpfs, cause that has a separate accounting/limitation approach.
4. The OOM killer now prints the memory of the killed process including the per
file pages which makes the whole things much more comprehensible.
5. I've added the per file pages to the different reports in RSS in procfs.
This has the interesting effect that tools like top suddenly give a much
more accurate overview of the memory use as well. This of course increases
the overhead of gathering those information quite a bit and I'm not sure how
feasible that is for up-streaming. On the other hand this once more clearly
shows that we need to do something about this issue.
Another rather interesting observation is that multiple subsystems (shmem,
tmpfs, ttm) came up with the same workaround of limiting the memory which can
be allocated through them to 50% of the whole system memory. Unfortunately
that isn't the same 50% and it doesn't apply everywhere, so you can still
easily crash the box.
Ideas and/or comments are really welcome.
Thanks,
Christian.
WARNING: multiple messages have this Message-ID (diff)
From: "Christian König" <ckoenig.leichtzumerken@gmail.com>
To: linux-media@vger.kernel.org, linux-kernel@vger.kernel.org,
intel-gfx@lists.freedesktop.org, amd-gfx@lists.freedesktop.org,
nouveau@lists.freedesktop.org, linux-tegra@vger.kernel.org,
linux-fsdevel@vger.kernel.org, linux-mm@kvack.org,
dri-devel@lists.freedesktop.org
Cc: mhocko@suse.com
Subject: [Intel-gfx] [RFC] Per file OOM-badness / RSS once more
Date: Fri, 24 Jun 2022 10:04:30 +0200 [thread overview]
Message-ID: <20220624080444.7619-1-christian.koenig@amd.com> (raw)
Hello everyone,
To summarize the issue I'm trying to address here: Processes can allocate
resources through a file descriptor without being held responsible for it.
I'm not explaining all the details again. See here for a more deeply
description of the problem: https://lwn.net/ml/linux-kernel/20220531100007.174649-1-christian.koenig@amd.com/
With this iteration I'm trying to address a bunch of the comments Michal Hocko
(thanks a lot for that) gave as well as giving some new ideas.
Changes made so far:
1. Renamed the callback into file_rss(). This is at least a start to better
describe what this is all about. I've been going back and forth over the
naming here, if you have any better idea please speak up.
2. Cleanups, e.g. now providing a helper function in the fs layer to sum up
all the pages allocated by the files in a file descriptor table.
3. Using the actual number of allocated pages for the shmem implementation
instead of just the size. I also tried to ignore shmem files which are part
of tmpfs, cause that has a separate accounting/limitation approach.
4. The OOM killer now prints the memory of the killed process including the per
file pages which makes the whole things much more comprehensible.
5. I've added the per file pages to the different reports in RSS in procfs.
This has the interesting effect that tools like top suddenly give a much
more accurate overview of the memory use as well. This of course increases
the overhead of gathering those information quite a bit and I'm not sure how
feasible that is for up-streaming. On the other hand this once more clearly
shows that we need to do something about this issue.
Another rather interesting observation is that multiple subsystems (shmem,
tmpfs, ttm) came up with the same workaround of limiting the memory which can
be allocated through them to 50% of the whole system memory. Unfortunately
that isn't the same 50% and it doesn't apply everywhere, so you can still
easily crash the box.
Ideas and/or comments are really welcome.
Thanks,
Christian.
next reply other threads:[~2022-06-24 8:04 UTC|newest]
Thread overview: 63+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-06-24 8:04 Christian König [this message]
2022-06-24 8:04 ` [Intel-gfx] [RFC] Per file OOM-badness / RSS once more Christian König
2022-06-24 8:04 ` [Nouveau] " Christian König
2022-06-24 8:04 ` [PATCH 01/14] fs: add per file RSS Christian König
2022-06-24 8:04 ` Christian König
2022-06-24 8:04 ` [Intel-gfx] " Christian König
2022-06-24 8:04 ` Christian König
2022-06-24 8:04 ` [Nouveau] " Christian König
2022-06-24 8:04 ` [PATCH 02/14] oom: take per file RSS into account Christian König
2022-06-24 8:04 ` Christian König
2022-06-24 8:04 ` [Intel-gfx] " Christian König
2022-06-24 8:04 ` Christian König
2022-06-24 8:04 ` [Nouveau] " Christian König
2022-06-24 8:04 ` [PATCH 03/14] proc: expose per file RSS Christian König
2022-06-24 8:04 ` [Intel-gfx] " Christian König
2022-06-24 8:04 ` [Nouveau] " Christian König
2022-06-24 8:04 ` [PATCH 04/14] mm: shmem: provide RSS for shmem files Christian König
2022-06-24 8:04 ` [Intel-gfx] " Christian König
2022-06-24 8:04 ` [Nouveau] " Christian König
2022-06-24 8:04 ` [PATCH 05/14] dma-buf: provide file RSS for DMA-buf files Christian König
2022-06-24 8:04 ` [Intel-gfx] " Christian König
2022-06-24 8:04 ` [Nouveau] " Christian König
2022-06-24 8:04 ` [PATCH 06/14] drm/gem: adjust per file RSS on handling buffers Christian König
2022-06-24 8:04 ` Christian König
2022-06-24 8:04 ` [Intel-gfx] " Christian König
2022-06-24 8:04 ` Christian König
2022-06-24 8:04 ` [Nouveau] " Christian König
2022-06-24 8:04 ` [PATCH 07/14] drm/gma500: use drm_file_rss Christian König
2022-06-24 8:04 ` [Nouveau] " Christian König
2022-06-24 8:04 ` [Intel-gfx] " Christian König
2022-06-24 8:04 ` [PATCH 08/14] drm/amdgpu: " Christian König
2022-06-24 8:04 ` Christian König
2022-06-24 8:04 ` [Intel-gfx] " Christian König
2022-06-24 8:04 ` Christian König
2022-06-24 8:04 ` [Nouveau] " Christian König
2022-06-24 8:04 ` [PATCH 09/14] drm/radeon: use drm_oom_badness Christian König
2022-06-24 8:04 ` [Nouveau] " Christian König
2022-06-24 8:04 ` [Intel-gfx] " Christian König
2022-06-28 10:34 ` Michel Dänzer
2022-06-28 10:34 ` [Nouveau] " Michel Dänzer
2022-06-28 10:34 ` [Intel-gfx] " Michel Dänzer
2022-06-24 8:04 ` [PATCH 10/14] drm/i915: use drm_file_rss Christian König
2022-06-24 8:04 ` [Nouveau] " Christian König
2022-06-24 8:04 ` [Intel-gfx] " Christian König
2022-06-24 8:04 ` [PATCH 11/14] drm/nouveau: " Christian König
2022-06-24 8:04 ` [Nouveau] " Christian König
2022-06-24 8:04 ` [Intel-gfx] " Christian König
2022-06-24 8:04 ` [PATCH 12/14] drm/omap: " Christian König
2022-06-24 8:04 ` [Nouveau] " Christian König
2022-06-24 8:04 ` [Intel-gfx] " Christian König
2022-06-24 8:04 ` [PATCH 13/14] drm/vmwgfx: " Christian König
2022-06-24 8:04 ` [Nouveau] " Christian König
2022-06-24 8:04 ` [Intel-gfx] " Christian König
2022-06-24 8:04 ` [PATCH 14/14] drm/tegra: " Christian König
2022-06-24 8:04 ` [Intel-gfx] " Christian König
2022-06-24 8:04 ` [Nouveau] " Christian König
2022-06-24 9:59 ` [RFC] Per file OOM-badness / RSS once more Michal Hocko
2022-06-24 9:59 ` [Nouveau] " Michal Hocko
2022-06-24 9:59 ` [Intel-gfx] " Michal Hocko
2022-06-24 9:59 ` Michal Hocko
2022-06-24 10:23 ` [Intel-gfx] ✗ Fi.CI.CHECKPATCH: warning for series starting with [01/14] fs: add per file RSS Patchwork
2022-06-24 10:23 ` [Intel-gfx] ✗ Fi.CI.SPARSE: " Patchwork
2022-06-24 10:47 ` [Intel-gfx] ✗ Fi.CI.BAT: failure " Patchwork
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220624080444.7619-1-christian.koenig@amd.com \
--to=ckoenig.leichtzumerken@gmail.com \
--cc=amd-gfx@lists.freedesktop.org \
--cc=dri-devel@lists.freedesktop.org \
--cc=intel-gfx@lists.freedesktop.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-media@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-tegra@vger.kernel.org \
--cc=mhocko@suse.com \
--cc=nouveau@lists.freedesktop.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.