* [RFC] fs/proc/meminfo: introduce Unaccounted statistic
@ 2016-10-20 12:11 Vlastimil Babka
2016-10-20 13:33 ` Michal Hocko
0 siblings, 1 reply; 7+ messages in thread
From: Vlastimil Babka @ 2016-10-20 12:11 UTC (permalink / raw)
To: linux-mm
Cc: linux-kernel, Andrew Morton, Vlastimil Babka, Michal Hocko,
Mel Gorman, Johannes Weiner, Kirill A. Shutemov, Hugh Dickins,
David Rientjes
The /proc/meminfo virtual file is a collection of various system-wide memory
usage statistics. One of the use cases for looking at the output is to see
whether the kernel might be leaking memory by direct allocations that are not
counted among any of the statistics. This is hovewer not immediately obvious,
because some fields are meant to add up to the MemTotal value, and others not.
Subtle changes also happen over time, e.g. the AnonPages field started
including THP's since commit 3cd14fcd3f12 ("thp: account anon transparent
huge pages into NR_ANON_PAGES") and the Cached field used to include hugetlb
until commit 4165b9b46181 ("hugetlb: do not account hugetlb pages as
NR_FILE_PAGES").
To make kernel memory leaks more obvious, this patch adds an Unaccounted field
whose value is calculated by subtracting a sum of fields that are supposed to
add up to MemTotal without overlap, from the MeTotal value. This should also
help anyone looking at the code to determine these fields easily.
Not-yet-signed-off-by: Vlastimil Babka <vbabka@suse.cz>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mel Gorman <mgorman@techsingularity.net>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: David Rientjes <rientjes@google.com>
---
Hi, I'm wondering if people would find this useful. If you think it is, and
to not make performance worse, I could also make sure in proper submission
that values are not read via global_page_state() multiple times etc...
fs/proc/meminfo.c | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c
index 8a428498d6b2..3fcd71d4d805 100644
--- a/fs/proc/meminfo.c
+++ b/fs/proc/meminfo.c
@@ -48,6 +48,7 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
unsigned long committed;
long cached;
long available;
+ long unaccounted;
unsigned long pages[NR_LRU_LISTS];
int lru;
@@ -65,6 +66,18 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
available = si_mem_available();
+ unaccounted = i.totalram - i.freeram
+ - global_node_page_state(NR_ANON_MAPPED)
+ - global_node_page_state(NR_FILE_PAGES)
+ - global_page_state(NR_PAGETABLE)
+ - global_page_state(NR_SLAB_RECLAIMABLE)
+ - global_page_state(NR_SLAB_UNRECLAIMABLE)
+ - (global_page_state(NR_KERNEL_STACK_KB)
+ >> (PAGE_SHIFT - 10))
+ - global_page_state(NR_BOUNCE)
+ - global_node_page_state(NR_WRITEBACK_TEMP)
+ - hugetlb_total_pages();
+
show_val_kb(m, "MemTotal: ", i.totalram);
show_val_kb(m, "MemFree: ", i.freeram);
show_val_kb(m, "MemAvailable: ", available);
@@ -119,6 +132,7 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
global_page_state(NR_PAGETABLE));
#ifdef CONFIG_QUICKLIST
show_val_kb(m, "Quicklists: ", quicklist_total_size());
+ unaccounted -= quicklist_total_size();
#endif
show_val_kb(m, "NFS_Unstable: ",
@@ -156,6 +170,10 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
hugetlb_report_meminfo(m);
+ if (unaccounted < 0)
+ unaccounted = 0;
+ show_val_kb(m, "Unaccounted: ", unaccounted);
+
arch_report_meminfo(m);
return 0;
--
2.10.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [RFC] fs/proc/meminfo: introduce Unaccounted statistic
2016-10-20 12:11 [RFC] fs/proc/meminfo: introduce Unaccounted statistic Vlastimil Babka
@ 2016-10-20 13:33 ` Michal Hocko
2016-10-20 22:59 ` Dave Chinner
2016-10-21 7:31 ` Vlastimil Babka
0 siblings, 2 replies; 7+ messages in thread
From: Michal Hocko @ 2016-10-20 13:33 UTC (permalink / raw)
To: Vlastimil Babka
Cc: linux-mm, linux-kernel, Andrew Morton, Mel Gorman,
Johannes Weiner, Kirill A. Shutemov, Hugh Dickins,
David Rientjes
On Thu 20-10-16 14:11:49, Vlastimil Babka wrote:
[...]
> Hi, I'm wondering if people would find this useful. If you think it is, and
> to not make performance worse, I could also make sure in proper submission
> that values are not read via global_page_state() multiple times etc...
I definitely find this information useful and hate to do the math all
the time but on the other hand this is quite fragile and I can imagine
we can easily forget to add something there and provide a misleading
information to the userspace. So I would be worried with a long term
maintainability of this.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [RFC] fs/proc/meminfo: introduce Unaccounted statistic
2016-10-20 13:33 ` Michal Hocko
@ 2016-10-20 22:59 ` Dave Chinner
2016-10-21 7:25 ` Vlastimil Babka
2016-10-21 7:36 ` Michal Hocko
2016-10-21 7:31 ` Vlastimil Babka
1 sibling, 2 replies; 7+ messages in thread
From: Dave Chinner @ 2016-10-20 22:59 UTC (permalink / raw)
To: Michal Hocko
Cc: Vlastimil Babka, linux-mm, linux-kernel, Andrew Morton,
Mel Gorman, Johannes Weiner, Kirill A. Shutemov, Hugh Dickins,
David Rientjes
On Thu, Oct 20, 2016 at 03:33:58PM +0200, Michal Hocko wrote:
> On Thu 20-10-16 14:11:49, Vlastimil Babka wrote:
> [...]
> > Hi, I'm wondering if people would find this useful. If you think it is, and
> > to not make performance worse, I could also make sure in proper submission
> > that values are not read via global_page_state() multiple times etc...
>
> I definitely find this information useful and hate to do the math all
> the time but on the other hand this is quite fragile and I can imagine
> we can easily forget to add something there and provide a misleading
> information to the userspace. So I would be worried with a long term
> maintainability of this.
This will result in valid memory usage by subsystems like the XFS
buffer cache being reported as "unaccounted". Given this cache
(whose size is shrinker controlled) can grow to gigabytes in size
under various metadata intensive workloads, there's every chance
that such reporting will make users incorrectly think they have a
massive memory leak....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [RFC] fs/proc/meminfo: introduce Unaccounted statistic
2016-10-20 22:59 ` Dave Chinner
@ 2016-10-21 7:25 ` Vlastimil Babka
2016-10-21 11:17 ` Dave Chinner
2016-10-21 7:36 ` Michal Hocko
1 sibling, 1 reply; 7+ messages in thread
From: Vlastimil Babka @ 2016-10-21 7:25 UTC (permalink / raw)
To: Dave Chinner, Michal Hocko
Cc: linux-mm, linux-kernel, Andrew Morton, Mel Gorman,
Johannes Weiner, Kirill A. Shutemov, Hugh Dickins,
David Rientjes
On 10/21/2016 12:59 AM, Dave Chinner wrote:
> On Thu, Oct 20, 2016 at 03:33:58PM +0200, Michal Hocko wrote:
>> On Thu 20-10-16 14:11:49, Vlastimil Babka wrote:
>> [...]
>> > Hi, I'm wondering if people would find this useful. If you think it is, and
>> > to not make performance worse, I could also make sure in proper submission
>> > that values are not read via global_page_state() multiple times etc...
>>
>> I definitely find this information useful and hate to do the math all
>> the time but on the other hand this is quite fragile and I can imagine
>> we can easily forget to add something there and provide a misleading
>> information to the userspace. So I would be worried with a long term
>> maintainability of this.
>
> This will result in valid memory usage by subsystems like the XFS
> buffer cache being reported as "unaccounted". Given this cache
> (whose size is shrinker controlled) can grow to gigabytes in size
> under various metadata intensive workloads, there's every chance
> that such reporting will make users incorrectly think they have a
> massive memory leak....
Is the XFS buffer cache accounted (and visible) somewhere then? I'd say getting
such large consumers to become visible on the same level as others would be
another advantage...
And yeah, I can even recall a bug report, where I had to do the calculation
myself and it looked like a big leak, and it took some effort to connect it to
xfs buffers. I'd very much welcome for it to be more obvious.
Vlastimil
> Cheers,
>
> Dave.
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [RFC] fs/proc/meminfo: introduce Unaccounted statistic
2016-10-20 13:33 ` Michal Hocko
2016-10-20 22:59 ` Dave Chinner
@ 2016-10-21 7:31 ` Vlastimil Babka
1 sibling, 0 replies; 7+ messages in thread
From: Vlastimil Babka @ 2016-10-21 7:31 UTC (permalink / raw)
To: Michal Hocko
Cc: linux-mm, linux-kernel, Andrew Morton, Mel Gorman,
Johannes Weiner, Kirill A. Shutemov, Hugh Dickins,
David Rientjes
On 10/20/2016 03:33 PM, Michal Hocko wrote:
> On Thu 20-10-16 14:11:49, Vlastimil Babka wrote:
> [...]
>> Hi, I'm wondering if people would find this useful. If you think it is, and
>> to not make performance worse, I could also make sure in proper submission
>> that values are not read via global_page_state() multiple times etc...
>
> I definitely find this information useful and hate to do the math all
> the time but on the other hand this is quite fragile and I can imagine
> we can easily forget to add something there and provide a misleading
> information to the userspace. So I would be worried with a long term
> maintainability of this.
I wouldn't fear this that much. Maybe even on the contrary - if we unknowingly
change the picture by misacounting something, or changing a counter to become
subset of another, and Unaccounted starts to give weird values, it will give us
hint that there's either a problem to fix, or a missed documentation update.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [RFC] fs/proc/meminfo: introduce Unaccounted statistic
2016-10-20 22:59 ` Dave Chinner
2016-10-21 7:25 ` Vlastimil Babka
@ 2016-10-21 7:36 ` Michal Hocko
1 sibling, 0 replies; 7+ messages in thread
From: Michal Hocko @ 2016-10-21 7:36 UTC (permalink / raw)
To: Dave Chinner
Cc: Vlastimil Babka, linux-mm, linux-kernel, Andrew Morton,
Mel Gorman, Johannes Weiner, Kirill A. Shutemov, Hugh Dickins,
David Rientjes
On Fri 21-10-16 09:59:29, Dave Chinner wrote:
> On Thu, Oct 20, 2016 at 03:33:58PM +0200, Michal Hocko wrote:
> > On Thu 20-10-16 14:11:49, Vlastimil Babka wrote:
> > [...]
> > > Hi, I'm wondering if people would find this useful. If you think it is, and
> > > to not make performance worse, I could also make sure in proper submission
> > > that values are not read via global_page_state() multiple times etc...
> >
> > I definitely find this information useful and hate to do the math all
> > the time but on the other hand this is quite fragile and I can imagine
> > we can easily forget to add something there and provide a misleading
> > information to the userspace. So I would be worried with a long term
> > maintainability of this.
>
> This will result in valid memory usage by subsystems like the XFS
> buffer cache being reported as "unaccounted".
I would argue that the file is more intended for developers than regular
users. Most of those counters simply require a deep knowledge of the MM
subsystem to interpret them correctly (yeah we have seen many reports
that the free memory is too low but we always managed to explain...).
So to me this is more a convenience thing for developers than anything
else. But my worry about maintainability still stands. People who are
adding new counters might easily forget to update this part, so I am not
actually sure it is a good idea long term. Getting a misleading data is
worse than pushing developers to do the math and scratch their heads
about what should be included IMHO.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [RFC] fs/proc/meminfo: introduce Unaccounted statistic
2016-10-21 7:25 ` Vlastimil Babka
@ 2016-10-21 11:17 ` Dave Chinner
0 siblings, 0 replies; 7+ messages in thread
From: Dave Chinner @ 2016-10-21 11:17 UTC (permalink / raw)
To: Vlastimil Babka
Cc: Michal Hocko, linux-mm, linux-kernel, Andrew Morton, Mel Gorman,
Johannes Weiner, Kirill A. Shutemov, Hugh Dickins,
David Rientjes
On Fri, Oct 21, 2016 at 09:25:10AM +0200, Vlastimil Babka wrote:
> On 10/21/2016 12:59 AM, Dave Chinner wrote:
> >On Thu, Oct 20, 2016 at 03:33:58PM +0200, Michal Hocko wrote:
> >>On Thu 20-10-16 14:11:49, Vlastimil Babka wrote:
> >>[...]
> >>> Hi, I'm wondering if people would find this useful. If you think it is, and
> >>> to not make performance worse, I could also make sure in proper submission
> >>> that values are not read via global_page_state() multiple times etc...
> >>
> >>I definitely find this information useful and hate to do the math all
> >>the time but on the other hand this is quite fragile and I can imagine
> >>we can easily forget to add something there and provide a misleading
> >>information to the userspace. So I would be worried with a long term
> >>maintainability of this.
> >
> >This will result in valid memory usage by subsystems like the XFS
> >buffer cache being reported as "unaccounted". Given this cache
> >(whose size is shrinker controlled) can grow to gigabytes in size
> >under various metadata intensive workloads, there's every chance
> >that such reporting will make users incorrectly think they have a
> >massive memory leak....
>
> Is the XFS buffer cache accounted (and visible) somewhere then? I'd
> say getting such large consumers to become visible on the same level
> as others would be another advantage...
It's handles are visible via the xfs_buf slab cache. By the time
you've got enough memory in the buffer cache for it to be noticed,
the xfs_buf slab is near the top of the list in slabtop.
Of course, because of the crazy way slub names caches, this can be
impossible to find because there isn't a "xfs_buf" slab cache that
shows up in slabtop. It'll end being called something like
"mnt_cache"....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2016-10-21 11:17 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-10-20 12:11 [RFC] fs/proc/meminfo: introduce Unaccounted statistic Vlastimil Babka
2016-10-20 13:33 ` Michal Hocko
2016-10-20 22:59 ` Dave Chinner
2016-10-21 7:25 ` Vlastimil Babka
2016-10-21 11:17 ` Dave Chinner
2016-10-21 7:36 ` Michal Hocko
2016-10-21 7:31 ` Vlastimil Babka
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).