Re: How to sample memory usage cheaply?

[Milian Wolff <milian.wolff@kdab.com>]

[-- Attachment #1: Type: text/plain, Size: 3688 bytes --]

On Donnerstag, 30. März 2017 22:04:04 CEST Benjamin King wrote:
> Hi,
> 
> I'd like to get a big picture of where a memory hogging process uses
> physical memory. I'm interested in call graphs, but in terms of Brendans
> treatise (http://www.brendangregg.com/FlameGraphs/memoryflamegraphs.html),
> I'd love to analyze page faults a bit before continuing with the more
> expensive tracing of malloc and friends.

I suggest you try out my heaptrack tool. While "expensive" it does work quite 
well even for applications that allocate a lot of heap memory. It's super easy 
to use, and if it works you don't need to venture into low-level profiling 
like you are describing below.

https://www.kdab.com/heaptrack-v1-0-0-release/

If you are actually using low-level stuff directly, i.e. have a custom memory 
pool implementation, heaptrack also offers an API similar to Valgrind such 
that you can annotate your custom heap allocations.

> My problem is that we mmap some readonly files more than once to save
> memory, but each individual mapping creates page faults.
> 
> This makes sense, but how can I measure physical memory properly, then?
> Parsing the Pss-Rows in /proc/<pid>/smaps does work, but seems a bit clumsy.
> Is there a better way (e.g. with callstacks) to measure physical memory
> growth for a process?

From what I understand, wouldn't you get this by tracing sbrk + mmap with call 
stacks? Do note that file-backed mmaps can be shared across processes, so 
you'll have to take that into account as done for Pss. But in my experience, 
when you want to improve a single application's memory consumption, it usually 
boils down to non-shared heap memory anyways. I.e. sbrk and anon mmaps.

But if you look at the callstacks for these syscalls, they usually point at 
the obvious places (i.e. mempools), but you won't see what is _actually_ using 
the memory of these pools. Heaptrack or massif is much better in that regard.

Hope that helps, happy profiling!

> PS:
>  Here is some measurement with a leaky toy program. It uses a 1GB
> zero-filled file and drops file system caches prior to the measurement to
> encourage major page faults for the first mapping only. Does not work at
> all: -----
> $ gcc -O0 mmap_faults.c
> $ fallocate -z -l $((1<<30)) 1gb_of_garbage.dat
> $ sudo sysctl -w vm.drop_caches=3
> vm.drop_caches = 3
> $ perf stat -eminor-faults,major-faults ./a.out
> 
>  Performance counter stats for './a.out':
> 
>             327,726      minor-faults
>                   1      major-faults
> $ cat mmap_faults.c
> #include <fcntl.h>
> #include <sys/mman.h>
> #include <unistd.h>
> 
> #define numMaps 20
> #define length 1u<<30
> #define path "1gb_of_garbage.dat"
> 
> int main()
> {
>   int sum = 0;
>   for ( int j = 0; j < numMaps; ++j )
>   {
>     const char *result =
>       (const char*)mmap( NULL, length, PROT_READ, MAP_PRIVATE,
>           open( path, O_RDONLY ), 0 );
> 
>     for ( int i = 0; i < length; i += 4096 )
>       sum += result[ i ];
>   }
>   return sum;
> }
> -----
> Shouldn't I see ~5 Million page faults (20GB/4K)?
> Shouldn't I see more major page faults?
> Same thing when garbage-file is filled from /dev/urandom
> Even weirder when MAP_POPULATE'ing the file
> --
> To unsubscribe from this list: send the line "unsubscribe linux-perf-users"
> in the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Milian Wolff | milian.wolff@kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5903 bytes --]