Re: [Qemu-devel] [PATCH 03/15] coroutine-ucontext: reduce stack size to 64kB

[Peter Lieven <pl@kamp.de>]

Am 28.06.2016 um 13:35 schrieb Dr. David Alan Gilbert:
> * Peter Lieven (pl@kamp.de) wrote:
>> Am 28.06.2016 um 12:57 schrieb Dr. David Alan Gilbert:
>>> * Paolo Bonzini (pbonzini@redhat.com) wrote:
>>>> On 28/06/2016 11:01, Peter Lieven wrote:
>>>>> evaluation with the recently introduced maximum stack size monitoring revealed
>>>>> that the actual used stack size was never above 4kB so allocating 1MB stack
>>>>> for each coroutine is a lot of wasted memory. So reduce the stack size to
>>>>> 64kB which should still give enough head room.
>>>> If we make the stack this much smaller, there is a non-zero chance of
>>>> smashing it.  You must add a guard page if you do this (actually more
>>>> than one because QEMU will happily have stack frames as big as 16 KB).
>>>> The stack counts for RSS but it's not actually allocated memory, so why
>>>> does it matter?
>>> I think I'd be interested in seeing the /proc/.../smaps before and after this
>>> change to see if anything is visible and if we can see the difference
>>> in rss etc.
>> Can you advise what in smaps should be especially looked at.
>>
>> As for RSS I can report hat the long term usage is significantly lower.
>> I had the strange observation that when the VM is running for some minutes
>> the RSS suddenly increases to the whole stack size.
> You can see the Rss of each mapping; if you knew where your stacks were
> it would be easy to see if it was the stacks that were Rss and if
> there was anything else odd about them.
> If you set hte mapping as growsdown then you can see the area that has a 'gd'
> in it's VmFlags.

Would you expect to see each 1MB allocation in smaps or is it possible that
the kernel merges some mappings to bigger ones?

And more importantly if the regions are merged Paolos comment about we
do not need a guard page would not be true because a coroutine stack could
grow into annother coroutines stack. Looking at the commit from Linus it
would also be good to have that guard page not having the gd flag.

Some of the regions above 1024kB have an RSS of exactly 4kB * (Size / 1024kB)
which leads to the assumption that it is a corouine stack where exactly one page
has been allocated.

I am asking because this is what I e.g. see for a Qemu VM with flags "gd":

cat /proc/5031/smaps | grep -B18 gd
7f808aee7000-7f808b9e6000 rw-p 00000000 00:00 0
Size:              11264 kB
Rss:                  44 kB
Pss:                  44 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        44 kB
Referenced:           44 kB
Anonymous:            44 kB
AnonHugePages:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me gd ac sd
--
7f808bb01000-7f8090000000 rw-p 00000000 00:00 0
Size:              70656 kB
Rss:                 276 kB
Pss:                 276 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:       276 kB
Referenced:          276 kB
Anonymous:           276 kB
AnonHugePages:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me gd ac sd
--
7f80940ff000-7f80943fe000 rw-p 00000000 00:00 0
Size:               3072 kB
Rss:                  12 kB
Pss:                  12 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        12 kB
Referenced:           12 kB
Anonymous:            12 kB
AnonHugePages:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me gd ac sd
--
7f8095700000-7f80957ff000 rw-p 00000000 00:00 0
Size:               1024 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me gd ac sd
--
7f8097301000-7f8097400000 rw-p 00000000 00:00 0
Size:               1024 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me gd ac sd
--
7f80974df000-7f80975de000 rw-p 00000000 00:00 0
Size:               1024 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me gd ac sd
7f809760c000-7f809770b000 rw-p 00000000 00:00 0
Size:               1024 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me gd ac sd
--
7f8097901000-7f8097a00000 rw-p 00000000 00:00 0
Size:               1024 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me gd ac sd
--
7f8097b01000-7f8097c00000 rw-p 00000000 00:00 0
Size:               1024 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me gd ac sd
--
7f8097d01000-7f8097e00000 rw-p 00000000 00:00 0
Size:               1024 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me gd ac sd
--
7f8197f01000-7f8198000000 rw-p 00000000 00:00 0
Size:               1024 kB
Rss:                   4 kB
Pss:                   4 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         4 kB
Referenced:            4 kB
Anonymous:             4 kB
AnonHugePages:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me gd ac sd
--
7f81b4001000-7f81b4200000 rw-p 00000000 00:00 0
Size:               2048 kB
Rss:                  20 kB
Pss:                  20 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        20 kB
Referenced:           20 kB
Anonymous:            20 kB
AnonHugePages:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me gd ac sd
--
7ffd337e2000-7ffd33805000 rw-p 00000000 00:00 0                          [stack]
Size:                144 kB
Rss:                  64 kB
Pss:                  64 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:        64 kB
Referenced:           64 kB
Anonymous:            64 kB
AnonHugePages:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me gd ac

Peter