Ambigiuous thread stack annotation in /proc/pid/[s]maps

Ambigiuous thread stack annotation in /proc/pid/[s]maps

[Jan Glauber]

Commit b764375 added annotations of thread stacks. This annotation can be
wrong depending on the memory layout of a task (will probably only happen
under 32 bit).

If a large allocation happens before the creation of a thread you can get:

root@box:~# cat /proc/2032/maps 
08048000-08049000 r-xp 00000000 08:01 3407931    /root/pt
08049000-0804a000 r--p 00000000 08:01 3407931    /root/pt
0804a000-0804b000 rw-p 00001000 08:01 3407931    /root/pt
093b8000-093d9000 rw-p 00000000 00:00 0          [heap]
363fe000-363ff000 ---p 00000000 00:00 0 
363ff000-b6c00000 rw-p 00000000 00:00 0          [stack:2036]
b6c00000-b6c21000 rw-p 00000000 00:00 0 
b6c21000-b6d00000 ---p 00000000 00:00 0 
b6d31000-b6d32000 ---p 00000000 00:00 0 
b6d32000-b7534000 rw-p 00000000 00:00 0          [stack:2033]
b7534000-b76d7000 r-xp 00000000 08:01 3145793    /lib/i386-linux-gnu/libc-2.15.so
b76d7000-b76d9000 r--p 001a3000 08:01 3145793    /lib/i386-linux-gnu/libc-2.15.so
b76d9000-b76da000 rw-p 001a5000 08:01 3145793    /lib/i386-linux-gnu/libc-2.15.so
b76da000-b76dd000 rw-p 00000000 00:00 0 
b76dd000-b76f4000 r-xp 00000000 08:01 3149883    /lib/i386-linux-gnu/libpthread-2.15.so
b76f4000-b76f5000 r--p 00016000 08:01 3149883    /lib/i386-linux-gnu/libpthread-2.15.so
b76f5000-b76f6000 rw-p 00017000 08:01 3149883    /lib/i386-linux-gnu/libpthread-2.15.so
b76f6000-b76f8000 rw-p 00000000 00:00 0 
b7707000-b770b000 rw-p 00000000 00:00 0 
b770b000-b770c000 r-xp 00000000 00:00 0          [vdso]
b770c000-b772c000 r-xp 00000000 08:01 3149887    /lib/i386-linux-gnu/ld-2.15.so
b772c000-b772d000 r--p 0001f000 08:01 3149887    /lib/i386-linux-gnu/ld-2.15.so
b772d000-b772e000 rw-p 00020000 08:01 3149887    /lib/i386-linux-gnu/ld-2.15.so
bfb15000-bfb36000 rw-p 00000000 00:00 0          [stack]

Now the stack for thread 2036 is amazingly large for a pthread. Actually it is
the result of:
1. malloc(2048 * 1024 * 1024)
2. pthread_create (with attr=NULL)

My theory is that this can happen anytime because a thread stack is just created
with mmap (MAP_PRIVATE | MAP_ANONYMOUS). The 1. malloc also resulted in a mmap 
(libc fallback on brk failure) with the same flags (MAP_PRIVATE | MAP_ANONYMOUS).

The kernel does not care about thread stacks and tries to merge vma's which seems
to happen here and which is correct from the kernel POV.

Any ideas how that can be fixed? The only solution that comes to my mind
is to prevent merging vma's that are used for thread stacks. There is already a
flag (MAP_STACK) which is set by the libc for mmap'ing thread stacks but the
kernel currently does not care. If the kernel gets an mmap request with
MAP_STACK set we could mark the VMA and avoid merging it with others.

Unfortunately there seems to be no bit left in the vm_flags to store the
MAP_STACK information...

--Jan

Re: Ambigiuous thread stack annotation in /proc/pid/[s]maps

[Siddhesh Poyarekar]

On 26 June 2013 17:13, Jan Glauber <jan.glauber@gmail.com> wrote:
> Any ideas how that can be fixed? The only solution that comes to my mind
> is to prevent merging vma's that are used for thread stacks. There is already a
> flag (MAP_STACK) which is set by the libc for mmap'ing thread stacks but the
> kernel currently does not care. If the kernel gets an mmap request with
> MAP_STACK set we could mark the VMA and avoid merging it with others.
>
> Unfortunately there seems to be no bit left in the vm_flags to store the
> MAP_STACK information...

The annotations essentially point out that the vma contains the stack
and not that the vma *is* the stack.  We'd get similar output with
makecontext/getcontext, where the stack may just be a portion of
memory in another vma.  I don't remember if I had explicitly mentioned
that during the original discussion.

Siddhesh
--
http://siddhesh.in

Re: Ambigiuous thread stack annotation in /proc/pid/[s]maps

[Jan Glauber]

On Wed, Jun 26, 2013 at 10:05:41PM +0530, Siddhesh Poyarekar wrote:
> On 26 June 2013 17:13, Jan Glauber <jan.glauber@gmail.com> wrote:
> > Any ideas how that can be fixed? The only solution that comes to my mind
> > is to prevent merging vma's that are used for thread stacks. There is already a
> > flag (MAP_STACK) which is set by the libc for mmap'ing thread stacks but the
> > kernel currently does not care. If the kernel gets an mmap request with
> > MAP_STACK set we could mark the VMA and avoid merging it with others.
> >
> > Unfortunately there seems to be no bit left in the vm_flags to store the
> > MAP_STACK information...
> 
> The annotations essentially point out that the vma contains the stack
> and not that the vma *is* the stack.  We'd get similar output with

But isn't that confusing to the user? At least it is to me. Imagine someone
who uses the maps or smaps output to determine the size of code, data and
stack of a process. Maybe it would be better to not print the stack:tid data
at all if the kernel cannot distinguish the vma's?
Is that behaviour documented anywhere?

> makecontext/getcontext, where the stack may just be a portion of
> memory in another vma.  I don't remember if I had explicitly mentioned
> that during the original discussion.

Never seen that makecontext stuff before. Do you have an example output how
the maps would look like if that is used?

--Jan
 
> Siddhesh
> --
> http://siddhesh.in

Re: Ambigiuous thread stack annotation in /proc/pid/[s]maps

[Siddhesh Poyarekar]

On 27 June 2013 21:32, Jan Glauber <jan.glauber@gmail.com> wrote:
> But isn't that confusing to the user? At least it is to me. Imagine someone
> who uses the maps or smaps output to determine the size of code, data and
> stack of a process. Maybe it would be better to not print the stack:tid data
> at all if the kernel cannot distinguish the vma's?
> Is that behaviour documented anywhere?

I'm afraid the documentation update I made to proc.txt did not mention
this.  In fact I went through the discussion thread and I don't think
I or anyone mentioned this in the thread either.  I think I assumed
back then that this was accepted since there was a point made that
vmas used by makecontext/swapcontext should also get marked correctly
with [stack].

However, I agree that you have a point about it being misleading.
Avoiding a vma merge is a possible solution, but we don't have flags
available any more to do that.  I'll try to think of another way.  In
the mean time I could add a note to the proc.txt documentation and
even adjust the language.  Would that be good enough or do you think
the patch should be reverted until I or someone else comes up with a
better solution?

> Never seen that makecontext stuff before. Do you have an example output how
> the maps would look like if that is used?

This is a sample program I cribbed from the makecontext man page and
modified slightly.  The stack is in the data area in this example.
There could be a case of a stack being with the main program stack or
even in the heap.

#include <ucontext.h>
#include <stdio.h>
#include <stdlib.h>

static ucontext_t uctx_main, uctx_func1, uctx_func2;

#define handle_error(msg) \
           do { perror(msg); exit(EXIT_FAILURE); } while (0)

static char func1_stack[16384];
static char func2_stack[16384];

static void func1(void)
{
        printf("func1: started\n");
        printf("func1: swapcontext(&uctx_func1, &uctx_func2)\n");
        if (swapcontext(&uctx_func1, &uctx_func2) == -1)
                handle_error("swapcontext");

        sleep (1);
        printf("func1: returning\n");
}

static void func2(void)
{
        printf("func2: started\n");
        printf("func2: swapcontext(&uctx_func2, &uctx_func1)\n");
        if (swapcontext(&uctx_func2, &uctx_func1) == -1)
                handle_error("swapcontext");
        sleep (1);
        printf("func2: returning\n");
}

int main(int argc, char *argv[])
{
        if (getcontext(&uctx_func1) == -1)
                handle_error("getcontext");
        uctx_func1.uc_stack.ss_sp = func1_stack;
        uctx_func1.uc_stack.ss_size = sizeof(func1_stack);
        uctx_func1.uc_link = &uctx_main;
        makecontext(&uctx_func1, func1, 0);

        if (getcontext(&uctx_func2) == -1)
                handle_error("getcontext");
        uctx_func2.uc_stack.ss_sp = func2_stack;
        uctx_func2.uc_stack.ss_size = sizeof(func2_stack);
        /* Successor context is f1(), unless argc > 1 */
        uctx_func2.uc_link = (argc > 1) ? NULL : &uctx_func1;
        makecontext(&uctx_func2, func2, 0);

        printf("main: swapcontext(&uctx_main, &uctx_func2)\n");
        if (swapcontext(&uctx_main, &uctx_func2) == -1)
                handle_error("swapcontext");

        printf("main: exiting\n");
        exit(EXIT_SUCCESS);
}


$./a.out
$ cat /proc/$(pgrep a.out)/maps

00400000-00401000 r-xp 00000000 fd:00 1704114
  /tmp/a.out
00601000-00602000 rw-p 00001000 fd:00 1704114
  /tmp/a.out
00602000-0060a000 rw-p 00000000 00:00 0
  [stack:7352]
7fac662d2000-7fac6647e000 r-xp 00000000 fd:00 524922
  /usr/lib64/libc-2.15.so
7fac6647e000-7fac6667e000 ---p 001ac000 fd:00 524922
  /usr/lib64/libc-2.15.so
7fac6667e000-7fac66682000 r--p 001ac000 fd:00 524922
  /usr/lib64/libc-2.15.so
7fac66682000-7fac66684000 rw-p 001b0000 fd:00 524922
  /usr/lib64/libc-2.15.so
7fac66684000-7fac66689000 rw-p 00000000 00:00 0
7fac66689000-7fac666a9000 r-xp 00000000 fd:00 524913
  /usr/lib64/ld-2.15.so
7fac6688f000-7fac66892000 rw-p 00000000 00:00 0
7fac668a6000-7fac668a8000 rw-p 00000000 00:00 0
7fac668a8000-7fac668a9000 r--p 0001f000 fd:00 524913
  /usr/lib64/ld-2.15.so
7fac668a9000-7fac668aa000 rw-p 00020000 fd:00 524913
  /usr/lib64/ld-2.15.so
7fac668aa000-7fac668ab000 rw-p 00000000 00:00 0
7fff5b8a8000-7fff5b8ca000 rw-p 00000000 00:00 0
7fff5b9fe000-7fff5ba00000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
  [vsyscall]


--
http://siddhesh.in

Re: Ambigiuous thread stack annotation in /proc/pid/[s]maps

[Jan Glauber]

On Thu, Jun 27, 2013 at 10:00:51PM +0530, Siddhesh Poyarekar wrote:
> On 27 June 2013 21:32, Jan Glauber <jan.glauber@gmail.com> wrote:
> > But isn't that confusing to the user? At least it is to me. Imagine someone
> > who uses the maps or smaps output to determine the size of code, data and
> > stack of a process. Maybe it would be better to not print the stack:tid data
> > at all if the kernel cannot distinguish the vma's?
> > Is that behaviour documented anywhere?
> 
> I'm afraid the documentation update I made to proc.txt did not mention
> this.  In fact I went through the discussion thread and I don't think
> I or anyone mentioned this in the thread either.  I think I assumed
> back then that this was accepted since there was a point made that
> vmas used by makecontext/swapcontext should also get marked correctly
> with [stack].
> 
> However, I agree that you have a point about it being misleading.
> Avoiding a vma merge is a possible solution, but we don't have flags
> available any more to do that.  I'll try to think of another way.  In
> the mean time I could add a note to the proc.txt documentation and
> even adjust the language.  Would that be good enough or do you think
> the patch should be reverted until I or someone else comes up with a
> better solution?

I think we should try to solve the problem first. If it turns out that
it is not possible to prevent these vma merges than we should document it.
Removing the annotation would IMHO be bad since it is really useful
information to the user.

Can anyone from the mm folks comment on the vm_flags situation (CC-ing linux-mm) ?
Would it be possible to re-use one of the defined flags? Or should we come up with a
crude hack?
 
> > Never seen that makecontext stuff before. Do you have an example output how
> > the maps would look like if that is used?
> 
> This is a sample program I cribbed from the makecontext man page and
> modified slightly.  The stack is in the data area in this example.
> There could be a case of a stack being with the main program stack or
> even in the heap.
> 
> #include <ucontext.h>
> #include <stdio.h>
> #include <stdlib.h>
> 
> static ucontext_t uctx_main, uctx_func1, uctx_func2;
> 
> #define handle_error(msg) \
>            do { perror(msg); exit(EXIT_FAILURE); } while (0)
> 
> static char func1_stack[16384];
> static char func2_stack[16384];
> 
> static void func1(void)
> {
>         printf("func1: started\n");
>         printf("func1: swapcontext(&uctx_func1, &uctx_func2)\n");
>         if (swapcontext(&uctx_func1, &uctx_func2) == -1)
>                 handle_error("swapcontext");
> 
>         sleep (1);
>         printf("func1: returning\n");
> }
> 
> static void func2(void)
> {
>         printf("func2: started\n");
>         printf("func2: swapcontext(&uctx_func2, &uctx_func1)\n");
>         if (swapcontext(&uctx_func2, &uctx_func1) == -1)
>                 handle_error("swapcontext");
>         sleep (1);
>         printf("func2: returning\n");
> }
> 
> int main(int argc, char *argv[])
> {
>         if (getcontext(&uctx_func1) == -1)
>                 handle_error("getcontext");
>         uctx_func1.uc_stack.ss_sp = func1_stack;
>         uctx_func1.uc_stack.ss_size = sizeof(func1_stack);
>         uctx_func1.uc_link = &uctx_main;
>         makecontext(&uctx_func1, func1, 0);
> 
>         if (getcontext(&uctx_func2) == -1)
>                 handle_error("getcontext");
>         uctx_func2.uc_stack.ss_sp = func2_stack;
>         uctx_func2.uc_stack.ss_size = sizeof(func2_stack);
>         /* Successor context is f1(), unless argc > 1 */
>         uctx_func2.uc_link = (argc > 1) ? NULL : &uctx_func1;
>         makecontext(&uctx_func2, func2, 0);
> 
>         printf("main: swapcontext(&uctx_main, &uctx_func2)\n");
>         if (swapcontext(&uctx_main, &uctx_func2) == -1)
>                 handle_error("swapcontext");
> 
>         printf("main: exiting\n");
>         exit(EXIT_SUCCESS);
> }
> 
> 
> $./a.out
> $ cat /proc/$(pgrep a.out)/maps
> 
> 00400000-00401000 r-xp 00000000 fd:00 1704114
>   /tmp/a.out
> 00601000-00602000 rw-p 00001000 fd:00 1704114
>   /tmp/a.out
> 00602000-0060a000 rw-p 00000000 00:00 0
>   [stack:7352]
> 7fac662d2000-7fac6647e000 r-xp 00000000 fd:00 524922
>   /usr/lib64/libc-2.15.so
> 7fac6647e000-7fac6667e000 ---p 001ac000 fd:00 524922
>   /usr/lib64/libc-2.15.so
> 7fac6667e000-7fac66682000 r--p 001ac000 fd:00 524922
>   /usr/lib64/libc-2.15.so
> 7fac66682000-7fac66684000 rw-p 001b0000 fd:00 524922
>   /usr/lib64/libc-2.15.so
> 7fac66684000-7fac66689000 rw-p 00000000 00:00 0
> 7fac66689000-7fac666a9000 r-xp 00000000 fd:00 524913
>   /usr/lib64/ld-2.15.so
> 7fac6688f000-7fac66892000 rw-p 00000000 00:00 0
> 7fac668a6000-7fac668a8000 rw-p 00000000 00:00 0
> 7fac668a8000-7fac668a9000 r--p 0001f000 fd:00 524913
>   /usr/lib64/ld-2.15.so
> 7fac668a9000-7fac668aa000 rw-p 00020000 fd:00 524913
>   /usr/lib64/ld-2.15.so
> 7fac668aa000-7fac668ab000 rw-p 00000000 00:00 0
> 7fff5b8a8000-7fff5b8ca000 rw-p 00000000 00:00 0
> 7fff5b9fe000-7fff5ba00000 r-xp 00000000 00:00 0                          [vdso]
> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
>   [vsyscall]
> 
> 
> --
> http://siddhesh.in

Re: Ambigiuous thread stack annotation in /proc/pid/[s]maps

[Jan Glauber]

On Thu, Jun 27, 2013 at 10:00:51PM +0530, Siddhesh Poyarekar wrote:
> On 27 June 2013 21:32, Jan Glauber <jan.glauber@gmail.com> wrote:
> > But isn't that confusing to the user? At least it is to me. Imagine someone
> > who uses the maps or smaps output to determine the size of code, data and
> > stack of a process. Maybe it would be better to not print the stack:tid data
> > at all if the kernel cannot distinguish the vma's?
> > Is that behaviour documented anywhere?
> 
> I'm afraid the documentation update I made to proc.txt did not mention
> this.  In fact I went through the discussion thread and I don't think
> I or anyone mentioned this in the thread either.  I think I assumed
> back then that this was accepted since there was a point made that
> vmas used by makecontext/swapcontext should also get marked correctly
> with [stack].
> 
> However, I agree that you have a point about it being misleading.
> Avoiding a vma merge is a possible solution, but we don't have flags
> available any more to do that.  I'll try to think of another way.  In
> the mean time I could add a note to the proc.txt documentation and
> even adjust the language.  Would that be good enough or do you think
> the patch should be reverted until I or someone else comes up with a
> better solution?

I think we should try to solve the problem first. If it turns out that
it is not possible to prevent these vma merges than we should document it.
Removing the annotation would IMHO be bad since it is really useful
information to the user.

Can anyone from the mm folks comment on the vm_flags situation (CC-ing linux-mm) ?
Would it be possible to re-use one of the defined flags? Or should we come up with a
crude hack?
 
> > Never seen that makecontext stuff before. Do you have an example output how
> > the maps would look like if that is used?
> 
> This is a sample program I cribbed from the makecontext man page and
> modified slightly.  The stack is in the data area in this example.
> There could be a case of a stack being with the main program stack or
> even in the heap.
> 
> #include <ucontext.h>
> #include <stdio.h>
> #include <stdlib.h>
> 
> static ucontext_t uctx_main, uctx_func1, uctx_func2;
> 
> #define handle_error(msg) \
>            do { perror(msg); exit(EXIT_FAILURE); } while (0)
> 
> static char func1_stack[16384];
> static char func2_stack[16384];
> 
> static void func1(void)
> {
>         printf("func1: started\n");
>         printf("func1: swapcontext(&uctx_func1, &uctx_func2)\n");
>         if (swapcontext(&uctx_func1, &uctx_func2) == -1)
>                 handle_error("swapcontext");
> 
>         sleep (1);
>         printf("func1: returning\n");
> }
> 
> static void func2(void)
> {
>         printf("func2: started\n");
>         printf("func2: swapcontext(&uctx_func2, &uctx_func1)\n");
>         if (swapcontext(&uctx_func2, &uctx_func1) == -1)
>                 handle_error("swapcontext");
>         sleep (1);
>         printf("func2: returning\n");
> }
> 
> int main(int argc, char *argv[])
> {
>         if (getcontext(&uctx_func1) == -1)
>                 handle_error("getcontext");
>         uctx_func1.uc_stack.ss_sp = func1_stack;
>         uctx_func1.uc_stack.ss_size = sizeof(func1_stack);
>         uctx_func1.uc_link = &uctx_main;
>         makecontext(&uctx_func1, func1, 0);
> 
>         if (getcontext(&uctx_func2) == -1)
>                 handle_error("getcontext");
>         uctx_func2.uc_stack.ss_sp = func2_stack;
>         uctx_func2.uc_stack.ss_size = sizeof(func2_stack);
>         /* Successor context is f1(), unless argc > 1 */
>         uctx_func2.uc_link = (argc > 1) ? NULL : &uctx_func1;
>         makecontext(&uctx_func2, func2, 0);
> 
>         printf("main: swapcontext(&uctx_main, &uctx_func2)\n");
>         if (swapcontext(&uctx_main, &uctx_func2) == -1)
>                 handle_error("swapcontext");
> 
>         printf("main: exiting\n");
>         exit(EXIT_SUCCESS);
> }
> 
> 
> $./a.out
> $ cat /proc/$(pgrep a.out)/maps
> 
> 00400000-00401000 r-xp 00000000 fd:00 1704114
>   /tmp/a.out
> 00601000-00602000 rw-p 00001000 fd:00 1704114
>   /tmp/a.out
> 00602000-0060a000 rw-p 00000000 00:00 0
>   [stack:7352]
> 7fac662d2000-7fac6647e000 r-xp 00000000 fd:00 524922
>   /usr/lib64/libc-2.15.so
> 7fac6647e000-7fac6667e000 ---p 001ac000 fd:00 524922
>   /usr/lib64/libc-2.15.so
> 7fac6667e000-7fac66682000 r--p 001ac000 fd:00 524922
>   /usr/lib64/libc-2.15.so
> 7fac66682000-7fac66684000 rw-p 001b0000 fd:00 524922
>   /usr/lib64/libc-2.15.so
> 7fac66684000-7fac66689000 rw-p 00000000 00:00 0
> 7fac66689000-7fac666a9000 r-xp 00000000 fd:00 524913
>   /usr/lib64/ld-2.15.so
> 7fac6688f000-7fac66892000 rw-p 00000000 00:00 0
> 7fac668a6000-7fac668a8000 rw-p 00000000 00:00 0
> 7fac668a8000-7fac668a9000 r--p 0001f000 fd:00 524913
>   /usr/lib64/ld-2.15.so
> 7fac668a9000-7fac668aa000 rw-p 00020000 fd:00 524913
>   /usr/lib64/ld-2.15.so
> 7fac668aa000-7fac668ab000 rw-p 00000000 00:00 0
> 7fff5b8a8000-7fff5b8ca000 rw-p 00000000 00:00 0
> 7fff5b9fe000-7fff5ba00000 r-xp 00000000 00:00 0                          [vdso]
> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
>   [vsyscall]
> 
> 
> --
> http://siddhesh.in

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>