[Qemu-devel] [PATCH V5 0/6]

[Qemu-devel] [PATCH V5 0/6]

[Peter Lieven]

I decided to split this from the rest of the Qemu RSS usage series as
it contains the more or less non contentious patches.

I omitted the MAP_GROWSDOWN flag in mmap as we are not 100% sure which
side effects it has.

I kept the guard page which is now nicely makes the stacks visible in
smaps. The old version of the relevent patch lacked the MAP_FIXED flag
in the second call to mmap.

The last patch which reduces the stack size of coroutines to 64kB
may be omitted if its found to risky.

v4->v5:
 - Patch 1: check if _SC_THREAD_STACK_MIN is defined
 - Patch 1: guard against sysconf(_SC_THREAD_STACK_MIN) returning -1 [Eric]

v3->v4:
 - Patch 1: add a static function to adjust the stack size [Richard]
 - Patch 1: round up the stack size to multiple of the pagesize.

v2->v3:
 - Patch 1,6: adjusted commit message to mention the guard page [Markus]

v1->v2:
 - Patch 1: added an architecture dependend guard page [Richard]
 - Patch 1: avoid stacks smaller than _SC_THREAD_STACK_MIN [Richard]
 - Patch 1: use mmap+mprotect instead of mmap+mmap [Richard]
 - Patch 5: u_int32_t -> uint32_t [Richard]
 - Patch 5: only available if stack grows down

Peter Lieven (6):
  oslib-posix: add helpers for stack alloc and free
  coroutine: add a macro for the coroutine stack size
  coroutine-ucontext: use helper for allocating stack memory
  coroutine-sigaltstack: use helper for allocating stack memory
  oslib-posix: add a configure switch to debug stack usage
  coroutine: reduce stack size to 64kB

 configure                    | 19 ++++++++++
 include/qemu/coroutine_int.h |  2 ++
 include/sysemu/os-posix.h    | 23 ++++++++++++
 util/coroutine-sigaltstack.c |  7 ++--
 util/coroutine-ucontext.c    |  9 +++--
 util/coroutine-win32.c       |  2 +-
 util/oslib-posix.c           | 83 ++++++++++++++++++++++++++++++++++++++++++++
 7 files changed, 135 insertions(+), 10 deletions(-)

-- 
1.9.1

[Qemu-devel] [PATCH V5 1/6] oslib-posix: add helpers for stack alloc and free

[Peter Lieven]

the allocated stack will be adjusted to the minimum supported stack size
by the OS and rounded up to be a multiple of the system pagesize.
Additionally an architecture dependent guard page is added to the stack
to catch stack overflows.

Signed-off-by: Peter Lieven <pl@kamp.de>
---
 include/sysemu/os-posix.h | 23 +++++++++++++++++++++++
 util/oslib-posix.c        | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 69 insertions(+)

diff --git a/include/sysemu/os-posix.h b/include/sysemu/os-posix.h
index 9c7dfdf..7630665 100644
--- a/include/sysemu/os-posix.h
+++ b/include/sysemu/os-posix.h
@@ -60,4 +60,27 @@ int qemu_utimens(const char *path, const qemu_timespec *times);
 
 bool is_daemonized(void);
 
+/**
+ * qemu_alloc_stack:
+ * @sz: size of required stack in bytes
+ *
+ * Allocate memory that can be used as a stack, for instance for
+ * coroutines. If the memory cannot be allocated, this function
+ * will abort (like g_malloc()).
+ *
+ * The allocated stack must be freed with qemu_free_stack().
+ *
+ * Returns: pointer to (the lowest address of) the stack memory.
+ */
+void *qemu_alloc_stack(size_t sz);
+
+/**
+ * qemu_free_stack:
+ * @stack: stack to free
+ * @sz: size of stack in bytes
+ *
+ * Free a stack allocated via qemu_alloc_stack().
+ */
+void qemu_free_stack(void *stack, size_t sz);
+
 #endif
diff --git a/util/oslib-posix.c b/util/oslib-posix.c
index e2e1d4d..2303ca6 100644
--- a/util/oslib-posix.c
+++ b/util/oslib-posix.c
@@ -497,3 +497,49 @@ pid_t qemu_fork(Error **errp)
     }
     return pid;
 }
+
+static size_t adjust_stack_size(size_t sz)
+{
+#ifdef _SC_THREAD_STACK_MIN
+    /* avoid stacks smaller than _SC_THREAD_STACK_MIN */
+    sz = MAX(MAX(sysconf(_SC_THREAD_STACK_MIN), 0), sz);
+#endif
+    /* adjust stack size to a multiple of the page size */
+    sz = ROUND_UP(sz, getpagesize());
+    return sz;
+}
+
+void *qemu_alloc_stack(size_t sz)
+{
+    void *ptr, *guardpage;
+    size_t pagesz = getpagesize();
+    sz = adjust_stack_size(sz);
+
+    ptr = mmap(NULL, sz, PROT_READ | PROT_WRITE,
+               MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+    if (ptr == MAP_FAILED) {
+        abort();
+    }
+
+#if defined(HOST_IA64)
+    /* separate register stack */
+    guardpage = ptr + (((sz - pagesz) / 2) & ~pagesz);
+#elif defined(HOST_HPPA)
+    /* stack grows up */
+    guardpage = ptr + sz - pagesz;
+#else
+    /* stack grows down */
+    guardpage = ptr;
+#endif
+    if (mprotect(guardpage, pagesz, PROT_NONE) != 0) {
+        abort();
+    }
+
+    return ptr;
+}
+
+void qemu_free_stack(void *stack, size_t sz)
+{
+    sz = adjust_stack_size(sz);
+    munmap(stack, sz);
+}
-- 
1.9.1

Re: [Qemu-devel] [PATCH V5 1/6] oslib-posix: add helpers for stack alloc and free

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 998 bytes --]

On 07/12/2016 10:23 AM, Peter Lieven wrote:
> the allocated stack will be adjusted to the minimum supported stack size
> by the OS and rounded up to be a multiple of the system pagesize.
> Additionally an architecture dependent guard page is added to the stack
> to catch stack overflows.
> 
> Signed-off-by: Peter Lieven <pl@kamp.de>
> ---
>  include/sysemu/os-posix.h | 23 +++++++++++++++++++++++
>  util/oslib-posix.c        | 46 ++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 69 insertions(+)
> 

> +static size_t adjust_stack_size(size_t sz)
> +{
> +#ifdef _SC_THREAD_STACK_MIN
> +    /* avoid stacks smaller than _SC_THREAD_STACK_MIN */
> +    sz = MAX(MAX(sysconf(_SC_THREAD_STACK_MIN), 0), sz);

MAX(MAX(int, int), unsigned)

is not transitive, but does the job. I won't request a rewrite.

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH V5 1/6] oslib-posix: add helpers for stack alloc and free

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3159 bytes --]

On Tue, Jul 12, 2016 at 06:23:01PM +0200, Peter Lieven wrote:
> the allocated stack will be adjusted to the minimum supported stack size
> by the OS and rounded up to be a multiple of the system pagesize.
> Additionally an architecture dependent guard page is added to the stack
> to catch stack overflows.
> 
> Signed-off-by: Peter Lieven <pl@kamp.de>
> ---
>  include/sysemu/os-posix.h | 23 +++++++++++++++++++++++
>  util/oslib-posix.c        | 46 ++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 69 insertions(+)
> 
> diff --git a/include/sysemu/os-posix.h b/include/sysemu/os-posix.h
> index 9c7dfdf..7630665 100644
> --- a/include/sysemu/os-posix.h
> +++ b/include/sysemu/os-posix.h
> @@ -60,4 +60,27 @@ int qemu_utimens(const char *path, const qemu_timespec *times);
>  
>  bool is_daemonized(void);
>  
> +/**
> + * qemu_alloc_stack:
> + * @sz: size of required stack in bytes
> + *
> + * Allocate memory that can be used as a stack, for instance for
> + * coroutines. If the memory cannot be allocated, this function
> + * will abort (like g_malloc()).
> + *
> + * The allocated stack must be freed with qemu_free_stack().
> + *
> + * Returns: pointer to (the lowest address of) the stack memory.
> + */
> +void *qemu_alloc_stack(size_t sz);
> +
> +/**
> + * qemu_free_stack:
> + * @stack: stack to free
> + * @sz: size of stack in bytes
> + *
> + * Free a stack allocated via qemu_alloc_stack().
> + */
> +void qemu_free_stack(void *stack, size_t sz);
> +
>  #endif
> diff --git a/util/oslib-posix.c b/util/oslib-posix.c
> index e2e1d4d..2303ca6 100644
> --- a/util/oslib-posix.c
> +++ b/util/oslib-posix.c
> @@ -497,3 +497,49 @@ pid_t qemu_fork(Error **errp)
>      }
>      return pid;
>  }
> +
> +static size_t adjust_stack_size(size_t sz)
> +{
> +#ifdef _SC_THREAD_STACK_MIN
> +    /* avoid stacks smaller than _SC_THREAD_STACK_MIN */
> +    sz = MAX(MAX(sysconf(_SC_THREAD_STACK_MIN), 0), sz);
> +#endif
> +    /* adjust stack size to a multiple of the page size */
> +    sz = ROUND_UP(sz, getpagesize());
> +    return sz;
> +}
> +
> +void *qemu_alloc_stack(size_t sz)
> +{
> +    void *ptr, *guardpage;
> +    size_t pagesz = getpagesize();
> +    sz = adjust_stack_size(sz);
> +
> +    ptr = mmap(NULL, sz, PROT_READ | PROT_WRITE,

It's cleaner to count for the guard page separately and give the caller
the sz bytes they expected:

  sz + pagesz

> +               MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
> +    if (ptr == MAP_FAILED) {
> +        abort();
> +    }
> +
> +#if defined(HOST_IA64)
> +    /* separate register stack */
> +    guardpage = ptr + (((sz - pagesz) / 2) & ~pagesz);
> +#elif defined(HOST_HPPA)
> +    /* stack grows up */
> +    guardpage = ptr + sz - pagesz;
> +#else
> +    /* stack grows down */
> +    guardpage = ptr;
> +#endif
> +    if (mprotect(guardpage, pagesz, PROT_NONE) != 0) {
> +        abort();
> +    }
> +
> +    return ptr;
> +}
> +
> +void qemu_free_stack(void *stack, size_t sz)
> +{
> +    sz = adjust_stack_size(sz);
> +    munmap(stack, sz);
> +}
> -- 
> 1.9.1
> 
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH V5 1/6] oslib-posix: add helpers for stack alloc and free

[Peter Lieven]

Am 08.08.2016 um 12:37 schrieb Stefan Hajnoczi:
> On Tue, Jul 12, 2016 at 06:23:01PM +0200, Peter Lieven wrote:
>> the allocated stack will be adjusted to the minimum supported stack size
>> by the OS and rounded up to be a multiple of the system pagesize.
>> Additionally an architecture dependent guard page is added to the stack
>> to catch stack overflows.
>>
>> Signed-off-by: Peter Lieven <pl@kamp.de>
>> ---
>>  include/sysemu/os-posix.h | 23 +++++++++++++++++++++++
>>  util/oslib-posix.c        | 46 ++++++++++++++++++++++++++++++++++++++++++++++
>>  2 files changed, 69 insertions(+)
>>
>> diff --git a/include/sysemu/os-posix.h b/include/sysemu/os-posix.h
>> index 9c7dfdf..7630665 100644
>> --- a/include/sysemu/os-posix.h
>> +++ b/include/sysemu/os-posix.h
>> @@ -60,4 +60,27 @@ int qemu_utimens(const char *path, const qemu_timespec *times);
>>  
>>  bool is_daemonized(void);
>>  
>> +/**
>> + * qemu_alloc_stack:
>> + * @sz: size of required stack in bytes
>> + *
>> + * Allocate memory that can be used as a stack, for instance for
>> + * coroutines. If the memory cannot be allocated, this function
>> + * will abort (like g_malloc()).
>> + *
>> + * The allocated stack must be freed with qemu_free_stack().
>> + *
>> + * Returns: pointer to (the lowest address of) the stack memory.
>> + */
>> +void *qemu_alloc_stack(size_t sz);
>> +
>> +/**
>> + * qemu_free_stack:
>> + * @stack: stack to free
>> + * @sz: size of stack in bytes
>> + *
>> + * Free a stack allocated via qemu_alloc_stack().
>> + */
>> +void qemu_free_stack(void *stack, size_t sz);
>> +
>>  #endif
>> diff --git a/util/oslib-posix.c b/util/oslib-posix.c
>> index e2e1d4d..2303ca6 100644
>> --- a/util/oslib-posix.c
>> +++ b/util/oslib-posix.c
>> @@ -497,3 +497,49 @@ pid_t qemu_fork(Error **errp)
>>      }
>>      return pid;
>>  }
>> +
>> +static size_t adjust_stack_size(size_t sz)
>> +{
>> +#ifdef _SC_THREAD_STACK_MIN
>> +    /* avoid stacks smaller than _SC_THREAD_STACK_MIN */
>> +    sz = MAX(MAX(sysconf(_SC_THREAD_STACK_MIN), 0), sz);
>> +#endif
>> +    /* adjust stack size to a multiple of the page size */
>> +    sz = ROUND_UP(sz, getpagesize());
>> +    return sz;
>> +}
>> +
>> +void *qemu_alloc_stack(size_t sz)
>> +{
>> +    void *ptr, *guardpage;
>> +    size_t pagesz = getpagesize();
>> +    sz = adjust_stack_size(sz);
>> +
>> +    ptr = mmap(NULL, sz, PROT_READ | PROT_WRITE,
> It's cleaner to count for the guard page separately and give the caller
> the sz bytes they expected:
>
>   sz + pagesz

I don't mind to change that, but the glibc stack allocation routine
does it exactly this way. (see glibc/nptl/allocatestack.c).
As we already used other parts (minimal stack size, positioning
of the guard page), we mimicked this as well.

Peter

Re: [Qemu-devel] [PATCH V5 1/6] oslib-posix: add helpers for stack alloc and free

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3126 bytes --]

On Mon, Aug 08, 2016 at 08:29:58PM +0200, Peter Lieven wrote:
> Am 08.08.2016 um 12:37 schrieb Stefan Hajnoczi:
> > On Tue, Jul 12, 2016 at 06:23:01PM +0200, Peter Lieven wrote:
> >> the allocated stack will be adjusted to the minimum supported stack size
> >> by the OS and rounded up to be a multiple of the system pagesize.
> >> Additionally an architecture dependent guard page is added to the stack
> >> to catch stack overflows.
> >>
> >> Signed-off-by: Peter Lieven <pl@kamp.de>
> >> ---
> >>  include/sysemu/os-posix.h | 23 +++++++++++++++++++++++
> >>  util/oslib-posix.c        | 46 ++++++++++++++++++++++++++++++++++++++++++++++
> >>  2 files changed, 69 insertions(+)
> >>
> >> diff --git a/include/sysemu/os-posix.h b/include/sysemu/os-posix.h
> >> index 9c7dfdf..7630665 100644
> >> --- a/include/sysemu/os-posix.h
> >> +++ b/include/sysemu/os-posix.h
> >> @@ -60,4 +60,27 @@ int qemu_utimens(const char *path, const qemu_timespec *times);
> >>  
> >>  bool is_daemonized(void);
> >>  
> >> +/**
> >> + * qemu_alloc_stack:
> >> + * @sz: size of required stack in bytes
> >> + *
> >> + * Allocate memory that can be used as a stack, for instance for
> >> + * coroutines. If the memory cannot be allocated, this function
> >> + * will abort (like g_malloc()).
> >> + *
> >> + * The allocated stack must be freed with qemu_free_stack().
> >> + *
> >> + * Returns: pointer to (the lowest address of) the stack memory.
> >> + */
> >> +void *qemu_alloc_stack(size_t sz);
> >> +
> >> +/**
> >> + * qemu_free_stack:
> >> + * @stack: stack to free
> >> + * @sz: size of stack in bytes
> >> + *
> >> + * Free a stack allocated via qemu_alloc_stack().
> >> + */
> >> +void qemu_free_stack(void *stack, size_t sz);
> >> +
> >>  #endif
> >> diff --git a/util/oslib-posix.c b/util/oslib-posix.c
> >> index e2e1d4d..2303ca6 100644
> >> --- a/util/oslib-posix.c
> >> +++ b/util/oslib-posix.c
> >> @@ -497,3 +497,49 @@ pid_t qemu_fork(Error **errp)
> >>      }
> >>      return pid;
> >>  }
> >> +
> >> +static size_t adjust_stack_size(size_t sz)
> >> +{
> >> +#ifdef _SC_THREAD_STACK_MIN
> >> +    /* avoid stacks smaller than _SC_THREAD_STACK_MIN */
> >> +    sz = MAX(MAX(sysconf(_SC_THREAD_STACK_MIN), 0), sz);
> >> +#endif
> >> +    /* adjust stack size to a multiple of the page size */
> >> +    sz = ROUND_UP(sz, getpagesize());
> >> +    return sz;
> >> +}
> >> +
> >> +void *qemu_alloc_stack(size_t sz)
> >> +{
> >> +    void *ptr, *guardpage;
> >> +    size_t pagesz = getpagesize();
> >> +    sz = adjust_stack_size(sz);
> >> +
> >> +    ptr = mmap(NULL, sz, PROT_READ | PROT_WRITE,
> > It's cleaner to count for the guard page separately and give the caller
> > the sz bytes they expected:
> >
> >   sz + pagesz
> 
> I don't mind to change that, but the glibc stack allocation routine
> does it exactly this way. (see glibc/nptl/allocatestack.c).
> As we already used other parts (minimal stack size, positioning
> of the guard page), we mimicked this as well.

Okay but please include a comment explaining the rationale.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[Qemu-devel] [PATCH V5 2/6] coroutine: add a macro for the coroutine stack size

[Peter Lieven]

Reviewed-by: Richard Henderson <rth@twiddle.net>
Signed-off-by: Peter Lieven <pl@kamp.de>
---
 include/qemu/coroutine_int.h | 2 ++
 util/coroutine-sigaltstack.c | 2 +-
 util/coroutine-ucontext.c    | 2 +-
 util/coroutine-win32.c       | 2 +-
 4 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/include/qemu/coroutine_int.h b/include/qemu/coroutine_int.h
index 42d6838..eac323a 100644
--- a/include/qemu/coroutine_int.h
+++ b/include/qemu/coroutine_int.h
@@ -28,6 +28,8 @@
 #include "qemu/queue.h"
 #include "qemu/coroutine.h"
 
+#define COROUTINE_STACK_SIZE (1 << 20)
+
 typedef enum {
     COROUTINE_YIELD = 1,
     COROUTINE_TERMINATE = 2,
diff --git a/util/coroutine-sigaltstack.c b/util/coroutine-sigaltstack.c
index a7c3366..9c2854c 100644
--- a/util/coroutine-sigaltstack.c
+++ b/util/coroutine-sigaltstack.c
@@ -143,7 +143,7 @@ static void coroutine_trampoline(int signal)
 
 Coroutine *qemu_coroutine_new(void)
 {
-    const size_t stack_size = 1 << 20;
+    const size_t stack_size = COROUTINE_STACK_SIZE;
     CoroutineUContext *co;
     CoroutineThreadState *coTS;
     struct sigaction sa;
diff --git a/util/coroutine-ucontext.c b/util/coroutine-ucontext.c
index 2bb7e10..31254ab 100644
--- a/util/coroutine-ucontext.c
+++ b/util/coroutine-ucontext.c
@@ -82,7 +82,7 @@ static void coroutine_trampoline(int i0, int i1)
 
 Coroutine *qemu_coroutine_new(void)
 {
-    const size_t stack_size = 1 << 20;
+    const size_t stack_size = COROUTINE_STACK_SIZE;
     CoroutineUContext *co;
     ucontext_t old_uc, uc;
     sigjmp_buf old_env;
diff --git a/util/coroutine-win32.c b/util/coroutine-win32.c
index 02e28e8..de6bd4f 100644
--- a/util/coroutine-win32.c
+++ b/util/coroutine-win32.c
@@ -71,7 +71,7 @@ static void CALLBACK coroutine_trampoline(void *co_)
 
 Coroutine *qemu_coroutine_new(void)
 {
-    const size_t stack_size = 1 << 20;
+    const size_t stack_size = COROUTINE_STACK_SIZE;
     CoroutineWin32 *co;
 
     co = g_malloc0(sizeof(*co));
-- 
1.9.1

Re: [Qemu-devel] [PATCH V5 2/6] coroutine: add a macro for the coroutine stack size

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 462 bytes --]

On Tue, Jul 12, 2016 at 06:23:02PM +0200, Peter Lieven wrote:
> Reviewed-by: Richard Henderson <rth@twiddle.net>
> Signed-off-by: Peter Lieven <pl@kamp.de>
> ---
>  include/qemu/coroutine_int.h | 2 ++
>  util/coroutine-sigaltstack.c | 2 +-
>  util/coroutine-ucontext.c    | 2 +-
>  util/coroutine-win32.c       | 2 +-
>  4 files changed, 5 insertions(+), 3 deletions(-)

Thanks, applied to my master tree:
https://github.com/stefanha/qemu/commits/master

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH V5 2/6] coroutine: add a macro for the coroutine stack size

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 486 bytes --]

On Tue, Jul 12, 2016 at 06:23:02PM +0200, Peter Lieven wrote:
> Reviewed-by: Richard Henderson <rth@twiddle.net>
> Signed-off-by: Peter Lieven <pl@kamp.de>
> ---
>  include/qemu/coroutine_int.h | 2 ++
>  util/coroutine-sigaltstack.c | 2 +-
>  util/coroutine-ucontext.c    | 2 +-
>  util/coroutine-win32.c       | 2 +-
>  4 files changed, 5 insertions(+), 3 deletions(-)

Sorry, please ignore the spurious "Applied" email I just sent.

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[Qemu-devel] [PATCH V5 3/6] coroutine-ucontext: use helper for allocating stack memory

[Peter Lieven]

Reviewed-by: Richard Henderson <rth@twiddle.net>
Signed-off-by: Peter Lieven <pl@kamp.de>
---
 util/coroutine-ucontext.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/util/coroutine-ucontext.c b/util/coroutine-ucontext.c
index 31254ab..b7dea8c 100644
--- a/util/coroutine-ucontext.c
+++ b/util/coroutine-ucontext.c
@@ -82,7 +82,6 @@ static void coroutine_trampoline(int i0, int i1)
 
 Coroutine *qemu_coroutine_new(void)
 {
-    const size_t stack_size = COROUTINE_STACK_SIZE;
     CoroutineUContext *co;
     ucontext_t old_uc, uc;
     sigjmp_buf old_env;
@@ -101,17 +100,17 @@ Coroutine *qemu_coroutine_new(void)
     }
 
     co = g_malloc0(sizeof(*co));
-    co->stack = g_malloc(stack_size);
+    co->stack = qemu_alloc_stack(COROUTINE_STACK_SIZE);
     co->base.entry_arg = &old_env; /* stash away our jmp_buf */
 
     uc.uc_link = &old_uc;
     uc.uc_stack.ss_sp = co->stack;
-    uc.uc_stack.ss_size = stack_size;
+    uc.uc_stack.ss_size = COROUTINE_STACK_SIZE;
     uc.uc_stack.ss_flags = 0;
 
 #ifdef CONFIG_VALGRIND_H
     co->valgrind_stack_id =
-        VALGRIND_STACK_REGISTER(co->stack, co->stack + stack_size);
+        VALGRIND_STACK_REGISTER(co->stack, co->stack + COROUTINE_STACK_SIZE);
 #endif
 
     arg.p = co;
@@ -149,7 +148,7 @@ void qemu_coroutine_delete(Coroutine *co_)
     valgrind_stack_deregister(co);
 #endif
 
-    g_free(co->stack);
+    qemu_free_stack(co->stack, COROUTINE_STACK_SIZE);
     g_free(co);
 }
 
-- 
1.9.1

Re: [Qemu-devel] [PATCH V5 3/6] coroutine-ucontext: use helper for allocating stack memory

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 308 bytes --]

On Tue, Jul 12, 2016 at 06:23:03PM +0200, Peter Lieven wrote:
> Reviewed-by: Richard Henderson <rth@twiddle.net>
> Signed-off-by: Peter Lieven <pl@kamp.de>
> ---
>  util/coroutine-ucontext.c | 9 ++++-----
>  1 file changed, 4 insertions(+), 5 deletions(-)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[Qemu-devel] [PATCH V5 4/6] coroutine-sigaltstack: use helper for allocating stack memory

[Peter Lieven]

Reviewed-by: Richard Henderson <rth@twiddle.net>
Signed-off-by: Peter Lieven <pl@kamp.de>
---
 util/coroutine-sigaltstack.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/util/coroutine-sigaltstack.c b/util/coroutine-sigaltstack.c
index 9c2854c..ccf4861 100644
--- a/util/coroutine-sigaltstack.c
+++ b/util/coroutine-sigaltstack.c
@@ -143,7 +143,6 @@ static void coroutine_trampoline(int signal)
 
 Coroutine *qemu_coroutine_new(void)
 {
-    const size_t stack_size = COROUTINE_STACK_SIZE;
     CoroutineUContext *co;
     CoroutineThreadState *coTS;
     struct sigaction sa;
@@ -164,7 +163,7 @@ Coroutine *qemu_coroutine_new(void)
      */
 
     co = g_malloc0(sizeof(*co));
-    co->stack = g_malloc(stack_size);
+    co->stack = qemu_alloc_stack(COROUTINE_STACK_SIZE);
     co->base.entry_arg = &old_env; /* stash away our jmp_buf */
 
     coTS = coroutine_get_thread_state();
@@ -189,7 +188,7 @@ Coroutine *qemu_coroutine_new(void)
      * Set the new stack.
      */
     ss.ss_sp = co->stack;
-    ss.ss_size = stack_size;
+    ss.ss_size = COROUTINE_STACK_SIZE;
     ss.ss_flags = 0;
     if (sigaltstack(&ss, &oss) < 0) {
         abort();
@@ -253,7 +252,7 @@ void qemu_coroutine_delete(Coroutine *co_)
 {
     CoroutineUContext *co = DO_UPCAST(CoroutineUContext, base, co_);
 
-    g_free(co->stack);
+    qemu_free_stack(co->stack, COROUTINE_STACK_SIZE);
     g_free(co);
 }
 
-- 
1.9.1

Re: [Qemu-devel] [PATCH V5 4/6] coroutine-sigaltstack: use helper for allocating stack memory

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 309 bytes --]

On Tue, Jul 12, 2016 at 06:23:04PM +0200, Peter Lieven wrote:
> Reviewed-by: Richard Henderson <rth@twiddle.net>
> Signed-off-by: Peter Lieven <pl@kamp.de>
> ---
>  util/coroutine-sigaltstack.c | 7 +++----
>  1 file changed, 3 insertions(+), 4 deletions(-)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[Qemu-devel] [PATCH V5 5/6] oslib-posix: add a configure switch to debug stack usage

[Peter Lieven]

this adds a knob to track the maximum stack usage of stacks
created by qemu_alloc_stack.

Signed-off-by: Peter Lieven <pl@kamp.de>
---
 configure          | 19 +++++++++++++++++++
 util/oslib-posix.c | 37 +++++++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+)

diff --git a/configure b/configure
index 5ada56d..a7ee2f3 100755
--- a/configure
+++ b/configure
@@ -296,6 +296,7 @@ libiscsi=""
 libnfs=""
 coroutine=""
 coroutine_pool=""
+debug_stack_usage="no"
 seccomp=""
 glusterfs=""
 glusterfs_xlator_opt="no"
@@ -1005,6 +1006,8 @@ for opt do
   ;;
   --enable-coroutine-pool) coroutine_pool="yes"
   ;;
+  --enable-debug-stack-usage) debug_stack_usage="yes"
+  ;;
   --disable-docs) docs="no"
   ;;
   --enable-docs) docs="yes"
@@ -4302,6 +4305,17 @@ if test "$coroutine" = "gthread" -a "$coroutine_pool" = "yes"; then
   error_exit "'gthread' coroutine backend does not support pool (use --disable-coroutine-pool)"
 fi
 
+if test "$debug_stack_usage" = "yes"; then
+  if test "$cpu" = "ia64" -o "$cpu" = "hppa"; then
+    error_exit "stack usage debugging is not supported for $cpu"
+  fi
+  if test "$coroutine_pool" = "yes"; then
+    echo "WARN: disabling coroutine pool for stack usage debugging"
+    coroutine_pool=no
+  fi
+fi
+
+
 ##########################################
 # check if we have open_by_handle_at
 
@@ -4879,6 +4893,7 @@ echo "QGA MSI support   $guest_agent_msi"
 echo "seccomp support   $seccomp"
 echo "coroutine backend $coroutine"
 echo "coroutine pool    $coroutine_pool"
+echo "debug stack usage $debug_stack_usage"
 echo "GlusterFS support $glusterfs"
 echo "Archipelago support $archipelago"
 echo "gcov              $gcov_tool"
@@ -5347,6 +5362,10 @@ else
   echo "CONFIG_COROUTINE_POOL=0" >> $config_host_mak
 fi
 
+if test "$debug_stack_usage" = "yes" ; then
+  echo "CONFIG_DEBUG_STACK_USAGE=y" >> $config_host_mak
+fi
+
 if test "$open_by_handle_at" = "yes" ; then
   echo "CONFIG_OPEN_BY_HANDLE=y" >> $config_host_mak
 fi
diff --git a/util/oslib-posix.c b/util/oslib-posix.c
index 2303ca6..e818d38 100644
--- a/util/oslib-posix.c
+++ b/util/oslib-posix.c
@@ -50,6 +50,10 @@
 
 #include <qemu/mmap-alloc.h>
 
+#ifdef CONFIG_DEBUG_STACK_USAGE
+#include "qemu/error-report.h"
+#endif
+
 int qemu_get_thread_id(void)
 {
 #if defined(__linux__)
@@ -512,6 +516,9 @@ static size_t adjust_stack_size(size_t sz)
 void *qemu_alloc_stack(size_t sz)
 {
     void *ptr, *guardpage;
+#ifdef CONFIG_DEBUG_STACK_USAGE
+    void *ptr2;
+#endif
     size_t pagesz = getpagesize();
     sz = adjust_stack_size(sz);
 
@@ -535,11 +542,41 @@ void *qemu_alloc_stack(size_t sz)
         abort();
     }
 
+#ifdef CONFIG_DEBUG_STACK_USAGE
+    for (ptr2 = ptr + pagesz; ptr2 < ptr + sz; ptr2 += sizeof(uint32_t)) {
+        *(uint32_t *)ptr2 = 0xdeadbeaf;
+    }
+#endif
+
     return ptr;
 }
 
+#ifdef CONFIG_DEBUG_STACK_USAGE
+static __thread unsigned int max_stack_usage;
+#endif
+
 void qemu_free_stack(void *stack, size_t sz)
 {
+#ifdef CONFIG_DEBUG_STACK_USAGE
+    unsigned int usage;
+    void *ptr;
+#endif
     sz = adjust_stack_size(sz);
+
+#ifdef CONFIG_DEBUG_STACK_USAGE
+    for (ptr = stack + getpagesize(); ptr < stack + sz;
+         ptr += sizeof(uint32_t)) {
+        if (*(uint32_t *)ptr != 0xdeadbeaf) {
+            break;
+        }
+    }
+    usage = sz - (uintptr_t) (ptr - stack);
+    if (usage > max_stack_usage) {
+        error_report("thread %d max stack usage increased from %u to %u",
+                     qemu_get_thread_id(), max_stack_usage, usage);
+        max_stack_usage = usage;
+    }
+#endif
+
     munmap(stack, sz);
 }
-- 
1.9.1

Re: [Qemu-devel] [PATCH V5 5/6] oslib-posix: add a configure switch to debug stack usage

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 419 bytes --]

On Tue, Jul 12, 2016 at 06:23:05PM +0200, Peter Lieven wrote:
> this adds a knob to track the maximum stack usage of stacks
> created by qemu_alloc_stack.
> 
> Signed-off-by: Peter Lieven <pl@kamp.de>
> ---
>  configure          | 19 +++++++++++++++++++
>  util/oslib-posix.c | 37 +++++++++++++++++++++++++++++++++++++
>  2 files changed, 56 insertions(+)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[Qemu-devel] [PATCH V5 6/6] coroutine: reduce stack size to 64kB

[Peter Lieven]

evaluation with the recently introduced maximum stack usage monitoring revealed
that the actual used stack size was never above 4kB so allocating 1MB stack
for each coroutine is a lot of wasted memory. So reduce the stack size to
64kB which should still give enough head room. The guard page added
in qemu_alloc_stack will catch a potential stack overflow introduced
by this commit.

Signed-off-by: Peter Lieven <pl@kamp.de>
---
 include/qemu/coroutine_int.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/qemu/coroutine_int.h b/include/qemu/coroutine_int.h
index eac323a..f84d777 100644
--- a/include/qemu/coroutine_int.h
+++ b/include/qemu/coroutine_int.h
@@ -28,7 +28,7 @@
 #include "qemu/queue.h"
 #include "qemu/coroutine.h"
 
-#define COROUTINE_STACK_SIZE (1 << 20)
+#define COROUTINE_STACK_SIZE (1 << 16)
 
 typedef enum {
     COROUTINE_YIELD = 1,
-- 
1.9.1

Re: [Qemu-devel] [PATCH V5 6/6] coroutine: reduce stack size to 64kB

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1169 bytes --]

On 07/12/2016 10:23 AM, Peter Lieven wrote:
> evaluation with the recently introduced maximum stack usage monitoring revealed
> that the actual used stack size was never above 4kB so allocating 1MB stack
> for each coroutine is a lot of wasted memory. So reduce the stack size to
> 64kB which should still give enough head room. The guard page added
> in qemu_alloc_stack will catch a potential stack overflow introduced
> by this commit.
> 
> Signed-off-by: Peter Lieven <pl@kamp.de>
> ---
>  include/qemu/coroutine_int.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Reviewed-by: Eric Blake <eblake@redhat.com>

> 
> diff --git a/include/qemu/coroutine_int.h b/include/qemu/coroutine_int.h
> index eac323a..f84d777 100644
> --- a/include/qemu/coroutine_int.h
> +++ b/include/qemu/coroutine_int.h
> @@ -28,7 +28,7 @@
>  #include "qemu/queue.h"
>  #include "qemu/coroutine.h"
>  
> -#define COROUTINE_STACK_SIZE (1 << 20)
> +#define COROUTINE_STACK_SIZE (1 << 16)
>  
>  typedef enum {
>      COROUTINE_YIELD = 1,
> 

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH V5 6/6] coroutine: reduce stack size to 64kB

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 663 bytes --]

On Tue, Jul 12, 2016 at 06:23:06PM +0200, Peter Lieven wrote:
> evaluation with the recently introduced maximum stack usage monitoring revealed
> that the actual used stack size was never above 4kB so allocating 1MB stack
> for each coroutine is a lot of wasted memory. So reduce the stack size to
> 64kB which should still give enough head room. The guard page added
> in qemu_alloc_stack will catch a potential stack overflow introduced
> by this commit.
> 
> Signed-off-by: Peter Lieven <pl@kamp.de>
> ---
>  include/qemu/coroutine_int.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH V5 0/6] coroutine: mmap stack memory and stack size

[Peter Lieven]

Am 12.07.2016 um 18:23 schrieb Peter Lieven:
> I decided to split this from the rest of the Qemu RSS usage series as
> it contains the more or less non contentious patches.
>
> I omitted the MAP_GROWSDOWN flag in mmap as we are not 100% sure which
> side effects it has.
>
> I kept the guard page which is now nicely makes the stacks visible in
> smaps. The old version of the relevent patch lacked the MAP_FIXED flag
> in the second call to mmap.
>
> The last patch which reduces the stack size of coroutines to 64kB
> may be omitted if its found to risky.
>
> v4->v5:
>  - Patch 1: check if _SC_THREAD_STACK_MIN is defined
>  - Patch 1: guard against sysconf(_SC_THREAD_STACK_MIN) returning -1 [Eric]
>
> v3->v4:
>  - Patch 1: add a static function to adjust the stack size [Richard]
>  - Patch 1: round up the stack size to multiple of the pagesize.
>
> v2->v3:
>  - Patch 1,6: adjusted commit message to mention the guard page [Markus]
>
> v1->v2:
>  - Patch 1: added an architecture dependend guard page [Richard]
>  - Patch 1: avoid stacks smaller than _SC_THREAD_STACK_MIN [Richard]
>  - Patch 1: use mmap+mprotect instead of mmap+mmap [Richard]
>  - Patch 5: u_int32_t -> uint32_t [Richard]
>  - Patch 5: only available if stack grows down
>
> Peter Lieven (6):
>   oslib-posix: add helpers for stack alloc and free
>   coroutine: add a macro for the coroutine stack size
>   coroutine-ucontext: use helper for allocating stack memory
>   coroutine-sigaltstack: use helper for allocating stack memory
>   oslib-posix: add a configure switch to debug stack usage
>   coroutine: reduce stack size to 64kB
>
>  configure                    | 19 ++++++++++
>  include/qemu/coroutine_int.h |  2 ++
>  include/sysemu/os-posix.h    | 23 ++++++++++++
>  util/coroutine-sigaltstack.c |  7 ++--
>  util/coroutine-ucontext.c    |  9 +++--
>  util/coroutine-win32.c       |  2 +-
>  util/oslib-posix.c           | 83 ++++++++++++++++++++++++++++++++++++++++++++
>  7 files changed, 135 insertions(+), 10 deletions(-)
>

Ping.

Peter