[Xenomai] Call to heapobj_pkg_init_private() in copperplate_init() locks unneeded pages into memory when CONFIG_XENO_PSHARED is set

[Xenomai] Call to heapobj_pkg_init_private() in copperplate_init() locks unneeded pages into memory when CONFIG_XENO_PSHARED is set

[Charles Kiorpes]

Hello,
I was noticing that my Xenomai 3.0.5 processes were consuming an
unusually large amount of physical memory space.

It appears that the call to heapobj_pkg_init_private() that occurs in
copperplate_init() is temporarily allocating a private heap of size
mem_pool_size in order to populate a static variable in
heapobj-tlsf.c, and then immediately deallocating this heap.  However,
this temporary allocation occurs after the call to mlockall() in
low_init(), so the pages remain locked in physical memory.  Because I
am using CONFIG_XENO_PSHARED, the memory pool is instead allocated out
of Linux shared memory, so these extra locked pages are never used.

>From my readthrough of the code, it appears that the call to
heapobj_pkg_init_private() doesn't need to do anything if
CONFIG_XENO_PSHARED is set, but I may be missing something.  Putting
in the following hack appears to fix the problem for me, and cause no
other problems in my particular use case:

diff --git a/lib/copperplate/heapobj-tlsf.c b/lib/copperplate/heapobj-tlsf.c
index e24db15..8990936 100644
--- a/lib/copperplate/heapobj-tlsf.c
+++ b/lib/copperplate/heapobj-tlsf.c
@@ -74,6 +74,12 @@ int heapobj_init_array_private(struct heapobj
*hobj, const char *name,
        return __bt(__heapobj_init_private(hobj, name, poolsz, NULL));
 }

+#ifdef CONFIG_XENO_PSHARED
+int heapobj_pkg_init_private(void)
+{
+       return 0;
+}
+#else
 int heapobj_pkg_init_private(void)
 {
        size_t size;
@@ -101,3 +107,4 @@ int heapobj_pkg_init_private(void)

        return 0;
 }
+#endif

I am running the latest "next" branch.

My config:
CONFIG_MMU=1
CONFIG_XENO_BUILD_ARGS=" 'CFLAGS=-march=armv7-a -mfpu=vfp3'
'LDFLAGS=-march=armv7-a -mfpu=vfp3' '--with-core=cobalt'
'--enable-pshared' '--host=arm-linux-gnueabihf' '-disable-debug'
'host_alias=arm-linux-gnueabihf'"
CONFIG_XENO_BUILD_STRING="x86_64-pc-linux-gnu"
CONFIG_XENO_COBALT=1
CONFIG_XENO_COMPILER="gcc version 6.2.1 20161016 (Linaro GCC 6.2-2016.11) "
CONFIG_XENO_DEFAULT_PERIOD=1000000
CONFIG_XENO_FORTIFY=1
CONFIG_XENO_HOST_STRING="arm-unknown-linux-gnueabihf"
CONFIG_XENO_LORES_CLOCK_DISABLED=1
CONFIG_XENO_PREFIX="/usr/xenomai"
CONFIG_XENO_PSHARED=1
CONFIG_XENO_RAW_CLOCK_ENABLED=1
CONFIG_XENO_REVISION_LEVEL=90
CONFIG_XENO_SANITY=1
CONFIG_XENO_TLSF=1
CONFIG_XENO_TLS_MODEL="initial-exec"
CONFIG_XENO_UAPI_LEVEL=15
CONFIG_XENO_VALGRIND_API=1
CONFIG_XENO_VERSION_MAJOR=3
CONFIG_XENO_VERSION_MINOR=0
CONFIG_XENO_VERSION_NAME="Xenomai -next"
CONFIG_XENO_VERSION_STRING="3.1-devel"
---
CONFIG_SMP is OFF
CONFIG_XENO_ASYNC_CANCEL is OFF
CONFIG_XENO_COPPERPLATE_CLOCK_RESTRICTED is OFF
CONFIG_XENO_DEBUG is OFF
CONFIG_XENO_DEBUG_FULL is OFF
CONFIG_XENO_LAZY_SETSCHED is OFF
CONFIG_XENO_LIBS_DLOPEN is OFF
CONFIG_XENO_MERCURY is OFF
CONFIG_XENO_REGISTRY is OFF
CONFIG_XENO_REGISTRY_ROOT is OFF
CONFIG_XENO_WORKAROUND_CONDVAR_PI is OFF
CONFIG_XENO_X86_VSYSCALL is OFF
---
PTHREAD_STACK_DEFAULT=65536
AUTOMATIC_BOOTSTRAP=1


Am I correct in my assessment that heapobj_pkg_init_private() doesn't
need to do anything when CONFIG_XENO_PSHARED is set, or are there
other code paths in which it would get called that my hack would
break?

Thank you in advance,
Charles Kiorpes

Re: [Xenomai] Call to heapobj_pkg_init_private() in copperplate_init() locks unneeded pages into memory when CONFIG_XENO_PSHARED is set

[Philippe Gerum]

On 11/01/2017 12:42 PM, Charles Kiorpes wrote:
> Hello,
> I was noticing that my Xenomai 3.0.5 processes were consuming an
> unusually large amount of physical memory space.
> 
> It appears that the call to heapobj_pkg_init_private() that occurs in
> copperplate_init() is temporarily allocating a private heap of size
> mem_pool_size in order to populate a static variable in
> heapobj-tlsf.c, and then immediately deallocating this heap.  However,
> this temporary allocation occurs after the call to mlockall() in
> low_init(), so the pages remain locked in physical memory.  Because I
> am using CONFIG_XENO_PSHARED, the memory pool is instead allocated out
> of Linux shared memory, so these extra locked pages are never used.
> 
>>From my readthrough of the code, it appears that the call to
> heapobj_pkg_init_private() doesn't need to do anything if
> CONFIG_XENO_PSHARED is set, but I may be missing something.  Putting
> in the following hack appears to fix the problem for me, and cause no
> other problems in my particular use case:
> 
> diff --git a/lib/copperplate/heapobj-tlsf.c b/lib/copperplate/heapobj-tlsf.c
> index e24db15..8990936 100644
> --- a/lib/copperplate/heapobj-tlsf.c
> +++ b/lib/copperplate/heapobj-tlsf.c

Good catch. However, some APIs may need to create private heaps along with a shared main heap, so we can't disable the init entirely. We can reduce the impact of creating the temporary heap in the shared case though. Could you try this alternative approach and let me know if that works for you?

diff --git a/lib/copperplate/heapobj-tlsf.c b/lib/copperplate/heapobj-tlsf.c
index e24db15..35999ae 100644
--- a/lib/copperplate/heapobj-tlsf.c
+++ b/lib/copperplate/heapobj-tlsf.c
@@ -19,6 +19,7 @@
 #include <errno.h>
 #include <stdio.h>
 #include <stdint.h>
+#include <unistd.h>
 #include "boilerplate/tlsf/tlsf.h"
 #include "copperplate/heapobj.h"
 #include "copperplate/debug.h"
@@ -76,7 +77,11 @@ int heapobj_init_array_private(struct heapobj *hobj, const char *name,
 
 int heapobj_pkg_init_private(void)
 {
-	size_t size;
+#ifdef CONFIG_XENO_PSHARED
+	size_t size = sysconf(_SC_PAGE_SIZE);
+#else
+	size_t size = __copperplate_setup_data.mem_pool;
+#endif
 	void *mem;
 
 	/*
@@ -88,8 +93,15 @@ int heapobj_pkg_init_private(void)
 	 * We include 1k of additional memory to cope with the
 	 * per-block overhead for an undefined number of individual
 	 * allocation requests. Ugly.
+	 *
+	 * CAUTION: in pshared mode, private heaps are subsidiary
+	 * storage pools, so no need to pre-commit as much memory as
+	 * we will be preallocating for the main shared pool,
+	 * especially with memory locking in effect. In that case,
+	 * creating a temporary single-page pool is enough to figure
+	 * out the allocation overhead.
 	 */
-	mem = tlsf_malloc(__copperplate_setup_data.mem_pool);
+	mem = tlsf_malloc(size);
 	size = init_memory_pool(__copperplate_setup_data.mem_pool, mem);
 	if (size == (size_t)-1)
 		panic("cannot initialize TLSF memory manager");

-- 
Philippe.

Re: [Xenomai] Call to heapobj_pkg_init_private() in copperplate_init() locks unneeded pages into memory when CONFIG_XENO_PSHARED is set

[Charles Kiorpes]

On Wed, Nov 1, 2017 at 4:58 PM, Philippe Gerum <rpm@xenomai.org> wrote:
> Good catch. However, some APIs may need to create private heaps along with a shared main heap, so we can't disable the init entirely. We can reduce the impact of creating the temporary heap in the shared case though. Could you try this alternative approach and let me know if that works for you?

Ah, ok.  Reducing the impact makes sense.

I did some testing with the provided patch.  I was seeing a segfault
in init_memory_pool(), because heapobj_pkg_init_private() passes
__copperplate_setup_data.mem_pool into init_memory_pool(), and also
uses that value to calculate the tlsf_pool_overhead.  Changing these
operations to use the reduced pool size eliminated the segfault.
Here's a modified version that makes sure that the variable pool size
is propagated throughout the function:

diff --git a/lib/copperplate/heapobj-tlsf.c b/lib/copperplate/heapobj-tlsf.c
index e24db15..3709852 100644
--- a/lib/copperplate/heapobj-tlsf.c
+++ b/lib/copperplate/heapobj-tlsf.c
@@ -19,6 +19,7 @@
 #include <errno.h>
 #include <stdio.h>
 #include <stdint.h>
+#include <unistd.h>
 #include "boilerplate/tlsf/tlsf.h"
 #include "copperplate/heapobj.h"
 #include "copperplate/debug.h"
@@ -76,7 +77,12 @@ int heapobj_init_array_private(struct heapobj
*hobj, const char *name,

 int heapobj_pkg_init_private(void)
 {
- size_t size;
+ #ifdef CONFIG_XENO_PSHARED
+ size_t alloc_size = sysconf(_SC_PAGE_SIZE);
+ #else
+ size_t alloc_size = __copperplate_setup_data.mem_pool;
+ #endif
+ size_t available_size;
  void *mem;

  /*
@@ -88,14 +94,21 @@ int heapobj_pkg_init_private(void)
  * We include 1k of additional memory to cope with the
  * per-block overhead for an undefined number of individual
  * allocation requests. Ugly.
+ *
+ * CAUTION: in pshared mode, private heaps are subsidiary
+ * storage pools, so no need to pre-commit as much memory as
+ * we will be preallocating for the main shared pool,
+ * especially with memory locking in effect. In that case,
+ * creating a temporary single-page pool is enough to figure
+ * out the allocation overhead.
  */
- mem = tlsf_malloc(__copperplate_setup_data.mem_pool);
- size = init_memory_pool(__copperplate_setup_data.mem_pool, mem);
- if (size == (size_t)-1)
+ mem = tlsf_malloc(alloc_size);
+ available_size = init_memory_pool(alloc_size, mem);
+ if (available_size == (size_t)-1)
  panic("cannot initialize TLSF memory manager");

  destroy_memory_pool(mem);
- tlsf_pool_overhead = __copperplate_setup_data.mem_pool - size;
+ tlsf_pool_overhead = alloc_size - available_size;
  tlsf_pool_overhead = (tlsf_pool_overhead + 1024) & ~15;
  tlsf_free(mem);


With these changes, my process-shared use case works fine, and the
actual memory usage matches my expectations.  I don't have test cases
for private heaps, so I can't speak to whether that functionality is
affected.

Thanks,
Charles Kiorpes

Re: [Xenomai] Call to heapobj_pkg_init_private() in copperplate_init() locks unneeded pages into memory when CONFIG_XENO_PSHARED is set

[Philippe Gerum]

On 11/02/2017 02:10 PM, Charles Kiorpes wrote:
> On Wed, Nov 1, 2017 at 4:58 PM, Philippe Gerum <rpm@xenomai.org> wrote:
>> Good catch. However, some APIs may need to create private heaps along with a shared main heap, so we can't disable the init entirely. We can reduce the impact of creating the temporary heap in the shared case though. Could you try this alternative approach and let me know if that works for you?
> 
> Ah, ok.  Reducing the impact makes sense.
> 
> I did some testing with the provided patch.  I was seeing a segfault
> in init_memory_pool(), because heapobj_pkg_init_private() passes
> __copperplate_setup_data.mem_pool into init_memory_pool(), and also
> uses that value to calculate the tlsf_pool_overhead.  Changing these
> operations to use the reduced pool size eliminated the segfault.
> Here's a modified version that makes sure that the variable pool size
> is propagated throughout the function:
> 
> diff --git a/lib/copperplate/heapobj-tlsf.c b/lib/copperplate/heapobj-tlsf.c
> index e24db15..3709852 100644
> --- a/lib/copperplate/heapobj-tlsf.c
> +++ b/lib/copperplate/heapobj-tlsf.c


Ok, let's merge this. Could you send back a complete patch formatted
with "git format-patch"? TIA,

-- 
Philippe.

Re: [Xenomai] Call to heapobj_pkg_init_private() in copperplate_init() locks unneeded pages into memory when CONFIG_XENO_PSHARED is set

[Charles Kiorpes]

> Ok, let's merge this. Could you send back a complete patch formatted
> with "git format-patch"? TIA,

The formatted patch has been sent to the mailing list, but it ended up
under a new thread.

Thanks for your help,
Charles Kiorpes