[Xenomai] what limits number of pSOS tasks using t_create?

[Xenomai] what limits number of pSOS tasks using t_create?

[Marcel Van Mierlo]

Hi,


Which parameters limit the number of psos skin tasks that can be created using t_create()? (ARM (Beagle), Xenomai 3.0).

At the moment I am getting ERR_NOTCB back after creating about 118 tasks (just calling t_create not t_start). In production we are hitting the limit at about 70 tasks...though sometimes we can go a bit higher.

>From the documentation I see:

* t_create() returns ERR_NOTCB upon memory allocation failure when
trying to get a task control block from the nucleus heap. There is
no control block limit for tasks.


So I adjusted the following:

CONFIG_XENO_OPT_SYS_HEAPSZ=1024
CONFIG_XENO_OPT_PRIVATE_HEAPSZ=128
CONFIG_XENO_OPT_SHARED_HEAPSZ=128

but increasing (doubling) these made no difference to maximum number of tasks which can be created...


This is the guts of the test:

      for (i = 0; i < limit; i++)
      {
        result = t_create("MAX_THREAD_TEST", 200, 0x1000, 0x1000, T_LOCAL, &tids[i]);
        if (result != 0)
          break;
      }

Re: [Xenomai] what limits number of pSOS tasks using t_create?

[Philippe Gerum]

On 05/13/2016 03:27 PM, Marcel Van Mierlo wrote:
> Hi,
> 
> 
> Which parameters limit the number of psos skin tasks that can be created using t_create()? (ARM (Beagle), Xenomai 3.0).
> 
> At the moment I am getting ERR_NOTCB back after creating about 118 tasks (just calling t_create not t_start). In production we are hitting the limit at about 70 tasks...though sometimes we can go a bit higher.
> 
>>From the documentation I see:
> 
> * t_create() returns ERR_NOTCB upon memory allocation failure when
> trying to get a task control block from the nucleus heap. There is
> no control block limit for tasks.
> 
> 
> So I adjusted the following:
> 
> CONFIG_XENO_OPT_SYS_HEAPSZ=1024
> CONFIG_XENO_OPT_PRIVATE_HEAPSZ=128
> CONFIG_XENO_OPT_SHARED_HEAPSZ=128
> 

You likely need to raise CONFIG_XENO_OPT_REGISTRY_NRSLOTS.

-- 
Philippe.

Re: [Xenomai] what limits number of pSOS tasks using t_create?

[Marcel Van Mierlo]

Hi Philippe, yes that worked.

I made a regression test on the maximum number of psos threads I could create. I found that once I start seeing ERR_NOTCB returned from t_create the maximum number of tasks I can create then drops, even after deleting all previous tasks - as if available memory for tasks becomes less after limit is hit.

For example, after tuning all four parameters below (including CONFIG_XENO_OPT_REGISTRY_NRSLOTS) I am able to reliably create 500 tasks under test conditions - if I push it to 600, say, and see ERR_NOTCB, then after deleting all tasks I can only create a maximum of around 300 tasks.

This is fine for me - I know I will never need more than 150 tasks, but seems something worth noting.



Cheers,
Marcel.
________________________________________
From: Philippe Gerum <rpm@xenomai.org>
Sent: 02 June 2016 14:44:44
To: Marcel Van Mierlo; xenomai@xenomai.org
Subject: Re: [Xenomai] what limits number of pSOS tasks using t_create?

On 05/13/2016 03:27 PM, Marcel Van Mierlo wrote:
> Hi,
>
>
> Which parameters limit the number of psos skin tasks that can be created using t_create()? (ARM (Beagle), Xenomai 3.0).
>
> At the moment I am getting ERR_NOTCB back after creating about 118 tasks (just calling t_create not t_start). In production we are hitting the limit at about 70 tasks...though sometimes we can go a bit higher.
>
>>From the documentation I see:
>
> * t_create() returns ERR_NOTCB upon memory allocation failure when
> trying to get a task control block from the nucleus heap. There is
> no control block limit for tasks.
>
>
> So I adjusted the following:
>
> CONFIG_XENO_OPT_SYS_HEAPSZ=1024
> CONFIG_XENO_OPT_PRIVATE_HEAPSZ=128
> CONFIG_XENO_OPT_SHARED_HEAPSZ=128
>

You likely need to raise CONFIG_XENO_OPT_REGISTRY_NRSLOTS.

--
Philippe.

Re: [Xenomai] what limits number of pSOS tasks using t_create?

[Philippe Gerum]

Hi,

On 06/06/2016 10:14 AM, Marcel Van Mierlo wrote:
> 
> Hi Philippe, yes that worked.
> 
> I made a regression test on the maximum number of psos threads I could create. I found that once I start seeing ERR_NOTCB returned from t_create the maximum number of tasks I can create then drops, even after deleting all previous tasks - as if available memory for tasks becomes less after limit is hit.
> 
> For example, after tuning all four parameters below (including CONFIG_XENO_OPT_REGISTRY_NRSLOTS) I am able to reliably create 500 tasks under test conditions - if I push it to 600, say, and see ERR_NOTCB, then after deleting all tasks I can only create a maximum of around 300 tasks.
> 
> This is fine for me - I know I will never need more than 150 tasks, but seems something worth noting.
> 

I cannot reproduce that one yet. A few questions:

- are you starting the tasks or only creating them before calling
t_delete() on the tid?

- do you observe this behavior when respawning the test program after
each creation+deletion sequence, or within a single execution of the
test program running multiple sequences?

- can you paste the output of ./your_program --dump-config?

Thanks,

-- 
Philippe.

Re: [Xenomai] what limits number of pSOS tasks using t_create?

[Marcel Van Mierlo]

Hi Philippe,

I'm just creating the tasks, not starting them. I am using googletest as a unit-test framework.

I see this during a single invocation of the test application. When I run the test again (on same running instance of Xenomai), the results are repeated. So seems related to resource usage of application instance only, and not global resources.

Regression tests, below. I ran them longer and notice that the maximum number of tasks I can create tends to increase between tests after being hit the first time...

MaxThreadsLimit_Pre and MaxThreadsLimit_Post are identical and create what should be a safe number of threads (750). MaxThreads tries to create 5000 threads and fails.
MaxThreadsLimit_Pre always succeeds. MaxThreads and MaxThreads_Post always fail - see test results, below.

Issue could be with memory management with googletest...but seems unlikely (as far as I can tell it isn't spawning threads).

If I monitor /proc/xenomai/heap, I see free "system heap" get pretty low during MasThreadsLimit_Pre, but interestingly it seems to generally stay higher for the other two tests...free shared and private heap stay healthy. As if some other internal resource is not getting returned to a free pool.

Here is the unit-test code. The test framework state is reset between invocation of each TEST instance, but they all run in the same application instance.

 11 #define MAXTHREAD_ITERATIONS  20
 12 #define NUM_THREADS_LIMIT     750
 13 #define NUM_THREADS           5000
 14
 15 TEST(Threads, MaxThreadsLimit_Pre) {
 16   int i;
 17
 18   for (i = 0; i < MAXTHREAD_ITERATIONS; i++)
 19   {
 20     EXPECT_EQ (MaxThreads(NUM_THREADS_LIMIT), NUM_THREADS_LIMIT);
 21   }
 22 }
 23
 24 TEST(Threads, MaxThreads) {
 25   int i;
 26
 27   for (i = 0; i < MAXTHREAD_ITERATIONS; i++)
 28   {
 29     EXPECT_EQ (MaxThreads(NUM_THREADS), NUM_THREADS);
 30   }
 31 }
 32
 33 TEST(Threads, MaxThreadsLimit_Post) {
 34   int i;
 35
 36   for (i = 0; i < MAXTHREAD_ITERATIONS; i++)
 37   {
 38     EXPECT_EQ (MaxThreads(NUM_THREADS_LIMIT), NUM_THREADS_LIMIT);
 39   }
 40 }

 11 #define MAX_TIDS (10000)
 12
 13 ULONG tids [MAX_TIDS];
 14 int MaxThreads(int limit)
 15 {
 16   int result  = 0;
 17   int i       = 0;
 18   int j       = 0;
 19   int dc      = 0; //delete count
 20
 21   if ((limit > 0) && (limit < MAX_TIDS))
 22   {
 23     bzero (tids, sizeof(tids));
 24
 25     for (i = 0; i < limit; i++)
 26     {
 27       result = t_create("MAX_THREAD_TEST", 200, 0x1000, 0x1000, T_LOCAL, &tids[i]);
 28
 29       if (result != 0)
 30         break;
 31     }
 32
 33     //cleardown allocated tasks
 34     for (j = 0; j < i; j++)
 35     {
 36       if (tids[j])
 37       {
 38         if (t_delete (tids[j]) == 0x00)
 39           dc++;
 40       }
 41     }
 42   }
 43
 44   return dc;
 45 }
 9 [----------] 3 tests from Threads
 10 [ RUN      ] Threads.MaxThreadsLimit_Pre
 11 [       OK ] Threads.MaxThreadsLimit_Pre (15521 ms)

 12 [ RUN      ] Threads.MaxThreads
 13       Expected: MaxThreads(5000) Which is: 847
 14 To be equal to: 5000
 15       Expected: MaxThreads(5000) Which is: 336
 16 To be equal to: 5000
 17       Expected: MaxThreads(5000) Which is: 338
 18 To be equal to: 5000
 19       Expected: MaxThreads(5000) Which is: 549
 20 To be equal to: 5000
 21       Expected: MaxThreads(5000) Which is: 560
 22 To be equal to: 5000
 23       Expected: MaxThreads(5000) Which is: 563
 24 To be equal to: 5000
 25       Expected: MaxThreads(5000) Which is: 571
 26 To be equal to: 5000
 27       Expected: MaxThreads(5000) Which is: 571
 28 To be equal to: 5000
 29       Expected: MaxThreads(5000) Which is: 573
 30 To be equal to: 5000
 31       Expected: MaxThreads(5000) Which is: 586
 32 To be equal to: 5000
 33       Expected: MaxThreads(5000) Which is: 588
 34 To be equal to: 5000
 35       Expected: MaxThreads(5000) Which is: 596
 36 To be equal to: 5000
 37       Expected: MaxThreads(5000) Which is: 619
 38 To be equal to: 5000
 39       Expected: MaxThreads(5000) Which is: 630
 40 To be equal to: 5000
 41       Expected: MaxThreads(5000) Which is: 634
 42 To be equal to: 5000
 43       Expected: MaxThreads(5000) Which is: 645
 44 To be equal to: 5000
 45       Expected: MaxThreads(5000) Which is: 647
 46 To be equal to: 5000
 47       Expected: MaxThreads(5000) Which is: 653
 48 To be equal to: 5000
 49       Expected: MaxThreads(5000) Which is: 663
 50 To be equal to: 5000
 51       Expected: MaxThreads(5000) Which is: 666
 52 To be equal to: 5000
 53 [  FAILED  ] Threads.MaxThreads (10090 ms)

 54 [ RUN      ] Threads.MaxThreadsLimit_Post
 55       Expected: MaxThreads(750) Which is: 675
 56 To be equal to: 750
 57       Expected: MaxThreads(750) Which is: 679
 58 To be equal to: 750
 59       Expected: MaxThreads(750) Which is: 672
 60 To be equal to: 750
 61       Expected: MaxThreads(750) Which is: 674
 62 To be equal to: 750
 63       Expected: MaxThreads(750) Which is: 678
 64 To be equal to: 750
 65       Expected: MaxThreads(750) Which is: 674
 66 To be equal to: 750
 67       Expected: MaxThreads(750) Which is: 676
 68 To be equal to: 750
 69       Expected: MaxThreads(750) Which is: 679
 70 To be equal to: 750
 71       Expected: MaxThreads(750) Which is: 672
 72 To be equal to: 750
 73       Expected: MaxThreads(750) Which is: 674
 74 To be equal to: 750
 75       Expected: MaxThreads(750) Which is: 678
 76 To be equal to: 750
 77       Expected: MaxThreads(750) Which is: 674
 78 To be equal to: 750
 79       Expected: MaxThreads(750) Which is: 676
 80 To be equal to: 750
 81       Expected: MaxThreads(750) Which is: 678
 82 To be equal to: 750
 83       Expected: MaxThreads(750) Which is: 677
 84 To be equal to: 750
 85       Expected: MaxThreads(750) Which is: 676
 86 To be equal to: 750
 87       Expected: MaxThreads(750) Which is: 676
 88 To be equal to: 750
 89       Expected: MaxThreads(750) Which is: 676
 90 To be equal to: 750
 91       Expected: MaxThreads(750) Which is: 676
 92 To be equal to: 750
 93       Expected: MaxThreads(750) Which is: 676
 94 To be equal to: 750
 95 [  FAILED  ] Threads.MaxThreadsLimit_Post (11411 ms)
 96 [----------] 3 tests from Threads (37023 ms total)

root@WPL:/var/wpl# ./wpl.bin --dump-config
based on Xenomai/cobalt v3.0 -- #5f6b32f (2015-10-31 20:57:40 +0100)
CONFIG_MMU=1
CONFIG_XENO_BUILD_ARGS=" 'CFLAGS=-march=armv7-a' 'CFLAGS=-mfpu=neon' 'CFLAGS=-fomit-frame-pointer' 'LDFLAGS=-march=armv7-a' '--with-core=cobalt' '--build=i686-pc-linux-gnu' '--host=arm-linux-gnueabihf' '-enable-lores-clock' '-enable-debug=symbols' 'build_alias=i686-pc-linux-gnu' 'host_alias=arm-linux-gnueabihf'"
CONFIG_XENO_BUILD_STRING="i686-pc-linux-gnu"
CONFIG_XENO_COBALT=1
CONFIG_XENO_COMPILER="gcc version 4.7.3 20121106 (prerelease) (crosstool-NG linaro-1.13.1-4.7-2012.11-20121123 - Linaro GCC 2012.11) "
CONFIG_XENO_DEFAULT_PERIOD=1000000
CONFIG_XENO_FORTIFY=1
CONFIG_XENO_HOST_STRING="arm-unknown-linux-gnueabihf"
CONFIG_XENO_PREFIX="/usr/xenomai"
CONFIG_XENO_RAW_CLOCK_ENABLED=1
CONFIG_XENO_REVISION_LEVEL=0
CONFIG_XENO_SANITY=1
CONFIG_XENO_TLSF=1
CONFIG_XENO_TLS_MODEL="initial-exec"
CONFIG_XENO_UAPI_LEVEL=14
CONFIG_XENO_VERSION_MAJOR=3
CONFIG_XENO_VERSION_MINOR=0
CONFIG_XENO_VERSION_NAME="Exact Zero"
CONFIG_XENO_VERSION_STRING="3.0"
CONFIG_XENO_X86_VSYSCALL=1
---
CONFIG_SMP is OFF
CONFIG_XENO_ASYNC_CANCEL is OFF
CONFIG_XENO_COPPERPLATE_CLOCK_RESTRICTED is OFF
CONFIG_XENO_DEBUG is OFF
CONFIG_XENO_DEBUG_FULL is OFF
CONFIG_XENO_LIBS_DLOPEN is OFF
CONFIG_XENO_LORES_CLOCK_DISABLED is OFF
CONFIG_XENO_MERCURY is OFF
CONFIG_XENO_PSHARED is OFF
CONFIG_XENO_REGISTRY is OFF
CONFIG_XENO_REGISTRY_ROOT is OFF
CONFIG_XENO_VALGRIND_API is OFF
CONFIG_XENO_WORKAROUND_CONDVAR_PI is OFF
---
PTHREAD_STACK_DEFAULT=65536
________________________________________
From: Philippe Gerum <rpm@xenomai.org>
Sent: 07 June 2016 16:03:24
To: Marcel Van Mierlo; xenomai@xenomai.org
Subject: Re: [Xenomai] what limits number of pSOS tasks using t_create?

Hi,

On 06/06/2016 10:14 AM, Marcel Van Mierlo wrote:
>
> Hi Philippe, yes that worked.
>
> I made a regression test on the maximum number of psos threads I could create. I found that once I start seeing ERR_NOTCB returned from t_create the maximum number of tasks I can create then drops, even after deleting all previous tasks - as if available memory for tasks becomes less after limit is hit.
>
> For example, after tuning all four parameters below (including CONFIG_XENO_OPT_REGISTRY_NRSLOTS) I am able to reliably create 500 tasks under test conditions - if I push it to 600, say, and see ERR_NOTCB, then after deleting all tasks I can only create a maximum of around 300 tasks.
>
> This is fine for me - I know I will never need more than 150 tasks, but seems something worth noting.
>

I cannot reproduce that one yet. A few questions:

- are you starting the tasks or only creating them before calling
t_delete() on the tid?

- do you observe this behavior when respawning the test program after
each creation+deletion sequence, or within a single execution of the
test program running multiple sequences?

- can you paste the output of ./your_program --dump-config?

Thanks,

--
Philippe.

Re: [Xenomai] what limits number of pSOS tasks using t_create?

[Philippe Gerum]

On 06/08/2016 11:11 AM, Marcel Van Mierlo wrote:

> root@WPL:/var/wpl# ./wpl.bin --dump-config
> based on Xenomai/cobalt v3.0 -- #5f6b32f (2015-10-31 20:57:40 +0100)

Before anything else, could you rebase your tests over the stable-3.0.x
branch from our git tree? The version you have been testing with so far
is obsolete; there is no point in debugging issues which have
potentially been fixed already.

-- 
Philippe.

Re: [Xenomai] what limits number of pSOS tasks using t_create?

[Marcel Van Mierlo]

OK, yep that sorted it. Thanks.

MaxThreadsLimit_Pre and MaxThreadsLimit_Post now always succeed, and MaxThreads(5000) fails consistently on 847 every time (which seems about right for the parameters I am using).

root@WPL:/var/wpl# ./wpl.bin --dump-config
based on Xenomai/cobalt v3.0.2 -- #83d7bfb (2016-06-08 10:51:06 +0200)


I noticed that after building Xenomai 3.0.2, updating kernel with prepare-kernel.sh, building kernel and installing Xenomai 3.0.2, dmesg still reports v3.0 (Exact Zero) - would have expected v3.0.2 to be reported?

root@WPL:~# dmesg | grep -i Xen
[    2.123665] [Xenomai] scheduling class idle registered.
[    2.123675] [Xenomai] scheduling class rt registered.
[    2.123815] I-pipe: head domain Xenomai registered.
[    2.127095] [Xenomai] Cobalt v3.0 (Exact Zero)

________________________________________
From: Philippe Gerum <rpm@xenomai.org>
Sent: 09 June 2016 08:17:35
To: Marcel Van Mierlo; xenomai@xenomai.org
Subject: Re: [Xenomai] what limits number of pSOS tasks using t_create?

On 06/08/2016 11:11 AM, Marcel Van Mierlo wrote:

> root@WPL:/var/wpl# ./wpl.bin --dump-config
> based on Xenomai/cobalt v3.0 -- #5f6b32f (2015-10-31 20:57:40 +0100)

Before anything else, could you rebase your tests over the stable-3.0.x
branch from our git tree? The version you have been testing with so far
is obsolete; there is no point in debugging issues which have
potentially been fixed already.

--
Philippe.

Re: [Xenomai] what limits number of pSOS tasks using t_create?

[Philippe Gerum]

On 06/09/2016 02:45 PM, Marcel Van Mierlo wrote:
> OK, yep that sorted it. Thanks.
> 
> MaxThreadsLimit_Pre and MaxThreadsLimit_Post now always succeed, and MaxThreads(5000) fails consistently on 847 every time (which seems about right for the parameters I am using).
> 
> root@WPL:/var/wpl# ./wpl.bin --dump-config
> based on Xenomai/cobalt v3.0.2 -- #83d7bfb (2016-06-08 10:51:06 +0200)
> 
> 
> I noticed that after building Xenomai 3.0.2, updating kernel with prepare-kernel.sh, building kernel and installing Xenomai 3.0.2, dmesg still reports v3.0 (Exact Zero) - would have expected v3.0.2 to be reported?
> 
> root@WPL:~# dmesg | grep -i Xen
> [    2.123665] [Xenomai] scheduling class idle registered.
> [    2.123675] [Xenomai] scheduling class rt registered.
> [    2.123815] I-pipe: head domain Xenomai registered.
> [    2.127095] [Xenomai] Cobalt v3.0 (Exact Zero)

You need to start from a fresh, un-prepared kernel tree.

-- 
Philippe.

Re: [Xenomai] what limits number of pSOS tasks using t_create?

[Marcel Van Mierlo]

Ah right. OK.

...looking at prepare-kernel.sh...it appears to skip updating the Xenomai version numbers if they are already present in init/Kconfig, though the Xenomai files appear to be copied anyway...

besides the i-pipe patches does Xenomai itself patch kernel files, besides Kconfig? looks pretty minimal if so. I wonder if it makes sense to allow kernel preparation of later Xenomai versions to a kernel source tree that's already been prepared...so there is a simpler upgrade path... in my experience applying i-pipe was the most risky aspect - I had to apply it to an already patched kernel and resolve some issues "by hand", and so is something that I *really* try and avoid doing, though I do want to stay current with Xenomai...

I guess another approach is to keep a clean kernel patched with i-pipe but pre Xenomai....all gets messy....


Marcel.



________________________________________
From: Philippe Gerum <rpm@xenomai.org>
Sent: 09 June 2016 13:52:13
To: Marcel Van Mierlo; xenomai@xenomai.org
Subject: Re: [Xenomai] what limits number of pSOS tasks using t_create?

On 06/09/2016 02:45 PM, Marcel Van Mierlo wrote:
> OK, yep that sorted it. Thanks.
>
> MaxThreadsLimit_Pre and MaxThreadsLimit_Post now always succeed, and MaxThreads(5000) fails consistently on 847 every time (which seems about right for the parameters I am using).
>
> root@WPL:/var/wpl# ./wpl.bin --dump-config
> based on Xenomai/cobalt v3.0.2 -- #83d7bfb (2016-06-08 10:51:06 +0200)
>
>
> I noticed that after building Xenomai 3.0.2, updating kernel with prepare-kernel.sh, building kernel and installing Xenomai 3.0.2, dmesg still reports v3.0 (Exact Zero) - would have expected v3.0.2 to be reported?
>
> root@WPL:~# dmesg | grep -i Xen
> [    2.123665] [Xenomai] scheduling class idle registered.
> [    2.123675] [Xenomai] scheduling class rt registered.
> [    2.123815] I-pipe: head domain Xenomai registered.
> [    2.127095] [Xenomai] Cobalt v3.0 (Exact Zero)

You need to start from a fresh, un-prepared kernel tree.

--
Philippe.