* [Buildroot] issue with timer_create and openblas after update buildroot from 2019_02 to 2021_02
@ 2021-04-07 9:01 Arthur Lambert
2021-04-08 22:59 ` Peter Seiderer
0 siblings, 1 reply; 6+ messages in thread
From: Arthur Lambert @ 2021-04-07 9:01 UTC (permalink / raw)
To: buildroot
I am trying to update buildroot from 2019_02 tag to 2021_02. After the
update I have
some issue with timer_create function from uclibc . I get -1 return error
with EAGAIN errno.
The error only occurred with SIGEV_THREAD type.
I was able to track the code responsible from the error without
understanding the reason
output/build/uclibc-1.0.37/libpthread/nptl/sysdeps/unix/sysv/linux/timer_create.c
line-150
>> {
>> /* Create the helper thread. */
>> pthread_once (&__helper_once, __start_helper_thread);
>> if (__helper_tid == 0)
>> {
>> /* No resources to start the helper thread. */
>> __set_errno (EAGAIN);
>> return -1;
>> }
I am was also able to reproduce the issue with a minimalist piece of code.
The issue seems
to be related to openblas. I dont see the link between BLAS and
thread/timer issue ??
>> #include <stdlib.h>
>> #include <unistd.h>
>> #include <stdio.h>
>> #include <signal.h>
>> #include <time.h>
>>
>>
>> int main(void)
>> {
>>
>> struct sigevent alarm_sigevt;
>> timer_t alarm_timer;
>> int ret = 0;
>>
>> alarm_sigevt.sigev_notify = SIGEV_THREAD;
>> alarm_sigevt.sigev_notify_function = NULL;
>> alarm_sigevt.sigev_notify_attributes = NULL;
>> alarm_sigevt.sigev_value.sival_ptr = NULL;
>> ret = timer_create(CLOCK_REALTIME, &alarm_sigevt, &alarm_timer);
>> printf ("ret : %d\n", ret);
>>
>> exit(EXIT_SUCCESS);
>> }
I compile the code like this :
/home/arthur/work/buildroot/atto_dreem3_dev/output/host/bin/arm-buildroot-linux-uclibcgnueabihf-gcc
test_timer.c -o test_timer
I run the binary on my arm target :
# ./test_timer
ret : 0
#
Everything is working fine here.
I just add openblas library now :
/home/arthur/work/buildroot/atto_dreem3_dev/output/host/bin/arm-buildroot-linux-uclibcgnueabihf-gcc
test_timer.c -o test_timer -lopenblas
I now run the new binary :
# ./test_timer
ret : -1
An issue with openblas which init too many timer ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20210407/9f6f31fe/attachment.html>
^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] issue with timer_create and openblas after update buildroot from 2019_02 to 2021_02
2021-04-07 9:01 [Buildroot] issue with timer_create and openblas after update buildroot from 2019_02 to 2021_02 Arthur Lambert
@ 2021-04-08 22:59 ` Peter Seiderer
2021-04-09 7:53 ` Arthur Lambert
0 siblings, 1 reply; 6+ messages in thread
From: Peter Seiderer @ 2021-04-08 22:59 UTC (permalink / raw)
To: buildroot
Hello Arthur,
On Wed, 7 Apr 2021 11:01:59 +0200, Arthur Lambert <lambertarthur22@gmail.com> wrote:
> I am trying to update buildroot from 2019_02 tag to 2021_02. After the
> update I have
> some issue with timer_create function from uclibc . I get -1 return error
> with EAGAIN errno.
> The error only occurred with SIGEV_THREAD type.
>
> I was able to track the code responsible from the error without
> understanding the reason
>
> output/build/uclibc-1.0.37/libpthread/nptl/sysdeps/unix/sysv/linux/timer_create.c
> line-150
>
> >> {
> >> /* Create the helper thread. */
> >> pthread_once (&__helper_once, __start_helper_thread);
> >> if (__helper_tid == 0)
> >> {
> >> /* No resources to start the helper thread. */
> >> __set_errno (EAGAIN);
> >> return -1;
> >> }
>
> I am was also able to reproduce the issue with a minimalist piece of code.
> The issue seems
> to be related to openblas. I dont see the link between BLAS and
> thread/timer issue ??
>
> >> #include <stdlib.h>
> >> #include <unistd.h>
> >> #include <stdio.h>
> >> #include <signal.h>
> >> #include <time.h>
> >>
> >>
> >> int main(void)
> >> {
> >>
> >> struct sigevent alarm_sigevt;
> >> timer_t alarm_timer;
> >> int ret = 0;
> >>
> >> alarm_sigevt.sigev_notify = SIGEV_THREAD;
> >> alarm_sigevt.sigev_notify_function = NULL;
> >> alarm_sigevt.sigev_notify_attributes = NULL;
> >> alarm_sigevt.sigev_value.sival_ptr = NULL;
> >> ret = timer_create(CLOCK_REALTIME, &alarm_sigevt, &alarm_timer);
> >> printf ("ret : %d\n", ret);
> >>
> >> exit(EXIT_SUCCESS);
> >> }
>
> I compile the code like this :
>
> /home/arthur/work/buildroot/atto_dreem3_dev/output/host/bin/arm-buildroot-linux-uclibcgnueabihf-gcc
> test_timer.c -o test_timer
>
> I run the binary on my arm target :
>
> # ./test_timer
> ret : 0
> #
>
> Everything is working fine here.
The strace log looks like the following:
[...]
235 stat("/lib/ld-uClibc.so.0", {st_mode=S_IFREG|0755, st_size=28408, ...}) = 0
235 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6f6b000
235 set_tls(0xb6f6b490) = 0
235 mprotect(0x20000, 4096, PROT_READ) = 0
235 mprotect(0xb6f40000, 4096, PROT_READ) = 0
235 mprotect(0xb6f6d000, 4096, PROT_READ) = 0
235 set_tid_address(0xb6f6b068) = 235
235 set_robust_list(0xb6f6b06c, 12) = 0
235 rt_sigaction(SIGRTMIN, {sa_handler=0xb6f239e8, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0xb6ed4908}, NULL, 8) = 0
235 rt_sigaction(SIGRT_1, {sa_handler=0xb6f23aac, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0xb6ed4908}, NULL, 8) = 0
235 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
235 ugetrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
235 ioctl(0, TCGETS, {B115200 opost isig icanon echo ...}) = 0
235 ioctl(1, TCGETS, {B115200 opost isig icanon echo ...}) = 0
235 rt_sigprocmask(SIG_SETMASK, ~[RT_1], [], 8) = 0
235 mmap2(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0xb6f67000
235 brk(NULL) = 0x1934000
235 brk(0x1935000) = 0x1935000
235 mprotect(0xb6f67000, 4096, PROT_NONE) = 0
235 clone(child_stack=0xb6f6a038, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[236], tls=0xb6f6a960, child_tidptr=0xb6f6a538) = 236
235 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
235 futex(0xb6f57824, FUTEX_WAKE_PRIVATE, 2147483647) = 0
235 timer_create(CLOCK_REALTIME, {sigev_value={sival_int=26427544, sival_ptr=0x1934098}, sigev_signo=SIGRTMIN, sigev_notify=SIGEV_THREAD_ID, sigev_notify_thread_id=236}, [0]) = 0
235 write(1, "ret : 0\n", 8) = 8
235 exit_group(0) = ?
236 +++ exited with 0 +++
235 +++ exited with 0 +++
> I just add openblas library now :
>
> /home/arthur/work/buildroot/atto_dreem3_dev/output/host/bin/arm-buildroot-linux-uclibcgnueabihf-gcc
> test_timer.c -o test_timer -lopenblas
>
> I now run the new binary :
>
> # ./test_timer
> ret : -1
>
> An issue with openblas which init too many timer ?
The strace log looks like the following:
[...]
240 stat("/lib/ld-uClibc.so.0", {st_mode=S_IFREG|0755, st_size=28408, ...}) = 0
240 mmap2(NULL, 64304, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6d0d000
240 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6f93000
240 set_tls(0xb6d0d490) = 0
240 mprotect(0x20000, 4096, PROT_READ) = 0
240 mprotect(0xb6f7b000, 8192, PROT_READ) = 0
240 mprotect(0xb6d9b000, 4096, PROT_READ) = 0
240 mprotect(0xb6f95000, 4096, PROT_READ) = 0
240 set_tid_address(0xb6d0d068) = 240
240 set_robust_list(0xb6d0d06c, 12) = 0
240 rt_sigaction(SIGRTMIN, {sa_handler=0xb6d7e9e8, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0xb6d2f908}, NULL, 8) = 0
240 rt_sigaction(SIGRT_1, {sa_handler=0xb6d7eaac, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0xb6d2f908}, NULL, 8) = 0
240 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
240 ugetrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
240 ioctl(0, TCGETS, {B115200 opost isig icanon echo ...}) = 0
240 ioctl(1, TCGETS, {B115200 opost isig icanon echo ...}) = 0
240 open("/sys/devices/system/cpu", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
240 fstat64(3, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
240 fcntl(3, F_SETFD, FD_CLOEXEC) = 0
240 brk(NULL) = 0x1485000
240 brk(0x1486000) = 0x1486000
240 brk(0x1487000) = 0x1487000
240 getdents64(3, 0xbed7fa28 /* 13 entries */, 4096) = 392
240 getdents64(3, 0xbed7fa28 /* 0 entries */, 4096) = 0
240 close(3) = 0
240 rt_sigprocmask(SIG_SETMASK, ~[RT_1], [], 8) = 0
240 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
240 futex(0xb6db2824, FUTEX_WAKE_PRIVATE, 2147483647) = 0
240 write(1, "ret : -1\n", 9) = 9
240 exit_group(0) = ?
240 +++ exited with 0 +++
No clone() and no timer_create() syscall...., but the syscall sequence from
open("/sys/devices/system/cpu",...) to getdents64()/close() looks like the
nprocessors_conf() implementation from uclibc-1.0.38/libc/unistd/sysconf.c...,
strange....
Regards,
Peter
^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] issue with timer_create and openblas after update buildroot from 2019_02 to 2021_02
2021-04-08 22:59 ` Peter Seiderer
@ 2021-04-09 7:53 ` Arthur Lambert
2021-04-09 19:18 ` Peter Seiderer
0 siblings, 1 reply; 6+ messages in thread
From: Arthur Lambert @ 2021-04-09 7:53 UTC (permalink / raw)
To: buildroot
I was able to fix the issue by checking the Openblas history commits. But I
don't really understand the issue...
The issue is coming from here:
https://github.com/buildroot/buildroot/commit/6f29cdeee40e556d26b791fabaff84a09e1a2d5d#diff-0995e357753caee10bbc553c2e7aa033d1df05c0c9799916729d4881d7735d26
By default, the multithreading setting is enabled on Openblas. This setting
is really confusing me. For me, I have to enable the multithreading setting
to use Openblas
in a multithreading context (this is my case !). But it is the opposite!
There is this information in the Openblas USAGE file :
>> #### How can I use OpenBLAS in multi-threaded applications?
>>
>> If your application is already multi-threaded, it will conflict with
OpenBLAS
>> multi-threading. Thus, you must set OpenBLAS to use single thread in any
of the
>> following ways:
>>
>> * `export OPENBLAS_NUM_THREADS=1` in the environment variables.
>> * Call `openblas_set_num_threads(1)` in the application on runtime.
>> * Build OpenBLAS single thread version, e.g. `make USE_THREAD=0`
So I have to add this in my defconfig to build Openblas with
singlethread support to fix the problem:
iff --git a/configs/xxxx_defconfig b/configs/dxxxxxx_defconfig
index 4714080d7e..a760627e2c 100644
--- a/configs/xxxxx_defconfig
+++ b/configs/xxxx_defconfig
@@ -45,6 +45,7 @@ BR2_PACKAGE_PAHO_MQTT_C=y
BR2_PACKAGE_ELFUTILS=y
BR2_PACKAGE_LIBUNWIND=y
BR2_PACKAGE_OPENBLAS=y
+# BR2_PACKAGE_OPENBLAS_USE_THREAD is not set
BR2_PACKAGE_PROTOBUF=y
BR2_PACKAGE_PROTOBUF_C=y
BR2_PACKAGE_HAVEGED=y
How this "conflict" can cause an error on the timer_create feature from
uclibc?
Is my change correct?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20210409/ed9ba3df/attachment.html>
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [Buildroot] issue with timer_create and openblas after update buildroot from 2019_02 to 2021_02
2021-04-09 7:53 ` Arthur Lambert
@ 2021-04-09 19:18 ` Peter Seiderer
2021-04-13 19:52 ` Peter Seiderer
0 siblings, 1 reply; 6+ messages in thread
From: Peter Seiderer @ 2021-04-09 19:18 UTC (permalink / raw)
To: buildroot
Hello Arthur,
On Fri, 9 Apr 2021 09:53:23 +0200, Arthur Lambert <lambertarthur22@gmail.com> wrote:
> I was able to fix the issue by checking the Openblas history commits. But I
> don't really understand the issue...
>
> The issue is coming from here:
> https://github.com/buildroot/buildroot/commit/6f29cdeee40e556d26b791fabaff84a09e1a2d5d#diff-0995e357753caee10bbc553c2e7aa033d1df05c0c9799916729d4881d7735d26
>
> By default, the multithreading setting is enabled on Openblas. This setting
> is really confusing me. For me, I have to enable the multithreading setting
> to use Openblas
> in a multithreading context (this is my case !). But it is the opposite!
> There is this information in the Openblas USAGE file :
>
> >> #### How can I use OpenBLAS in multi-threaded applications?
> >>
> >> If your application is already multi-threaded, it will conflict with
> OpenBLAS
> >> multi-threading. Thus, you must set OpenBLAS to use single thread in any
> of the
> >> following ways:
> >>
> >> * `export OPENBLAS_NUM_THREADS=1` in the environment variables.
> >> * Call `openblas_set_num_threads(1)` in the application on runtime.
> >> * Build OpenBLAS single thread version, e.g. `make USE_THREAD=0`
>
> So I have to add this in my defconfig to build Openblas with
> singlethread support to fix the problem:
>
> iff --git a/configs/xxxx_defconfig b/configs/dxxxxxx_defconfig
> index 4714080d7e..a760627e2c 100644
> --- a/configs/xxxxx_defconfig
> +++ b/configs/xxxx_defconfig
> @@ -45,6 +45,7 @@ BR2_PACKAGE_PAHO_MQTT_C=y
> BR2_PACKAGE_ELFUTILS=y
> BR2_PACKAGE_LIBUNWIND=y
> BR2_PACKAGE_OPENBLAS=y
> +# BR2_PACKAGE_OPENBLAS_USE_THREAD is not set
> BR2_PACKAGE_PROTOBUF=y
> BR2_PACKAGE_PROTOBUF_C=y
> BR2_PACKAGE_HAVEGED=y
>
> How this "conflict" can cause an error on the timer_create feature from
> uclibc?
This commit gives you only the chance to disable openblas threading support
(and circumvent the problem for you), but is not the decisive change
between buildroot 2019.02 and 2021.02 regarding uclibc/openblas, because
already the old/2019.02 version should have threading enabled (only without
an buildroot option to disable it)...
The version change from 2019.02 to 2021.02 are:
- openblas from 0.2.20 to 0.3.9
- uclibc from 1.0.31 to 1.0.38
- gcc/binutils/etc (did not yet check yet)?
But I did some debugging, with openblas linking the timer_create() call
fails because of a failing pthread_create() because of a stack-size
check in allocate_stack()...
Without openblas (o.k - one call via timer_create()/pthread_create()):
Breakpoint 1, allocate_stack (attr=0x7fed1079c8, pdp=0x7fed107920, stack=0x7fed107928)
at libpthread/nptl/allocatestack.c:472
472 pd = get_cached_stack (&size, &mem);
(gdb) p *attr
$1 = {schedparam = {__sched_priority = 0}, schedpolicy = 0, flags = 0, guardsize = 4096,
stackaddr = 0x0, stacksize = 16384, cpuset = 0x0, cpusetsize = 0}
(gdb) p size
$2 = 16384
(gdb) p __static_tls_size
$3 = 1712
With openblas (failure - three time o.k call from openblas, one failing call from timer_create()):
Breakpoint 1, allocate_stack (attr=0x7fa286de48 <default_attr>, pdp=0x7febf5fd80, stack=0x7febf5fd88)
at libpthread/nptl/allocatestack.c:472
472 pd = get_cached_stack (&size, &mem);
(gdb) p *attr
$1 = {schedparam = {__sched_priority = 0}, schedpolicy = 0, flags = 0, guardsize = 1, stackaddr = 0x0,
stacksize = 0, cpuset = 0x0, cpusetsize = 0}
(gdb) p size
$2 = 2097152
(gdb) p __static_tls_size
$3 = 63152
[...]
Thread 1 "test_timer_crea" hit Breakpoint 2, allocate_stack (attr=0x7febf61068, pdp=0x7febf60fc0,
stack=0x7febf60fc8) at libpthread/nptl/allocatestack.c:468
468 return EINVAL;
(gdb) p *attr
$4 = {schedparam = {__sched_priority = 0}, schedpolicy = 0, flags = 0, guardsize = 4096,
stackaddr = 0x0, stacksize = 16384, cpuset = 0x0, cpusetsize = 0}
(gdb) p size
$5 = 16384
(gdb) p __static_tls_size
$6 = 63152
- with openblas linking the thread-local-storage (__static_tls_size) is
increased from 1712 to 63152
- the openblas threads are created with default pthread_attr_t/nullptr,
the stack size size is calculated to 2097152 the check succeeds
- the timer_create pthread_create is called with pthread_attr_t created
via
66 (void) pthread_attr_init (&attr);
167 (void) pthread_attr_setstacksize (&attr, PTHREAD_STACK_MIN);
so stacksize is given as 16384, the check against the increased thread-local-
storage (63152 + guadsize + MINIMAL_REST_STACK + padding) fails...
Maybe a uclibc bug/defect...., or an expert question: is the pthread_create/
pthread_attr_t stacksize inclusive or exclusive the reserved size for
thread-local-storage usage (my judgment would be exclusive as the thread-local
storage size is dependents on external parameters/library linking etc.)?
> Is my change correct?
If it works for you yes ;-), but does not fix the underlying/generally problem...
Regards,
Peter
^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] issue with timer_create and openblas after update buildroot from 2019_02 to 2021_02
2021-04-09 19:18 ` Peter Seiderer
@ 2021-04-13 19:52 ` Peter Seiderer
2021-04-16 21:26 ` Peter Seiderer
0 siblings, 1 reply; 6+ messages in thread
From: Peter Seiderer @ 2021-04-13 19:52 UTC (permalink / raw)
To: buildroot
Hello Arthur, Waldemar,
On Fri, 9 Apr 2021 21:18:10 +0200, Peter Seiderer <ps.report@gmx.net> wrote:
> Hello Arthur,
>
> On Fri, 9 Apr 2021 09:53:23 +0200, Arthur Lambert <lambertarthur22@gmail.com> wrote:
>
> > I was able to fix the issue by checking the Openblas history commits. But I
> > don't really understand the issue...
> >
> > The issue is coming from here:
> > https://github.com/buildroot/buildroot/commit/6f29cdeee40e556d26b791fabaff84a09e1a2d5d#diff-0995e357753caee10bbc553c2e7aa033d1df05c0c9799916729d4881d7735d26
> >
> > By default, the multithreading setting is enabled on Openblas. This setting
> > is really confusing me. For me, I have to enable the multithreading setting
> > to use Openblas
> > in a multithreading context (this is my case !). But it is the opposite!
> > There is this information in the Openblas USAGE file :
> >
> > >> #### How can I use OpenBLAS in multi-threaded applications?
> > >>
> > >> If your application is already multi-threaded, it will conflict with
> > OpenBLAS
> > >> multi-threading. Thus, you must set OpenBLAS to use single thread in any
> > of the
> > >> following ways:
> > >>
> > >> * `export OPENBLAS_NUM_THREADS=1` in the environment variables.
> > >> * Call `openblas_set_num_threads(1)` in the application on runtime.
> > >> * Build OpenBLAS single thread version, e.g. `make USE_THREAD=0`
> >
> > So I have to add this in my defconfig to build Openblas with
> > singlethread support to fix the problem:
> >
> > iff --git a/configs/xxxx_defconfig b/configs/dxxxxxx_defconfig
> > index 4714080d7e..a760627e2c 100644
> > --- a/configs/xxxxx_defconfig
> > +++ b/configs/xxxx_defconfig
> > @@ -45,6 +45,7 @@ BR2_PACKAGE_PAHO_MQTT_C=y
> > BR2_PACKAGE_ELFUTILS=y
> > BR2_PACKAGE_LIBUNWIND=y
> > BR2_PACKAGE_OPENBLAS=y
> > +# BR2_PACKAGE_OPENBLAS_USE_THREAD is not set
> > BR2_PACKAGE_PROTOBUF=y
> > BR2_PACKAGE_PROTOBUF_C=y
> > BR2_PACKAGE_HAVEGED=y
> >
> > How this "conflict" can cause an error on the timer_create feature from
> > uclibc?
>
> This commit gives you only the chance to disable openblas threading support
> (and circumvent the problem for you), but is not the decisive change
> between buildroot 2019.02 and 2021.02 regarding uclibc/openblas, because
> already the old/2019.02 version should have threading enabled (only without
> an buildroot option to disable it)...
>
> The version change from 2019.02 to 2021.02 are:
>
> - openblas from 0.2.20 to 0.3.9
> - uclibc from 1.0.31 to 1.0.38
> - gcc/binutils/etc (did not yet check yet)?
>
> But I did some debugging, with openblas linking the timer_create() call
> fails because of a failing pthread_create() because of a stack-size
> check in allocate_stack()...
>
>
> Without openblas (o.k - one call via timer_create()/pthread_create()):
>
> Breakpoint 1, allocate_stack (attr=0x7fed1079c8, pdp=0x7fed107920, stack=0x7fed107928)
> at libpthread/nptl/allocatestack.c:472
> 472 pd = get_cached_stack (&size, &mem);
> (gdb) p *attr
> $1 = {schedparam = {__sched_priority = 0}, schedpolicy = 0, flags = 0, guardsize = 4096,
> stackaddr = 0x0, stacksize = 16384, cpuset = 0x0, cpusetsize = 0}
> (gdb) p size
> $2 = 16384
> (gdb) p __static_tls_size
> $3 = 1712
>
>
> With openblas (failure - three time o.k call from openblas, one failing call from timer_create()):
>
> Breakpoint 1, allocate_stack (attr=0x7fa286de48 <default_attr>, pdp=0x7febf5fd80, stack=0x7febf5fd88)
> at libpthread/nptl/allocatestack.c:472
> 472 pd = get_cached_stack (&size, &mem);
> (gdb) p *attr
> $1 = {schedparam = {__sched_priority = 0}, schedpolicy = 0, flags = 0, guardsize = 1, stackaddr = 0x0,
> stacksize = 0, cpuset = 0x0, cpusetsize = 0}
> (gdb) p size
> $2 = 2097152
> (gdb) p __static_tls_size
> $3 = 63152
>
> [...]
>
> Thread 1 "test_timer_crea" hit Breakpoint 2, allocate_stack (attr=0x7febf61068, pdp=0x7febf60fc0,
> stack=0x7febf60fc8) at libpthread/nptl/allocatestack.c:468
> 468 return EINVAL;
> (gdb) p *attr
> $4 = {schedparam = {__sched_priority = 0}, schedpolicy = 0, flags = 0, guardsize = 4096,
> stackaddr = 0x0, stacksize = 16384, cpuset = 0x0, cpusetsize = 0}
> (gdb) p size
> $5 = 16384
> (gdb) p __static_tls_size
> $6 = 63152
>
>
> - with openblas linking the thread-local-storage (__static_tls_size) is
> increased from 1712 to 63152
>
> - the openblas threads are created with default pthread_attr_t/nullptr,
> the stack size size is calculated to 2097152 the check succeeds
>
> - the timer_create pthread_create is called with pthread_attr_t created
> via
>
> 66 (void) pthread_attr_init (&attr);
> 167 (void) pthread_attr_setstacksize (&attr, PTHREAD_STACK_MIN);
>
> so stacksize is given as 16384, the check against the increased thread-local-
> storage (63152 + guadsize + MINIMAL_REST_STACK + padding) fails...
>
> Maybe a uclibc bug/defect...., or an expert question: is the pthread_create/
> pthread_attr_t stacksize inclusive or exclusive the reserved size for
> thread-local-storage usage (my judgment would be exclusive as the thread-local
> storage size is dependents on external parameters/library linking etc.)?
I think uclibc-ng needs something as implemented for glibc (see [1]), using
__static_tls_size + PTHREAD_STACK_MIN
as stack size for timer_create thread (take the thread local storage stack
part into account)..., draft patch like the following works for the openblas/
timer_create example:
diff --git a/libpthread/nptl/init.c b/libpthread/nptl/init.c
index 4959d5ed8..fefdda1a5 100644
--- a/libpthread/nptl/init.c
+++ b/libpthread/nptl/init.c
@@ -337,3 +337,9 @@ __pthread_initialize_minimal_internal (void)
}
strong_alias (__pthread_initialize_minimal_internal,
__pthread_initialize_minimal)
+
+size_t
+__pthread_get_minstack (const pthread_attr_t *attr)
+{
+ return __static_tls_size + PTHREAD_STACK_MIN;
+}
diff --git a/libpthread/nptl/pthreadP.h b/libpthread/nptl/pthreadP.h
index 13205512a..c686a4ca5 100644
--- a/libpthread/nptl/pthreadP.h
+++ b/libpthread/nptl/pthreadP.h
@@ -377,6 +377,7 @@ weak_function;
extern void __pthread_init_static_tls (struct link_map *) attribute_hidden;
+extern size_t __pthread_get_minstack (const pthread_attr_t *attr);
/* Namespace save aliases. */
extern int __pthread_getschedparam (pthread_t thread_id, int *policy,
diff --git a/libpthread/nptl/sysdeps/unix/sysv/linux/timer_routines.c b/libpthread/nptl/sysdeps/unix/sysv/linux/timer_routines.c
index 514913317..60f2a724c 100644
--- a/libpthread/nptl/sysdeps/unix/sysv/linux/timer_routines.c
+++ b/libpthread/nptl/sysdeps/unix/sysv/linux/timer_routines.c
@@ -164,7 +164,7 @@ __start_helper_thread (void)
and should go away automatically when canceled. */
pthread_attr_t attr;
(void) pthread_attr_init (&attr);
- (void) pthread_attr_setstacksize (&attr, PTHREAD_STACK_MIN);
+ (void) pthread_attr_setstacksize (&attr, __pthread_get_minstack (&attr));
/* Block all signals in the helper thread but SIGSETXID. To do this
thoroughly we temporarily have to block all signals here. The
Regards,
Peter
[1] https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=2c1094bd700e63a8d7f547b3f5495bedb55c0a08;hp=3b8dfc621bfd320c924a3cd597086d3473da1cf4
>
> > Is my change correct?
>
> If it works for you yes ;-), but does not fix the underlying/generally problem...
>
> Regards,
> Peter
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [Buildroot] issue with timer_create and openblas after update buildroot from 2019_02 to 2021_02
2021-04-13 19:52 ` Peter Seiderer
@ 2021-04-16 21:26 ` Peter Seiderer
0 siblings, 0 replies; 6+ messages in thread
From: Peter Seiderer @ 2021-04-16 21:26 UTC (permalink / raw)
To: buildroot
On Tue, 13 Apr 2021 21:52:39 +0200, Peter Seiderer <ps.report@gmx.net> wrote:
> Hello Arthur, Waldemar,
>
> On Fri, 9 Apr 2021 21:18:10 +0200, Peter Seiderer <ps.report@gmx.net> wrote:
>
> > Hello Arthur,
> >
> > On Fri, 9 Apr 2021 09:53:23 +0200, Arthur Lambert <lambertarthur22@gmail.com> wrote:
> >
> > > I was able to fix the issue by checking the Openblas history commits. But I
> > > don't really understand the issue...
> > >
> > > The issue is coming from here:
> > > https://github.com/buildroot/buildroot/commit/6f29cdeee40e556d26b791fabaff84a09e1a2d5d#diff-0995e357753caee10bbc553c2e7aa033d1df05c0c9799916729d4881d7735d26
> > >
> > > By default, the multithreading setting is enabled on Openblas. This setting
> > > is really confusing me. For me, I have to enable the multithreading setting
> > > to use Openblas
> > > in a multithreading context (this is my case !). But it is the opposite!
> > > There is this information in the Openblas USAGE file :
> > >
> > > >> #### How can I use OpenBLAS in multi-threaded applications?
> > > >>
> > > >> If your application is already multi-threaded, it will conflict with
> > > OpenBLAS
> > > >> multi-threading. Thus, you must set OpenBLAS to use single thread in any
> > > of the
> > > >> following ways:
> > > >>
> > > >> * `export OPENBLAS_NUM_THREADS=1` in the environment variables.
> > > >> * Call `openblas_set_num_threads(1)` in the application on runtime.
> > > >> * Build OpenBLAS single thread version, e.g. `make USE_THREAD=0`
> > >
> > > So I have to add this in my defconfig to build Openblas with
> > > singlethread support to fix the problem:
> > >
> > > iff --git a/configs/xxxx_defconfig b/configs/dxxxxxx_defconfig
> > > index 4714080d7e..a760627e2c 100644
> > > --- a/configs/xxxxx_defconfig
> > > +++ b/configs/xxxx_defconfig
> > > @@ -45,6 +45,7 @@ BR2_PACKAGE_PAHO_MQTT_C=y
> > > BR2_PACKAGE_ELFUTILS=y
> > > BR2_PACKAGE_LIBUNWIND=y
> > > BR2_PACKAGE_OPENBLAS=y
> > > +# BR2_PACKAGE_OPENBLAS_USE_THREAD is not set
> > > BR2_PACKAGE_PROTOBUF=y
> > > BR2_PACKAGE_PROTOBUF_C=y
> > > BR2_PACKAGE_HAVEGED=y
> > >
> > > How this "conflict" can cause an error on the timer_create feature from
> > > uclibc?
> >
> > This commit gives you only the chance to disable openblas threading support
> > (and circumvent the problem for you), but is not the decisive change
> > between buildroot 2019.02 and 2021.02 regarding uclibc/openblas, because
> > already the old/2019.02 version should have threading enabled (only without
> > an buildroot option to disable it)...
> >
> > The version change from 2019.02 to 2021.02 are:
> >
> > - openblas from 0.2.20 to 0.3.9
> > - uclibc from 1.0.31 to 1.0.38
> > - gcc/binutils/etc (did not yet check yet)?
> >
> > But I did some debugging, with openblas linking the timer_create() call
> > fails because of a failing pthread_create() because of a stack-size
> > check in allocate_stack()...
> >
> >
> > Without openblas (o.k - one call via timer_create()/pthread_create()):
> >
> > Breakpoint 1, allocate_stack (attr=0x7fed1079c8, pdp=0x7fed107920, stack=0x7fed107928)
> > at libpthread/nptl/allocatestack.c:472
> > 472 pd = get_cached_stack (&size, &mem);
> > (gdb) p *attr
> > $1 = {schedparam = {__sched_priority = 0}, schedpolicy = 0, flags = 0, guardsize = 4096,
> > stackaddr = 0x0, stacksize = 16384, cpuset = 0x0, cpusetsize = 0}
> > (gdb) p size
> > $2 = 16384
> > (gdb) p __static_tls_size
> > $3 = 1712
> >
> >
> > With openblas (failure - three time o.k call from openblas, one failing call from timer_create()):
> >
> > Breakpoint 1, allocate_stack (attr=0x7fa286de48 <default_attr>, pdp=0x7febf5fd80, stack=0x7febf5fd88)
> > at libpthread/nptl/allocatestack.c:472
> > 472 pd = get_cached_stack (&size, &mem);
> > (gdb) p *attr
> > $1 = {schedparam = {__sched_priority = 0}, schedpolicy = 0, flags = 0, guardsize = 1, stackaddr = 0x0,
> > stacksize = 0, cpuset = 0x0, cpusetsize = 0}
> > (gdb) p size
> > $2 = 2097152
> > (gdb) p __static_tls_size
> > $3 = 63152
> >
> > [...]
> >
> > Thread 1 "test_timer_crea" hit Breakpoint 2, allocate_stack (attr=0x7febf61068, pdp=0x7febf60fc0,
> > stack=0x7febf60fc8) at libpthread/nptl/allocatestack.c:468
> > 468 return EINVAL;
> > (gdb) p *attr
> > $4 = {schedparam = {__sched_priority = 0}, schedpolicy = 0, flags = 0, guardsize = 4096,
> > stackaddr = 0x0, stacksize = 16384, cpuset = 0x0, cpusetsize = 0}
> > (gdb) p size
> > $5 = 16384
> > (gdb) p __static_tls_size
> > $6 = 63152
> >
> >
> > - with openblas linking the thread-local-storage (__static_tls_size) is
> > increased from 1712 to 63152
> >
> > - the openblas threads are created with default pthread_attr_t/nullptr,
> > the stack size size is calculated to 2097152 the check succeeds
> >
> > - the timer_create pthread_create is called with pthread_attr_t created
> > via
> >
> > 66 (void) pthread_attr_init (&attr);
> > 167 (void) pthread_attr_setstacksize (&attr, PTHREAD_STACK_MIN);
> >
> > so stacksize is given as 16384, the check against the increased thread-local-
> > storage (63152 + guadsize + MINIMAL_REST_STACK + padding) fails...
> >
> > Maybe a uclibc bug/defect...., or an expert question: is the pthread_create/
> > pthread_attr_t stacksize inclusive or exclusive the reserved size for
> > thread-local-storage usage (my judgment would be exclusive as the thread-local
> > storage size is dependents on external parameters/library linking etc.)?
>
> I think uclibc-ng needs something as implemented for glibc (see [1]), using
>
> __static_tls_size + PTHREAD_STACK_MIN
>
> as stack size for timer_create thread (take the thread local storage stack
> part into account)..., draft patch like the following works for the openblas/
> timer_create example:
>
> diff --git a/libpthread/nptl/init.c b/libpthread/nptl/init.c
> index 4959d5ed8..fefdda1a5 100644
> --- a/libpthread/nptl/init.c
> +++ b/libpthread/nptl/init.c
> @@ -337,3 +337,9 @@ __pthread_initialize_minimal_internal (void)
> }
> strong_alias (__pthread_initialize_minimal_internal,
> __pthread_initialize_minimal)
> +
> +size_t
> +__pthread_get_minstack (const pthread_attr_t *attr)
> +{
> + return __static_tls_size + PTHREAD_STACK_MIN;
> +}
> diff --git a/libpthread/nptl/pthreadP.h b/libpthread/nptl/pthreadP.h
> index 13205512a..c686a4ca5 100644
> --- a/libpthread/nptl/pthreadP.h
> +++ b/libpthread/nptl/pthreadP.h
> @@ -377,6 +377,7 @@ weak_function;
>
> extern void __pthread_init_static_tls (struct link_map *) attribute_hidden;
>
> +extern size_t __pthread_get_minstack (const pthread_attr_t *attr);
>
> /* Namespace save aliases. */
> extern int __pthread_getschedparam (pthread_t thread_id, int *policy,
> diff --git a/libpthread/nptl/sysdeps/unix/sysv/linux/timer_routines.c b/libpthread/nptl/sysdeps/unix/sysv/linux/timer_routines.c
> index 514913317..60f2a724c 100644
> --- a/libpthread/nptl/sysdeps/unix/sysv/linux/timer_routines.c
> +++ b/libpthread/nptl/sysdeps/unix/sysv/linux/timer_routines.c
> @@ -164,7 +164,7 @@ __start_helper_thread (void)
> and should go away automatically when canceled. */
> pthread_attr_t attr;
> (void) pthread_attr_init (&attr);
> - (void) pthread_attr_setstacksize (&attr, PTHREAD_STACK_MIN);
> + (void) pthread_attr_setstacksize (&attr, __pthread_get_minstack (&attr));
>
> /* Block all signals in the helper thread but SIGSETXID. To do this
> thoroughly we temporarily have to block all signals here. The
>
Patch suggested upstream, see:
https://mailman.uclibc-ng.org/pipermail/devel/2021-April/002064.html
Regards,
Peter
>
> Regards,
> Peter
>
> [1] https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=2c1094bd700e63a8d7f547b3f5495bedb55c0a08;hp=3b8dfc621bfd320c924a3cd597086d3473da1cf4
>
> >
> > > Is my change correct?
> >
> > If it works for you yes ;-), but does not fix the underlying/generally problem...
> >
> > Regards,
> > Peter
> > _______________________________________________
> > buildroot mailing list
> > buildroot at busybox.net
> > http://lists.busybox.net/mailman/listinfo/buildroot
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-04-16 21:26 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-07 9:01 [Buildroot] issue with timer_create and openblas after update buildroot from 2019_02 to 2021_02 Arthur Lambert
2021-04-08 22:59 ` Peter Seiderer
2021-04-09 7:53 ` Arthur Lambert
2021-04-09 19:18 ` Peter Seiderer
2021-04-13 19:52 ` Peter Seiderer
2021-04-16 21:26 ` Peter Seiderer
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.