pseudo deadlock in rust via malloc backtrace

pseudo deadlock in rust via malloc backtrace

[Richard Purdie]

Just to note that where we've been seeing pseudo hangs in rust, it
seems jemalloc calls obtain_malloc_conf() which calls readlink() which
pseudo intercepts and then calls malloc() which loops back to jemalloc
which deadlocks. What fun.

#0  0x00007f3806c84bcf in __lll_lock_wait () from /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/sysroots-uninative/x86_64-linux/lib/libc.so.6
#1  0x00007f3806c8aec2 in pthread_mutex_lock () from /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/sysroots-uninative/x86_64-linux/lib/libc.so.6
#2  0x00005629da227a0e in malloc_mutex_lock_final (mutex=0x5629da45a288 <init_lock>) at include/jemalloc/internal/mutex.h:155
#3  _rjem_je_malloc_mutex_lock_slow (mutex=0x5629da45a288 <init_lock>) at src/mutex.c:85
#4  0x00005629da1ffddb in malloc_mutex_lock (tsdn=0x0, mutex=<optimized out>) at include/jemalloc/internal/mutex.h:221
#5  malloc_init_hard () at src/jemalloc.c:1739
#6  0x00005629da1fa2ee in malloc_init () at src/jemalloc.c:223
#7  imalloc_init_check (sopts=<optimized out>, dopts=<optimized out>) at src/jemalloc.c:2229
#8  imalloc (sopts=<optimized out>, dopts=<optimized out>) at src/jemalloc.c:2260
#9  _rjem_je_malloc_default (size=101) at src/jemalloc.c:2289
#10 0x00007f3806c998fe in strdup () from /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/sysroots-uninative/x86_64-linux/lib/libc.so.6
#11 0x00007f380b19da1b in pseudo_set_value () from /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so
#12 0x00007f380b19dbb9 in pseudo_init_util () from /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so
#13 0x00007f380b16bd01 in ?? () from /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so
#14 0x00007f380b19125d in readlink () from /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so
#15 0x00005629da1fe07b in obtain_malloc_conf (which_source=2, buf=0x7ffca80e7cc0 "\320~\016\250\374\177") at src/jemalloc.c:980
#16 malloc_conf_init_helper (sc_data=sc_data@entry=0x0, bin_shard_sizes=bin_shard_sizes@entry=0x0, initial_call=true, opts_cache=opts_cache@entry=0x7ffca80e6260, buf=<optimized out>, 
    buf@entry=0x7ffca80e7cc0 "\320~\016\250\374\177") at src/jemalloc.c:1038
#17 0x00005629da1fdcac in malloc_conf_init (sc_data=0x7ffca80e6310, bin_shard_sizes=0x7ffca80e6280) at src/jemalloc.c:1445
#18 malloc_init_hard_a0_locked () at src/jemalloc.c:1505
#19 0x00005629da1fff46 in malloc_init_hard () at src/jemalloc.c:1750
#20 0x00005629da1fa2ee in malloc_init () at src/jemalloc.c:223

Cheers,

Richard

Re: [OE-core] pseudo deadlock in rust via malloc backtrace

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 3585 bytes --]

On Thu, Aug 4, 2022 at 10:05 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> Just to note that where we've been seeing pseudo hangs in rust, it
> seems jemalloc calls obtain_malloc_conf() which calls readlink() which
> pseudo intercepts and then calls malloc() which loops back to jemalloc
> which deadlocks. What fun.


I guess pseudo is doing this extra malloc right ? Maybe disable
instrumentation in jemalloc when building non target packages. This might
avoid the issue at least

>
>
> #0  0x00007f3806c84bcf in __lll_lock_wait () from
> /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/sysroots-uninative/x86_64-linux/lib/libc.so.6
> #1  0x00007f3806c8aec2 in pthread_mutex_lock () from
> /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/sysroots-uninative/x86_64-linux/lib/libc.so.6
> #2  0x00005629da227a0e in malloc_mutex_lock_final (mutex=0x5629da45a288
> <init_lock>) at include/jemalloc/internal/mutex.h:155
> #3  _rjem_je_malloc_mutex_lock_slow (mutex=0x5629da45a288 <init_lock>) at
> src/mutex.c:85
> #4  0x00005629da1ffddb in malloc_mutex_lock (tsdn=0x0, mutex=<optimized
> out>) at include/jemalloc/internal/mutex.h:221
> #5  malloc_init_hard () at src/jemalloc.c:1739
> #6  0x00005629da1fa2ee in malloc_init () at src/jemalloc.c:223
> #7  imalloc_init_check (sopts=<optimized out>, dopts=<optimized out>) at
> src/jemalloc.c:2229
> #8  imalloc (sopts=<optimized out>, dopts=<optimized out>) at
> src/jemalloc.c:2260
> #9  _rjem_je_malloc_default (size=101) at src/jemalloc.c:2289
> #10 0x00007f3806c998fe in strdup () from
> /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/sysroots-uninative/x86_64-linux/lib/libc.so.6
> #11 0x00007f380b19da1b in pseudo_set_value () from
> /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so
> #12 0x00007f380b19dbb9 in pseudo_init_util () from
> /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so
> #13 0x00007f380b16bd01 in ?? () from
> /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so
> #14 0x00007f380b19125d in readlink () from
> /home/pokybuild/yocto-worker/qemumips64/build/build/tmp/sysroots-components/x86_64/pseudo-native/usr/lib/pseudo/lib64/libpseudo.so
> #15 0x00005629da1fe07b in obtain_malloc_conf (which_source=2,
> buf=0x7ffca80e7cc0 "\320~\016\250\374\177") at src/jemalloc.c:980
> #16 malloc_conf_init_helper (sc_data=sc_data@entry=0x0,
> bin_shard_sizes=bin_shard_sizes@entry=0x0, initial_call=true,
> opts_cache=opts_cache@entry=0x7ffca80e6260, buf=<optimized out>,
>     buf@entry=0x7ffca80e7cc0 "\320~\016\250\374\177") at
> src/jemalloc.c:1038
> #17 0x00005629da1fdcac in malloc_conf_init (sc_data=0x7ffca80e6310,
> bin_shard_sizes=0x7ffca80e6280) at src/jemalloc.c:1445
> #18 malloc_init_hard_a0_locked () at src/jemalloc.c:1505
> #19 0x00005629da1fff46 in malloc_init_hard () at src/jemalloc.c:1750
> #20 0x00005629da1fa2ee in malloc_init () at src/jemalloc.c:223
>
> Cheers,
>
> Richard
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#168839):
> https://lists.openembedded.org/g/openembedded-core/message/168839
> Mute This Topic: https://lists.openembedded.org/mt/92809824/1997914
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 4662 bytes --]

Re: [OE-core] pseudo deadlock in rust via malloc backtrace

[Richard Purdie]

On Thu, 2022-08-04 at 12:36 +0100, Khem Raj wrote:
> 
> 
> On Thu, Aug 4, 2022 at 10:05 AM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > Just to note that where we've been seeing pseudo hangs in rust, it
> > seems jemalloc calls obtain_malloc_conf() which calls readlink()
> > which
> > pseudo intercepts and then calls malloc() which loops back to
> > jemalloc
> > which deadlocks. What fun.
> > 
> 
> 
> I guess pseudo is doing this extra malloc right ?

Yes

>  Maybe disable instrumentation in jemalloc when building non target
> packages. This might avoid the issue at least 
> 

Sadly it doesn't help as it wants to load it isn't configuration and
the system setup from /proc too.

Cheers,

Richard

Re: [OE-core] pseudo deadlock in rust via malloc backtrace

[Mark Hatle]

On 8/4/22 11:27 AM, Richard Purdie wrote:
> On Thu, 2022-08-04 at 12:36 +0100, Khem Raj wrote:
>>
>>
>> On Thu, Aug 4, 2022 at 10:05 AM Richard Purdie
>> <richard.purdie@linuxfoundation.org> wrote:
>>> Just to note that where we've been seeing pseudo hangs in rust, it
>>> seems jemalloc calls obtain_malloc_conf() which calls readlink()
>>> which
>>> pseudo intercepts and then calls malloc() which loops back to
>>> jemalloc
>>> which deadlocks. What fun.
>>>

(Going way back) the way this was avoided in the distant past was having a 
static variable of whatever the 'max size' was already allocated in a few key 
locations and using that variable instead of a malloc.  (Of course due to it 
having a fixed length a size check and such needed to be done.)

I don't really know the context at this point for this item, but it's an 
alternative way to solve this problem, but will require memory to be allocated 
at compile time, either in a function or a small global pool for usage.

It does make me wonder if there are other function in the system that use malloc 
and if so could they use an alternative, if not the 'is_init' might be the only 
way to handle it.

--Mark

>>
>> I guess pseudo is doing this extra malloc right ?
> 
> Yes
> 
>>   Maybe disable instrumentation in jemalloc when building non target
>> packages. This might avoid the issue at least
>>
> 
> Sadly it doesn't help as it wants to load it isn't configuration and
> the system setup from /proc too.
> 
> Cheers,
> 
> Richard
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#168861): https://lists.openembedded.org/g/openembedded-core/message/168861
> Mute This Topic: https://lists.openembedded.org/mt/92809824/3616948
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [mark.hatle@kernel.crashing.org]
> -=-=-=-=-=-=-=-=-=-=-=-
>