smokey's fork tests hangs?

smokey's fork tests hangs?

[Lange Norbert]

Hello,

I have problems with the fork tests, they both hang for me.

# xeno smokey --run=11 --verbose=2
<hangs>

# xeno smokey --run=20 --verbose=2
no leak withthread
no leak withmutex
no leak withcond
no leak withsem
no leak withnamed sem
no leak withtimer
no leak withmq
<hangs>

When started under gdb, the parent is running into waitpid,
The child is stuck at an innocent locking snippet [1].

#0  __lll_lock_wait_private ()
    at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:63
#1  0x00007ffff7ed74de in __GI___register_atfork (
    prepare=0x7ffff7f5edc0 <atfork_lock>, prepare@entry=0x0,
    parent=parent@entry=0x0,
    child=child@entry=0x7ffff7f9f4e0 <cobalt_fork_handler>,
    dso_handle=0x7ffff7de0e5b <__lll_lock_wait_private+27>)
    at register-atfork.c:40
#2  0x00007ffff7faca9d in __pthread_atfork (prepare=prepare@entry=0x0,
    parent=parent@entry=0x0,
    child=child@entry=0x7ffff7f9f4e0 <cobalt_fork_handler>)
    at pthread_atfork.c:51
#3  0x00007ffff7f9f307 in __cobalt_init ()
    at /home/lano/buildroot/build/xenomai/lib/cobalt/init.c:212
#4  cobalt_init () at /home/lano/buildroot/build/xenomai/lib/cobalt/init.c:252
#5  0x00007ffff7f9f4fc in cobalt_fork_handler ()
    at /home/lano/buildroot/build/xenomai/lib/cobalt/init.c:192
#6  0x00007ffff7ed7898 in __run_fork_handlers (who=who@entry=atfork_run_child)
    at register-atfork.c:134
#7  0x00007ffff7e9b04a in __libc_fork () at ../sysdeps/nptl/fork.c:137
#8  0x0000000000408d0b in ?? ()
#9  0x00000000004051d1 in main ()

[1] - https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/x86_64/lowlevellock.S;h=71dd740202b1d6501a2ffd929aa99cbd99a019ef;hb=1e5c5303a522764d7e9d2302a60e4a32cdb902f1


Mit besten Grüßen / Kind regards

NORBERT LANGE
AT-DES

ANDRITZ HYDRO GmbH
Eibesbrunnergasse 20
1120 Vienna / AUSTRIA
p: +43 50805 56684
norbert.lange@andritz.com
andritz.com
________________________________

This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system.

ANDRITZ HYDRO GmbH


Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation

Firmensitz/ Registered seat: Wien

Firmenbuchgericht/ Court of registry: Handelsgericht Wien

Firmenbuchnummer/ Company registration: FN 61833 g

DVR: 0605077

UID-Nr.: ATU14756806


Thank You
________________________________

Re: smokey's fork tests hangs?

[Jan Kiszka]

On 07.03.19 15:55, Lange Norbert via Xenomai wrote:
> Hello,
> 
> I have problems with the fork tests, they both hang for me.
> 
> # xeno smokey --run=11 --verbose=2
> <hangs>
> 
> # xeno smokey --run=20 --verbose=2
> no leak withthread
> no leak withmutex
> no leak withcond
> no leak withsem
> no leak withnamed sem
> no leak withtimer
> no leak withmq
> <hangs>
> 
> When started under gdb, the parent is running into waitpid,
> The child is stuck at an innocent locking snippet [1].
> 
> #0  __lll_lock_wait_private ()
>      at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:63
> #1  0x00007ffff7ed74de in __GI___register_atfork (
>      prepare=0x7ffff7f5edc0 <atfork_lock>, prepare@entry=0x0,
>      parent=parent@entry=0x0,
>      child=child@entry=0x7ffff7f9f4e0 <cobalt_fork_handler>,
>      dso_handle=0x7ffff7de0e5b <__lll_lock_wait_private+27>)
>      at register-atfork.c:40
> #2  0x00007ffff7faca9d in __pthread_atfork (prepare=prepare@entry=0x0,
>      parent=parent@entry=0x0,
>      child=child@entry=0x7ffff7f9f4e0 <cobalt_fork_handler>)
>      at pthread_atfork.c:51
> #3  0x00007ffff7f9f307 in __cobalt_init ()
>      at /home/lano/buildroot/build/xenomai/lib/cobalt/init.c:212
> #4  cobalt_init () at /home/lano/buildroot/build/xenomai/lib/cobalt/init.c:252
> #5  0x00007ffff7f9f4fc in cobalt_fork_handler ()
>      at /home/lano/buildroot/build/xenomai/lib/cobalt/init.c:192
> #6  0x00007ffff7ed7898 in __run_fork_handlers (who=who@entry=atfork_run_child)
>      at register-atfork.c:134
> #7  0x00007ffff7e9b04a in __libc_fork () at ../sysdeps/nptl/fork.c:137
> #8  0x0000000000408d0b in ?? ()
> #9  0x00000000004051d1 in main ()
> 
> [1] - https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/x86_64/lowlevellock.S;h=71dd740202b1d6501a2ffd929aa99cbd99a019ef;hb=1e5c5303a522764d7e9d2302a60e4a32cdb902f1
> 

Not reproducible here with stable/3.0.x or next, and with ipipe-x86-4.14.y. What 
are your parameters?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

RE: smokey's fork tests hangs?

[Lange Norbert]

>
> Not reproducible here with stable/3.0.x or next, and with ipipe-x86-4.14.y.
> What are your parameters?

Not entirely upstream, but based on ipipe-core-4.14.89-x86-2 and xenomai master,
the difference is contained in the rt_igb driver, which is not even loaded.
Defconfig is attached.

I mostly suspect glibc as the relevant difference, I am using
glibc-2.28-69-g1e5c5303a522764d7e9d2302a60e4a32cdb902f1.

Looking at the strace the child process 1039 is stuck at FUTEX_WAIT_PRIVATE,
Don’t really know how to tackle this.

Norbert
________________________________

This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system.

ANDRITZ HYDRO GmbH


Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation

Firmensitz/ Registered seat: Wien

Firmenbuchgericht/ Court of registry: Handelsgericht Wien

Firmenbuchnummer/ Company registration: FN 61833 g

DVR: 0605077

UID-Nr.: ATU14756806


Thank You
________________________________
-------------- next part --------------
A non-text attachment was scrubbed...
Name: defconfig_rtnet_static
Type: application/octet-stream
Size: 13373 bytes
Desc: defconfig_rtnet_static
URL: <http://xenomai.org/pipermail/xenomai/attachments/20190308/5b62cac5/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: strace.tar.xz
Type: application/octet-stream
Size: 2740 bytes
Desc: strace.tar.xz
URL: <http://xenomai.org/pipermail/xenomai/attachments/20190308/5b62cac5/attachment-0001.obj>

Re: smokey's fork tests hangs?

[Jan Kiszka]

On 08.03.19 14:47, Lange Norbert wrote:
> 
>>
>> Not reproducible here with stable/3.0.x or next, and with ipipe-x86-4.14.y.
>> What are your parameters?
> 
> Not entirely upstream, but based on ipipe-core-4.14.89-x86-2 and xenomai master,
> the difference is contained in the rt_igb driver, which is not even loaded.
> Defconfig is attached.
> 
> I mostly suspect glibc as the relevant difference, I am using
> glibc-2.28-69-g1e5c5303a522764d7e9d2302a60e4a32cdb902f1.
> 
> Looking at the strace the child process 1039 is stuck at FUTEX_WAIT_PRIVATE,
> Don’t really know how to tackle this.

Is this a regression? Then try bisecting the causing commit.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

RE: smokey's fork tests hangs?

[Lange Norbert]

> -----Original Message-----
> From: Jan Kiszka <jan.kiszka@siemens.com>
> Sent: Freitag, 8. März 2019 14:59
> To: Lange Norbert <norbert.lange@andritz.com>; Xenomai
> (xenomai@xenomai.org) <xenomai@xenomai.org>
> Subject: Re: smokey's fork tests hangs?
>
> E-MAIL FROM A NON-ANDRITZ SOURCE: AS A SECURITY MEASURE, PLEASE
> EXERCISE CAUTION WITH E-MAIL CONTENT AND ANY LINKS OR
> ATTACHMENTS.
>
>
> On 08.03.19 14:47, Lange Norbert wrote:
> >
> >>
> >> Not reproducible here with stable/3.0.x or next, and with ipipe-x86-
> 4.14.y.
> >> What are your parameters?
> >
> > Not entirely upstream, but based on ipipe-core-4.14.89-x86-2 and
> > xenomai master, the difference is contained in the rt_igb driver, which is
> not even loaded.
> > Defconfig is attached.
> >
> > I mostly suspect glibc as the relevant difference, I am using
> > glibc-2.28-69-g1e5c5303a522764d7e9d2302a60e4a32cdb902f1.
> >
> > Looking at the strace the child process 1039 is stuck at
> > FUTEX_WAIT_PRIVATE, Don’t really know how to tackle this.
>
> Is this a regression? Then try bisecting the causing commit.

Not really, I haven't ran the smokey suite often (and only for specific tests).
I use buildroot for my rootfs, so I have no easy way of swapping around glibc versions.
I have not encountered other issues outside of those tests.

Which version of glibc is running on your end?

________________________________

This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system.

ANDRITZ HYDRO GmbH


Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation

Firmensitz/ Registered seat: Wien

Firmenbuchgericht/ Court of registry: Handelsgericht Wien

Firmenbuchnummer/ Company registration: FN 61833 g

DVR: 0605077

UID-Nr.: ATU14756806


Thank You
________________________________

Re: smokey's fork tests hangs?

[Jan Kiszka]

On 08.03.19 15:05, Lange Norbert wrote:
> 
> 
>> -----Original Message-----
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>> Sent: Freitag, 8. März 2019 14:59
>> To: Lange Norbert <norbert.lange@andritz.com>; Xenomai
>> (xenomai@xenomai.org) <xenomai@xenomai.org>
>> Subject: Re: smokey's fork tests hangs?
>>
>> E-MAIL FROM A NON-ANDRITZ SOURCE: AS A SECURITY MEASURE, PLEASE
>> EXERCISE CAUTION WITH E-MAIL CONTENT AND ANY LINKS OR
>> ATTACHMENTS.
>>
>>
>> On 08.03.19 14:47, Lange Norbert wrote:
>>>
>>>>
>>>> Not reproducible here with stable/3.0.x or next, and with ipipe-x86-
>> 4.14.y.
>>>> What are your parameters?
>>>
>>> Not entirely upstream, but based on ipipe-core-4.14.89-x86-2 and
>>> xenomai master, the difference is contained in the rt_igb driver, which is
>> not even loaded.
>>> Defconfig is attached.
>>>
>>> I mostly suspect glibc as the relevant difference, I am using
>>> glibc-2.28-69-g1e5c5303a522764d7e9d2302a60e4a32cdb902f1.
>>>
>>> Looking at the strace the child process 1039 is stuck at
>>> FUTEX_WAIT_PRIVATE, Don’t really know how to tackle this.
>>
>> Is this a regression? Then try bisecting the causing commit.
> 
> Not really, I haven't ran the smokey suite often (and only for specific tests).
> I use buildroot for my rootfs, so I have no easy way of swapping around glibc versions.
> I have not encountered other issues outside of those tests.
> 
> Which version of glibc is running on your end?
> 

Mine is in fact old: 2.19. Colleagues are on Debian 9 with 2.24 where this used 
to work but was not tested recently against master.

I haven't tried buildroot with Xenomai yet - is there a working 3.x recipe, and 
everything is out-of-the-box?

Regarding how to possibly debug this: If one thread is stuck, you could check if 
there are other threads in that application that may hold the lock. Or if the 
lock content is actually invalid and therefore blocking the caller (memory 
corruption). You should be able to read out the thread ID of the owner from a 
valid lock structure.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

RE: smokey's fork tests hangs?

[Lange Norbert]

> -----Original Message-----
> From: Jan Kiszka <jan.kiszka@siemens.com>
> Sent: Freitag, 8. März 2019 16:23
> To: Lange Norbert <norbert.lange@andritz.com>; Xenomai
> (xenomai@xenomai.org) <xenomai@xenomai.org>
> Subject: Re: smokey's fork tests hangs?
>
> E-MAIL FROM A NON-ANDRITZ SOURCE: AS A SECURITY MEASURE, PLEASE
> EXERCISE CAUTION WITH E-MAIL CONTENT AND ANY LINKS OR
> ATTACHMENTS.
>
>
> On 08.03.19 15:05, Lange Norbert wrote:
> >
> >
> >> -----Original Message-----
> >> From: Jan Kiszka <jan.kiszka@siemens.com>
> >> Sent: Freitag, 8. März 2019 14:59
> >> To: Lange Norbert <norbert.lange@andritz.com>; Xenomai
> >> (xenomai@xenomai.org) <xenomai@xenomai.org>
> >> Subject: Re: smokey's fork tests hangs?
> >>
> >> E-MAIL FROM A NON-ANDRITZ SOURCE: AS A SECURITY MEASURE,
> PLEASE
> >> EXERCISE CAUTION WITH E-MAIL CONTENT AND ANY LINKS OR
> ATTACHMENTS.
> >>
> >>
> >> On 08.03.19 14:47, Lange Norbert wrote:
> >>>
> >>>>
> >>>> Not reproducible here with stable/3.0.x or next, and with
> >>>> ipipe-x86-
> >> 4.14.y.
> >>>> What are your parameters?
> >>>
> >>> Not entirely upstream, but based on ipipe-core-4.14.89-x86-2 and
> >>> xenomai master, the difference is contained in the rt_igb driver,
> >>> which is
> >> not even loaded.
> >>> Defconfig is attached.
> >>>
> >>> I mostly suspect glibc as the relevant difference, I am using
> >>> glibc-2.28-69-g1e5c5303a522764d7e9d2302a60e4a32cdb902f1.
> >>>
> >>> Looking at the strace the child process 1039 is stuck at
> >>> FUTEX_WAIT_PRIVATE, Don’t really know how to tackle this.
> >>
> >> Is this a regression? Then try bisecting the causing commit.
> >
> > Not really, I haven't ran the smokey suite often (and only for specific tests).
> > I use buildroot for my rootfs, so I have no easy way of swapping around
> glibc versions.
> > I have not encountered other issues outside of those tests.
> >
> > Which version of glibc is running on your end?
> >
>
> Mine is in fact old: 2.19. Colleagues are on Debian 9 with 2.24 where this used
> to work but was not tested recently against master.
>
> I haven't tried buildroot with Xenomai yet - is there a working 3.x recipe, and
> everything is out-of-the-box?

Yes, and thats both a blessing(easy to start with) and a curse(by defaults builds everything inc. toolchain, packages mostly available in 1 version).
It has a configuration system similar to the Kernel, getting to a xenomai userspace would take the following steps:

make O=/tmp/c defconfig
make O=/tmp/c menuconfig
make O=/tmp/c

You would have to change in menuconfig:
Target options: x86_64
Toolchain -> C library: glibc
Target packages -> Real-Time: xenomai (set cobalt + test utils, set version to 3.0.8)

(or you drop the attached file into configs/xenomai_defconfig and run 'make O=/tmp/c xenomai_defconfig')

And you would need to remove the patch in package/xenomai as it won't apply with 3.0.8.

> Regarding how to possibly debug this: If one thread is stuck, you could check
> if there are other threads in that application that may hold the lock. Or if the
> lock content is actually invalid and therefore blocking the caller (memoryC
> corruption). You should be able to read out the thread ID of the owner from
> a valid lock structure.

I don’t know what the fork+exec sequence should do. I only know libc waits forever for a private futex,
potentially synchronizing with a thread that has not been created yet.
As said, I haven’t encountered any other issues and I am not able to sink a lot time into something
that’s not a problem right now. I would still like to narrow down the cause.

Norbert
________________________________

This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system.

ANDRITZ HYDRO GmbH


Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation

Firmensitz/ Registered seat: Wien

Firmenbuchgericht/ Court of registry: Handelsgericht Wien

Firmenbuchnummer/ Company registration: FN 61833 g

DVR: 0605077

UID-Nr.: ATU14756806


Thank You
________________________________
-------------- next part --------------
A non-text attachment was scrubbed...
Name: xenomai_defconfig
Type: application/octet-stream
Size: 242 bytes
Desc: xenomai_defconfig
URL: <http://xenomai.org/pipermail/xenomai/attachments/20190311/22a1cbff/attachment.obj>