RE: smokey's fork tests hangs with glibc 2.28+ ?

RE: smokey's fork tests hangs with glibc 2.28+ ?

[Lange Norbert]

Well, swapping out glibc with 2.27 removes the issue, so it has been introduced afterwards,
the current buildroot will use 2.28 so you should be able to build yourself a reproducer that way.
I will also attach a prebuilt rootfs once the clean rebuild is done.

Unfortunately the smokey suite is not available with the Mercury core, might be easier to isolate that way.

In the future, maybe some battletested glibc versions could be recommended for xenomai?

> -----Original Message-----
> From: Jan Kiszka <jan.kiszka@siemens.com>
> Sent: Freitag, 8. März 2019 16:23
> To: Lange Norbert <norbert.lange@andritz.com>; Xenomai
> (xenomai@xenomai.org) <xenomai@xenomai.org>
> Subject: Re: smokey's fork tests hangs?
>
> E-MAIL FROM A NON-ANDRITZ SOURCE: AS A SECURITY MEASURE, PLEASE
> EXERCISE CAUTION WITH E-MAIL CONTENT AND ANY LINKS OR
> ATTACHMENTS.
>
>
> On 08.03.19 15:05, Lange Norbert wrote:
> >
> >
> >> -----Original Message-----
> >> From: Jan Kiszka <jan.kiszka@siemens.com>
> >> Sent: Freitag, 8. März 2019 14:59
> >> To: Lange Norbert <norbert.lange@andritz.com>; Xenomai
> >> (xenomai@xenomai.org) <xenomai@xenomai.org>
> >> Subject: Re: smokey's fork tests hangs?
> >>
> >> E-MAIL FROM A NON-ANDRITZ SOURCE: AS A SECURITY MEASURE,
> PLEASE
> >> EXERCISE CAUTION WITH E-MAIL CONTENT AND ANY LINKS OR
> ATTACHMENTS.
> >>
> >>
> >> On 08.03.19 14:47, Lange Norbert wrote:
> >>>
> >>>>
> >>>> Not reproducible here with stable/3.0.x or next, and with
> >>>> ipipe-x86-
> >> 4.14.y.
> >>>> What are your parameters?
> >>>
> >>> Not entirely upstream, but based on ipipe-core-4.14.89-x86-2 and
> >>> xenomai master, the difference is contained in the rt_igb driver,
> >>> which is
> >> not even loaded.
> >>> Defconfig is attached.
> >>>
> >>> I mostly suspect glibc as the relevant difference, I am using
> >>> glibc-2.28-69-g1e5c5303a522764d7e9d2302a60e4a32cdb902f1.
> >>>
> >>> Looking at the strace the child process 1039 is stuck at
> >>> FUTEX_WAIT_PRIVATE, Don’t really know how to tackle this.
> >>
> >> Is this a regression? Then try bisecting the causing commit.
> >
> > Not really, I haven't ran the smokey suite often (and only for specific tests).
> > I use buildroot for my rootfs, so I have no easy way of swapping around
> glibc versions.
> > I have not encountered other issues outside of those tests.
> >
> > Which version of glibc is running on your end?
> >
>
> Mine is in fact old: 2.19. Colleagues are on Debian 9 with 2.24 where this used
> to work but was not tested recently against master.
>
> I haven't tried buildroot with Xenomai yet - is there a working 3.x recipe, and
> everything is out-of-the-box?
>
> Regarding how to possibly debug this: If one thread is stuck, you could check
> if there are other threads in that application that may hold the lock. Or if the
> lock content is actually invalid and therefore blocking the caller (memory
> corruption). You should be able to read out the thread ID of the owner from
> a valid lock structure.
>
> Jan
>
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate
> Competence Center Embedded Linux
________________________________

This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system.

ANDRITZ HYDRO GmbH


Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation

Firmensitz/ Registered seat: Wien

Firmenbuchgericht/ Court of registry: Handelsgericht Wien

Firmenbuchnummer/ Company registration: FN 61833 g

DVR: 0605077

UID-Nr.: ATU14756806


Thank You
________________________________

Re: smokey's fork tests hangs with glibc 2.28+ ?

[Jan Kiszka]

On 11.03.19 14:40, Lange Norbert wrote:
> Well, swapping out glibc with 2.27 removes the issue, so it has been introduced afterwards,
> the current buildroot will use 2.28 so you should be able to build yourself a reproducer that way.
> I will also attach a prebuilt rootfs once the clean rebuild is done.

Ok, thanks for narrowing that further down.

I just realized that buster (Debian 10) will use 2.28 as well. Will try that 
baseline as well.

> 
> Unfortunately the smokey suite is not available with the Mercury core, might be easier to isolate that way.
> 
> In the future, maybe some battletested glibc versions could be recommended for xenomai?

Maintained distributions are generally a good choice. My plan is to define and 
test reference images via https://gitlab.denx.de/Xenomai/xenomai-images, but we 
are not there yet.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

RE: smokey's fork tests hangs with glibc 2.28+ ?

[Lange Norbert]

Thats a rootfs which reproduces the issue. Identical setup, but glibc 2.27 will not reproduce.

> -----Original Message-----
> From: Jan Kiszka <jan.kiszka@siemens.com>
> Sent: Montag, 11. März 2019 14:50
> To: Lange Norbert <norbert.lange@andritz.com>; Xenomai
> (xenomai@xenomai.org) <xenomai@xenomai.org>
> Subject: Re: smokey's fork tests hangs with glibc 2.28+ ?
>
> E-MAIL FROM A NON-ANDRITZ SOURCE: AS A SECURITY MEASURE, PLEASE
> EXERCISE CAUTION WITH E-MAIL CONTENT AND ANY LINKS OR
> ATTACHMENTS.
>
>
> On 11.03.19 14:40, Lange Norbert wrote:
> > Well, swapping out glibc with 2.27 removes the issue, so it has been
> > introduced afterwards, the current buildroot will use 2.28 so you should be
> able to build yourself a reproducer that way.
> > I will also attach a prebuilt rootfs once the clean rebuild is done.
>
> Ok, thanks for narrowing that further down.
>
> I just realized that buster (Debian 10) will use 2.28 as well. Will try that
> baseline as well.
>
> >
> > Unfortunately the smokey suite is not available with the Mercury core,
> might be easier to isolate that way.
> >
> > In the future, maybe some battletested glibc versions could be
> recommended for xenomai?
>
> Maintained distributions are generally a good choice. My plan is to define and
> test reference images via https://gitlab.denx.de/Xenomai/xenomai-images,
> but we are not there yet.
>
> Jan
>
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate
> Competence Center Embedded Linux
________________________________

This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system.

ANDRITZ HYDRO GmbH


Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation

Firmensitz/ Registered seat: Wien

Firmenbuchgericht/ Court of registry: Handelsgericht Wien

Firmenbuchnummer/ Company registration: FN 61833 g

DVR: 0605077

UID-Nr.: ATU14756806


Thank You
________________________________
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rootfs.tar.xz
Type: application/octet-stream
Size: 1893344 bytes
Desc: rootfs.tar.xz
URL: <http://xenomai.org/pipermail/xenomai/attachments/20190311/aec534ed/attachment.obj>

Re: smokey's fork tests hangs with glibc 2.28+ ?

[Philippe Gerum]

On 3/11/19 3:08 PM, Lange Norbert via Xenomai wrote:
> Thats a rootfs which reproduces the issue. Identical setup, but glibc 2.27 will not reproduce.

The issue was introduced by [1], which already triggered a bug in the
glibc test suite [2]. In short, calling pthread_atfork() like
__cobalt_init() does over the context of a fork handler is now invalid,
because the glibc now holds the internal atfork_lock while running the
fork handlers. The cobalt fork handler needs to be registered earlier,
outside of such context.

This change was introduced between glibc-2.27.9000 and glibc-2.28.

[1] git://sourceware.org/git/glibc.git, 27761a104
[2] git://sourceware.org/git/glibc.git, 669ff911e

-- 
Philippe.

RE: smokey's fork tests hangs with glibc 2.28+ ?

[Lange Norbert]

Thanks Phillipe,

assume just for a moment that I know little of the issues,
is there any harm using glibc 2.28 for now, given that I don’t
run into those deadlocks or could there be some more
subtile breakage involved?

Norbert

> -----Original Message-----
> From: Philippe Gerum <rpm@xenomai.org>
> Sent: Montag, 11. März 2019 15:19
> To: Lange Norbert <norbert.lange@andritz.com>; Jan Kiszka
> <jan.kiszka@siemens.com>; Xenomai (xenomai@xenomai.org)
> <xenomai@xenomai.org>
> Subject: Re: smokey's fork tests hangs with glibc 2.28+ ?
>
> E-MAIL FROM A NON-ANDRITZ SOURCE: AS A SECURITY MEASURE, PLEASE
> EXERCISE CAUTION WITH E-MAIL CONTENT AND ANY LINKS OR
> ATTACHMENTS.
>
>
> On 3/11/19 3:08 PM, Lange Norbert via Xenomai wrote:
> > Thats a rootfs which reproduces the issue. Identical setup, but glibc 2.27 will
> not reproduce.
>
> The issue was introduced by [1], which already triggered a bug in the glibc
> test suite [2]. In short, calling pthread_atfork() like
> __cobalt_init() does over the context of a fork handler is now invalid,
> because the glibc now holds the internal atfork_lock while running the fork
> handlers. The cobalt fork handler needs to be registered earlier, outside of
> such context.
>
> This change was introduced between glibc-2.27.9000 and glibc-2.28.
>
> [1] git://sourceware.org/git/glibc.git, 27761a104 [2]
> git://sourceware.org/git/glibc.git, 669ff911e
>
> --
> Philippe.
________________________________

This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system.

ANDRITZ HYDRO GmbH


Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation

Firmensitz/ Registered seat: Wien

Firmenbuchgericht/ Court of registry: Handelsgericht Wien

Firmenbuchnummer/ Company registration: FN 61833 g

DVR: 0605077

UID-Nr.: ATU14756806


Thank You
________________________________

RE: smokey's fork tests hangs with glibc 2.28+ ?

[Lange Norbert]

Thanks Phillipe,

assume just for a moment that I know little of the issues,
is there any harm using glibc 2.28 for now, given that I don’t
run into those deadlocks or could there be some more
subtile breakage involved?

Norbert

> -----Original Message-----
> From: Philippe Gerum <rpm@xenomai.org>
> Sent: Montag, 11. März 2019 15:19
> To: Lange Norbert <norbert.lange@andritz.com>; Jan Kiszka
> <jan.kiszka@siemens.com>; Xenomai (xenomai@xenomai.org)
> <xenomai@xenomai.org>
> Subject: Re: smokey's fork tests hangs with glibc 2.28+ ?
>
> E-MAIL FROM A NON-ANDRITZ SOURCE: AS A SECURITY MEASURE, PLEASE
> EXERCISE CAUTION WITH E-MAIL CONTENT AND ANY LINKS OR
> ATTACHMENTS.
>
>
> On 3/11/19 3:08 PM, Lange Norbert via Xenomai wrote:
> > Thats a rootfs which reproduces the issue. Identical setup, but glibc 2.27 will
> not reproduce.
>
> The issue was introduced by [1], which already triggered a bug in the glibc
> test suite [2]. In short, calling pthread_atfork() like
> __cobalt_init() does over the context of a fork handler is now invalid,
> because the glibc now holds the internal atfork_lock while running the fork
> handlers. The cobalt fork handler needs to be registered earlier, outside of
> such context.
>
> This change was introduced between glibc-2.27.9000 and glibc-2.28.
>
> [1] git://sourceware.org/git/glibc.git, 27761a104 [2]
> git://sourceware.org/git/glibc.git, 669ff911e
>
> --
> Philippe.
________________________________

This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system.

ANDRITZ HYDRO GmbH


Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation

Firmensitz/ Registered seat: Wien

Firmenbuchgericht/ Court of registry: Handelsgericht Wien

Firmenbuchnummer/ Company registration: FN 61833 g

DVR: 0605077

UID-Nr.: ATU14756806


Thank You
________________________________

Re: smokey's fork tests hangs with glibc 2.28+ ?

[Philippe Gerum]

On 3/11/19 3:41 PM, Lange Norbert wrote:
> Thanks Phillipe,
> 
> assume just for a moment that I know little of the issues,
> is there any harm using glibc 2.28 for now, given that I don’t
> run into those deadlocks or could there be some more
> subtile breakage involved?

I don't know. This said, this question includes its own answer: 2.28 is
a recent release which few people have already used in a Xenomai
context, so any obvious bug we did not find yet might look subtle at
first glance.

-- 
Philippe.