Problems with perl 5.22

Problems with perl 5.22

[Gary Thomas]

I'm working on a package (amanda - the Advanced Maryland Archiving
system) that is written heavily in perl with swig interfaces to C.
This code ran great until the update to perl 5.22; it now dies a
horrible death on almost every activity.  These failures seem to
always be in the swig generated wrappers, but that may just be
where most of the work gets done.

I've narrowed this down to exactly the change to perl 5.22 from
5.20. Using bisect as well as experimentation (e.g. trying all
the compiler combinations that have occurred since a last good
version) and I can go from working to failing by only the change
in perl.

The interesting (scary) thing is that I've built amanda for my
target natively on my board running debian, including perl 5.22.
This means I can't say definitively  that perl 5.22 is the culprit
as on debian it runs fine.  So, it's got something to do with the
OE environment/porting/packaging of perl and not just the revision.

I've also tested this on multiple architectures (ARM, PowerPC) with
the same results - with perl 5.20 amanda works, with perl 5.22 it fails.

I've compared the actual 5.22.1 sources used by OE-core and debian
and they are subtly different, although I can't pinpoint any change
that might be responsible.

For the moment, I can just fall back to perl 5.20 for my target
that needs to run amanda, but this isn't a real solution (e.g.
in this state I can't propose my recipe to any layer as it's
totally broken with the current OE-core).  I'd like to see this
fixed but the amanda code (swig wrappers) are horribly complex
which makes debugging quite the challenge, not to mention they
may be about the only way to uncover the bug, whether it's in
amanda or perl.

Any suggestions on how to move forward?

Thanks for your thoughts

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Problems with perl 5.22

[Jens Rehsack]

[-- Attachment #1: Type: text/plain, Size: 2613 bytes --]


> Am 10.03.2016 um 06:13 schrieb Gary Thomas <gary@mlbassoc.com>:
> 
> I'm working on a package (amanda - the Advanced Maryland Archiving
> system) that is written heavily in perl with swig interfaces to C.
> This code ran great until the update to perl 5.22; it now dies a
> horrible death on almost every activity.  These failures seem to
> always be in the swig generated wrappers, but that may just be
> where most of the work gets done.
> 
> I've narrowed this down to exactly the change to perl 5.22 from
> 5.20. Using bisect as well as experimentation (e.g. trying all
> the compiler combinations that have occurred since a last good
> version) and I can go from working to failing by only the change
> in perl.
> 
> The interesting (scary) thing is that I've built amanda for my
> target natively on my board running debian, including perl 5.22.
> This means I can't say definitively  that perl 5.22 is the culprit
> as on debian it runs fine.  So, it's got something to do with the
> OE environment/porting/packaging of perl and not just the revision.
> 
> I've also tested this on multiple architectures (ARM, PowerPC) with
> the same results - with perl 5.20 amanda works, with perl 5.22 it fails.
> 
> I've compared the actual 5.22.1 sources used by OE-core and debian
> and they are subtly different, although I can't pinpoint any change
> that might be responsible.
> 
> For the moment, I can just fall back to perl 5.20 for my target
> that needs to run amanda, but this isn't a real solution (e.g.
> in this state I can't propose my recipe to any layer as it's
> totally broken with the current OE-core).  I'd like to see this
> fixed but the amanda code (swig wrappers) are horribly complex
> which makes debugging quite the challenge, not to mention they
> may be about the only way to uncover the bug, whether it's in
> amanda or perl.
> 
> Any suggestions on how to move forward?

Since I have no clue what's wrong and how it fails (backtrace
would point in some directions), several ideas might work:

How clean is your build location (we realize that often between
updates some files remain in our target images until we wipe
tmp/ - cleansstate for image doesn't help ...)?

Did you prove the library path's of your *.so's? Perl does
almost everything within libperl.so - build against wrong version
causes in weird crashes (scan DBI mailing list for admin's
build issues of DBI on AIX/HP-UX ...).

Maybe share your recipe can help to reproduce the problem
elsewhere and debug locally.

Cheers
--
Jens Rehsack - rehsack@gmail.com


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 859 bytes --]

Re: Problems with perl 5.22

[Gary Thomas]

On 2016-03-10 16:54, Jens Rehsack wrote:
>
>> Am 10.03.2016 um 06:13 schrieb Gary Thomas <gary@mlbassoc.com>:
>>
>> I'm working on a package (amanda - the Advanced Maryland Archiving
>> system) that is written heavily in perl with swig interfaces to C.
>> This code ran great until the update to perl 5.22; it now dies a
>> horrible death on almost every activity.  These failures seem to
>> always be in the swig generated wrappers, but that may just be
>> where most of the work gets done.
>>
>> I've narrowed this down to exactly the change to perl 5.22 from
>> 5.20. Using bisect as well as experimentation (e.g. trying all
>> the compiler combinations that have occurred since a last good
>> version) and I can go from working to failing by only the change
>> in perl.
>>
>> The interesting (scary) thing is that I've built amanda for my
>> target natively on my board running debian, including perl 5.22.
>> This means I can't say definitively  that perl 5.22 is the culprit
>> as on debian it runs fine.  So, it's got something to do with the
>> OE environment/porting/packaging of perl and not just the revision.
>>
>> I've also tested this on multiple architectures (ARM, PowerPC) with
>> the same results - with perl 5.20 amanda works, with perl 5.22 it fails.
>>
>> I've compared the actual 5.22.1 sources used by OE-core and debian
>> and they are subtly different, although I can't pinpoint any change
>> that might be responsible.
>>
>> For the moment, I can just fall back to perl 5.20 for my target
>> that needs to run amanda, but this isn't a real solution (e.g.
>> in this state I can't propose my recipe to any layer as it's
>> totally broken with the current OE-core).  I'd like to see this
>> fixed but the amanda code (swig wrappers) are horribly complex
>> which makes debugging quite the challenge, not to mention they
>> may be about the only way to uncover the bug, whether it's in
>> amanda or perl.
>>
>> Any suggestions on how to move forward?
>
> Since I have no clue what's wrong and how it fails (backtrace
> would point in some directions), several ideas might work:

The problem here is that the failures happen in the swig generated
files which are very difficult to debug (and don't really track
in the -dbg packages)

>
> How clean is your build location (we realize that often between
> updates some files remain in our target images until we wipe
> tmp/ - cleansstate for image doesn't help ...)?

100% clean, I've started from scratch many times

>
> Did you prove the library path's of your *.so's? Perl does
> almost everything within libperl.so - build against wrong version
> causes in weird crashes (scan DBI mailing list for admin's
> build issues of DBI on AIX/HP-UX ...).

As I said, this all works fine when I build [from scratch] with
perl-5.20 and not [from scratch] with perl 5.22

>
> Maybe share your recipe can help to reproduce the problem
> elsewhere and debug locally.
>

I'd be happy to share, perhaps privately if you're inclined?
It's a complicated setup and testing can be a bit tedious, but
I'm happy for any help.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Problems with perl 5.22

[Stephen Arnold]

As another point of information, I'm having similar issue with perl
5.22.1 on some arm boxes (native Gentoo builds) and even some odd
build differences between amd box and intel corei5 9 (I know,
weird...).

The issue I'm hitting is basically a horribly broken upstream
Configure script, and in my case it misdetects some libc functions as
"not available" and substitutes even more horribly broken replacement
functions.  Hacking the resulting config file *after* configure makes
it build successfully; seems to run okay but I'm not trying anything
like amanda to actually test it.

Like I said, it doesn't sound like your description but please
check/fix these two defines, or any others that are incorrect; this is
my ebuild post-configure hack:

sed -i \
                -e "s|d_fork='undef'|d_fork='define'|" \
                -e "s|d_memcpy='undef'|d_memcpy='define'|" \
                        "${S}"/config.sh

Steve

On Thu, Mar 10, 2016 at 8:00 AM, Gary Thomas <gary@mlbassoc.com> wrote:
> On 2016-03-10 16:54, Jens Rehsack wrote:
>>
>>
>>> Am 10.03.2016 um 06:13 schrieb Gary Thomas <gary@mlbassoc.com>:
>>>
>>> I'm working on a package (amanda - the Advanced Maryland Archiving
>>> system) that is written heavily in perl with swig interfaces to C.
>>> This code ran great until the update to perl 5.22; it now dies a
>>> horrible death on almost every activity.  These failures seem to
>>> always be in the swig generated wrappers, but that may just be
>>> where most of the work gets done.
>>>
>>> I've narrowed this down to exactly the change to perl 5.22 from
>>> 5.20. Using bisect as well as experimentation (e.g. trying all
>>> the compiler combinations that have occurred since a last good
>>> version) and I can go from working to failing by only the change
>>> in perl.
>>>
>>> The interesting (scary) thing is that I've built amanda for my
>>> target natively on my board running debian, including perl 5.22.
>>> This means I can't say definitively  that perl 5.22 is the culprit
>>> as on debian it runs fine.  So, it's got something to do with the
>>> OE environment/porting/packaging of perl and not just the revision.
>>>
>>> I've also tested this on multiple architectures (ARM, PowerPC) with
>>> the same results - with perl 5.20 amanda works, with perl 5.22 it fails.
>>>
>>> I've compared the actual 5.22.1 sources used by OE-core and debian
>>> and they are subtly different, although I can't pinpoint any change
>>> that might be responsible.
>>>
>>> For the moment, I can just fall back to perl 5.20 for my target
>>> that needs to run amanda, but this isn't a real solution (e.g.
>>> in this state I can't propose my recipe to any layer as it's
>>> totally broken with the current OE-core).  I'd like to see this
>>> fixed but the amanda code (swig wrappers) are horribly complex
>>> which makes debugging quite the challenge, not to mention they
>>> may be about the only way to uncover the bug, whether it's in
>>> amanda or perl.
>>>
>>> Any suggestions on how to move forward?
>>
>>
>> Since I have no clue what's wrong and how it fails (backtrace
>> would point in some directions), several ideas might work:
>
>
> The problem here is that the failures happen in the swig generated
> files which are very difficult to debug (and don't really track
> in the -dbg packages)
>
>>
>> How clean is your build location (we realize that often between
>> updates some files remain in our target images until we wipe
>> tmp/ - cleansstate for image doesn't help ...)?
>
>
> 100% clean, I've started from scratch many times
>
>>
>> Did you prove the library path's of your *.so's? Perl does
>> almost everything within libperl.so - build against wrong version
>> causes in weird crashes (scan DBI mailing list for admin's
>> build issues of DBI on AIX/HP-UX ...).
>
>
> As I said, this all works fine when I build [from scratch] with
> perl-5.20 and not [from scratch] with perl 5.22
>
>>
>> Maybe share your recipe can help to reproduce the problem
>> elsewhere and debug locally.
>>
>
> I'd be happy to share, perhaps privately if you're inclined?
> It's a complicated setup and testing can be a bit tedious, but
> I'm happy for any help.
>
>
> --
> ------------------------------------------------------------
> Gary Thomas                 |  Consulting for the
> MLB Associates              |    Embedded world
> ------------------------------------------------------------
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: Problems with perl 5.22

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 3154 bytes --]

Compare the output of configure run on working and non working versions.
Usually this is very helpful. See if its enabling or disabling something
new.
On Mar 10, 2016 10:55 PM, "Jens Rehsack" <rehsack@gmail.com> wrote:

>
> > Am 10.03.2016 um 06:13 schrieb Gary Thomas <gary@mlbassoc.com>:
> >
> > I'm working on a package (amanda - the Advanced Maryland Archiving
> > system) that is written heavily in perl with swig interfaces to C.
> > This code ran great until the update to perl 5.22; it now dies a
> > horrible death on almost every activity.  These failures seem to
> > always be in the swig generated wrappers, but that may just be
> > where most of the work gets done.
> >
> > I've narrowed this down to exactly the change to perl 5.22 from
> > 5.20. Using bisect as well as experimentation (e.g. trying all
> > the compiler combinations that have occurred since a last good
> > version) and I can go from working to failing by only the change
> > in perl.
> >
> > The interesting (scary) thing is that I've built amanda for my
> > target natively on my board running debian, including perl 5.22.
> > This means I can't say definitively  that perl 5.22 is the culprit
> > as on debian it runs fine.  So, it's got something to do with the
> > OE environment/porting/packaging of perl and not just the revision.
> >
> > I've also tested this on multiple architectures (ARM, PowerPC) with
> > the same results - with perl 5.20 amanda works, with perl 5.22 it fails.
> >
> > I've compared the actual 5.22.1 sources used by OE-core and debian
> > and they are subtly different, although I can't pinpoint any change
> > that might be responsible.
> >
> > For the moment, I can just fall back to perl 5.20 for my target
> > that needs to run amanda, but this isn't a real solution (e.g.
> > in this state I can't propose my recipe to any layer as it's
> > totally broken with the current OE-core).  I'd like to see this
> > fixed but the amanda code (swig wrappers) are horribly complex
> > which makes debugging quite the challenge, not to mention they
> > may be about the only way to uncover the bug, whether it's in
> > amanda or perl.
> >
> > Any suggestions on how to move forward?
>
> Since I have no clue what's wrong and how it fails (backtrace
> would point in some directions), several ideas might work:
>
> How clean is your build location (we realize that often between
> updates some files remain in our target images until we wipe
> tmp/ - cleansstate for image doesn't help ...)?
>
> Did you prove the library path's of your *.so's? Perl does
> almost everything within libperl.so - build against wrong version
> causes in weird crashes (scan DBI mailing list for admin's
> build issues of DBI on AIX/HP-UX ...).
>
> Maybe share your recipe can help to reproduce the problem
> elsewhere and debug locally.
>
> Cheers
> --
> Jens Rehsack - rehsack@gmail.com
>
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>

[-- Attachment #2: Type: text/html, Size: 3974 bytes --]

Re: Problems with perl 5.22

[Gary Thomas]

On 03/10/2016 04:54 PM, Jens Rehsack wrote:
>
>> Am 10.03.2016 um 06:13 schrieb Gary Thomas <gary@mlbassoc.com>:
>>
>> I'm working on a package (amanda - the Advanced Maryland Archiving
>> system) that is written heavily in perl with swig interfaces to C.
>> This code ran great until the update to perl 5.22; it now dies a
>> horrible death on almost every activity.  These failures seem to
>> always be in the swig generated wrappers, but that may just be
>> where most of the work gets done.
>>
>> I've narrowed this down to exactly the change to perl 5.22 from
>> 5.20. Using bisect as well as experimentation (e.g. trying all
>> the compiler combinations that have occurred since a last good
>> version) and I can go from working to failing by only the change
>> in perl.
>>
>> The interesting (scary) thing is that I've built amanda for my
>> target natively on my board running debian, including perl 5.22.
>> This means I can't say definitively  that perl 5.22 is the culprit
>> as on debian it runs fine.  So, it's got something to do with the
>> OE environment/porting/packaging of perl and not just the revision.
>>
>> I've also tested this on multiple architectures (ARM, PowerPC) with
>> the same results - with perl 5.20 amanda works, with perl 5.22 it fails.
>>
>> I've compared the actual 5.22.1 sources used by OE-core and debian
>> and they are subtly different, although I can't pinpoint any change
>> that might be responsible.
>>
>> For the moment, I can just fall back to perl 5.20 for my target
>> that needs to run amanda, but this isn't a real solution (e.g.
>> in this state I can't propose my recipe to any layer as it's
>> totally broken with the current OE-core).  I'd like to see this
>> fixed but the amanda code (swig wrappers) are horribly complex
>> which makes debugging quite the challenge, not to mention they
>> may be about the only way to uncover the bug, whether it's in
>> amanda or perl.
>>
>> Any suggestions on how to move forward?
>
> Since I have no clue what's wrong and how it fails (backtrace
> would point in some directions), several ideas might work:
>
> How clean is your build location (we realize that often between
> updates some files remain in our target images until we wipe
> tmp/ - cleansstate for image doesn't help ...)?
>
> Did you prove the library path's of your *.so's? Perl does
> almost everything within libperl.so - build against wrong version
> causes in weird crashes (scan DBI mailing list for admin's
> build issues of DBI on AIX/HP-UX ...).
>
> Maybe share your recipe can help to reproduce the problem
> elsewhere and debug locally.

I've spent quite a lot of time trying to come up with debuggable
scenario(s) on many different targets.  This is all packed up in
   https://github.com/GaryThomas/meta-amanda

If you give the README a look, you'll see that there are plenty
of issues to be looked at.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------