perl 5.22 and 32 bit targets

perl 5.22 and 32 bit targets

[Gary Thomas]

I hope this is the correct place to discuss this problem.  It
is all about a difference in behavior between a program built
using bitbake/OE (only OE-core is needed) vs building the program
on the target hardware itself.

I've been struggling with this problem since perl was upgraded
to version 5.22.  I'm working on Amanda (Advanced Maryland Archive
tool) which is written primarily in perl and uses swig interfaces
to access native C functions.  This code works great when using
the previous perl (5.20.x) but fails on all 32 bit targets with
perl 5.22

The interesting thing is that if I build Amanda on my target
directly (using SDK tools), it works perfectly even with perl
5.22, so it seems that there is some [subtle] difference between
building using bitbake/OE than when built on the self-hosted
target.  I've compared the builds and the only thing I could
find (from the output of configure) is a difference in sizeof(off_t)
Sadly, when I tried to adjust this in the OE build, it didn't
make any difference, but perhaps I didn't make this change
correctly or completely.

Anyway, I'm looking for some help to solve this.  I've put
all the relevant pieces and notes about the process at:
   https://github.com/GaryThomas/meta-amanda.git

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: perl 5.22 and 32 bit targets

[Khem Raj]

On Tue, Mar 22, 2016 at 9:53 PM, Gary Thomas <gary@mlbassoc.com> wrote:
> I hope this is the correct place to discuss this problem.  It
> is all about a difference in behavior between a program built
> using bitbake/OE (only OE-core is needed) vs building the program
> on the target hardware itself.
>
> I've been struggling with this problem since perl was upgraded
> to version 5.22.  I'm working on Amanda (Advanced Maryland Archive
> tool) which is written primarily in perl and uses swig interfaces
> to access native C functions.  This code works great when using
> the previous perl (5.20.x) but fails on all 32 bit targets with
> perl 5.22
>
> The interesting thing is that if I build Amanda on my target
> directly (using SDK tools), it works perfectly even with perl
> 5.22, so it seems that there is some [subtle] difference between
> building using bitbake/OE than when built on the self-hosted
> target.  I've compared the builds and the only thing I could
> find (from the output of configure) is a difference in sizeof(off_t)
> Sadly, when I tried to adjust this in the OE build, it didn't
> make any difference, but perhaps I didn't make this change
> correctly or completely.

do you have largefile support turned on ? if you do then it might
be detecting it wrongly during configure since we cache it to a
non-largefile case

so try to add something like

EXTRA_OECONF += "${@bb.utils.contains('DISTRO_FEATURES', 'largefile',
'ac_cv_sizeof_off_t=8', '', d)}"

while building perl or the affected program and see if that helps

>
> Anyway, I'm looking for some help to solve this.  I've put
> all the relevant pieces and notes about the process at:
>   https://github.com/GaryThomas/meta-amanda.git
>
> --
> ------------------------------------------------------------
> Gary Thomas                 |  Consulting for the
> MLB Associates              |    Embedded world
> ------------------------------------------------------------
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto

Re: perl 5.22 and 32 bit targets

[Gary Thomas]

On 2016-03-23 06:36, Khem Raj wrote:
> On Tue, Mar 22, 2016 at 9:53 PM, Gary Thomas <gary@mlbassoc.com> wrote:
>> I hope this is the correct place to discuss this problem.  It
>> is all about a difference in behavior between a program built
>> using bitbake/OE (only OE-core is needed) vs building the program
>> on the target hardware itself.
>>
>> I've been struggling with this problem since perl was upgraded
>> to version 5.22.  I'm working on Amanda (Advanced Maryland Archive
>> tool) which is written primarily in perl and uses swig interfaces
>> to access native C functions.  This code works great when using
>> the previous perl (5.20.x) but fails on all 32 bit targets with
>> perl 5.22
>>
>> The interesting thing is that if I build Amanda on my target
>> directly (using SDK tools), it works perfectly even with perl
>> 5.22, so it seems that there is some [subtle] difference between
>> building using bitbake/OE than when built on the self-hosted
>> target.  I've compared the builds and the only thing I could
>> find (from the output of configure) is a difference in sizeof(off_t)
>> Sadly, when I tried to adjust this in the OE build, it didn't
>> make any difference, but perhaps I didn't make this change
>> correctly or completely.
>
> do you have largefile support turned on ? if you do then it might
> be detecting it wrongly during configure since we cache it to a
> non-largefile case
>
> so try to add something like
>
> EXTRA_OECONF += "${@bb.utils.contains('DISTRO_FEATURES', 'largefile',
> 'ac_cv_sizeof_off_t=8', '', d)}"
>
> while building perl or the affected program and see if that helps

Thanks for the idea, but that didn't help.  I also forced some CFLAGS
to match, in particular:
   -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
but this didn't make any difference either.

>>
>> Anyway, I'm looking for some help to solve this.  I've put
>> all the relevant pieces and notes about the process at:
>>    https://github.com/GaryThomas/meta-amanda.git

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: perl 5.22 and 32 bit targets

[Gary Thomas]

On 2016-03-23 09:09, Gary Thomas wrote:
> On 2016-03-23 06:36, Khem Raj wrote:
>> On Tue, Mar 22, 2016 at 9:53 PM, Gary Thomas <gary@mlbassoc.com> wrote:
>>> I hope this is the correct place to discuss this problem.  It
>>> is all about a difference in behavior between a program built
>>> using bitbake/OE (only OE-core is needed) vs building the program
>>> on the target hardware itself.
>>>
>>> I've been struggling with this problem since perl was upgraded
>>> to version 5.22.  I'm working on Amanda (Advanced Maryland Archive
>>> tool) which is written primarily in perl and uses swig interfaces
>>> to access native C functions.  This code works great when using
>>> the previous perl (5.20.x) but fails on all 32 bit targets with
>>> perl 5.22
>>>
>>> The interesting thing is that if I build Amanda on my target
>>> directly (using SDK tools), it works perfectly even with perl
>>> 5.22, so it seems that there is some [subtle] difference between
>>> building using bitbake/OE than when built on the self-hosted
>>> target.  I've compared the builds and the only thing I could
>>> find (from the output of configure) is a difference in sizeof(off_t)
>>> Sadly, when I tried to adjust this in the OE build, it didn't
>>> make any difference, but perhaps I didn't make this change
>>> correctly or completely.
>>
>> do you have largefile support turned on ? if you do then it might
>> be detecting it wrongly during configure since we cache it to a
>> non-largefile case
>>
>> so try to add something like
>>
>> EXTRA_OECONF += "${@bb.utils.contains('DISTRO_FEATURES', 'largefile',
>> 'ac_cv_sizeof_off_t=8', '', d)}"
>>
>> while building perl or the affected program and see if that helps
>
> Thanks for the idea, but that didn't help.  I also forced some CFLAGS
> to match, in particular:
>    -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
> but this didn't make any difference either.
>

On a whim I just tried a little experiment where I took the *.o files
from the perl subdirectory (where all the swig shims live) from a working
(self-hosted) build and moved them to my bitbake/OE build.  I then touched
all the *.o and *.lo files in the perl tree to force a relink. I then ran
   % bitbake amanda -C compile && bitbake core-image-base
to my surprise, amanda works!  So the culprit lies somewhere within the
swig generated glue.  I've tried comparing these files before and I didn't
find anything other than cosmetic differences (mostly comments about the
name of the file processed, etc).  I've added this subtree to "results"
in my github layer in case someone can see what might be relevant.

Any ideas what might be different and make this swig generated glue fail?
Note that the swig interface files are rebuilt as part of the build process
and both bitbake/OE and self-hosted are using the same swig version.

>>>
>>> Anyway, I'm looking for some help to solve this.  I've put
>>> all the relevant pieces and notes about the process at:
>>>    https://github.com/GaryThomas/meta-amanda.git
>


-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: perl 5.22 and 32 bit targets

[Jens Rehsack]

[-- Attachment #1: Type: text/plain, Size: 3725 bytes --]


> Am 23.03.2016 um 09:40 schrieb Gary Thomas <gary@mlbassoc.com>:
> 
> On 2016-03-23 09:09, Gary Thomas wrote:
>> On 2016-03-23 06:36, Khem Raj wrote:
>>> On Tue, Mar 22, 2016 at 9:53 PM, Gary Thomas <gary@mlbassoc.com> wrote:
>>>> I hope this is the correct place to discuss this problem.  It
>>>> is all about a difference in behavior between a program built
>>>> using bitbake/OE (only OE-core is needed) vs building the program
>>>> on the target hardware itself.
>>>> 
>>>> I've been struggling with this problem since perl was upgraded
>>>> to version 5.22.  I'm working on Amanda (Advanced Maryland Archive
>>>> tool) which is written primarily in perl and uses swig interfaces
>>>> to access native C functions.  This code works great when using
>>>> the previous perl (5.20.x) but fails on all 32 bit targets with
>>>> perl 5.22
>>>> 
>>>> The interesting thing is that if I build Amanda on my target
>>>> directly (using SDK tools), it works perfectly even with perl
>>>> 5.22, so it seems that there is some [subtle] difference between
>>>> building using bitbake/OE than when built on the self-hosted
>>>> target.  I've compared the builds and the only thing I could
>>>> find (from the output of configure) is a difference in sizeof(off_t)
>>>> Sadly, when I tried to adjust this in the OE build, it didn't
>>>> make any difference, but perhaps I didn't make this change
>>>> correctly or completely.
>>> 
>>> do you have largefile support turned on ? if you do then it might
>>> be detecting it wrongly during configure since we cache it to a
>>> non-largefile case
>>> 
>>> so try to add something like
>>> 
>>> EXTRA_OECONF += "${@bb.utils.contains('DISTRO_FEATURES', 'largefile',
>>> 'ac_cv_sizeof_off_t=8', '', d)}"
>>> 
>>> while building perl or the affected program and see if that helps
>> 
>> Thanks for the idea, but that didn't help.  I also forced some CFLAGS
>> to match, in particular:
>>   -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
>> but this didn't make any difference either.
>> 
> 
> On a whim I just tried a little experiment where I took the *.o files
> from the perl subdirectory (where all the swig shims live) from a working
> (self-hosted) build and moved them to my bitbake/OE build.  I then touched
> all the *.o and *.lo files in the perl tree to force a relink. I then ran
>  % bitbake amanda -C compile && bitbake core-image-base
> to my surprise, amanda works!  So the culprit lies somewhere within the
> swig generated glue.  I've tried comparing these files before and I didn't
> find anything other than cosmetic differences (mostly comments about the
> name of the file processed, etc).  I've added this subtree to "results"
> in my github layer in case someone can see what might be relevant.
> 
> Any ideas what might be different and make this swig generated glue fail?
> Note that the swig interface files are rebuilt as part of the build process
> and both bitbake/OE and self-hosted are using the same swig version.

I digged a bit through your layer (while my up2date scanner over meta-cpan
blocks my build chain :P) and realized that you use perl-5.20.0 as it was
in poky. A "simple" downgrade would be more reasonable ... if reason applys
here in general :)

When you fail on cross-build and succeed in target build, try to compare the
C files and includes (even swig libraries) used.

It smells more like a "wrong source" than a "perl problem" (and even when
I never would read any python thread, the same problem would likely occur
there, too ^^).

Which perl headers are used in your build? To dig down, more logs would
be reasonable ...

Cheers
--
Jens Rehsack - rehsack@gmail.com


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 859 bytes --]

Re: perl 5.22 and 32 bit targets

[Gary Thomas]

On 2016-03-23 09:57, Jens Rehsack wrote:
>
>> Am 23.03.2016 um 09:40 schrieb Gary Thomas <gary@mlbassoc.com>:
>>
>> On 2016-03-23 09:09, Gary Thomas wrote:
>>> On 2016-03-23 06:36, Khem Raj wrote:
>>>> On Tue, Mar 22, 2016 at 9:53 PM, Gary Thomas <gary@mlbassoc.com> wrote:
>>>>> I hope this is the correct place to discuss this problem.  It
>>>>> is all about a difference in behavior between a program built
>>>>> using bitbake/OE (only OE-core is needed) vs building the program
>>>>> on the target hardware itself.
>>>>>
>>>>> I've been struggling with this problem since perl was upgraded
>>>>> to version 5.22.  I'm working on Amanda (Advanced Maryland Archive
>>>>> tool) which is written primarily in perl and uses swig interfaces
>>>>> to access native C functions.  This code works great when using
>>>>> the previous perl (5.20.x) but fails on all 32 bit targets with
>>>>> perl 5.22
>>>>>
>>>>> The interesting thing is that if I build Amanda on my target
>>>>> directly (using SDK tools), it works perfectly even with perl
>>>>> 5.22, so it seems that there is some [subtle] difference between
>>>>> building using bitbake/OE than when built on the self-hosted
>>>>> target.  I've compared the builds and the only thing I could
>>>>> find (from the output of configure) is a difference in sizeof(off_t)
>>>>> Sadly, when I tried to adjust this in the OE build, it didn't
>>>>> make any difference, but perhaps I didn't make this change
>>>>> correctly or completely.
>>>>
>>>> do you have largefile support turned on ? if you do then it might
>>>> be detecting it wrongly during configure since we cache it to a
>>>> non-largefile case
>>>>
>>>> so try to add something like
>>>>
>>>> EXTRA_OECONF += "${@bb.utils.contains('DISTRO_FEATURES', 'largefile',
>>>> 'ac_cv_sizeof_off_t=8', '', d)}"
>>>>
>>>> while building perl or the affected program and see if that helps
>>>
>>> Thanks for the idea, but that didn't help.  I also forced some CFLAGS
>>> to match, in particular:
>>>    -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
>>> but this didn't make any difference either.
>>>
>>
>> On a whim I just tried a little experiment where I took the *.o files
>> from the perl subdirectory (where all the swig shims live) from a working
>> (self-hosted) build and moved them to my bitbake/OE build.  I then touched
>> all the *.o and *.lo files in the perl tree to force a relink. I then ran
>>   % bitbake amanda -C compile && bitbake core-image-base
>> to my surprise, amanda works!  So the culprit lies somewhere within the
>> swig generated glue.  I've tried comparing these files before and I didn't
>> find anything other than cosmetic differences (mostly comments about the
>> name of the file processed, etc).  I've added this subtree to "results"
>> in my github layer in case someone can see what might be relevant.
>>
>> Any ideas what might be different and make this swig generated glue fail?
>> Note that the swig interface files are rebuilt as part of the build process
>> and both bitbake/OE and self-hosted are using the same swig version.
>
> I digged a bit through your layer (while my up2date scanner over meta-cpan
> blocks my build chain :P) and realized that you use perl-5.20.0 as it was
> in poky. A "simple" downgrade would be more reasonable ... if reason applys
> here in general :)

In practice, I am doing that.  However, I want to understand why perl 5.22
breaks things and get it fixed.

>
> When you fail on cross-build and succeed in target build, try to compare the
> C files and includes (even swig libraries) used.
>
> It smells more like a "wrong source" than a "perl problem" (and even when
> I never would read any python thread, the same problem would likely occur
> there, too ^^).
>
> Which perl headers are used in your build? To dig down, more logs would
> be reasonable ...

Everything comes from the same sources, same revisions, etc, as I'm using
either a bitbake/OE build or the embedded (self-hosted) version from the
same build plus SDK tools.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: perl 5.22 and 32 bit targets

[Jens Rehsack]

[-- Attachment #1: Type: text/plain, Size: 4832 bytes --]


> Am 23.03.2016 um 10:09 schrieb Gary Thomas <gary@mlbassoc.com>:
> 
> On 2016-03-23 09:57, Jens Rehsack wrote:
>> 
>>> Am 23.03.2016 um 09:40 schrieb Gary Thomas <gary@mlbassoc.com>:
>>> 
>>> On 2016-03-23 09:09, Gary Thomas wrote:
>>>> On 2016-03-23 06:36, Khem Raj wrote:
>>>>> On Tue, Mar 22, 2016 at 9:53 PM, Gary Thomas <gary@mlbassoc.com> wrote:
>>>>>> I hope this is the correct place to discuss this problem.  It
>>>>>> is all about a difference in behavior between a program built
>>>>>> using bitbake/OE (only OE-core is needed) vs building the program
>>>>>> on the target hardware itself.
>>>>>> 
>>>>>> I've been struggling with this problem since perl was upgraded
>>>>>> to version 5.22.  I'm working on Amanda (Advanced Maryland Archive
>>>>>> tool) which is written primarily in perl and uses swig interfaces
>>>>>> to access native C functions.  This code works great when using
>>>>>> the previous perl (5.20.x) but fails on all 32 bit targets with
>>>>>> perl 5.22
>>>>>> 
>>>>>> The interesting thing is that if I build Amanda on my target
>>>>>> directly (using SDK tools), it works perfectly even with perl
>>>>>> 5.22, so it seems that there is some [subtle] difference between
>>>>>> building using bitbake/OE than when built on the self-hosted
>>>>>> target.  I've compared the builds and the only thing I could
>>>>>> find (from the output of configure) is a difference in sizeof(off_t)
>>>>>> Sadly, when I tried to adjust this in the OE build, it didn't
>>>>>> make any difference, but perhaps I didn't make this change
>>>>>> correctly or completely.
>>>>> 
>>>>> do you have largefile support turned on ? if you do then it might
>>>>> be detecting it wrongly during configure since we cache it to a
>>>>> non-largefile case
>>>>> 
>>>>> so try to add something like
>>>>> 
>>>>> EXTRA_OECONF += "${@bb.utils.contains('DISTRO_FEATURES', 'largefile',
>>>>> 'ac_cv_sizeof_off_t=8', '', d)}"
>>>>> 
>>>>> while building perl or the affected program and see if that helps
>>>> 
>>>> Thanks for the idea, but that didn't help.  I also forced some CFLAGS
>>>> to match, in particular:
>>>>   -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
>>>> but this didn't make any difference either.
>>>> 
>>> 
>>> On a whim I just tried a little experiment where I took the *.o files
>>> from the perl subdirectory (where all the swig shims live) from a working
>>> (self-hosted) build and moved them to my bitbake/OE build.  I then touched
>>> all the *.o and *.lo files in the perl tree to force a relink. I then ran
>>>  % bitbake amanda -C compile && bitbake core-image-base
>>> to my surprise, amanda works!  So the culprit lies somewhere within the
>>> swig generated glue.  I've tried comparing these files before and I didn't
>>> find anything other than cosmetic differences (mostly comments about the
>>> name of the file processed, etc).  I've added this subtree to "results"
>>> in my github layer in case someone can see what might be relevant.
>>> 
>>> Any ideas what might be different and make this swig generated glue fail?
>>> Note that the swig interface files are rebuilt as part of the build process
>>> and both bitbake/OE and self-hosted are using the same swig version.
>> 
>> I digged a bit through your layer (while my up2date scanner over meta-cpan
>> blocks my build chain :P) and realized that you use perl-5.20.0 as it was
>> in poky. A "simple" downgrade would be more reasonable ... if reason applys
>> here in general :)
> 
> In practice, I am doing that.  However, I want to understand why perl 5.22
> breaks things and get it fixed.

I did a diff between your 5.20 and poky's 5.22 and realize some fixes applied
in 5.22 regarding library path's aren't applied in your copy. Maybe swig relies
on wrong library locations and when we know, we can fix.

So it's maybe not a 5.20 vs. 5.22 problem, it's maybe a weird swig setup problem.

>> When you fail on cross-build and succeed in target build, try to compare the
>> C files and includes (even swig libraries) used.
>> 
>> It smells more like a "wrong source" than a "perl problem" (and even when
>> I never would read any python thread, the same problem would likely occur
>> there, too ^^).
>> 
>> Which perl headers are used in your build? To dig down, more logs would
>> be reasonable ...
> 
> Everything comes from the same sources, same revisions, etc, as I'm using
> either a bitbake/OE build or the embedded (self-hosted) version from the
> same build plus SDK tools.

And your SDK does not include any host tools? Did you prove the intermediate
amanda build files (eg. generated by SWIG) for relicts from wrong source?
Did you check the logs which include directories had been used?

Cheers
--
Jens Rehsack - rehsack@gmail.com


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 859 bytes --]

Re: perl 5.22 and 32 bit targets

[Jens Rehsack]

[-- Attachment #1: Type: text/plain, Size: 6334 bytes --]


> Am 23.03.2016 um 10:14 schrieb Jens Rehsack <rehsack@gmail.com>:
> 
>> 
>> Am 23.03.2016 um 10:09 schrieb Gary Thomas <gary@mlbassoc.com>:
>> 
>> On 2016-03-23 09:57, Jens Rehsack wrote:
>>> 
>>>> Am 23.03.2016 um 09:40 schrieb Gary Thomas <gary@mlbassoc.com>:
>>>> 
>>>> On 2016-03-23 09:09, Gary Thomas wrote:
>>>>> On 2016-03-23 06:36, Khem Raj wrote:
>>>>>> On Tue, Mar 22, 2016 at 9:53 PM, Gary Thomas <gary@mlbassoc.com> wrote:
>>>>>>> I hope this is the correct place to discuss this problem.  It
>>>>>>> is all about a difference in behavior between a program built
>>>>>>> using bitbake/OE (only OE-core is needed) vs building the program
>>>>>>> on the target hardware itself.
>>>>>>> 
>>>>>>> I've been struggling with this problem since perl was upgraded
>>>>>>> to version 5.22.  I'm working on Amanda (Advanced Maryland Archive
>>>>>>> tool) which is written primarily in perl and uses swig interfaces
>>>>>>> to access native C functions.  This code works great when using
>>>>>>> the previous perl (5.20.x) but fails on all 32 bit targets with
>>>>>>> perl 5.22
>>>>>>> 
>>>>>>> The interesting thing is that if I build Amanda on my target
>>>>>>> directly (using SDK tools), it works perfectly even with perl
>>>>>>> 5.22, so it seems that there is some [subtle] difference between
>>>>>>> building using bitbake/OE than when built on the self-hosted
>>>>>>> target.  I've compared the builds and the only thing I could
>>>>>>> find (from the output of configure) is a difference in sizeof(off_t)
>>>>>>> Sadly, when I tried to adjust this in the OE build, it didn't
>>>>>>> make any difference, but perhaps I didn't make this change
>>>>>>> correctly or completely.
>>>>>> 
>>>>>> do you have largefile support turned on ? if you do then it might
>>>>>> be detecting it wrongly during configure since we cache it to a
>>>>>> non-largefile case
>>>>>> 
>>>>>> so try to add something like
>>>>>> 
>>>>>> EXTRA_OECONF += "${@bb.utils.contains('DISTRO_FEATURES', 'largefile',
>>>>>> 'ac_cv_sizeof_off_t=8', '', d)}"
>>>>>> 
>>>>>> while building perl or the affected program and see if that helps
>>>>> 
>>>>> Thanks for the idea, but that didn't help.  I also forced some CFLAGS
>>>>> to match, in particular:
>>>>>  -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
>>>>> but this didn't make any difference either.
>>>>> 
>>>> 
>>>> On a whim I just tried a little experiment where I took the *.o files
>>>> from the perl subdirectory (where all the swig shims live) from a working
>>>> (self-hosted) build and moved them to my bitbake/OE build.  I then touched
>>>> all the *.o and *.lo files in the perl tree to force a relink. I then ran
>>>> % bitbake amanda -C compile && bitbake core-image-base
>>>> to my surprise, amanda works!  So the culprit lies somewhere within the
>>>> swig generated glue.  I've tried comparing these files before and I didn't
>>>> find anything other than cosmetic differences (mostly comments about the
>>>> name of the file processed, etc).  I've added this subtree to "results"
>>>> in my github layer in case someone can see what might be relevant.
>>>> 
>>>> Any ideas what might be different and make this swig generated glue fail?
>>>> Note that the swig interface files are rebuilt as part of the build process
>>>> and both bitbake/OE and self-hosted are using the same swig version.
>>> 
>>> I digged a bit through your layer (while my up2date scanner over meta-cpan
>>> blocks my build chain :P) and realized that you use perl-5.20.0 as it was
>>> in poky. A "simple" downgrade would be more reasonable ... if reason applys
>>> here in general :)
>> 
>> In practice, I am doing that.  However, I want to understand why perl 5.22
>> breaks things and get it fixed.
> 
> I did a diff between your 5.20 and poky's 5.22 and realize some fixes applied
> in 5.22 regarding library path's aren't applied in your copy. Maybe swig relies
> on wrong library locations and when we know, we can fix.
> 
> So it's maybe not a 5.20 vs. 5.22 problem, it's maybe a weird swig setup problem.
> 
>>> When you fail on cross-build and succeed in target build, try to compare the
>>> C files and includes (even swig libraries) used.
>>> 
>>> It smells more like a "wrong source" than a "perl problem" (and even when
>>> I never would read any python thread, the same problem would likely occur
>>> there, too ^^).
>>> 
>>> Which perl headers are used in your build? To dig down, more logs would
>>> be reasonable ...
>> 
>> Everything comes from the same sources, same revisions, etc, as I'm using
>> either a bitbake/OE build or the embedded (self-hosted) version from the
>> same build plus SDK tools.
> 
> And your SDK does not include any host tools? Did you prove the intermediate
> amanda build files (eg. generated by SWIG) for relicts from wrong source?
> Did you check the logs which include directories had been used?

I give it a quick shot and got:

../../arm-poky-linux-gnueabi-libtool  --tag=CC   --mode=compile arm-poky-linux-gnueabi-gcc  -march=armv7-a -marm  -mthumb-interwork -mfloat-abi=hard -mfpu=neon -mtune=cortex-a9 --sysroot=/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie -DHAVE_CONFIG_H -I. -I../../config -I../../common-src -I../../common-src -I../../xfer-src -I../../gnulib -I../../ndmp-src -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/x86_64-linux/usr/lib/perl-native/perl/5.22.1/CORE   -fno-strict-aliasing -D_GNU_SOURCE -pthread -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie/usr/include/glib-2.0 -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie/usr/lib/glib-2.0/include -DSWIG  -O2 -pipe -g -feliminate-unused-debug-types -fipa-pta -ftree-partial-pre -ftree-loop-distribution -ftree-loop-distribute-patterns -fgcse-after-reload -fgcse-sm -fgcse-las  -fno-strict-aliasing  -c -o xferwrap.lo xferwrap.c

Do you see the "-I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/x86_64-linux/usr/lib/perl-native/perl/5.22.1/CORE" ?

Any further blames of perl-5.22? :)

Cheers
--
Jens Rehsack - rehsack@gmail.com


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 859 bytes --]

Re: perl 5.22 and 32 bit targets

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 6736 bytes --]

On Wed, Mar 23, 2016 at 10:48:38AM +0100, Jens Rehsack wrote:
> 
> > Am 23.03.2016 um 10:14 schrieb Jens Rehsack <rehsack@gmail.com>:
> > 
> >> 
> >> Am 23.03.2016 um 10:09 schrieb Gary Thomas <gary@mlbassoc.com>:
> >> 
> >> On 2016-03-23 09:57, Jens Rehsack wrote:
> >>> 
> >>>> Am 23.03.2016 um 09:40 schrieb Gary Thomas <gary@mlbassoc.com>:
> >>>> 
> >>>> On 2016-03-23 09:09, Gary Thomas wrote:
> >>>>> On 2016-03-23 06:36, Khem Raj wrote:
> >>>>>> On Tue, Mar 22, 2016 at 9:53 PM, Gary Thomas <gary@mlbassoc.com> wrote:
> >>>>>>> I hope this is the correct place to discuss this problem.  It
> >>>>>>> is all about a difference in behavior between a program built
> >>>>>>> using bitbake/OE (only OE-core is needed) vs building the program
> >>>>>>> on the target hardware itself.
> >>>>>>> 
> >>>>>>> I've been struggling with this problem since perl was upgraded
> >>>>>>> to version 5.22.  I'm working on Amanda (Advanced Maryland Archive
> >>>>>>> tool) which is written primarily in perl and uses swig interfaces
> >>>>>>> to access native C functions.  This code works great when using
> >>>>>>> the previous perl (5.20.x) but fails on all 32 bit targets with
> >>>>>>> perl 5.22
> >>>>>>> 
> >>>>>>> The interesting thing is that if I build Amanda on my target
> >>>>>>> directly (using SDK tools), it works perfectly even with perl
> >>>>>>> 5.22, so it seems that there is some [subtle] difference between
> >>>>>>> building using bitbake/OE than when built on the self-hosted
> >>>>>>> target.  I've compared the builds and the only thing I could
> >>>>>>> find (from the output of configure) is a difference in sizeof(off_t)
> >>>>>>> Sadly, when I tried to adjust this in the OE build, it didn't
> >>>>>>> make any difference, but perhaps I didn't make this change
> >>>>>>> correctly or completely.
> >>>>>> 
> >>>>>> do you have largefile support turned on ? if you do then it might
> >>>>>> be detecting it wrongly during configure since we cache it to a
> >>>>>> non-largefile case
> >>>>>> 
> >>>>>> so try to add something like
> >>>>>> 
> >>>>>> EXTRA_OECONF += "${@bb.utils.contains('DISTRO_FEATURES', 'largefile',
> >>>>>> 'ac_cv_sizeof_off_t=8', '', d)}"
> >>>>>> 
> >>>>>> while building perl or the affected program and see if that helps
> >>>>> 
> >>>>> Thanks for the idea, but that didn't help.  I also forced some CFLAGS
> >>>>> to match, in particular:
> >>>>>  -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
> >>>>> but this didn't make any difference either.
> >>>>> 
> >>>> 
> >>>> On a whim I just tried a little experiment where I took the *.o files
> >>>> from the perl subdirectory (where all the swig shims live) from a working
> >>>> (self-hosted) build and moved them to my bitbake/OE build.  I then touched
> >>>> all the *.o and *.lo files in the perl tree to force a relink. I then ran
> >>>> % bitbake amanda -C compile && bitbake core-image-base
> >>>> to my surprise, amanda works!  So the culprit lies somewhere within the
> >>>> swig generated glue.  I've tried comparing these files before and I didn't
> >>>> find anything other than cosmetic differences (mostly comments about the
> >>>> name of the file processed, etc).  I've added this subtree to "results"
> >>>> in my github layer in case someone can see what might be relevant.
> >>>> 
> >>>> Any ideas what might be different and make this swig generated glue fail?
> >>>> Note that the swig interface files are rebuilt as part of the build process
> >>>> and both bitbake/OE and self-hosted are using the same swig version.
> >>> 
> >>> I digged a bit through your layer (while my up2date scanner over meta-cpan
> >>> blocks my build chain :P) and realized that you use perl-5.20.0 as it was
> >>> in poky. A "simple" downgrade would be more reasonable ... if reason applys
> >>> here in general :)
> >> 
> >> In practice, I am doing that.  However, I want to understand why perl 5.22
> >> breaks things and get it fixed.
> > 
> > I did a diff between your 5.20 and poky's 5.22 and realize some fixes applied
> > in 5.22 regarding library path's aren't applied in your copy. Maybe swig relies
> > on wrong library locations and when we know, we can fix.
> > 
> > So it's maybe not a 5.20 vs. 5.22 problem, it's maybe a weird swig setup problem.
> > 
> >>> When you fail on cross-build and succeed in target build, try to compare the
> >>> C files and includes (even swig libraries) used.
> >>> 
> >>> It smells more like a "wrong source" than a "perl problem" (and even when
> >>> I never would read any python thread, the same problem would likely occur
> >>> there, too ^^).
> >>> 
> >>> Which perl headers are used in your build? To dig down, more logs would
> >>> be reasonable ...
> >> 
> >> Everything comes from the same sources, same revisions, etc, as I'm using
> >> either a bitbake/OE build or the embedded (self-hosted) version from the
> >> same build plus SDK tools.
> > 
> > And your SDK does not include any host tools? Did you prove the intermediate
> > amanda build files (eg. generated by SWIG) for relicts from wrong source?
> > Did you check the logs which include directories had been used?
> 
> I give it a quick shot and got:
> 
> ../../arm-poky-linux-gnueabi-libtool  --tag=CC   --mode=compile arm-poky-linux-gnueabi-gcc  -march=armv7-a -marm  -mthumb-interwork -mfloat-abi=hard -mfpu=neon -mtune=cortex-a9 --sysroot=/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie -DHAVE_CONFIG_H -I. -I../../config -I../../common-src -I../../common-src -I../../xfer-src -I../../gnulib -I../../ndmp-src -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/x86_64-linux/usr/lib/perl-native/perl/5.22.1/CORE   -fno-strict-aliasing -D_GNU_SOURCE -pthread -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie/usr/include/glib-2.0 -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie/usr/lib/glib-2.0/include -DSWIG  -O2 -pipe -g -feliminate-unused-debug-types -fipa-pta -ftree-partial-pre -ftree-loop-distribution -ftree-loop-distribute-patterns -fgcse-after-reload -fgcse-sm -fgcse-las  -fno-strict-aliasing  -c -o xferwrap.lo xferwrap.c
> 
> Do you see the "-I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/x86_64-linux/usr/lib/perl-native/perl/5.22.1/CORE" ?
> 
> Any further blames of perl-5.22? :)

I blame birds in PATH (for everything)
http://edition.cnn.com/2016/03/04/us/bird-poop-nuclear-power-shutdown/

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: perl 5.22 and 32 bit targets

[Gary Thomas]

On 2016-03-23 10:48, Jens Rehsack wrote:
>
>> Am 23.03.2016 um 10:14 schrieb Jens Rehsack <rehsack@gmail.com>:
>>
>>>
>>> Am 23.03.2016 um 10:09 schrieb Gary Thomas <gary@mlbassoc.com>:
>>>
>>> On 2016-03-23 09:57, Jens Rehsack wrote:
>>>>
>>>>> Am 23.03.2016 um 09:40 schrieb Gary Thomas <gary@mlbassoc.com>:
>>>>>
>>>>> On 2016-03-23 09:09, Gary Thomas wrote:
>>>>>> On 2016-03-23 06:36, Khem Raj wrote:
>>>>>>> On Tue, Mar 22, 2016 at 9:53 PM, Gary Thomas <gary@mlbassoc.com> wrote:
>>>>>>>> I hope this is the correct place to discuss this problem.  It
>>>>>>>> is all about a difference in behavior between a program built
>>>>>>>> using bitbake/OE (only OE-core is needed) vs building the program
>>>>>>>> on the target hardware itself.
>>>>>>>>
>>>>>>>> I've been struggling with this problem since perl was upgraded
>>>>>>>> to version 5.22.  I'm working on Amanda (Advanced Maryland Archive
>>>>>>>> tool) which is written primarily in perl and uses swig interfaces
>>>>>>>> to access native C functions.  This code works great when using
>>>>>>>> the previous perl (5.20.x) but fails on all 32 bit targets with
>>>>>>>> perl 5.22
>>>>>>>>
>>>>>>>> The interesting thing is that if I build Amanda on my target
>>>>>>>> directly (using SDK tools), it works perfectly even with perl
>>>>>>>> 5.22, so it seems that there is some [subtle] difference between
>>>>>>>> building using bitbake/OE than when built on the self-hosted
>>>>>>>> target.  I've compared the builds and the only thing I could
>>>>>>>> find (from the output of configure) is a difference in sizeof(off_t)
>>>>>>>> Sadly, when I tried to adjust this in the OE build, it didn't
>>>>>>>> make any difference, but perhaps I didn't make this change
>>>>>>>> correctly or completely.
>>>>>>>
>>>>>>> do you have largefile support turned on ? if you do then it might
>>>>>>> be detecting it wrongly during configure since we cache it to a
>>>>>>> non-largefile case
>>>>>>>
>>>>>>> so try to add something like
>>>>>>>
>>>>>>> EXTRA_OECONF += "${@bb.utils.contains('DISTRO_FEATURES', 'largefile',
>>>>>>> 'ac_cv_sizeof_off_t=8', '', d)}"
>>>>>>>
>>>>>>> while building perl or the affected program and see if that helps
>>>>>>
>>>>>> Thanks for the idea, but that didn't help.  I also forced some CFLAGS
>>>>>> to match, in particular:
>>>>>>   -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
>>>>>> but this didn't make any difference either.
>>>>>>
>>>>>
>>>>> On a whim I just tried a little experiment where I took the *.o files
>>>>> from the perl subdirectory (where all the swig shims live) from a working
>>>>> (self-hosted) build and moved them to my bitbake/OE build.  I then touched
>>>>> all the *.o and *.lo files in the perl tree to force a relink. I then ran
>>>>> % bitbake amanda -C compile && bitbake core-image-base
>>>>> to my surprise, amanda works!  So the culprit lies somewhere within the
>>>>> swig generated glue.  I've tried comparing these files before and I didn't
>>>>> find anything other than cosmetic differences (mostly comments about the
>>>>> name of the file processed, etc).  I've added this subtree to "results"
>>>>> in my github layer in case someone can see what might be relevant.
>>>>>
>>>>> Any ideas what might be different and make this swig generated glue fail?
>>>>> Note that the swig interface files are rebuilt as part of the build process
>>>>> and both bitbake/OE and self-hosted are using the same swig version.
>>>>
>>>> I digged a bit through your layer (while my up2date scanner over meta-cpan
>>>> blocks my build chain :P) and realized that you use perl-5.20.0 as it was
>>>> in poky. A "simple" downgrade would be more reasonable ... if reason applys
>>>> here in general :)
>>>
>>> In practice, I am doing that.  However, I want to understand why perl 5.22
>>> breaks things and get it fixed.
>>
>> I did a diff between your 5.20 and poky's 5.22 and realize some fixes applied
>> in 5.22 regarding library path's aren't applied in your copy. Maybe swig relies
>> on wrong library locations and when we know, we can fix.
>>
>> So it's maybe not a 5.20 vs. 5.22 problem, it's maybe a weird swig setup problem.
>>
>>>> When you fail on cross-build and succeed in target build, try to compare the
>>>> C files and includes (even swig libraries) used.
>>>>
>>>> It smells more like a "wrong source" than a "perl problem" (and even when
>>>> I never would read any python thread, the same problem would likely occur
>>>> there, too ^^).
>>>>
>>>> Which perl headers are used in your build? To dig down, more logs would
>>>> be reasonable ...
>>>
>>> Everything comes from the same sources, same revisions, etc, as I'm using
>>> either a bitbake/OE build or the embedded (self-hosted) version from the
>>> same build plus SDK tools.
>>
>> And your SDK does not include any host tools? Did you prove the intermediate
>> amanda build files (eg. generated by SWIG) for relicts from wrong source?
>> Did you check the logs which include directories had been used?
>
> I give it a quick shot and got:
>
> ../../arm-poky-linux-gnueabi-libtool  --tag=CC   --mode=compile arm-poky-linux-gnueabi-gcc  -march=armv7-a -marm  -mthumb-interwork -mfloat-abi=hard -mfpu=neon -mtune=cortex-a9 --sysroot=/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie -DHAVE_CONFIG_H -I. -I../../config -I../../common-src -I../../common-src -I../../xfer-src -I../../gnulib -I../../ndmp-src -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/x86_64-linux/usr/lib/perl-native/perl/5.22.1/CORE   -fno-strict-aliasing -D_GNU_SOURCE -pthread -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie/usr/include/glib-2.0 -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie/usr/lib/glib-2.0/include -DSWIG  -O2 -pipe -g -feliminate-unused-debug-types -fipa-pta -ftree-partial-pre -ftree-loop-distribution -ftree-loop-distribute-patterns -fgcse-afte
 r
-reload -fgcse-sm -fgcse-las  -fno-strict-aliasing  -c -o xferwrap.lo xferwrap.c
>
> Do you see the "-I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/x86_64-linux/usr/lib/perl-native/perl/5.22.1/CORE" ?
>
> Any further blames of perl-5.22? :)

I'm not sure what you are getting at here.  Indeed it is supposed to be
using perl-5.22.1 for everything.  The version 5.22.0 in meta-amanda layer
is just for testing and should not be used at the moment.

All of my comparisons and testing (see instructions in the README) are
for perl-5.22.1 only and in fact I've only been using the bits from bitbake/OE
and not building that any more (I found that it wasn't necessary - one only
needs to build amanda on the target)

Sorry for the confusion.


-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: perl 5.22 and 32 bit targets

[Jens Rehsack]

[-- Attachment #1: Type: text/plain, Size: 8202 bytes --]


> Am 23.03.2016 um 12:05 schrieb Gary Thomas <gary@mlbassoc.com>:
> 
> On 2016-03-23 10:48, Jens Rehsack wrote:
>> 
>>> Am 23.03.2016 um 10:14 schrieb Jens Rehsack <rehsack@gmail.com>:
>>> 
>>>> 
>>>> Am 23.03.2016 um 10:09 schrieb Gary Thomas <gary@mlbassoc.com>:
>>>> 
>>>> On 2016-03-23 09:57, Jens Rehsack wrote:
>>>>> 
>>>>>> Am 23.03.2016 um 09:40 schrieb Gary Thomas <gary@mlbassoc.com>:
>>>>>> 
>>>>>> On 2016-03-23 09:09, Gary Thomas wrote:
>>>>>>> On 2016-03-23 06:36, Khem Raj wrote:
>>>>>>>> On Tue, Mar 22, 2016 at 9:53 PM, Gary Thomas <gary@mlbassoc.com> wrote:
>>>>>>>>> I hope this is the correct place to discuss this problem.  It
>>>>>>>>> is all about a difference in behavior between a program built
>>>>>>>>> using bitbake/OE (only OE-core is needed) vs building the program
>>>>>>>>> on the target hardware itself.
>>>>>>>>> 
>>>>>>>>> I've been struggling with this problem since perl was upgraded
>>>>>>>>> to version 5.22.  I'm working on Amanda (Advanced Maryland Archive
>>>>>>>>> tool) which is written primarily in perl and uses swig interfaces
>>>>>>>>> to access native C functions.  This code works great when using
>>>>>>>>> the previous perl (5.20.x) but fails on all 32 bit targets with
>>>>>>>>> perl 5.22
>>>>>>>>> 
>>>>>>>>> The interesting thing is that if I build Amanda on my target
>>>>>>>>> directly (using SDK tools), it works perfectly even with perl
>>>>>>>>> 5.22, so it seems that there is some [subtle] difference between
>>>>>>>>> building using bitbake/OE than when built on the self-hosted
>>>>>>>>> target.  I've compared the builds and the only thing I could
>>>>>>>>> find (from the output of configure) is a difference in sizeof(off_t)
>>>>>>>>> Sadly, when I tried to adjust this in the OE build, it didn't
>>>>>>>>> make any difference, but perhaps I didn't make this change
>>>>>>>>> correctly or completely.
>>>>>>>> 
>>>>>>>> do you have largefile support turned on ? if you do then it might
>>>>>>>> be detecting it wrongly during configure since we cache it to a
>>>>>>>> non-largefile case
>>>>>>>> 
>>>>>>>> so try to add something like
>>>>>>>> 
>>>>>>>> EXTRA_OECONF += "${@bb.utils.contains('DISTRO_FEATURES', 'largefile',
>>>>>>>> 'ac_cv_sizeof_off_t=8', '', d)}"
>>>>>>>> 
>>>>>>>> while building perl or the affected program and see if that helps
>>>>>>> 
>>>>>>> Thanks for the idea, but that didn't help.  I also forced some CFLAGS
>>>>>>> to match, in particular:
>>>>>>>  -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
>>>>>>> but this didn't make any difference either.
>>>>>>> 
>>>>>> 
>>>>>> On a whim I just tried a little experiment where I took the *.o files
>>>>>> from the perl subdirectory (where all the swig shims live) from a working
>>>>>> (self-hosted) build and moved them to my bitbake/OE build.  I then touched
>>>>>> all the *.o and *.lo files in the perl tree to force a relink. I then ran
>>>>>> % bitbake amanda -C compile && bitbake core-image-base
>>>>>> to my surprise, amanda works!  So the culprit lies somewhere within the
>>>>>> swig generated glue.  I've tried comparing these files before and I didn't
>>>>>> find anything other than cosmetic differences (mostly comments about the
>>>>>> name of the file processed, etc).  I've added this subtree to "results"
>>>>>> in my github layer in case someone can see what might be relevant.
>>>>>> 
>>>>>> Any ideas what might be different and make this swig generated glue fail?
>>>>>> Note that the swig interface files are rebuilt as part of the build process
>>>>>> and both bitbake/OE and self-hosted are using the same swig version.
>>>>> 
>>>>> I digged a bit through your layer (while my up2date scanner over meta-cpan
>>>>> blocks my build chain :P) and realized that you use perl-5.20.0 as it was
>>>>> in poky. A "simple" downgrade would be more reasonable ... if reason applys
>>>>> here in general :)
>>>> 
>>>> In practice, I am doing that.  However, I want to understand why perl 5.22
>>>> breaks things and get it fixed.
>>> 
>>> I did a diff between your 5.20 and poky's 5.22 and realize some fixes applied
>>> in 5.22 regarding library path's aren't applied in your copy. Maybe swig relies
>>> on wrong library locations and when we know, we can fix.
>>> 
>>> So it's maybe not a 5.20 vs. 5.22 problem, it's maybe a weird swig setup problem.
>>> 
>>>>> When you fail on cross-build and succeed in target build, try to compare the
>>>>> C files and includes (even swig libraries) used.
>>>>> 
>>>>> It smells more like a "wrong source" than a "perl problem" (and even when
>>>>> I never would read any python thread, the same problem would likely occur
>>>>> there, too ^^).
>>>>> 
>>>>> Which perl headers are used in your build? To dig down, more logs would
>>>>> be reasonable ...
>>>> 
>>>> Everything comes from the same sources, same revisions, etc, as I'm using
>>>> either a bitbake/OE build or the embedded (self-hosted) version from the
>>>> same build plus SDK tools.
>>> 
>>> And your SDK does not include any host tools? Did you prove the intermediate
>>> amanda build files (eg. generated by SWIG) for relicts from wrong source?
>>> Did you check the logs which include directories had been used?
>> 
>> I give it a quick shot and got:
>> 
>> ../../arm-poky-linux-gnueabi-libtool  --tag=CC   --mode=compile arm-poky-linux-gnueabi-gcc  -march=armv7-a -marm  -mthumb-interwork -mfloat-abi=hard -mfpu=neon -mtune=cortex-a9 --sysroot=/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie -DHAVE_CONFIG_H -I. -I../../config -I../../common-src -I../../common-src -I../../xfer-src -I../../gnulib -I../../ndmp-src -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/x86_64-linux/usr/lib/perl-native/perl/5.22.1/CORE   -fno-strict-aliasing -D_GNU_SOURCE -pthread -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie/usr/include/glib-2.0 -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie/usr/lib/glib-2.0/include -DSWIG  -O2 -pipe -g -feliminate-unused-debug-types -fipa-pta -ftree-partial-pre -ftree-loop-distribution -ftree-loop-distribute-patterns -fgcse-afte
> r
> -reload -fgcse-sm -fgcse-las  -fno-strict-aliasing  -c -o xferwrap.lo xferwrap.c
>> 
>> Do you see the "-I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/x86_64-linux/usr/lib/perl-native/perl/5.22.1/CORE" ?
>> 
>> Any further blames of perl-5.22? :)
> 
> I'm not sure what you are getting at here.  Indeed it is supposed to be
> using perl-5.22.1 for everything.  The version 5.22.0 in meta-amanda layer
> is just for testing and should not be used at the moment.

Please compare word wise ...

-I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/x86_64-linux/usr/lib/perl-native/perl/5.22.1/CORE
-I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie/usr/include/glib-2.0

Probably this helps getting my point:

$ ls -l tmp/work
total 149
drwxrwxr-x  54 sno sno  54 Mar 22 23:49 all-poky-linux
drwxrwxr-x 269 sno sno 269 Mar 22 23:49 arm926ejse-poky-linux-gnueabi
drwxrwxr-x   3 sno sno   3 Mar 22 23:48 bohr-nand-poky-linux-gnueabi
drwxrwxr-x  19 sno sno  19 Mar 22 23:49 bohr-poky-linux-gnueabi
drwxrwxr-x   9 sno sno   9 Mar 22 21:38 cortexa9hf-vfp-neon-mx6qdl-poky-linux-gnueabi
drwxrwxr-x 326 sno sno 326 Mar 23 11:28 cortexa9hf-vfp-neon-poky-linux-gnueabi
drwxrwxr-x   3 sno sno   3 Mar 22 21:37 curie-emmc-poky-linux-gnueabi
drwxrwxr-x  29 sno sno  29 Mar 22 21:50 curie-poky-linux-gnueabi
drwxrwxr-x 196 sno sno 196 Mar 23 11:28 x86_64-linux

> All of my comparisons and testing (see instructions in the README) are
> for perl-5.22.1 only and in fact I've only been using the bits from bitbake/OE
> and not building that any more (I found that it wasn't necessary - one only
> needs to build amanda on the target)
> 
> Sorry for the confusion.

Cheers
--
Jens Rehsack - rehsack@gmail.com


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 859 bytes --]

Re: perl 5.22 and 32 bit targets

[Gary Thomas]

On 2016-03-23 12:43, Jens Rehsack wrote:
>
>> Am 23.03.2016 um 12:05 schrieb Gary Thomas <gary@mlbassoc.com>:
>>
>> On 2016-03-23 10:48, Jens Rehsack wrote:
>>>
>>>> Am 23.03.2016 um 10:14 schrieb Jens Rehsack <rehsack@gmail.com>:
>>>>
>>>>>
>>>>> Am 23.03.2016 um 10:09 schrieb Gary Thomas <gary@mlbassoc.com>:
>>>>>
>>>>> On 2016-03-23 09:57, Jens Rehsack wrote:
>>>>>>
>>>>>>> Am 23.03.2016 um 09:40 schrieb Gary Thomas <gary@mlbassoc.com>:
>>>>>>>
>>>>>>> On 2016-03-23 09:09, Gary Thomas wrote:
>>>>>>>> On 2016-03-23 06:36, Khem Raj wrote:
>>>>>>>>> On Tue, Mar 22, 2016 at 9:53 PM, Gary Thomas <gary@mlbassoc.com> wrote:
>>>>>>>>>> I hope this is the correct place to discuss this problem.  It
>>>>>>>>>> is all about a difference in behavior between a program built
>>>>>>>>>> using bitbake/OE (only OE-core is needed) vs building the program
>>>>>>>>>> on the target hardware itself.
>>>>>>>>>>
>>>>>>>>>> I've been struggling with this problem since perl was upgraded
>>>>>>>>>> to version 5.22.  I'm working on Amanda (Advanced Maryland Archive
>>>>>>>>>> tool) which is written primarily in perl and uses swig interfaces
>>>>>>>>>> to access native C functions.  This code works great when using
>>>>>>>>>> the previous perl (5.20.x) but fails on all 32 bit targets with
>>>>>>>>>> perl 5.22
>>>>>>>>>>
>>>>>>>>>> The interesting thing is that if I build Amanda on my target
>>>>>>>>>> directly (using SDK tools), it works perfectly even with perl
>>>>>>>>>> 5.22, so it seems that there is some [subtle] difference between
>>>>>>>>>> building using bitbake/OE than when built on the self-hosted
>>>>>>>>>> target.  I've compared the builds and the only thing I could
>>>>>>>>>> find (from the output of configure) is a difference in sizeof(off_t)
>>>>>>>>>> Sadly, when I tried to adjust this in the OE build, it didn't
>>>>>>>>>> make any difference, but perhaps I didn't make this change
>>>>>>>>>> correctly or completely.
>>>>>>>>>
>>>>>>>>> do you have largefile support turned on ? if you do then it might
>>>>>>>>> be detecting it wrongly during configure since we cache it to a
>>>>>>>>> non-largefile case
>>>>>>>>>
>>>>>>>>> so try to add something like
>>>>>>>>>
>>>>>>>>> EXTRA_OECONF += "${@bb.utils.contains('DISTRO_FEATURES', 'largefile',
>>>>>>>>> 'ac_cv_sizeof_off_t=8', '', d)}"
>>>>>>>>>
>>>>>>>>> while building perl or the affected program and see if that helps
>>>>>>>>
>>>>>>>> Thanks for the idea, but that didn't help.  I also forced some CFLAGS
>>>>>>>> to match, in particular:
>>>>>>>>   -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
>>>>>>>> but this didn't make any difference either.
>>>>>>>>
>>>>>>>
>>>>>>> On a whim I just tried a little experiment where I took the *.o files
>>>>>>> from the perl subdirectory (where all the swig shims live) from a working
>>>>>>> (self-hosted) build and moved them to my bitbake/OE build.  I then touched
>>>>>>> all the *.o and *.lo files in the perl tree to force a relink. I then ran
>>>>>>> % bitbake amanda -C compile && bitbake core-image-base
>>>>>>> to my surprise, amanda works!  So the culprit lies somewhere within the
>>>>>>> swig generated glue.  I've tried comparing these files before and I didn't
>>>>>>> find anything other than cosmetic differences (mostly comments about the
>>>>>>> name of the file processed, etc).  I've added this subtree to "results"
>>>>>>> in my github layer in case someone can see what might be relevant.
>>>>>>>
>>>>>>> Any ideas what might be different and make this swig generated glue fail?
>>>>>>> Note that the swig interface files are rebuilt as part of the build process
>>>>>>> and both bitbake/OE and self-hosted are using the same swig version.
>>>>>>
>>>>>> I digged a bit through your layer (while my up2date scanner over meta-cpan
>>>>>> blocks my build chain :P) and realized that you use perl-5.20.0 as it was
>>>>>> in poky. A "simple" downgrade would be more reasonable ... if reason applys
>>>>>> here in general :)
>>>>>
>>>>> In practice, I am doing that.  However, I want to understand why perl 5.22
>>>>> breaks things and get it fixed.
>>>>
>>>> I did a diff between your 5.20 and poky's 5.22 and realize some fixes applied
>>>> in 5.22 regarding library path's aren't applied in your copy. Maybe swig relies
>>>> on wrong library locations and when we know, we can fix.
>>>>
>>>> So it's maybe not a 5.20 vs. 5.22 problem, it's maybe a weird swig setup problem.
>>>>
>>>>>> When you fail on cross-build and succeed in target build, try to compare the
>>>>>> C files and includes (even swig libraries) used.
>>>>>>
>>>>>> It smells more like a "wrong source" than a "perl problem" (and even when
>>>>>> I never would read any python thread, the same problem would likely occur
>>>>>> there, too ^^).
>>>>>>
>>>>>> Which perl headers are used in your build? To dig down, more logs would
>>>>>> be reasonable ...
>>>>>
>>>>> Everything comes from the same sources, same revisions, etc, as I'm using
>>>>> either a bitbake/OE build or the embedded (self-hosted) version from the
>>>>> same build plus SDK tools.
>>>>
>>>> And your SDK does not include any host tools? Did you prove the intermediate
>>>> amanda build files (eg. generated by SWIG) for relicts from wrong source?
>>>> Did you check the logs which include directories had been used?
>>>
>>> I give it a quick shot and got:
>>>
>>> ../../arm-poky-linux-gnueabi-libtool  --tag=CC   --mode=compile arm-poky-linux-gnueabi-gcc  -march=armv7-a -marm  -mthumb-interwork -mfloat-abi=hard -mfpu=neon -mtune=cortex-a9 --sysroot=/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie -DHAVE_CONFIG_H -I. -I../../config -I../../common-src -I../../common-src -I../../xfer-src -I../../gnulib -I../../ndmp-src -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/x86_64-linux/usr/lib/perl-native/perl/5.22.1/CORE   -fno-strict-aliasing -D_GNU_SOURCE -pthread -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie/usr/include/glib-2.0 -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie/usr/lib/glib-2.0/include -DSWIG  -O2 -pipe -g -feliminate-unused-debug-types -fipa-pta -ftree-partial-pre -ftree-loop-distribution -ftree-loop-distribute-patterns -fgcse-af
 t
e
>> r
>> -reload -fgcse-sm -fgcse-las  -fno-strict-aliasing  -c -o xferwrap.lo xferwrap.c
>>>
>>> Do you see the "-I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/x86_64-linux/usr/lib/perl-native/perl/5.22.1/CORE" ?
>>>
>>> Any further blames of perl-5.22? :)
>>
>> I'm not sure what you are getting at here.  Indeed it is supposed to be
>> using perl-5.22.1 for everything.  The version 5.22.0 in meta-amanda layer
>> is just for testing and should not be used at the moment.
>
> Please compare word wise ...
>
> -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/x86_64-linux/usr/lib/perl-native/perl/5.22.1/CORE
> -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie/usr/include/glib-2.0
>
> Probably this helps getting my point:
>
> $ ls -l tmp/work
> total 149
> drwxrwxr-x  54 sno sno  54 Mar 22 23:49 all-poky-linux
> drwxrwxr-x 269 sno sno 269 Mar 22 23:49 arm926ejse-poky-linux-gnueabi
> drwxrwxr-x   3 sno sno   3 Mar 22 23:48 bohr-nand-poky-linux-gnueabi
> drwxrwxr-x  19 sno sno  19 Mar 22 23:49 bohr-poky-linux-gnueabi
> drwxrwxr-x   9 sno sno   9 Mar 22 21:38 cortexa9hf-vfp-neon-mx6qdl-poky-linux-gnueabi
> drwxrwxr-x 326 sno sno 326 Mar 23 11:28 cortexa9hf-vfp-neon-poky-linux-gnueabi
> drwxrwxr-x   3 sno sno   3 Mar 22 21:37 curie-emmc-poky-linux-gnueabi
> drwxrwxr-x  29 sno sno  29 Mar 22 21:50 curie-poky-linux-gnueabi
> drwxrwxr-x 196 sno sno 196 Mar 23 11:28 x86_64-linux
>
>> All of my comparisons and testing (see instructions in the README) are
>> for perl-5.22.1 only and in fact I've only been using the bits from bitbake/OE
>> and not building that any more (I found that it wasn't necessary - one only
>> needs to build amanda on the target)
>>
>> Sorry for the confusion.
>

Huzzah!  This seems to be on the right track.  I adjusted the PERL_INC
value in perl/Makefile (the *.h files used by the swig interfaces) to
point to the target ones and not the perl-native ones and now it works :-)

I'm not sure what's different between the two sets of include files, and
more interestingly why it worked with previous versions of perl, but this
does seem to be the culprit.

Now I just need to figure out how to convince the 'configure' process to
do it correctly.

Thanks for the help!

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: perl 5.22 and 32 bit targets

[Jens Rehsack]

[-- Attachment #1: Type: text/plain, Size: 9380 bytes --]


> Am 23.03.2016 um 13:07 schrieb Gary Thomas <gary@mlbassoc.com>:
> 
> On 2016-03-23 12:43, Jens Rehsack wrote:
>> 
>>> Am 23.03.2016 um 12:05 schrieb Gary Thomas <gary@mlbassoc.com>:
>>> 
>>> On 2016-03-23 10:48, Jens Rehsack wrote:
>>>> 
>>>>> Am 23.03.2016 um 10:14 schrieb Jens Rehsack <rehsack@gmail.com>:
>>>>> 
>>>>>> 
>>>>>> Am 23.03.2016 um 10:09 schrieb Gary Thomas <gary@mlbassoc.com>:
>>>>>> 
>>>>>> On 2016-03-23 09:57, Jens Rehsack wrote:
>>>>>>> 
>>>>>>>> Am 23.03.2016 um 09:40 schrieb Gary Thomas <gary@mlbassoc.com>:
>>>>>>>> 
>>>>>>>> On 2016-03-23 09:09, Gary Thomas wrote:
>>>>>>>>> On 2016-03-23 06:36, Khem Raj wrote:
>>>>>>>>>> On Tue, Mar 22, 2016 at 9:53 PM, Gary Thomas <gary@mlbassoc.com> wrote:
>>>>>>>>>>> I hope this is the correct place to discuss this problem.  It
>>>>>>>>>>> is all about a difference in behavior between a program built
>>>>>>>>>>> using bitbake/OE (only OE-core is needed) vs building the program
>>>>>>>>>>> on the target hardware itself.
>>>>>>>>>>> 
>>>>>>>>>>> I've been struggling with this problem since perl was upgraded
>>>>>>>>>>> to version 5.22.  I'm working on Amanda (Advanced Maryland Archive
>>>>>>>>>>> tool) which is written primarily in perl and uses swig interfaces
>>>>>>>>>>> to access native C functions.  This code works great when using
>>>>>>>>>>> the previous perl (5.20.x) but fails on all 32 bit targets with
>>>>>>>>>>> perl 5.22
>>>>>>>>>>> 
>>>>>>>>>>> The interesting thing is that if I build Amanda on my target
>>>>>>>>>>> directly (using SDK tools), it works perfectly even with perl
>>>>>>>>>>> 5.22, so it seems that there is some [subtle] difference between
>>>>>>>>>>> building using bitbake/OE than when built on the self-hosted
>>>>>>>>>>> target.  I've compared the builds and the only thing I could
>>>>>>>>>>> find (from the output of configure) is a difference in sizeof(off_t)
>>>>>>>>>>> Sadly, when I tried to adjust this in the OE build, it didn't
>>>>>>>>>>> make any difference, but perhaps I didn't make this change
>>>>>>>>>>> correctly or completely.
>>>>>>>>>> 
>>>>>>>>>> do you have largefile support turned on ? if you do then it might
>>>>>>>>>> be detecting it wrongly during configure since we cache it to a
>>>>>>>>>> non-largefile case
>>>>>>>>>> 
>>>>>>>>>> so try to add something like
>>>>>>>>>> 
>>>>>>>>>> EXTRA_OECONF += "${@bb.utils.contains('DISTRO_FEATURES', 'largefile',
>>>>>>>>>> 'ac_cv_sizeof_off_t=8', '', d)}"
>>>>>>>>>> 
>>>>>>>>>> while building perl or the affected program and see if that helps
>>>>>>>>> 
>>>>>>>>> Thanks for the idea, but that didn't help.  I also forced some CFLAGS
>>>>>>>>> to match, in particular:
>>>>>>>>>  -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
>>>>>>>>> but this didn't make any difference either.
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> On a whim I just tried a little experiment where I took the *.o files
>>>>>>>> from the perl subdirectory (where all the swig shims live) from a working
>>>>>>>> (self-hosted) build and moved them to my bitbake/OE build.  I then touched
>>>>>>>> all the *.o and *.lo files in the perl tree to force a relink. I then ran
>>>>>>>> % bitbake amanda -C compile && bitbake core-image-base
>>>>>>>> to my surprise, amanda works!  So the culprit lies somewhere within the
>>>>>>>> swig generated glue.  I've tried comparing these files before and I didn't
>>>>>>>> find anything other than cosmetic differences (mostly comments about the
>>>>>>>> name of the file processed, etc).  I've added this subtree to "results"
>>>>>>>> in my github layer in case someone can see what might be relevant.
>>>>>>>> 
>>>>>>>> Any ideas what might be different and make this swig generated glue fail?
>>>>>>>> Note that the swig interface files are rebuilt as part of the build process
>>>>>>>> and both bitbake/OE and self-hosted are using the same swig version.
>>>>>>> 
>>>>>>> I digged a bit through your layer (while my up2date scanner over meta-cpan
>>>>>>> blocks my build chain :P) and realized that you use perl-5.20.0 as it was
>>>>>>> in poky. A "simple" downgrade would be more reasonable ... if reason applys
>>>>>>> here in general :)
>>>>>> 
>>>>>> In practice, I am doing that.  However, I want to understand why perl 5.22
>>>>>> breaks things and get it fixed.
>>>>> 
>>>>> I did a diff between your 5.20 and poky's 5.22 and realize some fixes applied
>>>>> in 5.22 regarding library path's aren't applied in your copy. Maybe swig relies
>>>>> on wrong library locations and when we know, we can fix.
>>>>> 
>>>>> So it's maybe not a 5.20 vs. 5.22 problem, it's maybe a weird swig setup problem.
>>>>> 
>>>>>>> When you fail on cross-build and succeed in target build, try to compare the
>>>>>>> C files and includes (even swig libraries) used.
>>>>>>> 
>>>>>>> It smells more like a "wrong source" than a "perl problem" (and even when
>>>>>>> I never would read any python thread, the same problem would likely occur
>>>>>>> there, too ^^).
>>>>>>> 
>>>>>>> Which perl headers are used in your build? To dig down, more logs would
>>>>>>> be reasonable ...
>>>>>> 
>>>>>> Everything comes from the same sources, same revisions, etc, as I'm using
>>>>>> either a bitbake/OE build or the embedded (self-hosted) version from the
>>>>>> same build plus SDK tools.
>>>>> 
>>>>> And your SDK does not include any host tools? Did you prove the intermediate
>>>>> amanda build files (eg. generated by SWIG) for relicts from wrong source?
>>>>> Did you check the logs which include directories had been used?
>>>> 
>>>> I give it a quick shot and got:
>>>> 
>>>> ../../arm-poky-linux-gnueabi-libtool  --tag=CC   --mode=compile arm-poky-linux-gnueabi-gcc  -march=armv7-a -marm  -mthumb-interwork -mfloat-abi=hard -mfpu=neon -mtune=cortex-a9 --sysroot=/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie -DHAVE_CONFIG_H -I. -I../../config -I../../common-src -I../../common-src -I../../xfer-src -I../../gnulib -I../../ndmp-src -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/x86_64-linux/usr/lib/perl-native/perl/5.22.1/CORE   -fno-strict-aliasing -D_GNU_SOURCE -pthread -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie/usr/include/glib-2.0 -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie/usr/lib/glib-2.0/include -DSWIG  -O2 -pipe -g -feliminate-unused-debug-types -fipa-pta -ftree-partial-pre -ftree-loop-distribution -ftree-loop-distribute-patterns -fgcse-af
> t
> e
>>> r
>>> -reload -fgcse-sm -fgcse-las  -fno-strict-aliasing  -c -o xferwrap.lo xferwrap.c
>>>> 
>>>> Do you see the "-I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/x86_64-linux/usr/lib/perl-native/perl/5.22.1/CORE" ?
>>>> 
>>>> Any further blames of perl-5.22? :)
>>> 
>>> I'm not sure what you are getting at here.  Indeed it is supposed to be
>>> using perl-5.22.1 for everything.  The version 5.22.0 in meta-amanda layer
>>> is just for testing and should not be used at the moment.
>> 
>> Please compare word wise ...
>> 
>> -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/x86_64-linux/usr/lib/perl-native/perl/5.22.1/CORE
>> -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie/usr/include/glib-2.0
>> 
>> Probably this helps getting my point:
>> 
>> $ ls -l tmp/work
>> total 149
>> drwxrwxr-x  54 sno sno  54 Mar 22 23:49 all-poky-linux
>> drwxrwxr-x 269 sno sno 269 Mar 22 23:49 arm926ejse-poky-linux-gnueabi
>> drwxrwxr-x   3 sno sno   3 Mar 22 23:48 bohr-nand-poky-linux-gnueabi
>> drwxrwxr-x  19 sno sno  19 Mar 22 23:49 bohr-poky-linux-gnueabi
>> drwxrwxr-x   9 sno sno   9 Mar 22 21:38 cortexa9hf-vfp-neon-mx6qdl-poky-linux-gnueabi
>> drwxrwxr-x 326 sno sno 326 Mar 23 11:28 cortexa9hf-vfp-neon-poky-linux-gnueabi
>> drwxrwxr-x   3 sno sno   3 Mar 22 21:37 curie-emmc-poky-linux-gnueabi
>> drwxrwxr-x  29 sno sno  29 Mar 22 21:50 curie-poky-linux-gnueabi
>> drwxrwxr-x 196 sno sno 196 Mar 23 11:28 x86_64-linux
>> 
>>> All of my comparisons and testing (see instructions in the README) are
>>> for perl-5.22.1 only and in fact I've only been using the bits from bitbake/OE
>>> and not building that any more (I found that it wasn't necessary - one only
>>> needs to build amanda on the target)
>>> 
>>> Sorry for the confusion.
>> 
> 
> Huzzah!  This seems to be on the right track.  I adjusted the PERL_INC
> value in perl/Makefile (the *.h files used by the swig interfaces) to
> point to the target ones and not the perl-native ones and now it works :-)
> 
> I'm not sure what's different between the two sets of include files, and
> more interestingly why it worked with previous versions of perl, but this
> does seem to be the culprit.
> 
> Now I just need to figure out how to convince the 'configure' process to
> do it correctly.

IIRC, rrdtool is a good example to do it right. collectd had the same problem
as amanda had - wrong perl used ...
Maybe vice versa - I'm getting old and forget to remember :D

> Thanks for the help!

You're welcome.

Cheers
--
Jens Rehsack - rehsack@gmail.com


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 859 bytes --]

Re: perl 5.22 and 32 bit targets

[Gary Thomas]

On 2016-03-23 13:17, Jens Rehsack wrote:
>
>> Am 23.03.2016 um 13:07 schrieb Gary Thomas <gary@mlbassoc.com>:
>>
>> On 2016-03-23 12:43, Jens Rehsack wrote:
>>>
>>>> Am 23.03.2016 um 12:05 schrieb Gary Thomas <gary@mlbassoc.com>:
>>>>
>>>> On 2016-03-23 10:48, Jens Rehsack wrote:
>>>>>
>>>>>> Am 23.03.2016 um 10:14 schrieb Jens Rehsack <rehsack@gmail.com>:
>>>>>>
>>>>>>>
>>>>>>> Am 23.03.2016 um 10:09 schrieb Gary Thomas <gary@mlbassoc.com>:
>>>>>>>
>>>>>>> On 2016-03-23 09:57, Jens Rehsack wrote:
>>>>>>>>
>>>>>>>>> Am 23.03.2016 um 09:40 schrieb Gary Thomas <gary@mlbassoc.com>:
>>>>>>>>>
>>>>>>>>> On 2016-03-23 09:09, Gary Thomas wrote:
>>>>>>>>>> On 2016-03-23 06:36, Khem Raj wrote:
>>>>>>>>>>> On Tue, Mar 22, 2016 at 9:53 PM, Gary Thomas <gary@mlbassoc.com> wrote:
>>>>>>>>>>>> I hope this is the correct place to discuss this problem.  It
>>>>>>>>>>>> is all about a difference in behavior between a program built
>>>>>>>>>>>> using bitbake/OE (only OE-core is needed) vs building the program
>>>>>>>>>>>> on the target hardware itself.
>>>>>>>>>>>>
>>>>>>>>>>>> I've been struggling with this problem since perl was upgraded
>>>>>>>>>>>> to version 5.22.  I'm working on Amanda (Advanced Maryland Archive
>>>>>>>>>>>> tool) which is written primarily in perl and uses swig interfaces
>>>>>>>>>>>> to access native C functions.  This code works great when using
>>>>>>>>>>>> the previous perl (5.20.x) but fails on all 32 bit targets with
>>>>>>>>>>>> perl 5.22
>>>>>>>>>>>>
>>>>>>>>>>>> The interesting thing is that if I build Amanda on my target
>>>>>>>>>>>> directly (using SDK tools), it works perfectly even with perl
>>>>>>>>>>>> 5.22, so it seems that there is some [subtle] difference between
>>>>>>>>>>>> building using bitbake/OE than when built on the self-hosted
>>>>>>>>>>>> target.  I've compared the builds and the only thing I could
>>>>>>>>>>>> find (from the output of configure) is a difference in sizeof(off_t)
>>>>>>>>>>>> Sadly, when I tried to adjust this in the OE build, it didn't
>>>>>>>>>>>> make any difference, but perhaps I didn't make this change
>>>>>>>>>>>> correctly or completely.
>>>>>>>>>>>
>>>>>>>>>>> do you have largefile support turned on ? if you do then it might
>>>>>>>>>>> be detecting it wrongly during configure since we cache it to a
>>>>>>>>>>> non-largefile case
>>>>>>>>>>>
>>>>>>>>>>> so try to add something like
>>>>>>>>>>>
>>>>>>>>>>> EXTRA_OECONF += "${@bb.utils.contains('DISTRO_FEATURES', 'largefile',
>>>>>>>>>>> 'ac_cv_sizeof_off_t=8', '', d)}"
>>>>>>>>>>>
>>>>>>>>>>> while building perl or the affected program and see if that helps
>>>>>>>>>>
>>>>>>>>>> Thanks for the idea, but that didn't help.  I also forced some CFLAGS
>>>>>>>>>> to match, in particular:
>>>>>>>>>>   -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
>>>>>>>>>> but this didn't make any difference either.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On a whim I just tried a little experiment where I took the *.o files
>>>>>>>>> from the perl subdirectory (where all the swig shims live) from a working
>>>>>>>>> (self-hosted) build and moved them to my bitbake/OE build.  I then touched
>>>>>>>>> all the *.o and *.lo files in the perl tree to force a relink. I then ran
>>>>>>>>> % bitbake amanda -C compile && bitbake core-image-base
>>>>>>>>> to my surprise, amanda works!  So the culprit lies somewhere within the
>>>>>>>>> swig generated glue.  I've tried comparing these files before and I didn't
>>>>>>>>> find anything other than cosmetic differences (mostly comments about the
>>>>>>>>> name of the file processed, etc).  I've added this subtree to "results"
>>>>>>>>> in my github layer in case someone can see what might be relevant.
>>>>>>>>>
>>>>>>>>> Any ideas what might be different and make this swig generated glue fail?
>>>>>>>>> Note that the swig interface files are rebuilt as part of the build process
>>>>>>>>> and both bitbake/OE and self-hosted are using the same swig version.
>>>>>>>>
>>>>>>>> I digged a bit through your layer (while my up2date scanner over meta-cpan
>>>>>>>> blocks my build chain :P) and realized that you use perl-5.20.0 as it was
>>>>>>>> in poky. A "simple" downgrade would be more reasonable ... if reason applys
>>>>>>>> here in general :)
>>>>>>>
>>>>>>> In practice, I am doing that.  However, I want to understand why perl 5.22
>>>>>>> breaks things and get it fixed.
>>>>>>
>>>>>> I did a diff between your 5.20 and poky's 5.22 and realize some fixes applied
>>>>>> in 5.22 regarding library path's aren't applied in your copy. Maybe swig relies
>>>>>> on wrong library locations and when we know, we can fix.
>>>>>>
>>>>>> So it's maybe not a 5.20 vs. 5.22 problem, it's maybe a weird swig setup problem.
>>>>>>
>>>>>>>> When you fail on cross-build and succeed in target build, try to compare the
>>>>>>>> C files and includes (even swig libraries) used.
>>>>>>>>
>>>>>>>> It smells more like a "wrong source" than a "perl problem" (and even when
>>>>>>>> I never would read any python thread, the same problem would likely occur
>>>>>>>> there, too ^^).
>>>>>>>>
>>>>>>>> Which perl headers are used in your build? To dig down, more logs would
>>>>>>>> be reasonable ...
>>>>>>>
>>>>>>> Everything comes from the same sources, same revisions, etc, as I'm using
>>>>>>> either a bitbake/OE build or the embedded (self-hosted) version from the
>>>>>>> same build plus SDK tools.
>>>>>>
>>>>>> And your SDK does not include any host tools? Did you prove the intermediate
>>>>>> amanda build files (eg. generated by SWIG) for relicts from wrong source?
>>>>>> Did you check the logs which include directories had been used?
>>>>>
>>>>> I give it a quick shot and got:
>>>>>
>>>>> ../../arm-poky-linux-gnueabi-libtool  --tag=CC   --mode=compile arm-poky-linux-gnueabi-gcc  -march=armv7-a -marm  -mthumb-interwork -mfloat-abi=hard -mfpu=neon -mtune=cortex-a9 --sysroot=/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie -DHAVE_CONFIG_H -I. -I../../config -I../../common-src -I../../common-src -I../../xfer-src -I../../gnulib -I../../ndmp-src -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/x86_64-linux/usr/lib/perl-native/perl/5.22.1/CORE   -fno-strict-aliasing -D_GNU_SOURCE -pthread -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie/usr/include/glib-2.0 -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie/usr/lib/glib-2.0/include -DSWIG  -O2 -pipe -g -feliminate-unused-debug-types -fipa-pta -ftree-partial-pre -ftree-loop-distribution -ftree-loop-distribute-patterns -fgcse-
 a
f
>> t
>> e
>>>> r
>>>> -reload -fgcse-sm -fgcse-las  -fno-strict-aliasing  -c -o xferwrap.lo xferwrap.c
>>>>>
>>>>> Do you see the "-I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/x86_64-linux/usr/lib/perl-native/perl/5.22.1/CORE" ?
>>>>>
>>>>> Any further blames of perl-5.22? :)
>>>>
>>>> I'm not sure what you are getting at here.  Indeed it is supposed to be
>>>> using perl-5.22.1 for everything.  The version 5.22.0 in meta-amanda layer
>>>> is just for testing and should not be used at the moment.
>>>
>>> Please compare word wise ...
>>>
>>> -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/x86_64-linux/usr/lib/perl-native/perl/5.22.1/CORE
>>> -I/homes/sno/fsl-release-bsp/ornithologen-kann-man-mit-voegeln-eine-freude-machen/tmp/sysroots/curie/usr/include/glib-2.0
>>>
>>> Probably this helps getting my point:
>>>
>>> $ ls -l tmp/work
>>> total 149
>>> drwxrwxr-x  54 sno sno  54 Mar 22 23:49 all-poky-linux
>>> drwxrwxr-x 269 sno sno 269 Mar 22 23:49 arm926ejse-poky-linux-gnueabi
>>> drwxrwxr-x   3 sno sno   3 Mar 22 23:48 bohr-nand-poky-linux-gnueabi
>>> drwxrwxr-x  19 sno sno  19 Mar 22 23:49 bohr-poky-linux-gnueabi
>>> drwxrwxr-x   9 sno sno   9 Mar 22 21:38 cortexa9hf-vfp-neon-mx6qdl-poky-linux-gnueabi
>>> drwxrwxr-x 326 sno sno 326 Mar 23 11:28 cortexa9hf-vfp-neon-poky-linux-gnueabi
>>> drwxrwxr-x   3 sno sno   3 Mar 22 21:37 curie-emmc-poky-linux-gnueabi
>>> drwxrwxr-x  29 sno sno  29 Mar 22 21:50 curie-poky-linux-gnueabi
>>> drwxrwxr-x 196 sno sno 196 Mar 23 11:28 x86_64-linux
>>>
>>>> All of my comparisons and testing (see instructions in the README) are
>>>> for perl-5.22.1 only and in fact I've only been using the bits from bitbake/OE
>>>> and not building that any more (I found that it wasn't necessary - one only
>>>> needs to build amanda on the target)
>>>>
>>>> Sorry for the confusion.
>>>
>>
>> Huzzah!  This seems to be on the right track.  I adjusted the PERL_INC
>> value in perl/Makefile (the *.h files used by the swig interfaces) to
>> point to the target ones and not the perl-native ones and now it works :-)
>>
>> I'm not sure what's different between the two sets of include files, and
>> more interestingly why it worked with previous versions of perl, but this
>> does seem to be the culprit.
>>
>> Now I just need to figure out how to convince the 'configure' process to
>> do it correctly.
>
> IIRC, rrdtool is a good example to do it right. collectd had the same problem
> as amanda had - wrong perl used ...
> Maybe vice versa - I'm getting old and forget to remember :D

That did it.  I just had to add this line to my recipe
   export PERL_INC = "-I${STAGING_LIBDIR}${PERL_OWN_DIR}/perl/${@get_perl_version(d)}/CORE"
and a small patch to make it honor it (instead of always building the wrong path).  Now
it builds and runs great on a 32 bit target.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------