multilib support on Jethro: populate_sdk help

multilib support on Jethro: populate_sdk help

[Nemicolopterus Crypticus]

[-- Attachment #1: Type: text/plain, Size: 2917 bytes --]

Hello all,

We are having issues with our generated SDK files. We're using the jethro
branch, and have enabled multilib (see relevant portion of local.conf
below).

I can succesfully run the command to generate the SDK:
$ bitbake <recipe name> -c populate_sdk

But the output is incorrect. *While the aarch sysroots look good, the
sysroots for arm are completely missing. The environment setup script ends
up pointing to non-existent compilers. *Please let me know if you need more
information!

Steps to reproduce:
$ bitbake <recipe name> -c populate_sdk
$ cp tmp/deploy/sdk/<sdk-script>.sh /test/directory
$ cd /test/directory
$ bash <sdk-script>.sh
$ ls sysroots/
aarch64-poky-linux/   x86_64-pokysdk-linux/

Note the missing ARM directories.

Also, the setup script for arm sets this as the command for compiler:
 12 export CC="arm-pokymllib32-linux-gnueabi-gcc

But that compiler doesn't exist anywhere in the folder where I installed
everything, or environment!

So it appears that it's not completely or correctly packaging everything
for ARMv7.

The documentation does mention some environment variables
<http://www.yoctoproject.org/docs/2.0/mega-manual/mega-manual.html#sdk-dev-environment>,
but it's not clear how those related to multilib.

For instance, none of our recipes have "SDKIMAGE_FEATURES" defined, but
clearly the aarch64 sysroots are getting populated correctly (we
succesfully compiled and ran a simple hello world).

*Can anyone point me to an example, or discuss a successful use of
"multilib" on the Jethro branch (given desparate brainstorming below),
share some more verbose documentation, or ask follow-up questions?*


_____________________________________________________________
Desperate brainstorming below:
I did find this years-old patch:
https://patchwork.openembedded.org/patch/30941/
Notably, the do_populate_sdk method in
poky/meta/classes/populate_sdk_base.bbclass iterates over the values listed
in the MULTILIB_VARIANTS variable:

+	variants = d.getVar("MULTILIB_VARIANTS", True) or ""+	for item in
variants.split():+		# Load overrides from 'd' to avoid having to reset
the value...+		overrides = d.getVar("OVERRIDES", False) +
":virtclass-multilib-" + item+		localdata.setVar("OVERRIDES",
overrides)+		bb.data.update_data(localdata)+		bb.build.exec_func("create_sdk_files",
localdata)



The latest version in Jethro does not have anything equivalent to that, but
I did not dig deeply into the code or the commit history.




______________________________________________________________
More data:

local.conf relevant section:
#
# Multilib configuration
#
# This sets any packages preprended with lib32- to be built with
# the armv7a tuning (32 bit) instead of 64 bit aarch.
#
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "armv7at-neon"

[-- Attachment #2: Type: text/html, Size: 3754 bytes --]

Re: multilib support on Jethro: populate_sdk help

[Khem Raj]

[-- Attachment #1.1: Type: text/plain, Size: 3485 bytes --]


> On Jul 29, 2016, at 10:48 AM, Nemicolopterus Crypticus <ncrypticus@gmail.com> wrote:
> 
> Hello all,
> 
> We are having issues with our generated SDK files. We're using the jethro branch, and have enabled multilib (see relevant portion of local.conf below).
> 
> I can succesfully run the command to generate the SDK:
> $ bitbake <recipe name> -c populate_sdk
> 
> But the output is incorrect. While the aarch sysroots look good, the sysroots for arm are completely missing. The environment setup script ends up pointing to non-existent compilers. Please let me know if you need more information!

Thanks for report. It will be good if you can open a bugzilla entry for this. So it can be tracked.

> 
> Steps to reproduce:
> $ bitbake <recipe name> -c populate_sdk
> $ cp tmp/deploy/sdk/<sdk-script>.sh /test/directory
> $ cd /test/directory
> $ bash <sdk-script>.sh
> $ ls sysroots/
> aarch64-poky-linux/   x86_64-pokysdk-linux/
> 
> Note the missing ARM directories.
> 
> Also, the setup script for arm sets this as the command for compiler:
>  12 export CC="arm-pokymllib32-linux-gnueabi-gcc
> 
> But that compiler doesn't exist anywhere in the folder where I installed everything, or environment!
> 
> So it appears that it's not completely or correctly packaging everything for ARMv7.
> 
> The documentation does mention some environment variables <http://www.yoctoproject.org/docs/2.0/mega-manual/mega-manual.html#sdk-dev-environment>, but it's not clear how those related to multilib.
> 
> For instance, none of our recipes have "SDKIMAGE_FEATURES" defined, but clearly the aarch64 sysroots are getting populated correctly (we succesfully compiled and ran a simple hello world).
> 
> Can anyone point me to an example, or discuss a successful use of "multilib" on the Jethro branch (given desparate brainstorming below), share some more verbose documentation, or ask follow-up questions?
> 
> 
> _____________________________________________________________
> Desperate brainstorming below:
> I did find this years-old patch: https://patchwork.openembedded.org/patch/30941/ <https://patchwork.openembedded.org/patch/30941/>
> Notably, the do_populate_sdk method in poky/meta/classes/populate_sdk_base.bbclass iterates over the values listed in the MULTILIB_VARIANTS variable:
> 
> +	variants = d.getVar("MULTILIB_VARIANTS", True) or ""
> +	for item in variants.split():
> +		# Load overrides from 'd' to avoid having to reset the value...
> +		overrides = d.getVar("OVERRIDES", False) + ":virtclass-multilib-" + item
> +		localdata.setVar("OVERRIDES", overrides)
> +		bb.data.update_data(localdata)
> +		bb.build.exec_func("create_sdk_files", localdata)
> 
> 
> The latest version in Jethro does not have anything equivalent to that, but I did not dig deeply into the code or the commit history.
> 
> 
> 
> 
> ______________________________________________________________
> More data:
> 
> local.conf relevant section:
> #
> # Multilib configuration
> #
> # This sets any packages preprended with lib32- to be built with
> # the armv7a tuning (32 bit) instead of 64 bit aarch.
> #
> require conf/multilib.conf
> MULTILIBS = "multilib:lib32"
> DEFAULTTUNE_virtclass-multilib-lib32 = "armv7at-neon"
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core


[-- Attachment #1.2: Type: text/html, Size: 5626 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 211 bytes --]

Re: multilib support on Jethro: populate_sdk help

[Nemicolopterus Crypticus]

[-- Attachment #1: Type: text/plain, Size: 3788 bytes --]

Added: https://bugzilla.yoctoproject.org/show_bug.cgi?id=10054


Please let me know if I can try anything else or provide more info.

In the meantime, I'm going to try to disable multilib, and only build the
ARMv7 version of the SDK.


On Fri, Jul 29, 2016 at 12:07 PM, Khem Raj <raj.khem@gmail.com> wrote:

>
> On Jul 29, 2016, at 10:48 AM, Nemicolopterus Crypticus <
> ncrypticus@gmail.com> wrote:
>
> Hello all,
>
> We are having issues with our generated SDK files. We're using the jethro
> branch, and have enabled multilib (see relevant portion of local.conf
> below).
>
> I can succesfully run the command to generate the SDK:
> $ bitbake <recipe name> -c populate_sdk
>
> But the output is incorrect. *While the aarch sysroots look good, the
> sysroots for arm are completely missing. The environment setup script ends
> up pointing to non-existent compilers. *Please let me know if you need
> more information!
>
>
> Thanks for report. It will be good if you can open a bugzilla entry for
> this. So it can be tracked.
>
>
> Steps to reproduce:
> $ bitbake <recipe name> -c populate_sdk
> $ cp tmp/deploy/sdk/<sdk-script>.sh /test/directory
> $ cd /test/directory
> $ bash <sdk-script>.sh
> $ ls sysroots/
> aarch64-poky-linux/   x86_64-pokysdk-linux/
>
> Note the missing ARM directories.
>
> Also, the setup script for arm sets this as the command for compiler:
>  12 export CC="arm-pokymllib32-linux-gnueabi-gcc
>
> But that compiler doesn't exist anywhere in the folder where I installed
> everything, or environment!
>
> So it appears that it's not completely or correctly packaging everything
> for ARMv7.
>
> The documentation does mention some environment variables
> <http://www.yoctoproject.org/docs/2.0/mega-manual/mega-manual.html#sdk-dev-environment>,
> but it's not clear how those related to multilib.
>
> For instance, none of our recipes have "SDKIMAGE_FEATURES" defined, but
> clearly the aarch64 sysroots are getting populated correctly (we
> succesfully compiled and ran a simple hello world).
>
> *Can anyone point me to an example, or discuss a successful use of
> "multilib" on the Jethro branch (given desparate brainstorming below),
> share some more verbose documentation, or ask follow-up questions?*
>
>
> _____________________________________________________________
> Desperate brainstorming below:
> I did find this years-old patch:
> https://patchwork.openembedded.org/patch/30941/
> Notably, the do_populate_sdk method in
> poky/meta/classes/populate_sdk_base.bbclass iterates over the values listed
> in the MULTILIB_VARIANTS variable:
>
> +	variants = d.getVar("MULTILIB_VARIANTS", True) or ""+	for item in variants.split():+		# Load overrides from 'd' to avoid having to reset the value...+		overrides = d.getVar("OVERRIDES", False) + ":virtclass-multilib-" + item+		localdata.setVar("OVERRIDES", overrides)+		bb.data.update_data(localdata)+		bb.build.exec_func("create_sdk_files", localdata)
>
>
>
> The latest version in Jethro does not have anything equivalent to that,
> but I did not dig deeply into the code or the commit history.
>
>
>
>
> ______________________________________________________________
> More data:
>
> local.conf relevant section:
> #
> # Multilib configuration
> #
> # This sets any packages preprended with lib32- to be built with
> # the armv7a tuning (32 bit) instead of 64 bit aarch.
> #
> require conf/multilib.conf
> MULTILIBS = "multilib:lib32"
> DEFAULTTUNE_virtclass-multilib-lib32 = "armv7at-neon"
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>
>

[-- Attachment #2: Type: text/html, Size: 5452 bytes --]

Re: multilib support on Jethro: populate_sdk help

[Nemicolopterus Crypticus]

[-- Attachment #1: Type: text/plain, Size: 4574 bytes --]

Ok, I've made some progress.
I'm running
$bitbake lib32-<image name> -c populate_sdk

and it proceeds for a while.
We do have a few packages that are not ready to be built as 32-bit, and
those fail as compiler errors.

Is there a way to temporarily remove them from the list of packages to be
built?
Poking around I found DEFAULT_PREFERENCE*, but that looks like it's not
quite what I'm looking for.

Is there any easy way to quickly disable a particular package from an image
build?

* http://www.embeddedlinux.org.cn/OEManual/recipes_defaultpreference.html


On Fri, Jul 29, 2016 at 1:30 PM, Nemicolopterus Crypticus <
ncrypticus@gmail.com> wrote:

> Added: https://bugzilla.yoctoproject.org/show_bug.cgi?id=10054
>
>
> Please let me know if I can try anything else or provide more info.
>
> In the meantime, I'm going to try to disable multilib, and only build the
> ARMv7 version of the SDK.
>
>
> On Fri, Jul 29, 2016 at 12:07 PM, Khem Raj <raj.khem@gmail.com> wrote:
>
>>
>> On Jul 29, 2016, at 10:48 AM, Nemicolopterus Crypticus <
>> ncrypticus@gmail.com> wrote:
>>
>> Hello all,
>>
>> We are having issues with our generated SDK files. We're using the jethro
>> branch, and have enabled multilib (see relevant portion of local.conf
>> below).
>>
>> I can succesfully run the command to generate the SDK:
>> $ bitbake <recipe name> -c populate_sdk
>>
>> But the output is incorrect. *While the aarch sysroots look good, the
>> sysroots for arm are completely missing. The environment setup script ends
>> up pointing to non-existent compilers. *Please let me know if you need
>> more information!
>>
>>
>> Thanks for report. It will be good if you can open a bugzilla entry for
>> this. So it can be tracked.
>>
>>
>> Steps to reproduce:
>> $ bitbake <recipe name> -c populate_sdk
>> $ cp tmp/deploy/sdk/<sdk-script>.sh /test/directory
>> $ cd /test/directory
>> $ bash <sdk-script>.sh
>> $ ls sysroots/
>> aarch64-poky-linux/   x86_64-pokysdk-linux/
>>
>> Note the missing ARM directories.
>>
>> Also, the setup script for arm sets this as the command for compiler:
>>  12 export CC="arm-pokymllib32-linux-gnueabi-gcc
>>
>> But that compiler doesn't exist anywhere in the folder where I installed
>> everything, or environment!
>>
>> So it appears that it's not completely or correctly packaging everything
>> for ARMv7.
>>
>> The documentation does mention some environment variables
>> <http://www.yoctoproject.org/docs/2.0/mega-manual/mega-manual.html#sdk-dev-environment>,
>> but it's not clear how those related to multilib.
>>
>> For instance, none of our recipes have "SDKIMAGE_FEATURES" defined, but
>> clearly the aarch64 sysroots are getting populated correctly (we
>> succesfully compiled and ran a simple hello world).
>>
>> *Can anyone point me to an example, or discuss a successful use of
>> "multilib" on the Jethro branch (given desparate brainstorming below),
>> share some more verbose documentation, or ask follow-up questions?*
>>
>>
>> _____________________________________________________________
>> Desperate brainstorming below:
>> I did find this years-old patch:
>> https://patchwork.openembedded.org/patch/30941/
>> Notably, the do_populate_sdk method in
>> poky/meta/classes/populate_sdk_base.bbclass iterates over the values listed
>> in the MULTILIB_VARIANTS variable:
>>
>> +	variants = d.getVar("MULTILIB_VARIANTS", True) or ""+	for item in variants.split():+		# Load overrides from 'd' to avoid having to reset the value...+		overrides = d.getVar("OVERRIDES", False) + ":virtclass-multilib-" + item+		localdata.setVar("OVERRIDES", overrides)+		bb.data.update_data(localdata)+		bb.build.exec_func("create_sdk_files", localdata)
>>
>>
>>
>> The latest version in Jethro does not have anything equivalent to that,
>> but I did not dig deeply into the code or the commit history.
>>
>>
>>
>>
>> ______________________________________________________________
>> More data:
>>
>> local.conf relevant section:
>> #
>> # Multilib configuration
>> #
>> # This sets any packages preprended with lib32- to be built with
>> # the armv7a tuning (32 bit) instead of 64 bit aarch.
>> #
>> require conf/multilib.conf
>> MULTILIBS = "multilib:lib32"
>> DEFAULTTUNE_virtclass-multilib-lib32 = "armv7at-neon"
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 6622 bytes --]

Re: multilib support on Jethro: populate_sdk help

[Nemicolopterus Crypticus]

[-- Attachment #1: Type: text/plain, Size: 6401 bytes --]

Ok, I moved past that above error as well. I edited the image recipe, and
just commented out the packages that aren't ready for lib32 build yet.

Unfortunately, even when I build with lib32-<image name> -c populate_sdk,
the sysroots for armv7 aren't getting populated. Also, the environment
setup script does not work properly. (i.e., the environment variable $CC
points to a file that does not exist).

Here is the name of the output in tmp/deploy/sdk:
poky-glibc-x86_64-lib32-<image name>-aarch64-toolchain-2.0.1.sh

As you can see, the TUNE_PKGARCH is still set to aarch64. In fact, when I
first run the bitbake command, this is what I see:
$ bitbake lib32-<image name> -c populate_sdk
Build Configuration:
BB_VERSION        = "1.28.0"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "Ubuntu-14.04"
TARGET_SYS        = "aarch64-poky-linux"
MACHINE           = "qcom-bsp"
DISTRO            = "poky"
DISTRO_VERSION    = "2.0.1"
*TUNE_FEATURES     = "aarch64"*
TARGET_FPU        = ""

It seems this is coming from the MACHINE file, which includes this line:
require conf/machine/include/arm/arch-armv8.inc

That file just has a single line in it:
 requires conf/machine/include/arm/arch-arm64.inc

THAT file immediately sets the DEFAULTTUNE to aarch64.

So, how am I supposed to enable armv7 SDK to be built? Shouldn't the
DEFAULTTUNE be set to armv7-neon for this to work?
Am I missing something simple? I've read through the documentation a few
times, but fully admit that bitbake is large and complex and I may be
missing something.




On Mon, Aug 1, 2016 at 4:05 PM, Nemicolopterus Crypticus <
ncrypticus@gmail.com> wrote:

> Ok, I've made some progress.
> I'm running
> $bitbake lib32-<image name> -c populate_sdk
>
> and it proceeds for a while.
> We do have a few packages that are not ready to be built as 32-bit, and
> those fail as compiler errors.
>
> Is there a way to temporarily remove them from the list of packages to be
> built?
> Poking around I found DEFAULT_PREFERENCE*, but that looks like it's not
> quite what I'm looking for.
>
> Is there any easy way to quickly disable a particular package from an
> image build?
>
> * http://www.embeddedlinux.org.cn/OEManual/recipes_defaultpreference.html
>
>
> On Fri, Jul 29, 2016 at 1:30 PM, Nemicolopterus Crypticus <
> ncrypticus@gmail.com> wrote:
>
>> Added: https://bugzilla.yoctoproject.org/show_bug.cgi?id=10054
>>
>>
>> Please let me know if I can try anything else or provide more info.
>>
>> In the meantime, I'm going to try to disable multilib, and only build the
>> ARMv7 version of the SDK.
>>
>>
>> On Fri, Jul 29, 2016 at 12:07 PM, Khem Raj <raj.khem@gmail.com> wrote:
>>
>>>
>>> On Jul 29, 2016, at 10:48 AM, Nemicolopterus Crypticus <
>>> ncrypticus@gmail.com> wrote:
>>>
>>> Hello all,
>>>
>>> We are having issues with our generated SDK files. We're using the
>>> jethro branch, and have enabled multilib (see relevant portion of
>>> local.conf below).
>>>
>>> I can succesfully run the command to generate the SDK:
>>> $ bitbake <recipe name> -c populate_sdk
>>>
>>> But the output is incorrect. *While the aarch sysroots look good, the
>>> sysroots for arm are completely missing. The environment setup script ends
>>> up pointing to non-existent compilers. *Please let me know if you need
>>> more information!
>>>
>>>
>>> Thanks for report. It will be good if you can open a bugzilla entry for
>>> this. So it can be tracked.
>>>
>>>
>>> Steps to reproduce:
>>> $ bitbake <recipe name> -c populate_sdk
>>> $ cp tmp/deploy/sdk/<sdk-script>.sh /test/directory
>>> $ cd /test/directory
>>> $ bash <sdk-script>.sh
>>> $ ls sysroots/
>>> aarch64-poky-linux/   x86_64-pokysdk-linux/
>>>
>>> Note the missing ARM directories.
>>>
>>> Also, the setup script for arm sets this as the command for compiler:
>>>  12 export CC="arm-pokymllib32-linux-gnueabi-gcc
>>>
>>> But that compiler doesn't exist anywhere in the folder where I installed
>>> everything, or environment!
>>>
>>> So it appears that it's not completely or correctly packaging everything
>>> for ARMv7.
>>>
>>> The documentation does mention some environment variables
>>> <http://www.yoctoproject.org/docs/2.0/mega-manual/mega-manual.html#sdk-dev-environment>,
>>> but it's not clear how those related to multilib.
>>>
>>> For instance, none of our recipes have "SDKIMAGE_FEATURES" defined, but
>>> clearly the aarch64 sysroots are getting populated correctly (we
>>> succesfully compiled and ran a simple hello world).
>>>
>>> *Can anyone point me to an example, or discuss a successful use of
>>> "multilib" on the Jethro branch (given desparate brainstorming below),
>>> share some more verbose documentation, or ask follow-up questions?*
>>>
>>>
>>> _____________________________________________________________
>>> Desperate brainstorming below:
>>> I did find this years-old patch:
>>> https://patchwork.openembedded.org/patch/30941/
>>> Notably, the do_populate_sdk method in
>>> poky/meta/classes/populate_sdk_base.bbclass iterates over the values listed
>>> in the MULTILIB_VARIANTS variable:
>>>
>>> +	variants = d.getVar("MULTILIB_VARIANTS", True) or ""+	for item in variants.split():+		# Load overrides from 'd' to avoid having to reset the value...+		overrides = d.getVar("OVERRIDES", False) + ":virtclass-multilib-" + item+		localdata.setVar("OVERRIDES", overrides)+		bb.data.update_data(localdata)+		bb.build.exec_func("create_sdk_files", localdata)
>>>
>>>
>>>
>>> The latest version in Jethro does not have anything equivalent to that,
>>> but I did not dig deeply into the code or the commit history.
>>>
>>>
>>>
>>>
>>> ______________________________________________________________
>>> More data:
>>>
>>> local.conf relevant section:
>>> #
>>> # Multilib configuration
>>> #
>>> # This sets any packages preprended with lib32- to be built with
>>> # the armv7a tuning (32 bit) instead of 64 bit aarch.
>>> #
>>> require conf/multilib.conf
>>> MULTILIBS = "multilib:lib32"
>>> DEFAULTTUNE_virtclass-multilib-lib32 = "armv7at-neon"
>>> --
>>> _______________________________________________
>>> Openembedded-core mailing list
>>> Openembedded-core@lists.openembedded.org
>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>>
>>>
>>>
>>
>

[-- Attachment #2: Type: text/html, Size: 9058 bytes --]