Enabling uninative by default in oe-core?

Enabling uninative by default in oe-core?

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 1167 bytes --]

Hi,

Background: uninative is a class that downloads a precompiled host glibc
for use in the sysroot, thus isolating the native sysroot from the host
environment.  This means greater sstate reuse, as instead of native builds
being dependent on the host system they're able to be shared between all
hosts.  There is a reference tarball hosted on www.yoctoproject.org, and
the URL can be overridden by distros if you would prefer to build your own.

We enable this in Poky so that we get greater reuse on the autobuilders,
and due to some issues with the C++ ABI the eSDK generation in master now
requires uninative to be enabled.  The question is: do we now enable
uninative by default in oe-core's nodistro (pointing at the yoctoproject
tarball), or do we keep it disabled by default and require the user to
enable uninative if they wish to build an eSDK?

Personally I'm torn: I don't like eSDK not working out of the box, but I
don't really like oe-core nodistro depending on uninative.  Though enabling
uninative globally does mean everything works out of the box, so following
the principle of Least Surprise that's what we should do.

Ross

[-- Attachment #2: Type: text/html, Size: 1333 bytes --]

Re: Enabling uninative by default in oe-core?

[Khem Raj]

[-- Attachment #1.1: Type: text/plain, Size: 1552 bytes --]



On 11/17/16 9:31 AM, Burton, Ross wrote:
> Hi,
> 
> Background: uninative is a class that downloads a precompiled host glibc for
> use in the sysroot, thus isolating the native sysroot from the host
> environment.  This means greater sstate reuse, as instead of native builds
> being dependent on the host system they're able to be shared between all
> hosts.  There is a reference tarball hosted on www.yoctoproject.org
> <http://www.yoctoproject.org>, and the URL can be overridden by distros if you
> would prefer to build your own.
> 
> We enable this in Poky so that we get greater reuse on the autobuilders, and
> due to some issues with the C++ ABI the eSDK generation in master now requires
> uninative to be enabled.  The question is: do we now enable uninative by
> default in oe-core's nodistro (pointing at the yoctoproject tarball), or do we
> keep it disabled by default and require the user to enable uninative if they
> wish to build an eSDK?
> 
> Personally I'm torn: I don't like eSDK not working out of the box, but I don't
> really like oe-core nodistro depending on uninative.  Though enabling
> uninative globally does mean everything works out of the box, so following the
> principle of Least Surprise that's what we should do.

If we are supporing e-SDK in OE-Core then we should enable uninative too
on the same lines.

It does improve the user experience so I am in favor of adding it
unconditionally. May be tarball can be hosted on oe mirrors as well for
redundancy

> 
> Ross
> 
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 211 bytes --]

Re: [Openembedded-architecture] Enabling uninative by default in oe-core?

[Denys Dmytriyenko]

On Thu, Nov 17, 2016 at 10:06:46AM -0800, Khem Raj wrote:
> 
> 
> On 11/17/16 9:31 AM, Burton, Ross wrote:
> > Hi,
> > 
> > Background: uninative is a class that downloads a precompiled host glibc for
> > use in the sysroot, thus isolating the native sysroot from the host
> > environment.  This means greater sstate reuse, as instead of native builds
> > being dependent on the host system they're able to be shared between all
> > hosts.  There is a reference tarball hosted on www.yoctoproject.org
> > <http://www.yoctoproject.org>, and the URL can be overridden by distros if you
> > would prefer to build your own.
> > 
> > We enable this in Poky so that we get greater reuse on the autobuilders, and
> > due to some issues with the C++ ABI the eSDK generation in master now requires
> > uninative to be enabled.  The question is: do we now enable uninative by
> > default in oe-core's nodistro (pointing at the yoctoproject tarball), or do we
> > keep it disabled by default and require the user to enable uninative if they
> > wish to build an eSDK?
> > 
> > Personally I'm torn: I don't like eSDK not working out of the box, but I don't
> > really like oe-core nodistro depending on uninative.  Though enabling
> > uninative globally does mean everything works out of the box, so following the
> > principle of Least Surprise that's what we should do.
> 
> If we are supporing e-SDK in OE-Core then we should enable uninative too
> on the same lines.
> 
> It does improve the user experience so I am in favor of adding it
> unconditionally. May be tarball can be hosted on oe mirrors as well for
> redundancy

I still believe this new feature is moving to become mandatory a bit too 
soon...

-- 
Denys

Re: [Openembedded-architecture] Enabling uninative by default in oe-core?

[Khem Raj]

[-- Attachment #1.1: Type: text/plain, Size: 2082 bytes --]



On 11/17/16 10:50 AM, Denys Dmytriyenko wrote:
> On Thu, Nov 17, 2016 at 10:06:46AM -0800, Khem Raj wrote:
>>
>>
>> On 11/17/16 9:31 AM, Burton, Ross wrote:
>>> Hi,
>>>
>>> Background: uninative is a class that downloads a precompiled host glibc for
>>> use in the sysroot, thus isolating the native sysroot from the host
>>> environment.  This means greater sstate reuse, as instead of native builds
>>> being dependent on the host system they're able to be shared between all
>>> hosts.  There is a reference tarball hosted on www.yoctoproject.org
>>> <http://www.yoctoproject.org>, and the URL can be overridden by distros if you
>>> would prefer to build your own.
>>>
>>> We enable this in Poky so that we get greater reuse on the autobuilders, and
>>> due to some issues with the C++ ABI the eSDK generation in master now requires
>>> uninative to be enabled.  The question is: do we now enable uninative by
>>> default in oe-core's nodistro (pointing at the yoctoproject tarball), or do we
>>> keep it disabled by default and require the user to enable uninative if they
>>> wish to build an eSDK?
>>>
>>> Personally I'm torn: I don't like eSDK not working out of the box, but I don't
>>> really like oe-core nodistro depending on uninative.  Though enabling
>>> uninative globally does mean everything works out of the box, so following the
>>> principle of Least Surprise that's what we should do.
>>
>> If we are supporing e-SDK in OE-Core then we should enable uninative too
>> on the same lines.
>>
>> It does improve the user experience so I am in favor of adding it
>> unconditionally. May be tarball can be hosted on oe mirrors as well for
>> redundancy
> 
> I still believe this new feature is moving to become mandatory a bit too 
> soon...

perhaps keeping it optional for 1 release would be acceptable ? it also means
that we may not be able to develop core features using this and it would also
mean more testing matrix, which I believe is fine. Since we know non uninative
version wont get as much coverage.

> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 211 bytes --]

Re: Enabling uninative by default in oe-core?

[Nicolas Dechesne]

On Thu, Nov 17, 2016 at 7:06 PM, Khem Raj <raj.khem@gmail.com> wrote:
>> Background: uninative is a class that downloads a precompiled host glibc for
>> use in the sysroot, thus isolating the native sysroot from the host
>> environment.  This means greater sstate reuse, as instead of native builds
>> being dependent on the host system they're able to be shared between all
>> hosts.  There is a reference tarball hosted on www.yoctoproject.org
>> <http://www.yoctoproject.org>, and the URL can be overridden by distros if you
>> would prefer to build your own.
>>
>> We enable this in Poky so that we get greater reuse on the autobuilders, and
>> due to some issues with the C++ ABI the eSDK generation in master now requires
>> uninative to be enabled.  The question is: do we now enable uninative by
>> default in oe-core's nodistro (pointing at the yoctoproject tarball), or do we
>> keep it disabled by default and require the user to enable uninative if they
>> wish to build an eSDK?
>>
>> Personally I'm torn: I don't like eSDK not working out of the box, but I don't
>> really like oe-core nodistro depending on uninative.  Though enabling
>> uninative globally does mean everything works out of the box, so following the
>> principle of Least Surprise that's what we should do.
>
> If we are supporing e-SDK in OE-Core then we should enable uninative too
> on the same lines.
>
> It does improve the user experience so I am in favor of adding it
> unconditionally. May be tarball can be hosted on oe mirrors as well for
> redundancy


I am not sure how people would care about that (yet ;-) but uninative
does not work for arm64 (host).

Build Configuration:
BB_VERSION        = "1.32.0"
BUILD_SYS         = "aarch64-linux"
NATIVELSBSTRING   = "Debian-8.6"
TARGET_SYS        = "arm-oe-linux-gnueabi"
MACHINE           = "qemuarm"
DISTRO            = "nodistro"
DISTRO_VERSION    = "nodistro.0"
TUNE_FEATURES     = "arm armv5 thumb dsp"
TARGET_FPU        = "soft"
meta              = "master:9303d8055c45a0f6af295d70a6f6a8b9d8d8a7c9"

ERROR: Uninative selected but not configured correctly, please set
UNINATIVE_CHECKSUM[aarch64]

I don't know much about it, but it would be nice to fix that before we
enable it by default. I will try to have a look at it more closely..

Re: Enabling uninative by default in oe-core?

[Khem Raj]

[-- Attachment #1.1: Type: text/plain, Size: 2675 bytes --]



On 11/17/16 10:56 AM, Nicolas Dechesne wrote:
> On Thu, Nov 17, 2016 at 7:06 PM, Khem Raj <raj.khem@gmail.com> wrote:
>>> Background: uninative is a class that downloads a precompiled host glibc for
>>> use in the sysroot, thus isolating the native sysroot from the host
>>> environment.  This means greater sstate reuse, as instead of native builds
>>> being dependent on the host system they're able to be shared between all
>>> hosts.  There is a reference tarball hosted on www.yoctoproject.org
>>> <http://www.yoctoproject.org>, and the URL can be overridden by distros if you
>>> would prefer to build your own.
>>>
>>> We enable this in Poky so that we get greater reuse on the autobuilders, and
>>> due to some issues with the C++ ABI the eSDK generation in master now requires
>>> uninative to be enabled.  The question is: do we now enable uninative by
>>> default in oe-core's nodistro (pointing at the yoctoproject tarball), or do we
>>> keep it disabled by default and require the user to enable uninative if they
>>> wish to build an eSDK?
>>>
>>> Personally I'm torn: I don't like eSDK not working out of the box, but I don't
>>> really like oe-core nodistro depending on uninative.  Though enabling
>>> uninative globally does mean everything works out of the box, so following the
>>> principle of Least Surprise that's what we should do.
>>
>> If we are supporing e-SDK in OE-Core then we should enable uninative too
>> on the same lines.
>>
>> It does improve the user experience so I am in favor of adding it
>> unconditionally. May be tarball can be hosted on oe mirrors as well for
>> redundancy
> 
> 
> I am not sure how people would care about that (yet ;-) but uninative
> does not work for arm64 (host).
> 
> Build Configuration:
> BB_VERSION        = "1.32.0"
> BUILD_SYS         = "aarch64-linux"
> NATIVELSBSTRING   = "Debian-8.6"
> TARGET_SYS        = "arm-oe-linux-gnueabi"
> MACHINE           = "qemuarm"
> DISTRO            = "nodistro"
> DISTRO_VERSION    = "nodistro.0"
> TUNE_FEATURES     = "arm armv5 thumb dsp"
> TARGET_FPU        = "soft"
> meta              = "master:9303d8055c45a0f6af295d70a6f6a8b9d8d8a7c9"
> 
> ERROR: Uninative selected but not configured correctly, please set
> UNINATIVE_CHECKSUM[aarch64]
> 
> I don't know much about it, but it would be nice to fix that before we
> enable it by default. I will try to have a look at it more closely..
> 

Someone with aarch64 hardware in build boxes could take that up. I dont think
yocto project or many community members have access to hardware. I would be
happy if it informed me and continues without uninative.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 211 bytes --]

Re: [Openembedded-architecture] Enabling uninative by default in oe-core?

[Mark Hatle]

On 11/17/16 12:31 PM, Burton, Ross wrote:
> Hi,
> 
> Background: uninative is a class that downloads a precompiled host glibc for use
> in the sysroot, thus isolating the native sysroot from the host environment. 
> This means greater sstate reuse, as instead of native builds being dependent on
> the host system they're able to be shared between all hosts.  There is a
> reference tarball hosted on www.yoctoproject.org <http://www.yoctoproject.org>,
> and the URL can be overridden by distros if you would prefer to build your own.
> 
> We enable this in Poky so that we get greater reuse on the autobuilders, and due
> to some issues with the C++ ABI the eSDK generation in master now requires
> uninative to be enabled.  The question is: do we now enable uninative by default
> in oe-core's nodistro (pointing at the yoctoproject tarball), or do we keep it
> disabled by default and require the user to enable uninative if they wish to
> build an eSDK?
> 
> Personally I'm torn: I don't like eSDK not working out of the box, but I don't
> really like oe-core nodistro depending on uninative.  Though enabling uninative
> globally does mean everything works out of the box, so following the principle
> of Least Surprise that's what we should do.

I agree, I see both sides.

I'm tempted though to say, at a minimum it would be nice if oe-core had a
working (prebuilt) uninative -- or at least instructions for someone to build it
themselves.  Also corresponding instructions for the eSDK that says, BTW you
need this and this is how to do it.

That would let people use the uninative with oe-core... and maybe? let us vet
this so we can determine if it's the right thing to do by default in the future
or not.

(With that said, the stuff I'm working on is using the uninative -- and it's
solved a number of minor issues..)

--Mark

> Ross
> 
> 
> _______________________________________________
> Openembedded-architecture mailing list
> Openembedded-architecture@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
> 

Re: Enabling uninative by default in oe-core?

[Koen Kooi]

> Op 17 nov. 2016, om 18:31 heeft Burton, Ross <ross.burton@intel.com> het volgende geschreven:
> 
> Hi,
> 
> Background: uninative is a class that downloads a precompiled host glibc 

Why can’t OE build it on-demand? What’s next, requiring prebuilt toolchains?

regards,

Koen

Re: [Openembedded-architecture] Enabling uninative by default in oe-core?

[Richard Purdie]

On Fri, 2016-11-18 at 08:15 +0100, Koen Kooi wrote:
> > 
> > Op 17 nov. 2016, om 18:31 heeft Burton, Ross <ross.burton@intel.com
> > > het volgende geschreven:
> > 
> > Hi,
> > 
> > Background: uninative is a class that downloads a precompiled host
> > glibc 
> Why can’t OE build it on-demand? What’s next, requiring prebuilt
> toolchains?

Its a chicken and egg problem.

We could add a special "uninativesdk" BBCLASSEXTEND range of targets,
then require that it built before we build anything native (or anything
that needs a native tool). This would push build times up 'a little'
and I suspect might not be popular. So the reason we "can't" is that
its impractical, not an absolute technical constraint.

Cheers,

Richard

Re: Enabling uninative by default in oe-core?

[akuster808]

[-- Attachment #1: Type: text/plain, Size: 1877 bytes --]



On 11/17/2016 09:31 AM, Burton, Ross wrote:
> Hi,
>
> Background: uninative is a class that downloads a precompiled host 
> glibc for use in the sysroot, thus isolating the native sysroot from 
> the host environment.  This means greater sstate reuse, as instead of 
> native builds being dependent on the host system they're able to be 
> shared between all hosts.  There is a reference tarball hosted on 
> www.yoctoproject.org <http://www.yoctoproject.org>, and the URL can be 
> overridden by distros if you would prefer to build your own.
>
> We enable this in Poky so that we get greater reuse on the 
> autobuilders, and due to some issues with the C++ ABI the eSDK 
> generation in master now requires uninative to be enabled. The 
> question is: do we now enable uninative by default in oe-core's 
> nodistro (pointing at the yoctoproject tarball), or do we keep it 
> disabled by default and require the user to enable uninative if they 
> wish to build an eSDK?

If Poky wants the default to use a prebuilt uninative that is fine, but 
it should be not be the default in OE.  In the spirit of Bitbake, 
uninative should be a build dependency for eSDK with the option of using 
a prebuilt one.

>
> Personally I'm torn: I don't like eSDK not working out of the box, but 
> I don't really like oe-core nodistro depending on uninative.

Well Bitbake does not work out of the box on every host either. There 
are packages needing to exist on the host as a prerequisite. I see this 
in the same light.
Do we make "Buildtools" a  requirement to use Bitbake?


||||||
> Though enabling uninative globally does mean everything works out of 
> the box, so following the principle of Least Surprise that's what we 
> should do.

I see there are multiple versions, which one works with which release?


- Armin
>
> Ross
>
>


[-- Attachment #2: Type: text/html, Size: 3479 bytes --]

Re: [Openembedded-architecture] Enabling uninative by default in oe-core?

[Richard Purdie]

On Fri, 2016-11-18 at 08:28 -0800, akuster808 wrote:
> On 11/17/2016 09:31 AM, Burton, Ross wrote:
> > Background: uninative is a class that downloads a precompiled host
> > glibc for use in the sysroot, thus isolating the native sysroot
> > from the host environment.  This means greater sstate reuse, as
> > instead of native builds being dependent on the host system they're
> > able to be shared between all hosts.  There is a reference tarball
> > hosted on www.yoctoproject.org, and the URL can be overridden by
> > distros if you would prefer to build your own.
> > 
> > We enable this in Poky so that we get greater reuse on the
> > autobuilders, and due to some issues with the C++ ABI the eSDK
> > generation in master now requires uninative to be enabled.  The
> > question is: do we now enable uninative by default in oe-core's
> > nodistro (pointing at the yoctoproject tarball), or do we keep it
> > disabled by default and require the user to enable uninative if
> > they wish to build an eSDK?
>  
> If Poky wants the default to use a prebuilt uninative that is fine,
> but it should be not be the default in OE.  In the spirit of Bitbake,
> uninative should be a build dependency for eSDK with the option of
> using a prebuilt one.

Its not that simple. Using uninative requires certain options passed in
when compiling native recipes for example, e.g. to pick particular C++
abis. If you start the build without those set (since uninative is
disabled), you can't get native sstate built in the right way for it to
work with eSDK. We could of course add a new BBCLASSEXTEND, "native2"
which is native specially for use in the eSDK but that seems silly.

I guess we could move the configuration uninative requires into global
bitbake.conf but not require the actual binary shim to be enabled. That
would let eSDK work in OE-Core just not make OE-Core require uninative.
It would mean the compiler options for native would be a little
different. That might be an acceptable compromise?

> > Though enabling uninative globally does mean everything works out
> > of the box, so following the principle of Least Surprise that's
> > what we should do.
>  
> I see there are multiple versions, which one works with which
> release?

The one configured in meta/conf/distro/include/yocto-uninative.inc.
By putting a standard include there its easy for anyone to opt into
this should they wish to.

Cheers,

Richard

Re: [Openembedded-architecture] Enabling uninative by default in oe-core?

[Koen Kooi]

> Op 18 nov. 2016, om 19:06 heeft Richard Purdie <richard.purdie@linuxfoundation.org> het volgende geschreven:
> 
> On Fri, 2016-11-18 at 08:28 -0800, akuster808 wrote:
>> On 11/17/2016 09:31 AM, Burton, Ross wrote:
>>> Background: uninative is a class that downloads a precompiled host
>>> glibc for use in the sysroot, thus isolating the native sysroot
>>> from the host environment.  This means greater sstate reuse, as
>>> instead of native builds being dependent on the host system they're
>>> able to be shared between all hosts.  There is a reference tarball
>>> hosted on www.yoctoproject.org, and the URL can be overridden by
>>> distros if you would prefer to build your own.
>>> 
>>> We enable this in Poky so that we get greater reuse on the
>>> autobuilders, and due to some issues with the C++ ABI the eSDK
>>> generation in master now requires uninative to be enabled.  The
>>> question is: do we now enable uninative by default in oe-core's
>>> nodistro (pointing at the yoctoproject tarball), or do we keep it
>>> disabled by default and require the user to enable uninative if
>>> they wish to build an eSDK?
>>  
>> If Poky wants the default to use a prebuilt uninative that is fine,
>> but it should be not be the default in OE.  In the spirit of Bitbake,
>> uninative should be a build dependency for eSDK with the option of
>> using a prebuilt one.
> 
> Its not that simple. Using uninative requires certain options passed in
> when compiling native recipes for example, e.g. to pick particular C++
> abis. If you start the build without those set (since uninative is
> disabled), you can't get native sstate built in the right way for it to
> work with eSDK. We could of course add a new BBCLASSEXTEND, "native2"
> which is native specially for use in the eSDK but that seems silly.
> 
> I guess we could move the configuration uninative requires into global
> bitbake.conf but not require the actual binary shim to be enabled. That
> would let eSDK work in OE-Core just not make OE-Core require uninative.
> It would mean the compiler options for native would be a little
> different. That might be an acceptable compromise?

It would be for me.

regards,

Koen

Re: [Openembedded-architecture] Enabling uninative by default in oe-core?

[Denys Dmytriyenko]

On Fri, Nov 18, 2016 at 09:50:08PM +0100, Koen Kooi wrote:
> 
> > Op 18 nov. 2016, om 19:06 heeft Richard Purdie <richard.purdie@linuxfoundation.org> het volgende geschreven:
> > 
> > On Fri, 2016-11-18 at 08:28 -0800, akuster808 wrote:
> >> On 11/17/2016 09:31 AM, Burton, Ross wrote:
> >>> Background: uninative is a class that downloads a precompiled host
> >>> glibc for use in the sysroot, thus isolating the native sysroot
> >>> from the host environment.  This means greater sstate reuse, as
> >>> instead of native builds being dependent on the host system they're
> >>> able to be shared between all hosts.  There is a reference tarball
> >>> hosted on www.yoctoproject.org, and the URL can be overridden by
> >>> distros if you would prefer to build your own.
> >>> 
> >>> We enable this in Poky so that we get greater reuse on the
> >>> autobuilders, and due to some issues with the C++ ABI the eSDK
> >>> generation in master now requires uninative to be enabled.  The
> >>> question is: do we now enable uninative by default in oe-core's
> >>> nodistro (pointing at the yoctoproject tarball), or do we keep it
> >>> disabled by default and require the user to enable uninative if
> >>> they wish to build an eSDK?
> >>  
> >> If Poky wants the default to use a prebuilt uninative that is fine,
> >> but it should be not be the default in OE.  In the spirit of Bitbake,
> >> uninative should be a build dependency for eSDK with the option of
> >> using a prebuilt one.
> > 
> > Its not that simple. Using uninative requires certain options passed in
> > when compiling native recipes for example, e.g. to pick particular C++
> > abis. If you start the build without those set (since uninative is
> > disabled), you can't get native sstate built in the right way for it to
> > work with eSDK. We could of course add a new BBCLASSEXTEND, "native2"
> > which is native specially for use in the eSDK but that seems silly.
> > 
> > I guess we could move the configuration uninative requires into global
> > bitbake.conf but not require the actual binary shim to be enabled. That
> > would let eSDK work in OE-Core just not make OE-Core require uninative.
> > It would mean the compiler options for native would be a little
> > different. That might be an acceptable compromise?
> 
> It would be for me.

Agree, sounds like a good compromise.

-- 
Denys