Automatic prebuilt management

Automatic prebuilt management

[Loic Poulain]

[-- Attachment #1: Type: text/plain, Size: 1278 bytes --]

Hello,

Say a company works with an OE internal tree, with open-source and
closed-source packages. The company wants to release the tree so that its
customers can fully customize the images/distro. The closed sources being
only available internally, the company has to generate and handle prebuilt
binaries for proprietary packages. Ideally, with a unified public/internal
OE tree (same recipes for internal/public build).

This question is actually similar to an older thread:
https://marc.info/?l=openembedded-core&m=146779329804683

I started to work on this and added a 'generate-prebuilt' class which
generates a tarball of ${D} in deploy/prebuilts after do_install task.
Symmetrically, It's also possible to create an 'install-prebuilt' class
which bypasses do_fetch, do_unpack, ..., do_install with noexec flag and
instead uncompresses a previously generated prebuilt tarball into ${D} and
continue normally. But I would like something smarter, e.g. first trying to
check if the SRC_URI is available, if not switching on using the prebuilt
package if available (e.g. in a DIR_PREBUILT)...

Before going further is there already an existing solution for that? do you
have any recommendation on the easier/best way to achieve this?

Regards,
Loic

[-- Attachment #2: Type: text/html, Size: 1579 bytes --]

Re: Automatic prebuilt management

[chris.laplante]

> But I would like something smarter, e.g. first trying to check if the SRC_URI is available, if not switching on using the prebuilt package if available (e.g.
> in a DIR_PREBUILT)...
> 
> Before going further is there already an existing solution for that? do you have any recommendation on the easier/best way to
> achieve this?

I don't have a "best practice" recommendation to offer. But coincidentally I suggested something similar for upstreaming a few weeks ago. It was (rightfully) rejected as too "hacky" for inclusion in core code. Regardless, here's a bbclass we currently use:

addhandler skip_eventhandler
python skip_eventhandler() {
    try:
        bb.fetch2.Fetch(d.getVar("SRC_URI").split(), d)
    except bb.fetch2.FetchError:
        raise bb.parse.SkipRecipe("skip-inaccessible: could not access upstream repo; check SRC_URI, access rights, and network availability")
}
skip_eventhandler[eventmask] = "bb.event.RecipePreFinalise"


As was pointed out to me, this is a pretty heavy-handed approach - it can fail for any number of reasons, not just the URI actually being unavailable. So you may consider a more surgical check, e.g. actually trying to fetch the SRC_URI directly using raw GET requests.

To implement the handover, you'd have separate recipes for the build-from-source and the prebuilt case which PROVIDE the same package. You probably also want to set PREFERRED_PROVIDER in your layer conf to prefer the build-from-source recipe. 

Alternatively, you can do it all in a single recipe by rewriting the SRC_URI from within the event handler. But I think separate recipes is nicer.

Chris

Re: Automatic prebuilt management

[Loic Poulain]

[-- Attachment #1.1: Type: text/plain, Size: 2350 bytes --]

On Mon, 5 Aug 2019 at 18:39, chris.laplante@agilent.com <
chris.laplante@agilent.com> wrote:

> > But I would like something smarter, e.g. first trying to check if the
> SRC_URI is available, if not switching on using the prebuilt package if
> available (e.g.
> > in a DIR_PREBUILT)...
> >
> > Before going further is there already an existing solution for that? do
> you have any recommendation on the easier/best way to
> > achieve this?
>
> I don't have a "best practice" recommendation to offer. But coincidentally
> I suggested something similar for upstreaming a few weeks ago. It was
> (rightfully) rejected as too "hacky" for inclusion in core code.
> Regardless, here's a bbclass we currently use:
>
> addhandler skip_eventhandler
> python skip_eventhandler() {
>     try:
>         bb.fetch2.Fetch(d.getVar("SRC_URI").split(), d)
>     except bb.fetch2.FetchError:
>         raise bb.parse.SkipRecipe("skip-inaccessible: could not access
> upstream repo; check SRC_URI, access rights, and network availability")
> }
> skip_eventhandler[eventmask] = "bb.event.RecipePreFinalise"
>
>
> As was pointed out to me, this is a pretty heavy-handed approach - it can
> fail for any number of reasons, not just the URI actually being
> unavailable. So you may consider a more surgical check, e.g. actually
> trying to fetch the SRC_URI directly using raw GET requests.
>
> To implement the handover, you'd have separate recipes for the
> build-from-source and the prebuilt case which PROVIDE the same package. You
> probably also want to set PREFERRED_PROVIDER in your layer conf to prefer
> the build-from-source recipe.
>
> Alternatively, you can do it all in a single recipe by rewriting the
> SRC_URI from within the event handler. But I think separate recipes is
> nicer.
>

Well, I ended up with the attached class, which generates or install
prebuilt tarball. It's only a draft and I'll come back with a polished
version, but the idea is to look for the related prebuilt package into a
'PREBUILT_SRC_DIR' (if defined) and install it instead of fetching and
compiling from source. This allows having only one recipe. Then a company
just needs to provide closed-source software as a set of prebuilt tarballs
and customer need to point to the prebuilt dir (e.g. in local.conf).

Regards,
Loic

[-- Attachment #1.2: Type: text/html, Size: 2815 bytes --]

[-- Attachment #2: prebuilt.bbclass --]
[-- Type: application/octet-stream, Size: 2773 bytes --]

# This class allows to:
#     - Generate a prebuilt tarball of a package
#     - Use a previsouly generated prebuilt tarball instead of
#       fetching and compiling from source, when PREBUILT_SRC_DIR
#       is defined.
#
# A package has to inherits 'prebuilt' to generate a prebuilt tarball.
# Add PREBUILT_STRIP = "1" to generate stripped prebuilt (default is 0).
# To use prebuilt, define PREBUILT_SRC_DIR (e.g. in conf/local.conf):
#     PREBUILT_SRC_DIR = "/home/vendor/prebuilts"
#
# TODO License management

# Anonymous function needs to be executed each time so that runqueue
# can be updated
BB_DONT_CACHE = "1"

DEPLOY_DIR_PREBUILT ?= "${DEPLOY_DIR}/prebuilts"

PREBUILT_DIR="${WORKDIR}/prebuilt"
PREBUILT_BASENAME="${PN}_${PV}_prebuilt_${PACKAGE_ARCH}.tar.gz"
PREBUILT_DST="${PREBUILT_DIR}/${PREBUILT_BASENAME}"
PREBUILT_SRC="${PREBUILT_SRC_DIR}/${PREBUILT_BASENAME}"

PREBUILT_STRIP ?= "0"

# Install Prebuilt tarball
do_install_prebuilt[dirs] = "${D}"
do_install_prebuilt[doc] = "Populate Destination directory with prebuilt tarball content"

fakeroot do_install_prebuilt() {
    tar -xvzf ${PREBUILT_SRC} -C ${D}
}

# Generate Prebuilt tarball
DEPLOY_DIR_PREBUILT = "${DEPLOY_DIR}/prebuilts"
SSTATETASKS += "do_generate_prebuilt"
do_generate_prebuilt[dirs] = "${D}"
do_generate_prebuilt[cleandirs] = "${PREBUILT_DIR}"
do_generate_prebuilt[sstate-inputdirs] = "${PREBUILT_DIR}"
do_generate_prebuilt[sstate-outputdirs] = "${DEPLOY_DIR_PREBUILT}"
do_generate_prebuilt[stamp-extra-info] = "${MACHINE_ARCH}"
do_generate_prebuilt[doc] = "Create a tarball of prepopulated Destination directory"

do_generate_prebuilt() {
    cp -aPr ${D} ${PREBUILT_DIR}/${PN}

    if [ ${PREBUILT_STRIP} = "1" ]; then
        # TODO fine grained call to strip
        find ${PREBUILT_DIR}/${PN} -type f -exec ${STRIP} {} + || true
    fi

    tar --remove-files --owner 0 --group 0 -czvf ${PREBUILT_DST} -C ${PREBUILT_DIR}/${PN} . || [ $? -eq 1 ]
}

python do_generate_prebuilt_setscene() {
    sstate_setscene(d)
}

# In case of prebuilt usage, these tasks are discarded
PREBUILT_BYPASS_TASKS += "\
    do_fetch \
    do_unpack \
    do_patch \
    do_configure \
    do_compile \
    do_install \
    do_populate_lic \
"

python () {
    if d.getVar('PREBUILT_SRC_DIR') and os.path.isfile(d.getVar('PREBUILT_SRC')):
        # use prebuilt package
        for task in d.getVar('PREBUILT_BYPASS_TASKS').split():
            d.setVarFlag(task, 'noexec', '1')
        bb.build.addtask('do_install_prebuilt', 'do_populate_sysroot', 'do_install', d)
    elif d.getVar('DEPLOY_DIR_PREBUILT'):
        # generate prebuilt package
        bb.build.addtask('do_generate_prebuilt', 'do_rm_work', 'do_install', d)
        bb.build.addtask('do_generate_prebuilt_setscene', None, None, d)
}

Re: Automatic prebuilt management

[Andre McCurdy]

On Mon, Aug 5, 2019 at 9:06 AM Loic Poulain <loic.poulain@linaro.org> wrote:
> Say a company works with an OE internal tree, with open-source and closed-source packages. The company wants to release the tree so that its customers can fully customize the images/distro. The closed sources being only available internally, the company has to generate and handle prebuilt binaries for proprietary packages. Ideally, with a unified public/internal OE tree (same recipes for internal/public build).
>
> This question is actually similar to an older thread:
> https://marc.info/?l=openembedded-core&m=146779329804683

That very old thread was also referenced recently in a follow up where
I shared a later solution, see:

  http://lists.openembedded.org/pipermail/openembedded-core/2019-July/284896.html

> I started to work on this and added a 'generate-prebuilt' class which generates a tarball of ${D} in deploy/prebuilts after do_install task. Symmetrically, It's also possible to create an 'install-prebuilt' class which bypasses do_fetch, do_unpack, ..., do_install with noexec flag and instead uncompresses a previously generated prebuilt tarball into ${D} and continue normally. But I would like something smarter, e.g. first trying to check if the SRC_URI is available, if not switching on using the prebuilt package if available (e.g. in a DIR_PREBUILT)...
>
> Before going further is there already an existing solution for that? do you have any recommendation on the easier/best way to achieve this?

I did initially try the approach of having a single recipe which can
automatically support both building from source and extracting a
prebuilts tar file, but that (for me anyway) turned out to be a dead
end. Building from source requires build dependencies and config
options but extracting a prebuilt tar file does not, so the two end up
sharing very little... so 90% of the recipe ends up being conditional
on which mode it's running in. The solution I ended up with (see link
above) was for the class which creates the prebuilt tar file to also
create a dedicated mini recipe to extract it.

From my experience however an equally hard part of the problem is the
distribution of prebuilts. It starts off easy (you share a tar file
via email or an ftp site and the receiver manually copies to their
downloads directory...) but that doesn't scale if you need to make
regular updates. My solution is discussed a little more in the thread
linked to above.

Re: Automatic prebuilt management

[Loic Poulain]

[-- Attachment #1: Type: text/plain, Size: 3031 bytes --]

HI Andre,

On Fri, 23 Aug 2019 at 20:11, Andre McCurdy <armccurdy@gmail.com> wrote:

> On Mon, Aug 5, 2019 at 9:06 AM Loic Poulain <loic.poulain@linaro.org>
> wrote:
> > Say a company works with an OE internal tree, with open-source and
> closed-source packages. The company wants to release the tree so that its
> customers can fully customize the images/distro. The closed sources being
> only available internally, the company has to generate and handle prebuilt
> binaries for proprietary packages. Ideally, with a unified public/internal
> OE tree (same recipes for internal/public build).
> >
> > This question is actually similar to an older thread:
> > https://marc.info/?l=openembedded-core&m=146779329804683
>
> That very old thread was also referenced recently in a follow up where
> I shared a later solution, see:
>
>
> http://lists.openembedded.org/pipermail/openembedded-core/2019-July/284896.html


Thanks, I missed that one.


>
> > I started to work on this and added a 'generate-prebuilt' class which
> generates a tarball of ${D} in deploy/prebuilts after do_install task.
> Symmetrically, It's also possible to create an 'install-prebuilt' class
> which bypasses do_fetch, do_unpack, ..., do_install with noexec flag and
> instead uncompresses a previously generated prebuilt tarball into ${D} and
> continue normally. But I would like something smarter, e.g. first trying to
> check if the SRC_URI is available, if not switching on using the prebuilt
> package if available (e.g. in a DIR_PREBUILT)...
> >
> > Before going further is there already an existing solution for that? do
> you have any recommendation on the easier/best way to achieve this?
>
> I did initially try the approach of having a single recipe which can
> automatically support both building from source and extracting a
> prebuilts tar file, but that (for me anyway) turned out to be a dead
> end. Building from source requires build dependencies and config
> options but extracting a prebuilt tar file does not, so the two end up
> sharing very little... so 90% of the recipe ends up being conditional
> on which mode it's running in. The solution I ended up with (see link
> above) was for the class which creates the prebuilt tar file to also
> create a dedicated mini recipe to extract it.
>

> From my experience however an equally hard part of the problem is the
> distribution of prebuilts. It starts off easy (you share a tar file
> via email or an ftp site and the receiver manually copies to their
> downloads directory...) but that doesn't scale if you need to make
> regular updates.


Very good point. maybe having binaries also in a repo could help a bit
(e.g. using git LFS).


> My solution is discussed a little more in the thread
> linked to above.
>

Your solution is interesting and I'll probably jump on the thread with more
questions.
Having an upstream solution part of oe-core would be nice though.

Regards,
Loic

That's a very good point

[-- Attachment #2: Type: text/html, Size: 4212 bytes --]

Re: Automatic prebuilt management

[Richard Purdie]

On Fri, 2019-08-30 at 18:27 +0200, Loic Poulain wrote:
> HI Andre,
> 
> On Fri, 23 Aug 2019 at 20:11, Andre McCurdy <armccurdy@gmail.com>
> wrote:
> > On Mon, Aug 5, 2019 at 9:06 AM Loic Poulain <
> > loic.poulain@linaro.org> wrote:
> > > Say a company works with an OE internal tree, with open-source
> > and closed-source packages. The company wants to release the tree
> > so that its customers can fully customize the images/distro. The
> > closed sources being only available internally, the company has to
> > generate and handle prebuilt binaries for proprietary packages.
> > Ideally, with a unified public/internal OE tree (same recipes for
> > internal/public build).
> > >
> > > This question is actually similar to an older thread:
> > > https://marc.info/?l=openembedded-core&m=146779329804683
> > 
> > That very old thread was also referenced recently in a follow up
> > where
> > I shared a later solution, see:
> > 
> >   
> > http://lists.openembedded.org/pipermail/openembedded-core/2019-July/284896.html
> 
> Thanks, I missed that one.
> 
> > 
> > > I started to work on this and added a 'generate-prebuilt' class
> > which generates a tarball of ${D} in deploy/prebuilts after
> > do_install task. Symmetrically, It's also possible to create an
> > 'install-prebuilt' class which bypasses do_fetch, do_unpack, ...,
> > do_install with noexec flag and instead uncompresses a previously
> > generated prebuilt tarball into ${D} and continue normally. But I
> > would like something smarter, e.g. first trying to check if the
> > SRC_URI is available, if not switching on using the prebuilt
> > package if available (e.g. in a DIR_PREBUILT)...
> > >
> > > Before going further is there already an existing solution for
> > that? do you have any recommendation on the easier/best way to
> > achieve this?
> > 
> > I did initially try the approach of having a single recipe which
> > can
> > automatically support both building from source and extracting a
> > prebuilts tar file, but that (for me anyway) turned out to be a
> > dead
> > end. Building from source requires build dependencies and config
> > options but extracting a prebuilt tar file does not, so the two end
> > up
> > sharing very little... so 90% of the recipe ends up being
> > conditional
> > on which mode it's running in. The solution I ended up with (see
> > link
> > above) was for the class which creates the prebuilt tar file to
> > also
> > create a dedicated mini recipe to extract it. 
> > From my experience however an equally hard part of the problem is
> > the
> > distribution of prebuilts. It starts off easy (you share a tar file
> > via email or an ftp site and the receiver manually copies to their
> > downloads directory...) but that doesn't scale if you need to make
> > regular updates.
> 
> Very good point. maybe having binaries also in a repo could help a
> bit (e.g. using git LFS).
>  
> > My solution is discussed a little more in the thread
> > linked to above.
> 
> Your solution is interesting and I'll probably jump on the thread
> with more questions.
> Having an upstream solution part of oe-core would be nice though.

We'd be happy to have one, it will take someone to propose something
suitably general purpose that others can use along with tests.

Cheers,

Richard