Re: Automatic prebuilt management

[Loic Poulain <loic.poulain@linaro.org>]

[-- Attachment #1.1: Type: text/plain, Size: 2350 bytes --]

On Mon, 5 Aug 2019 at 18:39, chris.laplante@agilent.com <
chris.laplante@agilent.com> wrote:

> > But I would like something smarter, e.g. first trying to check if the
> SRC_URI is available, if not switching on using the prebuilt package if
> available (e.g.
> > in a DIR_PREBUILT)...
> >
> > Before going further is there already an existing solution for that? do
> you have any recommendation on the easier/best way to
> > achieve this?
>
> I don't have a "best practice" recommendation to offer. But coincidentally
> I suggested something similar for upstreaming a few weeks ago. It was
> (rightfully) rejected as too "hacky" for inclusion in core code.
> Regardless, here's a bbclass we currently use:
>
> addhandler skip_eventhandler
> python skip_eventhandler() {
>     try:
>         bb.fetch2.Fetch(d.getVar("SRC_URI").split(), d)
>     except bb.fetch2.FetchError:
>         raise bb.parse.SkipRecipe("skip-inaccessible: could not access
> upstream repo; check SRC_URI, access rights, and network availability")
> }
> skip_eventhandler[eventmask] = "bb.event.RecipePreFinalise"
>
>
> As was pointed out to me, this is a pretty heavy-handed approach - it can
> fail for any number of reasons, not just the URI actually being
> unavailable. So you may consider a more surgical check, e.g. actually
> trying to fetch the SRC_URI directly using raw GET requests.
>
> To implement the handover, you'd have separate recipes for the
> build-from-source and the prebuilt case which PROVIDE the same package. You
> probably also want to set PREFERRED_PROVIDER in your layer conf to prefer
> the build-from-source recipe.
>
> Alternatively, you can do it all in a single recipe by rewriting the
> SRC_URI from within the event handler. But I think separate recipes is
> nicer.
>

Well, I ended up with the attached class, which generates or install
prebuilt tarball. It's only a draft and I'll come back with a polished
version, but the idea is to look for the related prebuilt package into a
'PREBUILT_SRC_DIR' (if defined) and install it instead of fetching and
compiling from source. This allows having only one recipe. Then a company
just needs to provide closed-source software as a set of prebuilt tarballs
and customer need to point to the prebuilt dir (e.g. in local.conf).

Regards,
Loic

[-- Attachment #1.2: Type: text/html, Size: 2815 bytes --]

[-- Attachment #2: prebuilt.bbclass --]
[-- Type: application/octet-stream, Size: 2773 bytes --]

# This class allows to:
#     - Generate a prebuilt tarball of a package
#     - Use a previsouly generated prebuilt tarball instead of
#       fetching and compiling from source, when PREBUILT_SRC_DIR
#       is defined.
#
# A package has to inherits 'prebuilt' to generate a prebuilt tarball.
# Add PREBUILT_STRIP = "1" to generate stripped prebuilt (default is 0).
# To use prebuilt, define PREBUILT_SRC_DIR (e.g. in conf/local.conf):
#     PREBUILT_SRC_DIR = "/home/vendor/prebuilts"
#
# TODO License management

# Anonymous function needs to be executed each time so that runqueue
# can be updated
BB_DONT_CACHE = "1"

DEPLOY_DIR_PREBUILT ?= "${DEPLOY_DIR}/prebuilts"

PREBUILT_DIR="${WORKDIR}/prebuilt"
PREBUILT_BASENAME="${PN}_${PV}_prebuilt_${PACKAGE_ARCH}.tar.gz"
PREBUILT_DST="${PREBUILT_DIR}/${PREBUILT_BASENAME}"
PREBUILT_SRC="${PREBUILT_SRC_DIR}/${PREBUILT_BASENAME}"

PREBUILT_STRIP ?= "0"

# Install Prebuilt tarball
do_install_prebuilt[dirs] = "${D}"
do_install_prebuilt[doc] = "Populate Destination directory with prebuilt tarball content"

fakeroot do_install_prebuilt() {
    tar -xvzf ${PREBUILT_SRC} -C ${D}
}

# Generate Prebuilt tarball
DEPLOY_DIR_PREBUILT = "${DEPLOY_DIR}/prebuilts"
SSTATETASKS += "do_generate_prebuilt"
do_generate_prebuilt[dirs] = "${D}"
do_generate_prebuilt[cleandirs] = "${PREBUILT_DIR}"
do_generate_prebuilt[sstate-inputdirs] = "${PREBUILT_DIR}"
do_generate_prebuilt[sstate-outputdirs] = "${DEPLOY_DIR_PREBUILT}"
do_generate_prebuilt[stamp-extra-info] = "${MACHINE_ARCH}"
do_generate_prebuilt[doc] = "Create a tarball of prepopulated Destination directory"

do_generate_prebuilt() {
    cp -aPr ${D} ${PREBUILT_DIR}/${PN}

    if [ ${PREBUILT_STRIP} = "1" ]; then
        # TODO fine grained call to strip
        find ${PREBUILT_DIR}/${PN} -type f -exec ${STRIP} {} + || true
    fi

    tar --remove-files --owner 0 --group 0 -czvf ${PREBUILT_DST} -C ${PREBUILT_DIR}/${PN} . || [ $? -eq 1 ]
}

python do_generate_prebuilt_setscene() {
    sstate_setscene(d)
}

# In case of prebuilt usage, these tasks are discarded
PREBUILT_BYPASS_TASKS += "\
    do_fetch \
    do_unpack \
    do_patch \
    do_configure \
    do_compile \
    do_install \
    do_populate_lic \
"

python () {
    if d.getVar('PREBUILT_SRC_DIR') and os.path.isfile(d.getVar('PREBUILT_SRC')):
        # use prebuilt package
        for task in d.getVar('PREBUILT_BYPASS_TASKS').split():
            d.setVarFlag(task, 'noexec', '1')
        bb.build.addtask('do_install_prebuilt', 'do_populate_sysroot', 'do_install', d)
    elif d.getVar('DEPLOY_DIR_PREBUILT'):
        # generate prebuilt package
        bb.build.addtask('do_generate_prebuilt', 'do_rm_work', 'do_install', d)
        bb.build.addtask('do_generate_prebuilt_setscene', None, None, d)
}