* Automatic prebuilt management @ 2019-08-05 16:06 Loic Poulain 2019-08-05 16:39 ` chris.laplante 2019-08-23 18:11 ` Andre McCurdy 0 siblings, 2 replies; 6+ messages in thread From: Loic Poulain @ 2019-08-05 16:06 UTC (permalink / raw) To: openembedded-core [-- Attachment #1: Type: text/plain, Size: 1278 bytes --] Hello, Say a company works with an OE internal tree, with open-source and closed-source packages. The company wants to release the tree so that its customers can fully customize the images/distro. The closed sources being only available internally, the company has to generate and handle prebuilt binaries for proprietary packages. Ideally, with a unified public/internal OE tree (same recipes for internal/public build). This question is actually similar to an older thread: https://marc.info/?l=openembedded-core&m=146779329804683 I started to work on this and added a 'generate-prebuilt' class which generates a tarball of ${D} in deploy/prebuilts after do_install task. Symmetrically, It's also possible to create an 'install-prebuilt' class which bypasses do_fetch, do_unpack, ..., do_install with noexec flag and instead uncompresses a previously generated prebuilt tarball into ${D} and continue normally. But I would like something smarter, e.g. first trying to check if the SRC_URI is available, if not switching on using the prebuilt package if available (e.g. in a DIR_PREBUILT)... Before going further is there already an existing solution for that? do you have any recommendation on the easier/best way to achieve this? Regards, Loic [-- Attachment #2: Type: text/html, Size: 1579 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Automatic prebuilt management 2019-08-05 16:06 Automatic prebuilt management Loic Poulain @ 2019-08-05 16:39 ` chris.laplante 2019-08-23 9:49 ` Loic Poulain 2019-08-23 18:11 ` Andre McCurdy 1 sibling, 1 reply; 6+ messages in thread From: chris.laplante @ 2019-08-05 16:39 UTC (permalink / raw) To: Loic Poulain, openembedded-core > But I would like something smarter, e.g. first trying to check if the SRC_URI is available, if not switching on using the prebuilt package if available (e.g. > in a DIR_PREBUILT)... > > Before going further is there already an existing solution for that? do you have any recommendation on the easier/best way to > achieve this? I don't have a "best practice" recommendation to offer. But coincidentally I suggested something similar for upstreaming a few weeks ago. It was (rightfully) rejected as too "hacky" for inclusion in core code. Regardless, here's a bbclass we currently use: addhandler skip_eventhandler python skip_eventhandler() { try: bb.fetch2.Fetch(d.getVar("SRC_URI").split(), d) except bb.fetch2.FetchError: raise bb.parse.SkipRecipe("skip-inaccessible: could not access upstream repo; check SRC_URI, access rights, and network availability") } skip_eventhandler[eventmask] = "bb.event.RecipePreFinalise" As was pointed out to me, this is a pretty heavy-handed approach - it can fail for any number of reasons, not just the URI actually being unavailable. So you may consider a more surgical check, e.g. actually trying to fetch the SRC_URI directly using raw GET requests. To implement the handover, you'd have separate recipes for the build-from-source and the prebuilt case which PROVIDE the same package. You probably also want to set PREFERRED_PROVIDER in your layer conf to prefer the build-from-source recipe. Alternatively, you can do it all in a single recipe by rewriting the SRC_URI from within the event handler. But I think separate recipes is nicer. Chris ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Automatic prebuilt management 2019-08-05 16:39 ` chris.laplante @ 2019-08-23 9:49 ` Loic Poulain 0 siblings, 0 replies; 6+ messages in thread From: Loic Poulain @ 2019-08-23 9:49 UTC (permalink / raw) To: chris.laplante; +Cc: openembedded-core [-- Attachment #1.1: Type: text/plain, Size: 2350 bytes --] On Mon, 5 Aug 2019 at 18:39, chris.laplante@agilent.com < chris.laplante@agilent.com> wrote: > > But I would like something smarter, e.g. first trying to check if the > SRC_URI is available, if not switching on using the prebuilt package if > available (e.g. > > in a DIR_PREBUILT)... > > > > Before going further is there already an existing solution for that? do > you have any recommendation on the easier/best way to > > achieve this? > > I don't have a "best practice" recommendation to offer. But coincidentally > I suggested something similar for upstreaming a few weeks ago. It was > (rightfully) rejected as too "hacky" for inclusion in core code. > Regardless, here's a bbclass we currently use: > > addhandler skip_eventhandler > python skip_eventhandler() { > try: > bb.fetch2.Fetch(d.getVar("SRC_URI").split(), d) > except bb.fetch2.FetchError: > raise bb.parse.SkipRecipe("skip-inaccessible: could not access > upstream repo; check SRC_URI, access rights, and network availability") > } > skip_eventhandler[eventmask] = "bb.event.RecipePreFinalise" > > > As was pointed out to me, this is a pretty heavy-handed approach - it can > fail for any number of reasons, not just the URI actually being > unavailable. So you may consider a more surgical check, e.g. actually > trying to fetch the SRC_URI directly using raw GET requests. > > To implement the handover, you'd have separate recipes for the > build-from-source and the prebuilt case which PROVIDE the same package. You > probably also want to set PREFERRED_PROVIDER in your layer conf to prefer > the build-from-source recipe. > > Alternatively, you can do it all in a single recipe by rewriting the > SRC_URI from within the event handler. But I think separate recipes is > nicer. > Well, I ended up with the attached class, which generates or install prebuilt tarball. It's only a draft and I'll come back with a polished version, but the idea is to look for the related prebuilt package into a 'PREBUILT_SRC_DIR' (if defined) and install it instead of fetching and compiling from source. This allows having only one recipe. Then a company just needs to provide closed-source software as a set of prebuilt tarballs and customer need to point to the prebuilt dir (e.g. in local.conf). Regards, Loic [-- Attachment #1.2: Type: text/html, Size: 2815 bytes --] [-- Attachment #2: prebuilt.bbclass --] [-- Type: application/octet-stream, Size: 2773 bytes --] # This class allows to: # - Generate a prebuilt tarball of a package # - Use a previsouly generated prebuilt tarball instead of # fetching and compiling from source, when PREBUILT_SRC_DIR # is defined. # # A package has to inherits 'prebuilt' to generate a prebuilt tarball. # Add PREBUILT_STRIP = "1" to generate stripped prebuilt (default is 0). # To use prebuilt, define PREBUILT_SRC_DIR (e.g. in conf/local.conf): # PREBUILT_SRC_DIR = "/home/vendor/prebuilts" # # TODO License management # Anonymous function needs to be executed each time so that runqueue # can be updated BB_DONT_CACHE = "1" DEPLOY_DIR_PREBUILT ?= "${DEPLOY_DIR}/prebuilts" PREBUILT_DIR="${WORKDIR}/prebuilt" PREBUILT_BASENAME="${PN}_${PV}_prebuilt_${PACKAGE_ARCH}.tar.gz" PREBUILT_DST="${PREBUILT_DIR}/${PREBUILT_BASENAME}" PREBUILT_SRC="${PREBUILT_SRC_DIR}/${PREBUILT_BASENAME}" PREBUILT_STRIP ?= "0" # Install Prebuilt tarball do_install_prebuilt[dirs] = "${D}" do_install_prebuilt[doc] = "Populate Destination directory with prebuilt tarball content" fakeroot do_install_prebuilt() { tar -xvzf ${PREBUILT_SRC} -C ${D} } # Generate Prebuilt tarball DEPLOY_DIR_PREBUILT = "${DEPLOY_DIR}/prebuilts" SSTATETASKS += "do_generate_prebuilt" do_generate_prebuilt[dirs] = "${D}" do_generate_prebuilt[cleandirs] = "${PREBUILT_DIR}" do_generate_prebuilt[sstate-inputdirs] = "${PREBUILT_DIR}" do_generate_prebuilt[sstate-outputdirs] = "${DEPLOY_DIR_PREBUILT}" do_generate_prebuilt[stamp-extra-info] = "${MACHINE_ARCH}" do_generate_prebuilt[doc] = "Create a tarball of prepopulated Destination directory" do_generate_prebuilt() { cp -aPr ${D} ${PREBUILT_DIR}/${PN} if [ ${PREBUILT_STRIP} = "1" ]; then # TODO fine grained call to strip find ${PREBUILT_DIR}/${PN} -type f -exec ${STRIP} {} + || true fi tar --remove-files --owner 0 --group 0 -czvf ${PREBUILT_DST} -C ${PREBUILT_DIR}/${PN} . || [ $? -eq 1 ] } python do_generate_prebuilt_setscene() { sstate_setscene(d) } # In case of prebuilt usage, these tasks are discarded PREBUILT_BYPASS_TASKS += "\ do_fetch \ do_unpack \ do_patch \ do_configure \ do_compile \ do_install \ do_populate_lic \ " python () { if d.getVar('PREBUILT_SRC_DIR') and os.path.isfile(d.getVar('PREBUILT_SRC')): # use prebuilt package for task in d.getVar('PREBUILT_BYPASS_TASKS').split(): d.setVarFlag(task, 'noexec', '1') bb.build.addtask('do_install_prebuilt', 'do_populate_sysroot', 'do_install', d) elif d.getVar('DEPLOY_DIR_PREBUILT'): # generate prebuilt package bb.build.addtask('do_generate_prebuilt', 'do_rm_work', 'do_install', d) bb.build.addtask('do_generate_prebuilt_setscene', None, None, d) } ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Automatic prebuilt management 2019-08-05 16:06 Automatic prebuilt management Loic Poulain 2019-08-05 16:39 ` chris.laplante @ 2019-08-23 18:11 ` Andre McCurdy 2019-08-30 16:27 ` Loic Poulain 1 sibling, 1 reply; 6+ messages in thread From: Andre McCurdy @ 2019-08-23 18:11 UTC (permalink / raw) To: Loic Poulain; +Cc: OE Core mailing list On Mon, Aug 5, 2019 at 9:06 AM Loic Poulain <loic.poulain@linaro.org> wrote: > Say a company works with an OE internal tree, with open-source and closed-source packages. The company wants to release the tree so that its customers can fully customize the images/distro. The closed sources being only available internally, the company has to generate and handle prebuilt binaries for proprietary packages. Ideally, with a unified public/internal OE tree (same recipes for internal/public build). > > This question is actually similar to an older thread: > https://marc.info/?l=openembedded-core&m=146779329804683 That very old thread was also referenced recently in a follow up where I shared a later solution, see: http://lists.openembedded.org/pipermail/openembedded-core/2019-July/284896.html > I started to work on this and added a 'generate-prebuilt' class which generates a tarball of ${D} in deploy/prebuilts after do_install task. Symmetrically, It's also possible to create an 'install-prebuilt' class which bypasses do_fetch, do_unpack, ..., do_install with noexec flag and instead uncompresses a previously generated prebuilt tarball into ${D} and continue normally. But I would like something smarter, e.g. first trying to check if the SRC_URI is available, if not switching on using the prebuilt package if available (e.g. in a DIR_PREBUILT)... > > Before going further is there already an existing solution for that? do you have any recommendation on the easier/best way to achieve this? I did initially try the approach of having a single recipe which can automatically support both building from source and extracting a prebuilts tar file, but that (for me anyway) turned out to be a dead end. Building from source requires build dependencies and config options but extracting a prebuilt tar file does not, so the two end up sharing very little... so 90% of the recipe ends up being conditional on which mode it's running in. The solution I ended up with (see link above) was for the class which creates the prebuilt tar file to also create a dedicated mini recipe to extract it. From my experience however an equally hard part of the problem is the distribution of prebuilts. It starts off easy (you share a tar file via email or an ftp site and the receiver manually copies to their downloads directory...) but that doesn't scale if you need to make regular updates. My solution is discussed a little more in the thread linked to above. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Automatic prebuilt management 2019-08-23 18:11 ` Andre McCurdy @ 2019-08-30 16:27 ` Loic Poulain 2019-08-30 16:45 ` Richard Purdie 0 siblings, 1 reply; 6+ messages in thread From: Loic Poulain @ 2019-08-30 16:27 UTC (permalink / raw) To: Andre McCurdy; +Cc: OE Core mailing list [-- Attachment #1: Type: text/plain, Size: 3031 bytes --] HI Andre, On Fri, 23 Aug 2019 at 20:11, Andre McCurdy <armccurdy@gmail.com> wrote: > On Mon, Aug 5, 2019 at 9:06 AM Loic Poulain <loic.poulain@linaro.org> > wrote: > > Say a company works with an OE internal tree, with open-source and > closed-source packages. The company wants to release the tree so that its > customers can fully customize the images/distro. The closed sources being > only available internally, the company has to generate and handle prebuilt > binaries for proprietary packages. Ideally, with a unified public/internal > OE tree (same recipes for internal/public build). > > > > This question is actually similar to an older thread: > > https://marc.info/?l=openembedded-core&m=146779329804683 > > That very old thread was also referenced recently in a follow up where > I shared a later solution, see: > > > http://lists.openembedded.org/pipermail/openembedded-core/2019-July/284896.html Thanks, I missed that one. > > > I started to work on this and added a 'generate-prebuilt' class which > generates a tarball of ${D} in deploy/prebuilts after do_install task. > Symmetrically, It's also possible to create an 'install-prebuilt' class > which bypasses do_fetch, do_unpack, ..., do_install with noexec flag and > instead uncompresses a previously generated prebuilt tarball into ${D} and > continue normally. But I would like something smarter, e.g. first trying to > check if the SRC_URI is available, if not switching on using the prebuilt > package if available (e.g. in a DIR_PREBUILT)... > > > > Before going further is there already an existing solution for that? do > you have any recommendation on the easier/best way to achieve this? > > I did initially try the approach of having a single recipe which can > automatically support both building from source and extracting a > prebuilts tar file, but that (for me anyway) turned out to be a dead > end. Building from source requires build dependencies and config > options but extracting a prebuilt tar file does not, so the two end up > sharing very little... so 90% of the recipe ends up being conditional > on which mode it's running in. The solution I ended up with (see link > above) was for the class which creates the prebuilt tar file to also > create a dedicated mini recipe to extract it. > > From my experience however an equally hard part of the problem is the > distribution of prebuilts. It starts off easy (you share a tar file > via email or an ftp site and the receiver manually copies to their > downloads directory...) but that doesn't scale if you need to make > regular updates. Very good point. maybe having binaries also in a repo could help a bit (e.g. using git LFS). > My solution is discussed a little more in the thread > linked to above. > Your solution is interesting and I'll probably jump on the thread with more questions. Having an upstream solution part of oe-core would be nice though. Regards, Loic That's a very good point [-- Attachment #2: Type: text/html, Size: 4212 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Automatic prebuilt management 2019-08-30 16:27 ` Loic Poulain @ 2019-08-30 16:45 ` Richard Purdie 0 siblings, 0 replies; 6+ messages in thread From: Richard Purdie @ 2019-08-30 16:45 UTC (permalink / raw) To: Loic Poulain, Andre McCurdy; +Cc: OE Core mailing list On Fri, 2019-08-30 at 18:27 +0200, Loic Poulain wrote: > HI Andre, > > On Fri, 23 Aug 2019 at 20:11, Andre McCurdy <armccurdy@gmail.com> > wrote: > > On Mon, Aug 5, 2019 at 9:06 AM Loic Poulain < > > loic.poulain@linaro.org> wrote: > > > Say a company works with an OE internal tree, with open-source > > and closed-source packages. The company wants to release the tree > > so that its customers can fully customize the images/distro. The > > closed sources being only available internally, the company has to > > generate and handle prebuilt binaries for proprietary packages. > > Ideally, with a unified public/internal OE tree (same recipes for > > internal/public build). > > > > > > This question is actually similar to an older thread: > > > https://marc.info/?l=openembedded-core&m=146779329804683 > > > > That very old thread was also referenced recently in a follow up > > where > > I shared a later solution, see: > > > > > > http://lists.openembedded.org/pipermail/openembedded-core/2019-July/284896.html > > Thanks, I missed that one. > > > > > > I started to work on this and added a 'generate-prebuilt' class > > which generates a tarball of ${D} in deploy/prebuilts after > > do_install task. Symmetrically, It's also possible to create an > > 'install-prebuilt' class which bypasses do_fetch, do_unpack, ..., > > do_install with noexec flag and instead uncompresses a previously > > generated prebuilt tarball into ${D} and continue normally. But I > > would like something smarter, e.g. first trying to check if the > > SRC_URI is available, if not switching on using the prebuilt > > package if available (e.g. in a DIR_PREBUILT)... > > > > > > Before going further is there already an existing solution for > > that? do you have any recommendation on the easier/best way to > > achieve this? > > > > I did initially try the approach of having a single recipe which > > can > > automatically support both building from source and extracting a > > prebuilts tar file, but that (for me anyway) turned out to be a > > dead > > end. Building from source requires build dependencies and config > > options but extracting a prebuilt tar file does not, so the two end > > up > > sharing very little... so 90% of the recipe ends up being > > conditional > > on which mode it's running in. The solution I ended up with (see > > link > > above) was for the class which creates the prebuilt tar file to > > also > > create a dedicated mini recipe to extract it. > > From my experience however an equally hard part of the problem is > > the > > distribution of prebuilts. It starts off easy (you share a tar file > > via email or an ftp site and the receiver manually copies to their > > downloads directory...) but that doesn't scale if you need to make > > regular updates. > > Very good point. maybe having binaries also in a repo could help a > bit (e.g. using git LFS). > > > My solution is discussed a little more in the thread > > linked to above. > > Your solution is interesting and I'll probably jump on the thread > with more questions. > Having an upstream solution part of oe-core would be nice though. We'd be happy to have one, it will take someone to propose something suitably general purpose that others can use along with tests. Cheers, Richard ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2019-08-30 16:45 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-08-05 16:06 Automatic prebuilt management Loic Poulain 2019-08-05 16:39 ` chris.laplante 2019-08-23 9:49 ` Loic Poulain 2019-08-23 18:11 ` Andre McCurdy 2019-08-30 16:27 ` Loic Poulain 2019-08-30 16:45 ` Richard Purdie
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.