From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-f51.google.com (mail-ed1-f51.google.com [209.85.208.51]) by mail.openembedded.org (Postfix) with ESMTP id 486747EC92 for ; Fri, 30 Aug 2019 16:26:48 +0000 (UTC) Received: by mail-ed1-f51.google.com with SMTP id u6so2729991edq.6 for ; Fri, 30 Aug 2019 09:26:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=UI0nE8QX2rWtScTwG6J9AxDON3Xy4eHo8yCw/25Wk+A=; b=BnJEdn1gOVRx7DbFRZ1Q0MdAN3F9ABiJJQNVmOPpWqO77G7o+9WqFDIpPdSjER6Odq CoQiKBlK8NqA9vQcnl3w699JoF8ZP8ni2IvF54SOFVziJ1/2VK8tfMXO1gJM2nDHJCWU NOXFVQLR1Nr2ED1La7koV6VL1yseo0hbklYpnIapjXO3E7Q92ZqXuZ7NtbD6zKjkrpAr vUUjrJ3UtavM5z3bBdLDz+aD3VBICSV0/t6S7+fekyHUZZUKLSS2fORx2Agwxcc/UlEZ BPZF+gssxcDWNSquFjjzUTCZmR1z1T4RrlKtI696+A6E3H9GLg01iO8SiSviSJ5A9v6d j+8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=UI0nE8QX2rWtScTwG6J9AxDON3Xy4eHo8yCw/25Wk+A=; b=uQiDO6DMPVV2NBVKhIS0aFrXlPdyLdMYNVAM6W3zXvxJiEliYzrC1FJAMRd1AhwJgl 5u92h3ADUIsqkQ8ZD6DoEMwVmB/pTzdi5u4pVlYGZM90BUPeu3QP9xZAY5R98ZSiIotU D4fvWjQTLI06FgkLsR7OwZB4WBsEoVWYUFVaj9g6a62RjydyKvUD/WXEGdk8Jh9mn+h9 Gm7Lf3bTs6nTm2HljiqvF56rrYLcRf7uokCXS5GvakMTUpiPUGYROtk8rGJsHUqbhnw5 DfZGftGS7znZ1LiHGGAaDLERlxoauv+mDCgMOLr2Nhi4+O50jDekjJ4Fe/GPtZmpX9Ju nBtA== X-Gm-Message-State: APjAAAVaJ/4NwasK/ZxKGYcuHwWo6JF29SDw/mNqwLSXCOBK/JAv4vqB lzVf6opjgiHvsKENx0MnfGAl493w/EuPOHrS6BfEgA== X-Google-Smtp-Source: APXvYqwvLYxIpHt2n1wucJTQG3pf/g4tO/8yPBQx2mPoGiQo2rVcauwdH+l4xPNqLN1J3IaZleX36gyrzPIaFmcEN60= X-Received: by 2002:a17:906:1493:: with SMTP id x19mr7872714ejc.39.1567182408894; Fri, 30 Aug 2019 09:26:48 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Loic Poulain Date: Fri, 30 Aug 2019 18:27:27 +0200 Message-ID: To: Andre McCurdy Cc: OE Core mailing list Subject: Re: Automatic prebuilt management X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Aug 2019 16:26:48 -0000 Content-Type: multipart/alternative; boundary="000000000000c7df670591581673" --000000000000c7df670591581673 Content-Type: text/plain; charset="UTF-8" HI Andre, On Fri, 23 Aug 2019 at 20:11, Andre McCurdy wrote: > On Mon, Aug 5, 2019 at 9:06 AM Loic Poulain > wrote: > > Say a company works with an OE internal tree, with open-source and > closed-source packages. The company wants to release the tree so that its > customers can fully customize the images/distro. The closed sources being > only available internally, the company has to generate and handle prebuilt > binaries for proprietary packages. Ideally, with a unified public/internal > OE tree (same recipes for internal/public build). > > > > This question is actually similar to an older thread: > > https://marc.info/?l=openembedded-core&m=146779329804683 > > That very old thread was also referenced recently in a follow up where > I shared a later solution, see: > > > http://lists.openembedded.org/pipermail/openembedded-core/2019-July/284896.html Thanks, I missed that one. > > > I started to work on this and added a 'generate-prebuilt' class which > generates a tarball of ${D} in deploy/prebuilts after do_install task. > Symmetrically, It's also possible to create an 'install-prebuilt' class > which bypasses do_fetch, do_unpack, ..., do_install with noexec flag and > instead uncompresses a previously generated prebuilt tarball into ${D} and > continue normally. But I would like something smarter, e.g. first trying to > check if the SRC_URI is available, if not switching on using the prebuilt > package if available (e.g. in a DIR_PREBUILT)... > > > > Before going further is there already an existing solution for that? do > you have any recommendation on the easier/best way to achieve this? > > I did initially try the approach of having a single recipe which can > automatically support both building from source and extracting a > prebuilts tar file, but that (for me anyway) turned out to be a dead > end. Building from source requires build dependencies and config > options but extracting a prebuilt tar file does not, so the two end up > sharing very little... so 90% of the recipe ends up being conditional > on which mode it's running in. The solution I ended up with (see link > above) was for the class which creates the prebuilt tar file to also > create a dedicated mini recipe to extract it. > > From my experience however an equally hard part of the problem is the > distribution of prebuilts. It starts off easy (you share a tar file > via email or an ftp site and the receiver manually copies to their > downloads directory...) but that doesn't scale if you need to make > regular updates. Very good point. maybe having binaries also in a repo could help a bit (e.g. using git LFS). > My solution is discussed a little more in the thread > linked to above. > Your solution is interesting and I'll probably jump on the thread with more questions. Having an upstream solution part of oe-core would be nice though. Regards, Loic That's a very good point --000000000000c7df670591581673 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
HI Andre,

On Fri, 23 Aug 2019 at 20:11, Andre McCu= rdy <armccurdy@= gmail.com> wrote:
On Mon, Aug 5, 2019 at 9:06 AM Loic Poulain <loic.poulain@linaro.org>= ; wrote:
> Say a company works with an OE internal tree, with open-so= urce and closed-source packages. The company wants to release the tree so t= hat its customers can fully customize the images/distro. The closed sources= being only available internally, the company has to generate and handle pr= ebuilt binaries for proprietary packages. Ideally, with a unified public/in= ternal OE tree (same recipes for internal/public build).
>
> Th= is question is actually similar to an older thread:
> https://marc.info/?l=3Dopenembedded-core&m=3D14= 6779329804683

That very old thread was also referenced recently in a follow up where<= br>I shared a later solution, see:

=C2=A0 http://lis= ts.openembedded.org/pipermail/openembedded-core/2019-July/284896.html

Thanks, I missed that one.



> I started to work on this and added a 'generate-prebuilt' = class which generates a tarball of ${D} in deploy/prebuilts after do_instal= l task. Symmetrically, It's also possible to create an 'install-pre= built' class which bypasses do_fetch, do_unpack, ..., do_install with n= oexec flag and instead uncompresses a previously generated prebuilt tarball= into ${D} and continue normally. But I would like something smarter, e.g. = first trying to check if the SRC_URI is available, if not switching on usin= g the prebuilt package if available (e.g. in a DIR_PREBUILT)...
>
= > Before going further is there already an existing solution for that? d= o you have any recommendation on the easier/best way to achieve this?

I did initially try the approach of having a single recipe which canautomatically support both building from source and extracting a
prebui= lts tar file, but that (for me anyway) turned out to be a dead
end. Buil= ding from source requires build dependencies and config
options but extr= acting a prebuilt tar file does not, so the two end up
sharing very litt= le... so 90% of the recipe ends up being conditional
on which mode it= 9;s running in. The solution I ended up with (see link
above) was for th= e class which creates the prebuilt tar file to also
create a dedicated m= ini recipe to extract it.

From my experience however an equally hard part of the problem is thedistribution of prebuilts. It starts off easy (you share a tar file
vi= a email or an ftp site and the receiver manually copies to their
downloa= ds directory...) but that doesn't scale if you need to make
regular = updates.

Very good point. maybe having bina= ries also in a repo could help a bit (e.g. using git LFS).
=C2=A0=
My solution is disc= ussed a little more in the thread
linked to above.
=
Your solution is interesting and I'll probably jump on t= he thread with more questions.
Having an upstream solution part o= f oe-core would be nice though.

Regards,
Loic

That's a very good point
--000000000000c7df670591581673--