Re: [Openembedded-architecture] Sum up - Proposal: dealing with language-specific build tools/dependency management tools

[Mark Hatle <mark.hatle@windriver.com>]

On 3/16/17 4:30 AM, Paul Barker wrote:
> On Thu, 16 Mar 2017 09:17:57 +0100
> Josef Holzmayr <holzmayr@rsi-elektrotechnik.de> wrote:
> 
>> I'll give it a shot and try to sum up the current state of affairs in 
>> this discussion. In accordance to the "Package managers all the way 
>> down"-presentation, I'm gonna use the term LPM (for language package 
>> manager) for now on.
>>
>> *Requirements*
>> - repeatable and verifyable licensing of all dependencies that a LPM 
>> pulls in.
>> - locking down specific versions of packages and their dependencies for 
>> reproductible builds.
> 
> Agreed with these.
> 
>>
>> *Optionals*
>> - strict separation of fetch, compile, install stages. If a specifc LPM 
>> requires it, those might be intermingled or left out [Mark]
> 
> I think what we need is strict separation of fetch (allowed to access
> network) from the later tasks in a build (not allowed to access
> networks). We need to continue to support users behind corporate
> firewalls via mirroring and offline builds.

I agree here.  The 'network fetch', 'do something with it', 'deploy it'  Are
really the three logical steps we have.  The network fetch (as other pieces of
this thread mentioned) could include some specific dependency fetching or
whatever, but in the end this must be able to be mirrored and disable actual
network connections.  (network dependency fetching is possibly very tricky as
dependency scanning in bitbake happens very early, and there is no way to say
"please re-evaluate the dependencies on this branch".  [Maybe this is something
that needs to be considered for this type of work?)

--Mark

>> - opaque packaging: similar to static linking, we should at least have a 
>> way to bundle up a complete application into a single package. Maybe it 
>> might even be the default (like rust does it at the moment).
>> - leverage as much as possible of the functionality the LPMs provide 
>> instead of reimplementing it.
> 
> We should also aim for automatic generation of recipes from pypi,
> npmjs, crates.io, etc using devtool. I'm not sure how much of this has
> already been implemented, I'm not entirely up-to-date on devtool
> functionality.
> 
> Staying with NPM as the example, say we have a tree of 1000
> dependencies for a top-level application recipe. I understand that
> generating and then parsing 1000 recipes will bloat things and slow
> down the build. However, we do need some way to apply patches to those
> dependencies or fix metadata. For example, what if the license is
> stated wrong on npmjs? What if the dependency list itself is wrong? What
> I've always liked about OE is that you can fix this sort of problem in a
> recipe when you need to.
> 
>>
>> *Wish List*
>> - separating out the LPM infrastructure into one or more distinct 
>> layers, not treating it as OE/bitbake core functionality. [Paul]
>> - support for the use of multiple languages/LPMs inside a single recipe, 
>> hopefully even package. [myself]
>>
>> *Proposed Solutions*
>> - having lockdown files shipped with the recipes (in whatever form to be 
>> defined)
>> - leveraging the recipe system to resolve licensing.
>>
>> If we can boil things down to the common set that we all expect, it will 
>> in my opnion serve as a blueprint for the actual implementation to follow.
>>
>> Greetz,
> 
> Thanks,
> Paul
>