* Proposal: dealing with language-specific build tools/dependency management tools @ 2017-03-10 13:49 Alexander Kanavin 2017-03-10 14:30 ` [Openembedded-architecture] " Otavio Salvador ` (3 more replies) 0 siblings, 4 replies; 25+ messages in thread From: Alexander Kanavin @ 2017-03-10 13:49 UTC (permalink / raw) To: openembedded-architecture, Yocto Project Hello all, *Introduction* The new generation of programming languages (think node.js, Go, Rust) is a poor fit for the Yocto build model which follows the traditional Unix model. In particular, those new development environments have no problem with 'grabbing random stuff from the Internet' as a part of development and build process. However, Yocto has very strict rules about the build steps and what they can and can not do, and also a strict enforcement of license and version checks for every component that gets built. Those two models clash, and this is a proposal of how they could be reconciled. I'll also send a separate email that talks specifically about MEAN stack and how it could be supported as Yocto - take it as a specific example for all of the below. *Background* The traditional development model on Unix clearly separates installation of dependencies needed to develop a project from the development process itself. Typically, when one wants to build some project, first the project README needs to be inspected, and any required dependencies installed system-wide using the distribution package management's tool. When those dependencies change, usually this manifests itself in a previously unseen build failure which is again manually resolved by figuring out the missing dependency and installing it. This can be awkward, but it's how things have been done for decades, and Yocto's build system (with separate steps for fetching, unpacking, building, packaging etc.) is built around the assumption that most software can be built this way. Unfortunately, this situation is changing. The new development environments, such as Go, Rust or node.js see this approach as cumbersome and getting-in-the-way for developers. They want projects' setup to be as quick and automatic as possible - and it should also be cross-platform. So each such environment comes with a specialized tool which handles installation of dependencies and bypasses the distribution package management altogether. Typically these dependencies are fetched from the Internet and installed into the project tree. The details are hidden; it's assumed that developers don't want to know or care. In particular, specific versions of dependencies can be only weakly specified or ignored altogether (that is, the latest commit is always fetched), licensing is totally overlooked, a list of what was installed cannot be trivially obtained, and repeating the procedure the next day may result in a different set of code being pulled in, because someone somewhere added a commit to their github repo. This does not work well in Yocto context. Yocto project prides itself on being specific and exact about what gets build, how it gets built and what license is attached to each component. So we need to somehow enforce that with the new model, and avoid the situation where separate, incompatible, and difficult to grasp solutions are developed for each language environment. *Design considerations* 1. I would like recipes to remain short and sweet and clear. In particular, node.js projects can pull in hundreds of dependencies; I want to keep their metadata out of the recipe and somewhere else, for readability, clarity, and maintainability. 2. I don't want to implement custom fetchers, or otherwise re-implement (poorly) those language-specific build and dependency management tools. Let's use npm, cargo and go as much as we possibly can and let them do their job - yes, that also includes them fetching things from the internet for us. 3. When things need to be updated to a new version, manual editing of metadata should be avoided: when there are hundreds of dependencies, a tool should modify the metadata, and human should only inspect the changes. *How do we deal with this?* By introducing a lockdown file that lives next to the recipe. The concept is already implemented in npm, but needs to be made generic and come with a common API that is using the file to verify the build. *What is a lockdown file?* The file captures all of the recipe dependencies that are pulled in by the build tool. For each such dependency the following information is provided (this is really similar to what is in recipes, and that is on purpose: - name - description (optional) - verification data (this is specific to each language, but can be version, git commit id, a checksum and so on). The only requirement is that it maps to a unique set of code. - license string - license file path - license checksum *How is the lockdown file used?* 1. It needs to be generated in the first place when adding a new recipe. For example: bitbake -c generate_lockdown recipe would fetch and unpack the recipe code, then run npm/cargo/go to pull in the dependencies, then walk the project tree and generate the lockdown metadata. Sometimes the tools can help here somewhat, but other times they can be used only for fetching, and verification data has to be figured out by inspecting the tree with our custom-written code. This is the hard part that we have to deal with. 2. It can be used to perform a 'loose' build of the recipe that does not guarantee reproducibility. We have to accept this: some projects just don't care about it, and offer no support to those who want reproducibility. We should at least provide a way to build such projects in Yocto. The information in lockdown file is not enforced; it's merely compared against the actual build and any differences presented to the user as warnings. This is a recipe setting. 3. It can also be used to perform a 'strict' build of the recipe that enforces what is in the lockdown file. The information in the lockdown file is given to the language-specific tool to help it fetch the right things (whenever the tool makes it possible), and then is used to compare to what was fetched, but this time any mismatches stop the build. Exactly how this happens is specific to each language, and again, it is the hard bit that we need to deal with. 4. When a recipe is updated to a new version, the lockdown file needs to be updated as well. One possibility is to generate a new lockdown file (as in point 1), and then a human can compare that against the old lockdown file. bitbake -c update_lockdown recipe 5. Packaging Go by default is compiling everything into a static executable, so there are no separate packages. All dependencies' licenses should be rolled into the package: lockdown file tells what they are and where they are in the build tree. Other environments do install the dependencies somewhere in the system, so those should be packaged separately: lockdown file is used to get a list of them and attach licenses to them. Installation paths (things that FILES_ is set to) should typically be easy to figure out from dependency names. *Conclusion* This is only a preliminary idea: I understand that the devil is in the details, and there are plenty of details where things may not work out as planned, or there's something else I didn't think of that should be accounted for. So flame away! ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Proposal: dealing with language-specific build tools/dependency management tools 2017-03-10 13:49 Proposal: dealing with language-specific build tools/dependency management tools Alexander Kanavin @ 2017-03-10 14:30 ` Otavio Salvador 2017-03-10 14:48 ` Alexander Kanavin 2017-03-10 20:49 ` Trevor Woerner ` (2 subsequent siblings) 3 siblings, 1 reply; 25+ messages in thread From: Otavio Salvador @ 2017-03-10 14:30 UTC (permalink / raw) To: Alexander Kanavin; +Cc: Yocto Project, openembedded-architecture Hello Alexander, On Fri, Mar 10, 2017 at 10:49 AM, Alexander Kanavin <alexander.kanavin@linux.intel.com> wrote: > The new generation of programming languages (think node.js, Go, Rust) is a > poor fit for the Yocto build model which follows the traditional Unix model. > In particular, those new development environments have no problem with > 'grabbing random stuff from the Internet' as a part of development and build > process. However, Yocto has very strict rules about the build steps and what > they can and can not do, and also a strict enforcement of license and > version checks for every component that gets built. Those two models clash, > and this is a proposal of how they could be reconciled. When integrating the CHICKEN Scheme support onto the Yocto Project we dealt it using their installation tool but making the package of individual packages (eggs, in this specific case) as individual recipes. We went further and automated the recipe generation and this made it quite easy to maintain in long term. Take a look at: https://github.com/OSSystems/meta-chicken -- Otavio Salvador O.S. Systems http://www.ossystems.com.br http://code.ossystems.com.br Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750 ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Proposal: dealing with language-specific build tools/dependency management tools 2017-03-10 14:30 ` [Openembedded-architecture] " Otavio Salvador @ 2017-03-10 14:48 ` Alexander Kanavin 2017-03-10 14:58 ` Otavio Salvador 0 siblings, 1 reply; 25+ messages in thread From: Alexander Kanavin @ 2017-03-10 14:48 UTC (permalink / raw) To: Otavio Salvador; +Cc: Yocto Project, openembedded-architecture On 03/10/2017 04:30 PM, Otavio Salvador wrote: > When integrating the CHICKEN Scheme support onto the Yocto Project we > dealt it using their installation tool but making the package of > individual packages (eggs, in this specific case) as individual > recipes. We went further and automated the recipe generation and this > made it quite easy to maintain in long term. > > Take a look at: > > https://github.com/OSSystems/meta-chicken Thanks, even though my Scheme-fu isn't great :) I'd like to avoid generating entire separate recipes though, because that implies your custom-written tool would be figuring out where the dependency source came from in the first place, and what are its own dependencies, when creating the recipe, which can be tricky, breakage-prone guesswork. I want to use existing tools (like 'npm install') for getting the stuff from the network - we don't really need full recipes, we just want to know the licenses of the dependencies, and, if possible, lock them down to a specific version. Alex ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Proposal: dealing with language-specific build tools/dependency management tools 2017-03-10 14:48 ` Alexander Kanavin @ 2017-03-10 14:58 ` Otavio Salvador 2017-03-10 15:10 ` Alexander Kanavin ` (2 more replies) 0 siblings, 3 replies; 25+ messages in thread From: Otavio Salvador @ 2017-03-10 14:58 UTC (permalink / raw) To: Alexander Kanavin; +Cc: Yocto Project, openembedded-architecture On Fri, Mar 10, 2017 at 11:48 AM, Alexander Kanavin <alexander.kanavin@linux.intel.com> wrote: > On 03/10/2017 04:30 PM, Otavio Salvador wrote: >> >> When integrating the CHICKEN Scheme support onto the Yocto Project we >> dealt it using their installation tool but making the package of >> individual packages (eggs, in this specific case) as individual >> recipes. We went further and automated the recipe generation and this >> made it quite easy to maintain in long term. >> >> Take a look at: >> >> https://github.com/OSSystems/meta-chicken > > > Thanks, even though my Scheme-fu isn't great :) > > I'd like to avoid generating entire separate recipes though, because that > implies your custom-written tool would be figuring out where the dependency > source came from in the first place, and what are its own dependencies, when > creating the recipe, which can be tricky, breakage-prone guesswork. In fact not; as you generate the recipes for the dependencies, it goes recursively and is always good. > I want to use existing tools (like 'npm install') for getting the stuff from > the network - we don't really need full recipes, we just want to know the > licenses of the dependencies, and, if possible, lock them down to a specific > version. Well we initially thought this would suffice but consider a security flaw. As many apps may be using different versions of same package it becomes a nightmare to figure which ones are affected. If using dependencies it is fine, for free. -- Otavio Salvador O.S. Systems http://www.ossystems.com.br http://code.ossystems.com.br Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750 ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Proposal: dealing with language-specific build tools/dependency management tools 2017-03-10 14:58 ` Otavio Salvador @ 2017-03-10 15:10 ` Alexander Kanavin 2017-03-10 15:33 ` Derek Straka ` (3 more replies) 2017-03-10 16:23 ` Mark Hatle 2017-03-10 20:37 ` Josef Holzmayr 2 siblings, 4 replies; 25+ messages in thread From: Alexander Kanavin @ 2017-03-10 15:10 UTC (permalink / raw) To: Otavio Salvador; +Cc: Yocto Project, openembedded-architecture On 03/10/2017 04:58 PM, Otavio Salvador wrote: >> I'd like to avoid generating entire separate recipes though, because that >> implies your custom-written tool would be figuring out where the dependency >> source came from in the first place, and what are its own dependencies, when >> creating the recipe, which can be tricky, breakage-prone guesswork. > > In fact not; as you generate the recipes for the dependencies, it goes > recursively and is always good. Would it also be true for npm, Rust, Go, and other languages that will come along? In your specific case the metadata may be easily available to parse and convert to recipe form, but this many not hold in other situations. npm fetcher for instance was a nightmare to write, from what I've heard: http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/npm.py >> I want to use existing tools (like 'npm install') for getting the stuff from >> the network - we don't really need full recipes, we just want to know the >> licenses of the dependencies, and, if possible, lock them down to a specific >> version. > > Well we initially thought this would suffice but consider a security > flaw. As many apps may be using different versions of same package it > becomes a nightmare to figure which ones are affected. If using > dependencies it is fine, for free. The lockdown files would list the versions of the dependencies (if it is possible, which is not always true), so you can inspect those to see if something is vulnerable. In node.js or Go worlds the libraries are not reused between apps anyway, so it really doesn't matter if they're packaged as separate recipes or not (I didn't have time to check Rust, but as it's also using lockdown files, I believe the libraries are not reused either). Alex ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Proposal: dealing with language-specific build tools/dependency management tools 2017-03-10 15:10 ` Alexander Kanavin @ 2017-03-10 15:33 ` Derek Straka 2017-03-10 15:35 ` Derek Straka ` (2 subsequent siblings) 3 siblings, 0 replies; 25+ messages in thread From: Derek Straka @ 2017-03-10 15:33 UTC (permalink / raw) To: Alexander Kanavin Cc: Yocto Project, openembedded-architecture, Doug Goldstein, Otavio Salvador [-- Attachment #1: Type: text/plain, Size: 3549 bytes --] On Fri, Mar 10, 2017 at 10:10 AM, Alexander Kanavin < alexander.kanavin@linux.intel.com> wrote: > On 03/10/2017 04:58 PM, Otavio Salvador wrote: > >> I'd like to avoid generating entire separate recipes though, because that >>> implies your custom-written tool would be figuring out where the >>> dependency >>> source came from in the first place, and what are its own dependencies, >>> when >>> creating the recipe, which can be tricky, breakage-prone guesswork. >>> >> >> In fact not; as you generate the recipes for the dependencies, it goes >> recursively and is always good. >> > > Would it also be true for npm, Rust, Go, and other languages that will > come along? In your specific case the metadata may be easily available to > parse and convert to recipe form, but this many not hold in other > situations. > > npm fetcher for instance was a nightmare to write, from what I've heard: > > http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/ > lib/bb/fetch2/npm.py > > I want to use existing tools (like 'npm install') for getting the stuff >>> from >>> the network - we don't really need full recipes, we just want to know the >>> licenses of the dependencies, and, if possible, lock them down to a >>> specific >>> version. >>> >> >> Well we initially thought this would suffice but consider a security >> flaw. As many apps may be using different versions of same package it >> becomes a nightmare to figure which ones are affected. If using >> dependencies it is fine, for free. >> > > The lockdown files would list the versions of the dependencies (if it is > possible, which is not always true), so you can inspect those to see if > something is vulnerable. In node.js or Go worlds the libraries are not > reused between apps anyway, so it really doesn't matter if they're packaged > as separate recipes or not (I didn't have time to check Rust, but as it's > also using lockdown files, I believe the libraries are not reused either). > > I can chime in on how we do things in meta-rust. Right now, each application is statically linked against the crate library versions it calls out. At this point, the rust ABI is not stable between versions of the compiler, so we made the conscious decision to avoid dynamic libraries for the time being. We acknowledge this does increase the file system size, but we didn't want to have to deal with users trying to perform package updates on individual shared objects. We have our own cargo module that we maintain that helps users generate their bitbake recipe for a given package [1]. There was a small bit of work done on trying to create so files, but it became an unmanageable version nightmare isn't supported moving forward[2]. We also maintain our own custom fetcher for crates [3] and ran into some issues getting it totally supported without integrating it into the set of bitbake fetchers [4][5]. -Derek [1] - https://github.com/meta-rust/meta-rust [2] - https://github.com/cardoe/cargo-bitbake [3] - https://github.com/meta-rust/meta-rust/tree/master/recipes-core [4] - https://github.com/meta-rust/meta-rust/blob/master/lib/crate.py [5] - https://github.com/meta-rust/meta-rust/issues/136 [6] - https://bugzilla.yoctoproject.org/show_bug.cgi?id=10867 Alex > > _______________________________________________ > Openembedded-architecture mailing list > Openembedded-architecture@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-architecture > [-- Attachment #2: Type: text/html, Size: 5711 bytes --] ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Proposal: dealing with language-specific build tools/dependency management tools 2017-03-10 15:10 ` Alexander Kanavin 2017-03-10 15:33 ` Derek Straka @ 2017-03-10 15:35 ` Derek Straka 2017-03-13 8:25 ` Piotr Figiel 2017-03-13 17:51 ` Bruce, Henry 3 siblings, 0 replies; 25+ messages in thread From: Derek Straka @ 2017-03-10 15:35 UTC (permalink / raw) To: Yocto Project, openembedded-architecture [-- Attachment #1: Type: text/plain, Size: 3547 bytes --] On Fri, Mar 10, 2017 at 10:10 AM, Alexander Kanavin < alexander.kanavin@linux.intel.com> wrote: > On 03/10/2017 04:58 PM, Otavio Salvador wrote: > >> I'd like to avoid generating entire separate recipes though, because that >>> implies your custom-written tool would be figuring out where the >>> dependency >>> source came from in the first place, and what are its own dependencies, >>> when >>> creating the recipe, which can be tricky, breakage-prone guesswork. >>> >> >> In fact not; as you generate the recipes for the dependencies, it goes >> recursively and is always good. >> > > Would it also be true for npm, Rust, Go, and other languages that will > come along? In your specific case the metadata may be easily available to > parse and convert to recipe form, but this many not hold in other > situations. > > npm fetcher for instance was a nightmare to write, from what I've heard: > > http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/ > lib/bb/fetch2/npm.py > > I want to use existing tools (like 'npm install') for getting the stuff >>> from >>> the network - we don't really need full recipes, we just want to know the >>> licenses of the dependencies, and, if possible, lock them down to a >>> specific >>> version. >>> >> >> Well we initially thought this would suffice but consider a security >> flaw. As many apps may be using different versions of same package it >> becomes a nightmare to figure which ones are affected. If using >> dependencies it is fine, for free. >> > > The lockdown files would list the versions of the dependencies (if it is > possible, which is not always true), so you can inspect those to see if > something is vulnerable. In node.js or Go worlds the libraries are not > reused between apps anyway, so it really doesn't matter if they're packaged > as separate recipes or not (I didn't have time to check Rust, but as it's > also using lockdown files, I believe the libraries are not reused either). > > I can chime in on how we do things in meta-rust. Right now, each application is statically linked against the crate library versions it calls out. At this point, the rust ABI is not stable between versions of the compiler, so we made the conscious decision to avoid dynamic libraries for the time being. We acknowledge this does increase the file system size, but we didn't want to have to deal with users trying to perform package updates on individual shared objects. We have our own cargo module that we maintain that helps users generate their bitbake recipe for a given package [1]. There was a small bit of work done on trying to create so files, but it became an unmanageable version nightmare isn't supported moving forward[2]. We also maintain our own custom fetcher for crates [3] and ran into some issues getting it totally supported without integrating it into the set of bitbake fetchers [4][5]. -Derek [1] - https://github.com/meta-rust/meta-rust [2] - https://github.com/cardoe/cargo-bitbake [3] - https://github.com/meta-rust/meta-rust/tree/master/recipes-core [4] - https://github.com/meta-rust/meta-rust/blob/master/lib/crate.py [5] - https://github.com/meta-rust/meta-rust/issues/136 [6] - https://bugzilla.yoctoproject.org/show_bug.cgi?id=10867 > Alex > > _______________________________________________ > Openembedded-architecture mailing list > Openembedded-architecture@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-architecture > [-- Attachment #2: Type: text/html, Size: 6000 bytes --] ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Proposal: dealing with language-specific build tools/dependency management tools 2017-03-10 15:10 ` Alexander Kanavin 2017-03-10 15:33 ` Derek Straka 2017-03-10 15:35 ` Derek Straka @ 2017-03-13 8:25 ` Piotr Figiel 2017-03-13 17:51 ` Bruce, Henry 3 siblings, 0 replies; 25+ messages in thread From: Piotr Figiel @ 2017-03-13 8:25 UTC (permalink / raw) To: Alexander Kanavin Cc: Yocto Project, openembedded-architecture, Otavio Salvador Hi Alexander, first of all thanks for the efforts. Some comments to add to the mix: 2017-03-10 16:10 GMT+01:00 Alexander Kanavin <alexander.kanavin@linux.intel.com>: > The lockdown files would list the versions of the dependencies (if it is > possible, which is not always true), so you can inspect those to see if > something is vulnerable. In node.js or Go worlds the libraries are not > reused between apps anyway, so it really doesn't matter if they're packaged > as separate recipes or not (I didn't have time to check Rust, but as it's > also using lockdown files, I believe the libraries are not reused either). I don't know if you've heard of the lately popular idea of "microservices" - but basically it comes down to having multiple specialized node applications installed. In practice those applications often share the dependencies, so it would totally make sense to use the packages approach so that the dependencies don't need to be installed in multiple copies. Perhaps maybe not with recipe per npm package, but maybe with some more advanced bitbake magic or a post-rootfs hook to run deduplication. What works most of the time is using shrinkwrap to freeze the dependencies to some local npm mirror, so the concept of lockdown would make sense. Just a word of warning that it sometimes doesn't work that well - some of the npm packages (in the dependencies chain) may have hard-coded URI's to e.g. gitlab and shrinkwrap will keep those references instead of npm mirror. Also npm itself doesn't really check for consistency it only checks for versions, what can happen is that the contents may change but the version string may not. In terms of node yarn [1] seems to address some of the npm shortcomings but I'm not aware of any progress in regards of yocto integration. [1] https://yarnpkg.com/ Best regards, Piotr. ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Proposal: dealing with language-specific build tools/dependency management tools 2017-03-10 15:10 ` Alexander Kanavin ` (2 preceding siblings ...) 2017-03-13 8:25 ` Piotr Figiel @ 2017-03-13 17:51 ` Bruce, Henry 2017-03-16 10:25 ` Alexander Kanavin 3 siblings, 1 reply; 25+ messages in thread From: Bruce, Henry @ 2017-03-13 17:51 UTC (permalink / raw) To: openembedded-architecture, yocto On Fri, 2017-03-10 at 17:10 +0200, Alexander Kanavin wrote: Thanks for raising this topic. The problems we hit in adding node.js support clearly shows we would benefit from a common approach to supporting languages with their own and runtime and packaging. > > npm fetcher for instance was a nightmare to write, from what I've > heard: > > http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fe > tch2/npm.py Although I didn't write the code I worked with Paul Eggleton (who now owns it) on improving it, and yes it was hard to write. And it is becoming increasingly complex as we hit corner cases, indicating that the current architecture is not ideal. > > > I want to use existing tools (like 'npm install') for getting the > > > stuff from the network - we don't really need full recipes, we > > > just want to know the licenses of the dependencies, and, if > > > possible, lock them down to a specific version. I agree that leveraging the likes of 'npm install' will make life simpler but the problem with these operations is that they span a range of bitbake tasks. The reason we wrote an npm fetcher was to limit network access to the fetch task. This works in conjunction with an npm patch that enables the install command to run in the compile task without accessing the registry. Are you are proposing that we allow network access outside of the fetch task? I look forward to Paul's comments. Henry ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Proposal: dealing with language-specific build tools/dependency management tools 2017-03-13 17:51 ` Bruce, Henry @ 2017-03-16 10:25 ` Alexander Kanavin 0 siblings, 0 replies; 25+ messages in thread From: Alexander Kanavin @ 2017-03-16 10:25 UTC (permalink / raw) To: yocto, openembedded-architecture, Bruce, Henry On 03/13/2017 07:51 PM, Bruce, Henry wrote: > I agree that leveraging the likes of 'npm install' will make life > simpler but the problem with these operations is that they span a range > of bitbake tasks. The reason we wrote an npm fetcher was to limit > network access to the fetch task. This works in conjunction with an npm > patch that enables the install command to run in the compile task > without accessing the registry. Are you are proposing that we allow > network access outside of the fetch task? Yes, in a limited way. In addition to fetch, there would be a fetch_deps task, which is also allowed to access the network. Alex ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Proposal: dealing with language-specific build tools/dependency management tools 2017-03-10 14:58 ` Otavio Salvador 2017-03-10 15:10 ` Alexander Kanavin @ 2017-03-10 16:23 ` Mark Hatle 2017-03-10 20:37 ` Josef Holzmayr 2 siblings, 0 replies; 25+ messages in thread From: Mark Hatle @ 2017-03-10 16:23 UTC (permalink / raw) To: Otavio Salvador, Alexander Kanavin Cc: Yocto Project, openembedded-architecture On 3/10/17 8:58 AM, Otavio Salvador wrote: > On Fri, Mar 10, 2017 at 11:48 AM, Alexander Kanavin > <alexander.kanavin@linux.intel.com> wrote: >> On 03/10/2017 04:30 PM, Otavio Salvador wrote: >>> >>> When integrating the CHICKEN Scheme support onto the Yocto Project we >>> dealt it using their installation tool but making the package of >>> individual packages (eggs, in this specific case) as individual >>> recipes. We went further and automated the recipe generation and this >>> made it quite easy to maintain in long term. >>> >>> Take a look at: >>> >>> https://github.com/OSSystems/meta-chicken >> >> >> Thanks, even though my Scheme-fu isn't great :) >> >> I'd like to avoid generating entire separate recipes though, because that >> implies your custom-written tool would be figuring out where the dependency >> source came from in the first place, and what are its own dependencies, when >> creating the recipe, which can be tricky, breakage-prone guesswork. > > In fact not; as you generate the recipes for the dependencies, it goes > recursively and is always good. > >> I want to use existing tools (like 'npm install') for getting the stuff from >> the network - we don't really need full recipes, we just want to know the >> licenses of the dependencies, and, if possible, lock them down to a specific >> version. > > Well we initially thought this would suffice but consider a security > flaw. As many apps may be using different versions of same package it > becomes a nightmare to figure which ones are affected. If using > dependencies it is fine, for free. > I'm wondering if there may be any way to generate these recipes on the fly (say during ConfigParsed -- and then issue a reparse if things have changed?) I've got concerns with this "new model" of development specifically around: *) Bug fixes -) Security bugs especially When something is updated to fix a problem, how do we know we "got everything"? Something needs to flag the system to make it clear that not only the items itself, but all of the things it depends on are at the correct (revised) versions. The other problem is 'certified' devices. Various certification requirements don't allow just 'downloading new content whenever', but they often allow small targeted fixes. So in these cases, the user would likely want to know there was an update for a specific reason and then 'backport, and not change the version' to address their certification issues. (And yes, I know of companies that used to lock down version numbers of code, and simply upgrade anyway to get around this -- it's always possible, but not exactly within the spirit of the certification process.. targeted changes are much more reasonable in that environment.) *) License considerations A big problem in all of these new developer centric models is that they are designed for back-end service were the license doesn't matter (as much). For embedded devices, we have to have a clear lineage of the license, code and source (SRC_URI) to have any chance to "doing the right thing" or at worst (defending work during a lawsuit). *) Export/import control Unlike random open source projects, when you build a physical device there may export/import control that has to happen as well. Without the discrete components being clearly visible to the developers -- it is -very- difficult to satisfy the constraints of the various export/import requirements around the world. I think auto-generating recipe contents help with these. Having a good process behind the auto-generation can make the license and software "update" procedure a lot cleaner and provide the tracking needed for the export/import control. I do see we need a better model for these components as we move away from the core operating system and 'up the stack'.... better auto-generation and using the tooling provided, while yet conforming to the needs we have makes the most sense to me. (The problem of course is most of these seem to need an expert, or at least a highly interested developer to have a shot at working or keep working over time.... the developer needs to be able to explain for the OE the alternative model needed so we can collectively figure out how to do it best.) I think this is a good start at the discussion, but other then listing concerns, unfortunately I don't have any solutions to offer. --Mark ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Proposal: dealing with language-specific build tools/dependency management tools 2017-03-10 14:58 ` Otavio Salvador 2017-03-10 15:10 ` Alexander Kanavin 2017-03-10 16:23 ` Mark Hatle @ 2017-03-10 20:37 ` Josef Holzmayr 2 siblings, 0 replies; 25+ messages in thread From: Josef Holzmayr @ 2017-03-10 20:37 UTC (permalink / raw) To: Otavio Salvador, Alexander Kanavin Cc: Yocto Project, openembedded-architecture Hi Alexander, thanks for kicking off the topic, sounds like its kinda overdue. While I have no really good solution (d'oh!), please find below thoughts and bits and pieces about some of the points. *Recipes* My gut feeling say auto-generation of the recipes is a good way to go. Yet I am uncertain which set of functionality such an auto-generated recipe would provide. The obvious possibilities: Option 1: Only dependency/license tracking. Pro: provides exactly what it says - known and proven means to keep track of the dependencies and licenses. Contra: this would mean that we have a recipes that only do housekeeping, while the actual workload is taken care of in the main recipe. This seperation sounds like a massive headache Option 2: Full recipes Pro: would fit into the OE workflow. Contra: requires the sub-package managers to at least "play along". Think two applications having the same dependency. It would get installed once (globally/system-wide), and both applications have to use it. My interpretation is that this just is not how it works (looking at npm) Coming from there, one can go further, resulting in Option 1.5: Provide recipes for common base functionality. Those would have the all-in-one-locked-down approach, but are meant to be used as global dependencies. Example: the MEAN stack. Like its name says, it consist of 4 main pieces (-> possible recipes) which are needed by the application. Pro: reduces recipe number/bloat and makes dependencies readable. The mindset fits the classis 'library' thinking. Contra: the depending application would have to be packaged with the infrastructure in mind. So while the library recipes could rely on the locked down sub-package manager, the application would have to intently skip it and provide a custom installation. Which is an annyonce if you are application dev and packager in union - and a major pain point if you want to package some upstream application. *Lockdown* To me the approach sounds interesting, yet it comes with a couple of points to think about. - Feature set: having such a lockdown system implies/requires all sub-package managers to provide (at least) the functionality to fullfill the needs of the lockdown process/recreation. Is that something we can take for granted? - Multilanguage: imagine a package for example having some native go code, then nodejs bindings and then a npdejs application on top. How would that look like? Multiple lockdown files? What are the implications? *Sub-Package managers in general* We've seen the first (perl, php, python...) and second (npm, go, rust...) wave of those by now. A third one certainly will come one day. Taking a step back for a larger perspective, it sounds like what we actually need is some form of nested dependencies. Or scoped dependencies. Whatever we want to call it, because to me thats what it actually is. The dependencies we have now are always global. But especially the second wave things think different. Those sub-package managers do not care about the whole system. They care about their small part of it. So my interpretation is that we need to take that paradigm shift, and decide upon the actual implementation details afterwards. *Conclusion* Guess I raised more questions than I offered answers for. Sorry :-( Greetz (and try to enjoy the weekend) -- Josef Holzmayr Software Developer Embedded Systems Tel: +49 8444 9204-48 Fax: +49 8444 9204-50 R-S-I Elektrotechnik GmbH & Co. KG Woelkestrasse 11 D-85301 Schweitenkirchen www.rsi-elektrotechnik.de ——————————————— Amtsgericht Ingolstadt – GmbH: HRB 191328 – KG: HRA 170393 Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg Ust-IdNr: DE 128592548 _____________________________________________________________ Amtsgericht Ingolstadt - GmbH: HRB 191328 - KG: HRA 170363 Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg USt-IdNr.: DE 128592548 ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Proposal: dealing with language-specific build tools/dependency management tools 2017-03-10 13:49 Proposal: dealing with language-specific build tools/dependency management tools Alexander Kanavin 2017-03-10 14:30 ` [Openembedded-architecture] " Otavio Salvador @ 2017-03-10 20:49 ` Trevor Woerner 2017-03-11 13:07 ` Josef Holzmayr 2017-03-13 20:58 ` Paul Eggleton 2017-03-16 8:17 ` [Openembedded-architecture] Sum up - " Josef Holzmayr 3 siblings, 1 reply; 25+ messages in thread From: Trevor Woerner @ 2017-03-10 20:49 UTC (permalink / raw) To: Alexander Kanavin; +Cc: Yocto Project, openembedded-architecture Hi Alexander, Thanks for bringing up this important topic. There is no doubt we're seeing paradigm shifts in the way applications are written, built, and packaged; as well as a complete lack of interest in licensing. Although the trend is to not care about licensing, I believe it is vitally important that we do our best to keep track of all the licensing from every package that is pulled into an image. If we're pulling in >1000 npm packages just for one node app, then that means we should have >1000 item list of each dependency and their respective licenses. Although it makes a recipe look ugly, I wouldn't want to drop this functionality due to aesthetic concerns. Maybe the license list could be moved to another file that is required by the "main" recipe file? Maybe the list could be moved to the bottom of the file? In the case of node specifically, I don't think trying to create and maintain separate recipes for each and every dependency one might find in the npm registry would be a sane approach. Currently we embed the version info into the recipe filename. This will simply not scale to millions of npm packages, each with numerous versions. I've been playing with node a fair amount lately as it relates to OE and I have to say I've been quite impressed! These aren't easy things and I think there's a lot of good work happening. I've outlined some of my thoughts on my experiences[1]: http://lists.openembedded.org/pipermail/openembedded-core/2017-February/133432.html Other than these (short-term?) issues devtool seems to be on the right track (?) It does, for example, generate a lockdown.json file and an npm-shrinkwrap.json file automatically. All we need is the package.json from the app developer, and that can be auto-generated via npm. I think we have to accept that node developers are going to want to develop on the target device itself, and when they're done they can hand us the package.json file which we can run devtool on which will generate the recipe for us. As a short-term work-around, I've simply been creating an image with node+npm, running it on the device, copying over the package.json file, running npm install against it, then collecting up all the extra stuff that gets added to my image (as a result), and bundling all that into a platform-specific "bin_package" (bbclass). It works, but it's a multi-step process. If I could cut out some of those steps (once things from [1] are fixed), it would be an improvement. Best regards, Trevor [1] A short recap of those emails: Different paths seem to be followed depending on whether you point devtool at, say, a github repository versus a local checkout of the same project. That seems wrong. Also (as you've pointed out) RSS is messing all this up on master at the moment; but I assume this can/will get fixed? Things work fine on morty. Also, devtool gets tripped up when it encounters a license string that isn't found in its list of already-known license strings. This approach seems doomed to failure. It has to be able to recover gracefully and continue walking the dependency list without having to continuously add corner cases to the code. ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Proposal: dealing with language-specific build tools/dependency management tools 2017-03-10 20:49 ` Trevor Woerner @ 2017-03-11 13:07 ` Josef Holzmayr 0 siblings, 0 replies; 25+ messages in thread From: Josef Holzmayr @ 2017-03-11 13:07 UTC (permalink / raw) To: Trevor Woerner, Alexander Kanavin Cc: Yocto Project, openembedded-architecture Hi Trevor, On 10.03.2017 21:49, Trevor Woerner wrote: > Although the trend is to not care about licensing, I believe it is vitally > important that we do our best to keep track of all the licensing from every > package that is pulled into an image. If we're pulling in >1000 npm packages > just for one node app, then that means we should have >1000 item list of each > dependency and their respective licenses. Although it makes a recipe look > ugly, I wouldn't want to drop this functionality due to aesthetic concerns. > Maybe the license list could be moved to another file that is required by the > "main" recipe file? Maybe the list could be moved to the bottom of the file? Boiling that down, it sounds to me like the approach is the following: 1) Let the sub-package manager do its work as its meant to be. 2) If the sub-package manager supports version lockdown/shrinkwrapping. it shall be used. 3) The OE build process is only meant to take care of licensing. (This could basically be seen as an additional Option 0 to the mail from yesterday: License-only recipes. [1]) Sounds like an interesting option indeed. Keeping it in the recipe means, in an abstract manner, that we need support for sub-licensing. Might be a viable route to go . > In the case of node specifically, I don't think trying to create and maintain > separate recipes for each and every dependency one might find in the npm > registry would be a sane approach. Currently we embed the version info into > the recipe filename. This will simply not scale to millions of npm packages, > each with numerous versions. It will not scale for human inspection. For metadata that is algorithmically generated and used, I personally don't think the sheer number is a killer argument. > I've been playing with node a fair amount lately as it relates to OE and I > have to say I've been quite impressed! These aren't easy things and I think > there's a lot of good work happening. Totally agreed. But implicitly, we tend to see npm as the reference for such sub-package managers. Is this a good way? Alexanders approach was to find a concept that fits all such constructions. Maybe its also worthwhile to think along the opposite lines: Treat each and every of those sub-package managers completely on its own, with all its specialities? (And hope that their number stays low....) > Other than these (short-term?) issues devtool seems to be on the right > track (?) It does, for example, generate a lockdown.json file and an > npm-shrinkwrap.json file automatically. All we need is the package.json from > the app developer, and that can be auto-generated via npm. I think we have to > accept that node developers are going to want to develop on the target device > itself, and when they're done they can hand us the package.json file which we > can run devtool on which will generate the recipe for us. I'm not too convinced that this is a good way. Especially when it comes to node modules that contain some native code, this becomes very ugly. My experience is that auto-processing that stuff adds megabytes of clutter, while all that matters in the end is a binary that is a couple if kilobytes. So how would one tackle that? Carve that out as a separate recipe again? > As a short-term work-around, I've simply been creating an image with node+npm, > running it on the device, copying over the package.json file, running npm > install against it, then collecting up all the extra stuff that gets added > to my image (as a result), and bundling all that into a platform-specific > "bin_package" (bbclass). It works, but it's a multi-step process. If I could > cut out some of those steps (once things from [1] are fixed), it would be an > improvement. Yeah, thats an option. I am rather providing custom compile and install stages, as the applications I'm working on have similar requisites, but I didn't want to go multi-step/binary. Greetz PS: being tired of typing sub-package manager again and again, how shall we call this? SPM? Application Package Managers (APM)? [1] http://lists.openembedded.org/pipermail/openembedded-architecture/2017-March/000489.html -- Josef Holzmayr Software Developer Embedded Systems Tel: +49 8444 9204-48 Fax: +49 8444 9204-50 R-S-I Elektrotechnik GmbH & Co. KG Woelkestrasse 11 D-85301 Schweitenkirchen www.rsi-elektrotechnik.de ——————————————— Amtsgericht Ingolstadt – GmbH: HRB 191328 – KG: HRA 170393 Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg Ust-IdNr: DE 128592548 _____________________________________________________________ Amtsgericht Ingolstadt - GmbH: HRB 191328 - KG: HRA 170363 Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg USt-IdNr.: DE 128592548 ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Proposal: dealing with language-specific build tools/dependency management tools 2017-03-10 13:49 Proposal: dealing with language-specific build tools/dependency management tools Alexander Kanavin 2017-03-10 14:30 ` [Openembedded-architecture] " Otavio Salvador 2017-03-10 20:49 ` Trevor Woerner @ 2017-03-13 20:58 ` Paul Eggleton 2017-03-16 8:17 ` [Openembedded-architecture] Sum up - " Josef Holzmayr 3 siblings, 0 replies; 25+ messages in thread From: Paul Eggleton @ 2017-03-13 20:58 UTC (permalink / raw) To: openembedded-architecture; +Cc: Yocto Project Alex - thanks for kicking off the discussion here, this is something we definitely need to get a better handle on. My involvement with this is perhaps somewhat accidental - I ended up working on the fetcher that was implemented by someone else because I needed it to work for devtool integration, and got sucked into fixing a number of issues I found in the process. Unfortunately - other than python, naturally - I'm not that familiar with the actual languages or their package managers, I've only come into contact with node.js and npm through the work that I've done with the build system. This is a tough problem to solve. Maybe some of the other language package managers are more cooperative, I'm not sure, but npm *really* doesn't want to be used in the manner we're trying to use it - it insists on being able to go out to the registry and things get a bit ugly if you tell it not to. RSS- related issues aside (since I kind of have a path to fixing those) the latest pain was in 90cb980a1c49de99a0aec00c0cd5fc1e165490a7 when we shifted from a cp -a to a second invocation of npm install in order to get a more accurate install step - the side-effect has been we broke that step for certain modules, as the certain directives in the package.json file trigger querying the repository, but we've told it not to do that so it errors out. Another thing that is still a stumbling block for the "one package represents a tree of modules" that I've yet to properly resolve is how to deal with a "partial" tree, where you have one or more modules that you want to actually work on, i.e. you have another recipe to satisfy the dependency possibly under the control of devtool in your workspace. npm doesn't seem to provide any specific mechanism to help with this either. At the moment I'm not sure there's a solution with npm other than hacking the package.json file. Since it was brought up by Trevor a while back, I still have a todo item to go and look at yarn [1] to see if it really solved some of these issues for node.js in a less nasty way than npm. One thing I will say - I really want to see the fetchers (or at least, the custom bits) for these language package managers implemented in the metadata rather than in bitbake. This stuff moves fast, there are a growing number of these package managers, and it's awkward to have part of the implementation in one place and a significant portion of the rest of it (class and supporting recipes) in another, without which the fetcher is useless. However, at the same time we want to make sure we don't lose the ability to have mirror tarballs which are implemented on the bitbake side. With regard to the recipes generated by devtool, they do end up being a bit ugly because we package each underlying npm package individually. The reason I did it that way is to have each source and therefore each license represented in the image manifests. I am open to having a mode where we have it all in one package, though, but it seems to me that LICENSE must include all the licenses at parse time. I'm open to suggestions. Perhaps we could save the package.json file next to the recipe - is that going to be practical? It might be a bit easier to update at least. I agree that in addition to providing the lockdown we do definitely need to make updating these recipes easier, it's a bit awkward right now. Cheers, Paul [1] https://yarnpkg.com/ On Saturday, 11 March 2017 2:49:01 AM NZDT Alexander Kanavin wrote: > Hello all, > > *Introduction* > > The new generation of programming languages (think node.js, Go, Rust) is > a poor fit for the Yocto build model which follows the traditional Unix > model. In particular, those new development environments have no problem > with 'grabbing random stuff from the Internet' as a part of development > and build process. However, Yocto has very strict rules about the build > steps and what they can and can not do, and also a strict enforcement of > license and version checks for every component that gets built. Those > two models clash, and this is a proposal of how they could be reconciled. > > I'll also send a separate email that talks specifically about MEAN stack > and how it could be supported as Yocto - take it as a specific example > for all of the below. > > *Background* > > The traditional development model on Unix clearly separates installation > of dependencies needed to develop a project from the development process > itself. Typically, when one wants to build some project, first the > project README needs to be inspected, and any required dependencies > installed system-wide using the distribution package management's tool. > When those dependencies change, usually this manifests itself in a > previously unseen build failure which is again manually resolved by > figuring out the missing dependency and installing it. This can be > awkward, but it's how things have been done for decades, and Yocto's > build system (with separate steps for fetching, unpacking, building, > packaging etc.) is built around the assumption that most software can be > built this way. > > Unfortunately, this situation is changing. The new development > environments, such as Go, Rust or node.js see this approach as > cumbersome and getting-in-the-way for developers. They want projects' > setup to be as quick and automatic as possible - and it should also be > cross-platform. So each such environment comes with a specialized tool > which handles installation of dependencies and bypasses the distribution > package management altogether. Typically these dependencies are fetched > from the Internet and installed into the project tree. The details are > hidden; it's assumed that developers don't want to know or care. In > particular, specific versions of dependencies can be only weakly > specified or ignored altogether (that is, the latest commit is always > fetched), licensing is totally overlooked, a list of what was installed > cannot be trivially obtained, and repeating the procedure the next day > may result in a different set of code being pulled in, because someone > somewhere added a commit to their github repo. > > This does not work well in Yocto context. Yocto project prides itself on > being specific and exact about what gets build, how it gets built and > what license is attached to each component. So we need to somehow > enforce that with the new model, and avoid the situation where separate, > incompatible, and difficult to grasp solutions are developed for each > language environment. > > *Design considerations* > > 1. I would like recipes to remain short and sweet and clear. In > particular, node.js projects can pull in hundreds of dependencies; I > want to keep their metadata out of the recipe and somewhere else, for > readability, clarity, and maintainability. > > 2. I don't want to implement custom fetchers, or otherwise re-implement > (poorly) those language-specific build and dependency management tools. > Let's use npm, cargo and go as much as we possibly can and let them do > their job - yes, that also includes them fetching things from the > internet for us. > > 3. When things need to be updated to a new version, manual editing of > metadata should be avoided: when there are hundreds of dependencies, a > tool should modify the metadata, and human should only inspect the changes. > > *How do we deal with this?* > > By introducing a lockdown file that lives next to the recipe. The > concept is already implemented in npm, but needs to be made generic and > come with a common API that is using the file to verify the build. > > *What is a lockdown file?* > > The file captures all of the recipe dependencies that are pulled in by > the build tool. For each such dependency the following information is > provided (this is really similar to what is in recipes, and that is on > purpose: > > - name > - description (optional) > - verification data (this is specific to each language, but can be > version, git commit id, a checksum and so on). The only requirement is > that it maps to a unique set of code. > - license string > - license file path > - license checksum > > *How is the lockdown file used?* > > 1. It needs to be generated in the first place when adding a new recipe. > For example: > > bitbake -c generate_lockdown recipe > > would fetch and unpack the recipe code, then run npm/cargo/go to pull in > the dependencies, then walk the project tree and generate the lockdown > metadata. Sometimes the tools can help here somewhat, but other times > they can be used only for fetching, and verification data has to be > figured out by inspecting the tree with our custom-written code. This is > the hard part that we have to deal with. > > 2. It can be used to perform a 'loose' build of the recipe that does not > guarantee reproducibility. > > We have to accept this: some projects just don't care about it, and > offer no support to those who want reproducibility. We should at least > provide a way to build such projects in Yocto. The information in > lockdown file is not enforced; it's merely compared against the actual > build and any differences presented to the user as warnings. This is a > recipe setting. > > 3. It can also be used to perform a 'strict' build of the recipe that > enforces what is in the lockdown file. > > The information in the lockdown file is given to the language-specific > tool to help it fetch the right things (whenever the tool makes it > possible), and then is used to compare to what was fetched, but this > time any mismatches stop the build. Exactly how this happens is specific > to each language, and again, it is the hard bit that we need to deal with. > > 4. When a recipe is updated to a new version, the lockdown file needs to > be updated as well. > > One possibility is to generate a new lockdown file (as in point 1), and > then a human can compare that against the old lockdown file. > > bitbake -c update_lockdown recipe > > 5. Packaging > > Go by default is compiling everything into a static executable, so there > are no separate packages. All dependencies' licenses should be rolled > into the package: lockdown file tells what they are and where they are > in the build tree. > > Other environments do install the dependencies somewhere in the system, > so those should be packaged separately: lockdown file is used to get a > list of them and attach licenses to them. Installation paths (things > that FILES_ is set to) should typically be easy to figure out from > dependency names. > > *Conclusion* > > This is only a preliminary idea: I understand that the devil is in the > details, and there are plenty of details where things may not work out > as planned, or there's something else I didn't think of that should be > accounted for. So flame away! > > _______________________________________________ > Openembedded-architecture mailing list > Openembedded-architecture@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-architecture -- Paul Eggleton Intel Open Source Technology Centre ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Sum up - Proposal: dealing with language-specific build tools/dependency management tools 2017-03-10 13:49 Proposal: dealing with language-specific build tools/dependency management tools Alexander Kanavin ` (2 preceding siblings ...) 2017-03-13 20:58 ` Paul Eggleton @ 2017-03-16 8:17 ` Josef Holzmayr 2017-03-16 9:30 ` Paul Barker 2017-03-16 10:21 ` Alexander Kanavin 3 siblings, 2 replies; 25+ messages in thread From: Josef Holzmayr @ 2017-03-16 8:17 UTC (permalink / raw) To: Alexander Kanavin, openembedded-architecture, Yocto Project I'll give it a shot and try to sum up the current state of affairs in this discussion. In accordance to the "Package managers all the way down"-presentation, I'm gonna use the term LPM (for language package manager) for now on. *Requirements* - repeatable and verifyable licensing of all dependencies that a LPM pulls in. - locking down specific versions of packages and their dependencies for reproductible builds. *Optionals* - strict separation of fetch, compile, install stages. If a specifc LPM requires it, those might be intermingled or left out [Mark] - opaque packaging: similar to static linking, we should at least have a way to bundle up a complete application into a single package. Maybe it might even be the default (like rust does it at the moment). - leverage as much as possible of the functionality the LPMs provide instead of reimplementing it. *Wish List* - separating out the LPM infrastructure into one or more distinct layers, not treating it as OE/bitbake core functionality. [Paul] - support for the use of multiple languages/LPMs inside a single recipe, hopefully even package. [myself] *Proposed Solutions* - having lockdown files shipped with the recipes (in whatever form to be defined) - leveraging the recipe system to resolve licensing. If we can boil things down to the common set that we all expect, it will in my opnion serve as a blueprint for the actual implementation to follow. Greetz, -- Josef Holzmayr Software Developer Embedded Systems Tel: +49 8444 9204-48 Fax: +49 8444 9204-50 R-S-I Elektrotechnik GmbH & Co. KG Woelkestrasse 11 D-85301 Schweitenkirchen www.rsi-elektrotechnik.de ——————————————— Amtsgericht Ingolstadt – GmbH: HRB 191328 – KG: HRA 170393 Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg Ust-IdNr: DE 128592548 _____________________________________________________________ Amtsgericht Ingolstadt - GmbH: HRB 191328 - KG: HRA 170363 Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg USt-IdNr.: DE 128592548 ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Sum up - Proposal: dealing with language-specific build tools/dependency management tools 2017-03-16 8:17 ` [Openembedded-architecture] Sum up - " Josef Holzmayr @ 2017-03-16 9:30 ` Paul Barker 2017-03-16 10:35 ` Alexander Kanavin 2017-03-16 15:43 ` Mark Hatle 2017-03-16 10:21 ` Alexander Kanavin 1 sibling, 2 replies; 25+ messages in thread From: Paul Barker @ 2017-03-16 9:30 UTC (permalink / raw) To: Josef Holzmayr; +Cc: Yocto Project, openembedded-architecture On Thu, 16 Mar 2017 09:17:57 +0100 Josef Holzmayr <holzmayr@rsi-elektrotechnik.de> wrote: > I'll give it a shot and try to sum up the current state of affairs in > this discussion. In accordance to the "Package managers all the way > down"-presentation, I'm gonna use the term LPM (for language package > manager) for now on. > > *Requirements* > - repeatable and verifyable licensing of all dependencies that a LPM > pulls in. > - locking down specific versions of packages and their dependencies for > reproductible builds. Agreed with these. > > *Optionals* > - strict separation of fetch, compile, install stages. If a specifc LPM > requires it, those might be intermingled or left out [Mark] I think what we need is strict separation of fetch (allowed to access network) from the later tasks in a build (not allowed to access networks). We need to continue to support users behind corporate firewalls via mirroring and offline builds. > - opaque packaging: similar to static linking, we should at least have a > way to bundle up a complete application into a single package. Maybe it > might even be the default (like rust does it at the moment). > - leverage as much as possible of the functionality the LPMs provide > instead of reimplementing it. We should also aim for automatic generation of recipes from pypi, npmjs, crates.io, etc using devtool. I'm not sure how much of this has already been implemented, I'm not entirely up-to-date on devtool functionality. Staying with NPM as the example, say we have a tree of 1000 dependencies for a top-level application recipe. I understand that generating and then parsing 1000 recipes will bloat things and slow down the build. However, we do need some way to apply patches to those dependencies or fix metadata. For example, what if the license is stated wrong on npmjs? What if the dependency list itself is wrong? What I've always liked about OE is that you can fix this sort of problem in a recipe when you need to. > > *Wish List* > - separating out the LPM infrastructure into one or more distinct > layers, not treating it as OE/bitbake core functionality. [Paul] > - support for the use of multiple languages/LPMs inside a single recipe, > hopefully even package. [myself] > > *Proposed Solutions* > - having lockdown files shipped with the recipes (in whatever form to be > defined) > - leveraging the recipe system to resolve licensing. > > If we can boil things down to the common set that we all expect, it will > in my opnion serve as a blueprint for the actual implementation to follow. > > Greetz, Thanks, Paul ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Sum up - Proposal: dealing with language-specific build tools/dependency management tools 2017-03-16 9:30 ` Paul Barker @ 2017-03-16 10:35 ` Alexander Kanavin 2017-03-16 10:48 ` Jack Mitchell 2017-03-16 15:45 ` Mark Hatle 2017-03-16 15:43 ` Mark Hatle 1 sibling, 2 replies; 25+ messages in thread From: Alexander Kanavin @ 2017-03-16 10:35 UTC (permalink / raw) To: Paul Barker, Josef Holzmayr; +Cc: Yocto Project, openembedded-architecture On 03/16/2017 11:30 AM, Paul Barker wrote: > Staying with NPM as the example, say we have a tree of 1000 > dependencies for a top-level application recipe. I understand that > generating and then parsing 1000 recipes will bloat things and slow > down the build. However, we do need some way to apply patches to those > dependencies or fix metadata. For example, what if the license is > stated wrong on npmjs? What if the dependency list itself is wrong? What > I've always liked about OE is that you can fix this sort of problem in a > recipe when you need to. Uhm, no. Separate recipes are simply not gonna work for npm, or Go, or Rust without ugly hacking. Those 1000 deps are all pulled by 'npm install' into the source tree of the thing you want to build and then installed together with it into a private directory, having them as separate recipes would mean that you need to fetch and install them separately, which is simply not how npm operates. You can still patch them dependencies just fine if there's just one recipe; once they're pulled in, apply the patches. Alex ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Sum up - Proposal: dealing with language-specific build tools/dependency management tools 2017-03-16 10:35 ` Alexander Kanavin @ 2017-03-16 10:48 ` Jack Mitchell 2017-03-16 11:42 ` Alexander Kanavin 2017-03-16 19:41 ` Patrick Ohly 2017-03-16 15:45 ` Mark Hatle 1 sibling, 2 replies; 25+ messages in thread From: Jack Mitchell @ 2017-03-16 10:48 UTC (permalink / raw) Cc: yocto, openembedded-architecture On 16/03/17 10:35, Alexander Kanavin wrote: > On 03/16/2017 11:30 AM, Paul Barker wrote: >> Staying with NPM as the example, say we have a tree of 1000 >> dependencies for a top-level application recipe. I understand that >> generating and then parsing 1000 recipes will bloat things and slow >> down the build. However, we do need some way to apply patches to those >> dependencies or fix metadata. For example, what if the license is >> stated wrong on npmjs? What if the dependency list itself is wrong? What >> I've always liked about OE is that you can fix this sort of problem in a >> recipe when you need to. > > Uhm, no. Separate recipes are simply not gonna work for npm, or Go, or > Rust without ugly hacking. Those 1000 deps are all pulled by 'npm > install' into the source tree of the thing you want to build and then > installed together with it into a private directory, having them as > separate recipes would mean that you need to fetch and install them > separately, which is simply not how npm operates. > > You can still patch them dependencies just fine if there's just one > recipe; once they're pulled in, apply the patches. > > > Alex > An issue with not having separate recipes is going to cause problems with OTA package updates. I currently have a node package that I maintain for a client and it's so big (excess of 30,000 files with all deps) that we have to have a special update procedure of uninstalling the package, re-installing the new version and then performing a system update. A package update with 30,000 files on a quad core fast arm64 chip takes upwards of 40 minutes. It would be fantastic, if only the core node application and changed dependencies could be updated. Would it be possible to use something similar to the 'auto-package' stuff we already have to split out the inner package dependencies and have specific packages for each dependency but not a separate recipe? Cheers, Jack. ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Sum up - Proposal: dealing with language-specific build tools/dependency management tools 2017-03-16 10:48 ` Jack Mitchell @ 2017-03-16 11:42 ` Alexander Kanavin 2017-03-16 11:47 ` Alexander Kanavin 2017-03-16 19:41 ` Patrick Ohly 1 sibling, 1 reply; 25+ messages in thread From: Alexander Kanavin @ 2017-03-16 11:42 UTC (permalink / raw) To: yocto, openembedded-architecture On 03/16/2017 12:48 PM, Jack Mitchell wrote: > An issue with not having separate recipes is going to cause problems > with OTA package updates. I currently have a node package that I > maintain for a client and it's so big (excess of 30,000 files with all > deps) that we have to have a special update procedure of uninstalling > the package, re-installing the new version and then performing a system > update. A package update with 30,000 files on a quad core fast arm64 > chip takes upwards of 40 minutes. It would be fantastic, if only the > core node application and changed dependencies could be updated. > > Would it be possible to use something similar to the 'auto-package' > stuff we already have to split out the inner package dependencies and > have specific packages for each dependency but not a separate recipe? Yes, certainly. In fact, I think npm.bbclass already does it. Alex ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Sum up - Proposal: dealing with language-specific build tools/dependency management tools 2017-03-16 11:42 ` Alexander Kanavin @ 2017-03-16 11:47 ` Alexander Kanavin 0 siblings, 0 replies; 25+ messages in thread From: Alexander Kanavin @ 2017-03-16 11:47 UTC (permalink / raw) To: yocto, openembedded-architecture [-- Attachment #1: Type: text/plain, Size: 164 bytes --] By the way, Javascript developers do recognize that they have a framework problem. (source: http://thiswas.notinventedhe.re/on/2015-05-21 ) Alex [-- Attachment #2: 2015-05-21.png --] [-- Type: image/png, Size: 100528 bytes --] ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Sum up - Proposal: dealing with language-specific build tools/dependency management tools 2017-03-16 10:48 ` Jack Mitchell 2017-03-16 11:42 ` Alexander Kanavin @ 2017-03-16 19:41 ` Patrick Ohly 1 sibling, 0 replies; 25+ messages in thread From: Patrick Ohly @ 2017-03-16 19:41 UTC (permalink / raw) To: Jack Mitchell; +Cc: yocto, openembedded-architecture On Thu, 2017-03-16 at 10:48 +0000, Jack Mitchell wrote: > An issue with not having separate recipes is going to cause problems > with OTA package updates. I currently have a node package that I > maintain for a client and it's so big (excess of 30,000 files with all > deps) that we have to have a special update procedure of uninstalling > the package, re-installing the new version and then performing a system > update. A package update with 30,000 files on a quad core fast arm64 > chip takes upwards of 40 minutes. It would be fantastic, if only the > core node application and changed dependencies could be updated. If only some files changed, then going from a package-based update mechanism (where a package gets reinstalled when any file gets changed) to a file-based one (like swupd or OSTree) should also address the problem, because then the update detects exactly the modified files and only needs to replace those. Just a thought, in case that auto-packaging turns out to be too hard. -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Sum up - Proposal: dealing with language-specific build tools/dependency management tools 2017-03-16 10:35 ` Alexander Kanavin 2017-03-16 10:48 ` Jack Mitchell @ 2017-03-16 15:45 ` Mark Hatle 1 sibling, 0 replies; 25+ messages in thread From: Mark Hatle @ 2017-03-16 15:45 UTC (permalink / raw) To: Alexander Kanavin, Paul Barker, Josef Holzmayr Cc: Yocto Project, openembedded-architecture On 3/16/17 5:35 AM, Alexander Kanavin wrote: > On 03/16/2017 11:30 AM, Paul Barker wrote: >> Staying with NPM as the example, say we have a tree of 1000 >> dependencies for a top-level application recipe. I understand that >> generating and then parsing 1000 recipes will bloat things and slow >> down the build. However, we do need some way to apply patches to those >> dependencies or fix metadata. For example, what if the license is >> stated wrong on npmjs? What if the dependency list itself is wrong? What >> I've always liked about OE is that you can fix this sort of problem in a >> recipe when you need to. > > Uhm, no. Separate recipes are simply not gonna work for npm, or Go, or > Rust without ugly hacking. Those 1000 deps are all pulled by 'npm > install' into the source tree of the thing you want to build and then > installed together with it into a private directory, having them as > separate recipes would mean that you need to fetch and install them > separately, which is simply not how npm operates. > > You can still patch them dependencies just fine if there's just one > recipe; once they're pulled in, apply the patches. There really shouldn't be a reason why we can't have 'one fetch' for multiple 'recipes'. The concept might be a bit odd, but it also might work reasonably well if architected to work with the individual schemes. --Mark > > Alex > > _______________________________________________ > Openembedded-architecture mailing list > Openembedded-architecture@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-architecture > ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Sum up - Proposal: dealing with language-specific build tools/dependency management tools 2017-03-16 9:30 ` Paul Barker 2017-03-16 10:35 ` Alexander Kanavin @ 2017-03-16 15:43 ` Mark Hatle 1 sibling, 0 replies; 25+ messages in thread From: Mark Hatle @ 2017-03-16 15:43 UTC (permalink / raw) To: Paul Barker, Josef Holzmayr; +Cc: Yocto Project, openembedded-architecture On 3/16/17 4:30 AM, Paul Barker wrote: > On Thu, 16 Mar 2017 09:17:57 +0100 > Josef Holzmayr <holzmayr@rsi-elektrotechnik.de> wrote: > >> I'll give it a shot and try to sum up the current state of affairs in >> this discussion. In accordance to the "Package managers all the way >> down"-presentation, I'm gonna use the term LPM (for language package >> manager) for now on. >> >> *Requirements* >> - repeatable and verifyable licensing of all dependencies that a LPM >> pulls in. >> - locking down specific versions of packages and their dependencies for >> reproductible builds. > > Agreed with these. > >> >> *Optionals* >> - strict separation of fetch, compile, install stages. If a specifc LPM >> requires it, those might be intermingled or left out [Mark] > > I think what we need is strict separation of fetch (allowed to access > network) from the later tasks in a build (not allowed to access > networks). We need to continue to support users behind corporate > firewalls via mirroring and offline builds. I agree here. The 'network fetch', 'do something with it', 'deploy it' Are really the three logical steps we have. The network fetch (as other pieces of this thread mentioned) could include some specific dependency fetching or whatever, but in the end this must be able to be mirrored and disable actual network connections. (network dependency fetching is possibly very tricky as dependency scanning in bitbake happens very early, and there is no way to say "please re-evaluate the dependencies on this branch". [Maybe this is something that needs to be considered for this type of work?) --Mark >> - opaque packaging: similar to static linking, we should at least have a >> way to bundle up a complete application into a single package. Maybe it >> might even be the default (like rust does it at the moment). >> - leverage as much as possible of the functionality the LPMs provide >> instead of reimplementing it. > > We should also aim for automatic generation of recipes from pypi, > npmjs, crates.io, etc using devtool. I'm not sure how much of this has > already been implemented, I'm not entirely up-to-date on devtool > functionality. > > Staying with NPM as the example, say we have a tree of 1000 > dependencies for a top-level application recipe. I understand that > generating and then parsing 1000 recipes will bloat things and slow > down the build. However, we do need some way to apply patches to those > dependencies or fix metadata. For example, what if the license is > stated wrong on npmjs? What if the dependency list itself is wrong? What > I've always liked about OE is that you can fix this sort of problem in a > recipe when you need to. > >> >> *Wish List* >> - separating out the LPM infrastructure into one or more distinct >> layers, not treating it as OE/bitbake core functionality. [Paul] >> - support for the use of multiple languages/LPMs inside a single recipe, >> hopefully even package. [myself] >> >> *Proposed Solutions* >> - having lockdown files shipped with the recipes (in whatever form to be >> defined) >> - leveraging the recipe system to resolve licensing. >> >> If we can boil things down to the common set that we all expect, it will >> in my opnion serve as a blueprint for the actual implementation to follow. >> >> Greetz, > > Thanks, > Paul > ^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [Openembedded-architecture] Sum up - Proposal: dealing with language-specific build tools/dependency management tools 2017-03-16 8:17 ` [Openembedded-architecture] Sum up - " Josef Holzmayr 2017-03-16 9:30 ` Paul Barker @ 2017-03-16 10:21 ` Alexander Kanavin 1 sibling, 0 replies; 25+ messages in thread From: Alexander Kanavin @ 2017-03-16 10:21 UTC (permalink / raw) To: Josef Holzmayr, openembedded-architecture, Yocto Project On 03/16/2017 10:17 AM, Josef Holzmayr wrote: > - locking down specific versions of packages and their dependencies for > reproductible builds. Thanks for summarizing! I have just one comment: at least initially, locking down specific versions has to be optional. We certainly can try our best, but some environments offer no support for it, and even explicitly encourage the opposite approach (Go for instance); Yocto Project has no resources to fight against that. If someone wants to package an app, and lock it down in a reproducible way, they should take their concern directly to upstream. Alex ^ permalink raw reply [flat|nested] 25+ messages in thread
end of thread, other threads:[~2017-03-16 19:41 UTC | newest] Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-03-10 13:49 Proposal: dealing with language-specific build tools/dependency management tools Alexander Kanavin 2017-03-10 14:30 ` [Openembedded-architecture] " Otavio Salvador 2017-03-10 14:48 ` Alexander Kanavin 2017-03-10 14:58 ` Otavio Salvador 2017-03-10 15:10 ` Alexander Kanavin 2017-03-10 15:33 ` Derek Straka 2017-03-10 15:35 ` Derek Straka 2017-03-13 8:25 ` Piotr Figiel 2017-03-13 17:51 ` Bruce, Henry 2017-03-16 10:25 ` Alexander Kanavin 2017-03-10 16:23 ` Mark Hatle 2017-03-10 20:37 ` Josef Holzmayr 2017-03-10 20:49 ` Trevor Woerner 2017-03-11 13:07 ` Josef Holzmayr 2017-03-13 20:58 ` Paul Eggleton 2017-03-16 8:17 ` [Openembedded-architecture] Sum up - " Josef Holzmayr 2017-03-16 9:30 ` Paul Barker 2017-03-16 10:35 ` Alexander Kanavin 2017-03-16 10:48 ` Jack Mitchell 2017-03-16 11:42 ` Alexander Kanavin 2017-03-16 11:47 ` Alexander Kanavin 2017-03-16 19:41 ` Patrick Ohly 2017-03-16 15:45 ` Mark Hatle 2017-03-16 15:43 ` Mark Hatle 2017-03-16 10:21 ` Alexander Kanavin
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.