Re: [bitbake-devel] Improving npm(sw) fetcher & integration within Bitbake

[Stefan Herbrechtsmeier <stefan.herbrechtsmeier-oss@weidmueller.com>]

Am 06.11.2021 um 11:06 schrieb Richard Purdie:
> On Fri, 2021-11-05 at 16:02 +0000, Jasper Orschulko wrote:
>> Thanks @all for your input.
>>
>> It seems like there is a lot to be considered. When it comes to under-
>> the-hood development, the yocto project is fairly new to Martin and
>> myself. The term "bitbake requirements" has been used more than once in
>> this conversation, yet I am currently not aware of any formal
>> requirements. You wouldn't happen to have some documentation on this by
>> any chance? A common understanding of this and the actual usecases for
>> npm would be great, before we go into further details on how a possible
>> solution might look like.
> 
> We've used that term as there is a set of things which the bitbake fetcher is
> expected to provide in terms of workflows and user experiences, effectively what
> is it's API. Sadly there isn't a formal document, just a long understanding of
> what this looks like. It isn't a piece of the project that people have wanted to
> change too often so writing such documentation hasn't been a priority.
> 
> I can try and briefly summarise the expectations I can think of in the list
> below and we should perhaps just put this as a text file in the fetcher
> directory of the codebase. If others see anything I'm missing please add to the
> list.
> 
> Just to be clear, we're not trying to be awkward about making changes, there is
> just a long history with this codebase and end users tend to get upset if we
> break the API. Different users do have very different use cases from the code.
> 
> 
> a) network access for sources is only expected to happen in the do_fetch step.
> This is not enforced or tested but is required so that we can:
> 
>    i) audit the sources used (i.e. for license/manifest reasons)
>    ii) support offline builds with a suitable cache
>    iii) allow work to continue even with downtime upstream
>    iv) allow for changes upstream in incompatible ways
>    v) allow rebuilding of the software in X years time
> 
> b) network access is not expected in do_unpack
> 
> c) you can take DL_DIR and use it as a mirror for offline builds
> 
> d) access to the network is only made when explicitly configured in recipes
>     (e.g. use of AUTOREV, or use of git tags which change revision)
> 
> e) fetcher output is deterministic
>     (i.e. if you fetch configuration XXX now it will match in future exactly in
>      a clean build with a new DL_DIR)
> 
> f) network access is expected to work with the standard linux proxy variables
>     so that access behind firewalls works (the fetcher sets these in the
>     environment but only in the do_fetch tasks)
> 
> g) access during parsing has to be minimal, a "git ls-remote" for an AUTOREV
>     git recipe might be ok but you can't expect to checkout a git tree
> 
> h) we need to provide revision information during parsing such that a version
>     for the recipe can be constructed.
> 
> i) versions are expected to be able to increase in a way which sorts allowing
>     package feeds to operate (see PR server required for git revisions to sort)
> 
> j) API to query for possible version upgrades of a url is highly desireable to
>     allow out automated upgrage code to function (it is implied this does always
>     have network access)
> 
> k) Where fixes or changes to behaviour in the fetcher are made, we ask that
>     test cases are added (run with "bitbake-selftest bb.tests.fetch"). We do
>     have fairly extensive test coverage of the fetcher as it is the only way
>     to track all of it's corner cases, it still doesn't give entire coverage
>     though sadly.
> 
> Not all fetchers support all features, autorev is optional and doesn't make
> sense for some. Upgrade detection means different things in different contexts
> too.

Thanks for the list and I would like to see that as part of the bitbake 
source tree.

> Also, I did realise the npm fetcher tests simply don't work anymore. They're not
> run as standard as our infrastructure didn't have npm on it to run the tests so
> they have bitrotted :(.

Is it possible to add the nodejs recipe to the core?