Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Wed, 2021-11-10 at 13:52 +0000, Jasper Orschulko wrote:
> Hi Richard,
> 
> > When you say "fixed refspec", will that be a definitive sha revision
> > or a tag?
> > We always force resolution of tags as they tend to cause problems and
> > can change even if it is bad form.
> 
> that's a good point. Actually, Martin and I have just been discussing
> this, as we noticed that this point actually got "lost" during our
> implementation. We are currently working on fixing this. Good to know
> how you handle this. I will keep you posted.

Ok, it is good to be clear on that one. I know the fact we hit the network for
tags does concern some but it really is the only way to handle them.

> > This is potentially a big issue. Cloning operations during parsing is
> > pretty
> > horrible. We'd not expect any thing being written out like that
> > during a parse.
> > It would probably work "ok" for one recipe but if you start getting
> > the hundreds
> > of git recipes we have in some layers, it wouldn't scale if we
> > allowed that :(.
> > 
> > Not sure what to recommend here but it is definitely problematic.
> 
> Just to make sure that we are on the same page: This ONLY affects
> recipes which use the repo fetcher. And it ONLY clones the repository
> containing the repo manifest (which tend to be small in size).

Correct, we are on the same page. This is still quite problematic as the recipes
are meant to parse quickly and a repository clone is definitely not expected.

> So unless developers start using hundreds of repo-based recipes, which I
> find a very unlikely scenario, this should not be an issue.

Even ten recipes using this will show a degradation in parsing speed and I do
get a lot of complaints when parsing slows down for any reason. The user doesn't
expect this and it won't be visible what bitbake is doing (sitting at 99% parsed
for a period).

Also, the "tend to be small" implies someone will create a huge one at some
point even if that is a bad idea for whatever reasons, I just know how these
things end up going :(.

> Unfortunately, I don't see any other way to access the repo manifest
> file, as we need to calculate the commit hashes of the git repos
> referenced in the repo manifest file. Otherwise, it is impossible for
> us to determinate the necessity of an update when SRCREV =
> "${AUTOREV}". 

Some further questions:

* Does it only clone a repo in the AUTOREV case?
* Could it only obtain the manifest file somehow without a clone of the repo?

> However, I see one potential improvement here. Currently the cloning of
> the manifest repo is done on a per-recipe basis. E.g. this means if we
> have 10 recipes inheriting a bbclass containing a repo fetcher, we will
> clone 10 identical manifest repos. We'll work on improving this.

At least for wget or git, it is assumed that for a given url, there would be one
tarball/clone and that there is locking in place to share it between them. This
means you'll see do_fetch tasks for binutils, binutils-cross-XXX, nativesdk-
binutils and binutils-native and one will block the others but the fetch will
happen once and be shared between them. I guess with repo it may not be as
simple as that but we should try and share what we can if possible.

Cheers,

Richard