Re: [PATCH] fetch: git: add reference parameter

["Richard Purdie" <richard.purdie@linuxfoundation.org>]

On Mon, 2020-10-19 at 14:56 +0200, Rasmus Villemoes wrote:
> On 19/10/2020 14.28, Richard Purdie wrote:
> > > But, uh, how does this avoid the checksum/sstate issue?
> > > BBFETCH_GIT_REFERENCEDIR will need to be exempt somehow, right? I
> > > mean, I do want to be able to use sstate artifacts generated by
> > > our
> > > CI, but not setting up BBFETCH_GIT_REFERENCEDIR (at least not
> > > necessarily setting it up in exactly the same way).
> > 
> > The dependency code would have no idea the variable is being used
> > as it
> > doesn't know how to peek that deeply inside the fetcher.
> 
> Which is of course totally non-obvious for everybody but a handful of
> people.

Variable dependency tracking does not go into *any* library functions,
be it lib/oe/*, scripts/lib/* or bitbake itself. Whether obvious or
not, it is how it works and always has worked and the line is quite
clear.

>  But that means one might as well make it slightly more flexible and
> not force every reference repo to live in the same directory, and go
> back to the keyword-style
> 
> BBFETCH_GIT_REFERENCEDIR[linux-kernel] = "/some/path"
> BBFETCH_GIT_REFERENCEDIR[chromium] = "/totally/different/place"
> 
> But if you insist, it doesn't matter much, testing suggests that git
> is fine with the reference actually being a symlink to the real repo.

My preference is for simple API where we can. Users are much more used
to SSTATE_DIR = "XXX" rather than setting flags variables for each
repo.

> > > > This avoids all the messing with vardeps, keeps the SRC_URIs
> > > > simple
> > > > and
> > > > should generally be much more usable/scalable?
> > > > 
> > > > If BBFETCH_GIT_REFERENCEDIR is set, we'd also probably want to
> > > > make
> > > > successful clones push references into that directory to cache
> > > > them?
> > > 
> > > Absolutely not, the referenced repository is not to be modified
> > > in
> > > any way.
> > 
> > I'd like something which can be used by others so we need a way to
> > populate/update it. If its read-only in your case, fine, we can
> > detect
> > and not update but in the general case I'd like others to be able
> > to
> > benefit and use this too.
> 
> I really don't see the point of doing that (pushing "references" into
> the referenced repo). Why do you claim that we need a way to
> populate/update it? Keeping a  mirror of the upstream is exactly what
> we
> do under DL_DIR/git2/munged-path-to-upstream, why should some other
> local, but completely external to bitbake, repository also act like
> that? The purpose is just to speed up the initial clone (say, by
> having
> a clone of Linus' upstream repo lying around). I really don't get it.

You need to step back and think about how others might use this. I
think that most users will want to use a single common storage
location. They may have multiple BSPs, each using a kernel git repo so
being able to have a central store of those git references would be
useful. In the common case people will want the first clone to populate
it, the second clone to benefit from the speedup (regardless of which
BSP they build first, or whether they have local clone originally or
not).

I'd actually argue for making it part of the standard DL_DIR and drop
the references inside the other pieces of DL_DIR so its all more space
efficient. Its probably better to do this one step at a time though but
that would be a nice speedup/efficiency to ultimately have.

Whilst I appreciate this isn't exactly the problem you're trying to
solve, I'm interested in the solution which lets us be generally useful
to a majority of users.

Cheers,

Richard