Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3

[Jasper Orschulko <Jasper.Orschulko@iris-sensing.com>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi Richard and folks,

> The only other way I could imagine this to work would require us to
> keep the manifest file within the meta layer and do a somewhat
> "hacky"
> workaround:
> 
> It is possible to run `repo init` on a local git folder.
> As such, something like this could work:
> 
> 1) create temp folder (e.g. /tmp/repo-XXXXXXXXX)
> 2) run git init in temp folder
> 3) install manifest file into temp folder
> 4) add & commit
> 5) run repo init -u file://$temp-folder

today I discovered that repo also supports the --standalone-manifest
flag, which makes the workaround described above obsolete.
Thus, I believe supporting repo manifests from the layer repo only is
the cleanest and easiest way to implement this and we
wouldn't have to rely on cloning manifest repos during recipe parsing.

However, this would be a breaking change to the current repo fetcher
implementation, as it supports remote manifest repos.
I am pretty confident however that no one uses the arguably already
broken and unusable current repo fetcher implementation
in a production workload so this shouldn't be a major issue.

What are your thoughts on this?

- -- 
With best regards

Jasper Orschulko
DevOps Engineer

Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
Jasper.Orschulko@iris-sensing.com

• • • • • • • • • • • • • • • • • • • • • • • • • •

iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin

https://iris-sensing.com/





On Thu, 2021-11-11 at 12:42 +0100, Jasper Orschulko wrote:
> Hi Richard,
> 
> > Even ten recipes using this will show a degradation in parsing
> > speed
> > and I do
> > get a lot of complaints when parsing slows down for any reason. The
> > user doesn't
> > expect this and it won't be visible what bitbake is doing (sitting
> > at
> > 99% parsed
> > for a period).
> > 
> > Also, the "tend to be small" implies someone will create a huge one
> > at some
> > point even if that is a bad idea for whatever reasons, I just know
> > how these
> > things end up going :(.
> 
> I believe this is something that could be addressed in the
> documentation of the repo fetcher? Something along the lines of
> "Using
> the repo fetcher can increase the time spent in the parsing phase, as
> we need to clone and inspect the manifest repository. To avoid this,
> keep the size of your manifest repository at a minimum."
> 
> When this is done, the dalay is barely noticable. We have a testsetup
> with 24 recipes and reasonable parsing speeds.
> 
> > * Does it only clone a repo in the AUTOREV case?
> 
> No, it also clones with fixed refspecs, as we want to ensure that
> each
> of the git repos referenced in the manifest also uses fixed refspecs.
> Otherwise, the sources in the background might change, while the
> recipe
> still (correctly) references the same revision of the manifest repo.
> 
> > * Could it only obtain the manifest file somehow without a clone of
> > the repo?
> 
> Unfortunately not really. git does not offer a way to list the
> content
> of a file without cloning it beforehand.
> 
> The only other way I could imagine this to work would require us to
> keep the manifest file within the meta layer and do a somewhat
> "hacky"
> workaround:
> 
> It is possible to run `repo init` on a local git folder.
> As such, something like this could work:
> 
> 1) create temp folder (e.g. /tmp/repo-XXXXXXXXX)
> 2) run git init in temp folder
> 3) install manifest file into temp folder
> 4) add & commit
> 5) run repo init -u file://$temp-folder
> 
> The repo manifest could even still be used outside of yocto, by using
> the metalayer repo as basis for the repo init: `repo init -u
> $metalayer-repo -m $path-to-manifest-file`. Not pretty, but it works.
> 
> So if cloning within the parser is a dealbreaker for you, this option
> might be worth looking into.
> 
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGeYnwACgkQYgqew07V
MNWXFAgAkZ7+CRZadUItM+Ay8uePffysuQhlwxHaAgzx1H+4Pk/HKK9QpEy3jFWz
kODr3DPOWlf6eS+AcqpJXVYdMsTucvE34eLIWb8reFkrPQKMtxp0N6U5gqz7e+Bm
Wf4lS04tqGZzC/Ld2DOs/zEnGKEwXBRHMX/9j6YvVMYxwKjxghW2LRloWpzghduH
j0FTAp1aGKLhCQjqeEuNZRMT+rn0i8k6u8Lxel1wRHi2LNhyT3VD3NThjVL64acH
FF83jhaxQ0JxpsSn9ouvJD3wMJHi72rdfGzznuQHmFT0lOxO5AjlcIOtK3KisiAI
KQ/jEV/44WN0JO7OpmHrVyOSfPeVBA==
=H0FZ
-----END PGP SIGNATURE-----