Is there a way to speed up the initial fetch task?

Is there a way to speed up the initial fetch task?

[timotheus.giuliani]

[-- Attachment #1: Type: text/plain, Size: 722 bytes --]

Hi,

we run Yocto for pull request builds on cloud runners. Each run starts in a clean workspace from scratch.
There is no local download directory and no option for network drives. The latest SSTATE cache is provided via HTTP.
The changes are small in most cases so that the pull request build spends the most time fetching the sources.
All sources are provided by a mix of private git repositories and tarballs on a private HTTP source mirror.
Fetching big repositories like linux-yocto is in both cases slow, more than 5 minutes.
My question:
Is there a way to speed up the initial fetch task, e.g. by truncating the history of the repository and just fetching the sources needed to build?

Regards,

Timo

[-- Attachment #2: Type: text/html, Size: 830 bytes --]

Re: [yocto] Is there a way to speed up the initial fetch task?

[Alexander Kanavin]

On Thu, 1 Feb 2024 at 13:33, timotheus.giuliani via
lists.yoctoproject.org
<timotheus.giuliani=philips.com@lists.yoctoproject.org> wrote:
> we run Yocto for pull request builds on cloud runners. Each run starts in a clean workspace from scratch.
> There is no local download directory and no option for network drives. The latest SSTATE cache is provided via HTTP.
> The changes are small in most cases so that the pull request build spends the most time fetching the sources.
> All sources are provided by a mix of private git repositories and tarballs on a private HTTP source mirror.
> Fetching big repositories like linux-yocto is in both cases slow, more than 5 minutes.
> My question:
> Is there a way to speed up the initial fetch task, e.g. by truncating the history of the repository and just fetching the sources needed to build?

Don't throw away the download directory, and share it between
builders. This is tested, and expected.

Alex

Re: [yocto] Is there a way to speed up the initial fetch task?

[timotheus.giuliani]

[-- Attachment #1: Type: text/plain, Size: 273 bytes --]

It’s not possible to keep the download directory, since the build runs on cloud runners that are shared resources. Every time a PR is created a new runner instance with an empty workspace will be assigned. Therefore, I can’t reuse anything local from previous builds.

[-- Attachment #2: Type: text/html, Size: 386 bytes --]

RE: [yocto] Is there a way to speed up the initial fetch task?

[Marko, Peter]

[-- Attachment #1: Type: text/plain, Size: 573 bytes --]

> It’s not possible to keep the download directory, since the build runs on cloud runners that are shared resources. Every time a PR is created a new runner instance with an empty workspace will be assigned. Therefore, I can’t reuse anything local from previous builds.



In the same way how you provide sstate-cache mirror https server, you can also provide download mirror https server.



Try this in your distro:

INHERIT += "own-mirrors"

SOURCE_MIRROR_URL = "https://...”

BB_GIT_SHALLOW = “1”

BB_GIT_SHALLOW_DEPTH = “1”



Peter

[-- Attachment #2: Type: text/html, Size: 2828 bytes --]

Re: [yocto] Is there a way to speed up the initial fetch task?

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1387 bytes --]

> The latest SSTATE cache is provided via HTTP.
> All sources are provided by a mix of private git repositories and
tarballs on a private HTTP source mirror.

Is the "private HTTP source mirror" configured as PREMIRROR (not just
MIRROR, see own-mirrors.bbclass) to fetch from there before trying upstream
sites?

If it is, then is it as fast as the sstate mirror you're using? If the
network access is slow then you would see the same issue with sstate, so
I'm surprised that you worry only about fetched sources.

Regards,

On Thu, Feb 1, 2024 at 2:01 PM timotheus.giuliani via lists.yoctoproject.org
<timotheus.giuliani=philips.com@lists.yoctoproject.org> wrote:

> It’s not possible to keep the download directory, since the build runs on
> cloud runners that are shared resources. Every time a PR is created a new
> runner instance with an empty workspace will be assigned. Therefore, I
> can’t reuse anything local from previous builds.
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#62360):
> https://lists.yoctoproject.org/g/yocto/message/62360
> Mute This Topic: https://lists.yoctoproject.org/mt/104095616/3617156
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [
> martin.jansa@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 2287 bytes --]

Re: [yocto] Is there a way to speed up the initial fetch task?

[Alexander Kanavin]

You need to connect the instances to a shared NFS mount for the
downloads then. Ideally, for sstate as well.

Alex

On Thu, 1 Feb 2024 at 14:01, via lists.yoctoproject.org
<timotheus.giuliani=philips.com@lists.yoctoproject.org> wrote:
>
> It’s not possible to keep the download directory, since the build runs on cloud runners that are shared resources. Every time a PR is created a new runner instance with an empty workspace will be assigned. Therefore, I can’t reuse anything local from previous builds.

Re: [yocto] Is there a way to speed up the initial fetch task?

[timotheus.giuliani]

[-- Attachment #1: Type: text/plain, Size: 839 bytes --]

> Is the "private HTTP source mirror" configured as PREMIRROR (not just MIRROR, see own-mirrors.bbclass) to fetch from there before trying upstream sites?
The do_fetch logs shows that my mirror is used and not the upstream. own-mirrors.bbclass just prepends the mirror defined in SOURCE_MIRROR_URL for different protocols to the PREMIRRORS variable.
> If it is, then is it as fast as the sstate mirror you're using? If the network access is slow then you would see the same issue with sstate, so I'm surprised that you worry only about fetched sources.
Maybe it is, but the downloads are ten times bigger than sstate cache. Unfortunately I have no possibility to influence the network setup.
Maybe there is the possibility to bring the downloads size to the minimum necessary to build the current state, without history.

regards,

[-- Attachment #2: Type: text/html, Size: 884 bytes --]

Re: [yocto] Is there a way to speed up the initial fetch task?

[timotheus.giuliani]

[-- Attachment #1: Type: text/plain, Size: 109 bytes --]

These settings had no effect in my case.
Unfortunately will the initial fetch clones the hole repo anyhow.

[-- Attachment #2: Type: text/html, Size: 113 bytes --]

RE: [yocto] Is there a way to speed up the initial fetch task?

[Marko, Peter]

[-- Attachment #1: Type: text/plain, Size: 520 bytes --]


From: timotheus.giuliani via Lists.Yoctoproject.Org <timotheus.giuliani=philips.com@lists.yoctoproject.org>
Sent: Thursday, February 1, 2024 15:36
To: Marko, Peter (ADV D EU SK BFS1) <Peter.Marko@siemens.com>; yocto@lists.yoctoproject.org
Subject: Re: [yocto] Is there a way to speed up the initial fetch task?

> These settings had no effect in my case.
> Unfortunately will the initial fetch clones the hole repo anyhow.

You obviously need to setup and populate the mirror server before using it…
Peter

[-- Attachment #2: Type: text/html, Size: 2349 bytes --]

Re: [yocto] Is there a way to speed up the initial fetch task?

[Leon Woestenberg]

Hi Timotheus,

On Thu, Feb 1, 2024 at 2:01 PM timotheus.giuliani via
lists.yoctoproject.org
<timotheus.giuliani=philips.com@lists.yoctoproject.org> wrote:
>
> It’s not possible to keep the download directory, since the build runs on cloud runners that are shared resources. Every time a PR is created a new runner instance with an empty workspace will be assigned. Therefore, I can’t reuse anything local from previous builds.
>
Not possible?   Although not trivial to setup (depending on what cloud
provider), most cloud runners also allow to keep persistent data (from
a previous run) and pre-prune your fresh runner instance with this
persistent data.

downloads and even sstate-cache might be interesting, especially on
incremental builds.

Search terms: cloud runner cache share persistent

Off-topic for Yocto though, as this is quite general.

Regards, Leon

Re: [yocto] Is there a way to speed up the initial fetch task?

[Khem Raj]

I think if you add a pre-mirror and a CI jobs to populate it regularly
and adding
BB_GENERATE_MIRROR_TARBALLS = "1"

might alleviate some of the git fetching it might be doing on every
build. another option
would be to mount a shared nfs folder into the container as part of
build setup when build is
started, but yeah then its not stateless as much as you would like it to be.

On Thu, Feb 1, 2024 at 5:11 AM Peter Marko via lists.yoctoproject.org
<peter.marko=siemens.com@lists.yoctoproject.org> wrote:
>
> > It’s not possible to keep the download directory, since the build runs on cloud runners that are shared resources. Every time a PR is created a new runner instance with an empty workspace will be assigned. Therefore, I can’t reuse anything local from previous builds.
>
>
>
> In the same way how you provide sstate-cache mirror https server, you can also provide download mirror https server.
>
>
>
> Try this in your distro:
>
> INHERIT += "own-mirrors"
>
> SOURCE_MIRROR_URL = "https://...”
>
> BB_GIT_SHALLOW = “1”
>
> BB_GIT_SHALLOW_DEPTH = “1”
>
>
>
> Peter
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> You automatically follow any topics you start or reply to.
> View/Reply Online (#62361): https://lists.yoctoproject.org/g/yocto/message/62361
> Mute This Topic: https://lists.yoctoproject.org/mt/104095616/1997914
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [yocto] Is there a way to speed up the initial fetch task?

[Richard Purdie]

On Thu, 2024-02-01 at 06:31 -0800, timotheus.giuliani via
lists.yoctoproject.org wrote:
> > Is the "private HTTP source mirror" configured as PREMIRROR (not just MIRROR, see own-mirrors.bbclass) to fetch from there before trying upstream sites?
> The do_fetch logs shows that my mirror is used and not the upstream. own-mirrors.bbclass just prepends the mirror defined in SOURCE_MIRROR_URL for different protocols to the PREMIRRORS variable.
> > If it is, then is it as fast as the sstate mirror you're using? If the network access is slow then you would see the same issue with sstate, so I'm surprised that you worry only about fetched sources.
> Maybe it is, but the downloads are ten times bigger than sstate cache. Unfortunately I have no possibility to influence the network setup. 
> Maybe there is the possibility to bring the downloads size to the minimum necessary to build the current state, without history.

This is what Peter is suggesting with:

BB_GIT_SHALLOW = “1”
BB_GIT_SHALLOW_DEPTH = “1”

however you do need to populate such a mirror.

Cheers,

Richard