* Public project sstate/hash equivalence mirror now live
@ 2021-09-12 11:46 Richard Purdie
2021-09-12 14:55 ` [OE-core] " Armin Kuster
0 siblings, 1 reply; 6+ messages in thread
From: Richard Purdie @ 2021-09-12 11:46 UTC (permalink / raw)
To: openembedded-core; +Cc: Joshua Watt, Michael Halstead
We made the autobuilder sstate public a while ago but it wasn't useful after we
switched to hash equivalence by default. I'm pleased to be able to report that
we now have a read-only public server for the hash equivalence. We did have to
fix a small API omission and a couple of sstate related bugs but I'm pleased to
be able to report it is now operational and usable. An example config would be:
SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH"
BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687"
I've included an example image build below where it took 11 minutes for a
core-image-minimal image build. Obviously this is only of benefit if you're
building configurations that match the project autobuilder although hash
equivalence could be useful alone in reducing some build times.
I have updated the local.conf sample file in poky to match the new release.
We're not widely highlighting that information as yet as it remains to be seen
how well it scales and what the hosting impacts for it will look like but it is
there to experiment with to a wider audience such as this list.
The builds will sit for a long time whilst it queries hash equivalency and I
appreciate the UI feedback could be better there but one step at a time!
Cheers,
Richard
$ time bitbake core-image-minimal
Loading cache: 100% | | ETA: --:--:--
Loaded 0 entries from dependency cache.
Parsing recipes: 100% |######################################################################################################################################################| Time: 0:00:03
Parsing of 871 .bb files complete (0 cached, 871 parsed). 1519 targets, 49 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
Build Configuration:
BB_VERSION = "1.51.1"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "ubuntu-20.04"
TARGET_SYS = "x86_64-poky-linux"
MACHINE = "qemux86-64"
DISTRO = "poky"
DISTRO_VERSION = "3.3+snapshot-89e48765083f15bf79839a778d08a68f8aece51d"
TUNE_FEATURES = "m64 core2"
TARGET_FPU = ""
meta
meta-poky
meta-yocto-bsp
meta-selftest = "master:89e48765083f15bf79839a778d08a68f8aece51d"
Initialising tasks: 100% |###################################################################################################################################################| Time: 0:03:40
Checking sstate mirror object availability: 100% |###########################################################################################################################| Time: 0:00:28
Sstate summary: Wanted 1525 Local 0 Network 1515 Missed 10 Current 0 (99% match, 0% complete)
NOTE: Executing Tasks
NOTE: Tasks Summary: Attempted 3913 tasks of which 3603 didn't need to be rerun and all succeeded.
real 11m2.578s
user 0m42.927s
sys 0m9.963s
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [OE-core] Public project sstate/hash equivalence mirror now live
2021-09-12 11:46 Public project sstate/hash equivalence mirror now live Richard Purdie
@ 2021-09-12 14:55 ` Armin Kuster
2021-09-12 15:58 ` Richard Purdie
0 siblings, 1 reply; 6+ messages in thread
From: Armin Kuster @ 2021-09-12 14:55 UTC (permalink / raw)
To: Richard Purdie, openembedded-core; +Cc: Joshua Watt, Michael Halstead
On 9/12/21 4:46 AM, Richard Purdie wrote:
> We made the autobuilder sstate public a while ago but it wasn't useful after we
> switched to hash equivalence by default. I'm pleased to be able to report that
> we now have a read-only public server for the hash equivalence. We did have to
> fix a small API omission and a couple of sstate related bugs but I'm pleased to
> be able to report it is now operational and usable. An example config would be:
>
> SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH"
> BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687"
>
> I've included an example image build below where it took 11 minutes for a
> core-image-minimal image build. Obviously this is only of benefit if you're
> building configurations that match the project autobuilder although hash
> equivalence could be useful alone in reducing some build times.
What branches that support hash equivalence will be available other than
Master?
>
> I have updated the local.conf sample file in poky to match the new release.
> We're not widely highlighting that information as yet as it remains to be seen
> how well it scales and what the hosting impacts for it will look like but it is
> there to experiment with to a wider audience such as this list.
The should improve the out-of-the-box experience.
thanks,
> The builds will sit for a long time whilst it queries hash equivalency and I
> appreciate the UI feedback could be better there but one step at a time!
>
> Cheers,
>
> Richard
>
> $ time bitbake core-image-minimal
> Loading cache: 100% | | ETA: --:--:--
> Loaded 0 entries from dependency cache.
> Parsing recipes: 100% |######################################################################################################################################################| Time: 0:00:03
> Parsing of 871 .bb files complete (0 cached, 871 parsed). 1519 targets, 49 skipped, 0 masked, 0 errors.
> NOTE: Resolving any missing task queue dependencies
>
> Build Configuration:
> BB_VERSION = "1.51.1"
> BUILD_SYS = "x86_64-linux"
> NATIVELSBSTRING = "ubuntu-20.04"
> TARGET_SYS = "x86_64-poky-linux"
> MACHINE = "qemux86-64"
> DISTRO = "poky"
> DISTRO_VERSION = "3.3+snapshot-89e48765083f15bf79839a778d08a68f8aece51d"
> TUNE_FEATURES = "m64 core2"
> TARGET_FPU = ""
> meta
> meta-poky
> meta-yocto-bsp
> meta-selftest = "master:89e48765083f15bf79839a778d08a68f8aece51d"
>
> Initialising tasks: 100% |###################################################################################################################################################| Time: 0:03:40
> Checking sstate mirror object availability: 100% |###########################################################################################################################| Time: 0:00:28
> Sstate summary: Wanted 1525 Local 0 Network 1515 Missed 10 Current 0 (99% match, 0% complete)
> NOTE: Executing Tasks
> NOTE: Tasks Summary: Attempted 3913 tasks of which 3603 didn't need to be rerun and all succeeded.
>
> real 11m2.578s
> user 0m42.927s
> sys 0m9.963s
>
>
>
>
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [OE-core] Public project sstate/hash equivalence mirror now live
2021-09-12 14:55 ` [OE-core] " Armin Kuster
@ 2021-09-12 15:58 ` Richard Purdie
2021-09-12 16:01 ` Joshua Watt
0 siblings, 1 reply; 6+ messages in thread
From: Richard Purdie @ 2021-09-12 15:58 UTC (permalink / raw)
To: akuster808, openembedded-core; +Cc: Joshua Watt, Michael Halstead
On Sun, 2021-09-12 at 07:55 -0700, akuster808 wrote:
>
> On 9/12/21 4:46 AM, Richard Purdie wrote:
> > We made the autobuilder sstate public a while ago but it wasn't useful after we
> > switched to hash equivalence by default. I'm pleased to be able to report that
> > we now have a read-only public server for the hash equivalence. We did have to
> > fix a small API omission and a couple of sstate related bugs but I'm pleased to
> > be able to report it is now operational and usable. An example config would be:
> >
> > SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH"
> > BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687"
> >
> > I've included an example image build below where it took 11 minutes for a
> > core-image-minimal image build. Obviously this is only of benefit if you're
> > building configurations that match the project autobuilder although hash
> > equivalence could be useful alone in reducing some build times.
>
> What branches that support hash equivalence will be available other than
> Master?
The question doesn't make sense as hash equivalence doesn't have a "branch".
We're sharing any hash equivalence data generated from builds on the
autobuilder.
> > I have updated the local.conf sample file in poky to match the new release.
> > We're not widely highlighting that information as yet as it remains to be seen
> > how well it scales and what the hosting impacts for it will look like but it is
> > there to experiment with to a wider audience such as this list.
>
> The should improve the out-of-the-box experience.
Ultimately that is the hope.
Cheers,
Richard
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [OE-core] Public project sstate/hash equivalence mirror now live
2021-09-12 15:58 ` Richard Purdie
@ 2021-09-12 16:01 ` Joshua Watt
2021-09-14 15:27 ` Konrad Weihmann
0 siblings, 1 reply; 6+ messages in thread
From: Joshua Watt @ 2021-09-12 16:01 UTC (permalink / raw)
To: Richard Purdie; +Cc: akuster808, openembedded-core, Michael Halstead
[-- Attachment #1: Type: text/plain, Size: 2024 bytes --]
On Sun, Sep 12, 2021, 10:58 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:
> On Sun, 2021-09-12 at 07:55 -0700, akuster808 wrote:
> >
> > On 9/12/21 4:46 AM, Richard Purdie wrote:
> > > We made the autobuilder sstate public a while ago but it wasn't useful
> after we
> > > switched to hash equivalence by default. I'm pleased to be able to
> report that
> > > we now have a read-only public server for the hash equivalence. We did
> have to
> > > fix a small API omission and a couple of sstate related bugs but I'm
> pleased to
> > > be able to report it is now operational and usable. An example config
> would be:
> > >
> > > SSTATE_MIRRORS ?= "file://.*
> http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH"
> > > BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687"
> > >
> > > I've included an example image build below where it took 11 minutes
> for a
> > > core-image-minimal image build. Obviously this is only of benefit if
> you're
> > > building configurations that match the project autobuilder although
> hash
> > > equivalence could be useful alone in reducing some build times.
> >
> > What branches that support hash equivalence will be available other than
> > Master?
>
> The question doesn't make sense as hash equivalence doesn't have a
> "branch".
> We're sharing any hash equivalence data generated from builds on the
> autobuilder.
>
We should (?) backport the BB_HASHSERVE_UPSTREAM patch so that the
instructions to enable it make sense on older branches though.
> > > I have updated the local.conf sample file in poky to match the new
> release.
> > > We're not widely highlighting that information as yet as it remains to
> be seen
> > > how well it scales and what the hosting impacts for it will look like
> but it is
> > > there to experiment with to a wider audience such as this list.
> >
> > The should improve the out-of-the-box experience.
>
> Ultimately that is the hope.
>
> Cheers,
>
> Richard
>
>
[-- Attachment #2: Type: text/html, Size: 3012 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [OE-core] Public project sstate/hash equivalence mirror now live
2021-09-12 16:01 ` Joshua Watt
@ 2021-09-14 15:27 ` Konrad Weihmann
2021-09-14 16:26 ` Richard Purdie
0 siblings, 1 reply; 6+ messages in thread
From: Konrad Weihmann @ 2021-09-14 15:27 UTC (permalink / raw)
To: Joshua Watt, Richard Purdie
Cc: akuster808, openembedded-core, Michael Halstead
On 12.09.21 18:01, Joshua Watt wrote:
>
>
> On Sun, Sep 12, 2021, 10:58 AM Richard Purdie
> <richard.purdie@linuxfoundation.org
> <mailto:richard.purdie@linuxfoundation.org>> wrote:
>
> On Sun, 2021-09-12 at 07:55 -0700, akuster808 wrote:
> >
> > On 9/12/21 4:46 AM, Richard Purdie wrote:
> > > We made the autobuilder sstate public a while ago but it wasn't
> useful after we
> > > switched to hash equivalence by default. I'm pleased to be able
> to report that
> > > we now have a read-only public server for the hash equivalence.
> We did have to
> > > fix a small API omission and a couple of sstate related bugs
> but I'm pleased to
> > > be able to report it is now operational and usable. An example
> config would be:
> > >
> > > SSTATE_MIRRORS ?= "file://.*
> http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH
> <http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH>"
> > > BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687
> <http://typhoon.yocto.io:8687>"
> > >
> > > I've included an example image build below where it took 11
> minutes for a
> > > core-image-minimal image build. Obviously this is only of
> benefit if you're
> > > building configurations that match the project autobuilder
> although hash
> > > equivalence could be useful alone in reducing some build times.
> >
> > What branches that support hash equivalence will be available
> other than
> > Master?
>
> The question doesn't make sense as hash equivalence doesn't have a
> "branch".
> We're sharing any hash equivalence data generated from builds on the
> autobuilder.
>
>
> We should (?) backport the BB_HASHSERVE_UPSTREAM patch so that the
> instructions to enable it make sense on older branches though.
Yes, please - I'd really like to see that in dunfell and hardknott too
>
>
>
> > > I have updated the local.conf sample file in poky to match the
> new release.
> > > We're not widely highlighting that information as yet as it
> remains to be seen
> > > how well it scales and what the hosting impacts for it will
> look like but it is
> > > there to experiment with to a wider audience such as this list.
> >
> > The should improve the out-of-the-box experience.
>
> Ultimately that is the hope.
>
> Cheers,
>
> Richard
>
>
>
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [OE-core] Public project sstate/hash equivalence mirror now live
2021-09-14 15:27 ` Konrad Weihmann
@ 2021-09-14 16:26 ` Richard Purdie
0 siblings, 0 replies; 6+ messages in thread
From: Richard Purdie @ 2021-09-14 16:26 UTC (permalink / raw)
To: Konrad Weihmann, Joshua Watt
Cc: akuster808, openembedded-core, Michael Halstead
On Tue, 2021-09-14 at 17:27 +0200, Konrad Weihmann wrote:
>
> On 12.09.21 18:01, Joshua Watt wrote:
> >
> >
> > On Sun, Sep 12, 2021, 10:58 AM Richard Purdie
> > <richard.purdie@linuxfoundation.org
> > <mailto:richard.purdie@linuxfoundation.org>> wrote:
> >
> > On Sun, 2021-09-12 at 07:55 -0700, akuster808 wrote:
> > >
> > > On 9/12/21 4:46 AM, Richard Purdie wrote:
> > > > We made the autobuilder sstate public a while ago but it wasn't
> > useful after we
> > > > switched to hash equivalence by default. I'm pleased to be able
> > to report that
> > > > we now have a read-only public server for the hash equivalence.
> > We did have to
> > > > fix a small API omission and a couple of sstate related bugs
> > but I'm pleased to
> > > > be able to report it is now operational and usable. An example
> > config would be:
> > > >
> > > > SSTATE_MIRRORS ?= "file://.*
> > http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH
> > <http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH>"
> > > > BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687
> > <http://typhoon.yocto.io:8687>"
> > > >
> > > > I've included an example image build below where it took 11
> > minutes for a
> > > > core-image-minimal image build. Obviously this is only of
> > benefit if you're
> > > > building configurations that match the project autobuilder
> > although hash
> > > > equivalence could be useful alone in reducing some build times.
> > >
> > > What branches that support hash equivalence will be available
> > other than
> > > Master?
> >
> > The question doesn't make sense as hash equivalence doesn't have a
> > "branch".
> > We're sharing any hash equivalence data generated from builds on the
> > autobuilder.
> >
> >
> > We should (?) backport the BB_HASHSERVE_UPSTREAM patch so that the
> > instructions to enable it make sense on older branches though.
>
> Yes, please - I'd really like to see that in dunfell and hardknott too
Readonly hash equivalence is only in hardknott and master, it isn't in dunfell.
The commit in question:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=96b548a79d87120655da3ac5501b8ad4726cf1a4
This means the backport of the recent addition only really makes sense in the
context of hardknott.
Cheers,
Richard
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-09-14 16:26 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-12 11:46 Public project sstate/hash equivalence mirror now live Richard Purdie
2021-09-12 14:55 ` [OE-core] " Armin Kuster
2021-09-12 15:58 ` Richard Purdie
2021-09-12 16:01 ` Joshua Watt
2021-09-14 15:27 ` Konrad Weihmann
2021-09-14 16:26 ` Richard Purdie
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.