Re: [PATCH 01/42] gcc: Add gcc6 recipes

From: Christopher Larson <chris_larson@mentor.com>
To: Paul Eggleton <paul.eggleton@linux.intel.com>
Cc: Bruce Ashfield <bruce.ashfield@windriver.com>,
	Patches and discussions about the oe-core layer
	<openembedded-core@lists.openembedded.org>
Subject: Re: [PATCH 01/42] gcc: Add gcc6 recipes
Date: Thu, 2 Jun 2016 15:18:58 -0700	[thread overview]
Message-ID: <CABcZANmEPWQcfoCT2mQUJHWemPUG5V9-suomGzfT4y4JQt_+mw@mail.gmail.com> (raw)
In-Reply-To: <16461829.3PMlXGZ7eP@peggleto-mobl.ger.corp.intel.com>

[-- Attachment #1: Type: text/plain, Size: 5182 bytes --]

On Thu, Jun 2, 2016 at 12:24 PM, Paul Eggleton <
paul.eggleton@linux.intel.com> wrote:

> On Thu, 02 Jun 2016 15:19:37 Bruce Ashfield wrote:
> > On 2016-06-02 3:15 PM, Paul Eggleton wrote:
> > > On Thu, 02 Jun 2016 18:05:35 Martin Jansa wrote:
> > >> On Wed, Jun 01, 2016 at 10:23:35AM +1200, Paul Eggleton wrote:
> > >>> On Wed, 01 Jun 2016 10:20:23 Paul Eggleton wrote:
> > >>>> On Tue, 31 May 2016 11:12:21 Jussi Kukkonen wrote:
> > >>>>> On 11 May 2016 at 20:35, Khem Raj <raj.khem@gmail.com> wrote:
> > >>>>>> +#BASEURI ?= "${GNU_MIRROR}/gcc/gcc-${PV}/gcc-${PV}.tar.bz2"
> > >>>>>> +BASEURI ?= "git://
> > >>>>>> github.com/gcc-mirror/gcc;branch=gcc-6-branch;protocol=git"
> > >>>>>
> > >>>>> I guess this is where git2_github.com.gcc-mirror.gcc.tar.gz
> download
> > >>>>> comes from? It increased the size of my downloads-directory by
> >30% --
> > >>>>> and I must have quite a bit of old junk in that directory already.
> > >>>>>
> > >>>>> Is there no other solution to this than a 2.5G git copy (honest
> > >>>>> question, I don't know gcc)?
> > >>>>
> > >>>> We went down this road before and it wasn't great for users. There
> is
> > >>>> at
> > >>>> least a tarball for 6.1, we'd presumably need some patches on top of
> > >>>> that (~43 if we were to simply take what's in the gcc-6-branch
> between
> > >>>> 6.1 and bd9a826, minus the "daily bumps").
> > >>>
> > >>> Perhaps that was a little unclear - when I say "we went down this
> road
> > >>> before" I'm agreeing with Jussi - we pointed to a git branch in a
> > >>> previous release with the same resulting huge download for everyone,
> > >>> something I think we should avoid if at all possible.
> > >>
> > >> Maybe it's biggest but at least there is only one copy of it:
> > >>
> > >> top 10 in my downloads (archives and then git clones).
> > >>
> > >> 823M
> > >> /OE/downloads/git2_git.yoctoproject.org.linux-yocto-3.14.git.tar.gz
> > >> 831M
> > >> /OE/downloads/git2_git.yoctoproject.org.linux-yocto-3.10.git.tar.gz
> > >> 893M
> > >> /OE/downloads/git2_git.yoctoproject.org.linux-yocto-3.17.git.tar.gz
> > >> 934M
> > >> /OE/downloads/git2_git.yoctoproject.org.linux-yocto-3.19.git.tar.gz
> > >> 969M
> > >> /OE/downloads/git2_git.yoctoproject.org.linux-yocto-4.1.git.tar.gz
> > >> 1.2G
> > >> /OE/downloads/git2_git.yoctoproject.org.linux-yocto-dev.git.tar.gz
> > >>
> > >> 2.4G    /OE/downloads/git2_github.com.gcc-mirror.gcc.tar.gz
> > >>
> > >> 854M
> /OE/downloads/git2/github.com.qtproject.qtwebengine-chromium.git
> > >>
> > >> 877M    /OE/downloads/git2/git.yoctoproject.org.linux-yocto-3.10.git
> > >> 900M    /OE/downloads/git2/git.yoctoproject.org.linux-yocto-3.14.git
> > >> 921M    /OE/downloads/git2/git.yoctoproject.org.linux-yocto-3.17.git
> > >> 964M    /OE/downloads/git2/git.yoctoproject.org.linux-yocto-3.19.git
> > >> 999M    /OE/downloads/git2/git.yoctoproject.org.linux-yocto-4.1.git
> > >> 1.3G    /OE/downloads/git2/git.yoctoproject.org.linux-yocto-dev.git
> > >> 2.3G    /OE/downloads/git2/git.yoctoproject.org.linux-yocto-4.4.git
> > >
> > > No argument from me there - it does seem a little wasteful given that
> > > there must be a huge overlap between those repos.
> >
> > In other build systems, I've simply been able to reference a common
> > repository for the majority of the blobs. That isn't an option with
> > oe/bitbake (as far as I know).
>
> It's possible we could add functionality along those lines to at least save
> download time; disk usage would probably be unaffected though.
>
> > We could also do shallow clones, but I'm also unsure of how well that
> > is supported.
>
> I know Chris looked into adding this, I don't recall how far he got.

I did work on shallow support, yes.

Shallow mirror tarball support is on the bitbake list awaiting Richard's
review and merge for 2.2, assuming there are no major objections. There is
a caveat, however. Since git's shallow clone capability is limited to
specifying the depth, we can't really do shallow *clones*, as we don't know
the distance from branch HEAD to SRCREV at the time we're going to clone.

What we can support is shallow mirror tarballs, where a mirror is populated
with shallow mirror tarballs which the user's build will fetch, if
available, instead of full mirror tarballs, and this is what my patch
series implements. Of course, each time the recipe SRCREVs change, we'd get
new shallow tarballs that'd need fetching, enough of those would outweigh
the benefits vs just doing a straight up front clone, but I think the
benefit is clear. A shallow kernel tarball for a repo of this sort is
around 100 megs instead of a gig.

We've been using it at mentor for a while now with good results, so far.
I'd love to see someone other than us test out the patches, though, or give
some feedback on either the concept or the code :) See "[master][PATCH 0/3]
Implement support for shallow git mirror tarballs" on the bitbake-devel
list.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 6981 bytes --]