[PATCH RFC 00/21] Git repository sharing for kernel (and other) repos

[PATCH RFC 00/21] Git repository sharing for kernel (and other) repos

[Paul Gortmaker]

It doesn't seem that long ago (kernel 3.x era) that the kernel repo was
less than 1G in size, and while it wasn't actively shared, it was kind of
one of those "we'll deal with that later" type things.

But today, where more people care about CI/CD - if you make use of both
linux-yocto and linux-yocto-dev - well, you are looking at over 3G and
2.2G respectively.  So that is over 5G - and we still don't have any
effective sharing between them, or any formalized method for which another
derived kernel in another layer can share and avoid duplicating downloaded
data.  So, add in a 3rd party kernel, such as a SoC variant (e.g. r-pi
layer) and you've now got another 3G of kernel, and are up to 8G.

What I'm going to show is we can do all that in less than two.

OK, so what are we doing at a relatively high level?  We are treating
external repositories (mainline, stable, preempt-rt) as core building
blocks, and exposing them as "jumping off" reference points for our two
core kernel repositories, and for any other 3rd party kernel derivatives,
so that we never download the same common git objects twice.  In addition,
we selectively download/fetch from the stable and -rt repos so we only
pull a few megs and skip the dead end leaf nodes based on 2.x and 3.x and
4.x kernel versions we simply don't care about anymore.  We use the same
selective approach on our linux-yocto and linux-yocto-dev repos as well.

This is all achieved right at the fetch level, using what are essentially
"fetch-only" meta-packages that establish the core components to be
shared/referenced.  These fetch only recipes are in turn triggered by the
parent recipe having a fetch dependency on their desired reference.  

Right now, if you do:

     bitbake -c cleanall linux-yocto-dev ; bitbake -c fetch linux-yocto-dev

what will happen is that you will w-get 920M of a pre-mirror tarball of a
2014 copy of a 3.17 yocto-dev kernel and all its useless branches.  It
gets untarred (and deleted) and then a fetch chews and grinds on it doing
the mega uplift from 3.17 --> 5.12 for a final 2.2G repo spread across
five git packs (4 from 2014 and one new one).  This all takes 10-15
minutes even on a well connected fast machine.  Whee!

With this series, the above fetch will take ~30s because we never throw
away the mainline/stable/rt content, and hence only fetch less than 20MB
of yocto-dev specific git objects from the yocto server/repo.

The kernel as components:
-------------------------
Lets look at what we really need, in order to build any BSP on v5.10 from
linux-yocto, and similarly for v5.12 on linux-yocto-dev; showing the sizes
of repacked git bare repos, and references symbolically, with the core
mainline changes up to and including v5.10 as shared:

1.5G    git.kernel.org.torvalds.linux-5.10
35M     |--> git.kernel.org.stable.linux-5.10.y
7.1M         |--> git.kernel.org.preempt-rt.linux-5.10.y
38M               |--> git.yoctoproject.org.linux-yocto.git

        linux-5.10 (shared base)
148M    |--> git.kernel.org.torvalds.linux-master (v5.10 --> latest)
2.9M         |--> git.kernel.org.stable.linux-5.12.y
5.6M              |--> git.kernel.org.preempt-rt.linux-5.12.y
18M                    |--> git.yoctoproject.org.linux-yocto-dev.git

1516+35+7+38+148+3+6+18 = 1771.  So that is 1.7G.  Add in ~100MB more for
stable-5.4 and v5.4-rt and yocto v5.4 BSPs if you want them too.  That
would be pretty sweet, right?  A heck of a lot less than 5 gigs!   But can
we do that?  Actually, yes we can.  And we can open the door for other
sharing as well, and even split that big 1.5G chunk into smaller chunks.

A bit more on that sharing, since it is important, and easy to gloss over.
Consider the r-pi kernel repo, for no other reason than everyone is
familiar with it.  As of today, if you clone it w/o making any effort to
share/reference, you'll get just shy of 3G.  Instead, if you selectively
fetch the rpi-5.10 (and reference stable-5.10 above) you'll get 6M and
another 4M for rpi-5.12 (referencing linux-master above).  So you'd be
using 10M of that 3G (about 0.33%) but since we've not made it easy for
sharing, many custom SoC/SDK kernels are similar, and you clone a whole
bunch of stuff with 99% shared DNA with what you already have.

Similar sizes/factors are in play when we consider the linux-stable and
preempt-rt repositories - multiple gigs as a whole, but as we see above,
the selected v5.10 components are only 35M and 7M respectively.

Objectives:
-----------
There were some key goals at play, even if I sort of back-declare them
now with some level of revisionist reflection:

-enable sharing off of key universally public reference repos for everyone

-compartmentalize tech blocks to eliminate overlapping downloads of git objects

-reduce download size further where possible (exclusion of content/gc/repack)

-be ready to absorb further kernel growth and also EOL dead-end leaf nodes

-make only minimal specific objective based changes to the git fetcher

-remain compatible with the generally accepted Yocto workflows/design.

With that in mind, it makes sense to now consider the above requirements
in a bit more detail.

Consider the 1st and last together - share + compatible.  Adding a one
line patch to the git fetcher to support "--reference" would enable a
level of sharing but we'd have stuff spilling outside the normal download
paths, and absolute paths inside SRC_URIs and non-portable (pre)mirror
tarballs - failing miserably on the compatible goal.  So we start by only
allowing repos to reference others which are peers in the download dir -
no /home/bob/my-super-kernel type stuff.  This goes a long way towards
keeping a portable download dir and SRC_URIs clean of absolute paths, and
remaining compatible with the (pre)mirror type Yocto work flow and use
cases.  As will be seen, we don't even really expose the "--reference"
flag use outside of the internal fetcher code.

Technology blocks -- Looking back ~10 years, we did a lot more "kernel
patching" (via git am) than just using (and merging) pre-applied commits
in a branch of a technology feature - such as "stable" or "preempt-rt".
But now the Yocto tree builds on the -rt tree which builds on stable which
builds on mainline.  We can see how they are chained together in the
ascii nesting diagram above.  If we ignored this internal ordering, we
could end up with the stable content (git objects) duplicated inside the
preempt-rt repo and even duplicated in the linux-yocto repos themselves.
By exposing that building block on building block, we also get the
universally recognized share points (mainline, stable-5.x, rt-5.x) that
can be used by anyone to complete their full git object history.

Reduce/exclude - as time has moived on, more and more repos are of such
size and varying content, that using "--mirror" and "refs/*:refs/*" as a
"grab everything" approach simply doesn't scale.  So instead we allow
a selective clone/fetch of just what we need and exclude everything else.

Reduce/garbage-collect - there are opportunities for "low hanging fruit"
in terms of getting rid of unused references/tags in linux-yocto[-dev] but
since I can't directly control what is in those repos (or tar mirrors of
them) we'll have to pursue that outside of this changeset.

Reduce/repack - most people don't realize but the size of the pack you get
from a server depends on how generous the server is with its CPU time.
And that multiple packs can be significantly larger than one single pack.
While we can't control individual servers, we can (and should) consider
agressively repacking any repo we put into service for a (pre)mirror.  That
includes any repo content we encourage compartmentalizing in this changeset.

Growth - The epoch --> v5.10 block of mainline commits is static and need
not ever change from the group of objects we have in its pack today.  The
v5.10 is the effective merge-base between linux-yocto and -dev currently,
and as such makes a sensible line in the sand for sharing.  We currently
have v5.10+ up to mainline/master coverage in the next repo in the chain,
but it is trivial to create/insert a new block of static content covering
v5.10 to v5.13 between v5.10 and master in the future to absorb new growth
in manageable chunk sizes.  We already add/use the same technology here 
to opptionally "split" the v5.10 block for those who need sub-gigabyte
downloads (and repositories) for infrastructural reasons.

EOL - While the repo sizes for the stable and -rt chunks above may not
seem significant compared to the v5.10 basline size, they are specific to
a particular baseline as leaf-node content.  As such we can simply unlink
them from our SRC_URI driven reference chains after which they won't
appear (or download) for new users/builds but they will remain in people's
old download dirs and build workspaces.

Fetcher - we add support of sharing through a "--reference" like manner,
and enforce our relative path requirements there.  We also see a clear
need for selective clone/fetch as per above, so we allow clone args to
replace "--mirror" with "--single-branch" and an override of the fetch
args and their current default of everything via. "refs/*:refs/*".
Finally, if we want to have the stable-5.4, the stable-5.10 and the
stable-5.12 as separate content for independent introduction and EOL,
then we also have to allow them to be in separate repos in the download
dir, even though they all were sourced from the same server/path/repo.
This is achieved by allowing an optional recipe specified download name.

I won't go into more detail here, since the fetcher commits all have
proper commit logs and make sense in their own right, independent of the
larger overall goals described above.  Similarly, all the kernel recipe
changes provide the working example/context of how all the fetcher changes
are used.  So even though the two groups are separate repos, I've chosen
to present it all together against the poky repo, at least initially.

Next Steps:
-----------
With this being a functional implementation, it seems like a good time to
get other people looking at it.  Ideally step #1 will be getting general
agreement that this is something we need, something that is overdue,
and that the implementation as shown here makes sense in the absence of
any similar effort from anyone that does the same but in a better way.

From there, we'll want more people not just looking at it, but testing it
as well.   I know I want to write a commit (script?) that will avoid any
"transition tax" by prepopulating new repos with "old" already downloaded
git objects where we can.  And to add/do tests with my own popluated mirror
and NO_NETWORK, and also try to ensure nothing in BB_SHALLOW gets upset,
but I wasn't going to hold up starting a review of this any longer.

I suspect I can get some co-workers using/testing it too, but Yocto gets
used in a bunch of different ways by different groups, so we'll no doubt
have to do some additional fixups to ensure everybody gets the benefits of
this sharing.  But I'm hopeful that when people see the benefits above,
they'll pitch in to help take this the final mile by ensuring it works for
their use case as well.

I'm not too worried about pontificating out beyond that until we get past
the acceptance/testing hurdles outlined above.  So, please do have a read
of the commits, kick the tires, put on your bikeshedding clothes and grab
a brush, and lets see where it goes from here...

https://github.com/paulgortmaker/poky/compare/reference-RC1

---

Paul Gortmaker (21):
  bitbake: fetch2/git: allow override of clone args with GITCLONEARGS
  bitbake: fetch2/git: allow limiting upstream fetch refs to a subset
  bitbake: fetch2/git: allow optional git download name overrride
  bitbake: fetch2/git: allow specifying repos as static/unchanging
  bitbake: fetch2/git: ensure static repos have at least one refs/heads
  bitbake: fetch2/git: allow alt references within download dir
  bitbake: fetch2/git: append new altref line if/when SRC_URI changed value
  bitbake: fetch2/git: allow pack references within download dir
  bitbake: fetch2/git: use constant names for packs in static repos
  kernel: add basic boilerplate for fetch-only recipes
  kernel: add a fetch-only recipe for mainline v5.10 source
  kernel: allow splitting mainline v5.10 source download in two
  kernel: allow splitting mainline v5.10 source download in three
  kernel: allow splitting mainline v5.10 source download in four
  kernel: add recipe for linux-master (mainline latest)
  kernel: add stable fetch recipes for v5.4.x, v5.10.x and v5.12.x
  kernel: add preempt-rt fetch recipes for v5.4.x, v5.10.x and 5.12.x
  kernel: make v5.4.x Yocto recipes use shared source
  kernel: make v5.10.x Yocto recipes use shared source
  kernel: make linux-yocto-dev recipe use shared source
  kernel: disable (pre)mirror for linux-yocto and linux-yocto-dev

 .../bitbake-user-manual-fetching.rst          |  24 ++++
 bitbake/lib/bb/fetch2/git.py                  | 135 +++++++++++++++++-
 documentation/ref-manual/variables.rst        |  22 +++
 meta/recipes-kernel/linux/fetch-linux.inc     |  20 +++
 meta/recipes-kernel/linux/fetch-only.inc      |  20 +++
 meta/recipes-kernel/linux/fetch-rt.inc        |  25 ++++
 meta/recipes-kernel/linux/fetch-stable.inc    |  24 ++++
 meta/recipes-kernel/linux/linux-3.3.bb        |   9 ++
 meta/recipes-kernel/linux/linux-3.8.bb        |   9 ++
 meta/recipes-kernel/linux/linux-4.0.bb        |   9 ++
 meta/recipes-kernel/linux/linux-4.12.bb       |  10 ++
 meta/recipes-kernel/linux/linux-4.18.bb       |  10 ++
 meta/recipes-kernel/linux/linux-4.3.bb        |  10 ++
 meta/recipes-kernel/linux/linux-5.10.bb       |  38 +++++
 meta/recipes-kernel/linux/linux-master.bb     |  13 ++
 meta/recipes-kernel/linux/linux-rt-5.10.bb    |   9 ++
 meta/recipes-kernel/linux/linux-rt-5.12.bb    |  12 ++
 meta/recipes-kernel/linux/linux-rt-5.4.bb     |   9 ++
 meta/recipes-kernel/linux/linux-yocto-dev.bb  |  11 +-
 .../linux/linux-yocto-rt_5.10.bb              |   7 +-
 .../linux/linux-yocto-rt_5.4.bb               |   7 +-
 .../linux/linux-yocto-tiny_5.10.bb            |   7 +-
 .../linux/linux-yocto-tiny_5.4.bb             |   7 +-
 meta/recipes-kernel/linux/linux-yocto.inc     |  10 ++
 meta/recipes-kernel/linux/linux-yocto_5.10.bb |   7 +-
 meta/recipes-kernel/linux/linux-yocto_5.4.bb  |   7 +-
 meta/recipes-kernel/linux/stable-5.10.bb      |  10 ++
 meta/recipes-kernel/linux/stable-5.12.bb      |  16 +++
 meta/recipes-kernel/linux/stable-5.4.bb       |  11 ++
 29 files changed, 498 insertions(+), 10 deletions(-)
 create mode 100644 meta/recipes-kernel/linux/fetch-linux.inc
 create mode 100644 meta/recipes-kernel/linux/fetch-only.inc
 create mode 100644 meta/recipes-kernel/linux/fetch-rt.inc
 create mode 100644 meta/recipes-kernel/linux/fetch-stable.inc
 create mode 100644 meta/recipes-kernel/linux/linux-3.3.bb
 create mode 100644 meta/recipes-kernel/linux/linux-3.8.bb
 create mode 100644 meta/recipes-kernel/linux/linux-4.0.bb
 create mode 100644 meta/recipes-kernel/linux/linux-4.12.bb
 create mode 100644 meta/recipes-kernel/linux/linux-4.18.bb
 create mode 100644 meta/recipes-kernel/linux/linux-4.3.bb
 create mode 100644 meta/recipes-kernel/linux/linux-5.10.bb
 create mode 100644 meta/recipes-kernel/linux/linux-master.bb
 create mode 100644 meta/recipes-kernel/linux/linux-rt-5.10.bb
 create mode 100644 meta/recipes-kernel/linux/linux-rt-5.12.bb
 create mode 100644 meta/recipes-kernel/linux/linux-rt-5.4.bb
 create mode 100644 meta/recipes-kernel/linux/stable-5.10.bb
 create mode 100644 meta/recipes-kernel/linux/stable-5.12.bb
 create mode 100644 meta/recipes-kernel/linux/stable-5.4.bb

-- 
2.25.1

[PATCH 01/21] bitbake: fetch2/git: allow override of clone args with GITCLONEARGS

[Paul Gortmaker]

Currently there is no way to avoid the "--mirror" argument when doing an
initial clone.  This is problematic in that there are now a significant
number of repos out there for which we want the content, but creating a
full mirror clone isn't practical or efficient.

The two most common examples are:

1) Repos that have "ingested" another repo with no common ancestors - we
see this in "stable-queue" and "linux-rt-devel" repos[1] which started
as repos of queued format-patch output, but then also added the baseline
so the patches could be obtained "pre-applied" for easier automated testing.
Obviously, ingesting all of the linux history grows a repo by gigabytes.

2) Repos that have gone forward in time in a direction we don't care about,
say for example a flag day event, where everything was rewritten from C into
rust in one massive commit, or where a merge of a giant wad of orthogonal
history and/or binary/proprietary content "pollutes" the repo.

In both cases, we can solve the problem by using "--single-branch"
optionally in conjunction with "--branch <goodstuff>" in order to limit
the cloned content to a specific branch or tag.  We will only get
content up to that point, and any commits/ancestors leading up to that
point.  The clone is fully functional and without any of the limitatons
imposed by shallow clones.

So, in case #1 we simply choose the branch we want - raw unapplied
content, or applied and tagged (and perhaps signed) - but we aren't
subjected to downloading both branches.   In case #2 we simply specify
the last known "good" tag before the upstream went off in a direction
that we don't care about.

Note that default behaviour is unchanged - so this is an opt-in feature
which won't impact any existing recipes.  We also leave the door open
for it being URL specific via <name> since we know we have multi repo
SRC_URI in recipes-kernel already.

[1] git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git
    git://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 bitbake/lib/bb/fetch2/git.py           | 3 ++-
 documentation/ref-manual/variables.rst | 8 ++++++++
 2 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index e3ba80a3f52a..22281e2cfb98 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -345,10 +345,11 @@ class Git(FetchMethod):
 
         # If the repo still doesn't exist, fallback to cloning it
         if not os.path.exists(ud.clonedir):
+            extcloneargs = d.getVar('GITCLONEARGS_' + ud.names[0]) or d.getVar('GITCLONEARGS') or "--bare --mirror"
             # We do this since git will use a "-l" option automatically for local urls where possible
             if repourl.startswith("file://"):
                 repourl = repourl[7:]
-            clone_cmd = "LANG=C %s clone --bare --mirror %s %s --progress" % (ud.basecmd, shlex.quote(repourl), ud.clonedir)
+            clone_cmd = "LANG=C %s clone %s %s %s --progress" % (ud.basecmd, extcloneargs, shlex.quote(repourl), ud.clonedir)
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, clone_cmd, ud.url)
             progresshandler = GitProgressHandler(d)
diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst
index 0310429bdcab..cd2b77604e87 100644
--- a/documentation/ref-manual/variables.rst
+++ b/documentation/ref-manual/variables.rst
@@ -2649,6 +2649,14 @@ system and gives an overview of their function and contents.
    :term:`GDB`
       The minimal command and arguments to run the GNU Debugger.
 
+   :term:`GITCLONEARGS`
+      The arguments used to "git clone" when cloning a repo from an
+      an external server into the local download area.  The default is
+      "--bare --mirror".  Use of "--bare --single-branch --branch foo"
+      will limit what is cloned/downloaded just to the branch/tag "foo".
+      An URL specific GITCLONEARGS_<name> will take precedence over a
+      recipe wide GITCLONEARGS setting, if both are present.
+
    :term:`GITDIR`
       The directory in which a local copy of a Git repository is stored
       when it is cloned.
-- 
2.25.1

[PATCH 02/21] bitbake: fetch2/git: allow limiting upstream fetch refs to a subset

[Paul Gortmaker]

If a clone in the download directory is not static, and was created with
single-branch to avoid additonal unwanted content, then the fetcher will
come along and spoil that effort by unconditonally getting refs/* from
the server and downloading everything available at the server.

To that end, allow a companion variable that can optionally be used to
limit the fetch to a subset of references that are in line with the
desired content for the recipe in question, and so that large unwanted
downloads aren't inadvertently triggered during an update.

If not specified, the existing "fetch everything" behaviour remains.

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 bitbake/lib/bb/fetch2/git.py           | 3 ++-
 documentation/ref-manual/variables.rst | 8 ++++++++
 2 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index 22281e2cfb98..b54ec76d7174 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -362,7 +362,8 @@ class Git(FetchMethod):
               runfetchcmd("%s remote rm origin" % ud.basecmd, d, workdir=ud.clonedir)
 
             runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=ud.clonedir)
-            fetch_cmd = "LANG=C %s fetch -f --progress %s refs/*:refs/*" % (ud.basecmd, shlex.quote(repourl))
+            fetchrefs = d.getVar('GITFETCHREFS_' + ud.names[0]) or d.getVar('GITFETCHREFS') or "refs/*:refs/*"
+            fetch_cmd = "LANG=C %s fetch -f --progress %s %s" % (ud.basecmd, shlex.quote(repourl), fetchrefs)
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
             progresshandler = GitProgressHandler(d)
diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst
index cd2b77604e87..55028987c08a 100644
--- a/documentation/ref-manual/variables.rst
+++ b/documentation/ref-manual/variables.rst
@@ -2661,6 +2661,14 @@ system and gives an overview of their function and contents.
       The directory in which a local copy of a Git repository is stored
       when it is cloned.
 
+   :term:`GITFETCHREFS`
+      The refs requested during a "git fetch" when updating a repo from
+      an external server into the local download area.  The default is
+      "refs/*:refs/*".  Used in conjunction with GITCLONEARGS to limit what
+      is cloned/fetched from a repo otherwise dense with branches.  Note
+      that an URL specific GITFETCHREFS_<name> will take precedence over
+      a recipe wide GITFETCHREFS setting, if both are present.
+
    :term:`GLIBC_GENERATE_LOCALES`
       Specifies the list of GLIBC locales to generate should you not wish
       to generate all LIBC locals, which can be time consuming.
-- 
2.25.1

[PATCH 03/21] bitbake: fetch2/git: allow optional git download name overrride

[Paul Gortmaker]

Single repositories containing multiple independent origin commits are
becoming more common.  For example the rt development repo has patches
as "raw" format-patch output in a patch series, and also for ease of
automated testing, them all pre-applied (ff and non-ff) to the original
baseline (v5.10.8 in this case):

   linux-rt-devel$ git merge-base linux-5.10.y-rt linux-5.10.y-rt-rebase
   7a1519a74f3d0b06598fb95387688cde41e465d8
   linux-rt-devel$ git describe 7a1519a74f3d0b06598fb95387688cde41e465d8
   v5.10.8
   linux-rt-devel$ git merge-base linux-5.10.y-rt linux-5.10.y-rt-patches
   linux-rt-devel$
   linux-rt-devel$ git log --oneline linux-5.10.y-rt | tail -n1
   1da177e4c3f4 Linux-2.6.12-rc2
   linux-rt-devel$ git log --oneline linux-5.10.y-rt-patches | tail -n1
   96f1ac50158c [ANNOUNCE] 4.1.2-rt1

As can be seen, there are two different "epoch" commits and no history
or shared objects exist between the patches branch and the applied
branches (on the linux-stable baseline).

Since we construct the download directory name based on the server name
and the server's path to the repository, we currently can't refactor that
content into two locally independent paths/repositories in the download
dir for efficiency, ease of use and tarball/MIRROR handling.

To resolve this, we allow the recipe to optionally specify the path
component of the download dir name, so that a server with a repo of N
independent branches/histories can be used in N different recipes (or N
different SRC_URI lines) without overlap or name space collision in the
download dir.  Continuing with the above example, we could have:

  RT_REPO = "git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git"
  SRC_URI = "${RT_REPO};name=applied;dlname=.preempt-rt.linux-5.10.y \
             ${RT_REPO};name=patches;dlname=.preempt-rt.patches-5.10"

...resulting in the following independent repository dirs:
     downloads/git2/git.kernel.org.preempt-rt.linux-5.10.y
     downloads/git2/git.kernel.org.preempt-rt.patches-5.10

If the new "dlname" parameter is not specified, then the automatic name
generation functions just as it always has done and this change has no
effect on any existing download dirs.

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 .../bitbake-user-manual-fetching.rst              |  6 ++++++
 bitbake/lib/bb/fetch2/git.py                      | 15 ++++++++++++++-
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
index e9a5f336dfd2..0f12eaa3a81f 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
@@ -417,6 +417,12 @@ This fetcher supports the following parameters:
    is assumed to be "master". The number of branch parameters much match
    the number of name parameters.
 
+-  *"dlname":* The download name used in downloads/git2 (or GITDIR).  If
+    specified,  this will override the automatically constructed name
+    component from "thispath" for the URL git://some.host/thispath to the
+    user specified value.  The component from "some.host" remains unaffected.
+    Allows splitting of one repo into multiple independent download dirs.
+
 -  *"rev":* The revision to use for the checkout. The default is
    "master".
 
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index b54ec76d7174..de698a3d9f24 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -18,6 +18,13 @@ Supported SRC_URI options are:
    SRCREV_nameX = "xxxxxxxxxxxxxxxxxxxx"
    SRCREV_nameY = "YYYYYYYYYYYYYYYYYYYY"
 
+- dlname
+   The download name used in downloads/git2 (or GITDIR).  This will override
+   the automatically constructed name component from "thispath" for the URL
+   git://some.host/thispath to the user specified value.  The component from
+   "some.host" remains unaffected.  Allows splitting of one repo into multiple
+   independent download dirs.
+
 - tag
     The git tag to retrieve. The default is "master"
 
@@ -151,6 +158,8 @@ class Git(FetchMethod):
 
         ud.nobranch = ud.parm.get("nobranch","0") == "1"
 
+        ud.dlname = ud.parm.get("dlname","")
+
         # usehead implies nobranch
         ud.usehead = ud.parm.get("usehead","0") == "1"
         if ud.usehead:
@@ -242,7 +251,11 @@ class Git(FetchMethod):
                     ud.unresolvedrev[name] = ud.revisions[name]
                 ud.revisions[name] = self.latest_revision(ud, d, name)
 
-        gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_'))
+        if ud.dlname:
+            gitdlname = ud.dlname.replace('/', '.').replace('*', '.').replace(' ','_')
+        else:
+            gitdlname = ud.path.replace('/', '.').replace('*', '.').replace(' ','_')
+        gitsrcname = '%s%s' % (ud.host.replace(':', '.'), gitdlname)
         if gitsrcname.startswith('.'):
             gitsrcname = gitsrcname[1:]
 
-- 
2.25.1

[PATCH 04/21] bitbake: fetch2/git: allow specifying repos as static/unchanging

[Paul Gortmaker]

If we know a repository content has what we need, we can flag it as
static and then optimize accordingly for the future, by not trying to do
any fetch operations against it.  In a way, this can be thought of as
the special case of GITFETCHREFS = None.

There are multiple advantages to having this.  We can skip trying any
updates and speed things up for a project we know is EOL.  We can avoid
doing "ls-remote" operations against servers, easing use for
mirror/no-network users.  We can open the door to accepting tags in
SRC_URI more often as absolute references, instead of the less humanly
readable SHA.  Archives that change upstream in incompatible ways, or
have simply vanished, can exist as a snapshot in time after initial
creation via mirror tarball or manually.

Finally, we use the flexibility of git-config to store that static
information in the cloned repo itself, so we can tell at any time,
independent of the SRC_URI, that the repo has static content.

This will be used for reference purposes, but might also be used in the
download dir itself outside of BB recipes -- for example a script could
easily spot any static content and hence prioritize it as a possible
mirroring candidate.

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 .../bitbake-user-manual-fetching.rst          |  5 ++++
 bitbake/lib/bb/fetch2/git.py                  | 27 +++++++++++++++++++
 2 files changed, 32 insertions(+)

diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
index 0f12eaa3a81f..94f4cf3363f6 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
@@ -426,6 +426,11 @@ This fetcher supports the following parameters:
 -  *"rev":* The revision to use for the checkout. The default is
    "master".
 
+-  *"static":* The repository is a "one and done" - clone/untar and the
+   content is static and unchanging.  This allows optimizations like
+   skipping fetch/pull and doing "ls-remote" on servers.  Typically used
+   for repositories that will act as a commit library for other repos.
+
 -  *"tag":* Specifies a tag to use for the checkout. To correctly
    resolve tags, BitBake must access the network. For that reason, tags
    are often not used. As far as Git is concerned, the "tag" parameter
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index de698a3d9f24..7cba990c3be7 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -54,6 +54,11 @@ Supported SRC_URI options are:
    referring to commit which is valid in tag instead of branch.
    The default is "0", set nobranch=1 if needed.
 
+- static
+   Repository is of fixed/static content with no active/new development.
+   Once cloned, nothing will change, so optimize accordingly.  The default
+   is "0", set static=1 when appropriate.
+
 - usehead
    For local git:// urls to use the current branch HEAD as the revision for use with
    AUTOREV. Implies nobranch.
@@ -158,6 +163,8 @@ class Git(FetchMethod):
 
         ud.nobranch = ud.parm.get("nobranch","0") == "1"
 
+        ud.static = ud.parm.get("static","0") == "1"
+
         ud.dlname = ud.parm.get("dlname","")
 
         # usehead implies nobranch
@@ -167,6 +174,10 @@ class Git(FetchMethod):
                  raise bb.fetch2.ParameterError("The usehead option is only for use with local ('protocol=file') git repositories", ud.url)
             ud.nobranch = 1
 
+        if ud.static:
+            if ud.rebaseable:
+                 raise bb.fetch2.ParameterError("The rebaseable option is incompatible with the static option", ud.url)
+
         # bareclone implies nocheckout
         ud.bareclone = ud.parm.get("bareclone","0") == "1"
         if ud.bareclone:
@@ -311,6 +322,8 @@ class Git(FetchMethod):
     def clonedir_need_update(self, ud, d):
         if not os.path.exists(ud.clonedir):
             return True
+        if ud.static:
+            return False
         if ud.shallow and ud.write_shallow_tarballs and self.clonedir_need_shallow_revs(ud, d):
             return True
         for name in ud.names:
@@ -341,6 +354,17 @@ class Git(FetchMethod):
             return False
         return True
 
+    def get_git_config(self, ud, d, repo, cfgvar):
+        try:
+            output = runfetchcmd("%s config --get %s" % (ud.basecmd, cfgvar), d, workdir=repo)
+        except (bb.fetch2.FetchError,ValueError):
+            return ""
+        return output.split()[0]
+
+    def repo_is_static(self, ud, d, repo):
+        static = self.get_git_config(ud, d, repo, "bitbake.static")
+        return (static == "true")
+
     def download(self, ud, d):
         """Fetch url"""
 
@@ -368,6 +392,9 @@ class Git(FetchMethod):
             progresshandler = GitProgressHandler(d)
             runfetchcmd(clone_cmd, d, log=progresshandler)
 
+            if ud.static:
+                runfetchcmd("%s config --bool --add bitbake.static 1" % ud.basecmd, d, workdir=ud.clonedir)
+
         # Update the checkout if needed
         if self.clonedir_need_update(ud, d):
             output = runfetchcmd("%s remote" % ud.basecmd, d, quiet=True, workdir=ud.clonedir)
-- 
2.25.1

[PATCH 05/21] bitbake: fetch2/git: ensure static repos have at least one refs/heads

[Paul Gortmaker]

When you clone with "--single-branch --branch foo" one of two things
will happen:

If foo is a branch upstream you will get a local branch foo and HEAD
will contain "ref: refs/heads/foo".  Output from "git branch" will be:
	* foo

If foo is a tag upstream, you won't get any local branch, and HEAD
will contain the SHA of the tag foo.  Output from "git branch" will be:
	* (no branch)

In the latter case, people tend to assume it is some kind of indication
progress was interrupted and an error happened.

While not strictly necessary, in static repos, we point master at the
tag that was fetched if no branches are present, to avoid confusion.

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 bitbake/lib/bb/fetch2/git.py | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index 7cba990c3be7..6cd5d09b2ded 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -417,6 +417,15 @@ class Git(FetchMethod):
                 if exc.errno != errno.ENOENT:
                     raise
 
+        # fixup "* (no branch)" from "--single-branch --branch <tag>"
+        if ud.static:
+            output = runfetchcmd("%s show-ref --heads | wc -l" % ud.basecmd, d, workdir=ud.clonedir)
+            headless = output.split()[0] == "0"
+            if headless:
+                runfetchcmd("%s update-ref refs/heads/master HEAD" % ud.basecmd, d, workdir=ud.clonedir)
+                with open(os.path.join(ud.clonedir, "HEAD"), "w") as f:
+                    f.write('ref: refs/heads/master\n')
+
         for name in ud.names:
             if not self._contains_ref(ud, d, name, ud.clonedir):
                 raise bb.fetch2.FetchError("Unable to find revision %s in branch %s even from upstream" % (ud.revisions[name], ud.branches[name]))
-- 
2.25.1

[PATCH 06/21] bitbake: fetch2/git: allow alt references within download dir

[Paul Gortmaker]

It is no secret that we've needed to formalize some kind of sensible
sharing between git repos with common ancestry.  The opportunity for
sharing the mainline kernel base between linux-yocto and linux-yocto-dev
is the main obvious candidate.

Many of us have long been doing local hacks to pre-populate the
downloads dir and avoid lengthy/large downloads, and some have even
been kind enough to try and get a variant of that upstream[1].

But the problem is that it isn't as simple as adding a new "--reference"
switch to the fetcher and then walking away.  That will result in
non-portable and incompatible use cases with existing work flows.   We
have to put some thought into how we see this fitting into the larger
framework of things and in what ways we want to support use of it.

For example, we know we want relative paths in anything in the download
dir, but "clone --reference" will put an absolute path in the alternates
file by default.  We also don't want the SRC_URI to be containing any
absolute paths either.  We also want (pre)mirror stuff to keep working.

So we set some basic ground rules.  Firstly we don't even expose the
obvious "--reference" argument that some of us are used to at all.  That
is because we are going to have two types of references - the common one
based on objets/info/alternates (this commit) and a pack based one (the
subsequent commit).

We also restrict any references to be peers of the other content in
downloads - no /home/fred/super-SDK/kernel-4.19-rt/ type stuff.  The
idea is that once you factor out the common building blocks of source,
then your kernel (staying with that example) becomes a very small repo.
For example - all of the linux-yocto BSPs (standard and -rt) can be held
in a repo that is only 20-30 megabytes - small enough to attach to email.

Clones via alternates have some limitations - for example there is an
arbitrary limit of six[2] levels of nesting, so a pack reference is
introduced to resolve this subsequently.  However alternaltes make sense
for shared non-static (i.e. WIP repos) endpoints, like stable and -rt.

[1] https://lists.openembedded.org/g/bitbake-devel/topic/77645766
[2] https://github.com/git/git/commit/c2f493a4  # if (depth > 5)

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 .../bitbake-user-manual-fetching.rst          |  6 +++++
 bitbake/lib/bb/fetch2/git.py                  | 24 +++++++++++++++++++
 2 files changed, 30 insertions(+)

diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
index 94f4cf3363f6..12a68accc2c5 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
@@ -447,6 +447,12 @@ This fetcher supports the following parameters:
    parameter implies no branch and only works when the transfer protocol
    is ``file://``.
 
+-  *"altref":* Enables cloned ``git://`` URLs to use the named peer from
+   the downloads dir as a reference through the standard git alternates
+   file for clone (see ``-reference``) and subseqent operations.  Recipe
+   is responsible (via fetch dependency) for ensuring the reference is
+   populated/cloned prior to being called on to act as a reference.
+
 Here are some example URLs: ::
 
    SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1"
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index 6cd5d09b2ded..e742010229c3 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -59,6 +59,12 @@ Supported SRC_URI options are:
    Once cloned, nothing will change, so optimize accordingly.  The default
    is "0", set static=1 when appropriate.
 
+- altref
+   Repository is to use named peer from downloads dir as a reference through
+   the standard git objects/info/alternates file for clone and subseqent
+   operations.  Recipe is responsible (via fetch dependency) for ensuring the
+   reference is populated/cloned prior to being called on to act as a reference.
+
 - usehead
    For local git:// urls to use the current branch HEAD as the revision for use with
    AUTOREV. Implies nobranch.
@@ -165,6 +171,8 @@ class Git(FetchMethod):
 
         ud.static = ud.parm.get("static","0") == "1"
 
+        ud.altref = ud.parm.get("altref","")
+
         ud.dlname = ud.parm.get("dlname","")
 
         # usehead implies nobranch
@@ -380,9 +388,20 @@ class Git(FetchMethod):
 
         repourl = self._get_repo_url(ud)
 
+        refname = ud.altref
+        if refname:
+            alts = os.path.join(ud.clonedir, 'objects', 'info', 'alternates')
+            gitdir = d.getVar("GITDIR") or (d.getVar("DL_DIR") + "/git2")
+            refdir = os.path.join(gitdir, refname)
+            if not os.path.exists(refdir):
+                raise bb.fetch2.FetchError("Unable to find reference %s in %s" % (refname, gitdir))
+
         # If the repo still doesn't exist, fallback to cloning it
         if not os.path.exists(ud.clonedir):
             extcloneargs = d.getVar('GITCLONEARGS_' + ud.names[0]) or d.getVar('GITCLONEARGS') or "--bare --mirror"
+            if refname:
+                extcloneargs += " --reference %s" % refdir
+
             # We do this since git will use a "-l" option automatically for local urls where possible
             if repourl.startswith("file://"):
                 repourl = repourl[7:]
@@ -392,6 +411,11 @@ class Git(FetchMethod):
             progresshandler = GitProgressHandler(d)
             runfetchcmd(clone_cmd, d, log=progresshandler)
 
+            # Fix up reference to be relative for tarball/mirror portability
+            if ud.altref:
+                with open(alts, "w") as f:
+                    f.write("../../%s/objects\n" % ud.altref)
+
             if ud.static:
                 runfetchcmd("%s config --bool --add bitbake.static 1" % ud.basecmd, d, workdir=ud.clonedir)
 
-- 
2.25.1

[PATCH 07/21] bitbake: fetch2/git: append new altref line if/when SRC_URI changed value

[Paul Gortmaker]

A new repository that is cloned with "--reference" in turn creates an
objects/info/alternates file in the new repo containing the path back to
the reference repo objects.  This is normally a "one-and-done" type
operation and the reference never changes.

However we can envision where a package leaves behind a dead end
"stable" fork and re-aligns itself with the main branch, or a newer
stable fork.  In the case of kernel, a real world example would be
leaving behind v5.4.x in favour of v5.10.x for a baseline.

This commit is a consideration for those users - with an existing copy
in their downloads dir that is using a reference that has since been
updated in the SRC_URI of the package recipe.

If we removed the old reference and replaced it, we would break any old
build directories that had branches based on v5.4.x specific commits, so
that is clearly not an option.

If we simply did nothing, things would still work just fine for them;
any v5.10.x objects not found in the v5.4.x reference would be simply
obtained automatically from the server during a fetch update, and placed
in the endpoint repo that is using the older v5.4.x as a reference.

However, since alternates supports a multi-line format, we can simply
append the new reference.  This leaves any v5.4.x build "consumers"
functioning as-is, and also gives a path back to v5.10.x objects in the
new reference, so they need not be duplicated over into the endpoint repo.

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 bitbake/lib/bb/fetch2/git.py | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index e742010229c3..3b0c068bc020 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -421,6 +421,15 @@ class Git(FetchMethod):
 
         # Update the checkout if needed
         if self.clonedir_need_update(ud, d):
+            if ud.altref:
+                # if altref changed in SRC_URI then append new value.
+                alts = os.path.join(ud.clonedir, 'objects', 'info', 'alternates')
+                try:
+                    runfetchcmd("grep -q %s %s" % (ud.altref, alts), d, quiet=True, workdir=ud.clonedir)
+                except (bb.fetch2.FetchError,ValueError):
+                    with open(alts, "a") as f:
+                        f.write("../../%s/objects\n" % ud.altref)
+
             output = runfetchcmd("%s remote" % ud.basecmd, d, quiet=True, workdir=ud.clonedir)
             if "origin" in output:
               runfetchcmd("%s remote rm origin" % ud.basecmd, d, workdir=ud.clonedir)
-- 
2.25.1

[PATCH 08/21] bitbake: fetch2/git: allow pack references within download dir

[Paul Gortmaker]

Borrowing from "git pack-objects" manpage[1] we have:

   A packed archive is an efficient way to transfer a set of objects
   between two repositories as well as an access efficient archival
   format.  In a packed archive, an object is either stored as a
   compressed whole or as a difference from some other object. The
   latter is often called a delta.

   The packed archive format (.pack) is designed to be self-contained
   so that it can be unpacked without any further information. Therefore,
   each object that a delta depends upon must be present within the pack.

   A pack index file (.idx) is generated for fast, random access to
   the objects in the pack. Placing both the index file (.idx) and the
   packed archive (.pack) in the pack/ subdirectory [...] enables Git
   to read from the pack archive.

As stated, they are self-contained and can encapsulate an entire chunk
of development history starting at point A and ending at point B, which
makes them ideal not just for transferring, but also for sharing that
chunk of history between repositories with common ancestry.

When you ask a server for something, it sends a single pack, because
that is most efficient.  So, using the kernel as an example, lets say we
decide we want to use the 2005 git "epoch" as "A" and the v5.10 tag of a
recent release as "B" to delineate an initial block of history.  We can
grab that chunk of history with a simple:

git clone --bare --single-branch --branch v5.10 git://someserver/linux my-510-lib

and we now have all the history up to and including v5.10 tag in a repo
that we can use with further "clone --reference my-510-lib newstuff" as
a static unchanging library of all those objects for "newstuff" to use.

In making such a clone, Git stores the my-510-lib dir in the
objects/info/alternates of "newstuff" and all the refs and tags carry
forard into [packed]refs of "newstuff".  Subsequent uses of the
"newstuff" repo are only using the objects from "my-510-lib" and they
are conveniently all stored in one pack, as there are no loose objects.
In fact it is a waste of time even having git operations look for them.

This means we can simply remove the alternates file and link the pack
(and accompanying index file) into the pack dir of "newstuff".  And by
using symlinks with relative paths, we get a mirror friendly reference
with no data download duplication.  The "clone --reference" is simply
used as a convenience tool to do 99% of the setup work.  The linked in
pack is "just a normal pack" to the consumer of the commit library.

As such, we introduce a "packref" as a useful parallel addition to the
more well known "altref".  It is ideally suited to reference from static
repositories containing a known block of commit history.  Note that we
don't limit ourselves to a single pack, so that a packref can itself be
using another repo as its own packref and so on.  Hence there is no
arbitrary limit of a depth of 5 as there is with a reference using the
more common alternates (i.e. "altref") method.

This opens the door to insert an arbitrary number of split points between
A and B to fragment large repos, if desired.  An example that shows how
the bulk of kernel history can be split in two, three or four chunks
will be provided for recipes-kernel/linux subsequently.

[1] https://git-scm.com/docs/git-pack-objects

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 .../bitbake-user-manual-fetching.rst          |  7 +++++
 bitbake/lib/bb/fetch2/git.py                  | 31 ++++++++++++++++++-
 2 files changed, 37 insertions(+), 1 deletion(-)

diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
index 12a68accc2c5..db1d108f4736 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
@@ -453,6 +453,13 @@ This fetcher supports the following parameters:
    is responsible (via fetch dependency) for ensuring the reference is
    populated/cloned prior to being called on to act as a reference.
 
+-  *"packref":* Enables cloned ``git://`` URLs to use packs from the named
+   peer from the downloads dir as if they were their own packs of commit
+   history.  Similar to ``altref`` above, but suited for bare static repos
+   for which no loose objects outside of packs will ever be present.
+   Recipe is responsible (via fetch dependency) for ensuring the reference
+   is populated/cloned prior to being called on to act as a reference.
+
 Here are some example URLs: ::
 
    SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1"
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index 3b0c068bc020..8aec12df2bf8 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -65,6 +65,14 @@ Supported SRC_URI options are:
    operations.  Recipe is responsible (via fetch dependency) for ensuring the
    reference is populated/cloned prior to being called on to act as a reference.
 
+- packref
+   Repository is to use packs from named peer from downloads dir by linking
+   them into objects/pack/ of the new repository as if they were their own
+   packs of commit history.  Similar to altref above, but suited for bare
+   static repos for which no loose objects outside of packs will be present.
+   Recipe is responsible (via fetch dependency) for ensuring the reference
+   is populated/cloned prior to being called on to act as a reference.
+
 - usehead
    For local git:// urls to use the current branch HEAD as the revision for use with
    AUTOREV. Implies nobranch.
@@ -171,10 +179,15 @@ class Git(FetchMethod):
 
         ud.static = ud.parm.get("static","0") == "1"
 
+        ud.packref = ud.parm.get("packref","")
+
         ud.altref = ud.parm.get("altref","")
 
         ud.dlname = ud.parm.get("dlname","")
 
+        if ud.packref and ud.altref:
+            raise bb.fetch2.ParameterError("Please pick only ONE reference style: alternates OR packs", ud.url)
+
         # usehead implies nobranch
         ud.usehead = ud.parm.get("usehead","0") == "1"
         if ud.usehead:
@@ -373,6 +386,14 @@ class Git(FetchMethod):
         static = self.get_git_config(ud, d, repo, "bitbake.static")
         return (static == "true")
 
+    def create_pack_links(self, refname, refdir, dstdir):
+        dstpkdir = os.path.join(dstdir, 'objects', 'pack')
+        refpkdir = os.path.join(refdir, 'objects', 'pack')
+        for item in os.listdir(refpkdir):
+            dst = os.path.join(dstpkdir, item)
+            src = os.path.join('..', '..', '..', refname, 'objects', 'pack', item)
+            os.symlink(src, dst)
+
     def download(self, ud, d):
         """Fetch url"""
 
@@ -388,7 +409,9 @@ class Git(FetchMethod):
 
         repourl = self._get_repo_url(ud)
 
-        refname = ud.altref
+        # Both style of reference are the same for initial clone operation.
+        # It is the post-clone absolute to relative path fixup that differs.
+        refname = ud.altref or ud.packref
         if refname:
             alts = os.path.join(ud.clonedir, 'objects', 'info', 'alternates')
             gitdir = d.getVar("GITDIR") or (d.getVar("DL_DIR") + "/git2")
@@ -419,6 +442,12 @@ class Git(FetchMethod):
             if ud.static:
                 runfetchcmd("%s config --bool --add bitbake.static 1" % ud.basecmd, d, workdir=ud.clonedir)
 
+            if ud.packref:
+                if not self.repo_is_static(ud, d, refdir):
+                    raise bb.fetch2.FetchError("Pack reference '%s' is not marked as static content." % refname)
+                self.create_pack_links(refname, refdir, ud.clonedir)
+                os.remove(alts)
+
         # Update the checkout if needed
         if self.clonedir_need_update(ud, d):
             if ud.altref:
-- 
2.25.1

[PATCH 09/21] bitbake: fetch2/git: use constant names for packs in static repos

[Paul Gortmaker]

The "packref" support uses symlinks to "piggy-back" use of packs into
the pack namespace of the consumer from the originator (reference repo).

We use symlinks, so that mirror tarballs won't duplicate the referenced
data, as hardlinks would do.  However, that means we also assume that
the pack namespace of the originator remains constant.

What if one uses a reference from a mirror tarball, then disables
mirrors, does a "cleanall" on the reference for some obscure reason, and
then re-clones it from a server directly.  Will the pack links from the
consumer into the reference still point to a pack or be left dangling?

For simplicity, we temporarily ignored this assumption so that we could
deal with it (and this corner case related to it) separately - which we
do now.  To proceed, we first need to know a bit about pack names.

Most importantly, as you can see, pack names don't really matter at all:

   paul@hackbox:~$ git clone --bare git://git.yoctoproject.org/poky
   Cloning into bare repository 'poky.git'...
     [...]
   Resolving deltas: 100% (381097/381097), done.
   paul@hackbox:~$ cd poky.git/objects/pack/
   paul@hackbox:~/poky.git/objects/pack$ ls -l
   total 191100
   -r--r--r-- 1 paul paul  14250608 Mar 28 12:09 pack-db459c1574c5410a1002d8098745daee3a59599f.idx
   -r--r--r-- 1 paul paul 181429684 Mar 28 12:09 pack-db459c1574c5410a1002d8098745daee3a59599f.pack
   paul@hackbox:~/poky.git/objects/pack$ mv pack-db459c1574c5410a1002d8098745daee3a59599f.idx woot-fred-was-here-31337.idx
   paul@hackbox:~/poky.git/objects/pack$ mv pack-db459c1574c5410a1002d8098745daee3a59599f.pack woot-fred-was-here-31337.pack
   paul@hackbox:~/poky.git/objects/pack$ cd ../../
   paul@hackbox:~/poky.git$ git fsck
   Checking object directories: 100% (256/256), done.
   Checking objects: 100% (508912/508912), done.
   Checking connectivity: 508912, done.
   paul@hackbox:~/poky.git$ git show HEAD | head -n3
   commit a7e1bbaf6d7c5d1cf44069419860dabd78c02eec
   Author: Khem Raj <raj.khem@gmail.com>
   Date:   Thu Mar 25 18:11:07 2021 -0700
   paul@hackbox:~/poky.git$ echo $?
   0

This is because normal flow will have a clone/fetch feed stdin of
index-pack, and from the "--stdin" part of its manpage, we have:

   If <pack-file> is not specified, the pack is written to objects/pack/
   directory of the current Git repository with a default name determined
   from the pack content.

Two things to note: 1) we have always been free to choose the pack name
even if the average user probably always takes the default, and 2) the
man page is intentionally kind of vague on how that default is created.

We need to look at #2 in detail, in order to answer our question about
whether the pack names remain constant for the same object content.

The executive summary is "yes" for git v1.7 and older and "no" for git
v1.8+   So we need to look at both in order to decide what to do.

For the "old" git, the default pack name was simply "pack-" + the SHA1
of the sorted SHA1 object names themselves.  If you have an old pack
around you can confirm as I have done below:

   $ ls -l
   total 792
   -r--r--r-- 1 paul paul   1632 Feb 13  2013 pack-a9f793e9f46c138a27abb326ac68cb6e6397d0f0.idx
   -r--r--r-- 1 paul paul 803171 Feb 13  2013 pack-a9f793e9f46c138a27abb326ac68cb6e6397d0f0.pack
   $ git show-index < pack-a9f793e9f46c138a27abb326ac68cb6e6397d0f0.idx | cut -d " " -f 2 | xxd -r -p | sha1sum
   a9f793e9f46c138a27abb326ac68cb6e6397d0f0  -
   $ git verify-pack -v pack-a9f793e9f46c138a27abb326ac68cb6e6397d0f0.pack|grep '^[0-9a-f]\{40\}' | cut -d " " -f1  | sort | xxd -r -p | sha1sum
   a9f793e9f46c138a27abb326ac68cb6e6397d0f0  -

In both cases, we recover the pack name directly from the objects. There
is no extraneous information, like date, servername, etc. used in the
pack name.  And so it is invariant upon reclone/repack, even if the
compression level was changed - since the final result from "git
unpack-objects" is the same.

For newer git, which contain 1190a1ac[1] the choice was made to use the
SHA1 of the data as stored in the pack itself - the "trailer" SHA1.
If we look at our since-renamed pack friend Fred above:

   paul@hackbox:~/poky.git/objects/pack$ hexdump -C woot-fred-was-here-31337.pack  | tail -n3
   0ad065a0  db 45 9c 15 74 c5 41 0a  10 02 d8 09 87 45 da ee
   0ad065b0  3a 59 59 9f

...we can see we really didn't "lose" his original pack name, as the
trailer SHA1 db459c1574c5410a1002d8098745daee3a59599f is right there.

But this means that the pack name is based on transient data like
compression artifacts, and that no two clones will ever likely generate
the same pack name.  If this was a letter in an envelope, the SHA1 went
from being the contents of the letter, to also include how many times
the paper was folded, and the size and colour of the envelope.

This change was made to remove any possible confusion that pack name
equivalence meant binary byte equivalence across two packs, by virtually
ensuring we never got the same pack name twice.

However, in our case, a pack name that is essentially a random number is
unhelpful, and the namespace collision concerns of commit 1190a1ac are
even less of a concern for static content repositories.

So our static repos use pre v1.8 pack names, which reflect functional
equivalence between packs from clone to clone to mirror tarball snapshot
of a clone, because as demonstrated earlier, we are free to choose the
pack name(s) completely at will, so long as we keep it paired with its
index file.

This turns out to be much more robust than trying to preserve the
now random "new" pack names through self-updating symlinks based on
dependencies or anon python or anything similarly complex.

As a final optimization/convenience, we can simply drop the "xxd" from
the pipeline - and avoid introducing new native dependencies, or adding
it to the ASSUME_PROVIDED command list and hoping it is everywhere.

We can do this because the ASCII representation of the .idx hex values
contains the exact same pack object content information as the xxd
binary-converted version, but just in a less dense format.  So the SHA1
of that less packed info stream still gives us a unique signature of the
objects contained in the pack - just not the exact one the older git did.

[1] https://github.com/git/git/commit/1190a1ac

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 bitbake/lib/bb/fetch2/git.py | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index 8aec12df2bf8..2d693083cac4 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -386,6 +386,21 @@ class Git(FetchMethod):
         static = self.get_git_config(ud, d, repo, "bitbake.static")
         return (static == "true")
 
+    def rename_packs(self, ud, d, repo):
+        # Use git pack naming similar to pre v1.8 - where the name only
+        # depends on the objects within, not objs + compression artifacts.
+        pkdir = os.path.join(repo, 'objects', 'pack')
+
+        for idx in fnmatch.filter(os.listdir(pkdir), "*.idx"):
+            pk = os.path.join(pkdir, idx[:-3] + "pack")
+            idx = os.path.join(pkdir, idx)
+            # insert "xxd -p -r" before the sha1sum to get exactly v1.7 names.
+            cmd = "%s show-index < %s | cut -d' ' -f2 | sort | sha1sum" % (ud.basecmd, idx)
+            output = runfetchcmd(cmd, d, workdir=repo)
+            newname = os.path.join(pkdir, "pack-" + output[:40])
+            os.rename(pk, newname + ".pack")
+            os.rename(idx, newname + ".idx")
+
     def create_pack_links(self, refname, refdir, dstdir):
         dstpkdir = os.path.join(dstdir, 'objects', 'pack')
         refpkdir = os.path.join(refdir, 'objects', 'pack')
@@ -441,6 +456,7 @@ class Git(FetchMethod):
 
             if ud.static:
                 runfetchcmd("%s config --bool --add bitbake.static 1" % ud.basecmd, d, workdir=ud.clonedir)
+                self.rename_packs(ud, d, ud.clonedir)
 
             if ud.packref:
                 if not self.repo_is_static(ud, d, refdir):
-- 
2.25.1

[PATCH 10/21] kernel: add basic boilerplate for fetch-only recipes

[Paul Gortmaker]

With changes to the fetcher, we now have what we need to download
kernel sources in a way that is forward thinking for sharing and
growth and EOL of components.  This will be achieved by "fetch-only"
recipes that only exist to populate chunks of this "library" of source.

We may wish to reconsider locating 99% of this elsewhere, if use cases
outside the kernel eventually arise.

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 meta/recipes-kernel/linux/fetch-only.inc | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
 create mode 100644 meta/recipes-kernel/linux/fetch-only.inc

diff --git a/meta/recipes-kernel/linux/fetch-only.inc b/meta/recipes-kernel/linux/fetch-only.inc
new file mode 100644
index 000000000000..3333eef50713
--- /dev/null
+++ b/meta/recipes-kernel/linux/fetch-only.inc
@@ -0,0 +1,20 @@
+# There's nothing to do here, except fetch
+
+do_unpack[noexec] = "1"
+do_patch[noexec] = "1"
+do_configure[noexec] = "1"
+do_compile[noexec] = "1"
+do_install[noexec] = "1"
+
+inherit nopackages allarch meta
+
+deltask do_deploy_source_date_epoch
+deltask do_prepare_recipe_sysroot
+deltask do_populate_sysroot
+deltask do_populate_lic
+deltask do_rm_work
+
+EXCLUDE_FROM_WORLD = "1"
+KORG_SERVER ?= "git.kernel.org"
+KORG_PATH ?= "/pub/scm/linux/kernel/git"
+KORG_URLBASE ?= "git://${KORG_SERVER}${KORG_PATH}"
-- 
2.25.1

[PATCH 11/21] kernel: add a fetch-only recipe for mainline v5.10 source

[Paul Gortmaker]

With v5.10 being the newest baseline currently in use by linux-yocto and
with the download size being 1/2 the size of current linux-yocto itself,
the v5.10 makes a good initial line in the sand for blocking out the
source in a way to optimize sharing.

With this commit present, one can test via "bitbake -c fetch linux-5.10".

Pending recipes who will want this source present will add:
	do_fetch[depends] += "linux-5.10:do_fetch"

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 meta/recipes-kernel/linux/fetch-linux.inc | 7 +++++++
 meta/recipes-kernel/linux/linux-5.10.bb   | 9 +++++++++
 2 files changed, 16 insertions(+)
 create mode 100644 meta/recipes-kernel/linux/fetch-linux.inc
 create mode 100644 meta/recipes-kernel/linux/linux-5.10.bb

diff --git a/meta/recipes-kernel/linux/fetch-linux.inc b/meta/recipes-kernel/linux/fetch-linux.inc
new file mode 100644
index 000000000000..e3fdd0bb36c2
--- /dev/null
+++ b/meta/recipes-kernel/linux/fetch-linux.inc
@@ -0,0 +1,7 @@
+# Boilerplate to fetch up to a single tag of mainline
+
+require recipes-kernel/linux/fetch-only.inc
+
+GITCLONEARGS = "--bare --single-branch --branch v${KTAG}"
+DL_NAME = ".torvalds.linux-${KTAG}"
+SRC_URI = "${KORG_URLBASE}/torvalds/linux;tag=v${KTAG};nobranch=1;static=1;dlname=${DL_NAME}"
diff --git a/meta/recipes-kernel/linux/linux-5.10.bb b/meta/recipes-kernel/linux/linux-5.10.bb
new file mode 100644
index 000000000000..7f8a27bcb1f0
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-5.10.bb
@@ -0,0 +1,9 @@
+SUMMARY = "Linux kernel v5.10 source"
+DESCRIPTION = "Fetch source of the linux kernel up to the v5.10 tag"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
+
+KTAG = "5.10"
+
+require recipes-kernel/linux/fetch-linux.inc
-- 
2.25.1

[PATCH 12/21] kernel: allow splitting mainline v5.10 source download in two

[Paul Gortmaker]

With support for sharing packs, we can now optionally split the larger
mainline kernel content up to v5.10 (approx 1.5G) into two.  This may be
advantageous to people with release media constraints or unreliable
connections.

By setting "INITIAL_KERNEL_SPLIT = 1" in your configuration, you will
get a linux-4.0 clone and then the linux-5.10 clone in your downloads
directory.  From a size perspective, v4.0 is roughly the half way point
between when git use started in 2005, and v5.10 from 2020.

This split is achieved by manually selecting the midpoint tag and cloning
that portion 1st, and then cloning the 2nd 1/2, while using the packs in
the 1st 1/2 as a reference.

Packs deploy both zlib and delta compression - and as with all
compression technologies, splitting the input comes at a cost in terms
of reducing the compression.  So two packs will be bigger than one,
three packs bigger than two, and so on.  Representative numbers are
listed alongside the implementation in the change itself.

The 2nd 1/2 includes a fetch dependency on the 1st 1/2, so that things
will be serialiazed properly.  The dependency also links it into the
chain of events, so no other mentions of v4.0 are required.

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 meta/recipes-kernel/linux/fetch-linux.inc | 15 ++++++++++++++-
 meta/recipes-kernel/linux/linux-4.0.bb    |  9 +++++++++
 meta/recipes-kernel/linux/linux-5.10.bb   | 20 ++++++++++++++++++++
 3 files changed, 43 insertions(+), 1 deletion(-)
 create mode 100644 meta/recipes-kernel/linux/linux-4.0.bb

diff --git a/meta/recipes-kernel/linux/fetch-linux.inc b/meta/recipes-kernel/linux/fetch-linux.inc
index e3fdd0bb36c2..96653a2bea06 100644
--- a/meta/recipes-kernel/linux/fetch-linux.inc
+++ b/meta/recipes-kernel/linux/fetch-linux.inc
@@ -2,6 +2,19 @@
 
 require recipes-kernel/linux/fetch-only.inc
 
+python __anonymous () {
+    kref = d.getVar("KREF")
+    if not kref:
+        d.setVar("KORG_REF", "")
+        d.setVar("KORG_REFDEP", "")
+        return;
+    clonehost = d.getVar("KORG_SERVER")
+    d.setVar("KORG_REF", ";packref=%s.torvalds.linux-%s" % (clonehost, kref))
+    d.setVar("KORG_REFDEP", "linux-%s:do_fetch"% kref)
+}
+
+do_fetch[depends] += "${KORG_REFDEP}"
+
 GITCLONEARGS = "--bare --single-branch --branch v${KTAG}"
 DL_NAME = ".torvalds.linux-${KTAG}"
-SRC_URI = "${KORG_URLBASE}/torvalds/linux;tag=v${KTAG};nobranch=1;static=1;dlname=${DL_NAME}"
+SRC_URI = "${KORG_URLBASE}/torvalds/linux;tag=v${KTAG};nobranch=1;static=1;dlname=${DL_NAME}${KORG_REF}"
diff --git a/meta/recipes-kernel/linux/linux-4.0.bb b/meta/recipes-kernel/linux/linux-4.0.bb
new file mode 100644
index 000000000000..68dd054970ca
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-4.0.bb
@@ -0,0 +1,9 @@
+SUMMARY = "Linux kernel v4.0 source"
+DESCRIPTION = "Fetch source of the linux kernel up to the v4.0 tag"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=d7810fab7487fb0aad327b76f1be7cd7"
+
+KTAG = "4.0"
+
+require recipes-kernel/linux/fetch-linux.inc
diff --git a/meta/recipes-kernel/linux/linux-5.10.bb b/meta/recipes-kernel/linux/linux-5.10.bb
index 7f8a27bcb1f0..73e33d405376 100644
--- a/meta/recipes-kernel/linux/linux-5.10.bb
+++ b/meta/recipes-kernel/linux/linux-5.10.bb
@@ -6,4 +6,24 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
 
 KTAG = "5.10"
 
+# We optionally split the v5.10 download via adding refs and depends.
+# Fragmented downloads can be less apt to timeout, but delta compression
+# will lead to object duplication for > 1 objects/pack/*pack file and
+# an increase in overall on-disk totals.  The du -hsm for bare repos:
+#
+# No split:	1521	epoch->5.10
+# One split:	1730	epoch->4.0, 4.0->5.10 (854+877)
+#
+# Download sizes will vary depending on server compression settings.
+
+python __anonymous () {
+    splits = int(d.getVar("INITIAL_KERNEL_SPLIT") or 0)
+    if not splits:
+        return
+    if splits == 1:
+        d.setVar("KREF", "4.0")
+        return
+    bb.fatal("Unsupported number of v5.10 kernel content splits (%s)." % splits)
+}
+
 require recipes-kernel/linux/fetch-linux.inc
-- 
2.25.1

[PATCH 13/21] kernel: allow splitting mainline v5.10 source download in three

[Paul Gortmaker]

With a split in two done, extending to three is fairly trivial.  The
only difference is that the introduced "touch down" point of 4.12 has a
fetch depend/ref itself to the other touch down point of 3.8.  The parent
repo (v5.10) has no need of knowledge of what happens prior to 4.12.

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 meta/recipes-kernel/linux/linux-3.8.bb  |  9 +++++++++
 meta/recipes-kernel/linux/linux-4.12.bb | 10 ++++++++++
 meta/recipes-kernel/linux/linux-5.10.bb |  5 +++++
 3 files changed, 24 insertions(+)
 create mode 100644 meta/recipes-kernel/linux/linux-3.8.bb
 create mode 100644 meta/recipes-kernel/linux/linux-4.12.bb

diff --git a/meta/recipes-kernel/linux/linux-3.8.bb b/meta/recipes-kernel/linux/linux-3.8.bb
new file mode 100644
index 000000000000..58063ea51691
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-3.8.bb
@@ -0,0 +1,9 @@
+SUMMARY = "Linux kernel v3.8 source"
+DESCRIPTION = "Fetch source of the linux kernel up to the v3.8 tag"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=d7810fab7487fb0aad327b76f1be7cd7"
+
+KTAG = "3.8"
+
+require recipes-kernel/linux/fetch-linux.inc
diff --git a/meta/recipes-kernel/linux/linux-4.12.bb b/meta/recipes-kernel/linux/linux-4.12.bb
new file mode 100644
index 000000000000..6e8034a77f4a
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-4.12.bb
@@ -0,0 +1,10 @@
+SUMMARY = "Linux kernel v4.12 source"
+DESCRIPTION = "Fetch source of the linux kernel up to the v4.12 tag"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=d7810fab7487fb0aad327b76f1be7cd7"
+
+KTAG = "4.12"
+KREF = "3.8"
+
+require recipes-kernel/linux/fetch-linux.inc
diff --git a/meta/recipes-kernel/linux/linux-5.10.bb b/meta/recipes-kernel/linux/linux-5.10.bb
index 73e33d405376..373c6b8def2e 100644
--- a/meta/recipes-kernel/linux/linux-5.10.bb
+++ b/meta/recipes-kernel/linux/linux-5.10.bb
@@ -7,12 +7,14 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
 KTAG = "5.10"
 
 # We optionally split the v5.10 download via adding refs and depends.
+# Multiple splits are achieved by adding more chained dependencies.
 # Fragmented downloads can be less apt to timeout, but delta compression
 # will lead to object duplication for > 1 objects/pack/*pack file and
 # an increase in overall on-disk totals.  The du -hsm for bare repos:
 #
 # No split:	1521	epoch->5.10
 # One split:	1730	epoch->4.0, 4.0->5.10 (854+877)
+# Two splits:	1928	epoch->3.8, 3.8->4.12, 4.12->5.10 (628+647+653)
 #
 # Download sizes will vary depending on server compression settings.
 
@@ -23,6 +25,9 @@ python __anonymous () {
     if splits == 1:
         d.setVar("KREF", "4.0")
         return
+    if splits == 2:
+        d.setVar("KREF", "4.12")
+        return
     bb.fatal("Unsupported number of v5.10 kernel content splits (%s)." % splits)
 }
 
-- 
2.25.1

[PATCH 14/21] kernel: allow splitting mainline v5.10 source download in four

[Paul Gortmaker]

Extending from three to four roughly equal chunks introduces three new
split boundaries - 3.3 4.3 and 4.18, but is trivial to hook in.

This gets chunks down to roughly 1/2G and with the resulting loss of
compression efficiency, it doesn't seem to make sense to support
splitting things any further.  See the chunk sizes in the commit.

Document the new variable and its allowed value range since we don't
see adding any further split options.

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 documentation/ref-manual/variables.rst  |  6 ++++++
 meta/recipes-kernel/linux/linux-3.3.bb  |  9 +++++++++
 meta/recipes-kernel/linux/linux-4.18.bb | 10 ++++++++++
 meta/recipes-kernel/linux/linux-4.3.bb  | 10 ++++++++++
 meta/recipes-kernel/linux/linux-5.10.bb |  4 ++++
 5 files changed, 39 insertions(+)
 create mode 100644 meta/recipes-kernel/linux/linux-3.3.bb
 create mode 100644 meta/recipes-kernel/linux/linux-4.18.bb
 create mode 100644 meta/recipes-kernel/linux/linux-4.3.bb

diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst
index 55028987c08a..1226bbf77193 100644
--- a/documentation/ref-manual/variables.rst
+++ b/documentation/ref-manual/variables.rst
@@ -3441,6 +3441,12 @@ system and gives an overview of their function and contents.
       lines in the example are from individual recipes and show how the
       ``PR`` value is set.
 
+   :term:`INITIAL_KERNEL_SPLIT`
+      Specifies the number of times the initial kernel Git clone should be
+      split in order to avoid one single gigabyte+ download. Supported
+      values are 0 through 3 inclusive, resulting in one through four
+      separate chunks (repositories) in the download dir.
+
    :term:`INCOMPATIBLE_LICENSE`
       Specifies a space-separated list of license names (as they would
       appear in :term:`LICENSE`) that should be excluded
diff --git a/meta/recipes-kernel/linux/linux-3.3.bb b/meta/recipes-kernel/linux/linux-3.3.bb
new file mode 100644
index 000000000000..da355f9cd32c
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-3.3.bb
@@ -0,0 +1,9 @@
+SUMMARY = "Linux kernel v3.3 source"
+DESCRIPTION = "Fetch source of the linux kernel up to the v3.3 tag"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=d7810fab7487fb0aad327b76f1be7cd7"
+
+KTAG = "3.3"
+
+require recipes-kernel/linux/fetch-linux.inc
diff --git a/meta/recipes-kernel/linux/linux-4.18.bb b/meta/recipes-kernel/linux/linux-4.18.bb
new file mode 100644
index 000000000000..ea87f5e92b52
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-4.18.bb
@@ -0,0 +1,10 @@
+SUMMARY = "Linux kernel v4.18 source"
+DESCRIPTION = "Fetch source of the linux kernel up to the v4.18 tag"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=d7810fab7487fb0aad327b76f1be7cd7"
+
+KTAG = "4.18"
+KREF = "4.3"
+
+require recipes-kernel/linux/fetch-linux.inc
diff --git a/meta/recipes-kernel/linux/linux-4.3.bb b/meta/recipes-kernel/linux/linux-4.3.bb
new file mode 100644
index 000000000000..35e5fd48d8ba
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-4.3.bb
@@ -0,0 +1,10 @@
+SUMMARY = "Linux kernel v4.3 source"
+DESCRIPTION = "Fetch source of the linux kernel up to the v4.3 tag"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=d7810fab7487fb0aad327b76f1be7cd7"
+
+KTAG = "4.3"
+KREF = "3.3"
+
+require recipes-kernel/linux/fetch-linux.inc
diff --git a/meta/recipes-kernel/linux/linux-5.10.bb b/meta/recipes-kernel/linux/linux-5.10.bb
index 373c6b8def2e..e147c854477d 100644
--- a/meta/recipes-kernel/linux/linux-5.10.bb
+++ b/meta/recipes-kernel/linux/linux-5.10.bb
@@ -15,6 +15,7 @@ KTAG = "5.10"
 # No split:	1521	epoch->5.10
 # One split:	1730	epoch->4.0, 4.0->5.10 (854+877)
 # Two splits:	1928	epoch->3.8, 3.8->4.12, 4.12->5.10 (628+647+653)
+# Three splits:	2106	epoch->3.3, 3.3->4.3, 4.3->4.18, 4.18->5.10 (536+532+519+520)
 #
 # Download sizes will vary depending on server compression settings.
 
@@ -28,6 +29,9 @@ python __anonymous () {
     if splits == 2:
         d.setVar("KREF", "4.12")
         return
+    if splits == 3:
+        d.setVar("KREF", "4.18")
+        return
     bb.fatal("Unsupported number of v5.10 kernel content splits (%s)." % splits)
 }
 
-- 
2.25.1

[PATCH 15/21] kernel: add recipe for linux-master (mainline latest)

[Paul Gortmaker]

The idea for this recipe is to have a name "linux-master" that people
can depend on as a reference for mainline; today and ideally for as long
as we are maintaining it.  While the recipe is there for everybody,
the obvious user we are creating this for is the linux-yocto-dev repo.

At some point in the future, we'll carve off another chunk of post-v5.10
work, mark it static and put it alongside of v5.10 and let this mainline
recipe reference that.  But use of this "linux-master" will be 100%
transparent to the underlying backing store changes.

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 meta/recipes-kernel/linux/linux-master.bb | 13 +++++++++++++
 1 file changed, 13 insertions(+)
 create mode 100644 meta/recipes-kernel/linux/linux-master.bb

diff --git a/meta/recipes-kernel/linux/linux-master.bb b/meta/recipes-kernel/linux/linux-master.bb
new file mode 100644
index 000000000000..f674c2562e81
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-master.bb
@@ -0,0 +1,13 @@
+SUMMARY = "Linux kernel mainline source"
+DESCRIPTION = "Fetch source of the latest linux mainline kernel"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
+
+DL_NAME = ".torvalds.linux-master"
+KORG_REF = "git.kernel.org.torvalds.linux-5.10"
+SRC_URI = "${KORG_URLBASE}/torvalds/linux;rev=master;nobranch=1;dlname=${DL_NAME};packref=${KORG_REF}"
+
+do_fetch[depends] += "linux-5.10:do_fetch"
+
+require recipes-kernel/linux/fetch-only.inc
-- 
2.25.1

[PATCH 16/21] kernel: add stable fetch recipes for v5.4.x, v5.10.x and v5.12.x

[Paul Gortmaker]

One might wonder why there isn't just a stable-5.x to encompass all of
these, but by having them separate, we can retire/EOL them individually
without disruption.

Note that references are a "round up" to the nearest kernel recipe
containing the stable baseline - so stable-5.4 references linux-5.10 as
we didn't specifically call out linux-5.4 as an independent recipe.

Also, stable-5.12 doesn't exist yet, but we know it will, so we insert
a placeholder in for it, so that v5.12-rt can reference it w/o future
change; we just switch the one line in stable-5.12 once it exists.

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 meta/recipes-kernel/linux/fetch-stable.inc | 24 ++++++++++++++++++++++
 meta/recipes-kernel/linux/stable-5.10.bb   | 10 +++++++++
 meta/recipes-kernel/linux/stable-5.12.bb   | 16 +++++++++++++++
 meta/recipes-kernel/linux/stable-5.4.bb    | 11 ++++++++++
 4 files changed, 61 insertions(+)
 create mode 100644 meta/recipes-kernel/linux/fetch-stable.inc
 create mode 100644 meta/recipes-kernel/linux/stable-5.10.bb
 create mode 100644 meta/recipes-kernel/linux/stable-5.12.bb
 create mode 100644 meta/recipes-kernel/linux/stable-5.4.bb

diff --git a/meta/recipes-kernel/linux/fetch-stable.inc b/meta/recipes-kernel/linux/fetch-stable.inc
new file mode 100644
index 000000000000..051c35db8772
--- /dev/null
+++ b/meta/recipes-kernel/linux/fetch-stable.inc
@@ -0,0 +1,24 @@
+# Boilerplate to fetch a single branch of stable.
+# Reference is non-optional; there is no excuse to not use one.
+# Use a packref for a fixed tag reference; altref for master.
+
+require recipes-kernel/linux/fetch-only.inc
+
+python __anonymous () {
+    kref = d.getVar("KREF") or "master"
+    if kref == "master":
+        d.setVar("RTYPE", "alt")
+    else:
+        d.setVar("RTYPE", "pack")
+    clonehost = d.getVar("KORG_SERVER")
+    d.setVar("KORG_REF", "%s.torvalds.linux-%s" % (clonehost, kref))
+    d.setVar("KORG_REFDEP", "linux-%s:do_fetch"% kref)
+}
+
+do_fetch[depends] += "${KORG_REFDEP}"
+
+STABLE_REV ?= "${STABLE}"
+DL_NAME ?= ".stable.${STABLE}"
+GITCLONEARGS = "--bare --single-branch --branch ${STABLE}"
+GITFETCHREFS ?= "refs/heads/${STABLE}:refs/heads/${STABLE}"
+SRC_URI = "${KORG_URLBASE}/stable/linux-stable;rev=${STABLE_REV};nobranch=1;dlname=${DL_NAME};${RTYPE}ref=${KORG_REF}"
diff --git a/meta/recipes-kernel/linux/stable-5.10.bb b/meta/recipes-kernel/linux/stable-5.10.bb
new file mode 100644
index 000000000000..4dd09fd212d1
--- /dev/null
+++ b/meta/recipes-kernel/linux/stable-5.10.bb
@@ -0,0 +1,10 @@
+SUMMARY = "Linux stable kernel v5.10.y source"
+DESCRIPTION = "Fetch source of the stable linux kernel up to the end of the v5.10.y series"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
+
+STABLE = "linux-5.10.y"
+KREF = "5.10"
+
+require recipes-kernel/linux/fetch-stable.inc
diff --git a/meta/recipes-kernel/linux/stable-5.12.bb b/meta/recipes-kernel/linux/stable-5.12.bb
new file mode 100644
index 000000000000..18cb764dc3f3
--- /dev/null
+++ b/meta/recipes-kernel/linux/stable-5.12.bb
@@ -0,0 +1,16 @@
+SUMMARY = "Linux stable kernel v5.12.y source"
+DESCRIPTION = "Fetch source of the stable linux kernel up to the end of the v5.12.y series"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
+
+# The stable branch "linux-5.12.y" wont exist until late April or early May.
+# For now this will just create a linux-stable "empty" 5.12.x placeholder.
+#STABLE = "linux-5.12.y"
+STABLE = "master"
+DL_NAME = ".stable.linux-5.12.y"
+
+# mainline latest is the default stable reference if none is specified.
+# KREF = "master"
+
+require recipes-kernel/linux/fetch-stable.inc
diff --git a/meta/recipes-kernel/linux/stable-5.4.bb b/meta/recipes-kernel/linux/stable-5.4.bb
new file mode 100644
index 000000000000..98842bacc1ab
--- /dev/null
+++ b/meta/recipes-kernel/linux/stable-5.4.bb
@@ -0,0 +1,11 @@
+SUMMARY = "Linux stable kernel v5.4.y source"
+DESCRIPTION = "Fetch source of the stable linux kernel up to the end of the v5.4.y series"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
+
+STABLE = "linux-5.4.y"
+# For reference, round up to the nearest tag with a recipe
+KREF = "5.10"
+
+require recipes-kernel/linux/fetch-stable.inc
-- 
2.25.1

[PATCH 17/21] kernel: add preempt-rt fetch recipes for v5.4.x, v5.10.x and 5.12.x

[Paul Gortmaker]

Note that the v5.12 content currently is in the devel repo and will
eventually get moved over to the stable-rt repo.  When that move
happens, we'll need to update the SRC_URI accordingly.  [The v5.12-rt
content itself will remain fast-forward with respect to that which is
in devel currently.]

Also, since each reference their respective stable baseline repos,
which are still in maintenance and not static content, we use alternate
reference instead of pack reference back to stable.

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 meta/recipes-kernel/linux/fetch-rt.inc     | 25 ++++++++++++++++++++++
 meta/recipes-kernel/linux/linux-rt-5.10.bb |  9 ++++++++
 meta/recipes-kernel/linux/linux-rt-5.12.bb | 12 +++++++++++
 meta/recipes-kernel/linux/linux-rt-5.4.bb  |  9 ++++++++
 4 files changed, 55 insertions(+)
 create mode 100644 meta/recipes-kernel/linux/fetch-rt.inc
 create mode 100644 meta/recipes-kernel/linux/linux-rt-5.10.bb
 create mode 100644 meta/recipes-kernel/linux/linux-rt-5.12.bb
 create mode 100644 meta/recipes-kernel/linux/linux-rt-5.4.bb

diff --git a/meta/recipes-kernel/linux/fetch-rt.inc b/meta/recipes-kernel/linux/fetch-rt.inc
new file mode 100644
index 000000000000..18dbf0628640
--- /dev/null
+++ b/meta/recipes-kernel/linux/fetch-rt.inc
@@ -0,0 +1,25 @@
+# Boilerplate to fetch a single branch of preempt-rt
+# There are two different repos, and two branching name patterns.
+
+require recipes-kernel/linux/fetch-only.inc
+
+python __anonymous () {
+    rt = d.getVar("RT")
+    rt_repo = d.getVar("RT_REPO")
+    if rt_repo.endswith("devel"):
+        d.setVar("RT_BRANCH", "linux-%s.y-rt" % rt)
+    else:
+        d.setVar("RT_BRANCH", "v%s-rt" % rt)
+    clonehost = d.getVar("KORG_SERVER")
+    d.setVar("KORG_REF", "%s.stable.linux-%s.y" % (clonehost, rt))
+    d.setVar("KORG_REFDEP", "stable-%s:do_fetch" % rt)
+}
+
+do_fetch[depends] += "${KORG_REFDEP}"
+
+RT_REV ?= "${RT_BRANCH}"
+RT_REPO ?= "linux-stable-rt"
+GITCLONEARGS = "--bare --single-branch --branch ${RT_BRANCH}"
+GITFETCHREFS ?= "refs/heads/${RT_BRANCH}:refs/heads/${RT_BRANCH}"
+DL_NAME = ".preempt-rt.linux-${RT}.y"
+SRC_URI = "${KORG_URLBASE}/rt/${RT_REPO};rev=${RT_REV};nobranch=1;dlname=${DL_NAME};altref=${KORG_REF}"
diff --git a/meta/recipes-kernel/linux/linux-rt-5.10.bb b/meta/recipes-kernel/linux/linux-rt-5.10.bb
new file mode 100644
index 000000000000..04520101a443
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-rt-5.10.bb
@@ -0,0 +1,9 @@
+SUMMARY = "Linux preempt-rt v5.10.y kernel source"
+DESCRIPTION = "Fetch source of the preempt-rt linux kernel up to the end of the v5.10.y series"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
+
+RT = "5.10"
+
+require recipes-kernel/linux/fetch-rt.inc
diff --git a/meta/recipes-kernel/linux/linux-rt-5.12.bb b/meta/recipes-kernel/linux/linux-rt-5.12.bb
new file mode 100644
index 000000000000..88f2daf92032
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-rt-5.12.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Linux preempt-rt v5.12.y kernel source"
+DESCRIPTION = "Fetch source of the preempt-rt linux kernel up to the end of the v5.12.y series"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
+
+RT = "5.12"
+
+# Note: v5.12 RT will move to linux-stable-rt repo sometime
+RT_REPO = "linux-rt-devel"
+
+require recipes-kernel/linux/fetch-rt.inc
diff --git a/meta/recipes-kernel/linux/linux-rt-5.4.bb b/meta/recipes-kernel/linux/linux-rt-5.4.bb
new file mode 100644
index 000000000000..3d70be5aaa9b
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-rt-5.4.bb
@@ -0,0 +1,9 @@
+SUMMARY = "Linux preempt-rt v5.4.y kernel source"
+DESCRIPTION = "Fetch source of the preempt-rt linux kernel up to the end of the v5.4.y series"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
+
+RT = "5.4"
+
+require recipes-kernel/linux/fetch-rt.inc
-- 
2.25.1

[PATCH 18/21] kernel: make v5.4.x Yocto recipes use shared source

[Paul Gortmaker]

With all the heavy lifting done, all that is needed is to select the
right endpoint for our reference, and the right fetch wildcards to
ensure we only get what we need. We've got:

	linux-5.10 (superset contining v5.4)
	   --> stable-5.4
	      --> linux-5.4-rt
		--> linux-yocto_5.4 (and rt/tiny variants)

and hence the preempt-rt is the endpoint for the reference.

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 meta/recipes-kernel/linux/linux-yocto-rt_5.4.bb   | 7 ++++++-
 meta/recipes-kernel/linux/linux-yocto-tiny_5.4.bb | 7 ++++++-
 meta/recipes-kernel/linux/linux-yocto_5.4.bb      | 7 ++++++-
 3 files changed, 18 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-kernel/linux/linux-yocto-rt_5.4.bb b/meta/recipes-kernel/linux/linux-yocto-rt_5.4.bb
index 2ffc8ed5428b..373da8501f6b 100644
--- a/meta/recipes-kernel/linux/linux-yocto-rt_5.4.bb
+++ b/meta/recipes-kernel/linux/linux-yocto-rt_5.4.bb
@@ -14,7 +14,11 @@ python () {
 SRCREV_machine ?= "65bbe689d98a007848008be2c8edeb5fa8066829"
 SRCREV_meta ?= "19738ca97b999a3b150e2d34232bb44b6537348f"
 
-SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;branch=${KBRANCH};name=machine \
+KREF = "git.kernel.org.preempt-rt.linux-5.4.y"
+GITFETCHREFS_machine += " refs/heads/v5.4/*:refs/heads/v5.4/*"
+GITCLONEARGS_machine = "--bare --single-branch --branch v5.4/base"
+
+SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;name=machine;branch=${KBRANCH};altref=${KREF} \
            git://git.yoctoproject.org/yocto-kernel-cache;type=kmeta;name=meta;branch=yocto-5.4;destsuffix=${KMETA}"
 
 LINUX_VERSION ?= "5.4.107"
@@ -23,6 +27,7 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=bbea815ee2795b2f4230826c0c6b8814"
 
 DEPENDS += "${@bb.utils.contains('ARCH', 'x86', 'elfutils-native', '', d)}"
 DEPENDS += "openssl-native util-linux-native"
+do_fetch[depends] += "linux-rt-5.4:do_fetch"
 
 PV = "${LINUX_VERSION}+git${SRCPV}"
 
diff --git a/meta/recipes-kernel/linux/linux-yocto-tiny_5.4.bb b/meta/recipes-kernel/linux/linux-yocto-tiny_5.4.bb
index 2b6e35a69c10..40327039368f 100644
--- a/meta/recipes-kernel/linux/linux-yocto-tiny_5.4.bb
+++ b/meta/recipes-kernel/linux/linux-yocto-tiny_5.4.bb
@@ -11,6 +11,7 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=bbea815ee2795b2f4230826c0c6b8814"
 
 DEPENDS += "${@bb.utils.contains('ARCH', 'x86', 'elfutils-native', '', d)}"
 DEPENDS += "openssl-native util-linux-native"
+do_fetch[depends] += "linux-rt-5.4:do_fetch"
 
 KMETA = "kernel-meta"
 KCONF_BSP_AUDIT_LEVEL = "2"
@@ -21,7 +22,11 @@ SRCREV_meta ?= "19738ca97b999a3b150e2d34232bb44b6537348f"
 
 PV = "${LINUX_VERSION}+git${SRCPV}"
 
-SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;branch=${KBRANCH};name=machine \
+KREF = "git.kernel.org.preempt-rt.linux-5.4.y"
+GITFETCHREFS_machine += " refs/heads/v5.4/*:refs/heads/v5.4/*"
+GITCLONEARGS_machine = "--bare --single-branch --branch v5.4/base"
+
+SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;name=machine;branch=${KBRANCH};altref=${KREF} \
            git://git.yoctoproject.org/yocto-kernel-cache;type=kmeta;name=meta;branch=yocto-5.4;destsuffix=${KMETA}"
 
 COMPATIBLE_MACHINE = "qemux86|qemux86-64|qemuarm|qemuarmv5"
diff --git a/meta/recipes-kernel/linux/linux-yocto_5.4.bb b/meta/recipes-kernel/linux/linux-yocto_5.4.bb
index 245c3d574b44..938f3bc9e595 100644
--- a/meta/recipes-kernel/linux/linux-yocto_5.4.bb
+++ b/meta/recipes-kernel/linux/linux-yocto_5.4.bb
@@ -26,7 +26,11 @@ SRCREV_meta ?= "19738ca97b999a3b150e2d34232bb44b6537348f"
 # remap qemuarm to qemuarma15 for the 5.4 kernel
 # KMACHINE_qemuarm ?= "qemuarma15"
 
-SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;name=machine;branch=${KBRANCH}; \
+KREF = "git.kernel.org.preempt-rt.linux-5.4.y"
+GITFETCHREFS_machine += " refs/heads/v5.4/*:refs/heads/v5.4/*"
+GITCLONEARGS_machine = "--bare --single-branch --branch v5.4/base"
+
+SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;name=machine;branch=${KBRANCH};altref=${KREF}; \
            git://git.yoctoproject.org/yocto-kernel-cache;type=kmeta;name=meta;branch=yocto-5.4;destsuffix=${KMETA}"
 
 LIC_FILES_CHKSUM = "file://COPYING;md5=bbea815ee2795b2f4230826c0c6b8814"
@@ -34,6 +38,7 @@ LINUX_VERSION ?= "5.4.107"
 
 DEPENDS += "${@bb.utils.contains('ARCH', 'x86', 'elfutils-native', '', d)}"
 DEPENDS += "openssl-native util-linux-native"
+do_fetch[depends] += "linux-rt-5.4:do_fetch"
 
 PV = "${LINUX_VERSION}+git${SRCPV}"
 
-- 
2.25.1

[PATCH 19/21] kernel: make v5.10.x Yocto recipes use shared source

[Paul Gortmaker]

With all the heavy lifting done, all that is needed is to select the
right endpoint for our reference, and the right fetch wildcards to
ensure we only get what we need. We've got:

	linux-5.10
	   --> stable-5.10
	      --> linux-5.10-rt
		--> linux-yocto_5.10 (and rt/tiny variants)

and hence the preempt-rt is the endpoint for the reference.

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 meta/recipes-kernel/linux/linux-yocto-rt_5.10.bb   | 7 ++++++-
 meta/recipes-kernel/linux/linux-yocto-tiny_5.10.bb | 7 ++++++-
 meta/recipes-kernel/linux/linux-yocto_5.10.bb      | 7 ++++++-
 3 files changed, 18 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-kernel/linux/linux-yocto-rt_5.10.bb b/meta/recipes-kernel/linux/linux-yocto-rt_5.10.bb
index cb34887cda83..072b2eaabac8 100644
--- a/meta/recipes-kernel/linux/linux-yocto-rt_5.10.bb
+++ b/meta/recipes-kernel/linux/linux-yocto-rt_5.10.bb
@@ -14,7 +14,11 @@ python () {
 SRCREV_machine ?= "be2935bce35f9adb6d0e735d42651e81a5094adf"
 SRCREV_meta ?= "031f6c76e488a3563f35258c72ff1de3e25a512e"
 
-SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;branch=${KBRANCH};name=machine \
+KREF = "git.kernel.org.preempt-rt.linux-5.10.y"
+GITFETCHREFS_machine += " refs/heads/v5.10/*:refs/heads/v5.10/*"
+GITCLONEARGS_machine = "--bare --single-branch --branch v5.10/base"
+
+SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;name=machine;branch=${KBRANCH};altref=${KREF} \
            git://git.yoctoproject.org/yocto-kernel-cache;type=kmeta;name=meta;branch=yocto-5.10;destsuffix=${KMETA}"
 
 LINUX_VERSION ?= "5.10.25"
@@ -23,6 +27,7 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
 
 DEPENDS += "${@bb.utils.contains('ARCH', 'x86', 'elfutils-native', '', d)}"
 DEPENDS += "openssl-native util-linux-native"
+do_fetch[depends] += "linux-rt-5.10:do_fetch"
 
 PV = "${LINUX_VERSION}+git${SRCPV}"
 
diff --git a/meta/recipes-kernel/linux/linux-yocto-tiny_5.10.bb b/meta/recipes-kernel/linux/linux-yocto-tiny_5.10.bb
index 83e59b0ebb8f..14db1e578c1d 100644
--- a/meta/recipes-kernel/linux/linux-yocto-tiny_5.10.bb
+++ b/meta/recipes-kernel/linux/linux-yocto-tiny_5.10.bb
@@ -11,6 +11,7 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
 
 DEPENDS += "${@bb.utils.contains('ARCH', 'x86', 'elfutils-native', '', d)}"
 DEPENDS += "openssl-native util-linux-native"
+do_fetch[depends] += "linux-rt-5.10:do_fetch"
 
 KMETA = "kernel-meta"
 KCONF_BSP_AUDIT_LEVEL = "2"
@@ -21,7 +22,11 @@ SRCREV_meta ?= "031f6c76e488a3563f35258c72ff1de3e25a512e"
 
 PV = "${LINUX_VERSION}+git${SRCPV}"
 
-SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;branch=${KBRANCH};name=machine \
+KREF = "git.kernel.org.preempt-rt.linux-5.10.y"
+GITFETCHREFS_machine += " refs/heads/v5.10/*:refs/heads/v5.10/*"
+GITCLONEARGS_machine = "--bare --single-branch --branch v5.10/base"
+
+SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;name=machine;branch=${KBRANCH};altref=${KREF} \
            git://git.yoctoproject.org/yocto-kernel-cache;type=kmeta;name=meta;branch=yocto-5.10;destsuffix=${KMETA}"
 
 COMPATIBLE_MACHINE = "qemux86|qemux86-64|qemuarm|qemuarmv5"
diff --git a/meta/recipes-kernel/linux/linux-yocto_5.10.bb b/meta/recipes-kernel/linux/linux-yocto_5.10.bb
index 026e69511a60..c7a8cb360d2d 100644
--- a/meta/recipes-kernel/linux/linux-yocto_5.10.bb
+++ b/meta/recipes-kernel/linux/linux-yocto_5.10.bb
@@ -28,7 +28,11 @@ SRCREV_meta ?= "031f6c76e488a3563f35258c72ff1de3e25a512e"
 # remap qemuarm to qemuarma15 for the 5.8 kernel
 # KMACHINE_qemuarm ?= "qemuarma15"
 
-SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;name=machine;branch=${KBRANCH}; \
+KREF = "git.kernel.org.preempt-rt.linux-5.10.y"
+GITFETCHREFS_machine += " refs/heads/v5.10/*:refs/heads/v5.10/*"
+GITCLONEARGS_machine = "--bare --single-branch --branch v5.10/base"
+
+SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;name=machine;branch=${KBRANCH};altref=${KREF} \
            git://git.yoctoproject.org/yocto-kernel-cache;type=kmeta;name=meta;branch=yocto-5.10;destsuffix=${KMETA}"
 
 LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
@@ -37,6 +41,7 @@ LINUX_VERSION ?= "5.10.25"
 DEPENDS += "${@bb.utils.contains('ARCH', 'x86', 'elfutils-native', '', d)}"
 DEPENDS += "openssl-native util-linux-native"
 DEPENDS += "gmp-native"
+do_fetch[depends] += "linux-rt-5.10:do_fetch"
 
 PV = "${LINUX_VERSION}+git${SRCPV}"
 
-- 
2.25.1

[PATCH 20/21] kernel: make linux-yocto-dev recipe use shared source

[Paul Gortmaker]

At the moment, linux-yocto-dev is set up to reference v5.12-rt:

        mainline
           ---> stable-5.12
              ---> linux-rt-5.12
                 ---> linux-yocto-dev

We'll probably move to v5.13-rcN in late May and at that point we
can point it at referencing mainline directly while the respective
stable and -rt versions take shape, or perhaps create placeholders
in advance (as was just done for stable-5.12).

But the good thing is that we'll get v5.13-rcN via mainline, which is
already in the above chain, so we can take our time; there is no
pressure to reconsider the chaining of repos synchronously with the
kernel uprev.

And it is also constructive to consider the reverse case: where we are
pointing at mainline directly and a stable (or -rt) gets added/used.

        mainline
           ---> linux-yocto-dev

In this case, if linux-yocto-dev used a shiny new v5.13.1 stable,
without it yet being explicitly a part of the above chain, then the git
objects relating to that would simply flow from the server directly into
our linux-yocto-dev clone directly.  Nothing breaks or falls over.  So
there is no requirement to immediately react to creation of v5.13 stable
or preempt-rt recipes (even though they are trivial).

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 meta/recipes-kernel/linux/linux-yocto-dev.bb | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-kernel/linux/linux-yocto-dev.bb b/meta/recipes-kernel/linux/linux-yocto-dev.bb
index 8725473d1cd8..b4b91593aef8 100644
--- a/meta/recipes-kernel/linux/linux-yocto-dev.bb
+++ b/meta/recipes-kernel/linux/linux-yocto-dev.bb
@@ -19,7 +19,16 @@ include recipes-kernel/linux/linux-yocto-dev-revisions.inc
 KBRANCH = "standard/base"
 KMETA = "kernel-meta"
 
-SRC_URI = "git://git.yoctoproject.org/linux-yocto-dev.git;branch=${KBRANCH};name=machine \
+# When v5.13-rcN comes along, we can revector back to mainline
+#KREF = "git.kernel.org.torvalds.linux-master"
+#do_fetch[depends] += "linux-master:do_fetch"
+KREF = "git.kernel.org.preempt-rt.linux-5.12.y"
+do_fetch[depends] += "linux-rt-5.12:do_fetch"
+
+GITFETCHREFS_machine += " refs/heads/standard/*:refs/heads/standard/*"
+GITCLONEARGS_machine = "--bare --single-branch --branch master"
+
+SRC_URI = "git://git.yoctoproject.org/linux-yocto-dev.git;name=machine;branch=${KBRANCH};altref=${KREF} \
            git://git.yoctoproject.org/yocto-kernel-cache;type=kmeta;name=meta;branch=master;destsuffix=${KMETA}"
 
 # Set default SRCREVs. Both the machine and meta SRCREVs are statically set
-- 
2.25.1

[PATCH 21/21] kernel: disable (pre)mirror for linux-yocto and linux-yocto-dev

[Paul Gortmaker]

As per the comment in the file itself, we have to explicitly disable
(pre)mirror support for both kernel repositories, or else the fetcher
will unhelpfully volunteer to wget gigs of what amounts to largely
useless content that we don't want.

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 meta/recipes-kernel/linux/linux-yocto.inc | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/meta/recipes-kernel/linux/linux-yocto.inc b/meta/recipes-kernel/linux/linux-yocto.inc
index a85280990b43..5ec1b1ba7023 100644
--- a/meta/recipes-kernel/linux/linux-yocto.inc
+++ b/meta/recipes-kernel/linux/linux-yocto.inc
@@ -56,3 +56,13 @@ do_install_append(){
 
 # enable kernel-sample for oeqa/runtime/cases's ksample.py test
 KERNEL_FEATURES_append_qemuall=" features/kernel-sample/kernel-sample.scc"
+
+# For linux-yocto-dev, the existing v3.17 based premirror tarball is of no
+# value, and for the non-dev variants, we don't want to be going off and
+# w-getting the 3+ gigabyte (!!!) mirror tarball either, when we've got
+# a shared v5.10 repository already.  We can delete this in the future once
+# these old tarballs are gone.  But really we may never bother to mirror
+# either of them ever again, since they are (a) now small, and (b) leaf
+# nodes at the end of the chain where the content is regularly changing.
+PREMIRRORS = ""
+MIRRORS = ""
-- 
2.25.1

Re: [PATCH 11/21] kernel: add a fetch-only recipe for mainline v5.10 source

[Bruce Ashfield]

On Fri, Apr 2, 2021 at 1:16 PM Paul Gortmaker
<paul.gortmaker@windriver.com> wrote:
>
> With v5.10 being the newest baseline currently in use by linux-yocto and
> with the download size being 1/2 the size of current linux-yocto itself,
> the v5.10 makes a good initial line in the sand for blocking out the
> source in a way to optimize sharing.
>
> With this commit present, one can test via "bitbake -c fetch linux-5.10".
>
> Pending recipes who will want this source present will add:
>         do_fetch[depends] += "linux-5.10:do_fetch"
>
> Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
> ---
>  meta/recipes-kernel/linux/fetch-linux.inc | 7 +++++++
>  meta/recipes-kernel/linux/linux-5.10.bb   | 9 +++++++++
>  2 files changed, 16 insertions(+)
>  create mode 100644 meta/recipes-kernel/linux/fetch-linux.inc
>  create mode 100644 meta/recipes-kernel/linux/linux-5.10.bb
>
> diff --git a/meta/recipes-kernel/linux/fetch-linux.inc b/meta/recipes-kernel/linux/fetch-linux.inc
> new file mode 100644
> index 000000000000..e3fdd0bb36c2
> --- /dev/null
> +++ b/meta/recipes-kernel/linux/fetch-linux.inc
> @@ -0,0 +1,7 @@
> +# Boilerplate to fetch up to a single tag of mainline
> +
> +require recipes-kernel/linux/fetch-only.inc
> +
> +GITCLONEARGS = "--bare --single-branch --branch v${KTAG}"
> +DL_NAME = ".torvalds.linux-${KTAG}"
> +SRC_URI = "${KORG_URLBASE}/torvalds/linux;tag=v${KTAG};nobranch=1;static=1;dlname=${DL_NAME}"
> diff --git a/meta/recipes-kernel/linux/linux-5.10.bb b/meta/recipes-kernel/linux/linux-5.10.bb
> new file mode 100644
> index 000000000000..7f8a27bcb1f0
> --- /dev/null
> +++ b/meta/recipes-kernel/linux/linux-5.10.bb

A specific comment is that this should actually have "fetch-only" in
the name of the recipe.

We want to avoid any confusion about this being buildable, and that it
is not a jumping point
for kernel builds.

Trust me, someone will ask.

linux-yocto has v5.10/base which is vanilla -stable, and simply
because the name is
linux-yocto, people never realize that they can build a completely
stock greg-kh kernel
until they turn blue :D

Otherwise, I like the concept and can take it for a spin.

Bruce



> @@ -0,0 +1,9 @@
> +SUMMARY = "Linux kernel v5.10 source"
> +DESCRIPTION = "Fetch source of the linux kernel up to the v5.10 tag"
> +
> +LICENSE = "GPLv2"
> +LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
> +
> +KTAG = "5.10"
> +
> +require recipes-kernel/linux/fetch-linux.inc
> --
> 2.25.1
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [PATCH 15/21] kernel: add recipe for linux-master (mainline latest)

[Bruce Ashfield]

On Fri, Apr 2, 2021 at 1:16 PM Paul Gortmaker
<paul.gortmaker@windriver.com> wrote:
>
> The idea for this recipe is to have a name "linux-master" that people
> can depend on as a reference for mainline; today and ideally for as long
> as we are maintaining it.  While the recipe is there for everybody,
> the obvious user we are creating this for is the linux-yocto-dev repo.
>
> At some point in the future, we'll carve off another chunk of post-v5.10
> work, mark it static and put it alongside of v5.10 and let this mainline
> recipe reference that.  But use of this "linux-master" will be 100%
> transparent to the underlying backing store changes.
>
> Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
> ---
>  meta/recipes-kernel/linux/linux-master.bb | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
>  create mode 100644 meta/recipes-kernel/linux/linux-master.bb
>
> diff --git a/meta/recipes-kernel/linux/linux-master.bb b/meta/recipes-kernel/linux/linux-master.bb
> new file mode 100644
> index 000000000000..f674c2562e81
> --- /dev/null
> +++ b/meta/recipes-kernel/linux/linux-master.bb

I won't reply to all of these fetch only naming patches (maybe this is
the last one,
I didn't check), so this is my last one.

My same namespace comment applies here

Actually, thinking of it more, we could even have these both with fetch-only in
the recipe name AND in a directory structure that implies the same thing.

And thirdly, maybe we could taint the tasks in some way to prevent that jumping
point to building.

Since yes, you'll get someone trying to ask for support using this to build
version 'x' of mainline. I have the battle scars to prove it :D

Bruce

> @@ -0,0 +1,13 @@
> +SUMMARY = "Linux kernel mainline source"
> +DESCRIPTION = "Fetch source of the latest linux mainline kernel"
> +
> +LICENSE = "GPLv2"
> +LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
> +
> +DL_NAME = ".torvalds.linux-master"
> +KORG_REF = "git.kernel.org.torvalds.linux-5.10"
> +SRC_URI = "${KORG_URLBASE}/torvalds/linux;rev=master;nobranch=1;dlname=${DL_NAME};packref=${KORG_REF}"
> +
> +do_fetch[depends] += "linux-5.10:do_fetch"
> +
> +require recipes-kernel/linux/fetch-only.inc
> --
> 2.25.1
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [PATCH 21/21] kernel: disable (pre)mirror for linux-yocto and linux-yocto-dev

[Bruce Ashfield]

On Fri, Apr 2, 2021 at 1:16 PM Paul Gortmaker
<paul.gortmaker@windriver.com> wrote:
>
> As per the comment in the file itself, we have to explicitly disable
> (pre)mirror support for both kernel repositories, or else the fetcher
> will unhelpfully volunteer to wget gigs of what amounts to largely
> useless content that we don't want.
>
> Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
> ---
>  meta/recipes-kernel/linux/linux-yocto.inc | 10 ++++++++++
>  1 file changed, 10 insertions(+)
>
> diff --git a/meta/recipes-kernel/linux/linux-yocto.inc b/meta/recipes-kernel/linux/linux-yocto.inc
> index a85280990b43..5ec1b1ba7023 100644
> --- a/meta/recipes-kernel/linux/linux-yocto.inc
> +++ b/meta/recipes-kernel/linux/linux-yocto.inc
> @@ -56,3 +56,13 @@ do_install_append(){
>
>  # enable kernel-sample for oeqa/runtime/cases's ksample.py test
>  KERNEL_FEATURES_append_qemuall=" features/kernel-sample/kernel-sample.scc"
> +
> +# For linux-yocto-dev, the existing v3.17 based premirror tarball is of no
> +# value, and for the non-dev variants, we don't want to be going off and
> +# w-getting the 3+ gigabyte (!!!) mirror tarball either, when we've got
> +# a shared v5.10 repository already.  We can delete this in the future once
> +# these old tarballs are gone.  But really we may never bother to mirror
> +# either of them ever again, since they are (a) now small, and (b) leaf
> +# nodes at the end of the chain where the content is regularly changing.
> +PREMIRRORS = ""
> +MIRRORS = ""

I suppose these are already set somewhere else, so a ?= won't actually
override the defaults.

I was just trying to think of a way that someone might want to turn it
back on, and they'd have to use an override in a bbappend to do it ..
but I suppose that isn't any nastier than would be required with ?=

Bruce

> --
> 2.25.1
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos

[Richard Purdie]

On Fri, 2021-04-02 at 13:15 -0400, Paul Gortmaker wrote:
> Next Steps:
> -----------
> With this being a functional implementation, it seems like a good time to
> get other people looking at it.  Ideally step #1 will be getting general
> agreement that this is something we need, something that is overdue,
> and that the implementation as shown here makes sense in the absence of
> any similar effort from anyone that does the same but in a better way.
> 
> From there, we'll want more people not just looking at it, but testing it
> as well.   I know I want to write a commit (script?) that will avoid any
> "transition tax" by prepopulating new repos with "old" already downloaded
> git objects where we can.  And to add/do tests with my own popluated mirror
> and NO_NETWORK, and also try to ensure nothing in BB_SHALLOW gets upset,
> but I wasn't going to hold up starting a review of this any longer.
> 
> I suspect I can get some co-workers using/testing it too, but Yocto gets
> used in a bunch of different ways by different groups, so we'll no doubt
> have to do some additional fixups to ensure everybody gets the benefits of
> this sharing.  But I'm hopeful that when people see the benefits above,
> they'll pitch in to help take this the final mile by ensuring it works for
> their use case as well.
> 
> I'm not too worried about pontificating out beyond that until we get past
> the acceptance/testing hurdles outlined above.  So, please do have a read
> of the commits, kick the tires, put on your bikeshedding clothes and grab
> a brush, and lets see where it goes from here...
> 
> https://github.com/paulgortmaker/poky/compare/reference-RC1

I've only briefly looked through this but the more I look, the more worried
I'm getting which is never a good sign.

I totally agree with the use case and agree we need to do something in this 
area. I'm not sure the implementation looks right though, its clever but I
don't think its very usable to end users.

The warning signs to me are:

a) Needing new/confusing "fetch only" recipes
b) A large number of new options and variables to the fetcher
c) Needing recipes to change and people to migrate, potentially with scripts
   between old and new
d) Variable namespacing needs work

I'm very worried this confuses up the git fetcher code and nobody will be able
to tell what is going on any more :/.

What I'd envisaged was git urls having something like a "mainline-linux-kernel"
tag added in the url as a parameter and a table somewhere which meant the checkout
for this would share git objects in a common pool. There would be a variable 
mapping that name to git.kernel.org/pub/scm/linux/kernel/git somewhere but that
should be all that is needed.

No, this wouldn't cover 100% of artefacts but it should cover a majority of them
and be much simpler for users to comprehend. I haven't gone into this in detail
and perhaps I'm missing some problem that prevents it? :/

The other issue here is there are no tests. The bitbake fetcher code is one of
the few pieces of the project where we do have a fairly complete test suite and
we don't add things there without tests (see bitbake-selftest and lib/bb/tests/).

Cheers,

Richard

Re: [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos

[Paul Gortmaker]

[Re: [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos] On 02/04/2021 (Fri 23:14) Richard Purdie wrote:

> On Fri, 2021-04-02 at 13:15 -0400, Paul Gortmaker wrote:
> > Next Steps:
> > -----------
> > With this being a functional implementation, it seems like a good time to
> > get other people looking at it.  Ideally step #1 will be getting general
> > agreement that this is something we need, something that is overdue,
> > and that the implementation as shown here makes sense in the absence of
> > any similar effort from anyone that does the same but in a better way.
> > 
> > From there, we'll want more people not just looking at it, but testing it
> > as well.   I know I want to write a commit (script?) that will avoid any
> > "transition tax" by prepopulating new repos with "old" already downloaded
> > git objects where we can.  And to add/do tests with my own popluated mirror
> > and NO_NETWORK, and also try to ensure nothing in BB_SHALLOW gets upset,
> > but I wasn't going to hold up starting a review of this any longer.
> > 
> > I suspect I can get some co-workers using/testing it too, but Yocto gets
> > used in a bunch of different ways by different groups, so we'll no doubt
> > have to do some additional fixups to ensure everybody gets the benefits of
> > this sharing.  But I'm hopeful that when people see the benefits above,
> > they'll pitch in to help take this the final mile by ensuring it works for
> > their use case as well.
> > 
> > I'm not too worried about pontificating out beyond that until we get past
> > the acceptance/testing hurdles outlined above.  So, please do have a read
> > of the commits, kick the tires, put on your bikeshedding clothes and grab
> > a brush, and lets see where it goes from here...
> > 
> > https://github.com/paulgortmaker/poky/compare/reference-RC1
> 
> I've only briefly looked through this but the more I look, the more worried
> I'm getting which is never a good sign.

Well, firstly thanks for the quick response, and I hope you'll give it
another look when you have more time, because I'm not quite convinced
I've conveyed how it is to be used in a way that got through as intended.

> I totally agree with the use case and agree we need to do something in this 
> area. I'm not sure the implementation looks right though, its clever but I
> don't think its very usable to end users.

Not sure who you consider "end users".  I don't really consider myself
or Bruce or anyone else crafting up a kernel recipe from scratch as an
end user.  But even if an "end user" wants to do that, everything here
is opt-in.  If you want to go clone 3G of stuff from mysuperkernel.org
in your new recipe, then feel free.  Nothing stops them.

> The warning signs to me are:
> 
> a) Needing new/confusing "fetch only" recipes

Not sure how they are confusing - and not really new either; I stole the
idea of zeroing out all the tasks from kernel-devsrc if I recall...

> b) A large number of new options and variables to the fetcher

I'd disagree with the characterization of "large" - but if you do have
the time, please go look over the commit logs for each again.  They
all clearly indicate a necessary use case we don't handle well (or at
all) -- and we'll have to solve them one way or another.  I'm open to
considering alternate solutions for any of them.

> c) Needing recipes to change and people to migrate, potentially with scripts
>    between old and new

It feels like we are looking at two different things.  The users or
"people" don't really migrate and the only people changing things would
be a small subset of people who maintain kernel recipes - which is a
small group to begin with.  Recycling old git objects from existing
downloads would most likey be done in a transparent and hands free way.

> d) Variable namespacing needs work

OK - if given specifics it can and will be addressed.

> I'm very worried this confuses up the git fetcher code and nobody will be able
> to tell what is going on any more :/.

That kind of surprises me, to hear that after having looked at the shallow clone
changes to the fetcher, but again - I'm listening - what can we change
in the fetcher?  What is off limits?  The "pool" idea below doesn't address
the fact that "--mirror" is simply not viable for some repos, and that
is hard-coded into the fetcher.   Similar for the other fetcher changes.
And the "pool" idea is still going to have to teach the fetcher about
some kind of "--reference" because that *has* to be added at fetch/clone
time and not as some later afterthought post-fetch.

I could simply take over do_fetch for the kernel and leave the fetcher
untouched, but when the problems being solved weren't really kernel
specific, that didn't seem like the right approach.

> What I'd envisaged was git urls having something like a "mainline-linux-kernel"
> tag added in the url as a parameter and a table somewhere which meant the checkout
> for this would share git objects in a common pool. There would be a variable 
> mapping that name to git.kernel.org/pub/scm/linux/kernel/git somewhere but that
> should be all that is needed.

I saw this idea floated in the earlier thread that Randy pointed me at,
but I just don't see it being viable or the right solution even if one
could make it work.  How do you decide what lives in the pool?  If it is
just mainline, then it really isn't a pool.  Are we only solving for the
kernel, or are we making a solution for any repo that is unweildy?  What
triggers populating the pool?  Do you stuff all mainline and -rt and all
of stable in the pool?  Or just chunks of rt and stable?  If you aren't
tracking what is in there via encapsulating chunks somehow, how do you
expire stuff from the pool?  If it is one giant blob covering mainline
and linux-yocto objects, with no fetch-on-demand properties, then we are
back to the same old 3G-downloads-from-the-mirror problem.  How do you
split that monster?  The devil is in the details.

> No, this wouldn't cover 100% of artefacts but it should cover a majority of them
> and be much simpler for users to comprehend. I haven't gone into this in detail
> and perhaps I'm missing some problem that prevents it? :/

Well, I've thought about this a bunch, and I didn't come up with a whole
bunch of options to choose from that deliver what this delivers.

A mainline-only kernel-specific non-pool "pool" would be better than
nothing, but I'd still want to hear you flesh out how you thought it
would look and operate before I went out and tried to guess what you had
in mind.

Also, I'm still confused by this use of  "users" - what is here all
happens transparently in the background and users aren't aware of it any
more than they are aware of what the fetcher is currently doing today.

> The other issue here is there are no tests. The bitbake fetcher code is one of
> the few pieces of the project where we do have a fairly complete test suite and
> we don't add things there without tests (see bitbake-selftest and lib/bb/tests/).

Yes, of course; and I've been told that by several people in advance.
But as you can imagine, I wasn't going to write tests without 1st
floating the general concept as an RFC and seeing how that went.

Anyway, thanks for the initial scan and I do hope you have a chance to
revisit it in more detail and consider the various problems it solves.

Paul.
--

> 
> Cheers,
> 
> Richard
> 

Re: [PATCH 02/21] bitbake: fetch2/git: allow limiting upstream fetch refs to a subset

[Richard Purdie]

On Fri, 2021-04-02 at 13:15 -0400, Paul Gortmaker wrote:
> If a clone in the download directory is not static, and was created with
> single-branch to avoid additonal unwanted content, then the fetcher will
> come along and spoil that effort by unconditonally getting refs/* from
> the server and downloading everything available at the server.
> 
> To that end, allow a companion variable that can optionally be used to
> limit the fetch to a subset of references that are in line with the
> desired content for the recipe in question, and so that large unwanted
> downloads aren't inadvertently triggered during an update.
> 
> If not specified, the existing "fetch everything" behaviour remains.
> 
> Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
> ---
>  bitbake/lib/bb/fetch2/git.py           | 3 ++-
>  documentation/ref-manual/variables.rst | 8 ++++++++
>  2 files changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
> index 22281e2cfb98..b54ec76d7174 100644
> --- a/bitbake/lib/bb/fetch2/git.py
> +++ b/bitbake/lib/bb/fetch2/git.py
> @@ -362,7 +362,8 @@ class Git(FetchMethod):
>                runfetchcmd("%s remote rm origin" % ud.basecmd, d, workdir=ud.clonedir)
>  
> 
> 
> 
>              runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=ud.clonedir)
> -            fetch_cmd = "LANG=C %s fetch -f --progress %s refs/*:refs/*" % (ud.basecmd, shlex.quote(repourl))
> +            fetchrefs = d.getVar('GITFETCHREFS_' + ud.names[0]) or d.getVar('GITFETCHREFS') or "refs/*:refs/*"
> +            fetch_cmd = "LANG=C %s fetch -f --progress %s %s" % (ud.basecmd, shlex.quote(repourl), fetchrefs)
>              if ud.proto.lower() != 'file':
>                  bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
>              progresshandler = GitProgressHandler(d)

I think one of the reasons I'm unease about this series is that some of the 
changes break the existing fetcher functionality. Taking the above, I 
understand why you want to filter the references but this will break the
mirroring.

How? Imagine a repo has branches A and B and the recipe filters to A. This 
will result in a mirror tarball only with A in it. The mirror tarball name will
attempt to be reused by a recipe referencing with B and the mirroring 
mechanism will fail.

Mirroring is the reason we do use all pull the whole repo here. It isn't
good from an efficiency standpoint but it does work well for mirroring. This
piece of code from the fetcher kind of summarises it:

"""
gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace('
','_'))
if gitsrcname.startswith('.'):
    gitsrcname = gitsrcname[1:]

# for rebaseable git repo, it is necessary to keep mirror tar ball
# per revision, so that even the revision disappears from the
# upstream repo in the future, the mirror will remain intact and still
# contains the revision
if ud.rebaseable:
    for name in ud.names:
        gitsrcname = gitsrcname + '_' + ud.revisions[name]

mirrortarball = 'git2_%s.tar.gz' % gitsrcname
"""

To make the above "work", you'd need to rename the mirror tarball to be
branch specific and include A or B in the mirror tarball filename.

I also then wonder whether the form of GITFETCHREFS is good API. It is certainly
powerful however the code already knows which branch we're asking it to fetch
so we could just add a parameter to the url "clonebranchonly" which asks it
to only clone that branch and not all the references?

I appreciate that doesn't give you the way to say "heads/v5.4/*" but instead
you'd get something more specific like "v5.4/standard/base" but that would 
filter the fetch to me even more efficient to the particularly use case a
user would generally be working with (a specific machine). I appreciate that
you're not going to like a mirror tarball per kernel branch and even trying
to get the system to generate all the right tarballs would be a nightmare.

Another alternative could be a parameter like ";filterclone=heads/v5.4/" which
the fetcher could then translate to "refs/heads/v5.4/*:refs/heads/v5.4/*" 
internally.

I guess in summary this means I have two worries about the API. Ideally we 
want something which can be put simply in the URL itself avoiding external 
variables. We also want something which is end user readable, rather than 
being a direct copy of the git commandline. I'd prefer to avoid direct 
commandlines as they only encourage people to start using them to different 
things and then we end up breaking mirrors, have incomplete test coverage
and/or find it really hard to add/change functionality later.

Cheers,

Richard

Re: [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos

[Richard Purdie]

On Fri, 2021-04-02 at 21:44 -0400, Paul Gortmaker wrote:
> [Re: [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos] On 02/04/2021 (Fri 23:14) Richard Purdie wrote:
> 
> > On Fri, 2021-04-02 at 13:15 -0400, Paul Gortmaker wrote:
> > > Next Steps:
> > > -----------
> > > With this being a functional implementation, it seems like a good time to
> > > get other people looking at it.  Ideally step #1 will be getting general
> > > agreement that this is something we need, something that is overdue,
> > > and that the implementation as shown here makes sense in the absence of
> > > any similar effort from anyone that does the same but in a better way.
> > > 
> > > From there, we'll want more people not just looking at it, but testing it
> > > as well.   I know I want to write a commit (script?) that will avoid any
> > > "transition tax" by prepopulating new repos with "old" already downloaded
> > > git objects where we can.  And to add/do tests with my own popluated mirror
> > > and NO_NETWORK, and also try to ensure nothing in BB_SHALLOW gets upset,
> > > but I wasn't going to hold up starting a review of this any longer.
> > > 
> > > I suspect I can get some co-workers using/testing it too, but Yocto gets
> > > used in a bunch of different ways by different groups, so we'll no doubt
> > > have to do some additional fixups to ensure everybody gets the benefits of
> > > this sharing.  But I'm hopeful that when people see the benefits above,
> > > they'll pitch in to help take this the final mile by ensuring it works for
> > > their use case as well.
> > > 
> > > I'm not too worried about pontificating out beyond that until we get past
> > > the acceptance/testing hurdles outlined above.  So, please do have a read
> > > of the commits, kick the tires, put on your bikeshedding clothes and grab
> > > a brush, and lets see where it goes from here...
> > > 
> > > https://github.com/paulgortmaker/poky/compare/reference-RC1
> > 
> > I've only briefly looked through this but the more I look, the more worried
> > I'm getting which is never a good sign.
> 
> Well, firstly thanks for the quick response, and I hope you'll give it
> another look when you have more time, because I'm not quite convinced
> I've conveyed how it is to be used in a way that got through as intended.

> 
> > I totally agree with the use case and agree we need to do something in this 
> > area. I'm not sure the implementation looks right though, its clever but I
> > don't think its very usable to end users.
> 
> Not sure who you consider "end users".  I don't really consider myself
> or Bruce or anyone else crafting up a kernel recipe from scratch as an
> end user.

With the bitbake fetcher you are exposing an API. Anyone using that API is an
end user of that API. You and Bruce are, as is anyone writing a git:// url into
a recipe. I've drilled down into one of the changes in a separate mail to try 
and give a bit more of an idea about what I mean about usability of the API.

Yes, we can just add variables and hardcoded bits of git commandlines but we have
been here before with the old bitbake fetcher v1 code. As soon as you start bolting
in extra variables, things start to interact really badly. That change already
breaks mirroring in ways which isn't obvious even to you/Bruce.

This why I care quite a lot about how we expose the controls and we try and do it
in a way which gives you the capabilities you want/need but in a way which allows
us to change the fetcher in the future, allow people to understand what the fetcher
is doing from the configuration in the url and have decent tests.

> But even if an "end user" wants to do that, everything here is opt-in.  If you 
> want to go clone 3G of stuff from mysuperkernel.org in your new recipe, then feel 
> free.  Nothing stops them.

I'm not asking this to be made kernel only. I do want a generic API which everyone
can use. Ideally I want one which all end users and not just you+Bruce understand
too and part of that is making the user visible API simple and intuitive so everyone
can get in on the action. If that makes the fetcher a little more complex behind the
scenes, that is ok, we do ask for tests for that.

> > The warning signs to me are:
> > 
> > a) Needing new/confusing "fetch only" recipes
> 
> Not sure how they are confusing - and not really new either; I stole the
> idea of zeroing out all the tasks from kernel-devsrc if I recall...

I think you misunderstand me. I don't mean the recipes are confusing, I don't 
want them to exist at all. Consider these two ways of enabling alternates:

a) The user needs to add a new fetch-only recipe, update the SRC_URI, depend on 
the fetch only recipe, set several other variables

b) The user adds a parameter to the SRC_URI specifying the series it matches
and if not already there, adds an entry to the alternates mirror list

I would argue b) is a lot simpler to the end user and will allow more people
to join in using it.

> > b) A large number of new options and variables to the fetcher
> 
> I'd disagree with the characterization of "large" - but if you do have
> the time, please go look over the commit logs for each again.

You added two variables, GITCLONEARGS and GITFETCHREFS and four new parameters 
(dlname, static, altref, packref). The only variables currently in use in the 
fetcher are global behaviour ones, not url specific. There are currently 8 
documented parameters.

I've tried to explain why the variables worry me and why we don't see them in the
fetcher at all. A 50% increase in parameters is a large increase.

>   They all clearly indicate a necessary use case we don't handle well (or at
> all) -- and we'll have to solve them one way or another.  I'm open to
> considering alternate solutions for any of them.
> 
> > c) Needing recipes to change and people to migrate, potentially with scripts
> >    between old and new
> 
> It feels like we are looking at two different things.  The users or
> "people" don't really migrate and the only people changing things would
> be a small subset of people who maintain kernel recipes - which is a
> small group to begin with.  Recycling old git objects from existing
> downloads would most likey be done in a transparent and hands free way.
> 
> > d) Variable namespacing needs work
> 
> OK - if given specifics it can and will be addressed.
> 
> > I'm very worried this confuses up the git fetcher code and nobody will be able
> > to tell what is going on any more :/.
> 
> That kind of surprises me, to hear that after having looked at the shallow clone
> changes to the fetcher, but again - I'm listening - what can we change
> in the fetcher?  What is off limits?  The "pool" idea below doesn't address
> the fact that "--mirror" is simply not viable for some repos, and that
> is hard-coded into the fetcher.   Similar for the other fetcher changes.
> And the "pool" idea is still going to have to teach the fetcher about
> some kind of "--reference" because that *has* to be added at fetch/clone
> time and not as some later afterthought post-fetch.
> 
> I could simply take over do_fetch for the kernel and leave the fetcher
> untouched, but when the problems being solved weren't really kernel
> specific, that didn't seem like the right approach.

I think we're thinking about this from two different approaches. You're taking a 
design of what you want to happen and then pushing the fetcher around to do what 
you need. I think we need to look at how the user would configure and use this,
then work backwards to what the fetcher needs to do to support that.

There are a lot of moving parts here and it would help to simplify things down
to not try and solve every problem in one go. I'm going to focus on the alternates
issue here. This doesn't mean I don't understand the desire to be more efficient
about clones for example, I just need to start somewhere and see if we can move
forward.

The current way this is configured in your series is:

KREF = "git.kernel.org.preempt-rt.linux-5.4.y"
GITFETCHREFS_machine += " refs/heads/v5.4/*:refs/heads/v5.4/*"
GITCLONEARGS_machine = "--bare --single-branch --branch v5.4/base"

SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;name=machine;branch=${KBRANCH};altref=${KREF} \
           git://git.yoctoproject.org/yocto-kernel-cache;type=kmeta;name=meta;branch=yocto-5.4;destsuffix=${KMETA}"
do_fetch[depends] += "linux-rt-5.4:do_fetch"


and what I'd much prefer is something simpler like:


SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;name=machine;branch=${KBRANCH};altref=preempt-rt-linux-kernel \
           git://git.yoctoproject.org/yocto-kernel-cache;type=kmeta;name=meta;branch=yocto-5.4;destsuffix=${KMETA}"

BB_GIT_ALTREF_URLMAP[preempt-rt-linux-kernel] = "git.kernel.org.preempt-rt.linux-5.4.y"


Ignoring the lack of efficiency of the cloning bit, the usability win if we 
could do that is that anyone can start to benefit from the mirroring, just by
adding a single extra parameter to their git url.

The knowledge of the upstream clone repo can be encoded into one variable, 
centrally which everyone can use and I'd imagine we'd quickly build up a decent set
of mirror urls which everyone would build around. Its easy to extend/share.
I'd also note the variable is for central config to the fetcher, it isn't url
specific and it is namespaced as bitbake bitbake and git fetcher (which sounds
trivial but it is a huge problem in the wider code).

Now I'd defined how I'd like the user to see it, the question is can we implement 
that? I understand why you added extra 'fetch only' recipes, they are there 
basically to allow you to fetch a different url. I think it shouldn't be too 
painful to actually have the fetcher fetch that other url behind the scenes.
The gitsm fetcher (for all I dislike it) already does this.

The idea here would therefore be to teach the git fetcher that when it finds
configuration like this, it would look for and update a altref repo first
before it would then make its own clone.

Does that sound reasonable in principle at least? You'd agree that if we can, 
dropping the need for the "fetch-only" recipes would be good, ideally?

> > What I'd envisaged was git urls having something like a "mainline-linux-kernel"
> > tag added in the url as a parameter and a table somewhere which meant the checkout
> > for this would share git objects in a common pool. There would be a variable 
> > mapping that name to git.kernel.org/pub/scm/linux/kernel/git somewhere but that
> > should be all that is needed.
> 
> I saw this idea floated in the earlier thread that Randy pointed me at,
> but I just don't see it being viable or the right solution even if one
> could make it work.  How do you decide what lives in the pool?  If it is
> just mainline, then it really isn't a pool.  Are we only solving for the
> kernel, or are we making a solution for any repo that is unweildy?  What
> triggers populating the pool?  Do you stuff all mainline and -rt and all
> of stable in the pool?  Or just chunks of rt and stable?  If you aren't
> tracking what is in there via encapsulating chunks somehow, how do you
> expire stuff from the pool?  If it is one giant blob covering mainline
> and linux-yocto objects, with no fetch-on-demand properties, then we are
> back to the same old 3G-downloads-from-the-mirror problem.  How do you
> split that monster?  The devil is in the details.

Firstly, there isn't one pool, there are multiple "pools" each with some 
namespace. I was thinking they'd just track that particular upstream repo
and the name of the pool would indicate what you'd expect to find there.
Basically its like your altref parameter but I'm abstracting away their
creation/maintenance to be more magic.

The nice thing about altref "pools" is that they don't have mirroring 
constraints but lets try and come back to revision/branch/object filtering
once we get the above discussed.

> > No, this wouldn't cover 100% of artefacts but it should cover a majority of them
> > and be much simpler for users to comprehend. I haven't gone into this in detail
> > and perhaps I'm missing some problem that prevents it? :/
> 
> Well, I've thought about this a bunch, and I didn't come up with a whole
> bunch of options to choose from that deliver what this delivers.
> 
> A mainline-only kernel-specific non-pool "pool" would be better than
> nothing, but I'd still want to hear you flesh out how you thought it
> would look and operate before I went out and tried to guess what you had
> in mind.
> 
> Also, I'm still confused by this use of  "users" - what is here all
> happens transparently in the background and users aren't aware of it any
> more than they are aware of what the fetcher is currently doing today.

See if I managed to explain it better above.

> > The other issue here is there are no tests. The bitbake fetcher code is one of
> > the few pieces of the project where we do have a fairly complete test suite and
> > we don't add things there without tests (see bitbake-selftest and lib/bb/tests/).
> 
> Yes, of course; and I've been told that by several people in advance.
> But as you can imagine, I wasn't going to write tests without 1st
> floating the general concept as an RFC and seeing how that went.

You could have mentioned that you planned to add them :)

> Anyway, thanks for the initial scan and I do hope you have a chance to
> revisit it in more detail and consider the various problems it solves.

It was late and I clearly didn't convey what I'm thinking in that reply
or in the previous thread. I've tried to do so more above, I did spend 
half the night with this going around in my head and I've spent quite
some time trying to write it down this morning. See if that makes
my concerns clearer and whether what I'm thinking could be made to work.

Cheers,

Richard