* [PATCH 01/21] bitbake: fetch2/git: allow override of clone args with GITCLONEARGS
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
@ 2021-04-02 17:15 ` Paul Gortmaker
2021-04-02 17:15 ` [PATCH 02/21] bitbake: fetch2/git: allow limiting upstream fetch refs to a subset Paul Gortmaker
` (20 subsequent siblings)
21 siblings, 0 replies; 29+ messages in thread
From: Paul Gortmaker @ 2021-04-02 17:15 UTC (permalink / raw)
To: Bruce Ashfield, Richard Purdie; +Cc: linux-yocto, bitbake-devel
Currently there is no way to avoid the "--mirror" argument when doing an
initial clone. This is problematic in that there are now a significant
number of repos out there for which we want the content, but creating a
full mirror clone isn't practical or efficient.
The two most common examples are:
1) Repos that have "ingested" another repo with no common ancestors - we
see this in "stable-queue" and "linux-rt-devel" repos[1] which started
as repos of queued format-patch output, but then also added the baseline
so the patches could be obtained "pre-applied" for easier automated testing.
Obviously, ingesting all of the linux history grows a repo by gigabytes.
2) Repos that have gone forward in time in a direction we don't care about,
say for example a flag day event, where everything was rewritten from C into
rust in one massive commit, or where a merge of a giant wad of orthogonal
history and/or binary/proprietary content "pollutes" the repo.
In both cases, we can solve the problem by using "--single-branch"
optionally in conjunction with "--branch <goodstuff>" in order to limit
the cloned content to a specific branch or tag. We will only get
content up to that point, and any commits/ancestors leading up to that
point. The clone is fully functional and without any of the limitatons
imposed by shallow clones.
So, in case #1 we simply choose the branch we want - raw unapplied
content, or applied and tagged (and perhaps signed) - but we aren't
subjected to downloading both branches. In case #2 we simply specify
the last known "good" tag before the upstream went off in a direction
that we don't care about.
Note that default behaviour is unchanged - so this is an opt-in feature
which won't impact any existing recipes. We also leave the door open
for it being URL specific via <name> since we know we have multi repo
SRC_URI in recipes-kernel already.
[1] git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git
git://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
bitbake/lib/bb/fetch2/git.py | 3 ++-
documentation/ref-manual/variables.rst | 8 ++++++++
2 files changed, 10 insertions(+), 1 deletion(-)
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index e3ba80a3f52a..22281e2cfb98 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -345,10 +345,11 @@ class Git(FetchMethod):
# If the repo still doesn't exist, fallback to cloning it
if not os.path.exists(ud.clonedir):
+ extcloneargs = d.getVar('GITCLONEARGS_' + ud.names[0]) or d.getVar('GITCLONEARGS') or "--bare --mirror"
# We do this since git will use a "-l" option automatically for local urls where possible
if repourl.startswith("file://"):
repourl = repourl[7:]
- clone_cmd = "LANG=C %s clone --bare --mirror %s %s --progress" % (ud.basecmd, shlex.quote(repourl), ud.clonedir)
+ clone_cmd = "LANG=C %s clone %s %s %s --progress" % (ud.basecmd, extcloneargs, shlex.quote(repourl), ud.clonedir)
if ud.proto.lower() != 'file':
bb.fetch2.check_network_access(d, clone_cmd, ud.url)
progresshandler = GitProgressHandler(d)
diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst
index 0310429bdcab..cd2b77604e87 100644
--- a/documentation/ref-manual/variables.rst
+++ b/documentation/ref-manual/variables.rst
@@ -2649,6 +2649,14 @@ system and gives an overview of their function and contents.
:term:`GDB`
The minimal command and arguments to run the GNU Debugger.
+ :term:`GITCLONEARGS`
+ The arguments used to "git clone" when cloning a repo from an
+ an external server into the local download area. The default is
+ "--bare --mirror". Use of "--bare --single-branch --branch foo"
+ will limit what is cloned/downloaded just to the branch/tag "foo".
+ An URL specific GITCLONEARGS_<name> will take precedence over a
+ recipe wide GITCLONEARGS setting, if both are present.
+
:term:`GITDIR`
The directory in which a local copy of a Git repository is stored
when it is cloned.
--
2.25.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 02/21] bitbake: fetch2/git: allow limiting upstream fetch refs to a subset
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
2021-04-02 17:15 ` [PATCH 01/21] bitbake: fetch2/git: allow override of clone args with GITCLONEARGS Paul Gortmaker
@ 2021-04-02 17:15 ` Paul Gortmaker
2021-04-03 7:43 ` Richard Purdie
2021-04-02 17:15 ` [PATCH 03/21] bitbake: fetch2/git: allow optional git download name overrride Paul Gortmaker
` (19 subsequent siblings)
21 siblings, 1 reply; 29+ messages in thread
From: Paul Gortmaker @ 2021-04-02 17:15 UTC (permalink / raw)
To: Bruce Ashfield, Richard Purdie; +Cc: linux-yocto, bitbake-devel
If a clone in the download directory is not static, and was created with
single-branch to avoid additonal unwanted content, then the fetcher will
come along and spoil that effort by unconditonally getting refs/* from
the server and downloading everything available at the server.
To that end, allow a companion variable that can optionally be used to
limit the fetch to a subset of references that are in line with the
desired content for the recipe in question, and so that large unwanted
downloads aren't inadvertently triggered during an update.
If not specified, the existing "fetch everything" behaviour remains.
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
bitbake/lib/bb/fetch2/git.py | 3 ++-
documentation/ref-manual/variables.rst | 8 ++++++++
2 files changed, 10 insertions(+), 1 deletion(-)
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index 22281e2cfb98..b54ec76d7174 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -362,7 +362,8 @@ class Git(FetchMethod):
runfetchcmd("%s remote rm origin" % ud.basecmd, d, workdir=ud.clonedir)
runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=ud.clonedir)
- fetch_cmd = "LANG=C %s fetch -f --progress %s refs/*:refs/*" % (ud.basecmd, shlex.quote(repourl))
+ fetchrefs = d.getVar('GITFETCHREFS_' + ud.names[0]) or d.getVar('GITFETCHREFS') or "refs/*:refs/*"
+ fetch_cmd = "LANG=C %s fetch -f --progress %s %s" % (ud.basecmd, shlex.quote(repourl), fetchrefs)
if ud.proto.lower() != 'file':
bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
progresshandler = GitProgressHandler(d)
diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst
index cd2b77604e87..55028987c08a 100644
--- a/documentation/ref-manual/variables.rst
+++ b/documentation/ref-manual/variables.rst
@@ -2661,6 +2661,14 @@ system and gives an overview of their function and contents.
The directory in which a local copy of a Git repository is stored
when it is cloned.
+ :term:`GITFETCHREFS`
+ The refs requested during a "git fetch" when updating a repo from
+ an external server into the local download area. The default is
+ "refs/*:refs/*". Used in conjunction with GITCLONEARGS to limit what
+ is cloned/fetched from a repo otherwise dense with branches. Note
+ that an URL specific GITFETCHREFS_<name> will take precedence over
+ a recipe wide GITFETCHREFS setting, if both are present.
+
:term:`GLIBC_GENERATE_LOCALES`
Specifies the list of GLIBC locales to generate should you not wish
to generate all LIBC locals, which can be time consuming.
--
2.25.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* Re: [PATCH 02/21] bitbake: fetch2/git: allow limiting upstream fetch refs to a subset
2021-04-02 17:15 ` [PATCH 02/21] bitbake: fetch2/git: allow limiting upstream fetch refs to a subset Paul Gortmaker
@ 2021-04-03 7:43 ` Richard Purdie
0 siblings, 0 replies; 29+ messages in thread
From: Richard Purdie @ 2021-04-03 7:43 UTC (permalink / raw)
To: Paul Gortmaker, Bruce Ashfield; +Cc: linux-yocto, bitbake-devel
On Fri, 2021-04-02 at 13:15 -0400, Paul Gortmaker wrote:
> If a clone in the download directory is not static, and was created with
> single-branch to avoid additonal unwanted content, then the fetcher will
> come along and spoil that effort by unconditonally getting refs/* from
> the server and downloading everything available at the server.
>
> To that end, allow a companion variable that can optionally be used to
> limit the fetch to a subset of references that are in line with the
> desired content for the recipe in question, and so that large unwanted
> downloads aren't inadvertently triggered during an update.
>
> If not specified, the existing "fetch everything" behaviour remains.
>
> Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
> ---
> bitbake/lib/bb/fetch2/git.py | 3 ++-
> documentation/ref-manual/variables.rst | 8 ++++++++
> 2 files changed, 10 insertions(+), 1 deletion(-)
>
> diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
> index 22281e2cfb98..b54ec76d7174 100644
> --- a/bitbake/lib/bb/fetch2/git.py
> +++ b/bitbake/lib/bb/fetch2/git.py
> @@ -362,7 +362,8 @@ class Git(FetchMethod):
> runfetchcmd("%s remote rm origin" % ud.basecmd, d, workdir=ud.clonedir)
>
>
>
>
> runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=ud.clonedir)
> - fetch_cmd = "LANG=C %s fetch -f --progress %s refs/*:refs/*" % (ud.basecmd, shlex.quote(repourl))
> + fetchrefs = d.getVar('GITFETCHREFS_' + ud.names[0]) or d.getVar('GITFETCHREFS') or "refs/*:refs/*"
> + fetch_cmd = "LANG=C %s fetch -f --progress %s %s" % (ud.basecmd, shlex.quote(repourl), fetchrefs)
> if ud.proto.lower() != 'file':
> bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
> progresshandler = GitProgressHandler(d)
I think one of the reasons I'm unease about this series is that some of the
changes break the existing fetcher functionality. Taking the above, I
understand why you want to filter the references but this will break the
mirroring.
How? Imagine a repo has branches A and B and the recipe filters to A. This
will result in a mirror tarball only with A in it. The mirror tarball name will
attempt to be reused by a recipe referencing with B and the mirroring
mechanism will fail.
Mirroring is the reason we do use all pull the whole repo here. It isn't
good from an efficiency standpoint but it does work well for mirroring. This
piece of code from the fetcher kind of summarises it:
"""
gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace('
','_'))
if gitsrcname.startswith('.'):
gitsrcname = gitsrcname[1:]
# for rebaseable git repo, it is necessary to keep mirror tar ball
# per revision, so that even the revision disappears from the
# upstream repo in the future, the mirror will remain intact and still
# contains the revision
if ud.rebaseable:
for name in ud.names:
gitsrcname = gitsrcname + '_' + ud.revisions[name]
mirrortarball = 'git2_%s.tar.gz' % gitsrcname
"""
To make the above "work", you'd need to rename the mirror tarball to be
branch specific and include A or B in the mirror tarball filename.
I also then wonder whether the form of GITFETCHREFS is good API. It is certainly
powerful however the code already knows which branch we're asking it to fetch
so we could just add a parameter to the url "clonebranchonly" which asks it
to only clone that branch and not all the references?
I appreciate that doesn't give you the way to say "heads/v5.4/*" but instead
you'd get something more specific like "v5.4/standard/base" but that would
filter the fetch to me even more efficient to the particularly use case a
user would generally be working with (a specific machine). I appreciate that
you're not going to like a mirror tarball per kernel branch and even trying
to get the system to generate all the right tarballs would be a nightmare.
Another alternative could be a parameter like ";filterclone=heads/v5.4/" which
the fetcher could then translate to "refs/heads/v5.4/*:refs/heads/v5.4/*"
internally.
I guess in summary this means I have two worries about the API. Ideally we
want something which can be put simply in the URL itself avoiding external
variables. We also want something which is end user readable, rather than
being a direct copy of the git commandline. I'd prefer to avoid direct
commandlines as they only encourage people to start using them to different
things and then we end up breaking mirrors, have incomplete test coverage
and/or find it really hard to add/change functionality later.
Cheers,
Richard
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH 03/21] bitbake: fetch2/git: allow optional git download name overrride
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
2021-04-02 17:15 ` [PATCH 01/21] bitbake: fetch2/git: allow override of clone args with GITCLONEARGS Paul Gortmaker
2021-04-02 17:15 ` [PATCH 02/21] bitbake: fetch2/git: allow limiting upstream fetch refs to a subset Paul Gortmaker
@ 2021-04-02 17:15 ` Paul Gortmaker
2021-04-02 17:15 ` [PATCH 04/21] bitbake: fetch2/git: allow specifying repos as static/unchanging Paul Gortmaker
` (18 subsequent siblings)
21 siblings, 0 replies; 29+ messages in thread
From: Paul Gortmaker @ 2021-04-02 17:15 UTC (permalink / raw)
To: Bruce Ashfield, Richard Purdie; +Cc: linux-yocto, bitbake-devel
Single repositories containing multiple independent origin commits are
becoming more common. For example the rt development repo has patches
as "raw" format-patch output in a patch series, and also for ease of
automated testing, them all pre-applied (ff and non-ff) to the original
baseline (v5.10.8 in this case):
linux-rt-devel$ git merge-base linux-5.10.y-rt linux-5.10.y-rt-rebase
7a1519a74f3d0b06598fb95387688cde41e465d8
linux-rt-devel$ git describe 7a1519a74f3d0b06598fb95387688cde41e465d8
v5.10.8
linux-rt-devel$ git merge-base linux-5.10.y-rt linux-5.10.y-rt-patches
linux-rt-devel$
linux-rt-devel$ git log --oneline linux-5.10.y-rt | tail -n1
1da177e4c3f4 Linux-2.6.12-rc2
linux-rt-devel$ git log --oneline linux-5.10.y-rt-patches | tail -n1
96f1ac50158c [ANNOUNCE] 4.1.2-rt1
As can be seen, there are two different "epoch" commits and no history
or shared objects exist between the patches branch and the applied
branches (on the linux-stable baseline).
Since we construct the download directory name based on the server name
and the server's path to the repository, we currently can't refactor that
content into two locally independent paths/repositories in the download
dir for efficiency, ease of use and tarball/MIRROR handling.
To resolve this, we allow the recipe to optionally specify the path
component of the download dir name, so that a server with a repo of N
independent branches/histories can be used in N different recipes (or N
different SRC_URI lines) without overlap or name space collision in the
download dir. Continuing with the above example, we could have:
RT_REPO = "git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git"
SRC_URI = "${RT_REPO};name=applied;dlname=.preempt-rt.linux-5.10.y \
${RT_REPO};name=patches;dlname=.preempt-rt.patches-5.10"
...resulting in the following independent repository dirs:
downloads/git2/git.kernel.org.preempt-rt.linux-5.10.y
downloads/git2/git.kernel.org.preempt-rt.patches-5.10
If the new "dlname" parameter is not specified, then the automatic name
generation functions just as it always has done and this change has no
effect on any existing download dirs.
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
.../bitbake-user-manual-fetching.rst | 6 ++++++
bitbake/lib/bb/fetch2/git.py | 15 ++++++++++++++-
2 files changed, 20 insertions(+), 1 deletion(-)
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
index e9a5f336dfd2..0f12eaa3a81f 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
@@ -417,6 +417,12 @@ This fetcher supports the following parameters:
is assumed to be "master". The number of branch parameters much match
the number of name parameters.
+- *"dlname":* The download name used in downloads/git2 (or GITDIR). If
+ specified, this will override the automatically constructed name
+ component from "thispath" for the URL git://some.host/thispath to the
+ user specified value. The component from "some.host" remains unaffected.
+ Allows splitting of one repo into multiple independent download dirs.
+
- *"rev":* The revision to use for the checkout. The default is
"master".
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index b54ec76d7174..de698a3d9f24 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -18,6 +18,13 @@ Supported SRC_URI options are:
SRCREV_nameX = "xxxxxxxxxxxxxxxxxxxx"
SRCREV_nameY = "YYYYYYYYYYYYYYYYYYYY"
+- dlname
+ The download name used in downloads/git2 (or GITDIR). This will override
+ the automatically constructed name component from "thispath" for the URL
+ git://some.host/thispath to the user specified value. The component from
+ "some.host" remains unaffected. Allows splitting of one repo into multiple
+ independent download dirs.
+
- tag
The git tag to retrieve. The default is "master"
@@ -151,6 +158,8 @@ class Git(FetchMethod):
ud.nobranch = ud.parm.get("nobranch","0") == "1"
+ ud.dlname = ud.parm.get("dlname","")
+
# usehead implies nobranch
ud.usehead = ud.parm.get("usehead","0") == "1"
if ud.usehead:
@@ -242,7 +251,11 @@ class Git(FetchMethod):
ud.unresolvedrev[name] = ud.revisions[name]
ud.revisions[name] = self.latest_revision(ud, d, name)
- gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_'))
+ if ud.dlname:
+ gitdlname = ud.dlname.replace('/', '.').replace('*', '.').replace(' ','_')
+ else:
+ gitdlname = ud.path.replace('/', '.').replace('*', '.').replace(' ','_')
+ gitsrcname = '%s%s' % (ud.host.replace(':', '.'), gitdlname)
if gitsrcname.startswith('.'):
gitsrcname = gitsrcname[1:]
--
2.25.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 04/21] bitbake: fetch2/git: allow specifying repos as static/unchanging
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
` (2 preceding siblings ...)
2021-04-02 17:15 ` [PATCH 03/21] bitbake: fetch2/git: allow optional git download name overrride Paul Gortmaker
@ 2021-04-02 17:15 ` Paul Gortmaker
2021-04-02 17:15 ` [PATCH 05/21] bitbake: fetch2/git: ensure static repos have at least one refs/heads Paul Gortmaker
` (17 subsequent siblings)
21 siblings, 0 replies; 29+ messages in thread
From: Paul Gortmaker @ 2021-04-02 17:15 UTC (permalink / raw)
To: Bruce Ashfield, Richard Purdie; +Cc: linux-yocto, bitbake-devel
If we know a repository content has what we need, we can flag it as
static and then optimize accordingly for the future, by not trying to do
any fetch operations against it. In a way, this can be thought of as
the special case of GITFETCHREFS = None.
There are multiple advantages to having this. We can skip trying any
updates and speed things up for a project we know is EOL. We can avoid
doing "ls-remote" operations against servers, easing use for
mirror/no-network users. We can open the door to accepting tags in
SRC_URI more often as absolute references, instead of the less humanly
readable SHA. Archives that change upstream in incompatible ways, or
have simply vanished, can exist as a snapshot in time after initial
creation via mirror tarball or manually.
Finally, we use the flexibility of git-config to store that static
information in the cloned repo itself, so we can tell at any time,
independent of the SRC_URI, that the repo has static content.
This will be used for reference purposes, but might also be used in the
download dir itself outside of BB recipes -- for example a script could
easily spot any static content and hence prioritize it as a possible
mirroring candidate.
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
.../bitbake-user-manual-fetching.rst | 5 ++++
bitbake/lib/bb/fetch2/git.py | 27 +++++++++++++++++++
2 files changed, 32 insertions(+)
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
index 0f12eaa3a81f..94f4cf3363f6 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
@@ -426,6 +426,11 @@ This fetcher supports the following parameters:
- *"rev":* The revision to use for the checkout. The default is
"master".
+- *"static":* The repository is a "one and done" - clone/untar and the
+ content is static and unchanging. This allows optimizations like
+ skipping fetch/pull and doing "ls-remote" on servers. Typically used
+ for repositories that will act as a commit library for other repos.
+
- *"tag":* Specifies a tag to use for the checkout. To correctly
resolve tags, BitBake must access the network. For that reason, tags
are often not used. As far as Git is concerned, the "tag" parameter
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index de698a3d9f24..7cba990c3be7 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -54,6 +54,11 @@ Supported SRC_URI options are:
referring to commit which is valid in tag instead of branch.
The default is "0", set nobranch=1 if needed.
+- static
+ Repository is of fixed/static content with no active/new development.
+ Once cloned, nothing will change, so optimize accordingly. The default
+ is "0", set static=1 when appropriate.
+
- usehead
For local git:// urls to use the current branch HEAD as the revision for use with
AUTOREV. Implies nobranch.
@@ -158,6 +163,8 @@ class Git(FetchMethod):
ud.nobranch = ud.parm.get("nobranch","0") == "1"
+ ud.static = ud.parm.get("static","0") == "1"
+
ud.dlname = ud.parm.get("dlname","")
# usehead implies nobranch
@@ -167,6 +174,10 @@ class Git(FetchMethod):
raise bb.fetch2.ParameterError("The usehead option is only for use with local ('protocol=file') git repositories", ud.url)
ud.nobranch = 1
+ if ud.static:
+ if ud.rebaseable:
+ raise bb.fetch2.ParameterError("The rebaseable option is incompatible with the static option", ud.url)
+
# bareclone implies nocheckout
ud.bareclone = ud.parm.get("bareclone","0") == "1"
if ud.bareclone:
@@ -311,6 +322,8 @@ class Git(FetchMethod):
def clonedir_need_update(self, ud, d):
if not os.path.exists(ud.clonedir):
return True
+ if ud.static:
+ return False
if ud.shallow and ud.write_shallow_tarballs and self.clonedir_need_shallow_revs(ud, d):
return True
for name in ud.names:
@@ -341,6 +354,17 @@ class Git(FetchMethod):
return False
return True
+ def get_git_config(self, ud, d, repo, cfgvar):
+ try:
+ output = runfetchcmd("%s config --get %s" % (ud.basecmd, cfgvar), d, workdir=repo)
+ except (bb.fetch2.FetchError,ValueError):
+ return ""
+ return output.split()[0]
+
+ def repo_is_static(self, ud, d, repo):
+ static = self.get_git_config(ud, d, repo, "bitbake.static")
+ return (static == "true")
+
def download(self, ud, d):
"""Fetch url"""
@@ -368,6 +392,9 @@ class Git(FetchMethod):
progresshandler = GitProgressHandler(d)
runfetchcmd(clone_cmd, d, log=progresshandler)
+ if ud.static:
+ runfetchcmd("%s config --bool --add bitbake.static 1" % ud.basecmd, d, workdir=ud.clonedir)
+
# Update the checkout if needed
if self.clonedir_need_update(ud, d):
output = runfetchcmd("%s remote" % ud.basecmd, d, quiet=True, workdir=ud.clonedir)
--
2.25.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 05/21] bitbake: fetch2/git: ensure static repos have at least one refs/heads
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
` (3 preceding siblings ...)
2021-04-02 17:15 ` [PATCH 04/21] bitbake: fetch2/git: allow specifying repos as static/unchanging Paul Gortmaker
@ 2021-04-02 17:15 ` Paul Gortmaker
2021-04-02 17:15 ` [PATCH 06/21] bitbake: fetch2/git: allow alt references within download dir Paul Gortmaker
` (16 subsequent siblings)
21 siblings, 0 replies; 29+ messages in thread
From: Paul Gortmaker @ 2021-04-02 17:15 UTC (permalink / raw)
To: Bruce Ashfield, Richard Purdie; +Cc: linux-yocto, bitbake-devel
When you clone with "--single-branch --branch foo" one of two things
will happen:
If foo is a branch upstream you will get a local branch foo and HEAD
will contain "ref: refs/heads/foo". Output from "git branch" will be:
* foo
If foo is a tag upstream, you won't get any local branch, and HEAD
will contain the SHA of the tag foo. Output from "git branch" will be:
* (no branch)
In the latter case, people tend to assume it is some kind of indication
progress was interrupted and an error happened.
While not strictly necessary, in static repos, we point master at the
tag that was fetched if no branches are present, to avoid confusion.
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
bitbake/lib/bb/fetch2/git.py | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index 7cba990c3be7..6cd5d09b2ded 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -417,6 +417,15 @@ class Git(FetchMethod):
if exc.errno != errno.ENOENT:
raise
+ # fixup "* (no branch)" from "--single-branch --branch <tag>"
+ if ud.static:
+ output = runfetchcmd("%s show-ref --heads | wc -l" % ud.basecmd, d, workdir=ud.clonedir)
+ headless = output.split()[0] == "0"
+ if headless:
+ runfetchcmd("%s update-ref refs/heads/master HEAD" % ud.basecmd, d, workdir=ud.clonedir)
+ with open(os.path.join(ud.clonedir, "HEAD"), "w") as f:
+ f.write('ref: refs/heads/master\n')
+
for name in ud.names:
if not self._contains_ref(ud, d, name, ud.clonedir):
raise bb.fetch2.FetchError("Unable to find revision %s in branch %s even from upstream" % (ud.revisions[name], ud.branches[name]))
--
2.25.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 06/21] bitbake: fetch2/git: allow alt references within download dir
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
` (4 preceding siblings ...)
2021-04-02 17:15 ` [PATCH 05/21] bitbake: fetch2/git: ensure static repos have at least one refs/heads Paul Gortmaker
@ 2021-04-02 17:15 ` Paul Gortmaker
2021-04-02 17:15 ` [PATCH 07/21] bitbake: fetch2/git: append new altref line if/when SRC_URI changed value Paul Gortmaker
` (15 subsequent siblings)
21 siblings, 0 replies; 29+ messages in thread
From: Paul Gortmaker @ 2021-04-02 17:15 UTC (permalink / raw)
To: Bruce Ashfield, Richard Purdie; +Cc: linux-yocto, bitbake-devel
It is no secret that we've needed to formalize some kind of sensible
sharing between git repos with common ancestry. The opportunity for
sharing the mainline kernel base between linux-yocto and linux-yocto-dev
is the main obvious candidate.
Many of us have long been doing local hacks to pre-populate the
downloads dir and avoid lengthy/large downloads, and some have even
been kind enough to try and get a variant of that upstream[1].
But the problem is that it isn't as simple as adding a new "--reference"
switch to the fetcher and then walking away. That will result in
non-portable and incompatible use cases with existing work flows. We
have to put some thought into how we see this fitting into the larger
framework of things and in what ways we want to support use of it.
For example, we know we want relative paths in anything in the download
dir, but "clone --reference" will put an absolute path in the alternates
file by default. We also don't want the SRC_URI to be containing any
absolute paths either. We also want (pre)mirror stuff to keep working.
So we set some basic ground rules. Firstly we don't even expose the
obvious "--reference" argument that some of us are used to at all. That
is because we are going to have two types of references - the common one
based on objets/info/alternates (this commit) and a pack based one (the
subsequent commit).
We also restrict any references to be peers of the other content in
downloads - no /home/fred/super-SDK/kernel-4.19-rt/ type stuff. The
idea is that once you factor out the common building blocks of source,
then your kernel (staying with that example) becomes a very small repo.
For example - all of the linux-yocto BSPs (standard and -rt) can be held
in a repo that is only 20-30 megabytes - small enough to attach to email.
Clones via alternates have some limitations - for example there is an
arbitrary limit of six[2] levels of nesting, so a pack reference is
introduced to resolve this subsequently. However alternaltes make sense
for shared non-static (i.e. WIP repos) endpoints, like stable and -rt.
[1] https://lists.openembedded.org/g/bitbake-devel/topic/77645766
[2] https://github.com/git/git/commit/c2f493a4 # if (depth > 5)
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
.../bitbake-user-manual-fetching.rst | 6 +++++
bitbake/lib/bb/fetch2/git.py | 24 +++++++++++++++++++
2 files changed, 30 insertions(+)
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
index 94f4cf3363f6..12a68accc2c5 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
@@ -447,6 +447,12 @@ This fetcher supports the following parameters:
parameter implies no branch and only works when the transfer protocol
is ``file://``.
+- *"altref":* Enables cloned ``git://`` URLs to use the named peer from
+ the downloads dir as a reference through the standard git alternates
+ file for clone (see ``-reference``) and subseqent operations. Recipe
+ is responsible (via fetch dependency) for ensuring the reference is
+ populated/cloned prior to being called on to act as a reference.
+
Here are some example URLs: ::
SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1"
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index 6cd5d09b2ded..e742010229c3 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -59,6 +59,12 @@ Supported SRC_URI options are:
Once cloned, nothing will change, so optimize accordingly. The default
is "0", set static=1 when appropriate.
+- altref
+ Repository is to use named peer from downloads dir as a reference through
+ the standard git objects/info/alternates file for clone and subseqent
+ operations. Recipe is responsible (via fetch dependency) for ensuring the
+ reference is populated/cloned prior to being called on to act as a reference.
+
- usehead
For local git:// urls to use the current branch HEAD as the revision for use with
AUTOREV. Implies nobranch.
@@ -165,6 +171,8 @@ class Git(FetchMethod):
ud.static = ud.parm.get("static","0") == "1"
+ ud.altref = ud.parm.get("altref","")
+
ud.dlname = ud.parm.get("dlname","")
# usehead implies nobranch
@@ -380,9 +388,20 @@ class Git(FetchMethod):
repourl = self._get_repo_url(ud)
+ refname = ud.altref
+ if refname:
+ alts = os.path.join(ud.clonedir, 'objects', 'info', 'alternates')
+ gitdir = d.getVar("GITDIR") or (d.getVar("DL_DIR") + "/git2")
+ refdir = os.path.join(gitdir, refname)
+ if not os.path.exists(refdir):
+ raise bb.fetch2.FetchError("Unable to find reference %s in %s" % (refname, gitdir))
+
# If the repo still doesn't exist, fallback to cloning it
if not os.path.exists(ud.clonedir):
extcloneargs = d.getVar('GITCLONEARGS_' + ud.names[0]) or d.getVar('GITCLONEARGS') or "--bare --mirror"
+ if refname:
+ extcloneargs += " --reference %s" % refdir
+
# We do this since git will use a "-l" option automatically for local urls where possible
if repourl.startswith("file://"):
repourl = repourl[7:]
@@ -392,6 +411,11 @@ class Git(FetchMethod):
progresshandler = GitProgressHandler(d)
runfetchcmd(clone_cmd, d, log=progresshandler)
+ # Fix up reference to be relative for tarball/mirror portability
+ if ud.altref:
+ with open(alts, "w") as f:
+ f.write("../../%s/objects\n" % ud.altref)
+
if ud.static:
runfetchcmd("%s config --bool --add bitbake.static 1" % ud.basecmd, d, workdir=ud.clonedir)
--
2.25.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 07/21] bitbake: fetch2/git: append new altref line if/when SRC_URI changed value
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
` (5 preceding siblings ...)
2021-04-02 17:15 ` [PATCH 06/21] bitbake: fetch2/git: allow alt references within download dir Paul Gortmaker
@ 2021-04-02 17:15 ` Paul Gortmaker
2021-04-02 17:15 ` [PATCH 08/21] bitbake: fetch2/git: allow pack references within download dir Paul Gortmaker
` (14 subsequent siblings)
21 siblings, 0 replies; 29+ messages in thread
From: Paul Gortmaker @ 2021-04-02 17:15 UTC (permalink / raw)
To: Bruce Ashfield, Richard Purdie; +Cc: linux-yocto, bitbake-devel
A new repository that is cloned with "--reference" in turn creates an
objects/info/alternates file in the new repo containing the path back to
the reference repo objects. This is normally a "one-and-done" type
operation and the reference never changes.
However we can envision where a package leaves behind a dead end
"stable" fork and re-aligns itself with the main branch, or a newer
stable fork. In the case of kernel, a real world example would be
leaving behind v5.4.x in favour of v5.10.x for a baseline.
This commit is a consideration for those users - with an existing copy
in their downloads dir that is using a reference that has since been
updated in the SRC_URI of the package recipe.
If we removed the old reference and replaced it, we would break any old
build directories that had branches based on v5.4.x specific commits, so
that is clearly not an option.
If we simply did nothing, things would still work just fine for them;
any v5.10.x objects not found in the v5.4.x reference would be simply
obtained automatically from the server during a fetch update, and placed
in the endpoint repo that is using the older v5.4.x as a reference.
However, since alternates supports a multi-line format, we can simply
append the new reference. This leaves any v5.4.x build "consumers"
functioning as-is, and also gives a path back to v5.10.x objects in the
new reference, so they need not be duplicated over into the endpoint repo.
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
bitbake/lib/bb/fetch2/git.py | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index e742010229c3..3b0c068bc020 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -421,6 +421,15 @@ class Git(FetchMethod):
# Update the checkout if needed
if self.clonedir_need_update(ud, d):
+ if ud.altref:
+ # if altref changed in SRC_URI then append new value.
+ alts = os.path.join(ud.clonedir, 'objects', 'info', 'alternates')
+ try:
+ runfetchcmd("grep -q %s %s" % (ud.altref, alts), d, quiet=True, workdir=ud.clonedir)
+ except (bb.fetch2.FetchError,ValueError):
+ with open(alts, "a") as f:
+ f.write("../../%s/objects\n" % ud.altref)
+
output = runfetchcmd("%s remote" % ud.basecmd, d, quiet=True, workdir=ud.clonedir)
if "origin" in output:
runfetchcmd("%s remote rm origin" % ud.basecmd, d, workdir=ud.clonedir)
--
2.25.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 08/21] bitbake: fetch2/git: allow pack references within download dir
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
` (6 preceding siblings ...)
2021-04-02 17:15 ` [PATCH 07/21] bitbake: fetch2/git: append new altref line if/when SRC_URI changed value Paul Gortmaker
@ 2021-04-02 17:15 ` Paul Gortmaker
2021-04-02 17:15 ` [PATCH 09/21] bitbake: fetch2/git: use constant names for packs in static repos Paul Gortmaker
` (13 subsequent siblings)
21 siblings, 0 replies; 29+ messages in thread
From: Paul Gortmaker @ 2021-04-02 17:15 UTC (permalink / raw)
To: Bruce Ashfield, Richard Purdie; +Cc: linux-yocto, bitbake-devel
Borrowing from "git pack-objects" manpage[1] we have:
A packed archive is an efficient way to transfer a set of objects
between two repositories as well as an access efficient archival
format. In a packed archive, an object is either stored as a
compressed whole or as a difference from some other object. The
latter is often called a delta.
The packed archive format (.pack) is designed to be self-contained
so that it can be unpacked without any further information. Therefore,
each object that a delta depends upon must be present within the pack.
A pack index file (.idx) is generated for fast, random access to
the objects in the pack. Placing both the index file (.idx) and the
packed archive (.pack) in the pack/ subdirectory [...] enables Git
to read from the pack archive.
As stated, they are self-contained and can encapsulate an entire chunk
of development history starting at point A and ending at point B, which
makes them ideal not just for transferring, but also for sharing that
chunk of history between repositories with common ancestry.
When you ask a server for something, it sends a single pack, because
that is most efficient. So, using the kernel as an example, lets say we
decide we want to use the 2005 git "epoch" as "A" and the v5.10 tag of a
recent release as "B" to delineate an initial block of history. We can
grab that chunk of history with a simple:
git clone --bare --single-branch --branch v5.10 git://someserver/linux my-510-lib
and we now have all the history up to and including v5.10 tag in a repo
that we can use with further "clone --reference my-510-lib newstuff" as
a static unchanging library of all those objects for "newstuff" to use.
In making such a clone, Git stores the my-510-lib dir in the
objects/info/alternates of "newstuff" and all the refs and tags carry
forard into [packed]refs of "newstuff". Subsequent uses of the
"newstuff" repo are only using the objects from "my-510-lib" and they
are conveniently all stored in one pack, as there are no loose objects.
In fact it is a waste of time even having git operations look for them.
This means we can simply remove the alternates file and link the pack
(and accompanying index file) into the pack dir of "newstuff". And by
using symlinks with relative paths, we get a mirror friendly reference
with no data download duplication. The "clone --reference" is simply
used as a convenience tool to do 99% of the setup work. The linked in
pack is "just a normal pack" to the consumer of the commit library.
As such, we introduce a "packref" as a useful parallel addition to the
more well known "altref". It is ideally suited to reference from static
repositories containing a known block of commit history. Note that we
don't limit ourselves to a single pack, so that a packref can itself be
using another repo as its own packref and so on. Hence there is no
arbitrary limit of a depth of 5 as there is with a reference using the
more common alternates (i.e. "altref") method.
This opens the door to insert an arbitrary number of split points between
A and B to fragment large repos, if desired. An example that shows how
the bulk of kernel history can be split in two, three or four chunks
will be provided for recipes-kernel/linux subsequently.
[1] https://git-scm.com/docs/git-pack-objects
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
.../bitbake-user-manual-fetching.rst | 7 +++++
bitbake/lib/bb/fetch2/git.py | 31 ++++++++++++++++++-
2 files changed, 37 insertions(+), 1 deletion(-)
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
index 12a68accc2c5..db1d108f4736 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
@@ -453,6 +453,13 @@ This fetcher supports the following parameters:
is responsible (via fetch dependency) for ensuring the reference is
populated/cloned prior to being called on to act as a reference.
+- *"packref":* Enables cloned ``git://`` URLs to use packs from the named
+ peer from the downloads dir as if they were their own packs of commit
+ history. Similar to ``altref`` above, but suited for bare static repos
+ for which no loose objects outside of packs will ever be present.
+ Recipe is responsible (via fetch dependency) for ensuring the reference
+ is populated/cloned prior to being called on to act as a reference.
+
Here are some example URLs: ::
SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1"
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index 3b0c068bc020..8aec12df2bf8 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -65,6 +65,14 @@ Supported SRC_URI options are:
operations. Recipe is responsible (via fetch dependency) for ensuring the
reference is populated/cloned prior to being called on to act as a reference.
+- packref
+ Repository is to use packs from named peer from downloads dir by linking
+ them into objects/pack/ of the new repository as if they were their own
+ packs of commit history. Similar to altref above, but suited for bare
+ static repos for which no loose objects outside of packs will be present.
+ Recipe is responsible (via fetch dependency) for ensuring the reference
+ is populated/cloned prior to being called on to act as a reference.
+
- usehead
For local git:// urls to use the current branch HEAD as the revision for use with
AUTOREV. Implies nobranch.
@@ -171,10 +179,15 @@ class Git(FetchMethod):
ud.static = ud.parm.get("static","0") == "1"
+ ud.packref = ud.parm.get("packref","")
+
ud.altref = ud.parm.get("altref","")
ud.dlname = ud.parm.get("dlname","")
+ if ud.packref and ud.altref:
+ raise bb.fetch2.ParameterError("Please pick only ONE reference style: alternates OR packs", ud.url)
+
# usehead implies nobranch
ud.usehead = ud.parm.get("usehead","0") == "1"
if ud.usehead:
@@ -373,6 +386,14 @@ class Git(FetchMethod):
static = self.get_git_config(ud, d, repo, "bitbake.static")
return (static == "true")
+ def create_pack_links(self, refname, refdir, dstdir):
+ dstpkdir = os.path.join(dstdir, 'objects', 'pack')
+ refpkdir = os.path.join(refdir, 'objects', 'pack')
+ for item in os.listdir(refpkdir):
+ dst = os.path.join(dstpkdir, item)
+ src = os.path.join('..', '..', '..', refname, 'objects', 'pack', item)
+ os.symlink(src, dst)
+
def download(self, ud, d):
"""Fetch url"""
@@ -388,7 +409,9 @@ class Git(FetchMethod):
repourl = self._get_repo_url(ud)
- refname = ud.altref
+ # Both style of reference are the same for initial clone operation.
+ # It is the post-clone absolute to relative path fixup that differs.
+ refname = ud.altref or ud.packref
if refname:
alts = os.path.join(ud.clonedir, 'objects', 'info', 'alternates')
gitdir = d.getVar("GITDIR") or (d.getVar("DL_DIR") + "/git2")
@@ -419,6 +442,12 @@ class Git(FetchMethod):
if ud.static:
runfetchcmd("%s config --bool --add bitbake.static 1" % ud.basecmd, d, workdir=ud.clonedir)
+ if ud.packref:
+ if not self.repo_is_static(ud, d, refdir):
+ raise bb.fetch2.FetchError("Pack reference '%s' is not marked as static content." % refname)
+ self.create_pack_links(refname, refdir, ud.clonedir)
+ os.remove(alts)
+
# Update the checkout if needed
if self.clonedir_need_update(ud, d):
if ud.altref:
--
2.25.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 09/21] bitbake: fetch2/git: use constant names for packs in static repos
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
` (7 preceding siblings ...)
2021-04-02 17:15 ` [PATCH 08/21] bitbake: fetch2/git: allow pack references within download dir Paul Gortmaker
@ 2021-04-02 17:15 ` Paul Gortmaker
2021-04-02 17:15 ` [PATCH 10/21] kernel: add basic boilerplate for fetch-only recipes Paul Gortmaker
` (12 subsequent siblings)
21 siblings, 0 replies; 29+ messages in thread
From: Paul Gortmaker @ 2021-04-02 17:15 UTC (permalink / raw)
To: Bruce Ashfield, Richard Purdie; +Cc: linux-yocto, bitbake-devel
The "packref" support uses symlinks to "piggy-back" use of packs into
the pack namespace of the consumer from the originator (reference repo).
We use symlinks, so that mirror tarballs won't duplicate the referenced
data, as hardlinks would do. However, that means we also assume that
the pack namespace of the originator remains constant.
What if one uses a reference from a mirror tarball, then disables
mirrors, does a "cleanall" on the reference for some obscure reason, and
then re-clones it from a server directly. Will the pack links from the
consumer into the reference still point to a pack or be left dangling?
For simplicity, we temporarily ignored this assumption so that we could
deal with it (and this corner case related to it) separately - which we
do now. To proceed, we first need to know a bit about pack names.
Most importantly, as you can see, pack names don't really matter at all:
paul@hackbox:~$ git clone --bare git://git.yoctoproject.org/poky
Cloning into bare repository 'poky.git'...
[...]
Resolving deltas: 100% (381097/381097), done.
paul@hackbox:~$ cd poky.git/objects/pack/
paul@hackbox:~/poky.git/objects/pack$ ls -l
total 191100
-r--r--r-- 1 paul paul 14250608 Mar 28 12:09 pack-db459c1574c5410a1002d8098745daee3a59599f.idx
-r--r--r-- 1 paul paul 181429684 Mar 28 12:09 pack-db459c1574c5410a1002d8098745daee3a59599f.pack
paul@hackbox:~/poky.git/objects/pack$ mv pack-db459c1574c5410a1002d8098745daee3a59599f.idx woot-fred-was-here-31337.idx
paul@hackbox:~/poky.git/objects/pack$ mv pack-db459c1574c5410a1002d8098745daee3a59599f.pack woot-fred-was-here-31337.pack
paul@hackbox:~/poky.git/objects/pack$ cd ../../
paul@hackbox:~/poky.git$ git fsck
Checking object directories: 100% (256/256), done.
Checking objects: 100% (508912/508912), done.
Checking connectivity: 508912, done.
paul@hackbox:~/poky.git$ git show HEAD | head -n3
commit a7e1bbaf6d7c5d1cf44069419860dabd78c02eec
Author: Khem Raj <raj.khem@gmail.com>
Date: Thu Mar 25 18:11:07 2021 -0700
paul@hackbox:~/poky.git$ echo $?
0
This is because normal flow will have a clone/fetch feed stdin of
index-pack, and from the "--stdin" part of its manpage, we have:
If <pack-file> is not specified, the pack is written to objects/pack/
directory of the current Git repository with a default name determined
from the pack content.
Two things to note: 1) we have always been free to choose the pack name
even if the average user probably always takes the default, and 2) the
man page is intentionally kind of vague on how that default is created.
We need to look at #2 in detail, in order to answer our question about
whether the pack names remain constant for the same object content.
The executive summary is "yes" for git v1.7 and older and "no" for git
v1.8+ So we need to look at both in order to decide what to do.
For the "old" git, the default pack name was simply "pack-" + the SHA1
of the sorted SHA1 object names themselves. If you have an old pack
around you can confirm as I have done below:
$ ls -l
total 792
-r--r--r-- 1 paul paul 1632 Feb 13 2013 pack-a9f793e9f46c138a27abb326ac68cb6e6397d0f0.idx
-r--r--r-- 1 paul paul 803171 Feb 13 2013 pack-a9f793e9f46c138a27abb326ac68cb6e6397d0f0.pack
$ git show-index < pack-a9f793e9f46c138a27abb326ac68cb6e6397d0f0.idx | cut -d " " -f 2 | xxd -r -p | sha1sum
a9f793e9f46c138a27abb326ac68cb6e6397d0f0 -
$ git verify-pack -v pack-a9f793e9f46c138a27abb326ac68cb6e6397d0f0.pack|grep '^[0-9a-f]\{40\}' | cut -d " " -f1 | sort | xxd -r -p | sha1sum
a9f793e9f46c138a27abb326ac68cb6e6397d0f0 -
In both cases, we recover the pack name directly from the objects. There
is no extraneous information, like date, servername, etc. used in the
pack name. And so it is invariant upon reclone/repack, even if the
compression level was changed - since the final result from "git
unpack-objects" is the same.
For newer git, which contain 1190a1ac[1] the choice was made to use the
SHA1 of the data as stored in the pack itself - the "trailer" SHA1.
If we look at our since-renamed pack friend Fred above:
paul@hackbox:~/poky.git/objects/pack$ hexdump -C woot-fred-was-here-31337.pack | tail -n3
0ad065a0 db 45 9c 15 74 c5 41 0a 10 02 d8 09 87 45 da ee
0ad065b0 3a 59 59 9f
...we can see we really didn't "lose" his original pack name, as the
trailer SHA1 db459c1574c5410a1002d8098745daee3a59599f is right there.
But this means that the pack name is based on transient data like
compression artifacts, and that no two clones will ever likely generate
the same pack name. If this was a letter in an envelope, the SHA1 went
from being the contents of the letter, to also include how many times
the paper was folded, and the size and colour of the envelope.
This change was made to remove any possible confusion that pack name
equivalence meant binary byte equivalence across two packs, by virtually
ensuring we never got the same pack name twice.
However, in our case, a pack name that is essentially a random number is
unhelpful, and the namespace collision concerns of commit 1190a1ac are
even less of a concern for static content repositories.
So our static repos use pre v1.8 pack names, which reflect functional
equivalence between packs from clone to clone to mirror tarball snapshot
of a clone, because as demonstrated earlier, we are free to choose the
pack name(s) completely at will, so long as we keep it paired with its
index file.
This turns out to be much more robust than trying to preserve the
now random "new" pack names through self-updating symlinks based on
dependencies or anon python or anything similarly complex.
As a final optimization/convenience, we can simply drop the "xxd" from
the pipeline - and avoid introducing new native dependencies, or adding
it to the ASSUME_PROVIDED command list and hoping it is everywhere.
We can do this because the ASCII representation of the .idx hex values
contains the exact same pack object content information as the xxd
binary-converted version, but just in a less dense format. So the SHA1
of that less packed info stream still gives us a unique signature of the
objects contained in the pack - just not the exact one the older git did.
[1] https://github.com/git/git/commit/1190a1ac
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
bitbake/lib/bb/fetch2/git.py | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index 8aec12df2bf8..2d693083cac4 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -386,6 +386,21 @@ class Git(FetchMethod):
static = self.get_git_config(ud, d, repo, "bitbake.static")
return (static == "true")
+ def rename_packs(self, ud, d, repo):
+ # Use git pack naming similar to pre v1.8 - where the name only
+ # depends on the objects within, not objs + compression artifacts.
+ pkdir = os.path.join(repo, 'objects', 'pack')
+
+ for idx in fnmatch.filter(os.listdir(pkdir), "*.idx"):
+ pk = os.path.join(pkdir, idx[:-3] + "pack")
+ idx = os.path.join(pkdir, idx)
+ # insert "xxd -p -r" before the sha1sum to get exactly v1.7 names.
+ cmd = "%s show-index < %s | cut -d' ' -f2 | sort | sha1sum" % (ud.basecmd, idx)
+ output = runfetchcmd(cmd, d, workdir=repo)
+ newname = os.path.join(pkdir, "pack-" + output[:40])
+ os.rename(pk, newname + ".pack")
+ os.rename(idx, newname + ".idx")
+
def create_pack_links(self, refname, refdir, dstdir):
dstpkdir = os.path.join(dstdir, 'objects', 'pack')
refpkdir = os.path.join(refdir, 'objects', 'pack')
@@ -441,6 +456,7 @@ class Git(FetchMethod):
if ud.static:
runfetchcmd("%s config --bool --add bitbake.static 1" % ud.basecmd, d, workdir=ud.clonedir)
+ self.rename_packs(ud, d, ud.clonedir)
if ud.packref:
if not self.repo_is_static(ud, d, refdir):
--
2.25.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 10/21] kernel: add basic boilerplate for fetch-only recipes
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
` (8 preceding siblings ...)
2021-04-02 17:15 ` [PATCH 09/21] bitbake: fetch2/git: use constant names for packs in static repos Paul Gortmaker
@ 2021-04-02 17:15 ` Paul Gortmaker
2021-04-02 17:15 ` [PATCH 11/21] kernel: add a fetch-only recipe for mainline v5.10 source Paul Gortmaker
` (11 subsequent siblings)
21 siblings, 0 replies; 29+ messages in thread
From: Paul Gortmaker @ 2021-04-02 17:15 UTC (permalink / raw)
To: Bruce Ashfield, Richard Purdie; +Cc: linux-yocto, bitbake-devel
With changes to the fetcher, we now have what we need to download
kernel sources in a way that is forward thinking for sharing and
growth and EOL of components. This will be achieved by "fetch-only"
recipes that only exist to populate chunks of this "library" of source.
We may wish to reconsider locating 99% of this elsewhere, if use cases
outside the kernel eventually arise.
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
meta/recipes-kernel/linux/fetch-only.inc | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
create mode 100644 meta/recipes-kernel/linux/fetch-only.inc
diff --git a/meta/recipes-kernel/linux/fetch-only.inc b/meta/recipes-kernel/linux/fetch-only.inc
new file mode 100644
index 000000000000..3333eef50713
--- /dev/null
+++ b/meta/recipes-kernel/linux/fetch-only.inc
@@ -0,0 +1,20 @@
+# There's nothing to do here, except fetch
+
+do_unpack[noexec] = "1"
+do_patch[noexec] = "1"
+do_configure[noexec] = "1"
+do_compile[noexec] = "1"
+do_install[noexec] = "1"
+
+inherit nopackages allarch meta
+
+deltask do_deploy_source_date_epoch
+deltask do_prepare_recipe_sysroot
+deltask do_populate_sysroot
+deltask do_populate_lic
+deltask do_rm_work
+
+EXCLUDE_FROM_WORLD = "1"
+KORG_SERVER ?= "git.kernel.org"
+KORG_PATH ?= "/pub/scm/linux/kernel/git"
+KORG_URLBASE ?= "git://${KORG_SERVER}${KORG_PATH}"
--
2.25.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 11/21] kernel: add a fetch-only recipe for mainline v5.10 source
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
` (9 preceding siblings ...)
2021-04-02 17:15 ` [PATCH 10/21] kernel: add basic boilerplate for fetch-only recipes Paul Gortmaker
@ 2021-04-02 17:15 ` Paul Gortmaker
2021-04-02 20:13 ` Bruce Ashfield
2021-04-02 17:15 ` [PATCH 12/21] kernel: allow splitting mainline v5.10 source download in two Paul Gortmaker
` (10 subsequent siblings)
21 siblings, 1 reply; 29+ messages in thread
From: Paul Gortmaker @ 2021-04-02 17:15 UTC (permalink / raw)
To: Bruce Ashfield, Richard Purdie; +Cc: linux-yocto, bitbake-devel
With v5.10 being the newest baseline currently in use by linux-yocto and
with the download size being 1/2 the size of current linux-yocto itself,
the v5.10 makes a good initial line in the sand for blocking out the
source in a way to optimize sharing.
With this commit present, one can test via "bitbake -c fetch linux-5.10".
Pending recipes who will want this source present will add:
do_fetch[depends] += "linux-5.10:do_fetch"
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
meta/recipes-kernel/linux/fetch-linux.inc | 7 +++++++
meta/recipes-kernel/linux/linux-5.10.bb | 9 +++++++++
2 files changed, 16 insertions(+)
create mode 100644 meta/recipes-kernel/linux/fetch-linux.inc
create mode 100644 meta/recipes-kernel/linux/linux-5.10.bb
diff --git a/meta/recipes-kernel/linux/fetch-linux.inc b/meta/recipes-kernel/linux/fetch-linux.inc
new file mode 100644
index 000000000000..e3fdd0bb36c2
--- /dev/null
+++ b/meta/recipes-kernel/linux/fetch-linux.inc
@@ -0,0 +1,7 @@
+# Boilerplate to fetch up to a single tag of mainline
+
+require recipes-kernel/linux/fetch-only.inc
+
+GITCLONEARGS = "--bare --single-branch --branch v${KTAG}"
+DL_NAME = ".torvalds.linux-${KTAG}"
+SRC_URI = "${KORG_URLBASE}/torvalds/linux;tag=v${KTAG};nobranch=1;static=1;dlname=${DL_NAME}"
diff --git a/meta/recipes-kernel/linux/linux-5.10.bb b/meta/recipes-kernel/linux/linux-5.10.bb
new file mode 100644
index 000000000000..7f8a27bcb1f0
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-5.10.bb
@@ -0,0 +1,9 @@
+SUMMARY = "Linux kernel v5.10 source"
+DESCRIPTION = "Fetch source of the linux kernel up to the v5.10 tag"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
+
+KTAG = "5.10"
+
+require recipes-kernel/linux/fetch-linux.inc
--
2.25.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* Re: [PATCH 11/21] kernel: add a fetch-only recipe for mainline v5.10 source
2021-04-02 17:15 ` [PATCH 11/21] kernel: add a fetch-only recipe for mainline v5.10 source Paul Gortmaker
@ 2021-04-02 20:13 ` Bruce Ashfield
0 siblings, 0 replies; 29+ messages in thread
From: Bruce Ashfield @ 2021-04-02 20:13 UTC (permalink / raw)
To: Paul Gortmaker; +Cc: Richard Purdie, Linux Yocto, bitbake-devel
On Fri, Apr 2, 2021 at 1:16 PM Paul Gortmaker
<paul.gortmaker@windriver.com> wrote:
>
> With v5.10 being the newest baseline currently in use by linux-yocto and
> with the download size being 1/2 the size of current linux-yocto itself,
> the v5.10 makes a good initial line in the sand for blocking out the
> source in a way to optimize sharing.
>
> With this commit present, one can test via "bitbake -c fetch linux-5.10".
>
> Pending recipes who will want this source present will add:
> do_fetch[depends] += "linux-5.10:do_fetch"
>
> Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
> ---
> meta/recipes-kernel/linux/fetch-linux.inc | 7 +++++++
> meta/recipes-kernel/linux/linux-5.10.bb | 9 +++++++++
> 2 files changed, 16 insertions(+)
> create mode 100644 meta/recipes-kernel/linux/fetch-linux.inc
> create mode 100644 meta/recipes-kernel/linux/linux-5.10.bb
>
> diff --git a/meta/recipes-kernel/linux/fetch-linux.inc b/meta/recipes-kernel/linux/fetch-linux.inc
> new file mode 100644
> index 000000000000..e3fdd0bb36c2
> --- /dev/null
> +++ b/meta/recipes-kernel/linux/fetch-linux.inc
> @@ -0,0 +1,7 @@
> +# Boilerplate to fetch up to a single tag of mainline
> +
> +require recipes-kernel/linux/fetch-only.inc
> +
> +GITCLONEARGS = "--bare --single-branch --branch v${KTAG}"
> +DL_NAME = ".torvalds.linux-${KTAG}"
> +SRC_URI = "${KORG_URLBASE}/torvalds/linux;tag=v${KTAG};nobranch=1;static=1;dlname=${DL_NAME}"
> diff --git a/meta/recipes-kernel/linux/linux-5.10.bb b/meta/recipes-kernel/linux/linux-5.10.bb
> new file mode 100644
> index 000000000000..7f8a27bcb1f0
> --- /dev/null
> +++ b/meta/recipes-kernel/linux/linux-5.10.bb
A specific comment is that this should actually have "fetch-only" in
the name of the recipe.
We want to avoid any confusion about this being buildable, and that it
is not a jumping point
for kernel builds.
Trust me, someone will ask.
linux-yocto has v5.10/base which is vanilla -stable, and simply
because the name is
linux-yocto, people never realize that they can build a completely
stock greg-kh kernel
until they turn blue :D
Otherwise, I like the concept and can take it for a spin.
Bruce
> @@ -0,0 +1,9 @@
> +SUMMARY = "Linux kernel v5.10 source"
> +DESCRIPTION = "Fetch source of the linux kernel up to the v5.10 tag"
> +
> +LICENSE = "GPLv2"
> +LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
> +
> +KTAG = "5.10"
> +
> +require recipes-kernel/linux/fetch-linux.inc
> --
> 2.25.1
>
--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH 12/21] kernel: allow splitting mainline v5.10 source download in two
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
` (10 preceding siblings ...)
2021-04-02 17:15 ` [PATCH 11/21] kernel: add a fetch-only recipe for mainline v5.10 source Paul Gortmaker
@ 2021-04-02 17:15 ` Paul Gortmaker
2021-04-02 17:15 ` [PATCH 13/21] kernel: allow splitting mainline v5.10 source download in three Paul Gortmaker
` (9 subsequent siblings)
21 siblings, 0 replies; 29+ messages in thread
From: Paul Gortmaker @ 2021-04-02 17:15 UTC (permalink / raw)
To: Bruce Ashfield, Richard Purdie; +Cc: linux-yocto, bitbake-devel
With support for sharing packs, we can now optionally split the larger
mainline kernel content up to v5.10 (approx 1.5G) into two. This may be
advantageous to people with release media constraints or unreliable
connections.
By setting "INITIAL_KERNEL_SPLIT = 1" in your configuration, you will
get a linux-4.0 clone and then the linux-5.10 clone in your downloads
directory. From a size perspective, v4.0 is roughly the half way point
between when git use started in 2005, and v5.10 from 2020.
This split is achieved by manually selecting the midpoint tag and cloning
that portion 1st, and then cloning the 2nd 1/2, while using the packs in
the 1st 1/2 as a reference.
Packs deploy both zlib and delta compression - and as with all
compression technologies, splitting the input comes at a cost in terms
of reducing the compression. So two packs will be bigger than one,
three packs bigger than two, and so on. Representative numbers are
listed alongside the implementation in the change itself.
The 2nd 1/2 includes a fetch dependency on the 1st 1/2, so that things
will be serialiazed properly. The dependency also links it into the
chain of events, so no other mentions of v4.0 are required.
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
meta/recipes-kernel/linux/fetch-linux.inc | 15 ++++++++++++++-
meta/recipes-kernel/linux/linux-4.0.bb | 9 +++++++++
meta/recipes-kernel/linux/linux-5.10.bb | 20 ++++++++++++++++++++
3 files changed, 43 insertions(+), 1 deletion(-)
create mode 100644 meta/recipes-kernel/linux/linux-4.0.bb
diff --git a/meta/recipes-kernel/linux/fetch-linux.inc b/meta/recipes-kernel/linux/fetch-linux.inc
index e3fdd0bb36c2..96653a2bea06 100644
--- a/meta/recipes-kernel/linux/fetch-linux.inc
+++ b/meta/recipes-kernel/linux/fetch-linux.inc
@@ -2,6 +2,19 @@
require recipes-kernel/linux/fetch-only.inc
+python __anonymous () {
+ kref = d.getVar("KREF")
+ if not kref:
+ d.setVar("KORG_REF", "")
+ d.setVar("KORG_REFDEP", "")
+ return;
+ clonehost = d.getVar("KORG_SERVER")
+ d.setVar("KORG_REF", ";packref=%s.torvalds.linux-%s" % (clonehost, kref))
+ d.setVar("KORG_REFDEP", "linux-%s:do_fetch"% kref)
+}
+
+do_fetch[depends] += "${KORG_REFDEP}"
+
GITCLONEARGS = "--bare --single-branch --branch v${KTAG}"
DL_NAME = ".torvalds.linux-${KTAG}"
-SRC_URI = "${KORG_URLBASE}/torvalds/linux;tag=v${KTAG};nobranch=1;static=1;dlname=${DL_NAME}"
+SRC_URI = "${KORG_URLBASE}/torvalds/linux;tag=v${KTAG};nobranch=1;static=1;dlname=${DL_NAME}${KORG_REF}"
diff --git a/meta/recipes-kernel/linux/linux-4.0.bb b/meta/recipes-kernel/linux/linux-4.0.bb
new file mode 100644
index 000000000000..68dd054970ca
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-4.0.bb
@@ -0,0 +1,9 @@
+SUMMARY = "Linux kernel v4.0 source"
+DESCRIPTION = "Fetch source of the linux kernel up to the v4.0 tag"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=d7810fab7487fb0aad327b76f1be7cd7"
+
+KTAG = "4.0"
+
+require recipes-kernel/linux/fetch-linux.inc
diff --git a/meta/recipes-kernel/linux/linux-5.10.bb b/meta/recipes-kernel/linux/linux-5.10.bb
index 7f8a27bcb1f0..73e33d405376 100644
--- a/meta/recipes-kernel/linux/linux-5.10.bb
+++ b/meta/recipes-kernel/linux/linux-5.10.bb
@@ -6,4 +6,24 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
KTAG = "5.10"
+# We optionally split the v5.10 download via adding refs and depends.
+# Fragmented downloads can be less apt to timeout, but delta compression
+# will lead to object duplication for > 1 objects/pack/*pack file and
+# an increase in overall on-disk totals. The du -hsm for bare repos:
+#
+# No split: 1521 epoch->5.10
+# One split: 1730 epoch->4.0, 4.0->5.10 (854+877)
+#
+# Download sizes will vary depending on server compression settings.
+
+python __anonymous () {
+ splits = int(d.getVar("INITIAL_KERNEL_SPLIT") or 0)
+ if not splits:
+ return
+ if splits == 1:
+ d.setVar("KREF", "4.0")
+ return
+ bb.fatal("Unsupported number of v5.10 kernel content splits (%s)." % splits)
+}
+
require recipes-kernel/linux/fetch-linux.inc
--
2.25.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 13/21] kernel: allow splitting mainline v5.10 source download in three
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
` (11 preceding siblings ...)
2021-04-02 17:15 ` [PATCH 12/21] kernel: allow splitting mainline v5.10 source download in two Paul Gortmaker
@ 2021-04-02 17:15 ` Paul Gortmaker
2021-04-02 17:15 ` [PATCH 14/21] kernel: allow splitting mainline v5.10 source download in four Paul Gortmaker
` (8 subsequent siblings)
21 siblings, 0 replies; 29+ messages in thread
From: Paul Gortmaker @ 2021-04-02 17:15 UTC (permalink / raw)
To: Bruce Ashfield, Richard Purdie; +Cc: linux-yocto, bitbake-devel
With a split in two done, extending to three is fairly trivial. The
only difference is that the introduced "touch down" point of 4.12 has a
fetch depend/ref itself to the other touch down point of 3.8. The parent
repo (v5.10) has no need of knowledge of what happens prior to 4.12.
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
meta/recipes-kernel/linux/linux-3.8.bb | 9 +++++++++
meta/recipes-kernel/linux/linux-4.12.bb | 10 ++++++++++
meta/recipes-kernel/linux/linux-5.10.bb | 5 +++++
3 files changed, 24 insertions(+)
create mode 100644 meta/recipes-kernel/linux/linux-3.8.bb
create mode 100644 meta/recipes-kernel/linux/linux-4.12.bb
diff --git a/meta/recipes-kernel/linux/linux-3.8.bb b/meta/recipes-kernel/linux/linux-3.8.bb
new file mode 100644
index 000000000000..58063ea51691
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-3.8.bb
@@ -0,0 +1,9 @@
+SUMMARY = "Linux kernel v3.8 source"
+DESCRIPTION = "Fetch source of the linux kernel up to the v3.8 tag"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=d7810fab7487fb0aad327b76f1be7cd7"
+
+KTAG = "3.8"
+
+require recipes-kernel/linux/fetch-linux.inc
diff --git a/meta/recipes-kernel/linux/linux-4.12.bb b/meta/recipes-kernel/linux/linux-4.12.bb
new file mode 100644
index 000000000000..6e8034a77f4a
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-4.12.bb
@@ -0,0 +1,10 @@
+SUMMARY = "Linux kernel v4.12 source"
+DESCRIPTION = "Fetch source of the linux kernel up to the v4.12 tag"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=d7810fab7487fb0aad327b76f1be7cd7"
+
+KTAG = "4.12"
+KREF = "3.8"
+
+require recipes-kernel/linux/fetch-linux.inc
diff --git a/meta/recipes-kernel/linux/linux-5.10.bb b/meta/recipes-kernel/linux/linux-5.10.bb
index 73e33d405376..373c6b8def2e 100644
--- a/meta/recipes-kernel/linux/linux-5.10.bb
+++ b/meta/recipes-kernel/linux/linux-5.10.bb
@@ -7,12 +7,14 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
KTAG = "5.10"
# We optionally split the v5.10 download via adding refs and depends.
+# Multiple splits are achieved by adding more chained dependencies.
# Fragmented downloads can be less apt to timeout, but delta compression
# will lead to object duplication for > 1 objects/pack/*pack file and
# an increase in overall on-disk totals. The du -hsm for bare repos:
#
# No split: 1521 epoch->5.10
# One split: 1730 epoch->4.0, 4.0->5.10 (854+877)
+# Two splits: 1928 epoch->3.8, 3.8->4.12, 4.12->5.10 (628+647+653)
#
# Download sizes will vary depending on server compression settings.
@@ -23,6 +25,9 @@ python __anonymous () {
if splits == 1:
d.setVar("KREF", "4.0")
return
+ if splits == 2:
+ d.setVar("KREF", "4.12")
+ return
bb.fatal("Unsupported number of v5.10 kernel content splits (%s)." % splits)
}
--
2.25.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 14/21] kernel: allow splitting mainline v5.10 source download in four
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
` (12 preceding siblings ...)
2021-04-02 17:15 ` [PATCH 13/21] kernel: allow splitting mainline v5.10 source download in three Paul Gortmaker
@ 2021-04-02 17:15 ` Paul Gortmaker
2021-04-02 17:15 ` [PATCH 15/21] kernel: add recipe for linux-master (mainline latest) Paul Gortmaker
` (7 subsequent siblings)
21 siblings, 0 replies; 29+ messages in thread
From: Paul Gortmaker @ 2021-04-02 17:15 UTC (permalink / raw)
To: Bruce Ashfield, Richard Purdie; +Cc: linux-yocto, bitbake-devel
Extending from three to four roughly equal chunks introduces three new
split boundaries - 3.3 4.3 and 4.18, but is trivial to hook in.
This gets chunks down to roughly 1/2G and with the resulting loss of
compression efficiency, it doesn't seem to make sense to support
splitting things any further. See the chunk sizes in the commit.
Document the new variable and its allowed value range since we don't
see adding any further split options.
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
documentation/ref-manual/variables.rst | 6 ++++++
meta/recipes-kernel/linux/linux-3.3.bb | 9 +++++++++
meta/recipes-kernel/linux/linux-4.18.bb | 10 ++++++++++
meta/recipes-kernel/linux/linux-4.3.bb | 10 ++++++++++
meta/recipes-kernel/linux/linux-5.10.bb | 4 ++++
5 files changed, 39 insertions(+)
create mode 100644 meta/recipes-kernel/linux/linux-3.3.bb
create mode 100644 meta/recipes-kernel/linux/linux-4.18.bb
create mode 100644 meta/recipes-kernel/linux/linux-4.3.bb
diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst
index 55028987c08a..1226bbf77193 100644
--- a/documentation/ref-manual/variables.rst
+++ b/documentation/ref-manual/variables.rst
@@ -3441,6 +3441,12 @@ system and gives an overview of their function and contents.
lines in the example are from individual recipes and show how the
``PR`` value is set.
+ :term:`INITIAL_KERNEL_SPLIT`
+ Specifies the number of times the initial kernel Git clone should be
+ split in order to avoid one single gigabyte+ download. Supported
+ values are 0 through 3 inclusive, resulting in one through four
+ separate chunks (repositories) in the download dir.
+
:term:`INCOMPATIBLE_LICENSE`
Specifies a space-separated list of license names (as they would
appear in :term:`LICENSE`) that should be excluded
diff --git a/meta/recipes-kernel/linux/linux-3.3.bb b/meta/recipes-kernel/linux/linux-3.3.bb
new file mode 100644
index 000000000000..da355f9cd32c
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-3.3.bb
@@ -0,0 +1,9 @@
+SUMMARY = "Linux kernel v3.3 source"
+DESCRIPTION = "Fetch source of the linux kernel up to the v3.3 tag"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=d7810fab7487fb0aad327b76f1be7cd7"
+
+KTAG = "3.3"
+
+require recipes-kernel/linux/fetch-linux.inc
diff --git a/meta/recipes-kernel/linux/linux-4.18.bb b/meta/recipes-kernel/linux/linux-4.18.bb
new file mode 100644
index 000000000000..ea87f5e92b52
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-4.18.bb
@@ -0,0 +1,10 @@
+SUMMARY = "Linux kernel v4.18 source"
+DESCRIPTION = "Fetch source of the linux kernel up to the v4.18 tag"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=d7810fab7487fb0aad327b76f1be7cd7"
+
+KTAG = "4.18"
+KREF = "4.3"
+
+require recipes-kernel/linux/fetch-linux.inc
diff --git a/meta/recipes-kernel/linux/linux-4.3.bb b/meta/recipes-kernel/linux/linux-4.3.bb
new file mode 100644
index 000000000000..35e5fd48d8ba
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-4.3.bb
@@ -0,0 +1,10 @@
+SUMMARY = "Linux kernel v4.3 source"
+DESCRIPTION = "Fetch source of the linux kernel up to the v4.3 tag"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=d7810fab7487fb0aad327b76f1be7cd7"
+
+KTAG = "4.3"
+KREF = "3.3"
+
+require recipes-kernel/linux/fetch-linux.inc
diff --git a/meta/recipes-kernel/linux/linux-5.10.bb b/meta/recipes-kernel/linux/linux-5.10.bb
index 373c6b8def2e..e147c854477d 100644
--- a/meta/recipes-kernel/linux/linux-5.10.bb
+++ b/meta/recipes-kernel/linux/linux-5.10.bb
@@ -15,6 +15,7 @@ KTAG = "5.10"
# No split: 1521 epoch->5.10
# One split: 1730 epoch->4.0, 4.0->5.10 (854+877)
# Two splits: 1928 epoch->3.8, 3.8->4.12, 4.12->5.10 (628+647+653)
+# Three splits: 2106 epoch->3.3, 3.3->4.3, 4.3->4.18, 4.18->5.10 (536+532+519+520)
#
# Download sizes will vary depending on server compression settings.
@@ -28,6 +29,9 @@ python __anonymous () {
if splits == 2:
d.setVar("KREF", "4.12")
return
+ if splits == 3:
+ d.setVar("KREF", "4.18")
+ return
bb.fatal("Unsupported number of v5.10 kernel content splits (%s)." % splits)
}
--
2.25.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 15/21] kernel: add recipe for linux-master (mainline latest)
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
` (13 preceding siblings ...)
2021-04-02 17:15 ` [PATCH 14/21] kernel: allow splitting mainline v5.10 source download in four Paul Gortmaker
@ 2021-04-02 17:15 ` Paul Gortmaker
2021-04-02 20:16 ` Bruce Ashfield
2021-04-02 17:15 ` [PATCH 16/21] kernel: add stable fetch recipes for v5.4.x, v5.10.x and v5.12.x Paul Gortmaker
` (6 subsequent siblings)
21 siblings, 1 reply; 29+ messages in thread
From: Paul Gortmaker @ 2021-04-02 17:15 UTC (permalink / raw)
To: Bruce Ashfield, Richard Purdie; +Cc: linux-yocto, bitbake-devel
The idea for this recipe is to have a name "linux-master" that people
can depend on as a reference for mainline; today and ideally for as long
as we are maintaining it. While the recipe is there for everybody,
the obvious user we are creating this for is the linux-yocto-dev repo.
At some point in the future, we'll carve off another chunk of post-v5.10
work, mark it static and put it alongside of v5.10 and let this mainline
recipe reference that. But use of this "linux-master" will be 100%
transparent to the underlying backing store changes.
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
meta/recipes-kernel/linux/linux-master.bb | 13 +++++++++++++
1 file changed, 13 insertions(+)
create mode 100644 meta/recipes-kernel/linux/linux-master.bb
diff --git a/meta/recipes-kernel/linux/linux-master.bb b/meta/recipes-kernel/linux/linux-master.bb
new file mode 100644
index 000000000000..f674c2562e81
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-master.bb
@@ -0,0 +1,13 @@
+SUMMARY = "Linux kernel mainline source"
+DESCRIPTION = "Fetch source of the latest linux mainline kernel"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
+
+DL_NAME = ".torvalds.linux-master"
+KORG_REF = "git.kernel.org.torvalds.linux-5.10"
+SRC_URI = "${KORG_URLBASE}/torvalds/linux;rev=master;nobranch=1;dlname=${DL_NAME};packref=${KORG_REF}"
+
+do_fetch[depends] += "linux-5.10:do_fetch"
+
+require recipes-kernel/linux/fetch-only.inc
--
2.25.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* Re: [PATCH 15/21] kernel: add recipe for linux-master (mainline latest)
2021-04-02 17:15 ` [PATCH 15/21] kernel: add recipe for linux-master (mainline latest) Paul Gortmaker
@ 2021-04-02 20:16 ` Bruce Ashfield
0 siblings, 0 replies; 29+ messages in thread
From: Bruce Ashfield @ 2021-04-02 20:16 UTC (permalink / raw)
To: Paul Gortmaker; +Cc: Richard Purdie, Linux Yocto, bitbake-devel
On Fri, Apr 2, 2021 at 1:16 PM Paul Gortmaker
<paul.gortmaker@windriver.com> wrote:
>
> The idea for this recipe is to have a name "linux-master" that people
> can depend on as a reference for mainline; today and ideally for as long
> as we are maintaining it. While the recipe is there for everybody,
> the obvious user we are creating this for is the linux-yocto-dev repo.
>
> At some point in the future, we'll carve off another chunk of post-v5.10
> work, mark it static and put it alongside of v5.10 and let this mainline
> recipe reference that. But use of this "linux-master" will be 100%
> transparent to the underlying backing store changes.
>
> Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
> ---
> meta/recipes-kernel/linux/linux-master.bb | 13 +++++++++++++
> 1 file changed, 13 insertions(+)
> create mode 100644 meta/recipes-kernel/linux/linux-master.bb
>
> diff --git a/meta/recipes-kernel/linux/linux-master.bb b/meta/recipes-kernel/linux/linux-master.bb
> new file mode 100644
> index 000000000000..f674c2562e81
> --- /dev/null
> +++ b/meta/recipes-kernel/linux/linux-master.bb
I won't reply to all of these fetch only naming patches (maybe this is
the last one,
I didn't check), so this is my last one.
My same namespace comment applies here
Actually, thinking of it more, we could even have these both with fetch-only in
the recipe name AND in a directory structure that implies the same thing.
And thirdly, maybe we could taint the tasks in some way to prevent that jumping
point to building.
Since yes, you'll get someone trying to ask for support using this to build
version 'x' of mainline. I have the battle scars to prove it :D
Bruce
> @@ -0,0 +1,13 @@
> +SUMMARY = "Linux kernel mainline source"
> +DESCRIPTION = "Fetch source of the latest linux mainline kernel"
> +
> +LICENSE = "GPLv2"
> +LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
> +
> +DL_NAME = ".torvalds.linux-master"
> +KORG_REF = "git.kernel.org.torvalds.linux-5.10"
> +SRC_URI = "${KORG_URLBASE}/torvalds/linux;rev=master;nobranch=1;dlname=${DL_NAME};packref=${KORG_REF}"
> +
> +do_fetch[depends] += "linux-5.10:do_fetch"
> +
> +require recipes-kernel/linux/fetch-only.inc
> --
> 2.25.1
>
--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH 16/21] kernel: add stable fetch recipes for v5.4.x, v5.10.x and v5.12.x
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
` (14 preceding siblings ...)
2021-04-02 17:15 ` [PATCH 15/21] kernel: add recipe for linux-master (mainline latest) Paul Gortmaker
@ 2021-04-02 17:15 ` Paul Gortmaker
2021-04-02 17:15 ` [PATCH 17/21] kernel: add preempt-rt fetch recipes for v5.4.x, v5.10.x and 5.12.x Paul Gortmaker
` (5 subsequent siblings)
21 siblings, 0 replies; 29+ messages in thread
From: Paul Gortmaker @ 2021-04-02 17:15 UTC (permalink / raw)
To: Bruce Ashfield, Richard Purdie; +Cc: linux-yocto, bitbake-devel
One might wonder why there isn't just a stable-5.x to encompass all of
these, but by having them separate, we can retire/EOL them individually
without disruption.
Note that references are a "round up" to the nearest kernel recipe
containing the stable baseline - so stable-5.4 references linux-5.10 as
we didn't specifically call out linux-5.4 as an independent recipe.
Also, stable-5.12 doesn't exist yet, but we know it will, so we insert
a placeholder in for it, so that v5.12-rt can reference it w/o future
change; we just switch the one line in stable-5.12 once it exists.
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
meta/recipes-kernel/linux/fetch-stable.inc | 24 ++++++++++++++++++++++
meta/recipes-kernel/linux/stable-5.10.bb | 10 +++++++++
meta/recipes-kernel/linux/stable-5.12.bb | 16 +++++++++++++++
meta/recipes-kernel/linux/stable-5.4.bb | 11 ++++++++++
4 files changed, 61 insertions(+)
create mode 100644 meta/recipes-kernel/linux/fetch-stable.inc
create mode 100644 meta/recipes-kernel/linux/stable-5.10.bb
create mode 100644 meta/recipes-kernel/linux/stable-5.12.bb
create mode 100644 meta/recipes-kernel/linux/stable-5.4.bb
diff --git a/meta/recipes-kernel/linux/fetch-stable.inc b/meta/recipes-kernel/linux/fetch-stable.inc
new file mode 100644
index 000000000000..051c35db8772
--- /dev/null
+++ b/meta/recipes-kernel/linux/fetch-stable.inc
@@ -0,0 +1,24 @@
+# Boilerplate to fetch a single branch of stable.
+# Reference is non-optional; there is no excuse to not use one.
+# Use a packref for a fixed tag reference; altref for master.
+
+require recipes-kernel/linux/fetch-only.inc
+
+python __anonymous () {
+ kref = d.getVar("KREF") or "master"
+ if kref == "master":
+ d.setVar("RTYPE", "alt")
+ else:
+ d.setVar("RTYPE", "pack")
+ clonehost = d.getVar("KORG_SERVER")
+ d.setVar("KORG_REF", "%s.torvalds.linux-%s" % (clonehost, kref))
+ d.setVar("KORG_REFDEP", "linux-%s:do_fetch"% kref)
+}
+
+do_fetch[depends] += "${KORG_REFDEP}"
+
+STABLE_REV ?= "${STABLE}"
+DL_NAME ?= ".stable.${STABLE}"
+GITCLONEARGS = "--bare --single-branch --branch ${STABLE}"
+GITFETCHREFS ?= "refs/heads/${STABLE}:refs/heads/${STABLE}"
+SRC_URI = "${KORG_URLBASE}/stable/linux-stable;rev=${STABLE_REV};nobranch=1;dlname=${DL_NAME};${RTYPE}ref=${KORG_REF}"
diff --git a/meta/recipes-kernel/linux/stable-5.10.bb b/meta/recipes-kernel/linux/stable-5.10.bb
new file mode 100644
index 000000000000..4dd09fd212d1
--- /dev/null
+++ b/meta/recipes-kernel/linux/stable-5.10.bb
@@ -0,0 +1,10 @@
+SUMMARY = "Linux stable kernel v5.10.y source"
+DESCRIPTION = "Fetch source of the stable linux kernel up to the end of the v5.10.y series"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
+
+STABLE = "linux-5.10.y"
+KREF = "5.10"
+
+require recipes-kernel/linux/fetch-stable.inc
diff --git a/meta/recipes-kernel/linux/stable-5.12.bb b/meta/recipes-kernel/linux/stable-5.12.bb
new file mode 100644
index 000000000000..18cb764dc3f3
--- /dev/null
+++ b/meta/recipes-kernel/linux/stable-5.12.bb
@@ -0,0 +1,16 @@
+SUMMARY = "Linux stable kernel v5.12.y source"
+DESCRIPTION = "Fetch source of the stable linux kernel up to the end of the v5.12.y series"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
+
+# The stable branch "linux-5.12.y" wont exist until late April or early May.
+# For now this will just create a linux-stable "empty" 5.12.x placeholder.
+#STABLE = "linux-5.12.y"
+STABLE = "master"
+DL_NAME = ".stable.linux-5.12.y"
+
+# mainline latest is the default stable reference if none is specified.
+# KREF = "master"
+
+require recipes-kernel/linux/fetch-stable.inc
diff --git a/meta/recipes-kernel/linux/stable-5.4.bb b/meta/recipes-kernel/linux/stable-5.4.bb
new file mode 100644
index 000000000000..98842bacc1ab
--- /dev/null
+++ b/meta/recipes-kernel/linux/stable-5.4.bb
@@ -0,0 +1,11 @@
+SUMMARY = "Linux stable kernel v5.4.y source"
+DESCRIPTION = "Fetch source of the stable linux kernel up to the end of the v5.4.y series"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
+
+STABLE = "linux-5.4.y"
+# For reference, round up to the nearest tag with a recipe
+KREF = "5.10"
+
+require recipes-kernel/linux/fetch-stable.inc
--
2.25.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 17/21] kernel: add preempt-rt fetch recipes for v5.4.x, v5.10.x and 5.12.x
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
` (15 preceding siblings ...)
2021-04-02 17:15 ` [PATCH 16/21] kernel: add stable fetch recipes for v5.4.x, v5.10.x and v5.12.x Paul Gortmaker
@ 2021-04-02 17:15 ` Paul Gortmaker
2021-04-02 17:15 ` [PATCH 18/21] kernel: make v5.4.x Yocto recipes use shared source Paul Gortmaker
` (4 subsequent siblings)
21 siblings, 0 replies; 29+ messages in thread
From: Paul Gortmaker @ 2021-04-02 17:15 UTC (permalink / raw)
To: Bruce Ashfield, Richard Purdie; +Cc: linux-yocto, bitbake-devel
Note that the v5.12 content currently is in the devel repo and will
eventually get moved over to the stable-rt repo. When that move
happens, we'll need to update the SRC_URI accordingly. [The v5.12-rt
content itself will remain fast-forward with respect to that which is
in devel currently.]
Also, since each reference their respective stable baseline repos,
which are still in maintenance and not static content, we use alternate
reference instead of pack reference back to stable.
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
meta/recipes-kernel/linux/fetch-rt.inc | 25 ++++++++++++++++++++++
meta/recipes-kernel/linux/linux-rt-5.10.bb | 9 ++++++++
meta/recipes-kernel/linux/linux-rt-5.12.bb | 12 +++++++++++
meta/recipes-kernel/linux/linux-rt-5.4.bb | 9 ++++++++
4 files changed, 55 insertions(+)
create mode 100644 meta/recipes-kernel/linux/fetch-rt.inc
create mode 100644 meta/recipes-kernel/linux/linux-rt-5.10.bb
create mode 100644 meta/recipes-kernel/linux/linux-rt-5.12.bb
create mode 100644 meta/recipes-kernel/linux/linux-rt-5.4.bb
diff --git a/meta/recipes-kernel/linux/fetch-rt.inc b/meta/recipes-kernel/linux/fetch-rt.inc
new file mode 100644
index 000000000000..18dbf0628640
--- /dev/null
+++ b/meta/recipes-kernel/linux/fetch-rt.inc
@@ -0,0 +1,25 @@
+# Boilerplate to fetch a single branch of preempt-rt
+# There are two different repos, and two branching name patterns.
+
+require recipes-kernel/linux/fetch-only.inc
+
+python __anonymous () {
+ rt = d.getVar("RT")
+ rt_repo = d.getVar("RT_REPO")
+ if rt_repo.endswith("devel"):
+ d.setVar("RT_BRANCH", "linux-%s.y-rt" % rt)
+ else:
+ d.setVar("RT_BRANCH", "v%s-rt" % rt)
+ clonehost = d.getVar("KORG_SERVER")
+ d.setVar("KORG_REF", "%s.stable.linux-%s.y" % (clonehost, rt))
+ d.setVar("KORG_REFDEP", "stable-%s:do_fetch" % rt)
+}
+
+do_fetch[depends] += "${KORG_REFDEP}"
+
+RT_REV ?= "${RT_BRANCH}"
+RT_REPO ?= "linux-stable-rt"
+GITCLONEARGS = "--bare --single-branch --branch ${RT_BRANCH}"
+GITFETCHREFS ?= "refs/heads/${RT_BRANCH}:refs/heads/${RT_BRANCH}"
+DL_NAME = ".preempt-rt.linux-${RT}.y"
+SRC_URI = "${KORG_URLBASE}/rt/${RT_REPO};rev=${RT_REV};nobranch=1;dlname=${DL_NAME};altref=${KORG_REF}"
diff --git a/meta/recipes-kernel/linux/linux-rt-5.10.bb b/meta/recipes-kernel/linux/linux-rt-5.10.bb
new file mode 100644
index 000000000000..04520101a443
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-rt-5.10.bb
@@ -0,0 +1,9 @@
+SUMMARY = "Linux preempt-rt v5.10.y kernel source"
+DESCRIPTION = "Fetch source of the preempt-rt linux kernel up to the end of the v5.10.y series"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
+
+RT = "5.10"
+
+require recipes-kernel/linux/fetch-rt.inc
diff --git a/meta/recipes-kernel/linux/linux-rt-5.12.bb b/meta/recipes-kernel/linux/linux-rt-5.12.bb
new file mode 100644
index 000000000000..88f2daf92032
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-rt-5.12.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Linux preempt-rt v5.12.y kernel source"
+DESCRIPTION = "Fetch source of the preempt-rt linux kernel up to the end of the v5.12.y series"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
+
+RT = "5.12"
+
+# Note: v5.12 RT will move to linux-stable-rt repo sometime
+RT_REPO = "linux-rt-devel"
+
+require recipes-kernel/linux/fetch-rt.inc
diff --git a/meta/recipes-kernel/linux/linux-rt-5.4.bb b/meta/recipes-kernel/linux/linux-rt-5.4.bb
new file mode 100644
index 000000000000..3d70be5aaa9b
--- /dev/null
+++ b/meta/recipes-kernel/linux/linux-rt-5.4.bb
@@ -0,0 +1,9 @@
+SUMMARY = "Linux preempt-rt v5.4.y kernel source"
+DESCRIPTION = "Fetch source of the preempt-rt linux kernel up to the end of the v5.4.y series"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
+
+RT = "5.4"
+
+require recipes-kernel/linux/fetch-rt.inc
--
2.25.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 18/21] kernel: make v5.4.x Yocto recipes use shared source
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
` (16 preceding siblings ...)
2021-04-02 17:15 ` [PATCH 17/21] kernel: add preempt-rt fetch recipes for v5.4.x, v5.10.x and 5.12.x Paul Gortmaker
@ 2021-04-02 17:15 ` Paul Gortmaker
2021-04-02 17:15 ` [PATCH 19/21] kernel: make v5.10.x " Paul Gortmaker
` (3 subsequent siblings)
21 siblings, 0 replies; 29+ messages in thread
From: Paul Gortmaker @ 2021-04-02 17:15 UTC (permalink / raw)
To: Bruce Ashfield, Richard Purdie; +Cc: linux-yocto, bitbake-devel
With all the heavy lifting done, all that is needed is to select the
right endpoint for our reference, and the right fetch wildcards to
ensure we only get what we need. We've got:
linux-5.10 (superset contining v5.4)
--> stable-5.4
--> linux-5.4-rt
--> linux-yocto_5.4 (and rt/tiny variants)
and hence the preempt-rt is the endpoint for the reference.
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
meta/recipes-kernel/linux/linux-yocto-rt_5.4.bb | 7 ++++++-
meta/recipes-kernel/linux/linux-yocto-tiny_5.4.bb | 7 ++++++-
meta/recipes-kernel/linux/linux-yocto_5.4.bb | 7 ++++++-
3 files changed, 18 insertions(+), 3 deletions(-)
diff --git a/meta/recipes-kernel/linux/linux-yocto-rt_5.4.bb b/meta/recipes-kernel/linux/linux-yocto-rt_5.4.bb
index 2ffc8ed5428b..373da8501f6b 100644
--- a/meta/recipes-kernel/linux/linux-yocto-rt_5.4.bb
+++ b/meta/recipes-kernel/linux/linux-yocto-rt_5.4.bb
@@ -14,7 +14,11 @@ python () {
SRCREV_machine ?= "65bbe689d98a007848008be2c8edeb5fa8066829"
SRCREV_meta ?= "19738ca97b999a3b150e2d34232bb44b6537348f"
-SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;branch=${KBRANCH};name=machine \
+KREF = "git.kernel.org.preempt-rt.linux-5.4.y"
+GITFETCHREFS_machine += " refs/heads/v5.4/*:refs/heads/v5.4/*"
+GITCLONEARGS_machine = "--bare --single-branch --branch v5.4/base"
+
+SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;name=machine;branch=${KBRANCH};altref=${KREF} \
git://git.yoctoproject.org/yocto-kernel-cache;type=kmeta;name=meta;branch=yocto-5.4;destsuffix=${KMETA}"
LINUX_VERSION ?= "5.4.107"
@@ -23,6 +27,7 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=bbea815ee2795b2f4230826c0c6b8814"
DEPENDS += "${@bb.utils.contains('ARCH', 'x86', 'elfutils-native', '', d)}"
DEPENDS += "openssl-native util-linux-native"
+do_fetch[depends] += "linux-rt-5.4:do_fetch"
PV = "${LINUX_VERSION}+git${SRCPV}"
diff --git a/meta/recipes-kernel/linux/linux-yocto-tiny_5.4.bb b/meta/recipes-kernel/linux/linux-yocto-tiny_5.4.bb
index 2b6e35a69c10..40327039368f 100644
--- a/meta/recipes-kernel/linux/linux-yocto-tiny_5.4.bb
+++ b/meta/recipes-kernel/linux/linux-yocto-tiny_5.4.bb
@@ -11,6 +11,7 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=bbea815ee2795b2f4230826c0c6b8814"
DEPENDS += "${@bb.utils.contains('ARCH', 'x86', 'elfutils-native', '', d)}"
DEPENDS += "openssl-native util-linux-native"
+do_fetch[depends] += "linux-rt-5.4:do_fetch"
KMETA = "kernel-meta"
KCONF_BSP_AUDIT_LEVEL = "2"
@@ -21,7 +22,11 @@ SRCREV_meta ?= "19738ca97b999a3b150e2d34232bb44b6537348f"
PV = "${LINUX_VERSION}+git${SRCPV}"
-SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;branch=${KBRANCH};name=machine \
+KREF = "git.kernel.org.preempt-rt.linux-5.4.y"
+GITFETCHREFS_machine += " refs/heads/v5.4/*:refs/heads/v5.4/*"
+GITCLONEARGS_machine = "--bare --single-branch --branch v5.4/base"
+
+SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;name=machine;branch=${KBRANCH};altref=${KREF} \
git://git.yoctoproject.org/yocto-kernel-cache;type=kmeta;name=meta;branch=yocto-5.4;destsuffix=${KMETA}"
COMPATIBLE_MACHINE = "qemux86|qemux86-64|qemuarm|qemuarmv5"
diff --git a/meta/recipes-kernel/linux/linux-yocto_5.4.bb b/meta/recipes-kernel/linux/linux-yocto_5.4.bb
index 245c3d574b44..938f3bc9e595 100644
--- a/meta/recipes-kernel/linux/linux-yocto_5.4.bb
+++ b/meta/recipes-kernel/linux/linux-yocto_5.4.bb
@@ -26,7 +26,11 @@ SRCREV_meta ?= "19738ca97b999a3b150e2d34232bb44b6537348f"
# remap qemuarm to qemuarma15 for the 5.4 kernel
# KMACHINE_qemuarm ?= "qemuarma15"
-SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;name=machine;branch=${KBRANCH}; \
+KREF = "git.kernel.org.preempt-rt.linux-5.4.y"
+GITFETCHREFS_machine += " refs/heads/v5.4/*:refs/heads/v5.4/*"
+GITCLONEARGS_machine = "--bare --single-branch --branch v5.4/base"
+
+SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;name=machine;branch=${KBRANCH};altref=${KREF}; \
git://git.yoctoproject.org/yocto-kernel-cache;type=kmeta;name=meta;branch=yocto-5.4;destsuffix=${KMETA}"
LIC_FILES_CHKSUM = "file://COPYING;md5=bbea815ee2795b2f4230826c0c6b8814"
@@ -34,6 +38,7 @@ LINUX_VERSION ?= "5.4.107"
DEPENDS += "${@bb.utils.contains('ARCH', 'x86', 'elfutils-native', '', d)}"
DEPENDS += "openssl-native util-linux-native"
+do_fetch[depends] += "linux-rt-5.4:do_fetch"
PV = "${LINUX_VERSION}+git${SRCPV}"
--
2.25.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 19/21] kernel: make v5.10.x Yocto recipes use shared source
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
` (17 preceding siblings ...)
2021-04-02 17:15 ` [PATCH 18/21] kernel: make v5.4.x Yocto recipes use shared source Paul Gortmaker
@ 2021-04-02 17:15 ` Paul Gortmaker
2021-04-02 17:15 ` [PATCH 20/21] kernel: make linux-yocto-dev recipe " Paul Gortmaker
` (2 subsequent siblings)
21 siblings, 0 replies; 29+ messages in thread
From: Paul Gortmaker @ 2021-04-02 17:15 UTC (permalink / raw)
To: Bruce Ashfield, Richard Purdie; +Cc: linux-yocto, bitbake-devel
With all the heavy lifting done, all that is needed is to select the
right endpoint for our reference, and the right fetch wildcards to
ensure we only get what we need. We've got:
linux-5.10
--> stable-5.10
--> linux-5.10-rt
--> linux-yocto_5.10 (and rt/tiny variants)
and hence the preempt-rt is the endpoint for the reference.
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
meta/recipes-kernel/linux/linux-yocto-rt_5.10.bb | 7 ++++++-
meta/recipes-kernel/linux/linux-yocto-tiny_5.10.bb | 7 ++++++-
meta/recipes-kernel/linux/linux-yocto_5.10.bb | 7 ++++++-
3 files changed, 18 insertions(+), 3 deletions(-)
diff --git a/meta/recipes-kernel/linux/linux-yocto-rt_5.10.bb b/meta/recipes-kernel/linux/linux-yocto-rt_5.10.bb
index cb34887cda83..072b2eaabac8 100644
--- a/meta/recipes-kernel/linux/linux-yocto-rt_5.10.bb
+++ b/meta/recipes-kernel/linux/linux-yocto-rt_5.10.bb
@@ -14,7 +14,11 @@ python () {
SRCREV_machine ?= "be2935bce35f9adb6d0e735d42651e81a5094adf"
SRCREV_meta ?= "031f6c76e488a3563f35258c72ff1de3e25a512e"
-SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;branch=${KBRANCH};name=machine \
+KREF = "git.kernel.org.preempt-rt.linux-5.10.y"
+GITFETCHREFS_machine += " refs/heads/v5.10/*:refs/heads/v5.10/*"
+GITCLONEARGS_machine = "--bare --single-branch --branch v5.10/base"
+
+SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;name=machine;branch=${KBRANCH};altref=${KREF} \
git://git.yoctoproject.org/yocto-kernel-cache;type=kmeta;name=meta;branch=yocto-5.10;destsuffix=${KMETA}"
LINUX_VERSION ?= "5.10.25"
@@ -23,6 +27,7 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
DEPENDS += "${@bb.utils.contains('ARCH', 'x86', 'elfutils-native', '', d)}"
DEPENDS += "openssl-native util-linux-native"
+do_fetch[depends] += "linux-rt-5.10:do_fetch"
PV = "${LINUX_VERSION}+git${SRCPV}"
diff --git a/meta/recipes-kernel/linux/linux-yocto-tiny_5.10.bb b/meta/recipes-kernel/linux/linux-yocto-tiny_5.10.bb
index 83e59b0ebb8f..14db1e578c1d 100644
--- a/meta/recipes-kernel/linux/linux-yocto-tiny_5.10.bb
+++ b/meta/recipes-kernel/linux/linux-yocto-tiny_5.10.bb
@@ -11,6 +11,7 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
DEPENDS += "${@bb.utils.contains('ARCH', 'x86', 'elfutils-native', '', d)}"
DEPENDS += "openssl-native util-linux-native"
+do_fetch[depends] += "linux-rt-5.10:do_fetch"
KMETA = "kernel-meta"
KCONF_BSP_AUDIT_LEVEL = "2"
@@ -21,7 +22,11 @@ SRCREV_meta ?= "031f6c76e488a3563f35258c72ff1de3e25a512e"
PV = "${LINUX_VERSION}+git${SRCPV}"
-SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;branch=${KBRANCH};name=machine \
+KREF = "git.kernel.org.preempt-rt.linux-5.10.y"
+GITFETCHREFS_machine += " refs/heads/v5.10/*:refs/heads/v5.10/*"
+GITCLONEARGS_machine = "--bare --single-branch --branch v5.10/base"
+
+SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;name=machine;branch=${KBRANCH};altref=${KREF} \
git://git.yoctoproject.org/yocto-kernel-cache;type=kmeta;name=meta;branch=yocto-5.10;destsuffix=${KMETA}"
COMPATIBLE_MACHINE = "qemux86|qemux86-64|qemuarm|qemuarmv5"
diff --git a/meta/recipes-kernel/linux/linux-yocto_5.10.bb b/meta/recipes-kernel/linux/linux-yocto_5.10.bb
index 026e69511a60..c7a8cb360d2d 100644
--- a/meta/recipes-kernel/linux/linux-yocto_5.10.bb
+++ b/meta/recipes-kernel/linux/linux-yocto_5.10.bb
@@ -28,7 +28,11 @@ SRCREV_meta ?= "031f6c76e488a3563f35258c72ff1de3e25a512e"
# remap qemuarm to qemuarma15 for the 5.8 kernel
# KMACHINE_qemuarm ?= "qemuarma15"
-SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;name=machine;branch=${KBRANCH}; \
+KREF = "git.kernel.org.preempt-rt.linux-5.10.y"
+GITFETCHREFS_machine += " refs/heads/v5.10/*:refs/heads/v5.10/*"
+GITCLONEARGS_machine = "--bare --single-branch --branch v5.10/base"
+
+SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;name=machine;branch=${KBRANCH};altref=${KREF} \
git://git.yoctoproject.org/yocto-kernel-cache;type=kmeta;name=meta;branch=yocto-5.10;destsuffix=${KMETA}"
LIC_FILES_CHKSUM = "file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46"
@@ -37,6 +41,7 @@ LINUX_VERSION ?= "5.10.25"
DEPENDS += "${@bb.utils.contains('ARCH', 'x86', 'elfutils-native', '', d)}"
DEPENDS += "openssl-native util-linux-native"
DEPENDS += "gmp-native"
+do_fetch[depends] += "linux-rt-5.10:do_fetch"
PV = "${LINUX_VERSION}+git${SRCPV}"
--
2.25.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 20/21] kernel: make linux-yocto-dev recipe use shared source
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
` (18 preceding siblings ...)
2021-04-02 17:15 ` [PATCH 19/21] kernel: make v5.10.x " Paul Gortmaker
@ 2021-04-02 17:15 ` Paul Gortmaker
2021-04-02 17:15 ` [PATCH 21/21] kernel: disable (pre)mirror for linux-yocto and linux-yocto-dev Paul Gortmaker
2021-04-02 22:14 ` [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Richard Purdie
21 siblings, 0 replies; 29+ messages in thread
From: Paul Gortmaker @ 2021-04-02 17:15 UTC (permalink / raw)
To: Bruce Ashfield, Richard Purdie; +Cc: linux-yocto, bitbake-devel
At the moment, linux-yocto-dev is set up to reference v5.12-rt:
mainline
---> stable-5.12
---> linux-rt-5.12
---> linux-yocto-dev
We'll probably move to v5.13-rcN in late May and at that point we
can point it at referencing mainline directly while the respective
stable and -rt versions take shape, or perhaps create placeholders
in advance (as was just done for stable-5.12).
But the good thing is that we'll get v5.13-rcN via mainline, which is
already in the above chain, so we can take our time; there is no
pressure to reconsider the chaining of repos synchronously with the
kernel uprev.
And it is also constructive to consider the reverse case: where we are
pointing at mainline directly and a stable (or -rt) gets added/used.
mainline
---> linux-yocto-dev
In this case, if linux-yocto-dev used a shiny new v5.13.1 stable,
without it yet being explicitly a part of the above chain, then the git
objects relating to that would simply flow from the server directly into
our linux-yocto-dev clone directly. Nothing breaks or falls over. So
there is no requirement to immediately react to creation of v5.13 stable
or preempt-rt recipes (even though they are trivial).
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
meta/recipes-kernel/linux/linux-yocto-dev.bb | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/meta/recipes-kernel/linux/linux-yocto-dev.bb b/meta/recipes-kernel/linux/linux-yocto-dev.bb
index 8725473d1cd8..b4b91593aef8 100644
--- a/meta/recipes-kernel/linux/linux-yocto-dev.bb
+++ b/meta/recipes-kernel/linux/linux-yocto-dev.bb
@@ -19,7 +19,16 @@ include recipes-kernel/linux/linux-yocto-dev-revisions.inc
KBRANCH = "standard/base"
KMETA = "kernel-meta"
-SRC_URI = "git://git.yoctoproject.org/linux-yocto-dev.git;branch=${KBRANCH};name=machine \
+# When v5.13-rcN comes along, we can revector back to mainline
+#KREF = "git.kernel.org.torvalds.linux-master"
+#do_fetch[depends] += "linux-master:do_fetch"
+KREF = "git.kernel.org.preempt-rt.linux-5.12.y"
+do_fetch[depends] += "linux-rt-5.12:do_fetch"
+
+GITFETCHREFS_machine += " refs/heads/standard/*:refs/heads/standard/*"
+GITCLONEARGS_machine = "--bare --single-branch --branch master"
+
+SRC_URI = "git://git.yoctoproject.org/linux-yocto-dev.git;name=machine;branch=${KBRANCH};altref=${KREF} \
git://git.yoctoproject.org/yocto-kernel-cache;type=kmeta;name=meta;branch=master;destsuffix=${KMETA}"
# Set default SRCREVs. Both the machine and meta SRCREVs are statically set
--
2.25.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 21/21] kernel: disable (pre)mirror for linux-yocto and linux-yocto-dev
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
` (19 preceding siblings ...)
2021-04-02 17:15 ` [PATCH 20/21] kernel: make linux-yocto-dev recipe " Paul Gortmaker
@ 2021-04-02 17:15 ` Paul Gortmaker
2021-04-02 20:19 ` Bruce Ashfield
2021-04-02 22:14 ` [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Richard Purdie
21 siblings, 1 reply; 29+ messages in thread
From: Paul Gortmaker @ 2021-04-02 17:15 UTC (permalink / raw)
To: Bruce Ashfield, Richard Purdie; +Cc: linux-yocto, bitbake-devel
As per the comment in the file itself, we have to explicitly disable
(pre)mirror support for both kernel repositories, or else the fetcher
will unhelpfully volunteer to wget gigs of what amounts to largely
useless content that we don't want.
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
meta/recipes-kernel/linux/linux-yocto.inc | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/meta/recipes-kernel/linux/linux-yocto.inc b/meta/recipes-kernel/linux/linux-yocto.inc
index a85280990b43..5ec1b1ba7023 100644
--- a/meta/recipes-kernel/linux/linux-yocto.inc
+++ b/meta/recipes-kernel/linux/linux-yocto.inc
@@ -56,3 +56,13 @@ do_install_append(){
# enable kernel-sample for oeqa/runtime/cases's ksample.py test
KERNEL_FEATURES_append_qemuall=" features/kernel-sample/kernel-sample.scc"
+
+# For linux-yocto-dev, the existing v3.17 based premirror tarball is of no
+# value, and for the non-dev variants, we don't want to be going off and
+# w-getting the 3+ gigabyte (!!!) mirror tarball either, when we've got
+# a shared v5.10 repository already. We can delete this in the future once
+# these old tarballs are gone. But really we may never bother to mirror
+# either of them ever again, since they are (a) now small, and (b) leaf
+# nodes at the end of the chain where the content is regularly changing.
+PREMIRRORS = ""
+MIRRORS = ""
--
2.25.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* Re: [PATCH 21/21] kernel: disable (pre)mirror for linux-yocto and linux-yocto-dev
2021-04-02 17:15 ` [PATCH 21/21] kernel: disable (pre)mirror for linux-yocto and linux-yocto-dev Paul Gortmaker
@ 2021-04-02 20:19 ` Bruce Ashfield
0 siblings, 0 replies; 29+ messages in thread
From: Bruce Ashfield @ 2021-04-02 20:19 UTC (permalink / raw)
To: Paul Gortmaker; +Cc: Richard Purdie, Linux Yocto, bitbake-devel
On Fri, Apr 2, 2021 at 1:16 PM Paul Gortmaker
<paul.gortmaker@windriver.com> wrote:
>
> As per the comment in the file itself, we have to explicitly disable
> (pre)mirror support for both kernel repositories, or else the fetcher
> will unhelpfully volunteer to wget gigs of what amounts to largely
> useless content that we don't want.
>
> Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
> ---
> meta/recipes-kernel/linux/linux-yocto.inc | 10 ++++++++++
> 1 file changed, 10 insertions(+)
>
> diff --git a/meta/recipes-kernel/linux/linux-yocto.inc b/meta/recipes-kernel/linux/linux-yocto.inc
> index a85280990b43..5ec1b1ba7023 100644
> --- a/meta/recipes-kernel/linux/linux-yocto.inc
> +++ b/meta/recipes-kernel/linux/linux-yocto.inc
> @@ -56,3 +56,13 @@ do_install_append(){
>
> # enable kernel-sample for oeqa/runtime/cases's ksample.py test
> KERNEL_FEATURES_append_qemuall=" features/kernel-sample/kernel-sample.scc"
> +
> +# For linux-yocto-dev, the existing v3.17 based premirror tarball is of no
> +# value, and for the non-dev variants, we don't want to be going off and
> +# w-getting the 3+ gigabyte (!!!) mirror tarball either, when we've got
> +# a shared v5.10 repository already. We can delete this in the future once
> +# these old tarballs are gone. But really we may never bother to mirror
> +# either of them ever again, since they are (a) now small, and (b) leaf
> +# nodes at the end of the chain where the content is regularly changing.
> +PREMIRRORS = ""
> +MIRRORS = ""
I suppose these are already set somewhere else, so a ?= won't actually
override the defaults.
I was just trying to think of a way that someone might want to turn it
back on, and they'd have to use an override in a bbappend to do it ..
but I suppose that isn't any nastier than would be required with ?=
Bruce
> --
> 2.25.1
>
--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
` (20 preceding siblings ...)
2021-04-02 17:15 ` [PATCH 21/21] kernel: disable (pre)mirror for linux-yocto and linux-yocto-dev Paul Gortmaker
@ 2021-04-02 22:14 ` Richard Purdie
2021-04-03 1:44 ` Paul Gortmaker
21 siblings, 1 reply; 29+ messages in thread
From: Richard Purdie @ 2021-04-02 22:14 UTC (permalink / raw)
To: Paul Gortmaker, Bruce Ashfield; +Cc: linux-yocto, bitbake-devel
On Fri, 2021-04-02 at 13:15 -0400, Paul Gortmaker wrote:
> Next Steps:
> -----------
> With this being a functional implementation, it seems like a good time to
> get other people looking at it. Ideally step #1 will be getting general
> agreement that this is something we need, something that is overdue,
> and that the implementation as shown here makes sense in the absence of
> any similar effort from anyone that does the same but in a better way.
>
> From there, we'll want more people not just looking at it, but testing it
> as well. I know I want to write a commit (script?) that will avoid any
> "transition tax" by prepopulating new repos with "old" already downloaded
> git objects where we can. And to add/do tests with my own popluated mirror
> and NO_NETWORK, and also try to ensure nothing in BB_SHALLOW gets upset,
> but I wasn't going to hold up starting a review of this any longer.
>
> I suspect I can get some co-workers using/testing it too, but Yocto gets
> used in a bunch of different ways by different groups, so we'll no doubt
> have to do some additional fixups to ensure everybody gets the benefits of
> this sharing. But I'm hopeful that when people see the benefits above,
> they'll pitch in to help take this the final mile by ensuring it works for
> their use case as well.
>
> I'm not too worried about pontificating out beyond that until we get past
> the acceptance/testing hurdles outlined above. So, please do have a read
> of the commits, kick the tires, put on your bikeshedding clothes and grab
> a brush, and lets see where it goes from here...
>
> https://github.com/paulgortmaker/poky/compare/reference-RC1
I've only briefly looked through this but the more I look, the more worried
I'm getting which is never a good sign.
I totally agree with the use case and agree we need to do something in this
area. I'm not sure the implementation looks right though, its clever but I
don't think its very usable to end users.
The warning signs to me are:
a) Needing new/confusing "fetch only" recipes
b) A large number of new options and variables to the fetcher
c) Needing recipes to change and people to migrate, potentially with scripts
between old and new
d) Variable namespacing needs work
I'm very worried this confuses up the git fetcher code and nobody will be able
to tell what is going on any more :/.
What I'd envisaged was git urls having something like a "mainline-linux-kernel"
tag added in the url as a parameter and a table somewhere which meant the checkout
for this would share git objects in a common pool. There would be a variable
mapping that name to git.kernel.org/pub/scm/linux/kernel/git somewhere but that
should be all that is needed.
No, this wouldn't cover 100% of artefacts but it should cover a majority of them
and be much simpler for users to comprehend. I haven't gone into this in detail
and perhaps I'm missing some problem that prevents it? :/
The other issue here is there are no tests. The bitbake fetcher code is one of
the few pieces of the project where we do have a fairly complete test suite and
we don't add things there without tests (see bitbake-selftest and lib/bb/tests/).
Cheers,
Richard
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos
2021-04-02 22:14 ` [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Richard Purdie
@ 2021-04-03 1:44 ` Paul Gortmaker
2021-04-03 8:33 ` Richard Purdie
0 siblings, 1 reply; 29+ messages in thread
From: Paul Gortmaker @ 2021-04-03 1:44 UTC (permalink / raw)
To: Richard Purdie; +Cc: Bruce Ashfield, linux-yocto, bitbake-devel
[Re: [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos] On 02/04/2021 (Fri 23:14) Richard Purdie wrote:
> On Fri, 2021-04-02 at 13:15 -0400, Paul Gortmaker wrote:
> > Next Steps:
> > -----------
> > With this being a functional implementation, it seems like a good time to
> > get other people looking at it. Ideally step #1 will be getting general
> > agreement that this is something we need, something that is overdue,
> > and that the implementation as shown here makes sense in the absence of
> > any similar effort from anyone that does the same but in a better way.
> >
> > From there, we'll want more people not just looking at it, but testing it
> > as well. I know I want to write a commit (script?) that will avoid any
> > "transition tax" by prepopulating new repos with "old" already downloaded
> > git objects where we can. And to add/do tests with my own popluated mirror
> > and NO_NETWORK, and also try to ensure nothing in BB_SHALLOW gets upset,
> > but I wasn't going to hold up starting a review of this any longer.
> >
> > I suspect I can get some co-workers using/testing it too, but Yocto gets
> > used in a bunch of different ways by different groups, so we'll no doubt
> > have to do some additional fixups to ensure everybody gets the benefits of
> > this sharing. But I'm hopeful that when people see the benefits above,
> > they'll pitch in to help take this the final mile by ensuring it works for
> > their use case as well.
> >
> > I'm not too worried about pontificating out beyond that until we get past
> > the acceptance/testing hurdles outlined above. So, please do have a read
> > of the commits, kick the tires, put on your bikeshedding clothes and grab
> > a brush, and lets see where it goes from here...
> >
> > https://github.com/paulgortmaker/poky/compare/reference-RC1
>
> I've only briefly looked through this but the more I look, the more worried
> I'm getting which is never a good sign.
Well, firstly thanks for the quick response, and I hope you'll give it
another look when you have more time, because I'm not quite convinced
I've conveyed how it is to be used in a way that got through as intended.
> I totally agree with the use case and agree we need to do something in this
> area. I'm not sure the implementation looks right though, its clever but I
> don't think its very usable to end users.
Not sure who you consider "end users". I don't really consider myself
or Bruce or anyone else crafting up a kernel recipe from scratch as an
end user. But even if an "end user" wants to do that, everything here
is opt-in. If you want to go clone 3G of stuff from mysuperkernel.org
in your new recipe, then feel free. Nothing stops them.
> The warning signs to me are:
>
> a) Needing new/confusing "fetch only" recipes
Not sure how they are confusing - and not really new either; I stole the
idea of zeroing out all the tasks from kernel-devsrc if I recall...
> b) A large number of new options and variables to the fetcher
I'd disagree with the characterization of "large" - but if you do have
the time, please go look over the commit logs for each again. They
all clearly indicate a necessary use case we don't handle well (or at
all) -- and we'll have to solve them one way or another. I'm open to
considering alternate solutions for any of them.
> c) Needing recipes to change and people to migrate, potentially with scripts
> between old and new
It feels like we are looking at two different things. The users or
"people" don't really migrate and the only people changing things would
be a small subset of people who maintain kernel recipes - which is a
small group to begin with. Recycling old git objects from existing
downloads would most likey be done in a transparent and hands free way.
> d) Variable namespacing needs work
OK - if given specifics it can and will be addressed.
> I'm very worried this confuses up the git fetcher code and nobody will be able
> to tell what is going on any more :/.
That kind of surprises me, to hear that after having looked at the shallow clone
changes to the fetcher, but again - I'm listening - what can we change
in the fetcher? What is off limits? The "pool" idea below doesn't address
the fact that "--mirror" is simply not viable for some repos, and that
is hard-coded into the fetcher. Similar for the other fetcher changes.
And the "pool" idea is still going to have to teach the fetcher about
some kind of "--reference" because that *has* to be added at fetch/clone
time and not as some later afterthought post-fetch.
I could simply take over do_fetch for the kernel and leave the fetcher
untouched, but when the problems being solved weren't really kernel
specific, that didn't seem like the right approach.
> What I'd envisaged was git urls having something like a "mainline-linux-kernel"
> tag added in the url as a parameter and a table somewhere which meant the checkout
> for this would share git objects in a common pool. There would be a variable
> mapping that name to git.kernel.org/pub/scm/linux/kernel/git somewhere but that
> should be all that is needed.
I saw this idea floated in the earlier thread that Randy pointed me at,
but I just don't see it being viable or the right solution even if one
could make it work. How do you decide what lives in the pool? If it is
just mainline, then it really isn't a pool. Are we only solving for the
kernel, or are we making a solution for any repo that is unweildy? What
triggers populating the pool? Do you stuff all mainline and -rt and all
of stable in the pool? Or just chunks of rt and stable? If you aren't
tracking what is in there via encapsulating chunks somehow, how do you
expire stuff from the pool? If it is one giant blob covering mainline
and linux-yocto objects, with no fetch-on-demand properties, then we are
back to the same old 3G-downloads-from-the-mirror problem. How do you
split that monster? The devil is in the details.
> No, this wouldn't cover 100% of artefacts but it should cover a majority of them
> and be much simpler for users to comprehend. I haven't gone into this in detail
> and perhaps I'm missing some problem that prevents it? :/
Well, I've thought about this a bunch, and I didn't come up with a whole
bunch of options to choose from that deliver what this delivers.
A mainline-only kernel-specific non-pool "pool" would be better than
nothing, but I'd still want to hear you flesh out how you thought it
would look and operate before I went out and tried to guess what you had
in mind.
Also, I'm still confused by this use of "users" - what is here all
happens transparently in the background and users aren't aware of it any
more than they are aware of what the fetcher is currently doing today.
> The other issue here is there are no tests. The bitbake fetcher code is one of
> the few pieces of the project where we do have a fairly complete test suite and
> we don't add things there without tests (see bitbake-selftest and lib/bb/tests/).
Yes, of course; and I've been told that by several people in advance.
But as you can imagine, I wasn't going to write tests without 1st
floating the general concept as an RFC and seeing how that went.
Anyway, thanks for the initial scan and I do hope you have a chance to
revisit it in more detail and consider the various problems it solves.
Paul.
--
>
> Cheers,
>
> Richard
>
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos
2021-04-03 1:44 ` Paul Gortmaker
@ 2021-04-03 8:33 ` Richard Purdie
0 siblings, 0 replies; 29+ messages in thread
From: Richard Purdie @ 2021-04-03 8:33 UTC (permalink / raw)
To: Paul Gortmaker; +Cc: Bruce Ashfield, linux-yocto, bitbake-devel
On Fri, 2021-04-02 at 21:44 -0400, Paul Gortmaker wrote:
> [Re: [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos] On 02/04/2021 (Fri 23:14) Richard Purdie wrote:
>
> > On Fri, 2021-04-02 at 13:15 -0400, Paul Gortmaker wrote:
> > > Next Steps:
> > > -----------
> > > With this being a functional implementation, it seems like a good time to
> > > get other people looking at it. Ideally step #1 will be getting general
> > > agreement that this is something we need, something that is overdue,
> > > and that the implementation as shown here makes sense in the absence of
> > > any similar effort from anyone that does the same but in a better way.
> > >
> > > From there, we'll want more people not just looking at it, but testing it
> > > as well. I know I want to write a commit (script?) that will avoid any
> > > "transition tax" by prepopulating new repos with "old" already downloaded
> > > git objects where we can. And to add/do tests with my own popluated mirror
> > > and NO_NETWORK, and also try to ensure nothing in BB_SHALLOW gets upset,
> > > but I wasn't going to hold up starting a review of this any longer.
> > >
> > > I suspect I can get some co-workers using/testing it too, but Yocto gets
> > > used in a bunch of different ways by different groups, so we'll no doubt
> > > have to do some additional fixups to ensure everybody gets the benefits of
> > > this sharing. But I'm hopeful that when people see the benefits above,
> > > they'll pitch in to help take this the final mile by ensuring it works for
> > > their use case as well.
> > >
> > > I'm not too worried about pontificating out beyond that until we get past
> > > the acceptance/testing hurdles outlined above. So, please do have a read
> > > of the commits, kick the tires, put on your bikeshedding clothes and grab
> > > a brush, and lets see where it goes from here...
> > >
> > > https://github.com/paulgortmaker/poky/compare/reference-RC1
> >
> > I've only briefly looked through this but the more I look, the more worried
> > I'm getting which is never a good sign.
>
> Well, firstly thanks for the quick response, and I hope you'll give it
> another look when you have more time, because I'm not quite convinced
> I've conveyed how it is to be used in a way that got through as intended.
>
> > I totally agree with the use case and agree we need to do something in this
> > area. I'm not sure the implementation looks right though, its clever but I
> > don't think its very usable to end users.
>
> Not sure who you consider "end users". I don't really consider myself
> or Bruce or anyone else crafting up a kernel recipe from scratch as an
> end user.
With the bitbake fetcher you are exposing an API. Anyone using that API is an
end user of that API. You and Bruce are, as is anyone writing a git:// url into
a recipe. I've drilled down into one of the changes in a separate mail to try
and give a bit more of an idea about what I mean about usability of the API.
Yes, we can just add variables and hardcoded bits of git commandlines but we have
been here before with the old bitbake fetcher v1 code. As soon as you start bolting
in extra variables, things start to interact really badly. That change already
breaks mirroring in ways which isn't obvious even to you/Bruce.
This why I care quite a lot about how we expose the controls and we try and do it
in a way which gives you the capabilities you want/need but in a way which allows
us to change the fetcher in the future, allow people to understand what the fetcher
is doing from the configuration in the url and have decent tests.
> But even if an "end user" wants to do that, everything here is opt-in. If you
> want to go clone 3G of stuff from mysuperkernel.org in your new recipe, then feel
> free. Nothing stops them.
I'm not asking this to be made kernel only. I do want a generic API which everyone
can use. Ideally I want one which all end users and not just you+Bruce understand
too and part of that is making the user visible API simple and intuitive so everyone
can get in on the action. If that makes the fetcher a little more complex behind the
scenes, that is ok, we do ask for tests for that.
> > The warning signs to me are:
> >
> > a) Needing new/confusing "fetch only" recipes
>
> Not sure how they are confusing - and not really new either; I stole the
> idea of zeroing out all the tasks from kernel-devsrc if I recall...
I think you misunderstand me. I don't mean the recipes are confusing, I don't
want them to exist at all. Consider these two ways of enabling alternates:
a) The user needs to add a new fetch-only recipe, update the SRC_URI, depend on
the fetch only recipe, set several other variables
b) The user adds a parameter to the SRC_URI specifying the series it matches
and if not already there, adds an entry to the alternates mirror list
I would argue b) is a lot simpler to the end user and will allow more people
to join in using it.
> > b) A large number of new options and variables to the fetcher
>
> I'd disagree with the characterization of "large" - but if you do have
> the time, please go look over the commit logs for each again.
You added two variables, GITCLONEARGS and GITFETCHREFS and four new parameters
(dlname, static, altref, packref). The only variables currently in use in the
fetcher are global behaviour ones, not url specific. There are currently 8
documented parameters.
I've tried to explain why the variables worry me and why we don't see them in the
fetcher at all. A 50% increase in parameters is a large increase.
> They all clearly indicate a necessary use case we don't handle well (or at
> all) -- and we'll have to solve them one way or another. I'm open to
> considering alternate solutions for any of them.
>
> > c) Needing recipes to change and people to migrate, potentially with scripts
> > between old and new
>
> It feels like we are looking at two different things. The users or
> "people" don't really migrate and the only people changing things would
> be a small subset of people who maintain kernel recipes - which is a
> small group to begin with. Recycling old git objects from existing
> downloads would most likey be done in a transparent and hands free way.
>
> > d) Variable namespacing needs work
>
> OK - if given specifics it can and will be addressed.
>
> > I'm very worried this confuses up the git fetcher code and nobody will be able
> > to tell what is going on any more :/.
>
> That kind of surprises me, to hear that after having looked at the shallow clone
> changes to the fetcher, but again - I'm listening - what can we change
> in the fetcher? What is off limits? The "pool" idea below doesn't address
> the fact that "--mirror" is simply not viable for some repos, and that
> is hard-coded into the fetcher. Similar for the other fetcher changes.
> And the "pool" idea is still going to have to teach the fetcher about
> some kind of "--reference" because that *has* to be added at fetch/clone
> time and not as some later afterthought post-fetch.
>
> I could simply take over do_fetch for the kernel and leave the fetcher
> untouched, but when the problems being solved weren't really kernel
> specific, that didn't seem like the right approach.
I think we're thinking about this from two different approaches. You're taking a
design of what you want to happen and then pushing the fetcher around to do what
you need. I think we need to look at how the user would configure and use this,
then work backwards to what the fetcher needs to do to support that.
There are a lot of moving parts here and it would help to simplify things down
to not try and solve every problem in one go. I'm going to focus on the alternates
issue here. This doesn't mean I don't understand the desire to be more efficient
about clones for example, I just need to start somewhere and see if we can move
forward.
The current way this is configured in your series is:
KREF = "git.kernel.org.preempt-rt.linux-5.4.y"
GITFETCHREFS_machine += " refs/heads/v5.4/*:refs/heads/v5.4/*"
GITCLONEARGS_machine = "--bare --single-branch --branch v5.4/base"
SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;name=machine;branch=${KBRANCH};altref=${KREF} \
git://git.yoctoproject.org/yocto-kernel-cache;type=kmeta;name=meta;branch=yocto-5.4;destsuffix=${KMETA}"
do_fetch[depends] += "linux-rt-5.4:do_fetch"
and what I'd much prefer is something simpler like:
SRC_URI = "git://git.yoctoproject.org/linux-yocto.git;name=machine;branch=${KBRANCH};altref=preempt-rt-linux-kernel \
git://git.yoctoproject.org/yocto-kernel-cache;type=kmeta;name=meta;branch=yocto-5.4;destsuffix=${KMETA}"
BB_GIT_ALTREF_URLMAP[preempt-rt-linux-kernel] = "git.kernel.org.preempt-rt.linux-5.4.y"
Ignoring the lack of efficiency of the cloning bit, the usability win if we
could do that is that anyone can start to benefit from the mirroring, just by
adding a single extra parameter to their git url.
The knowledge of the upstream clone repo can be encoded into one variable,
centrally which everyone can use and I'd imagine we'd quickly build up a decent set
of mirror urls which everyone would build around. Its easy to extend/share.
I'd also note the variable is for central config to the fetcher, it isn't url
specific and it is namespaced as bitbake bitbake and git fetcher (which sounds
trivial but it is a huge problem in the wider code).
Now I'd defined how I'd like the user to see it, the question is can we implement
that? I understand why you added extra 'fetch only' recipes, they are there
basically to allow you to fetch a different url. I think it shouldn't be too
painful to actually have the fetcher fetch that other url behind the scenes.
The gitsm fetcher (for all I dislike it) already does this.
The idea here would therefore be to teach the git fetcher that when it finds
configuration like this, it would look for and update a altref repo first
before it would then make its own clone.
Does that sound reasonable in principle at least? You'd agree that if we can,
dropping the need for the "fetch-only" recipes would be good, ideally?
> > What I'd envisaged was git urls having something like a "mainline-linux-kernel"
> > tag added in the url as a parameter and a table somewhere which meant the checkout
> > for this would share git objects in a common pool. There would be a variable
> > mapping that name to git.kernel.org/pub/scm/linux/kernel/git somewhere but that
> > should be all that is needed.
>
> I saw this idea floated in the earlier thread that Randy pointed me at,
> but I just don't see it being viable or the right solution even if one
> could make it work. How do you decide what lives in the pool? If it is
> just mainline, then it really isn't a pool. Are we only solving for the
> kernel, or are we making a solution for any repo that is unweildy? What
> triggers populating the pool? Do you stuff all mainline and -rt and all
> of stable in the pool? Or just chunks of rt and stable? If you aren't
> tracking what is in there via encapsulating chunks somehow, how do you
> expire stuff from the pool? If it is one giant blob covering mainline
> and linux-yocto objects, with no fetch-on-demand properties, then we are
> back to the same old 3G-downloads-from-the-mirror problem. How do you
> split that monster? The devil is in the details.
Firstly, there isn't one pool, there are multiple "pools" each with some
namespace. I was thinking they'd just track that particular upstream repo
and the name of the pool would indicate what you'd expect to find there.
Basically its like your altref parameter but I'm abstracting away their
creation/maintenance to be more magic.
The nice thing about altref "pools" is that they don't have mirroring
constraints but lets try and come back to revision/branch/object filtering
once we get the above discussed.
> > No, this wouldn't cover 100% of artefacts but it should cover a majority of them
> > and be much simpler for users to comprehend. I haven't gone into this in detail
> > and perhaps I'm missing some problem that prevents it? :/
>
> Well, I've thought about this a bunch, and I didn't come up with a whole
> bunch of options to choose from that deliver what this delivers.
>
> A mainline-only kernel-specific non-pool "pool" would be better than
> nothing, but I'd still want to hear you flesh out how you thought it
> would look and operate before I went out and tried to guess what you had
> in mind.
>
> Also, I'm still confused by this use of "users" - what is here all
> happens transparently in the background and users aren't aware of it any
> more than they are aware of what the fetcher is currently doing today.
See if I managed to explain it better above.
> > The other issue here is there are no tests. The bitbake fetcher code is one of
> > the few pieces of the project where we do have a fairly complete test suite and
> > we don't add things there without tests (see bitbake-selftest and lib/bb/tests/).
>
> Yes, of course; and I've been told that by several people in advance.
> But as you can imagine, I wasn't going to write tests without 1st
> floating the general concept as an RFC and seeing how that went.
You could have mentioned that you planned to add them :)
> Anyway, thanks for the initial scan and I do hope you have a chance to
> revisit it in more detail and consider the various problems it solves.
It was late and I clearly didn't convey what I'm thinking in that reply
or in the previous thread. I've tried to do so more above, I did spend
half the night with this going around in my head and I've spent quite
some time trying to write it down this morning. See if that makes
my concerns clearer and whether what I'm thinking could be made to work.
Cheers,
Richard
^ permalink raw reply [flat|nested] 29+ messages in thread