All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Paul Gortmaker" <paul.gortmaker@windriver.com>
To: Bruce Ashfield <bruce.ashfield@gmail.com>,
	Richard Purdie <richard.purdie@linuxfoundation.org>
Cc: linux-yocto@lists.yoctoproject.org, bitbake-devel@lists.openembedded.org
Subject: [PATCH 03/21] bitbake: fetch2/git: allow optional git download name overrride
Date: Fri,  2 Apr 2021 13:15:39 -0400	[thread overview]
Message-ID: <20210402171557.981599-4-paul.gortmaker@windriver.com> (raw)
In-Reply-To: <20210402171557.981599-1-paul.gortmaker@windriver.com>

Single repositories containing multiple independent origin commits are
becoming more common.  For example the rt development repo has patches
as "raw" format-patch output in a patch series, and also for ease of
automated testing, them all pre-applied (ff and non-ff) to the original
baseline (v5.10.8 in this case):

   linux-rt-devel$ git merge-base linux-5.10.y-rt linux-5.10.y-rt-rebase
   7a1519a74f3d0b06598fb95387688cde41e465d8
   linux-rt-devel$ git describe 7a1519a74f3d0b06598fb95387688cde41e465d8
   v5.10.8
   linux-rt-devel$ git merge-base linux-5.10.y-rt linux-5.10.y-rt-patches
   linux-rt-devel$
   linux-rt-devel$ git log --oneline linux-5.10.y-rt | tail -n1
   1da177e4c3f4 Linux-2.6.12-rc2
   linux-rt-devel$ git log --oneline linux-5.10.y-rt-patches | tail -n1
   96f1ac50158c [ANNOUNCE] 4.1.2-rt1

As can be seen, there are two different "epoch" commits and no history
or shared objects exist between the patches branch and the applied
branches (on the linux-stable baseline).

Since we construct the download directory name based on the server name
and the server's path to the repository, we currently can't refactor that
content into two locally independent paths/repositories in the download
dir for efficiency, ease of use and tarball/MIRROR handling.

To resolve this, we allow the recipe to optionally specify the path
component of the download dir name, so that a server with a repo of N
independent branches/histories can be used in N different recipes (or N
different SRC_URI lines) without overlap or name space collision in the
download dir.  Continuing with the above example, we could have:

  RT_REPO = "git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git"
  SRC_URI = "${RT_REPO};name=applied;dlname=.preempt-rt.linux-5.10.y \
             ${RT_REPO};name=patches;dlname=.preempt-rt.patches-5.10"

...resulting in the following independent repository dirs:
     downloads/git2/git.kernel.org.preempt-rt.linux-5.10.y
     downloads/git2/git.kernel.org.preempt-rt.patches-5.10

If the new "dlname" parameter is not specified, then the automatic name
generation functions just as it always has done and this change has no
effect on any existing download dirs.

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
 .../bitbake-user-manual-fetching.rst              |  6 ++++++
 bitbake/lib/bb/fetch2/git.py                      | 15 ++++++++++++++-
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
index e9a5f336dfd2..0f12eaa3a81f 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
@@ -417,6 +417,12 @@ This fetcher supports the following parameters:
    is assumed to be "master". The number of branch parameters much match
    the number of name parameters.
 
+-  *"dlname":* The download name used in downloads/git2 (or GITDIR).  If
+    specified,  this will override the automatically constructed name
+    component from "thispath" for the URL git://some.host/thispath to the
+    user specified value.  The component from "some.host" remains unaffected.
+    Allows splitting of one repo into multiple independent download dirs.
+
 -  *"rev":* The revision to use for the checkout. The default is
    "master".
 
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index b54ec76d7174..de698a3d9f24 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -18,6 +18,13 @@ Supported SRC_URI options are:
    SRCREV_nameX = "xxxxxxxxxxxxxxxxxxxx"
    SRCREV_nameY = "YYYYYYYYYYYYYYYYYYYY"
 
+- dlname
+   The download name used in downloads/git2 (or GITDIR).  This will override
+   the automatically constructed name component from "thispath" for the URL
+   git://some.host/thispath to the user specified value.  The component from
+   "some.host" remains unaffected.  Allows splitting of one repo into multiple
+   independent download dirs.
+
 - tag
     The git tag to retrieve. The default is "master"
 
@@ -151,6 +158,8 @@ class Git(FetchMethod):
 
         ud.nobranch = ud.parm.get("nobranch","0") == "1"
 
+        ud.dlname = ud.parm.get("dlname","")
+
         # usehead implies nobranch
         ud.usehead = ud.parm.get("usehead","0") == "1"
         if ud.usehead:
@@ -242,7 +251,11 @@ class Git(FetchMethod):
                     ud.unresolvedrev[name] = ud.revisions[name]
                 ud.revisions[name] = self.latest_revision(ud, d, name)
 
-        gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_'))
+        if ud.dlname:
+            gitdlname = ud.dlname.replace('/', '.').replace('*', '.').replace(' ','_')
+        else:
+            gitdlname = ud.path.replace('/', '.').replace('*', '.').replace(' ','_')
+        gitsrcname = '%s%s' % (ud.host.replace(':', '.'), gitdlname)
         if gitsrcname.startswith('.'):
             gitsrcname = gitsrcname[1:]
 
-- 
2.25.1


  parent reply	other threads:[~2021-04-02 17:16 UTC|newest]

Thread overview: 29+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-04-02 17:15 [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Paul Gortmaker
2021-04-02 17:15 ` [PATCH 01/21] bitbake: fetch2/git: allow override of clone args with GITCLONEARGS Paul Gortmaker
2021-04-02 17:15 ` [PATCH 02/21] bitbake: fetch2/git: allow limiting upstream fetch refs to a subset Paul Gortmaker
2021-04-03  7:43   ` Richard Purdie
2021-04-02 17:15 ` Paul Gortmaker [this message]
2021-04-02 17:15 ` [PATCH 04/21] bitbake: fetch2/git: allow specifying repos as static/unchanging Paul Gortmaker
2021-04-02 17:15 ` [PATCH 05/21] bitbake: fetch2/git: ensure static repos have at least one refs/heads Paul Gortmaker
2021-04-02 17:15 ` [PATCH 06/21] bitbake: fetch2/git: allow alt references within download dir Paul Gortmaker
2021-04-02 17:15 ` [PATCH 07/21] bitbake: fetch2/git: append new altref line if/when SRC_URI changed value Paul Gortmaker
2021-04-02 17:15 ` [PATCH 08/21] bitbake: fetch2/git: allow pack references within download dir Paul Gortmaker
2021-04-02 17:15 ` [PATCH 09/21] bitbake: fetch2/git: use constant names for packs in static repos Paul Gortmaker
2021-04-02 17:15 ` [PATCH 10/21] kernel: add basic boilerplate for fetch-only recipes Paul Gortmaker
2021-04-02 17:15 ` [PATCH 11/21] kernel: add a fetch-only recipe for mainline v5.10 source Paul Gortmaker
2021-04-02 20:13   ` Bruce Ashfield
2021-04-02 17:15 ` [PATCH 12/21] kernel: allow splitting mainline v5.10 source download in two Paul Gortmaker
2021-04-02 17:15 ` [PATCH 13/21] kernel: allow splitting mainline v5.10 source download in three Paul Gortmaker
2021-04-02 17:15 ` [PATCH 14/21] kernel: allow splitting mainline v5.10 source download in four Paul Gortmaker
2021-04-02 17:15 ` [PATCH 15/21] kernel: add recipe for linux-master (mainline latest) Paul Gortmaker
2021-04-02 20:16   ` Bruce Ashfield
2021-04-02 17:15 ` [PATCH 16/21] kernel: add stable fetch recipes for v5.4.x, v5.10.x and v5.12.x Paul Gortmaker
2021-04-02 17:15 ` [PATCH 17/21] kernel: add preempt-rt fetch recipes for v5.4.x, v5.10.x and 5.12.x Paul Gortmaker
2021-04-02 17:15 ` [PATCH 18/21] kernel: make v5.4.x Yocto recipes use shared source Paul Gortmaker
2021-04-02 17:15 ` [PATCH 19/21] kernel: make v5.10.x " Paul Gortmaker
2021-04-02 17:15 ` [PATCH 20/21] kernel: make linux-yocto-dev recipe " Paul Gortmaker
2021-04-02 17:15 ` [PATCH 21/21] kernel: disable (pre)mirror for linux-yocto and linux-yocto-dev Paul Gortmaker
2021-04-02 20:19   ` Bruce Ashfield
2021-04-02 22:14 ` [PATCH RFC 00/21] Git repository sharing for kernel (and other) repos Richard Purdie
2021-04-03  1:44   ` Paul Gortmaker
2021-04-03  8:33     ` Richard Purdie

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210402171557.981599-4-paul.gortmaker@windriver.com \
    --to=paul.gortmaker@windriver.com \
    --cc=bitbake-devel@lists.openembedded.org \
    --cc=bruce.ashfield@gmail.com \
    --cc=linux-yocto@lists.yoctoproject.org \
    --cc=richard.purdie@linuxfoundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.