[PATCH 1/2] CI: maintain centralised repositories to speed fetching

* [PATCH 1/2] CI: maintain centralised repositories to speed fetching
@ 2021-03-30 16:43 Ross Burton
  2021-03-30 16:43 ` [PATCH 2/2] CI: move utility tasks to prep so they don't cause a fetch Ross Burton
  2021-03-31 12:57 ` [meta-arm] [PATCH 1/2] CI: maintain centralised repositories to speed fetching Jon Mason
  0 siblings, 2 replies; 3+ messages in thread
From: Ross Burton @ 2021-03-30 16:43 UTC (permalink / raw)
  To: meta-arm

Kas supports 'reference repositories' which are used as a local source
of git objects to speed up fetches.  At the beginning of the CI run we
can update these repositories once to speed up later fetches.

Change-Id: I138051fd3cf9b5675e0fa5007cd8088abd17db4e
Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 .gitlab-ci.yml   | 18 +++++++++++++++---
 kas/update-repos | 41 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+), 3 deletions(-)
 create mode 100755 kas/update-repos

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 4e247de..faf655d 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -3,6 +3,7 @@ image: $CI_REGISTRY/$CI_PROJECT_NAMESPACE/yocto-builder:master
 
   # First do a common bootstrap, and then build all the targets
 stages:
+  - prep
   - bootstrap
   - build
 
@@ -11,6 +12,7 @@ stages:
   stage: build
   variables:
     KAS_WORK_DIR: $CI_PROJECT_DIR/work
+    KAS_REPO_REF_DIR: $CI_BUILDS_DIR/persist/repos
     SSTATE_DIR: $CI_BUILDS_DIR/persist/sstate
     DL_DIR: $CI_BUILDS_DIR/persist/downloads
     BB_LOGCONFIG: $CI_PROJECT_DIR/kas/logging.yml
@@ -18,7 +20,7 @@ stages:
     - echo KAS_WORK_DIR = $KAS_WORK_DIR
     - echo SSTATE_DIR = $SSTATE_DIR
     - echo DL_DIR = $DL_DIR
-    - mkdir --verbose --parents $KAS_WORK_DIR $SSTATE_DIR $DL_DIR
+    - mkdir --verbose --parents $KAS_WORK_DIR $KAS_REPO_REF_DIR $SSTATE_DIR $DL_DIR
 
 # Generalised fragment to do a Kas build
 .build:
@@ -31,7 +33,17 @@ stages:
 
 
 #
-# First phase, bootstrap and machine coverage
+# Prep stage, update repositories once
+#
+update-repos:
+  extends: .setup
+  stage: prep
+  script:
+  - flock --verbose --timeout 60 $KAS_REPO_REF_DIR ./kas/update-repos
+
+
+#
+# Bootstrap stage, bootstrap and machine coverage
 #
 
 # Build a number of native tools first to ensure the other builders don't race
@@ -49,7 +61,7 @@ machine-coverage:
 
 
 #
-# Second phase, the actual build jobs
+# Build stage, the actual build jobs
 #
 
 corstone500:
diff --git a/kas/update-repos b/kas/update-repos
new file mode 100755
index 0000000..48711e3
--- /dev/null
+++ b/kas/update-repos
@@ -0,0 +1,41 @@
+#! /usr/bin/env python3
+
+# Update clones of the repositories we need in KAS_REPO_REF_DIR to speed up fetches
+
+import sys
+import os
+import subprocess
+import pathlib
+
+def repo_shortname(url):
+    # Taken from Kas (Repo.__getattr__) to ensure the logic is right
+    from urllib.parse import urlparse
+    url = urlparse(url)
+    return ('{url.netloc}{url.path}'
+            .format(url=url)
+            .replace('@', '.')
+            .replace(':', '.')
+            .replace('/', '.')
+            .replace('*', '.'))
+
+repositories = (
+    "https://git.yoctoproject.org/git/poky",
+    "https://git.openembedded.org/meta-openembedded",
+    "https://git.yoctoproject.org/git/meta-virtualization",
+    "https://git.yoctoproject.org/git/meta-zephyr",
+    "https://github.com/kraj/meta-clang",
+)
+
+if __name__ == "__main__":
+    if "KAS_REPO_REF_DIR" not in os.environ:
+        print("KAS_REPO_REF_DIR needs to be set")
+        sys.exit(1)
+
+    base_repodir = pathlib.Path(os.environ["KAS_REPO_REF_DIR"])
+
+    for repo in repositories:
+        repodir = base_repodir / repo_shortname(repo)
+        if repodir.exists():
+            subprocess.run(["git", "-C", repodir, "fetch"], check=True)
+        else:
+            subprocess.run(["git", "clone", "--bare", repo, repodir], check=True)
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 3+ messages in thread