[PATCH] CI: add CI_CLEAN_REPOS variable to allow cleaning the repo reference cache

[PATCH] CI: add CI_CLEAN_REPOS variable to allow cleaning the repo reference cache

[Ross Burton]

If the repository reference directory gets corrupted it's not easy to
wipe it, so add a variable CI_CLEAN_REPOS that if set in the pipeline
will clean the clones and re-fetch them.

Also, stop the fetch from detaching during the garbage collection, just
in case it was a long-running GC that got killed that caused the
corruption in the first place.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 .gitlab-ci.yml  | 3 ++-
 ci/update-repos | 8 +++++++-
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 9c52b106..28d0cc19 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -58,7 +58,8 @@ stages:
       - $CI_PROJECT_DIR/work/build/tmp/work*/**/testimage/*
 
 #
-# Prep stage, update repositories once
+# Prep stage, update repositories once.
+# Set the CI variable CI_CLEAN_REPOS=1 to refetch the respositories from scratch
 #
 update-repos:
   extends: .setup
diff --git a/ci/update-repos b/ci/update-repos
index 91ff1975..9487102d 100755
--- a/ci/update-repos
+++ b/ci/update-repos
@@ -4,6 +4,7 @@
 
 import sys
 import os
+import shutil
 import subprocess
 import pathlib
 
@@ -34,9 +35,14 @@ if __name__ == "__main__":
 
     for repo in repositories:
         repodir = base_repodir / repo_shortname(repo)
+
+        if "CI_CLEAN_REPOS" in os.environ:
+            print("Cleaning %s..." % repo)
+            shutil.rmtree(repodir, ignore_errors=True)
+
         if repodir.exists():
             print("Updating %s..." % repo)
-            subprocess.run(["git", "-C", repodir, "fetch"], check=True)
+            subprocess.run(["git", "-C", repodir, "-c", "gc.autoDetach=false", "fetch"], check=True)
         else:
             print("Cloning %s..." % repo)
             subprocess.run(["git", "clone", "--bare", repo, repodir], check=True)
-- 
2.34.1

Re: [PATCH] CI: add CI_CLEAN_REPOS variable to allow cleaning the repo reference cache

[Jon Mason]

On Mon, 27 Mar 2023 17:52:14 +0100, Ross Burton wrote:
> If the repository reference directory gets corrupted it's not easy to
> wipe it, so add a variable CI_CLEAN_REPOS that if set in the pipeline
> will clean the clones and re-fetch them.
> 
> Also, stop the fetch from detaching during the garbage collection, just
> in case it was a long-running GC that got killed that caused the
> corruption in the first place.

Applied, thanks!

[1/1] CI: add CI_CLEAN_REPOS variable to allow cleaning the repo reference cache
      commit: 2db12df861fda0896f196a101572d6300bb4cc78

Best regards,
-- 
Jon Mason <jon.mason@arm.com>