[PATCH v2] scripts/oe-setup-layers: Make efficiently idempotent

* [PATCH v2] scripts/oe-setup-layers: Make efficiently idempotent
@ 2023-01-17  1:59 Chuck Wolber
  2023-01-17 17:03 ` [OE-core] " Luca Ceresoli
  0 siblings, 1 reply; 5+ messages in thread
From: Chuck Wolber @ 2023-01-17  1:59 UTC (permalink / raw)
  To: openembedded-core; +Cc: Chuck Wolber

The effect of subsequent setup-layers executions is now either a NOOP
or the minimal set of changes required to ensure layers precisely match
the JSON configuration.

This change allows setup-layers to be incorporated into a team's
configuration management strategy. In particular, the configuration
JSON manages a "pinning policy" that documents the oversight of sources
of change (a requirement for embedded development in highly regulated
industries).

One model for this strategy would work as follows. Team level policy is
developed to regularly review upstream commits that occur between the
current upstream HEAD and the previously pinned revision. The JSON
configuration is periodically updated after a review, test, and approval
process. In the rare instance that an upstream change is considered
problematic, the bbappend mechanism can be used to make relevant
changes in the team's project repository. This approach also requires
that team developers regularly run the project repository copy of
setup-layers. This is most easily accomplished by including setup-layers
in a wrapper script that all team developers use to interact with the
bitbake tool suite (e.g. "bb bitbake foo-image"). Project level policy
and oversight is effectively "contained" within this wrapper script,
thereby reducing a significant source of human error.

Left unstated, but acknowledged here, are a number of nuances required
to successfully implement the above strategy.  The details are out of
scope for this explanation. What should be clear is that a larger
configuration management strategy can now benefit from the utility
provided by setup-layers.

Note: Neither the above configuration management strategy example nor
the change itself is intended to alter the original intent to use
"bitbake-layers create-layers-setup destdir" to keep pace with upstream
activity for those who wish to use it that way.

Signed-off-by: Chuck Wolber <chuck.wolber@boeing.com>
---
 scripts/oe-setup-layers | 55 +++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 46 insertions(+), 9 deletions(-)

diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers
index 6ecaffe..d0bc9f1 100755
--- a/scripts/oe-setup-layers
+++ b/scripts/oe-setup-layers
@@ -10,12 +10,42 @@
 # bitbake-layers create-layers-setup destdir
 #
 # It is recommended that you do not modify this file directly, but rather re-run the above command to get the freshest upstream copy.
+#
+# This script is idempotent. Subsequent runs only change what is necessary to
+# ensure your layers match your configuration.
 
 import argparse
 import json
 import os
 import subprocess
 
+def _is_layer_git_repo(layerdir):
+    git_dir = os.path.join(layerdir, ".git")
+    if not os.access(git_dir, os.R_OK):
+        return False
+    try:
+        return subprocess.check_output("git -C %s rev-parse --is-inside-git-dir" % git_dir, shell=True, stderr=subprocess.DEVNULL)
+    except subprocess.CalledProcessError:
+        return False
+
+def _is_layer_at_rev(layerdir, rev):
+    try:
+        curr_rev = subprocess.check_output("git -C %s rev-parse HEAD" % layerdir, shell=True, stderr=subprocess.DEVNULL)
+        if curr_rev.strip().decode("utf-8") == rev:
+            return True
+    except subprocess.CalledProcessError:
+        pass
+    return False
+
+def _is_layer_at_remote_uri(layerdir, remote, uri):
+    try:
+        curr_uri = subprocess.check_output("git -C %s remote get-url %s" % (layerdir, remote), shell=True, stderr=subprocess.DEVNULL)
+        if curr_uri.strip().decode("utf-8") == uri:
+            return True
+    except subprocess.CalledProcessError:
+        pass
+    return False
+
 def _do_checkout(args, json):
     layers = json['sources']
     for l_name in layers:
@@ -36,23 +66,30 @@ def _do_checkout(args, json):
         remotes = l_remote['remotes']
 
         print('\nSetting up source {}, revision {}, branch {}'.format(l_name, desc, branch))
-        cmd = 'git init -q {}'.format(layerdir)
-        print("Running '{}'".format(cmd))
-        subprocess.check_output(cmd, shell=True)
+        if not _is_layer_git_repo(layerdir):
+            cmd = 'git init -q {}'.format(layerdir)
+            print("Running '{}'".format(cmd))
+            subprocess.check_output(cmd, shell=True)
 
         for remote in remotes:
-            cmd = "git remote remove {} > /dev/null 2>&1; git remote add {} {}".format(remote, remote, remotes[remote]['uri'])
+            if not _is_layer_at_remote_uri(layerdir, remote, remotes[remote]['uri']):
+                cmd = "git remote remove {} > /dev/null 2>&1; git remote add {} {}".format(remote, remote, remotes[remote]['uri'])
+                print("Running '{}' in {}".format(cmd, layerdir))
+                subprocess.check_output(cmd, shell=True, cwd=layerdir)
+
+                cmd = "git fetch -q {} || true".format(remote)
+                print("Running '{}' in {}".format(cmd, layerdir))
+                subprocess.check_output(cmd, shell=True, cwd=layerdir)
+
+        if not _is_layer_at_rev(layerdir, rev):
+            cmd = "git fetch -q --all || true"
             print("Running '{}' in {}".format(cmd, layerdir))
             subprocess.check_output(cmd, shell=True, cwd=layerdir)
 
-            cmd = "git fetch -q {} || true".format(remote)
+            cmd = 'git checkout -q {}'.format(rev)
             print("Running '{}' in {}".format(cmd, layerdir))
             subprocess.check_output(cmd, shell=True, cwd=layerdir)
 
-        cmd = 'git checkout -q {}'.format(rev)
-        print("Running '{}' in {}".format(cmd, layerdir))
-        subprocess.check_output(cmd, shell=True, cwd=layerdir)
-
 parser = argparse.ArgumentParser(description="A self contained python script that fetches all the needed layers and sets them to correct revisions using data in a json format from a separate file. The json data can be created from an active build directory with 'bitbake-layers create-layers-setup destdir' and there's a sample file and a schema in meta/files/")
 
 parser.add_argument('--force-bootstraplayer-checkout', action='store_true',
-- 
1.8.3.1



^ permalink raw reply related	[flat|nested] 5+ messages in thread