[PATCH 0/2] externalsrc: Detect code changes in submodules

[PATCH 0/2] externalsrc: Detect code changes in submodules

[Douglas]

Further to 50ff9afb39, only detect code changes in submodules that are
subdirectories of the EXTERNALSRC directory.

The (undocumented) git submodule--helper returns a path
for each submodule relative to the top of the repo.
Don't add submodules that are not within our EXTERNALSRC subtree.

If we unpack one git repo inside another, like this:

    SRC_URI = "git://${GIT_SERVER}/repo1;name=repo1;destsuffix=repo1 \
               git://${GIT_SERVER}/repo2;name=repo2;destsuffix=repo1/repo2 \
               "

Git status reports, for repo1:

    Untracked files:
      (use "git add <file>..." to include in what will be committed)
    	repo2/

If we run `devtool modify` on this recipe, do_patch runs with:

    PATCHTOOL = "git"
    PATCH_COMMIT_FUNCTIONS = "1"

The `patch_task_postfunc` (patch.bbclass, line 82) runs a `git add .` on the
top-level repo1, leaving the checkout in an invalid state. The following git
warning does not appear in the log:

    $ git add .
    warning: adding embedded git repository: repo2
    hint: You've added another git repository inside your current repository.
    hint: Clones of the outer repository will not contain the contents of
    hint: the embedded repository and will not know how to obtain it.
    hint: If you meant to add a submodule, use:
    hint:
    hint: 	git submodule add <url> repo2
    hint:
    hint: If you added this path by mistake, you can remove it from the
    hint: index with:
    hint:
    hint: 	git rm --cached repo2
    hint:
    hint: See "git help submodule" for more information.

    $ git submodule status
    fatal: no submodule mapping found in .gitmodules for path 'repo2'

No further git submodule commands can be run on the checkout.

We could enhance the `patch_task_postfunc` to look for any embedded git
checkouts and add them as submodules, but this seems unnecessary complexity for
an obscure edge-case. Although the git repo is left in an invalid state with
respect to the submodules, it still serves the purpose required by devtool:
To take further commits, and generate patch files from them.

We are still able to run these commands to examine any submodules,
where git submodule--helper reports paths relative to the top of the checkout:

    $ git ls-files --stage | grep ^160000
    160000 5feee12d6e974dd8c0614cf5b593380b046439a5 0   repo2

    $ git submodule--helper list
    160000 5feee12d6e974dd8c0614cf5b593380b046439a5 0   repo2

When a recipe sets EXTERNALSRC to a subdirectory of the git checkout, we test
for the existence of the reported submodule paths within the EXTERNALSRC
directory.

The latest versions of git submodule--helper accept a path to a subdirectory and
correctly report no submodules within that subdirectory. Regrettably, we still
support git versions that don't accept a path to a subdirectory.

[YOCTO #14333]

Douglas Royds (2):
  Revert "externalsrc: Detect code changes in submodules"
  externalsrc: Detect code changes in submodules

 meta/classes/externalsrc.bbclass | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

-- 
2.25.1

[PATCH 1/2] Revert "externalsrc: Detect code changes in submodules"

[Douglas]

This reverts commit 4525310d49d115a37705f04ac5c03d639e5e8f8c.

Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
---
 meta/classes/externalsrc.bbclass | 16 +++++++---------
 1 file changed, 7 insertions(+), 9 deletions(-)

diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 54cc7edbae0..c7b2bf2f49a 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -217,16 +217,14 @@ def srctree_hash_files(d, srcdir=None):
             env['GIT_INDEX_FILE'] = tmp_index.name
             subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir, env=env)
             git_sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, env=env).decode("utf-8")
-            submodule_helper = subprocess.check_output(['git', 'submodule', 'status'], cwd=s_dir, env=env).decode("utf-8")
+            submodule_helper = subprocess.check_output(['git', 'submodule--helper', 'list'], cwd=s_dir, env=env).decode("utf-8")
             for line in submodule_helper.splitlines():
-                module_relpath = line.split()[1]
-                if not module_relpath.split('/')[0] == '..':
-                    module_dir = os.path.join(s_dir, module_relpath)
-                    proc = subprocess.Popen(['git', 'add', '-A', '.'], cwd=module_dir, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
-                    proc.communicate()
-                    proc = subprocess.Popen(['git', 'write-tree'], cwd=module_dir, env=env, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
-                    stdout, _ = proc.communicate()
-                    git_sha1 += stdout.decode("utf-8")
+                module_dir = os.path.join(s_dir, line.rsplit(maxsplit=1)[1])
+                proc = subprocess.Popen(['git', 'add', '-A', '.'], cwd=module_dir, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+                proc.communicate()
+                proc = subprocess.Popen(['git', 'write-tree'], cwd=module_dir, env=env, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
+                stdout, _ = proc.communicate()
+                git_sha1 += stdout.decode("utf-8")
             sha1 = hashlib.sha1(git_sha1.encode("utf-8")).hexdigest()
         with open(oe_hash_file, 'w') as fobj:
             fobj.write(sha1)
-- 
2.25.1

[PATCH 2/2] externalsrc: Detect code changes in submodules

[Douglas]

Further to 50ff9afb39, only detect code changes in submodules that are
subdirectories of the EXTERNALSRC directory.

The (undocumented) git submodule--helper returns a path
for each submodule relative to the top of the repo.
Don't add submodules that are not within our source subtree.

[YOCTO #14333]

Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
---
 meta/classes/externalsrc.bbclass | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index c7b2bf2f49a..3d6b80bee2b 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -220,11 +220,12 @@ def srctree_hash_files(d, srcdir=None):
             submodule_helper = subprocess.check_output(['git', 'submodule--helper', 'list'], cwd=s_dir, env=env).decode("utf-8")
             for line in submodule_helper.splitlines():
                 module_dir = os.path.join(s_dir, line.rsplit(maxsplit=1)[1])
-                proc = subprocess.Popen(['git', 'add', '-A', '.'], cwd=module_dir, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
-                proc.communicate()
-                proc = subprocess.Popen(['git', 'write-tree'], cwd=module_dir, env=env, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
-                stdout, _ = proc.communicate()
-                git_sha1 += stdout.decode("utf-8")
+                if os.path.isdir(module_dir):
+                    proc = subprocess.Popen(['git', 'add', '-A', '.'], cwd=module_dir, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+                    proc.communicate()
+                    proc = subprocess.Popen(['git', 'write-tree'], cwd=module_dir, env=env, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
+                    stdout, _ = proc.communicate()
+                    git_sha1 += stdout.decode("utf-8")
             sha1 = hashlib.sha1(git_sha1.encode("utf-8")).hexdigest()
         with open(oe_hash_file, 'w') as fobj:
             fobj.write(sha1)
-- 
2.25.1

Re: [OE-core] [PATCH 1/2] Revert "externalsrc: Detect code changes in submodules"

[Quentin Schulz]

Hi Douglas,

On Thu, Apr 08, 2021 at 01:08:54PM +1200, Douglas via lists.openembedded.org wrote:
> This reverts commit 4525310d49d115a37705f04ac5c03d639e5e8f8c.
> 

Why is this revert needed? The reason needs to be explicit in the commit
log.

Cheers,
Quentin