[PATCH] externalsrc: remove git submodule--helper list

[PATCH] externalsrc: remove git submodule--helper list

[Jacob Creedon]

This removes a call to an undocumented and now deprecated subcommand of
git submodule--helper and replaces it with a suppoorted one that gives an
equivalent list.

Signed-off-by: Jacob Creedon <jcreedon@google.com>
---
 meta/classes/externalsrc.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 3d6b80bee2..0401c75fa9 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -217,7 +217,7 @@ def srctree_hash_files(d, srcdir=None):
             env['GIT_INDEX_FILE'] = tmp_index.name
             subprocess.check_output(['git', 'add', '-A', '.'],
cwd=s_dir, env=env)
             git_sha1 = subprocess.check_output(['git', 'write-tree'],
cwd=s_dir, env=env).decode("utf-8")
-            submodule_helper = subprocess.check_output(['git',
'submodule--helper', 'list'], cwd=s_dir, env=env).decode("utf-8")
+            submodule_helper = subprocess.check_output(['git',
'submodule', '--quiet', 'foreach', 'echo $sm_path'], cwd=s_dir,
env=env).decode("utf-8")
             for line in submodule_helper.splitlines():
                 module_dir = os.path.join(s_dir, line.rsplit(maxsplit=1)[1])
                 if os.path.isdir(module_dir):
--
2.37.3

Re: [OE-core] [PATCH] externalsrc: remove git submodule--helper list

[Quentin Schulz]

Hi Jacob,

On 9/16/22 01:06, Jacob Creedon via lists.openembedded.org wrote:
> This removes a call to an undocumented and now deprecated subcommand of
> git submodule--helper and replaces it with a suppoorted one that gives an
> equivalent list.
> 
> Signed-off-by: Jacob Creedon <jcreedon@google.com>
> ---
>   meta/classes/externalsrc.bbclass | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
> index 3d6b80bee2..0401c75fa9 100644
> --- a/meta/classes/externalsrc.bbclass
> +++ b/meta/classes/externalsrc.bbclass
> @@ -217,7 +217,7 @@ def srctree_hash_files(d, srcdir=None):
>               env['GIT_INDEX_FILE'] = tmp_index.name
>               subprocess.check_output(['git', 'add', '-A', '.'],
> cwd=s_dir, env=env)
>               git_sha1 = subprocess.check_output(['git', 'write-tree'],
> cwd=s_dir, env=env).decode("utf-8")
> -            submodule_helper = subprocess.check_output(['git',
> 'submodule--helper', 'list'], cwd=s_dir, env=env).decode("utf-8")
> +            submodule_helper = subprocess.check_output(['git',
> 'submodule', '--quiet', 'foreach', 'echo $sm_path'], cwd=s_dir,
> env=env).decode("utf-8")

When was this added to git? We require version 1.8.3.1 or later right 
now. We would need to bump the requirement (and need to make sure that 
all supported distros actually have this git version available in their 
package feed) and update the documentation accordingly 
(https://git.yoctoproject.org/yocto-docs/tree/documentation/poky.yaml.in#n46) 
if it's not supported in git 1.8.3.1.

Cheers,
Quentin

Re: [OE-core] [PATCH] externalsrc: remove git submodule--helper list

[Richard Purdie]

On Thu, 2022-09-15 at 16:06 -0700, Jacob Creedon via lists.openembedded.org wrote:
> This removes a call to an undocumented and now deprecated subcommand of
> git submodule--helper and replaces it with a suppoorted one that gives an
> equivalent list.
> 
> Signed-off-by: Jacob Creedon <jcreedon@google.com>
> ---
>  meta/classes/externalsrc.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
> index 3d6b80bee2..0401c75fa9 100644
> --- a/meta/classes/externalsrc.bbclass
> +++ b/meta/classes/externalsrc.bbclass
> @@ -217,7 +217,7 @@ def srctree_hash_files(d, srcdir=None):
>              env['GIT_INDEX_FILE'] = tmp_index.name
>              subprocess.check_output(['git', 'add', '-A', '.'],
> cwd=s_dir, env=env)
>              git_sha1 = subprocess.check_output(['git', 'write-tree'],
> cwd=s_dir, env=env).decode("utf-8")
> -            submodule_helper = subprocess.check_output(['git',
> 'submodule--helper', 'list'], cwd=s_dir, env=env).decode("utf-8")
> +            submodule_helper = subprocess.check_output(['git',
> 'submodule', '--quiet', 'foreach', 'echo $sm_path'], cwd=s_dir,
> env=env).decode("utf-8")
>              for line in submodule_helper.splitlines():
>                  module_dir = os.path.join(s_dir, line.rsplit(maxsplit=1)[1])
>                  if os.path.isdir(module_dir):

I put this into our CI and it failed:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/4136
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/4079
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/4094
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/4150
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/132

which I could reproduce locally with:

"oe-selftest -r devtool.DevtoolAddTests.test_devtool_add_fetch_git"

I'd also note that your patch is line wrapped and your From address is
being mangled by the mailing list, probably due to DMARC issues. If you
could fix those for the new version that'd be great!

Cheers,

Richard

Re: [OE-core] [PATCH] externalsrc: remove git submodule--helper list

[Jacob Creedon]

> When was this added to git?

$sm_path was added in 2.19 (August 2018). Prior to that, $path was
used all the way back to when foreach was added to `git submodule` in
v1.6.1 (December 2008). I'm not sure I have the project level context
to decide whether or not we should be bumping the minimum git version,
or just shimming in a small script to check for $path vs. $sm_path. So
I'm open to guidance on the preferred solution here.

The impetus for this patch is that `submodule--helper list` was
recently removed from git in
https://github.com/git/git/commit/31955475d1c283120d5d84247eb3fd55d9f5fdd9.
I happened to run into this because I was running a bleeding edge
version of git. I suspect it will be a little while before this
version of git will hit distros.