[PATCH v2 0/2] Capture git submodules in mirror archiver

[PATCH v2 0/2] Capture git submodules in mirror archiver

[Paul Barker]

Changes in v2:

  * Use the git-submodule-test repo on git.yoctoproject.com instead of
    ovmf so that the test doesn't take forever.

  * Fix test_archiver_mode_mirror_gitsm_shallow. As the
    git-submodule-test recipe isn't likely to change we can be explicit
    with file names instead of using globs.

Paul Barker (2):
  selftest: git-submodule-test: New recipe for testing a gitsm SRC_URI
  archiver: Capture git submodules in mirror archiver

 .../git-submodule-test/git-submodule-test.bb  |  8 +++
 meta/classes/archiver.bbclass                 | 13 +++--
 meta/lib/oeqa/selftest/cases/archiver.py      | 54 +++++++++++++++++++
 3 files changed, 68 insertions(+), 7 deletions(-)
 create mode 100644 meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb

-- 
2.26.2

[PATCH v2 1/2] selftest: git-submodule-test: New recipe for testing a gitsm SRC_URI

[Paul Barker]

Signed-off-by: Paul Barker <pbarker@konsulko.com>
---
 .../recipes-test/git-submodule-test/git-submodule-test.bb | 8 ++++++++
 1 file changed, 8 insertions(+)
 create mode 100644 meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb

diff --git a/meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb b/meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb
new file mode 100644
index 0000000000..08089b4186
--- /dev/null
+++ b/meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb
@@ -0,0 +1,8 @@
+SUMMARY = "Test recipe for fetching git submodules"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI = "gitsm://git.yoctoproject.org/git-submodule-test"
+SRCREV = "a2885dd7d25380d23627e7544b7bbb55014b16ee"
-- 
2.26.2

[PATCH v2 2/2] archiver: Capture git submodules in mirror archiver

[Paul Barker]

Using the new Fetch.expanded_urldata() function we can get URL data for
all git submodules.

Signed-off-by: Paul Barker <pbarker@konsulko.com>
---
 meta/classes/archiver.bbclass            | 13 +++---
 meta/lib/oeqa/selftest/cases/archiver.py | 54 ++++++++++++++++++++++++
 2 files changed, 60 insertions(+), 7 deletions(-)

diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
index 780c562b68..c2c049c343 100644
--- a/meta/classes/archiver.bbclass
+++ b/meta/classes/archiver.bbclass
@@ -345,13 +345,12 @@ python do_ar_mirror() {
 
     fetcher = bb.fetch2.Fetch(src_uri, d)
 
-    for url in fetcher.urls:
-        if is_excluded(url):
-            bb.note('Skipping excluded url: %s' % (url))
+    for ud in fetcher.expanded_urldata():
+        if is_excluded(ud.url):
+            bb.note('Skipping excluded url: %s' % (ud.url))
             continue
 
-        bb.note('Archiving url: %s' % (url))
-        ud = fetcher.ud[url]
+        bb.note('Archiving url: %s' % (ud.url))
         ud.setup_localpath(d)
         localpath = None
 
@@ -367,7 +366,7 @@ python do_ar_mirror() {
         if len(ud.mirrortarballs) and not localpath:
             bb.warn('Mirror tarballs are listed for a source but none are present. ' \
                     'Falling back to original download.\n' \
-                    'SRC_URI = %s' % (url))
+                    'SRC_URI = %s' % (ud.url))
 
         # Check original download
         if not localpath:
@@ -376,7 +375,7 @@ python do_ar_mirror() {
 
         if not localpath or not os.path.exists(localpath):
             bb.fatal('Original download is missing for a source.\n' \
-                        'SRC_URI = %s' % (url))
+                        'SRC_URI = %s' % (ud.url))
 
         # We now have an appropriate localpath
         bb.note('Copying source mirror')
diff --git a/meta/lib/oeqa/selftest/cases/archiver.py b/meta/lib/oeqa/selftest/cases/archiver.py
index bc5447d2a3..37efed57c8 100644
--- a/meta/lib/oeqa/selftest/cases/archiver.py
+++ b/meta/lib/oeqa/selftest/cases/archiver.py
@@ -255,3 +255,57 @@ class Archiver(OESelftestTestCase):
             glob_str = os.path.join(bb_vars['DEPLOY_DIR_SRC'], 'mirror', target_file_name)
             glob_result = glob.glob(glob_str)
             self.assertTrue(glob_result, 'Missing archive file %s' % (target_file_name))
+
+    def test_archiver_mode_mirror_gitsm(self):
+        """
+        Test that the archiver correctly handles git submodules with
+        `ARCHIVER_MODE[src] = "mirror"`.
+        """
+        features = 'INHERIT += "archiver"\n'
+        features += 'ARCHIVER_MODE[src] = "mirror"\n'
+        features += 'ARCHIVER_MODE[mirror] = "combined"\n'
+        features += 'BB_GENERATE_MIRROR_TARBALLS = "1"\n'
+        features += 'COPYLEFT_LICENSE_INCLUDE = "*"\n'
+        self.write_config(features)
+
+        bitbake('-c clean git-submodule-test')
+        bitbake('-c deploy_archives -f git-submodule-test')
+
+        bb_vars = get_bb_vars(['DEPLOY_DIR_SRC'])
+        for target_file_name in [
+            'git2_git.yoctoproject.org.git-submodule-test.tar.gz',
+            'git2_git.yoctoproject.org.bitbake-gitsm-test1.tar.gz',
+            'git2_git.yoctoproject.org.bitbake-gitsm-test2.tar.gz',
+            'git2_git.openembedded.org.bitbake.tar.gz'
+        ]:
+            target_path = os.path.join(bb_vars['DEPLOY_DIR_SRC'], 'mirror', target_file_name)
+            self.assertTrue(os.path.exists(target_path))
+
+    def test_archiver_mode_mirror_gitsm_shallow(self):
+        """
+        Test that the archiver correctly handles git submodules with
+        `ARCHIVER_MODE[src] = "mirror"`.
+        """
+        features = 'INHERIT += "archiver"\n'
+        features += 'ARCHIVER_MODE[src] = "mirror"\n'
+        features += 'ARCHIVER_MODE[mirror] = "combined"\n'
+        features += 'BB_GENERATE_MIRROR_TARBALLS = "1"\n'
+        features += 'COPYLEFT_LICENSE_INCLUDE = "*"\n'
+        features += 'BB_GIT_SHALLOW = "1"\n'
+        features += 'BB_GENERATE_SHALLOW_TARBALLS = "1"\n'
+        features += 'DL_DIR = "${TOPDIR}/downloads-shallow"\n'
+        self.write_config(features)
+
+        bitbake('-c clean git-submodule-test')
+        bitbake('-c deploy_archives -f git-submodule-test')
+
+        bb_vars = get_bb_vars(['DEPLOY_DIR_SRC'])
+        for target_file_name in [
+            'gitsmshallow_git.yoctoproject.org.git-submodule-test_a2885dd-1_master.tar.gz',
+            'gitsmshallow_git.yoctoproject.org.bitbake-gitsm-test1_bare_120f4c7-1.tar.gz',
+            'gitsmshallow_git.yoctoproject.org.bitbake-gitsm-test2_bare_f66699e-1.tar.gz',
+            'gitsmshallow_git.openembedded.org.bitbake_bare_52a144a-1.tar.gz',
+            'gitsmshallow_git.openembedded.org.bitbake_bare_c39b997-1.tar.gz'
+        ]:
+            target_path = os.path.join(bb_vars['DEPLOY_DIR_SRC'], 'mirror', target_file_name)
+            self.assertTrue(os.path.exists(target_path))
-- 
2.26.2

Re: [OE-core] [PATCH v2 1/2] selftest: git-submodule-test: New recipe for testing a gitsm SRC_URI

[Richard Purdie]

Hi Paul,

On Thu, 2020-06-04 at 18:33 +0100, Paul Barker wrote:
> Signed-off-by: Paul Barker <pbarker@konsulko.com>
> ---
>  .../recipes-test/git-submodule-test/git-submodule-test.bb | 8 ++++++++
>  1 file changed, 8 insertions(+)
>  create mode 100644 meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb
> 
> diff --git a/meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb b/meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb
> new file mode 100644
> index 0000000000..08089b4186
> --- /dev/null
> +++ b/meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb
> @@ -0,0 +1,8 @@
> +SUMMARY = "Test recipe for fetching git submodules"
> +LICENSE = "MIT"
> +LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
> +
> +INHIBIT_DEFAULT_DEPS = "1"
> +
> +SRC_URI = "gitsm://git.yoctoproject.org/git-submodule-test"
> +SRCREV = "a2885dd7d25380d23627e7544b7bbb55014b16ee"

I didn't spot it at first but unfortunately this recipe is breaking the
autobuilder. I think there is a problem with gitsm:// modules not
mirroring correctly.

Here you see the a-quick build run:

bitbake universe -c fetch -k
(meta-selftest is included and SOURCE_MIRROR_FETCH = '1' set)

https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/1055/steps/10/logs/step1b

you do see do_fetch called for this recipe. Then later in the build it
does:

oe-selftest -r buildoptions.SourceMirroring.test_yocto_source_mirror

https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/1055/steps/12/logs/step1d

The intent is that we check YP can build from its mirrors. The test is
say it cannot.

Any ideas on what might be wrong?

Cheers,

Richard

Re: [OE-core] [PATCH v2 1/2] selftest: git-submodule-test: New recipe for testing a gitsm SRC_URI

[Paul Barker]

On Fri, 12 Jun 2020 at 08:34, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> Hi Paul,
>
> On Thu, 2020-06-04 at 18:33 +0100, Paul Barker wrote:
> > Signed-off-by: Paul Barker <pbarker@konsulko.com>
> > ---
> >  .../recipes-test/git-submodule-test/git-submodule-test.bb | 8 ++++++++
> >  1 file changed, 8 insertions(+)
> >  create mode 100644 meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb
> >
> > diff --git a/meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb b/meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb
> > new file mode 100644
> > index 0000000000..08089b4186
> > --- /dev/null
> > +++ b/meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb
> > @@ -0,0 +1,8 @@
> > +SUMMARY = "Test recipe for fetching git submodules"
> > +LICENSE = "MIT"
> > +LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
> > +
> > +INHIBIT_DEFAULT_DEPS = "1"
> > +
> > +SRC_URI = "gitsm://git.yoctoproject.org/git-submodule-test"
> > +SRCREV = "a2885dd7d25380d23627e7544b7bbb55014b16ee"
>
> I didn't spot it at first but unfortunately this recipe is breaking the
> autobuilder. I think there is a problem with gitsm:// modules not
> mirroring correctly.
>
> Here you see the a-quick build run:
>
> bitbake universe -c fetch -k
> (meta-selftest is included and SOURCE_MIRROR_FETCH = '1' set)
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/1055/steps/10/logs/step1b
>
> you do see do_fetch called for this recipe. Then later in the build it
> does:
>
> oe-selftest -r buildoptions.SourceMirroring.test_yocto_source_mirror
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/1055/steps/12/logs/step1d
>
> The intent is that we check YP can build from its mirrors. The test is
> say it cannot.
>
> Any ideas on what might be wrong?

ovmf also uses git submodules and that seems to work. Also, I can see
the relevant mirror tarballs are present on
downloads.yoctoproject.org:

http://downloads.yoctoproject.org/mirror/sources/git2_git.openembedded.org.bitbake.tar.gz
http://downloads.yoctoproject.org/mirror/sources/git2_git.yoctoproject.org.bitbake-gitsm-test1.tar.gz
http://downloads.yoctoproject.org/mirror/sources/git2_git.yoctoproject.org.bitbake-gitsm-test2.tar.gz
http://downloads.yoctoproject.org/mirror/sources/git2_git.yoctoproject.org.git-submodule-test.tar.gz

So it's not immediately obvious what's wrong there. I'll run that
selftest case locally and see if I can reproduce the issue.

Thanks,

-- 
Paul Barker
Konsulko Group