[PATCH 0/7] Add symlinks for git (shallow) tarballs when name differs due to mirror rewrite rules

[PATCH 0/7] Add symlinks for git (shallow) tarballs when name differs due to mirror rewrite rules

[Urs Fässler]

The git (shallow) tarballs are not found when certain mirror rewrite rules are
in use. This is due to the fact that the tarball is named according to the url
where it is downloaded from. The error occurs in the unpack step where BitBake
looks for a tarball named after the url as specified in the recipe. By adding
the symlinks BitBake finds the tarballs in the unpack step.

This is an alternative to the patches "Always use the url specified in the
recipe as a base for the git shallow tarball naming" from the 23.07.2018. This
solution is more aligned to the current implementation of BitBake.

Regards
Urs

Urs Fässler (7):
  fetch2/git: add tests to capture existing behavior wrt. naming of
    clone directories
  fetch2/git: add tests to capture existing behavior wrt. naming of
    mirror tarball
  fetch2/git: add tests to capture existing behavior wrt. naming of git
    shallow tarball
  fetch2/git: use intention revealing names for premirror tests
  fetch2: extract the function which ensures that a valid symlink exists
  fetch2: add symlinks for mirror tarballs
  fetch2: add symlink for shallow mirror tarballs

 lib/bb/fetch2/__init__.py |  31 +++--
 lib/bb/tests/fetch.py     | 235 ++++++++++++++++++++++++++++++++++++--
 2 files changed, 246 insertions(+), 20 deletions(-)

-- 
2.19.1

[PATCH 1/7] fetch2/git: add tests to capture existing behavior wrt. naming of clone directories

[Urs Fässler]

The mapping of the URLs to the local directory is not obvious. For easier
understanding, we add this tests to explicitly showing the mapping.

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
---
 lib/bb/tests/fetch.py | 49 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)

diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index ff66315a..53595312 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -463,6 +463,55 @@ class MirrorUriTest(FetcherTest):
                                 'https://BBBB/B/B/B/bitbake/bitbake-1.0.tar.gz',
                                 'http://AAAA/A/A/A/B/B/bitbake/bitbake-1.0.tar.gz'])
 
+
+class GitDownloadDirectoryNamingTest(FetcherTest):
+    def setUp(self):
+        super(GitDownloadDirectoryNamingTest, self).setUp()
+        self.recipe_url = "git://git.openembedded.org/bitbake"
+        self.recipe_dir = "git.openembedded.org.bitbake"
+        self.mirror_url = "git://github.com/openembedded/bitbake.git"
+        self.mirror_dir = "github.com.openembedded.bitbake.git"
+
+        self.d.setVar('SRCREV', '82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40')
+
+    def setup_mirror_rewrite(self):
+        self.d.setVar("PREMIRRORS", self.recipe_url + " " + self.mirror_url + " \n")
+
+    @skipIfNoNetwork()
+    def test_that_directory_is_named_after_recipe_url_when_no_mirroring_is_used(self):
+        self.setup_mirror_rewrite()
+        fetcher = bb.fetch.Fetch([self.recipe_url], self.d)
+
+        fetcher.download()
+
+        dir = os.listdir(self.dldir + "/git2")
+        self.assertIn(self.recipe_dir, dir)
+
+    @skipIfNoNetwork()
+    def test_that_directory_exists_for_mirrored_url_and_recipe_url_when_mirroring_is_used(self):
+        self.setup_mirror_rewrite()
+        fetcher = bb.fetch.Fetch([self.recipe_url], self.d)
+
+        fetcher.download()
+
+        dir = os.listdir(self.dldir + "/git2")
+        self.assertIn(self.mirror_dir, dir)
+        self.assertIn(self.recipe_dir, dir)
+
+    @skipIfNoNetwork()
+    def test_that_recipe_directory_and_mirrored_directory_exists_when_mirroring_is_used_and_the_mirrored_directory_already_exists(self):
+        self.setup_mirror_rewrite()
+        fetcher = bb.fetch.Fetch([self.mirror_url], self.d)
+        fetcher.download()
+        fetcher = bb.fetch.Fetch([self.recipe_url], self.d)
+
+        fetcher.download()
+
+        dir = os.listdir(self.dldir + "/git2")
+        self.assertIn(self.mirror_dir, dir)
+        self.assertIn(self.recipe_dir, dir)
+
+
 class FetcherLocalTest(FetcherTest):
     def setUp(self):
         def touch(fn):
-- 
2.19.1

[PATCH 3/7] fetch2/git: add tests to capture existing behavior wrt. naming of git shallow tarball

[Urs Fässler]

The mapping of the URLs to the local shallow tarballs is not obvious. For
easier understanding, we add this tests to explicitly showing the mapping.

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
---
 lib/bb/tests/fetch.py | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index dcc4d8ce..79be1be0 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -546,6 +546,41 @@ class TarballNamingTest(FetcherTest):
         self.assertIn(self.mirror_tarball, dir)
 
 
+class GitShallowTarballNamingTest(FetcherTest):
+    def setUp(self):
+        super(GitShallowTarballNamingTest, self).setUp()
+        self.recipe_url = "git://git.openembedded.org/bitbake"
+        self.recipe_tarball = "gitshallow_git.openembedded.org.bitbake_82ea737-1_master.tar.gz"
+        self.mirror_url = "git://github.com/openembedded/bitbake.git"
+        self.mirror_tarball = "gitshallow_github.com.openembedded.bitbake.git_82ea737-1_master.tar.gz"
+
+        self.d.setVar('BB_GIT_SHALLOW', '1')
+        self.d.setVar('BB_GENERATE_SHALLOW_TARBALLS', '1')
+        self.d.setVar('SRCREV', '82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40')
+
+    def setup_mirror_rewrite(self):
+        self.d.setVar("PREMIRRORS", self.recipe_url + " " + self.mirror_url + " \n")
+
+    @skipIfNoNetwork()
+    def test_that_the_tarball_is_named_after_recipe_url_when_no_mirroring_is_used(self):
+        fetcher = bb.fetch.Fetch([self.recipe_url], self.d)
+
+        fetcher.download()
+
+        dir = os.listdir(self.dldir)
+        self.assertIn(self.recipe_tarball, dir)
+
+    @skipIfNoNetwork()
+    def test_that_the_mirror_tarball_is_created_when_mirroring_is_used(self):
+        self.setup_mirror_rewrite()
+        fetcher = bb.fetch.Fetch([self.recipe_url], self.d)
+
+        fetcher.download()
+
+        dir = os.listdir(self.dldir)
+        self.assertIn(self.mirror_tarball, dir)
+
+
 class FetcherLocalTest(FetcherTest):
     def setUp(self):
         def touch(fn):
-- 
2.19.1

[PATCH 2/7] fetch2/git: add tests to capture existing behavior wrt. naming of mirror tarball

[Urs Fässler]

The mapping of the URLs to the local tarballs is not obvious. For easier
understanding, we add this tests to explicitly showing the mapping.

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
---
 lib/bb/tests/fetch.py | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 53595312..dcc4d8ce 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -512,6 +512,40 @@ class GitDownloadDirectoryNamingTest(FetcherTest):
         self.assertIn(self.recipe_dir, dir)
 
 
+class TarballNamingTest(FetcherTest):
+    def setUp(self):
+        super(TarballNamingTest, self).setUp()
+        self.recipe_url = "git://git.openembedded.org/bitbake"
+        self.recipe_tarball = "git2_git.openembedded.org.bitbake.tar.gz"
+        self.mirror_url = "git://github.com/openembedded/bitbake.git"
+        self.mirror_tarball = "git2_github.com.openembedded.bitbake.git.tar.gz"
+
+        self.d.setVar('BB_GENERATE_MIRROR_TARBALLS', '1')
+        self.d.setVar('SRCREV', '82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40')
+
+    def setup_mirror_rewrite(self):
+        self.d.setVar("PREMIRRORS", self.recipe_url + " " + self.mirror_url + " \n")
+
+    @skipIfNoNetwork()
+    def test_that_the_recipe_tarball_is_created_when_no_mirroring_is_used(self):
+        fetcher = bb.fetch.Fetch([self.recipe_url], self.d)
+
+        fetcher.download()
+
+        dir = os.listdir(self.dldir)
+        self.assertIn(self.recipe_tarball, dir)
+
+    @skipIfNoNetwork()
+    def test_that_the_mirror_tarball_is_created_when_mirroring_is_used(self):
+        self.setup_mirror_rewrite()
+        fetcher = bb.fetch.Fetch([self.recipe_url], self.d)
+
+        fetcher.download()
+
+        dir = os.listdir(self.dldir)
+        self.assertIn(self.mirror_tarball, dir)
+
+
 class FetcherLocalTest(FetcherTest):
     def setUp(self):
         def touch(fn):
-- 
2.19.1

[PATCH 4/7] fetch2/git: use intention revealing names for premirror tests

[Urs Fässler]

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
---
 lib/bb/tests/fetch.py | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 79be1be0..9bed06ba 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -869,27 +869,27 @@ class FetcherNetworkTest(FetcherTest):
         self.assertRaises(bb.fetch.ParameterError, self.gitfetcher, url, url)
 
     @skipIfNoNetwork()
-    def test_gitfetch_premirror(self):
-        url1 = "git://git.openembedded.org/bitbake"
-        url2 = "git://someserver.org/bitbake"
+    def test_gitfetch_finds_local_tarball_for_mirrored_url_when_previous_downloaded_by_the_recipe_url(self):
+        recipeurl = "git://git.openembedded.org/bitbake"
+        mirrorurl = "git://someserver.org/bitbake"
         self.d.setVar("PREMIRRORS", "git://someserver.org/bitbake git://git.openembedded.org/bitbake \n")
-        self.gitfetcher(url1, url2)
+        self.gitfetcher(recipeurl, mirrorurl)
 
     @skipIfNoNetwork()
-    def test_gitfetch_premirror2(self):
-        url1 = url2 = "git://someserver.org/bitbake"
+    def test_gitfetch_finds_local_tarball_when_previous_downloaded_from_a_premirror(self):
+        recipeurl = "git://someserver.org/bitbake"
         self.d.setVar("PREMIRRORS", "git://someserver.org/bitbake git://git.openembedded.org/bitbake \n")
-        self.gitfetcher(url1, url2)
+        self.gitfetcher(recipeurl, recipeurl)
 
     @skipIfNoNetwork()
-    def test_gitfetch_premirror3(self):
+    def test_gitfetch_finds_local_repository_when_premirror_rewrites_the_recipe_url(self):
         realurl = "git://git.openembedded.org/bitbake"
-        dummyurl = "git://someserver.org/bitbake"
+        recipeurl = "git://someserver.org/bitbake"
         self.sourcedir = self.unpackdir.replace("unpacked", "sourcemirror.git")
         os.chdir(self.tempdir)
         bb.process.run("git clone %s %s 2> /dev/null" % (realurl, self.sourcedir), shell=True)
-        self.d.setVar("PREMIRRORS", "%s git://%s;protocol=file \n" % (dummyurl, self.sourcedir))
-        self.gitfetcher(dummyurl, dummyurl)
+        self.d.setVar("PREMIRRORS", "%s git://%s;protocol=file \n" % (recipeurl, self.sourcedir))
+        self.gitfetcher(recipeurl, recipeurl)
 
     @skipIfNoNetwork()
     def test_git_submodule(self):
-- 
2.19.1

[PATCH 5/7] fetch2: extract the function which ensures that a valid symlink exists

[Urs Fässler]

For better readability and future use, we extract the function which
ensures that a given symlink exists.

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
---
 lib/bb/fetch2/__init__.py | 27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index a83526a5..458f3107 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -1016,16 +1016,7 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
                     origud.method.build_mirror_data(origud, ld)
             return origud.localpath
         # Otherwise the result is a local file:// and we symlink to it
-        if not os.path.exists(origud.localpath):
-            if os.path.islink(origud.localpath):
-                # Broken symbolic link
-                os.unlink(origud.localpath)
-
-            # As per above, in case two tasks end up here simultaneously.
-            try:
-                os.symlink(ud.localpath, origud.localpath)
-            except FileExistsError:
-                pass
+        ensure_symlink(ud.localpath, origud.localpath)
         update_stamp(origud, ld)
         return ud.localpath
 
@@ -1059,6 +1050,22 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
             bb.utils.unlockfile(lf)
 
 
+def ensure_symlink(target, link_name):
+    if not os.path.exists(link_name):
+        if os.path.islink(link_name):
+            # Broken symbolic link
+            os.unlink(link_name)
+
+        # In case this is executing without any file locks held (as is
+        # the case for file:// URLs), two tasks may end up here at the
+        # same time, in which case we do not want the second task to
+        # fail when the link has already been created by the first task.
+        try:
+            os.symlink(target, link_name)
+        except FileExistsError:
+            pass
+
+
 def try_mirrors(fetch, d, origud, mirrors, check = False):
     """
     Try to use a mirrored version of the sources.
-- 
2.19.1

[PATCH 6/7] fetch2: add symlinks for mirror tarballs

[Urs Fässler]

In the unpack step the mirror tarball is not found when the mirrored
tarball is named differently than the original tarball. This happens
due to mirror rewrite rules. To solve the problem we add symlinks for
the differently named tarballs.

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
---
 lib/bb/fetch2/__init__.py |  4 +++
 lib/bb/tests/fetch.py     | 53 ++++++++++++++++++++++++++++++++++++++-
 2 files changed, 56 insertions(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 458f3107..a1b03a95 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -1017,6 +1017,10 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
             return origud.localpath
         # Otherwise the result is a local file:// and we symlink to it
         ensure_symlink(ud.localpath, origud.localpath)
+
+        if os.path.exists(ud.fullmirror):
+            ensure_symlink(ud.fullmirror, origud.fullmirror)
+
         update_stamp(origud, ld)
         return ud.localpath
 
diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 9bed06ba..1cd4c65c 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -29,6 +29,20 @@ from bb.fetch2 import URI
 from bb.fetch2 import FetchMethod
 import bb
 
+
+def get_broken_symlinks(directory):
+    result = []
+
+    dir = os.listdir(directory)
+    for filename in dir:
+        fullname = directory + '/' + filename
+        broken = (not os.path.exists(fullname)) and os.path.islink(fullname)
+        if broken:
+            result += [filename]
+
+    return result
+
+
 def skipIfNoNetwork():
     if os.environ.get("BB_SKIP_NETTESTS") == "yes":
         return unittest.skip("Network tests being skipped")
@@ -536,7 +550,7 @@ class TarballNamingTest(FetcherTest):
         self.assertIn(self.recipe_tarball, dir)
 
     @skipIfNoNetwork()
-    def test_that_the_mirror_tarball_is_created_when_mirroring_is_used(self):
+    def test_that_tarball_for_mirrored_url_and_recipe_url_exists_when_mirroring_is_used(self):
         self.setup_mirror_rewrite()
         fetcher = bb.fetch.Fetch([self.recipe_url], self.d)
 
@@ -544,6 +558,7 @@ class TarballNamingTest(FetcherTest):
 
         dir = os.listdir(self.dldir)
         self.assertIn(self.mirror_tarball, dir)
+        self.assertIn(self.recipe_tarball, dir)
 
 
 class GitShallowTarballNamingTest(FetcherTest):
@@ -881,6 +896,12 @@ class FetcherNetworkTest(FetcherTest):
         self.d.setVar("PREMIRRORS", "git://someserver.org/bitbake git://git.openembedded.org/bitbake \n")
         self.gitfetcher(recipeurl, recipeurl)
 
+    @skipIfNoNetwork()
+    def test_gitfetch_finds_local_tarball_when_previous_downloaded_from_a_mirror(self):
+        recipeurl = "git://git.wrong.org/bitbake"
+        self.d.setVar("MIRRORS", "git://git.wrong.org/bitbake git://git.openembedded.org/bitbake \n")
+        self.gitfetcher(recipeurl, recipeurl)
+
     @skipIfNoNetwork()
     def test_gitfetch_finds_local_repository_when_premirror_rewrites_the_recipe_url(self):
         realurl = "git://git.openembedded.org/bitbake"
@@ -891,6 +912,36 @@ class FetcherNetworkTest(FetcherTest):
         self.d.setVar("PREMIRRORS", "%s git://%s;protocol=file \n" % (recipeurl, self.sourcedir))
         self.gitfetcher(recipeurl, recipeurl)
 
+    @skipIfNoNetwork()
+    def test_gitfetch_uses_the_git_tarball_when_using_mirror_rewrite_rules_and_the_git_clone_does_not_exist(self):
+        self.d.setVar('SRCREV', 'e5939ff608b95cdd4d0ab0e1935781ab9a276ac0')
+        self.d.setVar('BB_GENERATE_MIRROR_TARBALLS', '1')
+        self.d.setVar('PREMIRRORS', 'git://git.yoctoproject.org/.* git://git.yoctoproject.org/git/PATH;protocol=https \n')
+        fetcher = bb.fetch.Fetch(["git://git.yoctoproject.org/fstests"], self.d)
+        fetcher.download()
+        bb.utils.prunedir(self.dldir + '/git2')
+        bb.utils.prunedir(self.unpackdir)
+        self.d.setVar('PREMIRRORS', '')
+        self.d.setVar('BB_FETCH_PREMIRRORONLY', '1')
+
+        fetcher.download()
+
+        fetcher.unpack(self.unpackdir)
+        dir = os.listdir(self.unpackdir + "/git/")
+        self.assertIn("fstests.doap", dir)
+
+    @skipIfNoNetwork()
+    def test_that_no_broken_symlinks_exist_when_using_mirror_rewrite_rules_and_no_tarball(self):
+        self.d.setVar('SRCREV', 'e5939ff608b95cdd4d0ab0e1935781ab9a276ac0')
+        self.d.setVar('BB_GENERATE_MIRROR_TARBALLS', '0')
+        self.d.setVar('PREMIRRORS', 'git://git.yoctoproject.org/.* git://git.yoctoproject.org/git/PATH;protocol=https \n')
+        fetcher = bb.fetch.Fetch(["git://git.yoctoproject.org/fstests"], self.d)
+
+        fetcher.download()
+
+        broken_symlinks = get_broken_symlinks(self.dldir)
+        self.assertEqual([], broken_symlinks)
+
     @skipIfNoNetwork()
     def test_git_submodule(self):
         fetcher = bb.fetch.Fetch(["gitsm://git.yoctoproject.org/git-submodule-test;rev=f12e57f2edf0aa534cf1616fa983d165a92b0842"], self.d)
-- 
2.19.1

[PATCH 7/7] fetch2: add symlink for shallow mirror tarballs

[Urs Fässler]

In the unpack step the shallow mirror tarball is not found when the
mirrored tarball is named differently than the original tarball. This
happens due to mirror rewrite rules. To solve the problem we add
symlinks for the differently named tarballs.

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
---
 lib/bb/fetch2/__init__.py |  2 ++
 lib/bb/tests/fetch.py     | 44 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index a1b03a95..6cbaf413 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -1020,6 +1020,8 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
 
         if os.path.exists(ud.fullmirror):
             ensure_symlink(ud.fullmirror, origud.fullmirror)
+        if ud.shallow and os.path.exists(ud.fullshallow):
+            ensure_symlink(ud.fullshallow, origud.fullshallow)
 
         update_stamp(origud, ld)
         return ud.localpath
diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 1cd4c65c..bad497b4 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -595,6 +595,21 @@ class GitShallowTarballNamingTest(FetcherTest):
         dir = os.listdir(self.dldir)
         self.assertIn(self.mirror_tarball, dir)
 
+    @skipIfNoNetwork()
+    def test_that_tarball_exists_for_mirrored_url_and_recipe_url_when_mirroring_is_used_and_the_mirrored_tarball_already_exists(self):
+        self.setup_mirror_rewrite()
+        fetcher = bb.fetch.Fetch([self.mirror_url], self.d)
+        fetcher.download()
+        bb.utils.prunedir(self.dldir + '/git2')
+        bb.utils.prunedir(self.unpackdir)
+        fetcher = bb.fetch.Fetch([self.recipe_url], self.d)
+
+        fetcher.download()
+
+        dir = os.listdir(self.dldir)
+        self.assertIn(self.mirror_tarball, dir)
+        self.assertIn(self.recipe_tarball, dir)
+
 
 class FetcherLocalTest(FetcherTest):
     def setUp(self):
@@ -1763,3 +1778,32 @@ class GitShallowTest(FetcherTest):
 
         dir = os.listdir(self.unpackdir + "/git/")
         self.assertIn("fstests.doap", dir)
+
+    @skipIfNoNetwork()
+    def test_that_unpack_uses_the_git_shallow_tarball_when_using_mirror_rewrite_rules_and_the_git_clone_does_not_exist(self):
+        self.d.setVar('SRCREV', 'e5939ff608b95cdd4d0ab0e1935781ab9a276ac0')
+        self.d.setVar('BB_GIT_SHALLOW', '1')
+        self.d.setVar('BB_GENERATE_SHALLOW_TARBALLS', '1')
+        self.d.setVar('PREMIRRORS', 'git://git.yoctoproject.org/.* git://git.yoctoproject.org/git/PATH;protocol=https \n')
+        fetcher = bb.fetch.Fetch(["git://git.yoctoproject.org/fstests"], self.d)
+        fetcher.download()
+        bb.utils.remove(self.dldir + '/git2', recurse=True)
+
+        fetcher.unpack(self.unpackdir)
+
+        dir = os.listdir(self.unpackdir + "/git/")
+        self.assertIn("fstests.doap", dir)
+
+    @skipIfNoNetwork()
+    def test_that_no_broken_symlinks_exist_when_using_mirror_rewrite_rules_and_mirror_tarball_is_configured(self):
+        self.d.setVar('SRCREV', 'e5939ff608b95cdd4d0ab0e1935781ab9a276ac0')
+        self.d.setVar('BB_GIT_SHALLOW', '1')
+        self.d.setVar('BB_GENERATE_MIRROR_TARBALLS', '1')
+        self.d.delVar('BB_GENERATE_SHALLOW_TARBALLS')
+        self.d.setVar('PREMIRRORS', 'git://git.yoctoproject.org/.* git://git.yoctoproject.org/git/PATH;protocol=https \n')
+        fetcher = bb.fetch.Fetch(["git://git.yoctoproject.org/fstests"], self.d)
+
+        fetcher.download()
+
+        broken_symlinks = get_broken_symlinks(self.dldir)
+        self.assertEqual([], broken_symlinks)
-- 
2.19.1

Re: [PATCH 0/7] Add symlinks for git (shallow) tarballs when name differs due to mirror rewrite rules

[Richard Purdie]

On Mon, 2018-10-15 at 13:43 +0000, Urs Fässler wrote:
> The git (shallow) tarballs are not found when certain mirror rewrite rules are
> in use. This is due to the fact that the tarball is named according to the url
> where it is downloaded from. The error occurs in the unpack step where BitBake
> looks for a tarball named after the url as specified in the recipe. By adding
> the symlinks BitBake finds the tarballs in the unpack step.
> 
> This is an alternative to the patches "Always use the url specified in the
> recipe as a base for the git shallow tarball naming" from the 23.07.2018. This
> solution is more aligned to the current implementation of BitBake.

This patch series does look much more in keeping with the way bitbake
is written and is easier to review.

I tried it on the autobuilder and it resulted in:


https://autobuilder.yoctoproject.org/typhoon/#/builders/23/builds/120/steps/7/logs/step2b

I'm a bit nervous about accepting these in right now given we're so
close to releasing 2.6 :/.

Cheers,

Richard