Re: [PATCHv2] bb.fetch2.{git, hg}: remove tarball if it needs updating

Re: [PATCHv2] bb.fetch2.{git, hg}: remove tarball if it needs updating

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1091 bytes --]

On Tue, Sep 15, 2015 at 2:23 PM, Christopher Larson <kergoth@gmail.com>
wrote:

> From: Christopher Larson <chris_larson@mentor.com>
>
> We were maintaining state in the form of ud.repochanged to determine
> whether
> we need to write the tarball in the case where the tarball already exists,
> but
> this state is maintained only in memory. If we need to update the git repo,
> set ud.repochanged to True, and then are interrupted or killed, the tarball
> will then be out of date.
>
> Rather than maintaining this state, simply remove the out of date tarball
> when
> we update the git repo, and it will be re-created with updated content in
> build_mirror_data.
>
> [YOCTO #6366]
>
> Signed-off-by: Christopher Larson <chris_larson@mentor.com>


Ignore the v2, only removing it when write_tarballs is set could be
problematic if the repo is updated with write_tarballs not set, then it's
set afterward.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1591 bytes --]

[PATCHv2] bb.fetch2.{git, hg}: remove tarball if it needs updating

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

We were maintaining state in the form of ud.repochanged to determine whether
we need to write the tarball in the case where the tarball already exists, but
this state is maintained only in memory. If we need to update the git repo,
set ud.repochanged to True, and then are interrupted or killed, the tarball
will then be out of date.

Rather than maintaining this state, simply remove the out of date tarball when
we update the git repo, and it will be re-created with updated content in
build_mirror_data.

[YOCTO #6366]

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 bitbake/lib/bb/fetch2/git.py | 11 +++++++----
 bitbake/lib/bb/fetch2/hg.py  | 11 +++++++----
 2 files changed, 14 insertions(+), 8 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index 40658ff..9727041 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -181,8 +181,6 @@ class Git(FetchMethod):
     def download(self, ud, d):
         """Fetch url"""
 
-        ud.repochanged = not os.path.exists(ud.fullmirror)
-
         # If the checkout doesn't exist and the mirror tarball does, extract it
         if not os.path.exists(ud.clonedir) and os.path.exists(ud.fullmirror):
             bb.utils.mkdirhier(ud.clonedir)
@@ -220,7 +218,12 @@ class Git(FetchMethod):
             runfetchcmd(fetch_cmd, d)
             runfetchcmd("%s prune-packed" % ud.basecmd, d)
             runfetchcmd("%s pack-redundant --all | xargs -r rm" % ud.basecmd, d)
-            ud.repochanged = True
+            if ud.write_tarballs:
+                try:
+                    os.unlink(ud.fullmirror)
+                except OSError as exc:
+                    if exc.errno != errno.ENOENT:
+                        raise
         os.chdir(ud.clonedir)
         for name in ud.names:
             if not self._contains_ref(ud, d, name):
@@ -228,7 +231,7 @@ class Git(FetchMethod):
 
     def build_mirror_data(self, ud, d):
         # Generate a mirror tarball if needed
-        if ud.write_tarballs and (ud.repochanged or not os.path.exists(ud.fullmirror)):
+        if ud.write_tarballs and not os.path.exists(ud.fullmirror):
             # it's possible that this symlink points to read-only filesystem with PREMIRROR
             if os.path.islink(ud.fullmirror):
                 os.unlink(ud.fullmirror)
diff --git a/bitbake/lib/bb/fetch2/hg.py b/bitbake/lib/bb/fetch2/hg.py
index d978630..c880db2 100644
--- a/bitbake/lib/bb/fetch2/hg.py
+++ b/bitbake/lib/bb/fetch2/hg.py
@@ -163,8 +163,6 @@ class Hg(FetchMethod):
     def download(self, ud, d):
         """Fetch url"""
 
-        ud.repochanged = not os.path.exists(ud.fullmirror)
-
         logger.debug(2, "Fetch: checking for module directory '" + ud.moddir + "'")
 
         # If the checkout doesn't exist and the mirror tarball does, extract it
@@ -189,7 +187,12 @@ class Hg(FetchMethod):
                 logger.debug(1, "Running %s", pullcmd)
                 bb.fetch2.check_network_access(d, pullcmd, ud.url)
                 runfetchcmd(pullcmd, d)
-                ud.repochanged = True
+                if ud.write_tarballs == "1":
+                    try:
+                        os.unlink(ud.fullmirror)
+                    except OSError as exc:
+                        if exc.errno != errno.ENOENT:
+                            raise
 
         # No source found, clone it.
         if not os.path.exists(ud.moddir):
@@ -238,7 +241,7 @@ class Hg(FetchMethod):
 
     def build_mirror_data(self, ud, d):
         # Generate a mirror tarball if needed
-        if ud.write_tarballs == "1" and (ud.repochanged or not os.path.exists(ud.fullmirror)):
+        if ud.write_tarballs == "1" and not os.path.exists(ud.fullmirror):
             # it's possible that this symlink points to read-only filesystem with PREMIRROR
             if os.path.islink(ud.fullmirror):
                 os.unlink(ud.fullmirror)
-- 
2.2.1