[PATCH 0/3] fetch2/git: debug updates and minor cleanup

[PATCH 0/3] fetch2/git: debug updates and minor cleanup

[Darren Hart]

From: Darren Hart <dvhart@linux.intel.com>

Yu Ke,

Please consider the following patches against the fetch2/git fetcher. The one
point I would appreciate your review of is in 2/3. Is there a concern that
the urldata has not been initialized at the point where I made the change?

Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: dvhart/git2
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dvhart/git2

Thanks,
    Darren Hart <dvhart@linux.intel.com>
---


Darren Hart (3):
  fetch2/git: ensure network check log matches actual command
  fetch2/git: use urldata basecmd when available
  fetch2/git: use logging.debug() and clarify messages

 bitbake/lib/bb/fetch2/git.py |   24 +++++++++++++++---------
 1 files changed, 15 insertions(+), 9 deletions(-)

[PATCH 1/3] fetch2/git: ensure network check log matches actual command

[Darren Hart]

From: Darren Hart <dvhart@linux.intel.com>

The git command string logged via check_network_access() does not match
the actual command executed in a few places. Ensure that it does.

Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Cc: Yu Ke <ke.yu@intel.com>
---
 bitbake/lib/bb/fetch2/git.py |   16 ++++++++++------
 1 files changed, 10 insertions(+), 6 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index 82721c6..c6c4cc9 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -137,8 +137,10 @@ class Git(FetchMethod):
 
         # If the repo still doesn't exist, fallback to cloning it
         if not os.path.exists(ud.clonedir):
-            bb.fetch2.check_network_access(d, "git clone --bare %s%s" % (ud.host, ud.path))
-            runfetchcmd("%s clone --bare %s://%s%s%s %s" % (ud.basecmd, ud.proto, username, ud.host, ud.path, ud.clonedir), d)
+            clone_cmd = "%s clone --bare %s://%s%s%s %s" % \
+                  (ud.basecmd, ud.proto, username, ud.host, ud.path, ud.clonedir)
+            bb.fetch2.check_network_access(d, clone_cmd)
+            runfetchcmd(clone_cmd, d)
 
         os.chdir(ud.clonedir)
         # Update the checkout if needed
@@ -147,7 +149,6 @@ class Git(FetchMethod):
             if not self._contains_ref(ud.revisions[name], d):
                 needupdate = True
         if needupdate:
-            bb.fetch2.check_network_access(d, "git fetch %s%s" % (ud.host, ud.path), ud.url)
             try: 
                 runfetchcmd("%s remote prune origin" % ud.basecmd, d) 
                 runfetchcmd("%s remote rm origin" % ud.basecmd, d) 
@@ -155,7 +156,9 @@ class Git(FetchMethod):
                 logger.debug(1, "No Origin")
             
             runfetchcmd("%s remote add origin %s://%s%s%s" % (ud.basecmd, ud.proto, username, ud.host, ud.path), d)
-            runfetchcmd("%s fetch --all -t" % ud.basecmd, d)
+            fetch_cmd = "%s fetch --all -t" % ud.basecmd
+            bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
+            runfetchcmd(fetch_cmd, d)
             runfetchcmd("%s prune-packed" % ud.basecmd, d)
             runfetchcmd("%s pack-redundant --all | xargs -r rm" % ud.basecmd, d)
             ud.repochanged = True
@@ -216,9 +219,10 @@ class Git(FetchMethod):
         else:
             username = ""
 
-        bb.fetch2.check_network_access(d, "git ls-remote %s%s %s" % (ud.host, ud.path, ud.branches[name]))
         basecmd = data.getVar("FETCHCMD_git", d, True) or "git"
-        cmd = "%s ls-remote %s://%s%s%s %s" % (basecmd, ud.proto, username, ud.host, ud.path, ud.branches[name])
+        cmd = "%s ls-remote %s://%s%s%s %s" % \
+              (basecmd, ud.proto, username, ud.host, ud.path, ud.branches[name])
+        bb.fetch2.check_network_access(d, cmd)
         output = runfetchcmd(cmd, d, True)
         if not output:
             raise bb.fetch2.FetchError("The command %s gave empty output unexpectedly" % cmd, url)
-- 
1.7.1

[PATCH 2/3] fetch2/git: use urldata basecmd when available

[Darren Hart]

From: Darren Hart <dvhart@linux.intel.com>

If the urldata object is passed in, use it to determine basecmd rather
than looking it up again.

Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Cc: Yu Ke <ke.yu@intel.com>
---
 bitbake/lib/bb/fetch2/git.py |   15 +++++++--------
 1 files changed, 7 insertions(+), 8 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index c6c4cc9..7ffaa5e 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -137,10 +137,10 @@ class Git(FetchMethod):
 
         # If the repo still doesn't exist, fallback to cloning it
         if not os.path.exists(ud.clonedir):
-            clone_cmd = "%s clone --bare %s://%s%s%s %s" % \
+            cmd = "%s clone --bare %s://%s%s%s %s" % \
                   (ud.basecmd, ud.proto, username, ud.host, ud.path, ud.clonedir)
-            bb.fetch2.check_network_access(d, clone_cmd)
-            runfetchcmd(clone_cmd, d)
+            bb.fetch2.check_network_access(d, cmd)
+            runfetchcmd(cmd, d)
 
         os.chdir(ud.clonedir)
         # Update the checkout if needed
@@ -156,9 +156,9 @@ class Git(FetchMethod):
                 logger.debug(1, "No Origin")
             
             runfetchcmd("%s remote add origin %s://%s%s%s" % (ud.basecmd, ud.proto, username, ud.host, ud.path), d)
-            fetch_cmd = "%s fetch --all -t" % ud.basecmd
-            bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
-            runfetchcmd(fetch_cmd, d)
+            cmd = "%s fetch --all -t" % ud.basecmd
+            bb.fetch2.check_network_access(d, cmd, ud.url)
+            runfetchcmd(cmd, d)
             runfetchcmd("%s prune-packed" % ud.basecmd, d)
             runfetchcmd("%s pack-redundant --all | xargs -r rm" % ud.basecmd, d)
             ud.repochanged = True
@@ -219,9 +219,8 @@ class Git(FetchMethod):
         else:
             username = ""
 
-        basecmd = data.getVar("FETCHCMD_git", d, True) or "git"
         cmd = "%s ls-remote %s://%s%s%s %s" % \
-              (basecmd, ud.proto, username, ud.host, ud.path, ud.branches[name])
+              (ud.basecmd, ud.proto, username, ud.host, ud.path, ud.branches[name])
         bb.fetch2.check_network_access(d, cmd)
         output = runfetchcmd(cmd, d, True)
         if not output:
-- 
1.7.1

[PATCH 3/3] fetch2/git: use logging.debug() and clarify messages

[Darren Hart]

From: Darren Hart <dvhart@linux.intel.com>

Replace a call to print() with logging.debug() and flesh out the
message to clarify the state being reported.

Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Cc: Yu Ke <ke.yu@intel.com>
---
 bitbake/lib/bb/fetch2/git.py |    7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index 7ffaa5e..c14346b 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -241,10 +241,13 @@ class Git(FetchMethod):
         # Check if we have the rev already
 
         if not os.path.exists(ud.clonedir):
-            print("no repo")
+            logging.debug("GIT repository for %s does not exist in %s.  \
+                          Downloading.", url, ud.clonedir)
             self.download(None, ud, d)
             if not os.path.exists(ud.clonedir):
-                logger.error("GIT repository for %s doesn't exist in %s, cannot get sortable buildnumber, using old value", url, ud.clonedir)
+                logger.error("GIT repository for %s does not exist in %s after \
+                             download. Cannot get sortable buildnumber, using \
+                             old value", url, ud.clonedir)
                 return None
 
 
-- 
1.7.1

Re: [PATCH 0/3] fetch2/git: debug updates and minor cleanup

[Richard Purdie]

On Fri, 2011-05-20 at 16:06 -0700, Darren Hart wrote:
> From: Darren Hart <dvhart@linux.intel.com>
> 
> Yu Ke,
> 
> Please consider the following patches against the fetch2/git fetcher. The one
> point I would appreciate your review of is in 2/3. Is there a concern that
> the urldata has not been initialized at the point where I made the change?
> 
> Pull URL: git://git.pokylinux.org/poky-contrib.git
>   Branch: dvhart/git2
>   Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dvhart/git2
> 
> Thanks,
>     Darren Hart <dvhart@linux.intel.com>
> ---
> 
> 
> Darren Hart (3):
>   fetch2/git: ensure network check log matches actual command
>   fetch2/git: use logging.debug() and clarify messages

I merged these into bitbake master, thanks.

  fetch2/git: use urldata basecmd when available

This one we a had a good reason for, at least a long time ago. It needs
some further checking into the callers (and better documentation if it
is deliberate). The fetcher world has changed a lot since those times...

I'm therefore holding off on it for now.

Cheers,

Richard