All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] fetch/gitsm: avoid live submodule fetching during unpack()
@ 2018-05-10  2:33 Matt Hoosier
  2018-05-11 12:28 ` Burton, Ross
  0 siblings, 1 reply; 8+ messages in thread
From: Matt Hoosier @ 2018-05-10  2:33 UTC (permalink / raw)
  To: bitbake-devel

Although the submodules' histories have been fetched during the
do_fetch() phase, the mechanics used to clone the workdir copy
of the repo haven't been transferring the actual .git/modules
directory from the repo fetched into downloads/ during the
fetch task.

Fix that, and for good measure also explicitly tell Git to avoid
hitting the network during do_unpack() of the submodules.

[YOCTO #12739]

Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
---
 lib/bb/fetch2/gitsm.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
index 0aff1008..1f3fc443 100644
--- a/lib/bb/fetch2/gitsm.py
+++ b/lib/bb/fetch2/gitsm.py
@@ -132,4 +132,14 @@ class GitSM(Git):
 
         if self.uses_submodules(ud, d, ud.destdir):
             runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir)
-            runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir)
+
+            # Copy over the submodules' fetched histories too.
+            if ud.bareclone:
+                repo_conf = ud.destdir
+            else:
+                repo_conf = os.path.join(ud.destdir, '.git')
+            runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
+
+            # Careful not to hit the network during unpacking; all history should already
+            # be fetched.
+            runfetchcmd(ud.basecmd + " submodule update --init --recursive --no-fetch", d, workdir=ud.destdir)
-- 
2.13.6



^ permalink raw reply related	[flat|nested] 8+ messages in thread
* [PATCH] fetch/gitsm: avoid live submodule fetching during unpack()
@ 2018-05-11 13:53 Matt Hoosier
  2018-05-11 19:32 ` Matt Hoosier
  0 siblings, 1 reply; 8+ messages in thread
From: Matt Hoosier @ 2018-05-11 13:53 UTC (permalink / raw)
  To: bitbake-devel

Although the submodules' histories have been fetched during the
do_fetch() phase, the mechanics used to clone the workdir copy
of the repo haven't been transferring the actual .git/modules
directory from the repo fetched into downloads/ during the
fetch task.

Fix that, and for good measure also explicitly tell Git to avoid
hitting the network during do_unpack() of the submodules.

Rev2 changes:

- Fixed a failure when unpacking a shallow cloned mirror repository.

[YOCTO #12739]

Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
---
 lib/bb/fetch2/gitsm.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
index 0aff1008..1f3fc443 100644
--- a/lib/bb/fetch2/gitsm.py
+++ b/lib/bb/fetch2/gitsm.py
@@ -132,4 +132,14 @@ class GitSM(Git):
 
         if self.uses_submodules(ud, d, ud.destdir):
             runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir)
-            runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir)
+
+            # Copy over the submodules' fetched histories too.
+            if ud.bareclone:
+                repo_conf = ud.destdir
+            else:
+                repo_conf = os.path.join(ud.destdir, '.git')
+            runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
+
+            # Careful not to hit the network during unpacking; all history should already
+            # be fetched.
+            runfetchcmd(ud.basecmd + " submodule update --init --recursive --no-fetch", d, workdir=ud.destdir)
-- 
2.13.6



^ permalink raw reply related	[flat|nested] 8+ messages in thread
* [PATCH] fetch/gitsm: avoid live submodule fetching during unpack()
@ 2018-05-11 19:26 Matt Hoosier
  2018-05-14 13:18 ` Peter Kjellerstedt
  0 siblings, 1 reply; 8+ messages in thread
From: Matt Hoosier @ 2018-05-11 19:26 UTC (permalink / raw)
  To: bitbake-devel

Although the submodules' histories have been fetched during the
do_fetch() phase, the mechanics used to clone the workdir copy
of the repo haven't been transferring the actual .git/modules
directory from the repo fetched into downloads/ during the
fetch task.

Fix that, and for good measure also explicitly tell Git to avoid
hitting the network during do_unpack() of the submodules.

Rev3 changes:

- Really fix the shallow-clone issue claimed to be addressed in rev2.

Rev2 changes:

- Fixed a failure when unpacking a shallow cloned mirror repository.

[YOCTO #12739]

Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
---
 lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++-
 1 file changed, 27 insertions(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
index 0aff1008..3969f2f5 100644
--- a/lib/bb/fetch2/gitsm.py
+++ b/lib/bb/fetch2/gitsm.py
@@ -132,4 +132,30 @@ class GitSM(Git):
 
         if self.uses_submodules(ud, d, ud.destdir):
             runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir)
-            runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir)
+
+            updateflags = ""
+
+            # Copy over the submodules' fetched histories too.
+            if ud.bareclone:
+                repo_conf = ud.destdir
+            else:
+                repo_conf = os.path.join(ud.destdir, '.git')
+
+            if os.path.exists(ud.clonedir):
+                # This is not a copy unpacked from a shallow mirror clone. So
+                # the manual intervention to populate the .git/modules done
+                # in clone_shallow_local() won't have been done yet.
+                runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
+                updateflags += " --no-fetch"
+            elif os.path.exists(os.path.join(repo_conf, 'modules')):
+                # Unpacked from a shallow mirror clone. Manual population of
+                # .git/modules is already done.
+                updateflags += " --no-fetch"
+            else:
+                # This isn't fatal; git-submodule will just fetch it
+                # during do_unpack().
+                bb.error("submodule history not retrieved during do_fetch()")
+
+            # Careful not to hit the network during unpacking; all history should already
+            # be fetched.
+            runfetchcmd(ud.basecmd + " submodule update --init --recursive %s" % updateflags, d, workdir=ud.destdir)
-- 
2.13.6



^ permalink raw reply related	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2018-05-14 18:52 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-05-10  2:33 [PATCH] fetch/gitsm: avoid live submodule fetching during unpack() Matt Hoosier
2018-05-11 12:28 ` Burton, Ross
2018-05-11 13:55   ` Matt Hoosier
2018-05-11 13:53 Matt Hoosier
2018-05-11 19:32 ` Matt Hoosier
2018-05-11 19:26 Matt Hoosier
2018-05-14 13:18 ` Peter Kjellerstedt
2018-05-14 18:51   ` Matt Hoosier

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.