All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] fetch/gitsm: avoid live submodule fetching during unpack()
@ 2018-05-10  2:33 Matt Hoosier
  2018-05-11 12:28 ` Burton, Ross
  0 siblings, 1 reply; 8+ messages in thread
From: Matt Hoosier @ 2018-05-10  2:33 UTC (permalink / raw)
  To: bitbake-devel

Although the submodules' histories have been fetched during the
do_fetch() phase, the mechanics used to clone the workdir copy
of the repo haven't been transferring the actual .git/modules
directory from the repo fetched into downloads/ during the
fetch task.

Fix that, and for good measure also explicitly tell Git to avoid
hitting the network during do_unpack() of the submodules.

[YOCTO #12739]

Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
---
 lib/bb/fetch2/gitsm.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
index 0aff1008..1f3fc443 100644
--- a/lib/bb/fetch2/gitsm.py
+++ b/lib/bb/fetch2/gitsm.py
@@ -132,4 +132,14 @@ class GitSM(Git):
 
         if self.uses_submodules(ud, d, ud.destdir):
             runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir)
-            runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir)
+
+            # Copy over the submodules' fetched histories too.
+            if ud.bareclone:
+                repo_conf = ud.destdir
+            else:
+                repo_conf = os.path.join(ud.destdir, '.git')
+            runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
+
+            # Careful not to hit the network during unpacking; all history should already
+            # be fetched.
+            runfetchcmd(ud.basecmd + " submodule update --init --recursive --no-fetch", d, workdir=ud.destdir)
-- 
2.13.6



^ permalink raw reply related	[flat|nested] 8+ messages in thread

* Re: [PATCH] fetch/gitsm: avoid live submodule fetching during unpack()
  2018-05-10  2:33 [PATCH] fetch/gitsm: avoid live submodule fetching during unpack() Matt Hoosier
@ 2018-05-11 12:28 ` Burton, Ross
  2018-05-11 13:55   ` Matt Hoosier
  0 siblings, 1 reply; 8+ messages in thread
From: Burton, Ross @ 2018-05-11 12:28 UTC (permalink / raw)
  To: Matt Hoosier; +Cc: bitbake-devel

This breaks bitbake-selftest:

$ bitbake-selftest  bb.tests.fetch.GitShallowTest
........................E

ERROR: test_shallow_submodules (bb.tests.fetch.GitShallowTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ross/Yocto/poky/bitbake/lib/bb/tests/fetch.py", line
1353, in test_shallow_submodules
    fetcher, ud = self.fetch_shallow(uri)
  File "/home/ross/Yocto/poky/bitbake/lib/bb/tests/fetch.py", line
1198, in fetch_shallow
    fetcher, ud = self.fetch_and_unpack(uri)
  File "/home/ross/Yocto/poky/bitbake/lib/bb/tests/fetch.py", line
1179, in fetch_and_unpack
    fetcher.unpack(self.d.getVar('WORKDIR'))
  File "/home/ross/Yocto/poky/bitbake/lib/bb/fetch2/__init__.py", line
1762, in unpack
    ud.method.unpack(ud, root, self.d)
  File "/home/ross/Yocto/poky/bitbake/lib/bb/fetch2/gitsm.py", line
141, in unpack
    runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir,
'modules'), repo_conf), d)
  File "/home/ross/Yocto/poky/bitbake/lib/bb/fetch2/__init__.py", line
885, in runfetchcmd
    raise FetchError(error_message)
bb.fetch2.FetchError: Fetcher failure: Fetch command export
PSEUDO_DISABLED=1; cp -pr
/tmp/tmpxi2ydemw/download/git2/tmp.tmpxi2ydemw.gitsource/modules
/tmp/tmpxi2ydemw/git/.git failed with exit code 1, output:
cp: cannot stat
'/tmp/tmpxi2ydemw/download/git2/tmp.tmpxi2ydemw.gitsource/modules': No
such file or directory

Ross

On 10 May 2018 at 03:33, Matt Hoosier <matt.hoosier@gmail.com> wrote:
> Although the submodules' histories have been fetched during the
> do_fetch() phase, the mechanics used to clone the workdir copy
> of the repo haven't been transferring the actual .git/modules
> directory from the repo fetched into downloads/ during the
> fetch task.
>
> Fix that, and for good measure also explicitly tell Git to avoid
> hitting the network during do_unpack() of the submodules.
>
> [YOCTO #12739]
>
> Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
> ---
>  lib/bb/fetch2/gitsm.py | 12 +++++++++++-
>  1 file changed, 11 insertions(+), 1 deletion(-)
>
> diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
> index 0aff1008..1f3fc443 100644
> --- a/lib/bb/fetch2/gitsm.py
> +++ b/lib/bb/fetch2/gitsm.py
> @@ -132,4 +132,14 @@ class GitSM(Git):
>
>          if self.uses_submodules(ud, d, ud.destdir):
>              runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir)
> -            runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir)
> +
> +            # Copy over the submodules' fetched histories too.
> +            if ud.bareclone:
> +                repo_conf = ud.destdir
> +            else:
> +                repo_conf = os.path.join(ud.destdir, '.git')
> +            runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
> +
> +            # Careful not to hit the network during unpacking; all history should already
> +            # be fetched.
> +            runfetchcmd(ud.basecmd + " submodule update --init --recursive --no-fetch", d, workdir=ud.destdir)
> --
> 2.13.6
>
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] fetch/gitsm: avoid live submodule fetching during unpack()
  2018-05-11 12:28 ` Burton, Ross
@ 2018-05-11 13:55   ` Matt Hoosier
  0 siblings, 0 replies; 8+ messages in thread
From: Matt Hoosier @ 2018-05-11 13:55 UTC (permalink / raw)
  To: Burton, Ross; +Cc: bitbake-devel

[-- Attachment #1: Type: text/plain, Size: 3743 bytes --]

On Fri, May 11, 2018 at 7:28 AM, Burton, Ross <ross.burton@intel.com> wrote:

> This breaks bitbake-selftest:
>
> $ bitbake-selftest  bb.tests.fetch.GitShallowTest
> ........................E
>
> ERROR: test_shallow_submodules (bb.tests.fetch.GitShallowTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/ross/Yocto/poky/bitbake/lib/bb/tests/fetch.py", line
> 1353, in test_shallow_submodules
>     fetcher, ud = self.fetch_shallow(uri)
>   File "/home/ross/Yocto/poky/bitbake/lib/bb/tests/fetch.py", line
> 1198, in fetch_shallow
>     fetcher, ud = self.fetch_and_unpack(uri)
>   File "/home/ross/Yocto/poky/bitbake/lib/bb/tests/fetch.py", line
> 1179, in fetch_and_unpack
>     fetcher.unpack(self.d.getVar('WORKDIR'))
>   File "/home/ross/Yocto/poky/bitbake/lib/bb/fetch2/__init__.py", line
> 1762, in unpack
>     ud.method.unpack(ud, root, self.d)
>   File "/home/ross/Yocto/poky/bitbake/lib/bb/fetch2/gitsm.py", line
> 141, in unpack
>     runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir,
> 'modules'), repo_conf), d)
>   File "/home/ross/Yocto/poky/bitbake/lib/bb/fetch2/__init__.py", line
> 885, in runfetchcmd
>     raise FetchError(error_message)
> bb.fetch2.FetchError: Fetcher failure: Fetch command export
> PSEUDO_DISABLED=1; cp -pr
> /tmp/tmpxi2ydemw/download/git2/tmp.tmpxi2ydemw.gitsource/modules
> /tmp/tmpxi2ydemw/git/.git failed with exit code 1, output:
> cp: cannot stat
> '/tmp/tmpxi2ydemw/download/git2/tmp.tmpxi2ydemw.gitsource/modules': No
> such file or directory
>
> Ross
>

Thanks; I'd not considered shallow mirrored clones; rev2 fixes this.


>
> On 10 May 2018 at 03:33, Matt Hoosier <matt.hoosier@gmail.com> wrote:
> > Although the submodules' histories have been fetched during the
> > do_fetch() phase, the mechanics used to clone the workdir copy
> > of the repo haven't been transferring the actual .git/modules
> > directory from the repo fetched into downloads/ during the
> > fetch task.
> >
> > Fix that, and for good measure also explicitly tell Git to avoid
> > hitting the network during do_unpack() of the submodules.
> >
> > [YOCTO #12739]
> >
> > Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
> > ---
> >  lib/bb/fetch2/gitsm.py | 12 +++++++++++-
> >  1 file changed, 11 insertions(+), 1 deletion(-)
> >
> > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
> > index 0aff1008..1f3fc443 100644
> > --- a/lib/bb/fetch2/gitsm.py
> > +++ b/lib/bb/fetch2/gitsm.py
> > @@ -132,4 +132,14 @@ class GitSM(Git):
> >
> >          if self.uses_submodules(ud, d, ud.destdir):
> >              runfetchcmd(ud.basecmd + " checkout " +
> ud.revisions[ud.names[0]], d, workdir=ud.destdir)
> > -            runfetchcmd(ud.basecmd + " submodule update --init
> --recursive", d, workdir=ud.destdir)
> > +
> > +            # Copy over the submodules' fetched histories too.
> > +            if ud.bareclone:
> > +                repo_conf = ud.destdir
> > +            else:
> > +                repo_conf = os.path.join(ud.destdir, '.git')
> > +            runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir,
> 'modules'), repo_conf), d)
> > +
> > +            # Careful not to hit the network during unpacking; all
> history should already
> > +            # be fetched.
> > +            runfetchcmd(ud.basecmd + " submodule update --init
> --recursive --no-fetch", d, workdir=ud.destdir)
> > --
> > 2.13.6
> >
> > --
> > _______________________________________________
> > bitbake-devel mailing list
> > bitbake-devel@lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>

[-- Attachment #2: Type: text/html, Size: 5231 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] fetch/gitsm: avoid live submodule fetching during unpack()
  2018-05-14 13:18 ` Peter Kjellerstedt
@ 2018-05-14 18:51   ` Matt Hoosier
  0 siblings, 0 replies; 8+ messages in thread
From: Matt Hoosier @ 2018-05-14 18:51 UTC (permalink / raw)
  To: Peter Kjellerstedt; +Cc: bitbake-devel

[-- Attachment #1: Type: text/plain, Size: 3718 bytes --]

On Mon, May 14, 2018 at 8:18 AM, Peter Kjellerstedt <
peter.kjellerstedt@axis.com> wrote:

> > -----Original Message-----
> > From: bitbake-devel-bounces@lists.openembedded.org [mailto:bitbake-
> > devel-bounces@lists.openembedded.org] On Behalf Of Matt Hoosier
> > Sent: den 11 maj 2018 21:27
> > To: bitbake-devel@lists.openembedded.org
> > Subject: [bitbake-devel] [PATCH] fetch/gitsm: avoid live submodule
> > fetching during unpack()
> >
> > Although the submodules' histories have been fetched during the
> > do_fetch() phase, the mechanics used to clone the workdir copy
> > of the repo haven't been transferring the actual .git/modules
> > directory from the repo fetched into downloads/ during the
> > fetch task.
> >
> > Fix that, and for good measure also explicitly tell Git to avoid
> > hitting the network during do_unpack() of the submodules.
> >
> > Rev3 changes:
> >
> > - Really fix the shallow-clone issue claimed to be addressed in rev2.
> >
> > Rev2 changes:
> >
> > - Fixed a failure when unpacking a shallow cloned mirror repository.
>
> The Rev2 and Rev3 changes information should go below the --- below
> as it should not be part of the commit message as it relates to the
> patch submission process rather than the actual change. If any of it
> is relevant for the commit message, then the commit message should
> be changed as well.
>

Thanks; done. Resubmitted (as v4) to the list.


>
> > [YOCTO #12739]
> >
> > Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
> > ---
> >  lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++-
> >  1 file changed, 27 insertions(+), 1 deletion(-)
> >
> > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
> > index 0aff1008..3969f2f5 100644
> > --- a/lib/bb/fetch2/gitsm.py
> > +++ b/lib/bb/fetch2/gitsm.py
> > @@ -132,4 +132,30 @@ class GitSM(Git):
> >
> >          if self.uses_submodules(ud, d, ud.destdir):
> >              runfetchcmd(ud.basecmd + " checkout " +
> ud.revisions[ud.names[0]], d, workdir=ud.destdir)
> > -            runfetchcmd(ud.basecmd + " submodule update --init
> --recursive", d, workdir=ud.destdir)
> > +
> > +            updateflags = ""
> > +
> > +            # Copy over the submodules' fetched histories too.
> > +            if ud.bareclone:
> > +                repo_conf = ud.destdir
> > +            else:
> > +                repo_conf = os.path.join(ud.destdir, '.git')
> > +
> > +            if os.path.exists(ud.clonedir):
> > +                # This is not a copy unpacked from a shallow mirror
> clone. So
> > +                # the manual intervention to populate the .git/modules
> done
> > +                # in clone_shallow_local() won't have been done yet.
> > +                runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir,
> 'modules'), repo_conf), d)
> > +                updateflags += " --no-fetch"
> > +            elif os.path.exists(os.path.join(repo_conf, 'modules')):
> > +                # Unpacked from a shallow mirror clone. Manual
> population of
> > +                # .git/modules is already done.
> > +                updateflags += " --no-fetch"
> > +            else:
> > +                # This isn't fatal; git-submodule will just fetch it
> > +                # during do_unpack().
> > +                bb.error("submodule history not retrieved during
> do_fetch()")
> > +
> > +            # Careful not to hit the network during unpacking; all
> history should already
> > +            # be fetched.
> > +            runfetchcmd(ud.basecmd + " submodule update --init
> --recursive %s" % updateflags, d, workdir=ud.destdir)
> > --
> > 2.13.6
>
> //Peter
>
>

[-- Attachment #2: Type: text/html, Size: 5230 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] fetch/gitsm: avoid live submodule fetching during unpack()
  2018-05-11 19:26 Matt Hoosier
@ 2018-05-14 13:18 ` Peter Kjellerstedt
  2018-05-14 18:51   ` Matt Hoosier
  0 siblings, 1 reply; 8+ messages in thread
From: Peter Kjellerstedt @ 2018-05-14 13:18 UTC (permalink / raw)
  To: Matt Hoosier, bitbake-devel

> -----Original Message-----
> From: bitbake-devel-bounces@lists.openembedded.org [mailto:bitbake-
> devel-bounces@lists.openembedded.org] On Behalf Of Matt Hoosier
> Sent: den 11 maj 2018 21:27
> To: bitbake-devel@lists.openembedded.org
> Subject: [bitbake-devel] [PATCH] fetch/gitsm: avoid live submodule
> fetching during unpack()
> 
> Although the submodules' histories have been fetched during the
> do_fetch() phase, the mechanics used to clone the workdir copy
> of the repo haven't been transferring the actual .git/modules
> directory from the repo fetched into downloads/ during the
> fetch task.
> 
> Fix that, and for good measure also explicitly tell Git to avoid
> hitting the network during do_unpack() of the submodules.
> 
> Rev3 changes:
> 
> - Really fix the shallow-clone issue claimed to be addressed in rev2.
> 
> Rev2 changes:
> 
> - Fixed a failure when unpacking a shallow cloned mirror repository.

The Rev2 and Rev3 changes information should go below the --- below 
as it should not be part of the commit message as it relates to the 
patch submission process rather than the actual change. If any of it 
is relevant for the commit message, then the commit message should 
be changed as well.

> [YOCTO #12739]
> 
> Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
> ---
>  lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++-
>  1 file changed, 27 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
> index 0aff1008..3969f2f5 100644
> --- a/lib/bb/fetch2/gitsm.py
> +++ b/lib/bb/fetch2/gitsm.py
> @@ -132,4 +132,30 @@ class GitSM(Git):
> 
>          if self.uses_submodules(ud, d, ud.destdir):
>              runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir)
> -            runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir)
> +
> +            updateflags = ""
> +
> +            # Copy over the submodules' fetched histories too.
> +            if ud.bareclone:
> +                repo_conf = ud.destdir
> +            else:
> +                repo_conf = os.path.join(ud.destdir, '.git')
> +
> +            if os.path.exists(ud.clonedir):
> +                # This is not a copy unpacked from a shallow mirror clone. So
> +                # the manual intervention to populate the .git/modules done
> +                # in clone_shallow_local() won't have been done yet.
> +                runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
> +                updateflags += " --no-fetch"
> +            elif os.path.exists(os.path.join(repo_conf, 'modules')):
> +                # Unpacked from a shallow mirror clone. Manual population of
> +                # .git/modules is already done.
> +                updateflags += " --no-fetch"
> +            else:
> +                # This isn't fatal; git-submodule will just fetch it
> +                # during do_unpack().
> +                bb.error("submodule history not retrieved during do_fetch()")
> +
> +            # Careful not to hit the network during unpacking; all history should already
> +            # be fetched.
> +            runfetchcmd(ud.basecmd + " submodule update --init --recursive %s" % updateflags, d, workdir=ud.destdir)
> --
> 2.13.6

//Peter



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH] fetch/gitsm: avoid live submodule fetching during unpack()
  2018-05-11 13:53 Matt Hoosier
@ 2018-05-11 19:32 ` Matt Hoosier
  0 siblings, 0 replies; 8+ messages in thread
From: Matt Hoosier @ 2018-05-11 19:32 UTC (permalink / raw)
  To: bitbake-devel

[-- Attachment #1: Type: text/plain, Size: 2038 bytes --]

I had a snafu when prepping the submission of this revision. Ignore it;
I've submitted rev3 now:
http://lists.openembedded.org/pipermail/bitbake-devel/2018-May/009325.html.

On Fri, May 11, 2018 at 8:53 AM, Matt Hoosier <matt.hoosier@gmail.com>
wrote:

> Although the submodules' histories have been fetched during the
> do_fetch() phase, the mechanics used to clone the workdir copy
> of the repo haven't been transferring the actual .git/modules
> directory from the repo fetched into downloads/ during the
> fetch task.
>
> Fix that, and for good measure also explicitly tell Git to avoid
> hitting the network during do_unpack() of the submodules.
>
> Rev2 changes:
>
> - Fixed a failure when unpacking a shallow cloned mirror repository.
>
> [YOCTO #12739]
>
> Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
> ---
>  lib/bb/fetch2/gitsm.py | 12 +++++++++++-
>  1 file changed, 11 insertions(+), 1 deletion(-)
>
> diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
> index 0aff1008..1f3fc443 100644
> --- a/lib/bb/fetch2/gitsm.py
> +++ b/lib/bb/fetch2/gitsm.py
> @@ -132,4 +132,14 @@ class GitSM(Git):
>
>          if self.uses_submodules(ud, d, ud.destdir):
>              runfetchcmd(ud.basecmd + " checkout " +
> ud.revisions[ud.names[0]], d, workdir=ud.destdir)
> -            runfetchcmd(ud.basecmd + " submodule update --init
> --recursive", d, workdir=ud.destdir)
> +
> +            # Copy over the submodules' fetched histories too.
> +            if ud.bareclone:
> +                repo_conf = ud.destdir
> +            else:
> +                repo_conf = os.path.join(ud.destdir, '.git')
> +            runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir,
> 'modules'), repo_conf), d)
> +
> +            # Careful not to hit the network during unpacking; all
> history should already
> +            # be fetched.
> +            runfetchcmd(ud.basecmd + " submodule update --init
> --recursive --no-fetch", d, workdir=ud.destdir)
> --
> 2.13.6
>
>

[-- Attachment #2: Type: text/html, Size: 2752 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

* [PATCH] fetch/gitsm: avoid live submodule fetching during unpack()
@ 2018-05-11 19:26 Matt Hoosier
  2018-05-14 13:18 ` Peter Kjellerstedt
  0 siblings, 1 reply; 8+ messages in thread
From: Matt Hoosier @ 2018-05-11 19:26 UTC (permalink / raw)
  To: bitbake-devel

Although the submodules' histories have been fetched during the
do_fetch() phase, the mechanics used to clone the workdir copy
of the repo haven't been transferring the actual .git/modules
directory from the repo fetched into downloads/ during the
fetch task.

Fix that, and for good measure also explicitly tell Git to avoid
hitting the network during do_unpack() of the submodules.

Rev3 changes:

- Really fix the shallow-clone issue claimed to be addressed in rev2.

Rev2 changes:

- Fixed a failure when unpacking a shallow cloned mirror repository.

[YOCTO #12739]

Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
---
 lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++-
 1 file changed, 27 insertions(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
index 0aff1008..3969f2f5 100644
--- a/lib/bb/fetch2/gitsm.py
+++ b/lib/bb/fetch2/gitsm.py
@@ -132,4 +132,30 @@ class GitSM(Git):
 
         if self.uses_submodules(ud, d, ud.destdir):
             runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir)
-            runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir)
+
+            updateflags = ""
+
+            # Copy over the submodules' fetched histories too.
+            if ud.bareclone:
+                repo_conf = ud.destdir
+            else:
+                repo_conf = os.path.join(ud.destdir, '.git')
+
+            if os.path.exists(ud.clonedir):
+                # This is not a copy unpacked from a shallow mirror clone. So
+                # the manual intervention to populate the .git/modules done
+                # in clone_shallow_local() won't have been done yet.
+                runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
+                updateflags += " --no-fetch"
+            elif os.path.exists(os.path.join(repo_conf, 'modules')):
+                # Unpacked from a shallow mirror clone. Manual population of
+                # .git/modules is already done.
+                updateflags += " --no-fetch"
+            else:
+                # This isn't fatal; git-submodule will just fetch it
+                # during do_unpack().
+                bb.error("submodule history not retrieved during do_fetch()")
+
+            # Careful not to hit the network during unpacking; all history should already
+            # be fetched.
+            runfetchcmd(ud.basecmd + " submodule update --init --recursive %s" % updateflags, d, workdir=ud.destdir)
-- 
2.13.6



^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [PATCH] fetch/gitsm: avoid live submodule fetching during unpack()
@ 2018-05-11 13:53 Matt Hoosier
  2018-05-11 19:32 ` Matt Hoosier
  0 siblings, 1 reply; 8+ messages in thread
From: Matt Hoosier @ 2018-05-11 13:53 UTC (permalink / raw)
  To: bitbake-devel

Although the submodules' histories have been fetched during the
do_fetch() phase, the mechanics used to clone the workdir copy
of the repo haven't been transferring the actual .git/modules
directory from the repo fetched into downloads/ during the
fetch task.

Fix that, and for good measure also explicitly tell Git to avoid
hitting the network during do_unpack() of the submodules.

Rev2 changes:

- Fixed a failure when unpacking a shallow cloned mirror repository.

[YOCTO #12739]

Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
---
 lib/bb/fetch2/gitsm.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
index 0aff1008..1f3fc443 100644
--- a/lib/bb/fetch2/gitsm.py
+++ b/lib/bb/fetch2/gitsm.py
@@ -132,4 +132,14 @@ class GitSM(Git):
 
         if self.uses_submodules(ud, d, ud.destdir):
             runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir)
-            runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir)
+
+            # Copy over the submodules' fetched histories too.
+            if ud.bareclone:
+                repo_conf = ud.destdir
+            else:
+                repo_conf = os.path.join(ud.destdir, '.git')
+            runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
+
+            # Careful not to hit the network during unpacking; all history should already
+            # be fetched.
+            runfetchcmd(ud.basecmd + " submodule update --init --recursive --no-fetch", d, workdir=ud.destdir)
-- 
2.13.6



^ permalink raw reply related	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2018-05-14 18:52 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-05-10  2:33 [PATCH] fetch/gitsm: avoid live submodule fetching during unpack() Matt Hoosier
2018-05-11 12:28 ` Burton, Ross
2018-05-11 13:55   ` Matt Hoosier
2018-05-11 13:53 Matt Hoosier
2018-05-11 19:32 ` Matt Hoosier
2018-05-11 19:26 Matt Hoosier
2018-05-14 13:18 ` Peter Kjellerstedt
2018-05-14 18:51   ` Matt Hoosier

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.