[PATCH] bitbake: fetch2/gitsm: avoid live submodule fetching during unpack()

[PATCH] bitbake: fetch2/gitsm: avoid live submodule fetching during unpack()

[Matt Hoosier]

From: Matt Hoosier <matt.hoosier@garmin.com>

Although the submodules' histories have been fetched during the
do_fetch() phase, the mechanics used to clone the workdir copy
of the repo haven't been transferring the actual .git/modules
directory from the repo fetched into downloads/ during the
fetch task.

Fix that, and for good measure also explicitly tell Git to avoid
hitting the network during do_unpack() of the submodules.

Ref:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12739
---
 bitbake/lib/bb/fetch2/gitsm.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py
index 0aff1008e5..1f3fc44352 100644
--- a/bitbake/lib/bb/fetch2/gitsm.py
+++ b/bitbake/lib/bb/fetch2/gitsm.py
@@ -132,4 +132,14 @@ class GitSM(Git):
 
         if self.uses_submodules(ud, d, ud.destdir):
             runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir)
-            runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir)
+
+            # Copy over the submodules' fetched histories too.
+            if ud.bareclone:
+                repo_conf = ud.destdir
+            else:
+                repo_conf = os.path.join(ud.destdir, '.git')
+            runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
+
+            # Careful not to hit the network during unpacking; all history should already
+            # be fetched.
+            runfetchcmd(ud.basecmd + " submodule update --init --recursive --no-fetch", d, workdir=ud.destdir)
-- 
2.13.6

Re: [PATCH] bitbake: fetch2/gitsm: avoid live submodule fetching during unpack()

[Andre McCurdy]

On Wed, May 9, 2018 at 2:48 PM, Matt Hoosier <matt.hoosier@gmail.com> wrote:
> From: Matt Hoosier <matt.hoosier@garmin.com>
>
> Although the submodules' histories have been fetched during the
> do_fetch() phase, the mechanics used to clone the workdir copy
> of the repo haven't been transferring the actual .git/modules
> directory from the repo fetched into downloads/ during the
> fetch task.
>
> Fix that, and for good measure also explicitly tell Git to avoid
> hitting the network during do_unpack() of the submodules.
>
> Ref:
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12739

Patches for bitbake should be sent to the bitbake-devel mailing list:

  http://lists.openembedded.org/mailman/listinfo/bitbake-devel

> ---
>  bitbake/lib/bb/fetch2/gitsm.py | 12 +++++++++++-
>  1 file changed, 11 insertions(+), 1 deletion(-)
>
> diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py
> index 0aff1008e5..1f3fc44352 100644
> --- a/bitbake/lib/bb/fetch2/gitsm.py
> +++ b/bitbake/lib/bb/fetch2/gitsm.py
> @@ -132,4 +132,14 @@ class GitSM(Git):
>
>          if self.uses_submodules(ud, d, ud.destdir):
>              runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir)
> -            runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir)
> +
> +            # Copy over the submodules' fetched histories too.
> +            if ud.bareclone:
> +                repo_conf = ud.destdir
> +            else:
> +                repo_conf = os.path.join(ud.destdir, '.git')
> +            runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
> +
> +            # Careful not to hit the network during unpacking; all history should already
> +            # be fetched.
> +            runfetchcmd(ud.basecmd + " submodule update --init --recursive --no-fetch", d, workdir=ud.destdir)
> --
> 2.13.6
>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto

Re: [PATCH] bitbake: fetch2/gitsm: avoid live submodule fetching during unpack()

[Matt Hoosier]

[-- Attachment #1: Type: text/plain, Size: 2513 bytes --]

On Wed, May 9, 2018 at 6:27 PM, Andre McCurdy <armccurdy@gmail.com> wrote:

> On Wed, May 9, 2018 at 2:48 PM, Matt Hoosier <matt.hoosier@gmail.com>
> wrote:
> > From: Matt Hoosier <matt.hoosier@garmin.com>
> >
> > Although the submodules' histories have been fetched during the
> > do_fetch() phase, the mechanics used to clone the workdir copy
> > of the repo haven't been transferring the actual .git/modules
> > directory from the repo fetched into downloads/ during the
> > fetch task.
> >
> > Fix that, and for good measure also explicitly tell Git to avoid
> > hitting the network during do_unpack() of the submodules.
> >
> > Ref:
> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=12739
>
> Patches for bitbake should be sent to the bitbake-devel mailing list:
>
>   http://lists.openembedded.org/mailman/listinfo/bitbake-devel



Right, okay. How's that meant to work for stuff developed in-tree with
poky? Do you manually want the 'bitbake/' leading directory stripped off
before emailing the patch?



>
>
> > ---
> >  bitbake/lib/bb/fetch2/gitsm.py | 12 +++++++++++-
> >  1 file changed, 11 insertions(+), 1 deletion(-)
> >
> > diff --git a/bitbake/lib/bb/fetch2/gitsm.py
> b/bitbake/lib/bb/fetch2/gitsm.py
> > index 0aff1008e5..1f3fc44352 100644
> > --- a/bitbake/lib/bb/fetch2/gitsm.py
> > +++ b/bitbake/lib/bb/fetch2/gitsm.py
> > @@ -132,4 +132,14 @@ class GitSM(Git):
> >
> >          if self.uses_submodules(ud, d, ud.destdir):
> >              runfetchcmd(ud.basecmd + " checkout " +
> ud.revisions[ud.names[0]], d, workdir=ud.destdir)
> > -            runfetchcmd(ud.basecmd + " submodule update --init
> --recursive", d, workdir=ud.destdir)
> > +
> > +            # Copy over the submodules' fetched histories too.
> > +            if ud.bareclone:
> > +                repo_conf = ud.destdir
> > +            else:
> > +                repo_conf = os.path.join(ud.destdir, '.git')
> > +            runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir,
> 'modules'), repo_conf), d)
> > +
> > +            # Careful not to hit the network during unpacking; all
> history should already
> > +            # be fetched.
> > +            runfetchcmd(ud.basecmd + " submodule update --init
> --recursive --no-fetch", d, workdir=ud.destdir)
> > --
> > 2.13.6
> >
> > --
> > _______________________________________________
> > yocto mailing list
> > yocto@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/yocto
>

[-- Attachment #2: Type: text/html, Size: 3959 bytes --]

Re: [PATCH] bitbake: fetch2/gitsm: avoid live submodule fetching during unpack()

[Andre McCurdy]

On Wed, May 9, 2018 at 5:40 PM, Matt Hoosier <matt.hoosier@gmail.com> wrote:
> On Wed, May 9, 2018 at 6:27 PM, Andre McCurdy <armccurdy@gmail.com> wrote:
>>
>> On Wed, May 9, 2018 at 2:48 PM, Matt Hoosier <matt.hoosier@gmail.com>
>> wrote:
>> > From: Matt Hoosier <matt.hoosier@garmin.com>
>> >
>> > Although the submodules' histories have been fetched during the
>> > do_fetch() phase, the mechanics used to clone the workdir copy
>> > of the repo haven't been transferring the actual .git/modules
>> > directory from the repo fetched into downloads/ during the
>> > fetch task.
>> >
>> > Fix that, and for good measure also explicitly tell Git to avoid
>> > hitting the network during do_unpack() of the submodules.
>> >
>> > Ref:
>> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=12739
>>
>> Patches for bitbake should be sent to the bitbake-devel mailing list:
>>
>>   http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>
> Right, okay. How's that meant to work for stuff developed in-tree with poky?
> Do you manually want the 'bitbake/' leading directory stripped off before
> emailing the patch?

Yes, patches for bitbake should apply directly to a clone of the bitbake repo.

Manually editing a patch created from poky is certainly do-able, but
applying the patch to a genuine bikebake repo (with "git am -p2 ...")
and re-generating the patch from there may be safer. Whichever you
prefer.

>> > ---
>> >  bitbake/lib/bb/fetch2/gitsm.py | 12 +++++++++++-
>> >  1 file changed, 11 insertions(+), 1 deletion(-)
>> >
>> > diff --git a/bitbake/lib/bb/fetch2/gitsm.py
>> > b/bitbake/lib/bb/fetch2/gitsm.py
>> > index 0aff1008e5..1f3fc44352 100644
>> > --- a/bitbake/lib/bb/fetch2/gitsm.py
>> > +++ b/bitbake/lib/bb/fetch2/gitsm.py
>> > @@ -132,4 +132,14 @@ class GitSM(Git):
>> >
>> >          if self.uses_submodules(ud, d, ud.destdir):
>> >              runfetchcmd(ud.basecmd + " checkout " +
>> > ud.revisions[ud.names[0]], d, workdir=ud.destdir)
>> > -            runfetchcmd(ud.basecmd + " submodule update --init
>> > --recursive", d, workdir=ud.destdir)
>> > +
>> > +            # Copy over the submodules' fetched histories too.
>> > +            if ud.bareclone:
>> > +                repo_conf = ud.destdir
>> > +            else:
>> > +                repo_conf = os.path.join(ud.destdir, '.git')
>> > +            runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir,
>> > 'modules'), repo_conf), d)
>> > +
>> > +            # Careful not to hit the network during unpacking; all
>> > history should already
>> > +            # be fetched.
>> > +            runfetchcmd(ud.basecmd + " submodule update --init
>> > --recursive --no-fetch", d, workdir=ud.destdir)
>> > --
>> > 2.13.6
>> >
>> > --
>> > _______________________________________________
>> > yocto mailing list
>> > yocto@yoctoproject.org
>> > https://lists.yoctoproject.org/listinfo/yocto
>
>

Re: [PATCH] bitbake: fetch2/gitsm: avoid live submodule fetching during unpack()

[Burton, Ross]

On 10 May 2018 at 01:40, Matt Hoosier <matt.hoosier@gmail.com> wrote:
> On Wed, May 9, 2018 at 6:27 PM, Andre McCurdy <armccurdy@gmail.com> wrote:
>>
>> On Wed, May 9, 2018 at 2:48 PM, Matt Hoosier <matt.hoosier@gmail.com>
>> wrote:
>> > From: Matt Hoosier <matt.hoosier@garmin.com>
>> >
>> > Although the submodules' histories have been fetched during the
>> > do_fetch() phase, the mechanics used to clone the workdir copy
>> > of the repo haven't been transferring the actual .git/modules
>> > directory from the repo fetched into downloads/ during the
>> > fetch task.
>> >
>> > Fix that, and for good measure also explicitly tell Git to avoid
>> > hitting the network during do_unpack() of the submodules.
>> >
>> > Ref:
>> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=12739
>>
>> Patches for bitbake should be sent to the bitbake-devel mailing list:
>>
>>   http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>
>
>
> Right, okay. How's that meant to work for stuff developed in-tree with poky?
> Do you manually want the 'bitbake/' leading directory stripped off before
> emailing the patch?

If you're feeling lazy you can just send the patch to bitbake@ from a
poky repo and git will figure it out.  The same works the other way: a
bitbake patch applies fine to a poky repository.

Ross