Re: [PATCH] fetch/gitsm: avoid live submodule fetching during unpack()

From: "Burton, Ross" <ross.burton@intel.com>
To: Matt Hoosier <matt.hoosier@gmail.com>
Cc: bitbake-devel <bitbake-devel@lists.openembedded.org>
Subject: Re: [PATCH] fetch/gitsm: avoid live submodule fetching during unpack()
Date: Fri, 11 May 2018 13:28:34 +0100	[thread overview]
Message-ID: <CAJTo0LYFNknuqk_xyecvEgDFweFr5vXFwu3N5bNkLZDdcq=4Nw@mail.gmail.com> (raw)
In-Reply-To: <20180510023357.96409-1-matt.hoosier@gmail.com>

This breaks bitbake-selftest:

$ bitbake-selftest  bb.tests.fetch.GitShallowTest
........................E

ERROR: test_shallow_submodules (bb.tests.fetch.GitShallowTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ross/Yocto/poky/bitbake/lib/bb/tests/fetch.py", line
1353, in test_shallow_submodules
    fetcher, ud = self.fetch_shallow(uri)
  File "/home/ross/Yocto/poky/bitbake/lib/bb/tests/fetch.py", line
1198, in fetch_shallow
    fetcher, ud = self.fetch_and_unpack(uri)
  File "/home/ross/Yocto/poky/bitbake/lib/bb/tests/fetch.py", line
1179, in fetch_and_unpack
    fetcher.unpack(self.d.getVar('WORKDIR'))
  File "/home/ross/Yocto/poky/bitbake/lib/bb/fetch2/__init__.py", line
1762, in unpack
    ud.method.unpack(ud, root, self.d)
  File "/home/ross/Yocto/poky/bitbake/lib/bb/fetch2/gitsm.py", line
141, in unpack
    runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir,
'modules'), repo_conf), d)
  File "/home/ross/Yocto/poky/bitbake/lib/bb/fetch2/__init__.py", line
885, in runfetchcmd
    raise FetchError(error_message)
bb.fetch2.FetchError: Fetcher failure: Fetch command export
PSEUDO_DISABLED=1; cp -pr
/tmp/tmpxi2ydemw/download/git2/tmp.tmpxi2ydemw.gitsource/modules
/tmp/tmpxi2ydemw/git/.git failed with exit code 1, output:
cp: cannot stat
'/tmp/tmpxi2ydemw/download/git2/tmp.tmpxi2ydemw.gitsource/modules': No
such file or directory

Ross

On 10 May 2018 at 03:33, Matt Hoosier <matt.hoosier@gmail.com> wrote:
> Although the submodules' histories have been fetched during the
> do_fetch() phase, the mechanics used to clone the workdir copy
> of the repo haven't been transferring the actual .git/modules
> directory from the repo fetched into downloads/ during the
> fetch task.
>
> Fix that, and for good measure also explicitly tell Git to avoid
> hitting the network during do_unpack() of the submodules.
>
> [YOCTO #12739]
>
> Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
> ---
>  lib/bb/fetch2/gitsm.py | 12 +++++++++++-
>  1 file changed, 11 insertions(+), 1 deletion(-)
>
> diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
> index 0aff1008..1f3fc443 100644
> --- a/lib/bb/fetch2/gitsm.py
> +++ b/lib/bb/fetch2/gitsm.py
> @@ -132,4 +132,14 @@ class GitSM(Git):
>
>          if self.uses_submodules(ud, d, ud.destdir):
>              runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir)
> -            runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir)
> +
> +            # Copy over the submodules' fetched histories too.
> +            if ud.bareclone:
> +                repo_conf = ud.destdir
> +            else:
> +                repo_conf = os.path.join(ud.destdir, '.git')
> +            runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
> +
> +            # Careful not to hit the network during unpacking; all history should already
> +            # be fetched.
> +            runfetchcmd(ud.basecmd + " submodule update --init --recursive --no-fetch", d, workdir=ud.destdir)
> --
> 2.13.6
>
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel