Re: [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack()

[Matt Hoosier <matt.hoosier@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 6217 bytes --]

On Mon, May 14, 2018, 15:06 Burton, Ross <ross.burton@intel.com> wrote:

> Sorry, but:
>
> 2018-05-14 18:57:44,968 - oe-selftest - INFO - FAIL:
> test_devtool_add_fetch_git (devtool.DevtoolTests)
> 2018-05-14 18:57:44,968 - oe-selftest - INFO -
> ----------------------------------------------------------------------
> 2018-05-14 18:57:44,968 - oe-selftest - INFO - Traceback (most recent
> call last):
>   File
> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/core/decorator/__init__.py",
> line 32, in wrapped_f
>     return func(*args, **kwargs)
>   File
> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/devtool.py",
> line 403, in test_devtool_add_fetch_git
>     result = runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe,
> srcdir, url_rev))
>   File
> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/utils/commands.py",
> line 191, in runCmd
>     raise AssertionError("Command '%s' returned non-zero exit status
> %d:\n%s" % (command, result.status, exc_output))
> AssertionError: Command 'devtool add mraa /tmp/devtoolqa0aghorcw/mraa
> -f "gitsm://
> git.yoctoproject.org/mraa;rev=ae127b19a50aa54255e4330ccfdd9a5d058e581d"
> -V 1.5' returned non-zero exit status 1:
> ...
>
> ERROR: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export
>
> PATH="/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot/usr/bin/crossscripts:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bitbake/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/hosttools";
> export HOME="/home/pokybuild"; git -c core.fsyncobjectfiles=0
> submodule update --init --recursive  --no-fetch failed with exit code
> 1, output:
> Submodule 'doxygen2jsdoc'
> (https://github.com/arfoll/doxygen2jsdoc.git) registered for path
> 'doxygen2jsdoc'
> fatal: reference is not a tree: 9cc90b7976252b2d14b7956230c5870097e1f008
> Unable to checkout '9cc90b7976252b2d14b7956230c5870097e1f008' in
> submodule path 'doxygen2jsdoc'
>
> Ross
>
> On 14 May 2018 at 19:50, Matt Hoosier <matt.hoosier@gmail.com> wrote:
> > Although the submodules' histories have been fetched during the
> > do_fetch() phase, the mechanics used to clone the workdir copy
> > of the repo haven't been transferring the actual .git/modules
> > directory from the repo fetched into downloads/ during the
> > fetch task.
> >
> > Fix that, and for good measure also explicitly tell Git to avoid
> > hitting the network during do_unpack() of the submodules.
> >
> > [YOCTO #12739]
> >
> > Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
> > ---
> >  lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++-
> >  1 file changed, 27 insertions(+), 1 deletion(-)
> >
> > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
> > index 0aff1008..c5019e98 100644
> > --- a/lib/bb/fetch2/gitsm.py
> > +++ b/lib/bb/fetch2/gitsm.py
> > @@ -132,4 +132,30 @@ class GitSM(Git):
> >
> >          if self.uses_submodules(ud, d, ud.destdir):
> >              runfetchcmd(ud.basecmd + " checkout " +
> ud.revisions[ud.names[0]], d, workdir=ud.destdir)
> > -            runfetchcmd(ud.basecmd + " submodule update --init
> --recursive", d, workdir=ud.destdir)
> > +
> > +            updateflags = ""
> > +
> > +            # Copy over the submodules' fetched histories too.
> > +            if ud.bareclone:
> > +                repo_conf = ud.destdir
> > +            else:
> > +                repo_conf = os.path.join(ud.destdir, '.git')
> > +
> > +            if os.path.exists(ud.clonedir):
> > +                # This is not a copy unpacked from a shallow mirror
> clone. So
> > +                # the manual intervention to populate the .git/modules
> done
> > +                # in clone_shallow_local() won't have been done yet.
> > +                runfetchcmd("cp -fpPRH %s %s" %
> (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
> > +                updateflags += " --no-fetch"
> > +            elif os.path.exists(os.path.join(repo_conf, 'modules')):
> > +                # Unpacked from a shallow mirror clone. Manual
> population of
> > +                # .git/modules is already done.
> > +                updateflags += " --no-fetch"
> > +            else:
> > +                # This isn't fatal; git-submodule will just fetch it
> > +                # during do_unpack().
> > +                bb.error("submodule history not retrieved during
> do_fetch()")
> > +
> > +            # Careful not to hit the network during unpacking; all
> history should already
> > +            # be fetched.
> > +            runfetchcmd(ud.basecmd + " submodule update --init
> --recursive %s" % updateflags, d, workdir=ud.destdir)
> > --
> > 2.13.6
> >
> > --
> > _______________________________________________
> > bitbake-devel mailing list
> > bitbake-devel@lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/bitbake-devel


Okay, I'll have a look at that. Are there any other test suites I should
know about?

>
>

[-- Attachment #2: Type: text/html, Size: 8227 bytes --]