From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yb0-f174.google.com (mail-yb0-f174.google.com [209.85.213.174]) by mail.openembedded.org (Postfix) with ESMTP id 6C182786C0 for ; Mon, 14 May 2018 21:59:37 +0000 (UTC) Received: by mail-yb0-f174.google.com with SMTP id g140-v6so22365ybf.6 for ; Mon, 14 May 2018 14:59:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=BgwU5vZzL2TvTlGd1BSA8AKxVbnGFd0bzJq3nGV5VGI=; b=LB4eIUo2b+Hd8VAQRZRxXpeHY65rIz239tN3mBzbALWF4Hu/95LWe9XVJNZthdbUDn aD27IzLz0+B22pNaxNZEyS61zg4zXQIxIqRv+3egbvLOchxNdLIeo+dkR6o+47SgWza+ h3u8VUzGGT+91NYz9L0ay/kdst0hOjl3ZHtskFC6Q6C9SH+O2pbuaL/T1ivTJr4A7yez C5e/1fkyw3K9PTppN1Mv9tZyStYqFaRLleyr+oqjLKbmTHARlmXtviv7nTjE35VKUPG3 4y7kqK9x85UaZtgAKi9XHfVtOC2wv3G48foRNvwg6MxXtADqwL51bk/n4mHXXWNIG18Y GHSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=BgwU5vZzL2TvTlGd1BSA8AKxVbnGFd0bzJq3nGV5VGI=; b=fwa17YQOGnPkE4YEN+kxTu+FbAWImbfZr9ZqR8JawJNrx74IrK0argn1sgCi9ySY50 yTHh16rfdNFN/Zjs1tRejO7oCapuVhThmVURfeIySy5JeKM3KicTDmgPagvbvzOBLbHx dU/z2iK0Qa6FsJaWIjnBLDiFrDCynMaQbZ1sDAyL352/QFkGKUfU3YsNnidBlJq37ZcL AUWLJQ6kNEA4hBfW9exF8jxf9YQtuolCHnB83y4qtGhYXF3EZHohh7ByMvT43Y6aIi/3 HmaOzxXPuFqrJn0TYWGAakZwwKEmGXAPCE6VP7RVox5/+fDT6EGnpKU1o2yptN8KPyCj gSaw== X-Gm-Message-State: ALKqPwfo5k4VlEBmrf1iUQTiCeKO/BOs2qc0cp0rIEVfiJCYu+NADFGJ 4cNyFq9fbYl9QN14CsjpQP2clOGotBuWAlcuErg= X-Google-Smtp-Source: AB8JxZp0YAfgMx+3lfWnV/5qHdJGHFIZEJm0DyGg+EC8waLHZJkZKdgeVQeYlfeTJpUU816CyJKgGznQJs1/xUOc2z4= X-Received: by 2002:a5b:bc5:: with SMTP id c5-v6mr3265138ybr.429.1526335178584; Mon, 14 May 2018 14:59:38 -0700 (PDT) MIME-Version: 1.0 References: <20180514185036.104469-1-matt.hoosier@gmail.com> In-Reply-To: From: Matt Hoosier Date: Mon, 14 May 2018 16:59:26 -0500 Message-ID: To: "Burton, Ross" Cc: bitbake-devel Subject: Re: [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack() X-BeenThere: bitbake-devel@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussion that advance bitbake development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 May 2018 21:59:38 -0000 Content-Type: multipart/alternative; boundary="00000000000020b875056c319a77" --00000000000020b875056c319a77 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, May 14, 2018, 15:06 Burton, Ross wrote: > Sorry, but: > > 2018-05-14 18:57:44,968 - oe-selftest - INFO - FAIL: > test_devtool_add_fetch_git (devtool.DevtoolTests) > 2018-05-14 18:57:44,968 - oe-selftest - INFO - > ---------------------------------------------------------------------- > 2018-05-14 18:57:44,968 - oe-selftest - INFO - Traceback (most recent > call last): > File > "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build= /meta/lib/oeqa/core/decorator/__init__.py", > line 32, in wrapped_f > return func(*args, **kwargs) > File > "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build= /meta/lib/oeqa/selftest/cases/devtool.py", > line 403, in test_devtool_add_fetch_git > result =3D runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe, > srcdir, url_rev)) > File > "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build= /meta/lib/oeqa/utils/commands.py", > line 191, in runCmd > raise AssertionError("Command '%s' returned non-zero exit status > %d:\n%s" % (command, result.status, exc_output)) > AssertionError: Command 'devtool add mraa /tmp/devtoolqa0aghorcw/mraa > -f "gitsm:// > git.yoctoproject.org/mraa;rev=3Dae127b19a50aa54255e4330ccfdd9a5d058e581d" > -V 1.5' returned non-zero exit status 1: > ... > > ERROR: Fetcher failure: Fetch command export PSEUDO_DISABLED=3D1; export > > PATH=3D"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftes= t/build/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/y= octo-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts:/home/pokyb= uild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/wor= k/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin/x86_64-poky-linux:= /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bu= ild/tmp/work/recipetool-utucnynk/work/recipe-sysroot/usr/bin/crossscripts:/= home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bui= ld/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/sbin:/home/p= okybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp= /work/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin:/home/pokybuil= d/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/r= ecipetool-utucnynk/work/recipe-sysroot-native/sbin:/home/pokybuild/yocto-au= tobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-= utucnynk/work/recipe-sysroot-native/bin:/home/pokybuild/yocto-autobuilder/y= octo-worker/nightly-oe-selftest/build/bitbake/bin:/home/pokybuild/yocto-aut= obuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/hosttools"; > export HOME=3D"/home/pokybuild"; git -c core.fsyncobjectfiles=3D0 > submodule update --init --recursive --no-fetch failed with exit code > 1, output: > Submodule 'doxygen2jsdoc' > (https://github.com/arfoll/doxygen2jsdoc.git) registered for path > 'doxygen2jsdoc' > fatal: reference is not a tree: 9cc90b7976252b2d14b7956230c5870097e1f008 > Unable to checkout '9cc90b7976252b2d14b7956230c5870097e1f008' in > submodule path 'doxygen2jsdoc' > > Ross > > On 14 May 2018 at 19:50, Matt Hoosier wrote: > > Although the submodules' histories have been fetched during the > > do_fetch() phase, the mechanics used to clone the workdir copy > > of the repo haven't been transferring the actual .git/modules > > directory from the repo fetched into downloads/ during the > > fetch task. > > > > Fix that, and for good measure also explicitly tell Git to avoid > > hitting the network during do_unpack() of the submodules. > > > > [YOCTO #12739] > > > > Signed-off-by: Matt Hoosier > > --- > > lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++- > > 1 file changed, 27 insertions(+), 1 deletion(-) > > > > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py > > index 0aff1008..c5019e98 100644 > > --- a/lib/bb/fetch2/gitsm.py > > +++ b/lib/bb/fetch2/gitsm.py > > @@ -132,4 +132,30 @@ class GitSM(Git): > > > > if self.uses_submodules(ud, d, ud.destdir): > > runfetchcmd(ud.basecmd + " checkout " + > ud.revisions[ud.names[0]], d, workdir=3Dud.destdir) > > - runfetchcmd(ud.basecmd + " submodule update --init > --recursive", d, workdir=3Dud.destdir) > > + > > + updateflags =3D "" > > + > > + # Copy over the submodules' fetched histories too. > > + if ud.bareclone: > > + repo_conf =3D ud.destdir > > + else: > > + repo_conf =3D os.path.join(ud.destdir, '.git') > > + > > + if os.path.exists(ud.clonedir): > > + # This is not a copy unpacked from a shallow mirror > clone. So > > + # the manual intervention to populate the .git/modules > done > > + # in clone_shallow_local() won't have been done yet. > > + runfetchcmd("cp -fpPRH %s %s" % > (os.path.join(ud.clonedir, 'modules'), repo_conf), d) > > + updateflags +=3D " --no-fetch" > > + elif os.path.exists(os.path.join(repo_conf, 'modules')): > > + # Unpacked from a shallow mirror clone. Manual > population of > > + # .git/modules is already done. > > + updateflags +=3D " --no-fetch" > > + else: > > + # This isn't fatal; git-submodule will just fetch it > > + # during do_unpack(). > > + bb.error("submodule history not retrieved during > do_fetch()") > > + > > + # Careful not to hit the network during unpacking; all > history should already > > + # be fetched. > > + runfetchcmd(ud.basecmd + " submodule update --init > --recursive %s" % updateflags, d, workdir=3Dud.destdir) > > -- > > 2.13.6 > > > > -- > > _______________________________________________ > > bitbake-devel mailing list > > bitbake-devel@lists.openembedded.org > > http://lists.openembedded.org/mailman/listinfo/bitbake-devel Okay, I'll have a look at that. Are there any other test suites I should know about? > > --00000000000020b875056c319a77 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Mon, = May 14, 2018, 15:06 Burton, Ross <ross.burton@intel.com> wrote= :
Sorry, but:

2018-05-14 18:57:44,968 - oe-selftest - INFO - FAIL:
test_devtool_add_fetch_git (devtool.DevtoolTests)
2018-05-14 18:57:44,968 - oe-selftest - INFO -
----------------------------------------------------------------------
2018-05-14 18:57:44,968 - oe-selftest - INFO - Traceback (most recent
call last):
=C2=A0 File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe= -selftest/build/meta/lib/oeqa/core/decorator/__init__.py",
line 32, in wrapped_f
=C2=A0 =C2=A0 return func(*args, **kwargs)
=C2=A0 File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe= -selftest/build/meta/lib/oeqa/selftest/cases/devtool.py",
line 403, in test_devtool_add_fetch_git
=C2=A0 =C2=A0 result =3D runCmd('devtool add %s %s -f "%s" -V= 1.5' % (testrecipe,
srcdir, url_rev))
=C2=A0 File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe= -selftest/build/meta/lib/oeqa/utils/commands.py",
line 191, in runCmd
=C2=A0 =C2=A0 raise AssertionError("Command '%s' returned non-= zero exit status
%d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'devtool add mraa /tmp/devtoolqa0aghorcw/mraa -f "gitsm://git.yoctoproject.org/mraa;rev=3Dae127b19a50aa54255e4330cc= fdd9a5d058e581d"
-V 1.5' returned non-zero exit status 1:
...

ERROR: Fetcher failure: Fetch command export PSEUDO_DISABLED=3D1; export PATH=3D"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-self= test/build/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuil= d/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts:/home/po= kybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/= work/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin/x86_64-poky-lin= ux:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build= /build/tmp/work/recipetool-utucnynk/work/recipe-sysroot/usr/bin/crossscript= s:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/= build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/sbin:/hom= e/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/= tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin:/home/pokyb= uild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/wor= k/recipetool-utucnynk/work/recipe-sysroot-native/sbin:/home/pokybuild/yocto= -autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipeto= ol-utucnynk/work/recipe-sysroot-native/bin:/home/pokybuild/yocto-autobuilde= r/yocto-worker/nightly-oe-selftest/build/bitbake/bin:/home/pokybuild/yocto-= autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/hosttools"= ;;
export HOME=3D"/home/pokybuild"; git -c core.fsyncobjectfiles=3D0=
submodule update --init --recursive=C2=A0 --no-fetch failed with exit code<= br> 1, output:
Submodule 'doxygen2jsdoc'
(https://github.com/arfoll/doxygen2= jsdoc.git) registered for path
'doxygen2jsdoc'
fatal: reference is not a tree: 9cc90b7976252b2d14b7956230c5870097e1f008 Unable to checkout '9cc90b7976252b2d14b7956230c5870097e1f008' in submodule path 'doxygen2jsdoc'

Ross

On 14 May 2018 at 19:50, Matt Hoosier <matt.hoosier@gmail= .com> wrote:
> Although the submodules' histories have been fetched during the > do_fetch() phase, the mechanics used to clone the workdir copy
> of the repo haven't been transferring the actual .git/modules
> directory from the repo fetched into downloads/ during the
> fetch task.
>
> Fix that, and for good measure also explicitly tell Git to avoid
> hitting the network during do_unpack() of the submodules.
>
> [YOCTO #12739]
>
> Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com<= /a>>
> ---
>=C2=A0 lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++-
>=C2=A0 1 file changed, 27 insertions(+), 1 deletion(-)
>
> diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
> index 0aff1008..c5019e98 100644
> --- a/lib/bb/fetch2/gitsm.py
> +++ b/lib/bb/fetch2/gitsm.py
> @@ -132,4 +132,30 @@ class GitSM(Git):
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if self.uses_submodules(ud, d, ud.de= stdir):
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 runfetchcmd(ud.basecmd= + " checkout " + ud.revisions[ud.names[0]], d, workdir=3Dud.dest= dir)
> -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 runfetchcmd(ud.basecmd + &q= uot; submodule update --init --recursive", d, workdir=3Dud.destdir) > +
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 updateflags =3D ""= ;
> +
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # Copy over the submodules&= #39; fetched histories too.
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if ud.bareclone:
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 repo_conf =3D= ud.destdir
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 else:
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 repo_conf =3D= os.path.join(ud.destdir, '.git')
> +
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if os.path.exists(ud.cloned= ir):
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # This is not= a copy unpacked from a shallow mirror clone. So
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # the manual = intervention to populate the .git/modules done
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # in clone_sh= allow_local() won't have been done yet.
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 runfetchcmd(&= quot;cp -fpPRH %s %s" % (os.path.join(ud.clonedir, 'modules'),= repo_conf), d)
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 updateflags += =3D " --no-fetch"
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 elif os.path.exists(os.path= .join(repo_conf, 'modules')):
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # Unpacked fr= om a shallow mirror clone. Manual population of
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # .git/module= s is already done.
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 updateflags += =3D " --no-fetch"
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 else:
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # This isn= 9;t fatal; git-submodule will just fetch it
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # during do_u= npack().
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 bb.error(&quo= t;submodule history not retrieved during do_fetch()")
> +
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # Careful not to hit the ne= twork during unpacking; all history should already
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # be fetched.
> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 runfetchcmd(ud.basecmd + &q= uot; submodule update --init --recursive %s" % updateflags, d, workdir= =3Dud.destdir)
> --
> 2.13.6
>
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org<= br> > http://lists.= openembedded.org/mailman/listinfo/bitbake-devel

Okay, I'll have a look at that. Are there any other test sui= tes I should know about?

--00000000000020b875056c319a77--