From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yw0-f174.google.com (mail-yw0-f174.google.com [209.85.161.174]) by mail.openembedded.org (Postfix) with ESMTP id D280274966 for ; Thu, 17 May 2018 19:17:44 +0000 (UTC) Received: by mail-yw0-f174.google.com with SMTP id p144-v6so1667916ywg.7 for ; Thu, 17 May 2018 12:17:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=yxLAR7/YR5/UY7JKhVMxZTCFuAVwmDvbOkmt5IdI0a0=; b=vS5O0cTjx5UM/gveu9OgqZy8xPOi09/xkOHxNYZFjgGbQmK/A15WfuUdhe1gCIyWun Km99E48MiP1LAEq57tl12iFTcrP4aa7GRxcLjhX0ge/heNsWu9to0B1vMqIzCARB8wlh 1cPg3HhlrmQyy9V9MjyCfh/6Y/XUoQO2vTzW3r1EqmL7iaUjiHzFHxTfyXTsOs3WP0j4 DDfT2dIAe76fZnQx9RyQpXMElMjPG/HP49TGm+jy8aUFfFQbhimY6Zfh1spx/mVWBq4i NXMPfeHblA6tYlYySDGIDkDEPPSZTLR0kboLycZoJWu/QuAMFtwVGtSfr8P7VCfTjCfG ebGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=yxLAR7/YR5/UY7JKhVMxZTCFuAVwmDvbOkmt5IdI0a0=; b=HRKttsTglfw1aa/xBU67vZbzow4F0SmBw8vQ/fMXHYAr7eDYt/A8NaXD4a05iboXnS WNQaZqovTmPfNPRo2+MgRQZPWIfoBNoeCItMIiIVXdxBeEUd1g7mBdvjC0ey8wJ9xGMQ t0IJhsP3TS/PtAlKYYFFReieZhKBayNKSYXq4FvaM0f22rCcRmQHAsTxhClPyxQ7yEsO aSOlNYN4XSgb6OvtF9RZsnC/AMQd6H9GgLNuYxHZRpnctdrTT1t8xc0+FayRsbwtV5wy z6uo6URz7FxpzCXdKtQ8BFnPOgMb9imb5tUpaxLGiK9LMr2Lk9hpOcVy33ot/pN8yLuA vjXQ== X-Gm-Message-State: ALKqPwcHnbmuc3L4alyoAi7Fbqh7AUGz8kEWULcEhEdqzHP1tjHpJUXR /3jFZSdwdaeQd1eAAF1+C3rJ+WmGNaxyIaMhJxY= X-Google-Smtp-Source: AB8JxZr0sDCo+ZxHUhFRjSWrU5Rnv/ZLcAmL/HgVXxH3vDjt4iVyb7jQRXIxN6pGSvk1/k4vtYWv+9EbkrosV1b/d/w= X-Received: by 2002:a0d:e987:: with SMTP id s129-v6mr3296281ywe.510.1526584665335; Thu, 17 May 2018 12:17:45 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a25:748c:0:0:0:0:0 with HTTP; Thu, 17 May 2018 12:17:04 -0700 (PDT) In-Reply-To: References: <20180514185036.104469-1-matt.hoosier@gmail.com> From: Matt Hoosier Date: Thu, 17 May 2018 14:17:04 -0500 Message-ID: To: "Burton, Ross" Cc: bitbake-devel Subject: Re: [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack() X-BeenThere: bitbake-devel@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussion that advance bitbake development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 May 2018 19:17:45 -0000 Content-Type: multipart/alternative; boundary="000000000000b26f54056c6bb02d" --000000000000b26f54056c6bb02d Content-Type: text/plain; charset="UTF-8" Right; I got enough context from your copied/pasted failure report to duplicate it. I'm just wondering if there are other test batteries besides bitbake-selftest and oe-selftest that are gates to clear? -Matt On Thu, May 17, 2018 at 11:28 AM, Burton, Ross wrote: > Hi Matt, > > If you run oe-selftest -r devtool then it will run the tests which are > failing. > > Ross > > On 14 May 2018 at 22:59, Matt Hoosier wrote: > > On Mon, May 14, 2018, 15:06 Burton, Ross wrote: > >> > >> Sorry, but: > >> > >> 2018-05-14 18:57:44,968 - oe-selftest - INFO - FAIL: > >> test_devtool_add_fetch_git (devtool.DevtoolTests) > >> 2018-05-14 18:57:44,968 - oe-selftest - INFO - > >> ---------------------------------------------------------------------- > >> 2018-05-14 18:57:44,968 - oe-selftest - INFO - Traceback (most recent > >> call last): > >> File > >> "/home/pokybuild/yocto-autobuilder/yocto-worker/ > nightly-oe-selftest/build/meta/lib/oeqa/core/decorator/__init__.py", > >> line 32, in wrapped_f > >> return func(*args, **kwargs) > >> File > >> "/home/pokybuild/yocto-autobuilder/yocto-worker/ > nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/devtool.py", > >> line 403, in test_devtool_add_fetch_git > >> result = runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe, > >> srcdir, url_rev)) > >> File > >> "/home/pokybuild/yocto-autobuilder/yocto-worker/ > nightly-oe-selftest/build/meta/lib/oeqa/utils/commands.py", > >> line 191, in runCmd > >> raise AssertionError("Command '%s' returned non-zero exit status > >> %d:\n%s" % (command, result.status, exc_output)) > >> AssertionError: Command 'devtool add mraa /tmp/devtoolqa0aghorcw/mraa > >> -f > >> "gitsm://git.yoctoproject.org/mraa;rev=ae127b19a50aa54255e4330ccfdd9a > 5d058e581d" > >> -V 1.5' returned non-zero exit status 1: > >> ... > >> > >> ERROR: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export > >> > >> PATH="/home/pokybuild/yocto-autobuilder/yocto-worker/ > nightly-oe-selftest/build/build/tmp/sysroots-uninative/ > x86_64-linux/usr/bin:/home/pokybuild/yocto-autobuilder/ > yocto-worker/nightly-oe-selftest/build/scripts:/home/ > pokybuild/yocto-autobuilder/yocto-worker/nightly-oe- > selftest/build/build/tmp/work/recipetool-utucnynk/work/ > recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/ > pokybuild/yocto-autobuilder/yocto-worker/nightly-oe- > selftest/build/build/tmp/work/recipetool-utucnynk/work/ > recipe-sysroot/usr/bin/crossscripts:/home/pokybuild/ > yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/ > recipetool-utucnynk/work/recipe-sysroot-native/usr/ > sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/ > nightly-oe-selftest/build/build/tmp/work/recipetool- > utucnynk/work/recipe-sysroot-native/usr/bin:/home/ > pokybuild/yocto-autobuilder/yocto-worker/nightly-oe- > selftest/build/build/tmp/work/recipetool-utucnynk/work/ > recipe-sysroot-native/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/ > nightly-oe-selftest/build/build/tmp/work/recipetool- > utucnynk/work/recipe-sysroot-native/bin:/home/pokybuild/ > yocto-autobuilder/yocto-worker/nightly-oe-selftest/ > build/bitbake/bin:/home/pokybuild/yocto-autobuilder/ > yocto-worker/nightly-oe-selftest/build/build/tmp/hosttools"; > >> export HOME="/home/pokybuild"; git -c core.fsyncobjectfiles=0 > >> submodule update --init --recursive --no-fetch failed with exit code > >> 1, output: > >> Submodule 'doxygen2jsdoc' > >> (https://github.com/arfoll/doxygen2jsdoc.git) registered for path > >> 'doxygen2jsdoc' > >> fatal: reference is not a tree: 9cc90b7976252b2d14b7956230c587 > 0097e1f008 > >> Unable to checkout '9cc90b7976252b2d14b7956230c5870097e1f008' in > >> submodule path 'doxygen2jsdoc' > >> > >> Ross > >> > >> On 14 May 2018 at 19:50, Matt Hoosier wrote: > >> > Although the submodules' histories have been fetched during the > >> > do_fetch() phase, the mechanics used to clone the workdir copy > >> > of the repo haven't been transferring the actual .git/modules > >> > directory from the repo fetched into downloads/ during the > >> > fetch task. > >> > > >> > Fix that, and for good measure also explicitly tell Git to avoid > >> > hitting the network during do_unpack() of the submodules. > >> > > >> > [YOCTO #12739] > >> > > >> > Signed-off-by: Matt Hoosier > >> > --- > >> > lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++- > >> > 1 file changed, 27 insertions(+), 1 deletion(-) > >> > > >> > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py > >> > index 0aff1008..c5019e98 100644 > >> > --- a/lib/bb/fetch2/gitsm.py > >> > +++ b/lib/bb/fetch2/gitsm.py > >> > @@ -132,4 +132,30 @@ class GitSM(Git): > >> > > >> > if self.uses_submodules(ud, d, ud.destdir): > >> > runfetchcmd(ud.basecmd + " checkout " + > >> > ud.revisions[ud.names[0]], d, workdir=ud.destdir) > >> > - runfetchcmd(ud.basecmd + " submodule update --init > >> > --recursive", d, workdir=ud.destdir) > >> > + > >> > + updateflags = "" > >> > + > >> > + # Copy over the submodules' fetched histories too. > >> > + if ud.bareclone: > >> > + repo_conf = ud.destdir > >> > + else: > >> > + repo_conf = os.path.join(ud.destdir, '.git') > >> > + > >> > + if os.path.exists(ud.clonedir): > >> > + # This is not a copy unpacked from a shallow mirror > >> > clone. So > >> > + # the manual intervention to populate the > .git/modules > >> > done > >> > + # in clone_shallow_local() won't have been done yet. > >> > + runfetchcmd("cp -fpPRH %s %s" % > >> > (os.path.join(ud.clonedir, 'modules'), repo_conf), d) > >> > + updateflags += " --no-fetch" > >> > + elif os.path.exists(os.path.join(repo_conf, 'modules')): > >> > + # Unpacked from a shallow mirror clone. Manual > >> > population of > >> > + # .git/modules is already done. > >> > + updateflags += " --no-fetch" > >> > + else: > >> > + # This isn't fatal; git-submodule will just fetch it > >> > + # during do_unpack(). > >> > + bb.error("submodule history not retrieved during > >> > do_fetch()") > >> > + > >> > + # Careful not to hit the network during unpacking; all > >> > history should already > >> > + # be fetched. > >> > + runfetchcmd(ud.basecmd + " submodule update --init > >> > --recursive %s" % updateflags, d, workdir=ud.destdir) > >> > -- > >> > 2.13.6 > >> > > >> > -- > >> > _______________________________________________ > >> > bitbake-devel mailing list > >> > bitbake-devel@lists.openembedded.org > >> > http://lists.openembedded.org/mailman/listinfo/bitbake-devel > > > > > > Okay, I'll have a look at that. Are there any other test suites I should > > know about? > >> > >> > > > --000000000000b26f54056c6bb02d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Right; I got enough context from your copied/pasted failur= e report to duplicate it. I'm just wondering if there are other test ba= tteries besides bitbake-selftest and oe-selftest that are gates to clear?
-Matt

On Thu, May 17, 2018 at 11:28 AM, Burton, Ross = <ross.burton@= intel.com> wrote:
Hi Matt,=

If you run oe-selftest -r devtool then it will run the tests which are fail= ing.

Ross

On 14 May 2018 at 22:59, Matt Hoosier <matt.hoosier@gmail.com> wrote:
> On Mon, May 14, 2018, 15:06 Burton, Ross <ross.burton@intel.com> wrote:
>>
>> Sorry, but:
>>
>> 2018-05-14 18:57:44,968 - oe-selftest - INFO - FAIL:
>> test_devtool_add_fetch_git (devtool.DevtoolTests)
>> 2018-05-14 18:57:44,968 - oe-selftest - INFO -
>> ------------------------------------------------------------<= wbr>----------
>> 2018-05-14 18:57:44,968 - oe-selftest - INFO - Traceback (most rec= ent
>> call last):
>>=C2=A0 =C2=A0File
>> "/home/pokybuild/yocto-autobuilder/yocto-worker/nig= htly-oe-selftest/build/meta/lib/oeqa/core/decorator/__init__.py&q= uot;,
>> line 32, in wrapped_f
>>=C2=A0 =C2=A0 =C2=A0return func(*args, **kwargs)
>>=C2=A0 =C2=A0File
>> "/home/pokybuild/yocto-autobuilder/yocto-worker/nig= htly-oe-selftest/build/meta/lib/oeqa/selftest/cases/devtool.py&qu= ot;,
>> line 403, in test_devtool_add_fetch_git
>>=C2=A0 =C2=A0 =C2=A0result =3D runCmd('devtool add %s %s -f &qu= ot;%s" -V 1.5' % (testrecipe,
>> srcdir, url_rev))
>>=C2=A0 =C2=A0File
>> "/home/pokybuild/yocto-autobuilder/yocto-worker/nig= htly-oe-selftest/build/meta/lib/oeqa/utils/commands.py",
>> line 191, in runCmd
>>=C2=A0 =C2=A0 =C2=A0raise AssertionError("Command '%s'= returned non-zero exit status
>> %d:\n%s" % (command, result.status, exc_output))
>> AssertionError: Command 'devtool add mraa /tmp/devtoolqa0aghor= cw/mraa
>> -f
>> "gitsm://git.yoctoproject.org/mraa;rev=3Dae127b19a50aa54255e4330ccfdd9a<= wbr>5d058e581d"
>> -V 1.5' returned non-zero exit status 1:
>> ...
>>
>> ERROR: Fetcher failure: Fetch command export PSEUDO_DISABLED=3D1; = export
>>
>> PATH=3D"/home/pokybuild/yocto-autobuilder/yocto-worker/<= wbr>nightly-oe-selftest/build/build/tmp/sysroots-uninative/x86_64= -linux/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nig= htly-oe-selftest/build/scripts:/home/pokybuild/yocto-autobuilder/= yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipe= tool-utucnynk/work/recipe-sysroot-native/usr/bin/x86_64-poky-linu= x:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-= selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sys= root/usr/bin/crossscripts:/home/pokybuild/yocto-autobuilder/yocto= -worker/nightly-oe-selftest/build/build/tmp/work/recipetool-= utucnynk/work/recipe-sysroot-native/usr/sbin:/home/pokybuild/yoct= o-autobuilder/yocto-worker/nightly-oe-selftest/build/build/t= mp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin:/h= ome/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-self= test/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot= -native/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/= nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work= /recipe-sysroot-native/bin:/home/pokybuild/yocto-autobuilder/yoct= o-worker/nightly-oe-selftest/build/bitbake/bin:/home/pokybui= ld/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build= /tmp/hosttools";
>> export HOME=3D"/home/pokybuild"; git -c core.fsyncobject= files=3D0
>> submodule update --init --recursive=C2=A0 --no-fetch failed with e= xit code
>> 1, output:
>> Submodule 'doxygen2jsdoc'
>> (https://github.com/arfoll/doxygen2jsdoc.gi= t) registered for path
>> 'doxygen2jsdoc'
>> fatal: reference is not a tree: 9cc90b7976252b2d14b7956230c5870097e1f008
>> Unable to checkout '9cc90b7976252b2d14b7956230c58700= 97e1f008' in
>> submodule path 'doxygen2jsdoc'
>>
>> Ross
>>
>> On 14 May 2018 at 19:50, Matt Hoosier <matt.hoosier@gmail.com> wrote:
>> > Although the submodules' histories have been fetched duri= ng the
>> > do_fetch() phase, the mechanics used to clone the workdir cop= y
>> > of the repo haven't been transferring the actual .git/mod= ules
>> > directory from the repo fetched into downloads/ during the >> > fetch task.
>> >
>> > Fix that, and for good measure also explicitly tell Git to av= oid
>> > hitting the network during do_unpack() of the submodules.
>> >
>> > [YOCTO #12739]
>> >
>> > Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
>> > ---
>> >=C2=A0 lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++= -
>> >=C2=A0 1 file changed, 27 insertions(+), 1 deletion(-)
>> >
>> > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py<= br> >> > index 0aff1008..c5019e98 100644
>> > --- a/lib/bb/fetch2/gitsm.py
>> > +++ b/lib/bb/fetch2/gitsm.py
>> > @@ -132,4 +132,30 @@ class GitSM(Git):
>> >
>> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if self.uses_submodules(ud,= d, ud.destdir):
>> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 runfetchcmd(u= d.basecmd + " checkout " +
>> > ud.revisions[ud.names[0]], d, workdir=3Dud.destdir)
>> > -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 runfetchcmd(ud.bas= ecmd + " submodule update --init
>> > --recursive", d, workdir=3Dud.destdir)
>> > +
>> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 updateflags =3D &q= uot;"
>> > +
>> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # Copy over the su= bmodules' fetched histories too.
>> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if ud.bareclone: >> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 repo= _conf =3D ud.destdir
>> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 else:
>> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 repo= _conf =3D os.path.join(ud.destdir, '.git')
>> > +
>> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if os.path.exists(= ud.clonedir):
>> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # Th= is is not a copy unpacked from a shallow mirror
>> > clone. So
>> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # th= e manual intervention to populate the .git/modules
>> > done
>> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # in= clone_shallow_local() won't have been done yet.
>> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 runf= etchcmd("cp -fpPRH %s %s" %
>> > (os.path.join(ud.clonedir, 'modules'), repo_conf), d)=
>> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 upda= teflags +=3D " --no-fetch"
>> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 elif os.path.exist= s(os.path.join(repo_conf, 'modules')):
>> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # Un= packed from a shallow mirror clone. Manual
>> > population of
>> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # .g= it/modules is already done.
>> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 upda= teflags +=3D " --no-fetch"
>> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 else:
>> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # Th= is isn't fatal; git-submodule will just fetch it
>> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # du= ring do_unpack().
>> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 bb.e= rror("submodule history not retrieved during
>> > do_fetch()")
>> > +
>> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # Careful not to h= it the network during unpacking; all
>> > history should already
>> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # be fetched.
>> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 runfetchcmd(ud.bas= ecmd + " submodule update --init
>> > --recursive %s" % updateflags, d, workdir=3Dud.destdir)<= br> >> > --
>> > 2.13.6
>> >
>> > --
>> > _______________________________________________
>> > bitbake-devel mailing list
>> > bitba= ke-devel@lists.openembedded.org
>> > http://lists.openembedded.= org/mailman/listinfo/bitbake-devel
>
>
> Okay, I'll have a look at that. Are there any other test suites I = should
> know about?
>>
>>
>

--000000000000b26f54056c6bb02d--