* [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack() @ 2018-05-14 18:50 Matt Hoosier 2018-05-14 20:06 ` Burton, Ross 0 siblings, 1 reply; 6+ messages in thread From: Matt Hoosier @ 2018-05-14 18:50 UTC (permalink / raw) To: bitbake-devel Although the submodules' histories have been fetched during the do_fetch() phase, the mechanics used to clone the workdir copy of the repo haven't been transferring the actual .git/modules directory from the repo fetched into downloads/ during the fetch task. Fix that, and for good measure also explicitly tell Git to avoid hitting the network during do_unpack() of the submodules. [YOCTO #12739] Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com> --- lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py index 0aff1008..c5019e98 100644 --- a/lib/bb/fetch2/gitsm.py +++ b/lib/bb/fetch2/gitsm.py @@ -132,4 +132,30 @@ class GitSM(Git): if self.uses_submodules(ud, d, ud.destdir): runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir) - runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir) + + updateflags = "" + + # Copy over the submodules' fetched histories too. + if ud.bareclone: + repo_conf = ud.destdir + else: + repo_conf = os.path.join(ud.destdir, '.git') + + if os.path.exists(ud.clonedir): + # This is not a copy unpacked from a shallow mirror clone. So + # the manual intervention to populate the .git/modules done + # in clone_shallow_local() won't have been done yet. + runfetchcmd("cp -fpPRH %s %s" % (os.path.join(ud.clonedir, 'modules'), repo_conf), d) + updateflags += " --no-fetch" + elif os.path.exists(os.path.join(repo_conf, 'modules')): + # Unpacked from a shallow mirror clone. Manual population of + # .git/modules is already done. + updateflags += " --no-fetch" + else: + # This isn't fatal; git-submodule will just fetch it + # during do_unpack(). + bb.error("submodule history not retrieved during do_fetch()") + + # Careful not to hit the network during unpacking; all history should already + # be fetched. + runfetchcmd(ud.basecmd + " submodule update --init --recursive %s" % updateflags, d, workdir=ud.destdir) -- 2.13.6 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack() 2018-05-14 18:50 [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack() Matt Hoosier @ 2018-05-14 20:06 ` Burton, Ross 2018-05-14 21:59 ` Matt Hoosier 0 siblings, 1 reply; 6+ messages in thread From: Burton, Ross @ 2018-05-14 20:06 UTC (permalink / raw) To: Matt Hoosier; +Cc: bitbake-devel Sorry, but: 2018-05-14 18:57:44,968 - oe-selftest - INFO - FAIL: test_devtool_add_fetch_git (devtool.DevtoolTests) 2018-05-14 18:57:44,968 - oe-selftest - INFO - ---------------------------------------------------------------------- 2018-05-14 18:57:44,968 - oe-selftest - INFO - Traceback (most recent call last): File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/devtool.py", line 403, in test_devtool_add_fetch_git result = runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe, srcdir, url_rev)) File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/utils/commands.py", line 191, in runCmd raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output)) AssertionError: Command 'devtool add mraa /tmp/devtoolqa0aghorcw/mraa -f "gitsm://git.yoctoproject.org/mraa;rev=ae127b19a50aa54255e4330ccfdd9a5d058e581d" -V 1.5' returned non-zero exit status 1: ... ERROR: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export PATH="/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot/usr/bin/crossscripts:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bitbake/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/hosttools"; export HOME="/home/pokybuild"; git -c core.fsyncobjectfiles=0 submodule update --init --recursive --no-fetch failed with exit code 1, output: Submodule 'doxygen2jsdoc' (https://github.com/arfoll/doxygen2jsdoc.git) registered for path 'doxygen2jsdoc' fatal: reference is not a tree: 9cc90b7976252b2d14b7956230c5870097e1f008 Unable to checkout '9cc90b7976252b2d14b7956230c5870097e1f008' in submodule path 'doxygen2jsdoc' Ross On 14 May 2018 at 19:50, Matt Hoosier <matt.hoosier@gmail.com> wrote: > Although the submodules' histories have been fetched during the > do_fetch() phase, the mechanics used to clone the workdir copy > of the repo haven't been transferring the actual .git/modules > directory from the repo fetched into downloads/ during the > fetch task. > > Fix that, and for good measure also explicitly tell Git to avoid > hitting the network during do_unpack() of the submodules. > > [YOCTO #12739] > > Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com> > --- > lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++- > 1 file changed, 27 insertions(+), 1 deletion(-) > > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py > index 0aff1008..c5019e98 100644 > --- a/lib/bb/fetch2/gitsm.py > +++ b/lib/bb/fetch2/gitsm.py > @@ -132,4 +132,30 @@ class GitSM(Git): > > if self.uses_submodules(ud, d, ud.destdir): > runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir) > - runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir) > + > + updateflags = "" > + > + # Copy over the submodules' fetched histories too. > + if ud.bareclone: > + repo_conf = ud.destdir > + else: > + repo_conf = os.path.join(ud.destdir, '.git') > + > + if os.path.exists(ud.clonedir): > + # This is not a copy unpacked from a shallow mirror clone. So > + # the manual intervention to populate the .git/modules done > + # in clone_shallow_local() won't have been done yet. > + runfetchcmd("cp -fpPRH %s %s" % (os.path.join(ud.clonedir, 'modules'), repo_conf), d) > + updateflags += " --no-fetch" > + elif os.path.exists(os.path.join(repo_conf, 'modules')): > + # Unpacked from a shallow mirror clone. Manual population of > + # .git/modules is already done. > + updateflags += " --no-fetch" > + else: > + # This isn't fatal; git-submodule will just fetch it > + # during do_unpack(). > + bb.error("submodule history not retrieved during do_fetch()") > + > + # Careful not to hit the network during unpacking; all history should already > + # be fetched. > + runfetchcmd(ud.basecmd + " submodule update --init --recursive %s" % updateflags, d, workdir=ud.destdir) > -- > 2.13.6 > > -- > _______________________________________________ > bitbake-devel mailing list > bitbake-devel@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/bitbake-devel ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack() 2018-05-14 20:06 ` Burton, Ross @ 2018-05-14 21:59 ` Matt Hoosier 2018-05-17 16:28 ` Burton, Ross 0 siblings, 1 reply; 6+ messages in thread From: Matt Hoosier @ 2018-05-14 21:59 UTC (permalink / raw) To: Burton, Ross; +Cc: bitbake-devel [-- Attachment #1: Type: text/plain, Size: 6217 bytes --] On Mon, May 14, 2018, 15:06 Burton, Ross <ross.burton@intel.com> wrote: > Sorry, but: > > 2018-05-14 18:57:44,968 - oe-selftest - INFO - FAIL: > test_devtool_add_fetch_git (devtool.DevtoolTests) > 2018-05-14 18:57:44,968 - oe-selftest - INFO - > ---------------------------------------------------------------------- > 2018-05-14 18:57:44,968 - oe-selftest - INFO - Traceback (most recent > call last): > File > "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/core/decorator/__init__.py", > line 32, in wrapped_f > return func(*args, **kwargs) > File > "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/devtool.py", > line 403, in test_devtool_add_fetch_git > result = runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe, > srcdir, url_rev)) > File > "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/utils/commands.py", > line 191, in runCmd > raise AssertionError("Command '%s' returned non-zero exit status > %d:\n%s" % (command, result.status, exc_output)) > AssertionError: Command 'devtool add mraa /tmp/devtoolqa0aghorcw/mraa > -f "gitsm:// > git.yoctoproject.org/mraa;rev=ae127b19a50aa54255e4330ccfdd9a5d058e581d" > -V 1.5' returned non-zero exit status 1: > ... > > ERROR: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export > > PATH="/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot/usr/bin/crossscripts:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bitbake/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/hosttools"; > export HOME="/home/pokybuild"; git -c core.fsyncobjectfiles=0 > submodule update --init --recursive --no-fetch failed with exit code > 1, output: > Submodule 'doxygen2jsdoc' > (https://github.com/arfoll/doxygen2jsdoc.git) registered for path > 'doxygen2jsdoc' > fatal: reference is not a tree: 9cc90b7976252b2d14b7956230c5870097e1f008 > Unable to checkout '9cc90b7976252b2d14b7956230c5870097e1f008' in > submodule path 'doxygen2jsdoc' > > Ross > > On 14 May 2018 at 19:50, Matt Hoosier <matt.hoosier@gmail.com> wrote: > > Although the submodules' histories have been fetched during the > > do_fetch() phase, the mechanics used to clone the workdir copy > > of the repo haven't been transferring the actual .git/modules > > directory from the repo fetched into downloads/ during the > > fetch task. > > > > Fix that, and for good measure also explicitly tell Git to avoid > > hitting the network during do_unpack() of the submodules. > > > > [YOCTO #12739] > > > > Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com> > > --- > > lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++- > > 1 file changed, 27 insertions(+), 1 deletion(-) > > > > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py > > index 0aff1008..c5019e98 100644 > > --- a/lib/bb/fetch2/gitsm.py > > +++ b/lib/bb/fetch2/gitsm.py > > @@ -132,4 +132,30 @@ class GitSM(Git): > > > > if self.uses_submodules(ud, d, ud.destdir): > > runfetchcmd(ud.basecmd + " checkout " + > ud.revisions[ud.names[0]], d, workdir=ud.destdir) > > - runfetchcmd(ud.basecmd + " submodule update --init > --recursive", d, workdir=ud.destdir) > > + > > + updateflags = "" > > + > > + # Copy over the submodules' fetched histories too. > > + if ud.bareclone: > > + repo_conf = ud.destdir > > + else: > > + repo_conf = os.path.join(ud.destdir, '.git') > > + > > + if os.path.exists(ud.clonedir): > > + # This is not a copy unpacked from a shallow mirror > clone. So > > + # the manual intervention to populate the .git/modules > done > > + # in clone_shallow_local() won't have been done yet. > > + runfetchcmd("cp -fpPRH %s %s" % > (os.path.join(ud.clonedir, 'modules'), repo_conf), d) > > + updateflags += " --no-fetch" > > + elif os.path.exists(os.path.join(repo_conf, 'modules')): > > + # Unpacked from a shallow mirror clone. Manual > population of > > + # .git/modules is already done. > > + updateflags += " --no-fetch" > > + else: > > + # This isn't fatal; git-submodule will just fetch it > > + # during do_unpack(). > > + bb.error("submodule history not retrieved during > do_fetch()") > > + > > + # Careful not to hit the network during unpacking; all > history should already > > + # be fetched. > > + runfetchcmd(ud.basecmd + " submodule update --init > --recursive %s" % updateflags, d, workdir=ud.destdir) > > -- > > 2.13.6 > > > > -- > > _______________________________________________ > > bitbake-devel mailing list > > bitbake-devel@lists.openembedded.org > > http://lists.openembedded.org/mailman/listinfo/bitbake-devel Okay, I'll have a look at that. Are there any other test suites I should know about? > > [-- Attachment #2: Type: text/html, Size: 8227 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack() 2018-05-14 21:59 ` Matt Hoosier @ 2018-05-17 16:28 ` Burton, Ross 2018-05-17 19:17 ` Matt Hoosier 0 siblings, 1 reply; 6+ messages in thread From: Burton, Ross @ 2018-05-17 16:28 UTC (permalink / raw) To: Matt Hoosier; +Cc: bitbake-devel Hi Matt, If you run oe-selftest -r devtool then it will run the tests which are failing. Ross On 14 May 2018 at 22:59, Matt Hoosier <matt.hoosier@gmail.com> wrote: > On Mon, May 14, 2018, 15:06 Burton, Ross <ross.burton@intel.com> wrote: >> >> Sorry, but: >> >> 2018-05-14 18:57:44,968 - oe-selftest - INFO - FAIL: >> test_devtool_add_fetch_git (devtool.DevtoolTests) >> 2018-05-14 18:57:44,968 - oe-selftest - INFO - >> ---------------------------------------------------------------------- >> 2018-05-14 18:57:44,968 - oe-selftest - INFO - Traceback (most recent >> call last): >> File >> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/core/decorator/__init__.py", >> line 32, in wrapped_f >> return func(*args, **kwargs) >> File >> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/devtool.py", >> line 403, in test_devtool_add_fetch_git >> result = runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe, >> srcdir, url_rev)) >> File >> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/utils/commands.py", >> line 191, in runCmd >> raise AssertionError("Command '%s' returned non-zero exit status >> %d:\n%s" % (command, result.status, exc_output)) >> AssertionError: Command 'devtool add mraa /tmp/devtoolqa0aghorcw/mraa >> -f >> "gitsm://git.yoctoproject.org/mraa;rev=ae127b19a50aa54255e4330ccfdd9a5d058e581d" >> -V 1.5' returned non-zero exit status 1: >> ... >> >> ERROR: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export >> >> PATH="/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot/usr/bin/crossscripts:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bitbake/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/hosttools"; >> export HOME="/home/pokybuild"; git -c core.fsyncobjectfiles=0 >> submodule update --init --recursive --no-fetch failed with exit code >> 1, output: >> Submodule 'doxygen2jsdoc' >> (https://github.com/arfoll/doxygen2jsdoc.git) registered for path >> 'doxygen2jsdoc' >> fatal: reference is not a tree: 9cc90b7976252b2d14b7956230c5870097e1f008 >> Unable to checkout '9cc90b7976252b2d14b7956230c5870097e1f008' in >> submodule path 'doxygen2jsdoc' >> >> Ross >> >> On 14 May 2018 at 19:50, Matt Hoosier <matt.hoosier@gmail.com> wrote: >> > Although the submodules' histories have been fetched during the >> > do_fetch() phase, the mechanics used to clone the workdir copy >> > of the repo haven't been transferring the actual .git/modules >> > directory from the repo fetched into downloads/ during the >> > fetch task. >> > >> > Fix that, and for good measure also explicitly tell Git to avoid >> > hitting the network during do_unpack() of the submodules. >> > >> > [YOCTO #12739] >> > >> > Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com> >> > --- >> > lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++- >> > 1 file changed, 27 insertions(+), 1 deletion(-) >> > >> > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py >> > index 0aff1008..c5019e98 100644 >> > --- a/lib/bb/fetch2/gitsm.py >> > +++ b/lib/bb/fetch2/gitsm.py >> > @@ -132,4 +132,30 @@ class GitSM(Git): >> > >> > if self.uses_submodules(ud, d, ud.destdir): >> > runfetchcmd(ud.basecmd + " checkout " + >> > ud.revisions[ud.names[0]], d, workdir=ud.destdir) >> > - runfetchcmd(ud.basecmd + " submodule update --init >> > --recursive", d, workdir=ud.destdir) >> > + >> > + updateflags = "" >> > + >> > + # Copy over the submodules' fetched histories too. >> > + if ud.bareclone: >> > + repo_conf = ud.destdir >> > + else: >> > + repo_conf = os.path.join(ud.destdir, '.git') >> > + >> > + if os.path.exists(ud.clonedir): >> > + # This is not a copy unpacked from a shallow mirror >> > clone. So >> > + # the manual intervention to populate the .git/modules >> > done >> > + # in clone_shallow_local() won't have been done yet. >> > + runfetchcmd("cp -fpPRH %s %s" % >> > (os.path.join(ud.clonedir, 'modules'), repo_conf), d) >> > + updateflags += " --no-fetch" >> > + elif os.path.exists(os.path.join(repo_conf, 'modules')): >> > + # Unpacked from a shallow mirror clone. Manual >> > population of >> > + # .git/modules is already done. >> > + updateflags += " --no-fetch" >> > + else: >> > + # This isn't fatal; git-submodule will just fetch it >> > + # during do_unpack(). >> > + bb.error("submodule history not retrieved during >> > do_fetch()") >> > + >> > + # Careful not to hit the network during unpacking; all >> > history should already >> > + # be fetched. >> > + runfetchcmd(ud.basecmd + " submodule update --init >> > --recursive %s" % updateflags, d, workdir=ud.destdir) >> > -- >> > 2.13.6 >> > >> > -- >> > _______________________________________________ >> > bitbake-devel mailing list >> > bitbake-devel@lists.openembedded.org >> > http://lists.openembedded.org/mailman/listinfo/bitbake-devel > > > Okay, I'll have a look at that. Are there any other test suites I should > know about? >> >> > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack() 2018-05-17 16:28 ` Burton, Ross @ 2018-05-17 19:17 ` Matt Hoosier 2018-05-25 13:47 ` Matt Hoosier 0 siblings, 1 reply; 6+ messages in thread From: Matt Hoosier @ 2018-05-17 19:17 UTC (permalink / raw) To: Burton, Ross; +Cc: bitbake-devel [-- Attachment #1: Type: text/plain, Size: 7205 bytes --] Right; I got enough context from your copied/pasted failure report to duplicate it. I'm just wondering if there are other test batteries besides bitbake-selftest and oe-selftest that are gates to clear? -Matt On Thu, May 17, 2018 at 11:28 AM, Burton, Ross <ross.burton@intel.com> wrote: > Hi Matt, > > If you run oe-selftest -r devtool then it will run the tests which are > failing. > > Ross > > On 14 May 2018 at 22:59, Matt Hoosier <matt.hoosier@gmail.com> wrote: > > On Mon, May 14, 2018, 15:06 Burton, Ross <ross.burton@intel.com> wrote: > >> > >> Sorry, but: > >> > >> 2018-05-14 18:57:44,968 - oe-selftest - INFO - FAIL: > >> test_devtool_add_fetch_git (devtool.DevtoolTests) > >> 2018-05-14 18:57:44,968 - oe-selftest - INFO - > >> ---------------------------------------------------------------------- > >> 2018-05-14 18:57:44,968 - oe-selftest - INFO - Traceback (most recent > >> call last): > >> File > >> "/home/pokybuild/yocto-autobuilder/yocto-worker/ > nightly-oe-selftest/build/meta/lib/oeqa/core/decorator/__init__.py", > >> line 32, in wrapped_f > >> return func(*args, **kwargs) > >> File > >> "/home/pokybuild/yocto-autobuilder/yocto-worker/ > nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/devtool.py", > >> line 403, in test_devtool_add_fetch_git > >> result = runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe, > >> srcdir, url_rev)) > >> File > >> "/home/pokybuild/yocto-autobuilder/yocto-worker/ > nightly-oe-selftest/build/meta/lib/oeqa/utils/commands.py", > >> line 191, in runCmd > >> raise AssertionError("Command '%s' returned non-zero exit status > >> %d:\n%s" % (command, result.status, exc_output)) > >> AssertionError: Command 'devtool add mraa /tmp/devtoolqa0aghorcw/mraa > >> -f > >> "gitsm://git.yoctoproject.org/mraa;rev=ae127b19a50aa54255e4330ccfdd9a > 5d058e581d" > >> -V 1.5' returned non-zero exit status 1: > >> ... > >> > >> ERROR: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export > >> > >> PATH="/home/pokybuild/yocto-autobuilder/yocto-worker/ > nightly-oe-selftest/build/build/tmp/sysroots-uninative/ > x86_64-linux/usr/bin:/home/pokybuild/yocto-autobuilder/ > yocto-worker/nightly-oe-selftest/build/scripts:/home/ > pokybuild/yocto-autobuilder/yocto-worker/nightly-oe- > selftest/build/build/tmp/work/recipetool-utucnynk/work/ > recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/ > pokybuild/yocto-autobuilder/yocto-worker/nightly-oe- > selftest/build/build/tmp/work/recipetool-utucnynk/work/ > recipe-sysroot/usr/bin/crossscripts:/home/pokybuild/ > yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/ > recipetool-utucnynk/work/recipe-sysroot-native/usr/ > sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/ > nightly-oe-selftest/build/build/tmp/work/recipetool- > utucnynk/work/recipe-sysroot-native/usr/bin:/home/ > pokybuild/yocto-autobuilder/yocto-worker/nightly-oe- > selftest/build/build/tmp/work/recipetool-utucnynk/work/ > recipe-sysroot-native/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/ > nightly-oe-selftest/build/build/tmp/work/recipetool- > utucnynk/work/recipe-sysroot-native/bin:/home/pokybuild/ > yocto-autobuilder/yocto-worker/nightly-oe-selftest/ > build/bitbake/bin:/home/pokybuild/yocto-autobuilder/ > yocto-worker/nightly-oe-selftest/build/build/tmp/hosttools"; > >> export HOME="/home/pokybuild"; git -c core.fsyncobjectfiles=0 > >> submodule update --init --recursive --no-fetch failed with exit code > >> 1, output: > >> Submodule 'doxygen2jsdoc' > >> (https://github.com/arfoll/doxygen2jsdoc.git) registered for path > >> 'doxygen2jsdoc' > >> fatal: reference is not a tree: 9cc90b7976252b2d14b7956230c587 > 0097e1f008 > >> Unable to checkout '9cc90b7976252b2d14b7956230c5870097e1f008' in > >> submodule path 'doxygen2jsdoc' > >> > >> Ross > >> > >> On 14 May 2018 at 19:50, Matt Hoosier <matt.hoosier@gmail.com> wrote: > >> > Although the submodules' histories have been fetched during the > >> > do_fetch() phase, the mechanics used to clone the workdir copy > >> > of the repo haven't been transferring the actual .git/modules > >> > directory from the repo fetched into downloads/ during the > >> > fetch task. > >> > > >> > Fix that, and for good measure also explicitly tell Git to avoid > >> > hitting the network during do_unpack() of the submodules. > >> > > >> > [YOCTO #12739] > >> > > >> > Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com> > >> > --- > >> > lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++- > >> > 1 file changed, 27 insertions(+), 1 deletion(-) > >> > > >> > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py > >> > index 0aff1008..c5019e98 100644 > >> > --- a/lib/bb/fetch2/gitsm.py > >> > +++ b/lib/bb/fetch2/gitsm.py > >> > @@ -132,4 +132,30 @@ class GitSM(Git): > >> > > >> > if self.uses_submodules(ud, d, ud.destdir): > >> > runfetchcmd(ud.basecmd + " checkout " + > >> > ud.revisions[ud.names[0]], d, workdir=ud.destdir) > >> > - runfetchcmd(ud.basecmd + " submodule update --init > >> > --recursive", d, workdir=ud.destdir) > >> > + > >> > + updateflags = "" > >> > + > >> > + # Copy over the submodules' fetched histories too. > >> > + if ud.bareclone: > >> > + repo_conf = ud.destdir > >> > + else: > >> > + repo_conf = os.path.join(ud.destdir, '.git') > >> > + > >> > + if os.path.exists(ud.clonedir): > >> > + # This is not a copy unpacked from a shallow mirror > >> > clone. So > >> > + # the manual intervention to populate the > .git/modules > >> > done > >> > + # in clone_shallow_local() won't have been done yet. > >> > + runfetchcmd("cp -fpPRH %s %s" % > >> > (os.path.join(ud.clonedir, 'modules'), repo_conf), d) > >> > + updateflags += " --no-fetch" > >> > + elif os.path.exists(os.path.join(repo_conf, 'modules')): > >> > + # Unpacked from a shallow mirror clone. Manual > >> > population of > >> > + # .git/modules is already done. > >> > + updateflags += " --no-fetch" > >> > + else: > >> > + # This isn't fatal; git-submodule will just fetch it > >> > + # during do_unpack(). > >> > + bb.error("submodule history not retrieved during > >> > do_fetch()") > >> > + > >> > + # Careful not to hit the network during unpacking; all > >> > history should already > >> > + # be fetched. > >> > + runfetchcmd(ud.basecmd + " submodule update --init > >> > --recursive %s" % updateflags, d, workdir=ud.destdir) > >> > -- > >> > 2.13.6 > >> > > >> > -- > >> > _______________________________________________ > >> > bitbake-devel mailing list > >> > bitbake-devel@lists.openembedded.org > >> > http://lists.openembedded.org/mailman/listinfo/bitbake-devel > > > > > > Okay, I'll have a look at that. Are there any other test suites I should > > know about? > >> > >> > > > [-- Attachment #2: Type: text/html, Size: 10104 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack() 2018-05-17 19:17 ` Matt Hoosier @ 2018-05-25 13:47 ` Matt Hoosier 0 siblings, 0 replies; 6+ messages in thread From: Matt Hoosier @ 2018-05-25 13:47 UTC (permalink / raw) To: Burton, Ross; +Cc: bitbake-devel [-- Attachment #1: Type: text/plain, Size: 7615 bytes --] These test cases failures are fixed up in the v5 submission: http://lists.openembedded.org/pipermail/bitbake-devel/2018-May/009350.html. On Thu, May 17, 2018 at 2:17 PM, Matt Hoosier <matt.hoosier@gmail.com> wrote: > Right; I got enough context from your copied/pasted failure report to > duplicate it. I'm just wondering if there are other test batteries besides > bitbake-selftest and oe-selftest that are gates to clear? > > -Matt > > > On Thu, May 17, 2018 at 11:28 AM, Burton, Ross <ross.burton@intel.com> > wrote: > >> Hi Matt, >> >> If you run oe-selftest -r devtool then it will run the tests which are >> failing. >> >> Ross >> >> On 14 May 2018 at 22:59, Matt Hoosier <matt.hoosier@gmail.com> wrote: >> > On Mon, May 14, 2018, 15:06 Burton, Ross <ross.burton@intel.com> wrote: >> >> >> >> Sorry, but: >> >> >> >> 2018-05-14 18:57:44,968 - oe-selftest - INFO - FAIL: >> >> test_devtool_add_fetch_git (devtool.DevtoolTests) >> >> 2018-05-14 18:57:44,968 - oe-selftest - INFO - >> >> ---------------------------------------------------------------------- >> >> 2018-05-14 18:57:44,968 - oe-selftest - INFO - Traceback (most recent >> >> call last): >> >> File >> >> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe- >> selftest/build/meta/lib/oeqa/core/decorator/__init__.py", >> >> line 32, in wrapped_f >> >> return func(*args, **kwargs) >> >> File >> >> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe- >> selftest/build/meta/lib/oeqa/selftest/cases/devtool.py", >> >> line 403, in test_devtool_add_fetch_git >> >> result = runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe, >> >> srcdir, url_rev)) >> >> File >> >> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe- >> selftest/build/meta/lib/oeqa/utils/commands.py", >> >> line 191, in runCmd >> >> raise AssertionError("Command '%s' returned non-zero exit status >> >> %d:\n%s" % (command, result.status, exc_output)) >> >> AssertionError: Command 'devtool add mraa /tmp/devtoolqa0aghorcw/mraa >> >> -f >> >> "gitsm://git.yoctoproject.org/mraa;rev=ae127b19a50aa54255e43 >> 30ccfdd9a5d058e581d" >> >> -V 1.5' returned non-zero exit status 1: >> >> ... >> >> >> >> ERROR: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export >> >> >> >> PATH="/home/pokybuild/yocto-autobuilder/yocto-worker/nightly >> -oe-selftest/build/build/tmp/sysroots-uninative/x86_64- >> linux/usr/bin:/home/pokybuild/yocto-autobuilder/yocto- >> worker/nightly-oe-selftest/build/scripts:/home/pokybuild/ >> yocto-autobuilder/yocto-worker/nightly-oe-selftest/ >> build/build/tmp/work/recipetool-utucnynk/work/recipe- >> sysroot-native/usr/bin/x86_64-poky-linux:/home/pokybuild/ >> yocto-autobuilder/yocto-worker/nightly-oe-selftest/ >> build/build/tmp/work/recipetool-utucnynk/work/recipe- >> sysroot/usr/bin/crossscripts:/home/pokybuild/yocto- >> autobuilder/yocto-worker/nightly-oe-selftest/build/ >> build/tmp/work/recipetool-utucnynk/work/recipe-sysroot- >> native/usr/sbin:/home/pokybuild/yocto-autobuilder/ >> yocto-worker/nightly-oe-selftest/build/build/tmp/work/ >> recipetool-utucnynk/work/recipe-sysroot-native/usr/bin: >> /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/ >> build/tmp/work/recipetool-utucnynk/work/recipe-sysroot- >> native/sbin:/home/pokybuild/yocto-autobuilder/yocto- >> worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/ >> recipe-sysroot-native/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/ >> nightly-oe-selftest/build/bitbake/bin:/home/pokybuild/ >> yocto-autobuilder/yocto-worker/nightly-oe-selftest/ >> build/build/tmp/hosttools"; >> >> export HOME="/home/pokybuild"; git -c core.fsyncobjectfiles=0 >> >> submodule update --init --recursive --no-fetch failed with exit code >> >> 1, output: >> >> Submodule 'doxygen2jsdoc' >> >> (https://github.com/arfoll/doxygen2jsdoc.git) registered for path >> >> 'doxygen2jsdoc' >> >> fatal: reference is not a tree: 9cc90b7976252b2d14b7956230c587 >> 0097e1f008 >> >> Unable to checkout '9cc90b7976252b2d14b7956230c5870097e1f008' in >> >> submodule path 'doxygen2jsdoc' >> >> >> >> Ross >> >> >> >> On 14 May 2018 at 19:50, Matt Hoosier <matt.hoosier@gmail.com> wrote: >> >> > Although the submodules' histories have been fetched during the >> >> > do_fetch() phase, the mechanics used to clone the workdir copy >> >> > of the repo haven't been transferring the actual .git/modules >> >> > directory from the repo fetched into downloads/ during the >> >> > fetch task. >> >> > >> >> > Fix that, and for good measure also explicitly tell Git to avoid >> >> > hitting the network during do_unpack() of the submodules. >> >> > >> >> > [YOCTO #12739] >> >> > >> >> > Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com> >> >> > --- >> >> > lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++- >> >> > 1 file changed, 27 insertions(+), 1 deletion(-) >> >> > >> >> > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py >> >> > index 0aff1008..c5019e98 100644 >> >> > --- a/lib/bb/fetch2/gitsm.py >> >> > +++ b/lib/bb/fetch2/gitsm.py >> >> > @@ -132,4 +132,30 @@ class GitSM(Git): >> >> > >> >> > if self.uses_submodules(ud, d, ud.destdir): >> >> > runfetchcmd(ud.basecmd + " checkout " + >> >> > ud.revisions[ud.names[0]], d, workdir=ud.destdir) >> >> > - runfetchcmd(ud.basecmd + " submodule update --init >> >> > --recursive", d, workdir=ud.destdir) >> >> > + >> >> > + updateflags = "" >> >> > + >> >> > + # Copy over the submodules' fetched histories too. >> >> > + if ud.bareclone: >> >> > + repo_conf = ud.destdir >> >> > + else: >> >> > + repo_conf = os.path.join(ud.destdir, '.git') >> >> > + >> >> > + if os.path.exists(ud.clonedir): >> >> > + # This is not a copy unpacked from a shallow mirror >> >> > clone. So >> >> > + # the manual intervention to populate the >> .git/modules >> >> > done >> >> > + # in clone_shallow_local() won't have been done yet. >> >> > + runfetchcmd("cp -fpPRH %s %s" % >> >> > (os.path.join(ud.clonedir, 'modules'), repo_conf), d) >> >> > + updateflags += " --no-fetch" >> >> > + elif os.path.exists(os.path.join(repo_conf, >> 'modules')): >> >> > + # Unpacked from a shallow mirror clone. Manual >> >> > population of >> >> > + # .git/modules is already done. >> >> > + updateflags += " --no-fetch" >> >> > + else: >> >> > + # This isn't fatal; git-submodule will just fetch it >> >> > + # during do_unpack(). >> >> > + bb.error("submodule history not retrieved during >> >> > do_fetch()") >> >> > + >> >> > + # Careful not to hit the network during unpacking; all >> >> > history should already >> >> > + # be fetched. >> >> > + runfetchcmd(ud.basecmd + " submodule update --init >> >> > --recursive %s" % updateflags, d, workdir=ud.destdir) >> >> > -- >> >> > 2.13.6 >> >> > >> >> > -- >> >> > _______________________________________________ >> >> > bitbake-devel mailing list >> >> > bitbake-devel@lists.openembedded.org >> >> > http://lists.openembedded.org/mailman/listinfo/bitbake-devel >> > >> > >> > Okay, I'll have a look at that. Are there any other test suites I should >> > know about? >> >> >> >> >> > >> > > [-- Attachment #2: Type: text/html, Size: 10968 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-05-25 13:48 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-05-14 18:50 [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack() Matt Hoosier 2018-05-14 20:06 ` Burton, Ross 2018-05-14 21:59 ` Matt Hoosier 2018-05-17 16:28 ` Burton, Ross 2018-05-17 19:17 ` Matt Hoosier 2018-05-25 13:47 ` Matt Hoosier
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.