* [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack()
@ 2018-05-14 18:50 Matt Hoosier
2018-05-14 20:06 ` Burton, Ross
0 siblings, 1 reply; 6+ messages in thread
From: Matt Hoosier @ 2018-05-14 18:50 UTC (permalink / raw)
To: bitbake-devel
Although the submodules' histories have been fetched during the
do_fetch() phase, the mechanics used to clone the workdir copy
of the repo haven't been transferring the actual .git/modules
directory from the repo fetched into downloads/ during the
fetch task.
Fix that, and for good measure also explicitly tell Git to avoid
hitting the network during do_unpack() of the submodules.
[YOCTO #12739]
Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
---
lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++-
1 file changed, 27 insertions(+), 1 deletion(-)
diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
index 0aff1008..c5019e98 100644
--- a/lib/bb/fetch2/gitsm.py
+++ b/lib/bb/fetch2/gitsm.py
@@ -132,4 +132,30 @@ class GitSM(Git):
if self.uses_submodules(ud, d, ud.destdir):
runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir)
- runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir)
+
+ updateflags = ""
+
+ # Copy over the submodules' fetched histories too.
+ if ud.bareclone:
+ repo_conf = ud.destdir
+ else:
+ repo_conf = os.path.join(ud.destdir, '.git')
+
+ if os.path.exists(ud.clonedir):
+ # This is not a copy unpacked from a shallow mirror clone. So
+ # the manual intervention to populate the .git/modules done
+ # in clone_shallow_local() won't have been done yet.
+ runfetchcmd("cp -fpPRH %s %s" % (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
+ updateflags += " --no-fetch"
+ elif os.path.exists(os.path.join(repo_conf, 'modules')):
+ # Unpacked from a shallow mirror clone. Manual population of
+ # .git/modules is already done.
+ updateflags += " --no-fetch"
+ else:
+ # This isn't fatal; git-submodule will just fetch it
+ # during do_unpack().
+ bb.error("submodule history not retrieved during do_fetch()")
+
+ # Careful not to hit the network during unpacking; all history should already
+ # be fetched.
+ runfetchcmd(ud.basecmd + " submodule update --init --recursive %s" % updateflags, d, workdir=ud.destdir)
--
2.13.6
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack()
2018-05-14 18:50 [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack() Matt Hoosier
@ 2018-05-14 20:06 ` Burton, Ross
2018-05-14 21:59 ` Matt Hoosier
0 siblings, 1 reply; 6+ messages in thread
From: Burton, Ross @ 2018-05-14 20:06 UTC (permalink / raw)
To: Matt Hoosier; +Cc: bitbake-devel
Sorry, but:
2018-05-14 18:57:44,968 - oe-selftest - INFO - FAIL:
test_devtool_add_fetch_git (devtool.DevtoolTests)
2018-05-14 18:57:44,968 - oe-selftest - INFO -
----------------------------------------------------------------------
2018-05-14 18:57:44,968 - oe-selftest - INFO - Traceback (most recent
call last):
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/core/decorator/__init__.py",
line 32, in wrapped_f
return func(*args, **kwargs)
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/devtool.py",
line 403, in test_devtool_add_fetch_git
result = runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe,
srcdir, url_rev))
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/utils/commands.py",
line 191, in runCmd
raise AssertionError("Command '%s' returned non-zero exit status
%d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'devtool add mraa /tmp/devtoolqa0aghorcw/mraa
-f "gitsm://git.yoctoproject.org/mraa;rev=ae127b19a50aa54255e4330ccfdd9a5d058e581d"
-V 1.5' returned non-zero exit status 1:
...
ERROR: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export
PATH="/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot/usr/bin/crossscripts:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bitbake/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/hosttools";
export HOME="/home/pokybuild"; git -c core.fsyncobjectfiles=0
submodule update --init --recursive --no-fetch failed with exit code
1, output:
Submodule 'doxygen2jsdoc'
(https://github.com/arfoll/doxygen2jsdoc.git) registered for path
'doxygen2jsdoc'
fatal: reference is not a tree: 9cc90b7976252b2d14b7956230c5870097e1f008
Unable to checkout '9cc90b7976252b2d14b7956230c5870097e1f008' in
submodule path 'doxygen2jsdoc'
Ross
On 14 May 2018 at 19:50, Matt Hoosier <matt.hoosier@gmail.com> wrote:
> Although the submodules' histories have been fetched during the
> do_fetch() phase, the mechanics used to clone the workdir copy
> of the repo haven't been transferring the actual .git/modules
> directory from the repo fetched into downloads/ during the
> fetch task.
>
> Fix that, and for good measure also explicitly tell Git to avoid
> hitting the network during do_unpack() of the submodules.
>
> [YOCTO #12739]
>
> Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
> ---
> lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++-
> 1 file changed, 27 insertions(+), 1 deletion(-)
>
> diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
> index 0aff1008..c5019e98 100644
> --- a/lib/bb/fetch2/gitsm.py
> +++ b/lib/bb/fetch2/gitsm.py
> @@ -132,4 +132,30 @@ class GitSM(Git):
>
> if self.uses_submodules(ud, d, ud.destdir):
> runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir)
> - runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir)
> +
> + updateflags = ""
> +
> + # Copy over the submodules' fetched histories too.
> + if ud.bareclone:
> + repo_conf = ud.destdir
> + else:
> + repo_conf = os.path.join(ud.destdir, '.git')
> +
> + if os.path.exists(ud.clonedir):
> + # This is not a copy unpacked from a shallow mirror clone. So
> + # the manual intervention to populate the .git/modules done
> + # in clone_shallow_local() won't have been done yet.
> + runfetchcmd("cp -fpPRH %s %s" % (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
> + updateflags += " --no-fetch"
> + elif os.path.exists(os.path.join(repo_conf, 'modules')):
> + # Unpacked from a shallow mirror clone. Manual population of
> + # .git/modules is already done.
> + updateflags += " --no-fetch"
> + else:
> + # This isn't fatal; git-submodule will just fetch it
> + # during do_unpack().
> + bb.error("submodule history not retrieved during do_fetch()")
> +
> + # Careful not to hit the network during unpacking; all history should already
> + # be fetched.
> + runfetchcmd(ud.basecmd + " submodule update --init --recursive %s" % updateflags, d, workdir=ud.destdir)
> --
> 2.13.6
>
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack()
2018-05-14 20:06 ` Burton, Ross
@ 2018-05-14 21:59 ` Matt Hoosier
2018-05-17 16:28 ` Burton, Ross
0 siblings, 1 reply; 6+ messages in thread
From: Matt Hoosier @ 2018-05-14 21:59 UTC (permalink / raw)
To: Burton, Ross; +Cc: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 6217 bytes --]
On Mon, May 14, 2018, 15:06 Burton, Ross <ross.burton@intel.com> wrote:
> Sorry, but:
>
> 2018-05-14 18:57:44,968 - oe-selftest - INFO - FAIL:
> test_devtool_add_fetch_git (devtool.DevtoolTests)
> 2018-05-14 18:57:44,968 - oe-selftest - INFO -
> ----------------------------------------------------------------------
> 2018-05-14 18:57:44,968 - oe-selftest - INFO - Traceback (most recent
> call last):
> File
> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/core/decorator/__init__.py",
> line 32, in wrapped_f
> return func(*args, **kwargs)
> File
> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/devtool.py",
> line 403, in test_devtool_add_fetch_git
> result = runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe,
> srcdir, url_rev))
> File
> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/utils/commands.py",
> line 191, in runCmd
> raise AssertionError("Command '%s' returned non-zero exit status
> %d:\n%s" % (command, result.status, exc_output))
> AssertionError: Command 'devtool add mraa /tmp/devtoolqa0aghorcw/mraa
> -f "gitsm://
> git.yoctoproject.org/mraa;rev=ae127b19a50aa54255e4330ccfdd9a5d058e581d"
> -V 1.5' returned non-zero exit status 1:
> ...
>
> ERROR: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export
>
> PATH="/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot/usr/bin/crossscripts:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bitbake/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/hosttools";
> export HOME="/home/pokybuild"; git -c core.fsyncobjectfiles=0
> submodule update --init --recursive --no-fetch failed with exit code
> 1, output:
> Submodule 'doxygen2jsdoc'
> (https://github.com/arfoll/doxygen2jsdoc.git) registered for path
> 'doxygen2jsdoc'
> fatal: reference is not a tree: 9cc90b7976252b2d14b7956230c5870097e1f008
> Unable to checkout '9cc90b7976252b2d14b7956230c5870097e1f008' in
> submodule path 'doxygen2jsdoc'
>
> Ross
>
> On 14 May 2018 at 19:50, Matt Hoosier <matt.hoosier@gmail.com> wrote:
> > Although the submodules' histories have been fetched during the
> > do_fetch() phase, the mechanics used to clone the workdir copy
> > of the repo haven't been transferring the actual .git/modules
> > directory from the repo fetched into downloads/ during the
> > fetch task.
> >
> > Fix that, and for good measure also explicitly tell Git to avoid
> > hitting the network during do_unpack() of the submodules.
> >
> > [YOCTO #12739]
> >
> > Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
> > ---
> > lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++-
> > 1 file changed, 27 insertions(+), 1 deletion(-)
> >
> > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
> > index 0aff1008..c5019e98 100644
> > --- a/lib/bb/fetch2/gitsm.py
> > +++ b/lib/bb/fetch2/gitsm.py
> > @@ -132,4 +132,30 @@ class GitSM(Git):
> >
> > if self.uses_submodules(ud, d, ud.destdir):
> > runfetchcmd(ud.basecmd + " checkout " +
> ud.revisions[ud.names[0]], d, workdir=ud.destdir)
> > - runfetchcmd(ud.basecmd + " submodule update --init
> --recursive", d, workdir=ud.destdir)
> > +
> > + updateflags = ""
> > +
> > + # Copy over the submodules' fetched histories too.
> > + if ud.bareclone:
> > + repo_conf = ud.destdir
> > + else:
> > + repo_conf = os.path.join(ud.destdir, '.git')
> > +
> > + if os.path.exists(ud.clonedir):
> > + # This is not a copy unpacked from a shallow mirror
> clone. So
> > + # the manual intervention to populate the .git/modules
> done
> > + # in clone_shallow_local() won't have been done yet.
> > + runfetchcmd("cp -fpPRH %s %s" %
> (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
> > + updateflags += " --no-fetch"
> > + elif os.path.exists(os.path.join(repo_conf, 'modules')):
> > + # Unpacked from a shallow mirror clone. Manual
> population of
> > + # .git/modules is already done.
> > + updateflags += " --no-fetch"
> > + else:
> > + # This isn't fatal; git-submodule will just fetch it
> > + # during do_unpack().
> > + bb.error("submodule history not retrieved during
> do_fetch()")
> > +
> > + # Careful not to hit the network during unpacking; all
> history should already
> > + # be fetched.
> > + runfetchcmd(ud.basecmd + " submodule update --init
> --recursive %s" % updateflags, d, workdir=ud.destdir)
> > --
> > 2.13.6
> >
> > --
> > _______________________________________________
> > bitbake-devel mailing list
> > bitbake-devel@lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/bitbake-devel
Okay, I'll have a look at that. Are there any other test suites I should
know about?
>
>
[-- Attachment #2: Type: text/html, Size: 8227 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack()
2018-05-14 21:59 ` Matt Hoosier
@ 2018-05-17 16:28 ` Burton, Ross
2018-05-17 19:17 ` Matt Hoosier
0 siblings, 1 reply; 6+ messages in thread
From: Burton, Ross @ 2018-05-17 16:28 UTC (permalink / raw)
To: Matt Hoosier; +Cc: bitbake-devel
Hi Matt,
If you run oe-selftest -r devtool then it will run the tests which are failing.
Ross
On 14 May 2018 at 22:59, Matt Hoosier <matt.hoosier@gmail.com> wrote:
> On Mon, May 14, 2018, 15:06 Burton, Ross <ross.burton@intel.com> wrote:
>>
>> Sorry, but:
>>
>> 2018-05-14 18:57:44,968 - oe-selftest - INFO - FAIL:
>> test_devtool_add_fetch_git (devtool.DevtoolTests)
>> 2018-05-14 18:57:44,968 - oe-selftest - INFO -
>> ----------------------------------------------------------------------
>> 2018-05-14 18:57:44,968 - oe-selftest - INFO - Traceback (most recent
>> call last):
>> File
>> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/core/decorator/__init__.py",
>> line 32, in wrapped_f
>> return func(*args, **kwargs)
>> File
>> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/devtool.py",
>> line 403, in test_devtool_add_fetch_git
>> result = runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe,
>> srcdir, url_rev))
>> File
>> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/utils/commands.py",
>> line 191, in runCmd
>> raise AssertionError("Command '%s' returned non-zero exit status
>> %d:\n%s" % (command, result.status, exc_output))
>> AssertionError: Command 'devtool add mraa /tmp/devtoolqa0aghorcw/mraa
>> -f
>> "gitsm://git.yoctoproject.org/mraa;rev=ae127b19a50aa54255e4330ccfdd9a5d058e581d"
>> -V 1.5' returned non-zero exit status 1:
>> ...
>>
>> ERROR: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export
>>
>> PATH="/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot/usr/bin/crossscripts:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-native/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/bitbake/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/hosttools";
>> export HOME="/home/pokybuild"; git -c core.fsyncobjectfiles=0
>> submodule update --init --recursive --no-fetch failed with exit code
>> 1, output:
>> Submodule 'doxygen2jsdoc'
>> (https://github.com/arfoll/doxygen2jsdoc.git) registered for path
>> 'doxygen2jsdoc'
>> fatal: reference is not a tree: 9cc90b7976252b2d14b7956230c5870097e1f008
>> Unable to checkout '9cc90b7976252b2d14b7956230c5870097e1f008' in
>> submodule path 'doxygen2jsdoc'
>>
>> Ross
>>
>> On 14 May 2018 at 19:50, Matt Hoosier <matt.hoosier@gmail.com> wrote:
>> > Although the submodules' histories have been fetched during the
>> > do_fetch() phase, the mechanics used to clone the workdir copy
>> > of the repo haven't been transferring the actual .git/modules
>> > directory from the repo fetched into downloads/ during the
>> > fetch task.
>> >
>> > Fix that, and for good measure also explicitly tell Git to avoid
>> > hitting the network during do_unpack() of the submodules.
>> >
>> > [YOCTO #12739]
>> >
>> > Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
>> > ---
>> > lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++-
>> > 1 file changed, 27 insertions(+), 1 deletion(-)
>> >
>> > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
>> > index 0aff1008..c5019e98 100644
>> > --- a/lib/bb/fetch2/gitsm.py
>> > +++ b/lib/bb/fetch2/gitsm.py
>> > @@ -132,4 +132,30 @@ class GitSM(Git):
>> >
>> > if self.uses_submodules(ud, d, ud.destdir):
>> > runfetchcmd(ud.basecmd + " checkout " +
>> > ud.revisions[ud.names[0]], d, workdir=ud.destdir)
>> > - runfetchcmd(ud.basecmd + " submodule update --init
>> > --recursive", d, workdir=ud.destdir)
>> > +
>> > + updateflags = ""
>> > +
>> > + # Copy over the submodules' fetched histories too.
>> > + if ud.bareclone:
>> > + repo_conf = ud.destdir
>> > + else:
>> > + repo_conf = os.path.join(ud.destdir, '.git')
>> > +
>> > + if os.path.exists(ud.clonedir):
>> > + # This is not a copy unpacked from a shallow mirror
>> > clone. So
>> > + # the manual intervention to populate the .git/modules
>> > done
>> > + # in clone_shallow_local() won't have been done yet.
>> > + runfetchcmd("cp -fpPRH %s %s" %
>> > (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
>> > + updateflags += " --no-fetch"
>> > + elif os.path.exists(os.path.join(repo_conf, 'modules')):
>> > + # Unpacked from a shallow mirror clone. Manual
>> > population of
>> > + # .git/modules is already done.
>> > + updateflags += " --no-fetch"
>> > + else:
>> > + # This isn't fatal; git-submodule will just fetch it
>> > + # during do_unpack().
>> > + bb.error("submodule history not retrieved during
>> > do_fetch()")
>> > +
>> > + # Careful not to hit the network during unpacking; all
>> > history should already
>> > + # be fetched.
>> > + runfetchcmd(ud.basecmd + " submodule update --init
>> > --recursive %s" % updateflags, d, workdir=ud.destdir)
>> > --
>> > 2.13.6
>> >
>> > --
>> > _______________________________________________
>> > bitbake-devel mailing list
>> > bitbake-devel@lists.openembedded.org
>> > http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>
>
> Okay, I'll have a look at that. Are there any other test suites I should
> know about?
>>
>>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack()
2018-05-17 16:28 ` Burton, Ross
@ 2018-05-17 19:17 ` Matt Hoosier
2018-05-25 13:47 ` Matt Hoosier
0 siblings, 1 reply; 6+ messages in thread
From: Matt Hoosier @ 2018-05-17 19:17 UTC (permalink / raw)
To: Burton, Ross; +Cc: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 7205 bytes --]
Right; I got enough context from your copied/pasted failure report to
duplicate it. I'm just wondering if there are other test batteries besides
bitbake-selftest and oe-selftest that are gates to clear?
-Matt
On Thu, May 17, 2018 at 11:28 AM, Burton, Ross <ross.burton@intel.com>
wrote:
> Hi Matt,
>
> If you run oe-selftest -r devtool then it will run the tests which are
> failing.
>
> Ross
>
> On 14 May 2018 at 22:59, Matt Hoosier <matt.hoosier@gmail.com> wrote:
> > On Mon, May 14, 2018, 15:06 Burton, Ross <ross.burton@intel.com> wrote:
> >>
> >> Sorry, but:
> >>
> >> 2018-05-14 18:57:44,968 - oe-selftest - INFO - FAIL:
> >> test_devtool_add_fetch_git (devtool.DevtoolTests)
> >> 2018-05-14 18:57:44,968 - oe-selftest - INFO -
> >> ----------------------------------------------------------------------
> >> 2018-05-14 18:57:44,968 - oe-selftest - INFO - Traceback (most recent
> >> call last):
> >> File
> >> "/home/pokybuild/yocto-autobuilder/yocto-worker/
> nightly-oe-selftest/build/meta/lib/oeqa/core/decorator/__init__.py",
> >> line 32, in wrapped_f
> >> return func(*args, **kwargs)
> >> File
> >> "/home/pokybuild/yocto-autobuilder/yocto-worker/
> nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/devtool.py",
> >> line 403, in test_devtool_add_fetch_git
> >> result = runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe,
> >> srcdir, url_rev))
> >> File
> >> "/home/pokybuild/yocto-autobuilder/yocto-worker/
> nightly-oe-selftest/build/meta/lib/oeqa/utils/commands.py",
> >> line 191, in runCmd
> >> raise AssertionError("Command '%s' returned non-zero exit status
> >> %d:\n%s" % (command, result.status, exc_output))
> >> AssertionError: Command 'devtool add mraa /tmp/devtoolqa0aghorcw/mraa
> >> -f
> >> "gitsm://git.yoctoproject.org/mraa;rev=ae127b19a50aa54255e4330ccfdd9a
> 5d058e581d"
> >> -V 1.5' returned non-zero exit status 1:
> >> ...
> >>
> >> ERROR: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export
> >>
> >> PATH="/home/pokybuild/yocto-autobuilder/yocto-worker/
> nightly-oe-selftest/build/build/tmp/sysroots-uninative/
> x86_64-linux/usr/bin:/home/pokybuild/yocto-autobuilder/
> yocto-worker/nightly-oe-selftest/build/scripts:/home/
> pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-
> selftest/build/build/tmp/work/recipetool-utucnynk/work/
> recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/
> pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-
> selftest/build/build/tmp/work/recipetool-utucnynk/work/
> recipe-sysroot/usr/bin/crossscripts:/home/pokybuild/
> yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/
> recipetool-utucnynk/work/recipe-sysroot-native/usr/
> sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/
> nightly-oe-selftest/build/build/tmp/work/recipetool-
> utucnynk/work/recipe-sysroot-native/usr/bin:/home/
> pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-
> selftest/build/build/tmp/work/recipetool-utucnynk/work/
> recipe-sysroot-native/sbin:/home/pokybuild/yocto-autobuilder/yocto-worker/
> nightly-oe-selftest/build/build/tmp/work/recipetool-
> utucnynk/work/recipe-sysroot-native/bin:/home/pokybuild/
> yocto-autobuilder/yocto-worker/nightly-oe-selftest/
> build/bitbake/bin:/home/pokybuild/yocto-autobuilder/
> yocto-worker/nightly-oe-selftest/build/build/tmp/hosttools";
> >> export HOME="/home/pokybuild"; git -c core.fsyncobjectfiles=0
> >> submodule update --init --recursive --no-fetch failed with exit code
> >> 1, output:
> >> Submodule 'doxygen2jsdoc'
> >> (https://github.com/arfoll/doxygen2jsdoc.git) registered for path
> >> 'doxygen2jsdoc'
> >> fatal: reference is not a tree: 9cc90b7976252b2d14b7956230c587
> 0097e1f008
> >> Unable to checkout '9cc90b7976252b2d14b7956230c5870097e1f008' in
> >> submodule path 'doxygen2jsdoc'
> >>
> >> Ross
> >>
> >> On 14 May 2018 at 19:50, Matt Hoosier <matt.hoosier@gmail.com> wrote:
> >> > Although the submodules' histories have been fetched during the
> >> > do_fetch() phase, the mechanics used to clone the workdir copy
> >> > of the repo haven't been transferring the actual .git/modules
> >> > directory from the repo fetched into downloads/ during the
> >> > fetch task.
> >> >
> >> > Fix that, and for good measure also explicitly tell Git to avoid
> >> > hitting the network during do_unpack() of the submodules.
> >> >
> >> > [YOCTO #12739]
> >> >
> >> > Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
> >> > ---
> >> > lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++-
> >> > 1 file changed, 27 insertions(+), 1 deletion(-)
> >> >
> >> > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
> >> > index 0aff1008..c5019e98 100644
> >> > --- a/lib/bb/fetch2/gitsm.py
> >> > +++ b/lib/bb/fetch2/gitsm.py
> >> > @@ -132,4 +132,30 @@ class GitSM(Git):
> >> >
> >> > if self.uses_submodules(ud, d, ud.destdir):
> >> > runfetchcmd(ud.basecmd + " checkout " +
> >> > ud.revisions[ud.names[0]], d, workdir=ud.destdir)
> >> > - runfetchcmd(ud.basecmd + " submodule update --init
> >> > --recursive", d, workdir=ud.destdir)
> >> > +
> >> > + updateflags = ""
> >> > +
> >> > + # Copy over the submodules' fetched histories too.
> >> > + if ud.bareclone:
> >> > + repo_conf = ud.destdir
> >> > + else:
> >> > + repo_conf = os.path.join(ud.destdir, '.git')
> >> > +
> >> > + if os.path.exists(ud.clonedir):
> >> > + # This is not a copy unpacked from a shallow mirror
> >> > clone. So
> >> > + # the manual intervention to populate the
> .git/modules
> >> > done
> >> > + # in clone_shallow_local() won't have been done yet.
> >> > + runfetchcmd("cp -fpPRH %s %s" %
> >> > (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
> >> > + updateflags += " --no-fetch"
> >> > + elif os.path.exists(os.path.join(repo_conf, 'modules')):
> >> > + # Unpacked from a shallow mirror clone. Manual
> >> > population of
> >> > + # .git/modules is already done.
> >> > + updateflags += " --no-fetch"
> >> > + else:
> >> > + # This isn't fatal; git-submodule will just fetch it
> >> > + # during do_unpack().
> >> > + bb.error("submodule history not retrieved during
> >> > do_fetch()")
> >> > +
> >> > + # Careful not to hit the network during unpacking; all
> >> > history should already
> >> > + # be fetched.
> >> > + runfetchcmd(ud.basecmd + " submodule update --init
> >> > --recursive %s" % updateflags, d, workdir=ud.destdir)
> >> > --
> >> > 2.13.6
> >> >
> >> > --
> >> > _______________________________________________
> >> > bitbake-devel mailing list
> >> > bitbake-devel@lists.openembedded.org
> >> > http://lists.openembedded.org/mailman/listinfo/bitbake-devel
> >
> >
> > Okay, I'll have a look at that. Are there any other test suites I should
> > know about?
> >>
> >>
> >
>
[-- Attachment #2: Type: text/html, Size: 10104 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack()
2018-05-17 19:17 ` Matt Hoosier
@ 2018-05-25 13:47 ` Matt Hoosier
0 siblings, 0 replies; 6+ messages in thread
From: Matt Hoosier @ 2018-05-25 13:47 UTC (permalink / raw)
To: Burton, Ross; +Cc: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 7615 bytes --]
These test cases failures are fixed up in the v5 submission:
http://lists.openembedded.org/pipermail/bitbake-devel/2018-May/009350.html.
On Thu, May 17, 2018 at 2:17 PM, Matt Hoosier <matt.hoosier@gmail.com>
wrote:
> Right; I got enough context from your copied/pasted failure report to
> duplicate it. I'm just wondering if there are other test batteries besides
> bitbake-selftest and oe-selftest that are gates to clear?
>
> -Matt
>
>
> On Thu, May 17, 2018 at 11:28 AM, Burton, Ross <ross.burton@intel.com>
> wrote:
>
>> Hi Matt,
>>
>> If you run oe-selftest -r devtool then it will run the tests which are
>> failing.
>>
>> Ross
>>
>> On 14 May 2018 at 22:59, Matt Hoosier <matt.hoosier@gmail.com> wrote:
>> > On Mon, May 14, 2018, 15:06 Burton, Ross <ross.burton@intel.com> wrote:
>> >>
>> >> Sorry, but:
>> >>
>> >> 2018-05-14 18:57:44,968 - oe-selftest - INFO - FAIL:
>> >> test_devtool_add_fetch_git (devtool.DevtoolTests)
>> >> 2018-05-14 18:57:44,968 - oe-selftest - INFO -
>> >> ----------------------------------------------------------------------
>> >> 2018-05-14 18:57:44,968 - oe-selftest - INFO - Traceback (most recent
>> >> call last):
>> >> File
>> >> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-
>> selftest/build/meta/lib/oeqa/core/decorator/__init__.py",
>> >> line 32, in wrapped_f
>> >> return func(*args, **kwargs)
>> >> File
>> >> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-
>> selftest/build/meta/lib/oeqa/selftest/cases/devtool.py",
>> >> line 403, in test_devtool_add_fetch_git
>> >> result = runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe,
>> >> srcdir, url_rev))
>> >> File
>> >> "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-
>> selftest/build/meta/lib/oeqa/utils/commands.py",
>> >> line 191, in runCmd
>> >> raise AssertionError("Command '%s' returned non-zero exit status
>> >> %d:\n%s" % (command, result.status, exc_output))
>> >> AssertionError: Command 'devtool add mraa /tmp/devtoolqa0aghorcw/mraa
>> >> -f
>> >> "gitsm://git.yoctoproject.org/mraa;rev=ae127b19a50aa54255e43
>> 30ccfdd9a5d058e581d"
>> >> -V 1.5' returned non-zero exit status 1:
>> >> ...
>> >>
>> >> ERROR: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export
>> >>
>> >> PATH="/home/pokybuild/yocto-autobuilder/yocto-worker/nightly
>> -oe-selftest/build/build/tmp/sysroots-uninative/x86_64-
>> linux/usr/bin:/home/pokybuild/yocto-autobuilder/yocto-
>> worker/nightly-oe-selftest/build/scripts:/home/pokybuild/
>> yocto-autobuilder/yocto-worker/nightly-oe-selftest/
>> build/build/tmp/work/recipetool-utucnynk/work/recipe-
>> sysroot-native/usr/bin/x86_64-poky-linux:/home/pokybuild/
>> yocto-autobuilder/yocto-worker/nightly-oe-selftest/
>> build/build/tmp/work/recipetool-utucnynk/work/recipe-
>> sysroot/usr/bin/crossscripts:/home/pokybuild/yocto-
>> autobuilder/yocto-worker/nightly-oe-selftest/build/
>> build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-
>> native/usr/sbin:/home/pokybuild/yocto-autobuilder/
>> yocto-worker/nightly-oe-selftest/build/build/tmp/work/
>> recipetool-utucnynk/work/recipe-sysroot-native/usr/bin:
>> /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/
>> build/tmp/work/recipetool-utucnynk/work/recipe-sysroot-
>> native/sbin:/home/pokybuild/yocto-autobuilder/yocto-
>> worker/nightly-oe-selftest/build/build/tmp/work/recipetool-utucnynk/work/
>> recipe-sysroot-native/bin:/home/pokybuild/yocto-autobuilder/yocto-worker/
>> nightly-oe-selftest/build/bitbake/bin:/home/pokybuild/
>> yocto-autobuilder/yocto-worker/nightly-oe-selftest/
>> build/build/tmp/hosttools";
>> >> export HOME="/home/pokybuild"; git -c core.fsyncobjectfiles=0
>> >> submodule update --init --recursive --no-fetch failed with exit code
>> >> 1, output:
>> >> Submodule 'doxygen2jsdoc'
>> >> (https://github.com/arfoll/doxygen2jsdoc.git) registered for path
>> >> 'doxygen2jsdoc'
>> >> fatal: reference is not a tree: 9cc90b7976252b2d14b7956230c587
>> 0097e1f008
>> >> Unable to checkout '9cc90b7976252b2d14b7956230c5870097e1f008' in
>> >> submodule path 'doxygen2jsdoc'
>> >>
>> >> Ross
>> >>
>> >> On 14 May 2018 at 19:50, Matt Hoosier <matt.hoosier@gmail.com> wrote:
>> >> > Although the submodules' histories have been fetched during the
>> >> > do_fetch() phase, the mechanics used to clone the workdir copy
>> >> > of the repo haven't been transferring the actual .git/modules
>> >> > directory from the repo fetched into downloads/ during the
>> >> > fetch task.
>> >> >
>> >> > Fix that, and for good measure also explicitly tell Git to avoid
>> >> > hitting the network during do_unpack() of the submodules.
>> >> >
>> >> > [YOCTO #12739]
>> >> >
>> >> > Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
>> >> > ---
>> >> > lib/bb/fetch2/gitsm.py | 28 +++++++++++++++++++++++++++-
>> >> > 1 file changed, 27 insertions(+), 1 deletion(-)
>> >> >
>> >> > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
>> >> > index 0aff1008..c5019e98 100644
>> >> > --- a/lib/bb/fetch2/gitsm.py
>> >> > +++ b/lib/bb/fetch2/gitsm.py
>> >> > @@ -132,4 +132,30 @@ class GitSM(Git):
>> >> >
>> >> > if self.uses_submodules(ud, d, ud.destdir):
>> >> > runfetchcmd(ud.basecmd + " checkout " +
>> >> > ud.revisions[ud.names[0]], d, workdir=ud.destdir)
>> >> > - runfetchcmd(ud.basecmd + " submodule update --init
>> >> > --recursive", d, workdir=ud.destdir)
>> >> > +
>> >> > + updateflags = ""
>> >> > +
>> >> > + # Copy over the submodules' fetched histories too.
>> >> > + if ud.bareclone:
>> >> > + repo_conf = ud.destdir
>> >> > + else:
>> >> > + repo_conf = os.path.join(ud.destdir, '.git')
>> >> > +
>> >> > + if os.path.exists(ud.clonedir):
>> >> > + # This is not a copy unpacked from a shallow mirror
>> >> > clone. So
>> >> > + # the manual intervention to populate the
>> .git/modules
>> >> > done
>> >> > + # in clone_shallow_local() won't have been done yet.
>> >> > + runfetchcmd("cp -fpPRH %s %s" %
>> >> > (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
>> >> > + updateflags += " --no-fetch"
>> >> > + elif os.path.exists(os.path.join(repo_conf,
>> 'modules')):
>> >> > + # Unpacked from a shallow mirror clone. Manual
>> >> > population of
>> >> > + # .git/modules is already done.
>> >> > + updateflags += " --no-fetch"
>> >> > + else:
>> >> > + # This isn't fatal; git-submodule will just fetch it
>> >> > + # during do_unpack().
>> >> > + bb.error("submodule history not retrieved during
>> >> > do_fetch()")
>> >> > +
>> >> > + # Careful not to hit the network during unpacking; all
>> >> > history should already
>> >> > + # be fetched.
>> >> > + runfetchcmd(ud.basecmd + " submodule update --init
>> >> > --recursive %s" % updateflags, d, workdir=ud.destdir)
>> >> > --
>> >> > 2.13.6
>> >> >
>> >> > --
>> >> > _______________________________________________
>> >> > bitbake-devel mailing list
>> >> > bitbake-devel@lists.openembedded.org
>> >> > http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>> >
>> >
>> > Okay, I'll have a look at that. Are there any other test suites I should
>> > know about?
>> >>
>> >>
>> >
>>
>
>
[-- Attachment #2: Type: text/html, Size: 10968 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-05-25 13:48 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-05-14 18:50 [PATCH v4] fetch/gitsm: avoid live submodule fetching during unpack() Matt Hoosier
2018-05-14 20:06 ` Burton, Ross
2018-05-14 21:59 ` Matt Hoosier
2018-05-17 16:28 ` Burton, Ross
2018-05-17 19:17 ` Matt Hoosier
2018-05-25 13:47 ` Matt Hoosier
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.