* [PATCH] devtool: ensure temp directory is created
@ 2017-10-17 15:05 Joshua Lock
2017-10-17 16:00 ` Leonardo Sandoval
0 siblings, 1 reply; 6+ messages in thread
From: Joshua Lock @ 2017-10-17 15:05 UTC (permalink / raw)
To: openembedded-core
We often see nightly-oe-selftest on the Yocto Project Autobuilder fail in
test_devtool_modify_native with:
'devtool modify apt-native -x /tmp/devtoolqahlptvsfq/apt-native' returned
non-zero exit status 1
and the following traceback:
Traceback (most recent call last):
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/devtool", line 331, in <module>
ret = main()
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/devtool", line 318, in main
ret = args.func(args, config, basepath, workspace)
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/lib/devtool/standard.py", line 714, in modify
initial_rev = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil)
File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/lib/devtool/standard.py", line 545, in _extract_source
with open(os.path.join(tempdir, 'initial_rev'), 'r') as f:
Ensure the tempdir is created before we try and create any files there.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
scripts/lib/devtool/standard.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index beea0d4c274..38197251b42 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -494,6 +494,7 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
tempbasedir = d.getVar('WORKDIR')
bb.utils.mkdirhier(tempbasedir)
tempdir = tempfile.mkdtemp(prefix='devtooltmp-', dir=tempbasedir)
+ bb.utils.mkdirhier(tempdir)
try:
tinfoil.logger.setLevel(logging.WARNING)
--
2.13.6
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [PATCH] devtool: ensure temp directory is created
2017-10-17 15:05 [PATCH] devtool: ensure temp directory is created Joshua Lock
@ 2017-10-17 16:00 ` Leonardo Sandoval
2017-10-17 16:02 ` Joshua Lock
0 siblings, 1 reply; 6+ messages in thread
From: Leonardo Sandoval @ 2017-10-17 16:00 UTC (permalink / raw)
To: Joshua Lock; +Cc: openembedded-core
On Tue, 17 Oct 2017 16:05:06 +0100
Joshua Lock <joshua.g.lock@intel.com> wrote:
I would be nice that we keep the bugzilla ticket open (and included the tag on this patch) and investigate further this issue. Seems like in some distros, the temfile.mkdtemp is not doing what it is supposed thus we have this workaround (perhaps strace may lead the real reason)
> We often see nightly-oe-selftest on the Yocto Project Autobuilder fail in
> test_devtool_modify_native with:
>
> 'devtool modify apt-native -x /tmp/devtoolqahlptvsfq/apt-native' returned
> non-zero exit status 1
>
> and the following traceback:
>
> Traceback (most recent call last):
> File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/devtool", line 331, in <module>
> ret = main()
> File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/devtool", line 318, in main
> ret = args.func(args, config, basepath, workspace)
> File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/lib/devtool/standard.py", line 714, in modify
> initial_rev = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil)
> File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/lib/devtool/standard.py", line 545, in _extract_source
> with open(os.path.join(tempdir, 'initial_rev'), 'r') as f:
>
> Ensure the tempdir is created before we try and create any files there.
>
> Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
> ---
> scripts/lib/devtool/standard.py | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
> index beea0d4c274..38197251b42 100644
> --- a/scripts/lib/devtool/standard.py
> +++ b/scripts/lib/devtool/standard.py
> @@ -494,6 +494,7 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
> tempbasedir = d.getVar('WORKDIR')
> bb.utils.mkdirhier(tempbasedir)
> tempdir = tempfile.mkdtemp(prefix='devtooltmp-', dir=tempbasedir)
> + bb.utils.mkdirhier(tempdir)
> try:
> tinfoil.logger.setLevel(logging.WARNING)
>
> --
> 2.13.6
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
--
Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH] devtool: ensure temp directory is created
2017-10-17 16:00 ` Leonardo Sandoval
@ 2017-10-17 16:02 ` Joshua Lock
2017-10-17 16:12 ` Burton, Ross
0 siblings, 1 reply; 6+ messages in thread
From: Joshua Lock @ 2017-10-17 16:02 UTC (permalink / raw)
To: openembedded-core
On 17/10/17 17:00, Leonardo Sandoval wrote:
> On Tue, 17 Oct 2017 16:05:06 +0100
> Joshua Lock <joshua.g.lock@intel.com> wrote:
>
> I would be nice that we keep the bugzilla ticket open (and included the tag on this patch) and investigate further this issue. Seems like in some distros, the temfile.mkdtemp is not doing what it is supposed thus we have this workaround (perhaps strace may lead the real reason)
Agreed. I meant to write more before sending this patch (and copy Paul).
It's clearly not the right fix as the tempfile.mkdtemp() call should
create the directory, but I wanted to share my simple findings (that the
directory doesn't appear to be reliably created by the mkdtemp call, as
expected).
FWIW the failure has been observed on multiple host distros (Ubuntu
16.04, Fedora 26, Tumbleweed) and I could replicate it very easily on my
Fedora 26 host with an use of devtool modify -x.
Joshua
>
>> We often see nightly-oe-selftest on the Yocto Project Autobuilder fail in
>> test_devtool_modify_native with:
>>
>> 'devtool modify apt-native -x /tmp/devtoolqahlptvsfq/apt-native' returned
>> non-zero exit status 1
>>
>> and the following traceback:
>>
>> Traceback (most recent call last):
>> File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/devtool", line 331, in <module>
>> ret = main()
>> File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/devtool", line 318, in main
>> ret = args.func(args, config, basepath, workspace)
>> File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/lib/devtool/standard.py", line 714, in modify
>> initial_rev = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil)
>> File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/scripts/lib/devtool/standard.py", line 545, in _extract_source
>> with open(os.path.join(tempdir, 'initial_rev'), 'r') as f:
>>
>> Ensure the tempdir is created before we try and create any files there.
>>
>> Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
>> ---
>> scripts/lib/devtool/standard.py | 1 +
>> 1 file changed, 1 insertion(+)
>>
>> diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
>> index beea0d4c274..38197251b42 100644
>> --- a/scripts/lib/devtool/standard.py
>> +++ b/scripts/lib/devtool/standard.py
>> @@ -494,6 +494,7 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
>> tempbasedir = d.getVar('WORKDIR')
>> bb.utils.mkdirhier(tempbasedir)
>> tempdir = tempfile.mkdtemp(prefix='devtooltmp-', dir=tempbasedir)
>> + bb.utils.mkdirhier(tempdir)
>> try:
>> tinfoil.logger.setLevel(logging.WARNING)
>>
>> --
>> 2.13.6
>>
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH] devtool: ensure temp directory is created
2017-10-17 16:02 ` Joshua Lock
@ 2017-10-17 16:12 ` Burton, Ross
2017-10-17 16:34 ` Joshua Lock
0 siblings, 1 reply; 6+ messages in thread
From: Burton, Ross @ 2017-10-17 16:12 UTC (permalink / raw)
To: Joshua Lock; +Cc: OE-core
[-- Attachment #1: Type: text/plain, Size: 456 bytes --]
On 17 October 2017 at 17:02, Joshua Lock <joshua.g.lock@linux.intel.com>
wrote:
> FWIW the failure has been observed on multiple host distros (Ubuntu 16.04,
> Fedora 26, Tumbleweed) and I could replicate it very easily on my Fedora 26
> host with an use of devtool modify -x.
If you can replicate it on demand how about using a tracing tool to figure
out if a) the mkdir doesn't actually happen or b) something deletes the
directory.
Ross
[-- Attachment #2: Type: text/html, Size: 787 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] devtool: ensure temp directory is created
2017-10-17 16:12 ` Burton, Ross
@ 2017-10-17 16:34 ` Joshua Lock
2017-10-19 22:54 ` Paul Eggleton
0 siblings, 1 reply; 6+ messages in thread
From: Joshua Lock @ 2017-10-17 16:34 UTC (permalink / raw)
To: Burton, Ross; +Cc: OE-core
On 17/10/17 17:12, Burton, Ross wrote:
> On 17 October 2017 at 17:02, Joshua Lock <joshua.g.lock@linux.intel.com
> <mailto:joshua.g.lock@linux.intel.com>> wrote:
>
> FWIW the failure has been observed on multiple host distros (Ubuntu
> 16.04, Fedora 26, Tumbleweed) and I could replicate it very easily
> on my Fedora 26 host with an use of devtool modify -x.
>
>
> If you can replicate it on demand how about using a tracing tool to
> figure out if a) the mkdir doesn't actually happen or b) something
> deletes the directory.
It seems it's not as easy to reproduce as I thought. I can't reproduce
it at all today on either of my two systems.
I will attempt to gather some logs next time we see it on the autobuilder.
Joshua
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] devtool: ensure temp directory is created
2017-10-17 16:34 ` Joshua Lock
@ 2017-10-19 22:54 ` Paul Eggleton
0 siblings, 0 replies; 6+ messages in thread
From: Paul Eggleton @ 2017-10-19 22:54 UTC (permalink / raw)
To: openembedded-core
On Wednesday, 18 October 2017 5:34:14 AM NZDT Joshua Lock wrote:
>
> On 17/10/17 17:12, Burton, Ross wrote:
> > On 17 October 2017 at 17:02, Joshua Lock <joshua.g.lock@linux.intel.com
> > <mailto:joshua.g.lock@linux.intel.com>> wrote:
> >
> > FWIW the failure has been observed on multiple host distros (Ubuntu
> > 16.04, Fedora 26, Tumbleweed) and I could replicate it very easily
> > on my Fedora 26 host with an use of devtool modify -x.
> >
> >
> > If you can replicate it on demand how about using a tracing tool to
> > figure out if a) the mkdir doesn't actually happen or b) something
> > deletes the directory.
>
> It seems it's not as easy to reproduce as I thought. I can't reproduce
> it at all today on either of my two systems.
>
> I will attempt to gather some logs next time we see it on the autobuilder.
Just to follow up on this thread, creating the temp directory doesn't fix the
issue. I found and fixed the underlying problem yesterday, details on the bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12185
I've sent a fix to the bitbake list.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-10-19 22:54 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-10-17 15:05 [PATCH] devtool: ensure temp directory is created Joshua Lock
2017-10-17 16:00 ` Leonardo Sandoval
2017-10-17 16:02 ` Joshua Lock
2017-10-17 16:12 ` Burton, Ross
2017-10-17 16:34 ` Joshua Lock
2017-10-19 22:54 ` Paul Eggleton
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.