Re: [OE-core] [PATCH 6/7] native: Stop clearing PACKAGES

["Martin Jansa" <Martin.Jansa@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 7077 bytes --]

On Mon, Feb 1, 2021 at 9:21 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Sun, 2021-01-31 at 14:10 +0100, Martin Jansa wrote:
> > On Sun, Jan 31, 2021 at 10:34:48AM +0000, Richard Purdie wrote:
> > > Its the first time I've heard a report of that and it sounds a little
> > > worrying. What you say stuck, bitbake hangs? Is it using cpu? A process
> > > tree of what it looks like when stuck would be interesting if it is
> > > hanging and maybe the last things in any running or just finished task
> > > logs. It could be some issue with the way a failing task is handled?
> > >
> > > We've seen a lot of weird issues on the autobuilder but not that. Could
> > > you have something enabled in your setup for logging/reporting whch
> > > wouldn't be in ours and could be causing it?
> >
> > It doesn't seem to happen very often and I guess autobuilder doesn't
> > include meta-python2 (which is currently the only reproducer in my list
> > of layers).
>
> We don't use meta-python2 on the autobuilder so that could explain why
> we've not seen this.
>
> > I've tried to reproduce it with just 3 layers included:
> > BBLAYERS = " \
> >   /OE/build/oe-core/meta-python2 \
> >   /OE/build/oe-core/meta-openembedded/meta-oe \
> >   /OE/build/oe-core/openembedded-core/meta \
> > "
> >
> > and it happened on 3rd try (cache, tmpdir, sstate-cache were removed
> > only before 1st try)
> >
> > martin@jama:/OE/build/oe-core$ bitbake -k python-native
> > Loading cache: 100% |
>
>
>                       | ETA:  --:--:--
> > Loaded 0 entries from dependency cache.
> > ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/
> python-setuptools_42.0.2.bb: QA Issue:
> /OE/build/oe-core/meta-python2/recipes-devtools/python/
> python-setuptools_42.0.2.bb: Variable RPROVIDES is set as not being
> package specific, please fix this. [pkgvarcheck]
> > ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/
> python-setuptools_42.0.2.bb: Fatal QA errors found, failing task.
> > ERROR: Failed to parse recipe:
> /OE/build/oe-core/meta-python2/recipes-devtools/python/
> python-setuptools_42.0.2.bb
>
>              | ETA:  0:00:14
> >
> > Summary: There were 3 ERROR messages shown, returning a non-zero exit
> code.
> > martin@jama:/OE/build/oe-core$ bitbake -k python-native
> > NOTE: Reconnecting to bitbake server...
> > Loading cache: 100%
> |###################################################################################################################################################################################################################################|
> Time: 0:00:00
> > Loaded 845 entries from dependency cache.
> > ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/
> python-native_2.7.18.bb: Error executing a python function in <code>:
>
>                                      | ETA:  0:01:10
> >
> > The stack trace of python calls that resulted in this exception/failure
> was:
> > File: '<code>', lineno: 14, function: <module>
> >
>      0010:__anon_35__OE_build_oe_core_openembedded_core_meta_classes_devshell_bbclass(d)
> >
>      0011:__anon_151__OE_build_oe_core_openembedded_core_meta_classes_sstate_bbclass(d)
> >
>      0012:__anon_20__OE_build_oe_core_openembedded_core_meta_classes_blacklist_bbclass(d)
> >
>      0013:__anon_177__OE_build_oe_core_openembedded_core_meta_classes_siteinfo_bbclass(d)
> >  ***
> 0014:__anon_90__OE_build_oe_core_meta_python2_recipes_devtools_python_python_native_2_7_18_bb(d)
> > File: '/OE/build/oe-core/meta-python2/recipes-devtools/python/
> python-native_2.7.18.bb', lineno: 76, function:
> __anon_90__OE_build_oe_core_meta_python2_recipes_devtools_python_python_native_2_7_18_bb
> >      0072:        manifest_file.seek(json_start)
> >      0073:        manifest_str = manifest_file.read()
> >      0074:        python_manifest = json.loads(manifest_str)
> >      0075:
> >  *** 0076:    rprovides = d.getVar('RPROVIDES').split()
> >      0077:
> >      0078:    # Hardcoded since it cant be python-native-foo, should be
> python-foo-native
> >      0079:    pn = 'python'
> >      0080:
> > Exception: AttributeError: 'NoneType' object has no attribute 'split'
>
> This sounds like a similar issue to the one I fixed in the python3
> recipe:
>
>
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=cff7db890cdfab41cc8f74e3dc378660d9b6219e


Yes, I've already sent similar patch for meta-python2 yesterday (and today
asked for write access again as last e-mail from Tim clearly said he
doesn't want to maintain it anymore)


> > ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/
> python-setuptools_42.0.2.bb: QA Issue:
> /OE/build/oe-core/meta-python2/recipes-devtools/python/
> python-setuptools_42.0.2.bb: Variable RPROVIDES is set as not being
> package specific, please fix this. [pkgvarcheck]
> > ERROR: /OE/build/oe-core/meta-python2/recipes-devtools/python/
> python-setuptools_42.0.2.bb: Fatal QA errors found, failing task.
> > ERROR: Failed to parse recipe:
> /OE/build/oe-core/meta-python2/recipes-devtools/python/
> python-native_2.7.18.bb
>
>              | ETA:  0:00:30
>
> Which makes me wonder whether the real issue is a bug in bitbake when
> parsing fails. We did have an issue like that a while back but I
> thought it was fixed by changes both Joshua and I made.
>


> It sounds like the failed parsing is causing the parsing pool to fail
> to clean up properly and its hanging at exit. I don't know why :(
>


> I think after a certain number of tries, its not hitting the magic
> combination where the parse failure breaks the parsing pool.
>
>
> I think the reproducer will be some combination of parsing failures and
> a need to parse a larger number of recipes. I will give debug some
> further thought, I'm not entirely sure how to proceed on this one at
> the moment :/.
>
> I suspect if you're not hitting parse failures, things will calm down
> though.
>


Yes, I'm sorry if I wasn't clear, all these meta-python2 issues I've
mentioned are easy to fix and I already did (based on your changes in
oe-core), they were shown just as a case which sometimes leads to bitbake
to gettting stuck, with both fixes applied in meta-python2 I haven't
noticed bitbake getting stuck (with latest oe-core + bitbake).

In dunfell based builds I quite often see builds getting stuck on jenkins,
usually after parsing error caused by bad SRCREV (developers use reference
to gerrit review like SRCREV_pn-foo = "refs/changes/59/295159/96" but then
trigger many verification builds with it which include some builds where
foo's SRC_URI points to different repository
where refs/changes/59/295159/96 doesn't exit and git ls-remote fails during
"foo" parsing. But again this might be just the huge number of dunfell
builds we do with similar overrides and the root cause for getting stuck
might be the same as what we're seeing now with this meta-python2 in master.

Cheers,

[-- Attachment #2: Type: text/html, Size: 10167 bytes --]