Python 3 for Bitbake

Python 3 for Bitbake

[Richard Purdie]

We've come to a cross roads for bitbake and python. Currently we run on
python 2.7 but python 2.x is getting old and we really need to move to
3.X.

Whilst there are a number of tools out there which can translate
between the two, we have the complexity that we have python both in
bitbake itself and spread throughout the metadata. We could try and
migrate, attempting to support both versions but it would require a lot
of effort and I'm not sure we get much return for it. Its more likely
we'd end up with a lot of subtle bugs.

An alternative is we just have a flag day and switch, putting in
patches to bitbake and the metadata to migrate to 3.X in one go. The
advantage of this is much cleaner code without any workarounds for
compatibility.

I did some work on this a couple of years ago, I've dusted off those
patches and improved upon them to get to the ones in:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rpurdie/wip

Right now, this is enough to be able to run "bitbake bash", there are a
ton of things which still need to be done. The patches above are rather
rough, the aim just being to find out if we can get the basics working
which it appears we can.

My proposal is we decide to have the flag day, we queue up the patches
on a python3 branch both in oe-core and bitbake, then we switch when we
get successful autobuilder builds. I'd ideally like to do this quite
soon and get one with it (within a few weeks), leaving plenty of time
to handle issues and the other changes planned for this release cycle
and give other layers time to adapt.

I will likely push things so the basics in the core work, I'll then
need help for things like toaster, the QA framework, the supporting
tools (devtool, recipetool and so on).

Does anyone object to this?

Cheers,

Richard

Re: Python 3 for Bitbake

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2299 bytes --]

On Wed, May 4, 2016 at 8:56 AM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> We've come to a cross roads for bitbake and python. Currently we run on
> python 2.7 but python 2.x is getting old and we really need to move to
> 3.X.
>
> Whilst there are a number of tools out there which can translate
> between the two, we have the complexity that we have python both in
> bitbake itself and spread throughout the metadata. We could try and
> migrate, attempting to support both versions but it would require a lot
> of effort and I'm not sure we get much return for it. Its more likely
> we'd end up with a lot of subtle bugs.
>
> An alternative is we just have a flag day and switch, putting in
> patches to bitbake and the metadata to migrate to 3.X in one go. The
> advantage of this is much cleaner code without any workarounds for
> compatibility.
>
> I did some work on this a couple of years ago, I've dusted off those
> patches and improved upon them to get to the ones in:
>
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rpurdie/wip
>
> Right now, this is enough to be able to run "bitbake bash", there are a
> ton of things which still need to be done. The patches above are rather
> rough, the aim just being to find out if we can get the basics working
> which it appears we can.
>
> My proposal is we decide to have the flag day, we queue up the patches
> on a python3 branch both in oe-core and bitbake, then we switch when we
> get successful autobuilder builds. I'd ideally like to do this quite
> soon and get one with it (within a few weeks), leaving plenty of time
> to handle issues and the other changes planned for this release cycle
> and give other layers time to adapt.
>
> I will likely push things so the basics in the core work, I'll then
> need help for things like toaster, the QA framework, the supporting
> tools (devtool, recipetool and so on).
>
> Does anyone object to this?


I fully support this effort, and the use of a flag day to switch and not
maintain compatibility. We've wanted to move to 3 for a long time, after
all.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2901 bytes --]

Re: [Openembedded-architecture] Python 3 for Bitbake

[Mark Hatle]

On 5/4/16 3:56 AM, Richard Purdie wrote:
> My proposal is we decide to have the flag day, we queue up the patches
> on a python3 branch both in oe-core and bitbake, then we switch when we
> get successful autobuilder builds. I'd ideally like to do this quite
> soon and get one with it (within a few weeks), leaving plenty of time
> to handle issues and the other changes planned for this release cycle
> and give other layers time to adapt.
> 
> I will likely push things so the basics in the core work, I'll then
> need help for things like toaster, the QA framework, the supporting
> tools (devtool, recipetool and so on).
> 
> Does anyone object to this?

I think this is a good plan.  Flag day, and switch.

Question, do we need any type of identifier in layers to say they're "python3"
compatible or not?

I'd hate for someone to bring in a layer and find it doesn't work, without
understanding WHY it doesn't work.

Alternative would be to change the 'python' identifier to 'python3'..  Then if
'python' is parsed, we error?

--Mark

> Cheers,
> 
> Richard
> 
> 
> 
> _______________________________________________
> Openembedded-architecture mailing list
> Openembedded-architecture@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
> 

Re: [Openembedded-architecture] Python 3 for Bitbake

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 476 bytes --]

On 4 May 2016 at 22:29, Mark Hatle <mark.hatle@windriver.com> wrote:

> I'd hate for someone to bring in a layer and find it doesn't work, without
> understanding WHY it doesn't work.
>
> Alternative would be to change the 'python' identifier to 'python3'..
> Then if
> 'python' is parsed, we error?
>

That sounds like a lot of dull work.  If a layer's class is using python2
syntax and runs under python3 then it will generally throw exceptions,
right?

Ross

[-- Attachment #2: Type: text/html, Size: 912 bytes --]

Re: [Openembedded-architecture] Python 3 for Bitbake

[Tim Orling]

[-- Attachment #1: Type: text/plain, Size: 1228 bytes --]



> On May 4, 2016, at 2:34 PM, Burton, Ross <ross.burton@intel.com> wrote:
> 
> 
>> On 4 May 2016 at 22:29, Mark Hatle <mark.hatle@windriver.com> wrote:
>> I'd hate for someone to bring in a layer and find it doesn't work, without
>> understanding WHY it doesn't work.
>> 
>> Alternative would be to change the 'python' identifier to 'python3'..  Then if
>> 'python' is parsed, we error?
> 
> That sounds like a lot of dull work.  If a layer's class is using python2 syntax and runs under python3 then it will generally throw exceptions, right?
> 
Yes, you will see a lot of TypeError, SyntaxError and other exceptions. In fact, the try:catch handling for those exceptions is a frequent crutch for py2-py3 compatibility. The other biggest PITA is str vs unicode and friends. Look at the recent commits for pystatgrab (upstream) as an example.

Ripping off the band-aid sounds ok to me too. Flag day +1

--Tim

"Just as long as we aren't switching to perl6 at the same time."

> Ross
> _______________________________________________
> Openembedded-architecture mailing list
> Openembedded-architecture@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture

[-- Attachment #2: Type: text/html, Size: 2269 bytes --]

Re: [Openembedded-architecture] Python 3 for Bitbake

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 196 bytes --]

On 5 May 2016 at 06:10, Tim Orling <ticotimo@gmail.com> wrote:

> "Just as long as we aren't switching to perl6 at the same time."
>

I just spilt coffee over my desk you scoundrel.

Ross

[-- Attachment #2: Type: text/html, Size: 576 bytes --]

Re: [Openembedded-architecture] Python 3 for Bitbake

[Philip Balister]

On 05/04/2016 05:29 PM, Mark Hatle wrote:
> On 5/4/16 3:56 AM, Richard Purdie wrote:
>> My proposal is we decide to have the flag day, we queue up the patches
>> on a python3 branch both in oe-core and bitbake, then we switch when we
>> get successful autobuilder builds. I'd ideally like to do this quite
>> soon and get one with it (within a few weeks), leaving plenty of time
>> to handle issues and the other changes planned for this release cycle
>> and give other layers time to adapt.
>>
>> I will likely push things so the basics in the core work, I'll then
>> need help for things like toaster, the QA framework, the supporting
>> tools (devtool, recipetool and so on).
>>
>> Does anyone object to this?
> 
> I think this is a good plan.  Flag day, and switch.
> 
> Question, do we need any type of identifier in layers to say they're "python3"
> compatible or not?

Layers that are python2 will need krogoth branches. I'd expect master
branches to follow oe-core/master and jump to python3.

Philip


> 
> I'd hate for someone to bring in a layer and find it doesn't work, without
> understanding WHY it doesn't work.
> 
> Alternative would be to change the 'python' identifier to 'python3'..  Then if
> 'python' is parsed, we error?
> 
> --Mark
> 
>> Cheers,
>>
>> Richard
>>
>>
>>
>> _______________________________________________
>> Openembedded-architecture mailing list
>> Openembedded-architecture@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
>>
> 
> _______________________________________________
> Openembedded-architecture mailing list
> Openembedded-architecture@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
> 

Re: [Openembedded-architecture] Python 3 for Bitbake

[Mark Hatle]

On 5/5/16 7:13 AM, Philip Balister wrote:
> On 05/04/2016 05:29 PM, Mark Hatle wrote:
>> On 5/4/16 3:56 AM, Richard Purdie wrote:
>>> My proposal is we decide to have the flag day, we queue up the patches
>>> on a python3 branch both in oe-core and bitbake, then we switch when we
>>> get successful autobuilder builds. I'd ideally like to do this quite
>>> soon and get one with it (within a few weeks), leaving plenty of time
>>> to handle issues and the other changes planned for this release cycle
>>> and give other layers time to adapt.
>>>
>>> I will likely push things so the basics in the core work, I'll then
>>> need help for things like toaster, the QA framework, the supporting
>>> tools (devtool, recipetool and so on).
>>>
>>> Does anyone object to this?
>>
>> I think this is a good plan.  Flag day, and switch.
>>
>> Question, do we need any type of identifier in layers to say they're "python3"
>> compatible or not?
> 
> Layers that are python2 will need krogoth branches. I'd expect master
> branches to follow oe-core/master and jump to python3.

I don't disagree, but my experience has been that branching is late or never in
a lot of layers.

This might be a place where we need to plan to go through the layer index and at
least see if the layer parses -- and flag things that no longer parse?  (I don't
even know if this is something reasonable to attempt.)

My fear is just we've already got a lot of "broken" (without defining that)
layers in the index, and I think it will be worse once the python3 change
happens.  I'd really like to make sure we don't confuse people with various errors.

--Mark

> Philip
> 
> 
>>
>> I'd hate for someone to bring in a layer and find it doesn't work, without
>> understanding WHY it doesn't work.
>>
>> Alternative would be to change the 'python' identifier to 'python3'..  Then if
>> 'python' is parsed, we error?
>>
>> --Mark
>>
>>> Cheers,
>>>
>>> Richard
>>>
>>>
>>>
>>> _______________________________________________
>>> Openembedded-architecture mailing list
>>> Openembedded-architecture@lists.openembedded.org
>>> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
>>>
>>
>> _______________________________________________
>> Openembedded-architecture mailing list
>> Openembedded-architecture@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
>>

Re: [Openembedded-architecture] Python 3 for Bitbake

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2073 bytes --]

On Thu, May 5, 2016 at 8:13 AM, Mark Hatle <mark.hatle@windriver.com> wrote:

> On 5/5/16 7:13 AM, Philip Balister wrote:
> > On 05/04/2016 05:29 PM, Mark Hatle wrote:
> >> On 5/4/16 3:56 AM, Richard Purdie wrote:
> >>> My proposal is we decide to have the flag day, we queue up the patches
> >>> on a python3 branch both in oe-core and bitbake, then we switch when we
> >>> get successful autobuilder builds. I'd ideally like to do this quite
> >>> soon and get one with it (within a few weeks), leaving plenty of time
> >>> to handle issues and the other changes planned for this release cycle
> >>> and give other layers time to adapt.
> >>>
> >>> I will likely push things so the basics in the core work, I'll then
> >>> need help for things like toaster, the QA framework, the supporting
> >>> tools (devtool, recipetool and so on).
> >>>
> >>> Does anyone object to this?
> >>
> >> I think this is a good plan.  Flag day, and switch.
> >>
> >> Question, do we need any type of identifier in layers to say they're
> "python3"
> >> compatible or not?
> >
> > Layers that are python2 will need krogoth branches. I'd expect master
> > branches to follow oe-core/master and jump to python3.
>
> I don't disagree, but my experience has been that branching is late or
> never in
> a lot of layers.
>
> This might be a place where we need to plan to go through the layer index
> and at
> least see if the layer parses -- and flag things that no longer parse?  (I
> don't
> even know if this is something reasonable to attempt.)
>
> My fear is just we've already got a lot of "broken" (without defining that)
> layers in the index, and I think it will be worse once the python3 change
> happens.  I'd really like to make sure we don't confuse people with
> various errors.


Perhaps we should consider some sort of flag or dependency/versioning
statement in layer.conf?
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2723 bytes --]

Re: [Openembedded-architecture] Python 3 for Bitbake

[Otavio Salvador]

On Thu, May 5, 2016 at 12:18 PM, Christopher Larson <clarson@kergoth.com> wrote:
> On Thu, May 5, 2016 at 8:13 AM, Mark Hatle <mark.hatle@windriver.com> wrote:
>> My fear is just we've already got a lot of "broken" (without defining
>> that)
>> layers in the index, and I think it will be worse once the python3 change
>> happens.  I'd really like to make sure we don't confuse people with
>> various errors.
>
> Perhaps we should consider some sort of flag or dependency/versioning
> statement in layer.conf?

What about a bumping the OE-Core LAYERVERSION and force the check in the layers?

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [Openembedded-architecture] Python 3 for Bitbake

[Jan-Simon Möller]

Am Donnerstag, 5. Mai 2016, 10:13:21 schrieb Mark Hatle:
> >> Question, do we need any type of identifier in layers to say they're
> >> "python3" compatible or not?
> > 
> > Layers that are python2 will need krogoth branches. I'd expect master
> > branches to follow oe-core/master and jump to python3.
> 
> I don't disagree, but my experience has been that branching is late or never
> in a lot of layers.
> 
> This might be a place where we need to plan to go through the layer index
> and at least see if the layer parses -- and flag things that no longer
> parse?  (I don't even know if this is something reasonable to attempt.)
> 
> My fear is just we've already got a lot of "broken" (without defining that)
> layers in the index, and I think it will be worse once the python3 change
> happens.  I'd really like to make sure we don't confuse people with various
> errors.
> 

As it is a hard cut anyway, call it YP 3.0 (!) and start with an empty 
layerindex for 3.0 and only add layers that parse or are maintained.
Legacy is 1.x/2.x and will be a separate page.


--
Jan-Simon Möller
dl9pf@gmx.de

Re: [Openembedded-architecture] Python 3 for Bitbake

[Otavio Salvador]

On Thu, May 5, 2016 at 2:29 PM, Jan-Simon Möller <dl9pf@gmx.de> wrote:
> Am Donnerstag, 5. Mai 2016, 10:13:21 schrieb Mark Hatle:
>> >> Question, do we need any type of identifier in layers to say they're
>> >> "python3" compatible or not?
>> >
>> > Layers that are python2 will need krogoth branches. I'd expect master
>> > branches to follow oe-core/master and jump to python3.
>>
>> I don't disagree, but my experience has been that branching is late or never
>> in a lot of layers.
>>
>> This might be a place where we need to plan to go through the layer index
>> and at least see if the layer parses -- and flag things that no longer
>> parse?  (I don't even know if this is something reasonable to attempt.)
>>
>> My fear is just we've already got a lot of "broken" (without defining that)
>> layers in the index, and I think it will be worse once the python3 change
>> happens.  I'd really like to make sure we don't confuse people with various
>> errors.
>>
>
> As it is a hard cut anyway, call it YP 3.0 (!) and start with an empty
> layerindex for 3.0 and only add layers that parse or are maintained.
> Legacy is 1.x/2.x and will be a separate page.

Due the move to Python 3, using 3.0 as version does make sense.


-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [Openembedded-architecture] Python 3 for Bitbake

[akuster]

On 05/05/2016 10:29 AM, Jan-Simon Möller wrote:
> Am Donnerstag, 5. Mai 2016, 10:13:21 schrieb Mark Hatle:
>>>> Question, do we need any type of identifier in layers to say they're
>>>> "python3" compatible or not?
>>>
>>> Layers that are python2 will need krogoth branches. I'd expect master
>>> branches to follow oe-core/master and jump to python3.
>>
>> I don't disagree, but my experience has been that branching is late or never
>> in a lot of layers.
>>
>> This might be a place where we need to plan to go through the layer index
>> and at least see if the layer parses -- and flag things that no longer
>> parse?  (I don't even know if this is something reasonable to attempt.)
>>
>> My fear is just we've already got a lot of "broken" (without defining that)
>> layers in the index, and I think it will be worse once the python3 change
>> happens.  I'd really like to make sure we don't confuse people with various
>> errors.
>>
> 
> As it is a hard cut anyway, call it YP 3.0 (!) and start with an empty 
> layerindex for 3.0 and only add layers that parse or are maintained.
> Legacy is 1.x/2.x and will be a separate page.

That would be the lest disruptive but I fear the Python3 switch over
wont happen for a very long time. The same people doing that work would
most likely be active on or maintaining 2.x

- armin
> 
> 
> --
> Jan-Simon Möller
> dl9pf@gmx.de
> 

Re: Python 3 for Bitbake

[Richard Purdie]

On Wed, 2016-05-04 at 09:56 +0100, Richard Purdie wrote:
> My proposal is we decide to have the flag day, we queue up the 
> patches on a python3 branch both in oe-core and bitbake, then we 
> switch when we get successful autobuilder builds. I'd ideally like to 
> do this quite soon and get one with it (within a few weeks), leaving 
> plenty of time to handle issues and the other changes planned for 
> this release cycle and give other layers time to adapt.

To update, there are python3 branches in bitbake and OE-Core. With
those I've been able to "bitbake core-image-sato:do_rootfs core-image
-sato:do_populate_sdk" with the process and xmlrpc backends.

I've had a few issues to resolve, not least that we need to ensure
python is in utf-8 mode when it is started else files are opened as
ascii and all kinds of things break. Once started, you can't change the
default filesystem access mode and never will be able to for reasons
explained in a PEP.

I still haven't tried the various supporting tools, toaster, UIs other
than knotty, the oeqa framework.

I've separated out some patches and key fixes and submitted them, I
still have a monster patch which needs splitting up for both OE-Core
and bitbake though.

Help is welcome but a heads up on which area anyone is touching would
be appreciated. I know Ed plans to look at toaster.

Cheers,

Richard

Re: Python 3 for Bitbake

[Richard Purdie]

Where are we at?

The python3 branches in bitbake, OE-Core and poky are looking fairly
good from a build perspective. The run overnight on the autobuilder had
the following issues:

ppc specific sanity test failure:
https://autobuilder.yoctoproject.org/main/builders/nightly-ppc/builds/7
99
https:
//autobuilder.yoctoproject.org/main/builders/nightly-ppc-lsb/builds/770

weird intermittent multilib failure:
https:/
/autobuilder.yoctoproject.org/main/builders/nightly-multilib/builds/795

4 test failures in oe-selftest:
https://autobuilder.yoctoproject.org/main/builders/nightly-oe-selftest/
builds/517

which given the scope of the changes isn't so bad.

I have merged some of the pieces which are 2.7 safe into master. We've
had some issues with buildtools-tarball and the python3 it contains but
I think we're nearly there in resolving that. I don't believe toaster i
quite there yet either but getting close.

The question is therefore how much more work to do before those
branches are ready for merging? They do need a little more cosmetic
cleanup but I think things are very close.

I would like to get them merged sooner than later since they are quite
a nightmare to maintain.

Cheers,

Richard