source-less python

source-less python

[Yishai Jaffe]

[-- Attachment #1: Type: text/plain, Size: 434 bytes --]

Hi,
I was wondering if there has been talk about support for source-less python
on an image. Installing py and pyc files doubles the size of python on the
rootfs. I can imagine this being implemented as an image feature.
I know that in buildroot it is supported.
Was this discussed and decided against? Is this an open issue?
I have a working patch that implements this. I can submit it for review.

Thanks,
Yishai Jaffe
Yishai Jaffe

[-- Attachment #2: Type: text/html, Size: 716 bytes --]

Re: [yocto] source-less python

[Federico Pellegrin]

[-- Attachment #1: Type: text/plain, Size: 1946 bytes --]

Hello,
Just as a small reference since I raised some doubts and questions in
Buildroot community on this: there has been also some troubles to
understand the correctness or not there (as I found some packages with
problems due to this source-less management) and this then sparked, besides
discussions in the Buildroot mailing list (roughly end of July / beginning
of August if someone interested searching there), also an issue to the
Python community, which albeit some discussion I think never arrived to a
concrete conclusion. This is the issue:
https://github.com/python/cpython/issues/95827 (see also the linked one
maybe)

These are just my 2 cents to this discussion, then of course the Python
experts will probably chime in and describe the official/discussed position
of Yocto regarding to this, but just felt like mentioning that although
Buildroot is somehow offering this right now (and likely mostly works as
well!), it's not really a so clean and/or agreed solution.

Cheers,
Federico

Il giorno dom 13 nov 2022 alle ore 13:55 Yishai Jaffe <yishai1999@gmail.com>
ha scritto:

> Hi,
> I was wondering if there has been talk about support for source-less
> python on an image. Installing py and pyc files doubles the size of python
> on the rootfs. I can imagine this being implemented as an image feature.
> I know that in buildroot it is supported.
> Was this discussed and decided against? Is this an open issue?
> I have a working patch that implements this. I can submit it for review.
>
> Thanks,
> Yishai Jaffe
> Yishai Jaffe
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#58548):
> https://lists.yoctoproject.org/g/yocto/message/58548
> Mute This Topic: https://lists.yoctoproject.org/mt/94997538/5117409
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [
> fede@evolware.org]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 3038 bytes --]

Re: [yocto] source-less python

[Alexander Kanavin]

Generally we slim down python installations by not installing all of
the standard library, and rather having precise dependencies for
specific modules. Can you illustrate the kind of space savings that
can be gained in actual numbers?

Another issue is that this should be supported upstream and in the
wider python community. Is it?

Alex

On Sun, 13 Nov 2022 at 14:25, Federico Pellegrin <fede@evolware.org> wrote:
>
>
> Hello,
> Just as a small reference since I raised some doubts and questions in Buildroot community on this: there has been also some troubles to understand the correctness or not there (as I found some packages with problems due to this source-less management) and this then sparked, besides discussions in the Buildroot mailing list (roughly end of July / beginning of August if someone interested searching there), also an issue to the Python community, which albeit some discussion I think never arrived to a concrete conclusion. This is the issue: https://github.com/python/cpython/issues/95827 (see also the linked one maybe)
>
> These are just my 2 cents to this discussion, then of course the Python experts will probably chime in and describe the official/discussed position of Yocto regarding to this, but just felt like mentioning that although Buildroot is somehow offering this right now (and likely mostly works as well!), it's not really a so clean and/or agreed solution.
>
> Cheers,
> Federico
>
> Il giorno dom 13 nov 2022 alle ore 13:55 Yishai Jaffe <yishai1999@gmail.com> ha scritto:
>>
>> Hi,
>> I was wondering if there has been talk about support for source-less python on an image. Installing py and pyc files doubles the size of python on the rootfs. I can imagine this being implemented as an image feature.
>> I know that in buildroot it is supported.
>> Was this discussed and decided against? Is this an open issue?
>> I have a working patch that implements this. I can submit it for review.
>>
>> Thanks,
>> Yishai Jaffe
>> Yishai Jaffe
>>
>>
>>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#58549): https://lists.yoctoproject.org/g/yocto/message/58549
> Mute This Topic: https://lists.yoctoproject.org/mt/94997538/1686489
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [yocto] source-less python

[Federico Pellegrin]

[-- Attachment #1: Type: text/plain, Size: 4121 bytes --]

Hello Alex,

Il giorno dom 13 nov 2022 alle ore 16:12 Alexander Kanavin <
alex.kanavin@gmail.com> ha scritto:

> Generally we slim down python installations by not installing all of
> the standard library, and rather having precise dependencies for
> specific modules. Can you illustrate the kind of space savings that
> can be gained in actual numbers?
>

Just to go with some concrete numbers from a Buildroot example I have under
my hands, with just very basic Python 3.10 support (so no additional
modules, just what gets in with only BR2_PACKAGE_PYTHON3=y) the difference
of size you have between using PYC only (BR2_PACKAGE_PYTHON3_PYC_ONLY=y)
and source only (BR2_PACKAGE_PYTHON3_PY_ONLY=y) of the generated
/usr/lib/python3.10 is 14Mb vs 17Mb. So roughly a 20% ?

Besides the space savings, using the PYC only brings also a speedup in
startup of course.

But strictly on space savings: one should also take into account that the
difference we see is not counting the fact that very often then one has a
compressed filesystem underneath. This is especially true when you have
space constrained embedded devices where you'd like to save space, but not
necessarily that -20% is effective, once you take into account the
filesystem compression.

(please note: not the original thread initiator asking to support this in
Yocto, just chimed in soon after as dealed with this in different context
;-) )


>
> Another issue is that this should be supported upstream and in the
> wider python community. Is it?
>

That is the problem IMO: if you look at the linked github issue I posted
last, I don't think there is a full agreement. So while things mostly work
(and pretty well), there are some edge cases for sure (ie trivially
handling pyc compiled scripts or modules that could be executed as scripts
as well as modules). (here is some discussion:
http://lists.busybox.net/pipermail/buildroot/2022-August/648686.html )

Cheers,
Federico





>
> Alex
>
> On Sun, 13 Nov 2022 at 14:25, Federico Pellegrin <fede@evolware.org>
> wrote:
> >
> >
> > Hello,
> > Just as a small reference since I raised some doubts and questions in
> Buildroot community on this: there has been also some troubles to
> understand the correctness or not there (as I found some packages with
> problems due to this source-less management) and this then sparked, besides
> discussions in the Buildroot mailing list (roughly end of July / beginning
> of August if someone interested searching there), also an issue to the
> Python community, which albeit some discussion I think never arrived to a
> concrete conclusion. This is the issue:
> https://github.com/python/cpython/issues/95827 (see also the linked one
> maybe)
> >
> > These are just my 2 cents to this discussion, then of course the Python
> experts will probably chime in and describe the official/discussed position
> of Yocto regarding to this, but just felt like mentioning that although
> Buildroot is somehow offering this right now (and likely mostly works as
> well!), it's not really a so clean and/or agreed solution.
> >
> > Cheers,
> > Federico
> >
> > Il giorno dom 13 nov 2022 alle ore 13:55 Yishai Jaffe <
> yishai1999@gmail.com> ha scritto:
> >>
> >> Hi,
> >> I was wondering if there has been talk about support for source-less
> python on an image. Installing py and pyc files doubles the size of python
> on the rootfs. I can imagine this being implemented as an image feature.
> >> I know that in buildroot it is supported.
> >> Was this discussed and decided against? Is this an open issue?
> >> I have a working patch that implements this. I can submit it for review.
> >>
> >> Thanks,
> >> Yishai Jaffe
> >> Yishai Jaffe
> >>
> >>
> >>
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#58549):
> https://lists.yoctoproject.org/g/yocto/message/58549
> > Mute This Topic: https://lists.yoctoproject.org/mt/94997538/1686489
> > Group Owner: yocto+owner@lists.yoctoproject.org
> > Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [
> alex.kanavin@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>

[-- Attachment #2: Type: text/html, Size: 5956 bytes --]

Re: [yocto] source-less python

[Yishai Jaffe]

[-- Attachment #1: Type: text/plain, Size: 3074 bytes --]

Hi Alexander,

After some research, these are the numbers I came up with for compiling
core-image-minimal with python3-core:

Normal compilation:
tar.bz2 - 6.6MB
squashfs-xz - 6.1MB

With my pyc-only patch:
tar.bz2 - 5.8MB
squashfs-xz - 5.2MB

So that's about a 15% decrease in size.
Again, this is for an image only with python3-core. Logically that would
mean that if you had an image with more python packages it would be even a
bigger percentage.


Yishai Jaffe


On Sun, Nov 13, 2022 at 5:12 PM Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

> Generally we slim down python installations by not installing all of
> the standard library, and rather having precise dependencies for
> specific modules. Can you illustrate the kind of space savings that
> can be gained in actual numbers?
>
> Another issue is that this should be supported upstream and in the
> wider python community. Is it?
>
> Alex
>
> On Sun, 13 Nov 2022 at 14:25, Federico Pellegrin <fede@evolware.org>
> wrote:
> >
> >
> > Hello,
> > Just as a small reference since I raised some doubts and questions in
> Buildroot community on this: there has been also some troubles to
> understand the correctness or not there (as I found some packages with
> problems due to this source-less management) and this then sparked, besides
> discussions in the Buildroot mailing list (roughly end of July / beginning
> of August if someone interested searching there), also an issue to the
> Python community, which albeit some discussion I think never arrived to a
> concrete conclusion. This is the issue:
> https://github.com/python/cpython/issues/95827 (see also the linked one
> maybe)
> >
> > These are just my 2 cents to this discussion, then of course the Python
> experts will probably chime in and describe the official/discussed position
> of Yocto regarding to this, but just felt like mentioning that although
> Buildroot is somehow offering this right now (and likely mostly works as
> well!), it's not really a so clean and/or agreed solution.
> >
> > Cheers,
> > Federico
> >
> > Il giorno dom 13 nov 2022 alle ore 13:55 Yishai Jaffe <
> yishai1999@gmail.com> ha scritto:
> >>
> >> Hi,
> >> I was wondering if there has been talk about support for source-less
> python on an image. Installing py and pyc files doubles the size of python
> on the rootfs. I can imagine this being implemented as an image feature.
> >> I know that in buildroot it is supported.
> >> Was this discussed and decided against? Is this an open issue?
> >> I have a working patch that implements this. I can submit it for review.
> >>
> >> Thanks,
> >> Yishai Jaffe
> >> Yishai Jaffe
> >>
> >>
> >>
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#58549):
> https://lists.yoctoproject.org/g/yocto/message/58549
> > Mute This Topic: https://lists.yoctoproject.org/mt/94997538/1686489
> > Group Owner: yocto+owner@lists.yoctoproject.org
> > Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [
> alex.kanavin@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>

[-- Attachment #2: Type: text/html, Size: 4552 bytes --]

Re: [yocto] source-less python

[Alexander Kanavin]

I just wonder if this should really be in core. The standards for core
are high: it needs to be both testable and tested, and there's only so
many possible options and tweaks where things can regress that we can
take. On the other hand, there has not been much demand for it.

Alex

On Thu, 17 Nov 2022 at 12:13, Yishai Jaffe <yishai1999@gmail.com> wrote:
>
> Hi Alexander,
>
> After some research, these are the numbers I came up with for compiling core-image-minimal with python3-core:
>
> Normal compilation:
> tar.bz2 - 6.6MB
> squashfs-xz - 6.1MB
>
> With my pyc-only patch:
> tar.bz2 - 5.8MB
> squashfs-xz - 5.2MB
>
> So that's about a 15% decrease in size.
> Again, this is for an image only with python3-core. Logically that would mean that if you had an image with more python packages it would be even a bigger percentage.
>
>
> Yishai Jaffe
>
>
> On Sun, Nov 13, 2022 at 5:12 PM Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>>
>> Generally we slim down python installations by not installing all of
>> the standard library, and rather having precise dependencies for
>> specific modules. Can you illustrate the kind of space savings that
>> can be gained in actual numbers?
>>
>> Another issue is that this should be supported upstream and in the
>> wider python community. Is it?
>>
>> Alex
>>
>> On Sun, 13 Nov 2022 at 14:25, Federico Pellegrin <fede@evolware.org> wrote:
>> >
>> >
>> > Hello,
>> > Just as a small reference since I raised some doubts and questions in Buildroot community on this: there has been also some troubles to understand the correctness or not there (as I found some packages with problems due to this source-less management) and this then sparked, besides discussions in the Buildroot mailing list (roughly end of July / beginning of August if someone interested searching there), also an issue to the Python community, which albeit some discussion I think never arrived to a concrete conclusion. This is the issue: https://github.com/python/cpython/issues/95827 (see also the linked one maybe)
>> >
>> > These are just my 2 cents to this discussion, then of course the Python experts will probably chime in and describe the official/discussed position of Yocto regarding to this, but just felt like mentioning that although Buildroot is somehow offering this right now (and likely mostly works as well!), it's not really a so clean and/or agreed solution.
>> >
>> > Cheers,
>> > Federico
>> >
>> > Il giorno dom 13 nov 2022 alle ore 13:55 Yishai Jaffe <yishai1999@gmail.com> ha scritto:
>> >>
>> >> Hi,
>> >> I was wondering if there has been talk about support for source-less python on an image. Installing py and pyc files doubles the size of python on the rootfs. I can imagine this being implemented as an image feature.
>> >> I know that in buildroot it is supported.
>> >> Was this discussed and decided against? Is this an open issue?
>> >> I have a working patch that implements this. I can submit it for review.
>> >>
>> >> Thanks,
>> >> Yishai Jaffe
>> >> Yishai Jaffe
>> >>
>> >>
>> >>
>> >
>> > -=-=-=-=-=-=-=-=-=-=-=-
>> > Links: You receive all messages sent to this group.
>> > View/Reply Online (#58549): https://lists.yoctoproject.org/g/yocto/message/58549
>> > Mute This Topic: https://lists.yoctoproject.org/mt/94997538/1686489
>> > Group Owner: yocto+owner@lists.yoctoproject.org
>> > Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [alex.kanavin@gmail.com]
>> > -=-=-=-=-=-=-=-=-=-=-=-
>> >

Re: [yocto] source-less python

[Yishai Jaffe]

[-- Attachment #1: Type: text/plain, Size: 4436 bytes --]

I assume by core you mean openembedded-core/poky, yes?

So where would you add it to?

Seems to me like a very useful feature that many people would use if they
knew the option exists.

Also, I noticed that in the recipe for python there is a variable named
INCLUDE_PYC which does exactly what you think it would. This looks to me
like someone did think about the whole python size subject but didn't
exactly go through with it all the way. You can decide if to include pyc
files but not the opposite. Also, this feature which does exist only
applies to the python recipe but not to all other non-base python module
recipes. Maybe this can be added to setuptools3 bbclass?

Yishai Jaffe

On Thu, Nov 17, 2022, 1:24 PM Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

> I just wonder if this should really be in core. The standards for core
> are high: it needs to be both testable and tested, and there's only so
> many possible options and tweaks where things can regress that we can
> take. On the other hand, there has not been much demand for it.
>
> Alex
>
> On Thu, 17 Nov 2022 at 12:13, Yishai Jaffe <yishai1999@gmail.com> wrote:
> >
> > Hi Alexander,
> >
> > After some research, these are the numbers I came up with for compiling
> core-image-minimal with python3-core:
> >
> > Normal compilation:
> > tar.bz2 - 6.6MB
> > squashfs-xz - 6.1MB
> >
> > With my pyc-only patch:
> > tar.bz2 - 5.8MB
> > squashfs-xz - 5.2MB
> >
> > So that's about a 15% decrease in size.
> > Again, this is for an image only with python3-core. Logically that would
> mean that if you had an image with more python packages it would be even a
> bigger percentage.
> >
> >
> > Yishai Jaffe
> >
> >
> > On Sun, Nov 13, 2022 at 5:12 PM Alexander Kanavin <
> alex.kanavin@gmail.com> wrote:
> >>
> >> Generally we slim down python installations by not installing all of
> >> the standard library, and rather having precise dependencies for
> >> specific modules. Can you illustrate the kind of space savings that
> >> can be gained in actual numbers?
> >>
> >> Another issue is that this should be supported upstream and in the
> >> wider python community. Is it?
> >>
> >> Alex
> >>
> >> On Sun, 13 Nov 2022 at 14:25, Federico Pellegrin <fede@evolware.org>
> wrote:
> >> >
> >> >
> >> > Hello,
> >> > Just as a small reference since I raised some doubts and questions in
> Buildroot community on this: there has been also some troubles to
> understand the correctness or not there (as I found some packages with
> problems due to this source-less management) and this then sparked, besides
> discussions in the Buildroot mailing list (roughly end of July / beginning
> of August if someone interested searching there), also an issue to the
> Python community, which albeit some discussion I think never arrived to a
> concrete conclusion. This is the issue:
> https://github.com/python/cpython/issues/95827 (see also the linked one
> maybe)
> >> >
> >> > These are just my 2 cents to this discussion, then of course the
> Python experts will probably chime in and describe the official/discussed
> position of Yocto regarding to this, but just felt like mentioning that
> although Buildroot is somehow offering this right now (and likely mostly
> works as well!), it's not really a so clean and/or agreed solution.
> >> >
> >> > Cheers,
> >> > Federico
> >> >
> >> > Il giorno dom 13 nov 2022 alle ore 13:55 Yishai Jaffe <
> yishai1999@gmail.com> ha scritto:
> >> >>
> >> >> Hi,
> >> >> I was wondering if there has been talk about support for source-less
> python on an image. Installing py and pyc files doubles the size of python
> on the rootfs. I can imagine this being implemented as an image feature.
> >> >> I know that in buildroot it is supported.
> >> >> Was this discussed and decided against? Is this an open issue?
> >> >> I have a working patch that implements this. I can submit it for
> review.
> >> >>
> >> >> Thanks,
> >> >> Yishai Jaffe
> >> >> Yishai Jaffe
> >> >>
> >> >>
> >> >>
> >> >
> >> > -=-=-=-=-=-=-=-=-=-=-=-
> >> > Links: You receive all messages sent to this group.
> >> > View/Reply Online (#58549):
> https://lists.yoctoproject.org/g/yocto/message/58549
> >> > Mute This Topic: https://lists.yoctoproject.org/mt/94997538/1686489
> >> > Group Owner: yocto+owner@lists.yoctoproject.org
> >> > Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [
> alex.kanavin@gmail.com]
> >> > -=-=-=-=-=-=-=-=-=-=-=-
> >> >
>

[-- Attachment #2: Type: text/html, Size: 6477 bytes --]

Re: [yocto] source-less python

[Alexander Kanavin]

You can start by placing it in a separate layer?

Alex

On Thu, 17 Nov 2022 at 13:04, Yishai Jaffe <yishai1999@gmail.com> wrote:
>
> I assume by core you mean openembedded-core/poky, yes?
>
> So where would you add it to?
>
> Seems to me like a very useful feature that many people would use if they knew the option exists.
>
> Also, I noticed that in the recipe for python there is a variable named INCLUDE_PYC which does exactly what you think it would. This looks to me like someone did think about the whole python size subject but didn't exactly go through with it all the way. You can decide if to include pyc files but not the opposite. Also, this feature which does exist only applies to the python recipe but not to all other non-base python module recipes. Maybe this can be added to setuptools3 bbclass?
>
> Yishai Jaffe
>
> On Thu, Nov 17, 2022, 1:24 PM Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>>
>> I just wonder if this should really be in core. The standards for core
>> are high: it needs to be both testable and tested, and there's only so
>> many possible options and tweaks where things can regress that we can
>> take. On the other hand, there has not been much demand for it.
>>
>> Alex
>>
>> On Thu, 17 Nov 2022 at 12:13, Yishai Jaffe <yishai1999@gmail.com> wrote:
>> >
>> > Hi Alexander,
>> >
>> > After some research, these are the numbers I came up with for compiling core-image-minimal with python3-core:
>> >
>> > Normal compilation:
>> > tar.bz2 - 6.6MB
>> > squashfs-xz - 6.1MB
>> >
>> > With my pyc-only patch:
>> > tar.bz2 - 5.8MB
>> > squashfs-xz - 5.2MB
>> >
>> > So that's about a 15% decrease in size.
>> > Again, this is for an image only with python3-core. Logically that would mean that if you had an image with more python packages it would be even a bigger percentage.
>> >
>> >
>> > Yishai Jaffe
>> >
>> >
>> > On Sun, Nov 13, 2022 at 5:12 PM Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>> >>
>> >> Generally we slim down python installations by not installing all of
>> >> the standard library, and rather having precise dependencies for
>> >> specific modules. Can you illustrate the kind of space savings that
>> >> can be gained in actual numbers?
>> >>
>> >> Another issue is that this should be supported upstream and in the
>> >> wider python community. Is it?
>> >>
>> >> Alex
>> >>
>> >> On Sun, 13 Nov 2022 at 14:25, Federico Pellegrin <fede@evolware.org> wrote:
>> >> >
>> >> >
>> >> > Hello,
>> >> > Just as a small reference since I raised some doubts and questions in Buildroot community on this: there has been also some troubles to understand the correctness or not there (as I found some packages with problems due to this source-less management) and this then sparked, besides discussions in the Buildroot mailing list (roughly end of July / beginning of August if someone interested searching there), also an issue to the Python community, which albeit some discussion I think never arrived to a concrete conclusion. This is the issue: https://github.com/python/cpython/issues/95827 (see also the linked one maybe)
>> >> >
>> >> > These are just my 2 cents to this discussion, then of course the Python experts will probably chime in and describe the official/discussed position of Yocto regarding to this, but just felt like mentioning that although Buildroot is somehow offering this right now (and likely mostly works as well!), it's not really a so clean and/or agreed solution.
>> >> >
>> >> > Cheers,
>> >> > Federico
>> >> >
>> >> > Il giorno dom 13 nov 2022 alle ore 13:55 Yishai Jaffe <yishai1999@gmail.com> ha scritto:
>> >> >>
>> >> >> Hi,
>> >> >> I was wondering if there has been talk about support for source-less python on an image. Installing py and pyc files doubles the size of python on the rootfs. I can imagine this being implemented as an image feature.
>> >> >> I know that in buildroot it is supported.
>> >> >> Was this discussed and decided against? Is this an open issue?
>> >> >> I have a working patch that implements this. I can submit it for review.
>> >> >>
>> >> >> Thanks,
>> >> >> Yishai Jaffe
>> >> >> Yishai Jaffe
>> >> >>
>> >> >>
>> >> >>
>> >> >
>> >> > -=-=-=-=-=-=-=-=-=-=-=-
>> >> > Links: You receive all messages sent to this group.
>> >> > View/Reply Online (#58549): https://lists.yoctoproject.org/g/yocto/message/58549
>> >> > Mute This Topic: https://lists.yoctoproject.org/mt/94997538/1686489
>> >> > Group Owner: yocto+owner@lists.yoctoproject.org
>> >> > Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [alex.kanavin@gmail.com]
>> >> > -=-=-=-=-=-=-=-=-=-=-=-
>> >> >

Re: [yocto] source-less python

[Yishai Jaffe]

[-- Attachment #1: Type: text/plain, Size: 4990 bytes --]

What layer would you suggest?

Yishai Jaffe

On Thu, Nov 17, 2022, 2:21 PM Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

> You can start by placing it in a separate layer?
>
> Alex
>
> On Thu, 17 Nov 2022 at 13:04, Yishai Jaffe <yishai1999@gmail.com> wrote:
> >
> > I assume by core you mean openembedded-core/poky, yes?
> >
> > So where would you add it to?
> >
> > Seems to me like a very useful feature that many people would use if
> they knew the option exists.
> >
> > Also, I noticed that in the recipe for python there is a variable named
> INCLUDE_PYC which does exactly what you think it would. This looks to me
> like someone did think about the whole python size subject but didn't
> exactly go through with it all the way. You can decide if to include pyc
> files but not the opposite. Also, this feature which does exist only
> applies to the python recipe but not to all other non-base python module
> recipes. Maybe this can be added to setuptools3 bbclass?
> >
> > Yishai Jaffe
> >
> > On Thu, Nov 17, 2022, 1:24 PM Alexander Kanavin <alex.kanavin@gmail.com>
> wrote:
> >>
> >> I just wonder if this should really be in core. The standards for core
> >> are high: it needs to be both testable and tested, and there's only so
> >> many possible options and tweaks where things can regress that we can
> >> take. On the other hand, there has not been much demand for it.
> >>
> >> Alex
> >>
> >> On Thu, 17 Nov 2022 at 12:13, Yishai Jaffe <yishai1999@gmail.com>
> wrote:
> >> >
> >> > Hi Alexander,
> >> >
> >> > After some research, these are the numbers I came up with for
> compiling core-image-minimal with python3-core:
> >> >
> >> > Normal compilation:
> >> > tar.bz2 - 6.6MB
> >> > squashfs-xz - 6.1MB
> >> >
> >> > With my pyc-only patch:
> >> > tar.bz2 - 5.8MB
> >> > squashfs-xz - 5.2MB
> >> >
> >> > So that's about a 15% decrease in size.
> >> > Again, this is for an image only with python3-core. Logically that
> would mean that if you had an image with more python packages it would be
> even a bigger percentage.
> >> >
> >> >
> >> > Yishai Jaffe
> >> >
> >> >
> >> > On Sun, Nov 13, 2022 at 5:12 PM Alexander Kanavin <
> alex.kanavin@gmail.com> wrote:
> >> >>
> >> >> Generally we slim down python installations by not installing all of
> >> >> the standard library, and rather having precise dependencies for
> >> >> specific modules. Can you illustrate the kind of space savings that
> >> >> can be gained in actual numbers?
> >> >>
> >> >> Another issue is that this should be supported upstream and in the
> >> >> wider python community. Is it?
> >> >>
> >> >> Alex
> >> >>
> >> >> On Sun, 13 Nov 2022 at 14:25, Federico Pellegrin <fede@evolware.org>
> wrote:
> >> >> >
> >> >> >
> >> >> > Hello,
> >> >> > Just as a small reference since I raised some doubts and questions
> in Buildroot community on this: there has been also some troubles to
> understand the correctness or not there (as I found some packages with
> problems due to this source-less management) and this then sparked, besides
> discussions in the Buildroot mailing list (roughly end of July / beginning
> of August if someone interested searching there), also an issue to the
> Python community, which albeit some discussion I think never arrived to a
> concrete conclusion. This is the issue:
> https://github.com/python/cpython/issues/95827 (see also the linked one
> maybe)
> >> >> >
> >> >> > These are just my 2 cents to this discussion, then of course the
> Python experts will probably chime in and describe the official/discussed
> position of Yocto regarding to this, but just felt like mentioning that
> although Buildroot is somehow offering this right now (and likely mostly
> works as well!), it's not really a so clean and/or agreed solution.
> >> >> >
> >> >> > Cheers,
> >> >> > Federico
> >> >> >
> >> >> > Il giorno dom 13 nov 2022 alle ore 13:55 Yishai Jaffe <
> yishai1999@gmail.com> ha scritto:
> >> >> >>
> >> >> >> Hi,
> >> >> >> I was wondering if there has been talk about support for
> source-less python on an image. Installing py and pyc files doubles the
> size of python on the rootfs. I can imagine this being implemented as an
> image feature.
> >> >> >> I know that in buildroot it is supported.
> >> >> >> Was this discussed and decided against? Is this an open issue?
> >> >> >> I have a working patch that implements this. I can submit it for
> review.
> >> >> >>
> >> >> >> Thanks,
> >> >> >> Yishai Jaffe
> >> >> >> Yishai Jaffe
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >
> >> >> > -=-=-=-=-=-=-=-=-=-=-=-
> >> >> > Links: You receive all messages sent to this group.
> >> >> > View/Reply Online (#58549):
> https://lists.yoctoproject.org/g/yocto/message/58549
> >> >> > Mute This Topic:
> https://lists.yoctoproject.org/mt/94997538/1686489
> >> >> > Group Owner: yocto+owner@lists.yoctoproject.org
> >> >> > Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [
> alex.kanavin@gmail.com]
> >> >> > -=-=-=-=-=-=-=-=-=-=-=-
> >> >> >
>

[-- Attachment #2: Type: text/html, Size: 7564 bytes --]

Re: [yocto] source-less python

[Alexander Kanavin]

 Making your own! You can have it added to the layer index for better publicity.

Alex

On Thu, 17 Nov 2022 at 13:27, Yishai Jaffe <yishai1999@gmail.com> wrote:
>
> What layer would you suggest?
>
> Yishai Jaffe
>
> On Thu, Nov 17, 2022, 2:21 PM Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>>
>> You can start by placing it in a separate layer?
>>
>> Alex
>>
>> On Thu, 17 Nov 2022 at 13:04, Yishai Jaffe <yishai1999@gmail.com> wrote:
>> >
>> > I assume by core you mean openembedded-core/poky, yes?
>> >
>> > So where would you add it to?
>> >
>> > Seems to me like a very useful feature that many people would use if they knew the option exists.
>> >
>> > Also, I noticed that in the recipe for python there is a variable named INCLUDE_PYC which does exactly what you think it would. This looks to me like someone did think about the whole python size subject but didn't exactly go through with it all the way. You can decide if to include pyc files but not the opposite. Also, this feature which does exist only applies to the python recipe but not to all other non-base python module recipes. Maybe this can be added to setuptools3 bbclass?
>> >
>> > Yishai Jaffe
>> >
>> > On Thu, Nov 17, 2022, 1:24 PM Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>> >>
>> >> I just wonder if this should really be in core. The standards for core
>> >> are high: it needs to be both testable and tested, and there's only so
>> >> many possible options and tweaks where things can regress that we can
>> >> take. On the other hand, there has not been much demand for it.
>> >>
>> >> Alex
>> >>
>> >> On Thu, 17 Nov 2022 at 12:13, Yishai Jaffe <yishai1999@gmail.com> wrote:
>> >> >
>> >> > Hi Alexander,
>> >> >
>> >> > After some research, these are the numbers I came up with for compiling core-image-minimal with python3-core:
>> >> >
>> >> > Normal compilation:
>> >> > tar.bz2 - 6.6MB
>> >> > squashfs-xz - 6.1MB
>> >> >
>> >> > With my pyc-only patch:
>> >> > tar.bz2 - 5.8MB
>> >> > squashfs-xz - 5.2MB
>> >> >
>> >> > So that's about a 15% decrease in size.
>> >> > Again, this is for an image only with python3-core. Logically that would mean that if you had an image with more python packages it would be even a bigger percentage.
>> >> >
>> >> >
>> >> > Yishai Jaffe
>> >> >
>> >> >
>> >> > On Sun, Nov 13, 2022 at 5:12 PM Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>> >> >>
>> >> >> Generally we slim down python installations by not installing all of
>> >> >> the standard library, and rather having precise dependencies for
>> >> >> specific modules. Can you illustrate the kind of space savings that
>> >> >> can be gained in actual numbers?
>> >> >>
>> >> >> Another issue is that this should be supported upstream and in the
>> >> >> wider python community. Is it?
>> >> >>
>> >> >> Alex
>> >> >>
>> >> >> On Sun, 13 Nov 2022 at 14:25, Federico Pellegrin <fede@evolware.org> wrote:
>> >> >> >
>> >> >> >
>> >> >> > Hello,
>> >> >> > Just as a small reference since I raised some doubts and questions in Buildroot community on this: there has been also some troubles to understand the correctness or not there (as I found some packages with problems due to this source-less management) and this then sparked, besides discussions in the Buildroot mailing list (roughly end of July / beginning of August if someone interested searching there), also an issue to the Python community, which albeit some discussion I think never arrived to a concrete conclusion. This is the issue: https://github.com/python/cpython/issues/95827 (see also the linked one maybe)
>> >> >> >
>> >> >> > These are just my 2 cents to this discussion, then of course the Python experts will probably chime in and describe the official/discussed position of Yocto regarding to this, but just felt like mentioning that although Buildroot is somehow offering this right now (and likely mostly works as well!), it's not really a so clean and/or agreed solution.
>> >> >> >
>> >> >> > Cheers,
>> >> >> > Federico
>> >> >> >
>> >> >> > Il giorno dom 13 nov 2022 alle ore 13:55 Yishai Jaffe <yishai1999@gmail.com> ha scritto:
>> >> >> >>
>> >> >> >> Hi,
>> >> >> >> I was wondering if there has been talk about support for source-less python on an image. Installing py and pyc files doubles the size of python on the rootfs. I can imagine this being implemented as an image feature.
>> >> >> >> I know that in buildroot it is supported.
>> >> >> >> Was this discussed and decided against? Is this an open issue?
>> >> >> >> I have a working patch that implements this. I can submit it for review.
>> >> >> >>
>> >> >> >> Thanks,
>> >> >> >> Yishai Jaffe
>> >> >> >> Yishai Jaffe
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >
>> >> >> > -=-=-=-=-=-=-=-=-=-=-=-
>> >> >> > Links: You receive all messages sent to this group.
>> >> >> > View/Reply Online (#58549): https://lists.yoctoproject.org/g/yocto/message/58549
>> >> >> > Mute This Topic: https://lists.yoctoproject.org/mt/94997538/1686489
>> >> >> > Group Owner: yocto+owner@lists.yoctoproject.org
>> >> >> > Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [alex.kanavin@gmail.com]
>> >> >> > -=-=-=-=-=-=-=-=-=-=-=-
>> >> >> >

Re: [yocto] source-less python

[Yishai Jaffe]

[-- Attachment #1: Type: text/plain, Size: 5641 bytes --]

Seems like a shame not to add this feature for everyone to use...

Yishai Jaffe

On Thu, Nov 17, 2022, 2:34 PM Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

>  Making your own! You can have it added to the layer index for better
> publicity.
>
> Alex
>
> On Thu, 17 Nov 2022 at 13:27, Yishai Jaffe <yishai1999@gmail.com> wrote:
> >
> > What layer would you suggest?
> >
> > Yishai Jaffe
> >
> > On Thu, Nov 17, 2022, 2:21 PM Alexander Kanavin <alex.kanavin@gmail.com>
> wrote:
> >>
> >> You can start by placing it in a separate layer?
> >>
> >> Alex
> >>
> >> On Thu, 17 Nov 2022 at 13:04, Yishai Jaffe <yishai1999@gmail.com>
> wrote:
> >> >
> >> > I assume by core you mean openembedded-core/poky, yes?
> >> >
> >> > So where would you add it to?
> >> >
> >> > Seems to me like a very useful feature that many people would use if
> they knew the option exists.
> >> >
> >> > Also, I noticed that in the recipe for python there is a variable
> named INCLUDE_PYC which does exactly what you think it would. This looks to
> me like someone did think about the whole python size subject but didn't
> exactly go through with it all the way. You can decide if to include pyc
> files but not the opposite. Also, this feature which does exist only
> applies to the python recipe but not to all other non-base python module
> recipes. Maybe this can be added to setuptools3 bbclass?
> >> >
> >> > Yishai Jaffe
> >> >
> >> > On Thu, Nov 17, 2022, 1:24 PM Alexander Kanavin <
> alex.kanavin@gmail.com> wrote:
> >> >>
> >> >> I just wonder if this should really be in core. The standards for
> core
> >> >> are high: it needs to be both testable and tested, and there's only
> so
> >> >> many possible options and tweaks where things can regress that we can
> >> >> take. On the other hand, there has not been much demand for it.
> >> >>
> >> >> Alex
> >> >>
> >> >> On Thu, 17 Nov 2022 at 12:13, Yishai Jaffe <yishai1999@gmail.com>
> wrote:
> >> >> >
> >> >> > Hi Alexander,
> >> >> >
> >> >> > After some research, these are the numbers I came up with for
> compiling core-image-minimal with python3-core:
> >> >> >
> >> >> > Normal compilation:
> >> >> > tar.bz2 - 6.6MB
> >> >> > squashfs-xz - 6.1MB
> >> >> >
> >> >> > With my pyc-only patch:
> >> >> > tar.bz2 - 5.8MB
> >> >> > squashfs-xz - 5.2MB
> >> >> >
> >> >> > So that's about a 15% decrease in size.
> >> >> > Again, this is for an image only with python3-core. Logically that
> would mean that if you had an image with more python packages it would be
> even a bigger percentage.
> >> >> >
> >> >> >
> >> >> > Yishai Jaffe
> >> >> >
> >> >> >
> >> >> > On Sun, Nov 13, 2022 at 5:12 PM Alexander Kanavin <
> alex.kanavin@gmail.com> wrote:
> >> >> >>
> >> >> >> Generally we slim down python installations by not installing all
> of
> >> >> >> the standard library, and rather having precise dependencies for
> >> >> >> specific modules. Can you illustrate the kind of space savings
> that
> >> >> >> can be gained in actual numbers?
> >> >> >>
> >> >> >> Another issue is that this should be supported upstream and in the
> >> >> >> wider python community. Is it?
> >> >> >>
> >> >> >> Alex
> >> >> >>
> >> >> >> On Sun, 13 Nov 2022 at 14:25, Federico Pellegrin <
> fede@evolware.org> wrote:
> >> >> >> >
> >> >> >> >
> >> >> >> > Hello,
> >> >> >> > Just as a small reference since I raised some doubts and
> questions in Buildroot community on this: there has been also some troubles
> to understand the correctness or not there (as I found some packages with
> problems due to this source-less management) and this then sparked, besides
> discussions in the Buildroot mailing list (roughly end of July / beginning
> of August if someone interested searching there), also an issue to the
> Python community, which albeit some discussion I think never arrived to a
> concrete conclusion. This is the issue:
> https://github.com/python/cpython/issues/95827 (see also the linked one
> maybe)
> >> >> >> >
> >> >> >> > These are just my 2 cents to this discussion, then of course
> the Python experts will probably chime in and describe the
> official/discussed position of Yocto regarding to this, but just felt like
> mentioning that although Buildroot is somehow offering this right now (and
> likely mostly works as well!), it's not really a so clean and/or agreed
> solution.
> >> >> >> >
> >> >> >> > Cheers,
> >> >> >> > Federico
> >> >> >> >
> >> >> >> > Il giorno dom 13 nov 2022 alle ore 13:55 Yishai Jaffe <
> yishai1999@gmail.com> ha scritto:
> >> >> >> >>
> >> >> >> >> Hi,
> >> >> >> >> I was wondering if there has been talk about support for
> source-less python on an image. Installing py and pyc files doubles the
> size of python on the rootfs. I can imagine this being implemented as an
> image feature.
> >> >> >> >> I know that in buildroot it is supported.
> >> >> >> >> Was this discussed and decided against? Is this an open issue?
> >> >> >> >> I have a working patch that implements this. I can submit it
> for review.
> >> >> >> >>
> >> >> >> >> Thanks,
> >> >> >> >> Yishai Jaffe
> >> >> >> >> Yishai Jaffe
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >
> >> >> >> > -=-=-=-=-=-=-=-=-=-=-=-
> >> >> >> > Links: You receive all messages sent to this group.
> >> >> >> > View/Reply Online (#58549):
> https://lists.yoctoproject.org/g/yocto/message/58549
> >> >> >> > Mute This Topic:
> https://lists.yoctoproject.org/mt/94997538/1686489
> >> >> >> > Group Owner: yocto+owner@lists.yoctoproject.org
> >> >> >> > Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [
> alex.kanavin@gmail.com]
> >> >> >> > -=-=-=-=-=-=-=-=-=-=-=-
> >> >> >> >
>

[-- Attachment #2: Type: text/html, Size: 8976 bytes --]

Re: [yocto] source-less python

[Josef Holzmayr]

[-- Attachment #1: Type: text/plain, Size: 6107 bytes --]

It can always be added if is is found useful and robust. We tend to be
careful about taking things into core layers as long as those two concerns,
plus the maintenance commitment are not clear. Technical debt is a real
problem.

On Thu, Nov 17, 2022 at 2:05 PM Yishai Jaffe <yishai1999@gmail.com> wrote:

> Seems like a shame not to add this feature for everyone to use...
>
> Yishai Jaffe
>
> On Thu, Nov 17, 2022, 2:34 PM Alexander Kanavin <alex.kanavin@gmail.com>
> wrote:
>
>>  Making your own! You can have it added to the layer index for better
>> publicity.
>>
>> Alex
>>
>> On Thu, 17 Nov 2022 at 13:27, Yishai Jaffe <yishai1999@gmail.com> wrote:
>> >
>> > What layer would you suggest?
>> >
>> > Yishai Jaffe
>> >
>> > On Thu, Nov 17, 2022, 2:21 PM Alexander Kanavin <alex.kanavin@gmail.com>
>> wrote:
>> >>
>> >> You can start by placing it in a separate layer?
>> >>
>> >> Alex
>> >>
>> >> On Thu, 17 Nov 2022 at 13:04, Yishai Jaffe <yishai1999@gmail.com>
>> wrote:
>> >> >
>> >> > I assume by core you mean openembedded-core/poky, yes?
>> >> >
>> >> > So where would you add it to?
>> >> >
>> >> > Seems to me like a very useful feature that many people would use if
>> they knew the option exists.
>> >> >
>> >> > Also, I noticed that in the recipe for python there is a variable
>> named INCLUDE_PYC which does exactly what you think it would. This looks to
>> me like someone did think about the whole python size subject but didn't
>> exactly go through with it all the way. You can decide if to include pyc
>> files but not the opposite. Also, this feature which does exist only
>> applies to the python recipe but not to all other non-base python module
>> recipes. Maybe this can be added to setuptools3 bbclass?
>> >> >
>> >> > Yishai Jaffe
>> >> >
>> >> > On Thu, Nov 17, 2022, 1:24 PM Alexander Kanavin <
>> alex.kanavin@gmail.com> wrote:
>> >> >>
>> >> >> I just wonder if this should really be in core. The standards for
>> core
>> >> >> are high: it needs to be both testable and tested, and there's only
>> so
>> >> >> many possible options and tweaks where things can regress that we
>> can
>> >> >> take. On the other hand, there has not been much demand for it.
>> >> >>
>> >> >> Alex
>> >> >>
>> >> >> On Thu, 17 Nov 2022 at 12:13, Yishai Jaffe <yishai1999@gmail.com>
>> wrote:
>> >> >> >
>> >> >> > Hi Alexander,
>> >> >> >
>> >> >> > After some research, these are the numbers I came up with for
>> compiling core-image-minimal with python3-core:
>> >> >> >
>> >> >> > Normal compilation:
>> >> >> > tar.bz2 - 6.6MB
>> >> >> > squashfs-xz - 6.1MB
>> >> >> >
>> >> >> > With my pyc-only patch:
>> >> >> > tar.bz2 - 5.8MB
>> >> >> > squashfs-xz - 5.2MB
>> >> >> >
>> >> >> > So that's about a 15% decrease in size.
>> >> >> > Again, this is for an image only with python3-core. Logically
>> that would mean that if you had an image with more python packages it would
>> be even a bigger percentage.
>> >> >> >
>> >> >> >
>> >> >> > Yishai Jaffe
>> >> >> >
>> >> >> >
>> >> >> > On Sun, Nov 13, 2022 at 5:12 PM Alexander Kanavin <
>> alex.kanavin@gmail.com> wrote:
>> >> >> >>
>> >> >> >> Generally we slim down python installations by not installing
>> all of
>> >> >> >> the standard library, and rather having precise dependencies for
>> >> >> >> specific modules. Can you illustrate the kind of space savings
>> that
>> >> >> >> can be gained in actual numbers?
>> >> >> >>
>> >> >> >> Another issue is that this should be supported upstream and in
>> the
>> >> >> >> wider python community. Is it?
>> >> >> >>
>> >> >> >> Alex
>> >> >> >>
>> >> >> >> On Sun, 13 Nov 2022 at 14:25, Federico Pellegrin <
>> fede@evolware.org> wrote:
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > Hello,
>> >> >> >> > Just as a small reference since I raised some doubts and
>> questions in Buildroot community on this: there has been also some troubles
>> to understand the correctness or not there (as I found some packages with
>> problems due to this source-less management) and this then sparked, besides
>> discussions in the Buildroot mailing list (roughly end of July / beginning
>> of August if someone interested searching there), also an issue to the
>> Python community, which albeit some discussion I think never arrived to a
>> concrete conclusion. This is the issue:
>> https://github.com/python/cpython/issues/95827 (see also the linked one
>> maybe)
>> >> >> >> >
>> >> >> >> > These are just my 2 cents to this discussion, then of course
>> the Python experts will probably chime in and describe the
>> official/discussed position of Yocto regarding to this, but just felt like
>> mentioning that although Buildroot is somehow offering this right now (and
>> likely mostly works as well!), it's not really a so clean and/or agreed
>> solution.
>> >> >> >> >
>> >> >> >> > Cheers,
>> >> >> >> > Federico
>> >> >> >> >
>> >> >> >> > Il giorno dom 13 nov 2022 alle ore 13:55 Yishai Jaffe <
>> yishai1999@gmail.com> ha scritto:
>> >> >> >> >>
>> >> >> >> >> Hi,
>> >> >> >> >> I was wondering if there has been talk about support for
>> source-less python on an image. Installing py and pyc files doubles the
>> size of python on the rootfs. I can imagine this being implemented as an
>> image feature.
>> >> >> >> >> I know that in buildroot it is supported.
>> >> >> >> >> Was this discussed and decided against? Is this an open issue?
>> >> >> >> >> I have a working patch that implements this. I can submit it
>> for review.
>> >> >> >> >>
>> >> >> >> >> Thanks,
>> >> >> >> >> Yishai Jaffe
>> >> >> >> >> Yishai Jaffe
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> You automatically follow any topics you start or reply to.
> View/Reply Online (#58585):
> https://lists.yoctoproject.org/g/yocto/message/58585
> Mute This Topic: https://lists.yoctoproject.org/mt/94997538/4689568
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [
> jester@theyoctojester.info]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 9394 bytes --]

source-less python

[Aurum Nitrogen]

[-- Attachment #1: Type: text/plain, Size: 340 bytes --]

Hi,
I was wondering if there has been talk about support for source-less python
on an image. Installing py and pyc files doubles the size of python on the
rootfs. I can imagine this being implemented as an image feature.
I know that in buildroot it is supported.
Was this discussed and decided against? Is this an open issue?

Thanks,
John

[-- Attachment #2: Type: text/html, Size: 536 bytes --]

Re: [yocto] source-less python

[Alexander Kanavin]

I don't remember any such discussion. If you want this to happen, you
are welcome to develop the feature.

Alex

On Thu, 23 Jun 2022 at 16:39, Aurum Nitrogen <aurumnitrogen@gmail.com> wrote:
>
> Hi,
> I was wondering if there has been talk about support for source-less python on an image. Installing py and pyc files doubles the size of python on the rootfs. I can imagine this being implemented as an image feature.
> I know that in buildroot it is supported.
> Was this discussed and decided against? Is this an open issue?
>
> Thanks,
> John
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#57383): https://lists.yoctoproject.org/g/yocto/message/57383
> Mute This Topic: https://lists.yoctoproject.org/mt/91944424/1686489
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [yocto] source-less python

[Ross Burton]

On 23 Jun 2022, at 23:40, Aurum Nitrogen via lists.yoctoproject.org <aurumnitrogen=gmail.com@lists.yoctoproject.org> wrote:
>
> Hi,
> I was wondering if there has been talk about support for source-less python on an image. Installing py and pyc files doubles the size of python on the rootfs. I can imagine this being implemented as an image feature.
> I know that in buildroot it is supported.
> Was this discussed and decided against? Is this an open issue?


There’s an open issue: https://bugzilla.yoctoproject.org/show_bug.cgi?id=6434

The easiest implementation would be a rootfs-time postprocessing step where you compile every .py file, and then delete the .py.  Is this what buildroot does?

This does break feeds, but the alternative (of changing how packaging is done) would be a lot more invasive.

Ross
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [yocto] source-less python

[Aurum Nitrogen]

[-- Attachment #1.1: Type: text/plain, Size: 2288 bytes --]

Hi,
So after a little bit of research, I've implemented this feature in poky.
The way buildroot works, is that it doesn't create any .pyc files during
the build process of single packages but has a post rootfs hook that (If
that's what the user has configured) compiles the .py files into .pyc files
and removes the .py files.
The compilation to .pyc files is done using a small script that uses
python's py_compile module.

I have attached the diff to poky for implementation of this feature and the
pycompile.py script.
I would love to get your input on this.

Another thing I noticed while doing this research was that the python
recipe has a variable called INCLUDE_PYCS that decides if to include the
.pyc files in the package. This is nice but why not implement this in the
rest of the python package recipes? It can be added to setuptools3.bbclass
or something like that. What do you think?

Thanks a lot,

John

‫בתאריך יום ה׳, 23 ביוני 2022 ב-19:10 מאת ‪Ross Burton‬‏ <‪
Ross.Burton@arm.com‬‏>:‬

> On 23 Jun 2022, at 23:40, Aurum Nitrogen via lists.yoctoproject.org
> <aurumnitrogen=gmail.com@lists.yoctoproject.org> wrote:
> >
> > Hi,
> > I was wondering if there has been talk about support for source-less
> python on an image. Installing py and pyc files doubles the size of python
> on the rootfs. I can imagine this being implemented as an image feature.
> > I know that in buildroot it is supported.
> > Was this discussed and decided against? Is this an open issue?
>
>
> There’s an open issue:
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=6434
>
> The easiest implementation would be a rootfs-time postprocessing step
> where you compile every .py file, and then delete the .py.  Is this what
> buildroot does?
>
> This does break feeds, but the alternative (of changing how packaging is
> done) would be a lot more invasive.
>
> Ross
> IMPORTANT NOTICE: The contents of this email and any attachments are
> confidential and may also be privileged. If you are not the intended
> recipient, please notify the sender immediately and do not disclose the
> contents to any other person, use it for any purpose, or store or copy the
> information in any medium. Thank you.
>

[-- Attachment #1.2: Type: text/html, Size: 3109 bytes --]

[-- Attachment #2: pycompile.py --]
[-- Type: text/x-python, Size: 2902 bytes --]

#!/usr/bin/env python3

"""
Byte compile all .py files from provided directories. This script is an
alternative implementation of compileall.compile_dir written with
cross-compilation in mind.
"""

import argparse
import os
import py_compile
import re
import sys


def compile_one(host_path, strip_root=None, verbose=False):
    """
    Compile a .py file into a .pyc file located next to it.

    :arg host_path:
        Absolute path to the file to compile on the host running the build.
    :arg strip_root:
        Prefix to remove from the original source paths encoded in compiled
        files.
    :arg verbose:
        Print compiled file paths.
    """
    if os.path.islink(host_path) or not os.path.isfile(host_path):
        return  # only compile real files

    if not re.match(r"^[_A-Za-z][_A-Za-z0-9]*\.py$",
                    os.path.basename(host_path)):
        return  # only compile "importable" python modules

    if strip_root is not None:
        # determine the runtime path of the file (i.e.: relative path to root
        # dir prepended with "/").
        runtime_path = os.path.join("/", os.path.relpath(host_path, strip_root))
    else:
        runtime_path = host_path

    if verbose:
        print("  PYC  {}".format(runtime_path))

    # will raise an error if the file cannot be compiled
    py_compile.compile(host_path, cfile=host_path + "c",
                       dfile=runtime_path, doraise=True)


def existing_dir_abs(arg):
    """
    argparse type callback that checks that argument is a directory and returns
    its absolute path.
    """
    if not os.path.isdir(arg):
        raise argparse.ArgumentTypeError('no such directory: {!r}'.format(arg))
    return os.path.abspath(arg)


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("dirs", metavar="DIR", nargs="+", type=existing_dir_abs,
                        help="Directory to recursively scan and compile")
    parser.add_argument("--strip-root", metavar="ROOT", type=existing_dir_abs,
                        help="""
                        Prefix to remove from the original source paths encoded
                        in compiled files
                        """)
    parser.add_argument("--verbose", action="store_true",
                        help="Print compiled files")

    args = parser.parse_args()

    try:
        for d in args.dirs:
            if args.strip_root and ".." in os.path.relpath(d, args.strip_root):
                parser.error("DIR: not inside ROOT dir: {!r}".format(d))
            for parent, _, files in os.walk(d):
                for f in files:
                    compile_one(os.path.join(parent, f), args.strip_root,
                                args.verbose)

    except Exception as e:
        print("error: {}".format(e))
        return 1

    return 0


if __name__ == "__main__":
    sys.exit(main())

[-- Attachment #3: diff.patch --]
[-- Type: text/x-patch, Size: 2347 bytes --]

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index c2f3232027..f0a9fb11b8 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -33,7 +33,7 @@ INHIBIT_DEFAULT_DEPS = "1"
 # IMAGE_FEATURES may contain any available package group
 IMAGE_FEATURES ?= ""
 IMAGE_FEATURES[type] = "list"
-IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password allow-empty-password allow-root-login post-install-logging"
+IMAGE_FEATURES[validitems] += "pyc-only debug-tweaks read-only-rootfs read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password allow-empty-password allow-root-login post-install-logging"
 
 # Generate companion debugfs?
 IMAGE_GEN_DEBUGFS ?= "0"
diff --git a/meta/classes/rootfs-postcommands.bbclass b/meta/classes/rootfs-postcommands.bbclass
index 2310e86cdf..c33c4d0aed 100644
--- a/meta/classes/rootfs-postcommands.bbclass
+++ b/meta/classes/rootfs-postcommands.bbclass
@@ -17,6 +17,9 @@ ROOTFS_POSTPROCESS_COMMAND += "rootfs_update_timestamp; "
 # Tweak the mount options for rootfs in /etc/fstab if read-only-rootfs is enabled
 ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("IMAGE_FEATURES", "read-only-rootfs", "read_only_rootfs_hook; ", "",d)}'
 
+# Leave only pyc if pyc-only is enabled 
+ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("IMAGE_FEATURES", "pyc-only", "pyc_only_hook; ", "",d)}'
+
 # We also need to do the same for the kernel boot parameters,
 # otherwise kernel or initramfs end up mounting the rootfs read/write
 # (the default) if supported by the underlying storage.
@@ -139,6 +142,30 @@ read_only_rootfs_hook () {
 	fi
 }
 
+
+inherit python3native
+pyc_only_hook () {
+	# Create pyc files
+	bbwarn here
+	nativepython3 \
+		${COREBASE}/scripts/contrib/pycompile.py \
+		--verbose \
+		--strip-root ${IMAGE_ROOTFS} \
+		${IMAGE_ROOTFS}/usr/lib/python3.9
+
+	# Remove py files
+	bbwarn here2
+	find ${IMAGE_ROOTFS}/usr/lib/python3.9 -name '*.py' \
+		-print0 | \
+		xargs -0 --no-run-if-empty rm -f
+
+	# Remove pycache files
+	bbwarn here3
+	find ${IMAGE_ROOTFS}/usr/lib/python3.9 -name '__pycache__' \
+		-print0 | \
+		xargs -0 --no-run-if-empty rm -rf
+}
+
 #
 # This function is intended to disallow empty root password if 'debug-tweaks' is not in IMAGE_FEATURES.
 #