Reducing the size of the image by optimizing python

Reducing the size of the image by optimizing python

[Abhi Arora]

[-- Attachment #1: Type: text/plain, Size: 332 bytes --]

Hello,
I am having an embedded system. We have bsp with python 3.5 but we want to reduce its size further. I am planning to have only oyc files but not sure how it can be achieved in recipes. Also, I want to know if there is a other way to reduce the footprint.

Please help.

Get Outlook for Android<https://aka.ms/ghei36>


[-- Attachment #2: Type: text/html, Size: 1177 bytes --]

Re: Reducing the size of the image by optimizing python

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 791 bytes --]

On Fri, Oct 18, 2019 at 1:21 PM Abhi Arora <abhiarora4@live.com> wrote:

> Hello,
> I am having an embedded system. We have bsp with python 3.5 but we want to
> reduce its size further. I am planning to have only oyc files but not sure
> how it can be achieved in recipes. Also, I want to know if there is a other
> way to reduce the footprint.
>

Please start by eliminating modules first so find which modules are
required for your workloads and remove all other modules from image by
adding them to BAD_RECOMMENDATIONS from IMAGE_INSTALL

>
>
> Please help.
>
> Get Outlook for Android <https://aka.ms/ghei36>
>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

[-- Attachment #2: Type: text/html, Size: 2370 bytes --]

Re: Reducing the size of the image by optimizing python

[Abhi Arora]

[-- Attachment #1: Type: text/plain, Size: 1835 bytes --]

Thanks for the reply. Can you help me which file to modify to prevent downloading certain packages and modules using BAD_RECOMMENDATION?

And how to have only pyc files in the final image to improve speed and reduce footprint?

Following are the lines in my yocto project which install python packages.

IMAGE_INSTALL_append = " \
    python3 \
    python3-core \
    python3-misc \
    python3-async \
    python3-six \
    python3-websockets \
    python3-requests \
    python3-urllib3 \
    python3-chardet \
    python3-websocket-client \
    python3-pyopenssl \
    python3-simplejson \
    python3-cython \

Pardon for some stupid questions as I am new to yocto.

Get Outlook for Android<https://aka.ms/ghei36>

________________________________
From: Khem Raj <raj.khem@gmail.com>
Sent: Friday, October 18, 2019 2:28:23 PM
To: Abhi Arora <abhiarora4@live.com>
Cc: yocto@yoctoproject.org <yocto@yoctoproject.org>
Subject: Re: [yocto] Reducing the size of the image by optimizing python



On Fri, Oct 18, 2019 at 1:21 PM Abhi Arora <abhiarora4@live.com<mailto:abhiarora4@live.com>> wrote:
Hello,
I am having an embedded system. We have bsp with python 3.5 but we want to reduce its size further. I am planning to have only oyc files but not sure how it can be achieved in recipes. Also, I want to know if there is a other way to reduce the footprint.

Please start by eliminating modules first so find which modules are required for your workloads and remove all other modules from image by adding them to BAD_RECOMMENDATIONS from IMAGE_INSTALL


Please help.

Get Outlook for Android<https://aka.ms/ghei36>

--
_______________________________________________
yocto mailing list
yocto@yoctoproject.org<mailto:yocto@yoctoproject.org>
https://lists.yoctoproject.org/listinfo/yocto

[-- Attachment #2: Type: text/html, Size: 6749 bytes --]

Re: Reducing the size of the image by optimizing python

[Richard Purdie]

On Sat, 2019-10-19 at 12:45 +0000, Abhi Arora wrote:
> Thanks for the reply. Can you help me which file to modify to prevent
> downloading certain packages and modules using BAD_RECOMMENDATION? 
> 
> And how to have only pyc files in the final image to improve speed
> and reduce footprint?
> 
> Following are the lines in my yocto project which install python
> packages.
> 
> IMAGE_INSTALL_append = " \
>     python3 \
>     python3-core \
>     python3-misc \
>     python3-async \
>     python3-six \
>     python3-websockets \
>     python3-requests \
>     python3-urllib3 \
>     python3-chardet \
>     python3-websocket-client \
>     python3-pyopenssl \
>     python3-simplejson \
>     python3-cython \
> 
> Pardon for some stupid questions as I am new to yocto.

Take python3-misc out that list as its huge. See if anything breaks, if
so, try and install just what you need, not the large "catchall"
package.

Cheers,

Richard

Re: Reducing the size of the image by optimizing python

[Abhi Arora]

[-- Attachment #1: Type: text/plain, Size: 1628 bytes --]

Thanks your for the suggestion.

From where I can find out what modules and packages python3-misc downloads and installs? I am new to yocto and Don't know where to look for. I tried grep but didn't help me.

And how about optimization using pyc file? Is it doable?

Get Outlook for Android<https://aka.ms/ghei36>

________________________________
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Sent: Saturday, October 19, 2019 8:04:58 PM
To: Abhi Arora <abhiarora4@live.com>; Khem Raj <raj.khem@gmail.com>
Cc: yocto@yoctoproject.org <yocto@yoctoproject.org>
Subject: Re: [yocto] Reducing the size of the image by optimizing python

On Sat, 2019-10-19 at 12:45 +0000, Abhi Arora wrote:
> Thanks for the reply. Can you help me which file to modify to prevent
> downloading certain packages and modules using BAD_RECOMMENDATION?
>
> And how to have only pyc files in the final image to improve speed
> and reduce footprint?
>
> Following are the lines in my yocto project which install python
> packages.
>
> IMAGE_INSTALL_append = " \
>     python3 \
>     python3-core \
>     python3-misc \
>     python3-async \
>     python3-six \
>     python3-websockets \
>     python3-requests \
>     python3-urllib3 \
>     python3-chardet \
>     python3-websocket-client \
>     python3-pyopenssl \
>     python3-simplejson \
>     python3-cython \
>
> Pardon for some stupid questions as I am new to yocto.

Take python3-misc out that list as its huge. See if anything breaks, if
so, try and install just what you need, not the large "catchall"
package.

Cheers,

Richard


[-- Attachment #2: Type: text/html, Size: 3488 bytes --]

Re: Reducing the size of the image by optimizing python

[richard.purdie]

On Sat, 2019-10-19 at 14:55 +0000, Abhi Arora wrote:
> Thanks your for the suggestion. 
> 
> From where I can find out what modules and packages python3-misc
> downloads and installs? I am new to yocto and Don't know where to
> look for. I tried grep but didn't help me.

python3-misc is a package. You can look at its dependencies to see what
it adds to the image.

> And how about optimization using pyc file? Is it doable?

Start simple. If you have python3-misc installed its using a lot of
space and is the sensible place to start with trimming things down.

Optimising to just pyc files is an optimisation further than most
people find they need and will be much harder to do.

Cheers,

Richard

Re: Reducing the size of the image by optimizing python

[Yann Dirson]

You can also look at the package dependency graph (see the file
generated by bitbake -g) to get a better idea of what pulls what.

2019-10-19 17:10 UTC+02:00, richard.purdie@linuxfoundation.org
<richard.purdie@linuxfoundation.org>:
> On Sat, 2019-10-19 at 14:55 +0000, Abhi Arora wrote:
>> Thanks your for the suggestion.
>>
>> From where I can find out what modules and packages python3-misc
>> downloads and installs? I am new to yocto and Don't know where to
>> look for. I tried grep but didn't help me.
>
> python3-misc is a package. You can look at its dependencies to see what
> it adds to the image.
>
>> And how about optimization using pyc file? Is it doable?
>
> Start simple. If you have python3-misc installed its using a lot of
> space and is the sensible place to start with trimming things down.
>
> Optimising to just pyc files is an optimisation further than most
> people find they need and will be much harder to do.
>
> Cheers,
>
> Richard
>
>
>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>


-- 
Yann Dirson <yann@blade-group.com>
Blade / Shadow -- http://shadow.tech

Re: Reducing the size of the image by optimizing python

[Abhi Arora]

[-- Attachment #1: Type: text/plain, Size: 2198 bytes --]

Thank you for being patient.

I am unable to find out python3-misc package online. Neither "sudo apt-get install python3-misc" helps me nor "pip install python3-misc".

I am a newbie, please help me what kind of package it is and where to find it.

I am planning to install it in my host machine and see if I am able to run my scripts and manually add or remove modules and python packages to get minimal required system for my scripts.

For that I need to access python3-misc and what it installs.



Get Outlook for Android<https://aka.ms/ghei36>

________________________________
From: Yann Dirson <yann.dirson@blade-group.com>
Sent: Saturday, October 19, 2019 8:51:52 PM
To: richard.purdie@linuxfoundation.org <richard.purdie@linuxfoundation.org>
Cc: Abhi Arora <abhiarora4@live.com>; Khem Raj <raj.khem@gmail.com>; yocto@yoctoproject.org <yocto@yoctoproject.org>
Subject: Re: [yocto] Reducing the size of the image by optimizing python

You can also look at the package dependency graph (see the file
generated by bitbake -g) to get a better idea of what pulls what.

2019-10-19 17:10 UTC+02:00, richard.purdie@linuxfoundation.org
<richard.purdie@linuxfoundation.org>:
> On Sat, 2019-10-19 at 14:55 +0000, Abhi Arora wrote:
>> Thanks your for the suggestion.
>>
>> From where I can find out what modules and packages python3-misc
>> downloads and installs? I am new to yocto and Don't know where to
>> look for. I tried grep but didn't help me.
>
> python3-misc is a package. You can look at its dependencies to see what
> it adds to the image.
>
>> And how about optimization using pyc file? Is it doable?
>
> Start simple. If you have python3-misc installed its using a lot of
> space and is the sensible place to start with trimming things down.
>
> Optimising to just pyc files is an optimisation further than most
> people find they need and will be much harder to do.
>
> Cheers,
>
> Richard
>
>
>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>


--
Yann Dirson <yann@blade-group.com>
Blade / Shadow -- http://shadow.tech

[-- Attachment #2: Type: text/html, Size: 4984 bytes --]

Re: Reducing the size of the image by optimizing python

[Josef Holzmayr]

On Sat, Oct 19, 2019 at 04:32:26PM +0000, Abhi Arora wrote:
> I am planning to install it in my host machine and see if I am able to run my scripts and manually add or remove modules and python packages to get minimal required system for my scripts.
> 
> For that I need to access python3-misc and what it installs.

Its not something that directly corresponds to a standard desktop
distribution. Rather, its "everything that would otherwise be left
behind", see:

https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-devtools/python/python3_3.7.4.bb#n323

So its just like the advice was already: remove it, see what breaks, and
then add smaller bits and pieces again until everything works as
desired.

Greetz
> 
> 
> 
> Get Outlook for Android<https://aka.ms/ghei36>
> 
> ________________________________
> From: Yann Dirson <yann.dirson@blade-group.com>
> Sent: Saturday, October 19, 2019 8:51:52 PM
> To: richard.purdie@linuxfoundation.org <richard.purdie@linuxfoundation.org>
> Cc: Abhi Arora <abhiarora4@live.com>; Khem Raj <raj.khem@gmail.com>; yocto@yoctoproject.org <yocto@yoctoproject.org>
> Subject: Re: [yocto] Reducing the size of the image by optimizing python
> 
> You can also look at the package dependency graph (see the file
> generated by bitbake -g) to get a better idea of what pulls what.
> 
> 2019-10-19 17:10 UTC+02:00, richard.purdie@linuxfoundation.org
> <richard.purdie@linuxfoundation.org>:
> > On Sat, 2019-10-19 at 14:55 +0000, Abhi Arora wrote:
> >> Thanks your for the suggestion.
> >>
> >> From where I can find out what modules and packages python3-misc
> >> downloads and installs? I am new to yocto and Don't know where to
> >> look for. I tried grep but didn't help me.
> >
> > python3-misc is a package. You can look at its dependencies to see what
> > it adds to the image.
> >
> >> And how about optimization using pyc file? Is it doable?
> >
> > Start simple. If you have python3-misc installed its using a lot of
> > space and is the sensible place to start with trimming things down.
> >
> > Optimising to just pyc files is an optimisation further than most
> > people find they need and will be much harder to do.
> >
> > Cheers,
> >
> > Richard
> >
> >
> >
> > --
> > _______________________________________________
> > yocto mailing list
> > yocto@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/yocto
> >
> 
> 
> --
> Yann Dirson <yann@blade-group.com>
> Blade / Shadow -- http://shadow.tech

> -- 
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto


-- 
———————————————
Josef Holzmayr
Software Developer Embedded Systems

Tel: +49 8444 9204-48
Fax: +49 8444 9204-50

R-S-I Elektrotechnik GmbH & Co. KG
Woelkestrasse 11
D-85301 Schweitenkirchen
www.rsi-elektrotechnik.de
———————————————
Amtsgericht Ingolstadt – GmbH: HRB 191328 – KG: HRA 170393
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
Ust-IdNr: DE 128592548 

_____________________________________________________________
Amtsgericht Ingolstadt - GmbH: HRB 191328 - KG: HRA 170363
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
USt-IdNr.: DE 128592548

Re: Reducing the size of the image by optimizing python

[Abhi Arora]

[-- Attachment #1: Type: text/plain, Size: 3994 bytes --]

Hello,
I have listed the python modules and packages that I don't want. Someone recommended to list them with BAD_RECOMMENDATION. Can someone help me where this variable should go?

Thank you.

Get Outlook for Android<https://aka.ms/ghei36>

________________________________
From: Josef Holzmayr <holzmayr@rsi-elektrotechnik.de>
Sent: Sunday, October 20, 2019 4:29:00 PM
To: Abhi Arora <abhiarora4@live.com>
Cc: Yann Dirson <yann.dirson@blade-group.com>; richard.purdie@linuxfoundation.org <richard.purdie@linuxfoundation.org>; yocto@yoctoproject.org <yocto@yoctoproject.org>
Subject: Re: [yocto] Reducing the size of the image by optimizing python

On Sat, Oct 19, 2019 at 04:32:26PM +0000, Abhi Arora wrote:
> I am planning to install it in my host machine and see if I am able to run my scripts and manually add or remove modules and python packages to get minimal required system for my scripts.
>
> For that I need to access python3-misc and what it installs.

Its not something that directly corresponds to a standard desktop
distribution. Rather, its "everything that would otherwise be left
behind", see:

https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-devtools/python/python3_3.7.4.bb#n323

So its just like the advice was already: remove it, see what breaks, and
then add smaller bits and pieces again until everything works as
desired.

Greetz
>
>
>
> Get Outlook for Android<https://aka.ms/ghei36>
>
> ________________________________
> From: Yann Dirson <yann.dirson@blade-group.com>
> Sent: Saturday, October 19, 2019 8:51:52 PM
> To: richard.purdie@linuxfoundation.org <richard.purdie@linuxfoundation.org>
> Cc: Abhi Arora <abhiarora4@live.com>; Khem Raj <raj.khem@gmail.com>; yocto@yoctoproject.org <yocto@yoctoproject.org>
> Subject: Re: [yocto] Reducing the size of the image by optimizing python
>
> You can also look at the package dependency graph (see the file
> generated by bitbake -g) to get a better idea of what pulls what.
>
> 2019-10-19 17:10 UTC+02:00, richard.purdie@linuxfoundation.org
> <richard.purdie@linuxfoundation.org>:
> > On Sat, 2019-10-19 at 14:55 +0000, Abhi Arora wrote:
> >> Thanks your for the suggestion.
> >>
> >> From where I can find out what modules and packages python3-misc
> >> downloads and installs? I am new to yocto and Don't know where to
> >> look for. I tried grep but didn't help me.
> >
> > python3-misc is a package. You can look at its dependencies to see what
> > it adds to the image.
> >
> >> And how about optimization using pyc file? Is it doable?
> >
> > Start simple. If you have python3-misc installed its using a lot of
> > space and is the sensible place to start with trimming things down.
> >
> > Optimising to just pyc files is an optimisation further than most
> > people find they need and will be much harder to do.
> >
> > Cheers,
> >
> > Richard
> >
> >
> >
> > --
> > _______________________________________________
> > yocto mailing list
> > yocto@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/yocto
> >
>
>
> --
> Yann Dirson <yann@blade-group.com>
> Blade / Shadow -- http://shadow.tech

> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto


--
———————————————
Josef Holzmayr
Software Developer Embedded Systems

Tel: +49 8444 9204-48
Fax: +49 8444 9204-50

R-S-I Elektrotechnik GmbH & Co. KG
Woelkestrasse 11
D-85301 Schweitenkirchen
www.rsi-elektrotechnik.de<http://www.rsi-elektrotechnik.de>
———————————————
Amtsgericht Ingolstadt – GmbH: HRB 191328 – KG: HRA 170393
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
Ust-IdNr: DE 128592548

_____________________________________________________________
Amtsgericht Ingolstadt - GmbH: HRB 191328 - KG: HRA 170363
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
USt-IdNr.: DE 128592548


[-- Attachment #2: Type: text/html, Size: 6454 bytes --]

Re: Reducing the size of the image by optimizing python

[Ross Burton]

On 19/10/2019 13:45, Abhi Arora wrote:
> IMAGE_INSTALL_append = " \
>      python3 \

^^^^

This is installing "all of python".  If you want to trim your Python 
installation then don't do that, instead determine what libraries you 
*actually* need and explicitly add those.

Ross

Re: Reducing the size of the image by optimizing python

[Mike Looijmans]

> Optimising to just pyc files is an optimisation further than most
> people find they need and will be much harder to do.

It's actually quite simple, just add a bbappend to the python recipe that puts 
all .py files into the "dbg" or newly created "src" package. Or simply delete 
all .py files in a do_install_append.


Here's an example of the "src" approach:

https://github.com/OpenPLi/openpli-oe-core/blob/develop/meta-openpli/recipes-devtools/python/python_2.7.13.bbappend


Simply doing this in the bbappend might also work (untested):

do_install_append() {
	find ${D}${libdir}/python${PYTHON_MAJMIN}/ -name '*.py' -remove
}


Removing the ".py" files roughly cuts the disk space in half.


The "src" approach works on any package, and has the advantage that you can 
still install the source files for debugging and development.

Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
Postbus 440, 5680 AK Best
The Netherlands

T: +31 (0) 499 33 69 69
E: {E-mail
W: www.topicproducts.com

Please consider the environment before printing this e-mail