Building single package as image, respecting dependencies

Building single package as image, respecting dependencies

[Norman Stetter]

[-- Attachment #1.1: Type: text/plain, Size: 2353 bytes --]

Hi there,

I am currently working on a minimal CLI only system.
The image is based on 'core-image-base' using sysvinit and busybox.
To reduce image size and boot time, I removed as many packages as possible. For some application cases I will need python3 though.
My idea was to have python3 as some kind of add-on in a squashfs image, that can be mounted only when needed.

So I tried to build an image only containing the python3 package, but as little as possible otherwise, like this:

inherit image
IMAGE_FSTYPES = "squashfs-xz"
DEFAULT_TASK_PROVIDER = ""
DISTRO_EXTRA_RDEPENDS = ""
DISTRO_FEATURES = ""
POKY_DEFAULT_EXTRA_RDEPENDS = ""
IMAGE_FEATURES = ""
IMAGE_LINGUAS = ""
CORE_IMAGE_BASE_INSTALL = ""
RDEPENDS_${PN} = ""
PACKAGE_EXCLUDE = "busybox openssl run-postinsts update-rc.d"
VIRTUAL-RUNTIME_dev_manager = ""
VIRTUAL-RUNTIME_login_manager = ""
VIRTUAL-RUNTIME_init_manager = ""
VIRTUAL-RUNTIME_initscripts = ""
VIRTUAL-RUNTIME_keymaps = ""
VIRTUAL-RUNTIME_base-utils = ""
PREFERRED_PROVIDER_virtual/base-utils = ""

IMAGE_INSTALL = "python3"

But like this I can only manually exclude packages I already have in my main OS image. Some packages can't be excluded at all, as python3 depends on them and won't build if they are excluded.

Is there a way to have dependencies between images? So I could have the python-image build know which dependencies are already built into my OS image and therefore not include them itself?

Or would it be better to avoid building a second image and rather build 'python3' with my OS-image, adding it to PACKAGE_EXCLUDE and from within this build process pack all python3 files into an image? If I were to pursue this method, any suggestions on how to separate the python3 files from the rest of my rootfs, including the necessary dependencies?

Any advice and idea is much appreciated.

Mit freundlichen Grüßen / Best regards
Norman Stetter
SW ENWICKLUNG EMBEDDED SYSTEMS
Garz & Fricke GmbH

21079 Hamburg
Direct: +49 40 791899 - 477
Fax: +49 40 791899 - 39
Norman.Stetter@garz-fricke.com
www.garz-fricke.com<http://www.garz-fricke.com/>
WE MAKE IT YOURS!
[cid:image001.jpg@01D510A9.7F9CD640]
Sitz der Gesellschaft: D-21079 Hamburg
Registergericht: Amtsgericht Hamburg, HRB 60514
Geschäftsführer: Matthias Fricke, Manfred Garz, Marc-Michael Braun


[-- Attachment #1.2: Type: text/html, Size: 10942 bytes --]

[-- Attachment #2: image001.jpg --]
[-- Type: image/jpeg, Size: 14130 bytes --]

Re: Building single package as image, respecting dependencies

[Burton, Ross]

[-- Attachment #1.1: Type: text/plain, Size: 598 bytes --]

On Wed, 22 May 2019 at 14:28, Norman Stetter <Norman.Stetter@garz-fricke.com>
wrote:

> Is there a way to have dependencies between images? So I could have the
> python-image build know which dependencies are already built into my OS
> image and therefore not include them itself?
>

No.  An image is a self-contained file system.  About 50% of your image
recipe isn't helping your goal of an image that contains a single package.

I have to query your assumptions though: if you're going to have python3 in
a filessytem that you mount when required, how are you saving space?

Ross

[-- Attachment #1.2: Type: text/html, Size: 1097 bytes --]

[-- Attachment #2: image001.jpg --]
[-- Type: image/jpeg, Size: 14130 bytes --]

Re: Building single package as image, respecting dependencies

[Norman Stetter]

[-- Attachment #1.1: Type: text/plain, Size: 1944 bytes --]

Hi Ross,

thanks for clarifying that there are no dependencies between images.


Von: Burton, Ross [mailto:ross.burton@intel.com]
Gesendet: Thursday, May 23, 2019 2:15 PM
An: Norman Stetter <Norman.Stetter@garz-fricke.com>
Cc: yocto@yoctoproject.org
Betreff: Re: [yocto] Building single package as image, respecting dependencies

On Wed, 22 May 2019 at 14:28, Norman Stetter <Norman.Stetter@garz-fricke.com<mailto:Norman.Stetter@garz-fricke.com>> wrote:
Is there a way to have dependencies between images? So I could have the python-image build know which dependencies are already built into my OS image and therefore not include them itself?

No.  An image is a self-contained file system.  About 50% of your image recipe isn't helping your goal of an image that contains a single package.

What should be improved to achieve my goal of containing only the python3 package and as little dependencies as possible?

I have to query your assumptions though: if you're going to have python3 in a filessytem that you mount when required, how are you saving space?


That seems to be a misunderstanding. I want to keep the OS image as small as possible to reduce boot time. We only use python for some test scenarios, so it doesn’t have to be included in the OS image. I did some measurements and including python3 in the OS image increased the boot time by over 30%. So the idea is to have it separately, and only mount it, when needed.


Mit freundlichen Grüßen / Best regards
Norman Stetter
SW ENWICKLUNG EMBEDDED SYSTEMS
Garz & Fricke GmbH

21079 Hamburg
Direct: +49 40 791899 - 477
Fax: +49 40 791899 - 39
Norman.Stetter@garz-fricke.com
www.garz-fricke.com<http://www.garz-fricke.com/>
WE MAKE IT YOURS!
[cid:image001.jpg@01D51176.36275C60]
Sitz der Gesellschaft: D-21079 Hamburg
Registergericht: Amtsgericht Hamburg, HRB 60514
Geschäftsführer: Matthias Fricke, Manfred Garz, Marc-Michael Braun



[-- Attachment #1.2: Type: text/html, Size: 9293 bytes --]

[-- Attachment #2: image001.jpg --]
[-- Type: image/jpeg, Size: 14130 bytes --]

Re: Building single package as image, respecting dependencies

[Burton, Ross]

[-- Attachment #1.1: Type: text/plain, Size: 780 bytes --]

On Thu, 23 May 2019 at 13:46, Norman Stetter <Norman.Stetter@garz-fricke.com>
wrote:

> That seems to be a misunderstanding. I want to keep the OS image as small
> as possible to reduce boot time. We only use python for some test
> scenarios, so it doesn’t have to be included in the OS image. I did some
> measurements and including python3 in the OS image increased the boot time
> by over 30%. So the idea is to have it separately, and only mount it, when
> needed.
>

What filesystem are you using?  I wasn't aware of any modern file systems
where the number of files installed affected mount performance.

Could you just build two images and have a 'normal' image and a 'test
image'?  This is basically what the tools-testapps image feature is for.

Ross

[-- Attachment #1.2: Type: text/html, Size: 1368 bytes --]

[-- Attachment #2: image001.jpg --]
[-- Type: image/jpeg, Size: 14130 bytes --]

Re: Building single package as image, respecting dependencies

[Norman Stetter]

[-- Attachment #1.1: Type: text/plain, Size: 1997 bytes --]

For the OS image I use cpio.gz as file system. It gets booted as a RAMdisk. Sorry, forgot to mention that.

Building two different images doesn’t suit our use case. We install the image on a device and run different automated test suites on it, some of which need python. But for all of the testing time is critical.

I now have it running with the recipe I posted in the first mail in this thread. But I still can’t get rid of busybox in my python image.

Mit freundlichen Grüßen / Best regards
Norman Stetter
SW ENWICKLUNG EMBEDDED SYSTEMS
Garz & Fricke GmbH

21079 Hamburg
Direct: +49 40 791899 - 477
Fax: +49 40 791899 - 39
Norman.Stetter@garz-fricke.com
www.garz-fricke.com<http://www.garz-fricke.com/>
WE MAKE IT YOURS!
[cid:image001.jpg@01D5120A.5B66BBD0]
Sitz der Gesellschaft: D-21079 Hamburg
Registergericht: Amtsgericht Hamburg, HRB 60514
Geschäftsführer: Matthias Fricke, Manfred Garz, Marc-Michael Braun

Von: Burton, Ross [mailto:ross.burton@intel.com]
Gesendet: Thursday, May 23, 2019 3:46 PM
An: Norman Stetter <Norman.Stetter@garz-fricke.com>
Cc: yocto@yoctoproject.org
Betreff: Re: [yocto] Building single package as image, respecting dependencies

On Thu, 23 May 2019 at 13:46, Norman Stetter <Norman.Stetter@garz-fricke.com<mailto:Norman.Stetter@garz-fricke.com>> wrote:
That seems to be a misunderstanding. I want to keep the OS image as small as possible to reduce boot time. We only use python for some test scenarios, so it doesn’t have to be included in the OS image. I did some measurements and including python3 in the OS image increased the boot time by over 30%. So the idea is to have it separately, and only mount it, when needed.

What filesystem are you using?  I wasn't aware of any modern file systems where the number of files installed affected mount performance.

Could you just build two images and have a 'normal' image and a 'test image'?  This is basically what the tools-testapps image feature is for.

Ross

[-- Attachment #1.2: Type: text/html, Size: 8641 bytes --]

[-- Attachment #2: image001.jpg --]
[-- Type: image/jpeg, Size: 14130 bytes --]

Re: Building single package as image, respecting dependencies

[Burton, Ross]

[-- Attachment #1.1: Type: text/plain, Size: 388 bytes --]

On Fri, 24 May 2019 at 07:30, Norman Stetter <Norman.Stetter@garz-fricke.com>
wrote:

> For the OS image I use cpio.gz as file system. It gets booted as a
> RAMdisk. Sorry, forgot to mention that.
>

Would you consider switching to a proper filesystem?  The peril of using a
cpio.gz is that every file you add slows the boot down: using a proper
filesystem solves this.

Ross

[-- Attachment #1.2: Type: text/html, Size: 952 bytes --]

[-- Attachment #2: image001.jpg --]
[-- Type: image/jpeg, Size: 14130 bytes --]

Re: Building single package as image, respecting dependencies

[Norman Stetter]

[-- Attachment #1.1: Type: text/plain, Size: 1491 bytes --]

In fact the OS is just a minimal rescue OS. On a second partition we run our main OS, which is ‘properly’ installed.
So, I’m afraid using a ‘proper’ file system wouldn’t fit our use case. We run the rescue OS as RAMdisk to be able to self-update it without having to worry about modifying a mounted file system.

Norman



Mit freundlichen Grüßen / Best regards
Norman Stetter
SW ENWICKLUNG EMBEDDED SYSTEMS
Garz & Fricke GmbH

21079 Hamburg
Direct: +49 40 791899 - 477
Fax: +49 40 791899 - 39
Norman.Stetter@garz-fricke.com
www.garz-fricke.com<http://www.garz-fricke.com/>
WE MAKE IT YOURS!
[cid:image001.jpg@01D51228.BD16AF70]
Sitz der Gesellschaft: D-21079 Hamburg
Registergericht: Amtsgericht Hamburg, HRB 60514
Geschäftsführer: Matthias Fricke, Manfred Garz, Marc-Michael Braun

Von: Burton, Ross [mailto:ross.burton@intel.com]
Gesendet: Friday, May 24, 2019 11:45 AM
An: Norman Stetter <Norman.Stetter@garz-fricke.com>
Cc: yocto@yoctoproject.org
Betreff: Re: [yocto] Building single package as image, respecting dependencies

On Fri, 24 May 2019 at 07:30, Norman Stetter <Norman.Stetter@garz-fricke.com<mailto:Norman.Stetter@garz-fricke.com>> wrote:
For the OS image I use cpio.gz as file system. It gets booted as a RAMdisk. Sorry, forgot to mention that.

Would you consider switching to a proper filesystem?  The peril of using a cpio.gz is that every file you add slows the boot down: using a proper filesystem solves this.

Ross

[-- Attachment #1.2: Type: text/html, Size: 8207 bytes --]

[-- Attachment #2: image001.jpg --]
[-- Type: image/jpeg, Size: 14130 bytes --]

Re: Building single package as image, respecting dependencies

[Norman Stetter]

[-- Attachment #1.1: Type: text/plain, Size: 1355 bytes --]

Is there no way to simply force bitbake to only build the packages I want, ignoring dependencies? We used to build the image like this before, using PTXDist.

Everything else seems to be a dirty hack.

Mit freundlichen Grüßen / Best regards
Norman Stetter
SW ENWICKLUNG EMBEDDED SYSTEMS
Garz & Fricke GmbH

21079 Hamburg
Direct: +49 40 791899 - 477
Fax: +49 40 791899 - 39
Norman.Stetter@garz-fricke.com
www.garz-fricke.com<http://www.garz-fricke.com/>
WE MAKE IT YOURS!
[cid:image001.jpg@01D5124D.B0588270]
Sitz der Gesellschaft: D-21079 Hamburg
Registergericht: Amtsgericht Hamburg, HRB 60514
Geschäftsführer: Matthias Fricke, Manfred Garz, Marc-Michael Braun

Von: Burton, Ross [mailto:ross.burton@intel.com]
Gesendet: Friday, May 24, 2019 11:45 AM
An: Norman Stetter <Norman.Stetter@garz-fricke.com>
Cc: yocto@yoctoproject.org
Betreff: Re: [yocto] Building single package as image, respecting dependencies

On Fri, 24 May 2019 at 07:30, Norman Stetter <Norman.Stetter@garz-fricke.com<mailto:Norman.Stetter@garz-fricke.com>> wrote:
For the OS image I use cpio.gz as file system. It gets booted as a RAMdisk. Sorry, forgot to mention that.

Would you consider switching to a proper filesystem?  The peril of using a cpio.gz is that every file you add slows the boot down: using a proper filesystem solves this.

Ross

[-- Attachment #1.2: Type: text/html, Size: 7546 bytes --]

[-- Attachment #2: image001.jpg --]
[-- Type: image/jpeg, Size: 14130 bytes --]

Re: Building single package as image, respecting dependencies

[Paul Barker]

On 24/05/2019 15:28, Norman Stetter wrote:
> Is there no way to simply force bitbake to only build the packages I 
> want, ignoring dependencies? We used to build the image like this 
> before, using PTXDist.
> 
> Everything else seems to be a dirty hack.

Can you just collect the extra packages (ipk/deb/rpm as appropriate), 
put them in a package feed and install them on-target as required?

The package feed doesn't have to be on a webserver, you could put it in 
a directory in a separate partition on the target if that's what you need.

Thanks,

-- 
Paul Barker
Managing Director & Principal Engineer
Beta Five Ltd

Re: Building single package as image, respecting dependencies

[Norman Stetter]

Hello Paul,

thank you for your suggestion. I will look into it.

Meanwhile I worked around my issue by building all python3 packages, but excluding them from the rootfs. Then they get installed into a separate 'rootfs', which I pack as a squashfs image.

It might not be that elegant, but it works.

Regards,

NOrman


Mit freundlichen Grüßen / Best regards
Norman Stetter
SW ENWICKLUNG EMBEDDED SYSTEMS
Garz & Fricke GmbH

21079 Hamburg
Direct: +49 40 791899 - 477
Fax: +49 40 791899 - 39
Norman.Stetter@garz-fricke.com
www.garz-fricke.com
WE MAKE IT YOURS!

Sitz der Gesellschaft: D-21079 Hamburg
Registergericht: Amtsgericht Hamburg, HRB 60514
Geschäftsführer: Matthias Fricke, Manfred Garz, Marc-Michael Braun

> -----Ursprüngliche Nachricht-----
> Von: Paul Barker [mailto:paul@betafive.co.uk]
> Gesendet: Tuesday, May 28, 2019 11:37 AM
> An: Norman Stetter <Norman.Stetter@garz-fricke.com>; Burton, Ross
> <ross.burton@intel.com>
> Cc: yocto@yoctoproject.org
> Betreff: Re: [yocto] Building single package as image, respecting
> dependencies
> 
> On 24/05/2019 15:28, Norman Stetter wrote:
> > Is there no way to simply force bitbake to only build the packages I
> > want, ignoring dependencies? We used to build the image like this
> > before, using PTXDist.
> >
> > Everything else seems to be a dirty hack.
> 
> Can you just collect the extra packages (ipk/deb/rpm as appropriate), put
> them in a package feed and install them on-target as required?
> 
> The package feed doesn't have to be on a webserver, you could put it in a
> directory in a separate partition on the target if that's what you need.
> 
> Thanks,
> 
> --
> Paul Barker
> Managing Director & Principal Engineer
> Beta Five Ltd