Add package managers as a plugin

Add package managers as a plugin

[Fredrik Gustafsson]

Poky today has three different package managers, the well-known formats deb
and rpm is supported as well as ipkg that is good for embedded devices.

When building and having a good cache hit, a significant amount of time is
spent in the phase of generating a rootfs, which is really about the
performance of the package manager. ipkg is way slower than deb or rpm. To
save build time and also get a package manager that is suitanle for use on
targets where flash memory is a concern, support for apk is suggested.

However, it might or might not be what's wanted for Poky since it increases
the test matrix. Therefore this patch series refactors the package
management code so that it's possible to add more package managers in other
own layer. I will send another patch serie that will add apk.

Perfomance metrics below, (note that this includes build times).

APK
===
bitbake core-image-minimal
15.84s user 2.60s system 0% cpu 1:26:19.21 total
16.01s user 2.58s system 0% cpu 1:26:03.72 total
15.69s user 2.61s system 0% cpu 1:26:45.45 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -f -c do_rootfs
0.55s user 0.06s system 4% cpu 14.236 total
0.54s user 0.08s system 4% cpu 15.247 total
0.52s user 0.15s system 4% cpu 15.143 total

RPM
===
bitbake core-image-minimal
18.57s user 3.09s system 0% cpu 1:31:29.09 total
18.58s user 3.08s system 0% cpu 1:30:53.80 total
18.20s user 3.31s system 0% cpu 1:31:06.69 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -c do_rootfs -f
0.58s user 0.10s system 3% cpu 19.470 total
0.57s user 0.09s system 3% cpu 19.480 total
0.60s user 0.07s system 3% cpu 20.381 total

Re: [OE-core] Add package managers as a plugin

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 2148 bytes --]

Apologies, but these metrics do not look particularly convincing - 15s vs
20s? Can you try it with an image several gigabytes in size?
Core-image-sato-sdk-ptest is a good option.

Alex

On Thu, 25 Jun 2020 at 12:14, Fredrik Gustafsson <
fredrik.gustafsson@axis.com> wrote:

> Poky today has three different package managers, the well-known formats deb
> and rpm is supported as well as ipkg that is good for embedded devices.
>
> When building and having a good cache hit, a significant amount of time is
> spent in the phase of generating a rootfs, which is really about the
> performance of the package manager. ipkg is way slower than deb or rpm. To
> save build time and also get a package manager that is suitanle for use on
> targets where flash memory is a concern, support for apk is suggested.
>
> However, it might or might not be what's wanted for Poky since it increases
> the test matrix. Therefore this patch series refactors the package
> management code so that it's possible to add more package managers in other
> own layer. I will send another patch serie that will add apk.
>
> Perfomance metrics below, (note that this includes build times).
>
> APK
> ===
> bitbake core-image-minimal
> 15.84s user 2.60s system 0% cpu 1:26:19.21 total
> 16.01s user 2.58s system 0% cpu 1:26:03.72 total
> 15.69s user 2.61s system 0% cpu 1:26:45.45 total
>
> bitbake core-image-minimal -c cleansstate
> bitbake core-image-minimal -c clean
> bitbake core-image-minimal -f -c do_rootfs
> 0.55s user 0.06s system 4% cpu 14.236 total
> 0.54s user 0.08s system 4% cpu 15.247 total
> 0.52s user 0.15s system 4% cpu 15.143 total
>
> RPM
> ===
> bitbake core-image-minimal
> 18.57s user 3.09s system 0% cpu 1:31:29.09 total
> 18.58s user 3.08s system 0% cpu 1:30:53.80 total
> 18.20s user 3.31s system 0% cpu 1:31:06.69 total
>
> bitbake core-image-minimal -c cleansstate
> bitbake core-image-minimal -c clean
> bitbake core-image-minimal -c do_rootfs -f
> 0.58s user 0.10s system 3% cpu 19.470 total
> 0.57s user 0.09s system 3% cpu 19.480 total
> 0.60s user 0.07s system 3% cpu 20.381 total
> 
>

[-- Attachment #2: Type: text/html, Size: 2568 bytes --]

Re: [OE-core] Add package managers as a plugin

[Fredrik Gustafsson]

[-- Attachment #1: Type: text/plain, Size: 3535 bytes --]

Hi Alexander,

what are you finding strange with the numbers? Too fast or to big of a difference? Note that it's only the do_rootfs task that i measured.

I've tried the core-image-sato-sdk-ptest as you suggested. I run the following command multiple times:
bitbake core-image-sato-sdk-ptest -c cleansstate &&

bitbake core-image-sato-sdk-ptest -c clean &&
time bitbake core-image-sato-sdk-ptest -f -c do_rootfs

APK
====
0,85s user 0,16s system 0% cpu 1:13,66 total
0,87s user 0,14s system 0% cpu 1:13,28 total
0,89s user 0,11s system 0% cpu 1:13,10 total

RPM
===

0,67s user 0,12s system 0% cpu 2:02,48 total
0,68s user 0,11s system 0% cpu 2:03,39 total
0,73s user 0,09s system 0% cpu 1:59,87 total

5 seconds doesn't sound that much but it's 25% speedup. Since building is mostly cached but generating rootfs isn't this is a huge time consumption for our use case and our image takes around 4 minutes to build. Saving one minutes times the many builds we do is a huge win.

(Excuse my late reply, running core-image-sato-sdk-ptest uncovered a few bugs in the implementation of apk in OE. So thanks for having me testing this image)


/Fredrik


________________________________
From: Alexander Kanavin <alex.kanavin@gmail.com>
Sent: Thursday, June 25, 2020 12:24 PM
To: Fredrik Gustafsson
Cc: OE-core; tools-cfpbuild-internal; Hugo Cedervall
Subject: Re: [OE-core] Add package managers as a plugin

Apologies, but these metrics do not look particularly convincing - 15s vs 20s? Can you try it with an image several gigabytes in size? Core-image-sato-sdk-ptest is a good option.

Alex

On Thu, 25 Jun 2020 at 12:14, Fredrik Gustafsson <fredrik.gustafsson@axis.com<mailto:fredrik.gustafsson@axis.com>> wrote:
Poky today has three different package managers, the well-known formats deb
and rpm is supported as well as ipkg that is good for embedded devices.

When building and having a good cache hit, a significant amount of time is
spent in the phase of generating a rootfs, which is really about the
performance of the package manager. ipkg is way slower than deb or rpm. To
save build time and also get a package manager that is suitanle for use on
targets where flash memory is a concern, support for apk is suggested.

However, it might or might not be what's wanted for Poky since it increases
the test matrix. Therefore this patch series refactors the package
management code so that it's possible to add more package managers in other
own layer. I will send another patch serie that will add apk.

Perfomance metrics below, (note that this includes build times).

APK
===
bitbake core-image-minimal
15.84s user 2.60s system 0% cpu 1:26:19.21 total
16.01s user 2.58s system 0% cpu 1:26:03.72 total
15.69s user 2.61s system 0% cpu 1:26:45.45 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -f -c do_rootfs
0.55s user 0.06s system 4% cpu 14.236 total
0.54s user 0.08s system 4% cpu 15.247 total
0.52s user 0.15s system 4% cpu 15.143 total

RPM
===
bitbake core-image-minimal
18.57s user 3.09s system 0% cpu 1:31:29.09 total
18.58s user 3.08s system 0% cpu 1:30:53.80 total
18.20s user 3.31s system 0% cpu 1:31:06.69 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -c do_rootfs -f
0.58s user 0.10s system 3% cpu 19.470 total
0.57s user 0.09s system 3% cpu 19.480 total
0.60s user 0.07s system 3% cpu 20.381 total


[-- Attachment #2: Type: text/html, Size: 7617 bytes --]

Re: [OE-core] Add package managers as a plugin

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 4098 bytes --]

What I meant is that 15s vs 20s on a small image could have been 5 min vs
5min5s on a bigger image, so I wanted to see that the difference does scale
up with image size.

Can you to provide sdk-ptest numbers for deb and ipk as well?

Alex

On Tue, 30 Jun 2020 at 17:04, Fredrik Gustafsson <
Fredrik.Gustafsson@axis.com> wrote:

> Hi Alexander,
>
> what are you finding strange with the numbers? Too fast or to big of a
> difference? Note that it's only the do_rootfs task that i measured.
>
> I've tried the core-image-sato-sdk-ptest as you suggested. I run the
> following command multiple times:
> bitbake core-image-sato-sdk-ptest -c cleansstate &&
>
> bitbake core-image-sato-sdk-ptest -c clean &&
> time bitbake core-image-sato-sdk-ptest -f -c do_rootfs
>
>
> APK
> ====
> 0,85s user 0,16s system 0% cpu 1:13,66 total
> 0,87s user 0,14s system 0% cpu 1:13,28 total
> 0,89s user 0,11s system 0% cpu 1:13,10 total
>
> RPM
> ===
>
> 0,67s user 0,12s system 0% cpu 2:02,48 total
> 0,68s user 0,11s system 0% cpu 2:03,39 total
> 0,73s user 0,09s system 0% cpu 1:59,87 total
>
> 5 seconds doesn't sound that much but it's 25% speedup. Since building is
> mostly cached but generating rootfs isn't this is a huge time consumption
> for our use case and our image takes around 4 minutes to build. Saving one
> minutes times the many builds we do is a huge win.
>
> (Excuse my late reply, running core-image-sato-sdk-ptest uncovered a few
> bugs in the implementation of apk in OE. So thanks for having me testing
> this image)
>
>
> /Fredrik
>
>
> ------------------------------
> *From:* Alexander Kanavin <alex.kanavin@gmail.com>
> *Sent:* Thursday, June 25, 2020 12:24 PM
> *To:* Fredrik Gustafsson
> *Cc:* OE-core; tools-cfpbuild-internal; Hugo Cedervall
> *Subject:* Re: [OE-core] Add package managers as a plugin
>
> Apologies, but these metrics do not look particularly convincing - 15s vs
> 20s? Can you try it with an image several gigabytes in size?
> Core-image-sato-sdk-ptest is a good option.
>
> Alex
>
> On Thu, 25 Jun 2020 at 12:14, Fredrik Gustafsson <
> fredrik.gustafsson@axis.com> wrote:
>
>> Poky today has three different package managers, the well-known formats
>> deb
>> and rpm is supported as well as ipkg that is good for embedded devices.
>>
>> When building and having a good cache hit, a significant amount of time is
>> spent in the phase of generating a rootfs, which is really about the
>> performance of the package manager. ipkg is way slower than deb or rpm. To
>> save build time and also get a package manager that is suitanle for use on
>> targets where flash memory is a concern, support for apk is suggested.
>>
>> However, it might or might not be what's wanted for Poky since it
>> increases
>> the test matrix. Therefore this patch series refactors the package
>> management code so that it's possible to add more package managers in
>> other
>> own layer. I will send another patch serie that will add apk.
>>
>> Perfomance metrics below, (note that this includes build times).
>>
>> APK
>> ===
>> bitbake core-image-minimal
>> 15.84s user 2.60s system 0% cpu 1:26:19.21 total
>> 16.01s user 2.58s system 0% cpu 1:26:03.72 total
>> 15.69s user 2.61s system 0% cpu 1:26:45.45 total
>>
>> bitbake core-image-minimal -c cleansstate
>> bitbake core-image-minimal -c clean
>> bitbake core-image-minimal -f -c do_rootfs
>> 0.55s user 0.06s system 4% cpu 14.236 total
>> 0.54s user 0.08s system 4% cpu 15.247 total
>> 0.52s user 0.15s system 4% cpu 15.143 total
>>
>> RPM
>> ===
>> bitbake core-image-minimal
>> 18.57s user 3.09s system 0% cpu 1:31:29.09 total
>> 18.58s user 3.08s system 0% cpu 1:30:53.80 total
>> 18.20s user 3.31s system 0% cpu 1:31:06.69 total
>>
>> bitbake core-image-minimal -c cleansstate
>> bitbake core-image-minimal -c clean
>> bitbake core-image-minimal -c do_rootfs -f
>> 0.58s user 0.10s system 3% cpu 19.470 total
>> 0.57s user 0.09s system 3% cpu 19.480 total
>> 0.60s user 0.07s system 3% cpu 20.381 total
>> 
>>
>

[-- Attachment #2: Type: text/html, Size: 7891 bytes --]

Re: [OE-core] Add package managers as a plugin

[Fredrik Gustafsson]

[-- Attachment #1: Type: text/plain, Size: 4741 bytes --]

Hi Alex,

now I see what you mean. I got a script from Alex Stewart, here's the results for core-image-sato-sdk-ptest:

out.apk.1: 1:13.35
out.apk.2: 1:13.51
out.apk.3: 1:13.23
out.apk.4: 1:14.07
out.apk.5: 1:13.00
out.deb.1: 3:49.37
out.deb.2: 3:50.77
out.deb.3: 3:51.39
out.deb.4: 3:53.40
out.deb.5: 3:53.99
out.ipk.1: 2:38.99
out.ipk.2: 2:39.07
out.ipk.3: 2:35.34
out.ipk.4: 2:36.15
out.ipk.5: 2:34.55
out.rpm.1: 1:58.61
out.rpm.2: 1:59.42
out.rpm.3: 1:59.70
out.rpm.4: 1:58.96
out.rpm.5: 1:58.11


BR

Fredrik

________________________________
From: Alexander Kanavin <alex.kanavin@gmail.com>
Sent: Tuesday, June 30, 2020 5:43 PM
To: Fredrik Gustafsson
Cc: OE-core; tools-cfpbuild-internal; Hugo Cedervall
Subject: Re: [OE-core] Add package managers as a plugin

What I meant is that 15s vs 20s on a small image could have been 5 min vs 5min5s on a bigger image, so I wanted to see that the difference does scale up with image size.

Can you to provide sdk-ptest numbers for deb and ipk as well?

Alex

On Tue, 30 Jun 2020 at 17:04, Fredrik Gustafsson <Fredrik.Gustafsson@axis.com<mailto:Fredrik.Gustafsson@axis.com>> wrote:

Hi Alexander,

what are you finding strange with the numbers? Too fast or to big of a difference? Note that it's only the do_rootfs task that i measured.

I've tried the core-image-sato-sdk-ptest as you suggested. I run the following command multiple times:
bitbake core-image-sato-sdk-ptest -c cleansstate &&

bitbake core-image-sato-sdk-ptest -c clean &&
time bitbake core-image-sato-sdk-ptest -f -c do_rootfs

APK
====
0,85s user 0,16s system 0% cpu 1:13,66 total
0,87s user 0,14s system 0% cpu 1:13,28 total
0,89s user 0,11s system 0% cpu 1:13,10 total

RPM
===

0,67s user 0,12s system 0% cpu 2:02,48 total
0,68s user 0,11s system 0% cpu 2:03,39 total
0,73s user 0,09s system 0% cpu 1:59,87 total

5 seconds doesn't sound that much but it's 25% speedup. Since building is mostly cached but generating rootfs isn't this is a huge time consumption for our use case and our image takes around 4 minutes to build. Saving one minutes times the many builds we do is a huge win.

(Excuse my late reply, running core-image-sato-sdk-ptest uncovered a few bugs in the implementation of apk in OE. So thanks for having me testing this image)


/Fredrik


________________________________
From: Alexander Kanavin <alex.kanavin@gmail.com<mailto:alex.kanavin@gmail.com>>
Sent: Thursday, June 25, 2020 12:24 PM
To: Fredrik Gustafsson
Cc: OE-core; tools-cfpbuild-internal; Hugo Cedervall
Subject: Re: [OE-core] Add package managers as a plugin

Apologies, but these metrics do not look particularly convincing - 15s vs 20s? Can you try it with an image several gigabytes in size? Core-image-sato-sdk-ptest is a good option.

Alex

On Thu, 25 Jun 2020 at 12:14, Fredrik Gustafsson <fredrik.gustafsson@axis.com<mailto:fredrik.gustafsson@axis.com>> wrote:
Poky today has three different package managers, the well-known formats deb
and rpm is supported as well as ipkg that is good for embedded devices.

When building and having a good cache hit, a significant amount of time is
spent in the phase of generating a rootfs, which is really about the
performance of the package manager. ipkg is way slower than deb or rpm. To
save build time and also get a package manager that is suitanle for use on
targets where flash memory is a concern, support for apk is suggested.

However, it might or might not be what's wanted for Poky since it increases
the test matrix. Therefore this patch series refactors the package
management code so that it's possible to add more package managers in other
own layer. I will send another patch serie that will add apk.

Perfomance metrics below, (note that this includes build times).

APK
===
bitbake core-image-minimal
15.84s user 2.60s system 0% cpu 1:26:19.21 total
16.01s user 2.58s system 0% cpu 1:26:03.72 total
15.69s user 2.61s system 0% cpu 1:26:45.45 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -f -c do_rootfs
0.55s user 0.06s system 4% cpu 14.236 total
0.54s user 0.08s system 4% cpu 15.247 total
0.52s user 0.15s system 4% cpu 15.143 total

RPM
===
bitbake core-image-minimal
18.57s user 3.09s system 0% cpu 1:31:29.09 total
18.58s user 3.08s system 0% cpu 1:30:53.80 total
18.20s user 3.31s system 0% cpu 1:31:06.69 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -c do_rootfs -f
0.58s user 0.10s system 3% cpu 19.470 total
0.57s user 0.09s system 3% cpu 19.480 total
0.60s user 0.07s system 3% cpu 20.381 total


[-- Attachment #2: Type: text/html, Size: 9576 bytes --]

Re: [OE-core] Add package managers as a plugin

[Ross Burton]

On Thu, 25 Jun 2020 at 11:14, Fredrik Gustafsson
<fredrik.gustafsson@axis.com> wrote:
> Poky today has three different package managers, the well-known formats deb
> and rpm is supported as well as ipkg that is good for embedded devices.
>
> When building and having a good cache hit, a significant amount of time is
> spent in the phase of generating a rootfs, which is really about the
> performance of the package manager. ipkg is way slower than deb or rpm. To
> save build time and also get a package manager that is suitanle for use on
> targets where flash memory is a concern, support for apk is suggested.
>
> However, it might or might not be what's wanted for Poky since it increases
> the test matrix. Therefore this patch series refactors the package
> management code so that it's possible to add more package managers in other
> own layer. I will send another patch serie that will add apk.

As to the code review I mostly second what Paul said.

However, obviously you actually have strong feelings on this as the
series represents a fair amount of work.  What are the compelling
reasons to use apk?  Five seconds on core-image-minimal doesn't really
sell it to me, I don't think anyone ever claimed that opkg has been
performance optimised, dpkg isn't really used that much: maybe we
should re-evaluate what package managers are supported if apk has
feature parity but is significantly faster or has other advantages.

Ross

Re: [OE-core] Add package managers as a plugin

[Tim Orling]

[-- Attachment #1: Type: text/plain, Size: 1848 bytes --]

On Fri, Jun 26, 2020 at 3:53 AM Ross Burton <ross@burtonini.com> wrote:

> On Thu, 25 Jun 2020 at 11:14, Fredrik Gustafsson
> <fredrik.gustafsson@axis.com> wrote:
> > Poky today has three different package managers, the well-known formats
> deb
> > and rpm is supported as well as ipkg that is good for embedded devices.
> >
> > When building and having a good cache hit, a significant amount of time
> is
> > spent in the phase of generating a rootfs, which is really about the
> > performance of the package manager. ipkg is way slower than deb or rpm.
> To
> > save build time and also get a package manager that is suitanle for use
> on
> > targets where flash memory is a concern, support for apk is suggested.
> >
> > However, it might or might not be what's wanted for Poky since it
> increases
> > the test matrix. Therefore this patch series refactors the package
> > management code so that it's possible to add more package managers in
> other
> > own layer. I will send another patch serie that will add apk.
>
> As to the code review I mostly second what Paul said.
>
> However, obviously you actually have strong feelings on this as the
> series represents a fair amount of work.  What are the compelling
> reasons to use apk?  Five seconds on core-image-minimal doesn't really
> sell it to me, I don't think anyone ever claimed that opkg has been
> performance optimised, dpkg isn't really used that much: maybe we
> should re-evaluate what package managers are supported if apk has
> feature parity but is significantly faster or has other advantages.
>

FWIW, I was contemplating apk packaging because of heavy usage in Docker
containers based on Alpine Linux.

I haven’t reviewed the patch series, but the theme of adding apk as another
viable package type is welcome.


> Ross
> 
>

[-- Attachment #2: Type: text/html, Size: 2490 bytes --]

Re: [OE-core] Add package managers as a plugin

[Khem Raj]

On 6/26/20 3:52 AM, Ross Burton wrote:
> On Thu, 25 Jun 2020 at 11:14, Fredrik Gustafsson
> <fredrik.gustafsson@axis.com> wrote:
>> Poky today has three different package managers, the well-known formats deb
>> and rpm is supported as well as ipkg that is good for embedded devices.
>>
>> When building and having a good cache hit, a significant amount of time is
>> spent in the phase of generating a rootfs, which is really about the
>> performance of the package manager. ipkg is way slower than deb or rpm. To
>> save build time and also get a package manager that is suitanle for use on
>> targets where flash memory is a concern, support for apk is suggested.
>>
>> However, it might or might not be what's wanted for Poky since it increases
>> the test matrix. Therefore this patch series refactors the package
>> management code so that it's possible to add more package managers in other
>> own layer. I will send another patch serie that will add apk.
> 
> As to the code review I mostly second what Paul said.
> 
> However, obviously you actually have strong feelings on this as the
> series represents a fair amount of work.  What are the compelling
> reasons to use apk?  Five seconds on core-image-minimal doesn't really
> sell it to me, I don't think anyone ever claimed that opkg has been
> performance optimised, dpkg isn't really used that much: maybe we
> should re-evaluate what package managers are supported if apk has
> feature parity but is significantly faster or has other advantages.

25% is a lot, real world images are way more complex than reference 
images in reference distribution, we have images which use 7 to 10 mins 
in rootfs, I have seen a desire by many developers in optimizing build 
time so this could be a real help, especially where we dont care about 
package management at runtime, we have done many micro optimizations and 
rejected patches since they added to build time because we want to be 
sensitive for increasing build time. rootfs is one task where sstate 
caching does not help as well, while other package builds we can benefit 
from sstate, rootfs not so much hence such an optimizations, if we test 
it regularly perhaps can be debated, but it certainly seems interesting 
option to have.

> 
> Ross
> 
> 
> 
> 

Re: [OE-core] Add package managers as a plugin

[Fredrik Gustafsson]

Hi Ross,
those 5 seconds will increase to minutes for my use case and we build a lot
hence I hope this will save us a lot of computer power and engineering time.
For example I've sent numbers on building a bigger image (core-image-sato-sdk-ptest)
to Alex and Alex:
out.apk.1: 1:13.35
out.apk.2: 1:13.51
out.apk.3: 1:13.23
out.apk.4: 1:14.07
out.apk.5: 1:13.00
out.deb.1: 3:49.37
out.deb.2: 3:50.77
out.deb.3: 3:51.39
out.deb.4: 3:53.40
out.deb.5: 3:53.99
out.ipk.1: 2:38.99
out.ipk.2: 2:39.07
out.ipk.3: 2:35.34
out.ipk.4: 2:36.15
out.ipk.5: 2:34.55
out.rpm.1: 1:58.61
out.rpm.2: 1:59.42
out.rpm.3: 1:59.70
out.rpm.4: 1:58.96
out.rpm.5: 1:58.11

Also consider that if we've a target without python with limited space, rpm is out
of the question. So the real comparison would be between ipk and apk. Let's
say we can save 80 seconds in each build. Now multiply that with the number of
builds, and we do a lot (I guess we might approach 100 000 builds/week in the
next year). Then this will save 92.5 days build time each week.

That's why I'm passionated about this.

/Fredrik
________________________________________
From: Ross Burton <ross@burtonini.com>
Sent: Friday, June 26, 2020 12:52 PM
To: Fredrik Gustafsson
Cc: OE-core; tools-cfpbuild-internal; Hugo Cedervall
Subject: Re: [OE-core] Add package managers as a plugin

On Thu, 25 Jun 2020 at 11:14, Fredrik Gustafsson
<fredrik.gustafsson@axis.com> wrote:
> Poky today has three different package managers, the well-known formats deb
> and rpm is supported as well as ipkg that is good for embedded devices.
>
> When building and having a good cache hit, a significant amount of time is
> spent in the phase of generating a rootfs, which is really about the
> performance of the package manager. ipkg is way slower than deb or rpm. To
> save build time and also get a package manager that is suitanle for use on
> targets where flash memory is a concern, support for apk is suggested.
>
> However, it might or might not be what's wanted for Poky since it increases
> the test matrix. Therefore this patch series refactors the package
> management code so that it's possible to add more package managers in other
> own layer. I will send another patch serie that will add apk.

As to the code review I mostly second what Paul said.

However, obviously you actually have strong feelings on this as the
series represents a fair amount of work.  What are the compelling
reasons to use apk?  Five seconds on core-image-minimal doesn't really
sell it to me, I don't think anyone ever claimed that opkg has been
performance optimised, dpkg isn't really used that much: maybe we
should re-evaluate what package managers are supported if apk has
feature parity but is significantly faster or has other advantages.

Ross

Re: [OE-core] Add package managers as a plugin

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1898 bytes --]

On Tue, Jun 30, 2020 at 07:01:23PM +0000, Fredrik Gustafsson wrote:
> Hi Ross,
> those 5 seconds will increase to minutes for my use case and we build a lot
> hence I hope this will save us a lot of computer power and engineering time.
> For example I've sent numbers on building a bigger image (core-image-sato-sdk-ptest)
> to Alex and Alex:
> out.apk.1: 1:13.35
> out.apk.2: 1:13.51
> out.apk.3: 1:13.23
> out.apk.4: 1:14.07
> out.apk.5: 1:13.00
> out.deb.1: 3:49.37
> out.deb.2: 3:50.77
> out.deb.3: 3:51.39
> out.deb.4: 3:53.40
> out.deb.5: 3:53.99
> out.ipk.1: 2:38.99
> out.ipk.2: 2:39.07
> out.ipk.3: 2:35.34
> out.ipk.4: 2:36.15
> out.ipk.5: 2:34.55
> out.rpm.1: 1:58.61
> out.rpm.2: 1:59.42
> out.rpm.3: 1:59.70
> out.rpm.4: 1:58.96
> out.rpm.5: 1:58.11
> 
> Also consider that if we've a target without python with limited space, rpm is out
> of the question. So the real comparison would be between ipk and apk. Let's
> say we can save 80 seconds in each build. Now multiply that with the number of
> builds, and we do a lot (I guess we might approach 100 000 builds/week in the
> next year). Then this will save 92.5 days build time each week.

This is impressive.

Have you tried to profile where opkg spends most of the time compared to
apk? It would be nice to know if there is something fundamentally
different in how they handle packages or if apk is just better
optimized.

Now with opkg better maintained (thanks Alejandro!) there might be some
relatively low hanging optimalizations which could be implemented there
as well (for people who didn't implement the rest to switch to apk yet
:)).

I know at LGE we have some patches (some related to performance as well) which
unfortunately weren't ever upstreamed :/, some are at:
https://github.com/webosose/meta-webosose/tree/master/meta-webos/recipes-devtools/opkg/opkg

Cheers,

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

Re: [OE-core] Add package managers as a plugin

[Fredrik Gustafsson]

Hi Martin,
apk is quite different compared to other package managers. A traditional package manager download a package, saves it to a local cache, verifies it and then extract it. That's at least 3 reads and 2 writes of the data. Apk does this on the fly, so 1 read and 1 write of the data. I believe that it's a so fundamental design choice that opkg would have it very hard to compete for speed. The speed advantage is even bigger when installing over network since the extraction and verification is done wile the network is waiting for data. That means that apk has usually installed the package when an ordinary package manager just has downloaded it and should start to verify and install it.

So there's no secret sauce or awesome optimizations, it's "just" an other architectural design.

With that said, there's still some low hanging fruit for apk, for example package generation (my patch is doing 2 reads and 2 writes of all data) and index generation (duplicate reads and one extraction too much) that could have significant speedup. But let's make it possible to use apk at all before starting to tune it.

More information:
https://www.youtube.com/watch?v=sIG2P9k6EjA


BR
Fredrik
________________________________________
From: Martin Jansa <martin.jansa@gmail.com>
Sent: Tuesday, June 30, 2020 11:54 PM
To: Fredrik Gustafsson
Cc: Ross Burton; OE-core; tools-cfpbuild-internal
Subject: Re: [OE-core] Add package managers as a plugin

On Tue, Jun 30, 2020 at 07:01:23PM +0000, Fredrik Gustafsson wrote:
> Hi Ross,
> those 5 seconds will increase to minutes for my use case and we build a lot
> hence I hope this will save us a lot of computer power and engineering time.
> For example I've sent numbers on building a bigger image (core-image-sato-sdk-ptest)
> to Alex and Alex:
> out.apk.1: 1:13.35
> out.apk.2: 1:13.51
> out.apk.3: 1:13.23
> out.apk.4: 1:14.07
> out.apk.5: 1:13.00
> out.deb.1: 3:49.37
> out.deb.2: 3:50.77
> out.deb.3: 3:51.39
> out.deb.4: 3:53.40
> out.deb.5: 3:53.99
> out.ipk.1: 2:38.99
> out.ipk.2: 2:39.07
> out.ipk.3: 2:35.34
> out.ipk.4: 2:36.15
> out.ipk.5: 2:34.55
> out.rpm.1: 1:58.61
> out.rpm.2: 1:59.42
> out.rpm.3: 1:59.70
> out.rpm.4: 1:58.96
> out.rpm.5: 1:58.11
>
> Also consider that if we've a target without python with limited space, rpm is out
> of the question. So the real comparison would be between ipk and apk. Let's
> say we can save 80 seconds in each build. Now multiply that with the number of
> builds, and we do a lot (I guess we might approach 100 000 builds/week in the
> next year). Then this will save 92.5 days build time each week.

This is impressive.

Have you tried to profile where opkg spends most of the time compared to
apk? It would be nice to know if there is something fundamentally
different in how they handle packages or if apk is just better
optimized.

Now with opkg better maintained (thanks Alejandro!) there might be some
relatively low hanging optimalizations which could be implemented there
as well (for people who didn't implement the rest to switch to apk yet
:)).

I know at LGE we have some patches (some related to performance as well) which
unfortunately weren't ever upstreamed :/, some are at:
https://github.com/webosose/meta-webosose/tree/master/meta-webos/recipes-devtools/opkg/opkg

Cheers,

Re: [OE-core] Add package managers as a plugin

[Alejandro del Castillo]

Hi all,

Catching up on the conversation...impressive results coming out of apk. 
Haven't used apk enough to have an educated opinion, but by quickly 
glancing at the code, I believe it uses an ad hoc solver, which might 
struggle with corner cases (same issues we had with opkg, before 
switching to libsolv). For building images without package managers, its 
likely that this isn't a serious concern.

Regarding opkg, I haven't focus on build time so I get the feeling there 
might be some low hanging fruit. I plan to spend some time investigating 
and will report back. The mechanism used by apk (1 read, 1 write) can be 
added to opkg too...in general the idea with opkg is that you set it up 
with the trade off that make sense for you. So I can envision having a 
config option that enables apk-like behavior.

Regardless, its exciting to see changes that expand the ecosystem!

Cheers,

Alejandro


On 6/30/20 5:12 PM, Fredrik Gustafsson wrote:
> Hi Martin, apk is quite different compared to other package managers.
> A traditional package manager download a package, saves it to a local
> cache, verifies it and then extract it. That's at least 3 reads and 2
> writes of the data. Apk does this on the fly, so 1 read and 1 write
> of the data. I believe that it's a so fundamental design choice that
> opkg would have it very hard to compete for speed. The speed
> advantage is even bigger when installing over network since the
> extraction and verification is done wile the network is waiting for
> data. That means that apk has usually installed the package when an
> ordinary package manager just has downloaded it and should start to
> verify and install it.
> 
> So there's no secret sauce or awesome optimizations, it's "just" an
> other architectural design.
> 
> With that said, there's still some low hanging fruit for apk, for
> example package generation (my patch is doing 2 reads and 2 writes of
> all data) and index generation (duplicate reads and one extraction
> too much) that could have significant speedup. But let's make it
> possible to use apk at all before starting to tune it.
> 
> More information: 
> https://urldefense.com/v3/__https://www.youtube.com/watch?v=sIG2P9k6EjA__;!!FbZ0ZwI3Qg!4qK6Dao4S2RQqjMvNiVBB04AmB8DTgW7Ccfr39CwxKWkSMA8DjszgWpvvj5RVMCfLL158w$
>
> 
> 
> BR Fredrik ________________________________________ From: Martin
> Jansa <martin.jansa@gmail.com> Sent: Tuesday, June 30, 2020 11:54 PM 
> To: Fredrik Gustafsson Cc: Ross Burton; OE-core;
> tools-cfpbuild-internal Subject: Re: [OE-core] Add package managers
> as a plugin
> 
> On Tue, Jun 30, 2020 at 07:01:23PM +0000, Fredrik Gustafsson wrote:
>> Hi Ross, those 5 seconds will increase to minutes for my use case
>> and we build a lot hence I hope this will save us a lot of computer
>> power and engineering time. For example I've sent numbers on
>> building a bigger image (core-image-sato-sdk-ptest) to Alex and
>> Alex: out.apk.1: 1:13.35 out.apk.2: 1:13.51 out.apk.3: 1:13.23 
>> out.apk.4: 1:14.07 out.apk.5: 1:13.00 out.deb.1: 3:49.37 out.deb.2:
>> 3:50.77 out.deb.3: 3:51.39 out.deb.4: 3:53.40 out.deb.5: 3:53.99 
>> out.ipk.1: 2:38.99 out.ipk.2: 2:39.07 out.ipk.3: 2:35.34 out.ipk.4:
>> 2:36.15 out.ipk.5: 2:34.55 out.rpm.1: 1:58.61 out.rpm.2: 1:59.42 
>> out.rpm.3: 1:59.70 out.rpm.4: 1:58.96 out.rpm.5: 1:58.11
>> 
>> Also consider that if we've a target without python with limited
>> space, rpm is out of the question. So the real comparison would be
>> between ipk and apk. Let's say we can save 80 seconds in each
>> build. Now multiply that with the number of builds, and we do a lot
>> (I guess we might approach 100 000 builds/week in the next year).
>> Then this will save 92.5 days build time each week.
> 
> This is impressive.
> 
> Have you tried to profile where opkg spends most of the time compared
> to apk? It would be nice to know if there is something fundamentally 
> different in how they handle packages or if apk is just better 
> optimized.
> 
> Now with opkg better maintained (thanks Alejandro!) there might be
> some relatively low hanging optimalizations which could be
> implemented there as well (for people who didn't implement the rest
> to switch to apk yet :)).
> 
> I know at LGE we have some patches (some related to performance as
> well) which unfortunately weren't ever upstreamed :/, some are at: 
> https://urldefense.com/v3/__https://github.com/webosose/meta-webosose/tree/master/meta-webos/recipes-devtools/opkg/opkg__;!!FbZ0ZwI3Qg!4qK6Dao4S2RQqjMvNiVBB04AmB8DTgW7Ccfr39CwxKWkSMA8DjszgWpvvj5RVMDh3PHxpA$
>
>  Cheers,
> 
> 
> 
> 

-- 
Cheers,

Alejandro

Re: [OE-core] Add package managers as a plugin

[Andre McCurdy]

On Tue, Jun 30, 2020 at 2:54 PM Martin Jansa <Martin.Jansa@gmail.com> wrote:
> On Tue, Jun 30, 2020 at 07:01:23PM +0000, Fredrik Gustafsson wrote:
> > Hi Ross,
> > those 5 seconds will increase to minutes for my use case and we build a lot
> > hence I hope this will save us a lot of computer power and engineering time.
> > For example I've sent numbers on building a bigger image (core-image-sato-sdk-ptest)
> > to Alex and Alex:
> > out.apk.1: 1:13.35
> > out.apk.2: 1:13.51
> > out.apk.3: 1:13.23
> > out.apk.4: 1:14.07
> > out.apk.5: 1:13.00
> > out.deb.1: 3:49.37
> > out.deb.2: 3:50.77
> > out.deb.3: 3:51.39
> > out.deb.4: 3:53.40
> > out.deb.5: 3:53.99
> > out.ipk.1: 2:38.99
> > out.ipk.2: 2:39.07
> > out.ipk.3: 2:35.34
> > out.ipk.4: 2:36.15
> > out.ipk.5: 2:34.55
> > out.rpm.1: 1:58.61
> > out.rpm.2: 1:59.42
> > out.rpm.3: 1:59.70
> > out.rpm.4: 1:58.96
> > out.rpm.5: 1:58.11
> >
> > Also consider that if we've a target without python with limited space, rpm is out
> > of the question. So the real comparison would be between ipk and apk. Let's
> > say we can save 80 seconds in each build. Now multiply that with the number of
> > builds, and we do a lot (I guess we might approach 100 000 builds/week in the
> > next year). Then this will save 92.5 days build time each week.
>
> This is impressive.

Indeed. 10 builds per minute, 24hrs a day. But not as impressive as
the scale of the development team able to create enough changes to
trigger so many builds and the scale of the test infrastructure able
to consume them!

Assume one developer makes 10 changes a day and each change gets built
for 10 different targets... 100,000 builds per week implies a team of
over 140 developers.

Re: [OE-core] Add package managers as a plugin

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2670 bytes --]

Yes, I thought we at @lge do a lot of builds, but after checking our
statistics we average "only" around 30,000 builds per week in last 5 years.

But on the other hand, our typical image is quite big (between 300MB and
1GB for enduser image and between 700MG and 1.5GB for developer images
depending on the target MACHINE for tar.bz2 IMAGE_FSTYPE), but each build
is producing at least 2 images (enduser and devel), but e.g. TVs are
building 15 different rootfs for each MACHINE, so the number of do_rootfs
tasks is easily over 100,000 per week and I'm seeing average do_rootfs
times around 10-14 minutes for bigger images and 4 minutes for "small"
images.

I should definitely try apk to prevent global warming :).

On Wed, Jul 1, 2020 at 1:15 AM Andre McCurdy <armccurdy@gmail.com> wrote:

> On Tue, Jun 30, 2020 at 2:54 PM Martin Jansa <Martin.Jansa@gmail.com>
> wrote:
> > On Tue, Jun 30, 2020 at 07:01:23PM +0000, Fredrik Gustafsson wrote:
> > > Hi Ross,
> > > those 5 seconds will increase to minutes for my use case and we build
> a lot
> > > hence I hope this will save us a lot of computer power and engineering
> time.
> > > For example I've sent numbers on building a bigger image
> (core-image-sato-sdk-ptest)
> > > to Alex and Alex:
> > > out.apk.1: 1:13.35
> > > out.apk.2: 1:13.51
> > > out.apk.3: 1:13.23
> > > out.apk.4: 1:14.07
> > > out.apk.5: 1:13.00
> > > out.deb.1: 3:49.37
> > > out.deb.2: 3:50.77
> > > out.deb.3: 3:51.39
> > > out.deb.4: 3:53.40
> > > out.deb.5: 3:53.99
> > > out.ipk.1: 2:38.99
> > > out.ipk.2: 2:39.07
> > > out.ipk.3: 2:35.34
> > > out.ipk.4: 2:36.15
> > > out.ipk.5: 2:34.55
> > > out.rpm.1: 1:58.61
> > > out.rpm.2: 1:59.42
> > > out.rpm.3: 1:59.70
> > > out.rpm.4: 1:58.96
> > > out.rpm.5: 1:58.11
> > >
> > > Also consider that if we've a target without python with limited
> space, rpm is out
> > > of the question. So the real comparison would be between ipk and apk.
> Let's
> > > say we can save 80 seconds in each build. Now multiply that with the
> number of
> > > builds, and we do a lot (I guess we might approach 100 000 builds/week
> in the
> > > next year). Then this will save 92.5 days build time each week.
> >
> > This is impressive.
>
> Indeed. 10 builds per minute, 24hrs a day. But not as impressive as
> the scale of the development team able to create enough changes to
> trigger so many builds and the scale of the test infrastructure able
> to consume them!
>
> Assume one developer makes 10 changes a day and each change gets built
> for 10 different targets... 100,000 builds per week implies a team of
> over 140 developers.
>

[-- Attachment #2: Type: text/html, Size: 3347 bytes --]

Add package managers as a plugin

[Fredrik Gustafsson]

OE-core today has three different package managers, the well-known formats deb
and rpm is supported as well as ipkg that is good for embedded devices.

When building and having a good cache hit, a significant amount of time is
spent in the phase of generating a rootfs, which is really about the
performance of the package manager. To save build time and also get a
package manager that is suitanle for use on targets where flash memory is a
concern, support for apk is suggested.

However, it might or might not be what's wanted for OE-core since it increases
the test matrix. Therefore I will refactor the package management code to
allow a layer to add a new package manager without editing the meta layer.

This refactor will be divided into multiple patch series to be easier to review,
this is the second patch series.

Re: [OE-core] Add package managers as a plugin

[Paul Barker]

On Thu, 2 Jul 2020 at 20:29, Fredrik Gustafsson
<fredrik.gustafsson@axis.com> wrote:
>
> OE-core today has three different package managers, the well-known formats deb
> and rpm is supported as well as ipkg that is good for embedded devices.
>
> When building and having a good cache hit, a significant amount of time is
> spent in the phase of generating a rootfs, which is really about the
> performance of the package manager. To save build time and also get a
> package manager that is suitanle for use on targets where flash memory is a
> concern, support for apk is suggested.
>
> However, it might or might not be what's wanted for OE-core since it increases
> the test matrix. Therefore I will refactor the package management code to
> allow a layer to add a new package manager without editing the meta layer.
>
> This refactor will be divided into multiple patch series to be easier to review,
> this is the second patch series.

Sorry to be a pain but I think this series suffers from the opposite
problem to the previous version - it's only a partial move of some of
the code. I'd much rather review a series which does the complete
refactor of the existing code into separate modules without making any
functional changes. I'll let others give their opinions as well, I'm
happy to review this one if we do want to look at this piecemeal.

-- 
Paul Barker
Konsulko Group

Re: [OE-core] Add package managers as a plugin

[Fredrik Gustafsson]

________________________________________
From: Paul Barker <pbarker@konsulko.com>
Sent: Thursday, July 2, 2020 10:28 PM
To: Fredrik Gustafsson
Cc: openembedded-core; tools-cfpbuild-internal
Subject: Re: [OE-core] Add package managers as a plugin

On Thu, 2 Jul 2020 at 20:29, Fredrik Gustafsson
<fredrik.gustafsson@axis.com> wrote:
>
> OE-core today has three different package managers, the well-known formats deb
> and rpm is supported as well as ipkg that is good for embedded devices.
>
> When building and having a good cache hit, a significant amount of time is
> spent in the phase of generating a rootfs, which is really about the
> performance of the package manager. To save build time and also get a
> package manager that is suitanle for use on targets where flash memory is a
> concern, support for apk is suggested.
>
> However, it might or might not be what's wanted for OE-core since it increases
> the test matrix. Therefore I will refactor the package management code to
> allow a layer to add a new package manager without editing the meta layer.
>
> This refactor will be divided into multiple patch series to be easier to review,
> this is the second patch series.

Sorry to be a pain but I think this series suffers from the opposite
problem to the previous version - it's only a partial move of some of
the code. I'd much rather review a series which does the complete
refactor of the existing code into separate modules without making any
functional changes. I'll let others give their opinions as well, I'm
happy to review this one if we do want to look at this piecemeal.

--
Paul Barker
Konsulko Group

No problems, as said before I will do my best to make this easier to review.
I've sent a new patch series.

/Fredrik

Add package managers as a plugin

[Fredrik Gustafsson]

Poky today has three different package managers, the well-known formats deb
and rpm is supported as well as ipkg that is good for embedded devices.

When building and having a good cache hit, a significant amount of time is
spent in the phase of generating a rootfs, which is really about the
performance of the package manager. ipkg is way slower than deb or rpm. To
save build time and also get a package manager that is suitanle for use on
targets where flash memory is a concern, support for apk is suggested.

However, it might or might not be what's wanted for Poky since it increases
the test matrix. Therefore this patch series refactors the package
management code so that it's possible to add more package managers in other
own layer. I will send another patch serie that will add apk.

Perfomance metrics below, (note that this includes build times).

APK
===
bitbake core-image-minimal
15.84s user 2.60s system 0% cpu 1:26:19.21 total
16.01s user 2.58s system 0% cpu 1:26:03.72 total
15.69s user 2.61s system 0% cpu 1:26:45.45 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -f -c do_rootfs
0.55s user 0.06s system 4% cpu 14.236 total
0.54s user 0.08s system 4% cpu 15.247 total
0.52s user 0.15s system 4% cpu 15.143 total

RPM
===
bitbake core-image-minimal
18.57s user 3.09s system 0% cpu 1:31:29.09 total
18.58s user 3.08s system 0% cpu 1:30:53.80 total
18.20s user 3.31s system 0% cpu 1:31:06.69 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -c do_rootfs -f
0.58s user 0.10s system 3% cpu 19.470 total
0.57s user 0.09s system 3% cpu 19.480 total
0.60s user 0.07s system 3% cpu 20.381 total

Re: [OE-core] Add package managers as a plugin

[Paul Barker]

On Thu, 25 Jun 2020 at 11:21, Fredrik Gustafsson
<fredrik.gustafsson@axis.com> wrote:
>
> Poky today has three different package managers, the well-known formats deb
> and rpm is supported as well as ipkg that is good for embedded devices.
>
> When building and having a good cache hit, a significant amount of time is
> spent in the phase of generating a rootfs, which is really about the
> performance of the package manager. ipkg is way slower than deb or rpm. To
> save build time and also get a package manager that is suitanle for use on
> targets where flash memory is a concern, support for apk is suggested.
>
> However, it might or might not be what's wanted for Poky since it increases
> the test matrix. Therefore this patch series refactors the package
> management code so that it's possible to add more package managers in other
> own layer. I will send another patch serie that will add apk.
>
> Perfomance metrics below, (note that this includes build times).
>
> APK
> ===
> bitbake core-image-minimal
> 15.84s user 2.60s system 0% cpu 1:26:19.21 total
> 16.01s user 2.58s system 0% cpu 1:26:03.72 total
> 15.69s user 2.61s system 0% cpu 1:26:45.45 total
>
> bitbake core-image-minimal -c cleansstate
> bitbake core-image-minimal -c clean
> bitbake core-image-minimal -f -c do_rootfs
> 0.55s user 0.06s system 4% cpu 14.236 total
> 0.54s user 0.08s system 4% cpu 15.247 total
> 0.52s user 0.15s system 4% cpu 15.143 total
>
> RPM
> ===
> bitbake core-image-minimal
> 18.57s user 3.09s system 0% cpu 1:31:29.09 total
> 18.58s user 3.08s system 0% cpu 1:30:53.80 total
> 18.20s user 3.31s system 0% cpu 1:31:06.69 total
>
> bitbake core-image-minimal -c cleansstate
> bitbake core-image-minimal -c clean
> bitbake core-image-minimal -c do_rootfs -f
> 0.58s user 0.10s system 3% cpu 19.470 total
> 0.57s user 0.09s system 3% cpu 19.480 total
> 0.60s user 0.07s system 3% cpu 20.381 total

Some initial review comments incoming on all these patches. I've just
done a surface level reading, I think the way you're submitting this
still needs some work before it's in a state where we can properly
review the changes in detail. Right now it's too big and too easy to
get confused when looking through 26 large patches.

I'm very supportive of tidying this up and making it more dynamic, we
just need to make sure we don't break things along the way.

Thanks,

-- 
Paul Barker
Konsulko Group

Re: [OE-core] Add package managers as a plugin

[Fredrik Gustafsson]

Hi Paul,
thank you for your effort on this. It's appreciated! I'll go through ýour advice
and see if I can improve this series.

BR
Fredrik
________________________________________
From: Paul Barker <pbarker@konsulko.com>
Sent: Thursday, June 25, 2020 12:40 PM
To: Fredrik Gustafsson
Cc: openembedded-core; tools-cfpbuild-internal; Hugo Cedervall
Subject: Re: [OE-core] Add package managers as a plugin

On Thu, 25 Jun 2020 at 11:21, Fredrik Gustafsson
<fredrik.gustafsson@axis.com> wrote:
>
> Poky today has three different package managers, the well-known formats deb
> and rpm is supported as well as ipkg that is good for embedded devices.
>
> When building and having a good cache hit, a significant amount of time is
> spent in the phase of generating a rootfs, which is really about the
> performance of the package manager. ipkg is way slower than deb or rpm. To
> save build time and also get a package manager that is suitanle for use on
> targets where flash memory is a concern, support for apk is suggested.
>
> However, it might or might not be what's wanted for Poky since it increases
> the test matrix. Therefore this patch series refactors the package
> management code so that it's possible to add more package managers in other
> own layer. I will send another patch serie that will add apk.
>
> Perfomance metrics below, (note that this includes build times).
>
> APK
> ===
> bitbake core-image-minimal
> 15.84s user 2.60s system 0% cpu 1:26:19.21 total
> 16.01s user 2.58s system 0% cpu 1:26:03.72 total
> 15.69s user 2.61s system 0% cpu 1:26:45.45 total
>
> bitbake core-image-minimal -c cleansstate
> bitbake core-image-minimal -c clean
> bitbake core-image-minimal -f -c do_rootfs
> 0.55s user 0.06s system 4% cpu 14.236 total
> 0.54s user 0.08s system 4% cpu 15.247 total
> 0.52s user 0.15s system 4% cpu 15.143 total
>
> RPM
> ===
> bitbake core-image-minimal
> 18.57s user 3.09s system 0% cpu 1:31:29.09 total
> 18.58s user 3.08s system 0% cpu 1:30:53.80 total
> 18.20s user 3.31s system 0% cpu 1:31:06.69 total
>
> bitbake core-image-minimal -c cleansstate
> bitbake core-image-minimal -c clean
> bitbake core-image-minimal -c do_rootfs -f
> 0.58s user 0.10s system 3% cpu 19.470 total
> 0.57s user 0.09s system 3% cpu 19.480 total
> 0.60s user 0.07s system 3% cpu 20.381 total

Some initial review comments incoming on all these patches. I've just
done a surface level reading, I think the way you're submitting this
still needs some work before it's in a state where we can properly
review the changes in detail. Right now it's too big and too easy to
get confused when looking through 26 large patches.

I'm very supportive of tidying this up and making it more dynamic, we
just need to make sure we don't break things along the way.

Thanks,

--
Paul Barker
Konsulko Group

Add package managers as a plugin

[Fredrik Gustafsson]

Poky today has three different package managers, the well-known formats deb
and rpm is supported as well as ipkg that is good for embedded devices.

When building and having a good cache hit, a significant amount of time is
spent in the phase of generating a rootfs, which is really about the
performance of the package manager. ipkg is way slower than deb or rpm. To
save build time and also get a package manager that is suitanle for use on
targets where flash memory is a concern, support for apk is suggested.

However, it might or might not be what's wanted for Poky since it increases
the test matrix. Therefore this patch series refactors the package
management code so that it's possible to add more package managers in other
own layer. I will send another patch serie that will add apk.

Perfomance metrics below, (note that this includes build times).

APK
===
bitbake core-image-minimal
15.84s user 2.60s system 0% cpu 1:26:19.21 total
16.01s user 2.58s system 0% cpu 1:26:03.72 total
15.69s user 2.61s system 0% cpu 1:26:45.45 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -f -c do_rootfs
0.55s user 0.06s system 4% cpu 14.236 total
0.54s user 0.08s system 4% cpu 15.247 total
0.52s user 0.15s system 4% cpu 15.143 total

RPM
===
bitbake core-image-minimal
18.57s user 3.09s system 0% cpu 1:31:29.09 total
18.58s user 3.08s system 0% cpu 1:30:53.80 total
18.20s user 3.31s system 0% cpu 1:31:06.69 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -c do_rootfs -f
0.58s user 0.10s system 3% cpu 19.470 total
0.57s user 0.09s system 3% cpu 19.480 total
0.60s user 0.07s system 3% cpu 20.381 total

Re: [OE-core] Add package managers as a plugin

[Denys Dmytriyenko]

You mentioned Poky few times below - why is it Poky-specific? Since you are 
not submitting this change to Poky, but to OE-Core instead, can this be used 
by other OE distros?

-- 
Denys


On Tue, Jun 23, 2020 at 01:13:26PM +0200, Fredrik Gustafsson wrote:
> Poky today has three different package managers, the well-known formats deb
> and rpm is supported as well as ipkg that is good for embedded devices.
> 
> When building and having a good cache hit, a significant amount of time is
> spent in the phase of generating a rootfs, which is really about the
> performance of the package manager. ipkg is way slower than deb or rpm. To
> save build time and also get a package manager that is suitanle for use on
> targets where flash memory is a concern, support for apk is suggested.
> 
> However, it might or might not be what's wanted for Poky since it increases
> the test matrix. Therefore this patch series refactors the package
> management code so that it's possible to add more package managers in other
> own layer. I will send another patch serie that will add apk.
> 
> Perfomance metrics below, (note that this includes build times).
> 
> APK
> ===
> bitbake core-image-minimal
> 15.84s user 2.60s system 0% cpu 1:26:19.21 total
> 16.01s user 2.58s system 0% cpu 1:26:03.72 total
> 15.69s user 2.61s system 0% cpu 1:26:45.45 total
> 
> bitbake core-image-minimal -c cleansstate
> bitbake core-image-minimal -c clean
> bitbake core-image-minimal -f -c do_rootfs
> 0.55s user 0.06s system 4% cpu 14.236 total
> 0.54s user 0.08s system 4% cpu 15.247 total
> 0.52s user 0.15s system 4% cpu 15.143 total
> 
> RPM
> ===
> bitbake core-image-minimal
> 18.57s user 3.09s system 0% cpu 1:31:29.09 total
> 18.58s user 3.08s system 0% cpu 1:30:53.80 total
> 18.20s user 3.31s system 0% cpu 1:31:06.69 total
> 
> bitbake core-image-minimal -c cleansstate
> bitbake core-image-minimal -c clean
> bitbake core-image-minimal -c do_rootfs -f
> 0.58s user 0.10s system 3% cpu 19.470 total
> 0.57s user 0.09s system 3% cpu 19.480 total
> 0.60s user 0.07s system 3% cpu 20.381 total

> 

Re: [OE-core] Add package managers as a plugin

[Fredrik Gustafsson]

Hi Denys,
you're right. I was confusing poky and OE-core. I'll change the cover letter. Thanks!

I haven't tested it with other distros but I see no reason that it wouldn't work.

BR
Fredrik
________________________________________
From: Denys Dmytriyenko <denis@denix.org>
Sent: Tuesday, June 23, 2020 8:50 PM
To: Fredrik Gustafsson
Cc: openembedded-core@lists.openembedded.org; tools-cfpbuild-internal; Hugo Cedervall
Subject: Re: [OE-core] Add package managers as a plugin

You mentioned Poky few times below - why is it Poky-specific? Since you are
not submitting this change to Poky, but to OE-Core instead, can this be used
by other OE distros?

--
Denys


On Tue, Jun 23, 2020 at 01:13:26PM +0200, Fredrik Gustafsson wrote:
> Poky today has three different package managers, the well-known formats deb
> and rpm is supported as well as ipkg that is good for embedded devices.
>
> When building and having a good cache hit, a significant amount of time is
> spent in the phase of generating a rootfs, which is really about the
> performance of the package manager. ipkg is way slower than deb or rpm. To
> save build time and also get a package manager that is suitanle for use on
> targets where flash memory is a concern, support for apk is suggested.
>
> However, it might or might not be what's wanted for Poky since it increases
> the test matrix. Therefore this patch series refactors the package
> management code so that it's possible to add more package managers in other
> own layer. I will send another patch serie that will add apk.
>
> Perfomance metrics below, (note that this includes build times).
>
> APK
> ===
> bitbake core-image-minimal
> 15.84s user 2.60s system 0% cpu 1:26:19.21 total
> 16.01s user 2.58s system 0% cpu 1:26:03.72 total
> 15.69s user 2.61s system 0% cpu 1:26:45.45 total
>
> bitbake core-image-minimal -c cleansstate
> bitbake core-image-minimal -c clean
> bitbake core-image-minimal -f -c do_rootfs
> 0.55s user 0.06s system 4% cpu 14.236 total
> 0.54s user 0.08s system 4% cpu 15.247 total
> 0.52s user 0.15s system 4% cpu 15.143 total
>
> RPM
> ===
> bitbake core-image-minimal
> 18.57s user 3.09s system 0% cpu 1:31:29.09 total
> 18.58s user 3.08s system 0% cpu 1:30:53.80 total
> 18.20s user 3.31s system 0% cpu 1:31:06.69 total
>
> bitbake core-image-minimal -c cleansstate
> bitbake core-image-minimal -c clean
> bitbake core-image-minimal -c do_rootfs -f
> 0.58s user 0.10s system 3% cpu 19.470 total
> 0.57s user 0.09s system 3% cpu 19.480 total
> 0.60s user 0.07s system 3% cpu 20.381 total

> 

Re: [OE-core] Add package managers as a plugin

[Alex Stewart]

On 6/23/20 6:13 AM, Fredrik Gustafsson wrote:
> When building and having a good cache hit, a significant amount of time is
> spent in the phase of generating a rootfs, which is really about the
> performance of the package manager. ipkg is way slower than deb or rpm.
I was interested enough in your comment here to collect a small sampling 
of comparative do_rootfs runs on one of my dev machines.

I used a virgin poky clone and tried to replicate the steps you provided 
in your OP, timing only the execution of `bitbake -c do_rootfs 
core-image-minimal`. I used the debian `time` package for timing. Here's 
what I got:

# Summary
PM     Avg do_rootfs time (s)
--     ----------------------
deb    33.89
ipk    33.35
rpm    31.85

# Raw Data
deb    00:33.54
deb    00:34.66
deb    00:33.55
deb    00:33.23
deb    00:34.46
ipk    00:33.96
ipk    00:34.43
ipk    00:33.99
ipk    00:32.17
ipk    00:32.22
rpm    00:29.92
rpm    00:32.31
rpm    00:31.90
rpm    00:33.31
rpm    00:31.80

Based on those results, it seems like all the current PMs are similar in 
terms of execution time. Are there major differences between my test 
setup and yours? And do you already have a layer with APK available that 
I could compare against on my hardware?

This is the crude script I used to automate the test runs:

```
#!/bin/bash
set -euxo pipefail

TIME=/usr/bin/time

recipe=core-image-minimal

# Options are:
#  - 'package_deb' for debian style deb files
#  - 'package_ipk' for ipk files are used by opkg (a debian style 
embedded package manager)
#  - 'package_rpm' for rpm style packages
function set_pkg_man() {
     pkg_man=${1}
     sed -i "s/\(^PACKAGE_CLASSES ?= \)\"\(.*\)\"/\1\"$pkg_man\"/" 
conf/local.conf
}

function rebuild_recipe() {
     bitbake -c cleanall $recipe
     bitbake $recipe
}

function test_recipe() {
     local name=${1}
     bitbake -c cleansstate $recipe
     bitbake -c clean $recipe
     $TIME bitbake -c do_rootfs -f $recipe 2>&1 | tee outs/out.$name
}

# rpm
set_pkg_man package_rpm
rebuild_recipe

test_recipe rpm.1
test_recipe rpm.2
test_recipe rpm.3
test_recipe rpm.4
test_recipe rpm.5


# ipk
set_pkg_man package_ipk
rebuild_recipe

test_recipe ipk.1
test_recipe ipk.2
test_recipe ipk.3
test_recipe ipk.4
test_recipe ipk.5

# deb
set_pkg_man package_deb
rebuild_recipe

test_recipe deb.1
test_recipe deb.2
test_recipe deb.3
test_recipe deb.4
test_recipe deb.5
```

Thanks,

-- 
Alex Stewart
Software Engineer - LabVIEW Real-Time OS
National Instruments

alex.stewart@ni.com
office: +1(512)683-8522

Re: [OE-core] Add package managers as a plugin

[Fredrik Gustafsson]

[-- Attachment #1: Type: text/plain, Size: 5072 bytes --]

Hi Alex,
Thanks for supplying a script!

I added apk to your script and then ran
```
for f in `ls`; do
printf "$f: "; cat $f | tail -2 | head -1 | awk -F ' ' '{print $3}' | sed -e 's/elapsed//'
done | sort
```

The result is here:
out.apk.1: 0:13.35
out.apk.2: 0:12.89
out.apk.3: 0:12.99
out.apk.4: 0:12.87
out.apk.5: 0:13.02
out.deb.1: 0:20.27
out.deb.2: 0:20.30
out.deb.3: 0:20.33
out.deb.4: 0:20.06
out.deb.5: 0:20.17
out.ipk.1: 0:18.51
out.ipk.2: 0:18.78
out.ipk.3: 0:18.80
out.ipk.4: 0:18.99
out.ipk.5: 0:18.97
out.rpm.1: 0:16.80
out.rpm.2: 0:16.83
out.rpm.3: 0:16.69
out.rpm.4: 0:16.78
out.rpm.5: 0:17.78

That is not supportive of my statement of ipk being the slowest package manager.
That statement come from timing tests done with our proprietary version of poky
that is somewhat extended when it comes to the do_rootfs task. We're generating
a bit more artifacts than poky. However apk is still ~22% faster than rpm and ~40%
faster than deb according to the data above.

So I tried your script with core-image-sato-sdk-ptest instead that is a bigger image.
The results where more mixed. There is a huge difference between different package
managers:
out.apk.1: 1:13.35
out.apk.2: 1:13.51
out.apk.3: 1:13.23
out.apk.4: 1:14.07
out.apk.5: 1:13.00
out.deb.1: 3:49.37
out.deb.2: 3:50.77
out.deb.3: 3:51.39
out.deb.4: 3:53.40
out.deb.5: 3:53.99
out.ipk.1: 2:38.99
out.ipk.2: 2:39.07
out.ipk.3: 2:35.34
out.ipk.4: 2:36.15
out.ipk.5: 2:34.55
out.rpm.1: 1:58.61
out.rpm.2: 1:59.42
out.rpm.3: 1:59.70
out.rpm.4: 1:58.96
out.rpm.5: 1:58.11

But yeah, I should probably rephrase my cover letter. You'll find a layer for APK that will
apply on top of my patch serie attached to this email.

Also note that there's a difference between the time to generate a package. Which 
we completely ignore in this measurements. It's not as important since it's being
cached, but for a fresh build it will be a difference between different package managers.

BR
Fredrik

___________________________
From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> on behalf of Alex Stewart <alex.stewart@ni.com>
Sent: Wednesday, June 24, 2020 5:18 PM
To: Fredrik Gustafsson; openembedded-core@lists.openembedded.org
Cc: tools-cfpbuild-internal; Hugo Cedervall; Alejandro Del Castillo
Subject: Re: [OE-core] Add package managers as a plugin

On 6/23/20 6:13 AM, Fredrik Gustafsson wrote:
> When building and having a good cache hit, a significant amount of time is
> spent in the phase of generating a rootfs, which is really about the
> performance of the package manager. ipkg is way slower than deb or rpm.
I was interested enough in your comment here to collect a small sampling
of comparative do_rootfs runs on one of my dev machines.

I used a virgin poky clone and tried to replicate the steps you provided
in your OP, timing only the execution of `bitbake -c do_rootfs
core-image-minimal`. I used the debian `time` package for timing. Here's
what I got:

# Summary
PM     Avg do_rootfs time (s)
--     ----------------------
deb    33.89
ipk    33.35
rpm    31.85

# Raw Data
deb    00:33.54
deb    00:34.66
deb    00:33.55
deb    00:33.23
deb    00:34.46
ipk    00:33.96
ipk    00:34.43
ipk    00:33.99
ipk    00:32.17
ipk    00:32.22
rpm    00:29.92
rpm    00:32.31
rpm    00:31.90
rpm    00:33.31
rpm    00:31.80

Based on those results, it seems like all the current PMs are similar in
terms of execution time. Are there major differences between my test
setup and yours? And do you already have a layer with APK available that
I could compare against on my hardware?

This is the crude script I used to automate the test runs:

```
#!/bin/bash
set -euxo pipefail

TIME=/usr/bin/time

recipe=core-image-minimal

# Options are:
#  - 'package_deb' for debian style deb files
#  - 'package_ipk' for ipk files are used by opkg (a debian style
embedded package manager)
#  - 'package_rpm' for rpm style packages
function set_pkg_man() {
     pkg_man=${1}
     sed -i "s/\(^PACKAGE_CLASSES ?= \)\"\(.*\)\"/\1\"$pkg_man\"/"
conf/local.conf
}

function rebuild_recipe() {
     bitbake -c cleanall $recipe
     bitbake $recipe
}

function test_recipe() {
     local name=${1}
     bitbake -c cleansstate $recipe
     bitbake -c clean $recipe
     $TIME bitbake -c do_rootfs -f $recipe 2>&1 | tee outs/out.$name
}

# rpm
set_pkg_man package_rpm
rebuild_recipe

test_recipe rpm.1
test_recipe rpm.2
test_recipe rpm.3
test_recipe rpm.4
test_recipe rpm.5


# ipk
set_pkg_man package_ipk
rebuild_recipe

test_recipe ipk.1
test_recipe ipk.2
test_recipe ipk.3
test_recipe ipk.4
test_recipe ipk.5

# deb
set_pkg_man package_deb
rebuild_recipe

test_recipe deb.1
test_recipe deb.2
test_recipe deb.3
test_recipe deb.4
test_recipe deb.5
```

Thanks,

--
Alex Stewart
Software Engineer - LabVIEW Real-Time OS
National Instruments

alex.stewart@ni.com
office: +1(512)683-8522


[-- Attachment #2: meta-apk.tar.gz --]
[-- Type: application/gzip, Size: 31444 bytes --]

Re: [OE-core] Add package managers as a plugin

[Alex Stewart]

Hey Fredrik,

On 6/30/20 1:38 PM, Fredrik Gustafsson wrote:
> I added apk to your script and then ran
> <snip>
>
> That is not supportive of my statement of ipk being the slowest package manager.
> That statement come from timing tests done with our proprietary version of poky
> that is somewhat extended when it comes to the do_rootfs task. We're generating
> a bit more artifacts than poky.
Ah; good to know. Thanks for checking again! Sounds like it could be 
related to extant opkg bugs which degrade performance when you have many 
(read: thousands) of packages in the feed.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=13593

> So I tried your script with core-image-sato-sdk-ptest instead that is a bigger image.
> The results where more mixed. There is a huge difference between different package
> managers:
> <snip>
>
> But yeah, I should probably rephrase my cover letter. You'll find a layer for APK that will
> apply on top of my patch serie attached to this email.
Thanks for sending that; I'm interested to take a look at it. Those new 
performance numbers look impressive.

Good luck with the patchset.

-- 
Alex Stewart
Software Engineer - LabVIEW Real-Time OS
National Instruments

alex.stewart@ni.com
office: +1(512)683-8522