Add package managers as a plugin

Add package managers as a plugin

[Fredrik Gustafsson]

Poky today has three different package managers, the well-known formats deb
and rpm is supported as well as ipkg that is good for embedded devices.

When building and having a good cache hit, a significant amount of time is
spent in the phase of generating a rootfs, which is really about the
performance of the package manager. ipkg is way slower than deb or rpm. To
save build time and also get a package manager that is suitanle for use on
targets where flash memory is a concern, support for apk is suggested.

However, it might or might not be what's wanted for Poky since it increases
the test matrix. Therefore this patch series refactors the package
management code so that it's possible to add more package managers in other
own layer. I will send another patch serie that will add apk.

Perfomance metrics below, (note that this includes build times).

APK
===
bitbake core-image-minimal
15.84s user 2.60s system 0% cpu 1:26:19.21 total
16.01s user 2.58s system 0% cpu 1:26:03.72 total
15.69s user 2.61s system 0% cpu 1:26:45.45 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -f -c do_rootfs
0.55s user 0.06s system 4% cpu 14.236 total
0.54s user 0.08s system 4% cpu 15.247 total
0.52s user 0.15s system 4% cpu 15.143 total

RPM
===
bitbake core-image-minimal
18.57s user 3.09s system 0% cpu 1:31:29.09 total
18.58s user 3.08s system 0% cpu 1:30:53.80 total
18.20s user 3.31s system 0% cpu 1:31:06.69 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -c do_rootfs -f
0.58s user 0.10s system 3% cpu 19.470 total
0.57s user 0.09s system 3% cpu 19.480 total
0.60s user 0.07s system 3% cpu 20.381 total

Re: [OE-core] Add package managers as a plugin

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 2148 bytes --]

Apologies, but these metrics do not look particularly convincing - 15s vs
20s? Can you try it with an image several gigabytes in size?
Core-image-sato-sdk-ptest is a good option.

Alex

On Thu, 25 Jun 2020 at 12:14, Fredrik Gustafsson <
fredrik.gustafsson@axis.com> wrote:

> Poky today has three different package managers, the well-known formats deb
> and rpm is supported as well as ipkg that is good for embedded devices.
>
> When building and having a good cache hit, a significant amount of time is
> spent in the phase of generating a rootfs, which is really about the
> performance of the package manager. ipkg is way slower than deb or rpm. To
> save build time and also get a package manager that is suitanle for use on
> targets where flash memory is a concern, support for apk is suggested.
>
> However, it might or might not be what's wanted for Poky since it increases
> the test matrix. Therefore this patch series refactors the package
> management code so that it's possible to add more package managers in other
> own layer. I will send another patch serie that will add apk.
>
> Perfomance metrics below, (note that this includes build times).
>
> APK
> ===
> bitbake core-image-minimal
> 15.84s user 2.60s system 0% cpu 1:26:19.21 total
> 16.01s user 2.58s system 0% cpu 1:26:03.72 total
> 15.69s user 2.61s system 0% cpu 1:26:45.45 total
>
> bitbake core-image-minimal -c cleansstate
> bitbake core-image-minimal -c clean
> bitbake core-image-minimal -f -c do_rootfs
> 0.55s user 0.06s system 4% cpu 14.236 total
> 0.54s user 0.08s system 4% cpu 15.247 total
> 0.52s user 0.15s system 4% cpu 15.143 total
>
> RPM
> ===
> bitbake core-image-minimal
> 18.57s user 3.09s system 0% cpu 1:31:29.09 total
> 18.58s user 3.08s system 0% cpu 1:30:53.80 total
> 18.20s user 3.31s system 0% cpu 1:31:06.69 total
>
> bitbake core-image-minimal -c cleansstate
> bitbake core-image-minimal -c clean
> bitbake core-image-minimal -c do_rootfs -f
> 0.58s user 0.10s system 3% cpu 19.470 total
> 0.57s user 0.09s system 3% cpu 19.480 total
> 0.60s user 0.07s system 3% cpu 20.381 total
> 
>

[-- Attachment #2: Type: text/html, Size: 2568 bytes --]

Re: [OE-core] Add package managers as a plugin

[Ross Burton]

On Thu, 25 Jun 2020 at 11:14, Fredrik Gustafsson
<fredrik.gustafsson@axis.com> wrote:
> Poky today has three different package managers, the well-known formats deb
> and rpm is supported as well as ipkg that is good for embedded devices.
>
> When building and having a good cache hit, a significant amount of time is
> spent in the phase of generating a rootfs, which is really about the
> performance of the package manager. ipkg is way slower than deb or rpm. To
> save build time and also get a package manager that is suitanle for use on
> targets where flash memory is a concern, support for apk is suggested.
>
> However, it might or might not be what's wanted for Poky since it increases
> the test matrix. Therefore this patch series refactors the package
> management code so that it's possible to add more package managers in other
> own layer. I will send another patch serie that will add apk.

As to the code review I mostly second what Paul said.

However, obviously you actually have strong feelings on this as the
series represents a fair amount of work.  What are the compelling
reasons to use apk?  Five seconds on core-image-minimal doesn't really
sell it to me, I don't think anyone ever claimed that opkg has been
performance optimised, dpkg isn't really used that much: maybe we
should re-evaluate what package managers are supported if apk has
feature parity but is significantly faster or has other advantages.

Ross

Re: [OE-core] Add package managers as a plugin

[Tim Orling]

[-- Attachment #1: Type: text/plain, Size: 1848 bytes --]

On Fri, Jun 26, 2020 at 3:53 AM Ross Burton <ross@burtonini.com> wrote:

> On Thu, 25 Jun 2020 at 11:14, Fredrik Gustafsson
> <fredrik.gustafsson@axis.com> wrote:
> > Poky today has three different package managers, the well-known formats
> deb
> > and rpm is supported as well as ipkg that is good for embedded devices.
> >
> > When building and having a good cache hit, a significant amount of time
> is
> > spent in the phase of generating a rootfs, which is really about the
> > performance of the package manager. ipkg is way slower than deb or rpm.
> To
> > save build time and also get a package manager that is suitanle for use
> on
> > targets where flash memory is a concern, support for apk is suggested.
> >
> > However, it might or might not be what's wanted for Poky since it
> increases
> > the test matrix. Therefore this patch series refactors the package
> > management code so that it's possible to add more package managers in
> other
> > own layer. I will send another patch serie that will add apk.
>
> As to the code review I mostly second what Paul said.
>
> However, obviously you actually have strong feelings on this as the
> series represents a fair amount of work.  What are the compelling
> reasons to use apk?  Five seconds on core-image-minimal doesn't really
> sell it to me, I don't think anyone ever claimed that opkg has been
> performance optimised, dpkg isn't really used that much: maybe we
> should re-evaluate what package managers are supported if apk has
> feature parity but is significantly faster or has other advantages.
>

FWIW, I was contemplating apk packaging because of heavy usage in Docker
containers based on Alpine Linux.

I haven’t reviewed the patch series, but the theme of adding apk as another
viable package type is welcome.


> Ross
> 
>

[-- Attachment #2: Type: text/html, Size: 2490 bytes --]

Re: [OE-core] Add package managers as a plugin

[Fredrik Gustafsson]

[-- Attachment #1: Type: text/plain, Size: 3535 bytes --]

Hi Alexander,

what are you finding strange with the numbers? Too fast or to big of a difference? Note that it's only the do_rootfs task that i measured.

I've tried the core-image-sato-sdk-ptest as you suggested. I run the following command multiple times:
bitbake core-image-sato-sdk-ptest -c cleansstate &&

bitbake core-image-sato-sdk-ptest -c clean &&
time bitbake core-image-sato-sdk-ptest -f -c do_rootfs

APK
====
0,85s user 0,16s system 0% cpu 1:13,66 total
0,87s user 0,14s system 0% cpu 1:13,28 total
0,89s user 0,11s system 0% cpu 1:13,10 total

RPM
===

0,67s user 0,12s system 0% cpu 2:02,48 total
0,68s user 0,11s system 0% cpu 2:03,39 total
0,73s user 0,09s system 0% cpu 1:59,87 total

5 seconds doesn't sound that much but it's 25% speedup. Since building is mostly cached but generating rootfs isn't this is a huge time consumption for our use case and our image takes around 4 minutes to build. Saving one minutes times the many builds we do is a huge win.

(Excuse my late reply, running core-image-sato-sdk-ptest uncovered a few bugs in the implementation of apk in OE. So thanks for having me testing this image)


/Fredrik


________________________________
From: Alexander Kanavin <alex.kanavin@gmail.com>
Sent: Thursday, June 25, 2020 12:24 PM
To: Fredrik Gustafsson
Cc: OE-core; tools-cfpbuild-internal; Hugo Cedervall
Subject: Re: [OE-core] Add package managers as a plugin

Apologies, but these metrics do not look particularly convincing - 15s vs 20s? Can you try it with an image several gigabytes in size? Core-image-sato-sdk-ptest is a good option.

Alex

On Thu, 25 Jun 2020 at 12:14, Fredrik Gustafsson <fredrik.gustafsson@axis.com<mailto:fredrik.gustafsson@axis.com>> wrote:
Poky today has three different package managers, the well-known formats deb
and rpm is supported as well as ipkg that is good for embedded devices.

When building and having a good cache hit, a significant amount of time is
spent in the phase of generating a rootfs, which is really about the
performance of the package manager. ipkg is way slower than deb or rpm. To
save build time and also get a package manager that is suitanle for use on
targets where flash memory is a concern, support for apk is suggested.

However, it might or might not be what's wanted for Poky since it increases
the test matrix. Therefore this patch series refactors the package
management code so that it's possible to add more package managers in other
own layer. I will send another patch serie that will add apk.

Perfomance metrics below, (note that this includes build times).

APK
===
bitbake core-image-minimal
15.84s user 2.60s system 0% cpu 1:26:19.21 total
16.01s user 2.58s system 0% cpu 1:26:03.72 total
15.69s user 2.61s system 0% cpu 1:26:45.45 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -f -c do_rootfs
0.55s user 0.06s system 4% cpu 14.236 total
0.54s user 0.08s system 4% cpu 15.247 total
0.52s user 0.15s system 4% cpu 15.143 total

RPM
===
bitbake core-image-minimal
18.57s user 3.09s system 0% cpu 1:31:29.09 total
18.58s user 3.08s system 0% cpu 1:30:53.80 total
18.20s user 3.31s system 0% cpu 1:31:06.69 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -c do_rootfs -f
0.58s user 0.10s system 3% cpu 19.470 total
0.57s user 0.09s system 3% cpu 19.480 total
0.60s user 0.07s system 3% cpu 20.381 total


[-- Attachment #2: Type: text/html, Size: 7617 bytes --]

Re: [OE-core] Add package managers as a plugin

[Khem Raj]

On 6/26/20 3:52 AM, Ross Burton wrote:
> On Thu, 25 Jun 2020 at 11:14, Fredrik Gustafsson
> <fredrik.gustafsson@axis.com> wrote:
>> Poky today has three different package managers, the well-known formats deb
>> and rpm is supported as well as ipkg that is good for embedded devices.
>>
>> When building and having a good cache hit, a significant amount of time is
>> spent in the phase of generating a rootfs, which is really about the
>> performance of the package manager. ipkg is way slower than deb or rpm. To
>> save build time and also get a package manager that is suitanle for use on
>> targets where flash memory is a concern, support for apk is suggested.
>>
>> However, it might or might not be what's wanted for Poky since it increases
>> the test matrix. Therefore this patch series refactors the package
>> management code so that it's possible to add more package managers in other
>> own layer. I will send another patch serie that will add apk.
> 
> As to the code review I mostly second what Paul said.
> 
> However, obviously you actually have strong feelings on this as the
> series represents a fair amount of work.  What are the compelling
> reasons to use apk?  Five seconds on core-image-minimal doesn't really
> sell it to me, I don't think anyone ever claimed that opkg has been
> performance optimised, dpkg isn't really used that much: maybe we
> should re-evaluate what package managers are supported if apk has
> feature parity but is significantly faster or has other advantages.

25% is a lot, real world images are way more complex than reference 
images in reference distribution, we have images which use 7 to 10 mins 
in rootfs, I have seen a desire by many developers in optimizing build 
time so this could be a real help, especially where we dont care about 
package management at runtime, we have done many micro optimizations and 
rejected patches since they added to build time because we want to be 
sensitive for increasing build time. rootfs is one task where sstate 
caching does not help as well, while other package builds we can benefit 
from sstate, rootfs not so much hence such an optimizations, if we test 
it regularly perhaps can be debated, but it certainly seems interesting 
option to have.

> 
> Ross
> 
> 
> 
> 

Re: [OE-core] Add package managers as a plugin

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 4098 bytes --]

What I meant is that 15s vs 20s on a small image could have been 5 min vs
5min5s on a bigger image, so I wanted to see that the difference does scale
up with image size.

Can you to provide sdk-ptest numbers for deb and ipk as well?

Alex

On Tue, 30 Jun 2020 at 17:04, Fredrik Gustafsson <
Fredrik.Gustafsson@axis.com> wrote:

> Hi Alexander,
>
> what are you finding strange with the numbers? Too fast or to big of a
> difference? Note that it's only the do_rootfs task that i measured.
>
> I've tried the core-image-sato-sdk-ptest as you suggested. I run the
> following command multiple times:
> bitbake core-image-sato-sdk-ptest -c cleansstate &&
>
> bitbake core-image-sato-sdk-ptest -c clean &&
> time bitbake core-image-sato-sdk-ptest -f -c do_rootfs
>
>
> APK
> ====
> 0,85s user 0,16s system 0% cpu 1:13,66 total
> 0,87s user 0,14s system 0% cpu 1:13,28 total
> 0,89s user 0,11s system 0% cpu 1:13,10 total
>
> RPM
> ===
>
> 0,67s user 0,12s system 0% cpu 2:02,48 total
> 0,68s user 0,11s system 0% cpu 2:03,39 total
> 0,73s user 0,09s system 0% cpu 1:59,87 total
>
> 5 seconds doesn't sound that much but it's 25% speedup. Since building is
> mostly cached but generating rootfs isn't this is a huge time consumption
> for our use case and our image takes around 4 minutes to build. Saving one
> minutes times the many builds we do is a huge win.
>
> (Excuse my late reply, running core-image-sato-sdk-ptest uncovered a few
> bugs in the implementation of apk in OE. So thanks for having me testing
> this image)
>
>
> /Fredrik
>
>
> ------------------------------
> *From:* Alexander Kanavin <alex.kanavin@gmail.com>
> *Sent:* Thursday, June 25, 2020 12:24 PM
> *To:* Fredrik Gustafsson
> *Cc:* OE-core; tools-cfpbuild-internal; Hugo Cedervall
> *Subject:* Re: [OE-core] Add package managers as a plugin
>
> Apologies, but these metrics do not look particularly convincing - 15s vs
> 20s? Can you try it with an image several gigabytes in size?
> Core-image-sato-sdk-ptest is a good option.
>
> Alex
>
> On Thu, 25 Jun 2020 at 12:14, Fredrik Gustafsson <
> fredrik.gustafsson@axis.com> wrote:
>
>> Poky today has three different package managers, the well-known formats
>> deb
>> and rpm is supported as well as ipkg that is good for embedded devices.
>>
>> When building and having a good cache hit, a significant amount of time is
>> spent in the phase of generating a rootfs, which is really about the
>> performance of the package manager. ipkg is way slower than deb or rpm. To
>> save build time and also get a package manager that is suitanle for use on
>> targets where flash memory is a concern, support for apk is suggested.
>>
>> However, it might or might not be what's wanted for Poky since it
>> increases
>> the test matrix. Therefore this patch series refactors the package
>> management code so that it's possible to add more package managers in
>> other
>> own layer. I will send another patch serie that will add apk.
>>
>> Perfomance metrics below, (note that this includes build times).
>>
>> APK
>> ===
>> bitbake core-image-minimal
>> 15.84s user 2.60s system 0% cpu 1:26:19.21 total
>> 16.01s user 2.58s system 0% cpu 1:26:03.72 total
>> 15.69s user 2.61s system 0% cpu 1:26:45.45 total
>>
>> bitbake core-image-minimal -c cleansstate
>> bitbake core-image-minimal -c clean
>> bitbake core-image-minimal -f -c do_rootfs
>> 0.55s user 0.06s system 4% cpu 14.236 total
>> 0.54s user 0.08s system 4% cpu 15.247 total
>> 0.52s user 0.15s system 4% cpu 15.143 total
>>
>> RPM
>> ===
>> bitbake core-image-minimal
>> 18.57s user 3.09s system 0% cpu 1:31:29.09 total
>> 18.58s user 3.08s system 0% cpu 1:30:53.80 total
>> 18.20s user 3.31s system 0% cpu 1:31:06.69 total
>>
>> bitbake core-image-minimal -c cleansstate
>> bitbake core-image-minimal -c clean
>> bitbake core-image-minimal -c do_rootfs -f
>> 0.58s user 0.10s system 3% cpu 19.470 total
>> 0.57s user 0.09s system 3% cpu 19.480 total
>> 0.60s user 0.07s system 3% cpu 20.381 total
>> 
>>
>

[-- Attachment #2: Type: text/html, Size: 7891 bytes --]

Re: [OE-core] Add package managers as a plugin

[Fredrik Gustafsson]

[-- Attachment #1: Type: text/plain, Size: 4741 bytes --]

Hi Alex,

now I see what you mean. I got a script from Alex Stewart, here's the results for core-image-sato-sdk-ptest:

out.apk.1: 1:13.35
out.apk.2: 1:13.51
out.apk.3: 1:13.23
out.apk.4: 1:14.07
out.apk.5: 1:13.00
out.deb.1: 3:49.37
out.deb.2: 3:50.77
out.deb.3: 3:51.39
out.deb.4: 3:53.40
out.deb.5: 3:53.99
out.ipk.1: 2:38.99
out.ipk.2: 2:39.07
out.ipk.3: 2:35.34
out.ipk.4: 2:36.15
out.ipk.5: 2:34.55
out.rpm.1: 1:58.61
out.rpm.2: 1:59.42
out.rpm.3: 1:59.70
out.rpm.4: 1:58.96
out.rpm.5: 1:58.11


BR

Fredrik

________________________________
From: Alexander Kanavin <alex.kanavin@gmail.com>
Sent: Tuesday, June 30, 2020 5:43 PM
To: Fredrik Gustafsson
Cc: OE-core; tools-cfpbuild-internal; Hugo Cedervall
Subject: Re: [OE-core] Add package managers as a plugin

What I meant is that 15s vs 20s on a small image could have been 5 min vs 5min5s on a bigger image, so I wanted to see that the difference does scale up with image size.

Can you to provide sdk-ptest numbers for deb and ipk as well?

Alex

On Tue, 30 Jun 2020 at 17:04, Fredrik Gustafsson <Fredrik.Gustafsson@axis.com<mailto:Fredrik.Gustafsson@axis.com>> wrote:

Hi Alexander,

what are you finding strange with the numbers? Too fast or to big of a difference? Note that it's only the do_rootfs task that i measured.

I've tried the core-image-sato-sdk-ptest as you suggested. I run the following command multiple times:
bitbake core-image-sato-sdk-ptest -c cleansstate &&

bitbake core-image-sato-sdk-ptest -c clean &&
time bitbake core-image-sato-sdk-ptest -f -c do_rootfs

APK
====
0,85s user 0,16s system 0% cpu 1:13,66 total
0,87s user 0,14s system 0% cpu 1:13,28 total
0,89s user 0,11s system 0% cpu 1:13,10 total

RPM
===

0,67s user 0,12s system 0% cpu 2:02,48 total
0,68s user 0,11s system 0% cpu 2:03,39 total
0,73s user 0,09s system 0% cpu 1:59,87 total

5 seconds doesn't sound that much but it's 25% speedup. Since building is mostly cached but generating rootfs isn't this is a huge time consumption for our use case and our image takes around 4 minutes to build. Saving one minutes times the many builds we do is a huge win.

(Excuse my late reply, running core-image-sato-sdk-ptest uncovered a few bugs in the implementation of apk in OE. So thanks for having me testing this image)


/Fredrik


________________________________
From: Alexander Kanavin <alex.kanavin@gmail.com<mailto:alex.kanavin@gmail.com>>
Sent: Thursday, June 25, 2020 12:24 PM
To: Fredrik Gustafsson
Cc: OE-core; tools-cfpbuild-internal; Hugo Cedervall
Subject: Re: [OE-core] Add package managers as a plugin

Apologies, but these metrics do not look particularly convincing - 15s vs 20s? Can you try it with an image several gigabytes in size? Core-image-sato-sdk-ptest is a good option.

Alex

On Thu, 25 Jun 2020 at 12:14, Fredrik Gustafsson <fredrik.gustafsson@axis.com<mailto:fredrik.gustafsson@axis.com>> wrote:
Poky today has three different package managers, the well-known formats deb
and rpm is supported as well as ipkg that is good for embedded devices.

When building and having a good cache hit, a significant amount of time is
spent in the phase of generating a rootfs, which is really about the
performance of the package manager. ipkg is way slower than deb or rpm. To
save build time and also get a package manager that is suitanle for use on
targets where flash memory is a concern, support for apk is suggested.

However, it might or might not be what's wanted for Poky since it increases
the test matrix. Therefore this patch series refactors the package
management code so that it's possible to add more package managers in other
own layer. I will send another patch serie that will add apk.

Perfomance metrics below, (note that this includes build times).

APK
===
bitbake core-image-minimal
15.84s user 2.60s system 0% cpu 1:26:19.21 total
16.01s user 2.58s system 0% cpu 1:26:03.72 total
15.69s user 2.61s system 0% cpu 1:26:45.45 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -f -c do_rootfs
0.55s user 0.06s system 4% cpu 14.236 total
0.54s user 0.08s system 4% cpu 15.247 total
0.52s user 0.15s system 4% cpu 15.143 total

RPM
===
bitbake core-image-minimal
18.57s user 3.09s system 0% cpu 1:31:29.09 total
18.58s user 3.08s system 0% cpu 1:30:53.80 total
18.20s user 3.31s system 0% cpu 1:31:06.69 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -c do_rootfs -f
0.58s user 0.10s system 3% cpu 19.470 total
0.57s user 0.09s system 3% cpu 19.480 total
0.60s user 0.07s system 3% cpu 20.381 total


[-- Attachment #2: Type: text/html, Size: 9576 bytes --]

Re: [OE-core] Add package managers as a plugin

[Fredrik Gustafsson]

Hi Ross,
those 5 seconds will increase to minutes for my use case and we build a lot
hence I hope this will save us a lot of computer power and engineering time.
For example I've sent numbers on building a bigger image (core-image-sato-sdk-ptest)
to Alex and Alex:
out.apk.1: 1:13.35
out.apk.2: 1:13.51
out.apk.3: 1:13.23
out.apk.4: 1:14.07
out.apk.5: 1:13.00
out.deb.1: 3:49.37
out.deb.2: 3:50.77
out.deb.3: 3:51.39
out.deb.4: 3:53.40
out.deb.5: 3:53.99
out.ipk.1: 2:38.99
out.ipk.2: 2:39.07
out.ipk.3: 2:35.34
out.ipk.4: 2:36.15
out.ipk.5: 2:34.55
out.rpm.1: 1:58.61
out.rpm.2: 1:59.42
out.rpm.3: 1:59.70
out.rpm.4: 1:58.96
out.rpm.5: 1:58.11

Also consider that if we've a target without python with limited space, rpm is out
of the question. So the real comparison would be between ipk and apk. Let's
say we can save 80 seconds in each build. Now multiply that with the number of
builds, and we do a lot (I guess we might approach 100 000 builds/week in the
next year). Then this will save 92.5 days build time each week.

That's why I'm passionated about this.

/Fredrik
________________________________________
From: Ross Burton <ross@burtonini.com>
Sent: Friday, June 26, 2020 12:52 PM
To: Fredrik Gustafsson
Cc: OE-core; tools-cfpbuild-internal; Hugo Cedervall
Subject: Re: [OE-core] Add package managers as a plugin

On Thu, 25 Jun 2020 at 11:14, Fredrik Gustafsson
<fredrik.gustafsson@axis.com> wrote:
> Poky today has three different package managers, the well-known formats deb
> and rpm is supported as well as ipkg that is good for embedded devices.
>
> When building and having a good cache hit, a significant amount of time is
> spent in the phase of generating a rootfs, which is really about the
> performance of the package manager. ipkg is way slower than deb or rpm. To
> save build time and also get a package manager that is suitanle for use on
> targets where flash memory is a concern, support for apk is suggested.
>
> However, it might or might not be what's wanted for Poky since it increases
> the test matrix. Therefore this patch series refactors the package
> management code so that it's possible to add more package managers in other
> own layer. I will send another patch serie that will add apk.

As to the code review I mostly second what Paul said.

However, obviously you actually have strong feelings on this as the
series represents a fair amount of work.  What are the compelling
reasons to use apk?  Five seconds on core-image-minimal doesn't really
sell it to me, I don't think anyone ever claimed that opkg has been
performance optimised, dpkg isn't really used that much: maybe we
should re-evaluate what package managers are supported if apk has
feature parity but is significantly faster or has other advantages.

Ross

Re: [OE-core] Add package managers as a plugin

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1898 bytes --]

On Tue, Jun 30, 2020 at 07:01:23PM +0000, Fredrik Gustafsson wrote:
> Hi Ross,
> those 5 seconds will increase to minutes for my use case and we build a lot
> hence I hope this will save us a lot of computer power and engineering time.
> For example I've sent numbers on building a bigger image (core-image-sato-sdk-ptest)
> to Alex and Alex:
> out.apk.1: 1:13.35
> out.apk.2: 1:13.51
> out.apk.3: 1:13.23
> out.apk.4: 1:14.07
> out.apk.5: 1:13.00
> out.deb.1: 3:49.37
> out.deb.2: 3:50.77
> out.deb.3: 3:51.39
> out.deb.4: 3:53.40
> out.deb.5: 3:53.99
> out.ipk.1: 2:38.99
> out.ipk.2: 2:39.07
> out.ipk.3: 2:35.34
> out.ipk.4: 2:36.15
> out.ipk.5: 2:34.55
> out.rpm.1: 1:58.61
> out.rpm.2: 1:59.42
> out.rpm.3: 1:59.70
> out.rpm.4: 1:58.96
> out.rpm.5: 1:58.11
> 
> Also consider that if we've a target without python with limited space, rpm is out
> of the question. So the real comparison would be between ipk and apk. Let's
> say we can save 80 seconds in each build. Now multiply that with the number of
> builds, and we do a lot (I guess we might approach 100 000 builds/week in the
> next year). Then this will save 92.5 days build time each week.

This is impressive.

Have you tried to profile where opkg spends most of the time compared to
apk? It would be nice to know if there is something fundamentally
different in how they handle packages or if apk is just better
optimized.

Now with opkg better maintained (thanks Alejandro!) there might be some
relatively low hanging optimalizations which could be implemented there
as well (for people who didn't implement the rest to switch to apk yet
:)).

I know at LGE we have some patches (some related to performance as well) which
unfortunately weren't ever upstreamed :/, some are at:
https://github.com/webosose/meta-webosose/tree/master/meta-webos/recipes-devtools/opkg/opkg

Cheers,

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

Re: [OE-core] Add package managers as a plugin

[Fredrik Gustafsson]

Hi Martin,
apk is quite different compared to other package managers. A traditional package manager download a package, saves it to a local cache, verifies it and then extract it. That's at least 3 reads and 2 writes of the data. Apk does this on the fly, so 1 read and 1 write of the data. I believe that it's a so fundamental design choice that opkg would have it very hard to compete for speed. The speed advantage is even bigger when installing over network since the extraction and verification is done wile the network is waiting for data. That means that apk has usually installed the package when an ordinary package manager just has downloaded it and should start to verify and install it.

So there's no secret sauce or awesome optimizations, it's "just" an other architectural design.

With that said, there's still some low hanging fruit for apk, for example package generation (my patch is doing 2 reads and 2 writes of all data) and index generation (duplicate reads and one extraction too much) that could have significant speedup. But let's make it possible to use apk at all before starting to tune it.

More information:
https://www.youtube.com/watch?v=sIG2P9k6EjA


BR
Fredrik
________________________________________
From: Martin Jansa <martin.jansa@gmail.com>
Sent: Tuesday, June 30, 2020 11:54 PM
To: Fredrik Gustafsson
Cc: Ross Burton; OE-core; tools-cfpbuild-internal
Subject: Re: [OE-core] Add package managers as a plugin

On Tue, Jun 30, 2020 at 07:01:23PM +0000, Fredrik Gustafsson wrote:
> Hi Ross,
> those 5 seconds will increase to minutes for my use case and we build a lot
> hence I hope this will save us a lot of computer power and engineering time.
> For example I've sent numbers on building a bigger image (core-image-sato-sdk-ptest)
> to Alex and Alex:
> out.apk.1: 1:13.35
> out.apk.2: 1:13.51
> out.apk.3: 1:13.23
> out.apk.4: 1:14.07
> out.apk.5: 1:13.00
> out.deb.1: 3:49.37
> out.deb.2: 3:50.77
> out.deb.3: 3:51.39
> out.deb.4: 3:53.40
> out.deb.5: 3:53.99
> out.ipk.1: 2:38.99
> out.ipk.2: 2:39.07
> out.ipk.3: 2:35.34
> out.ipk.4: 2:36.15
> out.ipk.5: 2:34.55
> out.rpm.1: 1:58.61
> out.rpm.2: 1:59.42
> out.rpm.3: 1:59.70
> out.rpm.4: 1:58.96
> out.rpm.5: 1:58.11
>
> Also consider that if we've a target without python with limited space, rpm is out
> of the question. So the real comparison would be between ipk and apk. Let's
> say we can save 80 seconds in each build. Now multiply that with the number of
> builds, and we do a lot (I guess we might approach 100 000 builds/week in the
> next year). Then this will save 92.5 days build time each week.

This is impressive.

Have you tried to profile where opkg spends most of the time compared to
apk? It would be nice to know if there is something fundamentally
different in how they handle packages or if apk is just better
optimized.

Now with opkg better maintained (thanks Alejandro!) there might be some
relatively low hanging optimalizations which could be implemented there
as well (for people who didn't implement the rest to switch to apk yet
:)).

I know at LGE we have some patches (some related to performance as well) which
unfortunately weren't ever upstreamed :/, some are at:
https://github.com/webosose/meta-webosose/tree/master/meta-webos/recipes-devtools/opkg/opkg

Cheers,

Re: [OE-core] Add package managers as a plugin

[Andre McCurdy]

On Tue, Jun 30, 2020 at 2:54 PM Martin Jansa <Martin.Jansa@gmail.com> wrote:
> On Tue, Jun 30, 2020 at 07:01:23PM +0000, Fredrik Gustafsson wrote:
> > Hi Ross,
> > those 5 seconds will increase to minutes for my use case and we build a lot
> > hence I hope this will save us a lot of computer power and engineering time.
> > For example I've sent numbers on building a bigger image (core-image-sato-sdk-ptest)
> > to Alex and Alex:
> > out.apk.1: 1:13.35
> > out.apk.2: 1:13.51
> > out.apk.3: 1:13.23
> > out.apk.4: 1:14.07
> > out.apk.5: 1:13.00
> > out.deb.1: 3:49.37
> > out.deb.2: 3:50.77
> > out.deb.3: 3:51.39
> > out.deb.4: 3:53.40
> > out.deb.5: 3:53.99
> > out.ipk.1: 2:38.99
> > out.ipk.2: 2:39.07
> > out.ipk.3: 2:35.34
> > out.ipk.4: 2:36.15
> > out.ipk.5: 2:34.55
> > out.rpm.1: 1:58.61
> > out.rpm.2: 1:59.42
> > out.rpm.3: 1:59.70
> > out.rpm.4: 1:58.96
> > out.rpm.5: 1:58.11
> >
> > Also consider that if we've a target without python with limited space, rpm is out
> > of the question. So the real comparison would be between ipk and apk. Let's
> > say we can save 80 seconds in each build. Now multiply that with the number of
> > builds, and we do a lot (I guess we might approach 100 000 builds/week in the
> > next year). Then this will save 92.5 days build time each week.
>
> This is impressive.

Indeed. 10 builds per minute, 24hrs a day. But not as impressive as
the scale of the development team able to create enough changes to
trigger so many builds and the scale of the test infrastructure able
to consume them!

Assume one developer makes 10 changes a day and each change gets built
for 10 different targets... 100,000 builds per week implies a team of
over 140 developers.

Re: [OE-core] Add package managers as a plugin

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2670 bytes --]

Yes, I thought we at @lge do a lot of builds, but after checking our
statistics we average "only" around 30,000 builds per week in last 5 years.

But on the other hand, our typical image is quite big (between 300MB and
1GB for enduser image and between 700MG and 1.5GB for developer images
depending on the target MACHINE for tar.bz2 IMAGE_FSTYPE), but each build
is producing at least 2 images (enduser and devel), but e.g. TVs are
building 15 different rootfs for each MACHINE, so the number of do_rootfs
tasks is easily over 100,000 per week and I'm seeing average do_rootfs
times around 10-14 minutes for bigger images and 4 minutes for "small"
images.

I should definitely try apk to prevent global warming :).

On Wed, Jul 1, 2020 at 1:15 AM Andre McCurdy <armccurdy@gmail.com> wrote:

> On Tue, Jun 30, 2020 at 2:54 PM Martin Jansa <Martin.Jansa@gmail.com>
> wrote:
> > On Tue, Jun 30, 2020 at 07:01:23PM +0000, Fredrik Gustafsson wrote:
> > > Hi Ross,
> > > those 5 seconds will increase to minutes for my use case and we build
> a lot
> > > hence I hope this will save us a lot of computer power and engineering
> time.
> > > For example I've sent numbers on building a bigger image
> (core-image-sato-sdk-ptest)
> > > to Alex and Alex:
> > > out.apk.1: 1:13.35
> > > out.apk.2: 1:13.51
> > > out.apk.3: 1:13.23
> > > out.apk.4: 1:14.07
> > > out.apk.5: 1:13.00
> > > out.deb.1: 3:49.37
> > > out.deb.2: 3:50.77
> > > out.deb.3: 3:51.39
> > > out.deb.4: 3:53.40
> > > out.deb.5: 3:53.99
> > > out.ipk.1: 2:38.99
> > > out.ipk.2: 2:39.07
> > > out.ipk.3: 2:35.34
> > > out.ipk.4: 2:36.15
> > > out.ipk.5: 2:34.55
> > > out.rpm.1: 1:58.61
> > > out.rpm.2: 1:59.42
> > > out.rpm.3: 1:59.70
> > > out.rpm.4: 1:58.96
> > > out.rpm.5: 1:58.11
> > >
> > > Also consider that if we've a target without python with limited
> space, rpm is out
> > > of the question. So the real comparison would be between ipk and apk.
> Let's
> > > say we can save 80 seconds in each build. Now multiply that with the
> number of
> > > builds, and we do a lot (I guess we might approach 100 000 builds/week
> in the
> > > next year). Then this will save 92.5 days build time each week.
> >
> > This is impressive.
>
> Indeed. 10 builds per minute, 24hrs a day. But not as impressive as
> the scale of the development team able to create enough changes to
> trigger so many builds and the scale of the test infrastructure able
> to consume them!
>
> Assume one developer makes 10 changes a day and each change gets built
> for 10 different targets... 100,000 builds per week implies a team of
> over 140 developers.
>

[-- Attachment #2: Type: text/html, Size: 3347 bytes --]

Re: [OE-core] Add package managers as a plugin

[Alejandro del Castillo]

Hi all,

Catching up on the conversation...impressive results coming out of apk. 
Haven't used apk enough to have an educated opinion, but by quickly 
glancing at the code, I believe it uses an ad hoc solver, which might 
struggle with corner cases (same issues we had with opkg, before 
switching to libsolv). For building images without package managers, its 
likely that this isn't a serious concern.

Regarding opkg, I haven't focus on build time so I get the feeling there 
might be some low hanging fruit. I plan to spend some time investigating 
and will report back. The mechanism used by apk (1 read, 1 write) can be 
added to opkg too...in general the idea with opkg is that you set it up 
with the trade off that make sense for you. So I can envision having a 
config option that enables apk-like behavior.

Regardless, its exciting to see changes that expand the ecosystem!

Cheers,

Alejandro


On 6/30/20 5:12 PM, Fredrik Gustafsson wrote:
> Hi Martin, apk is quite different compared to other package managers.
> A traditional package manager download a package, saves it to a local
> cache, verifies it and then extract it. That's at least 3 reads and 2
> writes of the data. Apk does this on the fly, so 1 read and 1 write
> of the data. I believe that it's a so fundamental design choice that
> opkg would have it very hard to compete for speed. The speed
> advantage is even bigger when installing over network since the
> extraction and verification is done wile the network is waiting for
> data. That means that apk has usually installed the package when an
> ordinary package manager just has downloaded it and should start to
> verify and install it.
> 
> So there's no secret sauce or awesome optimizations, it's "just" an
> other architectural design.
> 
> With that said, there's still some low hanging fruit for apk, for
> example package generation (my patch is doing 2 reads and 2 writes of
> all data) and index generation (duplicate reads and one extraction
> too much) that could have significant speedup. But let's make it
> possible to use apk at all before starting to tune it.
> 
> More information: 
> https://urldefense.com/v3/__https://www.youtube.com/watch?v=sIG2P9k6EjA__;!!FbZ0ZwI3Qg!4qK6Dao4S2RQqjMvNiVBB04AmB8DTgW7Ccfr39CwxKWkSMA8DjszgWpvvj5RVMCfLL158w$
>
> 
> 
> BR Fredrik ________________________________________ From: Martin
> Jansa <martin.jansa@gmail.com> Sent: Tuesday, June 30, 2020 11:54 PM 
> To: Fredrik Gustafsson Cc: Ross Burton; OE-core;
> tools-cfpbuild-internal Subject: Re: [OE-core] Add package managers
> as a plugin
> 
> On Tue, Jun 30, 2020 at 07:01:23PM +0000, Fredrik Gustafsson wrote:
>> Hi Ross, those 5 seconds will increase to minutes for my use case
>> and we build a lot hence I hope this will save us a lot of computer
>> power and engineering time. For example I've sent numbers on
>> building a bigger image (core-image-sato-sdk-ptest) to Alex and
>> Alex: out.apk.1: 1:13.35 out.apk.2: 1:13.51 out.apk.3: 1:13.23 
>> out.apk.4: 1:14.07 out.apk.5: 1:13.00 out.deb.1: 3:49.37 out.deb.2:
>> 3:50.77 out.deb.3: 3:51.39 out.deb.4: 3:53.40 out.deb.5: 3:53.99 
>> out.ipk.1: 2:38.99 out.ipk.2: 2:39.07 out.ipk.3: 2:35.34 out.ipk.4:
>> 2:36.15 out.ipk.5: 2:34.55 out.rpm.1: 1:58.61 out.rpm.2: 1:59.42 
>> out.rpm.3: 1:59.70 out.rpm.4: 1:58.96 out.rpm.5: 1:58.11
>> 
>> Also consider that if we've a target without python with limited
>> space, rpm is out of the question. So the real comparison would be
>> between ipk and apk. Let's say we can save 80 seconds in each
>> build. Now multiply that with the number of builds, and we do a lot
>> (I guess we might approach 100 000 builds/week in the next year).
>> Then this will save 92.5 days build time each week.
> 
> This is impressive.
> 
> Have you tried to profile where opkg spends most of the time compared
> to apk? It would be nice to know if there is something fundamentally 
> different in how they handle packages or if apk is just better 
> optimized.
> 
> Now with opkg better maintained (thanks Alejandro!) there might be
> some relatively low hanging optimalizations which could be
> implemented there as well (for people who didn't implement the rest
> to switch to apk yet :)).
> 
> I know at LGE we have some patches (some related to performance as
> well) which unfortunately weren't ever upstreamed :/, some are at: 
> https://urldefense.com/v3/__https://github.com/webosose/meta-webosose/tree/master/meta-webos/recipes-devtools/opkg/opkg__;!!FbZ0ZwI3Qg!4qK6Dao4S2RQqjMvNiVBB04AmB8DTgW7Ccfr39CwxKWkSMA8DjszgWpvvj5RVMDh3PHxpA$
>
>  Cheers,
> 
> 
> 
> 

-- 
Cheers,

Alejandro

Add package managers as a plugin

[Fredrik Gustafsson]

OE-core today has three different package managers, the well-known formats deb
and rpm is supported as well as ipkg that is good for embedded devices.

When building and having a good cache hit, a significant amount of time is
spent in the phase of generating a rootfs, which is really about the
performance of the package manager. To save build time and also get a
package manager that is suitanle for use on targets where flash memory is a
concern, support for apk is suggested.

However, it might or might not be what's wanted for OE-core since it increases
the test matrix. Therefore I will refactor the package management code to
allow a layer to add a new package manager without editing the meta layer.

This refactor will be divided into multiple patch series to be easier to review,
this is the second patch series.

Add package managers as a plugin

[Fredrik Gustafsson]

OE-core today has three different package managers, the well-known formats deb
and rpm is supported as well as ipkg that is good for embedded devices.

When building and having a good cache hit, a significant amount of time is
spent in the phase of generating a rootfs, which is really about the
performance of the package manager. To save build time and also get a
package manager that is suitanle for use on targets where flash memory is a
concern, support for apk is suggested.

However, it might or might not be what's wanted for OE-core since it increases
the test matrix. Therefore I will refactor the package management code to
allow a layer to add a new package manager without editing the meta layer.

This refactor will be divided into multiple patch series to be easier to review,
this first one with a single patch.

Add package managers as a plugin

[Fredrik Gustafsson]

Poky today has three different package managers, the well-known formats deb
and rpm is supported as well as ipkg that is good for embedded devices.

When building and having a good cache hit, a significant amount of time is
spent in the phase of generating a rootfs, which is really about the
performance of the package manager. ipkg is way slower than deb or rpm. To
save build time and also get a package manager that is suitanle for use on
targets where flash memory is a concern, support for apk is suggested.

However, it might or might not be what's wanted for Poky since it increases
the test matrix. Therefore this patch series refactors the package
management code so that it's possible to add more package managers in other
own layer. I will send another patch serie that will add apk.

Perfomance metrics below, (note that this includes build times).

APK
===
bitbake core-image-minimal
15.84s user 2.60s system 0% cpu 1:26:19.21 total
16.01s user 2.58s system 0% cpu 1:26:03.72 total
15.69s user 2.61s system 0% cpu 1:26:45.45 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -f -c do_rootfs
0.55s user 0.06s system 4% cpu 14.236 total
0.54s user 0.08s system 4% cpu 15.247 total
0.52s user 0.15s system 4% cpu 15.143 total

RPM
===
bitbake core-image-minimal
18.57s user 3.09s system 0% cpu 1:31:29.09 total
18.58s user 3.08s system 0% cpu 1:30:53.80 total
18.20s user 3.31s system 0% cpu 1:31:06.69 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -c do_rootfs -f
0.58s user 0.10s system 3% cpu 19.470 total
0.57s user 0.09s system 3% cpu 19.480 total
0.60s user 0.07s system 3% cpu 20.381 total

Add package managers as a plugin

[Fredrik Gustafsson]

Poky today has three different package managers, the well-known formats deb
and rpm is supported as well as ipkg that is good for embedded devices.

When building and having a good cache hit, a significant amount of time is
spent in the phase of generating a rootfs, which is really about the
performance of the package manager. ipkg is way slower than deb or rpm. To
save build time and also get a package manager that is suitanle for use on
targets where flash memory is a concern, support for apk is suggested.

However, it might or might not be what's wanted for Poky since it increases
the test matrix. Therefore this patch series refactors the package
management code so that it's possible to add more package managers in other
own layer. I will send another patch serie that will add apk.

Perfomance metrics below, (note that this includes build times).

APK
===
bitbake core-image-minimal
15.84s user 2.60s system 0% cpu 1:26:19.21 total
16.01s user 2.58s system 0% cpu 1:26:03.72 total
15.69s user 2.61s system 0% cpu 1:26:45.45 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -f -c do_rootfs
0.55s user 0.06s system 4% cpu 14.236 total
0.54s user 0.08s system 4% cpu 15.247 total
0.52s user 0.15s system 4% cpu 15.143 total

RPM
===
bitbake core-image-minimal
18.57s user 3.09s system 0% cpu 1:31:29.09 total
18.58s user 3.08s system 0% cpu 1:30:53.80 total
18.20s user 3.31s system 0% cpu 1:31:06.69 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -c do_rootfs -f
0.58s user 0.10s system 3% cpu 19.470 total
0.57s user 0.09s system 3% cpu 19.480 total
0.60s user 0.07s system 3% cpu 20.381 total