Re: [OE-core] Add package managers as a plugin

From: "Fredrik Gustafsson" <fredrik.gustafsson@axis.com>
To: Martin Jansa <martin.jansa@gmail.com>
Cc: Ross Burton <ross@burtonini.com>,
	OE-core <openembedded-core@lists.openembedded.org>,
	tools-cfpbuild-internal <tools-cfpbuild-internal@axis.com>
Subject: Re: [OE-core] Add package managers as a plugin
Date: Tue, 30 Jun 2020 22:12:54 +0000	[thread overview]
Message-ID: <1593555173884.3272@axis.com> (raw)
In-Reply-To: <20200630215424.u77jq5frrnpwztfc@jama>

Hi Martin,
apk is quite different compared to other package managers. A traditional package manager download a package, saves it to a local cache, verifies it and then extract it. That's at least 3 reads and 2 writes of the data. Apk does this on the fly, so 1 read and 1 write of the data. I believe that it's a so fundamental design choice that opkg would have it very hard to compete for speed. The speed advantage is even bigger when installing over network since the extraction and verification is done wile the network is waiting for data. That means that apk has usually installed the package when an ordinary package manager just has downloaded it and should start to verify and install it.

So there's no secret sauce or awesome optimizations, it's "just" an other architectural design.

With that said, there's still some low hanging fruit for apk, for example package generation (my patch is doing 2 reads and 2 writes of all data) and index generation (duplicate reads and one extraction too much) that could have significant speedup. But let's make it possible to use apk at all before starting to tune it.

More information:
https://www.youtube.com/watch?v=sIG2P9k6EjA

BR
Fredrik
________________________________________
From: Martin Jansa <martin.jansa@gmail.com>
Sent: Tuesday, June 30, 2020 11:54 PM
To: Fredrik Gustafsson
Cc: Ross Burton; OE-core; tools-cfpbuild-internal
Subject: Re: [OE-core] Add package managers as a plugin

On Tue, Jun 30, 2020 at 07:01:23PM +0000, Fredrik Gustafsson wrote:
> Hi Ross,
> those 5 seconds will increase to minutes for my use case and we build a lot
> hence I hope this will save us a lot of computer power and engineering time.
> For example I've sent numbers on building a bigger image (core-image-sato-sdk-ptest)
> to Alex and Alex:
> out.apk.1: 1:13.35
> out.apk.2: 1:13.51
> out.apk.3: 1:13.23
> out.apk.4: 1:14.07
> out.apk.5: 1:13.00
> out.deb.1: 3:49.37
> out.deb.2: 3:50.77
> out.deb.3: 3:51.39
> out.deb.4: 3:53.40
> out.deb.5: 3:53.99
> out.ipk.1: 2:38.99
> out.ipk.2: 2:39.07
> out.ipk.3: 2:35.34
> out.ipk.4: 2:36.15
> out.ipk.5: 2:34.55
> out.rpm.1: 1:58.61
> out.rpm.2: 1:59.42
> out.rpm.3: 1:59.70
> out.rpm.4: 1:58.96
> out.rpm.5: 1:58.11
>
> Also consider that if we've a target without python with limited space, rpm is out
> of the question. So the real comparison would be between ipk and apk. Let's
> say we can save 80 seconds in each build. Now multiply that with the number of
> builds, and we do a lot (I guess we might approach 100 000 builds/week in the
> next year). Then this will save 92.5 days build time each week.

This is impressive.

Have you tried to profile where opkg spends most of the time compared to
apk? It would be nice to know if there is something fundamentally
different in how they handle packages or if apk is just better
optimized.

Now with opkg better maintained (thanks Alejandro!) there might be some
relatively low hanging optimalizations which could be implemented there
as well (for people who didn't implement the rest to switch to apk yet
:)).

I know at LGE we have some patches (some related to performance as well) which
unfortunately weren't ever upstreamed :/, some are at:
https://github.com/webosose/meta-webosose/tree/master/meta-webos/recipes-devtools/opkg/opkg

Cheers,