build dependency cycles in openembedded

build dependency cycles in openembedded

[Johannes Schauer]

Dear list,

I am analyzing build dependency graphs for rpm and deb based binary
distributions. I gave a talk about my work on this years FOSDEM [1]. To improve
my work I was looking into Gentoo and Openembedded.

While build dependency cycles in Gentoo are easily taken care of by manually
compiling some source packages with different USE flags, how is this topic
handled in OpenEmbedded?

Searching the wiki, mailinglist and internet I was not able to find good
sources which explain how serious the problem of build dependency cycles in
OpenEmbedded is when bootstrapping a new target from nothing. I also did not
find how build dependency cycles are dealt with, should they occur.

Could you direct me to some resources which document the status and handling of
build dependency cycles in OpenEmbedded?

Thanks!

cheers, josch

[1] https://fosdem.org/2013/schedule/event/debian_bootstrap/

Re: build dependency cycles in openembedded

[Takeshi Hamasaki]

Hi,

you can find the line like:
DEPENDS = "XXXXXX-native"
in some recipes.

So, you can describe dependency to native tools from target build if necessary.
I guess this is the answer for the sentence what I have read the
second paragraph of [1]:

> Since the full distribution for the new architecture is not yet available,
> dependency cycles have to be broken during cross as well as native compilation.

I can't answer "how" the dependency to XXXX-native chosen, in which
recipe and to which native tools.
When I start to try OpenEmbedded, the dependencies were already well-built.
Maybe there is someone who can answer for deeper question.

Regards,
-- 
Takeshi Hamasaki

2013/2/18 Johannes Schauer <j.schauer@email.de>:
> Dear list,
>
> I am analyzing build dependency graphs for rpm and deb based binary
> distributions. I gave a talk about my work on this years FOSDEM [1]. To improve
> my work I was looking into Gentoo and Openembedded.
>
> While build dependency cycles in Gentoo are easily taken care of by manually
> compiling some source packages with different USE flags, how is this topic
> handled in OpenEmbedded?
>
> Searching the wiki, mailinglist and internet I was not able to find good
> sources which explain how serious the problem of build dependency cycles in
> OpenEmbedded is when bootstrapping a new target from nothing. I also did not
> find how build dependency cycles are dealt with, should they occur.
>
> Could you direct me to some resources which document the status and handling of
> build dependency cycles in OpenEmbedded?
>
> Thanks!
>
> cheers, josch
>
> [1] https://fosdem.org/2013/schedule/event/debian_bootstrap/
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel

Re: build dependency cycles in openembedded

[Johannes Schauer]

Hi,

Quoting Takeshi Hamasaki (2013-02-18 13:26:38)
> you can find the line like:
> DEPENDS = "XXXXXX-native"
> in some recipes.
> 
> So, you can describe dependency to native tools from target build if necessary.

Sure, when cross compiling you can satisfy lots of dependencies by native
binaries but others need the foreign version. In Debian based distribution,
this distinction for cross build dependency resolution is made using
Multi-Arch.

Since some of the cross build dependencies can be satisfied by (already
available) native tools, the dependency cycle situation is *much* less severe
during cross compilation than it is during native compilation.

But still, even when one wants to cross compile a native, minimal Debian build
system (just containing the init, shell and compiler) one runs into build
dependency cycles because some dependencies have to be satisfied by foreign
(not yet built) binaries and thereby creating some dependency cycles.

Is it different in OpenEmbedded?

cheers, josch

Re: build dependency cycles in openembedded

[Takeshi Hamasaki]

Before continuing to write reply,
I want to make sure about the usage of the word "native":

in OpenEmbedded recipe, let's say, when you work on a x86-64 system to build
for target system which has ARM architecture,

if you want to borrow a xz tool from x86-64 environment: you can write
 DEPENDS = xz-native

if you depends on xz tool build on target environment: you just write
 DEPENDS = xz

Is this the custom you assume?

What I had to find was the word "native" is used to point the target environment
in [1].
-- 
Takeshi Hamasaki

2013/2/18 Johannes Schauer <j.schauer@email.de>:
> Hi,
>
> Quoting Takeshi Hamasaki (2013-02-18 13:26:38)
>> you can find the line like:
>> DEPENDS = "XXXXXX-native"
>> in some recipes.
>>
>> So, you can describe dependency to native tools from target build if necessary.
>
> Sure, when cross compiling you can satisfy lots of dependencies by native
> binaries but others need the foreign version. In Debian based distribution,
> this distinction for cross build dependency resolution is made using
> Multi-Arch.
>
> Since some of the cross build dependencies can be satisfied by (already
> available) native tools, the dependency cycle situation is *much* less severe
> during cross compilation than it is during native compilation.
>
> But still, even when one wants to cross compile a native, minimal Debian build
> system (just containing the init, shell and compiler) one runs into build
> dependency cycles because some dependencies have to be satisfied by foreign
> (not yet built) binaries and thereby creating some dependency cycles.
>
> Is it different in OpenEmbedded?
>
> cheers, josch
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel

Re: build dependency cycles in openembedded

[Johannes Schauer]

Hi,

Quoting Takeshi Hamasaki (2013-02-18 16:07:36)
> Before continuing to write reply,
> I want to make sure about the usage of the word "native":
> 
> in OpenEmbedded recipe, let's say, when you work on a x86-64 system to build
> for target system which has ARM architecture,
> 
> if you want to borrow a xz tool from x86-64 environment: you can write
>  DEPENDS = xz-native
> 
> if you depends on xz tool build on target environment: you just write
>  DEPENDS = xz
> 
> Is this the custom you assume?
> 
> What I had to find was the word "native" is used to point the target environment
> in [1].

The terminology can indeed become very confusing. In our setup we use the GNU
terminology for cross compilation.

       build machine
           The machine the package is built on.

       host machine
           The machine the package is built for.

In some tools for dependency analysis we develop (dose3) we also use the terms
"native" and "target". For cross compilation, native would be the build
architecture and target would be the host architecture. By your explanation
above, the terms seem to have the same meaning in OpenEmbedded so we seem to be
talking about the same things. :)

cheers, josch

Re: build dependency cycles in openembedded

[Marko Lindqvist]

On 18 February 2013 18:53, Johannes Schauer <j.schauer@email.de> wrote:
> Hi,
>
> Quoting Takeshi Hamasaki (2013-02-18 16:07:36)
>> Before continuing to write reply,
>> I want to make sure about the usage of the word "native":
>>
>> in OpenEmbedded recipe, let's say, when you work on a x86-64 system to build
>> for target system which has ARM architecture,
>>
>> if you want to borrow a xz tool from x86-64 environment: you can write
>>  DEPENDS = xz-native
>>
>> if you depends on xz tool build on target environment: you just write
>>  DEPENDS = xz
>>
>> Is this the custom you assume?
>>
>> What I had to find was the word "native" is used to point the target environment
>> in [1].
>
> The terminology can indeed become very confusing. In our setup we use the GNU
> terminology for cross compilation.
>
>        build machine
>            The machine the package is built on.
>
>        host machine
>            The machine the package is built for.
>
> In some tools for dependency analysis we develop (dose3) we also use the terms
> "native" and "target". For cross compilation, native would be the build
> architecture and target would be the host architecture. By your explanation
> above, the terms seem to have the same meaning in OpenEmbedded so we seem to be
> talking about the same things. :)

 "Native" indeed means binaries that can be run on build architecture.
I think term comes from the fact that they are build as native build,
not cross-compiled. I get more confused when I want to refer something
that's needed in build system prior to OpenEmbedded build, the OE
dependencies. I don't know what's the official term in OpenEmbedded
lingo, but I speak of "build host binaries". Anyway, "native" are
something that are still built as part of OpenEmbedded build process,
such as cross-compiler to be later used in producing binaries for the
target. That might be important for your original question about
cyclic (/infinite recursion) dependencies. OE has both native build
and cross-compile part, and problems can arise in native part too. One
example is that pkg-config version used has not been updated for a
while since build of newer version would depend on glib, which needs
pkg-config to build...


 - ML

Re: build dependency cycles in openembedded

[Johannes Schauer]

Hi,

Quoting Marko Lindqvist (2013-02-18 18:22:26)
> OE has both native build and cross-compile part, and problems can arise in
> native part too. One example is that pkg-config version used has not been
> updated for a while since build of newer version would depend on glib, which
> needs pkg-config to build...

Great example! The same loop exists in binary distributions. Though there it is
a bit more involved since the notion of binary and source packages exists.
Therefor the loop you mention is not between the pkg-config and glib source
packages but between the pkg-config source package and the glib development
headers binary package libglib2.0-dev. So while the glib source package can be
built without pkg-config if ./configure is called with manually supplied
arguments like this:

    ./configure \
        PKG_CONFIG=false \
        ZLIB_CFLAGS=-I/usr/include ZLIB_LIBS="-L/usr/lib -lz" \
        LIBFFI_CFLAGS=-I/usr/include LIBFFI_LIBS="-L/usr/lib -lffi" \
        --disable-libelf \
        --with-pcre=internal

The dependency loop does still exist because the binary package libglib2.0-dev
can only be installed if the pkg-config binary package exists. But the
pkg-config binary package can only be built from the pkg-config source package
using the uninstallable libglib2.0-dev binary package....

So while it is good to hear an example for a dependency cycle in OpenEmbedded,
I now wonder why I didnt find more instances where those impact the development
and deployment of the OpenEmbedded distribution for new platforms?

cheers, josch

Re: build dependency cycles in openembedded

[Andreas Müller]

On Mon, Feb 18, 2013 at 7:54 PM, Johannes Schauer <j.schauer@email.de> wrote:
> Hi,
>
> Quoting Marko Lindqvist (2013-02-18 18:22:26)
>> OE has both native build and cross-compile part, and problems can arise in
>> native part too. One example is that pkg-config version used has not been
>> updated for a while since build of newer version would depend on glib, which
>> needs pkg-config to build...
>
> Great example! The same loop exists in binary distributions. Though there it is
> a bit more involved since the notion of binary and source packages exists.
> Therefor the loop you mention is not between the pkg-config and glib source
> packages but between the pkg-config source package and the glib development
> headers binary package libglib2.0-dev. So while the glib source package can be
> built without pkg-config if ./configure is called with manually supplied
> arguments like this:
>
>     ./configure \
>         PKG_CONFIG=false \
>         ZLIB_CFLAGS=-I/usr/include ZLIB_LIBS="-L/usr/lib -lz" \
>         LIBFFI_CFLAGS=-I/usr/include LIBFFI_LIBS="-L/usr/lib -lffi" \
>         --disable-libelf \
>         --with-pcre=internal
>
> The dependency loop does still exist because the binary package libglib2.0-dev
> can only be installed if the pkg-config binary package exists. But the
> pkg-config binary package can only be built from the pkg-config source package
> using the uninstallable libglib2.0-dev binary package....
>
> So while it is good to hear an example for a dependency cycle in OpenEmbedded,
> I now wonder why I didnt find more instances where those impact the development
> and deployment of the OpenEmbedded distribution for new platforms?
>
> cheers, josch
>
FWIW long time ago in meta-oe commit
5b8522e3b592c96ec9325aff4bbaa68972e52d5b gvfs-gdu-volume-monitor was
spitted out into an own recipe to break dependency loop for gvfs.

Andreas

Re: build dependency cycles in openembedded

[Johannes Schauer]

Hi,

Quoting Andreas Müller (2013-02-18 21:15:43)
> FWIW long time ago in meta-oe commit 5b8522e3b592c96ec9325aff4bbaa68972e52d5b
> gvfs-gdu-volume-monitor was spitted out into an own recipe to break
> dependency loop for gvfs.

If I read this correctly, then this means that for some software packages,
multiple bitbake recipes exist for the purpose of breaking dependency cycles?

How does the generation of build order work? Which algorithm handles the
selection of recipes such that build dependency cycles are avoided? And once a
recipe which was used to break a dependency cycle was compiled, how is the
recompilation of the *full* package scheduled?

cheers, josch

Re: build dependency cycles in openembedded

[Khem Raj]

On (19/02/13 11:46), Johannes Schauer wrote:
> Hi,
> 
> Quoting Andreas Müller (2013-02-18 21:15:43)
> > FWIW long time ago in meta-oe commit 5b8522e3b592c96ec9325aff4bbaa68972e52d5b
> > gvfs-gdu-volume-monitor was spitted out into an own recipe to break
> > dependency loop for gvfs.
> 
> If I read this correctly, then this means that for some software packages,
> multiple bitbake recipes exist for the purpose of breaking dependency cycles?
> 
> How does the generation of build order work? Which algorithm handles the
> selection of recipes such that build dependency cycles are avoided? And once a
> recipe which was used to break a dependency cycle was compiled, how is the
> recompilation of the *full* package scheduled?

bitbake handles the built dependencies and runqueue generation which
decided the order or task execution. You should read about bitbkake
internals if you are interested in internal functioning of dep
resolution in bitbake.