[PATCH v4 0/4] cmake.bbclass improvements

[PATCH v4 0/4] cmake.bbclass improvements

[Pascal Bach]

This patchset is unmodified from v3. It is just rebased on top of master which
already includes a fixed version of libproxy and piglit.

I built all cmake based recipes in oe-core (qemuarm64) and did not find any more issues.

Pascal Bach (4):
  cmake.bbclass: use CMAKE_SYSTEM_LIBRARY_PATH instead of
    CMAKE_LIBRARY_PATH
  cmake.bbclass: search both sysroot-native and host for native packages
  cmake.bbclass: move CMAKE_NO_SYSTEM_FROM_IMPORTED to toolchain.cmake
  cmake.bbclass: allow cmake to find hosttools

 meta/classes/cmake.bbclass | 24 +++++++++++++++---------
 1 file changed, 15 insertions(+), 9 deletions(-)

-- 
2.11.0

[PATCH v4 1/4] cmake.bbclass: use CMAKE_SYSTEM_LIBRARY_PATH instead of CMAKE_LIBRARY_PATH

[Pascal Bach]

CMAKE_LIBRARY_PATH is intended to be set by projects.
CMAKE_SYSTEM_LIBRARY_PATH is better suited to be used in a toolchain
file.

Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
---
 meta/classes/cmake.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/cmake.bbclass b/meta/classes/cmake.bbclass
index fd40a9863e..86b1d0e9aa 100644
--- a/meta/classes/cmake.bbclass
+++ b/meta/classes/cmake.bbclass
@@ -106,7 +106,7 @@ set( CMAKE_INSTALL_RPATH ${OECMAKE_RPATH} )
 list(APPEND CMAKE_MODULE_PATH "${STAGING_DATADIR}/cmake/Modules/")
 
 # add for non /usr/lib libdir, e.g. /usr/lib64
-set( CMAKE_LIBRARY_PATH ${libdir} ${base_libdir})
+set( CMAKE_SYSTEM_LIBRARY_PATH ${libdir} ${base_libdir})
 
 EOF
 }
-- 
2.11.0

[PATCH v4 2/4] cmake.bbclass: search both sysroot-native and host for native packages

[Pascal Bach]

Certain headers and libraries like `math.h` an `-m` are only available on the
host as they are provided by the host toolchain.

This leads to issues that a find_library in CMake doesn't find the `m` library
of a find_path doesn't find `math.h`. This issue occurred in the wireshark recipe
for example.

This change enables CMake to also look on the host for libraries and includes when
building a native package.

Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
---
 meta/classes/cmake.bbclass | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/meta/classes/cmake.bbclass b/meta/classes/cmake.bbclass
index 86b1d0e9aa..ce3c0278ff 100644
--- a/meta/classes/cmake.bbclass
+++ b/meta/classes/cmake.bbclass
@@ -43,8 +43,8 @@ OECMAKE_RPATH ?= ""
 OECMAKE_PERLNATIVE_DIR ??= ""
 OECMAKE_EXTRA_ROOT_PATH ?= ""
 
-OECMAKE_FIND_ROOT_PATH_MODE_PROGRAM = "ONLY"
-OECMAKE_FIND_ROOT_PATH_MODE_PROGRAM_class-native = "BOTH"
+OECMAKE_FIND_ROOT_PATH_MODE = "ONLY"
+OECMAKE_FIND_ROOT_PATH_MODE_class-native = "BOTH"
 
 EXTRA_OECMAKE_append = " ${PACKAGECONFIG_CONFARGS}"
 
@@ -90,10 +90,10 @@ set( CMAKE_CXX_LINK_FLAGS "${OECMAKE_CXX_LINK_FLAGS}" CACHE STRING "LDFLAGS" )
 # only search in the paths provided so cmake doesnt pick
 # up libraries and tools from the native build machine
 set( CMAKE_FIND_ROOT_PATH ${STAGING_DIR_HOST} ${STAGING_DIR_NATIVE} ${CROSS_DIR} ${OECMAKE_PERLNATIVE_DIR} ${OECMAKE_EXTRA_ROOT_PATH} ${EXTERNAL_TOOLCHAIN})
-set( CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY )
-set( CMAKE_FIND_ROOT_PATH_MODE_PROGRAM ${OECMAKE_FIND_ROOT_PATH_MODE_PROGRAM} )
-set( CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY )
-set( CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY )
+set( CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ${OECMAKE_FIND_ROOT_PATH_MODE} )
+set( CMAKE_FIND_ROOT_PATH_MODE_PROGRAM ${OECMAKE_FIND_ROOT_PATH_MODE} )
+set( CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ${OECMAKE_FIND_ROOT_PATH_MODE} )
+set( CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ${OECMAKE_FIND_ROOT_PATH_MODE} )
 
 # Use qt.conf settings
 set( ENV{QT_CONF_PATH} ${WORKDIR}/qt.conf )
-- 
2.11.0

Re: [PATCH v4 2/4] cmake.bbclass: search both sysroot-native and host for native packages

[Burton, Ross]

I think I figured out why we're seeing odd failures on the
autobuilder, such as
https://autobuilder.yoctoproject.org/typhoon/#/builders/15/builds/125.

The important bit of the log is:

File "/home/pokybuild/yocto-worker/poky-tiny/build/build/tmp/work/qemux86-poky-linux-musl/core-image-minimal/1.0-r0/recipe-sysroot-native/usr/lib/python3.5/site-packages/libcomps/__init__.py",
line 1, in <module>
from ._libpycomps import *
ImportError: libpython3.6m.so.1.0: cannot open shared object file: No
such file or directory

The smoking gun is that oe-core doesn't have Python 3.6, but 3.5.

My suspicion is that for systems where the host has python3-devel
installed, libcomp-native's cmake is using the host python instead of
the native python in the sysroot.  I'm not sure yet if BOTH means it
searches the host before the sysroots, or if it is finding the python
binary in HOSTTOOLS and then asking that what the paths are.  Either
way, it ends up linking to the host libpython3.5.so and going into
sstate.  Another machine in the pool reuses this sstate but it doesn't
have Python 3.6 installed, so the library link is broken.

Ross

Re: [PATCH v4 2/4] cmake.bbclass: search both sysroot-native and host for native packages

[Pascal Bach]

On 17.10.2018 18:24, Burton, Ross wrote:
> I think I figured out why we're seeing odd failures on the
> autobuilder, such as
> https://autobuilder.yoctoproject.org/typhoon/#/builders/15/builds/125.
>
> The important bit of the log is:
>
> File "/home/pokybuild/yocto-worker/poky-tiny/build/build/tmp/work/qemux86-poky-linux-musl/core-image-minimal/1.0-r0/recipe-sysroot-native/usr/lib/python3.5/site-packages/libcomps/__init__.py",
> line 1, in <module>
> from ._libpycomps import *
> ImportError: libpython3.6m.so.1.0: cannot open shared object file: No
> such file or directory
>
> The smoking gun is that oe-core doesn't have Python 3.6, but 3.5.
>
> My suspicion is that for systems where the host has python3-devel
> installed, libcomp-native's cmake is using the host python instead of
> the native python in the sysroot.  I'm not sure yet if BOTH means it
> searches the host before the sysroots, or if it is finding the python
> binary in HOSTTOOLS and then asking that what the paths are.  Either
> way, it ends up linking to the host libpython3.5.so and going into
> sstate.  Another machine in the pool reuses this sstate but it doesn't
> have Python 3.6 installed, so the library link is broken.
That is definitively not the intention here.

For target builds it is pretty clear that there should be no usage of host dependencies.
For native it is not so clear to me. There seems to be some dependencies like the compiler and associated libs.

Maybe there is a way to bring this in a different way that is more selective instead of a general fallback to host as it would be with this patch?

What I'm traying to solve here was to get rid of these two lines in the wireshark recipe [1]:

-DM_INCLUDE_DIR=${includedir} \
-DM_LIBRARY=${libdir} \

As they effectively add /usr/include and /usr/lib to the search path of cmake.

Any toughts?

Pascal

[1] http://cgit.openembedded.org/meta-openembedded/tree/meta-networking/recipes-support/wireshark/wireshark_2.6.2.bb?h=master#n62

Re: [PATCH v4 2/4] cmake.bbclass: search both sysroot-native and host for native packages

[Burton, Ross]

On Thu, 18 Oct 2018 at 16:05, Pascal Bach <pascal.bach@siemens.com> wrote:
> > My suspicion is that for systems where the host has python3-devel
> > installed, libcomp-native's cmake is using the host python instead of
> > the native python in the sysroot.  I'm not sure yet if BOTH means it
> > searches the host before the sysroots, or if it is finding the python
> > binary in HOSTTOOLS and then asking that what the paths are.  Either
> > way, it ends up linking to the host libpython3.5.so and going into
> > sstate.  Another machine in the pool reuses this sstate but it doesn't
> > have Python 3.6 installed, so the library link is broken.
> That is definitively not the intention here.
>
> For target builds it is pretty clear that there should be no usage of host dependencies.
> For native it is not so clear to me. There seems to be some dependencies like the compiler and associated libs.

I think ensuring the ordering for native searches is sysroot then host
would be sufficient.

> What I'm traying to solve here was to get rid of these two lines in the wireshark recipe [1]:
>
> -DM_INCLUDE_DIR=${includedir} \
> -DM_LIBRARY=${libdir} \
>
> As they effectively add /usr/include and /usr/lib to the search path of cmake.

Well that's terrifying.  What is that trying to solve?  It's building
for the target, right? So why is it passing host paths?

Ross

[PATCH v4 3/4] cmake.bbclass: move CMAKE_NO_SYSTEM_FROM_IMPORTED to toolchain.cmake

[Pascal Bach]

The setting influences the build like other settings already in toolchain.cmake.
It is more appropriate to set it there instead of providing it as a random
command line parameter to CMake.

It also makes it easier to use the toolchain.cmake file independent of bitbake.
Like the devshell for example.

Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
---
 meta/classes/cmake.bbclass | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/classes/cmake.bbclass b/meta/classes/cmake.bbclass
index ce3c0278ff..0ef63795eb 100644
--- a/meta/classes/cmake.bbclass
+++ b/meta/classes/cmake.bbclass
@@ -108,6 +108,9 @@ list(APPEND CMAKE_MODULE_PATH "${STAGING_DATADIR}/cmake/Modules/")
 # add for non /usr/lib libdir, e.g. /usr/lib64
 set( CMAKE_SYSTEM_LIBRARY_PATH ${libdir} ${base_libdir})
 
+# avoid treating imports as system includes
+set( CMAKE_NO_SYSTEM_FROM_IMPORTED ON)
+
 EOF
 }
 
@@ -152,7 +155,6 @@ cmake_do_configure() {
 	  -DCMAKE_INSTALL_SO_NO_EXE=0 \
 	  -DCMAKE_TOOLCHAIN_FILE=${WORKDIR}/toolchain.cmake \
 	  -DCMAKE_VERBOSE_MAKEFILE=1 \
-	  -DCMAKE_NO_SYSTEM_FROM_IMPORTED=1 \
 	  ${EXTRA_OECMAKE} \
 	  -Wno-dev
 }
-- 
2.11.0

[PATCH v4 4/4] cmake.bbclass: allow cmake to find hosttools

[Pascal Bach]

Currently the generated toolchain file is unable to find hosttools as they
do not appear in the search paths.

Just adding HOSTTOOLS_DIR is not enough as binaries are located directly under
${HOSTTOOLS_DIR}. Like ${HOSTTOOLS_DIR}/git for example.
CMake however only searches in [s]bin sub directories of the paths specified in
CMAKE_FIND_ROOT_PATH. Explicitly adding / to CMAKE_SYSTEM_PROGRAM_PATH makes
CMake look in the right location.

Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
---
 meta/classes/cmake.bbclass | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/classes/cmake.bbclass b/meta/classes/cmake.bbclass
index 0ef63795eb..c8a079c417 100644
--- a/meta/classes/cmake.bbclass
+++ b/meta/classes/cmake.bbclass
@@ -89,7 +89,7 @@ set( CMAKE_CXX_LINK_FLAGS "${OECMAKE_CXX_LINK_FLAGS}" CACHE STRING "LDFLAGS" )
 
 # only search in the paths provided so cmake doesnt pick
 # up libraries and tools from the native build machine
-set( CMAKE_FIND_ROOT_PATH ${STAGING_DIR_HOST} ${STAGING_DIR_NATIVE} ${CROSS_DIR} ${OECMAKE_PERLNATIVE_DIR} ${OECMAKE_EXTRA_ROOT_PATH} ${EXTERNAL_TOOLCHAIN})
+set( CMAKE_FIND_ROOT_PATH ${STAGING_DIR_HOST} ${STAGING_DIR_NATIVE} ${CROSS_DIR} ${OECMAKE_PERLNATIVE_DIR} ${OECMAKE_EXTRA_ROOT_PATH} ${EXTERNAL_TOOLCHAIN} ${HOSTTOOLS_DIR})
 set( CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ${OECMAKE_FIND_ROOT_PATH_MODE} )
 set( CMAKE_FIND_ROOT_PATH_MODE_PROGRAM ${OECMAKE_FIND_ROOT_PATH_MODE} )
 set( CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ${OECMAKE_FIND_ROOT_PATH_MODE} )
@@ -108,6 +108,10 @@ list(APPEND CMAKE_MODULE_PATH "${STAGING_DATADIR}/cmake/Modules/")
 # add for non /usr/lib libdir, e.g. /usr/lib64
 set( CMAKE_SYSTEM_LIBRARY_PATH ${libdir} ${base_libdir})
 
+# by default CMake only looks in [s]bin subdirectories of CMAKE_FIND_ROOT_PATH
+# adding / makes CMake look for binaries in hosttools too.
+set( CMAKE_SYSTEM_PROGRAM_PATH /)
+
 # avoid treating imports as system includes
 set( CMAKE_NO_SYSTEM_FROM_IMPORTED ON)
 
-- 
2.11.0

Re: [PATCH v4 0/4] cmake.bbclass improvements

[Khem Raj]

On Wed, Oct 17, 2018 at 3:43 AM Pascal Bach <pascal.bach@siemens.com> wrote:
>
> This patchset is unmodified from v3. It is just rebased on top of master which
> already includes a fixed version of libproxy and piglit.
>
> I built all cmake based recipes in oe-core (qemuarm64) and did not find any more issues.
>

Can you run world builds on meta-openembedded repositories with
this series applied and also compile some other large projects which
use cmake like meta-browser

> Pascal Bach (4):
>   cmake.bbclass: use CMAKE_SYSTEM_LIBRARY_PATH instead of
>     CMAKE_LIBRARY_PATH
>   cmake.bbclass: search both sysroot-native and host for native packages
>   cmake.bbclass: move CMAKE_NO_SYSTEM_FROM_IMPORTED to toolchain.cmake
>   cmake.bbclass: allow cmake to find hosttools
>
>  meta/classes/cmake.bbclass | 24 +++++++++++++++---------
>  1 file changed, 15 insertions(+), 9 deletions(-)
>
> --
> 2.11.0
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH v4 0/4] cmake.bbclass improvements

[Pascal Bach]

On 17.10.2018 20:50, Khem Raj wrote:
> On Wed, Oct 17, 2018 at 3:43 AM Pascal Bach <pascal.bach@siemens.com> wrote:
>> This patchset is unmodified from v3. It is just rebased on top of master which
>> already includes a fixed version of libproxy and piglit.
>>
>> I built all cmake based recipes in oe-core (qemuarm64) and did not find any more issues.
>>
> Can you run world builds on meta-openembedded repositories with
> this series applied and also compile some other large projects which
> use cmake like meta-browser
I can try but usually I get into trouble with the proxy because there are always some recipes that fetch from some location via some protocol that doesn't work.

>
>> Pascal Bach (4):
>>   cmake.bbclass: use CMAKE_SYSTEM_LIBRARY_PATH instead of
>>     CMAKE_LIBRARY_PATH
>>   cmake.bbclass: search both sysroot-native and host for native packages
>>   cmake.bbclass: move CMAKE_NO_SYSTEM_FROM_IMPORTED to toolchain.cmake
>>   cmake.bbclass: allow cmake to find hosttools
>>
>>  meta/classes/cmake.bbclass | 24 +++++++++++++++---------
>>  1 file changed, 15 insertions(+), 9 deletions(-)
>>
>> --
>> 2.11.0
>>
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH v4 0/4] cmake.bbclass improvements

[Khem Raj]

On Thu, Oct 18, 2018 at 7:58 AM Pascal Bach <pascal.bach@siemens.com> wrote:
>
>
> On 17.10.2018 20:50, Khem Raj wrote:
> > On Wed, Oct 17, 2018 at 3:43 AM Pascal Bach <pascal.bach@siemens.com> wrote:
> >> This patchset is unmodified from v3. It is just rebased on top of master which
> >> already includes a fixed version of libproxy and piglit.
> >>
> >> I built all cmake based recipes in oe-core (qemuarm64) and did not find any more issues.
> >>
> > Can you run world builds on meta-openembedded repositories with
> > this series applied and also compile some other large projects which
> > use cmake like meta-browser
> I can try but usually I get into trouble with the proxy because there are always some recipes that fetch from some location via some protocol that doesn't work.

Once the current cycle of builds is done on OE
builders I will try to schedule this in OE builds and
see where it ends up. hopefully by this weekend.

>
> >
> >> Pascal Bach (4):
> >>   cmake.bbclass: use CMAKE_SYSTEM_LIBRARY_PATH instead of
> >>     CMAKE_LIBRARY_PATH
> >>   cmake.bbclass: search both sysroot-native and host for native packages
> >>   cmake.bbclass: move CMAKE_NO_SYSTEM_FROM_IMPORTED to toolchain.cmake
> >>   cmake.bbclass: allow cmake to find hosttools
> >>
> >>  meta/classes/cmake.bbclass | 24 +++++++++++++++---------
> >>  1 file changed, 15 insertions(+), 9 deletions(-)
> >>
> >> --
> >> 2.11.0
> >>
> >> --
> >> _______________________________________________
> >> Openembedded-core mailing list
> >> Openembedded-core@lists.openembedded.org
> >> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

Re: [PATCH v4 0/4] cmake.bbclass improvements

[Pascal Bach]

On 18.10.2018 17:28, Khem Raj wrote:
> On Thu, Oct 18, 2018 at 7:58 AM Pascal Bach <pascal.bach@siemens.com> wrote:
>>
>> On 17.10.2018 20:50, Khem Raj wrote:
>>> On Wed, Oct 17, 2018 at 3:43 AM Pascal Bach <pascal.bach@siemens.com> wrote:
>>>> This patchset is unmodified from v3. It is just rebased on top of master which
>>>> already includes a fixed version of libproxy and piglit.
>>>>
>>>> I built all cmake based recipes in oe-core (qemuarm64) and did not find any more issues.
>>>>
>>> Can you run world builds on meta-openembedded repositories with
>>> this series applied and also compile some other large projects which
>>> use cmake like meta-browser
>> I can try but usually I get into trouble with the proxy because there are always some recipes that fetch from some location via some protocol that doesn't work.
> Once the current cycle of builds is done on OE
> builders I will try to schedule this in OE builds and
> see where it ends up. hopefully by this weekend.
I was able to do a world build with all layers of meta-oe and there are a few failures:

Two of wich I think are related to cmake:
- libyui-ncurses => messing around with hardcoded path in their cmake setup
- civetweb => still trying to figure out what exactly goes wrong

The following I was unable to build due to fetch problems:
- oscam

I will try to send patches for these two recipes.

>>>> Pascal Bach (4):
>>>>   cmake.bbclass: use CMAKE_SYSTEM_LIBRARY_PATH instead of
>>>>     CMAKE_LIBRARY_PATH
>>>>   cmake.bbclass: search both sysroot-native and host for native packages
>>>>   cmake.bbclass: move CMAKE_NO_SYSTEM_FROM_IMPORTED to toolchain.cmake
>>>>   cmake.bbclass: allow cmake to find hosttools
>>>>
>>>>  meta/classes/cmake.bbclass | 24 +++++++++++++++---------
>>>>  1 file changed, 15 insertions(+), 9 deletions(-)
>>>>
>>>> --
>>>> 2.11.0
>>>>
>>>> --
>>>> _______________________________________________
>>>> Openembedded-core mailing list
>>>> Openembedded-core@lists.openembedded.org
>>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core