[PATCH] cmake: Export SSH_AUTH_SOCK for cmake at configure

[PATCH] cmake: Export SSH_AUTH_SOCK for cmake at configure

[Niko Mauno]

Update cmake_do_configure() to export a set SSH_AUTH_SOCK variable
before calling cmake.

Otherwise, if cmake call during cmake_do_configure() resorts to
ExternalProject directive containing a GIT_REPOSITORY entry, and git
authentication scheme is based on SSH agent forwarding, it fails
followingly

  | Cloning into 'foo'...
  | Permission denied (publickey).
  | fatal: Could not read from remote repository.
  |
  | Please make sure you have the correct access rights
  | and the repository exists.
  |
  | ...
  |
  | CMake Error at .../tmp/foo-gitclone.cmake:66 (message):
  |   Failed to clone repository: 'ssh://...

Signed-off-by: Niko Mauno <niko.mauno@vaisala.com>
---
 meta/classes/cmake.bbclass | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/classes/cmake.bbclass b/meta/classes/cmake.bbclass
index fcfd5dda4f..82d36be8ff 100644
--- a/meta/classes/cmake.bbclass
+++ b/meta/classes/cmake.bbclass
@@ -137,6 +137,11 @@ cmake_do_configure() {
 		oecmake_sitefile=
 	fi
 
+	# Allow cmake to perform eg. git clone in context where authentication relies on SSH agent forwarding
+	if [ "${SSH_AUTH_SOCK}" ] ; then
+		export SSH_AUTH_SOCK=${SSH_AUTH_SOCK}
+	fi
+
 	cmake \
 	  ${OECMAKE_GENERATOR_ARGS} \
 	  $oecmake_sitefile \
-- 
2.11.0

Re: [PATCH] cmake: Export SSH_AUTH_SOCK for cmake at configure

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2030 bytes --]

We should probably investigate if it's possible to shift those into
do_fetch eventually. Contacting upstream during do_configure is not kosher.

On Fri, May 25, 2018 at 5:52 AM Niko Mauno <niko.mauno@vaisala.com> wrote:

> Update cmake_do_configure() to export a set SSH_AUTH_SOCK variable
> before calling cmake.
>
> Otherwise, if cmake call during cmake_do_configure() resorts to
> ExternalProject directive containing a GIT_REPOSITORY entry, and git
> authentication scheme is based on SSH agent forwarding, it fails
> followingly
>
>   | Cloning into 'foo'...
>   | Permission denied (publickey).
>   | fatal: Could not read from remote repository.
>   |
>   | Please make sure you have the correct access rights
>   | and the repository exists.
>   |
>   | ...
>   |
>   | CMake Error at .../tmp/foo-gitclone.cmake:66 (message):
>   |   Failed to clone repository: 'ssh://...
>
> Signed-off-by: Niko Mauno <niko.mauno@vaisala.com>
> ---
>  meta/classes/cmake.bbclass | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/meta/classes/cmake.bbclass b/meta/classes/cmake.bbclass
> index fcfd5dda4f..82d36be8ff 100644
> --- a/meta/classes/cmake.bbclass
> +++ b/meta/classes/cmake.bbclass
> @@ -137,6 +137,11 @@ cmake_do_configure() {
>                 oecmake_sitefile=
>         fi
>
> +       # Allow cmake to perform eg. git clone in context where
> authentication relies on SSH agent forwarding
> +       if [ "${SSH_AUTH_SOCK}" ] ; then
> +               export SSH_AUTH_SOCK=${SSH_AUTH_SOCK}
> +       fi
> +
>         cmake \
>           ${OECMAKE_GENERATOR_ARGS} \
>           $oecmake_sitefile \
> --
> 2.11.0
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>


-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2869 bytes --]

Re: [PATCH] cmake: Export SSH_AUTH_SOCK for cmake at configure

[Hannu Lounento]

On 25/05/18 18:34, Christopher Larson wrote:
> We should probably investigate if it's possible to shift those into 
> do_fetch eventually. Contacting upstream during do_configure is not kosher.

The reason for fetching during configure is that the CMake build system 
in the git repository in question (later referred to as the "parent 
project") uses the ExternalProject keyword [1], which "creates a custom 
target to drive download, update/patch, configure, build, install and 
test steps of an external project" [1].

The external project that the aforementioned CMake build system uses is 
gtest (later referred to as the "child project"), which is many times 
built as part of a parent project. Usually git submodule is used to 
clone child projects, which works nicely with Bitbake with the gitsm 
fetcher. In this particular case the parent project uses the 
ExternalProject keyword instead, which is one of the options suggested 
in gtest's documentation [2]. While the parent project is not a public 
open source project, it is not in our direct control either and hence we 
cannot just change it to use git submodule instead.

Another option _might_ be to run CMake in two phases: the first pass 
would just run the ExternalProject keyword and the second the real 
configure step. We are not aware of a way to run CMake in multiple 
phases though. Also cmake.bbclass (currently) doesn't have anything 
related to the do_fetch step. Thirdly, I assume that running the 
ExternalProject keyword as the first pass during do_fetch would also 
configure and build the child project, which would then again conflict 
with Bitbake's later build steps such as do_configure and do_compile.

I think it would be a valid use case to support fetching by the 
ExternalProject keyword even during do_configure because a recipe 
maintainer does not necessarily have direct control over the upstream 
project.

If it is possible to resolve the issue e.g. by changing cmake.bbclass to 
call CMake differently or other means, any guidance is appreciated.

[1] https://cmake.org/cmake/help/latest/module/ExternalProject.html
[2] 
https://github.com/google/googletest/blob/master/googletest/README.md#incorporating-into-an-existing-cmake-project


Thanks,
-- 
Hannu Lounento
hannu.lounento@vaisala.com

> 
> On Fri, May 25, 2018 at 5:52 AM Niko Mauno <niko.mauno@vaisala.com 
> <mailto:niko.mauno@vaisala.com>> wrote:
> 
>     Update cmake_do_configure() to export a set SSH_AUTH_SOCK variable
>     before calling cmake.
> 
>     Otherwise, if cmake call during cmake_do_configure() resorts to
>     ExternalProject directive containing a GIT_REPOSITORY entry, and git
>     authentication scheme is based on SSH agent forwarding, it fails
>     followingly
> 
>        | Cloning into 'foo'...
>        | Permission denied (publickey).
>        | fatal: Could not read from remote repository.
>        |
>        | Please make sure you have the correct access rights
>        | and the repository exists.
>        |
>        | ...
>        |
>        | CMake Error at .../tmp/foo-gitclone.cmake:66 (message):
>        |   Failed to clone repository: 'ssh://...
> 
>     Signed-off-by: Niko Mauno <niko.mauno@vaisala.com
>     <mailto:niko.mauno@vaisala.com>>
>     ---
>       meta/classes/cmake.bbclass | 5 +++++
>       1 file changed, 5 insertions(+)
> 
>     diff --git a/meta/classes/cmake.bbclass b/meta/classes/cmake.bbclass
>     index fcfd5dda4f..82d36be8ff 100644
>     --- a/meta/classes/cmake.bbclass
>     +++ b/meta/classes/cmake.bbclass
>     @@ -137,6 +137,11 @@ cmake_do_configure() {
>                      oecmake_sitefile=
>              fi
> 
>     +       # Allow cmake to perform eg. git clone in context where
>     authentication relies on SSH agent forwarding
>     +       if [ "${SSH_AUTH_SOCK}" ] ; then
>     +               export SSH_AUTH_SOCK=${SSH_AUTH_SOCK}
>     +       fi
>     +
>              cmake \
>                ${OECMAKE_GENERATOR_ARGS} \
>                $oecmake_sitefile \
>     -- 
>     2.11.0
> 
>     -- 
>     _______________________________________________
>     Openembedded-core mailing list
>     Openembedded-core@lists.openembedded.org
>     <mailto:Openembedded-core@lists.openembedded.org>
>     http://lists.openembedded.org/mailman/listinfo/openembedded-core
> 
> 
> 
> -- 
> Christopher Larson
> kergoth at gmail dot com
> Founder - BitBake, OpenEmbedded, OpenZaurus
> Senior Software Engineer, Mentor Graphics
> 
> 

dealing with CMake ExternalProject (was: Re: [PATCH] cmake: Export SSH_AUTH_SOCK for cmake at configure)

[André Draszik]

On Mon, 2018-05-28 at 11:55 +0300, Hannu Lounento wrote:
> On 25/05/18 18:34, Christopher Larson wrote:
> > We should probably investigate if it's possible to shift those into 
> > do_fetch eventually. Contacting upstream during do_configure is not
> > kosher.
> 
> The reason for fetching during configure is that the CMake build system 
> in the git repository in question (later referred to as the "parent 
> project") uses the ExternalProject keyword [1], which "creates a custom 
> target to drive download, update/patch, configure, build, install and 
> test steps of an external project" [1].
> 
> The external project that the aforementioned CMake build system uses is 
> gtest (later referred to as the "child project"), which is many times 
> built as part of a parent project. Usually git submodule is used to 
> clone child projects, which works nicely with Bitbake with the gitsm 
> fetcher. In this particular case the parent project uses the 
> ExternalProject keyword instead, which is one of the options suggested 
> in gtest's documentation [2]. While the parent project is not a public 
> open source project, it is not in our direct control either and hence we 
> cannot just change it to use git submodule instead.

You can simply instruct ExternalProject to refer to an existing (local)
directory, instead of a remote git repository.  Either by patching it in
your parent project's recipe, or trying to get upstream to integrate such a
change:

Something along the lines of this works fine for me here:

CMakeLists.txt:

find_path(GTEST_SRC_LOCATION
    NAMES googlemock/src/gmock-all.cc googletest/src/gtest-all.cc
    HINTS ENV GTEST_SRC_ROOT
    PATHS "/usr/src/googletest"
    NO_DEFAULT_PATH
    )
if (GTEST_SRC_LOCATION)
    set(GTEST_GIT_REPOSITORY "" CACHE INTERNAL "googletest git repository")
    set(GTEST_GIT_TAG        "" CACHE INTERNAL "googletest git tag")
else ()
    set(GTEST_GIT_REPOSITORY "https://github.com/google/googletest.git" CACHE INTERNAL "googletest git repository")
    set(GTEST_GIT_TAG        "release-1.8.0" CACHE INTERNAL "googletest git tag")
    set(GTEST_SRC_LOCATION "${CMAKE_CURRENT_BINARY_DIR}/googletest-src" CACHE INTERNAL "googletest source location")
endif ()
configure_file(CMakeLists.txt.in googletest-download/CMakeLists.txt)


And CMakeLists.txt.in contains (amongst others):

include(ExternalProject)
ExternalProject_Add(googletest
  GIT_REPOSITORY    ${GTEST_GIT_REPOSITORY}
  GIT_TAG           ${GTEST_GIT_TAG}
  SOURCE_DIR        "${GTEST_SRC_LOCATION}"


This way, you can have an independent recipe that downloads gtest, and the
recipe of your parent project just needs to ensure to depend on the
downloaded gtest and set the GTEST_SRC_ROOT environment variable. There even
is a debian package for the gtest source code (hence the /usr/src/googletest
above), so not only yocto builds, but even local workstation builds outside
yocto would benefit from such a change.

No need for dramatic changes (such as switching to git submodules).


Cheers,
Andre'

Re: dealing with CMake ExternalProject

[Hannu Lounento]

On 28/05/18 15:21, André Draszik wrote:
> You can simply instruct ExternalProject to refer to an existing (local)
> directory, instead of a remote git repository.  Either by patching it in
> your parent project's recipe, or trying to get upstream to integrate such a
> change:
> 
> Something along the lines of this works fine for me here:

This looks like a reasonable change to the upstream project; we'll try it.

Thanks,
-- 
Hannu Lounento
hannu.lounento@vaisala.com