Extended SDK installation fails during git fetch of kernel source hosted on private repo

Extended SDK installation fails during git fetch of kernel source hosted on private repo

[Gabriele Favalessa]

[-- Attachment #1: Type: text/plain, Size: 5851 bytes --]

Hi,

I'm trying to install on ubuntu 17.04 an Extensible SDK that I have built,
and it stops with an error:

-------------------------------------------------------
sdk@sdk:~$ ./
acme-linux-glibc-x86_64-acme-image-datalogger-cortexa7hf-neon-toolchain-ext-2.5.1.sh
-D
…
+ sh -c . buildtools/environment-setup* >
/home/sdk/acme-linux_sdk/preparing_build_system.log && cd
/home/sdk/acme-linux_sdk/layers/poky && set /home/sdk/acme-linux_sdk && .
/home/sdk/acme-linux_sdk/layers/poky/oe-init-build-env
/home/sdk/acme-linux_sdk >>
/home/sdk/acme-linux_sdk/preparing_build_system.log && python
/home/sdk/acme-linux_sdk/ext-sdk-prepare.py
/home/sdk/acme-linux_sdk/preparing_build_system.log 'acme-image-datalogger
meta-extsdk-toolchain:do_populate_sysroot'
WARNING:
/home/sdk/acme-linux_sdk/layers/poky/meta-myboard/recipes-kernel/linux/
linux-imx_4.9.11.bb: Exception during build_dependencies for AUTOREV
WARNING:
/home/sdk/acme-linux_sdk/layers/poky/meta-myboard/recipes-kernel/linux/
linux-imx_4.9.11.bb: Error during finalise of
/home/sdk/acme-linux_sdk/layers/poky/meta-myboard/recipes-kernel/linux/
linux-imx_4.9.11.bb
ERROR: ExpansionError during parsing
/home/sdk/acme-linux_sdk/layers/poky/meta-myboard/recipes-kernel/linux/
linux-imx_4.9.11.bb
Traceback (most recent call last):
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression
was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError:
Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export
GIT_SSL_CAINFO="/home/sdk/acme-linux_sdk/buildtools/sysroots/x86_64-acmesdk-linux/etc/ssl/certs/ca-certificates.crt";
export
PATH="/home/sdk/acme-linux_sdk/layers/poky/scripts:/home/sdk/acme-linux_sdk/tmp/work/myboard-acme-linux-gnueabi/linux-imx/4.9.11-r0/recipe-sysroot-native/usr/bin/arm-acme-linux-gnueabi:/home/sdk/acme-linux_sdk/tmp/work/myboard-acme-linux-gnueabi/linux-imx/4.9.11-r0/recipe-sysroot/usr/bin/crossscripts:/home/sdk/acme-linux_sdk/tmp/work/myboard-acme-linux-gnueabi/linux-imx/4.9.11-r0/recipe-sysroot-native/usr/sbin:/home/sdk/acme-linux_sdk/tmp/work/myboard-acme-linux-gnueabi/linux-imx/4.9.11-r0/recipe-sysroot-native/usr/bin:/home/sdk/acme-linux_sdk/tmp/work/myboard-acme-linux-gnueabi/linux-imx/4.9.11-r0/recipe-sysroot-native/sbin:/home/sdk/acme-linux_sdk/tmp/work/myboard-acme-linux-gnueabi/linux-imx/4.9.11-r0/recipe-sysroot-native/bin:/home/sdk/acme-linux_sdk/layers/poky/bitbake/bin:/home/sdk/acme-linux_sdk/tmp/hosttools";
export HOME="/home/sdk"; git -c core.fsyncobjectfiles=0 ls-remote ssh://
git@github.com/Acme/linux-imx.git  failed with exit code 128, output:
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.


ERROR: SDK preparation failed: error log written to
/home/sdk/acme-linux_sdk/preparing_build_system.log
-------------------------------------------------------

I tested the failing git command `git -c ls-remote ssh://
git@github.com/Acme/linux-imx.git` directly from the command line and it
succeeds without errors. I do have the ssh key in ssh-agent, so that's not
surprise.

It looks like during the SDK installation the ssh key is needed but for
some reason the ssh-agent is not reachable (maybe the SSH_* env variables
are not propagated by the script?)

Is it unavoidable for the SDK to fetch from git during the installation? If
it is unavoidable, how can I make the script fetch from a private github
repo, given that the ssh-agent seems to be ignored?

Just out of curiosity, I copy&pasted the failing command in the shell, and
this time it ran without errors (again confirming that the ssh key is
present.)

-------------------------------------------------------
sdk@sdk:~$ sh -c . buildtools/environment-setup* >
/home/sdk/acme-linux_sdk/preparing_build_system.log && cd /home/sdk/
acme-linux_sdk/layers/poky && set /home/sdk/acme-linux_sdk && .
/home/sdk/acme-linux_sdk/layers/poky/oe-init-build-env
/home/sdk/acme-linux_sdk >>
/home/sdk/acme-linux_sdk/preparing_build_system.log && python
/home/sdk/acme-linux_sdk/ext-
sdk-prepare.py /home/sdk/acme-linux_sdk/preparing_build_system.log
'acme-image-datalogger meta-extsdk-toolchain:do_popu
late_sysroot'
Loading cache: 100%
|###################################################################################|
Time: 0:00:00
Parsing recipes: 100%
|#################################################################################|
Time: 0:01:32
Initialising tasks: 100%
|##############################################################################|
Time: 0:00:02
Checking sstate mirror object availability: 100%
|######################################################| Time: 0:00:00
Loading cache: 100%
|###################################################################################|
Time: 0:00:00
Parsing recipes: 100%
|#################################################################################|
Time: 0:00:07
Initialising tasks: 100%
|##############################################################################|
Time: 0:00:00
-------------------------------------------------------

However the resulting SDK installation seems to be broken anyway (probably
there are other steps after the one I re-run manually that where not
executed because the toplevel script stopped with an error):

-------------------------------------------------------
sdk@sdk:~/acme-linux_sdk$ source
environment-setup-cortexa7hf-neon-acme-linux-gnueabi
SDK environment now set up; additionally you may now run devtool to perform
development tasks.
Run devtool --help for further details.
ERROR: this SDK was not fully installed and needs reinstalling
-------------------------------------------------------

Thanks

Gabriele

[-- Attachment #2: Type: text/html, Size: 7075 bytes --]

Re: Extended SDK installation fails during git fetch of kernel source hosted on private repo

[Gabriele Favalessa]

[-- Attachment #1: Type: text/plain, Size: 6957 bytes --]

An update (and a workaround.)

What's happening is that during the execution of the install script for the
extended SDK, the env of the original shell is ignored. If your SDK refers
to private git repos (on github for example) and you normally authenticate
to github using ssh and ssh-agent, the SSH_AUTH_SOCK env variables needed
won't be set by the time git is invoked. The workaround is configuring the
authentication to github in .ssh/config and avoid the ssh-agent. (Yes, I
checked and SSH_AUTH_SOCK is exported by the shell :)

But the underlying problem is that env of the shell is ignored during
install, so potentially other useful variables could be ignored. Is that
intentional? Is there a way to prevent it?

Gabriele

On Wed, Aug 29, 2018 at 3:55 PM Gabriele Favalessa <
gabriele@analyticsfire.com> wrote:

> Hi,
>
> I'm trying to install on ubuntu 17.04 an Extensible SDK that I have built,
> and it stops with an error:
>
> -------------------------------------------------------
> sdk@sdk:~$ ./
> acme-linux-glibc-x86_64-acme-image-datalogger-cortexa7hf-neon-toolchain-ext-2.5.1.sh
> -D
> …
> + sh -c . buildtools/environment-setup* >
> /home/sdk/acme-linux_sdk/preparing_build_system.log && cd
> /home/sdk/acme-linux_sdk/layers/poky && set /home/sdk/acme-linux_sdk && .
> /home/sdk/acme-linux_sdk/layers/poky/oe-init-build-env
> /home/sdk/acme-linux_sdk >>
> /home/sdk/acme-linux_sdk/preparing_build_system.log && python
> /home/sdk/acme-linux_sdk/ext-sdk-prepare.py
> /home/sdk/acme-linux_sdk/preparing_build_system.log 'acme-image-datalogger
> meta-extsdk-toolchain:do_populate_sysroot'
> WARNING:
> /home/sdk/acme-linux_sdk/layers/poky/meta-myboard/recipes-kernel/linux/
> linux-imx_4.9.11.bb: Exception during build_dependencies for AUTOREV
> WARNING:
> /home/sdk/acme-linux_sdk/layers/poky/meta-myboard/recipes-kernel/linux/
> linux-imx_4.9.11.bb: Error during finalise of
> /home/sdk/acme-linux_sdk/layers/poky/meta-myboard/recipes-kernel/linux/
> linux-imx_4.9.11.bb
> ERROR: ExpansionError during parsing
> /home/sdk/acme-linux_sdk/layers/poky/meta-myboard/recipes-kernel/linux/
> linux-imx_4.9.11.bb
> Traceback (most recent call last):
> bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression
> was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError:
> Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export
> GIT_SSL_CAINFO="/home/sdk/acme-linux_sdk/buildtools/sysroots/x86_64-acmesdk-linux/etc/ssl/certs/ca-certificates.crt";
> export
> PATH="/home/sdk/acme-linux_sdk/layers/poky/scripts:/home/sdk/acme-linux_sdk/tmp/work/myboard-acme-linux-gnueabi/linux-imx/4.9.11-r0/recipe-sysroot-native/usr/bin/arm-acme-linux-gnueabi:/home/sdk/acme-linux_sdk/tmp/work/myboard-acme-linux-gnueabi/linux-imx/4.9.11-r0/recipe-sysroot/usr/bin/crossscripts:/home/sdk/acme-linux_sdk/tmp/work/myboard-acme-linux-gnueabi/linux-imx/4.9.11-r0/recipe-sysroot-native/usr/sbin:/home/sdk/acme-linux_sdk/tmp/work/myboard-acme-linux-gnueabi/linux-imx/4.9.11-r0/recipe-sysroot-native/usr/bin:/home/sdk/acme-linux_sdk/tmp/work/myboard-acme-linux-gnueabi/linux-imx/4.9.11-r0/recipe-sysroot-native/sbin:/home/sdk/acme-linux_sdk/tmp/work/myboard-acme-linux-gnueabi/linux-imx/4.9.11-r0/recipe-sysroot-native/bin:/home/sdk/acme-linux_sdk/layers/poky/bitbake/bin:/home/sdk/acme-linux_sdk/tmp/hosttools";
> export HOME="/home/sdk"; git -c core.fsyncobjectfiles=0 ls-remote ssh://
> git@github.com/Acme/linux-imx.git  failed with exit code 128, output:
> Permission denied (publickey).
> fatal: Could not read from remote repository.
>
> Please make sure you have the correct access rights
> and the repository exists.
>
>
> ERROR: SDK preparation failed: error log written to
> /home/sdk/acme-linux_sdk/preparing_build_system.log
> -------------------------------------------------------
>
> I tested the failing git command `git -c ls-remote ssh://
> git@github.com/Acme/linux-imx.git`
> <http://git@github.com/Acme/linux-imx.git> directly from the command line
> and it succeeds without errors. I do have the ssh key in ssh-agent, so
> that's not surprise.
>
> It looks like during the SDK installation the ssh key is needed but for
> some reason the ssh-agent is not reachable (maybe the SSH_* env variables
> are not propagated by the script?)
>
> Is it unavoidable for the SDK to fetch from git during the installation?
> If it is unavoidable, how can I make the script fetch from a private github
> repo, given that the ssh-agent seems to be ignored?
>
> Just out of curiosity, I copy&pasted the failing command in the shell, and
> this time it ran without errors (again confirming that the ssh key is
> present.)
>
> -------------------------------------------------------
> sdk@sdk:~$ sh -c . buildtools/environment-setup* >
> /home/sdk/acme-linux_sdk/preparing_build_system.log && cd /home/sdk/
> acme-linux_sdk/layers/poky && set /home/sdk/acme-linux_sdk && .
> /home/sdk/acme-linux_sdk/layers/poky/oe-init-build-env
> /home/sdk/acme-linux_sdk >>
> /home/sdk/acme-linux_sdk/preparing_build_system.log && python
> /home/sdk/acme-linux_sdk/ext-
> sdk-prepare.py /home/sdk/acme-linux_sdk/preparing_build_system.log
> 'acme-image-datalogger meta-extsdk-toolchain:do_popu
> late_sysroot'
> Loading cache: 100%
> |###################################################################################|
> Time: 0:00:00
> Parsing recipes: 100%
> |#################################################################################|
> Time: 0:01:32
> Initialising tasks: 100%
> |##############################################################################|
> Time: 0:00:02
> Checking sstate mirror object availability: 100%
> |######################################################| Time: 0:00:00
> Loading cache: 100%
> |###################################################################################|
> Time: 0:00:00
> Parsing recipes: 100%
> |#################################################################################|
> Time: 0:00:07
> Initialising tasks: 100%
> |##############################################################################|
> Time: 0:00:00
> -------------------------------------------------------
>
> However the resulting SDK installation seems to be broken anyway (probably
> there are other steps after the one I re-run manually that where not
> executed because the toplevel script stopped with an error):
>
> -------------------------------------------------------
> sdk@sdk:~/acme-linux_sdk$ source
> environment-setup-cortexa7hf-neon-acme-linux-gnueabi
> SDK environment now set up; additionally you may now run devtool to
> perform development tasks.
> Run devtool --help for further details.
> ERROR: this SDK was not fully installed and needs reinstalling
> -------------------------------------------------------
>
> Thanks
>
> Gabriele
>

[-- Attachment #2: Type: text/html, Size: 8399 bytes --]

Re: [yocto] Extended SDK installation fails during git fetch of kernel source hosted on private repo

[Andrea Galbusera]

Hi!

I had a similar issue in my backlog for ages... Finally hit it again
while updating some SDKs and found this thread from more than two
years ago. In my case, fetching during the SDK install script run was
triggered by at least one recipe which was setting SRCREV with
AUTOREV. Although using AUTOREV in generating SDKs is probably
debatable, I found your comments here a good starting point for
further investigation. Looking at the git history since then, it
reveals a recent commit [1] which is reworking (possibly with
different motivations) the way SDK install script is "cleaning" some
env variables at invocation.

I tested backporting this commit to an old pyro based project where I
was seeing SDK install failures with AUTOREV recipes and it solved the
original error.

I could find no issue in Bugzilla for this specific face of the
problem, even though the above commit descries itself as a rework of
the fix for [1]. I'm reporting to the list for the records, in case
someone else would happen to hit issues with using the ssh agent while
installing their SDKs.

Regards,

[1] https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=6d78b84029d0717354bf39e5c355fba1c14d8347
[2] https://bugzilla.yoctoproject.org/show_bug.cgi?id=8698

On Thu, Sep 6, 2018 at 12:18 PM Gabriele Favalessa
<gabriele@analyticsfire.com> wrote:
>
> An update (and a workaround.)
>
> What's happening is that during the execution of the install script for the extended SDK, the env of the original shell is ignored. If your SDK refers to private git repos (on github for example) and you normally authenticate to github using ssh and ssh-agent, the SSH_AUTH_SOCK env variables needed won't be set by the time git is invoked. The workaround is configuring the authentication to github in .ssh/config and avoid the ssh-agent. (Yes, I checked and SSH_AUTH_SOCK is exported by the shell :)
>
> But the underlying problem is that env of the shell is ignored during install, so potentially other useful variables could be ignored. Is that intentional? Is there a way to prevent it?
>
> Gabriele
>
> On Wed, Aug 29, 2018 at 3:55 PM Gabriele Favalessa <gabriele@analyticsfire.com> wrote:
>>
>> Hi,
>>
>> I'm trying to install on ubuntu 17.04 an Extensible SDK that I have built, and it stops with an error:
>>
>> -------------------------------------------------------
>> sdk@sdk:~$ ./acme-linux-glibc-x86_64-acme-image-datalogger-cortexa7hf-neon-toolchain-ext-2.5.1.sh -D
>> …
>> + sh -c . buildtools/environment-setup* > /home/sdk/acme-linux_sdk/preparing_build_system.log && cd /home/sdk/acme-linux_sdk/layers/poky && set /home/sdk/acme-linux_sdk && . /home/sdk/acme-linux_sdk/layers/poky/oe-init-build-env /home/sdk/acme-linux_sdk >> /home/sdk/acme-linux_sdk/preparing_build_system.log && python /home/sdk/acme-linux_sdk/ext-sdk-prepare.py /home/sdk/acme-linux_sdk/preparing_build_system.log 'acme-image-datalogger meta-extsdk-toolchain:do_populate_sysroot'
>> WARNING: /home/sdk/acme-linux_sdk/layers/poky/meta-myboard/recipes-kernel/linux/linux-imx_4.9.11.bb: Exception during build_dependencies for AUTOREV
>> WARNING: /home/sdk/acme-linux_sdk/layers/poky/meta-myboard/recipes-kernel/linux/linux-imx_4.9.11.bb: Error during finalise of /home/sdk/acme-linux_sdk/layers/poky/meta-myboard/recipes-kernel/linux/linux-imx_4.9.11.bb
>> ERROR: ExpansionError during parsing /home/sdk/acme-linux_sdk/layers/poky/meta-myboard/recipes-kernel/linux/linux-imx_4.9.11.bb
>> Traceback (most recent call last):
>> bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export GIT_SSL_CAINFO="/home/sdk/acme-linux_sdk/buildtools/sysroots/x86_64-acmesdk-linux/etc/ssl/certs/ca-certificates.crt"; export PATH="/home/sdk/acme-linux_sdk/layers/poky/scripts:/home/sdk/acme-linux_sdk/tmp/work/myboard-acme-linux-gnueabi/linux-imx/4.9.11-r0/recipe-sysroot-native/usr/bin/arm-acme-linux-gnueabi:/home/sdk/acme-linux_sdk/tmp/work/myboard-acme-linux-gnueabi/linux-imx/4.9.11-r0/recipe-sysroot/usr/bin/crossscripts:/home/sdk/acme-linux_sdk/tmp/work/myboard-acme-linux-gnueabi/linux-imx/4.9.11-r0/recipe-sysroot-native/usr/sbin:/home/sdk/acme-linux_sdk/tmp/work/myboard-acme-linux-gnueabi/linux-imx/4.9.11-r0/recipe-sysroot-native/usr/bin:/home/sdk/acme-linux_sdk/tmp/work/myboard-acme-linux-gnueabi/linux-imx/4.9.11-r0/recipe-sysroot-native/sbin:/home/sdk/acme-linux_sdk/tmp/work/myboard-acme-linux-gnueabi/linux-imx/4.9.11-r0/recipe-sysroot-native/bin:/home/sdk/acme-linux_sdk/layers/poky/bitbake/bin:/home/sdk/acme-linux_sdk/tmp/hosttools"; export HOME="/home/sdk"; git -c core.fsyncobjectfiles=0 ls-remote ssh://git@github.com/Acme/linux-imx.git  failed with exit code 128, output:
>> Permission denied (publickey).
>> fatal: Could not read from remote repository.
>>
>> Please make sure you have the correct access rights
>> and the repository exists.
>>
>>
>> ERROR: SDK preparation failed: error log written to /home/sdk/acme-linux_sdk/preparing_build_system.log
>> -------------------------------------------------------
>>
>> I tested the failing git command `git -c ls-remote ssh://git@github.com/Acme/linux-imx.git` directly from the command line and it succeeds without errors. I do have the ssh key in ssh-agent, so that's not surprise.
>>
>> It looks like during the SDK installation the ssh key is needed but for some reason the ssh-agent is not reachable (maybe the SSH_* env variables are not propagated by the script?)
>>
>> Is it unavoidable for the SDK to fetch from git during the installation? If it is unavoidable, how can I make the script fetch from a private github repo, given that the ssh-agent seems to be ignored?
>>
>> Just out of curiosity, I copy&pasted the failing command in the shell, and this time it ran without errors (again confirming that the ssh key is present.)
>>
>> -------------------------------------------------------
>> sdk@sdk:~$ sh -c . buildtools/environment-setup* > /home/sdk/acme-linux_sdk/preparing_build_system.log && cd /home/sdk/
>> acme-linux_sdk/layers/poky && set /home/sdk/acme-linux_sdk && . /home/sdk/acme-linux_sdk/layers/poky/oe-init-build-env
>> /home/sdk/acme-linux_sdk >> /home/sdk/acme-linux_sdk/preparing_build_system.log && python /home/sdk/acme-linux_sdk/ext-
>> sdk-prepare.py /home/sdk/acme-linux_sdk/preparing_build_system.log 'acme-image-datalogger meta-extsdk-toolchain:do_popu
>> late_sysroot'
>> Loading cache: 100% |###################################################################################| Time: 0:00:00
>> Parsing recipes: 100% |#################################################################################| Time: 0:01:32
>> Initialising tasks: 100% |##############################################################################| Time: 0:00:02
>> Checking sstate mirror object availability: 100% |######################################################| Time: 0:00:00
>> Loading cache: 100% |###################################################################################| Time: 0:00:00
>> Parsing recipes: 100% |#################################################################################| Time: 0:00:07
>> Initialising tasks: 100% |##############################################################################| Time: 0:00:00
>> -------------------------------------------------------
>>
>> However the resulting SDK installation seems to be broken anyway (probably there are other steps after the one I re-run manually that where not executed because the toplevel script stopped with an error):
>>
>> -------------------------------------------------------
>> sdk@sdk:~/acme-linux_sdk$ source environment-setup-cortexa7hf-neon-acme-linux-gnueabi
>> SDK environment now set up; additionally you may now run devtool to perform development tasks.
>> Run devtool --help for further details.
>> ERROR: this SDK was not fully installed and needs reinstalling
>> -------------------------------------------------------
>>
>> Thanks
>>
>> Gabriele
>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto