Fetch private gitlab repo using ssh with Yocto recipe #bitbake

Fetch private gitlab repo using ssh with Yocto recipe #bitbake

[hrsourabh011]

[-- Attachment #1: Type: text/plain, Size: 2048 bytes --]

I am trying to fetch a private gitlab repo within Yocto image recipe using SSH protocol. In my image recipe I have passed SRC_URI as:

SRC_URI = " \
       gitsm://git@git.example.com:2224/blah/blah/blah/blah;protocol=ssh;branch=master \
" But this results in the error:

ERROR: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export PATH="/root /build-swu-v2/tmp/sysroots-uninative/x86_64-linux/usr/bin:/root/sources/poky/scripts: /root/build-swu-v2/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/awsclient/1.4- r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/root/build-swu-v2/tmp /work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/awsclient/1.4-r0/recipe-sysroot/usr/bin /crossscripts:/root/build-swu-v2/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi /awsclient/1.4-r0/recipe-sysroot-native/usr/sbin:/root/build-swu-v2/tmp/work/cortexa7t2hf- neon-vfpv4-poky-linux-gnueabi/awsclient/1.4-r0/recipe-sysroot-native/usr/bin:/root/build- swu-v2/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/awsclient/1.4-r0/recipe-sysroot- native/sbin:/root/build-swu-v2/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi /awsclient/1.4-r0/recipe-sysroot-native/bin:/root/sources/poky/bitbake/bin:/root/build-swu- v2/tmp/hosttools"; export HOME="/root"; LANG=C git -c core.fsyncobjectfiles=0 clone --bare  --mirror "ssh://git@git.osb-connagtive.com:2224/iot/products/iot-device-suite_client/aws- iot-client" /root/downloads//git2/git@git.example.com:2224/blah/blah/blah/blah --progress failed with exit code 128, no output
ERROR: Bitbake Fetcher Error: FetchError('Unable to fetch URL from any source.',  'gitsm://git@git.example.com:2224/blah/blah/blah/blah;protocol=ssh;branch=master')
DEBUG: Python function base_do_fetch finished

But I am able to clone the repo using git clone.

SSH key is already added to the Gitlab. There is no config file in my ~/.ssh. Do I need to create a config file? What should be the content of the config file?

Can anyone please let me know how to resolve this?

Thanks in advance.

[-- Attachment #2: Type: text/html, Size: 2185 bytes --]

Re: [yocto] Fetch private gitlab repo using ssh with Yocto recipe #bitbake

[Sourabh Hegde]

[-- Attachment #1: Type: text/plain, Size: 1687 bytes --]

Hi Nicolas,

Thanks for your answer.

That's great. Even I am building inside a docker container. I tried with
creating a "config" file in .ssh directory. But I still have same issue.

Can you please let me know how to "forward SSH_AGENT into it to be able
to fetch from internal projects without the need to mount the key into the
container."? I never did that before.

Thanks in advance.

On Fri, Jan 28, 2022, 10:42 Nicolas Jeker <n.jeker@delisys.ch> wrote:

> On Tue, 2022-01-25 at 23:16 -0800, hrsourabh011@gmail.com wrote:
> > I am trying to fetch a private gitlab repo within Yocto image recipe
> > using SSH protocol. In my image recipe I have passed SRC_URI as:
> >
> > SRC_URI = " \
> >         gitsm://git@git.example.com:2224/blah/blah/blah/blah;protocol
> > =ssh;branch=master \
> > "
>
> I use almost the same, just without submodules.
>
> SRC_URI =
> "git://git@git.example.com:1234/group/project.git;protocol=ssh"
>
> It should "just work" if ssh is able to find your key. I often build in
> a docker container, so I have to forward SSH_AGENT into it to be able
> to fetch from internal projects without the need to mount the key into
> the container. I don't really have any insight for builds outside
> docker, if git clone works, the bitbake fetcher should too.
>
> > But this results in the error:
> >
> <snip>
> >
> > But I am able to clone the repo using git clone.
> > SSH key is already added to the Gitlab. There is no config file in my
> > ~/.ssh. Do I need to create a config file? What should be the content
> > of the config file?
>
> You should not need a ssh config file.
>
> > Can anyone please let me know how to resolve this?
> > Thanks in advance.
>
>

[-- Attachment #2: Type: text/html, Size: 2699 bytes --]

AW: [yocto] Fetch private gitlab repo using ssh with Yocto recipe #bitbake

[embedded (VIVAVIS AG)]

Hi,
 
> Von: yocto@lists.yoctoproject.org <yocto@lists.yoctoproject.org> Im Auftrag von Sourabh Hegde
> Gesendet: Freitag, 28. Januar 2022 10:47
>
> Can you please let me know how to "forward SSH_AGENT into it to be able
> to fetch from internal projects without the need to mount the key into the container."? I never did that before.

I use the following options within the Docker run command:
  -v $SSH_AUTH_SOCK:/ssh.socket \
  -e SSH_AUTH_SOCK=/ssh.socket \

Furthermore, I had to mount the .ssh folder into the container to make it working (be aware of security risk).
Additionally, you should check that uid, gid of the user in the container is the same on the host.

Regards,

Carsten

Re: [yocto] Fetch private gitlab repo using ssh with Yocto recipe #bitbake

[Erik Botö]

On Fri, Jan 28, 2022 at 11:50 AM Nicolas Jeker <n.jeker@delisys.ch> wrote:
>
> On Fri, 2022-01-28 at 10:27 +0000, VIVAVIS AG wrote:
> > Hi,
> >
> > > Von: yocto@lists.yoctoproject.org <yocto@lists.yoctoproject.org> Im
> > > Auftrag von Sourabh Hegde
> > > Gesendet: Freitag, 28. Januar 2022 10:47
> > >
> > > Can you please let me know how to "forward SSH_AGENT into it to be
> > > able
> > > to fetch from internal projects without the need to mount the key
> > > into the container."? I never did that before.
> >
> > I use the following options within the Docker run command:
> >   -v $SSH_AUTH_SOCK:/ssh.socket \
> >   -e SSH_AUTH_SOCK=/ssh.socket \
> >
>
> That's pretty much what I use.
>
> > Furthermore, I had to mount the .ssh folder into the container to
> > make it working (be aware of security risk).
> > Additionally, you should check that uid, gid of the user in the
> > container is the same on the host.
>
> I do something similar, my "problem" was that ssh needs the
> .ssh/known_hosts file with a matching entry in addition to your
> key/agent, but mounting the .ssh folder was not possible for me because
> of permissions. Currently, I just created a little script that wraps
> "oe-init-build-env" and populates the known_hosts file accordingly.
>
> mkdir -p ~/.ssh
>
> cat <<EOF >> ~/.ssh/known_hosts
> git.example.com ssh-ed25519 <base64key>
> EOF
>

I use my own Dockerfile based on crops/poky where I do the following,
which might be helpful if you also use this. It sets up the config
changes in /etc/skel/ since it creates users "on the fly" with
matching uid.

# Remove strict host key checking for ssh
# This is needed since the build will pull source over git-ssh
RUN mkdir -p /etc/skel/.ssh/
COPY ci-scripts/docker-stuff/config /etc/skel/.ssh/
RUN echo 'export GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null
-o StrictHostKeyChecking=no"' >> /etc/skel/.bashrc


The ci-scripts/docker-stuff/config file contains:
Host *
   StrictHostKeyChecking no
   UserKnownHostsFile=/dev/null

Now it was ages ago I set this up, and right now I can't really
understand why I basically do the same thing twice. So you'd have to
check which of the two things that actually solves the issue :-)

Cheers,
Erik


> > Regards,
> >
> > Carsten
> >
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#56009): https://lists.yoctoproject.org/g/yocto/message/56009
> Mute This Topic: https://lists.yoctoproject.org/mt/88691891/3618217
> Mute #bitbake:https://lists.yoctoproject.org/g/yocto/mutehashtag/bitbake
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [erik.boto@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [yocto] Fetch private gitlab repo using ssh with Yocto recipe #bitbake

[Khem Raj]

On Fri, Jan 28, 2022 at 2:27 AM VIVAVIS AG <embedded@vivavis.com> wrote:
>
> Hi,
>
> > Von: yocto@lists.yoctoproject.org <yocto@lists.yoctoproject.org> Im Auftrag von Sourabh Hegde
> > Gesendet: Freitag, 28. Januar 2022 10:47
> >
> > Can you please let me know how to "forward SSH_AGENT into it to be able
> > to fetch from internal projects without the need to mount the key into the container."? I never did that before.
>
> I use the following options within the Docker run command:
>   -v $SSH_AUTH_SOCK:/ssh.socket \
>   -e SSH_AUTH_SOCK=/ssh.socket \
>
> Furthermore, I had to mount the .ssh folder into the container to make it working (be aware of security risk).
> Additionally, you should check that uid, gid of the user in the container is the same on the host.
>

yeah something like that works, we use it for yoe which always uses
container to build
see

https://github.com/YoeDistro/yoe-distro/blob/master/envsetup.sh#L528-L541

> Regards,
>
> Carsten
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> You automatically follow any topics you start or reply to.
> View/Reply Online (#56008): https://lists.yoctoproject.org/g/yocto/message/56008
> Mute This Topic: https://lists.yoctoproject.org/mt/88691891/1997914
> Mute #bitbake:https://lists.yoctoproject.org/g/yocto/mutehashtag/bitbake
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: Fetch private gitlab repo using ssh with Yocto recipe #bitbake

[Sourabh Hegde]

[-- Attachment #1: Type: text/plain, Size: 2399 bytes --]

Hello @Nicolas @Erik @Khem,

Update from my side:

After following some discussion from other posts, I added "config" file.

~/.ssh/config:

Host git.example.com
HostName git.example.com
User git
PreferredAuthentications publickey
IdentityFile ~/.ssh/id_ed25519.pub
# LogLevel DEBUG3

Then I did "eval `ssh-agent -s`"

Then doing "ssh-add ~/.ssh/id_ed25519.pub" results in:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/root/.ssh/id_ed25519.pub' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.

Whereas the permissions are set as:

ls -l -a ~/.ssh

-rw-r--r-- 1 root root  157 Jan 31 10:48 config
-rw------- 1 root root  464 Jan 20 15:26 id_ed25519
-rw-r--r-- 1 root root  109 Jan 20 15:26 id_ed25519.pub
-rw-r--r-- 1 root root  888 Jan 26 08:43 known_hosts

"ssh-agent" is running

ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-lcft54A4nriC/agent.2833; export SSH_AUTH_SOCK;
SSH_AGENT_PID=2834; export SSH_AGENT_PID;
echo Agent pid 2834;

After doing these changes, when I try to "ssh -v git.example.com" to test the connection before running bitbake, I get

OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /root/.ssh/config
debug1: /root/.ssh/config line 1: Applying options for git.example.com
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to git.example.com [116.203.241.xxx] port 22.
debug1: connect to address 116.203.241.xxx port 22: Connection refused
ssh: connect to host git.example.com port 22: Connection refused

I don't understand what is the issue here.

@Nicolas Can you please let me know where and how to run below commands? Do I need to run them every time before fetching from gitlab?
-v $SSH_AUTH_SOCK:/ssh.socket \
-e SSH_AUTH_SOCK=/ssh.socket \

And also I already have "known_hosts" file with matching entries for key/agent pair.

Can you please let me know how to make this working?

Your help will be much appreciated.

Thanks in advance.

[-- Attachment #2: Type: text/html, Size: 2785 bytes --]