* gitsm fetching fails
@ 2018-11-11 21:04 Stefan Bigler
2018-11-13 15:26 ` Mark Hatle
0 siblings, 1 reply; 3+ messages in thread
From: Stefan Bigler @ 2018-11-11 21:04 UTC (permalink / raw)
To: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 19551 bytes --]
Hi
I'm updating from rocko to thud and recognized, that my projects with
git-submodules cannot be fetched anymore.
I tried to debug the fetch task, but I failed. I did a small example you
should have access and it should be easy to reproduce, the repo should be
public.
The recipe works fine with rocko but fails with the latest master branch of
poky.
I'm using gitlab for my repos. When I use https it works also with thud,
only when *ssh* is used then it fails.
Below you find the recipe, the console output and the important part of the
log.do_fetch
I hope somebody can give me a hint.
Thanks and Regards
Stefan
Here the recipe for https and shh
-----------------------
SUMMARY = "git submodule test"
DESCRIPTION = "recipe to test git submodule fetching"
SECTION = "testing"
LICENSE = "GPLv2"
LIC_FILES_CHKSUM = "file://COPYING;md5=fcb02dc552a041dee27e4b85c7396067"
# fetching module and submodule with https -> works
#SRC_URI = "gitsm://
gitlab.com/gitsm/gitsm-test.git;branch=access-https;protocol=https"
#SRCREV = "db15957a23abb6b8bc8ca1dc948a7bd6ef18b6f3"
# fetching module and submodule with ssh -> fails
SRC_URI = "gitsm://
git@gitlab.com/gitsm/gitsm-test.git;branch=master;protocol=ssh"
SRCREV = "5e400c09b7aff9bfdc2aabbb2d1cd07b94fdafec"
PV = "1.0+git${SRCPV}"
S = "${WORKDIR}/git/"
inherit cmake
Here the console:
-----------------------
Build Configuration:
BB_VERSION = "1.40.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "universal"
TARGET_SYS = "arm-poky-linux-gnueabi"
MACHINE = "beaglebone-yocto"
DISTRO = "poky"
DISTRO_VERSION = "2.6"
TUNE_FEATURES = "arm armv7a vfp neon callconvention-hard cortexa8"
TARGET_FPU = "hard"
meta
meta-poky
meta-yocto-bsp = "master:eddff2b361928e88e3628ebc22a1a0ebb119e01b"
Initialising tasks: 100%
|########################################################################################################################################################################|
Time: 0:00:00
Sstate summary: Wanted 6 Found 0 Missed 6 Current 72 (0% match, 92%
complete)
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
WARNING: gitsm-test-1.0+gitAUTOINC+5e400c09b7-r0 do_fetch: Failed to fetch
URL gitsm://git@gitlab.com/gitsm/gitsm-test.git;branch=master;protocol=ssh,
attempting MIRRORS if available
ERROR: gitsm-test-1.0+gitAUTOINC+5e400c09b7-r0 do_fetch: The URL:
'gitsm:Bigler/gitsm-test-submodule.git;protocol=git@gitlab.com;name=lib;bareclone=1;nocheckout=1;nobranch=1'
is invalid and cannot be interpreted
ERROR: gitsm-test-1.0+gitAUTOINC+5e400c09b7-r0 do_fetch: Fetcher failure
for URL: 'gitsm://
git@gitlab.com/gitsm/gitsm-test.git;branch=master;protocol=ssh'. Unable to
fetch URL from any source.
ERROR: gitsm-test-1.0+gitAUTOINC+5e400c09b7-r0 do_fetch: Function failed:
base_do_fetch
ERROR: Logfile of failure stored in:
/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/temp/log.do_fetch.28468
ERROR: Task
(/opt/yocto/poky/thud/poky-fetch-gitsm/meta-poky/recipes-testing/gitsm-test/gitsm-test_git.bb:do_fetch)
failed with exit code '1'
NOTE: Tasks Summary: Attempted 465 tasks of which 464 didn't need to be
rerun and 1 failed.
Summary: 1 task failed:
/opt/yocto/poky/thud/poky-fetch-gitsm/meta-poky/recipes-testing/gitsm-test/gitsm-test_git.bb:
do_fetch
Summary: There was 1 WARNING message shown.
Summary: There were 3 ERROR messages shown, returning a non-zero exit code.
Here the log (sorry it is long):
-------------------------------------
DEBUG: Executing python function clean_recipe_sysroot
DEBUG: Python function clean_recipe_sysroot finished
DEBUG: Executing python function extend_recipe_sysroot
NOTE: Direct dependencies are []
NOTE: Installed into sysroot: []
NOTE: Skipping as already exists in sysroot: []
DEBUG: Python function extend_recipe_sysroot finished
DEBUG: Executing python function do_fetch
DEBUG: Executing python function base_do_fetch
DEBUG: Trying PREMIRRORS
DEBUG: For url ['gitsm', 'gitlab.com', '/gitsm/gitsm-test.git', 'git', '',
OrderedDict([('branch', 'master'), ('protocol', 'ssh')])] comparing ['bzr',
'.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org',
'/mirror/sources/', '', '', OrderedDict()]
....
DEBUG: Fetching
http://downloads.yoctoproject.org/mirror/sources/git2_gitlab.com.gitsm.gitsm-test.git.tar.gz
using command '/usr/bin/env wget -t 2 -T 30 --passive-ftp
--no-check-certificate -P
/opt/yocto/poky/thud/poky-fetch-gitsm/build/downloads '
http://downloads.yoctoproject.org/mirror/sources/git2_gitlab.com.gitsm.gitsm-test.git.tar.gz
''
DEBUG: Fetcher accessed the network with the command /usr/bin/env wget -t 2
-T 30 --passive-ftp --no-check-certificate -P
/opt/yocto/poky/thud/poky-fetch-gitsm/build/downloads '
http://downloads.yoctoproject.org/mirror/sources/git2_gitlab.com.gitsm.gitsm-test.git.tar.gz
'
DEBUG: Running export PSEUDO_DISABLED=1; export
DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-O1awmP2O9l,guid=1b80485356e847617a9c3d615be850af";
export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh"; export
PATH="/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/scripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot/usr/bin/crossscripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/bitbake/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/hosttools";
export HOME="/home/stefan"; /usr/bin/env wget -t 2 -T 30 --passive-ftp
--no-check-certificate -P
/opt/yocto/poky/thud/poky-fetch-gitsm/build/downloads '
http://downloads.yoctoproject.org/mirror/sources/git2_gitlab.com.gitsm.gitsm-test.git.tar.gz'
--progress=dot -v
--2018-11-11 20:32:40--
http://downloads.yoctoproject.org/mirror/sources/git2_gitlab.com.gitsm.gitsm-test.git.tar.gz
Resolving downloads.yoctoproject.org (downloads.yoctoproject.org)...
198.145.29.63
Connecting to downloads.yoctoproject.org
(downloads.yoctoproject.org)|198.145.29.63|:80...
connected.
HTTP request sent, awaiting response... 404 Not Found
2018-11-11 20:32:41 ERROR 404: Not Found.
DEBUG: Mirror fetch failure for url
http://downloads.yoctoproject.org/mirror/sources/git2_gitlab.com.gitsm.gitsm-test.git.tar.gz
(original url: gitsm://
git@gitlab.com/gitsm/gitsm-test.git;branch=master;protocol=ssh)
DEBUG: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export
DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-O1awmP2O9l,guid=1b80485356e847617a9c3d615be850af";
export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh"; export
PATH="/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/scripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot/usr/bin/crossscripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/bitbake/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/hosttools";
export HOME="/home/stefan"; /usr/bin/env wget -t 2 -T 30 --passive-ftp
--no-check-certificate -P
/opt/yocto/poky/thud/poky-fetch-gitsm/build/downloads '
http://downloads.yoctoproject.org/mirror/sources/git2_gitlab.com.gitsm.gitsm-test.git.tar.gz'
--progress=dot -v failed with exit code 8, output:
--2018-11-11 20:32:40--
http://downloads.yoctoproject.org/mirror/sources/git2_gitlab.com.gitsm.gitsm-test.git.tar.gz
Resolving downloads.yoctoproject.org (downloads.yoctoproject.org)...
198.145.29.63
Connecting to downloads.yoctoproject.org
(downloads.yoctoproject.org)|198.145.29.63|:80...
connected.
HTTP request sent, awaiting response... 404 Not Found
2018-11-11 20:32:41 ERROR 404: Not Found.
DEBUG: Trying Upstream
DEBUG: Fetcher accessed the network with the command LANG=C git -c
core.fsyncobjectfiles=0 clone --bare --mirror ssh://
git@gitlab.com/gitsm/gitsm-test.git
/opt/yocto/poky/thud/poky-fetch-gitsm/build/downloads/git2/gitlab.com.gitsm.gitsm-test.git
--progress
DEBUG: Running export PSEUDO_DISABLED=1; export
DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-O1awmP2O9l,guid=1b80485356e847617a9c3d615be850af";
export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh"; export
PATH="/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/scripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot/usr/bin/crossscripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/bitbake/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/hosttools";
export HOME="/home/stefan"; LANG=C git -c core.fsyncobjectfiles=0 clone
--bare --mirror ssh://git@gitlab.com/gitsm/gitsm-test.git
/opt/yocto/poky/thud/poky-fetch-gitsm/build/downloads/git2/gitlab.com.gitsm.gitsm-test.git
--progress
Cloning into bare repository
'/opt/yocto/poky/thud/poky-fetch-gitsm/build/downloads/git2/gitlab.com.gitsm.gitsm-test.git'...
remote: Enumerating objects: 12, done.
remote: Counting objects: 100% (12/12), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 12 (delta 2), reused 0 (delta 0)
Receiving objects: 100% (12/12), 7.75 KiB | 0 bytes/s, done.
Resolving deltas: 100% (2/2), done.
Checking connectivity... done.
DEBUG: Running export PSEUDO_DISABLED=1; export
DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-O1awmP2O9l,guid=1b80485356e847617a9c3d615be850af";
export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh"; export
PATH="/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/scripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot/usr/bin/crossscripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/bitbake/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/hosttools";
export HOME="/home/stefan"; git -c core.fsyncobjectfiles=0 branch
--contains 5e400c09b7aff9bfdc2aabbb2d1cd07b94fdafec --list master 2>
/dev/null | wc -l
DEBUG: Running export PSEUDO_DISABLED=1; export
DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-O1awmP2O9l,guid=1b80485356e847617a9c3d615be850af";
export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh"; export
PATH="/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/scripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot/usr/bin/crossscripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/bitbake/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/hosttools";
export HOME="/home/stefan"; git -c core.fsyncobjectfiles=0 branch
--contains 5e400c09b7aff9bfdc2aabbb2d1cd07b94fdafec --list master 2>
/dev/null | wc -l
DEBUG: Running export PSEUDO_DISABLED=1; export
DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-O1awmP2O9l,guid=1b80485356e847617a9c3d615be850af";
export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh"; export
PATH="/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/scripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot/usr/bin/crossscripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/bitbake/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/hosttools";
export HOME="/home/stefan"; git -c core.fsyncobjectfiles=0 show
5e400c09b7aff9bfdc2aabbb2d1cd07b94fdafec:.gitmodules
DEBUG: Running export PSEUDO_DISABLED=1; export
DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-O1awmP2O9l,guid=1b80485356e847617a9c3d615be850af";
export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh"; export
PATH="/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/scripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot/usr/bin/crossscripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/bitbake/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/hosttools";
export HOME="/home/stefan"; git -c core.fsyncobjectfiles=0 ls-tree -z -d
5e400c09b7aff9bfdc2aabbb2d1cd07b94fdafec lib
WARNING: Failed to fetch URL gitsm://
git@gitlab.com/gitsm/gitsm-test.git;branch=master;protocol=ssh, attempting
MIRRORS if available
DEBUG: The URL: 'gitsm:Bigler/gitsm-test-submodule.git;protocol=
git@gitlab.com;name=lib;bareclone=1;nocheckout=1;nobranch=1' is invalid and
cannot be interpreted
DEBUG: Trying MIRRORS
DEBUG: For url ['gitsm', 'gitlab.com', '/gitsm/gitsm-test.git', 'git', '',
OrderedDict([('branch', 'master'), ('protocol', 'ssh')])] comparing ['ftp',
'.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org',
'/mirror/sources/', '', '', OrderedDict()]
...
DEBUG: For url ['gitsm', 'gitlab.com', '/gitsm/gitsm-test.git', 'git', '',
OrderedDict([('branch', 'master'), ('protocol', 'ssh')])] comparing ['git',
'.*', '/.*', '', '', OrderedDict()] to ['git', 'HOST', '/PATH', '', '',
OrderedDict([('protocol', 'https')])]
ERROR: The URL: 'gitsm:Bigler/gitsm-test-submodule.git;protocol=
git@gitlab.com;name=lib;bareclone=1;nocheckout=1;nobranch=1' is invalid and
cannot be interpreted
ERROR: Fetcher failure for URL: 'gitsm://
git@gitlab.com/gitsm/gitsm-test.git;branch=master;protocol=ssh'. Unable to
fetch URL from any source.
DEBUG: Python function base_do_fetch finished
DEBUG: Python function do_fetch finished
ERROR: Function failed: base_do_fetch
[-- Attachment #2: Type: text/html, Size: 23584 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: gitsm fetching fails
2018-11-11 21:04 gitsm fetching fails Stefan Bigler
@ 2018-11-13 15:26 ` Mark Hatle
0 siblings, 0 replies; 3+ messages in thread
From: Mark Hatle @ 2018-11-13 15:26 UTC (permalink / raw)
To: bitbake-devel
On 11/11/18 3:04 PM, Stefan Bigler wrote:
> Hi
> I'm updating from rocko to thud and recognized, that my projects with
> git-submodules cannot be fetched anymore.
> I tried to debug the fetch task, but I failed. I did a small example you should
> have access and it should be easy to reproduce, the repo should be public.
> The recipe works fine with rocko but fails with the latest master branch of poky.
> I'm using gitlab for my repos. When I use https it works also with thud, only
> when *ssh* is used then it fails.
> Below you find the recipe, the console output and the important part of the
> log.do_fetch
> I hope somebody can give me a hint.
The previous fetcher handed off everything to git and did not do all of the
validation and network caching/fetching like it should have. This hid a lot of
problems and prevented the system from being able to use declared mirror resources.
The new fetcher iterates over the items and calls the regular git fetcher to
pull down the various components. This ensures that the download directory and
mirrors are used appropriately.
Below the message:
> ERROR: gitsm-test-1.0+gitAUTOINC+5e400c09b7-r0 do_fetch: The URL:
> 'gitsm:Bigler/gitsm-test-submodule.git;protocol=git@gitlab.com
indicates that the submodule is not defined properly. If this is legal, then it
may require some kind of a translation that does not exist.
See: http://git.openembedded.org/bitbake/tree/lib/bb/fetch2/gitsm.py
Look at lines 86 though 96. This is where the submodule URLs are constructed.
The current code is expected the submodules to be defined with URIs, NOT SSH
'strings'.
i.e.:
http://user@host/path/file
ssh://user@host/path/file
NOT
user@host:path/file
It appears that this submodule reference is using the second format, instead of
the actual URI format. gitlab must be doing something different if it works on
http but not ssh, since typically the .gitmodules are declared in a singular
format by the author of the repository and don't change, even if the fetch
method of the initial repository changes.
Since you have a way to reproduce this, I'd suggest working through that small
section of code and trying to determine if it's an invalid URI (SSH style) and
then translating it to a URI for this processing. (If you can make a reproducer
publicly available, then I can try to fix it instead.)
Looking at the git.py, it says the following protocols are supported:
"git", "http", "https", "file", "ssh" and "rsync"
So checking the protocol field for one of those, failing it find it should go
into a translation step most likely. The result (I would assume) will either be
file or ssh. If there is a ':', then most likely it's an ssh.. and the rules to
make this into a proper URI should be fairly simple.
--Mark
> Thanks and Regards
> Stefan
>
> Here the recipe for https and shh
> -----------------------
> SUMMARY = "git submodule test"
> DESCRIPTION = "recipe to test git submodule fetching"
> SECTION = "testing"
> LICENSE = "GPLv2"
> LIC_FILES_CHKSUM = "file://COPYING;md5=fcb02dc552a041dee27e4b85c7396067"
>
> # fetching module and submodule with https -> works
> #SRC_URI =
> "gitsm://gitlab.com/gitsm/gitsm-test.git;branch=access-https;protocol=https
> <http://gitlab.com/gitsm/gitsm-test.git;branch=access-https;protocol=https>"
> #SRCREV = "db15957a23abb6b8bc8ca1dc948a7bd6ef18b6f3"
>
> # fetching module and submodule with ssh -> fails
> SRC_URI =
> "gitsm://git@gitlab.com/gitsm/gitsm-test.git;branch=master;protocol=ssh
> <http://git@gitlab.com/gitsm/gitsm-test.git;branch=master;protocol=ssh>"
> SRCREV = "5e400c09b7aff9bfdc2aabbb2d1cd07b94fdafec"
>
> PV = "1.0+git${SRCPV}"
> S = "${WORKDIR}/git/"
>
> inherit cmake
>
>
> Here the console:
> -----------------------
> Build Configuration:
> BB_VERSION = "1.40.0"
> BUILD_SYS = "x86_64-linux"
> NATIVELSBSTRING = "universal"
> TARGET_SYS = "arm-poky-linux-gnueabi"
> MACHINE = "beaglebone-yocto"
> DISTRO = "poky"
> DISTRO_VERSION = "2.6"
> TUNE_FEATURES = "arm armv7a vfp neon callconvention-hard cortexa8"
> TARGET_FPU = "hard"
> meta
> meta-poky
> meta-yocto-bsp = "master:eddff2b361928e88e3628ebc22a1a0ebb119e01b"
>
> Initialising tasks: 100%
> |########################################################################################################################################################################|
> Time: 0:00:00
> Sstate summary: Wanted 6 Found 0 Missed 6 Current 72 (0% match, 92% complete)
> NOTE: Executing SetScene Tasks
> NOTE: Executing RunQueue Tasks
> WARNING: gitsm-test-1.0+gitAUTOINC+5e400c09b7-r0 do_fetch: Failed to fetch URL
> gitsm://git@gitlab.com/gitsm/gitsm-test.git;branch=master;protocol=ssh
> <http://git@gitlab.com/gitsm/gitsm-test.git;branch=master;protocol=ssh>,
> attempting MIRRORS if available
> ERROR: gitsm-test-1.0+gitAUTOINC+5e400c09b7-r0 do_fetch: The URL:
> 'gitsm:Bigler/gitsm-test-submodule.git;protocol=git@gitlab.com
> <mailto:git@gitlab.com>;name=lib;bareclone=1;nocheckout=1;nobranch=1' is invalid
> and cannot be interpreted
> ERROR: gitsm-test-1.0+gitAUTOINC+5e400c09b7-r0 do_fetch: Fetcher failure for
> URL: 'gitsm://git@gitlab.com/gitsm/gitsm-test.git;branch=master;protocol=ssh
> <http://git@gitlab.com/gitsm/gitsm-test.git;branch=master;protocol=ssh>'. Unable
> to fetch URL from any source.
> ERROR: gitsm-test-1.0+gitAUTOINC+5e400c09b7-r0 do_fetch: Function failed:
> base_do_fetch
> ERROR: Logfile of failure stored in:
> /opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/temp/log.do_fetch.28468
> ERROR: Task
> (/opt/yocto/poky/thud/poky-fetch-gitsm/meta-poky/recipes-testing/gitsm-test/gitsm-test_git.bb:do_fetch)
> failed with exit code '1'
> NOTE: Tasks Summary: Attempted 465 tasks of which 464 didn't need to be rerun
> and 1 failed.
>
> Summary: 1 task failed:
>
> /opt/yocto/poky/thud/poky-fetch-gitsm/meta-poky/recipes-testing/gitsm-test/gitsm-test_git.bb:do_fetch
> Summary: There was 1 WARNING message shown.
> Summary: There were 3 ERROR messages shown, returning a non-zero exit code.
>
> Here the log (sorry it is long):
> -------------------------------------
> DEBUG: Executing python function clean_recipe_sysroot
> DEBUG: Python function clean_recipe_sysroot finished
> DEBUG: Executing python function extend_recipe_sysroot
> NOTE: Direct dependencies are []
> NOTE: Installed into sysroot: []
> NOTE: Skipping as already exists in sysroot: []
> DEBUG: Python function extend_recipe_sysroot finished
> DEBUG: Executing python function do_fetch
> DEBUG: Executing python function base_do_fetch
> DEBUG: Trying PREMIRRORS
> DEBUG: For url ['gitsm', 'gitlab.com <http://gitlab.com>',
> '/gitsm/gitsm-test.git', 'git', '', OrderedDict([('branch', 'master'),
> ('protocol', 'ssh')])] comparing ['bzr', '.*', '/.*', '', '', OrderedDict()] to
> ['http', 'downloads.yoctoproject.org <http://downloads.yoctoproject.org>',
> '/mirror/sources/', '', '', OrderedDict()]
> ....
> DEBUG: Fetching
> http://downloads.yoctoproject.org/mirror/sources/git2_gitlab.com.gitsm.gitsm-test.git.tar.gz
> using command '/usr/bin/env wget -t 2 -T 30 --passive-ftp --no-check-certificate
> -P /opt/yocto/poky/thud/poky-fetch-gitsm/build/downloads
> 'http://downloads.yoctoproject.org/mirror/sources/git2_gitlab.com.gitsm.gitsm-test.git.tar.gz''
> DEBUG: Fetcher accessed the network with the command /usr/bin/env wget -t 2 -T
> 30 --passive-ftp --no-check-certificate -P
> /opt/yocto/poky/thud/poky-fetch-gitsm/build/downloads
> 'http://downloads.yoctoproject.org/mirror/sources/git2_gitlab.com.gitsm.gitsm-test.git.tar.gz'
> DEBUG: Running export PSEUDO_DISABLED=1; export
> DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-O1awmP2O9l,guid=1b80485356e847617a9c3d615be850af";
> export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh"; export
> PATH="/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/scripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot/usr/bin/crossscripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/bitbake/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/hosttools";
> export HOME="/home/stefan"; /usr/bin/env wget -t 2 -T 30 --passive-ftp
> --no-check-certificate -P /opt/yocto/poky/thud/poky-fetch-gitsm/build/downloads
> 'http://downloads.yoctoproject.org/mirror/sources/git2_gitlab.com.gitsm.gitsm-test.git.tar.gz'
> --progress=dot -v
> --2018-11-11 20:32:40--
> http://downloads.yoctoproject.org/mirror/sources/git2_gitlab.com.gitsm.gitsm-test.git.tar.gz
> Resolving downloads.yoctoproject.org <http://downloads.yoctoproject.org>
> (downloads.yoctoproject.org <http://downloads.yoctoproject.org>)... 198.145.29.63
> Connecting to downloads.yoctoproject.org <http://downloads.yoctoproject.org>
> (downloads.yoctoproject.org
> <http://downloads.yoctoproject.org>)|198.145.29.63|:80... connected.
> HTTP request sent, awaiting response... 404 Not Found
> 2018-11-11 20:32:41 ERROR 404: Not Found.
>
> DEBUG: Mirror fetch failure for url
> http://downloads.yoctoproject.org/mirror/sources/git2_gitlab.com.gitsm.gitsm-test.git.tar.gz
> (original url:
> gitsm://git@gitlab.com/gitsm/gitsm-test.git;branch=master;protocol=ssh
> <http://git@gitlab.com/gitsm/gitsm-test.git;branch=master;protocol=ssh>)
> DEBUG: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export
> DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-O1awmP2O9l,guid=1b80485356e847617a9c3d615be850af";
> export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh"; export
> PATH="/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/scripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot/usr/bin/crossscripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/bitbake/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/hosttools";
> export HOME="/home/stefan"; /usr/bin/env wget -t 2 -T 30 --passive-ftp
> --no-check-certificate -P /opt/yocto/poky/thud/poky-fetch-gitsm/build/downloads
> 'http://downloads.yoctoproject.org/mirror/sources/git2_gitlab.com.gitsm.gitsm-test.git.tar.gz'
> --progress=dot -v failed with exit code 8, output:
> --2018-11-11 20:32:40--
> http://downloads.yoctoproject.org/mirror/sources/git2_gitlab.com.gitsm.gitsm-test.git.tar.gz
> Resolving downloads.yoctoproject.org <http://downloads.yoctoproject.org>
> (downloads.yoctoproject.org <http://downloads.yoctoproject.org>)... 198.145.29.63
> Connecting to downloads.yoctoproject.org <http://downloads.yoctoproject.org>
> (downloads.yoctoproject.org
> <http://downloads.yoctoproject.org>)|198.145.29.63|:80... connected.
> HTTP request sent, awaiting response... 404 Not Found
> 2018-11-11 20:32:41 ERROR 404: Not Found.
>
>
> DEBUG: Trying Upstream
> DEBUG: Fetcher accessed the network with the command LANG=C git -c
> core.fsyncobjectfiles=0 clone --bare --mirror
> ssh://git@gitlab.com/gitsm/gitsm-test.git
> <http://git@gitlab.com/gitsm/gitsm-test.git>
> /opt/yocto/poky/thud/poky-fetch-gitsm/build/downloads/git2/gitlab.com.gitsm.gitsm-test.git
> --progress
> DEBUG: Running export PSEUDO_DISABLED=1; export
> DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-O1awmP2O9l,guid=1b80485356e847617a9c3d615be850af";
> export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh"; export
> PATH="/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/scripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot/usr/bin/crossscripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/bitbake/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/hosttools";
> export HOME="/home/stefan"; LANG=C git -c core.fsyncobjectfiles=0 clone --bare
> --mirror ssh://git@gitlab.com/gitsm/gitsm-test.git
> <http://git@gitlab.com/gitsm/gitsm-test.git>
> /opt/yocto/poky/thud/poky-fetch-gitsm/build/downloads/git2/gitlab.com.gitsm.gitsm-test.git
> --progress
> Cloning into bare repository
> '/opt/yocto/poky/thud/poky-fetch-gitsm/build/downloads/git2/gitlab.com.gitsm.gitsm-test.git'...
> remote: Enumerating objects: 12, done.
> remote: Counting objects: 100% (12/12), done.
> remote: Compressing objects: 100% (10/10), done.
> remote: Total 12 (delta 2), reused 0 (delta 0)
> Receiving objects: 100% (12/12), 7.75 KiB | 0 bytes/s, done.
> Resolving deltas: 100% (2/2), done.
> Checking connectivity... done.
> DEBUG: Running export PSEUDO_DISABLED=1; export
> DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-O1awmP2O9l,guid=1b80485356e847617a9c3d615be850af";
> export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh"; export
> PATH="/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/scripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot/usr/bin/crossscripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/bitbake/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/hosttools";
> export HOME="/home/stefan"; git -c core.fsyncobjectfiles=0 branch --contains
> 5e400c09b7aff9bfdc2aabbb2d1cd07b94fdafec --list master 2> /dev/null | wc -l
> DEBUG: Running export PSEUDO_DISABLED=1; export
> DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-O1awmP2O9l,guid=1b80485356e847617a9c3d615be850af";
> export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh"; export
> PATH="/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/scripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot/usr/bin/crossscripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/bitbake/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/hosttools";
> export HOME="/home/stefan"; git -c core.fsyncobjectfiles=0 branch --contains
> 5e400c09b7aff9bfdc2aabbb2d1cd07b94fdafec --list master 2> /dev/null | wc -l
> DEBUG: Running export PSEUDO_DISABLED=1; export
> DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-O1awmP2O9l,guid=1b80485356e847617a9c3d615be850af";
> export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh"; export
> PATH="/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/scripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot/usr/bin/crossscripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/bitbake/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/hosttools";
> export HOME="/home/stefan"; git -c core.fsyncobjectfiles=0 show
> 5e400c09b7aff9bfdc2aabbb2d1cd07b94fdafec:.gitmodules
> DEBUG: Running export PSEUDO_DISABLED=1; export
> DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-O1awmP2O9l,guid=1b80485356e847617a9c3d615be850af";
> export SSH_AUTH_SOCK="/run/user/1000/keyring/ssh"; export
> PATH="/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/scripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot/usr/bin/crossscripts:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/usr/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/sbin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/gitsm-test/1.0+gitAUTOINC+5e400c09b7-r0/recipe-sysroot-native/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/bitbake/bin:/opt/yocto/poky/thud/poky-fetch-gitsm/build/tmp/hosttools";
> export HOME="/home/stefan"; git -c core.fsyncobjectfiles=0 ls-tree -z -d
> 5e400c09b7aff9bfdc2aabbb2d1cd07b94fdafec lib
> WARNING: Failed to fetch URL
> gitsm://git@gitlab.com/gitsm/gitsm-test.git;branch=master;protocol=ssh
> <http://git@gitlab.com/gitsm/gitsm-test.git;branch=master;protocol=ssh>,
> attempting MIRRORS if available
> DEBUG: The URL: 'gitsm:Bigler/gitsm-test-submodule.git;protocol=git@gitlab.com
> <mailto:git@gitlab.com>;name=lib;bareclone=1;nocheckout=1;nobranch=1' is invalid
> and cannot be interpreted
> DEBUG: Trying MIRRORS
> DEBUG: For url ['gitsm', 'gitlab.com <http://gitlab.com>',
> '/gitsm/gitsm-test.git', 'git', '', OrderedDict([('branch', 'master'),
> ('protocol', 'ssh')])] comparing ['ftp', '.*', '/.*', '', '', OrderedDict()] to
> ['http', 'downloads.yoctoproject.org <http://downloads.yoctoproject.org>',
> '/mirror/sources/', '', '', OrderedDict()]
> ...
> DEBUG: For url ['gitsm', 'gitlab.com <http://gitlab.com>',
> '/gitsm/gitsm-test.git', 'git', '', OrderedDict([('branch', 'master'),
> ('protocol', 'ssh')])] comparing ['git', '.*', '/.*', '', '', OrderedDict()] to
> ['git', 'HOST', '/PATH', '', '', OrderedDict([('protocol', 'https')])]
> ERROR: The URL: 'gitsm:Bigler/gitsm-test-submodule.git;protocol=git@gitlab.com
> <mailto:git@gitlab.com>;name=lib;bareclone=1;nocheckout=1;nobranch=1' is invalid
> and cannot be interpreted
> ERROR: Fetcher failure for URL:
> 'gitsm://git@gitlab.com/gitsm/gitsm-test.git;branch=master;protocol=ssh
> <http://git@gitlab.com/gitsm/gitsm-test.git;branch=master;protocol=ssh>'. Unable
> to fetch URL from any source.
> DEBUG: Python function base_do_fetch finished
> DEBUG: Python function do_fetch finished
> ERROR: Function failed: base_do_fetch
>
>
>
>
^ permalink raw reply [flat|nested] 3+ messages in thread
* gitsm fetching fails
@ 2018-11-15 8:27 Stefan Bigler
0 siblings, 0 replies; 3+ messages in thread
From: Stefan Bigler @ 2018-11-15 8:27 UTC (permalink / raw)
To: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 4137 bytes --]
Hi Mark
Thank you for the explanation and sorry for the late response from my side.
With your help I was successful.
> On 11/11/18 3:04 PM, Stefan Bigler wrote:
> > Hi
> > I'm updating from rocko to thud and recognized, that my projects with
> > git-submodules cannot be fetched anymore.
> > I tried to debug the fetch task, but I failed. I did a small example
you should
> > have access and it should be easy to reproduce, the repo should be
public.
> > The recipe works fine with rocko but fails with the latest master
branch of poky.
> > I'm using gitlab for my repos. When I use https it works also with
thud, only
> > when *ssh* is used then it fails.
> > Below you find the recipe, the console output and the important part of
the
> > log.do_fetch
> > I hope somebody can give me a hint.
> The previous fetcher handed off everything to git and did not do all of
the
> validation and network caching/fetching like it should have. This hid a
lot of
> problems and prevented the system from being able to use declared mirror
resources.
>
> The new fetcher iterates over the items and calls the regular git fetcher
to
> pull down the various components. This ensures that the download
directory and
> mirrors are used appropriately.
>
> Below the message:
> > ERROR: gitsm-test-1.0+gitAUTOINC+5e400c09b7-r0 do_fetch: The URL:
> > 'gitsm:Bigler/gitsm-test-submodule.git;protocol=git at gitlab.com
> indicates that the submodule is not defined properly. If this is legal,
then it
> may require some kind of a translation that does not exist.
> See: http://git.openembedded.org/bitbake/tree/lib/bb/fetch2/gitsm.py
> Look at lines 86 though 96. This is where the submodule URLs are
constructed.
> The current code is expected the submodules to be defined with URIs, NOT
SSH
> 'strings'.
> i.e.:
> http://user@host/path/file
> ssh://user@host/path/file
> NOT
> user at host:path/file
> It appears that this submodule reference is using the second format,
instead of
> the actual URI format. gitlab must be doing something different if it
works on
> http but not ssh, since typically the .gitmodules are declared in a
singular
> format by the author of the repository and don't change, even if the fetch
> method of the initial repository changes.
If I change the .gitmodules to
[submodule "lib"]
path = lib
- url = git@gitlab.com:gitsm/gitsm-test-submodule.git
+ url = ssh://git@gitlab.com/gitsm/gitsm-test-submodule.git
then it works as expected
> Since you have a way to reproduce this, I'd suggest working through that
small
> section of code and trying to determine if it's an invalid URI (SSH
style) and
> then translating it to a URI for this processing. (If you can make a
reproducer
> publicly available, then I can try to fix it instead.)
The repos are available here
https://gitlab.com/gitsm/gitsm-test/tree/master
> Looking at the git.py, it says the following protocols are supported:
> "git", "http", "https", "file", "ssh" and "rsync"
> So checking the protocol field for one of those, failing it find it
should go
> into a translation step most likely. The result (I would assume) will
either be
> file or ssh. If there is a ':', then most likely it's an ssh.. and the
rules to
> make this into a proper URI should be fairly simple.
With the patch below I *hardcoded* the gitlab format and it works as well,
but of course all other will fail.
I do not know if there is a good way to detect if secondary format is used
and then do the conversion.
Anyway, I can live with the standard format and I can change it in my repos.
bitbake/lib/bb/fetch2/gitsm.py
@@ -88,8 +88,17 @@ class GitSM(Git):
module_hash = module_hash.split()[2]
# Build new SRC_URI
- proto = uris[module].split(':', 1)[0]
- url = uris[module].replace('%s:' % proto, 'gitsm:', 1)
+ proto = "ssh"
+ url = "gitsm://"
+ url += uris[module].replace(':', '/', 1)
+
url += ';protocol=%s' % proto
-- Stefan
[-- Attachment #2: Type: text/html, Size: 5021 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2018-11-15 8:27 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-11-11 21:04 gitsm fetching fails Stefan Bigler
2018-11-13 15:26 ` Mark Hatle
2018-11-15 8:27 Stefan Bigler
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.