gitsm Fetcher update

gitsm Fetcher update

[Jens Rehsack | External Miltenyi Biotec]

Hi Mark,

after your last update to gitsm fetcher, it seems it cannot handle submodules not using the master branch correctly:

--- BEGIN INITIAL ERROR
WARNING: automacs-ui-git-r0 do_fetch: Failed to fetch URL gitsm://mgegit.miltenyibiotec.de:7999/umsa/automacs-ui.git;protocol=ssh;branch=develop;rev=fddb26a33d3f949d0863cffc5306146ab8e5a9a8, attempting MIRRORS if available
ERROR: automacs-ui-git-r0 do_fetch: Fetcher failure for URL: 'gitsm://git@mgegit.miltenyibiotec.de:7999/cpsm/cmake-modules.git;protocol=ssh;name=cmake/modules;qbareclone=1;nocheckout=1'. Unable to fetch URL from any source.
ERROR: automacs-ui-git-r0 do_fetch: Fetcher failure for URL: 'gitsm://mgegit.miltenyibiotec.de:7999/umsa/automacs-ui.git;protocol=ssh;branch=develop;rev=fddb26a33d3f949d0863cffc5306146ab8e5a9a8'. Unable to fetch URL from any source.
ERROR: automacs-ui-git-r0 do_fetch: Function failed: base_do_fetch
ERROR: Logfile of failure stored in: /home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/temp/log.do_fetch.5452
ERROR: Task (/home/sno/gpw-community-bsp/sources/meta-miltenyi/recipes-ui/automacs-ui/automacs-ui_git.bb:do_fetch) failed with exit code '1'

DEBUG: Running export PSEUDO_DISABLED=1; export GIT_PROXY_COMMAND="oe-git-proxy"; export NO_PROXY="*.miltenyibiotec.de,*.miltenyibiotec.com"; export https_proxy="http://proxy2:8080/"; export http_proxy="http://proxy2:8080/"; export PATH="/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/sno/gpw-community-bsp/sources/poky/scripts:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot/opt/miltenyi/bin/crossscripts:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/usr/sbin:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/usr/bin:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/sbin:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/bin:/home/sno/gpw-community-bsp/sources/poky/bitbake/bin:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/hosttools"; export HOME="/home/sno"; git -c core.fsyncobjectfiles=0 branch --contains c6c052603e2e3ea334c5a71e3584855c5535995f --list master 2> /dev/null | wc -l
--- END INITIAL ERROR

I cloned the appropriate project locally to see what's going on:

sno@yoctopoc:~/prj/automacs-ui/cmake/modules$ git -c core.fsyncobjectfiles=0 branch --contains c6c052603e2e3ea334c5a71e3584855c5535995f --list
* (HEAD detached at c6c0526)
  develop
sno@yoctopoc:~/prj/automacs-ui/cmake/modules$ git -c core.fsyncobjectfiles=0 branch --contains c6c052603e2e3ea334c5a71e3584855c5535995f --list master
sno@yoctopoc:~/prj/automacs-ui/cmake/modules$

The I tried to reverse engineer the gitsm fetcher and added:

poky$ git diff
diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py
index 51f8d0e68c..221af934c3 100644
--- a/bitbake/lib/bb/fetch2/gitsm.py
+++ b/bitbake/lib/bb/fetch2/gitsm.py
@@ -79,6 +79,7 @@ class GitSM(Git):
             url += ';protocol=%s' % proto
             url += ";name=%s" % module
             url += ";qbareclone=1;nocheckout=1"
+            url += ";nobranch=1"

             ld = d.createCopy()
             # Not necessary to set SRC_URI, since we're passing the URI to

which results into 

--- BEGIN FOLLOW UP ERROR
ERROR: automacs-ui-git-r0 do_unpack: Fetcher failure: Submodule cmake/modules does not exist in /home/sno/gpw-community-bsp/mops-yocto-platform/downloads/git2/mgegit.miltenyibiotec.de.7999.umsa.automacs-ui.git/modules/cmake/modules or /home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/git/.git/modules/cmake/modules.
ERROR: automacs-ui-git-r0 do_unpack: Function failed: base_do_unpack
ERROR: Logfile of failure stored in: /home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/temp/log.do_unpack.6551
ERROR: Task (/home/sno/gpw-community-bsp/sources/meta-miltenyi/recipes-ui/automacs-ui/automacs-ui_git.bb:do_unpack) failed with exit code '1'
--- END FOLLOW UP ERROR

because it uses the submodule names as archive name:

$ tar xzf downloads/git2_mgegit.miltenyibiotec.de.7999.<TAB>
git2_mgegit.miltenyibiotec.de.7999.cpsm.cmake-modules.git.tar.gz
git2_mgegit.miltenyibiotec.de.7999.cpsm.mcs-lib.git.tar.gz
git2_mgegit.miltenyibiotec.de.7999.umsa.corert.git.tar.gz
git2_mgegit.miltenyibiotec.de.7999.umsa.protos.git.tar.gz

Cheers,
Jens

Re: gitsm Fetcher update

[Mark Hatle]

On 10/1/18 3:52 AM, Jens Rehsack | External Miltenyi Biotec wrote:
> Hi Mark,
> 
> after your last update to gitsm fetcher, it seems it cannot handle submodules not using the master branch correctly:

I'm not sure I understand below what is going wrong.

Does it not checkout the primary repository in the correct branch, or does this
get checked out -- and it's the submodules that get 'confused' between the
pointed to commit and the branches?

The system recursively processes the git submodules, while interrogating it's
way through the system.  If the failure is on a submode (not the main
repository) it would be something in this block...

http://git.openembedded.org/bitbake/tree/lib/bb/fetch2/gitsm.py

47 - 108

Looking at the code, line 81.

            # Build new SRC_URI
            proto = uris[module].split(':', 1)[0]
            url = uris[module].replace('%s:' % proto, 'gitsm:', 1)
            url += ';protocol=%s' % proto
            url += ";name=%s" % module
            url += ";qbareclone=1;nocheckout=1"


It looks like a 'q' accidentally made it into the patch.  Can you remove this
and try it?   This is where the system constructs SRC_URI for the recursive
(submodule) download.  The next block defines the SRCREV and related items and
re-calls fetch.

Nowhere in there do we call or set anything related to the branches.  Maybe
"nobranch=1" needs to be added to the generated SRC_URI?

If you can give me a recipe that used to work through 'do_unpack', then I should
be able to attempt to replicate it locally and work through the problems if the
above isn't enough of a place to look.

(It also shows a missing test case, one that we need to add -- once we
understand what part and why it's failing.)

--Mark

> --- BEGIN INITIAL ERROR
> WARNING: automacs-ui-git-r0 do_fetch: Failed to fetch URL gitsm://mgegit.miltenyibiotec.de:7999/umsa/automacs-ui.git;protocol=ssh;branch=develop;rev=fddb26a33d3f949d0863cffc5306146ab8e5a9a8, attempting MIRRORS if available
> ERROR: automacs-ui-git-r0 do_fetch: Fetcher failure for URL: 'gitsm://git@mgegit.miltenyibiotec.de:7999/cpsm/cmake-modules.git;protocol=ssh;name=cmake/modules;qbareclone=1;nocheckout=1'. Unable to fetch URL from any source.
> ERROR: automacs-ui-git-r0 do_fetch: Fetcher failure for URL: 'gitsm://mgegit.miltenyibiotec.de:7999/umsa/automacs-ui.git;protocol=ssh;branch=develop;rev=fddb26a33d3f949d0863cffc5306146ab8e5a9a8'. Unable to fetch URL from any source.
> ERROR: automacs-ui-git-r0 do_fetch: Function failed: base_do_fetch
> ERROR: Logfile of failure stored in: /home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/temp/log.do_fetch.5452
> ERROR: Task (/home/sno/gpw-community-bsp/sources/meta-miltenyi/recipes-ui/automacs-ui/automacs-ui_git.bb:do_fetch) failed with exit code '1'
> 
> DEBUG: Running export PSEUDO_DISABLED=1; export GIT_PROXY_COMMAND="oe-git-proxy"; export NO_PROXY="*.miltenyibiotec.de,*.miltenyibiotec.com"; export https_proxy="http://proxy2:8080/"; export http_proxy="http://proxy2:8080/"; export PATH="/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/sno/gpw-community-bsp/sources/poky/scripts:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot/opt/miltenyi/bin/crossscripts:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/usr/sbin:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/usr/bin:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/sbin:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/bin:/home/sno/gpw-community-bsp/sources/poky/bitbake/bin:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/hosttools"; export HOME="/home/sno"; git -c core.fsyncobjectfiles=0 branch --contains c6c052603e2e3ea334c5a71e3584855c5535995f --list master 2> /dev/null | wc -l
> --- END INITIAL ERROR
> 
> I cloned the appropriate project locally to see what's going on:
> 
> sno@yoctopoc:~/prj/automacs-ui/cmake/modules$ git -c core.fsyncobjectfiles=0 branch --contains c6c052603e2e3ea334c5a71e3584855c5535995f --list
> * (HEAD detached at c6c0526)
>   develop
> sno@yoctopoc:~/prj/automacs-ui/cmake/modules$ git -c core.fsyncobjectfiles=0 branch --contains c6c052603e2e3ea334c5a71e3584855c5535995f --list master
> sno@yoctopoc:~/prj/automacs-ui/cmake/modules$
> 
> The I tried to reverse engineer the gitsm fetcher and added:
> 
> poky$ git diff
> diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py
> index 51f8d0e68c..221af934c3 100644
> --- a/bitbake/lib/bb/fetch2/gitsm.py
> +++ b/bitbake/lib/bb/fetch2/gitsm.py
> @@ -79,6 +79,7 @@ class GitSM(Git):
>              url += ';protocol=%s' % proto
>              url += ";name=%s" % module
>              url += ";qbareclone=1;nocheckout=1"
> +            url += ";nobranch=1"
> 
>              ld = d.createCopy()
>              # Not necessary to set SRC_URI, since we're passing the URI to
> 
> which results into 
> 
> --- BEGIN FOLLOW UP ERROR
> ERROR: automacs-ui-git-r0 do_unpack: Fetcher failure: Submodule cmake/modules does not exist in /home/sno/gpw-community-bsp/mops-yocto-platform/downloads/git2/mgegit.miltenyibiotec.de.7999.umsa.automacs-ui.git/modules/cmake/modules or /home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/git/.git/modules/cmake/modules.
> ERROR: automacs-ui-git-r0 do_unpack: Function failed: base_do_unpack
> ERROR: Logfile of failure stored in: /home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/temp/log.do_unpack.6551
> ERROR: Task (/home/sno/gpw-community-bsp/sources/meta-miltenyi/recipes-ui/automacs-ui/automacs-ui_git.bb:do_unpack) failed with exit code '1'
> --- END FOLLOW UP ERROR
> 
> because it uses the submodule names as archive name:
> 
> $ tar xzf downloads/git2_mgegit.miltenyibiotec.de.7999.<TAB>
> git2_mgegit.miltenyibiotec.de.7999.cpsm.cmake-modules.git.tar.gz
> git2_mgegit.miltenyibiotec.de.7999.cpsm.mcs-lib.git.tar.gz
> git2_mgegit.miltenyibiotec.de.7999.umsa.corert.git.tar.gz
> git2_mgegit.miltenyibiotec.de.7999.umsa.protos.git.tar.gz
> 
> Cheers,
> Jens
> 

Re: gitsm Fetcher update

[Jens Rehsack]

[-- Attachment #1.1: Type: text/plain, Size: 7632 bytes --]



> Am 01.10.2018 um 15:34 schrieb Mark Hatle <mark.hatle@windriver.com>:
> 
> On 10/1/18 3:52 AM, Jens Rehsack | External Miltenyi Biotec wrote:
>> Hi Mark,
>> 
>> after your last update to gitsm fetcher, it seems it cannot handle submodules not using the master branch correctly:
> 
> I'm not sure I understand below what is going wrong.
> 
> Does it not checkout the primary repository in the correct branch, or does this
> get checked out -- and it's the submodules that get 'confused' between the
> pointed to commit and the branches?

The primary repository is - as far as I can see from the logs - checked out correctly.
It just tries to apply the branch 'master' to the submodule. The primary repository has
the branch 'develop' - what makes me a bit curios.

> The system recursively processes the git submodules, while interrogating it's
> way through the system.  If the failure is on a submode (not the main
> repository) it would be something in this block...
> 
> http://git.openembedded.org/bitbake/tree/lib/bb/fetch2/gitsm.py
> 
> 47 - 108
> 
> Looking at the code, line 81.
> 
>            # Build new SRC_URI
>            proto = uris[module].split(':', 1)[0]
>            url = uris[module].replace('%s:' % proto, 'gitsm:', 1)
>            url += ';protocol=%s' % proto
>            url += ";name=%s" % module
>            url += ";qbareclone=1;nocheckout=1"
> 
> 
> It looks like a 'q' accidentally made it into the patch.  Can you remove this
> and try it?   This is where the system constructs SRC_URI for the recursive
> (submodule) download.  The next block defines the SRCREV and related items and
> re-calls fetch.

Will do tomorrow when I'm back in company and report back.

> Nowhere in there do we call or set anything related to the branches.  Maybe
> "nobranch=1" needs to be added to the generated SRC_URI?

I tried that and this ended up in wrong names for the archive files created
for the download cache (see below).

> If you can give me a recipe that used to work through 'do_unpack', then I should
> be able to attempt to replicate it locally and work through the problems if the
> above isn't enough of a place to look.

Unfortunately we have only proprietary recipes using that. I can try create
one for a perl module I maintain, if you like.
You will need meta-cpan for that - can you confirm that meta-cpan works
for you in general (https://github.com/meta-cpan <https://github.com/meta-cpan>)?

> (It also shows a missing test case, one that we need to add -- once we
> understand what part and why it's failing.)
> 
> --Mark

Best regards,
Jens

>> --- BEGIN INITIAL ERROR
>> WARNING: automacs-ui-git-r0 do_fetch: Failed to fetch URL gitsm://mgegit.miltenyibiotec.de:7999/umsa/automacs-ui.git;protocol=ssh;branch=develop;rev=fddb26a33d3f949d0863cffc5306146ab8e5a9a8, attempting MIRRORS if available
>> ERROR: automacs-ui-git-r0 do_fetch: Fetcher failure for URL: 'gitsm://git@mgegit.miltenyibiotec.de:7999/cpsm/cmake-modules.git;protocol=ssh;name=cmake/modules;qbareclone=1;nocheckout=1'. Unable to fetch URL from any source.
>> ERROR: automacs-ui-git-r0 do_fetch: Fetcher failure for URL: 'gitsm://mgegit.miltenyibiotec.de:7999/umsa/automacs-ui.git;protocol=ssh;branch=develop;rev=fddb26a33d3f949d0863cffc5306146ab8e5a9a8'. Unable to fetch URL from any source.
>> ERROR: automacs-ui-git-r0 do_fetch: Function failed: base_do_fetch
>> ERROR: Logfile of failure stored in: /home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/temp/log.do_fetch.5452
>> ERROR: Task (/home/sno/gpw-community-bsp/sources/meta-miltenyi/recipes-ui/automacs-ui/automacs-ui_git.bb:do_fetch) failed with exit code '1'
>> 
>> DEBUG: Running export PSEUDO_DISABLED=1; export GIT_PROXY_COMMAND="oe-git-proxy"; export NO_PROXY="*.miltenyibiotec.de,*.miltenyibiotec.com"; export https_proxy="http://proxy2:8080/"; export http_proxy="http://proxy2:8080/"; export PATH="/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/sno/gpw-community-bsp/sources/poky/scripts:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot/opt/miltenyi/bin/crossscripts:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/usr/sbin:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/usr/bin:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/sbin:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/recipe-sysroot-native/bin:/home/sno/gpw-community-bsp/sources/poky/bitbake/bin:/home/sno/gpw-community-bsp/mops-yocto-platform/tmp/hosttools"; export HOME="/home/sno"; git -c core.fsyncobjectfiles=0 branch --contains c6c052603e2e3ea334c5a71e3584855c5535995f --list master 2> /dev/null | wc -l
>> --- END INITIAL ERROR
>> 
>> I cloned the appropriate project locally to see what's going on:
>> 
>> sno@yoctopoc:~/prj/automacs-ui/cmake/modules$ git -c core.fsyncobjectfiles=0 branch --contains c6c052603e2e3ea334c5a71e3584855c5535995f --list
>> * (HEAD detached at c6c0526)
>>  develop
>> sno@yoctopoc:~/prj/automacs-ui/cmake/modules$ git -c core.fsyncobjectfiles=0 branch --contains c6c052603e2e3ea334c5a71e3584855c5535995f --list master
>> sno@yoctopoc:~/prj/automacs-ui/cmake/modules$
>> 
>> The I tried to reverse engineer the gitsm fetcher and added:
>> 
>> poky$ git diff
>> diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py
>> index 51f8d0e68c..221af934c3 100644
>> --- a/bitbake/lib/bb/fetch2/gitsm.py
>> +++ b/bitbake/lib/bb/fetch2/gitsm.py
>> @@ -79,6 +79,7 @@ class GitSM(Git):
>>             url += ';protocol=%s' % proto
>>             url += ";name=%s" % module
>>             url += ";qbareclone=1;nocheckout=1"
>> +            url += ";nobranch=1"
>> 
>>             ld = d.createCopy()
>>             # Not necessary to set SRC_URI, since we're passing the URI to
>> 
>> which results into
>> 
>> --- BEGIN FOLLOW UP ERROR
>> ERROR: automacs-ui-git-r0 do_unpack: Fetcher failure: Submodule cmake/modules does not exist in /home/sno/gpw-community-bsp/mops-yocto-platform/downloads/git2/mgegit.miltenyibiotec.de.7999.umsa.automacs-ui.git/modules/cmake/modules or /home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/git/.git/modules/cmake/modules.
>> ERROR: automacs-ui-git-r0 do_unpack: Function failed: base_do_unpack
>> ERROR: Logfile of failure stored in: /home/sno/gpw-community-bsp/mops-yocto-platform/tmp/work/core2-64-poky-linux/automacs-ui/git-r0/temp/log.do_unpack.6551
>> ERROR: Task (/home/sno/gpw-community-bsp/sources/meta-miltenyi/recipes-ui/automacs-ui/automacs-ui_git.bb:do_unpack) failed with exit code '1'
>> --- END FOLLOW UP ERROR
>> 
>> because it uses the submodule names as archive name:
>> 
>> $ tar xzf downloads/git2_mgegit.miltenyibiotec.de.7999.<TAB>
>> git2_mgegit.miltenyibiotec.de.7999.cpsm.cmake-modules.git.tar.gz
>> git2_mgegit.miltenyibiotec.de.7999.cpsm.mcs-lib.git.tar.gz
>> git2_mgegit.miltenyibiotec.de.7999.umsa.corert.git.tar.gz
>> git2_mgegit.miltenyibiotec.de.7999.umsa.protos.git.tar.gz
>> 
>> Cheers,
>> Jens
>> 
> 

--
Jens Rehsack - rehsack@gmail.com


[-- Attachment #1.2: Type: text/html, Size: 12302 bytes --]

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 849 bytes --]