Using gitsm fetcher and PREMIRROR

Using gitsm fetcher and PREMIRROR

[Konrad Scherer]

Hello,

I recently found a strange problem involving the gitsm fetcher and our PREMIRRORS and I was able to 
reproduce it using poky.

 > git clone git://git.yoctoproject.org/poky poky1
 > cd poky1
 > . oe-init-build-env
 > bitbake ovmf

The ovmf recipe has git submodules inside git submodules. This step works fine.

I then exposed the downloads directory over http (http://<host>/downloads) and the downloads/git2 
directory over http with git-http-backend enabled (http://<host>/git).

I noticed that there are shallow clone tarballs in downloads that match the git repos in 
downloads/git2. For example:

downloads/git2/boringssl.googlesource.com.boringssl/
downloads/git2_boringssl.googlesource.com.boringssl.tar.gz

Since these are duplicates of the git repos, I deleted the tar.gz files

 > rm -f downloads/git2_*

<new shell>

 > git clone git://git.yoctoproject.org/poky poky2
 > cd poky2
 > . oe-init-build-env
 > cat >> conf/local.conf <<EOF
WRS_MIRROR_HOST = "<host>"
BB_ALLOWED_NETWORKS = "${WRS_MIRROR_HOST}"

BB_NO_NETWORK = '0'
BB_FETCH_PREMIRRORONLY = '1'

PREMIRRORS_append = " \
      .*://.*/.* http://${WRS_MIRROR_HOST}/downloads/ \n \
      git://.*/.* git://${WRS_MIRROR_HOST}/git/MIRRORNAME;protocol=http \n \
      gitsm://.*/.* git://${WRS_MIRROR_HOST}/git/MIRRORNAME;protocol=http \n \
"
CONNECTIVITY_CHECK_URIS = ""
EOF
 > bitbake ovmf

This fails with:

ERROR: ovmf-edk2-stable201905-r0 do_unpack: gitsm: submodule unpack failed: UnpackError Unpack 
failure for URL: 
'gitsm://github.com/openssl/openssl;protocol=https;name=CryptoPkg/Library/OpensslLib/openssl;subpath=CryptoPkg/Library/OpensslLib/openssl;bareclone=1;nobranch=1'. 
No up to date source found: clone directory not available or not up to date: 
/ala-lpggp22/kscherer/gitsm/poky2/build/downloads/git2/github.com.openssl.openssl; shallow clone not 
enabled

If I leave the shallow clone tarballs in the PREMIRROR the build succeeds. The other recipes that 
use git are fetched properly. The logs don't have any warnings before the error and I verified that 
the objects are indeed present in the repo.

Are my PREMIRROR settings correct? Is this a bug in the gitsm fetcher?

Any help and insight appreciated.

-- 
Konrad Scherer, MTS, Linux Products Group, Wind River

Re: Using gitsm fetcher and PREMIRROR

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 3112 bytes --]

Don’t have time for a full reply at the moment, I’ll take a look in the morning, but I did want to note that git2_boringssl.googlesource.com.boringssl.tar.gz is a full git mirror tarball as generated by BB_GENERATE_MIRROR_TARBALLS, and the premirrors fetching will download and use them if you want to skip the git-http-backend approach entirely. They are *not* shallow in any way. Only gitshallow_*.tar.gz are shallow tarballs.
On Oct 25, 2019, 1:39 PM -0700, Konrad Scherer <Konrad.Scherer@windriver.com>, wrote:
> Hello,
>
> I recently found a strange problem involving the gitsm fetcher and our PREMIRRORS and I was able to
> reproduce it using poky.
>
> > git clone git://git.yoctoproject.org/poky poky1
> > cd poky1
> > . oe-init-build-env
> > bitbake ovmf
>
> The ovmf recipe has git submodules inside git submodules. This step works fine.
>
> I then exposed the downloads directory over http (http://<host>/downloads) and the downloads/git2
> directory over http with git-http-backend enabled (http://<host>/git).
>
> I noticed that there are shallow clone tarballs in downloads that match the git repos in
> downloads/git2. For example:
>
> downloads/git2/boringssl.googlesource.com.boringssl/
> downloads/git2_boringssl.googlesource.com.boringssl.tar.gz
>
> Since these are duplicates of the git repos, I deleted the tar.gz files
>
> > rm -f downloads/git2_*
>
> <new shell>
>
> > git clone git://git.yoctoproject.org/poky poky2
> > cd poky2
> > . oe-init-build-env
> > cat >> conf/local.conf <<EOF
> WRS_MIRROR_HOST = "<host>"
> BB_ALLOWED_NETWORKS = "${WRS_MIRROR_HOST}"
>
> BB_NO_NETWORK = '0'
> BB_FETCH_PREMIRRORONLY = '1'
>
> PREMIRRORS_append = " \
> .*://.*/.* http://${WRS_MIRROR_HOST}/downloads/ \n \
> git://.*/.* git://${WRS_MIRROR_HOST}/git/MIRRORNAME;protocol=http \n \
> gitsm://.*/.* git://${WRS_MIRROR_HOST}/git/MIRRORNAME;protocol=http \n \
> "
> CONNECTIVITY_CHECK_URIS = ""
> EOF
> > bitbake ovmf
>
> This fails with:
>
> ERROR: ovmf-edk2-stable201905-r0 do_unpack: gitsm: submodule unpack failed: UnpackError Unpack
> failure for URL:
> 'gitsm://github.com/openssl/openssl;protocol=https;name=CryptoPkg/Library/OpensslLib/openssl;subpath=CryptoPkg/Library/OpensslLib/openssl;bareclone=1;nobranch=1'.
> No up to date source found: clone directory not available or not up to date:
> /ala-lpggp22/kscherer/gitsm/poky2/build/downloads/git2/github.com.openssl.openssl; shallow clone not
> enabled
>
> If I leave the shallow clone tarballs in the PREMIRROR the build succeeds. The other recipes that
> use git are fetched properly. The logs don't have any warnings before the error and I verified that
> the objects are indeed present in the repo.
>
> Are my PREMIRROR settings correct? Is this a bug in the gitsm fetcher?
>
> Any help and insight appreciated.
>
> --
> Konrad Scherer, MTS, Linux Products Group, Wind River
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel

[-- Attachment #2: Type: text/html, Size: 4325 bytes --]

Re: Using gitsm fetcher and PREMIRROR

[Konrad Scherer]

On 10/27/19 10:34 PM, Christopher Larson wrote:
> Don’t have time for a full reply at the moment, I’ll take a look in the morning, but I did want to note 
> that git2_boringssl.googlesource.com.boringssl.tar.gz is a full git mirror tarball as generated by 
> BB_GENERATE_MIRROR_TARBALLS, and the premirrors fetching will download and use them if you want to skip the 
> git-http-backend approach entirely. They are *not* shallow in any way. Only gitshallow_*.tar.gz are shallow tarballs.

Thanks for the response and the clarification. I got confused by the mention of shallow clones in the error message.

Any idea why the gitsm fetcher seems to depend on the mirror tarballs and doesn't recreate them from the available git 
repository?

Thank you!

-- 
Konrad Scherer, MTS, Linux Products Group, Wind River