git submodule fetcher regression

git submodule fetcher regression

[Stefan Agner]

[resend, this time with me being subscriber of bitbake-devel]

Hi,

It seems that 9c6b39adf978 ("fetch2/gitsm.py: Rework the git submodule fetcher") broke git submodules in some cases. My particular case is aktualizr from meta-updater (https://github.com/advancedtelematic/meta-updater). Fetching aktualizr reliably fails for me now:

ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_unpack: Fetcher failure: Submodule tests/tuf-test-vectors does not exist in /home/ags/oe/downloads/git2/github.com.advancedtelematic.aktualizr/modules/tests/tuf-test-vectors or
/home/ags/oe/build-colibri-imx7/tmp-oe/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/git/.git/modules/tests/tuf-test-vectors.                         
ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_unpack: Function failed: base_do_unpack
ERROR: Logfile of failure stored in: /home/ags/oe/build-colibri-imx7/tmp-oe/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/temp/log.do_unpack.35143
ERROR: Task (/home/ags/oe/build-colibri-imx7/conf/../../layers/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb:do_unpack) failed with exit code '1'

Any idea what that could be?

--
Stefan

Re: git submodule fetcher regression

[Bonnans, Laurent]

Hello Stefan,

On 10/04/2018 06:08 PM, Stefan Agner wrote:
> Hi,
>
> It seems that 9c6b39adf978 ("fetch2/gitsm.py: Rework the git submodule fetcher") broke git submodules in some cases. My particular case is aktualizr from meta-updater (https://github.com/advancedtelematic/meta-updater). Fetching aktualizr reliably fails for me now:
>
> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_unpack: Fetcher failure: Submodule tests/tuf-test-vectors does not exist in /home/ags/oe/downloads/git2/github.com.advancedtelematic.aktualizr/modules/tests/tuf-test-vectors or
> /home/ags/oe/build-colibri-imx7/tmp-oe/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/git/.git/modules/tests/tuf-test-vectors.
> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_unpack: Function failed: base_do_unpack
> ERROR: Logfile of failure stored in: /home/ags/oe/build-colibri-imx7/tmp-oe/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/temp/log.do_unpack.35143
> ERROR: Task (/home/ags/oe/build-colibri-imx7/conf/../../layers/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb:do_unpack) failed with exit code '1'
>
> Any idea what that could be?
>
> --
> Stefan

We also noticed the problem and supplied a fix to the bitbake-devel 
mailing list (actually, there were two separate issues in our case). The 
commits are currently sitting in master-next branches of bitbake and 
poky and should, in all likelihood, move to master soon.

Laurent

Re: git submodule fetcher regression

[Stefan Agner]

Hi Laurent,

On 04.10.2018 18:28, Bonnans, Laurent wrote:
> Hello Stefan,
> 
> On 10/04/2018 06:08 PM, Stefan Agner wrote:
>> Hi,
>>
>> It seems that 9c6b39adf978 ("fetch2/gitsm.py: Rework the git submodule fetcher") broke git submodules in some cases. My particular case is aktualizr from meta-updater (https://github.com/advancedtelematic/meta-updater). Fetching aktualizr reliably fails for me now:
>>
>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_unpack: Fetcher failure: Submodule tests/tuf-test-vectors does not exist in /home/ags/oe/downloads/git2/github.com.advancedtelematic.aktualizr/modules/tests/tuf-test-vectors or
>> /home/ags/oe/build-colibri-imx7/tmp-oe/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/git/.git/modules/tests/tuf-test-vectors.
>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_unpack: Function failed: base_do_unpack
>> ERROR: Logfile of failure stored in: /home/ags/oe/build-colibri-imx7/tmp-oe/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/temp/log.do_unpack.35143
>> ERROR: Task (/home/ags/oe/build-colibri-imx7/conf/../../layers/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb:do_unpack) failed with exit code '1'
>>
>> Any idea what that could be?
>>
>> --
>> Stefan
> 
> We also noticed the problem and supplied a fix to the bitbake-devel 
> mailing list (actually, there were two separate issues in our case). The 
> commits are currently sitting in master-next branches of bitbake and 
> poky and should, in all likelihood, move to master soon.

I guess you refer to:
fetch2/gitsm: checkout submodules recursively
fetch2/gitsm: handle submodules nested inside subdirs

WARNING: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Failed to fetch URL gitsm://github.com/cajun-rat/HdrHistogram_c;protocol=https;name=third_party/HdrHistogram_c;bareclone=1;nocheckout=1, attempting MIRRORS if available
ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure: Unable to find revision 0b778a98c90ce55285cb1cb6fe50e2b25dfd57ae in branch master even from upstream
WARNING: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Failed to fetch URL gitsm://github.com/advancedtelematic/aktualizr;branch=master, attempting MIRRORS if available
ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure for URL: 'gitsm://github.com/cajun-rat/HdrHistogram_c;protocol=https;name=third_party/HdrHistogram_c;bareclone=1;nocheckout=1'. Unable to fetch URL from any source.
ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure for URL: 'gitsm://github.com/advancedtelematic/aktualizr;branch=master'. Unable to fetch URL from any source.
ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Function failed: base_do_fetch
ERROR: Logfile of failure stored in: /home/ags/oe/build-colibri-imx7/tmp-tordy/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/temp/log.do_fetch.41673
ERROR: Task (/home/ags/oe/build-colibri-imx7/conf/../../layers/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb:do_fetch) failed with exit code '1'

(I am using meta-updater c4902394cc6673b07ad5ee87570e02afbdf7b022 and bitbake master-next as of now)

I did use -c cleanall before, which hopefully should be sufficient?

--
Stefan

Re: git submodule fetcher regression

[Mark Hatle]

On 10/4/18 11:08 AM, Stefan Agner wrote:
> [resend, this time with me being subscriber of bitbake-devel]
> 
> Hi,
> 
> It seems that 9c6b39adf978 ("fetch2/gitsm.py: Rework the git submodule fetcher") broke git submodules in some cases. My particular case is aktualizr from meta-updater (https://github.com/advancedtelematic/meta-updater). Fetching aktualizr reliably fails for me now:
> 
> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_unpack: Fetcher failure: Submodule tests/tuf-test-vectors does not exist in /home/ags/oe/downloads/git2/github.com.advancedtelematic.aktualizr/modules/tests/tuf-test-vectors or
> /home/ags/oe/build-colibri-imx7/tmp-oe/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/git/.git/modules/tests/tuf-test-vectors.                         
> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_unpack: Function failed: base_do_unpack
> ERROR: Logfile of failure stored in: /home/ags/oe/build-colibri-imx7/tmp-oe/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/temp/log.do_unpack.35143
> ERROR: Task (/home/ags/oe/build-colibri-imx7/conf/../../layers/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb:do_unpack) failed with exit code '1'
> 
> Any idea what that could be?

See the mailing list archives (or master-next branch).  There are a small series
of pending patches.  Please try these.

3 from Laurent Bonnans
1 from me

--Mark

> --
> Stefan
> 

Re: git submodule fetcher regression

[Mark Hatle]

On 10/4/18 11:45 AM, Stefan Agner wrote:
> Hi Laurent,
> 
> On 04.10.2018 18:28, Bonnans, Laurent wrote:
>> Hello Stefan,
>>
>> On 10/04/2018 06:08 PM, Stefan Agner wrote:
>>> Hi,
>>>
>>> It seems that 9c6b39adf978 ("fetch2/gitsm.py: Rework the git submodule fetcher") broke git submodules in some cases. My particular case is aktualizr from meta-updater (https://github.com/advancedtelematic/meta-updater). Fetching aktualizr reliably fails for me now:
>>>
>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_unpack: Fetcher failure: Submodule tests/tuf-test-vectors does not exist in /home/ags/oe/downloads/git2/github.com.advancedtelematic.aktualizr/modules/tests/tuf-test-vectors or
>>> /home/ags/oe/build-colibri-imx7/tmp-oe/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/git/.git/modules/tests/tuf-test-vectors.
>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_unpack: Function failed: base_do_unpack
>>> ERROR: Logfile of failure stored in: /home/ags/oe/build-colibri-imx7/tmp-oe/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/temp/log.do_unpack.35143
>>> ERROR: Task (/home/ags/oe/build-colibri-imx7/conf/../../layers/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb:do_unpack) failed with exit code '1'
>>>
>>> Any idea what that could be?
>>>
>>> --
>>> Stefan
>>
>> We also noticed the problem and supplied a fix to the bitbake-devel 
>> mailing list (actually, there were two separate issues in our case). The 
>> commits are currently sitting in master-next branches of bitbake and 
>> poky and should, in all likelihood, move to master soon.
> 
> I guess you refer to:
> fetch2/gitsm: checkout submodules recursively
> fetch2/gitsm: handle submodules nested inside subdirs
> 
> WARNING: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Failed to fetch URL gitsm://github.com/cajun-rat/HdrHistogram_c;protocol=https;name=third_party/HdrHistogram_c;bareclone=1;nocheckout=1, attempting MIRRORS if available
> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure: Unable to find revision 0b778a98c90ce55285cb1cb6fe50e2b25dfd57ae in branch master even from upstream
> WARNING: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Failed to fetch URL gitsm://github.com/advancedtelematic/aktualizr;branch=master, attempting MIRRORS if available
> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure for URL: 'gitsm://github.com/cajun-rat/HdrHistogram_c;protocol=https;name=third_party/HdrHistogram_c;bareclone=1;nocheckout=1'. Unable to fetch URL from any source.
> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure for URL: 'gitsm://github.com/advancedtelematic/aktualizr;branch=master'. Unable to fetch URL from any source.
> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Function failed: base_do_fetch
> ERROR: Logfile of failure stored in: /home/ags/oe/build-colibri-imx7/tmp-tordy/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/temp/log.do_fetch.41673
> ERROR: Task (/home/ags/oe/build-colibri-imx7/conf/../../layers/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb:do_fetch) failed with exit code '1'
> 
> (I am using meta-updater c4902394cc6673b07ad5ee87570e02afbdf7b022 and bitbake master-next as of now)
> 
> I did use -c cleanall before, which hopefully should be sufficient?

If you are still having issues from master-next (bitbake), can you distill down
a reproducer recipe?

All it takes is a minimal recipe with a SRC_URI line that used to work for you
through the do_unpack stage.  (I don't care about anything after do_unpack...)

typical reproducer may look like YP Bug 12941:

SUMMARY  = "Library to access Blu-Ray disk"
SECTION = "misc"
HOMEPAGE = "http://videolan.org"
LICENSE = "GPLv2"
LIC_FILES_CHKSUM="file://COPYING;md5=435ed639f84d4585d93824e7da3d85da"

DEPENDS = "libxml2"

SRC_URI = "gitsm://git.videolan.org/libbluray.git"

SRCREV = "${AUTOREV}"


--Mark

> --
> Stefan
> 

Re: git submodule fetcher regression

[Ricardo Salveti]

On Thu, Oct 4, 2018 at 1:47 PM Mark Hatle <mark.hatle@windriver.com> wrote:
>
> On 10/4/18 11:08 AM, Stefan Agner wrote:
> > [resend, this time with me being subscriber of bitbake-devel]
> >
> > Hi,
> >
> > It seems that 9c6b39adf978 ("fetch2/gitsm.py: Rework the git submodule fetcher") broke git submodules in some cases. My particular case is aktualizr from meta-updater (https://github.com/advancedtelematic/meta-updater). Fetching aktualizr reliably fails for me now:
> >
> > ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_unpack: Fetcher failure: Submodule tests/tuf-test-vectors does not exist in /home/ags/oe/downloads/git2/github.com.advancedtelematic.aktualizr/modules/tests/tuf-test-vectors or
> > /home/ags/oe/build-colibri-imx7/tmp-oe/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/git/.git/modules/tests/tuf-test-vectors.
> > ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_unpack: Function failed: base_do_unpack
> > ERROR: Logfile of failure stored in: /home/ags/oe/build-colibri-imx7/tmp-oe/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/temp/log.do_unpack.35143
> > ERROR: Task (/home/ags/oe/build-colibri-imx7/conf/../../layers/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb:do_unpack) failed with exit code '1'
> >
> > Any idea what that could be?
>
> See the mailing list archives (or master-next branch).  There are a small series
> of pending patches.  Please try these.
>
> 3 from Laurent Bonnans
> 1 from me

Tested with latest bitbake from master-next and latest aktualizr from
meta-updater and it seems functional again, looks like the patches
from Laurent are already enough (up to the checkout submodules
recursively).

Trying a completely clean build now to see if there are any other regressions.

Cheers,
-- 
Ricardo Salveti

Re: git submodule fetcher regression

[Bonnans, Laurent]

Hello Stefan,


On 10/04/2018 06:58 PM, Mark Hatle wrote:
> On 10/4/18 11:45 AM, Stefan Agner wrote:
>> I guess you refer to:
>> fetch2/gitsm: checkout submodules recursively
>> fetch2/gitsm: handle submodules nested inside subdirs
>>
>> WARNING: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Failed to fetch URL gitsm://github.com/cajun-rat/HdrHistogram_c;protocol=https;name=third_party/HdrHistogram_c;bareclone=1;nocheckout=1, attempting MIRRORS if available
>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure: Unable to find revision 0b778a98c90ce55285cb1cb6fe50e2b25dfd57ae in branch master even from upstream
>> WARNING: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Failed to fetch URL gitsm://github.com/advancedtelematic/aktualizr;branch=master, attempting MIRRORS if available
>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure for URL: 'gitsm://github.com/cajun-rat/HdrHistogram_c;protocol=https;name=third_party/HdrHistogram_c;bareclone=1;nocheckout=1'. Unable to fetch URL from any source.
>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure for URL: 'gitsm://github.com/advancedtelematic/aktualizr;branch=master'. Unable to fetch URL from any source.
>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Function failed: base_do_fetch
>> ERROR: Logfile of failure stored in: /home/ags/oe/build-colibri-imx7/tmp-tordy/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/temp/log.do_fetch.41673
>> ERROR: Task (/home/ags/oe/build-colibri-imx7/conf/../../layers/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb:do_fetch) failed with exit code '1'
>>
>> (I am using meta-updater c4902394cc6673b07ad5ee87570e02afbdf7b022 and bitbake master-next as of now)
>>
>> I did use -c cleanall before, which hopefully should be sufficient?
This should be solved in meta-updater 
8a44c8a26966d0cb9b57ea1ef4c6f25c2a944dd8 and beyond. I believe it's the 
same issue reported by Jens Rehsack on Oct 1.
We had a simple workaround in this case, so we did not investigate the 
core bitbake issue at that time.
> If you are still having issues from master-next (bitbake), can you distill down
> a reproducer recipe?
>
> All it takes is a minimal recipe with a SRC_URI line that used to work for you
> through the do_unpack stage.  (I don't care about anything after do_unpack...)
>
> typical reproducer may look like YP Bug 12941:
>
> SUMMARY  = "Library to access Blu-Ray disk"
> SECTION = "misc"
> HOMEPAGE = "http://videolan.org"
> LICENSE = "GPLv2"
> LIC_FILES_CHKSUM="file://COPYING;md5=435ed639f84d4585d93824e7da3d85da"
>
> DEPENDS = "libxml2"
>
> SRC_URI = "gitsm://git.videolan.org/libbluray.git"
>
> SRCREV = "${AUTOREV}"
>
>
> --Mark
>
>
Hello Mark, here is a short reproducer (aktualizr.bb)


SUMMARY = "Aktualizr SOTA Client"
DESCRIPTION = "SOTA Client application written in C++"
HOMEPAGE = "https://github.com/advancedtelematic/aktualizr"
SECTION = "base"
LICENSE = "MPL-2.0"
LIC_FILES_CHKSUM = 
"file://${S}/LICENSE;md5=9741c346eef56131163e13b9db1241b3"

PV = "1.0+git${SRCPV}"

SRC_URI = " \
   gitsm://github.com/advancedtelematic/aktualizr;branch=${BRANCH} \
   "
SRCREV = "79246b930a1b5b6f956b69f74f1f0c62060061df"
BRANCH ?= "master"


Then, `aktualizr -c unpack` fails with the same error Stefan described.

The root cause is this submodule declaration:

[submodule "third_party/HdrHistogram_c"]
     path = third_party/HdrHistogram_c
     url = https://github.com/cajun-rat/HdrHistogram_c

Which is set to the commit 0b778a98c90ce55285cb1cb6fe50e2b25dfd57ae, 
current HEAD of the branch `fix-const-cast` but not merged in 
cajun-rat's master.

Please let me know if you need more details.


Also, while investigating these issues, I noticed that `cleanall` now 
does not remove the fetched submodules like it would be expected. So, 
running `fetch` again only download the main project and will reuse 
cached submodules repositories if up to date.


Laurent

Re: git submodule fetcher regression

[Stefan Agner]

Hi Mark,

On 04.10.2018 20:51, Ricardo Salveti wrote:
> On Thu, Oct 4, 2018 at 1:47 PM Mark Hatle <mark.hatle@windriver.com> wrote:
>>
>> On 10/4/18 11:08 AM, Stefan Agner wrote:
>> > [resend, this time with me being subscriber of bitbake-devel]
>> >
>> > Hi,
>> >
>> > It seems that 9c6b39adf978 ("fetch2/gitsm.py: Rework the git submodule fetcher") broke git submodules in some cases. My particular case is aktualizr from meta-updater (https://github.com/advancedtelematic/meta-updater). Fetching aktualizr reliably fails for me now:
>> >
>> > ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_unpack: Fetcher failure: Submodule tests/tuf-test-vectors does not exist in /home/ags/oe/downloads/git2/github.com.advancedtelematic.aktualizr/modules/tests/tuf-test-vectors or
>> > /home/ags/oe/build-colibri-imx7/tmp-oe/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/git/.git/modules/tests/tuf-test-vectors.
>> > ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_unpack: Function failed: base_do_unpack
>> > ERROR: Logfile of failure stored in: /home/ags/oe/build-colibri-imx7/tmp-oe/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/temp/log.do_unpack.35143
>> > ERROR: Task (/home/ags/oe/build-colibri-imx7/conf/../../layers/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb:do_unpack) failed with exit code '1'
>> >
>> > Any idea what that could be?
>>
>> See the mailing list archives (or master-next branch).  There are a small series
>> of pending patches.  Please try these.
>>
>> 3 from Laurent Bonnans
>> 1 from me
> 
> Tested with latest bitbake from master-next and latest aktualizr from
> meta-updater and it seems functional again, looks like the patches
> from Laurent are already enough (up to the checkout submodules
> recursively).
> 
> Trying a completely clean build now to see if there are any other regressions.
> 

Tried todays bitbake master, which is fd9893c338df ("gitsm: Allow relative URIs when fetching"), containing the four commits mentioned above.

On my setup, building aktualizr, this fails to fetch source ref a94a15e31f3b973966b1afc9081fc1b17a3b9d48, while it works for a newer ref 512ad74c0b5339ca7775d8c9461b565a9e6ff5b3 (which presumably is what Ricardo tried).

This is a rather minimal recipe to reproduce the fetcher issue:

SUMMARY = "Aktualizr SOTA Client"
DESCRIPTION = "SOTA Client application written in C++"
HOMEPAGE = "https://github.com/advancedtelematic/aktualizr"
SECTION = "base"
LICENSE = "MPL-2.0"
LIC_FILES_CHKSUM = "file://${S}/LICENSE;md5=9741c346eef56131163e13b9db1241b3"

PV = "1.0+git${SRCPV}"
PR = "7"

SRC_URI = " \
  gitsm://github.com/advancedtelematic/aktualizr;branch=${BRANCH} \
  "
SRCREV = "a94a15e31f3b973966b1afc9081fc1b17a3b9d48"
BRANCH ?= "master"

S = "${WORKDIR}/git"

inherit cmake

--
Stefan

Re: git submodule fetcher regression

[Stefan Agner]

On 05.10.2018 10:38, Bonnans, Laurent wrote:
> Hello Stefan,
> 
> 
> On 10/04/2018 06:58 PM, Mark Hatle wrote:
>> On 10/4/18 11:45 AM, Stefan Agner wrote:
>>> I guess you refer to:
>>> fetch2/gitsm: checkout submodules recursively
>>> fetch2/gitsm: handle submodules nested inside subdirs
>>>
>>> WARNING: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Failed to fetch URL gitsm://github.com/cajun-rat/HdrHistogram_c;protocol=https;name=third_party/HdrHistogram_c;bareclone=1;nocheckout=1, attempting MIRRORS if available
>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure: Unable to find revision 0b778a98c90ce55285cb1cb6fe50e2b25dfd57ae in branch master even from upstream
>>> WARNING: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Failed to fetch URL gitsm://github.com/advancedtelematic/aktualizr;branch=master, attempting MIRRORS if available
>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure for URL: 'gitsm://github.com/cajun-rat/HdrHistogram_c;protocol=https;name=third_party/HdrHistogram_c;bareclone=1;nocheckout=1'. Unable to fetch URL from any source.
>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure for URL: 'gitsm://github.com/advancedtelematic/aktualizr;branch=master'. Unable to fetch URL from any source.
>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Function failed: base_do_fetch
>>> ERROR: Logfile of failure stored in: /home/ags/oe/build-colibri-imx7/tmp-tordy/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/temp/log.do_fetch.41673
>>> ERROR: Task (/home/ags/oe/build-colibri-imx7/conf/../../layers/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb:do_fetch) failed with exit code '1'
>>>
>>> (I am using meta-updater c4902394cc6673b07ad5ee87570e02afbdf7b022 and bitbake master-next as of now)
>>>
>>> I did use -c cleanall before, which hopefully should be sufficient?
> This should be solved in meta-updater 
> 8a44c8a26966d0cb9b57ea1ef4c6f25c2a944dd8 and beyond. I believe it's the 
> same issue reported by Jens Rehsack on Oct 1.
> We had a simple workaround in this case, so we did not investigate the 
> core bitbake issue at that time.
>> If you are still having issues from master-next (bitbake), can you distill down
>> a reproducer recipe?
>>
>> All it takes is a minimal recipe with a SRC_URI line that used to work for you
>> through the do_unpack stage.  (I don't care about anything after do_unpack...)
>>
>> typical reproducer may look like YP Bug 12941:
>>
>> SUMMARY  = "Library to access Blu-Ray disk"
>> SECTION = "misc"
>> HOMEPAGE = "http://videolan.org"
>> LICENSE = "GPLv2"
>> LIC_FILES_CHKSUM="file://COPYING;md5=435ed639f84d4585d93824e7da3d85da"
>>
>> DEPENDS = "libxml2"
>>
>> SRC_URI = "gitsm://git.videolan.org/libbluray.git"
>>
>> SRCREV = "${AUTOREV}"
>>
>>
>> --Mark
>>
>>
> Hello Mark, here is a short reproducer (aktualizr.bb)
> 
> 
> SUMMARY = "Aktualizr SOTA Client"
> DESCRIPTION = "SOTA Client application written in C++"
> HOMEPAGE = "https://github.com/advancedtelematic/aktualizr"
> SECTION = "base"
> LICENSE = "MPL-2.0"
> LIC_FILES_CHKSUM = 
> "file://${S}/LICENSE;md5=9741c346eef56131163e13b9db1241b3"
> 
> PV = "1.0+git${SRCPV}"
> 
> SRC_URI = " \
>    gitsm://github.com/advancedtelematic/aktualizr;branch=${BRANCH} \
>    "
> SRCREV = "79246b930a1b5b6f956b69f74f1f0c62060061df"
> BRANCH ?= "master"
> 
> 
> Then, `aktualizr -c unpack` fails with the same error Stefan described.
> 
> The root cause is this submodule declaration:
> 
> [submodule "third_party/HdrHistogram_c"]
>      path = third_party/HdrHistogram_c
>      url = https://github.com/cajun-rat/HdrHistogram_c
> 
> Which is set to the commit 0b778a98c90ce55285cb1cb6fe50e2b25dfd57ae, 
> current HEAD of the branch `fix-const-cast` but not merged in 
> cajun-rat's master.
> 
> Please let me know if you need more details.
> 
> 
> Also, while investigating these issues, I noticed that `cleanall` now 
> does not remove the fetched submodules like it would be expected. So, 
> running `fetch` again only download the main project and will reuse 
> cached submodules repositories if up to date.

That just raced with my email :-)

Yes that seems to be the issue I see. Thanks Laurent for looking closer into that.

--
Stefan

Re: git submodule fetcher regression

[Stefan Agner]

Hi Mark, Hi Laurent,

On 05.10.2018 10:38, Bonnans, Laurent wrote:
> Hello Stefan,
> 
> 
> On 10/04/2018 06:58 PM, Mark Hatle wrote:
>> On 10/4/18 11:45 AM, Stefan Agner wrote:
>>> I guess you refer to:
>>> fetch2/gitsm: checkout submodules recursively
>>> fetch2/gitsm: handle submodules nested inside subdirs
>>>
>>> WARNING: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Failed to fetch URL gitsm://github.com/cajun-rat/HdrHistogram_c;protocol=https;name=third_party/HdrHistogram_c;bareclone=1;nocheckout=1, attempting MIRRORS if available
>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure: Unable to find revision 0b778a98c90ce55285cb1cb6fe50e2b25dfd57ae in branch master even from upstream
>>> WARNING: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Failed to fetch URL gitsm://github.com/advancedtelematic/aktualizr;branch=master, attempting MIRRORS if available
>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure for URL: 'gitsm://github.com/cajun-rat/HdrHistogram_c;protocol=https;name=third_party/HdrHistogram_c;bareclone=1;nocheckout=1'. Unable to fetch URL from any source.
>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure for URL: 'gitsm://github.com/advancedtelematic/aktualizr;branch=master'. Unable to fetch URL from any source.
>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Function failed: base_do_fetch
>>> ERROR: Logfile of failure stored in: /home/ags/oe/build-colibri-imx7/tmp-tordy/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/temp/log.do_fetch.41673
>>> ERROR: Task (/home/ags/oe/build-colibri-imx7/conf/../../layers/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb:do_fetch) failed with exit code '1'
>>>
>>> (I am using meta-updater c4902394cc6673b07ad5ee87570e02afbdf7b022 and bitbake master-next as of now)
>>>
>>> I did use -c cleanall before, which hopefully should be sufficient?
> This should be solved in meta-updater 
> 8a44c8a26966d0cb9b57ea1ef4c6f25c2a944dd8 and beyond. I believe it's the 
> same issue reported by Jens Rehsack on Oct 1.
> We had a simple workaround in this case, so we did not investigate the 
> core bitbake issue at that time.
>> If you are still having issues from master-next (bitbake), can you distill down
>> a reproducer recipe?
>>
>> All it takes is a minimal recipe with a SRC_URI line that used to work for you
>> through the do_unpack stage.  (I don't care about anything after do_unpack...)
>>
>> typical reproducer may look like YP Bug 12941:
>>
>> SUMMARY  = "Library to access Blu-Ray disk"
>> SECTION = "misc"
>> HOMEPAGE = "http://videolan.org"
>> LICENSE = "GPLv2"
>> LIC_FILES_CHKSUM="file://COPYING;md5=435ed639f84d4585d93824e7da3d85da"
>>
>> DEPENDS = "libxml2"
>>
>> SRC_URI = "gitsm://git.videolan.org/libbluray.git"
>>
>> SRCREV = "${AUTOREV}"
>>
>>
>> --Mark
>>
>>
> Hello Mark, here is a short reproducer (aktualizr.bb)
> 
> 
> SUMMARY = "Aktualizr SOTA Client"
> DESCRIPTION = "SOTA Client application written in C++"
> HOMEPAGE = "https://github.com/advancedtelematic/aktualizr"
> SECTION = "base"
> LICENSE = "MPL-2.0"
> LIC_FILES_CHKSUM = 
> "file://${S}/LICENSE;md5=9741c346eef56131163e13b9db1241b3"
> 
> PV = "1.0+git${SRCPV}"
> 
> SRC_URI = " \
>    gitsm://github.com/advancedtelematic/aktualizr;branch=${BRANCH} \
>    "
> SRCREV = "79246b930a1b5b6f956b69f74f1f0c62060061df"
> BRANCH ?= "master"
> 
> 
> Then, `aktualizr -c unpack` fails with the same error Stefan described.
> 
> The root cause is this submodule declaration:
> 
> [submodule "third_party/HdrHistogram_c"]
>      path = third_party/HdrHistogram_c
>      url = https://github.com/cajun-rat/HdrHistogram_c
> 
> Which is set to the commit 0b778a98c90ce55285cb1cb6fe50e2b25dfd57ae, 
> current HEAD of the branch `fix-const-cast` but not merged in 
> cajun-rat's master.
> 
> Please let me know if you need more details.
> 
> 
> Also, while investigating these issues, I noticed that `cleanall` now 
> does not remove the fetched submodules like it would be expected. So, 
> running `fetch` again only download the main project and will reuse 
> cached submodules repositories if up to date.

Any update on this?

Still the case in bitbake master-next...

--
Stefan

Re: git submodule fetcher regression

[Mark Hatle]

On 10/12/18 8:50 AM, Stefan Agner wrote:
> Hi Mark, Hi Laurent,
> 
> On 05.10.2018 10:38, Bonnans, Laurent wrote:
>> Hello Stefan,
>>
>>
>> On 10/04/2018 06:58 PM, Mark Hatle wrote:
>>> On 10/4/18 11:45 AM, Stefan Agner wrote:
>>>> I guess you refer to:
>>>> fetch2/gitsm: checkout submodules recursively
>>>> fetch2/gitsm: handle submodules nested inside subdirs
>>>>
>>>> WARNING: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Failed to fetch URL gitsm://github.com/cajun-rat/HdrHistogram_c;protocol=https;name=third_party/HdrHistogram_c;bareclone=1;nocheckout=1, attempting MIRRORS if available
>>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure: Unable to find revision 0b778a98c90ce55285cb1cb6fe50e2b25dfd57ae in branch master even from upstream
>>>> WARNING: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Failed to fetch URL gitsm://github.com/advancedtelematic/aktualizr;branch=master, attempting MIRRORS if available
>>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure for URL: 'gitsm://github.com/cajun-rat/HdrHistogram_c;protocol=https;name=third_party/HdrHistogram_c;bareclone=1;nocheckout=1'. Unable to fetch URL from any source.
>>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure for URL: 'gitsm://github.com/advancedtelematic/aktualizr;branch=master'. Unable to fetch URL from any source.
>>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Function failed: base_do_fetch
>>>> ERROR: Logfile of failure stored in: /home/ags/oe/build-colibri-imx7/tmp-tordy/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/temp/log.do_fetch.41673
>>>> ERROR: Task (/home/ags/oe/build-colibri-imx7/conf/../../layers/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb:do_fetch) failed with exit code '1'
>>>>
>>>> (I am using meta-updater c4902394cc6673b07ad5ee87570e02afbdf7b022 and bitbake master-next as of now)
>>>>
>>>> I did use -c cleanall before, which hopefully should be sufficient?
>> This should be solved in meta-updater 
>> 8a44c8a26966d0cb9b57ea1ef4c6f25c2a944dd8 and beyond. I believe it's the 
>> same issue reported by Jens Rehsack on Oct 1.
>> We had a simple workaround in this case, so we did not investigate the 
>> core bitbake issue at that time.
>>> If you are still having issues from master-next (bitbake), can you distill down
>>> a reproducer recipe?
>>>
>>> All it takes is a minimal recipe with a SRC_URI line that used to work for you
>>> through the do_unpack stage.  (I don't care about anything after do_unpack...)
>>>
>>> typical reproducer may look like YP Bug 12941:
>>>
>>> SUMMARY  = "Library to access Blu-Ray disk"
>>> SECTION = "misc"
>>> HOMEPAGE = "http://videolan.org"
>>> LICENSE = "GPLv2"
>>> LIC_FILES_CHKSUM="file://COPYING;md5=435ed639f84d4585d93824e7da3d85da"
>>>
>>> DEPENDS = "libxml2"
>>>
>>> SRC_URI = "gitsm://git.videolan.org/libbluray.git"
>>>
>>> SRCREV = "${AUTOREV}"
>>>
>>>
>>> --Mark
>>>
>>>
>> Hello Mark, here is a short reproducer (aktualizr.bb)
>>
>>
>> SUMMARY = "Aktualizr SOTA Client"
>> DESCRIPTION = "SOTA Client application written in C++"
>> HOMEPAGE = "https://github.com/advancedtelematic/aktualizr"
>> SECTION = "base"
>> LICENSE = "MPL-2.0"
>> LIC_FILES_CHKSUM = 
>> "file://${S}/LICENSE;md5=9741c346eef56131163e13b9db1241b3"
>>
>> PV = "1.0+git${SRCPV}"
>>
>> SRC_URI = " \
>>    gitsm://github.com/advancedtelematic/aktualizr;branch=${BRANCH} \
>>    "
>> SRCREV = "79246b930a1b5b6f956b69f74f1f0c62060061df"
>> BRANCH ?= "master"
>>
>>
>> Then, `aktualizr -c unpack` fails with the same error Stefan described.
>>
>> The root cause is this submodule declaration:
>>
>> [submodule "third_party/HdrHistogram_c"]
>>      path = third_party/HdrHistogram_c
>>      url = https://github.com/cajun-rat/HdrHistogram_c
>>
>> Which is set to the commit 0b778a98c90ce55285cb1cb6fe50e2b25dfd57ae, 
>> current HEAD of the branch `fix-const-cast` but not merged in 
>> cajun-rat's master.
>>
>> Please let me know if you need more details.
>>
>>
>> Also, while investigating these issues, I noticed that `cleanall` now 
>> does not remove the fetched submodules like it would be expected. So, 
>> running `fetch` again only download the main project and will reuse 
>> cached submodules repositories if up to date.
> 
> Any update on this?

I don't have an immediate solution for the clear issue.  I'm hoping someone else
might have an idea there.

Since there repository is up-to-date (it has the commit id of what the submodule
wants) it should be safe for most users, unless they truly need to refetch for
another reason.

I'd consider this a low priority issue, but definitely a bug.

--Mark

> Still the case in bitbake master-next...
> 
> --
> Stefan
> 

Re: git submodule fetcher regression

[Stefan Agner]

On 12.10.2018 16:50, Mark Hatle wrote:
> On 10/12/18 8:50 AM, Stefan Agner wrote:
>> Hi Mark, Hi Laurent,
>>
>> On 05.10.2018 10:38, Bonnans, Laurent wrote:
>>> Hello Stefan,
>>>
>>>
>>> On 10/04/2018 06:58 PM, Mark Hatle wrote:
>>>> On 10/4/18 11:45 AM, Stefan Agner wrote:
>>>>> I guess you refer to:
>>>>> fetch2/gitsm: checkout submodules recursively
>>>>> fetch2/gitsm: handle submodules nested inside subdirs
>>>>>
>>>>> WARNING: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Failed to fetch URL gitsm://github.com/cajun-rat/HdrHistogram_c;protocol=https;name=third_party/HdrHistogram_c;bareclone=1;nocheckout=1, attempting MIRRORS if available
>>>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure: Unable to find revision 0b778a98c90ce55285cb1cb6fe50e2b25dfd57ae in branch master even from upstream
>>>>> WARNING: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Failed to fetch URL gitsm://github.com/advancedtelematic/aktualizr;branch=master, attempting MIRRORS if available
>>>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure for URL: 'gitsm://github.com/cajun-rat/HdrHistogram_c;protocol=https;name=third_party/HdrHistogram_c;bareclone=1;nocheckout=1'. Unable to fetch URL from any source.
>>>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Fetcher failure for URL: 'gitsm://github.com/advancedtelematic/aktualizr;branch=master'. Unable to fetch URL from any source.
>>>>> ERROR: aktualizr-1.0+gitAUTOINC+a94a15e31f-7 do_fetch: Function failed: base_do_fetch
>>>>> ERROR: Logfile of failure stored in: /home/ags/oe/build-colibri-imx7/tmp-tordy/work/armv7at2hf-neon-lmp-linux-gnueabi/aktualizr/1.0+gitAUTOINC+a94a15e31f-7/temp/log.do_fetch.41673
>>>>> ERROR: Task (/home/ags/oe/build-colibri-imx7/conf/../../layers/meta-updater/recipes-sota/aktualizr/aktualizr_git.bb:do_fetch) failed with exit code '1'
>>>>>
>>>>> (I am using meta-updater c4902394cc6673b07ad5ee87570e02afbdf7b022 and bitbake master-next as of now)
>>>>>
>>>>> I did use -c cleanall before, which hopefully should be sufficient?
>>> This should be solved in meta-updater
>>> 8a44c8a26966d0cb9b57ea1ef4c6f25c2a944dd8 and beyond. I believe it's the
>>> same issue reported by Jens Rehsack on Oct 1.
>>> We had a simple workaround in this case, so we did not investigate the
>>> core bitbake issue at that time.
>>>> If you are still having issues from master-next (bitbake), can you distill down
>>>> a reproducer recipe?
>>>>
>>>> All it takes is a minimal recipe with a SRC_URI line that used to work for you
>>>> through the do_unpack stage.  (I don't care about anything after do_unpack...)
>>>>
>>>> typical reproducer may look like YP Bug 12941:
>>>>
>>>> SUMMARY  = "Library to access Blu-Ray disk"
>>>> SECTION = "misc"
>>>> HOMEPAGE = "http://videolan.org"
>>>> LICENSE = "GPLv2"
>>>> LIC_FILES_CHKSUM="file://COPYING;md5=435ed639f84d4585d93824e7da3d85da"
>>>>
>>>> DEPENDS = "libxml2"
>>>>
>>>> SRC_URI = "gitsm://git.videolan.org/libbluray.git"
>>>>
>>>> SRCREV = "${AUTOREV}"
>>>>
>>>>
>>>> --Mark
>>>>
>>>>
>>> Hello Mark, here is a short reproducer (aktualizr.bb)
>>>
>>>
>>> SUMMARY = "Aktualizr SOTA Client"
>>> DESCRIPTION = "SOTA Client application written in C++"
>>> HOMEPAGE = "https://github.com/advancedtelematic/aktualizr"
>>> SECTION = "base"
>>> LICENSE = "MPL-2.0"
>>> LIC_FILES_CHKSUM =
>>> "file://${S}/LICENSE;md5=9741c346eef56131163e13b9db1241b3"
>>>
>>> PV = "1.0+git${SRCPV}"
>>>
>>> SRC_URI = " \
>>>    gitsm://github.com/advancedtelematic/aktualizr;branch=${BRANCH} \
>>>    "
>>> SRCREV = "79246b930a1b5b6f956b69f74f1f0c62060061df"
>>> BRANCH ?= "master"
>>>
>>>
>>> Then, `aktualizr -c unpack` fails with the same error Stefan described.
>>>
>>> The root cause is this submodule declaration:
>>>
>>> [submodule "third_party/HdrHistogram_c"]
>>>      path = third_party/HdrHistogram_c
>>>      url = https://github.com/cajun-rat/HdrHistogram_c
>>>
>>> Which is set to the commit 0b778a98c90ce55285cb1cb6fe50e2b25dfd57ae,
>>> current HEAD of the branch `fix-const-cast` but not merged in
>>> cajun-rat's master.
>>>
>>> Please let me know if you need more details.
>>>
>>>
>>> Also, while investigating these issues, I noticed that `cleanall` now
>>> does not remove the fetched submodules like it would be expected. So,
>>> running `fetch` again only download the main project and will reuse
>>> cached submodules repositories if up to date.
>>
>> Any update on this?
> 
> I don't have an immediate solution for the clear issue.  I'm hoping someone else
> might have an idea there.

Sorry, I meant the issue above, where fetching fails...

Fetching that hash used to work before the gitsm rework, so this is a
regression...

--
Stefan

> 
> Since there repository is up-to-date (it has the commit id of what the submodule
> wants) it should be safe for most users, unless they truly need to refetch for
> another reason.
> 
> I'd consider this a low priority issue, but definitely a bug.
> 
> --Mark
> 
>> Still the case in bitbake master-next...
>>
>> --
>> Stefan
>>

Re: git submodule fetcher regression

[Andreas Müller]

Hi,

sorry for being late on this but I think I have a recipe failing 100% [1]

To gather more information I did checkout
7d5af23d560b38fb5558d4ac51c3c9611d6d904f. This is the commit right
before 9c6b39adf9781fa6745f48913a97c859fa37eb5b where gism fetcher
rework started: It works fine without issues so upstream can be
assumed working.

Hope this helps

Andreas

[1] https://github.com/schnitzeltony/meta-qt5-extra/blob/master/recipes-misc/recipes-multimedia/wolf-shaper/wolf-shaper_0.1.6.bb