[meta-raspberrypi][PATCH] firmware.inc: fetch from SVN instead of Git

[meta-raspberrypi][PATCH] firmware.inc: fetch from SVN instead of Git

[Jonathan Liu]

Reduces the download time significantly.

Signed-off-by: Jonathan Liu <net147@gmail.com>
---
 recipes-bsp/common/firmware.inc | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/recipes-bsp/common/firmware.inc b/recipes-bsp/common/firmware.inc
index ad3176a..100df3a 100644
--- a/recipes-bsp/common/firmware.inc
+++ b/recipes-bsp/common/firmware.inc
@@ -1,7 +1,7 @@
-RPIFW_SRCREV ?= "e42a747e8d5c4a2fb3e837d0924c7cc39999936a"
+RPIFW_SRCREV ?= "461"
 RPIFW_DATE ?= "20150206"
-RPIFW_SRC_URI ?= "git://github.com/raspberrypi/firmware.git;protocol=git;branch=master"
-RPIFW_S ?= "${WORKDIR}/git"
+RPIFW_SRC_URI ?= "svn://github.com/raspberrypi/firmware;protocol=https;module=trunk"
+RPIFW_S ?= "${WORKDIR}/trunk"
 
 SRC_URI = "${RPIFW_SRC_URI}"
 SRCREV = "${RPIFW_SRCREV}"
-- 
2.4.4

Re: [meta-raspberrypi][PATCH] firmware.inc: fetch from SVN instead of Git

[Nikolay Dimitrov]

Hi Jonathan,

On 07/04/2015 06:48 AM, Jonathan Liu wrote:
> Reduces the download time significantly.
>
> Signed-off-by: Jonathan Liu <net147@gmail.com>
> ---
>   recipes-bsp/common/firmware.inc | 6 +++---
>   1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/recipes-bsp/common/firmware.inc b/recipes-bsp/common/firmware.inc
> index ad3176a..100df3a 100644
> --- a/recipes-bsp/common/firmware.inc
> +++ b/recipes-bsp/common/firmware.inc
> @@ -1,7 +1,7 @@
> -RPIFW_SRCREV ?= "e42a747e8d5c4a2fb3e837d0924c7cc39999936a"
> +RPIFW_SRCREV ?= "461"
>   RPIFW_DATE ?= "20150206"
> -RPIFW_SRC_URI ?= "git://github.com/raspberrypi/firmware.git;protocol=git;branch=master"
> -RPIFW_S ?= "${WORKDIR}/git"
> +RPIFW_SRC_URI ?= "svn://github.com/raspberrypi/firmware;protocol=https;module=trunk"
> +RPIFW_S ?= "${WORKDIR}/trunk"
>
>   SRC_URI = "${RPIFW_SRC_URI}"
>   SRCREV = "${RPIFW_SRCREV}"
>

Just tested the download times for this repo:

git (deep clone): 24m
svn: 3m8s
git (shallow clone): 40s

Isn't it possible to use "--depth 1" for git clone? I would rather
prefer seeing this instead of just giving up and going back to svn...

Regards,
Nikolay

Re: [meta-raspberrypi][PATCH] firmware.inc: fetch from SVN instead of Git

[Petter Mabäcker]

On 07/04/2015 07:29 AM, Nikolay Dimitrov wrote:
> Hi Jonathan,
>
> On 07/04/2015 06:48 AM, Jonathan Liu wrote:
>> Reduces the download time significantly.
>>
>> Signed-off-by: Jonathan Liu <net147@gmail.com>
>> ---
>>   recipes-bsp/common/firmware.inc | 6 +++---
>>   1 file changed, 3 insertions(+), 3 deletions(-)
>>
>> diff --git a/recipes-bsp/common/firmware.inc 
>> b/recipes-bsp/common/firmware.inc
>> index ad3176a..100df3a 100644
>> --- a/recipes-bsp/common/firmware.inc
>> +++ b/recipes-bsp/common/firmware.inc
>> @@ -1,7 +1,7 @@
>> -RPIFW_SRCREV ?= "e42a747e8d5c4a2fb3e837d0924c7cc39999936a"
>> +RPIFW_SRCREV ?= "461"
>>   RPIFW_DATE ?= "20150206"
>> -RPIFW_SRC_URI ?= 
>> "git://github.com/raspberrypi/firmware.git;protocol=git;branch=master"
>> -RPIFW_S ?= "${WORKDIR}/git"
>> +RPIFW_SRC_URI ?= 
>> "svn://github.com/raspberrypi/firmware;protocol=https;module=trunk"
>> +RPIFW_S ?= "${WORKDIR}/trunk"
>>
>>   SRC_URI = "${RPIFW_SRC_URI}"
>>   SRCREV = "${RPIFW_SRCREV}"
>>
>
> Just tested the download times for this repo:
>
> git (deep clone): 24m
> svn: 3m8s
> git (shallow clone): 40s
>
> Isn't it possible to use "--depth 1" for git clone? I would rather
> prefer seeing this instead of just giving up and going back to svn...
>
> Regards,
> Nikolay
Jonathan, I understand your problem, I have pointed out 
"https://github.com/raspberrypi/firmware/archive/${RPIFW_SRCREV}.zip" 
ontop of HEAD when building in order to get rid of this problem. But I 
agree with Nikolay that going back to svn feels like a step in the wrong 
direction and instead we should push for a changle like the one 
discussed in the thread: '[yocto] [meta-raspberrypi][PATCH] 
firmware.inc: Fetch a zip instead of cloning a git repo'

or by using "--depth=1" (which isn't supported in the bitbake git 
fetcher for the moment..).

I will try to push for that one of the solutions (--depth=1 or .zip) is 
merged soon so we once for all can get rid of this annoying problem.

BR,
Petter

Re: [meta-raspberrypi][PATCH] firmware.inc: fetch from SVN instead of Git

[Jonathan Liu]

Hi Nikolay,

On 4/07/2015 3:29 PM, Nikolay Dimitrov wrote:
> Hi Jonathan,
>
> On 07/04/2015 06:48 AM, Jonathan Liu wrote:
>> Reduces the download time significantly.
>>
>> Signed-off-by: Jonathan Liu <net147@gmail.com>
>> ---
>>   recipes-bsp/common/firmware.inc | 6 +++---
>>   1 file changed, 3 insertions(+), 3 deletions(-)
>>
>> diff --git a/recipes-bsp/common/firmware.inc 
>> b/recipes-bsp/common/firmware.inc
>> index ad3176a..100df3a 100644
>> --- a/recipes-bsp/common/firmware.inc
>> +++ b/recipes-bsp/common/firmware.inc
>> @@ -1,7 +1,7 @@
>> -RPIFW_SRCREV ?= "e42a747e8d5c4a2fb3e837d0924c7cc39999936a"
>> +RPIFW_SRCREV ?= "461"
>>   RPIFW_DATE ?= "20150206"
>> -RPIFW_SRC_URI ?= 
>> "git://github.com/raspberrypi/firmware.git;protocol=git;branch=master"
>> -RPIFW_S ?= "${WORKDIR}/git"
>> +RPIFW_SRC_URI ?= 
>> "svn://github.com/raspberrypi/firmware;protocol=https;module=trunk"
>> +RPIFW_S ?= "${WORKDIR}/trunk"
>>
>>   SRC_URI = "${RPIFW_SRC_URI}"
>>   SRCREV = "${RPIFW_SRCREV}"
>>
>
> Just tested the download times for this repo:
>
> git (deep clone): 24m
> svn: 3m8s
> git (shallow clone): 40s
>
> Isn't it possible to use "--depth 1" for git clone? I would rather
> prefer seeing this instead of just giving up and going back to svn...
>
> Regards,
> Nikolay
How would you clone a specific Git revision using --depth 1?

Regards,
Jonathan

Re: [meta-raspberrypi][PATCH] firmware.inc: fetch from SVN instead of Git

[Nikolay Dimitrov]

Hi Jonathan,

On 07/06/2015 04:41 PM, Jonathan Liu wrote:
> Hi Nikolay,
>
> On 4/07/2015 3:29 PM, Nikolay Dimitrov wrote:
>> Hi Jonathan,
>>
>> On 07/04/2015 06:48 AM, Jonathan Liu wrote:
>>> Reduces the download time significantly.
>>>
>>> Signed-off-by: Jonathan Liu <net147@gmail.com>
>>> ---
>>>   recipes-bsp/common/firmware.inc | 6 +++---
>>>   1 file changed, 3 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/recipes-bsp/common/firmware.inc
>>> b/recipes-bsp/common/firmware.inc
>>> index ad3176a..100df3a 100644
>>> --- a/recipes-bsp/common/firmware.inc
>>> +++ b/recipes-bsp/common/firmware.inc
>>> @@ -1,7 +1,7 @@
>>> -RPIFW_SRCREV ?= "e42a747e8d5c4a2fb3e837d0924c7cc39999936a"
>>> +RPIFW_SRCREV ?= "461"
>>>   RPIFW_DATE ?= "20150206"
>>> -RPIFW_SRC_URI ?=
>>> "git://github.com/raspberrypi/firmware.git;protocol=git;branch=master"
>>> -RPIFW_S ?= "${WORKDIR}/git"
>>> +RPIFW_SRC_URI ?=
>>> "svn://github.com/raspberrypi/firmware;protocol=https;module=trunk"
>>> +RPIFW_S ?= "${WORKDIR}/trunk"
>>>
>>>   SRC_URI = "${RPIFW_SRC_URI}"
>>>   SRCREV = "${RPIFW_SRCREV}"
>>>
>>
>> Just tested the download times for this repo:
>>
>> git (deep clone): 24m
>> svn: 3m8s
>> git (shallow clone): 40s
>>
>> Isn't it possible to use "--depth 1" for git clone? I would rather
>> prefer seeing this instead of just giving up and going back to svn...
>>
>> Regards,
>> Nikolay
> How would you clone a specific Git revision using --depth 1?
>
> Regards,
> Jonathan

git fetch --depth=1 git://repo.git $SHA1

Available in the latest git code.

A point to be made is that this feature requires cooperation from the
server side (uploadpack.allowReachableSHA1InWant config variable) which
can complicate thing somewhat, but we can probably start pushing the
server provider to accelerate the process :D.

Regards,
Nikolay

Re: [meta-raspberrypi][PATCH] firmware.inc: fetch from SVN instead of Git

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 275 bytes --]

On 6 July 2015 at 17:18, Nikolay Dimitrov <picmaster@mail.bg> wrote:

> git fetch --depth=1 git://repo.git $SHA1
>
> Available in the latest git code.
>

How latest, remembering that oe-core supports LTS distros which are likely
to have old releases of git.

Ross

[-- Attachment #2: Type: text/html, Size: 682 bytes --]

Re: [meta-raspberrypi][PATCH] firmware.inc: fetch from SVN instead of Git

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 490 bytes --]

On 6 July 2015 at 17:23, Burton, Ross <ross.burton@intel.com> wrote:

> How latest, remembering that oe-core supports LTS distros which are likely
> to have old releases of git.
>

Assuming I'm driving it correctly:

[git init, git remote add origin ...]
$ git fetch  --depth=1 origin beeafee030a38380c65a9a664003ba50f87aefbf
error: no such remote ref beeafee030a38380c65a9a664003ba50f87aefbf
$ git --version
git version 2.1.4

That's with the latest Debian release.

Ross

[-- Attachment #2: Type: text/html, Size: 1001 bytes --]

Re: [meta-raspberrypi][PATCH] firmware.inc: fetch from SVN instead of Git

[Nikolay Dimitrov]

Hi Ross,

On 07/06/2015 07:26 PM, Burton, Ross wrote:
>
> On 6 July 2015 at 17:23, Burton, Ross <ross.burton@intel.com
> <mailto:ross.burton@intel.com>> wrote:
>
>     How latest, remembering that oe-core supports LTS distros which are
>     likely to have old releases of git.
>
>
> Assuming I'm driving it correctly:
>
> [git init, git remote add origin ...]
> $ git fetch  --depth=1 origin beeafee030a38380c65a9a664003ba50f87aefbf
> error: no such remote ref beeafee030a38380c65a9a664003ba50f87aefbf
> $ git --version
> git version 2.1.4
>
> That's with the latest Debian release.
>
> Ross

Well, the commits that add this feature are from May-June.2015. I fully
agree with you that LTS is a factor that can't be ignored.

Kind regards,
Nikolay

Re: [meta-raspberrypi][PATCH] firmware.inc: fetch from SVN instead of Git

[Nikolay Dimitrov]

Hi Ross,

On 07/06/2015 07:58 PM, Nikolay Dimitrov wrote:
> Hi Ross,
>
> On 07/06/2015 07:26 PM, Burton, Ross wrote:
>>
>> On 6 July 2015 at 17:23, Burton, Ross <ross.burton@intel.com
>> <mailto:ross.burton@intel.com>> wrote:
>>
>>     How latest, remembering that oe-core supports LTS distros which are
>>     likely to have old releases of git.
>>
>>
>> Assuming I'm driving it correctly:
>>
>> [git init, git remote add origin ...]
>> $ git fetch  --depth=1 origin beeafee030a38380c65a9a664003ba50f87aefbf
>> error: no such remote ref beeafee030a38380c65a9a664003ba50f87aefbf
>> $ git --version
>> git version 2.1.4
>>
>> That's with the latest Debian release.
>>
>> Ross
>
> Well, the commits that add this feature are from May-June.2015. I fully
> agree with you that LTS is a factor that can't be ignored.
>
> Kind regards,
> Nikolay

Forgot to answer you question - the expected versions to include this 
feature are 2.5.0/2.5.1.

Regards,
Nikolay

Re: [meta-raspberrypi][PATCH] firmware.inc: fetch from SVN instead of Git

[Andrei Gherzan]

[-- Attachment #1: Type: text/plain, Size: 1214 bytes --]

On Mon, Jul 6, 2015 at 6:59 PM, Nikolay Dimitrov <picmaster@mail.bg> wrote:

> Hi Ross,
>
> On 07/06/2015 07:58 PM, Nikolay Dimitrov wrote:
>
>> Hi Ross,
>>
>> On 07/06/2015 07:26 PM, Burton, Ross wrote:
>>
>>>
>>> On 6 July 2015 at 17:23, Burton, Ross <ross.burton@intel.com
>>> <mailto:ross.burton@intel.com>> wrote:
>>>
>>>     How latest, remembering that oe-core supports LTS distros which are
>>>     likely to have old releases of git.
>>>
>>>
>>> Assuming I'm driving it correctly:
>>>
>>> [git init, git remote add origin ...]
>>> $ git fetch  --depth=1 origin beeafee030a38380c65a9a664003ba50f87aefbf
>>> error: no such remote ref beeafee030a38380c65a9a664003ba50f87aefbf
>>> $ git --version
>>> git version 2.1.4
>>>
>>> That's with the latest Debian release.
>>>
>>> Ross
>>>
>>
>> Well, the commits that add this feature are from May-June.2015. I fully
>> agree with you that LTS is a factor that can't be ignored.
>>
>> Kind regards,
>> Nikolay
>>
>
> Forgot to answer you question - the expected versions to include this
> feature are 2.5.0/2.5.1.
>
>
So there is no support for depth clones until 2.5.0? I didn't really
understand.

--
Andrei Gherzan

[-- Attachment #2: Type: text/html, Size: 2255 bytes --]

Re: [meta-raspberrypi][PATCH] firmware.inc: fetch from SVN instead of Git

[Nikolay Dimitrov]

Hi Andrei,

On 07/09/2015 11:19 PM, Andrei Gherzan wrote:
> On Mon, Jul 6, 2015 at 6:59 PM, Nikolay Dimitrov <picmaster@mail.bg
> <mailto:picmaster@mail.bg>> wrote:
>
>     Hi Ross,
>
>     On 07/06/2015 07:58 PM, Nikolay Dimitrov wrote:
>
>         Hi Ross,
>
>         On 07/06/2015 07:26 PM, Burton, Ross wrote:
>
>
>             On 6 July 2015 at 17:23, Burton, Ross <ross.burton@intel.com
>             <mailto:ross.burton@intel.com>
>             <mailto:ross.burton@intel.com
>             <mailto:ross.burton@intel.com>>> wrote:
>
>                  How latest, remembering that oe-core supports LTS
>             distros which are
>                  likely to have old releases of git.
>
>
>             Assuming I'm driving it correctly:
>
>             [git init, git remote add origin ...]
>             $ git fetch  --depth=1 origin
>             beeafee030a38380c65a9a664003ba50f87aefbf
>             error: no such remote ref
>             beeafee030a38380c65a9a664003ba50f87aefbf
>             $ git --version
>             git version 2.1.4
>
>             That's with the latest Debian release.
>
>             Ross
>
>
>         Well, the commits that add this feature are from May-June.2015.
>         I fully
>         agree with you that LTS is a factor that can't be ignored.
>
>         Kind regards,
>         Nikolay
>
>
>     Forgot to answer you question - the expected versions to include
>     this feature are 2.5.0/2.5.1. <http://2.5.1.>
>
>
> So there is no support for depth clones until 2.5.0? I didn't really
> understand.

Well, shallow clones are supported but only for branch tips, which is
not what we need. This feature for shallow cloning a specific revision
is available only in git 2.5.0+. Also, this feature needs a server-side
configuration option to be enabled, in order to work properly.

Regards,
Nikolay

Re: [meta-raspberrypi][PATCH] firmware.inc: fetch from SVN instead of Git

[Andrei Gherzan]

On Mon, Aug 10, 2015 at 01:03:57AM -0700, Khem Raj wrote:
> On Sun, Aug 9, 2015 at 4:57 PM, Andrei Gherzan <andrei@gherzan.ro> wrote:
> > Hi,
> >
> > On Mon, Jul 20, 2015 at 09:59:30AM -0600, Gary Thomas wrote:
> >> On 2015-07-19 15:34, Andrei Gherzan wrote:
> >> >Hello,
> >> >
> >> >--
> >> >Andrei Gherzan
> >> >
> >> >On Thu, Jul 16, 2015 at 7:53 PM, Jon Szymaniak <jon.szymaniak@gmail.com <mailto:jon.szymaniak@gmail.com>> wrote:
> >> >
> >> >    Hi all,
> >> >
> >> >        >
> >> >        > So there is no support for depth clones until 2.5.0? I didn't really
> >> >        > understand.
> >> >
> >> >        Well, shallow clones are supported but only for branch tips, which is
> >> >        not what we need. This feature for shallow cloning a specific revision
> >> >        is available only in git 2.5.0+. Also, this feature needs a server-side
> >> >        configuration option to be enabled, in order to work properly.
> >> >
> >> >        Regards,
> >> >        Nikolay
> >> >
> >> >
> >> >    I'd argue that this whole issue with the whole meta-raspberrypi firmware.inc download is more than just slow, inconvenient download.  I've left builds running all night (8+
> >> >    hours on a 30Mib/s residential link) that just hang on this, usually timing out.  I initially thought it was just me, but am hearing others confirm this as well.
> >> >
> >> >    As such, I just wanted to continue this conversation.  It sounds like git fetch's --depth is the best option on the table, but has the issues Nikolay has described.  What are
> >> >    your thoughts on the following?
> >> >
> >> >    (1) We create a git-native and build a version that supports this the fetch depth?
> >> >
> >> >    I suspect this could be made to work, but haven't dug into what dependencies git may have and how that would play out on various LTS distros. My knee-jerk is that has too high
> >> >    of a risk/benefit ration, given that we're talking about 1 repo.
> >> >
> >> >    (2) We update the git fetcher to check the git version and support a depth= option if the git version is sufficient.  If it is not, we spit out a warning and fall back to the
> >> >    current behavior.
> >> >
> >> >    Neither (1) nor (2) address Nikolay's point that --depth requires server-side support.  However, I'd argue this is something you'd be testing and verifying when writing the
> >> >    recipe.  Is this a reasonable assertion? How likely is it that a server supporting this would suddenly be re-configured?
> >> >
> >> >    (3) We request that the upstream maintainer of meta-raspberrypi use the GitHub Release feature [a] to post a tarball of a known checksum at somewhat regular intervals.  I'm
> >> >    told by a few package maintainers that while the tarballs that it generates for specific changesets are subject to change, that the tarballs it autogenerates when using its
> >> >    Releases feature do not.  However, I have not confirmed this.  If this is false, then one can upload a tarball with known checksums to the release as an attachment; I would be
> >> >    *shocked* if they touch your attachments.
> >> >
> >> >    While (3) is a nice "not our problem" solution, I think (2) might have some benefits for other recipes later.  Any other ideas?
> >> >
> >> >
> >> >Definitely 2 is the best opinion in my opinion too. I'm wondering if there is any work started in this direction.
> >> >
> >>
> >> It would sure be nice to get this fixed!  The latest download
> >> (I always save the tarballs for the next time) is over 4GB!!
> >>
> >
> > So what is the conlcusion on this? Should we switch on SVN temporary?
>
> Step back a bit and look at what this firmware repo is all about. Its
> bunch of binaries in there. Git is a bad choice for such a repo. Since
> binary blobs whenever updates will add that much of data to size of
> metadata and soon it will go into tens of gigabytes so its a building
> avalanche. Ideally this firmware should have been released as tarballs
> in first place. Can you work with owners who are releasing this in
> this form to reconsider releasing it in discrete tarballs ?

This is not under our control. I will try to talk with rpi people:
https://github.com/raspberrypi/firmware/issues/455

Khem, could you add a comment there too?

Thanks,

--
Andrei Gherzan

Re: [meta-raspberrypi][PATCH] firmware.inc: fetch from SVN instead of Git

[Khem Raj]

On Sun, Aug 9, 2015 at 4:57 PM, Andrei Gherzan <andrei@gherzan.ro> wrote:
> Hi,
>
> On Mon, Jul 20, 2015 at 09:59:30AM -0600, Gary Thomas wrote:
>> On 2015-07-19 15:34, Andrei Gherzan wrote:
>> >Hello,
>> >
>> >--
>> >Andrei Gherzan
>> >
>> >On Thu, Jul 16, 2015 at 7:53 PM, Jon Szymaniak <jon.szymaniak@gmail.com <mailto:jon.szymaniak@gmail.com>> wrote:
>> >
>> >    Hi all,
>> >
>> >        >
>> >        > So there is no support for depth clones until 2.5.0? I didn't really
>> >        > understand.
>> >
>> >        Well, shallow clones are supported but only for branch tips, which is
>> >        not what we need. This feature for shallow cloning a specific revision
>> >        is available only in git 2.5.0+. Also, this feature needs a server-side
>> >        configuration option to be enabled, in order to work properly.
>> >
>> >        Regards,
>> >        Nikolay
>> >
>> >
>> >    I'd argue that this whole issue with the whole meta-raspberrypi firmware.inc download is more than just slow, inconvenient download.  I've left builds running all night (8+
>> >    hours on a 30Mib/s residential link) that just hang on this, usually timing out.  I initially thought it was just me, but am hearing others confirm this as well.
>> >
>> >    As such, I just wanted to continue this conversation.  It sounds like git fetch's --depth is the best option on the table, but has the issues Nikolay has described.  What are
>> >    your thoughts on the following?
>> >
>> >    (1) We create a git-native and build a version that supports this the fetch depth?
>> >
>> >    I suspect this could be made to work, but haven't dug into what dependencies git may have and how that would play out on various LTS distros. My knee-jerk is that has too high
>> >    of a risk/benefit ration, given that we're talking about 1 repo.
>> >
>> >    (2) We update the git fetcher to check the git version and support a depth= option if the git version is sufficient.  If it is not, we spit out a warning and fall back to the
>> >    current behavior.
>> >
>> >    Neither (1) nor (2) address Nikolay's point that --depth requires server-side support.  However, I'd argue this is something you'd be testing and verifying when writing the
>> >    recipe.  Is this a reasonable assertion? How likely is it that a server supporting this would suddenly be re-configured?
>> >
>> >    (3) We request that the upstream maintainer of meta-raspberrypi use the GitHub Release feature [a] to post a tarball of a known checksum at somewhat regular intervals.  I'm
>> >    told by a few package maintainers that while the tarballs that it generates for specific changesets are subject to change, that the tarballs it autogenerates when using its
>> >    Releases feature do not.  However, I have not confirmed this.  If this is false, then one can upload a tarball with known checksums to the release as an attachment; I would be
>> >    *shocked* if they touch your attachments.
>> >
>> >    While (3) is a nice "not our problem" solution, I think (2) might have some benefits for other recipes later.  Any other ideas?
>> >
>> >
>> >Definitely 2 is the best opinion in my opinion too. I'm wondering if there is any work started in this direction.
>> >
>>
>> It would sure be nice to get this fixed!  The latest download
>> (I always save the tarballs for the next time) is over 4GB!!
>>
>
> So what is the conlcusion on this? Should we switch on SVN temporary?

Step back a bit and look at what this firmware repo is all about. Its
bunch of binaries in there. Git is a bad choice for such a repo. Since
binary blobs whenever updates will add that much of data to size of
metadata and soon it will go into tens of gigabytes so its a building
avalanche. Ideally this firmware should have been released as tarballs
in first place. Can you work with owners who are releasing this in
this form to reconsider releasing it in discrete tarballs ?

Re: [meta-raspberrypi][PATCH] firmware.inc: fetch from SVN instead of Git

[Andrei Gherzan]

Hi,

On Mon, Jul 20, 2015 at 09:59:30AM -0600, Gary Thomas wrote:
> On 2015-07-19 15:34, Andrei Gherzan wrote:
> >Hello,
> >
> >--
> >Andrei Gherzan
> >
> >On Thu, Jul 16, 2015 at 7:53 PM, Jon Szymaniak <jon.szymaniak@gmail.com <mailto:jon.szymaniak@gmail.com>> wrote:
> >
> >    Hi all,
> >
> >        >
> >        > So there is no support for depth clones until 2.5.0? I didn't really
> >        > understand.
> >
> >        Well, shallow clones are supported but only for branch tips, which is
> >        not what we need. This feature for shallow cloning a specific revision
> >        is available only in git 2.5.0+. Also, this feature needs a server-side
> >        configuration option to be enabled, in order to work properly.
> >
> >        Regards,
> >        Nikolay
> >
> >
> >    I'd argue that this whole issue with the whole meta-raspberrypi firmware.inc download is more than just slow, inconvenient download.  I've left builds running all night (8+
> >    hours on a 30Mib/s residential link) that just hang on this, usually timing out.  I initially thought it was just me, but am hearing others confirm this as well.
> >
> >    As such, I just wanted to continue this conversation.  It sounds like git fetch's --depth is the best option on the table, but has the issues Nikolay has described.  What are
> >    your thoughts on the following?
> >
> >    (1) We create a git-native and build a version that supports this the fetch depth?
> >
> >    I suspect this could be made to work, but haven't dug into what dependencies git may have and how that would play out on various LTS distros. My knee-jerk is that has too high
> >    of a risk/benefit ration, given that we're talking about 1 repo.
> >
> >    (2) We update the git fetcher to check the git version and support a depth= option if the git version is sufficient.  If it is not, we spit out a warning and fall back to the
> >    current behavior.
> >
> >    Neither (1) nor (2) address Nikolay's point that --depth requires server-side support.  However, I'd argue this is something you'd be testing and verifying when writing the
> >    recipe.  Is this a reasonable assertion? How likely is it that a server supporting this would suddenly be re-configured?
> >
> >    (3) We request that the upstream maintainer of meta-raspberrypi use the GitHub Release feature [a] to post a tarball of a known checksum at somewhat regular intervals.  I'm
> >    told by a few package maintainers that while the tarballs that it generates for specific changesets are subject to change, that the tarballs it autogenerates when using its
> >    Releases feature do not.  However, I have not confirmed this.  If this is false, then one can upload a tarball with known checksums to the release as an attachment; I would be
> >    *shocked* if they touch your attachments.
> >
> >    While (3) is a nice "not our problem" solution, I think (2) might have some benefits for other recipes later.  Any other ideas?
> >
> >
> >Definitely 2 is the best opinion in my opinion too. I'm wondering if there is any work started in this direction.
> >
>
> It would sure be nice to get this fixed!  The latest download
> (I always save the tarballs for the next time) is over 4GB!!
>

So what is the conlcusion on this? Should we switch on SVN temporary?

--
Andrei Gherzan

Re: [meta-raspberrypi][PATCH] firmware.inc: fetch from SVN instead of Git

[Gary Thomas]

On 2015-07-19 15:34, Andrei Gherzan wrote:
> Hello,
>
> --
> Andrei Gherzan
>
> On Thu, Jul 16, 2015 at 7:53 PM, Jon Szymaniak <jon.szymaniak@gmail.com <mailto:jon.szymaniak@gmail.com>> wrote:
>
>     Hi all,
>
>         >
>         > So there is no support for depth clones until 2.5.0? I didn't really
>         > understand.
>
>         Well, shallow clones are supported but only for branch tips, which is
>         not what we need. This feature for shallow cloning a specific revision
>         is available only in git 2.5.0+. Also, this feature needs a server-side
>         configuration option to be enabled, in order to work properly.
>
>         Regards,
>         Nikolay
>
>
>     I'd argue that this whole issue with the whole meta-raspberrypi firmware.inc download is more than just slow, inconvenient download.  I've left builds running all night (8+
>     hours on a 30Mib/s residential link) that just hang on this, usually timing out.  I initially thought it was just me, but am hearing others confirm this as well.
>
>     As such, I just wanted to continue this conversation.  It sounds like git fetch's --depth is the best option on the table, but has the issues Nikolay has described.  What are
>     your thoughts on the following?
>
>     (1) We create a git-native and build a version that supports this the fetch depth?
>
>     I suspect this could be made to work, but haven't dug into what dependencies git may have and how that would play out on various LTS distros. My knee-jerk is that has too high
>     of a risk/benefit ration, given that we're talking about 1 repo.
>
>     (2) We update the git fetcher to check the git version and support a depth= option if the git version is sufficient.  If it is not, we spit out a warning and fall back to the
>     current behavior.
>
>     Neither (1) nor (2) address Nikolay's point that --depth requires server-side support.  However, I'd argue this is something you'd be testing and verifying when writing the
>     recipe.  Is this a reasonable assertion? How likely is it that a server supporting this would suddenly be re-configured?
>
>     (3) We request that the upstream maintainer of meta-raspberrypi use the GitHub Release feature [a] to post a tarball of a known checksum at somewhat regular intervals.  I'm
>     told by a few package maintainers that while the tarballs that it generates for specific changesets are subject to change, that the tarballs it autogenerates when using its
>     Releases feature do not.  However, I have not confirmed this.  If this is false, then one can upload a tarball with known checksums to the release as an attachment; I would be
>     *shocked* if they touch your attachments.
>
>     While (3) is a nice "not our problem" solution, I think (2) might have some benefits for other recipes later.  Any other ideas?
>
>
> Definitely 2 is the best opinion in my opinion too. I'm wondering if there is any work started in this direction.
>

It would sure be nice to get this fixed!  The latest download
(I always save the tarballs for the next time) is over 4GB!!

-rw-rw-r-- 1 gthomas gthomas 4194568645 Jul 20 09:44 /local/rpi2_2015-03-05/downloads/git2_github.com.raspberrypi.firmware.git.tar.gz


-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: [meta-raspberrypi][PATCH] firmware.inc: fetch from SVN instead of Git

[Andrei Gherzan]

[-- Attachment #1: Type: text/plain, Size: 2845 bytes --]

Hello,

--
Andrei Gherzan

On Thu, Jul 16, 2015 at 7:53 PM, Jon Szymaniak <jon.szymaniak@gmail.com>
wrote:

> Hi all,
>
>
>> >
>> > So there is no support for depth clones until 2.5.0? I didn't really
>> > understand.
>>
>> Well, shallow clones are supported but only for branch tips, which is
>> not what we need. This feature for shallow cloning a specific revision
>> is available only in git 2.5.0+. Also, this feature needs a server-side
>> configuration option to be enabled, in order to work properly.
>>
>> Regards,
>> Nikolay
>>
>
> I'd argue that this whole issue with the whole meta-raspberrypi
> firmware.inc download is more than just slow, inconvenient download.  I've
> left builds running all night (8+ hours on a 30Mib/s residential link) that
> just hang on this, usually timing out.  I initially thought it was just me,
> but am hearing others confirm this as well.
>
> As such, I just wanted to continue this conversation.  It sounds like git
> fetch's --depth is the best option on the table, but has the issues Nikolay
> has described.  What are your thoughts on the following?
>
> (1) We create a git-native and build a version that supports this the
> fetch depth?
>
> I suspect this could be made to work, but haven't dug into what
> dependencies git may have and how that would play out on various LTS
> distros. My knee-jerk is that has too high of a risk/benefit ration, given
> that we're talking about 1 repo.
>
> (2) We update the git fetcher to check the git version and support a
> depth= option if the git version is sufficient.  If it is not, we spit out
> a warning and fall back to the current behavior.
>
> Neither (1) nor (2) address Nikolay's point that --depth requires
> server-side support.  However, I'd argue this is something you'd be testing
> and verifying when writing the recipe.  Is this a reasonable assertion? How
> likely is it that a server supporting this would suddenly be re-configured?
>
> (3) We request that the upstream maintainer of meta-raspberrypi use the
> GitHub Release feature [a] to post a tarball of a known checksum at
> somewhat regular intervals.  I'm told by a few package maintainers that
> while the tarballs that it generates for specific changesets are subject to
> change, that the tarballs it autogenerates when using its Releases feature
> do not.  However, I have not confirmed this.  If this is false, then one
> can upload a tarball with known checksums to the release as an attachment;
> I would be *shocked* if they touch your attachments.
>
> While (3) is a nice "not our problem" solution, I think (2) might have
> some benefits for other recipes later.  Any other ideas?
>
>
Definitely 2 is the best opinion in my opinion too. I'm wondering if there
is any work started in this direction.

[-- Attachment #2: Type: text/html, Size: 3681 bytes --]

Re: [meta-raspberrypi][PATCH] firmware.inc: fetch from SVN instead of Git

[Jon Szymaniak]

[-- Attachment #1: Type: text/plain, Size: 2602 bytes --]

Hi all,


> >
> > So there is no support for depth clones until 2.5.0? I didn't really
> > understand.
>
> Well, shallow clones are supported but only for branch tips, which is
> not what we need. This feature for shallow cloning a specific revision
> is available only in git 2.5.0+. Also, this feature needs a server-side
> configuration option to be enabled, in order to work properly.
>
> Regards,
> Nikolay
>

I'd argue that this whole issue with the whole meta-raspberrypi
firmware.inc download is more than just slow, inconvenient download.  I've
left builds running all night (8+ hours on a 30Mib/s residential link) that
just hang on this, usually timing out.  I initially thought it was just me,
but am hearing others confirm this as well.

As such, I just wanted to continue this conversation.  It sounds like git
fetch's --depth is the best option on the table, but has the issues Nikolay
has described.  What are your thoughts on the following?

(1) We create a git-native and build a version that supports this the fetch
depth?

I suspect this could be made to work, but haven't dug into what
dependencies git may have and how that would play out on various LTS
distros. My knee-jerk is that has too high of a risk/benefit ration, given
that we're talking about 1 repo.

(2) We update the git fetcher to check the git version and support a depth=
option if the git version is sufficient.  If it is not, we spit out a
warning and fall back to the current behavior.

Neither (1) nor (2) address Nikolay's point that --depth requires
server-side support.  However, I'd argue this is something you'd be testing
and verifying when writing the recipe.  Is this a reasonable assertion? How
likely is it that a server supporting this would suddenly be re-configured?

(3) We request that the upstream maintainer of meta-raspberrypi use the
GitHub Release feature [a] to post a tarball of a known checksum at
somewhat regular intervals.  I'm told by a few package maintainers that
while the tarballs that it generates for specific changesets are subject to
change, that the tarballs it autogenerates when using its Releases feature
do not.  However, I have not confirmed this.  If this is false, then one
can upload a tarball with known checksums to the release as an attachment;
I would be *shocked* if they touch your attachments.

While (3) is a nice "not our problem" solution, I think (2) might have some
benefits for other recipes later.  Any other ideas?

Best regards,
Jon

[a] https://help.github.com/articles/creating-releases/

[-- Attachment #2: Type: text/html, Size: 3204 bytes --]