Fetch failure for source at kernel.org

Fetch failure for source at kernel.org

[Philip Balister]

I'm finally trying to get started with oe-core. I'm using a setup 
created with the setup-script created by Koen. Everything looks ok until 
it tried to build cpufrequtils. The recipe tries to fetch the source 
from a git repo at kernel.org. Obviously, the fetch fails.

What is odd, in the "good old days" I would expect this config to fall 
back and use the Angstrom source mirror, but apparently this is not the 
case with oe-core.

Is this a bug or a feature? If this is a bug, is there a plan to fix it? 
If it is a feature, how are we supposed to do builds if servers go away? 
What do we do about GPL compliance?

Thanks,

Philip

Re: Fetch failure for source at kernel.org

[Mark Hatle]

On 9/21/11 2:38 PM, Philip Balister wrote:
> I'm finally trying to get started with oe-core. I'm using a setup 
> created with the setup-script created by Koen. Everything looks ok until 
> it tried to build cpufrequtils. The recipe tries to fetch the source 
> from a git repo at kernel.org. Obviously, the fetch fails.
> 
> What is odd, in the "good old days" I would expect this config to fall 
> back and use the Angstrom source mirror, but apparently this is not the 
> case with oe-core.
> 
> Is this a bug or a feature? If this is a bug, is there a plan to fix it? 
> If it is a feature, how are we supposed to do builds if servers go away? 
> What do we do about GPL compliance?

This is a feature, as people did not want oe-core tied to a given mirror.

Myself, I use:

PREMIRRORS ?= "\
bzr://.*/.*   http://autobuilder.yoctoproject.org/sources/ \n \
cvs://.*/.*   http://autobuilder.yoctoproject.org/sources/ \n \
git://.*/.*   http://autobuilder.yoctoproject.org/sources/ \n \
hg://.*/.*    http://autobuilder.yoctoproject.org/sources/ \n \
osc://.*/.*   http://autobuilder.yoctoproject.org/sources/ \n \
p4://.*/.*    http://autobuilder.yoctoproject.org/sources/ \n \
svk://.*/.*   http://autobuilder.yoctoproject.org/sources/ \n \
svn://.*/.*   http://autobuilder.yoctoproject.org/sources/ \n"

MIRRORS =+ "\
ftp://.*/.*    http://autobuilder.yoctoproject.org/sources/ \n \
http://.*/.*     http://autobuilder.yoctoproject.org/sources/ \n \
https://.*/.*    http://autobuilder.yoctoproject.org/sources/ \n"

PREMIRRORS are tried before the upstream source, MIRRORS are tried only after a
failure.  (Please correct me someone if I have that wrong.)

--Mark

> Thanks,
> 
> Philip
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: Fetch failure for source at kernel.org

[Koen Kooi]

Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven:

> On 9/21/11 2:38 PM, Philip Balister wrote:
>> I'm finally trying to get started with oe-core. I'm using a setup 
>> created with the setup-script created by Koen. Everything looks ok until 
>> it tried to build cpufrequtils. The recipe tries to fetch the source 
>> from a git repo at kernel.org. Obviously, the fetch fails.
>> 
>> What is odd, in the "good old days" I would expect this config to fall 
>> back and use the Angstrom source mirror, but apparently this is not the 
>> case with oe-core.
>> 
>> Is this a bug or a feature? If this is a bug, is there a plan to fix it? 
>> If it is a feature, how are we supposed to do builds if servers go away? 
>> What do we do about GPL compliance?
> 
> This is a feature, as people did not want oe-core tied to a given mirror.

This is the "OE core doesn't generate versioned git tarballs" problem people keep talking about. I know I can edit each and every recipe to add ';rebasable=true', but I'd like a global solution before the GPL police starts looking at the angstrom source mirrors.

regards,

Koen

Re: Fetch failure for source at kernel.org

[Richard Purdie]

On Wed, 2011-09-21 at 21:49 +0200, Koen Kooi wrote:
> Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven:
> 
> > On 9/21/11 2:38 PM, Philip Balister wrote:
> >> I'm finally trying to get started with oe-core. I'm using a setup 
> >> created with the setup-script created by Koen. Everything looks ok until 
> >> it tried to build cpufrequtils. The recipe tries to fetch the source 
> >> from a git repo at kernel.org. Obviously, the fetch fails.
> >> 
> >> What is odd, in the "good old days" I would expect this config to fall 
> >> back and use the Angstrom source mirror, but apparently this is not the 
> >> case with oe-core.
> >> 
> >> Is this a bug or a feature? If this is a bug, is there a plan to fix it? 
> >> If it is a feature, how are we supposed to do builds if servers go away? 
> >> What do we do about GPL compliance?
> > 
> > This is a feature, as people did not want oe-core tied to a given mirror.
> 
> This is the "OE core doesn't generate versioned git tarballs" problem
> people keep talking about. I know I can edit each and every recipe to
> add ';rebasable=true', but I'd like a global solution before the GPL
> police starts looking at the angstrom source mirrors.

Is this a versioned tarball problem or just the tarballs not being
generated at all?

Are you setting BB_GENERATE_MIRROR_TARBALLS?

Cheers,

Richard

Re: Fetch failure for source at kernel.org

[Koen Kooi]

Op 22 sep. 2011, om 02:26 heeft Richard Purdie het volgende geschreven:

> On Wed, 2011-09-21 at 21:49 +0200, Koen Kooi wrote:
>> Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven:
>> 
>>> On 9/21/11 2:38 PM, Philip Balister wrote:
>>>> I'm finally trying to get started with oe-core. I'm using a setup 
>>>> created with the setup-script created by Koen. Everything looks ok until 
>>>> it tried to build cpufrequtils. The recipe tries to fetch the source 
>>>> from a git repo at kernel.org. Obviously, the fetch fails.
>>>> 
>>>> What is odd, in the "good old days" I would expect this config to fall 
>>>> back and use the Angstrom source mirror, but apparently this is not the 
>>>> case with oe-core.
>>>> 
>>>> Is this a bug or a feature? If this is a bug, is there a plan to fix it? 
>>>> If it is a feature, how are we supposed to do builds if servers go away? 
>>>> What do we do about GPL compliance?
>>> 
>>> This is a feature, as people did not want oe-core tied to a given mirror.
>> 
>> This is the "OE core doesn't generate versioned git tarballs" problem
>> people keep talking about. I know I can edit each and every recipe to
>> add ';rebasable=true', but I'd like a global solution before the GPL
>> police starts looking at the angstrom source mirrors.
> 
> Is this a versioned tarball problem or just the tarballs not being
> generated at all?

versioned tarball problem, since the source mirror won't overwrite files with the same name, so unversioned ones are both useless and bandwidth wasting.

Re: Fetch failure for source at kernel.org

[Richard Purdie]

On Thu, 2011-09-22 at 08:40 +0200, Koen Kooi wrote:
> Op 22 sep. 2011, om 02:26 heeft Richard Purdie het volgende geschreven:
> 
> > On Wed, 2011-09-21 at 21:49 +0200, Koen Kooi wrote:
> >> Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven:
> >> 
> >>> On 9/21/11 2:38 PM, Philip Balister wrote:
> >>>> I'm finally trying to get started with oe-core. I'm using a setup 
> >>>> created with the setup-script created by Koen. Everything looks ok until 
> >>>> it tried to build cpufrequtils. The recipe tries to fetch the source 
> >>>> from a git repo at kernel.org. Obviously, the fetch fails.
> >>>> 
> >>>> What is odd, in the "good old days" I would expect this config to fall 
> >>>> back and use the Angstrom source mirror, but apparently this is not the 
> >>>> case with oe-core.
> >>>> 
> >>>> Is this a bug or a feature? If this is a bug, is there a plan to fix it? 
> >>>> If it is a feature, how are we supposed to do builds if servers go away? 
> >>>> What do we do about GPL compliance?
> >>> 
> >>> This is a feature, as people did not want oe-core tied to a given mirror.
> >> 
> >> This is the "OE core doesn't generate versioned git tarballs" problem
> >> people keep talking about. I know I can edit each and every recipe to
> >> add ';rebasable=true', but I'd like a global solution before the GPL
> >> police starts looking at the angstrom source mirrors.
> > 
> > Is this a versioned tarball problem or just the tarballs not being
> > generated at all?
> 
> versioned tarball problem, since the source mirror won't overwrite
> files with the same name, so unversioned ones are both useless and
> bandwidth wasting.

They seem to be working perfectly well for Yocto...

They're also not bandwidth wasting since the fetcher is efficient and
will incrementally update something that is incomplete.

Anyhow, this is the first time someone has explained that the mirror
tarball improvements that were made to bitbake have not done everything
they desire. If you want someone on the Yocto team to look at this I'd
suggest a bug report or I'll take patches to bitbake (as always).

Cheers,

Richard

Re: Fetch failure for source at kernel.org

[Koen Kooi]

Op 22 sep. 2011, om 09:03 heeft Richard Purdie het volgende geschreven:

> On Thu, 2011-09-22 at 08:40 +0200, Koen Kooi wrote:
>> Op 22 sep. 2011, om 02:26 heeft Richard Purdie het volgende geschreven:
>> 
>>> On Wed, 2011-09-21 at 21:49 +0200, Koen Kooi wrote:
>>>> Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven:
>>>> 
>>>>> On 9/21/11 2:38 PM, Philip Balister wrote:
>>>>>> I'm finally trying to get started with oe-core. I'm using a setup 
>>>>>> created with the setup-script created by Koen. Everything looks ok until 
>>>>>> it tried to build cpufrequtils. The recipe tries to fetch the source 
>>>>>> from a git repo at kernel.org. Obviously, the fetch fails.
>>>>>> 
>>>>>> What is odd, in the "good old days" I would expect this config to fall 
>>>>>> back and use the Angstrom source mirror, but apparently this is not the 
>>>>>> case with oe-core.
>>>>>> 
>>>>>> Is this a bug or a feature? If this is a bug, is there a plan to fix it? 
>>>>>> If it is a feature, how are we supposed to do builds if servers go away? 
>>>>>> What do we do about GPL compliance?
>>>>> 
>>>>> This is a feature, as people did not want oe-core tied to a given mirror.
>>>> 
>>>> This is the "OE core doesn't generate versioned git tarballs" problem
>>>> people keep talking about. I know I can edit each and every recipe to
>>>> add ';rebasable=true', but I'd like a global solution before the GPL
>>>> police starts looking at the angstrom source mirrors.
>>> 
>>> Is this a versioned tarball problem or just the tarballs not being
>>> generated at all?
>> 
>> versioned tarball problem, since the source mirror won't overwrite
>> files with the same name, so unversioned ones are both useless and
>> bandwidth wasting.
> 
> They seem to be working perfectly well for Yocto...
> 
> They're also not bandwidth wasting

They are, since for the kernel you need to re-upload the 300MB tarball to the source mirror for mirroring to work, even if there were no real changes but bitbake repacked it. I know that versioned tarballs aren't terribly efficient either, but at least they are deterministic.

The angstrom source mirror checks for duplicate filenames to avoid introducing checksum changes due to upstream stupidity or corrupted downloads. 

> since the fetcher is efficient and
> will incrementally update something that is incomplete.

Which won't work currently since kernel.org is down.

> Anyhow, this is the first time someone has explained that the mirror
> tarball improvements that were made to bitbake have not done everything
> they desire. If you want someone on the Yocto team to look at this I'd
> suggest a bug report or I'll take patches to bitbake (as always).

But you do acknowledge the problem? FWIW, I've been arguing for a global method of enabled versioned tarballs since I first started using oe-core (called differently back then). I know linux-yocto stretches the fetcher to its limits, but I'd like this change for all the recipes using a single git tree with a single srcrev.

regards,

Koen

Re: Fetch failure for source at kernel.org

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3502 bytes --]

On Thu, Sep 22, 2011 at 10:28:02AM +0200, Koen Kooi wrote:
> 
> Op 22 sep. 2011, om 09:03 heeft Richard Purdie het volgende geschreven:
> 
> > On Thu, 2011-09-22 at 08:40 +0200, Koen Kooi wrote:
> >> Op 22 sep. 2011, om 02:26 heeft Richard Purdie het volgende geschreven:
> >> 
> >>> On Wed, 2011-09-21 at 21:49 +0200, Koen Kooi wrote:
> >>>> Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven:
> >>>> 
> >>>>> On 9/21/11 2:38 PM, Philip Balister wrote:
> >>>>>> I'm finally trying to get started with oe-core. I'm using a setup 
> >>>>>> created with the setup-script created by Koen. Everything looks ok until 
> >>>>>> it tried to build cpufrequtils. The recipe tries to fetch the source 
> >>>>>> from a git repo at kernel.org. Obviously, the fetch fails.
> >>>>>> 
> >>>>>> What is odd, in the "good old days" I would expect this config to fall 
> >>>>>> back and use the Angstrom source mirror, but apparently this is not the 
> >>>>>> case with oe-core.
> >>>>>> 
> >>>>>> Is this a bug or a feature? If this is a bug, is there a plan to fix it? 
> >>>>>> If it is a feature, how are we supposed to do builds if servers go away? 
> >>>>>> What do we do about GPL compliance?
> >>>>> 
> >>>>> This is a feature, as people did not want oe-core tied to a given mirror.
> >>>> 
> >>>> This is the "OE core doesn't generate versioned git tarballs" problem
> >>>> people keep talking about. I know I can edit each and every recipe to
> >>>> add ';rebasable=true', but I'd like a global solution before the GPL
> >>>> police starts looking at the angstrom source mirrors.
> >>> 
> >>> Is this a versioned tarball problem or just the tarballs not being
> >>> generated at all?
> >> 
> >> versioned tarball problem, since the source mirror won't overwrite
> >> files with the same name, so unversioned ones are both useless and
> >> bandwidth wasting.
> > 
> > They seem to be working perfectly well for Yocto...
> > 
> > They're also not bandwidth wasting
> 
> They are, since for the kernel you need to re-upload the 300MB tarball to the source mirror for mirroring to work, even if there were no real changes but bitbake repacked it. I know that versioned tarballs aren't terribly efficient either, but at least they are deterministic.
> 
> The angstrom source mirror checks for duplicate filenames to avoid introducing checksum changes due to upstream stupidity or corrupted downloads. 
> 
> > since the fetcher is efficient and
> > will incrementally update something that is incomplete.
> 
> Which won't work currently since kernel.org is down.
> 
> > Anyhow, this is the first time someone has explained that the mirror
> > tarball improvements that were made to bitbake have not done everything
> > they desire. If you want someone on the Yocto team to look at this I'd
> > suggest a bug report or I'll take patches to bitbake (as always).
> 
> But you do acknowledge the problem? FWIW, I've been arguing for a global method of enabled versioned tarballs since I first started using oe-core (called differently back then). I know linux-yocto stretches the fetcher to its limits, but I'd like this change for all the recipes using a single git tree with a single srcrev.

FWIW: I would also like to have versioned git checkouts in fetcher2 as
it was in old fetcher. It would be more consitent with svn versioned
checkouts which are still created.

Regards,
-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: Fetch failure for source at kernel.org

[Koen Kooi]

Op 22 sep. 2011, om 10:45 heeft Martin Jansa het volgende geschreven:

> On Thu, Sep 22, 2011 at 10:28:02AM +0200, Koen Kooi wrote:
>> 
>> Op 22 sep. 2011, om 09:03 heeft Richard Purdie het volgende geschreven:
>> 
>>> On Thu, 2011-09-22 at 08:40 +0200, Koen Kooi wrote:
>>>> Op 22 sep. 2011, om 02:26 heeft Richard Purdie het volgende geschreven:
>>>> 
>>>>> On Wed, 2011-09-21 at 21:49 +0200, Koen Kooi wrote:
>>>>>> Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven:
>>>>>> 
>>>>>>> On 9/21/11 2:38 PM, Philip Balister wrote:
>>>>>>>> I'm finally trying to get started with oe-core. I'm using a setup 
>>>>>>>> created with the setup-script created by Koen. Everything looks ok until 
>>>>>>>> it tried to build cpufrequtils. The recipe tries to fetch the source 
>>>>>>>> from a git repo at kernel.org. Obviously, the fetch fails.
>>>>>>>> 
>>>>>>>> What is odd, in the "good old days" I would expect this config to fall 
>>>>>>>> back and use the Angstrom source mirror, but apparently this is not the 
>>>>>>>> case with oe-core.
>>>>>>>> 
>>>>>>>> Is this a bug or a feature? If this is a bug, is there a plan to fix it? 
>>>>>>>> If it is a feature, how are we supposed to do builds if servers go away? 
>>>>>>>> What do we do about GPL compliance?
>>>>>>> 
>>>>>>> This is a feature, as people did not want oe-core tied to a given mirror.
>>>>>> 
>>>>>> This is the "OE core doesn't generate versioned git tarballs" problem
>>>>>> people keep talking about. I know I can edit each and every recipe to
>>>>>> add ';rebasable=true', but I'd like a global solution before the GPL
>>>>>> police starts looking at the angstrom source mirrors.
>>>>> 
>>>>> Is this a versioned tarball problem or just the tarballs not being
>>>>> generated at all?
>>>> 
>>>> versioned tarball problem, since the source mirror won't overwrite
>>>> files with the same name, so unversioned ones are both useless and
>>>> bandwidth wasting.
>>> 
>>> They seem to be working perfectly well for Yocto...
>>> 
>>> They're also not bandwidth wasting
>> 
>> They are, since for the kernel you need to re-upload the 300MB tarball to the source mirror for mirroring to work, even if there were no real changes but bitbake repacked it. I know that versioned tarballs aren't terribly efficient either, but at least they are deterministic.
>> 
>> The angstrom source mirror checks for duplicate filenames to avoid introducing checksum changes due to upstream stupidity or corrupted downloads. 
>> 
>>> since the fetcher is efficient and
>>> will incrementally update something that is incomplete.
>> 
>> Which won't work currently since kernel.org is down.
>> 
>>> Anyhow, this is the first time someone has explained that the mirror
>>> tarball improvements that were made to bitbake have not done everything
>>> they desire. If you want someone on the Yocto team to look at this I'd
>>> suggest a bug report or I'll take patches to bitbake (as always).
>> 
>> But you do acknowledge the problem? FWIW, I've been arguing for a global method of enabled versioned tarballs since I first started using oe-core (called differently back then). I know linux-yocto stretches the fetcher to its limits, but I'd like this change for all the recipes using a single git tree with a single srcrev.
> 
> FWIW: I would also like to have versioned git checkouts in fetcher2 as
> it was in old fetcher. It would be more consitent with svn versioned
> checkouts which are still created.

http://patches.openembedded.org/patch/11971/ adds a global flag to generate such tarballs, but I don't know if the mirroring logic will actually try to fetch those or if fetch2 will actually try to unpack them when present, but at least we can generate the tarballs now :)

regards,

Koen

Re: Fetch failure for source at kernel.org

[Richard Purdie]

On Thu, 2011-09-22 at 10:28 +0200, Koen Kooi wrote:
> Op 22 sep. 2011, om 09:03 heeft Richard Purdie het volgende geschreven:
> > On Thu, 2011-09-22 at 08:40 +0200, Koen Kooi wrote:
> >> Op 22 sep. 2011, om 02:26 heeft Richard Purdie het volgende geschreven:
> >>> On Wed, 2011-09-21 at 21:49 +0200, Koen Kooi wrote:
> >>>> Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven:
> >>>> 
> >>>>> On 9/21/11 2:38 PM, Philip Balister wrote:
> >>>>>> I'm finally trying to get started with oe-core. I'm using a setup 
> >>>>>> created with the setup-script created by Koen. Everything looks ok until 
> >>>>>> it tried to build cpufrequtils. The recipe tries to fetch the source 
> >>>>>> from a git repo at kernel.org. Obviously, the fetch fails.
> >>>>>> 
> >>>>>> What is odd, in the "good old days" I would expect this config to fall 
> >>>>>> back and use the Angstrom source mirror, but apparently this is not the 
> >>>>>> case with oe-core.
> >>>>>> 
> >>>>>> Is this a bug or a feature? If this is a bug, is there a plan to fix it? 
> >>>>>> If it is a feature, how are we supposed to do builds if servers go away? 
> >>>>>> What do we do about GPL compliance?
> >>>>> 
> >>>>> This is a feature, as people did not want oe-core tied to a given mirror.
> >>>> 
> >>>> This is the "OE core doesn't generate versioned git tarballs" problem
> >>>> people keep talking about. I know I can edit each and every recipe to
> >>>> add ';rebasable=true', but I'd like a global solution before the GPL
> >>>> police starts looking at the angstrom source mirrors.
> >>> 
> >>> Is this a versioned tarball problem or just the tarballs not being
> >>> generated at all?
> >> 
> >> versioned tarball problem, since the source mirror won't overwrite
> >> files with the same name, so unversioned ones are both useless and
> >> bandwidth wasting.
> > 
> > They seem to be working perfectly well for Yocto...
> > 
> > They're also not bandwidth wasting
> 
> They are, since for the kernel you need to re-upload the 300MB tarball
> to the source mirror for mirroring to work, even if there were no real
> changes but bitbake repacked it. I know that versioned tarballs aren't
> terribly efficient either, but at least they are deterministic.

Well, it depends on your perspective, I was looking at this from a
user's point of view. The server's bandwidth is nothing compared to the
users pulling the data.

> The angstrom source mirror checks for duplicate filenames to avoid
> introducing checksum changes due to upstream stupidity or corrupted
> downloads. 

Fair enough but you could make an exception for the git scm tarballs
(which follow a fairly identifiable pattern).

> > since the fetcher is efficient and
> > will incrementally update something that is incomplete.
> 
> Which won't work currently since kernel.org is down.

This just means there aren't appropriate git:// -> git:// mappings in
people's mirrors files (including Yocto). If git:// -> git:// mappings
don't work, we should fix that but they should.

> > Anyhow, this is the first time someone has explained that the mirror
> > tarball improvements that were made to bitbake have not done everything
> > they desire. If you want someone on the Yocto team to look at this I'd
> > suggest a bug report or I'll take patches to bitbake (as always).
> 
> But you do acknowledge the problem? FWIW, I've been arguing for a
> global method of enabled versioned tarballs since I first started
> using oe-core (called differently back then). I know linux-yocto
> stretches the fetcher to its limits, but I'd like this change for all
> the recipes using a single git tree with a single srcrev.

Now someone has explained the issue, yes, I can see why it is causing
problems. I do think some of them are of people's own choosing but I'm
trying to be flexible and not make judgement on that.

So we can do something along the lines of what you after but there are
the following considerations:

a) The versioned tarballs do need to be different to what we had before 
in that the tarball will be of a .git directory containing the git
metadata, not just a checkout. Why? This is so we can incrementally
update a git repo if the user changes the source revision as one
example. We're dealing with the git fetcher and limiting ourselves to a
flat one dimension doesn't make sense.

b) We need to agree what order the tarballs on the server should be
searched for. The "obvious" order would be:

1. Specific Revision X tarball
2. generic mirror tarball
3. git clone

The problems come about when we have an existing git checkout which
doesn't contain the revision we want. Currently we can just update and
assume that will bring in the revision. If we now support people doing
rebases and other things, we now at least in theory have to:

1. See if the revision exists locally
2. Try pulling
3. If that doesn't work look for a versioned tarball
4. Blow away all fetcher git status from DL_DIR
5. Extract the tarball

My big concern with this is rather than having some general
archictecture and standard way of doing this, the fetcher once again
becomes a set of special case if A do B but if C do D first then do E
but don't forget to do A if Z happens type code.

So what I'm saying is I don't want the fetcher to become the
unmaintainable mess the that the code paths in fetch1 were. I don't
honestly know how to add a code path like the above and avoid this :(

The patch on the bitbake list doesn't consider any of this complexity or
the corner cases :(

Cheers,

Richard

Re: Fetch failure for source at kernel.org

[Koen Kooi]

Op 22 sep 2011, om 12:20 heeft Richard Purdie het volgende geschreven:

> On Thu, 2011-09-22 at 10:28 +0200, Koen Kooi wrote:
>> Op 22 sep. 2011, om 09:03 heeft Richard Purdie het volgende  
>> geschreven:
>>> On Thu, 2011-09-22 at 08:40 +0200, Koen Kooi wrote:
>>>> Op 22 sep. 2011, om 02:26 heeft Richard Purdie het volgende  
>>>> geschreven:
>>>>> On Wed, 2011-09-21 at 21:49 +0200, Koen Kooi wrote:
>>>>>> Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende  
>>>>>> geschreven:
>>>>>>
>>>>>>> On 9/21/11 2:38 PM, Philip Balister wrote:
>>>>>>>> I'm finally trying to get started with oe-core. I'm using a  
>>>>>>>> setup
>>>>>>>> created with the setup-script created by Koen. Everything  
>>>>>>>> looks ok until
>>>>>>>> it tried to build cpufrequtils. The recipe tries to fetch the  
>>>>>>>> source
>>>>>>>> from a git repo at kernel.org. Obviously, the fetch fails.
>>>>>>>>
>>>>>>>> What is odd, in the "good old days" I would expect this  
>>>>>>>> config to fall
>>>>>>>> back and use the Angstrom source mirror, but apparently this  
>>>>>>>> is not the
>>>>>>>> case with oe-core.
>>>>>>>>
>>>>>>>> Is this a bug or a feature? If this is a bug, is there a plan  
>>>>>>>> to fix it?
>>>>>>>> If it is a feature, how are we supposed to do builds if  
>>>>>>>> servers go away?
>>>>>>>> What do we do about GPL compliance?
>>>>>>>
>>>>>>> This is a feature, as people did not want oe-core tied to a  
>>>>>>> given mirror.
>>>>>>
>>>>>> This is the "OE core doesn't generate versioned git tarballs"  
>>>>>> problem
>>>>>> people keep talking about. I know I can edit each and every  
>>>>>> recipe to
>>>>>> add ';rebasable=true', but I'd like a global solution before  
>>>>>> the GPL
>>>>>> police starts looking at the angstrom source mirrors.
>>>>>
>>>>> Is this a versioned tarball problem or just the tarballs not being
>>>>> generated at all?
>>>>
>>>> versioned tarball problem, since the source mirror won't overwrite
>>>> files with the same name, so unversioned ones are both useless and
>>>> bandwidth wasting.
>>>
>>> They seem to be working perfectly well for Yocto...
>>>
>>> They're also not bandwidth wasting
>>
>> They are, since for the kernel you need to re-upload the 300MB  
>> tarball
>> to the source mirror for mirroring to work, even if there were no  
>> real
>> changes but bitbake repacked it. I know that versioned tarballs  
>> aren't
>> terribly efficient either, but at least they are deterministic.
>
> Well, it depends on your perspective, I was looking at this from a
> user's point of view. The server's bandwidth is nothing compared to  
> the
> users pulling the data.
>
>> The angstrom source mirror checks for duplicate filenames to avoid
>> introducing checksum changes due to upstream stupidity or corrupted
>> downloads.
>
> Fair enough but you could make an exception for the git scm tarballs
> (which follow a fairly identifiable pattern).

That still opens the door for people to upload "old" tarballs with  
less revs in them. No solution is perfect, though.

>>> since the fetcher is efficient and
>>> will incrementally update something that is incomplete.
>>
>> Which won't work currently since kernel.org is down.
>
> This just means there aren't appropriate git:// -> git:// mappings in
> people's mirrors files (including Yocto). If git:// -> git:// mappings
> don't work, we should fix that but they should.

You mean something like git.kernel.org/scm/torvalds/linux-2.6.git ->  
github.com/torvalds/linux.git ?

>>> Anyhow, this is the first time someone has explained that the mirror
>>> tarball improvements that were made to bitbake have not done  
>>> everything
>>> they desire. If you want someone on the Yocto team to look at this  
>>> I'd
>>> suggest a bug report or I'll take patches to bitbake (as always).
>>
>> But you do acknowledge the problem? FWIW, I've been arguing for a
>> global method of enabled versioned tarballs since I first started
>> using oe-core (called differently back then). I know linux-yocto
>> stretches the fetcher to its limits, but I'd like this change for all
>> the recipes using a single git tree with a single srcrev.
>
> Now someone has explained the issue, yes, I can see why it is causing
> problems. I do think some of them are of people's own choosing but I'm
> trying to be flexible and not make judgement on that.
>
> So we can do something along the lines of what you after but there are
> the following considerations:
>
> a) The versioned tarballs do need to be different to what we had  
> before
> in that the tarball will be of a .git directory containing the git
> metadata, not just a checkout. Why? This is so we can incrementally
> update a git repo if the user changes the source revision as one
> example. We're dealing with the git fetcher and limiting ourselves  
> to a
> flat one dimension doesn't make sense.

Agreed on that. The bitbake patch I sent does at least do that :)

> b) We need to agree what order the tarballs on the server should be
> searched for. The "obvious" order would be:
>
> 1. Specific Revision X tarball
> 2. generic mirror tarball
> 3. git clone
>
> The problems come about when we have an existing git checkout which
> doesn't contain the revision we want.

There currently is a bug in the fetcher that makes it hard to use now  
that kernel.org is down. The situation:

I have a recipe that pulls from git (e.g. cpufrequtils.git from  
kernel.org) with SRCREV fixed to e.g. deadbeef. DL_DIR/git/ 
git.kernel.org.utils.cpufrequtils has revision deadbeef. When I do a - 
c fetch -f it will try to do an update and fail since kernel.org is  
down. I would have expected it to check the existing bare clone for  
deadbeef first and not try updating if it is present.

Anyway, I agree on the search order above.

>  Currently we can just update and
> assume that will bring in the revision. If we now support people doing
> rebases and other things, we now at least in theory have to:
>
> 1. See if the revision exists locally
> 2. Try pulling
> 3. If that doesn't work look for a versioned tarball
> 4. Blow away all fetcher git status from DL_DIR
> 5. Extract the tarball

Since git supports local fetches the above can be:

1) see if rev exists
2) try pulling
3) try versioned tarball
4) extract tarball into tmpdir, fetch objects into DL_DIR/git2 repo

That would stop things from being blown away. It does allow the  
versioned snapshots to differ, but they will always have the rev they  
were versioned with.

> My big concern with this is rather than having some general
> archictecture and standard way of doing this, the fetcher once again
> becomes a set of special case if A do B but if C do D first then do E
> but don't forget to do A if Z happens type code.

I know virtually nothing of the fetcher code, but I think treating  
every tarball as a local mirror that gets fetched into DL_DIR/git2/ 
repodir would avoid a lot of special casing.

> So what I'm saying is I don't want the fetcher to become the
> unmaintainable mess the that the code paths in fetch1 were. I don't
> honestly know how to add a code path like the above and avoid this :(
>
> The patch on the bitbake list doesn't consider any of this  
> complexity or
> the corner cases

Sadly it's the best thing I can accomplish with my python skills :( It  
does solve my #1 need, which is an easy way (ls | grep) to check for  
GPL compliance.

regards,

Koen

Re: Fetch failure for source at kernel.org

[Richard Purdie]

On Thu, 2011-09-22 at 13:27 +0200, Koen Kooi wrote:
> Op 22 sep 2011, om 12:20 heeft Richard Purdie het volgende geschreven:
> 
> > On Thu, 2011-09-22 at 10:28 +0200, Koen Kooi wrote:
> >> Op 22 sep. 2011, om 09:03 heeft Richard Purdie het volgende  
> > This just means there aren't appropriate git:// -> git:// mappings in
> > people's mirrors files (including Yocto). If git:// -> git:// mappings
> > don't work, we should fix that but they should.
> 
> You mean something like git.kernel.org/scm/torvalds/linux-2.6.git ->  
> github.com/torvalds/linux.git ?

Yes. Thinking about this, the current fetcher might not be optimised in
making this as efficient as possible but it should work and if it
doesn't, we should make it.

> > Now someone has explained the issue, yes, I can see why it is causing
> > problems. I do think some of them are of people's own choosing but I'm
> > trying to be flexible and not make judgement on that.
> >
> > So we can do something along the lines of what you after but there are
> > the following considerations:
> >
> > a) The versioned tarballs do need to be different to what we had  
> > before
> > in that the tarball will be of a .git directory containing the git
> > metadata, not just a checkout. Why? This is so we can incrementally
> > update a git repo if the user changes the source revision as one
> > example. We're dealing with the git fetcher and limiting ourselves  
> > to a
> > flat one dimension doesn't make sense.
> 
> Agreed on that. The bitbake patch I sent does at least do that :)

Right :)

I just want to be clear we're doing something different to what was
there before (and why).

> > b) We need to agree what order the tarballs on the server should be
> > searched for. The "obvious" order would be:
> >
> > 1. Specific Revision X tarball
> > 2. generic mirror tarball
> > 3. git clone
> >
> > The problems come about when we have an existing git checkout which
> > doesn't contain the revision we want.
> 
> There currently is a bug in the fetcher that makes it hard to use now  
> that kernel.org is down. The situation:
> 
> I have a recipe that pulls from git (e.g. cpufrequtils.git from  
> kernel.org) with SRCREV fixed to e.g. deadbeef. DL_DIR/git/ 
> git.kernel.org.utils.cpufrequtils has revision deadbeef. When I do a - 
> c fetch -f it will try to do an update and fail since kernel.org is  
> down. I would have expected it to check the existing bare clone for  
> deadbeef first and not try updating if it is present.

Agreed, it shouldn't be hitting the network unless the revision is
floating. That is bad :/.

> Anyway, I agree on the search order above.
> 
> >  Currently we can just update and
> > assume that will bring in the revision. If we now support people doing
> > rebases and other things, we now at least in theory have to:
> >
> > 1. See if the revision exists locally
> > 2. Try pulling
> > 3. If that doesn't work look for a versioned tarball
> > 4. Blow away all fetcher git status from DL_DIR
> > 5. Extract the tarball
> 
> Since git supports local fetches the above can be:
> 
> 1) see if rev exists
> 2) try pulling
> 3) try versioned tarball
> 4) extract tarball into tmpdir, fetch objects into DL_DIR/git2 repo
> 
> That would stop things from being blown away. It does allow the  
> versioned snapshots to differ, but they will always have the rev they  
> were versioned with.

This is a much better approach. Its quite a big difference from anything
we've done before but the approach should fit in better. I need to think
on it more...

> > My big concern with this is rather than having some general
> > archictecture and standard way of doing this, the fetcher once again
> > becomes a set of special case if A do B but if C do D first then do E
> > but don't forget to do A if Z happens type code.
> 
> I know virtually nothing of the fetcher code, but I think treating  
> every tarball as a local mirror that gets fetched into DL_DIR/git2/ 
> repodir would avoid a lot of special casing.

Yes, its nicer. We keep falling into the trap of thinking about the
non-distributed SCM cases but we have more capability with git and it
makes sense to use it.

> > So what I'm saying is I don't want the fetcher to become the
> > unmaintainable mess the that the code paths in fetch1 were. I don't
> > honestly know how to add a code path like the above and avoid this :(
> >
> > The patch on the bitbake list doesn't consider any of this  
> > complexity or
> > the corner cases
> 
> Sadly it's the best thing I can accomplish with my python skills :( It  
> does solve my #1 need, which is an easy way (ls | grep) to check for  
> GPL compliance.

Ok. Can you at least file a bug in the yocto bugzilla with a summary of
this discussion please? :) That should ensure the issue gets more
attention since with the best intentions, I can't remember everything.

Cheers,

Richard

Re: Fetch failure for source at kernel.org

[Koen Kooi]

Op 22 sep. 2011, om 13:42 heeft Richard Purdie het volgende geschreven:

[..]

> Ok. Can you at least file a bug in the yocto bugzilla with a summary of
> this discussion please? :) That should ensure the issue gets more
> attention since with the best intentions, I can't remember everything.

I filed it as http://bugzilla.pokylinux.org/show_bug.cgi?id=1511