OEDEM 2009 summary: Death to checksums.ini?

OEDEM 2009 summary: Death to checksums.ini?

[Phil Blundell]

The current checksums.ini arrangement has a number of issues:

 - single monolithic file is a rich source of merge conflicts
 - concrete URIs require many duplicate entries for different mirrors
 - can be annoying for folks using overlays and/or collections
 - storing the checksum separately from the rest of the .bb file makes
cherry-picking harder than it needs to be
 - large amount of churn in checksums.ini can make it hard to spot cases
where checksums were changed rather than just added.

It was proposed that, rather than storing checksums centrally in
checksums.ini, they should be placed in the individual .bb file to which
the checksum relates.  Bitbake already has a certain level of support
for reading checksums from SRC_URI and it would seem natural to try to
make use of that. 

There was some discussion around alternative proposals of storing the
checksums in separate files within the recipes/ directory.  These
proposals didn't seem to offer any real advantage over storing the
checksums within the .bb file (and, importantly, didn't really solve the
issues around recipe copying/merging) so they were not pursued further.

Some concerns were raised around .inc files and other places where
multiple recipes shared a single definition of SRC_URI, but it seems
like these can all be addressed with strategic use of variable
substitutions.

Also, some concerns were raised over the disk space impact of the
lengthened SRC_URIs.  However, the net increase in disk space usage
seems like it will be marginal at worst: many packages will actually
wind up using less space with the new arrangement.

There was a side discussion around providing a separate mechanism for a
site-local cache of checksums, to be used solely for verifying that a
particular source archive has not changed from one build to the next.
This cache would not be checked in anywhere or distributed.  RP noted
that this was the original intent of checksums.ini.  Agreed to park this
for now since it is independent of (though somewhat related to) the
topic of shared checksums.

Conclusions:

- checksums are clearly part of the metadata, it seems like they
naturally belong in the .bb file. 

- there was general acceptance that the checksums belong in SRC_URI.  

Next steps:

- figure out a way to implement sha256sum checking, either by extending
the code in bitbake's fetcher or by providing equivalent functionality
in base.bbclass

- work out a migration strategy: is it feasible to splice the existing
checksums into the SRC_URIs programmatically?  RP thinks yes.  PB
suggests leaving the existing checksums.ini as read-only and switching
to checksums incrementally for new packages.  RP: can make a git hook to
allow deletion from checksums.ini but no other changes.

Re: OEDEM 2009 summary: Death to checksums.ini?

[Holger Hans Peter Freyther]

On Tuesday 10 November 2009 17:55:40 Phil Blundell wrote:
> The current checksums.ini arrangement has a number of issues:
> 
>  - single monolithic file is a rich source of merge conflicts
>  - concrete URIs require many duplicate entries for different mirrors
>  - can be annoying for folks using overlays and/or collections
>  - storing the checksum separately from the rest of the .bb file makes
> cherry-picking harder than it needs to be
>  - large amount of churn in checksums.ini can make it hard to spot cases
> where checksums were changed rather than just added.
> 
> It was proposed that, rather than storing checksums centrally in
> checksums.ini, they should be placed in the individual .bb file to which
> the checksum relates.  Bitbake already has a certain level of support
> for reading checksums from SRC_URI and it would seem natural to try to
> make use of that.

*sigh*

SRC_URI = "http://example.org/${PN}-${PV}.tar,bz2"

how do you want to handle these? What happens if you place a checksum in the 
inc file? Do you want to propose removing SRC_URI from .ini files and put them 
back to the .bb files?


z.

Re: OEDEM 2009 summary: Death to checksums.ini?

[Frans Meulenbroeks]

2009/11/11 Holger Hans Peter Freyther <holger+oe@freyther.de>:
> On Tuesday 10 November 2009 17:55:40 Phil Blundell wrote:
>> The current checksums.ini arrangement has a number of issues:
>>
>>  - single monolithic file is a rich source of merge conflicts
>>  - concrete URIs require many duplicate entries for different mirrors
>>  - can be annoying for folks using overlays and/or collections
>>  - storing the checksum separately from the rest of the .bb file makes
>> cherry-picking harder than it needs to be
>>  - large amount of churn in checksums.ini can make it hard to spot cases
>> where checksums were changed rather than just added.
>>
>> It was proposed that, rather than storing checksums centrally in
>> checksums.ini, they should be placed in the individual .bb file to which
>> the checksum relates.  Bitbake already has a certain level of support
>> for reading checksums from SRC_URI and it would seem natural to try to
>> make use of that.
>
> *sigh*
>
> SRC_URI = "http://example.org/${PN}-${PV}.tar,bz2"
>
> how do you want to handle these? What happens if you place a checksum in the
> inc file? Do you want to propose removing SRC_URI from .ini files and put them
> back to the .bb files?

While I think the monolithic checksums.ini file has its problems (I've
cursed more than once that someone changed the fiel just wwhen I had
changed it), I think Holger has a point with ${PN}-{$PV}
One now needs to explicitly add the checksum to the recipe.

Would it be better to have a separate checksums.ini file in the
directory with the packages (and I know that for some dirs like perl
this still can become substantial)..
With some machinery we could even generate the checksum automatically
if a new package or version is added (NOT if the checksum is alreayd
there and is changed).

How about that?

Frans.

Re: OEDEM 2009 summary: Death to checksums.ini?

[Phil Blundell]

On Wed, 2009-11-11 at 02:06 +0100, Holger Hans Peter Freyther wrote:
> SRC_URI = "http://example.org/${PN}-${PV}.tar,bz2"
> 
> how do you want to handle these? What happens if you place a checksum in the 
> inc file? Do you want to propose removing SRC_URI from .ini files and put them 
> back to the .bb files?

SRC_URI = "http://example.org/${PN}-${PV}.tar,bz2;md5sum=${MD5SUM}"

You can define ${MD5SUM} whereever you like.

p.

Re: OEDEM 2009 summary: Death to checksums.ini?

[Holger Hans Peter Freyther]

On Wednesday 11 November 2009 09:24:09 Phil Blundell wrote:
> On Wed, 2009-11-11 at 02:06 +0100, Holger Hans Peter Freyther wrote:
> > SRC_URI = "http://example.org/${PN}-${PV}.tar,bz2"
> >
> > how do you want to handle these? What happens if you place a checksum in
> > the inc file? Do you want to propose removing SRC_URI from .ini files and
> > put them back to the .bb files?
> 
> SRC_URI = "http://example.org/${PN}-${PV}.tar,bz2;md5sum=${MD5SUM}"
> 
> You can define ${MD5SUM} whereever you like.

This will create an even bigger mess. Sometimes you need to download two 
things, this means you will end up with A_MD5SUM, B_MD5SUM, A_SHASUM, 
B_SHASUM. The main problem with the above is that in contrast to a well defined 
checksums.ini file we will end up with n-variants of the above trick.


I agree that conceptually the checksum belongs to the URI, but putting it into
the URI is just creating a horrible mess. It has issues with .inc files, adding 
a shasum will make the URI not fit in any terminal...

The best alternatives so far where:
	- Place the checksums into the dir of the recipe
	- Use a MD5SUM_${URL} = "", SHA256SUM_${URL} = "" syntax


z.

Re: OEDEM 2009 summary: Death to checksums.ini?

[Phil Blundell]

On Wed, 2009-11-11 at 09:44 +0100, Holger Hans Peter Freyther wrote:
> This will create an even bigger mess. Sometimes you need to download two 
> things, this means you will end up with A_MD5SUM, B_MD5SUM, A_SHASUM, 
> B_SHASUM. The main problem with the above is that in contrast to a well defined 
> checksums.ini file we will end up with n-variants of the above trick.

The number of recipes where multiple items need to be downloaded and
checksummed is small: this is a tiny minority of the total.  So,
although I agree that this case will become more ugly, I don't think
this is going to be a common enough problem that it will represent a
very big deal.

> I agree that conceptually the checksum belongs to the URI, but putting it into
> the URI is just creating a horrible mess. It has issues with .inc files, adding 
> a shasum will make the URI not fit in any terminal...
> 
> The best alternatives so far where:
> 	- Place the checksums into the dir of the recipe
> 	- Use a MD5SUM_${URL} = "", SHA256SUM_${URL} = "" syntax

I would be happy with the latter of those suggestions.  I don't think
the former really addresses the problems with the current checksums.ini.

p.

Re: OEDEM 2009 summary: Death to checksums.ini?

[Frans Meulenbroeks]

2009/11/11 Phil Blundell <philb@gnu.org>:
> On Wed, 2009-11-11 at 09:44 +0100, Holger Hans Peter Freyther wrote:
>> This will create an even bigger mess. Sometimes you need to download two
>> things, this means you will end up with A_MD5SUM, B_MD5SUM, A_SHASUM,
>> B_SHASUM. The main problem with the above is that in contrast to a well defined
>> checksums.ini file we will end up with n-variants of the above trick.
>
> The number of recipes where multiple items need to be downloaded and
> checksummed is small: this is a tiny minority of the total.  So,
> although I agree that this case will become more ugly, I don't think
> this is going to be a common enough problem that it will represent a
> very big deal.
>
>> I agree that conceptually the checksum belongs to the URI, but putting it into
>> the URI is just creating a horrible mess. It has issues with .inc files, adding
>> a shasum will make the URI not fit in any terminal...
>>
>> The best alternatives so far where:
>>       - Place the checksums into the dir of the recipe
>>       - Use a MD5SUM_${URL} = "", SHA256SUM_${URL} = "" syntax
>
> I would be happy with the latter of those suggestions.  I don't think
> the former really addresses the problems with the current checksums.ini.

I'd like things to work with minimal effort, so I would encourage a
solution where, if you create a new recipe, the resulting checksums
are automatically added (automating the dumb and error prone cut &
paste process).
In that view having the checksums in a separate file would help
(although it will require adding a file to the commit which can be
forgotten).

Frans.

Re: OEDEM 2009 summary: Death to checksums.ini?

[Holger Hans Peter Freyther]

On Wednesday 11 November 2009 10:17:42 Phil Blundell wrote:
> On Wed, 2009-11-11 at 09:44 +0100, Holger Hans Peter Freyther wrote:
> > This will create an even bigger mess. Sometimes you need to download two
> > things, this means you will end up with A_MD5SUM, B_MD5SUM, A_SHASUM,
> > B_SHASUM. The main problem with the above is that in contrast to a well
> > defined checksums.ini file we will end up with n-variants of the above
> > trick.
> 
> The number of recipes where multiple items need to be downloaded and
> checksummed is small: this is a tiny minority of the total.  So,
> although I agree that this case will become more ugly, I don't think
> this is going to be a common enough problem that it will represent a
> very big deal.

It depends. Currently the instructions to update the checksums.ini are 
straight forward and branch free, with the above it is impossible to write a 
branch free documentation that everyone can follow. :)


PS: Let me come up with a patch introducing the new variables..

Re: OEDEM 2009 summary: Death to checksums.ini?

[Andrea Adami]

>>There was some discussion around alternative proposals of storing the
>> checksums in separate files within the recipes/ directory.

>> The best alternatives so far where:
>>       - Place the checksums into the dir of the recipe
>>       - Use a MD5SUM_${URL} = "", SHA256SUM_${URL} = "" syntax
>
> I would be happy with the latter of those suggestions.  I don't think
> the former really addresses the problems with the current checksums.ini.

Yes, I sponsor the idea of a single Manifest in each dir.

Let apart filetype specifier, example here:
http://www.gentoo.org/proj/en/glep/glep-0044-extras/manifest2-example.txt

Having a Manifest listing the filename, the size and some checksums on
a single line seems to me very easy to parse for an automated script
wich could refresh/generate the whole file.

Finally, a question: why do we have full URL in the checksum-file?
I can't see the real advantages in case of checksum mismatch...

My (euro)2 cents

Andrea

Re: OEDEM 2009 summary: Death to checksums.ini?

[Michael 'Mickey' Lauer]

Am Mittwoch, den 11.11.2009, 09:17 +0000 schrieb Phil Blundell:
> On Wed, 2009-11-11 at 09:44 +0100, Holger Hans Peter Freyther wrote:
> > This will create an even bigger mess. Sometimes you need to download two 
> > things, this means you will end up with A_MD5SUM, B_MD5SUM, A_SHASUM, 
> > B_SHASUM. The main problem with the above is that in contrast to a well defined 
> > checksums.ini file we will end up with n-variants of the above trick.
> 
> The number of recipes where multiple items need to be downloaded and
> checksummed is small: this is a tiny minority of the total.  So,
> although I agree that this case will become more ugly, I don't think
> this is going to be a common enough problem that it will represent a
> very big deal.
> 
> > I agree that conceptually the checksum belongs to the URI, but putting it into
> > the URI is just creating a horrible mess. It has issues with .inc files, adding 
> > a shasum will make the URI not fit in any terminal...
> > 
> > The best alternatives so far where:
> > 	- Place the checksums into the dir of the recipe
> > 	- Use a MD5SUM_${URL} = "", SHA256SUM_${URL} = "" syntax
> 
> I would be happy with the latter of those suggestions.  I don't think
> the former really addresses the problems with the current checksums.ini.

I agree here, the latter syntax seems well.

:M:

Re: OEDEM 2009 summary: Death to checksums.ini?

[Phil Blundell]

On Wed, 2009-11-11 at 10:43 +0100, Holger Hans Peter Freyther wrote:
> On Wednesday 11 November 2009 10:17:42 Phil Blundell wrote:
> > On Wed, 2009-11-11 at 09:44 +0100, Holger Hans Peter Freyther wrote:
> > > This will create an even bigger mess. Sometimes you need to download two
> > > things, this means you will end up with A_MD5SUM, B_MD5SUM, A_SHASUM,
> > > B_SHASUM. The main problem with the above is that in contrast to a well
> > > defined checksums.ini file we will end up with n-variants of the above
> > > trick.
> > 
> > The number of recipes where multiple items need to be downloaded and
> > checksummed is small: this is a tiny minority of the total.  So,
> > although I agree that this case will become more ugly, I don't think
> > this is going to be a common enough problem that it will represent a
> > very big deal.
> 
> It depends. Currently the instructions to update the checksums.ini are 
> straight forward and branch free, with the above it is impossible to write a 
> branch free documentation that everyone can follow. :)

True.  On the other hand, if you are making a recipe with multiple
downloaded tarballs then you are already some way outside what can be
achieved using a straightforward "painting by numbers" approach.  I'm
not sure that the checksums make a material difference to that.

> PS: Let me come up with a patch introducing the new variables..

That'd be great.

p.

Re: OEDEM 2009 summary: Death to checksums.ini?

[Holger Hans Peter Freyther]

On Wednesday 11 November 2009 14:31:21 Phil Blundell wrote:
> On Wed, 2009-11-11 at 10:43 +0100, Holger Hans Peter Freyther wrote:
> > On Wednesday 11 November 2009 10:17:42 Phil Blundell wrote:
> > > On Wed, 2009-11-11 at 09:44 +0100, Holger Hans Peter Freyther wrote:

> > It depends. Currently the instructions to update the checksums.ini are
> > straight forward and branch free, with the above it is impossible to
> > write a branch free documentation that everyone can follow. :)

True

Re: OEDEM 2009 summary: Death to checksums.ini?

[Holger Hans Peter Freyther]

On Wednesday 11 November 2009 14:37:46 Holger Hans Peter Freyther wrote:
> On Wednesday 11 November 2009 14:31:21 Phil Blundell wrote:
> > On Wed, 2009-11-11 at 10:43 +0100, Holger Hans Peter Freyther wrote:
> > > On Wednesday 11 November 2009 10:17:42 Phil Blundell wrote:
> > > > On Wed, 2009-11-11 at 09:44 +0100, Holger Hans Peter Freyther wrote:
> > >
> > > It depends. Currently the instructions to update the checksums.ini are
> > > straight forward and branch free, with the above it is impossible to
> > > write a branch free documentation that everyone can follow. :)
> 
> True

ouch, it is a good sign to go to bed... I have quoted myself now.. 

z.

Re: OEDEM 2009 summary: Death to checksums.ini?

[Richard Purdie]

On Wed, 2009-11-11 at 09:44 +0100, Holger Hans Peter Freyther wrote:
> On Wednesday 11 November 2009 09:24:09 Phil Blundell wrote:
> > On Wed, 2009-11-11 at 02:06 +0100, Holger Hans Peter Freyther wrote:
> > > SRC_URI = "http://example.org/${PN}-${PV}.tar,bz2"
> > >
> > > how do you want to handle these? What happens if you place a checksum in
> > > the inc file? Do you want to propose removing SRC_URI from .ini files and
> > > put them back to the .bb files?
> > 
> > SRC_URI = "http://example.org/${PN}-${PV}.tar,bz2;md5sum=${MD5SUM}"
> > 
> > You can define ${MD5SUM} whereever you like.
> 
> This will create an even bigger mess. Sometimes you need to download two 
> things, this means you will end up with A_MD5SUM, B_MD5SUM, A_SHASUM, 
> B_SHASUM. The main problem with the above is that in contrast to a well defined 
> checksums.ini file we will end up with n-variants of the above trick.
> 
> 
> I agree that conceptually the checksum belongs to the URI, but putting it into
> the URI is just creating a horrible mess. It has issues with .inc files, adding 
> a shasum will make the URI not fit in any terminal...
> 
> The best alternatives so far where:
> 	- Place the checksums into the dir of the recipe
> 	- Use a MD5SUM_${URL} = "", SHA256SUM_${URL} = "" syntax

Might I also propose:

SCR_URI = "xyz://abc.com/efg.tgz;name=bar"

MD5SUM_bar = ""

or maybe :

MD5SUM[bar] = ""

Cheers,

Richard

Re: OEDEM 2009 summary: Death to checksums.ini?

[Phil Blundell]

On Wed, 2009-11-11 at 14:17 +0000, Richard Purdie wrote:
> SCR_URI = "xyz://abc.com/efg.tgz;name=bar"
> 
> MD5SUM_bar = ""
> 
> or maybe :
> 
> MD5SUM[bar] = ""

The labelling is a good idea, although I am not especially keen on
either of those notations.  How about:

SRC_URI = "xyz://abc.com/efg.tgz;name=bar"
SRC_URI[md5sum:bar] = "..."
SRC_URI[sha256sum:bar] = "..."

p.

Re: OEDEM 2009 summary: Death to checksums.ini?

[Frans Meulenbroeks]

2009/11/11 Phil Blundell <philb@gnu.org>:
> On Wed, 2009-11-11 at 14:17 +0000, Richard Purdie wrote:
>> SCR_URI = "xyz://abc.com/efg.tgz;name=bar"
>>
>> MD5SUM_bar = ""
>>
>> or maybe :
>>
>> MD5SUM[bar] = ""
>
> The labelling is a good idea, although I am not especially keen on
> either of those notations.  How about:
>
> SRC_URI = "xyz://abc.com/efg.tgz;name=bar"
> SRC_URI[md5sum:bar] = "..."
> SRC_URI[sha256sum:bar] = "..."
>
Would we still be able to use PN and PV in the md5sum:bar etc?

Re: OEDEM 2009 summary: Death to checksums.ini?

[Phil Blundell]

On Wed, 2009-11-11 at 17:01 +0100, Frans Meulenbroeks wrote:
> 2009/11/11 Phil Blundell <philb@gnu.org>:
> > On Wed, 2009-11-11 at 14:17 +0000, Richard Purdie wrote:
> >> SCR_URI = "xyz://abc.com/efg.tgz;name=bar"
> >>
> >> MD5SUM_bar = ""
> >>
> >> or maybe :
> >>
> >> MD5SUM[bar] = ""
> >
> > The labelling is a good idea, although I am not especially keen on
> > either of those notations.  How about:
> >
> > SRC_URI = "xyz://abc.com/efg.tgz;name=bar"
> > SRC_URI[md5sum:bar] = "..."
> > SRC_URI[sha256sum:bar] = "..."
> >
> Would we still be able to use PN and PV in the md5sum:bar etc?

Not directly, though it isn't entirely obvious to me why this would be a
useful thing to do.  Did you have a situation in mind where this would
be desirable?

p.

Re: OEDEM 2009 summary: Death to checksums.ini?

[Frans Meulenbroeks]

2009/11/11 Phil Blundell <pb@reciva.com>:
> On Wed, 2009-11-11 at 17:01 +0100, Frans Meulenbroeks wrote:
>> 2009/11/11 Phil Blundell <philb@gnu.org>:
>> > On Wed, 2009-11-11 at 14:17 +0000, Richard Purdie wrote:
>> >> SCR_URI = "xyz://abc.com/efg.tgz;name=bar"
>> >>
>> >> MD5SUM_bar = ""
>> >>
>> >> or maybe :
>> >>
>> >> MD5SUM[bar] = ""
>> >
>> > The labelling is a good idea, although I am not especially keen on
>> > either of those notations.  How about:
>> >
>> > SRC_URI = "xyz://abc.com/efg.tgz;name=bar"
>> > SRC_URI[md5sum:bar] = "..."
>> > SRC_URI[sha256sum:bar] = "..."
>> >
>> Would we still be able to use PN and PV in the md5sum:bar etc?
>
> Not directly, though it isn't entirely obvious to me why this would be a
> useful thing to do.  Did you have a situation in mind where this would
> be desirable?
>
I am somewhat lazy so if a new version of a recipe has to be made I'd
like to touch as little as possible.
(the less I have to do the less chance of an error occurring)
So preferably I would use PV an PN in the url and not copy it from
somewhere (as that is more error prone too.

(btw I can also envision a new bitbake task which will rewrite the checksums).

Frans

Re: OEDEM 2009 summary: Death to checksums.ini?

[Esben Haabendal]

On Wed, Nov 11, 2009 at 10:17 AM, Phil Blundell <philb@gnu.org> wrote:
> On Wed, 2009-11-11 at 09:44 +0100, Holger Hans Peter Freyther wrote:
>> This will create an even bigger mess. Sometimes you need to download two
>> things, this means you will end up with A_MD5SUM, B_MD5SUM, A_SHASUM,
>> B_SHASUM. The main problem with the above is that in contrast to a well defined
>> checksums.ini file we will end up with n-variants of the above trick.
>
> The number of recipes where multiple items need to be downloaded and
> checksummed is small: this is a tiny minority of the total.  So,
> although I agree that this case will become more ugly, I don't think
> this is going to be a common enough problem that it will represent a
> very big deal.

That seems to depend on how must weigth you put on clean vs. "messy"
solutions.

>> I agree that conceptually the checksum belongs to the URI, but putting it into
>> the URI is just creating a horrible mess. It has issues with .inc files, adding
>> a shasum will make the URI not fit in any terminal...
>>
>> The best alternatives so far where:
>>       - Place the checksums into the dir of the recipe
>>       - Use a MD5SUM_${URL} = "", SHA256SUM_${URL} = "" syntax
>
> I would be happy with the latter of those suggestions.  I don't think
> the former really addresses the problems with the current checksums.ini.

Which part of the current checksums.ini problems does the former not resolve?

Isn't the main issue with checksums.ini is the merge nightmare it poses?
I don't see how this will be a big problem with per recipe checksums
files.

A per recipe checksums file is also the one solution I have seen
that would reduce manual work needed for OE developers to
actually maintain checksums.  And face it, we are for the most part lazy :-D

As I also made clear on OEDEM, I think it is important to make things
simpler and easier to work with.  Not cause more mess and come
up with additional hurdless for OE developers.

/Esben

Re: OEDEM 2009 summary: Death to checksums.ini?

[Phil Blundell]

On Wed, 2009-11-11 at 18:37 +0100, Frans Meulenbroeks wrote:
> 2009/11/11 Phil Blundell <pb@reciva.com>:
> > On Wed, 2009-11-11 at 17:01 +0100, Frans Meulenbroeks wrote:
> >> 2009/11/11 Phil Blundell <philb@gnu.org>:
> >> > On Wed, 2009-11-11 at 14:17 +0000, Richard Purdie wrote:
> >> >> SCR_URI = "xyz://abc.com/efg.tgz;name=bar"
> >> >>
> >> >> MD5SUM_bar = ""
> >> >>
> >> >> or maybe :
> >> >>
> >> >> MD5SUM[bar] = ""
> >> >
> >> > The labelling is a good idea, although I am not especially keen on
> >> > either of those notations.  How about:
> >> >
> >> > SRC_URI = "xyz://abc.com/efg.tgz;name=bar"
> >> > SRC_URI[md5sum:bar] = "..."
> >> > SRC_URI[sha256sum:bar] = "..."
> >> >
> >> Would we still be able to use PN and PV in the md5sum:bar etc?
> >
> > Not directly, though it isn't entirely obvious to me why this would be a
> > useful thing to do.  Did you have a situation in mind where this would
> > be desirable?
> >
> I am somewhat lazy so if a new version of a recipe has to be made I'd
> like to touch as little as possible.
> (the less I have to do the less chance of an error occurring)
> So preferably I would use PV an PN in the url and not copy it from
> somewhere (as that is more error prone too.

That should still work fine.  The purpose of the "name=bar" thing is to
decouple the checksum from the exact URI in use.  So, you can write:

SRC_URI = "${MIRROR}/somedir/${PN}-${PV}.tar.gz;name=mytarball \
           ${OTHERMIRROR}/${PN}-${PV}-extra-files.tar.gz;name=extrafiles"

and then, in another file if you prefer, add:

SRC_URI[md5sum:mytarball] = "d3b07384d113edec49eaa6238ad5ff00"
SRC_URI[md5sum:extrafiles] = "c157a79031e1c40f85931829bc5fc552"

p.

Re: OEDEM 2009 summary: Death to checksums.ini?

[Frans Meulenbroeks]

2009/11/11 Phil Blundell <philb@gnu.org>:
> On Wed, 2009-11-11 at 18:37 +0100, Frans Meulenbroeks wrote:
>> 2009/11/11 Phil Blundell <pb@reciva.com>:
>> > On Wed, 2009-11-11 at 17:01 +0100, Frans Meulenbroeks wrote:
>> >> 2009/11/11 Phil Blundell <philb@gnu.org>:
>> >> > On Wed, 2009-11-11 at 14:17 +0000, Richard Purdie wrote:
>> >> >> SCR_URI = "xyz://abc.com/efg.tgz;name=bar"
>> >> >>
>> >> >> MD5SUM_bar = ""
>> >> >>
>> >> >> or maybe :
>> >> >>
>> >> >> MD5SUM[bar] = ""
>> >> >
>> >> > The labelling is a good idea, although I am not especially keen on
>> >> > either of those notations.  How about:
>> >> >
>> >> > SRC_URI = "xyz://abc.com/efg.tgz;name=bar"
>> >> > SRC_URI[md5sum:bar] = "..."
>> >> > SRC_URI[sha256sum:bar] = "..."
>> >> >
>> >> Would we still be able to use PN and PV in the md5sum:bar etc?
>> >
>> > Not directly, though it isn't entirely obvious to me why this would be a
>> > useful thing to do.  Did you have a situation in mind where this would
>> > be desirable?
>> >
>> I am somewhat lazy so if a new version of a recipe has to be made I'd
>> like to touch as little as possible.
>> (the less I have to do the less chance of an error occurring)
>> So preferably I would use PV an PN in the url and not copy it from
>> somewhere (as that is more error prone too.
>
> That should still work fine.  The purpose of the "name=bar" thing is to
> decouple the checksum from the exact URI in use.  So, you can write:
>
> SRC_URI = "${MIRROR}/somedir/${PN}-${PV}.tar.gz;name=mytarball \
>           ${OTHERMIRROR}/${PN}-${PV}-extra-files.tar.gz;name=extrafiles"
>
> and then, in another file if you prefer, add:
>
> SRC_URI[md5sum:mytarball] = "d3b07384d113edec49eaa6238ad5ff00"
> SRC_URI[md5sum:extrafiles] = "c157a79031e1c40f85931829bc5fc552"
>

Ah ok, now I understand. I thought at the place of bar I would get the
url, not a name tag associated to the url.

Thanks for clarifying , Frans

Re: OEDEM 2009 summary: Death to checksums.ini?

[Ricardo Salveti de Araujo]

On Wed, Nov 11, 2009 at 7:06 AM, Andrea Adami <andrea.adami@gmail.com> wrote:
>>>There was some discussion around alternative proposals of storing the
>>> checksums in separate files within the recipes/ directory.
>
>>> The best alternatives so far where:
>>>       - Place the checksums into the dir of the recipe
>>>       - Use a MD5SUM_${URL} = "", SHA256SUM_${URL} = "" syntax
>>
>> I would be happy with the latter of those suggestions.  I don't think
>> the former really addresses the problems with the current checksums.ini.
>
> Yes, I sponsor the idea of a single Manifest in each dir.
>
> Let apart filetype specifier, example here:
> http://www.gentoo.org/proj/en/glep/glep-0044-extras/manifest2-example.txt
>
> Having a Manifest listing the filename, the size and some checksums on
> a single line seems to me very easy to parse for an automated script
> wich could refresh/generate the whole file.

I also like the way Gentoo handle with this problem, but I guess that
if we go that way we'd still have trouble regarding merging and
conflicts, like we current have with checksums.

Adding the checksum metadata in the recipe itself makes easier the
life of a lot of people, we just need to identify the easier and
prettier way of doing that.

Cheers,
-- 
Ricardo Salveti de Araujo

Re: OEDEM 2009 summary: Death to checksums.ini?

[Phil Blundell]

[-- Attachment #1: Type: text/plain, Size: 846 bytes --]

On Wed, 2009-11-11 at 20:34 +0000, Phil Blundell wrote:
> That should still work fine.  The purpose of the "name=bar" thing is to
> decouple the checksum from the exact URI in use.  So, you can write:
> 
> SRC_URI = "${MIRROR}/somedir/${PN}-${PV}.tar.gz;name=mytarball \
>            ${OTHERMIRROR}/${PN}-${PV}-extra-files.tar.gz;name=extrafiles"
> 
> and then, in another file if you prefer, add:
> 
> SRC_URI[md5sum:mytarball] = "d3b07384d113edec49eaa6238ad5ff00"
> SRC_URI[md5sum:extrafiles] = "c157a79031e1c40f85931829bc5fc552"

It turns out that colons in flag names aren't actually accepted by the
parser, but something similar with dots is OK.  So, here's a patch to
make it work.  The modified notation is actually:

SRC_URI[mytarball.md5sum] = "d3b07384d113edec49eaa6238ad5ff00"

and likewise for sha256sum.

p.



[-- Attachment #2: base.patch --]
[-- Type: text/x-patch, Size: 3057 bytes --]

diff --git a/classes/base.bbclass b/classes/base.bbclass
index 9d063f2..9508754 100644
--- a/classes/base.bbclass
+++ b/classes/base.bbclass
@@ -56,6 +56,50 @@ def base_chk_load_parser(config_paths):
 
     return parser
 
+def base_chk_file_vars(parser, localpath, params, data):
+    try:
+        name = params["name"]
+    except KeyError:
+        return False
+    flagName = "%s.md5sum" % name
+    want_md5sum = bb.data.getVarFlag("SRC_URI", flagName, data)
+    flagName = "%s.sha256sum" % name
+    want_sha256sum = bb.data.getVarFlag("SRC_URI", flagName, data)
+
+    if (want_sha256sum == None and want_md5sum == None):
+        # no checksums to check, nothing to do
+        return False
+
+    if not os.path.exists(localpath):
+        localpath = base_path_out(localpath, data)
+        bb.note("The localpath does not exist '%s'" % localpath)
+        raise Exception("The path does not exist '%s'" % localpath)
+
+    if want_md5sum:
+        try:
+	    md5pipe = os.popen('PATH=%s md5sum %s' % (bb.data.getVar('PATH', data, True), localpath))
+            md5data = (md5pipe.readline().split() or [ "" ])[0]
+            md5pipe.close()
+        except OSError, e:
+            raise Exception("Executing md5sum failed")
+        if want_md5sum != md5data:
+            bb.note("The MD5Sums did not match. Wanted: '%s' and Got: '%s'" % (want_md5sum, md5data))
+            raise Exception("MD5 Sums do not match. Wanted: '%s' Got: '%s'" % (want_md5sum, md5data))
+
+    if want_sha256sum:
+        try:
+            shapipe = os.popen('PATH=%s oe_sha256sum %s' % (bb.data.getVar('PATH', data, True), localpath))
+            shadata = (shapipe.readline().split() or [ "" ])[0]
+            shapipe.close()
+        except OSError, e:
+            raise Exception("Executing shasum failed")
+        if want_sha256sum != sha256data:
+            bb.note("The SHA256Sums did not match. Wanted: '%s' and Got: '%s'" % (want_sha256sum, sha256data))
+            raise Exception("SHA256 Sums do not match. Wanted: '%s' Got: '%s'" % (want_sha256sum, sha256data))
+
+    return True
+
+
 def base_chk_file(parser, pn, pv, src_uri, localpath, data):
     no_checksum = False
     # Try PN-PV-SRC_URI first and then try PN-SRC_URI
@@ -639,11 +683,11 @@ python base_do_fetch() {
 	# Check each URI
 	for url in src_uri.split():
 		localpath = bb.data.expand(bb.fetch.localpath(url, localdata), localdata)
-		(type,host,path,_,_,_) = bb.decodeurl(url)
+		(type,host,path,_,_,params) = bb.decodeurl(url)
 		uri = "%s://%s%s" % (type,host,path)
 		try:
 			if type == "http" or type == "https" or type == "ftp" or type == "ftps":
-				if not base_chk_file(parser, pn, pv,uri, localpath, d):
+				if not (base_chk_file_vars(parser, localpath, params, d) or base_chk_file(parser, pn, pv,uri, localpath, d)):
 					if not bb.data.getVar("OE_ALLOW_INSECURE_DOWNLOADS",d, True):
 						bb.fatal("%s-%s: %s has no entry in conf/checksums.ini, not checking URI" % (pn,pv,uri))
 					else:

Re: OEDEM 2009 summary: Death to checksums.ini?

[Bernhard Reutner-Fischer]

On Thu, Nov 12, 2009 at 06:36:02PM +0000, Phil Blundell wrote:
>On Wed, 2009-11-11 at 20:34 +0000, Phil Blundell wrote:

>diff --git a/classes/base.bbclass b/classes/base.bbclass
>index 9d063f2..9508754 100644
>--- a/classes/base.bbclass
>+++ b/classes/base.bbclass
>@@ -56,6 +56,50 @@ def base_chk_load_parser(config_paths):
> 
>     return parser
> 
>+def base_chk_file_vars(parser, localpath, params, data):
>+    try:
>+        name = params["name"]
>+    except KeyError:
>+        return False
>+    flagName = "%s.md5sum" % name
>+    want_md5sum = bb.data.getVarFlag("SRC_URI", flagName, data)
>+    flagName = "%s.sha256sum" % name
>+    want_sha256sum = bb.data.getVarFlag("SRC_URI", flagName, data)
>+
>+    if (want_sha256sum == None and want_md5sum == None):
>+        # no checksums to check, nothing to do
>+        return False
>+
>+    if not os.path.exists(localpath):
>+        localpath = base_path_out(localpath, data)
>+        bb.note("The localpath does not exist '%s'" % localpath)
>+        raise Exception("The path does not exist '%s'" % localpath)
>+
>+    if want_md5sum:
>+        try:
>+	    md5pipe = os.popen('PATH=%s md5sum %s' % (bb.data.getVar('PATH', data, True), localpath))
>+            md5data = (md5pipe.readline().split() or [ "" ])[0]
>+            md5pipe.close()
>+        except OSError, e:
>+            raise Exception("Executing md5sum failed")
>+        if want_md5sum != md5data:
>+            bb.note("The MD5Sums did not match. Wanted: '%s' and Got: '%s'" % (want_md5sum, md5data))

please s/and Got/and got/g

>+            raise Exception("MD5 Sums do not match. Wanted: '%s' Got: '%s'" % (want_md5sum, md5data))

Why are you providing the same information as a note and in the
exception text?
>+
>+    if want_sha256sum:
>+        try:
>+            shapipe = os.popen('PATH=%s oe_sha256sum %s' % (bb.data.getVar('PATH', data, True), localpath))
>+            shadata = (shapipe.readline().split() or [ "" ])[0]
>+            shapipe.close()
>+        except OSError, e:
>+            raise Exception("Executing shasum failed")

sha256sum
[]
>@@ -639,11 +683,11 @@ python base_do_fetch() {
> 	# Check each URI
> 	for url in src_uri.split():
> 		localpath = bb.data.expand(bb.fetch.localpath(url, localdata), localdata)
>-		(type,host,path,_,_,_) = bb.decodeurl(url)
>+		(type,host,path,_,_,params) = bb.decodeurl(url)
> 		uri = "%s://%s%s" % (type,host,path)
> 		try:
> 			if type == "http" or type == "https" or type == "ftp" or type == "ftps":

if type in ["http", "https", "ftp", "ftps"]:
(while you're at it :)

cheers,
Bernhard

Re: OEDEM 2009 summary: Death to checksums.ini?

[Phil Blundell]

On Thu, 2009-11-12 at 20:18 +0100, Bernhard Reutner-Fischer wrote:
> >+            raise Exception("MD5 Sums do not match. Wanted: '%s' Got: '%s'" % (want_md5sum, md5data))
> 
> Why are you providing the same information as a note and in the
> exception text?

This is the way those diagnostics have always been done (see the
existing code in base.bbclass).  In practice I don't think the exception
text is used for anything but it seems fairly harmless to go on
providing it.  If we're going to change that then it should presumably
be done for both forms of the checksum code, not just for this one.

p.

Re: OEDEM 2009 summary: Death to checksums.ini?

[Phil Blundell]

On Thu, 2009-11-12 at 18:36 +0000, Phil Blundell wrote:
> It turns out that colons in flag names aren't actually accepted by the
> parser, but something similar with dots is OK.  So, here's a patch to
> make it work.  The modified notation is actually:
> 
> SRC_URI[mytarball.md5sum] = "d3b07384d113edec49eaa6238ad5ff00"

I've committed this patch now.  Also, by way of example, I've converted
busybox 1.14.3 to use the new notation.

Share and enjoy.

p.

Re: OEDEM 2009 summary: Death to checksums.ini?

[Holger Hans Peter Freyther]

On Friday 13 November 2009 18:47:04 Phil Blundell wrote:
> On Thu, 2009-11-12 at 18:36 +0000, Phil Blundell wrote:
> > It turns out that colons in flag names aren't actually accepted by the
> > parser, but something similar with dots is OK.  So, here's a patch to
> > make it work.  The modified notation is actually:
> >
> > SRC_URI[mytarball.md5sum] = "d3b07384d113edec49eaa6238ad5ff00"
> 
> I've committed this patch now.  Also, by way of example, I've converted
> busybox 1.14.3 to use the new notation.


thanks