File conflicts in DL_DIR

File conflicts in DL_DIR

[Phil Blundell]

I just ran into an annoying problem with two recipes whose upstreams
both use the same poor naming scheme for their downloadable tarballs
which causes the sources to collide in DL_DIR.  One of them is eudev,
which has:

SRC_URI = "https://github.com/gentoo/${BPN}/archive/v${PV}.tar.gz

This shows up as a source checksum error when the second package finds
what it thinks are its sources in DL_DIR, but of course they are not
the files that it wanted.

I couldn't immediately find any easy way to work around this
difficulty.  Setting "localpath=eudev_v{$PV}.tar.gz" in eudev's SRC_URI
looks superficially like it ought to do the right thing, but although
this changes the location that the fetcher will search for the
downloaded file it doesn't actually cause wget to put the file there so
this doesn't work either.

I suppose I could define DL_DIR in terms of ${PN}, which would
presumably fix this problem at the cost of causing all my previous
downloads to be repeated.  Or I could stop using eudev, or wait for one
of the two packages to release a new version. :-)  Any better
suggestions?

thanks

p.

Re: File conflicts in DL_DIR

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1693 bytes --]

On Wed, Oct 12, 2016 at 06:01:05PM +0100, Phil Blundell wrote:
> I just ran into an annoying problem with two recipes whose upstreams
> both use the same poor naming scheme for their downloadable tarballs
> which causes the sources to collide in DL_DIR.  One of them is eudev,
> which has:
> 
> SRC_URI = "https://github.com/gentoo/${BPN}/archive/v${PV}.tar.gz

Github archives are regenerated from time to time and then the checksums
are changed, so 3rd option is to use git fetcher with SRCREV matching the
has of v${PV} tag.

> This shows up as a source checksum error when the second package finds
> what it thinks are its sources in DL_DIR, but of course they are not
> the files that it wanted.
> 
> I couldn't immediately find any easy way to work around this
> difficulty.  Setting "localpath=eudev_v{$PV}.tar.gz" in eudev's SRC_URI
> looks superficially like it ought to do the right thing, but although
> this changes the location that the fetcher will search for the
> downloaded file it doesn't actually cause wget to put the file there so
> this doesn't work either.
> 
> I suppose I could define DL_DIR in terms of ${PN}, which would
> presumably fix this problem at the cost of causing all my previous
> downloads to be repeated.  Or I could stop using eudev, or wait for one
> of the two packages to release a new version. :-)  Any better
> suggestions?
> 
> thanks
> 
> p.
> 
> -- 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 169 bytes --]

Re: File conflicts in DL_DIR

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2216 bytes --]

On Wed, Oct 12, 2016 at 10:01 AM, Phil Blundell <pb@pbcl.net> wrote:

> I just ran into an annoying problem with two recipes whose upstreams
> both use the same poor naming scheme for their downloadable tarballs
> which causes the sources to collide in DL_DIR.  One of them is eudev,
> which has:
>
> SRC_URI = "https://github.com/gentoo/${BPN}/archive/v${PV}.tar.gz
>
> This shows up as a source checksum error when the second package finds
> what it thinks are its sources in DL_DIR, but of course they are not
> the files that it wanted.
>
> I couldn't immediately find any easy way to work around this
> difficulty.  Setting "localpath=eudev_v{$PV}.tar.gz" in eudev's SRC_URI
> looks superficially like it ought to do the right thing, but although
> this changes the location that the fetcher will search for the
> downloaded file it doesn't actually cause wget to put the file there so
> this doesn't work either.
>

I think that’s just a bug.. The fetcher should fully obey the url
parameters we support, so we should fix the wget fetcher to properly obey
localpath.

I suppose I could define DL_DIR in terms of ${PN}, which would
> presumably fix this problem at the cost of causing all my previous
> downloads to be repeated.  Or I could stop using eudev, or wait for one
> of the two packages to release a new version. :-)  Any better
> suggestions?
>

One option would be to change DL_DIR, but use PREMIRRORS for compatibility
with existing DL_DIRs, i.e.:
DL_DIR_BASE = “${TOPDIR}/downloads”
DL_DIR = “${DL_DIR_BASE}/${BPN}”
PREMIRRORS_prepend = “file://.*/ file://${DL_DIR_BASE}/\n “

or similar (untested).

Of course, this conflict will occur on a mirror too, so that’s a concern if
you care about mirror population.

Long term, I’d like to see our DL_DIR as a more capable download cache.
Download every file to a unique name based on its checksum, then link those
into a ${BPN}-separated area with their real names. Then add a separate
tool to populate a mirror from DL_DIR.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 3321 bytes --]

Re: File conflicts in DL_DIR

[Matt Madison]

On Wed, Oct 12, 2016 at 10:01 AM, Phil Blundell <pb@pbcl.net> wrote:
> I just ran into an annoying problem with two recipes whose upstreams
> both use the same poor naming scheme for their downloadable tarballs
> which causes the sources to collide in DL_DIR.  One of them is eudev,
> which has:
>
> SRC_URI = "https://github.com/gentoo/${BPN}/archive/v${PV}.tar.gz
>
> This shows up as a source checksum error when the second package finds
> what it thinks are its sources in DL_DIR, but of course they are not
> the files that it wanted.
>
> I couldn't immediately find any easy way to work around this
> difficulty.  Setting "localpath=eudev_v{$PV}.tar.gz" in eudev's SRC_URI
> looks superficially like it ought to do the right thing, but although
> this changes the location that the fetcher will search for the
> downloaded file it doesn't actually cause wget to put the file there so
> this doesn't work either.

I use ";downloadfilename=${BPN}-${PV}.tar.gz" to deal with this
problem, and it's worked well for me.

-Matt

>
> I suppose I could define DL_DIR in terms of ${PN}, which would
> presumably fix this problem at the cost of causing all my previous
> downloads to be repeated.  Or I could stop using eudev, or wait for one
> of the two packages to release a new version. :-)  Any better
> suggestions?
>
> thanks
>
> p.
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: File conflicts in DL_DIR

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 339 bytes --]

On 12 October 2016 at 18:30, Martin Jansa <martin.jansa@gmail.com> wrote:

> Github archives are regenerated from time to time and then the checksums
> are changed, so 3rd option is to use git fetcher with SRCREV matching the
> has of v${PV} tag.
>

Note that Github maintain that this isn't the case for .tar.gz archives.

Ross

[-- Attachment #2: Type: text/html, Size: 784 bytes --]

Re: File conflicts in DL_DIR

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1214 bytes --]

On Wed, Oct 12, 2016 at 08:11:57PM +0100, Burton, Ross wrote:
> On 12 October 2016 at 18:30, Martin Jansa <martin.jansa@gmail.com> wrote:
> 
> > Github archives are regenerated from time to time and then the checksums
> > are changed, so 3rd option is to use git fetcher with SRCREV matching the
> > has of v${PV} tag.
> >
> 
> Note that Github maintain that this isn't the case for .tar.gz archives.

I've checked git log of my layers and from last 3 cases where I was
replacing github archive with git SRC_URI it was .tar.gz in two cases and
.zip in one (and they were really failing to fetch because checksums
mismatch).

They might maintain that they don't regenerate them anymore, but if you have
older recipe with checksums which were valid 2 years ago you can still
be surprised by new checksums (I was 3 times in last year by github.com,
last month from few more cases which were still using googlecode.com and
direct release URL stopped working).

Unfortunately this always happens close (or worse shortly after) our
internal release when we switch to different premirror directory and
recipes start failing to fetch.

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 169 bytes --]

Re: File conflicts in DL_DIR

[Phil Blundell]

On Wed, 2016-10-12 at 11:03 -0700, Matt Madison wrote:
> I use ";downloadfilename=${BPN}-${PV}.tar.gz" to deal with this
> problem, and it's worked well for me.

Thanks, that works for me.

p.