[Buildroot] Different site methods for the main package and patches

[Buildroot] Different site methods for the main package and patches

[Frank Hunleth]

I've run into an issue where I'd like to retrieve a package using git
and then patch it using a diff file from another site. In my case, the
package is the Linux kernel.

This is possible to reproduce using the raspberrypi_defconfig and
adding the BR2_LINUX_KERNEL_PATCH line below. Here are the relevant
lines:

BR2_LINUX_KERNEL=y
BR2_LINUX_KERNEL_CUSTOM_GIT=y
BR2_LINUX_KERNEL_CUSTOM_REPO_URL="git://github.com/raspberrypi/linux.git"
BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION="3bff11d4d4b8dc28cb9ce81449c989466ba27198"
BR2_LINUX_KERNEL_PATCH="http://download.filesystems.org/unionfs/unionfs-2.x-latest/unionfs-2.5.12_for_3.10.21.diff.gz"

The call that fails is in package/pkg-download.mk in DOWNLOAD_INNER.
There's a test whether LINUX_SITE_METHOD is defined (which it is) and
to use it for the $scheme. This is correct to download the kernel, but
incorrect when going through the kernel patches. If I could undefine
LINUX_SITE_METHOD, my use case would work since the method would be
determined via the URI, but obviously other use cases would break.

For patches, I'd almost think that the <package>_SITE_METHOD shouldn't
be used at all, since multiple patches could be specified that come
from different places. I don't know what the ramifications of that
change would be, though, or if there's a simpler solution.

Any ideas?

Thanks,
Frank

[Buildroot] Different site methods for the main package and patches

[Arnout Vandecappelle]

On 24/03/14 01:47, Frank Hunleth wrote:
> I've run into an issue where I'd like to retrieve a package using git
> and then patch it using a diff file from another site. In my case, the
> package is the Linux kernel.
> 
> This is possible to reproduce using the raspberrypi_defconfig and
> adding the BR2_LINUX_KERNEL_PATCH line below. Here are the relevant
> lines:
> 
> BR2_LINUX_KERNEL=y
> BR2_LINUX_KERNEL_CUSTOM_GIT=y
> BR2_LINUX_KERNEL_CUSTOM_REPO_URL="git://github.com/raspberrypi/linux.git"
> BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION="3bff11d4d4b8dc28cb9ce81449c989466ba27198"
> BR2_LINUX_KERNEL_PATCH="http://download.filesystems.org/unionfs/unionfs-2.x-latest/unionfs-2.5.12_for_3.10.21.diff.gz"
> 
> The call that fails is in package/pkg-download.mk in DOWNLOAD_INNER.
> There's a test whether LINUX_SITE_METHOD is defined (which it is) and
> to use it for the $scheme. This is correct to download the kernel, but
> incorrect when going through the kernel patches. If I could undefine
> LINUX_SITE_METHOD, my use case would work since the method would be
> determined via the URI, but obviously other use cases would break.
> 
> For patches, I'd almost think that the <package>_SITE_METHOD shouldn't
> be used at all, since multiple patches could be specified that come
> from different places. I don't know what the ramifications of that
> change would be, though, or if there's a simpler solution.

 You're probably right about that. The problem is that the DOWNLOAD macro
cannot know if it is given a patch or the package itself. So it should
probably be changed to get _SITE_METHOD as an optional argument.

 Yann, perhaps you could take that up into your download method rework
series?

 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] Different site methods for the main package and patches

[Thomas Petazzoni]

Dear Arnout Vandecappelle,

On Wed, 26 Mar 2014 18:46:16 +0100, Arnout Vandecappelle wrote:

>  You're probably right about that. The problem is that the DOWNLOAD macro
> cannot know if it is given a patch or the package itself. So it should
> probably be changed to get _SITE_METHOD as an optional argument.
> 
>  Yann, perhaps you could take that up into your download method rework
> series?

Generally speaking, I'm a bit unhappy about how things work on the
download front:

 *) I don't really understand why we have a separation between _SITE
    and _SOURCE, and why our infra assumes that _PATCH is relative to
    _SITE. Why don't we simply make _SOURCE and _PATCH full URLs
    instead? This way _PATCH can list multiple patches coming from
    different locations, not necessarily the upstream location of the
    package.

 *) Why is the DOWNLOAD macro looking at _SITE_METHOD ? It believe it
    should be a macro that remains independent of the package
    infrastructure, and it should simply take as argument the
    information that it needs, rather than poking directly into
    _SITE_METHOD.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Different site methods for the main package and patches

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

Hi,

 > Generally speaking, I'm a bit unhappy about how things work on the
 > download front:

 >  *) I don't really understand why we have a separation between _SITE
 >     and _SOURCE, and why our infra assumes that _PATCH is relative to
 >     _SITE. Why don't we simply make _SOURCE and _PATCH full URLs
 >     instead? This way _PATCH can list multiple patches coming from
 >     different locations, not necessarily the upstream location of the
 >     package.

Some of it is historical, but it also makes it easier to support
the primary/backup site features.

-- 
Bye, Peter Korsgaard

[Buildroot] Different site methods for the main package and patches

[rjbarnet at rockwellcollins.com]

Peter/Thomas/Yann,

Peter Korsgaard <jacmet@uclibc.org> wrote on 03/26/2014 05:42:20 PM:

> >>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> 
writes:
> 
> Hi,
> 
>  > Generally speaking, I'm a bit unhappy about how things work on the
>  > download front:
> 
>  >  *) I don't really understand why we have a separation between _SITE
>  >     and _SOURCE, and why our infra assumes that _PATCH is relative to
>  >     _SITE. Why don't we simply make _SOURCE and _PATCH full URLs
>  >     instead? This way _PATCH can list multiple patches coming from
>  >     different locations, not necessarily the upstream location of the
>  >     package.
> 
> Some of it is historical, but it also makes it easier to support
> the primary/backup site features.

The primary/backup site feature is useful when using Buildroot on build 
machines behind a firewall. You are able to create and host a local mirror 
of http://sources.buildroot.net/ which makes managing firewall rules much 
easier. Any modification to the download infrastructure would need to 
support this feature (preferably by still defining BR2_PRIMARY_SITE).

Thanks,
-Ryan

[Buildroot] Different site methods for the main package and patches

[Thomas Petazzoni]

Dear Peter Korsgaard,

On Wed, 26 Mar 2014 23:42:20 +0100, Peter Korsgaard wrote:

>  >  *) I don't really understand why we have a separation between _SITE
>  >     and _SOURCE, and why our infra assumes that _PATCH is relative to
>  >     _SITE. Why don't we simply make _SOURCE and _PATCH full URLs
>  >     instead? This way _PATCH can list multiple patches coming from
>  >     different locations, not necessarily the upstream location of the
>  >     package.
> 
> Some of it is historical, but it also makes it easier to support
> the primary/backup site features.

Even if:

BLEH_SOURCE = http://foobar.com/bleh-$(BLEH_VERSION).tar.gz

we can do:

	$(notdir $(BLEH_SOURCE))

to retrieve just the filename and look it up on the primary/backup
sites, no?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Different site methods for the main package and patches

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

Hi,

 > Even if:

 > BLEH_SOURCE = http://foobar.com/bleh-$(BLEH_VERSION).tar.gz

 > we can do:

 > 	$(notdir $(BLEH_SOURCE))

 > to retrieve just the filename and look it up on the primary/backup
 > sites, no?

True. As I said, it's mainly historical.

-- 
Bye, Peter Korsgaard