[Buildroot] [RFC 0/2] Add support for specifying a custom kernel directory

[Buildroot] [RFC 0/2] Add support for specifying a custom kernel directory

[Florian Fainelli]

Hi,

This series allows specifying a custom kernel directory which could be an
existing git tree. This is particularly useful when doing kernel development
and having an existing git checkout where development is happening.

Ideally, I would like to avoid doing the rsync operation involved with the "local" method
and instead just have a symbolic link from /path/to/kernel -> output/build/linux.

Comments welcome!

Florian Fainelli (2):
  pkg-generic: Don't check for trailing slashes for local method
  linux: Add support for specifying a custom directory

 linux/Config.in                        | 10 ++++++++++
 linux/linux.mk                         |  3 +++
 package/linux-headers/linux-headers.mk |  5 +++++
 package/pkg-generic.mk                 |  2 ++
 4 files changed, 20 insertions(+)

-- 
2.7.4

[Buildroot] [RFC 1/2] pkg-generic: Don't check for trailing slashes for local method

[Florian Fainelli]

Local methods might be used to specify a path to a directory, which could
contain trailing slashes, don't bother checking those.

Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
---
 package/pkg-generic.mk | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index cca94ba338fb..4c090f2daed3 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -998,9 +998,11 @@ $$(error $(2)_SITE cannot be empty when $(2)_SOURCE is not)
 endif
 endif
 
+ifneq ($$($(2)_SITE_METHOD),local)
 ifeq ($$(patsubst %/,ERROR,$$($(2)_SITE)),ERROR)
 $$(error $(2)_SITE ($$($(2)_SITE)) cannot have a trailing slash)
 endif
+endif
 
 ifneq ($$($(2)_HELP_CMDS),)
 HELP_PACKAGES += $(2)
-- 
2.7.4

[Buildroot] [RFC 2/2] linux: Add support for specifying a custom directory

[Florian Fainelli]

Add the ability to specify a path to a custom directory where kernel sources
may be contained. This is useful when doing kernel development in an existing
git tree.

Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
---
 linux/Config.in                        | 10 ++++++++++
 linux/linux.mk                         |  3 +++
 package/linux-headers/linux-headers.mk |  5 +++++
 3 files changed, 18 insertions(+)

diff --git a/linux/Config.in b/linux/Config.in
index 33ae944557b6..7d4dcddc7ea7 100644
--- a/linux/Config.in
+++ b/linux/Config.in
@@ -87,6 +87,12 @@ config BR2_LINUX_KERNEL_CUSTOM_SVN
 	  This option allows Buildroot to get the Linux kernel source
 	  code from a Subversion repository.
 
+config BR2_LINUX_KERNEL_CUSTOM_DIRECTORY
+	bool "Custom directory"
+	help
+	  This option allows Buildroot to use the Linux kernel source
+	  code from an existing directory.
+
 endchoice
 
 config BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE
@@ -97,6 +103,10 @@ config BR2_LINUX_KERNEL_CUSTOM_TARBALL_LOCATION
 	string "URL of custom kernel tarball"
 	depends on BR2_LINUX_KERNEL_CUSTOM_TARBALL
 
+config BR2_LINUX_KERNEL_CUSTOM_DIRECTORY_LOCATION
+	string "Path of the custom kernel source"
+	depends on BR2_LINUX_KERNEL_CUSTOM_DIRECTORY
+
 if BR2_LINUX_KERNEL_CUSTOM_GIT || BR2_LINUX_KERNEL_CUSTOM_HG || BR2_LINUX_KERNEL_CUSTOM_SVN
 
 config BR2_LINUX_KERNEL_CUSTOM_REPO_URL
diff --git a/linux/linux.mk b/linux/linux.mk
index bd5589bae02d..192fd0d3ffb3 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -29,6 +29,9 @@ LINUX_SITE_METHOD = hg
 else ifeq ($(BR2_LINUX_KERNEL_CUSTOM_SVN),y)
 LINUX_SITE = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_REPO_URL))
 LINUX_SITE_METHOD = svn
+else ifeq ($(BR2_LINUX_KERNEL_CUSTOM_DIRECTORY),y)
+LINUX_SITE = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_DIRECTORY_LOCATION))
+LINUX_SITE_METHOD = local
 else ifeq ($(BR2_LINUX_KERNEL_LATEST_CIP_VERSION),y)
 LINUX_SITE = git://git.kernel.org/pub/scm/linux/kernel/git/bwh/linux-cip.git
 else ifneq ($(findstring -rc,$(LINUX_VERSION)),)
diff --git a/package/linux-headers/linux-headers.mk b/package/linux-headers/linux-headers.mk
index f1e379060834..e42af8a4d672 100644
--- a/package/linux-headers/linux-headers.mk
+++ b/package/linux-headers/linux-headers.mk
@@ -26,6 +26,9 @@ LINUX_HEADERS_SITE = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_REPO_URL))
 LINUX_HEADERS_SITE_METHOD = hg
 # use same hg tarball as linux kernel
 LINUX_HEADERS_SOURCE = linux-$(LINUX_HEADERS_VERSION).tar.gz
+else ifeq ($(BR2_LINUX_KERNEL_CUSTOM_DIRECTORY),y)
+LINUX_HEADERS_SITE = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_DIRECTORY_LOCATION))
+LINUX_HEADERS_SITE_METHOD = local
 else
 LINUX_HEADERS_SOURCE = linux-$(LINUX_HEADERS_VERSION).tar.xz
 # In X.Y.Z, get X and Y. We replace dots and dashes by spaces in order
@@ -122,6 +125,7 @@ define LINUX_HEADERS_INSTALL_STAGING_CMDS
 			headers_install)
 endef
 
+ifneq ($(BR2_LINUX_KERNEL_CUSTOM_DIRECTORY),y)
 ifeq ($(BR2_KERNEL_HEADERS_VERSION)$(BR2_KERNEL_HEADERS_AS_KERNEL),y)
 define LINUX_HEADERS_CHECK_VERSION
 	$(call check_kernel_headers_version,\
@@ -130,5 +134,6 @@ define LINUX_HEADERS_CHECK_VERSION
 endef
 LINUX_HEADERS_POST_INSTALL_STAGING_HOOKS += LINUX_HEADERS_CHECK_VERSION
 endif
+endif
 
 $(eval $(generic-package))
-- 
2.7.4

[Buildroot] [RFC 2/2] linux: Add support for specifying a custom directory

[Yann E. MORIN]

Florian, All,

On 2017-10-28 19:06 -0700, Florian Fainelli spake thusly:
> Add the ability to specify a path to a custom directory where kernel sources
> may be contained. This is useful when doing kernel development in an existing
> git tree.

This case is covered by the override-sourcedir mechanism.

Create a file (by default, local.mk) in your config directory (the one
with the Buildroot .config file), and edit this file with:

    LINUX_OVERRIDE_SRCDIR = /path/to/your/linux

and Buildroot will use that as a rsync source, instead of downloading
the kernel sources.

See also e782cd5b1bc (Revert "Added local directory as source of kernel
code") for more in-depth explanations. ;-)

Regards,
Yann E. MORIN.

> Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
> ---
>  linux/Config.in                        | 10 ++++++++++
>  linux/linux.mk                         |  3 +++
>  package/linux-headers/linux-headers.mk |  5 +++++
>  3 files changed, 18 insertions(+)
> 
> diff --git a/linux/Config.in b/linux/Config.in
> index 33ae944557b6..7d4dcddc7ea7 100644
> --- a/linux/Config.in
> +++ b/linux/Config.in
> @@ -87,6 +87,12 @@ config BR2_LINUX_KERNEL_CUSTOM_SVN
>  	  This option allows Buildroot to get the Linux kernel source
>  	  code from a Subversion repository.
>  
> +config BR2_LINUX_KERNEL_CUSTOM_DIRECTORY
> +	bool "Custom directory"
> +	help
> +	  This option allows Buildroot to use the Linux kernel source
> +	  code from an existing directory.
> +
>  endchoice
>  
>  config BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE
> @@ -97,6 +103,10 @@ config BR2_LINUX_KERNEL_CUSTOM_TARBALL_LOCATION
>  	string "URL of custom kernel tarball"
>  	depends on BR2_LINUX_KERNEL_CUSTOM_TARBALL
>  
> +config BR2_LINUX_KERNEL_CUSTOM_DIRECTORY_LOCATION
> +	string "Path of the custom kernel source"
> +	depends on BR2_LINUX_KERNEL_CUSTOM_DIRECTORY
> +
>  if BR2_LINUX_KERNEL_CUSTOM_GIT || BR2_LINUX_KERNEL_CUSTOM_HG || BR2_LINUX_KERNEL_CUSTOM_SVN
>  
>  config BR2_LINUX_KERNEL_CUSTOM_REPO_URL
> diff --git a/linux/linux.mk b/linux/linux.mk
> index bd5589bae02d..192fd0d3ffb3 100644
> --- a/linux/linux.mk
> +++ b/linux/linux.mk
> @@ -29,6 +29,9 @@ LINUX_SITE_METHOD = hg
>  else ifeq ($(BR2_LINUX_KERNEL_CUSTOM_SVN),y)
>  LINUX_SITE = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_REPO_URL))
>  LINUX_SITE_METHOD = svn
> +else ifeq ($(BR2_LINUX_KERNEL_CUSTOM_DIRECTORY),y)
> +LINUX_SITE = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_DIRECTORY_LOCATION))
> +LINUX_SITE_METHOD = local
>  else ifeq ($(BR2_LINUX_KERNEL_LATEST_CIP_VERSION),y)
>  LINUX_SITE = git://git.kernel.org/pub/scm/linux/kernel/git/bwh/linux-cip.git
>  else ifneq ($(findstring -rc,$(LINUX_VERSION)),)
> diff --git a/package/linux-headers/linux-headers.mk b/package/linux-headers/linux-headers.mk
> index f1e379060834..e42af8a4d672 100644
> --- a/package/linux-headers/linux-headers.mk
> +++ b/package/linux-headers/linux-headers.mk
> @@ -26,6 +26,9 @@ LINUX_HEADERS_SITE = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_REPO_URL))
>  LINUX_HEADERS_SITE_METHOD = hg
>  # use same hg tarball as linux kernel
>  LINUX_HEADERS_SOURCE = linux-$(LINUX_HEADERS_VERSION).tar.gz
> +else ifeq ($(BR2_LINUX_KERNEL_CUSTOM_DIRECTORY),y)
> +LINUX_HEADERS_SITE = $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_DIRECTORY_LOCATION))
> +LINUX_HEADERS_SITE_METHOD = local
>  else
>  LINUX_HEADERS_SOURCE = linux-$(LINUX_HEADERS_VERSION).tar.xz
>  # In X.Y.Z, get X and Y. We replace dots and dashes by spaces in order
> @@ -122,6 +125,7 @@ define LINUX_HEADERS_INSTALL_STAGING_CMDS
>  			headers_install)
>  endef
>  
> +ifneq ($(BR2_LINUX_KERNEL_CUSTOM_DIRECTORY),y)
>  ifeq ($(BR2_KERNEL_HEADERS_VERSION)$(BR2_KERNEL_HEADERS_AS_KERNEL),y)
>  define LINUX_HEADERS_CHECK_VERSION
>  	$(call check_kernel_headers_version,\
> @@ -130,5 +134,6 @@ define LINUX_HEADERS_CHECK_VERSION
>  endef
>  LINUX_HEADERS_POST_INSTALL_STAGING_HOOKS += LINUX_HEADERS_CHECK_VERSION
>  endif
> +endif
>  
>  $(eval $(generic-package))
> -- 
> 2.7.4
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [RFC 0/2] Add support for specifying a custom kernel directory

[Yann E. MORIN]

Florian, All,

On 2017-10-28 19:06 -0700, Florian Fainelli spake thusly:
> This series allows specifying a custom kernel directory which could be an
> existing git tree. This is particularly useful when doing kernel development
> and having an existing git checkout where development is happening.

As I replied to the patch itself: this is already covered by the
opverride-srcdir mechanism, so I've marked this series as rejected in
our patchwork.

> Ideally, I would like to avoid doing the rsync operation involved with the "local" method
> and instead just have a symbolic link from /path/to/kernel -> output/build/linux.

The first rsync may take a few seconds, true, but subs=equent ones will
be much faster if all you do is change a few files, especially on a hot
directory, so there is not much gain to have with a local symlink.

OTOH, I am currently working on out-of-tree per-pacakge build, which
will (eventually) allow us to make use of the O= mechanism to build the
kernel out-of-tree, and get rid of the rsync altogether. Don't hold your
breath, though. ;-)

Regards,
Yann E. MORIN.

> Comments welcome!
> 
> Florian Fainelli (2):
>   pkg-generic: Don't check for trailing slashes for local method
>   linux: Add support for specifying a custom directory
> 
>  linux/Config.in                        | 10 ++++++++++
>  linux/linux.mk                         |  3 +++
>  package/linux-headers/linux-headers.mk |  5 +++++
>  package/pkg-generic.mk                 |  2 ++
>  4 files changed, 20 insertions(+)
> 
> -- 
> 2.7.4
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [RFC 2/2] linux: Add support for specifying a custom directory

[Thomas Petazzoni]

Hello,

On Sun, 29 Oct 2017 08:26:49 +0100, Yann E. MORIN wrote:

> On 2017-10-28 19:06 -0700, Florian Fainelli spake thusly:
> > Add the ability to specify a path to a custom directory where kernel sources
> > may be contained. This is useful when doing kernel development in an existing
> > git tree.  
> 
> This case is covered by the override-sourcedir mechanism.
> 
> Create a file (by default, local.mk) in your config directory (the one
> with the Buildroot .config file), and edit this file with:
> 
>     LINUX_OVERRIDE_SRCDIR = /path/to/your/linux
> 
> and Buildroot will use that as a rsync source, instead of downloading
> the kernel sources.
> 
> See also e782cd5b1bc (Revert "Added local directory as source of kernel
> code") for more in-depth explanations. ;-)

Fully agreed with Yann here: there is no point in adding a
Linux-specific solution for this use case, as we already have a much
more general solution that works for all packages, including the
'linux' package.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [RFC 0/2] Add support for specifying a custom kernel directory

[Florian Fainelli]

Hi,

On 10/29/2017 12:33 AM, Yann E. MORIN wrote:
> Florian, All,
> 
> On 2017-10-28 19:06 -0700, Florian Fainelli spake thusly:
>> This series allows specifying a custom kernel directory which could be an
>> existing git tree. This is particularly useful when doing kernel development
>> and having an existing git checkout where development is happening.
> 
> As I replied to the patch itself: this is already covered by the
> opverride-srcdir mechanism, so I've marked this series as rejected in
> our patchwork.
> 
>> Ideally, I would like to avoid doing the rsync operation involved with the "local" method
>> and instead just have a symbolic link from /path/to/kernel -> output/build/linux.
> 
> The first rsync may take a few seconds, true, but subs=equent ones will
> be much faster if all you do is change a few files, especially on a hot
> directory, so there is not much gain to have with a local symlink.

It is more about not duplicating sources unnecessarily. If you have all
developers on your system do the same thing, you would end-up with a lot
of copies of the same files, and if your filesystem does not already
offer de-duplication, that becomes additional space used. The copy
aspect is also potentially an issue since Linux kernel sources are
significant enough this can become a concern over time.

> 
> OTOH, I am currently working on out-of-tree per-pacakge build, which
> will (eventually) allow us to make use of the O= mechanism to build the
> kernel out-of-tree, and get rid of the rsync altogether. Don't hold your
> breath, though. ;-)

OK, that's good to hear. Feel free to CC when you get there and I will
happily try to test that too.

Thanks!
-- 
Florian

[Buildroot] [RFC 2/2] linux: Add support for specifying a custom directory

[Florian Fainelli]

Hi,

On 10/29/2017 02:12 AM, Thomas Petazzoni wrote:
> Hello,
> 
> On Sun, 29 Oct 2017 08:26:49 +0100, Yann E. MORIN wrote:
> 
>> On 2017-10-28 19:06 -0700, Florian Fainelli spake thusly:
>>> Add the ability to specify a path to a custom directory where kernel sources
>>> may be contained. This is useful when doing kernel development in an existing
>>> git tree.  
>>
>> This case is covered by the override-sourcedir mechanism.
>>
>> Create a file (by default, local.mk) in your config directory (the one
>> with the Buildroot .config file), and edit this file with:
>>
>>     LINUX_OVERRIDE_SRCDIR = /path/to/your/linux
>>
>> and Buildroot will use that as a rsync source, instead of downloading
>> the kernel sources.
>>
>> See also e782cd5b1bc (Revert "Added local directory as source of kernel
>> code") for more in-depth explanations. ;-)
> 
> Fully agreed with Yann here: there is no point in adding a
> Linux-specific solution for this use case, as we already have a much
> more general solution that works for all packages, including the
> 'linux' package.

That seems fine in premise, but there are still a few angles that just
don't feel right or I am most certainly just misusing this:

- we still need to specify a "custom" kernel version, and apparently it
cannot be empty and just mean "does not matter", that's extremely
impractical, if the workflow involves a bisection over several kernel
versions, there is no way I am specifying the kernel version every time
I run git bisect

- how about kernel-headers? My local.mk now has this:

LINUX_OVERRIDE_SRCDIR = /home/florian/dev/linux
LINUX_HEADERS_OVERRIDE_SRCDIR = /home/florian/dev/linux

but it still won't let me build with "Same as kernel being built", see
below [1]

- if someone is seeing any issues with Buildroot using this (or anything
for that matter), it would a lot easier for you to just tell them to
send their .config file, now you also need to ask them whether they are
using local.mk, and send it too, that assumes people read "how to submit
bugs" documentation anyway

[1]:
package/linux-headers/linux-headers.mk:134: *** LINUX_HEADERS_SITE
cannot be empty when LINUX_HEADERS_SOURCE is not.  Stop.
Makefile:79: recipe for target '_all' failed
make: *** [_all] Error 2
zsh: exit 2     make

-- 
Florian

[Buildroot] [RFC 2/2] linux: Add support for specifying a custom directory

[Yann E. MORIN]

Florian, All,

On 2017-10-29 10:26 -0700, Florian Fainelli spake thusly:
> On 10/29/2017 02:12 AM, Thomas Petazzoni wrote:
> > Hello,
> > 
> > On Sun, 29 Oct 2017 08:26:49 +0100, Yann E. MORIN wrote:
> > 
> >> On 2017-10-28 19:06 -0700, Florian Fainelli spake thusly:
> >>> Add the ability to specify a path to a custom directory where kernel sources
> >>> may be contained. This is useful when doing kernel development in an existing
> >>> git tree.  
> >>
> >> This case is covered by the override-sourcedir mechanism.
> >>
> >> Create a file (by default, local.mk) in your config directory (the one
> >> with the Buildroot .config file), and edit this file with:
> >>
> >>     LINUX_OVERRIDE_SRCDIR = /path/to/your/linux
> >>
> >> and Buildroot will use that as a rsync source, instead of downloading
> >> the kernel sources.
> >>
> >> See also e782cd5b1bc (Revert "Added local directory as source of kernel
> >> code") for more in-depth explanations. ;-)
> > 
> > Fully agreed with Yann here: there is no point in adding a
> > Linux-specific solution for this use case, as we already have a much
> > more general solution that works for all packages, including the
> > 'linux' package.
> 
> That seems fine in premise, but there are still a few angles that just
> don't feel right or I am most certainly just misusing this:
> 
> - we still need to specify a "custom" kernel version, and apparently it
> cannot be empty and just mean "does not matter", that's extremely
> impractical, if the workflow involves a bisection over several kernel
> versions, there is no way I am specifying the kernel version every time
> I run git bisect

Just leave trhe version at its default value; it is only ever used to
know what to download in the first place, and since you use an override,
it will download nothing.

> - how about kernel-headers? My local.mk now has this:
> 
> LINUX_OVERRIDE_SRCDIR = /home/florian/dev/linux
> LINUX_HEADERS_OVERRIDE_SRCDIR = /home/florian/dev/linux
> 
> but it still won't let me build with "Same as kernel being built", see
> below [1]

In that case, you have a bigger problem: you will have to rebuild the
toolchain every time, as the headers are changing. And thus, you will
have to rebuild everything else (all packages).

And if/when your bisect crosses kernel version boundaries, you'll have
to adjust the "Custom kernel headers series" as well too.

So I'd say that you would probably be more impacted by the build time
first, no?

So, for bisecting a kernel, I'd try to not have to rebuild the toolchain
every tine, so stick with a static headers version.

Unless you are explicitly trying to bisect a headers problem...

> - if someone is seeing any issues with Buildroot using this (or anything
> for that matter), it would a lot easier for you to just tell them to
> send their .config file, now you also need to ask them whether they are
> using local.mk, and send it too, that assumes people read "how to submit
> bugs" documentation anyway

Meh, you know what we say about documentation? No one reads it! ;-)

But yeah, we've had to deal with that a few times, and it soon becomes
apparent that the user is using override-srcdir, because the paths then
read something like 'linux-custom', which is easy to spot.

> [1]:
> package/linux-headers/linux-headers.mk:134: *** LINUX_HEADERS_SITE
> cannot be empty when LINUX_HEADERS_SOURCE is not.  Stop.
> Makefile:79: recipe for target '_all' failed
> make: *** [_all] Error 2
> zsh: exit 2     make

Hmm... Even when you keep the kernel version to its default value?

and even then, I would not be too much concerned about this, because if
you change the headers, you will need to rebuild the toolchain (see
above for the rest of the story).

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [RFC 2/2] linux: Add support for specifying a custom directory

[Florian Fainelli]

Le 10/29/17 ? 11:12, Yann E. MORIN a ?crit?:
> Florian, All,
> 
> On 2017-10-29 10:26 -0700, Florian Fainelli spake thusly:
>> On 10/29/2017 02:12 AM, Thomas Petazzoni wrote:
>>> Hello,
>>>
>>> On Sun, 29 Oct 2017 08:26:49 +0100, Yann E. MORIN wrote:
>>>
>>>> On 2017-10-28 19:06 -0700, Florian Fainelli spake thusly:
>>>>> Add the ability to specify a path to a custom directory where kernel sources
>>>>> may be contained. This is useful when doing kernel development in an existing
>>>>> git tree.  
>>>>
>>>> This case is covered by the override-sourcedir mechanism.
>>>>
>>>> Create a file (by default, local.mk) in your config directory (the one
>>>> with the Buildroot .config file), and edit this file with:
>>>>
>>>>     LINUX_OVERRIDE_SRCDIR = /path/to/your/linux
>>>>
>>>> and Buildroot will use that as a rsync source, instead of downloading
>>>> the kernel sources.
>>>>
>>>> See also e782cd5b1bc (Revert "Added local directory as source of kernel
>>>> code") for more in-depth explanations. ;-)
>>>
>>> Fully agreed with Yann here: there is no point in adding a
>>> Linux-specific solution for this use case, as we already have a much
>>> more general solution that works for all packages, including the
>>> 'linux' package.
>>
>> That seems fine in premise, but there are still a few angles that just
>> don't feel right or I am most certainly just misusing this:
>>
>> - we still need to specify a "custom" kernel version, and apparently it
>> cannot be empty and just mean "does not matter", that's extremely
>> impractical, if the workflow involves a bisection over several kernel
>> versions, there is no way I am specifying the kernel version every time
>> I run git bisect
> 
> Just leave trhe version at its default value; it is only ever used to
> know what to download in the first place, and since you use an override,
> it will download nothing.
> 
>> - how about kernel-headers? My local.mk now has this:
>>
>> LINUX_OVERRIDE_SRCDIR = /home/florian/dev/linux
>> LINUX_HEADERS_OVERRIDE_SRCDIR = /home/florian/dev/linux
>>
>> but it still won't let me build with "Same as kernel being built", see
>> below [1]
> 
> In that case, you have a bigger problem: you will have to rebuild the
> toolchain every time, as the headers are changing. And thus, you will
> have to rebuild everything else (all packages).
> 
> And if/when your bisect crosses kernel version boundaries, you'll have
> to adjust the "Custom kernel headers series" as well too.
> 
> So I'd say that you would probably be more impacted by the build time
> first, no?
> 
> So, for bisecting a kernel, I'd try to not have to rebuild the toolchain
> every tine, so stick with a static headers version.
> 
> Unless you are explicitly trying to bisect a headers problem...

Not in that particular case, but fair enough, setting a "fixed" kernel
headers version that is known to be lower than or equal to the kernel
being built works for me. This might be something to keep in mind when
bisecting, but usually it is not a problem.

> 
>> - if someone is seeing any issues with Buildroot using this (or anything
>> for that matter), it would a lot easier for you to just tell them to
>> send their .config file, now you also need to ask them whether they are
>> using local.mk, and send it too, that assumes people read "how to submit
>> bugs" documentation anyway
> 
> Meh, you know what we say about documentation? No one reads it! ;-)

Why read, such a waste of time, better ask the relevant developers
instead ;)

> 
> But yeah, we've had to deal with that a few times, and it soon becomes
> apparent that the user is using override-srcdir, because the paths then
> read something like 'linux-custom', which is easy to spot.
> 
>> [1]:
>> package/linux-headers/linux-headers.mk:134: *** LINUX_HEADERS_SITE
>> cannot be empty when LINUX_HEADERS_SOURCE is not.  Stop.
>> Makefile:79: recipe for target '_all' failed
>> make: *** [_all] Error 2
>> zsh: exit 2     make
> 
> Hmm... Even when you keep the kernel version to its default value?

That was also because I had set the "custom kernel version" to an empty
string, which could have indicated "don't care" but was still needed.
This is also probably fine to set to anything once you set your kernel
headers to a particular version.

It seems like we should not prompted any kernel version in case since we
use a local kernel directory, or, the version should be derived directly
from the kernel  directory so it does not have to be manually specified
at all but that's a tad more complicated.

Thanks for your help, it's been pretty interesting to use another build
system thus far :)
-- 
Florian

[Buildroot] [RFC 2/2] linux: Add support for specifying a custom directory

[Yann E. MORIN]

Florian, All,

On 2017-10-29 20:27 -0700, Florian Fainelli spake thusly:
> Le 10/29/17 ? 11:12, Yann E. MORIN a ?crit?:
> > On 2017-10-29 10:26 -0700, Florian Fainelli spake thusly:
> >> package/linux-headers/linux-headers.mk:134: *** LINUX_HEADERS_SITE
> >> cannot be empty when LINUX_HEADERS_SOURCE is not.  Stop.
> >> Makefile:79: recipe for target '_all' failed
> >> make: *** [_all] Error 2
> >> zsh: exit 2     make
> > 
> > Hmm... Even when you keep the kernel version to its default value?
> 
> That was also because I had set the "custom kernel version" to an empty
> string, which could have indicated "don't care" but was still needed.
> This is also probably fine to set to anything once you set your kernel
> headers to a particular version.
> 
> It seems like we should not prompted any kernel version in case since we
> use a local kernel directory, or, the version should be derived directly
> from the kernel  directory so it does not have to be manually specified
> at all but that's a tad more complicated.

But in truth, the override-srcdir mechanism is just that: an _override_
of the configured values, to be used only for development.

So, the premise is that you start off from a "known" configuration, and
override just that one or two packages you are working on.

> Thanks for your help, it's been pretty interesting to use another build
> system thus far :)

I was rather surprised (in a good sense!) to see you here, yes. Trying
the Dark Side of the Force, are we? ;-]

See ya!

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [RFC 2/2] linux: Add support for specifying a custom directory

[Florian Fainelli]

On 10/30/2017 03:32 PM, Yann E. MORIN wrote:
> Florian, All,
> 
> On 2017-10-29 20:27 -0700, Florian Fainelli spake thusly:
>> Le 10/29/17 ? 11:12, Yann E. MORIN a ?crit?:
>>> On 2017-10-29 10:26 -0700, Florian Fainelli spake thusly:
>>>> package/linux-headers/linux-headers.mk:134: *** LINUX_HEADERS_SITE
>>>> cannot be empty when LINUX_HEADERS_SOURCE is not.  Stop.
>>>> Makefile:79: recipe for target '_all' failed
>>>> make: *** [_all] Error 2
>>>> zsh: exit 2     make
>>>
>>> Hmm... Even when you keep the kernel version to its default value?
>>
>> That was also because I had set the "custom kernel version" to an empty
>> string, which could have indicated "don't care" but was still needed.
>> This is also probably fine to set to anything once you set your kernel
>> headers to a particular version.
>>
>> It seems like we should not prompted any kernel version in case since we
>> use a local kernel directory, or, the version should be derived directly
>> from the kernel  directory so it does not have to be manually specified
>> at all but that's a tad more complicated.
> 
> But in truth, the override-srcdir mechanism is just that: an _override_
> of the configured values, to be used only for development.

OK. So by the same token, I should not expect that rsync operation
syncing my development tree to output/build/linux-custom to work past
the initial build/configuration? Right now it does not, I can make as
many changes as I want in the directory reference from local.mk and
output/build/linux-custom remains the same as it was the first time it
got rsync'd, is that really expected?

I really wonder how you guys are doing serious kernel development with
buildroot?

> 
> So, the premise is that you start off from a "known" configuration, and
> override just that one or two packages you are working on.
> 
>> Thanks for your help, it's been pretty interesting to use another build
>> system thus far :)
> 
> I was rather surprised (in a good sense!) to see you here, yes. Trying
> the Dark Side of the Force, are we? ;-]

Yes, and so far, it's quite dark, looks about as friendly as Dagoba :)
-- 
Florian

[Buildroot] [RFC 2/2] linux: Add support for specifying a custom directory

[Yann E. MORIN]

On 3 November 2017 06:03:32 CET, Florian Fainelli <f.fainelli@gmail.com> wrote:
>OK. So by the same token, I should not expect that rsync operation
>syncing my development tree to output/build/linux-custom to work past
>the initial build/configuration? Right now it does not, I can make as
>many changes as I want in the directory reference from local.mk and
>output/build/linux-custom remains the same as it was the first time it
>got rsync'd, is that really expected?

Sorta, yes.

>I really wonder how you guys are doing serious kernel development with
>buildroot?

'make linux-rebuild' is what you're looking for.

If you need 'oldconfig' to be run, 'make linux-reconfigure' you will want instead.

Regards,
Yann E. MORIN.
Florian, All,
-- 
Sent from my phone. Please excuse my brevity.

[Buildroot] [RFC 2/2] linux: Add support for specifying a custom directory

[Thomas Petazzoni]

Hello,

On Thu, 2 Nov 2017 22:03:32 -0700, Florian Fainelli wrote:

> OK. So by the same token, I should not expect that rsync operation
> syncing my development tree to output/build/linux-custom to work past
> the initial build/configuration? Right now it does not, I can make as
> many changes as I want in the directory reference from local.mk and
> output/build/linux-custom remains the same as it was the first time it
> got rsync'd, is that really expected?
> 
> I really wonder how you guys are doing serious kernel development with
> buildroot?

Come on Florian, such a silly and uninformed statement coming from you?
I could expect that from a less seasoned open-source contributor, but
from you, I'm really surprised.

This is explained very clearly in the section "8.12.6. Using Buildroot
during development" of the manual,
https://buildroot.org/downloads/manual/manual.html, which states:

"""

  When Buildroot finds that for a given package, an
  <pkg>_OVERRIDE_SRCDIR has been defined, it will no longer attempt to
  download, extract and patch the package. Instead, it will directly
  use the source code available in in the specified directory and make
  clean will not touch this directory. This allows to point Buildroot
  to your own directories, that can be managed by Git, Subversion, or
  any other version control system. To achieve this, Buildroot will use
  rsync to copy the source code of the component from the specified
  <pkg>_OVERRIDE_SRCDIR to output/build/<package>-custom/.

  This mechanism is best used in conjunction with the make <pkg>-rebuild
  and make <pkg>-reconfigure targets. A make <pkg>-rebuild all sequence
  will rsync the source code from <pkg>_OVERRIDE_SRCDIR to
  output/build/<package>-custom (thanks to rsync, only the modified
  files are copied), and restart the build process of just this package.

"""

This is also explained in details in the Buildroot training materials,
http://free-electrons.com/doc/training/buildroot/buildroot-slides.pdf
slide 269 and following.

So, before making such statements, could you perhaps read the
documentation first?

Also, it's worth mentioning that when I do active kernel development, I
personally don't build my kernel within Buildroot, but separately. I
use Buildroot to build the root filesystem (often a simple initramfs),
but I build the kernel separately. The kernel build process is simple
enough that building it separately is often easier.

<pkg>_OVERRIDE_SRCDIR can however be useful if you're building kernel
modules, as it ensures they are installed in the root filesystem.

However, I believe <pkg>_OVERRIDE_SRCDIR is particularly useful when
you're actively developing/debugging a user-space application/library
that has a complicated build process. In this case, having Buildroot
trigger the build is easier.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com