[Buildroot] [RFC] Override source directories

[Buildroot] [RFC] Override source directories

[Thomas Petazzoni]

Hello,

Here is a set of RFC patches that implement the basic mechanism I have
been referring to since a couple of weeks/months about allowing users
to override the source directory for packages.

From an user perspective, it allows ones to create a "local.mk" file
in the Buildroot main source directory, with variable definitions such
as :

 ZLIB_OVERRIDE_SRCDIR = /tmp/zlib
 MYAPP_OVERRIDE_SRCDIR = /home/foobar/project/myapp
 LIBGLIB2_OVERRIDE_SRCDIR = /home/foobar/git/libglib2

At build time, Buildroot would use those directories at the source for
the packages, instead of downloading, extracing and patching them in
the usual way.

Internally, this is implemented inside our package infrastructure, so
this mechanism should work for all packages that rely on the
infrastructure. The GENTARGETS macro makes sure that the package build
process bypasses the download, extract and patch step when
<pkg>_OVERRIDE_SRCDIR is defined, and instead a symlinking step is
added. It's very simple.

This series of patches also add the <pkg>-reconfigure and
<pkg>-rebuild targets, which makes it easy for users to restart the
package build process at the compilation step or at the configuration
step, without having to mess up with stamp file details.

There are still a few issues to solve and discuss, though :

 * Is local.mk inside the main source directory the good solution?
   Should we instead provide a configuration option to set the path to
   the override file? Or another solution?

 * As the mechanism is implemented today, "make clean" removes the
   symbolic link $(O)/build/<pkg>-<version>, but not the stamp files
   created inside this source directory. So next time the build is
   started, things will go wrong. To solve this, the only solution I
   see is to separate the directory with the stamp files from the
   source directory, at least in the case of overriden source
   directories.

 * Other things?

Feedback, comments and ideas welcome.

Regards,

Thomas

The following changes since commit 7234c23967fa70b760c64700e56bb1778019fbeb:

  CHANGES: update with recent changes (2011-05-18 21:02:30 +0200)

are available in the git repository at:
  http://free-electrons.com/~thomas/buildroot.git for-2011.08/sourcedir

Thomas Petazzoni (4):
      package: enhance infrastructure to support source dir override
      package: remove stamp files in <pkg>-dirclean
      package: add <pkg>-rebuild and <pkg>-reconfigure
      Include a local.mk for package overrides

 Makefile                    |    1 +
 package/Makefile.package.in |   30 ++++++++++++++++++++++++++++++
 2 files changed, 31 insertions(+), 0 deletions(-)

Thanks,
-- 
Thomas Petazzoni

[Buildroot] [PATCH 1/4] package: enhance infrastructure to support source dir override

[Thomas Petazzoni]

When a variable <pkg>_OVERRIDE_SRCDIR is defined, then Buildroot will
no longer try to download, extract and patch the package. It will
simply use the value of this variable as the source directory for the
package. A symbolic link to this directory will be created, so that
for Buildroot, the package sources are still in
$(O)/build/pkg-version/.

This can be used to tell Buildroot that the sources for a given
package are inside some directory that you control, and which can be
versioned in Git/SVN, or handled in whichever way you want.

Note that Buildroot will still create its .stamp_* files and do the
build inside this source directory. Will out-of-tree builds would have
been possible with AUTOTARGETS, it wouldn't work well in a generic way
for GENTARGETS packages. That's the reason why we've choosen to keep
an in-tree build solution.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 package/Makefile.package.in |   16 ++++++++++++++++
 1 files changed, 16 insertions(+), 0 deletions(-)

diff --git a/package/Makefile.package.in b/package/Makefile.package.in
index bf4e1b4..1afc18d 100644
--- a/package/Makefile.package.in
+++ b/package/Makefile.package.in
@@ -250,6 +250,13 @@ $(BUILD_DIR)/%/.stamp_extracted:
 	$(foreach hook,$($(PKG)_POST_EXTRACT_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 
+# Set up symbolic link to the source directory if the
+# <pkg>_OVERRIDE_SRCDIR feature is used.
+$(BUILD_DIR)/%/.stamp_symlinked:
+	@$(call MESSAGE,"Symlinking to source dir")
+	ln -s $(SRCDIR) $(@D)
+	$(Q)touch $@
+
 # Patch
 #
 # The NOHOSTPKG variable is the uppercased package name, without the
@@ -418,6 +425,7 @@ $(2)_TARGET_INSTALL_STAGING =	$$($(2)_DIR)/.stamp_staging_installed
 $(2)_TARGET_INSTALL_HOST =      $$($(2)_DIR)/.stamp_host_installed
 $(2)_TARGET_BUILD =		$$($(2)_DIR)/.stamp_built
 $(2)_TARGET_CONFIGURE =		$$($(2)_DIR)/.stamp_configured
+$(2)_TARGET_SYMLINK =		$$($(2)_DIR)/.stamp_symlinked
 $(2)_TARGET_PATCH =		$$($(2)_DIR)/.stamp_patched
 $(2)_TARGET_EXTRACT =		$$($(2)_DIR)/.stamp_extracted
 $(2)_TARGET_SOURCE =		$$($(2)_DIR)/.stamp_downloaded
@@ -463,8 +471,15 @@ $(1)-install-host:      $(1)-build $$($(2)_TARGET_INSTALL_HOST)
 $(1)-build:		$(1)-configure \
 			$$($(2)_TARGET_BUILD)
 
+ifeq ($($(2)_OVERRIDE_SRCDIR),)
 $(1)-configure:		$(1)-patch \
 			$$($(2)_TARGET_CONFIGURE)
+else
+$(1)-configure:		$(1)-symlink \
+			$$($(2)_TARGET_CONFIGURE)
+endif
+
+$(1)-symlink:		$$($(2)_TARGET_SYMLINK)
 
 $(1)-patch:		$(1)-extract $$($(2)_TARGET_PATCH)
 
@@ -492,6 +507,7 @@ $$($(2)_TARGET_INSTALL_STAGING):	PKG=$(2)
 $$($(2)_TARGET_INSTALL_HOST):           PKG=$(2)
 $$($(2)_TARGET_BUILD):			PKG=$(2)
 $$($(2)_TARGET_CONFIGURE):		PKG=$(2)
+$$($(2)_TARGET_SYMLINK):		SRCDIR=$$($(2)_OVERRIDE_SRCDIR)
 $$($(2)_TARGET_PATCH):			PKG=$(2)
 $$($(2)_TARGET_PATCH):			NOHOSTPKG=$(3)
 $$($(2)_TARGET_EXTRACT):		PKG=$(2)
-- 
1.7.1

[Buildroot] [PATCH 2/4] package: remove stamp files in <pkg>-dirclean

[Thomas Petazzoni]

When the package build directory was a real directory, then removing
the directory was enough to remove all its stamp files. However, now
that the build directory can in fact be a symbolic link to the real
source directory, we need to explicitly remove the stamp files before
removing the symbolic link. Otherwise, next time the symbolic is
re-created, some old stamp files would be found.

Unfortunately, there is still a bug: if one does a global "make
clean", the symbolic links are removed since $(BUILD_DIR) is removed,
but not the stamp files. Should we create the stamp files outside of
the package build directory ?

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 package/Makefile.package.in |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/package/Makefile.package.in b/package/Makefile.package.in
index 1afc18d..702395a 100644
--- a/package/Makefile.package.in
+++ b/package/Makefile.package.in
@@ -335,6 +335,7 @@ $(BUILD_DIR)/%/.stamp_uninstalled:
 
 # Remove package sources
 $(BUILD_DIR)/%/.stamp_dircleaned:
+	rm -f $(@D)/.stamp_*
 	rm -Rf $(@D)
 
 ################################################################################
-- 
1.7.1

[Buildroot] [PATCH 3/4] package: add <pkg>-rebuild and <pkg>-reconfigure

[Thomas Petazzoni]

We are often asked "how can I restart the build of a package ?" or
"how can I restart the build of package from the configure part
?". Obviously, tweaking with stamp files is possible, but not very
user friendly.

Therefore this patch adds two new per-package targets: <pkg>-rebuild
and <pkg>-reconfigure. They will respectively restart the
build/install or the configure/build/install of the package <pkg>.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 package/Makefile.package.in |   13 +++++++++++++
 1 files changed, 13 insertions(+), 0 deletions(-)

diff --git a/package/Makefile.package.in b/package/Makefile.package.in
index 702395a..8abfd8b 100644
--- a/package/Makefile.package.in
+++ b/package/Makefile.package.in
@@ -501,6 +501,19 @@ $(1)-clean:		$(1)-uninstall \
 
 $(1)-dirclean:		$$($(2)_TARGET_DIRCLEAN)
 
+$(1)-clean-for-rebuild:
+			rm -f $$($(2)_TARGET_BUILD)
+			rm -f $$($(2)_TARGET_INSTALL_STAGING)
+			rm -f $$($(2)_TARGET_INSTALL_TARGET)
+			rm -f $$($(2)_TARGET_INSTALL_HOST)
+
+$(1)-rebuild:		$(1)-clean-for-rebuild $(1)
+
+$(1)-clean-for-reconfigure: $(1)-clean-for-rebuild
+			rm -f $$($(2)_TARGET_CONFIGURE)
+
+$(1)-reconfigure:	$(1)-clean-for-reconfigure $(1)
+
 # define the PKG variable for all targets, containing the
 # uppercase package variable prefix
 $$($(2)_TARGET_INSTALL_TARGET):		PKG=$(2)
-- 
1.7.1

[Buildroot] [PATCH 4/4] Include a local.mk for package overrides

[Thomas Petazzoni]

The user can now create a local.mk file at the top of the Buildroot
source tree to override the source directory for various packages.

We could probably turn this into a configuration option, so that a
different override file can be used for different projects using the
same Buildroot source tree.

An example local.mk file:

 ZLIB_OVERRIDE_SRCDIR = /tmp/zlib

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 Makefile |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/Makefile b/Makefile
index 5f61b93..07a51d8 100644
--- a/Makefile
+++ b/Makefile
@@ -318,6 +318,7 @@ else ifeq ($(BR2_TOOLCHAIN_CTNG),y)
 include toolchain/toolchain-crosstool-ng.mk
 endif
 
+-include local.mk
 include package/*/*.mk
 
 include boot/common.mk
-- 
1.7.1

[Buildroot] [RFC] Override source directories

[Benoît Mauduit]

On Wed, May 18, 2011 at 11:48 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Hello,

Hi Thomas,

> >From an user perspective, it allows ones to create a "local.mk" file
> in the Buildroot main source directory, with variable definitions such
> as :
>
> ?ZLIB_OVERRIDE_SRCDIR = /tmp/zlib
> ?MYAPP_OVERRIDE_SRCDIR = /home/foobar/project/myapp
> ?LIBGLIB2_OVERRIDE_SRCDIR = /home/foobar/git/libglib2
>
> At build time, Buildroot would use those directories at the source for
> the packages, instead of downloading, extracing and patching them in
> the usual way.
>
> Internally, this is implemented inside our package infrastructure, so
> this mechanism should work for all packages that rely on the
> infrastructure. The GENTARGETS macro makes sure that the package build
> process bypasses the download, extract and patch step when
> <pkg>_OVERRIDE_SRCDIR is defined, and instead a symlinking step is
> added. It's very simple.

I do a similar thing but with a python wrapper, so i think it would be
great to have this feature directly inside Buildroot.

> This series of patches also add the <pkg>-reconfigure and
> <pkg>-rebuild targets, which makes it easy for users to restart the
> package build process at the compilation step or at the configuration
> step, without having to mess up with stamp file details.

Very useful for a user point of view :)

> There are still a few issues to solve and discuss, though :
>
> ?* Is local.mk inside the main source directory the good solution?
> ? Should we instead provide a configuration option to set the path to
> ? the override file? Or another solution?

The advantage to have this file on top of Buildroot directory is that
we can easily find it and made local changes.
But a configuration option could be more comprehensive for end user,
and add flexibility.

> ?* As the mechanism is implemented today, "make clean" removes the
> ? symbolic link $(O)/build/<pkg>-<version>, but not the stamp files
> ? created inside this source directory. So next time the build is
> ? started, things will go wrong. To solve this, the only solution I
> ? see is to separate the directory with the stamp files from the
> ? source directory, at least in the case of overriden source
> ? directories.

Maybe i am wrong, but it can be easy to parse the previous local.mk
file and call the updated <pkg>-dirclean rules just for local package
before make clean call 'rm' ?

I will be happy to see this feature inside Buildroot :)

Regards,

-- 
Benoit Mauduit

[Buildroot] [RFC] Override source directories

[Alper Yıldırım]

Hi Thomas,

On Thu, May 19, 2011 at 12:48 AM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Hello,
>
> Here is a set of RFC patches that implement the basic mechanism I have
> been referring to since a couple of weeks/months about allowing users
> to override the source directory for packages.
>
> >From an user perspective, it allows ones to create a "local.mk" file
> in the Buildroot main source directory, with variable definitions such
> as :
>
> ?ZLIB_OVERRIDE_SRCDIR = /tmp/zlib
> ?MYAPP_OVERRIDE_SRCDIR = /home/foobar/project/myapp
> ?LIBGLIB2_OVERRIDE_SRCDIR = /home/foobar/git/libglib2
>
> At build time, Buildroot would use those directories at the source for
> the packages, instead of downloading, extracing and patching them in
> the usual way.
>
> Internally, this is implemented inside our package infrastructure, so
> this mechanism should work for all packages that rely on the
> infrastructure. The GENTARGETS macro makes sure that the package build
> process bypasses the download, extract and patch step when
> <pkg>_OVERRIDE_SRCDIR is defined, and instead a symlinking step is
> added. It's very simple.
>

It is definitely what i want from buildroot. This feature is a must when working
with local application repositories.

> This series of patches also add the <pkg>-reconfigure and
> <pkg>-rebuild targets, which makes it easy for users to restart the
> package build process at the compilation step or at the configuration
> step, without having to mess up with stamp file details.
>
> There are still a few issues to solve and discuss, though :
>
> ?* Is local.mk inside the main source directory the good solution?
> ? Should we instead provide a configuration option to set the path to
> ? the override file? Or another solution?

I prefer having a configuration option for this, it will be more flexible.

> ?* As the mechanism is implemented today, "make clean" removes the
> ? symbolic link $(O)/build/<pkg>-<version>, but not the stamp files
> ? created inside this source directory. So next time the build is
> ? started, things will go wrong. To solve this, the only solution I
> ? see is to separate the directory with the stamp files from the
> ? source directory, at least in the case of overriden source
> ? directories.

I guess, it won't hurt to have an extra stamp directory for locally overridden
source directories.

> ?* Other things?
>
> Feedback, comments and ideas welcome.
>

When do you plan to release this feature.
Please don't wait too much.

Cheers,
Alper

> Regards,
>
> Thomas
>
> The following changes since commit 7234c23967fa70b760c64700e56bb1778019fbeb:
>
> ?CHANGES: update with recent changes (2011-05-18 21:02:30 +0200)
>
> are available in the git repository at:
> ?http://free-electrons.com/~thomas/buildroot.git for-2011.08/sourcedir
>
> Thomas Petazzoni (4):
> ? ? ?package: enhance infrastructure to support source dir override
> ? ? ?package: remove stamp files in <pkg>-dirclean
> ? ? ?package: add <pkg>-rebuild and <pkg>-reconfigure
> ? ? ?Include a local.mk for package overrides
>
> ?Makefile ? ? ? ? ? ? ? ? ? ?| ? ?1 +
> ?package/Makefile.package.in | ? 30 ++++++++++++++++++++++++++++++
> ?2 files changed, 31 insertions(+), 0 deletions(-)
>
> Thanks,
> --
> Thomas Petazzoni
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
>

[Buildroot] [RFC] Override source directories

[Patryk Benderz]

Dnia 2011-05-18, ?ro o godzinie 23:48 +0200, Thomas Petazzoni pisze:
> Hello,
Hi Thomas,
[cut]
> Feedback, comments and ideas welcome.
Any chance you could implement <pkg>-remove target, parallel to
<pkg>-rebuild target? Because this is what I really miss in Buildroot. I
often find some packages useless, and I have to build from scratch to
remove them. I suppose others would also appreciate "remove package"
function.
-- 
Patryk "LeadMan" Benderz
Linux Registered User #377521
()  ascii ribbon campaign - against html e-mail 
/\  www.asciiribbon.org   - against proprietary attachments

[Buildroot] [RFC] Override source directories

[Thomas Petazzoni]

Hello Patryk,

Le Thu, 09 Jun 2011 08:38:46 +0200,
Patryk Benderz <Patryk.Benderz@esp.pl> a ?crit :

> > Feedback, comments and ideas welcome.
> Any chance you could implement <pkg>-remove target, parallel to
> <pkg>-rebuild target? Because this is what I really miss in
> Buildroot. I often find some packages useless, and I have to build
> from scratch to remove them. I suppose others would also appreciate
> "remove package" function.

This is also on the TODO-list, and some experiments have been made to
implement this, but it is a very different work than the one I'm
currently proposing through this patch set. As Buildroot is designed
today, it does not know which package installed which files in
TARGET_DIR and STAGING_DIR, which makes it impossible for Buildroot to
remove a particular package. The package infrastructure needs to be
extended to maintain the list of the files installed by each package
(this is relatively simple). But then, when removing a package, you
need to ensure that all reverse-dependencies of this package are also
removed, or at least recompiled. This is not completely trivial to
implement.

Regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH 1/4] package: enhance infrastructure to support source dir override

[Alper YILDIRIM]

Hello Thomas,

We started using your "source override" patches, but we encountered a
problem.
Package dependencies are not build when a package source is overriden. 
To overcome this problem we made the following change in Makefile.package.in

ifeq ($($(2)_OVERRIDE_SRCDIR),)
$(1)-configure:		$(1)-patch \
			$$($(2)_TARGET_CONFIGURE)
else
$(1)-configure:		$(1)-symlink \
+                      $(1)-depends \
			$$($(2)_TARGET_CONFIGURE)
endif

$(1)-symlink:		$$($(2)_TARGET_SYMLINK)

Cheers,
Alper


Thomas Petazzoni-2 wrote:
> 
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
>  package/Makefile.package.in |   16 ++++++++++++++++
>  1 files changed, 16 insertions(+), 0 deletions(-)
> 
> diff --git a/package/Makefile.package.in b/package/Makefile.package.in
> index bf4e1b4..1afc18d 100644
> --- a/package/Makefile.package.in
> +++ b/package/Makefile.package.in
> @@ -250,6 +250,13 @@ $(BUILD_DIR)/%/.stamp_extracted:
>  	$(foreach hook,$($(PKG)_POST_EXTRACT_HOOKS),$(call $(hook))$(sep))
>  	$(Q)touch $@
>  
> +# Set up symbolic link to the source directory if the
> +# <pkg>_OVERRIDE_SRCDIR feature is used.
> +$(BUILD_DIR)/%/.stamp_symlinked:
> +	@$(call MESSAGE,"Symlinking to source dir")
> +	ln -s $(SRCDIR) $(@D)
> +	$(Q)touch $@
> +
>  # Patch
>  #
>  # The NOHOSTPKG variable is the uppercased package name, without the
> @@ -418,6 +425,7 @@ $(2)_TARGET_INSTALL_STAGING =
> $$($(2)_DIR)/.stamp_staging_installed
>  $(2)_TARGET_INSTALL_HOST =      $$($(2)_DIR)/.stamp_host_installed
>  $(2)_TARGET_BUILD =		$$($(2)_DIR)/.stamp_built
>  $(2)_TARGET_CONFIGURE =		$$($(2)_DIR)/.stamp_configured
> +$(2)_TARGET_SYMLINK =		$$($(2)_DIR)/.stamp_symlinked
>  $(2)_TARGET_PATCH =		$$($(2)_DIR)/.stamp_patched
>  $(2)_TARGET_EXTRACT =		$$($(2)_DIR)/.stamp_extracted
>  $(2)_TARGET_SOURCE =		$$($(2)_DIR)/.stamp_downloaded
> @@ -463,8 +471,15 @@ $(1)-install-host:      $(1)-build
> $$($(2)_TARGET_INSTALL_HOST)
>  $(1)-build:		$(1)-configure \
>  			$$($(2)_TARGET_BUILD)
>  
> +ifeq ($($(2)_OVERRIDE_SRCDIR),)
>  $(1)-configure:		$(1)-patch \
>  			$$($(2)_TARGET_CONFIGURE)
> +else
> +$(1)-configure:		$(1)-symlink \
> +			$$($(2)_TARGET_CONFIGURE)
> +endif
> +
> +$(1)-symlink:		$$($(2)_TARGET_SYMLINK)
>  
>  $(1)-patch:		$(1)-extract $$($(2)_TARGET_PATCH)
>  
> @@ -492,6 +507,7 @@ $$($(2)_TARGET_INSTALL_STAGING):	PKG=$(2)
>  $$($(2)_TARGET_INSTALL_HOST):           PKG=$(2)
>  $$($(2)_TARGET_BUILD):			PKG=$(2)
>  $$($(2)_TARGET_CONFIGURE):		PKG=$(2)
> +$$($(2)_TARGET_SYMLINK):		SRCDIR=$$($(2)_OVERRIDE_SRCDIR)
>  $$($(2)_TARGET_PATCH):			PKG=$(2)
>  $$($(2)_TARGET_PATCH):			NOHOSTPKG=$(3)
>  $$($(2)_TARGET_EXTRACT):		PKG=$(2)
> -- 
> 
> 

-- 
View this message in context: http://old.nabble.com/-RFC--Override-source-directories-tp31651180p31919307.html
Sent from the Buildroot (busybox) mailing list archive at Nabble.com.

[Buildroot] [PATCH 1/4] package: enhance infrastructure to support source dir override

[Thomas Petazzoni]

Hello Alper,

Le Fri, 24 Jun 2011 05:04:56 -0700 (PDT),
Alper YILDIRIM <yildirimalper@gmail.com> a ?crit :

> We started using your "source override" patches, but we encountered a
> problem.
> Package dependencies are not build when a package source is
> overriden. To overcome this problem we made the following change in
> Makefile.package.in

Yes, this is correct. I also found the problem a few days ago, and
fixed it locally, but I haven't had the time to post a newer version of
the patch set, because I'm fixing other issues with this new feature.

Regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com