* [Buildroot] [PATCH 1/1] Makefile: fix SDK relocation for per-package-dirs
@ 2022-11-10 18:47 Brandon Maier via buildroot
2022-11-24 14:35 ` Arnout Vandecappelle
0 siblings, 1 reply; 4+ messages in thread
From: Brandon Maier via buildroot @ 2022-11-10 18:47 UTC (permalink / raw)
To: buildroot; +Cc: Brandon Maier, Thomas Petazzoni
The relocate-sdk.sh script does not work correctly when
BR2_PER_PACKAGE_DIRECTORIES is enabled. relocate-sdk.sh expects
everything to point at $HOST_DIR, but each package will be pointing at
its $(O)/per-package/*/host.
Before packing up the SDK, scrub the HOST_DIR to replace all matches of
the per-package directory to point at the final HOST_DIR.
Signed-off-by: Brandon Maier <brandon.maier@collins.com>
---
DEVELOPERS | 1 +
Makefile | 1 +
support/scripts/fix-perpkg | 31 +++++++++++++++++++++++++++++++
3 files changed, 33 insertions(+)
create mode 100755 support/scripts/fix-perpkg
diff --git a/DEVELOPERS b/DEVELOPERS
index 81e6cd54ab..238f8754e3 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -488,6 +488,7 @@ F: package/ncdu/
N: Brandon Maier <brandon.maier@collins.com>
F: package/vmtouch/
+F: support/scripts/fix-perpkg
N: Brock Williams <brock@cottonwoodcomputer.com>
F: package/pdmenu/
diff --git a/Makefile b/Makefile
index 7c1c07a2e4..929042396a 100644
--- a/Makefile
+++ b/Makefile
@@ -595,6 +595,7 @@ prepare-sdk: world
@$(call MESSAGE,"Rendering the SDK relocatable")
PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) $(TOPDIR)/support/scripts/fix-rpath host
PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) $(TOPDIR)/support/scripts/fix-rpath staging
+ PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) $(TOPDIR)/support/scripts/fix-perpkg
$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
mkdir -p $(HOST_DIR)/share/buildroot
echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
diff --git a/support/scripts/fix-perpkg b/support/scripts/fix-perpkg
new file mode 100755
index 0000000000..56d7d7a45d
--- /dev/null
+++ b/support/scripts/fix-perpkg
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+
+usage() {
+ cat <<-EOF >&2
+ Usage: fix-perpkg
+
+ This script rewrites file's PER_PACKAGE_DIR to point at the HOST_DIR
+ before they are bundled into an SDK. So that relocate-sdk.sh can find
+ and relocate them.
+ EOF
+ exit 1
+}
+
+if [ "$#" -gt 1 ]; then
+ usage
+fi
+
+# Check if per-package-directories is disabled
+if ! [ -e "$PER_PACKAGE_DIR" ]; then
+ exit
+fi
+
+# Make sure `file` uses the right language
+export LC_ALL=C
+
+# Replace the old path with the new one in all text files
+grep -lr "${PER_PACKAGE_DIR}/[-_a-zA-Z0-9]\+/host" "${HOST_DIR}" \
+ | file --mime-type -f - \
+ | grep ': \+text/[-_a-zA-Z0-9]*$' \
+ | sed 's|: \+text/[-_a-zA-Z0-9]*$||' \
+ | xargs -r sed -i "s|${PER_PACKAGE_DIR}/[-_a-zA-Z0-9]\+/host|${HOST_DIR}|g"
--
2.38.1
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 4+ messages in thread* Re: [Buildroot] [PATCH 1/1] Makefile: fix SDK relocation for per-package-dirs
2022-11-10 18:47 [Buildroot] [PATCH 1/1] Makefile: fix SDK relocation for per-package-dirs Brandon Maier via buildroot
@ 2022-11-24 14:35 ` Arnout Vandecappelle
2022-11-28 20:40 ` [Buildroot] [External] " Maier, Brandon L Collins via buildroot
0 siblings, 1 reply; 4+ messages in thread
From: Arnout Vandecappelle @ 2022-11-24 14:35 UTC (permalink / raw)
To: Brandon Maier, buildroot; +Cc: Thomas Petazzoni
Hi Brandon,
I was going to apply, but I have a few comments/improvements that I prefer if
you bring them to completion.
On 10/11/2022 19:47, Brandon Maier via buildroot wrote:
> The relocate-sdk.sh script does not work correctly when
> BR2_PER_PACKAGE_DIRECTORIES is enabled. relocate-sdk.sh expects
> everything to point at $HOST_DIR, but each package will be pointing at
> its $(O)/per-package/*/host.
>
> Before packing up the SDK, scrub the HOST_DIR to replace all matches of
> the per-package directory to point at the final HOST_DIR.
>
> Signed-off-by: Brandon Maier <brandon.maier@collins.com>
> ---
> DEVELOPERS | 1 +
> Makefile | 1 +
> support/scripts/fix-perpkg | 31 +++++++++++++++++++++++++++++++
> 3 files changed, 33 insertions(+)
> create mode 100755 support/scripts/fix-perpkg
>
> diff --git a/DEVELOPERS b/DEVELOPERS
> index 81e6cd54ab..238f8754e3 100644
> --- a/DEVELOPERS
> +++ b/DEVELOPERS
> @@ -488,6 +488,7 @@ F: package/ncdu/
>
> N: Brandon Maier <brandon.maier@collins.com>
> F: package/vmtouch/
> +F: support/scripts/fix-perpkg
>
> N: Brock Williams <brock@cottonwoodcomputer.com>
> F: package/pdmenu/
> diff --git a/Makefile b/Makefile
> index 7c1c07a2e4..929042396a 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -595,6 +595,7 @@ prepare-sdk: world
> @$(call MESSAGE,"Rendering the SDK relocatable")
> PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) $(TOPDIR)/support/scripts/fix-rpath host
> PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) $(TOPDIR)/support/scripts/fix-rpath staging
> + PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) $(TOPDIR)/support/scripts/fix-perpkg
I'm not sure if it's really useful to make a separate script out of this -
it's just a simple pipeline after all.
> $(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
> mkdir -p $(HOST_DIR)/share/buildroot
> echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
> diff --git a/support/scripts/fix-perpkg b/support/scripts/fix-perpkg
> new file mode 100755
> index 0000000000..56d7d7a45d
> --- /dev/null
> +++ b/support/scripts/fix-perpkg
> @@ -0,0 +1,31 @@
> +#!/usr/bin/env bash
> +
> +usage() {
> + cat <<-EOF >&2
> + Usage: fix-perpkg
> +
> + This script rewrites file's PER_PACKAGE_DIR to point at the HOST_DIR
> + before they are bundled into an SDK. So that relocate-sdk.sh can find
> + and relocate them.
> + EOF
> + exit 1
> +}
> +
> +if [ "$#" -gt 1 ]; then
> + usage
> +fi
> +
> +# Check if per-package-directories is disabled
> +if ! [ -e "$PER_PACKAGE_DIR" ]; then
PER_PACKAGE_DIR is always set, even if per-package-directories is disabled, so
the comment is wrong. I anyway don't think it's necessary to check for this - if
PPD is disabled, then there will never be a match.
> + exit
> +fi
> +
> +# Make sure `file` uses the right language
> +export LC_ALL=C
> +
> +# Replace the old path with the new one in all text files
> +grep -lr "${PER_PACKAGE_DIR}/[-_a-zA-Z0-9]\+/host" "${HOST_DIR}" \
Check what is used in PPD_FIXUP_PATHS:
grep --binary-files=without-match -lr '$(PER_PACKAGE_DIR)/[^/]\+/' $(HOST_DIR)
The --binary-files=without-match speeds things up a lot because it will
immediately skip large binary files.
PPD_FIXUP_PATHS also uses a shell loop instead of xargs; not sure which one is
better.
> + | file --mime-type -f - \
> + | grep ': \+text/[-_a-zA-Z0-9]*$' \
> + | sed 's|: \+text/[-_a-zA-Z0-9]*$||' \
You can merge those two in a single
| sed -n '/\(.*\): \+text\/[-_a-zA-Z0-9]*$/s//\1/p' \
> + | xargs -r sed -i "s|${PER_PACKAGE_DIR}/[-_a-zA-Z0-9]\+/host|${HOST_DIR}|g"
There is no constraint on package names containing only those symbols, so
better use [^/]\+ instead.
However, I think it would be better to refactor fix-perpkg against
PPD_FIXUP_PATHS. It only differens in the replacement part of the final sed, so
that can easily be done with a make function.
Regards,
Arnout
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [Buildroot] [External] Re: [PATCH 1/1] Makefile: fix SDK relocation for per-package-dirs
2022-11-24 14:35 ` Arnout Vandecappelle
@ 2022-11-28 20:40 ` Maier, Brandon L Collins via buildroot
0 siblings, 0 replies; 4+ messages in thread
From: Maier, Brandon L Collins via buildroot @ 2022-11-28 20:40 UTC (permalink / raw)
To: Arnout Vandecappelle, buildroot; +Cc: Thomas Petazzoni
> From: Arnout Vandecappelle <arnout@mind.be>
> On 10/11/2022 19:47, Brandon Maier via buildroot wrote:
> > +# Replace the old path with the new one in all text files
> > +grep -lr "${PER_PACKAGE_DIR}/[-_a-zA-Z0-9]\+/host" "${HOST_DIR}" \
>
> Check what is used in PPD_FIXUP_PATHS:
>
> grep --binary-files=without-match -lr '$(PER_PACKAGE_DIR)/[^/]\+/'
> $(HOST_DIR)
>
> The --binary-files=without-match speeds things up a lot because it will
> immediately skip large binary files.
>
> PPD_FIXUP_PATHS also uses a shell loop instead of xargs; not sure which
> one is
> better.
Before switching to --binary-files, the shell for-loop was much slower closer to
a minute for me. With --binary-files it's about 5 seconds, which is closer to the
xargs which is about 1 second. So I don't mind just using the PPD_FIXUP_PATHS
code.
>
> However, I think it would be better to refactor fix-perpkg against
> PPD_FIXUP_PATHS. It only differens in the replacement part of the final sed,
> so
> that can easily be done with a make function.
Agreed with all comments. And reusing PPD_FIXUP_PATHS will address them.
Will send updated patch shortly.
Thanks,
Brandon
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 4+ messages in thread
* [Buildroot] [PATCH 1/1] Makefile: fix SDK relocation for per-package-dirs
@ 2022-11-28 20:46 Brandon Maier via buildroot
0 siblings, 0 replies; 4+ messages in thread
From: Brandon Maier via buildroot @ 2022-11-28 20:46 UTC (permalink / raw)
To: buildroot; +Cc: Brandon Maier
The relocate-sdk.sh script does not work correctly when
BR2_PER_PACKAGE_DIRECTORIES is enabled. relocate-sdk.sh expects
everything to point at $HOST_DIR, but each package will be pointing at
its $(O)/per-package/*/host.
Use the same command for scrubing host paths during the build, to scrub
to the final host directory location.
Signed-off-by: Brandon Maier <brandon.maier@collins.com>
---
v2:
- drop original implementation in favor of reusing the existing
PPD_FIXUP_PATHS code.
---
Makefile | 1 +
package/pkg-generic.mk | 12 +-----------
package/pkg-utils.mk | 15 +++++++++++++++
3 files changed, 17 insertions(+), 11 deletions(-)
diff --git a/Makefile b/Makefile
index 8d980db3e4..8bf7ed7ad8 100644
--- a/Makefile
+++ b/Makefile
@@ -609,6 +609,7 @@ prepare-sdk: world
@$(call MESSAGE,"Rendering the SDK relocatable")
PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) $(TOPDIR)/support/scripts/fix-rpath host
PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) $(TOPDIR)/support/scripts/fix-rpath staging
+ $(call ppd-fixup-paths,$(BASE_DIR))
$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
mkdir -p $(HOST_DIR)/share/buildroot
echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 8af51f2724..f61076cddf 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -92,18 +92,8 @@ endif
ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
-# Ensure files like .la, .pc, .pri, .cmake, and so on, point to the
-# proper staging and host directories for the current package: find
-# all text files that contain the PPD root, and replace it with the
-# current package's PPD.
define PPD_FIXUP_PATHS
- $(Q)grep --binary-files=without-match -lrZ '$(PER_PACKAGE_DIR)/[^/]\+/' $(HOST_DIR) \
- |while read -d '' f; do \
- file -b --mime-type "$${f}" | grep -q '^text/' || continue; \
- printf '%s\0' "$${f}"; \
- done \
- |xargs -0 --no-run-if-empty \
- $(SED) 's:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g'
+ $(call ppd-fixup-paths,$(PER_PACKAGE_DIR)/$($(PKG)_NAME))
endef
# Remove python's pre-compiled "sysconfigdata", as it may contain paths to
diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index 2951e466c1..77d406fc03 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -233,6 +233,21 @@ define prepare-per-package-directory
$(call per-package-rsync,$(1),host,$(HOST_DIR))
$(call per-package-rsync,$(1),target,$(TARGET_DIR))
endef
+
+# Ensure files like .la, .pc, .pri, .cmake, and so on, point to the
+# proper staging and host directories for the current package: find
+# all text files that contain the PPD root, and replace it with the
+# current package's PPD.
+# $1: destination root directory containing host and staging
+define ppd-fixup-paths
+ $(Q)grep --binary-files=without-match -lrZ '$(PER_PACKAGE_DIR)/[^/]\+/' $(HOST_DIR) \
+ |while read -d '' f; do \
+ file -b --mime-type "$${f}" | grep -q '^text/' || continue; \
+ printf '%s\0' "$${f}"; \
+ done \
+ |xargs -0 --no-run-if-empty \
+ $(SED) 's:$(PER_PACKAGE_DIR)/[^/]\+/:$(1)/:g'
+endef
endif
#
--
2.38.1
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 4+ messages in thread
end of thread, other threads:[~2022-11-28 21:05 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-11-10 18:47 [Buildroot] [PATCH 1/1] Makefile: fix SDK relocation for per-package-dirs Brandon Maier via buildroot
2022-11-24 14:35 ` Arnout Vandecappelle
2022-11-28 20:40 ` [Buildroot] [External] " Maier, Brandon L Collins via buildroot
2022-11-28 20:46 [Buildroot] " Brandon Maier via buildroot
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.