[PATCH 1/3] insane.bbclass: Make do_qa_staging check shebangs

[PATCH 1/3] insane.bbclass: Make do_qa_staging check shebangs

[Paulo Neves]

As reported in the bug report [1], there was no check for shebang sizes
on native scripts and now this is fixed.

The path scope of qa_staging needed to be increased from just
checking libdir to all the relevant SYSROOT_DIRS.

It is possible to skip this check through INSANE_SKIP.

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
---
 meta/classes/insane.bbclass | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 9ca84bace9..dd0cf2a769 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -630,6 +630,11 @@ def qa_check_staged(path,d):
         bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
         skip_pkgconfig = True
 
+    skip_shebang_size = False
+    if 'shebang-size' in skip:
+        bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
+        skip_shebang_size = True
+
     # find all .la and .pc files
     # read the content
     # and check for stuff that looks wrong
@@ -651,6 +656,13 @@ def qa_check_staged(path,d):
                         error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
                         oe.qa.handle_error("pkgconfig", error_msg, d)
 
+            if not skip_shebang_size:
+                errors = {}
+                package_qa_check_shebang_size(path, "", d, None, errors)
+                for e in errors:
+                    oe.qa.handle_error(e, errors[e], d)
+
+
 # Run all package-wide warnfuncs and errorfuncs
 def package_qa_package(warnfuncs, errorfuncs, package, d):
     warnings = {}
@@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
 
 python do_qa_staging() {
     bb.note("QA checking staging")
-    qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
+    sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
+    for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
+        qa_check_staged(os.path.join(sysroot_destdir, sysroot_dir), d)
     oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
 }
 
-- 
2.25.1

[PATCH 2/3] oeqa/selftest: Add test for shebang overflow

[Paulo Neves]

Make sure we do not stage any executable with a bigger shebang
than 128. Fixes [1]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
---
 .../sysroot-test/sysroot-shebang-test_1.0.bb         | 12 ++++++++++++
 meta/lib/oeqa/selftest/cases/sysroot.py              | 10 ++++++++++
 2 files changed, 22 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb

diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
new file mode 100644
index 0000000000..6c834be897
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Check that shebang does not exceed 128 characters"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    echo '#!BiM3cnVd1Amtv6PG+FynrQiVMbZnX5ELgF21q3EkuB+44JEGWtq8TvBJ7EGidfVs3eR3wVOUbLnjYDlKUWcm7YC/ute7f+KDHbwxziRUSUBZAUqgjiQdfQ0HnxajI0ozbM863E9JV9k13yZKYfh9/zR77Y6Dl4Dd3zOWS75LSpkAXV' > ${D}${bindir}/max-shebang
+    chmod 755 ${D}${bindir}/max-shebang
+}
+
+BBCLASSEXTEND = "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index 315d1a61c2..9457f1e3ac 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -35,3 +35,13 @@ TESTSTRING:pn-sysroot-test-arch1 = "%s"
 TESTSTRING:pn-sysroot-test-arch2 = "%s"
 """ % (uuid1, uuid2))
         bitbake("sysroot-test")
+
+    def test_sysroot_max_shebang(self):
+        """
+        Summary:   Check max shebang triggers. To confirm [YOCTO #11053] is closed.
+        Expected:  Fail when a shebang bigger than the max shebang-size is reached.
+        Author:    Paulo Neves <ptsneves@gmail.com>
+        """
+        expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
+        res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
-- 
2.25.1

[PATCH 3/3] oeqa/selftest: Test staged .la and .pc files

[Paulo Neves]

These files are checked by qa_check_staged but there was no
test cases for whether the tests actually worked. Now there
are.
---
 .../sysroot-test/sysroot-la-test_1.0.bb       | 16 ++++++++++
 .../sysroot-test/sysroot-pc-test_1.0.bb       | 12 +++++++
 meta/lib/oeqa/selftest/cases/sysroot.py       | 32 +++++++++++++++++++
 3 files changed, 60 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb

diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
new file mode 100644
index 0000000000..21f06782fb
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
@@ -0,0 +1,16 @@
+SUMMARY = "Produce a broken la file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+# remove-libtool.bbclass is inherited by default and removes all
+# .la files which for this test we specifically do not want.
+REMOVE_LIBTOOL_LA = "0"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/la-test.la
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
new file mode 100644
index 0000000000..e748310fc4
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Produce a broken pc file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/test.pc
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index 9457f1e3ac..588fc8c713 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -45,3 +45,35 @@ TESTSTRING:pn-sysroot-test-arch2 = "%s"
         expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
         res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
         self.assertTrue(expected in res.output, msg=res.output)
+
+    def test_sysroot_la(self):
+        """
+        Summary:   Check that workdir paths are not contained in .la files.
+        Expected:  Fail when a workdir path is found in the file content.
+        Author:    Paulo Neves <ptsneves@gmail.com>
+        """
+        expected = "la-test.la failed sanity test (workdir) in path"
+
+        res = bitbake("sysroot-la-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+
+        res = bitbake("sysroot-la-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+
+    def test_sysroot_pkgconfig(self):
+        """
+        Summary:   Check that tmpdir paths are not contained in .pc files.
+        Expected:  Fail when a tmpdir path is found in the file content.
+        Author:    Paulo Neves <ptsneves@gmail.com>
+        """
+        expected = "test.pc failed sanity test (tmpdir) in path"
+
+        res = bitbake("sysroot-pc-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
+        self.assertTrue(expected in res.output, msg=res.output)
+
+        res = bitbake("sysroot-pc-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
-- 
2.25.1

[PATCH v2 1/5] python: Avoid shebang overflow on python-config.py

[Paulo Neves]

The native path may be too big, leading to shebang
overflow. Just use the #!/usr/bin/env python3.

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 ...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
 .../recipes-devtools/python/python3_3.10.4.bb |  1 +
 2 files changed, 30 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch

diff --git a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
new file mode 100644
index 0000000000..9f23278a30
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
@@ -0,0 +1,29 @@
+From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
+From: Paulo Neves <ptsneves@gmail.com>
+Date: Tue, 7 Jun 2022 16:16:41 +0200
+Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
+
+The whole native path may be too big, leading to shebang
+overflow. Let's just use the env shebang.
+
+Upstream-Status: Inappropriate [distribution]
+---
+ Makefile.pre.in | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
+ 	@ # Substitution happens here, as the completely-expanded BINDIR
+ 	@ # is not available in configure
+ 	sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
++	@ # Otherwise we might get huge shebangs with native paths
++	sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
+ 	@ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
+ 	LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
+ 	@  # In OpenEmbedded, always use the python version of the script, the shell
+-- 
+2.25.1
+
diff --git a/meta/recipes-devtools/python/python3_3.10.4.bb b/meta/recipes-devtools/python/python3_3.10.4.bb
index 6bd3a6aba8..357025f856 100644
--- a/meta/recipes-devtools/python/python3_3.10.4.bb
+++ b/meta/recipes-devtools/python/python3_3.10.4.bb
@@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
            file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
            file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
            file://deterministic_imports.patch \
+           file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
            "
 
 SRC_URI:append:class-native = " \
-- 
2.25.1

[PATCH v2 2/5] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2

[Paulo Neves]

A native path can cause a shebang overflow on gtkdoc-mkhtml.
Replace it with /usr/bin/env.

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb b/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
index 150d2c0b23..392913fcc6 100644
--- a/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
+++ b/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
@@ -43,6 +43,7 @@ do_install:append () {
         ${datadir}/gtk-doc/python/gtkdoc/config.py; do
         sed -e 's,${RECIPE_SYSROOT_NATIVE}/usr/bin/pkg-config,${bindir}/pkg-config,' \
             -e 's,${HOSTTOOLS_DIR}/python3,${bindir}/python3,' \
+            -e '1s|^#!.*|#!/usr/bin/env python3|' \
             -i ${D}$fn
     done
 }
-- 
2.25.1

[PATCH v2 3/5] insane.bbclass: Make do_qa_staging check shebangs

[Paulo Neves]

As reported in the bug report [1], there was no check for shebang
sizes on native scripts and now this is fixed.

The path scope of the qa_staging was increased from just checking
libdir to all the relevant SYSROOT_DIRS.

It is possible to skip this check through INSANE_SKIP.

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 meta/classes/insane.bbclass | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 9ca84bace9..b2951a48fe 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -630,6 +630,11 @@ def qa_check_staged(path,d):
         bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
         skip_pkgconfig = True
 
+    skip_shebang_size = False
+    if 'shebang-size' in skip:
+        bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
+        skip_shebang_size = True
+
     # find all .la and .pc files
     # read the content
     # and check for stuff that looks wrong
@@ -651,6 +656,13 @@ def qa_check_staged(path,d):
                         error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
                         oe.qa.handle_error("pkgconfig", error_msg, d)
 
+            if not skip_shebang_size:
+                errors = {}
+                package_qa_check_shebang_size(path, "", d, None, errors)
+                for e in errors:
+                    oe.qa.handle_error(e, errors[e], d)
+
+
 # Run all package-wide warnfuncs and errorfuncs
 def package_qa_package(warnfuncs, errorfuncs, package, d):
     warnings = {}
@@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
 
 python do_qa_staging() {
     bb.note("QA checking staging")
-    qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
+    sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
+    for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
+        qa_check_staged(sysroot_destdir + sysroot_dir, d)
     oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
 }
 
-- 
2.25.1

[PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow

[Paulo Neves]

Make sure we do not stage any executable with a bigger shebang
than 128. Fixes [1]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 .../sysroot-test/sysroot-shebang-test_1.0.bb         | 12 ++++++++++++
 meta/lib/oeqa/selftest/cases/sysroot.py              | 10 ++++++++++
 2 files changed, 22 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb

diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
new file mode 100644
index 0000000000..6c834be897
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Check that shebang does not exceed 128 characters"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    echo '#!BiM3cnVd1Amtv6PG+FynrQiVMbZnX5ELgF21q3EkuB+44JEGWtq8TvBJ7EGidfVs3eR3wVOUbLnjYDlKUWcm7YC/ute7f+KDHbwxziRUSUBZAUqgjiQdfQ0HnxajI0ozbM863E9JV9k13yZKYfh9/zR77Y6Dl4Dd3zOWS75LSpkAXV' > ${D}${bindir}/max-shebang
+    chmod 755 ${D}${bindir}/max-shebang
+}
+
+BBCLASSEXTEND = "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index 315d1a61c2..9457f1e3ac 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -35,3 +35,13 @@ TESTSTRING:pn-sysroot-test-arch1 = "%s"
 TESTSTRING:pn-sysroot-test-arch2 = "%s"
 """ % (uuid1, uuid2))
         bitbake("sysroot-test")
+
+    def test_sysroot_max_shebang(self):
+        """
+        Summary:   Check max shebang triggers. To confirm [YOCTO #11053] is closed.
+        Expected:  Fail when a shebang bigger than the max shebang-size is reached.
+        Author:    Paulo Neves <ptsneves@gmail.com>
+        """
+        expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
+        res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
-- 
2.25.1

[PATCH v2 5/5] oeqa/selftest: Test staged .la and .pc files

[Paulo Neves]

These files are checked by qa_check_staged but there was no
test cases for whether the tests actually worked. Now there
are.

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 .../sysroot-test/sysroot-la-test_1.0.bb       | 16 ++++++++++
 .../sysroot-test/sysroot-pc-test_1.0.bb       | 12 +++++++
 meta/lib/oeqa/selftest/cases/sysroot.py       | 32 +++++++++++++++++++
 3 files changed, 60 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb

diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
new file mode 100644
index 0000000000..21f06782fb
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
@@ -0,0 +1,16 @@
+SUMMARY = "Produce a broken la file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+# remove-libtool.bbclass is inherited by default and removes all
+# .la files which for this test we specifically do not want.
+REMOVE_LIBTOOL_LA = "0"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/la-test.la
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
new file mode 100644
index 0000000000..e748310fc4
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Produce a broken pc file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/test.pc
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index 9457f1e3ac..588fc8c713 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -45,3 +45,35 @@ TESTSTRING:pn-sysroot-test-arch2 = "%s"
         expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
         res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
         self.assertTrue(expected in res.output, msg=res.output)
+
+    def test_sysroot_la(self):
+        """
+        Summary:   Check that workdir paths are not contained in .la files.
+        Expected:  Fail when a workdir path is found in the file content.
+        Author:    Paulo Neves <ptsneves@gmail.com>
+        """
+        expected = "la-test.la failed sanity test (workdir) in path"
+
+        res = bitbake("sysroot-la-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+
+        res = bitbake("sysroot-la-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+
+    def test_sysroot_pkgconfig(self):
+        """
+        Summary:   Check that tmpdir paths are not contained in .pc files.
+        Expected:  Fail when a tmpdir path is found in the file content.
+        Author:    Paulo Neves <ptsneves@gmail.com>
+        """
+        expected = "test.pc failed sanity test (tmpdir) in path"
+
+        res = bitbake("sysroot-pc-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
+        self.assertTrue(expected in res.output, msg=res.output)
+
+        res = bitbake("sysroot-pc-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
-- 
2.25.1

Re: [OE-core] [PATCH v2 1/5] python: Avoid shebang overflow on python-config.py

[Alexander Kanavin]

I'm not sure this is Inappropriate. Can you send this to upstream nevertheless?

Alex

On Tue, 7 Jun 2022 at 17:11, Paulo Neves <ptsneves@gmail.com> wrote:
>
> The native path may be too big, leading to shebang
> overflow. Just use the #!/usr/bin/env python3.
>
> Signed-off-by: Paulo Neves <ptsneves@gmail.com>
> ---
>  ...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
>  .../recipes-devtools/python/python3_3.10.4.bb |  1 +
>  2 files changed, 30 insertions(+)
>  create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
>
> diff --git a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
> new file mode 100644
> index 0000000000..9f23278a30
> --- /dev/null
> +++ b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
> @@ -0,0 +1,29 @@
> +From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
> +From: Paulo Neves <ptsneves@gmail.com>
> +Date: Tue, 7 Jun 2022 16:16:41 +0200
> +Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
> +
> +The whole native path may be too big, leading to shebang
> +overflow. Let's just use the env shebang.
> +
> +Upstream-Status: Inappropriate [distribution]
> +---
> + Makefile.pre.in | 2 ++
> + 1 file changed, 2 insertions(+)
> +
> +diff --git a/Makefile.pre.in b/Makefile.pre.in
> +index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
> +--- a/Makefile.pre.in
> ++++ b/Makefile.pre.in
> +@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
> +       @ # Substitution happens here, as the completely-expanded BINDIR
> +       @ # is not available in configure
> +       sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
> ++      @ # Otherwise we might get huge shebangs with native paths
> ++      sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
> +       @ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
> +       LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
> +       @  # In OpenEmbedded, always use the python version of the script, the shell
> +--
> +2.25.1
> +
> diff --git a/meta/recipes-devtools/python/python3_3.10.4.bb b/meta/recipes-devtools/python/python3_3.10.4.bb
> index 6bd3a6aba8..357025f856 100644
> --- a/meta/recipes-devtools/python/python3_3.10.4.bb
> +++ b/meta/recipes-devtools/python/python3_3.10.4.bb
> @@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
>             file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
>             file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
>             file://deterministic_imports.patch \
> +           file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
>             "
>
>  SRC_URI:append:class-native = " \
> --
> 2.25.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#166670): https://lists.openembedded.org/g/openembedded-core/message/166670
> Mute This Topic: https://lists.openembedded.org/mt/91602889/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH v2 1/5] python: Avoid shebang overflow on python-config.py

[Paulo Neves]

[-- Attachment #1: Type: text/plain, Size: 4038 bytes --]

I think this patch will never be accepted because it hard-codes and depends
on env. I recall that red hat for example does not accept the use of env,
possibly for security reasons.

Paulo Neves

On Tue, Jun 7, 2022, 18:53 Alexander Kanavin <alex.kanavin@gmail.com> wrote:

> I'm not sure this is Inappropriate. Can you send this to upstream
> nevertheless?
>
> Alex
>
> On Tue, 7 Jun 2022 at 17:11, Paulo Neves <ptsneves@gmail.com> wrote:
> >
> > The native path may be too big, leading to shebang
> > overflow. Just use the #!/usr/bin/env python3.
> >
> > Signed-off-by: Paulo Neves <ptsneves@gmail.com>
> > ---
> >  ...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
> >  .../recipes-devtools/python/python3_3.10.4.bb |  1 +
> >  2 files changed, 30 insertions(+)
> >  create mode 100644
> meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
> >
> > diff --git
> a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
> b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
> > new file mode 100644
> > index 0000000000..9f23278a30
> > --- /dev/null
> > +++
> b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
> > @@ -0,0 +1,29 @@
> > +From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
> > +From: Paulo Neves <ptsneves@gmail.com>
> > +Date: Tue, 7 Jun 2022 16:16:41 +0200
> > +Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
> > +
> > +The whole native path may be too big, leading to shebang
> > +overflow. Let's just use the env shebang.
> > +
> > +Upstream-Status: Inappropriate [distribution]
> > +---
> > + Makefile.pre.in | 2 ++
> > + 1 file changed, 2 insertions(+)
> > +
> > +diff --git a/Makefile.pre.in b/Makefile.pre.in
> > +index
> f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176
> 100644
> > +--- a/Makefile.pre.in
> > ++++ b/Makefile.pre.in
> > +@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in
> Misc/python-config.sh
> > +       @ # Substitution happens here, as the completely-expanded BINDIR
> > +       @ # is not available in configure
> > +       sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," <
> $(srcdir)/Misc/python-config.in >python-config.py
> > ++      @ # Otherwise we might get huge shebangs with native paths
> > ++      sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
> > +       @ # Replace makefile compat. variable references with shell
> script compat. ones; $(VAR) -> ${VAR}
> > +       LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' <
> Misc/python-config.sh >python-config
> > +       @  # In OpenEmbedded, always use the python version of the
> script, the shell
> > +--
> > +2.25.1
> > +
> > diff --git a/meta/recipes-devtools/python/python3_3.10.4.bb
> b/meta/recipes-devtools/python/python3_3.10.4.bb
> > index 6bd3a6aba8..357025f856 100644
> > --- a/meta/recipes-devtools/python/python3_3.10.4.bb
> > +++ b/meta/recipes-devtools/python/python3_3.10.4.bb
> > @@ -34,6 +34,7 @@ SRC_URI = "
> http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
> >
>  file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
> >
>  file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
> >             file://deterministic_imports.patch \
> > +           file://0001-Avoid-shebang-overflow-on-python-config.py.patch
> \
> >             "
> >
> >  SRC_URI:append:class-native = " \
> > --
> > 2.25.1
> >
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#166670):
> https://lists.openembedded.org/g/openembedded-core/message/166670
> > Mute This Topic: https://lists.openembedded.org/mt/91602889/1686489
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> alex.kanavin@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>

[-- Attachment #2: Type: text/html, Size: 6776 bytes --]

Re: [OE-core] [PATCH v2 1/5] python: Avoid shebang overflow on python-config.py

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 4477 bytes --]

Please submit it nevertheless. If upstream says no, we'll just mark it as
denied, but it’s not appropriate to declare it inappropriate a priori.

Alex

On Tue 7. Jun 2022 at 23.03, Paulo Neves <ptsneves@gmail.com> wrote:

> I think this patch will never be accepted because it hard-codes and
> depends on env. I recall that red hat for example does not accept the use
> of env, possibly for security reasons.
>
> Paulo Neves
>
> On Tue, Jun 7, 2022, 18:53 Alexander Kanavin <alex.kanavin@gmail.com>
> wrote:
>
>> I'm not sure this is Inappropriate. Can you send this to upstream
>> nevertheless?
>>
>> Alex
>>
>> On Tue, 7 Jun 2022 at 17:11, Paulo Neves <ptsneves@gmail.com> wrote:
>> >
>> > The native path may be too big, leading to shebang
>> > overflow. Just use the #!/usr/bin/env python3.
>> >
>> > Signed-off-by: Paulo Neves <ptsneves@gmail.com>
>> > ---
>> >  ...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
>> >  .../recipes-devtools/python/python3_3.10.4.bb |  1 +
>> >  2 files changed, 30 insertions(+)
>> >  create mode 100644
>> meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
>> >
>> > diff --git
>> a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
>> b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
>> > new file mode 100644
>> > index 0000000000..9f23278a30
>> > --- /dev/null
>> > +++
>> b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
>> > @@ -0,0 +1,29 @@
>> > +From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
>> > +From: Paulo Neves <ptsneves@gmail.com>
>> > +Date: Tue, 7 Jun 2022 16:16:41 +0200
>> > +Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
>> > +
>> > +The whole native path may be too big, leading to shebang
>> > +overflow. Let's just use the env shebang.
>> > +
>> > +Upstream-Status: Inappropriate [distribution]
>> > +---
>> > + Makefile.pre.in | 2 ++
>> > + 1 file changed, 2 insertions(+)
>> > +
>> > +diff --git a/Makefile.pre.in b/Makefile.pre.in
>> > +index
>> f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176
>> 100644
>> > +--- a/Makefile.pre.in
>> > ++++ b/Makefile.pre.in
>> > +@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in
>> Misc/python-config.sh
>> > +       @ # Substitution happens here, as the completely-expanded BINDIR
>> > +       @ # is not available in configure
>> > +       sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," <
>> $(srcdir)/Misc/python-config.in >python-config.py
>> > ++      @ # Otherwise we might get huge shebangs with native paths
>> > ++      sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
>> > +       @ # Replace makefile compat. variable references with shell
>> script compat. ones; $(VAR) -> ${VAR}
>> > +       LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' <
>> Misc/python-config.sh >python-config
>> > +       @  # In OpenEmbedded, always use the python version of the
>> script, the shell
>> > +--
>> > +2.25.1
>> > +
>> > diff --git a/meta/recipes-devtools/python/python3_3.10.4.bb
>> b/meta/recipes-devtools/python/python3_3.10.4.bb
>> > index 6bd3a6aba8..357025f856 100644
>> > --- a/meta/recipes-devtools/python/python3_3.10.4.bb
>> > +++ b/meta/recipes-devtools/python/python3_3.10.4.bb
>> > @@ -34,6 +34,7 @@ SRC_URI = "
>> http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
>> >
>>  file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
>> >
>>  file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
>> >             file://deterministic_imports.patch \
>> > +
>>  file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
>> >             "
>> >
>> >  SRC_URI:append:class-native = " \
>> > --
>> > 2.25.1
>> >
>> >
>> > -=-=-=-=-=-=-=-=-=-=-=-
>> > Links: You receive all messages sent to this group.
>> > View/Reply Online (#166670):
>> https://lists.openembedded.org/g/openembedded-core/message/166670
>> > Mute This Topic: https://lists.openembedded.org/mt/91602889/1686489
>> > Group Owner: openembedded-core+owner@lists.openembedded.org
>> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
>> alex.kanavin@gmail.com]
>> > -=-=-=-=-=-=-=-=-=-=-=-
>> >
>>
>

[-- Attachment #2: Type: text/html, Size: 7481 bytes --]

Re: [OE-core] [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow

[Luca Ceresoli]

Hi Paulo,

On Tue,  7 Jun 2022 17:11:22 +0200
"Paulo Neves" <ptsneves@gmail.com> wrote:

> Make sure we do not stage any executable with a bigger shebang
> than 128. Fixes [1]
> 
> [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
> 
> Signed-off-by: Paulo Neves <ptsneves@gmail.com>

This check seems to be working very well!! It triggered a huge amount
of build failures on the autobuilders due to libcheck having a shebang
too long in the checkmk script, e.g.:

#! /home/pokybuild/yocto-worker/genericx86-64-alt/build/build/tmp/work/x86_64-linux/libcheck-native/0.15.2-r0/recipe-sysroot-native/usr/bin/gawk -f

Here are a few logs:

https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/14/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/11/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/11/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/12/logs/errors

It would be great if you could add another patch to your series to fix
libcheck, and also to do 'bitbake world' to test as many packages as
possible before discovering from the autobuilders.

Thank you!

-- 
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow

[Paulo Neves]

On 6/8/22 14:53, Luca Ceresoli wrote:
> Hi Paulo,
>
> On Tue,  7 Jun 2022 17:11:22 +0200
> "Paulo Neves" <ptsneves@gmail.com> wrote:
>
>> Make sure we do not stage any executable with a bigger shebang
>> than 128. Fixes [1]
>>
>> [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
>>
>> Signed-off-by: Paulo Neves <ptsneves@gmail.com>
> This check seems to be working very well!! It triggered a huge amount
> of build failures on the autobuilders due to libcheck having a shebang
> too long in the checkmk script, e.g.:
>
> #! /home/pokybuild/yocto-worker/genericx86-64-alt/build/build/tmp/work/x86_64-linux/libcheck-native/0.15.2-r0/recipe-sysroot-native/usr/bin/gawk -f
>
> Here are a few logs:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/14/logs/errors
> https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/11/logs/errors
> https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/11/logs/errors
> https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/12/logs/errors

I am not seeing an immediate way to fix this optimally. The go-to 
solution to this class of issues is to just make
the #!/usr/bin/env <interpreter>. The issue is that there is an extra 
-f, which with /usr/bin/env, will not work. The awk manual also implies 
this is not optional for standalone scripts. I think we can create a 
wrapper, or maybe we already have such a wrapper?
>
> It would be great if you could add another patch to your series to fix
> libcheck, and also to do 'bitbake world' to test as many packages as
> possible before discovering from the autobuilders.
It takes quite a while on my computer and often i get out of disk space. 
I tried asking the linaro guys for a tuxsuite token but no answer yet. 
If you have some way of getting resources to make builds let me know.
> Thank you!

Thank you
Paulo Neves

Re: [OE-core] [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow

[Luca Ceresoli]

On Wed, 8 Jun 2022 16:45:20 +0200
"Paulo Neves" <ptsneves@gmail.com> wrote:

> On 6/8/22 14:53, Luca Ceresoli wrote:
> > Hi Paulo,
> >
> > On Tue,  7 Jun 2022 17:11:22 +0200
> > "Paulo Neves" <ptsneves@gmail.com> wrote:
> >  
> >> Make sure we do not stage any executable with a bigger shebang
> >> than 128. Fixes [1]
> >>
> >> [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
> >>
> >> Signed-off-by: Paulo Neves <ptsneves@gmail.com>  
> > This check seems to be working very well!! It triggered a huge amount
> > of build failures on the autobuilders due to libcheck having a shebang
> > too long in the checkmk script, e.g.:
> >
> > #! /home/pokybuild/yocto-worker/genericx86-64-alt/build/build/tmp/work/x86_64-linux/libcheck-native/0.15.2-r0/recipe-sysroot-native/usr/bin/gawk -f
> >
> > Here are a few logs:
> >
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/14/logs/errors
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/11/logs/errors
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/11/logs/errors
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/12/logs/errors  
> 
> I am not seeing an immediate way to fix this optimally. The go-to 
> solution to this class of issues is to just make
> the #!/usr/bin/env <interpreter>. The issue is that there is an extra 
> -f, which with /usr/bin/env, will not work. The awk manual also implies 
> this is not optional for standalone scripts. I think we can create a 
> wrapper, or maybe we already have such a wrapper?
> >
> > It would be great if you could add another patch to your series to fix
> > libcheck, and also to do 'bitbake world' to test as many packages as
> > possible before discovering from the autobuilders.  
> It takes quite a while on my computer and often i get out of disk space. 
> I tried asking the linaro guys for a tuxsuite token but no answer yet. 
> If you have some way of getting resources to make builds let me know.

Did you enable rm_work?

-- 
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow

[Luca Ceresoli]

Hi Paulo,

On Wed, 8 Jun 2022 14:53:05 +0200
"Luca Ceresoli via lists.openembedded.org"
<luca.ceresoli=bootlin.com@lists.openembedded.org> wrote:

> Hi Paulo,
> 
> On Tue,  7 Jun 2022 17:11:22 +0200
> "Paulo Neves" <ptsneves@gmail.com> wrote:
> 
> > Make sure we do not stage any executable with a bigger shebang
> > than 128. Fixes [1]
> > 
> > [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
> > 
> > Signed-off-by: Paulo Neves <ptsneves@gmail.com>  
> 
> This check seems to be working very well!! It triggered a huge amount
> of build failures on the autobuilders due to libcheck having a shebang
> too long in the checkmk script, e.g.:
> 
> #! /home/pokybuild/yocto-worker/genericx86-64-alt/build/build/tmp/work/x86_64-linux/libcheck-native/0.15.2-r0/recipe-sysroot-native/usr/bin/gawk -f
> 
> Here are a few logs:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/14/logs/errors
> https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/11/logs/errors
> https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/11/logs/errors
> https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/12/logs/errors
> 
> It would be great if you could add another patch to your series to fix
> libcheck, and also to do 'bitbake world' to test as many packages as
> possible before discovering from the autobuilders.

Here are more failures:

https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/12/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/62/builds/5340/steps/12/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/62/builds/5340/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/13/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/13/logs/stdio

This time the error is:

stdio: ERROR: core-image-sato-1.0-r0 do_testsdk: The toolchain <...> is not built. Build it before running the tests: 'bitbake <image> -c populate_sdk' .

I'm not sure exactly how your code triggers such error, but it appeared
when testing on the autobuilders with this patch series and disappeared
when I removed only these 5 patches, thus it seems related.

-- 
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[PATCH v3 1/7] python: Avoid shebang overflow on python-config.py

[Paulo Neves]

The native path may be too big, leading to shebang
overflow. Just use the #!/usr/bin/env python3.

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 ...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
 .../recipes-devtools/python/python3_3.10.4.bb |  1 +
 2 files changed, 30 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch

diff --git a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
new file mode 100644
index 0000000000..9f23278a30
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
@@ -0,0 +1,29 @@
+From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
+From: Paulo Neves <ptsneves@gmail.com>
+Date: Tue, 7 Jun 2022 16:16:41 +0200
+Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
+
+The whole native path may be too big, leading to shebang
+overflow. Let's just use the env shebang.
+
+Upstream-Status: Inappropriate [distribution]
+---
+ Makefile.pre.in | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
+ 	@ # Substitution happens here, as the completely-expanded BINDIR
+ 	@ # is not available in configure
+ 	sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
++	@ # Otherwise we might get huge shebangs with native paths
++	sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
+ 	@ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
+ 	LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
+ 	@  # In OpenEmbedded, always use the python version of the script, the shell
+-- 
+2.25.1
+
diff --git a/meta/recipes-devtools/python/python3_3.10.4.bb b/meta/recipes-devtools/python/python3_3.10.4.bb
index 6bd3a6aba8..357025f856 100644
--- a/meta/recipes-devtools/python/python3_3.10.4.bb
+++ b/meta/recipes-devtools/python/python3_3.10.4.bb
@@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
            file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
            file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
            file://deterministic_imports.patch \
+           file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
            "
 
 SRC_URI:append:class-native = " \
-- 
2.25.1

[PATCH v3 2/7] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2

[Paulo Neves]

A native path can cause a shebang overflow on gtkdoc-mkhtml.
Replace it with /usr/bin/env.

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb b/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
index 150d2c0b23..392913fcc6 100644
--- a/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
+++ b/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
@@ -43,6 +43,7 @@ do_install:append () {
         ${datadir}/gtk-doc/python/gtkdoc/config.py; do
         sed -e 's,${RECIPE_SYSROOT_NATIVE}/usr/bin/pkg-config,${bindir}/pkg-config,' \
             -e 's,${HOSTTOOLS_DIR}/python3,${bindir}/python3,' \
+            -e '1s|^#!.*|#!/usr/bin/env python3|' \
             -i ${D}$fn
     done
 }
-- 
2.25.1

[PATCH v3 3/7] insane.bbclass: Make do_qa_staging check shebangs

[Paulo Neves]

As reported in the bug report [1], there was no check for shebang
sizes on native scripts and now this is fixed.

The path scope of the qa_staging was increased from just checking
libdir to all the relevant SYSROOT_DIRS.

It is possible to skip this check through INSANE_SKIP.

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 meta/classes/insane.bbclass | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 9ca84bace9..b2951a48fe 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -630,6 +630,11 @@ def qa_check_staged(path,d):
         bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
         skip_pkgconfig = True
 
+    skip_shebang_size = False
+    if 'shebang-size' in skip:
+        bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
+        skip_shebang_size = True
+
     # find all .la and .pc files
     # read the content
     # and check for stuff that looks wrong
@@ -651,6 +656,13 @@ def qa_check_staged(path,d):
                         error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
                         oe.qa.handle_error("pkgconfig", error_msg, d)
 
+            if not skip_shebang_size:
+                errors = {}
+                package_qa_check_shebang_size(path, "", d, None, errors)
+                for e in errors:
+                    oe.qa.handle_error(e, errors[e], d)
+
+
 # Run all package-wide warnfuncs and errorfuncs
 def package_qa_package(warnfuncs, errorfuncs, package, d):
     warnings = {}
@@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
 
 python do_qa_staging() {
     bb.note("QA checking staging")
-    qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
+    sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
+    for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
+        qa_check_staged(sysroot_destdir + sysroot_dir, d)
     oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
 }
 
-- 
2.25.1

[PATCH v3 4/7] oeqa/selftest: Add test for shebang overflow

[Paulo Neves]

Make sure we do not stage any executable with a bigger shebang
than 128. Fixes [1]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 .../sysroot-test/sysroot-shebang-test_1.0.bb         | 12 ++++++++++++
 meta/lib/oeqa/selftest/cases/sysroot.py              | 10 ++++++++++
 2 files changed, 22 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb

diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
new file mode 100644
index 0000000000..6c834be897
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Check that shebang does not exceed 128 characters"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    echo '#!BiM3cnVd1Amtv6PG+FynrQiVMbZnX5ELgF21q3EkuB+44JEGWtq8TvBJ7EGidfVs3eR3wVOUbLnjYDlKUWcm7YC/ute7f+KDHbwxziRUSUBZAUqgjiQdfQ0HnxajI0ozbM863E9JV9k13yZKYfh9/zR77Y6Dl4Dd3zOWS75LSpkAXV' > ${D}${bindir}/max-shebang
+    chmod 755 ${D}${bindir}/max-shebang
+}
+
+BBCLASSEXTEND = "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index 315d1a61c2..9457f1e3ac 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -35,3 +35,13 @@ TESTSTRING:pn-sysroot-test-arch1 = "%s"
 TESTSTRING:pn-sysroot-test-arch2 = "%s"
 """ % (uuid1, uuid2))
         bitbake("sysroot-test")
+
+    def test_sysroot_max_shebang(self):
+        """
+        Summary:   Check max shebang triggers. To confirm [YOCTO #11053] is closed.
+        Expected:  Fail when a shebang bigger than the max shebang-size is reached.
+        Author:    Paulo Neves <ptsneves@gmail.com>
+        """
+        expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
+        res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
-- 
2.25.1

[PATCH v3 5/7] oeqa/selftest: Test staged .la and .pc files

[Paulo Neves]

These files are checked by qa_check_staged but there was no
test cases for whether the tests actually worked. Now there
are.

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 .../sysroot-test/sysroot-la-test_1.0.bb       | 16 ++++++++++
 .../sysroot-test/sysroot-pc-test_1.0.bb       | 12 +++++++
 meta/lib/oeqa/selftest/cases/sysroot.py       | 32 +++++++++++++++++++
 3 files changed, 60 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb

diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
new file mode 100644
index 0000000000..21f06782fb
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
@@ -0,0 +1,16 @@
+SUMMARY = "Produce a broken la file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+# remove-libtool.bbclass is inherited by default and removes all
+# .la files which for this test we specifically do not want.
+REMOVE_LIBTOOL_LA = "0"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/la-test.la
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
new file mode 100644
index 0000000000..e748310fc4
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Produce a broken pc file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/test.pc
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index 9457f1e3ac..588fc8c713 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -45,3 +45,35 @@ TESTSTRING:pn-sysroot-test-arch2 = "%s"
         expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
         res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
         self.assertTrue(expected in res.output, msg=res.output)
+
+    def test_sysroot_la(self):
+        """
+        Summary:   Check that workdir paths are not contained in .la files.
+        Expected:  Fail when a workdir path is found in the file content.
+        Author:    Paulo Neves <ptsneves@gmail.com>
+        """
+        expected = "la-test.la failed sanity test (workdir) in path"
+
+        res = bitbake("sysroot-la-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+
+        res = bitbake("sysroot-la-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+
+    def test_sysroot_pkgconfig(self):
+        """
+        Summary:   Check that tmpdir paths are not contained in .pc files.
+        Expected:  Fail when a tmpdir path is found in the file content.
+        Author:    Paulo Neves <ptsneves@gmail.com>
+        """
+        expected = "test.pc failed sanity test (tmpdir) in path"
+
+        res = bitbake("sysroot-pc-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
+        self.assertTrue(expected in res.output, msg=res.output)
+
+        res = bitbake("sysroot-pc-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
-- 
2.25.1

[PATCH v3 6/7] utils: Add cmdline_shebang_wrapper util.

[Paulo Neves]

Useful to work around shebang relocation issues, where
shebangs are too long or have arguments in them, thus preventing them
from using the /usr/bin/env shebang.
---
 .../wrapper/cmdline-shebang-wrapper-test.bb   | 21 ++++++++++++
 .../recipes-test/wrapper/files/test.awk       |  2 ++
 meta/classes/utils.bbclass                    | 34 +++++++++++++++++++
 meta/lib/oeqa/selftest/cases/wrapper.py       | 11 ++++++
 4 files changed, 68 insertions(+)
 create mode 100644 meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
 create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
 create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py

diff --git a/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
new file mode 100644
index 0000000000..302eea8901
--- /dev/null
+++ b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
@@ -0,0 +1,21 @@
+FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
+SUMMARY = "Check that create_cmdline_shebang works"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI += "file://test.awk"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
+    sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
+    create_cmdline_shebang_wrapper ${D}${bindir}/test
+    if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
+        bbfatal "Wrapper is broken"
+    else
+        bbnote "Wrapper is good"
+    fi
+}
+
+BBCLASSEXTEND = "native"
diff --git a/meta-selftest/recipes-test/wrapper/files/test.awk b/meta-selftest/recipes-test/wrapper/files/test.awk
new file mode 100644
index 0000000000..91429197b1
--- /dev/null
+++ b/meta-selftest/recipes-test/wrapper/files/test.awk
@@ -0,0 +1,2 @@
+#! @AWK_BIN@ -f
+BEGIN { print "Don't Panic!" }
diff --git a/meta/classes/utils.bbclass b/meta/classes/utils.bbclass
index b4eb3d38ab..b617632d9f 100644
--- a/meta/classes/utils.bbclass
+++ b/meta/classes/utils.bbclass
@@ -184,6 +184,40 @@ END
 	chmod +x $cmd
 }
 
+create_cmdline_shebang_wrapper () {
+	# Create a wrapper script where commandline options are needed
+	#
+	# These are useful to work around shebang relocation issues, where shebangs are too
+  # long or have arguments in them, thus preventing them from using the /usr/bin/env
+	# shebang
+	#
+	# Usage: create_cmdline_wrapper FILENAME <extra-options>
+
+	cmd=$1
+	shift
+
+	echo "Generating wrapper script for $cmd"
+
+  # Strip #! and get remaining interpreter + arg
+  argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )")"
+  # strip the shebang from the real script as we do not want it to be usable anyway
+  tail -n +2 $cmd > $cmd.real
+	cmdname=$(basename $cmd)
+	dirname=$(dirname $cmd)
+	cmdoptions=$@
+	if [ "${base_prefix}" != "" ]; then
+		relpath=`python3 -c "import os; print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
+		cmdoptions=`echo $@ | sed -e "s:${base_prefix}:\\$realdir/$relpath:g"`
+	fi
+	cat <<END >$cmd
+#!/usr/bin/env bash
+realpath=\`readlink -fn \$0\`
+realdir=\`dirname \$realpath\`
+exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real $cmdoptions "\$@"
+END
+	chmod +x $cmd
+}
+
 create_wrapper () {
 	# Create a wrapper script where extra environment variables are needed
 	#
diff --git a/meta/lib/oeqa/selftest/cases/wrapper.py b/meta/lib/oeqa/selftest/cases/wrapper.py
new file mode 100644
index 0000000000..6de63310c0
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/wrapper.py
@@ -0,0 +1,11 @@
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake
+
+class WrapperTests(OESelftestTestCase):
+    def test_shebang_wrapper(self):
+        """
+        Summary:   Build a recipe which will fail if the cmdline_shebang_wrapper function is defective.
+        Expected:  Exit status to be 0.
+        Author:    Paulo Neves <ptsneves@gmail.com>
+        """
+        res = bitbake("cmdline-shebang-wrapper-test -c install", ignore_status=False)
-- 
2.25.1

[PATCH v3 7/7] libcheck: Fix too long shebang for native case.

[Paulo Neves]

It requires a shebang wrapper due to the fact that awk interpreter
has an argument.
---
 meta/recipes-support/libcheck/libcheck_0.15.2.bb | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/recipes-support/libcheck/libcheck_0.15.2.bb b/meta/recipes-support/libcheck/libcheck_0.15.2.bb
index 188d689cc3..1393aa2a1c 100644
--- a/meta/recipes-support/libcheck/libcheck_0.15.2.bb
+++ b/meta/recipes-support/libcheck/libcheck_0.15.2.bb
@@ -23,6 +23,9 @@ CACHED_CONFIGUREVARS += "ac_cv_path_AWK_PATH=${bindir}/gawk"
 
 RREPLACES:${PN} = "check (<= 0.9.5)"
 
+do_install:append:class-native() {
+    create_cmdline_shebang_wrapper ${D}${bindir}/checkmk
+}
 BBCLASSEXTEND = "native nativesdk"
 
 PACKAGES =+ "checkmk"
-- 
2.25.1

Re: [OE-core] [PATCH v3 1/7] python: Avoid shebang overflow on python-config.py

[Alexander Kanavin]

As requested, please submit upstream.

Alex

On Fri, 10 Jun 2022 at 23:43, Paulo Neves <ptsneves@gmail.com> wrote:
>
> The native path may be too big, leading to shebang
> overflow. Just use the #!/usr/bin/env python3.
>
> Signed-off-by: Paulo Neves <ptsneves@gmail.com>
> ---
>  ...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
>  .../recipes-devtools/python/python3_3.10.4.bb |  1 +
>  2 files changed, 30 insertions(+)
>  create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
>
> diff --git a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
> new file mode 100644
> index 0000000000..9f23278a30
> --- /dev/null
> +++ b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
> @@ -0,0 +1,29 @@
> +From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
> +From: Paulo Neves <ptsneves@gmail.com>
> +Date: Tue, 7 Jun 2022 16:16:41 +0200
> +Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
> +
> +The whole native path may be too big, leading to shebang
> +overflow. Let's just use the env shebang.
> +
> +Upstream-Status: Inappropriate [distribution]
> +---
> + Makefile.pre.in | 2 ++
> + 1 file changed, 2 insertions(+)
> +
> +diff --git a/Makefile.pre.in b/Makefile.pre.in
> +index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
> +--- a/Makefile.pre.in
> ++++ b/Makefile.pre.in
> +@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
> +       @ # Substitution happens here, as the completely-expanded BINDIR
> +       @ # is not available in configure
> +       sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
> ++      @ # Otherwise we might get huge shebangs with native paths
> ++      sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
> +       @ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
> +       LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
> +       @  # In OpenEmbedded, always use the python version of the script, the shell
> +--
> +2.25.1
> +
> diff --git a/meta/recipes-devtools/python/python3_3.10.4.bb b/meta/recipes-devtools/python/python3_3.10.4.bb
> index 6bd3a6aba8..357025f856 100644
> --- a/meta/recipes-devtools/python/python3_3.10.4.bb
> +++ b/meta/recipes-devtools/python/python3_3.10.4.bb
> @@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
>             file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
>             file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
>             file://deterministic_imports.patch \
> +           file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
>             "
>
>  SRC_URI:append:class-native = " \
> --
> 2.25.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#166813): https://lists.openembedded.org/g/openembedded-core/message/166813
> Mute This Topic: https://lists.openembedded.org/mt/91678611/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH v3 1/7] python: Avoid shebang overflow on python-config.py

[Paulo Neves]

As requested
https://github.com/python/cpython/pull/93760

Paulo Neves

On 6/13/22 09:30, Alexander Kanavin wrote:
> As requested, please submit upstream.
>
> Alex
>
> On Fri, 10 Jun 2022 at 23:43, Paulo Neves <ptsneves@gmail.com> wrote:
>> The native path may be too big, leading to shebang
>> overflow. Just use the #!/usr/bin/env python3.
>>
>> Signed-off-by: Paulo Neves <ptsneves@gmail.com>
>> ---
>>   ...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
>>   .../recipes-devtools/python/python3_3.10.4.bb |  1 +
>>   2 files changed, 30 insertions(+)
>>   create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
>>
>> diff --git a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
>> new file mode 100644
>> index 0000000000..9f23278a30
>> --- /dev/null
>> +++ b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
>> @@ -0,0 +1,29 @@
>> +From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
>> +From: Paulo Neves <ptsneves@gmail.com>
>> +Date: Tue, 7 Jun 2022 16:16:41 +0200
>> +Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
>> +
>> +The whole native path may be too big, leading to shebang
>> +overflow. Let's just use the env shebang.
>> +
>> +Upstream-Status: Inappropriate [distribution]
>> +---
>> + Makefile.pre.in | 2 ++
>> + 1 file changed, 2 insertions(+)
>> +
>> +diff --git a/Makefile.pre.in b/Makefile.pre.in
>> +index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
>> +--- a/Makefile.pre.in
>> ++++ b/Makefile.pre.in
>> +@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
>> +       @ # Substitution happens here, as the completely-expanded BINDIR
>> +       @ # is not available in configure
>> +       sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
>> ++      @ # Otherwise we might get huge shebangs with native paths
>> ++      sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
>> +       @ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
>> +       LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
>> +       @  # In OpenEmbedded, always use the python version of the script, the shell
>> +--
>> +2.25.1
>> +
>> diff --git a/meta/recipes-devtools/python/python3_3.10.4.bb b/meta/recipes-devtools/python/python3_3.10.4.bb
>> index 6bd3a6aba8..357025f856 100644
>> --- a/meta/recipes-devtools/python/python3_3.10.4.bb
>> +++ b/meta/recipes-devtools/python/python3_3.10.4.bb
>> @@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
>>              file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
>>              file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
>>              file://deterministic_imports.patch \
>> +           file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
>>              "
>>
>>   SRC_URI:append:class-native = " \
>> --
>> 2.25.1
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#166813): https://lists.openembedded.org/g/openembedded-core/message/166813
>> Mute This Topic: https://lists.openembedded.org/mt/91678611/1686489
>> Group Owner: openembedded-core+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>

Re: [OE-core] [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow

[Paulo Neves]

Hey Luca,

I sent a new patchset and from my tests it all should be good now.

Paulo Neves

On 6/8/22 17:06, Luca Ceresoli wrote:
> Hi Paulo,
>
> On Wed, 8 Jun 2022 14:53:05 +0200
> "Luca Ceresoli via lists.openembedded.org"
> <luca.ceresoli=bootlin.com@lists.openembedded.org> wrote:
>
>> Hi Paulo,
>>
>> On Tue,  7 Jun 2022 17:11:22 +0200
>> "Paulo Neves" <ptsneves@gmail.com> wrote:
>>
>>> Make sure we do not stage any executable with a bigger shebang
>>> than 128. Fixes [1]
>>>
>>> [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
>>>
>>> Signed-off-by: Paulo Neves <ptsneves@gmail.com>
>> This check seems to be working very well!! It triggered a huge amount
>> of build failures on the autobuilders due to libcheck having a shebang
>> too long in the checkmk script, e.g.:
>>
>> #! /home/pokybuild/yocto-worker/genericx86-64-alt/build/build/tmp/work/x86_64-linux/libcheck-native/0.15.2-r0/recipe-sysroot-native/usr/bin/gawk -f
>>
>> Here are a few logs:
>>
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/14/logs/errors
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/11/logs/errors
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/11/logs/errors
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/12/logs/errors
>>
>> It would be great if you could add another patch to your series to fix
>> libcheck, and also to do 'bitbake world' to test as many packages as
>> possible before discovering from the autobuilders.
> Here are more failures:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/12/logs/errors
> https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/12/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/62/builds/5340/steps/12/logs/errors
> https://autobuilder.yoctoproject.org/typhoon/#/builders/62/builds/5340/steps/12/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/13/logs/errors
> https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/13/logs/stdio
>
> This time the error is:
>
> stdio: ERROR: core-image-sato-1.0-r0 do_testsdk: The toolchain <...> is not built. Build it before running the tests: 'bitbake <image> -c populate_sdk' .
>
> I'm not sure exactly how your code triggers such error, but it appeared
> when testing on the autobuilders with this patch series and disappeared
> when I removed only these 5 patches, thus it seems related.
>

Re: [OE-core] [PATCH v3 1/7] python: Avoid shebang overflow on python-config.py

[Alexander Kanavin]

Thanks, can you resend with updated Upstream-Status that includes the link?

Alex

On Mon, 13 Jun 2022 at 12:32, Paulo Neves <ptsneves@gmail.com> wrote:
>
> As requested
> https://github.com/python/cpython/pull/93760
>
> Paulo Neves
>
> On 6/13/22 09:30, Alexander Kanavin wrote:
> > As requested, please submit upstream.
> >
> > Alex
> >
> > On Fri, 10 Jun 2022 at 23:43, Paulo Neves <ptsneves@gmail.com> wrote:
> >> The native path may be too big, leading to shebang
> >> overflow. Just use the #!/usr/bin/env python3.
> >>
> >> Signed-off-by: Paulo Neves <ptsneves@gmail.com>
> >> ---
> >>   ...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
> >>   .../recipes-devtools/python/python3_3.10.4.bb |  1 +
> >>   2 files changed, 30 insertions(+)
> >>   create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
> >>
> >> diff --git a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
> >> new file mode 100644
> >> index 0000000000..9f23278a30
> >> --- /dev/null
> >> +++ b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
> >> @@ -0,0 +1,29 @@
> >> +From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
> >> +From: Paulo Neves <ptsneves@gmail.com>
> >> +Date: Tue, 7 Jun 2022 16:16:41 +0200
> >> +Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
> >> +
> >> +The whole native path may be too big, leading to shebang
> >> +overflow. Let's just use the env shebang.
> >> +
> >> +Upstream-Status: Inappropriate [distribution]
> >> +---
> >> + Makefile.pre.in | 2 ++
> >> + 1 file changed, 2 insertions(+)
> >> +
> >> +diff --git a/Makefile.pre.in b/Makefile.pre.in
> >> +index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
> >> +--- a/Makefile.pre.in
> >> ++++ b/Makefile.pre.in
> >> +@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
> >> +       @ # Substitution happens here, as the completely-expanded BINDIR
> >> +       @ # is not available in configure
> >> +       sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
> >> ++      @ # Otherwise we might get huge shebangs with native paths
> >> ++      sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
> >> +       @ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
> >> +       LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
> >> +       @  # In OpenEmbedded, always use the python version of the script, the shell
> >> +--
> >> +2.25.1
> >> +
> >> diff --git a/meta/recipes-devtools/python/python3_3.10.4.bb b/meta/recipes-devtools/python/python3_3.10.4.bb
> >> index 6bd3a6aba8..357025f856 100644
> >> --- a/meta/recipes-devtools/python/python3_3.10.4.bb
> >> +++ b/meta/recipes-devtools/python/python3_3.10.4.bb
> >> @@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
> >>              file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
> >>              file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
> >>              file://deterministic_imports.patch \
> >> +           file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
> >>              "
> >>
> >>   SRC_URI:append:class-native = " \
> >> --
> >> 2.25.1
> >>
> >>
> >> -=-=-=-=-=-=-=-=-=-=-=-
> >> Links: You receive all messages sent to this group.
> >> View/Reply Online (#166813): https://lists.openembedded.org/g/openembedded-core/message/166813
> >> Mute This Topic: https://lists.openembedded.org/mt/91678611/1686489
> >> Group Owner: openembedded-core+owner@lists.openembedded.org
> >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> >> -=-=-=-=-=-=-=-=-=-=-=-
> >>
>

[PATCH v4 1/7] python: Avoid shebang overflow on python-config.py

[Paulo Neves]

The native path may be too big, leading to shebang
overflow. Just use the #!/usr/bin/env python3.

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 ...shebang-overflow-on-python-config.py.patch | 33 +++++++++++++++++++
 .../recipes-devtools/python/python3_3.10.4.bb |  1 +
 2 files changed, 34 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch

diff --git a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
new file mode 100644
index 0000000000..921da8de7c
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
@@ -0,0 +1,33 @@
+From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
+From: Paulo Neves <ptsneves@gmail.com>
+Date: Tue, 7 Jun 2022 16:16:41 +0200
+Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
+
+The whole native path may be too big, leading to shebang
+overflow. Let's just use the env shebang.
+
+Denial reason: [1]
+
+Upstream-Status: Denied [distribution]
+
+[1] https://github.com/python/cpython/pull/93760#pullrequestreview-1005365737
+---
+ Makefile.pre.in | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
+ 	@ # Substitution happens here, as the completely-expanded BINDIR
+ 	@ # is not available in configure
+ 	sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
++	@ # Otherwise we might get huge shebangs with native paths
++	sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
+ 	@ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
+ 	LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
+ 	@  # In OpenEmbedded, always use the python version of the script, the shell
+-- 
+2.25.1
+
diff --git a/meta/recipes-devtools/python/python3_3.10.4.bb b/meta/recipes-devtools/python/python3_3.10.4.bb
index 6bd3a6aba8..357025f856 100644
--- a/meta/recipes-devtools/python/python3_3.10.4.bb
+++ b/meta/recipes-devtools/python/python3_3.10.4.bb
@@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
            file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
            file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
            file://deterministic_imports.patch \
+           file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
            "
 
 SRC_URI:append:class-native = " \
-- 
2.25.1

[PATCH v4 2/7] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2

[Paulo Neves]

A native path can cause a shebang overflow on gtkdoc-mkhtml.
Replace it with /usr/bin/env.

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb b/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
index 150d2c0b23..392913fcc6 100644
--- a/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
+++ b/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
@@ -43,6 +43,7 @@ do_install:append () {
         ${datadir}/gtk-doc/python/gtkdoc/config.py; do
         sed -e 's,${RECIPE_SYSROOT_NATIVE}/usr/bin/pkg-config,${bindir}/pkg-config,' \
             -e 's,${HOSTTOOLS_DIR}/python3,${bindir}/python3,' \
+            -e '1s|^#!.*|#!/usr/bin/env python3|' \
             -i ${D}$fn
     done
 }
-- 
2.25.1

[PATCH v4 3/7] insane.bbclass: Make do_qa_staging check shebangs

[Paulo Neves]

As reported in the bug report [1], there was no check for shebang
sizes on native scripts and now this is fixed.

The path scope of the qa_staging was increased from just checking
libdir to all the relevant SYSROOT_DIRS.

It is possible to skip this check through INSANE_SKIP.

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 meta/classes/insane.bbclass | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 9ca84bace9..b2951a48fe 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -630,6 +630,11 @@ def qa_check_staged(path,d):
         bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
         skip_pkgconfig = True
 
+    skip_shebang_size = False
+    if 'shebang-size' in skip:
+        bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
+        skip_shebang_size = True
+
     # find all .la and .pc files
     # read the content
     # and check for stuff that looks wrong
@@ -651,6 +656,13 @@ def qa_check_staged(path,d):
                         error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
                         oe.qa.handle_error("pkgconfig", error_msg, d)
 
+            if not skip_shebang_size:
+                errors = {}
+                package_qa_check_shebang_size(path, "", d, None, errors)
+                for e in errors:
+                    oe.qa.handle_error(e, errors[e], d)
+
+
 # Run all package-wide warnfuncs and errorfuncs
 def package_qa_package(warnfuncs, errorfuncs, package, d):
     warnings = {}
@@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
 
 python do_qa_staging() {
     bb.note("QA checking staging")
-    qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
+    sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
+    for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
+        qa_check_staged(sysroot_destdir + sysroot_dir, d)
     oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
 }
 
-- 
2.25.1

[PATCH v4 4/7] oeqa/selftest: Add test for shebang overflow

[Paulo Neves]

Make sure we do not stage any executable with a bigger shebang
than 128. Fixes [1]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 .../sysroot-test/sysroot-shebang-test_1.0.bb         | 12 ++++++++++++
 meta/lib/oeqa/selftest/cases/sysroot.py              | 10 ++++++++++
 2 files changed, 22 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb

diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
new file mode 100644
index 0000000000..6c834be897
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Check that shebang does not exceed 128 characters"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    echo '#!BiM3cnVd1Amtv6PG+FynrQiVMbZnX5ELgF21q3EkuB+44JEGWtq8TvBJ7EGidfVs3eR3wVOUbLnjYDlKUWcm7YC/ute7f+KDHbwxziRUSUBZAUqgjiQdfQ0HnxajI0ozbM863E9JV9k13yZKYfh9/zR77Y6Dl4Dd3zOWS75LSpkAXV' > ${D}${bindir}/max-shebang
+    chmod 755 ${D}${bindir}/max-shebang
+}
+
+BBCLASSEXTEND = "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index 315d1a61c2..9457f1e3ac 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -35,3 +35,13 @@ TESTSTRING:pn-sysroot-test-arch1 = "%s"
 TESTSTRING:pn-sysroot-test-arch2 = "%s"
 """ % (uuid1, uuid2))
         bitbake("sysroot-test")
+
+    def test_sysroot_max_shebang(self):
+        """
+        Summary:   Check max shebang triggers. To confirm [YOCTO #11053] is closed.
+        Expected:  Fail when a shebang bigger than the max shebang-size is reached.
+        Author:    Paulo Neves <ptsneves@gmail.com>
+        """
+        expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
+        res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
-- 
2.25.1

[PATCH v4 5/7] oeqa/selftest: Test staged .la and .pc files

[Paulo Neves]

These files are checked by qa_check_staged but there was no
test cases for whether the tests actually worked. Now there
are.

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 .../sysroot-test/sysroot-la-test_1.0.bb       | 16 ++++++++++
 .../sysroot-test/sysroot-pc-test_1.0.bb       | 12 +++++++
 meta/lib/oeqa/selftest/cases/sysroot.py       | 32 +++++++++++++++++++
 3 files changed, 60 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb

diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
new file mode 100644
index 0000000000..21f06782fb
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
@@ -0,0 +1,16 @@
+SUMMARY = "Produce a broken la file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+# remove-libtool.bbclass is inherited by default and removes all
+# .la files which for this test we specifically do not want.
+REMOVE_LIBTOOL_LA = "0"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/la-test.la
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
new file mode 100644
index 0000000000..e748310fc4
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Produce a broken pc file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/test.pc
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index 9457f1e3ac..588fc8c713 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -45,3 +45,35 @@ TESTSTRING:pn-sysroot-test-arch2 = "%s"
         expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
         res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
         self.assertTrue(expected in res.output, msg=res.output)
+
+    def test_sysroot_la(self):
+        """
+        Summary:   Check that workdir paths are not contained in .la files.
+        Expected:  Fail when a workdir path is found in the file content.
+        Author:    Paulo Neves <ptsneves@gmail.com>
+        """
+        expected = "la-test.la failed sanity test (workdir) in path"
+
+        res = bitbake("sysroot-la-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+
+        res = bitbake("sysroot-la-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+
+    def test_sysroot_pkgconfig(self):
+        """
+        Summary:   Check that tmpdir paths are not contained in .pc files.
+        Expected:  Fail when a tmpdir path is found in the file content.
+        Author:    Paulo Neves <ptsneves@gmail.com>
+        """
+        expected = "test.pc failed sanity test (tmpdir) in path"
+
+        res = bitbake("sysroot-pc-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
+        self.assertTrue(expected in res.output, msg=res.output)
+
+        res = bitbake("sysroot-pc-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
-- 
2.25.1

[PATCH v4 6/7] utils: Add cmdline_shebang_wrapper util.

[Paulo Neves]

Useful to work around shebang relocation issues, where
shebangs are too long or have arguments in them, thus preventing them
from using the /usr/bin/env shebang.
---
 .../wrapper/cmdline-shebang-wrapper-test.bb   | 21 ++++++++++++
 .../recipes-test/wrapper/files/test.awk       |  2 ++
 meta/classes/utils.bbclass                    | 34 +++++++++++++++++++
 meta/lib/oeqa/selftest/cases/wrapper.py       | 11 ++++++
 4 files changed, 68 insertions(+)
 create mode 100644 meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
 create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
 create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py

diff --git a/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
new file mode 100644
index 0000000000..302eea8901
--- /dev/null
+++ b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
@@ -0,0 +1,21 @@
+FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
+SUMMARY = "Check that create_cmdline_shebang works"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI += "file://test.awk"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
+    sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
+    create_cmdline_shebang_wrapper ${D}${bindir}/test
+    if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
+        bbfatal "Wrapper is broken"
+    else
+        bbnote "Wrapper is good"
+    fi
+}
+
+BBCLASSEXTEND = "native"
diff --git a/meta-selftest/recipes-test/wrapper/files/test.awk b/meta-selftest/recipes-test/wrapper/files/test.awk
new file mode 100644
index 0000000000..91429197b1
--- /dev/null
+++ b/meta-selftest/recipes-test/wrapper/files/test.awk
@@ -0,0 +1,2 @@
+#! @AWK_BIN@ -f
+BEGIN { print "Don't Panic!" }
diff --git a/meta/classes/utils.bbclass b/meta/classes/utils.bbclass
index b4eb3d38ab..b617632d9f 100644
--- a/meta/classes/utils.bbclass
+++ b/meta/classes/utils.bbclass
@@ -184,6 +184,40 @@ END
 	chmod +x $cmd
 }
 
+create_cmdline_shebang_wrapper () {
+	# Create a wrapper script where commandline options are needed
+	#
+	# These are useful to work around shebang relocation issues, where shebangs are too
+  # long or have arguments in them, thus preventing them from using the /usr/bin/env
+	# shebang
+	#
+	# Usage: create_cmdline_wrapper FILENAME <extra-options>
+
+	cmd=$1
+	shift
+
+	echo "Generating wrapper script for $cmd"
+
+  # Strip #! and get remaining interpreter + arg
+  argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )")"
+  # strip the shebang from the real script as we do not want it to be usable anyway
+  tail -n +2 $cmd > $cmd.real
+	cmdname=$(basename $cmd)
+	dirname=$(dirname $cmd)
+	cmdoptions=$@
+	if [ "${base_prefix}" != "" ]; then
+		relpath=`python3 -c "import os; print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
+		cmdoptions=`echo $@ | sed -e "s:${base_prefix}:\\$realdir/$relpath:g"`
+	fi
+	cat <<END >$cmd
+#!/usr/bin/env bash
+realpath=\`readlink -fn \$0\`
+realdir=\`dirname \$realpath\`
+exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real $cmdoptions "\$@"
+END
+	chmod +x $cmd
+}
+
 create_wrapper () {
 	# Create a wrapper script where extra environment variables are needed
 	#
diff --git a/meta/lib/oeqa/selftest/cases/wrapper.py b/meta/lib/oeqa/selftest/cases/wrapper.py
new file mode 100644
index 0000000000..6de63310c0
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/wrapper.py
@@ -0,0 +1,11 @@
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake
+
+class WrapperTests(OESelftestTestCase):
+    def test_shebang_wrapper(self):
+        """
+        Summary:   Build a recipe which will fail if the cmdline_shebang_wrapper function is defective.
+        Expected:  Exit status to be 0.
+        Author:    Paulo Neves <ptsneves@gmail.com>
+        """
+        res = bitbake("cmdline-shebang-wrapper-test -c install", ignore_status=False)
-- 
2.25.1

[PATCH v4 7/7] libcheck: Fix too long shebang for native case.

[Paulo Neves]

It requires a shebang wrapper due to the fact that awk interpreter
has an argument.
---
 meta/recipes-support/libcheck/libcheck_0.15.2.bb | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/recipes-support/libcheck/libcheck_0.15.2.bb b/meta/recipes-support/libcheck/libcheck_0.15.2.bb
index 188d689cc3..1393aa2a1c 100644
--- a/meta/recipes-support/libcheck/libcheck_0.15.2.bb
+++ b/meta/recipes-support/libcheck/libcheck_0.15.2.bb
@@ -23,6 +23,9 @@ CACHED_CONFIGUREVARS += "ac_cv_path_AWK_PATH=${bindir}/gawk"
 
 RREPLACES:${PN} = "check (<= 0.9.5)"
 
+do_install:append:class-native() {
+    create_cmdline_shebang_wrapper ${D}${bindir}/checkmk
+}
 BBCLASSEXTEND = "native nativesdk"
 
 PACKAGES =+ "checkmk"
-- 
2.25.1

Re: [OE-core] [PATCH v4 6/7] utils: Add cmdline_shebang_wrapper util.

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 5050 bytes --]

FILESEXTRAPATHS:prepend doesn't seem to be needed and LICENSE is really
CLOSED?

On Tue, Jun 14, 2022 at 3:17 PM Paulo Neves <ptsneves@gmail.com> wrote:

> Useful to work around shebang relocation issues, where
> shebangs are too long or have arguments in them, thus preventing them
> from using the /usr/bin/env shebang.
> ---
>  .../wrapper/cmdline-shebang-wrapper-test.bb   | 21 ++++++++++++
>  .../recipes-test/wrapper/files/test.awk       |  2 ++
>  meta/classes/utils.bbclass                    | 34 +++++++++++++++++++
>  meta/lib/oeqa/selftest/cases/wrapper.py       | 11 ++++++
>  4 files changed, 68 insertions(+)
>  create mode 100644 meta-selftest/recipes-test/wrapper/
> cmdline-shebang-wrapper-test.bb
>  create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
>  create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py
>
> diff --git a/meta-selftest/recipes-test/wrapper/
> cmdline-shebang-wrapper-test.bb b/meta-selftest/recipes-test/wrapper/
> cmdline-shebang-wrapper-test.bb
> new file mode 100644
> index 0000000000..302eea8901
> --- /dev/null
> +++ b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
> @@ -0,0 +1,21 @@
> +FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
> +SUMMARY = "Check that create_cmdline_shebang works"
> +LICENSE = "CLOSED"
> +INHIBIT_DEFAULT_DEPS = "1"
> +
> +SRC_URI += "file://test.awk"
> +
> +EXCLUDE_FROM_WORLD = "1"
> +do_install() {
> +    install -d ${D}${bindir}
> +    install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
> +    sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
> +    create_cmdline_shebang_wrapper ${D}${bindir}/test
> +    if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
> +        bbfatal "Wrapper is broken"
> +    else
> +        bbnote "Wrapper is good"
> +    fi
> +}
> +
> +BBCLASSEXTEND = "native"
> diff --git a/meta-selftest/recipes-test/wrapper/files/test.awk
> b/meta-selftest/recipes-test/wrapper/files/test.awk
> new file mode 100644
> index 0000000000..91429197b1
> --- /dev/null
> +++ b/meta-selftest/recipes-test/wrapper/files/test.awk
> @@ -0,0 +1,2 @@
> +#! @AWK_BIN@ -f
> +BEGIN { print "Don't Panic!" }
> diff --git a/meta/classes/utils.bbclass b/meta/classes/utils.bbclass
> index b4eb3d38ab..b617632d9f 100644
> --- a/meta/classes/utils.bbclass
> +++ b/meta/classes/utils.bbclass
> @@ -184,6 +184,40 @@ END
>         chmod +x $cmd
>  }
>
> +create_cmdline_shebang_wrapper () {
> +       # Create a wrapper script where commandline options are needed
> +       #
> +       # These are useful to work around shebang relocation issues, where
> shebangs are too
> +  # long or have arguments in them, thus preventing them from using the
> /usr/bin/env
> +       # shebang
> +       #
> +       # Usage: create_cmdline_wrapper FILENAME <extra-options>
> +
> +       cmd=$1
> +       shift
> +
> +       echo "Generating wrapper script for $cmd"
> +
> +  # Strip #! and get remaining interpreter + arg
> +  argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )")"
> +  # strip the shebang from the real script as we do not want it to be
> usable anyway
> +  tail -n +2 $cmd > $cmd.real
> +       cmdname=$(basename $cmd)
> +       dirname=$(dirname $cmd)
> +       cmdoptions=$@
> +       if [ "${base_prefix}" != "" ]; then
> +               relpath=`python3 -c "import os;
> print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
> +               cmdoptions=`echo $@ | sed -e
> "s:${base_prefix}:\\$realdir/$relpath:g"`
> +       fi
> +       cat <<END >$cmd
> +#!/usr/bin/env bash
> +realpath=\`readlink -fn \$0\`
> +realdir=\`dirname \$realpath\`
> +exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real $cmdoptions
> "\$@"
> +END
> +       chmod +x $cmd
> +}
> +
>  create_wrapper () {
>         # Create a wrapper script where extra environment variables are
> needed
>         #
> diff --git a/meta/lib/oeqa/selftest/cases/wrapper.py
> b/meta/lib/oeqa/selftest/cases/wrapper.py
> new file mode 100644
> index 0000000000..6de63310c0
> --- /dev/null
> +++ b/meta/lib/oeqa/selftest/cases/wrapper.py
> @@ -0,0 +1,11 @@
> +from oeqa.selftest.case import OESelftestTestCase
> +from oeqa.utils.commands import bitbake
> +
> +class WrapperTests(OESelftestTestCase):
> +    def test_shebang_wrapper(self):
> +        """
> +        Summary:   Build a recipe which will fail if the
> cmdline_shebang_wrapper function is defective.
> +        Expected:  Exit status to be 0.
> +        Author:    Paulo Neves <ptsneves@gmail.com>
> +        """
> +        res = bitbake("cmdline-shebang-wrapper-test -c install",
> ignore_status=False)
> --
> 2.25.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#166887):
> https://lists.openembedded.org/g/openembedded-core/message/166887
> Mute This Topic: https://lists.openembedded.org/mt/91748692/3617156
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> Martin.Jansa@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 7092 bytes --]

Re: [OE-core] [PATCH v4 6/7] utils: Add cmdline_shebang_wrapper util.

[Paulo Neves]

[-- Attachment #1: Type: text/plain, Size: 6066 bytes --]

Regarding the FILESEXTRAPATHS:prepend  i was convinced that 
${THISDIR}/files was not automatically added and was needed for the 
test.awk file. I guess I have been cargo culting for some time on this. 
Will remove it.

This is a recipe for a test and if i add a license different than CLOSED 
than i need to add a license file, which seems a bit overkill for a one 
line. Am I wrong? What license should it even be? Dont Panic is a common 
string used in awk docs.

Paulo Neves

On 6/14/22 15:39, Martin Jansa wrote:
> FILESEXTRAPATHS:prepend doesn't seem to be needed and LICENSE is 
> really CLOSED?
>
> On Tue, Jun 14, 2022 at 3:17 PM Paulo Neves <ptsneves@gmail.com> wrote:
>
>     Useful to work around shebang relocation issues, where
>     shebangs are too long or have arguments in them, thus preventing them
>     from using the /usr/bin/env shebang.
>     ---
>      .../wrapper/cmdline-shebang-wrapper-test.bb
>     <http://cmdline-shebang-wrapper-test.bb>  | 21 ++++++++++++
>      .../recipes-test/wrapper/files/test.awk       |  2 ++
>      meta/classes/utils.bbclass                    | 34
>     +++++++++++++++++++
>      meta/lib/oeqa/selftest/cases/wrapper.py       | 11 ++++++
>      4 files changed, 68 insertions(+)
>      create mode 100644
>     meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
>     <http://cmdline-shebang-wrapper-test.bb>
>      create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
>      create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py
>
>     diff --git
>     a/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
>     <http://cmdline-shebang-wrapper-test.bb>
>     b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
>     <http://cmdline-shebang-wrapper-test.bb>
>     new file mode 100644
>     index 0000000000..302eea8901
>     --- /dev/null
>     +++
>     b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
>     <http://cmdline-shebang-wrapper-test.bb>
>     @@ -0,0 +1,21 @@
>     +FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
>     +SUMMARY = "Check that create_cmdline_shebang works"
>     +LICENSE = "CLOSED"
>     +INHIBIT_DEFAULT_DEPS = "1"
>     +
>     +SRC_URI += "file://test.awk"
>     +
>     +EXCLUDE_FROM_WORLD = "1"
>     +do_install() {
>     +    install -d ${D}${bindir}
>     +    install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
>     +    sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
>     +    create_cmdline_shebang_wrapper ${D}${bindir}/test
>     +    if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
>     +        bbfatal "Wrapper is broken"
>     +    else
>     +        bbnote "Wrapper is good"
>     +    fi
>     +}
>     +
>     +BBCLASSEXTEND = "native"
>     diff --git a/meta-selftest/recipes-test/wrapper/files/test.awk
>     b/meta-selftest/recipes-test/wrapper/files/test.awk
>     new file mode 100644
>     index 0000000000..91429197b1
>     --- /dev/null
>     +++ b/meta-selftest/recipes-test/wrapper/files/test.awk
>     @@ -0,0 +1,2 @@
>     +#! @AWK_BIN@ -f
>     +BEGIN { print "Don't Panic!" }
>     diff --git a/meta/classes/utils.bbclass b/meta/classes/utils.bbclass
>     index b4eb3d38ab..b617632d9f 100644
>     --- a/meta/classes/utils.bbclass
>     +++ b/meta/classes/utils.bbclass
>     @@ -184,6 +184,40 @@ END
>             chmod +x $cmd
>      }
>
>     +create_cmdline_shebang_wrapper () {
>     +       # Create a wrapper script where commandline options are needed
>     +       #
>     +       # These are useful to work around shebang relocation
>     issues, where shebangs are too
>     +  # long or have arguments in them, thus preventing them from
>     using the /usr/bin/env
>     +       # shebang
>     +       #
>     +       # Usage: create_cmdline_wrapper FILENAME <extra-options>
>     +
>     +       cmd=$1
>     +       shift
>     +
>     +       echo "Generating wrapper script for $cmd"
>     +
>     +  # Strip #! and get remaining interpreter + arg
>     +  argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )")"
>     +  # strip the shebang from the real script as we do not want it
>     to be usable anyway
>     +  tail -n +2 $cmd > $cmd.real
>     +       cmdname=$(basename $cmd)
>     +       dirname=$(dirname $cmd)
>     +       cmdoptions=$@
>     +       if [ "${base_prefix}" != "" ]; then
>     +               relpath=`python3 -c "import os;
>     print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
>     +               cmdoptions=`echo $@ | sed -e
>     "s:${base_prefix}:\\$realdir/$relpath:g"`
>     +       fi
>     +       cat <<END >$cmd
>     +#!/usr/bin/env bash
>     +realpath=\`readlink -fn \$0\`
>     +realdir=\`dirname \$realpath\`
>     +exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real
>     $cmdoptions "\$@"
>     +END
>     +       chmod +x $cmd
>     +}
>     +
>      create_wrapper () {
>             # Create a wrapper script where extra environment
>     variables are needed
>             #
>     diff --git a/meta/lib/oeqa/selftest/cases/wrapper.py
>     b/meta/lib/oeqa/selftest/cases/wrapper.py
>     new file mode 100644
>     index 0000000000..6de63310c0
>     --- /dev/null
>     +++ b/meta/lib/oeqa/selftest/cases/wrapper.py
>     @@ -0,0 +1,11 @@
>     +from oeqa.selftest.case import OESelftestTestCase
>     +from oeqa.utils.commands import bitbake
>     +
>     +class WrapperTests(OESelftestTestCase):
>     +    def test_shebang_wrapper(self):
>     +        """
>     +        Summary:   Build a recipe which will fail if the
>     cmdline_shebang_wrapper function is defective.
>     +        Expected:  Exit status to be 0.
>     +        Author:    Paulo Neves <ptsneves@gmail.com>
>     +        """
>     +        res = bitbake("cmdline-shebang-wrapper-test -c install",
>     ignore_status=False)
>     -- 
>     2.25.1
>
>
>     
>

[-- Attachment #2: Type: text/html, Size: 8661 bytes --]

Re: [OE-core] [PATCH v4 6/7] utils: Add cmdline_shebang_wrapper util.

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 6292 bytes --]

You can use bitbake-getvar to see the default FILESPATH, but "files" next
to recipe is included by default (I prefer to use BPN directory for
slightly faster lookup - unless the files are shared between different
recipes). FILESEXTRAPATHS is usually only needed from bbappends which add
new files to SRC_URI.

For LICENSE you can use MIT and
LIC_FILES_CHKSUM =
"file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
like many "pure-metadata" recipes in oe-core.

On Tue, Jun 14, 2022 at 3:46 PM Paulo Neves <ptsneves@gmail.com> wrote:

> Regarding the FILESEXTRAPATHS:prepend  i was convinced that
> ${THISDIR}/files was not automatically added and was needed for the
> test.awk file. I guess I have been cargo culting for some time on this.
> Will remove it.
>
> This is a recipe for a test and if i add a license different than CLOSED
> than i need to add a license file, which seems a bit overkill for a one
> line. Am I wrong? What license should it even be? Dont Panic is a common
> string used in awk docs.
>
> Paulo Neves
>
> On 6/14/22 15:39, Martin Jansa wrote:
>
> FILESEXTRAPATHS:prepend doesn't seem to be needed and LICENSE is really
> CLOSED?
>
> On Tue, Jun 14, 2022 at 3:17 PM Paulo Neves <ptsneves@gmail.com> wrote:
>
>> Useful to work around shebang relocation issues, where
>> shebangs are too long or have arguments in them, thus preventing them
>> from using the /usr/bin/env shebang.
>> ---
>>  .../wrapper/cmdline-shebang-wrapper-test.bb   | 21 ++++++++++++
>>  .../recipes-test/wrapper/files/test.awk       |  2 ++
>>  meta/classes/utils.bbclass                    | 34 +++++++++++++++++++
>>  meta/lib/oeqa/selftest/cases/wrapper.py       | 11 ++++++
>>  4 files changed, 68 insertions(+)
>>  create mode 100644 meta-selftest/recipes-test/wrapper/
>> cmdline-shebang-wrapper-test.bb
>>  create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
>>  create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py
>>
>> diff --git a/meta-selftest/recipes-test/wrapper/
>> cmdline-shebang-wrapper-test.bb b/meta-selftest/recipes-test/wrapper/
>> cmdline-shebang-wrapper-test.bb
>> new file mode 100644
>> index 0000000000..302eea8901
>> --- /dev/null
>> +++ b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
>> @@ -0,0 +1,21 @@
>> +FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
>> +SUMMARY = "Check that create_cmdline_shebang works"
>> +LICENSE = "CLOSED"
>> +INHIBIT_DEFAULT_DEPS = "1"
>> +
>> +SRC_URI += "file://test.awk"
>> +
>> +EXCLUDE_FROM_WORLD = "1"
>> +do_install() {
>> +    install -d ${D}${bindir}
>> +    install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
>> +    sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
>> +    create_cmdline_shebang_wrapper ${D}${bindir}/test
>> +    if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
>> +        bbfatal "Wrapper is broken"
>> +    else
>> +        bbnote "Wrapper is good"
>> +    fi
>> +}
>> +
>> +BBCLASSEXTEND = "native"
>> diff --git a/meta-selftest/recipes-test/wrapper/files/test.awk
>> b/meta-selftest/recipes-test/wrapper/files/test.awk
>> new file mode 100644
>> index 0000000000..91429197b1
>> --- /dev/null
>> +++ b/meta-selftest/recipes-test/wrapper/files/test.awk
>> @@ -0,0 +1,2 @@
>> +#! @AWK_BIN@ -f
>> +BEGIN { print "Don't Panic!" }
>> diff --git a/meta/classes/utils.bbclass b/meta/classes/utils.bbclass
>> index b4eb3d38ab..b617632d9f 100644
>> --- a/meta/classes/utils.bbclass
>> +++ b/meta/classes/utils.bbclass
>> @@ -184,6 +184,40 @@ END
>>         chmod +x $cmd
>>  }
>>
>> +create_cmdline_shebang_wrapper () {
>> +       # Create a wrapper script where commandline options are needed
>> +       #
>> +       # These are useful to work around shebang relocation issues,
>> where shebangs are too
>> +  # long or have arguments in them, thus preventing them from using the
>> /usr/bin/env
>> +       # shebang
>> +       #
>> +       # Usage: create_cmdline_wrapper FILENAME <extra-options>
>> +
>> +       cmd=$1
>> +       shift
>> +
>> +       echo "Generating wrapper script for $cmd"
>> +
>> +  # Strip #! and get remaining interpreter + arg
>> +  argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )")"
>> +  # strip the shebang from the real script as we do not want it to be
>> usable anyway
>> +  tail -n +2 $cmd > $cmd.real
>> +       cmdname=$(basename $cmd)
>> +       dirname=$(dirname $cmd)
>> +       cmdoptions=$@
>> +       if [ "${base_prefix}" != "" ]; then
>> +               relpath=`python3 -c "import os;
>> print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
>> +               cmdoptions=`echo $@ | sed -e
>> "s:${base_prefix}:\\$realdir/$relpath:g"`
>> +       fi
>> +       cat <<END >$cmd
>> +#!/usr/bin/env bash
>> +realpath=\`readlink -fn \$0\`
>> +realdir=\`dirname \$realpath\`
>> +exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real $cmdoptions
>> "\$@"
>> +END
>> +       chmod +x $cmd
>> +}
>> +
>>  create_wrapper () {
>>         # Create a wrapper script where extra environment variables are
>> needed
>>         #
>> diff --git a/meta/lib/oeqa/selftest/cases/wrapper.py
>> b/meta/lib/oeqa/selftest/cases/wrapper.py
>> new file mode 100644
>> index 0000000000..6de63310c0
>> --- /dev/null
>> +++ b/meta/lib/oeqa/selftest/cases/wrapper.py
>> @@ -0,0 +1,11 @@
>> +from oeqa.selftest.case import OESelftestTestCase
>> +from oeqa.utils.commands import bitbake
>> +
>> +class WrapperTests(OESelftestTestCase):
>> +    def test_shebang_wrapper(self):
>> +        """
>> +        Summary:   Build a recipe which will fail if the
>> cmdline_shebang_wrapper function is defective.
>> +        Expected:  Exit status to be 0.
>> +        Author:    Paulo Neves <ptsneves@gmail.com>
>> +        """
>> +        res = bitbake("cmdline-shebang-wrapper-test -c install",
>> ignore_status=False)
>> --
>> 2.25.1
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#166887):
>> https://lists.openembedded.org/g/openembedded-core/message/166887
>> Mute This Topic: https://lists.openembedded.org/mt/91748692/3617156
>> Group Owner: openembedded-core+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
>> Martin.Jansa@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 10324 bytes --]

[PATCH v5 1/7] python: Avoid shebang overflow on python-config.py

[Paulo Neves]

The native path may be too big, leading to shebang
overflow. Just use the #!/usr/bin/env python3.

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 ...shebang-overflow-on-python-config.py.patch | 33 +++++++++++++++++++
 .../recipes-devtools/python/python3_3.10.4.bb |  1 +
 2 files changed, 34 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch

diff --git a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
new file mode 100644
index 0000000000..921da8de7c
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
@@ -0,0 +1,33 @@
+From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
+From: Paulo Neves <ptsneves@gmail.com>
+Date: Tue, 7 Jun 2022 16:16:41 +0200
+Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
+
+The whole native path may be too big, leading to shebang
+overflow. Let's just use the env shebang.
+
+Denial reason: [1]
+
+Upstream-Status: Denied [distribution]
+
+[1] https://github.com/python/cpython/pull/93760#pullrequestreview-1005365737
+---
+ Makefile.pre.in | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
+ 	@ # Substitution happens here, as the completely-expanded BINDIR
+ 	@ # is not available in configure
+ 	sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
++	@ # Otherwise we might get huge shebangs with native paths
++	sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
+ 	@ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
+ 	LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
+ 	@  # In OpenEmbedded, always use the python version of the script, the shell
+-- 
+2.25.1
+
diff --git a/meta/recipes-devtools/python/python3_3.10.4.bb b/meta/recipes-devtools/python/python3_3.10.4.bb
index 6bd3a6aba8..357025f856 100644
--- a/meta/recipes-devtools/python/python3_3.10.4.bb
+++ b/meta/recipes-devtools/python/python3_3.10.4.bb
@@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
            file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
            file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
            file://deterministic_imports.patch \
+           file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
            "
 
 SRC_URI:append:class-native = " \
-- 
2.25.1

[PATCH v5 2/7] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2

[Paulo Neves]

A native path can cause a shebang overflow on gtkdoc-mkhtml.
Replace it with /usr/bin/env.

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb b/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
index 150d2c0b23..392913fcc6 100644
--- a/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
+++ b/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
@@ -43,6 +43,7 @@ do_install:append () {
         ${datadir}/gtk-doc/python/gtkdoc/config.py; do
         sed -e 's,${RECIPE_SYSROOT_NATIVE}/usr/bin/pkg-config,${bindir}/pkg-config,' \
             -e 's,${HOSTTOOLS_DIR}/python3,${bindir}/python3,' \
+            -e '1s|^#!.*|#!/usr/bin/env python3|' \
             -i ${D}$fn
     done
 }
-- 
2.25.1

[PATCH v5 3/7] insane.bbclass: Make do_qa_staging check shebangs

[Paulo Neves]

As reported in the bug report [1], there was no check for shebang
sizes on native scripts and now this is fixed.

The path scope of the qa_staging was increased from just checking
libdir to all the relevant SYSROOT_DIRS.

It is possible to skip this check through INSANE_SKIP.

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 meta/classes/insane.bbclass | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 9ca84bace9..b2951a48fe 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -630,6 +630,11 @@ def qa_check_staged(path,d):
         bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
         skip_pkgconfig = True
 
+    skip_shebang_size = False
+    if 'shebang-size' in skip:
+        bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
+        skip_shebang_size = True
+
     # find all .la and .pc files
     # read the content
     # and check for stuff that looks wrong
@@ -651,6 +656,13 @@ def qa_check_staged(path,d):
                         error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
                         oe.qa.handle_error("pkgconfig", error_msg, d)
 
+            if not skip_shebang_size:
+                errors = {}
+                package_qa_check_shebang_size(path, "", d, None, errors)
+                for e in errors:
+                    oe.qa.handle_error(e, errors[e], d)
+
+
 # Run all package-wide warnfuncs and errorfuncs
 def package_qa_package(warnfuncs, errorfuncs, package, d):
     warnings = {}
@@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
 
 python do_qa_staging() {
     bb.note("QA checking staging")
-    qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
+    sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
+    for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
+        qa_check_staged(sysroot_destdir + sysroot_dir, d)
     oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
 }
 
-- 
2.25.1

[PATCH v5 4/7] oeqa/selftest: Add test for shebang overflow

[Paulo Neves]

Make sure we do not stage any executable with a bigger shebang
than 128. Fixes [1]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 .../sysroot-test/sysroot-shebang-test_1.0.bb         | 12 ++++++++++++
 meta/lib/oeqa/selftest/cases/sysroot.py              | 10 ++++++++++
 2 files changed, 22 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb

diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
new file mode 100644
index 0000000000..6c834be897
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Check that shebang does not exceed 128 characters"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    echo '#!BiM3cnVd1Amtv6PG+FynrQiVMbZnX5ELgF21q3EkuB+44JEGWtq8TvBJ7EGidfVs3eR3wVOUbLnjYDlKUWcm7YC/ute7f+KDHbwxziRUSUBZAUqgjiQdfQ0HnxajI0ozbM863E9JV9k13yZKYfh9/zR77Y6Dl4Dd3zOWS75LSpkAXV' > ${D}${bindir}/max-shebang
+    chmod 755 ${D}${bindir}/max-shebang
+}
+
+BBCLASSEXTEND = "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index 315d1a61c2..9457f1e3ac 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -35,3 +35,13 @@ TESTSTRING:pn-sysroot-test-arch1 = "%s"
 TESTSTRING:pn-sysroot-test-arch2 = "%s"
 """ % (uuid1, uuid2))
         bitbake("sysroot-test")
+
+    def test_sysroot_max_shebang(self):
+        """
+        Summary:   Check max shebang triggers. To confirm [YOCTO #11053] is closed.
+        Expected:  Fail when a shebang bigger than the max shebang-size is reached.
+        Author:    Paulo Neves <ptsneves@gmail.com>
+        """
+        expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
+        res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
-- 
2.25.1

[PATCH v5 5/7] oeqa/selftest: Test staged .la and .pc files

[Paulo Neves]

These files are checked by qa_check_staged but there was no
test cases for whether the tests actually worked. Now there
are.

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 .../sysroot-test/sysroot-la-test_1.0.bb       | 16 ++++++++++
 .../sysroot-test/sysroot-pc-test_1.0.bb       | 12 +++++++
 meta/lib/oeqa/selftest/cases/sysroot.py       | 32 +++++++++++++++++++
 3 files changed, 60 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb

diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
new file mode 100644
index 0000000000..21f06782fb
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
@@ -0,0 +1,16 @@
+SUMMARY = "Produce a broken la file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+# remove-libtool.bbclass is inherited by default and removes all
+# .la files which for this test we specifically do not want.
+REMOVE_LIBTOOL_LA = "0"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/la-test.la
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
new file mode 100644
index 0000000000..e748310fc4
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Produce a broken pc file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/test.pc
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index 9457f1e3ac..588fc8c713 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -45,3 +45,35 @@ TESTSTRING:pn-sysroot-test-arch2 = "%s"
         expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
         res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
         self.assertTrue(expected in res.output, msg=res.output)
+
+    def test_sysroot_la(self):
+        """
+        Summary:   Check that workdir paths are not contained in .la files.
+        Expected:  Fail when a workdir path is found in the file content.
+        Author:    Paulo Neves <ptsneves@gmail.com>
+        """
+        expected = "la-test.la failed sanity test (workdir) in path"
+
+        res = bitbake("sysroot-la-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+
+        res = bitbake("sysroot-la-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+
+    def test_sysroot_pkgconfig(self):
+        """
+        Summary:   Check that tmpdir paths are not contained in .pc files.
+        Expected:  Fail when a tmpdir path is found in the file content.
+        Author:    Paulo Neves <ptsneves@gmail.com>
+        """
+        expected = "test.pc failed sanity test (tmpdir) in path"
+
+        res = bitbake("sysroot-pc-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
+        self.assertTrue(expected in res.output, msg=res.output)
+
+        res = bitbake("sysroot-pc-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
-- 
2.25.1

[PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util.

[Paulo Neves]

Useful to work around shebang relocation issues, where
shebangs are too long or have arguments in them, thus preventing them
from using the /usr/bin/env shebang.
---
 .../wrapper/cmdline-shebang-wrapper-test.bb   | 21 ++++++++++++
 .../recipes-test/wrapper/files/test.awk       |  2 ++
 meta/classes/utils.bbclass                    | 34 +++++++++++++++++++
 meta/lib/oeqa/selftest/cases/wrapper.py       | 11 ++++++
 4 files changed, 68 insertions(+)
 create mode 100644 meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
 create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
 create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py

diff --git a/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
new file mode 100644
index 0000000000..c4126a41fc
--- /dev/null
+++ b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
@@ -0,0 +1,21 @@
+SUMMARY = "Check that create_cmdline_shebang works"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI += "file://test.awk"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
+    sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
+    create_cmdline_shebang_wrapper ${D}${bindir}/test
+    if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
+        bbfatal "Wrapper is broken"
+    else
+        bbnote "Wrapper is good"
+    fi
+}
+
+BBCLASSEXTEND = "native"
diff --git a/meta-selftest/recipes-test/wrapper/files/test.awk b/meta-selftest/recipes-test/wrapper/files/test.awk
new file mode 100644
index 0000000000..91429197b1
--- /dev/null
+++ b/meta-selftest/recipes-test/wrapper/files/test.awk
@@ -0,0 +1,2 @@
+#! @AWK_BIN@ -f
+BEGIN { print "Don't Panic!" }
diff --git a/meta/classes/utils.bbclass b/meta/classes/utils.bbclass
index b4eb3d38ab..b617632d9f 100644
--- a/meta/classes/utils.bbclass
+++ b/meta/classes/utils.bbclass
@@ -184,6 +184,40 @@ END
 	chmod +x $cmd
 }
 
+create_cmdline_shebang_wrapper () {
+	# Create a wrapper script where commandline options are needed
+	#
+	# These are useful to work around shebang relocation issues, where shebangs are too
+  # long or have arguments in them, thus preventing them from using the /usr/bin/env
+	# shebang
+	#
+	# Usage: create_cmdline_wrapper FILENAME <extra-options>
+
+	cmd=$1
+	shift
+
+	echo "Generating wrapper script for $cmd"
+
+  # Strip #! and get remaining interpreter + arg
+  argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )")"
+  # strip the shebang from the real script as we do not want it to be usable anyway
+  tail -n +2 $cmd > $cmd.real
+	cmdname=$(basename $cmd)
+	dirname=$(dirname $cmd)
+	cmdoptions=$@
+	if [ "${base_prefix}" != "" ]; then
+		relpath=`python3 -c "import os; print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
+		cmdoptions=`echo $@ | sed -e "s:${base_prefix}:\\$realdir/$relpath:g"`
+	fi
+	cat <<END >$cmd
+#!/usr/bin/env bash
+realpath=\`readlink -fn \$0\`
+realdir=\`dirname \$realpath\`
+exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real $cmdoptions "\$@"
+END
+	chmod +x $cmd
+}
+
 create_wrapper () {
 	# Create a wrapper script where extra environment variables are needed
 	#
diff --git a/meta/lib/oeqa/selftest/cases/wrapper.py b/meta/lib/oeqa/selftest/cases/wrapper.py
new file mode 100644
index 0000000000..6de63310c0
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/wrapper.py
@@ -0,0 +1,11 @@
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake
+
+class WrapperTests(OESelftestTestCase):
+    def test_shebang_wrapper(self):
+        """
+        Summary:   Build a recipe which will fail if the cmdline_shebang_wrapper function is defective.
+        Expected:  Exit status to be 0.
+        Author:    Paulo Neves <ptsneves@gmail.com>
+        """
+        res = bitbake("cmdline-shebang-wrapper-test -c install", ignore_status=False)
-- 
2.25.1

[PATCH v5 7/7] libcheck: Fix too long shebang for native case.

[Paulo Neves]

It requires a shebang wrapper due to the fact that awk interpreter
has an argument.
---
 meta/recipes-support/libcheck/libcheck_0.15.2.bb | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/recipes-support/libcheck/libcheck_0.15.2.bb b/meta/recipes-support/libcheck/libcheck_0.15.2.bb
index 188d689cc3..1393aa2a1c 100644
--- a/meta/recipes-support/libcheck/libcheck_0.15.2.bb
+++ b/meta/recipes-support/libcheck/libcheck_0.15.2.bb
@@ -23,6 +23,9 @@ CACHED_CONFIGUREVARS += "ac_cv_path_AWK_PATH=${bindir}/gawk"
 
 RREPLACES:${PN} = "check (<= 0.9.5)"
 
+do_install:append:class-native() {
+    create_cmdline_shebang_wrapper ${D}${bindir}/checkmk
+}
 BBCLASSEXTEND = "native nativesdk"
 
 PACKAGES =+ "checkmk"
-- 
2.25.1

Re: [OE-core] [PATCH v4 6/7] utils: Add cmdline_shebang_wrapper util.

[Paulo Neves]

[-- Attachment #1: Type: text/plain, Size: 7685 bytes --]

Thanks for the great tips. Yeah the habit of using the FILESPATH is that 
mostly I work on bbappends. Today i learned.

I sent a new v5 with the corrections you mention.

Paulo Neves

On 6/14/22 16:17, Martin Jansa wrote:
> You can use bitbake-getvar to see the default FILESPATH, but "files" 
> next to recipe is included by default (I prefer to use BPN directory 
> for slightly faster lookup - unless the files are shared between 
> different recipes). FILESEXTRAPATHS is usually only needed from 
> bbappends which add new files to SRC_URI.
>
> For LICENSE you can use MIT and
> LIC_FILES_CHKSUM = 
> "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
> like many "pure-metadata" recipes in oe-core.
>
> On Tue, Jun 14, 2022 at 3:46 PM Paulo Neves <ptsneves@gmail.com> wrote:
>
>     Regarding the FILESEXTRAPATHS:prepend  i was convinced that
>     ${THISDIR}/files was not automatically added and was needed for
>     the test.awk file. I guess I have been cargo culting for some time
>     on this. Will remove it.
>
>     This is a recipe for a test and if i add a license different than
>     CLOSED than i need to add a license file, which seems a bit
>     overkill for a one line. Am I wrong? What license should it even
>     be? Dont Panic is a common string used in awk docs.
>
>     Paulo Neves
>
>     On 6/14/22 15:39, Martin Jansa wrote:
>>     FILESEXTRAPATHS:prepend doesn't seem to be needed and LICENSE is
>>     really CLOSED?
>>
>>     On Tue, Jun 14, 2022 at 3:17 PM Paulo Neves <ptsneves@gmail.com>
>>     wrote:
>>
>>         Useful to work around shebang relocation issues, where
>>         shebangs are too long or have arguments in them, thus
>>         preventing them
>>         from using the /usr/bin/env shebang.
>>         ---
>>          .../wrapper/cmdline-shebang-wrapper-test.bb
>>         <http://cmdline-shebang-wrapper-test.bb>  | 21 ++++++++++++
>>          .../recipes-test/wrapper/files/test.awk       |  2 ++
>>          meta/classes/utils.bbclass                    | 34
>>         +++++++++++++++++++
>>          meta/lib/oeqa/selftest/cases/wrapper.py       | 11 ++++++
>>          4 files changed, 68 insertions(+)
>>          create mode 100644
>>         meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
>>         <http://cmdline-shebang-wrapper-test.bb>
>>          create mode 100644
>>         meta-selftest/recipes-test/wrapper/files/test.awk
>>          create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py
>>
>>         diff --git
>>         a/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
>>         <http://cmdline-shebang-wrapper-test.bb>
>>         b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
>>         <http://cmdline-shebang-wrapper-test.bb>
>>         new file mode 100644
>>         index 0000000000..302eea8901
>>         --- /dev/null
>>         +++
>>         b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
>>         <http://cmdline-shebang-wrapper-test.bb>
>>         @@ -0,0 +1,21 @@
>>         +FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
>>         +SUMMARY = "Check that create_cmdline_shebang works"
>>         +LICENSE = "CLOSED"
>>         +INHIBIT_DEFAULT_DEPS = "1"
>>         +
>>         +SRC_URI += "file://test.awk"
>>         +
>>         +EXCLUDE_FROM_WORLD = "1"
>>         +do_install() {
>>         +    install -d ${D}${bindir}
>>         +    install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
>>         +    sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
>>         +    create_cmdline_shebang_wrapper ${D}${bindir}/test
>>         +    if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
>>         +        bbfatal "Wrapper is broken"
>>         +    else
>>         +        bbnote "Wrapper is good"
>>         +    fi
>>         +}
>>         +
>>         +BBCLASSEXTEND = "native"
>>         diff --git
>>         a/meta-selftest/recipes-test/wrapper/files/test.awk
>>         b/meta-selftest/recipes-test/wrapper/files/test.awk
>>         new file mode 100644
>>         index 0000000000..91429197b1
>>         --- /dev/null
>>         +++ b/meta-selftest/recipes-test/wrapper/files/test.awk
>>         @@ -0,0 +1,2 @@
>>         +#! @AWK_BIN@ -f
>>         +BEGIN { print "Don't Panic!" }
>>         diff --git a/meta/classes/utils.bbclass
>>         b/meta/classes/utils.bbclass
>>         index b4eb3d38ab..b617632d9f 100644
>>         --- a/meta/classes/utils.bbclass
>>         +++ b/meta/classes/utils.bbclass
>>         @@ -184,6 +184,40 @@ END
>>                 chmod +x $cmd
>>          }
>>
>>         +create_cmdline_shebang_wrapper () {
>>         +       # Create a wrapper script where commandline options
>>         are needed
>>         +       #
>>         +       # These are useful to work around shebang relocation
>>         issues, where shebangs are too
>>         +  # long or have arguments in them, thus preventing them
>>         from using the /usr/bin/env
>>         +       # shebang
>>         +       #
>>         +       # Usage: create_cmdline_wrapper FILENAME <extra-options>
>>         +
>>         +       cmd=$1
>>         +       shift
>>         +
>>         +       echo "Generating wrapper script for $cmd"
>>         +
>>         +  # Strip #! and get remaining interpreter + arg
>>         +  argument="$(basename "$(head -n1 $cmd | sed -e 's|#![
>>         ]*||g' )")"
>>         +  # strip the shebang from the real script as we do not want
>>         it to be usable anyway
>>         +  tail -n +2 $cmd > $cmd.real
>>         +       cmdname=$(basename $cmd)
>>         +       dirname=$(dirname $cmd)
>>         +       cmdoptions=$@
>>         +       if [ "${base_prefix}" != "" ]; then
>>         +               relpath=`python3 -c "import os;
>>         print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
>>         +               cmdoptions=`echo $@ | sed -e
>>         "s:${base_prefix}:\\$realdir/$relpath:g"`
>>         +       fi
>>         +       cat <<END >$cmd
>>         +#!/usr/bin/env bash
>>         +realpath=\`readlink -fn \$0\`
>>         +realdir=\`dirname \$realpath\`
>>         +exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real
>>         $cmdoptions "\$@"
>>         +END
>>         +       chmod +x $cmd
>>         +}
>>         +
>>          create_wrapper () {
>>                 # Create a wrapper script where extra environment
>>         variables are needed
>>                 #
>>         diff --git a/meta/lib/oeqa/selftest/cases/wrapper.py
>>         b/meta/lib/oeqa/selftest/cases/wrapper.py
>>         new file mode 100644
>>         index 0000000000..6de63310c0
>>         --- /dev/null
>>         +++ b/meta/lib/oeqa/selftest/cases/wrapper.py
>>         @@ -0,0 +1,11 @@
>>         +from oeqa.selftest.case import OESelftestTestCase
>>         +from oeqa.utils.commands import bitbake
>>         +
>>         +class WrapperTests(OESelftestTestCase):
>>         +    def test_shebang_wrapper(self):
>>         +        """
>>         +        Summary:   Build a recipe which will fail if the
>>         cmdline_shebang_wrapper function is defective.
>>         +        Expected:  Exit status to be 0.
>>         +        Author:    Paulo Neves <ptsneves@gmail.com>
>>         +        """
>>         +        res = bitbake("cmdline-shebang-wrapper-test -c
>>         install", ignore_status=False)
>>         -- 
>>         2.25.1
>>
>>
>>         
>>
>

[-- Attachment #2: Type: text/html, Size: 12012 bytes --]

Re: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util.

[Ross Burton]

On 14 Jun 2022, at 16:11, Paulo Neves via lists.openembedded.org <ptsneves=gmail.com@lists.openembedded.org> wrote:
> +create_cmdline_shebang_wrapper () {

The indentation in this function is a mix of tabs and spaces, please unify.

> +  argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )”)”


[ ]* seems overly, complex, no need for the brackets. I’d anchor to the beginning of the line, and remove the g as there’s only one expected.

Might also be wise to assert that you found something here, so running this on an ELF fails with an obvious error.

Ross
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [OE-core] [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow

[Ross Burton]

> +        expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
> +        res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
> +        self.assertTrue(expected in res.output, msg=res.output)

I err towards paranoia and would also check that res.status != 0.

Ross

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [OE-core] [PATCH 1/3] insane.bbclass: Make do_qa_staging check shebangs

[Ross Burton]

> +            if not skip_shebang_size:
> +                errors = {}
> +                package_qa_check_shebang_size(path, "", d, None, errors)
> +                for e in errors:
> +                    oe.qa.handle_error(e, errors[e], d)

Where is package_qa_check_shebang_size() defined?

Ross
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [OE-core] [PATCH 1/3] insane.bbclass: Make do_qa_staging check shebangs

[Ross Burton]

Never mind, can’t read :)

> On 15 Jun 2022, at 12:07, Ross Burton <Ross.Burton@arm.com> wrote:
>
>> +            if not skip_shebang_size:
>> +                errors = {}
>> +                package_qa_check_shebang_size(path, "", d, None, errors)
>> +                for e in errors:
>> +                    oe.qa.handle_error(e, errors[e], d)
>
> Where is package_qa_check_shebang_size() defined?
>
> Ross

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [OE-core] [PATCH 1/3] insane.bbclass: Make do_qa_staging check shebangs

[Paulo Neves]

:)

The function should probably be refactored so it does not imply it is 
only useful for packaging.

Paulo Neves

On 6/15/22 13:08, Ross Burton wrote:
> Never mind, can’t read :)
>
>> On 15 Jun 2022, at 12:07, Ross Burton <Ross.Burton@arm.com> wrote:
>>
>>> +            if not skip_shebang_size:
>>> +                errors = {}
>>> +                package_qa_check_shebang_size(path, "", d, None, errors)
>>> +                for e in errors:
>>> +                    oe.qa.handle_error(e, errors[e], d)
>> Where is package_qa_check_shebang_size() defined?
>>
>> Ross
> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

RE: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util.

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-
> core@lists.openembedded.org> On Behalf Of Ross Burton
> Sent: den 15 juni 2022 12:52
> To: ptsneves@gmail.com
> Cc: openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper
> util.
> 
> On 14 Jun 2022, at 16:11, Paulo Neves via lists.openembedded.org
> <ptsneves=gmail.com@lists.openembedded.org> wrote:
> > +create_cmdline_shebang_wrapper () {
> 
> The indentation in this function is a mix of tabs and spaces, please
> unify.
> 
> > +  argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )”)”
> 
> [ ]* seems overly, complex, no need for the brackets. I’d anchor to the
> beginning of the line, and remove the g as there’s only one expected.

This should work:

argument=$(basename "$(sed -ne 's/^#! *//p;q')" $cmd)

However, if the first line is something like "#!/usr/bin/env python3", then 
$argument will be "env python3" and somehow I assume that is not exactly 
what the rest of the code expects.

> 
> Might also be wise to assert that you found something here, so running
> this on an ELF fails with an obvious error.
> 
> Ross

//Peter

Re: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util.

[Paulo Neves]

On 6/15/22 15:11, Peter Kjellerstedt wrote:
>> -----Original Message-----
>> From: openembedded-core@lists.openembedded.org <openembedded-
>> core@lists.openembedded.org> On Behalf Of Ross Burton
>> Sent: den 15 juni 2022 12:52
>> To: ptsneves@gmail.com
>> Cc: openembedded-core@lists.openembedded.org
>> Subject: Re: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper
>> util.
>>
>> On 14 Jun 2022, at 16:11, Paulo Neves via lists.openembedded.org
>> <ptsneves=gmail.com@lists.openembedded.org> wrote:
>>> +create_cmdline_shebang_wrapper () {
>> The indentation in this function is a mix of tabs and spaces, please
>> unify.
>>
>>> +  argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )”)”
>> [ ]* seems overly, complex, no need for the brackets. I’d anchor to the
>> beginning of the line, and remove the g as there’s only one expected.
> This should work:
>
> argument=$(basename "$(sed -ne 's/^#! *//p;q')" $cmd)
>
> However, if the first line is something like "#!/usr/bin/env python3", then
> $argument will be "env python3" and somehow I assume that is not exactly
> what the rest of the code expects.

The use case is that whatever is in shebang should be extracted and ran 
verbatim, so truncating like your suggestion can work but has slightly 
different semantic.
>> Might also be wise to assert that you found something here, so running
>> this on an ELF fails with an obvious error.
>>
>> Ross
> //Peter
>
Paulo Neves

RE: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util.

[Peter Kjellerstedt]

> -----Original Message-----
> From: Paulo Neves <ptsneves@gmail.com>
> Sent: den 15 juni 2022 15:51
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; Ross Burton
> <ross.burton@arm.com>
> Cc: openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper
> util.
> 
> On 6/15/22 15:11, Peter Kjellerstedt wrote:
> >> -----Original Message-----
> >> From: openembedded-core@lists.openembedded.org <openembedded-
> >> core@lists.openembedded.org> On Behalf Of Ross Burton
> >> Sent: den 15 juni 2022 12:52
> >> To: ptsneves@gmail.com
> >> Cc: openembedded-core@lists.openembedded.org
> >> Subject: Re: [OE-core] [PATCH v5 6/7] utils: Add
> cmdline_shebang_wrapper
> >> util.
> >>
> >> On 14 Jun 2022, at 16:11, Paulo Neves via lists.openembedded.org
> >> <ptsneves=gmail.com@lists.openembedded.org> wrote:
> >>> +create_cmdline_shebang_wrapper () {
> >> The indentation in this function is a mix of tabs and spaces, please
> >> unify.
> >>
> >>> +  argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )”)”
> >> [ ]* seems overly, complex, no need for the brackets. I’d anchor to the
> >> beginning of the line, and remove the g as there’s only one expected.
> > This should work:
> >
> > argument=$(basename "$(sed -ne 's/^#! *//p;q')" $cmd)
> >
> > However, if the first line is something like "#!/usr/bin/env python3", then
> > $argument will be "env python3" and somehow I assume that is not exactly
> > what the rest of the code expects.
> 
> The use case is that whatever is in shebang should be extracted and ran
> verbatim, so truncating like your suggestion can work but has slightly
> different semantic.

Umm, my suggested line above should give exactly the same output as yours, 
only with fewer commands...

> >> Might also be wise to assert that you found something here, so running
> >> this on an ELF fails with an obvious error.
> >>
> >> Ross
> > //Peter
> >
> Paulo Neves

//Peter

Re: [OE-core] [PATCH v5 3/7] insane.bbclass: Make do_qa_staging check shebangs

[Richard Purdie]

On Tue, 2022-06-14 at 17:11 +0200, Paulo Neves wrote:
> As reported in the bug report [1], there was no check for shebang
> sizes on native scripts and now this is fixed.
> 
> The path scope of the qa_staging was increased from just checking
> libdir to all the relevant SYSROOT_DIRS.
> 
> It is possible to skip this check through INSANE_SKIP.
> 
> [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
> 
> Signed-off-by: Paulo Neves <ptsneves@gmail.com>
> ---
>  meta/classes/insane.bbclass | 16 +++++++++++++++-
>  1 file changed, 15 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
> index 9ca84bace9..b2951a48fe 100644
> --- a/meta/classes/insane.bbclass
> +++ b/meta/classes/insane.bbclass
> @@ -630,6 +630,11 @@ def qa_check_staged(path,d):
>          bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
>          skip_pkgconfig = True
>  
> +    skip_shebang_size = False
> +    if 'shebang-size' in skip:
> +        bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
> +        skip_shebang_size = True
> +
>      # find all .la and .pc files
>      # read the content
>      # and check for stuff that looks wrong
> @@ -651,6 +656,13 @@ def qa_check_staged(path,d):
>                          error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
>                          oe.qa.handle_error("pkgconfig", error_msg, d)
>  
> +            if not skip_shebang_size:
> +                errors = {}
> +                package_qa_check_shebang_size(path, "", d, None, errors)
> +                for e in errors:
> +                    oe.qa.handle_error(e, errors[e], d)
> +
> +
>  # Run all package-wide warnfuncs and errorfuncs
>  def package_qa_package(warnfuncs, errorfuncs, package, d):
>      warnings = {}
> @@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
>  
>  python do_qa_staging() {
>      bb.note("QA checking staging")
> -    qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
> +    sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
> +    for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
> +        qa_check_staged(sysroot_destdir + sysroot_dir, d)
>      oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
>  }


I'm a little worried about the performance implications of this, we're
going from scanning files in libdir to scanning nearly all files in the
sysroots, reading from many of them. In isolation that doesn't seem
much but I suspect the IO will impact builds overall.

That leaves me a little torn on this change :/

Cheers,

Richard

Re: [OE-core] [PATCH v5 3/7] insane.bbclass: Make do_qa_staging check shebangs

[Paulo Neves]

[-- Attachment #1: Type: text/plain, Size: 3139 bytes --]

Yes, when I was making the change I also realized the scope of the check
was quite bigger than before. I am going to deliver the final changes
requested and leave it up to you to decide on the merge. Let me know your
decision so I can close the bug as well.

Paulo Neves

On Fri, Jun 17, 2022, 18:50 Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Tue, 2022-06-14 at 17:11 +0200, Paulo Neves wrote:
> > As reported in the bug report [1], there was no check for shebang
> > sizes on native scripts and now this is fixed.
> >
> > The path scope of the qa_staging was increased from just checking
> > libdir to all the relevant SYSROOT_DIRS.
> >
> > It is possible to skip this check through INSANE_SKIP.
> >
> > [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
> >
> > Signed-off-by: Paulo Neves <ptsneves@gmail.com>
> > ---
> >  meta/classes/insane.bbclass | 16 +++++++++++++++-
> >  1 file changed, 15 insertions(+), 1 deletion(-)
> >
> > diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
> > index 9ca84bace9..b2951a48fe 100644
> > --- a/meta/classes/insane.bbclass
> > +++ b/meta/classes/insane.bbclass
> > @@ -630,6 +630,11 @@ def qa_check_staged(path,d):
> >          bb.note("Recipe %s skipping qa checking: pkgconfig" %
> d.getVar('PN'))
> >          skip_pkgconfig = True
> >
> > +    skip_shebang_size = False
> > +    if 'shebang-size' in skip:
> > +        bb.note("Recipe %s skipping qa checkking: shebang-size" %
> d.getVar('PN'))
> > +        skip_shebang_size = True
> > +
> >      # find all .la and .pc files
> >      # read the content
> >      # and check for stuff that looks wrong
> > @@ -651,6 +656,13 @@ def qa_check_staged(path,d):
> >                          error_msg = "%s failed sanity test (tmpdir) in
> path %s" % (file,root)
> >                          oe.qa.handle_error("pkgconfig", error_msg, d)
> >
> > +            if not skip_shebang_size:
> > +                errors = {}
> > +                package_qa_check_shebang_size(path, "", d, None, errors)
> > +                for e in errors:
> > +                    oe.qa.handle_error(e, errors[e], d)
> > +
> > +
> >  # Run all package-wide warnfuncs and errorfuncs
> >  def package_qa_package(warnfuncs, errorfuncs, package, d):
> >      warnings = {}
> > @@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
> >
> >  python do_qa_staging() {
> >      bb.note("QA checking staging")
> > -    qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
> > +    sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
> > +    for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
> > +        qa_check_staged(sysroot_destdir + sysroot_dir, d)
> >      oe.qa.exit_with_message_if_errors("QA staging was broken by the
> package built above", d)
> >  }
>
>
> I'm a little worried about the performance implications of this, we're
> going from scanning files in libdir to scanning nearly all files in the
> sysroots, reading from many of them. In isolation that doesn't seem
> much but I suspect the IO will impact builds overall.
>
> That leaves me a little torn on this change :/
>
> Cheers,
>
> Richard
>
>

[-- Attachment #2: Type: text/html, Size: 4289 bytes --]

Re: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util.

[Paulo Neves]

You are right never mind.

Paulo Neves

On 6/16/22 00:16, Peter Kjellerstedt wrote:
>> -----Original Message-----
>> From: Paulo Neves <ptsneves@gmail.com>
>> Sent: den 15 juni 2022 15:51
>> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; Ross Burton
>> <ross.burton@arm.com>
>> Cc: openembedded-core@lists.openembedded.org
>> Subject: Re: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper
>> util.
>>
>> On 6/15/22 15:11, Peter Kjellerstedt wrote:
>>>> -----Original Message-----
>>>> From: openembedded-core@lists.openembedded.org <openembedded-
>>>> core@lists.openembedded.org> On Behalf Of Ross Burton
>>>> Sent: den 15 juni 2022 12:52
>>>> To: ptsneves@gmail.com
>>>> Cc: openembedded-core@lists.openembedded.org
>>>> Subject: Re: [OE-core] [PATCH v5 6/7] utils: Add
>> cmdline_shebang_wrapper
>>>> util.
>>>>
>>>> On 14 Jun 2022, at 16:11, Paulo Neves via lists.openembedded.org
>>>> <ptsneves=gmail.com@lists.openembedded.org> wrote:
>>>>> +create_cmdline_shebang_wrapper () {
>>>> The indentation in this function is a mix of tabs and spaces, please
>>>> unify.
>>>>
>>>>> +  argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )”)”
>>>> [ ]* seems overly, complex, no need for the brackets. I’d anchor to the
>>>> beginning of the line, and remove the g as there’s only one expected.
>>> This should work:
>>>
>>> argument=$(basename "$(sed -ne 's/^#! *//p;q')" $cmd)
>>>
>>> However, if the first line is something like "#!/usr/bin/env python3", then
>>> $argument will be "env python3" and somehow I assume that is not exactly
>>> what the rest of the code expects.
>> The use case is that whatever is in shebang should be extracted and ran
>> verbatim, so truncating like your suggestion can work but has slightly
>> different semantic.
> Umm, my suggested line above should give exactly the same output as yours,
> only with fewer commands...
>
>>>> Might also be wise to assert that you found something here, so running
>>>> this on an ELF fails with an obvious error.
>>>>
>>>> Ross
>>> //Peter
>>>
>> Paulo Neves
> //Peter
>

[PATCH v6 1/7] python: Avoid shebang overflow on python-config.py

[Paulo Neves]

The native path may be too big, leading to shebang
overflow. Just use the #!/usr/bin/env python3.

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 ...shebang-overflow-on-python-config.py.patch | 33 +++++++++++++++++++
 .../recipes-devtools/python/python3_3.10.4.bb |  1 +
 2 files changed, 34 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch

diff --git a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
new file mode 100644
index 0000000000..921da8de7c
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
@@ -0,0 +1,33 @@
+From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
+From: Paulo Neves <ptsneves@gmail.com>
+Date: Tue, 7 Jun 2022 16:16:41 +0200
+Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
+
+The whole native path may be too big, leading to shebang
+overflow. Let's just use the env shebang.
+
+Denial reason: [1]
+
+Upstream-Status: Denied [distribution]
+
+[1] https://github.com/python/cpython/pull/93760#pullrequestreview-1005365737
+---
+ Makefile.pre.in | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
+ 	@ # Substitution happens here, as the completely-expanded BINDIR
+ 	@ # is not available in configure
+ 	sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
++	@ # Otherwise we might get huge shebangs with native paths
++	sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
+ 	@ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
+ 	LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
+ 	@  # In OpenEmbedded, always use the python version of the script, the shell
+-- 
+2.25.1
+
diff --git a/meta/recipes-devtools/python/python3_3.10.4.bb b/meta/recipes-devtools/python/python3_3.10.4.bb
index 6bd3a6aba8..357025f856 100644
--- a/meta/recipes-devtools/python/python3_3.10.4.bb
+++ b/meta/recipes-devtools/python/python3_3.10.4.bb
@@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
            file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
            file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
            file://deterministic_imports.patch \
+           file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
            "
 
 SRC_URI:append:class-native = " \
-- 
2.25.1

[PATCH v6 2/7] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2

[Paulo Neves]

A native path can cause a shebang overflow on gtkdoc-mkhtml.
Replace it with /usr/bin/env.

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb b/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
index 150d2c0b23..392913fcc6 100644
--- a/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
+++ b/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
@@ -43,6 +43,7 @@ do_install:append () {
         ${datadir}/gtk-doc/python/gtkdoc/config.py; do
         sed -e 's,${RECIPE_SYSROOT_NATIVE}/usr/bin/pkg-config,${bindir}/pkg-config,' \
             -e 's,${HOSTTOOLS_DIR}/python3,${bindir}/python3,' \
+            -e '1s|^#!.*|#!/usr/bin/env python3|' \
             -i ${D}$fn
     done
 }
-- 
2.25.1

[PATCH v6 3/7] insane.bbclass: Make do_qa_staging check shebangs

[Paulo Neves]

As reported in the bug report [1], there was no check for shebang
sizes on native scripts and now this is fixed.

The path scope of the qa_staging was increased from just checking
libdir to all the relevant SYSROOT_DIRS.

It is possible to skip this check through INSANE_SKIP.

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 meta/classes/insane.bbclass | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 9ca84bace9..b2951a48fe 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -630,6 +630,11 @@ def qa_check_staged(path,d):
         bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
         skip_pkgconfig = True
 
+    skip_shebang_size = False
+    if 'shebang-size' in skip:
+        bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
+        skip_shebang_size = True
+
     # find all .la and .pc files
     # read the content
     # and check for stuff that looks wrong
@@ -651,6 +656,13 @@ def qa_check_staged(path,d):
                         error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
                         oe.qa.handle_error("pkgconfig", error_msg, d)
 
+            if not skip_shebang_size:
+                errors = {}
+                package_qa_check_shebang_size(path, "", d, None, errors)
+                for e in errors:
+                    oe.qa.handle_error(e, errors[e], d)
+
+
 # Run all package-wide warnfuncs and errorfuncs
 def package_qa_package(warnfuncs, errorfuncs, package, d):
     warnings = {}
@@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
 
 python do_qa_staging() {
     bb.note("QA checking staging")
-    qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
+    sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
+    for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
+        qa_check_staged(sysroot_destdir + sysroot_dir, d)
     oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
 }
 
-- 
2.25.1

[PATCH v6 4/7] oeqa/selftest: Add test for shebang overflow

[Paulo Neves]

Make sure we do not stage any executable with a bigger shebang
than 128. Fixes [1]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 .../sysroot-test/sysroot-shebang-test_1.0.bb         | 12 ++++++++++++
 meta/lib/oeqa/selftest/cases/sysroot.py              | 11 +++++++++++
 2 files changed, 23 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb

diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
new file mode 100644
index 0000000000..6c834be897
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Check that shebang does not exceed 128 characters"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    echo '#!BiM3cnVd1Amtv6PG+FynrQiVMbZnX5ELgF21q3EkuB+44JEGWtq8TvBJ7EGidfVs3eR3wVOUbLnjYDlKUWcm7YC/ute7f+KDHbwxziRUSUBZAUqgjiQdfQ0HnxajI0ozbM863E9JV9k13yZKYfh9/zR77Y6Dl4Dd3zOWS75LSpkAXV' > ${D}${bindir}/max-shebang
+    chmod 755 ${D}${bindir}/max-shebang
+}
+
+BBCLASSEXTEND = "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index 315d1a61c2..a17fd1818a 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -35,3 +35,14 @@ TESTSTRING:pn-sysroot-test-arch1 = "%s"
 TESTSTRING:pn-sysroot-test-arch2 = "%s"
 """ % (uuid1, uuid2))
         bitbake("sysroot-test")
+
+    def test_sysroot_max_shebang(self):
+        """
+        Summary:   Check max shebang triggers. To confirm [YOCTO #11053] is closed.
+        Expected:  Fail when a shebang bigger than the max shebang-size is reached.
+        Author:    Paulo Neves <ptsneves@gmail.com>
+        """
+        expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
+        res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue(res.status != 0)
-- 
2.25.1

[PATCH v6 5/7] oeqa/selftest: Test staged .la and .pc files

[Paulo Neves]

These files are checked by qa_check_staged but there was no
test cases for whether the tests actually worked. Now there
are.

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 .../sysroot-test/sysroot-la-test_1.0.bb       | 16 +++++++++
 .../sysroot-test/sysroot-pc-test_1.0.bb       | 12 +++++++
 meta/lib/oeqa/selftest/cases/sysroot.py       | 36 +++++++++++++++++++
 3 files changed, 64 insertions(+)
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
 create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb

diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
new file mode 100644
index 0000000000..21f06782fb
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
@@ -0,0 +1,16 @@
+SUMMARY = "Produce a broken la file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+# remove-libtool.bbclass is inherited by default and removes all
+# .la files which for this test we specifically do not want.
+REMOVE_LIBTOOL_LA = "0"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/la-test.la
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
new file mode 100644
index 0000000000..e748310fc4
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Produce a broken pc file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+do_install() {
+    install -d ${D}${libdir}/test/
+    echo '${WORKDIR}' > ${D}${libdir}/test/test.pc
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index a17fd1818a..294ba4a4a0 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -46,3 +46,39 @@ TESTSTRING:pn-sysroot-test-arch2 = "%s"
         res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
         self.assertTrue(expected in res.output, msg=res.output)
         self.assertTrue(res.status != 0)
+
+    def test_sysroot_la(self):
+        """
+        Summary:   Check that workdir paths are not contained in .la files.
+        Expected:  Fail when a workdir path is found in the file content.
+        Author:    Paulo Neves <ptsneves@gmail.com>
+        """
+        expected = "la-test.la failed sanity test (workdir) in path"
+
+        res = bitbake("sysroot-la-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+        self.assertTrue(res.status != 0)
+
+        res = bitbake("sysroot-la-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[la]' in res.output, msg=res.output)
+        self.assertTrue(res.status != 0)
+
+    def test_sysroot_pkgconfig(self):
+        """
+        Summary:   Check that tmpdir paths are not contained in .pc files.
+        Expected:  Fail when a tmpdir path is found in the file content.
+        Author:    Paulo Neves <ptsneves@gmail.com>
+        """
+        expected = "test.pc failed sanity test (tmpdir) in path"
+
+        res = bitbake("sysroot-pc-test -c populate_sysroot", ignore_status=True)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue(res.status != 0)
+
+        res = bitbake("sysroot-pc-test-native -c populate_sysroot", ignore_status=True)
+        self.assertTrue(expected in res.output, msg=res.output)
+        self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
+        self.assertTrue(res.status != 0)
-- 
2.25.1

[PATCH v6 6/7] utils: Add cmdline_shebang_wrapper util.

[Paulo Neves]

Useful to work around shebang relocation issues, where
shebangs are too long or have arguments in them, thus preventing them
from using the /usr/bin/env shebang.
---
 .../wrapper/cmdline-shebang-wrapper-test.bb   | 21 ++++++++++++
 .../recipes-test/wrapper/files/test.awk       |  2 ++
 meta/classes/utils.bbclass                    | 34 +++++++++++++++++++
 meta/lib/oeqa/selftest/cases/wrapper.py       | 11 ++++++
 4 files changed, 68 insertions(+)
 create mode 100644 meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
 create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
 create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py

diff --git a/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
new file mode 100644
index 0000000000..c4126a41fc
--- /dev/null
+++ b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
@@ -0,0 +1,21 @@
+SUMMARY = "Check that create_cmdline_shebang works"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI += "file://test.awk"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+    install -d ${D}${bindir}
+    install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
+    sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
+    create_cmdline_shebang_wrapper ${D}${bindir}/test
+    if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
+        bbfatal "Wrapper is broken"
+    else
+        bbnote "Wrapper is good"
+    fi
+}
+
+BBCLASSEXTEND = "native"
diff --git a/meta-selftest/recipes-test/wrapper/files/test.awk b/meta-selftest/recipes-test/wrapper/files/test.awk
new file mode 100644
index 0000000000..91429197b1
--- /dev/null
+++ b/meta-selftest/recipes-test/wrapper/files/test.awk
@@ -0,0 +1,2 @@
+#! @AWK_BIN@ -f
+BEGIN { print "Don't Panic!" }
diff --git a/meta/classes/utils.bbclass b/meta/classes/utils.bbclass
index b4eb3d38ab..b58c22771f 100644
--- a/meta/classes/utils.bbclass
+++ b/meta/classes/utils.bbclass
@@ -184,6 +184,40 @@ END
 	chmod +x $cmd
 }
 
+create_cmdline_shebang_wrapper () {
+	# Create a wrapper script where commandline options are needed
+	#
+	# These are useful to work around shebang relocation issues, where shebangs are too
+	# long or have arguments in them, thus preventing them from using the /usr/bin/env
+	# shebang
+	#
+	# Usage: create_cmdline_wrapper FILENAME <extra-options>
+
+	cmd=$1
+	shift
+
+	echo "Generating wrapper script for $cmd"
+
+	# Strip #! and get remaining interpreter + arg
+	argument="$(sed -ne 's/^#! *//p;q' $cmd)"
+	# strip the shebang from the real script as we do not want it to be usable anyway
+	tail -n +2 $cmd > $cmd.real
+	cmdname=$(basename $cmd)
+	dirname=$(dirname $cmd)
+	cmdoptions=$@
+	if [ "${base_prefix}" != "" ]; then
+		relpath=`python3 -c "import os; print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
+		cmdoptions=`echo $@ | sed -e "s:${base_prefix}:\\$realdir/$relpath:g"`
+	fi
+	cat <<END >$cmd
+#!/usr/bin/env bash
+realpath=\`readlink -fn \$0\`
+realdir=\`dirname \$realpath\`
+exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real $cmdoptions "\$@"
+END
+	chmod +x $cmd
+}
+
 create_wrapper () {
 	# Create a wrapper script where extra environment variables are needed
 	#
diff --git a/meta/lib/oeqa/selftest/cases/wrapper.py b/meta/lib/oeqa/selftest/cases/wrapper.py
new file mode 100644
index 0000000000..6de63310c0
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/wrapper.py
@@ -0,0 +1,11 @@
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake
+
+class WrapperTests(OESelftestTestCase):
+    def test_shebang_wrapper(self):
+        """
+        Summary:   Build a recipe which will fail if the cmdline_shebang_wrapper function is defective.
+        Expected:  Exit status to be 0.
+        Author:    Paulo Neves <ptsneves@gmail.com>
+        """
+        res = bitbake("cmdline-shebang-wrapper-test -c install", ignore_status=False)
-- 
2.25.1

[PATCH v6 7/7] libcheck: Fix too long shebang for native case.

[Paulo Neves]

It requires a shebang wrapper due to the fact that awk interpreter
has an argument.
---
 meta/recipes-support/libcheck/libcheck_0.15.2.bb | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/recipes-support/libcheck/libcheck_0.15.2.bb b/meta/recipes-support/libcheck/libcheck_0.15.2.bb
index 188d689cc3..1393aa2a1c 100644
--- a/meta/recipes-support/libcheck/libcheck_0.15.2.bb
+++ b/meta/recipes-support/libcheck/libcheck_0.15.2.bb
@@ -23,6 +23,9 @@ CACHED_CONFIGUREVARS += "ac_cv_path_AWK_PATH=${bindir}/gawk"
 
 RREPLACES:${PN} = "check (<= 0.9.5)"
 
+do_install:append:class-native() {
+    create_cmdline_shebang_wrapper ${D}${bindir}/checkmk
+}
 BBCLASSEXTEND = "native nativesdk"
 
 PACKAGES =+ "checkmk"
-- 
2.25.1

Re: [OE-core] [PATCH v5 3/7] insane.bbclass: Make do_qa_staging check shebangs

[Richard Purdie]

On Fri, 2022-06-17 at 17:50 +0100, Richard Purdie via
lists.openembedded.org wrote:
> On Tue, 2022-06-14 at 17:11 +0200, Paulo Neves wrote:
> > As reported in the bug report [1], there was no check for shebang
> > sizes on native scripts and now this is fixed.
> > 
> > The path scope of the qa_staging was increased from just checking
> > libdir to all the relevant SYSROOT_DIRS.
> > 
> > It is possible to skip this check through INSANE_SKIP.
> > 
> > [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
> > 
> > Signed-off-by: Paulo Neves <ptsneves@gmail.com>
> > ---
> >  meta/classes/insane.bbclass | 16 +++++++++++++++-
> >  1 file changed, 15 insertions(+), 1 deletion(-)
> > 
> > diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
> > index 9ca84bace9..b2951a48fe 100644
> > --- a/meta/classes/insane.bbclass
> > +++ b/meta/classes/insane.bbclass
> > @@ -630,6 +630,11 @@ def qa_check_staged(path,d):
> >          bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
> >          skip_pkgconfig = True
> >  
> > +    skip_shebang_size = False
> > +    if 'shebang-size' in skip:
> > +        bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
> > +        skip_shebang_size = True
> > +
> >      # find all .la and .pc files
> >      # read the content
> >      # and check for stuff that looks wrong
> > @@ -651,6 +656,13 @@ def qa_check_staged(path,d):
> >                          error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
> >                          oe.qa.handle_error("pkgconfig", error_msg, d)
> >  
> > +            if not skip_shebang_size:
> > +                errors = {}
> > +                package_qa_check_shebang_size(path, "", d, None, errors)
> > +                for e in errors:
> > +                    oe.qa.handle_error(e, errors[e], d)
> > +
> > +
> >  # Run all package-wide warnfuncs and errorfuncs
> >  def package_qa_package(warnfuncs, errorfuncs, package, d):
> >      warnings = {}
> > @@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
> >  
> >  python do_qa_staging() {
> >      bb.note("QA checking staging")
> > -    qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
> > +    sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
> > +    for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
> > +        qa_check_staged(sysroot_destdir + sysroot_dir, d)
> >      oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
> >  }
> 
> 
> I'm a little worried about the performance implications of this, we're
> going from scanning files in libdir to scanning nearly all files in the
> sysroots, reading from many of them. In isolation that doesn't seem
> much but I suspect the IO will impact builds overall.
> 
> That leaves me a little torn on this change :/

Sorry about the delay on this. My concern isn't an issue as I wasn't
remembering the exact context of this code correctly. The patches have
therefore merged, thanks!

Cheers,

Richard

Re: [OE-core] [PATCH v6 6/7] utils: Add cmdline_shebang_wrapper util.

[Jacob Kroon]

On 6/19/22 21:20, Paulo Neves wrote:
> Useful to work around shebang relocation issues, where
> shebangs are too long or have arguments in them, thus preventing them
> from using the /usr/bin/env shebang.
> ---
>   .../wrapper/cmdline-shebang-wrapper-test.bb   | 21 ++++++++++++
>   .../recipes-test/wrapper/files/test.awk       |  2 ++
>   meta/classes/utils.bbclass                    | 34 +++++++++++++++++++
>   meta/lib/oeqa/selftest/cases/wrapper.py       | 11 ++++++
>   4 files changed, 68 insertions(+)
>   create mode 100644 meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
>   create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
>   create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py
> 
> diff --git a/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
> new file mode 100644
> index 0000000000..c4126a41fc
> --- /dev/null
> +++ b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
> @@ -0,0 +1,21 @@
> +SUMMARY = "Check that create_cmdline_shebang works"
> +LICENSE = "MIT"
> +LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
> +INHIBIT_DEFAULT_DEPS = "1"
> +
> +SRC_URI += "file://test.awk"
> +
> +EXCLUDE_FROM_WORLD = "1"
> +do_install() {
> +    install -d ${D}${bindir}
> +    install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
> +    sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
> +    create_cmdline_shebang_wrapper ${D}${bindir}/test
> +    if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
> +        bbfatal "Wrapper is broken"
> +    else
> +        bbnote "Wrapper is good"
> +    fi
> +}
> +
> +BBCLASSEXTEND = "native"
> diff --git a/meta-selftest/recipes-test/wrapper/files/test.awk b/meta-selftest/recipes-test/wrapper/files/test.awk
> new file mode 100644
> index 0000000000..91429197b1
> --- /dev/null
> +++ b/meta-selftest/recipes-test/wrapper/files/test.awk
> @@ -0,0 +1,2 @@
> +#! @AWK_BIN@ -f
> +BEGIN { print "Don't Panic!" }
> diff --git a/meta/classes/utils.bbclass b/meta/classes/utils.bbclass
> index b4eb3d38ab..b58c22771f 100644
> --- a/meta/classes/utils.bbclass
> +++ b/meta/classes/utils.bbclass
> @@ -184,6 +184,40 @@ END
>   	chmod +x $cmd
>   }
>   
> +create_cmdline_shebang_wrapper () {
> +	# Create a wrapper script where commandline options are needed
> +	#
> +	# These are useful to work around shebang relocation issues, where shebangs are too
> +	# long or have arguments in them, thus preventing them from using the /usr/bin/env
> +	# shebang
> +	#
> +	# Usage: create_cmdline_wrapper FILENAME <extra-options>
> +
> +	cmd=$1
> +	shift
> +
> +	echo "Generating wrapper script for $cmd"
> +
> +	# Strip #! and get remaining interpreter + arg
> +	argument="$(sed -ne 's/^#! *//p;q' $cmd)"
> +	# strip the shebang from the real script as we do not want it to be usable anyway
> +	tail -n +2 $cmd > $cmd.real
> +	cmdname=$(basename $cmd)
> +	dirname=$(dirname $cmd)
> +	cmdoptions=$@
> +	if [ "${base_prefix}" != "" ]; then
> +		relpath=`python3 -c "import os; print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
> +		cmdoptions=`echo $@ | sed -e "s:${base_prefix}:\\$realdir/$relpath:g"`
> +	fi
> +	cat <<END >$cmd
> +#!/usr/bin/env bash
> +realpath=\`readlink -fn \$0\`
> +realdir=\`dirname \$realpath\`
> +exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real $cmdoptions "\$@"
> +END
> +	chmod +x $cmd
> +}
> +

Maybe this has already been raised before, but the wrapper above does a 
couple of addtitional forks, which could be avoided if we use a python 
wrapper script instead.

Jacob

>   create_wrapper () {
>   	# Create a wrapper script where extra environment variables are needed
>   	#
> diff --git a/meta/lib/oeqa/selftest/cases/wrapper.py b/meta/lib/oeqa/selftest/cases/wrapper.py
> new file mode 100644
> index 0000000000..6de63310c0
> --- /dev/null
> +++ b/meta/lib/oeqa/selftest/cases/wrapper.py
> @@ -0,0 +1,11 @@
> +from oeqa.selftest.case import OESelftestTestCase
> +from oeqa.utils.commands import bitbake
> +
> +class WrapperTests(OESelftestTestCase):
> +    def test_shebang_wrapper(self):
> +        """
> +        Summary:   Build a recipe which will fail if the cmdline_shebang_wrapper function is defective.
> +        Expected:  Exit status to be 0.
> +        Author:    Paulo Neves <ptsneves@gmail.com>
> +        """
> +        res = bitbake("cmdline-shebang-wrapper-test -c install", ignore_status=False)
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#167044): https://lists.openembedded.org/g/openembedded-core/message/167044
> Mute This Topic: https://lists.openembedded.org/mt/91863579/4454410
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [jacob.kroon@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>