[PATCH 0/3] RPM bug fixes...

[PATCH 0/3] RPM bug fixes...

[Mark Hatle]

This set fixes a few problems that I recently noticed with RPM5.

The first item is a pure bug fix, the second and third are either bugs or an 
enhancements depending on how you look at it.

To test the first bug, you will need to enable 'lua' in the rpm recipe's
PACKAGECONFIG.  You can then verify behavor by running:

rpm --eval "%{_defaultdocdir} %{lua:print 'after'}"

If 'after' appears BEFORE the defaultdocdir value, then the bug is not fixed.

For the second/third items, if you install the necessary runtime environment
to use rpmbuild, you can attempt to build various spec files that use the
updated variables and verify the usage was correct.  (Not all spec files use
__cc, __cxx, etc.. so you may have to hand craft them.)

Mark Hatle (3):
  rpm: Fix lua 'print' statement capture
  rpm: Rebrand rpm custom macro paths to be distro specific
  rpm: Generate per distribution and multilib macro files

 .../rpm/rpm/rpm-lua-fix-print.patch                | 104 +++++++++++++++++++++
 meta/recipes-devtools/rpm/rpm_5.4.14.bb            |  87 ++++++++++++++++-
 2 files changed, 188 insertions(+), 3 deletions(-)
 create mode 100644 meta/recipes-devtools/rpm/rpm/rpm-lua-fix-print.patch

-- 
1.9.3

[PATCH 1/3] rpm: Fix lua 'print' statement capture

[Mark Hatle]

The print statement should capture the output and send it to the script
processing engine, and not display it directly to the screen.

Note, this is only a bug if 'lua' support has been enabled in the RPM
recipe's PACKAGECONFIG.

This patch is from: http://rpm5.org/cvs/patchset?cn=17671

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 .../rpm/rpm/rpm-lua-fix-print.patch                | 104 +++++++++++++++++++++
 meta/recipes-devtools/rpm/rpm_5.4.14.bb            |   1 +
 2 files changed, 105 insertions(+)
 create mode 100644 meta/recipes-devtools/rpm/rpm/rpm-lua-fix-print.patch

diff --git a/meta/recipes-devtools/rpm/rpm/rpm-lua-fix-print.patch b/meta/recipes-devtools/rpm/rpm/rpm-lua-fix-print.patch
new file mode 100644
index 0000000..7ab49e9
--- /dev/null
+++ b/meta/recipes-devtools/rpm/rpm/rpm-lua-fix-print.patch
@@ -0,0 +1,104 @@
+Lua 'print' statement is not working properly inside of RPM 5
+
+The print statement should capture the output and send it to the script
+processing engine, and not display it directly to the screen.
+
+This patch is from: http://rpm5.org/cvs/patchset?cn=17671
+
+Upstream-Status: backport (patchset 17671 from rpm5.org)
+
+Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
+
+Index: rpm-5.4.14/CHANGES
+===================================================================
+--- rpm-5.4.14.orig/CHANGES
++++ rpm-5.4.14/CHANGES
+@@ -1,3 +1,4 @@
++    - jbj: lua: fix: resurrect output capture with lua-5.2.
+     - jbj: verify: fix: broken logic for %ghost avoidance (Mark Hatle).
+ 
+ 5.4.13 -> 5.4.14:
+Index: rpm-5.4.14/rpmio/rpmlua.c
+===================================================================
+--- rpm-5.4.14.orig/rpmio/rpmlua.c
++++ rpm-5.4.14/rpmio/rpmlua.c
+@@ -175,7 +175,7 @@ rpmlua rpmluaNew(void)
+     };
+     /*@=readonlytrans =nullassign @*/
+     /*@observer@*/ /*@unchecked@*/
+-    const luaL_Reg *lib = lualibs;
++    const luaL_Reg *lib;
+     char *path_buf;
+     char *path_next;
+     char *path;
+@@ -190,31 +190,34 @@ rpmlua rpmluaNew(void)
+ 
+     luaL_openlibs(L);
+ 
+-    for (; lib->name; lib++) {
++    for (lib = lualibs; lib->name; lib++) {
+ 	luaL_requiref(L, lib->name, lib->func, 1);
++	lua_pop(L, 1);
+     }
+ 
+     {	const char * _lua_path = rpmGetPath(rpmluaPath, NULL);
+  	if (_lua_path != NULL) {
++#if defined(LUA_GLOBALSINDEX)
+ 	    lua_pushliteral(L, "LUA_PATH");
+ 	    lua_pushstring(L, _lua_path);
++	    lua_rawset(L, LUA_GLOBALSINDEX);
++#else
++	    lua_pushstring(L, _lua_path);
++	    lua_setglobal(L, "LUA_PATH");
++#endif
+ 	    _lua_path = _free(_lua_path);
+ 	}
+     }
+ 
+ #if defined(LUA_GLOBALSINDEX)
+-    lua_rawset(L, LUA_GLOBALSINDEX);
+-#else
+-    lua_pushglobaltable(L);
+-#endif
+     lua_pushliteral(L, "print");
+     lua_pushcfunction(L, rpm_print);
+-
+-#if defined(LUA_GLOBALSINDEX)
+     lua_rawset(L, LUA_GLOBALSINDEX);
+ #else
+-    lua_pushglobaltable(L);
++    lua_pushcfunction(L, rpm_print);
++    lua_setglobal(L, "print");
+ #endif
++
+     rpmluaSetData(lua, "lua", lua);
+ 
+     /* load all standard RPM Lua script files */
+@@ -351,6 +354,9 @@ void rpmluaSetVar(rpmlua _lua, rpmluav v
+ #if defined(LUA_GLOBALSINDEX)
+ 	if (lua->pushsize == 0)
+ 	    lua_pushvalue(L, LUA_GLOBALSINDEX);
++#else
++	if (lua->pushsize == 0)
++	    lua_pushglobaltable(L);
+ #endif
+ 	if (pushvar(L, var->keyType, &var->key) != -1) {
+ 	    if (pushvar(L, var->valueType, &var->value) != -1)
+@@ -1039,14 +1045,15 @@ static int rpm_print (lua_State *L)
+     lua_getglobal(L, "tostring");
+     for (i = 1; i <= n; i++) {
+ 	const char *s;
++	size_t l;
+ 	lua_pushvalue(L, -1);  /* function to be called */
+ 	lua_pushvalue(L, i);   /* value to print */
+ 	lua_call(L, 1, 1);
+-	s = lua_tostring(L, -1);  /* get result */
++	s = lua_tolstring(L, -1, &l);  /* get result */
+ 	if (s == NULL)
+ 	    return luaL_error(L, "`tostring' must return a string to `print'");
+ 	if (lua->storeprint) {
+-	    size_t sl = lua_rawlen(L, -1);
++	    size_t sl = l;
+ 	    if ((size_t)(lua->printbufused+sl+1) > lua->printbufsize) {
+ 		lua->printbufsize += sl+512;
+ 		lua->printbuf = (char *) xrealloc(lua->printbuf, lua->printbufsize);
diff --git a/meta/recipes-devtools/rpm/rpm_5.4.14.bb b/meta/recipes-devtools/rpm/rpm_5.4.14.bb
index 75b1ae2..eac3b8f 100644
--- a/meta/recipes-devtools/rpm/rpm_5.4.14.bb
+++ b/meta/recipes-devtools/rpm/rpm_5.4.14.bb
@@ -92,6 +92,7 @@ SRC_URI = "http://www.rpm5.org/files/rpm/rpm-5.4/rpm-5.4.14-0.20131024.src.rpm;e
 	   file://rpm-realpath.patch \
 	   file://0001-using-poptParseArgvString-to-parse-the-_gpg_check_pa.patch \
 	   file://no-ldflags-in-pkgconfig.patch \
+	   file://rpm-lua-fix-print.patch \
 	  "
 
 # Uncomment the following line to enable platform score debugging
-- 
1.9.3

[PATCH 2/3] rpm: Rebrand rpm custom macro paths to be distro specific

[Mark Hatle]

RPM was previously hard coded to use the 'poky' directory.  We really want
the directory name to be distribution specific.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 meta/recipes-devtools/rpm/rpm_5.4.14.bb | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-devtools/rpm/rpm_5.4.14.bb b/meta/recipes-devtools/rpm/rpm_5.4.14.bb
index eac3b8f..fb81f12 100644
--- a/meta/recipes-devtools/rpm/rpm_5.4.14.bb
+++ b/meta/recipes-devtools/rpm/rpm_5.4.14.bb
@@ -108,9 +108,9 @@ inherit autotools gettext
 acpaths = "-I ${S}/db/dist/aclocal -I ${S}/db/dist/aclocal_java"
 
 # Specify the default rpm macros in terms of adjustable variables
-rpm_macros = "%{_usrlibrpm}/macros:%{_usrlibrpm}/poky/macros:%{_usrlibrpm}/poky/%{_target}/macros:%{_etcrpm}/macros.*:%{_etcrpm}/macros:%{_etcrpm}/%{_target}/macros:~/.oerpmmacros"
-rpm_macros_class-native = "%{_usrlibrpm}/macros:%{_usrlibrpm}/poky/macros:%{_usrlibrpm}/poky/%{_target}/macros:~/.oerpmmacros"
-rpm_macros_class-nativesdk = "%{_usrlibrpm}/macros:%{_usrlibrpm}/poky/macros:%{_usrlibrpm}/poky/%{_target}/macros:~/.oerpmmacros"
+rpm_macros = "%{_usrlibrpm}/macros:%{_usrlibrpm}/${DISTRO}/macros:%{_usrlibrpm}/${DISTRO}/%{_target}/macros:%{_etcrpm}/macros.*:%{_etcrpm}/macros:%{_etcrpm}/%{_target}/macros:~/.oerpmmacros"
+rpm_macros_class-native = "%{_usrlibrpm}/macros:%{_usrlibrpm}/${DISTRO}/macros:%{_usrlibrpm}/${DISTRO}/%{_target}/macros:~/.oerpmmacros"
+rpm_macros_class-nativesdk = "%{_usrlibrpm}/macros:%{_usrlibrpm}/${DISTRO}/macros:%{_usrlibrpm}/${DISTRO}/%{_target}/macros:~/.oerpmmacros"
 
 # sqlite lua tcl augeas nss gcrypt neon xz xar keyutils perl selinux
 
-- 
1.9.3

[PATCH 3/3] rpm: Generate per distribution and multilib macro files

[Mark Hatle]

The distribution and multilib macro files are required for rpmbuild, on the
target, to work properly.  These override the default settings from the
upstream macro files with the proper values for the OE configuration.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 meta/recipes-devtools/rpm/rpm_5.4.14.bb | 80 +++++++++++++++++++++++++++++++++
 1 file changed, 80 insertions(+)

diff --git a/meta/recipes-devtools/rpm/rpm_5.4.14.bb b/meta/recipes-devtools/rpm/rpm_5.4.14.bb
index fb81f12..35c86bd 100644
--- a/meta/recipes-devtools/rpm/rpm_5.4.14.bb
+++ b/meta/recipes-devtools/rpm/rpm_5.4.14.bb
@@ -107,6 +107,9 @@ inherit autotools gettext
 
 acpaths = "-I ${S}/db/dist/aclocal -I ${S}/db/dist/aclocal_java"
 
+# The local distribution macro directory
+distromacrodir = "${libdir}/rpm/${DISTRO}"
+
 # Specify the default rpm macros in terms of adjustable variables
 rpm_macros = "%{_usrlibrpm}/macros:%{_usrlibrpm}/${DISTRO}/macros:%{_usrlibrpm}/${DISTRO}/%{_target}/macros:%{_etcrpm}/macros.*:%{_etcrpm}/macros:%{_etcrpm}/%{_target}/macros:~/.oerpmmacros"
 rpm_macros_class-native = "%{_usrlibrpm}/macros:%{_usrlibrpm}/${DISTRO}/macros:%{_usrlibrpm}/${DISTRO}/%{_target}/macros:~/.oerpmmacros"
@@ -331,6 +334,7 @@ FILES_${PN}-build = "${prefix}/src/rpm \
 		${libdir}/rpm/vpkg-provides.sh \
 		${libdir}/rpm/vpkg-provides2.sh \
 		${libdir}/rpm/perfile_rpmdeps.sh \
+		${distromacrodir} \
 		"
 RDEPENDS_${PN} = "base-files run-postinsts"
 RDEPENDS_${PN}_class-native = ""
@@ -472,6 +476,82 @@ do_install_append() {
 
 }
 
+do_install_append_class-target() {
+	# Create and install distribution specific macros
+	mkdir -p ${D}/${distromacrodir}
+	cat << EOF > ${D}/${distromacrodir}/macros
+%_defaultdocdir		${docdir}
+
+%_prefix                ${prefix}
+%_exec_prefix           ${exec_prefix}
+%_datarootdir           ${datadir}
+%_bindir                ${bindir}
+%_sbindir               ${sbindir}
+%_libexecdir            %{_libdir}/%{name}
+%_datadir               ${datadir}
+%_sysconfdir            ${sysconfdir}
+%_sharedstatedir        ${sharedstatedir}
+%_localstatedir         ${localstatedir}
+%_lib                   lib
+%_libdir                %{_exec_prefix}/%{_lib}
+%_includedir            ${includedir}
+%_oldincludedir         ${oldincludedir}
+%_infodir               ${infodir}
+%_mandir                ${mandir}
+%_localedir             %{_libdir}/locale
+EOF
+
+	# Create and install multilib specific macros
+	${@multilib_rpmmacros(d)}
+}
+
+def multilib_rpmmacros(d):
+    localdata = d.createCopy()
+    # We need to clear the TOOLCHAIN_OPTIONS (--sysroot)
+    localdata.delVar('TOOLCHAIN_OPTIONS')
+
+    # Set it to the consistent of 'd'.
+    localdata.setVar('distromacrodir', d.getVar('distromacrodir', True))
+    localdata.setVar('WORKDIR', d.getVar('WORKDIR', True))
+
+    ret = gen_arch_macro(localdata)
+
+    variants = d.getVar("MULTILIB_VARIANTS", True) or ""
+    for item in variants.split():
+        # Load overrides from 'd' to avoid having to reset the value...
+        overrides = d.getVar("OVERRIDES", False) + ":virtclass-multilib-" + item
+        localdata.setVar("OVERRIDES", overrides)
+        localdata.setVar("MLPREFIX", item + "-")
+        bb.data.update_data(localdata)
+        ret += gen_arch_macro(localdata)
+    return ret
+
+def gen_arch_macro(d):
+    # Generate shell script to produce the file as part of do_install
+    val  = "mkdir -p ${D}/${distromacrodir}/${TARGET_ARCH}-${TARGET_OS}\n"
+    val += "cat << EOF > ${D}/${distromacrodir}/${TARGET_ARCH}-${TARGET_OS}/macros\n"
+    val += "%_lib               ${baselib}\n"
+    val += "%_libdir            ${libdir}\n"
+    val += "%_localedir         ${localedir}\n"
+    val += "\n"
+    val += "# Toolchain configuration\n"
+    val += "%TOOLCHAIN_OPTIONS  %{nil}\n"
+    val += "%__ar               ${@d.getVar('AR', True).replace('$','%')}\n"
+    val += "%__as               ${@d.getVar('AS', True).replace('$','%')}\n"
+    val += "%__cc               ${@d.getVar('CC', True).replace('$','%')}\n"
+    val += "%__cpp              ${@d.getVar('CPP', True).replace('$','%')}\n"
+    val += "%__cxx              ${@d.getVar('CXX', True).replace('$','%')}\n"
+    val += "%__ld               ${@d.getVar('LD', True).replace('$','%')}\n"
+    val += "%__nm               ${@d.getVar('NM', True).replace('$','%')}\n"
+    val += "%__objcopy          ${@d.getVar('OBJCOPY', True).replace('$','%')}\n"
+    val += "%__objdump          ${@d.getVar('OBJDUMP', True).replace('$','%')}\n"
+    val += "%__ranlib           ${@d.getVar('RANLIB', True).replace('$','%')}\n"
+    val += "%__strip            ${@d.getVar('STRIP', True).replace('$','%')}\n"
+    val += "EOF\n"
+    val += "\n"
+    return d.expand(val)
+
+
 add_native_wrapper() {
         create_wrapper ${D}/${bindir}/rpm \
 		RPM_USRLIBRPM='`dirname $''realpath`'/${@os.path.relpath(d.getVar('libdir', True), d.getVar('bindir', True))}/rpm \
-- 
1.9.3

Re: [PATCH 3/3] rpm: Generate per distribution and multilib macro files

[Richard Purdie]

On Wed, 2015-06-17 at 12:04 -0500, Mark Hatle wrote:
> The distribution and multilib macro files are required for rpmbuild, on the
> target, to work properly.  These override the default settings from the
> upstream macro files with the proper values for the OE configuration.
> 
> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
[...]
> +def multilib_rpmmacros(d):
> +    localdata = d.createCopy()
> +    # We need to clear the TOOLCHAIN_OPTIONS (--sysroot)
> +    localdata.delVar('TOOLCHAIN_OPTIONS')
> +
> +    # Set it to the consistent of 'd'.
> +    localdata.setVar('distromacrodir', d.getVar('distromacrodir', True))
> +    localdata.setVar('WORKDIR', d.getVar('WORKDIR', True))
> +
> +    ret = gen_arch_macro(localdata)
> +
> +    variants = d.getVar("MULTILIB_VARIANTS", True) or ""
> +    for item in variants.split():
> +        # Load overrides from 'd' to avoid having to reset the value...
> +        overrides = d.getVar("OVERRIDES", False) + ":virtclass-multilib-" + item
> +        localdata.setVar("OVERRIDES", overrides)
> +        localdata.setVar("MLPREFIX", item + "-")
> +        bb.data.update_data(localdata)
> +        ret += gen_arch_macro(localdata)
> +    return ret

This doesn't look right. Usually there should be a loop of the form:

for x in y:
    localdata = d.createCopy()
    overrides = d.getVar("OVERRIDES", False) + f(x)
    localdata.setVar("OVERRIDES", overrides)
    bb.data.update_data(localdata)

where overrides is modified in the copy. If you don't do this, the
overrides will continue to be applied in the datastore with the way the
datastore currently works. This may happen to work but won't be correct.

As it happens, the datastore changes I've proposed will handle this
better but right now, the code above doesn't do what I think you'd want
it too. If its been copied from somewhere, that source also likely needs
fixing.

Cheers,

Richard