[PATCH] package_rpm: Fix useradd preinst ordering issues

[PATCH] package_rpm: Fix useradd preinst ordering issues

[Richard Purdie]

We were already having occasional ordering issues with package_rpm.
Fixing the ldconfig postinstall issue pushed rpm over the cliff and
totally broke rpm builds with the packages getting installed in
effectively a random order and the useradd preinstalls getting executed
out of order and breaking.

The only explanation I can find for this is that rpm is special. It will
happily run a preinst for a package without any of that package's
dependencies being present regardless of whether there are any circular
dependency issues or not. I attempted various ways of solving this such
as ordering the total_solution.manifest in creative ways but the bottom
line is RPM ignores this. It takes little account of any request to
ignore /bin/sh dependencies for the purposes of constructing the final
image.

The end result is we're having to install the base-passwd, base-files
and shadow packages first (if there is a request to install them), then
install any other packages.

It this wasn't in the middle of a release I'd be rewriting this bbclass
file, its horrible.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index 16a2c87..1b0f6f2 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -166,22 +167,23 @@ rpm_common_comand () {
 # install or remove the pkg
 rpm_update_pkg () {
 
+    manifest=$1
+    btmanifest=$manifest.bt
     local target_rootfs="${INSTALL_ROOTFS_RPM}"
 
     # Save the rpm's build time for incremental image generation, and the file
     # would be moved to ${T}
-    rm -f ${target_rootfs}/install/total_solution_bt.manifest
-    for i in `cat ${target_rootfs}/install/total_solution.manifest`; do
+    rm -f $btmanifest
+    for i in `cat $manifest`; do
         # Use "rpm" rather than "${RPM}" here, since we don't need the
         # '--dbpath' option
-        echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`" >> \
-            ${target_rootfs}/install/total_solution_bt.manifest
+        echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`" >> $btmanifest
     done
 
     # Only install the different pkgs if incremental image generation is set
     if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f ${T}/total_solution_bt.manifest -a \
         "${IMAGE_PKGTYPE}" = "rpm" ]; then
-        cur_list="${target_rootfs}/install/total_solution_bt.manifest"
+        cur_list="$btmanifest"
         pre_list="${T}/total_solution_bt.manifest"
         sort -u $cur_list -o $cur_list
         sort -u $pre_list -o $pre_list
@@ -203,8 +205,7 @@ rpm_update_pkg () {
             -Uvh ${target_rootfs}/install/incremental.manifest
     else
         # Attempt to install
-        rpm_common_comand --replacepkgs \
-            -Uhv ${target_rootfs}/install/total_solution.manifest
+        rpm_common_comand --replacepkgs -Uhv $manifest
     fi
 }
 
@@ -440,14 +441,7 @@ package_install_internal_rpm () {
 
 	fi
 
-	# If base-passwd or shadow are in the list of packages to install,
-	# ensure they are installed first to support later packages that
-	# may create custom users/groups (fixes Yocto bug #2127)
-	infile=${target_rootfs}/install/install_solution.manifest
-	outfile=${target_rootfs}/install/total_solution.manifest
-	cat $infile | grep /base-passwd-[0-9] > $outfile || true
-	cat $infile | grep /shadow-[0-9] >> $outfile || true
-	cat $infile | grep -v /shadow-[0-9] | grep -v /base-passwd-[0-9] >> $outfile || true
+	cat ${target_rootfs}/install/install_solution.manifest > ${target_rootfs}/install/total_solution.manifest
 	cat ${target_rootfs}/install/install_multilib_solution.manifest >> ${target_rootfs}/install/total_solution.manifest
 
 	# Construct install scriptlet wrapper
@@ -474,8 +468,45 @@ EOF
 
 	chmod 0755 ${WORKDIR}/scriptlet_wrapper
 
-    rpm_update_pkg
+	# RPM is special. It can't handle dependencies and preinstall scripts correctly. Its
+	# probably a feature. The only way to convince rpm to actually run the preinstall scripts 
+	# for base-passwd and shadow first before installing packages that depend on these packages 
+	# is to do two image installs, installing one set of packages, then the other.
+	if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f ${T}/total_solution_bt.manifest ]; then
+		echo "Skipping pre install due to exisitng image"
+	else
+		echo "# Intial Install manifest" > ${target_rootfs}/install/initial_install.manifest
+		echo "Installing base dependencies first (base-passwd, base-files and shadow) since rpm is special"
+		grep /base-passwd-[0-9] ${target_rootfs}/install/total_solution.manifest >> ${target_rootfs}/install/initial_install.manifest || true
+		grep /base-files-[0-9] ${target_rootfs}/install/total_solution.manifest >> ${target_rootfs}/install/initial_install.manifest || true		
+		grep /shadow-[0-9] ${target_rootfs}/install/total_solution.manifest >> ${target_rootfs}/install/initial_install.manifest || true
+
+		# Generate an install solution by doing a --justdb install, then recreate it with
+		# an actual package install!
+		mkdir -p ${target_rootfs}/initial
+
+		${RPM} --predefine "_rpmds_sysinfo_path ${target_rootfs}/etc/rpm/sysinfo" \
+			--predefine "_rpmrc_platform_path ${target_rootfs}/etc/rpm/platform" \
+			-D "_dbpath ${target_rootfs}/initial" -D "`cat ${confbase}-base_archs.macro`" \
+			-D "__dbi_txn create nofsync" \
+			-U --justdb --noscripts --notriggers --noparentdirs --nolinktos --ignoresize \
+			${target_rootfs}/install/initial_install.manifest
+
+		${RPM} -D "_dbpath ${target_rootfs}/initial" -qa --yaml \
+			-D "__dbi_txn create nofsync private" \
+			| grep -i 'Packageorigin' | cut -d : -f 2 > ${target_rootfs}/install/initial_solution.manifest
+
+		rpm_update_pkg ${target_rootfs}/install/initial_solution.manifest
+		
+		grep -Fv -f ${target_rootfs}/install/initial_solution.manifest ${target_rootfs}/install/total_solution.manifest > ${target_rootfs}/install/total_solution.manifest.new
+		mv ${target_rootfs}/install/total_solution.manifest.new ${target_rootfs}/install/total_solution.manifest
+		
+		rm -rf ${target_rootfs}/initial
+	fi
+
+	echo "Installing main solution manifest (${target_rootfs}/install/total_solution.manifest)"
 
+	rpm_update_pkg ${target_rootfs}/install/total_solution.manifest
 }
 
 python write_specfile () {

Re: [PATCH] package_rpm: Fix useradd preinst ordering issues

[Mark Hatle]

On 4/11/12 4:31 PM, Richard Purdie wrote:
> We were already having occasional ordering issues with package_rpm.
> Fixing the ldconfig postinstall issue pushed rpm over the cliff and
> totally broke rpm builds with the packages getting installed in
> effectively a random order and the useradd preinstalls getting executed
> out of order and breaking.
>
> The only explanation I can find for this is that rpm is special. It will
> happily run a preinst for a package without any of that package's
> dependencies being present regardless of whether there are any circular
> dependency issues or not. I attempted various ways of solving this such
> as ordering the total_solution.manifest in creative ways but the bottom
> line is RPM ignores this. It takes little account of any request to
> ignore /bin/sh dependencies for the purposes of constructing the final
> image.
>
> The end result is we're having to install the base-passwd, base-files
> and shadow packages first (if there is a request to install them), then
> install any other packages.
>
> It this wasn't in the middle of a release I'd be rewriting this bbclass
> file, its horrible.
>
> Signed-off-by: Richard Purdie<richard.purdie@linuxfoundation.org>

I've reviewed this.  I don't see anything wrong with this, other then the pain 
of having to do this.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>

--Mark

>
> diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
> index 16a2c87..1b0f6f2 100644
> --- a/meta/classes/package_rpm.bbclass
> +++ b/meta/classes/package_rpm.bbclass
> @@ -166,22 +167,23 @@ rpm_common_comand () {
>   # install or remove the pkg
>   rpm_update_pkg () {
>
> +    manifest=$1
> +    btmanifest=$manifest.bt
>       local target_rootfs="${INSTALL_ROOTFS_RPM}"
>
>       # Save the rpm's build time for incremental image generation, and the file
>       # would be moved to ${T}
> -    rm -f ${target_rootfs}/install/total_solution_bt.manifest
> -    for i in `cat ${target_rootfs}/install/total_solution.manifest`; do
> +    rm -f $btmanifest
> +    for i in `cat $manifest`; do
>           # Use "rpm" rather than "${RPM}" here, since we don't need the
>           # '--dbpath' option
> -        echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`">>  \
> -            ${target_rootfs}/install/total_solution_bt.manifest
> +        echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`">>  $btmanifest
>       done
>
>       # Only install the different pkgs if incremental image generation is set
>       if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f ${T}/total_solution_bt.manifest -a \
>           "${IMAGE_PKGTYPE}" = "rpm" ]; then
> -        cur_list="${target_rootfs}/install/total_solution_bt.manifest"
> +        cur_list="$btmanifest"
>           pre_list="${T}/total_solution_bt.manifest"
>           sort -u $cur_list -o $cur_list
>           sort -u $pre_list -o $pre_list
> @@ -203,8 +205,7 @@ rpm_update_pkg () {
>               -Uvh ${target_rootfs}/install/incremental.manifest
>       else
>           # Attempt to install
> -        rpm_common_comand --replacepkgs \
> -            -Uhv ${target_rootfs}/install/total_solution.manifest
> +        rpm_common_comand --replacepkgs -Uhv $manifest
>       fi
>   }
>
> @@ -440,14 +441,7 @@ package_install_internal_rpm () {
>
>   	fi
>
> -	# If base-passwd or shadow are in the list of packages to install,
> -	# ensure they are installed first to support later packages that
> -	# may create custom users/groups (fixes Yocto bug #2127)
> -	infile=${target_rootfs}/install/install_solution.manifest
> -	outfile=${target_rootfs}/install/total_solution.manifest
> -	cat $infile | grep /base-passwd-[0-9]>  $outfile || true
> -	cat $infile | grep /shadow-[0-9]>>  $outfile || true
> -	cat $infile | grep -v /shadow-[0-9] | grep -v /base-passwd-[0-9]>>  $outfile || true
> +	cat ${target_rootfs}/install/install_solution.manifest>  ${target_rootfs}/install/total_solution.manifest
>   	cat ${target_rootfs}/install/install_multilib_solution.manifest>>  ${target_rootfs}/install/total_solution.manifest
>
>   	# Construct install scriptlet wrapper
> @@ -474,8 +468,45 @@ EOF
>
>   	chmod 0755 ${WORKDIR}/scriptlet_wrapper
>
> -    rpm_update_pkg
> +	# RPM is special. It can't handle dependencies and preinstall scripts correctly. Its
> +	# probably a feature. The only way to convince rpm to actually run the preinstall scripts
> +	# for base-passwd and shadow first before installing packages that depend on these packages
> +	# is to do two image installs, installing one set of packages, then the other.
> +	if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f ${T}/total_solution_bt.manifest ]; then
> +		echo "Skipping pre install due to exisitng image"
> +	else
> +		echo "# Intial Install manifest">  ${target_rootfs}/install/initial_install.manifest
> +		echo "Installing base dependencies first (base-passwd, base-files and shadow) since rpm is special"
> +		grep /base-passwd-[0-9] ${target_rootfs}/install/total_solution.manifest>>  ${target_rootfs}/install/initial_install.manifest || true
> +		grep /base-files-[0-9] ${target_rootfs}/install/total_solution.manifest>>  ${target_rootfs}/install/initial_install.manifest || true		
> +		grep /shadow-[0-9] ${target_rootfs}/install/total_solution.manifest>>  ${target_rootfs}/install/initial_install.manifest || true
> +
> +		# Generate an install solution by doing a --justdb install, then recreate it with
> +		# an actual package install!
> +		mkdir -p ${target_rootfs}/initial
> +
> +		${RPM} --predefine "_rpmds_sysinfo_path ${target_rootfs}/etc/rpm/sysinfo" \
> +			--predefine "_rpmrc_platform_path ${target_rootfs}/etc/rpm/platform" \
> +			-D "_dbpath ${target_rootfs}/initial" -D "`cat ${confbase}-base_archs.macro`" \
> +			-D "__dbi_txn create nofsync" \
> +			-U --justdb --noscripts --notriggers --noparentdirs --nolinktos --ignoresize \
> +			${target_rootfs}/install/initial_install.manifest
> +
> +		${RPM} -D "_dbpath ${target_rootfs}/initial" -qa --yaml \
> +			-D "__dbi_txn create nofsync private" \
> +			| grep -i 'Packageorigin' | cut -d : -f 2>  ${target_rootfs}/install/initial_solution.manifest
> +
> +		rpm_update_pkg ${target_rootfs}/install/initial_solution.manifest
> +		
> +		grep -Fv -f ${target_rootfs}/install/initial_solution.manifest ${target_rootfs}/install/total_solution.manifest>  ${target_rootfs}/install/total_solution.manifest.new
> +		mv ${target_rootfs}/install/total_solution.manifest.new ${target_rootfs}/install/total_solution.manifest
> +		
> +		rm -rf ${target_rootfs}/initial
> +	fi
> +
> +	echo "Installing main solution manifest (${target_rootfs}/install/total_solution.manifest)"
>
> +	rpm_update_pkg ${target_rootfs}/install/total_solution.manifest
>   }
>
>   python write_specfile () {
>

Re: [PATCH] package_rpm: Fix useradd preinst ordering issues

[Steve Sakoman]

On Wed, Apr 11, 2012 at 2:42 PM, Mark Hatle <mark.hatle@windriver.com> wrote:
> On 4/11/12 4:31 PM, Richard Purdie wrote:
>>
>> We were already having occasional ordering issues with package_rpm.
>> Fixing the ldconfig postinstall issue pushed rpm over the cliff and
>> totally broke rpm builds with the packages getting installed in
>> effectively a random order and the useradd preinstalls getting executed
>> out of order and breaking.
>>
>> The only explanation I can find for this is that rpm is special. It will
>> happily run a preinst for a package without any of that package's
>> dependencies being present regardless of whether there are any circular
>> dependency issues or not. I attempted various ways of solving this such
>> as ordering the total_solution.manifest in creative ways but the bottom
>> line is RPM ignores this. It takes little account of any request to
>> ignore /bin/sh dependencies for the purposes of constructing the final
>> image.
>>
>> The end result is we're having to install the base-passwd, base-files
>> and shadow packages first (if there is a request to install them), then
>> install any other packages.
>>
>> It this wasn't in the middle of a release I'd be rewriting this bbclass
>> file, its horrible.
>>
>> Signed-off-by: Richard Purdie<richard.purdie@linuxfoundation.org>
>
>
> I've reviewed this.  I don't see anything wrong with this, other then the
> pain of having to do this.
>
> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>

FWIW, after pulling current poky this morning all of my image builds
are failing with errors like this:

| Installing base dependencies first (base-passwd, base-files and
shadow) since rpm is special
| error: Failed dependencies:
| 	/bin/sh is needed by base-passwd-3.5.24-r0.armv7a
| 	/bin/sh is needed by shadow-4.1.4.3-r8.armv7a
| 	/bin/sh is needed by libgcc1-4.6.3+svnr184847-r24.armv7a
| 	/bin/sh is needed by libc6-2.13-r23+svnr15508.armv7a

I'll investigate, but since it is late in the release process thought
I would mention the issue.

Steve



>> diff --git a/meta/classes/package_rpm.bbclass
>> b/meta/classes/package_rpm.bbclass
>> index 16a2c87..1b0f6f2 100644
>> --- a/meta/classes/package_rpm.bbclass
>> +++ b/meta/classes/package_rpm.bbclass
>> @@ -166,22 +167,23 @@ rpm_common_comand () {
>>  # install or remove the pkg
>>  rpm_update_pkg () {
>>
>> +    manifest=$1
>> +    btmanifest=$manifest.bt
>>      local target_rootfs="${INSTALL_ROOTFS_RPM}"
>>
>>      # Save the rpm's build time for incremental image generation, and the
>> file
>>      # would be moved to ${T}
>> -    rm -f ${target_rootfs}/install/total_solution_bt.manifest
>> -    for i in `cat ${target_rootfs}/install/total_solution.manifest`; do
>> +    rm -f $btmanifest
>> +    for i in `cat $manifest`; do
>>          # Use "rpm" rather than "${RPM}" here, since we don't need the
>>          # '--dbpath' option
>> -        echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`">>  \
>> -            ${target_rootfs}/install/total_solution_bt.manifest
>> +        echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`">>  $btmanifest
>>      done
>>
>>      # Only install the different pkgs if incremental image generation is
>> set
>>      if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f
>> ${T}/total_solution_bt.manifest -a \
>>          "${IMAGE_PKGTYPE}" = "rpm" ]; then
>> -        cur_list="${target_rootfs}/install/total_solution_bt.manifest"
>> +        cur_list="$btmanifest"
>>          pre_list="${T}/total_solution_bt.manifest"
>>          sort -u $cur_list -o $cur_list
>>          sort -u $pre_list -o $pre_list
>> @@ -203,8 +205,7 @@ rpm_update_pkg () {
>>              -Uvh ${target_rootfs}/install/incremental.manifest
>>      else
>>          # Attempt to install
>> -        rpm_common_comand --replacepkgs \
>> -            -Uhv ${target_rootfs}/install/total_solution.manifest
>> +        rpm_common_comand --replacepkgs -Uhv $manifest
>>      fi
>>  }
>>
>> @@ -440,14 +441,7 @@ package_install_internal_rpm () {
>>
>>        fi
>>
>> -       # If base-passwd or shadow are in the list of packages to install,
>> -       # ensure they are installed first to support later packages that
>> -       # may create custom users/groups (fixes Yocto bug #2127)
>> -       infile=${target_rootfs}/install/install_solution.manifest
>> -       outfile=${target_rootfs}/install/total_solution.manifest
>> -       cat $infile | grep /base-passwd-[0-9]>  $outfile || true
>> -       cat $infile | grep /shadow-[0-9]>>  $outfile || true
>> -       cat $infile | grep -v /shadow-[0-9] | grep -v /base-passwd-[0-9]>>
>>  $outfile || true
>> +       cat ${target_rootfs}/install/install_solution.manifest>
>>  ${target_rootfs}/install/total_solution.manifest
>>        cat ${target_rootfs}/install/install_multilib_solution.manifest>>
>>  ${target_rootfs}/install/total_solution.manifest
>>
>>        # Construct install scriptlet wrapper
>> @@ -474,8 +468,45 @@ EOF
>>
>>        chmod 0755 ${WORKDIR}/scriptlet_wrapper
>>
>> -    rpm_update_pkg
>> +       # RPM is special. It can't handle dependencies and preinstall
>> scripts correctly. Its
>> +       # probably a feature. The only way to convince rpm to actually run
>> the preinstall scripts
>> +       # for base-passwd and shadow first before installing packages that
>> depend on these packages
>> +       # is to do two image installs, installing one set of packages,
>> then the other.
>> +       if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f
>> ${T}/total_solution_bt.manifest ]; then
>> +               echo "Skipping pre install due to exisitng image"
>> +       else
>> +               echo "# Intial Install manifest">
>>  ${target_rootfs}/install/initial_install.manifest
>> +               echo "Installing base dependencies first (base-passwd,
>> base-files and shadow) since rpm is special"
>> +               grep /base-passwd-[0-9]
>> ${target_rootfs}/install/total_solution.manifest>>
>>  ${target_rootfs}/install/initial_install.manifest || true
>> +               grep /base-files-[0-9]
>> ${target_rootfs}/install/total_solution.manifest>>
>>  ${target_rootfs}/install/initial_install.manifest || true
>> +               grep /shadow-[0-9]
>> ${target_rootfs}/install/total_solution.manifest>>
>>  ${target_rootfs}/install/initial_install.manifest || true
>> +
>> +               # Generate an install solution by doing a --justdb
>> install, then recreate it with
>> +               # an actual package install!
>> +               mkdir -p ${target_rootfs}/initial
>> +
>> +               ${RPM} --predefine "_rpmds_sysinfo_path
>> ${target_rootfs}/etc/rpm/sysinfo" \
>> +                       --predefine "_rpmrc_platform_path
>> ${target_rootfs}/etc/rpm/platform" \
>> +                       -D "_dbpath ${target_rootfs}/initial" -D "`cat
>> ${confbase}-base_archs.macro`" \
>> +                       -D "__dbi_txn create nofsync" \
>> +                       -U --justdb --noscripts --notriggers
>> --noparentdirs --nolinktos --ignoresize \
>> +                       ${target_rootfs}/install/initial_install.manifest
>> +
>> +               ${RPM} -D "_dbpath ${target_rootfs}/initial" -qa --yaml \
>> +                       -D "__dbi_txn create nofsync private" \
>> +                       | grep -i 'Packageorigin' | cut -d : -f 2>
>>  ${target_rootfs}/install/initial_solution.manifest
>> +
>> +               rpm_update_pkg
>> ${target_rootfs}/install/initial_solution.manifest
>> +
>> +               grep -Fv -f
>> ${target_rootfs}/install/initial_solution.manifest
>> ${target_rootfs}/install/total_solution.manifest>
>>  ${target_rootfs}/install/total_solution.manifest.new
>> +               mv ${target_rootfs}/install/total_solution.manifest.new
>> ${target_rootfs}/install/total_solution.manifest
>> +
>> +               rm -rf ${target_rootfs}/initial
>> +       fi
>> +
>> +       echo "Installing main solution manifest
>> (${target_rootfs}/install/total_solution.manifest)"
>>
>> +       rpm_update_pkg ${target_rootfs}/install/total_solution.manifest
>>  }
>>
>>  python write_specfile () {
>>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [PATCH] package_rpm: Fix useradd preinst ordering issues

[Steve Sakoman]

On Thu, Apr 12, 2012 at 6:39 AM, Steve Sakoman <sakoman@gmail.com> wrote:

> FWIW, after pulling current poky this morning all of my image builds
> are failing with errors like this:
>
> | Installing base dependencies first (base-passwd, base-files and
> shadow) since rpm is special
> | error: Failed dependencies:
> |       /bin/sh is needed by base-passwd-3.5.24-r0.armv7a
> |       /bin/sh is needed by shadow-4.1.4.3-r8.armv7a
> |       /bin/sh is needed by libgcc1-4.6.3+svnr184847-r24.armv7a
> |       /bin/sh is needed by libc6-2.13-r23+svnr15508.armv7a
>
> I'll investigate, but since it is late in the release process thought
> I would mention the issue.

I have no idea if this is the proper fix, but adding bash to the list
of packages to install first fixed the issue for me:

diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index fd00fb1..a5f0b06 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -475,7 +475,8 @@ EOF
                echo "Skipping pre install due to exisitng image"
        else
                echo "# Initial Install manifest" >
${target_rootfs}/install/initial_install.manifest
-               echo "Installing base dependencies first (base-passwd,
base-files and shadow) since rpm is special"
+               echo "Installing base dependencies first (bash,
base-passwd, base-files and shadow) since rpm is special"
+               grep /bash-[0-9]
${target_rootfs}/install/total_solution.manifest >>
${target_rootfs}/install/initial_install.manife
                grep /base-passwd-[0-9]
${target_rootfs}/install/total_solution.manifest >>
${target_rootfs}/install/initial_install
                grep /base-files-[0-9]
${target_rootfs}/install/total_solution.manifest >>
${target_rootfs}/install/initial_install.
                grep /shadow-[0-9]
${target_rootfs}/install/total_solution.manifest >>
${target_rootfs}/install/initial_install.mani

I'm concerned we are on a slippery slope here!

Steve


>>> diff --git a/meta/classes/package_rpm.bbclass
>>> b/meta/classes/package_rpm.bbclass
>>> index 16a2c87..1b0f6f2 100644
>>> --- a/meta/classes/package_rpm.bbclass
>>> +++ b/meta/classes/package_rpm.bbclass
>>> @@ -166,22 +167,23 @@ rpm_common_comand () {
>>>  # install or remove the pkg
>>>  rpm_update_pkg () {
>>>
>>> +    manifest=$1
>>> +    btmanifest=$manifest.bt
>>>      local target_rootfs="${INSTALL_ROOTFS_RPM}"
>>>
>>>      # Save the rpm's build time for incremental image generation, and the
>>> file
>>>      # would be moved to ${T}
>>> -    rm -f ${target_rootfs}/install/total_solution_bt.manifest
>>> -    for i in `cat ${target_rootfs}/install/total_solution.manifest`; do
>>> +    rm -f $btmanifest
>>> +    for i in `cat $manifest`; do
>>>          # Use "rpm" rather than "${RPM}" here, since we don't need the
>>>          # '--dbpath' option
>>> -        echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`">>  \
>>> -            ${target_rootfs}/install/total_solution_bt.manifest
>>> +        echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`">>  $btmanifest
>>>      done
>>>
>>>      # Only install the different pkgs if incremental image generation is
>>> set
>>>      if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f
>>> ${T}/total_solution_bt.manifest -a \
>>>          "${IMAGE_PKGTYPE}" = "rpm" ]; then
>>> -        cur_list="${target_rootfs}/install/total_solution_bt.manifest"
>>> +        cur_list="$btmanifest"
>>>          pre_list="${T}/total_solution_bt.manifest"
>>>          sort -u $cur_list -o $cur_list
>>>          sort -u $pre_list -o $pre_list
>>> @@ -203,8 +205,7 @@ rpm_update_pkg () {
>>>              -Uvh ${target_rootfs}/install/incremental.manifest
>>>      else
>>>          # Attempt to install
>>> -        rpm_common_comand --replacepkgs \
>>> -            -Uhv ${target_rootfs}/install/total_solution.manifest
>>> +        rpm_common_comand --replacepkgs -Uhv $manifest
>>>      fi
>>>  }
>>>
>>> @@ -440,14 +441,7 @@ package_install_internal_rpm () {
>>>
>>>        fi
>>>
>>> -       # If base-passwd or shadow are in the list of packages to install,
>>> -       # ensure they are installed first to support later packages that
>>> -       # may create custom users/groups (fixes Yocto bug #2127)
>>> -       infile=${target_rootfs}/install/install_solution.manifest
>>> -       outfile=${target_rootfs}/install/total_solution.manifest
>>> -       cat $infile | grep /base-passwd-[0-9]>  $outfile || true
>>> -       cat $infile | grep /shadow-[0-9]>>  $outfile || true
>>> -       cat $infile | grep -v /shadow-[0-9] | grep -v /base-passwd-[0-9]>>
>>>  $outfile || true
>>> +       cat ${target_rootfs}/install/install_solution.manifest>
>>>  ${target_rootfs}/install/total_solution.manifest
>>>        cat ${target_rootfs}/install/install_multilib_solution.manifest>>
>>>  ${target_rootfs}/install/total_solution.manifest
>>>
>>>        # Construct install scriptlet wrapper
>>> @@ -474,8 +468,45 @@ EOF
>>>
>>>        chmod 0755 ${WORKDIR}/scriptlet_wrapper
>>>
>>> -    rpm_update_pkg
>>> +       # RPM is special. It can't handle dependencies and preinstall
>>> scripts correctly. Its
>>> +       # probably a feature. The only way to convince rpm to actually run
>>> the preinstall scripts
>>> +       # for base-passwd and shadow first before installing packages that
>>> depend on these packages
>>> +       # is to do two image installs, installing one set of packages,
>>> then the other.
>>> +       if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f
>>> ${T}/total_solution_bt.manifest ]; then
>>> +               echo "Skipping pre install due to exisitng image"
>>> +       else
>>> +               echo "# Intial Install manifest">
>>>  ${target_rootfs}/install/initial_install.manifest
>>> +               echo "Installing base dependencies first (base-passwd,
>>> base-files and shadow) since rpm is special"
>>> +               grep /base-passwd-[0-9]
>>> ${target_rootfs}/install/total_solution.manifest>>
>>>  ${target_rootfs}/install/initial_install.manifest || true
>>> +               grep /base-files-[0-9]
>>> ${target_rootfs}/install/total_solution.manifest>>
>>>  ${target_rootfs}/install/initial_install.manifest || true
>>> +               grep /shadow-[0-9]
>>> ${target_rootfs}/install/total_solution.manifest>>
>>>  ${target_rootfs}/install/initial_install.manifest || true
>>> +
>>> +               # Generate an install solution by doing a --justdb
>>> install, then recreate it with
>>> +               # an actual package install!
>>> +               mkdir -p ${target_rootfs}/initial
>>> +
>>> +               ${RPM} --predefine "_rpmds_sysinfo_path
>>> ${target_rootfs}/etc/rpm/sysinfo" \
>>> +                       --predefine "_rpmrc_platform_path
>>> ${target_rootfs}/etc/rpm/platform" \
>>> +                       -D "_dbpath ${target_rootfs}/initial" -D "`cat
>>> ${confbase}-base_archs.macro`" \
>>> +                       -D "__dbi_txn create nofsync" \
>>> +                       -U --justdb --noscripts --notriggers
>>> --noparentdirs --nolinktos --ignoresize \
>>> +                       ${target_rootfs}/install/initial_install.manifest
>>> +
>>> +               ${RPM} -D "_dbpath ${target_rootfs}/initial" -qa --yaml \
>>> +                       -D "__dbi_txn create nofsync private" \
>>> +                       | grep -i 'Packageorigin' | cut -d : -f 2>
>>>  ${target_rootfs}/install/initial_solution.manifest
>>> +
>>> +               rpm_update_pkg
>>> ${target_rootfs}/install/initial_solution.manifest
>>> +
>>> +               grep -Fv -f
>>> ${target_rootfs}/install/initial_solution.manifest
>>> ${target_rootfs}/install/total_solution.manifest>
>>>  ${target_rootfs}/install/total_solution.manifest.new
>>> +               mv ${target_rootfs}/install/total_solution.manifest.new
>>> ${target_rootfs}/install/total_solution.manifest
>>> +
>>> +               rm -rf ${target_rootfs}/initial
>>> +       fi
>>> +
>>> +       echo "Installing main solution manifest
>>> (${target_rootfs}/install/total_solution.manifest)"
>>>
>>> +       rpm_update_pkg ${target_rootfs}/install/total_solution.manifest
>>>  }
>>>
>>>  python write_specfile () {
>>>
>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [PATCH] package_rpm: Fix useradd preinst ordering issues

[Mark Hatle]

On 4/12/12 10:36 AM, Steve Sakoman wrote:
> On Thu, Apr 12, 2012 at 6:39 AM, Steve Sakoman<sakoman@gmail.com>  wrote:

That is very odd, the system is supposed to identify all of the dependencies 
needed for the first set of packages... (the three base-passwd, base-files and 
shadow).

The end result is a list of 10-12 binaries to be installed that meet those 
requirements, as well as the original three requested packages.

We also shouldn't specify bash because busybox provides /bin/sh in a lot of 
configurations.

Which image did you try to build and I'll see what I can replicate here.

--Mark

>> FWIW, after pulling current poky this morning all of my image builds
>> are failing with errors like this:
>>
>> | Installing base dependencies first (base-passwd, base-files and
>> shadow) since rpm is special
>> | error: Failed dependencies:
>> |       /bin/sh is needed by base-passwd-3.5.24-r0.armv7a
>> |       /bin/sh is needed by shadow-4.1.4.3-r8.armv7a
>> |       /bin/sh is needed by libgcc1-4.6.3+svnr184847-r24.armv7a
>> |       /bin/sh is needed by libc6-2.13-r23+svnr15508.armv7a
>>
>> I'll investigate, but since it is late in the release process thought
>> I would mention the issue.
>
> I have no idea if this is the proper fix, but adding bash to the list
> of packages to install first fixed the issue for me:
>
> diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
> index fd00fb1..a5f0b06 100644
> --- a/meta/classes/package_rpm.bbclass
> +++ b/meta/classes/package_rpm.bbclass
> @@ -475,7 +475,8 @@ EOF
>                  echo "Skipping pre install due to exisitng image"
>          else
>                  echo "# Initial Install manifest">
> ${target_rootfs}/install/initial_install.manifest
> -               echo "Installing base dependencies first (base-passwd,
> base-files and shadow) since rpm is special"
> +               echo "Installing base dependencies first (bash,
> base-passwd, base-files and shadow) since rpm is special"
> +               grep /bash-[0-9]
> ${target_rootfs}/install/total_solution.manifest>>
> ${target_rootfs}/install/initial_install.manife
>                  grep /base-passwd-[0-9]
> ${target_rootfs}/install/total_solution.manifest>>
> ${target_rootfs}/install/initial_install
>                  grep /base-files-[0-9]
> ${target_rootfs}/install/total_solution.manifest>>
> ${target_rootfs}/install/initial_install.
>                  grep /shadow-[0-9]
> ${target_rootfs}/install/total_solution.manifest>>
> ${target_rootfs}/install/initial_install.mani
>
> I'm concerned we are on a slippery slope here!
>
> Steve
>
>
>>>> diff --git a/meta/classes/package_rpm.bbclass
>>>> b/meta/classes/package_rpm.bbclass
>>>> index 16a2c87..1b0f6f2 100644
>>>> --- a/meta/classes/package_rpm.bbclass
>>>> +++ b/meta/classes/package_rpm.bbclass
>>>> @@ -166,22 +167,23 @@ rpm_common_comand () {
>>>>   # install or remove the pkg
>>>>   rpm_update_pkg () {
>>>>
>>>> +    manifest=$1
>>>> +    btmanifest=$manifest.bt
>>>>       local target_rootfs="${INSTALL_ROOTFS_RPM}"
>>>>
>>>>       # Save the rpm's build time for incremental image generation, and the
>>>> file
>>>>       # would be moved to ${T}
>>>> -    rm -f ${target_rootfs}/install/total_solution_bt.manifest
>>>> -    for i in `cat ${target_rootfs}/install/total_solution.manifest`; do
>>>> +    rm -f $btmanifest
>>>> +    for i in `cat $manifest`; do
>>>>           # Use "rpm" rather than "${RPM}" here, since we don't need the
>>>>           # '--dbpath' option
>>>> -        echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`">>    \
>>>> -            ${target_rootfs}/install/total_solution_bt.manifest
>>>> +        echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`">>    $btmanifest
>>>>       done
>>>>
>>>>       # Only install the different pkgs if incremental image generation is
>>>> set
>>>>       if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f
>>>> ${T}/total_solution_bt.manifest -a \
>>>>           "${IMAGE_PKGTYPE}" = "rpm" ]; then
>>>> -        cur_list="${target_rootfs}/install/total_solution_bt.manifest"
>>>> +        cur_list="$btmanifest"
>>>>           pre_list="${T}/total_solution_bt.manifest"
>>>>           sort -u $cur_list -o $cur_list
>>>>           sort -u $pre_list -o $pre_list
>>>> @@ -203,8 +205,7 @@ rpm_update_pkg () {
>>>>               -Uvh ${target_rootfs}/install/incremental.manifest
>>>>       else
>>>>           # Attempt to install
>>>> -        rpm_common_comand --replacepkgs \
>>>> -            -Uhv ${target_rootfs}/install/total_solution.manifest
>>>> +        rpm_common_comand --replacepkgs -Uhv $manifest
>>>>       fi
>>>>   }
>>>>
>>>> @@ -440,14 +441,7 @@ package_install_internal_rpm () {
>>>>
>>>>         fi
>>>>
>>>> -       # If base-passwd or shadow are in the list of packages to install,
>>>> -       # ensure they are installed first to support later packages that
>>>> -       # may create custom users/groups (fixes Yocto bug #2127)
>>>> -       infile=${target_rootfs}/install/install_solution.manifest
>>>> -       outfile=${target_rootfs}/install/total_solution.manifest
>>>> -       cat $infile | grep /base-passwd-[0-9]>    $outfile || true
>>>> -       cat $infile | grep /shadow-[0-9]>>    $outfile || true
>>>> -       cat $infile | grep -v /shadow-[0-9] | grep -v /base-passwd-[0-9]>>
>>>>   $outfile || true
>>>> +       cat ${target_rootfs}/install/install_solution.manifest>
>>>>   ${target_rootfs}/install/total_solution.manifest
>>>>         cat ${target_rootfs}/install/install_multilib_solution.manifest>>
>>>>   ${target_rootfs}/install/total_solution.manifest
>>>>
>>>>         # Construct install scriptlet wrapper
>>>> @@ -474,8 +468,45 @@ EOF
>>>>
>>>>         chmod 0755 ${WORKDIR}/scriptlet_wrapper
>>>>
>>>> -    rpm_update_pkg
>>>> +       # RPM is special. It can't handle dependencies and preinstall
>>>> scripts correctly. Its
>>>> +       # probably a feature. The only way to convince rpm to actually run
>>>> the preinstall scripts
>>>> +       # for base-passwd and shadow first before installing packages that
>>>> depend on these packages
>>>> +       # is to do two image installs, installing one set of packages,
>>>> then the other.
>>>> +       if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f
>>>> ${T}/total_solution_bt.manifest ]; then
>>>> +               echo "Skipping pre install due to exisitng image"
>>>> +       else
>>>> +               echo "# Intial Install manifest">
>>>>   ${target_rootfs}/install/initial_install.manifest
>>>> +               echo "Installing base dependencies first (base-passwd,
>>>> base-files and shadow) since rpm is special"
>>>> +               grep /base-passwd-[0-9]
>>>> ${target_rootfs}/install/total_solution.manifest>>
>>>>   ${target_rootfs}/install/initial_install.manifest || true
>>>> +               grep /base-files-[0-9]
>>>> ${target_rootfs}/install/total_solution.manifest>>
>>>>   ${target_rootfs}/install/initial_install.manifest || true
>>>> +               grep /shadow-[0-9]
>>>> ${target_rootfs}/install/total_solution.manifest>>
>>>>   ${target_rootfs}/install/initial_install.manifest || true
>>>> +
>>>> +               # Generate an install solution by doing a --justdb
>>>> install, then recreate it with
>>>> +               # an actual package install!
>>>> +               mkdir -p ${target_rootfs}/initial
>>>> +
>>>> +               ${RPM} --predefine "_rpmds_sysinfo_path
>>>> ${target_rootfs}/etc/rpm/sysinfo" \
>>>> +                       --predefine "_rpmrc_platform_path
>>>> ${target_rootfs}/etc/rpm/platform" \
>>>> +                       -D "_dbpath ${target_rootfs}/initial" -D "`cat
>>>> ${confbase}-base_archs.macro`" \
>>>> +                       -D "__dbi_txn create nofsync" \
>>>> +                       -U --justdb --noscripts --notriggers
>>>> --noparentdirs --nolinktos --ignoresize \
>>>> +                       ${target_rootfs}/install/initial_install.manifest
>>>> +
>>>> +               ${RPM} -D "_dbpath ${target_rootfs}/initial" -qa --yaml \
>>>> +                       -D "__dbi_txn create nofsync private" \
>>>> +                       | grep -i 'Packageorigin' | cut -d : -f 2>
>>>>   ${target_rootfs}/install/initial_solution.manifest
>>>> +
>>>> +               rpm_update_pkg
>>>> ${target_rootfs}/install/initial_solution.manifest
>>>> +
>>>> +               grep -Fv -f
>>>> ${target_rootfs}/install/initial_solution.manifest
>>>> ${target_rootfs}/install/total_solution.manifest>
>>>>   ${target_rootfs}/install/total_solution.manifest.new
>>>> +               mv ${target_rootfs}/install/total_solution.manifest.new
>>>> ${target_rootfs}/install/total_solution.manifest
>>>> +
>>>> +               rm -rf ${target_rootfs}/initial
>>>> +       fi
>>>> +
>>>> +       echo "Installing main solution manifest
>>>> (${target_rootfs}/install/total_solution.manifest)"
>>>>
>>>> +       rpm_update_pkg ${target_rootfs}/install/total_solution.manifest
>>>>   }
>>>>
>>>>   python write_specfile () {
>>>>
>>>
>>>
>>> _______________________________________________
>>> Openembedded-core mailing list
>>> Openembedded-core@lists.openembedded.org
>>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [PATCH] package_rpm: Fix useradd preinst ordering issues

[Steve Sakoman]

On Thu, Apr 12, 2012 at 8:46 AM, Mark Hatle <mark.hatle@windriver.com> wrote:
> On 4/12/12 10:36 AM, Steve Sakoman wrote:
>>
>> On Thu, Apr 12, 2012 at 6:39 AM, Steve Sakoman<sakoman@gmail.com>  wrote:
>
>
> That is very odd, the system is supposed to identify all of the dependencies
> needed for the first set of packages... (the three base-passwd, base-files
> and shadow).

Perhaps the issue is with the bash package -- for some reason rpm
doesn't realize that bash provides /bin/sh?

> The end result is a list of 10-12 binaries to be installed that meet those
> requirements, as well as the original three requested packages.
>
> We also shouldn't specify bash because busybox provides /bin/sh in a lot of
> configurations.
>
> Which image did you try to build and I'll see what I can replicate here.

The images I built are not standard oe-core/yocto images, they are
custom images that don't include busybox.  Perhaps that is why I am
seeing the issue and others aren't.

If you are really interested in replicating let me know and I can
provide you with pointers to my image recipes.

Steve

Re: [PATCH] package_rpm: Fix useradd preinst ordering issues

[Richard Purdie]

On Thu, 2012-04-12 at 08:55 -0700, Steve Sakoman wrote:
> On Thu, Apr 12, 2012 at 8:46 AM, Mark Hatle <mark.hatle@windriver.com> wrote:
> > On 4/12/12 10:36 AM, Steve Sakoman wrote:
> >>
> >> On Thu, Apr 12, 2012 at 6:39 AM, Steve Sakoman<sakoman@gmail.com>  wrote:
> >
> >
> > That is very odd, the system is supposed to identify all of the dependencies
> > needed for the first set of packages... (the three base-passwd, base-files
> > and shadow).
> 
> Perhaps the issue is with the bash package -- for some reason rpm
> doesn't realize that bash provides /bin/sh?
> 
> > The end result is a list of 10-12 binaries to be installed that meet those
> > requirements, as well as the original three requested packages.
> >
> > We also shouldn't specify bash because busybox provides /bin/sh in a lot of
> > configurations.
> >
> > Which image did you try to build and I'll see what I can replicate here.
> 
> The images I built are not standard oe-core/yocto images, they are
> custom images that don't include busybox.  Perhaps that is why I am
> seeing the issue and others aren't.
> 
> If you are really interested in replicating let me know and I can
> provide you with pointers to my image recipes.

For what its worth, bash here does Provides: /bin/sh so I'm a little
puzzled about what is going on here. If its not doing that, it would
certainly cause the error you're reporting...

This is with master? Did you do anything like disable the per file
dependencies?

Cheers,

Richard

Re: [PATCH] package_rpm: Fix useradd preinst ordering issues

[Mark Hatle]

On 4/12/12 10:55 AM, Steve Sakoman wrote:
> On Thu, Apr 12, 2012 at 8:46 AM, Mark Hatle<mark.hatle@windriver.com>  wrote:
>> On 4/12/12 10:36 AM, Steve Sakoman wrote:
>>>
>>> On Thu, Apr 12, 2012 at 6:39 AM, Steve Sakoman<sakoman@gmail.com>    wrote:
>>
>>
>> That is very odd, the system is supposed to identify all of the dependencies
>> needed for the first set of packages... (the three base-passwd, base-files
>> and shadow).
>
> Perhaps the issue is with the bash package -- for some reason rpm
> doesn't realize that bash provides /bin/sh?
>
>> The end result is a list of 10-12 binaries to be installed that meet those
>> requirements, as well as the original three requested packages.
>>
>> We also shouldn't specify bash because busybox provides /bin/sh in a lot of
>> configurations.
>>
>> Which image did you try to build and I'll see what I can replicate here.
>
> The images I built are not standard oe-core/yocto images, they are
> custom images that don't include busybox.  Perhaps that is why I am
> seeing the issue and others aren't.
>
> If you are really interested in replicating let me know and I can
> provide you with pointers to my image recipes.

Ya, if you could point me to the image recipe that eliminates BB and uses the 
discrete versions of things I'll look into this.

I can try to do it with hob, but I'm not sure of everything I'd have to select.

--Mark

> Steve

Re: [PATCH] package_rpm: Fix useradd preinst ordering issues

[Steve Sakoman]

> Ya, if you could point me to the image recipe that eliminates BB and uses
> the discrete versions of things I'll look into this.
>
> I can try to do it with hob, but I'm not sure of everything I'd have to
> select.

Here is one of the images that shows the issue:

http://www.sakoman.com/cgi-bin/gitweb.cgi?p=meta-sakoman.git;a=blob_plain;f=recipes-sakoman/images/sakoman-systemd-image.bb;h=d1f1202528af6ed73171a9a26148f18da9ee389c;hb=HEAD

It pulls in a few things from meta-oe and my meta-sakoman layer so
this may be more trouble that you want to deal with . . .

Steve

Re: [PATCH] package_rpm: Fix useradd preinst ordering issues

[Steve Sakoman]

On Thu, Apr 12, 2012 at 9:08 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:

> For what its worth, bash here does Provides: /bin/sh so I'm a little
> puzzled about what is going on here. If its not doing that, it would
> certainly cause the error you're reporting...

Hmm . . . while I see:

Provides: /bin/sh

in bash.spec, it is a different story in the pkgdata/runtime files.

busybox has:

FILERPROVIDES_/bin/busybox_busybox: <snip many filenames> /bin/sh
<snip many filenames>

While bash doesn't have a FILERPROVIDES entry at all.

Could this be why rpm doesn't realize bash provides /bin/sh?

Steve

Re: [PATCH] package_rpm: Fix useradd preinst ordering issues

[Mark Hatle]

On 4/12/12 12:06 PM, Steve Sakoman wrote:
> On Thu, Apr 12, 2012 at 9:08 AM, Richard Purdie
> <richard.purdie@linuxfoundation.org>  wrote:
>
>> For what its worth, bash here does Provides: /bin/sh so I'm a little
>> puzzled about what is going on here. If its not doing that, it would
>> certainly cause the error you're reporting...
>
> Hmm . . . while I see:
>
> Provides: /bin/sh
>
> in bash.spec, it is a different story in the pkgdata/runtime files.
>
> busybox has:
>
> FILERPROVIDES_/bin/busybox_busybox:<snip many filenames>  /bin/sh
> <snip many filenames>
>
> While bash doesn't have a FILERPROVIDES entry at all.
>
> Could this be why rpm doesn't realize bash provides /bin/sh?

They should both be equally valid for the purpose of dependency resolution.

--Mark

> Steve