[PATCH] lib/oe/rootfs: Fix DEBUGFS generation when using opkg

[PATCH] lib/oe/rootfs: Fix DEBUGFS generation when using opkg

[Richard Purdie]

When enabling extra DEBUGFS image generation with opkg, errors are seen like:

ERROR: core-image-minimal-1.0-r0 do_rootfs: Cannot get the installed packages list. Command '/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/opkg -f /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/opkg.conf -o /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs  --force_postinstall --prefer-arch-to-version   status' returned 0 and stderr:
Collected errors:
 * file_md5sum_alloc: Failed to open file /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/syslog-startup.conf.busybox: No such file or directory.
 * file_md5sum_alloc: Failed to open file /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/fstab: No such file or directory.

basically for all CONFFILES in the image. This is due to the file rearranging
the rootfs generation code does. If we preserve the /etc directory,
the avoids the problem.

We need to tell copyfile to preserve symlinks since some are present in /etc.

[YOCTO #9490]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index a95e1b7..0546c1e 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -122,7 +122,7 @@ class Rootfs(object):
         bb.note("  Copying back package database...")
         for dir in dirs:
             bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(dir))
-            shutil.copytree(self.image_rootfs + '-orig' + dir, self.image_rootfs + dir)
+            shutil.copytree(self.image_rootfs + '-orig' + dir, self.image_rootfs + dir, symlinks=True)
 
         cpath = oe.cachedpath.CachedPath()
         # Copy files located in /usr/lib/debug or /usr/src/debug
@@ -907,7 +907,7 @@ class OpkgRootfs(DpkgOpkgRootfs):
 
         self.pm.install_complementary()
 
-        self._setup_dbg_rootfs(['/var/lib/opkg'])
+        self._setup_dbg_rootfs(['/etc', '/var/lib/opkg'])
 
         execute_pre_post_process(self.d, opkg_post_process_cmds)
 

Re: [PATCH] lib/oe/rootfs: Fix DEBUGFS generation when using opkg

[Anders Darander]

Hi,
* Richard Purdie <richard.purdie@linuxfoundation.org> [160422 15:52]:

> When enabling extra DEBUGFS image generation with opkg, errors are seen like:

> ERROR: core-image-minimal-1.0-r0 do_rootfs: Cannot get the installed packages list. Command '/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/opkg -f /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/opkg.conf -o /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs  --force_postinstall --prefer-arch-to-version   status' returned 0 and stderr:
> Collected errors:
>  * file_md5sum_alloc: Failed to open file /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/syslog-startup.conf.busybox: No such file or directory.
>  * file_md5sum_alloc: Failed to open file /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/fstab: No such file or directory.

> basically for all CONFFILES in the image. This is due to the file rearranging
> the rootfs generation code does. If we preserve the /etc directory,
> the avoids the problem.

> We need to tell copyfile to preserve symlinks since some are present in /etc.

Thanks for the patch. It fixes the issue on e.g. core-image-minimal.

Unfortunately, on more complex images, we'll still fail with e.g.:

ERROR: core-image-minimal-1.0-r0 do_rootfs: Cannot get the installed packages list. Command '/mnt/cs-builds/anders/poky-play/build/tmp-glibc/sysroots/x86_64-linux/usr/bin/opkg -f /mnt/cs-builds/anders/poky-play/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/opkg.conf -o /mnt/cs-builds/anders/poky-play/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs --force_postinstall --prefer-arch-to-version   status' returned 0 and stderr:
Collected errors:
 * file_md5sum_alloc: Failed to open file /mnt/cs-builds/anders/poky-play/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs/usr/lib/ssl/openssl.cnf: No such file or directory.

I'd guess that all packages that have CONFFILES outside of /etc will
suffer from this...

I've updated #9490 with this info.

Cheers,
Anders
-- 
Anders Darander, Senior System Architect
ChargeStorm AB / eStorm AB

Re: [PATCH] lib/oe/rootfs: Fix DEBUGFS generation when using opkg

[Patrick Ohly]

On Fri, 2016-04-22 at 14:51 +0100, Richard Purdie wrote:
> When enabling extra DEBUGFS image generation with opkg, errors are seen like:
> 
> ERROR: core-image-minimal-1.0-r0 do_rootfs: Cannot get the installed packages list. Command '/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/opkg -f /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/opkg.conf -o /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs  --force_postinstall --prefer-arch-to-version   status' returned 0 and stderr:
> Collected errors:
>  * file_md5sum_alloc: Failed to open
> file /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/syslog-startup.conf.busybox: No such file or directory.
>  * file_md5sum_alloc: Failed to open
> file /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/fstab: No such file or directory.
> 
> basically for all CONFFILES in the image. This is due to the file
> rearranging
> the rootfs generation code does. If we preserve the /etc directory,
> the avoids the problem.

It avoids the problem, but it does not address the root cause (IMHO).
Should opkg really complain about missing configuration files? It is
perhaps an edge case, but besides editing a configuration file
*removing* it entirely may also be a valid user modification.

The usage of opkg here is during image building, but the operation
itself (opkg status) is not specific to image creation and thus should
be able to handle arbitrary states of /etc and the config files in
general.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH] lib/oe/rootfs: Fix DEBUGFS generation when using opkg

[Alejandro del Castillo]

On 05/02/2016 08:39 AM, Patrick Ohly wrote:
> On Fri, 2016-04-22 at 14:51 +0100, Richard Purdie wrote:
>> When enabling extra DEBUGFS image generation with opkg, errors are seen like:
>>
>> ERROR: core-image-minimal-1.0-r0 do_rootfs: Cannot get the installed packages list. Command '/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/opkg -f /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/opkg.conf -o /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs  --force_postinstall --prefer-arch-to-version   status' returned 0 and stderr:
>> Collected errors:
>>  * file_md5sum_alloc: Failed to open
>> file /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/syslog-startup.conf.busybox: No such file or directory.
>>  * file_md5sum_alloc: Failed to open
>> file /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/fstab: No such file or directory.
>>
>> basically for all CONFFILES in the image. This is due to the file
>> rearranging
>> the rootfs generation code does. If we preserve the /etc directory,
>> the avoids the problem.
> 
> It avoids the problem, but it does not address the root cause (IMHO).
> Should opkg really complain about missing configuration files? It is
> perhaps an edge case, but besides editing a configuration file
> *removing* it entirely may also be a valid user modification.
> 
> The usage of opkg here is during image building, but the operation
> itself (opkg status) is not specific to image creation and thus should
> be able to handle arbitrary states of /etc and the config files in
> general.

Opkg status is supposed to give you the state of all your installed packages.
Currently is pretty rough data: it dumps into stdout the internal metadata that
tracks package info and promotes to error any inconsistency, like conffiles not
being present.

Looking at pacakge_manager.py, opkg status is being called on OpkgPkgsList.
Seems to me that if the purpose is to get all installed packages, then "opkg
list-installed" is the correct command. Looking at dpkg (DpkgPkgsList),
"dpkg-query -W" is being called, which I think maps to opkg list-installed, not
to opkg status.

-- 
Cheers,

Alejandro

Re: [PATCH] lib/oe/rootfs: Fix DEBUGFS generation when using opkg

[Patrick Ohly]

On Mon, 2016-05-02 at 11:15 -0500, Alejandro del Castillo wrote:
> 
> On 05/02/2016 08:39 AM, Patrick Ohly wrote:
> > On Fri, 2016-04-22 at 14:51 +0100, Richard Purdie wrote:
> >> When enabling extra DEBUGFS image generation with opkg, errors are seen like:
> >>
> >> ERROR: core-image-minimal-1.0-r0 do_rootfs: Cannot get the installed packages list. Command '/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/opkg -f /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/opkg.conf -o /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs  --force_postinstall --prefer-arch-to-version   status' returned 0 and stderr:
> >> Collected errors:
> >>  * file_md5sum_alloc: Failed to open
> >> file /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/syslog-startup.conf.busybox: No such file or directory.
> >>  * file_md5sum_alloc: Failed to open
> >> file /media/build1/poky/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/fstab: No such file or directory.
> >>
> >> basically for all CONFFILES in the image. This is due to the file
> >> rearranging
> >> the rootfs generation code does. If we preserve the /etc directory,
> >> the avoids the problem.
> > 
> > It avoids the problem, but it does not address the root cause (IMHO).
> > Should opkg really complain about missing configuration files? It is
> > perhaps an edge case, but besides editing a configuration file
> > *removing* it entirely may also be a valid user modification.
> > 
> > The usage of opkg here is during image building, but the operation
> > itself (opkg status) is not specific to image creation and thus should
> > be able to handle arbitrary states of /etc and the config files in
> > general.
> 
> Opkg status is supposed to give you the state of all your installed packages.
> Currently is pretty rough data: it dumps into stdout the internal metadata that
> tracks package info and promotes to error any inconsistency, like conffiles not
> being present.

But how does opkg know that "config file not present" is an error? It
depends on the the semantic of the file and thus the package providing
the file. It's not declared explicitly, so it is a fairly arbitrary
judgment call to treat "missing" as error.

> Looking at pacakge_manager.py, opkg status is being called on OpkgPkgsList.
> Seems to me that if the purpose is to get all installed packages, then "opkg
> list-installed" is the correct command. Looking at dpkg (DpkgPkgsList),
> "dpkg-query -W" is being called, which I think maps to opkg list-installed, not
> to opkg status.

That indeed seems better: output should be smaller, faster to run and it
avoids the ambiguity about missing config files.



-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH] lib/oe/rootfs: Fix DEBUGFS generation when using opkg

[Alejandro del Castillo]

On 05/03/2016 02:49 AM, Patrick Ohly wrote:
> But how does opkg know that "config file not present" is an error? It
> depends on the the semantic of the file and thus the package providing
> the file. It's not declared explicitly, so it is a fairly arbitrary
> judgment call to treat "missing" as error.

I agree. What I meant when I said that "opkg status" output is pretty rough is
that it is making blanket assumptions, and dumping unprocessed metadata. Dpkg -V
analyses the md5sum of each file to determine if it changed, and reports changed
files. While opkg currently only stores the md5sum of conffiles, I think opkg
status should behave similarly to dpkg -V (for conffiles, at least).

Do you mind filing a bug ticket on bugzilla for opkg?

-- 
Cheers,

Alejandro