possible bug with dpkg-native with sstate-cache

possible bug with dpkg-native with sstate-cache

[Anders Oleson]

[-- Attachment #1: Type: text/plain, Size: 2325 bytes --]

I have found what I think is a sneaky nasty bug when using package_deb with
SSTATE_MIRRORS on RPM based build hosts. Below is a short description of what
I found and our workaround. A patch that adds a patch that is applied to the
packager that packages the real packager.  To avoid this sneaking failure mode
when building without building when using the cache (its all so very
meta, btw.).

Don't ask how we found this; it wasn't pretty.

Note: the fix approach below actually patches dpkg on the target too,
but I do not
know how to easily make the patch only apply to dpkg-native and not also dpkg.
But in this case it should be fine anyhow; there is no reason that dpkg on the
target should not behave this way too. I have checked the upstream and the
lines in question have never been touched - it is possible they may want to
eventually upstream this too (???).

For future knowledge, is there an easy way to have a SRC_URI list that is
different for dpkg-native than for dpkg?

Also, this is my first post to the group, so apologies if this is not the
right place or way to upload a potential patch; set me straight if so.

Thanks, guys.

Anders.

    The dpkg binary contains a hard-coded CONFIGDIR path which normally points
    to /etc/dpkg. However, if /etc/dpkg/dpkg.cfg.d doesn't exist (ENOTDIR) it
    fails with an error, even though it does not fail when the directory exists
    but is empty (ENOENT).

    Normally this is not an issue, but when dpkg-native is built, CONFIGDIR
    points to a directory in the sysroot under work. If that binary is later
    cached in the sstate-cache and mirrored to another host that does not
    contain the same work directory, the dpkg binary fails due to this missing
    directory during certain packaging steps involving apt-get. This "leaks"
    information about the build host into the sstate-cache in an unintended way
    that causes a failure.

    The dpkg utility does not currently allow a command line or environment
    override of CONFIGDIR, so this patches dpkg so that a missing directory is
    the same as an empty one and the failure is avoided. The dpkg-native is
    always passed command line arguments and does not need, nor contain, any
    configuration data in this directory in the first place

[-- Attachment #2: patch.txt --]
[-- Type: text/plain, Size: 2901 bytes --]

commit 48441318ced298e58ba2ea4ae1e2332259dcb6f8
Author: Anders <anders@openpuma.org>
Date:   Tue Dec 13 20:10:25 2016 -0800

    patch to workaround sstate-cache bug in dpkg
    
    The dpkg binary contains a hard-coded CONFIGDIR path which normally points to
    /etc/dpkg. However, if /etc/dpkg/dpkg.cfg.d doesn't exist (ENOTDIR) it fails
    with an error, even though it does not fail when the directory exists but is
    empty (ENOENT).
    
    Normally this is not an issue, but when dpkg-native is built, CONFIGDIR points
    to a directory in the sysroot under work. If that binary is later cached in
    the sstate-cache and mirrored to another host that does not contain the same
    work directory, the dpkg binary fails due to this missing directory during
    certain packaging steps involving apt-get. This "leaks" information about the
    build host into the sstate-cache in an unintended way that causes a failure.
    
    The dpkg utility does not currently allow a command line or environment
    override of CONFIGDIR, so this patches dpkg so that a missing directory is the
    same as an empty one and the failure is avoided. The dpkg-native is always
    passed command line arguments and does not need, nor contain, any
    configuration data in this directory in the first place.

diff --git a/meta/recipes-devtools/dpkg/dpkg/allow-missing-config-dir.patch b/meta/recipes-devtools/dpkg/dpkg/allow-missing-config-dir.patch
new file mode 100644
index 0000000..8a52065
--- /dev/null
+++ b/meta/recipes-devtools/dpkg/dpkg/allow-missing-config-dir.patch
@@ -0,0 +1,16 @@
+Index: dpkg-1.18.7/lib/dpkg/options.c
+===================================================================
+--- dpkg-1.18.7.orig/lib/dpkg/options.c
++++ dpkg-1.18.7/lib/dpkg/options.c
+@@ -172,11 +172,8 @@ dpkg_options_load_dir(const char *prog,
+ 
+   dlist_n = scandir(dirname, &dlist, valid_config_filename, alphasort);
+   if (dlist_n < 0) {
+-    if (errno == ENOENT) {
+       free(dirname);
+       return;
+-    } else
+-      ohshite(_("error opening configuration directory '%s'"), dirname);
+   }
+ 
+   for (i = 0; i < dlist_n; i++) {
diff --git a/meta/recipes-devtools/dpkg/dpkg_1.18.7.bb b/meta/recipes-devtools/dpkg/dpkg_1.18.7.bb
index 28fdc13..981fc6b 100644
--- a/meta/recipes-devtools/dpkg/dpkg_1.18.7.bb
+++ b/meta/recipes-devtools/dpkg/dpkg_1.18.7.bb
@@ -7,6 +7,7 @@ SRC_URI = "http://snapshot.debian.org/archive/debian/20160509T100042Z/pool/main/
            file://arch_pm.patch \
            file://dpkg-configure.service \
            file://add_armeb_triplet_entry.patch \
+           file://allow-missing-config-dir.patch \
            file://0002-Adapt-to-linux-wrs-kernel-version-which-has-characte.patch \
            file://0003-Our-pre-postinsts-expect-D-to-be-set-when-running-in.patch \
            file://0004-The-lutimes-function-doesn-t-work-properly-for-all-s.patch \

Re: possible bug with dpkg-native with sstate-cache

[Jussi Kukkonen]

[-- Attachment #1: Type: text/plain, Size: 2315 bytes --]

On 14 December 2016 at 06:59, Anders Oleson <anders@openpuma.org> wrote:

> I have found what I think is a sneaky nasty bug when using package_deb with
> SSTATE_MIRRORS on RPM based build hosts. Below is a short description of
> what
> I found and our workaround. A patch that adds a patch that is applied to
> the
> packager that packages the real packager.  To avoid this sneaking failure
> mode
> when building without building when using the cache (its all so very
> meta, btw.).

Don't ask how we found this; it wasn't pretty.
>
> Note: the fix approach below actually patches dpkg on the target too,
> but I do not
> know how to easily make the patch only apply to dpkg-native and not also
> dpkg.
> But in this case it should be fine anyhow; there is no reason that dpkg on
> the
> target should not behave this way too. I have checked the upstream and the
> lines in question have never been touched - it is possible they may want to
> eventually upstream this too (???).
>

I have to point out that the upstream developers are usually not going to
come to us looking for patches :) If we want patches upstreamed we have to
go to them and in fact there is a bit of process to to track this, see
below.

For future knowledge, is there an easy way to have a SRC_URI list that is
> different for dpkg-native than for dpkg?
>

I think this is actually already used in dpkg recipe:
    SRC_URI_append_class-native = "..."
but as you said this doesn't seem required in this case.

Also, this is my first post to the group, so apologies if this is not the
> right place or way to upload a potential patch; set me straight if so.
>

Thanks and no problem: openembedded-core@lists.openembedded.org would be
the correct place for meta/ patches. Please take a look at

http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#how-to-submit-a-change
    http://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines

Some specific things you may want to look at:
* Send the commit as part of an email, not as attachment (git "send-email"
  and/or "create-pull-request/send-pull-request" script are good tools)
* Signed-off-by tag on both the commit and in any patch files in it
* Upstream-Status tag in any patch files in the commit

Thanks,
 Jussi

[-- Attachment #2: Type: text/html, Size: 3630 bytes --]

Re: possible bug with dpkg-native with sstate-cache

[Anders Oleson]

On Wed, Dec 14, 2016 at 2:01 AM, Jussi Kukkonen
<jussi.kukkonen@intel.com> wrote:
> On 14 December 2016 at 06:59, Anders Oleson <anders@openpuma.org> wrote:
>>
>> I have found what I think is a sneaky nasty bug when using package_deb
>> with
>> SSTATE_MIRRORS on RPM based build hosts. Below is a short description of
>> what
>> I found and our workaround. A patch that adds a patch that is applied to
>> the
>> packager that packages the real packager.  To avoid this sneaking failure
>> mode
>> when building without building when using the cache (its all so very
>> meta, btw.).
>>
>> Don't ask how we found this; it wasn't pretty.
>>
>> Note: the fix approach below actually patches dpkg on the target too,
>> but I do not
>> know how to easily make the patch only apply to dpkg-native and not also
>> dpkg.
>> But in this case it should be fine anyhow; there is no reason that dpkg on
>> the
>> target should not behave this way too. I have checked the upstream and the
>> lines in question have never been touched - it is possible they may want
>> to
>> eventually upstream this too (???).
>
>
> I have to point out that the upstream developers are usually not going to
> come to us looking for patches :) If we want patches upstreamed we have to
> go to them and in fact there is a bit of process to to track this, see
> below.

Yes, I'm more or less thinking out loud whether it would be something
they care about.
I'm not sure they would, but I may ask and see. There's a minor issue
in that the
error message would disappear in case of a real misconfiguration, but
this morning I
see that on my Ubuntu box that the directory is empty anyhow.

>
>> For future knowledge, is there an easy way to have a SRC_URI list that is
>> different for dpkg-native than for dpkg?
>
>
> I think this is actually already used in dpkg recipe:
>     SRC_URI_append_class-native = "..."
> but as you said this doesn't seem required in this case.
>

OK, wasn't sure if this was a good idea and I missed the example. So
that all makes sense.
By the time I formulated the question I was 1/2 way there...

>> Also, this is my first post to the group, so apologies if this is not the
>> right place or way to upload a potential patch; set me straight if so.
>
>
> Thanks and no problem: openembedded-core@lists.openembedded.org would be the
> correct place for meta/ patches. Please take a look at
>
> http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#how-to-submit-a-change
>     http://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
>
> Some specific things you may want to look at:
> * Send the commit as part of an email, not as attachment (git "send-email"
>   and/or "create-pull-request/send-pull-request" script are good tools)
> * Signed-off-by tag on both the commit and in any patch files in it
> * Upstream-Status tag in any patch files in the commit
>
> Thanks,
>  Jussi

Will do. It may take a bit to process all that, but I will.