[PATCH] package_deb: Fix rootfs generation in non Debian-based hosts

[PATCH] package_deb: Fix rootfs generation in non Debian-based hosts

[Aníbal Limón]

Since we have support of log checking in deb/ipk [1] rootfs generation
in non Debian-based hosts fails because apt-ftparchive generates a
warn when not find /etc/apt/apt.conf.d/ (available in Debian-based
hosts).

In order to fix,

	package_manager.py: DpkgPMIndexer add support for export APT_CONF
	to environment.

[1] http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=86aec93902af2e2d7d73ca9a643707fcca45055c

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta/lib/oe/package_manager.py | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index c9a8084..395c0d01 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -169,7 +169,14 @@ class OpkgIndexer(Indexer):
 
 
 class DpkgIndexer(Indexer):
+    def __init__(self, d, deploy_dir, apt_conf_file=None):
+        self.apt_conf_file = apt_conf_file
+        Indexer.__init__(self, d, deploy_dir)
+
     def write_index(self):
+        if not self.apt_conf_file is None:
+            os.environ['APT_CONFIG'] = self.apt_conf_file
+
         pkg_archs = self.d.getVar('PACKAGE_ARCHS', True)
         if pkg_archs is not None:
             arch_list = pkg_archs.split()
@@ -1507,7 +1514,7 @@ class DpkgPM(PackageManager):
 
         self._create_configs(archs, base_archs)
 
-        self.indexer = DpkgIndexer(self.d, self.deploy_dir)
+        self.indexer = DpkgIndexer(self.d, self.deploy_dir, self.apt_conf_file)
 
     """
     This function will change a package's status in /var/lib/dpkg/status file.
-- 
1.8.4.5

[PATCH] package_manager/deb: apt-ftparchive needs a valid config

[Andreas Oberritter]

Hello Aníbal,

On 23.03.2015 17:05, Aníbal Limón wrote:
> Since we have support of log checking in deb/ipk [1] rootfs generation
> in non Debian-based hosts fails because apt-ftparchive generates a
> warn when not find /etc/apt/apt.conf.d/ (available in Debian-based
> hosts).
> 
> In order to fix,
> 
> 	package_manager.py: DpkgPMIndexer add support for export APT_CONF
> 	to environment.
> 
> [1] http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=86aec93902af2e2d7d73ca9a643707fcca45055c
> 
> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
> ---
>  meta/lib/oe/package_manager.py | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
> index c9a8084..395c0d01 100644
> --- a/meta/lib/oe/package_manager.py
> +++ b/meta/lib/oe/package_manager.py
> @@ -169,7 +169,14 @@ class OpkgIndexer(Indexer):
>  
>  
>  class DpkgIndexer(Indexer):
> +    def __init__(self, d, deploy_dir, apt_conf_file=None):
> +        self.apt_conf_file = apt_conf_file
> +        Indexer.__init__(self, d, deploy_dir)
> +
>      def write_index(self):
> +        if not self.apt_conf_file is None:
> +            os.environ['APT_CONFIG'] = self.apt_conf_file
> +
>          pkg_archs = self.d.getVar('PACKAGE_ARCHS', True)
>          if pkg_archs is not None:
>              arch_list = pkg_archs.split()
> @@ -1507,7 +1514,7 @@ class DpkgPM(PackageManager):
>  
>          self._create_configs(archs, base_archs)
>  
> -        self.indexer = DpkgIndexer(self.d, self.deploy_dir)
> +        self.indexer = DpkgIndexer(self.d, self.deploy_dir, self.apt_conf_file)
>  
>      """
>      This function will change a package's status in /var/lib/dpkg/status file.
> 

unfortunately, this approach doesn't work with "bitbake package-index", which
calls generate_index_files() in package_manager.py. That's why I implemented
a quite ugly solution covering all cases:


From 668fe52b58aba6ab47e4712ee460ed7dd06c948f Mon Sep 17 00:00:00 2001
From: Andreas Oberritter <obi@opendreambox.org>
Date: Wed, 8 Oct 2014 15:52:22 +0200
Subject: [PATCH] package_manager/deb: apt-ftparchive needs a valid config

Without a valid config, apt-ftparchive aborts. Since apt-ftparchive
defaults to /etc/apt if APT_CONFIG is unset, this is only an issue
non non-Debian build hosts.

Signed-off-by: Andreas Oberritter <obi@opendreambox.org>
---
 meta/lib/oe/package_manager.py | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index d4ac174..71f909c 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -182,6 +182,28 @@ class DpkgIndexer(Indexer):
         all_mlb_pkg_arch_list = (self.d.getVar('ALL_MULTILIB_PACKAGE_ARCHS', True) or "").replace('-', '_').split()
         arch_list.extend(arch for arch in all_mlb_pkg_arch_list if arch not in arch_list)
 
+        apt_conf_dir = self.d.expand("${APTCONF_TARGET}/apt-ftparchive")
+        apt_conf_file = os.path.join(apt_conf_dir, "apt.conf")
+
+        bb.utils.mkdirhier(apt_conf_dir)
+        bb.utils.mkdirhier(apt_conf_dir + "/lists/partial/")
+        bb.utils.mkdirhier(apt_conf_dir + "/apt.conf.d/")
+        bb.utils.mkdirhier(apt_conf_dir + "/preferences.d/")
+
+        with open(os.path.join(apt_conf_dir, "preferences"), "w") as prefs_file:
+            pass
+        with open(os.path.join(apt_conf_dir, "sources.list"), "w+") as sources_file:
+            pass
+
+        with open(apt_conf_file, "w") as apt_conf:
+            with open(self.d.expand("${STAGING_ETCDIR_NATIVE}/apt/apt.conf.sample")) as apt_conf_sample:
+                for line in apt_conf_sample.read().split("\n"):
+                    line = re.sub("#ROOTFS#", "/dev/null", line)
+                    line = re.sub("#APTCONF#", apt_conf_dir, line)
+                    apt_conf.write(line + "\n")
+
+        os.environ['APT_CONFIG'] = apt_conf_file
+
         apt_ftparchive = bb.utils.which(os.getenv('PATH'), "apt-ftparchive")
         gzip = bb.utils.which(os.getenv('PATH'), "gzip")
 
-- 
1.9.1

Re: [PATCH] package_manager/deb: apt-ftparchive needs a valid config

[Aníbal Limón]

Hi Andreas,

Your solution is better but i think we need to generalize the code in 
DpkgPM::_create_configs to avoid code duplicate.

Comments?

Cheers,
     alimon

On 24/03/15 05:58, Andreas Oberritter wrote:
> Hello Aníbal,
>
> On 23.03.2015 17:05, Aníbal Limón wrote:
>> Since we have support of log checking in deb/ipk [1] rootfs generation
>> in non Debian-based hosts fails because apt-ftparchive generates a
>> warn when not find /etc/apt/apt.conf.d/ (available in Debian-based
>> hosts).
>>
>> In order to fix,
>>
>> 	package_manager.py: DpkgPMIndexer add support for export APT_CONF
>> 	to environment.
>>
>> [1] http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=86aec93902af2e2d7d73ca9a643707fcca45055c
>>
>> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
>> ---
>>   meta/lib/oe/package_manager.py | 9 ++++++++-
>>   1 file changed, 8 insertions(+), 1 deletion(-)
>>
>> diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
>> index c9a8084..395c0d01 100644
>> --- a/meta/lib/oe/package_manager.py
>> +++ b/meta/lib/oe/package_manager.py
>> @@ -169,7 +169,14 @@ class OpkgIndexer(Indexer):
>>   
>>   
>>   class DpkgIndexer(Indexer):
>> +    def __init__(self, d, deploy_dir, apt_conf_file=None):
>> +        self.apt_conf_file = apt_conf_file
>> +        Indexer.__init__(self, d, deploy_dir)
>> +
>>       def write_index(self):
>> +        if not self.apt_conf_file is None:
>> +            os.environ['APT_CONFIG'] = self.apt_conf_file
>> +
>>           pkg_archs = self.d.getVar('PACKAGE_ARCHS', True)
>>           if pkg_archs is not None:
>>               arch_list = pkg_archs.split()
>> @@ -1507,7 +1514,7 @@ class DpkgPM(PackageManager):
>>   
>>           self._create_configs(archs, base_archs)
>>   
>> -        self.indexer = DpkgIndexer(self.d, self.deploy_dir)
>> +        self.indexer = DpkgIndexer(self.d, self.deploy_dir, self.apt_conf_file)
>>   
>>       """
>>       This function will change a package's status in /var/lib/dpkg/status file.
>>
> unfortunately, this approach doesn't work with "bitbake package-index", which
> calls generate_index_files() in package_manager.py. That's why I implemented
> a quite ugly solution covering all cases:
>
>
>  From 668fe52b58aba6ab47e4712ee460ed7dd06c948f Mon Sep 17 00:00:00 2001
> From: Andreas Oberritter <obi@opendreambox.org>
> Date: Wed, 8 Oct 2014 15:52:22 +0200
> Subject: [PATCH] package_manager/deb: apt-ftparchive needs a valid config
>
> Without a valid config, apt-ftparchive aborts. Since apt-ftparchive
> defaults to /etc/apt if APT_CONFIG is unset, this is only an issue
> non non-Debian build hosts.
>
> Signed-off-by: Andreas Oberritter <obi@opendreambox.org>
> ---
>   meta/lib/oe/package_manager.py | 22 ++++++++++++++++++++++
>   1 file changed, 22 insertions(+)
>
> diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
> index d4ac174..71f909c 100644
> --- a/meta/lib/oe/package_manager.py
> +++ b/meta/lib/oe/package_manager.py
> @@ -182,6 +182,28 @@ class DpkgIndexer(Indexer):
>           all_mlb_pkg_arch_list = (self.d.getVar('ALL_MULTILIB_PACKAGE_ARCHS', True) or "").replace('-', '_').split()
>           arch_list.extend(arch for arch in all_mlb_pkg_arch_list if arch not in arch_list)
>   
> +        apt_conf_dir = self.d.expand("${APTCONF_TARGET}/apt-ftparchive")
> +        apt_conf_file = os.path.join(apt_conf_dir, "apt.conf")
> +
> +        bb.utils.mkdirhier(apt_conf_dir)
> +        bb.utils.mkdirhier(apt_conf_dir + "/lists/partial/")
> +        bb.utils.mkdirhier(apt_conf_dir + "/apt.conf.d/")
> +        bb.utils.mkdirhier(apt_conf_dir + "/preferences.d/")
> +
> +        with open(os.path.join(apt_conf_dir, "preferences"), "w") as prefs_file:
> +            pass
> +        with open(os.path.join(apt_conf_dir, "sources.list"), "w+") as sources_file:
> +            pass
> +
> +        with open(apt_conf_file, "w") as apt_conf:
> +            with open(self.d.expand("${STAGING_ETCDIR_NATIVE}/apt/apt.conf.sample")) as apt_conf_sample:
> +                for line in apt_conf_sample.read().split("\n"):
> +                    line = re.sub("#ROOTFS#", "/dev/null", line)
> +                    line = re.sub("#APTCONF#", apt_conf_dir, line)
> +                    apt_conf.write(line + "\n")
> +
> +        os.environ['APT_CONFIG'] = apt_conf_file
> +
>           apt_ftparchive = bb.utils.which(os.getenv('PATH'), "apt-ftparchive")
>           gzip = bb.utils.which(os.getenv('PATH'), "gzip")
>   

Re: [PATCH] package_manager/deb: apt-ftparchive needs a valid config

[Aníbal Limón]

Hi again,

On 24/03/15 10:34, Aníbal Limón wrote:
> Hi Andreas,
>
> Your solution is better but i think we need to generalize the code in 
> DpkgPM::_create_configs to avoid code duplicate.

I reviewed the code and the issue is why PM's and Indexer's are 
different classes i think they need to share common information.
For now your patch is the best solution, now i'm doing a little 
improvements and i'll send v2 with your signature.

Cheers,
     alimon

>
> Comments?
>
> Cheers,
>     alimon
>
> On 24/03/15 05:58, Andreas Oberritter wrote:
>> Hello Aníbal,
>>
>> On 23.03.2015 17:05, Aníbal Limón wrote:
>>> Since we have support of log checking in deb/ipk [1] rootfs generation
>>> in non Debian-based hosts fails because apt-ftparchive generates a
>>> warn when not find /etc/apt/apt.conf.d/ (available in Debian-based
>>> hosts).
>>>
>>> In order to fix,
>>>
>>>     package_manager.py: DpkgPMIndexer add support for export APT_CONF
>>>     to environment.
>>>
>>> [1] 
>>> http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=86aec93902af2e2d7d73ca9a643707fcca45055c
>>>
>>> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
>>> ---
>>>   meta/lib/oe/package_manager.py | 9 ++++++++-
>>>   1 file changed, 8 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/meta/lib/oe/package_manager.py 
>>> b/meta/lib/oe/package_manager.py
>>> index c9a8084..395c0d01 100644
>>> --- a/meta/lib/oe/package_manager.py
>>> +++ b/meta/lib/oe/package_manager.py
>>> @@ -169,7 +169,14 @@ class OpkgIndexer(Indexer):
>>>       class DpkgIndexer(Indexer):
>>> +    def __init__(self, d, deploy_dir, apt_conf_file=None):
>>> +        self.apt_conf_file = apt_conf_file
>>> +        Indexer.__init__(self, d, deploy_dir)
>>> +
>>>       def write_index(self):
>>> +        if not self.apt_conf_file is None:
>>> +            os.environ['APT_CONFIG'] = self.apt_conf_file
>>> +
>>>           pkg_archs = self.d.getVar('PACKAGE_ARCHS', True)
>>>           if pkg_archs is not None:
>>>               arch_list = pkg_archs.split()
>>> @@ -1507,7 +1514,7 @@ class DpkgPM(PackageManager):
>>>             self._create_configs(archs, base_archs)
>>>   -        self.indexer = DpkgIndexer(self.d, self.deploy_dir)
>>> +        self.indexer = DpkgIndexer(self.d, self.deploy_dir, 
>>> self.apt_conf_file)
>>>         """
>>>       This function will change a package's status in 
>>> /var/lib/dpkg/status file.
>>>
>> unfortunately, this approach doesn't work with "bitbake 
>> package-index", which
>> calls generate_index_files() in package_manager.py. That's why I 
>> implemented
>> a quite ugly solution covering all cases:
>>
>>
>>  From 668fe52b58aba6ab47e4712ee460ed7dd06c948f Mon Sep 17 00:00:00 2001
>> From: Andreas Oberritter <obi@opendreambox.org>
>> Date: Wed, 8 Oct 2014 15:52:22 +0200
>> Subject: [PATCH] package_manager/deb: apt-ftparchive needs a valid 
>> config
>>
>> Without a valid config, apt-ftparchive aborts. Since apt-ftparchive
>> defaults to /etc/apt if APT_CONFIG is unset, this is only an issue
>> non non-Debian build hosts.
>>
>> Signed-off-by: Andreas Oberritter <obi@opendreambox.org>
>> ---
>>   meta/lib/oe/package_manager.py | 22 ++++++++++++++++++++++
>>   1 file changed, 22 insertions(+)
>>
>> diff --git a/meta/lib/oe/package_manager.py 
>> b/meta/lib/oe/package_manager.py
>> index d4ac174..71f909c 100644
>> --- a/meta/lib/oe/package_manager.py
>> +++ b/meta/lib/oe/package_manager.py
>> @@ -182,6 +182,28 @@ class DpkgIndexer(Indexer):
>>           all_mlb_pkg_arch_list = 
>> (self.d.getVar('ALL_MULTILIB_PACKAGE_ARCHS', True) or 
>> "").replace('-', '_').split()
>>           arch_list.extend(arch for arch in all_mlb_pkg_arch_list if 
>> arch not in arch_list)
>>   +        apt_conf_dir = 
>> self.d.expand("${APTCONF_TARGET}/apt-ftparchive")
>> +        apt_conf_file = os.path.join(apt_conf_dir, "apt.conf")
>> +
>> +        bb.utils.mkdirhier(apt_conf_dir)
>> +        bb.utils.mkdirhier(apt_conf_dir + "/lists/partial/")
>> +        bb.utils.mkdirhier(apt_conf_dir + "/apt.conf.d/")
>> +        bb.utils.mkdirhier(apt_conf_dir + "/preferences.d/")
>> +
>> +        with open(os.path.join(apt_conf_dir, "preferences"), "w") as 
>> prefs_file:
>> +            pass
>> +        with open(os.path.join(apt_conf_dir, "sources.list"), "w+") 
>> as sources_file:
>> +            pass
>> +
>> +        with open(apt_conf_file, "w") as apt_conf:
>> +            with 
>> open(self.d.expand("${STAGING_ETCDIR_NATIVE}/apt/apt.conf.sample")) 
>> as apt_conf_sample:
>> +                for line in apt_conf_sample.read().split("\n"):
>> +                    line = re.sub("#ROOTFS#", "/dev/null", line)
>> +                    line = re.sub("#APTCONF#", apt_conf_dir, line)
>> +                    apt_conf.write(line + "\n")
>> +
>> +        os.environ['APT_CONFIG'] = apt_conf_file
>> +
>>           apt_ftparchive = bb.utils.which(os.getenv('PATH'), 
>> "apt-ftparchive")
>>           gzip = bb.utils.which(os.getenv('PATH'), "gzip")
>

Re: [PATCH] package_manager/deb: apt-ftparchive needs a valid config

[Andreas Oberritter]

On 24.03.2015 18:33, Aníbal Limón wrote:
> I reviewed the code and the issue is why PM's and Indexer's are
> different classes i think they need to share common information.
> For now your patch is the best solution, now i'm doing a little
> improvements and i'll send v2 with your signature.

Thank you, Aníbal!

Btw., I think your development machine has a wrong timezone setting.
Your patches appear to have a time offset of about seven hours.

Regards,
Andreas