[PATCH 0/2] Change the way of CONFFILES handling

[PATCH 0/2] Change the way of CONFFILES handling

[Chen Qi]

This patchset include two patches.
The first one is the basic change.
The second one provides an example of how to fix CONFFILES in each recipe after this change, if the previously value of
CONFFILES in that recipe is not correct or redundant.

The following changes since commit 68ddb28a68ceb59cd1ed322c16143827ce1ac712:

  distrodata_class: checkpkg make usage of latest_versionstring methods in bitbake fetcher (2014-11-28 13:59:52 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib ChenQi/CONFFILES
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=ChenQi/CONFFILES

Chen Qi (2):
  packaging: allow globs in CONFFILES
  base-files: fix CONFFILES

 meta/classes/package.bbclass                      | 35 +++++++++++++++++++++++
 meta/classes/package_deb.bbclass                  |  2 +-
 meta/classes/package_ipk.bbclass                  |  2 +-
 meta/classes/package_rpm.bbclass                  |  2 +-
 meta/conf/bitbake.conf                            |  2 ++
 meta/recipes-core/base-files/base-files_3.0.14.bb |  3 --
 6 files changed, 40 insertions(+), 6 deletions(-)

-- 
1.9.1

[PATCH 1/2] packaging: allow globs in CONFFILES

[Chen Qi]

Allow globs in CONFFILES and default CONFFILES to '${sysconfdir}'.

Previously, the CONFFILES variable is a list of config files to be
packaged. However, as these files may come directly from the source
instead of our WORKDIR, it's a very common mistake that we forget to set
this variable in our recipes. This will lead to unexpected behavior
when doing an on-target upgrade. For example, we modify the /etc/login.defs
locally and then do an upgrade. The file will be replaced, which is
obviously not correct.

This patch changes the way of CONFFILES handling. After this  change,
the CONFFILES can take the same form as FILES. That means, we don't
have to list a bunch of files for CONFFILES. It will just be expanded
like the FILES variable.

As almost all files under /etc are basically configuration files, we
provide a default value for CONFFILES.

    CONNFFILES = "${sysconfdir}"

In this way, we don't need to modify every recipe to set the CONFFILES
variable. Of course, setting CONFFILES in recipes take precedence over
the CONFFILES. For example, if the recipe author decides that package A
should only treat files under ${sysconfdir}/default/ as config files,
he/she can write like this.

    CONFFILES_A = "${sysconfdir}/default"

The above situation should not be common. As according to FHS, the /etc
directory is a place for all system related configuration files.

[YOCTO #5200]

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/classes/package.bbclass     | 35 +++++++++++++++++++++++++++++++++++
 meta/classes/package_deb.bbclass |  2 +-
 meta/classes/package_ipk.bbclass |  2 +-
 meta/classes/package_rpm.bbclass |  2 +-
 meta/conf/bitbake.conf           |  2 ++
 5 files changed, 40 insertions(+), 3 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 510617c..ae5ca20 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -237,6 +237,41 @@ python () {
         d.appendVarFlag('do_package', 'deptask', " do_packagedata")
 }
 
+# Called in package_<rpm,ipk,deb>.bbclass to get the correct list of configuration files
+def get_conffiles(pkg, d):
+    import glob
+    pkgdest = d.getVar('PKGDEST', True)
+    conf_list = []
+    root = os.path.join(pkgdest, pkg)
+    cwd = os.getcwd()
+    os.chdir(root)
+    conffiles = (d.getVar('CONFFILES_%s' % pkg, True) or d.getVar('CONFFILES', True) or "").split()
+    for conffile in conffiles:
+        if os.path.isabs(conffile):
+            conffile = '.' + conffile
+        if not conffile.startswith('./'):
+            conffile = './' + conffile
+        if not os.path.islink(conffile) and os.path.isdir(conffile):
+            newconffiles = [ os.path.join(conffile, x) for x in os.listdir(conffile) ]
+            if newconffiles:
+                conffiles += newconffiles
+                continue
+        globbed = glob.glob(conffile)
+        if globbed:
+            if [ conffile ] != globbed:
+                conffile += globbed
+                continue
+        if (not os.path.islink(conffile)) and (not os.path.exists(conffile)):
+            continue
+        if conffile in conf_list:
+            continue
+        conf_list.append(conffile)
+    # Remove the leading './'
+    for i in range(0, len(conf_list)):
+        conf_list[i] = conf_list[i][1:]
+    os.chdir(cwd)
+    return conf_list
+
 def splitdebuginfo(file, debugfile, debugsrcdir, sourcefile, d):
     # Function to split a single file into two components, one is the stripped
     # target system binary, the other contains any debugging information. The
diff --git a/meta/classes/package_deb.bbclass b/meta/classes/package_deb.bbclass
index 5b5f7e2..9d7c59b 100644
--- a/meta/classes/package_deb.bbclass
+++ b/meta/classes/package_deb.bbclass
@@ -262,7 +262,7 @@ python do_package_deb () {
             scriptfile.close()
             os.chmod(os.path.join(controldir, script), 0755)
 
-        conffiles_str = localdata.getVar("CONFFILES", True)
+        conffiles_str = ' '.join(get_conffiles(pkg, d))
         if conffiles_str:
             try:
                 conffiles = open(os.path.join(controldir, 'conffiles'), 'w')
diff --git a/meta/classes/package_ipk.bbclass b/meta/classes/package_ipk.bbclass
index 44fd3eb..dba6804 100644
--- a/meta/classes/package_ipk.bbclass
+++ b/meta/classes/package_ipk.bbclass
@@ -226,7 +226,7 @@ python do_package_ipk () {
             scriptfile.close()
             os.chmod(os.path.join(controldir, script), 0755)
 
-        conffiles_str = localdata.getVar("CONFFILES", True)
+        conffiles_str = ' '.join(get_conffiles(pkg, d))
         if conffiles_str:
             try:
                 conffiles = open(os.path.join(controldir, 'conffiles'), 'w')
diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index 92ddf7a..b87e634 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -324,7 +324,7 @@ python write_specfile () {
 
         bb.data.update_data(localdata)
 
-        conffiles = (localdata.getVar('CONFFILES', True) or "").split()
+        conffiles = get_conffiles(pkg, d)
         dirfiles = localdata.getVar('DIRFILES', True)
         if dirfiles is not None:
             dirfiles = dirfiles.split()
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 0ccaac0..dd3b41b 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -265,6 +265,8 @@ PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE
 PACKAGES_DYNAMIC = "^${PN}-locale-.*"
 FILES = ""
 
+CONFFILES = "${sysconfdir}"
+
 FILES_${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \
             ${sysconfdir} ${sharedstatedir} ${localstatedir} \
             ${base_bindir}/* ${base_sbindir}/* \
-- 
1.9.1

[PATCH 2/2] base-files: fix CONFFILES

[Chen Qi]

Remove CONFFILES setting in this recipe and use the default value.
In this way, all files under /etc would be treated as config files.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/recipes-core/base-files/base-files_3.0.14.bb | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/meta/recipes-core/base-files/base-files_3.0.14.bb b/meta/recipes-core/base-files/base-files_3.0.14.bb
index 07f5c54..d896a75 100644
--- a/meta/recipes-core/base-files/base-files_3.0.14.bb
+++ b/meta/recipes-core/base-files/base-files_3.0.14.bb
@@ -149,6 +149,3 @@ FILES_${PN} = "/"
 FILES_${PN}-doc = "${docdir} ${datadir}/common-licenses"
 
 PACKAGE_ARCH = "${MACHINE_ARCH}"
-
-CONFFILES_${PN} = "${sysconfdir}/fstab ${@['', '${sysconfdir}/hostname'][(d.getVar('hostname', True) != '')]} ${sysconfdir}/shells"
-
-- 
1.9.1

Re: [PATCH 1/2] packaging: allow globs in CONFFILES

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 319 bytes --]

On 1 December 2014 at 06:01, Chen Qi <Qi.Chen@windriver.com> wrote:

> +def get_conffiles(pkg, d):
>

Presumably this function is mostly a copy-and-paste of the FILES_* globbing
code also in package.bbclass.  Can the common code be extracted into a
shared function so that the two copies don't diverge?

Ross

[-- Attachment #2: Type: text/html, Size: 730 bytes --]

Re: [PATCH 1/2] packaging: allow globs in CONFFILES

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 747 bytes --]

On 1 December 2014 at 06:01, Chen Qi <Qi.Chen@windriver.com> wrote:

> Allow globs in CONFFILES and default CONFFILES to '${sysconfdir}'.
>

If I apply this patch and build core-image-sato with package_ipk:

| *** Error: CONTROL/conffiles mentions conffile /etc/depmod.d which does
not exist
| *** Error: CONTROL/conffiles mentions conffile /etc/modprobe.d which does
not existERROR: Task 1120 (/home/ross/Yocto/poky/meta/recipes-kernel/kmod/
kmod_git.bb, do_package_write_ipk) failed with exit code '1'

| *** Error: CONTROL/conffiles mentions conffile /etc/udev/hwdb.d which
does not exist
ERROR: Task 558 (/home/ross/Yocto/poky/meta/recipes-core/systemd/
systemd_216.bb, do_package_write_ipk) failed with exit code '1'

Ross

[-- Attachment #2: Type: text/html, Size: 1551 bytes --]

Re: [PATCH 1/2] packaging: allow globs in CONFFILES

[ChenQi]

[-- Attachment #1: Type: text/plain, Size: 1076 bytes --]

On 12/04/2014 04:48 AM, Burton, Ross wrote:
>
> On 1 December 2014 at 06:01, Chen Qi <Qi.Chen@windriver.com 
> <mailto:Qi.Chen@windriver.com>> wrote:
>
>     Allow globs in CONFFILES and default CONFFILES to '${sysconfdir}'.
>
>
> If I apply this patch and build core-image-sato with package_ipk:
>
> | *** Error: CONTROL/conffiles mentions conffile /etc/depmod.d which 
> does not exist
> | *** Error: CONTROL/conffiles mentions conffile /etc/modprobe.d which 
> does not existERROR: Task 1120 
> (/home/ross/Yocto/poky/meta/recipes-kernel/kmod/kmod_git.bb 
> <http://kmod_git.bb>, do_package_write_ipk) failed with exit code '1'
>
> | *** Error: CONTROL/conffiles mentions conffile /etc/udev/hwdb.d 
> which does not exist
> ERROR: Task 558 
> (/home/ross/Yocto/poky/meta/recipes-core/systemd/systemd_216.bb 
> <http://systemd_216.bb>, do_package_write_ipk) failed with exit code '1'
>
> Ross

Hi Ross,

Thanks for pointing these problems out.
I'll send out a V2 with common code extracted and the above problem fixed.

Best Regards,
Chen Qi

[-- Attachment #2: Type: text/html, Size: 2824 bytes --]