[PATCH 0/4] Persistent /var/log support

[PATCH 0/4] Persistent /var/log support

[Chen Qi]

The following changes since commit a675b2c89e477af088faee9b3be96eae19a85f0b:

  sanity.bbclass: fix logging of an error (2016-11-15 15:18:50 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib ChenQi/persistent-var-log
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=ChenQi/persistent-var-log

Chen Qi (4):
  bitbake.conf: add VOLATILE_LOG_DIR variable
  base-files: respect VOLATILE_LOG_DIR
  initscripts: support persistent /var/log
  package.bbclass: support persistent /var/log

 meta/classes/package.bbclass                       |  2 +-
 meta/conf/bitbake.conf                             |  3 +
 meta/files/fs-perms-persistent-log.txt             | 69 ++++++++++++++++++++++
 meta/recipes-core/base-files/base-files_3.0.14.bb  |  4 +-
 .../initscripts/initscripts-1.0/volatiles          |  1 -
 meta/recipes-core/initscripts/initscripts_1.0.bb   |  3 +
 6 files changed, 78 insertions(+), 4 deletions(-)
 create mode 100644 meta/files/fs-perms-persistent-log.txt

-- 
1.9.1

[PATCH 1/4] bitbake.conf: add VOLATILE_LOG_DIR variable

[Chen Qi]

The default value is "yes" which results in the /var/log being a link
pointing to /var/volatile/log which is on tmpfs.

The user could override this value to "no" which causes /var/log to be
a directory on persistent storage.

[YOCTO #6132]

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/conf/bitbake.conf | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 1472e8f..794f422 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -83,6 +83,9 @@ USRBINPATH_class-nativesdk = "/usr/bin"
 # Root home directory
 ROOT_HOME ??= "/home/root"
 
+# If set to "yes", /var/log links to /var/volatile/log; otherwise, /var/log is on persistent storage.
+VOLATILE_LOG_DIR ?= "yes"
+
 ##################################################################
 # Architecture-dependent build variables.
 ##################################################################
-- 
1.9.1

[PATCH 2/4] base-files: respect VOLATILE_LOG_DIR

[Chen Qi]

Respect VOLATILE_LOG_DIR variable. In this way, if the user overrides
this variable to be "no", /var/log on the final image would reside on
persistent storage.

[YOCTO #6132]

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/recipes-core/base-files/base-files_3.0.14.bb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/base-files/base-files_3.0.14.bb b/meta/recipes-core/base-files/base-files_3.0.14.bb
index 5333110..7cc8e8c 100644
--- a/meta/recipes-core/base-files/base-files_3.0.14.bb
+++ b/meta/recipes-core/base-files/base-files_3.0.14.bb
@@ -41,7 +41,7 @@ dirs755 = "/bin /boot /dev ${sysconfdir} ${sysconfdir}/default \
            ${localstatedir}/backups ${localstatedir}/lib \
            /sys ${localstatedir}/lib/misc ${localstatedir}/spool \
            ${localstatedir}/volatile \
-           ${localstatedir}/volatile/log \
+           ${localstatedir}/${@['', 'volatile/'][d.getVar('VOLATILE_LOG_DIR', True) == 'yes']}log \
            /home ${prefix}/src ${localstatedir}/local \
            /media"
 
@@ -52,7 +52,7 @@ dirs755-lsb = "/srv  \
                ${prefix}/lib/locale"
 dirs2775-lsb = "/var/mail"
 
-volatiles = "log tmp"
+volatiles = "${@['', 'log'][d.getVar('VOLATILE_LOG_DIR', True) == 'yes']} tmp"
 conffiles = "${sysconfdir}/debian_version ${sysconfdir}/host.conf \
              ${sysconfdir}/issue /${sysconfdir}/issue.net \
              ${sysconfdir}/nsswitch.conf ${sysconfdir}/profile \
-- 
1.9.1

[PATCH 3/4] initscripts: support persistent /var/log

[Chen Qi]

Respect VOLATILE_VAR_LOG variable so that if it's set to "no", we could
have persistent /var/log on the final image.

[YOCTO #6132]

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/recipes-core/initscripts/initscripts-1.0/volatiles | 1 -
 meta/recipes-core/initscripts/initscripts_1.0.bb        | 3 +++
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-core/initscripts/initscripts-1.0/volatiles b/meta/recipes-core/initscripts/initscripts-1.0/volatiles
index 297245d..6cccab7 100644
--- a/meta/recipes-core/initscripts/initscripts-1.0/volatiles
+++ b/meta/recipes-core/initscripts/initscripts-1.0/volatiles
@@ -25,7 +25,6 @@ d root root 1777 /run/lock none
 d root root 0755 /var/volatile/log none
 d root root 1777 /var/volatile/tmp none
 l root root 1777 /var/lock /run/lock
-l root root 0755 /var/log /var/volatile/log
 l root root 0755 /var/run /run
 l root root 1777 /var/tmp /var/volatile/tmp
 l root root 1777 /tmp /var/tmp
diff --git a/meta/recipes-core/initscripts/initscripts_1.0.bb b/meta/recipes-core/initscripts/initscripts_1.0.bb
index 8f110b0..14cdf6a 100644
--- a/meta/recipes-core/initscripts/initscripts_1.0.bb
+++ b/meta/recipes-core/initscripts/initscripts_1.0.bb
@@ -102,6 +102,9 @@ do_install () {
 	install -m 0755    ${WORKDIR}/read-only-rootfs-hook.sh ${D}${sysconfdir}/init.d
 	install -m 0755    ${WORKDIR}/save-rtc.sh	${D}${sysconfdir}/init.d
 	install -m 0644    ${WORKDIR}/volatiles		${D}${sysconfdir}/default/volatiles/00_core
+	if [ "${VOLATILE_LOG_DIR}" = "log" ]; then
+		echo "l root root 0755 /var/log /var/volatile/log" >> ${D}${sysconfdir}/default/volatiles/00_core
+	fi
 	install -m 0755    ${WORKDIR}/dmesg.sh		${D}${sysconfdir}/init.d
 	install -m 0644    ${WORKDIR}/logrotate-dmesg.conf ${D}${sysconfdir}/
 
-- 
1.9.1

[PATCH 4/4] package.bbclass: support persistent /var/log

[Chen Qi]

Add a new file, fs-perms-persistent-log.txt, which treats /var/log
as a directory instead of a link.

Modify package.bbclass to use this file if VOLATILE_LOG_DIR is not set to "yes".

[YOCTO #6132]

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/classes/package.bbclass           |  2 +-
 meta/files/fs-perms-persistent-log.txt | 69 ++++++++++++++++++++++++++++++++++
 2 files changed, 70 insertions(+), 1 deletion(-)
 create mode 100644 meta/files/fs-perms-persistent-log.txt

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index a6f0a7a..9a423f8 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -733,7 +733,7 @@ python fixup_perms () {
         bbpath = d.getVar('BBPATH', True)
         fs_perms_tables = d.getVar('FILESYSTEM_PERMS_TABLES', True)
         if not fs_perms_tables:
-            fs_perms_tables = 'files/fs-perms.txt'
+            fs_perms_tables = ['files/fs-perms-persistent-log.txt', 'files/fs-perms.txt'][d.getVar('VOLATILE_LOG_DIR', True) == 'yes']
         for conf_file in fs_perms_tables.split():
             str += " %s" % bb.utils.which(bbpath, conf_file)
         return str
diff --git a/meta/files/fs-perms-persistent-log.txt b/meta/files/fs-perms-persistent-log.txt
new file mode 100644
index 0000000..2487a68
--- /dev/null
+++ b/meta/files/fs-perms-persistent-log.txt
@@ -0,0 +1,69 @@
+# This file contains a list of files and directories with known permissions.
+# It is used by the packaging class to ensure that the permissions, owners and
+# group of listed files and directories are in sync across the system.
+#
+# The format of this file 
+#
+#<path>	<mode>	<uid>	<gid>	<walk>	<fmode>	<fuid>	<fgid>
+#
+# or
+#
+#<path> link <target>
+#
+# <path>: directory path
+# <mode>: mode for directory
+# <uid>:  uid for directory
+# <gid>:  gid for directory
+# <walk>: recursively walk the directory?  true or false
+# <fmode>: if walking, new mode for files
+# <fuid>:  if walking, new uid for files
+# <fgid>:  if walking, new gid for files
+# <target>: turn the directory into a symlink point to target
+#
+# in mode, uid or gid, a "-" means don't change any existing values
+#
+# /usr/src		0755	root	root	false	-	-	-
+# /usr/share/man	0755	root	root	true	0644	root	root
+
+# Note: all standard config directories are automatically assigned "0755 root root false - - -"
+
+# Documentation should always be corrected
+${mandir}		0755	root	root	true	0644	root	root
+${infodir}		0755	root	root	true	0644	root	root
+${docdir}		0755	root	root	true	0644	root	root
+${datadir}/gtk-doc	0755	root	root	true	0644	root	root
+
+# Fixup locales
+${datadir}/locale	0755	root	root	true	0644	root	root
+
+# Cleanup headers
+${includedir}		0755	root	root	true	0644	root	root
+${oldincludedir}	0755	root	root	true	0644	root	root
+
+# Cleanup debug src
+/usr/src/debug		0755	root	root	true	-	root	root
+
+# Items from base-files
+# Links
+${localstatedir}/run	link	/run
+${localstatedir}/lock	link	/run/lock
+${localstatedir}/tmp	link	volatile/tmp
+
+/home				0755	root	root	false - - -
+/srv				0755	root	root	false - - -
+${prefix}/src			0755	root	root	false - - -
+${localstatedir}/local		0755	root	root	false - - -
+
+# Special permissions from base-files
+# Set 1777
+/tmp				01777	root	root	false - - -
+${localstatedir}/volatile/tmp	01777	root	root	false - - -
+
+# Set 0700
+${ROOT_HOME}			0700	root	root	false - - -
+
+# Set 755-lsb
+/srv				0755	root	root	false - - -
+
+# Set 2775-lsb
+/var/mail			02775	root	mail	false - - -
-- 
1.9.1

Re: [PATCH 0/4] Persistent /var/log support

[Jack Mitchell]

On 22/11/16 10:10, Chen Qi wrote:
> The following changes since commit a675b2c89e477af088faee9b3be96eae19a85f0b:
>
>   sanity.bbclass: fix logging of an error (2016-11-15 15:18:50 +0000)
>
> are available in the git repository at:
>
>   git://git.openembedded.org/openembedded-core-contrib ChenQi/persistent-var-log
>   http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=ChenQi/persistent-var-log
>
> Chen Qi (4):
>   bitbake.conf: add VOLATILE_LOG_DIR variable
>   base-files: respect VOLATILE_LOG_DIR
>   initscripts: support persistent /var/log
>   package.bbclass: support persistent /var/log
>
>  meta/classes/package.bbclass                       |  2 +-
>  meta/conf/bitbake.conf                             |  3 +
>  meta/files/fs-perms-persistent-log.txt             | 69 ++++++++++++++++++++++
>  meta/recipes-core/base-files/base-files_3.0.14.bb  |  4 +-
>  .../initscripts/initscripts-1.0/volatiles          |  1 -
>  meta/recipes-core/initscripts/initscripts_1.0.bb   |  3 +
>  6 files changed, 78 insertions(+), 4 deletions(-)
>  create mode 100644 meta/files/fs-perms-persistent-log.txt
>

Ack from me, I don't know if this is the best way to do it, but it is 
way way way too hard to get persistent logging in OE. I've had to 
implement it in numerous ways, multiple times over the past few years 
and it shouldn't be that hard.

Re: [PATCH 2/4] base-files: respect VOLATILE_LOG_DIR

[Patrick Ohly]

On Tue, 2016-11-22 at 18:10 +0800, Chen Qi wrote:
> -           ${localstatedir}/volatile/log \
> +           ${localstatedir}/${@['',
> 'volatile/'][d.getVar('VOLATILE_LOG_DIR', True) == 'yes']}log \

I find the "if/else" expression easier to read, there's no need for an
explicit d.getVar(), and there's an utility method for variables which
are meant to be booleans:
   ${@ 'volatile/' if oe.types.boolean('${VOLATILE_LOG_DIR'}) else '' }

Using the utility function ensures that typos are caught and allows
several other values besides yes/no.

The change itself of course works either way. Just my 2 cents ;-}

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH 3/4] initscripts: support persistent /var/log

[Patrick Ohly]

On Tue, 2016-11-22 at 18:10 +0800, Chen Qi wrote:
> ${WORKDIR}/volatiles         ${D}${sysconfdir}/default/volatiles/00_core
> +       if [ "${VOLATILE_LOG_DIR}" = "log" ]; then
> +               echo "l root root 0755 /var/log /var/volatile/log" >>
> ${D}${sysconfdir}/default/volatiles/00_core
> +       fi

Why are you checking for "log" here? I thought VOLATILE_LOG_DIR was
supposed to be a boolean?

Using oe.types.boolean in shell code is a bit awkward, but would be
necessary for the sake of consistency when allowing VOLATILE_LOG_DIR to
have more than just yes/no. Untested (relies on bool->str conversion):

if [ ${@ oe.types.boolean('${VOLATILE_LOG_DIR}') } = True ]; then

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH 3/4] initscripts: support persistent /var/log

[ChenQi]

On 12/12/2016 04:05 PM, Patrick Ohly wrote:
> On Tue, 2016-11-22 at 18:10 +0800, Chen Qi wrote:
>> ${WORKDIR}/volatiles         ${D}${sysconfdir}/default/volatiles/00_core
>> +       if [ "${VOLATILE_LOG_DIR}" = "log" ]; then
>> +               echo "l root root 0755 /var/log /var/volatile/log" >>
>> ${D}${sysconfdir}/default/volatiles/00_core
>> +       fi
> Why are you checking for "log" here? I thought VOLATILE_LOG_DIR was
> supposed to be a boolean?
>
> Using oe.types.boolean in shell code is a bit awkward, but would be
> necessary for the sake of consistency when allowing VOLATILE_LOG_DIR to
> have more than just yes/no. Untested (relies on bool->str conversion):
>
> if [ ${@ oe.types.boolean('${VOLATILE_LOG_DIR}') } = True ]; then
>

Thanks for your suggestion.

I'll send out V2.

Best Regards,

Chen Qi

[PATCH 3/4] initscripts: support persistent /var/log

[Chen Qi]

Respect VOLATILE_VAR_LOG variable so that if it's set to any valid boolean
false value, we could have persistent /var/log on the final image.

[YOCTO #6132]

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/recipes-core/initscripts/initscripts-1.0/volatiles | 1 -
 meta/recipes-core/initscripts/initscripts_1.0.bb        | 3 +++
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-core/initscripts/initscripts-1.0/volatiles b/meta/recipes-core/initscripts/initscripts-1.0/volatiles
index bc17c45..2011066 100644
--- a/meta/recipes-core/initscripts/initscripts-1.0/volatiles
+++ b/meta/recipes-core/initscripts/initscripts-1.0/volatiles
@@ -27,7 +27,6 @@ d root root 1777 /run/lock none
 d root root 0755 /var/volatile/log none
 d root root 1777 /var/volatile/tmp none
 l root root 1777 /var/lock /run/lock
-l root root 0755 /var/log /var/volatile/log
 l root root 0755 /var/run /run
 l root root 1777 /var/tmp /var/volatile/tmp
 l root root 1777 /tmp /var/tmp
diff --git a/meta/recipes-core/initscripts/initscripts_1.0.bb b/meta/recipes-core/initscripts/initscripts_1.0.bb
index a65f1b2..7ab0d2b 100644
--- a/meta/recipes-core/initscripts/initscripts_1.0.bb
+++ b/meta/recipes-core/initscripts/initscripts_1.0.bb
@@ -102,6 +102,9 @@ do_install () {
 	install -m 0755    ${WORKDIR}/read-only-rootfs-hook.sh ${D}${sysconfdir}/init.d
 	install -m 0755    ${WORKDIR}/save-rtc.sh	${D}${sysconfdir}/init.d
 	install -m 0644    ${WORKDIR}/volatiles		${D}${sysconfdir}/default/volatiles/00_core
+	if [ ${@ oe.types.boolean('${VOLATILE_LOG_DIR}') } = True ]; then
+		echo "l root root 0755 /var/log /var/volatile/log" >> ${D}${sysconfdir}/default/volatiles/00_core
+	fi
 	install -m 0755    ${WORKDIR}/dmesg.sh		${D}${sysconfdir}/init.d
 	install -m 0644    ${WORKDIR}/logrotate-dmesg.conf ${D}${sysconfdir}/
 
-- 
1.9.1

[PATCH 3/4] initscripts: support persistent /var/log

[Chen Qi]

Respect VOLATILE_VAR_LOG variable so that if it's set to "", we could
have persistent /var/log on the final image.

[YOCTO #6132]

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/recipes-core/initscripts/initscripts-1.0/volatiles | 1 -
 meta/recipes-core/initscripts/initscripts_1.0.bb        | 3 +++
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-core/initscripts/initscripts-1.0/volatiles b/meta/recipes-core/initscripts/initscripts-1.0/volatiles
index 297245d..6cccab7 100644
--- a/meta/recipes-core/initscripts/initscripts-1.0/volatiles
+++ b/meta/recipes-core/initscripts/initscripts-1.0/volatiles
@@ -25,7 +25,6 @@ d root root 1777 /run/lock none
 d root root 0755 /var/volatile/log none
 d root root 1777 /var/volatile/tmp none
 l root root 1777 /var/lock /run/lock
-l root root 0755 /var/log /var/volatile/log
 l root root 0755 /var/run /run
 l root root 1777 /var/tmp /var/volatile/tmp
 l root root 1777 /tmp /var/tmp
diff --git a/meta/recipes-core/initscripts/initscripts_1.0.bb b/meta/recipes-core/initscripts/initscripts_1.0.bb
index dfb75b2..14f4167 100644
--- a/meta/recipes-core/initscripts/initscripts_1.0.bb
+++ b/meta/recipes-core/initscripts/initscripts_1.0.bb
@@ -98,6 +98,9 @@ do_install () {
 	install -m 0755    ${WORKDIR}/read-only-rootfs-hook.sh ${D}${sysconfdir}/init.d
 	install -m 0755    ${WORKDIR}/save-rtc.sh	${D}${sysconfdir}/init.d
 	install -m 0644    ${WORKDIR}/volatiles		${D}${sysconfdir}/default/volatiles/00_core
+	if [ "${VOLATILE_LOG_DIR}" = "log" ]; then
+		echo "l root root 0755 /var/log /var/volatile/log" >> ${D}${sysconfdir}/default/volatiles/00_core
+	fi
 	install -m 0755    ${WORKDIR}/dmesg.sh		${D}${sysconfdir}/init.d
 	install -m 0644    ${WORKDIR}/logrotate-dmesg.conf ${D}${sysconfdir}/
 
-- 
1.9.1