Kernel crashing and log buffers...

Kernel crashing and log buffers...

[Robin Getz]

On 17 Oct 2007, after much discussion and debate, Mike added add two new 
functions for reading the kernel log buffer (from kernel space).

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=0b15d04af3dd996035d8fa81fc849d049171f9c3

The intention was for them to be used by recovery/dump/debug code so the 
kernel log can be easily retrieved/parsed by the bootloader (or another 
kernel) in a crash scenario.

I was going to push the arch specific recovery/dump/debug code that uses them 
upstream (yeah, it has been a little while - but anyway...) it was removed 
since then by Roel Kluin ...

21 Oct 2008:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=acff181d3574244e651913df77332e897b88bff4

Before I ask Andrew to add it back, I thought I would make sure it was still a 
useful function, and did everything everyone wanted - and wasn't deemed 
unnecessary by a feature/function that I wasn't aware of - like the next 
thing...

I saw the patch Grant sent recently - Add Alternative Log Buffer Support for 
printk Messages (in Nov2008 to the embedded list, and Jan 2009 to lkml) - but 
I couldn't find any followups - and it doesn't seem to be in Linus's tree.

http://thread.gmane.org/gmane.linux.kernel.embedded/1358/focus=1373

http://lkml.org/lkml/2009/1/21/250

I can see the desire on Wolfgang & Grant's part - for not needing the copy 
from/to - (you never have to worry about crashing "nicely" - the kernel 
panics, but you still need to copy memory around - potentially causing all 
kinds of secondary issues - and masking the real reason the crash occurred).

But for the majority of the case - the copy from/to would work much better 
than what we have in mainstream today...


I would be interested in Paul and Russell - how have you solved this issue? 
(Or do your kernel's never crash? :)

Thanks
-Robin

Re: Kernel crashing and log buffers...

[David VomLehn]

On Wed, Jun 10, 2009 at 09:26:40PM -0400, Robin Getz wrote:
> On 17 Oct 2007, after much discussion and debate, Mike added add two new 
> functions for reading the kernel log buffer (from kernel space).
> 
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=0b15d04af3dd996035d8fa81fc849d049171f9c3
> 
> The intention was for them to be used by recovery/dump/debug code so the 
> kernel log can be easily retrieved/parsed by the bootloader (or another 
> kernel) in a crash scenario.
> 
> I was going to push the arch specific recovery/dump/debug code that uses them 
> upstream (yeah, it has been a little while - but anyway...) it was removed 
> since then by Roel Kluin ...
> 
> 21 Oct 2008:
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=acff181d3574244e651913df77332e897b88bff4
> 
> Before I ask Andrew to add it back, I thought I would make sure it was still a 
> useful function, and did everything everyone wanted - and wasn't deemed 
> unnecessary by a feature/function that I wasn't aware of - like the next 
> thing...
> 
> I saw the patch Grant sent recently - Add Alternative Log Buffer Support for 
> printk Messages (in Nov2008 to the embedded list, and Jan 2009 to lkml) - but 
> I couldn't find any followups - and it doesn't seem to be in Linus's tree.
> 
> http://thread.gmane.org/gmane.linux.kernel.embedded/1358/focus=1373
> 
> http://lkml.org/lkml/2009/1/21/250
> 
> I can see the desire on Wolfgang & Grant's part - for not needing the copy 
> from/to - (you never have to worry about crashing "nicely" - the kernel 
> panics, but you still need to copy memory around - potentially causing all 
> kinds of secondary issues - and masking the real reason the crash occurred).
> 
> But for the majority of the case - the copy from/to would work much better 
> than what we have in mainstream today...
> 
> 
> I would be interested in Paul and Russell - how have you solved this issue? 
> (Or do your kernel's never crash? :)

Our kernel does crash, so we have to do this. I had submitted a patch a
while ago that tweaks emit_log_char, but this was bounced, reasonably,
because it would be interferring with the great majority of people who
are not interested in capturing log output. The suggestion was made that
we do this with a console driver. I've recently got a version working, as
a module, but I've only started testing of the version integrated into the
kernel tree.  Still, post early, post often, so I've appended a version
of the patch here to see if this seems to be the right direction.

Regardless whether my particular patch seems like the right way to do this,
we do need to solve this problem for the embedded world. It is one of the
few areas where we have needs not shared by the rest of the kernel community.

> -Robin
------------------------------------ CUT HERE --------------------------------
Provide functions for capturing console output for storage. The primary user
is likely to be embedded systems that don't have the storage for core dumps
but do have a need to log kernel panic information for later evaluation. It
offers two main areas of functionality:

o	It can maintain a circular log of console output so that kernel log
	messages written before panic() was called can be retrieved to be
	added to the failure log.
o	A function can be registered to store output from printk() in a
	persistent location, such as a reserved location in RAM.  Then,
	printk() can be used either directly, to print state information, or
	indirectly, through standard functions like dump_stack() and
	show_regs().

During normal operation, we use the circular logging. When we crash, almost
the first thing we do is to switch to storing output. This goes in a memory
buffer that is preserved over reboots. We then write a detailed crash
report using printk() and functions that use printk(). We retrieve the last
n lines of the log before the crash and print it, so that gets captured
in the log, too.

Signed-off-by: David VomLehn <dvomlehn@cisco.com>
---
 drivers/char/Kconfig       |   14 +++
 drivers/char/Makefile      |    2 +
 drivers/char/conslogger.c  |  233 ++++++++++++++++++++++++++++++++++++++++++++
 include/linux/conslogger.h |   52 ++++++++++
 4 files changed, 301 insertions(+), 0 deletions(-)

diff --git a/drivers/char/Kconfig b/drivers/char/Kconfig
index 735bbe2..882ee57 100644
--- a/drivers/char/Kconfig
+++ b/drivers/char/Kconfig
@@ -684,6 +684,20 @@ config HVCS
 	  which will also be compiled when this driver is built as a
 	  module.
 
+config CONSLOGGER
+	tristate "Pseudo-console for capturing console output"
+	depends on PRINTK
+	default n
+	help
+	  This contains a pseudo-console to record and divert kernel console
+	  output, which is probably of most used to embedded systems. When
+	  a system crashes, it can divert printk output for logging information
+	  about the failure in some persistent location. Then the output from
+	  any function that uses printk() to display information, such as
+	  dump_stack() can be stored in the failure log. It also stores
+	  console output in a circular buffer so that that last <n> messages
+	  can be added to the failure log.
+
 config IBM_BSR
 	tristate "IBM POWER Barrier Synchronization Register support"
 	depends on PPC_PSERIES
diff --git a/drivers/char/Makefile b/drivers/char/Makefile
index 9caf5b5..ca62934 100644
--- a/drivers/char/Makefile
+++ b/drivers/char/Makefile
@@ -111,6 +111,8 @@ obj-$(CONFIG_PS3_FLASH)		+= ps3flash.o
 obj-$(CONFIG_JS_RTC)		+= js-rtc.o
 js-rtc-y = rtc.o
 
+obj-$(CONFIG_CONSLOGGER)	+= conslogger.o
+
 # Files generated that shall be removed upon make clean
 clean-files := consolemap_deftbl.c defkeymap.c
 
diff --git a/drivers/char/conslogger.c b/drivers/char/conslogger.c
new file mode 100644
index 0000000..ad44dc8
--- /dev/null
+++ b/drivers/char/conslogger.c
@@ -0,0 +1,233 @@
+/*
+ *				conslogger.c
+ *
+ * Console log diversion
+ *
+ * Copyright (C) 2005-2009 Scientific-Atlanta, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+ *
+ * Author:       David VomLehn
+ *
+ * This offers two functionalities. One is to continually record output from
+ * printk in a buffer for retrieval of history, while the second is to call
+ * a registered function so that printk output can be stored.
+ */
+
+#include <linux/kernel.h>
+#include <linux/console.h>
+#include <linux/gfp.h>
+#include <linux/slab.h>
+#include <linux/compiler.h>		/* For barrier() */
+#include <linux/conslogger.h>
+
+/* Return a value for indexing into the conslog array */
+#define BUF_IDX(conslog, idx)	((idx) & (conslog)->mask)
+
+static void conslog_write(struct console *c, const char *p, unsigned n);
+
+/**
+ * conslog_register - create and register a logging console
+ * @order:	Power of two of the log buffer size
+ *
+ * Returns an ERR_VALUE on error or a pointer to a &struct conslog on success.
+ */
+struct conslog *conslog_register(unsigned order)
+{
+	struct conslog *conslog;
+
+	conslog = kmalloc(sizeof(*conslog), GFP_KERNEL);
+
+	if (conslog == NULL)
+		conslog = ERR_PTR(-ENOMEM);
+
+	else {
+		size_t	size;
+		char	*buf;
+
+		memset(conslog, 0, sizeof(*conslog));
+		size = 1 << order;
+		buf = kmalloc(size, GFP_KERNEL);
+
+		if (buf == NULL) {
+			kfree(conslog);
+			conslog = ERR_PTR(-ENOMEM);
+		}
+
+		else {
+			conslog->size = size;
+			conslog->buf = buf;
+			conslog->mask = conslog->size - 1;
+			conslog->state = CONSLOG_DISABLE;
+
+			/* Initialize the console part of the structure */
+			conslog->console.data = conslog;
+
+			strlcpy(conslog->console.name, "console_logger",
+				sizeof(conslog->console.name));
+			conslog->console.write = conslog_write;
+			conslog->console.flags = CON_ENABLED;
+			conslog->console.index = -1;
+			register_console(&conslog->console);
+		}
+	}
+
+	return conslog;
+}
+
+/**
+ * conslog_unregister - Unregister a console logger
+ * @conslog:	Value returned by conslog_register
+ */
+int conslog_unregister(struct conslog *conslog)
+{
+	int	ret;
+
+	ret = unregister_console(&conslog->console);
+
+	if (conslog->buf != NULL)
+		kfree(conslog->buf);
+
+	return ret;
+}
+
+/**
+ * conslog_record - enable or disable storage of console to circular buffer
+ * @conslog:	Pointer to &struct conslog returned by conslog_register()
+ * @record:	True if data should be stored in the circular buffer, false
+ *		to disable data storage.
+ */
+void conslog_record(struct conslog *conslog, bool record)
+{
+	conslog->state = record ? CONSLOG_RECORD : CONSLOG_DISABLE;
+}
+
+/**
+ * conslog_divert - define a function to call with console output
+ * @conslog:	Pointer to &struct conslog returned by conslog_register()
+ * @divert:	Pointer to the function to call
+ */
+void conslog_divert(struct conslog *conslog,
+	void (*divert)(const char *p, size_t n))
+{
+	conslog->divert = divert;
+	conslog->state = CONSLOG_DIVERT;
+}
+
+/**
+ * conslog_write - console write function
+ * @c:	Pointer to the &console structure
+ * @p:	Pointer to data to write
+ * @n:	Number of bytes to write
+ *
+ * Called with console write data, which it either stores in the buffer,
+ * passes to the diversion function, or just ignores.
+ */
+static void conslog_write(struct console *c, const char *p, unsigned n)
+{
+	struct conslog *conslog;
+	static bool busy;
+
+	if (busy)
+		return;
+
+	busy = true;
+	barrier();
+	conslog = c->data;
+
+	switch (conslog->state) {
+	case CONSLOG_RECORD:
+		while (n-- != 0) {
+			conslog->buf[BUF_IDX(conslog, conslog->idx)] = *p++;
+			conslog->idx++;
+			if (unlikely(conslog->idx == conslog->size))
+				conslog->wrapped = true;
+		}
+		break;
+
+	case CONSLOG_DIVERT:
+		conslog->divert(p, n);
+		break;
+
+	case CONSLOG_DISABLE:
+		break;
+	}
+	busy = false;
+	barrier();
+}
+
+/**
+ * conslog_last - extract the last @n lines from recorded data
+ * @conslog:	Pointer to the &struct consolog from which to extract lines
+ * @nlines:	Number of lines to extract
+ * @report:	Function to report lines
+ *
+ * It is best to call this with recording disabled, though this is not
+ * enforced. See conslog_record().
+ */
+void conslog_last(struct conslog *conslog, unsigned nlines,
+	void (*report)(const char *p, unsigned cnt))
+{
+	unsigned	i;
+	unsigned	newest;
+	unsigned	oldest;
+	unsigned	n;
+
+	if (nlines == 0)
+		return;
+
+	oldest = conslog->wrapped ? conslog->idx - conslog->size : 0;
+
+	/* Set newest to be the index of the character most recently added to
+	 * the buffer, but if the most recent line ends with a newline, skip
+	 * the newline. This makes things work correctly even if the last
+	 * line was incomplete. */
+	newest = conslog->idx - 1;
+	if (newest != oldest && conslog->buf[BUF_IDX(conslog, newest)] == '\n')
+		newest--;
+
+	/* Scan backwards for the requested number of lines */
+	for (i = newest, n = 0; n != nlines && i != oldest; i--) {
+		if (conslog->buf[BUF_IDX(conslog, i)] == '\n')
+			n++;
+	}
+
+	/* We may be part-way through a line, in which case we need to scan
+	 * forward until we find a newline. Then we skip that newline. */
+	while (i != newest && conslog->buf[BUF_IDX(conslog, i)] != '\n')
+		i++;
+	i++;
+
+	/* We have n lines, let's print them */
+	for (; n != 0; n--) {
+		size_t	start;
+
+		start = i;
+
+		/* Search until we find the end of the line or we reach the
+		 * end of data. */
+		while (i != newest && conslog->buf[BUF_IDX(conslog, i)] != '\n')
+			i++;
+
+		/* If we stopped because i equaled newest, we need to increment
+		 * once more to get the correct end. */
+		if (i == newest)
+			i++;
+
+		report(conslog->buf + BUF_IDX(conslog, start), i - start);
+		i++;			/* Skip the newline */
+	}
+}
+
diff --git a/include/linux/conslogger.h b/include/linux/conslogger.h
new file mode 100644
index 0000000..77bcb90
--- /dev/null
+++ b/include/linux/conslogger.h
@@ -0,0 +1,52 @@
+/*
+ *				conslog.h
+ *
+ * Definitions for using the console diverter
+ *
+ * Copyright (C) 2005-2009 Scientific-Atlanta, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+ *
+ * Author:       David VomLehn
+ */
+
+#ifndef _INCLUDE_LINUX_CONSLOG_H_
+#define _INCLUDE_LINUX_CONSLOG_H_
+#include <linux/console.h>
+#include <linux/err.h>
+
+enum conslog_state {
+	CONSLOG_DISABLE, CONSLOG_RECORD, CONSLOG_DIVERT
+};
+
+struct conslog {
+	enum conslog_state state;
+	char		*buf;
+	unsigned	idx;
+	bool		wrapped;
+	size_t		mask;
+	size_t		size;
+	void		(*divert)(const char *p, size_t n);
+	struct console	console;
+};
+
+extern struct conslog *conslog_register(unsigned order);
+extern int conslog_unregister(struct conslog *conslog);
+extern void conslog_record(struct conslog *conslog, bool record);
+extern void conslog_divert(struct conslog *conslog,
+	void (*divert)(const char *p, size_t n));
+extern void conslog_last(struct conslog *conslog, unsigned nlines,
+	void (*report)(const char *p, unsigned cnt));
+#endif

Re: Kernel crashing and log buffers...

[Tim Bird]

David VomLehn wrote:
> Our kernel does crash, so we have to do this. I had submitted a patch a
> while ago that tweaks emit_log_char, but this was bounced, reasonably,
> because it would be interferring with the great majority of people who
> are not interested in capturing log output. The suggestion was made that
> we do this with a console driver. I've recently got a version working, as
> a module, but I've only started testing of the version integrated into the
> kernel tree.  Still, post early, post often, so I've appended a version
> of the patch here to see if this seems to be the right direction.
> 
> Regardless whether my particular patch seems like the right way to do this,
> we do need to solve this problem for the embedded world. It is one of the
> few areas where we have needs not shared by the rest of the kernel community.

In general, I like this approach to solve this problem.  I spend most of
my time in the lab with prototype boards that use a serial console, so my
terminal history has always been my crashdump repository. ;-)

This approach it unobtrusive to the rest of the log buffer system.  I never
did like the patches that introduced other ways to hack into the log
buffer.  The printk/log buffer code path is already convoluted enough,
and I didn't like the idea of adding additional hooks.  this approach
leverages an already existing hook.

I don't have much comment on the code specifics, but I have one nit
with the config help...

> diff --git a/drivers/char/Kconfig b/drivers/char/Kconfig
> index 735bbe2..882ee57 100644
> --- a/drivers/char/Kconfig
> +++ b/drivers/char/Kconfig
> @@ -684,6 +684,20 @@ config HVCS
>  	  which will also be compiled when this driver is built as a
>  	  module.
>  
> +config CONSLOGGER
> +	tristate "Pseudo-console for capturing console output"
> +	depends on PRINTK
> +	default n
> +	help
> +	  This contains a pseudo-console to record and divert kernel console
> +	  output, which is probably of most used to embedded systems. When
should be "most use" (no 'd')

> +	  a system crashes, it can divert printk output for logging information
> +	  about the failure in some persistent location. Then the output from
> +	  any function that uses printk() to display information, such as
> +	  dump_stack() can be stored in the failure log. It also stores
> +	  console output in a circular buffer so that that last <n> messages
> +	  can be added to the failure log.
> +
>  config IBM_BSR
>  	tristate "IBM POWER Barrier Synchronization Register support"
>  	depends on PPC_PSERIES

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: Kernel crashing and log buffers...

[Robin Getz]

On Thu 11 Jun 2009 13:53, David VomLehn pondered:
> On Wed, Jun 10, 2009 at 09:26:40PM -0400, Robin Getz wrote:
> > On 17 Oct 2007, after much discussion and debate, Mike added add 
> > two new functions for reading the kernel log buffer (from kernel space).
> > 
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=0b15d04af3dd996035d8fa81fc849d049171f9c3
> > 
> > The intention was for them to be used by recovery/dump/debug code
> > so the kernel log can be easily retrieved/parsed by the bootloader
> > (or another kernel) in a crash scenario.
> > 
> > I was going to push the arch specific recovery/dump/debug code 
> > that uses them upstream (yeah, it has been a little while - but 
> > anyway...) it was removed since then by Roel Kluin ...
> > 
> > 21 Oct 2008:
> > 
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=acff181d3574244e651913df77332e897b88bff4
> > 
> > Before I ask Andrew to add it back, I thought I would make sure it
> > was still a useful function, and did everything everyone wanted - 
> > and wasn't deemed unnecessary by a feature/function that I wasn't
> > aware of - like the next thing...
> > 
> > I saw the patch Grant sent recently - Add Alternative Log Buffer
> > Support for printk Messages (in Nov2008 to the embedded list, and 
> > Jan 2009 to lkml) - but I couldn't find any followups - and it 
> > doesn't seem to be in Linus's tree.
> > 
> > http://thread.gmane.org/gmane.linux.kernel.embedded/1358/focus=1373
> > 
> > http://lkml.org/lkml/2009/1/21/250
> > 
> > I can see the desire on Wolfgang & Grant's part - for not needing 
> > the copy from/to - (you never have to worry about crashing "nicely" - 
> > the kernel panics, but you still need to copy memory around - 
> > potentially causing all kinds of secondary issues - and masking the
> > real reason the crash occurred).
> > 
> > But for the majority of the case - the copy from/to would work much
> > better than what we have in mainstream today...
> > 
> > 
> > I would be interested in Paul and Russell - how have you solved this
> > issue? (Or do your kernel's never crash? :)
> 
> Our kernel does crash, so we have to do this. I had submitted a patch a
> while ago that tweaks emit_log_char, but this was bounced, reasonably,
> because it would be interferring with the great majority of people who
> are not interested in capturing log output. The suggestion was made that
> we do this with a console driver. I've recently got a version working,
> as a module, but I've only started testing of the version integrated 
> into the kernel tree.  Still, post early, post often, so I've appended 
> a version of the patch here to see if this seems to be the right 
> direction. 

Yeah - it looks like n people have done this n ways. 

I think even Andrew M had mentioned that he had done something in a past 
embedded life.

I think what you have would do what I need it to do - I'm not so sure about 
what Wolfgang/Grant had in mind (- since they wanted also to dump buffers 
from the bootloader into the console for syslog processing)...

> Regardless whether my particular patch seems like the right way to do
> this, we do need to solve this problem for the embedded world. It is
> one of the few areas where we have needs not shared by the rest of
> the kernel community.

I'm not so sure - googling for things last night led me to this:

http://www.faqs.org/patents/app/20090044051
EXTRACTING LOG AND TRACE BUFFERS IN THE EVENT OF SYSTEM CRASHES
Hariprasad V. Nellitheertha  - Linux Technology Center, India Software Labs, 
IBM India.

So - I think as Paul stated in a previous conversation - the high availability 
needs of servers are similar to the high availability needs of embedded...

Hari - did any of the work you did end up in mainline?

Thanks
-Robin

Re: Kernel crashing and log buffers...

[David VomLehn]

On Thu, Jun 11, 2009 at 02:52:38PM -0400, Robin Getz wrote:
> On Thu 11 Jun 2009 13:53, David VomLehn pondered:
> > On Wed, Jun 10, 2009 at 09:26:40PM -0400, Robin Getz wrote:
...
> > Our kernel does crash, so we have to do this. I had submitted a patch a
> > while ago that tweaks emit_log_char, but this was bounced, reasonably,
> > because it would be interferring with the great majority of people who
> > are not interested in capturing log output. The suggestion was made that
> > we do this with a console driver. I've recently got a version working,
> > as a module, but I've only started testing of the version integrated 
> > into the kernel tree.  Still, post early, post often, so I've appended 
> > a version of the patch here to see if this seems to be the right 
> > direction. 
> 
> Yeah - it looks like n people have done this n ways. 
> 
> I think even Andrew M had mentioned that he had done something in a past 
> embedded life.

It ain't hard, but I think we need a common implementation to go into the
kernel.

> I think what you have would do what I need it to do - I'm not so sure about 
> what Wolfgang/Grant had in mind (- since they wanted also to dump buffers 
> from the bootloader into the console for syslog processing)...

This is definitely different what what Wolfgang/Grant had in mind.

> > Regardless whether my particular patch seems like the right way to do
> > this, we do need to solve this problem for the embedded world. It is
> > one of the few areas where we have needs not shared by the rest of
> > the kernel community.
> 
> I'm not so sure - googling for things last night led me to this:
> 
> http://www.faqs.org/patents/app/20090044051
> EXTRACTING LOG AND TRACE BUFFERS IN THE EVENT OF SYSTEM CRASHES
> Hariprasad V. Nellitheertha  - Linux Technology Center, India Software Labs, 
> IBM India.
> 
> So - I think as Paul stated in a previous conversation - the high availability 
> needs of servers are similar to the high availability needs of embedded...

This is also different as it involves continuous logging of system state
rather than producing crash logs. The continuous logging is definitely of
wider interest. I did some work a couple of years ago to see what
it would take to specify the buffer for LTT FDR data, though this work was
dropped as lower priority, and I never got back to it.

> -Robin

David

Re: Kernel crashing and log buffers...

[David VomLehn]

On Thu, Jun 11, 2009 at 11:22:57AM -0700, Tim Bird wrote:
> David VomLehn wrote:
> > Our kernel does crash, so we have to do this. I had submitted a patch a
> > while ago that tweaks emit_log_char, but this was bounced, reasonably,
> > because it would be interferring with the great majority of people who
> > are not interested in capturing log output. The suggestion was made that
> > we do this with a console driver. I've recently got a version working, as
> > a module, but I've only started testing of the version integrated into the
> > kernel tree.  Still, post early, post often, so I've appended a version
> > of the patch here to see if this seems to be the right direction.
> > 
> > Regardless whether my particular patch seems like the right way to do this,
> > we do need to solve this problem for the embedded world. It is one of the
> > few areas where we have needs not shared by the rest of the kernel community.
> 
> In general, I like this approach to solve this problem.  I spend most of
> my time in the lab with prototype boards that use a serial console, so my
> terminal history has always been my crashdump repository. ;-)
> 
> This approach it unobtrusive to the rest of the log buffer system.  I never
> did like the patches that introduced other ways to hack into the log
> buffer.  The printk/log buffer code path is already convoluted enough,
> and I didn't like the idea of adding additional hooks.  this approach
> leverages an already existing hook.

The one thing that is missing over some other approaches is that log output
captured by this method will be filtered, just like all of the other consoles,
by the printk priority. It might be useful to have the ability to specify
the priority level for each console. So, you could have only KERN_CRIT and
above going to your serial console, but configure the logging console to
capture everything down to KERN_WARNING. This is a refinement, though, and I
didn't want to make things overly complex without a good feeling that it was
necessary.

> I don't have much comment on the code specifics, but I have one nit
> with the config help...
...
> > +	help
> > +	  This contains a pseudo-console to record and divert kernel console
> > +	  output, which is probably of most used to embedded systems. When
> should be "most use" (no 'd')

Thanks for picking the nit!

> Tim Bird

David VomLehn

Re: Kernel crashing and log buffers...

[Grant Erickson]

On 6/11/09 11:52 AM, Robin Getz wrote:
> On Thu 11 Jun 2009 13:53, David VomLehn pondered:
>> On Wed, Jun 10, 2009 at 09:26:40PM -0400, Robin Getz wrote:
>>> I would be interested in Paul and Russell - how have you solved this
>>> issue? (Or do your kernel's never crash? :)
>> 
>> Our kernel does crash, so we have to do this. I had submitted a patch a
>> while ago that tweaks emit_log_char, but this was bounced, reasonably,
>> because it would be interferring with the great majority of people who
>> are not interested in capturing log output. The suggestion was made that
>> we do this with a console driver. I've recently got a version working,
>> as a module, but I've only started testing of the version integrated
>> into the kernel tree.  Still, post early, post often, so I've appended
>> a version of the patch here to see if this seems to be the right
>> direction. 
> 
> Yeah - it looks like n people have done this n ways.
> 
> I think even Andrew M had mentioned that he had done something in a past
> embedded life.
> 
> I think what you have would do what I need it to do - I'm not so sure about
> what Wolfgang/Grant had in mind (- since they wanted also to dump buffers
> from the bootloader into the console for syslog processing)...

Regrettably, I have not had the time to follow-up on this since my last
commentary in March:

    http://lkml.org/lkml/2009/3/31/293

However, from what I can glean of the discussion, you are correct that this
does not precisely cover what Wolfgang (originally) or, later, I was working
towards in that neither boot loader logs nor round-trip log buffering
(bootloader to kernel and back again) are covered.

However, the value and what's being proposed here and what I and others
proposed--at least in the embedded kingdom--seems high and the features and
implementation needn't be mutually exclusive.

Regards,

Grant

Re: Kernel crashing and log buffers...

[Mike Frysinger]

On Thu, Jun 11, 2009 at 13:53, David VomLehn wrote:
> On Wed, Jun 10, 2009 at 09:26:40PM -0400, Robin Getz wrote:
>> On 17 Oct 2007, after much discussion and debate, Mike added add two new
>> functions for reading the kernel log buffer (from kernel space).
>>
>> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=0b15d04af3dd996035d8fa81fc849d049171f9c3
>>
>> The intention was for them to be used by recovery/dump/debug code so the
>> kernel log can be easily retrieved/parsed by the bootloader (or another
>> kernel) in a crash scenario.
>>
>> I was going to push the arch specific recovery/dump/debug code that uses them
>> upstream (yeah, it has been a little while - but anyway...) it was removed
>> since then by Roel Kluin ...
>>
>> 21 Oct 2008:
>> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=acff181d3574244e651913df77332e897b88bff4
>>
>> Before I ask Andrew to add it back, I thought I would make sure it was still a
>> useful function, and did everything everyone wanted - and wasn't deemed
>> unnecessary by a feature/function that I wasn't aware of - like the next
>> thing...
>>
>> I saw the patch Grant sent recently - Add Alternative Log Buffer Support for
>> printk Messages (in Nov2008 to the embedded list, and Jan 2009 to lkml) - but
>> I couldn't find any followups - and it doesn't seem to be in Linus's tree.
>>
>> http://thread.gmane.org/gmane.linux.kernel.embedded/1358/focus=1373
>>
>> http://lkml.org/lkml/2009/1/21/250
>>
>> I can see the desire on Wolfgang & Grant's part - for not needing the copy
>> from/to - (you never have to worry about crashing "nicely" - the kernel
>> panics, but you still need to copy memory around - potentially causing all
>> kinds of secondary issues - and masking the real reason the crash occurred).
>>
>> But for the majority of the case - the copy from/to would work much better
>> than what we have in mainstream today...
>>
>>
>> I would be interested in Paul and Russell - how have you solved this issue?
>> (Or do your kernel's never crash? :)
>
> Our kernel does crash, so we have to do this. I had submitted a patch a
> while ago that tweaks emit_log_char, but this was bounced, reasonably,
> because it would be interferring with the great majority of people who
> are not interested in capturing log output. The suggestion was made that
> we do this with a console driver. I've recently got a version working, as
> a module, but I've only started testing of the version integrated into the
> kernel tree.  Still, post early, post often, so I've appended a version
> of the patch here to see if this seems to be the right direction.

i guess there's two distinct crash types: early and late.  your
console driver seems like it should be able to handle the "late"
variety ok (i.e. the kernel is up and running and at some point, it
blows up).  but it doesnt address the early crashes which is what
we're currently looking for (i.e. kernel crash occurs before the
serial console is up and running).

i think extending this driver to include early printk support should
address that ?
-mike

Re: Kernel crashing and log buffers...

[David VomLehn]

On Thu, Jun 11, 2009 at 03:57:56PM -0700, Grant Erickson wrote:
> On 6/11/09 11:52 AM, Robin Getz wrote:
> > On Thu 11 Jun 2009 13:53, David VomLehn pondered:
> >> On Wed, Jun 10, 2009 at 09:26:40PM -0400, Robin Getz wrote:
> >>> I would be interested in Paul and Russell - how have you solved this
> >>> issue? (Or do your kernel's never crash? :)
> >> 
> >> Our kernel does crash, so we have to do this. I had submitted a patch a
> >> while ago that tweaks emit_log_char, but this was bounced, reasonably,
> >> because it would be interferring with the great majority of people who
> >> are not interested in capturing log output. The suggestion was made that
> >> we do this with a console driver. I've recently got a version working,
> >> as a module, but I've only started testing of the version integrated
> >> into the kernel tree.  Still, post early, post often, so I've appended
> >> a version of the patch here to see if this seems to be the right
> >> direction. 
> > 
> > Yeah - it looks like n people have done this n ways.
> > 
> > I think even Andrew M had mentioned that he had done something in a past
> > embedded life.
> > 
> > I think what you have would do what I need it to do - I'm not so sure about
> > what Wolfgang/Grant had in mind (- since they wanted also to dump buffers
> > from the bootloader into the console for syslog processing)...
> 
> Regrettably, I have not had the time to follow-up on this since my last
> commentary in March:
> 
>     http://lkml.org/lkml/2009/3/31/293
> 
> However, from what I can glean of the discussion, you are correct that this
> does not precisely cover what Wolfgang (originally) or, later, I was working
> towards in that neither boot loader logs nor round-trip log buffering
> (bootloader to kernel and back again) are covered.
> 
> However, the value and what's being proposed here and what I and others
> proposed--at least in the embedded kingdom--seems high and the features and
> implementation needn't be mutually exclusive.

I agree. In this case, we have three different approaches to three different
problems (bootloader buffers, continuous state logging, and in-kernel capture
of logging). All are problems that need to be solved. Personally, I'd love
to have the continuous state logging, though I don't need the bootloader
buffer part.

> Grant

Re: Kernel crashing and log buffers...

[David VomLehn]

On Thu, Jun 11, 2009 at 07:27:16PM -0400, Mike Frysinger wrote:
> On Thu, Jun 11, 2009 at 13:53, David VomLehn wrote:
> > On Wed, Jun 10, 2009 at 09:26:40PM -0400, Robin Getz wrote:
...
> > Our kernel does crash, so we have to do this. I had submitted a patch a
> > while ago that tweaks emit_log_char, but this was bounced, reasonably,
> > because it would be interferring with the great majority of people who
> > are not interested in capturing log output. The suggestion was made that
> > we do this with a console driver. I've recently got a version working, as
> > a module, but I've only started testing of the version integrated into the
> > kernel tree.  Still, post early, post often, so I've appended a version
> > of the patch here to see if this seems to be the right direction.
> 
> i guess there's two distinct crash types: early and late.  your
> console driver seems like it should be able to handle the "late"
> variety ok (i.e. the kernel is up and running and at some point, it
> blows up).  but it doesnt address the early crashes which is what
> we're currently looking for (i.e. kernel crash occurs before the
> serial console is up and running).
> 
> i think extending this driver to include early printk support should
> address that ?

If you really want to see really early output, I don't think this will give
you more benefit than simply looking at log_buf through your favorite kernel
debugging tool. Which is something I do with some regularity, so I know that
it can be a bit of a pain. It may be helpful if you can say a bit more about
the problem you are trying to solve.

> -mike

David VomLehn

Re: Kernel crashing and log buffers...

[Robin Getz]

On Thu 11 Jun 2009 21:07, David VomLehn pondered:
> It may be helpful if you can say a bit more about
> the problem you are trying to solve.

Yeah, I guess defining the goals is a good place to start (and 
the hardest to come to agreement on).

 - early crash analysis - before the kernel has set up the console 
   or console drivers, (which happens after interrupts, caches, & 
   memory, are set up) crashes are pretty difficult to figure out. 
   Typically - the kernel crashes or hangs without any output on 
   the console.
   What the end users experiences is typically...
   "Starting Kernel at = xxxx"
   and nothing else. The valuable info (if any) is stuck in the 
   kernel's log buffer.

   earlyprintk helps with this, however - this assumes that the 
   kernel has set up the early_printk console properly. If there
   is a problem doing that (wrong driver compiled in, wrong pin 
   muxing, etc), then the info is still stuck (unseen) in the
   kernel's log buffer.

   - but is only to be supported on:
   Documentation/kernel-parameters.txt
       earlyprintk=    [X86-32,X86-64,SH,BLACKFIN]
                        earlyprintk=vga
                        earlyprintk=serial[,ttySn[,baudrate]]
                        earlyprintk=dbgp

   see:
   arch/x86/kernel/early_printk.c
   arch/sh/kernel/early_printk.c
   arch/blackfin/kernel/early_printk.c

   For giggles - put a panic() in your arch/*/kernel/setup.c:setup_arch()
   With earlyprintk - you still get to see what happened (if the 
   earlyprintk console is working OK).

   Problem:
    Even with earlyprintk - there are times when things not getting
    out of the system, and are stuck in the printk __log_buf.

   David's patch might help with this - as architectures could register, 
   and write all the prink data into a buffer in memory - where it could
   be read out by the bootloader, or future kernels - but it would need
   some tweaks to support early functionality (but I don't think much).

 - late crash analysis - Sometimes - even after the kernel is up
   and running properly - a device driver does a bad thing. 
   Normally - complete kernel dumps can come out a serial
   console, and you can you your favourite serial application
   to scroll back and determine what is going on.

   In many embedded devices - this is not a possibility, since
   all the serial ports are in use (Irda, Bluetooth, GPS, etc).

   In devices that are deployed in the field - this is not a
   possibility, since nothing is connected to the serial 
   console (if it exists) for logging.

   Even when there is no console device, or anything attached to the
   console device, everything is in the __log_buf.

   Problem:
   There doesn't appear (to me) to be anything in mainline to help 
   with this.

   David's patch might help with this - as architectures could register, 
   and write all the prink data into a buffer in memory - where it could
   be read out by the bootloader, or future kernels.

 - allowing board startup information (POST) created/output from
   the boot loader to be transfered to the standard/normal kernel
   log processing features (syslog/dmesg/etc).

   There doesn't appear (to me) to be anything in mainline to help 
   with this.
 
   David's patch could help with this - as architectures could register, 
   and if a magic number was in the start of the buffer, it could just
   strnlen the buffer, and then do a printk on it. The downside is it would
   not be at the start of the buffer - as Grant's and Wolfgang's suggestion.
   But I'm not sure this is a requirement???


Is there anything I'm missing?

Re: Kernel crashing and log buffers...

[Wolfgang Denk]

Dear David VomLehn,

In message <20090612003302.GA23715@cuplxvomd02.corp.sa.net> you wrote:
>
> > However, the value and what's being proposed here and what I and others
> > proposed--at least in the embedded kingdom--seems high and the features and
> > implementation needn't be mutually exclusive.
> 
> I agree. In this case, we have three different approaches to three different
> problems (bootloader buffers, continuous state logging, and in-kernel capture
> of logging). All are problems that need to be solved. Personally, I'd love
> to have the continuous state logging, though I don't need the bootloader
> buffer part.

Question is: do we need three different solutions, or can we come up
with one solution that covers all?

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
"If it ain't broke, don't fix it."                       - Bert Lantz

Re: Kernel crashing and log buffers...

[Russell King]

On Wed, Jun 10, 2009 at 09:26:40PM -0400, Robin Getz wrote:
> I would be interested in Paul and Russell - how have you solved this issue? 
> (Or do your kernel's never crash? :)

We haven't any dedicated solution to reading out the crash messages.

What we do have is a very basic platform specific serial IO implementation
which we call "low level debug" - it just provides a number of functions
(eg, printascii).  These functions are callable from assembly or C, and
can work whether the MMU is enabled or not.

People can enable the low level debug build option and patch into
kernel/printk.c to hook it up (or place printascii etc where-ever they
choose.)  It has the advantage that it is entirely synchronous from
every context.

Yes, it requires a little bit of effort on the part of the developer to
hook it up - so far no one has complained about the process of hooking
this facility up in their kernel.

You can use this to debug SMP bringup, but you risk having each CPU
stomping on each other's messages - so it's in no way perfect.

The other way I've seen people read out crash messages is using a
debugger to dump the kernel's log buffer directly.  That seems to work
as well as any other method, and has the advantage that it doesn't
require any kernel modifications.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

Re: Kernel crashing and log buffers...

[Jamie Lokier]

Robin Getz wrote:
>  - late crash analysis - Sometimes - even after the kernel is up
>    and running properly - a device driver does a bad thing. 
>    Normally - complete kernel dumps can come out a serial
>    console, and you can you your favourite serial application
>    to scroll back and determine what is going on.
> 
>    In many embedded devices - this is not a possibility, since
>    all the serial ports are in use (Irda, Bluetooth, GPS, etc).

There's another thing with serial ports:

In embedded devices, I've seen serial drivers hooked up to the console
output either synchronously or asynchronously.

Synchronously, every printk() waits to be sent to the serial port, and
this slows the kernel down in normal operation, and especially boot
time.

Asynchronously, every printk() does into the log buffer without delay,
and is sent over the serial port as fast as that can.  It doesn't slow
the kernel down much.

When it's done asynchronously, if the kernel crashes you don't always
get the last output prior to the crash, even with a serial terminal
listening in your lab.  But if you do it synchronously, it slows
things down.

-- Jamie

Re: Kernel crashing and log buffers...

[Wolfgang Denk]

Dear Russell King,

In message <20090613102642.GB7976@flint.arm.linux.org.uk> you wrote:
>
> The other way I've seen people read out crash messages is using a
> debugger to dump the kernel's log buffer directly.  That seems to work
> as well as any other method, and has the advantage that it doesn't
> require any kernel modifications.

This works well in the lab during hardware bringup or BSP development.

But we are also interested in a solution that allows to get  more  or
less  automatic access to the log buffer content after a crash - when
you have several ten thousand systems in the field,  such  a  feature
can save you a lot of money.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de

Re: Kernel crashing and log buffers...

[Robin Getz]

On Sat 13 Jun 2009 14:59, Wolfgang Denk pondered:
> Dear Russell King,
> 
> In message <20090613102642.GB7976@flint.arm.linux.org.uk> you wrote:
> >
> > The other way I've seen people read out crash messages is using a
> > debugger to dump the kernel's log buffer directly.  That seems to work
> > as well as any other method, and has the advantage that it doesn't
> > require any kernel modifications.
> 
> This works well in the lab during hardware bringup or BSP development.
> 
> But we are also interested in a solution that allows to get  more  or
> less  automatic access to the log buffer content after a crash - when
> you have several ten thousand systems in the field,  such  a  feature
> can save you a lot of money.

in this case - is it true that providing the kernel crash log buffer to the 
next running kernel - is actually more important than just giving it to the 
bootloader? (Since the bootloader may not have the facilities to 
email/post/netcat it to a developer - like the next running kernel might?)

In this case - the Bootloader being able to read the buffer, understand that 
there was a crash, and be able to pass the buffer (with the crash message) 
into the next running kernel - where that kernel can package it up and do 
anything it wants to it - would be a requirement?

-Robin

Re: Kernel crashing and log buffers...

[Robin Getz]

OK - so after a bit more digging into things (and a poke by Mike) - I think 
most of the stuff already exists.


Normal run time issues:
--------------------------
 - MTD_OOPS - on 2007-06-28 this was added to Linus's tree:
    http://lkml.org/lkml/2007/6/18/234
    CONFIG_MTD_OOPS
      tristate "Log panic/oops to an MTD buffer"
        depends on MTD
        help
          This enables panic and oops messages to be logged to a circular
          buffer in a flash partition where it can be read back at some
          later point.

          To use, add console=ttyMTDx to the kernel command line,
          where x is the MTD device number to use.

Note - this currently _only_ does things when oops_in_progress is set, but 
should be easy to extend to all - if that what people want.

 - mtd devices can be backed by RAM.
    CONFIG_MTD_RAM

 - mtd devices can be backed by flash (if you want things to be persistant
     across power cycles)...



early boot issues:
----------------------
 - CONFIG_EARLY_PRINTK
   - early printk just defined a console - and is supported by:
      alpha, blackfin, microblaze, mips, powerpc, sh, x86
   - it's pretty trivial to support a memory based buffer - some
     archs already support it.

I think this only leaves Wolfgang's desire for memory buffers from the 
bootloader to get (somehow) into the kernel's log buffer for syslog 
processing...

Anyone else agree?

Re: Kernel crashing and log buffers...

[David VomLehn]

On Fri, Jun 26, 2009 at 10:39:50AM -0400, Robin Getz wrote:
> OK - so after a bit more digging into things (and a poke by Mike) - I think 
> most of the stuff already exists.
> 
> Normal run time issues:
> --------------------------
>  - MTD_OOPS - on 2007-06-28 this was added to Linus's tree:
>     http://lkml.org/lkml/2007/6/18/234
>     CONFIG_MTD_OOPS
>       tristate "Log panic/oops to an MTD buffer"
...
> early boot issues:
> ----------------------
>  - CONFIG_EARLY_PRINTK
>    - early printk just defined a console - and is supported by:
>       alpha, blackfin, microblaze, mips, powerpc, sh, x86
>    - it's pretty trivial to support a memory based buffer - some
>      archs already support it.
> 
> I think this only leaves Wolfgang's desire for memory buffers from the 
> bootloader to get (somehow) into the kernel's log buffer for syslog 
> processing...
> 
> Anyone else agree?

Almost. A couple of us also want memory for "flight data record" FDR data
for doing continuous logging. This would, ideally, be either uncached or cached
in such a way that data is guaranteed to be written to memory in the event of
a watchdog timer-induced system reset.

David VL

Re: Kernel crashing and log buffers...

[Mike Frysinger]

On Fri, Jun 26, 2009 at 13:42, David VomLehn wrote:
> On Fri, Jun 26, 2009 at 10:39:50AM -0400, Robin Getz wrote:
>> OK - so after a bit more digging into things (and a poke by Mike) - I think
>> most of the stuff already exists.
>>
>> Normal run time issues:
>> --------------------------
>>  - MTD_OOPS - on 2007-06-28 this was added to Linus's tree:
>>     http://lkml.org/lkml/2007/6/18/234
>>     CONFIG_MTD_OOPS
>>       tristate "Log panic/oops to an MTD buffer"
> ...
>> early boot issues:
>> ----------------------
>>  - CONFIG_EARLY_PRINTK
>>    - early printk just defined a console - and is supported by:
>>       alpha, blackfin, microblaze, mips, powerpc, sh, x86
>>    - it's pretty trivial to support a memory based buffer - some
>>      archs already support it.
>>
>> I think this only leaves Wolfgang's desire for memory buffers from the
>> bootloader to get (somehow) into the kernel's log buffer for syslog
>> processing...
>>
>> Anyone else agree?
>
> Almost. A couple of us also want memory for "flight data record" FDR data
> for doing continuous logging. This would, ideally, be either uncached or cached
> in such a way that data is guaranteed to be written to memory in the event of
> a watchdog timer-induced system reset.

can none of the existing mtd devices provide uncached ram ?
otherwise, create a mtd map that simply calls data flush after every
write.  then the existing mtd stack can provide what you want.
-mike

Re: Kernel crashing and log buffers...

[Robin Getz]

On Fri 26 Jun 2009 13:42, David VomLehn pondered:
> On Fri, Jun 26, 2009 at 10:39:50AM -0400, Robin Getz wrote:
> > OK - so after a bit more digging into things (and a poke by Mike) - I
> think 
> > most of the stuff already exists.
> > 
> > Normal run time issues:
> > --------------------------
> >  - MTD_OOPS - on 2007-06-28 this was added to Linus's tree:
> >     http://lkml.org/lkml/2007/6/18/234
> >     CONFIG_MTD_OOPS
> >       tristate "Log panic/oops to an MTD buffer"
> ...
> > early boot issues:
> > ----------------------
> >  - CONFIG_EARLY_PRINTK
> >    - early printk just defined a console - and is supported by:
> >       alpha, blackfin, microblaze, mips, powerpc, sh, x86
> >    - it's pretty trivial to support a memory based buffer - some
> >      archs already support it.
> > 
> > I think this only leaves Wolfgang's desire for memory buffers from the
> 
> > bootloader to get (somehow) into the kernel's log buffer for syslog 
> > processing...
> > 
> > Anyone else agree?
> 
> Almost. A couple of us also want memory for "flight data record" FDR
> data for doing continuous logging. This would, ideally, be either uncached
> or cached in such a way that data is guaranteed to be written to memory in
> the event of a watchdog timer-induced system reset.

Many of the early_printk implementations accept a "keep" option (which does 
not remove the console/buffer - and keeps it around forever) - I'm adding 
that to the Blackfin implementation as we speak - but it is already there on 
sh.

I expect that we could add something to the MTD_OOPS, as to allways write as 
well (console=ttyMTDx,log_level) or something like that if the backing needed 
to be flash.

Would that satisfy things?

-Robin