[PATCH v2 0/2] Add capability to dump fdt blob for arm64 platforms

[PATCH v2 0/2] Add capability to dump fdt blob for arm64 platforms

[Bhupesh Sharma]

Changes since v1:
----------------
 - v1 can be viewed here: http://lists.infradead.org/pipermail/kexec/2018-April/020407.html
 - No functional changes: Just added a cover letter to explain the
   background better and also capture some details on where I found this
   patchset handy. Also added some dtb dumps logs from 'kexec -p -d' for
   reference (with this patchset applied) for clarity.

While working on a couple of issues related to primary kernel crash on freescale
and huawei arm64 boards, I noticed that the primary kernel crashed before it could reach
the command prompt but was able to launch some early initramfs scriptware. 

In the initial initramfs scriptware crashkernel loading was automated along
with auto load of other userspace applications (for e.g. on the freescale board
there are networking applications like ODP/DPDK which can be launched automatically via
scriptware).

I was hoping that the crashkernel would be able to load when the primary kernel crashes,
and using the crash core dump thus obtained, I would be able to debug the problem which
caused the primary kernel to crash late in the boot flow (before reaching the boot prompt).

Unfortunately currently we can experience an early crash in crashkernel itself
(on such example is the 'acpi table access' issue in the arm64 crashkernel
which we discussed some time back upstream
<https://www.spinics.net/lists/arm-kernel/msg616632.html>):

In such cases, we have no opportunity to obtain the crash core dump which can be
used to debug the primary kernel crash.

Now, looking at just the panic messages from the crashkernel in such cases is sometimes
not very useful in debugging what might have caused it to crash when the primary kernel
is able to atleast boot past that point on the same hardware platform.

Debugging the issue closer (especially on the request for help on the freescale board), I
realized that the crashkernel crash may be caused by improper/buggy fixing of 'dtb'
being passed to the crashkernel - especially the 'linux,usable-memory-range' property.

For such cases, I found that dumping the dtb blob entries from kexec-tools is
a useful debugging tip as I could identify the 'linux,usable-memory-range'
property did not contain ACPI RECLAIM region entries.

Please note that since the primary kernel crashes before the command prompt
can be reached, it is not possible to run a dtc interpreter there (and it
also adds the requirement for an additional 'dtc' tool to be present in the initramfs).

Also, it might not be possible to always correctly time the 'dtc' interpreter loading
via the initramfs scriptware and store the binary/hex output to a storage device
just after the crashkernel is loaded via 'kexec -p' as the storage driver itself
might have panick'ed during the meanwhile.

In view of the above, it would be useful to dump the fdt blob being passed to the second
(kexec/kdump) kernel when '-d' flag is specified while invoking kexec/kdump. This allows
 one to look at the device-tree fields that is being passed to the secondary
kernel and accordingly debug issues.

This can be specially useful for the arm64 case, where we are still fixing up some issues
with the upstream kexec-tools/arm64 kernel.

I loathe to keep this patch locally and apply it locally on top of the upstream 'kexec-tools'
patches when debugging such issues, so it would be probably good to have this feature
available in upstream itself.

Here is an example output of the dtb dump(on an arm64 board), on serial console with
the patchset applied and 'kexec -p' launched used with a '-d' flag using initramfs scriptware:

<..snip..>

setup_2nd_dtb: found /sys/firmware/fdt
 / {
    #size-cells = <0x00000002>;
    #address-cells = <0x00000002>;
    chosen {
        linux,usable-memory-range = <0x00000000 0xdfe00000 0x00000000 0x20000000>;
        linux,elfcorehdr = <0x00000000 0xffdf0000 0x00000000 0x00001400>;
        kaslr-seed = <0x00000000 0x00000000>;
        linux,uefi-mmap-desc-ver = <0x00000001>;
        linux,uefi-mmap-desc-size = <0x00000030>;
        linux,uefi-mmap-size = <0x000020a0>;
        linux,uefi-mmap-start = <0x00000000 0x07a81018>;
        linux,uefi-system-table = <0x00000000 0x17fc0018>;
        bootargs = <0x726f6f74 0x3d2f6465 0x762f6d61 0x70706572 0x2f726865 0x6c5f7175 0x616c636f 0x6d6d2d2d 0x616d6265 0x7277696e 0x672d2d72 0x65702d2d 0x31352d72 0x6f6f7420 0x726f2072 0x642e6c76 0x6d2e6c76 0x3d726865 0x6c5f7175 0x616c636f 0x6d6d2d61 0x6d626572 0x77696e67 0x2d726570 0x2d31352f 0x726f6f74 0x2072642e 0x6c766d2e 0x6c763d72 0x68656c5f 0x7175616c 0x636f6d6d 0x2d616d62 0x65727769 0x6e672d72 0x65702d31 0x352f7377 0x61700000>;
        linux,initrd-end = <0x00000000 0x05e8a7a1>;
        linux,initrd-start = <0x00000000 0x04b49000>;
    };
 };

<..snip..>

Bhupesh Sharma (2):
  dt-ops: Add helper API to dump fdt blob
  kexec-arm64: Add functionality to dump 2nd dtb

 kexec/arch/arm64/kexec-arm64.c |   3 +-
 kexec/dt-ops.c                 | 133 +++++++++++++++++++++++++++++++++++++++++
 kexec/dt-ops.h                 |   1 +
 3 files changed, 136 insertions(+), 1 deletion(-)

-- 
2.7.4


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH v2 1/2] dt-ops: Add helper API to dump fdt blob

[Bhupesh Sharma]

At several occasions it would be useful to dump the fdt
blob being passed to the second (kexec/kdump) kernel
when '-d' flag is specified while invoking kexec/kdump.

This allows one to look at the device-tree fields that
is being passed to the secondary kernel and accordingly
debug issues.

This can be specially useful for the arm64 case, where
kexec_load() or kdump passes important information like
'linux,usable-memory' ranges to the second kernel, and
the correctness of the ranges can be verified by
looking at the device-tree dump with '-d' flag specified.

Signed-off-by: Bhupesh Sharma <bhsharma@redhat.com>
---
 kexec/dt-ops.c | 133 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 kexec/dt-ops.h |   1 +
 2 files changed, 134 insertions(+)

diff --git a/kexec/dt-ops.c b/kexec/dt-ops.c
index 915dbf55afd2..caf6f8df6e6b 100644
--- a/kexec/dt-ops.c
+++ b/kexec/dt-ops.c
@@ -8,6 +8,10 @@
 #include "kexec.h"
 #include "dt-ops.h"
 
+#define ALIGN(x, a)	(((x) + ((a) - 1)) & ~((a) - 1))
+#define PALIGN(p, a)	((void *)(ALIGN((unsigned long)(p), (a))))
+#define GET_CELL(p)	(p += 4, *((const uint32_t *)(p-4)))
+
 static const char n_chosen[] = "/chosen";
 
 static const char p_bootargs[] = "bootargs";
@@ -143,3 +147,132 @@ int dtb_delete_property(char *dtb, const char *node, const char *prop)
 
 	return result;
 }
+
+static uint64_t is_printable_string(const void* data, uint64_t len)
+{
+	const char *s = data;
+	const char *ss;
+
+	/* Check for zero length strings */
+	if (len == 0) {
+		return 0;
+	}
+
+	/* String must be terminated with a '\0' */
+	if (s[len - 1] != '\0') {
+		return 0;
+	}
+
+	ss = s;
+	while (*s) {
+		s++;
+	}
+
+	/* Traverse till we hit a '\0' or reach 'len' */
+	if (*s != '\0' || (s + 1 - ss) < len) {
+		return 0;
+	}
+
+	return 1;
+}
+
+static void print_data(const char* data, uint64_t len)
+{
+	uint64_t i;
+	const char *p = data;
+
+	/* Check for non-zero length */
+	if (len == 0)
+		return;
+
+	if (is_printable_string(data, len)) {
+		dbgprintf(" = \"%s\"", (const char *)data);
+	} else if ((len % 4) == 0) {
+		dbgprintf(" = <");
+		for (i = 0; i < len; i += 4) {
+			dbgprintf("0x%08x%s",
+					fdt32_to_cpu(GET_CELL(p)),
+					i < (len - 4) ? " " : "");
+		}
+		dbgprintf(">");
+	} else {
+		dbgprintf(" = [");
+		for (i = 0; i < len; i++)
+			dbgprintf("%02x%s", *p++,
+					i < len - 1 ? " " : "");
+		dbgprintf("]");
+	}
+}
+
+void dump_fdt(void* fdt)
+{
+	struct fdt_header *bph;
+	const char* p_struct;
+	const char* p_strings;
+	const char* p;
+	const char* s;
+	const char* t;
+	uint32_t off_dt;
+	uint32_t off_str;
+	uint32_t tag;
+	uint64_t sz;
+	uint64_t depth;
+	uint64_t shift;
+	uint32_t version;
+
+	depth = 0;
+	shift = 4;
+
+	bph = fdt;
+	off_dt = fdt32_to_cpu(bph->off_dt_struct);
+	off_str = fdt32_to_cpu(bph->off_dt_strings);
+	p_struct = (const char*)fdt + off_dt;
+	p_strings = (const char*)fdt + off_str;
+	version = fdt32_to_cpu(bph->version);
+
+	p = p_struct;
+	while ((tag = fdt32_to_cpu(GET_CELL(p))) != FDT_END) {
+
+		if (tag == FDT_BEGIN_NODE) {
+			s = p;
+			p = PALIGN(p + strlen(s) + 1, 4);
+
+			if (*s == '\0')
+				s = "/";
+
+			dbgprintf("%*s%s {\n", (int)(depth * shift), " ", s);
+
+			depth++;
+			continue;
+		}
+
+		if (tag == FDT_END_NODE) {
+			depth--;
+
+			dbgprintf("%*s};\n", (int)(depth * shift), " ");
+			continue;
+		}
+
+		if (tag == FDT_NOP) {
+			dbgprintf("%*s// [NOP]\n", (int)(depth * shift), " ");
+			continue;
+		}
+
+		if (tag != FDT_PROP) {
+			dbgprintf("%*s ** Unknown tag 0x%08x\n",
+					(int)(depth * shift), " ", tag);
+			break;
+		}
+		sz = fdt32_to_cpu(GET_CELL(p));
+		s = p_strings + fdt32_to_cpu(GET_CELL(p));
+		if (version < 16 && sz >= 8)
+			p = PALIGN(p, 8);
+		t = p;
+
+		p = PALIGN(p + sz, 4);
+
+		dbgprintf("%*s%s", (int)(depth * shift), " ", s);
+		print_data(t, sz);
+		dbgprintf(";\n");
+	}
+}
diff --git a/kexec/dt-ops.h b/kexec/dt-ops.h
index e70d15d8ffbf..25b9b569f2b7 100644
--- a/kexec/dt-ops.h
+++ b/kexec/dt-ops.h
@@ -9,5 +9,6 @@ int dtb_set_property(char **dtb, off_t *dtb_size, const char *node,
 	const char *prop, const void *value, int value_len);
 
 int dtb_delete_property(char *dtb, const char *node, const char *prop);
+void dump_fdt(void *fdt) ;
 
 #endif
-- 
2.7.4


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH v2 2/2] kexec-arm64: Add functionality to dump 2nd dtb

[Bhupesh Sharma]

Since during the arm64 kexec_load()/kdump invocation,
the dtb is passed to the second kernel, it is sometimes useful
to dump the dtb contents (to verify the correctness
of the same).

This patch adds this feature which is enabled when '-d' flag is
used with kexec command line invocation.

Signed-off-by: Bhupesh Sharma <bhsharma@redhat.com>
---
 kexec/arch/arm64/kexec-arm64.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/kexec/arch/arm64/kexec-arm64.c b/kexec/arch/arm64/kexec-arm64.c
index 62f37585b788..1b54718465b9 100644
--- a/kexec/arch/arm64/kexec-arm64.c
+++ b/kexec/arch/arm64/kexec-arm64.c
@@ -477,9 +477,10 @@ static int setup_2nd_dtb(struct dtb *dtb, char *command_line, int on_crash)
 		dtb->size = fdt_totalsize(new_buf);
 	}
 
+	dbgprintf("%s: found %s\n", __func__, dtb->path);
+	dump_fdt(dtb->buf);
 	dump_reservemap(dtb);
 
-
 	return result;
 
 on_error:
-- 
2.7.4


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH v2 0/2] Add capability to dump fdt blob for arm64 platforms

[AKASHI Takahiro]

Bhupesh,

On Mon, Apr 23, 2018 at 03:56:10PM +0530, Bhupesh Sharma wrote:
> Changes since v1:
> ----------------
>  - v1 can be viewed here: http://lists.infradead.org/pipermail/kexec/2018-April/020407.html
>  - No functional changes: Just added a cover letter to explain the
>    background better and also capture some details on where I found this
>    patchset handy. Also added some dtb dumps logs from 'kexec -p -d' for
>    reference (with this patchset applied) for clarity.
> 
> While working on a couple of issues related to primary kernel crash on freescale
> and huawei arm64 boards, I noticed that the primary kernel crashed before it could reach
> the command prompt but was able to launch some early initramfs scriptware. 
> 
> In the initial initramfs scriptware crashkernel loading was automated along
> with auto load of other userspace applications (for e.g. on the freescale board
> there are networking applications like ODP/DPDK which can be launched automatically via
> scriptware).
> 
> I was hoping that the crashkernel would be able to load when the primary kernel crashes,
> and using the crash core dump thus obtained, I would be able to debug the problem which
> caused the primary kernel to crash late in the boot flow (before reaching the boot prompt).
> 
> Unfortunately currently we can experience an early crash in crashkernel itself
> (on such example is the 'acpi table access' issue in the arm64 crashkernel
> which we discussed some time back upstream
> <https://www.spinics.net/lists/arm-kernel/msg616632.html>):
> 
> In such cases, we have no opportunity to obtain the crash core dump which can be
> used to debug the primary kernel crash.
> 
> Now, looking at just the panic messages from the crashkernel in such cases is sometimes
> not very useful in debugging what might have caused it to crash when the primary kernel
> is able to atleast boot past that point on the same hardware platform.
> 
> Debugging the issue closer (especially on the request for help on the freescale board), I
> realized that the crashkernel crash may be caused by improper/buggy fixing of 'dtb'
> being passed to the crashkernel - especially the 'linux,usable-memory-range' property.
> 
> For such cases, I found that dumping the dtb blob entries from kexec-tools is
> a useful debugging tip as I could identify the 'linux,usable-memory-range'
> property did not contain ACPI RECLAIM region entries.
> 
> Please note that since the primary kernel crashes before the command prompt
> can be reached, it is not possible to run a dtc interpreter there (and it
> also adds the requirement for an additional 'dtc' tool to be present in the initramfs).
> 
> Also, it might not be possible to always correctly time the 'dtc' interpreter loading
> via the initramfs scriptware and store the binary/hex output to a storage device
> just after the crashkernel is loaded via 'kexec -p' as the storage driver itself
> might have panick'ed during the meanwhile.
> 
> In view of the above, it would be useful to dump the fdt blob being passed to the second
> (kexec/kdump) kernel when '-d' flag is specified while invoking kexec/kdump. This allows
>  one to look at the device-tree fields that is being passed to the secondary
> kernel and accordingly debug issues.
> 
> This can be specially useful for the arm64 case, where we are still fixing up some issues
> with the upstream kexec-tools/arm64 kernel.
> 
> I loathe to keep this patch locally and apply it locally on top of the upstream 'kexec-tools'
> patches when debugging such issues, so it would be probably good to have this feature
> available in upstream itself.
> 
> Here is an example output of the dtb dump(on an arm64 board), on serial console with
> the patchset applied and 'kexec -p' launched used with a '-d' flag using initramfs scriptware:
> 
> <..snip..>
> 
> setup_2nd_dtb: found /sys/firmware/fdt
>  / {
>     #size-cells = <0x00000002>;
>     #address-cells = <0x00000002>;
>     chosen {
>         linux,usable-memory-range = <0x00000000 0xdfe00000 0x00000000 0x20000000>;
>         linux,elfcorehdr = <0x00000000 0xffdf0000 0x00000000 0x00001400>;
>         kaslr-seed = <0x00000000 0x00000000>;
>         linux,uefi-mmap-desc-ver = <0x00000001>;
>         linux,uefi-mmap-desc-size = <0x00000030>;
>         linux,uefi-mmap-size = <0x000020a0>;
>         linux,uefi-mmap-start = <0x00000000 0x07a81018>;
>         linux,uefi-system-table = <0x00000000 0x17fc0018>;
>         bootargs = <0x726f6f74 0x3d2f6465 0x762f6d61 0x70706572 0x2f726865 0x6c5f7175 0x616c636f 0x6d6d2d2d 0x616d6265 0x7277696e 0x672d2d72 0x65702d2d 0x31352d72 0x6f6f7420 0x726f2072 0x642e6c76 0x6d2e6c76 0x3d726865 0x6c5f7175 0x616c636f 0x6d6d2d61 0x6d626572 0x77696e67 0x2d726570 0x2d31352f 0x726f6f74 0x2072642e 0x6c766d2e 0x6c763d72 0x68656c5f 0x7175616c 0x636f6d6d 0x2d616d62 0x65727769 0x6e672d72 0x65702d31 0x352f7377 0x61700000>;

Why not print "ascii" string here?

-Takahiro AKASHI

>         linux,initrd-end = <0x00000000 0x05e8a7a1>;
>         linux,initrd-start = <0x00000000 0x04b49000>;
>     };
>  };
> 
> <..snip..>
> 
> Bhupesh Sharma (2):
>   dt-ops: Add helper API to dump fdt blob
>   kexec-arm64: Add functionality to dump 2nd dtb
> 
>  kexec/arch/arm64/kexec-arm64.c |   3 +-
>  kexec/dt-ops.c                 | 133 +++++++++++++++++++++++++++++++++++++++++
>  kexec/dt-ops.h                 |   1 +
>  3 files changed, 136 insertions(+), 1 deletion(-)
> 
> -- 
> 2.7.4
> 

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH v2 0/2] Add capability to dump fdt blob for arm64 platforms

[Bhupesh Sharma]

Hi Akashi,

Thanks for the review.

On Thu, Apr 26, 2018 at 1:26 PM, AKASHI Takahiro
<takahiro.akashi@linaro.org> wrote:
> Bhupesh,
>
> On Mon, Apr 23, 2018 at 03:56:10PM +0530, Bhupesh Sharma wrote:
>> Changes since v1:
>> ----------------
>>  - v1 can be viewed here: http://lists.infradead.org/pipermail/kexec/2018-April/020407.html
>>  - No functional changes: Just added a cover letter to explain the
>>    background better and also capture some details on where I found this
>>    patchset handy. Also added some dtb dumps logs from 'kexec -p -d' for
>>    reference (with this patchset applied) for clarity.
>>
>> While working on a couple of issues related to primary kernel crash on freescale
>> and huawei arm64 boards, I noticed that the primary kernel crashed before it could reach
>> the command prompt but was able to launch some early initramfs scriptware.
>>
>> In the initial initramfs scriptware crashkernel loading was automated along
>> with auto load of other userspace applications (for e.g. on the freescale board
>> there are networking applications like ODP/DPDK which can be launched automatically via
>> scriptware).
>>
>> I was hoping that the crashkernel would be able to load when the primary kernel crashes,
>> and using the crash core dump thus obtained, I would be able to debug the problem which
>> caused the primary kernel to crash late in the boot flow (before reaching the boot prompt).
>>
>> Unfortunately currently we can experience an early crash in crashkernel itself
>> (on such example is the 'acpi table access' issue in the arm64 crashkernel
>> which we discussed some time back upstream
>> <https://www.spinics.net/lists/arm-kernel/msg616632.html>):
>>
>> In such cases, we have no opportunity to obtain the crash core dump which can be
>> used to debug the primary kernel crash.
>>
>> Now, looking at just the panic messages from the crashkernel in such cases is sometimes
>> not very useful in debugging what might have caused it to crash when the primary kernel
>> is able to atleast boot past that point on the same hardware platform.
>>
>> Debugging the issue closer (especially on the request for help on the freescale board), I
>> realized that the crashkernel crash may be caused by improper/buggy fixing of 'dtb'
>> being passed to the crashkernel - especially the 'linux,usable-memory-range' property.
>>
>> For such cases, I found that dumping the dtb blob entries from kexec-tools is
>> a useful debugging tip as I could identify the 'linux,usable-memory-range'
>> property did not contain ACPI RECLAIM region entries.
>>
>> Please note that since the primary kernel crashes before the command prompt
>> can be reached, it is not possible to run a dtc interpreter there (and it
>> also adds the requirement for an additional 'dtc' tool to be present in the initramfs).
>>
>> Also, it might not be possible to always correctly time the 'dtc' interpreter loading
>> via the initramfs scriptware and store the binary/hex output to a storage device
>> just after the crashkernel is loaded via 'kexec -p' as the storage driver itself
>> might have panick'ed during the meanwhile.
>>
>> In view of the above, it would be useful to dump the fdt blob being passed to the second
>> (kexec/kdump) kernel when '-d' flag is specified while invoking kexec/kdump. This allows
>>  one to look at the device-tree fields that is being passed to the secondary
>> kernel and accordingly debug issues.
>>
>> This can be specially useful for the arm64 case, where we are still fixing up some issues
>> with the upstream kexec-tools/arm64 kernel.
>>
>> I loathe to keep this patch locally and apply it locally on top of the upstream 'kexec-tools'
>> patches when debugging such issues, so it would be probably good to have this feature
>> available in upstream itself.
>>
>> Here is an example output of the dtb dump(on an arm64 board), on serial console with
>> the patchset applied and 'kexec -p' launched used with a '-d' flag using initramfs scriptware:
>>
>> <..snip..>
>>
>> setup_2nd_dtb: found /sys/firmware/fdt
>>  / {
>>     #size-cells = <0x00000002>;
>>     #address-cells = <0x00000002>;
>>     chosen {
>>         linux,usable-memory-range = <0x00000000 0xdfe00000 0x00000000 0x20000000>;
>>         linux,elfcorehdr = <0x00000000 0xffdf0000 0x00000000 0x00001400>;
>>         kaslr-seed = <0x00000000 0x00000000>;
>>         linux,uefi-mmap-desc-ver = <0x00000001>;
>>         linux,uefi-mmap-desc-size = <0x00000030>;
>>         linux,uefi-mmap-size = <0x000020a0>;
>>         linux,uefi-mmap-start = <0x00000000 0x07a81018>;
>>         linux,uefi-system-table = <0x00000000 0x17fc0018>;
>>         bootargs = <0x726f6f74 0x3d2f6465 0x762f6d61 0x70706572 0x2f726865 0x6c5f7175 0x616c636f 0x6d6d2d2d 0x616d6265 0x7277696e 0x672d2d72 0x65702d2d 0x31352d72 0x6f6f7420 0x726f2072 0x642e6c76 0x6d2e6c76 0x3d726865 0x6c5f7175 0x616c636f 0x6d6d2d61 0x6d626572 0x77696e67 0x2d726570 0x2d31352f 0x726f6f74 0x2072642e 0x6c766d2e 0x6c763d72 0x68656c5f 0x7175616c 0x636f6d6d 0x2d616d62 0x65727769 0x6e672d72 0x65702d31 0x352f7377 0x61700000>;
>
> Why not print "ascii" string here?

Sure, I can fix that in the v3. I will wait for other comments on v2
(since on the v1 we had several queries regarding the use case
definition), for a day or two (so that they can be addressed in the
new version) and then will spin out a v3.

Thanks,
Bhupesh


>>         linux,initrd-end = <0x00000000 0x05e8a7a1>;
>>         linux,initrd-start = <0x00000000 0x04b49000>;
>>     };
>>  };
>>
>> <..snip..>
>>
>> Bhupesh Sharma (2):
>>   dt-ops: Add helper API to dump fdt blob
>>   kexec-arm64: Add functionality to dump 2nd dtb
>>
>>  kexec/arch/arm64/kexec-arm64.c |   3 +-
>>  kexec/dt-ops.c                 | 133 +++++++++++++++++++++++++++++++++++++++++
>>  kexec/dt-ops.h                 |   1 +
>>  3 files changed, 136 insertions(+), 1 deletion(-)
>>
>> --
>> 2.7.4
>>

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec