Handling spin table in kdump

Handling spin table in kdump

[Suzuki K. Poulose]

Hi

I came across the following issue while testing Kdump on an SMP 
board(Currituck) running a non-SMP kernel. Even though the kernel is UP,
the device-tree has the nodes for second CPU and the related details.


The kexec tool adds the spin table area as a reserved section in the 
device tree for the dump capture kernel. This value is read from the 
'cpu-release-addr'.

But now, if the spin table is not located within the 'Reserved region' 
for the crash kernel, the dump capture kernel would fail to boot, 
hitting a BUG in mm/bootmem.c as in [1].

This is because we try to reserve a region which is not available to the 
kernel.

So I am wondering how is this handled really on an SMP board (Fsl_bookE).

There are two possible solutions :
1) Do not reserve the regions for the spin-table, as we will use
only the crashing CPU in the second kernel(maxcpus=1).


2) Add the spin-table region to the available memory regions passed
to the kernel by kexec-tools.

I have tested (1) and it works fine for me. Yet to test (2).


Thoughts ?


Thanks
Suzuki



[1] Kernel Bug
----------------


Linux version 3.3.0-rc5 (root@suzukikp.in.ibm.com) (gcc version 4.3.4 
[gcc-4_3-branch revision 152973] (GCC) ) #12 Tue May 22 18:03:01 IST2
Found legacy serial port 0 for /plb/opb/serial@10000000
   mem=20010000000, taddr=20010000000, irq=0, clk=1851851, speed=115200
------------[ cut here ]------------
kernel BUG at mm/bootmem.c:351!
Vector: 700 (Program Check) at [c8a61e90]
     pc: c847f91c: mark_bootmem+0xa0/0x14c
     lr: c8472670: do_init_bootmem+0x1ac/0x218
     sp: c8a61f40
    msr: 21000
   current = 0xc8a4a500
     pid   = 0, comm = swapper
kernel BUG at mm/bootmem.c:351!
enter ? for help
[c8a61f70] c8472670 do_init_bootmem+0x1ac/0x218
[c8a61f90] c847025c setup_arch+0x1bc/0x234
[c8a61fb0] c846b62c start_kernel+0x98/0x358
[c8a61ff0] c80000b4 _start+0xb4/0xf8

Re: Handling spin table in kdump

[McClintock Matthew-B29882]

On Tue, May 22, 2012 at 7:42 AM, Suzuki K. Poulose <suzuki@in.ibm.com> wrot=
e:
> Hi
>
> I came across the following issue while testing Kdump on an SMP
> board(Currituck) running a non-SMP kernel. Even though the kernel is UP,
> the device-tree has the nodes for second CPU and the related details.
>
>
> The kexec tool adds the spin table area as a reserved section in the devi=
ce
> tree for the dump capture kernel. This value is read from the
> 'cpu-release-addr'.
>
> But now, if the spin table is not located within the 'Reserved region' fo=
r
> the crash kernel, the dump capture kernel would fail to boot, hitting a B=
UG
> in mm/bootmem.c as in [1].
>
> This is because we try to reserve a region which is not available to the
> kernel.
>
> So I am wondering how is this handled really on an SMP board (Fsl_bookE).
>
> There are two possible solutions :
> 1) Do not reserve the regions for the spin-table, as we will use
> only the crashing CPU in the second kernel(maxcpus=3D1).
>
>
> 2) Add the spin-table region to the available memory regions passed
> to the kernel by kexec-tools.
>
> I have tested (1) and it works fine for me. Yet to test (2).
>
>
> Thoughts ?

I would think option 1 is acceptable. The crash kernel will never
attempt to use the memory outside of the crash kernel region therefore
it does not need to be reserved.

However, I thought we were able to boot SMP into the crash kernel on
our parts... not sure how that effects things (although the maxcpus=3D1
here is a perfectly acceptable and safer thing to do anyways)

-M


>
>
> Thanks
> Suzuki
>
>
>
> [1] Kernel Bug
> ----------------
>
>
> Linux version 3.3.0-rc5 (root@suzukikp.in.ibm.com) (gcc version 4.3.4
> [gcc-4_3-branch revision 152973] (GCC) ) #12 Tue May 22 18:03:01 IST2
> Found legacy serial port 0 for /plb/opb/serial@10000000
> =A0mem=3D20010000000, taddr=3D20010000000, irq=3D0, clk=3D1851851, speed=
=3D115200
> ------------[ cut here ]------------
> kernel BUG at mm/bootmem.c:351!
> Vector: 700 (Program Check) at [c8a61e90]
> =A0 =A0pc: c847f91c: mark_bootmem+0xa0/0x14c
> =A0 =A0lr: c8472670: do_init_bootmem+0x1ac/0x218
> =A0 =A0sp: c8a61f40
> =A0 msr: 21000
> =A0current =3D 0xc8a4a500
> =A0 =A0pid =A0 =3D 0, comm =3D swapper
> kernel BUG at mm/bootmem.c:351!
> enter ? for help
> [c8a61f70] c8472670 do_init_bootmem+0x1ac/0x218
> [c8a61f90] c847025c setup_arch+0x1bc/0x234
> [c8a61fb0] c846b62c start_kernel+0x98/0x358
> [c8a61ff0] c80000b4 _start+0xb4/0xf8
>
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev=

[PATCH] [ppc] Do not reserve cpu spin-table for crash kernel

[Suzuki K. Poulose]

As of now, the kexec reserves the spin-table for all the CPUs
on an SMP machine. The spin-table is pointed to by the 
cpu-release-addr property in the device-tree. Reserving the
spin-table in the crash kernel will cause a BUG(), if the table
lies outside the memory reserved for the crashkernel.

Disable reserving the spin-table regions and use maxcpus=1 to 
use only the crashing CPU to boot the crash kernel.

Signed-off-by: Suzuki K. Poulose <suzuki@in.ibm.com>
---

 kexec/arch/ppc/crashdump-powerpc.c |   19 +++++++++++++------
 kexec/arch/ppc/fixup_dtb.c         |    4 ++++
 2 files changed, 17 insertions(+), 6 deletions(-)

diff --git a/kexec/arch/ppc/crashdump-powerpc.c b/kexec/arch/ppc/crashdump-powerpc.c
index 1bef69b..4c8c75d 100644
--- a/kexec/arch/ppc/crashdump-powerpc.c
+++ b/kexec/arch/ppc/crashdump-powerpc.c
@@ -262,10 +262,19 @@ static void ulltoa(unsigned long long i, char *str)
 	}
 }
 
+/* Append str to cmdline */
+static void add_cmdline(char *cmdline, char *str)
+{
+	int cmdlen = strlen(cmdline) + strlen(str);
+	if (cmdlen > (COMMAND_LINE_SIZE - 1))
+		die("Command line overflow\n");
+	strcat(cmdline, str);
+}
+
 static int add_cmdline_param(char *cmdline, unsigned long long addr,
 				char *cmdstr, char *byte)
 {
-	int cmdlen, len, align = 1024;
+	int align = 1024;
 	char str[COMMAND_LINE_SIZE], *ptr;
 
 	/* Passing in =xxxK / =xxxM format. Saves space required in cmdline.*/
@@ -284,11 +293,8 @@ static int add_cmdline_param(char *cmdline, unsigned long long addr,
 	ptr += strlen(str);
 	ulltoa(addr, ptr);
 	strcat(str, byte);
-	len = strlen(str);
-	cmdlen = strlen(cmdline) + len;
-	if (cmdlen > (COMMAND_LINE_SIZE - 1))
-		die("Command line overflow\n");
-	strcat(cmdline, str);
+
+	add_cmdline(cmdline, str);
 
 	dbgprintf("Command line after adding elfcorehdr: %s\n", cmdline);
 
@@ -365,6 +371,7 @@ int load_crashdump_segments(struct kexec_info *info, char *mod_cmdline,
 	 */
 	add_cmdline_param(mod_cmdline, elfcorehdr, " elfcorehdr=", "K");
 	add_cmdline_param(mod_cmdline, saved_max_mem, " savemaxmem=", "M");
+	add_cmdline(mod_cmdline, " maxcpus=1");
 	return 0;
 }
 
diff --git a/kexec/arch/ppc/fixup_dtb.c b/kexec/arch/ppc/fixup_dtb.c
index e9890a4..f832026 100644
--- a/kexec/arch/ppc/fixup_dtb.c
+++ b/kexec/arch/ppc/fixup_dtb.c
@@ -172,6 +172,9 @@ static void fixup_reserve_regions(struct kexec_info *info, char *blob_buf)
 		}
 	}
 
+#if 0
+	/* XXX: Do not reserve spin-table for CPUs. */
+
 	/* Add reserve regions for cpu-release-addr */
 	nodeoffset = fdt_node_offset_by_prop_value(blob_buf, -1, "device_type", "cpu", 4);
 	while (nodeoffset != -FDT_ERR_NOTFOUND) {
@@ -201,6 +204,7 @@ static void fixup_reserve_regions(struct kexec_info *info, char *blob_buf)
 		nodeoffset = fdt_node_offset_by_prop_value(blob_buf, nodeoffset,
 				"device_type", "cpu", 4);
 	}
+#endif
 
 out:
 	print_fdt_reserve_regions(blob_buf);

Re: [PATCH] [ppc] Do not reserve cpu spin-table for crash kernel

[Suzuki K. Poulose]

On 05/24/2012 11:39 AM, Suzuki K. Poulose wrote:
> As of now, the kexec reserves the spin-table for all the CPUs
> on an SMP machine. The spin-table is pointed to by the
> cpu-release-addr property in the device-tree. Reserving the
> spin-table in the crash kernel will cause a BUG(), if the table
> lies outside the memory reserved for the crashkernel.
>
> Disable reserving the spin-table regions and use maxcpus=1 to
> use only the crashing CPU to boot the crash kernel.
>
> Signed-off-by: Suzuki K. Poulose<suzuki@in.ibm.com>

Simon,

Any response on this one ?

I have tested this on a Currituck board (476, SMP) with a UP kernel.
Without this patch, the secondary kernel hits 'PANIC' in boot while
trying to reserve a memory(the spin table), outside the memory
range(crash reserve).


Thanks
Suzuki

> ---
>
>   kexec/arch/ppc/crashdump-powerpc.c |   19 +++++++++++++------
>   kexec/arch/ppc/fixup_dtb.c         |    4 ++++
>   2 files changed, 17 insertions(+), 6 deletions(-)
>
> diff --git a/kexec/arch/ppc/crashdump-powerpc.c b/kexec/arch/ppc/crashdump-powerpc.c
> index 1bef69b..4c8c75d 100644
> --- a/kexec/arch/ppc/crashdump-powerpc.c
> +++ b/kexec/arch/ppc/crashdump-powerpc.c
> @@ -262,10 +262,19 @@ static void ulltoa(unsigned long long i, char *str)
>   	}
>   }
>
> +/* Append str to cmdline */
> +static void add_cmdline(char *cmdline, char *str)
> +{
> +	int cmdlen = strlen(cmdline) + strlen(str);
> +	if (cmdlen>  (COMMAND_LINE_SIZE - 1))
> +		die("Command line overflow\n");
> +	strcat(cmdline, str);
> +}
> +
>   static int add_cmdline_param(char *cmdline, unsigned long long addr,
>   				char *cmdstr, char *byte)
>   {
> -	int cmdlen, len, align = 1024;
> +	int align = 1024;
>   	char str[COMMAND_LINE_SIZE], *ptr;
>
>   	/* Passing in =xxxK / =xxxM format. Saves space required in cmdline.*/
> @@ -284,11 +293,8 @@ static int add_cmdline_param(char *cmdline, unsigned long long addr,
>   	ptr += strlen(str);
>   	ulltoa(addr, ptr);
>   	strcat(str, byte);
> -	len = strlen(str);
> -	cmdlen = strlen(cmdline) + len;
> -	if (cmdlen>  (COMMAND_LINE_SIZE - 1))
> -		die("Command line overflow\n");
> -	strcat(cmdline, str);
> +
> +	add_cmdline(cmdline, str);
>
>   	dbgprintf("Command line after adding elfcorehdr: %s\n", cmdline);
>
> @@ -365,6 +371,7 @@ int load_crashdump_segments(struct kexec_info *info, char *mod_cmdline,
>   	 */
>   	add_cmdline_param(mod_cmdline, elfcorehdr, " elfcorehdr=", "K");
>   	add_cmdline_param(mod_cmdline, saved_max_mem, " savemaxmem=", "M");
> +	add_cmdline(mod_cmdline, " maxcpus=1");
>   	return 0;
>   }
>
> diff --git a/kexec/arch/ppc/fixup_dtb.c b/kexec/arch/ppc/fixup_dtb.c
> index e9890a4..f832026 100644
> --- a/kexec/arch/ppc/fixup_dtb.c
> +++ b/kexec/arch/ppc/fixup_dtb.c
> @@ -172,6 +172,9 @@ static void fixup_reserve_regions(struct kexec_info *info, char *blob_buf)
>   		}
>   	}
>
> +#if 0
> +	/* XXX: Do not reserve spin-table for CPUs. */
> +
>   	/* Add reserve regions for cpu-release-addr */
>   	nodeoffset = fdt_node_offset_by_prop_value(blob_buf, -1, "device_type", "cpu", 4);
>   	while (nodeoffset != -FDT_ERR_NOTFOUND) {
> @@ -201,6 +204,7 @@ static void fixup_reserve_regions(struct kexec_info *info, char *blob_buf)
>   		nodeoffset = fdt_node_offset_by_prop_value(blob_buf, nodeoffset,
>   				"device_type", "cpu", 4);
>   	}
> +#endif
>
>   out:
>   	print_fdt_reserve_regions(blob_buf);
>
>
> _______________________________________________
> kexec mailing list
> kexec@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexec
>

Re: [PATCH] [ppc] Do not reserve cpu spin-table for crash kernel

[Simon Horman]

On Thu, May 24, 2012 at 11:39:45AM +0530, Suzuki K. Poulose wrote:
> As of now, the kexec reserves the spin-table for all the CPUs
> on an SMP machine. The spin-table is pointed to by the 
> cpu-release-addr property in the device-tree. Reserving the
> spin-table in the crash kernel will cause a BUG(), if the table
> lies outside the memory reserved for the crashkernel.
> 
> Disable reserving the spin-table regions and use maxcpus=1 to 
> use only the crashing CPU to boot the crash kernel.
> 
> Signed-off-by: Suzuki K. Poulose <suzuki@in.ibm.com>

Thanks, applied.

Sorry for not seeing this (much) earlier.