[next-20190530] Boot failure on PowerPC

[next-20190530] Boot failure on PowerPC

[Sachin Sant]

Latest next fails to boot with a kernel panic on POWER9.

[   33.689332] Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: write_irq_affinity.isra.5+0x15c/0x160
[   33.689346] CPU: 35 PID: 4907 Comm: irqbalance Not tainted 5.2.0-rc2-next-20190530-autotest-autotest #1
[   33.689352] Call Trace:
[   33.689356] [c0000018d974bab0] [c000000000b5328c] dump_stack+0xb0/0xf4 (unreliable)
[   33.689364] [c0000018d974baf0] [c000000000120694] panic+0x16c/0x408
[   33.689370] [c0000018d974bb80] [c00000000012010c] __stack_chk_fail+0x2c/0x30
[   33.689376] [c0000018d974bbe0] [c0000000001b859c] write_irq_affinity.isra.5+0x15c/0x160
[   33.689383] [c0000018d974bd30] [c0000000004d6f30] proc_reg_write+0x90/0x110
[   33.689388] [c0000018d974bd60] [c00000000041453c] __vfs_write+0x3c/0x70
[   33.689394] [c0000018d974bd80] [c000000000418650] vfs_write+0xd0/0x250
[   33.689399] [c0000018d974bdd0] [c000000000418a2c] ksys_write+0x7c/0x130
[   33.689405] [c0000018d974be20] [c00000000000b688] system_call+0x5c/0x70

Machine boots till login prompt and then panics few seconds later.

Last known next build was May 24th. Will attempt few builds till May 30 to
narrow down this problem.

Thanks
-Sachin

Re: [next-20190530] Boot failure on PowerPC

[Michael Ellerman]

Sachin Sant <sachinp@linux.vnet.ibm.com> writes:
> Latest next fails to boot with a kernel panic on POWER9.
>
> [   33.689332] Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: write_irq_affinity.isra.5+0x15c/0x160
> [   33.689346] CPU: 35 PID: 4907 Comm: irqbalance Not tainted 5.2.0-rc2-next-20190530-autotest-autotest #1
> [   33.689352] Call Trace:
> [   33.689356] [c0000018d974bab0] [c000000000b5328c] dump_stack+0xb0/0xf4 (unreliable)
> [   33.689364] [c0000018d974baf0] [c000000000120694] panic+0x16c/0x408
> [   33.689370] [c0000018d974bb80] [c00000000012010c] __stack_chk_fail+0x2c/0x30
> [   33.689376] [c0000018d974bbe0] [c0000000001b859c] write_irq_affinity.isra.5+0x15c/0x160
> [   33.689383] [c0000018d974bd30] [c0000000004d6f30] proc_reg_write+0x90/0x110
> [   33.689388] [c0000018d974bd60] [c00000000041453c] __vfs_write+0x3c/0x70
> [   33.689394] [c0000018d974bd80] [c000000000418650] vfs_write+0xd0/0x250
> [   33.689399] [c0000018d974bdd0] [c000000000418a2c] ksys_write+0x7c/0x130
> [   33.689405] [c0000018d974be20] [c00000000000b688] system_call+0x5c/0x70
>
> Machine boots till login prompt and then panics few seconds later.
>
> Last known next build was May 24th. Will attempt few builds till May 30 to
> narrow down this problem.

My CI was fine with next-20190529 (9a15d2e3fd03e3).

cheers

Re: [next-20190530] Boot failure on PowerPC

[Dexuan-Linux Cui]

On Fri, May 31, 2019 at 6:52 AM Michael Ellerman <mpe@ellerman.id.au> wrote:
>
> Sachin Sant <sachinp@linux.vnet.ibm.com> writes:
> > Latest next fails to boot with a kernel panic on POWER9.
> >
> > [   33.689332] Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: write_irq_affinity.isra.5+0x15c/0x160
> > [   33.689346] CPU: 35 PID: 4907 Comm: irqbalance Not tainted 5.2.0-rc2-next-20190530-autotest-autotest #1
> > [   33.689352] Call Trace:
> > [   33.689356] [c0000018d974bab0] [c000000000b5328c] dump_stack+0xb0/0xf4 (unreliable)
> > [   33.689364] [c0000018d974baf0] [c000000000120694] panic+0x16c/0x408
> > [   33.689370] [c0000018d974bb80] [c00000000012010c] __stack_chk_fail+0x2c/0x30
> > [   33.689376] [c0000018d974bbe0] [c0000000001b859c] write_irq_affinity.isra.5+0x15c/0x160
> > [   33.689383] [c0000018d974bd30] [c0000000004d6f30] proc_reg_write+0x90/0x110
> > [   33.689388] [c0000018d974bd60] [c00000000041453c] __vfs_write+0x3c/0x70
> > [   33.689394] [c0000018d974bd80] [c000000000418650] vfs_write+0xd0/0x250
> > [   33.689399] [c0000018d974bdd0] [c000000000418a2c] ksys_write+0x7c/0x130
> > [   33.689405] [c0000018d974be20] [c00000000000b688] system_call+0x5c/0x70
> >
> > Machine boots till login prompt and then panics few seconds later.
> >
> > Last known next build was May 24th. Will attempt few builds till May 30 to
> > narrow down this problem.
>
> My CI was fine with next-20190529 (9a15d2e3fd03e3).
>
> cheers

Hi Sachin,
It looks this patch may fix the issue:
https://lkml.org/lkml/2019/5/30/1630 , but I'm not sure.

Thanks
-- Dexuan

Re: [next-20190530] Boot failure on PowerPC

[Sachin Sant]

> On 31-May-2019, at 11:43 PM, Dexuan-Linux Cui <dexuan.linux@gmail.com> wrote:
> 
> On Fri, May 31, 2019 at 6:52 AM Michael Ellerman <mpe@ellerman.id.au> wrote:
>>> 
>>> Machine boots till login prompt and then panics few seconds later.
>>> 
>>> Last known next build was May 24th. Will attempt few builds till May 30 to
>>> narrow down this problem.
>> 
>> My CI was fine with next-20190529 (9a15d2e3fd03e3).
>> 
>> cheers
> 
> Hi Sachin,
> It looks this patch may fix the issue:
> https://lkml.org/lkml/2019/5/30/1630 , but I'm not sure.

It does not help fix the kernel panic issue, but it fixes the get_swap_device warning
messages during the boot.

Thanks
-Sachin

RE: [next-20190530] Boot failure on PowerPC

[Lili Deng (Wicresoft North America Ltd)]

Hi Sachin,

I verified below patch against Ubuntu 18.04, didn't hit the kernel panic any more, could you please let know how did you verify?

Thanks,
Lili
-----Original Message-----
From: Sachin Sant <sachinp@linux.vnet.ibm.com> 
Sent: Monday, June 3, 2019 11:12 AM
To: Dexuan-Linux Cui <dexuan.linux@gmail.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>; linuxppc-dev@lists.ozlabs.org; linux-next@vger.kernel.org; Dexuan Cui <decui@microsoft.com>; Lili Deng (Wicresoft North America Ltd) <v-lide@microsoft.com>
Subject: Re: [next-20190530] Boot failure on PowerPC



> On 31-May-2019, at 11:43 PM, Dexuan-Linux Cui <dexuan.linux@gmail.com> wrote:
> 
> On Fri, May 31, 2019 at 6:52 AM Michael Ellerman <mpe@ellerman.id.au> wrote:
>>> 
>>> Machine boots till login prompt and then panics few seconds later.
>>> 
>>> Last known next build was May 24th. Will attempt few builds till May 
>>> 30 to narrow down this problem.
>> 
>> My CI was fine with next-20190529 (9a15d2e3fd03e3).
>> 
>> cheers
> 
> Hi Sachin,
> It looks this patch may fix the issue:
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flkml.org%2Flkml%2F2019%2F5%2F30%2F1630&amp;data=02%7C01%7Cv-lide%40microsoft.com%7C66e1ef6017aa461703f808d6e7d148cd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C636951283393233385&amp;sdata=IJFhtvL2Bd87HCoMZ7oWL%2Bar6NY%2FfPbmdCZMT%2BJz5t4%3D&amp;reserved=0 , but I'm not sure.

It does not help fix the kernel panic issue, but it fixes the get_swap_device warning messages during the boot.

Thanks
-Sachin

RE: [next-20190530] Boot failure on PowerPC

[Lili Deng (Wicresoft North America Ltd)]

ACK for ' but it fixes the get_swap_device warning messages during the boot.'
Double check for kernel panic issue, without this patch, it seems not repro in my local manual environment, so please ignore my previous mail for 'the patch fixes the kernel panic'.
Sorry for the confusion. 

-----Original Message-----
From: Lili Deng (Wicresoft North America Ltd) 
Sent: Monday, June 3, 2019 11:24 AM
To: Sachin Sant <sachinp@linux.vnet.ibm.com>; Dexuan-Linux Cui <dexuan.linux@gmail.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>; linuxppc-dev@lists.ozlabs.org; linux-next@vger.kernel.org; Dexuan Cui <decui@microsoft.com>
Subject: RE: [next-20190530] Boot failure on PowerPC

Hi Sachin,

I verified below patch against Ubuntu 18.04, didn't hit the kernel panic any more, could you please let know how did you verify?

Thanks,
Lili
-----Original Message-----
From: Sachin Sant <sachinp@linux.vnet.ibm.com> 
Sent: Monday, June 3, 2019 11:12 AM
To: Dexuan-Linux Cui <dexuan.linux@gmail.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>; linuxppc-dev@lists.ozlabs.org; linux-next@vger.kernel.org; Dexuan Cui <decui@microsoft.com>; Lili Deng (Wicresoft North America Ltd) <v-lide@microsoft.com>
Subject: Re: [next-20190530] Boot failure on PowerPC



> On 31-May-2019, at 11:43 PM, Dexuan-Linux Cui <dexuan.linux@gmail.com> wrote:
> 
> On Fri, May 31, 2019 at 6:52 AM Michael Ellerman <mpe@ellerman.id.au> wrote:
>>> 
>>> Machine boots till login prompt and then panics few seconds later.
>>> 
>>> Last known next build was May 24th. Will attempt few builds till May 
>>> 30 to narrow down this problem.
>> 
>> My CI was fine with next-20190529 (9a15d2e3fd03e3).
>> 
>> cheers
> 
> Hi Sachin,
> It looks this patch may fix the issue:
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flkml.org%2Flkml%2F2019%2F5%2F30%2F1630&amp;data=02%7C01%7Cv-lide%40microsoft.com%7C66e1ef6017aa461703f808d6e7d148cd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C636951283393233385&amp;sdata=IJFhtvL2Bd87HCoMZ7oWL%2Bar6NY%2FfPbmdCZMT%2BJz5t4%3D&amp;reserved=0 , but I'm not sure.

It does not help fix the kernel panic issue, but it fixes the get_swap_device warning messages during the boot.

Thanks
-Sachin