* Re: [powerpc][next-20210727] Boot failure - kernel BUG at arch/powerpc/kernel/interrupt.c:98!
2021-07-28 17:35 ` [powerpc][next-20210727] Boot failure - kernel BUG at arch/powerpc/kernel/interrupt.c:98! Nathan Chancellor
@ 2021-07-29 4:08 ` Nicholas Piggin
2021-07-29 4:21 ` Sachin Sant
2021-07-29 16:13 ` Will Deacon
2 siblings, 0 replies; 7+ messages in thread
From: Nicholas Piggin @ 2021-07-29 4:08 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk, Nathan Chancellor, Sachin Sant, Will Deacon
Cc: linuxppc-dev, iommu, linux-next, Claire Chang, Robin Murphy,
Christoph Hellwig, Anshuman Khandual
Excerpts from Nathan Chancellor's message of July 29, 2021 3:35 am:
> On Wed, Jul 28, 2021 at 01:31:06PM +0530, Sachin Sant wrote:
>> linux-next fails to boot on Power server (POWER8/POWER9). Following traces
>> are seen during boot
>>
>> [ 0.010799] software IO TLB: tearing down default memory pool
>> [ 0.010805] ------------[ cut here ]------------
>> [ 0.010808] kernel BUG at arch/powerpc/kernel/interrupt.c:98!
>> [ 0.010812] Oops: Exception in kernel mode, sig: 5 [#1]
>> [ 0.010816] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
>> [ 0.010820] Modules linked in:
>> [ 0.010824] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.14.0-rc3-next-20210727 #1
>> [ 0.010830] NIP: c000000000032cfc LR: c00000000000c764 CTR: c00000000000c670
>> [ 0.010834] REGS: c000000003603b10 TRAP: 0700 Not tainted (5.14.0-rc3-next-20210727)
>> [ 0.010838] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 28000222 XER: 00000002
>> [ 0.010848] CFAR: c00000000000c760 IRQMASK: 3
>> [ 0.010848] GPR00: c00000000000c764 c000000003603db0 c0000000029bd000 0000000000000001
>> [ 0.010848] GPR04: 0000000000000a68 0000000000000400 c000000003603868 ffffffffffffffff
>> [ 0.010848] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000003
>> [ 0.010848] GPR12: ffffffffffffffff c00000001ec9ee80 c000000000012a28 0000000000000000
>> [ 0.010848] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
>> [ 0.010848] GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
>> [ 0.010848] GPR24: 000000000000f134 0000000000000000 ffffffffffffffff c000000003603868
>> [ 0.010848] GPR28: 0000000000000400 0000000000000a68 c00000000202e9c0 c000000003603e80
>> [ 0.010896] NIP [c000000000032cfc] system_call_exception+0x8c/0x2e0
>> [ 0.010901] LR [c00000000000c764] system_call_common+0xf4/0x258
>> [ 0.010907] Call Trace:
>> [ 0.010909] [c000000003603db0] [c00000000016a6dc] calculate_sigpending+0x4c/0xe0 (unreliable)
>> [ 0.010915] [c000000003603e10] [c00000000000c764] system_call_common+0xf4/0x258
>> [ 0.010921] --- interrupt: c00 at kvm_template_end+0x4/0x8
>> [ 0.010926] NIP: c000000000092dec LR: c000000000114fc8 CTR: 0000000000000000
>> [ 0.010930] REGS: c000000003603e80 TRAP: 0c00 Not tainted (5.14.0-rc3-next-20210727)
>> [ 0.010934] MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE> CR: 28000222 XER: 00000000
>> [ 0.010943] IRQMASK: 0
>> [ 0.010943] GPR00: c00000000202e9c0 c000000003603b00 c0000000029bd000 000000000000f134
>> [ 0.010943] GPR04: 0000000000000a68 0000000000000400 c000000003603868 ffffffffffffffff
>> [ 0.010943] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
>> [ 0.010943] GPR12: 0000000000000000 c00000001ec9ee80 c000000000012a28 0000000000000000
>> [ 0.010943] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
>> [ 0.010943] GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
>> [ 0.010943] GPR24: c0000000020033c4 c00000000110afc0 c000000002081950 c000000003277d40
>> [ 0.010943] GPR28: 0000000000000000 c00000000a680000 0000000004000000 00000000000d0000
>> [ 0.010989] NIP [c000000000092dec] kvm_template_end+0x4/0x8
>> [ 0.010993] LR [c000000000114fc8] set_memory_encrypted+0x38/0x60
>> [ 0.010999] --- interrupt: c00
>> [ 0.011001] [c000000003603b00] [c00000000000c764] system_call_common+0xf4/0x258 (unreliable)
>> [ 0.011008] Instruction dump:
>> [ 0.011011] 694a0003 312affff 7d495110 0b0a0000 60000000 60000000 e87f0108 68690002
>> [ 0.011019] 7929ffe2 0b090000 68634000 786397e2 <0b030000> e93f0138 792907e0 0b090000
>> [ 0.011029] ---[ end trace a20ad55589efcb10 ]---
>> [ 0.012297]
>> [ 1.012304] Kernel panic - not syncing: Fatal exception
>>
>> next-20210723 was good. The boot failure seems to have been introduced with next-20210726.
>>
>> I have attached the boot log.
>
> I noticed this with OpenSUSE's ppc64le config [1] and my bisect landed on
> commit ad6c00283163 ("swiotlb: Free tbl memory in swiotlb_exit()"). That
> series just keeps on giving... Adding some people from that thread to
> this one. Original thread:
> https://lore.kernel.org/r/1905CD70-7656-42AE-99E2-A31FC3812EAC@linux.vnet.ibm.com/
This is because powerpc's set_memory_encrypted makes an ultracall but it
does not exist on that processor.
x86's set_memory_encrypted/decrypted have
/* Nothing to do if memory encryption is not active */
if (!mem_encrypt_active())
return 0;
Probably powerpc should just do that too.
Thanks,
Nick
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [powerpc][next-20210727] Boot failure - kernel BUG at arch/powerpc/kernel/interrupt.c:98!
2021-07-28 17:35 ` [powerpc][next-20210727] Boot failure - kernel BUG at arch/powerpc/kernel/interrupt.c:98! Nathan Chancellor
2021-07-29 4:08 ` Nicholas Piggin
@ 2021-07-29 4:21 ` Sachin Sant
2021-07-29 16:13 ` Will Deacon
2 siblings, 0 replies; 7+ messages in thread
From: Sachin Sant @ 2021-07-29 4:21 UTC (permalink / raw)
To: Nathan Chancellor
Cc: Konrad Rzeszutek Wilk, Will Deacon, Robin Murphy, iommu,
linux-next, Claire Chang, linuxppc-dev, Christoph Hellwig
> On 28-Jul-2021, at 11:05 PM, Nathan Chancellor <nathan@kernel.org> wrote:
>
> On Wed, Jul 28, 2021 at 01:31:06PM +0530, Sachin Sant wrote:
>> linux-next fails to boot on Power server (POWER8/POWER9). Following traces
>> are seen during boot
>>
>> [ 0.010799] software IO TLB: tearing down default memory pool
>> [ 0.010805] ------------[ cut here ]------------
>> [ 0.010808] kernel BUG at arch/powerpc/kernel/interrupt.c:98!
>> [ 0.010812] Oops: Exception in kernel mode, sig: 5 [#1]
…….
>
> I noticed this with OpenSUSE's ppc64le config [1] and my bisect landed on
> commit ad6c00283163 ("swiotlb: Free tbl memory in swiotlb_exit()"). That
Indeed. Thanks Nathan.
Bisect points to this commit. Reverting the commit allows the kernel to boot.
Thanks
-Sachin
> series just keeps on giving... Adding some people from that thread to
> this one. Original thread:
> https://lore.kernel.org/r/1905CD70-7656-42AE-99E2-A31FC3812EAC@linux.vnet.ibm.com/
>
> [1]: https://github.com/openSUSE/kernel-source/raw/master/config/ppc64le/default
>
> Cheers,
> Nathan
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [powerpc][next-20210727] Boot failure - kernel BUG at arch/powerpc/kernel/interrupt.c:98!
2021-07-28 17:35 ` [powerpc][next-20210727] Boot failure - kernel BUG at arch/powerpc/kernel/interrupt.c:98! Nathan Chancellor
2021-07-29 4:08 ` Nicholas Piggin
2021-07-29 4:21 ` Sachin Sant
@ 2021-07-29 16:13 ` Will Deacon
2021-07-29 16:35 ` Konrad Rzeszutek Wilk
2021-07-30 5:17 ` Sachin Sant
2 siblings, 2 replies; 7+ messages in thread
From: Will Deacon @ 2021-07-29 16:13 UTC (permalink / raw)
To: Nathan Chancellor
Cc: Sachin Sant, Konrad Rzeszutek Wilk, Robin Murphy, iommu,
linux-next, Claire Chang, linuxppc-dev, Christoph Hellwig
On Wed, Jul 28, 2021 at 10:35:34AM -0700, Nathan Chancellor wrote:
> On Wed, Jul 28, 2021 at 01:31:06PM +0530, Sachin Sant wrote:
> > next-20210723 was good. The boot failure seems to have been introduced with next-20210726.
> >
> > I have attached the boot log.
>
> I noticed this with OpenSUSE's ppc64le config [1] and my bisect landed on
> commit ad6c00283163 ("swiotlb: Free tbl memory in swiotlb_exit()"). That
> series just keeps on giving...
Yes, but look how handy our new print is!
[ 0.010799] software IO TLB: tearing down default memory pool
[ 0.010805] ------------[ cut here ]------------
[ 0.010808] kernel BUG at arch/powerpc/kernel/interrupt.c:98!
Following Nick's suggestion, the diff below should help? I don't have a
relevant box on which I can test it though.
Will
--->8
diff --git a/arch/powerpc/platforms/pseries/svm.c b/arch/powerpc/platforms/pseries/svm.c
index 1d829e257996..87f001b4c4e4 100644
--- a/arch/powerpc/platforms/pseries/svm.c
+++ b/arch/powerpc/platforms/pseries/svm.c
@@ -63,6 +63,9 @@ void __init svm_swiotlb_init(void)
int set_memory_encrypted(unsigned long addr, int numpages)
{
+ if (!mem_encrypt_active())
+ return 0;
+
if (!PAGE_ALIGNED(addr))
return -EINVAL;
@@ -73,6 +76,9 @@ int set_memory_encrypted(unsigned long addr, int numpages)
int set_memory_decrypted(unsigned long addr, int numpages)
{
+ if (!mem_encrypt_active())
+ return 0;
+
if (!PAGE_ALIGNED(addr))
return -EINVAL;
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
^ permalink raw reply related [flat|nested] 7+ messages in thread* Re: [powerpc][next-20210727] Boot failure - kernel BUG at arch/powerpc/kernel/interrupt.c:98!
2021-07-29 16:13 ` Will Deacon
@ 2021-07-29 16:35 ` Konrad Rzeszutek Wilk
2021-07-29 19:05 ` Nathan Chancellor
2021-07-30 5:17 ` Sachin Sant
1 sibling, 1 reply; 7+ messages in thread
From: Konrad Rzeszutek Wilk @ 2021-07-29 16:35 UTC (permalink / raw)
To: Will Deacon
Cc: Sachin Sant, Robin Murphy, Nathan Chancellor, iommu, linux-next,
Claire Chang, linuxppc-dev, Christoph Hellwig
On Thu, Jul 29, 2021 at 05:13:36PM +0100, Will Deacon wrote:
> On Wed, Jul 28, 2021 at 10:35:34AM -0700, Nathan Chancellor wrote:
> > On Wed, Jul 28, 2021 at 01:31:06PM +0530, Sachin Sant wrote:
> > > next-20210723 was good. The boot failure seems to have been introduced with next-20210726.
> > >
> > > I have attached the boot log.
> >
> > I noticed this with OpenSUSE's ppc64le config [1] and my bisect landed on
> > commit ad6c00283163 ("swiotlb: Free tbl memory in swiotlb_exit()"). That
> > series just keeps on giving...
Low-level across platform do that. And thank you for testing it and
finding this bug. Please let me know if the patch works so I can add it
in in the patch series.
>
> Yes, but look how handy our new print is!
:)
>
> [ 0.010799] software IO TLB: tearing down default memory pool
> [ 0.010805] ------------[ cut here ]------------
> [ 0.010808] kernel BUG at arch/powerpc/kernel/interrupt.c:98!
>
> Following Nick's suggestion, the diff below should help? I don't have a
> relevant box on which I can test it though.
>
> Will
>
> --->8
>
> diff --git a/arch/powerpc/platforms/pseries/svm.c b/arch/powerpc/platforms/pseries/svm.c
> index 1d829e257996..87f001b4c4e4 100644
> --- a/arch/powerpc/platforms/pseries/svm.c
> +++ b/arch/powerpc/platforms/pseries/svm.c
> @@ -63,6 +63,9 @@ void __init svm_swiotlb_init(void)
>
> int set_memory_encrypted(unsigned long addr, int numpages)
> {
> + if (!mem_encrypt_active())
> + return 0;
> +
> if (!PAGE_ALIGNED(addr))
> return -EINVAL;
>
> @@ -73,6 +76,9 @@ int set_memory_encrypted(unsigned long addr, int numpages)
>
> int set_memory_decrypted(unsigned long addr, int numpages)
> {
> + if (!mem_encrypt_active())
> + return 0;
> +
> if (!PAGE_ALIGNED(addr))
> return -EINVAL;
>
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [powerpc][next-20210727] Boot failure - kernel BUG at arch/powerpc/kernel/interrupt.c:98!
2021-07-29 16:35 ` Konrad Rzeszutek Wilk
@ 2021-07-29 19:05 ` Nathan Chancellor
0 siblings, 0 replies; 7+ messages in thread
From: Nathan Chancellor @ 2021-07-29 19:05 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk, Will Deacon
Cc: Sachin Sant, Robin Murphy, iommu, linux-next, Claire Chang,
linuxppc-dev, Christoph Hellwig
On 7/29/2021 9:35 AM, Konrad Rzeszutek Wilk wrote:
> On Thu, Jul 29, 2021 at 05:13:36PM +0100, Will Deacon wrote:
>> On Wed, Jul 28, 2021 at 10:35:34AM -0700, Nathan Chancellor wrote:
>>> On Wed, Jul 28, 2021 at 01:31:06PM +0530, Sachin Sant wrote:
>>>> next-20210723 was good. The boot failure seems to have been introduced with next-20210726.
>>>>
>>>> I have attached the boot log.
>>>
>>> I noticed this with OpenSUSE's ppc64le config [1] and my bisect landed on
>>> commit ad6c00283163 ("swiotlb: Free tbl memory in swiotlb_exit()"). That
>>> series just keeps on giving...
>
> Low-level across platform do that. And thank you for testing it and
> finding this bug. Please let me know if the patch works so I can add it
> in in the patch series.
That was not meant to sound as sarcastic as it did so my apologies for
that :(
Will's patch looks good to me in QEMU, I do not have a bare metal POWER
system to test it on.
Tested-by: Nathan Chancellor <nathan@kernel.org>
>>
>> Yes, but look how handy our new print is!
>
> :)
>>
>> [ 0.010799] software IO TLB: tearing down default memory pool
>> [ 0.010805] ------------[ cut here ]------------
>> [ 0.010808] kernel BUG at arch/powerpc/kernel/interrupt.c:98!
>>
>> Following Nick's suggestion, the diff below should help? I don't have a
>> relevant box on which I can test it though.
>>
>> Will
>>
>> --->8
>>
>> diff --git a/arch/powerpc/platforms/pseries/svm.c b/arch/powerpc/platforms/pseries/svm.c
>> index 1d829e257996..87f001b4c4e4 100644
>> --- a/arch/powerpc/platforms/pseries/svm.c
>> +++ b/arch/powerpc/platforms/pseries/svm.c
>> @@ -63,6 +63,9 @@ void __init svm_swiotlb_init(void)
>>
>> int set_memory_encrypted(unsigned long addr, int numpages)
>> {
>> + if (!mem_encrypt_active())
>> + return 0;
>> +
>> if (!PAGE_ALIGNED(addr))
>> return -EINVAL;
>>
>> @@ -73,6 +76,9 @@ int set_memory_encrypted(unsigned long addr, int numpages)
>>
>> int set_memory_decrypted(unsigned long addr, int numpages)
>> {
>> + if (!mem_encrypt_active())
>> + return 0;
>> +
>> if (!PAGE_ALIGNED(addr))
>> return -EINVAL;
>>
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [powerpc][next-20210727] Boot failure - kernel BUG at arch/powerpc/kernel/interrupt.c:98!
2021-07-29 16:13 ` Will Deacon
2021-07-29 16:35 ` Konrad Rzeszutek Wilk
@ 2021-07-30 5:17 ` Sachin Sant
1 sibling, 0 replies; 7+ messages in thread
From: Sachin Sant @ 2021-07-30 5:17 UTC (permalink / raw)
To: Will Deacon
Cc: Konrad Rzeszutek Wilk, Robin Murphy, Nathan Chancellor, iommu,
linux-next, Claire Chang, linuxppc-dev, Christoph Hellwig
> On 29-Jul-2021, at 9:43 PM, Will Deacon <will@kernel.org> wrote:
>
> On Wed, Jul 28, 2021 at 10:35:34AM -0700, Nathan Chancellor wrote:
>> On Wed, Jul 28, 2021 at 01:31:06PM +0530, Sachin Sant wrote:
>>> next-20210723 was good. The boot failure seems to have been introduced with next-20210726.
>>>
>>> I have attached the boot log.
>>
>> I noticed this with OpenSUSE's ppc64le config [1] and my bisect landed on
>> commit ad6c00283163 ("swiotlb: Free tbl memory in swiotlb_exit()"). That
>> series just keeps on giving...
>
> Yes, but look how handy our new print is!
>
> [ 0.010799] software IO TLB: tearing down default memory pool
> [ 0.010805] ------------[ cut here ]------------
> [ 0.010808] kernel BUG at arch/powerpc/kernel/interrupt.c:98!
>
> Following Nick's suggestion, the diff below should help? I don't have a
> relevant box on which I can test it though.
>
Thanks for the fix. This fixes the reported problem for me.
Tested successfully on both PowerVM LPAR as well as bare metal environment.
Reported-by: Sachin Sant <sachinp@linux.vnet.ibm.com>
Tested-by: Sachin Sant <sachinp@linux.vnet.ibm.com>
> Will
>
> --->8
>
> diff --git a/arch/powerpc/platforms/pseries/svm.c b/arch/powerpc/platforms/pseries/svm.c
> index 1d829e257996..87f001b4c4e4 100644
> --- a/arch/powerpc/platforms/pseries/svm.c
> +++ b/arch/powerpc/platforms/pseries/svm.c
> @@ -63,6 +63,9 @@ void __init svm_swiotlb_init(void)
>
> int set_memory_encrypted(unsigned long addr, int numpages)
> {
> + if (!mem_encrypt_active())
> + return 0;
> +
> if (!PAGE_ALIGNED(addr))
> return -EINVAL;
>
> @@ -73,6 +76,9 @@ int set_memory_encrypted(unsigned long addr, int numpages)
>
> int set_memory_decrypted(unsigned long addr, int numpages)
> {
> + if (!mem_encrypt_active())
> + return 0;
> +
> if (!PAGE_ALIGNED(addr))
> return -EINVAL;
>
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
^ permalink raw reply [flat|nested] 7+ messages in thread