pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[Sebastian Andrzej Siewior]

With qemu-system-ppc64le -machine pseries -smp 4 I get:

|#  chrt 1 hackbench
|Running in process mode with 10 groups using 40 file descriptors each (== 400 tasks)
|Each sender will pass 100 messages of 100 bytes
| Oops: Exception in kernel mode, sig: 4 [#1]
| LE PAGE_SIZE=64K MMU=Hash PREEMPT SMP NR_CPUS=2048 NUMA pSeries
| Modules linked in:
| CPU: 0 PID: 629 Comm: hackbench Not tainted 5.1.0-rc2 #71
| NIP:  c000000000046978 LR: c000000000046a38 CTR: c0000000000b0150
| REGS: c0000001fffeb8e0 TRAP: 0700   Not tainted  (5.1.0-rc2)
| MSR:  8000000000089033 <SF,EE,ME,IR,DR,RI,LE>  CR: 42000874  XER: 00000000
| CFAR: c000000000046a34 IRQMASK: 1
| GPR00: c0000000000b0170 c0000001fffebb70 c000000000a6ba00 0000000028000000
…
| NIP [c000000000046978] doorbell_core_ipi+0x28/0x30
| LR [c000000000046a38] doorbell_try_core_ipi+0xb8/0xf0
| Call Trace:
| [c0000001fffebb70] [c0000001fffebba0] 0xc0000001fffebba0 (unreliable)
| [c0000001fffebba0] [c0000000000b0170] smp_pseries_cause_ipi+0x20/0x70
| [c0000001fffebbd0] [c00000000004b02c] arch_send_call_function_single_ipi+0x8c/0xa0
| [c0000001fffebbf0] [c0000000001de600] irq_work_queue_on+0xe0/0x130
| [c0000001fffebc30] [c0000000001340c8] rto_push_irq_work_func+0xc8/0x120
…
| Instruction dump:
| 60000000 60000000 3c4c00a2 384250b0 3d220009 392949c8 81290000 3929ffff
| 7d231838 7c0004ac 5463017e 64632800 <7c00191c> 4e800020 3c4c00a2 38425080
| ---[ end trace eb842b544538cbdf ]---

and I was wondering whether this is a qemu bug or the kernel is using an
opcode it should rather not. If I skip doorbell_try_core_ipi() in
smp_pseries_cause_ipi() then there is no crash. The comment says "POWER9
should not use this handler" so…

(This is QEMU emulator version 3.1.0 (Debian 1:3.1+dfsg-6))

Sebastian

Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[Cédric Le Goater]

On 3/27/19 1:36 PM, Sebastian Andrzej Siewior wrote:
> With qemu-system-ppc64le -machine pseries -smp 4 I get:
> 
> |#  chrt 1 hackbench
> |Running in process mode with 10 groups using 40 file descriptors each (== 400 tasks)
> |Each sender will pass 100 messages of 100 bytes
> | Oops: Exception in kernel mode, sig: 4 [#1]
> | LE PAGE_SIZE=64K MMU=Hash PREEMPT SMP NR_CPUS=2048 NUMA pSeries
> | Modules linked in:
> | CPU: 0 PID: 629 Comm: hackbench Not tainted 5.1.0-rc2 #71
> | NIP:  c000000000046978 LR: c000000000046a38 CTR: c0000000000b0150
> | REGS: c0000001fffeb8e0 TRAP: 0700   Not tainted  (5.1.0-rc2)
> | MSR:  8000000000089033 <SF,EE,ME,IR,DR,RI,LE>  CR: 42000874  XER: 00000000
> | CFAR: c000000000046a34 IRQMASK: 1
> | GPR00: c0000000000b0170 c0000001fffebb70 c000000000a6ba00 0000000028000000
> …
> | NIP [c000000000046978] doorbell_core_ipi+0x28/0x30
> | LR [c000000000046a38] doorbell_try_core_ipi+0xb8/0xf0
> | Call Trace:
> | [c0000001fffebb70] [c0000001fffebba0] 0xc0000001fffebba0 (unreliable)
> | [c0000001fffebba0] [c0000000000b0170] smp_pseries_cause_ipi+0x20/0x70
> | [c0000001fffebbd0] [c00000000004b02c] arch_send_call_function_single_ipi+0x8c/0xa0
> | [c0000001fffebbf0] [c0000000001de600] irq_work_queue_on+0xe0/0x130
> | [c0000001fffebc30] [c0000000001340c8] rto_push_irq_work_func+0xc8/0x120
> …
> | Instruction dump:
> | 60000000 60000000 3c4c00a2 384250b0 3d220009 392949c8 81290000 3929ffff
> | 7d231838 7c0004ac 5463017e 64632800 <7c00191c> 4e800020 3c4c00a2 38425080
> | ---[ end trace eb842b544538cbdf ]---
> 
> and I was wondering whether this is a qemu bug or the kernel is using an
> opcode it should rather not. If I skip doorbell_try_core_ipi() in
> smp_pseries_cause_ipi() then there is no crash. The comment says "POWER9
> should not use this handler" so…

I would say Linux is using a msgsndp instruction which is not implemented
in QEMU TCG. But why have we started using dbells in Linux ? 

C. 

Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[Cédric Le Goater]

On 3/27/19 5:37 PM, Cédric Le Goater wrote:
> On 3/27/19 1:36 PM, Sebastian Andrzej Siewior wrote:
>> With qemu-system-ppc64le -machine pseries -smp 4 I get:
>>
>> |#  chrt 1 hackbench
>> |Running in process mode with 10 groups using 40 file descriptors each (== 400 tasks)
>> |Each sender will pass 100 messages of 100 bytes
>> | Oops: Exception in kernel mode, sig: 4 [#1]
>> | LE PAGE_SIZE=64K MMU=Hash PREEMPT SMP NR_CPUS=2048 NUMA pSeries
>> | Modules linked in:
>> | CPU: 0 PID: 629 Comm: hackbench Not tainted 5.1.0-rc2 #71
>> | NIP:  c000000000046978 LR: c000000000046a38 CTR: c0000000000b0150
>> | REGS: c0000001fffeb8e0 TRAP: 0700   Not tainted  (5.1.0-rc2)
>> | MSR:  8000000000089033 <SF,EE,ME,IR,DR,RI,LE>  CR: 42000874  XER: 00000000
>> | CFAR: c000000000046a34 IRQMASK: 1
>> | GPR00: c0000000000b0170 c0000001fffebb70 c000000000a6ba00 0000000028000000
>> …
>> | NIP [c000000000046978] doorbell_core_ipi+0x28/0x30
>> | LR [c000000000046a38] doorbell_try_core_ipi+0xb8/0xf0
>> | Call Trace:
>> | [c0000001fffebb70] [c0000001fffebba0] 0xc0000001fffebba0 (unreliable)
>> | [c0000001fffebba0] [c0000000000b0170] smp_pseries_cause_ipi+0x20/0x70
>> | [c0000001fffebbd0] [c00000000004b02c] arch_send_call_function_single_ipi+0x8c/0xa0
>> | [c0000001fffebbf0] [c0000000001de600] irq_work_queue_on+0xe0/0x130
>> | [c0000001fffebc30] [c0000000001340c8] rto_push_irq_work_func+0xc8/0x120
>> …
>> | Instruction dump:
>> | 60000000 60000000 3c4c00a2 384250b0 3d220009 392949c8 81290000 3929ffff
>> | 7d231838 7c0004ac 5463017e 64632800 <7c00191c> 4e800020 3c4c00a2 38425080
>> | ---[ end trace eb842b544538cbdf ]---
>>
>> and I was wondering whether this is a qemu bug or the kernel is using an
>> opcode it should rather not. If I skip doorbell_try_core_ipi() in
>> smp_pseries_cause_ipi() then there is no crash. The comment says "POWER9
>> should not use this handler" so…
> 
> I would say Linux is using a msgsndp instruction which is not implemented
> in QEMU TCG. But why have we started using dbells in Linux ? 

ah. It seems arch_local_irq_restore() / replay_interrupt() generated
some interrupt.

C.

Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[Suraj Jitindar Singh]

On Wed, 2019-03-27 at 17:51 +0100, Cédric Le Goater wrote:
> On 3/27/19 5:37 PM, Cédric Le Goater wrote:
> > On 3/27/19 1:36 PM, Sebastian Andrzej Siewior wrote:
> > > With qemu-system-ppc64le -machine pseries -smp 4 I get:
> > > 
> > > > #  chrt 1 hackbench
> > > > Running in process mode with 10 groups using 40 file
> > > > descriptors each (== 400 tasks)
> > > > Each sender will pass 100 messages of 100 bytes
> > > > Oops: Exception in kernel mode, sig: 4 [#1]
> > > > LE PAGE_SIZE=64K MMU=Hash PREEMPT SMP NR_CPUS=2048 NUMA pSeries
> > > > Modules linked in:
> > > > CPU: 0 PID: 629 Comm: hackbench Not tainted 5.1.0-rc2 #71
> > > > NIP:  c000000000046978 LR: c000000000046a38 CTR:
> > > > c0000000000b0150
> > > > REGS: c0000001fffeb8e0 TRAP: 0700   Not tainted  (5.1.0-rc2)
> > > > MSR:  8000000000089033 <SF,EE,ME,IR,DR,RI,LE>  CR:
> > > > 42000874  XER: 00000000
> > > > CFAR: c000000000046a34 IRQMASK: 1
> > > > GPR00: c0000000000b0170 c0000001fffebb70 c000000000a6ba00
> > > > 0000000028000000
> > > 
> > > …
> > > > NIP [c000000000046978] doorbell_core_ipi+0x28/0x30
> > > > LR [c000000000046a38] doorbell_try_core_ipi+0xb8/0xf0
> > > > Call Trace:
> > > > [c0000001fffebb70] [c0000001fffebba0] 0xc0000001fffebba0
> > > > (unreliable)
> > > > [c0000001fffebba0] [c0000000000b0170]
> > > > smp_pseries_cause_ipi+0x20/0x70
> > > > [c0000001fffebbd0] [c00000000004b02c]
> > > > arch_send_call_function_single_ipi+0x8c/0xa0
> > > > [c0000001fffebbf0] [c0000000001de600]
> > > > irq_work_queue_on+0xe0/0x130
> > > > [c0000001fffebc30] [c0000000001340c8]
> > > > rto_push_irq_work_func+0xc8/0x120
> > > 
> > > …
> > > > Instruction dump:
> > > > 60000000 60000000 3c4c00a2 384250b0 3d220009 392949c8 81290000
> > > > 3929ffff
> > > > 7d231838 7c0004ac 5463017e 64632800 <7c00191c> 4e800020
> > > > 3c4c00a2 38425080
> > > > ---[ end trace eb842b544538cbdf ]---
> > > 
> > > and I was wondering whether this is a qemu bug or the kernel is
> > > using an
> > > opcode it should rather not. If I skip doorbell_try_core_ipi() in
> > > smp_pseries_cause_ipi() then there is no crash. The comment says
> > > "POWER9
> > > should not use this handler" so…
> > 
> > I would say Linux is using a msgsndp instruction which is not
> > implemented
> > in QEMU TCG. But why have we started using dbells in Linux ? 

Yeah the kernel must have used msgsndp which isn't implemented for TCG
yet. We use doorbells in linux but only for threads which are on the
same core.
And when I try to construct a situation with more than 1 thread per
core (e.g. -smp 4,threads=4), I get "TCG cannot support more than 1
thread/core on a pseries machine".

So I wonder why the guest thinks it can use msgsndp...

> 
> ah. It seems arch_local_irq_restore() / replay_interrupt() generated
> some interrupt.
> 
> C.
> 

Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[Sebastian Andrzej Siewior]

On 2019-03-29 16:20:51 [+1100], Suraj Jitindar Singh wrote:
> 
> Yeah the kernel must have used msgsndp which isn't implemented for TCG
> yet. We use doorbells in linux but only for threads which are on the
> same core.

It is msgsndp as per instruction decode.

> And when I try to construct a situation with more than 1 thread per
> core (e.g. -smp 4,threads=4), I get "TCG cannot support more than 1
> thread/core on a pseries machine".
>
> So I wonder why the guest thinks it can use msgsndp...

I see
|# cat /sys/devices/system/cpu/cpu*/topology/thread_siblings_list
|0
|1
|2
|3

so this works all well. The problem is when a CPU sends an IPI to itself
then linux's doorbell_try_core_ipi() considers this as a valid sibling
and here we have the msgsndp.

Sebastian

Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[Nicholas Piggin]

Suraj Jitindar Singh's on March 29, 2019 3:20 pm:
> On Wed, 2019-03-27 at 17:51 +0100, Cédric Le Goater wrote:
>> On 3/27/19 5:37 PM, Cédric Le Goater wrote:
>> > On 3/27/19 1:36 PM, Sebastian Andrzej Siewior wrote:
>> > > With qemu-system-ppc64le -machine pseries -smp 4 I get:
>> > > 
>> > > > #  chrt 1 hackbench
>> > > > Running in process mode with 10 groups using 40 file
>> > > > descriptors each (== 400 tasks)
>> > > > Each sender will pass 100 messages of 100 bytes
>> > > > Oops: Exception in kernel mode, sig: 4 [#1]
>> > > > LE PAGE_SIZE=64K MMU=Hash PREEMPT SMP NR_CPUS=2048 NUMA pSeries
>> > > > Modules linked in:
>> > > > CPU: 0 PID: 629 Comm: hackbench Not tainted 5.1.0-rc2 #71
>> > > > NIP:  c000000000046978 LR: c000000000046a38 CTR:
>> > > > c0000000000b0150
>> > > > REGS: c0000001fffeb8e0 TRAP: 0700   Not tainted  (5.1.0-rc2)
>> > > > MSR:  8000000000089033 <SF,EE,ME,IR,DR,RI,LE>  CR:
>> > > > 42000874  XER: 00000000
>> > > > CFAR: c000000000046a34 IRQMASK: 1
>> > > > GPR00: c0000000000b0170 c0000001fffebb70 c000000000a6ba00
>> > > > 0000000028000000
>> > > 
>> > > …
>> > > > NIP [c000000000046978] doorbell_core_ipi+0x28/0x30
>> > > > LR [c000000000046a38] doorbell_try_core_ipi+0xb8/0xf0
>> > > > Call Trace:
>> > > > [c0000001fffebb70] [c0000001fffebba0] 0xc0000001fffebba0
>> > > > (unreliable)
>> > > > [c0000001fffebba0] [c0000000000b0170]
>> > > > smp_pseries_cause_ipi+0x20/0x70
>> > > > [c0000001fffebbd0] [c00000000004b02c]
>> > > > arch_send_call_function_single_ipi+0x8c/0xa0
>> > > > [c0000001fffebbf0] [c0000000001de600]
>> > > > irq_work_queue_on+0xe0/0x130
>> > > > [c0000001fffebc30] [c0000000001340c8]
>> > > > rto_push_irq_work_func+0xc8/0x120
>> > > 
>> > > …
>> > > > Instruction dump:
>> > > > 60000000 60000000 3c4c00a2 384250b0 3d220009 392949c8 81290000
>> > > > 3929ffff
>> > > > 7d231838 7c0004ac 5463017e 64632800 <7c00191c> 4e800020
>> > > > 3c4c00a2 38425080
>> > > > ---[ end trace eb842b544538cbdf ]---

This is unusual and causing powerpc code to crash because the rt
scheduler is telling irq_work_queue_on to queue work on this CPU.
Is that something allowed? There's no warnings in there but it must
be a rarely tested path, would it be better to ban it?

Steven is this queue_work_on to self by design?

>> > > 
>> > > and I was wondering whether this is a qemu bug or the kernel is
>> > > using an
>> > > opcode it should rather not. If I skip doorbell_try_core_ipi() in
>> > > smp_pseries_cause_ipi() then there is no crash. The comment says
>> > > "POWER9
>> > > should not use this handler" so…
>> > 
>> > I would say Linux is using a msgsndp instruction which is not
>> > implemented
>> > in QEMU TCG. But why have we started using dbells in Linux ? 
> 
> Yeah the kernel must have used msgsndp which isn't implemented for TCG
> yet. We use doorbells in linux but only for threads which are on the
> same core.
> And when I try to construct a situation with more than 1 thread per
> core (e.g. -smp 4,threads=4), I get "TCG cannot support more than 1
> thread/core on a pseries machine".
> 
> So I wonder why the guest thinks it can use msgsndp...

IPI to self evidently. Under TCG it really should implement the
instruction or remove the DBELL feature.

Thanks,
Nick

Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[Steven Rostedt]

[ Added Peter ]

On Fri, 29 Mar 2019 19:13:55 +1000
Nicholas Piggin <npiggin@gmail.com> wrote:

> Suraj Jitindar Singh's on March 29, 2019 3:20 pm:
> > On Wed, 2019-03-27 at 17:51 +0100, Cédric Le Goater wrote:  
> >> On 3/27/19 5:37 PM, Cédric Le Goater wrote:  
> >> > On 3/27/19 1:36 PM, Sebastian Andrzej Siewior wrote:  
> >> > > With qemu-system-ppc64le -machine pseries -smp 4 I get:
> >> > >   
> >> > > > #  chrt 1 hackbench
> >> > > > Running in process mode with 10 groups using 40 file
> >> > > > descriptors each (== 400 tasks)
> >> > > > Each sender will pass 100 messages of 100 bytes
> >> > > > Oops: Exception in kernel mode, sig: 4 [#1]
> >> > > > LE PAGE_SIZE=64K MMU=Hash PREEMPT SMP NR_CPUS=2048 NUMA pSeries
> >> > > > Modules linked in:
> >> > > > CPU: 0 PID: 629 Comm: hackbench Not tainted 5.1.0-rc2 #71
> >> > > > NIP:  c000000000046978 LR: c000000000046a38 CTR:
> >> > > > c0000000000b0150
> >> > > > REGS: c0000001fffeb8e0 TRAP: 0700   Not tainted  (5.1.0-rc2)
> >> > > > MSR:  8000000000089033 <SF,EE,ME,IR,DR,RI,LE>  CR:
> >> > > > 42000874  XER: 00000000
> >> > > > CFAR: c000000000046a34 IRQMASK: 1
> >> > > > GPR00: c0000000000b0170 c0000001fffebb70 c000000000a6ba00
> >> > > > 0000000028000000  
> >> > > 
> >> > > …  
> >> > > > NIP [c000000000046978] doorbell_core_ipi+0x28/0x30
> >> > > > LR [c000000000046a38] doorbell_try_core_ipi+0xb8/0xf0
> >> > > > Call Trace:
> >> > > > [c0000001fffebb70] [c0000001fffebba0] 0xc0000001fffebba0
> >> > > > (unreliable)
> >> > > > [c0000001fffebba0] [c0000000000b0170]
> >> > > > smp_pseries_cause_ipi+0x20/0x70
> >> > > > [c0000001fffebbd0] [c00000000004b02c]
> >> > > > arch_send_call_function_single_ipi+0x8c/0xa0
> >> > > > [c0000001fffebbf0] [c0000000001de600]
> >> > > > irq_work_queue_on+0xe0/0x130
> >> > > > [c0000001fffebc30] [c0000000001340c8]
> >> > > > rto_push_irq_work_func+0xc8/0x120  
> >> > > 
> >> > > …  
> >> > > > Instruction dump:
> >> > > > 60000000 60000000 3c4c00a2 384250b0 3d220009 392949c8 81290000
> >> > > > 3929ffff
> >> > > > 7d231838 7c0004ac 5463017e 64632800 <7c00191c> 4e800020
> >> > > > 3c4c00a2 38425080
> >> > > > ---[ end trace eb842b544538cbdf ]---  
> 
> This is unusual and causing powerpc code to crash because the rt
> scheduler is telling irq_work_queue_on to queue work on this CPU.
> Is that something allowed? There's no warnings in there but it must
> be a rarely tested path, would it be better to ban it?
> 
> Steven is this queue_work_on to self by design?

I just figured that a irq_work_queue_on() would default to just
irq_work_queue() if cpu == smp_processor_id().

I'm sure this case has been tested (on x86), as I stressed this quite a
bit, and all it takes is to have an RT task queued on the CPU
processing the current "push" logic when it's not holding the rq locks
and read that the current CPU now has an overloaded RT task.

If calling irq_work_queue_on() is really bad to do to the same CPU,
then we can work on changing the logic to do something different if the
CPU returned from rto_next_cpu() is the current CPU.

Or would it be possible to just change irq_work_queue_on() to call
irq_work_queue() if cpu == smp_processor_id()?

-- Steve


> 
> >> > > 
> >> > > and I was wondering whether this is a qemu bug or the kernel is
> >> > > using an
> >> > > opcode it should rather not. If I skip doorbell_try_core_ipi() in
> >> > > smp_pseries_cause_ipi() then there is no crash. The comment says
> >> > > "POWER9
> >> > > should not use this handler" so…  
> >> > 
> >> > I would say Linux is using a msgsndp instruction which is not
> >> > implemented
> >> > in QEMU TCG. But why have we started using dbells in Linux ?   
> > 
> > Yeah the kernel must have used msgsndp which isn't implemented for TCG
> > yet. We use doorbells in linux but only for threads which are on the
> > same core.
> > And when I try to construct a situation with more than 1 thread per
> > core (e.g. -smp 4,threads=4), I get "TCG cannot support more than 1
> > thread/core on a pseries machine".
> > 
> > So I wonder why the guest thinks it can use msgsndp...  
> 
> IPI to self evidently. Under TCG it really should implement the
> instruction or remove the DBELL feature.
> 
> Thanks,
> Nick
> 

Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[Nicholas Piggin]

Steven Rostedt's on March 30, 2019 1:31 am:
> [ Added Peter ]
> 
> On Fri, 29 Mar 2019 19:13:55 +1000
> Nicholas Piggin <npiggin@gmail.com> wrote:
> 
>> Suraj Jitindar Singh's on March 29, 2019 3:20 pm:
>> > On Wed, 2019-03-27 at 17:51 +0100, Cédric Le Goater wrote:  
>> >> On 3/27/19 5:37 PM, Cédric Le Goater wrote:  
>> >> > On 3/27/19 1:36 PM, Sebastian Andrzej Siewior wrote:  
>> >> > > With qemu-system-ppc64le -machine pseries -smp 4 I get:
>> >> > >   
>> >> > > > #  chrt 1 hackbench
>> >> > > > Running in process mode with 10 groups using 40 file
>> >> > > > descriptors each (== 400 tasks)
>> >> > > > Each sender will pass 100 messages of 100 bytes
>> >> > > > Oops: Exception in kernel mode, sig: 4 [#1]
>> >> > > > LE PAGE_SIZE=64K MMU=Hash PREEMPT SMP NR_CPUS=2048 NUMA pSeries
>> >> > > > Modules linked in:
>> >> > > > CPU: 0 PID: 629 Comm: hackbench Not tainted 5.1.0-rc2 #71
>> >> > > > NIP:  c000000000046978 LR: c000000000046a38 CTR:
>> >> > > > c0000000000b0150
>> >> > > > REGS: c0000001fffeb8e0 TRAP: 0700   Not tainted  (5.1.0-rc2)
>> >> > > > MSR:  8000000000089033 <SF,EE,ME,IR,DR,RI,LE>  CR:
>> >> > > > 42000874  XER: 00000000
>> >> > > > CFAR: c000000000046a34 IRQMASK: 1
>> >> > > > GPR00: c0000000000b0170 c0000001fffebb70 c000000000a6ba00
>> >> > > > 0000000028000000  
>> >> > > 
>> >> > > …  
>> >> > > > NIP [c000000000046978] doorbell_core_ipi+0x28/0x30
>> >> > > > LR [c000000000046a38] doorbell_try_core_ipi+0xb8/0xf0
>> >> > > > Call Trace:
>> >> > > > [c0000001fffebb70] [c0000001fffebba0] 0xc0000001fffebba0
>> >> > > > (unreliable)
>> >> > > > [c0000001fffebba0] [c0000000000b0170]
>> >> > > > smp_pseries_cause_ipi+0x20/0x70
>> >> > > > [c0000001fffebbd0] [c00000000004b02c]
>> >> > > > arch_send_call_function_single_ipi+0x8c/0xa0
>> >> > > > [c0000001fffebbf0] [c0000000001de600]
>> >> > > > irq_work_queue_on+0xe0/0x130
>> >> > > > [c0000001fffebc30] [c0000000001340c8]
>> >> > > > rto_push_irq_work_func+0xc8/0x120  
>> >> > > 
>> >> > > …  
>> >> > > > Instruction dump:
>> >> > > > 60000000 60000000 3c4c00a2 384250b0 3d220009 392949c8 81290000
>> >> > > > 3929ffff
>> >> > > > 7d231838 7c0004ac 5463017e 64632800 <7c00191c> 4e800020
>> >> > > > 3c4c00a2 38425080
>> >> > > > ---[ end trace eb842b544538cbdf ]---  
>> 
>> This is unusual and causing powerpc code to crash because the rt
>> scheduler is telling irq_work_queue_on to queue work on this CPU.
>> Is that something allowed? There's no warnings in there but it must
>> be a rarely tested path, would it be better to ban it?
>> 
>> Steven is this queue_work_on to self by design?
> 
> I just figured that a irq_work_queue_on() would default to just
> irq_work_queue() if cpu == smp_processor_id().

Yeah it actually doesn't, it raises a CALL_FUNCTION IPI on the
current CPU. Maybe it should do that.

> 
> I'm sure this case has been tested (on x86), as I stressed this quite a
> bit, and all it takes is to have an RT task queued on the CPU
> processing the current "push" logic when it's not holding the rq locks
> and read that the current CPU now has an overloaded RT task.
> 
> If calling irq_work_queue_on() is really bad to do to the same CPU,
> then we can work on changing the logic to do something different if the
> CPU returned from rto_next_cpu() is the current CPU.

I don't actually know that it is bad. It does seem to violate the
principle of least surprise for arch code though.

From this bug report, I'm guessing it's pretty rare case to hit so
I would worry about subtle bugs. And I just cc'ed you in case your
code was actually not intending to do this.

> Or would it be possible to just change irq_work_queue_on() to call
> irq_work_queue() if cpu == smp_processor_id()?

Something like this?

kernel/irq_work: Do not raise an IPI when queueing work on the local CPU

The QEMU powerpc/pseries machine model was not expecting a self-IPI,
and it may be a bit surprising thing to do, so have irq_work_queue_on
do local queueing when target is current CPU.

Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 kernel/irq_work.c | 78 ++++++++++++++++++++++++++---------------------
 1 file changed, 43 insertions(+), 35 deletions(-)

diff --git a/kernel/irq_work.c b/kernel/irq_work.c
index 6b7cdf17ccf8..f0e539d0f879 100644
--- a/kernel/irq_work.c
+++ b/kernel/irq_work.c
@@ -56,61 +56,69 @@ void __weak arch_irq_work_raise(void)
 	 */
 }
 
-/*
- * Enqueue the irq_work @work on @cpu unless it's already pending
- * somewhere.
- *
- * Can be re-enqueued while the callback is still in progress.
- */
-bool irq_work_queue_on(struct irq_work *work, int cpu)
+/* Enqueue on current CPU, work must already be claimed and preempt disabled */
+static void __irq_work_queue(struct irq_work *work)
 {
-	/* All work should have been flushed before going offline */
-	WARN_ON_ONCE(cpu_is_offline(cpu));
-
-#ifdef CONFIG_SMP
-
-	/* Arch remote IPI send/receive backend aren't NMI safe */
-	WARN_ON_ONCE(in_nmi());
+	/* If the work is "lazy", handle it from next tick if any */
+	if (work->flags & IRQ_WORK_LAZY) {
+		if (llist_add(&work->llnode, this_cpu_ptr(&lazy_list)) &&
+		    tick_nohz_tick_stopped())
+			arch_irq_work_raise();
+	} else {
+		if (llist_add(&work->llnode, this_cpu_ptr(&raised_list)))
+			arch_irq_work_raise();
+	}
+}
 
+/* Enqueue the irq work @work on the current CPU */
+bool irq_work_queue(struct irq_work *work)
+{
 	/* Only queue if not already pending */
 	if (!irq_work_claim(work))
 		return false;
 
-	if (llist_add(&work->llnode, &per_cpu(raised_list, cpu)))
-		arch_send_call_function_single_ipi(cpu);
-
-#else /* #ifdef CONFIG_SMP */
-	irq_work_queue(work);
-#endif /* #else #ifdef CONFIG_SMP */
+	/* Queue the entry and raise the IPI if needed. */
+	preempt_disable();
+	__irq_work_queue(work);
+	preempt_enable();
 
 	return true;
 }
+EXPORT_SYMBOL_GPL(irq_work_queue);
 
-/* Enqueue the irq work @work on the current CPU */
-bool irq_work_queue(struct irq_work *work)
+/*
+ * Enqueue the irq_work @work on @cpu unless it's already pending
+ * somewhere.
+ *
+ * Can be re-enqueued while the callback is still in progress.
+ */
+bool irq_work_queue_on(struct irq_work *work, int cpu)
 {
+#ifndef CONFIG_SMP
+	return irq_work_queue(work);
+
+#else /* #ifndef CONFIG_SMP */
+	/* All work should have been flushed before going offline */
+	WARN_ON_ONCE(cpu_is_offline(cpu));
+
 	/* Only queue if not already pending */
 	if (!irq_work_claim(work))
 		return false;
 
-	/* Queue the entry and raise the IPI if needed. */
 	preempt_disable();
-
-	/* If the work is "lazy", handle it from next tick if any */
-	if (work->flags & IRQ_WORK_LAZY) {
-		if (llist_add(&work->llnode, this_cpu_ptr(&lazy_list)) &&
-		    tick_nohz_tick_stopped())
-			arch_irq_work_raise();
-	} else {
-		if (llist_add(&work->llnode, this_cpu_ptr(&raised_list)))
-			arch_irq_work_raise();
-	}
-
+	if (cpu != smp_processor_id()) {
+		/* Arch remote IPI send/receive backend aren't NMI safe */
+		WARN_ON_ONCE(in_nmi());
+		if (llist_add(&work->llnode, &per_cpu(raised_list, cpu)))
+			arch_send_call_function_single_ipi(cpu);
+	} else
+		__irq_work_queue(work);
 	preempt_enable();
 
 	return true;
+#endif /* #else #ifndef CONFIG_SMP */
 }
-EXPORT_SYMBOL_GPL(irq_work_queue);
+
 
 bool irq_work_needs_cpu(void)
 {
-- 
2.20.1

Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[Peter Zijlstra]

+ fweisbec, who did the remote bits

On Sat, Mar 30, 2019 at 01:10:28PM +1000, Nicholas Piggin wrote:
> Something like this?
> 
> kernel/irq_work: Do not raise an IPI when queueing work on the local CPU
> 
> The QEMU powerpc/pseries machine model was not expecting a self-IPI,
> and it may be a bit surprising thing to do, so have irq_work_queue_on
> do local queueing when target is current CPU.

This seems OK to me.

> Suggested-by: Steven Rostedt <rostedt@goodmis.org>
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>

Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>

> ---
>  kernel/irq_work.c | 78 ++++++++++++++++++++++++++---------------------
>  1 file changed, 43 insertions(+), 35 deletions(-)
> 
> diff --git a/kernel/irq_work.c b/kernel/irq_work.c
> index 6b7cdf17ccf8..f0e539d0f879 100644
> --- a/kernel/irq_work.c
> +++ b/kernel/irq_work.c
> @@ -56,61 +56,69 @@ void __weak arch_irq_work_raise(void)
>  	 */
>  }
>  
> -/*
> - * Enqueue the irq_work @work on @cpu unless it's already pending
> - * somewhere.
> - *
> - * Can be re-enqueued while the callback is still in progress.
> - */
> -bool irq_work_queue_on(struct irq_work *work, int cpu)
> +/* Enqueue on current CPU, work must already be claimed and preempt disabled */
> +static void __irq_work_queue(struct irq_work *work)
>  {
> -	/* All work should have been flushed before going offline */
> -	WARN_ON_ONCE(cpu_is_offline(cpu));
> -
> -#ifdef CONFIG_SMP
> -
> -	/* Arch remote IPI send/receive backend aren't NMI safe */
> -	WARN_ON_ONCE(in_nmi());
> +	/* If the work is "lazy", handle it from next tick if any */
> +	if (work->flags & IRQ_WORK_LAZY) {
> +		if (llist_add(&work->llnode, this_cpu_ptr(&lazy_list)) &&
> +		    tick_nohz_tick_stopped())
> +			arch_irq_work_raise();
> +	} else {
> +		if (llist_add(&work->llnode, this_cpu_ptr(&raised_list)))
> +			arch_irq_work_raise();
> +	}
> +}
>  
> +/* Enqueue the irq work @work on the current CPU */
> +bool irq_work_queue(struct irq_work *work)
> +{
>  	/* Only queue if not already pending */
>  	if (!irq_work_claim(work))
>  		return false;
>  
> -	if (llist_add(&work->llnode, &per_cpu(raised_list, cpu)))
> -		arch_send_call_function_single_ipi(cpu);
> -
> -#else /* #ifdef CONFIG_SMP */
> -	irq_work_queue(work);
> -#endif /* #else #ifdef CONFIG_SMP */
> +	/* Queue the entry and raise the IPI if needed. */
> +	preempt_disable();
> +	__irq_work_queue(work);
> +	preempt_enable();
>  
>  	return true;
>  }
> +EXPORT_SYMBOL_GPL(irq_work_queue);
>  
> -/* Enqueue the irq work @work on the current CPU */
> -bool irq_work_queue(struct irq_work *work)
> +/*
> + * Enqueue the irq_work @work on @cpu unless it's already pending
> + * somewhere.
> + *
> + * Can be re-enqueued while the callback is still in progress.
> + */
> +bool irq_work_queue_on(struct irq_work *work, int cpu)
>  {
> +#ifndef CONFIG_SMP
> +	return irq_work_queue(work);
> +
> +#else /* #ifndef CONFIG_SMP */
> +	/* All work should have been flushed before going offline */
> +	WARN_ON_ONCE(cpu_is_offline(cpu));
> +
>  	/* Only queue if not already pending */
>  	if (!irq_work_claim(work))
>  		return false;
>  
> -	/* Queue the entry and raise the IPI if needed. */
>  	preempt_disable();
> -
> -	/* If the work is "lazy", handle it from next tick if any */
> -	if (work->flags & IRQ_WORK_LAZY) {
> -		if (llist_add(&work->llnode, this_cpu_ptr(&lazy_list)) &&
> -		    tick_nohz_tick_stopped())
> -			arch_irq_work_raise();
> -	} else {
> -		if (llist_add(&work->llnode, this_cpu_ptr(&raised_list)))
> -			arch_irq_work_raise();
> -	}
> -
> +	if (cpu != smp_processor_id()) {
> +		/* Arch remote IPI send/receive backend aren't NMI safe */
> +		WARN_ON_ONCE(in_nmi());
> +		if (llist_add(&work->llnode, &per_cpu(raised_list, cpu)))
> +			arch_send_call_function_single_ipi(cpu);
> +	} else
> +		__irq_work_queue(work);
>  	preempt_enable();
>  
>  	return true;
> +#endif /* #else #ifndef CONFIG_SMP */
>  }
> -EXPORT_SYMBOL_GPL(irq_work_queue);
> +
>  
>  bool irq_work_needs_cpu(void)
>  {
> -- 
> 2.20.1
> 

Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[Nicholas Piggin]

Peter Zijlstra's on April 1, 2019 6:38 pm:
> 
> + fweisbec, who did the remote bits
> 
> On Sat, Mar 30, 2019 at 01:10:28PM +1000, Nicholas Piggin wrote:
>> Something like this?
>> 
>> kernel/irq_work: Do not raise an IPI when queueing work on the local CPU
>> 
>> The QEMU powerpc/pseries machine model was not expecting a self-IPI,
>> and it may be a bit surprising thing to do, so have irq_work_queue_on
>> do local queueing when target is current CPU.
> 
> This seems OK to me.
> 
>> Suggested-by: Steven Rostedt <rostedt@goodmis.org>
>> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
> 
> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>

Thanks for taking a look.

Sebastian, are you able to test if this patch solves your problem?

Thanks,
Nick

Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[Sebastian Andrzej Siewior]

On 2019-04-05 02:25:44 [+1000], Nicholas Piggin wrote:
> Sebastian, are you able to test if this patch solves your problem?

yes, it does.

Tested-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>

> Thanks,
> Nick

Sebastian

Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[Frederic Weisbecker]

On Mon, Apr 01, 2019 at 10:38:27AM +0200, Peter Zijlstra wrote:
> 
> + fweisbec, who did the remote bits
> 
> On Sat, Mar 30, 2019 at 01:10:28PM +1000, Nicholas Piggin wrote:
> > diff --git a/kernel/irq_work.c b/kernel/irq_work.c
> > index 6b7cdf17ccf8..f0e539d0f879 100644
> > --- a/kernel/irq_work.c
> > +++ b/kernel/irq_work.c
> > -/* Enqueue the irq work @work on the current CPU */
> > -bool irq_work_queue(struct irq_work *work)
> > +/*
> > + * Enqueue the irq_work @work on @cpu unless it's already pending
> > + * somewhere.
> > + *
> > + * Can be re-enqueued while the callback is still in progress.
> > + */
> > +bool irq_work_queue_on(struct irq_work *work, int cpu)
> >  {
> > +#ifndef CONFIG_SMP
> > +	return irq_work_queue(work);
> > +

I'd suggest to use "if (!IS_ENABLED(CONFIG_SMP))" here to avoid the large
ifdeffery.

> > +#else /* #ifndef CONFIG_SMP */
> > +	/* All work should have been flushed before going offline */
> > +	WARN_ON_ONCE(cpu_is_offline(cpu));
> > +
> >  	/* Only queue if not already pending */
> >  	if (!irq_work_claim(work))
> >  		return false;
> >  
> > -	/* Queue the entry and raise the IPI if needed. */
> >  	preempt_disable();
> > -
> > -	/* If the work is "lazy", handle it from next tick if any */
> > -	if (work->flags & IRQ_WORK_LAZY) {
> > -		if (llist_add(&work->llnode, this_cpu_ptr(&lazy_list)) &&
> > -		    tick_nohz_tick_stopped())
> > -			arch_irq_work_raise();
> > -	} else {
> > -		if (llist_add(&work->llnode, this_cpu_ptr(&raised_list)))
> > -			arch_irq_work_raise();
> > -	}
> > -
> > +	if (cpu != smp_processor_id()) {
> > +		/* Arch remote IPI send/receive backend aren't NMI safe */
> > +		WARN_ON_ONCE(in_nmi());
> > +		if (llist_add(&work->llnode, &per_cpu(raised_list, cpu)))
> > +			arch_send_call_function_single_ipi(cpu);
> > +	} else
> > +		__irq_work_queue(work);

Also perhaps rename __irq_work_queue() to irq_work_queue_local() to make it
instantly clearer to reviewers.

Other than those cosmetic changes,

  Reviewed-by: Frederic Weisbecker <frederic@kernel.org>

Thanks.

Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[Nicholas Piggin]

Frederic Weisbecker's on April 6, 2019 10:06 am:
> On Mon, Apr 01, 2019 at 10:38:27AM +0200, Peter Zijlstra wrote:
>> 
>> + fweisbec, who did the remote bits
>> 
>> On Sat, Mar 30, 2019 at 01:10:28PM +1000, Nicholas Piggin wrote:
>> > diff --git a/kernel/irq_work.c b/kernel/irq_work.c
>> > index 6b7cdf17ccf8..f0e539d0f879 100644
>> > --- a/kernel/irq_work.c
>> > +++ b/kernel/irq_work.c
>> > -/* Enqueue the irq work @work on the current CPU */
>> > -bool irq_work_queue(struct irq_work *work)
>> > +/*
>> > + * Enqueue the irq_work @work on @cpu unless it's already pending
>> > + * somewhere.
>> > + *
>> > + * Can be re-enqueued while the callback is still in progress.
>> > + */
>> > +bool irq_work_queue_on(struct irq_work *work, int cpu)
>> >  {
>> > +#ifndef CONFIG_SMP
>> > +	return irq_work_queue(work);
>> > +
> 
> I'd suggest to use "if (!IS_ENABLED(CONFIG_SMP))" here to avoid the large
> ifdeffery.

Sadly you can't do it because arch_send_call_function_single_ipi is
not defined for !SMP. I made your suggested name change though (with
the __ prefix because work needs to be claimed and preempt disabled).

Thanks,
Nick

Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[Jason A. Donenfeld]

Hi folks,

I'm actually still experiencing this sporadically in the WireGuard test 
suite, which you can see being run on https://build.wireguard.com/ . 
About 50% of the time the powerpc64 build will fail at a place like this:

[   65.147823] Oops: Exception in kernel mode, sig: 4 [#1]
[   65.149198] LE PAGE_SIZE=4K MMU=Hash PREEMPT SMP NR_CPUS=4 NUMA pSeries
[   65.149595] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.5.0-rc1+ #1
[   65.149745] NIP:  c000000000033330 LR: c00000000007eda0 CTR: 
c00000000007ed80
[   65.149934] REGS: c000000000a47970 TRAP: 0700   Not tainted  (5.5.0-rc1+)
[   65.150032] MSR:  800000000288b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> 
CR: 48008288  XER: 00000000
[   65.150352] CFAR: c0000000000332bc IRQMASK: 1
[   65.150352] GPR00: c000000000036508 c000000000a47c00 c000000000a4c100 
0000000000000001
[   65.150352] GPR04: c000000000a50998 0000000000000000 c000000000a50908 
000000000f509000
[   65.150352] GPR08: 0000000028000000 0000000000000000 0000000000000000 
c00000000ff24f00
[   65.150352] GPR12: c00000000007ed80 c000000000ad9000 0000000000000000 
0000000000000000
[   65.150352] GPR16: 00000000008c9190 00000000008c94a8 00000000008c92f8 
00000000008c98b0
[   65.150352] GPR20: 00000000008f2f88 fffffffffffffffd 0000000000000014 
0000000000e6c100
[   65.150352] GPR24: 0000000000e6c100 0000000000000001 0000000000000000 
c000000000a50998
[   65.150352] GPR28: c000000000a9e280 c000000000a50aa4 0000000000000002 
0000000000000000
[   65.151591] NIP [c000000000033330] doorbell_try_core_ipi+0xd0/0xf0
[   65.151750] LR [c00000000007eda0] smp_pseries_cause_ipi+0x20/0x70
[   65.151913] Call Trace:
[   65.152109] [c000000000a47c00] [c0000000000c7c9c] 
_nohz_idle_balance+0xbc/0x300 (unreliable)
[   65.152370] [c000000000a47c30] [c000000000036508] 
smp_send_reschedule+0x98/0xb0
[   65.152711] [c000000000a47c50] [c0000000000c1634] kick_ilb+0x114/0x140
[   65.152962] [c000000000a47ca0] [c0000000000c86d8] 
newidle_balance+0x4e8/0x500
[   65.153213] [c000000000a47d20] [c0000000000c8788] 
pick_next_task_fair+0x48/0x3a0
[   65.153424] [c000000000a47d80] [c000000000466620] __schedule+0xf0/0x430
[   65.153612] [c000000000a47de0] [c000000000466b04] schedule_idle+0x34/0x70
[   65.153786] [c000000000a47e10] [c0000000000c0bc8] do_idle+0x1a8/0x220
[   65.154121] [c000000000a47e70] [c0000000000c0e94] 
cpu_startup_entry+0x34/0x40
[   65.154313] [c000000000a47ea0] [c00000000000ef1c] rest_init+0x10c/0x124
[   65.154414] [c000000000a47ee0] [c000000000500004] 
start_kernel+0x568/0x594
[   65.154585] [c000000000a47f90] [c00000000000a7cc] 
start_here_common+0x1c/0x330
[   65.154854] Instruction dump:
[   65.155191] 38210030 e8010010 7c0803a6 4e800020 3d220004 39295228 
81290000 3929ffff
[   65.155498] 7d284038 7c0004ac 5508017e 65082800 <7c00411c> e94d0178 
812a0000 3929ffff
[   65.156155] ---[ end trace 6180d12e268ffdaf ]---
[   65.185452]
[   66.187490] Kernel panic - not syncing: Fatal exception

This is with "qemu-system-ppc64 -smp 4 -machine pseries" on QEMU 4.0.0.

I'm not totally sure what's going on here. I'm emulating a pseries, and 
using that with qemu's pseries model, and I see that selecting the 
pseries forces the selection of 'config PPC_DOORBELL' (twice in the same 
section, actually). Then inside the kernel there appears to be some 
runtime CPU check for doorbell support. Is this a case in which QEMU is 
advertising doorbell support that TCG doesn't have? Or is something else 
happening here?

Thanks,
Jason

Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[Sebastian Andrzej Siewior]

On 2019-12-19 11:41:21 [+0100], Jason A. Donenfeld wrote:
> Hi folks,
Hi,

so this should duct tape it:

diff --git a/arch/powerpc/kernel/dbell.c b/arch/powerpc/kernel/dbell.c
index f17ff1200eaae..ec044bdf362a1 100644
--- a/arch/powerpc/kernel/dbell.c
+++ b/arch/powerpc/kernel/dbell.c
@@ -60,16 +60,8 @@ void doorbell_core_ipi(int cpu)
  */
 int doorbell_try_core_ipi(int cpu)
 {
-	int this_cpu = get_cpu();
 	int ret = 0;
 
-	if (cpumask_test_cpu(cpu, cpu_sibling_mask(this_cpu))) {
-		doorbell_core_ipi(cpu);
-		ret = 1;
-	}
-
-	put_cpu();
-
 	return ret;
 }
 

> This is with "qemu-system-ppc64 -smp 4 -machine pseries" on QEMU 4.0.0.

Interesting. I didn't get v5.4 to boot a while ago and didn't have the
time to look into it.

> I'm not totally sure what's going on here. I'm emulating a pseries, and
> using that with qemu's pseries model, and I see that selecting the pseries
> forces the selection of 'config PPC_DOORBELL' (twice in the same section,
> actually). Then inside the kernel there appears to be some runtime CPU check
> for doorbell support. Is this a case in which QEMU is advertising doorbell
> support that TCG doesn't have? Or is something else happening here?

Based on my understanding is that the doorbell feature is part of the
architecture. It can be used to signal other siblings on the same CPU.
qemu TCG doesn't support that and does not allow to announce multiple
siblings on the same CPU. However, the kernel uses this interface if it
tries to send an interrupt to itself (the same CPU) so everything
matches.
Last time I run into this, the interface was change so the kernel das
not send an IPI to itself. This changed now for another function…

> Thanks,
> Jason

Sebastian

Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[Jason A. Donenfeld]

On Thu, Dec 19, 2019 at 12:13 PM Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
> Based on my understanding is that the doorbell feature is part of the
> architecture. It can be used to signal other siblings on the same CPU.
> qemu TCG doesn't support that and does not allow to announce multiple
> siblings on the same CPU. However, the kernel uses this interface if it
> tries to send an interrupt to itself (the same CPU) so everything
> matches.
> Last time I run into this, the interface was change so the kernel das
> not send an IPI to itself. This changed now for another function…

One way of fixing this is to just "not use the feature", as you seem
to be suggesting.

But actually shouldn't there be some CPU feature detection available?
Something like -- QEMU advertises to the kernel that it supports or
doesn't support doorbells, and the kernel then avoids those paths if
the CPU feature flag isn't present?

Jason

Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[Michael Ellerman]

"Jason A. Donenfeld" <Jason@zx2c4.com> writes:
> Hi folks,
>
> I'm actually still experiencing this sporadically in the WireGuard test 
> suite, which you can see being run on https://build.wireguard.com/ . 

Fancy dashboard you got there :)

> About 50% of the time the powerpc64 build will fail at a place like this:
>
> [   65.147823] Oops: Exception in kernel mode, sig: 4 [#1]
> [   65.149198] LE PAGE_SIZE=4K MMU=Hash PREEMPT SMP NR_CPUS=4 NUMA pSeries
> [   65.149595] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.5.0-rc1+ #1
> [   65.149745] NIP:  c000000000033330 LR: c00000000007eda0 CTR: c00000000007ed80
> [   65.149934] REGS: c000000000a47970 TRAP: 0700   Not tainted  (5.5.0-rc1+)
> [   65.150032] MSR:  800000000288b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> > CR: 48008288  XER: 00000000
> [   65.150352] CFAR: c0000000000332bc IRQMASK: 1
> [   65.150352] GPR00: c000000000036508 c000000000a47c00 c000000000a4c100 0000000000000001
> [   65.150352] GPR04: c000000000a50998 0000000000000000 c000000000a50908 000000000f509000
> [   65.150352] GPR08: 0000000028000000 0000000000000000 0000000000000000 c00000000ff24f00
> [   65.150352] GPR12: c00000000007ed80 c000000000ad9000 0000000000000000 0000000000000000
> [   65.150352] GPR16: 00000000008c9190 00000000008c94a8 00000000008c92f8 00000000008c98b0
> [   65.150352] GPR20: 00000000008f2f88 fffffffffffffffd 0000000000000014 0000000000e6c100
> [   65.150352] GPR24: 0000000000e6c100 0000000000000001 0000000000000000 c000000000a50998
> [   65.150352] GPR28: c000000000a9e280 c000000000a50aa4 0000000000000002 0000000000000000
> [   65.151591] NIP [c000000000033330] doorbell_try_core_ipi+0xd0/0xf0
> [   65.151750] LR [c00000000007eda0] smp_pseries_cause_ipi+0x20/0x70
> [   65.151913] Call Trace:
> [   65.152109] [c000000000a47c00] [c0000000000c7c9c] _nohz_idle_balance+0xbc/0x300 (unreliable)
> [   65.152370] [c000000000a47c30] [c000000000036508] smp_send_reschedule+0x98/0xb0
> [   65.152711] [c000000000a47c50] [c0000000000c1634] kick_ilb+0x114/0x140
> [   65.152962] [c000000000a47ca0] [c0000000000c86d8] newidle_balance+0x4e8/0x500
> [   65.153213] [c000000000a47d20] [c0000000000c8788] pick_next_task_fair+0x48/0x3a0
> [   65.153424] [c000000000a47d80] [c000000000466620] __schedule+0xf0/0x430
> [   65.153612] [c000000000a47de0] [c000000000466b04] schedule_idle+0x34/0x70
> [   65.153786] [c000000000a47e10] [c0000000000c0bc8] do_idle+0x1a8/0x220
> [   65.154121] [c000000000a47e70] [c0000000000c0e94] cpu_startup_entry+0x34/0x40
> [   65.154313] [c000000000a47ea0] [c00000000000ef1c] rest_init+0x10c/0x124
> [   65.154414] [c000000000a47ee0] [c000000000500004] start_kernel+0x568/0x594
> [   65.154585] [c000000000a47f90] [c00000000000a7cc] start_here_common+0x1c/0x330
> [   65.154854] Instruction dump:
> [   65.155191] 38210030 e8010010 7c0803a6 4e800020 3d220004 39295228 81290000 3929ffff
> [   65.155498] 7d284038 7c0004ac 5508017e 65082800 <7c00411c> e94d0178 812a0000 3929ffff
                                                      ^
Again the faulting instruction there is "msgsndp r8"

> [   65.156155] ---[ end trace 6180d12e268ffdaf ]---
> [   65.185452]
> [   66.187490] Kernel panic - not syncing: Fatal exception
>
> This is with "qemu-system-ppc64 -smp 4 -machine pseries" on QEMU 4.0.0.
>
> I'm not totally sure what's going on here. I'm emulating a pseries, and 
> using that with qemu's pseries model, and I see that selecting the 
> pseries forces the selection of 'config PPC_DOORBELL' (twice in the same 
> section, actually).

Noted.

> Then inside the kernel there appears to be some runtime CPU check for
> doorbell support.

Not really. The kernel looks at the CPU revision (PVR) and decides that
it has doorbell support.

> Is this a case in which QEMU is advertising doorbell support that TCG
> doesn't have? Or is something else happening here?

It's a gap in the emulation I guess. qemu doesn't emulate msgsndp, but
it really should because that's a supported instruction since Power8.

I suspect msgsndp wasn't implemented for TCG because TCG doesn't support
more than one thread per core, and you can only send doorbells to other
threads in the same core, and therefore there is no reason to ever use
msgsndp.

That's the message Suraj mentioned up thread, eg:

  $ qemu-system-ppc64 -nographic -vga none -M pseries -smp 2,threads=2 -cpu POWER8 -kernel build~/vmlinux
  qemu-system-ppc64: TCG cannot support more than 1 thread/core on a pseries machine


But I guess we've hit another case of a CPU sending itself an IPI, and
the way the sibling masks are done, CPUs are siblings of themselves, so
the sibling test passes, eg:

int doorbell_try_core_ipi(int cpu)
{
	int this_cpu = get_cpu();
	int ret = 0;

	if (cpumask_test_cpu(cpu, cpu_sibling_mask(this_cpu))) {
		doorbell_core_ipi(cpu);



In which case this patch should fix it.

diff --git a/arch/powerpc/kernel/dbell.c b/arch/powerpc/kernel/dbell.c
index f17ff1200eaa..e45cb9bba193 100644
--- a/arch/powerpc/kernel/dbell.c
+++ b/arch/powerpc/kernel/dbell.c
@@ -63,7 +63,7 @@ int doorbell_try_core_ipi(int cpu)
 	int this_cpu = get_cpu();
 	int ret = 0;
 
-	if (cpumask_test_cpu(cpu, cpu_sibling_mask(this_cpu))) {
+	if (cpu != this_cpu && cpumask_test_cpu(cpu, cpu_sibling_mask(this_cpu))) {
 		doorbell_core_ipi(cpu);
 		ret = 1;
 	}


The other option would be we disable CPU_FTR_DBELL if we detect we're
running under TCG. But I'm not sure we have a particularly clean way to
detect that.

cheers

Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[Cédric Le Goater]

On 19/12/2019 13:45, Michael Ellerman wrote:
> "Jason A. Donenfeld" <Jason@zx2c4.com> writes:
>> Hi folks,
>>
>> I'm actually still experiencing this sporadically in the WireGuard test 
>> suite, which you can see being run on https://build.wireguard.com/ . 
> 
> Fancy dashboard you got there :)
> 
>> About 50% of the time the powerpc64 build will fail at a place like this:
>>
>> [   65.147823] Oops: Exception in kernel mode, sig: 4 [#1]
>> [   65.149198] LE PAGE_SIZE=4K MMU=Hash PREEMPT SMP NR_CPUS=4 NUMA pSeries
>> [   65.149595] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.5.0-rc1+ #1
>> [   65.149745] NIP:  c000000000033330 LR: c00000000007eda0 CTR: c00000000007ed80
>> [   65.149934] REGS: c000000000a47970 TRAP: 0700   Not tainted  (5.5.0-rc1+)
>> [   65.150032] MSR:  800000000288b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> > CR: 48008288  XER: 00000000
>> [   65.150352] CFAR: c0000000000332bc IRQMASK: 1
>> [   65.150352] GPR00: c000000000036508 c000000000a47c00 c000000000a4c100 0000000000000001
>> [   65.150352] GPR04: c000000000a50998 0000000000000000 c000000000a50908 000000000f509000
>> [   65.150352] GPR08: 0000000028000000 0000000000000000 0000000000000000 c00000000ff24f00
>> [   65.150352] GPR12: c00000000007ed80 c000000000ad9000 0000000000000000 0000000000000000
>> [   65.150352] GPR16: 00000000008c9190 00000000008c94a8 00000000008c92f8 00000000008c98b0
>> [   65.150352] GPR20: 00000000008f2f88 fffffffffffffffd 0000000000000014 0000000000e6c100
>> [   65.150352] GPR24: 0000000000e6c100 0000000000000001 0000000000000000 c000000000a50998
>> [   65.150352] GPR28: c000000000a9e280 c000000000a50aa4 0000000000000002 0000000000000000
>> [   65.151591] NIP [c000000000033330] doorbell_try_core_ipi+0xd0/0xf0
>> [   65.151750] LR [c00000000007eda0] smp_pseries_cause_ipi+0x20/0x70
>> [   65.151913] Call Trace:
>> [   65.152109] [c000000000a47c00] [c0000000000c7c9c] _nohz_idle_balance+0xbc/0x300 (unreliable)
>> [   65.152370] [c000000000a47c30] [c000000000036508] smp_send_reschedule+0x98/0xb0
>> [   65.152711] [c000000000a47c50] [c0000000000c1634] kick_ilb+0x114/0x140
>> [   65.152962] [c000000000a47ca0] [c0000000000c86d8] newidle_balance+0x4e8/0x500
>> [   65.153213] [c000000000a47d20] [c0000000000c8788] pick_next_task_fair+0x48/0x3a0
>> [   65.153424] [c000000000a47d80] [c000000000466620] __schedule+0xf0/0x430
>> [   65.153612] [c000000000a47de0] [c000000000466b04] schedule_idle+0x34/0x70
>> [   65.153786] [c000000000a47e10] [c0000000000c0bc8] do_idle+0x1a8/0x220
>> [   65.154121] [c000000000a47e70] [c0000000000c0e94] cpu_startup_entry+0x34/0x40
>> [   65.154313] [c000000000a47ea0] [c00000000000ef1c] rest_init+0x10c/0x124
>> [   65.154414] [c000000000a47ee0] [c000000000500004] start_kernel+0x568/0x594
>> [   65.154585] [c000000000a47f90] [c00000000000a7cc] start_here_common+0x1c/0x330
>> [   65.154854] Instruction dump:
>> [   65.155191] 38210030 e8010010 7c0803a6 4e800020 3d220004 39295228 81290000 3929ffff
>> [   65.155498] 7d284038 7c0004ac 5508017e 65082800 <7c00411c> e94d0178 812a0000 3929ffff
>                                                       ^
> Again the faulting instruction there is "msgsndp r8"
> 
>> [   65.156155] ---[ end trace 6180d12e268ffdaf ]---
>> [   65.185452]
>> [   66.187490] Kernel panic - not syncing: Fatal exception
>>
>> This is with "qemu-system-ppc64 -smp 4 -machine pseries" on QEMU 4.0.0.
>>
>> I'm not totally sure what's going on here. I'm emulating a pseries, and 
>> using that with qemu's pseries model, and I see that selecting the 
>> pseries forces the selection of 'config PPC_DOORBELL' (twice in the same 
>> section, actually).
> 
> Noted.
> 
>> Then inside the kernel there appears to be some runtime CPU check for
>> doorbell support.
> 
> Not really. The kernel looks at the CPU revision (PVR) and decides that
> it has doorbell support.
> 
>> Is this a case in which QEMU is advertising doorbell support that TCG
>> doesn't have? Or is something else happening here?
> 
> It's a gap in the emulation I guess. qemu doesn't emulate msgsndp, but
> it really should because that's a supported instruction since Power8.

There is a patch for msgsndp in my tree you could try : 

  https://github.com/legoater/qemu/tree/powernv-5.0

Currently being reviewed. I have to address some remarks from David before
it can be merged.

> I suspect msgsndp wasn't implemented for TCG because TCG doesn't support
> more than one thread per core, and you can only send doorbells to other
> threads in the same core, and therefore there is no reason to ever use
> msgsndp.

There is a need now with KVM emulation under TCG, but, yes, QEMU still lacks
SMT support.

> That's the message Suraj mentioned up thread, eg:
> 
>   $ qemu-system-ppc64 -nographic -vga none -M pseries -smp 2,threads=2 -cpu POWER8 -kernel build~/vmlinux
>   qemu-system-ppc64: TCG cannot support more than 1 thread/core on a pseries machine
> 
> 
> But I guess we've hit another case of a CPU sending itself an IPI, and
> the way the sibling masks are done, CPUs are siblings of themselves, so
> the sibling test passes, eg:
> 
> int doorbell_try_core_ipi(int cpu)
> {
> 	int this_cpu = get_cpu();
> 	int ret = 0;
> 
> 	if (cpumask_test_cpu(cpu, cpu_sibling_mask(this_cpu))) {
> 		doorbell_core_ipi(cpu);
> 
> 
> 
> In which case this patch should fix it.
> 
> diff --git a/arch/powerpc/kernel/dbell.c b/arch/powerpc/kernel/dbell.c
> index f17ff1200eaa..e45cb9bba193 100644
> --- a/arch/powerpc/kernel/dbell.c
> +++ b/arch/powerpc/kernel/dbell.c
> @@ -63,7 +63,7 @@ int doorbell_try_core_ipi(int cpu)
>  	int this_cpu = get_cpu();
>  	int ret = 0;
>  
> -	if (cpumask_test_cpu(cpu, cpu_sibling_mask(this_cpu))) {
> +	if (cpu != this_cpu && cpumask_test_cpu(cpu, cpu_sibling_mask(this_cpu))) {
>  		doorbell_core_ipi(cpu);
>  		ret = 1;
>  	}
> 
> 
> The other option would be we disable CPU_FTR_DBELL if we detect we're
> running under TCG. But I'm not sure we have a particularly clean way to
> detect that.

does the pseries kernel support cpufeatures in the DT ?

Cheers,

C.

Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[David? Gibson]

[-- Attachment #1: Type: text/plain, Size: 6776 bytes --]

On Thu, Dec 19, 2019 at 02:08:29PM +0100, Cédric Le Goater wrote:
> On 19/12/2019 13:45, Michael Ellerman wrote:
> > "Jason A. Donenfeld" <Jason@zx2c4.com> writes:
> >> Hi folks,
> >>
> >> I'm actually still experiencing this sporadically in the WireGuard test 
> >> suite, which you can see being run on https://build.wireguard.com/ . 
> > 
> > Fancy dashboard you got there :)
> > 
> >> About 50% of the time the powerpc64 build will fail at a place like this:
> >>
> >> [   65.147823] Oops: Exception in kernel mode, sig: 4 [#1]
> >> [   65.149198] LE PAGE_SIZE=4K MMU=Hash PREEMPT SMP NR_CPUS=4 NUMA pSeries
> >> [   65.149595] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.5.0-rc1+ #1
> >> [   65.149745] NIP:  c000000000033330 LR: c00000000007eda0 CTR: c00000000007ed80
> >> [   65.149934] REGS: c000000000a47970 TRAP: 0700   Not tainted  (5.5.0-rc1+)
> >> [   65.150032] MSR:  800000000288b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> > CR: 48008288  XER: 00000000
> >> [   65.150352] CFAR: c0000000000332bc IRQMASK: 1
> >> [   65.150352] GPR00: c000000000036508 c000000000a47c00 c000000000a4c100 0000000000000001
> >> [   65.150352] GPR04: c000000000a50998 0000000000000000 c000000000a50908 000000000f509000
> >> [   65.150352] GPR08: 0000000028000000 0000000000000000 0000000000000000 c00000000ff24f00
> >> [   65.150352] GPR12: c00000000007ed80 c000000000ad9000 0000000000000000 0000000000000000
> >> [   65.150352] GPR16: 00000000008c9190 00000000008c94a8 00000000008c92f8 00000000008c98b0
> >> [   65.150352] GPR20: 00000000008f2f88 fffffffffffffffd 0000000000000014 0000000000e6c100
> >> [   65.150352] GPR24: 0000000000e6c100 0000000000000001 0000000000000000 c000000000a50998
> >> [   65.150352] GPR28: c000000000a9e280 c000000000a50aa4 0000000000000002 0000000000000000
> >> [   65.151591] NIP [c000000000033330] doorbell_try_core_ipi+0xd0/0xf0
> >> [   65.151750] LR [c00000000007eda0] smp_pseries_cause_ipi+0x20/0x70
> >> [   65.151913] Call Trace:
> >> [   65.152109] [c000000000a47c00] [c0000000000c7c9c] _nohz_idle_balance+0xbc/0x300 (unreliable)
> >> [   65.152370] [c000000000a47c30] [c000000000036508] smp_send_reschedule+0x98/0xb0
> >> [   65.152711] [c000000000a47c50] [c0000000000c1634] kick_ilb+0x114/0x140
> >> [   65.152962] [c000000000a47ca0] [c0000000000c86d8] newidle_balance+0x4e8/0x500
> >> [   65.153213] [c000000000a47d20] [c0000000000c8788] pick_next_task_fair+0x48/0x3a0
> >> [   65.153424] [c000000000a47d80] [c000000000466620] __schedule+0xf0/0x430
> >> [   65.153612] [c000000000a47de0] [c000000000466b04] schedule_idle+0x34/0x70
> >> [   65.153786] [c000000000a47e10] [c0000000000c0bc8] do_idle+0x1a8/0x220
> >> [   65.154121] [c000000000a47e70] [c0000000000c0e94] cpu_startup_entry+0x34/0x40
> >> [   65.154313] [c000000000a47ea0] [c00000000000ef1c] rest_init+0x10c/0x124
> >> [   65.154414] [c000000000a47ee0] [c000000000500004] start_kernel+0x568/0x594
> >> [   65.154585] [c000000000a47f90] [c00000000000a7cc] start_here_common+0x1c/0x330
> >> [   65.154854] Instruction dump:
> >> [   65.155191] 38210030 e8010010 7c0803a6 4e800020 3d220004 39295228 81290000 3929ffff
> >> [   65.155498] 7d284038 7c0004ac 5508017e 65082800 <7c00411c> e94d0178 812a0000 3929ffff
> >                                                       ^
> > Again the faulting instruction there is "msgsndp r8"
> > 
> >> [   65.156155] ---[ end trace 6180d12e268ffdaf ]---
> >> [   65.185452]
> >> [   66.187490] Kernel panic - not syncing: Fatal exception
> >>
> >> This is with "qemu-system-ppc64 -smp 4 -machine pseries" on QEMU 4.0.0.
> >>
> >> I'm not totally sure what's going on here. I'm emulating a pseries, and 
> >> using that with qemu's pseries model, and I see that selecting the 
> >> pseries forces the selection of 'config PPC_DOORBELL' (twice in the same 
> >> section, actually).
> > 
> > Noted.
> > 
> >> Then inside the kernel there appears to be some runtime CPU check for
> >> doorbell support.
> > 
> > Not really. The kernel looks at the CPU revision (PVR) and decides that
> > it has doorbell support.
> > 
> >> Is this a case in which QEMU is advertising doorbell support that TCG
> >> doesn't have? Or is something else happening here?
> > 
> > It's a gap in the emulation I guess. qemu doesn't emulate msgsndp, but
> > it really should because that's a supported instruction since Power8.
> 
> There is a patch for msgsndp in my tree you could try : 
> 
>   https://github.com/legoater/qemu/tree/powernv-5.0
> 
> Currently being reviewed. I have to address some remarks from David before
> it can be merged.

Right.  It needs some polish, but I expect we'll have this merged in
the not too distant future.

> > I suspect msgsndp wasn't implemented for TCG because TCG doesn't support
> > more than one thread per core, and you can only send doorbells to other
> > threads in the same core, and therefore there is no reason to ever use
> > msgsndp.
> 
> There is a need now with KVM emulation under TCG, but, yes, QEMU still lacks
> SMT support.
> 
> > That's the message Suraj mentioned up thread, eg:
> > 
> >   $ qemu-system-ppc64 -nographic -vga none -M pseries -smp 2,threads=2 -cpu POWER8 -kernel build~/vmlinux
> >   qemu-system-ppc64: TCG cannot support more than 1 thread/core on a pseries machine
> > 
> > 
> > But I guess we've hit another case of a CPU sending itself an IPI, and
> > the way the sibling masks are done, CPUs are siblings of themselves, so
> > the sibling test passes, eg:
> > 
> > int doorbell_try_core_ipi(int cpu)
> > {
> > 	int this_cpu = get_cpu();
> > 	int ret = 0;
> > 
> > 	if (cpumask_test_cpu(cpu, cpu_sibling_mask(this_cpu))) {
> > 		doorbell_core_ipi(cpu);
> > 
> > 
> > 
> > In which case this patch should fix it.
> > 
> > diff --git a/arch/powerpc/kernel/dbell.c b/arch/powerpc/kernel/dbell.c
> > index f17ff1200eaa..e45cb9bba193 100644
> > --- a/arch/powerpc/kernel/dbell.c
> > +++ b/arch/powerpc/kernel/dbell.c
> > @@ -63,7 +63,7 @@ int doorbell_try_core_ipi(int cpu)
> >  	int this_cpu = get_cpu();
> >  	int ret = 0;
> >  
> > -	if (cpumask_test_cpu(cpu, cpu_sibling_mask(this_cpu))) {
> > +	if (cpu != this_cpu && cpumask_test_cpu(cpu, cpu_sibling_mask(this_cpu))) {
> >  		doorbell_core_ipi(cpu);
> >  		ret = 1;
> >  	}
> > 
> > 
> > The other option would be we disable CPU_FTR_DBELL if we detect we're
> > running under TCG. But I'm not sure we have a particularly clean way to
> > detect that.
> 
> does the pseries kernel support cpufeatures in the DT ?
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[Jason A. Donenfeld]

On Thu, Dec 19, 2019 at 1:52 PM Michael Ellerman <mpe@ellerman.id.au> wrote:
> diff --git a/arch/powerpc/kernel/dbell.c b/arch/powerpc/kernel/dbell.c
> index f17ff1200eaa..e45cb9bba193 100644
> --- a/arch/powerpc/kernel/dbell.c
> +++ b/arch/powerpc/kernel/dbell.c
> @@ -63,7 +63,7 @@ int doorbell_try_core_ipi(int cpu)
>         int this_cpu = get_cpu();
>         int ret = 0;
>
> -       if (cpumask_test_cpu(cpu, cpu_sibling_mask(this_cpu))) {
> +       if (cpu != this_cpu && cpumask_test_cpu(cpu, cpu_sibling_mask(this_cpu))) {
>                 doorbell_core_ipi(cpu);
>                 ret = 1;
>         }

I realize the best solution is that nice powernv branch that will
eventually be merged for qemu/tcg. But, maybe the above would be a
decent idea to upstream? It seems like that's a case of a superfluous
doorbell?

Jason

Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[Jason A. Donenfeld]

On Thu, Dec 19, 2019 at 2:08 PM Cédric Le Goater <clg@kaod.org> wrote:>
> There is a patch for msgsndp in my tree you could try :
>
>   https://github.com/legoater/qemu/tree/powernv-5.0
>
> Currently being reviewed. I have to address some remarks from David before
> it can be merged.

Thanks. I've cherry-picked
https://github.com/legoater/qemu/commit/910c9ea5ecc and can confirm
that I no longer receive the crashes. QEMU 5.1 or 5.0.1 release, I
guess?

Jason

Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[David? Gibson]

[-- Attachment #1: Type: text/plain, Size: 839 bytes --]

On Fri, Dec 20, 2019 at 12:32:06PM +0100, Jason A. Donenfeld wrote:
> On Thu, Dec 19, 2019 at 2:08 PM Cédric Le Goater <clg@kaod.org> wrote:>
> > There is a patch for msgsndp in my tree you could try :
> >
> >   https://github.com/legoater/qemu/tree/powernv-5.0
> >
> > Currently being reviewed. I have to address some remarks from David before
> > it can be merged.
> 
> Thanks. I've cherry-picked
> https://github.com/legoater/qemu/commit/910c9ea5ecc and can confirm
> that I no longer receive the crashes. QEMU 5.1 or 5.0.1 release, I
> guess?

Unless the revision takes much longer than I expect, I'd anticipate it
in 5.0.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Qemu-ppc] pseries on qemu-system-ppc64le crashes in doorbell_core_ipi()

[Cédric Le Goater]

On 12/20/19 12:32 PM, Jason A. Donenfeld wrote:
> On Thu, Dec 19, 2019 at 2:08 PM Cédric Le Goater <clg@kaod.org> wrote:>
>> There is a patch for msgsndp in my tree you could try :
>>
>>   https://github.com/legoater/qemu/tree/powernv-5.0
>>
>> Currently being reviewed. I have to address some remarks from David before
>> it can be merged.
> 
> Thanks. I've cherry-picked
> https://github.com/legoater/qemu/commit/910c9ea5ecc and can confirm
> that I no longer receive the crashes. QEMU 5.1 or 5.0.1 release, I
> guess?

QEMU 5.1 I would say. That's enough time to address the comments.

Michael's patch to drop self IPIs is also interesting for Linux. I wonder
how we can reach that point. 

Thanks,

C.