Re: [RFC PATCH] hw/arm/virt: Support NMI injection

[Gavin Shan <gshan@redhat.com>]

On 1/28/20 9:56 PM, Auger Eric wrote:
> Hi Marc,
> On 1/28/20 10:25 AM, Marc Zyngier wrote:
>> Gavin, Eric,
>>
>> On 2020-01-28 08:05, Auger Eric wrote:
>>> Hi,
>>>
>>> On 1/28/20 7:48 AM, Gavin Shan wrote:
>>>> [including more folks into the discussion]
>>>>
>>>>> On Fri, 17 Jan 2020 at 14:00, Peter Maydell <peter.maydell@linaro.org>
>>>>> wrote:
>>>>>> On Thu, 19 Dec 2019 at 04:06, Gavin Shan <gshan@redhat.com> wrote:
>>>>>>> This supports NMI injection for virtual machine and currently it's
>>>>>>> only
>>>>>>> supported on GICv3 controller, which is emulated by qemu or host
>>>>>>> kernel.
>>>>>>> The design is highlighted as below:
>>>>>>>
>>>>>>> * The NMI is identified by its priority (0x20). In the guest (linux)
>>>>>>> kernel, the GICC_PMR is set to 0x80, to block all interrupts except
>>>>>>> the NMIs when the external interrupt is disabled. It means the FIQ
>>>>>>> and IRQ bit in PSTATE isn't touched when the functionality (NMI) is
>>>>>>> functional.
>>>>>>> * LPIs aren't considered as NMIs because of their nature. It means
>>>>>>> NMI
>>>>>>> is either SPI or PPI. Besides, the NMIs are injected in round-robin
>>>>>>> fashion is there are multiple NMIs existing.
>>>>>>> * When the GICv3 controller is emulated by qemu, the interrupt states
>>>>>>> (e.g. enabled, priority) is fetched from the corresponding data
>>>>>>> struct
>>>>>>> directly. However, we have to pause all CPUs to fetch the interrupt
>>>>>>> states from host in advance if the GICv3 controller is emulated by
>>>>>>> host.
>>>>>>>
>>>>>>> The testing scenario is to tweak guest (linux) kernel where the
>>>>>>> pl011 SPI
>>>>>>> can be enabled as NMI by request_nmi(). Check "/proc/interrupts"
>>>>>>> after injecting
>>>>>>> several NMIs, to see if the interrupt count is increased or not. The
>>>>>>> result
>>>>>>> is just as expected.
>>>>>>>
>>>>>
>>>>> So, QEMU is trying to emulate actual hardware. None of this
>>>>> looks to me like what GICv3 hardware does... If you want to
>>>>> have the virt board send an interrupt, do it the usual way
>>>>> by wiring up a qemu_irq from some device to the GIC, please.
>>>>> (More generally, there is no concept of an "NMI" in the GIC;
>>>>> there are just interrupts at varying possible guest-programmable
>>>>> priority levels.)
>>>>>
>>>>
>>>> Peter, I missed to read your reply in time and apologies for late
>>>> response.
>>>>
>>>> Yes, there is no concept of "NMI" in the GIC from hardware perspective.
>>>> However, NMI has been supported from the software by kernel commit
>>>> bc3c03ccb4641 ("arm64: Enable the support of pseudo-NMIs"). The NMIs
>>>> have higher priority than normal ones. NMIs are deliverable after
>>>> local_irq_disable() because the SYS_ICC_PMR_EL1 is tweaked so that
>>>> normal interrupts are masked only.
>>
>> And none of that is an NMI. This is a completely SW-defined mechanism,
>> and you can't rely on this to inject something that would behave as
>> a NMI in in a guest. I thought the "pseudo" prefix would give it away :-(.
>>

Marc, thanks for the explanation.

>>>>
>>>> It's unclear about the purpose of "nmi" QMP/HMP command. It's why I
>>>> put a RFC tag. The command has been supported by multiple architects
>>>> including x86/ppc. However, they are having different behaviors. The
>>>> system will be restarted on ppc with this command, but a NMI is injected
>>>> through LAPIC on x86. So I'm not sure what architect (system reset on
>>>> ppc or injecting NMI on x86) aarch64 should follow.
>>
>> The x86 NMI has no equivalent on ARM, full stop. And the only thing that
>> the ARM implementation should follow is the letter of the architecture,
>> without added concepts.
>>
>>> arm_pmu driver was reworked to use pseudo-NMIs. I don't know the exact
>>> status of this work though
>>> (https://patchwork.kernel.org/cover/11047407/). So we cannot use any
>>> random NMI for guest injection.
>>>
>>> I wonder whether we should implement the KVM_NMI vcpu ioctl once we have
>>> agreed on which behavior is expected upon NMI injection. However the
>>> kernel doc says this ioctl only is well defined if "KVM_CREATE_IRQCHIP
>>> has not been called" (?).
>>
>> But what architectural concept would you map your KVM_NMI to? The number
>> of things you can do is pretty limited:
>>
>> - Reset: we already have this
>> - Interrupt: you don't get to decide the priority or the group
>> - SError: Pretty much fatal in all cases
>>
>> You *could* try something like SDEI [1], but that's a pretty terrible
>> interface too.
> 
> Thank you for the pointer.
> 
> So Gavin, not sure the QEMU QMP/HMP NMI command is relevant on ARM (at
> least at this point)?
> 

Yes, the primary concern is what behavior we should have for ARM when
QMP/HMP "nmi" command is executed. After that's determined, I can dig
into SDEI if needed.

As Alexey said in another reply, it's used to force the guest to have
crash dump or drop into in-kernel debugger (xmon) on PowerPC. However,
x86 guest will receive NMI after the command is issued. This RFC patch
is following x86 to inject "pseudo" NMIs, but it seems incorrect. So
the question is what behavior we should have for ARM when QMP/HMP "nmi"
command is issued? I'm expecting more input in this regard :)

Thanks,
Gavin

> Thanks
> 
> Eric
> 
> 
>>
>>          M.
>>
>> [1]
>> https://static.docs.arm.com/den0054/a/ARM_DEN0054A_Software_Delegated_Exception_Interface.pdf
>>
>