On 04.10.21 13:02, Marc Zyngier wrote:
> On Mon, 04 Oct 2021 11:30:06 +0100,
> Lukas Jünger <lukas.juenger@ice.rwth-aachen.de> wrote:
>> [1  <text/plain (en-US); utf-8 (quoted-printable)>]
>> On 04.10.21 12:24, Marc Zyngier wrote:
>>> Hi Lukas,
>> Hi Mark,
>>
>> Thanks for your quick reply.
>>
>>> On Mon, 04 Oct 2021 11:07:47 +0100,
>>> Lukas Jünger <lukas.juenger@ice.rwth-aachen.de> wrote:
>>>> Hello,
>>>>
>>>> I am trying to run an emulator that uses KVM on arm64 to execute
>>>> code. The emulator contains a userspace model of a GICv2 IRQ
>>>> controller. The platform that I am running on (n1sdp) has a
>>> N1-SDP? My condolences...
>> Is there more to this?
> How do you like the PCI patches? :D
Ah, that's what you were alluding to. PCI+ARM seems to be tricky 
somehow. The SynQuacer dev box as well as the ROCKPro 64 I was using 
before also had PCI issues.
>>>> GICv3. When I boot Linux in the emulator I run into
>>>> gic_check_cpu_features()  in drivers/irqchip/irq-gic.c, which taints
>>>> the kernel as the host uses system registers to communicate with the
>>>> host GICv3. I saw that ICC_SRE_ELx can be used to force MMIO, but
>>>> setting this from inside the VM did not work and using KVM_SET_ONE_REG
>>>> failed with error.
>>> N1-SDP doesn't implement the MMIO interface at all, and our GIC
>>> emulation doesn't either. Both are valid implementations.
>>>
>>>> Is there a way to use a userspace GICv2 model with KVM on a GICv3 host
>>>> without tainting?
>>> The tainting happens because you have created a VM with a GICv3
>>> irqchip (at some point, your VMM calls into KVM to create a device
>>> with the KVM_DEV_TYPE_ARM_VGIC_V3 attribute). The guest then sees that
>>> GICv3 is enabled (ICC_SRE_ELx.SRE==1), and yet you somehow expose a
>>> GICv2 to the guest (either via DT or ACPI). That's illegal.
>>>
>>> If you want a userspace interrupt controller, you need prevent the
>>> creation of an in-kernel interrupt controller, which is a change in
>>> your VMM or maybe a configuration change.
>> I'm not using an in-kernel irq controller, at least I don't set one
>> up. This is all custom, so no QEMU etc. The GICv2 is also a custom
>> model that lives in user space. The guest gets a DT telling it that
>> there is a GICv2 and it should access it via MMIO. This all used to
>> work on Raspberry Pi 3
> RPI3 doesn't have a GIC at all, so the example is a bit moot.
True. Fair point.
>> and Socionext Synquacer.
> This one however is more interesting, as it has a GICv3 + v2 compat.
>
>> The port to N1-SDP is
>> giving me trouble. I understand why it is tainting the kernel, I was
>> just wondering if I could somehow tell KVM to set this up correctly,
>> e.g. by setting the ICC_SRE_ELx.
> KVM doesn't *set* ICC_SRE_EL1.SRE. It is RAO/WI on this machine, which
> is perfectly legal. However, KVM traps this access and emulates it
> (access_gic_sre() returns vcpu->arch.vgic_cpu.vgic_v3.vgic_sre).
>
> So if you see ICC_SRE_EL1.SRE==1 in your guest, that's because
> vgic_sre is set to something that is non-zero. The only way for this
> bit to be set is in vgic_v3_enable(), which has the following code:
>
> <quote>
> 	if (vcpu->kvm->arch.vgic.vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3) {
> 		vgic_v3->vgic_sre = (ICC_SRE_EL1_DIB |
> 				     ICC_SRE_EL1_DFB |
> 				     ICC_SRE_EL1_SRE);
> 		vcpu->arch.vgic_cpu.pendbaser = INITIAL_PENDBASER_VALUE;
> 	} else {
> 		vgic_v3->vgic_sre = 0;
> 	}
> </quote>
>
> So short of a terrible bug that would dump random values in this
> structure, you are setting vgic_model to a GICv3 implementation. This
> can only be done from userspace if you are creating a GICv3 irqchip.
>
> Without seeing what your userspace does, I'm afraid I can't help you
> much further. Can you please provide some traces of what it does? A
> strace dump would certainly help.

Could it be that this is because I use KVM_ARM_PREFERRED_TARGET and init 
the vcpu from this config?

I have attached an strace log file.

>
> Thanks,
>
> 	M.

Thank you for your help,

Lukas