KVM/VFIO passthrough not working when TRIM_UNUSED_KSYMS is enabled

KVM/VFIO passthrough not working when TRIM_UNUSED_KSYMS is enabled

[Gunnar Eggen]

Hi,

It's a bit unclear what subsystem is to blame for this problem, so I'm 
sending this to both KVM, VFIO and Module support.

The problem is that trimming unused symbols in the kernel breaks VFIO 
passthrough on x86/amd64 at least. If the option TRIM_UNUSED_KSYMS is 
enabled you will see the following error when trying to start a VM in 
QEmu with any pcie device passed via VFIO:

qemu-system-x86_64: -device vfio-pci,host=04:00.0: Failed to add group 
25 to KVM VFIO device: Invalid argument

The error will not stop the VM from launching, but it will break things 
in mysterious ways when e.g. installing graphics drivers.
No external modules is involved in this, so I would guess that there is 
some dependency that the trimming is missing in some way.

With the introduction of UNUSED_KSYMS_WHITELIST in the latest kernels, 
and some talk about making trimming symbols the default in the future, 
it would be great if we could get this fixed or at least identify the 
problematic symbols so that they could be whitelisted if needed.

Steps to reproduce:

1 - Have a kernel where TRIM_UNUSED_KSYMS is enabled
2 - Start a VM in QEmu/KVM with a pcie device passed through via vfio-pci

This is a common issue that keeps popping up on user forums related to 
vfio passthrough, so it should be fairly simple to reproduce.

Let me know if you want more details or perhaps my kernel config or 
trimmed system map to test with.

Best regards,
Gunnar

Re: KVM/VFIO passthrough not working when TRIM_UNUSED_KSYMS is enabled

[Paolo Bonzini]

On 04/07/20 23:03, Gunnar Eggen wrote:
> Hi,
> 
> It's a bit unclear what subsystem is to blame for this problem, so I'm
> sending this to both KVM, VFIO and Module support.
> 
> The problem is that trimming unused symbols in the kernel breaks VFIO
> passthrough on x86/amd64 at least. If the option TRIM_UNUSED_KSYMS is
> enabled you will see the following error when trying to start a VM in
> QEmu with any pcie device passed via VFIO:
> 
> qemu-system-x86_64: -device vfio-pci,host=04:00.0: Failed to add group
> 25 to KVM VFIO device: Invalid argument
> 
> The error will not stop the VM from launching, but it will break things
> in mysterious ways when e.g. installing graphics drivers.
> No external modules is involved in this, so I would guess that there is
> some dependency that the trimming is missing in some way.
> 
> With the introduction of UNUSED_KSYMS_WHITELIST in the latest kernels,
> and some talk about making trimming symbols the default in the future,
> it would be great if we could get this fixed or at least identify the
> problematic symbols so that they could be whitelisted if needed.

They are:
- vfio_group_get_external_user
- vfio_external_group_match_file
- vfio_group_put_external_user
- vfio_group_set_kvm
- vfio_external_check_extension
- vfio_external_user_iommu_id

and also (unrelated but breaking other stuff):
- mdev_get_iommu_device
- mdev_bus_type

However, UNUSED_KSYMS_WHITELIST seems the wrong tool for this.  We would
need to have something that says: "if KVM && VFIO, then include these
symbols", for example a macro "IMPORT_SYMBOL" that would be processed by
cmd_undef_syms.

Paolo

> Steps to reproduce:
> 
> 1 - Have a kernel where TRIM_UNUSED_KSYMS is enabled
> 2 - Start a VM in QEmu/KVM with a pcie device passed through via vfio-pci
> 
> This is a common issue that keeps popping up on user forums related to
> vfio passthrough, so it should be fairly simple to reproduce.
> 
> Let me know if you want more details or perhaps my kernel config or
> trimmed system map to test with.
> 
> Best regards,
> Gunnar
> 

Re: KVM/VFIO passthrough not working when TRIM_UNUSED_KSYMS is enabled

[Gunnar Eggen]

Great! I can confirm that whitelisting the symbols you listed fixes the 
problem.

I hope we get a permanent solution for this eventually, but in the 
meantime we have a workaround.

Thanks so much!


On 05.07.2020 07:44, Paolo Bonzini wrote:
> On 04/07/20 23:03, Gunnar Eggen wrote:
>> Hi,
>>
>> It's a bit unclear what subsystem is to blame for this problem, so I'm
>> sending this to both KVM, VFIO and Module support.
>>
>> The problem is that trimming unused symbols in the kernel breaks VFIO
>> passthrough on x86/amd64 at least. If the option TRIM_UNUSED_KSYMS is
>> enabled you will see the following error when trying to start a VM in
>> QEmu with any pcie device passed via VFIO:
>>
>> qemu-system-x86_64: -device vfio-pci,host=04:00.0: Failed to add group
>> 25 to KVM VFIO device: Invalid argument
>>
>> The error will not stop the VM from launching, but it will break things
>> in mysterious ways when e.g. installing graphics drivers.
>> No external modules is involved in this, so I would guess that there is
>> some dependency that the trimming is missing in some way.
>>
>> With the introduction of UNUSED_KSYMS_WHITELIST in the latest kernels,
>> and some talk about making trimming symbols the default in the future,
>> it would be great if we could get this fixed or at least identify the
>> problematic symbols so that they could be whitelisted if needed.
> They are:
> - vfio_group_get_external_user
> - vfio_external_group_match_file
> - vfio_group_put_external_user
> - vfio_group_set_kvm
> - vfio_external_check_extension
> - vfio_external_user_iommu_id
>
> and also (unrelated but breaking other stuff):
> - mdev_get_iommu_device
> - mdev_bus_type
>
> However, UNUSED_KSYMS_WHITELIST seems the wrong tool for this.  We would
> need to have something that says: "if KVM && VFIO, then include these
> symbols", for example a macro "IMPORT_SYMBOL" that would be processed by
> cmd_undef_syms.
>
> Paolo
>
>> Steps to reproduce:
>>
>> 1 - Have a kernel where TRIM_UNUSED_KSYMS is enabled
>> 2 - Start a VM in QEmu/KVM with a pcie device passed through via vfio-pci
>>
>> This is a common issue that keeps popping up on user forums related to
>> vfio passthrough, so it should be fairly simple to reproduce.
>>
>> Let me know if you want more details or perhaps my kernel config or
>> trimmed system map to test with.
>>
>> Best regards,
>> Gunnar
>>