memory overcomittment with sr-iov device assighment

memory overcomittment with sr-iov device assighment

[alex.nlnnfn]

Hello list,

I looked into Xen documentation and also Xen wiki and I could't find a definitive answer if Xen supports memory over-commitment when VMs use SR-IOV device assignment (passthrough). Memory over-commitment I mean giving VMs more RAM than is available in the host.

I know that ESX doesn't support it, and also QEMU/KVM pins all RAM when a device is directly assigned to a VM (VFIO_IOMMU_MAP_DMA ioctl). I have two questions:

1. Does Xen supports memory over commitment when all VMs are using direct device assignment e.g., SR-IOV?
2. Would you please point me to the code that does the pinning?

Thanks,
Alex

Re: memory overcomittment with sr-iov device assighment

[Jan Beulich]

On 07.06.2022 05:59, alex.nlnnfn@proton.me wrote:
> I looked into Xen documentation and also Xen wiki and I could't find a definitive answer if Xen supports memory over-commitment when VMs use SR-IOV device assignment (passthrough). Memory over-commitment I mean giving VMs more RAM than is available in the host.
> 
> I know that ESX doesn't support it, and also QEMU/KVM pins all RAM when a device is directly assigned to a VM (VFIO_IOMMU_MAP_DMA ioctl). I have two questions:
> 
> 1. Does Xen supports memory over commitment when all VMs are using direct device assignment e.g., SR-IOV?

What you describe looks to match mem-paging / mem-sharing in Xen. Neither is
compatible with device pass-through (SR-IOV or not is irrelevant), and both
are only in "experimental" state anyway.

> 2. Would you please point me to the code that does the pinning?

That's a KVM concept with no direct equivalent in Xen (largely due to Xen
being a type-1 hypervisor, while KVM is a type-2 one).

Jan

Re: memory overcomittment with sr-iov device assighment

[Andrew Cooper]

On 07/06/2022 04:59, alex.nlnnfn@proton.me wrote:
> Hello list,
>
> I looked into Xen documentation and also Xen wiki and I could't find a definitive answer if Xen supports memory over-commitment when VMs use SR-IOV device assignment (passthrough). Memory over-commitment I mean giving VMs more RAM than is available in the host.
>
> I know that ESX doesn't support it, and also QEMU/KVM pins all RAM when a device is directly assigned to a VM (VFIO_IOMMU_MAP_DMA ioctl). I have two questions:
>
> 1. Does Xen supports memory over commitment when all VMs are using direct device assignment e.g., SR-IOV?

No.  Memory overcommit is fundamentally incompatible with having real
devices.

On the CPU side, EPT_VIOLATION/EPT_MISCONFIG are faults, just like #PF,
so we can play games with swapping out some other part of the guest and
paging in the part which is currently being accessed.

But IOMMU violations are not restartable.  We can't just take an IOMMU
fault, and shuffle the guests memory, because the PCIe protocol has
timeouts.  These aren't generally long enough to even send an interrupt
to request software help, let alone page one part out and another part in.

For an IOMMU mapping to exists, it must point at real RAM, because any
DMA targetting it cannot be paused and delayed for later.

~Andrew

Re: memory overcomittment with sr-iov device assighment

[Alex Nln]

------- Original Message -------
On Tuesday, June 7th, 2022 at 3:04 AM, Andrew Cooper <Andrew.Cooper3@citrix.com> wrote:
> But IOMMU violations are not restartable. We can't just take an IOMMU
> fault, and shuffle the guests memory, because the PCIe protocol has
> timeouts. These aren't generally long enough to even send an interrupt
> to request software help, let alone page one part out and another part in.
>
> For an IOMMU mapping to exists, it must point at real RAM, because any
> DMA targetting it cannot be paused and delayed for later.
>

Thanks for the information!

Speaking about IOMMU. Can Xen emulate virtual IOMMU? Just thinking out loud.
If Xen exposes a virtual IOMMU to a guest VM, wouldn't it be possible
for the hypervisor to pin VM's pages that are mapped (dynamically)
by the guest virtual IOMMU? But I guess it will eliminate the performance
benefits of direct device assignment.

Re: memory overcomittment with sr-iov device assighment

[Andrew Cooper]

On 07/06/2022 20:22, Alex Nln wrote:
> ------- Original Message -------
> On Tuesday, June 7th, 2022 at 3:04 AM, Andrew Cooper <Andrew.Cooper3@citrix.com> wrote:
>> But IOMMU violations are not restartable. We can't just take an IOMMU
>> fault, and shuffle the guests memory, because the PCIe protocol has
>> timeouts. These aren't generally long enough to even send an interrupt
>> to request software help, let alone page one part out and another part in.
>>
>> For an IOMMU mapping to exists, it must point at real RAM, because any
>> DMA targetting it cannot be paused and delayed for later.
>>
> Thanks for the information!
>
> Speaking about IOMMU. Can Xen emulate virtual IOMMU? Just thinking out loud.
> If Xen exposes a virtual IOMMU to a guest VM, wouldn't it be possible
> for the hypervisor to pin VM's pages that are mapped (dynamically)
> by the guest virtual IOMMU? But I guess it will eliminate the performance
> benefits of direct device assignment.

As Jan pointed out, there's no such thing as pinning like that.  Xen is
not KVM.  See
https://xenbits.xen.org/docs/latest/admin-guide/introduction.html


But "can you only deal with the subset mapped in a virtual IOMMU" is a
reasonable question.  The answer is no, for two reasons.

First, what do you do if the guest maps it's whole guest physical
address space in the virtual IOMMU?  There's no "sorry - not got enough
memory to arrange that" error available in either Intel or AMD's IOMMU
spec, because there's no such thing as memory overcommit in a real system.

Secondly and more importantly, the at-reset behaviour of an IOMMU is no
translation, which for a guest means that DMA can go anywhere in the
guest physical address space, meaning the whole guest physical address
space needs mapping into the IOMMU pagetables,

~Andrew