* memory overcomittment with sr-iov device assighment @ 2022-06-07 3:59 alex.nlnnfn 2022-06-07 7:11 ` Jan Beulich 2022-06-07 10:04 ` Andrew Cooper 0 siblings, 2 replies; 5+ messages in thread From: alex.nlnnfn @ 2022-06-07 3:59 UTC (permalink / raw) To: xen-devel Hello list, I looked into Xen documentation and also Xen wiki and I could't find a definitive answer if Xen supports memory over-commitment when VMs use SR-IOV device assignment (passthrough). Memory over-commitment I mean giving VMs more RAM than is available in the host. I know that ESX doesn't support it, and also QEMU/KVM pins all RAM when a device is directly assigned to a VM (VFIO_IOMMU_MAP_DMA ioctl). I have two questions: 1. Does Xen supports memory over commitment when all VMs are using direct device assignment e.g., SR-IOV? 2. Would you please point me to the code that does the pinning? Thanks, Alex ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: memory overcomittment with sr-iov device assighment 2022-06-07 3:59 memory overcomittment with sr-iov device assighment alex.nlnnfn @ 2022-06-07 7:11 ` Jan Beulich 2022-06-07 10:04 ` Andrew Cooper 1 sibling, 0 replies; 5+ messages in thread From: Jan Beulich @ 2022-06-07 7:11 UTC (permalink / raw) To: alex.nlnnfn; +Cc: xen-devel On 07.06.2022 05:59, alex.nlnnfn@proton.me wrote: > I looked into Xen documentation and also Xen wiki and I could't find a definitive answer if Xen supports memory over-commitment when VMs use SR-IOV device assignment (passthrough). Memory over-commitment I mean giving VMs more RAM than is available in the host. > > I know that ESX doesn't support it, and also QEMU/KVM pins all RAM when a device is directly assigned to a VM (VFIO_IOMMU_MAP_DMA ioctl). I have two questions: > > 1. Does Xen supports memory over commitment when all VMs are using direct device assignment e.g., SR-IOV? What you describe looks to match mem-paging / mem-sharing in Xen. Neither is compatible with device pass-through (SR-IOV or not is irrelevant), and both are only in "experimental" state anyway. > 2. Would you please point me to the code that does the pinning? That's a KVM concept with no direct equivalent in Xen (largely due to Xen being a type-1 hypervisor, while KVM is a type-2 one). Jan ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: memory overcomittment with sr-iov device assighment 2022-06-07 3:59 memory overcomittment with sr-iov device assighment alex.nlnnfn 2022-06-07 7:11 ` Jan Beulich @ 2022-06-07 10:04 ` Andrew Cooper 2022-06-07 19:22 ` Alex Nln 1 sibling, 1 reply; 5+ messages in thread From: Andrew Cooper @ 2022-06-07 10:04 UTC (permalink / raw) To: alex.nlnnfn, xen-devel On 07/06/2022 04:59, alex.nlnnfn@proton.me wrote: > Hello list, > > I looked into Xen documentation and also Xen wiki and I could't find a definitive answer if Xen supports memory over-commitment when VMs use SR-IOV device assignment (passthrough). Memory over-commitment I mean giving VMs more RAM than is available in the host. > > I know that ESX doesn't support it, and also QEMU/KVM pins all RAM when a device is directly assigned to a VM (VFIO_IOMMU_MAP_DMA ioctl). I have two questions: > > 1. Does Xen supports memory over commitment when all VMs are using direct device assignment e.g., SR-IOV? No. Memory overcommit is fundamentally incompatible with having real devices. On the CPU side, EPT_VIOLATION/EPT_MISCONFIG are faults, just like #PF, so we can play games with swapping out some other part of the guest and paging in the part which is currently being accessed. But IOMMU violations are not restartable. We can't just take an IOMMU fault, and shuffle the guests memory, because the PCIe protocol has timeouts. These aren't generally long enough to even send an interrupt to request software help, let alone page one part out and another part in. For an IOMMU mapping to exists, it must point at real RAM, because any DMA targetting it cannot be paused and delayed for later. ~Andrew ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: memory overcomittment with sr-iov device assighment 2022-06-07 10:04 ` Andrew Cooper @ 2022-06-07 19:22 ` Alex Nln 2022-06-07 19:47 ` Andrew Cooper 0 siblings, 1 reply; 5+ messages in thread From: Alex Nln @ 2022-06-07 19:22 UTC (permalink / raw) To: Andrew Cooper; +Cc: xen-devel ------- Original Message ------- On Tuesday, June 7th, 2022 at 3:04 AM, Andrew Cooper <Andrew.Cooper3@citrix.com> wrote: > But IOMMU violations are not restartable. We can't just take an IOMMU > fault, and shuffle the guests memory, because the PCIe protocol has > timeouts. These aren't generally long enough to even send an interrupt > to request software help, let alone page one part out and another part in. > > For an IOMMU mapping to exists, it must point at real RAM, because any > DMA targetting it cannot be paused and delayed for later. > Thanks for the information! Speaking about IOMMU. Can Xen emulate virtual IOMMU? Just thinking out loud. If Xen exposes a virtual IOMMU to a guest VM, wouldn't it be possible for the hypervisor to pin VM's pages that are mapped (dynamically) by the guest virtual IOMMU? But I guess it will eliminate the performance benefits of direct device assignment. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: memory overcomittment with sr-iov device assighment 2022-06-07 19:22 ` Alex Nln @ 2022-06-07 19:47 ` Andrew Cooper 0 siblings, 0 replies; 5+ messages in thread From: Andrew Cooper @ 2022-06-07 19:47 UTC (permalink / raw) To: Alex Nln; +Cc: xen-devel On 07/06/2022 20:22, Alex Nln wrote: > ------- Original Message ------- > On Tuesday, June 7th, 2022 at 3:04 AM, Andrew Cooper <Andrew.Cooper3@citrix.com> wrote: >> But IOMMU violations are not restartable. We can't just take an IOMMU >> fault, and shuffle the guests memory, because the PCIe protocol has >> timeouts. These aren't generally long enough to even send an interrupt >> to request software help, let alone page one part out and another part in. >> >> For an IOMMU mapping to exists, it must point at real RAM, because any >> DMA targetting it cannot be paused and delayed for later. >> > Thanks for the information! > > Speaking about IOMMU. Can Xen emulate virtual IOMMU? Just thinking out loud. > If Xen exposes a virtual IOMMU to a guest VM, wouldn't it be possible > for the hypervisor to pin VM's pages that are mapped (dynamically) > by the guest virtual IOMMU? But I guess it will eliminate the performance > benefits of direct device assignment. As Jan pointed out, there's no such thing as pinning like that. Xen is not KVM. See https://xenbits.xen.org/docs/latest/admin-guide/introduction.html But "can you only deal with the subset mapped in a virtual IOMMU" is a reasonable question. The answer is no, for two reasons. First, what do you do if the guest maps it's whole guest physical address space in the virtual IOMMU? There's no "sorry - not got enough memory to arrange that" error available in either Intel or AMD's IOMMU spec, because there's no such thing as memory overcommit in a real system. Secondly and more importantly, the at-reset behaviour of an IOMMU is no translation, which for a guest means that DMA can go anywhere in the guest physical address space, meaning the whole guest physical address space needs mapping into the IOMMU pagetables, ~Andrew ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2022-06-07 19:47 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-06-07 3:59 memory overcomittment with sr-iov device assighment alex.nlnnfn 2022-06-07 7:11 ` Jan Beulich 2022-06-07 10:04 ` Andrew Cooper 2022-06-07 19:22 ` Alex Nln 2022-06-07 19:47 ` Andrew Cooper
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.