Re: issues with emulated PCI MMIO backed by host memory under KVM

[Andrew Jones <drjones@redhat.com>]

On Tue, Jun 28, 2016 at 02:25:19PM +0100, Catalin Marinas wrote:
> On Tue, Jun 28, 2016 at 03:19:14PM +0200, Ard Biesheuvel wrote:
> > On 28 June 2016 at 15:10, Catalin Marinas <catalin.marinas@arm.com> wrote:
> > > On Tue, Jun 28, 2016 at 02:20:43PM +0200, Christoffer Dall wrote:
> > >> On Tue, Jun 28, 2016 at 01:06:36PM +0200, Laszlo Ersek wrote:
> > >> > On 06/28/16 12:04, Christoffer Dall wrote:
> > >> > > On Mon, Jun 27, 2016 at 03:57:28PM +0200, Ard Biesheuvel wrote:
> > >> > >> So if vga-pci.c is the only problematic device, for which a reasonable
> > >> > >> alternative exists (virtio-gpu), I think the only feasible solution is
> > >> > >> to educate QEMU not to allow RAM memslots being exposed via PCI BARs
> > >> > >> when running under KVM/ARM.
> > >> > >
> > >> > > It would be good if we could support vga-pci under KVM/ARM, but if
> > >> > > there's no other way than rewriting the arm64 kernel's memory mappings
> > >> > > completely, then probably we're stuck there, unfortunately.
> > >
> > > Just to be clear, the behaviour of mismatched memory attributes is
> > > defined in the ARM ARM and so far Linux worked fine with such cacheable
> > > vs non-cacheable (as long as only one of them is accessed *or* cache
> > > maintenance is performed accordingly). I don't think the arm64 kernel
> > > memory map needs to be rewritten.
> > 
> > That would suggest that having an uncached userland mapping in QEMU
> > and an uncached kernel mapping in the guest would be ok as long as we
> > don't access the host kernel's cacheable alias?
> 
> Yes, from an architecture perspective. Many framebuffer drivers already
> work in a similar way and map the framebuffer memory in user as
> non-cacheable. Of course, one difference is that the other agent
> accessing the memory is a DMA device rather than the CPU.
> 
> > In that case, Drew's approach would be feasible, and the
> > pci_register_bar() function in QEMU could be modified to force the
> > userland mapping and the stage2 mapping to 'device' [when running
> > under KVM/ARM] if it refers to a memslot that is backed by host
> > memory.
> 
> Device or normal non-cacheable (depending on the unaligned access
> requirements).
> 
> Since such memory is allocated by Qemu (rather than a kernel driver),
> KVM would need to mark the pages as reserved so that they are not moved
> around by the host kernel, especially since it would use the cacheable
> alias.
> 
> Another issue is taking care of the host kernel merging adjacent vmas
> since we only want to apply the attributes to a single region.

I also experimented with dropping the KVM memslot flag in favor of an
madvise flag, allowing us to avoid vma merging problems. I forget if
I dropped that for any other reason than I thought it would generate
too much hate mail... Or maybe it was because I ended up needing to
add a new mprotect flag too, which I was quite sure would generate hate
mail, even though I found precedence for it

  ef3d3246a0d0 powerpc/mm: Add Strong Access Ordering support

I have experimental patches (somewhere, they don't seem to be on this
laptop) for that stuff, but I can't recall what was still broken with
them in the end. I just recall it still didn't work, which is why I
never posted even a crazy RFC.

Thanks,
drew