Re: [RFC 1/2] vfio/pci: keep the prefetchable attribute of a BAR region in VMA

From: Marc Zyngier <maz@kernel.org>
To: Shanker R Donthineni <sdonthineni@nvidia.com>
Cc: Vikram Sethi <vsethi@nvidia.com>,
	Alex Williamson <alex.williamson@redhat.com>,
	Will Deacon <will@kernel.org>,
	Catalin Marinas <catalin.marinas@arm.com>,
	"Christoffer\ Dall" <christoffer.dall@arm.com>,
	"linux-arm-kernel@lists.infradead.org"
	<linux-arm-kernel@lists.infradead.org>,
	"kvmarm@lists.cs.columbia.edu" <kvmarm@lists.cs.columbia.edu>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	Jason Sequeira <jsequeira@nvidia.com>
Subject: Re: [RFC 1/2] vfio/pci: keep the prefetchable attribute of a BAR region in VMA
Date: Mon, 03 May 2021 10:50:41 +0100	[thread overview]
Message-ID: <87czu8uowe.wl-maz@kernel.org> (raw)
In-Reply-To: <49e26646-9f05-ccb8-f5c1-73a92ab79972@nvidia.com>

On Sat, 01 May 2021 12:36:11 +0100,
Shanker R Donthineni <sdonthineni@nvidia.com> wrote:
> 
> Hi Marc,
> 
> On 5/1/21 4:30 AM, Marc Zyngier wrote:
> >> I think Device GRE has some practical problems.
> >> 1. A lot of userspace code which is used to getting write combined
> >> mappings to GPU memory from kernel drivers does memcpy/memset on it
> >> which can insert ldp/stp which can crash on Device Memory Type. From
> >> a quick search I didn't find a memcpy_io or memset_io in
> >> glibc. Perhaps there are some other functions available, but a lot
> >> of userspace applications that work on x86 and ARM baremetal won't
> >> work on ARM VMs without such changes. Changes to all of userspace
> >> may not always be practical, specially if linking to binaries
> > This seems to go against what Alex was hinting at earlier, which is
> > that unaligned accesses were not expected on prefetchable regions, and
> > Shanker latter confirming that it was an actual bug. Where do we stand
> > here?
> >
> We agreed to call it a driver bug if it's not following Linux
> write-combining API ioremap_wc() semantics. So far I didn't find
> whether unaligned accesses allowed or not for WC regions explicitly
> in Linux documentation.

And that's exactly the kind of problem I want clarification on before
we add *anything* to KVM. Proper, unambiguous definition of what WC is
on the CPU side, and how it maps onto PCI. Without such a definition,
we're just driving blind.

> Page faults due to driver unaligned accesses
> in kernel space will be under driver control, we'll fix it.
> 
> Driver uses the architecture agnostic functions that are available
> in the Linux kernel and expecting the same behavior in VM vs
> Baremetal. We would like to keep the driver implementation is
> architecture-independent as much as possible and support VM
> unaware. For ARM64, VM's ioremap_wc() definition doesn't match
> baremetal.

You are mixing two things: what Linux/arm64 gives to kernel drivers,
and what KVM, as an implementation of the ARMv8 architecture, gives to
virtual machines. There is zero reason for the two to match if there
is no definition of what we need to provide.

> We don't have any control over the userspace
> applications/drivers/libraries as Vikram saying. Another example GCC
> memset() function uses 'DC ZVA' which triggers an alignment fault if
> the actual memory type is device_xxx.

Again, you're talking about an application, and I'm talking about how
to map a nebulous concept that originated on a foreign architecture
onto something that is entirely different. So please drop the "that's
how my SW works", and instead give me a good definition of what WC
means in architectural terms.

Thanks

	M.

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel