Re: [GIT preview] for-6.3/cxl-ram-region

From: Dan Williams <dan.j.williams@intel.com>
To: Gregory Price <gregory.price@memverge.com>,
	Jonathan Cameron <Jonathan.Cameron@huawei.com>
Cc: Fan Ni <fan.ni@samsung.com>,
	"Verma, Vishal L" <vishal.l.verma@intel.com>,
	"Williams, Dan J" <dan.j.williams@intel.com>,
	"linux-cxl@vger.kernel.org" <linux-cxl@vger.kernel.org>,
	Adam Manzanares <a.manzanares@samsung.com>,
	"dave@stgolabs.net" <dave@stgolabs.net>
Subject: Re: [GIT preview] for-6.3/cxl-ram-region
Date: Thu, 2 Feb 2023 10:18:33 -0800	[thread overview]
Message-ID: <63dbfe79dbb44_ea22229468@dwillia2-xfh.jf.intel.com.notmuch> (raw)
In-Reply-To: <Y9riPw7ccA34FQG6@memverge.com>

Gregory Price wrote:
[..]
> Some speculation here:
> 
> The crux of the issue, as i understand it, is the invalidation path.
> MMIO doesn't traditionally have a mechanism to tell the caches "hey
> i got updated, boot this cache line", so whenever your compiler accesses
> MMIO it - at best - does a fetch-and-discard, meaning that instruction
> translations can never be cached.  That's the source of the slow down on
> the QEMU side, you're constantly re-compiling the translations.
> 
> On the KVM side, it likely requires a VMExit to handle the MMIO, and
> when it sees that it's an instruction fetch it probably just falls back
> to emulator mode to execute the instruction before re-entering.  Maybe
> there's a mild optimization where it continues executing until it leaves
> that MMIO region, but you're still getting QEMU performance over KVM.
> 
> So that all makes sense to me.
> 
> To me, the solution here isn't to change QEMU, it's to change the kernel
> to try to get it to aggressively keep executable regions out of CXL by
> marking CXL regions into a new zone type that essentially says "Use this
> as a last resort only for X pages".  But that would likely require
> adding migration code to the likes of mprotect and friends.

No, this can't be the path forward as far as I can see. QEMU is a test
vehicle for CXL enabling, there's no expectation that QEMU is running
CXL emulation in production. The quirks of how the QEMU-CXL memory
behaves are not something the kernel should worry about mitigating. CXL
is "System RAM" especially in the case when it is mapped by
platform-firmware. If it's not suitable to be treated as "System RAM"
then the onus is on platform-firmware to keep it out of the general
purpose pool.

> In the meantime, sure would be nice to have a userland program that
> grooms software to detect this problem and migrate X pages to DRAM.

As far as software is concerned it may not even be able to discern that
DRAM is DDR vs CXL attached. Something is wrong if that distinction
matters in practice outside of NUMA policy or dedicated device access.