CXL 2.0 memory device design

From: Ben Widawsky <ben@bwidawsk.net>
To: "Philippe Mathieu-Daudé" <f4bug@amsat.org>,
	"Markus Armbruster" <armbru@redhat.com>,
	"Igor Mammedov" <imammedo@redhat.com>,
	qemu-devel@nongnu.org
Cc: Dan Williams <dan.j.williams@intel.com>,
	Jonathan Cameron <Jonathan.Cameron@huawei.com>
Subject: CXL 2.0 memory device design
Date: Wed, 17 Mar 2021 14:40:58 -0700	[thread overview]
Message-ID: <20210317214045.4xrwlhfvyczhxvc5@mail.bwidawsk.net> (raw)

Phil, Igor, Markus

TL;DR: What to do about multiple capacities in a single device, and what to do
about interleave?

I've hacked together a basic CXL 2.0 implementation which exposes a CXL "Type 3"
memory device (CXL 2.0 Chapter 2.3). For what we were trying to do this was
sufficient. There are two main capabilities that CXL spec exposes which I've not
implemented that I'd like to start working toward and am realizing that I what I
have so far might not be able to carry forward to that next milestone.

Capability 1. A CXL memory device may have both a volatile, and a persistent
	      capacity. https://bwidawsk.net/HDM_decoders.svg (lower right
	      side). The current work only supports a single persistent
	      capacity.
Capability 2. CXL topologies can be interleaved. Basic example:
              https://bwidawsk.net/HDM_decoders.svg (lower left side)

Memory regions are configured via a CXL spec defined HDM decoder. The HDM
decoder which is minimally implemented supports all the functionality mentioned
above (base, size, interleave, type, etc.). A CXL component may have up to 10
HDMs.

What I have today: https://bwidawsk.net/QEMU_objects.svg
There's a single memory backend device for each host bridge. That backend is
passed to any CXL component that is part of the hierarchy underneath that
hostbridge. In the case of a Type 3 device memory capacity a subregion is
created for that capacity from within the main backend. The device itself
implements the TYPE_MEMORY_DEVICE interface. This allows me to utilize the
existing memory region code to determine the next free address, and warn on
overlaps. It hopefully will help when I'm ready to support hotplug.

Where I've gotten stuck: A Memory Device expects only to have one region of
memory. Trying to add a second breaks pretty much everything.

I'm hoping to start the discussion about what the right way to emulate this in
QEMU. Ideally something upstreamable would be great. I think adding a secondary
(or more) capacity to a memory class device is doable, but probably not the
right approach.

For context, I've posted v3 previously. Here's a link to v4 which has some minor
changes as well as moving back to using subregions instead of aliases:
https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0v4

Thanks.
Ben