Re: [PATCH 00/15] cxl: Add support for Restricted CXL hosts (RCD mode)

[Dan Williams <dan.j.williams@intel.com>]

Jonathan Zhang (Infra) wrote:
> 
> 
> > On Sep 8, 2022, at 12:51 PM, Dan Williams <dan.j.williams@intel.com> wrote:
> > 
> > !-------------------------------------------------------------------|
> >  This Message Is From an External Sender
> > 
> > |-------------------------------------------------------------------!
> > 
> > Jonathan Zhang (Infra) wrote:
> >> 
> >> 
> >>> On Sep 7, 2022, at 10:43 PM, Dan Williams <dan.j.williams@intel.com> wrote:
> >>> 
> >>> Apologies for the delay in getting to this I had hoped to be able to
> >>> finish up some other DAX work to focus on this, but time is getting
> >>> short so I will need to do both in parallel.
> >>> 
> >>> Robert Richter wrote:
> >>>> In Restricted CXL Device (RCD) mode (formerly referred to as CXL 1.1)
> >>>> the PCIe enumeration hierarchy is different from CXL VH Enumeration
> >>>> (formerly referred to as 2.0, for both modes see CXL spec 3.0: 9.11
> >>>> and 9.12, [1]). This series adds support for RCD mode. It implements
> >>>> the detection of Restricted CXL Hosts (RCHs) and its corresponding
> >>>> Restricted CXL Devices (RCDs). It does the necessary enumeration of
> >>>> ports and connects the endpoints. With all the plumbing an RCH/RCD
> >>>> pair is registered at the Linux CXL bus and becomes visible in sysfs
> >>>> in the same way as CXL VH hosts and devices do already. RCDs are
> >>>> brought up as CXL endpoints and bound to subsequent drivers such as
> >>>> cxl_mem.
> >>>> 
> >>>> For CXL VH the host driver (cxl_acpi) starts host bridge discovery
> >>>> once the ACPI0017 CXL root device is detected and then searches for
> >>>> ACPI0016 host bridges to enable CXL. In RCD mode an ACPI0017 device
> >>>> might not necessarily exist 
> >>> 
> >>> That's a broken BIOS as far as I can see. No ACPI0017 == no OS CXL
> >>> services and the CXL aspects of the device need to be 100% managed by
> >>> the BIOS. You can still run the cxl_pci driver in that case for mailbox
> >>> operation, but error handling must be firmware-first without ACPI0017.
> >> Firmware-first or OS-first applies to CXL protocol error handling. For CXL 
> >> memory error handling, the device generates a DRAM error record, the OS
> >> parses such record and act accordingly. According to CXL spec (section
> >> 8.2.9.2.1.2 DRAM Event Record), DPA but not HPA is in such record. The OS
> >> needs to translate such DPA into HPA to act on. I am taking this as an example
> >> to show that OS CXL services is needed.
> >> Instead of using ACPI0016 to tell whether the system is under RCH mode,
> >> I suppose one way is to check “CXL version” field of CHBS structure in CEDT?
> > 
> > Unless the OS has negotiated CXL _OSC the BIOS owns the event retrieval
> > and translating it from DPA to HPA. I do want to add OS CXL services to
> > Linux, but only in the case when the BIOS is actively enabling OS native
> > address translation which includes populating ACPI0017, CFMWS, and
> > devices with the HDM decoder capability registers instead of DVSEC range
> > registers. Everything else is early-gen CXL that is 100% BIOS supported,
> > similar to DDR where a driver is not expected.
> 
> 
> It makes sense that the BIOS and OS need to negotiate CXL _OSC so that OS
> would take care of address translation. That being said, only DVSEC range 
> register (but not HDM decoder capability register) is available when the device is in
> RCRB mode (section 9.11.8 figure 9-7) attached to a RCH. This type of
> configuration needs to be supported with OS CXL service.
> 

So that figure does have the HDM capabilty pictured in the RCD upstream
port. However, Table 8-22 does seem to incidate that Type 3 D1 devices
are not permitted to have an HDM Decoder Capabilitiy Structure.

However that then leave me confused about figure 9-8 as that shows an
HDM decoder capability in the BAR and not the RCRB. Is that picture
wrong with respect what Table 8-22 indicates?