Re: [PATCH 19/19] tools/testing/cxl: Emulate a CXL accelerator with local memory

From: Jonathan Cameron <Jonathan.Cameron@Huawei.com>
To: Vikram Sethi <vsethi@nvidia.com>
Cc: Dan Williams <dan.j.williams@intel.com>,
	"linux-cxl@vger.kernel.org" <linux-cxl@vger.kernel.org>,
	"ira.weiny@intel.com" <ira.weiny@intel.com>,
	"navneet.singh@intel.com" <navneet.singh@intel.com>
Subject: Re: [PATCH 19/19] tools/testing/cxl: Emulate a CXL accelerator with local memory
Date: Thu, 8 Jun 2023 16:22:03 +0100	[thread overview]
Message-ID: <20230608162203.00003788@Huawei.com> (raw)
In-Reply-To: <BYAPR12MB33364671B3FD0721174A3F49BD50A@BYAPR12MB3336.namprd12.prod.outlook.com>

On Thu, 8 Jun 2023 14:34:48 +0000
Vikram Sethi <vsethi@nvidia.com> wrote:

> > From: Jonathan Cameron <Jonathan.Cameron@Huawei.com>
> > Sent: Thursday, June 8, 2023 5:47 AM
> > To: Vikram Sethi <vsethi@nvidia.com>
> > Cc: Dan Williams <dan.j.williams@intel.com>; linux-cxl@vger.kernel.org;
> > ira.weiny@intel.com; navneet.singh@intel.com
> > Subject: Re: [PATCH 19/19] tools/testing/cxl: Emulate a CXL accelerator with
> > local memory
> > On Wed, 7 Jun 2023 21:09:21 +0000
> > Vikram Sethi <vsethi@nvidia.com> wrote:
> >   
> > > Thanks for posting this Dan.  
> > > > From: Dan Williams <dan.j.williams@intel.com>
> > > > Sent: Sunday, June 4, 2023 6:33 PM
> > > > To: linux-cxl@vger.kernel.org
> > > > Cc: ira.weiny@intel.com; navneet.singh@intel.com
> > > > Subject: [PATCH 19/19] tools/testing/cxl: Emulate a CXL accelerator
> > > > with local memory
> > > >
> > > > Mock-up a device that does not have a standard mailbox, i.e. a
> > > > device that does not implement the CXL memory-device class code, but
> > > > wants to map "device" memory (aka Type-2, aka HDM-D[B], aka  
> > accelerator memory).  
> > > >
> > > > +
> > > > +       cxlrd = cxl_hpa_freespace(endpoint, &endpoint->host_bridge, 1,
> > > > +                                 CXL_DECODER_F_RAM | CXL_DECODER_F_TYPE2,
> > > > +                                 &max);
> > > > +  
> > >  
> >   
> > > IIUC, finding free HPA space is for the case where the platform FW has
> > > not already allocated it and initialized the HDM ranges in the device
> > > decoders, correct? If the accelerator driver recognized that FW had
> > > initialized its HPA ranges in the device decoders (without
> > > committing/locking the decoders), could it skip the cxl_hpa_freespace
> > > call? It would seem reasonable for FW to init the decoder but not
> > > commit/lock it.  
> > 
> > I'd find it a bit odd if firmware did a partial job...
> > Why do you think it might?  To pass a hint to the kernel?
> >   

> Firmware could certainly initialize, commit, and lock the decoder for
> accelerators that are soldered on the motherboard. I just wasn't sure
> if the CXL core code could deal with a committed and locked decoder. 

It can for type 3 devices, I haven't checked it still applies for
this type 2 code.

> I was also thinking about chiplets within a package with new
> specifications like UCIe where it is possible that chip designers
> assigned a fixed HPA range in the chip address map to a CXL device
> chiplet's HDM. Would it be sufficient for FW to convey this by
> committing and locking the decoders, or would we need some new ACPI
> flags telling the kernel that this device's decoders can really
> decode a fixed HPA range and not to change the fixed values? A
> similar notion exists in PCIe of fixed BARs called enhanced
> allocation with hardwired addresses.

If they are hard wired then FW should just lock them I think.

Jonathan