Re: [RFC PATCH 00/15] Region driver

[Dan Williams <dan.j.williams@intel.com>]

Dan Williams wrote:
> Jonathan Cameron wrote:
> > On Wed, 22 Jun 2022 22:40:48 -0700
> > Dan Williams <dan.j.williams@intel.com> wrote:
> > 
> > > Jonathan Cameron wrote:
> > > > ....
> > > >   
> > > > > > Hi Ben,
> > > > > >
> > > > > > I finally got around to actually trying this out on top of Dan's recent fix set
> > > > > > (I rebased it from the cxl/preview branch on kernel.org).
> > > > > >
> > > > > > I'm not having much luck actually bring up a region.
> > > > > >
> > > > > > The patch set refers to configuring the end point decoders, but all their
> > > > > > sysfs attributes are read only.  Am I missing a dependency somewhere or
> > > > > > is the intent that this series is part of the solution only?
> > > > > >
> > > > > > I'm confused!    
> > > > > 
> > > > > There's a new series that's being reviewed internally before going to the list:
> > > > > 
> > > > > https://gitlab.com/bwidawsk/linux/-/tree/cxl_region-redux3
> > > > > 
> > > > > Given the proximity to the merge window opening and the need to get
> > > > > the "mem_enabled" series staged, I asked Ben to hold it back from the
> > > > > list for now.
> > > > > 
> > > > > There are some changes I am folding into it, but I hope to send it out
> > > > > in the next few days after "mem_enabled" is finalized.  
> > > > 
> > > > Hi Dan,
> > > > 
> > > > I switched from an earlier version of the region code over to a rebase of the tree.
> > > > Two issues below you may already have fixed.
> > > > 
> > > > The second is a carry over from an earlier set so I haven't tested
> > > > without it but looks like it's still valid.
> > > > 
> > > > Anyhow, thought it might save some cycles to preempt you sending
> > > > out the series if these issues are still present.
> > > > 
> > > > Minimal testing so far on these with 2 hb, 2 rp, 4 directly connected
> > > > devices, but once you post I'll test more extensively.  I've not
> > > > really thought about the below much, so might not be best way to fix.
> > > > 
> > > > Found a bug in QEMU code as well (missing write masks for the
> > > > target list registers) - will post fix for that shortly.  
> > > 
> > > Hi Jonathan,
> > > 
> > > Tomorrow I'll post the tranche to the list, but wanted to let you and
> > > others watching that that the 'preview' branch [1] now has the proposed
> > > initial region support. Once the bots give the thumbs up I'll send it
> > > along.
> > > 
> > > To date I've only tested it with cxl_test and an internal test vehicle.
> > > The cxl_test script I used to setup and teardown a x8 interleave across
> > > x2 host bridges and x4 switches is:
> > 
> > Thanks.  Trivial feedback from a very quick play (busy day).
> > 
> > Bit odd that regionX/size is once write - get an error even if
> > writing same value to it twice.
> 
> Ah true, that should just silently succeed.

I fixed this one.

> 
> > Also not debugged yet but on just got a null pointer dereference on
> > 
> > echo decoder3.0 > target0
> > 
> > Beyond a stacktrace pointing at store_targetN and dereference is of
> > 0x00008 no idea yet.
> 
> The compiler unfortunately does a good job inlining the entirety of all the
> leaf functions beneath store_targetN() so I have found myself needing to
> sprinkle "noinline" to get better back traces.
> 
> > 
> > I was testing with a slightly modified version of a nasty script
> > I was using to test with Ben's code previously.  Might well be
> > doing something wrong but obviously need to fix that crash anyway!
> 
> Most definitely.

I tried to reproduce this one, but unfortunately, "worked for me". So
send along more reproduction details when you get a chance, but I'll
proceed with posting the series for now. I tried the following on my
QEMU config to reproduce:

# cxl reserve-dpa -t pmem mem0
{
  "memdev":"mem0",
  "pmem_size":"512.00 MiB (536.87 MB)",
  "ram_size":0,
  "serial":"0",
  "host":"0000:35:00.0",
  "decoder":{
    "decoder":"decoder2.0",
    "state":"disabled",
    "dpa_size":"512.00 MiB (536.87 MB)",
    "mode":"pmem"
  }
}
# echo region1 > /sys/bus/cxl/devices/decoder0.0/create_pmem_region
# echo 1 > /sys/bus/cxl/devices/region1/interleave_ways 
# echo 256 > /sys/bus/cxl/devices/region1/interleave_granularity 
# echo $((512 << 20)) > /sys/bus/cxl/devices/region1/size
# echo decoder2.0 > /sys/bus/cxl/devices/region1/target0