"release early" preview of DCD enabling

"release early" preview of DCD enabling

[Dan Williams]

Here is an early draft of the DCD work that Navneet has spearheaded.
Given the interest level I thought it best to do the remaining
development and refinement of the functionality in the open.  For now
this is just a git branch that will become a patchkit on the mailing
list in a few weeks.

Questions are welcome, but I do not recommend a formal review until the
patches hit the mailing list.

This has had some checkout on an internal functional model, I expect
QEMU and/or cxl_test grows an emulation of DCD before this patchkit is
committed for mainline.

https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/log/?h=for-6.5/dcd-preview

Re: "release early" preview of DCD enabling

[Gregory Price]

On Sun, Apr 09, 2023 at 12:07:36AM -0700, Dan Williams wrote:
> Here is an early draft of the DCD work that Navneet has spearheaded.
> Given the interest level I thought it best to do the remaining
> development and refinement of the functionality in the open.  For now
> this is just a git branch that will become a patchkit on the mailing
> list in a few weeks.
> 
> Questions are welcome, but I do not recommend a formal review until the
> patches hit the mailing list.
> 
> This has had some checkout on an internal functional model, I expect
> QEMU and/or cxl_test grows an emulation of DCD before this patchkit is
> committed for mainline.
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/log/?h=for-6.5/dcd-preview

Awesome!

for the QEMU folks, has anyone be actively working on a DCD device? I
was going to start looking at it, but didn't want to duplicate work.

~Gregory

Re: "release early" preview of DCD enabling

[Jonathan Cameron]

On Tue, 11 Apr 2023 13:14:56 -0400
Gregory Price <gregory.price@memverge.com> wrote:

> On Sun, Apr 09, 2023 at 12:07:36AM -0700, Dan Williams wrote:
> > Here is an early draft of the DCD work that Navneet has spearheaded.
> > Given the interest level I thought it best to do the remaining
> > development and refinement of the functionality in the open.  For now
> > this is just a git branch that will become a patchkit on the mailing
> > list in a few weeks.
> > 
> > Questions are welcome, but I do not recommend a formal review until the
> > patches hit the mailing list.
> > 
> > This has had some checkout on an internal functional model, I expect
> > QEMU and/or cxl_test grows an emulation of DCD before this patchkit is
> > committed for mainline.
> > 
> > https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/log/?h=for-6.5/dcd-preview  
> 
> Awesome!
> 
> for the QEMU folks, has anyone be actively working on a DCD device? I
> was going to start looking at it, but didn't want to duplicate work.

Not that I know of.  Was thinking I might hack some of it together, but
not gotten to it yet. 

For and end goal I think we want:

a) Add all the config stuff for DCD regions - perhaps hardcode one initially.
b) Device that generates DCD events (don't care on interface to make them happen but maybe start
   with QMP similar to injection interfaces).
c) Device that maps the DCD extents through to doing the correct read and write behavior for
   reads that aren't in DCD extents.
d) A standards based way to poke the FM-API. Tunneling and / or MCTP CCIs


Can probably split this up though so if multiple people are hacking on it they don't clash.

(a) and (b) needed to do anything useful with kernel side.
(c) needed for testing long term, but don't care initially.

(d) mostly separable - needed for a 'nice' test setup but not for basic functionality testing.

If you want to take a look at (a) and (b) and maybe (c) that would be great.

There is lots of independent stuff to do for (d) so maybe I'll focus on that in short term?

How's that work for you?

Jonathan


> 
> ~Gregory

Re: "release early" preview of DCD enabling

[Fan Ni]

On Fri, Apr 14, 2023 at 01:04:14PM +0100, Jonathan Cameron wrote:

> On Tue, 11 Apr 2023 13:14:56 -0400
> Gregory Price <gregory.price@memverge.com> wrote:
> 
> > On Sun, Apr 09, 2023 at 12:07:36AM -0700, Dan Williams wrote:
> > > Here is an early draft of the DCD work that Navneet has spearheaded.
> > > Given the interest level I thought it best to do the remaining
> > > development and refinement of the functionality in the open.  For now
> > > this is just a git branch that will become a patchkit on the mailing
> > > list in a few weeks.
> > > 
> > > Questions are welcome, but I do not recommend a formal review until the
> > > patches hit the mailing list.
> > > 
> > > This has had some checkout on an internal functional model, I expect
> > > QEMU and/or cxl_test grows an emulation of DCD before this patchkit is
> > > committed for mainline.
> > > 
> > > https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/log/?h=for-6.5/dcd-preview  
> > 
> > Awesome!
> > 
> > for the QEMU folks, has anyone be actively working on a DCD device? I
> > was going to start looking at it, but didn't want to duplicate work.
> 
> Not that I know of.  Was thinking I might hack some of it together, but
> not gotten to it yet. 
> 
> For and end goal I think we want:
> 
> a) Add all the config stuff for DCD regions - perhaps hardcode one initially.
> b) Device that generates DCD events (don't care on interface to make them happen but maybe start
>    with QMP similar to injection interfaces).
> c) Device that maps the DCD extents through to doing the correct read and write behavior for
>    reads that aren't in DCD extents.
> d) A standards based way to poke the FM-API. Tunneling and / or MCTP CCIs
> 
> 
> Can probably split this up though so if multiple people are hacking on it they don't clash.
> 
> (a) and (b) needed to do anything useful with kernel side.
> (c) needed for testing long term, but don't care initially.
> 
> (d) mostly separable - needed for a 'nice' test setup but not for basic functionality testing.
> 
> If you want to take a look at (a) and (b) and maybe (c) that would be great.
> 
> There is lots of independent stuff to do for (d) so maybe I'll focus on that in short term?
> 
> How's that work for you?
> 
> Jonathan
> 

Hi Jonathan and Gregory,
I have some initial code for DCD support for qemu emulation. It
mainly includes following
1. DCD mailbox command support at qemu side;
2. QMP interface to support DCD event like adding/removing dynamic
capacity
3. DCD Device representative for Qemu emulation (based on
cxl_type3.c)

The code is only draft now and not completed or tested with early
enabling code posted by Dan in the thread.
If you are interested, feel free to use it as base and carry it on.
I will also continue to work on it.
My next step would be to check the kernel side DCD code and
try to fix anything missing/incorrect at qemu side, and then see if we can do
some very simple DCD test with both kernel and qemu support.

Here is the code:
https://github.com/moking/qemu-dev/tree/dcd

Fan

> 
> > 
> > ~Gregory
> 
> 

Re: "release early" preview of DCD enabling

[nifan]

Since my last post, I updated the code, and it can do some simple
end-to-end dcd test with kernel dcd support.

See below for more details.

The 04/17/2023 16:41, Fan Ni wrote:
> On Fri, Apr 14, 2023 at 01:04:14PM +0100, Jonathan Cameron wrote:
> 
> > On Tue, 11 Apr 2023 13:14:56 -0400
> > Gregory Price <gregory.price@memverge.com> wrote:
> > 
> > > On Sun, Apr 09, 2023 at 12:07:36AM -0700, Dan Williams wrote:
> > > > Here is an early draft of the DCD work that Navneet has spearheaded.
> > > > Given the interest level I thought it best to do the remaining
> > > > development and refinement of the functionality in the open.  For now
> > > > this is just a git branch that will become a patchkit on the mailing
> > > > list in a few weeks.
> > > > 
> > > > Questions are welcome, but I do not recommend a formal review until the
> > > > patches hit the mailing list.
> > > > 
> > > > This has had some checkout on an internal functional model, I expect
> > > > QEMU and/or cxl_test grows an emulation of DCD before this patchkit is
> > > > committed for mainline.
> > > > 
> > > > https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/log/?h=for-6.5/dcd-preview  
> > > 
> > > Awesome!
> > > 
> > > for the QEMU folks, has anyone be actively working on a DCD device? I
> > > was going to start looking at it, but didn't want to duplicate work.
> > 
> > Not that I know of.  Was thinking I might hack some of it together, but
> > not gotten to it yet. 
> > 
> > For and end goal I think we want:
> > 
> > a) Add all the config stuff for DCD regions - perhaps hardcode one initially.
This subtask is there in my code.
> > b) Device that generates DCD events (don't care on interface to make them happen but maybe start
> >    with QMP similar to injection interfaces).
Two QMP interfaces are added to add/release dc extents. At the this
time, we keep the interface as simple as possible to do the work.
> > c) Device that maps the DCD extents through to doing the correct read and write behavior for
> >    reads that aren't in DCD extents.
Not implemented yet. 
> > d) A standards based way to poke the FM-API. Tunneling and / or MCTP CCIs
Not touched yet. 
> > 
> > 
> > Can probably split this up though so if multiple people are hacking on it they don't clash.
> > 
> > (a) and (b) needed to do anything useful with kernel side.
> > (c) needed for testing long term, but don't care initially.
> > 
> > (d) mostly separable - needed for a 'nice' test setup but not for basic functionality testing.
> > 
> > If you want to take a look at (a) and (b) and maybe (c) that would be great.
> > 
> > There is lots of independent stuff to do for (d) so maybe I'll focus on that in short term?
> > 
> > How's that work for you?
> > 
> > Jonathan
> > 
> 
> Hi Jonathan and Gregory,
> I have some initial code for DCD support for qemu emulation. It
> mainly includes following
> 1. DCD mailbox command support at qemu side;
> 2. QMP interface to support DCD event like adding/removing dynamic
> capacity
> 3. DCD Device representative for Qemu emulation (based on
> cxl_type3.c)
> 
> The code is only draft now and not completed or tested with early
> enabling code posted by Dan in the thread.
> If you are interested, feel free to use it as base and carry it on.
> I will also continue to work on it.
> My next step would be to check the kernel side DCD code and
> try to fix anything missing/incorrect at qemu side, and then see if we can do
> some very simple DCD test with both kernel and qemu support.
> 
> Here is the code:
> https://github.com/moking/qemu-dev/tree/dcd
> 
> Fan

Initially I prototyped the dcd device as a new device class and found it
may be not a good choice and will need a lot of code change at qemu
side. Instead, in the new version I extended type3 device implemention with
dc capability. Now it is able to do some simple test together with kernel
side dcd support. 

Here is the qemu code branch for the new version:  
https://github.com/moking/qemu-dev/tree/dcd-v2

The code is not clean at all and not ready for review. It is for people who
are interested and want to have a early try of dcd support at kernel side or
who want to use it as base to carry the dcd feature on.

What we can do with the code together with kernel dcd code:
1. load the cxl modules touching identify memory device code change at
kernel; 
2. touch the code path of the mailbox command to get dc region
info (spec 3.0, 8.2.9.8.9) on both kernel and qemu side;
3. touch the code path of the mailbox command to get dc extent list info
4. touch the dc region related code to create dc region
5. use qmp interface to add one dc exent and the kernel was able to
create dax device for the extent, and the memory can show as offline RAM.
And the extent can be released with the qmp dc extent release interface.
Re-add a released dc extent has issue.
6. NOTE: real read/write access to the dynamic capacity is not supported yet.

What operations have been tested:
1. install the cxl modules 
	modprobe -a cxl_acpi cxl_core cxl_pci cxl_port cxl_mem

2. create dc regions:

region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region)
echo $region> /sys/bus/cxl/devices/decoder0.0/create_dc_region
echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity
echo 1 > /sys/bus/cxl/devices/$region/interleave_ways

echo "dc" >/sys/bus/cxl/devices/decoder2.0/mode
echo 0x10000000 >/sys/bus/cxl/devices/decoder2.0/dpa_size

echo 0x10000000 > /sys/bus/cxl/devices/$region/size
echo  "decoder2.0" > /sys/bus/cxl/devices/$region/target0
echo 1 > /sys/bus/cxl/devices/$region/commit
echo $region > /sys/bus/cxl/drivers/cxl_region/bind

root@bgt-140510-bm03:/home/fan/cxl/tools-and-scripts# cxl list
[
  {
    "memdevs":[
      {
        "memdev":"mem0",
        "pmem_size":536870912,
        "ram_size":0,
        "serial":0,
        "host":"0000:0d:00.0"
      }
    ]
  },
  {
    "regions":[
      {
        "region":"region0",
        "resource":45365592064,
        "size":268435456,
        "interleave_ways":1,
        "interleave_granularity":256,
        "decode_state":"commit"
      }
    ]
  }
]


3. add one dc extent (128MB) through qmp interface

{ "execute": "qmp_capabilities" }                                               
                                                                                
{ "execute": "cxl-add-dynamic-capacity-event",                                  
    "arguments": {                                                              
         "path": "/machine/peripheral/cxl-pmem0",                               
         "num-extent": 1                                                        
    }                                                                           
}                  

root@bgt-140510-bm03:/home/fan/cxl/tools-and-scripts# lsmem 
RANGE                                  SIZE   STATE REMOVABLE BLOCK
0x0000000000000000-0x000000007fffffff    2G  online       yes  0-15
0x0000000100000000-0x000000027fffffff    6G  online       yes 32-79
0x0000000a90000000-0x0000000a97ffffff  128M offline             338

Memory block size:       128M
Total online memory:       8G
Total offline memory:    128M


4. release the dc extent (128MB) added through qmp interface

{ "execute": "cxl-release-dynamic-capacity-event",                                                                                                                                                                                                            
    "arguments": {                                                              
         "path": "/machine/peripheral/cxl-pmem0",                               
         "num-extent": 1                                                        
    }                                                                           
}                                                                               
~                     
root@bgt-140510-bm03:/home/fan/cxl/tools-and-scripts# lsmem 
RANGE                                 SIZE  STATE REMOVABLE BLOCK
0x0000000000000000-0x000000007fffffff   2G online       yes  0-15
0x0000000100000000-0x000000027fffffff   6G online       yes 32-79

Memory block size:       128M
Total online memory:       8G
Total offline memory:      0B



QEMU command cxl configuration:

RP1="-object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=512M \
-object memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/cxltest.raw,size=512M \
-object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=512M \
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \                            
-device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \          
-device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,dc-memdev=cxl-mem2,id=cxl-pmem0,num-dc-regions=1\
-M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=8k"


The code is tested with the posted kernel dcd support:
https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/log/?h=for-6.5/dcd-preview

To make the test work, we have made the following changes to the following
commit of above kernel code branch:

-----------------------------------------------------------------------------
commit f425bc34c600e2a3721d6560202962ec41622815 (origin/for-6.5/dcd-preview)
Author: Navneet Singh <navneet.singh@intel.com>
Date:   Mon Mar 6 08:45:59 2023 -0500

    cxl/mem: Trace Dynamic capacity Event Record
-----------------------------------------------------------------------------



diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index 5f04bbc18af5..5f421d3c5cef 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -68,6 +68,7 @@ static struct cxl_mem_command cxl_mem_commands[CXL_MEM_COMMAND_ID_MAX] = {
 	CXL_CMD(SCAN_MEDIA, 0x11, 0, 0),
 	CXL_CMD(GET_SCAN_MEDIA, 0, CXL_VARIABLE_PAYLOAD, 0),
 	CXL_CMD(GET_DC_EXTENT_LIST, 0x8, CXL_VARIABLE_PAYLOAD, 0),
+	CXL_CMD(GET_DC_CONFIG, 0x2, CXL_VARIABLE_PAYLOAD, 0),
 };
 
 /*
diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
index 291c716abd49..ae10e3cf43a1 100644
--- a/drivers/cxl/core/region.c
+++ b/drivers/cxl/core/region.c
@@ -194,7 +194,7 @@ static int cxl_region_manage_dc(struct cxl_region *cxlr)
 		}
 		cxlds->dc_list_gen_num = extent_gen_num;
 		dev_dbg(cxlds->dev, "No of preallocated extents :%d\n", rc);
-		enable_irq(cxlds->cxl_irq[CXL_EVENT_TYPE_DCD]);
+		/*enable_irq(cxlds->cxl_irq[CXL_EVENT_TYPE_DCD]);*/
 	}
 	return 0;
 err:
@@ -2810,7 +2810,8 @@ int cxl_add_dc_extent(struct cxl_dev_state *cxlds, struct resource *alloc_dpa_re
 				dev_dax->align, memremap_compat_align()))) {
 		rc = alloc_dev_dax_range(dev_dax, hpa,
 					resource_size(alloc_dpa_res));
-		return rc;
+		if (rc)
+			return rc;
 	}
 
 	rc = xa_insert(&cxlr_dc->dax_dev_list, hpa, dev_dax, GFP_KERNEL);
diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
index 9e45b1056022..653bec203838 100644
--- a/drivers/cxl/pci.c
+++ b/drivers/cxl/pci.c
@@ -659,7 +659,7 @@ static int cxl_event_irqsetup(struct cxl_dev_state *cxlds)
 
 	/* Driver enables DCD interrupt after creating the dc cxl_region */
 	rc = cxl_event_req_irq(cxlds, policy.dyncap_settings, CXL_EVENT_TYPE_DCD,
-					IRQF_SHARED | IRQF_ONESHOT | IRQF_NO_AUTOEN);
+					IRQF_SHARED | IRQF_ONESHOT);
 	if (rc) {
 		dev_err(cxlds->dev, "Failed to get interrupt for event dc log\n");
 		return rc;
diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
index 6ca85861750c..910a48259239 100644
--- a/include/uapi/linux/cxl_mem.h
+++ b/include/uapi/linux/cxl_mem.h
@@ -47,6 +47,7 @@
 	___C(SCAN_MEDIA, "Scan Media"),                                   \
 	___C(GET_SCAN_MEDIA, "Get Scan Media Results"),                   \
 	___C(GET_DC_EXTENT_LIST, "Get dynamic capacity extents"),         \
+	___C(GET_DC_CONFIG, "Get dynamic capacity configuration"),         \
 	___C(MAX, "invalid / last command")
 
 #define ___C(a, b) CXL_MEM_COMMAND_ID_##a

> 
> > 
> > > 
> > > ~Gregory
> > 
> > 

-- 
Fan Ni