[PATCH 0/3] CXL: Add dynamic capacity device test support in kdevops

[PATCH 0/3] CXL: Add dynamic capacity device test support in kdevops

[fan.ni]

From: Fan Ni <fan.ni@gmx.us>

This patch series include 3 patches for enabling dynamic capacity device (DCD)
test support in kdevops.

Patch 1: Add a DCD device topology for users to select for DCD emulation.
Patch 2: Add options to setup QMP interface for Qemu, through which users
can send control commands to Qemu instance. Currently, adding and releasing
dc extents to dc regions are implemented through QMP interface.
Patch 3: Add a new make command "cxl-create-dc-region" to create a dc region
so later we can add/release dc extents to the region through QMP interface.

Fan Ni (3):
  CXL: Add a new topology to support DCD emulation
  qemu: Add QMP connection option for QMP interface access
  CXL: Add support to create a DC region for DCD

 .../cxl/tasks/cxl-create-dc-region/main.yml   | 27 +++++++++++++++++++
 playbooks/roles/cxl/tasks/main.yml            |  4 +++
 playbooks/roles/gen_nodes/defaults/main.yml   |  4 +++
 .../roles/gen_nodes/templates/Vagrantfile.j2  | 19 ++++++++++---
 scripts/gen-nodes.Makefile                    | 12 +++++++++
 vagrant/Kconfig                               | 24 +++++++++++++++++
 workflows/cxl/Makefile                        |  6 +++++
 7 files changed, 93 insertions(+), 3 deletions(-)
 create mode 100644 playbooks/roles/cxl/tasks/cxl-create-dc-region/main.yml

--
2.40.1

[PATCH 1/3] CXL: Add a new topology to support DCD emulation

[fan.ni]

From: Fan Ni <fan.ni@gmx.us>

The change adds a new cxl topology to enable dynamic capacity device
emulation in Qemu. The device includes no static volatile/non-volatile
capacity but 2 dynamic capacity regions where dynamic capacity extents
can be added. Also, increase the fix memory window size and host backend size
which gives more room for testing as the dc extents needs to be a least one
memory block (256MB).

Signed-off-by: Fan Ni <fan.ni@samsung.com>
---
 playbooks/roles/gen_nodes/defaults/main.yml        |  1 +
 playbooks/roles/gen_nodes/templates/Vagrantfile.j2 | 11 ++++++++---
 scripts/gen-nodes.Makefile                         |  3 +++
 vagrant/Kconfig                                    |  9 +++++++++
 4 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/playbooks/roles/gen_nodes/defaults/main.yml b/playbooks/roles/gen_nodes/defaults/main.yml
index c57effe7..55827e5b 100644
--- a/playbooks/roles/gen_nodes/defaults/main.yml
+++ b/playbooks/roles/gen_nodes/defaults/main.yml
@@ -51,6 +51,7 @@ libvirt_enable_cxl: False
 libvirt_enable_cxl_demo_topo1: False
 libvirt_enable_cxl_demo_topo2: False
 libvirt_enable_cxl_switch_topo1: False
+libvirt_enable_cxl_dcd_topo1: False
 libvirt_extra_drive_id_prefix: 'drv'
 libvirt_extra_storage_drive_nvme: True
 libvirt_extra_storage_drive_virtio: False
diff --git a/playbooks/roles/gen_nodes/templates/Vagrantfile.j2 b/playbooks/roles/gen_nodes/templates/Vagrantfile.j2
index 7ed59ff7..124cabd6 100644
--- a/playbooks/roles/gen_nodes/templates/Vagrantfile.j2
+++ b/playbooks/roles/gen_nodes/templates/Vagrantfile.j2
@@ -292,7 +292,7 @@ Vagrant.configure("2") do |config|

         cxl_mem1_path = cxl_path + "/cxl-mem1.raw"
         libvirt.qemuargs :value => "-object"
-        libvirt.qemuargs :value => "memory-backend-file,id=kdevops-cxl-mem1,share=on,mem-path=#{cxl_mem1_path},size=256M"
+        libvirt.qemuargs :value => "memory-backend-file,id=kdevops-cxl-mem1,share=on,mem-path=#{cxl_mem1_path},size=4G"

 	# Label Storage Area, used to store CXL namespace labels and region labels
         cxl_lsa1_path = cxl_path + "/cxl-lsa1.raw"
@@ -319,12 +319,17 @@ Vagrant.configure("2") do |config|
 	libvirt.qemuargs :value => "cxl-downstream,port=1,bus=kdevops_cxl_us0,id=kdevops_cxl_swport1,chassis=0,slot=5"
 	libvirt.qemuargs :value => "-device"
 	libvirt.qemuargs :value => "cxl-type3,bus=kdevops_cxl_swport0,memdev=kdevops-cxl-mem1,lsa=kdevops-cxl-lsa1,id=kdevops-cxl-pmem0"
-{% else %} # switch topo1
+{% else %} # !switch topo1
+	{% if not libvirt_enable_cxl_dcd_topo1 %}
         libvirt.qemuargs :value => "-device"
         libvirt.qemuargs :value => "cxl-type3,bus=kdevops_cxl_root_port0,memdev=kdevops-cxl-mem1,lsa=kdevops-cxl-lsa1,id=kdevops-cxl-pmem0"
+	{% else %} # dcd_topo1
+        libvirt.qemuargs :value => "-device"
+        libvirt.qemuargs :value => "cxl-type3,bus=kdevops_cxl_root_port0,nonvolatile-dc-memdev=kdevops-cxl-mem1,lsa=kdevops-cxl-lsa1,id=kdevops-cxl-dcd0,num-dc-regions=2"
+	{% endif %} # dcd topo1
 {% endif %} # !switch topo1
         libvirt.qemuargs :value => "-M"
-        libvirt.qemuargs :value => "cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=512M"
+        libvirt.qemuargs :value => "cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=8G"
 {% endif %} # CXL
 {% endif %}

diff --git a/scripts/gen-nodes.Makefile b/scripts/gen-nodes.Makefile
index 533b19db..eebd0b2b 100644
--- a/scripts/gen-nodes.Makefile
+++ b/scripts/gen-nodes.Makefile
@@ -199,6 +199,9 @@ endif # QEMU_ENABLE_CXL_DEMO_TOPOLOGY_2
 ifeq (y,$(CONFIG_QEMU_ENABLE_CXL_SWITCH_TOPOLOGY_1))
 GEN_NODES_EXTRA_ARGS += libvirt_enable_cxl_switch_topo1='True'
 endif # QEMU_ENABLE_CXL_SWITCH_TOPOLOGY_1
+ifeq (y,$(CONFIG_QEMU_ENABLE_CXL_DEMO_DCD_TOPOLOGY_1))
+GEN_NODES_EXTRA_ARGS += libvirt_enable_cxl_dcd_topo1='True'
+endif # QEMU_ENABLE_CXL_DEMO_DCD_TOPOLOGY_1
 endif # CONFIG_QEMU_ENABLE_CXL

 endif # CONFIG_LIBVIRT_MACHINE_TYPE_Q35
diff --git a/vagrant/Kconfig b/vagrant/Kconfig
index 3bf096fc..4e382264 100644
--- a/vagrant/Kconfig
+++ b/vagrant/Kconfig
@@ -1666,6 +1666,15 @@ config QEMU_ENABLE_CXL_SWITCH_TOPOLOGY_1
 	  is connected to one of the down stream ports. The upstream port
 	  is connected to a root port on the host bridge.

+config QEMU_ENABLE_CXL_DEMO_DCD_TOPOLOGY_1
+	bool "CXL DCD demo directly attached to a single-port HB"
+	help
+	  This topology adds a DCD device in the topology, directly attached to
+	  a Host bridge with only one root port.
+	  The device has zero (volatile or non-volatile) static capacity
+	  and 2 dynamic capacity regions which host dynamic extents  to be
+	  added.
+
 endchoice

 endif # QEMU_ENABLE_CXL
--
2.40.1

[PATCH 2/3] qemu: Add QMP connection option for QMP interface access

[fan.ni]

From: Fan Ni <fan.ni@gmx.us>

In Qemu, QMP interface allows applications to control a QEMU instance.
For CXL emulation, QMP interface has been used to inject poison,
add/release dynamic capacity extents, etc.

With the change, we add a Qemu configuration option to allow user to use
the QMP interface. By default, we start QMP on a TCP socket and let Qemu
instance go through without waiting for QMP connection.

Signed-off-by: Fan Ni <fan.ni@samsung.com>
---
 playbooks/roles/gen_nodes/defaults/main.yml       |  3 +++
 .../roles/gen_nodes/templates/Vagrantfile.j2      |  8 ++++++++
 scripts/gen-nodes.Makefile                        |  9 +++++++++
 vagrant/Kconfig                                   | 15 +++++++++++++++
 4 files changed, 35 insertions(+)

diff --git a/playbooks/roles/gen_nodes/defaults/main.yml b/playbooks/roles/gen_nodes/defaults/main.yml
index 55827e5b..ef5e0c26 100644
--- a/playbooks/roles/gen_nodes/defaults/main.yml
+++ b/playbooks/roles/gen_nodes/defaults/main.yml
@@ -48,6 +48,9 @@ libvirt_qemu_group: libvirt-qemu
 libvirt_machine_type: 'pc'
 libvirt_host_passthrough: False
 libvirt_enable_cxl: False
+libvirt_enable_qmp: False
+libvirt_qemu_qmp_string: ""
+libvirt_qemu_qmp_wait: False
 libvirt_enable_cxl_demo_topo1: False
 libvirt_enable_cxl_demo_topo2: False
 libvirt_enable_cxl_switch_topo1: False
diff --git a/playbooks/roles/gen_nodes/templates/Vagrantfile.j2 b/playbooks/roles/gen_nodes/templates/Vagrantfile.j2
index 124cabd6..20cdd592 100644
--- a/playbooks/roles/gen_nodes/templates/Vagrantfile.j2
+++ b/playbooks/roles/gen_nodes/templates/Vagrantfile.j2
@@ -330,6 +330,14 @@ Vagrant.configure("2") do |config|
 {% endif %} # !switch topo1
         libvirt.qemuargs :value => "-M"
         libvirt.qemuargs :value => "cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=8G"
+{% if libvirt_enable_qmp %}
+        libvirt.qemuargs :value => "-qmp"
+{% if libvirt_qemu_qmp_wait %}
+        libvirt.qemuargs :value => "{{ libvirt_qemu_qmp_string }},wait=on"
+{% else %} # !qmp_wait
+        libvirt.qemuargs :value => "{{ libvirt_qemu_qmp_string }},wait=off"
+{% endif %} # libvirt_qemu_qmp_wait
+{% endif %} # libvirt_enable_qmp
 {% endif %} # CXL
 {% endif %}

diff --git a/scripts/gen-nodes.Makefile b/scripts/gen-nodes.Makefile
index eebd0b2b..8906b49f 100644
--- a/scripts/gen-nodes.Makefile
+++ b/scripts/gen-nodes.Makefile
@@ -190,6 +190,15 @@ GEN_NODES_EXTRA_ARGS += libvirt_machine_type='q35'

 ifeq (y,$(CONFIG_QEMU_ENABLE_CXL))
 GEN_NODES_EXTRA_ARGS += libvirt_enable_cxl='True'
+ifeq (y,$(CONFIG_QEMU_START_QMP_ON_TCP_SOCKET))
+GEN_NODES_EXTRA_ARGS += libvirt_enable_qmp='True'
+GEN_NODES_EXTRA_ARGS += libvirt_qemu_qmp_string='$(subst ",,$(CONFIG_QEMU_QMP_COMMAND_LINE_STRING))'
+ifeq (y,$(CONFIG_QEMU_QMP_WAIT_ON))
+GEN_NODES_EXTRA_ARGS += libvirt_qemu_qmp_wait='True'
+else
+GEN_NODES_EXTRA_ARGS += libvirt_qemu_qmp_wait='False'
+endif # CONFIG_QEMU_QMP_WAIT_ON
+endif # CONFIG_QEMU_START_QMP_ON_TCP_SOCKET
 ifeq (y,$(CONFIG_QEMU_ENABLE_CXL_DEMO_TOPOLOGY_1))
 GEN_NODES_EXTRA_ARGS += libvirt_enable_cxl_demo_topo1='True'
 endif # QEMU_ENABLE_CXL_DEMO_TOPOLOGY_1
diff --git a/vagrant/Kconfig b/vagrant/Kconfig
index 4e382264..c2783935 100644
--- a/vagrant/Kconfig
+++ b/vagrant/Kconfig
@@ -1635,6 +1635,21 @@ config QEMU_ENABLE_CXL

 if QEMU_ENABLE_CXL

+config QEMU_START_QMP_ON_TCP_SOCKET
+	bool "Start QMP on a TCP socket"
+	default n
+	if QEMU_START_QMP_ON_TCP_SOCKET
+	config QEMU_QMP_COMMAND_LINE_STRING
+	string "Qemu command line string for qmp"
+	default "tcp:localhost:4444,server"
+	help
+	  Option for qmp interface (from https://wiki.qemu.org/Documentation/QMP).
+
+	config QEMU_QMP_WAIT_ON
+	bool "Let Qemu instance wait for qmp connection"
+	default n
+	endif # QEMU_START_QMP_ON_TCP_SOCKET
+
 choice
 	prompt "CXL topology to enable"
 	default QEMU_ENABLE_CXL_DEMO_TOPOLOGY_1
--
2.40.1

[PATCH 3/3] CXL: Add support to create a DC region for DCD

[fan.ni]

From: Fan Ni <fan.ni@gmx.us>

For a dynamic capacity device (DCD), before a dynamic capacity (dc) extent
can be added and used by the system, a dc region needs to be created to cover
the dc address range.

This change add support to a new make command "cxl-create-dc-region" for
creating a dc region.

Signed-off-by: Fan Ni <fan.ni@gmx.us>
---
 .../cxl/tasks/cxl-create-dc-region/main.yml   | 27 +++++++++++++++++++
 playbooks/roles/cxl/tasks/main.yml            |  4 +++
 workflows/cxl/Makefile                        |  6 +++++
 3 files changed, 37 insertions(+)
 create mode 100644 playbooks/roles/cxl/tasks/cxl-create-dc-region/main.yml

diff --git a/playbooks/roles/cxl/tasks/cxl-create-dc-region/main.yml b/playbooks/roles/cxl/tasks/cxl-create-dc-region/main.yml
new file mode 100644
index 00000000..daed8fa5
--- /dev/null
+++ b/playbooks/roles/cxl/tasks/cxl-create-dc-region/main.yml
@@ -0,0 +1,27 @@
+---
+
+- name: Create a DC region before DC extents can be added
+  become: yes
+  become_method: sudo
+  shell: "rid=0; \
+          region=$(cat /sys/bus/cxl/devices/decoder0.0/create_dc_region); \
+          echo $region > /sys/bus/cxl/devices/decoder0.0/create_dc_region; \
+          echo 256 > /sys/bus/cxl/devices/$region/interleave_granularity; \
+          echo 1 > /sys/bus/cxl/devices/$region/interleave_ways; \
+          echo dc$rid >/sys/bus/cxl/devices/decoder2.0/mode; \
+          echo 0x40000000 >/sys/bus/cxl/devices/decoder2.0/dpa_size; \
+          echo 0x40000000 > /sys/bus/cxl/devices/$region/size; \
+          echo  decoder2.0 > /sys/bus/cxl/devices/$region/target0; \
+          echo 1 > /sys/bus/cxl/devices/$region/commit; \
+          echo $region > /sys/bus/cxl/drivers/cxl_region/bind"
+  tags: [ 'cxl-create-dc-region' ]
+
+- name: Show the DC region created
+  become: no
+  shell: "cxl list -uR"
+  register: command_output
+  tags: [ 'cxl-create-dc-region' ]
+
+- debug:
+      var: command_output.stdout_lines
+  tags: [ 'cxl-create-dc-region' ]
diff --git a/playbooks/roles/cxl/tasks/main.yml b/playbooks/roles/cxl/tasks/main.yml
index a75c20a9..164bc2ad 100644
--- a/playbooks/roles/cxl/tasks/main.yml
+++ b/playbooks/roles/cxl/tasks/main.yml
@@ -57,6 +57,10 @@
   include_tasks: cxl-mem-setup/main.yml
   tags: [ 'cxl-mem-setup' ]

+- name: Create a dynamic capacity region for further dc extent add operations
+  include_tasks: cxl-create-dc-region/main.yml
+  tags: [ 'cxl-create-dc-region' ]
+
 - name: Set the path to cxl workflow
   set_fact:
     cxl_workflow_target: "../workflows/cxl/"
diff --git a/workflows/cxl/Makefile b/workflows/cxl/Makefile
index 476ecc5e..c71e7a86 100644
--- a/workflows/cxl/Makefile
+++ b/workflows/cxl/Makefile
@@ -43,6 +43,11 @@ cxl-mem-setup:
 		-f 30 -i hosts playbooks/cxl.yml \
 		--tags cxl-mem-setup $(LIMIT_HOSTS)

+cxl-create-dc-region:
+	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
+		-f 30 -i hosts playbooks/cxl.yml \
+		--tags cxl-create-dc-region $(LIMIT_HOSTS)
+
 cxl-help-menu:
 	@echo "cxl development:"
 	@echo "cxl                 - Git clones ncdtl to compile and install the /usr/bin/cxl binary"
@@ -50,6 +55,7 @@ cxl-help-menu:
 	@echo "cxl-test-meson      - Runs the ndctl meson unit tests for the CXL test suite"
 	@echo "cxl-results         - Get results for all tests runs"
 	@echo "cxl-mem-setup	   - Convert cxl memory to cpu-less numa node"
+	@echo "cxl-create-dc-region	   - Create a dc region so dc extents can be added further"

 cxl-help-end:
 	@echo ""
--
2.40.1

Re: [PATCH 0/3] CXL: Add dynamic capacity device test support in kdevops

[Fan Ni]

On Tue, Aug 29, 2023 at 01:36:31PM -0700, fan.ni@gmx.us wrote:
> From: Fan Ni <fan.ni@gmx.us>
>

FYI. Just noticed a revised patch series for DCD from kernel side has
been posted for review. That can lead to changes for this patch series
(PATCH 3), so the review can be postponed. I will update here once I
reviewed and tested the revised DCD kernel patches.


Fan

> This patch series include 3 patches for enabling dynamic capacity device (DCD)
> test support in kdevops.
>
> Patch 1: Add a DCD device topology for users to select for DCD emulation.
> Patch 2: Add options to setup QMP interface for Qemu, through which users
> can send control commands to Qemu instance. Currently, adding and releasing
> dc extents to dc regions are implemented through QMP interface.
> Patch 3: Add a new make command "cxl-create-dc-region" to create a dc region
> so later we can add/release dc extents to the region through QMP interface.
>
> Fan Ni (3):
>   CXL: Add a new topology to support DCD emulation
>   qemu: Add QMP connection option for QMP interface access
>   CXL: Add support to create a DC region for DCD
>
>  .../cxl/tasks/cxl-create-dc-region/main.yml   | 27 +++++++++++++++++++
>  playbooks/roles/cxl/tasks/main.yml            |  4 +++
>  playbooks/roles/gen_nodes/defaults/main.yml   |  4 +++
>  .../roles/gen_nodes/templates/Vagrantfile.j2  | 19 ++++++++++---
>  scripts/gen-nodes.Makefile                    | 12 +++++++++
>  vagrant/Kconfig                               | 24 +++++++++++++++++
>  workflows/cxl/Makefile                        |  6 +++++
>  7 files changed, 93 insertions(+), 3 deletions(-)
>  create mode 100644 playbooks/roles/cxl/tasks/cxl-create-dc-region/main.yml
>
> --
> 2.40.1
>