* [PATCH v4 00/14] Use Intel DSA accelerator to offload zero page checking in multifd live migration.
@ 2024-04-25 2:21 Hao Xiang
2024-04-25 2:21 ` [PATCH v4 01/14] meson: Introduce new instruction set enqcmd to the build system Hao Xiang
` (14 more replies)
0 siblings, 15 replies; 32+ messages in thread
From: Hao Xiang @ 2024-04-25 2:21 UTC (permalink / raw)
To: marcandre.lureau, peterx, farosas, armbru, lvivier, qemu-devel; +Cc: Hao Xiang
v4
* Rebase on top of 85b597413d4370cb168f711192eaef2eb70535ac.
* A separate "multifd zero page checking" patchset was split from this
patchset's v3 and got merged into master. v4 re-applied the rest of all
commits on top of that patchset, re-factored and re-tested.
https://lore.kernel.org/all/20240311180015.3359271-1-hao.xiang@linux.dev/
* There are some feedback from v3 I likely overlooked.
v3
* Rebase on top of 7425b6277f12e82952cede1f531bfc689bf77fb1.
* Fix error/warning from checkpatch.pl
* Fix use-after-free bug when multifd-dsa-accel option is not set.
* Handle error from dsa_init and correctly propogate the error.
* Remove unnecessary call to dsa_stop.
* Detect availability of DSA feature at compile time.
* Implement a generic batch_task structure and a DSA specific one dsa_batch_task.
* Remove all exit() calls and propagate errors correctly.
* Use bytes instead of page count to configure multifd-packet-size option.
v2
* Rebase on top of 3e01f1147a16ca566694b97eafc941d62fa1e8d8.
* Leave Juan's changes in their original form instead of squashing them.
* Add a new commit to refactor the multifd_send_thread function to prepare for introducing the DSA offload functionality.
* Use page count to configure multifd-packet-size option.
* Don't use the FLAKY flag in DSA tests.
* Test if DSA integration test is setup correctly and skip the test if
* not.
* Fixed broken link in the previous patch cover.
* Background:
I posted an RFC about DSA offloading in QEMU:
https://patchew.org/QEMU/20230529182001.2232069-1-hao.xiang@bytedance.com/
This patchset implements the DSA offloading on zero page checking in
multifd live migration code path.
* Overview:
Intel Data Streaming Accelerator(DSA) is introduced in Intel's 4th generation
Xeon server, aka Sapphire Rapids.
https://cdrdv2-public.intel.com/671116/341204-intel-data-streaming-accelerator-spec.pdf
https://www.intel.com/content/www/us/en/content-details/759709/intel-data-streaming-accelerator-user-guide.html
One of the things DSA can do is to offload memory comparison workload from
CPU to DSA accelerator hardware. This patchset implements a solution to offload
QEMU's zero page checking from CPU to DSA accelerator hardware. We gain
two benefits from this change:
1. Reduces CPU usage in multifd live migration workflow across all use
cases.
2. Reduces migration total time in some use cases.
* Design:
These are the logical steps to perform DSA offloading:
1. Configure DSA accelerators and create user space openable DSA work
queues via the idxd driver.
2. Map DSA's work queue into a user space address space.
3. Fill an in-memory task descriptor to describe the memory operation.
4. Use dedicated CPU instruction _enqcmd to queue a task descriptor to
the work queue.
5. Pull the task descriptor's completion status field until the task
completes.
6. Check return status.
The memory operation is now totally done by the accelerator hardware but
the new workflow introduces overheads. The overhead is the extra cost CPU
prepares and submits the task descriptors and the extra cost CPU pulls for
completion. The design is around minimizing these two overheads.
1. In order to reduce the overhead on task preparation and submission,
we use batch descriptors. A batch descriptor will contain N individual
zero page checking tasks where the default N is 128 (default packet size
/ page size) and we can increase N by setting the packet size via a new
migration option.
2. The multifd sender threads prepares and submits batch tasks to DSA
hardware and it waits on a synchronization object for task completion.
Whenever a DSA task is submitted, the task structure is added to a
thread safe queue. It's safe to have multiple multifd sender threads to
submit tasks concurrently.
3. Multiple DSA hardware devices can be used. During multifd initialization,
every sender thread will be assigned a DSA device to work with. We
use a round-robin scheme to evenly distribute the work across all used
DSA devices.
4. Use a dedicated thread dsa_completion to perform busy pulling for all
DSA task completions. The thread keeps dequeuing DSA tasks from the
thread safe queue. The thread blocks when there is no outstanding DSA
task. When pulling for completion of a DSA task, the thread uses CPU
instruction _mm_pause between the iterations of a busy loop to save some
CPU power as well as optimizing core resources for the other hypercore.
5. DSA accelerator can encounter errors. The most popular error is a
page fault. We have tested using devices to handle page faults but
performance is bad. Right now, if DSA hits a page fault, we fallback to
use CPU to complete the rest of the work. The CPU fallback is done in
the multifd sender thread.
6. Added a new migration option multifd-dsa-accel to set the DSA device
path. If set, the multifd workflow will leverage the DSA devices for
offloading.
7. Added a new migration option multifd-normal-page-ratio to make
multifd live migration easier to test. Setting a normal page ratio will
make live migration recognize a zero page as a normal page and send
the entire payload over the network. If we want to send a large network
payload and analyze throughput, this option is useful.
8. Added a new migration option multifd-packet-size. This can increase
the number of pages being zero page checked and sent over the network.
The extra synchronization between the sender threads and the dsa
completion thread is an overhead. Using a large packet size can reduce
that overhead.
* Performance:
We use two Intel 4th generation Xeon servers for testing.
Architecture: x86_64
CPU(s): 192
Thread(s) per core: 2
Core(s) per socket: 48
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 143
Model name: Intel(R) Xeon(R) Platinum 8457C
Stepping: 8
CPU MHz: 2538.624
CPU max MHz: 3800.0000
CPU min MHz: 800.0000
We perform multifd live migration with below setup:
1. VM has 100GB memory.
2. Use the new migration option multifd-set-normal-page-ratio to control the total
size of the payload sent over the network.
3. Use 8 multifd channels.
4. Use tcp for live migration.
4. Use CPU to perform zero page checking as the baseline.
5. Use one DSA device to offload zero page checking to compare with the baseline.
6. Use "perf sched record" and "perf sched timehist" to analyze CPU usage.
A) Scenario 1: 50% (50GB) normal pages on an 100GB vm.
CPU usage
|---------------|---------------|---------------|---------------|
| |comm |runtime(msec) |totaltime(msec)|
|---------------|---------------|---------------|---------------|
|Baseline |live_migration |5657.58 | |
| |multifdsend_0 |3931.563 | |
| |multifdsend_1 |4405.273 | |
| |multifdsend_2 |3941.968 | |
| |multifdsend_3 |5032.975 | |
| |multifdsend_4 |4533.865 | |
| |multifdsend_5 |4530.461 | |
| |multifdsend_6 |5171.916 | |
| |multifdsend_7 |4722.769 |41922 |
|---------------|---------------|---------------|---------------|
|DSA |live_migration |6129.168 | |
| |multifdsend_0 |2954.717 | |
| |multifdsend_1 |2766.359 | |
| |multifdsend_2 |2853.519 | |
| |multifdsend_3 |2740.717 | |
| |multifdsend_4 |2824.169 | |
| |multifdsend_5 |2966.908 | |
| |multifdsend_6 |2611.137 | |
| |multifdsend_7 |3114.732 | |
| |dsa_completion |3612.564 |32568 |
|---------------|---------------|---------------|---------------|
Baseline total runtime is calculated by adding up all multifdsend_X
and live_migration threads runtime. DSA offloading total runtime is
calculated by adding up all multifdsend_X, live_migration and
dsa_completion threads runtime. 41922 msec VS 32568 msec runtime and
that is 23% total CPU usage savings.
Latency
|---------------|---------------|---------------|---------------|---------------|---------------|
| |total time |down time |throughput |transferred-ram|total-ram |
|---------------|---------------|---------------|---------------|---------------|---------------|
|Baseline |10343 ms |161 ms |41007.00 mbps |51583797 kb |102400520 kb |
|---------------|---------------|---------------|---------------|-------------------------------|
|DSA offload |9535 ms |135 ms |46554.40 mbps |53947545 kb |102400520 kb |
|---------------|---------------|---------------|---------------|---------------|---------------|
Total time is 8% faster and down time is 16% faster.
B) Scenario 2: 100% (100GB) zero pages on an 100GB vm.
CPU usage
|---------------|---------------|---------------|---------------|
| |comm |runtime(msec) |totaltime(msec)|
|---------------|---------------|---------------|---------------|
|Baseline |live_migration |4860.718 | |
| |multifdsend_0 |748.875 | |
| |multifdsend_1 |898.498 | |
| |multifdsend_2 |787.456 | |
| |multifdsend_3 |764.537 | |
| |multifdsend_4 |785.687 | |
| |multifdsend_5 |756.941 | |
| |multifdsend_6 |774.084 | |
| |multifdsend_7 |782.900 |11154 |
|---------------|---------------|-------------------------------|
|DSA offloading |live_migration |3846.976 | |
| |multifdsend_0 |191.880 | |
| |multifdsend_1 |166.331 | |
| |multifdsend_2 |168.528 | |
| |multifdsend_3 |197.831 | |
| |multifdsend_4 |169.580 | |
| |multifdsend_5 |167.984 | |
| |multifdsend_6 |198.042 | |
| |multifdsend_7 |170.624 | |
| |dsa_completion |3428.669 |8700 |
|---------------|---------------|---------------|---------------|
Baseline total runtime is 11154 msec and DSA offloading total runtime is
8700 msec. That is 22% CPU savings.
Latency
|--------------------------------------------------------------------------------------------|
| |total time |down time |throughput |transferred-ram|total-ram |
|---------------|---------------|---------------|---------------|---------------|------------|
|Baseline |4867 ms |20 ms |1.51 mbps |565 kb |102400520 kb|
|---------------|---------------|---------------|---------------|----------------------------|
|DSA offload |3888 ms |18 ms |1.89 mbps |565 kb |102400520 kb|
|---------------|---------------|---------------|---------------|---------------|------------|
Total time 20% faster and down time 10% faster.
* Testing:
1. Added unit tests for cover the added code path in dsa.c
2. Added integration tests to cover multifd live migration using DSA
offloading.
* Patchset
Apply this patchset on top of commit
85b597413d4370cb168f711192eaef2eb70535ac
Hao Xiang (14):
meson: Introduce new instruction set enqcmd to the build system.
util/dsa: Add dependency idxd.
util/dsa: Implement DSA device start and stop logic.
util/dsa: Implement DSA task enqueue and dequeue.
util/dsa: Implement DSA task asynchronous completion thread model.
util/dsa: Implement zero page checking in DSA task.
util/dsa: Implement DSA task asynchronous submission and wait for
completion.
migration/multifd: Add new migration option for multifd DSA
offloading.
migration/multifd: Prepare to introduce DSA acceleration on the
multifd path.
migration/multifd: Enable DSA offloading in multifd sender path.
migration/multifd: Add migration option set packet size.
migration/multifd: Enable set packet size migration option.
util/dsa: Add unit test coverage for Intel DSA task submission and
completion.
migration/multifd: Add integration tests for multifd with Intel DSA
offloading.
include/qemu/dsa.h | 180 +++++
linux-headers/linux/idxd.h | 356 ++++++++++
meson.build | 14 +
meson_options.txt | 2 +
migration/migration-hmp-cmds.c | 15 +
migration/multifd-zero-page.c | 99 ++-
migration/multifd-zlib.c | 6 +-
migration/multifd-zstd.c | 6 +-
migration/multifd.c | 38 +-
migration/multifd.h | 6 +-
migration/options.c | 66 ++
migration/options.h | 2 +
qapi/migration.json | 43 +-
scripts/meson-buildoptions.sh | 3 +
tests/qtest/migration-test.c | 77 ++-
tests/unit/meson.build | 6 +
tests/unit/test-dsa.c | 499 ++++++++++++++
util/dsa.c | 1170 ++++++++++++++++++++++++++++++++
util/meson.build | 1 +
19 files changed, 2568 insertions(+), 21 deletions(-)
create mode 100644 include/qemu/dsa.h
create mode 100644 linux-headers/linux/idxd.h
create mode 100644 tests/unit/test-dsa.c
create mode 100644 util/dsa.c
--
2.30.2
^ permalink raw reply [flat|nested] 32+ messages in thread
* [PATCH v4 01/14] meson: Introduce new instruction set enqcmd to the build system.
2024-04-25 2:21 [PATCH v4 00/14] Use Intel DSA accelerator to offload zero page checking in multifd live migration Hao Xiang
@ 2024-04-25 2:21 ` Hao Xiang
2024-04-25 18:50 ` Fabiano Rosas
2024-04-25 2:21 ` [PATCH v4 02/14] util/dsa: Add dependency idxd Hao Xiang
` (13 subsequent siblings)
14 siblings, 1 reply; 32+ messages in thread
From: Hao Xiang @ 2024-04-25 2:21 UTC (permalink / raw)
To: marcandre.lureau, peterx, farosas, armbru, lvivier, qemu-devel; +Cc: Hao Xiang
Enable instruction set enqcmd in build.
Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
---
meson.build | 14 ++++++++++++++
meson_options.txt | 2 ++
scripts/meson-buildoptions.sh | 3 +++
3 files changed, 19 insertions(+)
diff --git a/meson.build b/meson.build
index 95cee7046e..9e008ddc34 100644
--- a/meson.build
+++ b/meson.build
@@ -2824,6 +2824,20 @@ config_host_data.set('CONFIG_AVX512BW_OPT', get_option('avx512bw') \
int main(int argc, char *argv[]) { return bar(argv[0]); }
'''), error_message: 'AVX512BW not available').allowed())
+config_host_data.set('CONFIG_DSA_OPT', get_option('enqcmd') \
+ .require(have_cpuid_h, error_message: 'cpuid.h not available, cannot enable ENQCMD') \
+ .require(cc.links('''
+ #include <stdint.h>
+ #include <cpuid.h>
+ #include <immintrin.h>
+ static int __attribute__((target("enqcmd"))) bar(void *a) {
+ uint64_t dst[8] = { 0 };
+ uint64_t src[8] = { 0 };
+ return _enqcmd(dst, src);
+ }
+ int main(int argc, char *argv[]) { return bar(argv[argc - 1]); }
+ '''), error_message: 'ENQCMD not available').allowed())
+
# For both AArch64 and AArch32, detect if builtins are available.
config_host_data.set('CONFIG_ARM_AES_BUILTIN', cc.compiles('''
#include <arm_neon.h>
diff --git a/meson_options.txt b/meson_options.txt
index b5c0bad9e7..63c1bf815b 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -121,6 +121,8 @@ option('avx512f', type: 'feature', value: 'disabled',
description: 'AVX512F optimizations')
option('avx512bw', type: 'feature', value: 'auto',
description: 'AVX512BW optimizations')
+option('enqcmd', type: 'feature', value: 'disabled',
+ description: 'MENQCMD optimizations')
option('keyring', type: 'feature', value: 'auto',
description: 'Linux keyring support')
option('libkeyutils', type: 'feature', value: 'auto',
diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh
index 5ace33f167..2cdfc84455 100644
--- a/scripts/meson-buildoptions.sh
+++ b/scripts/meson-buildoptions.sh
@@ -93,6 +93,7 @@ meson_options_help() {
printf "%s\n" ' avx2 AVX2 optimizations'
printf "%s\n" ' avx512bw AVX512BW optimizations'
printf "%s\n" ' avx512f AVX512F optimizations'
+ printf "%s\n" ' enqcmd ENQCMD optimizations'
printf "%s\n" ' blkio libblkio block device driver'
printf "%s\n" ' bochs bochs image format support'
printf "%s\n" ' bpf eBPF support'
@@ -239,6 +240,8 @@ _meson_option_parse() {
--disable-avx512bw) printf "%s" -Davx512bw=disabled ;;
--enable-avx512f) printf "%s" -Davx512f=enabled ;;
--disable-avx512f) printf "%s" -Davx512f=disabled ;;
+ --enable-enqcmd) printf "%s" -Denqcmd=enabled ;;
+ --disable-enqcmd) printf "%s" -Denqcmd=disabled ;;
--enable-gcov) printf "%s" -Db_coverage=true ;;
--disable-gcov) printf "%s" -Db_coverage=false ;;
--enable-lto) printf "%s" -Db_lto=true ;;
--
2.30.2
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [PATCH v4 02/14] util/dsa: Add dependency idxd.
2024-04-25 2:21 [PATCH v4 00/14] Use Intel DSA accelerator to offload zero page checking in multifd live migration Hao Xiang
2024-04-25 2:21 ` [PATCH v4 01/14] meson: Introduce new instruction set enqcmd to the build system Hao Xiang
@ 2024-04-25 2:21 ` Hao Xiang
2024-04-25 20:33 ` Fabiano Rosas
2024-04-25 2:21 ` [PATCH v4 03/14] util/dsa: Implement DSA device start and stop logic Hao Xiang
` (12 subsequent siblings)
14 siblings, 1 reply; 32+ messages in thread
From: Hao Xiang @ 2024-04-25 2:21 UTC (permalink / raw)
To: marcandre.lureau, peterx, farosas, armbru, lvivier, qemu-devel; +Cc: Hao Xiang
Idxd is the device driver for DSA (Intel Data Streaming
Accelerator). The driver is fully functioning since Linux
kernel 5.19. This change adds the driver's header file used
for userspace development.
Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
---
| 356 +++++++++++++++++++++++++++++++++++++
1 file changed, 356 insertions(+)
create mode 100644 linux-headers/linux/idxd.h
--git a/linux-headers/linux/idxd.h b/linux-headers/linux/idxd.h
new file mode 100644
index 0000000000..1d553bedbd
--- /dev/null
+++ b/linux-headers/linux/idxd.h
@@ -0,0 +1,356 @@
+/* SPDX-License-Identifier: LGPL-2.1 WITH Linux-syscall-note */
+/* Copyright(c) 2019 Intel Corporation. All rights rsvd. */
+#ifndef _USR_IDXD_H_
+#define _USR_IDXD_H_
+
+#ifdef __KERNEL__
+#include <linux/types.h>
+#else
+#include <stdint.h>
+#endif
+
+/* Driver command error status */
+enum idxd_scmd_stat {
+ IDXD_SCMD_DEV_ENABLED = 0x80000010,
+ IDXD_SCMD_DEV_NOT_ENABLED = 0x80000020,
+ IDXD_SCMD_WQ_ENABLED = 0x80000021,
+ IDXD_SCMD_DEV_DMA_ERR = 0x80020000,
+ IDXD_SCMD_WQ_NO_GRP = 0x80030000,
+ IDXD_SCMD_WQ_NO_NAME = 0x80040000,
+ IDXD_SCMD_WQ_NO_SVM = 0x80050000,
+ IDXD_SCMD_WQ_NO_THRESH = 0x80060000,
+ IDXD_SCMD_WQ_PORTAL_ERR = 0x80070000,
+ IDXD_SCMD_WQ_RES_ALLOC_ERR = 0x80080000,
+ IDXD_SCMD_PERCPU_ERR = 0x80090000,
+ IDXD_SCMD_DMA_CHAN_ERR = 0x800a0000,
+ IDXD_SCMD_CDEV_ERR = 0x800b0000,
+ IDXD_SCMD_WQ_NO_SWQ_SUPPORT = 0x800c0000,
+ IDXD_SCMD_WQ_NONE_CONFIGURED = 0x800d0000,
+ IDXD_SCMD_WQ_NO_SIZE = 0x800e0000,
+ IDXD_SCMD_WQ_NO_PRIV = 0x800f0000,
+ IDXD_SCMD_WQ_IRQ_ERR = 0x80100000,
+ IDXD_SCMD_WQ_USER_NO_IOMMU = 0x80110000,
+};
+
+#define IDXD_SCMD_SOFTERR_MASK 0x80000000
+#define IDXD_SCMD_SOFTERR_SHIFT 16
+
+/* Descriptor flags */
+#define IDXD_OP_FLAG_FENCE 0x0001
+#define IDXD_OP_FLAG_BOF 0x0002
+#define IDXD_OP_FLAG_CRAV 0x0004
+#define IDXD_OP_FLAG_RCR 0x0008
+#define IDXD_OP_FLAG_RCI 0x0010
+#define IDXD_OP_FLAG_CRSTS 0x0020
+#define IDXD_OP_FLAG_CR 0x0080
+#define IDXD_OP_FLAG_CC 0x0100
+#define IDXD_OP_FLAG_ADDR1_TCS 0x0200
+#define IDXD_OP_FLAG_ADDR2_TCS 0x0400
+#define IDXD_OP_FLAG_ADDR3_TCS 0x0800
+#define IDXD_OP_FLAG_CR_TCS 0x1000
+#define IDXD_OP_FLAG_STORD 0x2000
+#define IDXD_OP_FLAG_DRDBK 0x4000
+#define IDXD_OP_FLAG_DSTS 0x8000
+
+/* IAX */
+#define IDXD_OP_FLAG_RD_SRC2_AECS 0x010000
+#define IDXD_OP_FLAG_RD_SRC2_2ND 0x020000
+#define IDXD_OP_FLAG_WR_SRC2_AECS_COMP 0x040000
+#define IDXD_OP_FLAG_WR_SRC2_AECS_OVFL 0x080000
+#define IDXD_OP_FLAG_SRC2_STS 0x100000
+#define IDXD_OP_FLAG_CRC_RFC3720 0x200000
+
+/* Opcode */
+enum dsa_opcode {
+ DSA_OPCODE_NOOP = 0,
+ DSA_OPCODE_BATCH,
+ DSA_OPCODE_DRAIN,
+ DSA_OPCODE_MEMMOVE,
+ DSA_OPCODE_MEMFILL,
+ DSA_OPCODE_COMPARE,
+ DSA_OPCODE_COMPVAL,
+ DSA_OPCODE_CR_DELTA,
+ DSA_OPCODE_AP_DELTA,
+ DSA_OPCODE_DUALCAST,
+ DSA_OPCODE_CRCGEN = 0x10,
+ DSA_OPCODE_COPY_CRC,
+ DSA_OPCODE_DIF_CHECK,
+ DSA_OPCODE_DIF_INS,
+ DSA_OPCODE_DIF_STRP,
+ DSA_OPCODE_DIF_UPDT,
+ DSA_OPCODE_CFLUSH = 0x20,
+};
+
+enum iax_opcode {
+ IAX_OPCODE_NOOP = 0,
+ IAX_OPCODE_DRAIN = 2,
+ IAX_OPCODE_MEMMOVE,
+ IAX_OPCODE_DECOMPRESS = 0x42,
+ IAX_OPCODE_COMPRESS,
+ IAX_OPCODE_CRC64,
+ IAX_OPCODE_ZERO_DECOMP_32 = 0x48,
+ IAX_OPCODE_ZERO_DECOMP_16,
+ IAX_OPCODE_ZERO_COMP_32 = 0x4c,
+ IAX_OPCODE_ZERO_COMP_16,
+ IAX_OPCODE_SCAN = 0x50,
+ IAX_OPCODE_SET_MEMBER,
+ IAX_OPCODE_EXTRACT,
+ IAX_OPCODE_SELECT,
+ IAX_OPCODE_RLE_BURST,
+ IAX_OPCODE_FIND_UNIQUE,
+ IAX_OPCODE_EXPAND,
+};
+
+/* Completion record status */
+enum dsa_completion_status {
+ DSA_COMP_NONE = 0,
+ DSA_COMP_SUCCESS,
+ DSA_COMP_SUCCESS_PRED,
+ DSA_COMP_PAGE_FAULT_NOBOF,
+ DSA_COMP_PAGE_FAULT_IR,
+ DSA_COMP_BATCH_FAIL,
+ DSA_COMP_BATCH_PAGE_FAULT,
+ DSA_COMP_DR_OFFSET_NOINC,
+ DSA_COMP_DR_OFFSET_ERANGE,
+ DSA_COMP_DIF_ERR,
+ DSA_COMP_BAD_OPCODE = 0x10,
+ DSA_COMP_INVALID_FLAGS,
+ DSA_COMP_NOZERO_RESERVE,
+ DSA_COMP_XFER_ERANGE,
+ DSA_COMP_DESC_CNT_ERANGE,
+ DSA_COMP_DR_ERANGE,
+ DSA_COMP_OVERLAP_BUFFERS,
+ DSA_COMP_DCAST_ERR,
+ DSA_COMP_DESCLIST_ALIGN,
+ DSA_COMP_INT_HANDLE_INVAL,
+ DSA_COMP_CRA_XLAT,
+ DSA_COMP_CRA_ALIGN,
+ DSA_COMP_ADDR_ALIGN,
+ DSA_COMP_PRIV_BAD,
+ DSA_COMP_TRAFFIC_CLASS_CONF,
+ DSA_COMP_PFAULT_RDBA,
+ DSA_COMP_HW_ERR1,
+ DSA_COMP_HW_ERR_DRB,
+ DSA_COMP_TRANSLATION_FAIL,
+};
+
+enum iax_completion_status {
+ IAX_COMP_NONE = 0,
+ IAX_COMP_SUCCESS,
+ IAX_COMP_PAGE_FAULT_IR = 0x04,
+ IAX_COMP_ANALYTICS_ERROR = 0x0a,
+ IAX_COMP_OUTBUF_OVERFLOW,
+ IAX_COMP_BAD_OPCODE = 0x10,
+ IAX_COMP_INVALID_FLAGS,
+ IAX_COMP_NOZERO_RESERVE,
+ IAX_COMP_INVALID_SIZE,
+ IAX_COMP_OVERLAP_BUFFERS = 0x16,
+ IAX_COMP_INT_HANDLE_INVAL = 0x19,
+ IAX_COMP_CRA_XLAT,
+ IAX_COMP_CRA_ALIGN,
+ IAX_COMP_ADDR_ALIGN,
+ IAX_COMP_PRIV_BAD,
+ IAX_COMP_TRAFFIC_CLASS_CONF,
+ IAX_COMP_PFAULT_RDBA,
+ IAX_COMP_HW_ERR1,
+ IAX_COMP_HW_ERR_DRB,
+ IAX_COMP_TRANSLATION_FAIL,
+ IAX_COMP_PRS_TIMEOUT,
+ IAX_COMP_WATCHDOG,
+ IAX_COMP_INVALID_COMP_FLAG = 0x30,
+ IAX_COMP_INVALID_FILTER_FLAG,
+ IAX_COMP_INVALID_INPUT_SIZE,
+ IAX_COMP_INVALID_NUM_ELEMS,
+ IAX_COMP_INVALID_SRC1_WIDTH,
+ IAX_COMP_INVALID_INVERT_OUT,
+};
+
+#define DSA_COMP_STATUS_MASK 0x7f
+#define DSA_COMP_STATUS_WRITE 0x80
+
+struct dsa_hw_desc {
+ uint32_t pasid:20;
+ uint32_t rsvd:11;
+ uint32_t priv:1;
+ uint32_t flags:24;
+ uint32_t opcode:8;
+ uint64_t completion_addr;
+ union {
+ uint64_t src_addr;
+ uint64_t rdback_addr;
+ uint64_t pattern;
+ uint64_t desc_list_addr;
+ };
+ union {
+ uint64_t dst_addr;
+ uint64_t rdback_addr2;
+ uint64_t src2_addr;
+ uint64_t comp_pattern;
+ };
+ union {
+ uint32_t xfer_size;
+ uint32_t desc_count;
+ };
+ uint16_t int_handle;
+ uint16_t rsvd1;
+ union {
+ uint8_t expected_res;
+ /* create delta record */
+ struct {
+ uint64_t delta_addr;
+ uint32_t max_delta_size;
+ uint32_t delt_rsvd;
+ uint8_t expected_res_mask;
+ };
+ uint32_t delta_rec_size;
+ uint64_t dest2;
+ /* CRC */
+ struct {
+ uint32_t crc_seed;
+ uint32_t crc_rsvd;
+ uint64_t seed_addr;
+ };
+ /* DIF check or strip */
+ struct {
+ uint8_t src_dif_flags;
+ uint8_t dif_chk_res;
+ uint8_t dif_chk_flags;
+ uint8_t dif_chk_res2[5];
+ uint32_t chk_ref_tag_seed;
+ uint16_t chk_app_tag_mask;
+ uint16_t chk_app_tag_seed;
+ };
+ /* DIF insert */
+ struct {
+ uint8_t dif_ins_res;
+ uint8_t dest_dif_flag;
+ uint8_t dif_ins_flags;
+ uint8_t dif_ins_res2[13];
+ uint32_t ins_ref_tag_seed;
+ uint16_t ins_app_tag_mask;
+ uint16_t ins_app_tag_seed;
+ };
+ /* DIF update */
+ struct {
+ uint8_t src_upd_flags;
+ uint8_t upd_dest_flags;
+ uint8_t dif_upd_flags;
+ uint8_t dif_upd_res[5];
+ uint32_t src_ref_tag_seed;
+ uint16_t src_app_tag_mask;
+ uint16_t src_app_tag_seed;
+ uint32_t dest_ref_tag_seed;
+ uint16_t dest_app_tag_mask;
+ uint16_t dest_app_tag_seed;
+ };
+
+ uint8_t op_specific[24];
+ };
+} __attribute__((packed));
+
+struct iax_hw_desc {
+ uint32_t pasid:20;
+ uint32_t rsvd:11;
+ uint32_t priv:1;
+ uint32_t flags:24;
+ uint32_t opcode:8;
+ uint64_t completion_addr;
+ uint64_t src1_addr;
+ uint64_t dst_addr;
+ uint32_t src1_size;
+ uint16_t int_handle;
+ union {
+ uint16_t compr_flags;
+ uint16_t decompr_flags;
+ };
+ uint64_t src2_addr;
+ uint32_t max_dst_size;
+ uint32_t src2_size;
+ uint32_t filter_flags;
+ uint32_t num_inputs;
+} __attribute__((packed));
+
+struct dsa_raw_desc {
+ uint64_t field[8];
+} __attribute__((packed));
+
+/*
+ * The status field will be modified by hardware, therefore it should be
+ * volatile and prevent the compiler from optimize the read.
+ */
+struct dsa_completion_record {
+ volatile uint8_t status;
+ union {
+ uint8_t result;
+ uint8_t dif_status;
+ };
+ uint16_t rsvd;
+ uint32_t bytes_completed;
+ uint64_t fault_addr;
+ union {
+ /* common record */
+ struct {
+ uint32_t invalid_flags:24;
+ uint32_t rsvd2:8;
+ };
+
+ uint32_t delta_rec_size;
+ uint64_t crc_val;
+
+ /* DIF check & strip */
+ struct {
+ uint32_t dif_chk_ref_tag;
+ uint16_t dif_chk_app_tag_mask;
+ uint16_t dif_chk_app_tag;
+ };
+
+ /* DIF insert */
+ struct {
+ uint64_t dif_ins_res;
+ uint32_t dif_ins_ref_tag;
+ uint16_t dif_ins_app_tag_mask;
+ uint16_t dif_ins_app_tag;
+ };
+
+ /* DIF update */
+ struct {
+ uint32_t dif_upd_src_ref_tag;
+ uint16_t dif_upd_src_app_tag_mask;
+ uint16_t dif_upd_src_app_tag;
+ uint32_t dif_upd_dest_ref_tag;
+ uint16_t dif_upd_dest_app_tag_mask;
+ uint16_t dif_upd_dest_app_tag;
+ };
+
+ uint8_t op_specific[16];
+ };
+} __attribute__((packed));
+
+struct dsa_raw_completion_record {
+ uint64_t field[4];
+} __attribute__((packed));
+
+struct iax_completion_record {
+ volatile uint8_t status;
+ uint8_t error_code;
+ uint16_t rsvd;
+ uint32_t bytes_completed;
+ uint64_t fault_addr;
+ uint32_t invalid_flags;
+ uint32_t rsvd2;
+ uint32_t output_size;
+ uint8_t output_bits;
+ uint8_t rsvd3;
+ uint16_t xor_csum;
+ uint32_t crc;
+ uint32_t min;
+ uint32_t max;
+ uint32_t sum;
+ uint64_t rsvd4[2];
+} __attribute__((packed));
+
+struct iax_raw_completion_record {
+ uint64_t field[8];
+} __attribute__((packed));
+
+#endif
--
2.30.2
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [PATCH v4 03/14] util/dsa: Implement DSA device start and stop logic.
2024-04-25 2:21 [PATCH v4 00/14] Use Intel DSA accelerator to offload zero page checking in multifd live migration Hao Xiang
2024-04-25 2:21 ` [PATCH v4 01/14] meson: Introduce new instruction set enqcmd to the build system Hao Xiang
2024-04-25 2:21 ` [PATCH v4 02/14] util/dsa: Add dependency idxd Hao Xiang
@ 2024-04-25 2:21 ` Hao Xiang
2024-04-25 14:21 ` Daniel P. Berrangé
` (3 more replies)
2024-04-25 2:21 ` [PATCH v4 04/14] util/dsa: Implement DSA task enqueue and dequeue Hao Xiang
` (11 subsequent siblings)
14 siblings, 4 replies; 32+ messages in thread
From: Hao Xiang @ 2024-04-25 2:21 UTC (permalink / raw)
To: marcandre.lureau, peterx, farosas, armbru, lvivier, qemu-devel
Cc: Hao Xiang, Bryan Zhang
* DSA device open and close.
* DSA group contains multiple DSA devices.
* DSA group configure/start/stop/clean.
Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
---
include/qemu/dsa.h | 72 +++++++++++
util/dsa.c | 316 +++++++++++++++++++++++++++++++++++++++++++++
util/meson.build | 1 +
3 files changed, 389 insertions(+)
create mode 100644 include/qemu/dsa.h
create mode 100644 util/dsa.c
diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
new file mode 100644
index 0000000000..f15c05ee85
--- /dev/null
+++ b/include/qemu/dsa.h
@@ -0,0 +1,72 @@
+#ifndef QEMU_DSA_H
+#define QEMU_DSA_H
+
+#include "qemu/error-report.h"
+#include "qemu/thread.h"
+#include "qemu/queue.h"
+
+#ifdef CONFIG_DSA_OPT
+
+#pragma GCC push_options
+#pragma GCC target("enqcmd")
+
+#include <linux/idxd.h>
+#include "x86intrin.h"
+
+/**
+ * @brief Initializes DSA devices.
+ *
+ * @param dsa_parameter A list of DSA device path from migration parameter.
+ *
+ * @return int Zero if successful, otherwise non zero.
+ */
+int dsa_init(const char *dsa_parameter);
+
+/**
+ * @brief Start logic to enable using DSA.
+ */
+void dsa_start(void);
+
+/**
+ * @brief Stop the device group and the completion thread.
+ */
+void dsa_stop(void);
+
+/**
+ * @brief Clean up system resources created for DSA offloading.
+ */
+void dsa_cleanup(void);
+
+/**
+ * @brief Check if DSA is running.
+ *
+ * @return True if DSA is running, otherwise false.
+ */
+bool dsa_is_running(void);
+
+#else
+
+static inline bool dsa_is_running(void)
+{
+ return false;
+}
+
+static inline int dsa_init(const char *dsa_parameter)
+{
+ if (dsa_parameter != NULL && strlen(dsa_parameter) != 0) {
+ error_report("DSA not supported.");
+ return -1;
+ }
+
+ return 0;
+}
+
+static inline void dsa_start(void) {}
+
+static inline void dsa_stop(void) {}
+
+static inline void dsa_cleanup(void) {}
+
+#endif
+
+#endif
diff --git a/util/dsa.c b/util/dsa.c
new file mode 100644
index 0000000000..05bbf8e31a
--- /dev/null
+++ b/util/dsa.c
@@ -0,0 +1,316 @@
+/*
+ * Use Intel Data Streaming Accelerator to offload certain background
+ * operations.
+ *
+ * Copyright (c) 2023 Hao Xiang <hao.xiang@bytedance.com>
+ * Bryan Zhang <bryan.zhang@bytedance.com>
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/queue.h"
+#include "qemu/memalign.h"
+#include "qemu/lockable.h"
+#include "qemu/cutils.h"
+#include "qemu/dsa.h"
+#include "qemu/bswap.h"
+#include "qemu/error-report.h"
+#include "qemu/rcu.h"
+
+#ifdef CONFIG_DSA_OPT
+
+#pragma GCC push_options
+#pragma GCC target("enqcmd")
+
+#include <linux/idxd.h>
+#include "x86intrin.h"
+
+#define DSA_WQ_SIZE 4096
+#define MAX_DSA_DEVICES 16
+
+typedef QSIMPLEQ_HEAD(dsa_task_queue, dsa_batch_task) dsa_task_queue;
+
+struct dsa_device {
+ void *work_queue;
+};
+
+struct dsa_device_group {
+ struct dsa_device *dsa_devices;
+ int num_dsa_devices;
+ /* The index of the next DSA device to be used. */
+ uint32_t device_allocator_index;
+ bool running;
+ QemuMutex task_queue_lock;
+ QemuCond task_queue_cond;
+ dsa_task_queue task_queue;
+};
+
+uint64_t max_retry_count;
+static struct dsa_device_group dsa_group;
+
+
+/**
+ * @brief This function opens a DSA device's work queue and
+ * maps the DSA device memory into the current process.
+ *
+ * @param dsa_wq_path A pointer to the DSA device work queue's file path.
+ * @return A pointer to the mapped memory, or MAP_FAILED on failure.
+ */
+static void *
+map_dsa_device(const char *dsa_wq_path)
+{
+ void *dsa_device;
+ int fd;
+
+ fd = open(dsa_wq_path, O_RDWR);
+ if (fd < 0) {
+ error_report("Open %s failed with errno = %d.",
+ dsa_wq_path, errno);
+ return MAP_FAILED;
+ }
+ dsa_device = mmap(NULL, DSA_WQ_SIZE, PROT_WRITE,
+ MAP_SHARED | MAP_POPULATE, fd, 0);
+ close(fd);
+ if (dsa_device == MAP_FAILED) {
+ error_report("mmap failed with errno = %d.", errno);
+ return MAP_FAILED;
+ }
+ return dsa_device;
+}
+
+/**
+ * @brief Initializes a DSA device structure.
+ *
+ * @param instance A pointer to the DSA device.
+ * @param work_queue A pointer to the DSA work queue.
+ */
+static void
+dsa_device_init(struct dsa_device *instance,
+ void *dsa_work_queue)
+{
+ instance->work_queue = dsa_work_queue;
+}
+
+/**
+ * @brief Cleans up a DSA device structure.
+ *
+ * @param instance A pointer to the DSA device to cleanup.
+ */
+static void
+dsa_device_cleanup(struct dsa_device *instance)
+{
+ if (instance->work_queue != MAP_FAILED) {
+ munmap(instance->work_queue, DSA_WQ_SIZE);
+ }
+}
+
+/**
+ * @brief Initializes a DSA device group.
+ *
+ * @param group A pointer to the DSA device group.
+ * @param dsa_parameter A list of DSA device path from are separated by space
+ * character migration parameter. Multiple DSA device path.
+ *
+ * @return Zero if successful, non-zero otherwise.
+ */
+static int
+dsa_device_group_init(struct dsa_device_group *group,
+ const char *dsa_parameter)
+{
+ if (dsa_parameter == NULL || strlen(dsa_parameter) == 0) {
+ return 0;
+ }
+
+ int ret = 0;
+ char *local_dsa_parameter = g_strdup(dsa_parameter);
+ const char *dsa_path[MAX_DSA_DEVICES];
+ int num_dsa_devices = 0;
+ char delim[2] = " ";
+
+ char *current_dsa_path = strtok(local_dsa_parameter, delim);
+
+ while (current_dsa_path != NULL) {
+ dsa_path[num_dsa_devices++] = current_dsa_path;
+ if (num_dsa_devices == MAX_DSA_DEVICES) {
+ break;
+ }
+ current_dsa_path = strtok(NULL, delim);
+ }
+
+ group->dsa_devices =
+ g_new0(struct dsa_device, num_dsa_devices);
+ group->num_dsa_devices = num_dsa_devices;
+ group->device_allocator_index = 0;
+
+ group->running = false;
+ qemu_mutex_init(&group->task_queue_lock);
+ qemu_cond_init(&group->task_queue_cond);
+ QSIMPLEQ_INIT(&group->task_queue);
+
+ void *dsa_wq = MAP_FAILED;
+ for (int i = 0; i < num_dsa_devices; i++) {
+ dsa_wq = map_dsa_device(dsa_path[i]);
+ if (dsa_wq == MAP_FAILED) {
+ error_report("map_dsa_device failed MAP_FAILED.");
+ ret = -1;
+ goto exit;
+ }
+ dsa_device_init(&dsa_group.dsa_devices[i], dsa_wq);
+ }
+
+exit:
+ g_free(local_dsa_parameter);
+ return ret;
+}
+
+/**
+ * @brief Starts a DSA device group.
+ *
+ * @param group A pointer to the DSA device group.
+ */
+static void
+dsa_device_group_start(struct dsa_device_group *group)
+{
+ group->running = true;
+}
+
+/**
+ * @brief Stops a DSA device group.
+ *
+ * @param group A pointer to the DSA device group.
+ */
+__attribute__((unused))
+static void
+dsa_device_group_stop(struct dsa_device_group *group)
+{
+ group->running = false;
+}
+
+/**
+ * @brief Cleans up a DSA device group.
+ *
+ * @param group A pointer to the DSA device group.
+ */
+static void
+dsa_device_group_cleanup(struct dsa_device_group *group)
+{
+ if (!group->dsa_devices) {
+ return;
+ }
+ for (int i = 0; i < group->num_dsa_devices; i++) {
+ dsa_device_cleanup(&group->dsa_devices[i]);
+ }
+ g_free(group->dsa_devices);
+ group->dsa_devices = NULL;
+
+ qemu_mutex_destroy(&group->task_queue_lock);
+ qemu_cond_destroy(&group->task_queue_cond);
+}
+
+/**
+ * @brief Returns the next available DSA device in the group.
+ *
+ * @param group A pointer to the DSA device group.
+ *
+ * @return struct dsa_device* A pointer to the next available DSA device
+ * in the group.
+ */
+__attribute__((unused))
+static struct dsa_device *
+dsa_device_group_get_next_device(struct dsa_device_group *group)
+{
+ if (group->num_dsa_devices == 0) {
+ return NULL;
+ }
+ uint32_t current = qatomic_fetch_inc(&group->device_allocator_index);
+ current %= group->num_dsa_devices;
+ return &group->dsa_devices[current];
+}
+
+/**
+ * @brief Check if DSA is running.
+ *
+ * @return True if DSA is running, otherwise false.
+ */
+bool dsa_is_running(void)
+{
+ return false;
+}
+
+static void
+dsa_globals_init(void)
+{
+ max_retry_count = UINT64_MAX;
+}
+
+/**
+ * @brief Initializes DSA devices.
+ *
+ * @param dsa_parameter A list of DSA device path from migration parameter.
+ *
+ * @return int Zero if successful, otherwise non zero.
+ */
+int dsa_init(const char *dsa_parameter)
+{
+ dsa_globals_init();
+
+ return dsa_device_group_init(&dsa_group, dsa_parameter);
+}
+
+/**
+ * @brief Start logic to enable using DSA.
+ *
+ */
+void dsa_start(void)
+{
+ if (dsa_group.num_dsa_devices == 0) {
+ return;
+ }
+ if (dsa_group.running) {
+ return;
+ }
+ dsa_device_group_start(&dsa_group);
+}
+
+/**
+ * @brief Stop the device group and the completion thread.
+ *
+ */
+void dsa_stop(void)
+{
+ struct dsa_device_group *group = &dsa_group;
+
+ if (!group->running) {
+ return;
+ }
+}
+
+/**
+ * @brief Clean up system resources created for DSA offloading.
+ *
+ */
+void dsa_cleanup(void)
+{
+ dsa_stop();
+ dsa_device_group_cleanup(&dsa_group);
+}
+
+#endif
+
diff --git a/util/meson.build b/util/meson.build
index 2ad57b10ba..144c6812e5 100644
--- a/util/meson.build
+++ b/util/meson.build
@@ -88,6 +88,7 @@ if have_block or have_ga
endif
if have_block
util_ss.add(files('aio-wait.c'))
+ util_ss.add(files('dsa.c'))
util_ss.add(files('buffer.c'))
util_ss.add(files('bufferiszero.c'))
util_ss.add(files('hbitmap.c'))
--
2.30.2
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [PATCH v4 04/14] util/dsa: Implement DSA task enqueue and dequeue.
2024-04-25 2:21 [PATCH v4 00/14] Use Intel DSA accelerator to offload zero page checking in multifd live migration Hao Xiang
` (2 preceding siblings ...)
2024-04-25 2:21 ` [PATCH v4 03/14] util/dsa: Implement DSA device start and stop logic Hao Xiang
@ 2024-04-25 2:21 ` Hao Xiang
2024-04-25 20:55 ` Fabiano Rosas
2024-04-25 21:48 ` Fabiano Rosas
2024-04-25 2:21 ` [PATCH v4 05/14] util/dsa: Implement DSA task asynchronous completion thread model Hao Xiang
` (10 subsequent siblings)
14 siblings, 2 replies; 32+ messages in thread
From: Hao Xiang @ 2024-04-25 2:21 UTC (permalink / raw)
To: marcandre.lureau, peterx, farosas, armbru, lvivier, qemu-devel; +Cc: Hao Xiang
* Use a safe thread queue for DSA task enqueue/dequeue.
* Implement DSA task submission.
* Implement DSA batch task submission.
Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
---
include/qemu/dsa.h | 28 +++++++
util/dsa.c | 201 +++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 229 insertions(+)
diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
index f15c05ee85..37cae8d9d2 100644
--- a/include/qemu/dsa.h
+++ b/include/qemu/dsa.h
@@ -13,6 +13,34 @@
#include <linux/idxd.h>
#include "x86intrin.h"
+typedef enum DsaTaskType {
+ DSA_TASK = 0,
+ DSA_BATCH_TASK
+} DsaTaskType;
+
+typedef enum DsaTaskStatus {
+ DSA_TASK_READY = 0,
+ DSA_TASK_PROCESSING,
+ DSA_TASK_COMPLETION
+} DsaTaskStatus;
+
+typedef void (*dsa_completion_fn)(void *);
+
+typedef struct dsa_batch_task {
+ struct dsa_hw_desc batch_descriptor;
+ struct dsa_hw_desc *descriptors;
+ struct dsa_completion_record batch_completion __attribute__((aligned(32)));
+ struct dsa_completion_record *completions;
+ struct dsa_device_group *group;
+ struct dsa_device *device;
+ dsa_completion_fn completion_callback;
+ QemuSemaphore sem_task_complete;
+ DsaTaskType task_type;
+ DsaTaskStatus status;
+ int batch_size;
+ QSIMPLEQ_ENTRY(dsa_batch_task) entry;
+} dsa_batch_task;
+
/**
* @brief Initializes DSA devices.
*
diff --git a/util/dsa.c b/util/dsa.c
index 05bbf8e31a..75739a1af6 100644
--- a/util/dsa.c
+++ b/util/dsa.c
@@ -244,6 +244,205 @@ dsa_device_group_get_next_device(struct dsa_device_group *group)
return &group->dsa_devices[current];
}
+/**
+ * @brief Empties out the DSA task queue.
+ *
+ * @param group A pointer to the DSA device group.
+ */
+static void
+dsa_empty_task_queue(struct dsa_device_group *group)
+{
+ qemu_mutex_lock(&group->task_queue_lock);
+ dsa_task_queue *task_queue = &group->task_queue;
+ while (!QSIMPLEQ_EMPTY(task_queue)) {
+ QSIMPLEQ_REMOVE_HEAD(task_queue, entry);
+ }
+ qemu_mutex_unlock(&group->task_queue_lock);
+}
+
+/**
+ * @brief Adds a task to the DSA task queue.
+ *
+ * @param group A pointer to the DSA device group.
+ * @param context A pointer to the DSA task to enqueue.
+ *
+ * @return int Zero if successful, otherwise a proper error code.
+ */
+static int
+dsa_task_enqueue(struct dsa_device_group *group,
+ struct dsa_batch_task *task)
+{
+ dsa_task_queue *task_queue = &group->task_queue;
+ QemuMutex *task_queue_lock = &group->task_queue_lock;
+ QemuCond *task_queue_cond = &group->task_queue_cond;
+
+ bool notify = false;
+
+ qemu_mutex_lock(task_queue_lock);
+
+ if (!group->running) {
+ error_report("DSA: Tried to queue task to stopped device queue.");
+ qemu_mutex_unlock(task_queue_lock);
+ return -1;
+ }
+
+ /* The queue is empty. This enqueue operation is a 0->1 transition. */
+ if (QSIMPLEQ_EMPTY(task_queue)) {
+ notify = true;
+ }
+
+ QSIMPLEQ_INSERT_TAIL(task_queue, task, entry);
+
+ /* We need to notify the waiter for 0->1 transitions. */
+ if (notify) {
+ qemu_cond_signal(task_queue_cond);
+ }
+
+ qemu_mutex_unlock(task_queue_lock);
+
+ return 0;
+}
+
+/**
+ * @brief Takes a DSA task out of the task queue.
+ *
+ * @param group A pointer to the DSA device group.
+ * @return dsa_batch_task* The DSA task being dequeued.
+ */
+__attribute__((unused))
+static struct dsa_batch_task *
+dsa_task_dequeue(struct dsa_device_group *group)
+{
+ struct dsa_batch_task *task = NULL;
+ dsa_task_queue *task_queue = &group->task_queue;
+ QemuMutex *task_queue_lock = &group->task_queue_lock;
+ QemuCond *task_queue_cond = &group->task_queue_cond;
+
+ qemu_mutex_lock(task_queue_lock);
+
+ while (true) {
+ if (!group->running) {
+ goto exit;
+ }
+ task = QSIMPLEQ_FIRST(task_queue);
+ if (task != NULL) {
+ break;
+ }
+ qemu_cond_wait(task_queue_cond, task_queue_lock);
+ }
+
+ QSIMPLEQ_REMOVE_HEAD(task_queue, entry);
+
+exit:
+ qemu_mutex_unlock(task_queue_lock);
+ return task;
+}
+
+/**
+ * @brief Submits a DSA work item to the device work queue.
+ *
+ * @param wq A pointer to the DSA work queue's device memory.
+ * @param descriptor A pointer to the DSA work item descriptor.
+ *
+ * @return Zero if successful, non-zero otherwise.
+ */
+static int
+submit_wi_int(void *wq, struct dsa_hw_desc *descriptor)
+{
+ uint64_t retry = 0;
+
+ _mm_sfence();
+
+ while (true) {
+ if (_enqcmd(wq, descriptor) == 0) {
+ break;
+ }
+ retry++;
+ if (retry > max_retry_count) {
+ error_report("Submit work retry %lu times.", retry);
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+/**
+ * @brief Synchronously submits a DSA work item to the
+ * device work queue.
+ *
+ * @param wq A pointer to the DSA worjk queue's device memory.
+ * @param descriptor A pointer to the DSA work item descriptor.
+ *
+ * @return int Zero if successful, non-zero otherwise.
+ */
+__attribute__((unused))
+static int
+submit_wi(void *wq, struct dsa_hw_desc *descriptor)
+{
+ return submit_wi_int(wq, descriptor);
+}
+
+/**
+ * @brief Asynchronously submits a DSA work item to the
+ * device work queue.
+ *
+ * @param task A pointer to the buffer zero task.
+ *
+ * @return int Zero if successful, non-zero otherwise.
+ */
+__attribute__((unused))
+static int
+submit_wi_async(struct dsa_batch_task *task)
+{
+ struct dsa_device_group *device_group = task->group;
+ struct dsa_device *device_instance = task->device;
+ int ret;
+
+ assert(task->task_type == DSA_TASK);
+
+ task->status = DSA_TASK_PROCESSING;
+
+ ret = submit_wi_int(device_instance->work_queue,
+ &task->descriptors[0]);
+ if (ret != 0) {
+ return ret;
+ }
+
+ return dsa_task_enqueue(device_group, task);
+}
+
+/**
+ * @brief Asynchronously submits a DSA batch work item to the
+ * device work queue.
+ *
+ * @param dsa_batch_task A pointer to the batch buffer zero task.
+ *
+ * @return int Zero if successful, non-zero otherwise.
+ */
+__attribute__((unused))
+static int
+submit_batch_wi_async(struct dsa_batch_task *batch_task)
+{
+ struct dsa_device_group *device_group = batch_task->group;
+ struct dsa_device *device_instance = batch_task->device;
+ int ret;
+
+ assert(batch_task->task_type == DSA_BATCH_TASK);
+ assert(batch_task->batch_descriptor.desc_count <= batch_task->batch_size);
+ assert(batch_task->status == DSA_TASK_READY);
+
+ batch_task->status = DSA_TASK_PROCESSING;
+
+ ret = submit_wi_int(device_instance->work_queue,
+ &batch_task->batch_descriptor);
+ if (ret != 0) {
+ return ret;
+ }
+
+ return dsa_task_enqueue(device_group, batch_task);
+}
+
/**
* @brief Check if DSA is running.
*
@@ -300,6 +499,8 @@ void dsa_stop(void)
if (!group->running) {
return;
}
+
+ dsa_empty_task_queue(group);
}
/**
--
2.30.2
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [PATCH v4 05/14] util/dsa: Implement DSA task asynchronous completion thread model.
2024-04-25 2:21 [PATCH v4 00/14] Use Intel DSA accelerator to offload zero page checking in multifd live migration Hao Xiang
` (3 preceding siblings ...)
2024-04-25 2:21 ` [PATCH v4 04/14] util/dsa: Implement DSA task enqueue and dequeue Hao Xiang
@ 2024-04-25 2:21 ` Hao Xiang
2024-04-25 2:21 ` [PATCH v4 06/14] util/dsa: Implement zero page checking in DSA task Hao Xiang
` (9 subsequent siblings)
14 siblings, 0 replies; 32+ messages in thread
From: Hao Xiang @ 2024-04-25 2:21 UTC (permalink / raw)
To: marcandre.lureau, peterx, farosas, armbru, lvivier, qemu-devel; +Cc: Hao Xiang
* Create a dedicated thread for DSA task completion.
* DSA completion thread runs a loop and poll for completed tasks.
* Start and stop DSA completion thread during DSA device start stop.
User space application can directly submit task to Intel DSA
accelerator by writing to DSA's device memory (mapped in user space).
Once a task is submitted, the device starts processing it and write
the completion status back to the task. A user space application can
poll the task's completion status to check for completion. This change
uses a dedicated thread to perform DSA task completion checking.
Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
---
include/qemu/dsa.h | 1 +
util/dsa.c | 274 ++++++++++++++++++++++++++++++++++++++++++++-
2 files changed, 274 insertions(+), 1 deletion(-)
diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
index 37cae8d9d2..2513192a2b 100644
--- a/include/qemu/dsa.h
+++ b/include/qemu/dsa.h
@@ -38,6 +38,7 @@ typedef struct dsa_batch_task {
DsaTaskType task_type;
DsaTaskStatus status;
int batch_size;
+ bool *results;
QSIMPLEQ_ENTRY(dsa_batch_task) entry;
} dsa_batch_task;
diff --git a/util/dsa.c b/util/dsa.c
index 75739a1af6..003c4f47d9 100644
--- a/util/dsa.c
+++ b/util/dsa.c
@@ -44,6 +44,7 @@
#define DSA_WQ_SIZE 4096
#define MAX_DSA_DEVICES 16
+#define DSA_COMPLETION_THREAD "dsa_completion"
typedef QSIMPLEQ_HEAD(dsa_task_queue, dsa_batch_task) dsa_task_queue;
@@ -62,8 +63,18 @@ struct dsa_device_group {
dsa_task_queue task_queue;
};
+struct dsa_completion_thread {
+ bool stopping;
+ bool running;
+ QemuThread thread;
+ int thread_id;
+ QemuSemaphore sem_init_done;
+ struct dsa_device_group *group;
+};
+
uint64_t max_retry_count;
static struct dsa_device_group dsa_group;
+static struct dsa_completion_thread completion_thread;
/**
@@ -443,6 +454,265 @@ submit_batch_wi_async(struct dsa_batch_task *batch_task)
return dsa_task_enqueue(device_group, batch_task);
}
+/**
+ * @brief Poll for the DSA work item completion.
+ *
+ * @param completion A pointer to the DSA work item completion record.
+ * @param opcode The DSA opcode.
+ *
+ * @return Zero if successful, non-zero otherwise.
+ */
+static int
+poll_completion(struct dsa_completion_record *completion,
+ enum dsa_opcode opcode)
+{
+ uint8_t status;
+ uint64_t retry = 0;
+
+ while (true) {
+ /* The DSA operation completes successfully or fails. */
+ status = completion->status;
+ if (status == DSA_COMP_SUCCESS ||
+ status == DSA_COMP_PAGE_FAULT_NOBOF ||
+ status == DSA_COMP_BATCH_PAGE_FAULT ||
+ status == DSA_COMP_BATCH_FAIL) {
+ break;
+ } else if (status != DSA_COMP_NONE) {
+ error_report("DSA opcode %d failed with status = %d.",
+ opcode, status);
+ return 1;
+ }
+ retry++;
+ if (retry > max_retry_count) {
+ error_report("DSA wait for completion retry %lu times.", retry);
+ return 1;
+ }
+ _mm_pause();
+ }
+
+ return 0;
+}
+
+/**
+ * @brief Complete a single DSA task in the batch task.
+ *
+ * @param task A pointer to the batch task structure.
+ *
+ * @return Zero if successful, otherwise non-zero.
+ */
+static int
+poll_task_completion(struct dsa_batch_task *task)
+{
+ assert(task->task_type == DSA_TASK);
+
+ struct dsa_completion_record *completion = &task->completions[0];
+ uint8_t status;
+ int ret;
+
+ ret = poll_completion(completion, task->descriptors[0].opcode);
+ if (ret != 0) {
+ goto exit;
+ }
+
+ status = completion->status;
+ if (status == DSA_COMP_SUCCESS) {
+ task->results[0] = (completion->result == 0);
+ goto exit;
+ }
+
+ assert(status == DSA_COMP_PAGE_FAULT_NOBOF);
+
+exit:
+ return ret;
+}
+
+/**
+ * @brief Poll a batch task status until it completes. If DSA task doesn't
+ * complete properly, use CPU to complete the task.
+ *
+ * @param batch_task A pointer to the DSA batch task.
+ *
+ * @return Zero if successful, otherwise non-zero.
+ */
+static int
+poll_batch_task_completion(struct dsa_batch_task *batch_task)
+{
+ struct dsa_completion_record *batch_completion =
+ &batch_task->batch_completion;
+ struct dsa_completion_record *completion;
+ uint8_t batch_status;
+ uint8_t status;
+ bool *results = batch_task->results;
+ uint32_t count = batch_task->batch_descriptor.desc_count;
+ int ret;
+
+ ret = poll_completion(batch_completion,
+ batch_task->batch_descriptor.opcode);
+ if (ret != 0) {
+ goto exit;
+ }
+
+ batch_status = batch_completion->status;
+
+ if (batch_status == DSA_COMP_SUCCESS) {
+ if (batch_completion->bytes_completed == count) {
+ /*
+ * Let's skip checking for each descriptors' completion status
+ * if the batch descriptor says all succedded.
+ */
+ for (int i = 0; i < count; i++) {
+ assert(batch_task->completions[i].status == DSA_COMP_SUCCESS);
+ results[i] = (batch_task->completions[i].result == 0);
+ }
+ goto exit;
+ }
+ } else {
+ assert(batch_status == DSA_COMP_BATCH_FAIL ||
+ batch_status == DSA_COMP_BATCH_PAGE_FAULT);
+ }
+
+ for (int i = 0; i < count; i++) {
+
+ completion = &batch_task->completions[i];
+ status = completion->status;
+
+ if (status == DSA_COMP_SUCCESS) {
+ results[i] = (completion->result == 0);
+ continue;
+ }
+
+ assert(status == DSA_COMP_PAGE_FAULT_NOBOF);
+
+ if (status != DSA_COMP_PAGE_FAULT_NOBOF) {
+ error_report("Unexpected DSA completion status = %u.", status);
+ ret = 1;
+ goto exit;
+ }
+ }
+
+exit:
+ return ret;
+}
+
+/**
+ * @brief Handles an asynchronous DSA batch task completion.
+ *
+ * @param task A pointer to the batch buffer zero task structure.
+ */
+static void
+dsa_batch_task_complete(struct dsa_batch_task *batch_task)
+{
+ batch_task->status = DSA_TASK_COMPLETION;
+ batch_task->completion_callback(batch_task);
+}
+
+/**
+ * @brief The function entry point called by a dedicated DSA
+ * work item completion thread.
+ *
+ * @param opaque A pointer to the thread context.
+ *
+ * @return void* Not used.
+ */
+static void *
+dsa_completion_loop(void *opaque)
+{
+ struct dsa_completion_thread *thread_context =
+ (struct dsa_completion_thread *)opaque;
+ struct dsa_batch_task *batch_task;
+ struct dsa_device_group *group = thread_context->group;
+ int ret;
+
+ rcu_register_thread();
+
+ thread_context->thread_id = qemu_get_thread_id();
+ qemu_sem_post(&thread_context->sem_init_done);
+
+ while (thread_context->running) {
+ batch_task = dsa_task_dequeue(group);
+ assert(batch_task != NULL || !group->running);
+ if (!group->running) {
+ assert(!thread_context->running);
+ break;
+ }
+ if (batch_task->task_type == DSA_TASK) {
+ ret = poll_task_completion(batch_task);
+ } else {
+ assert(batch_task->task_type == DSA_BATCH_TASK);
+ ret = poll_batch_task_completion(batch_task);
+ }
+
+ if (ret != 0) {
+ goto exit;
+ }
+
+ dsa_batch_task_complete(batch_task);
+ }
+
+exit:
+ if (ret != 0) {
+ error_report("DSA completion thread exited due to internal error.");
+ }
+ rcu_unregister_thread();
+ return NULL;
+}
+
+/**
+ * @brief Initializes a DSA completion thread.
+ *
+ * @param completion_thread A pointer to the completion thread context.
+ * @param group A pointer to the DSA device group.
+ */
+static void
+dsa_completion_thread_init(
+ struct dsa_completion_thread *completion_thread,
+ struct dsa_device_group *group)
+{
+ completion_thread->stopping = false;
+ completion_thread->running = true;
+ completion_thread->thread_id = -1;
+ qemu_sem_init(&completion_thread->sem_init_done, 0);
+ completion_thread->group = group;
+
+ qemu_thread_create(&completion_thread->thread,
+ DSA_COMPLETION_THREAD,
+ dsa_completion_loop,
+ completion_thread,
+ QEMU_THREAD_JOINABLE);
+
+ /* Wait for initialization to complete */
+ qemu_sem_wait(&completion_thread->sem_init_done);
+}
+
+/**
+ * @brief Stops the completion thread (and implicitly, the device group).
+ *
+ * @param opaque A pointer to the completion thread.
+ */
+static void dsa_completion_thread_stop(void *opaque)
+{
+ struct dsa_completion_thread *thread_context =
+ (struct dsa_completion_thread *)opaque;
+
+ struct dsa_device_group *group = thread_context->group;
+
+ qemu_mutex_lock(&group->task_queue_lock);
+
+ thread_context->stopping = true;
+ thread_context->running = false;
+
+ /* Prevent the compiler from setting group->running first. */
+ barrier();
+ dsa_device_group_stop(group);
+
+ qemu_cond_signal(&group->task_queue_cond);
+ qemu_mutex_unlock(&group->task_queue_lock);
+
+ qemu_thread_join(&thread_context->thread);
+
+ qemu_sem_destroy(&thread_context->sem_init_done);
+}
+
/**
* @brief Check if DSA is running.
*
@@ -450,7 +720,7 @@ submit_batch_wi_async(struct dsa_batch_task *batch_task)
*/
bool dsa_is_running(void)
{
- return false;
+ return completion_thread.running;
}
static void
@@ -486,6 +756,7 @@ void dsa_start(void)
return;
}
dsa_device_group_start(&dsa_group);
+ dsa_completion_thread_init(&completion_thread, &dsa_group);
}
/**
@@ -500,6 +771,7 @@ void dsa_stop(void)
return;
}
+ dsa_completion_thread_stop(&completion_thread);
dsa_empty_task_queue(group);
}
--
2.30.2
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [PATCH v4 06/14] util/dsa: Implement zero page checking in DSA task.
2024-04-25 2:21 [PATCH v4 00/14] Use Intel DSA accelerator to offload zero page checking in multifd live migration Hao Xiang
` (4 preceding siblings ...)
2024-04-25 2:21 ` [PATCH v4 05/14] util/dsa: Implement DSA task asynchronous completion thread model Hao Xiang
@ 2024-04-25 2:21 ` Hao Xiang
2024-04-25 2:21 ` [PATCH v4 07/14] util/dsa: Implement DSA task asynchronous submission and wait for completion Hao Xiang
` (8 subsequent siblings)
14 siblings, 0 replies; 32+ messages in thread
From: Hao Xiang @ 2024-04-25 2:21 UTC (permalink / raw)
To: marcandre.lureau, peterx, farosas, armbru, lvivier, qemu-devel
Cc: Hao Xiang, Bryan Zhang
Create DSA task with operation code DSA_OPCODE_COMPVAL.
Here we create two types of DSA tasks, a single DSA task and
a batch DSA task. Batch DSA task reduces task submission overhead
and hence should be the default option. However, due to the way DSA
hardware works, a DSA batch task must contain at least two individual
tasks. There are times we need to submit a single task and hence a
single DSA task submission is also required.
Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
---
include/qemu/dsa.h | 18 ++++
util/dsa.c | 247 +++++++++++++++++++++++++++++++++++++++++----
2 files changed, 244 insertions(+), 21 deletions(-)
diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
index 2513192a2b..645e6fc367 100644
--- a/include/qemu/dsa.h
+++ b/include/qemu/dsa.h
@@ -73,6 +73,24 @@ void dsa_cleanup(void);
*/
bool dsa_is_running(void);
+/**
+ * @brief Initializes a buffer zero batch task.
+ *
+ * @param task A pointer to the batch task to initialize.
+ * @param results A pointer to an array of zero page checking results.
+ * @param batch_size The number of DSA tasks in the batch.
+ */
+void
+buffer_zero_batch_task_init(struct dsa_batch_task *task,
+ bool *results, int batch_size);
+
+/**
+ * @brief Performs the proper cleanup on a DSA batch task.
+ *
+ * @param task A pointer to the batch task to cleanup.
+ */
+void buffer_zero_batch_task_destroy(struct dsa_batch_task *task);
+
#else
static inline bool dsa_is_running(void)
diff --git a/util/dsa.c b/util/dsa.c
index 003c4f47d9..9db4cfcf1d 100644
--- a/util/dsa.c
+++ b/util/dsa.c
@@ -76,6 +76,7 @@ uint64_t max_retry_count;
static struct dsa_device_group dsa_group;
static struct dsa_completion_thread completion_thread;
+static void buffer_zero_dsa_completion(void *context);
/**
* @brief This function opens a DSA device's work queue and
@@ -207,7 +208,6 @@ dsa_device_group_start(struct dsa_device_group *group)
*
* @param group A pointer to the DSA device group.
*/
-__attribute__((unused))
static void
dsa_device_group_stop(struct dsa_device_group *group)
{
@@ -243,7 +243,6 @@ dsa_device_group_cleanup(struct dsa_device_group *group)
* @return struct dsa_device* A pointer to the next available DSA device
* in the group.
*/
-__attribute__((unused))
static struct dsa_device *
dsa_device_group_get_next_device(struct dsa_device_group *group)
{
@@ -320,7 +319,6 @@ dsa_task_enqueue(struct dsa_device_group *group,
* @param group A pointer to the DSA device group.
* @return dsa_batch_task* The DSA task being dequeued.
*/
-__attribute__((unused))
static struct dsa_batch_task *
dsa_task_dequeue(struct dsa_device_group *group)
{
@@ -378,22 +376,6 @@ submit_wi_int(void *wq, struct dsa_hw_desc *descriptor)
return 0;
}
-/**
- * @brief Synchronously submits a DSA work item to the
- * device work queue.
- *
- * @param wq A pointer to the DSA worjk queue's device memory.
- * @param descriptor A pointer to the DSA work item descriptor.
- *
- * @return int Zero if successful, non-zero otherwise.
- */
-__attribute__((unused))
-static int
-submit_wi(void *wq, struct dsa_hw_desc *descriptor)
-{
- return submit_wi_int(wq, descriptor);
-}
-
/**
* @brief Asynchronously submits a DSA work item to the
* device work queue.
@@ -402,7 +384,6 @@ submit_wi(void *wq, struct dsa_hw_desc *descriptor)
*
* @return int Zero if successful, non-zero otherwise.
*/
-__attribute__((unused))
static int
submit_wi_async(struct dsa_batch_task *task)
{
@@ -431,7 +412,6 @@ submit_wi_async(struct dsa_batch_task *task)
*
* @return int Zero if successful, non-zero otherwise.
*/
-__attribute__((unused))
static int
submit_batch_wi_async(struct dsa_batch_task *batch_task)
{
@@ -713,6 +693,231 @@ static void dsa_completion_thread_stop(void *opaque)
qemu_sem_destroy(&thread_context->sem_init_done);
}
+/**
+ * @brief Initializes a buffer zero comparison DSA task.
+ *
+ * @param descriptor A pointer to the DSA task descriptor.
+ * @param completion A pointer to the DSA task completion record.
+ */
+static void
+buffer_zero_task_init_int(struct dsa_hw_desc *descriptor,
+ struct dsa_completion_record *completion)
+{
+ descriptor->opcode = DSA_OPCODE_COMPVAL;
+ descriptor->flags = IDXD_OP_FLAG_RCR | IDXD_OP_FLAG_CRAV;
+ descriptor->comp_pattern = (uint64_t)0;
+ descriptor->completion_addr = (uint64_t)completion;
+}
+
+/**
+ * @brief Initializes a buffer zero batch task.
+ *
+ * @param task A pointer to the batch task to initialize.
+ * @param results A pointer to an array of zero page checking results.
+ * @param batch_size The number of DSA tasks in the batch.
+ */
+void
+buffer_zero_batch_task_init(struct dsa_batch_task *task,
+ bool *results, int batch_size)
+{
+ int descriptors_size = sizeof(*task->descriptors) * batch_size;
+ memset(task, 0, sizeof(*task));
+
+ task->descriptors =
+ (struct dsa_hw_desc *)qemu_memalign(64, descriptors_size);
+ memset(task->descriptors, 0, descriptors_size);
+ task->completions = (struct dsa_completion_record *)qemu_memalign(
+ 32, sizeof(*task->completions) * batch_size);
+ task->results = results;
+ task->batch_size = batch_size;
+
+ task->batch_completion.status = DSA_COMP_NONE;
+ task->batch_descriptor.completion_addr = (uint64_t)&task->batch_completion;
+ /* TODO: Ensure that we never send a batch with count <= 1 */
+ task->batch_descriptor.desc_count = 0;
+ task->batch_descriptor.opcode = DSA_OPCODE_BATCH;
+ task->batch_descriptor.flags = IDXD_OP_FLAG_RCR | IDXD_OP_FLAG_CRAV;
+ task->batch_descriptor.desc_list_addr = (uintptr_t)task->descriptors;
+ task->status = DSA_TASK_READY;
+ task->group = &dsa_group;
+ task->device = dsa_device_group_get_next_device(&dsa_group);
+
+ for (int i = 0; i < task->batch_size; i++) {
+ buffer_zero_task_init_int(&task->descriptors[i],
+ &task->completions[i]);
+ }
+
+ qemu_sem_init(&task->sem_task_complete, 0);
+ task->completion_callback = buffer_zero_dsa_completion;
+}
+
+/**
+ * @brief Performs the proper cleanup on a DSA batch task.
+ *
+ * @param task A pointer to the batch task to cleanup.
+ */
+void
+buffer_zero_batch_task_destroy(struct dsa_batch_task *task)
+{
+ qemu_vfree(task->descriptors);
+ qemu_vfree(task->completions);
+ task->results = NULL;
+
+ qemu_sem_destroy(&task->sem_task_complete);
+}
+
+/**
+ * @brief Resets a buffer zero comparison DSA batch task.
+ *
+ * @param task A pointer to the batch task.
+ * @param count The number of DSA tasks this batch task will contain.
+ */
+static void
+buffer_zero_batch_task_reset(struct dsa_batch_task *task, size_t count)
+{
+ task->batch_completion.status = DSA_COMP_NONE;
+ task->batch_descriptor.desc_count = count;
+ task->task_type = DSA_BATCH_TASK;
+ task->status = DSA_TASK_READY;
+}
+
+/**
+ * @brief Sets a buffer zero comparison DSA task.
+ *
+ * @param descriptor A pointer to the DSA task descriptor.
+ * @param buf A pointer to the memory buffer.
+ * @param len The length of the buffer.
+ */
+static void
+buffer_zero_task_set_int(struct dsa_hw_desc *descriptor,
+ const void *buf,
+ size_t len)
+{
+ struct dsa_completion_record *completion =
+ (struct dsa_completion_record *)descriptor->completion_addr;
+
+ descriptor->xfer_size = len;
+ descriptor->src_addr = (uintptr_t)buf;
+ completion->status = 0;
+ completion->result = 0;
+}
+
+/**
+ * @brief Resets a buffer zero comparison DSA batch task.
+ *
+ * @param task A pointer to the DSA batch task.
+ */
+static void
+buffer_zero_task_reset(struct dsa_batch_task *task)
+{
+ task->completions[0].status = DSA_COMP_NONE;
+ task->task_type = DSA_TASK;
+ task->status = DSA_TASK_READY;
+}
+
+/**
+ * @brief Sets a buffer zero comparison DSA task.
+ *
+ * @param task A pointer to the DSA task.
+ * @param buf A pointer to the memory buffer.
+ * @param len The buffer length.
+ */
+static void
+buffer_zero_task_set(struct dsa_batch_task *task,
+ const void *buf,
+ size_t len)
+{
+ buffer_zero_task_reset(task);
+ buffer_zero_task_set_int(&task->descriptors[0], buf, len);
+}
+
+/**
+ * @brief Sets a buffer zero comparison batch task.
+ *
+ * @param batch_task A pointer to the batch task.
+ * @param buf An array of memory buffers.
+ * @param count The number of buffers in the array.
+ * @param len The length of the buffers.
+ */
+static void
+buffer_zero_batch_task_set(struct dsa_batch_task *batch_task,
+ const void **buf, size_t count, size_t len)
+{
+ assert(count > 0);
+ assert(count <= batch_task->batch_size);
+
+ buffer_zero_batch_task_reset(batch_task, count);
+ for (int i = 0; i < count; i++) {
+ buffer_zero_task_set_int(&batch_task->descriptors[i], buf[i], len);
+ }
+}
+
+/**
+ * @brief Asychronously perform a buffer zero DSA operation.
+ *
+ * @param task A pointer to the batch task structure.
+ * @param buf A pointer to the memory buffer.
+ * @param len The length of the memory buffer.
+ *
+ * @return int Zero if successful, otherwise an appropriate error code.
+ */
+__attribute__((unused))
+static int
+buffer_zero_dsa_async(struct dsa_batch_task *task,
+ const void *buf, size_t len)
+{
+ buffer_zero_task_set(task, buf, len);
+
+ return submit_wi_async(task);
+}
+
+/**
+ * @brief Sends a memory comparison batch task to a DSA device and wait
+ * for completion.
+ *
+ * @param batch_task The batch task to be submitted to DSA device.
+ * @param buf An array of memory buffers to check for zero.
+ * @param count The number of buffers.
+ * @param len The buffer length.
+ */
+__attribute__((unused))
+static int
+buffer_zero_dsa_batch_async(struct dsa_batch_task *batch_task,
+ const void **buf, size_t count, size_t len)
+{
+ assert(count <= batch_task->batch_size);
+ buffer_zero_batch_task_set(batch_task, buf, count, len);
+
+ return submit_batch_wi_async(batch_task);
+}
+
+/**
+ * @brief The completion callback function for buffer zero
+ * comparison DSA task completion.
+ *
+ * @param context A pointer to the callback context.
+ */
+static void
+buffer_zero_dsa_completion(void *context)
+{
+ assert(context != NULL);
+
+ struct dsa_batch_task *task = (struct dsa_batch_task *)context;
+ qemu_sem_post(&task->sem_task_complete);
+}
+
+/**
+ * @brief Wait for the asynchronous DSA task to complete.
+ *
+ * @param batch_task A pointer to the buffer zero comparison batch task.
+ */
+__attribute__((unused))
+static void
+buffer_zero_dsa_wait(struct dsa_batch_task *batch_task)
+{
+ qemu_sem_wait(&batch_task->sem_task_complete);
+}
+
/**
* @brief Check if DSA is running.
*
--
2.30.2
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [PATCH v4 07/14] util/dsa: Implement DSA task asynchronous submission and wait for completion.
2024-04-25 2:21 [PATCH v4 00/14] Use Intel DSA accelerator to offload zero page checking in multifd live migration Hao Xiang
` (5 preceding siblings ...)
2024-04-25 2:21 ` [PATCH v4 06/14] util/dsa: Implement zero page checking in DSA task Hao Xiang
@ 2024-04-25 2:21 ` Hao Xiang
2024-05-01 18:59 ` Peter Xu
2024-04-25 2:21 ` [PATCH v4 08/14] migration/multifd: Add new migration option for multifd DSA offloading Hao Xiang
` (7 subsequent siblings)
14 siblings, 1 reply; 32+ messages in thread
From: Hao Xiang @ 2024-04-25 2:21 UTC (permalink / raw)
To: marcandre.lureau, peterx, farosas, armbru, lvivier, qemu-devel
Cc: Hao Xiang, Bryan Zhang
* Add a DSA task completion callback.
* DSA completion thread will call the tasks's completion callback
on every task/batch task completion.
* DSA submission path to wait for completion.
* Implement CPU fallback if DSA is not able to complete the task.
Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
---
include/qemu/dsa.h | 14 +++++
util/dsa.c | 147 ++++++++++++++++++++++++++++++++++++++++++++-
2 files changed, 158 insertions(+), 3 deletions(-)
diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
index 645e6fc367..e002652879 100644
--- a/include/qemu/dsa.h
+++ b/include/qemu/dsa.h
@@ -91,6 +91,20 @@ buffer_zero_batch_task_init(struct dsa_batch_task *task,
*/
void buffer_zero_batch_task_destroy(struct dsa_batch_task *task);
+/**
+ * @brief Performs buffer zero comparison on a DSA batch task asynchronously.
+ *
+ * @param batch_task A pointer to the batch task.
+ * @param buf An array of memory buffers.
+ * @param count The number of buffers in the array.
+ * @param len The buffer length.
+ *
+ * @return Zero if successful, otherwise non-zero.
+ */
+int
+buffer_is_zero_dsa_batch_async(struct dsa_batch_task *batch_task,
+ const void **buf, size_t count, size_t len);
+
#else
static inline bool dsa_is_running(void)
diff --git a/util/dsa.c b/util/dsa.c
index 9db4cfcf1d..5a2bf33651 100644
--- a/util/dsa.c
+++ b/util/dsa.c
@@ -473,6 +473,57 @@ poll_completion(struct dsa_completion_record *completion,
return 0;
}
+/**
+ * @brief Helper function to use CPU to complete a single
+ * zero page checking task.
+ *
+ * @param completion A pointer to a DSA task completion record.
+ * @param descriptor A pointer to a DSA task descriptor.
+ * @param result A pointer to the result of a zero page checking.
+ */
+static void
+task_cpu_fallback_int(struct dsa_completion_record *completion,
+ struct dsa_hw_desc *descriptor, bool *result)
+{
+ const uint8_t *buf;
+ size_t len;
+
+ if (completion->status == DSA_COMP_SUCCESS) {
+ return;
+ }
+
+ /*
+ * DSA was able to partially complete the operation. Check the
+ * result. If we already know this is not a zero page, we can
+ * return now.
+ */
+ if (completion->bytes_completed != 0 && completion->result != 0) {
+ *result = false;
+ return;
+ }
+
+ /* Let's fallback to use CPU to complete it. */
+ buf = (const uint8_t *)descriptor->src_addr;
+ len = descriptor->xfer_size;
+ *result = buffer_is_zero(buf + completion->bytes_completed,
+ len - completion->bytes_completed);
+}
+
+/**
+ * @brief Use CPU to complete a single zero page checking task.
+ *
+ * @param task A pointer to the task.
+ */
+static void
+task_cpu_fallback(struct dsa_batch_task *task)
+{
+ assert(task->task_type == DSA_TASK);
+
+ task_cpu_fallback_int(&task->completions[0],
+ &task->descriptors[0],
+ &task->results[0]);
+}
+
/**
* @brief Complete a single DSA task in the batch task.
*
@@ -574,6 +625,47 @@ exit:
return ret;
}
+/**
+ * @brief Use CPU to complete the zero page checking batch task.
+ *
+ * @param batch_task A pointer to the batch task.
+ */
+static void
+batch_task_cpu_fallback(struct dsa_batch_task *batch_task)
+{
+ assert(batch_task->task_type == DSA_BATCH_TASK);
+
+ struct dsa_completion_record *batch_completion =
+ &batch_task->batch_completion;
+ struct dsa_completion_record *completion;
+ uint8_t status;
+ bool *results = batch_task->results;
+ uint32_t count = batch_task->batch_descriptor.desc_count;
+
+ /* DSA is able to complete the entire batch task. */
+ if (batch_completion->status == DSA_COMP_SUCCESS) {
+ assert(count == batch_completion->bytes_completed);
+ return;
+ }
+
+ /*
+ * DSA encounters some error and is not able to complete
+ * the entire batch task. Use CPU fallback.
+ */
+ for (int i = 0; i < count; i++) {
+
+ completion = &batch_task->completions[i];
+ status = completion->status;
+
+ assert(status == DSA_COMP_SUCCESS ||
+ status == DSA_COMP_PAGE_FAULT_NOBOF);
+
+ task_cpu_fallback_int(completion,
+ &batch_task->descriptors[i],
+ &results[i]);
+ }
+}
+
/**
* @brief Handles an asynchronous DSA batch task completion.
*
@@ -861,7 +953,6 @@ buffer_zero_batch_task_set(struct dsa_batch_task *batch_task,
*
* @return int Zero if successful, otherwise an appropriate error code.
*/
-__attribute__((unused))
static int
buffer_zero_dsa_async(struct dsa_batch_task *task,
const void *buf, size_t len)
@@ -880,7 +971,6 @@ buffer_zero_dsa_async(struct dsa_batch_task *task,
* @param count The number of buffers.
* @param len The buffer length.
*/
-__attribute__((unused))
static int
buffer_zero_dsa_batch_async(struct dsa_batch_task *batch_task,
const void **buf, size_t count, size_t len)
@@ -911,13 +1001,29 @@ buffer_zero_dsa_completion(void *context)
*
* @param batch_task A pointer to the buffer zero comparison batch task.
*/
-__attribute__((unused))
static void
buffer_zero_dsa_wait(struct dsa_batch_task *batch_task)
{
qemu_sem_wait(&batch_task->sem_task_complete);
}
+/**
+ * @brief Use CPU to complete the zero page checking task if DSA
+ * is not able to complete it.
+ *
+ * @param batch_task A pointer to the batch task.
+ */
+static void
+buffer_zero_cpu_fallback(struct dsa_batch_task *batch_task)
+{
+ if (batch_task->task_type == DSA_TASK) {
+ task_cpu_fallback(batch_task);
+ } else {
+ assert(batch_task->task_type == DSA_BATCH_TASK);
+ batch_task_cpu_fallback(batch_task);
+ }
+}
+
/**
* @brief Check if DSA is running.
*
@@ -990,5 +1096,40 @@ void dsa_cleanup(void)
dsa_device_group_cleanup(&dsa_group);
}
+/**
+ * @brief Performs buffer zero comparison on a DSA batch task asynchronously.
+ *
+ * @param batch_task A pointer to the batch task.
+ * @param buf An array of memory buffers.
+ * @param count The number of buffers in the array.
+ * @param len The buffer length.
+ *
+ * @return Zero if successful, otherwise non-zero.
+ */
+int
+buffer_is_zero_dsa_batch_async(struct dsa_batch_task *batch_task,
+ const void **buf, size_t count, size_t len)
+{
+ if (count <= 0 || count > batch_task->batch_size) {
+ return -1;
+ }
+
+ assert(batch_task != NULL);
+ assert(len != 0);
+ assert(buf != NULL);
+
+ if (count == 1) {
+ /* DSA doesn't take batch operation with only 1 task. */
+ buffer_zero_dsa_async(batch_task, buf[0], len);
+ } else {
+ buffer_zero_dsa_batch_async(batch_task, buf, count, len);
+ }
+
+ buffer_zero_dsa_wait(batch_task);
+ buffer_zero_cpu_fallback(batch_task);
+
+ return 0;
+}
+
#endif
--
2.30.2
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [PATCH v4 08/14] migration/multifd: Add new migration option for multifd DSA offloading.
2024-04-25 2:21 [PATCH v4 00/14] Use Intel DSA accelerator to offload zero page checking in multifd live migration Hao Xiang
` (6 preceding siblings ...)
2024-04-25 2:21 ` [PATCH v4 07/14] util/dsa: Implement DSA task asynchronous submission and wait for completion Hao Xiang
@ 2024-04-25 2:21 ` Hao Xiang
2024-04-25 14:17 ` Daniel P. Berrangé
2024-04-25 2:21 ` [PATCH v4 09/14] migration/multifd: Prepare to introduce DSA acceleration on the multifd path Hao Xiang
` (6 subsequent siblings)
14 siblings, 1 reply; 32+ messages in thread
From: Hao Xiang @ 2024-04-25 2:21 UTC (permalink / raw)
To: marcandre.lureau, peterx, farosas, armbru, lvivier, qemu-devel; +Cc: Hao Xiang
Intel DSA offloading is an optional feature that turns on if
proper hardware and software stack is available. To turn on
DSA offloading in multifd live migration:
multifd-dsa-accel="[dsa_dev_path1] [dsa_dev_path2] ... [dsa_dev_pathX]"
This feature is turned off by default.
Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
---
migration/migration-hmp-cmds.c | 8 ++++++++
migration/options.c | 30 ++++++++++++++++++++++++++++++
migration/options.h | 1 +
qapi/migration.json | 26 +++++++++++++++++++++++---
4 files changed, 62 insertions(+), 3 deletions(-)
diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c
index 7e96ae6ffd..7e9bb278c9 100644
--- a/migration/migration-hmp-cmds.c
+++ b/migration/migration-hmp-cmds.c
@@ -358,6 +358,9 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
monitor_printf(mon, "%s: '%s'\n",
MigrationParameter_str(MIGRATION_PARAMETER_TLS_AUTHZ),
params->tls_authz);
+ monitor_printf(mon, "%s: '%s'\n",
+ MigrationParameter_str(MIGRATION_PARAMETER_MULTIFD_DSA_ACCEL),
+ params->multifd_dsa_accel);
if (params->has_block_bitmap_mapping) {
const BitmapMigrationNodeAliasList *bmnal;
@@ -622,6 +625,11 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
p->has_block_incremental = true;
visit_type_bool(v, param, &p->block_incremental, &err);
break;
+ case MIGRATION_PARAMETER_MULTIFD_DSA_ACCEL:
+ p->multifd_dsa_accel = g_new0(StrOrNull, 1);
+ p->multifd_dsa_accel->type = QTYPE_QSTRING;
+ visit_type_str(v, param, &p->multifd_dsa_accel->u.s, &err);
+ break;
case MIGRATION_PARAMETER_MULTIFD_CHANNELS:
p->has_multifd_channels = true;
visit_type_uint8(v, param, &p->multifd_channels, &err);
diff --git a/migration/options.c b/migration/options.c
index 239f5ecfb4..dc8642df81 100644
--- a/migration/options.c
+++ b/migration/options.c
@@ -182,6 +182,8 @@ Property migration_properties[] = {
DEFINE_PROP_ZERO_PAGE_DETECTION("zero-page-detection", MigrationState,
parameters.zero_page_detection,
ZERO_PAGE_DETECTION_MULTIFD),
+ DEFINE_PROP_STRING("multifd-dsa-accel", MigrationState,
+ parameters.multifd_dsa_accel),
/* Migration capabilities */
DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE),
@@ -920,6 +922,13 @@ const char *migrate_tls_creds(void)
return s->parameters.tls_creds;
}
+const char *migrate_multifd_dsa_accel(void)
+{
+ MigrationState *s = migrate_get_current();
+
+ return s->parameters.multifd_dsa_accel;
+}
+
const char *migrate_tls_hostname(void)
{
MigrationState *s = migrate_get_current();
@@ -1060,6 +1069,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
params->mode = s->parameters.mode;
params->has_zero_page_detection = true;
params->zero_page_detection = s->parameters.zero_page_detection;
+ params->multifd_dsa_accel = g_strdup(s->parameters.multifd_dsa_accel ?
+ s->parameters.multifd_dsa_accel : "");
return params;
}
@@ -1068,6 +1079,7 @@ void migrate_params_init(MigrationParameters *params)
{
params->tls_hostname = g_strdup("");
params->tls_creds = g_strdup("");
+ params->multifd_dsa_accel = g_strdup("");
/* Set has_* up only for parameter checks */
params->has_compress_level = true;
@@ -1416,6 +1428,11 @@ static void migrate_params_test_apply(MigrateSetParameters *params,
if (params->has_zero_page_detection) {
dest->zero_page_detection = params->zero_page_detection;
}
+
+ if (params->multifd_dsa_accel) {
+ assert(params->multifd_dsa_accel->type == QTYPE_QSTRING);
+ dest->multifd_dsa_accel = params->multifd_dsa_accel->u.s;
+ }
}
static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
@@ -1570,6 +1587,13 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
if (params->has_zero_page_detection) {
s->parameters.zero_page_detection = params->zero_page_detection;
}
+
+ if (params->multifd_dsa_accel) {
+ g_free(s->parameters.multifd_dsa_accel);
+ assert(params->multifd_dsa_accel->type == QTYPE_QSTRING);
+ s->parameters.multifd_dsa_accel =
+ g_strdup(params->multifd_dsa_accel->u.s);
+ }
}
void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp)
@@ -1595,6 +1619,12 @@ void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp)
params->tls_authz->type = QTYPE_QSTRING;
params->tls_authz->u.s = strdup("");
}
+ if (params->multifd_dsa_accel
+ && params->multifd_dsa_accel->type == QTYPE_QNULL) {
+ qobject_unref(params->multifd_dsa_accel->u.n);
+ params->multifd_dsa_accel->type = QTYPE_QSTRING;
+ params->multifd_dsa_accel->u.s = strdup("");
+ }
migrate_params_test_apply(params, &tmp);
diff --git a/migration/options.h b/migration/options.h
index ab8199e207..1cb3393be9 100644
--- a/migration/options.h
+++ b/migration/options.h
@@ -91,6 +91,7 @@ const char *migrate_tls_creds(void);
const char *migrate_tls_hostname(void);
uint64_t migrate_xbzrle_cache_size(void);
ZeroPageDetection migrate_zero_page_detection(void);
+const char *migrate_multifd_dsa_accel(void);
/* parameters setters */
diff --git a/qapi/migration.json b/qapi/migration.json
index 8c65b90328..934fa8839e 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -914,6 +914,12 @@
# See description in @ZeroPageDetection. Default is 'multifd'.
# (since 9.0)
#
+# @multifd-dsa-accel: If enabled, use DSA accelerator offloading for
+# certain memory operations. Enable DSA accelerator offloading by
+# setting this string to a list of DSA device path separated by space
+# characters. Setting this string to an empty string means disabling
+# DSA accelerator offloading. Defaults to an empty string. (since 9.2)
+#
# Features:
#
# @deprecated: Member @block-incremental is deprecated. Use
@@ -937,7 +943,7 @@
'cpu-throttle-initial', 'cpu-throttle-increment',
'cpu-throttle-tailslow',
'tls-creds', 'tls-hostname', 'tls-authz', 'max-bandwidth',
- 'avail-switchover-bandwidth', 'downtime-limit',
+ 'avail-switchover-bandwidth', 'downtime-limit', 'multifd-dsa-accel',
{ 'name': 'x-checkpoint-delay', 'features': [ 'unstable' ] },
{ 'name': 'block-incremental', 'features': [ 'deprecated' ] },
'multifd-channels',
@@ -1122,6 +1128,12 @@
# See description in @ZeroPageDetection. Default is 'multifd'.
# (since 9.0)
#
+# @multifd-dsa-accel: If enabled, use DSA accelerator offloading for
+# certain memory operations. Enable DSA accelerator offloading by
+# setting this string to a list of DSA device path separated by space
+# characters. Setting this string to an empty string means disabling
+# DSA accelerator offloading. Defaults to an empty string. (since 9.2)
+#
# Features:
#
# @deprecated: Member @block-incremental is deprecated. Use
@@ -1176,7 +1188,8 @@
'features': [ 'unstable' ] },
'*vcpu-dirty-limit': 'uint64',
'*mode': 'MigMode',
- '*zero-page-detection': 'ZeroPageDetection'} }
+ '*zero-page-detection': 'ZeroPageDetection',
+ '*multifd-dsa-accel': 'StrOrNull'} }
##
# @migrate-set-parameters:
@@ -1354,6 +1367,12 @@
# See description in @ZeroPageDetection. Default is 'multifd'.
# (since 9.0)
#
+# @multifd-dsa-accel: If enabled, use DSA accelerator offloading for
+# certain memory operations. Enable DSA accelerator offloading by
+# setting this string to a list of DSA device path separated by space
+# characters. Setting this string to an empty string means disabling
+# DSA accelerator offloading. Defaults to an empty string. (since 9.2)
+#
# Features:
#
# @deprecated: Member @block-incremental is deprecated. Use
@@ -1405,7 +1424,8 @@
'features': [ 'unstable' ] },
'*vcpu-dirty-limit': 'uint64',
'*mode': 'MigMode',
- '*zero-page-detection': 'ZeroPageDetection'} }
+ '*zero-page-detection': 'ZeroPageDetection',
+ '*multifd-dsa-accel': 'str'} }
##
# @query-migrate-parameters:
--
2.30.2
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [PATCH v4 09/14] migration/multifd: Prepare to introduce DSA acceleration on the multifd path.
2024-04-25 2:21 [PATCH v4 00/14] Use Intel DSA accelerator to offload zero page checking in multifd live migration Hao Xiang
` (7 preceding siblings ...)
2024-04-25 2:21 ` [PATCH v4 08/14] migration/multifd: Add new migration option for multifd DSA offloading Hao Xiang
@ 2024-04-25 2:21 ` Hao Xiang
2024-05-01 19:18 ` Peter Xu
2024-04-25 2:21 ` [PATCH v4 10/14] migration/multifd: Enable DSA offloading in multifd sender path Hao Xiang
` (5 subsequent siblings)
14 siblings, 1 reply; 32+ messages in thread
From: Hao Xiang @ 2024-04-25 2:21 UTC (permalink / raw)
To: marcandre.lureau, peterx, farosas, armbru, lvivier, qemu-devel; +Cc: Hao Xiang
1. Refactor multifd_send_thread function.
2. Introduce the batch task structure in MultiFDSendParams.
Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
---
include/qemu/dsa.h | 51 +++++++++++++++++++++++++++++++++++++++++++--
migration/multifd.c | 5 +++++
migration/multifd.h | 2 ++
util/dsa.c | 51 ++++++++++++++++++++++++++++++++++++++-------
4 files changed, 99 insertions(+), 10 deletions(-)
diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
index e002652879..0c36e93016 100644
--- a/include/qemu/dsa.h
+++ b/include/qemu/dsa.h
@@ -2,6 +2,7 @@
#define QEMU_DSA_H
#include "qemu/error-report.h"
+#include "exec/cpu-common.h"
#include "qemu/thread.h"
#include "qemu/queue.h"
@@ -42,6 +43,21 @@ typedef struct dsa_batch_task {
QSIMPLEQ_ENTRY(dsa_batch_task) entry;
} dsa_batch_task;
+#endif
+
+struct batch_task {
+#ifdef CONFIG_DSA_OPT
+ /* Address of each pages in pages */
+ ram_addr_t *addr;
+ /* Zero page checking results */
+ bool *results;
+ /* Batch task DSA specific implementation */
+ struct dsa_batch_task *dsa_batch;
+#endif
+};
+
+#ifdef CONFIG_DSA_OPT
+
/**
* @brief Initializes DSA devices.
*
@@ -74,7 +90,7 @@ void dsa_cleanup(void);
bool dsa_is_running(void);
/**
- * @brief Initializes a buffer zero batch task.
+ * @brief Initializes a buffer zero DSA batch task.
*
* @param task A pointer to the batch task to initialize.
* @param results A pointer to an array of zero page checking results.
@@ -102,9 +118,26 @@ void buffer_zero_batch_task_destroy(struct dsa_batch_task *task);
* @return Zero if successful, otherwise non-zero.
*/
int
-buffer_is_zero_dsa_batch_async(struct dsa_batch_task *batch_task,
+buffer_is_zero_dsa_batch_async(struct batch_task *batch_task,
const void **buf, size_t count, size_t len);
+/**
+ * @brief Initializes a general buffer zero batch task.
+ *
+ * @param batch_size The number of zero page checking tasks in the batch.
+ * @return A pointer to the general batch task initialized.
+ */
+struct batch_task *
+batch_task_init(int batch_size);
+
+/**
+ * @brief Destroys a general buffer zero batch task.
+ *
+ * @param task A pointer to the general batch task to destroy.
+ */
+void
+batch_task_destroy(struct batch_task *task);
+
#else
static inline bool dsa_is_running(void)
@@ -128,6 +161,20 @@ static inline void dsa_stop(void) {}
static inline void dsa_cleanup(void) {}
+static inline int
+buffer_is_zero_dsa_batch_async(struct batch_task *batch_task,
+ const void **buf, size_t count, size_t len)
+{
+ exit(1);
+}
+
+static inline struct batch_task *batch_task_init(int batch_size)
+{
+ return NULL;
+}
+
+static inline void batch_task_destroy(struct batch_task *task) {}
+
#endif
#endif
diff --git a/migration/multifd.c b/migration/multifd.c
index f317bff077..cfd3a92f6c 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -13,6 +13,8 @@
#include "qemu/osdep.h"
#include "qemu/cutils.h"
#include "qemu/rcu.h"
+#include "qemu/dsa.h"
+#include "qemu/memalign.h"
#include "exec/target_page.h"
#include "sysemu/sysemu.h"
#include "exec/ramblock.h"
@@ -780,6 +782,8 @@ static bool multifd_send_cleanup_channel(MultiFDSendParams *p, Error **errp)
p->name = NULL;
multifd_pages_clear(p->pages);
p->pages = NULL;
+ batch_task_destroy(p->batch_task);
+ p->batch_task = NULL;
p->packet_len = 0;
g_free(p->packet);
p->packet = NULL;
@@ -1172,6 +1176,7 @@ bool multifd_send_setup(void)
qemu_sem_init(&p->sem_sync, 0);
p->id = i;
p->pages = multifd_pages_init(page_count);
+ p->batch_task = batch_task_init(page_count);
if (use_packets) {
p->packet_len = sizeof(MultiFDPacket_t)
diff --git a/migration/multifd.h b/migration/multifd.h
index c9d9b09239..16e27db5e9 100644
--- a/migration/multifd.h
+++ b/migration/multifd.h
@@ -135,6 +135,8 @@ typedef struct {
* pending_job != 0 -> multifd_channel can use it.
*/
MultiFDPages_t *pages;
+ /* Zero page checking batch task */
+ struct batch_task *batch_task;
/* thread local variables. No locking required */
diff --git a/util/dsa.c b/util/dsa.c
index 5a2bf33651..4f695e58af 100644
--- a/util/dsa.c
+++ b/util/dsa.c
@@ -802,7 +802,7 @@ buffer_zero_task_init_int(struct dsa_hw_desc *descriptor,
}
/**
- * @brief Initializes a buffer zero batch task.
+ * @brief Initializes a buffer zero DSA batch task.
*
* @param task A pointer to the batch task to initialize.
* @param results A pointer to an array of zero page checking results.
@@ -1107,29 +1107,64 @@ void dsa_cleanup(void)
* @return Zero if successful, otherwise non-zero.
*/
int
-buffer_is_zero_dsa_batch_async(struct dsa_batch_task *batch_task,
+buffer_is_zero_dsa_batch_async(struct batch_task *batch_task,
const void **buf, size_t count, size_t len)
{
- if (count <= 0 || count > batch_task->batch_size) {
+ struct dsa_batch_task *dsa_batch = batch_task->dsa_batch;
+
+ if (count <= 0 || count > dsa_batch->batch_size) {
return -1;
}
- assert(batch_task != NULL);
+ assert(dsa_batch != NULL);
assert(len != 0);
assert(buf != NULL);
if (count == 1) {
/* DSA doesn't take batch operation with only 1 task. */
- buffer_zero_dsa_async(batch_task, buf[0], len);
+ buffer_zero_dsa_async(dsa_batch, buf[0], len);
} else {
- buffer_zero_dsa_batch_async(batch_task, buf, count, len);
+ buffer_zero_dsa_batch_async(dsa_batch, buf, count, len);
}
- buffer_zero_dsa_wait(batch_task);
- buffer_zero_cpu_fallback(batch_task);
+ buffer_zero_dsa_wait(dsa_batch);
+ buffer_zero_cpu_fallback(dsa_batch);
return 0;
}
+/**
+ * @brief Initializes a general buffer zero batch task.
+ *
+ * @param batch_size The number of zero page checking tasks in the batch.
+ * @return A pointer to the general batch task initialized.
+ */
+struct batch_task *
+batch_task_init(int batch_size)
+{
+ struct batch_task *task = g_malloc0(sizeof(struct batch_task));
+ task->addr = g_new0(ram_addr_t, batch_size);
+ task->results = g_new0(bool, batch_size);
+ task->dsa_batch = qemu_memalign(64, sizeof(struct dsa_batch_task));
+ buffer_zero_batch_task_init(task->dsa_batch, task->results, batch_size);
+
+ return task;
+}
+
+/**
+ * @brief Destroys a general buffer zero batch task.
+ *
+ * @param task A pointer to the general batch task to destroy.
+ */
+void
+batch_task_destroy(struct batch_task *task)
+{
+ g_free(task->addr);
+ g_free(task->results);
+ buffer_zero_batch_task_destroy(task->dsa_batch);
+ qemu_vfree(task->dsa_batch);
+ g_free(task);
+}
+
#endif
--
2.30.2
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [PATCH v4 10/14] migration/multifd: Enable DSA offloading in multifd sender path.
2024-04-25 2:21 [PATCH v4 00/14] Use Intel DSA accelerator to offload zero page checking in multifd live migration Hao Xiang
` (8 preceding siblings ...)
2024-04-25 2:21 ` [PATCH v4 09/14] migration/multifd: Prepare to introduce DSA acceleration on the multifd path Hao Xiang
@ 2024-04-25 2:21 ` Hao Xiang
2024-04-25 14:29 ` Daniel P. Berrangé
` (2 more replies)
2024-04-25 2:21 ` [PATCH v4 11/14] migration/multifd: Add migration option set packet size Hao Xiang
` (4 subsequent siblings)
14 siblings, 3 replies; 32+ messages in thread
From: Hao Xiang @ 2024-04-25 2:21 UTC (permalink / raw)
To: marcandre.lureau, peterx, farosas, armbru, lvivier, qemu-devel; +Cc: Hao Xiang
Multifd sender path gets an array of pages queued by the migration
thread. It performs zero page checking on every page in the array.
The pages are classfied as either a zero page or a normal page. This
change uses Intel DSA to offload the zero page checking from CPU to
the DSA accelerator. The sender thread submits a batch of pages to DSA
hardware and waits for the DSA completion thread to signal for work
completion.
Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
---
migration/multifd-zero-page.c | 99 +++++++++++++++++++++++++++++++++--
migration/multifd.c | 27 +++++++++-
migration/multifd.h | 1 +
3 files changed, 120 insertions(+), 7 deletions(-)
diff --git a/migration/multifd-zero-page.c b/migration/multifd-zero-page.c
index e1b8370f88..4f426289e4 100644
--- a/migration/multifd-zero-page.c
+++ b/migration/multifd-zero-page.c
@@ -37,25 +37,83 @@ static void swap_page_offset(ram_addr_t *pages_offset, int a, int b)
}
/**
- * multifd_send_zero_page_detect: Perform zero page detection on all pages.
+ * zero_page_detect_cpu: Perform zero page detection using CPU.
*
* Sorts normal pages before zero pages in p->pages->offset and updates
* p->pages->normal_num.
*
* @param p A pointer to the send params.
*/
-void multifd_send_zero_page_detect(MultiFDSendParams *p)
+static void zero_page_detect_cpu(MultiFDSendParams *p)
{
MultiFDPages_t *pages = p->pages;
RAMBlock *rb = pages->block;
int i = 0;
int j = pages->num - 1;
- if (!multifd_zero_page_enabled()) {
- pages->normal_num = pages->num;
+ /*
+ * Sort the page offset array by moving all normal pages to
+ * the left and all zero pages to the right of the array.
+ */
+ while (i <= j) {
+ uint64_t offset = pages->offset[i];
+
+ if (!buffer_is_zero(rb->host + offset, p->page_size)) {
+ i++;
+ continue;
+ }
+
+ swap_page_offset(pages->offset, i, j);
+ ram_release_page(rb->idstr, offset);
+ j--;
+ }
+
+ pages->normal_num = i;
+}
+
+
+#ifdef CONFIG_DSA_OPT
+
+static void swap_result(bool *results, int a, int b)
+{
+ bool temp;
+
+ if (a == b) {
return;
}
+ temp = results[a];
+ results[a] = results[b];
+ results[b] = temp;
+}
+
+/**
+ * zero_page_detect_dsa: Perform zero page detection using
+ * Intel Data Streaming Accelerator (DSA).
+ *
+ * Sorts normal pages before zero pages in p->pages->offset and updates
+ * p->pages->normal_num.
+ *
+ * @param p A pointer to the send params.
+ */
+static void zero_page_detect_dsa(MultiFDSendParams *p)
+{
+ MultiFDPages_t *pages = p->pages;
+ RAMBlock *rb = pages->block;
+ bool *results = p->batch_task->results;
+
+ for (int i = 0; i < p->pages->num; i++) {
+ p->batch_task->addr[i] = (ram_addr_t)(rb->host + p->pages->offset[i]);
+ }
+
+ buffer_is_zero_dsa_batch_async(p->batch_task,
+ (const void **)p->batch_task->addr,
+ p->pages->num,
+ p->page_size);
+
+ int i = 0;
+ int j = pages->num - 1;
+
/*
* Sort the page offset array by moving all normal pages to
* the left and all zero pages to the right of the array.
@@ -63,11 +121,12 @@ void multifd_send_zero_page_detect(MultiFDSendParams *p)
while (i <= j) {
uint64_t offset = pages->offset[i];
- if (!buffer_is_zero(rb->host + offset, p->page_size)) {
+ if (!results[i]) {
i++;
continue;
}
+ swap_result(results, i, j);
swap_page_offset(pages->offset, i, j);
ram_release_page(rb->idstr, offset);
j--;
@@ -76,6 +135,15 @@ void multifd_send_zero_page_detect(MultiFDSendParams *p)
pages->normal_num = i;
}
+#else
+
+static void zero_page_detect_dsa(MultiFDSendParams *p)
+{
+ exit(1);
+}
+
+#endif
+
void multifd_recv_zero_page_process(MultiFDRecvParams *p)
{
for (int i = 0; i < p->zero_num; i++) {
@@ -87,3 +155,24 @@ void multifd_recv_zero_page_process(MultiFDRecvParams *p)
}
}
}
+
+/**
+ * multifd_send_zero_page_detect: Perform zero page detection on all pages.
+ *
+ * @param p A pointer to the send params.
+ */
+void multifd_send_zero_page_detect(MultiFDSendParams *p)
+{
+ MultiFDPages_t *pages = p->pages;
+
+ if (!multifd_zero_page_enabled()) {
+ pages->normal_num = pages->num;
+ return;
+ }
+
+ if (dsa_is_running()) {
+ zero_page_detect_dsa(p);
+ } else {
+ zero_page_detect_cpu(p);
+ }
+}
diff --git a/migration/multifd.c b/migration/multifd.c
index cfd3a92f6c..7316643d0a 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -818,6 +818,8 @@ void multifd_send_shutdown(void)
multifd_send_terminate_threads();
+ dsa_cleanup();
+
for (i = 0; i < migrate_multifd_channels(); i++) {
MultiFDSendParams *p = &multifd_send_state->params[i];
Error *local_err = NULL;
@@ -1155,11 +1157,20 @@ bool multifd_send_setup(void)
uint32_t page_count = MULTIFD_PACKET_SIZE / qemu_target_page_size();
bool use_packets = multifd_use_packets();
uint8_t i;
+ const char *dsa_parameter = migrate_multifd_dsa_accel();
if (!migrate_multifd()) {
return true;
}
+ if (dsa_init(dsa_parameter)) {
+ error_setg(&local_err, "multifd: Sender failed to initialize DSA.");
+ error_report_err(local_err);
+ return false;
+ }
+
+ dsa_start();
+
thread_count = migrate_multifd_channels();
multifd_send_state = g_malloc0(sizeof(*multifd_send_state));
multifd_send_state->params = g_new0(MultiFDSendParams, thread_count);
@@ -1393,6 +1404,7 @@ void multifd_recv_cleanup(void)
qemu_thread_join(&p->thread);
}
}
+ dsa_cleanup();
for (i = 0; i < migrate_multifd_channels(); i++) {
multifd_recv_cleanup_channel(&multifd_recv_state->params[i]);
}
@@ -1568,6 +1580,9 @@ int multifd_recv_setup(Error **errp)
uint32_t page_count = MULTIFD_PACKET_SIZE / qemu_target_page_size();
bool use_packets = multifd_use_packets();
uint8_t i;
+ const char *dsa_parameter = migrate_multifd_dsa_accel();
+ int ret;
+ Error *local_err = NULL;
/*
* Return successfully if multiFD recv state is already initialised
@@ -1577,6 +1592,15 @@ int multifd_recv_setup(Error **errp)
return 0;
}
+ ret = dsa_init(dsa_parameter);
+ if (ret != 0) {
+ error_setg(&local_err, "multifd: Receiver failed to initialize DSA.");
+ error_propagate(errp, local_err);
+ return ret;
+ }
+
+ dsa_start();
+
thread_count = migrate_multifd_channels();
multifd_recv_state = g_malloc0(sizeof(*multifd_recv_state));
multifd_recv_state->params = g_new0(MultiFDRecvParams, thread_count);
@@ -1616,13 +1640,12 @@ int multifd_recv_setup(Error **errp)
for (i = 0; i < thread_count; i++) {
MultiFDRecvParams *p = &multifd_recv_state->params[i];
- int ret;
-
ret = multifd_recv_state->ops->recv_setup(p, errp);
if (ret) {
return ret;
}
}
+
return 0;
}
diff --git a/migration/multifd.h b/migration/multifd.h
index 16e27db5e9..b3717fae24 100644
--- a/migration/multifd.h
+++ b/migration/multifd.h
@@ -14,6 +14,7 @@
#define QEMU_MIGRATION_MULTIFD_H
#include "ram.h"
+#include "qemu/dsa.h"
typedef struct MultiFDRecvData MultiFDRecvData;
--
2.30.2
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [PATCH v4 11/14] migration/multifd: Add migration option set packet size.
2024-04-25 2:21 [PATCH v4 00/14] Use Intel DSA accelerator to offload zero page checking in multifd live migration Hao Xiang
` (9 preceding siblings ...)
2024-04-25 2:21 ` [PATCH v4 10/14] migration/multifd: Enable DSA offloading in multifd sender path Hao Xiang
@ 2024-04-25 2:21 ` Hao Xiang
2024-05-01 19:36 ` Peter Xu
2024-04-25 2:21 ` [PATCH v4 12/14] migration/multifd: Enable set packet size migration option Hao Xiang
` (3 subsequent siblings)
14 siblings, 1 reply; 32+ messages in thread
From: Hao Xiang @ 2024-04-25 2:21 UTC (permalink / raw)
To: marcandre.lureau, peterx, farosas, armbru, lvivier, qemu-devel; +Cc: Hao Xiang
The current multifd packet size is 128 * 4kb. This change adds
an option to set the packet size. Both sender and receiver needs
to set the same packet size for things to work.
Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
---
migration/options.c | 36 ++++++++++++++++++++++++++++++++++++
migration/options.h | 1 +
qapi/migration.json | 21 ++++++++++++++++++---
3 files changed, 55 insertions(+), 3 deletions(-)
diff --git a/migration/options.c b/migration/options.c
index dc8642df81..a9deb079eb 100644
--- a/migration/options.c
+++ b/migration/options.c
@@ -79,6 +79,12 @@
#define DEFAULT_MIGRATE_ANNOUNCE_ROUNDS 5
#define DEFAULT_MIGRATE_ANNOUNCE_STEP 100
+/*
+ * Parameter for multifd packet size.
+ */
+#define DEFAULT_MIGRATE_MULTIFD_PACKET_SIZE (128 * 4 * 1024)
+#define MAX_MIGRATE_MULTIFD_PACKET_SIZE (1023 * 4 * 1024)
+
#define DEFINE_PROP_MIG_CAP(name, x) \
DEFINE_PROP_BOOL(name, MigrationState, capabilities[x], false)
@@ -184,6 +190,9 @@ Property migration_properties[] = {
ZERO_PAGE_DETECTION_MULTIFD),
DEFINE_PROP_STRING("multifd-dsa-accel", MigrationState,
parameters.multifd_dsa_accel),
+ DEFINE_PROP_SIZE("multifd-packet-size", MigrationState,
+ parameters.multifd_packet_size,
+ DEFAULT_MIGRATE_MULTIFD_PACKET_SIZE),
/* Migration capabilities */
DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE),
@@ -879,6 +888,13 @@ int migrate_multifd_channels(void)
return s->parameters.multifd_channels;
}
+uint64_t migrate_multifd_packet_size(void)
+{
+ MigrationState *s = migrate_get_current();
+
+ return s->parameters.multifd_packet_size;
+}
+
MultiFDCompression migrate_multifd_compression(void)
{
MigrationState *s = migrate_get_current();
@@ -1031,6 +1047,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
params->x_checkpoint_delay = s->parameters.x_checkpoint_delay;
params->has_block_incremental = true;
params->block_incremental = s->parameters.block_incremental;
+ params->has_multifd_packet_size = true;
+ params->multifd_packet_size = s->parameters.multifd_packet_size;
params->has_multifd_channels = true;
params->multifd_channels = s->parameters.multifd_channels;
params->has_multifd_compression = true;
@@ -1094,6 +1112,7 @@ void migrate_params_init(MigrationParameters *params)
params->has_downtime_limit = true;
params->has_x_checkpoint_delay = true;
params->has_block_incremental = true;
+ params->has_multifd_packet_size = true;
params->has_multifd_channels = true;
params->has_multifd_compression = true;
params->has_multifd_zlib_level = true;
@@ -1195,6 +1214,17 @@ bool migrate_params_check(MigrationParameters *params, Error **errp)
/* x_checkpoint_delay is now always positive */
+ if (params->has_multifd_packet_size &&
+ ((params->multifd_packet_size < DEFAULT_MIGRATE_MULTIFD_PACKET_SIZE) ||
+ (params->multifd_packet_size > MAX_MIGRATE_MULTIFD_PACKET_SIZE) ||
+ (params->multifd_packet_size % qemu_target_page_size() != 0))) {
+ error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
+ "multifd_packet_size",
+ "a value between 524288 and 4190208, "
+ "must be a multiple of guest VM's page size.");
+ return false;
+ }
+
if (params->has_multifd_channels && (params->multifd_channels < 1)) {
error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
"multifd_channels",
@@ -1374,6 +1404,9 @@ static void migrate_params_test_apply(MigrateSetParameters *params,
if (params->has_block_incremental) {
dest->block_incremental = params->block_incremental;
}
+ if (params->has_multifd_packet_size) {
+ dest->multifd_packet_size = params->multifd_packet_size;
+ }
if (params->has_multifd_channels) {
dest->multifd_channels = params->multifd_channels;
}
@@ -1524,6 +1557,9 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
" use blockdev-mirror with NBD instead");
s->parameters.block_incremental = params->block_incremental;
}
+ if (params->has_multifd_packet_size) {
+ s->parameters.multifd_packet_size = params->multifd_packet_size;
+ }
if (params->has_multifd_channels) {
s->parameters.multifd_channels = params->multifd_channels;
}
diff --git a/migration/options.h b/migration/options.h
index 1cb3393be9..23995e6608 100644
--- a/migration/options.h
+++ b/migration/options.h
@@ -92,6 +92,7 @@ const char *migrate_tls_hostname(void);
uint64_t migrate_xbzrle_cache_size(void);
ZeroPageDetection migrate_zero_page_detection(void);
const char *migrate_multifd_dsa_accel(void);
+uint64_t migrate_multifd_packet_size(void);
/* parameters setters */
diff --git a/qapi/migration.json b/qapi/migration.json
index 934fa8839e..39d609c394 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -920,6 +920,10 @@
# characters. Setting this string to an empty string means disabling
# DSA accelerator offloading. Defaults to an empty string. (since 9.2)
#
+# @multifd-packet-size: Packet size in bytes used to migrate data.
+# The value needs to be a multiple of guest VM's page size.
+# The default value is 524288 and max value is 4190208. (Since 9.2)
+#
# Features:
#
# @deprecated: Member @block-incremental is deprecated. Use
@@ -954,7 +958,8 @@
{ 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] },
'vcpu-dirty-limit',
'mode',
- 'zero-page-detection'] }
+ 'zero-page-detection',
+ 'multifd-packet-size'] }
##
# @MigrateSetParameters:
@@ -1134,6 +1139,10 @@
# characters. Setting this string to an empty string means disabling
# DSA accelerator offloading. Defaults to an empty string. (since 9.2)
#
+# @multifd-packet-size: Packet size in bytes used to migrate data.
+# The value needs to be a multiple of guest VM's page size.
+# The default value is 524288 and max value is 4190208. (Since 9.2)
+#
# Features:
#
# @deprecated: Member @block-incremental is deprecated. Use
@@ -1189,7 +1198,8 @@
'*vcpu-dirty-limit': 'uint64',
'*mode': 'MigMode',
'*zero-page-detection': 'ZeroPageDetection',
- '*multifd-dsa-accel': 'StrOrNull'} }
+ '*multifd-dsa-accel': 'StrOrNull',
+ '*multifd-packet-size' : 'uint64'} }
##
# @migrate-set-parameters:
@@ -1373,6 +1383,10 @@
# characters. Setting this string to an empty string means disabling
# DSA accelerator offloading. Defaults to an empty string. (since 9.2)
#
+# @multifd-packet-size: Packet size in bytes used to migrate data.
+# The value needs to be a multiple of guest VM's page size.
+# The default value is 524288 and max value is 4190208. (Since 9.2)
+#
# Features:
#
# @deprecated: Member @block-incremental is deprecated. Use
@@ -1425,7 +1439,8 @@
'*vcpu-dirty-limit': 'uint64',
'*mode': 'MigMode',
'*zero-page-detection': 'ZeroPageDetection',
- '*multifd-dsa-accel': 'str'} }
+ '*multifd-dsa-accel': 'str',
+ '*multifd-packet-size': 'uint64'} }
##
# @query-migrate-parameters:
--
2.30.2
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [PATCH v4 12/14] migration/multifd: Enable set packet size migration option.
2024-04-25 2:21 [PATCH v4 00/14] Use Intel DSA accelerator to offload zero page checking in multifd live migration Hao Xiang
` (10 preceding siblings ...)
2024-04-25 2:21 ` [PATCH v4 11/14] migration/multifd: Add migration option set packet size Hao Xiang
@ 2024-04-25 2:21 ` Hao Xiang
2024-04-25 2:21 ` [PATCH v4 13/14] util/dsa: Add unit test coverage for Intel DSA task submission and completion Hao Xiang
` (2 subsequent siblings)
14 siblings, 0 replies; 32+ messages in thread
From: Hao Xiang @ 2024-04-25 2:21 UTC (permalink / raw)
To: marcandre.lureau, peterx, farosas, armbru, lvivier, qemu-devel; +Cc: Hao Xiang
During live migration, if the latency between sender and receiver
is high and bandwidth is also high (a long and fat pipe), using a bigger
packet size can help reduce migration total time. In addition, Intel
DSA offloading performs better with a large batch task. Providing an
option to set the packet size is useful for performance tuning.
Set the option:
migrate_set_parameter multifd-packet-size 4190208
Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
---
migration/migration-hmp-cmds.c | 7 +++++++
migration/multifd-zlib.c | 6 ++++--
migration/multifd-zstd.c | 6 ++++--
migration/multifd.c | 6 ++++--
migration/multifd.h | 3 ---
5 files changed, 19 insertions(+), 9 deletions(-)
diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c
index 7e9bb278c9..053ad0283a 100644
--- a/migration/migration-hmp-cmds.c
+++ b/migration/migration-hmp-cmds.c
@@ -338,6 +338,9 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
monitor_printf(mon, "%s: %s\n",
MigrationParameter_str(MIGRATION_PARAMETER_BLOCK_INCREMENTAL),
params->block_incremental ? "on" : "off");
+ monitor_printf(mon, "%s: %" PRIu64 "\n",
+ MigrationParameter_str(MIGRATION_PARAMETER_MULTIFD_PACKET_SIZE),
+ params->multifd_packet_size);
monitor_printf(mon, "%s: %u\n",
MigrationParameter_str(MIGRATION_PARAMETER_MULTIFD_CHANNELS),
params->multifd_channels);
@@ -630,6 +633,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
p->multifd_dsa_accel->type = QTYPE_QSTRING;
visit_type_str(v, param, &p->multifd_dsa_accel->u.s, &err);
break;
+ case MIGRATION_PARAMETER_MULTIFD_PACKET_SIZE:
+ p->has_multifd_packet_size = true;
+ visit_type_size(v, param, &p->multifd_packet_size, &err);
+ break;
case MIGRATION_PARAMETER_MULTIFD_CHANNELS:
p->has_multifd_channels = true;
visit_type_uint8(v, param, &p->multifd_channels, &err);
diff --git a/migration/multifd-zlib.c b/migration/multifd-zlib.c
index 737a9645d2..2880d35841 100644
--- a/migration/multifd-zlib.c
+++ b/migration/multifd-zlib.c
@@ -49,6 +49,7 @@ static int zlib_send_setup(MultiFDSendParams *p, Error **errp)
struct zlib_data *z = g_new0(struct zlib_data, 1);
z_stream *zs = &z->zs;
const char *err_msg;
+ uint64_t multifd_packet_size = migrate_multifd_packet_size();
zs->zalloc = Z_NULL;
zs->zfree = Z_NULL;
@@ -58,7 +59,7 @@ static int zlib_send_setup(MultiFDSendParams *p, Error **errp)
goto err_free_z;
}
/* This is the maximum size of the compressed buffer */
- z->zbuff_len = compressBound(MULTIFD_PACKET_SIZE);
+ z->zbuff_len = compressBound(multifd_packet_size);
z->zbuff = g_try_malloc(z->zbuff_len);
if (!z->zbuff) {
err_msg = "out of memory for zbuff";
@@ -193,6 +194,7 @@ out:
*/
static int zlib_recv_setup(MultiFDRecvParams *p, Error **errp)
{
+ uint64_t multifd_packet_size = migrate_multifd_packet_size();
struct zlib_data *z = g_new0(struct zlib_data, 1);
z_stream *zs = &z->zs;
@@ -207,7 +209,7 @@ static int zlib_recv_setup(MultiFDRecvParams *p, Error **errp)
return -1;
}
/* To be safe, we reserve twice the size of the packet */
- z->zbuff_len = MULTIFD_PACKET_SIZE * 2;
+ z->zbuff_len = multifd_packet_size * 2;
z->zbuff = g_try_malloc(z->zbuff_len);
if (!z->zbuff) {
inflateEnd(zs);
diff --git a/migration/multifd-zstd.c b/migration/multifd-zstd.c
index 256858df0a..edc738afbb 100644
--- a/migration/multifd-zstd.c
+++ b/migration/multifd-zstd.c
@@ -49,6 +49,7 @@ struct zstd_data {
*/
static int zstd_send_setup(MultiFDSendParams *p, Error **errp)
{
+ uint64_t multifd_packet_size = migrate_multifd_packet_size();
struct zstd_data *z = g_new0(struct zstd_data, 1);
int res;
@@ -69,7 +70,7 @@ static int zstd_send_setup(MultiFDSendParams *p, Error **errp)
return -1;
}
/* This is the maximum size of the compressed buffer */
- z->zbuff_len = ZSTD_compressBound(MULTIFD_PACKET_SIZE);
+ z->zbuff_len = ZSTD_compressBound(multifd_packet_size);
z->zbuff = g_try_malloc(z->zbuff_len);
if (!z->zbuff) {
ZSTD_freeCStream(z->zcs);
@@ -182,6 +183,7 @@ out:
*/
static int zstd_recv_setup(MultiFDRecvParams *p, Error **errp)
{
+ uint64_t multifd_packet_size = migrate_multifd_packet_size();
struct zstd_data *z = g_new0(struct zstd_data, 1);
int ret;
@@ -203,7 +205,7 @@ static int zstd_recv_setup(MultiFDRecvParams *p, Error **errp)
}
/* To be safe, we reserve twice the size of the packet */
- z->zbuff_len = MULTIFD_PACKET_SIZE * 2;
+ z->zbuff_len = multifd_packet_size * 2;
z->zbuff = g_try_malloc(z->zbuff_len);
if (!z->zbuff) {
ZSTD_freeDStream(z->zds);
diff --git a/migration/multifd.c b/migration/multifd.c
index 7316643d0a..2796646087 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -1154,7 +1154,8 @@ bool multifd_send_setup(void)
MigrationState *s = migrate_get_current();
Error *local_err = NULL;
int thread_count, ret = 0;
- uint32_t page_count = MULTIFD_PACKET_SIZE / qemu_target_page_size();
+ uint32_t page_count =
+ migrate_multifd_packet_size() / qemu_target_page_size();
bool use_packets = multifd_use_packets();
uint8_t i;
const char *dsa_parameter = migrate_multifd_dsa_accel();
@@ -1577,7 +1578,8 @@ static void *multifd_recv_thread(void *opaque)
int multifd_recv_setup(Error **errp)
{
int thread_count;
- uint32_t page_count = MULTIFD_PACKET_SIZE / qemu_target_page_size();
+ uint32_t page_count =
+ migrate_multifd_packet_size() / qemu_target_page_size();
bool use_packets = multifd_use_packets();
uint8_t i;
const char *dsa_parameter = migrate_multifd_dsa_accel();
diff --git a/migration/multifd.h b/migration/multifd.h
index b3717fae24..97d4095b6a 100644
--- a/migration/multifd.h
+++ b/migration/multifd.h
@@ -42,9 +42,6 @@ MultiFDRecvData *multifd_get_recv_data(void);
#define MULTIFD_FLAG_ZLIB (1 << 1)
#define MULTIFD_FLAG_ZSTD (2 << 1)
-/* This value needs to be a multiple of qemu_target_page_size() */
-#define MULTIFD_PACKET_SIZE (512 * 1024)
-
typedef struct {
uint32_t magic;
uint32_t version;
--
2.30.2
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [PATCH v4 13/14] util/dsa: Add unit test coverage for Intel DSA task submission and completion.
2024-04-25 2:21 [PATCH v4 00/14] Use Intel DSA accelerator to offload zero page checking in multifd live migration Hao Xiang
` (11 preceding siblings ...)
2024-04-25 2:21 ` [PATCH v4 12/14] migration/multifd: Enable set packet size migration option Hao Xiang
@ 2024-04-25 2:21 ` Hao Xiang
2024-04-25 2:21 ` [PATCH v4 14/14] migration/multifd: Add integration tests for multifd with Intel DSA offloading Hao Xiang
2024-05-01 19:54 ` [PATCH v4 00/14] Use Intel DSA accelerator to offload zero page checking in multifd live migration Peter Xu
14 siblings, 0 replies; 32+ messages in thread
From: Hao Xiang @ 2024-04-25 2:21 UTC (permalink / raw)
To: marcandre.lureau, peterx, farosas, armbru, lvivier, qemu-devel
Cc: Hao Xiang, Bryan Zhang
* Test DSA start and stop path.
* Test DSA configure and cleanup path.
* Test DSA task submission and completion path.
Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
---
tests/unit/meson.build | 6 +
tests/unit/test-dsa.c | 499 +++++++++++++++++++++++++++++++++++++++++
2 files changed, 505 insertions(+)
create mode 100644 tests/unit/test-dsa.c
diff --git a/tests/unit/meson.build b/tests/unit/meson.build
index 26c109c968..1d4d48898b 100644
--- a/tests/unit/meson.build
+++ b/tests/unit/meson.build
@@ -49,6 +49,12 @@ tests = {
'test-interval-tree': [],
}
+if config_host_data.get('CONFIG_DSA_OPT')
+ tests += {
+ 'test-dsa': [],
+ }
+endif
+
if have_system or have_tools
tests += {
'test-qmp-event': [testqapi],
diff --git a/tests/unit/test-dsa.c b/tests/unit/test-dsa.c
new file mode 100644
index 0000000000..0f2092767d
--- /dev/null
+++ b/tests/unit/test-dsa.c
@@ -0,0 +1,499 @@
+/*
+ * Test DSA functions.
+ *
+ * Copyright (c) 2023 Hao Xiang <hao.xiang@bytedance.com>
+ * Copyright (c) 2023 Bryan Zhang <bryan.zhang@bytedance.com>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see <http://www.gnu.org/licenses/>.
+ */
+#include "qemu/osdep.h"
+#include "qemu/host-utils.h"
+
+#include "qemu/cutils.h"
+#include "qemu/memalign.h"
+#include "qemu/dsa.h"
+
+/*
+ * TODO Communicate that DSA must be configured to support this batch size.
+ * TODO Alternatively, poke the DSA device to figure out batch size.
+ */
+#define batch_size 128
+#define page_size 4096
+
+#define oversized_batch_size (batch_size + 1)
+#define num_devices 2
+#define max_buffer_size (64 * 1024)
+
+/* TODO Make these not-hardcoded. */
+static const char *path1 = "/dev/dsa/wq4.0";
+static const char *path2 = "/dev/dsa/wq4.0 /dev/dsa/wq4.1";
+
+static struct batch_task *task;
+
+/* A helper for running a single task and checking for correctness. */
+static void do_single_task(void)
+{
+ task = batch_task_init(batch_size);
+ char buf[page_size];
+ char *ptr = buf;
+
+ buffer_is_zero_dsa_batch_async(task,
+ (const void **)&ptr,
+ 1,
+ page_size);
+ g_assert(task->results[0] == buffer_is_zero(buf, page_size));
+
+ batch_task_destroy(task);
+}
+
+static void test_single_zero(void)
+{
+ g_assert(!dsa_init(path1));
+ dsa_start();
+
+ task = batch_task_init(batch_size);
+
+ char buf[page_size];
+ char *ptr = buf;
+
+ memset(buf, 0x0, page_size);
+ buffer_is_zero_dsa_batch_async(task,
+ (const void **)&ptr,
+ 1, page_size);
+ g_assert(task->results[0]);
+
+ batch_task_destroy(task);
+
+ dsa_cleanup();
+}
+
+static void test_single_zero_async(void)
+{
+ test_single_zero();
+}
+
+static void test_single_nonzero(void)
+{
+ g_assert(!dsa_init(path1));
+ dsa_start();
+
+ task = batch_task_init(batch_size);
+
+ char buf[page_size];
+ char *ptr = buf;
+
+ memset(buf, 0x1, page_size);
+ buffer_is_zero_dsa_batch_async(task,
+ (const void **)&ptr,
+ 1, page_size);
+ g_assert(!task->results[0]);
+
+ batch_task_destroy(task);
+
+ dsa_cleanup();
+}
+
+static void test_single_nonzero_async(void)
+{
+ test_single_nonzero();
+}
+
+/* count == 0 should return quickly without calling into DSA. */
+static void test_zero_count_async(void)
+{
+ char buf[page_size];
+ buffer_is_zero_dsa_batch_async(task,
+ (const void **)&buf,
+ 0,
+ page_size);
+}
+
+static void test_null_task_async(void)
+{
+ if (g_test_subprocess()) {
+ g_assert(!dsa_init(path1));
+
+ char buf[page_size * batch_size];
+ char *addrs[batch_size];
+ for (int i = 0; i < batch_size; i++) {
+ addrs[i] = buf + (page_size * i);
+ }
+
+ buffer_is_zero_dsa_batch_async(NULL, (const void **)addrs,
+ batch_size,
+ page_size);
+ } else {
+ g_test_trap_subprocess(NULL, 0, 0);
+ g_test_trap_assert_failed();
+ }
+}
+
+static void test_oversized_batch(void)
+{
+ g_assert(!dsa_init(path1));
+ dsa_start();
+
+ task = batch_task_init(batch_size);
+
+ char buf[page_size * oversized_batch_size];
+ char *addrs[batch_size];
+ for (int i = 0; i < oversized_batch_size; i++) {
+ addrs[i] = buf + (page_size * i);
+ }
+
+ int ret = buffer_is_zero_dsa_batch_async(task,
+ (const void **)addrs,
+ oversized_batch_size,
+ page_size);
+ g_assert(ret != 0);
+
+ batch_task_destroy(task);
+
+ dsa_cleanup();
+}
+
+static void test_oversized_batch_async(void)
+{
+ test_oversized_batch();
+}
+
+static void test_zero_len_async(void)
+{
+ if (g_test_subprocess()) {
+ g_assert(!dsa_init(path1));
+
+ task = batch_task_init(batch_size);
+
+ char buf[page_size];
+
+ buffer_is_zero_dsa_batch_async(task,
+ (const void **)&buf,
+ 1,
+ 0);
+
+ batch_task_destroy(task);
+ } else {
+ g_test_trap_subprocess(NULL, 0, 0);
+ g_test_trap_assert_failed();
+ }
+}
+
+static void test_null_buf_async(void)
+{
+ if (g_test_subprocess()) {
+ g_assert(!dsa_init(path1));
+
+ task = batch_task_init(batch_size);
+
+ buffer_is_zero_dsa_batch_async(task, NULL, 1, page_size);
+
+ batch_task_destroy(task);
+ } else {
+ g_test_trap_subprocess(NULL, 0, 0);
+ g_test_trap_assert_failed();
+ }
+}
+
+static void test_batch(void)
+{
+ g_assert(!dsa_init(path1));
+ dsa_start();
+
+ task = batch_task_init(batch_size);
+
+ char buf[page_size * batch_size];
+ char *addrs[batch_size];
+ for (int i = 0; i < batch_size; i++) {
+ addrs[i] = buf + (page_size * i);
+ }
+
+ /*
+ * Using whatever is on the stack is somewhat random.
+ * Manually set some pages to zero and some to nonzero.
+ */
+ memset(buf + 0, 0, page_size * 10);
+ memset(buf + (10 * page_size), 0xff, page_size * 10);
+
+ buffer_is_zero_dsa_batch_async(task,
+ (const void **)addrs,
+ batch_size,
+ page_size);
+
+ bool is_zero;
+ for (int i = 0; i < batch_size; i++) {
+ is_zero = buffer_is_zero((const void *)&buf[page_size * i], page_size);
+ g_assert(task->results[i] == is_zero);
+ }
+
+ batch_task_destroy(task);
+
+ dsa_cleanup();
+}
+
+static void test_batch_async(void)
+{
+ test_batch();
+}
+
+static void test_page_fault(void)
+{
+ g_assert(!dsa_init(path1));
+ dsa_start();
+
+ char *buf[2];
+ int prot = PROT_READ | PROT_WRITE;
+ int flags = MAP_SHARED | MAP_ANON;
+ buf[0] = (char *)mmap(NULL, page_size * batch_size, prot, flags, -1, 0);
+ assert(buf[0] != MAP_FAILED);
+ buf[1] = (char *)malloc(page_size * batch_size);
+ assert(buf[1] != NULL);
+
+ for (int j = 0; j < 2; j++) {
+ task = batch_task_init(batch_size);
+
+ char *addrs[batch_size];
+ for (int i = 0; i < batch_size; i++) {
+ addrs[i] = buf[j] + (page_size * i);
+ }
+
+ buffer_is_zero_dsa_batch_async(task,
+ (const void **)addrs,
+ batch_size,
+ page_size);
+
+ bool is_zero;
+ for (int i = 0; i < batch_size; i++) {
+ is_zero = buffer_is_zero((const void *)&buf[j][page_size * i],
+ page_size);
+ g_assert(task->results[i] == is_zero);
+ }
+ batch_task_destroy(task);
+ }
+
+ assert(!munmap(buf[0], page_size * batch_size));
+ free(buf[1]);
+ dsa_cleanup();
+}
+
+static void test_various_buffer_sizes(void)
+{
+ g_assert(!dsa_init(path1));
+ dsa_start();
+
+ char *buf = malloc(max_buffer_size * batch_size);
+ char *addrs[batch_size];
+
+ for (int len = 16; len <= max_buffer_size; len *= 2) {
+ task = batch_task_init(batch_size);
+
+ for (int i = 0; i < batch_size; i++) {
+ addrs[i] = buf + (len * i);
+ }
+
+ buffer_is_zero_dsa_batch_async(task,
+ (const void **)addrs,
+ batch_size,
+ len);
+
+ bool is_zero;
+ for (int j = 0; j < batch_size; j++) {
+ is_zero = buffer_is_zero((const void *)&buf[len * j], len);
+ g_assert(task->results[j] == is_zero);
+ }
+
+ batch_task_destroy(task);
+ }
+
+ free(buf);
+
+ dsa_cleanup();
+}
+
+static void test_various_buffer_sizes_async(void)
+{
+ test_various_buffer_sizes();
+}
+
+static void test_double_start_stop(void)
+{
+ g_assert(!dsa_init(path1));
+ /* Double start */
+ dsa_start();
+ dsa_start();
+ g_assert(dsa_is_running());
+ do_single_task();
+
+ /* Double stop */
+ dsa_stop();
+ g_assert(!dsa_is_running());
+ dsa_stop();
+ g_assert(!dsa_is_running());
+
+ /* Restart */
+ dsa_start();
+ g_assert(dsa_is_running());
+ do_single_task();
+ dsa_cleanup();
+}
+
+static void test_is_running(void)
+{
+ g_assert(!dsa_init(path1));
+
+ g_assert(!dsa_is_running());
+ dsa_start();
+ g_assert(dsa_is_running());
+ dsa_stop();
+ g_assert(!dsa_is_running());
+ dsa_cleanup();
+}
+
+static void test_multiple_engines(void)
+{
+ g_assert(!dsa_init(path2));
+ dsa_start();
+
+ struct batch_task *tasks[num_devices];
+ char bufs[num_devices][page_size * batch_size];
+ char *addrs[num_devices][batch_size];
+
+ /*
+ * This is a somewhat implementation-specific way
+ * of testing that the tasks have unique engines
+ * assigned to them.
+ */
+ tasks[0] = batch_task_init(batch_size);
+ tasks[1] = batch_task_init(batch_size);
+ g_assert(tasks[0]->dsa_batch->device != tasks[1]->dsa_batch->device);
+
+ for (int i = 0; i < num_devices; i++) {
+ for (int j = 0; j < batch_size; j++) {
+ addrs[i][j] = bufs[i] + (page_size * j);
+ }
+
+ buffer_is_zero_dsa_batch_async(tasks[i],
+ (const void **)addrs[i],
+ batch_size, page_size);
+
+ bool is_zero;
+ for (int j = 0; j < batch_size; j++) {
+ is_zero = buffer_is_zero((const void *)&bufs[i][page_size * j],
+ page_size);
+ g_assert(tasks[i]->results[j] == is_zero);
+ }
+ }
+
+ batch_task_destroy(tasks[0]);
+ batch_task_destroy(tasks[1]);
+
+ dsa_cleanup();
+}
+
+static void test_configure_dsa_twice(void)
+{
+ g_assert(!dsa_init(path2));
+ g_assert(!dsa_init(path2));
+ dsa_start();
+ do_single_task();
+ dsa_cleanup();
+}
+
+static void test_configure_dsa_bad_path(void)
+{
+ const char *bad_path = "/not/a/real/path";
+ g_assert(dsa_init(bad_path));
+}
+
+static void test_cleanup_before_configure(void)
+{
+ dsa_cleanup();
+ g_assert(!dsa_init(path2));
+}
+
+static void test_configure_dsa_num_devices(void)
+{
+ g_assert(!dsa_init(path1));
+ dsa_start();
+
+ do_single_task();
+ dsa_stop();
+ dsa_cleanup();
+}
+
+static void test_cleanup_twice(void)
+{
+ g_assert(!dsa_init(path2));
+ dsa_cleanup();
+ dsa_cleanup();
+
+ g_assert(!dsa_init(path2));
+ dsa_start();
+ do_single_task();
+ dsa_cleanup();
+}
+
+static int check_test_setup(void)
+{
+ const char *path[2] = {path1, path2};
+ for (int i = 0; i < sizeof(path) / sizeof(char *); i++) {
+ if (dsa_init(path[i])) {
+ return -1;
+ }
+ dsa_cleanup();
+ }
+ return 0;
+}
+
+int main(int argc, char **argv)
+{
+ g_test_init(&argc, &argv, NULL);
+
+ if (check_test_setup() != 0) {
+ /*
+ * This test requires extra setup. The current
+ * setup is not correct. Just skip this test
+ * for now.
+ */
+ exit(0);
+ }
+
+ if (num_devices > 1) {
+ g_test_add_func("/dsa/multiple_engines", test_multiple_engines);
+ }
+
+ g_test_add_func("/dsa/async/batch", test_batch_async);
+ g_test_add_func("/dsa/async/various_buffer_sizes",
+ test_various_buffer_sizes_async);
+ g_test_add_func("/dsa/async/null_buf", test_null_buf_async);
+ g_test_add_func("/dsa/async/zero_len", test_zero_len_async);
+ g_test_add_func("/dsa/async/oversized_batch", test_oversized_batch_async);
+ g_test_add_func("/dsa/async/zero_count", test_zero_count_async);
+ g_test_add_func("/dsa/async/single_zero", test_single_zero_async);
+ g_test_add_func("/dsa/async/single_nonzero", test_single_nonzero_async);
+ g_test_add_func("/dsa/async/null_task", test_null_task_async);
+ g_test_add_func("/dsa/async/page_fault", test_page_fault);
+
+ g_test_add_func("/dsa/double_start_stop", test_double_start_stop);
+ g_test_add_func("/dsa/is_running", test_is_running);
+
+ g_test_add_func("/dsa/configure_dsa_twice", test_configure_dsa_twice);
+ g_test_add_func("/dsa/configure_dsa_bad_path", test_configure_dsa_bad_path);
+ g_test_add_func("/dsa/cleanup_before_configure",
+ test_cleanup_before_configure);
+ g_test_add_func("/dsa/configure_dsa_num_devices",
+ test_configure_dsa_num_devices);
+ g_test_add_func("/dsa/cleanup_twice", test_cleanup_twice);
+
+ return g_test_run();
+}
--
2.30.2
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [PATCH v4 14/14] migration/multifd: Add integration tests for multifd with Intel DSA offloading.
2024-04-25 2:21 [PATCH v4 00/14] Use Intel DSA accelerator to offload zero page checking in multifd live migration Hao Xiang
` (12 preceding siblings ...)
2024-04-25 2:21 ` [PATCH v4 13/14] util/dsa: Add unit test coverage for Intel DSA task submission and completion Hao Xiang
@ 2024-04-25 2:21 ` Hao Xiang
2024-05-01 19:54 ` [PATCH v4 00/14] Use Intel DSA accelerator to offload zero page checking in multifd live migration Peter Xu
14 siblings, 0 replies; 32+ messages in thread
From: Hao Xiang @ 2024-04-25 2:21 UTC (permalink / raw)
To: marcandre.lureau, peterx, farosas, armbru, lvivier, qemu-devel
Cc: Hao Xiang, Bryan Zhang
* Add test case to start and complete multifd live migration with DSA
offloading enabled.
* Add test case to start and cancel multifd live migration with DSA
offloading enabled.
Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
---
tests/qtest/migration-test.c | 77 +++++++++++++++++++++++++++++++++++-
1 file changed, 76 insertions(+), 1 deletion(-)
diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c
index 5d6d8cd634..354c5f26f8 100644
--- a/tests/qtest/migration-test.c
+++ b/tests/qtest/migration-test.c
@@ -616,6 +616,12 @@ typedef struct {
bool suspend_me;
} MigrateStart;
+/*
+ * It requires separate steps to configure and enable DSA device.
+ * This test assumes that the configuration is done already.
+ */
+static const char *dsa_dev_path = "/dev/dsa/wq4.0";
+
/*
* A hook that runs after the src and dst QEMUs have been
* created, but before the migration is started. This can
@@ -3025,7 +3031,7 @@ static void test_multifd_tcp_tls_x509_reject_anon_client(void)
*
* And see that it works
*/
-static void test_multifd_tcp_cancel(void)
+static void test_multifd_tcp_cancel_common(bool use_dsa)
{
MigrateStart args = {
.hide_stderr = true,
@@ -3045,6 +3051,10 @@ static void test_multifd_tcp_cancel(void)
migrate_set_capability(from, "multifd", true);
migrate_set_capability(to, "multifd", true);
+ if (use_dsa) {
+ migrate_set_parameter_str(from, "multifd-dsa-accel", dsa_dev_path);
+ }
+
/* Start incoming migration from the 1st socket */
migrate_incoming_qmp(to, "tcp:127.0.0.1:0", "{}");
@@ -3094,6 +3104,48 @@ static void test_multifd_tcp_cancel(void)
test_migrate_end(from, to2, true);
}
+/*
+ * This test does:
+ * source target
+ * migrate_incoming
+ * migrate
+ * migrate_cancel
+ * launch another target
+ * migrate
+ *
+ * And see that it works
+ */
+static void test_multifd_tcp_cancel(void)
+{
+ test_multifd_tcp_cancel_common(false);
+}
+
+#ifdef CONFIG_DSA_OPT
+
+static void *test_migrate_precopy_tcp_multifd_start_dsa(QTestState *from,
+ QTestState *to)
+{
+ migrate_set_parameter_str(from, "multifd-dsa-accel", dsa_dev_path);
+ return test_migrate_precopy_tcp_multifd_start_common(from, to, "none");
+}
+
+static void test_multifd_tcp_zero_page_dsa(void)
+{
+ MigrateCommon args = {
+ .listen_uri = "defer",
+ .start_hook = test_migrate_precopy_tcp_multifd_start_dsa,
+ };
+
+ test_precopy_common(&args);
+}
+
+static void test_multifd_tcp_cancel_dsa(void)
+{
+ test_multifd_tcp_cancel_common(true);
+}
+
+#endif
+
static void calc_dirty_rate(QTestState *who, uint64_t calc_time)
{
qtest_qmp_assert_success(who,
@@ -3518,6 +3570,19 @@ static bool kvm_dirty_ring_supported(void)
#endif
}
+#ifdef CONFIG_DSA_OPT
+static int test_dsa_setup(void)
+{
+ int fd;
+ fd = open(dsa_dev_path, O_RDWR);
+ if (fd < 0) {
+ return -1;
+ }
+ close(fd);
+ return 0;
+}
+#endif
+
int main(int argc, char **argv)
{
bool has_kvm, has_tcg;
@@ -3752,6 +3817,16 @@ int main(int argc, char **argv)
test_multifd_tcp_zero_page_legacy);
migration_test_add("/migration/multifd/tcp/plain/zero-page/none",
test_multifd_tcp_no_zero_page);
+
+#ifdef CONFIG_DSA_OPT
+ if (g_str_equal(arch, "x86_64") && test_dsa_setup() == 0) {
+ migration_test_add("/migration/multifd/tcp/plain/zero-page/dsa",
+ test_multifd_tcp_zero_page_dsa);
+ migration_test_add("/migration/multifd/tcp/plain/cancel/dsa",
+ test_multifd_tcp_cancel_dsa);
+ }
+#endif
+
migration_test_add("/migration/multifd/tcp/plain/cancel",
test_multifd_tcp_cancel);
migration_test_add("/migration/multifd/tcp/plain/zlib",
--
2.30.2
^ permalink raw reply related [flat|nested] 32+ messages in thread
* Re: [PATCH v4 08/14] migration/multifd: Add new migration option for multifd DSA offloading.
2024-04-25 2:21 ` [PATCH v4 08/14] migration/multifd: Add new migration option for multifd DSA offloading Hao Xiang
@ 2024-04-25 14:17 ` Daniel P. Berrangé
2024-04-26 9:16 ` Markus Armbruster
0 siblings, 1 reply; 32+ messages in thread
From: Daniel P. Berrangé @ 2024-04-25 14:17 UTC (permalink / raw)
To: Hao Xiang; +Cc: marcandre.lureau, peterx, farosas, armbru, lvivier, qemu-devel
On Thu, Apr 25, 2024 at 02:21:11AM +0000, Hao Xiang wrote:
> Intel DSA offloading is an optional feature that turns on if
> proper hardware and software stack is available. To turn on
> DSA offloading in multifd live migration:
>
> multifd-dsa-accel="[dsa_dev_path1] [dsa_dev_path2] ... [dsa_dev_pathX]"
>
> This feature is turned off by default.
>
> Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
> ---
> migration/migration-hmp-cmds.c | 8 ++++++++
> migration/options.c | 30 ++++++++++++++++++++++++++++++
> migration/options.h | 1 +
> qapi/migration.json | 26 +++++++++++++++++++++++---
> 4 files changed, 62 insertions(+), 3 deletions(-)
> diff --git a/qapi/migration.json b/qapi/migration.json
> index 8c65b90328..934fa8839e 100644
> --- a/qapi/migration.json
> +++ b/qapi/migration.json
> @@ -914,6 +914,12 @@
> # See description in @ZeroPageDetection. Default is 'multifd'.
> # (since 9.0)
> #
> +# @multifd-dsa-accel: If enabled, use DSA accelerator offloading for
> +# certain memory operations. Enable DSA accelerator offloading by
> +# setting this string to a list of DSA device path separated by space
> +# characters. Setting this string to an empty string means disabling
> +# DSA accelerator offloading. Defaults to an empty string. (since 9.2)
Passing a list of paths as a single space separate string is a
design anti-pattern. This needs to use a list type at the QAPI
level.
Also I don't think we need add 'multifd' on the name - all
new features are for multifd.
Overall it should be called 'dsa-accel-path' I thjink
> @@ -1122,6 +1128,12 @@
> # See description in @ZeroPageDetection. Default is 'multifd'.
> # (since 9.0)
> #
> +# @multifd-dsa-accel: If enabled, use DSA accelerator offloading for
> +# certain memory operations. Enable DSA accelerator offloading by
> +# setting this string to a list of DSA device path separated by space
> +# characters. Setting this string to an empty string means disabling
> +# DSA accelerator offloading. Defaults to an empty string. (since 9.2)
> +#
> # Features:
> #
> # @deprecated: Member @block-incremental is deprecated. Use
> @@ -1176,7 +1188,8 @@
> 'features': [ 'unstable' ] },
> '*vcpu-dirty-limit': 'uint64',
> '*mode': 'MigMode',
> - '*zero-page-detection': 'ZeroPageDetection'} }
> + '*zero-page-detection': 'ZeroPageDetection',
> + '*multifd-dsa-accel': 'StrOrNull'} }
This needs to be
['str'] not 'StrOrNull'
>
> ##
> # @migrate-set-parameters:
> @@ -1354,6 +1367,12 @@
> # See description in @ZeroPageDetection. Default is 'multifd'.
> # (since 9.0)
> #
> +# @multifd-dsa-accel: If enabled, use DSA accelerator offloading for
> +# certain memory operations. Enable DSA accelerator offloading by
> +# setting this string to a list of DSA device path separated by space
> +# characters. Setting this string to an empty string means disabling
> +# DSA accelerator offloading. Defaults to an empty string. (since 9.2)
> +#
> # Features:
> #
> # @deprecated: Member @block-incremental is deprecated. Use
> @@ -1405,7 +1424,8 @@
> 'features': [ 'unstable' ] },
> '*vcpu-dirty-limit': 'uint64',
> '*mode': 'MigMode',
> - '*zero-page-detection': 'ZeroPageDetection'} }
> + '*zero-page-detection': 'ZeroPageDetection',
> + '*multifd-dsa-accel': 'str'} }
Liekewise needs to be
['str']
Having mgmt apps pass in the path every time though, feels like
overkill. Surely there's a standard path that QEMU should use
by default, and should only require flag to turn on its usage.
IOW, why not extend the ZeroPageDetection enum, to have a further
entry for 'dsa' to request ue of dsa accel. Passing paths could
be optional.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v4 03/14] util/dsa: Implement DSA device start and stop logic.
2024-04-25 2:21 ` [PATCH v4 03/14] util/dsa: Implement DSA device start and stop logic Hao Xiang
@ 2024-04-25 14:21 ` Daniel P. Berrangé
2024-04-25 14:25 ` Daniel P. Berrangé
` (2 subsequent siblings)
3 siblings, 0 replies; 32+ messages in thread
From: Daniel P. Berrangé @ 2024-04-25 14:21 UTC (permalink / raw)
To: Hao Xiang
Cc: marcandre.lureau, peterx, farosas, armbru, lvivier, qemu-devel,
Bryan Zhang
On Thu, Apr 25, 2024 at 02:21:06AM +0000, Hao Xiang wrote:
> * DSA device open and close.
> * DSA group contains multiple DSA devices.
> * DSA group configure/start/stop/clean.
>
> Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
> Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
> ---
> include/qemu/dsa.h | 72 +++++++++++
> util/dsa.c | 316 +++++++++++++++++++++++++++++++++++++++++++++
> util/meson.build | 1 +
> 3 files changed, 389 insertions(+)
> create mode 100644 include/qemu/dsa.h
> create mode 100644 util/dsa.c
>
> diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
> new file mode 100644
> index 0000000000..f15c05ee85
> --- /dev/null
> +++ b/include/qemu/dsa.h
> @@ -0,0 +1,72 @@
Missing license header.
> +#ifndef QEMU_DSA_H
> +#define QEMU_DSA_H
> +
> +#include "qemu/error-report.h"
> +#include "qemu/thread.h"
> +#include "qemu/queue.h"
> +
> +#ifdef CONFIG_DSA_OPT
> +
> +#pragma GCC push_options
> +#pragma GCC target("enqcmd")
> +
> +#include <linux/idxd.h>
> +#include "x86intrin.h"
> +
> +/**
> + * @brief Initializes DSA devices.
> + *
> + * @param dsa_parameter A list of DSA device path from migration parameter.
> + *
> + * @return int Zero if successful, otherwise non zero.
> + */
> +int dsa_init(const char *dsa_parameter);
> +
> +/**
> + * @brief Start logic to enable using DSA.
> + */
> +void dsa_start(void);
> +
> +/**
> + * @brief Stop the device group and the completion thread.
> + */
> +void dsa_stop(void);
> +
> +/**
> + * @brief Clean up system resources created for DSA offloading.
> + */
> +void dsa_cleanup(void);
> +
> +/**
> + * @brief Check if DSA is running.
> + *
> + * @return True if DSA is running, otherwise false.
> + */
> +bool dsa_is_running(void);
> +
> +#else
> +
> +static inline bool dsa_is_running(void)
> +{
> + return false;
> +}
> +
> +static inline int dsa_init(const char *dsa_parameter)
> +{
> + if (dsa_parameter != NULL && strlen(dsa_parameter) != 0) {
> + error_report("DSA not supported.");
Using error_report in this code is undesirable, as it means the
migration code has no way to feed error information back to the
mgmt app.
*all* the APIs in dsa.h/dsa.c that can have errors, need to have
an "Error **errp" parameter, so useful info can be propagated
back the caller in the migration code.
> + return -1;
> + }
> +
> + return 0;
> +}
> +
> +static inline void dsa_start(void) {}
> +
> +static inline void dsa_stop(void) {}
> +
> +static inline void dsa_cleanup(void) {}
> +
> +#endif
> +
> +#endif
> diff --git a/util/dsa.c b/util/dsa.c
> new file mode 100644
> index 0000000000..05bbf8e31a
> --- /dev/null
> +++ b/util/dsa.c
> @@ -0,0 +1,316 @@
> +/*
> + * Use Intel Data Streaming Accelerator to offload certain background
> + * operations.
> + *
> + * Copyright (c) 2023 Hao Xiang <hao.xiang@bytedance.com>
> + * Bryan Zhang <bryan.zhang@bytedance.com>
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a copy
> + * of this software and associated documentation files (the "Software"), to deal
> + * in the Software without restriction, including without limitation the rights
> + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
> + * copies of the Software, and to permit persons to whom the Software is
> + * furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
> + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
> + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
> + * THE SOFTWARE.
> + */
This is an MIT license header, but QEMU's standard license is GPL-2.0-or-later.
Please keep new contributions under QEMU's normal license unless there's a
reason why you must differ.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v4 03/14] util/dsa: Implement DSA device start and stop logic.
2024-04-25 2:21 ` [PATCH v4 03/14] util/dsa: Implement DSA device start and stop logic Hao Xiang
2024-04-25 14:21 ` Daniel P. Berrangé
@ 2024-04-25 14:25 ` Daniel P. Berrangé
2024-04-25 14:32 ` Daniel P. Berrangé
2024-04-25 21:22 ` Fabiano Rosas
3 siblings, 0 replies; 32+ messages in thread
From: Daniel P. Berrangé @ 2024-04-25 14:25 UTC (permalink / raw)
To: Hao Xiang
Cc: marcandre.lureau, peterx, farosas, armbru, lvivier, qemu-devel,
Bryan Zhang
On Thu, Apr 25, 2024 at 02:21:06AM +0000, Hao Xiang wrote:
> * DSA device open and close.
> * DSA group contains multiple DSA devices.
> * DSA group configure/start/stop/clean.
>
> Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
> Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
> ---
> include/qemu/dsa.h | 72 +++++++++++
> util/dsa.c | 316 +++++++++++++++++++++++++++++++++++++++++++++
> util/meson.build | 1 +
> 3 files changed, 389 insertions(+)
> create mode 100644 include/qemu/dsa.h
> create mode 100644 util/dsa.c
>
> diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
> new file mode 100644
> index 0000000000..f15c05ee85
> --- /dev/null
> +++ b/include/qemu/dsa.h
> @@ -0,0 +1,72 @@
> +#ifndef QEMU_DSA_H
> +#define QEMU_DSA_H
> +
> +#include "qemu/error-report.h"
> +#include "qemu/thread.h"
> +#include "qemu/queue.h"
> +
> +#ifdef CONFIG_DSA_OPT
> +
> +#pragma GCC push_options
> +#pragma GCC target("enqcmd")
You've tested in meson.build that the compiler supports
'enqcmd' which is good. I'm not seeing anything that
tests whether the host running this code supports 'enqcmd'.
There needs to be a CPUID check for this at runtime.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v4 10/14] migration/multifd: Enable DSA offloading in multifd sender path.
2024-04-25 2:21 ` [PATCH v4 10/14] migration/multifd: Enable DSA offloading in multifd sender path Hao Xiang
@ 2024-04-25 14:29 ` Daniel P. Berrangé
2024-04-25 15:39 ` Fabiano Rosas
2024-05-01 19:25 ` Peter Xu
2 siblings, 0 replies; 32+ messages in thread
From: Daniel P. Berrangé @ 2024-04-25 14:29 UTC (permalink / raw)
To: Hao Xiang; +Cc: marcandre.lureau, peterx, farosas, armbru, lvivier, qemu-devel
On Thu, Apr 25, 2024 at 02:21:13AM +0000, Hao Xiang wrote:
> Multifd sender path gets an array of pages queued by the migration
> thread. It performs zero page checking on every page in the array.
> The pages are classfied as either a zero page or a normal page. This
> change uses Intel DSA to offload the zero page checking from CPU to
> the DSA accelerator. The sender thread submits a batch of pages to DSA
> hardware and waits for the DSA completion thread to signal for work
> completion.
>
> Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
> ---
> migration/multifd-zero-page.c | 99 +++++++++++++++++++++++++++++++++--
> migration/multifd.c | 27 +++++++++-
> migration/multifd.h | 1 +
> 3 files changed, 120 insertions(+), 7 deletions(-)
>
> diff --git a/migration/multifd-zero-page.c b/migration/multifd-zero-page.c
> index e1b8370f88..4f426289e4 100644
> --- a/migration/multifd-zero-page.c
> +++ b/migration/multifd-zero-page.c
> diff --git a/migration/multifd.c b/migration/multifd.c
> index cfd3a92f6c..7316643d0a 100644
> --- a/migration/multifd.c
> +++ b/migration/multifd.c
> @@ -818,6 +818,8 @@ void multifd_send_shutdown(void)
>
> multifd_send_terminate_threads();
>
> + dsa_cleanup();
> +
> for (i = 0; i < migrate_multifd_channels(); i++) {
> MultiFDSendParams *p = &multifd_send_state->params[i];
> Error *local_err = NULL;
> @@ -1155,11 +1157,20 @@ bool multifd_send_setup(void)
> uint32_t page_count = MULTIFD_PACKET_SIZE / qemu_target_page_size();
> bool use_packets = multifd_use_packets();
> uint8_t i;
> + const char *dsa_parameter = migrate_multifd_dsa_accel();
>
> if (!migrate_multifd()) {
> return true;
> }
>
> + if (dsa_init(dsa_parameter)) {
> + error_setg(&local_err, "multifd: Sender failed to initialize DSA.");
> + error_report_err(local_err);
> + return false;
> + }
This is an example of why all the dsa functions need to report
a via an "Error **err" parameter. The error reported
here is useless as it lacks any meaningful information of what
went wrong.
The multifd_send_setup method itself needs a "Error **errp"
param so it can pass it back up to be reoprted too.
> +
> + dsa_start();
> +
> thread_count = migrate_multifd_channels();
> multifd_send_state = g_malloc0(sizeof(*multifd_send_state));
> multifd_send_state->params = g_new0(MultiFDSendParams, thread_count);
> @@ -1393,6 +1404,7 @@ void multifd_recv_cleanup(void)
> qemu_thread_join(&p->thread);
> }
> }
> + dsa_cleanup();
> for (i = 0; i < migrate_multifd_channels(); i++) {
> multifd_recv_cleanup_channel(&multifd_recv_state->params[i]);
> }
> @@ -1568,6 +1580,9 @@ int multifd_recv_setup(Error **errp)
> uint32_t page_count = MULTIFD_PACKET_SIZE / qemu_target_page_size();
> bool use_packets = multifd_use_packets();
> uint8_t i;
> + const char *dsa_parameter = migrate_multifd_dsa_accel();
> + int ret;
> + Error *local_err = NULL;
>
> /*
> * Return successfully if multiFD recv state is already initialised
> @@ -1577,6 +1592,15 @@ int multifd_recv_setup(Error **errp)
> return 0;
> }
>
> + ret = dsa_init(dsa_parameter);
> + if (ret != 0) {
> + error_setg(&local_err, "multifd: Receiver failed to initialize DSA.");
> + error_propagate(errp, local_err);
> + return ret;
> + }
> +
> + dsa_start();
> +
> thread_count = migrate_multifd_channels();
> multifd_recv_state = g_malloc0(sizeof(*multifd_recv_state));
> multifd_recv_state->params = g_new0(MultiFDRecvParams, thread_count);
> @@ -1616,13 +1640,12 @@ int multifd_recv_setup(Error **errp)
>
> for (i = 0; i < thread_count; i++) {
> MultiFDRecvParams *p = &multifd_recv_state->params[i];
> - int ret;
> -
> ret = multifd_recv_state->ops->recv_setup(p, errp);
> if (ret) {
> return ret;
> }
> }
> +
> return 0;
> }
>
> diff --git a/migration/multifd.h b/migration/multifd.h
> index 16e27db5e9..b3717fae24 100644
> --- a/migration/multifd.h
> +++ b/migration/multifd.h
> @@ -14,6 +14,7 @@
> #define QEMU_MIGRATION_MULTIFD_H
>
> #include "ram.h"
> +#include "qemu/dsa.h"
>
> typedef struct MultiFDRecvData MultiFDRecvData;
>
> --
> 2.30.2
>
>
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v4 03/14] util/dsa: Implement DSA device start and stop logic.
2024-04-25 2:21 ` [PATCH v4 03/14] util/dsa: Implement DSA device start and stop logic Hao Xiang
2024-04-25 14:21 ` Daniel P. Berrangé
2024-04-25 14:25 ` Daniel P. Berrangé
@ 2024-04-25 14:32 ` Daniel P. Berrangé
2024-04-25 21:22 ` Fabiano Rosas
3 siblings, 0 replies; 32+ messages in thread
From: Daniel P. Berrangé @ 2024-04-25 14:32 UTC (permalink / raw)
To: Hao Xiang
Cc: marcandre.lureau, peterx, farosas, armbru, lvivier, qemu-devel,
Bryan Zhang
On Thu, Apr 25, 2024 at 02:21:06AM +0000, Hao Xiang wrote:
> * DSA device open and close.
> * DSA group contains multiple DSA devices.
> * DSA group configure/start/stop/clean.
>
> Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
> Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
> ---
> include/qemu/dsa.h | 72 +++++++++++
> util/dsa.c | 316 +++++++++++++++++++++++++++++++++++++++++++++
> util/meson.build | 1 +
> 3 files changed, 389 insertions(+)
> create mode 100644 include/qemu/dsa.h
> create mode 100644 util/dsa.c
>
> diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
> new file mode 100644
> index 0000000000..f15c05ee85
> --- /dev/null
> +++ b/include/qemu/dsa.h
> @@ -0,0 +1,72 @@
> +#ifndef QEMU_DSA_H
> +#define QEMU_DSA_H
> +
> +#include "qemu/error-report.h"
> +#include "qemu/thread.h"
> +#include "qemu/queue.h"
> +
> +#ifdef CONFIG_DSA_OPT
> +
> +#pragma GCC push_options
> +#pragma GCC target("enqcmd")
> +
> +#include <linux/idxd.h>
> +#include "x86intrin.h"
> +
> +/**
> + * @brief Initializes DSA devices.
> + *
> + * @param dsa_parameter A list of DSA device path from migration parameter.
> + *
> + * @return int Zero if successful, otherwise non zero.
> + */
> +int dsa_init(const char *dsa_parameter);
BTW, all these methods should also use 'qemu_dsa_' as a name
prefix, not merely 'dsa_'. The latter is too generic, and
likely to clash with naming of APIs implemnenting 'dsa'
crypto, as well as withthe kernel's dsa devoce header.
Likewise best practice for the structs in the dsa.c file
to also use 'QemuDsa' as a nameprefix, not merely 'Dsa'.
> +
> +/**
> + * @brief Start logic to enable using DSA.
> + */
> +void dsa_start(void);
> +
> +/**
> + * @brief Stop the device group and the completion thread.
> + */
> +void dsa_stop(void);
> +
> +/**
> + * @brief Clean up system resources created for DSA offloading.
> + */
> +void dsa_cleanup(void);
> +
> +/**
> + * @brief Check if DSA is running.
> + *
> + * @return True if DSA is running, otherwise false.
> + */
> +bool dsa_is_running(void);
> +
> +#else
> +
> +static inline bool dsa_is_running(void)
> +{
> + return false;
> +}
> +
> +static inline int dsa_init(const char *dsa_parameter)
> +{
> + if (dsa_parameter != NULL && strlen(dsa_parameter) != 0) {
> + error_report("DSA not supported.");
> + return -1;
> + }
> +
> + return 0;
> +}
> +
> +static inline void dsa_start(void) {}
> +
> +static inline void dsa_stop(void) {}
> +
> +static inline void dsa_cleanup(void) {}
> +
> +#endif
> +
> +#endif
> diff --git a/util/dsa.c b/util/dsa.c
> new file mode 100644
> index 0000000000..05bbf8e31a
> --- /dev/null
> +++ b/util/dsa.c
> @@ -0,0 +1,316 @@
> +/*
> + * Use Intel Data Streaming Accelerator to offload certain background
> + * operations.
> + *
> + * Copyright (c) 2023 Hao Xiang <hao.xiang@bytedance.com>
> + * Bryan Zhang <bryan.zhang@bytedance.com>
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a copy
> + * of this software and associated documentation files (the "Software"), to deal
> + * in the Software without restriction, including without limitation the rights
> + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
> + * copies of the Software, and to permit persons to whom the Software is
> + * furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
> + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
> + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
> + * THE SOFTWARE.
> + */
> +
> +#include "qemu/osdep.h"
> +#include "qemu/queue.h"
> +#include "qemu/memalign.h"
> +#include "qemu/lockable.h"
> +#include "qemu/cutils.h"
> +#include "qemu/dsa.h"
> +#include "qemu/bswap.h"
> +#include "qemu/error-report.h"
> +#include "qemu/rcu.h"
> +
> +#ifdef CONFIG_DSA_OPT
> +
> +#pragma GCC push_options
> +#pragma GCC target("enqcmd")
> +
> +#include <linux/idxd.h>
> +#include "x86intrin.h"
> +
> +#define DSA_WQ_SIZE 4096
> +#define MAX_DSA_DEVICES 16
> +
> +typedef QSIMPLEQ_HEAD(dsa_task_queue, dsa_batch_task) dsa_task_queue;
> +
> +struct dsa_device {
> + void *work_queue;
> +};
> +
> +struct dsa_device_group {
IMHO preferable to use initial-upper case for struct
names, to distinguish from method names. ie
QemuDsaDeviceGroup
also I'd suggest they should all be typedef'd too,
so its not repeating 'struct <blah>' everywhere.
> + struct dsa_device *dsa_devices;
> + int num_dsa_devices;
> + /* The index of the next DSA device to be used. */
> + uint32_t device_allocator_index;
> + bool running;
> + QemuMutex task_queue_lock;
> + QemuCond task_queue_cond;
> + dsa_task_queue task_queue;
> +};
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v4 10/14] migration/multifd: Enable DSA offloading in multifd sender path.
2024-04-25 2:21 ` [PATCH v4 10/14] migration/multifd: Enable DSA offloading in multifd sender path Hao Xiang
2024-04-25 14:29 ` Daniel P. Berrangé
@ 2024-04-25 15:39 ` Fabiano Rosas
2024-05-01 19:25 ` Peter Xu
2 siblings, 0 replies; 32+ messages in thread
From: Fabiano Rosas @ 2024-04-25 15:39 UTC (permalink / raw)
To: Hao Xiang, marcandre.lureau, peterx, armbru, lvivier, qemu-devel
Cc: Hao Xiang
Hao Xiang <hao.xiang@linux.dev> writes:
> Multifd sender path gets an array of pages queued by the migration
> thread. It performs zero page checking on every page in the array.
> The pages are classfied as either a zero page or a normal page. This
> change uses Intel DSA to offload the zero page checking from CPU to
> the DSA accelerator. The sender thread submits a batch of pages to DSA
> hardware and waits for the DSA completion thread to signal for work
> completion.
>
> Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
> ---
> migration/multifd-zero-page.c | 99 +++++++++++++++++++++++++++++++++--
> migration/multifd.c | 27 +++++++++-
> migration/multifd.h | 1 +
> 3 files changed, 120 insertions(+), 7 deletions(-)
>
> diff --git a/migration/multifd-zero-page.c b/migration/multifd-zero-page.c
> index e1b8370f88..4f426289e4 100644
> --- a/migration/multifd-zero-page.c
> +++ b/migration/multifd-zero-page.c
> @@ -37,25 +37,83 @@ static void swap_page_offset(ram_addr_t *pages_offset, int a, int b)
> }
>
> /**
> - * multifd_send_zero_page_detect: Perform zero page detection on all pages.
> + * zero_page_detect_cpu: Perform zero page detection using CPU.
> *
> * Sorts normal pages before zero pages in p->pages->offset and updates
> * p->pages->normal_num.
> *
> * @param p A pointer to the send params.
> */
> -void multifd_send_zero_page_detect(MultiFDSendParams *p)
> +static void zero_page_detect_cpu(MultiFDSendParams *p)
> {
> MultiFDPages_t *pages = p->pages;
> RAMBlock *rb = pages->block;
> int i = 0;
> int j = pages->num - 1;
>
> - if (!multifd_zero_page_enabled()) {
> - pages->normal_num = pages->num;
> + /*
> + * Sort the page offset array by moving all normal pages to
> + * the left and all zero pages to the right of the array.
> + */
> + while (i <= j) {
> + uint64_t offset = pages->offset[i];
> +
> + if (!buffer_is_zero(rb->host + offset, p->page_size)) {
> + i++;
> + continue;
> + }
> +
> + swap_page_offset(pages->offset, i, j);
> + ram_release_page(rb->idstr, offset);
> + j--;
> + }
> +
> + pages->normal_num = i;
> +}
> +
> +
> +#ifdef CONFIG_DSA_OPT
> +
> +static void swap_result(bool *results, int a, int b)
> +{
> + bool temp;
> +
> + if (a == b) {
> return;
> }
>
> + temp = results[a];
> + results[a] = results[b];
> + results[b] = temp;
> +}
> +
> +/**
> + * zero_page_detect_dsa: Perform zero page detection using
> + * Intel Data Streaming Accelerator (DSA).
> + *
> + * Sorts normal pages before zero pages in p->pages->offset and updates
> + * p->pages->normal_num.
> + *
> + * @param p A pointer to the send params.
> + */
> +static void zero_page_detect_dsa(MultiFDSendParams *p)
> +{
> + MultiFDPages_t *pages = p->pages;
> + RAMBlock *rb = pages->block;
> + bool *results = p->batch_task->results;
At this point I don't see the need to carry p->batch_task in the multifd
code. results here could be taken from the
buffer_is_zero_dsa_batch_async call below.
> +
> + for (int i = 0; i < p->pages->num; i++) {
> + p->batch_task->addr[i] = (ram_addr_t)(rb->host + p->pages->offset[i]);
... and this is just what p->iov[0].iov_base contains.
> + }
> +
> + buffer_is_zero_dsa_batch_async(p->batch_task,
This is not async. You're using the result right after this call. Leave
the 'async' for the functions called from whithin this one, which do the
actual wait part.
> + (const void **)p->batch_task->addr,
> + p->pages->num,
> + p->page_size);
> +
> + int i = 0;
> + int j = pages->num - 1;
> +
> /*
> * Sort the page offset array by moving all normal pages to
> * the left and all zero pages to the right of the array.
> @@ -63,11 +121,12 @@ void multifd_send_zero_page_detect(MultiFDSendParams *p)
> while (i <= j) {
> uint64_t offset = pages->offset[i];
>
> - if (!buffer_is_zero(rb->host + offset, p->page_size)) {
> + if (!results[i]) {
> i++;
> continue;
> }
>
> + swap_result(results, i, j);
> swap_page_offset(pages->offset, i, j);
> ram_release_page(rb->idstr, offset);
> j--;
> @@ -76,6 +135,15 @@ void multifd_send_zero_page_detect(MultiFDSendParams *p)
> pages->normal_num = i;
> }
>
> +#else
> +
> +static void zero_page_detect_dsa(MultiFDSendParams *p)
> +{
> + exit(1);
> +}
> +
> +#endif
> +
> void multifd_recv_zero_page_process(MultiFDRecvParams *p)
> {
> for (int i = 0; i < p->zero_num; i++) {
> @@ -87,3 +155,24 @@ void multifd_recv_zero_page_process(MultiFDRecvParams *p)
> }
> }
> }
> +
> +/**
> + * multifd_send_zero_page_detect: Perform zero page detection on all pages.
> + *
> + * @param p A pointer to the send params.
> + */
> +void multifd_send_zero_page_detect(MultiFDSendParams *p)
> +{
> + MultiFDPages_t *pages = p->pages;
> +
> + if (!multifd_zero_page_enabled()) {
> + pages->normal_num = pages->num;
> + return;
> + }
> +
> + if (dsa_is_running()) {
> + zero_page_detect_dsa(p);
> + } else {
> + zero_page_detect_cpu(p);
> + }
> +}
> diff --git a/migration/multifd.c b/migration/multifd.c
> index cfd3a92f6c..7316643d0a 100644
> --- a/migration/multifd.c
> +++ b/migration/multifd.c
> @@ -818,6 +818,8 @@ void multifd_send_shutdown(void)
>
> multifd_send_terminate_threads();
>
> + dsa_cleanup();
> +
> for (i = 0; i < migrate_multifd_channels(); i++) {
> MultiFDSendParams *p = &multifd_send_state->params[i];
> Error *local_err = NULL;
> @@ -1155,11 +1157,20 @@ bool multifd_send_setup(void)
> uint32_t page_count = MULTIFD_PACKET_SIZE / qemu_target_page_size();
> bool use_packets = multifd_use_packets();
> uint8_t i;
> + const char *dsa_parameter = migrate_multifd_dsa_accel();
>
> if (!migrate_multifd()) {
> return true;
> }
>
> + if (dsa_init(dsa_parameter)) {
> + error_setg(&local_err, "multifd: Sender failed to initialize DSA.");
> + error_report_err(local_err);
> + return false;
> + }
> +
> + dsa_start();
> +
> thread_count = migrate_multifd_channels();
> multifd_send_state = g_malloc0(sizeof(*multifd_send_state));
> multifd_send_state->params = g_new0(MultiFDSendParams, thread_count);
> @@ -1393,6 +1404,7 @@ void multifd_recv_cleanup(void)
> qemu_thread_join(&p->thread);
> }
> }
> + dsa_cleanup();
> for (i = 0; i < migrate_multifd_channels(); i++) {
> multifd_recv_cleanup_channel(&multifd_recv_state->params[i]);
> }
> @@ -1568,6 +1580,9 @@ int multifd_recv_setup(Error **errp)
> uint32_t page_count = MULTIFD_PACKET_SIZE / qemu_target_page_size();
> bool use_packets = multifd_use_packets();
> uint8_t i;
> + const char *dsa_parameter = migrate_multifd_dsa_accel();
> + int ret;
> + Error *local_err = NULL;
>
> /*
> * Return successfully if multiFD recv state is already initialised
> @@ -1577,6 +1592,15 @@ int multifd_recv_setup(Error **errp)
> return 0;
> }
>
> + ret = dsa_init(dsa_parameter);
> + if (ret != 0) {
> + error_setg(&local_err, "multifd: Receiver failed to initialize DSA.");
> + error_propagate(errp, local_err);
> + return ret;
> + }
> +
> + dsa_start();
> +
> thread_count = migrate_multifd_channels();
> multifd_recv_state = g_malloc0(sizeof(*multifd_recv_state));
> multifd_recv_state->params = g_new0(MultiFDRecvParams, thread_count);
> @@ -1616,13 +1640,12 @@ int multifd_recv_setup(Error **errp)
>
> for (i = 0; i < thread_count; i++) {
> MultiFDRecvParams *p = &multifd_recv_state->params[i];
> - int ret;
> -
> ret = multifd_recv_state->ops->recv_setup(p, errp);
> if (ret) {
> return ret;
> }
> }
> +
> return 0;
> }
>
> diff --git a/migration/multifd.h b/migration/multifd.h
> index 16e27db5e9..b3717fae24 100644
> --- a/migration/multifd.h
> +++ b/migration/multifd.h
> @@ -14,6 +14,7 @@
> #define QEMU_MIGRATION_MULTIFD_H
>
> #include "ram.h"
> +#include "qemu/dsa.h"
>
> typedef struct MultiFDRecvData MultiFDRecvData;
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v4 01/14] meson: Introduce new instruction set enqcmd to the build system.
2024-04-25 2:21 ` [PATCH v4 01/14] meson: Introduce new instruction set enqcmd to the build system Hao Xiang
@ 2024-04-25 18:50 ` Fabiano Rosas
0 siblings, 0 replies; 32+ messages in thread
From: Fabiano Rosas @ 2024-04-25 18:50 UTC (permalink / raw)
To: Hao Xiang, marcandre.lureau, peterx, armbru, lvivier, qemu-devel
Cc: Hao Xiang
Hao Xiang <hao.xiang@linux.dev> writes:
> Enable instruction set enqcmd in build.
>
> Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
> ---
> meson.build | 14 ++++++++++++++
> meson_options.txt | 2 ++
> scripts/meson-buildoptions.sh | 3 +++
> 3 files changed, 19 insertions(+)
>
> diff --git a/meson.build b/meson.build
> index 95cee7046e..9e008ddc34 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -2824,6 +2824,20 @@ config_host_data.set('CONFIG_AVX512BW_OPT', get_option('avx512bw') \
> int main(int argc, char *argv[]) { return bar(argv[0]); }
> '''), error_message: 'AVX512BW not available').allowed())
>
> +config_host_data.set('CONFIG_DSA_OPT', get_option('enqcmd') \
> + .require(have_cpuid_h, error_message: 'cpuid.h not available, cannot enable ENQCMD') \
> + .require(cc.links('''
> + #include <stdint.h>
> + #include <cpuid.h>
> + #include <immintrin.h>
> + static int __attribute__((target("enqcmd"))) bar(void *a) {
> + uint64_t dst[8] = { 0 };
> + uint64_t src[8] = { 0 };
> + return _enqcmd(dst, src);
> + }
> + int main(int argc, char *argv[]) { return bar(argv[argc - 1]); }
> + '''), error_message: 'ENQCMD not available').allowed())
> +
> # For both AArch64 and AArch32, detect if builtins are available.
> config_host_data.set('CONFIG_ARM_AES_BUILTIN', cc.compiles('''
> #include <arm_neon.h>
> diff --git a/meson_options.txt b/meson_options.txt
> index b5c0bad9e7..63c1bf815b 100644
> --- a/meson_options.txt
> +++ b/meson_options.txt
> @@ -121,6 +121,8 @@ option('avx512f', type: 'feature', value: 'disabled',
> description: 'AVX512F optimizations')
> option('avx512bw', type: 'feature', value: 'auto',
> description: 'AVX512BW optimizations')
> +option('enqcmd', type: 'feature', value: 'disabled',
> + description: 'MENQCMD optimizations')
s/MENQCMD/ENQCMD/
> option('keyring', type: 'feature', value: 'auto',
> description: 'Linux keyring support')
> option('libkeyutils', type: 'feature', value: 'auto',
> diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh
> index 5ace33f167..2cdfc84455 100644
> --- a/scripts/meson-buildoptions.sh
> +++ b/scripts/meson-buildoptions.sh
> @@ -93,6 +93,7 @@ meson_options_help() {
> printf "%s\n" ' avx2 AVX2 optimizations'
> printf "%s\n" ' avx512bw AVX512BW optimizations'
> printf "%s\n" ' avx512f AVX512F optimizations'
> + printf "%s\n" ' enqcmd ENQCMD optimizations'
> printf "%s\n" ' blkio libblkio block device driver'
> printf "%s\n" ' bochs bochs image format support'
> printf "%s\n" ' bpf eBPF support'
> @@ -239,6 +240,8 @@ _meson_option_parse() {
> --disable-avx512bw) printf "%s" -Davx512bw=disabled ;;
> --enable-avx512f) printf "%s" -Davx512f=enabled ;;
> --disable-avx512f) printf "%s" -Davx512f=disabled ;;
> + --enable-enqcmd) printf "%s" -Denqcmd=enabled ;;
> + --disable-enqcmd) printf "%s" -Denqcmd=disabled ;;
> --enable-gcov) printf "%s" -Db_coverage=true ;;
> --disable-gcov) printf "%s" -Db_coverage=false ;;
> --enable-lto) printf "%s" -Db_lto=true ;;
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v4 02/14] util/dsa: Add dependency idxd.
2024-04-25 2:21 ` [PATCH v4 02/14] util/dsa: Add dependency idxd Hao Xiang
@ 2024-04-25 20:33 ` Fabiano Rosas
0 siblings, 0 replies; 32+ messages in thread
From: Fabiano Rosas @ 2024-04-25 20:33 UTC (permalink / raw)
To: Hao Xiang, marcandre.lureau, peterx, armbru, lvivier, qemu-devel
Cc: Hao Xiang
Hao Xiang <hao.xiang@linux.dev> writes:
> Idxd is the device driver for DSA (Intel Data Streaming
> Accelerator). The driver is fully functioning since Linux
> kernel 5.19. This change adds the driver's header file used
> for userspace development.
Have you looked at the update-linux-headers script?
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v4 04/14] util/dsa: Implement DSA task enqueue and dequeue.
2024-04-25 2:21 ` [PATCH v4 04/14] util/dsa: Implement DSA task enqueue and dequeue Hao Xiang
@ 2024-04-25 20:55 ` Fabiano Rosas
2024-04-25 21:48 ` Fabiano Rosas
1 sibling, 0 replies; 32+ messages in thread
From: Fabiano Rosas @ 2024-04-25 20:55 UTC (permalink / raw)
To: Hao Xiang, marcandre.lureau, peterx, armbru, lvivier, qemu-devel
Cc: Hao Xiang
Hao Xiang <hao.xiang@linux.dev> writes:
> * Use a safe thread queue for DSA task enqueue/dequeue.
> * Implement DSA task submission.
> * Implement DSA batch task submission.
>
> Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
> ---
> include/qemu/dsa.h | 28 +++++++
> util/dsa.c | 201 +++++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 229 insertions(+)
>
> diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
> index f15c05ee85..37cae8d9d2 100644
> --- a/include/qemu/dsa.h
> +++ b/include/qemu/dsa.h
> @@ -13,6 +13,34 @@
> #include <linux/idxd.h>
> #include "x86intrin.h"
>
> +typedef enum DsaTaskType {
> + DSA_TASK = 0,
> + DSA_BATCH_TASK
> +} DsaTaskType;
> +
> +typedef enum DsaTaskStatus {
> + DSA_TASK_READY = 0,
> + DSA_TASK_PROCESSING,
> + DSA_TASK_COMPLETION
> +} DsaTaskStatus;
> +
> +typedef void (*dsa_completion_fn)(void *);
> +
> +typedef struct dsa_batch_task {
> + struct dsa_hw_desc batch_descriptor;
> + struct dsa_hw_desc *descriptors;
> + struct dsa_completion_record batch_completion __attribute__((aligned(32)));
> + struct dsa_completion_record *completions;
> + struct dsa_device_group *group;
> + struct dsa_device *device;
> + dsa_completion_fn completion_callback;
> + QemuSemaphore sem_task_complete;
> + DsaTaskType task_type;
> + DsaTaskStatus status;
> + int batch_size;
> + QSIMPLEQ_ENTRY(dsa_batch_task) entry;
> +} dsa_batch_task;
> +
> /**
> * @brief Initializes DSA devices.
> *
> diff --git a/util/dsa.c b/util/dsa.c
> index 05bbf8e31a..75739a1af6 100644
> --- a/util/dsa.c
> +++ b/util/dsa.c
> @@ -244,6 +244,205 @@ dsa_device_group_get_next_device(struct dsa_device_group *group)
> return &group->dsa_devices[current];
> }
>
> +/**
> + * @brief Empties out the DSA task queue.
> + *
> + * @param group A pointer to the DSA device group.
> + */
> +static void
> +dsa_empty_task_queue(struct dsa_device_group *group)
> +{
> + qemu_mutex_lock(&group->task_queue_lock);
> + dsa_task_queue *task_queue = &group->task_queue;
> + while (!QSIMPLEQ_EMPTY(task_queue)) {
> + QSIMPLEQ_REMOVE_HEAD(task_queue, entry);
> + }
> + qemu_mutex_unlock(&group->task_queue_lock);
> +}
> +
> +/**
> + * @brief Adds a task to the DSA task queue.
> + *
> + * @param group A pointer to the DSA device group.
> + * @param context A pointer to the DSA task to enqueue.
This is wrong^
> + *
> + * @return int Zero if successful, otherwise a proper error code.
> + */
> +static int
> +dsa_task_enqueue(struct dsa_device_group *group,
> + struct dsa_batch_task *task)
> +{
> + dsa_task_queue *task_queue = &group->task_queue;
> + QemuMutex *task_queue_lock = &group->task_queue_lock;
> + QemuCond *task_queue_cond = &group->task_queue_cond;
It's more idiomatic to not hold any of these in a variable, just access
them directly.
> +
> + bool notify = false;
> +
> + qemu_mutex_lock(task_queue_lock);
> +
> + if (!group->running) {
> + error_report("DSA: Tried to queue task to stopped device queue.");
> + qemu_mutex_unlock(task_queue_lock);
> + return -1;
> + }
> +
> + /* The queue is empty. This enqueue operation is a 0->1 transition. */
> + if (QSIMPLEQ_EMPTY(task_queue)) {
> + notify = true;
> + }
> +
> + QSIMPLEQ_INSERT_TAIL(task_queue, task, entry);
> +
> + /* We need to notify the waiter for 0->1 transitions. */
> + if (notify) {
> + qemu_cond_signal(task_queue_cond);
> + }
> +
> + qemu_mutex_unlock(task_queue_lock);
> +
> + return 0;
> +}
> +
> +/**
> + * @brief Takes a DSA task out of the task queue.
> + *
> + * @param group A pointer to the DSA device group.
> + * @return dsa_batch_task* The DSA task being dequeued.
> + */
> +__attribute__((unused))
> +static struct dsa_batch_task *
> +dsa_task_dequeue(struct dsa_device_group *group)
> +{
> + struct dsa_batch_task *task = NULL;
> + dsa_task_queue *task_queue = &group->task_queue;
> + QemuMutex *task_queue_lock = &group->task_queue_lock;
> + QemuCond *task_queue_cond = &group->task_queue_cond;
Same here.
> +
> + qemu_mutex_lock(task_queue_lock);
> +
> + while (true) {
> + if (!group->running) {
> + goto exit;
> + }
> + task = QSIMPLEQ_FIRST(task_queue);
> + if (task != NULL) {
> + break;
> + }
> + qemu_cond_wait(task_queue_cond, task_queue_lock);
> + }
> +
> + QSIMPLEQ_REMOVE_HEAD(task_queue, entry);
> +
> +exit:
> + qemu_mutex_unlock(task_queue_lock);
> + return task;
> +}
> +
> +/**
> + * @brief Submits a DSA work item to the device work queue.
> + *
> + * @param wq A pointer to the DSA work queue's device memory.
> + * @param descriptor A pointer to the DSA work item descriptor.
> + *
> + * @return Zero if successful, non-zero otherwise.
> + */
> +static int
> +submit_wi_int(void *wq, struct dsa_hw_desc *descriptor)
> +{
> + uint64_t retry = 0;
> +
> + _mm_sfence();
> +
> + while (true) {
> + if (_enqcmd(wq, descriptor) == 0) {
> + break;
> + }
> + retry++;
> + if (retry > max_retry_count) {
> + error_report("Submit work retry %lu times.", retry);
> + return -1;
> + }
> + }
> +
> + return 0;
> +}
> +
> +/**
> + * @brief Synchronously submits a DSA work item to the
> + * device work queue.
> + *
> + * @param wq A pointer to the DSA worjk queue's device memory.
s/worjk/work/
> + * @param descriptor A pointer to the DSA work item descriptor.
> + *
> + * @return int Zero if successful, non-zero otherwise.
> + */
> +__attribute__((unused))
> +static int
> +submit_wi(void *wq, struct dsa_hw_desc *descriptor)
> +{
> + return submit_wi_int(wq, descriptor);
> +}
> +
> +/**
> + * @brief Asynchronously submits a DSA work item to the
> + * device work queue.
> + *
> + * @param task A pointer to the buffer zero task.
> + *
> + * @return int Zero if successful, non-zero otherwise.
> + */
> +__attribute__((unused))
> +static int
> +submit_wi_async(struct dsa_batch_task *task)
> +{
> + struct dsa_device_group *device_group = task->group;
> + struct dsa_device *device_instance = task->device;
> + int ret;
> +
> + assert(task->task_type == DSA_TASK);
> +
> + task->status = DSA_TASK_PROCESSING;
> +
> + ret = submit_wi_int(device_instance->work_queue,
> + &task->descriptors[0]);
> + if (ret != 0) {
> + return ret;
> + }
> +
> + return dsa_task_enqueue(device_group, task);
> +}
> +
> +/**
> + * @brief Asynchronously submits a DSA batch work item to the
> + * device work queue.
> + *
> + * @param dsa_batch_task A pointer to the batch buffer zero task.
s/buffer zero //
> + *
> + * @return int Zero if successful, non-zero otherwise.
> + */
> +__attribute__((unused))
> +static int
> +submit_batch_wi_async(struct dsa_batch_task *batch_task)
> +{
> + struct dsa_device_group *device_group = batch_task->group;
> + struct dsa_device *device_instance = batch_task->device;
> + int ret;
> +
> + assert(batch_task->task_type == DSA_BATCH_TASK);
> + assert(batch_task->batch_descriptor.desc_count <= batch_task->batch_size);
> + assert(batch_task->status == DSA_TASK_READY);
> +
> + batch_task->status = DSA_TASK_PROCESSING;
> +
> + ret = submit_wi_int(device_instance->work_queue,
> + &batch_task->batch_descriptor);
> + if (ret != 0) {
> + return ret;
> + }
> +
> + return dsa_task_enqueue(device_group, batch_task);
> +}
> +
> /**
> * @brief Check if DSA is running.
> *
> @@ -300,6 +499,8 @@ void dsa_stop(void)
> if (!group->running) {
> return;
> }
> +
> + dsa_empty_task_queue(group);
> }
>
> /**
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v4 03/14] util/dsa: Implement DSA device start and stop logic.
2024-04-25 2:21 ` [PATCH v4 03/14] util/dsa: Implement DSA device start and stop logic Hao Xiang
` (2 preceding siblings ...)
2024-04-25 14:32 ` Daniel P. Berrangé
@ 2024-04-25 21:22 ` Fabiano Rosas
3 siblings, 0 replies; 32+ messages in thread
From: Fabiano Rosas @ 2024-04-25 21:22 UTC (permalink / raw)
To: Hao Xiang, marcandre.lureau, peterx, armbru, lvivier, qemu-devel
Cc: Hao Xiang, Bryan Zhang
Hao Xiang <hao.xiang@linux.dev> writes:
> * DSA device open and close.
> * DSA group contains multiple DSA devices.
> * DSA group configure/start/stop/clean.
>
> Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
> Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
> ---
> include/qemu/dsa.h | 72 +++++++++++
> util/dsa.c | 316 +++++++++++++++++++++++++++++++++++++++++++++
> util/meson.build | 1 +
> 3 files changed, 389 insertions(+)
> create mode 100644 include/qemu/dsa.h
> create mode 100644 util/dsa.c
>
> diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
> new file mode 100644
> index 0000000000..f15c05ee85
> --- /dev/null
> +++ b/include/qemu/dsa.h
> @@ -0,0 +1,72 @@
> +#ifndef QEMU_DSA_H
> +#define QEMU_DSA_H
> +
> +#include "qemu/error-report.h"
> +#include "qemu/thread.h"
> +#include "qemu/queue.h"
> +
> +#ifdef CONFIG_DSA_OPT
> +
> +#pragma GCC push_options
> +#pragma GCC target("enqcmd")
> +
> +#include <linux/idxd.h>
> +#include "x86intrin.h"
> +
> +/**
> + * @brief Initializes DSA devices.
> + *
> + * @param dsa_parameter A list of DSA device path from migration parameter.
> + *
> + * @return int Zero if successful, otherwise non zero.
> + */
> +int dsa_init(const char *dsa_parameter);
> +
> +/**
> + * @brief Start logic to enable using DSA.
> + */
> +void dsa_start(void);
> +
> +/**
> + * @brief Stop the device group and the completion thread.
> + */
> +void dsa_stop(void);
> +
> +/**
> + * @brief Clean up system resources created for DSA offloading.
> + */
> +void dsa_cleanup(void);
> +
> +/**
> + * @brief Check if DSA is running.
> + *
> + * @return True if DSA is running, otherwise false.
> + */
> +bool dsa_is_running(void);
> +
> +#else
> +
> +static inline bool dsa_is_running(void)
> +{
> + return false;
> +}
> +
> +static inline int dsa_init(const char *dsa_parameter)
> +{
> + if (dsa_parameter != NULL && strlen(dsa_parameter) != 0) {
> + error_report("DSA not supported.");
> + return -1;
> + }
> +
> + return 0;
> +}
> +
> +static inline void dsa_start(void) {}
> +
> +static inline void dsa_stop(void) {}
> +
> +static inline void dsa_cleanup(void) {}
> +
> +#endif
> +
> +#endif
> diff --git a/util/dsa.c b/util/dsa.c
> new file mode 100644
> index 0000000000..05bbf8e31a
> --- /dev/null
> +++ b/util/dsa.c
> @@ -0,0 +1,316 @@
> +/*
> + * Use Intel Data Streaming Accelerator to offload certain background
> + * operations.
> + *
> + * Copyright (c) 2023 Hao Xiang <hao.xiang@bytedance.com>
> + * Bryan Zhang <bryan.zhang@bytedance.com>
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a copy
> + * of this software and associated documentation files (the "Software"), to deal
> + * in the Software without restriction, including without limitation the rights
> + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
> + * copies of the Software, and to permit persons to whom the Software is
> + * furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
> + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
> + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
> + * THE SOFTWARE.
> + */
> +
> +#include "qemu/osdep.h"
> +#include "qemu/queue.h"
> +#include "qemu/memalign.h"
> +#include "qemu/lockable.h"
> +#include "qemu/cutils.h"
> +#include "qemu/dsa.h"
> +#include "qemu/bswap.h"
> +#include "qemu/error-report.h"
> +#include "qemu/rcu.h"
> +
> +#ifdef CONFIG_DSA_OPT
This should be done in meson.build. Here you're allowing an empty object
to be generated just so we can expose the struct batch_task to
multifd. In another patch I suggested we stop using it, then we can put
the whole dsa.c under CONFIG_DSA (we can drop the OPT as well) in
meson.build.
> +
> +#pragma GCC push_options
> +#pragma GCC target("enqcmd")
> +
> +#include <linux/idxd.h>
> +#include "x86intrin.h"
> +
> +#define DSA_WQ_SIZE 4096
> +#define MAX_DSA_DEVICES 16
> +
> +typedef QSIMPLEQ_HEAD(dsa_task_queue, dsa_batch_task) dsa_task_queue;
> +
> +struct dsa_device {
> + void *work_queue;
> +};
> +
> +struct dsa_device_group {
> + struct dsa_device *dsa_devices;
> + int num_dsa_devices;
> + /* The index of the next DSA device to be used. */
> + uint32_t device_allocator_index;
> + bool running;
> + QemuMutex task_queue_lock;
> + QemuCond task_queue_cond;
> + dsa_task_queue task_queue;
> +};
> +
> +uint64_t max_retry_count;
> +static struct dsa_device_group dsa_group;
> +
> +
> +/**
> + * @brief This function opens a DSA device's work queue and
> + * maps the DSA device memory into the current process.
> + *
> + * @param dsa_wq_path A pointer to the DSA device work queue's file path.
> + * @return A pointer to the mapped memory, or MAP_FAILED on failure.
> + */
> +static void *
> +map_dsa_device(const char *dsa_wq_path)
> +{
> + void *dsa_device;
> + int fd;
> +
> + fd = open(dsa_wq_path, O_RDWR);
> + if (fd < 0) {
> + error_report("Open %s failed with errno = %d.",
> + dsa_wq_path, errno);
> + return MAP_FAILED;
> + }
> + dsa_device = mmap(NULL, DSA_WQ_SIZE, PROT_WRITE,
> + MAP_SHARED | MAP_POPULATE, fd, 0);
> + close(fd);
> + if (dsa_device == MAP_FAILED) {
> + error_report("mmap failed with errno = %d.", errno);
> + return MAP_FAILED;
> + }
> + return dsa_device;
> +}
> +
> +/**
> + * @brief Initializes a DSA device structure.
> + *
> + * @param instance A pointer to the DSA device.
> + * @param work_queue A pointer to the DSA work queue.
> + */
> +static void
> +dsa_device_init(struct dsa_device *instance,
> + void *dsa_work_queue)
> +{
> + instance->work_queue = dsa_work_queue;
> +}
> +
> +/**
> + * @brief Cleans up a DSA device structure.
> + *
> + * @param instance A pointer to the DSA device to cleanup.
> + */
> +static void
> +dsa_device_cleanup(struct dsa_device *instance)
> +{
> + if (instance->work_queue != MAP_FAILED) {
> + munmap(instance->work_queue, DSA_WQ_SIZE);
> + }
> +}
> +
> +/**
> + * @brief Initializes a DSA device group.
> + *
> + * @param group A pointer to the DSA device group.
> + * @param dsa_parameter A list of DSA device path from are separated by space
> + * character migration parameter. Multiple DSA device path.
> + *
> + * @return Zero if successful, non-zero otherwise.
> + */
> +static int
> +dsa_device_group_init(struct dsa_device_group *group,
> + const char *dsa_parameter)
> +{
> + if (dsa_parameter == NULL || strlen(dsa_parameter) == 0) {
> + return 0;
> + }
> +
> + int ret = 0;
> + char *local_dsa_parameter = g_strdup(dsa_parameter);
> + const char *dsa_path[MAX_DSA_DEVICES];
> + int num_dsa_devices = 0;
> + char delim[2] = " ";
> +
> + char *current_dsa_path = strtok(local_dsa_parameter, delim);
> +
> + while (current_dsa_path != NULL) {
> + dsa_path[num_dsa_devices++] = current_dsa_path;
> + if (num_dsa_devices == MAX_DSA_DEVICES) {
> + break;
> + }
> + current_dsa_path = strtok(NULL, delim);
> + }
> +
> + group->dsa_devices =
> + g_new0(struct dsa_device, num_dsa_devices);
> + group->num_dsa_devices = num_dsa_devices;
> + group->device_allocator_index = 0;
> +
> + group->running = false;
> + qemu_mutex_init(&group->task_queue_lock);
> + qemu_cond_init(&group->task_queue_cond);
> + QSIMPLEQ_INIT(&group->task_queue);
> +
> + void *dsa_wq = MAP_FAILED;
> + for (int i = 0; i < num_dsa_devices; i++) {
> + dsa_wq = map_dsa_device(dsa_path[i]);
> + if (dsa_wq == MAP_FAILED) {
> + error_report("map_dsa_device failed MAP_FAILED.");
> + ret = -1;
> + goto exit;
> + }
> + dsa_device_init(&dsa_group.dsa_devices[i], dsa_wq);
> + }
> +
> +exit:
> + g_free(local_dsa_parameter);
> + return ret;
> +}
> +
> +/**
> + * @brief Starts a DSA device group.
> + *
> + * @param group A pointer to the DSA device group.
> + */
> +static void
> +dsa_device_group_start(struct dsa_device_group *group)
> +{
> + group->running = true;
> +}
> +
> +/**
> + * @brief Stops a DSA device group.
> + *
> + * @param group A pointer to the DSA device group.
> + */
> +__attribute__((unused))
> +static void
> +dsa_device_group_stop(struct dsa_device_group *group)
> +{
> + group->running = false;
> +}
> +
> +/**
> + * @brief Cleans up a DSA device group.
> + *
> + * @param group A pointer to the DSA device group.
> + */
> +static void
> +dsa_device_group_cleanup(struct dsa_device_group *group)
> +{
> + if (!group->dsa_devices) {
> + return;
> + }
> + for (int i = 0; i < group->num_dsa_devices; i++) {
> + dsa_device_cleanup(&group->dsa_devices[i]);
> + }
> + g_free(group->dsa_devices);
> + group->dsa_devices = NULL;
> +
> + qemu_mutex_destroy(&group->task_queue_lock);
> + qemu_cond_destroy(&group->task_queue_cond);
> +}
> +
> +/**
> + * @brief Returns the next available DSA device in the group.
> + *
> + * @param group A pointer to the DSA device group.
> + *
> + * @return struct dsa_device* A pointer to the next available DSA device
> + * in the group.
> + */
> +__attribute__((unused))
> +static struct dsa_device *
> +dsa_device_group_get_next_device(struct dsa_device_group *group)
> +{
> + if (group->num_dsa_devices == 0) {
> + return NULL;
> + }
> + uint32_t current = qatomic_fetch_inc(&group->device_allocator_index);
> + current %= group->num_dsa_devices;
> + return &group->dsa_devices[current];
> +}
> +
> +/**
> + * @brief Check if DSA is running.
> + *
> + * @return True if DSA is running, otherwise false.
> + */
> +bool dsa_is_running(void)
> +{
> + return false;
> +}
> +
> +static void
> +dsa_globals_init(void)
> +{
> + max_retry_count = UINT64_MAX;
> +}
> +
> +/**
> + * @brief Initializes DSA devices.
> + *
> + * @param dsa_parameter A list of DSA device path from migration parameter.
> + *
> + * @return int Zero if successful, otherwise non zero.
> + */
> +int dsa_init(const char *dsa_parameter)
> +{
> + dsa_globals_init();
> +
> + return dsa_device_group_init(&dsa_group, dsa_parameter);
> +}
> +
> +/**
> + * @brief Start logic to enable using DSA.
> + *
> + */
> +void dsa_start(void)
> +{
> + if (dsa_group.num_dsa_devices == 0) {
> + return;
> + }
> + if (dsa_group.running) {
> + return;
> + }
> + dsa_device_group_start(&dsa_group);
> +}
> +
> +/**
> + * @brief Stop the device group and the completion thread.
> + *
> + */
> +void dsa_stop(void)
> +{
> + struct dsa_device_group *group = &dsa_group;
> +
> + if (!group->running) {
> + return;
> + }
> +}
> +
> +/**
> + * @brief Clean up system resources created for DSA offloading.
> + *
> + */
> +void dsa_cleanup(void)
> +{
> + dsa_stop();
> + dsa_device_group_cleanup(&dsa_group);
> +}
> +
> +#endif
> +
> diff --git a/util/meson.build b/util/meson.build
> index 2ad57b10ba..144c6812e5 100644
> --- a/util/meson.build
> +++ b/util/meson.build
> @@ -88,6 +88,7 @@ if have_block or have_ga
> endif
> if have_block
> util_ss.add(files('aio-wait.c'))
> + util_ss.add(files('dsa.c'))
> util_ss.add(files('buffer.c'))
> util_ss.add(files('bufferiszero.c'))
> util_ss.add(files('hbitmap.c'))
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v4 04/14] util/dsa: Implement DSA task enqueue and dequeue.
2024-04-25 2:21 ` [PATCH v4 04/14] util/dsa: Implement DSA task enqueue and dequeue Hao Xiang
2024-04-25 20:55 ` Fabiano Rosas
@ 2024-04-25 21:48 ` Fabiano Rosas
1 sibling, 0 replies; 32+ messages in thread
From: Fabiano Rosas @ 2024-04-25 21:48 UTC (permalink / raw)
To: Hao Xiang, marcandre.lureau, peterx, armbru, lvivier, qemu-devel
Cc: Hao Xiang
Hao Xiang <hao.xiang@linux.dev> writes:
> * Use a safe thread queue for DSA task enqueue/dequeue.
> * Implement DSA task submission.
> * Implement DSA batch task submission.
>
> Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
> ---
> include/qemu/dsa.h | 28 +++++++
> util/dsa.c | 201 +++++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 229 insertions(+)
>
> diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
> index f15c05ee85..37cae8d9d2 100644
> --- a/include/qemu/dsa.h
> +++ b/include/qemu/dsa.h
> @@ -13,6 +13,34 @@
> #include <linux/idxd.h>
> #include "x86intrin.h"
>
> +typedef enum DsaTaskType {
> + DSA_TASK = 0,
> + DSA_BATCH_TASK
> +} DsaTaskType;
> +
> +typedef enum DsaTaskStatus {
> + DSA_TASK_READY = 0,
> + DSA_TASK_PROCESSING,
> + DSA_TASK_COMPLETION
> +} DsaTaskStatus;
> +
> +typedef void (*dsa_completion_fn)(void *);
> +
> +typedef struct dsa_batch_task {
> + struct dsa_hw_desc batch_descriptor;
> + struct dsa_hw_desc *descriptors;
> + struct dsa_completion_record batch_completion __attribute__((aligned(32)));
> + struct dsa_completion_record *completions;
> + struct dsa_device_group *group;
> + struct dsa_device *device;
> + dsa_completion_fn completion_callback;
> + QemuSemaphore sem_task_complete;
> + DsaTaskType task_type;
> + DsaTaskStatus status;
> + int batch_size;
> + QSIMPLEQ_ENTRY(dsa_batch_task) entry;
> +} dsa_batch_task;
> +
> /**
> * @brief Initializes DSA devices.
> *
> diff --git a/util/dsa.c b/util/dsa.c
> index 05bbf8e31a..75739a1af6 100644
> --- a/util/dsa.c
> +++ b/util/dsa.c
> @@ -244,6 +244,205 @@ dsa_device_group_get_next_device(struct dsa_device_group *group)
> return &group->dsa_devices[current];
> }
>
> +/**
> + * @brief Empties out the DSA task queue.
> + *
> + * @param group A pointer to the DSA device group.
> + */
> +static void
> +dsa_empty_task_queue(struct dsa_device_group *group)
> +{
> + qemu_mutex_lock(&group->task_queue_lock);
> + dsa_task_queue *task_queue = &group->task_queue;
> + while (!QSIMPLEQ_EMPTY(task_queue)) {
> + QSIMPLEQ_REMOVE_HEAD(task_queue, entry);
> + }
> + qemu_mutex_unlock(&group->task_queue_lock);
> +}
> +
> +/**
> + * @brief Adds a task to the DSA task queue.
> + *
> + * @param group A pointer to the DSA device group.
> + * @param context A pointer to the DSA task to enqueue.
> + *
> + * @return int Zero if successful, otherwise a proper error code.
> + */
> +static int
> +dsa_task_enqueue(struct dsa_device_group *group,
> + struct dsa_batch_task *task)
> +{
> + dsa_task_queue *task_queue = &group->task_queue;
> + QemuMutex *task_queue_lock = &group->task_queue_lock;
> + QemuCond *task_queue_cond = &group->task_queue_cond;
> +
> + bool notify = false;
> +
> + qemu_mutex_lock(task_queue_lock);
> +
> + if (!group->running) {
> + error_report("DSA: Tried to queue task to stopped device queue.");
> + qemu_mutex_unlock(task_queue_lock);
> + return -1;
> + }
> +
> + /* The queue is empty. This enqueue operation is a 0->1 transition. */
> + if (QSIMPLEQ_EMPTY(task_queue)) {
> + notify = true;
> + }
> +
> + QSIMPLEQ_INSERT_TAIL(task_queue, task, entry);
> +
> + /* We need to notify the waiter for 0->1 transitions. */
> + if (notify) {
> + qemu_cond_signal(task_queue_cond);
> + }
> +
> + qemu_mutex_unlock(task_queue_lock);
> +
> + return 0;
> +}
> +
> +/**
> + * @brief Takes a DSA task out of the task queue.
> + *
> + * @param group A pointer to the DSA device group.
> + * @return dsa_batch_task* The DSA task being dequeued.
> + */
> +__attribute__((unused))
> +static struct dsa_batch_task *
> +dsa_task_dequeue(struct dsa_device_group *group)
> +{
> + struct dsa_batch_task *task = NULL;
> + dsa_task_queue *task_queue = &group->task_queue;
> + QemuMutex *task_queue_lock = &group->task_queue_lock;
> + QemuCond *task_queue_cond = &group->task_queue_cond;
> +
> + qemu_mutex_lock(task_queue_lock);
> +
> + while (true) {
> + if (!group->running) {
> + goto exit;
> + }
> + task = QSIMPLEQ_FIRST(task_queue);
> + if (task != NULL) {
> + break;
> + }
> + qemu_cond_wait(task_queue_cond, task_queue_lock);
> + }
> +
> + QSIMPLEQ_REMOVE_HEAD(task_queue, entry);
> +
> +exit:
> + qemu_mutex_unlock(task_queue_lock);
> + return task;
> +}
> +
> +/**
> + * @brief Submits a DSA work item to the device work queue.
> + *
> + * @param wq A pointer to the DSA work queue's device memory.
> + * @param descriptor A pointer to the DSA work item descriptor.
> + *
> + * @return Zero if successful, non-zero otherwise.
> + */
> +static int
> +submit_wi_int(void *wq, struct dsa_hw_desc *descriptor)
> +{
> + uint64_t retry = 0;
> +
> + _mm_sfence();
> +
> + while (true) {
> + if (_enqcmd(wq, descriptor) == 0) {
> + break;
> + }
> + retry++;
> + if (retry > max_retry_count) {
You missed my comment in v2 that max_retry_count is UINT64_MAX.
> + error_report("Submit work retry %lu times.", retry);
> + return -1;
> + }
> + }
> +
> + return 0;
> +}
> +
> +/**
> + * @brief Synchronously submits a DSA work item to the
> + * device work queue.
> + *
> + * @param wq A pointer to the DSA worjk queue's device memory.
> + * @param descriptor A pointer to the DSA work item descriptor.
> + *
> + * @return int Zero if successful, non-zero otherwise.
> + */
> +__attribute__((unused))
> +static int
> +submit_wi(void *wq, struct dsa_hw_desc *descriptor)
> +{
> + return submit_wi_int(wq, descriptor);
> +}
> +
> +/**
> + * @brief Asynchronously submits a DSA work item to the
> + * device work queue.
> + *
> + * @param task A pointer to the buffer zero task.
> + *
> + * @return int Zero if successful, non-zero otherwise.
> + */
> +__attribute__((unused))
> +static int
> +submit_wi_async(struct dsa_batch_task *task)
> +{
> + struct dsa_device_group *device_group = task->group;
> + struct dsa_device *device_instance = task->device;
> + int ret;
> +
> + assert(task->task_type == DSA_TASK);
> +
> + task->status = DSA_TASK_PROCESSING;
> +
> + ret = submit_wi_int(device_instance->work_queue,
> + &task->descriptors[0]);
> + if (ret != 0) {
> + return ret;
> + }
> +
> + return dsa_task_enqueue(device_group, task);
> +}
> +
> +/**
> + * @brief Asynchronously submits a DSA batch work item to the
> + * device work queue.
> + *
> + * @param dsa_batch_task A pointer to the batch buffer zero task.
> + *
> + * @return int Zero if successful, non-zero otherwise.
> + */
> +__attribute__((unused))
> +static int
> +submit_batch_wi_async(struct dsa_batch_task *batch_task)
> +{
> + struct dsa_device_group *device_group = batch_task->group;
> + struct dsa_device *device_instance = batch_task->device;
> + int ret;
> +
> + assert(batch_task->task_type == DSA_BATCH_TASK);
> + assert(batch_task->batch_descriptor.desc_count <= batch_task->batch_size);
> + assert(batch_task->status == DSA_TASK_READY);
> +
> + batch_task->status = DSA_TASK_PROCESSING;
> +
> + ret = submit_wi_int(device_instance->work_queue,
> + &batch_task->batch_descriptor);
> + if (ret != 0) {
> + return ret;
> + }
> +
> + return dsa_task_enqueue(device_group, batch_task);
> +}
> +
> /**
> * @brief Check if DSA is running.
> *
> @@ -300,6 +499,8 @@ void dsa_stop(void)
> if (!group->running) {
> return;
> }
> +
> + dsa_empty_task_queue(group);
> }
>
> /**
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v4 08/14] migration/multifd: Add new migration option for multifd DSA offloading.
2024-04-25 14:17 ` Daniel P. Berrangé
@ 2024-04-26 9:16 ` Markus Armbruster
0 siblings, 0 replies; 32+ messages in thread
From: Markus Armbruster @ 2024-04-26 9:16 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Hao Xiang, marcandre.lureau, peterx, farosas, armbru, lvivier,
qemu-devel
Daniel P. Berrangé <berrange@redhat.com> writes:
> On Thu, Apr 25, 2024 at 02:21:11AM +0000, Hao Xiang wrote:
>> Intel DSA offloading is an optional feature that turns on if
>> proper hardware and software stack is available. To turn on
>> DSA offloading in multifd live migration:
>>
>> multifd-dsa-accel="[dsa_dev_path1] [dsa_dev_path2] ... [dsa_dev_pathX]"
>>
>> This feature is turned off by default.
>>
>> Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
>> ---
>> migration/migration-hmp-cmds.c | 8 ++++++++
>> migration/options.c | 30 ++++++++++++++++++++++++++++++
>> migration/options.h | 1 +
>> qapi/migration.json | 26 +++++++++++++++++++++++---
>> 4 files changed, 62 insertions(+), 3 deletions(-)
>
>> diff --git a/qapi/migration.json b/qapi/migration.json
>> index 8c65b90328..934fa8839e 100644
>> --- a/qapi/migration.json
>> +++ b/qapi/migration.json
>> @@ -914,6 +914,12 @@
>> # See description in @ZeroPageDetection. Default is 'multifd'.
>> # (since 9.0)
>> #
>> +# @multifd-dsa-accel: If enabled, use DSA accelerator offloading for
>> +# certain memory operations. Enable DSA accelerator offloading by
>> +# setting this string to a list of DSA device path separated by space
>> +# characters. Setting this string to an empty string means disabling
>> +# DSA accelerator offloading. Defaults to an empty string. (since 9.2)
>
> Passing a list of paths as a single space separate string is a
> design anti-pattern. This needs to use a list type at the QAPI
> level.
Yup.
> Also I don't think we need add 'multifd' on the name - all
> new features are for multifd.
>
> Overall it should be called 'dsa-accel-path' I thjink
Moreover, docs/devel/qapi-code-gen.rst:
For legibility, wrap text paragraphs so every line is at most 70
characters long.
Separate sentences with two spaces.
[...]
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v4 07/14] util/dsa: Implement DSA task asynchronous submission and wait for completion.
2024-04-25 2:21 ` [PATCH v4 07/14] util/dsa: Implement DSA task asynchronous submission and wait for completion Hao Xiang
@ 2024-05-01 18:59 ` Peter Xu
0 siblings, 0 replies; 32+ messages in thread
From: Peter Xu @ 2024-05-01 18:59 UTC (permalink / raw)
To: Hao Xiang
Cc: marcandre.lureau, farosas, armbru, lvivier, qemu-devel, Bryan Zhang
On Thu, Apr 25, 2024 at 02:21:10AM +0000, Hao Xiang wrote:
> +/**
> + * @brief Performs buffer zero comparison on a DSA batch task asynchronously.
> + *
> + * @param batch_task A pointer to the batch task.
> + * @param buf An array of memory buffers.
> + * @param count The number of buffers in the array.
> + * @param len The buffer length.
> + *
> + * @return Zero if successful, otherwise non-zero.
> + */
> +int
> +buffer_is_zero_dsa_batch_async(struct dsa_batch_task *batch_task,
> + const void **buf, size_t count, size_t len)
It says it's "async", but then..
> +{
> + if (count <= 0 || count > batch_task->batch_size) {
> + return -1;
> + }
> +
> + assert(batch_task != NULL);
> + assert(len != 0);
> + assert(buf != NULL);
> +
> + if (count == 1) {
> + /* DSA doesn't take batch operation with only 1 task. */
> + buffer_zero_dsa_async(batch_task, buf[0], len);
> + } else {
> + buffer_zero_dsa_batch_async(batch_task, buf, count, len);
> + }
> +
> + buffer_zero_dsa_wait(batch_task);
... it waits always.
Wrong function name?
> + buffer_zero_cpu_fallback(batch_task);
Is this introducing yet another path even if it internally still uses
buffer_is_zero()?
Can we allow buffer_is_zero_dsa_batch_async() (or when it's renamed) fail
directly with a hint that it should fallback? Ultimately something like:
if (dsa_is_running() && zero_page_detect_dsa(p)) {
/* Succeeded */
return;
}
/* Use cpu detection by default, or as fallback */
zero_page_detect_cpu();
> +
> + return 0;
> +}
> +
> #endif
>
> --
> 2.30.2
>
>
--
Peter Xu
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v4 09/14] migration/multifd: Prepare to introduce DSA acceleration on the multifd path.
2024-04-25 2:21 ` [PATCH v4 09/14] migration/multifd: Prepare to introduce DSA acceleration on the multifd path Hao Xiang
@ 2024-05-01 19:18 ` Peter Xu
0 siblings, 0 replies; 32+ messages in thread
From: Peter Xu @ 2024-05-01 19:18 UTC (permalink / raw)
To: Hao Xiang; +Cc: marcandre.lureau, farosas, armbru, lvivier, qemu-devel
On Thu, Apr 25, 2024 at 02:21:12AM +0000, Hao Xiang wrote:
> 1. Refactor multifd_send_thread function.
> 2. Introduce the batch task structure in MultiFDSendParams.
>
> Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
> ---
> include/qemu/dsa.h | 51 +++++++++++++++++++++++++++++++++++++++++++--
> migration/multifd.c | 5 +++++
> migration/multifd.h | 2 ++
> util/dsa.c | 51 ++++++++++++++++++++++++++++++++++++++-------
> 4 files changed, 99 insertions(+), 10 deletions(-)
>
> diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
> index e002652879..0c36e93016 100644
> --- a/include/qemu/dsa.h
> +++ b/include/qemu/dsa.h
> @@ -2,6 +2,7 @@
> #define QEMU_DSA_H
>
> #include "qemu/error-report.h"
> +#include "exec/cpu-common.h"
> #include "qemu/thread.h"
> #include "qemu/queue.h"
>
> @@ -42,6 +43,21 @@ typedef struct dsa_batch_task {
> QSIMPLEQ_ENTRY(dsa_batch_task) entry;
> } dsa_batch_task;
>
> +#endif
> +
> +struct batch_task {
> +#ifdef CONFIG_DSA_OPT
> + /* Address of each pages in pages */
> + ram_addr_t *addr;
> + /* Zero page checking results */
> + bool *results;
> + /* Batch task DSA specific implementation */
> + struct dsa_batch_task *dsa_batch;
> +#endif
> +};
> +
> +#ifdef CONFIG_DSA_OPT
> +
> /**
> * @brief Initializes DSA devices.
> *
> @@ -74,7 +90,7 @@ void dsa_cleanup(void);
> bool dsa_is_running(void);
>
> /**
> - * @brief Initializes a buffer zero batch task.
> + * @brief Initializes a buffer zero DSA batch task.
> *
> * @param task A pointer to the batch task to initialize.
> * @param results A pointer to an array of zero page checking results.
> @@ -102,9 +118,26 @@ void buffer_zero_batch_task_destroy(struct dsa_batch_task *task);
> * @return Zero if successful, otherwise non-zero.
> */
> int
> -buffer_is_zero_dsa_batch_async(struct dsa_batch_task *batch_task,
> +buffer_is_zero_dsa_batch_async(struct batch_task *batch_task,
> const void **buf, size_t count, size_t len);
>
> +/**
> + * @brief Initializes a general buffer zero batch task.
> + *
> + * @param batch_size The number of zero page checking tasks in the batch.
> + * @return A pointer to the general batch task initialized.
> + */
> +struct batch_task *
> +batch_task_init(int batch_size);
> +
> +/**
> + * @brief Destroys a general buffer zero batch task.
> + *
> + * @param task A pointer to the general batch task to destroy.
> + */
> +void
> +batch_task_destroy(struct batch_task *task);
> +
> #else
>
> static inline bool dsa_is_running(void)
> @@ -128,6 +161,20 @@ static inline void dsa_stop(void) {}
>
> static inline void dsa_cleanup(void) {}
>
> +static inline int
> +buffer_is_zero_dsa_batch_async(struct batch_task *batch_task,
> + const void **buf, size_t count, size_t len)
> +{
> + exit(1);
> +}
> +
> +static inline struct batch_task *batch_task_init(int batch_size)
> +{
> + return NULL;
> +}
> +
> +static inline void batch_task_destroy(struct batch_task *task) {}
I feel like there're too many things exported for DSA.
For example, at least buffer_is_zero_dsa_batch_async() looks like not
needed to be exported, maybe what should be exported is
zero_page_detect_dsa()?
We also should avoid accessing dsa internal fields in multifd*.c generic
code, for example, I think we should avoid things like below:
MultiFDSendParams:
struct batch_task *batch_task;
multifd_send_setup:
if (dsa_init(dsa_parameter)) {
error_setg(&local_err, "multifd: Sender failed to initialize DSA.");
error_report_err(local_err);
return false;
}
dsa_start();
...
for (each_thread)
p->batch_task = batch_task_init(page_count);
This is way too ugly...
We should have one multifd_dsa_send_setup() and call it once and for all,
internally you can do whatever you want, rewalk the thread pool and init
states.
The name "batch_task" isn't clear either on being consumed by DSA. I'd
think something like "dsa_state" better.
So instead of above like:
struct batch_task {
#ifdef CONFIG_DSA_OPT
/* Address of each pages in pages */
ram_addr_t *addr;
/* Zero page checking results */
bool *results;
/* Batch task DSA specific implementation */
struct dsa_batch_task *dsa_batch;
#endif
};
The fields should always be defined (say, dsa_state), then:
struct dsa_state {
/* Address of each pages in pages */
ram_addr_t *addr;
/* Zero page checking results */
bool *results;
/* Batch task DSA specific implementation */
struct dsa_batch_task *dsa_batch;
};
MultiFDSendParams:
...
#ifdef CONFIG_DSA_OPT
struct dsa_state *dsa_state;
#endif
> +
> #endif
>
> #endif
> diff --git a/migration/multifd.c b/migration/multifd.c
> index f317bff077..cfd3a92f6c 100644
> --- a/migration/multifd.c
> +++ b/migration/multifd.c
> @@ -13,6 +13,8 @@
> #include "qemu/osdep.h"
> #include "qemu/cutils.h"
> #include "qemu/rcu.h"
> +#include "qemu/dsa.h"
> +#include "qemu/memalign.h"
> #include "exec/target_page.h"
> #include "sysemu/sysemu.h"
> #include "exec/ramblock.h"
> @@ -780,6 +782,8 @@ static bool multifd_send_cleanup_channel(MultiFDSendParams *p, Error **errp)
> p->name = NULL;
> multifd_pages_clear(p->pages);
> p->pages = NULL;
> + batch_task_destroy(p->batch_task);
> + p->batch_task = NULL;
Again, please try to export as less DSA relevant functions as possible.
Here IMHO we only need one dsa_state_destroy() on multifd_send_state, do
whatever inside.
> p->packet_len = 0;
> g_free(p->packet);
> p->packet = NULL;
> @@ -1172,6 +1176,7 @@ bool multifd_send_setup(void)
> qemu_sem_init(&p->sem_sync, 0);
> p->id = i;
> p->pages = multifd_pages_init(page_count);
> + p->batch_task = batch_task_init(page_count);
>
> if (use_packets) {
> p->packet_len = sizeof(MultiFDPacket_t)
> diff --git a/migration/multifd.h b/migration/multifd.h
> index c9d9b09239..16e27db5e9 100644
> --- a/migration/multifd.h
> +++ b/migration/multifd.h
> @@ -135,6 +135,8 @@ typedef struct {
> * pending_job != 0 -> multifd_channel can use it.
> */
> MultiFDPages_t *pages;
> + /* Zero page checking batch task */
> + struct batch_task *batch_task;
>
> /* thread local variables. No locking required */
>
> diff --git a/util/dsa.c b/util/dsa.c
> index 5a2bf33651..4f695e58af 100644
> --- a/util/dsa.c
> +++ b/util/dsa.c
> @@ -802,7 +802,7 @@ buffer_zero_task_init_int(struct dsa_hw_desc *descriptor,
> }
>
> /**
> - * @brief Initializes a buffer zero batch task.
> + * @brief Initializes a buffer zero DSA batch task.
> *
> * @param task A pointer to the batch task to initialize.
> * @param results A pointer to an array of zero page checking results.
> @@ -1107,29 +1107,64 @@ void dsa_cleanup(void)
> * @return Zero if successful, otherwise non-zero.
> */
> int
> -buffer_is_zero_dsa_batch_async(struct dsa_batch_task *batch_task,
> +buffer_is_zero_dsa_batch_async(struct batch_task *batch_task,
> const void **buf, size_t count, size_t len)
> {
> - if (count <= 0 || count > batch_task->batch_size) {
> + struct dsa_batch_task *dsa_batch = batch_task->dsa_batch;
> +
> + if (count <= 0 || count > dsa_batch->batch_size) {
> return -1;
> }
>
> - assert(batch_task != NULL);
> + assert(dsa_batch != NULL);
> assert(len != 0);
> assert(buf != NULL);
>
> if (count == 1) {
> /* DSA doesn't take batch operation with only 1 task. */
> - buffer_zero_dsa_async(batch_task, buf[0], len);
> + buffer_zero_dsa_async(dsa_batch, buf[0], len);
> } else {
> - buffer_zero_dsa_batch_async(batch_task, buf, count, len);
> + buffer_zero_dsa_batch_async(dsa_batch, buf, count, len);
> }
>
> - buffer_zero_dsa_wait(batch_task);
> - buffer_zero_cpu_fallback(batch_task);
> + buffer_zero_dsa_wait(dsa_batch);
> + buffer_zero_cpu_fallback(dsa_batch);
>
> return 0;
> }
>
> +/**
> + * @brief Initializes a general buffer zero batch task.
> + *
> + * @param batch_size The number of zero page checking tasks in the batch.
> + * @return A pointer to the general batch task initialized.
> + */
> +struct batch_task *
> +batch_task_init(int batch_size)
> +{
> + struct batch_task *task = g_malloc0(sizeof(struct batch_task));
> + task->addr = g_new0(ram_addr_t, batch_size);
> + task->results = g_new0(bool, batch_size);
> + task->dsa_batch = qemu_memalign(64, sizeof(struct dsa_batch_task));
> + buffer_zero_batch_task_init(task->dsa_batch, task->results, batch_size);
> +
> + return task;
> +}
> +
> +/**
> + * @brief Destroys a general buffer zero batch task.
> + *
> + * @param task A pointer to the general batch task to destroy.
> + */
> +void
> +batch_task_destroy(struct batch_task *task)
> +{
> + g_free(task->addr);
> + g_free(task->results);
> + buffer_zero_batch_task_destroy(task->dsa_batch);
> + qemu_vfree(task->dsa_batch);
> + g_free(task);
> +}
> +
> #endif
>
> --
> 2.30.2
>
>
--
Peter Xu
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v4 10/14] migration/multifd: Enable DSA offloading in multifd sender path.
2024-04-25 2:21 ` [PATCH v4 10/14] migration/multifd: Enable DSA offloading in multifd sender path Hao Xiang
2024-04-25 14:29 ` Daniel P. Berrangé
2024-04-25 15:39 ` Fabiano Rosas
@ 2024-05-01 19:25 ` Peter Xu
2 siblings, 0 replies; 32+ messages in thread
From: Peter Xu @ 2024-05-01 19:25 UTC (permalink / raw)
To: Hao Xiang; +Cc: marcandre.lureau, farosas, armbru, lvivier, qemu-devel
On Thu, Apr 25, 2024 at 02:21:13AM +0000, Hao Xiang wrote:
> Multifd sender path gets an array of pages queued by the migration
> thread. It performs zero page checking on every page in the array.
> The pages are classfied as either a zero page or a normal page. This
> change uses Intel DSA to offload the zero page checking from CPU to
> the DSA accelerator. The sender thread submits a batch of pages to DSA
> hardware and waits for the DSA completion thread to signal for work
> completion.
>
> Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
> ---
> migration/multifd-zero-page.c | 99 +++++++++++++++++++++++++++++++++--
> migration/multifd.c | 27 +++++++++-
> migration/multifd.h | 1 +
> 3 files changed, 120 insertions(+), 7 deletions(-)
>
> diff --git a/migration/multifd-zero-page.c b/migration/multifd-zero-page.c
> index e1b8370f88..4f426289e4 100644
> --- a/migration/multifd-zero-page.c
> +++ b/migration/multifd-zero-page.c
> @@ -37,25 +37,83 @@ static void swap_page_offset(ram_addr_t *pages_offset, int a, int b)
> }
>
> /**
> - * multifd_send_zero_page_detect: Perform zero page detection on all pages.
> + * zero_page_detect_cpu: Perform zero page detection using CPU.
> *
> * Sorts normal pages before zero pages in p->pages->offset and updates
> * p->pages->normal_num.
> *
> * @param p A pointer to the send params.
> */
> -void multifd_send_zero_page_detect(MultiFDSendParams *p)
> +static void zero_page_detect_cpu(MultiFDSendParams *p)
> {
> MultiFDPages_t *pages = p->pages;
> RAMBlock *rb = pages->block;
> int i = 0;
> int j = pages->num - 1;
>
> - if (!multifd_zero_page_enabled()) {
> - pages->normal_num = pages->num;
> + /*
> + * Sort the page offset array by moving all normal pages to
> + * the left and all zero pages to the right of the array.
> + */
> + while (i <= j) {
> + uint64_t offset = pages->offset[i];
> +
> + if (!buffer_is_zero(rb->host + offset, p->page_size)) {
> + i++;
> + continue;
> + }
> +
> + swap_page_offset(pages->offset, i, j);
> + ram_release_page(rb->idstr, offset);
> + j--;
> + }
> +
> + pages->normal_num = i;
> +}
> +
> +
> +#ifdef CONFIG_DSA_OPT
> +
> +static void swap_result(bool *results, int a, int b)
> +{
> + bool temp;
> +
> + if (a == b) {
> return;
> }
>
> + temp = results[a];
> + results[a] = results[b];
> + results[b] = temp;
> +}
> +
> +/**
> + * zero_page_detect_dsa: Perform zero page detection using
> + * Intel Data Streaming Accelerator (DSA).
> + *
> + * Sorts normal pages before zero pages in p->pages->offset and updates
> + * p->pages->normal_num.
> + *
> + * @param p A pointer to the send params.
> + */
> +static void zero_page_detect_dsa(MultiFDSendParams *p)
> +{
> + MultiFDPages_t *pages = p->pages;
> + RAMBlock *rb = pages->block;
> + bool *results = p->batch_task->results;
> +
> + for (int i = 0; i < p->pages->num; i++) {
> + p->batch_task->addr[i] = (ram_addr_t)(rb->host + p->pages->offset[i]);
> + }
> +
> + buffer_is_zero_dsa_batch_async(p->batch_task,
> + (const void **)p->batch_task->addr,
> + p->pages->num,
> + p->page_size);
> +
> + int i = 0;
> + int j = pages->num - 1;
> +
> /*
> * Sort the page offset array by moving all normal pages to
> * the left and all zero pages to the right of the array.
> @@ -63,11 +121,12 @@ void multifd_send_zero_page_detect(MultiFDSendParams *p)
> while (i <= j) {
> uint64_t offset = pages->offset[i];
>
> - if (!buffer_is_zero(rb->host + offset, p->page_size)) {
> + if (!results[i]) {
> i++;
> continue;
> }
>
> + swap_result(results, i, j);
> swap_page_offset(pages->offset, i, j);
> ram_release_page(rb->idstr, offset);
> j--;
> @@ -76,6 +135,15 @@ void multifd_send_zero_page_detect(MultiFDSendParams *p)
> pages->normal_num = i;
> }
>
> +#else
> +
> +static void zero_page_detect_dsa(MultiFDSendParams *p)
> +{
> + exit(1);
> +}
> +
> +#endif
> +
> void multifd_recv_zero_page_process(MultiFDRecvParams *p)
> {
> for (int i = 0; i < p->zero_num; i++) {
> @@ -87,3 +155,24 @@ void multifd_recv_zero_page_process(MultiFDRecvParams *p)
> }
> }
> }
> +
> +/**
> + * multifd_send_zero_page_detect: Perform zero page detection on all pages.
> + *
> + * @param p A pointer to the send params.
> + */
> +void multifd_send_zero_page_detect(MultiFDSendParams *p)
> +{
> + MultiFDPages_t *pages = p->pages;
> +
> + if (!multifd_zero_page_enabled()) {
> + pages->normal_num = pages->num;
> + return;
> + }
> +
> + if (dsa_is_running()) {
> + zero_page_detect_dsa(p);
> + } else {
> + zero_page_detect_cpu(p);
> + }
> +}
> diff --git a/migration/multifd.c b/migration/multifd.c
> index cfd3a92f6c..7316643d0a 100644
> --- a/migration/multifd.c
> +++ b/migration/multifd.c
> @@ -818,6 +818,8 @@ void multifd_send_shutdown(void)
>
> multifd_send_terminate_threads();
>
> + dsa_cleanup();
> +
> for (i = 0; i < migrate_multifd_channels(); i++) {
> MultiFDSendParams *p = &multifd_send_state->params[i];
> Error *local_err = NULL;
> @@ -1155,11 +1157,20 @@ bool multifd_send_setup(void)
> uint32_t page_count = MULTIFD_PACKET_SIZE / qemu_target_page_size();
> bool use_packets = multifd_use_packets();
> uint8_t i;
> + const char *dsa_parameter = migrate_multifd_dsa_accel();
>
> if (!migrate_multifd()) {
> return true;
> }
>
> + if (dsa_init(dsa_parameter)) {
> + error_setg(&local_err, "multifd: Sender failed to initialize DSA.");
> + error_report_err(local_err);
> + return false;
> + }
> +
> + dsa_start();
Commented on this in another reply, let's condense them into a single dsa
call. That call should also be nested into a flag to know at least dsa is
enabled:
if (multifd_dsa_enabled()) {
multifd_dsa_init();
}
Multifd used to have some legacy code where it randomly call multifd_*
functions directly in generic migration code. Not a good example to
follow there..
Also, I'd think it better if we don't introduce global vars for dsa, it
should be put under multifd_send_state (or recv_state). So anything like
dsa_start() should always take that pointer first.
> +
> thread_count = migrate_multifd_channels();
> multifd_send_state = g_malloc0(sizeof(*multifd_send_state));
> multifd_send_state->params = g_new0(MultiFDSendParams, thread_count);
> @@ -1393,6 +1404,7 @@ void multifd_recv_cleanup(void)
> qemu_thread_join(&p->thread);
> }
> }
> + dsa_cleanup();
Same here.
> for (i = 0; i < migrate_multifd_channels(); i++) {
> multifd_recv_cleanup_channel(&multifd_recv_state->params[i]);
> }
> @@ -1568,6 +1580,9 @@ int multifd_recv_setup(Error **errp)
> uint32_t page_count = MULTIFD_PACKET_SIZE / qemu_target_page_size();
> bool use_packets = multifd_use_packets();
> uint8_t i;
> + const char *dsa_parameter = migrate_multifd_dsa_accel();
> + int ret;
> + Error *local_err = NULL;
>
> /*
> * Return successfully if multiFD recv state is already initialised
> @@ -1577,6 +1592,15 @@ int multifd_recv_setup(Error **errp)
> return 0;
> }
>
> + ret = dsa_init(dsa_parameter);
> + if (ret != 0) {
> + error_setg(&local_err, "multifd: Receiver failed to initialize DSA.");
> + error_propagate(errp, local_err);
> + return ret;
> + }
> +
> + dsa_start();
Same here.
in the dsa case, it'll be even better if you can find a way to unify
send/recv, as IIUC they do work similarly, setup() some dsa stuff, do some
zero page detection, cleanup() some dsa stuff. They look all the same
irrelevant of src/dst. I think it's nice if we can merge them.
> +
> thread_count = migrate_multifd_channels();
> multifd_recv_state = g_malloc0(sizeof(*multifd_recv_state));
> multifd_recv_state->params = g_new0(MultiFDRecvParams, thread_count);
> @@ -1616,13 +1640,12 @@ int multifd_recv_setup(Error **errp)
>
> for (i = 0; i < thread_count; i++) {
> MultiFDRecvParams *p = &multifd_recv_state->params[i];
> - int ret;
> -
> ret = multifd_recv_state->ops->recv_setup(p, errp);
> if (ret) {
> return ret;
> }
> }
> +
> return 0;
> }
>
> diff --git a/migration/multifd.h b/migration/multifd.h
> index 16e27db5e9..b3717fae24 100644
> --- a/migration/multifd.h
> +++ b/migration/multifd.h
> @@ -14,6 +14,7 @@
> #define QEMU_MIGRATION_MULTIFD_H
>
> #include "ram.h"
> +#include "qemu/dsa.h"
>
> typedef struct MultiFDRecvData MultiFDRecvData;
>
> --
> 2.30.2
>
>
--
Peter Xu
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v4 11/14] migration/multifd: Add migration option set packet size.
2024-04-25 2:21 ` [PATCH v4 11/14] migration/multifd: Add migration option set packet size Hao Xiang
@ 2024-05-01 19:36 ` Peter Xu
0 siblings, 0 replies; 32+ messages in thread
From: Peter Xu @ 2024-05-01 19:36 UTC (permalink / raw)
To: Hao Xiang; +Cc: marcandre.lureau, farosas, armbru, lvivier, qemu-devel
On Thu, Apr 25, 2024 at 02:21:14AM +0000, Hao Xiang wrote:
> The current multifd packet size is 128 * 4kb. This change adds
> an option to set the packet size. Both sender and receiver needs
> to set the same packet size for things to work.
>
> Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
> ---
> migration/options.c | 36 ++++++++++++++++++++++++++++++++++++
> migration/options.h | 1 +
> qapi/migration.json | 21 ++++++++++++++++++---
> 3 files changed, 55 insertions(+), 3 deletions(-)
>
> diff --git a/migration/options.c b/migration/options.c
> index dc8642df81..a9deb079eb 100644
> --- a/migration/options.c
> +++ b/migration/options.c
> @@ -79,6 +79,12 @@
> #define DEFAULT_MIGRATE_ANNOUNCE_ROUNDS 5
> #define DEFAULT_MIGRATE_ANNOUNCE_STEP 100
>
> +/*
> + * Parameter for multifd packet size.
> + */
> +#define DEFAULT_MIGRATE_MULTIFD_PACKET_SIZE (128 * 4 * 1024)
> +#define MAX_MIGRATE_MULTIFD_PACKET_SIZE (1023 * 4 * 1024)
> +
> #define DEFINE_PROP_MIG_CAP(name, x) \
> DEFINE_PROP_BOOL(name, MigrationState, capabilities[x], false)
>
> @@ -184,6 +190,9 @@ Property migration_properties[] = {
> ZERO_PAGE_DETECTION_MULTIFD),
> DEFINE_PROP_STRING("multifd-dsa-accel", MigrationState,
> parameters.multifd_dsa_accel),
> + DEFINE_PROP_SIZE("multifd-packet-size", MigrationState,
> + parameters.multifd_packet_size,
> + DEFAULT_MIGRATE_MULTIFD_PACKET_SIZE),
Having such knob looks all fine, but I feel like this patch is half-baked,
no? There seems to have another part in the next patch. Maybe they need
to be squashed together.
>
> /* Migration capabilities */
> DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE),
> @@ -879,6 +888,13 @@ int migrate_multifd_channels(void)
> return s->parameters.multifd_channels;
> }
>
> +uint64_t migrate_multifd_packet_size(void)
> +{
> + MigrationState *s = migrate_get_current();
> +
> + return s->parameters.multifd_packet_size;
> +}
> +
> MultiFDCompression migrate_multifd_compression(void)
> {
> MigrationState *s = migrate_get_current();
> @@ -1031,6 +1047,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
> params->x_checkpoint_delay = s->parameters.x_checkpoint_delay;
> params->has_block_incremental = true;
> params->block_incremental = s->parameters.block_incremental;
> + params->has_multifd_packet_size = true;
> + params->multifd_packet_size = s->parameters.multifd_packet_size;
> params->has_multifd_channels = true;
> params->multifd_channels = s->parameters.multifd_channels;
> params->has_multifd_compression = true;
> @@ -1094,6 +1112,7 @@ void migrate_params_init(MigrationParameters *params)
> params->has_downtime_limit = true;
> params->has_x_checkpoint_delay = true;
> params->has_block_incremental = true;
> + params->has_multifd_packet_size = true;
> params->has_multifd_channels = true;
> params->has_multifd_compression = true;
> params->has_multifd_zlib_level = true;
> @@ -1195,6 +1214,17 @@ bool migrate_params_check(MigrationParameters *params, Error **errp)
>
> /* x_checkpoint_delay is now always positive */
>
> + if (params->has_multifd_packet_size &&
> + ((params->multifd_packet_size < DEFAULT_MIGRATE_MULTIFD_PACKET_SIZE) ||
> + (params->multifd_packet_size > MAX_MIGRATE_MULTIFD_PACKET_SIZE) ||
> + (params->multifd_packet_size % qemu_target_page_size() != 0))) {
> + error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
> + "multifd_packet_size",
> + "a value between 524288 and 4190208, "
We should reference the macros here.
> + "must be a multiple of guest VM's page size.");
> + return false;
> + }
> +
> if (params->has_multifd_channels && (params->multifd_channels < 1)) {
> error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
> "multifd_channels",
> @@ -1374,6 +1404,9 @@ static void migrate_params_test_apply(MigrateSetParameters *params,
> if (params->has_block_incremental) {
> dest->block_incremental = params->block_incremental;
> }
> + if (params->has_multifd_packet_size) {
> + dest->multifd_packet_size = params->multifd_packet_size;
> + }
> if (params->has_multifd_channels) {
> dest->multifd_channels = params->multifd_channels;
> }
> @@ -1524,6 +1557,9 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
> " use blockdev-mirror with NBD instead");
> s->parameters.block_incremental = params->block_incremental;
> }
> + if (params->has_multifd_packet_size) {
> + s->parameters.multifd_packet_size = params->multifd_packet_size;
> + }
> if (params->has_multifd_channels) {
> s->parameters.multifd_channels = params->multifd_channels;
> }
> diff --git a/migration/options.h b/migration/options.h
> index 1cb3393be9..23995e6608 100644
> --- a/migration/options.h
> +++ b/migration/options.h
> @@ -92,6 +92,7 @@ const char *migrate_tls_hostname(void);
> uint64_t migrate_xbzrle_cache_size(void);
> ZeroPageDetection migrate_zero_page_detection(void);
> const char *migrate_multifd_dsa_accel(void);
> +uint64_t migrate_multifd_packet_size(void);
>
> /* parameters setters */
>
> diff --git a/qapi/migration.json b/qapi/migration.json
> index 934fa8839e..39d609c394 100644
> --- a/qapi/migration.json
> +++ b/qapi/migration.json
> @@ -920,6 +920,10 @@
> # characters. Setting this string to an empty string means disabling
> # DSA accelerator offloading. Defaults to an empty string. (since 9.2)
> #
> +# @multifd-packet-size: Packet size in bytes used to migrate data.
> +# The value needs to be a multiple of guest VM's page size.
Maybe just call it "guest page size".
> +# The default value is 524288 and max value is 4190208. (Since 9.2)
IMHO we can avoid mentioning these in QAPI. This will be a very, very,
developer oriented value: if the default isn't the best to the majority of
people, we should change the default. Not easy for an admin to understand
what is this about.
I'm even thinking whether we should only expose it via one migration debug
option (-global migration.multifd-packet-size only), rather exporting it in
QMP or even HMP. Or do you want this actually to be tunable for real?
> +#
> # Features:
> #
> # @deprecated: Member @block-incremental is deprecated. Use
> @@ -954,7 +958,8 @@
> { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] },
> 'vcpu-dirty-limit',
> 'mode',
> - 'zero-page-detection'] }
> + 'zero-page-detection',
> + 'multifd-packet-size'] }
>
> ##
> # @MigrateSetParameters:
> @@ -1134,6 +1139,10 @@
> # characters. Setting this string to an empty string means disabling
> # DSA accelerator offloading. Defaults to an empty string. (since 9.2)
> #
> +# @multifd-packet-size: Packet size in bytes used to migrate data.
> +# The value needs to be a multiple of guest VM's page size.
> +# The default value is 524288 and max value is 4190208. (Since 9.2)
> +#
> # Features:
> #
> # @deprecated: Member @block-incremental is deprecated. Use
> @@ -1189,7 +1198,8 @@
> '*vcpu-dirty-limit': 'uint64',
> '*mode': 'MigMode',
> '*zero-page-detection': 'ZeroPageDetection',
> - '*multifd-dsa-accel': 'StrOrNull'} }
> + '*multifd-dsa-accel': 'StrOrNull',
> + '*multifd-packet-size' : 'uint64'} }
>
> ##
> # @migrate-set-parameters:
> @@ -1373,6 +1383,10 @@
> # characters. Setting this string to an empty string means disabling
> # DSA accelerator offloading. Defaults to an empty string. (since 9.2)
> #
> +# @multifd-packet-size: Packet size in bytes used to migrate data.
> +# The value needs to be a multiple of guest VM's page size.
> +# The default value is 524288 and max value is 4190208. (Since 9.2)
> +#
> # Features:
> #
> # @deprecated: Member @block-incremental is deprecated. Use
> @@ -1425,7 +1439,8 @@
> '*vcpu-dirty-limit': 'uint64',
> '*mode': 'MigMode',
> '*zero-page-detection': 'ZeroPageDetection',
> - '*multifd-dsa-accel': 'str'} }
> + '*multifd-dsa-accel': 'str',
> + '*multifd-packet-size': 'uint64'} }
>
> ##
> # @query-migrate-parameters:
> --
> 2.30.2
>
>
--
Peter Xu
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [PATCH v4 00/14] Use Intel DSA accelerator to offload zero page checking in multifd live migration.
2024-04-25 2:21 [PATCH v4 00/14] Use Intel DSA accelerator to offload zero page checking in multifd live migration Hao Xiang
` (13 preceding siblings ...)
2024-04-25 2:21 ` [PATCH v4 14/14] migration/multifd: Add integration tests for multifd with Intel DSA offloading Hao Xiang
@ 2024-05-01 19:54 ` Peter Xu
14 siblings, 0 replies; 32+ messages in thread
From: Peter Xu @ 2024-05-01 19:54 UTC (permalink / raw)
To: Hao Xiang; +Cc: marcandre.lureau, farosas, armbru, lvivier, qemu-devel
On Thu, Apr 25, 2024 at 02:21:03AM +0000, Hao Xiang wrote:
> 7. Added a new migration option multifd-normal-page-ratio to make
> multifd live migration easier to test. Setting a normal page ratio will
> make live migration recognize a zero page as a normal page and send
> the entire payload over the network. If we want to send a large network
> payload and analyze throughput, this option is useful.
I didn't see this when quickly going through the series. It's even
mentioned in test results later. Is it removed?
Thanks,
--
Peter Xu
^ permalink raw reply [flat|nested] 32+ messages in thread
end of thread, other threads:[~2024-05-01 19:54 UTC | newest]
Thread overview: 32+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-04-25 2:21 [PATCH v4 00/14] Use Intel DSA accelerator to offload zero page checking in multifd live migration Hao Xiang
2024-04-25 2:21 ` [PATCH v4 01/14] meson: Introduce new instruction set enqcmd to the build system Hao Xiang
2024-04-25 18:50 ` Fabiano Rosas
2024-04-25 2:21 ` [PATCH v4 02/14] util/dsa: Add dependency idxd Hao Xiang
2024-04-25 20:33 ` Fabiano Rosas
2024-04-25 2:21 ` [PATCH v4 03/14] util/dsa: Implement DSA device start and stop logic Hao Xiang
2024-04-25 14:21 ` Daniel P. Berrangé
2024-04-25 14:25 ` Daniel P. Berrangé
2024-04-25 14:32 ` Daniel P. Berrangé
2024-04-25 21:22 ` Fabiano Rosas
2024-04-25 2:21 ` [PATCH v4 04/14] util/dsa: Implement DSA task enqueue and dequeue Hao Xiang
2024-04-25 20:55 ` Fabiano Rosas
2024-04-25 21:48 ` Fabiano Rosas
2024-04-25 2:21 ` [PATCH v4 05/14] util/dsa: Implement DSA task asynchronous completion thread model Hao Xiang
2024-04-25 2:21 ` [PATCH v4 06/14] util/dsa: Implement zero page checking in DSA task Hao Xiang
2024-04-25 2:21 ` [PATCH v4 07/14] util/dsa: Implement DSA task asynchronous submission and wait for completion Hao Xiang
2024-05-01 18:59 ` Peter Xu
2024-04-25 2:21 ` [PATCH v4 08/14] migration/multifd: Add new migration option for multifd DSA offloading Hao Xiang
2024-04-25 14:17 ` Daniel P. Berrangé
2024-04-26 9:16 ` Markus Armbruster
2024-04-25 2:21 ` [PATCH v4 09/14] migration/multifd: Prepare to introduce DSA acceleration on the multifd path Hao Xiang
2024-05-01 19:18 ` Peter Xu
2024-04-25 2:21 ` [PATCH v4 10/14] migration/multifd: Enable DSA offloading in multifd sender path Hao Xiang
2024-04-25 14:29 ` Daniel P. Berrangé
2024-04-25 15:39 ` Fabiano Rosas
2024-05-01 19:25 ` Peter Xu
2024-04-25 2:21 ` [PATCH v4 11/14] migration/multifd: Add migration option set packet size Hao Xiang
2024-05-01 19:36 ` Peter Xu
2024-04-25 2:21 ` [PATCH v4 12/14] migration/multifd: Enable set packet size migration option Hao Xiang
2024-04-25 2:21 ` [PATCH v4 13/14] util/dsa: Add unit test coverage for Intel DSA task submission and completion Hao Xiang
2024-04-25 2:21 ` [PATCH v4 14/14] migration/multifd: Add integration tests for multifd with Intel DSA offloading Hao Xiang
2024-05-01 19:54 ` [PATCH v4 00/14] Use Intel DSA accelerator to offload zero page checking in multifd live migration Peter Xu
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.