From: Salil Mehta via <qemu-devel@nongnu.org>
To: <qemu-devel@nongnu.org>, <qemu-arm@nongnu.org>
Cc: <salil.mehta@huawei.com>, <maz@kernel.org>,
<jean-philippe@linaro.org>, <jonathan.cameron@huawei.com>,
<lpieralisi@kernel.org>, <peter.maydell@linaro.org>,
<richard.henderson@linaro.org>, <imammedo@redhat.com>,
<andrew.jones@linux.dev>, <david@redhat.com>, <philmd@linaro.org>,
<eric.auger@redhat.com>, <will@kernel.org>, <ardb@kernel.org>,
<oliver.upton@linux.dev>, <pbonzini@redhat.com>, <mst@redhat.com>,
<gshan@redhat.com>, <rafael@kernel.org>,
<borntraeger@linux.ibm.com>, <alex.bennee@linaro.org>,
<linux@armlinux.org.uk>, <darren@os.amperecomputing.com>,
<ilkka@os.amperecomputing.com>, <vishnu@os.amperecomputing.com>,
<karl.heubaum@oracle.com>, <miguel.luis@oracle.com>,
<salil.mehta@opnsrc.net>, <zhukeqian1@huawei.com>,
<wangxiongfeng2@huawei.com>, <wangyanan55@huawei.com>,
<jiakernel2@gmail.com>, <maobibo@loongson.cn>,
<lixianglai@loongson.cn>
Subject: [PATCH RFC V2 06/37] arm/virt, kvm: Pre-create disabled possible vCPUs @machine init
Date: Tue, 26 Sep 2023 11:04:05 +0100 [thread overview]
Message-ID: <20230926100436.28284-7-salil.mehta@huawei.com> (raw)
In-Reply-To: <20230926100436.28284-1-salil.mehta@huawei.com>
In ARMv8 architecture, GIC needs all the vCPUs to be created and present when
it is initialized. This is because:
1. GICC and MPIDR association must be fixed at the VM initialization time.
This is represented by register GIC_TYPER(mp_afffinity, proc_num)
2. GICC(cpu interfaces), GICR(redistributors) etc all must be initialized
at the boot time as well.
3. Memory regions associated with GICR etc. cannot be changed(add/del/mod)
after VM has inited.
This patch adds the support to pre-create all such possible vCPUs within the
host using the KVM interface as part of the virt machine initialization. These
vCPUs could later be attached to QOM/ACPI while they are actually hot plugged
and made present.
Co-developed-by: Salil Mehta <salil.mehta@huawei.com>
Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
Co-developed-by: Keqian Zhu <zhukeqian1@huawei.com>
Signed-off-by: Keqian Zhu <zhukeqian1@huawei.com>
Reported-by: Vishnu Pajjuri <vishnu@os.amperecomputing.com>
[VP: Identified CPU stall issue & suggested probable fix]
Signed-off-by: Salil Mehta <salil.mehta@huawei.com>
---
hw/arm/virt.c | 53 +++++++++++++++++++++++++++++++++++++++++--
include/hw/core/cpu.h | 1 +
target/arm/cpu64.c | 1 +
target/arm/kvm.c | 32 ++++++++++++++++++++++++++
target/arm/kvm64.c | 9 +++++++-
target/arm/kvm_arm.h | 11 +++++++++
6 files changed, 104 insertions(+), 3 deletions(-)
diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 3668ad27ec..6ba131b799 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -2293,8 +2293,10 @@ static void machvirt_init(MachineState *machine)
assert(possible_cpus->len == max_cpus);
for (n = 0; n < possible_cpus->len; n++) {
Object *cpuobj;
+ CPUState *cs;
cpuobj = object_new(possible_cpus->cpus[n].type);
+ cs = CPU(cpuobj);
aarch64 &= object_property_get_bool(cpuobj, "aarch64", NULL);
object_property_set_int(cpuobj, "socket-id",
@@ -2306,8 +2308,55 @@ static void machvirt_init(MachineState *machine)
object_property_set_int(cpuobj, "thread-id",
virt_get_thread_id(machine, n), NULL);
- qdev_realize(DEVICE(cpuobj), NULL, &error_fatal);
- object_unref(cpuobj);
+ if (n < smp_cpus) {
+ qdev_realize(DEVICE(cpuobj), NULL, &error_fatal);
+ object_unref(cpuobj);
+ } else {
+ CPUArchId *cpu_slot;
+
+ /* handling for vcpus which are yet to be hot-plugged */
+ cs->cpu_index = n;
+ cpu_slot = virt_find_cpu_slot(machine, cs->cpu_index);
+
+ /*
+ * ARM host vCPU features need to be fixed at the boot time. But as
+ * per current approach this CPU object will be destroyed during
+ * cpu_post_init(). During hotplug of vCPUs these properties are
+ * initialized again.
+ */
+ virt_cpu_set_properties(cpuobj, cpu_slot, &error_fatal);
+
+ /*
+ * For KVM, we shall be pre-creating the now disabled/un-plugged
+ * possbile host vcpus and park them till the time they are
+ * actually hot plugged. This is required to pre-size the host
+ * GICC and GICR with the all possible vcpus for this VM.
+ */
+ if (kvm_enabled()) {
+ kvm_arm_create_host_vcpu(ARM_CPU(cs));
+ }
+ /*
+ * Add disabled vCPU to CPU slot during the init phase of the virt
+ * machine
+ * 1. We need this ARMCPU object during the GIC init. This object
+ * will facilitate in pre-realizing the GIC. Any info like
+ * mp-affinity(required to derive gicr_type) etc. could still be
+ * fetched while preserving QOM abstraction akin to realized
+ * vCPUs.
+ * 2. Now, after initialization of the virt machine is complete we
+ * could use two approaches to deal with this ARMCPU object:
+ * (i) re-use this ARMCPU object during hotplug of this vCPU.
+ * OR
+ * (ii) defer release this ARMCPU object after gic has been
+ * initialized or during pre-plug phase when a vCPU is
+ * hotplugged.
+ *
+ * We will use the (ii) approach and release the ARMCPU objects
+ * after GIC and machine has been fully initialized during
+ * machine_init_done() phase.
+ */
+ cpu_slot->cpu = OBJECT(cs);
+ }
}
fdt_add_timer_nodes(vms);
fdt_add_cpu_nodes(vms);
diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
index e5af79950c..b2201a98ee 100644
--- a/include/hw/core/cpu.h
+++ b/include/hw/core/cpu.h
@@ -401,6 +401,7 @@ struct CPUState {
uint32_t kvm_fetch_index;
uint64_t dirty_pages;
int kvm_vcpu_stats_fd;
+ VMChangeStateEntry *vmcse;
/* Use by accel-block: CPU is executing an ioctl() */
QemuLockCnt in_ioctl_lock;
diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
index a660e3f483..3a38e7ccaf 100644
--- a/target/arm/cpu64.c
+++ b/target/arm/cpu64.c
@@ -748,6 +748,7 @@ static void aarch64_cpu_initfn(Object *obj)
* enabled explicitly
*/
cs->disabled = true;
+ cs->thread_id = 0;
}
static void aarch64_cpu_finalizefn(Object *obj)
diff --git a/target/arm/kvm.c b/target/arm/kvm.c
index b4c7654f49..0e1d0692b1 100644
--- a/target/arm/kvm.c
+++ b/target/arm/kvm.c
@@ -637,6 +637,38 @@ void kvm_arm_reset_vcpu(ARMCPU *cpu)
write_list_to_cpustate(cpu);
}
+void kvm_arm_create_host_vcpu(ARMCPU *cpu)
+{
+ CPUState *cs = CPU(cpu);
+ unsigned long vcpu_id = cs->cpu_index;
+ int ret;
+
+ ret = kvm_create_vcpu(cs);
+ if (ret < 0) {
+ error_report("Failed to create host vcpu %ld", vcpu_id);
+ abort();
+ }
+
+ /*
+ * Initialize the vCPU in the host. This will reset the sys regs
+ * for this vCPU and related registers like MPIDR_EL1 etc. also
+ * gets programmed during this call to host. These are referred
+ * later while setting device attributes of the GICR during GICv3
+ * reset
+ */
+ ret = kvm_arch_init_vcpu(cs);
+ if (ret < 0) {
+ error_report("Failed to initialize host vcpu %ld", vcpu_id);
+ abort();
+ }
+
+ /*
+ * park the created vCPU. shall be used during kvm_get_vcpu() when
+ * threads are created during realization of ARM vCPUs.
+ */
+ kvm_park_vcpu(cs);
+}
+
/*
* Update KVM's MP_STATE based on what QEMU thinks it is
*/
diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index 94bbd9661f..364cc21f81 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -566,7 +566,14 @@ int kvm_arch_init_vcpu(CPUState *cs)
return -EINVAL;
}
- qemu_add_vm_change_state_handler(kvm_arm_vm_state_change, cs);
+ /*
+ * Install VM change handler only when vCPU thread has been spawned
+ * i.e. vCPU is being realized
+ */
+ if (cs->thread_id) {
+ cs->vmcse = qemu_add_vm_change_state_handler(kvm_arm_vm_state_change,
+ cs);
+ }
/* Determine init features for this CPU */
memset(cpu->kvm_init_features, 0, sizeof(cpu->kvm_init_features));
diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
index 051a0da41c..31408499b3 100644
--- a/target/arm/kvm_arm.h
+++ b/target/arm/kvm_arm.h
@@ -163,6 +163,17 @@ void kvm_arm_cpu_post_load(ARMCPU *cpu);
*/
void kvm_arm_reset_vcpu(ARMCPU *cpu);
+/**
+ * kvm_arm_create_host_vcpu:
+ * @cpu: ARMCPU
+ *
+ * Called at to pre create all possible kvm vCPUs within the the host at the
+ * virt machine init time. This will also init this pre-created vCPU and
+ * hence result in vCPU reset at host. These pre created and inited vCPUs
+ * shall be parked for use when ARM vCPUs are actually realized.
+ */
+void kvm_arm_create_host_vcpu(ARMCPU *cpu);
+
/**
* kvm_arm_init_serror_injection:
* @cs: CPUState
--
2.34.1
next prev parent reply other threads:[~2023-09-26 10:08 UTC|newest]
Thread overview: 146+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-09-26 10:03 [PATCH RFC V2 00/37] Support of Virtual CPU Hotplug for ARMv8 Arch Salil Mehta via
2023-09-26 10:04 ` [PATCH RFC V2 01/37] arm/virt, target/arm: Add new ARMCPU {socket, cluster, core, thread}-id property Salil Mehta via
2023-09-26 23:57 ` [PATCH RFC V2 01/37] arm/virt,target/arm: Add new ARMCPU {socket,cluster,core,thread}-id property Gavin Shan
2023-10-02 9:53 ` Salil Mehta via
2023-10-02 9:53 ` Salil Mehta
2023-10-03 5:05 ` Gavin Shan
2023-09-26 10:04 ` [PATCH RFC V2 02/37] cpus-common: Add common CPU utility for possible vCPUs Salil Mehta via
2023-09-27 3:54 ` Gavin Shan
2023-10-02 10:21 ` Salil Mehta via
2023-10-02 10:21 ` Salil Mehta
2023-10-03 5:34 ` Gavin Shan
2023-09-26 10:04 ` [PATCH RFC V2 03/37] hw/arm/virt: Move setting of common CPU properties in a function Salil Mehta via
2023-09-27 5:16 ` Gavin Shan
2023-10-02 10:24 ` Salil Mehta via
2023-10-02 10:24 ` Salil Mehta
2023-10-10 6:46 ` Shaoqin Huang
2023-10-10 9:47 ` Salil Mehta via
2023-10-10 9:47 ` Salil Mehta
2023-09-26 10:04 ` [PATCH RFC V2 04/37] arm/virt, target/arm: Machine init time change common to vCPU {cold|hot}-plug Salil Mehta via
2023-09-27 6:28 ` [PATCH RFC V2 04/37] arm/virt,target/arm: " Gavin Shan
2023-10-02 16:12 ` Salil Mehta via
2023-10-02 16:12 ` Salil Mehta
2024-01-16 15:59 ` Jonathan Cameron via
2023-09-27 6:30 ` Gavin Shan
2023-10-02 10:27 ` Salil Mehta via
2023-10-02 10:27 ` Salil Mehta
2023-09-26 10:04 ` [PATCH RFC V2 05/37] accel/kvm: Extract common KVM vCPU {creation, parking} code Salil Mehta via
2023-09-27 6:51 ` [PATCH RFC V2 05/37] accel/kvm: Extract common KVM vCPU {creation,parking} code Gavin Shan
2023-10-02 16:20 ` Salil Mehta via
2023-10-02 16:20 ` Salil Mehta
2023-10-03 5:39 ` Gavin Shan
2023-09-26 10:04 ` Salil Mehta via [this message]
2023-09-27 10:04 ` [PATCH RFC V2 06/37] arm/virt,kvm: Pre-create disabled possible vCPUs @machine init Gavin Shan
2023-10-02 16:39 ` Salil Mehta via
2023-10-02 16:39 ` Salil Mehta
2023-09-26 10:04 ` [PATCH RFC V2 07/37] arm/virt, gicv3: Changes to pre-size GIC with possible vcpus " Salil Mehta via
2023-09-28 0:14 ` Gavin Shan
2023-10-16 16:15 ` Salil Mehta via
2023-10-16 16:15 ` Salil Mehta
2023-09-26 10:04 ` [PATCH RFC V2 08/37] arm/virt: Init PMU at host for all possible vcpus Salil Mehta via
2023-09-26 10:04 ` [PATCH RFC V2 09/37] hw/acpi: Move CPU ctrl-dev MMIO region len macro to common header file Salil Mehta via
2023-09-28 0:19 ` Gavin Shan
2023-10-16 16:20 ` Salil Mehta via
2023-10-16 16:20 ` Salil Mehta
2023-09-26 10:04 ` [PATCH RFC V2 10/37] arm/acpi: Enable ACPI support for vcpu hotplug Salil Mehta via
2023-09-28 0:25 ` Gavin Shan
2023-10-16 21:23 ` Salil Mehta via
2023-10-16 21:23 ` Salil Mehta
2023-09-26 10:04 ` [PATCH RFC V2 11/37] hw/acpi: Add ACPI CPU hotplug init stub Salil Mehta via
2023-09-28 0:28 ` Gavin Shan
2023-10-16 21:27 ` Salil Mehta via
2023-10-16 21:27 ` Salil Mehta
2023-09-26 10:04 ` [PATCH RFC V2 12/37] hw/acpi: Use qemu_present_cpu() API in ACPI CPU hotplug init Salil Mehta via
2023-09-28 0:40 ` Gavin Shan
2023-10-16 21:41 ` Salil Mehta via
2023-10-16 21:41 ` Salil Mehta
2023-09-26 10:04 ` [PATCH RFC V2 13/37] hw/acpi: Init GED framework with cpu hotplug events Salil Mehta via
2023-09-28 0:56 ` Gavin Shan
2023-10-16 21:44 ` Salil Mehta via
2023-10-16 21:44 ` Salil Mehta
2023-09-26 10:04 ` [PATCH RFC V2 14/37] arm/virt: Add cpu hotplug events to GED during creation Salil Mehta via
2023-09-28 1:03 ` Gavin Shan
2023-10-16 21:46 ` Salil Mehta via
2023-10-16 21:46 ` Salil Mehta
2023-09-26 10:04 ` [PATCH RFC V2 15/37] arm/virt: Create GED dev before *disabled* CPU Objs are destroyed Salil Mehta via
2023-09-28 1:08 ` Gavin Shan
2023-10-16 21:54 ` Salil Mehta via
2023-10-16 21:54 ` Salil Mehta
2023-09-26 10:04 ` [PATCH RFC V2 16/37] hw/acpi: Update CPUs AML with cpu-(ctrl)dev change Salil Mehta via
2023-09-28 1:26 ` Gavin Shan
2023-10-16 21:57 ` Salil Mehta via
2023-10-16 21:57 ` Salil Mehta
2023-09-26 10:04 ` [PATCH RFC V2 17/37] arm/virt/acpi: Build CPUs AML with CPU Hotplug support Salil Mehta via
2023-09-28 1:36 ` Gavin Shan
2023-10-16 22:05 ` Salil Mehta via
2023-10-16 22:05 ` Salil Mehta
2023-09-26 10:04 ` [PATCH RFC V2 18/37] arm/virt: Make ARM vCPU *present* status ACPI *persistent* Salil Mehta via
2023-09-28 23:18 ` Gavin Shan
2023-10-16 22:33 ` Salil Mehta via
2023-10-16 22:33 ` Salil Mehta
2023-09-26 10:04 ` [PATCH RFC V2 19/37] hw/acpi: ACPI/AML Changes to reflect the correct _STA.{PRES, ENA} Bits to Guest Salil Mehta via
2023-09-28 23:33 ` [PATCH RFC V2 19/37] hw/acpi: ACPI/AML Changes to reflect the correct _STA.{PRES,ENA} " Gavin Shan
2023-10-16 22:59 ` Salil Mehta via
2023-10-16 22:59 ` Salil Mehta
2024-01-17 21:46 ` [PATCH RFC V2 19/37] hw/acpi: ACPI/AML Changes to reflect the correct _STA.{PRES, ENA} " Jonathan Cameron via
2023-09-26 10:04 ` [PATCH RFC V2 20/37] hw/acpi: Update GED _EVT method AML with cpu scan Salil Mehta via
2023-09-28 23:35 ` Gavin Shan
2023-10-16 23:01 ` Salil Mehta via
2023-10-16 23:01 ` Salil Mehta
2023-09-26 10:04 ` [PATCH RFC V2 21/37] hw/arm: MADT Tbl change to size the guest with possible vCPUs Salil Mehta via
2023-09-28 23:43 ` Gavin Shan
2023-10-16 23:15 ` Salil Mehta via
2023-10-16 23:15 ` Salil Mehta
2023-09-26 10:04 ` [PATCH RFC V2 22/37] hw/acpi: Make _MAT method optional Salil Mehta via
2023-09-28 23:50 ` Gavin Shan
2023-10-16 23:17 ` Salil Mehta via
2023-10-16 23:17 ` Salil Mehta
2023-09-26 10:04 ` [PATCH RFC V2 23/37] arm/virt: Release objects for *disabled* possible vCPUs after init Salil Mehta via
2023-09-28 23:57 ` Gavin Shan
2023-10-16 23:28 ` Salil Mehta via
2023-10-16 23:28 ` Salil Mehta
2023-09-26 10:04 ` [PATCH RFC V2 24/37] hw/acpi: Update ACPI GED framework to support vCPU Hotplug Salil Mehta via
2023-09-26 11:02 ` Michael S. Tsirkin
2023-09-26 11:37 ` Salil Mehta via
2023-09-26 12:00 ` Michael S. Tsirkin
2023-09-26 12:27 ` Salil Mehta via
2023-09-26 13:02 ` lixianglai
2023-09-26 10:04 ` [PATCH RFC V2 25/37] arm/virt: Add/update basic hot-(un)plug framework Salil Mehta via
2023-09-29 0:20 ` Gavin Shan
2023-10-16 23:40 ` Salil Mehta via
2023-10-16 23:40 ` Salil Mehta
2023-09-26 10:04 ` [PATCH RFC V2 26/37] arm/virt: Changes to (un)wire GICC<->vCPU IRQs during hot-(un)plug Salil Mehta via
2023-09-26 10:04 ` [PATCH RFC V2 27/37] hw/arm, gicv3: Changes to update GIC with vCPU hot-plug notification Salil Mehta via
2023-09-26 10:04 ` [PATCH RFC V2 28/37] hw/intc/arm-gicv3*: Changes required to (re)init the vCPU register info Salil Mehta via
2023-09-26 10:04 ` [PATCH RFC V2 29/37] arm/virt: Update the guest(via GED) about CPU hot-(un)plug events Salil Mehta via
2023-09-29 0:30 ` Gavin Shan
2023-10-16 23:48 ` Salil Mehta via
2023-10-16 23:48 ` Salil Mehta
2023-09-26 10:04 ` [PATCH RFC V2 30/37] hw/arm: Changes required for reset and to support next boot Salil Mehta via
2023-09-26 10:04 ` [PATCH RFC V2 31/37] physmem, gdbstub: Common helping funcs/changes to *unrealize* vCPU Salil Mehta via
2023-10-03 6:33 ` [PATCH RFC V2 31/37] physmem,gdbstub: " Philippe Mathieu-Daudé
2023-10-03 10:22 ` Salil Mehta via
2023-10-03 10:22 ` Salil Mehta
2023-10-04 9:17 ` Salil Mehta via
2023-10-04 9:17 ` Salil Mehta
2023-09-26 10:36 ` [PATCH RFC V2 32/37] target/arm: Add support of *unrealize* ARMCPU during vCPU Hot-unplug Salil Mehta via
2023-09-26 10:36 ` [PATCH RFC V2 33/37] target/arm/kvm: Write CPU state back to KVM on reset Salil Mehta via
2023-09-26 10:36 ` [PATCH RFC V2 34/37] target/arm/kvm, tcg: Register/Handle SMCCC hypercall exits to VMM/Qemu Salil Mehta via
2023-09-29 4:15 ` [PATCH RFC V2 34/37] target/arm/kvm,tcg: " Gavin Shan
2023-10-17 0:03 ` Salil Mehta via
2023-10-17 0:03 ` Salil Mehta
2023-09-26 10:36 ` [PATCH RFC V2 35/37] hw/arm: Support hotplug capability check using _OSC method Salil Mehta via
2023-09-29 4:23 ` Gavin Shan
2023-10-17 0:13 ` Salil Mehta via
2023-10-17 0:13 ` Salil Mehta
2023-09-26 10:36 ` [PATCH RFC V2 36/37] tcg/mttcg: enable threads to unregister in tcg_ctxs[] Salil Mehta via
2023-09-26 10:36 ` [PATCH RFC V2 37/37] hw/arm/virt: Expose cold-booted CPUs as MADT GICC Enabled Salil Mehta via
2023-10-11 10:23 ` [PATCH RFC V2 00/37] Support of Virtual CPU Hotplug for ARMv8 Arch Vishnu Pajjuri
2023-10-11 10:32 ` Salil Mehta via
2023-10-11 10:32 ` Salil Mehta
2023-10-11 11:08 ` Vishnu Pajjuri
2023-10-11 20:15 ` Salil Mehta
2023-10-12 17:02 ` Miguel Luis
2023-10-12 17:54 ` Salil Mehta via
2023-10-12 17:54 ` Salil Mehta
2023-10-13 10:43 ` Miguel Luis
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230926100436.28284-7-salil.mehta@huawei.com \
--to=qemu-devel@nongnu.org \
--cc=alex.bennee@linaro.org \
--cc=andrew.jones@linux.dev \
--cc=ardb@kernel.org \
--cc=borntraeger@linux.ibm.com \
--cc=darren@os.amperecomputing.com \
--cc=david@redhat.com \
--cc=eric.auger@redhat.com \
--cc=gshan@redhat.com \
--cc=ilkka@os.amperecomputing.com \
--cc=imammedo@redhat.com \
--cc=jean-philippe@linaro.org \
--cc=jiakernel2@gmail.com \
--cc=jonathan.cameron@huawei.com \
--cc=karl.heubaum@oracle.com \
--cc=linux@armlinux.org.uk \
--cc=lixianglai@loongson.cn \
--cc=lpieralisi@kernel.org \
--cc=maobibo@loongson.cn \
--cc=maz@kernel.org \
--cc=miguel.luis@oracle.com \
--cc=mst@redhat.com \
--cc=oliver.upton@linux.dev \
--cc=pbonzini@redhat.com \
--cc=peter.maydell@linaro.org \
--cc=philmd@linaro.org \
--cc=qemu-arm@nongnu.org \
--cc=rafael@kernel.org \
--cc=richard.henderson@linaro.org \
--cc=salil.mehta@huawei.com \
--cc=salil.mehta@opnsrc.net \
--cc=vishnu@os.amperecomputing.com \
--cc=wangxiongfeng2@huawei.com \
--cc=wangyanan55@huawei.com \
--cc=will@kernel.org \
--cc=zhukeqian1@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).