[PATCH RFC v4 00/15] Add riscv kvm accel support

[PATCH RFC v4 00/15] Add riscv kvm accel support

[Yifei Jiang]

This series adds both riscv32 and riscv64 kvm support, and implements
migration based on riscv. It is based on temporarily unaccepted kvm:
https://github.com/kvm-riscv/linux (lastest version v15).

This series depends on above pending changes which haven't yet been
accepted, so this QEMU patch series is treated as RFC patches until
that dependency has been dealt with.

Compared to RFC v3, the time scaling is supported in this series. The
new feature also requires the following patches:
[1] Bugfix in kvm v15
https://lkml.org/lkml/2020/11/30/245
[2] kvm patches:
[PATCH RFC 0/3] Implement guest time scaling in RISC-V KVM

Several steps to use this:
1. Build emulation
$ ./configure --target-list=riscv64-softmmu
$ make -j$(nproc)

2. Build kernel
https://github.com/kvm-riscv/linux

3. Build QEMU VM
Cross built in riscv toolchain.
$ PKG_CONFIG_LIBDIR=<toolchain pkgconfig path>
$ export PKG_CONFIG_SYSROOT_DIR=<toolchain sysroot path>
$ ./configure --target-list=riscv64-softmmu --enable-kvm \
--cross-prefix=riscv64-linux-gnu- --disable-libiscsi --disable-glusterfs \
--disable-libusb --disable-usb-redir --audio-drv-list= --disable-opengl \
--disable-libxml2
$ make -j$(nproc)

4. Start emulation
$ ./qemu-system-riscv64 -M virt -m 4096M -cpu rv64,x-h=true -nographic \
        -name guest=riscv-hyp,debug-threads=on \
        -smp 4 \
        -bios ./fw_jump.bin \
        -kernel ./Image \
        -drive file=./hyp.img,format=raw,id=hd0 \
        -device virtio-blk-device,drive=hd0 \
        -append "root=/dev/vda rw console=ttyS0 earlycon=sbi"

5. Start kvm-acceled QEMU VM in emulation
$ ./qemu-system-riscv64 -M virt,accel=kvm -m 1024M -cpu host -nographic \
        -name guest=riscv-guset \
        -smp 2 \
        -bios none \
        -kernel ./Image \
        -drive file=./guest.img,format=raw,id=hd0 \
        -device virtio-blk-device,drive=hd0 \
        -append "root=/dev/vda rw console=ttyS0 earlycon=sbi"

Changes since RFC v3
- Rebase on QEMU v5.2.0-rc2 and kvm-riscv linux v15.
- Add time scaling support(New patches 13, 14 and 15).
- Fix the bug that guest vm can't reboot.

Changes since RFC v2
- Fix checkpatch error at target/riscv/sbi_ecall_interface.h.
- Add riscv migration support.

Changes since RFC v1
- Add separate SBI ecall interface header.
- Add riscv32 kvm accel support.

Yifei Jiang (15):
  linux-header: Update linux/kvm.h
  target/riscv: Add target/riscv/kvm.c to place the public kvm interface
  target/riscv: Implement function kvm_arch_init_vcpu
  target/riscv: Implement kvm_arch_get_registers
  target/riscv: Implement kvm_arch_put_registers
  target/riscv: Support start kernel directly by KVM
  hw/riscv: PLIC update external interrupt by KVM when kvm enabled
  target/riscv: Handle KVM_EXIT_RISCV_SBI exit
  target/riscv: Add host cpu type
  target/riscv: Add kvm_riscv_get/put_regs_timer
  target/riscv: Implement virtual time adjusting with vm state changing
  target/riscv: Support virtual time context synchronization
  target/riscv: Introduce dynamic time frequency for virt machine
  target/riscv: Synchronize vcpu's frequency with KVM
  target/riscv: Add time frequency migration support

 hw/intc/sifive_plic.c              |  31 +-
 hw/riscv/virt.c                    |  26 +-
 linux-headers/linux/kvm.h          |   8 +
 meson.build                        |   2 +
 target/riscv/cpu.c                 |  13 +
 target/riscv/cpu.h                 |  12 +
 target/riscv/kvm.c                 | 617 +++++++++++++++++++++++++++++
 target/riscv/kvm_riscv.h           |  25 ++
 target/riscv/machine.c             |  23 ++
 target/riscv/meson.build           |   1 +
 target/riscv/sbi_ecall_interface.h |  72 ++++
 11 files changed, 817 insertions(+), 13 deletions(-)
 create mode 100644 target/riscv/kvm.c
 create mode 100644 target/riscv/kvm_riscv.h
 create mode 100644 target/riscv/sbi_ecall_interface.h

-- 
2.19.1

[PATCH RFC v4 01/15] linux-header: Update linux/kvm.h

[Yifei Jiang]

Update linux-headers/linux/kvm.h from
https://github.com/avpatel/linux/tree/riscv_kvm_v15.
Only use this header file, so here do not update all linux headers by
update-linux-headers.sh before above KVM series is accepted.

Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
---
 linux-headers/linux/kvm.h | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
index 56ce14ad20..ddeedcf3df 100644
--- a/linux-headers/linux/kvm.h
+++ b/linux-headers/linux/kvm.h
@@ -250,6 +250,7 @@ struct kvm_hyperv_exit {
 #define KVM_EXIT_ARM_NISV         28
 #define KVM_EXIT_X86_RDMSR        29
 #define KVM_EXIT_X86_WRMSR        30
+#define KVM_EXIT_RISCV_SBI        31
 
 /* For KVM_EXIT_INTERNAL_ERROR */
 /* Emulate instruction failed. */
@@ -426,6 +427,13 @@ struct kvm_run {
 			__u32 index; /* kernel -> user */
 			__u64 data; /* kernel <-> user */
 		} msr;
+		/* KVM_EXIT_RISCV_SBI */
+		struct {
+			unsigned long extension_id;
+			unsigned long function_id;
+			unsigned long args[6];
+			unsigned long ret[2];
+		} riscv_sbi;
 		/* Fix the size of the union. */
 		char padding[256];
 	};
-- 
2.19.1

[PATCH RFC v4 02/15] target/riscv: Add target/riscv/kvm.c to place the public kvm interface

[Yifei Jiang]

Add target/riscv/kvm.c to place kvm_arch_* function needed by
kvm/kvm-all.c. Meanwhile, add kvm support in meson.build file.

Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
---
 meson.build              |   2 +
 target/riscv/kvm.c       | 128 +++++++++++++++++++++++++++++++++++++++
 target/riscv/meson.build |   1 +
 3 files changed, 131 insertions(+)
 create mode 100644 target/riscv/kvm.c

diff --git a/meson.build b/meson.build
index e3386196ba..14728b77ab 100644
--- a/meson.build
+++ b/meson.build
@@ -69,6 +69,8 @@ elif cpu in ['ppc', 'ppc64']
   kvm_targets = ['ppc-softmmu', 'ppc64-softmmu']
 elif cpu in ['mips', 'mips64']
   kvm_targets = ['mips-softmmu', 'mipsel-softmmu', 'mips64-softmmu', 'mips64el-softmmu']
+elif cpu in ['riscv32', 'riscv64']
+  kvm_targets = ['riscv32-softmmu', 'riscv64-softmmu']
 else
   kvm_targets = []
 endif
diff --git a/target/riscv/kvm.c b/target/riscv/kvm.c
new file mode 100644
index 0000000000..8c386d9acf
--- /dev/null
+++ b/target/riscv/kvm.c
@@ -0,0 +1,128 @@
+/*
+ * RISC-V implementation of KVM hooks
+ *
+ * Copyright (c) 2020 Huawei Technologies Co., Ltd
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2 or later, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "qemu/osdep.h"
+#include <sys/ioctl.h>
+
+#include <linux/kvm.h>
+
+#include "qemu-common.h"
+#include "qemu/timer.h"
+#include "qemu/error-report.h"
+#include "qemu/main-loop.h"
+#include "sysemu/sysemu.h"
+#include "sysemu/kvm.h"
+#include "sysemu/kvm_int.h"
+#include "cpu.h"
+#include "trace.h"
+#include "hw/pci/pci.h"
+#include "exec/memattrs.h"
+#include "exec/address-spaces.h"
+#include "hw/boards.h"
+#include "hw/irq.h"
+#include "qemu/log.h"
+#include "hw/loader.h"
+
+const KVMCapabilityInfo kvm_arch_required_capabilities[] = {
+    KVM_CAP_LAST_INFO
+};
+
+int kvm_arch_get_registers(CPUState *cs)
+{
+    return 0;
+}
+
+int kvm_arch_put_registers(CPUState *cs, int level)
+{
+    return 0;
+}
+
+int kvm_arch_release_virq_post(int virq)
+{
+    return 0;
+}
+
+int kvm_arch_fixup_msi_route(struct kvm_irq_routing_entry *route,
+                             uint64_t address, uint32_t data, PCIDevice *dev)
+{
+    return 0;
+}
+
+int kvm_arch_destroy_vcpu(CPUState *cs)
+{
+    return 0;
+}
+
+unsigned long kvm_arch_vcpu_id(CPUState *cpu)
+{
+    return cpu->cpu_index;
+}
+
+void kvm_arch_init_irq_routing(KVMState *s)
+{
+}
+
+int kvm_arch_init_vcpu(CPUState *cs)
+{
+    return 0;
+}
+
+int kvm_arch_msi_data_to_gsi(uint32_t data)
+{
+    abort();
+}
+
+int kvm_arch_add_msi_route_post(struct kvm_irq_routing_entry *route,
+                                int vector, PCIDevice *dev)
+{
+    return 0;
+}
+
+int kvm_arch_init(MachineState *ms, KVMState *s)
+{
+    return 0;
+}
+
+int kvm_arch_irqchip_create(KVMState *s)
+{
+    return 0;
+}
+
+int kvm_arch_process_async_events(CPUState *cs)
+{
+    return 0;
+}
+
+void kvm_arch_pre_run(CPUState *cs, struct kvm_run *run)
+{
+}
+
+MemTxAttrs kvm_arch_post_run(CPUState *cs, struct kvm_run *run)
+{
+    return MEMTXATTRS_UNSPECIFIED;
+}
+
+bool kvm_arch_stop_on_emulation_error(CPUState *cs)
+{
+    return true;
+}
+
+int kvm_arch_handle_exit(CPUState *cs, struct kvm_run *run)
+{
+    return 0;
+}
diff --git a/target/riscv/meson.build b/target/riscv/meson.build
index 14a5c62dac..12e9c074af 100644
--- a/target/riscv/meson.build
+++ b/target/riscv/meson.build
@@ -23,6 +23,7 @@ riscv_ss.add(files(
   'vector_helper.c',
   'translate.c',
 ))
+riscv_ss.add(when: 'CONFIG_KVM', if_true: files('kvm.c'))
 
 riscv_softmmu_ss = ss.source_set()
 riscv_softmmu_ss.add(files(
-- 
2.19.1

[PATCH RFC v4 03/15] target/riscv: Implement function kvm_arch_init_vcpu

[Yifei Jiang]

Get isa info from kvm while kvm init.

Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
---
 target/riscv/kvm.c | 27 ++++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)

diff --git a/target/riscv/kvm.c b/target/riscv/kvm.c
index 8c386d9acf..86660ba81b 100644
--- a/target/riscv/kvm.c
+++ b/target/riscv/kvm.c
@@ -38,6 +38,18 @@
 #include "qemu/log.h"
 #include "hw/loader.h"
 
+static __u64 kvm_riscv_reg_id(__u64 type, __u64 idx)
+{
+    __u64 id = KVM_REG_RISCV | type | idx;
+
+#if defined(TARGET_RISCV32)
+    id |= KVM_REG_SIZE_U32;
+#elif defined(TARGET_RISCV64)
+    id |= KVM_REG_SIZE_U64;
+#endif
+    return id;
+}
+
 const KVMCapabilityInfo kvm_arch_required_capabilities[] = {
     KVM_CAP_LAST_INFO
 };
@@ -79,7 +91,20 @@ void kvm_arch_init_irq_routing(KVMState *s)
 
 int kvm_arch_init_vcpu(CPUState *cs)
 {
-    return 0;
+    int ret = 0;
+    target_ulong isa;
+    RISCVCPU *cpu = RISCV_CPU(cs);
+    CPURISCVState *env = &cpu->env;
+    __u64 id;
+
+    id = kvm_riscv_reg_id(KVM_REG_RISCV_CONFIG, KVM_REG_RISCV_CONFIG_REG(isa));
+    ret = kvm_get_one_reg(cs, id, &isa);
+    if (ret) {
+        return ret;
+    }
+    env->misa = isa;
+
+    return ret;
 }
 
 int kvm_arch_msi_data_to_gsi(uint32_t data)
-- 
2.19.1

Re: [PATCH RFC v4 03/15] target/riscv: Implement function kvm_arch_init_vcpu

[Alistair Francis]

On Thu, Dec 3, 2020 at 4:55 AM Yifei Jiang <jiangyifei@huawei.com> wrote:
>
> Get isa info from kvm while kvm init.
>
> Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
> Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
> ---
>  target/riscv/kvm.c | 27 ++++++++++++++++++++++++++-
>  1 file changed, 26 insertions(+), 1 deletion(-)
>
> diff --git a/target/riscv/kvm.c b/target/riscv/kvm.c
> index 8c386d9acf..86660ba81b 100644
> --- a/target/riscv/kvm.c
> +++ b/target/riscv/kvm.c
> @@ -38,6 +38,18 @@
>  #include "qemu/log.h"
>  #include "hw/loader.h"
>
> +static __u64 kvm_riscv_reg_id(__u64 type, __u64 idx)
> +{
> +    __u64 id = KVM_REG_RISCV | type | idx;
> +
> +#if defined(TARGET_RISCV32)
> +    id |= KVM_REG_SIZE_U32;
> +#elif defined(TARGET_RISCV64)
> +    id |= KVM_REG_SIZE_U64;
> +#endif

There is a series on list (I'll send a v2 out later today) that starts
to remove these #ifdef for the RISC-V XLEN. Next time you rebase it
would be great if you can use that and hopefully remove this.

Alistair

> +    return id;
> +}
> +
>  const KVMCapabilityInfo kvm_arch_required_capabilities[] = {
>      KVM_CAP_LAST_INFO
>  };
> @@ -79,7 +91,20 @@ void kvm_arch_init_irq_routing(KVMState *s)
>
>  int kvm_arch_init_vcpu(CPUState *cs)
>  {
> -    return 0;
> +    int ret = 0;
> +    target_ulong isa;
> +    RISCVCPU *cpu = RISCV_CPU(cs);
> +    CPURISCVState *env = &cpu->env;
> +    __u64 id;
> +
> +    id = kvm_riscv_reg_id(KVM_REG_RISCV_CONFIG, KVM_REG_RISCV_CONFIG_REG(isa));
> +    ret = kvm_get_one_reg(cs, id, &isa);
> +    if (ret) {
> +        return ret;
> +    }
> +    env->misa = isa;
> +
> +    return ret;
>  }
>
>  int kvm_arch_msi_data_to_gsi(uint32_t data)
> --
> 2.19.1
>
>

[PATCH RFC v4 04/15] target/riscv: Implement kvm_arch_get_registers

[Yifei Jiang]

Get GPR CSR and FP registers from kvm by KVM_GET_ONE_REG ioctl.

Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
---
 target/riscv/kvm.c | 150 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 149 insertions(+), 1 deletion(-)

diff --git a/target/riscv/kvm.c b/target/riscv/kvm.c
index 86660ba81b..e679619e79 100644
--- a/target/riscv/kvm.c
+++ b/target/riscv/kvm.c
@@ -50,13 +50,161 @@ static __u64 kvm_riscv_reg_id(__u64 type, __u64 idx)
     return id;
 }
 
+#define RISCV_CORE_REG(name)  kvm_riscv_reg_id(KVM_REG_RISCV_CORE, \
+                 KVM_REG_RISCV_CORE_REG(name))
+
+#define RISCV_CSR_REG(name)  kvm_riscv_reg_id(KVM_REG_RISCV_CSR, \
+                 KVM_REG_RISCV_CSR_REG(name))
+
+#define RISCV_FP_F_REG(idx)  kvm_riscv_reg_id(KVM_REG_RISCV_FP_F, idx)
+
+#define RISCV_FP_D_REG(idx)  kvm_riscv_reg_id(KVM_REG_RISCV_FP_D, idx)
+
+static int kvm_riscv_get_regs_core(CPUState *cs)
+{
+    int ret = 0;
+    int i;
+    target_ulong reg;
+    CPURISCVState *env = &RISCV_CPU(cs)->env;
+
+    ret = kvm_get_one_reg(cs, RISCV_CORE_REG(regs.pc), &reg);
+    if (ret) {
+        return ret;
+    }
+    env->pc = reg;
+
+    for (i = 1; i < 32; i++) {
+        __u64 id = kvm_riscv_reg_id(KVM_REG_RISCV_CORE, i);
+        ret = kvm_get_one_reg(cs, id, &reg);
+        if (ret) {
+            return ret;
+        }
+        env->gpr[i] = reg;
+    }
+
+    return ret;
+}
+
+static int kvm_riscv_get_regs_csr(CPUState *cs)
+{
+    int ret = 0;
+    target_ulong reg;
+    CPURISCVState *env = &RISCV_CPU(cs)->env;
+
+    ret = kvm_get_one_reg(cs, RISCV_CSR_REG(sstatus), &reg);
+    if (ret) {
+        return ret;
+    }
+    env->mstatus = reg;
+
+    ret = kvm_get_one_reg(cs, RISCV_CSR_REG(sie), &reg);
+    if (ret) {
+        return ret;
+    }
+    env->mie = reg;
+
+    ret = kvm_get_one_reg(cs, RISCV_CSR_REG(stvec), &reg);
+    if (ret) {
+        return ret;
+    }
+    env->stvec = reg;
+
+    ret = kvm_get_one_reg(cs, RISCV_CSR_REG(sscratch), &reg);
+    if (ret) {
+        return ret;
+    }
+    env->sscratch = reg;
+
+    ret = kvm_get_one_reg(cs, RISCV_CSR_REG(sepc), &reg);
+    if (ret) {
+        return ret;
+    }
+    env->sepc = reg;
+
+    ret = kvm_get_one_reg(cs, RISCV_CSR_REG(scause), &reg);
+    if (ret) {
+        return ret;
+    }
+    env->scause = reg;
+
+    ret = kvm_get_one_reg(cs, RISCV_CSR_REG(stval), &reg);
+    if (ret) {
+        return ret;
+    }
+    env->sbadaddr = reg;
+
+    ret = kvm_get_one_reg(cs, RISCV_CSR_REG(sip), &reg);
+    if (ret) {
+        return ret;
+    }
+    env->mip = reg;
+
+    ret = kvm_get_one_reg(cs, RISCV_CSR_REG(satp), &reg);
+    if (ret) {
+        return ret;
+    }
+    env->satp = reg;
+
+    return ret;
+}
+
+static int kvm_riscv_get_regs_fp(CPUState *cs)
+{
+    int ret = 0;
+    int i;
+    CPURISCVState *env = &RISCV_CPU(cs)->env;
+
+    if (riscv_has_ext(env, RVD)) {
+        uint64_t reg;
+        for (i = 0; i < 32; i++) {
+            ret = kvm_get_one_reg(cs, RISCV_FP_D_REG(i), &reg);
+            if (ret) {
+                return ret;
+            }
+            env->fpr[i] = reg;
+        }
+        return ret;
+    }
+
+    if (riscv_has_ext(env, RVF)) {
+        uint32_t reg;
+        for (i = 0; i < 32; i++) {
+            ret = kvm_get_one_reg(cs, RISCV_FP_F_REG(i), &reg);
+            if (ret) {
+                return ret;
+            }
+            env->fpr[i] = reg;
+        }
+        return ret;
+    }
+
+    return ret;
+}
+
 const KVMCapabilityInfo kvm_arch_required_capabilities[] = {
     KVM_CAP_LAST_INFO
 };
 
 int kvm_arch_get_registers(CPUState *cs)
 {
-    return 0;
+    int ret = 0;
+
+    ret = kvm_riscv_get_regs_core(cs);
+    if (ret) {
+        return ret;
+    }
+
+    ret = kvm_riscv_get_regs_csr(cs);
+    if (ret) {
+        return ret;
+    }
+
+    ret = kvm_riscv_get_regs_fp(cs);
+    if (ret) {
+        return ret;
+    }
+
+    return ret;
 }
 
 int kvm_arch_put_registers(CPUState *cs, int level)
-- 
2.19.1

[PATCH RFC v4 05/15] target/riscv: Implement kvm_arch_put_registers

[Yifei Jiang]

Put GPR CSR and FP registers to kvm by KVM_SET_ONE_REG ioctl

Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
---
 target/riscv/kvm.c | 142 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 141 insertions(+), 1 deletion(-)

diff --git a/target/riscv/kvm.c b/target/riscv/kvm.c
index e679619e79..8b206ce99c 100644
--- a/target/riscv/kvm.c
+++ b/target/riscv/kvm.c
@@ -85,6 +85,31 @@ static int kvm_riscv_get_regs_core(CPUState *cs)
     return ret;
 }
 
+static int kvm_riscv_put_regs_core(CPUState *cs)
+{
+    int ret = 0;
+    int i;
+    target_ulong reg;
+    CPURISCVState *env = &RISCV_CPU(cs)->env;
+
+    reg = env->pc;
+    ret = kvm_set_one_reg(cs, RISCV_CORE_REG(regs.pc), &reg);
+    if (ret) {
+        return ret;
+    }
+
+    for (i = 1; i < 32; i++) {
+        __u64 id = kvm_riscv_reg_id(KVM_REG_RISCV_CORE, i);
+        reg = env->gpr[i];
+        ret = kvm_set_one_reg(cs, id, &reg);
+        if (ret) {
+            return ret;
+        }
+    }
+
+    return ret;
+}
+
 static int kvm_riscv_get_regs_csr(CPUState *cs)
 {
     int ret = 0;
@@ -148,6 +173,70 @@ static int kvm_riscv_get_regs_csr(CPUState *cs)
     return ret;
 }
 
+static int kvm_riscv_put_regs_csr(CPUState *cs)
+{
+    int ret = 0;
+    target_ulong reg;
+    CPURISCVState *env = &RISCV_CPU(cs)->env;
+
+    reg = env->mstatus;
+    ret = kvm_set_one_reg(cs, RISCV_CSR_REG(sstatus), &reg);
+    if (ret) {
+        return ret;
+    }
+
+    reg = env->mie;
+    ret = kvm_set_one_reg(cs, RISCV_CSR_REG(sie), &reg);
+    if (ret) {
+        return ret;
+    }
+
+    reg = env->stvec;
+    ret = kvm_set_one_reg(cs, RISCV_CSR_REG(stvec), &reg);
+    if (ret) {
+        return ret;
+    }
+
+    reg = env->sscratch;
+    ret = kvm_set_one_reg(cs, RISCV_CSR_REG(sscratch), &reg);
+    if (ret) {
+        return ret;
+    }
+
+    reg = env->sepc;
+    ret = kvm_set_one_reg(cs, RISCV_CSR_REG(sepc), &reg);
+    if (ret) {
+        return ret;
+    }
+
+    reg = env->scause;
+    ret = kvm_set_one_reg(cs, RISCV_CSR_REG(scause), &reg);
+    if (ret) {
+        return ret;
+    }
+
+    reg = env->sbadaddr;
+    ret = kvm_set_one_reg(cs, RISCV_CSR_REG(stval), &reg);
+    if (ret) {
+        return ret;
+    }
+
+    reg = env->mip;
+    ret = kvm_set_one_reg(cs, RISCV_CSR_REG(sip), &reg);
+    if (ret) {
+        return ret;
+    }
+
+    reg = env->satp;
+    ret = kvm_set_one_reg(cs, RISCV_CSR_REG(satp), &reg);
+    if (ret) {
+        return ret;
+    }
+
+    return ret;
+}
+
+
 static int kvm_riscv_get_regs_fp(CPUState *cs)
 {
     int ret = 0;
@@ -181,6 +270,40 @@ static int kvm_riscv_get_regs_fp(CPUState *cs)
     return ret;
 }
 
+static int kvm_riscv_put_regs_fp(CPUState *cs)
+{
+    int ret = 0;
+    int i;
+    CPURISCVState *env = &RISCV_CPU(cs)->env;
+
+    if (riscv_has_ext(env, RVD)) {
+        uint64_t reg;
+        for (i = 0; i < 32; i++) {
+            reg = env->fpr[i];
+            ret = kvm_set_one_reg(cs, RISCV_FP_D_REG(i), &reg);
+            if (ret) {
+                return ret;
+            }
+        }
+        return ret;
+    }
+
+    if (riscv_has_ext(env, RVF)) {
+        uint32_t reg;
+        for (i = 0; i < 32; i++) {
+            reg = env->fpr[i];
+            ret = kvm_set_one_reg(cs, RISCV_FP_F_REG(i), &reg);
+            if (ret) {
+                return ret;
+            }
+        }
+        return ret;
+    }
+
+    return ret;
+}
+
+
 const KVMCapabilityInfo kvm_arch_required_capabilities[] = {
     KVM_CAP_LAST_INFO
 };
@@ -209,7 +332,24 @@ int kvm_arch_get_registers(CPUState *cs)
 
 int kvm_arch_put_registers(CPUState *cs, int level)
 {
-    return 0;
+    int ret = 0;
+
+    ret = kvm_riscv_put_regs_core(cs);
+    if (ret) {
+        return ret;
+    }
+
+    ret = kvm_riscv_put_regs_csr(cs);
+    if (ret) {
+        return ret;
+    }
+
+    ret = kvm_riscv_put_regs_fp(cs);
+    if (ret) {
+        return ret;
+    }
+
+    return ret;
 }
 
 int kvm_arch_release_virq_post(int virq)
-- 
2.19.1

[PATCH RFC v4 06/15] target/riscv: Support start kernel directly by KVM

[Yifei Jiang]

Get kernel and fdt start address in virt.c, and pass them to KVM
when cpu reset. In addition, add kvm_riscv.h to place riscv specific
interface.

Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
---
 hw/riscv/virt.c          |  8 ++++++++
 target/riscv/cpu.c       |  4 ++++
 target/riscv/cpu.h       |  3 +++
 target/riscv/kvm.c       | 15 +++++++++++++++
 target/riscv/kvm_riscv.h | 24 ++++++++++++++++++++++++
 5 files changed, 54 insertions(+)
 create mode 100644 target/riscv/kvm_riscv.h

diff --git a/hw/riscv/virt.c b/hw/riscv/virt.c
index 25cea7aa67..47b7018193 100644
--- a/hw/riscv/virt.c
+++ b/hw/riscv/virt.c
@@ -42,6 +42,7 @@
 #include "sysemu/sysemu.h"
 #include "hw/pci/pci.h"
 #include "hw/pci-host/gpex.h"
+#include "sysemu/kvm.h"
 
 #if defined(TARGET_RISCV32)
 # define BIOS_FILENAME "opensbi-riscv32-generic-fw_dynamic.bin"
@@ -511,6 +512,7 @@ static void virt_machine_init(MachineState *machine)
     uint64_t kernel_entry;
     DeviceState *mmio_plic, *virtio_plic, *pcie_plic;
     int i, j, base_hartid, hart_count;
+    CPUState *cs;
 
     /* Check socket count limit */
     if (VIRT_SOCKETS_MAX < riscv_socket_count(machine)) {
@@ -660,6 +662,12 @@ static void virt_machine_init(MachineState *machine)
                               virt_memmap[VIRT_MROM].size, kernel_entry,
                               fdt_load_addr, s->fdt);
 
+    for (cs = first_cpu; cs; cs = CPU_NEXT(cs)) {
+        RISCVCPU *riscv_cpu = RISCV_CPU(cs);
+        riscv_cpu->env.kernel_addr = kernel_entry;
+        riscv_cpu->env.fdt_addr = fdt_load_addr;
+    }
+
     /* SiFive Test MMIO device */
     sifive_test_create(memmap[VIRT_TEST].base);
 
diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c
index 6a0264fc6b..faee98a58c 100644
--- a/target/riscv/cpu.c
+++ b/target/riscv/cpu.c
@@ -29,6 +29,7 @@
 #include "hw/qdev-properties.h"
 #include "migration/vmstate.h"
 #include "fpu/softfloat-helpers.h"
+#include "kvm_riscv.h"
 
 /* RISC-V CPU definitions */
 
@@ -330,6 +331,9 @@ static void riscv_cpu_reset(DeviceState *dev)
     cs->exception_index = EXCP_NONE;
     env->load_res = -1;
     set_default_nan_mode(1, &env->fp_status);
+#ifdef CONFIG_KVM
+    kvm_riscv_reset_vcpu(cpu);
+#endif
 }
 
 static void riscv_cpu_disas_set_info(CPUState *s, disassemble_info *info)
diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h
index c0a326c843..ad1c90f798 100644
--- a/target/riscv/cpu.h
+++ b/target/riscv/cpu.h
@@ -233,6 +233,9 @@ struct CPURISCVState {
 
     /* Fields from here on are preserved across CPU reset. */
     QEMUTimer *timer; /* Internal timer */
+
+    hwaddr kernel_addr;
+    hwaddr fdt_addr;
 };
 
 OBJECT_DECLARE_TYPE(RISCVCPU, RISCVCPUClass,
diff --git a/target/riscv/kvm.c b/target/riscv/kvm.c
index 8b206ce99c..6250ca0c7d 100644
--- a/target/riscv/kvm.c
+++ b/target/riscv/kvm.c
@@ -37,6 +37,7 @@
 #include "hw/irq.h"
 #include "qemu/log.h"
 #include "hw/loader.h"
+#include "kvm_riscv.h"
 
 static __u64 kvm_riscv_reg_id(__u64 type, __u64 idx)
 {
@@ -439,3 +440,17 @@ int kvm_arch_handle_exit(CPUState *cs, struct kvm_run *run)
 {
     return 0;
 }
+
+void kvm_riscv_reset_vcpu(RISCVCPU *cpu)
+{
+    CPURISCVState *env = &cpu->env;
+
+    if (!kvm_enabled()) {
+        return;
+    }
+    env->pc = cpu->env.kernel_addr;
+    env->gpr[10] = kvm_arch_vcpu_id(CPU(cpu)); /* a0 */
+    env->gpr[11] = cpu->env.fdt_addr;          /* a1 */
+    env->satp = 0;
+}
+
diff --git a/target/riscv/kvm_riscv.h b/target/riscv/kvm_riscv.h
new file mode 100644
index 0000000000..f38c82bf59
--- /dev/null
+++ b/target/riscv/kvm_riscv.h
@@ -0,0 +1,24 @@
+/*
+ * QEMU KVM support -- RISC-V specific functions.
+ *
+ * Copyright (c) 2020 Huawei Technologies Co., Ltd
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2 or later, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef QEMU_KVM_RISCV_H
+#define QEMU_KVM_RISCV_H
+
+void kvm_riscv_reset_vcpu(RISCVCPU *cpu);
+
+#endif
-- 
2.19.1

Re: [PATCH RFC v4 06/15] target/riscv: Support start kernel directly by KVM

[Alistair Francis]

On Thu, Dec 3, 2020 at 4:58 AM Yifei Jiang <jiangyifei@huawei.com> wrote:
>
> Get kernel and fdt start address in virt.c, and pass them to KVM
> when cpu reset. In addition, add kvm_riscv.h to place riscv specific
> interface.

This doesn't seem right. Why do we need to do this? Other
architectures don't seem to do this.

Writing to the CPU from the board like this looks fishy and probably
breaks some QOM rules.

Alistair

>
> Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
> Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
> ---
>  hw/riscv/virt.c          |  8 ++++++++
>  target/riscv/cpu.c       |  4 ++++
>  target/riscv/cpu.h       |  3 +++
>  target/riscv/kvm.c       | 15 +++++++++++++++
>  target/riscv/kvm_riscv.h | 24 ++++++++++++++++++++++++
>  5 files changed, 54 insertions(+)
>  create mode 100644 target/riscv/kvm_riscv.h
>
> diff --git a/hw/riscv/virt.c b/hw/riscv/virt.c
> index 25cea7aa67..47b7018193 100644
> --- a/hw/riscv/virt.c
> +++ b/hw/riscv/virt.c
> @@ -42,6 +42,7 @@
>  #include "sysemu/sysemu.h"
>  #include "hw/pci/pci.h"
>  #include "hw/pci-host/gpex.h"
> +#include "sysemu/kvm.h"
>
>  #if defined(TARGET_RISCV32)
>  # define BIOS_FILENAME "opensbi-riscv32-generic-fw_dynamic.bin"
> @@ -511,6 +512,7 @@ static void virt_machine_init(MachineState *machine)
>      uint64_t kernel_entry;
>      DeviceState *mmio_plic, *virtio_plic, *pcie_plic;
>      int i, j, base_hartid, hart_count;
> +    CPUState *cs;
>
>      /* Check socket count limit */
>      if (VIRT_SOCKETS_MAX < riscv_socket_count(machine)) {
> @@ -660,6 +662,12 @@ static void virt_machine_init(MachineState *machine)
>                                virt_memmap[VIRT_MROM].size, kernel_entry,
>                                fdt_load_addr, s->fdt);
>
> +    for (cs = first_cpu; cs; cs = CPU_NEXT(cs)) {
> +        RISCVCPU *riscv_cpu = RISCV_CPU(cs);
> +        riscv_cpu->env.kernel_addr = kernel_entry;
> +        riscv_cpu->env.fdt_addr = fdt_load_addr;
> +    }
> +
>      /* SiFive Test MMIO device */
>      sifive_test_create(memmap[VIRT_TEST].base);
>
> diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c
> index 6a0264fc6b..faee98a58c 100644
> --- a/target/riscv/cpu.c
> +++ b/target/riscv/cpu.c
> @@ -29,6 +29,7 @@
>  #include "hw/qdev-properties.h"
>  #include "migration/vmstate.h"
>  #include "fpu/softfloat-helpers.h"
> +#include "kvm_riscv.h"
>
>  /* RISC-V CPU definitions */
>
> @@ -330,6 +331,9 @@ static void riscv_cpu_reset(DeviceState *dev)
>      cs->exception_index = EXCP_NONE;
>      env->load_res = -1;
>      set_default_nan_mode(1, &env->fp_status);
> +#ifdef CONFIG_KVM
> +    kvm_riscv_reset_vcpu(cpu);
> +#endif
>  }
>
>  static void riscv_cpu_disas_set_info(CPUState *s, disassemble_info *info)
> diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h
> index c0a326c843..ad1c90f798 100644
> --- a/target/riscv/cpu.h
> +++ b/target/riscv/cpu.h
> @@ -233,6 +233,9 @@ struct CPURISCVState {
>
>      /* Fields from here on are preserved across CPU reset. */
>      QEMUTimer *timer; /* Internal timer */
> +
> +    hwaddr kernel_addr;
> +    hwaddr fdt_addr;
>  };
>
>  OBJECT_DECLARE_TYPE(RISCVCPU, RISCVCPUClass,
> diff --git a/target/riscv/kvm.c b/target/riscv/kvm.c
> index 8b206ce99c..6250ca0c7d 100644
> --- a/target/riscv/kvm.c
> +++ b/target/riscv/kvm.c
> @@ -37,6 +37,7 @@
>  #include "hw/irq.h"
>  #include "qemu/log.h"
>  #include "hw/loader.h"
> +#include "kvm_riscv.h"
>
>  static __u64 kvm_riscv_reg_id(__u64 type, __u64 idx)
>  {
> @@ -439,3 +440,17 @@ int kvm_arch_handle_exit(CPUState *cs, struct kvm_run *run)
>  {
>      return 0;
>  }
> +
> +void kvm_riscv_reset_vcpu(RISCVCPU *cpu)
> +{
> +    CPURISCVState *env = &cpu->env;
> +
> +    if (!kvm_enabled()) {
> +        return;
> +    }
> +    env->pc = cpu->env.kernel_addr;
> +    env->gpr[10] = kvm_arch_vcpu_id(CPU(cpu)); /* a0 */
> +    env->gpr[11] = cpu->env.fdt_addr;          /* a1 */
> +    env->satp = 0;
> +}
> +
> diff --git a/target/riscv/kvm_riscv.h b/target/riscv/kvm_riscv.h
> new file mode 100644
> index 0000000000..f38c82bf59
> --- /dev/null
> +++ b/target/riscv/kvm_riscv.h
> @@ -0,0 +1,24 @@
> +/*
> + * QEMU KVM support -- RISC-V specific functions.
> + *
> + * Copyright (c) 2020 Huawei Technologies Co., Ltd
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms and conditions of the GNU General Public License,
> + * version 2 or later, as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope it will be useful, but WITHOUT
> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
> + * more details.
> + *
> + * You should have received a copy of the GNU General Public License along with
> + * this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#ifndef QEMU_KVM_RISCV_H
> +#define QEMU_KVM_RISCV_H
> +
> +void kvm_riscv_reset_vcpu(RISCVCPU *cpu);
> +
> +#endif
> --
> 2.19.1
>
>

RE: [PATCH RFC v4 06/15] target/riscv: Support start kernel directly by KVM

[Jiangyifei]

> -----Original Message-----
> From: Alistair Francis [mailto:alistair23@gmail.com]
> Sent: Wednesday, December 9, 2020 6:19 AM
> To: Jiangyifei <jiangyifei@huawei.com>
> Cc: qemu-devel@nongnu.org Developers <qemu-devel@nongnu.org>; open
> list:RISC-V <qemu-riscv@nongnu.org>; Zhangxiaofeng (F)
> <victor.zhangxiaofeng@huawei.com>; Sagar Karandikar
> <sagark@eecs.berkeley.edu>; open list:Overall <kvm@vger.kernel.org>;
> libvir-list@redhat.com; Bastian Koppelmann
> <kbastian@mail.uni-paderborn.de>; Anup Patel <anup.patel@wdc.com>;
> yinyipeng <yinyipeng1@huawei.com>; Alistair Francis
> <Alistair.Francis@wdc.com>; kvm-riscv@lists.infradead.org; Palmer Dabbelt
> <palmer@dabbelt.com>; dengkai (A) <dengkai1@huawei.com>; Wubin (H)
> <wu.wubin@huawei.com>; Zhanghailiang <zhang.zhanghailiang@huawei.com>
> Subject: Re: [PATCH RFC v4 06/15] target/riscv: Support start kernel directly by
> KVM
> 
> On Thu, Dec 3, 2020 at 4:58 AM Yifei Jiang <jiangyifei@huawei.com> wrote:
> >
> > Get kernel and fdt start address in virt.c, and pass them to KVM when
> > cpu reset. In addition, add kvm_riscv.h to place riscv specific
> > interface.
> 
> This doesn't seem right. Why do we need to do this? Other architectures don't
> seem to do this.
> 
> Writing to the CPU from the board like this looks fishy and probably breaks
> some QOM rules.
> 
> Alistair
> 

Sorry for the delayed reply.

We need to set the starting address of the kernel and fdt to vcpu, which is implemented
by firmware bootloader under other architectures and RISC-V emulators. However, the
RISC-V virtual machine does not have bootloader, so we boot the kernel directly.
In the future, we will support firmware loading.

Before supporting the firmware bootloader, we can add a public interface instead of
modifying the CPU instance directly to comply with the QOM rules.

Yifei

> >
> > Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
> > Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
> > ---
> >  hw/riscv/virt.c          |  8 ++++++++
> >  target/riscv/cpu.c       |  4 ++++
> >  target/riscv/cpu.h       |  3 +++
> >  target/riscv/kvm.c       | 15 +++++++++++++++
> >  target/riscv/kvm_riscv.h | 24 ++++++++++++++++++++++++
> >  5 files changed, 54 insertions(+)
> >  create mode 100644 target/riscv/kvm_riscv.h
> >
> > diff --git a/hw/riscv/virt.c b/hw/riscv/virt.c index
> > 25cea7aa67..47b7018193 100644
> > --- a/hw/riscv/virt.c
> > +++ b/hw/riscv/virt.c
> > @@ -42,6 +42,7 @@
> >  #include "sysemu/sysemu.h"
> >  #include "hw/pci/pci.h"
> >  #include "hw/pci-host/gpex.h"
> > +#include "sysemu/kvm.h"
> >
> >  #if defined(TARGET_RISCV32)
> >  # define BIOS_FILENAME "opensbi-riscv32-generic-fw_dynamic.bin"
> > @@ -511,6 +512,7 @@ static void virt_machine_init(MachineState
> *machine)
> >      uint64_t kernel_entry;
> >      DeviceState *mmio_plic, *virtio_plic, *pcie_plic;
> >      int i, j, base_hartid, hart_count;
> > +    CPUState *cs;
> >
> >      /* Check socket count limit */
> >      if (VIRT_SOCKETS_MAX < riscv_socket_count(machine)) { @@ -660,6
> > +662,12 @@ static void virt_machine_init(MachineState *machine)
> >                                virt_memmap[VIRT_MROM].size,
> kernel_entry,
> >                                fdt_load_addr, s->fdt);
> >
> > +    for (cs = first_cpu; cs; cs = CPU_NEXT(cs)) {
> > +        RISCVCPU *riscv_cpu = RISCV_CPU(cs);
> > +        riscv_cpu->env.kernel_addr = kernel_entry;
> > +        riscv_cpu->env.fdt_addr = fdt_load_addr;
> > +    }
> > +
> >      /* SiFive Test MMIO device */
> >      sifive_test_create(memmap[VIRT_TEST].base);
> >
> > diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c index
> > 6a0264fc6b..faee98a58c 100644
> > --- a/target/riscv/cpu.c
> > +++ b/target/riscv/cpu.c
> > @@ -29,6 +29,7 @@
> >  #include "hw/qdev-properties.h"
> >  #include "migration/vmstate.h"
> >  #include "fpu/softfloat-helpers.h"
> > +#include "kvm_riscv.h"
> >
> >  /* RISC-V CPU definitions */
> >
> > @@ -330,6 +331,9 @@ static void riscv_cpu_reset(DeviceState *dev)
> >      cs->exception_index = EXCP_NONE;
> >      env->load_res = -1;
> >      set_default_nan_mode(1, &env->fp_status);
> > +#ifdef CONFIG_KVM
> > +    kvm_riscv_reset_vcpu(cpu);
> > +#endif
> >  }
> >
> >  static void riscv_cpu_disas_set_info(CPUState *s, disassemble_info
> > *info) diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h index
> > c0a326c843..ad1c90f798 100644
> > --- a/target/riscv/cpu.h
> > +++ b/target/riscv/cpu.h
> > @@ -233,6 +233,9 @@ struct CPURISCVState {
> >
> >      /* Fields from here on are preserved across CPU reset. */
> >      QEMUTimer *timer; /* Internal timer */
> > +
> > +    hwaddr kernel_addr;
> > +    hwaddr fdt_addr;
> >  };
> >
> >  OBJECT_DECLARE_TYPE(RISCVCPU, RISCVCPUClass, diff --git
> > a/target/riscv/kvm.c b/target/riscv/kvm.c index 8b206ce99c..6250ca0c7d
> > 100644
> > --- a/target/riscv/kvm.c
> > +++ b/target/riscv/kvm.c
> > @@ -37,6 +37,7 @@
> >  #include "hw/irq.h"
> >  #include "qemu/log.h"
> >  #include "hw/loader.h"
> > +#include "kvm_riscv.h"
> >
> >  static __u64 kvm_riscv_reg_id(__u64 type, __u64 idx)  { @@ -439,3
> > +440,17 @@ int kvm_arch_handle_exit(CPUState *cs, struct kvm_run *run)
> > {
> >      return 0;
> >  }
> > +
> > +void kvm_riscv_reset_vcpu(RISCVCPU *cpu) {
> > +    CPURISCVState *env = &cpu->env;
> > +
> > +    if (!kvm_enabled()) {
> > +        return;
> > +    }
> > +    env->pc = cpu->env.kernel_addr;
> > +    env->gpr[10] = kvm_arch_vcpu_id(CPU(cpu)); /* a0 */
> > +    env->gpr[11] = cpu->env.fdt_addr;          /* a1 */
> > +    env->satp = 0;
> > +}
> > +
> > diff --git a/target/riscv/kvm_riscv.h b/target/riscv/kvm_riscv.h new
> > file mode 100644 index 0000000000..f38c82bf59
> > --- /dev/null
> > +++ b/target/riscv/kvm_riscv.h
> > @@ -0,0 +1,24 @@
> > +/*
> > + * QEMU KVM support -- RISC-V specific functions.
> > + *
> > + * Copyright (c) 2020 Huawei Technologies Co., Ltd
> > + *
> > + * This program is free software; you can redistribute it and/or
> > +modify it
> > + * under the terms and conditions of the GNU General Public License,
> > + * version 2 or later, as published by the Free Software Foundation.
> > + *
> > + * This program is distributed in the hope it will be useful, but
> > +WITHOUT
> > + * ANY WARRANTY; without even the implied warranty of
> MERCHANTABILITY
> > +or
> > + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public
> > +License for
> > + * more details.
> > + *
> > + * You should have received a copy of the GNU General Public License
> > +along with
> > + * this program.  If not, see <http://www.gnu.org/licenses/>.
> > + */
> > +
> > +#ifndef QEMU_KVM_RISCV_H
> > +#define QEMU_KVM_RISCV_H
> > +
> > +void kvm_riscv_reset_vcpu(RISCVCPU *cpu);
> > +
> > +#endif
> > --
> > 2.19.1
> >
> >

[PATCH RFC v4 07/15] hw/riscv: PLIC update external interrupt by KVM when kvm enabled

[Yifei Jiang]

Only support supervisor external interrupt currently.

Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
---
 hw/intc/sifive_plic.c    | 31 ++++++++++++++++++++++---------
 target/riscv/kvm.c       | 19 +++++++++++++++++++
 target/riscv/kvm_riscv.h |  1 +
 3 files changed, 42 insertions(+), 9 deletions(-)

diff --git a/hw/intc/sifive_plic.c b/hw/intc/sifive_plic.c
index 97a1a27a9a..a419ca3a3c 100644
--- a/hw/intc/sifive_plic.c
+++ b/hw/intc/sifive_plic.c
@@ -31,6 +31,8 @@
 #include "target/riscv/cpu.h"
 #include "sysemu/sysemu.h"
 #include "migration/vmstate.h"
+#include "sysemu/kvm.h"
+#include "kvm_riscv.h"
 
 #define RISCV_DEBUG_PLIC 0
 
@@ -147,15 +149,26 @@ static void sifive_plic_update(SiFivePLICState *plic)
             continue;
         }
         int level = sifive_plic_irqs_pending(plic, addrid);
-        switch (mode) {
-        case PLICMode_M:
-            riscv_cpu_update_mip(RISCV_CPU(cpu), MIP_MEIP, BOOL_TO_MASK(level));
-            break;
-        case PLICMode_S:
-            riscv_cpu_update_mip(RISCV_CPU(cpu), MIP_SEIP, BOOL_TO_MASK(level));
-            break;
-        default:
-            break;
+        if (kvm_enabled()) {
+            if (mode == PLICMode_M) {
+                continue;
+            }
+#ifdef CONFIG_KVM
+            kvm_riscv_set_irq(RISCV_CPU(cpu), IRQ_S_EXT, level);
+#endif
+        } else {
+            switch (mode) {
+            case PLICMode_M:
+                riscv_cpu_update_mip(RISCV_CPU(cpu),
+                                     MIP_MEIP, BOOL_TO_MASK(level));
+                break;
+            case PLICMode_S:
+                riscv_cpu_update_mip(RISCV_CPU(cpu),
+                                     MIP_SEIP, BOOL_TO_MASK(level));
+                break;
+            default:
+                break;
+            }
         }
     }
 
diff --git a/target/riscv/kvm.c b/target/riscv/kvm.c
index 6250ca0c7d..b01ff0754c 100644
--- a/target/riscv/kvm.c
+++ b/target/riscv/kvm.c
@@ -454,3 +454,22 @@ void kvm_riscv_reset_vcpu(RISCVCPU *cpu)
     env->satp = 0;
 }
 
+void kvm_riscv_set_irq(RISCVCPU *cpu, int irq, int level)
+{
+    int ret;
+    unsigned virq = level ? KVM_INTERRUPT_SET : KVM_INTERRUPT_UNSET;
+
+    if (irq != IRQ_S_EXT) {
+        return;
+    }
+
+    if (!kvm_enabled()) {
+        return;
+    }
+
+    ret = kvm_vcpu_ioctl(CPU(cpu), KVM_INTERRUPT, &virq);
+    if (ret < 0) {
+        perror("Set irq failed");
+        abort();
+    }
+}
diff --git a/target/riscv/kvm_riscv.h b/target/riscv/kvm_riscv.h
index f38c82bf59..ed281bdce0 100644
--- a/target/riscv/kvm_riscv.h
+++ b/target/riscv/kvm_riscv.h
@@ -20,5 +20,6 @@
 #define QEMU_KVM_RISCV_H
 
 void kvm_riscv_reset_vcpu(RISCVCPU *cpu);
+void kvm_riscv_set_irq(RISCVCPU *cpu, int irq, int level);
 
 #endif
-- 
2.19.1

Re: [PATCH RFC v4 07/15] hw/riscv: PLIC update external interrupt by KVM when kvm enabled

[Alistair Francis]

On Thu, Dec 3, 2020 at 4:47 AM Yifei Jiang <jiangyifei@huawei.com> wrote:
>
> Only support supervisor external interrupt currently.
>
> Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
> Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
> ---
>  hw/intc/sifive_plic.c    | 31 ++++++++++++++++++++++---------
>  target/riscv/kvm.c       | 19 +++++++++++++++++++
>  target/riscv/kvm_riscv.h |  1 +
>  3 files changed, 42 insertions(+), 9 deletions(-)
>
> diff --git a/hw/intc/sifive_plic.c b/hw/intc/sifive_plic.c
> index 97a1a27a9a..a419ca3a3c 100644
> --- a/hw/intc/sifive_plic.c
> +++ b/hw/intc/sifive_plic.c
> @@ -31,6 +31,8 @@
>  #include "target/riscv/cpu.h"
>  #include "sysemu/sysemu.h"
>  #include "migration/vmstate.h"
> +#include "sysemu/kvm.h"
> +#include "kvm_riscv.h"
>
>  #define RISCV_DEBUG_PLIC 0
>
> @@ -147,15 +149,26 @@ static void sifive_plic_update(SiFivePLICState *plic)
>              continue;
>          }
>          int level = sifive_plic_irqs_pending(plic, addrid);
> -        switch (mode) {
> -        case PLICMode_M:
> -            riscv_cpu_update_mip(RISCV_CPU(cpu), MIP_MEIP, BOOL_TO_MASK(level));
> -            break;
> -        case PLICMode_S:
> -            riscv_cpu_update_mip(RISCV_CPU(cpu), MIP_SEIP, BOOL_TO_MASK(level));
> -            break;
> -        default:
> -            break;
> +        if (kvm_enabled()) {
> +            if (mode == PLICMode_M) {
> +                continue;
> +            }
> +#ifdef CONFIG_KVM
> +            kvm_riscv_set_irq(RISCV_CPU(cpu), IRQ_S_EXT, level);
> +#endif

What if kvm_enalbed() is true, but CONFIG_KVM isn't defined?

Alistair

> +        } else {
> +            switch (mode) {
> +            case PLICMode_M:
> +                riscv_cpu_update_mip(RISCV_CPU(cpu),
> +                                     MIP_MEIP, BOOL_TO_MASK(level));
> +                break;
> +            case PLICMode_S:
> +                riscv_cpu_update_mip(RISCV_CPU(cpu),
> +                                     MIP_SEIP, BOOL_TO_MASK(level));
> +                break;
> +            default:
> +                break;
> +            }
>          }
>      }
>
> diff --git a/target/riscv/kvm.c b/target/riscv/kvm.c
> index 6250ca0c7d..b01ff0754c 100644
> --- a/target/riscv/kvm.c
> +++ b/target/riscv/kvm.c
> @@ -454,3 +454,22 @@ void kvm_riscv_reset_vcpu(RISCVCPU *cpu)
>      env->satp = 0;
>  }
>
> +void kvm_riscv_set_irq(RISCVCPU *cpu, int irq, int level)
> +{
> +    int ret;
> +    unsigned virq = level ? KVM_INTERRUPT_SET : KVM_INTERRUPT_UNSET;
> +
> +    if (irq != IRQ_S_EXT) {
> +        return;
> +    }
> +
> +    if (!kvm_enabled()) {
> +        return;
> +    }
> +
> +    ret = kvm_vcpu_ioctl(CPU(cpu), KVM_INTERRUPT, &virq);
> +    if (ret < 0) {
> +        perror("Set irq failed");
> +        abort();
> +    }
> +}
> diff --git a/target/riscv/kvm_riscv.h b/target/riscv/kvm_riscv.h
> index f38c82bf59..ed281bdce0 100644
> --- a/target/riscv/kvm_riscv.h
> +++ b/target/riscv/kvm_riscv.h
> @@ -20,5 +20,6 @@
>  #define QEMU_KVM_RISCV_H
>
>  void kvm_riscv_reset_vcpu(RISCVCPU *cpu);
> +void kvm_riscv_set_irq(RISCVCPU *cpu, int irq, int level);
>
>  #endif
> --
> 2.19.1
>
>

RE: [PATCH RFC v4 07/15] hw/riscv: PLIC update external interrupt by KVM when kvm enabled

[Jiangyifei]

> -----Original Message-----
> From: Alistair Francis [mailto:alistair23@gmail.com]
> Sent: Wednesday, December 9, 2020 6:30 AM
> To: Jiangyifei <jiangyifei@huawei.com>
> Cc: qemu-devel@nongnu.org Developers <qemu-devel@nongnu.org>; open
> list:RISC-V <qemu-riscv@nongnu.org>; Zhangxiaofeng (F)
> <victor.zhangxiaofeng@huawei.com>; Sagar Karandikar
> <sagark@eecs.berkeley.edu>; open list:Overall <kvm@vger.kernel.org>;
> libvir-list@redhat.com; Bastian Koppelmann
> <kbastian@mail.uni-paderborn.de>; Anup Patel <anup.patel@wdc.com>;
> yinyipeng <yinyipeng1@huawei.com>; Alistair Francis
> <Alistair.Francis@wdc.com>; kvm-riscv@lists.infradead.org; Palmer Dabbelt
> <palmer@dabbelt.com>; dengkai (A) <dengkai1@huawei.com>; Wubin (H)
> <wu.wubin@huawei.com>; Zhanghailiang <zhang.zhanghailiang@huawei.com>
> Subject: Re: [PATCH RFC v4 07/15] hw/riscv: PLIC update external interrupt by
> KVM when kvm enabled
> 
> On Thu, Dec 3, 2020 at 4:47 AM Yifei Jiang <jiangyifei@huawei.com> wrote:
> >
> > Only support supervisor external interrupt currently.
> >
> > Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
> > Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
> > ---
> >  hw/intc/sifive_plic.c    | 31 ++++++++++++++++++++++---------
> >  target/riscv/kvm.c       | 19 +++++++++++++++++++
> >  target/riscv/kvm_riscv.h |  1 +
> >  3 files changed, 42 insertions(+), 9 deletions(-)
> >
> > diff --git a/hw/intc/sifive_plic.c b/hw/intc/sifive_plic.c index
> > 97a1a27a9a..a419ca3a3c 100644
> > --- a/hw/intc/sifive_plic.c
> > +++ b/hw/intc/sifive_plic.c
> > @@ -31,6 +31,8 @@
> >  #include "target/riscv/cpu.h"
> >  #include "sysemu/sysemu.h"
> >  #include "migration/vmstate.h"
> > +#include "sysemu/kvm.h"
> > +#include "kvm_riscv.h"
> >
> >  #define RISCV_DEBUG_PLIC 0
> >
> > @@ -147,15 +149,26 @@ static void sifive_plic_update(SiFivePLICState *plic)
> >              continue;
> >          }
> >          int level = sifive_plic_irqs_pending(plic, addrid);
> > -        switch (mode) {
> > -        case PLICMode_M:
> > -            riscv_cpu_update_mip(RISCV_CPU(cpu), MIP_MEIP,
> BOOL_TO_MASK(level));
> > -            break;
> > -        case PLICMode_S:
> > -            riscv_cpu_update_mip(RISCV_CPU(cpu), MIP_SEIP,
> BOOL_TO_MASK(level));
> > -            break;
> > -        default:
> > -            break;
> > +        if (kvm_enabled()) {
> > +            if (mode == PLICMode_M) {
> > +                continue;
> > +            }
> > +#ifdef CONFIG_KVM
> > +            kvm_riscv_set_irq(RISCV_CPU(cpu), IRQ_S_EXT, level);
> > +#endif
> 
> What if kvm_enalbed() is true, but CONFIG_KVM isn't defined?
> 
> Alistair
> 

Impossible. It will cause compilation failure without CONFIG_KVM. We also
introduce kvm-stub.c to solve the compilation failure like other architectures.

We will introduce kvm-stub.c in next series.

Yifei

> > +        } else {
> > +            switch (mode) {
> > +            case PLICMode_M:
> > +                riscv_cpu_update_mip(RISCV_CPU(cpu),
> > +                                     MIP_MEIP,
> BOOL_TO_MASK(level));
> > +                break;
> > +            case PLICMode_S:
> > +                riscv_cpu_update_mip(RISCV_CPU(cpu),
> > +                                     MIP_SEIP,
> BOOL_TO_MASK(level));
> > +                break;
> > +            default:
> > +                break;
> > +            }
> >          }
> >      }
> >
> > diff --git a/target/riscv/kvm.c b/target/riscv/kvm.c index
> > 6250ca0c7d..b01ff0754c 100644
> > --- a/target/riscv/kvm.c
> > +++ b/target/riscv/kvm.c
> > @@ -454,3 +454,22 @@ void kvm_riscv_reset_vcpu(RISCVCPU *cpu)
> >      env->satp = 0;
> >  }
> >
> > +void kvm_riscv_set_irq(RISCVCPU *cpu, int irq, int level) {
> > +    int ret;
> > +    unsigned virq = level ? KVM_INTERRUPT_SET :
> KVM_INTERRUPT_UNSET;
> > +
> > +    if (irq != IRQ_S_EXT) {
> > +        return;
> > +    }
> > +
> > +    if (!kvm_enabled()) {
> > +        return;
> > +    }
> > +
> > +    ret = kvm_vcpu_ioctl(CPU(cpu), KVM_INTERRUPT, &virq);
> > +    if (ret < 0) {
> > +        perror("Set irq failed");
> > +        abort();
> > +    }
> > +}
> > diff --git a/target/riscv/kvm_riscv.h b/target/riscv/kvm_riscv.h index
> > f38c82bf59..ed281bdce0 100644
> > --- a/target/riscv/kvm_riscv.h
> > +++ b/target/riscv/kvm_riscv.h
> > @@ -20,5 +20,6 @@
> >  #define QEMU_KVM_RISCV_H
> >
> >  void kvm_riscv_reset_vcpu(RISCVCPU *cpu);
> > +void kvm_riscv_set_irq(RISCVCPU *cpu, int irq, int level);
> >
> >  #endif
> > --
> > 2.19.1
> >
> >

[PATCH RFC v4 08/15] target/riscv: Handle KVM_EXIT_RISCV_SBI exit

[Yifei Jiang]

Use char-fe to handle console sbi call, which implement early
console io while apply 'earlycon=sbi' into kernel parameters.

Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
---
 target/riscv/kvm.c                 | 42 ++++++++++++++++-
 target/riscv/sbi_ecall_interface.h | 72 ++++++++++++++++++++++++++++++
 2 files changed, 113 insertions(+), 1 deletion(-)
 create mode 100644 target/riscv/sbi_ecall_interface.h

diff --git a/target/riscv/kvm.c b/target/riscv/kvm.c
index b01ff0754c..afd99b3315 100644
--- a/target/riscv/kvm.c
+++ b/target/riscv/kvm.c
@@ -38,6 +38,8 @@
 #include "qemu/log.h"
 #include "hw/loader.h"
 #include "kvm_riscv.h"
+#include "sbi_ecall_interface.h"
+#include "chardev/char-fe.h"
 
 static __u64 kvm_riscv_reg_id(__u64 type, __u64 idx)
 {
@@ -436,9 +438,47 @@ bool kvm_arch_stop_on_emulation_error(CPUState *cs)
     return true;
 }
 
+static int kvm_riscv_handle_sbi(struct kvm_run *run)
+{
+    int ret = 0;
+    unsigned char ch;
+    switch (run->riscv_sbi.extension_id) {
+    case SBI_EXT_0_1_CONSOLE_PUTCHAR:
+        ch = run->riscv_sbi.args[0];
+        qemu_chr_fe_write(serial_hd(0)->be, &ch, sizeof(ch));
+        break;
+    case SBI_EXT_0_1_CONSOLE_GETCHAR:
+        ret = qemu_chr_fe_read_all(serial_hd(0)->be, &ch, sizeof(ch));
+        if (ret == sizeof(ch)) {
+            run->riscv_sbi.args[0] = ch;
+        } else {
+            run->riscv_sbi.args[0] = -1;
+        }
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP,
+                      "%s: un-handled SBI EXIT, specific reasons is %lu\n",
+                      __func__, run->riscv_sbi.extension_id);
+        ret = -1;
+        break;
+    }
+    return ret;
+}
+
 int kvm_arch_handle_exit(CPUState *cs, struct kvm_run *run)
 {
-    return 0;
+    int ret = 0;
+    switch (run->exit_reason) {
+    case KVM_EXIT_RISCV_SBI:
+        ret = kvm_riscv_handle_sbi(run);
+        break;
+    default:
+        qemu_log_mask(LOG_UNIMP, "%s: un-handled exit reason %d\n",
+                      __func__, run->exit_reason);
+        ret = -1;
+        break;
+    }
+    return ret;
 }
 
 void kvm_riscv_reset_vcpu(RISCVCPU *cpu)
diff --git a/target/riscv/sbi_ecall_interface.h b/target/riscv/sbi_ecall_interface.h
new file mode 100644
index 0000000000..fb1a3fa8f2
--- /dev/null
+++ b/target/riscv/sbi_ecall_interface.h
@@ -0,0 +1,72 @@
+/*
+ * SPDX-License-Identifier: BSD-2-Clause
+ *
+ * Copyright (c) 2019 Western Digital Corporation or its affiliates.
+ *
+ * Authors:
+ *   Anup Patel <anup.patel@wdc.com>
+ */
+
+#ifndef __SBI_ECALL_INTERFACE_H__
+#define __SBI_ECALL_INTERFACE_H__
+
+/* clang-format off */
+
+/* SBI Extension IDs */
+#define SBI_EXT_0_1_SET_TIMER           0x0
+#define SBI_EXT_0_1_CONSOLE_PUTCHAR     0x1
+#define SBI_EXT_0_1_CONSOLE_GETCHAR     0x2
+#define SBI_EXT_0_1_CLEAR_IPI           0x3
+#define SBI_EXT_0_1_SEND_IPI            0x4
+#define SBI_EXT_0_1_REMOTE_FENCE_I      0x5
+#define SBI_EXT_0_1_REMOTE_SFENCE_VMA   0x6
+#define SBI_EXT_0_1_REMOTE_SFENCE_VMA_ASID 0x7
+#define SBI_EXT_0_1_SHUTDOWN            0x8
+#define SBI_EXT_BASE                    0x10
+#define SBI_EXT_TIME                    0x54494D45
+#define SBI_EXT_IPI                     0x735049
+#define SBI_EXT_RFENCE                  0x52464E43
+#define SBI_EXT_HSM                     0x48534D
+
+/* SBI function IDs for BASE extension*/
+#define SBI_EXT_BASE_GET_SPEC_VERSION   0x0
+#define SBI_EXT_BASE_GET_IMP_ID         0x1
+#define SBI_EXT_BASE_GET_IMP_VERSION    0x2
+#define SBI_EXT_BASE_PROBE_EXT          0x3
+#define SBI_EXT_BASE_GET_MVENDORID      0x4
+#define SBI_EXT_BASE_GET_MARCHID        0x5
+#define SBI_EXT_BASE_GET_MIMPID         0x6
+
+/* SBI function IDs for TIME extension*/
+#define SBI_EXT_TIME_SET_TIMER          0x0
+
+/* SBI function IDs for IPI extension*/
+#define SBI_EXT_IPI_SEND_IPI            0x0
+
+/* SBI function IDs for RFENCE extension*/
+#define SBI_EXT_RFENCE_REMOTE_FENCE_I       0x0
+#define SBI_EXT_RFENCE_REMOTE_SFENCE_VMA    0x1
+#define SBI_EXT_RFENCE_REMOTE_SFENCE_VMA_ASID  0x2
+#define SBI_EXT_RFENCE_REMOTE_HFENCE_GVMA   0x3
+#define SBI_EXT_RFENCE_REMOTE_HFENCE_GVMA_VMID 0x4
+#define SBI_EXT_RFENCE_REMOTE_HFENCE_VVMA   0x5
+#define SBI_EXT_RFENCE_REMOTE_HFENCE_VVMA_ASID 0x6
+
+/* SBI function IDs for HSM extension */
+#define SBI_EXT_HSM_HART_START          0x0
+#define SBI_EXT_HSM_HART_STOP           0x1
+#define SBI_EXT_HSM_HART_GET_STATUS     0x2
+
+#define SBI_HSM_HART_STATUS_STARTED     0x0
+#define SBI_HSM_HART_STATUS_STOPPED     0x1
+#define SBI_HSM_HART_STATUS_START_PENDING   0x2
+#define SBI_HSM_HART_STATUS_STOP_PENDING    0x3
+
+#define SBI_SPEC_VERSION_MAJOR_OFFSET   24
+#define SBI_SPEC_VERSION_MAJOR_MASK     0x7f
+#define SBI_SPEC_VERSION_MINOR_MASK     0xffffff
+#define SBI_EXT_VENDOR_START            0x09000000
+#define SBI_EXT_VENDOR_END              0x09FFFFFF
+/* clang-format on */
+
+#endif
-- 
2.19.1

[PATCH RFC v4 09/15] target/riscv: Add host cpu type

[Yifei Jiang]

Currently, host cpu is inherited simply.

Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
---
 target/riscv/cpu.c | 6 ++++++
 target/riscv/cpu.h | 1 +
 2 files changed, 7 insertions(+)

diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c
index faee98a58c..439dc89ee7 100644
--- a/target/riscv/cpu.c
+++ b/target/riscv/cpu.c
@@ -186,6 +186,10 @@ static void rv32_imafcu_nommu_cpu_init(Object *obj)
 
 #endif
 
+static void riscv_host_cpu_init(Object *obj)
+{
+}
+
 static ObjectClass *riscv_cpu_class_by_name(const char *cpu_model)
 {
     ObjectClass *oc;
@@ -641,10 +645,12 @@ static const TypeInfo riscv_cpu_type_infos[] = {
     DEFINE_CPU(TYPE_RISCV_CPU_SIFIVE_E31,       rvxx_sifive_e_cpu_init),
     DEFINE_CPU(TYPE_RISCV_CPU_SIFIVE_E34,       rv32_imafcu_nommu_cpu_init),
     DEFINE_CPU(TYPE_RISCV_CPU_SIFIVE_U34,       rvxx_sifive_u_cpu_init),
+    DEFINE_CPU(TYPE_RISCV_CPU_HOST,             riscv_host_cpu_init),
 #elif defined(TARGET_RISCV64)
     DEFINE_CPU(TYPE_RISCV_CPU_BASE64,           riscv_base_cpu_init),
     DEFINE_CPU(TYPE_RISCV_CPU_SIFIVE_E51,       rvxx_sifive_e_cpu_init),
     DEFINE_CPU(TYPE_RISCV_CPU_SIFIVE_U54,       rvxx_sifive_u_cpu_init),
+    DEFINE_CPU(TYPE_RISCV_CPU_HOST,             riscv_host_cpu_init),
 #endif
 };
 
diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h
index ad1c90f798..4288898019 100644
--- a/target/riscv/cpu.h
+++ b/target/riscv/cpu.h
@@ -43,6 +43,7 @@
 #define TYPE_RISCV_CPU_SIFIVE_E51       RISCV_CPU_TYPE_NAME("sifive-e51")
 #define TYPE_RISCV_CPU_SIFIVE_U34       RISCV_CPU_TYPE_NAME("sifive-u34")
 #define TYPE_RISCV_CPU_SIFIVE_U54       RISCV_CPU_TYPE_NAME("sifive-u54")
+#define TYPE_RISCV_CPU_HOST             RISCV_CPU_TYPE_NAME("host")
 
 #define RV32 ((target_ulong)1 << (TARGET_LONG_BITS - 2))
 #define RV64 ((target_ulong)2 << (TARGET_LONG_BITS - 2))
-- 
2.19.1

Re: [PATCH RFC v4 09/15] target/riscv: Add host cpu type

[Alistair Francis]

On Thu, Dec 3, 2020 at 4:55 AM Yifei Jiang <jiangyifei@huawei.com> wrote:
>
> Currently, host cpu is inherited simply.
>
> Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
> Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
> ---
>  target/riscv/cpu.c | 6 ++++++
>  target/riscv/cpu.h | 1 +
>  2 files changed, 7 insertions(+)
>
> diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c
> index faee98a58c..439dc89ee7 100644
> --- a/target/riscv/cpu.c
> +++ b/target/riscv/cpu.c
> @@ -186,6 +186,10 @@ static void rv32_imafcu_nommu_cpu_init(Object *obj)
>
>  #endif
>
> +static void riscv_host_cpu_init(Object *obj)
> +{
> +}
> +
>  static ObjectClass *riscv_cpu_class_by_name(const char *cpu_model)
>  {
>      ObjectClass *oc;
> @@ -641,10 +645,12 @@ static const TypeInfo riscv_cpu_type_infos[] = {
>      DEFINE_CPU(TYPE_RISCV_CPU_SIFIVE_E31,       rvxx_sifive_e_cpu_init),
>      DEFINE_CPU(TYPE_RISCV_CPU_SIFIVE_E34,       rv32_imafcu_nommu_cpu_init),
>      DEFINE_CPU(TYPE_RISCV_CPU_SIFIVE_U34,       rvxx_sifive_u_cpu_init),
> +    DEFINE_CPU(TYPE_RISCV_CPU_HOST,             riscv_host_cpu_init),
>  #elif defined(TARGET_RISCV64)
>      DEFINE_CPU(TYPE_RISCV_CPU_BASE64,           riscv_base_cpu_init),
>      DEFINE_CPU(TYPE_RISCV_CPU_SIFIVE_E51,       rvxx_sifive_e_cpu_init),
>      DEFINE_CPU(TYPE_RISCV_CPU_SIFIVE_U54,       rvxx_sifive_u_cpu_init),
> +    DEFINE_CPU(TYPE_RISCV_CPU_HOST,             riscv_host_cpu_init),

Shouldn't this only be included if KVM is configured? Also it should
be shared between RV32 and RV64.

Alistair

>  #endif
>  };
>
> diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h
> index ad1c90f798..4288898019 100644
> --- a/target/riscv/cpu.h
> +++ b/target/riscv/cpu.h
> @@ -43,6 +43,7 @@
>  #define TYPE_RISCV_CPU_SIFIVE_E51       RISCV_CPU_TYPE_NAME("sifive-e51")
>  #define TYPE_RISCV_CPU_SIFIVE_U34       RISCV_CPU_TYPE_NAME("sifive-u34")
>  #define TYPE_RISCV_CPU_SIFIVE_U54       RISCV_CPU_TYPE_NAME("sifive-u54")
> +#define TYPE_RISCV_CPU_HOST             RISCV_CPU_TYPE_NAME("host")
>
>  #define RV32 ((target_ulong)1 << (TARGET_LONG_BITS - 2))
>  #define RV64 ((target_ulong)2 << (TARGET_LONG_BITS - 2))
> --
> 2.19.1
>
>

RE: [PATCH RFC v4 09/15] target/riscv: Add host cpu type

[Jiangyifei]

> -----Original Message-----
> From: Alistair Francis [mailto:alistair23@gmail.com]
> Sent: Wednesday, December 9, 2020 6:22 AM
> To: Jiangyifei <jiangyifei@huawei.com>
> Cc: qemu-devel@nongnu.org Developers <qemu-devel@nongnu.org>; open
> list:RISC-V <qemu-riscv@nongnu.org>; Zhangxiaofeng (F)
> <victor.zhangxiaofeng@huawei.com>; Sagar Karandikar
> <sagark@eecs.berkeley.edu>; open list:Overall <kvm@vger.kernel.org>;
> libvir-list@redhat.com; Bastian Koppelmann
> <kbastian@mail.uni-paderborn.de>; Anup Patel <anup.patel@wdc.com>;
> yinyipeng <yinyipeng1@huawei.com>; Alistair Francis
> <Alistair.Francis@wdc.com>; kvm-riscv@lists.infradead.org; Palmer Dabbelt
> <palmer@dabbelt.com>; dengkai (A) <dengkai1@huawei.com>; Wubin (H)
> <wu.wubin@huawei.com>; Zhanghailiang <zhang.zhanghailiang@huawei.com>
> Subject: Re: [PATCH RFC v4 09/15] target/riscv: Add host cpu type
> 
> On Thu, Dec 3, 2020 at 4:55 AM Yifei Jiang <jiangyifei@huawei.com> wrote:
> >
> > Currently, host cpu is inherited simply.
> >
> > Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
> > Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
> > ---
> >  target/riscv/cpu.c | 6 ++++++
> >  target/riscv/cpu.h | 1 +
> >  2 files changed, 7 insertions(+)
> >
> > diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c index
> > faee98a58c..439dc89ee7 100644
> > --- a/target/riscv/cpu.c
> > +++ b/target/riscv/cpu.c
> > @@ -186,6 +186,10 @@ static void rv32_imafcu_nommu_cpu_init(Object
> > *obj)
> >
> >  #endif
> >
> > +static void riscv_host_cpu_init(Object *obj) { }
> > +
> >  static ObjectClass *riscv_cpu_class_by_name(const char *cpu_model)  {
> >      ObjectClass *oc;
> > @@ -641,10 +645,12 @@ static const TypeInfo riscv_cpu_type_infos[] = {
> >      DEFINE_CPU(TYPE_RISCV_CPU_SIFIVE_E31,
> rvxx_sifive_e_cpu_init),
> >      DEFINE_CPU(TYPE_RISCV_CPU_SIFIVE_E34,
> rv32_imafcu_nommu_cpu_init),
> >      DEFINE_CPU(TYPE_RISCV_CPU_SIFIVE_U34,
> rvxx_sifive_u_cpu_init),
> > +    DEFINE_CPU(TYPE_RISCV_CPU_HOST,
> riscv_host_cpu_init),
> >  #elif defined(TARGET_RISCV64)
> >      DEFINE_CPU(TYPE_RISCV_CPU_BASE64,
> riscv_base_cpu_init),
> >      DEFINE_CPU(TYPE_RISCV_CPU_SIFIVE_E51,
> rvxx_sifive_e_cpu_init),
> >      DEFINE_CPU(TYPE_RISCV_CPU_SIFIVE_U54,
> rvxx_sifive_u_cpu_init),
> > +    DEFINE_CPU(TYPE_RISCV_CPU_HOST,
> riscv_host_cpu_init),
> 
> Shouldn't this only be included if KVM is configured? Also it should be shared
> between RV32 and RV64.
> 
> Alistair
> 

Yes, It should be included by CONFIG_KVM and be shared between RV32 and RV64.

Yifei

> >  #endif
> >  };
> >
> > diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h index
> > ad1c90f798..4288898019 100644
> > --- a/target/riscv/cpu.h
> > +++ b/target/riscv/cpu.h
> > @@ -43,6 +43,7 @@
> >  #define TYPE_RISCV_CPU_SIFIVE_E51
> RISCV_CPU_TYPE_NAME("sifive-e51")
> >  #define TYPE_RISCV_CPU_SIFIVE_U34
> RISCV_CPU_TYPE_NAME("sifive-u34")
> >  #define TYPE_RISCV_CPU_SIFIVE_U54
> RISCV_CPU_TYPE_NAME("sifive-u54")
> > +#define TYPE_RISCV_CPU_HOST
> RISCV_CPU_TYPE_NAME("host")
> >
> >  #define RV32 ((target_ulong)1 << (TARGET_LONG_BITS - 2))  #define
> > RV64 ((target_ulong)2 << (TARGET_LONG_BITS - 2))
> > --
> > 2.19.1
> >
> >

[PATCH RFC v4 10/15] target/riscv: Add kvm_riscv_get/put_regs_timer

[Yifei Jiang]

Add kvm_riscv_get/put_regs_timer to synchronize virtual time context
from KVM.

To set register of RISCV_TIMER_REG(state) will occur a error from KVM
on kvm_timer_state == 0. It's better to adapt in KVM, but it doesn't matter
that adaping in QEMU.

Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
---
 target/riscv/cpu.h |  6 ++++
 target/riscv/kvm.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 78 insertions(+)

diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h
index 4288898019..16d6050ead 100644
--- a/target/riscv/cpu.h
+++ b/target/riscv/cpu.h
@@ -237,6 +237,12 @@ struct CPURISCVState {
 
     hwaddr kernel_addr;
     hwaddr fdt_addr;
+
+    /* kvm timer */
+    bool kvm_timer_dirty;
+    uint64_t kvm_timer_time;
+    uint64_t kvm_timer_compare;
+    uint64_t kvm_timer_state;
 };
 
 OBJECT_DECLARE_TYPE(RISCVCPU, RISCVCPUClass,
diff --git a/target/riscv/kvm.c b/target/riscv/kvm.c
index afd99b3315..79228eb726 100644
--- a/target/riscv/kvm.c
+++ b/target/riscv/kvm.c
@@ -59,6 +59,9 @@ static __u64 kvm_riscv_reg_id(__u64 type, __u64 idx)
 #define RISCV_CSR_REG(name)  kvm_riscv_reg_id(KVM_REG_RISCV_CSR, \
                  KVM_REG_RISCV_CSR_REG(name))
 
+#define RISCV_TIMER_REG(name)  kvm_riscv_reg_id(KVM_REG_RISCV_TIMER, \
+                 KVM_REG_RISCV_TIMER_REG(name))
+
 #define RISCV_FP_F_REG(idx)  kvm_riscv_reg_id(KVM_REG_RISCV_FP_F, idx)
 
 #define RISCV_FP_D_REG(idx)  kvm_riscv_reg_id(KVM_REG_RISCV_FP_D, idx)
@@ -306,6 +309,75 @@ static int kvm_riscv_put_regs_fp(CPUState *cs)
     return ret;
 }
 
+static void kvm_riscv_get_regs_timer(CPUState *cs)
+{
+    int ret;
+    uint64_t reg;
+    CPURISCVState *env = &RISCV_CPU(cs)->env;
+
+    if (env->kvm_timer_dirty) {
+        return;
+    }
+
+    ret = kvm_get_one_reg(cs, RISCV_TIMER_REG(time), &reg);
+    if (ret) {
+        abort();
+    }
+    env->kvm_timer_time = reg;
+
+    ret = kvm_get_one_reg(cs, RISCV_TIMER_REG(compare), &reg);
+    if (ret) {
+        abort();
+    }
+    env->kvm_timer_compare = reg;
+
+    ret = kvm_get_one_reg(cs, RISCV_TIMER_REG(state), &reg);
+    if (ret) {
+        abort();
+    }
+    env->kvm_timer_state = reg;
+
+    env->kvm_timer_dirty = true;
+}
+
+static void kvm_riscv_put_regs_timer(CPUState *cs)
+{
+    int ret;
+    uint64_t reg;
+    CPURISCVState *env = &RISCV_CPU(cs)->env;
+
+    if (!env->kvm_timer_dirty) {
+        return;
+    }
+
+    reg = env->kvm_timer_time;
+    ret = kvm_set_one_reg(cs, RISCV_TIMER_REG(time), &reg);
+    if (ret) {
+        abort();
+    }
+
+    reg = env->kvm_timer_compare;
+    ret = kvm_set_one_reg(cs, RISCV_TIMER_REG(compare), &reg);
+    if (ret) {
+        abort();
+    }
+
+    /*
+     * To set register of RISCV_TIMER_REG(state) will occur a error from KVM
+     * on env->kvm_timer_state == 0, It's better to adapt in KVM, but it
+     * doesn't matter that adaping in QEMU now.
+     * TODO If KVM changes, adapt here.
+     */
+    if (env->kvm_timer_state) {
+        reg = env->kvm_timer_state;
+        ret = kvm_set_one_reg(cs, RISCV_TIMER_REG(state), &reg);
+        if (ret) {
+            abort();
+        }
+    }
+
+    env->kvm_timer_dirty = false;
+}
 
 const KVMCapabilityInfo kvm_arch_required_capabilities[] = {
     KVM_CAP_LAST_INFO
-- 
2.19.1

[PATCH RFC v4 11/15] target/riscv: Implement virtual time adjusting with vm state changing

[Yifei Jiang]

We hope that virtual time adjusts with vm state changing. When a vm
is stopped, guest virtual time should stop counting and kvm_timer
should be stopped. When the vm is resumed, guest virtual time should
continue to count and kvm_timer should be restored.

Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
---
 target/riscv/kvm.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/target/riscv/kvm.c b/target/riscv/kvm.c
index 79228eb726..1e16d24544 100644
--- a/target/riscv/kvm.c
+++ b/target/riscv/kvm.c
@@ -40,6 +40,7 @@
 #include "kvm_riscv.h"
 #include "sbi_ecall_interface.h"
 #include "chardev/char-fe.h"
+#include "sysemu/runstate.h"
 
 static __u64 kvm_riscv_reg_id(__u64 type, __u64 idx)
 {
@@ -448,6 +449,17 @@ unsigned long kvm_arch_vcpu_id(CPUState *cpu)
     return cpu->cpu_index;
 }
 
+static void kvm_riscv_vm_state_change(void *opaque, int running, RunState state)
+{
+    CPUState *cs = opaque;
+
+    if (running) {
+        kvm_riscv_put_regs_timer(cs);
+    } else {
+        kvm_riscv_get_regs_timer(cs);
+    }
+}
+
 void kvm_arch_init_irq_routing(KVMState *s)
 {
 }
@@ -460,6 +472,8 @@ int kvm_arch_init_vcpu(CPUState *cs)
     CPURISCVState *env = &cpu->env;
     __u64 id;
 
+    qemu_add_vm_change_state_handler(kvm_riscv_vm_state_change, cs);
+
     id = kvm_riscv_reg_id(KVM_REG_RISCV_CONFIG, KVM_REG_RISCV_CONFIG_REG(isa));
     ret = kvm_get_one_reg(cs, id, &isa);
     if (ret) {
-- 
2.19.1

[PATCH RFC v4 12/15] target/riscv: Support virtual time context synchronization

[Yifei Jiang]

Add virtual time context description to vmstate_riscv_cpu. After cpu being
loaded, virtual time context is updated to KVM.

Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
---
 target/riscv/machine.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/target/riscv/machine.c b/target/riscv/machine.c
index 44d4015bd6..ef2d5395a8 100644
--- a/target/riscv/machine.c
+++ b/target/riscv/machine.c
@@ -138,10 +138,20 @@ static const VMStateDescription vmstate_hyper = {
     }
 };
 
+static int cpu_post_load(void *opaque, int version_id)
+{
+    RISCVCPU *cpu = opaque;
+    CPURISCVState *env = &cpu->env;
+
+    env->kvm_timer_dirty = true;
+    return 0;
+}
+
 const VMStateDescription vmstate_riscv_cpu = {
     .name = "cpu",
     .version_id = 1,
     .minimum_version_id = 1,
+    .post_load = cpu_post_load,
     .fields = (VMStateField[]) {
         VMSTATE_UINTTL_ARRAY(env.gpr, RISCVCPU, 32),
         VMSTATE_UINT64_ARRAY(env.fpr, RISCVCPU, 32),
@@ -185,6 +195,10 @@ const VMStateDescription vmstate_riscv_cpu = {
         VMSTATE_UINT64(env.mtohost, RISCVCPU),
         VMSTATE_UINT64(env.timecmp, RISCVCPU),
 
+        VMSTATE_UINT64(env.kvm_timer_time, RISCVCPU),
+        VMSTATE_UINT64(env.kvm_timer_compare, RISCVCPU),
+        VMSTATE_UINT64(env.kvm_timer_state, RISCVCPU),
+
         VMSTATE_END_OF_LIST()
     },
     .subsections = (const VMStateDescription * []) {
-- 
2.19.1

[PATCH RFC v4 13/15] target/riscv: Introduce dynamic time frequency for virt machine

[Yifei Jiang]

Currently, time base frequency was fixed as SIFIVE_CLINT_TIMEBASE_FREQ.
Here introduce "time-frequency" property to set time base frequency dynamically
of which default value is still SIFIVE_CLINT_TIMEBASE_FREQ. The virt machine
uses frequency of the first cpu to create clint and fdt.

Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
---
 hw/riscv/virt.c    | 18 ++++++++++++++----
 target/riscv/cpu.c |  3 +++
 target/riscv/cpu.h |  2 ++
 3 files changed, 19 insertions(+), 4 deletions(-)

diff --git a/hw/riscv/virt.c b/hw/riscv/virt.c
index 47b7018193..788a7237b6 100644
--- a/hw/riscv/virt.c
+++ b/hw/riscv/virt.c
@@ -178,7 +178,7 @@ static void create_pcie_irq_map(void *fdt, char *nodename,
 }
 
 static void create_fdt(RISCVVirtState *s, const struct MemmapEntry *memmap,
-    uint64_t mem_size, const char *cmdline)
+    uint64_t mem_size, const char *cmdline, uint64_t timebase_frequency)
 {
     void *fdt;
     int i, cpu, socket;
@@ -225,7 +225,7 @@ static void create_fdt(RISCVVirtState *s, const struct MemmapEntry *memmap,
 
     qemu_fdt_add_subnode(fdt, "/cpus");
     qemu_fdt_setprop_cell(fdt, "/cpus", "timebase-frequency",
-                          SIFIVE_CLINT_TIMEBASE_FREQ);
+                          timebase_frequency);
     qemu_fdt_setprop_cell(fdt, "/cpus", "#size-cells", 0x0);
     qemu_fdt_setprop_cell(fdt, "/cpus", "#address-cells", 0x1);
     qemu_fdt_add_subnode(fdt, "/cpus/cpu-map");
@@ -510,6 +510,7 @@ static void virt_machine_init(MachineState *machine)
     target_ulong firmware_end_addr, kernel_start_addr;
     uint32_t fdt_load_addr;
     uint64_t kernel_entry;
+    uint64_t timebase_frequency = 0;
     DeviceState *mmio_plic, *virtio_plic, *pcie_plic;
     int i, j, base_hartid, hart_count;
     CPUState *cs;
@@ -553,12 +554,20 @@ static void virt_machine_init(MachineState *machine)
                                 hart_count, &error_abort);
         sysbus_realize(SYS_BUS_DEVICE(&s->soc[i]), &error_abort);
 
+        if (!timebase_frequency) {
+            timebase_frequency = RISCV_CPU(first_cpu)->env.frequency;
+        }
+        /* If vcpu's time frequency is not specified, we use default frequency */
+        if (!timebase_frequency) {
+            timebase_frequency = SIFIVE_CLINT_TIMEBASE_FREQ;
+        }
+
         /* Per-socket CLINT */
         sifive_clint_create(
             memmap[VIRT_CLINT].base + i * memmap[VIRT_CLINT].size,
             memmap[VIRT_CLINT].size, base_hartid, hart_count,
             SIFIVE_SIP_BASE, SIFIVE_TIMECMP_BASE, SIFIVE_TIME_BASE,
-            SIFIVE_CLINT_TIMEBASE_FREQ, true);
+            timebase_frequency, true);
 
         /* Per-socket PLIC hart topology configuration string */
         plic_hart_config_len =
@@ -610,7 +619,8 @@ static void virt_machine_init(MachineState *machine)
         main_mem);
 
     /* create device tree */
-    create_fdt(s, memmap, machine->ram_size, machine->kernel_cmdline);
+    create_fdt(s, memmap, machine->ram_size, machine->kernel_cmdline,
+               timebase_frequency);
 
     /* boot rom */
     memory_region_init_rom(mask_rom, NULL, "riscv_virt_board.mrom",
diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c
index 439dc89ee7..66f35bcbbf 100644
--- a/target/riscv/cpu.c
+++ b/target/riscv/cpu.c
@@ -494,6 +494,8 @@ static void riscv_cpu_realize(DeviceState *dev, Error **errp)
 
     riscv_cpu_register_gdb_regs_for_features(cs);
 
+    env->user_frequency = env->frequency;
+
     qemu_init_vcpu(cs);
     cpu_reset(cs);
 
@@ -531,6 +533,7 @@ static Property riscv_cpu_properties[] = {
     DEFINE_PROP_BOOL("mmu", RISCVCPU, cfg.mmu, true),
     DEFINE_PROP_BOOL("pmp", RISCVCPU, cfg.pmp, true),
     DEFINE_PROP_UINT64("resetvec", RISCVCPU, cfg.resetvec, DEFAULT_RSTVEC),
+    DEFINE_PROP_UINT64("time-frequency", RISCVCPU, env.frequency, 0),
     DEFINE_PROP_END_OF_LIST(),
 };
 
diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h
index 16d6050ead..f5b6c34176 100644
--- a/target/riscv/cpu.h
+++ b/target/riscv/cpu.h
@@ -243,6 +243,8 @@ struct CPURISCVState {
     uint64_t kvm_timer_time;
     uint64_t kvm_timer_compare;
     uint64_t kvm_timer_state;
+    uint64_t user_frequency;
+    uint64_t frequency;
 };
 
 OBJECT_DECLARE_TYPE(RISCVCPU, RISCVCPUClass,
-- 
2.19.1

Re: [PATCH RFC v4 13/15] target/riscv: Introduce dynamic time frequency for virt machine

[Alistair Francis]

On Thu, Dec 3, 2020 at 4:57 AM Yifei Jiang <jiangyifei@huawei.com> wrote:
>
> Currently, time base frequency was fixed as SIFIVE_CLINT_TIMEBASE_FREQ.
> Here introduce "time-frequency" property to set time base frequency dynamically
> of which default value is still SIFIVE_CLINT_TIMEBASE_FREQ. The virt machine
> uses frequency of the first cpu to create clint and fdt.
>
> Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
> Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
> ---
>  hw/riscv/virt.c    | 18 ++++++++++++++----
>  target/riscv/cpu.c |  3 +++
>  target/riscv/cpu.h |  2 ++
>  3 files changed, 19 insertions(+), 4 deletions(-)
>
> diff --git a/hw/riscv/virt.c b/hw/riscv/virt.c
> index 47b7018193..788a7237b6 100644
> --- a/hw/riscv/virt.c
> +++ b/hw/riscv/virt.c
> @@ -178,7 +178,7 @@ static void create_pcie_irq_map(void *fdt, char *nodename,
>  }
>
>  static void create_fdt(RISCVVirtState *s, const struct MemmapEntry *memmap,
> -    uint64_t mem_size, const char *cmdline)
> +    uint64_t mem_size, const char *cmdline, uint64_t timebase_frequency)
>  {
>      void *fdt;
>      int i, cpu, socket;
> @@ -225,7 +225,7 @@ static void create_fdt(RISCVVirtState *s, const struct MemmapEntry *memmap,
>
>      qemu_fdt_add_subnode(fdt, "/cpus");
>      qemu_fdt_setprop_cell(fdt, "/cpus", "timebase-frequency",
> -                          SIFIVE_CLINT_TIMEBASE_FREQ);
> +                          timebase_frequency);
>      qemu_fdt_setprop_cell(fdt, "/cpus", "#size-cells", 0x0);
>      qemu_fdt_setprop_cell(fdt, "/cpus", "#address-cells", 0x1);
>      qemu_fdt_add_subnode(fdt, "/cpus/cpu-map");
> @@ -510,6 +510,7 @@ static void virt_machine_init(MachineState *machine)
>      target_ulong firmware_end_addr, kernel_start_addr;
>      uint32_t fdt_load_addr;
>      uint64_t kernel_entry;
> +    uint64_t timebase_frequency = 0;
>      DeviceState *mmio_plic, *virtio_plic, *pcie_plic;
>      int i, j, base_hartid, hart_count;
>      CPUState *cs;
> @@ -553,12 +554,20 @@ static void virt_machine_init(MachineState *machine)
>                                  hart_count, &error_abort);
>          sysbus_realize(SYS_BUS_DEVICE(&s->soc[i]), &error_abort);
>
> +        if (!timebase_frequency) {
> +            timebase_frequency = RISCV_CPU(first_cpu)->env.frequency;
> +        }
> +        /* If vcpu's time frequency is not specified, we use default frequency */
> +        if (!timebase_frequency) {
> +            timebase_frequency = SIFIVE_CLINT_TIMEBASE_FREQ;
> +        }
> +
>          /* Per-socket CLINT */
>          sifive_clint_create(
>              memmap[VIRT_CLINT].base + i * memmap[VIRT_CLINT].size,
>              memmap[VIRT_CLINT].size, base_hartid, hart_count,
>              SIFIVE_SIP_BASE, SIFIVE_TIMECMP_BASE, SIFIVE_TIME_BASE,
> -            SIFIVE_CLINT_TIMEBASE_FREQ, true);
> +            timebase_frequency, true);
>
>          /* Per-socket PLIC hart topology configuration string */
>          plic_hart_config_len =
> @@ -610,7 +619,8 @@ static void virt_machine_init(MachineState *machine)
>          main_mem);
>
>      /* create device tree */
> -    create_fdt(s, memmap, machine->ram_size, machine->kernel_cmdline);
> +    create_fdt(s, memmap, machine->ram_size, machine->kernel_cmdline,
> +               timebase_frequency);
>
>      /* boot rom */
>      memory_region_init_rom(mask_rom, NULL, "riscv_virt_board.mrom",
> diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c
> index 439dc89ee7..66f35bcbbf 100644
> --- a/target/riscv/cpu.c
> +++ b/target/riscv/cpu.c
> @@ -494,6 +494,8 @@ static void riscv_cpu_realize(DeviceState *dev, Error **errp)
>
>      riscv_cpu_register_gdb_regs_for_features(cs);
>
> +    env->user_frequency = env->frequency;
> +
>      qemu_init_vcpu(cs);
>      cpu_reset(cs);
>
> @@ -531,6 +533,7 @@ static Property riscv_cpu_properties[] = {
>      DEFINE_PROP_BOOL("mmu", RISCVCPU, cfg.mmu, true),
>      DEFINE_PROP_BOOL("pmp", RISCVCPU, cfg.pmp, true),
>      DEFINE_PROP_UINT64("resetvec", RISCVCPU, cfg.resetvec, DEFAULT_RSTVEC),
> +    DEFINE_PROP_UINT64("time-frequency", RISCVCPU, env.frequency, 0),

Why not set the default to SIFIVE_CLINT_TIMEBASE_FREQ?

Also, QEMU now has a clock API, is using that instead a better option?

Alistair

>      DEFINE_PROP_END_OF_LIST(),
>  };
>
> diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h
> index 16d6050ead..f5b6c34176 100644
> --- a/target/riscv/cpu.h
> +++ b/target/riscv/cpu.h
> @@ -243,6 +243,8 @@ struct CPURISCVState {
>      uint64_t kvm_timer_time;
>      uint64_t kvm_timer_compare;
>      uint64_t kvm_timer_state;
> +    uint64_t user_frequency;
> +    uint64_t frequency;
>  };
>
>  OBJECT_DECLARE_TYPE(RISCVCPU, RISCVCPUClass,
> --
> 2.19.1
>
>

RE: [PATCH RFC v4 13/15] target/riscv: Introduce dynamic time frequency for virt machine

[Jiangyifei]

> -----Original Message-----
> From: Alistair Francis [mailto:alistair23@gmail.com]
> Sent: Wednesday, December 9, 2020 6:26 AM
> To: Jiangyifei <jiangyifei@huawei.com>
> Cc: qemu-devel@nongnu.org Developers <qemu-devel@nongnu.org>; open
> list:RISC-V <qemu-riscv@nongnu.org>; Zhangxiaofeng (F)
> <victor.zhangxiaofeng@huawei.com>; Sagar Karandikar
> <sagark@eecs.berkeley.edu>; open list:Overall <kvm@vger.kernel.org>;
> libvir-list@redhat.com; Bastian Koppelmann
> <kbastian@mail.uni-paderborn.de>; Anup Patel <anup.patel@wdc.com>;
> yinyipeng <yinyipeng1@huawei.com>; Alistair Francis
> <Alistair.Francis@wdc.com>; kvm-riscv@lists.infradead.org; Palmer Dabbelt
> <palmer@dabbelt.com>; dengkai (A) <dengkai1@huawei.com>; Wubin (H)
> <wu.wubin@huawei.com>; Zhanghailiang <zhang.zhanghailiang@huawei.com>
> Subject: Re: [PATCH RFC v4 13/15] target/riscv: Introduce dynamic time
> frequency for virt machine
> 
> On Thu, Dec 3, 2020 at 4:57 AM Yifei Jiang <jiangyifei@huawei.com> wrote:
> >
> > Currently, time base frequency was fixed as SIFIVE_CLINT_TIMEBASE_FREQ.
> > Here introduce "time-frequency" property to set time base frequency
> > dynamically of which default value is still
> > SIFIVE_CLINT_TIMEBASE_FREQ. The virt machine uses frequency of the first
> cpu to create clint and fdt.
> >
> > Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
> > Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
> > ---
> >  hw/riscv/virt.c    | 18 ++++++++++++++----
> >  target/riscv/cpu.c |  3 +++
> >  target/riscv/cpu.h |  2 ++
> >  3 files changed, 19 insertions(+), 4 deletions(-)
> >
> > diff --git a/hw/riscv/virt.c b/hw/riscv/virt.c index
> > 47b7018193..788a7237b6 100644
> > --- a/hw/riscv/virt.c
> > +++ b/hw/riscv/virt.c
> > @@ -178,7 +178,7 @@ static void create_pcie_irq_map(void *fdt, char
> > *nodename,  }
> >
> >  static void create_fdt(RISCVVirtState *s, const struct MemmapEntry
> *memmap,
> > -    uint64_t mem_size, const char *cmdline)
> > +    uint64_t mem_size, const char *cmdline, uint64_t
> > + timebase_frequency)
> >  {
> >      void *fdt;
> >      int i, cpu, socket;
> > @@ -225,7 +225,7 @@ static void create_fdt(RISCVVirtState *s, const
> > struct MemmapEntry *memmap,
> >
> >      qemu_fdt_add_subnode(fdt, "/cpus");
> >      qemu_fdt_setprop_cell(fdt, "/cpus", "timebase-frequency",
> > -                          SIFIVE_CLINT_TIMEBASE_FREQ);
> > +                          timebase_frequency);
> >      qemu_fdt_setprop_cell(fdt, "/cpus", "#size-cells", 0x0);
> >      qemu_fdt_setprop_cell(fdt, "/cpus", "#address-cells", 0x1);
> >      qemu_fdt_add_subnode(fdt, "/cpus/cpu-map"); @@ -510,6 +510,7
> @@
> > static void virt_machine_init(MachineState *machine)
> >      target_ulong firmware_end_addr, kernel_start_addr;
> >      uint32_t fdt_load_addr;
> >      uint64_t kernel_entry;
> > +    uint64_t timebase_frequency = 0;
> >      DeviceState *mmio_plic, *virtio_plic, *pcie_plic;
> >      int i, j, base_hartid, hart_count;
> >      CPUState *cs;
> > @@ -553,12 +554,20 @@ static void virt_machine_init(MachineState
> *machine)
> >                                  hart_count, &error_abort);
> >          sysbus_realize(SYS_BUS_DEVICE(&s->soc[i]), &error_abort);
> >
> > +        if (!timebase_frequency) {
> > +            timebase_frequency = RISCV_CPU(first_cpu)->env.frequency;
> > +        }
> > +        /* If vcpu's time frequency is not specified, we use default
> frequency */
> > +        if (!timebase_frequency) {
> > +            timebase_frequency = SIFIVE_CLINT_TIMEBASE_FREQ;
> > +        }
> > +
> >          /* Per-socket CLINT */
> >          sifive_clint_create(
> >              memmap[VIRT_CLINT].base + i *
> memmap[VIRT_CLINT].size,
> >              memmap[VIRT_CLINT].size, base_hartid, hart_count,
> >              SIFIVE_SIP_BASE, SIFIVE_TIMECMP_BASE,
> SIFIVE_TIME_BASE,
> > -            SIFIVE_CLINT_TIMEBASE_FREQ, true);
> > +            timebase_frequency, true);
> >
> >          /* Per-socket PLIC hart topology configuration string */
> >          plic_hart_config_len =
> > @@ -610,7 +619,8 @@ static void virt_machine_init(MachineState
> *machine)
> >          main_mem);
> >
> >      /* create device tree */
> > -    create_fdt(s, memmap, machine->ram_size,
> machine->kernel_cmdline);
> > +    create_fdt(s, memmap, machine->ram_size,
> machine->kernel_cmdline,
> > +               timebase_frequency);
> >
> >      /* boot rom */
> >      memory_region_init_rom(mask_rom, NULL, "riscv_virt_board.mrom",
> > diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c index
> > 439dc89ee7..66f35bcbbf 100644
> > --- a/target/riscv/cpu.c
> > +++ b/target/riscv/cpu.c
> > @@ -494,6 +494,8 @@ static void riscv_cpu_realize(DeviceState *dev,
> > Error **errp)
> >
> >      riscv_cpu_register_gdb_regs_for_features(cs);
> >
> > +    env->user_frequency = env->frequency;
> > +
> >      qemu_init_vcpu(cs);
> >      cpu_reset(cs);
> >
> > @@ -531,6 +533,7 @@ static Property riscv_cpu_properties[] = {
> >      DEFINE_PROP_BOOL("mmu", RISCVCPU, cfg.mmu, true),
> >      DEFINE_PROP_BOOL("pmp", RISCVCPU, cfg.pmp, true),
> >      DEFINE_PROP_UINT64("resetvec", RISCVCPU, cfg.resetvec,
> > DEFAULT_RSTVEC),
> > +    DEFINE_PROP_UINT64("time-frequency", RISCVCPU, env.frequency, 0),
> 
> Why not set the default to SIFIVE_CLINT_TIMEBASE_FREQ?
> 

When the time frequency is not specified, it will follow the host or the migration
source. And we define 0 as equivalent to not specified time frequency.

> Also, QEMU now has a clock API, is using that instead a better option?
> 

Sorry, I didn't find the clock API. Could you tell me what the API is.
I think that the time frequency is option of KVM VCPU. So it is appropriate to put this
option in the CPU.

Yifei

> Alistair
> 
> >      DEFINE_PROP_END_OF_LIST(),
> >  };
> >
> > diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h index
> > 16d6050ead..f5b6c34176 100644
> > --- a/target/riscv/cpu.h
> > +++ b/target/riscv/cpu.h
> > @@ -243,6 +243,8 @@ struct CPURISCVState {
> >      uint64_t kvm_timer_time;
> >      uint64_t kvm_timer_compare;
> >      uint64_t kvm_timer_state;
> > +    uint64_t user_frequency;
> > +    uint64_t frequency;
> >  };
> >
> >  OBJECT_DECLARE_TYPE(RISCVCPU, RISCVCPUClass,
> > --
> > 2.19.1
> >
> >

Re: [PATCH RFC v4 13/15] target/riscv: Introduce dynamic time frequency for virt machine

[Alistair Francis]

On Mon, Dec 14, 2020 at 11:31 PM Jiangyifei <jiangyifei@huawei.com> wrote:
>
>
> > -----Original Message-----
> > From: Alistair Francis [mailto:alistair23@gmail.com]
> > Sent: Wednesday, December 9, 2020 6:26 AM
> > To: Jiangyifei <jiangyifei@huawei.com>
> > Cc: qemu-devel@nongnu.org Developers <qemu-devel@nongnu.org>; open
> > list:RISC-V <qemu-riscv@nongnu.org>; Zhangxiaofeng (F)
> > <victor.zhangxiaofeng@huawei.com>; Sagar Karandikar
> > <sagark@eecs.berkeley.edu>; open list:Overall <kvm@vger.kernel.org>;
> > libvir-list@redhat.com; Bastian Koppelmann
> > <kbastian@mail.uni-paderborn.de>; Anup Patel <anup.patel@wdc.com>;
> > yinyipeng <yinyipeng1@huawei.com>; Alistair Francis
> > <Alistair.Francis@wdc.com>; kvm-riscv@lists.infradead.org; Palmer Dabbelt
> > <palmer@dabbelt.com>; dengkai (A) <dengkai1@huawei.com>; Wubin (H)
> > <wu.wubin@huawei.com>; Zhanghailiang <zhang.zhanghailiang@huawei.com>
> > Subject: Re: [PATCH RFC v4 13/15] target/riscv: Introduce dynamic time
> > frequency for virt machine
> >
> > On Thu, Dec 3, 2020 at 4:57 AM Yifei Jiang <jiangyifei@huawei.com> wrote:
> > >
> > > Currently, time base frequency was fixed as SIFIVE_CLINT_TIMEBASE_FREQ.
> > > Here introduce "time-frequency" property to set time base frequency
> > > dynamically of which default value is still
> > > SIFIVE_CLINT_TIMEBASE_FREQ. The virt machine uses frequency of the first
> > cpu to create clint and fdt.
> > >
> > > Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
> > > Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
> > > ---
> > >  hw/riscv/virt.c    | 18 ++++++++++++++----
> > >  target/riscv/cpu.c |  3 +++
> > >  target/riscv/cpu.h |  2 ++
> > >  3 files changed, 19 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/hw/riscv/virt.c b/hw/riscv/virt.c index
> > > 47b7018193..788a7237b6 100644
> > > --- a/hw/riscv/virt.c
> > > +++ b/hw/riscv/virt.c
> > > @@ -178,7 +178,7 @@ static void create_pcie_irq_map(void *fdt, char
> > > *nodename,  }
> > >
> > >  static void create_fdt(RISCVVirtState *s, const struct MemmapEntry
> > *memmap,
> > > -    uint64_t mem_size, const char *cmdline)
> > > +    uint64_t mem_size, const char *cmdline, uint64_t
> > > + timebase_frequency)
> > >  {
> > >      void *fdt;
> > >      int i, cpu, socket;
> > > @@ -225,7 +225,7 @@ static void create_fdt(RISCVVirtState *s, const
> > > struct MemmapEntry *memmap,
> > >
> > >      qemu_fdt_add_subnode(fdt, "/cpus");
> > >      qemu_fdt_setprop_cell(fdt, "/cpus", "timebase-frequency",
> > > -                          SIFIVE_CLINT_TIMEBASE_FREQ);
> > > +                          timebase_frequency);
> > >      qemu_fdt_setprop_cell(fdt, "/cpus", "#size-cells", 0x0);
> > >      qemu_fdt_setprop_cell(fdt, "/cpus", "#address-cells", 0x1);
> > >      qemu_fdt_add_subnode(fdt, "/cpus/cpu-map"); @@ -510,6 +510,7
> > @@
> > > static void virt_machine_init(MachineState *machine)
> > >      target_ulong firmware_end_addr, kernel_start_addr;
> > >      uint32_t fdt_load_addr;
> > >      uint64_t kernel_entry;
> > > +    uint64_t timebase_frequency = 0;
> > >      DeviceState *mmio_plic, *virtio_plic, *pcie_plic;
> > >      int i, j, base_hartid, hart_count;
> > >      CPUState *cs;
> > > @@ -553,12 +554,20 @@ static void virt_machine_init(MachineState
> > *machine)
> > >                                  hart_count, &error_abort);
> > >          sysbus_realize(SYS_BUS_DEVICE(&s->soc[i]), &error_abort);
> > >
> > > +        if (!timebase_frequency) {
> > > +            timebase_frequency = RISCV_CPU(first_cpu)->env.frequency;
> > > +        }
> > > +        /* If vcpu's time frequency is not specified, we use default
> > frequency */
> > > +        if (!timebase_frequency) {
> > > +            timebase_frequency = SIFIVE_CLINT_TIMEBASE_FREQ;
> > > +        }
> > > +
> > >          /* Per-socket CLINT */
> > >          sifive_clint_create(
> > >              memmap[VIRT_CLINT].base + i *
> > memmap[VIRT_CLINT].size,
> > >              memmap[VIRT_CLINT].size, base_hartid, hart_count,
> > >              SIFIVE_SIP_BASE, SIFIVE_TIMECMP_BASE,
> > SIFIVE_TIME_BASE,
> > > -            SIFIVE_CLINT_TIMEBASE_FREQ, true);
> > > +            timebase_frequency, true);
> > >
> > >          /* Per-socket PLIC hart topology configuration string */
> > >          plic_hart_config_len =
> > > @@ -610,7 +619,8 @@ static void virt_machine_init(MachineState
> > *machine)
> > >          main_mem);
> > >
> > >      /* create device tree */
> > > -    create_fdt(s, memmap, machine->ram_size,
> > machine->kernel_cmdline);
> > > +    create_fdt(s, memmap, machine->ram_size,
> > machine->kernel_cmdline,
> > > +               timebase_frequency);
> > >
> > >      /* boot rom */
> > >      memory_region_init_rom(mask_rom, NULL, "riscv_virt_board.mrom",
> > > diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c index
> > > 439dc89ee7..66f35bcbbf 100644
> > > --- a/target/riscv/cpu.c
> > > +++ b/target/riscv/cpu.c
> > > @@ -494,6 +494,8 @@ static void riscv_cpu_realize(DeviceState *dev,
> > > Error **errp)
> > >
> > >      riscv_cpu_register_gdb_regs_for_features(cs);
> > >
> > > +    env->user_frequency = env->frequency;
> > > +
> > >      qemu_init_vcpu(cs);
> > >      cpu_reset(cs);
> > >
> > > @@ -531,6 +533,7 @@ static Property riscv_cpu_properties[] = {
> > >      DEFINE_PROP_BOOL("mmu", RISCVCPU, cfg.mmu, true),
> > >      DEFINE_PROP_BOOL("pmp", RISCVCPU, cfg.pmp, true),
> > >      DEFINE_PROP_UINT64("resetvec", RISCVCPU, cfg.resetvec,
> > > DEFAULT_RSTVEC),
> > > +    DEFINE_PROP_UINT64("time-frequency", RISCVCPU, env.frequency, 0),
> >
> > Why not set the default to SIFIVE_CLINT_TIMEBASE_FREQ?
> >
>
> When the time frequency is not specified, it will follow the host or the migration
> source. And we define 0 as equivalent to not specified time frequency.
>
> > Also, QEMU now has a clock API, is using that instead a better option?
> >
>
> Sorry, I didn't find the clock API. Could you tell me what the API is.
> I think that the time frequency is option of KVM VCPU. So it is appropriate to put this
> option in the CPU.

The clock API is documented here:
https://gitlab.com/qemu-project/qemu/-/blob/master/docs/devel/clocks.rst

I'm not sure if it applies to KVM, but it is at least worth considering.

Alistair

>
> Yifei
>
> > Alistair
> >
> > >      DEFINE_PROP_END_OF_LIST(),
> > >  };
> > >
> > > diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h index
> > > 16d6050ead..f5b6c34176 100644
> > > --- a/target/riscv/cpu.h
> > > +++ b/target/riscv/cpu.h
> > > @@ -243,6 +243,8 @@ struct CPURISCVState {
> > >      uint64_t kvm_timer_time;
> > >      uint64_t kvm_timer_compare;
> > >      uint64_t kvm_timer_state;
> > > +    uint64_t user_frequency;
> > > +    uint64_t frequency;
> > >  };
> > >
> > >  OBJECT_DECLARE_TYPE(RISCVCPU, RISCVCPUClass,
> > > --
> > > 2.19.1
> > >
> > >

[PATCH RFC v4 14/15] target/riscv: Synchronize vcpu's frequency with KVM

[Yifei Jiang]

If vcpu's frequency is specified by cpu option 'frequency', it
will be set into KVM by KVM_SET_ONE_REG ioctl. Otherwise, vcpu's
frequency will follow KVM by KVM_GET_ONE_REG ioctl.

Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
---
 target/riscv/kvm.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/target/riscv/kvm.c b/target/riscv/kvm.c
index 1e16d24544..3499efd4eb 100644
--- a/target/riscv/kvm.c
+++ b/target/riscv/kvm.c
@@ -409,6 +409,8 @@ int kvm_arch_get_registers(CPUState *cs)
 int kvm_arch_put_registers(CPUState *cs, int level)
 {
     int ret = 0;
+    RISCVCPU *cpu = RISCV_CPU(cs);
+    CPURISCVState *env = &cpu->env;
 
     ret = kvm_riscv_put_regs_core(cs);
     if (ret) {
@@ -425,6 +427,10 @@ int kvm_arch_put_registers(CPUState *cs, int level)
         return ret;
     }
 
+    if (env->frequency) {
+        ret = kvm_set_one_reg(cs, RISCV_TIMER_REG(frequency), &env->frequency);
+    }
+
     return ret;
 }
 
@@ -481,6 +487,17 @@ int kvm_arch_init_vcpu(CPUState *cs)
     }
     env->misa = isa;
 
+    /*
+     * Synchronize vcpu's frequency with KVM. If vcpu's frequency is specified
+     * by cpu option 'frequency', this will be set to KVM. Otherwise, vcpu's
+     * frequency will follow KVM.
+     */
+    if (env->user_frequency) {
+        ret = kvm_set_one_reg(cs, RISCV_TIMER_REG(frequency), &env->frequency);
+    } else {
+        ret = kvm_get_one_reg(cs, RISCV_TIMER_REG(frequency), &env->frequency);
+    }
+
     return ret;
 }
 
-- 
2.19.1

[PATCH RFC v4 15/15] target/riscv: Add time frequency migration support

[Yifei Jiang]

If vcpu's time frequency is not specified by CPU option 'time-frequency'
on the destination, the time frequency of destination will follow
the source.

If vcpu's time frequency specified by CPU option 'time-frequency' on the
destination is different from migrated time frequency. The migration
will be abort.

Signed-off-by: Yifei Jiang <jiangyifei@huawei.com>
Signed-off-by: Yipeng Yin <yinyipeng1@huawei.com>
---
 target/riscv/machine.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/target/riscv/machine.c b/target/riscv/machine.c
index ef2d5395a8..6955542fef 100644
--- a/target/riscv/machine.c
+++ b/target/riscv/machine.c
@@ -144,6 +144,13 @@ static int cpu_post_load(void *opaque, int version_id)
     CPURISCVState *env = &cpu->env;
 
     env->kvm_timer_dirty = true;
+
+    if (env->user_frequency && env->user_frequency != env->frequency) {
+        error_report("Mismatch between user-specified time frequency and "
+                     "migrated time frequency");
+        return -EINVAL;
+    }
+
     return 0;
 }
 
@@ -198,6 +205,7 @@ const VMStateDescription vmstate_riscv_cpu = {
         VMSTATE_UINT64(env.kvm_timer_time, RISCVCPU),
         VMSTATE_UINT64(env.kvm_timer_compare, RISCVCPU),
         VMSTATE_UINT64(env.kvm_timer_state, RISCVCPU),
+        VMSTATE_UINT64(env.frequency, RISCVCPU),
 
         VMSTATE_END_OF_LIST()
     },
-- 
2.19.1