[Qemu-devel] [PULL] RISC-V QEMU Port Submission

[Qemu-devel] [PULL] RISC-V QEMU Port Submission

[Michael Clark]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

The following changes since commit 0a773d55ac76c5aa89ed9187a3bc5af8c5c2a6d0:

  maintainers: Add myself as a OpenBSD maintainer (2018-02-23 12:05:07 +0000)

are available in the git repository at:

  https://github.com/riscv/riscv-qemu.git tags/riscv-qemu-upstream-v7

for you to fetch changes up to 170a9d412ca1eb3b7ae6f6c1ff86dcbdff0fd7a8:

  RISC-V Build Infrastructure (2018-02-27 11:09:43 +1300)

- ----------------------------------------------------------------
QEMU RISC-V Emulation Support (RV64GC, RV32GC)

With this reelase we have contacted all contributors and have received
agreement to re-license their work as GPLv2+. This release also updates
linux-user/riscv/termbits.h which should fix S390 builds. The spike_v1.9
machine has been renamed to spike_v1.9.1 to match the privileged ISA
version and spike_v1.10 has been made the default machine. All substantial
work in this series has been reviewed or acknowledged and subsequent to
submitting this series we will be submitting signed a pull request.

The git tree for the v7 patch series tree (squashed and rebased):

- - https://github.com/riscv/riscv-qemu/tree/qemu-upstream-v7

The git tree for the v1-v6 patch series with review commit history:

- - https://github.com/riscv/riscv-qemu/tree/qemu-upstream-v6
- - https://github.com/riscv/riscv-qemu/tree/qemu-upstream-v5
- - https://github.com/riscv/riscv-qemu/tree/qemu-upstream-v4
- - https://github.com/riscv/riscv-qemu/tree/qemu-upstream-v3
- - https://github.com/riscv/riscv-qemu/tree/qemu-upstream-v2
- - https://github.com/riscv/riscv-qemu/tree/qemu-upstream-v1

*** Known Issues ***

- - Disassembler has some checkpatch warnings for the sake of code brevity
- - scripts/qemu-binfmt-conf.sh has checkpatch warnings due to line length
- - PMP (Physical Memory Protection) is as-of-yet unused and needs testing

*** Changelog ***

v7

- - Make spike_v1.10 the default machine
- - Rename spike_v1.9 to spike_v1.9.1 to match privileged spec version
- - Remove empty target/riscv/trace-events file
- - Monitor ROM 32-bit reset code needs to be target endian
- - Add TARGET_TIOCGPTPEER to linux-user/riscv/termbits.h
- - Add -initrd support to the virt board
- - Fix naming in spike machine interface header
- - Update copyright notice on RISC-V Spike machines
- - Update copyright notice on RISC-V HTIF Console device
- - Change CPU Core and translator to GPLv2+
- - Change RISC-V Disassembler to GPLv2+
- - Change SiFive Test Finisher to GPLv2+
- - Change SiFive CLINT to GPLv2+
- - Change SiFive PRCI to GPLv2+
- - Change SiFive PLIC to GPLv2+
- - Change RISC-V spike machines to GPLv2+
- - Change RISC-V virt machine to GPLv2+
- - Change SiFive E300 machine to GPLv2+
- - Change SiFive U500 machine to GPLv2+
- - Change RISC-V Hart Array to GPLv2+
- - Change RISC-V HTIF device to GPLv2+
- - Change SiFiveUART device to GPLv2+

v6

- - Drop IEEE 754-201x minimumNumber/maximumNumber for fmin/fmax
- - Remove some unnecessary commented debug statements
- - Change RISCV_CPU_TYPE_NAME to use riscv-cpu suffix
- - Define all CPU variants for linux-user
- - qemu_log calls require trailing \n
- - Replace PLIC printfs with qemu_log
- - Tear out unused HTIF code and eliminate shouting debug messages
- - Fix illegal instruction when sfence.vma is passed (rs2) arguments
- - Make updates to PTE accessed and dirty bits atomic
- - Only require atomic PTE updates on MTTCG enabled guests
- - Page fault if accessed or dirty bits can't be updated
- - Fix get_physical_address PTE reads and writes on riscv32
- - Remove erroneous comments from the PLIC
- - Default enable MTTCG
- - Make WFI less conservative
- - Unify local interrupt handling
- - Expunge HTIF interrupts
- - Always access mstatus.mip under a lock
- - Don't implement rdtime/rdtimeh in system mode (bbl emulates them)
- - Implement insreth/cycleh for rv32 and always enable user-mode counters
- - Add GDB stub support for reading and writing CSRs
- - Rename ENABLE_CHARDEV #ifdef from HTIF code
- - Replace bad HTIF ELF code with load_elf symbol callback
- - Convert chained if else fault handlers to switch statements
- - Use RISCV exception codes for linux-user page faults

v5

- - Implement NaN-boxing for flw, set high order bits to 1
- - Use float_muladd_negate_* flags to floatXX_muladd
- - Use IEEE 754-201x minimumNumber/maximumNumber for fmin/fmax
- - Fix TARGET_NR_syscalls
- - Update linux-user/riscv/syscall_nr.h
- - Fix FENCE.I, needs to terminate translation block
- - Adjust unusual convention for interruptno >= 0

v4

- - Add @riscv: since 2.12 to CpuInfoArch
- - Remove misleading little-endian comment from load_kernel
- - Rename cpu-model property to cpu-type
- - Drop some unnecessary inline function attributes
- - Don't allow GDB to set value of x0 register
- - Remove unnecessary empty property lists
- - Add Test Finisher device to implement poweroff in virt machine
- - Implement priv ISA v1.10 trap and sret/mret xPIE/xIE behavior
- - Store fflags data in fp_status
- - Purge runtime users of helper_raise_exception
- - Fix validate_csr
- - Tidy gen_jalr
- - Tidy immediate shifts
- - Add gen_exception_inst_addr_mis
- - Add gen_exception_debug
- - Add gen_exception_illegal
- - Tidy helper_fclass_*
- - Split rounding mode setting to a new function
- - Enforce MSTATUS_FS via TB flags
- - Implement acquire/release barrier semantics
- - Use atomic operations as required
- - Fix FENCE and FENCE_I
- - Remove commented code from spike machines
- - PAGE_WRITE permissions can be set on loads if page is already dirty
- - The result of format conversion on an NaN must be a quiet NaN
- - Add missing process_queued_cpu_work to riscv linux-user
- - Remove float(32|64)_classify from cpu.h
- - Removed nonsensical unions aliasing the same type
- - Use uintN_t instead of uintN_fast_t in fpu_helper.c
- - Use macros for FPU exception values in softfloat_flags_to_riscv
- - Move code to set round mode into set_fp_round_mode function
- - Convert set_fp_exceptions from a macro to an inline function
- - Convert round mode helper into an inline function
- - Make fpu_helper ieee_rm array static const
- - Include cpu_mmu_index in cpu_get_tb_cpu_state flags
- - Eliminate MPRV influence on mmu_index
- - Remove unrecoverable do_unassigned_access function
- - Only update PTE accessed and dirty bits if necessary
- - Remove unnecessary tlb_flush in set_mode as mode is in mmu_idx
- - Remove buggy support for misa writes. misa writes are optional
  and are not implemented in any known hardware
- - Always set PTE read or execute permissions during page walk
- - Reorder helper function declarations to match order in helper.c
- - Remove redundant variable declaration in get_physical_address
- - Remove duplicated code from get_physical_address
- - Use mmu_idx instead of mem_idx in riscv_cpu_get_phys_page_debug

v3

- - Fix indentation in PMP and HTIF debug macros
- - Fix disassembler checkpatch open brace '{' on next line errors
- - Fix trailing statements on next line in decode_inst_decompress
- - NOTE: the other checkpatch issues have been reviewed previously

v2

- - Remove redundant NULL terminators from disassembler register arrays
- - Change disassembler register name arrays to const
- - Refine disassembler internal function names
- - Update dates in disassembler copyright message
- - Remove #ifdef CONFIG_USER_ONLY version of cpu_has_work
- - Use ULL suffix on 64-bit constants
- - Move riscv_cpu_mmu_index from cpu.h to helper.c
- - Move riscv_cpu_hw_interrupts_pending from cpu.h to helper.c
- - Remove redundant TARGET_HAS_ICE from cpu.h
- - Use qemu_irq instead of void* for irq definition in cpu.h
- - Remove duplicate typedef from struct CPURISCVState
- - Remove redundant g_strdup from cpu_register
- - Remove redundant tlb_flush from riscv_cpu_reset
- - Remove redundant mode calculation from get_physical_address
- - Remove redundant debug mode printf and dcsr comment
- - Remove redundant clearing of MSB for bare physical addresses
- - Use g_assert_not_reached for invalid mode in get_physical_address
- - Use g_assert_not_reached for unreachable checks in get_physical_address
- - Use g_assert_not_reached for unreachable type in raise_mmu_exception
- - Return exception instead of aborting for misaligned fetches
- - Move exception defines from cpu.h to cpu_bits.h
- - Remove redundant breakpoint control definitions from cpu_bits.h
- - Implement riscv_cpu_unassigned_access exception handling
- - Log and raise exceptions for unimplemented CSRs
- - Match Spike HTIF exit behavior - don’t print TEST-PASSED
- - Make frm,fflags,fcsr writes trap when mstatus.FS is clear
- - Use g_assert_not_reached for unreachable invalid mode
- - Make hret,uret,dret generate illegal instructions
- - Move riscv_cpu_dump_state and int/fpr regnames to cpu.c
- - Lift interrupt flag and mask into constants in cpu_bits.h
- - Change trap debugging to use qemu_log_mask LOG_TRACE
- - Change CSR debugging to use qemu_log_mask LOG_TRACE
- - Change PMP debugging to use qemu_log_mask LOG_TRACE
- - Remove commented code from pmp.c
- - Change CpuInfoRISCV qapi schema docs to Since 2.12
- - Change RV feature macro to use target_ulong cast
- - Remove riscv_feature and instead use misa extension flags
- - Make riscv_flush_icache_syscall a no-op
- - Undo checkpatch whitespace fixes in unrelated linux-user code
- - Remove redudant constants and tidy up cpu_bits.h
- - Make helper_fence_i a no-op
- - Move include "exec/cpu-all" to end of cpu.h
- - Rename set_privilege to riscv_set_mode
- - Move redundant forward declaration for cpu_riscv_translate_address
- - Remove TCGV_UNUSED from riscv_translate_init
- - Add comment to pmp.c stating the code is untested and currently unused
- - Use ctz to simplify decoding of PMP NAPOT address ranges
- - Change pmp_is_in_range to use than equal for end addresses
- - Fix off by one error in pmp_update_rule
- - Rearrange PMP_DEBUG so that formatting is compile-time checked
- - Rearrange trap debugging so that formatting is compile-time checked
- - Rearrange PLIC debugging so that formatting is compile-time checked
- - Use qemu_log/qemu_log_mask for HTIF logging and debugging
- - Move exception and interrupt names into cpu.c
- - Add Palmer Dabbelt as a RISC-V Maintainer
- - Rebase against current qemu master branch

v1

- - initial version based on forward port from riscv-qemu repository

*** Background ***

"RISC-V is an open, free ISA enabling a new era of processor innovation
through open standard collaboration. Born in academia and research,
RISC-V ISA delivers a new level of free, extensible software and
hardware freedom on architecture, paving the way for the next 50 years
of computing design and innovation."

The QEMU RISC-V port has been developed and maintained out-of-tree for
several years by Sagar Karandikar and Bastian Koppelmann. The RISC-V
Privileged specification has evolved substantially over this period but
has recently been solidifying. The RISC-V Base ISA has been frozon for
some time and the Privileged ISA, GCC toolchain and Linux ABI are now
quite stable. I have recently joined Sagar and Bastian as a RISC-V QEMU
Maintainer and hope to support upstreaming the port.

There are multiple vendors taping out, preparing to ship, or shipping
silicon that implements the RISC-V Privileged ISA Version 1.10. There
are also several RISC-V Soft-IP cores implementing Privileged ISA
Version 1.10 that run on FPGA such as SiFive's Freedom U500 Platform
and the U54‑MC RISC-V Core IP, among many more implementations from a
variety of vendors. See https://riscv.org/ for more details.

RISC-V support was upstreamed in binutils 2.28 and GCC 7.1 in the first
half of 2016. RISC-V support is now available in LLVM top-of-tree and
the RISC-V Linux port was accepted into Linux 4.15-rc1 late last year
and is available in the Linux 4.15 release. GLIBC 2.27 added support
for the RISC-V ISA running on Linux (requires at least binutils-2.30,
gcc-7.3.0, and linux-4.15). We believe it is timely to submit the
RISC-V QEMU port for upstream review with the goal of incorporating
RISC-V support into the upcoming QEMU 2.12 release.

The RISC-V QEMU port is still under active development, mostly with
respect to device emulation, the addition of Hypervisor support as
specified in the RISC-V Draft Privileged ISA Version 1.11, and Vector
support once the first draft is finalized later this year. We believe
now is the appropriate time for RISC-V QEMU development to be carried
out in the main QEMU repository as the code will benefit from more
rigorous review. The RISC-V QEMU port currently supports all the ISA
extensions that have been finalized and frozen in the Base ISA.

Blog post about recent additions to RISC-V QEMU: https://goo.gl/fJ4zgk

The RISC-V QEMU wiki: https://github.com/riscv/riscv-qemu/wiki

Instructions for building a busybox+dropbear root image, BBL (Berkeley
Boot Loader) and linux kernel image for use with the RISC-V QEMU
'virt' machine: https://github.com/michaeljclark/busybear-linux

*** Overview ***

The RISC-V QEMU port implements the following specifications:

- - RISC-V Instruction Set Manual Volume I: User-Level ISA Version 2.2
- - RISC-V Instruction Set Manual Volume II: Privileged ISA Version 1.9.1
- - RISC-V Instruction Set Manual Volume II: Privileged ISA Version 1.10

The RISC-V QEMU port supports the following instruction set extensions:

- - RV32GC with Supervisor-mode and User-mode (RV32IMAFDCSU)
- - RV64GC with Supervisor-mode and User-mode (RV64IMAFDCSU)

The RISC-V QEMU port adds the following targets to QEMU:

- - riscv32-softmmu
- - riscv64-softmmu
- - riscv32-linux-user
- - riscv64-linux-user

The RISC-V QEMU port supports the following hardware:

- - HTIF Console (Host Target Interface)
- - SiFive CLINT (Core Local Interruptor) for Timer interrupts and IPIs
- - SiFive PLIC (Platform Level Interrupt Controller)
- - SiFive Test (Test Finisher) for exiting simulation
- - SiFive UART, PRCI, AON, PWM, QSPI support is partially implemented
- - VirtIO MMIO (GPEX PCI support will be added in a future patch)
- - Generic 16550A UART emulation using 'hw/char/serial.c'
- - MTTCG and SMP support (PLIC and CLINT) on the 'virt' machine

The RISC-V QEMU full system emulator supports 5 machines:

- - 'spike_v1.9';  CLINT, PLIC, HTIF console, config-string, Priv v1.9.1
- - 'spike_v1.10'; CLINT, PLIC, HTIF console, device-tree, Priv v1.10
- - 'sifive_e300'; CLINT, PLIC, SiFive UART, HiFive1 compat, Priv v1.10
- - 'sifive_u500'; CLINT, PLIC, SiFive UART, device-tree, Priv v1.10
- - 'virt'; CLINT, PLIC, 16550A UART, VirtIO, device-tree, Priv v1.10

This is a list of RISC-V QEMU Port Contributors:

- - Alex Suykov
- - Andreas Schwab
- - Antony Pavlov
- - Bastian Koppelmann
- - Bruce Hoult
- - Chih-Min Chao
- - Daire McNamara
- - Darius Rad
- - David Abdurachmanov
- - Hesham Almatary
- - Ivan Griffin
- - Jim Wilson
- - Kito Cheng
- - Michael Clark
- - Palmer Dabbelt
- - Richard Henderson
- - Sagar Karandikar
- - Shea Levy
- - Stefan O'Rear

Notes:

- - contributor email addresses available off-list on request.
- - checkpatch has been run on all 23 patches.
- - checkpatch exceptions are noted in patches that have errors.
- - passes "make check" on full build for all targets
- - tested riscv-linux-4.6.2 on 'spike_v1.9' machine
- - tested riscv-linux-4.15 on 'spike_v1.10' and 'virt' machines
- - tested SiFive HiFive1 binaries in 'sifive_e300' machine
- - tested RV64 on 32-bit i386

- ----------------------------------------------------------------
Michael Clark (23):
      RISC-V Maintainers
      RISC-V ELF Machine Definition
      RISC-V CPU Core Definition
      RISC-V Disassembler
      RISC-V CPU Helpers
      RISC-V FPU Support
      RISC-V GDB Stub
      RISC-V TCG Code Generation
      RISC-V Physical Memory Protection
      RISC-V Linux User Emulation
      Add symbol table callback interface to load_elf
      RISC-V HTIF Console
      RISC-V HART Array
      SiFive RISC-V CLINT Block
      SiFive RISC-V PLIC Block
      RISC-V Spike Machines
      SiFive RISC-V Test Finisher
      RISC-V VirtIO Machine
      SiFive RISC-V UART Device
      SiFive RISC-V PRCI Block
      SiFive Freedom E300 RISC-V Machine
      SiFive Freedom U500 RISC-V Machine
      RISC-V Build Infrastructure

 MAINTAINERS                            |   11 +
 arch_init.c                            |    2 +
 configure                              |   13 +
 cpus.c                                 |    6 +
 default-configs/riscv32-linux-user.mak |    1 +
 default-configs/riscv32-softmmu.mak    |    4 +
 default-configs/riscv64-linux-user.mak |    1 +
 default-configs/riscv64-softmmu.mak    |    4 +
 disas.c                                |    2 +
 disas/Makefile.objs                    |    1 +
 disas/riscv.c                          | 3048 ++++++++++++++++++++++++++++++++
 fpu/softfloat-specialize.h             |    7 +-
 hw/core/loader.c                       |   18 +-
 hw/riscv/Makefile.objs                 |   12 +
 hw/riscv/riscv_hart.c                  |   89 +
 hw/riscv/riscv_htif.c                  |  258 +++
 hw/riscv/sifive_clint.c                |  254 +++
 hw/riscv/sifive_e300.c                 |  236 +++
 hw/riscv/sifive_plic.c                 |  505 ++++++
 hw/riscv/sifive_prci.c                 |   89 +
 hw/riscv/sifive_test.c                 |   93 +
 hw/riscv/sifive_u500.c                 |  341 ++++
 hw/riscv/sifive_uart.c                 |  176 ++
 hw/riscv/spike_v1_09.c                 |  202 +++
 hw/riscv/spike_v1_10.c                 |  277 +++
 hw/riscv/virt.c                        |  420 +++++
 include/disas/bfd.h                    |    2 +
 include/elf.h                          |    2 +
 include/hw/elf_ops.h                   |   34 +-
 include/hw/loader.h                    |   17 +-
 include/hw/riscv/riscv_hart.h          |   39 +
 include/hw/riscv/riscv_htif.h          |   61 +
 include/hw/riscv/sifive_clint.h        |   50 +
 include/hw/riscv/sifive_e300.h         |   73 +
 include/hw/riscv/sifive_plic.h         |   85 +
 include/hw/riscv/sifive_prci.h         |   37 +
 include/hw/riscv/sifive_test.h         |   42 +
 include/hw/riscv/sifive_u500.h         |   63 +
 include/hw/riscv/sifive_uart.h         |   71 +
 include/hw/riscv/spike.h               |   45 +
 include/hw/riscv/virt.h                |   68 +
 include/sysemu/arch_init.h             |    1 +
 linux-user/elfload.c                   |   22 +
 linux-user/main.c                      |   99 ++
 linux-user/riscv/syscall_nr.h          |  287 +++
 linux-user/riscv/target_cpu.h          |   18 +
 linux-user/riscv/target_signal.h       |   23 +
 linux-user/riscv/target_structs.h      |   46 +
 linux-user/riscv/target_syscall.h      |   56 +
 linux-user/riscv/termbits.h            |  222 +++
 linux-user/signal.c                    |  203 ++-
 linux-user/syscall.c                   |    2 +
 linux-user/syscall_defs.h              |   13 +-
 qapi-schema.json                       |   16 +-
 scripts/qemu-binfmt-conf.sh            |   13 +-
 target/riscv/Makefile.objs             |    1 +
 target/riscv/cpu.c                     |  390 ++++
 target/riscv/cpu.h                     |  275 +++
 target/riscv/cpu_bits.h                |  416 +++++
 target/riscv/cpu_user.h                |   13 +
 target/riscv/fpu_helper.c              |  373 ++++
 target/riscv/gdbstub.c                 |   62 +
 target/riscv/helper.c                  |  503 ++++++
 target/riscv/helper.h                  |   78 +
 target/riscv/instmap.h                 |  364 ++++
 target/riscv/op_helper.c               |  655 +++++++
 target/riscv/pmp.c                     |  380 ++++
 target/riscv/pmp.h                     |   64 +
 target/riscv/translate.c               | 1974 +++++++++++++++++++++
 69 files changed, 13301 insertions(+), 27 deletions(-)
 create mode 100644 default-configs/riscv32-linux-user.mak
 create mode 100644 default-configs/riscv32-softmmu.mak
 create mode 100644 default-configs/riscv64-linux-user.mak
 create mode 100644 default-configs/riscv64-softmmu.mak
 create mode 100644 disas/riscv.c
 create mode 100644 hw/riscv/Makefile.objs
 create mode 100644 hw/riscv/riscv_hart.c
 create mode 100644 hw/riscv/riscv_htif.c
 create mode 100644 hw/riscv/sifive_clint.c
 create mode 100644 hw/riscv/sifive_e300.c
 create mode 100644 hw/riscv/sifive_plic.c
 create mode 100644 hw/riscv/sifive_prci.c
 create mode 100644 hw/riscv/sifive_test.c
 create mode 100644 hw/riscv/sifive_u500.c
 create mode 100644 hw/riscv/sifive_uart.c
 create mode 100644 hw/riscv/spike_v1_09.c
 create mode 100644 hw/riscv/spike_v1_10.c
 create mode 100644 hw/riscv/virt.c
 create mode 100644 include/hw/riscv/riscv_hart.h
 create mode 100644 include/hw/riscv/riscv_htif.h
 create mode 100644 include/hw/riscv/sifive_clint.h
 create mode 100644 include/hw/riscv/sifive_e300.h
 create mode 100644 include/hw/riscv/sifive_plic.h
 create mode 100644 include/hw/riscv/sifive_prci.h
 create mode 100644 include/hw/riscv/sifive_test.h
 create mode 100644 include/hw/riscv/sifive_u500.h
 create mode 100644 include/hw/riscv/sifive_uart.h
 create mode 100644 include/hw/riscv/spike.h
 create mode 100644 include/hw/riscv/virt.h
 create mode 100644 linux-user/riscv/syscall_nr.h
 create mode 100644 linux-user/riscv/target_cpu.h
 create mode 100644 linux-user/riscv/target_signal.h
 create mode 100644 linux-user/riscv/target_structs.h
 create mode 100644 linux-user/riscv/target_syscall.h
 create mode 100644 linux-user/riscv/termbits.h
 create mode 100644 target/riscv/Makefile.objs
 create mode 100644 target/riscv/cpu.c
 create mode 100644 target/riscv/cpu.h
 create mode 100644 target/riscv/cpu_bits.h
 create mode 100644 target/riscv/cpu_user.h
 create mode 100644 target/riscv/fpu_helper.c
 create mode 100644 target/riscv/gdbstub.c
 create mode 100644 target/riscv/helper.c
 create mode 100644 target/riscv/helper.h
 create mode 100644 target/riscv/instmap.h
 create mode 100644 target/riscv/op_helper.c
 create mode 100644 target/riscv/pmp.c
 create mode 100644 target/riscv/pmp.h
 create mode 100644 target/riscv/translate.c
-----BEGIN PGP SIGNATURE-----

iF0EARECAB0WIQR8mZMOsXzYugc9Xvpr8dezV+8+TwUCWpSiWgAKCRBr8dezV+8+
T4PVAJ9KlofNs1hS84ZxELwerF58DluL2QCfVh8I2NKTs7CECPpvjCI+L1sqX1c=
=97Kj
-----END PGP SIGNATURE-----

Re: [Qemu-devel] [PULL] RISC-V QEMU Port Submission

[Peter Maydell]

On 27 February 2018 at 00:15, Michael Clark <mjc@sifive.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> The following changes since commit 0a773d55ac76c5aa89ed9187a3bc5af8c5c2a6d0:
>
>   maintainers: Add myself as a OpenBSD maintainer (2018-02-23 12:05:07 +0000)
>
> are available in the git repository at:
>
>   https://github.com/riscv/riscv-qemu.git tags/riscv-qemu-upstream-v7
>
> for you to fetch changes up to 170a9d412ca1eb3b7ae6f6c1ff86dcbdff0fd7a8:
>
>   RISC-V Build Infrastructure (2018-02-27 11:09:43 +1300)
>
> - ----------------------------------------------------------------
> QEMU RISC-V Emulation Support (RV64GC, RV32GC)

Hi; thanks for this pull request. Unfortunately it seems to
be missing Signed-off-by: tags. Every commit needs to have
the Signed-off-by: tags from the people who contributed code to
it, indicating that they're OK with the code going into QEMU.
(If the work was done by and copyright a company then you don't
need to provide signoffs from every person at the company who
worked on the code if you don't want to.)

> The spike_v1.9
> machine has been renamed to spike_v1.9.1 to match the privileged ISA
> version and spike_v1.10 has been made the default machine.

I'm confused about this. Generally QEMU boards should model
hardware, and the board shouldn't care about the ISA versions.
Versioned board names in QEMU generally follow _QEMU_'s versioning,
and indicate that a board is identical to whatever we modelled
in that earlier QEMU version, for VM migration compatibility.
Board renames for minor ISA version bumps sounds like there's going
to be a lot of churn and breakage -- is this stuff really ready?
(Also, should we really have two different board source files
for two different ISA versions? I would have expected these to
share a source file to share code.)

I did a test build and there are some compile errors:

/home/pm215/qemu/linux-user/main.c:38:24: fatal error: target_elf.h:
No such file or directory
 #include "target_elf.h"
                        ^
compilation terminated.

This is because your patchset has a clash with commit 542ca4349878a2e
which has just merged to master, and refactors out an ifdef ladder,
so now all targets supporting linux-user need to provide a
linux-user/$ARCH/target_elf.h file. Could you fix that up and rebase,
please?

thanks
-- PMM

Re: [Qemu-devel] [patches] Re: [PULL] RISC-V QEMU Port Submission

[Stef O'Rear]

On Tue, Feb 27, 2018 at 6:01 AM, Peter Maydell <peter.maydell@linaro.org> wrote:
> On 27 February 2018 at 00:15, Michael Clark <mjc@sifive.com> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> The following changes since commit 0a773d55ac76c5aa89ed9187a3bc5af8c5c2a6d0:
>>
>>   maintainers: Add myself as a OpenBSD maintainer (2018-02-23 12:05:07 +0000)
>>
>> are available in the git repository at:
>>
>>   https://github.com/riscv/riscv-qemu.git tags/riscv-qemu-upstream-v7
>>
>> for you to fetch changes up to 170a9d412ca1eb3b7ae6f6c1ff86dcbdff0fd7a8:
>>
>>   RISC-V Build Infrastructure (2018-02-27 11:09:43 +1300)
>>
>> - ----------------------------------------------------------------
>> QEMU RISC-V Emulation Support (RV64GC, RV32GC)
>
> Hi; thanks for this pull request. Unfortunately it seems to
> be missing Signed-off-by: tags. Every commit needs to have
> the Signed-off-by: tags from the people who contributed code to
> it, indicating that they're OK with the code going into QEMU.
> (If the work was done by and copyright a company then you don't
> need to provide signoffs from every person at the company who
> worked on the code if you don't want to.)

I'll add mine.

>> The spike_v1.9
>> machine has been renamed to spike_v1.9.1 to match the privileged ISA
>> version and spike_v1.10 has been made the default machine.
>
> I'm confused about this. Generally QEMU boards should model
> hardware, and the board shouldn't care about the ISA versions.

The spike boards model the Berkeley architectural simulator "spike"
(https://github.com/riscv/riscv-isa-sim), which does not have a formal
release process or version numbers so we are using the ISA version as
a proxy for spike's version.

When physical boards are released with full documentation I presume we
will be adding board definitions for them, and they will imply
specific ISA versions.

> Versioned board names in QEMU generally follow _QEMU_'s versioning,
> and indicate that a board is identical to whatever we modelled
> in that earlier QEMU version, for VM migration compatibility.

In this case we're handling two logically distinct boards.  We could
combine them and implement a parameter; I was having trouble finding a
suitable example to follow earlier but it looks like gic-version in
hw/arm/virt.c is one.  This seems like a bad thing to change this late
in the review though?

> Board renames for minor ISA version bumps sounds like there's going
> to be a lot of churn and breakage -- is this stuff really ready?
> (Also, should we really have two different board source files
> for two different ISA versions? I would have expected these to
> share a source file to share code.)

1.10 is the version we have committed to long term support for; it
matches all public hardware the upstream Linux port, so it seems
appropriate to use for QEMU.

1.9.1 was the version supported by riscv-qemu at the time Michael
Clark took over maintainership; we have not removed support for it
because we cannot prove that there is nobody depending on it, although
I do not use it myself and do not know anyone else who does, so I
would not personably object to removing it if that were required.

Combining spike_v1.10 and spike_v1.9.1 would also be an option amenable to us.

-s

Re: [Qemu-devel] [PULL] RISC-V QEMU Port Submission

[Igor Mammedov]

On Tue, 27 Feb 2018 14:01:05 +0000
Peter Maydell <peter.maydell@linaro.org> wrote:

> On 27 February 2018 at 00:15, Michael Clark <mjc@sifive.com> wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > The following changes since commit 0a773d55ac76c5aa89ed9187a3bc5af8c5c2a6d0:
> >
> >   maintainers: Add myself as a OpenBSD maintainer (2018-02-23 12:05:07 +0000)
> >
> > are available in the git repository at:
> >
> >   https://github.com/riscv/riscv-qemu.git tags/riscv-qemu-upstream-v7
> >
> > for you to fetch changes up to 170a9d412ca1eb3b7ae6f6c1ff86dcbdff0fd7a8:
> >
> >   RISC-V Build Infrastructure (2018-02-27 11:09:43 +1300)
> >
> > - ----------------------------------------------------------------
> > QEMU RISC-V Emulation Support (RV64GC, RV32GC)  
> 
> Hi; thanks for this pull request. Unfortunately it seems to
> be missing Signed-off-by: tags. Every commit needs to have
> the Signed-off-by: tags from the people who contributed code to
> it, indicating that they're OK with the code going into QEMU.
> (If the work was done by and copyright a company then you don't
> need to provide signoffs from every person at the company who
> worked on the code if you don't want to.)
> 
> > The spike_v1.9
> > machine has been renamed to spike_v1.9.1 to match the privileged ISA
> > version and spike_v1.10 has been made the default machine.  
> 
> I'm confused about this. Generally QEMU boards should model
> hardware, and the board shouldn't care about the ISA versions.
> Versioned board names in QEMU generally follow _QEMU_'s versioning,
> and indicate that a board is identical to whatever we modelled
> in that earlier QEMU version, for VM migration compatibility.
> Board renames for minor ISA version bumps sounds like there's going
> to be a lot of churn and breakage -- is this stuff really ready?
> (Also, should we really have two different board source files
> for two different ISA versions? I would have expected these to
> share a source file to share code.)
> 
> I did a test build and there are some compile errors:
> 
> /home/pm215/qemu/linux-user/main.c:38:24: fatal error: target_elf.h:
> No such file or directory
>  #include "target_elf.h"
>                         ^
> compilation terminated.
> 
> This is because your patchset has a clash with commit 542ca4349878a2e
> which has just merged to master, and refactors out an ifdef ladder,
> so now all targets supporting linux-user need to provide a
> linux-user/$ARCH/target_elf.h file. Could you fix that up and rebase,
> please?
also '[PATCH v7 03/23] RISC-V CPU Core Definition' still hasn't addressed
comment http://lists.gnu.org/archive/html/qemu-devel/2018-02/msg06412.html
which isn't fixed since it was first pointed out (v4).

I'd prefer that being fixed before merge so another people
won't have to clean it up later after original authors,
When they try to generalize cpu_type -> cpu_model conversion.

> 
> thanks
> -- PMM
> 

Re: [Qemu-devel] [patches] Re: [PULL] RISC-V QEMU Port Submission

[Peter Maydell]

On 27 February 2018 at 15:50, Stef O'Rear <sorear2@gmail.com> wrote:
> On Tue, Feb 27, 2018 at 6:01 AM, Peter Maydell <peter.maydell@linaro.org> wrote:
>> On 27 February 2018 at 00:15, Michael Clark <mjc@sifive.com> wrote:
>>> The spike_v1.9
>>> machine has been renamed to spike_v1.9.1 to match the privileged ISA
>>> version and spike_v1.10 has been made the default machine.
>>
>> I'm confused about this. Generally QEMU boards should model
>> hardware, and the board shouldn't care about the ISA versions.
>
> The spike boards model the Berkeley architectural simulator "spike"
> (https://github.com/riscv/riscv-isa-sim), which does not have a formal
> release process or version numbers so we are using the ISA version as
> a proxy for spike's version.
>
> When physical boards are released with full documentation I presume we
> will be adding board definitions for them, and they will imply
> specific ISA versions.
>
>> Versioned board names in QEMU generally follow _QEMU_'s versioning,
>> and indicate that a board is identical to whatever we modelled
>> in that earlier QEMU version, for VM migration compatibility.
>
> In this case we're handling two logically distinct boards.  We could
> combine them and implement a parameter; I was having trouble finding a
> suitable example to follow earlier but it looks like gic-version in
> hw/arm/virt.c is one.  This seems like a bad thing to change this late
> in the review though?

You don't need to make them one board with a command line option
if that doesn't suit -- for instance hw/arm/vexpress.c defines
multiple board models that are variants on each other and
share a lot of code. That said, see below...

>> Board renames for minor ISA version bumps sounds like there's going
>> to be a lot of churn and breakage -- is this stuff really ready?
>> (Also, should we really have two different board source files
>> for two different ISA versions? I would have expected these to
>> share a source file to share code.)
>
> 1.10 is the version we have committed to long term support for; it
> matches all public hardware the upstream Linux port, so it seems
> appropriate to use for QEMU.
>
> 1.9.1 was the version supported by riscv-qemu at the time Michael
> Clark took over maintainership; we have not removed support for it
> because we cannot prove that there is nobody depending on it, although
> I do not use it myself and do not know anyone else who does, so I
> would not personably object to removing it if that were required.

I would rather not have stray legacy old versions in QEMU just
because we think maybe somebody might be using them. If 1.10
is the long-term-support committed version, then I think we
should just have a model of that. Anybody who for some reason is
still stuck on an older unsupported version gets to find out
what "unsupported" means; they can always keep using whatever
old QEMU code base they've been using up til now, presumably.

thanks
-- PMM

Re: [Qemu-devel] [patches] Re: [PULL] RISC-V QEMU Port Submission

[Michael Clark]

On Wed, Feb 28, 2018 at 4:50 AM, Stef O'Rear <sorear2@gmail.com> wrote:

> On Tue, Feb 27, 2018 at 6:01 AM, Peter Maydell <peter.maydell@linaro.org>
> wrote:
> > On 27 February 2018 at 00:15, Michael Clark <mjc@sifive.com> wrote:
> >> -----BEGIN PGP SIGNED MESSAGE-----
> >> Hash: SHA1
> >>
> >> The following changes since commit 0a773d55ac76c5aa89ed9187a3bc5a
> f8c5c2a6d0:
> >>
> >>   maintainers: Add myself as a OpenBSD maintainer (2018-02-23 12:05:07
> +0000)
> >>
> >> are available in the git repository at:
> >>
> >>   https://github.com/riscv/riscv-qemu.git tags/riscv-qemu-upstream-v7
> >>
> >> for you to fetch changes up to 170a9d412ca1eb3b7ae6f6c1ff86dc
> bdff0fd7a8:
> >>
> >>   RISC-V Build Infrastructure (2018-02-27 11:09:43 +1300)
> >>
> >> - ----------------------------------------------------------------
> >> QEMU RISC-V Emulation Support (RV64GC, RV32GC)
> >
> > Hi; thanks for this pull request. Unfortunately it seems to
> > be missing Signed-off-by: tags. Every commit needs to have
> > the Signed-off-by: tags from the people who contributed code to
> > it, indicating that they're OK with the code going into QEMU.
> > (If the work was done by and copyright a company then you don't
> > need to provide signoffs from every person at the company who
> > worked on the code if you don't want to.)
>
> I'll add mine.
>
> >> The spike_v1.9
> >> machine has been renamed to spike_v1.9.1 to match the privileged ISA
> >> version and spike_v1.10 has been made the default machine.
> >
> > I'm confused about this. Generally QEMU boards should model
> > hardware, and the board shouldn't care about the ISA versions.
>
> The spike boards model the Berkeley architectural simulator "spike"
> (https://github.com/riscv/riscv-isa-sim), which does not have a formal
> release process or version numbers so we are using the ISA version as
> a proxy for spike's version.
>
> When physical boards are released with full documentation I presume we
> will be adding board definitions for them, and they will imply
> specific ISA versions.
>
> > Versioned board names in QEMU generally follow _QEMU_'s versioning,
> > and indicate that a board is identical to whatever we modelled
> > in that earlier QEMU version, for VM migration compatibility.
>
> In this case we're handling two logically distinct boards.  We could
> combine them and implement a parameter; I was having trouble finding a
> suitable example to follow earlier but it looks like gic-version in
> hw/arm/virt.c is one.  This seems like a bad thing to change this late
> in the review though?
>
> > Board renames for minor ISA version bumps sounds like there's going
> > to be a lot of churn and breakage -- is this stuff really ready?
> > (Also, should we really have two different board source files
> > for two different ISA versions? I would have expected these to
> > share a source file to share code.)
>
> 1.10 is the version we have committed to long term support for; it
> matches all public hardware the upstream Linux port, so it seems
> appropriate to use for QEMU.
>
> 1.9.1 was the version supported by riscv-qemu at the time Michael
> Clark took over maintainership; we have not removed support for it
> because we cannot prove that there is nobody depending on it, although
> I do not use it myself and do not know anyone else who does, so I
> would not personably object to removing it if that were required.
>
> Combining spike_v1.10 and spike_v1.9.1 would also be an option amenable to
> us.
>

I've just talked to SiFive about this. They have agreed that we can remove
the sifive_e300 and sifive_u500 boards from the patch series that we are
going to submit upstream again later this week or early next week. These
machines and their devices are pretty easy for us to maintain in the riscv
or a sifive repo. This trims the number of machines from 5 to 3 and lets us
remove the SiFiveUART and SiFivePRCI from the next patch series we are
going to submit. e.g. v8

SiFive have indicated that they would like to keep privileged ISA v1.9.1
support. It's likely the RISC-V foundation would also like us to start
supporting backwards compatibility from this point. Removing support for a
specification version only 4 months after the latest specification has been
implemented is too severe of a deprecation period. They have said they
would like QEMU to support at least 2 specification versions so we won't
consider removing privileged ISA v1.9.1 support until privileged ISA v1.11
has been released and implemented. There are still several OS ports and
private tape-outs and test chips that target privileged ISA v1.9.1. In
fact, someone may very well add privileged ISA v1.9 and privileged ISA v1.7
support, perhaps as a computing history project. The published
specifications are all available however the chips implementing these
versions of the spec are mostly test chips. Nevertheless, they are part of
RISC-V history.

With respect to combining them, we could investigate triggering the config
string vs flattened device-tree, based on a restricted set of cpu models.
e.g. make spike-10901 and spike-11000 CPU models and have the spike board
supply config string or flattened device-tree based on the cpu model it is
being invoked with.

However I'm inclined to leave it as it is, at this point. It is not
something that we can't change in the future once the code is in-tree.

There will be strong resistance to removing privileged ISA v1.9.1. The
privileged
ISA specification version number is currently the only strong contract as
to the behaviour of the privileged control and status registers. The Base
ISA however is pretty much rock-solid before we started work on QEMU.
Privileged
ISA specification v1.10 will likely not make backwards incompatible
changes, however this does not change our stance on maintaining
compatibility with the official RISC-V ISA Simulator, for both privileged
ISA v1.9.1 and privileged ISA v1.10.

It's likely going forward that we won't see version compatibility breaks,
but in the process of implementing this discipline, it means we must keep
support for historical privileged ISA versions in one of the simulators, as
a runtime flag, not just in its commit history.

With respect to the other issues. I'm going to be seeking Signed-off-by
from the various contributors for the next patch series. v8

BTW somewhat coincidentally, the binary translator I wrote; RV8, which is
practicaly twice as fast as QEMU only supports privileged ISA v1.9.1 and I
personally want to keep binary compatiblity with it. This is as an aside to
the desire of foundation members that wish to preserve privileged ISA
v1.9.1 support. In RV8, we yet don't have binary translation on the
privileged ISA _yet_, but when we do, we will implement shadow paging, so
out privileged ISA support will be Hard MMU. RISC-V page tables are close
enough to x86_64 pages tables such that we could emulate them using shadow
paging vs softmmu, and with the addition of a register allocator (rv8 uses
a static register allocation from the 31 RISC-V registers to the 16 x86_64
registers) it's likely we'll get simulation performance up around 10
Billion Instructions per second (currently we're peaking at approximately 6
Billion RISC-V instructions per second on a Broadwell Core i7 5557U).
Unfortunaly I haven't had much time to spend on rv8 as QEMU is what SiFive
want me to support. One day we'll add Hard MMU for the translated target...
in fact we could consider doing this in QEMU?... The core rv8 engine is
much faster but it has terrible device support. It's possibly easier to
implement Hard MMU emulation support in an alternate translator? maybe we
could use KVM and do Hard MMU for RISC-V in QEMU?

- https://rv8.io/
- https://rv8.io/bench
- https://anarch128.org/~mclark/rv8-carrv.pdf
- https://anarch128.org/~mclark/rv8-slides.pdf

Re: [Qemu-devel] [patches] Re: [PULL] RISC-V QEMU Port Submission

[Michael Clark]

On Wed, Feb 28, 2018 at 6:50 AM, Peter Maydell <peter.maydell@linaro.org>
wrote:

> On 27 February 2018 at 15:50, Stef O'Rear <sorear2@gmail.com> wrote:
> > On Tue, Feb 27, 2018 at 6:01 AM, Peter Maydell <peter.maydell@linaro.org>
> wrote:
> >> On 27 February 2018 at 00:15, Michael Clark <mjc@sifive.com> wrote:
> >>> The spike_v1.9
> >>> machine has been renamed to spike_v1.9.1 to match the privileged ISA
> >>> version and spike_v1.10 has been made the default machine.
> >>
> >> I'm confused about this. Generally QEMU boards should model
> >> hardware, and the board shouldn't care about the ISA versions.
> >
> > The spike boards model the Berkeley architectural simulator "spike"
> > (https://github.com/riscv/riscv-isa-sim), which does not have a formal
> > release process or version numbers so we are using the ISA version as
> > a proxy for spike's version.
> >
> > When physical boards are released with full documentation I presume we
> > will be adding board definitions for them, and they will imply
> > specific ISA versions.
> >
> >> Versioned board names in QEMU generally follow _QEMU_'s versioning,
> >> and indicate that a board is identical to whatever we modelled
> >> in that earlier QEMU version, for VM migration compatibility.
> >
> > In this case we're handling two logically distinct boards.  We could
> > combine them and implement a parameter; I was having trouble finding a
> > suitable example to follow earlier but it looks like gic-version in
> > hw/arm/virt.c is one.  This seems like a bad thing to change this late
> > in the review though?
>
> You don't need to make them one board with a command line option
> if that doesn't suit -- for instance hw/arm/vexpress.c defines
> multiple board models that are variants on each other and
> share a lot of code. That said, see below...
>
> >> Board renames for minor ISA version bumps sounds like there's going
> >> to be a lot of churn and breakage -- is this stuff really ready?
> >> (Also, should we really have two different board source files
> >> for two different ISA versions? I would have expected these to
> >> share a source file to share code.)
> >
> > 1.10 is the version we have committed to long term support for; it
> > matches all public hardware the upstream Linux port, so it seems
> > appropriate to use for QEMU.
> >
> > 1.9.1 was the version supported by riscv-qemu at the time Michael
> > Clark took over maintainership; we have not removed support for it
> > because we cannot prove that there is nobody depending on it, although
> > I do not use it myself and do not know anyone else who does, so I
> > would not personably object to removing it if that were required.
>
> I would rather not have stray legacy old versions in QEMU just
> because we think maybe somebody might be using them. If 1.10
> is the long-term-support committed version, then I think we
> should just have a model of that. Anybody who for some reason is
> still stuck on an older unsupported version gets to find out
> what "unsupported" means; they can always keep using whatever
> old QEMU code base they've been using up til now, presumably.
>

SiFive are happy to support privileged ISA v1.9.1.

I don't think the branch we maintain will easily merge with a branch that
has privileged ISA v1.9.1 torn out (the only version that actually worked 4
months ago).

If we can't submit our port with privileged ISA v1.9.1 suport then thats
going to put a big spanner in the works.

We've made a pretty strong choice to not break backwards compatibility
going forward and privileged ISA v1.9.1 is the line in the sand so to
speak. i.e. the QEMU port is still compatible with binaries from the v1.9.1
ISA spec published in November 2016 which has been implemented by many
folk. We have to have a much more reasonable deprecation period. Software
such as GDB and OpenOCD continue to support privileged ISA v1.9.1 and have
specific fallback code paths, as well as the OpenOCD port having support
for two versions of the debug spec.

Re: [Qemu-devel] [PULL] RISC-V QEMU Port Submission

[Michael Clark]

On Wed, Feb 28, 2018 at 3:01 AM, Peter Maydell <peter.maydell@linaro.org>
wrote:

> On 27 February 2018 at 00:15, Michael Clark <mjc@sifive.com> wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > The following changes since commit 0a773d55ac76c5aa89ed9187a3bc5a
> f8c5c2a6d0:
> >
> >   maintainers: Add myself as a OpenBSD maintainer (2018-02-23 12:05:07
> +0000)
> >
> > are available in the git repository at:
> >
> >   https://github.com/riscv/riscv-qemu.git tags/riscv-qemu-upstream-v7
> >
> > for you to fetch changes up to 170a9d412ca1eb3b7ae6f6c1ff86dcbdff0fd7a8:
> >
> >   RISC-V Build Infrastructure (2018-02-27 11:09:43 +1300)
> >
> > - ----------------------------------------------------------------
> > QEMU RISC-V Emulation Support (RV64GC, RV32GC)
>
> Hi; thanks for this pull request. Unfortunately it seems to
> be missing Signed-off-by: tags. Every commit needs to have
> the Signed-off-by: tags from the people who contributed code to
> it, indicating that they're OK with the code going into QEMU.
> (If the work was done by and copyright a company then you don't
> need to provide signoffs from every person at the company who
> worked on the code if you don't want to.)
>
> > The spike_v1.9
> > machine has been renamed to spike_v1.9.1 to match the privileged ISA
> > version and spike_v1.10 has been made the default machine.
>
> I'm confused about this. Generally QEMU boards should model
> hardware, and the board shouldn't care about the ISA versions.
> Versioned board names in QEMU generally follow _QEMU_'s versioning,
> and indicate that a board is identical to whatever we modelled
> in that earlier QEMU version, for VM migration compatibility.
> Board renames for minor ISA version bumps sounds like there's going
> to be a lot of churn and breakage -- is this stuff really ready?
> (Also, should we really have two different board source files
> for two different ISA versions? I would have expected these to
> share a source file to share code.)
>
> I did a test build and there are some compile errors:
>
> /home/pm215/qemu/linux-user/main.c:38:24: fatal error: target_elf.h:
> No such file or directory
>  #include "target_elf.h"
>                         ^
> compilation terminated.
>
> This is because your patchset has a clash with commit 542ca4349878a2e
> which has just merged to master, and refactors out an ifdef ladder,
> so now all targets supporting linux-user need to provide a
> linux-user/$ARCH/target_elf.h file. Could you fix that up and rebase,
> please?
>

No worries. I'll rebase and submit a v8 patch series very soon.

I've just discussed with SiFive, and they wish to remove a couple of
machines and devices from the v8 patch series. They want to get the chip
model, SOC and board model right before they submit them upstream.

This is fine, as most folk seem to want to use "virt" to run Linux and we
have the "spike" machines that match the RISC-V Foundation ISA Simulator.

SiFive's boards are for customers that are using their MCUs without MMUs.
We can wait until they are fully baked before we submit them. The machines
are quite easy to maintain on a file level such that they won't cause much
trouble with them being in downstream repos. Other RISC-V vendors also
probably want to submit their boards and core models at some point too, if
they choose to support QEMU...

Re: [Qemu-devel] [patches] Re: [PULL] RISC-V QEMU Port Submission

[Michael Clark]

On Wed, Feb 28, 2018 at 5:00 AM, Igor Mammedov <imammedo@redhat.com> wrote:

> On Tue, 27 Feb 2018 14:01:05 +0000
> Peter Maydell <peter.maydell@linaro.org> wrote:
>
> > On 27 February 2018 at 00:15, Michael Clark <mjc@sifive.com> wrote:
> > > -----BEGIN PGP SIGNED MESSAGE-----
> > > Hash: SHA1
> > >
> > > The following changes since commit 0a773d55ac76c5aa89ed9187a3bc5a
> f8c5c2a6d0:
> > >
> > >   maintainers: Add myself as a OpenBSD maintainer (2018-02-23 12:05:07
> +0000)
> > >
> > > are available in the git repository at:
> > >
> > >   https://github.com/riscv/riscv-qemu.git tags/riscv-qemu-upstream-v7
> > >
> > > for you to fetch changes up to 170a9d412ca1eb3b7ae6f6c1ff86dc
> bdff0fd7a8:
> > >
> > >   RISC-V Build Infrastructure (2018-02-27 11:09:43 +1300)
> > >
> > > - ----------------------------------------------------------------
> > > QEMU RISC-V Emulation Support (RV64GC, RV32GC)
> >
> > Hi; thanks for this pull request. Unfortunately it seems to
> > be missing Signed-off-by: tags. Every commit needs to have
> > the Signed-off-by: tags from the people who contributed code to
> > it, indicating that they're OK with the code going into QEMU.
> > (If the work was done by and copyright a company then you don't
> > need to provide signoffs from every person at the company who
> > worked on the code if you don't want to.)
> >
> > > The spike_v1.9
> > > machine has been renamed to spike_v1.9.1 to match the privileged ISA
> > > version and spike_v1.10 has been made the default machine.
> >
> > I'm confused about this. Generally QEMU boards should model
> > hardware, and the board shouldn't care about the ISA versions.
> > Versioned board names in QEMU generally follow _QEMU_'s versioning,
> > and indicate that a board is identical to whatever we modelled
> > in that earlier QEMU version, for VM migration compatibility.
> > Board renames for minor ISA version bumps sounds like there's going
> > to be a lot of churn and breakage -- is this stuff really ready?
> > (Also, should we really have two different board source files
> > for two different ISA versions? I would have expected these to
> > share a source file to share code.)
> >
> > I did a test build and there are some compile errors:
> >
> > /home/pm215/qemu/linux-user/main.c:38:24: fatal error: target_elf.h:
> > No such file or directory
> >  #include "target_elf.h"
> >                         ^
> > compilation terminated.
> >
> > This is because your patchset has a clash with commit 542ca4349878a2e
> > which has just merged to master, and refactors out an ifdef ladder,
> > so now all targets supporting linux-user need to provide a
> > linux-user/$ARCH/target_elf.h file. Could you fix that up and rebase,
> > please?
> also '[PATCH v7 03/23] RISC-V CPU Core Definition' still hasn't addressed
> comment http://lists.gnu.org/archive/html/qemu-devel/2018-02/msg06412.html
> which isn't fixed since it was first pointed out (v4).
>
> I'd prefer that being fixed before merge so another people
> won't have to clean it up later after original authors,
> When they try to generalize cpu_type -> cpu_model conversion.
>

I re-read the email and it doesn't seem clear what you want us to do. I
changed the CPU suffix to a prefix as you requested. The rest of the CPU
initialisation is "modelled" on arm not sh4.

If you want to make a pull request, please use this branch:

- https://github.com/riscv/riscv-qemu/tree/qemu-upstream-v7

Re: [Qemu-devel] [patches] Re: [PULL] RISC-V QEMU Port Submission

[Igor Mammedov]

On Wed, 28 Feb 2018 13:41:25 +1300
Michael Clark <mjc@sifive.com> wrote:

> On Wed, Feb 28, 2018 at 5:00 AM, Igor Mammedov <imammedo@redhat.com> wrote:
> 
> > On Tue, 27 Feb 2018 14:01:05 +0000
> > Peter Maydell <peter.maydell@linaro.org> wrote:
> >  
> > > On 27 February 2018 at 00:15, Michael Clark <mjc@sifive.com> wrote:  
> > > > -----BEGIN PGP SIGNED MESSAGE-----
> > > > Hash: SHA1
> > > >
> > > > The following changes since commit 0a773d55ac76c5aa89ed9187a3bc5a  
> > f8c5c2a6d0:  
> > > >
> > > >   maintainers: Add myself as a OpenBSD maintainer (2018-02-23 12:05:07  
> > +0000)  
> > > >
> > > > are available in the git repository at:
> > > >
> > > >   https://github.com/riscv/riscv-qemu.git tags/riscv-qemu-upstream-v7
> > > >
> > > > for you to fetch changes up to 170a9d412ca1eb3b7ae6f6c1ff86dc  
> > bdff0fd7a8:  
> > > >
> > > >   RISC-V Build Infrastructure (2018-02-27 11:09:43 +1300)
> > > >
> > > > - ----------------------------------------------------------------
> > > > QEMU RISC-V Emulation Support (RV64GC, RV32GC)  
> > >
> > > Hi; thanks for this pull request. Unfortunately it seems to
> > > be missing Signed-off-by: tags. Every commit needs to have
> > > the Signed-off-by: tags from the people who contributed code to
> > > it, indicating that they're OK with the code going into QEMU.
> > > (If the work was done by and copyright a company then you don't
> > > need to provide signoffs from every person at the company who
> > > worked on the code if you don't want to.)
> > >  
> > > > The spike_v1.9
> > > > machine has been renamed to spike_v1.9.1 to match the privileged ISA
> > > > version and spike_v1.10 has been made the default machine.  
> > >
> > > I'm confused about this. Generally QEMU boards should model
> > > hardware, and the board shouldn't care about the ISA versions.
> > > Versioned board names in QEMU generally follow _QEMU_'s versioning,
> > > and indicate that a board is identical to whatever we modelled
> > > in that earlier QEMU version, for VM migration compatibility.
> > > Board renames for minor ISA version bumps sounds like there's going
> > > to be a lot of churn and breakage -- is this stuff really ready?
> > > (Also, should we really have two different board source files
> > > for two different ISA versions? I would have expected these to
> > > share a source file to share code.)
> > >
> > > I did a test build and there are some compile errors:
> > >
> > > /home/pm215/qemu/linux-user/main.c:38:24: fatal error: target_elf.h:
> > > No such file or directory
> > >  #include "target_elf.h"
> > >                         ^
> > > compilation terminated.
> > >
> > > This is because your patchset has a clash with commit 542ca4349878a2e
> > > which has just merged to master, and refactors out an ifdef ladder,
> > > so now all targets supporting linux-user need to provide a
> > > linux-user/$ARCH/target_elf.h file. Could you fix that up and rebase,
> > > please?  
> > also '[PATCH v7 03/23] RISC-V CPU Core Definition' still hasn't addressed
> > comment http://lists.gnu.org/archive/html/qemu-devel/2018-02/msg06412.html
> > which isn't fixed since it was first pointed out (v4).
> >
> > I'd prefer that being fixed before merge so another people
> > won't have to clean it up later after original authors,
> > When they try to generalize cpu_type -> cpu_model conversion.
> >  
> 
> I re-read the email and it doesn't seem clear what you want us to do.
> I changed the CPU suffix to a prefix as you requested. The rest of the CPU
> initialisation is "modelled" on arm not sh4.
Not addressed comment is not related to CPU suffix, it's a separate issue.

I haven't had time to clean up that part of ARM cpu initialization yet,
so it still uses old approach, but you've been pointed to right
approach rather early (v4) but ignored it.

There is no point to use legacy approach with new patches
that could block other people and would force them to cleanup
before doing what they intended to do.

I've pointed to sh4 as example that you should use for new CPU
types internalization. Beside making RISC-V consistent with
direction that part of code moves to, it will also simplify patch.

Please ask if something still unclear.

> If you want to make a pull request, please use this branch:
> 
> - https://github.com/riscv/riscv-qemu/tree/qemu-upstream-v7

Re: [Qemu-devel] [patches] Re: [PULL] RISC-V QEMU Port Submission

[Peter Maydell]

On 28 February 2018 at 00:09, Michael Clark <mjc@sifive.com> wrote:
> I've just talked to SiFive about this. They have agreed that we can remove
> the sifive_e300 and sifive_u500 boards from the patch series that we are
> going to submit upstream again later this week or early next week. These
> machines and their devices are pretty easy for us to maintain in the riscv
> or a sifive repo. This trims the number of machines from 5 to 3 and lets us
> remove the SiFiveUART and SiFivePRCI from the next patch series we are
> going to submit. e.g. v8

Models of boards which people actively want and are using are fine
(though indeed you can save them for a later round of patches if you
like). And it sounds like the 1.9.1 stuff is genuinely wanted, so
thanks for the clarification there. What I want to avoid is boards
going into QEMU just because you happened to have them already. Once
a board model goes into QEMU it's a commitment to supporting it for
the long term, and getting rid of it again is hard.

> However I'm inclined to leave it as it is, at this point. It is not
> something that we can't change in the future once the code is in-tree.

With my 'upstream dev' hat on, I tend to be suspicious of this
line of argument, because in a lot of cases what tends to happen
is that the code for some new target or device goes in-tree, and
then the people who worked on submitting it disappear, or never
actually do get round to refactoring[*]. You get more leeway for
making this argument the longer you've been around and participating
in QEMU development, because then you have a track record of
following up on things.

[*] in fact we're currently discussing deleting support for
a couple of target architectures that were basically "once the
code went into mainline nothing further was ever done to it except
global-refactorings and other tree wide maintenance by other
QEMU developers".

thanks
-- PMM

Re: [Qemu-devel] [patches] Re: [PULL] RISC-V QEMU Port Submission

[Peter Maydell]

On 28 February 2018 at 11:53, Peter Maydell <peter.maydell@linaro.org> wrote:
> With my 'upstream dev' hat on, I tend to be suspicious of this
> line of argument, because in a lot of cases what tends to happen
> is that the code for some new target or device goes in-tree, and
> then the people who worked on submitting it disappear, or never
> actually do get round to refactoring[*]. You get more leeway for
> making this argument the longer you've been around and participating
> in QEMU development, because then you have a track record of
> following up on things.

That said, I can probably live with it in this particular instance,
given when the 2.12 codefreeze deadline is.

thanks
-- PMM

Re: [Qemu-devel] [patches] Re: [PULL] RISC-V QEMU Port Submission

[Michael Clark]

On Thu, Mar 1, 2018 at 12:53 AM, Peter Maydell <peter.maydell@linaro.org>
wrote:

> On 28 February 2018 at 00:09, Michael Clark <mjc@sifive.com> wrote:
> > I've just talked to SiFive about this. They have agreed that we can
> remove
> > the sifive_e300 and sifive_u500 boards from the patch series that we are
> > going to submit upstream again later this week or early next week. These
> > machines and their devices are pretty easy for us to maintain in the
> riscv
> > or a sifive repo. This trims the number of machines from 5 to 3 and lets
> us
> > remove the SiFiveUART and SiFivePRCI from the next patch series we are
> > going to submit. e.g. v8
>
> Models of boards which people actively want and are using are fine
> (though indeed you can save them for a later round of patches if you
> like). And it sounds like the 1.9.1 stuff is genuinely wanted, so
> thanks for the clarification there. What I want to avoid is boards
> going into QEMU just because you happened to have them already. Once
> a board model goes into QEMU it's a commitment to supporting it for
> the long term, and getting rid of it again is hard.


Most folk in the community at large are interested in the 'spike' and
'virt' machines. Well, the linux distributors are currently targetting
'virt' as it has working network and block storage.

It's mostly SiFive customers that are interested in the SiFive machines.

SiFive do intend to submit their machines upstream however we've decided
that we want to review the naming of the machine/board/SOC in light of
several new models, as there are infact more MCU/SOC models than are
currently represented in the RISC-V QEMU port (E21, E31, E51, U54, U54MC).
Those are the SOCs and then there are boards like the HiFive1, the LoFive,
the HiFive Unleashed and several others (the 7th RISC-V Foundation Workshop
had an electronic badge with the FE310 designed by antmicro). SiFive might
like to review the naming and re-jig the SOC / board relationship.
Presently we created two generic boards to represent the MMU-less E-series
(sifive_e300) and the U-series (sifive_u500) however that doesn't cover the
E21 and E51. As well as silicon, there are soft-cores, including soft-cores
from other vendors (who have not yet submitted anything to the QEMU port).
After reflecting on this we don't think the naming of the two sifive
machines is quite right and they are not yet complete emulations. The
sifive_u500
is supposed to model the HiFive Unleashed however we don't have device
emulation for all of the hardware widgets yet, and some of the drivers are
not yet in upstream Linux. Linus essentially accepted the generic core,
which is what we are now suggesting for QEMU. We are totally happy to
submit them if folk don't mind us potentially renaming them later. This is
2 patches for 2 machines and 2 or 3 devices. It would reduce the patch
series from 23 patches to 18 patches and we'd maintain the 5 files and
associated headers out-of tree until things firm up. The core of the port
is pretty solid as Fedora have build stage 4 images using SMP on the 'virt'
machine.


> > However I'm inclined to leave it as it is, at this point. It is not
> > something that we can't change in the future once the code is in-tree.
>
> With my 'upstream dev' hat on, I tend to be suspicious of this
> line of argument, because in a lot of cases what tends to happen
> is that the code for some new target or device goes in-tree, and
> then the people who worked on submitting it disappear, or never
> actually do get round to refactoring[*]. You get more leeway for
> making this argument the longer you've been around and participating
> in QEMU development, because then you have a track record of
> following up on things.
>
> [*] in fact we're currently discussing deleting support for
> a couple of target architectures that were basically "once the
> code went into mainline nothing further was ever done to it except
> global-refactorings and other tree wide maintenance by other
> QEMU developers".


I might explore coalescing the two spike boards before the v8 spin then.

I was thinking about using a global to enable the backwards compatibility
mode. It seems -global is for driver properties? Is it appropriate to use a
global property to change the behaviour of a machine?

Okay, I see raspi.c contains raspi, raspi 2, raspi 3, and it just defines
multiple machines in the same file and passes a version number to routines
that are shared. We could go with that approach.

Re: [Qemu-devel] [patches] Re: [PULL] RISC-V QEMU Port Submission

[Emilio G. Cota]

On Wed, Feb 28, 2018 at 13:09:11 +1300, Michael Clark wrote:
> BTW somewhat coincidentally, the binary translator I wrote; RV8, which is
> practicaly twice as fast as QEMU only supports privileged ISA v1.9.1 and I
> personally want to keep binary compatiblity with it.
(snip)
> - https://rv8.io/
> - https://rv8.io/bench
> - https://anarch128.org/~mclark/rv8-carrv.pdf
> - https://anarch128.org/~mclark/rv8-slides.pdf

What QEMU versions did you use for those comparisons? I wonder if
the recent indirect branch handling improvements were included in those
(this work was merged in 2.10 for aarch64). Also, 2.7 is quite a bit
faster than previous versions for user-mode due to the use of QHT,
although you probably used a later version.

BTW after the merge you might want to look into optimizing indirect
branches (and cross-page direct jumps in softmmu) for riscv in qemu.
See examples with
	$ git log -Stcg_gen_lookup_and_goto_ptr

Cheers,

		Emilio

Re: [Qemu-devel] [patches] Re: [PULL] RISC-V QEMU Port Submission

[Michael Clark]

On Thu, Mar 1, 2018 at 11:26 AM, Emilio G. Cota <cota@braap.org> wrote:

> On Wed, Feb 28, 2018 at 13:09:11 +1300, Michael Clark wrote:
> > BTW somewhat coincidentally, the binary translator I wrote; RV8, which is
> > practicaly twice as fast as QEMU only supports privileged ISA v1.9.1 and
> I
> > personally want to keep binary compatiblity with it.
> (snip)
> > - https://rv8.io/
> > - https://rv8.io/bench
> > - https://anarch128.org/~mclark/rv8-carrv.pdf
> > - https://anarch128.org/~mclark/rv8-slides.pdf
>
> What QEMU versions did you use for those comparisons? I wonder if
> the recent indirect branch handling improvements were included in those
> (this work was merged in 2.10 for aarch64). Also, 2.7 is quite a bit
> faster than previous versions for user-mode due to the use of QHT,
> although you probably used a later version.
>

Yes I noticed indirect branch handling was very slow in the QEMU verison I
tested. I have highly optimised assembly stubs that implement a direct
mapped translation cache and translation cache miss fallback to C code that
does a fast hash map lookup.


> BTW after the merge you might want to look into optimizing indirect
> branches (and cross-page direct jumps in softmmu) for riscv in qemu.
> See examples with
>         $ git log -Stcg_gen_lookup_and_goto_ptr
>

It was qemu-2.7.50 (late 2016). The benchmarks were generated mid last year.

I can run the benchmarks again... Has it doubled in speed?

Note: I don't even have a register allocator. I've assigned RISC-V RVC
register to hard registers (compiler optimises to choose compressable
registers first) and the remainder are in spill slots that in many cases
can be embedded as memory operands in the x86_64 translation. i.e. no
explicit reload, we let the micro-architecture crack these into micro-ops,
as they help to keep up code density.

I think I can get close to double again with tiered optimization and a good
register allocator (lift RISC-V asm to SSA form). It's also a hotspot
interpreter, which is definately faster than compiling all code, as I
benchmarked it. It profiles and only translates hot paths, so code that
only runs a few iterations is not translated. When I did eager transaltion
I got a slow-down.

Re: [Qemu-devel] [patches] Re: [PULL] RISC-V QEMU Port Submission

[Michael Clark]

On Thu, Mar 1, 2018 at 9:40 AM, Michael Clark <mjc@sifive.com> wrote:

>
>
> On Thu, Mar 1, 2018 at 12:53 AM, Peter Maydell <peter.maydell@linaro.org>
> wrote:
>
>> On 28 February 2018 at 00:09, Michael Clark <mjc@sifive.com> wrote:
>> > I've just talked to SiFive about this. They have agreed that we can
>> remove
>> > the sifive_e300 and sifive_u500 boards from the patch series that we are
>> > going to submit upstream again later this week or early next week. These
>> > machines and their devices are pretty easy for us to maintain in the
>> riscv
>> > or a sifive repo. This trims the number of machines from 5 to 3 and
>> lets us
>> > remove the SiFiveUART and SiFivePRCI from the next patch series we are
>> > going to submit. e.g. v8
>>
>> Models of boards which people actively want and are using are fine
>> (though indeed you can save them for a later round of patches if you
>> like). And it sounds like the 1.9.1 stuff is genuinely wanted, so
>> thanks for the clarification there. What I want to avoid is boards
>> going into QEMU just because you happened to have them already. Once
>> a board model goes into QEMU it's a commitment to supporting it for
>> the long term, and getting rid of it again is hard.
>
>
> Most folk in the community at large are interested in the 'spike' and
> 'virt' machines. Well, the linux distributors are currently targetting
> 'virt' as it has working network and block storage.
>
> It's mostly SiFive customers that are interested in the SiFive machines.
>
> SiFive do intend to submit their machines upstream however we've decided
> that we want to review the naming of the machine/board/SOC in light of
> several new models, as there are infact more MCU/SOC models than are
> currently represented in the RISC-V QEMU port (E21, E31, E51, U54, U54MC).
> Those are the SOCs and then there are boards like the HiFive1, the LoFive,
> the HiFive Unleashed and several others (the 7th RISC-V Foundation Workshop
> had an electronic badge with the FE310 designed by antmicro). SiFive might
> like to review the naming and re-jig the SOC / board relationship.
> Presently we created two generic boards to represent the MMU-less E-series
> (sifive_e300) and the U-series (sifive_u500) however that doesn't cover the
> E21 and E51. As well as silicon, there are soft-cores, including soft-cores
> from other vendors (who have not yet submitted anything to the QEMU port).
> After reflecting on this we don't think the naming of the two sifive
> machines is quite right and they are not yet complete emulations. The sifive_u500
> is supposed to model the HiFive Unleashed however we don't have device
> emulation for all of the hardware widgets yet, and some of the drivers are
> not yet in upstream Linux. Linus essentially accepted the generic core,
> which is what we are now suggesting for QEMU. We are totally happy to
> submit them if folk don't mind us potentially renaming them later. This
> is 2 patches for 2 machines and 2 or 3 devices. It would reduce the patch
> series from 23 patches to 18 patches and we'd maintain the 5 files and
> associated headers out-of tree until things firm up. The core of the port
> is pretty solid as Fedora have build stage 4 images using SMP on the 'virt'
> machine.
>

We had internal discussions about the SiFive machines and SiFive decided to
rename SiFive E300 (sifive_e300.c) and SiFive U500 (sifive_u500.c) to
SiFive E Series (sifive_e.c) and SiFive U Series (sifive_u.c). This was the
result of synchronising with the product marketing folk.

- sifive_e will instantiate a e31 nommu core when run in qemu-system-riscv32
- sifive_e will instantiate a e51 nommu core when run in qemu-system-riscv64
- sifive_u will instantiate a u34 mmu core when run in qemu-system-riscv32
- sifive_u will instantiate a u54 mmu core when run in qemu-system-riscv64

The riscv port is interesting because the machines are designed to be
instantiated with either 32-bit or 64-bit cores. SiFive decided they wanted
to get this right before we contributed the SiFive boards. We think we now
have a model we are satisfied with submittting.

We did some last minute changes to strengthen the riscv-qemu port
specificaiton compliance. The nommu machines worked previously but the lack
of an mmu was not enforced. We've now disabled mmu on cores that lack mmu
(SiFive E series). We've also made some changes to strengthen specification
compliance. It is possible for a core to implement S mode and U mode
(Supervisor and User privilege mode) either with or without an mmu. We've
now matches the hardware behaviour and discard mstatus.mpp writes for modes
that are not supported by the core, making it not possible to enter S mode
on the E series cores. We are pretty confident that these are low risk
changes, and we have tested that Linux still works fine in the 'virt'
machine. The 'virt' machine is likely to be the most popular machine for
use by the Linux community.

Given SiFive has SOCs and Soft-core IP, we would like to in the future make
the sifive_u and sifive_e configurable with different memory maps and
hardware widgets, via runtime configuration (command line switches or
configuration file). The intention is for SiFive to be able to model custom
soft-core IP configurations in the  SiFive E series (Everywhere) SiFive U
series (Unleashed). The primary difference between these two machines is
that E series is a nommu MCU and U series is an application processor
capable of running Linux.


> > However I'm inclined to leave it as it is, at this point. It is not
>> > something that we can't change in the future once the code is in-tree.
>>
>
I've coalesced the two spike machines into spike.c and they now share
common code, with two different init routines, one privileged ISA v1.9.1
pre device-tree and one privileged ISA v1.10 with device-tree. We could
refactor further to eliminate some small sections of duplicated code,
although that is a relatively minor issue.

The spike machines are designed to match the riscv-isa-sim (spike)
behaviour and we have created retrospective tags for priv-1.9.1 as we will
be adding this to our test suite. SiFive is committed to maintaining binary
compatibility from privileged ISA v1.9.1 forwards, which in this case
requires command line switches to select the historical behaviour. We are
using this as the starting point of a discipline to not break backwards
compatibility.

With my 'upstream dev' hat on, I tend to be suspicious of this
>> line of argument, because in a lot of cases what tends to happen
>> is that the code for some new target or device goes in-tree, and
>> then the people who worked on submitting it disappear, or never
>> actually do get round to refactoring[*]. You get more leeway for
>> making this argument the longer you've been around and participating
>> in QEMU development, because then you have a track record of
>> following up on things.
>>
>> [*] in fact we're currently discussing deleting support for
>> a couple of target architectures that were basically "once the
>> code went into mainline nothing further was ever done to it except
>> global-refactorings and other tree wide maintenance by other
>> QEMU developers".
>
>
> I might explore coalescing the two spike boards before the v8 spin then.
>
> I was thinking about using a global to enable the backwards compatibility
> mode. It seems -global is for driver properties? Is it appropriate to use
> a global property to change the behaviour of a machine?
>
> Okay, I see raspi.c contains raspi, raspi 2, raspi 3, and it just defines
> multiple machines in the same file and passes a version number to routines
> that are shared. We could go with that approach.
>

I sent the patches for final review as we had been asked to add
Signed-off-by for all of the contributors, which is why I sent out a v8
patch series. The changes are relatively small and are primarily related to
adding the SiFive cores, and making sure that the -cpu help returns the
correct list of cores in qemu-system-riscv32 and qemu-system-riscv64. The
changes were made so that we could keep the two SiFive boards in the port,
so I kind of went out of my way to get these changes in before our final
patch submission.

I am working on some very minor cleanups that I'm working on in my own tree
after going over the code about 8 times:

- https://github.com/michaeljclark/riscv-qemu/commits/qemu-devel

I believe the riscv-qemu-upstream-v8 branch is good to pull. I can hold off
on any other changes for now and we can just still in a rebase loop,
although if we have to rebase again I might be inclined to include the
small set of cleanups that i've noticed on my 9th time over the code.

Apologies if the v8 pull request was premature, but I believe the series
has been rock solid since v6, and subsequent to that the changes have been
license changes, contributor sign offs and some last minute changes to make
it possible for SiFive to submit the SiFive boards upstream with the
correct naming, after internal review.

Thanks,
Michael.

Re: [Qemu-devel] [patches] Re: [PULL] RISC-V QEMU Port Submission

[Emilio G. Cota]

On Sat, Mar 03, 2018 at 02:26:12 +1300, Michael Clark wrote:
> It was qemu-2.7.50 (late 2016). The benchmarks were generated mid last year.
> 
> I can run the benchmarks again... Has it doubled in speed?

It depends on the benchmarks. Small-ish benchmarks such as rv8-bench
show about a 1.5x speedup since QEMU v2.6.0 for Aarch64:

                Aarch64 rv8-bench performance under QEMU user-mode              
                  Host: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz                
                                                                                
  4.5 +-+----+------+------+------+------+------+------+------+------+----+-+   
      |                                                 ++                  |   
    4 +-+..........v2.8.0.........v2.9.0........v2.10.0.%%.....v2.11.0....+-+   
  3.5 +-+...............................................%%@...............+-+   
      |                                                 %%@                 |   
    3 +-+...............................................%%@...............+-+   
  2.5 +-+...............................................%%@...............+-+   
      |                     ++                        $$$%@                 |   
    2 +-+.................$$$%@......................##.$%@...............+-+   
      |                  ##+$%@ ##$$%@               ## $%@                 |   
  1.5 +-+..+++%%@.......**#.$%@.##.$%@++++%%@........##.$%@........##$$%@.+-+   
    1 +-+.**#$$%@+##$$%@**#.$%@**#.$%@**#$$%@**#$$%@**#.$%@**#$$%@**#.$%@.+-+   
      |   **# $%@**# $%@**# $%@**# $%@**# $%@**#+$%@**# $%@**# $%@**# $%@   |   
  0.5 +-+-**#$$%@**#$$%@**#$$%@**#$$%@**#$$%@**#$$%@**#$$%@**#$$%@**#$$%@-+-+   
          aes bigidhrystone  miniz   norx primes  qsort sha512geomean           
  png: https://imgur.com/Agr5CJd

SPEC06int shows a larger improvement, up to ~2x avg speedup for the train
set:
          Aarch64 SPEC06int (train set) performance under QEMU user-mode        
                  Host: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz                
                                                                                
    4 +-+--+----+----+----+----+----+----+----+----+----+----+----+----+--+-+   
      |    %%                                           ++                  |   
  3.5 +-+..%%@.....v2.8.0.........v2.9.0........v2.10.0.%%+....v2.11.0....+-+   
      |    %%@                                          %%@       ++        |   
    3 +-+..%%@.......++.................................%%@.......%%+.....+-+   
      |   +$%@        |+                                %%@       %%@       |   
  2.5 +-+.##%@.......%%@...............................+$%@.......%%@.....+-+   
    2 +-+.##%@......+%%@.......%%@.................%%@.##%@.......%%@..++.+-+   
      |   ##%@  %%@ ##%@      +$%@       %%@       %%@ ##%@       $%@  %%@  |   
  1.5 +-+**#%@.##%@.##%@......##%@.......$%@.......$%@.##%@..%%+.##%@.##%@+-+   
      |  **#%@**#%@**#%@  +++**#%@      ##%@  ++  ##%@**#%@ ##%@ ##%@ ##%@  |   
    1 +-+**#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@+##%@**#%@+##%@**#%@**#%@+-+   
      |  **#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@  |   
  0.5 +-+**#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@+-+   
       401.bzi403.g429445.g456.h462.libq464.h471.omn4483.xalancbgeomean         
  png: https://imgur.com/JknVT5H

Note that the test set is less sensitive to the changes:
  https://imgur.com/W7CT0eO

Running small benchmarks (such as SPEC "test" or rv8-bench) is
very useful to get quick feedback on optimizations. However, some
of these runs are still dominated by parts of the code that aren't
that relevant -- for instance, some of them take so little time to
run that the major contributor to execution time is memory allocation.
Therefore, when publishing results it's best to stick with larger
benchmarks that run for longer (e.g. SPEC "train" set), which are more
sensitive to DBT performance.

I tried running some other benchmarks, such as nbench[1], under rv-jit.
I quickly get a "bus error" though -- don't know if I'm doing anything
wrong, or maybe compiling with the glibc cross-compiler I used
to build riscv linux isn't supported.
I managed though to run rv8-bench on both rv-jit and qemu (v8 patchset);
rv-jit is 1.30x faster on average for those, although note I dropped
qsort because it wasn't working properly on rv-jit:

               rv8-bench performance under rv-jit and QEMU user-mode            
                  Host: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz                
             [qsort does not finish cleanly for rv8, so I dropped it.]          
                                                                                
    3 +-+-----+-------+------+-------+-------+-------+------+-------+-----+-+   
  2.5 +-+..................*****..........................................+-+   
      |                    *-+-* b1bae23b7c2                                |   
    2 +-+..................*...*...................+-+-+..................+-+   
  1.5 +-+...........*****..*...*...................*****..................+-+   
      |     *****   *-+-*  *   *   *****           *   *   ++-+   *****     |   
    1 +-+...*-+-*...*...*..*...*...*...*...*****...*...*...****...*...*...+-+   
  0.5 +-+---*****---*****--*****---*****---*****---*****---****---*****---+-+   
           aes  bigidhrystone   miniz    norx  primes sha512 geomean            
  png: https://imgur.com/rLmTH3L

> I think I can get close to double again with tiered optimization and a good
> register allocator (lift RISC-V asm to SSA form). It's also a hotspot
> interpreter, which is definately faster than compiling all code, as I
> benchmarked it. It profiles and only translates hot paths, so code that
> only runs a few iterations is not translated. When I did eager transaltion
> I got a slow-down.

Yes, hotspot is great for real-life workloads (e.g. booting a system). Note
though that most benchmarks (e.g. SPEC) don't translate code that often;
most execution time is spent in loops and therefore the quality of
the generated code does matter. Hotspot detection of TBs/traces is great
for this as well, because it allows you to spend more resources generating
higher-quality code--for instance, see HQEMU[2].

Thanks,

		Emilio

[1] https://github.com/cota/nbench
[2] http://www.iis.sinica.edu.tw/papers/dyhong/18243-F.pdf
PS. One page with all the png's: https://imgur.com/a/5P5zj

Re: [Qemu-devel] [patches] Re: [PULL] RISC-V QEMU Port Submission

[Michael Clark]

On Tue, Mar 6, 2018 at 8:00 AM, Emilio G. Cota <cota@braap.org> wrote:

> On Sat, Mar 03, 2018 at 02:26:12 +1300, Michael Clark wrote:
> > It was qemu-2.7.50 (late 2016). The benchmarks were generated mid last
> year.
> >
> > I can run the benchmarks again... Has it doubled in speed?
>
> It depends on the benchmarks. Small-ish benchmarks such as rv8-bench
> show about a 1.5x speedup since QEMU v2.6.0 for Aarch64:
>
>                 Aarch64 rv8-bench performance under QEMU user-mode
>                   Host: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
>
>   4.5 +-+----+------+------+------+------+------+------+------+---
> ---+----+-+
>       |                                                 ++
>   |
>     4 +-+..........v2.8.0.........v2.9.0........v2.10.0.%%.....
> v2.11.0....+-+
>   3.5 +-+...............................................%%@.......
> ........+-+
>       |                                                 %%@
>  |
>     3 +-+...............................................%%@.......
> ........+-+
>   2.5 +-+...............................................%%@.......
> ........+-+
>       |                     ++                        $$$%@
>  |
>     2 +-+.................$$$%@......................##.$%@.......
> ........+-+
>       |                  ##+$%@ ##$$%@               ## $%@
>  |
>   1.5 +-+..+++%%@.......**#.$%@.##.$%@++++%%@........##.$%@.......
> .##$$%@.+-+
>     1 +-+.**#$$%@+##$$%@**#.$%@**#.$%@**#$$%@**#$$%@**#.$%@**#$$%@
> **#.$%@.+-+
>       |   **# $%@**# $%@**# $%@**# $%@**# $%@**#+$%@**# $%@**# $%@**# $%@
>  |
>   0.5 +-+-**#$$%@**#$$%@**#$$%@**#$$%@**#$$%@**#$$%@**#$$%@**#$$%@
> **#$$%@-+-+
>           aes bigidhrystone  miniz   norx primes  qsort sha512geomean
>   png: https://imgur.com/Agr5CJd
>
> SPEC06int shows a larger improvement, up to ~2x avg speedup for the train
> set:
>           Aarch64 SPEC06int (train set) performance under QEMU user-mode
>                   Host: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
>
>     4 +-+--+----+----+----+----+----+----+----+----+----+----+----
> +----+--+-+
>       |    %%                                           ++
>   |
>   3.5 +-+..%%@.....v2.8.0.........v2.9.0........v2.10.0.%%+....
> v2.11.0....+-+
>       |    %%@                                          %%@       ++
>   |
>     3 +-+..%%@.......++.................................%%@.......
> %%+.....+-+
>       |   +$%@        |+                                %%@       %%@
>  |
>   2.5 +-+.##%@.......%%@...............................+$%@.......
> %%@.....+-+
>     2 +-+.##%@......+%%@.......%%@.................%%@.##%@.......
> %%@..++.+-+
>       |   ##%@  %%@ ##%@      +$%@       %%@       %%@ ##%@       $%@
> %%@  |
>   1.5 +-+**#%@.##%@.##%@......##%@.......$%@.......$%@.##%@..%%+.#
> #%@.##%@+-+
>       |  **#%@**#%@**#%@  +++**#%@      ##%@  ++  ##%@**#%@ ##%@ ##%@
> ##%@  |
>     1 +-+**#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@+##%@**#%@+##%@**
> #%@**#%@+-+
>       |  **#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@
> |
>   0.5 +-+**#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@**#%@**
> #%@**#%@+-+
>        401.bzi403.g429445.g456.h462.libq464.h471.omn4483.xalancbgeomean
>   png: https://imgur.com/JknVT5H
>
> Note that the test set is less sensitive to the changes:
>   https://imgur.com/W7CT0eO
>
> Running small benchmarks (such as SPEC "test" or rv8-bench) is
> very useful to get quick feedback on optimizations. However, some
> of these runs are still dominated by parts of the code that aren't
> that relevant -- for instance, some of them take so little time to
> run that the major contributor to execution time is memory allocation.
> Therefore, when publishing results it's best to stick with larger
> benchmarks that run for longer (e.g. SPEC "train" set), which are more
> sensitive to DBT performance.
>
> I tried running some other benchmarks, such as nbench[1], under rv-jit.
> I quickly get a "bus error" though -- don't know if I'm doing anything
> wrong, or maybe compiling with the glibc cross-compiler I used
> to build riscv linux isn't supported.
> I managed though to run rv8-bench on both rv-jit and qemu (v8 patchset);
> rv-jit is 1.30x faster on average for those, although note I dropped
> qsort because it wasn't working properly on rv-jit:
>

That's interesting. I know from some analysis that the current slow-down in
rv8 is mostly from accessing statically spilled registers (which in many
cases we embed in x86 memory operands to keep up code density, and make use
of the instruction cracker and uop cache in Intel's front-end). The
slowdown is mostly L1 cache latency vs register access latency given we are
emulating 31 registers on a 16 register host with a static register
allocation (based on the compiler register allocation order which optimizes
for the RVC accessible registers). With the addition of a register
allocator, I am sure I can make rv8 substantially faster. perhaps 1.7x

The user-mode emulation in rv8 is very limited, and so far has been
targetted at running rv8-bench compiled with musl-riscv-toolchain
(musl-libc). It has also been tested somewhat with newlib.

- https://github.com/rv8-io/musl-riscv
- https://github.com/rv8-io/musl-riscv-toolchain

I haven't really tested glibc.  It is what I would call a late stage
proof-of-concept, research simulator.

The user-mode simulator was a good way to bring-up the easy part of the
JIT. The next step is register allocation with a hot-spot tiered
optizimation strategy. i.e. interp -> T1 -> T2, where T2 lifts the RISC-V
code to SSA form and does register allocation, and inter-trace swaps based
on the traces entry and exit live register mappings.

When I get time I'd like to implement hardmmu emulation of the RISC-V
privileged ISA. I have a plan to use CR4.PCIDE and PCID to run M, S and U
mode in Ring 3 with different address space IDs. We can take advantage of
various architectural optimisations that are more difficult with a
multi-target translator. i.e. my intended baseline for the (vapourware)
privileged mode translator is Broadwell. i.e. x86_64 + CR4.PCIDE = 1. I
should be able to emulate RISC-V ASID using a LRU/MRU for ASID. I'll also
emulate sv39 page tables, but use 4 level page tables in the first level
page tables of the x86_64 host (not EPT), so that the translator can live
in address space unreachable by the guest, and also so that the translator
can run as a regular guest, in fact inside qemu/kvm for x86_64. i.e. so I
don't need to futz around with Hyperkit/HyperV/HAX/KVM, etc. It will just
be a kernel that can load an RV32/RV64 boot loader in privileged mode.

The problem is SiFive want me to spend most of my time on QEMU, so it's a
weekend project, however most weekends recently have been spent on the
RISC-V QEMU port.

I do hope I have time to spend on rv8 in the future. I will re-run the
rv8-bench suite with a more recent version of QEMU and upload the results.
It could also be useful to track performance differences between different
versions of QEMU. I've automated the generation of charts and tables so it
is pretty easy for me to regenerate the results with new compiler versions
and emulator versions.

               rv8-bench performance under rv-jit and QEMU user-mode
>                   Host: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
>              [qsort does not finish cleanly for rv8, so I dropped it.]
>
>     3 +-+-----+-------+------+-------+-------+-------+------+-----
> --+-----+-+
>   2.5 +-+..................*****..................................
> ........+-+
>       |                    *-+-* b1bae23b7c2
>   |
>     2 +-+..................*...*...................+-+-+..........
> ........+-+
>   1.5 +-+...........*****..*...*...................*****..........
> ........+-+
>       |     *****   *-+-*  *   *   *****           *   *   ++-+   *****
>  |
>     1 +-+...*-+-*...*...*..*...*...*...*...*****...*...*...****...
> *...*...+-+
>   0.5 +-+---*****---*****--*****---*****---*****---*****---****---
> *****---+-+
>            aes  bigidhrystone   miniz    norx  primes sha512 geomean
>   png: https://imgur.com/rLmTH3L
>
> > I think I can get close to double again with tiered optimization and a
> good
> > register allocator (lift RISC-V asm to SSA form). It's also a hotspot
> > interpreter, which is definately faster than compiling all code, as I
> > benchmarked it. It profiles and only translates hot paths, so code that
> > only runs a few iterations is not translated. When I did eager
> transaltion
> > I got a slow-down.
>
> Yes, hotspot is great for real-life workloads (e.g. booting a system). Note
> though that most benchmarks (e.g. SPEC) don't translate code that often;
> most execution time is spent in loops and therefore the quality of
> the generated code does matter. Hotspot detection of TBs/traces is great
> for this as well, because it allows you to spend more resources generating
> higher-quality code--for instance, see HQEMU[2].
>
> Thanks,
>
>                 Emilio
>
> [1] https://github.com/cota/nbench
> [2] http://www.iis.sinica.edu.tw/papers/dyhong/18243-F.pdf
> PS. One page with all the png's: https://imgur.com/a/5P5zj
>
>
Regards,
Michael.